2024-03-06T07:07:02.071 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:07:02.071 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042 --verbose --description fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} --name leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/orig.config.yaml 2024-03-06T07:07:02.122 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:07:02.206 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042 branch: wip-lusov-qdb description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} email: null first_in_suite: false job_id: '7583042' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MDS_DAMAGE\) - \(FS_DEGRADED\) sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: disabled: true install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUptUXOT7IP+rlzPS9z+dJBf1CjRETBgacSwiKFjPRRFP2vQO1vvZ/eNWBQcHv29vpje+bcaZkurfh9o6Rui0E= smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLs9x8j8hoye4v7cxEJMQ6KkZsfY8EpCDIT4JcwJqJWa2nvNoegFCUiOVIwOGvganVWn+4aJUZjR+AISPQXVEc4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_failover teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:00:27 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:07:02.207 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:07:02.207 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:07:02.207 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:07:02.209 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:07:02.233 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:07:02.233 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:07:02.233 INFO:teuthology.packaging:ref: None 2024-03-06T07:07:02.233 INFO:teuthology.packaging:tag: None 2024-03-06T07:07:02.233 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:07:02.233 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:07:02.234 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:07:02.521 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:07:02.523 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:07:02.532 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:07:02.532 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:07:02.540 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:07:02.556 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:07:02.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:07:02.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:53.122975', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUptUXOT7IP+rlzPS9z+dJBf1CjRETBgacSwiKFjPRRFP2vQO1vvZ/eNWBQcHv29vpje+bcaZkurfh9o6Rui0E='} 2024-03-06T07:07:02.612 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:53.124550', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLs9x8j8hoye4v7cxEJMQ6KkZsfY8EpCDIT4JcwJqJWa2nvNoegFCUiOVIwOGvganVWn+4aJUZjR+AISPQXVEc4='} 2024-03-06T07:07:02.612 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:07:02.618 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-06T07:07:02.618 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-03-06T07:07:02.618 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:07:02.688 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f96c4a520d0>, signals=[15]) 2024-03-06T07:07:02.688 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:07:02.696 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:07:02.696 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:07:02.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:07:02.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:07:02.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:07:02.876 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:07:02.883 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-03-06T07:07:02.900 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-03-06T07:07:02.900 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-03-06T07:07:02.956 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-03-06T07:07:02.956 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-03-06T07:07:02.956 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-03-06T07:07:02.956 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:07:02.956 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:07:02.957 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:07:02.958 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-03-06T07:07:02.980 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-03-06T07:07:02.998 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-03-06T07:07:02.998 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="9" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="9" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:07:03.055 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:07:03.056 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:07:03.056 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:07:03.056 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-03-06T07:07:03.080 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:07:03.091 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:07:03.124 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:07:03.124 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:07:03.126 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:07:03.142 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:07:03.149 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:07:03.149 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:07:03.184 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:07:03.294 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:07:03.355 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:07:03.370 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-06T07:07:03.370 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2024-03-06T07:07:03.370 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-03-06T07:07:03.371 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:07:03.371 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:07:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:07:03.388 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-06T07:07:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:07:03.389 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-06T07:07:03.445 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:07:03.445 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-361.el9.x86_64 2024-03-06T07:07:03.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-03-06T07:07:03.446 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:07:03.446 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.14.0-361.el9.x86_64 2024-03-06T07:07:03.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2024-03-06T07:07:06.045 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-03-06T07:07:06.061 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-03-06T07:07:07.470 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-06T07:07:07.754 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-06T07:07:09.053 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.0 MB 00:00 2024-03-06T07:07:09.178 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-06T07:07:14.661 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 4.2 MB/s | 19 MB 00:04 2024-03-06T07:07:18.620 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-06T07:07:20.229 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-03-06T07:07:21.950 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-06T07:07:26.219 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-06T07:07:26.938 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-06T07:07:29.162 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:07:29.162 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:07:29.263 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-06T07:07:29.271 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-06T07:07:29.271 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2024-03-06T07:07:29.272 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:07:29.273 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 100 M 2024-03-06T07:07:29.273 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 123 M 2024-03-06T07:07:29.273 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-06T07:07:30.279 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-06T07:07:30.964 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.9 MB/s | 6.0 MB 00:01 2024-03-06T07:07:32.738 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-06T07:07:33.848 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-06T07:07:38.815 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-06T07:07:41.089 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:07:41.089 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:07:41.237 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-06T07:07:41.247 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-06T07:07:41.247 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-06T07:07:41.247 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-03-06T07:07:41.248 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:07:41.249 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 100 M 2024-03-06T07:07:41.249 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 123 M 2024-03-06T07:07:41.249 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-06T07:07:41.834 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-03-06T07:07:43.584 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 9.1 MB/s | 21 MB 00:02 2024-03-06T07:07:46.888 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.2 MB/s | 21 MB 00:17 2024-03-06T07:07:47.017 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 6.4 MB/s | 33 MB 00:05 2024-03-06T07:07:52.669 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 3.5 MB/s | 39 MB 00:11 2024-03-06T07:07:52.670 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:07:52.670 INFO:teuthology.orchestra.run.smithi094.stdout:Total 8.7 MB/s | 100 MB 00:11 2024-03-06T07:07:53.552 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-06T07:07:53.935 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-06T07:07:53.935 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-06T07:07:55.581 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-06T07:07:55.581 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-06T07:07:58.532 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-06T07:07:59.220 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:07:59.373 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:00.178 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 1.1 MB/s | 33 MB 00:29 2024-03-06T07:08:01.236 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 1.2 MB/s | 39 MB 00:31 2024-03-06T07:08:01.237 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:08:01.237 INFO:teuthology.orchestra.run.smithi145.stdout:Total 3.1 MB/s | 100 MB 00:31 2024-03-06T07:08:02.117 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-06T07:08:02.226 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:02.422 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:02.496 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-06T07:08:02.496 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-06T07:08:03.857 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-06T07:08:03.857 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-06T07:08:06.808 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-06T07:08:07.198 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:07.325 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:07.560 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:08:07.695 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:10.528 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:12.097 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:15.417 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:15.608 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:20.379 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:41.285 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:41.285 INFO:teuthology.orchestra.run.smithi094.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:08:41.286 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:08:41.370 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:41.842 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:41.842 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:08:41.842 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:41.842 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:42.255 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:42.255 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:08:42.255 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-06T07:08:42.255 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:42.255 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:08:42.256 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:08:42.256 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:08:42.256 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:08:42.256 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-06T07:08:42.767 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:08:42.923 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:42.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:08:42.924 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:08:42.925 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:08:42.925 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:08:42.925 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-06T07:08:42.925 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:08:42.925 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:08:43.043 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:43.044 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:08:43.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-03-06T07:08:43.083 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-03-06T07:08:43.083 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-03-06T07:08:43.083 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-03-06T07:08:43.083 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 68.el9 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8264254 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:08:43.084 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:08:43.085 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:08:43.086 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:08:43.086 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:08:43.086 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:08:43.777 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-03-06T07:08:45.186 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:08:45.212 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-03-06T07:08:45.214 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-06T07:08:45.229 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.iNi8YAYB5Y 2024-03-06T07:08:45.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iNi8YAYB5Y 2024-03-06T07:08:45.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.iNi8YAYB5Y 2024-03-06T07:08:45.440 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.iNi8YAYB5Y is 6KB 2024-03-06T07:08:45.488 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.iNi8YAYB5Y 2024-03-06T07:08:45.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:08:45.567 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:08:45.567 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:08:45.567 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:08:45.568 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:08:45.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:08:45.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-03-06T07:08:49.979 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:49.979 INFO:teuthology.orchestra.run.smithi145.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:08:49.979 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:08:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:08:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:08:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:08:50.891 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-06T07:08:51.420 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:08:51.563 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:51.564 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:08:51.564 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:08:51.564 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:08:51.564 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:08:51.564 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-03-06T07:08:51.565 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:08:51.565 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:08:51.686 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:08:51.686 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:08:51.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2024-03-06T07:08:51.725 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.06 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 68.el9 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Group : Unspecified 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 8264254 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:08:51.726 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : builder@centos.org 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:08:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:08:51.728 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:08:51.728 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:08:51.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:08:52.339 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-03-06T07:08:53.692 INFO:teuthology.orchestra.run.smithi145.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:08:53.724 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-03-06T07:08:53.727 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-06T07:08:53.741 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.UjHSh5KHWH 2024-03-06T07:08:53.741 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UjHSh5KHWH 2024-03-06T07:08:53.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.UjHSh5KHWH 2024-03-06T07:08:53.948 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.UjHSh5KHWH is 6KB 2024-03-06T07:08:53.996 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.UjHSh5KHWH 2024-03-06T07:08:54.010 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:08:54.074 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:08:54.074 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:08:54.074 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:08:54.074 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:08:54.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:08:54.228 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-03-06T07:09:15.736 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-06T07:09:15.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:09:15.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:24.238 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-06T07:09:24.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-06T07:09:24.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:34.141 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-03-06T07:09:42.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-06T07:09:43.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:09:43.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:46.205 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-03-06T07:09:51.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-06T07:09:51.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:54.652 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-06T07:09:58.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:09:58.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:01.276 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-03-06T07:10:06.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-06T07:10:06.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:16.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:10:16.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:19.356 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2024-03-06T07:10:37.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:10:37.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:37.776 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-06T07:10:38.168 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:10:38.169 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:10:38.169 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-06T07:10:38.185 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:10:38.185 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:10:38.185 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:10:38.185 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:10:38.498 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-06T07:10:38.916 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-06T07:10:38.916 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:10:38.917 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-06T07:10:38.930 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:10:38.930 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:10:38.930 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:10:38.931 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:10:39.187 DEBUG:teuthology.parallel:result is None 2024-03-06T07:10:39.931 DEBUG:teuthology.parallel:result is None 2024-03-06T07:10:39.931 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:10:39.938 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:10:39.938 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:10:39.941 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:10:39.956 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:10:39.962 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:10:39.972 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:10:39.972 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:10:39.998 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:10:40.048 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:10:40.055 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:10:40.055 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:10:40.074 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:10:40.149 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:10:40.191 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:10:40.204 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:10:40.561 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:10:40.563 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:10:40.570 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:10:40.570 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:10:40.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:10:40.624 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:10:40.633 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:10:40.634 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:10:40.644 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:10:40.670 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:10:40.745 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:10:40.805 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:10:40.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:10:40.892 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:10:40.892 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:10:40.960 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:10:41.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:10:41.081 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:10:41.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:10:41.177 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:10:41.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:10:41.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-03-06T07:10:41.247 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-03-06T07:10:41.297 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:10:41.332 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:10:41.807 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:10:41.815 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:10:41.815 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:10:41.825 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:10:41.833 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-03-06T07:10:41.899 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-03-06T07:10:41.901 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2024-03-06T07:10:41.987 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2024-03-06T07:10:41.989 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:10:41.989 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-03-06T07:10:42.009 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-03-06T07:10:42.009 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2024-03-06T07:10:42.030 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2024-03-06T07:10:42.031 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi145.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:10:42.031 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:10:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:42.078 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:10:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:42.105 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:10:42.105 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:10:42.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:10:42.199 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:10:42.209 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:10:42.300 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:10:42.365 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:14:57.282 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2024-03-06T07:14:57.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:14:57.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:14:57.380 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-06T07:14:57.458 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-06T07:14:57.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-06T07:14:57.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:14:57.555 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-06T07:14:57.635 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-06T07:14:57.636 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:14:57.648 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:14:57.648 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:14:57.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:14:57.651 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:14:57.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:14:57.680 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:14:57.696 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:14:57.709 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:14:57.725 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-03-06T07:14:57.726 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:14:57.739 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-03-06T07:14:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:14:57.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2024-03-06T07:14:57.770 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2024-03-06T07:14:57.771 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:14:57.786 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:14:57.803 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:14:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:14:57.830 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:14:57.830 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-06T07:14:57.830 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.830 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.830 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.830 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.857 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:14:57.861 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:14:57.861 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-06T07:14:57.861 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.861 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.861 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.861 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:14:57.862 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:14:57.871 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:14:57.871 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:14:57.871 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:14:57.871 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:14:57.877 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:14:57.877 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:14:57.878 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:14:57.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:14:57.880 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:14:57.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:14:58.038 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:14:58.038 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:14:58.042 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:14:58.043 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:14:58.433 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:14:58.433 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:14:58.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:14:58.441 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:14:58.441 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:14:58.441 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:14:58.462 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:14:58.463 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:14:58.470 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:14:58.470 DEBUG:teuthology.orchestra.run.smithi145:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:14:58.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:14:58.539 DEBUG:teuthology.orchestra.run.smithi145:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:14:58.612 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:14:58.620 DEBUG:teuthology.orchestra.run.smithi145:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:14:58.679 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-03-06T07:14:58.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-03-06T07:14:58.687 INFO:teuthology.orchestra.run.smithi145.stdout:check_obsoletes = 1 2024-03-06T07:14:58.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum clean all 2024-03-06T07:14:58.928 INFO:teuthology.orchestra.run.smithi094.stdout:74 files removed 2024-03-06T07:14:58.928 INFO:teuthology.orchestra.run.smithi145.stdout:74 files removed 2024-03-06T07:14:58.954 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:14:58.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:15:00.173 INFO:teuthology.orchestra.run.smithi145.stdout:ceph packages for x86_64 92 kB/s | 83 kB 00:00 2024-03-06T07:15:00.324 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 79 kB/s | 83 kB 00:01 2024-03-06T07:15:00.890 INFO:teuthology.orchestra.run.smithi145.stdout:ceph noarch packages 15 kB/s | 10 kB 00:00 2024-03-06T07:15:01.159 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-06T07:15:01.732 INFO:teuthology.orchestra.run.smithi145.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-06T07:15:01.925 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-06T07:15:02.318 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:15:02.376 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-06T07:15:03.718 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-06T07:15:03.760 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-06T07:15:04.576 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-06T07:15:07.800 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-06T07:15:20.569 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-06T07:15:22.550 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-06T07:15:24.885 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.1 MB 00:00 2024-03-06T07:15:26.236 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 15 kB 00:00 2024-03-06T07:15:27.096 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-03-06T07:15:28.210 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-03-06T07:15:29.861 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-06T07:15:30.996 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-06T07:15:32.021 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-06T07:15:34.557 INFO:teuthology.orchestra.run.smithi145.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:15:34.558 INFO:teuthology.orchestra.run.smithi145.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:15:34.572 INFO:teuthology.orchestra.run.smithi145.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:15:34.573 INFO:teuthology.orchestra.run.smithi145.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:15:34.574 INFO:teuthology.orchestra.run.smithi145.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:15:34.575 INFO:teuthology.orchestra.run.smithi145.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:15:34.577 INFO:teuthology.orchestra.run.smithi145.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:15:34.580 INFO:teuthology.orchestra.run.smithi145.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:15:34.581 INFO:teuthology.orchestra.run.smithi145.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:15:34.660 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-06T07:15:34.677 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-06T07:15:34.677 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-06T07:15:34.677 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:15:34.678 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:15:34.679 INFO:teuthology.orchestra.run.smithi145.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:15:34.680 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:15:34.680 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:15:34.680 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:15:34.680 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:15:34.680 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:15:34.680 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:15:34.681 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:15:34.681 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:15:34.681 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:15:34.682 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:15:34.683 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:15:34.684 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:15:34.689 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:15:34.690 INFO:teuthology.orchestra.run.smithi145.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:15:34.691 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:15:34.692 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:15:34.692 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:15:34.693 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:15:34.694 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:15:34.695 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:15:34.696 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:15:34.697 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout:Installing weak dependencies: 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-06T07:15:34.698 INFO:teuthology.orchestra.run.smithi145.stdout:Install 150 Packages 2024-03-06T07:15:34.699 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 26 Packages 2024-03-06T07:15:34.699 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:15:34.699 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 224 M 2024-03-06T07:15:34.699 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-06T07:15:35.197 INFO:teuthology.orchestra.run.smithi145.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-03-06T07:15:35.970 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-06T07:15:36.164 INFO:teuthology.orchestra.run.smithi145.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 919 kB/s | 888 kB 00:00 2024-03-06T07:15:36.324 INFO:teuthology.orchestra.run.smithi145.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 918 kB/s | 145 kB 00:00 2024-03-06T07:15:37.431 INFO:teuthology.orchestra.run.smithi145.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 2.1 MB/s | 2.3 MB 00:01 2024-03-06T07:15:38.190 INFO:teuthology.orchestra.run.smithi145.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.0 MB/s | 1.5 MB 00:00 2024-03-06T07:15:38.574 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:15:38.575 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:15:38.588 INFO:teuthology.orchestra.run.smithi094.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:15:38.588 INFO:teuthology.orchestra.run.smithi094.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:15:38.589 INFO:teuthology.orchestra.run.smithi094.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:15:38.590 INFO:teuthology.orchestra.run.smithi094.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:15:38.592 INFO:teuthology.orchestra.run.smithi094.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:15:38.594 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:15:38.595 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:15:38.665 INFO:teuthology.orchestra.run.smithi145.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.4 MB/s | 5.3 MB 00:03 2024-03-06T07:15:38.676 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-06T07:15:38.693 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-06T07:15:38.693 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-06T07:15:38.693 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:15:38.694 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:15:38.695 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-03-06T07:15:38.696 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:15:38.697 INFO:teuthology.orchestra.run.smithi094.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:15:38.698 INFO:teuthology.orchestra.run.smithi094.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:15:38.699 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:15:38.700 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:15:38.701 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:15:38.702 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:15:38.703 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:15:38.704 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:15:38.705 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:15:38.706 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:15:38.707 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:15:38.707 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:15:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:15:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:15:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:15:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:15:38.709 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-03-06T07:15:38.709 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:15:38.710 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:15:38.710 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-06T07:15:38.710 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-06T07:15:38.710 INFO:teuthology.orchestra.run.smithi094.stdout:Install 150 Packages 2024-03-06T07:15:38.710 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 26 Packages 2024-03-06T07:15:38.710 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:15:38.710 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 224 M 2024-03-06T07:15:38.711 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-06T07:15:39.355 INFO:teuthology.orchestra.run.smithi094.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-06T07:15:39.782 INFO:teuthology.orchestra.run.smithi145.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 2.9 MB/s | 4.7 MB 00:01 2024-03-06T07:15:40.571 INFO:teuthology.orchestra.run.smithi094.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 730 kB/s | 888 kB 00:01 2024-03-06T07:15:40.772 INFO:teuthology.orchestra.run.smithi094.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 726 kB/s | 145 kB 00:00 2024-03-06T07:15:42.440 INFO:teuthology.orchestra.run.smithi145.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 3.7 MB/s | 9.8 MB 00:02 2024-03-06T07:15:42.515 INFO:teuthology.orchestra.run.smithi145.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 336 kB/s | 25 kB 00:00 2024-03-06T07:15:43.363 INFO:teuthology.orchestra.run.smithi094.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 913 kB/s | 2.3 MB 00:02 2024-03-06T07:15:44.948 INFO:teuthology.orchestra.run.smithi145.stdout:(10/176): ceph-common-19.0.0-1739.g1bc250c7.el9 2.0 MB/s | 20 MB 00:10 2024-03-06T07:15:45.190 INFO:teuthology.orchestra.run.smithi145.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 913 kB/s | 220 kB 00:00 2024-03-06T07:15:45.282 INFO:teuthology.orchestra.run.smithi145.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 295 kB/s | 27 kB 00:00 2024-03-06T07:15:45.366 INFO:teuthology.orchestra.run.smithi145.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 375 kB/s | 31 kB 00:00 2024-03-06T07:15:45.447 INFO:teuthology.orchestra.run.smithi094.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 734 kB/s | 1.5 MB 00:02 2024-03-06T07:15:45.891 INFO:teuthology.orchestra.run.smithi145.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.4 MB/s | 747 kB 00:00 2024-03-06T07:15:46.058 INFO:teuthology.orchestra.run.smithi145.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 981 kB/s | 163 kB 00:00 2024-03-06T07:15:46.233 INFO:teuthology.orchestra.run.smithi145.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 724 kB/s | 127 kB 00:00 2024-03-06T07:15:46.272 INFO:teuthology.orchestra.run.smithi094.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 743 kB/s | 5.3 MB 00:07 2024-03-06T07:15:46.608 INFO:teuthology.orchestra.run.smithi145.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 1.3 MB/s | 501 kB 00:00 2024-03-06T07:15:49.991 INFO:teuthology.orchestra.run.smithi145.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.5 MB/s | 5.1 MB 00:03 2024-03-06T07:15:50.083 INFO:teuthology.orchestra.run.smithi145.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 493 kB/s | 45 kB 00:00 2024-03-06T07:15:50.163 INFO:teuthology.orchestra.run.smithi094.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 1.0 MB/s | 4.7 MB 00:04 2024-03-06T07:15:50.225 INFO:teuthology.orchestra.run.smithi145.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 927 kB/s | 131 kB 00:00 2024-03-06T07:15:50.384 INFO:teuthology.orchestra.run.smithi145.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.0 MB/s | 162 kB 00:00 2024-03-06T07:15:50.759 INFO:teuthology.orchestra.run.smithi145.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 856 kB/s | 321 kB 00:00 2024-03-06T07:15:51.051 INFO:teuthology.orchestra.run.smithi145.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 1.0 MB/s | 298 kB 00:00 2024-03-06T07:15:51.209 INFO:teuthology.orchestra.run.smithi145.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 627 kB/s | 99 kB 00:00 2024-03-06T07:15:51.302 INFO:teuthology.orchestra.run.smithi145.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 929 kB/s | 85 kB 00:00 2024-03-06T07:15:52.910 INFO:teuthology.orchestra.run.smithi145.stdout:(26/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 4.6 MB/s | 47 MB 00:10 2024-03-06T07:15:52.985 INFO:teuthology.orchestra.run.smithi145.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 2.2 MB/s | 171 kB 00:00 2024-03-06T07:15:53.060 INFO:teuthology.orchestra.run.smithi145.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 420 kB/s | 31 kB 00:00 2024-03-06T07:15:53.127 INFO:teuthology.orchestra.run.smithi145.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 2.0 MB/s | 138 kB 00:00 2024-03-06T07:15:53.494 INFO:teuthology.orchestra.run.smithi145.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 4.9 MB/s | 1.8 MB 00:00 2024-03-06T07:15:53.797 INFO:teuthology.orchestra.run.smithi094.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 1.4 MB/s | 20 MB 00:14 2024-03-06T07:15:53.905 INFO:teuthology.orchestra.run.smithi094.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 233 kB/s | 25 kB 00:00 2024-03-06T07:15:54.045 INFO:teuthology.orchestra.run.smithi145.stdout:(31/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 3.1 MB 00:02 2024-03-06T07:15:54.286 INFO:teuthology.orchestra.run.smithi145.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.0 MB/s | 248 kB 00:00 2024-03-06T07:15:54.362 INFO:teuthology.orchestra.run.smithi145.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 657 kB/s | 49 kB 00:00 2024-03-06T07:15:54.454 INFO:teuthology.orchestra.run.smithi145.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 167 kB/s | 15 kB 00:00 2024-03-06T07:15:54.679 INFO:teuthology.orchestra.run.smithi145.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.1 MB/s | 264 kB 00:00 2024-03-06T07:15:54.997 INFO:teuthology.orchestra.run.smithi094.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 2.0 MB/s | 9.8 MB 00:04 2024-03-06T07:15:55.021 INFO:teuthology.orchestra.run.smithi145.stdout:(36/176): ceph-mgr-diskprediction-local-19.0.0- 4.9 MB/s | 7.4 MB 00:01 2024-03-06T07:15:55.096 INFO:teuthology.orchestra.run.smithi145.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 244 kB/s | 18 kB 00:00 2024-03-06T07:15:55.097 INFO:teuthology.orchestra.run.smithi094.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 2.2 MB/s | 220 kB 00:00 2024-03-06T07:15:55.197 INFO:teuthology.orchestra.run.smithi094.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 271 kB/s | 27 kB 00:00 2024-03-06T07:15:55.213 INFO:teuthology.orchestra.run.smithi145.stdout:(38/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.4 MB/s | 748 kB 00:00 2024-03-06T07:15:55.247 INFO:teuthology.orchestra.run.smithi145.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-06T07:15:55.273 INFO:teuthology.orchestra.run.smithi094.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 417 kB/s | 31 kB 00:00 2024-03-06T07:15:55.297 INFO:teuthology.orchestra.run.smithi145.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 920 kB 00:00 2024-03-06T07:15:55.322 INFO:teuthology.orchestra.run.smithi145.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-03-06T07:15:55.348 INFO:teuthology.orchestra.run.smithi145.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 408 kB/s | 20 kB 00:00 2024-03-06T07:15:55.376 INFO:teuthology.orchestra.run.smithi145.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-06T07:15:55.390 INFO:teuthology.orchestra.run.smithi145.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 770 kB/s | 32 kB 00:00 2024-03-06T07:15:55.474 INFO:teuthology.orchestra.run.smithi145.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-06T07:15:55.499 INFO:teuthology.orchestra.run.smithi145.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 159 kB/s | 17 kB 00:00 2024-03-06T07:15:55.516 INFO:teuthology.orchestra.run.smithi145.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 241 kB/s | 10 kB 00:00 2024-03-06T07:15:55.569 INFO:teuthology.orchestra.run.smithi145.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.2 MB/s | 159 kB 00:00 2024-03-06T07:15:55.592 INFO:teuthology.orchestra.run.smithi145.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-06T07:15:55.615 INFO:teuthology.orchestra.run.smithi094.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 2.1 MB/s | 747 kB 00:00 2024-03-06T07:15:55.642 INFO:teuthology.orchestra.run.smithi145.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.0 MB/s | 661 kB 00:00 2024-03-06T07:15:55.667 INFO:teuthology.orchestra.run.smithi145.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 578 kB 00:00 2024-03-06T07:15:55.684 INFO:teuthology.orchestra.run.smithi145.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 597 kB/s | 24 kB 00:00 2024-03-06T07:15:55.701 INFO:teuthology.orchestra.run.smithi145.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-06T07:15:55.715 INFO:teuthology.orchestra.run.smithi094.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.6 MB/s | 163 kB 00:00 2024-03-06T07:15:55.726 INFO:teuthology.orchestra.run.smithi145.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-06T07:15:55.752 INFO:teuthology.orchestra.run.smithi145.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-06T07:15:55.768 INFO:teuthology.orchestra.run.smithi145.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 827 kB/s | 34 kB 00:00 2024-03-06T07:15:55.841 INFO:teuthology.orchestra.run.smithi094.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.0 MB/s | 127 kB 00:00 2024-03-06T07:15:55.894 INFO:teuthology.orchestra.run.smithi145.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-03-06T07:15:55.952 INFO:teuthology.orchestra.run.smithi145.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-03-06T07:15:55.994 INFO:teuthology.orchestra.run.smithi145.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2024-03-06T07:15:56.036 INFO:teuthology.orchestra.run.smithi145.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 9.8 MB/s | 811 kB 00:00 2024-03-06T07:15:56.079 INFO:teuthology.orchestra.run.smithi145.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 1.6 MB/s | 134 kB 00:00 2024-03-06T07:15:56.107 INFO:teuthology.orchestra.run.smithi094.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 1.8 MB/s | 501 kB 00:00 2024-03-06T07:15:56.271 INFO:teuthology.orchestra.run.smithi145.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 24 MB/s | 5.7 MB 00:00 2024-03-06T07:15:56.304 INFO:teuthology.orchestra.run.smithi145.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-06T07:15:56.321 INFO:teuthology.orchestra.run.smithi145.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-06T07:15:56.513 INFO:teuthology.orchestra.run.smithi145.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 227 kB 00:00 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi145.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi145.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 244 kB/s | 24 kB 00:00 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi145.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 359 kB/s | 18 kB 00:00 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi145.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi145.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-06T07:15:56.724 INFO:teuthology.orchestra.run.smithi094.stdout:(18/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 1.6 MB/s | 17 MB 00:10 2024-03-06T07:15:56.808 INFO:teuthology.orchestra.run.smithi094.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 543 kB/s | 45 kB 00:00 2024-03-06T07:15:56.831 INFO:teuthology.orchestra.run.smithi145.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-06T07:15:56.857 INFO:teuthology.orchestra.run.smithi145.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-06T07:15:56.890 INFO:teuthology.orchestra.run.smithi145.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-06T07:15:56.924 INFO:teuthology.orchestra.run.smithi145.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-06T07:15:56.950 INFO:teuthology.orchestra.run.smithi145.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 700 kB/s | 41 kB 00:00 2024-03-06T07:15:56.983 INFO:teuthology.orchestra.run.smithi145.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-06T07:15:57.008 INFO:teuthology.orchestra.run.smithi145.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-06T07:15:57.042 INFO:teuthology.orchestra.run.smithi145.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-06T07:15:57.067 INFO:teuthology.orchestra.run.smithi145.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 807 kB/s | 47 kB 00:00 2024-03-06T07:15:57.075 INFO:teuthology.orchestra.run.smithi094.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 493 kB/s | 131 kB 00:00 2024-03-06T07:15:57.092 INFO:teuthology.orchestra.run.smithi145.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-06T07:15:57.325 INFO:teuthology.orchestra.run.smithi094.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 647 kB/s | 162 kB 00:00 2024-03-06T07:15:57.518 INFO:teuthology.orchestra.run.smithi145.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 91 kB/s | 39 kB 00:00 2024-03-06T07:15:57.567 INFO:teuthology.orchestra.run.smithi094.stdout:(22/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 3.5 MB/s | 5.1 MB 00:01 2024-03-06T07:15:57.709 INFO:teuthology.orchestra.run.smithi094.stdout:(23/176): python3-rados-19.0.0-1739.g1bc250c7.e 837 kB/s | 321 kB 00:00 2024-03-06T07:15:57.810 INFO:teuthology.orchestra.run.smithi145.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-03-06T07:15:57.827 INFO:teuthology.orchestra.run.smithi145.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 133 kB/s | 41 kB 00:00 2024-03-06T07:15:57.834 INFO:teuthology.orchestra.run.smithi094.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 793 kB/s | 99 kB 00:00 2024-03-06T07:15:57.885 INFO:teuthology.orchestra.run.smithi145.stdout:(84/176): qatlib-service-23.11.0-1.el9.x86_64.r 583 kB/s | 34 kB 00:00 2024-03-06T07:15:57.911 INFO:teuthology.orchestra.run.smithi145.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 215 kB 00:00 2024-03-06T07:15:57.944 INFO:teuthology.orchestra.run.smithi145.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-03-06T07:15:57.968 INFO:teuthology.orchestra.run.smithi094.stdout:(25/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 744 kB/s | 298 kB 00:00 2024-03-06T07:15:57.986 INFO:teuthology.orchestra.run.smithi145.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-06T07:15:57.993 INFO:teuthology.orchestra.run.smithi094.stdout:(26/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 537 kB/s | 85 kB 00:00 2024-03-06T07:15:58.020 INFO:teuthology.orchestra.run.smithi145.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 746 kB/s | 56 kB 00:00 2024-03-06T07:15:58.046 INFO:teuthology.orchestra.run.smithi145.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 87 kB 00:00 2024-03-06T07:15:58.079 INFO:teuthology.orchestra.run.smithi145.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-06T07:15:58.104 INFO:teuthology.orchestra.run.smithi145.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 754 kB/s | 44 kB 00:00 2024-03-06T07:15:58.146 INFO:teuthology.orchestra.run.smithi145.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.1 MB/s | 79 kB 00:00 2024-03-06T07:15:58.180 INFO:teuthology.orchestra.run.smithi145.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-03-06T07:15:58.219 INFO:teuthology.orchestra.run.smithi094.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 758 kB/s | 171 kB 00:00 2024-03-06T07:15:58.239 INFO:teuthology.orchestra.run.smithi145.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 8.5 MB/s | 802 kB 00:00 2024-03-06T07:15:58.264 INFO:teuthology.orchestra.run.smithi145.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 194 kB 00:00 2024-03-06T07:15:58.286 INFO:teuthology.orchestra.run.smithi094.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 472 kB/s | 31 kB 00:00 2024-03-06T07:15:58.289 INFO:teuthology.orchestra.run.smithi145.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-06T07:15:58.315 INFO:teuthology.orchestra.run.smithi145.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-06T07:15:58.373 INFO:teuthology.orchestra.run.smithi145.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-06T07:15:58.398 INFO:teuthology.orchestra.run.smithi145.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-06T07:15:58.424 INFO:teuthology.orchestra.run.smithi145.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-06T07:15:58.427 INFO:teuthology.orchestra.run.smithi094.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 974 kB/s | 138 kB 00:00 2024-03-06T07:15:58.449 INFO:teuthology.orchestra.run.smithi145.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:15:58.474 INFO:teuthology.orchestra.run.smithi145.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:15:58.641 INFO:teuthology.orchestra.run.smithi145.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-03-06T07:15:58.850 INFO:teuthology.orchestra.run.smithi145.stdout:(104/176): protobuf-compiler-3.14.0-13.el9.x86_ 4.1 MB/s | 863 kB 00:00 2024-03-06T07:15:58.852 INFO:teuthology.orchestra.run.smithi094.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 3.5 MB/s | 3.1 MB 00:00 2024-03-06T07:15:59.058 INFO:teuthology.orchestra.run.smithi145.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 403 kB/s | 245 kB 00:00 2024-03-06T07:15:59.175 INFO:teuthology.orchestra.run.smithi145.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-06T07:15:59.200 INFO:teuthology.orchestra.run.smithi145.stdout:(107/176): gperftools-libs-2.9.1-2.el9.x86_64.r 2.1 MB/s | 309 kB 00:00 2024-03-06T07:15:59.334 INFO:teuthology.orchestra.run.smithi145.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-03-06T07:15:59.851 INFO:teuthology.orchestra.run.smithi145.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 50 kB/s | 26 kB 00:00 2024-03-06T07:15:59.870 INFO:teuthology.orchestra.run.smithi094.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.2 MB/s | 1.8 MB 00:01 2024-03-06T07:16:00.002 INFO:teuthology.orchestra.run.smithi145.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 325 kB/s | 49 kB 00:00 2024-03-06T07:16:00.110 INFO:teuthology.orchestra.run.smithi145.stdout:(111/176): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-06T07:16:00.135 INFO:teuthology.orchestra.run.smithi145.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 513 kB/s | 67 kB 00:00 2024-03-06T07:16:00.211 INFO:teuthology.orchestra.run.smithi145.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-06T07:16:00.253 INFO:teuthology.orchestra.run.smithi094.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 649 kB/s | 248 kB 00:00 2024-03-06T07:16:00.320 INFO:teuthology.orchestra.run.smithi094.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 740 kB/s | 49 kB 00:00 2024-03-06T07:16:00.378 INFO:teuthology.orchestra.run.smithi094.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 263 kB/s | 15 kB 00:00 2024-03-06T07:16:00.763 INFO:teuthology.orchestra.run.smithi094.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 688 kB/s | 264 kB 00:00 2024-03-06T07:16:00.853 INFO:teuthology.orchestra.run.smithi145.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 838 kB 00:00 2024-03-06T07:16:00.912 INFO:teuthology.orchestra.run.smithi145.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 782 kB/s | 548 kB 00:00 2024-03-06T07:16:00.928 INFO:teuthology.orchestra.run.smithi145.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 582 kB/s | 43 kB 00:00 2024-03-06T07:16:00.970 INFO:teuthology.orchestra.run.smithi145.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-03-06T07:16:00.995 INFO:teuthology.orchestra.run.smithi145.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-06T07:16:01.046 INFO:teuthology.orchestra.run.smithi145.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-06T07:16:01.071 INFO:teuthology.orchestra.run.smithi145.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-06T07:16:01.138 INFO:teuthology.orchestra.run.smithi145.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-06T07:16:01.164 INFO:teuthology.orchestra.run.smithi145.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-03-06T07:16:01.264 INFO:teuthology.orchestra.run.smithi145.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-06T07:16:01.821 INFO:teuthology.orchestra.run.smithi094.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 707 kB/s | 748 kB 00:01 2024-03-06T07:16:01.921 INFO:teuthology.orchestra.run.smithi094.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 183 kB/s | 18 kB 00:00 2024-03-06T07:16:02.080 INFO:teuthology.orchestra.run.smithi094.stdout:(38/176): ceph-mgr-diskprediction-local-19.0.0- 2.3 MB/s | 7.4 MB 00:03 2024-03-06T07:16:02.180 INFO:teuthology.orchestra.run.smithi094.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 662 kB 00:00 2024-03-06T07:16:02.222 INFO:teuthology.orchestra.run.smithi094.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 6.4 MB/s | 920 kB 00:00 2024-03-06T07:16:02.247 INFO:teuthology.orchestra.run.smithi094.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-06T07:16:02.272 INFO:teuthology.orchestra.run.smithi094.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 406 kB/s | 20 kB 00:00 2024-03-06T07:16:02.298 INFO:teuthology.orchestra.run.smithi094.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-06T07:16:02.323 INFO:teuthology.orchestra.run.smithi094.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-03-06T07:16:02.407 INFO:teuthology.orchestra.run.smithi094.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-06T07:16:02.414 INFO:teuthology.orchestra.run.smithi145.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 1.6 MB/s | 2.0 MB 00:01 2024-03-06T07:16:02.424 INFO:teuthology.orchestra.run.smithi094.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 175 kB/s | 17 kB 00:00 2024-03-06T07:16:02.439 INFO:teuthology.orchestra.run.smithi145.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 123 kB/s | 144 kB 00:01 2024-03-06T07:16:02.457 INFO:teuthology.orchestra.run.smithi094.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 304 kB/s | 10 kB 00:00 2024-03-06T07:16:02.483 INFO:teuthology.orchestra.run.smithi094.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-06T07:16:02.508 INFO:teuthology.orchestra.run.smithi094.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-06T07:16:02.543 INFO:teuthology.orchestra.run.smithi094.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-06T07:16:02.548 INFO:teuthology.orchestra.run.smithi145.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-06T07:16:02.566 INFO:teuthology.orchestra.run.smithi094.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-06T07:16:02.574 INFO:teuthology.orchestra.run.smithi145.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 135 kB/s | 18 kB 00:00 2024-03-06T07:16:02.583 INFO:teuthology.orchestra.run.smithi094.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 599 kB/s | 24 kB 00:00 2024-03-06T07:16:02.600 INFO:teuthology.orchestra.run.smithi094.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-06T07:16:02.648 INFO:teuthology.orchestra.run.smithi094.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 282 kB 00:00 2024-03-06T07:16:02.676 INFO:teuthology.orchestra.run.smithi094.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-06T07:16:02.682 INFO:teuthology.orchestra.run.smithi145.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-03-06T07:16:02.701 INFO:teuthology.orchestra.run.smithi094.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 656 kB/s | 34 kB 00:00 2024-03-06T07:16:02.775 INFO:teuthology.orchestra.run.smithi145.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 97 kB/s | 19 kB 00:00 2024-03-06T07:16:02.799 INFO:teuthology.orchestra.run.smithi145.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-06T07:16:02.801 INFO:teuthology.orchestra.run.smithi094.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-06T07:16:02.860 INFO:teuthology.orchestra.run.smithi094.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-03-06T07:16:02.902 INFO:teuthology.orchestra.run.smithi094.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 16 MB/s | 1.5 MB 00:00 2024-03-06T07:16:02.936 INFO:teuthology.orchestra.run.smithi094.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-06T07:16:02.961 INFO:teuthology.orchestra.run.smithi094.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-06T07:16:03.161 INFO:teuthology.orchestra.run.smithi094.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-06T07:16:03.203 INFO:teuthology.orchestra.run.smithi094.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-03-06T07:16:03.228 INFO:teuthology.orchestra.run.smithi094.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-06T07:16:03.437 INFO:teuthology.orchestra.run.smithi094.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-06T07:16:03.462 INFO:teuthology.orchestra.run.smithi094.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 972 kB/s | 227 kB 00:00 2024-03-06T07:16:03.487 INFO:teuthology.orchestra.run.smithi094.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 490 kB/s | 24 kB 00:00 2024-03-06T07:16:03.513 INFO:teuthology.orchestra.run.smithi094.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-06T07:16:03.546 INFO:teuthology.orchestra.run.smithi094.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-06T07:16:03.572 INFO:teuthology.orchestra.run.smithi094.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 539 kB/s | 32 kB 00:00 2024-03-06T07:16:03.755 INFO:teuthology.orchestra.run.smithi094.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-06T07:16:03.781 INFO:teuthology.orchestra.run.smithi094.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-03-06T07:16:03.806 INFO:teuthology.orchestra.run.smithi094.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-06T07:16:03.831 INFO:teuthology.orchestra.run.smithi094.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-06T07:16:03.857 INFO:teuthology.orchestra.run.smithi094.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 819 kB/s | 41 kB 00:00 2024-03-06T07:16:03.890 INFO:teuthology.orchestra.run.smithi094.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-06T07:16:03.916 INFO:teuthology.orchestra.run.smithi094.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-06T07:16:03.941 INFO:teuthology.orchestra.run.smithi094.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:16:03.966 INFO:teuthology.orchestra.run.smithi094.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-06T07:16:03.991 INFO:teuthology.orchestra.run.smithi094.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-06T07:16:04.225 INFO:teuthology.orchestra.run.smithi145.stdout:(131/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x 674 kB/s | 17 MB 00:25 2024-03-06T07:16:04.275 INFO:teuthology.orchestra.run.smithi094.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 136 kB/s | 39 kB 00:00 2024-03-06T07:16:04.508 INFO:teuthology.orchestra.run.smithi094.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-06T07:16:04.534 INFO:teuthology.orchestra.run.smithi094.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-06T07:16:04.559 INFO:teuthology.orchestra.run.smithi094.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-03-06T07:16:04.584 INFO:teuthology.orchestra.run.smithi094.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 677 kB/s | 34 kB 00:00 2024-03-06T07:16:04.609 INFO:teuthology.orchestra.run.smithi094.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 866 kB/s | 43 kB 00:00 2024-03-06T07:16:04.635 INFO:teuthology.orchestra.run.smithi094.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-06T07:16:04.660 INFO:teuthology.orchestra.run.smithi094.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-06T07:16:04.677 INFO:teuthology.orchestra.run.smithi094.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-06T07:16:04.702 INFO:teuthology.orchestra.run.smithi094.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-06T07:16:04.728 INFO:teuthology.orchestra.run.smithi094.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-06T07:16:04.744 INFO:teuthology.orchestra.run.smithi094.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-06T07:16:04.770 INFO:teuthology.orchestra.run.smithi094.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-03-06T07:16:04.803 INFO:teuthology.orchestra.run.smithi094.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:16:04.829 INFO:teuthology.orchestra.run.smithi094.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 194 kB 00:00 2024-03-06T07:16:04.854 INFO:teuthology.orchestra.run.smithi094.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-06T07:16:04.887 INFO:teuthology.orchestra.run.smithi094.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-06T07:16:04.899 INFO:teuthology.orchestra.run.smithi145.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 498 kB/s | 1.0 MB 00:02 2024-03-06T07:16:04.946 INFO:teuthology.orchestra.run.smithi094.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-06T07:16:04.958 INFO:teuthology.orchestra.run.smithi145.stdout:(133/176): python3-logutils-0.3.5-21.el9.noarch 21 kB/s | 46 kB 00:02 2024-03-06T07:16:04.971 INFO:teuthology.orchestra.run.smithi094.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-06T07:16:04.999 INFO:teuthology.orchestra.run.smithi094.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-06T07:16:05.022 INFO:teuthology.orchestra.run.smithi094.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:16:05.047 INFO:teuthology.orchestra.run.smithi094.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:16:05.217 INFO:teuthology.orchestra.run.smithi145.stdout:(134/176): python3-more-itertools-8.12.0-2.el9. 79 kB/s | 79 kB 00:00 2024-03-06T07:16:05.234 INFO:teuthology.orchestra.run.smithi145.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 173 kB/s | 58 kB 00:00 2024-03-06T07:16:05.318 INFO:teuthology.orchestra.run.smithi094.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 82 kB/s | 22 kB 00:00 2024-03-06T07:16:05.384 INFO:teuthology.orchestra.run.smithi145.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 640 kB/s | 272 kB 00:00 2024-03-06T07:16:05.389 INFO:teuthology.orchestra.run.smithi094.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 668 kB/s | 245 kB 00:00 2024-03-06T07:16:05.401 INFO:teuthology.orchestra.run.smithi145.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 89 kB/s | 16 kB 00:00 2024-03-06T07:16:05.451 INFO:teuthology.orchestra.run.smithi145.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 414 kB/s | 90 kB 00:00 2024-03-06T07:16:05.468 INFO:teuthology.orchestra.run.smithi145.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-03-06T07:16:05.493 INFO:teuthology.orchestra.run.smithi145.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-06T07:16:05.582 INFO:teuthology.orchestra.run.smithi094.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-03-06T07:16:05.619 INFO:teuthology.orchestra.run.smithi145.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-03-06T07:16:05.669 INFO:teuthology.orchestra.run.smithi145.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-03-06T07:16:05.694 INFO:teuthology.orchestra.run.smithi145.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 178 kB/s | 36 kB 00:00 2024-03-06T07:16:05.720 INFO:teuthology.orchestra.run.smithi145.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-06T07:16:05.782 INFO:teuthology.orchestra.run.smithi094.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-06T07:16:05.845 INFO:teuthology.orchestra.run.smithi145.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-06T07:16:05.874 INFO:teuthology.orchestra.run.smithi094.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-06T07:16:05.903 INFO:teuthology.orchestra.run.smithi145.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 430 kB/s | 90 kB 00:00 2024-03-06T07:16:05.958 INFO:teuthology.orchestra.run.smithi094.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 823 kB/s | 309 kB 00:00 2024-03-06T07:16:06.033 INFO:teuthology.orchestra.run.smithi094.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-06T07:16:06.108 INFO:teuthology.orchestra.run.smithi094.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-06T07:16:06.183 INFO:teuthology.orchestra.run.smithi094.stdout:(111/176): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-06T07:16:06.208 INFO:teuthology.orchestra.run.smithi094.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-06T07:16:06.212 INFO:teuthology.orchestra.run.smithi145.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 868 kB/s | 427 kB 00:00 2024-03-06T07:16:06.262 INFO:teuthology.orchestra.run.smithi145.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 48 kB/s | 20 kB 00:00 2024-03-06T07:16:06.300 INFO:teuthology.orchestra.run.smithi094.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-06T07:16:06.434 INFO:teuthology.orchestra.run.smithi094.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-03-06T07:16:06.492 INFO:teuthology.orchestra.run.smithi094.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-06T07:16:06.543 INFO:teuthology.orchestra.run.smithi094.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-03-06T07:16:06.593 INFO:teuthology.orchestra.run.smithi094.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-06T07:16:06.643 INFO:teuthology.orchestra.run.smithi094.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-06T07:16:06.694 INFO:teuthology.orchestra.run.smithi094.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-06T07:16:06.761 INFO:teuthology.orchestra.run.smithi094.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-06T07:16:06.803 INFO:teuthology.orchestra.run.smithi094.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-06T07:16:06.844 INFO:teuthology.orchestra.run.smithi094.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 424 kB/s | 35 kB 00:00 2024-03-06T07:16:06.895 INFO:teuthology.orchestra.run.smithi094.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-06T07:16:06.995 INFO:teuthology.orchestra.run.smithi094.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-06T07:16:07.070 INFO:teuthology.orchestra.run.smithi094.stdout:(125/176): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-03-06T07:16:07.096 INFO:teuthology.orchestra.run.smithi094.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-06T07:16:07.162 INFO:teuthology.orchestra.run.smithi094.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-06T07:16:07.188 INFO:teuthology.orchestra.run.smithi094.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-03-06T07:16:07.229 INFO:teuthology.orchestra.run.smithi145.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 145 kB/s | 192 kB 00:01 2024-03-06T07:16:07.247 INFO:teuthology.orchestra.run.smithi094.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 233 kB/s | 19 kB 00:00 2024-03-06T07:16:07.272 INFO:teuthology.orchestra.run.smithi094.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-06T07:16:07.389 INFO:teuthology.orchestra.run.smithi094.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-06T07:16:07.414 INFO:teuthology.orchestra.run.smithi094.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-03-06T07:16:07.498 INFO:teuthology.orchestra.run.smithi094.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 726 kB/s | 79 kB 00:00 2024-03-06T07:16:07.515 INFO:teuthology.orchestra.run.smithi094.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 574 kB/s | 58 kB 00:00 2024-03-06T07:16:07.546 INFO:teuthology.orchestra.run.smithi145.stdout:(150/176): librados2-19.0.0-1739.g1bc250c7.el9. 2.6 MB/s | 3.4 MB 00:01 2024-03-06T07:16:07.606 INFO:teuthology.orchestra.run.smithi094.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-06T07:16:07.623 INFO:teuthology.orchestra.run.smithi094.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-06T07:16:07.646 INFO:teuthology.orchestra.run.smithi145.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 676 kB/s | 68 kB 00:00 2024-03-06T07:16:07.690 INFO:teuthology.orchestra.run.smithi094.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-06T07:16:07.707 INFO:teuthology.orchestra.run.smithi094.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 338 kB/s | 28 kB 00:00 2024-03-06T07:16:07.772 INFO:teuthology.orchestra.run.smithi145.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 18 MB/s | 2.3 MB 00:00 2024-03-06T07:16:07.816 INFO:teuthology.orchestra.run.smithi094.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 246 kB/s | 31 kB 00:00 2024-03-06T07:16:07.925 INFO:teuthology.orchestra.run.smithi094.stdout:(140/176): python3-rsa-4.9-2.el9.noarch.rpm 547 kB/s | 59 kB 00:00 2024-03-06T07:16:07.955 INFO:teuthology.orchestra.run.smithi145.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 9.9 MB/s | 1.8 MB 00:00 2024-03-06T07:16:07.989 INFO:teuthology.orchestra.run.smithi145.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:16:07.991 INFO:teuthology.orchestra.run.smithi094.stdout:(141/176): python3-tempora-5.0.0-2.el9.noarch.r 539 kB/s | 36 kB 00:00 2024-03-06T07:16:08.039 INFO:teuthology.orchestra.run.smithi145.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-06T07:16:08.075 INFO:teuthology.orchestra.run.smithi094.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 611 kB/s | 51 kB 00:00 2024-03-06T07:16:08.081 INFO:teuthology.orchestra.run.smithi145.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 69 kB 00:00 2024-03-06T07:16:08.106 INFO:teuthology.orchestra.run.smithi145.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-03-06T07:16:08.148 INFO:teuthology.orchestra.run.smithi145.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:16:08.167 INFO:teuthology.orchestra.run.smithi094.stdout:(143/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2024-03-06T07:16:08.198 INFO:teuthology.orchestra.run.smithi145.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-03-06T07:16:08.257 INFO:teuthology.orchestra.run.smithi145.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:16:08.259 INFO:teuthology.orchestra.run.smithi094.stdout:(144/176): python3-websocket-client-1.2.3-2.el9 978 kB/s | 90 kB 00:00 2024-03-06T07:16:08.307 INFO:teuthology.orchestra.run.smithi145.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.5 MB/s | 433 kB 00:00 2024-03-06T07:16:08.318 INFO:teuthology.orchestra.run.smithi094.stdout:(145/176): python3-routes-2.5.1-5.el9.noarch.rp 309 kB/s | 188 kB 00:00 2024-03-06T07:16:08.333 INFO:teuthology.orchestra.run.smithi145.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 877 kB/s | 22 kB 00:00 2024-03-06T07:16:08.359 INFO:teuthology.orchestra.run.smithi094.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-06T07:16:08.366 INFO:teuthology.orchestra.run.smithi145.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 107 kB 00:00 2024-03-06T07:16:08.401 INFO:teuthology.orchestra.run.smithi094.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-06T07:16:08.408 INFO:teuthology.orchestra.run.smithi145.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-03-06T07:16:08.450 INFO:teuthology.orchestra.run.smithi145.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-03-06T07:16:08.453 INFO:teuthology.orchestra.run.smithi094.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-06T07:16:08.484 INFO:teuthology.orchestra.run.smithi145.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.8 MB/s | 63 kB 00:00 2024-03-06T07:16:08.509 INFO:teuthology.orchestra.run.smithi145.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 28 kB 00:00 2024-03-06T07:16:08.593 INFO:teuthology.orchestra.run.smithi145.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-03-06T07:16:08.644 INFO:teuthology.orchestra.run.smithi094.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-06T07:16:08.718 INFO:teuthology.orchestra.run.smithi145.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-06T07:16:08.843 INFO:teuthology.orchestra.run.smithi145.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 18 MB/s | 2.2 MB 00:00 2024-03-06T07:16:08.902 INFO:teuthology.orchestra.run.smithi145.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 7.4 MB/s | 438 kB 00:00 2024-03-06T07:16:08.944 INFO:teuthology.orchestra.run.smithi145.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 716 kB/s | 30 kB 00:00 2024-03-06T07:16:08.994 INFO:teuthology.orchestra.run.smithi145.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:16:09.044 INFO:teuthology.orchestra.run.smithi094.stdout:(150/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 3.1 MB/s | 47 MB 00:15 2024-03-06T07:16:09.086 INFO:teuthology.orchestra.run.smithi094.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-06T07:16:09.086 INFO:teuthology.orchestra.run.smithi145.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-03-06T07:16:09.203 INFO:teuthology.orchestra.run.smithi094.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 20 MB/s | 2.3 MB 00:00 2024-03-06T07:16:09.286 INFO:teuthology.orchestra.run.smithi094.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-06T07:16:09.320 INFO:teuthology.orchestra.run.smithi094.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:16:09.362 INFO:teuthology.orchestra.run.smithi094.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:16:09.395 INFO:teuthology.orchestra.run.smithi094.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:16:09.421 INFO:teuthology.orchestra.run.smithi094.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.4 MB/s | 35 kB 00:00 2024-03-06T07:16:09.463 INFO:teuthology.orchestra.run.smithi094.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:16:09.508 INFO:teuthology.orchestra.run.smithi094.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:16:09.571 INFO:teuthology.orchestra.run.smithi094.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 12 MB/s | 802 kB 00:00 2024-03-06T07:16:09.622 INFO:teuthology.orchestra.run.smithi094.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.5 MB/s | 433 kB 00:00 2024-03-06T07:16:09.647 INFO:teuthology.orchestra.run.smithi094.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 877 kB/s | 22 kB 00:00 2024-03-06T07:16:09.689 INFO:teuthology.orchestra.run.smithi094.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-03-06T07:16:09.731 INFO:teuthology.orchestra.run.smithi094.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-03-06T07:16:09.765 INFO:teuthology.orchestra.run.smithi094.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.9 MB/s | 135 kB 00:00 2024-03-06T07:16:09.790 INFO:teuthology.orchestra.run.smithi094.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 2.5 MB/s | 63 kB 00:00 2024-03-06T07:16:09.815 INFO:teuthology.orchestra.run.smithi094.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 28 kB 00:00 2024-03-06T07:16:09.907 INFO:teuthology.orchestra.run.smithi094.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-06T07:16:10.057 INFO:teuthology.orchestra.run.smithi094.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-06T07:16:10.108 INFO:teuthology.orchestra.run.smithi094.stdout:(170/176): librados2-19.0.0-1739.g1bc250c7.el9. 2.0 MB/s | 3.4 MB 00:01 2024-03-06T07:16:10.225 INFO:teuthology.orchestra.run.smithi094.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 13 MB/s | 2.2 MB 00:00 2024-03-06T07:16:10.253 INFO:teuthology.orchestra.run.smithi145.stdout:(175/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.0 MB/s | 3.1 MB 00:03 2024-03-06T07:16:10.358 INFO:teuthology.orchestra.run.smithi094.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 1.7 MB/s | 438 kB 00:00 2024-03-06T07:16:10.384 INFO:teuthology.orchestra.run.smithi094.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 190 kB/s | 30 kB 00:00 2024-03-06T07:16:10.417 INFO:teuthology.orchestra.run.smithi094.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-06T07:16:10.492 INFO:teuthology.orchestra.run.smithi094.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.8 MB/s | 1.1 MB 00:00 2024-03-06T07:16:11.426 INFO:teuthology.orchestra.run.smithi094.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.1 MB/s | 3.1 MB 00:02 2024-03-06T07:16:11.431 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:16:11.431 INFO:teuthology.orchestra.run.smithi094.stdout:Total 6.8 MB/s | 224 MB 00:32 2024-03-06T07:16:11.475 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-06T07:16:11.686 INFO:teuthology.orchestra.run.smithi145.stdout:(176/176): thrift-0.15.0-2.el9.x86_64.rpm 296 kB/s | 1.6 MB 00:05 2024-03-06T07:16:11.692 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:16:11.692 INFO:teuthology.orchestra.run.smithi145.stdout:Total 6.0 MB/s | 224 MB 00:37 2024-03-06T07:16:11.740 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-06T07:16:11.788 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-06T07:16:11.788 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-06T07:16:12.055 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-06T07:16:12.055 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-06T07:16:13.749 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-06T07:16:13.749 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-06T07:16:14.047 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-06T07:16:14.048 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-06T07:16:20.728 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-06T07:16:20.822 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:16:20.861 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:16:20.920 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:16:20.943 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:16:20.990 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:16:20.997 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-06T07:16:21.028 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:16:21.082 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:16:21.083 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:16:21.115 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:16:21.116 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:16:21.141 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:16:21.163 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:16:21.189 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:16:21.189 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:16:21.242 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:16:21.280 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:16:21.338 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:16:21.376 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:16:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:16:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:16:21.517 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:16:21.600 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:16:21.721 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:16:21.768 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:16:21.772 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:16:21.844 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:16:21.858 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:16:21.899 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:16:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:16:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:16:22.060 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:16:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:16:22.118 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:16:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:16:22.147 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:16:22.249 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:16:22.273 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:16:22.294 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:16:22.329 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:16:22.356 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:16:22.359 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:16:22.382 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:16:22.384 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:16:22.398 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:16:22.487 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:16:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:16:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:16:22.591 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:16:22.616 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:16:22.622 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:16:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:16:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:16:22.669 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:16:22.836 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:16:22.837 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:16:22.868 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:16:22.908 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:16:22.968 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:16:23.020 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:16:23.096 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:16:23.139 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:16:23.287 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:16:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:23.393 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:23.466 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:23.508 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:16:23.604 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:16:23.635 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:16:23.680 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:16:23.702 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:16:23.748 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:16:23.788 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:16:23.788 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:16:23.823 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:16:23.863 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:16:23.886 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:16:23.923 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:16:23.927 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:16:23.962 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:16:23.998 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:16:24.014 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:16:24.047 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:16:24.052 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:16:24.075 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:16:24.083 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:16:24.118 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:16:24.143 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:16:24.158 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:16:24.179 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:16:24.222 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:16:24.223 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:16:24.252 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:16:24.260 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:16:24.301 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:16:24.319 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:16:24.334 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:16:24.361 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:16:24.366 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:16:24.430 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:16:24.439 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:16:24.469 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:16:24.510 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:16:24.539 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:16:24.579 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:16:24.607 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:16:24.635 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:16:24.653 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:16:24.726 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:16:24.869 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:16:24.923 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:16:25.145 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:16:25.230 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:16:25.308 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:16:25.359 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:16:25.366 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:16:25.406 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:16:25.443 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:16:25.443 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:16:25.477 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:16:25.526 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:16:25.579 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:16:25.641 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:16:25.653 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:16:25.696 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:16:25.704 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:16:25.742 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:16:25.781 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:16:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:16:25.930 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:16:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:16:26.096 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:16:26.135 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:16:26.213 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:16:26.257 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:16:26.394 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:16:26.444 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:16:26.499 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:16:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:16:26.571 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:16:26.603 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:16:26.639 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:16:26.686 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:16:26.722 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:16:26.765 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:16:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:16:26.809 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:16:26.891 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:16:26.935 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:16:26.941 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:16:26.988 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:16:27.018 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:16:27.059 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:16:27.076 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:16:27.130 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:16:27.154 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:16:27.187 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:16:27.201 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:16:27.242 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:16:27.260 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:16:27.267 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:16:27.291 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:16:27.292 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:16:27.327 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:16:27.372 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:16:27.375 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:16:27.410 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:16:27.453 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:16:27.493 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:16:27.504 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:16:27.609 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:16:27.637 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:16:27.676 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:16:27.696 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:16:27.705 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:16:27.747 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:16:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:16:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:16:27.874 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:16:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:16:27.946 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:16:27.971 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:16:28.047 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:16:28.068 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:16:28.180 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:16:28.279 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:16:28.329 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:16:28.377 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:16:28.401 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:16:28.433 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:16:28.454 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:16:28.584 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:16:28.617 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:16:28.620 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:16:28.660 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:16:28.675 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:16:28.693 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:16:28.727 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:16:28.758 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:16:28.786 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:16:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:16:28.821 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:16:28.856 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:16:28.860 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:16:28.889 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:16:28.891 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:16:28.921 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:16:28.921 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:16:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:16:28.997 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:16:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:16:29.032 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:16:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:16:29.065 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:16:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:16:29.107 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:16:29.121 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:16:29.150 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:16:29.153 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:16:29.183 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:16:29.238 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:16:29.260 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:16:29.293 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:16:29.320 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:16:29.326 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:16:29.368 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:16:29.372 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:16:29.402 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:16:29.404 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:16:29.451 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:16:29.490 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:16:29.548 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:16:29.582 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:16:29.597 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:16:29.625 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:16:29.656 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:16:29.704 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:16:29.801 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:16:29.849 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:16:30.318 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:16:30.369 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:16:30.403 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:16:30.451 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:16:30.540 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:16:30.578 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:16:30.596 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:16:30.630 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:16:30.663 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:16:30.712 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:16:30.772 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:16:30.798 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:16:30.799 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:16:30.839 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:16:30.875 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:16:31.016 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:16:31.054 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:16:31.134 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:16:31.519 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:16:31.639 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:16:31.777 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:16:31.893 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:16:32.855 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:16:32.936 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:16:32.963 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:16:33.113 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:16:33.182 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:16:33.209 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:16:33.276 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:16:33.301 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:16:33.391 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:16:33.423 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:16:33.460 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:16:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:16:33.555 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:16:33.646 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:16:33.677 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:16:33.713 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:16:33.914 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:16:33.946 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:16:34.017 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:16:34.039 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:16:34.166 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:16:34.192 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:16:34.262 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:16:34.285 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:16:36.299 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:36.339 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:36.511 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:36.550 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:36.884 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:36.925 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:16:37.014 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:16:37.014 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:16:37.014 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:16:37.015 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:37.043 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:16:37.083 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:16:37.120 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:16:37.212 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:16:37.212 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:16:37.212 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:16:37.212 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:37.246 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:16:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:16:48.700 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /sys 2024-03-06T07:16:48.700 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /proc 2024-03-06T07:16:48.700 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /mnt 2024-03-06T07:16:48.701 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /var/tmp 2024-03-06T07:16:48.701 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /home 2024-03-06T07:16:48.701 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /root 2024-03-06T07:16:48.701 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /tmp 2024-03-06T07:16:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:48.873 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:16:48.888 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:16:48.888 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:48.888 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:16:48.888 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-03-06T07:16:48.897 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:49.068 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:16:49.082 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:16:49.082 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:49.083 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:16:49.083 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:16:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:16:50.377 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:50.377 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:16:50.377 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:16:50.377 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:16:50.377 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:50.561 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:16:50.586 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:16:50.586 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:50.586 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:16:50.587 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:16:50.587 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:16:50.587 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:50.610 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:16:50.633 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:16:50.633 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:50.633 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:16:50.633 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:16:50.633 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:16:50.633 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:50.812 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:16:50.834 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:16:50.835 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:50.835 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:16:50.835 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:16:50.835 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:16:50.835 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:51.069 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:16:51.091 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:16:51.091 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:51.092 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:16:51.092 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:16:51.092 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:16:51.092 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:51.179 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:16:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:16:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:16:51.302 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:51.302 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:16:51.302 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:16:51.302 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:16:51.302 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:51.423 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:16:51.631 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:16:51.661 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:16:51.709 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:16:51.769 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:16:51.792 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:16:51.848 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:16:51.880 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:16:51.911 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:16:51.972 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:16:52.003 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:16:52.017 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:16:52.043 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:16:52.231 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:16:52.262 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:16:52.997 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:16:53.018 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:16:53.118 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:16:53.222 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:16:53.254 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:16:53.280 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:16:53.312 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:16:53.334 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:16:53.335 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:53.335 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:16:53.335 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:16:53.335 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:16:53.335 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:53.354 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:16:53.379 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:16:53.390 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:16:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:16:53.517 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:16:53.539 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:16:53.562 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:16:53.562 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:53.562 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:16:53.563 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:16:53.563 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:16:53.563 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:53.609 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:16:53.620 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:16:53.673 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:16:54.306 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:16:54.331 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:16:54.353 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:16:54.353 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:54.353 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:16:54.354 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:16:54.354 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:16:54.354 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:54.398 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:16:54.420 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:16:54.420 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:54.420 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:16:54.420 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:54.465 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:16:54.487 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:16:54.487 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:54.487 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:16:54.488 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:16:54.488 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:16:54.488 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:54.539 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:16:54.578 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:16:54.600 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:16:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:16:54.600 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:16:54.601 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:16:54.601 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:54.645 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:16:54.667 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:16:54.667 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:54.668 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:16:54.668 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:54.712 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:16:54.734 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:16:54.734 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:54.734 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:16:54.735 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:16:54.735 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:16:54.735 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:54.811 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:16:54.834 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:16:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:16:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:16:54.834 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:16:54.834 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:16:55.046 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:16:55.068 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:16:55.069 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:16:55.069 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:16:55.069 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:16:55.069 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:16:55.069 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:16:59.212 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:16:59.263 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:16:59.293 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:16:59.319 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:16:59.524 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:16:59.527 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:16:59.568 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:16:59.598 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:16:59.634 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:16:59.634 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:16:59.675 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:16:59.732 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:16:59.793 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:16:59.826 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:16:59.869 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:16:59.911 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:16:59.942 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:16:59.954 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:16:59.983 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:16:59.985 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:17:00.014 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:17:00.039 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:17:00.086 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:17:00.099 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:17:00.137 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:17:00.163 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:17:00.183 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:17:00.226 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:17:00.270 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:17:00.301 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:17:00.330 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:17:00.412 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:17:00.468 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:17:00.495 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:17:00.786 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:00.828 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:00.847 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:01.074 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:01.119 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:01.139 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:17:01.654 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:17:01.731 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:17:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:17:01.868 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:17:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:17:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:17:01.955 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:17:01.960 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:17:01.986 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:17:01.986 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:17:02.023 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:17:02.062 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:17:02.064 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:17:02.091 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:17:02.121 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:17:02.121 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:17:02.146 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:17:02.177 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:17:02.202 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:17:02.206 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:17:02.227 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:17:02.232 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:17:02.254 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:17:02.254 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:17:02.265 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:17:02.290 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:17:02.324 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:17:02.338 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:17:02.358 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:17:02.366 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:17:02.392 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:17:02.405 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:17:02.405 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:17:02.432 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:17:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:17:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:17:02.490 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:17:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:17:02.516 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:17:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:17:02.550 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:17:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:17:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:02.575 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:17:02.609 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:17:02.643 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:17:02.677 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:17:02.715 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:17:02.748 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:17:02.751 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:02.751 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:17:02.773 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:17:02.800 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:17:02.824 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:17:02.849 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:17:02.856 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:02.856 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:02.859 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:17:02.891 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:17:02.918 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:17:02.943 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:17:02.969 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:17:03.042 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:17:03.044 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:17:03.044 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:17:03.127 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:17:03.159 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:17:03.185 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:17:03.212 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:17:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:17:03.302 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:17:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:17:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:17:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:17:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:17:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:17:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:17:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:17:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:17:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:17:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:17:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:17:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:17:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:17:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:17:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:17:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:17:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:17:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:17:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:17:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:17:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:17:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:17:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:17:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:17:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:17:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:17:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:17:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:17:08.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:17:08.407 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:17:08.408 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:17:08.408 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:17:08.408 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:17:08.408 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:17:08.408 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:17:08.408 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:17:08.424 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:17:08.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:17:08.426 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:17:08.427 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:17:08.428 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:17:08.429 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:17:08.430 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:17:08.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:17:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:17:08.433 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:17:08.434 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:17:08.435 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:17:10.698 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:10.699 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.700 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.701 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:17:10.702 INFO:teuthology.orchestra.run.smithi145.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:17:10.703 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:17:10.704 INFO:teuthology.orchestra.run.smithi145.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:17:10.705 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:17:10.706 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:17:10.707 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:17:10.708 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:17:10.709 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:17:10.710 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:17:10.711 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:10.712 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-06T07:17:11.190 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:17:11.191 INFO:teuthology.orchestra.run.smithi094.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.192 INFO:teuthology.orchestra.run.smithi094.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.193 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.194 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:17:11.195 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:17:11.196 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:17:11.197 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:17:11.198 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:17:11.199 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:17:11.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:17:11.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:17:11.202 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.203 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:17:11.204 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:17:11.204 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:17:11.204 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:17:11.204 INFO:teuthology.orchestra.run.smithi094.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:17:11.204 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:17:11.204 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:17:11.205 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:17:11.205 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:11.205 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-06T07:17:11.371 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:11.720 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:11.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:17:11.868 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:17:11.891 INFO:teuthology.orchestra.run.smithi094.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:17:11.892 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:17:11.892 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:17:11.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:17:12.112 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:17:12.134 INFO:teuthology.orchestra.run.smithi145.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:17:12.134 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:17:12.135 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:17:12.136 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:17:12.137 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:12.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:17:12.168 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:12.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:17:12.204 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:17:12.204 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:12.204 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:17:12.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:17:12.305 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:12.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:17:12.333 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:17:12.401 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:17:12.401 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:12.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:17:12.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:17:12.526 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:12.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:17:12.552 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:17:12.618 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:17:12.618 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:12.619 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:17:12.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:17:12.715 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:12.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:17:12.741 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:17:12.808 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:17:12.914 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:17:12.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-03-06T07:17:12.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /var/log/ceph 2024-03-06T07:17:12.944 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:17:12.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:17:12.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:17:13.010 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:17:13.010 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:17:13.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:17:13.084 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:17:13.085 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:17:13.085 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:17:13.085 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:13.085 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:17:13.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:17:13.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:17:13.281 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:13.281 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:17:13.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:17:13.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:17:13.446 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:17:13.446 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:17:13.447 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042', 'branch': 'wip-lusov-qdb', 'description': 'fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}', 'email': None, 'first_in_suite': False, 'job_id': '7583042', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'disabled': True}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUptUXOT7IP+rlzPS9z+dJBf1CjRETBgacSwiKFjPRRFP2vQO1vvZ/eNWBQcHv29vpje+bcaZkurfh9o6Rui0E=', 'smithi145.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLs9x8j8hoye4v7cxEJMQ6KkZsfY8EpCDIT4JcwJqJWa2nvNoegFCUiOVIwOGvganVWn+4aJUZjR+AISPQXVEc4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'kclient': None}, {'exec': {'mon.a': ['ceph config set mgr mgr/crash/warn_recent_interval 0']}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_failover']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:00:27', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:17:13.447 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:17:13.450 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:17:13.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:17:13.505 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:17:13.563 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:13.563 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:17:13.585 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:17:13.585 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-03-06T07:17:13.639 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:17:13.639 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:13.640 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-06T07:17:13.640 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:13.640 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:13.640 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-06 07:17:03.627450453 +0000 2024-03-06T07:17:13.640 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-06 07:14:33.820326511 +0000 2024-03-06T07:17:13.640 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-06 07:14:33.820326511 +0000 2024-03-06T07:17:13.640 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-06 07:14:33.820326511 +0000 2024-03-06T07:17:13.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:17:13.704 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-06T07:17:13.705 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-06T07:17:13.705 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000122749 s, 4.2 MB/s 2024-03-06T07:17:13.706 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:17:13.762 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-03-06T07:17:13.817 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:17:13.817 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:13.817 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-06T07:17:13.817 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:13.817 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:13.818 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-06 07:17:03.628450437 +0000 2024-03-06T07:17:13.818 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-06 07:14:34.161320939 +0000 2024-03-06T07:17:13.818 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-06 07:14:34.161320939 +0000 2024-03-06T07:17:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-06 07:14:34.161320939 +0000 2024-03-06T07:17:13.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:17:13.883 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-06T07:17:13.883 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-06T07:17:13.883 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.00010095 s, 5.1 MB/s 2024-03-06T07:17:13.885 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:17:13.940 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-06 07:17:03.628450437 +0000 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-06 07:14:34.507315285 +0000 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-06 07:14:34.507315285 +0000 2024-03-06T07:17:13.996 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-06 07:14:34.507315285 +0000 2024-03-06T07:17:13.997 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:17:14.061 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-06T07:17:14.061 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-06T07:17:14.061 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000100316 s, 5.1 MB/s 2024-03-06T07:17:14.062 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:17:14.118 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-03-06T07:17:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:17:14.173 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:17:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-06 07:17:03.628450437 +0000 2024-03-06T07:17:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-06 07:14:34.845309762 +0000 2024-03-06T07:17:14.174 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-06 07:14:34.845309762 +0000 2024-03-06T07:17:14.174 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-06 07:14:34.845309762 +0000 2024-03-06T07:17:14.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:17:14.239 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-06T07:17:14.239 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-06T07:17:14.239 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000118937 s, 4.3 MB/s 2024-03-06T07:17:14.240 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:17:14.296 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:14.296 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:14.296 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:17:14.313 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:17:14.313 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-03-06T07:17:14.369 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:17:14.369 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:14.369 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-06T07:17:14.369 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:14.369 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:14.369 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-06 07:17:03.383908355 +0000 2024-03-06T07:17:14.369 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-06 07:14:33.800100773 +0000 2024-03-06T07:17:14.370 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-06 07:14:33.800100773 +0000 2024-03-06T07:17:14.370 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-06 07:14:33.800100773 +0000 2024-03-06T07:17:14.370 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:17:14.436 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-06T07:17:14.436 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-06T07:17:14.436 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000208953 s, 2.5 MB/s 2024-03-06T07:17:14.438 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:17:14.494 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-03-06T07:17:14.549 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-06 07:17:03.384908333 +0000 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-06 07:14:34.142093007 +0000 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-06 07:14:34.142093007 +0000 2024-03-06T07:17:14.550 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-06 07:14:34.142093007 +0000 2024-03-06T07:17:14.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:17:14.615 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-06T07:17:14.615 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-06T07:17:14.615 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000187866 s, 2.7 MB/s 2024-03-06T07:17:14.616 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:17:14.672 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-03-06T07:17:14.727 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:17:14.727 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:14.727 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-06T07:17:14.727 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:14.728 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:14.728 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-06 07:17:03.384908333 +0000 2024-03-06T07:17:14.728 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-06 07:14:34.485085218 +0000 2024-03-06T07:17:14.728 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-06 07:14:34.485085218 +0000 2024-03-06T07:17:14.728 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-06 07:14:34.485085218 +0000 2024-03-06T07:17:14.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:17:14.793 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-06T07:17:14.793 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-06T07:17:14.793 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000210526 s, 2.4 MB/s 2024-03-06T07:17:14.794 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:17:14.849 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-06 07:17:03.384908333 +0000 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-06 07:14:34.823077543 +0000 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-06 07:14:34.823077543 +0000 2024-03-06T07:17:14.904 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-06 07:14:34.823077543 +0000 2024-03-06T07:17:14.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:17:14.969 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-06T07:17:14.969 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-06T07:17:14.970 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000244525 s, 2.1 MB/s 2024-03-06T07:17:14.970 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:17:15.026 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:15.026 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:17:15.026 INFO:tasks.ceph:Generating config... 2024-03-06T07:17:15.027 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:17:15.027 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:17:15.027 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:17:15.027 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:17:15.028 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:17:15.028 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:17:15.028 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:17:15.029 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:17:15.029 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:17:15.029 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:17:15.029 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:17:15.029 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:17:15.029 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:17:15.030 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:17:15.030 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:17:15.030 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:17:15.030 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:17:15.030 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:17:15.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:17:15.084 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:17:15.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:17:15.133 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:17:15.160 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.94'), ('mon.b', '172.21.15.145'), ('mon.c', '[v2:172.21.15.145:3301,v1:172.21.15.145:6790]')] 2024-03-06T07:17:15.160 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.94,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:17:15.161 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:15.161 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:17:15.215 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.94 --add b 172.21.15.145 --addv c '[v2:172.21.15.145:3301,v1:172.21.15.145:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: generated fsid 00dede11-ad16-411b-be77-3a57b1483d8f 2024-03-06T07:17:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = quincy 2024-03-06T07:17:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 0 2024-03-06T07:17:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:fsid 00dede11-ad16-411b-be77-3a57b1483d8f 2024-03-06T07:17:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:last_changed 2024-03-06T07:17:15.298430+0000 2024-03-06T07:17:15.299 INFO:teuthology.orchestra.run.smithi094.stdout:created 2024-03-06T07:17:15.298430+0000 2024-03-06T07:17:15.300 INFO:teuthology.orchestra.run.smithi094.stdout:min_mon_release 17 (quincy) 2024-03-06T07:17:15.300 INFO:teuthology.orchestra.run.smithi094.stdout:election_strategy: 1 2024-03-06T07:17:15.300 INFO:teuthology.orchestra.run.smithi094.stdout:0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2024-03-06T07:17:15.300 INFO:teuthology.orchestra.run.smithi094.stdout:1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-03-06T07:17:15.300 INFO:teuthology.orchestra.run.smithi094.stdout:2: [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] mon.c 2024-03-06T07:17:15.300 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:17:15.301 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:17:15.317 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 00dede11-ad16-411b-be77-3a57b1483d8f... 2024-03-06T07:17:15.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:17:15.360 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:17:15.407 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-03-06T07:17:15.407 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2024-03-06T07:17:15.408 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:17:15.409 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2024-03-06T07:17:15.410 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "172.21.15.94,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on skipped message = False 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 00dede11-ad16-411b-be77-3a57b1483d8f 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:17:15.411 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug trim objects = true 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.412 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression mode = aggressive 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 20 2024-03-06T07:17:15.413 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 20 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 10 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: osd op complaint time = 180 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2024-03-06T07:17:15.414 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:17:15.415 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:17:15.416 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: mon op complaint time = 120 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: client mount timeout = 600 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: debug client = 20 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: rados mon op timeout = 900 2024-03-06T07:17:15.417 INFO:teuthology.orchestra.run.smithi094.stdout: rados osd op timeout = 900 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.d] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout:[mds] 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout: debug mds = 20 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout: debug mds balancer = 20 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout: mds debug frag = True 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout: mds debug scatterstat = True 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout: mds op complaint time = 180 2024-03-06T07:17:15.418 INFO:teuthology.orchestra.run.smithi094.stdout: mds verify scatter = True 2024-03-06T07:17:15.419 INFO:teuthology.orchestra.run.smithi094.stdout: osd op complaint time = 180 2024-03-06T07:17:15.419 INFO:teuthology.orchestra.run.smithi094.stdout: rados mon op timeout = 900 2024-03-06T07:17:15.419 INFO:teuthology.orchestra.run.smithi094.stdout: rados osd op timeout = 900 2024-03-06T07:17:15.419 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-03-06T07:17:15.419 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-03-06T07:17:15.419 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:17:15.420 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:17:15.421 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-03-06T07:17:15.422 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "172.21.15.94,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on skipped message = False 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 00dede11-ad16-411b-be77-3a57b1483d8f 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-03-06T07:17:15.423 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug trim objects = true 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-03-06T07:17:15.424 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-03-06T07:17:15.425 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore compression mode = aggressive 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-03-06T07:17:15.426 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-03-06T07:17:15.427 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:17:15.428 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: mon op complaint time = 120 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: client mount timeout = 600 2024-03-06T07:17:15.429 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout:[mds] 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds = 20 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds balancer = 20 2024-03-06T07:17:15.430 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-06T07:17:15.431 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug frag = True 2024-03-06T07:17:15.431 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug scatterstat = True 2024-03-06T07:17:15.431 INFO:teuthology.orchestra.run.smithi145.stdout: mds op complaint time = 180 2024-03-06T07:17:15.431 INFO:teuthology.orchestra.run.smithi145.stdout: mds verify scatter = True 2024-03-06T07:17:15.431 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-03-06T07:17:15.431 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-03-06T07:17:15.431 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-03-06T07:17:15.432 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:17:15.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:17:15.528 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:17:15.529 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:15.529 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:17:15.551 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:15.551 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:17:15.606 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:17:15.606 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:15.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:17:15.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:17:15.685 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:15.686 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:15.743 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:17:15.743 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:15.743 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:17:15.743 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:17:15.783 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:15.783 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:15.838 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:17:15.838 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:17:15.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:17:15.893 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:17:15.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:17:15.959 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:17:15.961 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:17:15.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:17:16.016 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:17:16.018 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:17:16.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:17:16.139 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:17:16.141 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:17:16.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:17:16.223 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:17:16.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:17:16.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:17:16.347 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:17:16.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:17:16.375 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:17:16.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:17:16.417 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:17:16.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:17:16.475 INFO:teuthology.orchestra.run.smithi145.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:17:16.490 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:17:16.490 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:17:16.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:17:16.517 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:16.517 INFO:tasks.ceph:role: osd.0 2024-03-06T07:17:16.517 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:17:16.517 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:17:16.583 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:16.583 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:16.583 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:16.583 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:16.583 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:16.583 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:16.583 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:16.584 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:16.584 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:16.584 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:16.618 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-06T07:17:16.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-06T07:17:16.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:17:16.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:17:16.767 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:17:16.834 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:16.834 INFO:tasks.ceph:role: osd.1 2024-03-06T07:17:16.834 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:17:16.835 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:17:16.900 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:16.901 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:16.933 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-06T07:17:16.935 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-06T07:17:16.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:17:17.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:17:17.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:17:17.142 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:17.142 INFO:tasks.ceph:role: osd.2 2024-03-06T07:17:17.142 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:17:17.142 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:17:17.208 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:17.208 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:17.208 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:17.209 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:17.209 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:17.209 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:17.209 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:17.240 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-06T07:17:17.242 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-06T07:17:17.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:17:17.312 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:17:17.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:17:17.450 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:17.451 INFO:tasks.ceph:role: osd.3 2024-03-06T07:17:17.451 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:17:17.451 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:17:17.516 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:17.516 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:17.517 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:17.549 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-06T07:17:17.550 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-06T07:17:17.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:17:17.627 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:17:17.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:17.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:17.796+0000 7f6e78413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:17:17.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:17.796+0000 7f6e78413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:17:17.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:17.796+0000 7f6e78413640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:17:17.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:17.796+0000 7f6e78413640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:17:17.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:17.796+0000 7f6e78413640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:17:17.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:17.797+0000 7f6e78413640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:17:20.651 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:17:20.683 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:20.772+0000 7f09f360f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:17:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:20.772+0000 7f09f360f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:17:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:20.772+0000 7f09f360f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:17:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:20.772+0000 7f09f360f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:17:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:20.772+0000 7f09f360f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:17:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:20.772+0000 7f09f360f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:17:23.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:17:23.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:23.743+0000 7fc8e1048640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:17:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:23.743+0000 7fc8e1048640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:17:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:23.743+0000 7fc8e1048640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:17:23.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:23.743+0000 7fc8e1048640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:17:23.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:23.743+0000 7fc8e1048640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:17:23.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:23.743+0000 7fc8e1048640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:17:26.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:17:26.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:26.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:26.715+0000 7ff79180c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:17:26.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:26.715+0000 7ff79180c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:17:26.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:26.715+0000 7ff79180c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:17:26.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:26.715+0000 7ff79180c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:17:26.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:26.715+0000 7ff79180c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:17:26.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:26.715+0000 7ff79180c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:17:29.564 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:17:29.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:17:29.629 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:29.629 INFO:tasks.ceph:role: osd.4 2024-03-06T07:17:29.629 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:17:29.630 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:17:29.706 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:29.706 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:29.706 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:29.706 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:29.706 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:29.707 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:29.707 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:29.707 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:29.707 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:29.707 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:30.215 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-06T07:17:30.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-06T07:17:30.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:17:30.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:17:30.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:17:30.474 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:30.474 INFO:tasks.ceph:role: osd.5 2024-03-06T07:17:30.475 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:17:30.475 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:17:30.549 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:30.550 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:30.988 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-06T07:17:30.991 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-06T07:17:30.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:17:31.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:17:31.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:17:31.189 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:31.190 INFO:tasks.ceph:role: osd.6 2024-03-06T07:17:31.190 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:17:31.190 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:17:31.263 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:31.264 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:31.264 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:31.264 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:31.264 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:31.264 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:31.265 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:31.265 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:31.265 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:31.265 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:31.743 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-06T07:17:31.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-06T07:17:31.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:17:31.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:17:31.911 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:17:31.978 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:17:31.978 INFO:tasks.ceph:role: osd.7 2024-03-06T07:17:31.978 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:17:31.978 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:17:32.052 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:17:32.052 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:17:32.052 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:17:32.052 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:17:32.053 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:17:32.053 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:17:32.053 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:17:32.053 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:17:32.053 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:17:32.053 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:17:32.537 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-06T07:17:32.539 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-06T07:17:32.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:17:32.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:17:32.706 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:32.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:32.804+0000 7facc6813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:17:32.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:32.804+0000 7facc6813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:17:32.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:32.804+0000 7facc6813640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:17:32.806 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:32.804+0000 7facc6813640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:17:32.807 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:32.804+0000 7facc6813640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:17:32.807 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:32.806+0000 7facc6813640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:17:35.656 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:17:35.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:35.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:35.776+0000 7fa126aae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:17:35.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:35.777+0000 7fa126aae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:17:35.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:35.777+0000 7fa126aae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:17:35.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:35.777+0000 7fa126aae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:17:35.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:35.777+0000 7fa126aae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:17:35.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:35.777+0000 7fa126aae640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:17:38.624 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:17:38.655 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:38.748 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:38.746+0000 7fb460613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:17:38.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:38.747+0000 7fb460613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:17:38.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:38.747+0000 7fb460613640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:17:38.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:38.747+0000 7fb460613640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:17:38.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:38.747+0000 7fb460613640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:17:38.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:38.747+0000 7fb460613640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:17:41.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:17:41.633 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:41.726 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:41.724+0000 7fcb72a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:17:41.727 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:41.725+0000 7fcb72a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:17:41.727 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:41.725+0000 7fcb72a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:17:41.727 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:41.725+0000 7fcb72a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:17:41.727 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:41.725+0000 7fcb72a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:17:41.727 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:41.725+0000 7fcb72a13640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:17:43.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:17:43.452 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:17:44.573 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:17:44.603 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:17:44.603 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:44.603 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:17:44.630 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:44.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:17:44.698 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:44.698 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:17:44.764 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:44.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:17:44.831 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:44.831 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:17:44.897 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:44.898 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:17:44.965 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:44.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:17:45.032 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:17:45.065 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.065 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:17:45.136 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.137 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:17:45.206 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.206 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:17:45.275 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:17:45.344 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.344 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:17:45.412 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.412 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:17:45.480 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:45.480 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:17:45.497 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:45.497 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:17:45.536 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:17:45.536 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:17:45.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:17:45.580 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.y] 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg== 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a] 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg== 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c] 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA== 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA== 2024-03-06T07:17:45.581 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q== 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg== 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ== 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA== 2024-03-06T07:17:45.582 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b] 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA== 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.d] 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ== 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.4] 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q== 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.5] 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw== 2024-03-06T07:17:45.583 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.6] 2024-03-06T07:17:45.584 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g== 2024-03-06T07:17:45.584 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.7] 2024-03-06T07:17:45.584 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g== 2024-03-06T07:17:45.584 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:17:45.584 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:17:45.584 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2024-03-06T07:17:45.584 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg== 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.y] 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg== 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg== 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA== 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.0] 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA== 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.1] 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q== 2024-03-06T07:17:45.602 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.2] 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg== 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.3] 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ== 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.x] 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA== 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA== 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ== 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.4] 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q== 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.5] 2024-03-06T07:17:45.603 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw== 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.6] 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g== 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.7] 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g== 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout:[client.0] 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout:[client.1] 2024-03-06T07:17:45.604 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg== 2024-03-06T07:17:45.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:17:45.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:17:45.698 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:45.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:45.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:45.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:45.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:45.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:45.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:45.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:45.919 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:45.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:45.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:45.975 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.026 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:17:46.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:17:46.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:46.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:46.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:46.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:17:46.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.199 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.362 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:17:46.416 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:17:46.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:17:46.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:17:46.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:17:46.531 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:17:46.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:17:46.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:17:46.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:17:46.684 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:17:46.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:17:46.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:17:46.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:17:46.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:17:47.002 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:17:47.028 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:47.030 DEBUG:teuthology.orchestra.run.smithi145:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:17:47.083 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:17:47.083 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:17:47.083 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:17:47.086 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:17:47.086 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:17:47.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:17:47.125 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:17:47.126 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:17:47.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:17:47.127 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:17:47.128 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:17:47.129 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:17:47.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:17:47.132 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:17:47.132 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:17:47.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:17:47.134 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:17:47.135 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:17:47.135 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:17:47.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.234+0000 7f3aba1ae640 1 Processor -- start 2024-03-06T07:17:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.234+0000 7f3aba1ae640 1 -- start start 2024-03-06T07:17:47.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3aba1ae640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:47.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3aba1ae640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:47.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3aba1ae640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab4073580 con 0x7f3ab40c7f60 2024-03-06T07:17:47.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3aba1ae640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab4070b20 con 0x7f3ab41505f0 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3aba1ae640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab40701f0 con 0x7f3ab40c6710 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54696/0 (socket says 172.21.15.94:54696) 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 learned_addr learned my addr 172.21.15.94:0/1359229058 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.235+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:47.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.236+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:17:47.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.236+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:47.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.236+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:47.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.236+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:47.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.436+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:47.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.436+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:47.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.436+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:47.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.436+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:47.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.436+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:17:47.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.437+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:47.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.437+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:47.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.437+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:47.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.437+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:47.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.437+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:17:47.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.837+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:47.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.837+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:47.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.837+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:47.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.837+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:47.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.837+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:17:47.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.838+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:47.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.838+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:47.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.838+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:47.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.838+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:47.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:47.838+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:17:48.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.638+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:48.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.638+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.638+0000 7f3ab37fe640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.638+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.638+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:17:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.639+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.639+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.639+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.639+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:48.639+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:17:50.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 msgr2=0x7f3ab40c6b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:50.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.145:3300/0 conn(0x7f3ab40c6710 0x7f3ab40c6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:50.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 -- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 msgr2=0x7f3ab4156ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:50.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 --2- 172.21.15.94:0/1359229058 >> v2:172.21.15.94:3300/0 conn(0x7f3ab41505f0 0x7f3ab4156ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:50.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 -- 172.21.15.94:0/1359229058 >> v1:172.21.15.145:6789/0 conn(0x7f3ab40c7f60 legacy=0x7f3ab4072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 -- 172.21.15.94:0/1359229058 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab40704d0 con 0x7f3aa00036b0 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.235+0000 7f3ab17fa640 1 -- 172.21.15.94:0/1359229058 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab4070b20 con 0x7f3aa00072e0 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.236+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.236+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.236+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.236+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:50.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.236+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:50.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.437+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:50.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.437+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:17:50.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.437+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:50.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.437+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:50.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.437+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:17:50.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.838+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:50.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.838+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:17:50.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.838+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:50.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.838+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:50.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:50.838+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:17:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:51.640+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:51.640+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:17:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:51.640+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:51.640+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:51.640+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:17:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317832192 0 0) 0x7f3ab40704d0 con 0x7f3aa00036b0 2024-03-06T07:17:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a9c003520 con 0x7f3aa00036b0 2024-03-06T07:17:52.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1419605471 0 0) 0x7f3a9c003520 con 0x7f3aa00036b0 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 msgr2=0x7f3aa00097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3aa00072e0 0x7f3aa00097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa000af60 con 0x7f3aa00036b0 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f3aa400f040 con 0x7f3aa00036b0 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 >> v1:172.21.15.94:6789/0 conn(0x7f3aa00036b0 legacy=0x7f3aa0005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 reap_dead start 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3a9c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.418+0000 7f3ab27fc640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3a9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab27fc640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3a9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab4070b20 con 0x7f3a9c005290 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c00c1e0 con 0x7f3a9c0039b0 2024-03-06T07:17:52.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c00c3e0 con 0x7f3a9c004800 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3a9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3a9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3a9c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 msgr2=0x7f3a9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3a9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 msgr2=0x7f3a9c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3a9c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.419+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1359229058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c003520 con 0x7f3a9c005290 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.420+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3a9c00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3ab406bf40 tx=0x7f3a9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad88f5d93f7ad4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.420+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac03a070 con 0x7f3a9c005290 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.420+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac035410 con 0x7f3a9c005290 2024-03-06T07:17:52.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.420+0000 7f3ab27fc640 1 -- 172.21.15.94:0/1359229058 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac003db0 con 0x7f3a9c005290 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1359229058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 msgr2=0x7f3a9c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3a9c00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3ab406bf40 tx=0x7f3a9c005830 comp rx=0 tx=0).stop 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1359229058 shutdown_connections 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3a9c00b9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3a9c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1359229058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3a9c003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1359229058 >> 172.21.15.94:0/1359229058 conn(0x7f3ab40aa5c0 msgr2=0x7f3ab40aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1359229058 shutdown_connections 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.421+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1359229058 wait complete. 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.423+0000 7f3aba1ae640 1 Processor -- start 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.423+0000 7f3aba1ae640 1 -- start start 2024-03-06T07:17:52.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3aba1ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3ab40c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3aba1ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3ab4155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3aba1ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3ab40aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3aba1ae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab41509d0 con 0x7f3a9c005290 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3aba1ae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab406fec0 con 0x7f3a9c0039b0 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3aba1ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab4150b50 con 0x7f3a9c004800 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3ab40aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3ab40aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35016/0 (socket says 172.21.15.94:35016) 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1660669668 learned_addr learned my addr 172.21.15.94:0/1660669668 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3ab4155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab37fe640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3ab40c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 msgr2=0x7f3ab4155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab2ffd640 1 -- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 msgr2=0x7f3ab4155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:52.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 msgr2=0x7f3ab4155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3ab4155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab2ffd640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3ab4155b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3ab4155b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 msgr2=0x7f3ab40c1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3ab40c1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 -- 172.21.15.94:0/1660669668 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab41e8990 con 0x7f3a9c005290 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.424+0000 7f3ab3fff640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3ab40aced0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3aac002230 tx=0x7f3aac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.425+0000 7f3a9bfff640 1 -- 172.21.15.94:0/1660669668 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac03a040 con 0x7f3a9c005290 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.425+0000 7f3a9bfff640 1 -- 172.21.15.94:0/1660669668 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac048070 con 0x7f3a9c005290 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.425+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab41e8c30 con 0x7f3a9c005290 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.425+0000 7f3a9bfff640 1 -- 172.21.15.94:0/1660669668 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac041ad0 con 0x7f3a9c005290 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.425+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab40ae400 con 0x7f3a9c005290 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.425+0000 7f3a9bfff640 1 -- 172.21.15.94:0/1660669668 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3aac03d040 con 0x7f3a9c005290 2024-03-06T07:17:52.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.425+0000 7f3a9bfff640 1 -- 172.21.15.94:0/1660669668 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3aac04d9b0 con 0x7f3a9c005290 2024-03-06T07:17:52.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.426+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab404f330 con 0x7f3a9c005290 2024-03-06T07:17:52.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.428+0000 7f3a9bfff640 1 -- 172.21.15.94:0/1660669668 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3aac04d460 con 0x7f3a9c005290 2024-03-06T07:17:52.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.493+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3ab40ae6d0 con 0x7f3a9c005290 2024-03-06T07:17:52.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3a9bfff640 1 -- 172.21.15.94:0/1660669668 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3aac041c70 con 0x7f3a9c005290 2024-03-06T07:17:52.496 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2024-03-06T07:17:52.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 msgr2=0x7f3ab40aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:52.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3ab40aced0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3aac002230 tx=0x7f3aac004a40 comp rx=0 tx=0).stop 2024-03-06T07:17:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 shutdown_connections 2024-03-06T07:17:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a9c005290 0x7f3ab40aced0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a9c004800 0x7f3ab4155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3aba1ae640 1 --2- 172.21.15.94:0/1660669668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a9c0039b0 0x7f3ab40c1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.495+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 >> 172.21.15.94:0/1660669668 conn(0x7f3ab40aa5c0 msgr2=0x7f3ab40c2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.496+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 shutdown_connections 2024-03-06T07:17:52.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:52.496+0000 7f3aba1ae640 1 -- 172.21.15.94:0/1660669668 wait complete. 2024-03-06T07:17:52.508 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:17:52.509 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:17:52.509 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:52.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:17:52.537 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:52.537 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:17:52.604 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:52.604 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:17:52.671 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:17:52.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:17:52.737 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:52.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:17:52.763 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:52.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:17:52.830 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:52.830 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:17:52.897 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:17:52.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:17:52.912 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:52.909+0000 7fe171640040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:17:52.938 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 51d664c3-2eb9-4a1e-b855-ad028426728c 0 2024-03-06T07:17:52.969 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:52.966+0000 7fe171640040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:17:53.075 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.065+0000 7f54856d4640 1 Processor -- start 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f54856d4640 1 -- start start 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f54856d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f54800725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f54856d4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800c85f0 con 0x7f5480072ae0 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f54856d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800a7e70 con 0x7f5480155b70 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f54856d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54800a8430 con 0x7f54800c8210 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f547effd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f54800725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f547f7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5480155b70 0x7f54800c5430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:57120/0 (socket says 172.21.15.145:57120) 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f547f7fe640 1 -- 172.21.15.145:0/1516623366 learned_addr learned my addr 172.21.15.145:0/1516623366 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-06T07:17:53.076 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f547effd640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 msgr2=0x7f54800725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f547effd640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 msgr2=0x7f54800725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f547effd640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f54800725a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.066+0000 7f547effd640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f54800725a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148270732 0 0) 0x7f54800a7e70 con 0x7f5480155b70 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f546c003520 con 0x7f5480155b70 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144862617 0 0) 0x7f54800c85f0 con 0x7f5480072ae0 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54800a7e70 con 0x7f5480072ae0 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 <== mon.1 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 300109934 0 0) 0x7f546c003520 con 0x7f5480155b70 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 msgr2=0x7f54800725a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f54800725a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 >> v1:172.21.15.94:6789/0 conn(0x7f5480072ae0 legacy=0x7f54800c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54800a8150 con 0x7f5480155b70 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 <== mon.1 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f54780019f0 con 0x7f5480155b70 2024-03-06T07:17:53.077 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 >> v1:172.21.15.145:6789/0 conn(0x7f5480155b70 legacy=0x7f54800c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f546c003b40 0x7f546c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f546c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f546c004d40 0x7f546c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54800a8430 con 0x7f546c003b40 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c00bb50 con 0x7f546c004d40 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c00bd50 con 0x7f54800c8210 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547f7fe640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f546c004d40 0x7f546c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547effd640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f546c003b40 0x7f546c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f5477fff640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f546c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f5477fff640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 msgr2=0x7f546c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f5477fff640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 msgr2=0x7f546c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f5477fff640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f546c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.078 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f5477fff640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f546c004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547f7fe640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 msgr2=0x7f546c004800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547f7fe640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54800c8210 0x7f546c004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547f7fe640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f546c003b40 msgr2=0x7f546c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547f7fe640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f546c003b40 0x7f546c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.067+0000 7f547f7fe640 1 -- 172.21.15.145:0/1516623366 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c003520 con 0x7f546c004d40 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.068+0000 7f547f7fe640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f546c004d40 0x7f546c0051c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f547800a7c0 tx=0x7f546c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3a10c69f5eb2dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.068+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547800a410 con 0x7f546c004d40 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.068+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5478034650 con 0x7f546c004d40 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.068+0000 7f547e7fc640 1 -- 172.21.15.145:0/1516623366 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478034a40 con 0x7f546c004d40 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.069+0000 7f54856d4640 1 -- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f546c004d40 msgr2=0x7f546c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.069+0000 7f54856d4640 1 --2- 172.21.15.145:0/1516623366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f546c004d40 0x7f546c0051c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f547800a7c0 tx=0x7f546c005830 comp rx=0 tx=0).stop 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.069+0000 7f547f7fe640 1 -- 172.21.15.145:0/1516623366 reap_dead start 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.069+0000 7f54856d4640 1 -- 172.21.15.145:0/1516623366 shutdown_connections 2024-03-06T07:17:53.079 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.069+0000 7f54856d4640 1 -- 172.21.15.145:0/1516623366 >> 172.21.15.145:0/1516623366 conn(0x7f54800c88f0 msgr2=0x7f54800c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.069+0000 7f54856d4640 1 -- 172.21.15.145:0/1516623366 shutdown_connections 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.069+0000 7f54856d4640 1 -- 172.21.15.145:0/1516623366 wait complete. 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.070+0000 7f54856d4640 1 Processor -- start 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.070+0000 7f54856d4640 1 -- start start 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54856d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5480156860 0x7f5480156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54856d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54801eb540 0x7f54800aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54856d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 0x7f54800aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54856d4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5480073210 con 0x7f5480156860 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54856d4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5480073090 con 0x7f54801eb540 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54856d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5480073390 con 0x7f5480152360 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547f7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 0x7f54800aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f5477fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54801eb540 0x7f54800aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.080 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547f7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 0x7f54800aafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:55244/0 (socket says 172.21.15.145:55244) 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547f7fe640 1 -- 172.21.15.145:0/3824380560 learned_addr learned my addr 172.21.15.145:0/3824380560 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547f7fe640 1 -- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 msgr2=0x7f54800aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547f7fe640 1 -- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 msgr2=0x7f54800aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547f7fe640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 0x7f54800aafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547f7fe640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 0x7f54800aafc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f547effd640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5480156860 0x7f5480156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f5477fff640 1 -- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 msgr2=0x7f54800aafc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f5477fff640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 0x7f54800aafc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f5477fff640 1 -- 172.21.15.145:0/3824380560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5480156860 msgr2=0x7f5480156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f5477fff640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5480156860 0x7f5480156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f5477fff640 1 -- 172.21.15.145:0/3824380560 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54801eefa0 con 0x7f54801eb540 2024-03-06T07:17:53.081 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f5477fff640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54801eb540 0x7f54800aaa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f546800e550 tx=0x7f546800ea10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54777fe640 1 -- 172.21.15.145:0/3824380560 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54680052e0 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54777fe640 1 -- 172.21.15.145:0/3824380560 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5468018070 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54777fe640 1 -- 172.21.15.145:0/3824380560 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468003830 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.071+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54801ee1e0 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.072+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54801ee7a0 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.072+0000 7f54777fe640 1 -- 172.21.15.145:0/3824380560 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f54680039d0 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.072+0000 7f54777fe640 1 -- 172.21.15.145:0/3824380560 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5468013420 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.073+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548004f330 con 0x7f54801eb540 2024-03-06T07:17:53.082 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.075+0000 7f54777fe640 1 -- 172.21.15.145:0/3824380560 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f546801d460 con 0x7f54801eb540 2024-03-06T07:17:53.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.145+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "51d664c3-2eb9-4a1e-b855-ad028426728c", "id": 0} v 0) v1 -- 0x7f54800c2130 con 0x7f54801eb540 2024-03-06T07:17:53.153 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.150+0000 7f54777fe640 1 -- 172.21.15.145:0/3824380560 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "51d664c3-2eb9-4a1e-b855-ad028426728c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f546801d640 con 0x7f54801eb540 2024-03-06T07:17:53.153 INFO:teuthology.orchestra.run.smithi145.stdout:0 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54801eb540 msgr2=0x7f54800aaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54801eb540 0x7f54800aaa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f546800e550 tx=0x7f546800ea10 comp rx=0 tx=0).stop 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 shutdown_connections 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5480152360 0x7f54800aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54801eb540 0x7f54800aaa80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 --2- 172.21.15.145:0/3824380560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5480156860 0x7f5480156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 >> 172.21.15.145:0/3824380560 conn(0x7f54800c88f0 msgr2=0x7f54800c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 shutdown_connections 2024-03-06T07:17:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.151+0000 7f54856d4640 1 -- 172.21.15.145:0/3824380560 wait complete. 2024-03-06T07:17:53.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new a739cd3d-4564-4d3d-8624-a4d4ddbc0e14 1 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14ad73e640 1 Processor -- start 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14ad73e640 1 -- start start 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14ad73e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14a80b0070 0x7f14a8155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14ad73e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f14a80c5380 0x7f14a80c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14ad73e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a80b0450 con 0x7f14a806ca70 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14ad73e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a806c020 con 0x7f14a80b0070 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14ad73e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a806cf20 con 0x7f14a80c5380 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.264+0000 7f14a77fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f14a80c5380 0x7f14a80c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.270 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.265+0000 7f14a77fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f14a80c5380 0x7f14a80c5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34896/0 (socket says 172.21.15.145:34896) 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.265+0000 7f14a77fe640 1 -- 172.21.15.145:0/4139147300 learned_addr learned my addr 172.21.15.145:0/4139147300 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.265+0000 7f14a6ffd640 1 --2- 172.21.15.145:0/4139147300 >> v2:172.21.15.94:3300/0 conn(0x7f14a80b0070 0x7f14a8155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.265+0000 7f14a77fe640 1 -- 172.21.15.145:0/4139147300 >> v2:172.21.15.94:3300/0 conn(0x7f14a80b0070 msgr2=0x7f14a8155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.265+0000 7f14a77fe640 1 --2- 172.21.15.145:0/4139147300 >> v2:172.21.15.94:3300/0 conn(0x7f14a80b0070 0x7f14a8155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.265+0000 7f14a77fe640 1 -- 172.21.15.145:0/4139147300 >> v1:172.21.15.94:6789/0 conn(0x7f14a806ca70 legacy=0x7f14a80c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.265+0000 7f14a77fe640 1 -- 172.21.15.145:0/4139147300 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a80c77f0 con 0x7f14a80c5380 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14a77fe640 1 --2- 172.21.15.145:0/4139147300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a80c5380 0x7f14a80c5790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f14a80be030 tx=0x7f14a002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91d177d9d6e39422 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14a67fc640 1 -- 172.21.15.145:0/4139147300 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a002ca80 con 0x7f14a80c5380 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14a67fc640 1 -- 172.21.15.145:0/4139147300 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a002cc20 con 0x7f14a80c5380 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14a67fc640 1 -- 172.21.15.145:0/4139147300 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a00343f0 con 0x7f14a80c5380 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 -- 172.21.15.145:0/4139147300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a80c5380 msgr2=0x7f14a80c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 --2- 172.21.15.145:0/4139147300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a80c5380 0x7f14a80c5790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f14a80be030 tx=0x7f14a002bee0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 -- 172.21.15.145:0/4139147300 shutdown_connections 2024-03-06T07:17:53.271 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 --2- 172.21.15.145:0/4139147300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a80c5380 0x7f14a80c5790 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 --2- 172.21.15.145:0/4139147300 >> v2:172.21.15.94:3300/0 conn(0x7f14a80b0070 0x7f14a8155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 -- 172.21.15.145:0/4139147300 >> 172.21.15.145:0/4139147300 conn(0x7f14a80ac590 msgr2=0x7f14a80ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 -- 172.21.15.145:0/4139147300 shutdown_connections 2024-03-06T07:17:53.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.266+0000 7f14ad73e640 1 -- 172.21.15.145:0/4139147300 wait complete. 2024-03-06T07:17:53.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 Processor -- start 2024-03-06T07:17:53.272 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 -- start start 2024-03-06T07:17:53.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a806ca70 0x7f14a81547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.274 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a80b0070 0x7f14a81edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 0x7f14a814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a806c550 con 0x7f14a80b0070 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a806c270 con 0x7f14a806ca70 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14ad73e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a806da80 con 0x7f14a80c5380 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a6ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a806ca70 0x7f14a81547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a6ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a806ca70 0x7f14a81547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34904/0 (socket says 172.21.15.145:34904) 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a6ffd640 1 -- 172.21.15.145:0/2543392198 learned_addr learned my addr 172.21.15.145:0/2543392198 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a77fe640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 0x7f14a814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f149ffff640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a80b0070 0x7f14a81edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a77fe640 1 -- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 msgr2=0x7f14a814dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a77fe640 1 -- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 msgr2=0x7f14a814dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a77fe640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 0x7f14a814dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.268+0000 7f14a77fe640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 0x7f14a814dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14a6ffd640 1 -- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 msgr2=0x7f14a814dc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14a6ffd640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 0x7f14a814dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14a6ffd640 1 -- 172.21.15.145:0/2543392198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a80b0070 msgr2=0x7f14a81edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14a6ffd640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a80b0070 0x7f14a81edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14a6ffd640 1 -- 172.21.15.145:0/2543392198 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a80c4020 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14a6ffd640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a806ca70 0x7f14a81547d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f149800e9b0 tx=0x7f149800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f149f7fe640 1 -- 172.21.15.145:0/2543392198 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498017070 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f149f7fe640 1 -- 172.21.15.145:0/2543392198 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149800c9a0 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a80c42c0 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f149f7fe640 1 -- 172.21.15.145:0/2543392198 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800cd90 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a80c1740 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f149f7fe640 1 -- 172.21.15.145:0/2543392198 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f149801a070 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.269+0000 7f149f7fe640 1 -- 172.21.15.145:0/2543392198 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f1498013820 con 0x7f14a806ca70 2024-03-06T07:17:53.276 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.270+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a804f330 con 0x7f14a806ca70 2024-03-06T07:17:53.277 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.272+0000 7f149f7fe640 1 -- 172.21.15.145:0/2543392198 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1498006700 con 0x7f14a806ca70 2024-03-06T07:17:53.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.345+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a739cd3d-4564-4d3d-8624-a4d4ddbc0e14", "id": 1} v 0) v1 -- 0x7f14a806d860 con 0x7f14a806ca70 2024-03-06T07:17:53.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f149f7fe640 1 -- 172.21.15.145:0/2543392198 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a739cd3d-4564-4d3d-8624-a4d4ddbc0e14", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1498006020 con 0x7f14a806ca70 2024-03-06T07:17:53.355 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-03-06T07:17:53.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a806ca70 msgr2=0x7f14a81547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f14ad73e640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a806ca70 0x7f14a81547d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f149800e9b0 tx=0x7f149800ee70 comp rx=0 tx=0).stop 2024-03-06T07:17:53.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 shutdown_connections 2024-03-06T07:17:53.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f14ad73e640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a80c5380 0x7f14a814dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f14ad73e640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a80b0070 0x7f14a81edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f14ad73e640 1 --2- 172.21.15.145:0/2543392198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a806ca70 0x7f14a81547d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.353+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 >> 172.21.15.145:0/2543392198 conn(0x7f14a80ac590 msgr2=0x7f14a8151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.354+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 shutdown_connections 2024-03-06T07:17:53.357 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.354+0000 7f14ad73e640 1 -- 172.21.15.145:0/2543392198 wait complete. 2024-03-06T07:17:53.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 93c1acd5-d24b-42b9-8654-dfec597d57b3 2 2024-03-06T07:17:53.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.467+0000 7f5b198f4640 1 Processor -- start 2024-03-06T07:17:53.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.467+0000 7f5b198f4640 1 -- start start 2024-03-06T07:17:53.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b198f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 0x7f5b1406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b198f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5b140c6080 0x7f5b14072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.476 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b198f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5b14072bc0 0x7f5b14153030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b198f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1406ec50 con 0x7f5b14072bc0 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b198f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b14156650 con 0x7f5b140c6080 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b198f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b141568f0 con 0x7f5b140c4e60 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5b140c6080 0x7f5b14072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b12ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 0x7f5b1406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5b140c6080 0x7f5b14072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34908/0 (socket says 172.21.15.145:34908) 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b12ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 0x7f5b1406f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:55256/0 (socket says 172.21.15.145:55256) 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 -- 172.21.15.145:0/2696639352 learned_addr learned my addr 172.21.15.145:0/2696639352 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b137fe640 1 --2- 172.21.15.145:0/2696639352 >> v2:172.21.15.94:3300/0 conn(0x7f5b14072bc0 0x7f5b14153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b12ffd640 1 -- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 msgr2=0x7f5b1406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b12ffd640 1 -- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 msgr2=0x7f5b1406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b12ffd640 1 --2- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 0x7f5b1406f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b12ffd640 1 --2- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 0x7f5b1406f7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.477 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 -- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 msgr2=0x7f5b1406f7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 --2- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 0x7f5b1406f7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 -- 172.21.15.145:0/2696639352 >> v2:172.21.15.94:3300/0 conn(0x7f5b14072bc0 msgr2=0x7f5b14153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 --2- 172.21.15.145:0/2696639352 >> v2:172.21.15.94:3300/0 conn(0x7f5b14072bc0 0x7f5b14153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.468+0000 7f5b127fc640 1 -- 172.21.15.145:0/2696639352 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14156bd0 con 0x7f5b140c6080 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.469+0000 7f5b127fc640 1 --2- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c6080 0x7f5b14072680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5b140ab830 tx=0x7f5b0402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05f89eddf36717f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.469+0000 7f5b11ffb640 1 -- 172.21.15.145:0/2696639352 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04035020 con 0x7f5b140c6080 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.469+0000 7f5b11ffb640 1 -- 172.21.15.145:0/2696639352 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b04031870 con 0x7f5b140c6080 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.469+0000 7f5b11ffb640 1 -- 172.21.15.145:0/2696639352 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04031c40 con 0x7f5b140c6080 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 -- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c6080 msgr2=0x7f5b14072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c6080 0x7f5b14072680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5b140ab830 tx=0x7f5b0402b910 comp rx=0 tx=0).stop 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 -- 172.21.15.145:0/2696639352 shutdown_connections 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2696639352 >> v2:172.21.15.94:3300/0 conn(0x7f5b14072bc0 0x7f5b14153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c6080 0x7f5b14072680 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2696639352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c4e60 0x7f5b1406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 -- 172.21.15.145:0/2696639352 >> 172.21.15.145:0/2696639352 conn(0x7f5b140c5560 msgr2=0x7f5b140c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 -- 172.21.15.145:0/2696639352 shutdown_connections 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.470+0000 7f5b198f4640 1 -- 172.21.15.145:0/2696639352 wait complete. 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.471+0000 7f5b198f4640 1 Processor -- start 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.471+0000 7f5b198f4640 1 -- start start 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b14072bc0 0x7f5b141e9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c4e60 0x7f5b141ecd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 0x7f5b141e00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140c87d0 con 0x7f5b14072bc0 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b14157190 con 0x7f5b140c4e60 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b14153570 con 0x7f5b140c6080 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c4e60 0x7f5b141ecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b137fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 0x7f5b141e00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c4e60 0x7f5b141ecd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34918/0 (socket says 172.21.15.145:34918) 2024-03-06T07:17:53.479 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 -- 172.21.15.145:0/2314312617 learned_addr learned my addr 172.21.15.145:0/2314312617 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b12ffd640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b14072bc0 0x7f5b141e9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b137fe640 1 -- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 msgr2=0x7f5b141e00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b137fe640 1 -- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 msgr2=0x7f5b141e00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b137fe640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 0x7f5b141e00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b137fe640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 0x7f5b141e00e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 -- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 msgr2=0x7f5b141e00e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 0x7f5b141e00e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 -- 172.21.15.145:0/2314312617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b14072bc0 msgr2=0x7f5b141e9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b14072bc0 0x7f5b141e9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 -- 172.21.15.145:0/2314312617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b141ed2d0 con 0x7f5b140c4e60 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b127fc640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c4e60 0x7f5b141ecd90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b04000c00 tx=0x7f5b040046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5afbfff640 1 -- 172.21.15.145:0/2314312617 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04035040 con 0x7f5b140c4e60 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5afbfff640 1 -- 172.21.15.145:0/2314312617 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b04030d00 con 0x7f5b140c4e60 2024-03-06T07:17:53.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b141558f0 con 0x7f5b140c4e60 2024-03-06T07:17:53.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5afbfff640 1 -- 172.21.15.145:0/2314312617 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b040456a0 con 0x7f5b140c4e60 2024-03-06T07:17:53.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.472+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b140abd40 con 0x7f5b140c4e60 2024-03-06T07:17:53.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.473+0000 7f5afbfff640 1 -- 172.21.15.145:0/2314312617 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5b04038020 con 0x7f5b140c4e60 2024-03-06T07:17:53.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.473+0000 7f5afbfff640 1 -- 172.21.15.145:0/2314312617 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f5b0403aca0 con 0x7f5b140c4e60 2024-03-06T07:17:53.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.474+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1404f330 con 0x7f5b140c4e60 2024-03-06T07:17:53.482 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.476+0000 7f5afbfff640 1 -- 172.21.15.145:0/2314312617 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5b04053080 con 0x7f5b140c4e60 2024-03-06T07:17:53.522 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.519+0000 7fe171640040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:17:53.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.547+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "93c1acd5-d24b-42b9-8654-dfec597d57b3", "id": 2} v 0) v1 -- 0x7f5b140c8330 con 0x7f5b140c4e60 2024-03-06T07:17:53.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.550+0000 7f5afbfff640 1 -- 172.21.15.145:0/2314312617 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93c1acd5-d24b-42b9-8654-dfec597d57b3", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5b04045db0 con 0x7f5b140c4e60 2024-03-06T07:17:53.552 INFO:teuthology.orchestra.run.smithi145.stdout:2 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c4e60 msgr2=0x7f5b141ecd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c4e60 0x7f5b141ecd90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b04000c00 tx=0x7f5b040046c0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 shutdown_connections 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b140c6080 0x7f5b141e00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b140c4e60 0x7f5b141ecd90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 --2- 172.21.15.145:0/2314312617 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b14072bc0 0x7f5b141e9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 >> 172.21.15.145:0/2314312617 conn(0x7f5b140c5560 msgr2=0x7f5b140c8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 shutdown_connections 2024-03-06T07:17:53.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.551+0000 7f5b198f4640 1 -- 172.21.15.145:0/2314312617 wait complete. 2024-03-06T07:17:53.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new e268a931-5385-498e-8bc4-28a44c6bf4cf 3 2024-03-06T07:17:53.584 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.581+0000 7fe171640040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:17:53.631 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.628+0000 7fe171640040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:17:53.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.666+0000 7fab6fa9d640 1 Processor -- start 2024-03-06T07:17:53.669 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.666+0000 7fab6fa9d640 1 -- start start 2024-03-06T07:17:53.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6fa9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab680b0070 0x7fab68155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.670 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6fa9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6fa9d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab680b0450 con 0x7fab680c5380 2024-03-06T07:17:53.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6fa9d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6806c020 con 0x7fab680b0070 2024-03-06T07:17:53.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6fa9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6806cf20 con 0x7fab6806ca70 2024-03-06T07:17:53.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d839640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab680b0070 0x7fab68155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.671 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d038640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6e03a640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fab680c5380 0x7fab6814e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:57126/0 (socket says 172.21.15.145:57126) 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6e03a640 1 -- 172.21.15.145:0/3084471946 learned_addr learned my addr 172.21.15.145:0/3084471946 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d038640 1 -- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 msgr2=0x7fab680c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d038640 1 -- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 msgr2=0x7fab680c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d038640 1 --2- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d038640 1 --2- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c8900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d839640 1 -- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 msgr2=0x7fab680c8900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d839640 1 --2- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c8900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d839640 1 -- 172.21.15.145:0/3084471946 >> v1:172.21.15.145:6789/0 conn(0x7fab680c5380 legacy=0x7fab6814e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6d839640 1 -- 172.21.15.145:0/3084471946 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab680c77f0 con 0x7fab680b0070 2024-03-06T07:17:53.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.667+0000 7fab6c837640 1 -- 172.21.15.145:0/3084471946 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47732140 0 0) 0x7fab60001490 con 0x7fab680c5380 2024-03-06T07:17:53.674 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:53.670+0000 7f8ab7440040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:17:53.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6d839640 1 --2- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab68155c30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fab580097a0 tx=0x7fab5802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22cc663d27676ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6c837640 1 -- 172.21.15.145:0/3084471946 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580315a0 con 0x7fab680b0070 2024-03-06T07:17:53.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6c837640 1 -- 172.21.15.145:0/3084471946 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab58037070 con 0x7fab680b0070 2024-03-06T07:17:53.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6c837640 1 -- 172.21.15.145:0/3084471946 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58031960 con 0x7fab680b0070 2024-03-06T07:17:53.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6fa9d640 1 -- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 msgr2=0x7fab68155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.680 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6fa9d640 1 --2- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab68155c30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fab580097a0 tx=0x7fab5802b840 comp rx=0 tx=0).stop 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6fa9d640 1 -- 172.21.15.145:0/3084471946 shutdown_connections 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6fa9d640 1 --2- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6fa9d640 1 --2- 172.21.15.145:0/3084471946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab68155c30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.669+0000 7fab6fa9d640 1 -- 172.21.15.145:0/3084471946 >> 172.21.15.145:0/3084471946 conn(0x7fab680ac590 msgr2=0x7fab680ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.670+0000 7fab6fa9d640 1 -- 172.21.15.145:0/3084471946 shutdown_connections 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.670+0000 7fab6fa9d640 1 -- 172.21.15.145:0/3084471946 wait complete. 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 Processor -- start 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 -- start start 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab681ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab680c5380 0x7fab681ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6806c550 con 0x7fab680c5380 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6806c270 con 0x7fab680b0070 2024-03-06T07:17:53.681 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6fa9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6806da80 con 0x7fab6806ca70 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d038640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab681ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d839640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d038640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab681ee1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34926/0 (socket says 172.21.15.145:34926) 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d038640 1 -- 172.21.15.145:0/627183696 learned_addr learned my addr 172.21.15.145:0/627183696 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d839640 1 -- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 msgr2=0x7fab680c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d839640 1 -- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 msgr2=0x7fab680c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d839640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6e03a640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab680c5380 0x7fab681ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.671+0000 7fab6d839640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c24a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6d038640 1 -- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 msgr2=0x7fab680c24a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6d038640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c24a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6d038640 1 -- 172.21.15.145:0/627183696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab680c5380 msgr2=0x7fab681ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6d038640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab680c5380 0x7fab681ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6d038640 1 -- 172.21.15.145:0/627183696 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68159580 con 0x7fab680b0070 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6d038640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab681ee1c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab6400d7d0 tx=0x7fab6400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab5e7fc640 1 -- 172.21.15.145:0/627183696 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400b6c0 con 0x7fab680b0070 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab5e7fc640 1 -- 172.21.15.145:0/627183696 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab64019070 con 0x7fab680b0070 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab680c4040 con 0x7fab680b0070 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab5e7fc640 1 -- 172.21.15.145:0/627183696 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400bac0 con 0x7fab680b0070 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab680c4600 con 0x7fab680b0070 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab5e7fc640 1 -- 172.21.15.145:0/627183696 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fab6400bc60 con 0x7fab680b0070 2024-03-06T07:17:53.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.672+0000 7fab5e7fc640 1 -- 172.21.15.145:0/627183696 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fab640149b0 con 0x7fab680b0070 2024-03-06T07:17:53.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.673+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6804f330 con 0x7fab680b0070 2024-03-06T07:17:53.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.677+0000 7fab5e7fc640 1 -- 172.21.15.145:0/627183696 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fab64014d00 con 0x7fab680b0070 2024-03-06T07:17:53.690 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.687+0000 7fe171640040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:17:53.724 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:53.722+0000 7f8ab7440040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:17:53.733 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.731+0000 7fe171640040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:17:53.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.749+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e268a931-5385-498e-8bc4-28a44c6bf4cf", "id": 3} v 0) v1 -- 0x7fab6806d860 con 0x7fab680b0070 2024-03-06T07:17:53.755 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.752+0000 7fab5e7fc640 1 -- 172.21.15.145:0/627183696 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e268a931-5385-498e-8bc4-28a44c6bf4cf", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fab640145f0 con 0x7fab680b0070 2024-03-06T07:17:53.755 INFO:teuthology.orchestra.run.smithi145.stdout:3 2024-03-06T07:17:53.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 msgr2=0x7fab681ee1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab681ee1c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab6400d7d0 tx=0x7fab6400dc90 comp rx=0 tx=0).stop 2024-03-06T07:17:53.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 shutdown_connections 2024-03-06T07:17:53.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab680c5380 0x7fab681ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab680b0070 0x7fab681ee1c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.757 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 --2- 172.21.15.145:0/627183696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab6806ca70 0x7fab680c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.758 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 >> 172.21.15.145:0/627183696 conn(0x7fab680ac590 msgr2=0x7fab68154e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.758 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 shutdown_connections 2024-03-06T07:17:53.758 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.753+0000 7fab6fa9d640 1 -- 172.21.15.145:0/627183696 wait complete. 2024-03-06T07:17:53.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new f3040779-55db-4733-a4d0-cc47e70d3ec8 4 2024-03-06T07:17:53.855 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.853+0000 7fe171640040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:17:53.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.905+0000 7f6244ba0640 1 Processor -- start 2024-03-06T07:17:53.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.905+0000 7f6244ba0640 1 -- start start 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6244ba0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f62400c1370 0x7f62401590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6244ba0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6240152960 con 0x7f624006afc0 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6244ba0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6240155ea0 con 0x7f6240152580 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6244ba0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6240155970 con 0x7f62400c1370 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6237fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f62400c1370 0x7f62401590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f623f7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6240152580 0x7f6240071fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:57140/0 (socket says 172.21.15.145:57140) 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6237fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f62400c1370 0x7f62401590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34936/0 (socket says 172.21.15.145:34936) 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f623f7fe640 1 -- 172.21.15.145:0/185387882 learned_addr learned my addr 172.21.15.145:0/185387882 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6237fff640 1 -- 172.21.15.145:0/185387882 >> v1:172.21.15.145:6789/0 conn(0x7f6240152580 legacy=0x7f6240071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6237fff640 1 -- 172.21.15.145:0/185387882 >> v1:172.21.15.94:6789/0 conn(0x7f624006afc0 legacy=0x7f624015f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.906+0000 7f6237fff640 1 -- 172.21.15.145:0/185387882 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6240158610 con 0x7f62400c1370 2024-03-06T07:17:53.910 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.907+0000 7f6237fff640 1 --2- 172.21.15.145:0/185387882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62401590b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f62280097a0 tx=0x7f622802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1463a052e001779 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.907+0000 7f623effd640 1 -- 172.21.15.145:0/185387882 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228035020 con 0x7f62400c1370 2024-03-06T07:17:53.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.907+0000 7f623effd640 1 -- 172.21.15.145:0/185387882 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6228031830 con 0x7f62400c1370 2024-03-06T07:17:53.911 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.907+0000 7f623effd640 1 -- 172.21.15.145:0/185387882 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228031c00 con 0x7f62400c1370 2024-03-06T07:17:53.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.908+0000 7f6244ba0640 1 -- 172.21.15.145:0/185387882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 msgr2=0x7f62401590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.914 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.908+0000 7f6244ba0640 1 --2- 172.21.15.145:0/185387882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62401590b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f62280097a0 tx=0x7f622802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.908+0000 7f6244ba0640 1 -- 172.21.15.145:0/185387882 shutdown_connections 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.908+0000 7f6244ba0640 1 --2- 172.21.15.145:0/185387882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62401590b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.908+0000 7f6244ba0640 1 -- 172.21.15.145:0/185387882 >> 172.21.15.145:0/185387882 conn(0x7f624006c6d0 msgr2=0x7f62400c3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.908+0000 7f6244ba0640 1 -- 172.21.15.145:0/185387882 shutdown_connections 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.908+0000 7f6244ba0640 1 -- 172.21.15.145:0/185387882 wait complete. 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.909+0000 7f6244ba0640 1 Processor -- start 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.909+0000 7f6244ba0640 1 -- start start 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6244ba0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624006afc0 0x7f62401e1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6244ba0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62400c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6244ba0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 0x7f62400c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6244ba0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6240158b90 con 0x7f624006afc0 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6244ba0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401588b0 con 0x7f62400c1370 2024-03-06T07:17:53.915 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6244ba0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62400726b0 con 0x7f6240152580 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62400c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62400c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34950/0 (socket says 172.21.15.145:34950) 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 -- 172.21.15.145:0/3337317324 learned_addr learned my addr 172.21.15.145:0/3337317324 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623ffff640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 0x7f62400c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623f7fe640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624006afc0 0x7f62401e1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623ffff640 1 -- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 msgr2=0x7f62400c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623ffff640 1 -- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 msgr2=0x7f62400c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623ffff640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 0x7f62400c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623ffff640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 0x7f62400c6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 -- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 msgr2=0x7f62400c6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 0x7f62400c6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 -- 172.21.15.145:0/3337317324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624006afc0 msgr2=0x7f62401e1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624006afc0 0x7f62401e1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 -- 172.21.15.145:0/3337317324 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62401ee6f0 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623f7fe640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624006afc0 0x7f62401e1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f6237fff640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62400c5bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f622802bee0 tx=0x7f62280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.910+0000 7f623cff9640 1 -- 172.21.15.145:0/3337317324 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228041070 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.911+0000 7f623cff9640 1 -- 172.21.15.145:0/3337317324 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622803a950 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.911+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62401ee9d0 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.911+0000 7f623cff9640 1 -- 172.21.15.145:0/3337317324 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6228035040 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.911+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62401eef90 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.911+0000 7f623cff9640 1 -- 172.21.15.145:0/3337317324 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f622804c060 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.911+0000 7f623cff9640 1 -- 172.21.15.145:0/3337317324 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f622803b940 con 0x7f62400c1370 2024-03-06T07:17:53.917 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.912+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62400693c0 con 0x7f62400c1370 2024-03-06T07:17:53.918 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.913+0000 7f623cff9640 1 -- 172.21.15.145:0/3337317324 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f622803bb90 con 0x7f62400c1370 2024-03-06T07:17:53.930 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.927+0000 7fe171640040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:17:53.977 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:53.975+0000 7fe171640040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:17:53.989 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.986+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f3040779-55db-4733-a4d0-cc47e70d3ec8", "id": 4} v 0) v1 -- 0x7f62400c0a80 con 0x7f62400c1370 2024-03-06T07:17:53.992 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.989+0000 7f623cff9640 1 -- 172.21.15.145:0/3337317324 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f3040779-55db-4733-a4d0-cc47e70d3ec8", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f622803bd70 con 0x7f62400c1370 2024-03-06T07:17:53.992 INFO:teuthology.orchestra.run.smithi145.stdout:4 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 msgr2=0x7f62400c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62400c5bf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f622802bee0 tx=0x7f62280046c0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 shutdown_connections 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6240152580 0x7f62400c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62400c1370 0x7f62400c5bf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 --2- 172.21.15.145:0/3337317324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624006afc0 0x7f62401e1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 >> 172.21.15.145:0/3337317324 conn(0x7f624006c6d0 msgr2=0x7f6240072920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 shutdown_connections 2024-03-06T07:17:53.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:53.990+0000 7f6244ba0640 1 -- 172.21.15.145:0/3337317324 wait complete. 2024-03-06T07:17:54.004 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new b95cc0c1-7cbe-4f68-b823-7e3eace28e7e 5 2024-03-06T07:17:54.034 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:54.032+0000 7fe171640040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:17:54.084 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:54.081+0000 7fe171640040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:17:54.105 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.103+0000 7f3264cdf640 1 Processor -- start 2024-03-06T07:17:54.106 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.103+0000 7f3264cdf640 1 -- start start 2024-03-06T07:17:54.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f3264cdf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f326006c240 0x7f3260159110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f3264cdf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f326006dc80 0x7f32600c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.107 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f3264cdf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3260155e50 con 0x7f326006e140 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f3264cdf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3260159650 con 0x7f326006c240 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f3264cdf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3260155c00 con 0x7f326006dc80 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325dd9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f326006dc80 0x7f32600c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325dd9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f326006dc80 0x7f32600c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34966/0 (socket says 172.21.15.145:34966) 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325dd9b640 1 -- 172.21.15.145:0/4134574322 learned_addr learned my addr 172.21.15.145:0/4134574322 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325e59c640 1 --2- 172.21.15.145:0/4134574322 >> v2:172.21.15.94:3300/0 conn(0x7f326006c240 0x7f3260159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325dd9b640 1 -- 172.21.15.145:0/4134574322 >> v2:172.21.15.94:3300/0 conn(0x7f326006c240 msgr2=0x7f3260159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325dd9b640 1 --2- 172.21.15.145:0/4134574322 >> v2:172.21.15.94:3300/0 conn(0x7f326006c240 0x7f3260159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325dd9b640 1 -- 172.21.15.145:0/4134574322 >> v1:172.21.15.145:6789/0 conn(0x7f326006e140 legacy=0x7f3260152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.104+0000 7f325dd9b640 1 -- 172.21.15.145:0/4134574322 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260158910 con 0x7f326006dc80 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.105+0000 7f325dd9b640 1 --2- 172.21.15.145:0/4134574322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32600c1a40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3254009870 tx=0x7f325402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31336d4b89fef83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.105+0000 7f325d59a640 1 -- 172.21.15.145:0/4134574322 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254035020 con 0x7f326006dc80 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.105+0000 7f325d59a640 1 -- 172.21.15.145:0/4134574322 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3254031870 con 0x7f326006dc80 2024-03-06T07:17:54.108 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.105+0000 7f325d59a640 1 -- 172.21.15.145:0/4134574322 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254031c40 con 0x7f326006dc80 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 -- 172.21.15.145:0/4134574322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 msgr2=0x7f32600c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 --2- 172.21.15.145:0/4134574322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32600c1a40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3254009870 tx=0x7f325402b910 comp rx=0 tx=0).stop 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 -- 172.21.15.145:0/4134574322 shutdown_connections 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 --2- 172.21.15.145:0/4134574322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32600c1a40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 --2- 172.21.15.145:0/4134574322 >> v2:172.21.15.94:3300/0 conn(0x7f326006c240 0x7f3260159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 -- 172.21.15.145:0/4134574322 >> 172.21.15.145:0/4134574322 conn(0x7f32600c3b40 msgr2=0x7f32600c3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 -- 172.21.15.145:0/4134574322 shutdown_connections 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.106+0000 7f3264cdf640 1 -- 172.21.15.145:0/4134574322 wait complete. 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.107+0000 7f3264cdf640 1 Processor -- start 2024-03-06T07:17:54.110 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.107+0000 7f3264cdf640 1 -- start start 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f3264cdf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326006c240 0x7f32601479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f3264cdf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32601ede40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f3264cdf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 0x7f32601ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f3264cdf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326006f0e0 con 0x7f326006c240 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f3264cdf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3260158c80 con 0x7f326006dc80 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f3264cdf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326006f260 con 0x7f326006e140 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32601ede40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325ed9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 0x7f32601ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32601ede40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34974/0 (socket says 172.21.15.145:34974) 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 -- 172.21.15.145:0/960580370 learned_addr learned my addr 172.21.15.145:0/960580370 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325e59c640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326006c240 0x7f32601479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.113 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325ed9d640 1 -- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 msgr2=0x7f32601ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325ed9d640 1 -- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 msgr2=0x7f32601ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325ed9d640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 0x7f32601ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325ed9d640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 0x7f32601ee380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 -- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 msgr2=0x7f32601ee380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 0x7f32601ee380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 -- 172.21.15.145:0/960580370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326006c240 msgr2=0x7f32601479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326006c240 0x7f32601479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 -- 172.21.15.145:0/960580370 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326014d610 con 0x7f326006dc80 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325e59c640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326006c240 0x7f32601479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f325dd9b640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32601ede40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3254000c00 tx=0x7f32540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f324effd640 1 -- 172.21.15.145:0/960580370 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254040070 con 0x7f326006dc80 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f324effd640 1 -- 172.21.15.145:0/960580370 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3254030d00 con 0x7f326006dc80 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f324effd640 1 -- 172.21.15.145:0/960580370 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254035040 con 0x7f326006dc80 2024-03-06T07:17:54.114 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.108+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326014df50 con 0x7f326006dc80 2024-03-06T07:17:54.115 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.109+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3260151910 con 0x7f326006dc80 2024-03-06T07:17:54.115 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.109+0000 7f324effd640 1 -- 172.21.15.145:0/960580370 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f325404b060 con 0x7f326006dc80 2024-03-06T07:17:54.115 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.109+0000 7f324effd640 1 -- 172.21.15.145:0/960580370 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f3254044980 con 0x7f326006dc80 2024-03-06T07:17:54.115 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.110+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326014cee0 con 0x7f326006dc80 2024-03-06T07:17:54.115 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.112+0000 7f324effd640 1 -- 172.21.15.145:0/960580370 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3254063dd0 con 0x7f326006dc80 2024-03-06T07:17:54.186 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.184+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b95cc0c1-7cbe-4f68-b823-7e3eace28e7e", "id": 5} v 0) v1 -- 0x7f326004f330 con 0x7f326006dc80 2024-03-06T07:17:54.189 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.187+0000 7f324effd640 1 -- 172.21.15.145:0/960580370 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b95cc0c1-7cbe-4f68-b823-7e3eace28e7e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3254052030 con 0x7f326006dc80 2024-03-06T07:17:54.190 INFO:teuthology.orchestra.run.smithi145.stdout:5 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 msgr2=0x7f32601ede40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32601ede40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3254000c00 tx=0x7f32540046c0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 shutdown_connections 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f326006e140 0x7f32601ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f326006dc80 0x7f32601ede40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 --2- 172.21.15.145:0/960580370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f326006c240 0x7f32601479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 >> 172.21.15.145:0/960580370 conn(0x7f32600c3b40 msgr2=0x7f32600c0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 shutdown_connections 2024-03-06T07:17:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.188+0000 7f3264cdf640 1 -- 172.21.15.145:0/960580370 wait complete. 2024-03-06T07:17:54.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new c57d10da-d807-4079-91b6-0a0db0c33c60 6 2024-03-06T07:17:54.268 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.266+0000 7f8ab7440040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:17:54.321 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.319+0000 7f8ab7440040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:17:54.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 Processor -- start 2024-03-06T07:17:54.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 -- start start 2024-03-06T07:17:54.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 0x7fd4980c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd49806e630 0x7fd498152170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd4981526b0 0x7fd498154f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4980c17d0 con 0x7fd49806e630 2024-03-06T07:17:54.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd498151ce0 con 0x7fd4981526b0 2024-03-06T07:17:54.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.342+0000 7fd49de55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980731c0 con 0x7fd4980c6d20 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd4977fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 0x7fd4980c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd4981526b0 0x7fd498154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd4981526b0 0x7fd498154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34982/0 (socket says 172.21.15.145:34982) 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd4977fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 0x7fd4980c2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:55310/0 (socket says 172.21.15.145:55310) 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 -- 172.21.15.145:0/3993423891 learned_addr learned my addr 172.21.15.145:0/3993423891 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd496ffd640 1 --2- 172.21.15.145:0/3993423891 >> v2:172.21.15.94:3300/0 conn(0x7fd49806e630 0x7fd498152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd4977fe640 1 -- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 msgr2=0x7fd4980c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd4977fe640 1 -- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 msgr2=0x7fd4980c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd4977fe640 1 --2- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 0x7fd4980c2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd4977fe640 1 --2- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 0x7fd4980c2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 -- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 msgr2=0x7fd4980c2c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:54.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 --2- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 0x7fd4980c2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 -- 172.21.15.145:0/3993423891 >> v2:172.21.15.94:3300/0 conn(0x7fd49806e630 msgr2=0x7fd498152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 --2- 172.21.15.145:0/3993423891 >> v2:172.21.15.94:3300/0 conn(0x7fd49806e630 0x7fd498152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.343+0000 7fd497fff640 1 -- 172.21.15.145:0/3993423891 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980a6070 con 0x7fd4981526b0 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd497fff640 1 --2- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981526b0 0x7fd498154f00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4980691e0 tx=0x7fd49002bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74b14446712f8fd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd4967fc640 1 -- 172.21.15.145:0/3993423891 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49002c950 con 0x7fd4981526b0 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd4967fc640 1 -- 172.21.15.145:0/3993423891 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49002caf0 con 0x7fd4981526b0 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd4967fc640 1 -- 172.21.15.145:0/3993423891 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4900343f0 con 0x7fd4981526b0 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 -- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981526b0 msgr2=0x7fd498154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 --2- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981526b0 0x7fd498154f00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4980691e0 tx=0x7fd49002bdb0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 -- 172.21.15.145:0/3993423891 shutdown_connections 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 --2- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981526b0 0x7fd498154f00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 --2- 172.21.15.145:0/3993423891 >> v2:172.21.15.94:3300/0 conn(0x7fd49806e630 0x7fd498152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 --2- 172.21.15.145:0/3993423891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c6d20 0x7fd4980c2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.350 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 -- 172.21.15.145:0/3993423891 >> 172.21.15.145:0/3993423891 conn(0x7fd49806cbf0 msgr2=0x7fd49806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.350 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.344+0000 7fd49de55640 1 -- 172.21.15.145:0/3993423891 shutdown_connections 2024-03-06T07:17:54.350 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.345+0000 7fd49de55640 1 -- 172.21.15.145:0/3993423891 wait complete. 2024-03-06T07:17:54.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 Processor -- start 2024-03-06T07:17:54.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 -- start start 2024-03-06T07:17:54.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 0x7fd498072810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4980c6d20 0x7fd4981ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981dc190 0x7fd49814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980c7100 con 0x7fd4980c6d20 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd498072e70 con 0x7fd4981dc190 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd49de55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980c7280 con 0x7fd49806e630 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd497fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981dc190 0x7fd49814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.346+0000 7fd4977fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 0x7fd498072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981dc190 0x7fd49814df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:34986/0 (socket says 172.21.15.145:34986) 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 -- 172.21.15.145:0/1713642479 learned_addr learned my addr 172.21.15.145:0/1713642479 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd496ffd640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4980c6d20 0x7fd4981ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd4977fe640 1 -- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 msgr2=0x7fd498072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd4977fe640 1 -- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 msgr2=0x7fd498072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd4977fe640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 0x7fd498072810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd4977fe640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 0x7fd498072810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 -- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 msgr2=0x7fd498072810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 0x7fd498072810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.353 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 -- 172.21.15.145:0/1713642479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4980c6d20 msgr2=0x7fd4981ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4980c6d20 0x7fd4981ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 -- 172.21.15.145:0/1713642479 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4981eebc0 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd497fff640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981dc190 0x7fd49814df80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd490036040 tx=0x7fd490004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd47ffff640 1 -- 172.21.15.145:0/1713642479 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490044020 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd47ffff640 1 -- 172.21.15.145:0/1713642479 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4900406b0 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4980a9e20 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd47ffff640 1 -- 172.21.15.145:0/1713642479 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490009510 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd498070bf0 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.347+0000 7fd47ffff640 1 -- 172.21.15.145:0/1713642479 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd490047020 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.348+0000 7fd47ffff640 1 -- 172.21.15.145:0/1713642479 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fd49002c960 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.348+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49804f330 con 0x7fd4981dc190 2024-03-06T07:17:54.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.352+0000 7fd47ffff640 1 -- 172.21.15.145:0/1713642479 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd49004e030 con 0x7fd4981dc190 2024-03-06T07:17:54.366 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.365+0000 7f8ab7440040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:17:54.421 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.419+0000 7f8ab7440040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:17:54.426 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.423+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c57d10da-d807-4079-91b6-0a0db0c33c60", "id": 6} v 0) v1 -- 0x7fd4980c4b30 con 0x7fd4981dc190 2024-03-06T07:17:54.429 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.427+0000 7fd47ffff640 1 -- 172.21.15.145:0/1713642479 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c57d10da-d807-4079-91b6-0a0db0c33c60", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd4981eebc0 con 0x7fd4981dc190 2024-03-06T07:17:54.429 INFO:teuthology.orchestra.run.smithi145.stdout:6 2024-03-06T07:17:54.430 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981dc190 msgr2=0x7fd49814df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981dc190 0x7fd49814df80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd490036040 tx=0x7fd490004a40 comp rx=0 tx=0).stop 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 shutdown_connections 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4981dc190 0x7fd49814df80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4980c6d20 0x7fd4981ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 --2- 172.21.15.145:0/1713642479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd49806e630 0x7fd498072810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 >> 172.21.15.145:0/1713642479 conn(0x7fd49806cbf0 msgr2=0x7fd4980a7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 shutdown_connections 2024-03-06T07:17:54.431 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.428+0000 7fd49de55640 1 -- 172.21.15.145:0/1713642479 wait complete. 2024-03-06T07:17:54.441 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 6b61b374-fd8d-4b4a-a15b-cc008337e57c 7 2024-03-06T07:17:54.464 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.462+0000 7f8ab7440040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:17:54.481 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:54.478+0000 7fe171640040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:17:54.542 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.539+0000 7f5c27aa4640 1 Processor -- start 2024-03-06T07:17:54.542 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c27aa4640 1 -- start start 2024-03-06T07:17:54.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c27aa4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 0x7f5c200c0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c27aa4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c200a6cc0 0x7f5c2006e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c27aa4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c20155270 con 0x7f5c200c0bc0 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c27aa4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c20155e80 con 0x7f5c200a6cc0 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c27aa4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c20155550 con 0x7f5c200c5690 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c2503f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 0x7f5c200c0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c2503f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 0x7f5c200c0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:54044/0 (socket says 172.21.15.145:54044) 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.540+0000 7f5c2503f640 1 -- 172.21.15.145:0/2824694738 learned_addr learned my addr 172.21.15.145:0/2824694738 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c2503f640 1 -- 172.21.15.145:0/2824694738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 msgr2=0x7f5c200c0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c2503f640 1 -- 172.21.15.145:0/2824694738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 msgr2=0x7f5c200c0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c2503f640 1 --2- 172.21.15.145:0/2824694738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 0x7f5c200c0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c2503f640 1 --2- 172.21.15.145:0/2824694738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 0x7f5c200c0fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c26041640 1 --2- 172.21.15.145:0/2824694738 >> v2:172.21.15.94:3300/0 conn(0x7f5c200a6cc0 0x7f5c2006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c26041640 1 -- 172.21.15.145:0/2824694738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 msgr2=0x7f5c200c0fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c26041640 1 --2- 172.21.15.145:0/2824694738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 0x7f5c200c0fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.545 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c26041640 1 -- 172.21.15.145:0/2824694738 >> v1:172.21.15.94:6789/0 conn(0x7f5c200c0bc0 legacy=0x7f5c2006db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.546 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c26041640 1 -- 172.21.15.145:0/2824694738 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c20155830 con 0x7f5c200a6cc0 2024-03-06T07:17:54.546 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.541+0000 7f5c2483e640 1 -- 172.21.15.145:0/2824694738 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239976196 0 0) 0x7f5c20155270 con 0x7f5c200c0bc0 2024-03-06T07:17:54.546 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c26041640 1 --2- 172.21.15.145:0/2824694738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 0x7f5c2006e280 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009d90 tx=0x7f5c1802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c870663177bad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.546 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c2483e640 1 -- 172.21.15.145:0/2824694738 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18039070 con 0x7f5c200a6cc0 2024-03-06T07:17:54.546 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c2483e640 1 -- 172.21.15.145:0/2824694738 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1802cb30 con 0x7f5c200a6cc0 2024-03-06T07:17:54.546 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c2483e640 1 -- 172.21.15.145:0/2824694738 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18034440 con 0x7f5c200a6cc0 2024-03-06T07:17:54.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 -- 172.21.15.145:0/2824694738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 msgr2=0x7f5c2006e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 --2- 172.21.15.145:0/2824694738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 0x7f5c2006e280 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009d90 tx=0x7f5c1802bd80 comp rx=0 tx=0).stop 2024-03-06T07:17:54.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 -- 172.21.15.145:0/2824694738 shutdown_connections 2024-03-06T07:17:54.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 --2- 172.21.15.145:0/2824694738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 0x7f5c2006e280 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 --2- 172.21.15.145:0/2824694738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c5690 0x7f5c200c0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 -- 172.21.15.145:0/2824694738 >> 172.21.15.145:0/2824694738 conn(0x7f5c200a9540 msgr2=0x7f5c200a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.549 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 -- 172.21.15.145:0/2824694738 shutdown_connections 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.542+0000 7f5c27aa4640 1 -- 172.21.15.145:0/2824694738 wait complete. 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.543+0000 7f5c27aa4640 1 Processor -- start 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c27aa4640 1 -- start start 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c27aa4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 0x7f5c201eb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c27aa4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 0x7f5c200a8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c27aa4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200c5690 0x7f5c201dfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c27aa4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2006f410 con 0x7f5c200a6cc0 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c27aa4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2006e7c0 con 0x7f5c200c5690 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c27aa4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c20152ab0 con 0x7f5c200c0bc0 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c2503f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 0x7f5c200a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c26041640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200c5690 0x7f5c201dfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c2503f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 0x7f5c200a8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:54046/0 (socket says 172.21.15.145:54046) 2024-03-06T07:17:54.550 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c2503f640 1 -- 172.21.15.145:0/686910692 learned_addr learned my addr 172.21.15.145:0/686910692 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c2503f640 1 -- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 msgr2=0x7f5c200a8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c25840640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 0x7f5c201eb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c2503f640 1 -- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 msgr2=0x7f5c200a8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c2503f640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 0x7f5c200a8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.544+0000 7f5c2503f640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 0x7f5c200a8670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c26041640 1 -- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 msgr2=0x7f5c200a8670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c26041640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 0x7f5c200a8670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c26041640 1 -- 172.21.15.145:0/686910692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 msgr2=0x7f5c201eb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c26041640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 0x7f5c201eb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c26041640 1 -- 172.21.15.145:0/686910692 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c201ebbe0 con 0x7f5c200c5690 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c26041640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200c5690 0x7f5c201dfb50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5c18033000 tx=0x7f5c18004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c167fc640 1 -- 172.21.15.145:0/686910692 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c18039050 con 0x7f5c200c5690 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c167fc640 1 -- 172.21.15.145:0/686910692 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c18045070 con 0x7f5c200c5690 2024-03-06T07:17:54.551 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2014e8a0 con 0x7f5c200c5690 2024-03-06T07:17:54.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c167fc640 1 -- 172.21.15.145:0/686910692 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1803e850 con 0x7f5c200c5690 2024-03-06T07:17:54.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2014d280 con 0x7f5c200c5690 2024-03-06T07:17:54.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c167fc640 1 -- 172.21.15.145:0/686910692 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5c1803e9f0 con 0x7f5c200c5690 2024-03-06T07:17:54.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.545+0000 7f5c167fc640 1 -- 172.21.15.145:0/686910692 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f5c18040ab0 con 0x7f5c200c5690 2024-03-06T07:17:54.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.546+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2004f330 con 0x7f5c200c5690 2024-03-06T07:17:54.552 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.548+0000 7f5c167fc640 1 -- 172.21.15.145:0/686910692 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5c1803ee20 con 0x7f5c200c5690 2024-03-06T07:17:54.579 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.576+0000 7f8ab7440040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:17:54.621 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:54.618+0000 7fe171640040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:17:54.623 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.621+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6b61b374-fd8d-4b4a-a15b-cc008337e57c", "id": 7} v 0) v1 -- 0x7f5c2014cd50 con 0x7f5c200c5690 2024-03-06T07:17:54.626 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.624+0000 7f5c167fc640 1 -- 172.21.15.145:0/686910692 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6b61b374-fd8d-4b4a-a15b-cc008337e57c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5c1804a460 con 0x7f5c200c5690 2024-03-06T07:17:54.627 INFO:teuthology.orchestra.run.smithi145.stdout:7 2024-03-06T07:17:54.628 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200c5690 msgr2=0x7f5c201dfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.628 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200c5690 0x7f5c201dfb50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5c18033000 tx=0x7f5c18004a40 comp rx=0 tx=0).stop 2024-03-06T07:17:54.628 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 shutdown_connections 2024-03-06T07:17:54.629 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200c5690 0x7f5c201dfb50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.629 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200c0bc0 0x7f5c200a8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.629 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 --2- 172.21.15.145:0/686910692 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c200a6cc0 0x7f5c201eb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.629 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 >> 172.21.15.145:0/686910692 conn(0x7f5c200a9540 msgr2=0x7f5c200a9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.629 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 shutdown_connections 2024-03-06T07:17:54.629 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-06T07:17:54.625+0000 7f5c27aa4640 1 -- 172.21.15.145:0/686910692 wait complete. 2024-03-06T07:17:54.638 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:17:54.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:17:54.640 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:17:54.640 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:17:54.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:17:54.642 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:17:54.643 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:17:54.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:17:54.645 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:17:54.645 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:17:54.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:17:54.647 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:17:54.647 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:17:54.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:17:54.651 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.650+0000 7f8ab7440040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:17:54.681 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:17:54.682 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:17:54.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:17:54.684 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:17:54.684 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:17:54.684 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:17:54.687 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:17:54.687 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:17:54.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:17:54.689 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:17:54.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:17:54.723 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.721+0000 7f8ab7440040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:17:54.755 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:54.753+0000 7fe171640040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:17:54.799 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.798+0000 7f8ab7440040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:17:54.806 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:54.804+0000 7fe171640040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:17:54.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.811+0000 7f9662956640 1 Processor -- start 2024-03-06T07:17:54.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.811+0000 7f9662956640 1 -- start start 2024-03-06T07:17:54.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.811+0000 7f9662956640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f965c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.811+0000 7f9662956640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c0c73d0 con 0x7f965c0c88c0 2024-03-06T07:17:54.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.811+0000 7f9662956640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965c0be2e0 con 0x7f965c158b60 2024-03-06T07:17:54.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.811+0000 7f9662956640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c159700 con 0x7f965c14f360 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f965c0c88c0 0x7f965c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53742/0 (socket says 172.21.15.94:53742) 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965b7fe640 1 -- 172.21.15.94:0/3352448064 learned_addr learned my addr 172.21.15.94:0/3352448064 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965bfff640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f965c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151810443 0 0) 0x7f965c0c73d0 con 0x7f965c0c88c0 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9644003520 con 0x7f965c0c88c0 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965bfff640 1 -- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 msgr2=0x7f965c155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965bfff640 1 -- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 msgr2=0x7f965c155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965bfff640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f965c155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965bfff640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f965c155c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238765753 0 0) 0x7f965c0be2e0 con 0x7f965c158b60 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f965c0c73d0 con 0x7f965c158b60 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1894564087 0 0) 0x7f9644003520 con 0x7f965c0c88c0 2024-03-06T07:17:54.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 msgr2=0x7f965c155c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f965c155c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 >> v1:172.21.15.145:6789/0 conn(0x7f965c158b60 legacy=0x7f965c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c0c7670 con 0x7f965c0c88c0 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f964c001670 con 0x7f965c0c88c0 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 >> v1:172.21.15.94:6789/0 conn(0x7f965c0c88c0 legacy=0x7f965c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9644003b70 0x7f9644003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f9644004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f96440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c159700 con 0x7f9644004d40 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964400bb80 con 0x7f9644003b70 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964400bd80 con 0x7f965c14f360 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f9660ef3640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f96440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965bfff640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9644003b70 0x7f9644003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f965b7fe640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f9644004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f9660ef3640 1 -- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 msgr2=0x7f9644004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f9660ef3640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c14f360 0x7f9644004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f9660ef3640 1 -- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9644003b70 msgr2=0x7f9644003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f9660ef3640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9644003b70 0x7f9644003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.812+0000 7f9660ef3640 1 -- 172.21.15.94:0/3352448064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9644003520 con 0x7f9644004d40 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.813+0000 7f9660ef3640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f96440051c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9654001fa0 tx=0x7f9644005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988995117f0f7391 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.813+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9654001bf0 con 0x7f9644004d40 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.813+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9654003b70 con 0x7f9644004d40 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.813+0000 7f965affd640 1 -- 172.21.15.94:0/3352448064 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96540050d0 con 0x7f9644004d40 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.814+0000 7f9662956640 1 -- 172.21.15.94:0/3352448064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 msgr2=0x7f96440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.814+0000 7f9662956640 1 --2- 172.21.15.94:0/3352448064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f96440051c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9654001fa0 tx=0x7f9644005830 comp rx=0 tx=0).stop 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.814+0000 7f9660ef3640 1 -- 172.21.15.94:0/3352448064 reap_dead start 2024-03-06T07:17:54.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.814+0000 7f9662956640 1 -- 172.21.15.94:0/3352448064 shutdown_connections 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.814+0000 7f9662956640 1 -- 172.21.15.94:0/3352448064 >> 172.21.15.94:0/3352448064 conn(0x7f965c0ac570 msgr2=0x7f965c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.814+0000 7f9662956640 1 -- 172.21.15.94:0/3352448064 shutdown_connections 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.814+0000 7f9662956640 1 -- 172.21.15.94:0/3352448064 wait complete. 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.815+0000 7f9662956640 1 Processor -- start 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.815+0000 7f9662956640 1 -- start start 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9662956640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f965c0c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9662956640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f965c0c4440 0x7f965c1edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9662956640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 0x7f965c1ec910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9662956640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0c7c30 con 0x7f9644004d40 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9662956640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0c5a50 con 0x7f965c0c4440 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9662956640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965c0c5bd0 con 0x7f965c1ee500 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f965c0c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f965c0c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35078/0 (socket says 172.21.15.94:35078) 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 -- 172.21.15.94:0/2639637143 learned_addr learned my addr 172.21.15.94:0/2639637143 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9660ef3640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 0x7f965c1ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965b7fe640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f965c0c4440 0x7f965c1edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9660ef3640 1 -- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 msgr2=0x7f965c1ec910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9660ef3640 1 -- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 msgr2=0x7f965c1ec910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9660ef3640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 0x7f965c1ec910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f9660ef3640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 0x7f965c1ec910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 -- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 msgr2=0x7f965c1ec910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 0x7f965c1ec910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 -- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f965c0c4440 msgr2=0x7f965c1edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f965c0c4440 0x7f965c1edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965bfff640 1 -- 172.21.15.94:0/2639637143 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c1eea70 con 0x7f9644004d40 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.816+0000 7f965b7fe640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f965c0c4440 0x7f965c1edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:17:54.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f965bfff640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f965c0c3f00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9650010900 tx=0x7f9650010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f9658ff9640 1 -- 172.21.15.94:0/2639637143 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965001a020 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f9658ff9640 1 -- 172.21.15.94:0/2639637143 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965000ea20 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965c1eed50 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f9658ff9640 1 -- 172.21.15.94:0/2639637143 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965000ee10 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965c0c0510 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f9658ff9640 1 -- 172.21.15.94:0/2639637143 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f965001d020 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.817+0000 7f9658ff9640 1 -- 172.21.15.94:0/2639637143 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f965001f400 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.818+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965c04f330 con 0x7f9644004d40 2024-03-06T07:17:54.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.820+0000 7f9658ff9640 1 -- 172.21.15.94:0/2639637143 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9650024020 con 0x7f9644004d40 2024-03-06T07:17:54.850 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:54.848+0000 7f8ab7440040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:17:54.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.881+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f965c0c8d50 con 0x7f9644004d40 2024-03-06T07:17:54.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.881+0000 7f9658ff9640 1 -- 172.21.15.94:0/2639637143 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f9650014430 con 0x7f9644004d40 2024-03-06T07:17:54.883 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:17:54.884 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":10,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:17:54.623758+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:17:54.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 msgr2=0x7f965c0c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f965c0c3f00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9650010900 tx=0x7f9650010dc0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 shutdown_connections 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f965c1ee500 0x7f965c1ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f965c0c4440 0x7f965c1edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 --2- 172.21.15.94:0/2639637143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9644004d40 0x7f965c0c3f00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 >> 172.21.15.94:0/2639637143 conn(0x7f965c0ac570 msgr2=0x7f965c14e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 shutdown_connections 2024-03-06T07:17:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:17:54.882+0000 7f9662956640 1 -- 172.21.15.94:0/2639637143 wait complete. 2024-03-06T07:17:54.895 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:17:54.895 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:17:54.919 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:54.916+0000 7fe171640040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:17:55.012 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:55.009+0000 7fe171640040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:17:55.167 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:55.165+0000 7fe171640040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:17:55.209 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:55.206+0000 7fe171640040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:17:55.233 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.231+0000 7f8ab7440040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:17:55.263 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-03-06T07:17:55.261+0000 7fe557a8b640 -1 Falling back to public interface 2024-03-06T07:17:55.267 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-03-06T07:17:55.265+0000 7f790b837640 -1 Falling back to public interface 2024-03-06T07:17:55.267 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-03-06T07:17:55.265+0000 7faee94f9640 -1 Falling back to public interface 2024-03-06T07:17:55.275 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-03-06T07:17:55.273+0000 7f55e0114640 -1 Falling back to public interface 2024-03-06T07:17:55.290 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-03-06T07:17:55.287+0000 7f34ea1bd640 -1 Falling back to public interface 2024-03-06T07:17:55.292 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-03-06T07:17:55.290+0000 7fae6d613640 -1 Falling back to public interface 2024-03-06T07:17:55.299 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-03-06T07:17:55.297+0000 7f3c4d024640 -1 Falling back to public interface 2024-03-06T07:17:55.312 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-03-06T07:17:55.309+0000 7f95ed193640 -1 Falling back to public interface 2024-03-06T07:17:55.368 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:55.366+0000 7fe171640040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:17:55.369 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.368+0000 7f8ab7440040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:17:55.485 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.483+0000 7f8ab7440040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:17:55.531 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.529+0000 7f8ab7440040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:17:55.643 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.641+0000 7f8ab7440040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:17:55.700 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:17:55.739 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.737+0000 7f8ab7440040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:17:55.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.787+0000 7f42ab2b5640 1 Processor -- start 2024-03-06T07:17:55.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.788+0000 7f42ab2b5640 1 -- start start 2024-03-06T07:17:55.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.788+0000 7f42ab2b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f42a40c4bd0 0x7f42a4156f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.788+0000 7f42ab2b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 0x7f42a40c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.788+0000 7f42ab2b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a40c8430 con 0x7f42a4070360 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.788+0000 7f42ab2b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a40c81e0 con 0x7f42a40c4bd0 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.788+0000 7f42ab2b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a41564b0 con 0x7f42a40c7c90 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9051640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f42a40c4bd0 0x7f42a4156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9852640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 0x7f42a40c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a8850640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f42a4070360 0x7f42a4152ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:45676/0 (socket says 172.21.15.94:45676) 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a8850640 1 -- 172.21.15.94:0/1150758539 learned_addr learned my addr 172.21.15.94:0/1150758539 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9852640 1 -- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 msgr2=0x7f42a40c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9852640 1 -- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 msgr2=0x7f42a40c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9852640 1 --2- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 0x7f42a40c53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9852640 1 --2- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 0x7f42a40c53e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:55.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9051640 1 -- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 msgr2=0x7f42a40c53e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9051640 1 --2- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 0x7f42a40c53e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9051640 1 -- 172.21.15.94:0/1150758539 >> v1:172.21.15.145:6789/0 conn(0x7f42a4070360 legacy=0x7f42a4152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.789+0000 7f42a9051640 1 -- 172.21.15.94:0/1150758539 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a4156700 con 0x7f42a40c4bd0 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f42a9051640 1 --2- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a4156f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42a00097a0 tx=0x7f42a002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e5ccd72a41931d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f429bfff640 1 -- 172.21.15.94:0/1150758539 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0038070 con 0x7f42a40c4bd0 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f429bfff640 1 -- 172.21.15.94:0/1150758539 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0031700 con 0x7f42a40c4bd0 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f429bfff640 1 -- 172.21.15.94:0/1150758539 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0031ad0 con 0x7f42a40c4bd0 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f42ab2b5640 1 -- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 msgr2=0x7f42a4156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f42ab2b5640 1 --2- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a4156f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42a00097a0 tx=0x7f42a002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f42ab2b5640 1 -- 172.21.15.94:0/1150758539 shutdown_connections 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f42ab2b5640 1 --2- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a40c7c90 0x7f42a40c53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f42ab2b5640 1 --2- 172.21.15.94:0/1150758539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a4156f00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.790+0000 7f42ab2b5640 1 -- 172.21.15.94:0/1150758539 >> 172.21.15.94:0/1150758539 conn(0x7f42a40c8a50 msgr2=0x7f42a40c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:55.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.791+0000 7f42ab2b5640 1 -- 172.21.15.94:0/1150758539 shutdown_connections 2024-03-06T07:17:55.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.791+0000 7f42ab2b5640 1 -- 172.21.15.94:0/1150758539 wait complete. 2024-03-06T07:17:55.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 Processor -- start 2024-03-06T07:17:55.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 -- start start 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 0x7f42a40c2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a414b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42a40c7c90 0x7f42a40a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4157440 con 0x7f42a40c7c90 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4156a70 con 0x7f42a40c4bd0 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42ab2b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40c5920 con 0x7f42a4070360 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42a9852640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42a40c7c90 0x7f42a40a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42a9051640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 0x7f42a40c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42a8850640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a414b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42a9852640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42a40c7c90 0x7f42a40a8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35092/0 (socket says 172.21.15.94:35092) 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42a9051640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 0x7f42a40c2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:35728/0 (socket says 172.21.15.94:35728) 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.792+0000 7f42a9852640 1 -- 172.21.15.94:0/206477939 learned_addr learned my addr 172.21.15.94:0/206477939 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:17:55.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a9051640 1 -- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 msgr2=0x7f42a40c2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a9051640 1 -- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 msgr2=0x7f42a40c2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a9051640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 0x7f42a40c2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a9051640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 0x7f42a40c2640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a8850640 1 -- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 msgr2=0x7f42a40c2640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a8850640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 0x7f42a40c2640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a8850640 1 -- 172.21.15.94:0/206477939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42a40c7c90 msgr2=0x7f42a40a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a8850640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42a40c7c90 0x7f42a40a8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a8850640 1 -- 172.21.15.94:0/206477939 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a40a9d50 con 0x7f42a40c4bd0 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a9852640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42a40c7c90 0x7f42a40a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42a8850640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a414b630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4294000f60 tx=0x7f429400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f4299ffb640 1 -- 172.21.15.94:0/206477939 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294017070 con 0x7f42a40c4bd0 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f4299ffb640 1 -- 172.21.15.94:0/206477939 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4294005ce0 con 0x7f42a40c4bd0 2024-03-06T07:17:55.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a40aa030 con 0x7f42a40c4bd0 2024-03-06T07:17:55.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f4299ffb640 1 -- 172.21.15.94:0/206477939 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294007310 con 0x7f42a40c4bd0 2024-03-06T07:17:55.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a40c2120 con 0x7f42a40c4bd0 2024-03-06T07:17:55.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.793+0000 7f4299ffb640 1 -- 172.21.15.94:0/206477939 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f429401a070 con 0x7f42a40c4bd0 2024-03-06T07:17:55.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.794+0000 7f4299ffb640 1 -- 172.21.15.94:0/206477939 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f4294012d80 con 0x7f42a40c4bd0 2024-03-06T07:17:55.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.794+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a404f330 con 0x7f42a40c4bd0 2024-03-06T07:17:55.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.796+0000 7f4299ffb640 1 -- 172.21.15.94:0/206477939 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f42940074b0 con 0x7f42a40c4bd0 2024-03-06T07:17:55.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.858+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42a4073000 con 0x7f42a40c4bd0 2024-03-06T07:17:55.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.858+0000 7f4299ffb640 1 -- 172.21.15.94:0/206477939 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f4294007690 con 0x7f42a40c4bd0 2024-03-06T07:17:55.860 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-06T07:17:55.861 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:17:54.623758+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:17:55.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 msgr2=0x7f42a414b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a414b630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4294000f60 tx=0x7f429400c800 comp rx=0 tx=0).stop 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 shutdown_connections 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42a40c7c90 0x7f42a40a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42a40c4bd0 0x7f42a414b630 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 --2- 172.21.15.94:0/206477939 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42a4070360 0x7f42a40c2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 >> 172.21.15.94:0/206477939 conn(0x7f42a40c8a50 msgr2=0x7f42a414e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 shutdown_connections 2024-03-06T07:17:55.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:17:55.859+0000 7f42ab2b5640 1 -- 172.21.15.94:0/206477939 wait complete. 2024-03-06T07:17:55.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:17:55.901 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.899+0000 7f8ab7440040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:17:55.943 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:55.941+0000 7f8ab7440040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:17:56.098 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:56.096+0000 7f8ab7440040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:17:56.173 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:56.170+0000 7fe171640040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:17:56.449 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:56.447+0000 7fe171640040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:17:56.500 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:56.497+0000 7fe171640040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:17:56.613 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:56.610+0000 7fe171640040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:17:56.918 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:56.916+0000 7f8ab7440040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:17:57.192 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:57.190+0000 7f8ab7440040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:17:57.244 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:57.242+0000 7f8ab7440040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:17:57.359 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-06T07:17:57.357+0000 7f8ab7440040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:17:57.640 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:57.638+0000 7fe11228c640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:17:57.640 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-03-06T07:17:57.638+0000 7fe11228c640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:17:58.741 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-03-06T07:17:58.739+0000 7fe557a8b640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:17:58.773 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-03-06T07:17:58.771+0000 7f790b837640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:17:58.788 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-03-06T07:17:58.784+0000 7fae6d613640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:17:58.791 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-03-06T07:17:58.790+0000 7faee94f9640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:17:58.792 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-03-06T07:17:58.790+0000 7f34ea1bd640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:17:58.804 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-03-06T07:17:58.802+0000 7f55e0114640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:17:58.813 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-03-06T07:17:58.810+0000 7f3c4d024640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:17:58.837 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-03-06T07:17:58.835+0000 7f95ed193640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:18:01.279 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-03-06T07:18:01.277+0000 7f55dc0aa640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:18:01.294 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-03-06T07:18:01.292+0000 7f55d3eca640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:01.298 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-03-06T07:18:01.296+0000 7fe554235640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:18:01.308 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-03-06T07:18:01.307+0000 7fe54b841640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:01.310 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-03-06T07:18:01.308+0000 7f79077cd640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:18:01.328 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-03-06T07:18:01.326+0000 7f78ff5ed640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:01.336 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-03-06T07:18:01.334+0000 7f95e9129640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:18:01.349 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-03-06T07:18:01.346+0000 7f3c497ce640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:18:01.354 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-03-06T07:18:01.352+0000 7f95e0f49640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:01.367 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-03-06T07:18:01.365+0000 7f3c40dda640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:01.376 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-03-06T07:18:01.375+0000 7faee548f640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:18:01.385 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-03-06T07:18:01.382+0000 7f34e6153640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:18:01.388 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-03-06T07:18:01.387+0000 7faedd2af640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:01.402 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-03-06T07:18:01.400+0000 7f34ddf73640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:01.404 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-03-06T07:18:01.402+0000 7fae68ee5640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:18:01.420 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-03-06T07:18:01.418+0000 7fae60d05640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:18:02.679 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:02.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.781+0000 7f68d0edf640 1 Processor -- start 2024-03-06T07:18:02.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.781+0000 7f68d0edf640 1 -- start start 2024-03-06T07:18:02.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68d0edf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68cc1528e0 0x7f68cc0c2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:02.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68d0edf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f68cc0c20e0 0x7f68cc0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:02.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68d0edf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc1553b0 con 0x7f68cc0c8d40 2024-03-06T07:18:02.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68d0edf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68cc155690 con 0x7f68cc1528e0 2024-03-06T07:18:02.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68d0edf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68cc155970 con 0x7f68cc0c20e0 2024-03-06T07:18:02.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68ca59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68cc1528e0 0x7f68cc0c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:02.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68ca59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68cc1528e0 0x7f68cc0c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52340/0 (socket says 172.21.15.94:52340) 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68ca59c640 1 -- 172.21.15.94:0/1437435714 learned_addr learned my addr 172.21.15.94:0/1437435714 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68c9d9b640 1 --2- 172.21.15.94:0/1437435714 >> v2:172.21.15.145:3300/0 conn(0x7f68cc0c20e0 0x7f68cc0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68ca59c640 1 -- 172.21.15.94:0/1437435714 >> v2:172.21.15.145:3300/0 conn(0x7f68cc0c20e0 msgr2=0x7f68cc0c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68ca59c640 1 --2- 172.21.15.94:0/1437435714 >> v2:172.21.15.145:3300/0 conn(0x7f68cc0c20e0 0x7f68cc0c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68ca59c640 1 -- 172.21.15.94:0/1437435714 >> v1:172.21.15.145:6789/0 conn(0x7f68cc0c8d40 legacy=0x7f68cc158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.782+0000 7f68ca59c640 1 -- 172.21.15.94:0/1437435714 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc155c50 con 0x7f68cc1528e0 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.783+0000 7f68ca59c640 1 --2- 172.21.15.94:0/1437435714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc1528e0 0x7f68cc0c2960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0c37d0 tx=0x7f68bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b7a1344a94ea27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.783+0000 7f68c959a640 1 -- 172.21.15.94:0/1437435714 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc035020 con 0x7f68cc1528e0 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.783+0000 7f68c959a640 1 -- 172.21.15.94:0/1437435714 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68bc031870 con 0x7f68cc1528e0 2024-03-06T07:18:02.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.783+0000 7f68c959a640 1 -- 172.21.15.94:0/1437435714 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc031c40 con 0x7f68cc1528e0 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 -- 172.21.15.94:0/1437435714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc1528e0 msgr2=0x7f68cc0c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 --2- 172.21.15.94:0/1437435714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc1528e0 0x7f68cc0c2960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0c37d0 tx=0x7f68bc02b910 comp rx=0 tx=0).stop 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 -- 172.21.15.94:0/1437435714 shutdown_connections 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 --2- 172.21.15.94:0/1437435714 >> v2:172.21.15.145:3300/0 conn(0x7f68cc0c20e0 0x7f68cc0c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 --2- 172.21.15.94:0/1437435714 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc1528e0 0x7f68cc0c2960 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 -- 172.21.15.94:0/1437435714 >> 172.21.15.94:0/1437435714 conn(0x7f68cc0672b0 msgr2=0x7f68cc072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 -- 172.21.15.94:0/1437435714 shutdown_connections 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.784+0000 7f68d0edf640 1 -- 172.21.15.94:0/1437435714 wait complete. 2024-03-06T07:18:02.789 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.785+0000 7f68d0edf640 1 Processor -- start 2024-03-06T07:18:02.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.785+0000 7f68d0edf640 1 -- start start 2024-03-06T07:18:02.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68d0edf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc0c20e0 0x7f68cc150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:02.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68d0edf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68cc0c8d40 0x7f68cc14d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:02.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68d0edf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68cc1528e0 0x7f68cc14dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:02.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68d0edf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc159020 con 0x7f68cc0c20e0 2024-03-06T07:18:02.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68d0edf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc158ea0 con 0x7f68cc1528e0 2024-03-06T07:18:02.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68d0edf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc1591a0 con 0x7f68cc0c8d40 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc0c20e0 0x7f68cc150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc0c20e0 0x7f68cc150fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52350/0 (socket says 172.21.15.94:52350) 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 -- 172.21.15.94:0/2171470484 learned_addr learned my addr 172.21.15.94:0/2171470484 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 -- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68cc0c8d40 msgr2=0x7f68cc14d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68cc0c8d40 0x7f68cc14d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 -- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68cc1528e0 msgr2=0x7f68cc14dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68cad9d640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68cc1528e0 0x7f68cc14dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68cc1528e0 0x7f68cc14dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 -- 172.21.15.94:0/2171470484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc14e2d0 con 0x7f68cc0c20e0 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68cad9d640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68cc1528e0 0x7f68cc14dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.786+0000 7f68ca59c640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc0c20e0 0x7f68cc150fd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f68bc000c00 tx=0x7f68bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:02.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.787+0000 7f68baffd640 1 -- 172.21.15.94:0/2171470484 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc040070 con 0x7f68cc0c20e0 2024-03-06T07:18:02.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.787+0000 7f68baffd640 1 -- 172.21.15.94:0/2171470484 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68bc030d00 con 0x7f68cc0c20e0 2024-03-06T07:18:02.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.787+0000 7f68baffd640 1 -- 172.21.15.94:0/2171470484 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc035040 con 0x7f68cc0c20e0 2024-03-06T07:18:02.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.787+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc157f00 con 0x7f68cc0c20e0 2024-03-06T07:18:02.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.787+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc1e1f80 con 0x7f68cc0c20e0 2024-03-06T07:18:02.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.788+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc04f330 con 0x7f68cc0c20e0 2024-03-06T07:18:02.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.788+0000 7f68baffd640 1 -- 172.21.15.94:0/2171470484 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f68bc038020 con 0x7f68cc0c20e0 2024-03-06T07:18:02.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.788+0000 7f68baffd640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68ac06b440 0x7f68ac06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:02.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.789+0000 7f68baffd640 1 -- 172.21.15.94:0/2171470484 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f68bc0ab950 con 0x7f68cc0c20e0 2024-03-06T07:18:02.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.789+0000 7f68c9d9b640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68ac06b440 0x7f68ac06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:02.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.789+0000 7f68c9d9b640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68ac06b440 0x7f68ac06d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0703c0 tx=0x7f68c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:02.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.793+0000 7f68baffd640 1 -- 172.21.15.94:0/2171470484 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68bc07ae10 con 0x7f68cc0c20e0 2024-03-06T07:18:02.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.968+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68cc0c1ec0 con 0x7f68cc0c20e0 2024-03-06T07:18:02.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.969+0000 7f68baffd640 1 -- 172.21.15.94:0/2171470484 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10909 (secure 0 0 0) 0x7f68bc07a710 con 0x7f68cc0c20e0 2024-03-06T07:18:02.971 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-06T07:18:02.971 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":13,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:02.299336+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:02.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68ac06b440 msgr2=0x7f68ac06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:02.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68ac06b440 0x7f68ac06d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0703c0 tx=0x7f68c0008040 comp rx=0 tx=0).stop 2024-03-06T07:18:02.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc0c20e0 msgr2=0x7f68cc150fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:02.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc0c20e0 0x7f68cc150fd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f68bc000c00 tx=0x7f68bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 shutdown_connections 2024-03-06T07:18:02.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68ac06b440 0x7f68ac06d900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68cc1528e0 0x7f68cc14dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68cc0c8d40 0x7f68cc14d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 --2- 172.21.15.94:0/2171470484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68cc0c20e0 0x7f68cc150fd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:02.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 >> 172.21.15.94:0/2171470484 conn(0x7f68cc0672b0 msgr2=0x7f68cc06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:02.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 shutdown_connections 2024-03-06T07:18:02.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-06T07:18:02.971+0000 7f68d0edf640 1 -- 172.21.15.94:0/2171470484 wait complete. 2024-03-06T07:18:02.982 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:18:02.983 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:18:02.983 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:18:02.983 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:18:02.985 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:18:02.985 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:18:02.986 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:18:02.987 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:18:02.987 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:18:02.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:18:02.990 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:18:02.991 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:18:02.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:18:02.993 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:18:02.993 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:18:02.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:03.061 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:18:03.062 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:18:03.077 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:18:03.078 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:18:03.078 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:18:03.075+0000 7f5727514640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:18:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.098+0000 7f4cb65d0640 1 Processor -- start 2024-03-06T07:18:03.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.098+0000 7f4cb65d0640 1 -- start start 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4cb65d0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4cb014f360 0x7f4cb0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4cb65d0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00c73d0 con 0x7f4cb00c88c0 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4cb65d0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00be2e0 con 0x7f4cb0158b60 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4cb65d0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb0159700 con 0x7f4cb014f360 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4caffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4cb014f360 0x7f4cb0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4caf7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4cb00c88c0 0x7f4cb00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56600/0 (socket says 172.21.15.94:56600) 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4caf7fe640 1 -- 172.21.15.94:0/943767191 learned_addr learned my addr 172.21.15.94:0/943767191 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4caffff640 1 -- 172.21.15.94:0/943767191 >> v1:172.21.15.145:6789/0 conn(0x7f4cb0158b60 legacy=0x7f4cb0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4caffff640 1 -- 172.21.15.94:0/943767191 >> v1:172.21.15.94:6789/0 conn(0x7f4cb00c88c0 legacy=0x7f4cb00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.099+0000 7f4caffff640 1 -- 172.21.15.94:0/943767191 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00c7670 con 0x7f4cb014f360 2024-03-06T07:18:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.100+0000 7f4caffff640 1 --2- 172.21.15.94:0/943767191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb0155c10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00097a0 tx=0x7f4ca002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fb9c19859ee458 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.100+0000 7f4caeffd640 1 -- 172.21.15.94:0/943767191 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0035020 con 0x7f4cb014f360 2024-03-06T07:18:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.100+0000 7f4caeffd640 1 -- 172.21.15.94:0/943767191 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca0031830 con 0x7f4cb014f360 2024-03-06T07:18:03.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.100+0000 7f4caeffd640 1 -- 172.21.15.94:0/943767191 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0031c00 con 0x7f4cb014f360 2024-03-06T07:18:03.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.101+0000 7f4cb65d0640 1 -- 172.21.15.94:0/943767191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 msgr2=0x7f4cb0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.101+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/943767191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb0155c10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00097a0 tx=0x7f4ca002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.101+0000 7f4cb65d0640 1 -- 172.21.15.94:0/943767191 shutdown_connections 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.101+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/943767191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb0155c10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.101+0000 7f4cb65d0640 1 -- 172.21.15.94:0/943767191 >> 172.21.15.94:0/943767191 conn(0x7f4cb00ac570 msgr2=0x7f4cb00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.101+0000 7f4cb65d0640 1 -- 172.21.15.94:0/943767191 shutdown_connections 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.101+0000 7f4cb65d0640 1 -- 172.21.15.94:0/943767191 wait complete. 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.102+0000 7f4cb65d0640 1 Processor -- start 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.102+0000 7f4cb65d0640 1 -- start start 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4cb00c88c0 0x7f4cb01eb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb01578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4cb0158b60 0x7f4cb00af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00c7c30 con 0x7f4cb014f360 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00c5a50 con 0x7f4cb0158b60 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb00c5bd0 con 0x7f4cb00c88c0 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb01578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb01578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52392/0 (socket says 172.21.15.94:52392) 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 -- 172.21.15.94:0/1978789833 learned_addr learned my addr 172.21.15.94:0/1978789833 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 -- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4cb00c88c0 msgr2=0x7f4cb01eb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb4b6d640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4cb0158b60 0x7f4cb00af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4cb00c88c0 0x7f4cb01eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 -- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4cb0158b60 msgr2=0x7f4cb00af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4cb0158b60 0x7f4cb00af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 -- 172.21.15.94:0/1978789833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00c1030 con 0x7f4cb014f360 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4caf7fe640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb01578d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c500 tx=0x7f4ca400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cacff9640 1 -- 172.21.15.94:0/1978789833 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4017070 con 0x7f4cb014f360 2024-03-06T07:18:03.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cacff9640 1 -- 172.21.15.94:0/1978789833 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4005ce0 con 0x7f4cb014f360 2024-03-06T07:18:03.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb01ef810 con 0x7f4cb014f360 2024-03-06T07:18:03.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cacff9640 1 -- 172.21.15.94:0/1978789833 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40072f0 con 0x7f4cb014f360 2024-03-06T07:18:03.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.103+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb01eef70 con 0x7f4cb014f360 2024-03-06T07:18:03.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.104+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb0075300 con 0x7f4cb014f360 2024-03-06T07:18:03.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.105+0000 7f4cacff9640 1 -- 172.21.15.94:0/1978789833 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f4ca401a070 con 0x7f4cb014f360 2024-03-06T07:18:03.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.105+0000 7f4cacff9640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c9c06b440 0x7f4c9c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.105+0000 7f4cacff9640 1 -- 172.21.15.94:0/1978789833 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f4ca4088e00 con 0x7f4cb014f360 2024-03-06T07:18:03.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.105+0000 7f4caffff640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c9c06b440 0x7f4c9c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.106+0000 7f4caffff640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c9c06b440 0x7f4c9c06d900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0036000 tx=0x7f4ca0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:03.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.109+0000 7f4cacff9640 1 -- 172.21.15.94:0/1978789833 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cb0075300 con 0x7f4cb014f360 2024-03-06T07:18:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.281+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cb0068e00 con 0x7f4cb014f360 2024-03-06T07:18:03.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.281+0000 7f4cacff9640 1 -- 172.21.15.94:0/1978789833 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10909 (secure 0 0 0) 0x7f4cb0068e00 con 0x7f4cb014f360 2024-03-06T07:18:03.283 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:03.284 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":13,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:02.299336+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c9c06b440 msgr2=0x7f4c9c06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c9c06b440 0x7f4c9c06d900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0036000 tx=0x7f4ca0030000 comp rx=0 tx=0).stop 2024-03-06T07:18:03.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 msgr2=0x7f4cb01578d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb01578d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c500 tx=0x7f4ca400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 shutdown_connections 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c9c06b440 0x7f4c9c06d900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4cb0158b60 0x7f4cb00af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb014f360 0x7f4cb01578d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 --2- 172.21.15.94:0/1978789833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4cb00c88c0 0x7f4cb01eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 >> 172.21.15.94:0/1978789833 conn(0x7f4cb00ac570 msgr2=0x7f4cb014dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 shutdown_connections 2024-03-06T07:18:03.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.283+0000 7f4cb65d0640 1 -- 172.21.15.94:0/1978789833 wait complete. 2024-03-06T07:18:03.297 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-06T07:18:03.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:03.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.396+0000 7fcf38943640 1 Processor -- start 2024-03-06T07:18:03.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.396+0000 7fcf38943640 1 -- start start 2024-03-06T07:18:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf38943640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf340cac50 0x7fcf340cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf38943640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcf34158880 0x7fcf34158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf38943640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf340c5380 con 0x7fcf3414f1e0 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf38943640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf340c5960 con 0x7fcf34158880 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf38943640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3414f5c0 con 0x7fcf340cac50 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf3259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcf3414f1e0 0x7fcf34155b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56614/0 (socket says 172.21.15.94:56614) 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf32d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcf34158880 0x7fcf34158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf3259c640 1 -- 172.21.15.94:0/3808206711 learned_addr learned my addr 172.21.15.94:0/3808206711 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf31d9b640 1 --2- 172.21.15.94:0/3808206711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf340cac50 0x7fcf340cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.397+0000 7fcf3159a640 1 -- 172.21.15.94:0/3808206711 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062618744 0 0) 0x7fcf340c5380 con 0x7fcf3414f1e0 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf3159a640 1 -- 172.21.15.94:0/3808206711 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c003520 con 0x7fcf3414f1e0 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf32d9d640 1 -- 172.21.15.94:0/3808206711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf340cac50 msgr2=0x7fcf340cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf32d9d640 1 --2- 172.21.15.94:0/3808206711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf340cac50 0x7fcf340cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf32d9d640 1 -- 172.21.15.94:0/3808206711 >> v1:172.21.15.94:6789/0 conn(0x7fcf3414f1e0 legacy=0x7fcf34155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf32d9d640 1 -- 172.21.15.94:0/3808206711 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf340c8be0 con 0x7fcf34158880 2024-03-06T07:18:03.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf32d9d640 1 --2- 172.21.15.94:0/3808206711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf34158880 0x7fcf34158cb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf240097a0 tx=0x7fcf2402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63063701cdf358c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf3159a640 1 -- 172.21.15.94:0/3808206711 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24038070 con 0x7fcf34158880 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf3159a640 1 -- 172.21.15.94:0/3808206711 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf24031700 con 0x7fcf34158880 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.398+0000 7fcf3159a640 1 -- 172.21.15.94:0/3808206711 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24031ab0 con 0x7fcf34158880 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 -- 172.21.15.94:0/3808206711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf34158880 msgr2=0x7fcf34158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 --2- 172.21.15.94:0/3808206711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf34158880 0x7fcf34158cb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcf240097a0 tx=0x7fcf2402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 -- 172.21.15.94:0/3808206711 shutdown_connections 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 --2- 172.21.15.94:0/3808206711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf34158880 0x7fcf34158cb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 --2- 172.21.15.94:0/3808206711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf340cac50 0x7fcf340cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 -- 172.21.15.94:0/3808206711 >> 172.21.15.94:0/3808206711 conn(0x7fcf340728d0 msgr2=0x7fcf34072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 -- 172.21.15.94:0/3808206711 shutdown_connections 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.399+0000 7fcf38943640 1 -- 172.21.15.94:0/3808206711 wait complete. 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.400+0000 7fcf38943640 1 Processor -- start 2024-03-06T07:18:03.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.400+0000 7fcf38943640 1 -- start start 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf38943640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf340cac50 0x7fcf340b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf38943640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf3414f1e0 0x7fcf341480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf38943640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf34158880 0x7fcf341e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf38943640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340c8ec0 con 0x7fcf3414f1e0 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf38943640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3406d2a0 con 0x7fcf340cac50 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf38943640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf340c8860 con 0x7fcf34158880 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf3414f1e0 0x7fcf341480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf3414f1e0 0x7fcf341480b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52418/0 (socket says 172.21.15.94:52418) 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 -- 172.21.15.94:0/2825339845 learned_addr learned my addr 172.21.15.94:0/2825339845 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf3259c640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf340cac50 0x7fcf340b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf32d9d640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf34158880 0x7fcf341e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 -- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf34158880 msgr2=0x7fcf341e7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf34158880 0x7fcf341e7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 -- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf340cac50 msgr2=0x7fcf340b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf340cac50 0x7fcf340b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 -- 172.21.15.94:0/2825339845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf341472b0 con 0x7fcf3414f1e0 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf32d9d640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf34158880 0x7fcf341e7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf3259c640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf340cac50 0x7fcf340b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.401+0000 7fcf31d9b640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf3414f1e0 0x7fcf341480b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800efc0 tx=0x7fcf2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.402+0000 7fcf22ffd640 1 -- 172.21.15.94:0/2825339845 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28018020 con 0x7fcf3414f1e0 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.402+0000 7fcf22ffd640 1 -- 172.21.15.94:0/2825339845 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28014480 con 0x7fcf3414f1e0 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.402+0000 7fcf22ffd640 1 -- 172.21.15.94:0/2825339845 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28014850 con 0x7fcf3414f1e0 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.402+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf341eb710 con 0x7fcf3414f1e0 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.402+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf341eb430 con 0x7fcf3414f1e0 2024-03-06T07:18:03.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.403+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3404f330 con 0x7fcf3414f1e0 2024-03-06T07:18:03.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.403+0000 7fcf22ffd640 1 -- 172.21.15.94:0/2825339845 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fcf28005d10 con 0x7fcf3414f1e0 2024-03-06T07:18:03.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.403+0000 7fcf22ffd640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b440 0x7fcf1406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:03.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.403+0000 7fcf22ffd640 1 -- 172.21.15.94:0/2825339845 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4846+0+0 (secure 0 0 0) 0x7fcf2808a760 con 0x7fcf3414f1e0 2024-03-06T07:18:03.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.404+0000 7fcf3259c640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b440 0x7fcf1406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:03.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.404+0000 7fcf3259c640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b440 0x7fcf1406d900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c005880 tx=0x7fcf2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:03.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.408+0000 7fcf22ffd640 1 -- 172.21.15.94:0/2825339845 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf28022070 con 0x7fcf3414f1e0 2024-03-06T07:18:03.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.612+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf3406cfa0 con 0x7fcf3414f1e0 2024-03-06T07:18:03.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.613+0000 7fcf22ffd640 1 -- 172.21.15.94:0/2825339845 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fcf341472b0 con 0x7fcf3414f1e0 2024-03-06T07:18:03.616 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:03.616 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 2 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b440 msgr2=0x7fcf1406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b440 0x7fcf1406d900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c005880 tx=0x7fcf2c009040 comp rx=0 tx=0).stop 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf3414f1e0 msgr2=0x7fcf341480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf3414f1e0 0x7fcf341480b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800efc0 tx=0x7fcf2800c490 comp rx=0 tx=0).stop 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 shutdown_connections 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b440 0x7fcf1406d900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf34158880 0x7fcf341e7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf3414f1e0 0x7fcf341480b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 --2- 172.21.15.94:0/2825339845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf340cac50 0x7fcf340b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:03.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 >> 172.21.15.94:0/2825339845 conn(0x7fcf340728d0 msgr2=0x7fcf340c9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:03.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 shutdown_connections 2024-03-06T07:18:03.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:03.615+0000 7fcf38943640 1 -- 172.21.15.94:0/2825339845 wait complete. 2024-03-06T07:18:05.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:05.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.733+0000 7f988cabb640 1 Processor -- start 2024-03-06T07:18:05.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.733+0000 7f988cabb640 1 -- start start 2024-03-06T07:18:05.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.733+0000 7f988cabb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9888072720 0x7f9888156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988cabb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 0x7f98880c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988cabb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9888157340 con 0x7f988806f7b0 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988cabb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98880c3740 con 0x7f9888072720 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988cabb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9888156d80 con 0x7f98881508c0 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f9885d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f988806f7b0 0x7f98881501b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56616/0 (socket says 172.21.15.94:56616) 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f9885d9b640 1 -- 172.21.15.94:0/3811808279 learned_addr learned my addr 172.21.15.94:0/3811808279 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988659c640 1 --2- 172.21.15.94:0/3811808279 >> v2:172.21.15.145:3300/0 conn(0x7f9888072720 0x7f9888156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f9886d9d640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 0x7f98880c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832471117 0 0) 0x7f9888157340 con 0x7f988806f7b0 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9870003520 con 0x7f988806f7b0 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943257487 0 0) 0x7f9870003520 con 0x7f988806f7b0 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 msgr2=0x7f98880c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 0x7f98880c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 >> v2:172.21.15.145:3300/0 conn(0x7f9888072720 msgr2=0x7f9888156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 --2- 172.21.15.94:0/3811808279 >> v2:172.21.15.145:3300/0 conn(0x7f9888072720 0x7f9888156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888157060 con 0x7f988806f7b0 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f987c001670 con 0x7f988806f7b0 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 >> v1:172.21.15.94:6789/0 conn(0x7f988806f7b0 legacy=0x7f98881501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98700039b0 0x7f9870003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 0x7f9870004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9870004d40 0x7f98700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98880c3740 con 0x7f98700039b0 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9888156d80 con 0x7f9870004d40 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987000b9c0 con 0x7f98881508c0 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f988659c640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98700039b0 0x7f9870003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f9885d9b640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 0x7f9870004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.734+0000 7f9886d9d640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9870004d40 0x7f98700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988659c640 1 -- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 msgr2=0x7f9870004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988659c640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98881508c0 0x7f9870004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988659c640 1 -- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9870004d40 msgr2=0x7f98700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988659c640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9870004d40 0x7f98700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988659c640 1 -- 172.21.15.94:0/3811808279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9870003520 con 0x7f98700039b0 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988659c640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98700039b0 0x7f9870003dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9878009ac0 tx=0x7f9870005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bafb277291d7d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878039070 con 0x7f98700039b0 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987802c710 con 0x7f98700039b0 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.735+0000 7f988559a640 1 -- 172.21.15.94:0/3811808279 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878009690 con 0x7f98700039b0 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.736+0000 7f988cabb640 1 -- 172.21.15.94:0/3811808279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98700039b0 msgr2=0x7f9870003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.736+0000 7f988cabb640 1 --2- 172.21.15.94:0/3811808279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98700039b0 0x7f9870003dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9878009ac0 tx=0x7f9870005830 comp rx=0 tx=0).stop 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.736+0000 7f9886d9d640 1 -- 172.21.15.94:0/3811808279 reap_dead start 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.736+0000 7f988cabb640 1 -- 172.21.15.94:0/3811808279 shutdown_connections 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.736+0000 7f988cabb640 1 -- 172.21.15.94:0/3811808279 >> 172.21.15.94:0/3811808279 conn(0x7f98880aa630 msgr2=0x7f98880aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.736+0000 7f988cabb640 1 -- 172.21.15.94:0/3811808279 shutdown_connections 2024-03-06T07:18:05.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.736+0000 7f988cabb640 1 -- 172.21.15.94:0/3811808279 wait complete. 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.737+0000 7f988cabb640 1 Processor -- start 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.737+0000 7f988cabb640 1 -- start start 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f988cabb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98700039b0 0x7f98881ee920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f988cabb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f988806c370 0x7f98880c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f988cabb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f988806dce0 0x7f98881eb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f988cabb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9888156af0 con 0x7f988806c370 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f988cabb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9888156970 con 0x7f988806dce0 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f988cabb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9888072b00 con 0x7f98700039b0 2024-03-06T07:18:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f988806c370 0x7f98880c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f988806c370 0x7f98880c1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52426/0 (socket says 172.21.15.94:52426) 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 -- 172.21.15.94:0/1668742597 learned_addr learned my addr 172.21.15.94:0/1668742597 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 -- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98700039b0 msgr2=0x7f98881ee920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98700039b0 0x7f98881ee920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 -- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f988806dce0 msgr2=0x7f98881eb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f988806dce0 0x7f98881eb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 -- 172.21.15.94:0/1668742597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888154d30 con 0x7f988806c370 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.738+0000 7f9885d9b640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f988806c370 0x7f98880c1f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f987c007340 tx=0x7f987c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.739+0000 7f9876ffd640 1 -- 172.21.15.94:0/1668742597 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00f8c0 con 0x7f988806c370 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.739+0000 7f9876ffd640 1 -- 172.21.15.94:0/1668742597 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c018070 con 0x7f988806c370 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.739+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9888155010 con 0x7f988806c370 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.739+0000 7f9876ffd640 1 -- 172.21.15.94:0/1668742597 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00fcc0 con 0x7f988806c370 2024-03-06T07:18:05.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.739+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98880c2460 con 0x7f988806c370 2024-03-06T07:18:05.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.740+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988804f330 con 0x7f988806c370 2024-03-06T07:18:05.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.740+0000 7f9876ffd640 1 -- 172.21.15.94:0/1668742597 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f987c01d460 con 0x7f988806c370 2024-03-06T07:18:05.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.740+0000 7f9876ffd640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f986806b4d0 0x7f986806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:05.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.740+0000 7f9876ffd640 1 -- 172.21.15.94:0/1668742597 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f987c089f00 con 0x7f988806c370 2024-03-06T07:18:05.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.741+0000 7f988659c640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f986806b4d0 0x7f986806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:05.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.741+0000 7f988659c640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f986806b4d0 0x7f986806d990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9878036000 tx=0x7f987802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:05.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.745+0000 7f9876ffd640 1 -- 172.21.15.94:0/1668742597 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f987c00fe60 con 0x7f988806c370 2024-03-06T07:18:05.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.952+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98880c41c0 con 0x7f988806c370 2024-03-06T07:18:05.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.952+0000 7f9876ffd640 1 -- 172.21.15.94:0/1668742597 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3698 (secure 0 0 0) 0x7f987c059160 con 0x7f988806c370 2024-03-06T07:18:05.955 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:05.955 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":34136,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:18:05.955 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 3 2024-03-06T07:18:05.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f986806b4d0 msgr2=0x7f986806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f986806b4d0 0x7f986806d990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9878036000 tx=0x7f987802e000 comp rx=0 tx=0).stop 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f988806c370 msgr2=0x7f98880c1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f988806c370 0x7f98880c1f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f987c007340 tx=0x7f987c007800 comp rx=0 tx=0).stop 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 shutdown_connections 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f986806b4d0 0x7f986806d990 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f988806dce0 0x7f98881eb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f988806c370 0x7f98880c1f20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 --2- 172.21.15.94:0/1668742597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98700039b0 0x7f98881ee920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 >> 172.21.15.94:0/1668742597 conn(0x7f98880aa630 msgr2=0x7f98880c4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 shutdown_connections 2024-03-06T07:18:05.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:05.954+0000 7f988cabb640 1 -- 172.21.15.94:0/1668742597 wait complete. 2024-03-06T07:18:05.967 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:06.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.065+0000 7f3fac925640 1 Processor -- start 2024-03-06T07:18:06.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.065+0000 7f3fac925640 1 -- start start 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fac925640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8072720 0x7f3fa81564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fac925640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3fa8156a00 0x7f3fa8150290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fac925640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa8156f90 con 0x7f3fa81507d0 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fac925640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa80c38d0 con 0x7f3fa8156a00 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fac925640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa8157270 con 0x7f3fa8072720 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa6d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3fa81507d0 0x7f3fa80c3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56628/0 (socket says 172.21.15.94:56628) 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa6d9d640 1 -- 172.21.15.94:0/3787025709 learned_addr learned my addr 172.21.15.94:0/3787025709 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa659c640 1 --2- 172.21.15.94:0/3787025709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8072720 0x7f3fa81564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3f9ffff640 1 --2- 172.21.15.94:0/3787025709 >> v2:172.21.15.145:3300/0 conn(0x7f3fa8156a00 0x7f3fa8150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa5d9b640 1 -- 172.21.15.94:0/3787025709 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342505316 0 0) 0x7f3fa8156f90 con 0x7f3fa81507d0 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa5d9b640 1 -- 172.21.15.94:0/3787025709 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f94003520 con 0x7f3fa81507d0 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa659c640 1 -- 172.21.15.94:0/3787025709 >> v2:172.21.15.145:3300/0 conn(0x7f3fa8156a00 msgr2=0x7f3fa8150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa659c640 1 --2- 172.21.15.94:0/3787025709 >> v2:172.21.15.145:3300/0 conn(0x7f3fa8156a00 0x7f3fa8150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa659c640 1 -- 172.21.15.94:0/3787025709 >> v1:172.21.15.94:6789/0 conn(0x7f3fa81507d0 legacy=0x7f3fa80c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3fa659c640 1 -- 172.21.15.94:0/3787025709 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa80c3280 con 0x7f3fa8072720 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.066+0000 7f3f9ffff640 1 --2- 172.21.15.94:0/3787025709 >> v2:172.21.15.145:3300/0 conn(0x7f3fa8156a00 0x7f3fa8150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.067+0000 7f3fa659c640 1 --2- 172.21.15.94:0/3787025709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8072720 0x7f3fa81564c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f980097d0 tx=0x7f3f9802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=837abc9f5002355e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.067+0000 7f3fa5d9b640 1 -- 172.21.15.94:0/3787025709 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98039070 con 0x7f3fa8072720 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.067+0000 7f3fa5d9b640 1 -- 172.21.15.94:0/3787025709 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f980316f0 con 0x7f3fa8072720 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.067+0000 7f3fa5d9b640 1 -- 172.21.15.94:0/3787025709 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98031aa0 con 0x7f3fa8072720 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 -- 172.21.15.94:0/3787025709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8072720 msgr2=0x7f3fa81564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 --2- 172.21.15.94:0/3787025709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8072720 0x7f3fa81564c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f980097d0 tx=0x7f3f9802b790 comp rx=0 tx=0).stop 2024-03-06T07:18:06.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 -- 172.21.15.94:0/3787025709 shutdown_connections 2024-03-06T07:18:06.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 --2- 172.21.15.94:0/3787025709 >> v2:172.21.15.145:3300/0 conn(0x7f3fa8156a00 0x7f3fa8150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 --2- 172.21.15.94:0/3787025709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8072720 0x7f3fa81564c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 -- 172.21.15.94:0/3787025709 >> 172.21.15.94:0/3787025709 conn(0x7f3fa80aa630 msgr2=0x7f3fa80aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:06.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 -- 172.21.15.94:0/3787025709 shutdown_connections 2024-03-06T07:18:06.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.068+0000 7f3fac925640 1 -- 172.21.15.94:0/3787025709 wait complete. 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.069+0000 7f3fac925640 1 Processor -- start 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.069+0000 7f3fac925640 1 -- start start 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fa8072720 0x7f3fa80c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fa81507d0 0x7f3fa8149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8156a00 0x7f3fa8149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa8072b00 con 0x7f3fa8072720 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa806f8f0 con 0x7f3fa81507d0 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa8072c80 con 0x7f3fa8156a00 2024-03-06T07:18:06.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fa8072720 0x7f3fa80c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fa8072720 0x7f3fa80c1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52432/0 (socket says 172.21.15.94:52432) 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 -- 172.21.15.94:0/2072271405 learned_addr learned my addr 172.21.15.94:0/2072271405 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa6d9d640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8156a00 0x7f3fa8149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3f9ffff640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fa81507d0 0x7f3fa8149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 -- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8156a00 msgr2=0x7f3fa8149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8156a00 0x7f3fa8149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 -- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fa81507d0 msgr2=0x7f3fa8149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fa81507d0 0x7f3fa8149a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 -- 172.21.15.94:0/2072271405 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa806d5e0 con 0x7f3fa8072720 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fa659c640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fa8072720 0x7f3fa80c1f20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f98037040 tx=0x7f3f980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3f9effd640 1 -- 172.21.15.94:0/2072271405 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98039040 con 0x7f3fa8072720 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3f9effd640 1 -- 172.21.15.94:0/2072271405 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f98004230 con 0x7f3fa8072720 2024-03-06T07:18:06.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa80ac1e0 con 0x7f3fa8072720 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3f9effd640 1 -- 172.21.15.94:0/2072271405 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98003610 con 0x7f3fa8072720 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.070+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa80c2460 con 0x7f3fa8072720 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.071+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa806d7b0 con 0x7f3fa8072720 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.072+0000 7f3f9effd640 1 -- 172.21.15.94:0/2072271405 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f9803c070 con 0x7f3fa8072720 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.072+0000 7f3f9effd640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f8806b4d0 0x7f3f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.072+0000 7f3f9effd640 1 -- 172.21.15.94:0/2072271405 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3f980ab890 con 0x7f3fa8072720 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.072+0000 7f3f9ffff640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f8806b4d0 0x7f3f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.073+0000 7f3f9ffff640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f8806b4d0 0x7f3f8806d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009560 tx=0x7f3f90009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.076+0000 7f3f9effd640 1 -- 172.21.15.94:0/2072271405 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f9804c030 con 0x7f3fa8072720 2024-03-06T07:18:06.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.249+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fa804f330 con 0x7f3fa8072720 2024-03-06T07:18:06.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.250+0000 7f3f9effd640 1 -- 172.21.15.94:0/2072271405 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12884 (secure 0 0 0) 0x7f3f98043030 con 0x7f3fa8072720 2024-03-06T07:18:06.252 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:06.252 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":16,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:05.390725+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:06.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f8806b4d0 msgr2=0x7f3f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f8806b4d0 0x7f3f8806d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009560 tx=0x7f3f90009190 comp rx=0 tx=0).stop 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fa8072720 msgr2=0x7f3fa80c1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fa8072720 0x7f3fa80c1f20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f98037040 tx=0x7f3f980046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 shutdown_connections 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f8806b4d0 0x7f3f8806d990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fa8156a00 0x7f3fa8149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fa81507d0 0x7f3fa8149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 --2- 172.21.15.94:0/2072271405 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fa8072720 0x7f3fa80c1f20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 >> 172.21.15.94:0/2072271405 conn(0x7f3fa80aa630 msgr2=0x7f3fa80a9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 shutdown_connections 2024-03-06T07:18:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.252+0000 7f3fac925640 1 -- 172.21.15.94:0/2072271405 wait complete. 2024-03-06T07:18:06.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:18:06.265 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:06.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.365+0000 7f3859afb640 1 Processor -- start 2024-03-06T07:18:06.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.365+0000 7f3859afb640 1 -- start start 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3859afb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f385406c1f0 0x7f38540c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3859afb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385406cad0 0x7f3854154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3859afb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38540c2790 con 0x7f385414ef70 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3859afb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38540c3090 con 0x7f385406c1f0 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3859afb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385414f350 con 0x7f385406cad0 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f38537fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f385414ef70 0x7f38540c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56634/0 (socket says 172.21.15.94:56634) 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f38537fe640 1 -- 172.21.15.94:0/2105983882 learned_addr learned my addr 172.21.15.94:0/2105983882 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3852ffd640 1 --2- 172.21.15.94:0/2105983882 >> v2:172.21.15.145:3300/0 conn(0x7f385406c1f0 0x7f38540c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3853fff640 1 --2- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385406cad0 0x7f3854154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f38527fc640 1 -- 172.21.15.94:0/2105983882 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426857830 0 0) 0x7f38540c2790 con 0x7f385414ef70 2024-03-06T07:18:06.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f38527fc640 1 -- 172.21.15.94:0/2105983882 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f383c004630 con 0x7f385414ef70 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3852ffd640 1 -- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385406cad0 msgr2=0x7f3854154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3852ffd640 1 --2- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385406cad0 0x7f3854154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3852ffd640 1 -- 172.21.15.94:0/2105983882 >> v1:172.21.15.94:6789/0 conn(0x7f385414ef70 legacy=0x7f38540c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.366+0000 7f3852ffd640 1 -- 172.21.15.94:0/2105983882 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540c1540 con 0x7f385406c1f0 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.367+0000 7f3852ffd640 1 --2- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406c1f0 0x7f38540c1820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38440097a0 tx=0x7f384402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c85e24d11810f56e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.367+0000 7f38527fc640 1 -- 172.21.15.94:0/2105983882 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38440315a0 con 0x7f385406c1f0 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.367+0000 7f38527fc640 1 -- 172.21.15.94:0/2105983882 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3844037070 con 0x7f385406c1f0 2024-03-06T07:18:06.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.367+0000 7f38527fc640 1 -- 172.21.15.94:0/2105983882 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844031960 con 0x7f385406c1f0 2024-03-06T07:18:06.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 -- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406c1f0 msgr2=0x7f38540c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 --2- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406c1f0 0x7f38540c1820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38440097a0 tx=0x7f384402b840 comp rx=0 tx=0).stop 2024-03-06T07:18:06.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 -- 172.21.15.94:0/2105983882 shutdown_connections 2024-03-06T07:18:06.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 --2- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385406cad0 0x7f3854154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 --2- 172.21.15.94:0/2105983882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406c1f0 0x7f38540c1820 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 -- 172.21.15.94:0/2105983882 >> 172.21.15.94:0/2105983882 conn(0x7f38540c6710 msgr2=0x7f38540c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 -- 172.21.15.94:0/2105983882 shutdown_connections 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.368+0000 7f3859afb640 1 -- 172.21.15.94:0/2105983882 wait complete. 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 Processor -- start 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 -- start start 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f385406c1f0 0x7f38541ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406cad0 0x7f385414ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385414ef70 0x7f385414b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3854155750 con 0x7f385406c1f0 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385406cf60 con 0x7f385406cad0 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.369+0000 7f3859afb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3854155170 con 0x7f385414ef70 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f385406c1f0 0x7f38541ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f385406c1f0 0x7f38541ee2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52448/0 (socket says 172.21.15.94:52448) 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 -- 172.21.15.94:0/300870228 learned_addr learned my addr 172.21.15.94:0/300870228 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f3852ffd640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406cad0 0x7f385414ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f3853fff640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385414ef70 0x7f385414b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 -- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385414ef70 msgr2=0x7f385414b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385414ef70 0x7f385414b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 -- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406cad0 msgr2=0x7f385414ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406cad0 0x7f385414ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 -- 172.21.15.94:0/300870228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3854142a60 con 0x7f385406c1f0 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f3853fff640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385414ef70 0x7f385414b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f38537fe640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f385406c1f0 0x7f38541ee2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f384800c390 tx=0x7f384800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f383bfff640 1 -- 172.21.15.94:0/300870228 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848017070 con 0x7f385406c1f0 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f383bfff640 1 -- 172.21.15.94:0/300870228 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3848005ce0 con 0x7f385406c1f0 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38541eed20 con 0x7f385406c1f0 2024-03-06T07:18:06.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f383bfff640 1 -- 172.21.15.94:0/300870228 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38480072f0 con 0x7f385406c1f0 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.370+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385414b7d0 con 0x7f385406c1f0 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.371+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385404f330 con 0x7f385406c1f0 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.371+0000 7f383bfff640 1 -- 172.21.15.94:0/300870228 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f384801a070 con 0x7f385406c1f0 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.372+0000 7f383bfff640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f384006b4d0 0x7f384006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.372+0000 7f383bfff640 1 -- 172.21.15.94:0/300870228 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3848089220 con 0x7f385406c1f0 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.372+0000 7f3852ffd640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f384006b4d0 0x7f384006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.372+0000 7f3852ffd640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f384006b4d0 0x7f384006d990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3844036000 tx=0x7f384403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.376+0000 7f383bfff640 1 -- 172.21.15.94:0/300870228 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3848057da0 con 0x7f385406c1f0 2024-03-06T07:18:06.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.547+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38540c1110 con 0x7f385406c1f0 2024-03-06T07:18:06.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.547+0000 7f383bfff640 1 -- 172.21.15.94:0/300870228 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f384805e0f0 con 0x7f385406c1f0 2024-03-06T07:18:06.549 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:06.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f384006b4d0 msgr2=0x7f384006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f384006b4d0 0x7f384006d990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3844036000 tx=0x7f384403d040 comp rx=0 tx=0).stop 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f385406c1f0 msgr2=0x7f38541ee2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f385406c1f0 0x7f38541ee2a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f384800c390 tx=0x7f384800c850 comp rx=0 tx=0).stop 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 shutdown_connections 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f384006b4d0 0x7f384006d990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f385414ef70 0x7f385414b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f385406cad0 0x7f385414ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 --2- 172.21.15.94:0/300870228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f385406c1f0 0x7f38541ee2a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 >> 172.21.15.94:0/300870228 conn(0x7f38540c6710 msgr2=0x7f38540c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 shutdown_connections 2024-03-06T07:18:06.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.549+0000 7f3859afb640 1 -- 172.21.15.94:0/300870228 wait complete. 2024-03-06T07:18:06.562 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:18:06.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.660+0000 7f68daadf640 1 Processor -- start 2024-03-06T07:18:06.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.660+0000 7f68daadf640 1 -- start start 2024-03-06T07:18:06.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68daadf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68d4152960 0x7f68d406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68daadf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d40ac260 0x7f68d41559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68daadf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d4152d40 con 0x7f68d41592d0 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68daadf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d4066e30 con 0x7f68d4152960 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68daadf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40c1dc0 con 0x7f68d40ac260 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d887b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68d4152960 0x7f68d406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d887b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68d4152960 0x7f68d406dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52460/0 (socket says 172.21.15.94:52460) 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d887b640 1 -- 172.21.15.94:0/476386957 learned_addr learned my addr 172.21.15.94:0/476386957 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d907c640 1 --2- 172.21.15.94:0/476386957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d40ac260 0x7f68d41559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d887b640 1 -- 172.21.15.94:0/476386957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d40ac260 msgr2=0x7f68d41559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d887b640 1 --2- 172.21.15.94:0/476386957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d40ac260 0x7f68d41559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d887b640 1 -- 172.21.15.94:0/476386957 >> v1:172.21.15.145:6789/0 conn(0x7f68d41592d0 legacy=0x7f68d41552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.661+0000 7f68d887b640 1 -- 172.21.15.94:0/476386957 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40c1b50 con 0x7f68d4152960 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68d887b640 1 --2- 172.21.15.94:0/476386957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d4152960 0x7f68d406dc90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009870 tx=0x7f68c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe4576850bd8923 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68d37fe640 1 -- 172.21.15.94:0/476386957 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8034070 con 0x7f68d4152960 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68d37fe640 1 -- 172.21.15.94:0/476386957 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c8030840 con 0x7f68d4152960 2024-03-06T07:18:06.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68d37fe640 1 -- 172.21.15.94:0/476386957 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8030c10 con 0x7f68d4152960 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68daadf640 1 -- 172.21.15.94:0/476386957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d4152960 msgr2=0x7f68d406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68daadf640 1 --2- 172.21.15.94:0/476386957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d4152960 0x7f68d406dc90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009870 tx=0x7f68c802b980 comp rx=0 tx=0).stop 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68daadf640 1 -- 172.21.15.94:0/476386957 shutdown_connections 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68daadf640 1 --2- 172.21.15.94:0/476386957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d40ac260 0x7f68d41559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68daadf640 1 --2- 172.21.15.94:0/476386957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d4152960 0x7f68d406dc90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.662+0000 7f68daadf640 1 -- 172.21.15.94:0/476386957 >> 172.21.15.94:0/476386957 conn(0x7f68d40c70b0 msgr2=0x7f68d40c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.663+0000 7f68daadf640 1 -- 172.21.15.94:0/476386957 shutdown_connections 2024-03-06T07:18:06.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.663+0000 7f68daadf640 1 -- 172.21.15.94:0/476386957 wait complete. 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 Processor -- start 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 -- start start 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d40ac260 0x7f68d41df090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68d4152960 0x7f68d40c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d41eba00 0x7f68d41eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40c20a0 con 0x7f68d40ac260 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4159760 con 0x7f68d4152960 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68daadf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40c2220 con 0x7f68d41eba00 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68d887b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d40ac260 0x7f68d41df090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68d887b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d40ac260 0x7f68d41df090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52470/0 (socket says 172.21.15.94:52470) 2024-03-06T07:18:06.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.664+0000 7f68d887b640 1 -- 172.21.15.94:0/3864217751 learned_addr learned my addr 172.21.15.94:0/3864217751 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d3fff640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68d4152960 0x7f68d40c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d907c640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d41eba00 0x7f68d41eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d887b640 1 -- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d41eba00 msgr2=0x7f68d41eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d887b640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d41eba00 0x7f68d41eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d887b640 1 -- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68d4152960 msgr2=0x7f68d40c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d887b640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68d4152960 0x7f68d40c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d887b640 1 -- 172.21.15.94:0/3864217751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d4154e50 con 0x7f68d40ac260 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d887b640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d40ac260 0x7f68d41df090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f68c8000c00 tx=0x7f68c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d17fa640 1 -- 172.21.15.94:0/3864217751 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8048070 con 0x7f68d40ac260 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d17fa640 1 -- 172.21.15.94:0/3864217751 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c8030640 con 0x7f68d40ac260 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68d17fa640 1 -- 172.21.15.94:0/3864217751 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8034040 con 0x7f68d40ac260 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d4155130 con 0x7f68d40ac260 2024-03-06T07:18:06.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.665+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d40c4630 con 0x7f68d40ac260 2024-03-06T07:18:06.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.666+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d404f330 con 0x7f68d40ac260 2024-03-06T07:18:06.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.666+0000 7f68d17fa640 1 -- 172.21.15.94:0/3864217751 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68c804b070 con 0x7f68d40ac260 2024-03-06T07:18:06.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.667+0000 7f68d17fa640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68b806b500 0x7f68b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.667+0000 7f68d17fa640 1 -- 172.21.15.94:0/3864217751 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f68c80abd40 con 0x7f68d40ac260 2024-03-06T07:18:06.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.667+0000 7f68d3fff640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68b806b500 0x7f68b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.667+0000 7f68d3fff640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68b806b500 0x7f68b806d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f68d40c6980 tx=0x7f68c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.671+0000 7f68d17fa640 1 -- 172.21.15.94:0/3864217751 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68c807afa0 con 0x7f68d40ac260 2024-03-06T07:18:06.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.845+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f68d4155300 con 0x7f68d40ac260 2024-03-06T07:18:06.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.845+0000 7f68d17fa640 1 -- 172.21.15.94:0/3864217751 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f68c8030810 con 0x7f68d40ac260 2024-03-06T07:18:06.847 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:06.847 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:18:06.847 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:18:06.847 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:18:06.848 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:18:06.848 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:18:06.848 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:18:06.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68b806b500 msgr2=0x7f68b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68b806b500 0x7f68b806d9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f68d40c6980 tx=0x7f68c0009040 comp rx=0 tx=0).stop 2024-03-06T07:18:06.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d40ac260 msgr2=0x7f68d41df090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d40ac260 0x7f68d41df090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f68c8000c00 tx=0x7f68c8004730 comp rx=0 tx=0).stop 2024-03-06T07:18:06.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 shutdown_connections 2024-03-06T07:18:06.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68b806b500 0x7f68b806d9c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68d41eba00 0x7f68d41eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68d4152960 0x7f68d40c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 --2- 172.21.15.94:0/3864217751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68d40ac260 0x7f68d41df090 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 >> 172.21.15.94:0/3864217751 conn(0x7f68d40c70b0 msgr2=0x7f68d406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:06.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 shutdown_connections 2024-03-06T07:18:06.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.847+0000 7f68daadf640 1 -- 172.21.15.94:0/3864217751 wait complete. 2024-03-06T07:18:06.859 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:18:06.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:18:06.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.961+0000 7f3358cb5640 1 Processor -- start 2024-03-06T07:18:06.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3358cb5640 1 -- start start 2024-03-06T07:18:06.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3358cb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f335406c340 0x7f3354152860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3358cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 0x7f335414f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3358cb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33540c7060 con 0x7f335414f540 2024-03-06T07:18:06.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3358cb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33540c2b10 con 0x7f335406c340 2024-03-06T07:18:06.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3358cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33540c2440 con 0x7f33540a6cc0 2024-03-06T07:18:06.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3352d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f335414f540 0x7f3354155370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56640/0 (socket says 172.21.15.94:56640) 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.962+0000 7f3352d9d640 1 -- 172.21.15.94:0/957344819 learned_addr learned my addr 172.21.15.94:0/957344819 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f334bfff640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 0x7f335414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f335259c640 1 --2- 172.21.15.94:0/957344819 >> v2:172.21.15.145:3300/0 conn(0x7f335406c340 0x7f3354152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488115900 0 0) 0x7f33540c7060 con 0x7f335414f540 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3340003520 con 0x7f335414f540 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478964427 0 0) 0x7f3340003520 con 0x7f335414f540 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 msgr2=0x7f335414f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 0x7f335414f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 >> v2:172.21.15.145:3300/0 conn(0x7f335406c340 msgr2=0x7f3354152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 --2- 172.21.15.94:0/957344819 >> v2:172.21.15.145:3300/0 conn(0x7f335406c340 0x7f3354152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33540c7340 con 0x7f335414f540 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f334c0019f0 con 0x7f335414f540 2024-03-06T07:18:06.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 >> v1:172.21.15.94:6789/0 conn(0x7f335414f540 legacy=0x7f3354155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 0x7f33400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3340003f30 0x7f3340004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340004d40 0x7f33400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33540c2b10 con 0x7f3340003f30 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33540c2440 con 0x7f3340004d40 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334000baa0 con 0x7f33540a6cc0 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f334bfff640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3340003f30 0x7f3340004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f335259c640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 0x7f33400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3352d9d640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340004d40 0x7f33400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f334bfff640 1 -- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 msgr2=0x7f33400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f334bfff640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 0x7f33400039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f334bfff640 1 -- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340004d40 msgr2=0x7f33400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f334bfff640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340004d40 0x7f33400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f334bfff640 1 -- 172.21.15.94:0/957344819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340003520 con 0x7f3340003f30 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f3352d9d640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340004d40 0x7f33400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.963+0000 7f335259c640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540a6cc0 0x7f33400039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.964+0000 7f334bfff640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3340003f30 0x7f3340004800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f333c009ac0 tx=0x7f3340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642f39eaa04b6c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.964+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c03b070 con 0x7f3340003f30 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.964+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c02e8c0 con 0x7f3340003f30 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.964+0000 7f3351d9b640 1 -- 172.21.15.94:0/957344819 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c009690 con 0x7f3340003f30 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.965+0000 7f3358cb5640 1 -- 172.21.15.94:0/957344819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3340003f30 msgr2=0x7f3340004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.965+0000 7f3358cb5640 1 --2- 172.21.15.94:0/957344819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3340003f30 0x7f3340004800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f333c009ac0 tx=0x7f3340005830 comp rx=0 tx=0).stop 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.965+0000 7f3352d9d640 1 -- 172.21.15.94:0/957344819 reap_dead start 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.965+0000 7f3358cb5640 1 -- 172.21.15.94:0/957344819 shutdown_connections 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.965+0000 7f3358cb5640 1 -- 172.21.15.94:0/957344819 >> 172.21.15.94:0/957344819 conn(0x7f33540a9540 msgr2=0x7f33540a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.965+0000 7f3358cb5640 1 -- 172.21.15.94:0/957344819 shutdown_connections 2024-03-06T07:18:06.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.965+0000 7f3358cb5640 1 -- 172.21.15.94:0/957344819 wait complete. 2024-03-06T07:18:06.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.966+0000 7f3358cb5640 1 Processor -- start 2024-03-06T07:18:06.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.966+0000 7f3358cb5640 1 -- start start 2024-03-06T07:18:06.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340003f30 0x7f3354072610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3354144070 0x7f33541e9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3354072b50 0x7f33541ea3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335406c8a0 con 0x7f3354072b50 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335406c720 con 0x7f3340003f30 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335414f000 con 0x7f3354144070 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3354072b50 0x7f33541ea3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3354072b50 0x7f33541ea3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52484/0 (socket says 172.21.15.94:52484) 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 -- 172.21.15.94:0/1730113737 learned_addr learned my addr 172.21.15.94:0/1730113737 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 -- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3354144070 msgr2=0x7f33541e9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3354144070 0x7f33541e9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 -- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340003f30 msgr2=0x7f3354072610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f335259c640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340003f30 0x7f3354072610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340003f30 0x7f3354072610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 -- 172.21.15.94:0/1730113737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33541505c0 con 0x7f3354072b50 2024-03-06T07:18:06.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f335259c640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340003f30 0x7f3354072610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3352d9d640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3354072b50 0x7f33541ea3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f335406f480 tx=0x7f334c007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f334affd640 1 -- 172.21.15.94:0/1730113737 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c00ea80 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f334affd640 1 -- 172.21.15.94:0/1730113737 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c018070 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33541518b0 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f334affd640 1 -- 172.21.15.94:0/1730113737 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c00ee60 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.967+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3354150b50 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.968+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335404f330 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.969+0000 7f334affd640 1 -- 172.21.15.94:0/1730113737 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f334c01d460 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.969+0000 7f334affd640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f333406b4d0 0x7f333406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.969+0000 7f334affd640 1 -- 172.21.15.94:0/1730113737 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f334c089e80 con 0x7f3354072b50 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.969+0000 7f335259c640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f333406b4d0 0x7f333406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:06.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.970+0000 7f335259c640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f333406b4d0 0x7f333406d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3344009b00 tx=0x7f3344008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:06.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:06.973+0000 7f334affd640 1 -- 172.21.15.94:0/1730113737 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f334c022020 con 0x7f3354072b50 2024-03-06T07:18:07.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.145+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f335406e470 con 0x7f3354072b50 2024-03-06T07:18:07.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.496+0000 7f334affd640 1 -- 172.21.15.94:0/1730113737 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f334c0590e0 con 0x7f3354072b50 2024-03-06T07:18:07.498 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f333406b4d0 msgr2=0x7f333406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f333406b4d0 0x7f333406d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3344009b00 tx=0x7f3344008040 comp rx=0 tx=0).stop 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3354072b50 msgr2=0x7f33541ea3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3354072b50 0x7f33541ea3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f335406f480 tx=0x7f334c007bc0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 shutdown_connections 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f333406b4d0 0x7f333406d990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3354072b50 0x7f33541ea3a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3354144070 0x7f33541e9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 --2- 172.21.15.94:0/1730113737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3340003f30 0x7f3354072610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 >> 172.21.15.94:0/1730113737 conn(0x7f33540a9540 msgr2=0x7f33540c5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:07.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 shutdown_connections 2024-03-06T07:18:07.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.498+0000 7f3358cb5640 1 -- 172.21.15.94:0/1730113737 wait complete. 2024-03-06T07:18:07.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-06T07:18:07.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.620+0000 7fdf139c7640 1 Processor -- start 2024-03-06T07:18:07.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.620+0000 7fdf139c7640 1 -- start start 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.620+0000 7fdf139c7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdf0c1529d0 0x7fdf0c0c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.620+0000 7fdf139c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0c152db0 con 0x7fdf0c072cd0 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.620+0000 7fdf139c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0c1590f0 con 0x7fdf0c0c57e0 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.620+0000 7fdf139c7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf0c0c6dd0 con 0x7fdf0c1529d0 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf11763640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdf0c1529d0 0x7fdf0c0c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf11763640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdf0c1529d0 0x7fdf0c0c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52496/0 (socket says 172.21.15.94:52496) 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf11f64640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdf0c072cd0 0x7fdf0c15faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56650/0 (socket says 172.21.15.94:56650) 2024-03-06T07:18:07.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf11763640 1 -- 172.21.15.94:0/2144780736 learned_addr learned my addr 172.21.15.94:0/2144780736 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:07.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf11763640 1 -- 172.21.15.94:0/2144780736 >> v1:172.21.15.145:6789/0 conn(0x7fdf0c0c57e0 legacy=0x7fdf0c155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:07.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf11763640 1 -- 172.21.15.94:0/2144780736 >> v1:172.21.15.94:6789/0 conn(0x7fdf0c072cd0 legacy=0x7fdf0c15faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:07.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf03fff640 1 -- 172.21.15.94:0/2144780736 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510304161 0 0) 0x7fdf04001490 con 0x7fdf0c072cd0 2024-03-06T07:18:07.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.621+0000 7fdf11763640 1 -- 172.21.15.94:0/2144780736 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c159650 con 0x7fdf0c1529d0 2024-03-06T07:18:07.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf11763640 1 --2- 172.21.15.94:0/2144780736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c1529d0 0x7fdf0c0c52a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009870 tx=0x7fdf0802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4a57752e3f2895 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:07.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf03fff640 1 -- 172.21.15.94:0/2144780736 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0802e070 con 0x7fdf0c1529d0 2024-03-06T07:18:07.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf03fff640 1 -- 172.21.15.94:0/2144780736 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf08037070 con 0x7fdf0c1529d0 2024-03-06T07:18:07.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf03fff640 1 -- 172.21.15.94:0/2144780736 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08031a00 con 0x7fdf0c1529d0 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf139c7640 1 -- 172.21.15.94:0/2144780736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c1529d0 msgr2=0x7fdf0c0c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf139c7640 1 --2- 172.21.15.94:0/2144780736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c1529d0 0x7fdf0c0c52a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009870 tx=0x7fdf0802b880 comp rx=0 tx=0).stop 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf139c7640 1 -- 172.21.15.94:0/2144780736 shutdown_connections 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf139c7640 1 --2- 172.21.15.94:0/2144780736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c1529d0 0x7fdf0c0c52a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf139c7640 1 -- 172.21.15.94:0/2144780736 >> 172.21.15.94:0/2144780736 conn(0x7fdf0c0aa630 msgr2=0x7fdf0c0aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf139c7640 1 -- 172.21.15.94:0/2144780736 shutdown_connections 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.622+0000 7fdf139c7640 1 -- 172.21.15.94:0/2144780736 wait complete. 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.623+0000 7fdf139c7640 1 Processor -- start 2024-03-06T07:18:07.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.623+0000 7fdf139c7640 1 -- start start 2024-03-06T07:18:07.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf139c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c072cd0 0x7fdf0c14bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:07.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf139c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0c0c57e0 0x7fdf0c06d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:07.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf139c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0c1529d0 0x7fdf0c0c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:07.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf139c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c159370 con 0x7fdf0c072cd0 2024-03-06T07:18:07.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf139c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0c6940 con 0x7fdf0c0c57e0 2024-03-06T07:18:07.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf139c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0723b0 con 0x7fdf0c1529d0 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11763640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c072cd0 0x7fdf0c14bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11763640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c072cd0 0x7fdf0c14bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52506/0 (socket says 172.21.15.94:52506) 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11763640 1 -- 172.21.15.94:0/3187952345 learned_addr learned my addr 172.21.15.94:0/3187952345 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11f64640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0c1529d0 0x7fdf0c0c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf10f62640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0c0c57e0 0x7fdf0c06d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11763640 1 -- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0c1529d0 msgr2=0x7fdf0c0c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11763640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0c1529d0 0x7fdf0c0c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11763640 1 -- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0c0c57e0 msgr2=0x7fdf0c06d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.624+0000 7fdf11763640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0c0c57e0 0x7fdf0c06d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.625+0000 7fdf11763640 1 -- 172.21.15.94:0/3187952345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c0c4900 con 0x7fdf0c072cd0 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.625+0000 7fdf11763640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c072cd0 0x7fdf0c14bff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf0802b8c0 tx=0x7fdf080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.625+0000 7fdf01ffb640 1 -- 172.21.15.94:0/3187952345 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08032880 con 0x7fdf0c072cd0 2024-03-06T07:18:07.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.625+0000 7fdf01ffb640 1 -- 172.21.15.94:0/3187952345 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf08032a40 con 0x7fdf0c072cd0 2024-03-06T07:18:07.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.625+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c0c4be0 con 0x7fdf0c072cd0 2024-03-06T07:18:07.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.625+0000 7fdf01ffb640 1 -- 172.21.15.94:0/3187952345 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08032e30 con 0x7fdf0c072cd0 2024-03-06T07:18:07.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.625+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c14d3b0 con 0x7fdf0c072cd0 2024-03-06T07:18:07.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.626+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c158550 con 0x7fdf0c072cd0 2024-03-06T07:18:07.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.627+0000 7fdf01ffb640 1 -- 172.21.15.94:0/3187952345 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf0804c060 con 0x7fdf0c072cd0 2024-03-06T07:18:07.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.627+0000 7fdf01ffb640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdef006b4d0 0x7fdef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:07.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.627+0000 7fdf01ffb640 1 -- 172.21.15.94:0/3187952345 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fdf080abb00 con 0x7fdf0c072cd0 2024-03-06T07:18:07.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.627+0000 7fdf10f62640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdef006b4d0 0x7fdef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:07.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.628+0000 7fdf10f62640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdef006b4d0 0x7fdef006d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdefc006fd0 tx=0x7fdefc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:07.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.631+0000 7fdf01ffb640 1 -- 172.21.15.94:0/3187952345 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf0c158550 con 0x7fdf0c072cd0 2024-03-06T07:18:07.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:07.803+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fdf0c04f330 con 0x7fdf0c072cd0 2024-03-06T07:18:08.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.498+0000 7fdf01ffb640 1 -- 172.21.15.94:0/3187952345 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7fdf08040070 con 0x7fdf0c072cd0 2024-03-06T07:18:08.500 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:18:08.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdef006b4d0 msgr2=0x7fdef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:08.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdef006b4d0 0x7fdef006d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdefc006fd0 tx=0x7fdefc008040 comp rx=0 tx=0).stop 2024-03-06T07:18:08.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c072cd0 msgr2=0x7fdf0c14bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c072cd0 0x7fdf0c14bff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf0802b8c0 tx=0x7fdf080046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 shutdown_connections 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdef006b4d0 0x7fdef006d990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0c1529d0 0x7fdf0c0c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0c0c57e0 0x7fdf0c06d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 --2- 172.21.15.94:0/3187952345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf0c072cd0 0x7fdf0c14bff0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 >> 172.21.15.94:0/3187952345 conn(0x7fdf0c0aa630 msgr2=0x7fdf0c1501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.501+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 shutdown_connections 2024-03-06T07:18:08.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.502+0000 7fdf139c7640 1 -- 172.21.15.94:0/3187952345 wait complete. 2024-03-06T07:18:08.528 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:18:08.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.630+0000 7fead1650640 1 Processor -- start 2024-03-06T07:18:08.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.630+0000 7fead1650640 1 -- start start 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7fead1650640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 0x7feacc153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7fead1650640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7feacc06ee40 0x7feacc1565f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7fead1650640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc153000 con 0x7feacc06c4f0 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7fead1650640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc0abf70 con 0x7feacc06ee40 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7fead1650640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc153280 con 0x7feacc149950 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7feacb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feacc06c4f0 0x7feacc0c69b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41496/0 (socket says 172.21.15.94:41496) 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7feacb7fe640 1 -- 172.21.15.94:0/2832376350 learned_addr learned my addr 172.21.15.94:0/2832376350 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:08.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405848135 0 0) 0x7feac4001110 con 0x7feacc06c4f0 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab4003520 con 0x7feacc06c4f0 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7feacaffd640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 0x7feacc153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.631+0000 7feaca7fc640 1 --2- 172.21.15.94:0/2832376350 >> v2:172.21.15.145:3300/0 conn(0x7feacc06ee40 0x7feacc1565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345500858 0 0) 0x7feab4003520 con 0x7feacc06c4f0 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 msgr2=0x7feacc153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 0x7feacc153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 >> v2:172.21.15.145:3300/0 conn(0x7feacc06ee40 msgr2=0x7feacc1565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 --2- 172.21.15.94:0/2832376350 >> v2:172.21.15.145:3300/0 conn(0x7feacc06ee40 0x7feacc1565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc156b30 con 0x7feacc06c4f0 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7feac4001840 con 0x7feacc06c4f0 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 >> v1:172.21.15.94:6789/0 conn(0x7feacc06c4f0 legacy=0x7feacc0c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feab40039b0 0x7feab4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 0x7feab4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feab4004d40 0x7feab40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0abf70 con 0x7feab4004d40 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc153280 con 0x7feab40039b0 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab400b9c0 con 0x7feacc149950 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feacb7fe640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feab4004d40 0x7feab40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feacaffd640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feab40039b0 0x7feab4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feaca7fc640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 0x7feab4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feacb7fe640 1 -- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 msgr2=0x7feab4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feacb7fe640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc149950 0x7feab4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feacb7fe640 1 -- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feab40039b0 msgr2=0x7feab4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feacb7fe640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feab40039b0 0x7feab4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.632+0000 7feacb7fe640 1 -- 172.21.15.94:0/2832376350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4003520 con 0x7feab4004d40 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.633+0000 7feacb7fe640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feab4004d40 0x7feab40051c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feac400c490 tx=0x7feab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf951034a864a007 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.633+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400c0e0 con 0x7feab4004d40 2024-03-06T07:18:08.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.633+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac4030070 con 0x7feab4004d40 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.633+0000 7feac9ffb640 1 -- 172.21.15.94:0/2832376350 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac40058d0 con 0x7feab4004d40 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.633+0000 7fead1650640 1 -- 172.21.15.94:0/2832376350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feab4004d40 msgr2=0x7feab40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.633+0000 7fead1650640 1 --2- 172.21.15.94:0/2832376350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feab4004d40 0x7feab40051c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feac400c490 tx=0x7feab4005830 comp rx=0 tx=0).stop 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.634+0000 7feacb7fe640 1 -- 172.21.15.94:0/2832376350 reap_dead start 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.634+0000 7fead1650640 1 -- 172.21.15.94:0/2832376350 shutdown_connections 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.634+0000 7fead1650640 1 -- 172.21.15.94:0/2832376350 >> 172.21.15.94:0/2832376350 conn(0x7feacc0c45d0 msgr2=0x7feacc0c49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.634+0000 7fead1650640 1 -- 172.21.15.94:0/2832376350 shutdown_connections 2024-03-06T07:18:08.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.634+0000 7fead1650640 1 -- 172.21.15.94:0/2832376350 wait complete. 2024-03-06T07:18:08.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.635+0000 7fead1650640 1 Processor -- start 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.635+0000 7fead1650640 1 -- start start 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.635+0000 7fead1650640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc155c00 0x7feacc14f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.635+0000 7fead1650640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feacc148f40 0x7feacc1484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.635+0000 7fead1650640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feacc1489e0 0x7feacc1ec470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7fead1650640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc149eb0 con 0x7feacc1489e0 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7fead1650640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc149d30 con 0x7feacc148f40 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7fead1650640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc06e360 con 0x7feacc155c00 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feacc1489e0 0x7feacc1ec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feacc1489e0 0x7feacc1ec470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41150/0 (socket says 172.21.15.94:41150) 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 -- 172.21.15.94:0/2116796682 learned_addr learned my addr 172.21.15.94:0/2116796682 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacaffd640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc155c00 0x7feacc14f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feaca7fc640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feacc148f40 0x7feacc1484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 -- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc155c00 msgr2=0x7feacc14f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc155c00 0x7feacc14f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 -- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feacc148f40 msgr2=0x7feacc1484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feacc148f40 0x7feacc1484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 -- 172.21.15.94:0/2116796682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc14fa40 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feacb7fe640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feacc1489e0 0x7feacc1ec470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feac4001b40 tx=0x7feac4001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feab37fe640 1 -- 172.21.15.94:0/2116796682 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4004cb0 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feab37fe640 1 -- 172.21.15.94:0/2116796682 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac4004e50 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7feab37fe640 1 -- 172.21.15.94:0/2116796682 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4007270 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc14fd20 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.636+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc0c0d40 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.637+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc06ce30 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.638+0000 7feab37fe640 1 -- 172.21.15.94:0/2116796682 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feac4007410 con 0x7feacc1489e0 2024-03-06T07:18:08.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.638+0000 7feab37fe640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feab806b500 0x7feab806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:08.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.638+0000 7feab37fe640 1 -- 172.21.15.94:0/2116796682 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7feac40ab080 con 0x7feacc1489e0 2024-03-06T07:18:08.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.638+0000 7feacaffd640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feab806b500 0x7feab806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:08.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.639+0000 7feacaffd640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feab806b500 0x7feab806d9c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feabc014510 tx=0x7feabc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:08.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.643+0000 7feab37fe640 1 -- 172.21.15.94:0/2116796682 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feacc06ce30 con 0x7feacc1489e0 2024-03-06T07:18:08.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:08.849+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feacc04f330 con 0x7feacc1489e0 2024-03-06T07:18:09.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.523+0000 7feab37fe640 1 -- 172.21.15.94:0/2116796682 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7feacc153000 con 0x7feacc1489e0 2024-03-06T07:18:09.528 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feab806b500 msgr2=0x7feab806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feab806b500 0x7feab806d9c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feabc014510 tx=0x7feabc00b040 comp rx=0 tx=0).stop 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feacc1489e0 msgr2=0x7feacc1ec470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feacc1489e0 0x7feacc1ec470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feac4001b40 tx=0x7feac4001c10 comp rx=0 tx=0).stop 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 shutdown_connections 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feab806b500 0x7feab806d9c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feacc1489e0 0x7feacc1ec470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feacc148f40 0x7feacc1484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 --2- 172.21.15.94:0/2116796682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feacc155c00 0x7feacc14f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 >> 172.21.15.94:0/2116796682 conn(0x7feacc0c45d0 msgr2=0x7feacc0c3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:09.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 shutdown_connections 2024-03-06T07:18:09.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.529+0000 7fead1650640 1 -- 172.21.15.94:0/2116796682 wait complete. 2024-03-06T07:18:09.548 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-06T07:18:09.549 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-06T07:18:09.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.655+0000 7f5ff1cb5640 1 Processor -- start 2024-03-06T07:18:09.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.655+0000 7f5ff1cb5640 1 -- start start 2024-03-06T07:18:09.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5ff1cb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5fec0c22b0 0x7f5fec151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5ff1cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 0x7f5fec153820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5ff1cb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec06e850 con 0x7f5fec0a9830 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5ff1cb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec06d2b0 con 0x7f5fec0c22b0 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5ff1cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec06d590 con 0x7f5fec153390 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5feaffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5fec0a9830 0x7f5fec14ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41510/0 (socket says 172.21.15.94:41510) 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5feaffd640 1 -- 172.21.15.94:0/188741777 learned_addr learned my addr 172.21.15.94:0/188741777 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5febfff640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 0x7f5fec153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5feb7fe640 1 --2- 172.21.15.94:0/188741777 >> v2:172.21.15.145:3300/0 conn(0x7f5fec0c22b0 0x7f5fec151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071023028 0 0) 0x7f5fe0001110 con 0x7f5fec0a9830 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd4003520 con 0x7f5fec0a9830 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2163929577 0 0) 0x7f5fd4003520 con 0x7f5fec0a9830 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 msgr2=0x7f5fec153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 0x7f5fec153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 >> v2:172.21.15.145:3300/0 conn(0x7f5fec0c22b0 msgr2=0x7f5fec151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 --2- 172.21.15.94:0/188741777 >> v2:172.21.15.145:3300/0 conn(0x7f5fec0c22b0 0x7f5fec151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec06d870 con 0x7f5fec0a9830 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f5fe0001840 con 0x7f5fec0a9830 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 >> v1:172.21.15.94:6789/0 conn(0x7f5fec0a9830 legacy=0x7f5fec14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd40039b0 0x7f5fd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 0x7f5fd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec06d2b0 con 0x7f5fd4004800 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec06d590 con 0x7f5fd40039b0 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.656+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd400b9c0 con 0x7f5fec153390 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5feaffd640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5feb7fe640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd40039b0 0x7f5fd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5febfff640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 0x7f5fd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5feaffd640 1 -- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 msgr2=0x7f5fd40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5feaffd640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec153390 0x7f5fd40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5feaffd640 1 -- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd40039b0 msgr2=0x7f5fd4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5feaffd640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd40039b0 0x7f5fd4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.657+0000 7f5feaffd640 1 -- 172.21.15.94:0/188741777 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4003520 con 0x7f5fd4004800 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5feaffd640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fd4004c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000c4c0 tx=0x7f5fd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3062409cc377b950 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe000c0e0 con 0x7f5fd4004800 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe0030070 con 0x7f5fd4004800 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5fea7fc640 1 -- 172.21.15.94:0/188741777 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe00058d0 con 0x7f5fd4004800 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/188741777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 msgr2=0x7f5fd4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5ff1cb5640 1 --2- 172.21.15.94:0/188741777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fd4004c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000c4c0 tx=0x7f5fd4005830 comp rx=0 tx=0).stop 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5febfff640 1 -- 172.21.15.94:0/188741777 reap_dead start 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/188741777 shutdown_connections 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/188741777 >> 172.21.15.94:0/188741777 conn(0x7f5fec0c65b0 msgr2=0x7f5fec0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/188741777 shutdown_connections 2024-03-06T07:18:09.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.658+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/188741777 wait complete. 2024-03-06T07:18:09.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.659+0000 7f5ff1cb5640 1 Processor -- start 2024-03-06T07:18:09.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5ff1cb5640 1 -- start start 2024-03-06T07:18:09.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5ff1cb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fec1ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5ff1cb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fec0ab790 0x7f5fec0bfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5ff1cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec1537a0 0x7f5fec1ea550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5ff1cb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec14f7e0 con 0x7f5fd4004800 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5ff1cb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec14f660 con 0x7f5fec0ab790 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5ff1cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0c2690 con 0x7f5fec1537a0 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5feb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fec1ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5feb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fec1ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41162/0 (socket says 172.21.15.94:41162) 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5feb7fe640 1 -- 172.21.15.94:0/488299221 learned_addr learned my addr 172.21.15.94:0/488299221 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5feaffd640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fec0ab790 0x7f5fec0bfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5febfff640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec1537a0 0x7f5fec1ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5feb7fe640 1 -- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec1537a0 msgr2=0x7f5fec1ea550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.660+0000 7f5feb7fe640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec1537a0 0x7f5fec1ea550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5feb7fe640 1 -- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fec0ab790 msgr2=0x7f5fec0bfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:09.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5feb7fe640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fec0ab790 0x7f5fec0bfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5feb7fe640 1 -- 172.21.15.94:0/488299221 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec155d50 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5feaffd640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fec0ab790 0x7f5fec0bfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5feb7fe640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fec1ee6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc011910 tx=0x7f5fdc011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5fd3fff640 1 -- 172.21.15.94:0/488299221 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc00faf0 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5fd3fff640 1 -- 172.21.15.94:0/488299221 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc00fc90 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec1edbe0 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5fd3fff640 1 -- 172.21.15.94:0/488299221 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc014440 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.661+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec1ee1a0 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.662+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec04f330 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.662+0000 7f5fd3fff640 1 -- 172.21.15.94:0/488299221 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fdc012040 con 0x7f5fd4004800 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.663+0000 7f5fd3fff640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fd806b4d0 0x7f5fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:09.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.663+0000 7f5fd3fff640 1 -- 172.21.15.94:0/488299221 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fdc08a790 con 0x7f5fd4004800 2024-03-06T07:18:09.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.663+0000 7f5feaffd640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fd806b4d0 0x7f5fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:09.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.663+0000 7f5feaffd640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fd806b4d0 0x7f5fd806d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0034000 tx=0x7f5fe0009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:09.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.667+0000 7f5fd3fff640 1 -- 172.21.15.94:0/488299221 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fdc019030 con 0x7f5fd4004800 2024-03-06T07:18:09.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:09.847+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f5fec0c0110 con 0x7f5fd4004800 2024-03-06T07:18:10.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.519+0000 7f5fd3fff640 1 -- 172.21.15.94:0/488299221 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f5fdc0596a0 con 0x7f5fd4004800 2024-03-06T07:18:10.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fd806b4d0 msgr2=0x7f5fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fd806b4d0 0x7f5fd806d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0034000 tx=0x7f5fe0009030 comp rx=0 tx=0).stop 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 msgr2=0x7f5fec1ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fec1ee6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc011910 tx=0x7f5fdc011dd0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 shutdown_connections 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fd806b4d0 0x7f5fd806d990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fec1537a0 0x7f5fec1ea550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fec0ab790 0x7f5fec0bfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 --2- 172.21.15.94:0/488299221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd4004800 0x7f5fec1ee6c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.521+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 >> 172.21.15.94:0/488299221 conn(0x7f5fec0c65b0 msgr2=0x7f5fec14d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.522+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 shutdown_connections 2024-03-06T07:18:10.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.522+0000 7f5ff1cb5640 1 -- 172.21.15.94:0/488299221 wait complete. 2024-03-06T07:18:10.536 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:18:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.643+0000 7f72f4d0c640 1 Processor -- start 2024-03-06T07:18:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.643+0000 7f72f4d0c640 1 -- start start 2024-03-06T07:18:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72f4d0c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f72f006dfc0 0x7f72f0165460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72f4d0c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f00c4020 con 0x7f72f006c1f0 2024-03-06T07:18:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72f4d0c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f0165fd0 con 0x7f72f00c3c40 2024-03-06T07:18:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72f4d0c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f0163aa0 con 0x7f72f006dfc0 2024-03-06T07:18:10.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72edd9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f72f006c1f0 0x7f72f0163390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41522/0 (socket says 172.21.15.94:41522) 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72eed9d640 1 --2- 172.21.15.94:0/745974217 >> v2:172.21.15.94:3300/0 conn(0x7f72f006dfc0 0x7f72f0165460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72edd9b640 1 -- 172.21.15.94:0/745974217 learned_addr learned my addr 172.21.15.94:0/745974217 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.644+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239259541 0 0) 0x7f72f00c4020 con 0x7f72f006c1f0 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.741+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d8003500 con 0x7f72f006c1f0 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.741+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206270735 0 0) 0x7f72f0165fd0 con 0x7f72f00c3c40 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.741+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f00c4020 con 0x7f72f00c3c40 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.741+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592632167 0 0) 0x7f72d8003500 con 0x7f72f006c1f0 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.741+0000 7f72eed9d640 1 -- 172.21.15.94:0/745974217 >> v1:172.21.15.145:6789/0 conn(0x7f72f00c3c40 legacy=0x7f72f01628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.741+0000 7f72eed9d640 1 -- 172.21.15.94:0/745974217 >> v1:172.21.15.94:6789/0 conn(0x7f72f006c1f0 legacy=0x7f72f0163390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:10.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.741+0000 7f72eed9d640 1 -- 172.21.15.94:0/745974217 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f00c5e50 con 0x7f72f006dfc0 2024-03-06T07:18:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.742+0000 7f72eed9d640 1 --2- 172.21.15.94:0/745974217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f006dfc0 0x7f72f0165460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72e8007670 tx=0x7f72e802bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c81ef2fbaf512c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:10.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.742+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4001670 con 0x7f72f006dfc0 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.742+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e80092e0 con 0x7f72f006dfc0 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.742+0000 7f72ed59a640 1 -- 172.21.15.94:0/745974217 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8003990 con 0x7f72f006dfc0 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.743+0000 7f72f4d0c640 1 -- 172.21.15.94:0/745974217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f006dfc0 msgr2=0x7f72f0165460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.743+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/745974217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f006dfc0 0x7f72f0165460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72e8007670 tx=0x7f72e802bce0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.743+0000 7f72f4d0c640 1 -- 172.21.15.94:0/745974217 shutdown_connections 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.743+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/745974217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f006dfc0 0x7f72f0165460 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.743+0000 7f72f4d0c640 1 -- 172.21.15.94:0/745974217 >> 172.21.15.94:0/745974217 conn(0x7f72f00c6710 msgr2=0x7f72f00c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:10.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.743+0000 7f72f4d0c640 1 -- 172.21.15.94:0/745974217 shutdown_connections 2024-03-06T07:18:10.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.743+0000 7f72f4d0c640 1 -- 172.21.15.94:0/745974217 wait complete. 2024-03-06T07:18:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.744+0000 7f72f4d0c640 1 Processor -- start 2024-03-06T07:18:10.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.744+0000 7f72f4d0c640 1 -- start start 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72f4d0c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72f006c1f0 0x7f72f00c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72f4d0c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72f006dfc0 0x7f72f0071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72f4d0c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f00c3c40 0x7f72f0072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72f4d0c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f01662b0 con 0x7f72f00c3c40 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72f4d0c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f01659a0 con 0x7f72f006c1f0 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72f4d0c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f0165b20 con 0x7f72f006dfc0 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f00c3c40 0x7f72f0072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f00c3c40 0x7f72f0072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41190/0 (socket says 172.21.15.94:41190) 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 -- 172.21.15.94:0/3645615864 learned_addr learned my addr 172.21.15.94:0/3645615864 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:10.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72edd9b640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72f006dfc0 0x7f72f0071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72ee59c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72f006c1f0 0x7f72f00c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 -- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72f006dfc0 msgr2=0x7f72f0071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72f006dfc0 0x7f72f0071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 -- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72f006c1f0 msgr2=0x7f72f00c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72f006c1f0 0x7f72f00c2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72eed9d640 1 -- 172.21.15.94:0/3645615864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f00a8540 con 0x7f72f00c3c40 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.745+0000 7f72edd9b640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72f006dfc0 0x7f72f0071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.746+0000 7f72eed9d640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f00c3c40 0x7f72f0072050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009e90 tx=0x7f72e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.746+0000 7f72deffd640 1 -- 172.21.15.94:0/3645615864 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8041030 con 0x7f72f00c3c40 2024-03-06T07:18:10.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.746+0000 7f72deffd640 1 -- 172.21.15.94:0/3645615864 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e8007990 con 0x7f72f00c3c40 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.746+0000 7f72deffd640 1 -- 172.21.15.94:0/3645615864 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8007b80 con 0x7f72f00c3c40 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.746+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f01f7650 con 0x7f72f00c3c40 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.746+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f00c3160 con 0x7f72f00c3c40 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.747+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f01f7440 con 0x7f72f00c3c40 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.747+0000 7f72deffd640 1 -- 172.21.15.94:0/3645615864 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e8044070 con 0x7f72f00c3c40 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.748+0000 7f72deffd640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72d006b4d0 0x7f72d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.748+0000 7f72deffd640 1 -- 172.21.15.94:0/3645615864 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f72e80ab8a0 con 0x7f72f00c3c40 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.748+0000 7f72ee59c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72d006b4d0 0x7f72d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:10.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.748+0000 7f72ee59c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72d006b4d0 0x7f72d006d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72e00011e0 tx=0x7f72e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:10.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.752+0000 7f72deffd640 1 -- 172.21.15.94:0/3645615864 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72e804b070 con 0x7f72f00c3c40 2024-03-06T07:18:10.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:10.930+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f72f004f330 con 0x7f72f00c3c40 2024-03-06T07:18:12.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.540+0000 7f72deffd640 1 -- 172.21.15.94:0/3645615864 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f72e807a680 con 0x7f72f00c3c40 2024-03-06T07:18:12.542 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data_ec' created 2024-03-06T07:18:12.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72d006b4d0 msgr2=0x7f72d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:12.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72d006b4d0 0x7f72d006d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72e00011e0 tx=0x7f72e0009040 comp rx=0 tx=0).stop 2024-03-06T07:18:12.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f00c3c40 msgr2=0x7f72f0072050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:12.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f00c3c40 0x7f72f0072050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009e90 tx=0x7f72e8004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:12.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 shutdown_connections 2024-03-06T07:18:12.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72d006b4d0 0x7f72d006d990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f00c3c40 0x7f72f0072050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72f006dfc0 0x7f72f0071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 --2- 172.21.15.94:0/3645615864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72f006c1f0 0x7f72f00c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 >> 172.21.15.94:0/3645615864 conn(0x7f72f00c6710 msgr2=0x7f72f00a6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:12.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 shutdown_connections 2024-03-06T07:18:12.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.544+0000 7f72f4d0c640 1 -- 172.21.15.94:0/3645615864 wait complete. 2024-03-06T07:18:12.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-06T07:18:12.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.665+0000 7f94eceb1640 1 Processor -- start 2024-03-06T07:18:12.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.665+0000 7f94eceb1640 1 -- start start 2024-03-06T07:18:12.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.666+0000 7f94eceb1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f94e80c2200 0x7f94e80c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:12.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.666+0000 7f94eceb1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f94e80a6cc0 0x7f94e8155460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:12.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.666+0000 7f94eceb1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e81559a0 0x7f94e80c0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.666+0000 7f94eceb1640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e81596c0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.666+0000 7f94eceb1640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e8158ab0 con 0x7f94e80c2200 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.666+0000 7f94eceb1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8158d90 con 0x7f94e81559a0 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.666+0000 7f94e5d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f94e80a6cc0 0x7f94e8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f94e80a6cc0 0x7f94e8155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41206/0 (socket says 172.21.15.94:41206) 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 -- 172.21.15.94:0/4187531478 learned_addr learned my addr 172.21.15.94:0/4187531478 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e6d9d640 1 --2- 172.21.15.94:0/4187531478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e81559a0 0x7f94e80c0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e659c640 1 --2- 172.21.15.94:0/4187531478 >> v2:172.21.15.145:3300/0 conn(0x7f94e80c2200 0x7f94e80c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 -- 172.21.15.94:0/4187531478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e81559a0 msgr2=0x7f94e80c0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 --2- 172.21.15.94:0/4187531478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e81559a0 0x7f94e80c0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 -- 172.21.15.94:0/4187531478 >> v2:172.21.15.145:3300/0 conn(0x7f94e80c2200 msgr2=0x7f94e80c25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 --2- 172.21.15.94:0/4187531478 >> v2:172.21.15.145:3300/0 conn(0x7f94e80c2200 0x7f94e80c25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 -- 172.21.15.94:0/4187531478 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8159070 con 0x7f94e80a6cc0 2024-03-06T07:18:12.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.667+0000 7f94e5d9b640 1 --2- 172.21.15.94:0/4187531478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e8155460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94e806d190 tx=0x7f94dc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fa523c7e737061 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:12.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94e559a640 1 -- 172.21.15.94:0/4187531478 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc02c9e0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94e559a640 1 -- 172.21.15.94:0/4187531478 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc02cb80 con 0x7f94e80a6cc0 2024-03-06T07:18:12.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94e559a640 1 -- 172.21.15.94:0/4187531478 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0343f0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 -- 172.21.15.94:0/4187531478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 msgr2=0x7f94e8155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 --2- 172.21.15.94:0/4187531478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e8155460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94e806d190 tx=0x7f94dc02be40 comp rx=0 tx=0).stop 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 -- 172.21.15.94:0/4187531478 shutdown_connections 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 --2- 172.21.15.94:0/4187531478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e81559a0 0x7f94e80c0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 --2- 172.21.15.94:0/4187531478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e8155460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 --2- 172.21.15.94:0/4187531478 >> v2:172.21.15.145:3300/0 conn(0x7f94e80c2200 0x7f94e80c25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 -- 172.21.15.94:0/4187531478 >> 172.21.15.94:0/4187531478 conn(0x7f94e80c4300 msgr2=0x7f94e80c46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 -- 172.21.15.94:0/4187531478 shutdown_connections 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.668+0000 7f94eceb1640 1 -- 172.21.15.94:0/4187531478 wait complete. 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 Processor -- start 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 -- start start 2024-03-06T07:18:12.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e81eb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e80c2200 0x7f94e80a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94e81559a0 0x7f94e8150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8073580 con 0x7f94e80a6cc0 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e806be10 con 0x7f94e81559a0 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94eceb1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8158740 con 0x7f94e80c2200 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e81eb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e81eb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41212/0 (socket says 172.21.15.94:41212) 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94e659c640 1 -- 172.21.15.94:0/2892223339 learned_addr learned my addr 172.21.15.94:0/2892223339 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94e5d9b640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e80c2200 0x7f94e80a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.670+0000 7f94e6d9d640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94e81559a0 0x7f94e8150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94e659c640 1 -- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e80c2200 msgr2=0x7f94e80a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94e659c640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e80c2200 0x7f94e80a8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94e659c640 1 -- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94e81559a0 msgr2=0x7f94e8150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:12.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94e659c640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94e81559a0 0x7f94e8150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94e659c640 1 -- 172.21.15.94:0/2892223339 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e814c2b0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94e659c640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e81eb6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94e000e8c0 tx=0x7f94e000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94d6ffd640 1 -- 172.21.15.94:0/2892223339 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0017070 con 0x7f94e80a6cc0 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94d6ffd640 1 -- 172.21.15.94:0/2892223339 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e000c8e0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e814bfd0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94d6ffd640 1 -- 172.21.15.94:0/2892223339 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e000ccb0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.671+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e81eaee0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.672+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e804f330 con 0x7f94e80a6cc0 2024-03-06T07:18:12.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.672+0000 7f94d6ffd640 1 -- 172.21.15.94:0/2892223339 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94e001a070 con 0x7f94e80a6cc0 2024-03-06T07:18:12.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.673+0000 7f94d6ffd640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f94c806b4d0 0x7f94c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:12.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.673+0000 7f94d6ffd640 1 -- 172.21.15.94:0/2892223339 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f94e008a6d0 con 0x7f94e80a6cc0 2024-03-06T07:18:12.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.673+0000 7f94e5d9b640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f94c806b4d0 0x7f94c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:12.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.673+0000 7f94e5d9b640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f94c806b4d0 0x7f94c806d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f94dc035000 tx=0x7f94dc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:12.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.677+0000 7f94d6ffd640 1 -- 172.21.15.94:0/2892223339 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94e0058c70 con 0x7f94e80a6cc0 2024-03-06T07:18:12.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:12.848+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f94e80c5700 con 0x7f94e80a6cc0 2024-03-06T07:18:13.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:18:13.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:18:13.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.544+0000 7f94d6ffd640 1 -- 172.21.15.94:0/2892223339 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f94e005cd10 con 0x7f94e80a6cc0 2024-03-06T07:18:13.546 INFO:teuthology.orchestra.run.smithi094.stderr:set pool 4 allow_ec_overwrites to true 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.548+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f94c806b4d0 msgr2=0x7f94c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.548+0000 7f94eceb1640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f94c806b4d0 0x7f94c806d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f94dc035000 tx=0x7f94dc030000 comp rx=0 tx=0).stop 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.548+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 msgr2=0x7f94e81eb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.548+0000 7f94eceb1640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e81eb6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f94e000e8c0 tx=0x7f94e000ed80 comp rx=0 tx=0).stop 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.549+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 shutdown_connections 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.549+0000 7f94eceb1640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f94c806b4d0 0x7f94c806d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.549+0000 7f94eceb1640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94e81559a0 0x7f94e8150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.549+0000 7f94eceb1640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94e80c2200 0x7f94e80a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.549+0000 7f94eceb1640 1 --2- 172.21.15.94:0/2892223339 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94e80a6cc0 0x7f94e81eb6e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.549+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 >> 172.21.15.94:0/2892223339 conn(0x7f94e80c4300 msgr2=0x7f94e806bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:13.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.549+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 shutdown_connections 2024-03-06T07:18:13.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.550+0000 7f94eceb1640 1 -- 172.21.15.94:0/2892223339 wait complete. 2024-03-06T07:18:13.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-06T07:18:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.751+0000 7fea7b4c2640 1 Processor -- start 2024-03-06T07:18:13.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.752+0000 7fea7b4c2640 1 -- start start 2024-03-06T07:18:13.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.752+0000 7fea7b4c2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fea740c26c0 0x7fea74155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.752+0000 7fea7b4c2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fea740c0470 0x7fea74152350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.752+0000 7fea7b4c2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740670c0 con 0x7fea740c2aa0 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.752+0000 7fea7b4c2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea7406b740 con 0x7fea740c0470 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.752+0000 7fea7b4c2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea7406b0f0 con 0x7fea740c26c0 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea79a5f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fea740c0470 0x7fea74152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea79a5f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fea740c0470 0x7fea74152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41222/0 (socket says 172.21.15.94:41222) 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea79a5f640 1 -- 172.21.15.94:0/2205830268 learned_addr learned my addr 172.21.15.94:0/2205830268 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea7925e640 1 --2- 172.21.15.94:0/2205830268 >> v2:172.21.15.145:3300/0 conn(0x7fea740c26c0 0x7fea74155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea79a5f640 1 -- 172.21.15.94:0/2205830268 >> v2:172.21.15.145:3300/0 conn(0x7fea740c26c0 msgr2=0x7fea74155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea79a5f640 1 --2- 172.21.15.94:0/2205830268 >> v2:172.21.15.145:3300/0 conn(0x7fea740c26c0 0x7fea74155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea79a5f640 1 -- 172.21.15.94:0/2205830268 >> v1:172.21.15.145:6789/0 conn(0x7fea740c2aa0 legacy=0x7fea7406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:13.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.753+0000 7fea79a5f640 1 -- 172.21.15.94:0/2205830268 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7406b3d0 con 0x7fea740c0470 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea79a5f640 1 --2- 172.21.15.94:0/2205830268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c0470 0x7fea74152350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea74067560 tx=0x7fea6c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ae0b9f00427145 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea6bfff640 1 -- 172.21.15.94:0/2205830268 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c02c9f0 con 0x7fea740c0470 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea6bfff640 1 -- 172.21.15.94:0/2205830268 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6c02cb90 con 0x7fea740c0470 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea6bfff640 1 -- 172.21.15.94:0/2205830268 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0343f0 con 0x7fea740c0470 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 -- 172.21.15.94:0/2205830268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c0470 msgr2=0x7fea74152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/2205830268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c0470 0x7fea74152350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea74067560 tx=0x7fea6c02be50 comp rx=0 tx=0).stop 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 -- 172.21.15.94:0/2205830268 shutdown_connections 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/2205830268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c0470 0x7fea74152350 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/2205830268 >> v2:172.21.15.145:3300/0 conn(0x7fea740c26c0 0x7fea74155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 -- 172.21.15.94:0/2205830268 >> 172.21.15.94:0/2205830268 conn(0x7fea740c3b40 msgr2=0x7fea740c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 -- 172.21.15.94:0/2205830268 shutdown_connections 2024-03-06T07:18:13.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.754+0000 7fea7b4c2640 1 -- 172.21.15.94:0/2205830268 wait complete. 2024-03-06T07:18:13.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.755+0000 7fea7b4c2640 1 Processor -- start 2024-03-06T07:18:13.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.755+0000 7fea7b4c2640 1 -- start start 2024-03-06T07:18:13.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea7b4c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea740c0470 0x7fea7414a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:13.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea7b4c2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c26c0 0x7fea741397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea7b4c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea740c2aa0 0x7fea74139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea7b4c2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea740c20b0 con 0x7fea740c26c0 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea7b4c2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea740c1f30 con 0x7fea740c2aa0 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea7b4c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea74066d00 con 0x7fea740c0470 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c26c0 0x7fea741397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c26c0 0x7fea741397f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41228/0 (socket says 172.21.15.94:41228) 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 -- 172.21.15.94:0/151310063 learned_addr learned my addr 172.21.15.94:0/151310063 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea7925e640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea740c0470 0x7fea7414a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea79a5f640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea740c2aa0 0x7fea74139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 -- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea740c0470 msgr2=0x7fea7414a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea740c0470 0x7fea7414a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 -- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea740c2aa0 msgr2=0x7fea74139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea740c2aa0 0x7fea74139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:13.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.756+0000 7fea78a5d640 1 -- 172.21.15.94:0/151310063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea741ee940 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.757+0000 7fea78a5d640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c26c0 0x7fea741397f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea6400c3c0 tx=0x7fea6400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.757+0000 7fea69ffb640 1 -- 172.21.15.94:0/151310063 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea640052d0 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.757+0000 7fea69ffb640 1 -- 172.21.15.94:0/151310063 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea64005490 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.757+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea741eec20 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.757+0000 7fea69ffb640 1 -- 172.21.15.94:0/151310063 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64007260 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.757+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea740ab0d0 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.758+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7404f330 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.758+0000 7fea69ffb640 1 -- 172.21.15.94:0/151310063 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea64007400 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.758+0000 7fea69ffb640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fea5806b500 0x7fea5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.758+0000 7fea69ffb640 1 -- 172.21.15.94:0/151310063 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fea640898a0 con 0x7fea740c26c0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.759+0000 7fea7925e640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fea5806b500 0x7fea5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:13.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.759+0000 7fea7925e640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fea5806b500 0x7fea5806d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea70009b40 tx=0x7fea70009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:13.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.763+0000 7fea69ffb640 1 -- 172.21.15.94:0/151310063 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea64057e40 con 0x7fea740c26c0 2024-03-06T07:18:13.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:13.977+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fea74068fb0 con 0x7fea740c26c0 2024-03-06T07:18:14.075 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:18:14.074+0000 7f5721508640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:18:14.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.545+0000 7fea69ffb640 1 -- 172.21.15.94:0/151310063 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7fea6405bee0 con 0x7fea740c26c0 2024-03-06T07:18:14.547 INFO:teuthology.orchestra.run.smithi094.stderr:added data pool 4 to fsmap 2024-03-06T07:18:14.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fea5806b500 msgr2=0x7fea5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fea5806b500 0x7fea5806d9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea70009b40 tx=0x7fea70009190 comp rx=0 tx=0).stop 2024-03-06T07:18:14.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c26c0 msgr2=0x7fea741397f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c26c0 0x7fea741397f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea6400c3c0 tx=0x7fea6400c880 comp rx=0 tx=0).stop 2024-03-06T07:18:14.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 shutdown_connections 2024-03-06T07:18:14.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fea5806b500 0x7fea5806d9c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea740c2aa0 0x7fea74139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fea740c26c0 0x7fea741397f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 --2- 172.21.15.94:0/151310063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea740c0470 0x7fea7414a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 >> 172.21.15.94:0/151310063 conn(0x7fea740c3b40 msgr2=0x7fea740c0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:14.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 shutdown_connections 2024-03-06T07:18:14.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.548+0000 7fea7b4c2640 1 -- 172.21.15.94:0/151310063 wait complete. 2024-03-06T07:18:14.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:14.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.663+0000 7fc031637640 1 Processor -- start 2024-03-06T07:18:14.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.663+0000 7fc031637640 1 -- start start 2024-03-06T07:18:14.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.663+0000 7fc031637640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc02c0c88c0 0x7fc02c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:14.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.663+0000 7fc031637640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c158b60 0x7fc02c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:14.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.663+0000 7fc031637640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0c73d0 con 0x7fc02c14f360 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.663+0000 7fc031637640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02c0c8dd0 con 0x7fc02c0c88c0 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.663+0000 7fc031637640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0be2e0 con 0x7fc02c158b60 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc0227fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc02c0c88c0 0x7fc02c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc0227fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc02c0c88c0 0x7fc02c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41232/0 (socket says 172.21.15.94:41232) 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc0227fc640 1 -- 172.21.15.94:0/1960428642 learned_addr learned my addr 172.21.15.94:0/1960428642 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc02b7fe640 1 --2- 172.21.15.94:0/1960428642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c158b60 0x7fc02c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc0227fc640 1 -- 172.21.15.94:0/1960428642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c158b60 msgr2=0x7fc02c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc0227fc640 1 --2- 172.21.15.94:0/1960428642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c158b60 0x7fc02c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc0227fc640 1 -- 172.21.15.94:0/1960428642 >> v1:172.21.15.145:6789/0 conn(0x7fc02c14f360 legacy=0x7fc02c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc0227fc640 1 -- 172.21.15.94:0/1960428642 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0c7670 con 0x7fc02c0c88c0 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.664+0000 7fc02a7fc640 1 -- 172.21.15.94:0/1960428642 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590398127 0 0) 0x7fc02c0c73d0 con 0x7fc02c14f360 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc0227fc640 1 --2- 172.21.15.94:0/1960428642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c0c88c0 0x7fc02c155c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc014009870 tx=0x7fc01402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4e2f774ee5a9df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:14.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc02a7fc640 1 -- 172.21.15.94:0/1960428642 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014035020 con 0x7fc02c0c88c0 2024-03-06T07:18:14.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc02a7fc640 1 -- 172.21.15.94:0/1960428642 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc014031640 con 0x7fc02c0c88c0 2024-03-06T07:18:14.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc02a7fc640 1 -- 172.21.15.94:0/1960428642 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01402d040 con 0x7fc02c0c88c0 2024-03-06T07:18:14.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 -- 172.21.15.94:0/1960428642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c0c88c0 msgr2=0x7fc02c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 --2- 172.21.15.94:0/1960428642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c0c88c0 0x7fc02c155c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc014009870 tx=0x7fc01402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 -- 172.21.15.94:0/1960428642 shutdown_connections 2024-03-06T07:18:14.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 --2- 172.21.15.94:0/1960428642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c158b60 0x7fc02c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 --2- 172.21.15.94:0/1960428642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c0c88c0 0x7fc02c155c10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 -- 172.21.15.94:0/1960428642 >> 172.21.15.94:0/1960428642 conn(0x7fc02c0ac570 msgr2=0x7fc02c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 -- 172.21.15.94:0/1960428642 shutdown_connections 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.665+0000 7fc031637640 1 -- 172.21.15.94:0/1960428642 wait complete. 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.666+0000 7fc031637640 1 Processor -- start 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.666+0000 7fc031637640 1 -- start start 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc031637640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c0c88c0 0x7fc02c0c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc031637640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc02c14f360 0x7fc02c1ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc031637640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c158b60 0x7fc02c14b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc031637640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0c5a50 con 0x7fc02c158b60 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc031637640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0c7c30 con 0x7fc02c14f360 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc031637640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0c5bd0 con 0x7fc02c0c88c0 2024-03-06T07:18:14.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c158b60 0x7fc02c14b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c158b60 0x7fc02c14b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41246/0 (socket says 172.21.15.94:41246) 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02b7fe640 1 -- 172.21.15.94:0/2887724744 learned_addr learned my addr 172.21.15.94:0/2887724744 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc0227fc640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc02c14f360 0x7fc02c1ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02affd640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c0c88c0 0x7fc02c0c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02b7fe640 1 -- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c0c88c0 msgr2=0x7fc02c0c11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02b7fe640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c0c88c0 0x7fc02c0c11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02b7fe640 1 -- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc02c14f360 msgr2=0x7fc02c1ee2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.667+0000 7fc02b7fe640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc02c14f360 0x7fc02c1ee2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.668+0000 7fc02b7fe640 1 -- 172.21.15.94:0/2887724744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c1eedb0 con 0x7fc02c158b60 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.668+0000 7fc02b7fe640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c158b60 0x7fc02c14b930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc02400efc0 tx=0x7fc02400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.668+0000 7fc023fff640 1 -- 172.21.15.94:0/2887724744 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024018020 con 0x7fc02c158b60 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.668+0000 7fc023fff640 1 -- 172.21.15.94:0/2887724744 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024014480 con 0x7fc02c158b60 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.668+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c1ec140 con 0x7fc02c158b60 2024-03-06T07:18:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.668+0000 7fc023fff640 1 -- 172.21.15.94:0/2887724744 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024014870 con 0x7fc02c158b60 2024-03-06T07:18:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.668+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c0c0be0 con 0x7fc02c158b60 2024-03-06T07:18:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.669+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c0c0e70 con 0x7fc02c158b60 2024-03-06T07:18:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.669+0000 7fc023fff640 1 -- 172.21.15.94:0/2887724744 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc024005d10 con 0x7fc02c158b60 2024-03-06T07:18:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.670+0000 7fc023fff640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc00c06b4d0 0x7fc00c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:14.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.670+0000 7fc023fff640 1 -- 172.21.15.94:0/2887724744 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc02408b310 con 0x7fc02c158b60 2024-03-06T07:18:14.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.670+0000 7fc02affd640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc00c06b4d0 0x7fc00c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:14.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.670+0000 7fc02affd640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc00c06b4d0 0x7fc00c06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000f60 tx=0x7fc01c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.674+0000 7fc023fff640 1 -- 172.21.15.94:0/2887724744 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc024022030 con 0x7fc02c158b60 2024-03-06T07:18:14.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.888+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc02c1ebf30 con 0x7fc02c158b60 2024-03-06T07:18:14.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.888+0000 7fc023fff640 1 -- 172.21.15.94:0/2887724744 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4981 (secure 0 0 0) 0x7fc024010040 con 0x7fc02c158b60 2024-03-06T07:18:14.891 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:14.891 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:13.978510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 7 2024-03-06T07:18:14.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc00c06b4d0 msgr2=0x7fc00c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc00c06b4d0 0x7fc00c06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000f60 tx=0x7fc01c009040 comp rx=0 tx=0).stop 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c158b60 msgr2=0x7fc02c14b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c158b60 0x7fc02c14b930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc02400efc0 tx=0x7fc02400c490 comp rx=0 tx=0).stop 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 shutdown_connections 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc00c06b4d0 0x7fc00c06d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc02c158b60 0x7fc02c14b930 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc02c14f360 0x7fc02c1ee2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 --2- 172.21.15.94:0/2887724744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc02c0c88c0 0x7fc02c0c11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.890+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 >> 172.21.15.94:0/2887724744 conn(0x7fc02c0ac570 msgr2=0x7fc02c0c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.891+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 shutdown_connections 2024-03-06T07:18:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:14.891+0000 7fc031637640 1 -- 172.21.15.94:0/2887724744 wait complete. 2024-03-06T07:18:14.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.002+0000 7ff4364a4640 1 Processor -- start 2024-03-06T07:18:15.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff4364a4640 1 -- start start 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff4364a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff4301506b0 0x7ff430153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff4364a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4300acc00 0x7ff4300c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff4364a4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff430070aa0 con 0x7ff430156ea0 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff4364a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff430156450 con 0x7ff4300acc00 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff4364a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff430156a10 con 0x7ff4301506b0 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff434a41640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4300acc00 0x7ff4300c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff42ffff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff4301506b0 0x7ff430153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff434a41640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4300acc00 0x7ff4300c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41258/0 (socket says 172.21.15.94:41258) 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.003+0000 7ff434a41640 1 -- 172.21.15.94:0/1385130920 learned_addr learned my addr 172.21.15.94:0/1385130920 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.004+0000 7ff434a41640 1 -- 172.21.15.94:0/1385130920 >> v2:172.21.15.145:3300/0 conn(0x7ff4301506b0 msgr2=0x7ff430153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.004+0000 7ff434a41640 1 --2- 172.21.15.94:0/1385130920 >> v2:172.21.15.145:3300/0 conn(0x7ff4301506b0 0x7ff430153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.004+0000 7ff434a41640 1 -- 172.21.15.94:0/1385130920 >> v1:172.21.15.145:6789/0 conn(0x7ff430156ea0 legacy=0x7ff4300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.004+0000 7ff434a41640 1 -- 172.21.15.94:0/1385130920 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430070d80 con 0x7ff4300acc00 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.004+0000 7ff434a41640 1 --2- 172.21.15.94:0/1385130920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4300acc00 0x7ff4300c54a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff4300c97e0 tx=0x7ff42802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f54c9a9f1b5b186 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff42effd640 1 -- 172.21.15.94:0/1385130920 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42802cb20 con 0x7ff4300acc00 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff42effd640 1 -- 172.21.15.94:0/1385130920 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42802ccc0 con 0x7ff4300acc00 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff42effd640 1 -- 172.21.15.94:0/1385130920 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428034610 con 0x7ff4300acc00 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 -- 172.21.15.94:0/1385130920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4300acc00 msgr2=0x7ff4300c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1385130920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4300acc00 0x7ff4300c54a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff4300c97e0 tx=0x7ff42802bf80 comp rx=0 tx=0).stop 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 -- 172.21.15.94:0/1385130920 shutdown_connections 2024-03-06T07:18:15.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1385130920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4300acc00 0x7ff4300c54a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1385130920 >> v2:172.21.15.145:3300/0 conn(0x7ff4301506b0 0x7ff430153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 -- 172.21.15.94:0/1385130920 >> 172.21.15.94:0/1385130920 conn(0x7ff43006b1a0 msgr2=0x7ff43006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 -- 172.21.15.94:0/1385130920 shutdown_connections 2024-03-06T07:18:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.005+0000 7ff4364a4640 1 -- 172.21.15.94:0/1385130920 wait complete. 2024-03-06T07:18:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.006+0000 7ff4364a4640 1 Processor -- start 2024-03-06T07:18:15.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.006+0000 7ff4364a4640 1 -- start start 2024-03-06T07:18:15.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff4364a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4300acc00 0x7ff4301e89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff4364a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4301506b0 0x7ff4300c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff4364a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff430156ea0 0x7ff430151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff4364a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300ad1d0 con 0x7ff430156ea0 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff4364a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300ad050 con 0x7ff4300acc00 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff4364a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300ad350 con 0x7ff4301506b0 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff430156ea0 0x7ff430151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff430156ea0 0x7ff430151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41264/0 (socket says 172.21.15.94:41264) 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 -- 172.21.15.94:0/1717268669 learned_addr learned my addr 172.21.15.94:0/1717268669 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff42f7fe640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4301506b0 0x7ff4300c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff42ffff640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4300acc00 0x7ff4301e89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 -- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4301506b0 msgr2=0x7ff4300c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4301506b0 0x7ff4300c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 -- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4300acc00 msgr2=0x7ff4301e89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4300acc00 0x7ff4301e89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 -- 172.21.15.94:0/1717268669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430149800 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff42ffff640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4300acc00 0x7ff4301e89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.007+0000 7ff434a41640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff430156ea0 0x7ff430151f60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff428009f40 tx=0x7ff428004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.008+0000 7ff42cff9640 1 -- 172.21.15.94:0/1717268669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428043020 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.008+0000 7ff42cff9640 1 -- 172.21.15.94:0/1717268669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42803e450 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.008+0000 7ff42cff9640 1 -- 172.21.15.94:0/1717268669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428009510 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.008+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4300c7d20 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.008+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff430149f00 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.009+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4301ecde0 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.009+0000 7ff42cff9640 1 -- 172.21.15.94:0/1717268669 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff428046020 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.009+0000 7ff42cff9640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff41c06b4d0 0x7ff41c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.009+0000 7ff42cff9640 1 -- 172.21.15.94:0/1717268669 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff4280acb60 con 0x7ff430156ea0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.010+0000 7ff42ffff640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff41c06b4d0 0x7ff41c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.010+0000 7ff42ffff640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff41c06b4d0 0x7ff41c06d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4300c85c0 tx=0x7ff424008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.014+0000 7ff42cff9640 1 -- 172.21.15.94:0/1717268669 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4301ecde0 con 0x7ff430156ea0 2024-03-06T07:18:15.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.187+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff43004f330 con 0x7ff430156ea0 2024-03-06T07:18:15.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.188+0000 7ff42cff9640 1 -- 172.21.15.94:0/1717268669 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7ff42804d030 con 0x7ff430156ea0 2024-03-06T07:18:15.190 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:15.191 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:15.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff41c06b4d0 msgr2=0x7ff41c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff41c06b4d0 0x7ff41c06d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff4300c85c0 tx=0x7ff424008040 comp rx=0 tx=0).stop 2024-03-06T07:18:15.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff430156ea0 msgr2=0x7ff430151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff430156ea0 0x7ff430151f60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff428009f40 tx=0x7ff428004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 shutdown_connections 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff41c06b4d0 0x7ff41c06d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff430156ea0 0x7ff430151f60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4301506b0 0x7ff4300c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 --2- 172.21.15.94:0/1717268669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4300acc00 0x7ff4301e89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 >> 172.21.15.94:0/1717268669 conn(0x7ff43006b1a0 msgr2=0x7ff430152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 shutdown_connections 2024-03-06T07:18:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.190+0000 7ff4364a4640 1 -- 172.21.15.94:0/1717268669 wait complete. 2024-03-06T07:18:15.203 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:15.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.301+0000 7f9041eb5640 1 Processor -- start 2024-03-06T07:18:15.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.301+0000 7f9041eb5640 1 -- start start 2024-03-06T07:18:15.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f9041eb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f903c150920 0x7f903c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f9041eb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c0c2e70 0x7f903c15dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f9041eb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903c067340 con 0x7f903c06c720 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f9041eb5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903c150d00 con 0x7f903c150920 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f9041eb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0c2010 con 0x7f903c0c2e70 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f903c150920 0x7f903c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f903c150920 0x7f903c0c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41268/0 (socket says 172.21.15.94:41268) 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903b7fe640 1 -- 172.21.15.94:0/1200373584 learned_addr learned my addr 172.21.15.94:0/1200373584 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903bfff640 1 --2- 172.21.15.94:0/1200373584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c0c2e70 0x7f903c15dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903b7fe640 1 -- 172.21.15.94:0/1200373584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c0c2e70 msgr2=0x7f903c15dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903b7fe640 1 --2- 172.21.15.94:0/1200373584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c0c2e70 0x7f903c15dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903b7fe640 1 -- 172.21.15.94:0/1200373584 >> v1:172.21.15.145:6789/0 conn(0x7f903c06c720 legacy=0x7f903c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.302+0000 7f903b7fe640 1 -- 172.21.15.94:0/1200373584 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c067060 con 0x7f903c150920 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.303+0000 7f903b7fe640 1 --2- 172.21.15.94:0/1200373584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c150920 0x7f903c0c2930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f902c009870 tx=0x7f902c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17d24efb815ee91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.303+0000 7f903a7fc640 1 -- 172.21.15.94:0/1200373584 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c035020 con 0x7f903c150920 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.303+0000 7f903a7fc640 1 -- 172.21.15.94:0/1200373584 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c031640 con 0x7f903c150920 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.303+0000 7f903a7fc640 1 -- 172.21.15.94:0/1200373584 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c02d040 con 0x7f903c150920 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 -- 172.21.15.94:0/1200373584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c150920 msgr2=0x7f903c0c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 --2- 172.21.15.94:0/1200373584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c150920 0x7f903c0c2930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f902c009870 tx=0x7f902c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 -- 172.21.15.94:0/1200373584 shutdown_connections 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 --2- 172.21.15.94:0/1200373584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c0c2e70 0x7f903c15dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 --2- 172.21.15.94:0/1200373584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c150920 0x7f903c0c2930 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 -- 172.21.15.94:0/1200373584 >> 172.21.15.94:0/1200373584 conn(0x7f903c0c70b0 msgr2=0x7f903c0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:15.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 -- 172.21.15.94:0/1200373584 shutdown_connections 2024-03-06T07:18:15.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.304+0000 7f9041eb5640 1 -- 172.21.15.94:0/1200373584 wait complete. 2024-03-06T07:18:15.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 Processor -- start 2024-03-06T07:18:15.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 -- start start 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c06c720 0x7f903c0c0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c2e70 0x7f903c1eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c150920 0x7f903c1eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c073210 con 0x7f903c06c720 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c073090 con 0x7f903c0c2e70 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.305+0000 7f9041eb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0c1b40 con 0x7f903c150920 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c06c720 0x7f903c0c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c06c720 0x7f903c0c0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41274/0 (socket says 172.21.15.94:41274) 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 -- 172.21.15.94:0/2537286586 learned_addr learned my addr 172.21.15.94:0/2537286586 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903affd640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c2e70 0x7f903c1eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903bfff640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c150920 0x7f903c1eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 -- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c150920 msgr2=0x7f903c1eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c150920 0x7f903c1eb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 -- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c2e70 msgr2=0x7f903c1eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c2e70 0x7f903c1eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 -- 172.21.15.94:0/2537286586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0abe20 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f903b7fe640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c06c720 0x7f903c0c0750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c00 tx=0x7f902c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f9023fff640 1 -- 172.21.15.94:0/2537286586 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c040070 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f9023fff640 1 -- 172.21.15.94:0/2537286586 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c0399f0 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c0ac100 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f9023fff640 1 -- 172.21.15.94:0/2537286586 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c035040 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.306+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c152830 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.307+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c04f330 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.308+0000 7f9023fff640 1 -- 172.21.15.94:0/2537286586 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f902c04b060 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.308+0000 7f9023fff640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.308+0000 7f9023fff640 1 -- 172.21.15.94:0/2537286586 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f902c0abc20 con 0x7f903c06c720 2024-03-06T07:18:15.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.308+0000 7f903affd640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.309+0000 7f903affd640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902806b4d0 0x7f902806d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f903c14ed50 tx=0x7f9030009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.312+0000 7f9023fff640 1 -- 172.21.15.94:0/2537286586 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f902c07a1a0 con 0x7f903c06c720 2024-03-06T07:18:15.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.485+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f903c068450 con 0x7f903c06c720 2024-03-06T07:18:15.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.486+0000 7f9023fff640 1 -- 172.21.15.94:0/2537286586 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f902c07e240 con 0x7f903c06c720 2024-03-06T07:18:15.488 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:15.489 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902806b4d0 msgr2=0x7f902806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902806b4d0 0x7f902806d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f903c14ed50 tx=0x7f9030009040 comp rx=0 tx=0).stop 2024-03-06T07:18:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c06c720 msgr2=0x7f903c0c0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c06c720 0x7f903c0c0750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c00 tx=0x7f902c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 shutdown_connections 2024-03-06T07:18:15.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c150920 0x7f903c1eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c2e70 0x7f903c1eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 --2- 172.21.15.94:0/2537286586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c06c720 0x7f903c0c0750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 >> 172.21.15.94:0/2537286586 conn(0x7f903c0c70b0 msgr2=0x7f903c06dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 shutdown_connections 2024-03-06T07:18:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.488+0000 7f9041eb5640 1 -- 172.21.15.94:0/2537286586 wait complete. 2024-03-06T07:18:15.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:15.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.703+0000 7f57d7e2a640 1 Processor -- start 2024-03-06T07:18:15.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d7e2a640 1 -- start start 2024-03-06T07:18:15.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d7e2a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f57d0155c80 0x7f57d006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d7e2a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f57d00720a0 0x7f57d006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d7e2a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00c1ae0 con 0x7f57d0152520 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d7e2a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d00c1dc0 con 0x7f57d0155c80 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d7e2a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d006e450 con 0x7f57d00720a0 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d53c5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f57d0155c80 0x7f57d006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d53c5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f57d0155c80 0x7f57d006ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41288/0 (socket says 172.21.15.94:41288) 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.704+0000 7f57d53c5640 1 -- 172.21.15.94:0/922179614 learned_addr learned my addr 172.21.15.94:0/922179614 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.705+0000 7f57d63c7640 1 --2- 172.21.15.94:0/922179614 >> v2:172.21.15.145:3300/0 conn(0x7f57d00720a0 0x7f57d006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.705+0000 7f57d53c5640 1 -- 172.21.15.94:0/922179614 >> v2:172.21.15.145:3300/0 conn(0x7f57d00720a0 msgr2=0x7f57d006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.705+0000 7f57d53c5640 1 --2- 172.21.15.94:0/922179614 >> v2:172.21.15.145:3300/0 conn(0x7f57d00720a0 0x7f57d006afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.705+0000 7f57d53c5640 1 -- 172.21.15.94:0/922179614 >> v1:172.21.15.145:6789/0 conn(0x7f57d0152520 legacy=0x7f57d00c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.705+0000 7f57d53c5640 1 -- 172.21.15.94:0/922179614 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d006e730 con 0x7f57d0155c80 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.705+0000 7f57d53c5640 1 --2- 172.21.15.94:0/922179614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d0155c80 0x7f57d006ef30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f57d0070f30 tx=0x7f57cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0571537490aeebd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d4bc4640 1 -- 172.21.15.94:0/922179614 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc034070 con 0x7f57d0155c80 2024-03-06T07:18:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d4bc4640 1 -- 172.21.15.94:0/922179614 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc030860 con 0x7f57d0155c80 2024-03-06T07:18:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d4bc4640 1 -- 172.21.15.94:0/922179614 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc030c30 con 0x7f57d0155c80 2024-03-06T07:18:15.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 -- 172.21.15.94:0/922179614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d0155c80 msgr2=0x7f57d006ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/922179614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d0155c80 0x7f57d006ef30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f57d0070f30 tx=0x7f57cc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 -- 172.21.15.94:0/922179614 shutdown_connections 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/922179614 >> v2:172.21.15.145:3300/0 conn(0x7f57d00720a0 0x7f57d006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/922179614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d0155c80 0x7f57d006ef30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 -- 172.21.15.94:0/922179614 >> 172.21.15.94:0/922179614 conn(0x7f57d00c76b0 msgr2=0x7f57d00c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 -- 172.21.15.94:0/922179614 shutdown_connections 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.706+0000 7f57d7e2a640 1 -- 172.21.15.94:0/922179614 wait complete. 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.707+0000 7f57d7e2a640 1 Processor -- start 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.707+0000 7f57d7e2a640 1 -- start start 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d7e2a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d00720a0 0x7f57d014fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d7e2a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57d0152520 0x7f57d01ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d7e2a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57d0155c80 0x7f57d014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d7e2a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0152a80 con 0x7f57d00720a0 2024-03-06T07:18:15.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d7e2a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0152900 con 0x7f57d0152520 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d7e2a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d006b500 con 0x7f57d0155c80 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d00720a0 0x7f57d014fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d00720a0 0x7f57d014fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41298/0 (socket says 172.21.15.94:41298) 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 -- 172.21.15.94:0/3966296497 learned_addr learned my addr 172.21.15.94:0/3966296497 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d53c5640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57d0152520 0x7f57d01ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d63c7640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57d0155c80 0x7f57d014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 -- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57d0155c80 msgr2=0x7f57d014ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57d0155c80 0x7f57d014ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 -- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57d0152520 msgr2=0x7f57d01ee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57d0152520 0x7f57d01ee3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 -- 172.21.15.94:0/3966296497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d01ef070 con 0x7f57d00720a0 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.708+0000 7f57d5bc6640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d00720a0 0x7f57d014fd20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f57c000c430 tx=0x7f57c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.709+0000 7f57c67fc640 1 -- 172.21.15.94:0/3966296497 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c00052d0 con 0x7f57d00720a0 2024-03-06T07:18:15.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.709+0000 7f57c67fc640 1 -- 172.21.15.94:0/3966296497 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c0005490 con 0x7f57d00720a0 2024-03-06T07:18:15.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.709+0000 7f57c67fc640 1 -- 172.21.15.94:0/3966296497 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0007260 con 0x7f57d00720a0 2024-03-06T07:18:15.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.709+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d00a6e00 con 0x7f57d00720a0 2024-03-06T07:18:15.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.709+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d01eed90 con 0x7f57d00720a0 2024-03-06T07:18:15.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.709+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d00a6fd0 con 0x7f57d00720a0 2024-03-06T07:18:15.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.710+0000 7f57c67fc640 1 -- 172.21.15.94:0/3966296497 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57c0007400 con 0x7f57d00720a0 2024-03-06T07:18:15.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.710+0000 7f57c67fc640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f57b406b4d0 0x7f57b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:15.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.710+0000 7f57c67fc640 1 -- 172.21.15.94:0/3966296497 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f57c00898f0 con 0x7f57d00720a0 2024-03-06T07:18:15.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.710+0000 7f57d53c5640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f57b406b4d0 0x7f57b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:15.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.711+0000 7f57d53c5640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f57b406b4d0 0x7f57b406d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f57cc036000 tx=0x7f57cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:15.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.715+0000 7f57c67fc640 1 -- 172.21.15.94:0/3966296497 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57c0057e70 con 0x7f57d00720a0 2024-03-06T07:18:15.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.921+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57d004f330 con 0x7f57d00720a0 2024-03-06T07:18:15.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.921+0000 7f57c67fc640 1 -- 172.21.15.94:0/3966296497 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4981 (secure 0 0 0) 0x7f57c005bf10 con 0x7f57d00720a0 2024-03-06T07:18:15.923 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:15.923 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:13.978510+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:15.924 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 7 2024-03-06T07:18:15.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f57b406b4d0 msgr2=0x7f57b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f57b406b4d0 0x7f57b406d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f57cc036000 tx=0x7f57cc008030 comp rx=0 tx=0).stop 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d00720a0 msgr2=0x7f57d014fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d00720a0 0x7f57d014fd20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f57c000c430 tx=0x7f57c000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 shutdown_connections 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f57b406b4d0 0x7f57b406d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57d0155c80 0x7f57d014ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57d0152520 0x7f57d01ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 --2- 172.21.15.94:0/3966296497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57d00720a0 0x7f57d014fd20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 >> 172.21.15.94:0/3966296497 conn(0x7f57d00c76b0 msgr2=0x7f57d00c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 shutdown_connections 2024-03-06T07:18:15.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:15.923+0000 7f57d7e2a640 1 -- 172.21.15.94:0/3966296497 wait complete. 2024-03-06T07:18:15.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:16.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.034+0000 7f6058efb640 1 Processor -- start 2024-03-06T07:18:16.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.034+0000 7f6058efb640 1 -- start start 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f6058efb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f60540c1990 0x7f6054160b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f6058efb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60540c1dc0 con 0x7f6054155c20 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f6058efb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605406afc0 con 0x7f6054152920 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f6058efb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605406b2a0 con 0x7f60540c1990 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f6051d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6054155c20 0x7f60540c1280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41534/0 (socket says 172.21.15.94:41534) 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f6051d9b640 1 -- 172.21.15.94:0/2159272233 learned_addr learned my addr 172.21.15.94:0/2159272233 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f6052d9d640 1 --2- 172.21.15.94:0/2159272233 >> v2:172.21.15.145:3300/0 conn(0x7f60540c1990 0x7f6054160b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f605159a640 1 -- 172.21.15.94:0/2159272233 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270802441 0 0) 0x7f60540c1dc0 con 0x7f6054155c20 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.035+0000 7f605159a640 1 -- 172.21.15.94:0/2159272233 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c003520 con 0x7f6054155c20 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.036+0000 7f6052d9d640 1 -- 172.21.15.94:0/2159272233 >> v1:172.21.15.145:6789/0 conn(0x7f6054152920 legacy=0x7f60540c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.036+0000 7f605159a640 1 -- 172.21.15.94:0/2159272233 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891685532 0 0) 0x7f605406afc0 con 0x7f6054152920 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.036+0000 7f6052d9d640 1 -- 172.21.15.94:0/2159272233 >> v1:172.21.15.94:6789/0 conn(0x7f6054155c20 legacy=0x7f60540c1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.036+0000 7f6052d9d640 1 -- 172.21.15.94:0/2159272233 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605406b580 con 0x7f60540c1990 2024-03-06T07:18:16.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.036+0000 7f6052d9d640 1 --2- 172.21.15.94:0/2159272233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60540c1990 0x7f6054160b30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f604c0097a0 tx=0x7f604c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=300939a29b29850b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.036+0000 7f605159a640 1 -- 172.21.15.94:0/2159272233 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c034070 con 0x7f60540c1990 2024-03-06T07:18:16.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f605159a640 1 -- 172.21.15.94:0/2159272233 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c030610 con 0x7f60540c1990 2024-03-06T07:18:16.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f605159a640 1 -- 172.21.15.94:0/2159272233 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c0309c0 con 0x7f60540c1990 2024-03-06T07:18:16.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f6058efb640 1 -- 172.21.15.94:0/2159272233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60540c1990 msgr2=0x7f6054160b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f6058efb640 1 --2- 172.21.15.94:0/2159272233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60540c1990 0x7f6054160b30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f604c0097a0 tx=0x7f604c02b960 comp rx=0 tx=0).stop 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f6058efb640 1 -- 172.21.15.94:0/2159272233 shutdown_connections 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f6058efb640 1 --2- 172.21.15.94:0/2159272233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60540c1990 0x7f6054160b30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f6058efb640 1 -- 172.21.15.94:0/2159272233 >> 172.21.15.94:0/2159272233 conn(0x7f60540c4290 msgr2=0x7f60540c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f6058efb640 1 -- 172.21.15.94:0/2159272233 shutdown_connections 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.037+0000 7f6058efb640 1 -- 172.21.15.94:0/2159272233 wait complete. 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.038+0000 7f6058efb640 1 Processor -- start 2024-03-06T07:18:16.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.038+0000 7f6058efb640 1 -- start start 2024-03-06T07:18:16.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6058efb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60540c1990 0x7f6054153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6058efb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6054152920 0x7f6054145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6058efb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6054155c20 0x7f6054145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6058efb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60541560b0 con 0x7f6054152920 2024-03-06T07:18:16.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6058efb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540c0d60 con 0x7f6054155c20 2024-03-06T07:18:16.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6058efb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540c28a0 con 0x7f60540c1990 2024-03-06T07:18:16.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6054152920 0x7f6054145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6054152920 0x7f6054145700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41312/0 (socket says 172.21.15.94:41312) 2024-03-06T07:18:16.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 -- 172.21.15.94:0/3016585772 learned_addr learned my addr 172.21.15.94:0/3016585772 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f605259c640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60540c1990 0x7f6054153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6052d9d640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6054155c20 0x7f6054145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 -- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60540c1990 msgr2=0x7f6054153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60540c1990 0x7f6054153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 -- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6054155c20 msgr2=0x7f6054145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6054155c20 0x7f6054145c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6051d9b640 1 -- 172.21.15.94:0/3016585772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60541484f0 con 0x7f6054152920 2024-03-06T07:18:16.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.039+0000 7f6052d9d640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6054155c20 0x7f6054145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:16.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.040+0000 7f6051d9b640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6054152920 0x7f6054145700 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f604800c390 tx=0x7f604800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.040+0000 7f6042ffd640 1 -- 172.21.15.94:0/3016585772 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048017070 con 0x7f6054152920 2024-03-06T07:18:16.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.040+0000 7f6042ffd640 1 -- 172.21.15.94:0/3016585772 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6048005ce0 con 0x7f6054152920 2024-03-06T07:18:16.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.040+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6054147730 con 0x7f6054152920 2024-03-06T07:18:16.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.040+0000 7f6042ffd640 1 -- 172.21.15.94:0/3016585772 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480072f0 con 0x7f6054152920 2024-03-06T07:18:16.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.040+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6054147cf0 con 0x7f6054152920 2024-03-06T07:18:16.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.041+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605404f330 con 0x7f6054152920 2024-03-06T07:18:16.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.041+0000 7f6042ffd640 1 -- 172.21.15.94:0/3016585772 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f604801a070 con 0x7f6054152920 2024-03-06T07:18:16.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.042+0000 7f6042ffd640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.042+0000 7f6042ffd640 1 -- 172.21.15.94:0/3016585772 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6048089740 con 0x7f6054152920 2024-03-06T07:18:16.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.042+0000 7f605259c640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.042+0000 7f605259c640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f603406b4d0 0x7f603406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f60440012e0 tx=0x7f6044009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.046+0000 7f6042ffd640 1 -- 172.21.15.94:0/3016585772 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6048057cc0 con 0x7f6054152920 2024-03-06T07:18:16.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.218+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6054068460 con 0x7f6054152920 2024-03-06T07:18:16.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.219+0000 7f6042ffd640 1 -- 172.21.15.94:0/3016585772 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f604805bd60 con 0x7f6054152920 2024-03-06T07:18:16.221 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:16.221 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:16.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f603406b4d0 msgr2=0x7f603406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f603406b4d0 0x7f603406d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f60440012e0 tx=0x7f6044009040 comp rx=0 tx=0).stop 2024-03-06T07:18:16.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6054152920 msgr2=0x7f6054145700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6054152920 0x7f6054145700 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f604800c390 tx=0x7f604800c850 comp rx=0 tx=0).stop 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 shutdown_connections 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f603406b4d0 0x7f603406d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6054155c20 0x7f6054145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6054152920 0x7f6054145700 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 --2- 172.21.15.94:0/3016585772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60540c1990 0x7f6054153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 >> 172.21.15.94:0/3016585772 conn(0x7f60540c4290 msgr2=0x7f605414c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 shutdown_connections 2024-03-06T07:18:16.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.221+0000 7f6058efb640 1 -- 172.21.15.94:0/3016585772 wait complete. 2024-03-06T07:18:16.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:18:08.851499+0000', 'modified': '2024-03-06T07:18:13.978510+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34136': {'gid': 34136, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/2223795314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2223795314}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2223795314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 34136, 'qdb_cluster': [34136]} 2024-03-06T07:18:16.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:18:16.636 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:18:16.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.737+0000 7fa343c9a640 1 Processor -- start 2024-03-06T07:18:16.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.737+0000 7fa343c9a640 1 -- start start 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa343c9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c0ab860 0x7fa33c0c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa343c9a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa33c0ac2b0 0x7fa33c1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa343c9a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c1553c0 con 0x7fa33c06c240 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa343c9a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33c1556a0 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa343c9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c155980 con 0x7fa33c0ab860 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa342237640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa33c0ac2b0 0x7fa33c1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341235640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c0ab860 0x7fa33c0c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341235640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c0ab860 0x7fa33c0c4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:37954/0 (socket says 172.21.15.94:37954) 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa342237640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa33c0ac2b0 0x7fa33c1588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:37218/0 (socket says 172.21.15.94:37218) 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341235640 1 -- 172.21.15.94:0/3094181065 learned_addr learned my addr 172.21.15.94:0/3094181065 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341a36640 1 --1- 172.21.15.94:0/3094181065 >> v1:172.21.15.145:6789/0 conn(0x7fa33c06c240 0x7fa33c0ab150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37660/0 (socket says 172.21.15.94:37660) 2024-03-06T07:18:16.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341235640 1 -- 172.21.15.94:0/3094181065 >> v2:172.21.15.145:3300/0 conn(0x7fa33c0ac2b0 msgr2=0x7fa33c1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341235640 1 --2- 172.21.15.94:0/3094181065 >> v2:172.21.15.145:3300/0 conn(0x7fa33c0ac2b0 0x7fa33c1588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341235640 1 -- 172.21.15.94:0/3094181065 >> v1:172.21.15.145:6789/0 conn(0x7fa33c06c240 legacy=0x7fa33c0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa341235640 1 -- 172.21.15.94:0/3094181065 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c155c60 con 0x7fa33c0ab860 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.738+0000 7fa342237640 1 --2- 172.21.15.94:0/3094181065 >> v2:172.21.15.145:3300/0 conn(0x7fa33c0ac2b0 0x7fa33c1588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.739+0000 7fa341235640 1 --2- 172.21.15.94:0/3094181065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c0ab860 0x7fa33c0c4250 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009870 tx=0x7fa32c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=132e4414383680c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.739+0000 7fa340a34640 1 -- 172.21.15.94:0/3094181065 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c035020 con 0x7fa33c0ab860 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.739+0000 7fa340a34640 1 -- 172.21.15.94:0/3094181065 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c031640 con 0x7fa33c0ab860 2024-03-06T07:18:16.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.739+0000 7fa340a34640 1 -- 172.21.15.94:0/3094181065 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c02d040 con 0x7fa33c0ab860 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.739+0000 7fa343c9a640 1 -- 172.21.15.94:0/3094181065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c0ab860 msgr2=0x7fa33c0c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.739+0000 7fa343c9a640 1 --2- 172.21.15.94:0/3094181065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c0ab860 0x7fa33c0c4250 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa32c009870 tx=0x7fa32c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.740+0000 7fa343c9a640 1 -- 172.21.15.94:0/3094181065 shutdown_connections 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.740+0000 7fa343c9a640 1 --2- 172.21.15.94:0/3094181065 >> v2:172.21.15.145:3300/0 conn(0x7fa33c0ac2b0 0x7fa33c1588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.740+0000 7fa343c9a640 1 --2- 172.21.15.94:0/3094181065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c0ab860 0x7fa33c0c4250 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.740+0000 7fa343c9a640 1 -- 172.21.15.94:0/3094181065 >> 172.21.15.94:0/3094181065 conn(0x7fa33c0c7670 msgr2=0x7fa33c0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.740+0000 7fa343c9a640 1 -- 172.21.15.94:0/3094181065 shutdown_connections 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.740+0000 7fa343c9a640 1 -- 172.21.15.94:0/3094181065 wait complete. 2024-03-06T07:18:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 Processor -- start 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 -- start start 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c06c240 0x7fa33c0728c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0ab860 0x7fa33c1570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0ac2b0 0x7fa33c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c0c4830 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c156180 con 0x7fa33c0ab860 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa343c9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c06c620 con 0x7fa33c06c240 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.741+0000 7fa342237640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0ac2b0 0x7fa33c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa341a36640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c06c240 0x7fa33c0728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0ac2b0 0x7fa33c1575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41314/0 (socket says 172.21.15.94:41314) 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 -- 172.21.15.94:0/2753675518 learned_addr learned my addr 172.21.15.94:0/2753675518 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa341235640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0ab860 0x7fa33c1570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 -- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c06c240 msgr2=0x7fa33c0728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c06c240 0x7fa33c0728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 -- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0ab860 msgr2=0x7fa33c1570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0ab860 0x7fa33c1570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 -- 172.21.15.94:0/2753675518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c1eeda0 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa341235640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0ab860 0x7fa33c1570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa342237640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0ac2b0 0x7fa33c1575f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa33400efc0 tx=0x7fa33400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa3327fc640 1 -- 172.21.15.94:0/2753675518 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334018030 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa3327fc640 1 -- 172.21.15.94:0/2753675518 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33400cdf0 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa3327fc640 1 -- 172.21.15.94:0/2753675518 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334003bb0 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c0c6dc0 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.742+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c1ee500 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.744+0000 7fa3327fc640 1 -- 172.21.15.94:0/2753675518 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa334005d10 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.744+0000 7fa3327fc640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b4d0 0x7fa32006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.744+0000 7fa3327fc640 1 -- 172.21.15.94:0/2753675518 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa33408ad90 con 0x7fa33c0ac2b0 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.744+0000 7fa341a36640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b4d0 0x7fa32006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.745+0000 7fa341a36640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b4d0 0x7fa32006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa338007fd0 tx=0x7fa338007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.745+0000 7fa346522740 1 Processor -- start 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.745+0000 7fa346522740 1 -- start start 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa346522740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29cf50 0x55e4bc29aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29d330 0x55e4bc29af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29b4b0 0x55e4bc297680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa346522740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc2016b0 con 0x55e4bc29cf50 2024-03-06T07:18:16.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc1aaa60 con 0x55e4bc29d330 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc1fd750 con 0x55e4bc29b4b0 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29cf50 0x55e4bc29aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29cf50 0x55e4bc29aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41320/0 (socket says 172.21.15.94:41320) 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 -- 172.21.15.94:0/3304719503 learned_addr learned my addr 172.21.15.94:0/3304719503 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa342237640 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29b4b0 0x55e4bc297680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341235640 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29d330 0x55e4bc29af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 -- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29b4b0 msgr2=0x55e4bc297680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29b4b0 0x55e4bc297680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 -- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29d330 msgr2=0x55e4bc29af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29d330 0x55e4bc29af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 -- 172.21.15.94:0/3304719503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4bc1c8f30 con 0x55e4bc29cf50 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341235640 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29d330 0x55e4bc29af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:16.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa341a36640 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29cf50 0x55e4bc29aa30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa338007840 tx=0x7fa338013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301223d32d940899 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa31f7fe640 1 -- 172.21.15.94:0/3304719503 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338026070 con 0x55e4bc29cf50 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa31f7fe640 1 -- 172.21.15.94:0/3304719503 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa338013b80 con 0x55e4bc29cf50 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.746+0000 7fa31f7fe640 1 -- 172.21.15.94:0/3304719503 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338001d40 con 0x55e4bc29cf50 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 -- 172.21.15.94:0/3304719503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29cf50 msgr2=0x55e4bc29aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29cf50 0x55e4bc29aa30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa338007840 tx=0x7fa338013000 comp rx=0 tx=0).stop 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 -- 172.21.15.94:0/3304719503 shutdown_connections 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29b4b0 0x55e4bc297680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29d330 0x55e4bc29af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 --2- 172.21.15.94:0/3304719503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29cf50 0x55e4bc29aa30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 -- 172.21.15.94:0/3304719503 >> 172.21.15.94:0/3304719503 conn(0x55e4bc2997d0 msgr2=0x55e4bc2be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 -- 172.21.15.94:0/3304719503 shutdown_connections 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.747+0000 7fa346522740 1 -- 172.21.15.94:0/3304719503 wait complete. 2024-03-06T07:18:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 -- start start 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 Processor -- start 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2d8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc2bf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc2d5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc2bf230 con 0x55e4bc29d330 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc1bdea0 con 0x55e4bc29cf50 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.748+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc2bec70 con 0x55e4bc2c2a30 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa346522740 10 client.0 _ll_register_callbacks cb 0x55e4bc1bd870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc2bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341a36640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2d8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc2bf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41332/0 (socket says 172.21.15.94:41332) 2024-03-06T07:18:16.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 -- 172.21.15.94:0/2082828071 learned_addr learned my addr 172.21.15.94:0/2082828071 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341a36640 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2d8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:37244/0 (socket says 172.21.15.94:37244) 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa342237640 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc2d5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 -- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 msgr2=0x55e4bc2d5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc2d5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 -- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 msgr2=0x55e4bc2d8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2d8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 -- 172.21.15.94:0/2082828071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4bc1be270 con 0x55e4bc29d330 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa341235640 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc2bf600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa32c02bf00 tx=0x7fa32c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906f5cd33cb32db0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa31cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa31cff9640 1 -- 172.21.15.94:0/2082828071 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c04a060 con 0x55e4bc29d330 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa31cff9640 1 -- 172.21.15.94:0/2082828071 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa32c043920 con 0x55e4bc29d330 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa338013b80 con 0x55e4bc29d330 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa31cff9640 1 -- 172.21.15.94:0/2082828071 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa32c035040 con 0x55e4bc29d330 2024-03-06T07:18:16.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa346522740 10 client.4404 fetch_fsmap learned FSMap version 7 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55e4bbf162d0 con 0x55e4bc29d330 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa31cff9640 1 -- 172.21.15.94:0/2082828071 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa338013b80 con 0x55e4bc29d330 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa31cff9640 1 -- 172.21.15.94:0/2082828071 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7fa32c043b20 con 0x55e4bc29d330 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa346522740 10 client.4404 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.749+0000 7fa346522740 10 client.4404 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.750+0000 7fa346522740 20 client.4404 populate_metadata read hostname 'smithi094' 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.750+0000 7fa346522740 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2ea2f0 0x55e4bc2f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.750+0000 7fa346522740 4 client.4404 mds_command: new command op to 34136 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.750+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 --> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4bc20ebf0 con 0x55e4bc2ea2f0 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.750+0000 7fa341a36640 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2ea2f0 0x55e4bc2f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.750+0000 7fa341a36640 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2ea2f0 0x55e4bc2f84c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=922d1075507543f3 server_cookie=c821726b773b7359 in_seq=0 out_seq=0 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.750+0000 7fa31cff9640 10 client.4404 ms_handle_connect on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa31cff9640 1 -- 172.21.15.94:0/2082828071 <== mds.0 v2:172.21.15.145:6835/2223795314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e4bc20ebf0 con 0x55e4bc2ea2f0 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa31cff9640 10 client.4404 handle_command_reply: tid=0 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 client.4404 shutdown 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 msgr2=0x55e4bc2bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc2bf600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa32c02bf00 tx=0x7fa32c004730 comp rx=0 tx=0).stop 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 shutdown_connections 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2ea2f0 0x55e4bc2f84c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc2d5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc2bf600 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 --2- 172.21.15.94:0/2082828071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2d8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa31cff9640 0 client.4404 ms_handle_reset on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 >> 172.21.15.94:0/2082828071 conn(0x55e4bc29b4b0 msgr2=0x55e4bc296510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 shutdown_connections 2024-03-06T07:18:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 1 -- 172.21.15.94:0/2082828071 wait complete. 2024-03-06T07:18:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.751+0000 7fa346522740 20 client.4404 trim_cache size 0 max 16384 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 Processor -- start 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 -- start start 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2fd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29d330 0x55e4bc296510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc2c2a30 0x55e4bc2c11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c1f7d90 con 0x55e4bc2c2a30 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c1f7190 con 0x55e4bc29cf50 2024-03-06T07:18:16.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c1f7790 con 0x55e4bc29d330 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa342237640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc2c2a30 0x55e4bc2c11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa342237640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc2c2a30 0x55e4bc2c11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41346/0 (socket says 172.21.15.94:41346) 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa341235640 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29d330 0x55e4bc296510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa342237640 1 -- 172.21.15.94:0/2954029736 learned_addr learned my addr 172.21.15.94:0/2954029736 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.768+0000 7fa341a36640 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2fd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa342237640 1 -- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29d330 msgr2=0x55e4bc296510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa342237640 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29d330 0x55e4bc296510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa342237640 1 -- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 msgr2=0x55e4bc2fd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa342237640 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2fd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa342237640 1 -- 172.21.15.94:0/2954029736 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4bc2d9910 con 0x55e4bc2c2a30 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa342237640 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc2c2a30 0x55e4bc2c11d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa3340613c0 tx=0x7fa334055660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1bb9448dd2f30b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa31effd640 1 -- 172.21.15.94:0/2954029736 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334087070 con 0x55e4bc2c2a30 2024-03-06T07:18:16.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa31effd640 1 -- 172.21.15.94:0/2954029736 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334057590 con 0x55e4bc2c2a30 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa31effd640 1 -- 172.21.15.94:0/2954029736 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334074070 con 0x55e4bc2c2a30 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 -- 172.21.15.94:0/2954029736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc2c2a30 msgr2=0x55e4bc2c11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc2c2a30 0x55e4bc2c11d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa3340613c0 tx=0x7fa334055660 comp rx=0 tx=0).stop 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 -- 172.21.15.94:0/2954029736 shutdown_connections 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc2c2a30 0x55e4bc2c11d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc29d330 0x55e4bc296510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 --2- 172.21.15.94:0/2954029736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc2fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 -- 172.21.15.94:0/2954029736 >> 172.21.15.94:0/2954029736 conn(0x55e4bc29b4b0 msgr2=0x55e4bc2c3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 -- 172.21.15.94:0/2954029736 shutdown_connections 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.769+0000 7fa346522740 1 -- 172.21.15.94:0/2954029736 wait complete. 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.770+0000 7fa346522740 1 -- start start 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.770+0000 7fa346522740 1 Processor -- start 2024-03-06T07:18:16.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc346d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc296db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc297700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc2d9ae0 con 0x55e4bc29d330 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc2ed3e0 con 0x55e4bc29cf50 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e4bc2edbe0 con 0x55e4bc2c2a30 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 10 client.0 _ll_register_callbacks cb 0x7fa32c0091e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc296db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc296db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41360/0 (socket says 172.21.15.94:41360) 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 -- 172.21.15.94:0/332326736 learned_addr learned my addr 172.21.15.94:0/332326736 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341a36640 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc346d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa342237640 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc297700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 -- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 msgr2=0x55e4bc297700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc297700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 -- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 msgr2=0x55e4bc346d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc346d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 -- 172.21.15.94:0/332326736 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4bbfe06b0 con 0x55e4bc29d330 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa341235640 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc296db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0317c0 tx=0x7fa32c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1919c353708f93d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa31cff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa31cff9640 1 -- 172.21.15.94:0/332326736 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c000e60 con 0x55e4bc29d330 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa31cff9640 1 -- 172.21.15.94:0/332326736 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa32c004480 con 0x55e4bc29d330 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.771+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa334057590 con 0x55e4bc29d330 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa31cff9640 1 -- 172.21.15.94:0/332326736 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa32c043930 con 0x55e4bc29d330 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 10 client.4410 fetch_fsmap learned FSMap version 7 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55e4bc2f3d50 con 0x55e4bc29d330 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa31cff9640 1 -- 172.21.15.94:0/332326736 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334057590 con 0x55e4bc29d330 2024-03-06T07:18:16.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa31cff9640 1 -- 172.21.15.94:0/332326736 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 7) v1 ==== 3322+0+0 (secure 0 0 0) 0x7fa32c05b050 con 0x55e4bc29d330 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 10 client.4410 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 10 client.4410 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 20 client.4410 populate_metadata read hostname 'smithi094' 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2fc4e0 0x55e4bc2fc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 4 client.4410 mds_command: new command op to 34136 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 --> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e4bc208f40 con 0x55e4bc2fc4e0 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa341a36640 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2fc4e0 0x55e4bc2fc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa341a36640 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2fc4e0 0x55e4bc2fc8c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17feae27c2bd29d4 server_cookie=32e15b4286d82036 in_seq=0 out_seq=0 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa31cff9640 10 client.4410 ms_handle_connect on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa31cff9640 1 -- 172.21.15.94:0/332326736 <== mds.0 v2:172.21.15.145:6835/2223795314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55e4bc208f40 con 0x55e4bc2fc4e0 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa31cff9640 10 client.4410 handle_command_reply: tid=0 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.772+0000 7fa346522740 1 client.4410 shutdown 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 msgr2=0x55e4bc296db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc296db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0317c0 tx=0x7fa32c031000 comp rx=0 tx=0).stop 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 shutdown_connections 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x55e4bc2fc4e0 0x55e4bc2fc8c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55e4bc2c2a30 0x55e4bc297700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e4bc29d330 0x55e4bc296db0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 --2- 172.21.15.94:0/332326736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55e4bc29cf50 0x55e4bc346d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa31cff9640 0 client.4410 ms_handle_reset on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 >> 172.21.15.94:0/332326736 conn(0x55e4bc29b4b0 msgr2=0x55e4bc2c3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 shutdown_connections 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 1 -- 172.21.15.94:0/332326736 wait complete. 2024-03-06T07:18:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa346522740 20 client.4410 trim_cache size 0 max 16384 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 34136, 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 7.2479214519999999, 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 7, 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 24, 2024-03-06T07:18:16.783 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 23, 2024-03-06T07:18:16.784 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 13.695709920000001 2024-03-06T07:18:16.784 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:18:16.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b4d0 msgr2=0x7fa32006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b4d0 0x7fa32006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa338007fd0 tx=0x7fa338007e80 comp rx=0 tx=0).stop 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0ac2b0 msgr2=0x7fa33c1575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0ac2b0 0x7fa33c1575f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa33400efc0 tx=0x7fa33400c490 comp rx=0 tx=0).stop 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 shutdown_connections 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b4d0 0x7fa32006d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0ac2b0 0x7fa33c1575f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0ab860 0x7fa33c1570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 --2- 172.21.15.94:0/2753675518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c06c240 0x7fa33c0728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 >> 172.21.15.94:0/2753675518 conn(0x7fa33c0c7670 msgr2=0x7fa33c0c2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 shutdown_connections 2024-03-06T07:18:16.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.773+0000 7fa343c9a640 1 -- 172.21.15.94:0/2753675518 wait complete. 2024-03-06T07:18:16.786 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:18:16.787 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:16.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:18:16.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6e0241c640 1 Processor -- start 2024-03-06T07:18:16.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6e0241c640 1 -- start start 2024-03-06T07:18:16.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6e0241c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6dfc0c3410 0x7f6dfc152b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6e0241c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc1596a0 con 0x7f6dfc06c240 2024-03-06T07:18:16.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6e0241c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc1553a0 con 0x7f6dfc159210 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6e0241c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dfc155680 con 0x7f6dfc0c3410 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6dfbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6dfc06c240 0x7f6dfc152470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41542/0 (socket says 172.21.15.94:41542) 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6dfbfff640 1 -- 172.21.15.94:0/4162196466 learned_addr learned my addr 172.21.15.94:0/4162196466 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.885+0000 7f6dfb7fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6dfc0c3410 0x7f6dfc152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.886+0000 7f6dfaffd640 1 -- 172.21.15.94:0/4162196466 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259108356 0 0) 0x7f6dfc1596a0 con 0x7f6dfc06c240 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.886+0000 7f6dfaffd640 1 -- 172.21.15.94:0/4162196466 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de4003520 con 0x7f6dfc06c240 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.886+0000 7f6dfaffd640 1 -- 172.21.15.94:0/4162196466 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604062522 0 0) 0x7f6dfc1553a0 con 0x7f6dfc159210 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.886+0000 7f6dfb7fe640 1 -- 172.21.15.94:0/4162196466 >> v1:172.21.15.145:6789/0 conn(0x7f6dfc159210 legacy=0x7f6dfc0c19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.886+0000 7f6dfb7fe640 1 -- 172.21.15.94:0/4162196466 >> v1:172.21.15.94:6789/0 conn(0x7f6dfc06c240 legacy=0x7f6dfc152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.886+0000 7f6dfb7fe640 1 -- 172.21.15.94:0/4162196466 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc155960 con 0x7f6dfc0c3410 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6dfb7fe640 1 --2- 172.21.15.94:0/4162196466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc0c3410 0x7f6dfc152b80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6df00097a0 tx=0x7f6df002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfca72edc95878d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6dfaffd640 1 -- 172.21.15.94:0/4162196466 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0035020 con 0x7f6dfc0c3410 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6dfaffd640 1 -- 172.21.15.94:0/4162196466 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df0031830 con 0x7f6dfc0c3410 2024-03-06T07:18:16.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6dfaffd640 1 -- 172.21.15.94:0/4162196466 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0031be0 con 0x7f6dfc0c3410 2024-03-06T07:18:16.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6e0241c640 1 -- 172.21.15.94:0/4162196466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc0c3410 msgr2=0x7f6dfc152b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6e0241c640 1 --2- 172.21.15.94:0/4162196466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc0c3410 0x7f6dfc152b80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6df00097a0 tx=0x7f6df002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6e0241c640 1 -- 172.21.15.94:0/4162196466 shutdown_connections 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6e0241c640 1 --2- 172.21.15.94:0/4162196466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc0c3410 0x7f6dfc152b80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6e0241c640 1 -- 172.21.15.94:0/4162196466 >> 172.21.15.94:0/4162196466 conn(0x7f6dfc0c3ad0 msgr2=0x7f6dfc0c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6e0241c640 1 -- 172.21.15.94:0/4162196466 shutdown_connections 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.887+0000 7f6e0241c640 1 -- 172.21.15.94:0/4162196466 wait complete. 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.888+0000 7f6e0241c640 1 Processor -- start 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e0241c640 1 -- start start 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e0241c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc06c240 0x7f6dfc158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e0241c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dfc0c3410 0x7f6dfc156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e0241c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dfc159210 0x7f6dfc156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e0241c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc06c620 con 0x7f6dfc159210 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e0241c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc06d7c0 con 0x7f6dfc06c240 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e0241c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc06c7a0 con 0x7f6dfc0c3410 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e009b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dfc159210 0x7f6dfc156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6dfbfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc06c240 0x7f6dfc158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e009b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dfc159210 0x7f6dfc156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41364/0 (socket says 172.21.15.94:41364) 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6e009b9640 1 -- 172.21.15.94:0/1425273728 learned_addr learned my addr 172.21.15.94:0/1425273728 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.889+0000 7f6dfb7fe640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dfc0c3410 0x7f6dfc156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e009b9640 1 -- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dfc0c3410 msgr2=0x7f6dfc156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e009b9640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dfc0c3410 0x7f6dfc156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e009b9640 1 -- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc06c240 msgr2=0x7f6dfc158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e009b9640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc06c240 0x7f6dfc158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e009b9640 1 -- 172.21.15.94:0/1425273728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc1de940 con 0x7f6dfc159210 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6dfbfff640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc06c240 0x7f6dfc158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e009b9640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dfc159210 0x7f6dfc156b40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c7b0 tx=0x7f6df400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6df8ff9640 1 -- 172.21.15.94:0/1425273728 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4003ba0 con 0x7f6dfc159210 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6df8ff9640 1 -- 172.21.15.94:0/1425273728 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df4003d40 con 0x7f6dfc159210 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc1dec20 con 0x7f6dfc159210 2024-03-06T07:18:16.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6df8ff9640 1 -- 172.21.15.94:0/1425273728 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4005270 con 0x7f6dfc159210 2024-03-06T07:18:16.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.890+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc158aa0 con 0x7f6dfc159210 2024-03-06T07:18:16.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.891+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc04f330 con 0x7f6dfc159210 2024-03-06T07:18:16.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.891+0000 7f6df8ff9640 1 -- 172.21.15.94:0/1425273728 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6df4005410 con 0x7f6dfc159210 2024-03-06T07:18:16.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.892+0000 7f6df8ff9640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6de806b500 0x7f6de806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:16.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.892+0000 7f6df8ff9640 1 -- 172.21.15.94:0/1425273728 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6df408a270 con 0x7f6dfc159210 2024-03-06T07:18:16.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.892+0000 7f6dfbfff640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6de806b500 0x7f6de806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:16.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.892+0000 7f6dfbfff640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6de806b500 0x7f6de806d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6dec005880 tx=0x7f6dec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:16.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:16.896+0000 7f6df8ff9640 1 -- 172.21.15.94:0/1425273728 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6df4058770 con 0x7f6dfc159210 2024-03-06T07:18:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.105+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6dfc06da00 con 0x7f6dfc159210 2024-03-06T07:18:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.105+0000 7f6df8ff9640 1 -- 172.21.15.94:0/1425273728 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6df405c810 con 0x7f6dfc159210 2024-03-06T07:18:17.107 INFO:teuthology.orchestra.run.smithi094.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6de806b500 msgr2=0x7f6de806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6de806b500 0x7f6de806d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6dec005880 tx=0x7f6dec009040 comp rx=0 tx=0).stop 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dfc159210 msgr2=0x7f6dfc156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dfc159210 0x7f6dfc156b40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c7b0 tx=0x7f6df400cbb0 comp rx=0 tx=0).stop 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 shutdown_connections 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6de806b500 0x7f6de806d9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dfc159210 0x7f6dfc156b40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dfc0c3410 0x7f6dfc156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 --2- 172.21.15.94:0/1425273728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dfc06c240 0x7f6dfc158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:17.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 >> 172.21.15.94:0/1425273728 conn(0x7f6dfc0c3ad0 msgr2=0x7f6dfc153940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:17.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 shutdown_connections 2024-03-06T07:18:17.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:17.107+0000 7f6e0241c640 1 -- 172.21.15.94:0/1425273728 wait complete. 2024-03-06T07:18:17.120 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:18:17.120 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:18:17.120 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2024-03-06T07:18:17.120 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-06T07:18:17.120 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:17.121 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:18:17.121 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:18:17.121 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:18:17.121 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-06T07:18:17.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.121 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:17.137 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f0 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f1 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42729sec preferred_lft 42729sec 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::ec4:7aff:fe8f:d763/64 scope link 2024-03-06T07:18:17.138 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:17.139 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:18:17.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.139 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:17.139 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:17.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:18:17.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr flush dev ceph-brx 2024-03-06T07:18:17.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set ceph-brx up 2024-03-06T07:18:17.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:18:17.140 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:17.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.353 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:18:17.381 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-03-06T07:18:17.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.383 DEBUG:teuthology.orchestra.run.smithi094:> route 2024-03-06T07:18:17.446 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2024-03-06T07:18:17.446 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:18:17.446 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:18:17.446 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:18:17.446 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:18:17.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.447 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:17.447 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:17.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:18:17.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:18:17.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:18:17.448 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:17.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.710 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-03-06T07:18:17.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.726 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2024-03-06T07:18:17.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.782 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:17.782 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:17.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:18:17.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-06T07:18:17.783 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:17.864 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-06T07:18:17.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:17.864 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:17.864 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:17.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-06T07:18:17.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:18:17.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-06T07:18:17.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-06T07:18:17.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-06T07:18:17.865 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:18.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:18.015 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:18.015 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:18.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2024-03-06T07:18:18.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:18:18.015 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:18.059 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-06T07:18:18.059 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:18:18.059 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:18.112 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-06T07:18:18.112 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:18:18.112 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:18.166 DEBUG:teuthology.orchestra.run.smithi094:> sudo modprobe fuse 2024-03-06T07:18:18.234 DEBUG:teuthology.orchestra.run.smithi094:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:18:18.289 INFO:teuthology.orchestra.run.smithi094.stdout:/proc 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/sys 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/dev 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/security 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/shm 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/pts 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/run 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/cgroup 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/pstore 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/bpf 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/ 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/selinux 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:18:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/hugepages 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/mqueue 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/debug 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/tracing 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/fuse/connections 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/config 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/1000 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:18:18.291 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:18:18.292 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:18:18.292 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns 2024-03-06T07:18:18.292 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:18:18.292 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:18:18.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:18.292 DEBUG:teuthology.orchestra.run.smithi094:> ls /sys/fs/fuse/connections 2024-03-06T07:18:18.344 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:18:18.345 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-06T07:18:18.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo modprobe fuse 2024-03-06T07:18:18.413 DEBUG:teuthology.orchestra.run.smithi094:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:18:18.470 INFO:teuthology.orchestra.run.smithi094.stdout:/proc 2024-03-06T07:18:18.470 INFO:teuthology.orchestra.run.smithi094.stdout:/sys 2024-03-06T07:18:18.470 INFO:teuthology.orchestra.run.smithi094.stdout:/dev 2024-03-06T07:18:18.470 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/security 2024-03-06T07:18:18.470 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/shm 2024-03-06T07:18:18.470 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/pts 2024-03-06T07:18:18.470 INFO:teuthology.orchestra.run.smithi094.stdout:/run 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/cgroup 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/pstore 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/bpf 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/ 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/selinux 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/hugepages 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/dev/mqueue 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/debug 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/tracing 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/fs/fuse/connections 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:18:18.471 INFO:teuthology.orchestra.run.smithi094.stdout:/sys/kernel/config 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/run/user/1000 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:18:18.472 INFO:teuthology.orchestra.run.smithi094.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:18:18.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:18.473 DEBUG:teuthology.orchestra.run.smithi094:> ls /sys/fs/fuse/connections 2024-03-06T07:18:18.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:2024-03-06T07:18:18.471+0000 7fc2e164c480 -1 init, newargv = 0x55b861744bf0 newargc=15 2024-03-06T07:18:18.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[43009]: starting ceph client 2024-03-06T07:18:18.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[43009]: starting fuse 2024-03-06T07:18:18.488 INFO:teuthology.orchestra.run.smithi094.stdout:41 2024-03-06T07:18:18.488 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:18:18.488 DEBUG:teuthology.orchestra.run.smithi094:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> import glob 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> import re 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> import os 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> import subprocess 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> def _find_admin_socket(client_name): 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> files = glob.glob(asok_path) 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> # Given a non-glob path, it better be there 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> if "*" not in asok_path: 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> assert(len(files) == 1) 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> return files[0] 2024-03-06T07:18:18.489 DEBUG:teuthology.orchestra.run.smithi094:> 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> for f in files: 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> contents = proc_f.read() 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> if mountpoint in contents: 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> return f 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> print(_find_admin_socket("client.admin")) 2024-03-06T07:18:18.490 DEBUG:teuthology.orchestra.run.smithi094:> ' 2024-03-06T07:18:18.608 INFO:teuthology.orchestra.run.smithi094.stdout:/var/run/ceph/ceph-client.admin.43009.asok 2024-03-06T07:18:18.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:18:18.618 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.43009.asok 2024-03-06T07:18:18.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:18.619 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.43009.asok status 2024-03-06T07:18:18.755 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:18:18.755 INFO:teuthology.orchestra.run.smithi094.stdout: "metadata": { 2024-03-06T07:18:18.755 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:18:18.755 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:18:18.755 INFO:teuthology.orchestra.run.smithi094.stdout: "entity_id": "admin", 2024-03-06T07:18:18.755 INFO:teuthology.orchestra.run.smithi094.stdout: "hostname": "smithi094", 2024-03-06T07:18:18.755 INFO:teuthology.orchestra.run.smithi094.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "pid": "43009", 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "root": "/" 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "dentry_count": 0, 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "dentry_pinned_count": 0, 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 4425, 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "inst": { 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "name": { 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "client", 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "num": 4425 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": { 2024-03-06T07:18:18.756 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 2635284269 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": { 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "type": "v1", 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "nonce": 2635284269 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "inst_str": "client.4425 192.168.0.1:0/2635284269", 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "addr_str": "192.168.0.1:0/2635284269", 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "inode_count": 1, 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "mds_epoch": 7, 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_epoch": 24, 2024-03-06T07:18:18.757 INFO:teuthology.orchestra.run.smithi094.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:18:18.758 INFO:teuthology.orchestra.run.smithi094.stdout: "blocklisted": false, 2024-03-06T07:18:18.758 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs" 2024-03-06T07:18:18.758 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:18:18.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:18.763 DEBUG:teuthology.orchestra.run.smithi094:> stat --file-system '--printf=%T 2024-03-06T07:18:18.764 DEBUG:teuthology.orchestra.run.smithi094:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:18.779 INFO:teuthology.orchestra.run.smithi094.stdout:fuseblk 2024-03-06T07:18:18.779 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:18.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:18.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:18.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:18.848 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-06T07:18:18.916 INFO:teuthology.orchestra.run.smithi094.stdout:# file: . 2024-03-06T07:18:18.916 INFO:teuthology.orchestra.run.smithi094.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-06T07:18:18.916 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:18.916 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:18:18.916 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:18.971 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:19.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.111+0000 7f959e16c640 1 Processor -- start 2024-03-06T07:18:19.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.111+0000 7f959e16c640 1 -- start start 2024-03-06T07:18:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.111+0000 7f959e16c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9598072720 0x7f95981564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.111+0000 7f959e16c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95980c3d60 0x7f95980c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f959e16c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9598156f90 con 0x7f9598156a00 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f959e16c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95980c38d0 con 0x7f95980c3d60 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f959e16c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9598157270 con 0x7f9598072720 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9597fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95980c3d60 0x7f95980c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9596ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9598156a00 0x7f9598150290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59366/0 (socket says 172.21.15.94:59366) 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9597fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95980c3d60 0x7f95980c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55946/0 (socket says 172.21.15.94:55946) 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9596ffd640 1 -- 172.21.15.94:0/3371218936 learned_addr learned my addr 172.21.15.94:0/3371218936 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f95977fe640 1 --2- 172.21.15.94:0/3371218936 >> v2:172.21.15.145:3300/0 conn(0x7f9598072720 0x7f95981564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f95967fc640 1 -- 172.21.15.94:0/3371218936 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788944296 0 0) 0x7f9598156f90 con 0x7f9598156a00 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f95967fc640 1 -- 172.21.15.94:0/3371218936 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580003520 con 0x7f9598156a00 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9597fff640 1 -- 172.21.15.94:0/3371218936 >> v2:172.21.15.145:3300/0 conn(0x7f9598072720 msgr2=0x7f95981564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9597fff640 1 --2- 172.21.15.94:0/3371218936 >> v2:172.21.15.145:3300/0 conn(0x7f9598072720 0x7f95981564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9597fff640 1 -- 172.21.15.94:0/3371218936 >> v1:172.21.15.94:6789/0 conn(0x7f9598156a00 legacy=0x7f9598150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.112+0000 7f9597fff640 1 -- 172.21.15.94:0/3371218936 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95980c3280 con 0x7f95980c3d60 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f9597fff640 1 --2- 172.21.15.94:0/3371218936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95980c3d60 0x7f95980c41b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95980c9850 tx=0x7f959002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0c1a332ce90058 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f95967fc640 1 -- 172.21.15.94:0/3371218936 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590039070 con 0x7f95980c3d60 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f95967fc640 1 -- 172.21.15.94:0/3371218936 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959002cbf0 con 0x7f95980c3d60 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f95967fc640 1 -- 172.21.15.94:0/3371218936 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590034440 con 0x7f95980c3d60 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 -- 172.21.15.94:0/3371218936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95980c3d60 msgr2=0x7f95980c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 --2- 172.21.15.94:0/3371218936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95980c3d60 0x7f95980c41b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95980c9850 tx=0x7f959002be40 comp rx=0 tx=0).stop 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 -- 172.21.15.94:0/3371218936 shutdown_connections 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 --2- 172.21.15.94:0/3371218936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95980c3d60 0x7f95980c41b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 --2- 172.21.15.94:0/3371218936 >> v2:172.21.15.145:3300/0 conn(0x7f9598072720 0x7f95981564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 -- 172.21.15.94:0/3371218936 >> 172.21.15.94:0/3371218936 conn(0x7f95980aa630 msgr2=0x7f95980aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 -- 172.21.15.94:0/3371218936 shutdown_connections 2024-03-06T07:18:19.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.113+0000 7f959e16c640 1 -- 172.21.15.94:0/3371218936 wait complete. 2024-03-06T07:18:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.114+0000 7f959e16c640 1 Processor -- start 2024-03-06T07:18:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f959e16c640 1 -- start start 2024-03-06T07:18:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f959e16c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9598072720 0x7f95981ec360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f959e16c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95980c3d60 0x7f95980a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f959e16c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9598156a00 0x7f95981495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f959e16c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95981509a0 con 0x7f9598072720 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f959e16c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959806f8f0 con 0x7f95980c3d60 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f959e16c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9598150b20 con 0x7f9598156a00 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f95977fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9598072720 0x7f95981ec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f95977fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9598072720 0x7f95981ec360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55954/0 (socket says 172.21.15.94:55954) 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f95977fe640 1 -- 172.21.15.94:0/652525393 learned_addr learned my addr 172.21.15.94:0/652525393 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f9596ffd640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95980c3d60 0x7f95980a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.115+0000 7f9597fff640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9598156a00 0x7f95981495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f95977fe640 1 -- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9598156a00 msgr2=0x7f95981495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f95977fe640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9598156a00 0x7f95981495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f95977fe640 1 -- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95980c3d60 msgr2=0x7f95980a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f95977fe640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95980c3d60 0x7f95980a9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f95977fe640 1 -- 172.21.15.94:0/652525393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598150e90 con 0x7f9598072720 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f95977fe640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9598072720 0x7f95981ec360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f958800e9b0 tx=0x7f958800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f957ffff640 1 -- 172.21.15.94:0/652525393 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588017070 con 0x7f9598072720 2024-03-06T07:18:19.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f957ffff640 1 -- 172.21.15.94:0/652525393 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958800c9a0 con 0x7f9598072720 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9598151170 con 0x7f9598072720 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f957ffff640 1 -- 172.21.15.94:0/652525393 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800cd90 con 0x7f9598072720 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.116+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95980aa070 con 0x7f9598072720 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.117+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95980a83a0 con 0x7f9598072720 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.117+0000 7f957ffff640 1 -- 172.21.15.94:0/652525393 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f958801a070 con 0x7f9598072720 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.118+0000 7f957ffff640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958406b4d0 0x7f958406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.118+0000 7f957ffff640 1 -- 172.21.15.94:0/652525393 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f958808a7e0 con 0x7f9598072720 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.118+0000 7f9596ffd640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958406b4d0 0x7f958406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.118+0000 7f9596ffd640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958406b4d0 0x7f958406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f95980aa370 tx=0x7f958c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.122+0000 7f957ffff640 1 -- 172.21.15.94:0/652525393 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95980a83a0 con 0x7f9598072720 2024-03-06T07:18:19.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.294+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f959804f330 con 0x7f9598072720 2024-03-06T07:18:19.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.295+0000 7f957ffff640 1 -- 172.21.15.94:0/652525393 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f9588058d60 con 0x7f9598072720 2024-03-06T07:18:19.298 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:19.298 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958406b4d0 msgr2=0x7f958406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958406b4d0 0x7f958406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f95980aa370 tx=0x7f958c009040 comp rx=0 tx=0).stop 2024-03-06T07:18:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9598072720 msgr2=0x7f95981ec360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9598072720 0x7f95981ec360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f958800e9b0 tx=0x7f958800ee70 comp rx=0 tx=0).stop 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 shutdown_connections 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958406b4d0 0x7f958406d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9598156a00 0x7f95981495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95980c3d60 0x7f95980a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 --2- 172.21.15.94:0/652525393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9598072720 0x7f95981ec360 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 >> 172.21.15.94:0/652525393 conn(0x7f95980aa630 msgr2=0x7f959814b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 shutdown_connections 2024-03-06T07:18:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.297+0000 7f959e16c640 1 -- 172.21.15.94:0/652525393 wait complete. 2024-03-06T07:18:19.310 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:19.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:19.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.411+0000 7fcfcd7bc640 1 Processor -- start 2024-03-06T07:18:19.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.411+0000 7fcfcd7bc640 1 -- start start 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfcd7bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc80c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfcd7bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc80c3ef0 con 0x7fcfc806c900 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfcd7bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc81556d0 con 0x7fcfc80c5990 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfcd7bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc81559b0 con 0x7fcfc80ab170 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc67fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcfc806c900 0x7fcfc80a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59382/0 (socket says 172.21.15.94:59382) 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc67fc640 1 -- 172.21.15.94:0/2975375463 learned_addr learned my addr 172.21.15.94:0/2975375463 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc77fe640 1 --2- 172.21.15.94:0/2975375463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc80c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc5ffb640 1 -- 172.21.15.94:0/2975375463 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588217481 0 0) 0x7fcfc80c3ef0 con 0x7fcfc806c900 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc5ffb640 1 -- 172.21.15.94:0/2975375463 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb0003520 con 0x7fcfc806c900 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc77fe640 1 -- 172.21.15.94:0/2975375463 >> v1:172.21.15.145:6789/0 conn(0x7fcfc80c5990 legacy=0x7fcfc806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc77fe640 1 -- 172.21.15.94:0/2975375463 >> v1:172.21.15.94:6789/0 conn(0x7fcfc806c900 legacy=0x7fcfc80a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.412+0000 7fcfc77fe640 1 -- 172.21.15.94:0/2975375463 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc8155c90 con 0x7fcfc80ab170 2024-03-06T07:18:19.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.413+0000 7fcfc77fe640 1 --2- 172.21.15.94:0/2975375463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc80c0ab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009dc0 tx=0x7fcfc002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e11462a27e678b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.413+0000 7fcfc5ffb640 1 -- 172.21.15.94:0/2975375463 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0039070 con 0x7fcfc80ab170 2024-03-06T07:18:19.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.413+0000 7fcfc5ffb640 1 -- 172.21.15.94:0/2975375463 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc002ccd0 con 0x7fcfc80ab170 2024-03-06T07:18:19.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.413+0000 7fcfc5ffb640 1 -- 172.21.15.94:0/2975375463 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0034640 con 0x7fcfc80ab170 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.414+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/2975375463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 msgr2=0x7fcfc80c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.414+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/2975375463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc80c0ab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009dc0 tx=0x7fcfc002bf20 comp rx=0 tx=0).stop 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.414+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/2975375463 shutdown_connections 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.414+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/2975375463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc80c0ab0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.414+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/2975375463 >> 172.21.15.94:0/2975375463 conn(0x7fcfc80c6710 msgr2=0x7fcfc80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.414+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/2975375463 shutdown_connections 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.414+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/2975375463 wait complete. 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.415+0000 7fcfcd7bc640 1 Processor -- start 2024-03-06T07:18:19.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.415+0000 7fcfcd7bc640 1 -- start start 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc806c900 0x7fcfc81e8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc8148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc80c5990 0x7fcfc8151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc8156610 con 0x7fcfc806c900 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc8156490 con 0x7fcfc80c5990 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc80c3af0 con 0x7fcfc80ab170 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc806c900 0x7fcfc81e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc806c900 0x7fcfc81e8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55960/0 (socket says 172.21.15.94:55960) 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 -- 172.21.15.94:0/176252380 learned_addr learned my addr 172.21.15.94:0/176252380 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc67fc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc8148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc77fe640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc80c5990 0x7fcfc8151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 -- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 msgr2=0x7fcfc8148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc8148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 -- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc80c5990 msgr2=0x7fcfc8151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc80c5990 0x7fcfc8151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 -- 172.21.15.94:0/176252380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc81ec470 con 0x7fcfc806c900 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfc6ffd640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc806c900 0x7fcfc81e8450 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c400 tx=0x7fcfb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfaf7fe640 1 -- 172.21.15.94:0/176252380 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80052d0 con 0x7fcfc806c900 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfaf7fe640 1 -- 172.21.15.94:0/176252380 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8005490 con 0x7fcfc806c900 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfaf7fe640 1 -- 172.21.15.94:0/176252380 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8007260 con 0x7fcfc806c900 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc81ec710 con 0x7fcfc806c900 2024-03-06T07:18:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.416+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc8149270 con 0x7fcfc806c900 2024-03-06T07:18:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.417+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc804f330 con 0x7fcfc806c900 2024-03-06T07:18:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.418+0000 7fcfaf7fe640 1 -- 172.21.15.94:0/176252380 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfb8007400 con 0x7fcfc806c900 2024-03-06T07:18:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.418+0000 7fcfaf7fe640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfb406b4d0 0x7fcfb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.418+0000 7fcfaf7fe640 1 -- 172.21.15.94:0/176252380 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcfb8089980 con 0x7fcfc806c900 2024-03-06T07:18:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.418+0000 7fcfc67fc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfb406b4d0 0x7fcfb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.419+0000 7fcfc67fc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfb406b4d0 0x7fcfb406d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80c1f90 tx=0x7fcfbc007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.423+0000 7fcfaf7fe640 1 -- 172.21.15.94:0/176252380 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfb8057f00 con 0x7fcfc806c900 2024-03-06T07:18:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.594+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfc80c1740 con 0x7fcfc806c900 2024-03-06T07:18:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.594+0000 7fcfaf7fe640 1 -- 172.21.15.94:0/176252380 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfb805bfa0 con 0x7fcfc806c900 2024-03-06T07:18:19.596 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfb406b4d0 msgr2=0x7fcfb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfb406b4d0 0x7fcfb406d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80c1f90 tx=0x7fcfbc007d30 comp rx=0 tx=0).stop 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc806c900 msgr2=0x7fcfc81e8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc806c900 0x7fcfc81e8450 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c400 tx=0x7fcfb800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 shutdown_connections 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfb406b4d0 0x7fcfb406d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc80c5990 0x7fcfc8151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc80ab170 0x7fcfc8148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 --2- 172.21.15.94:0/176252380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc806c900 0x7fcfc81e8450 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 >> 172.21.15.94:0/176252380 conn(0x7fcfc80c6710 msgr2=0x7fcfc806c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 shutdown_connections 2024-03-06T07:18:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.596+0000 7fcfcd7bc640 1 -- 172.21.15.94:0/176252380 wait complete. 2024-03-06T07:18:19.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:19.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.710+0000 7f7a68dac640 1 Processor -- start 2024-03-06T07:18:19.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.710+0000 7f7a68dac640 1 -- start start 2024-03-06T07:18:19.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a68dac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a641528d0 0x7f7a640724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a68dac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a64152cb0 con 0x7f7a64072a10 2024-03-06T07:18:19.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a68dac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a640c38b0 con 0x7f7a640c67f0 2024-03-06T07:18:19.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a68dac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641594f0 con 0x7f7a641528d0 2024-03-06T07:18:19.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a5bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7a64072a10 0x7f7a64155220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59398/0 (socket says 172.21.15.94:59398) 2024-03-06T07:18:19.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a5bfff640 1 -- 172.21.15.94:0/2287757172 learned_addr learned my addr 172.21.15.94:0/2287757172 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:19.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a6259c640 1 --2- 172.21.15.94:0/2287757172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a641528d0 0x7f7a640724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a61d9b640 1 -- 172.21.15.94:0/2287757172 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079063610 0 0) 0x7f7a64152cb0 con 0x7f7a64072a10 2024-03-06T07:18:19.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a61d9b640 1 -- 172.21.15.94:0/2287757172 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a50003520 con 0x7f7a64072a10 2024-03-06T07:18:19.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a61d9b640 1 -- 172.21.15.94:0/2287757172 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208722264 0 0) 0x7f7a640c38b0 con 0x7f7a640c67f0 2024-03-06T07:18:19.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a61d9b640 1 -- 172.21.15.94:0/2287757172 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a64152cb0 con 0x7f7a640c67f0 2024-03-06T07:18:19.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a6259c640 1 -- 172.21.15.94:0/2287757172 >> v1:172.21.15.145:6789/0 conn(0x7f7a640c67f0 legacy=0x7f7a6415fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a6259c640 1 -- 172.21.15.94:0/2287757172 >> v1:172.21.15.94:6789/0 conn(0x7f7a64072a10 legacy=0x7f7a64155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.711+0000 7f7a6259c640 1 -- 172.21.15.94:0/2287757172 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a640a7f70 con 0x7f7a641528d0 2024-03-06T07:18:19.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.712+0000 7f7a6259c640 1 --2- 172.21.15.94:0/2287757172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a641528d0 0x7f7a640724d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a540097d0 tx=0x7f7a5402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=152b2c8336448d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.712+0000 7f7a61d9b640 1 -- 172.21.15.94:0/2287757172 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54034070 con 0x7f7a641528d0 2024-03-06T07:18:19.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.712+0000 7f7a61d9b640 1 -- 172.21.15.94:0/2287757172 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a54030880 con 0x7f7a641528d0 2024-03-06T07:18:19.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.712+0000 7f7a61d9b640 1 -- 172.21.15.94:0/2287757172 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54030c30 con 0x7f7a641528d0 2024-03-06T07:18:19.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.712+0000 7f7a68dac640 1 -- 172.21.15.94:0/2287757172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a641528d0 msgr2=0x7f7a640724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.712+0000 7f7a68dac640 1 --2- 172.21.15.94:0/2287757172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a641528d0 0x7f7a640724d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a540097d0 tx=0x7f7a5402b9c0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.713+0000 7f7a68dac640 1 -- 172.21.15.94:0/2287757172 shutdown_connections 2024-03-06T07:18:19.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.713+0000 7f7a68dac640 1 --2- 172.21.15.94:0/2287757172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a641528d0 0x7f7a640724d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.713+0000 7f7a68dac640 1 -- 172.21.15.94:0/2287757172 >> 172.21.15.94:0/2287757172 conn(0x7f7a640aa6a0 msgr2=0x7f7a640aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:19.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.713+0000 7f7a68dac640 1 -- 172.21.15.94:0/2287757172 shutdown_connections 2024-03-06T07:18:19.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.713+0000 7f7a68dac640 1 -- 172.21.15.94:0/2287757172 wait complete. 2024-03-06T07:18:19.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 Processor -- start 2024-03-06T07:18:19.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 -- start start 2024-03-06T07:18:19.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7a64072a10 0x7f7a64150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a640c67f0 0x7f7a64150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a641528d0 0x7f7a64144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641591a0 con 0x7f7a64072a10 2024-03-06T07:18:19.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640a8250 con 0x7f7a641528d0 2024-03-06T07:18:19.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a68dac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640c4ef0 con 0x7f7a640c67f0 2024-03-06T07:18:19.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.714+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7a64072a10 0x7f7a64150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7a64072a10 0x7f7a64150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55970/0 (socket says 172.21.15.94:55970) 2024-03-06T07:18:19.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 -- 172.21.15.94:0/3787776317 learned_addr learned my addr 172.21.15.94:0/3787776317 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:19.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a62d9d640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a641528d0 0x7f7a64144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a5bfff640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a640c67f0 0x7f7a64150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 -- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a640c67f0 msgr2=0x7f7a64150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a640c67f0 0x7f7a64150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 -- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a641528d0 msgr2=0x7f7a64144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a641528d0 0x7f7a64144c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 -- 172.21.15.94:0/3787776317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a641514f0 con 0x7f7a64072a10 2024-03-06T07:18:19.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a6259c640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7a64072a10 0x7f7a64150a70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7a5402ba00 tx=0x7f7a540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a5affd640 1 -- 172.21.15.94:0/3787776317 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54034040 con 0x7f7a64072a10 2024-03-06T07:18:19.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a5affd640 1 -- 172.21.15.94:0/3787776317 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5403ada0 con 0x7f7a64072a10 2024-03-06T07:18:19.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a5affd640 1 -- 172.21.15.94:0/3787776317 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54006e10 con 0x7f7a64072a10 2024-03-06T07:18:19.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a641578b0 con 0x7f7a64072a10 2024-03-06T07:18:19.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.715+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a64147b30 con 0x7f7a64072a10 2024-03-06T07:18:19.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.716+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6404f330 con 0x7f7a64072a10 2024-03-06T07:18:19.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.717+0000 7f7a5affd640 1 -- 172.21.15.94:0/3787776317 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a54037070 con 0x7f7a64072a10 2024-03-06T07:18:19.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.717+0000 7f7a5affd640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:19.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.717+0000 7f7a5affd640 1 -- 172.21.15.94:0/3787776317 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7a540ac580 con 0x7f7a64072a10 2024-03-06T07:18:19.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.717+0000 7f7a5bfff640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:19.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.718+0000 7f7a5bfff640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a4406b4d0 0x7f7a4406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a64158400 tx=0x7f7a4c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:19.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.721+0000 7f7a5affd640 1 -- 172.21.15.94:0/3787776317 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a5403f070 con 0x7f7a64072a10 2024-03-06T07:18:19.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.894+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a64073120 con 0x7f7a64072a10 2024-03-06T07:18:19.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.894+0000 7f7a5affd640 1 -- 172.21.15.94:0/3787776317 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7a5407ab00 con 0x7f7a64072a10 2024-03-06T07:18:19.897 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a4406b4d0 msgr2=0x7f7a4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a4406b4d0 0x7f7a4406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a64158400 tx=0x7f7a4c007d30 comp rx=0 tx=0).stop 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7a64072a10 msgr2=0x7f7a64150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7a64072a10 0x7f7a64150a70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7a5402ba00 tx=0x7f7a540046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 shutdown_connections 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a641528d0 0x7f7a64144c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a640c67f0 0x7f7a64150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 --2- 172.21.15.94:0/3787776317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7a64072a10 0x7f7a64150a70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 >> 172.21.15.94:0/3787776317 conn(0x7f7a640aa6a0 msgr2=0x7f7a6414b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 shutdown_connections 2024-03-06T07:18:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:19.896+0000 7f7a68dac640 1 -- 172.21.15.94:0/3787776317 wait complete. 2024-03-06T07:18:19.910 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:20.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.010+0000 7f9bbf37a640 1 Processor -- start 2024-03-06T07:18:20.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.010+0000 7f9bbf37a640 1 -- start start 2024-03-06T07:18:20.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.010+0000 7f9bbf37a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9bb80c5990 0x7f9bb80c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.010+0000 7f9bbf37a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb806c900 0x7f9bb80a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.010+0000 7f9bbf37a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb80c3ef0 con 0x7f9bb80ab170 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.010+0000 7f9bbf37a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb81556d0 con 0x7f9bb80c5990 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.010+0000 7f9bbf37a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb81559b0 con 0x7f9bb806c900 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd116640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9bb80c5990 0x7f9bb80c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd917640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb806c900 0x7f9bb80a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd116640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9bb80c5990 0x7f9bb80c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:40016/0 (socket says 172.21.15.94:40016) 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd116640 1 -- 172.21.15.94:0/2899357415 learned_addr learned my addr 172.21.15.94:0/2899357415 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbc915640 1 --1- 172.21.15.94:0/2899357415 >> v1:172.21.15.145:6789/0 conn(0x7f9bb80ab170 0x7f9bb806c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:58220/0 (socket says 172.21.15.94:58220) 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd116640 1 -- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb806c900 msgr2=0x7f9bb80a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd116640 1 --2- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb806c900 0x7f9bb80a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd116640 1 -- 172.21.15.94:0/2899357415 >> v1:172.21.15.145:6789/0 conn(0x7f9bb80ab170 legacy=0x7f9bb806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd116640 1 -- 172.21.15.94:0/2899357415 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8155c90 con 0x7f9bb80c5990 2024-03-06T07:18:20.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.011+0000 7f9bbd917640 1 --2- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb806c900 0x7f9bb80a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:20.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9bbd116640 1 --2- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb80c0ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d150 tx=0x7f9bb002e160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c216718b96afd54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9baffff640 1 -- 172.21.15.94:0/2899357415 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb000b0d0 con 0x7f9bb80c5990 2024-03-06T07:18:20.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9baffff640 1 -- 172.21.15.94:0/2899357415 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb000b270 con 0x7f9bb80c5990 2024-03-06T07:18:20.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9baffff640 1 -- 172.21.15.94:0/2899357415 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0009600 con 0x7f9bb80c5990 2024-03-06T07:18:20.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9bbf37a640 1 -- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 msgr2=0x7f9bb80c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb80c0ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d150 tx=0x7f9bb002e160 comp rx=0 tx=0).stop 2024-03-06T07:18:20.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9bbf37a640 1 -- 172.21.15.94:0/2899357415 shutdown_connections 2024-03-06T07:18:20.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb806c900 0x7f9bb80a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/2899357415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb80c0ab0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.012+0000 7f9bbf37a640 1 -- 172.21.15.94:0/2899357415 >> 172.21.15.94:0/2899357415 conn(0x7f9bb80c6710 msgr2=0x7f9bb80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:20.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.013+0000 7f9bbf37a640 1 -- 172.21.15.94:0/2899357415 shutdown_connections 2024-03-06T07:18:20.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.013+0000 7f9bbf37a640 1 -- 172.21.15.94:0/2899357415 wait complete. 2024-03-06T07:18:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 Processor -- start 2024-03-06T07:18:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 -- start start 2024-03-06T07:18:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb806c900 0x7f9bb80a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb80ab170 0x7f9bb81489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb8148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8156610 con 0x7f9bb806c900 2024-03-06T07:18:20.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8156490 con 0x7f9bb80c5990 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbf37a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb80c3af0 con 0x7f9bb80ab170 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbd116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb806c900 0x7f9bb80a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbd116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb806c900 0x7f9bb80a75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55982/0 (socket says 172.21.15.94:55982) 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbd116640 1 -- 172.21.15.94:0/972418610 learned_addr learned my addr 172.21.15.94:0/972418610 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbd917640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb8148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.014+0000 7f9bbc915640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb80ab170 0x7f9bb81489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbd116640 1 -- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb80ab170 msgr2=0x7f9bb81489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbd116640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb80ab170 0x7f9bb81489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbd116640 1 -- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 msgr2=0x7f9bb8148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbd116640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb8148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbd116640 1 -- 172.21.15.94:0/972418610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb80706a0 con 0x7f9bb806c900 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbc915640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb80ab170 0x7f9bb81489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbd917640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb8148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbd116640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb806c900 0x7f9bb80a75f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d270 tx=0x7f9bb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9badffb640 1 -- 172.21.15.94:0/972418610 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0045020 con 0x7f9bb806c900 2024-03-06T07:18:20.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9badffb640 1 -- 172.21.15.94:0/972418610 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb0009e30 con 0x7f9bb806c900 2024-03-06T07:18:20.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb8070980 con 0x7f9bb806c900 2024-03-06T07:18:20.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9badffb640 1 -- 172.21.15.94:0/972418610 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb000b0d0 con 0x7f9bb806c900 2024-03-06T07:18:20.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.015+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb8070f40 con 0x7f9bb806c900 2024-03-06T07:18:20.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.016+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb804f330 con 0x7f9bb806c900 2024-03-06T07:18:20.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.017+0000 7f9badffb640 1 -- 172.21.15.94:0/972418610 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bb0048020 con 0x7f9bb806c900 2024-03-06T07:18:20.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.017+0000 7f9badffb640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b9c06b4d0 0x7f9b9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.017+0000 7f9badffb640 1 -- 172.21.15.94:0/972418610 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9bb00aef70 con 0x7f9bb806c900 2024-03-06T07:18:20.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.017+0000 7f9bbc915640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b9c06b4d0 0x7f9b9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.018+0000 7f9bbc915640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b9c06b4d0 0x7f9b9c06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9bb814fda0 tx=0x7f9ba8007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.021+0000 7f9badffb640 1 -- 172.21.15.94:0/972418610 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bb004f030 con 0x7f9bb806c900 2024-03-06T07:18:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.192+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9bb8071590 con 0x7f9bb806c900 2024-03-06T07:18:20.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.193+0000 7f9badffb640 1 -- 172.21.15.94:0/972418610 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9bb007dbd0 con 0x7f9bb806c900 2024-03-06T07:18:20.195 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b9c06b4d0 msgr2=0x7f9b9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b9c06b4d0 0x7f9b9c06d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9bb814fda0 tx=0x7f9ba8007e80 comp rx=0 tx=0).stop 2024-03-06T07:18:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb806c900 msgr2=0x7f9bb80a75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb806c900 0x7f9bb80a75f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000d270 tx=0x7f9bb0004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 shutdown_connections 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b9c06b4d0 0x7f9b9c06d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bb80c5990 0x7f9bb8148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bb80ab170 0x7f9bb81489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 --2- 172.21.15.94:0/972418610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bb806c900 0x7f9bb80a75f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 >> 172.21.15.94:0/972418610 conn(0x7f9bb80c6710 msgr2=0x7f9bb80c4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 shutdown_connections 2024-03-06T07:18:20.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.195+0000 7f9bbf37a640 1 -- 172.21.15.94:0/972418610 wait complete. 2024-03-06T07:18:20.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:20.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.308+0000 7fc758f0c640 1 Processor -- start 2024-03-06T07:18:20.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.308+0000 7fc758f0c640 1 -- start start 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.309+0000 7fc758f0c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc75406d750 0x7fc7540732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.309+0000 7fc758f0c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc754156060 con 0x7fc75406db30 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.309+0000 7fc758f0c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7540a8430 con 0x7fc7540c3a70 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.309+0000 7fc758f0c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc754155b30 con 0x7fc75406d750 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.309+0000 7fc752d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc75406db30 0x7fc7540c1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59408/0 (socket says 172.21.15.94:59408) 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.309+0000 7fc752d9d640 1 -- 172.21.15.94:0/1097162158 learned_addr learned my addr 172.21.15.94:0/1097162158 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.309+0000 7fc75259c640 1 --2- 172.21.15.94:0/1097162158 >> v2:172.21.15.145:3300/0 conn(0x7fc75406d750 0x7fc7540732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.310+0000 7fc75259c640 1 -- 172.21.15.94:0/1097162158 >> v1:172.21.15.145:6789/0 conn(0x7fc7540c3a70 legacy=0x7fc754072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.310+0000 7fc75259c640 1 -- 172.21.15.94:0/1097162158 >> v1:172.21.15.94:6789/0 conn(0x7fc75406db30 legacy=0x7fc7540c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.310+0000 7fc75159a640 1 -- 172.21.15.94:0/1097162158 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357871438 0 0) 0x7fc754156060 con 0x7fc75406db30 2024-03-06T07:18:20.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.310+0000 7fc75259c640 1 -- 172.21.15.94:0/1097162158 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc754155d80 con 0x7fc75406d750 2024-03-06T07:18:20.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.310+0000 7fc75259c640 1 --2- 172.21.15.94:0/1097162158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406d750 0x7fc7540732a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097a0 tx=0x7fc74c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abcbd6c9f7d831cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc75159a640 1 -- 172.21.15.94:0/1097162158 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0315a0 con 0x7fc75406d750 2024-03-06T07:18:20.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc75159a640 1 -- 172.21.15.94:0/1097162158 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c037070 con 0x7fc75406d750 2024-03-06T07:18:20.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc75159a640 1 -- 172.21.15.94:0/1097162158 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c031960 con 0x7fc75406d750 2024-03-06T07:18:20.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc758f0c640 1 -- 172.21.15.94:0/1097162158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406d750 msgr2=0x7fc7540732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc758f0c640 1 --2- 172.21.15.94:0/1097162158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406d750 0x7fc7540732a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097a0 tx=0x7fc74c02b840 comp rx=0 tx=0).stop 2024-03-06T07:18:20.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc758f0c640 1 -- 172.21.15.94:0/1097162158 shutdown_connections 2024-03-06T07:18:20.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc758f0c640 1 --2- 172.21.15.94:0/1097162158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406d750 0x7fc7540732a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc758f0c640 1 -- 172.21.15.94:0/1097162158 >> 172.21.15.94:0/1097162158 conn(0x7fc7540c5560 msgr2=0x7fc7540c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:20.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc758f0c640 1 -- 172.21.15.94:0/1097162158 shutdown_connections 2024-03-06T07:18:20.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.311+0000 7fc758f0c640 1 -- 172.21.15.94:0/1097162158 wait complete. 2024-03-06T07:18:20.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.312+0000 7fc758f0c640 1 Processor -- start 2024-03-06T07:18:20.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.312+0000 7fc758f0c640 1 -- start start 2024-03-06T07:18:20.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc758f0c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc75406d750 0x7fc75414d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc758f0c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406db30 0x7fc7541eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc758f0c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7540c3a70 0x7fc7540acd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc758f0c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540c2720 con 0x7fc75406d750 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc758f0c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540c25a0 con 0x7fc75406db30 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc758f0c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540a7cd0 con 0x7fc7540c3a70 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc75259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc75406d750 0x7fc75414d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc75259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc75406d750 0x7fc75414d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55984/0 (socket says 172.21.15.94:55984) 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc75259c640 1 -- 172.21.15.94:0/825873487 learned_addr learned my addr 172.21.15.94:0/825873487 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc752d9d640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7540c3a70 0x7fc7540acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.313+0000 7fc751d9b640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406db30 0x7fc7541eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc75259c640 1 -- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7540c3a70 msgr2=0x7fc7540acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc75259c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7540c3a70 0x7fc7540acd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc75259c640 1 -- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406db30 msgr2=0x7fc7541eb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc75259c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406db30 0x7fc7541eb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc75259c640 1 -- 172.21.15.94:0/825873487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75414dac0 con 0x7fc75406d750 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc75259c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc75406d750 0x7fc75414d3c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097a0 tx=0x7fc74c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc742ffd640 1 -- 172.21.15.94:0/825873487 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c004230 con 0x7fc75406d750 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc742ffd640 1 -- 172.21.15.94:0/825873487 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c0043f0 con 0x7fc75406d750 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75414dda0 con 0x7fc75406d750 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc742ffd640 1 -- 172.21.15.94:0/825873487 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c032880 con 0x7fc75406d750 2024-03-06T07:18:20.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.314+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75414fc80 con 0x7fc75406d750 2024-03-06T07:18:20.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.315+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75404f330 con 0x7fc75406d750 2024-03-06T07:18:20.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.315+0000 7fc742ffd640 1 -- 172.21.15.94:0/825873487 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc74c032a20 con 0x7fc75406d750 2024-03-06T07:18:20.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.316+0000 7fc742ffd640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc73406b4d0 0x7fc73406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.316+0000 7fc742ffd640 1 -- 172.21.15.94:0/825873487 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc74c02f030 con 0x7fc75406d750 2024-03-06T07:18:20.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.316+0000 7fc751d9b640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc73406b4d0 0x7fc73406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.316+0000 7fc751d9b640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc73406b4d0 0x7fc73406d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7540c6b30 tx=0x7fc748009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.320+0000 7fc742ffd640 1 -- 172.21.15.94:0/825873487 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc74c041070 con 0x7fc75406d750 2024-03-06T07:18:20.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.492+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc7541544f0 con 0x7fc75406d750 2024-03-06T07:18:20.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.492+0000 7fc742ffd640 1 -- 172.21.15.94:0/825873487 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc74c07abf0 con 0x7fc75406d750 2024-03-06T07:18:20.495 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:20.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc73406b4d0 msgr2=0x7fc73406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc73406b4d0 0x7fc73406d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7540c6b30 tx=0x7fc748009040 comp rx=0 tx=0).stop 2024-03-06T07:18:20.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc75406d750 msgr2=0x7fc75414d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc75406d750 0x7fc75414d3c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097a0 tx=0x7fc74c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 shutdown_connections 2024-03-06T07:18:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc73406b4d0 0x7fc73406d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7540c3a70 0x7fc7540acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75406db30 0x7fc7541eb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 --2- 172.21.15.94:0/825873487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc75406d750 0x7fc75414d3c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 >> 172.21.15.94:0/825873487 conn(0x7fc7540c5560 msgr2=0x7fc754151370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 shutdown_connections 2024-03-06T07:18:20.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.494+0000 7fc758f0c640 1 -- 172.21.15.94:0/825873487 wait complete. 2024-03-06T07:18:20.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:18:20.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.606+0000 7fee7711e640 1 Processor -- start 2024-03-06T07:18:20.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.606+0000 7fee7711e640 1 -- start start 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee7711e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee701561a0 0x7fee701529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee7711e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee7014f110 0x7fee7014f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee7711e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee700a9990 con 0x7fee700c6ab0 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee7711e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee70155d10 con 0x7fee7014f110 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee7711e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee701556c0 con 0x7fee701561a0 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee756bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee7014f110 0x7fee7014f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee756bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee7014f110 0x7fee7014f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55992/0 (socket says 172.21.15.94:55992) 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee756bb640 1 -- 172.21.15.94:0/5825968 learned_addr learned my addr 172.21.15.94:0/5825968 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee6ffff640 1 --2- 172.21.15.94:0/5825968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee701561a0 0x7fee701529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee756bb640 1 -- 172.21.15.94:0/5825968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee701561a0 msgr2=0x7fee701529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:20.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee756bb640 1 --2- 172.21.15.94:0/5825968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee701561a0 0x7fee701529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee756bb640 1 -- 172.21.15.94:0/5825968 >> v1:172.21.15.145:6789/0 conn(0x7fee700c6ab0 legacy=0x7fee700a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee756bb640 1 -- 172.21.15.94:0/5825968 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee701559a0 con 0x7fee7014f110 2024-03-06T07:18:20.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.607+0000 7fee6f7fe640 1 -- 172.21.15.94:0/5825968 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271050996 0 0) 0x7fee700a9990 con 0x7fee700c6ab0 2024-03-06T07:18:20.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.608+0000 7fee756bb640 1 --2- 172.21.15.94:0/5825968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7014f110 0x7fee7014f570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fee68009e50 tx=0x7fee6802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34751fea1930b0bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.608+0000 7fee6f7fe640 1 -- 172.21.15.94:0/5825968 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6802ca70 con 0x7fee7014f110 2024-03-06T07:18:20.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.608+0000 7fee6f7fe640 1 -- 172.21.15.94:0/5825968 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6802cc10 con 0x7fee7014f110 2024-03-06T07:18:20.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.608+0000 7fee6f7fe640 1 -- 172.21.15.94:0/5825968 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee680343f0 con 0x7fee7014f110 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 -- 172.21.15.94:0/5825968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7014f110 msgr2=0x7fee7014f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 --2- 172.21.15.94:0/5825968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7014f110 0x7fee7014f570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fee68009e50 tx=0x7fee6802bed0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 -- 172.21.15.94:0/5825968 shutdown_connections 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 --2- 172.21.15.94:0/5825968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee7014f110 0x7fee7014f570 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 --2- 172.21.15.94:0/5825968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee701561a0 0x7fee701529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 -- 172.21.15.94:0/5825968 >> 172.21.15.94:0/5825968 conn(0x7fee7006b1a0 msgr2=0x7fee7006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 -- 172.21.15.94:0/5825968 shutdown_connections 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.609+0000 7fee7711e640 1 -- 172.21.15.94:0/5825968 wait complete. 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.610+0000 7fee7711e640 1 Processor -- start 2024-03-06T07:18:20.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.610+0000 7fee7711e640 1 -- start start 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee7711e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee700c6ab0 0x7fee70141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee7711e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee7014f110 0x7fee700c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee7711e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee701561a0 0x7fee701e8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee7711e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700aa600 con 0x7fee701561a0 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee7711e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700c2a90 con 0x7fee7014f110 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee7711e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee701ebd00 con 0x7fee700c6ab0 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee701561a0 0x7fee701e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee701561a0 0x7fee701e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56004/0 (socket says 172.21.15.94:56004) 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 -- 172.21.15.94:0/3494578793 learned_addr learned my addr 172.21.15.94:0/3494578793 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee6ffff640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee7014f110 0x7fee700c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee74eba640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee700c6ab0 0x7fee70141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 -- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee700c6ab0 msgr2=0x7fee70141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee700c6ab0 0x7fee70141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 -- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee7014f110 msgr2=0x7fee700c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee7014f110 0x7fee700c4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 -- 172.21.15.94:0/3494578793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee701ebf50 con 0x7fee701561a0 2024-03-06T07:18:20.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee6ffff640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee7014f110 0x7fee700c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee756bb640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee701561a0 0x7fee701e8790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fee68036040 tx=0x7fee68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee6d7fa640 1 -- 172.21.15.94:0/3494578793 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68043020 con 0x7fee701561a0 2024-03-06T07:18:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee6d7fa640 1 -- 172.21.15.94:0/3494578793 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6803e450 con 0x7fee701561a0 2024-03-06T07:18:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.611+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee701ec230 con 0x7fee701561a0 2024-03-06T07:18:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.612+0000 7fee6d7fa640 1 -- 172.21.15.94:0/3494578793 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68009510 con 0x7fee701561a0 2024-03-06T07:18:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.612+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee701421a0 con 0x7fee701561a0 2024-03-06T07:18:20.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.613+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7004f330 con 0x7fee701561a0 2024-03-06T07:18:20.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.613+0000 7fee6d7fa640 1 -- 172.21.15.94:0/3494578793 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee68046020 con 0x7fee701561a0 2024-03-06T07:18:20.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.613+0000 7fee6d7fa640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee5c06b4d0 0x7fee5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.613+0000 7fee6d7fa640 1 -- 172.21.15.94:0/3494578793 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fee680acae0 con 0x7fee701561a0 2024-03-06T07:18:20.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.613+0000 7fee74eba640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee5c06b4d0 0x7fee5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.614+0000 7fee74eba640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee5c06b4d0 0x7fee5c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee60000f60 tx=0x7fee60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.617+0000 7fee6d7fa640 1 -- 172.21.15.94:0/3494578793 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee6804d030 con 0x7fee701561a0 2024-03-06T07:18:20.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.802+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fee700c4a30 con 0x7fee701561a0 2024-03-06T07:18:20.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.802+0000 7fee6d7fa640 1 -- 172.21.15.94:0/3494578793 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee6807b180 con 0x7fee701561a0 2024-03-06T07:18:20.805 INFO:teuthology.orchestra.run.smithi094.stderr:listed 0 entries 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee5c06b4d0 msgr2=0x7fee5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee5c06b4d0 0x7fee5c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee60000f60 tx=0x7fee60009040 comp rx=0 tx=0).stop 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee701561a0 msgr2=0x7fee701e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee701561a0 0x7fee701e8790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fee68036040 tx=0x7fee68004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 shutdown_connections 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee5c06b4d0 0x7fee5c06d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee701561a0 0x7fee701e8790 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee7014f110 0x7fee700c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 --2- 172.21.15.94:0/3494578793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee700c6ab0 0x7fee70141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 >> 172.21.15.94:0/3494578793 conn(0x7fee7006b1a0 msgr2=0x7fee7006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:20.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 shutdown_connections 2024-03-06T07:18:20.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.804+0000 7fee7711e640 1 -- 172.21.15.94:0/3494578793 wait complete. 2024-03-06T07:18:20.817 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:18:20.818 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:20.833 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:20.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 Processor -- start 2024-03-06T07:18:20.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 -- start start 2024-03-06T07:18:20.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6ba80c88c0 0x7f6ba80c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6ba80c5880 0x7f6ba8158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba80c8dd0 con 0x7f6ba80c5880 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba80c73d0 con 0x7f6ba80c88c0 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.972+0000 7f6bada5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80be2e0 con 0x7f6ba814f360 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6ba80c5880 0x7f6ba8158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6ba80c5880 0x7f6ba8158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56010/0 (socket says 172.21.15.94:56010) 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 -- 172.21.15.94:0/2233181681 learned_addr learned my addr 172.21.15.94:0/2233181681 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba67fc640 1 --2- 172.21.15.94:0/2233181681 >> v2:172.21.15.145:3300/0 conn(0x7f6ba80c88c0 0x7f6ba80c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba6ffd640 1 --2- 172.21.15.94:0/2233181681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 -- 172.21.15.94:0/2233181681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 msgr2=0x7f6ba8155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 --2- 172.21.15.94:0/2233181681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 -- 172.21.15.94:0/2233181681 >> v2:172.21.15.145:3300/0 conn(0x7f6ba80c88c0 msgr2=0x7f6ba80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 --2- 172.21.15.94:0/2233181681 >> v2:172.21.15.145:3300/0 conn(0x7f6ba80c88c0 0x7f6ba80c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.973+0000 7f6ba77fe640 1 -- 172.21.15.94:0/2233181681 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80c7670 con 0x7f6ba80c5880 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6ba77fe640 1 --2- 172.21.15.94:0/2233181681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c5880 0x7f6ba8158b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80cbac0 tx=0x7f6ba002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e53166348b43be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6ba5ffb640 1 -- 172.21.15.94:0/2233181681 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba002c9e0 con 0x7f6ba80c5880 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6ba5ffb640 1 -- 172.21.15.94:0/2233181681 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba002cb80 con 0x7f6ba80c5880 2024-03-06T07:18:20.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6ba5ffb640 1 -- 172.21.15.94:0/2233181681 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba00343f0 con 0x7f6ba80c5880 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 -- 172.21.15.94:0/2233181681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c5880 msgr2=0x7f6ba8158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2233181681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c5880 0x7f6ba8158b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80cbac0 tx=0x7f6ba002be40 comp rx=0 tx=0).stop 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 -- 172.21.15.94:0/2233181681 shutdown_connections 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2233181681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c5880 0x7f6ba8158b60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2233181681 >> v2:172.21.15.145:3300/0 conn(0x7f6ba80c88c0 0x7f6ba80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2233181681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 -- 172.21.15.94:0/2233181681 >> 172.21.15.94:0/2233181681 conn(0x7f6ba80ac570 msgr2=0x7f6ba80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 -- 172.21.15.94:0/2233181681 shutdown_connections 2024-03-06T07:18:20.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.974+0000 7f6bada5c640 1 -- 172.21.15.94:0/2233181681 wait complete. 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.975+0000 7f6bada5c640 1 Processor -- start 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6bada5c640 1 -- start start 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6bada5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ba80c5880 0x7f6ba81ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6bada5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c88c0 0x7f6ba80ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6bada5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6bada5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba81590a0 con 0x7f6ba80c88c0 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6bada5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80c7c30 con 0x7f6ba80c5880 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6bada5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8159220 con 0x7f6ba814f360 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6ba67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c88c0 0x7f6ba80ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6ba67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c88c0 0x7f6ba80ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56016/0 (socket says 172.21.15.94:56016) 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6ba67fc640 1 -- 172.21.15.94:0/2224788113 learned_addr learned my addr 172.21.15.94:0/2224788113 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:20.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6ba6ffd640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ba80c5880 0x7f6ba81ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.976+0000 7f6ba77fe640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6ba67fc640 1 -- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 msgr2=0x7f6ba8156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6ba67fc640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6ba67fc640 1 -- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ba80c5880 msgr2=0x7f6ba81ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6ba67fc640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ba80c5880 0x7f6ba81ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6ba67fc640 1 -- 172.21.15.94:0/2224788113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80b0560 con 0x7f6ba80c88c0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6ba67fc640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c88c0 0x7f6ba80ab7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00e860 tx=0x7f6b9c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6b8ffff640 1 -- 172.21.15.94:0/2224788113 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c017070 con 0x7f6ba80c88c0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6b8ffff640 1 -- 172.21.15.94:0/2224788113 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9c00c940 con 0x7f6ba80c88c0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba80ac2d0 con 0x7f6ba80c88c0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6b8ffff640 1 -- 172.21.15.94:0/2224788113 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00cd10 con 0x7f6ba80c88c0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.977+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba8157d60 con 0x7f6ba80c88c0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.978+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba804f330 con 0x7f6ba80c88c0 2024-03-06T07:18:20.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.978+0000 7f6b8ffff640 1 -- 172.21.15.94:0/2224788113 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b9c01a070 con 0x7f6ba80c88c0 2024-03-06T07:18:20.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.979+0000 7f6b8ffff640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6b9406b4d0 0x7f6b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:20.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.979+0000 7f6b8ffff640 1 -- 172.21.15.94:0/2224788113 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6b9c08a420 con 0x7f6ba80c88c0 2024-03-06T07:18:20.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.979+0000 7f6ba6ffd640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6b9406b4d0 0x7f6b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:20.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.979+0000 7f6ba6ffd640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6b9406b4d0 0x7f6b9406d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80ac540 tx=0x7f6b98009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:20.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:20.983+0000 7f6b8ffff640 1 -- 172.21.15.94:0/2224788113 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b9c0589a0 con 0x7f6ba80c88c0 2024-03-06T07:18:21.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.154+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ba8158b00 con 0x7f6ba80c88c0 2024-03-06T07:18:21.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.155+0000 7f6b8ffff640 1 -- 172.21.15.94:0/2224788113 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f6b9c05ca40 con 0x7f6ba80c88c0 2024-03-06T07:18:21.158 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:21.158 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6b9406b4d0 msgr2=0x7f6b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6b9406b4d0 0x7f6b9406d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80ac540 tx=0x7f6b98009190 comp rx=0 tx=0).stop 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c88c0 msgr2=0x7f6ba80ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c88c0 0x7f6ba80ab7d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00e860 tx=0x7f6b9c00ed20 comp rx=0 tx=0).stop 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 shutdown_connections 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6b9406b4d0 0x7f6b9406d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ba814f360 0x7f6ba8156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ba80c88c0 0x7f6ba80ab7d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 --2- 172.21.15.94:0/2224788113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ba80c5880 0x7f6ba81ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.157+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 >> 172.21.15.94:0/2224788113 conn(0x7f6ba80ac570 msgr2=0x7f6ba80ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:21.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.158+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 shutdown_connections 2024-03-06T07:18:21.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.158+0000 7f6bada5c640 1 -- 172.21.15.94:0/2224788113 wait complete. 2024-03-06T07:18:21.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:21.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:21.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.272+0000 7fa3818ed640 1 Processor -- start 2024-03-06T07:18:21.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.272+0000 7fa3818ed640 1 -- start start 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa3818ed640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa37c0b0070 0x7fa37c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa3818ed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa37c0c5380 0x7fa37c0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa3818ed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0b0450 con 0x7fa37c06ca70 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa3818ed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c06c020 con 0x7fa37c0c5380 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa3818ed640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c06cf20 con 0x7fa37c0b0070 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa37c0c5380 0x7fa37c0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa37c0c5380 0x7fa37c0c5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56030/0 (socket says 172.21.15.94:56030) 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa373fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa37c06ca70 0x7fa37c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59410/0 (socket says 172.21.15.94:59410) 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37b7fe640 1 -- 172.21.15.94:0/2937851974 learned_addr learned my addr 172.21.15.94:0/2937851974 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:21.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37affd640 1 --2- 172.21.15.94:0/2937851974 >> v2:172.21.15.145:3300/0 conn(0x7fa37c0b0070 0x7fa37c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37b7fe640 1 -- 172.21.15.94:0/2937851974 >> v2:172.21.15.145:3300/0 conn(0x7fa37c0b0070 msgr2=0x7fa37c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37b7fe640 1 --2- 172.21.15.94:0/2937851974 >> v2:172.21.15.145:3300/0 conn(0x7fa37c0b0070 0x7fa37c155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37b7fe640 1 -- 172.21.15.94:0/2937851974 >> v1:172.21.15.94:6789/0 conn(0x7fa37c06ca70 legacy=0x7fa37c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37b7fe640 1 -- 172.21.15.94:0/2937851974 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0c77f0 con 0x7fa37c0c5380 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.273+0000 7fa37a7fc640 1 -- 172.21.15.94:0/2937851974 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899470347 0 0) 0x7fa364001110 con 0x7fa37c06ca70 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.274+0000 7fa37b7fe640 1 --2- 172.21.15.94:0/2937851974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c5790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0aa3d0 tx=0x7fa37402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a43fd6bc263ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.274+0000 7fa37a7fc640 1 -- 172.21.15.94:0/2937851974 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37402ca80 con 0x7fa37c0c5380 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.274+0000 7fa37a7fc640 1 -- 172.21.15.94:0/2937851974 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37402cc20 con 0x7fa37c0c5380 2024-03-06T07:18:21.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.274+0000 7fa37a7fc640 1 -- 172.21.15.94:0/2937851974 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740343f0 con 0x7fa37c0c5380 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.274+0000 7fa3818ed640 1 -- 172.21.15.94:0/2937851974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 msgr2=0x7fa37c0c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.274+0000 7fa3818ed640 1 --2- 172.21.15.94:0/2937851974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c5790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0aa3d0 tx=0x7fa37402bee0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.275+0000 7fa3818ed640 1 -- 172.21.15.94:0/2937851974 shutdown_connections 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.275+0000 7fa3818ed640 1 --2- 172.21.15.94:0/2937851974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c5790 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.275+0000 7fa3818ed640 1 --2- 172.21.15.94:0/2937851974 >> v2:172.21.15.145:3300/0 conn(0x7fa37c0b0070 0x7fa37c155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.275+0000 7fa3818ed640 1 -- 172.21.15.94:0/2937851974 >> 172.21.15.94:0/2937851974 conn(0x7fa37c0ac590 msgr2=0x7fa37c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.275+0000 7fa3818ed640 1 -- 172.21.15.94:0/2937851974 shutdown_connections 2024-03-06T07:18:21.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.275+0000 7fa3818ed640 1 -- 172.21.15.94:0/2937851974 wait complete. 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 Processor -- start 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 -- start start 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa37c06ca70 0x7fa37c0be510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa37c0b0070 0x7fa37c1545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c06c550 con 0x7fa37c0c5380 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c06c270 con 0x7fa37c0b0070 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa3818ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c06da80 con 0x7fa37c06ca70 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa37b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa37b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56046/0 (socket says 172.21.15.94:56046) 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa37b7fe640 1 -- 172.21.15.94:0/103795452 learned_addr learned my addr 172.21.15.94:0/103795452 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa37affd640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa37c06ca70 0x7fa37c0be510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.276+0000 7fa373fff640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa37c0b0070 0x7fa37c1545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa37b7fe640 1 -- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa37c06ca70 msgr2=0x7fa37c0be510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa37b7fe640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa37c06ca70 0x7fa37c0be510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa37b7fe640 1 -- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa37c0b0070 msgr2=0x7fa37c1545f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa37b7fe640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa37c0b0070 0x7fa37c1545f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa37b7fe640 1 -- 172.21.15.94:0/103795452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0c42b0 con 0x7fa37c0c5380 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa373fff640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa37c0b0070 0x7fa37c1545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa37b7fe640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c35e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa374036040 tx=0x7fa374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa378ff9640 1 -- 172.21.15.94:0/103795452 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374043020 con 0x7fa37c0c5380 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa378ff9640 1 -- 172.21.15.94:0/103795452 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37403e450 con 0x7fa37c0c5380 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c0c4590 con 0x7fa37c0c5380 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa378ff9640 1 -- 172.21.15.94:0/103795452 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374009510 con 0x7fa37c0c5380 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.277+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c0c1c70 con 0x7fa37c0c5380 2024-03-06T07:18:21.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.278+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c04f330 con 0x7fa37c0c5380 2024-03-06T07:18:21.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.279+0000 7fa378ff9640 1 -- 172.21.15.94:0/103795452 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa374046020 con 0x7fa37c0c5380 2024-03-06T07:18:21.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.279+0000 7fa378ff9640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa35c06b4d0 0x7fa35c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.279+0000 7fa378ff9640 1 -- 172.21.15.94:0/103795452 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa3740acac0 con 0x7fa37c0c5380 2024-03-06T07:18:21.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.279+0000 7fa37affd640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa35c06b4d0 0x7fa35c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.280+0000 7fa37affd640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa35c06b4d0 0x7fa35c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009b40 tx=0x7fa36c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.283+0000 7fa378ff9640 1 -- 172.21.15.94:0/103795452 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa37404d030 con 0x7fa37c0c5380 2024-03-06T07:18:21.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.453+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa37c06d860 con 0x7fa37c0c5380 2024-03-06T07:18:21.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.454+0000 7fa378ff9640 1 -- 172.21.15.94:0/103795452 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa37407b160 con 0x7fa37c0c5380 2024-03-06T07:18:21.456 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:21.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa35c06b4d0 msgr2=0x7fa35c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa35c06b4d0 0x7fa35c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009b40 tx=0x7fa36c009190 comp rx=0 tx=0).stop 2024-03-06T07:18:21.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 msgr2=0x7fa37c0c35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c35e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa374036040 tx=0x7fa374004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:21.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 shutdown_connections 2024-03-06T07:18:21.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa35c06b4d0 0x7fa35c06d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa37c0c5380 0x7fa37c0c35e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa37c0b0070 0x7fa37c1545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 --2- 172.21.15.94:0/103795452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa37c06ca70 0x7fa37c0be510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 >> 172.21.15.94:0/103795452 conn(0x7fa37c0ac590 msgr2=0x7fa37c0ae5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:21.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 shutdown_connections 2024-03-06T07:18:21.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.456+0000 7fa3818ed640 1 -- 172.21.15.94:0/103795452 wait complete. 2024-03-06T07:18:21.469 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:21.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.571+0000 7ff7c02e1640 1 Processor -- start 2024-03-06T07:18:21.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.571+0000 7ff7c02e1640 1 -- start start 2024-03-06T07:18:21.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7c02e1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b81539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7c02e1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff7b80c4740 0x7ff7b80701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7c02e1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7b80c69a0 0x7ff7b8072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7c02e1640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b8153f00 con 0x7ff7b80c69a0 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7c02e1640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b80c62a0 con 0x7ff7b80c4740 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7c02e1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b80706e0 con 0x7ff7b80a8290 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7b80c69a0 0x7ff7b8072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7b80c69a0 0x7ff7b8072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56054/0 (socket says 172.21.15.94:56054) 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 -- 172.21.15.94:0/473261480 learned_addr learned my addr 172.21.15.94:0/473261480 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7bd87c640 1 --2- 172.21.15.94:0/473261480 >> v2:172.21.15.145:3300/0 conn(0x7ff7b80c4740 0x7ff7b80701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be07d640 1 --2- 172.21.15.94:0/473261480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b81539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 -- 172.21.15.94:0/473261480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 msgr2=0x7ff7b81539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 --2- 172.21.15.94:0/473261480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b81539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 -- 172.21.15.94:0/473261480 >> v2:172.21.15.145:3300/0 conn(0x7ff7b80c4740 msgr2=0x7ff7b80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 --2- 172.21.15.94:0/473261480 >> v2:172.21.15.145:3300/0 conn(0x7ff7b80c4740 0x7ff7b80701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.572+0000 7ff7be87e640 1 -- 172.21.15.94:0/473261480 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b80c4c00 con 0x7ff7b80c69a0 2024-03-06T07:18:21.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.573+0000 7ff7be87e640 1 --2- 172.21.15.94:0/473261480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c69a0 0x7ff7b8072680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80c6620 tx=0x7ff7a802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77db10a83edacd65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.573+0000 7ff7bd07b640 1 -- 172.21.15.94:0/473261480 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a802c950 con 0x7ff7b80c69a0 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.573+0000 7ff7bd07b640 1 -- 172.21.15.94:0/473261480 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a802caf0 con 0x7ff7b80c69a0 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.573+0000 7ff7bd07b640 1 -- 172.21.15.94:0/473261480 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80343f0 con 0x7ff7b80c69a0 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 -- 172.21.15.94:0/473261480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c69a0 msgr2=0x7ff7b8072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/473261480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c69a0 0x7ff7b8072680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80c6620 tx=0x7ff7a802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 -- 172.21.15.94:0/473261480 shutdown_connections 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/473261480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c69a0 0x7ff7b8072680 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/473261480 >> v2:172.21.15.145:3300/0 conn(0x7ff7b80c4740 0x7ff7b80701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/473261480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b81539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 -- 172.21.15.94:0/473261480 >> 172.21.15.94:0/473261480 conn(0x7ff7b80c5560 msgr2=0x7ff7b80c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 -- 172.21.15.94:0/473261480 shutdown_connections 2024-03-06T07:18:21.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.574+0000 7ff7c02e1640 1 -- 172.21.15.94:0/473261480 wait complete. 2024-03-06T07:18:21.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.575+0000 7ff7c02e1640 1 Processor -- start 2024-03-06T07:18:21.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.575+0000 7ff7c02e1640 1 -- start start 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b806d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c4740 0x7ff7b81ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7b80c69a0 0x7ff7b81eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b806fc80 con 0x7ff7b80c4740 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b806f9a0 con 0x7ff7b80c69a0 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b80c6e10 con 0x7ff7b80a8290 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c4740 0x7ff7b81ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c4740 0x7ff7b81ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56060/0 (socket says 172.21.15.94:56060) 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 -- 172.21.15.94:0/2743303796 learned_addr learned my addr 172.21.15.94:0/2743303796 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7be87e640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7b80c69a0 0x7ff7b81eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7be07d640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b806d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 -- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 msgr2=0x7ff7b806d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b806d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 -- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7b80c69a0 msgr2=0x7ff7b81eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7b80c69a0 0x7ff7b81eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 -- 172.21.15.94:0/2743303796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b8150ee0 con 0x7ff7b80c4740 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7bd87c640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c4740 0x7ff7b81ee890 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00e9e0 tx=0x7ff7ac00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7b6ffd640 1 -- 172.21.15.94:0/2743303796 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac017070 con 0x7ff7b80c4740 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7b6ffd640 1 -- 172.21.15.94:0/2743303796 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac00c9d0 con 0x7ff7b80c4740 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7b6ffd640 1 -- 172.21.15.94:0/2743303796 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac00cdc0 con 0x7ff7b80c4740 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b81eb9c0 con 0x7ff7b80c4740 2024-03-06T07:18:21.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.576+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b806da00 con 0x7ff7b80c4740 2024-03-06T07:18:21.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.577+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b8075300 con 0x7ff7b80c4740 2024-03-06T07:18:21.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.578+0000 7ff7b6ffd640 1 -- 172.21.15.94:0/2743303796 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7ac01a070 con 0x7ff7b80c4740 2024-03-06T07:18:21.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.578+0000 7ff7b6ffd640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7a406b4d0 0x7ff7a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.578+0000 7ff7b6ffd640 1 -- 172.21.15.94:0/2743303796 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff7ac08a3b0 con 0x7ff7b80c4740 2024-03-06T07:18:21.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.578+0000 7ff7be07d640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7a406b4d0 0x7ff7a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.579+0000 7ff7be07d640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7a406b4d0 0x7ff7a406d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009b40 tx=0x7ff7b0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.582+0000 7ff7b6ffd640 1 -- 172.21.15.94:0/2743303796 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7b8075300 con 0x7ff7b80c4740 2024-03-06T07:18:21.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.754+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff7b804f330 con 0x7ff7b80c4740 2024-03-06T07:18:21.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.754+0000 7ff7b6ffd640 1 -- 172.21.15.94:0/2743303796 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff7ac058930 con 0x7ff7b80c4740 2024-03-06T07:18:21.756 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7a406b4d0 msgr2=0x7ff7a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7a406b4d0 0x7ff7a406d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009b40 tx=0x7ff7b0009190 comp rx=0 tx=0).stop 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c4740 msgr2=0x7ff7b81ee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c4740 0x7ff7b81ee890 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00e9e0 tx=0x7ff7ac00eea0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 shutdown_connections 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7a406b4d0 0x7ff7a406d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7b80c69a0 0x7ff7b81eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7b80c4740 0x7ff7b81ee890 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 --2- 172.21.15.94:0/2743303796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7b80a8290 0x7ff7b806d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.756+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 >> 172.21.15.94:0/2743303796 conn(0x7ff7b80c5560 msgr2=0x7ff7b80aa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.757+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 shutdown_connections 2024-03-06T07:18:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.757+0000 7ff7c02e1640 1 -- 172.21.15.94:0/2743303796 wait complete. 2024-03-06T07:18:21.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:21.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.871+0000 7fed4bb7a640 1 Processor -- start 2024-03-06T07:18:21.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.871+0000 7fed4bb7a640 1 -- start start 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed4bb7a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed440c5880 0x7fed440c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed4bb7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed44073190 0x7fed44156530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed4bb7a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed44072600 con 0x7fed440c5c60 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed4bb7a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed440723b0 con 0x7fed440c5880 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed4bb7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed440728e0 con 0x7fed44073190 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed49916640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed440c5880 0x7fed440c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed49916640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed440c5880 0x7fed440c6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56064/0 (socket says 172.21.15.94:56064) 2024-03-06T07:18:21.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed49916640 1 -- 172.21.15.94:0/1003852901 learned_addr learned my addr 172.21.15.94:0/1003852901 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed4a117640 1 --2- 172.21.15.94:0/1003852901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed44073190 0x7fed44156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed49916640 1 -- 172.21.15.94:0/1003852901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed44073190 msgr2=0x7fed44156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed49916640 1 --2- 172.21.15.94:0/1003852901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed44073190 0x7fed44156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed49916640 1 -- 172.21.15.94:0/1003852901 >> v1:172.21.15.145:6789/0 conn(0x7fed440c5c60 legacy=0x7fed440c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.872+0000 7fed49916640 1 -- 172.21.15.94:0/1003852901 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed44153af0 con 0x7fed440c5880 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.873+0000 7fed49916640 1 --2- 172.21.15.94:0/1003852901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed440c5880 0x7fed440c6ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fed40009870 tx=0x7fed4002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f53b790452c115 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.873+0000 7fed48914640 1 -- 172.21.15.94:0/1003852901 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed400317c0 con 0x7fed440c5880 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.873+0000 7fed48914640 1 -- 172.21.15.94:0/1003852901 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed40037070 con 0x7fed440c5880 2024-03-06T07:18:21.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.873+0000 7fed48914640 1 -- 172.21.15.94:0/1003852901 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40031b80 con 0x7fed440c5880 2024-03-06T07:18:21.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 -- 172.21.15.94:0/1003852901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed440c5880 msgr2=0x7fed440c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/1003852901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed440c5880 0x7fed440c6ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fed40009870 tx=0x7fed4002b860 comp rx=0 tx=0).stop 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 -- 172.21.15.94:0/1003852901 shutdown_connections 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/1003852901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed44073190 0x7fed44156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/1003852901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed440c5880 0x7fed440c6ca0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 -- 172.21.15.94:0/1003852901 >> 172.21.15.94:0/1003852901 conn(0x7fed440aa630 msgr2=0x7fed440aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 -- 172.21.15.94:0/1003852901 shutdown_connections 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.874+0000 7fed4bb7a640 1 -- 172.21.15.94:0/1003852901 wait complete. 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.875+0000 7fed4bb7a640 1 Processor -- start 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.875+0000 7fed4bb7a640 1 -- start start 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed44073190 0x7fed44154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed440c5880 0x7fed4414b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed440c5c60 0x7fed44156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed440c78f0 con 0x7fed44073190 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed440c6780 con 0x7fed440c5880 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed44156a70 con 0x7fed440c5c60 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed44073190 0x7fed44154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed44073190 0x7fed44154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56080/0 (socket says 172.21.15.94:56080) 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 -- 172.21.15.94:0/2020607038 learned_addr learned my addr 172.21.15.94:0/2020607038 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4a117640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed440c5c60 0x7fed44156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49115640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed440c5880 0x7fed4414b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 -- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed440c5c60 msgr2=0x7fed44156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed440c5c60 0x7fed44156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 -- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed440c5880 msgr2=0x7fed4414b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed440c5880 0x7fed4414b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 -- 172.21.15.94:0/2020607038 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed44152be0 con 0x7fed44073190 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed49916640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed44073190 0x7fed44154290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed3a7fc640 1 -- 172.21.15.94:0/2020607038 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40032650 con 0x7fed44073190 2024-03-06T07:18:21.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed3a7fc640 1 -- 172.21.15.94:0/2020607038 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed400327f0 con 0x7fed44073190 2024-03-06T07:18:21.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed3a7fc640 1 -- 172.21.15.94:0/2020607038 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40032be0 con 0x7fed44073190 2024-03-06T07:18:21.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed441e85c0 con 0x7fed44073190 2024-03-06T07:18:21.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.876+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed440ac270 con 0x7fed44073190 2024-03-06T07:18:21.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.877+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4404f330 con 0x7fed44073190 2024-03-06T07:18:21.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.878+0000 7fed3a7fc640 1 -- 172.21.15.94:0/2020607038 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed4004c060 con 0x7fed44073190 2024-03-06T07:18:21.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.878+0000 7fed3a7fc640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed2806b4d0 0x7fed2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:21.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.878+0000 7fed3a7fc640 1 -- 172.21.15.94:0/2020607038 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fed400abaa0 con 0x7fed44073190 2024-03-06T07:18:21.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.878+0000 7fed49115640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed2806b4d0 0x7fed2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:21.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.879+0000 7fed49115640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed2806b4d0 0x7fed2806d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed44152ee0 tx=0x7fed34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:21.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:21.882+0000 7fed3a7fc640 1 -- 172.21.15.94:0/2020607038 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed44152be0 con 0x7fed44073190 2024-03-06T07:18:22.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.058+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed44152910 con 0x7fed44073190 2024-03-06T07:18:22.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.058+0000 7fed3a7fc640 1 -- 172.21.15.94:0/2020607038 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed40040070 con 0x7fed44073190 2024-03-06T07:18:22.060 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:22.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed2806b4d0 msgr2=0x7fed2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed2806b4d0 0x7fed2806d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed44152ee0 tx=0x7fed34009040 comp rx=0 tx=0).stop 2024-03-06T07:18:22.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed44073190 msgr2=0x7fed44154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed44073190 0x7fed44154290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed400046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 shutdown_connections 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed2806b4d0 0x7fed2806d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed440c5c60 0x7fed44156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed440c5880 0x7fed4414b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 --2- 172.21.15.94:0/2020607038 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed44073190 0x7fed44154290 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 >> 172.21.15.94:0/2020607038 conn(0x7fed440aa630 msgr2=0x7fed44073e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 shutdown_connections 2024-03-06T07:18:22.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.060+0000 7fed4bb7a640 1 -- 172.21.15.94:0/2020607038 wait complete. 2024-03-06T07:18:22.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:22.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a6e1b6640 1 Processor -- start 2024-03-06T07:18:22.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a6e1b6640 1 -- start start 2024-03-06T07:18:22.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a6e1b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3a6814f580 0x7f3a68153420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a6e1b6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3a68155520 0x7f3a68155950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a6e1b6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a680c2d80 con 0x7f3a6806dca0 2024-03-06T07:18:22.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a6e1b6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a680c2b30 con 0x7f3a68155520 2024-03-06T07:18:22.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a6e1b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a680c3d70 con 0x7f3a6814f580 2024-03-06T07:18:22.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a67fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3a68155520 0x7f3a68155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a67fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3a68155520 0x7f3a68155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56084/0 (socket says 172.21.15.94:56084) 2024-03-06T07:18:22.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a67fff640 1 -- 172.21.15.94:0/3706679673 learned_addr learned my addr 172.21.15.94:0/3706679673 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:22.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a677fe640 1 --2- 172.21.15.94:0/3706679673 >> v2:172.21.15.145:3300/0 conn(0x7f3a6814f580 0x7f3a68153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.174+0000 7f3a66ffd640 1 --1- 172.21.15.94:0/3706679673 >> v1:172.21.15.145:6789/0 conn(0x7f3a6806dca0 0x7f3a6806f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:58262/0 (socket says 172.21.15.94:58262) 2024-03-06T07:18:22.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.175+0000 7f3a67fff640 1 -- 172.21.15.94:0/3706679673 >> v2:172.21.15.145:3300/0 conn(0x7f3a6814f580 msgr2=0x7f3a68153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:22.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.175+0000 7f3a67fff640 1 --2- 172.21.15.94:0/3706679673 >> v2:172.21.15.145:3300/0 conn(0x7f3a6814f580 0x7f3a68153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.175+0000 7f3a67fff640 1 -- 172.21.15.94:0/3706679673 >> v1:172.21.15.145:6789/0 conn(0x7f3a6806dca0 legacy=0x7f3a6806f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.175+0000 7f3a67fff640 1 -- 172.21.15.94:0/3706679673 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a680c3a00 con 0x7f3a68155520 2024-03-06T07:18:22.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.175+0000 7f3a67fff640 1 --2- 172.21.15.94:0/3706679673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a68155950 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a680aa1a0 tx=0x7f3a6002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f0ca3d2f1a6409 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:22.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a667fc640 1 -- 172.21.15.94:0/3706679673 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6002c9f0 con 0x7f3a68155520 2024-03-06T07:18:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a667fc640 1 -- 172.21.15.94:0/3706679673 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6002cb90 con 0x7f3a68155520 2024-03-06T07:18:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a667fc640 1 -- 172.21.15.94:0/3706679673 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a600343f0 con 0x7f3a68155520 2024-03-06T07:18:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/3706679673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 msgr2=0x7f3a68155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/3706679673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a68155950 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3a680aa1a0 tx=0x7f3a6002be50 comp rx=0 tx=0).stop 2024-03-06T07:18:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/3706679673 shutdown_connections 2024-03-06T07:18:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/3706679673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a68155950 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/3706679673 >> v2:172.21.15.145:3300/0 conn(0x7f3a6814f580 0x7f3a68153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/3706679673 >> 172.21.15.94:0/3706679673 conn(0x7f3a680a94d0 msgr2=0x7f3a680a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:22.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/3706679673 shutdown_connections 2024-03-06T07:18:22.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.176+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/3706679673 wait complete. 2024-03-06T07:18:22.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.177+0000 7f3a6e1b6640 1 Processor -- start 2024-03-06T07:18:22.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.177+0000 7f3a6e1b6640 1 -- start start 2024-03-06T07:18:22.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a6e1b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a6806dca0 0x7f3a6814f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a6e1b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a6814f580 0x7f3a681ddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a6e1b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a6814dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a6e1b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a680c3060 con 0x7f3a68155520 2024-03-06T07:18:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a6e1b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6814f960 con 0x7f3a6814f580 2024-03-06T07:18:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a6e1b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a68155e90 con 0x7f3a6806dca0 2024-03-06T07:18:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a67fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a6814dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a67fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a6814dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56092/0 (socket says 172.21.15.94:56092) 2024-03-06T07:18:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a67fff640 1 -- 172.21.15.94:0/1124597541 learned_addr learned my addr 172.21.15.94:0/1124597541 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a66ffd640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a6814f580 0x7f3a681ddbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.178+0000 7f3a677fe640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a6806dca0 0x7f3a6814f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a67fff640 1 -- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a6806dca0 msgr2=0x7f3a6814f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a67fff640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a6806dca0 0x7f3a6814f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a67fff640 1 -- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a6814f580 msgr2=0x7f3a681ddbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a67fff640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a6814f580 0x7f3a681ddbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a67fff640 1 -- 172.21.15.94:0/1124597541 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a681e75a0 con 0x7f3a68155520 2024-03-06T07:18:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a67fff640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a6814dd70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3a6002c460 tx=0x7f3a60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a4ffff640 1 -- 172.21.15.94:0/1124597541 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60044020 con 0x7f3a68155520 2024-03-06T07:18:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a4ffff640 1 -- 172.21.15.94:0/1124597541 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a60040470 con 0x7f3a68155520 2024-03-06T07:18:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a681e7840 con 0x7f3a68155520 2024-03-06T07:18:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a4ffff640 1 -- 172.21.15.94:0/1124597541 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60009510 con 0x7f3a68155520 2024-03-06T07:18:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.179+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a681db940 con 0x7f3a68155520 2024-03-06T07:18:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.180+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6804f330 con 0x7f3a68155520 2024-03-06T07:18:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.180+0000 7f3a4ffff640 1 -- 172.21.15.94:0/1124597541 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a60047020 con 0x7f3a68155520 2024-03-06T07:18:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.181+0000 7f3a4ffff640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5406b4d0 0x7f3a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.181+0000 7f3a4ffff640 1 -- 172.21.15.94:0/1124597541 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3a600acb90 con 0x7f3a68155520 2024-03-06T07:18:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.181+0000 7f3a677fe640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5406b4d0 0x7f3a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.181+0000 7f3a677fe640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5406b4d0 0x7f3a5406d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a580133e0 tx=0x7f3a5800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.185+0000 7f3a4ffff640 1 -- 172.21.15.94:0/1124597541 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a6004e030 con 0x7f3a68155520 2024-03-06T07:18:22.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.357+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3a680c1e00 con 0x7f3a68155520 2024-03-06T07:18:22.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.357+0000 7f3a4ffff640 1 -- 172.21.15.94:0/1124597541 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3a6007b110 con 0x7f3a68155520 2024-03-06T07:18:22.359 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:22.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5406b4d0 msgr2=0x7f3a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5406b4d0 0x7f3a5406d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a580133e0 tx=0x7f3a5800d040 comp rx=0 tx=0).stop 2024-03-06T07:18:22.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 msgr2=0x7f3a6814dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a6814dd70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3a6002c460 tx=0x7f3a60004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:22.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 shutdown_connections 2024-03-06T07:18:22.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5406b4d0 0x7f3a5406d990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68155520 0x7f3a6814dd70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a6814f580 0x7f3a681ddbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 --2- 172.21.15.94:0/1124597541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a6806dca0 0x7f3a6814f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 >> 172.21.15.94:0/1124597541 conn(0x7f3a680a94d0 msgr2=0x7f3a680a79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:22.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 shutdown_connections 2024-03-06T07:18:22.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.359+0000 7f3a6e1b6640 1 -- 172.21.15.94:0/1124597541 wait complete. 2024-03-06T07:18:22.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:18:22.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.472+0000 7fde07464640 1 Processor -- start 2024-03-06T07:18:22.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.473+0000 7fde07464640 1 -- start start 2024-03-06T07:18:22.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.473+0000 7fde07464640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fde0006f550 0x7fde001593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.473+0000 7fde07464640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde00072cd0 0x7fde0006fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.473+0000 7fde07464640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde000733f0 con 0x7fde00152720 2024-03-06T07:18:22.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.473+0000 7fde07464640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde00155ca0 con 0x7fde00072cd0 2024-03-06T07:18:22.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.473+0000 7fde07464640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde001561d0 con 0x7fde0006f550 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.473+0000 7fde049ff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fde0006f550 0x7fde001593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fde05200640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fde00152720 0x7fde000c77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:58264/0 (socket says 172.21.15.94:58264) 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fde05200640 1 -- 172.21.15.94:0/1902892189 learned_addr learned my addr 172.21.15.94:0/1902892189 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fde05a01640 1 --2- 172.21.15.94:0/1902892189 >> v2:172.21.15.94:3300/0 conn(0x7fde00072cd0 0x7fde0006fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fde049ff640 1 -- 172.21.15.94:0/1902892189 >> v2:172.21.15.94:3300/0 conn(0x7fde00072cd0 msgr2=0x7fde0006fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fde049ff640 1 --2- 172.21.15.94:0/1902892189 >> v2:172.21.15.94:3300/0 conn(0x7fde00072cd0 0x7fde0006fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fde049ff640 1 -- 172.21.15.94:0/1902892189 >> v1:172.21.15.145:6789/0 conn(0x7fde00152720 legacy=0x7fde000c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fddf3fff640 1 -- 172.21.15.94:0/1902892189 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994533393 0 0) 0x7fde000733f0 con 0x7fde00152720 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.474+0000 7fde049ff640 1 -- 172.21.15.94:0/1902892189 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00155ef0 con 0x7fde0006f550 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.475+0000 7fde049ff640 1 --2- 172.21.15.94:0/1902892189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 0x7fde001593c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde0006bd00 tx=0x7fddf402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1b2b6b8912ca26b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.475+0000 7fddf3fff640 1 -- 172.21.15.94:0/1902892189 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf40315a0 con 0x7fde0006f550 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.475+0000 7fddf3fff640 1 -- 172.21.15.94:0/1902892189 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf4037070 con 0x7fde0006f550 2024-03-06T07:18:22.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.475+0000 7fddf3fff640 1 -- 172.21.15.94:0/1902892189 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4031960 con 0x7fde0006f550 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.475+0000 7fde07464640 1 -- 172.21.15.94:0/1902892189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 msgr2=0x7fde001593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.475+0000 7fde07464640 1 --2- 172.21.15.94:0/1902892189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 0x7fde001593c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde0006bd00 tx=0x7fddf402b840 comp rx=0 tx=0).stop 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.476+0000 7fde07464640 1 -- 172.21.15.94:0/1902892189 shutdown_connections 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.476+0000 7fde07464640 1 --2- 172.21.15.94:0/1902892189 >> v2:172.21.15.94:3300/0 conn(0x7fde00072cd0 0x7fde0006fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.476+0000 7fde07464640 1 --2- 172.21.15.94:0/1902892189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 0x7fde001593c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.476+0000 7fde07464640 1 -- 172.21.15.94:0/1902892189 >> 172.21.15.94:0/1902892189 conn(0x7fde0006ea40 msgr2=0x7fde0006afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.476+0000 7fde07464640 1 -- 172.21.15.94:0/1902892189 shutdown_connections 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.476+0000 7fde07464640 1 -- 172.21.15.94:0/1902892189 wait complete. 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 Processor -- start 2024-03-06T07:18:22.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 -- start start 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 0x7fde001ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde00072cd0 0x7fde000a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde00152720 0x7fde001e7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde00158ea0 con 0x7fde00072cd0 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde00158bc0 con 0x7fde0006f550 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde07464640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde000c7ef0 con 0x7fde00152720 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde049ff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde00072cd0 0x7fde000a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde049ff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde00072cd0 0x7fde000a98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56116/0 (socket says 172.21.15.94:56116) 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.477+0000 7fde049ff640 1 -- 172.21.15.94:0/654856024 learned_addr learned my addr 172.21.15.94:0/654856024 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde05200640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 0x7fde001ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde05a01640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde00152720 0x7fde001e7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde049ff640 1 -- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde00152720 msgr2=0x7fde001e7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde049ff640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde00152720 0x7fde001e7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde049ff640 1 -- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 msgr2=0x7fde001ea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde049ff640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 0x7fde001ea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde049ff640 1 -- 172.21.15.94:0/654856024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00068fa0 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde049ff640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde00072cd0 0x7fde000a98c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fddf4000c00 tx=0x7fddf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fddf1ffb640 1 -- 172.21.15.94:0/654856024 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4004230 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fddf1ffb640 1 -- 172.21.15.94:0/654856024 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf40043d0 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde001edfe0 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fddf1ffb640 1 -- 172.21.15.94:0/654856024 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4003850 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.478+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde001ee5a0 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.479+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0004f330 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.480+0000 7fddf1ffb640 1 -- 172.21.15.94:0/654856024 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddf40039f0 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.480+0000 7fddf1ffb640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.480+0000 7fddf1ffb640 1 -- 172.21.15.94:0/654856024 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fddf402f030 con 0x7fde00072cd0 2024-03-06T07:18:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.480+0000 7fde05200640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:22.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.481+0000 7fde05200640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0012e0 tx=0x7fddfc001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.484+0000 7fddf1ffb640 1 -- 172.21.15.94:0/654856024 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddf4041070 con 0x7fde00072cd0 2024-03-06T07:18:22.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.666+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fde00072ab0 con 0x7fde00072cd0 2024-03-06T07:18:22.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.666+0000 7fddf1ffb640 1 -- 172.21.15.94:0/654856024 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fddf407aa60 con 0x7fde00072cd0 2024-03-06T07:18:22.668 INFO:teuthology.orchestra.run.smithi094.stderr:listed 0 entries 2024-03-06T07:18:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 msgr2=0x7fdde406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0012e0 tx=0x7fddfc001630 comp rx=0 tx=0).stop 2024-03-06T07:18:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde00072cd0 msgr2=0x7fde000a98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde00072cd0 0x7fde000a98c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fddf4000c00 tx=0x7fddf40046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 shutdown_connections 2024-03-06T07:18:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde00152720 0x7fde001e7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde00072cd0 0x7fde000a98c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 --2- 172.21.15.94:0/654856024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006f550 0x7fde001ea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 >> 172.21.15.94:0/654856024 conn(0x7fde0006ea40 msgr2=0x7fde0014bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 shutdown_connections 2024-03-06T07:18:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:22.668+0000 7fde07464640 1 -- 172.21.15.94:0/654856024 wait complete. 2024-03-06T07:18:22.681 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi094.front.sepia.ceph.com... 2024-03-06T07:18:22.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:22.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-06T07:18:22.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi094.stderr:ceph-fuse[43009]: fuse finished with error 0 and tester_r 0 2024-03-06T07:18:22.709 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:18:28.810 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:18:28.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:28.811 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-06T07:18:28.839 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:28.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.987+0000 7f4e9fc78640 1 Processor -- start 2024-03-06T07:18:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.987+0000 7f4e9fc78640 1 -- start start 2024-03-06T07:18:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9fc78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e98150920 0x7f4e980c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9fc78640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e98067340 con 0x7f4e980c2e70 2024-03-06T07:18:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9fc78640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e98150d00 con 0x7f4e9806c720 2024-03-06T07:18:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9fc78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980c2010 con 0x7f4e98150920 2024-03-06T07:18:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9e215640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4e980c2e70 0x7f4e9815dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41372/0 (socket says 172.21.15.94:41372) 2024-03-06T07:18:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9e215640 1 -- 172.21.15.94:0/2873032970 learned_addr learned my addr 172.21.15.94:0/2873032970 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9da14640 1 --2- 172.21.15.94:0/2873032970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e98150920 0x7f4e980c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9ca12640 1 -- 172.21.15.94:0/2873032970 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437577862 0 0) 0x7f4e98067340 con 0x7f4e980c2e70 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9ca12640 1 -- 172.21.15.94:0/2873032970 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e84003520 con 0x7f4e980c2e70 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9ca12640 1 -- 172.21.15.94:0/2873032970 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217592776 0 0) 0x7f4e98150d00 con 0x7f4e9806c720 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9ca12640 1 -- 172.21.15.94:0/2873032970 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e98067340 con 0x7f4e9806c720 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9da14640 1 -- 172.21.15.94:0/2873032970 >> v1:172.21.15.145:6789/0 conn(0x7f4e9806c720 legacy=0x7f4e98071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9da14640 1 -- 172.21.15.94:0/2873032970 >> v1:172.21.15.94:6789/0 conn(0x7f4e980c2e70 legacy=0x7f4e9815dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.988+0000 7f4e9da14640 1 -- 172.21.15.94:0/2873032970 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98067060 con 0x7f4e98150920 2024-03-06T07:18:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.989+0000 7f4e9da14640 1 --2- 172.21.15.94:0/2873032970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e98150920 0x7f4e980c2930 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4e940097d0 tx=0x7f4e9402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83396a2d4ff00a27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.989+0000 7f4e9ca12640 1 -- 172.21.15.94:0/2873032970 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94035020 con 0x7f4e98150920 2024-03-06T07:18:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.989+0000 7f4e9ca12640 1 -- 172.21.15.94:0/2873032970 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94031800 con 0x7f4e98150920 2024-03-06T07:18:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.989+0000 7f4e9ca12640 1 -- 172.21.15.94:0/2873032970 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94031bd0 con 0x7f4e98150920 2024-03-06T07:18:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.990+0000 7f4e9fc78640 1 -- 172.21.15.94:0/2873032970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e98150920 msgr2=0x7f4e980c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.990+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/2873032970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e98150920 0x7f4e980c2930 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4e940097d0 tx=0x7f4e9402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:18:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.990+0000 7f4e9fc78640 1 -- 172.21.15.94:0/2873032970 shutdown_connections 2024-03-06T07:18:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.990+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/2873032970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e98150920 0x7f4e980c2930 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.990+0000 7f4e9fc78640 1 -- 172.21.15.94:0/2873032970 >> 172.21.15.94:0/2873032970 conn(0x7f4e980c70b0 msgr2=0x7f4e980c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.990+0000 7f4e9fc78640 1 -- 172.21.15.94:0/2873032970 shutdown_connections 2024-03-06T07:18:28.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.990+0000 7f4e9fc78640 1 -- 172.21.15.94:0/2873032970 wait complete. 2024-03-06T07:18:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.991+0000 7f4e9fc78640 1 Processor -- start 2024-03-06T07:18:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.991+0000 7f4e9fc78640 1 -- start start 2024-03-06T07:18:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9fc78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e9806c720 0x7f4e98152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9fc78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e980c2e70 0x7f4e9814be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9fc78640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98150920 0x7f4e981e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9fc78640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98073210 con 0x7f4e980c2e70 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9fc78640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98073090 con 0x7f4e98150920 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9fc78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980c1b40 con 0x7f4e9806c720 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e980c2e70 0x7f4e9814be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e980c2e70 0x7f4e9814be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40674/0 (socket says 172.21.15.94:40674) 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 -- 172.21.15.94:0/4094262121 learned_addr learned my addr 172.21.15.94:0/4094262121 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9e215640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98150920 0x7f4e981e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9da14640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e9806c720 0x7f4e98152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 -- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e9806c720 msgr2=0x7f4e98152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e9806c720 0x7f4e98152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 -- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98150920 msgr2=0x7f4e981e9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98150920 0x7f4e981e9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 -- 172.21.15.94:0/4094262121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9814c970 con 0x7f4e980c2e70 2024-03-06T07:18:28.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9d213640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e980c2e70 0x7f4e9814be40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800c460 tx=0x7f4e8800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e8e7fc640 1 -- 172.21.15.94:0/4094262121 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88017070 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e8e7fc640 1 -- 172.21.15.94:0/4094262121 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e880052f0 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e981ed450 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.992+0000 7f4e8e7fc640 1 -- 172.21.15.94:0/4094262121 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88003830 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.993+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e98154d90 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.994+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e98069460 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.994+0000 7f4e8e7fc640 1 -- 172.21.15.94:0/4094262121 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e8801a070 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.994+0000 7f4e8e7fc640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e7c06b4d0 0x7f4e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.995+0000 7f4e8e7fc640 1 -- 172.21.15.94:0/4094262121 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4e880897f0 con 0x7f4e980c2e70 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.995+0000 7f4e9da14640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e7c06b4d0 0x7f4e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.995+0000 7f4e9da14640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e7c06b4d0 0x7f4e7c06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e94036000 tx=0x7f4e94008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:29.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:28.999+0000 7f4e8e7fc640 1 -- 172.21.15.94:0/4094262121 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e98069460 con 0x7f4e980c2e70 2024-03-06T07:18:29.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.171+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e9804f330 con 0x7f4e980c2e70 2024-03-06T07:18:29.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.172+0000 7f4e8e7fc640 1 -- 172.21.15.94:0/4094262121 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f4e88057d70 con 0x7f4e980c2e70 2024-03-06T07:18:29.175 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:29.175 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:29.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.174+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e7c06b4d0 msgr2=0x7f4e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.174+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e7c06b4d0 0x7f4e7c06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e94036000 tx=0x7f4e94008030 comp rx=0 tx=0).stop 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.174+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e980c2e70 msgr2=0x7f4e9814be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.174+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e980c2e70 0x7f4e9814be40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800c460 tx=0x7f4e8800c920 comp rx=0 tx=0).stop 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.174+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 shutdown_connections 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.175+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e7c06b4d0 0x7f4e7c06d990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.175+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e98150920 0x7f4e981e9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.175+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e980c2e70 0x7f4e9814be40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.175+0000 7f4e9fc78640 1 --2- 172.21.15.94:0/4094262121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e9806c720 0x7f4e98152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.175+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 >> 172.21.15.94:0/4094262121 conn(0x7f4e980c70b0 msgr2=0x7f4e98148c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.175+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 shutdown_connections 2024-03-06T07:18:29.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.175+0000 7f4e9fc78640 1 -- 172.21.15.94:0/4094262121 wait complete. 2024-03-06T07:18:29.189 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.287+0000 7f486c888640 1 Processor -- start 2024-03-06T07:18:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.287+0000 7f486c888640 1 -- start start 2024-03-06T07:18:29.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.287+0000 7f486c888640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f486814f790 0x7f48681529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486c888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48680c1aa0 0x7f48680c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486c888640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48680c28d0 con 0x7f48680c5c60 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486c888640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48680c2680 con 0x7f486814f790 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486c888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48680c2bb0 con 0x7f48680c1aa0 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f4866d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f48680c5c60 0x7f4868155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43718/0 (socket says 172.21.15.94:43718) 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f4866d9d640 1 -- 172.21.15.94:0/3386503921 learned_addr learned my addr 172.21.15.94:0/3386503921 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486659c640 1 --2- 172.21.15.94:0/3386503921 >> v2:172.21.15.145:3300/0 conn(0x7f486814f790 0x7f48681529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f485ffff640 1 --2- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48680c1aa0 0x7f48680c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486659c640 1 -- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48680c1aa0 msgr2=0x7f48680c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486659c640 1 --2- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48680c1aa0 0x7f48680c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486659c640 1 -- 172.21.15.94:0/3386503921 >> v1:172.21.15.145:6789/0 conn(0x7f48680c5c60 legacy=0x7f4868155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.288+0000 7f486659c640 1 -- 172.21.15.94:0/3386503921 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680c2e90 con 0x7f486814f790 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f486659c640 1 --2- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f486814f790 0x7f48681529d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f48580097a0 tx=0x7f485802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcfd09da9d33bff2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f4865d9b640 1 -- 172.21.15.94:0/3386503921 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858035020 con 0x7f486814f790 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f4865d9b640 1 -- 172.21.15.94:0/3386503921 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4858031830 con 0x7f486814f790 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f4865d9b640 1 -- 172.21.15.94:0/3386503921 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4858031c00 con 0x7f486814f790 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f486c888640 1 -- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f486814f790 msgr2=0x7f48681529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f486c888640 1 --2- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f486814f790 0x7f48681529d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f48580097a0 tx=0x7f485802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f486c888640 1 -- 172.21.15.94:0/3386503921 shutdown_connections 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f486c888640 1 --2- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48680c1aa0 0x7f48680c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f486c888640 1 --2- 172.21.15.94:0/3386503921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f486814f790 0x7f48681529d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.289+0000 7f486c888640 1 -- 172.21.15.94:0/3386503921 >> 172.21.15.94:0/3386503921 conn(0x7f48680c6710 msgr2=0x7f48680c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:29.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.290+0000 7f486c888640 1 -- 172.21.15.94:0/3386503921 shutdown_connections 2024-03-06T07:18:29.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.290+0000 7f486c888640 1 -- 172.21.15.94:0/3386503921 wait complete. 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 Processor -- start 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 -- start start 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48680c1aa0 0x7f48680bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48680c5c60 0x7f486814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f486814f790 0x7f486814ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486806cc00 con 0x7f48680c1aa0 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48680c0d60 con 0x7f48680c5c60 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486c888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486806cd80 con 0x7f486814f790 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.291+0000 7f486659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48680c1aa0 0x7f48680bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48680c1aa0 0x7f48680bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40690/0 (socket says 172.21.15.94:40690) 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 -- 172.21.15.94:0/113196224 learned_addr learned my addr 172.21.15.94:0/113196224 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f485ffff640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48680c5c60 0x7f486814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f4866d9d640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f486814f790 0x7f486814ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 -- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f486814f790 msgr2=0x7f486814ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f486814f790 0x7f486814ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 -- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48680c5c60 msgr2=0x7f486814e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48680c5c60 0x7f486814e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 -- 172.21.15.94:0/113196224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48680c05e0 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486659c640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48680c1aa0 0x7f48680bfdb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f48580097a0 tx=0x7f48580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f485effd640 1 -- 172.21.15.94:0/113196224 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485803a970 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f485effd640 1 -- 172.21.15.94:0/113196224 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485803ab10 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4868072f50 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f485effd640 1 -- 172.21.15.94:0/113196224 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580443f0 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.292+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48681eb750 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.293+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48681ebb30 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.293+0000 7f485effd640 1 -- 172.21.15.94:0/113196224 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f485804b060 con 0x7f48680c1aa0 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.294+0000 7f485effd640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f484806b4d0 0x7f484806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.294+0000 7f485effd640 1 -- 172.21.15.94:0/113196224 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f48580abef0 con 0x7f48680c1aa0 2024-03-06T07:18:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.294+0000 7f485ffff640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f484806b4d0 0x7f484806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.294+0000 7f485ffff640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f484806b4d0 0x7f484806d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4868156e80 tx=0x7f4850009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:29.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.298+0000 7f485effd640 1 -- 172.21.15.94:0/113196224 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48681ebb30 con 0x7f48680c1aa0 2024-03-06T07:18:29.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.470+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f486804f330 con 0x7f48680c1aa0 2024-03-06T07:18:29.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.471+0000 7f485effd640 1 -- 172.21.15.94:0/113196224 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f48680c05e0 con 0x7f48680c1aa0 2024-03-06T07:18:29.473 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:29.473 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f484806b4d0 msgr2=0x7f484806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f484806b4d0 0x7f484806d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4868156e80 tx=0x7f4850009190 comp rx=0 tx=0).stop 2024-03-06T07:18:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48680c1aa0 msgr2=0x7f48680bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48680c1aa0 0x7f48680bfdb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f48580097a0 tx=0x7f48580046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 shutdown_connections 2024-03-06T07:18:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f484806b4d0 0x7f484806d990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f486814f790 0x7f486814ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48680c5c60 0x7f486814e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 --2- 172.21.15.94:0/113196224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48680c1aa0 0x7f48680bfdb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 >> 172.21.15.94:0/113196224 conn(0x7f48680c6710 msgr2=0x7f48680a6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:29.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 shutdown_connections 2024-03-06T07:18:29.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.473+0000 7f486c888640 1 -- 172.21.15.94:0/113196224 wait complete. 2024-03-06T07:18:29.486 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:18:29.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.589+0000 7f9dcad02640 1 Processor -- start 2024-03-06T07:18:29.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dcad02640 1 -- start start 2024-03-06T07:18:29.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dcad02640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9dc41528d0 0x7f9dc40724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dcad02640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc4155220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dcad02640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc4152cb0 con 0x7f9dc40c67f0 2024-03-06T07:18:29.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dcad02640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc40c38b0 con 0x7f9dc41528d0 2024-03-06T07:18:29.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dcad02640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc41594f0 con 0x7f9dc4072a10 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dc8a9e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9dc41528d0 0x7f9dc40724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dc929f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9dc40c67f0 0x7f9dc4155760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41376/0 (socket says 172.21.15.94:41376) 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dc8a9e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9dc41528d0 0x7f9dc40724d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40700/0 (socket says 172.21.15.94:40700) 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dc929f640 1 -- 172.21.15.94:0/2727687572 learned_addr learned my addr 172.21.15.94:0/2727687572 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.590+0000 7f9dc3fff640 1 --2- 172.21.15.94:0/2727687572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.591+0000 7f9dc8a9e640 1 -- 172.21.15.94:0/2727687572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 msgr2=0x7f9dc4155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.591+0000 7f9dc37fe640 1 -- 172.21.15.94:0/2727687572 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235361307 0 0) 0x7f9dc4152cb0 con 0x7f9dc40c67f0 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.591+0000 7f9dc8a9e640 1 --2- 172.21.15.94:0/2727687572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc4155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.591+0000 7f9dc8a9e640 1 -- 172.21.15.94:0/2727687572 >> v1:172.21.15.94:6789/0 conn(0x7f9dc40c67f0 legacy=0x7f9dc4155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.591+0000 7f9dc8a9e640 1 -- 172.21.15.94:0/2727687572 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc40a7f70 con 0x7f9dc41528d0 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.591+0000 7f9dc3fff640 1 --2- 172.21.15.94:0/2727687572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc4155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dc8a9e640 1 --2- 172.21.15.94:0/2727687572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc40724d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0097a0 tx=0x7f9dac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b51850b9cebfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dc37fe640 1 -- 172.21.15.94:0/2727687572 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac034070 con 0x7f9dc41528d0 2024-03-06T07:18:29.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dc37fe640 1 -- 172.21.15.94:0/2727687572 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac030610 con 0x7f9dc41528d0 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dc37fe640 1 -- 172.21.15.94:0/2727687572 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac0309e0 con 0x7f9dc41528d0 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 -- 172.21.15.94:0/2727687572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 msgr2=0x7f9dc40724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 --2- 172.21.15.94:0/2727687572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc40724d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0097a0 tx=0x7f9dac02b960 comp rx=0 tx=0).stop 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 -- 172.21.15.94:0/2727687572 shutdown_connections 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 --2- 172.21.15.94:0/2727687572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc4155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 --2- 172.21.15.94:0/2727687572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc40724d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 -- 172.21.15.94:0/2727687572 >> 172.21.15.94:0/2727687572 conn(0x7f9dc40aa6a0 msgr2=0x7f9dc40aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 -- 172.21.15.94:0/2727687572 shutdown_connections 2024-03-06T07:18:29.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.592+0000 7f9dcad02640 1 -- 172.21.15.94:0/2727687572 wait complete. 2024-03-06T07:18:29.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.593+0000 7f9dcad02640 1 Processor -- start 2024-03-06T07:18:29.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.593+0000 7f9dcad02640 1 -- start start 2024-03-06T07:18:29.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dcad02640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc40acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dcad02640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dc40c67f0 0x7f9dc4153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dcad02640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc4145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dcad02640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc41591a0 con 0x7f9dc41528d0 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dcad02640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40a8250 con 0x7f9dc40c67f0 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dcad02640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40c4ef0 con 0x7f9dc4072a10 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc4145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc4145040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40704/0 (socket says 172.21.15.94:40704) 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 -- 172.21.15.94:0/236166051 learned_addr learned my addr 172.21.15.94:0/236166051 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc8a9e640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc40acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc3fff640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dc40c67f0 0x7f9dc4153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 -- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 msgr2=0x7f9dc40acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc40acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 -- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dc40c67f0 msgr2=0x7f9dc4153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dc40c67f0 0x7f9dc4153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:29.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.594+0000 7f9dc929f640 1 -- 172.21.15.94:0/236166051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc40abb40 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.595+0000 7f9dc929f640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc4145040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c780 tx=0x7f9dbc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.595+0000 7f9dc17fa640 1 -- 172.21.15.94:0/236166051 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc018070 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.595+0000 7f9dc17fa640 1 -- 172.21.15.94:0/236166051 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc00a640 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.595+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc40abe20 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.595+0000 7f9dc17fa640 1 -- 172.21.15.94:0/236166051 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc00aab0 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.595+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc41539a0 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.596+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc40c4310 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.596+0000 7f9dc17fa640 1 -- 172.21.15.94:0/236166051 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dbc01b070 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.597+0000 7f9dc17fa640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9da406b500 0x7f9da406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.597+0000 7f9dc8a9e640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9da406b500 0x7f9da406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.597+0000 7f9dc17fa640 1 -- 172.21.15.94:0/236166051 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9dbc0899f0 con 0x7f9dc41528d0 2024-03-06T07:18:29.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.597+0000 7f9dc8a9e640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9da406b500 0x7f9da406d9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9dac036000 tx=0x7f9dac008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:29.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.601+0000 7f9dc17fa640 1 -- 172.21.15.94:0/236166051 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dc40c4310 con 0x7f9dc41528d0 2024-03-06T07:18:29.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:29.811+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9dc406c270 con 0x7f9dc41528d0 2024-03-06T07:18:30.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.710+0000 7f9dc17fa640 1 -- 172.21.15.94:0/236166051 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9dc406c270 con 0x7f9dc41528d0 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.713+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9da406b500 msgr2=0x7f9da406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.713+0000 7f9dcad02640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9da406b500 0x7f9da406d9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9dac036000 tx=0x7f9dac008030 comp rx=0 tx=0).stop 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.713+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 msgr2=0x7f9dc4145040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.713+0000 7f9dcad02640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc4145040 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c780 tx=0x7f9dbc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 shutdown_connections 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9da406b500 0x7f9da406d9c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dc41528d0 0x7f9dc4145040 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dc40c67f0 0x7f9dc4153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:30.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 --2- 172.21.15.94:0/236166051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dc4072a10 0x7f9dc40acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:30.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 >> 172.21.15.94:0/236166051 conn(0x7f9dc40aa6a0 msgr2=0x7f9dc406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:30.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 shutdown_connections 2024-03-06T07:18:30.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.714+0000 7f9dcad02640 1 -- 172.21.15.94:0/236166051 wait complete. 2024-03-06T07:18:30.729 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-06T07:18:30.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:18:30.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.835+0000 7f187a3ac640 1 Processor -- start 2024-03-06T07:18:30.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.835+0000 7f187a3ac640 1 -- start start 2024-03-06T07:18:30.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f187a3ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18740c3a70 0x7f18740732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:30.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f187a3ac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1874156060 con 0x7f187406db30 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f187a3ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18740a8430 con 0x7f187406d750 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f187a3ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1874155b30 con 0x7f18740c3a70 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f18737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18740c3a70 0x7f18740732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f1878949640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f187406db30 0x7f18740c1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41390/0 (socket says 172.21.15.94:41390) 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f18737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18740c3a70 0x7f18740732a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40710/0 (socket says 172.21.15.94:40710) 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.836+0000 7f1878949640 1 -- 172.21.15.94:0/428460355 learned_addr learned my addr 172.21.15.94:0/428460355 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.837+0000 7f18737fe640 1 -- 172.21.15.94:0/428460355 >> v1:172.21.15.145:6789/0 conn(0x7f187406d750 legacy=0x7f1874072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.837+0000 7f18737fe640 1 -- 172.21.15.94:0/428460355 >> v1:172.21.15.94:6789/0 conn(0x7f187406db30 legacy=0x7f18740c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.837+0000 7f1872ffd640 1 -- 172.21.15.94:0/428460355 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014023707 0 0) 0x7f18740a8430 con 0x7f187406d750 2024-03-06T07:18:30.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.837+0000 7f18737fe640 1 -- 172.21.15.94:0/428460355 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874155d80 con 0x7f18740c3a70 2024-03-06T07:18:30.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.837+0000 7f18737fe640 1 --2- 172.21.15.94:0/428460355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740732a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f18680097a0 tx=0x7f186802b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53c46491b6662c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:30.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.837+0000 7f1872ffd640 1 -- 172.21.15.94:0/428460355 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868037070 con 0x7f18740c3a70 2024-03-06T07:18:30.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f1872ffd640 1 -- 172.21.15.94:0/428460355 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1868030710 con 0x7f18740c3a70 2024-03-06T07:18:30.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f1872ffd640 1 -- 172.21.15.94:0/428460355 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868030ae0 con 0x7f18740c3a70 2024-03-06T07:18:30.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f187a3ac640 1 -- 172.21.15.94:0/428460355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 msgr2=0x7f18740732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:30.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f187a3ac640 1 --2- 172.21.15.94:0/428460355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740732a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f18680097a0 tx=0x7f186802b680 comp rx=0 tx=0).stop 2024-03-06T07:18:30.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f187a3ac640 1 -- 172.21.15.94:0/428460355 shutdown_connections 2024-03-06T07:18:30.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f187a3ac640 1 --2- 172.21.15.94:0/428460355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740732a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:30.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f187a3ac640 1 -- 172.21.15.94:0/428460355 >> 172.21.15.94:0/428460355 conn(0x7f18740c5560 msgr2=0x7f18740c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:30.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f187a3ac640 1 -- 172.21.15.94:0/428460355 shutdown_connections 2024-03-06T07:18:30.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.838+0000 7f187a3ac640 1 -- 172.21.15.94:0/428460355 wait complete. 2024-03-06T07:18:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.839+0000 7f187a3ac640 1 Processor -- start 2024-03-06T07:18:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.839+0000 7f187a3ac640 1 -- start start 2024-03-06T07:18:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f187a3ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f187406d750 0x7f187414aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f187a3ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f187406db30 0x7f18740abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f187a3ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f187a3ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740c2720 con 0x7f18740c3a70 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f187a3ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740c25a0 con 0x7f187406db30 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f187a3ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18740a7cd0 con 0x7f187406d750 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40718/0 (socket says 172.21.15.94:40718) 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 -- 172.21.15.94:0/1690338974 learned_addr learned my addr 172.21.15.94:0/1690338974 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f18737fe640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f187406db30 0x7f18740abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1873fff640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f187406d750 0x7f187414aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 -- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f187406d750 msgr2=0x7f187414aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f187406d750 0x7f187414aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 -- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f187406db30 msgr2=0x7f18740abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f187406db30 0x7f18740abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 -- 172.21.15.94:0/1690338974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18741564c0 con 0x7f18740c3a70 2024-03-06T07:18:30.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.840+0000 7f1878949640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740c8540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f186c00caf0 tx=0x7f186c002090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:30.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.841+0000 7f1870ff9640 1 -- 172.21.15.94:0/1690338974 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c018070 con 0x7f18740c3a70 2024-03-06T07:18:30.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.841+0000 7f1870ff9640 1 -- 172.21.15.94:0/1690338974 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c00a700 con 0x7f18740c3a70 2024-03-06T07:18:30.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.841+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187414b210 con 0x7f18740c3a70 2024-03-06T07:18:30.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.841+0000 7f1870ff9640 1 -- 172.21.15.94:0/1690338974 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c00aaf0 con 0x7f18740c3a70 2024-03-06T07:18:30.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.841+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18741eb030 con 0x7f18740c3a70 2024-03-06T07:18:30.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.842+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187404f330 con 0x7f18740c3a70 2024-03-06T07:18:30.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.842+0000 7f1870ff9640 1 -- 172.21.15.94:0/1690338974 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f186c01b070 con 0x7f18740c3a70 2024-03-06T07:18:30.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.842+0000 7f1870ff9640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f186006b4d0 0x7f186006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:30.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.843+0000 7f1870ff9640 1 -- 172.21.15.94:0/1690338974 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f186c089d10 con 0x7f18740c3a70 2024-03-06T07:18:30.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.843+0000 7f1873fff640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f186006b4d0 0x7f186006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:30.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.843+0000 7f1873fff640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f186006b4d0 0x7f186006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1864000f60 tx=0x7f1864009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:30.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:30.847+0000 7f1870ff9640 1 -- 172.21.15.94:0/1690338974 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f186c058170 con 0x7f18740c3a70 2024-03-06T07:18:31.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.054+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f18740c2fc0 con 0x7f18740c3a70 2024-03-06T07:18:31.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.714+0000 7f1870ff9640 1 -- 172.21.15.94:0/1690338974 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f186c05c210 con 0x7f18740c3a70 2024-03-06T07:18:31.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f186006b4d0 msgr2=0x7f186006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:31.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f186006b4d0 0x7f186006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1864000f60 tx=0x7f1864009040 comp rx=0 tx=0).stop 2024-03-06T07:18:31.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 msgr2=0x7f18740c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740c8540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f186c00caf0 tx=0x7f186c002090 comp rx=0 tx=0).stop 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 shutdown_connections 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f186006b4d0 0x7f186006d990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18740c3a70 0x7f18740c8540 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f187406db30 0x7f18740abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 --2- 172.21.15.94:0/1690338974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f187406d750 0x7f187414aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 >> 172.21.15.94:0/1690338974 conn(0x7f18740c5560 msgr2=0x7f18740c1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 shutdown_connections 2024-03-06T07:18:31.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.716+0000 7f187a3ac640 1 -- 172.21.15.94:0/1690338974 wait complete. 2024-03-06T07:18:31.730 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:18:31.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 Processor -- start 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 -- start start 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa17414f360 0x7fa174155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa1740c88c0 0x7fa1740c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1740c5880 0x7fa174158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1740c8dd0 con 0x7fa1740c5880 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1740c73d0 con 0x7fa1740c88c0 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.830+0000 7fa1792c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740be2e0 con 0x7fa17414f360 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1740c5880 0x7fa174158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1740c5880 0x7fa174158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40730/0 (socket says 172.21.15.94:40730) 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 -- 172.21.15.94:0/2036619531 learned_addr learned my addr 172.21.15.94:0/2036619531 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa172ffd640 1 --2- 172.21.15.94:0/2036619531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa17414f360 0x7fa174155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1727fc640 1 --2- 172.21.15.94:0/2036619531 >> v2:172.21.15.145:3300/0 conn(0x7fa1740c88c0 0x7fa1740c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 -- 172.21.15.94:0/2036619531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa17414f360 msgr2=0x7fa174155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 --2- 172.21.15.94:0/2036619531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa17414f360 0x7fa174155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 -- 172.21.15.94:0/2036619531 >> v2:172.21.15.145:3300/0 conn(0x7fa1740c88c0 msgr2=0x7fa1740c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 --2- 172.21.15.94:0/2036619531 >> v2:172.21.15.145:3300/0 conn(0x7fa1740c88c0 0x7fa1740c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.831+0000 7fa1737fe640 1 -- 172.21.15.94:0/2036619531 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1740c7670 con 0x7fa1740c5880 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1737fe640 1 --2- 172.21.15.94:0/2036619531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1740c5880 0x7fa174158b60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa1740cbac0 tx=0x7fa16c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2773e051b3740b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa171ffb640 1 -- 172.21.15.94:0/2036619531 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c02c9e0 con 0x7fa1740c5880 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa171ffb640 1 -- 172.21.15.94:0/2036619531 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c02cb80 con 0x7fa1740c5880 2024-03-06T07:18:31.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa171ffb640 1 -- 172.21.15.94:0/2036619531 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c0343f0 con 0x7fa1740c5880 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 -- 172.21.15.94:0/2036619531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1740c5880 msgr2=0x7fa174158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 --2- 172.21.15.94:0/2036619531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1740c5880 0x7fa174158b60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa1740cbac0 tx=0x7fa16c02be40 comp rx=0 tx=0).stop 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 -- 172.21.15.94:0/2036619531 shutdown_connections 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 --2- 172.21.15.94:0/2036619531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1740c5880 0x7fa174158b60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 --2- 172.21.15.94:0/2036619531 >> v2:172.21.15.145:3300/0 conn(0x7fa1740c88c0 0x7fa1740c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 --2- 172.21.15.94:0/2036619531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa17414f360 0x7fa174155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 -- 172.21.15.94:0/2036619531 >> 172.21.15.94:0/2036619531 conn(0x7fa1740ac570 msgr2=0x7fa1740ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 -- 172.21.15.94:0/2036619531 shutdown_connections 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.832+0000 7fa1792c3640 1 -- 172.21.15.94:0/2036619531 wait complete. 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 Processor -- start 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 -- start start 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1740c5880 0x7fa1741ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1740c88c0 0x7fa1740ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa17414f360 0x7fa174156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1741590a0 con 0x7fa17414f360 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740c7c30 con 0x7fa1740c5880 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1792c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa174159220 con 0x7fa1740c88c0 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa17414f360 0x7fa174156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1727fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1740c88c0 0x7fa1740ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa172ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1740c5880 0x7fa1741ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa17414f360 0x7fa174156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40732/0 (socket says 172.21.15.94:40732) 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.834+0000 7fa1737fe640 1 -- 172.21.15.94:0/3432310058 learned_addr learned my addr 172.21.15.94:0/3432310058 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1737fe640 1 -- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1740c88c0 msgr2=0x7fa1740ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1737fe640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1740c88c0 0x7fa1740ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1737fe640 1 -- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1740c5880 msgr2=0x7fa1741ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1737fe640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1740c5880 0x7fa1741ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1737fe640 1 -- 172.21.15.94:0/3432310058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1740b0560 con 0x7fa17414f360 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1737fe640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa17414f360 0x7fa174156ea0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa16c036040 tx=0x7fa16c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa15b7fe640 1 -- 172.21.15.94:0/3432310058 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c043020 con 0x7fa17414f360 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa15b7fe640 1 -- 172.21.15.94:0/3432310058 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c03e450 con 0x7fa17414f360 2024-03-06T07:18:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1740ac2d0 con 0x7fa17414f360 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa15b7fe640 1 -- 172.21.15.94:0/3432310058 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c009510 con 0x7fa17414f360 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.835+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa174157d60 con 0x7fa17414f360 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.836+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17404f330 con 0x7fa17414f360 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.837+0000 7fa15b7fe640 1 -- 172.21.15.94:0/3432310058 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa16c046020 con 0x7fa17414f360 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.837+0000 7fa15b7fe640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa16006b4d0 0x7fa16006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.837+0000 7fa15b7fe640 1 -- 172.21.15.94:0/3432310058 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa16c0aca80 con 0x7fa17414f360 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.837+0000 7fa172ffd640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa16006b4d0 0x7fa16006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:31.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.838+0000 7fa172ffd640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa16006b4d0 0x7fa16006d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1740ac540 tx=0x7fa164009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:31.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:31.841+0000 7fa15b7fe640 1 -- 172.21.15.94:0/3432310058 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa16c04d030 con 0x7fa17414f360 2024-03-06T07:18:32.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.052+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa174158b00 con 0x7fa17414f360 2024-03-06T07:18:32.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.052+0000 7fa15b7fe640 1 -- 172.21.15.94:0/3432310058 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7fa16c07b000 con 0x7fa17414f360 2024-03-06T07:18:32.054 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:32.055 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:32.055 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa16006b4d0 msgr2=0x7fa16006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa16006b4d0 0x7fa16006d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1740ac540 tx=0x7fa164009190 comp rx=0 tx=0).stop 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa17414f360 msgr2=0x7fa174156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa17414f360 0x7fa174156ea0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa16c036040 tx=0x7fa16c004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 shutdown_connections 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa16006b4d0 0x7fa16006d990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa17414f360 0x7fa174156ea0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1740c88c0 0x7fa1740ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 --2- 172.21.15.94:0/3432310058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1740c5880 0x7fa1741ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.054+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 >> 172.21.15.94:0/3432310058 conn(0x7fa1740ac570 msgr2=0x7fa1740ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:32.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.055+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 shutdown_connections 2024-03-06T07:18:32.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.055+0000 7fa1792c3640 1 -- 172.21.15.94:0/3432310058 wait complete. 2024-03-06T07:18:32.067 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:32.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.164+0000 7f03cdc88640 1 Processor -- start 2024-03-06T07:18:32.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03cdc88640 1 -- start start 2024-03-06T07:18:32.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03cdc88640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f03c81506b0 0x7f03c8153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03cdc88640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8070aa0 con 0x7f03c8156ea0 2024-03-06T07:18:32.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03cdc88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8070d80 con 0x7f03c80acc00 2024-03-06T07:18:32.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03cdc88640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c8156450 con 0x7f03c81506b0 2024-03-06T07:18:32.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03c77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f03c81506b0 0x7f03c8153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03c6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f03c8156ea0 0x7f03c80702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41406/0 (socket says 172.21.15.94:41406) 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.165+0000 7f03c6ffd640 1 -- 172.21.15.94:0/2034801589 learned_addr learned my addr 172.21.15.94:0/2034801589 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.166+0000 7f03c77fe640 1 -- 172.21.15.94:0/2034801589 >> v1:172.21.15.145:6789/0 conn(0x7f03c80acc00 legacy=0x7f03c80c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.166+0000 7f03c77fe640 1 -- 172.21.15.94:0/2034801589 >> v1:172.21.15.94:6789/0 conn(0x7f03c8156ea0 legacy=0x7f03c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.166+0000 7f03c67fc640 1 -- 172.21.15.94:0/2034801589 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766746730 0 0) 0x7f03c8070aa0 con 0x7f03c8156ea0 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.166+0000 7f03c77fe640 1 -- 172.21.15.94:0/2034801589 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c81566a0 con 0x7f03c81506b0 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.166+0000 7f03c77fe640 1 --2- 172.21.15.94:0/2034801589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c8153990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6dcf995cf9b64a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03c67fc640 1 -- 172.21.15.94:0/2034801589 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8034070 con 0x7f03c81506b0 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03c67fc640 1 -- 172.21.15.94:0/2034801589 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8030610 con 0x7f03c81506b0 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03c67fc640 1 -- 172.21.15.94:0/2034801589 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b80309e0 con 0x7f03c81506b0 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03cdc88640 1 -- 172.21.15.94:0/2034801589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 msgr2=0x7f03c8153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03cdc88640 1 --2- 172.21.15.94:0/2034801589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c8153990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b802b960 comp rx=0 tx=0).stop 2024-03-06T07:18:32.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03cdc88640 1 -- 172.21.15.94:0/2034801589 shutdown_connections 2024-03-06T07:18:32.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03cdc88640 1 --2- 172.21.15.94:0/2034801589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c8153990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03cdc88640 1 -- 172.21.15.94:0/2034801589 >> 172.21.15.94:0/2034801589 conn(0x7f03c806b1a0 msgr2=0x7f03c806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:32.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03cdc88640 1 -- 172.21.15.94:0/2034801589 shutdown_connections 2024-03-06T07:18:32.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.167+0000 7f03cdc88640 1 -- 172.21.15.94:0/2034801589 wait complete. 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.168+0000 7f03cdc88640 1 Processor -- start 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.168+0000 7f03cdc88640 1 -- start start 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03cdc88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03c80acc00 0x7f03c814be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03cdc88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c81498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03cdc88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03c8156ea0 0x7f03c8149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03cdc88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80ad050 con 0x7f03c81506b0 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03cdc88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8156a10 con 0x7f03c8156ea0 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03cdc88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80ad1d0 con 0x7f03c80acc00 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03c6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c81498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03c6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c81498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40744/0 (socket says 172.21.15.94:40744) 2024-03-06T07:18:32.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03c6ffd640 1 -- 172.21.15.94:0/1462084325 learned_addr learned my addr 172.21.15.94:0/1462084325 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03c77fe640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03c80acc00 0x7f03c814be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.169+0000 7f03c7fff640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03c8156ea0 0x7f03c8149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03c6ffd640 1 -- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03c80acc00 msgr2=0x7f03c814be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03c6ffd640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03c80acc00 0x7f03c814be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03c6ffd640 1 -- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03c8156ea0 msgr2=0x7f03c8149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03c6ffd640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03c8156ea0 0x7f03c8149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03c6ffd640 1 -- 172.21.15.94:0/1462084325 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c81e8b80 con 0x7f03c81506b0 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03c6ffd640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c81498d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c3c0 tx=0x7f03bc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03affff640 1 -- 172.21.15.94:0/1462084325 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0074e0 con 0x7f03c81506b0 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03affff640 1 -- 172.21.15.94:0/1462084325 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc0047e0 con 0x7f03c81506b0 2024-03-06T07:18:32.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03affff640 1 -- 172.21.15.94:0/1462084325 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc004bd0 con 0x7f03c81506b0 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c80aae90 con 0x7f03c81506b0 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.170+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c80a88b0 con 0x7f03c81506b0 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.171+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c804f330 con 0x7f03c81506b0 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.171+0000 7f03affff640 1 -- 172.21.15.94:0/1462084325 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03bc011070 con 0x7f03c81506b0 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.172+0000 7f03affff640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f03b406b500 0x7f03b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.172+0000 7f03affff640 1 -- 172.21.15.94:0/1462084325 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f03bc089ed0 con 0x7f03c81506b0 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.172+0000 7f03c77fe640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f03b406b500 0x7f03b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.172+0000 7f03c77fe640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f03b406b500 0x7f03b406d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03b8036000 tx=0x7f03b803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.176+0000 7f03affff640 1 -- 172.21.15.94:0/1462084325 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03bc058450 con 0x7f03c81506b0 2024-03-06T07:18:32.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.347+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03c80700a0 con 0x7f03c81506b0 2024-03-06T07:18:32.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.349+0000 7f03affff640 1 -- 172.21.15.94:0/1462084325 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f03bc05c4f0 con 0x7f03c81506b0 2024-03-06T07:18:32.351 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:32.351 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.350+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f03b406b500 msgr2=0x7f03b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.350+0000 7f03cdc88640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f03b406b500 0x7f03b406d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f03b8036000 tx=0x7f03b803b040 comp rx=0 tx=0).stop 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.350+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 msgr2=0x7f03c81498d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.350+0000 7f03cdc88640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c81498d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c3c0 tx=0x7f03bc00c880 comp rx=0 tx=0).stop 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 shutdown_connections 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f03b406b500 0x7f03b406d9c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03c8156ea0 0x7f03c8149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03c81506b0 0x7f03c81498d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 --2- 172.21.15.94:0/1462084325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03c80acc00 0x7f03c814be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 >> 172.21.15.94:0/1462084325 conn(0x7f03c806b1a0 msgr2=0x7f03c8152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:32.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 shutdown_connections 2024-03-06T07:18:32.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.351+0000 7f03cdc88640 1 -- 172.21.15.94:0/1462084325 wait complete. 2024-03-06T07:18:32.364 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:18:32.364 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:18:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.461+0000 7fb458288640 1 Processor -- start 2024-03-06T07:18:32.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.461+0000 7fb458288640 1 -- start start 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb458288640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5480 0x7fb450155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb458288640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb4500c5860 0x7fb45006f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb458288640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4500c5cc0 con 0x7fb4501586f0 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb458288640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45006df40 con 0x7fb4500c5860 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb458288640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4501591c0 con 0x7fb4500c5480 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb455823640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb4500c5860 0x7fb45006f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb456024640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5480 0x7fb450155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb456825640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fb4501586f0 0x7fb4500724f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43740/0 (socket says 172.21.15.94:43740) 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb455823640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb4500c5860 0x7fb45006f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41564/0 (socket says 172.21.15.94:41564) 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb456024640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5480 0x7fb450155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54224/0 (socket says 172.21.15.94:54224) 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.462+0000 7fb456825640 1 -- 172.21.15.94:0/1464464735 learned_addr learned my addr 172.21.15.94:0/1464464735 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455823640 1 -- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5480 msgr2=0x7fb450155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:32.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455823640 1 --2- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5480 0x7fb450155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455823640 1 -- 172.21.15.94:0/1464464735 >> v1:172.21.15.145:6789/0 conn(0x7fb4501586f0 legacy=0x7fb4500724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455022640 1 -- 172.21.15.94:0/1464464735 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193990671 0 0) 0x7fb4500c5cc0 con 0x7fb4501586f0 2024-03-06T07:18:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455823640 1 -- 172.21.15.94:0/1464464735 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4501594a0 con 0x7fb4500c5860 2024-03-06T07:18:32.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb456024640 1 --2- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5480 0x7fb450155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455823640 1 --2- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5860 0x7fb45006f6a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4400097a0 tx=0x7fb44002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=750c777636bdec10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455022640 1 -- 172.21.15.94:0/1464464735 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4400315a0 con 0x7fb4500c5860 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.463+0000 7fb455022640 1 -- 172.21.15.94:0/1464464735 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb440037070 con 0x7fb4500c5860 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb455022640 1 -- 172.21.15.94:0/1464464735 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440031960 con 0x7fb4500c5860 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 -- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5860 msgr2=0x7fb45006f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 --2- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5860 0x7fb45006f6a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4400097a0 tx=0x7fb44002b840 comp rx=0 tx=0).stop 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 -- 172.21.15.94:0/1464464735 shutdown_connections 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 --2- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5860 0x7fb45006f6a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 --2- 172.21.15.94:0/1464464735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5480 0x7fb450155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 -- 172.21.15.94:0/1464464735 >> 172.21.15.94:0/1464464735 conn(0x7fb45006b1a0 msgr2=0x7fb45006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 -- 172.21.15.94:0/1464464735 shutdown_connections 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.464+0000 7fb458288640 1 -- 172.21.15.94:0/1464464735 wait complete. 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.465+0000 7fb458288640 1 Processor -- start 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.465+0000 7fb458288640 1 -- start start 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb458288640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5480 0x7fb4501ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb458288640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5860 0x7fb4500c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb458288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4501586f0 0x7fb4500ac820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb458288640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450152a20 con 0x7fb4501586f0 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb458288640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450155680 con 0x7fb4500c5480 2024-03-06T07:18:32.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb458288640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb450155050 con 0x7fb4500c5860 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5480 0x7fb4501ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456825640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4501586f0 0x7fb4500ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5480 0x7fb4501ebed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41580/0 (socket says 172.21.15.94:41580) 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 -- 172.21.15.94:0/3920923125 learned_addr learned my addr 172.21.15.94:0/3920923125 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb455823640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5860 0x7fb4500c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 -- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5860 msgr2=0x7fb4500c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5860 0x7fb4500c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 -- 172.21.15.94:0/3920923125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4501586f0 msgr2=0x7fb4500ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4501586f0 0x7fb4500ac820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.466+0000 7fb456024640 1 -- 172.21.15.94:0/3920923125 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4501ec410 con 0x7fb4500c5480 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.467+0000 7fb456024640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5480 0x7fb4501ebed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb44800efc0 tx=0x7fb44800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.467+0000 7fb446ffd640 1 -- 172.21.15.94:0/3920923125 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448018020 con 0x7fb4500c5480 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.467+0000 7fb446ffd640 1 -- 172.21.15.94:0/3920923125 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb448014680 con 0x7fb4500c5480 2024-03-06T07:18:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.467+0000 7fb446ffd640 1 -- 172.21.15.94:0/3920923125 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448014a50 con 0x7fb4500c5480 2024-03-06T07:18:32.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.467+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4501eb6f0 con 0x7fb4500c5480 2024-03-06T07:18:32.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.467+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb450156850 con 0x7fb4500c5480 2024-03-06T07:18:32.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.468+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4501eb500 con 0x7fb4500c5480 2024-03-06T07:18:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.468+0000 7fb446ffd640 1 -- 172.21.15.94:0/3920923125 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb448005d10 con 0x7fb4500c5480 2024-03-06T07:18:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.468+0000 7fb446ffd640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb43406b500 0x7fb43406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.469+0000 7fb446ffd640 1 -- 172.21.15.94:0/3920923125 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb44808b1f0 con 0x7fb4500c5480 2024-03-06T07:18:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.469+0000 7fb455823640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb43406b500 0x7fb43406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.469+0000 7fb455823640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb43406b500 0x7fb43406d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb440036000 tx=0x7fb440008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.474+0000 7fb446ffd640 1 -- 172.21.15.94:0/3920923125 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb448022030 con 0x7fb4500c5480 2024-03-06T07:18:32.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.641+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb450156a90 con 0x7fb43406b500 2024-03-06T07:18:32.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.657+0000 7fb446ffd640 1 -- 172.21.15.94:0/3920923125 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332611 (secure 0 0 0) 0x7fb440037040 con 0x7fb43406b500 2024-03-06T07:18:32.664 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:32.666 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:18:32.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb43406b500 msgr2=0x7fb43406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb43406b500 0x7fb43406d9c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb440036000 tx=0x7fb440008030 comp rx=0 tx=0).stop 2024-03-06T07:18:32.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5480 msgr2=0x7fb4501ebed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5480 0x7fb4501ebed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb44800efc0 tx=0x7fb44800c490 comp rx=0 tx=0).stop 2024-03-06T07:18:32.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 shutdown_connections 2024-03-06T07:18:32.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb43406b500 0x7fb43406d9c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4501586f0 0x7fb4500ac820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4500c5860 0x7fb4500c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 --2- 172.21.15.94:0/3920923125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4500c5480 0x7fb4501ebed0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 >> 172.21.15.94:0/3920923125 conn(0x7fb45006b1a0 msgr2=0x7fb4500c8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:32.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 shutdown_connections 2024-03-06T07:18:32.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.660+0000 7fb458288640 1 -- 172.21.15.94:0/3920923125 wait complete. 2024-03-06T07:18:32.674 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-06T07:18:32.639796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226688,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754748032,"statfs":{"total":773094113280,"available":772861984768,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002245"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719950+0000","last_change":"2024-03-06T07:18:08.523161+0000","last_active":"2024-03-06T07:18:14.719950+0000","last_peered":"2024-03-06T07:18:14.719950+0000","last_clean":"2024-03-06T07:18:14.719950+0000","last_became_active":"2024-03-06T07:18:08.522883+0000","last_became_peered":"2024-03-06T07:18:08.522883+0000","last_unstale":"2024-03-06T07:18:14.719950+0000","last_undegraded":"2024-03-06T07:18:14.719950+0000","last_fullsized":"2024-03-06T07:18:14.719950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720669+0000","last_change":"2024-03-06T07:18:09.526639+0000","last_active":"2024-03-06T07:18:14.720669+0000","last_peered":"2024-03-06T07:18:14.720669+0000","last_clean":"2024-03-06T07:18:14.720669+0000","last_became_active":"2024-03-06T07:18:09.526380+0000","last_became_peered":"2024-03-06T07:18:09.526380+0000","last_unstale":"2024-03-06T07:18:14.720669+0000","last_undegraded":"2024-03-06T07:18:14.720669+0000","last_fullsized":"2024-03-06T07:18:14.720669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720585+0000","last_change":"2024-03-06T07:18:13.788926+0000","last_active":"2024-03-06T07:18:14.720585+0000","last_peered":"2024-03-06T07:18:14.720585+0000","last_clean":"2024-03-06T07:18:14.720585+0000","last_became_active":"2024-03-06T07:18:13.788572+0000","last_became_peered":"2024-03-06T07:18:13.788572+0000","last_unstale":"2024-03-06T07:18:14.720585+0000","last_undegraded":"2024-03-06T07:18:14.720585+0000","last_fullsized":"2024-03-06T07:18:14.720585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:00.875495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720880+0000","last_change":"2024-03-06T07:18:08.517732+0000","last_active":"2024-03-06T07:18:14.720880+0000","last_peered":"2024-03-06T07:18:14.720880+0000","last_clean":"2024-03-06T07:18:14.720880+0000","last_became_active":"2024-03-06T07:18:08.517469+0000","last_became_peered":"2024-03-06T07:18:08.517469+0000","last_unstale":"2024-03-06T07:18:14.720880+0000","last_undegraded":"2024-03-06T07:18:14.720880+0000","last_fullsized":"2024-03-06T07:18:14.720880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719676+0000","last_change":"2024-03-06T07:18:09.531502+0000","last_active":"2024-03-06T07:18:14.719676+0000","last_peered":"2024-03-06T07:18:14.719676+0000","last_clean":"2024-03-06T07:18:14.719676+0000","last_became_active":"2024-03-06T07:18:09.531231+0000","last_became_peered":"2024-03-06T07:18:09.531231+0000","last_unstale":"2024-03-06T07:18:14.719676+0000","last_undegraded":"2024-03-06T07:18:14.719676+0000","last_fullsized":"2024-03-06T07:18:14.719676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551608+0000","last_change":"2024-03-06T07:18:13.565825+0000","last_active":"2024-03-06T07:18:14.551608+0000","last_peered":"2024-03-06T07:18:14.551608+0000","last_clean":"2024-03-06T07:18:14.551608+0000","last_became_active":"2024-03-06T07:18:13.565250+0000","last_became_peered":"2024-03-06T07:18:13.565250+0000","last_unstale":"2024-03-06T07:18:14.551608+0000","last_undegraded":"2024-03-06T07:18:14.551608+0000","last_fullsized":"2024-03-06T07:18:14.551608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:15.265063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718863+0000","last_change":"2024-03-06T07:18:08.523848+0000","last_active":"2024-03-06T07:18:14.718863+0000","last_peered":"2024-03-06T07:18:14.718863+0000","last_clean":"2024-03-06T07:18:14.718863+0000","last_became_active":"2024-03-06T07:18:08.523455+0000","last_became_peered":"2024-03-06T07:18:08.523455+0000","last_unstale":"2024-03-06T07:18:14.718863+0000","last_undegraded":"2024-03-06T07:18:14.718863+0000","last_fullsized":"2024-03-06T07:18:14.718863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718515+0000","last_change":"2024-03-06T07:18:09.531072+0000","last_active":"2024-03-06T07:18:14.718515+0000","last_peered":"2024-03-06T07:18:14.718515+0000","last_clean":"2024-03-06T07:18:14.718515+0000","last_became_active":"2024-03-06T07:18:09.530815+0000","last_became_peered":"2024-03-06T07:18:09.530815+0000","last_unstale":"2024-03-06T07:18:14.718515+0000","last_undegraded":"2024-03-06T07:18:14.718515+0000","last_fullsized":"2024-03-06T07:18:14.718515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551688+0000","last_change":"2024-03-06T07:18:13.563305+0000","last_active":"2024-03-06T07:18:14.551688+0000","last_peered":"2024-03-06T07:18:14.551688+0000","last_clean":"2024-03-06T07:18:14.551688+0000","last_became_active":"2024-03-06T07:18:13.562936+0000","last_became_peered":"2024-03-06T07:18:13.562936+0000","last_unstale":"2024-03-06T07:18:14.551688+0000","last_undegraded":"2024-03-06T07:18:14.551688+0000","last_fullsized":"2024-03-06T07:18:14.551688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:02.828593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718486+0000","last_change":"2024-03-06T07:18:08.520455+0000","last_active":"2024-03-06T07:18:14.718486+0000","last_peered":"2024-03-06T07:18:14.718486+0000","last_clean":"2024-03-06T07:18:14.718486+0000","last_became_active":"2024-03-06T07:18:08.520170+0000","last_became_peered":"2024-03-06T07:18:08.520170+0000","last_unstale":"2024-03-06T07:18:14.718486+0000","last_undegraded":"2024-03-06T07:18:14.718486+0000","last_fullsized":"2024-03-06T07:18:14.718486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719134+0000","last_change":"2024-03-06T07:18:09.536183+0000","last_active":"2024-03-06T07:18:14.719134+0000","last_peered":"2024-03-06T07:18:14.719134+0000","last_clean":"2024-03-06T07:18:14.719134+0000","last_became_active":"2024-03-06T07:18:09.535516+0000","last_became_peered":"2024-03-06T07:18:09.535516+0000","last_unstale":"2024-03-06T07:18:14.719134+0000","last_undegraded":"2024-03-06T07:18:14.719134+0000","last_fullsized":"2024-03-06T07:18:14.719134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551978+0000","last_change":"2024-03-06T07:18:13.564931+0000","last_active":"2024-03-06T07:18:14.551978+0000","last_peered":"2024-03-06T07:18:14.551978+0000","last_clean":"2024-03-06T07:18:14.551978+0000","last_became_active":"2024-03-06T07:18:13.564588+0000","last_became_peered":"2024-03-06T07:18:13.564588+0000","last_unstale":"2024-03-06T07:18:14.551978+0000","last_undegraded":"2024-03-06T07:18:14.551978+0000","last_fullsized":"2024-03-06T07:18:14.551978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:41.700550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550818+0000","last_change":"2024-03-06T07:18:09.516785+0000","last_active":"2024-03-06T07:18:14.550818+0000","last_peered":"2024-03-06T07:18:14.550818+0000","last_clean":"2024-03-06T07:18:14.550818+0000","last_became_active":"2024-03-06T07:18:09.516494+0000","last_became_peered":"2024-03-06T07:18:09.516494+0000","last_unstale":"2024-03-06T07:18:14.550818+0000","last_undegraded":"2024-03-06T07:18:14.550818+0000","last_fullsized":"2024-03-06T07:18:14.550818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719130+0000","last_change":"2024-03-06T07:18:08.523549+0000","last_active":"2024-03-06T07:18:14.719130+0000","last_peered":"2024-03-06T07:18:14.719130+0000","last_clean":"2024-03-06T07:18:14.719130+0000","last_became_active":"2024-03-06T07:18:08.523264+0000","last_became_peered":"2024-03-06T07:18:08.523264+0000","last_unstale":"2024-03-06T07:18:14.719130+0000","last_undegraded":"2024-03-06T07:18:14.719130+0000","last_fullsized":"2024-03-06T07:18:14.719130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:02.351247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552173+0000","last_change":"2024-03-06T07:18:13.563114+0000","last_active":"2024-03-06T07:18:14.552173+0000","last_peered":"2024-03-06T07:18:14.552173+0000","last_clean":"2024-03-06T07:18:14.552173+0000","last_became_active":"2024-03-06T07:18:13.562736+0000","last_became_peered":"2024-03-06T07:18:13.562736+0000","last_unstale":"2024-03-06T07:18:14.552173+0000","last_undegraded":"2024-03-06T07:18:14.552173+0000","last_fullsized":"2024-03-06T07:18:14.552173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:20.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720134+0000","last_change":"2024-03-06T07:18:09.524088+0000","last_active":"2024-03-06T07:18:14.720134+0000","last_peered":"2024-03-06T07:18:14.720134+0000","last_clean":"2024-03-06T07:18:14.720134+0000","last_became_active":"2024-03-06T07:18:09.523821+0000","last_became_peered":"2024-03-06T07:18:09.523821+0000","last_unstale":"2024-03-06T07:18:14.720134+0000","last_undegraded":"2024-03-06T07:18:14.720134+0000","last_fullsized":"2024-03-06T07:18:14.720134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719053+0000","last_change":"2024-03-06T07:18:08.516137+0000","last_active":"2024-03-06T07:18:14.719053+0000","last_peered":"2024-03-06T07:18:14.719053+0000","last_clean":"2024-03-06T07:18:14.719053+0000","last_became_active":"2024-03-06T07:18:08.515841+0000","last_became_peered":"2024-03-06T07:18:08.515841+0000","last_unstale":"2024-03-06T07:18:14.719053+0000","last_undegraded":"2024-03-06T07:18:14.719053+0000","last_fullsized":"2024-03-06T07:18:14.719053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551452+0000","last_change":"2024-03-06T07:18:13.564851+0000","last_active":"2024-03-06T07:18:14.551452+0000","last_peered":"2024-03-06T07:18:14.551452+0000","last_clean":"2024-03-06T07:18:14.551452+0000","last_became_active":"2024-03-06T07:18:13.564494+0000","last_became_peered":"2024-03-06T07:18:13.564494+0000","last_unstale":"2024-03-06T07:18:14.551452+0000","last_undegraded":"2024-03-06T07:18:14.551452+0000","last_fullsized":"2024-03-06T07:18:14.551452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:27.757638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720452+0000","last_change":"2024-03-06T07:18:09.527318+0000","last_active":"2024-03-06T07:18:14.720452+0000","last_peered":"2024-03-06T07:18:14.720452+0000","last_clean":"2024-03-06T07:18:14.720452+0000","last_became_active":"2024-03-06T07:18:09.527061+0000","last_became_peered":"2024-03-06T07:18:09.527061+0000","last_unstale":"2024-03-06T07:18:14.720452+0000","last_undegraded":"2024-03-06T07:18:14.720452+0000","last_fullsized":"2024-03-06T07:18:14.720452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719621+0000","last_change":"2024-03-06T07:18:08.515303+0000","last_active":"2024-03-06T07:18:14.719621+0000","last_peered":"2024-03-06T07:18:14.719621+0000","last_clean":"2024-03-06T07:18:14.719621+0000","last_became_active":"2024-03-06T07:18:08.514178+0000","last_became_peered":"2024-03-06T07:18:08.514178+0000","last_unstale":"2024-03-06T07:18:14.719621+0000","last_undegraded":"2024-03-06T07:18:14.719621+0000","last_fullsized":"2024-03-06T07:18:14.719621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719788+0000","last_change":"2024-03-06T07:18:13.788851+0000","last_active":"2024-03-06T07:18:14.719788+0000","last_peered":"2024-03-06T07:18:14.719788+0000","last_clean":"2024-03-06T07:18:14.719788+0000","last_became_active":"2024-03-06T07:18:13.788492+0000","last_became_peered":"2024-03-06T07:18:13.788492+0000","last_unstale":"2024-03-06T07:18:14.719788+0000","last_undegraded":"2024-03-06T07:18:14.719788+0000","last_fullsized":"2024-03-06T07:18:14.719788+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552468+0000","last_change":"2024-03-06T07:18:09.517957+0000","last_active":"2024-03-06T07:18:14.552468+0000","last_peered":"2024-03-06T07:18:14.552468+0000","last_clean":"2024-03-06T07:18:14.552468+0000","last_became_active":"2024-03-06T07:18:09.517197+0000","last_became_peered":"2024-03-06T07:18:09.517197+0000","last_unstale":"2024-03-06T07:18:14.552468+0000","last_undegraded":"2024-03-06T07:18:14.552468+0000","last_fullsized":"2024-03-06T07:18:14.552468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:31.051748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718952+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.718952+0000","last_peered":"2024-03-06T07:18:14.718952+0000","last_clean":"2024-03-06T07:18:14.718952+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.718952+0000","last_undegraded":"2024-03-06T07:18:14.718952+0000","last_fullsized":"2024-03-06T07:18:14.718952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552149+0000","last_change":"2024-03-06T07:18:13.565073+0000","last_active":"2024-03-06T07:18:14.552149+0000","last_peered":"2024-03-06T07:18:14.552149+0000","last_clean":"2024-03-06T07:18:14.552149+0000","last_became_active":"2024-03-06T07:18:13.564708+0000","last_became_peered":"2024-03-06T07:18:13.564708+0000","last_unstale":"2024-03-06T07:18:14.552149+0000","last_undegraded":"2024-03-06T07:18:14.552149+0000","last_fullsized":"2024-03-06T07:18:14.552149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:20.796366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719443+0000","last_change":"2024-03-06T07:18:09.520925+0000","last_active":"2024-03-06T07:18:14.719443+0000","last_peered":"2024-03-06T07:18:14.719443+0000","last_clean":"2024-03-06T07:18:14.719443+0000","last_became_active":"2024-03-06T07:18:09.520658+0000","last_became_peered":"2024-03-06T07:18:09.520658+0000","last_unstale":"2024-03-06T07:18:14.719443+0000","last_undegraded":"2024-03-06T07:18:14.719443+0000","last_fullsized":"2024-03-06T07:18:14.719443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551744+0000","last_change":"2024-03-06T07:18:08.519229+0000","last_active":"2024-03-06T07:18:14.551744+0000","last_peered":"2024-03-06T07:18:14.551744+0000","last_clean":"2024-03-06T07:18:14.551744+0000","last_became_active":"2024-03-06T07:18:08.518936+0000","last_became_peered":"2024-03-06T07:18:08.518936+0000","last_unstale":"2024-03-06T07:18:14.551744+0000","last_undegraded":"2024-03-06T07:18:14.551744+0000","last_fullsized":"2024-03-06T07:18:14.551744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552379+0000","last_change":"2024-03-06T07:18:13.787076+0000","last_active":"2024-03-06T07:18:14.552379+0000","last_peered":"2024-03-06T07:18:14.552379+0000","last_clean":"2024-03-06T07:18:14.552379+0000","last_became_active":"2024-03-06T07:18:13.786733+0000","last_became_peered":"2024-03-06T07:18:13.786733+0000","last_unstale":"2024-03-06T07:18:14.552379+0000","last_undegraded":"2024-03-06T07:18:14.552379+0000","last_fullsized":"2024-03-06T07:18:14.552379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720898+0000","last_change":"2024-03-06T07:18:09.534800+0000","last_active":"2024-03-06T07:18:14.720898+0000","last_peered":"2024-03-06T07:18:14.720898+0000","last_clean":"2024-03-06T07:18:14.720898+0000","last_became_active":"2024-03-06T07:18:09.534408+0000","last_became_peered":"2024-03-06T07:18:09.534408+0000","last_unstale":"2024-03-06T07:18:14.720898+0000","last_undegraded":"2024-03-06T07:18:14.720898+0000","last_fullsized":"2024-03-06T07:18:14.720898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719429+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.719429+0000","last_peered":"2024-03-06T07:18:14.719429+0000","last_clean":"2024-03-06T07:18:14.719429+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.719429+0000","last_undegraded":"2024-03-06T07:18:14.719429+0000","last_fullsized":"2024-03-06T07:18:14.719429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720202+0000","last_change":"2024-03-06T07:18:13.785863+0000","last_active":"2024-03-06T07:18:14.720202+0000","last_peered":"2024-03-06T07:18:14.720202+0000","last_clean":"2024-03-06T07:18:14.720202+0000","last_became_active":"2024-03-06T07:18:13.785533+0000","last_became_peered":"2024-03-06T07:18:13.785533+0000","last_unstale":"2024-03-06T07:18:14.720202+0000","last_undegraded":"2024-03-06T07:18:14.720202+0000","last_fullsized":"2024-03-06T07:18:14.720202+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:43.382716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718962+0000","last_change":"2024-03-06T07:18:09.523934+0000","last_active":"2024-03-06T07:18:14.718962+0000","last_peered":"2024-03-06T07:18:14.718962+0000","last_clean":"2024-03-06T07:18:14.718962+0000","last_became_active":"2024-03-06T07:18:09.523663+0000","last_became_peered":"2024-03-06T07:18:09.523663+0000","last_unstale":"2024-03-06T07:18:14.718962+0000","last_undegraded":"2024-03-06T07:18:14.718962+0000","last_fullsized":"2024-03-06T07:18:14.718962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718957+0000","last_change":"2024-03-06T07:18:08.520839+0000","last_active":"2024-03-06T07:18:14.718957+0000","last_peered":"2024-03-06T07:18:14.718957+0000","last_clean":"2024-03-06T07:18:14.718957+0000","last_became_active":"2024-03-06T07:18:08.520578+0000","last_became_peered":"2024-03-06T07:18:08.520578+0000","last_unstale":"2024-03-06T07:18:14.718957+0000","last_undegraded":"2024-03-06T07:18:14.718957+0000","last_fullsized":"2024-03-06T07:18:14.718957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:21.462756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718986+0000","last_change":"2024-03-06T07:18:13.787099+0000","last_active":"2024-03-06T07:18:14.718986+0000","last_peered":"2024-03-06T07:18:14.718986+0000","last_clean":"2024-03-06T07:18:14.718986+0000","last_became_active":"2024-03-06T07:18:13.786694+0000","last_became_peered":"2024-03-06T07:18:13.786694+0000","last_unstale":"2024-03-06T07:18:14.718986+0000","last_undegraded":"2024-03-06T07:18:14.718986+0000","last_fullsized":"2024-03-06T07:18:14.718986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719219+0000","last_change":"2024-03-06T07:18:09.531609+0000","last_active":"2024-03-06T07:18:14.719219+0000","last_peered":"2024-03-06T07:18:14.719219+0000","last_clean":"2024-03-06T07:18:14.719219+0000","last_became_active":"2024-03-06T07:18:09.531318+0000","last_became_peered":"2024-03-06T07:18:09.531318+0000","last_unstale":"2024-03-06T07:18:14.719219+0000","last_undegraded":"2024-03-06T07:18:14.719219+0000","last_fullsized":"2024-03-06T07:18:14.719219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719042+0000","last_change":"2024-03-06T07:18:08.516897+0000","last_active":"2024-03-06T07:18:14.719042+0000","last_peered":"2024-03-06T07:18:14.719042+0000","last_clean":"2024-03-06T07:18:14.719042+0000","last_became_active":"2024-03-06T07:18:08.516647+0000","last_became_peered":"2024-03-06T07:18:08.516647+0000","last_unstale":"2024-03-06T07:18:14.719042+0000","last_undegraded":"2024-03-06T07:18:14.719042+0000","last_fullsized":"2024-03-06T07:18:14.719042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718719+0000","last_change":"2024-03-06T07:18:13.564264+0000","last_active":"2024-03-06T07:18:14.718719+0000","last_peered":"2024-03-06T07:18:14.718719+0000","last_clean":"2024-03-06T07:18:14.718719+0000","last_became_active":"2024-03-06T07:18:13.563923+0000","last_became_peered":"2024-03-06T07:18:13.563923+0000","last_unstale":"2024-03-06T07:18:14.718719+0000","last_undegraded":"2024-03-06T07:18:14.718719+0000","last_fullsized":"2024-03-06T07:18:14.718719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552093+0000","last_change":"2024-03-06T07:18:09.519337+0000","last_active":"2024-03-06T07:18:14.552093+0000","last_peered":"2024-03-06T07:18:14.552093+0000","last_clean":"2024-03-06T07:18:14.552093+0000","last_became_active":"2024-03-06T07:18:09.519042+0000","last_became_peered":"2024-03-06T07:18:09.519042+0000","last_unstale":"2024-03-06T07:18:14.552093+0000","last_undegraded":"2024-03-06T07:18:14.552093+0000","last_fullsized":"2024-03-06T07:18:14.552093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:48.003076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718514+0000","last_change":"2024-03-06T07:18:08.521877+0000","last_active":"2024-03-06T07:18:14.718514+0000","last_peered":"2024-03-06T07:18:14.718514+0000","last_clean":"2024-03-06T07:18:14.718514+0000","last_became_active":"2024-03-06T07:18:08.521583+0000","last_became_peered":"2024-03-06T07:18:08.521583+0000","last_unstale":"2024-03-06T07:18:14.718514+0000","last_undegraded":"2024-03-06T07:18:14.718514+0000","last_fullsized":"2024-03-06T07:18:14.718514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718907+0000","last_change":"2024-03-06T07:18:13.566418+0000","last_active":"2024-03-06T07:18:14.718907+0000","last_peered":"2024-03-06T07:18:14.718907+0000","last_clean":"2024-03-06T07:18:14.718907+0000","last_became_active":"2024-03-06T07:18:13.566013+0000","last_became_peered":"2024-03-06T07:18:13.566013+0000","last_unstale":"2024-03-06T07:18:14.718907+0000","last_undegraded":"2024-03-06T07:18:14.718907+0000","last_fullsized":"2024-03-06T07:18:14.718907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719777+0000","last_change":"2024-03-06T07:18:09.522608+0000","last_active":"2024-03-06T07:18:14.719777+0000","last_peered":"2024-03-06T07:18:14.719777+0000","last_clean":"2024-03-06T07:18:14.719777+0000","last_became_active":"2024-03-06T07:18:09.522360+0000","last_became_peered":"2024-03-06T07:18:09.522360+0000","last_unstale":"2024-03-06T07:18:14.719777+0000","last_undegraded":"2024-03-06T07:18:14.719777+0000","last_fullsized":"2024-03-06T07:18:14.719777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548304+0000","last_change":"2024-03-06T07:18:08.512125+0000","last_active":"2024-03-06T07:18:14.548304+0000","last_peered":"2024-03-06T07:18:14.548304+0000","last_clean":"2024-03-06T07:18:14.548304+0000","last_became_active":"2024-03-06T07:18:08.511827+0000","last_became_peered":"2024-03-06T07:18:08.511827+0000","last_unstale":"2024-03-06T07:18:14.548304+0000","last_undegraded":"2024-03-06T07:18:14.548304+0000","last_fullsized":"2024-03-06T07:18:14.548304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719927+0000","last_change":"2024-03-06T07:18:13.573031+0000","last_active":"2024-03-06T07:18:14.719927+0000","last_peered":"2024-03-06T07:18:14.719927+0000","last_clean":"2024-03-06T07:18:14.719927+0000","last_became_active":"2024-03-06T07:18:13.572562+0000","last_became_peered":"2024-03-06T07:18:13.572562+0000","last_unstale":"2024-03-06T07:18:14.719927+0000","last_undegraded":"2024-03-06T07:18:14.719927+0000","last_fullsized":"2024-03-06T07:18:14.719927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:55.484210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719484+0000","last_change":"2024-03-06T07:18:09.518341+0000","last_active":"2024-03-06T07:18:14.719484+0000","last_peered":"2024-03-06T07:18:14.719484+0000","last_clean":"2024-03-06T07:18:14.719484+0000","last_became_active":"2024-03-06T07:18:09.518030+0000","last_became_peered":"2024-03-06T07:18:09.518030+0000","last_unstale":"2024-03-06T07:18:14.719484+0000","last_undegraded":"2024-03-06T07:18:14.719484+0000","last_fullsized":"2024-03-06T07:18:14.719484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:13.874406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718993+0000","last_change":"2024-03-06T07:18:08.522022+0000","last_active":"2024-03-06T07:18:14.718993+0000","last_peered":"2024-03-06T07:18:14.718993+0000","last_clean":"2024-03-06T07:18:14.718993+0000","last_became_active":"2024-03-06T07:18:08.521764+0000","last_became_peered":"2024-03-06T07:18:08.521764+0000","last_unstale":"2024-03-06T07:18:14.718993+0000","last_undegraded":"2024-03-06T07:18:14.718993+0000","last_fullsized":"2024-03-06T07:18:14.718993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719331+0000","last_change":"2024-03-06T07:18:13.785137+0000","last_active":"2024-03-06T07:18:14.719331+0000","last_peered":"2024-03-06T07:18:14.719331+0000","last_clean":"2024-03-06T07:18:14.719331+0000","last_became_active":"2024-03-06T07:18:13.784706+0000","last_became_peered":"2024-03-06T07:18:13.784706+0000","last_unstale":"2024-03-06T07:18:14.719331+0000","last_undegraded":"2024-03-06T07:18:14.719331+0000","last_fullsized":"2024-03-06T07:18:14.719331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719805+0000","last_change":"2024-03-06T07:18:09.532012+0000","last_active":"2024-03-06T07:18:14.719805+0000","last_peered":"2024-03-06T07:18:14.719805+0000","last_clean":"2024-03-06T07:18:14.719805+0000","last_became_active":"2024-03-06T07:18:09.531724+0000","last_became_peered":"2024-03-06T07:18:09.531724+0000","last_unstale":"2024-03-06T07:18:14.719805+0000","last_undegraded":"2024-03-06T07:18:14.719805+0000","last_fullsized":"2024-03-06T07:18:14.719805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718407+0000","last_change":"2024-03-06T07:18:08.515757+0000","last_active":"2024-03-06T07:18:14.718407+0000","last_peered":"2024-03-06T07:18:14.718407+0000","last_clean":"2024-03-06T07:18:14.718407+0000","last_became_active":"2024-03-06T07:18:08.515494+0000","last_became_peered":"2024-03-06T07:18:08.515494+0000","last_unstale":"2024-03-06T07:18:14.718407+0000","last_undegraded":"2024-03-06T07:18:14.718407+0000","last_fullsized":"2024-03-06T07:18:14.718407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719078+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:18:14.719078+0000","last_peered":"2024-03-06T07:18:14.719078+0000","last_clean":"2024-03-06T07:18:14.719078+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:18:14.719078+0000","last_undegraded":"2024-03-06T07:18:14.719078+0000","last_fullsized":"2024-03-06T07:18:14.719078+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551436+0000","last_change":"2024-03-06T07:18:13.786445+0000","last_active":"2024-03-06T07:18:14.551436+0000","last_peered":"2024-03-06T07:18:14.551436+0000","last_clean":"2024-03-06T07:18:14.551436+0000","last_became_active":"2024-03-06T07:18:13.786085+0000","last_became_peered":"2024-03-06T07:18:13.786085+0000","last_unstale":"2024-03-06T07:18:14.551436+0000","last_undegraded":"2024-03-06T07:18:14.551436+0000","last_fullsized":"2024-03-06T07:18:14.551436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718724+0000","last_change":"2024-03-06T07:18:09.532185+0000","last_active":"2024-03-06T07:18:14.718724+0000","last_peered":"2024-03-06T07:18:14.718724+0000","last_clean":"2024-03-06T07:18:14.718724+0000","last_became_active":"2024-03-06T07:18:09.531883+0000","last_became_peered":"2024-03-06T07:18:09.531883+0000","last_unstale":"2024-03-06T07:18:14.718724+0000","last_undegraded":"2024-03-06T07:18:14.718724+0000","last_fullsized":"2024-03-06T07:18:14.718724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548537+0000","last_change":"2024-03-06T07:18:08.518305+0000","last_active":"2024-03-06T07:18:14.548537+0000","last_peered":"2024-03-06T07:18:14.548537+0000","last_clean":"2024-03-06T07:18:14.548537+0000","last_became_active":"2024-03-06T07:18:08.517977+0000","last_became_peered":"2024-03-06T07:18:08.517977+0000","last_unstale":"2024-03-06T07:18:14.548537+0000","last_undegraded":"2024-03-06T07:18:14.548537+0000","last_fullsized":"2024-03-06T07:18:14.548537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550763+0000","last_change":"2024-03-06T07:18:13.785349+0000","last_active":"2024-03-06T07:18:14.550763+0000","last_peered":"2024-03-06T07:18:14.550763+0000","last_clean":"2024-03-06T07:18:14.550763+0000","last_became_active":"2024-03-06T07:18:13.784931+0000","last_became_peered":"2024-03-06T07:18:13.784931+0000","last_unstale":"2024-03-06T07:18:14.550763+0000","last_undegraded":"2024-03-06T07:18:14.550763+0000","last_fullsized":"2024-03-06T07:18:14.550763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718607+0000","last_change":"2024-03-06T07:18:09.532114+0000","last_active":"2024-03-06T07:18:14.718607+0000","last_peered":"2024-03-06T07:18:14.718607+0000","last_clean":"2024-03-06T07:18:14.718607+0000","last_became_active":"2024-03-06T07:18:09.531373+0000","last_became_peered":"2024-03-06T07:18:09.531373+0000","last_unstale":"2024-03-06T07:18:14.718607+0000","last_undegraded":"2024-03-06T07:18:14.718607+0000","last_fullsized":"2024-03-06T07:18:14.718607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718980+0000","last_change":"2024-03-06T07:18:08.520471+0000","last_active":"2024-03-06T07:18:14.718980+0000","last_peered":"2024-03-06T07:18:14.718980+0000","last_clean":"2024-03-06T07:18:14.718980+0000","last_became_active":"2024-03-06T07:18:08.520156+0000","last_became_peered":"2024-03-06T07:18:08.520156+0000","last_unstale":"2024-03-06T07:18:14.718980+0000","last_undegraded":"2024-03-06T07:18:14.718980+0000","last_fullsized":"2024-03-06T07:18:14.718980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719199+0000","last_change":"2024-03-06T07:18:13.569753+0000","last_active":"2024-03-06T07:18:14.719199+0000","last_peered":"2024-03-06T07:18:14.719199+0000","last_clean":"2024-03-06T07:18:14.719199+0000","last_became_active":"2024-03-06T07:18:13.569369+0000","last_became_peered":"2024-03-06T07:18:13.569369+0000","last_unstale":"2024-03-06T07:18:14.719199+0000","last_undegraded":"2024-03-06T07:18:14.719199+0000","last_fullsized":"2024-03-06T07:18:14.719199+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719153+0000","last_change":"2024-03-06T07:18:09.536431+0000","last_active":"2024-03-06T07:18:14.719153+0000","last_peered":"2024-03-06T07:18:14.719153+0000","last_clean":"2024-03-06T07:18:14.719153+0000","last_became_active":"2024-03-06T07:18:09.536167+0000","last_became_peered":"2024-03-06T07:18:09.536167+0000","last_unstale":"2024-03-06T07:18:14.719153+0000","last_undegraded":"2024-03-06T07:18:14.719153+0000","last_fullsized":"2024-03-06T07:18:14.719153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.441770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548593+0000","last_change":"2024-03-06T07:18:08.511825+0000","last_active":"2024-03-06T07:18:14.548593+0000","last_peered":"2024-03-06T07:18:14.548593+0000","last_clean":"2024-03-06T07:18:14.548593+0000","last_became_active":"2024-03-06T07:18:08.511363+0000","last_became_peered":"2024-03-06T07:18:08.511363+0000","last_unstale":"2024-03-06T07:18:14.548593+0000","last_undegraded":"2024-03-06T07:18:14.548593+0000","last_fullsized":"2024-03-06T07:18:14.548593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718008+0000","last_change":"2024-03-06T07:18:13.786468+0000","last_active":"2024-03-06T07:18:14.718008+0000","last_peered":"2024-03-06T07:18:14.718008+0000","last_clean":"2024-03-06T07:18:14.718008+0000","last_became_active":"2024-03-06T07:18:13.786129+0000","last_became_peered":"2024-03-06T07:18:13.786129+0000","last_unstale":"2024-03-06T07:18:14.718008+0000","last_undegraded":"2024-03-06T07:18:14.718008+0000","last_fullsized":"2024-03-06T07:18:14.718008+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:42.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718705+0000","last_change":"2024-03-06T07:18:09.534236+0000","last_active":"2024-03-06T07:18:14.718705+0000","last_peered":"2024-03-06T07:18:14.718705+0000","last_clean":"2024-03-06T07:18:14.718705+0000","last_became_active":"2024-03-06T07:18:09.533090+0000","last_became_peered":"2024-03-06T07:18:09.533090+0000","last_unstale":"2024-03-06T07:18:14.718705+0000","last_undegraded":"2024-03-06T07:18:14.718705+0000","last_fullsized":"2024-03-06T07:18:14.718705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552342+0000","last_change":"2024-03-06T07:18:08.519410+0000","last_active":"2024-03-06T07:18:14.552342+0000","last_peered":"2024-03-06T07:18:14.552342+0000","last_clean":"2024-03-06T07:18:14.552342+0000","last_became_active":"2024-03-06T07:18:08.519115+0000","last_became_peered":"2024-03-06T07:18:08.519115+0000","last_unstale":"2024-03-06T07:18:14.552342+0000","last_undegraded":"2024-03-06T07:18:14.552342+0000","last_fullsized":"2024-03-06T07:18:14.552342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:22.247661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552289+0000","last_change":"2024-03-06T07:18:13.563952+0000","last_active":"2024-03-06T07:18:14.552289+0000","last_peered":"2024-03-06T07:18:14.552289+0000","last_clean":"2024-03-06T07:18:14.552289+0000","last_became_active":"2024-03-06T07:18:13.563578+0000","last_became_peered":"2024-03-06T07:18:13.563578+0000","last_unstale":"2024-03-06T07:18:14.552289+0000","last_undegraded":"2024-03-06T07:18:14.552289+0000","last_fullsized":"2024-03-06T07:18:14.552289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718657+0000","last_change":"2024-03-06T07:18:09.526419+0000","last_active":"2024-03-06T07:18:14.718657+0000","last_peered":"2024-03-06T07:18:14.718657+0000","last_clean":"2024-03-06T07:18:14.718657+0000","last_became_active":"2024-03-06T07:18:09.526165+0000","last_became_peered":"2024-03-06T07:18:09.526165+0000","last_unstale":"2024-03-06T07:18:14.718657+0000","last_undegraded":"2024-03-06T07:18:14.718657+0000","last_fullsized":"2024-03-06T07:18:14.718657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548110+0000","last_change":"2024-03-06T07:18:08.508444+0000","last_active":"2024-03-06T07:18:14.548110+0000","last_peered":"2024-03-06T07:18:14.548110+0000","last_clean":"2024-03-06T07:18:14.548110+0000","last_became_active":"2024-03-06T07:18:08.508153+0000","last_became_peered":"2024-03-06T07:18:08.508153+0000","last_unstale":"2024-03-06T07:18:14.548110+0000","last_undegraded":"2024-03-06T07:18:14.548110+0000","last_fullsized":"2024-03-06T07:18:14.548110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719075+0000","last_change":"2024-03-06T07:18:13.567057+0000","last_active":"2024-03-06T07:18:14.719075+0000","last_peered":"2024-03-06T07:18:14.719075+0000","last_clean":"2024-03-06T07:18:14.719075+0000","last_became_active":"2024-03-06T07:18:13.566702+0000","last_became_peered":"2024-03-06T07:18:13.566702+0000","last_unstale":"2024-03-06T07:18:14.719075+0000","last_undegraded":"2024-03-06T07:18:14.719075+0000","last_fullsized":"2024-03-06T07:18:14.719075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550710+0000","last_change":"2024-03-06T07:18:09.520527+0000","last_active":"2024-03-06T07:18:14.550710+0000","last_peered":"2024-03-06T07:18:14.550710+0000","last_clean":"2024-03-06T07:18:14.550710+0000","last_became_active":"2024-03-06T07:18:09.520233+0000","last_became_peered":"2024-03-06T07:18:09.520233+0000","last_unstale":"2024-03-06T07:18:14.550710+0000","last_undegraded":"2024-03-06T07:18:14.550710+0000","last_fullsized":"2024-03-06T07:18:14.550710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550736+0000","last_change":"2024-03-06T07:18:08.521908+0000","last_active":"2024-03-06T07:18:14.550736+0000","last_peered":"2024-03-06T07:18:14.550736+0000","last_clean":"2024-03-06T07:18:14.550736+0000","last_became_active":"2024-03-06T07:18:08.521324+0000","last_became_peered":"2024-03-06T07:18:08.521324+0000","last_unstale":"2024-03-06T07:18:14.550736+0000","last_undegraded":"2024-03-06T07:18:14.550736+0000","last_fullsized":"2024-03-06T07:18:14.550736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720936+0000","last_change":"2024-03-06T07:18:13.788800+0000","last_active":"2024-03-06T07:18:14.720936+0000","last_peered":"2024-03-06T07:18:14.720936+0000","last_clean":"2024-03-06T07:18:14.720936+0000","last_became_active":"2024-03-06T07:18:13.788435+0000","last_became_peered":"2024-03-06T07:18:13.788435+0000","last_unstale":"2024-03-06T07:18:14.720936+0000","last_undegraded":"2024-03-06T07:18:14.720936+0000","last_fullsized":"2024-03-06T07:18:14.720936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:36.842720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719105+0000","last_change":"2024-03-06T07:18:09.527698+0000","last_active":"2024-03-06T07:18:14.719105+0000","last_peered":"2024-03-06T07:18:14.719105+0000","last_clean":"2024-03-06T07:18:14.719105+0000","last_became_active":"2024-03-06T07:18:09.527407+0000","last_became_peered":"2024-03-06T07:18:09.527407+0000","last_unstale":"2024-03-06T07:18:14.719105+0000","last_undegraded":"2024-03-06T07:18:14.719105+0000","last_fullsized":"2024-03-06T07:18:14.719105+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718447+0000","last_change":"2024-03-06T07:18:08.518380+0000","last_active":"2024-03-06T07:18:14.718447+0000","last_peered":"2024-03-06T07:18:14.718447+0000","last_clean":"2024-03-06T07:18:14.718447+0000","last_became_active":"2024-03-06T07:18:08.517571+0000","last_became_peered":"2024-03-06T07:18:08.517571+0000","last_unstale":"2024-03-06T07:18:14.718447+0000","last_undegraded":"2024-03-06T07:18:14.718447+0000","last_fullsized":"2024-03-06T07:18:14.718447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720838+0000","last_change":"2024-03-06T07:18:13.785111+0000","last_active":"2024-03-06T07:18:14.720838+0000","last_peered":"2024-03-06T07:18:14.720838+0000","last_clean":"2024-03-06T07:18:14.720838+0000","last_became_active":"2024-03-06T07:18:13.784751+0000","last_became_peered":"2024-03-06T07:18:13.784751+0000","last_unstale":"2024-03-06T07:18:14.720838+0000","last_undegraded":"2024-03-06T07:18:14.720838+0000","last_fullsized":"2024-03-06T07:18:14.720838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.733802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719140+0000","last_change":"2024-03-06T07:18:09.526243+0000","last_active":"2024-03-06T07:18:14.719140+0000","last_peered":"2024-03-06T07:18:14.719140+0000","last_clean":"2024-03-06T07:18:14.719140+0000","last_became_active":"2024-03-06T07:18:09.525985+0000","last_became_peered":"2024-03-06T07:18:09.525985+0000","last_unstale":"2024-03-06T07:18:14.719140+0000","last_undegraded":"2024-03-06T07:18:14.719140+0000","last_fullsized":"2024-03-06T07:18:14.719140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:20.656356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719309+0000","last_change":"2024-03-06T07:18:08.520844+0000","last_active":"2024-03-06T07:18:14.719309+0000","last_peered":"2024-03-06T07:18:14.719309+0000","last_clean":"2024-03-06T07:18:14.719309+0000","last_became_active":"2024-03-06T07:18:08.520318+0000","last_became_peered":"2024-03-06T07:18:08.520318+0000","last_unstale":"2024-03-06T07:18:14.719309+0000","last_undegraded":"2024-03-06T07:18:14.719309+0000","last_fullsized":"2024-03-06T07:18:14.719309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:28.274132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719091+0000","last_change":"2024-03-06T07:18:13.785334+0000","last_active":"2024-03-06T07:18:14.719091+0000","last_peered":"2024-03-06T07:18:14.719091+0000","last_clean":"2024-03-06T07:18:14.719091+0000","last_became_active":"2024-03-06T07:18:13.784953+0000","last_became_peered":"2024-03-06T07:18:13.784953+0000","last_unstale":"2024-03-06T07:18:14.719091+0000","last_undegraded":"2024-03-06T07:18:14.719091+0000","last_fullsized":"2024-03-06T07:18:14.719091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:18.038907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718773+0000","last_change":"2024-03-06T07:18:09.533771+0000","last_active":"2024-03-06T07:18:14.718773+0000","last_peered":"2024-03-06T07:18:14.718773+0000","last_clean":"2024-03-06T07:18:14.718773+0000","last_became_active":"2024-03-06T07:18:09.533520+0000","last_became_peered":"2024-03-06T07:18:09.533520+0000","last_unstale":"2024-03-06T07:18:14.718773+0000","last_undegraded":"2024-03-06T07:18:14.718773+0000","last_fullsized":"2024-03-06T07:18:14.718773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:18.912751+0000","last_change":"2024-03-06T07:18:08.511813+0000","last_active":"2024-03-06T07:18:18.912751+0000","last_peered":"2024-03-06T07:18:18.912751+0000","last_clean":"2024-03-06T07:18:18.912751+0000","last_became_active":"2024-03-06T07:18:08.511552+0000","last_became_peered":"2024-03-06T07:18:08.511552+0000","last_unstale":"2024-03-06T07:18:18.912751+0000","last_undegraded":"2024-03-06T07:18:18.912751+0000","last_fullsized":"2024-03-06T07:18:18.912751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718960+0000","last_change":"2024-03-06T07:18:13.567483+0000","last_active":"2024-03-06T07:18:14.718960+0000","last_peered":"2024-03-06T07:18:14.718960+0000","last_clean":"2024-03-06T07:18:14.718960+0000","last_became_active":"2024-03-06T07:18:13.566829+0000","last_became_peered":"2024-03-06T07:18:13.566829+0000","last_unstale":"2024-03-06T07:18:14.718960+0000","last_undegraded":"2024-03-06T07:18:14.718960+0000","last_fullsized":"2024-03-06T07:18:14.718960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551865+0000","last_change":"2024-03-06T07:18:09.525404+0000","last_active":"2024-03-06T07:18:14.551865+0000","last_peered":"2024-03-06T07:18:14.551865+0000","last_clean":"2024-03-06T07:18:14.551865+0000","last_became_active":"2024-03-06T07:18:09.525127+0000","last_became_peered":"2024-03-06T07:18:09.525127+0000","last_unstale":"2024-03-06T07:18:14.551865+0000","last_undegraded":"2024-03-06T07:18:14.551865+0000","last_fullsized":"2024-03-06T07:18:14.551865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718876+0000","last_change":"2024-03-06T07:18:08.518033+0000","last_active":"2024-03-06T07:18:14.718876+0000","last_peered":"2024-03-06T07:18:14.718876+0000","last_clean":"2024-03-06T07:18:14.718876+0000","last_became_active":"2024-03-06T07:18:08.517752+0000","last_became_peered":"2024-03-06T07:18:08.517752+0000","last_unstale":"2024-03-06T07:18:14.718876+0000","last_undegraded":"2024-03-06T07:18:14.718876+0000","last_fullsized":"2024-03-06T07:18:14.718876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551959+0000","last_change":"2024-03-06T07:18:13.561392+0000","last_active":"2024-03-06T07:18:14.551959+0000","last_peered":"2024-03-06T07:18:14.551959+0000","last_clean":"2024-03-06T07:18:14.551959+0000","last_became_active":"2024-03-06T07:18:13.561014+0000","last_became_peered":"2024-03-06T07:18:13.561014+0000","last_unstale":"2024-03-06T07:18:14.551959+0000","last_undegraded":"2024-03-06T07:18:14.551959+0000","last_fullsized":"2024-03-06T07:18:14.551959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:43.183715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719832+0000","last_change":"2024-03-06T07:18:09.531541+0000","last_active":"2024-03-06T07:18:14.719832+0000","last_peered":"2024-03-06T07:18:14.719832+0000","last_clean":"2024-03-06T07:18:14.719832+0000","last_became_active":"2024-03-06T07:18:09.531248+0000","last_became_peered":"2024-03-06T07:18:09.531248+0000","last_unstale":"2024-03-06T07:18:14.719832+0000","last_undegraded":"2024-03-06T07:18:14.719832+0000","last_fullsized":"2024-03-06T07:18:14.719832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719237+0000","last_change":"2024-03-06T07:18:08.519552+0000","last_active":"2024-03-06T07:18:14.719237+0000","last_peered":"2024-03-06T07:18:14.719237+0000","last_clean":"2024-03-06T07:18:14.719237+0000","last_became_active":"2024-03-06T07:18:08.519273+0000","last_became_peered":"2024-03-06T07:18:08.519273+0000","last_unstale":"2024-03-06T07:18:14.719237+0000","last_undegraded":"2024-03-06T07:18:14.719237+0000","last_fullsized":"2024-03-06T07:18:14.719237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720395+0000","last_change":"2024-03-06T07:18:13.564953+0000","last_active":"2024-03-06T07:18:14.720395+0000","last_peered":"2024-03-06T07:18:14.720395+0000","last_clean":"2024-03-06T07:18:14.720395+0000","last_became_active":"2024-03-06T07:18:13.564601+0000","last_became_peered":"2024-03-06T07:18:13.564601+0000","last_unstale":"2024-03-06T07:18:14.720395+0000","last_undegraded":"2024-03-06T07:18:14.720395+0000","last_fullsized":"2024-03-06T07:18:14.720395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:31.246872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718301+0000","last_change":"2024-03-06T07:18:09.535125+0000","last_active":"2024-03-06T07:18:14.718301+0000","last_peered":"2024-03-06T07:18:14.718301+0000","last_clean":"2024-03-06T07:18:14.718301+0000","last_became_active":"2024-03-06T07:18:09.534863+0000","last_became_peered":"2024-03-06T07:18:09.534863+0000","last_unstale":"2024-03-06T07:18:14.718301+0000","last_undegraded":"2024-03-06T07:18:14.718301+0000","last_fullsized":"2024-03-06T07:18:14.718301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551568+0000","last_change":"2024-03-06T07:18:08.517122+0000","last_active":"2024-03-06T07:18:14.551568+0000","last_peered":"2024-03-06T07:18:14.551568+0000","last_clean":"2024-03-06T07:18:14.551568+0000","last_became_active":"2024-03-06T07:18:08.516566+0000","last_became_peered":"2024-03-06T07:18:08.516566+0000","last_unstale":"2024-03-06T07:18:14.551568+0000","last_undegraded":"2024-03-06T07:18:14.551568+0000","last_fullsized":"2024-03-06T07:18:14.551568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720688+0000","last_change":"2024-03-06T07:18:13.789985+0000","last_active":"2024-03-06T07:18:14.720688+0000","last_peered":"2024-03-06T07:18:14.720688+0000","last_clean":"2024-03-06T07:18:14.720688+0000","last_became_active":"2024-03-06T07:18:13.789627+0000","last_became_peered":"2024-03-06T07:18:13.789627+0000","last_unstale":"2024-03-06T07:18:14.720688+0000","last_undegraded":"2024-03-06T07:18:14.720688+0000","last_fullsized":"2024-03-06T07:18:14.720688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:22:45.216688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718902+0000","last_change":"2024-03-06T07:18:13.786557+0000","last_active":"2024-03-06T07:18:14.718902+0000","last_peered":"2024-03-06T07:18:14.718902+0000","last_clean":"2024-03-06T07:18:14.718902+0000","last_became_active":"2024-03-06T07:18:13.786179+0000","last_became_peered":"2024-03-06T07:18:13.786179+0000","last_unstale":"2024-03-06T07:18:14.718902+0000","last_undegraded":"2024-03-06T07:18:14.718902+0000","last_fullsized":"2024-03-06T07:18:14.718902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719435+0000","last_change":"2024-03-06T07:18:09.528134+0000","last_active":"2024-03-06T07:18:14.719435+0000","last_peered":"2024-03-06T07:18:14.719435+0000","last_clean":"2024-03-06T07:18:14.719435+0000","last_became_active":"2024-03-06T07:18:09.527831+0000","last_became_peered":"2024-03-06T07:18:09.527831+0000","last_unstale":"2024-03-06T07:18:14.719435+0000","last_undegraded":"2024-03-06T07:18:14.719435+0000","last_fullsized":"2024-03-06T07:18:14.719435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718145+0000","last_change":"2024-03-06T07:18:08.520786+0000","last_active":"2024-03-06T07:18:14.718145+0000","last_peered":"2024-03-06T07:18:14.718145+0000","last_clean":"2024-03-06T07:18:14.718145+0000","last_became_active":"2024-03-06T07:18:08.520392+0000","last_became_peered":"2024-03-06T07:18:08.520392+0000","last_unstale":"2024-03-06T07:18:14.718145+0000","last_undegraded":"2024-03-06T07:18:14.718145+0000","last_fullsized":"2024-03-06T07:18:14.718145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551680+0000","last_change":"2024-03-06T07:18:13.566639+0000","last_active":"2024-03-06T07:18:14.551680+0000","last_peered":"2024-03-06T07:18:14.551680+0000","last_clean":"2024-03-06T07:18:14.551680+0000","last_became_active":"2024-03-06T07:18:13.566321+0000","last_became_peered":"2024-03-06T07:18:13.566321+0000","last_unstale":"2024-03-06T07:18:14.551680+0000","last_undegraded":"2024-03-06T07:18:14.551680+0000","last_fullsized":"2024-03-06T07:18:14.551680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720366+0000","last_change":"2024-03-06T07:18:09.523023+0000","last_active":"2024-03-06T07:18:14.720366+0000","last_peered":"2024-03-06T07:18:14.720366+0000","last_clean":"2024-03-06T07:18:14.720366+0000","last_became_active":"2024-03-06T07:18:09.522736+0000","last_became_peered":"2024-03-06T07:18:09.522736+0000","last_unstale":"2024-03-06T07:18:14.720366+0000","last_undegraded":"2024-03-06T07:18:14.720366+0000","last_fullsized":"2024-03-06T07:18:14.720366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720831+0000","last_change":"2024-03-06T07:18:08.516225+0000","last_active":"2024-03-06T07:18:14.720831+0000","last_peered":"2024-03-06T07:18:14.720831+0000","last_clean":"2024-03-06T07:18:14.720831+0000","last_became_active":"2024-03-06T07:18:08.515908+0000","last_became_peered":"2024-03-06T07:18:08.515908+0000","last_unstale":"2024-03-06T07:18:14.720831+0000","last_undegraded":"2024-03-06T07:18:14.720831+0000","last_fullsized":"2024-03-06T07:18:14.720831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550914+0000","last_change":"2024-03-06T07:18:08.513484+0000","last_active":"2024-03-06T07:18:14.550914+0000","last_peered":"2024-03-06T07:18:14.550914+0000","last_clean":"2024-03-06T07:18:14.550914+0000","last_became_active":"2024-03-06T07:18:08.513195+0000","last_became_peered":"2024-03-06T07:18:08.513195+0000","last_unstale":"2024-03-06T07:18:14.550914+0000","last_undegraded":"2024-03-06T07:18:14.550914+0000","last_fullsized":"2024-03-06T07:18:14.550914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719247+0000","last_change":"2024-03-06T07:18:09.518972+0000","last_active":"2024-03-06T07:18:14.719247+0000","last_peered":"2024-03-06T07:18:14.719247+0000","last_clean":"2024-03-06T07:18:14.719247+0000","last_became_active":"2024-03-06T07:18:09.518673+0000","last_became_peered":"2024-03-06T07:18:09.518673+0000","last_unstale":"2024-03-06T07:18:14.719247+0000","last_undegraded":"2024-03-06T07:18:14.719247+0000","last_fullsized":"2024-03-06T07:18:14.719247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548431+0000","last_change":"2024-03-06T07:18:13.569839+0000","last_active":"2024-03-06T07:18:14.548431+0000","last_peered":"2024-03-06T07:18:14.548431+0000","last_clean":"2024-03-06T07:18:14.548431+0000","last_became_active":"2024-03-06T07:18:13.569443+0000","last_became_peered":"2024-03-06T07:18:13.569443+0000","last_unstale":"2024-03-06T07:18:14.548431+0000","last_undegraded":"2024-03-06T07:18:14.548431+0000","last_fullsized":"2024-03-06T07:18:14.548431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:45.425994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720231+0000","last_change":"2024-03-06T07:18:08.519883+0000","last_active":"2024-03-06T07:18:14.720231+0000","last_peered":"2024-03-06T07:18:14.720231+0000","last_clean":"2024-03-06T07:18:14.720231+0000","last_became_active":"2024-03-06T07:18:08.519579+0000","last_became_peered":"2024-03-06T07:18:08.519579+0000","last_unstale":"2024-03-06T07:18:14.720231+0000","last_undegraded":"2024-03-06T07:18:14.720231+0000","last_fullsized":"2024-03-06T07:18:14.720231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551887+0000","last_change":"2024-03-06T07:18:09.520603+0000","last_active":"2024-03-06T07:18:14.551887+0000","last_peered":"2024-03-06T07:18:14.551887+0000","last_clean":"2024-03-06T07:18:14.551887+0000","last_became_active":"2024-03-06T07:18:09.520258+0000","last_became_peered":"2024-03-06T07:18:09.520258+0000","last_unstale":"2024-03-06T07:18:14.551887+0000","last_undegraded":"2024-03-06T07:18:14.551887+0000","last_fullsized":"2024-03-06T07:18:14.551887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720637+0000","last_change":"2024-03-06T07:18:13.564140+0000","last_active":"2024-03-06T07:18:14.720637+0000","last_peered":"2024-03-06T07:18:14.720637+0000","last_clean":"2024-03-06T07:18:14.720637+0000","last_became_active":"2024-03-06T07:18:13.563770+0000","last_became_peered":"2024-03-06T07:18:13.563770+0000","last_unstale":"2024-03-06T07:18:14.720637+0000","last_undegraded":"2024-03-06T07:18:14.720637+0000","last_fullsized":"2024-03-06T07:18:14.720637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:03.956217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550619+0000","last_change":"2024-03-06T07:18:09.520176+0000","last_active":"2024-03-06T07:18:14.550619+0000","last_peered":"2024-03-06T07:18:14.550619+0000","last_clean":"2024-03-06T07:18:14.550619+0000","last_became_active":"2024-03-06T07:18:09.519911+0000","last_became_peered":"2024-03-06T07:18:09.519911+0000","last_unstale":"2024-03-06T07:18:14.550619+0000","last_undegraded":"2024-03-06T07:18:14.550619+0000","last_fullsized":"2024-03-06T07:18:14.550619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548193+0000","last_change":"2024-03-06T07:18:09.524681+0000","last_active":"2024-03-06T07:18:14.548193+0000","last_peered":"2024-03-06T07:18:14.548193+0000","last_clean":"2024-03-06T07:18:14.548193+0000","last_became_active":"2024-03-06T07:18:09.524427+0000","last_became_peered":"2024-03-06T07:18:09.524427+0000","last_unstale":"2024-03-06T07:18:14.548193+0000","last_undegraded":"2024-03-06T07:18:14.548193+0000","last_fullsized":"2024-03-06T07:18:14.548193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.531928+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531670+0000","last_became_peered":"2024-03-06T07:18:09.531670+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719143+0000","last_change":"2024-03-06T07:18:09.523431+0000","last_active":"2024-03-06T07:18:14.719143+0000","last_peered":"2024-03-06T07:18:14.719143+0000","last_clean":"2024-03-06T07:18:14.719143+0000","last_became_active":"2024-03-06T07:18:09.523169+0000","last_became_peered":"2024-03-06T07:18:09.523169+0000","last_unstale":"2024-03-06T07:18:14.719143+0000","last_undegraded":"2024-03-06T07:18:14.719143+0000","last_fullsized":"2024-03-06T07:18:14.719143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720897+0000","last_change":"2024-03-06T07:18:09.525359+0000","last_active":"2024-03-06T07:18:14.720897+0000","last_peered":"2024-03-06T07:18:14.720897+0000","last_clean":"2024-03-06T07:18:14.720897+0000","last_became_active":"2024-03-06T07:18:09.525069+0000","last_became_peered":"2024-03-06T07:18:09.525069+0000","last_unstale":"2024-03-06T07:18:14.720897+0000","last_undegraded":"2024-03-06T07:18:14.720897+0000","last_fullsized":"2024-03-06T07:18:14.720897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718709+0000","last_change":"2024-03-06T07:18:09.528783+0000","last_active":"2024-03-06T07:18:14.718709+0000","last_peered":"2024-03-06T07:18:14.718709+0000","last_clean":"2024-03-06T07:18:14.718709+0000","last_became_active":"2024-03-06T07:18:09.528488+0000","last_became_peered":"2024-03-06T07:18:09.528488+0000","last_unstale":"2024-03-06T07:18:14.718709+0000","last_undegraded":"2024-03-06T07:18:14.718709+0000","last_fullsized":"2024-03-06T07:18:14.718709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551739+0000","last_change":"2024-03-06T07:18:09.526930+0000","last_active":"2024-03-06T07:18:14.551739+0000","last_peered":"2024-03-06T07:18:14.551739+0000","last_clean":"2024-03-06T07:18:14.551739+0000","last_became_active":"2024-03-06T07:18:09.526575+0000","last_became_peered":"2024-03-06T07:18:09.526575+0000","last_unstale":"2024-03-06T07:18:14.551739+0000","last_undegraded":"2024-03-06T07:18:14.551739+0000","last_fullsized":"2024-03-06T07:18:14.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720809+0000","last_change":"2024-03-06T07:18:09.532541+0000","last_active":"2024-03-06T07:18:14.720809+0000","last_peered":"2024-03-06T07:18:14.720809+0000","last_clean":"2024-03-06T07:18:14.720809+0000","last_became_active":"2024-03-06T07:18:09.532260+0000","last_became_peered":"2024-03-06T07:18:09.532260+0000","last_unstale":"2024-03-06T07:18:14.720809+0000","last_undegraded":"2024-03-06T07:18:14.720809+0000","last_fullsized":"2024-03-06T07:18:14.720809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718061+0000","last_change":"2024-03-06T07:18:09.530812+0000","last_active":"2024-03-06T07:18:14.718061+0000","last_peered":"2024-03-06T07:18:14.718061+0000","last_clean":"2024-03-06T07:18:14.718061+0000","last_became_active":"2024-03-06T07:18:09.530493+0000","last_became_peered":"2024-03-06T07:18:09.530493+0000","last_unstale":"2024-03-06T07:18:14.718061+0000","last_undegraded":"2024-03-06T07:18:14.718061+0000","last_fullsized":"2024-03-06T07:18:14.718061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720741+0000","last_change":"2024-03-06T07:18:09.525260+0000","last_active":"2024-03-06T07:18:14.720741+0000","last_peered":"2024-03-06T07:18:14.720741+0000","last_clean":"2024-03-06T07:18:14.720741+0000","last_became_active":"2024-03-06T07:18:09.524996+0000","last_became_peered":"2024-03-06T07:18:09.524996+0000","last_unstale":"2024-03-06T07:18:14.720741+0000","last_undegraded":"2024-03-06T07:18:14.720741+0000","last_fullsized":"2024-03-06T07:18:14.720741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718326+0000","last_change":"2024-03-06T07:18:09.539784+0000","last_active":"2024-03-06T07:18:14.718326+0000","last_peered":"2024-03-06T07:18:14.718326+0000","last_clean":"2024-03-06T07:18:14.718326+0000","last_became_active":"2024-03-06T07:18:09.538642+0000","last_became_peered":"2024-03-06T07:18:09.538642+0000","last_unstale":"2024-03-06T07:18:14.718326+0000","last_undegraded":"2024-03-06T07:18:14.718326+0000","last_fullsized":"2024-03-06T07:18:14.718326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719031+0000","last_change":"2024-03-06T07:18:09.526156+0000","last_active":"2024-03-06T07:18:14.719031+0000","last_peered":"2024-03-06T07:18:14.719031+0000","last_clean":"2024-03-06T07:18:14.719031+0000","last_became_active":"2024-03-06T07:18:09.525877+0000","last_became_peered":"2024-03-06T07:18:09.525877+0000","last_unstale":"2024-03-06T07:18:14.719031+0000","last_undegraded":"2024-03-06T07:18:14.719031+0000","last_fullsized":"2024-03-06T07:18:14.719031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551188+0000","last_change":"2024-03-06T07:18:09.516685+0000","last_active":"2024-03-06T07:18:14.551188+0000","last_peered":"2024-03-06T07:18:14.551188+0000","last_clean":"2024-03-06T07:18:14.551188+0000","last_became_active":"2024-03-06T07:18:09.516377+0000","last_became_peered":"2024-03-06T07:18:09.516377+0000","last_unstale":"2024-03-06T07:18:14.551188+0000","last_undegraded":"2024-03-06T07:18:14.551188+0000","last_fullsized":"2024-03-06T07:18:14.551188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718906+0000","last_change":"2024-03-06T07:18:09.524856+0000","last_active":"2024-03-06T07:18:14.718906+0000","last_peered":"2024-03-06T07:18:14.718906+0000","last_clean":"2024-03-06T07:18:14.718906+0000","last_became_active":"2024-03-06T07:18:09.523916+0000","last_became_peered":"2024-03-06T07:18:09.523916+0000","last_unstale":"2024-03-06T07:18:14.718906+0000","last_undegraded":"2024-03-06T07:18:14.718906+0000","last_fullsized":"2024-03-06T07:18:14.718906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548544+0000","last_change":"2024-03-06T07:18:09.518046+0000","last_active":"2024-03-06T07:18:14.548544+0000","last_peered":"2024-03-06T07:18:14.548544+0000","last_clean":"2024-03-06T07:18:14.548544+0000","last_became_active":"2024-03-06T07:18:09.517756+0000","last_became_peered":"2024-03-06T07:18:09.517756+0000","last_unstale":"2024-03-06T07:18:14.548544+0000","last_undegraded":"2024-03-06T07:18:14.548544+0000","last_fullsized":"2024-03-06T07:18:14.548544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719287+0000","last_change":"2024-03-06T07:18:09.537036+0000","last_active":"2024-03-06T07:18:14.719287+0000","last_peered":"2024-03-06T07:18:14.719287+0000","last_clean":"2024-03-06T07:18:14.719287+0000","last_became_active":"2024-03-06T07:18:09.536576+0000","last_became_peered":"2024-03-06T07:18:09.536576+0000","last_unstale":"2024-03-06T07:18:14.719287+0000","last_undegraded":"2024-03-06T07:18:14.719287+0000","last_fullsized":"2024-03-06T07:18:14.719287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.717960+0000","last_change":"2024-03-06T07:18:09.527339+0000","last_active":"2024-03-06T07:18:14.717960+0000","last_peered":"2024-03-06T07:18:14.717960+0000","last_clean":"2024-03-06T07:18:14.717960+0000","last_became_active":"2024-03-06T07:18:09.526873+0000","last_became_peered":"2024-03-06T07:18:09.526873+0000","last_unstale":"2024-03-06T07:18:14.717960+0000","last_undegraded":"2024-03-06T07:18:14.717960+0000","last_fullsized":"2024-03-06T07:18:14.717960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719530+0000","last_change":"2024-03-06T07:18:09.522712+0000","last_active":"2024-03-06T07:18:14.719530+0000","last_peered":"2024-03-06T07:18:14.719530+0000","last_clean":"2024-03-06T07:18:14.719530+0000","last_became_active":"2024-03-06T07:18:09.522433+0000","last_became_peered":"2024-03-06T07:18:09.522433+0000","last_unstale":"2024-03-06T07:18:14.719530+0000","last_undegraded":"2024-03-06T07:18:14.719530+0000","last_fullsized":"2024-03-06T07:18:14.719530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719097+0000","last_change":"2024-03-06T07:18:09.522057+0000","last_active":"2024-03-06T07:18:14.719097+0000","last_peered":"2024-03-06T07:18:14.719097+0000","last_clean":"2024-03-06T07:18:14.719097+0000","last_became_active":"2024-03-06T07:18:09.521270+0000","last_became_peered":"2024-03-06T07:18:09.521270+0000","last_unstale":"2024-03-06T07:18:14.719097+0000","last_undegraded":"2024-03-06T07:18:14.719097+0000","last_fullsized":"2024-03-06T07:18:14.719097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:57.186351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720044+0000","last_change":"2024-03-06T07:18:09.526002+0000","last_active":"2024-03-06T07:18:14.720044+0000","last_peered":"2024-03-06T07:18:14.720044+0000","last_clean":"2024-03-06T07:18:14.720044+0000","last_became_active":"2024-03-06T07:18:09.525717+0000","last_became_peered":"2024-03-06T07:18:09.525717+0000","last_unstale":"2024-03-06T07:18:14.720044+0000","last_undegraded":"2024-03-06T07:18:14.720044+0000","last_fullsized":"2024-03-06T07:18:14.720044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551860+0000","last_change":"2024-03-06T07:18:09.521766+0000","last_active":"2024-03-06T07:18:14.551860+0000","last_peered":"2024-03-06T07:18:14.551860+0000","last_clean":"2024-03-06T07:18:14.551860+0000","last_became_active":"2024-03-06T07:18:09.521159+0000","last_became_peered":"2024-03-06T07:18:09.521159+0000","last_unstale":"2024-03-06T07:18:14.551860+0000","last_undegraded":"2024-03-06T07:18:14.551860+0000","last_fullsized":"2024-03-06T07:18:14.551860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:57.265260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551251+0000","last_change":"2024-03-06T07:18:09.533157+0000","last_active":"2024-03-06T07:18:14.551251+0000","last_peered":"2024-03-06T07:18:14.551251+0000","last_clean":"2024-03-06T07:18:14.551251+0000","last_became_active":"2024-03-06T07:18:09.532868+0000","last_became_peered":"2024-03-06T07:18:09.532868+0000","last_unstale":"2024-03-06T07:18:14.551251+0000","last_undegraded":"2024-03-06T07:18:14.551251+0000","last_fullsized":"2024-03-06T07:18:14.551251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551648+0000","last_change":"2024-03-06T07:18:09.532473+0000","last_active":"2024-03-06T07:18:14.551648+0000","last_peered":"2024-03-06T07:18:14.551648+0000","last_clean":"2024-03-06T07:18:14.551648+0000","last_became_active":"2024-03-06T07:18:09.532191+0000","last_became_peered":"2024-03-06T07:18:09.532191+0000","last_unstale":"2024-03-06T07:18:14.551648+0000","last_undegraded":"2024-03-06T07:18:14.551648+0000","last_fullsized":"2024-03-06T07:18:14.551648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719062+0000","last_change":"2024-03-06T07:18:09.533061+0000","last_active":"2024-03-06T07:18:14.719062+0000","last_peered":"2024-03-06T07:18:14.719062+0000","last_clean":"2024-03-06T07:18:14.719062+0000","last_became_active":"2024-03-06T07:18:09.532742+0000","last_became_peered":"2024-03-06T07:18:09.532742+0000","last_unstale":"2024-03-06T07:18:14.719062+0000","last_undegraded":"2024-03-06T07:18:14.719062+0000","last_fullsized":"2024-03-06T07:18:14.719062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:25.345923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720283+0000","last_change":"2024-03-06T07:18:09.532342+0000","last_active":"2024-03-06T07:18:14.720283+0000","last_peered":"2024-03-06T07:18:14.720283+0000","last_clean":"2024-03-06T07:18:14.720283+0000","last_became_active":"2024-03-06T07:18:09.532062+0000","last_became_peered":"2024-03-06T07:18:09.532062+0000","last_unstale":"2024-03-06T07:18:14.720283+0000","last_undegraded":"2024-03-06T07:18:14.720283+0000","last_fullsized":"2024-03-06T07:18:14.720283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718610+0000","last_change":"2024-03-06T07:18:09.525768+0000","last_active":"2024-03-06T07:18:14.718610+0000","last_peered":"2024-03-06T07:18:14.718610+0000","last_clean":"2024-03-06T07:18:14.718610+0000","last_became_active":"2024-03-06T07:18:09.524604+0000","last_became_peered":"2024-03-06T07:18:09.524604+0000","last_unstale":"2024-03-06T07:18:14.718610+0000","last_undegraded":"2024-03-06T07:18:14.718610+0000","last_fullsized":"2024-03-06T07:18:14.718610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718219+0000","last_change":"2024-03-06T07:18:09.531193+0000","last_active":"2024-03-06T07:18:14.718219+0000","last_peered":"2024-03-06T07:18:14.718219+0000","last_clean":"2024-03-06T07:18:14.718219+0000","last_became_active":"2024-03-06T07:18:09.530931+0000","last_became_peered":"2024-03-06T07:18:09.530931+0000","last_unstale":"2024-03-06T07:18:14.718219+0000","last_undegraded":"2024-03-06T07:18:14.718219+0000","last_fullsized":"2024-03-06T07:18:14.718219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718236+0000","last_change":"2024-03-06T07:18:09.524492+0000","last_active":"2024-03-06T07:18:14.718236+0000","last_peered":"2024-03-06T07:18:14.718236+0000","last_clean":"2024-03-06T07:18:14.718236+0000","last_became_active":"2024-03-06T07:18:09.524202+0000","last_became_peered":"2024-03-06T07:18:09.524202+0000","last_unstale":"2024-03-06T07:18:14.718236+0000","last_undegraded":"2024-03-06T07:18:14.718236+0000","last_fullsized":"2024-03-06T07:18:14.718236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.532118+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531771+0000","last_became_peered":"2024-03-06T07:18:09.531771+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719327+0000","last_change":"2024-03-06T07:18:09.519755+0000","last_active":"2024-03-06T07:18:14.719327+0000","last_peered":"2024-03-06T07:18:14.719327+0000","last_clean":"2024-03-06T07:18:14.719327+0000","last_became_active":"2024-03-06T07:18:09.519475+0000","last_became_peered":"2024-03-06T07:18:09.519475+0000","last_unstale":"2024-03-06T07:18:14.719327+0000","last_undegraded":"2024-03-06T07:18:14.719327+0000","last_fullsized":"2024-03-06T07:18:14.719327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719345+0000","last_change":"2024-03-06T07:18:09.524863+0000","last_active":"2024-03-06T07:18:14.719345+0000","last_peered":"2024-03-06T07:18:14.719345+0000","last_clean":"2024-03-06T07:18:14.719345+0000","last_became_active":"2024-03-06T07:18:09.524588+0000","last_became_peered":"2024-03-06T07:18:09.524588+0000","last_unstale":"2024-03-06T07:18:14.719345+0000","last_undegraded":"2024-03-06T07:18:14.719345+0000","last_fullsized":"2024-03-06T07:18:14.719345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718889+0000","last_change":"2024-03-06T07:18:09.528727+0000","last_active":"2024-03-06T07:18:14.718889+0000","last_peered":"2024-03-06T07:18:14.718889+0000","last_clean":"2024-03-06T07:18:14.718889+0000","last_became_active":"2024-03-06T07:18:09.528474+0000","last_became_peered":"2024-03-06T07:18:09.528474+0000","last_unstale":"2024-03-06T07:18:14.718889+0000","last_undegraded":"2024-03-06T07:18:14.718889+0000","last_fullsized":"2024-03-06T07:18:14.718889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:18:32.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:18:32.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.812+0000 7f4ab0513640 1 Processor -- start 2024-03-06T07:18:32.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.812+0000 7f4ab0513640 1 -- start start 2024-03-06T07:18:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.813+0000 7f4ab0513640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4aa8159100 0x7f4aa80c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.813+0000 7f4ab0513640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa80c7c60 con 0x7f4aa81527b0 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.813+0000 7f4ab0513640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa80c8060 con 0x7f4aa80c7180 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.813+0000 7f4ab0513640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa80c8df0 con 0x7f4aa8159100 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.813+0000 7f4aaeab0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4aa8159100 0x7f4aa80c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.813+0000 7f4aae2af640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4aa81527b0 0x7f4aa8155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41418/0 (socket says 172.21.15.94:41418) 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.813+0000 7f4aae2af640 1 -- 172.21.15.94:0/2173915967 learned_addr learned my addr 172.21.15.94:0/2173915967 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aaeab0640 1 -- 172.21.15.94:0/2173915967 >> v1:172.21.15.145:6789/0 conn(0x7f4aa80c7180 legacy=0x7f4aa80c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aaeab0640 1 -- 172.21.15.94:0/2173915967 >> v1:172.21.15.94:6789/0 conn(0x7f4aa81527b0 legacy=0x7f4aa8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aad2ad640 1 -- 172.21.15.94:0/2173915967 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762165518 0 0) 0x7f4aa4001490 con 0x7f4aa81527b0 2024-03-06T07:18:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aaeab0640 1 -- 172.21.15.94:0/2173915967 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa80c76a0 con 0x7f4aa8159100 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aaeab0640 1 --2- 172.21.15.94:0/2173915967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa80c88b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7982a375796a2559 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aad2ad640 1 -- 172.21.15.94:0/2173915967 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0035020 con 0x7f4aa8159100 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aad2ad640 1 -- 172.21.15.94:0/2173915967 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0031830 con 0x7f4aa8159100 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.814+0000 7f4aad2ad640 1 -- 172.21.15.94:0/2173915967 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0031c00 con 0x7f4aa8159100 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.815+0000 7f4ab0513640 1 -- 172.21.15.94:0/2173915967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 msgr2=0x7f4aa80c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.815+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2173915967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa80c88b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.815+0000 7f4ab0513640 1 -- 172.21.15.94:0/2173915967 shutdown_connections 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.815+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2173915967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa80c88b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.815+0000 7f4ab0513640 1 -- 172.21.15.94:0/2173915967 >> 172.21.15.94:0/2173915967 conn(0x7f4aa80ac560 msgr2=0x7f4aa80ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.815+0000 7f4ab0513640 1 -- 172.21.15.94:0/2173915967 shutdown_connections 2024-03-06T07:18:32.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.815+0000 7f4ab0513640 1 -- 172.21.15.94:0/2173915967 wait complete. 2024-03-06T07:18:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.816+0000 7f4ab0513640 1 Processor -- start 2024-03-06T07:18:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.816+0000 7f4ab0513640 1 -- start start 2024-03-06T07:18:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4ab0513640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aa80c7180 0x7f4aa81eeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4ab0513640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aa81527b0 0x7f4aa81580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4ab0513640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa814b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4ab0513640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8155600 con 0x7f4aa8159100 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4ab0513640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8155320 con 0x7f4aa80c7180 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4ab0513640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8152b90 con 0x7f4aa81527b0 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa814b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa814b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40772/0 (socket says 172.21.15.94:40772) 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aae2af640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aa80c7180 0x7f4aa81eeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 -- 172.21.15.94:0/2392101362 learned_addr learned my addr 172.21.15.94:0/2392101362 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aadaae640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aa81527b0 0x7f4aa81580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 -- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aa81527b0 msgr2=0x7f4aa81580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aa81527b0 0x7f4aa81580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 -- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aa80c7180 msgr2=0x7f4aa81eeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aa80c7180 0x7f4aa81eeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 -- 172.21.15.94:0/2392101362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa814f440 con 0x7f4aa8159100 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.817+0000 7f4aaeab0640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa814b9b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0034040 tx=0x7f4aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.818+0000 7f4a9effd640 1 -- 172.21.15.94:0/2392101362 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0041070 con 0x7f4aa8159100 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.818+0000 7f4a9effd640 1 -- 172.21.15.94:0/2392101362 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0004250 con 0x7f4aa8159100 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.818+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa814f720 con 0x7f4aa8159100 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.818+0000 7f4a9effd640 1 -- 172.21.15.94:0/2392101362 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0035040 con 0x7f4aa8159100 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.818+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa80c2ad0 con 0x7f4aa8159100 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.818+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa804f330 con 0x7f4aa8159100 2024-03-06T07:18:32.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.819+0000 7f4a9effd640 1 -- 172.21.15.94:0/2392101362 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4aa0038020 con 0x7f4aa8159100 2024-03-06T07:18:32.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.819+0000 7f4a9effd640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:32.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.819+0000 7f4a9effd640 1 -- 172.21.15.94:0/2392101362 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4aa00abf30 con 0x7f4aa8159100 2024-03-06T07:18:32.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.820+0000 7f4aae2af640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:32.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.820+0000 7f4aae2af640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40012e0 tx=0x7f4aa4007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:32.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.824+0000 7f4a9effd640 1 -- 172.21.15.94:0/2392101362 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4aa007a390 con 0x7f4aa8159100 2024-03-06T07:18:32.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:32.993+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4aa81504a0 con 0x7f4a8c06b4d0 2024-03-06T07:18:33.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.000+0000 7f4a9effd640 1 -- 172.21.15.94:0/2392101362 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332611 (secure 0 0 0) 0x7f4aa81504a0 con 0x7f4a8c06b4d0 2024-03-06T07:18:33.010 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:33.012 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a8c06b4d0 msgr2=0x7f4a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40012e0 tx=0x7f4aa4007ed0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 msgr2=0x7f4aa814b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa814b9b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0034040 tx=0x7f4aa00046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 shutdown_connections 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a8c06b4d0 0x7f4a8c06d990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4aa8159100 0x7f4aa814b9b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aa81527b0 0x7f4aa81580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 --2- 172.21.15.94:0/2392101362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aa80c7180 0x7f4aa81eeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 >> 172.21.15.94:0/2392101362 conn(0x7f4aa80ac560 msgr2=0x7f4aa8150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:33.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 shutdown_connections 2024-03-06T07:18:33.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.004+0000 7f4ab0513640 1 -- 172.21.15.94:0/2392101362 wait complete. 2024-03-06T07:18:33.017 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-06T07:18:32.639796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226688,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754748032,"statfs":{"total":773094113280,"available":772861984768,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002245"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719950+0000","last_change":"2024-03-06T07:18:08.523161+0000","last_active":"2024-03-06T07:18:14.719950+0000","last_peered":"2024-03-06T07:18:14.719950+0000","last_clean":"2024-03-06T07:18:14.719950+0000","last_became_active":"2024-03-06T07:18:08.522883+0000","last_became_peered":"2024-03-06T07:18:08.522883+0000","last_unstale":"2024-03-06T07:18:14.719950+0000","last_undegraded":"2024-03-06T07:18:14.719950+0000","last_fullsized":"2024-03-06T07:18:14.719950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720669+0000","last_change":"2024-03-06T07:18:09.526639+0000","last_active":"2024-03-06T07:18:14.720669+0000","last_peered":"2024-03-06T07:18:14.720669+0000","last_clean":"2024-03-06T07:18:14.720669+0000","last_became_active":"2024-03-06T07:18:09.526380+0000","last_became_peered":"2024-03-06T07:18:09.526380+0000","last_unstale":"2024-03-06T07:18:14.720669+0000","last_undegraded":"2024-03-06T07:18:14.720669+0000","last_fullsized":"2024-03-06T07:18:14.720669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720585+0000","last_change":"2024-03-06T07:18:13.788926+0000","last_active":"2024-03-06T07:18:14.720585+0000","last_peered":"2024-03-06T07:18:14.720585+0000","last_clean":"2024-03-06T07:18:14.720585+0000","last_became_active":"2024-03-06T07:18:13.788572+0000","last_became_peered":"2024-03-06T07:18:13.788572+0000","last_unstale":"2024-03-06T07:18:14.720585+0000","last_undegraded":"2024-03-06T07:18:14.720585+0000","last_fullsized":"2024-03-06T07:18:14.720585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:00.875495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720880+0000","last_change":"2024-03-06T07:18:08.517732+0000","last_active":"2024-03-06T07:18:14.720880+0000","last_peered":"2024-03-06T07:18:14.720880+0000","last_clean":"2024-03-06T07:18:14.720880+0000","last_became_active":"2024-03-06T07:18:08.517469+0000","last_became_peered":"2024-03-06T07:18:08.517469+0000","last_unstale":"2024-03-06T07:18:14.720880+0000","last_undegraded":"2024-03-06T07:18:14.720880+0000","last_fullsized":"2024-03-06T07:18:14.720880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719676+0000","last_change":"2024-03-06T07:18:09.531502+0000","last_active":"2024-03-06T07:18:14.719676+0000","last_peered":"2024-03-06T07:18:14.719676+0000","last_clean":"2024-03-06T07:18:14.719676+0000","last_became_active":"2024-03-06T07:18:09.531231+0000","last_became_peered":"2024-03-06T07:18:09.531231+0000","last_unstale":"2024-03-06T07:18:14.719676+0000","last_undegraded":"2024-03-06T07:18:14.719676+0000","last_fullsized":"2024-03-06T07:18:14.719676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551608+0000","last_change":"2024-03-06T07:18:13.565825+0000","last_active":"2024-03-06T07:18:14.551608+0000","last_peered":"2024-03-06T07:18:14.551608+0000","last_clean":"2024-03-06T07:18:14.551608+0000","last_became_active":"2024-03-06T07:18:13.565250+0000","last_became_peered":"2024-03-06T07:18:13.565250+0000","last_unstale":"2024-03-06T07:18:14.551608+0000","last_undegraded":"2024-03-06T07:18:14.551608+0000","last_fullsized":"2024-03-06T07:18:14.551608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:15.265063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718863+0000","last_change":"2024-03-06T07:18:08.523848+0000","last_active":"2024-03-06T07:18:14.718863+0000","last_peered":"2024-03-06T07:18:14.718863+0000","last_clean":"2024-03-06T07:18:14.718863+0000","last_became_active":"2024-03-06T07:18:08.523455+0000","last_became_peered":"2024-03-06T07:18:08.523455+0000","last_unstale":"2024-03-06T07:18:14.718863+0000","last_undegraded":"2024-03-06T07:18:14.718863+0000","last_fullsized":"2024-03-06T07:18:14.718863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718515+0000","last_change":"2024-03-06T07:18:09.531072+0000","last_active":"2024-03-06T07:18:14.718515+0000","last_peered":"2024-03-06T07:18:14.718515+0000","last_clean":"2024-03-06T07:18:14.718515+0000","last_became_active":"2024-03-06T07:18:09.530815+0000","last_became_peered":"2024-03-06T07:18:09.530815+0000","last_unstale":"2024-03-06T07:18:14.718515+0000","last_undegraded":"2024-03-06T07:18:14.718515+0000","last_fullsized":"2024-03-06T07:18:14.718515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551688+0000","last_change":"2024-03-06T07:18:13.563305+0000","last_active":"2024-03-06T07:18:14.551688+0000","last_peered":"2024-03-06T07:18:14.551688+0000","last_clean":"2024-03-06T07:18:14.551688+0000","last_became_active":"2024-03-06T07:18:13.562936+0000","last_became_peered":"2024-03-06T07:18:13.562936+0000","last_unstale":"2024-03-06T07:18:14.551688+0000","last_undegraded":"2024-03-06T07:18:14.551688+0000","last_fullsized":"2024-03-06T07:18:14.551688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:02.828593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718486+0000","last_change":"2024-03-06T07:18:08.520455+0000","last_active":"2024-03-06T07:18:14.718486+0000","last_peered":"2024-03-06T07:18:14.718486+0000","last_clean":"2024-03-06T07:18:14.718486+0000","last_became_active":"2024-03-06T07:18:08.520170+0000","last_became_peered":"2024-03-06T07:18:08.520170+0000","last_unstale":"2024-03-06T07:18:14.718486+0000","last_undegraded":"2024-03-06T07:18:14.718486+0000","last_fullsized":"2024-03-06T07:18:14.718486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719134+0000","last_change":"2024-03-06T07:18:09.536183+0000","last_active":"2024-03-06T07:18:14.719134+0000","last_peered":"2024-03-06T07:18:14.719134+0000","last_clean":"2024-03-06T07:18:14.719134+0000","last_became_active":"2024-03-06T07:18:09.535516+0000","last_became_peered":"2024-03-06T07:18:09.535516+0000","last_unstale":"2024-03-06T07:18:14.719134+0000","last_undegraded":"2024-03-06T07:18:14.719134+0000","last_fullsized":"2024-03-06T07:18:14.719134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551978+0000","last_change":"2024-03-06T07:18:13.564931+0000","last_active":"2024-03-06T07:18:14.551978+0000","last_peered":"2024-03-06T07:18:14.551978+0000","last_clean":"2024-03-06T07:18:14.551978+0000","last_became_active":"2024-03-06T07:18:13.564588+0000","last_became_peered":"2024-03-06T07:18:13.564588+0000","last_unstale":"2024-03-06T07:18:14.551978+0000","last_undegraded":"2024-03-06T07:18:14.551978+0000","last_fullsized":"2024-03-06T07:18:14.551978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:41.700550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550818+0000","last_change":"2024-03-06T07:18:09.516785+0000","last_active":"2024-03-06T07:18:14.550818+0000","last_peered":"2024-03-06T07:18:14.550818+0000","last_clean":"2024-03-06T07:18:14.550818+0000","last_became_active":"2024-03-06T07:18:09.516494+0000","last_became_peered":"2024-03-06T07:18:09.516494+0000","last_unstale":"2024-03-06T07:18:14.550818+0000","last_undegraded":"2024-03-06T07:18:14.550818+0000","last_fullsized":"2024-03-06T07:18:14.550818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719130+0000","last_change":"2024-03-06T07:18:08.523549+0000","last_active":"2024-03-06T07:18:14.719130+0000","last_peered":"2024-03-06T07:18:14.719130+0000","last_clean":"2024-03-06T07:18:14.719130+0000","last_became_active":"2024-03-06T07:18:08.523264+0000","last_became_peered":"2024-03-06T07:18:08.523264+0000","last_unstale":"2024-03-06T07:18:14.719130+0000","last_undegraded":"2024-03-06T07:18:14.719130+0000","last_fullsized":"2024-03-06T07:18:14.719130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:02.351247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552173+0000","last_change":"2024-03-06T07:18:13.563114+0000","last_active":"2024-03-06T07:18:14.552173+0000","last_peered":"2024-03-06T07:18:14.552173+0000","last_clean":"2024-03-06T07:18:14.552173+0000","last_became_active":"2024-03-06T07:18:13.562736+0000","last_became_peered":"2024-03-06T07:18:13.562736+0000","last_unstale":"2024-03-06T07:18:14.552173+0000","last_undegraded":"2024-03-06T07:18:14.552173+0000","last_fullsized":"2024-03-06T07:18:14.552173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:20.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720134+0000","last_change":"2024-03-06T07:18:09.524088+0000","last_active":"2024-03-06T07:18:14.720134+0000","last_peered":"2024-03-06T07:18:14.720134+0000","last_clean":"2024-03-06T07:18:14.720134+0000","last_became_active":"2024-03-06T07:18:09.523821+0000","last_became_peered":"2024-03-06T07:18:09.523821+0000","last_unstale":"2024-03-06T07:18:14.720134+0000","last_undegraded":"2024-03-06T07:18:14.720134+0000","last_fullsized":"2024-03-06T07:18:14.720134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719053+0000","last_change":"2024-03-06T07:18:08.516137+0000","last_active":"2024-03-06T07:18:14.719053+0000","last_peered":"2024-03-06T07:18:14.719053+0000","last_clean":"2024-03-06T07:18:14.719053+0000","last_became_active":"2024-03-06T07:18:08.515841+0000","last_became_peered":"2024-03-06T07:18:08.515841+0000","last_unstale":"2024-03-06T07:18:14.719053+0000","last_undegraded":"2024-03-06T07:18:14.719053+0000","last_fullsized":"2024-03-06T07:18:14.719053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551452+0000","last_change":"2024-03-06T07:18:13.564851+0000","last_active":"2024-03-06T07:18:14.551452+0000","last_peered":"2024-03-06T07:18:14.551452+0000","last_clean":"2024-03-06T07:18:14.551452+0000","last_became_active":"2024-03-06T07:18:13.564494+0000","last_became_peered":"2024-03-06T07:18:13.564494+0000","last_unstale":"2024-03-06T07:18:14.551452+0000","last_undegraded":"2024-03-06T07:18:14.551452+0000","last_fullsized":"2024-03-06T07:18:14.551452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:27.757638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720452+0000","last_change":"2024-03-06T07:18:09.527318+0000","last_active":"2024-03-06T07:18:14.720452+0000","last_peered":"2024-03-06T07:18:14.720452+0000","last_clean":"2024-03-06T07:18:14.720452+0000","last_became_active":"2024-03-06T07:18:09.527061+0000","last_became_peered":"2024-03-06T07:18:09.527061+0000","last_unstale":"2024-03-06T07:18:14.720452+0000","last_undegraded":"2024-03-06T07:18:14.720452+0000","last_fullsized":"2024-03-06T07:18:14.720452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719621+0000","last_change":"2024-03-06T07:18:08.515303+0000","last_active":"2024-03-06T07:18:14.719621+0000","last_peered":"2024-03-06T07:18:14.719621+0000","last_clean":"2024-03-06T07:18:14.719621+0000","last_became_active":"2024-03-06T07:18:08.514178+0000","last_became_peered":"2024-03-06T07:18:08.514178+0000","last_unstale":"2024-03-06T07:18:14.719621+0000","last_undegraded":"2024-03-06T07:18:14.719621+0000","last_fullsized":"2024-03-06T07:18:14.719621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719788+0000","last_change":"2024-03-06T07:18:13.788851+0000","last_active":"2024-03-06T07:18:14.719788+0000","last_peered":"2024-03-06T07:18:14.719788+0000","last_clean":"2024-03-06T07:18:14.719788+0000","last_became_active":"2024-03-06T07:18:13.788492+0000","last_became_peered":"2024-03-06T07:18:13.788492+0000","last_unstale":"2024-03-06T07:18:14.719788+0000","last_undegraded":"2024-03-06T07:18:14.719788+0000","last_fullsized":"2024-03-06T07:18:14.719788+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552468+0000","last_change":"2024-03-06T07:18:09.517957+0000","last_active":"2024-03-06T07:18:14.552468+0000","last_peered":"2024-03-06T07:18:14.552468+0000","last_clean":"2024-03-06T07:18:14.552468+0000","last_became_active":"2024-03-06T07:18:09.517197+0000","last_became_peered":"2024-03-06T07:18:09.517197+0000","last_unstale":"2024-03-06T07:18:14.552468+0000","last_undegraded":"2024-03-06T07:18:14.552468+0000","last_fullsized":"2024-03-06T07:18:14.552468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:31.051748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718952+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.718952+0000","last_peered":"2024-03-06T07:18:14.718952+0000","last_clean":"2024-03-06T07:18:14.718952+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.718952+0000","last_undegraded":"2024-03-06T07:18:14.718952+0000","last_fullsized":"2024-03-06T07:18:14.718952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552149+0000","last_change":"2024-03-06T07:18:13.565073+0000","last_active":"2024-03-06T07:18:14.552149+0000","last_peered":"2024-03-06T07:18:14.552149+0000","last_clean":"2024-03-06T07:18:14.552149+0000","last_became_active":"2024-03-06T07:18:13.564708+0000","last_became_peered":"2024-03-06T07:18:13.564708+0000","last_unstale":"2024-03-06T07:18:14.552149+0000","last_undegraded":"2024-03-06T07:18:14.552149+0000","last_fullsized":"2024-03-06T07:18:14.552149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:20.796366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719443+0000","last_change":"2024-03-06T07:18:09.520925+0000","last_active":"2024-03-06T07:18:14.719443+0000","last_peered":"2024-03-06T07:18:14.719443+0000","last_clean":"2024-03-06T07:18:14.719443+0000","last_became_active":"2024-03-06T07:18:09.520658+0000","last_became_peered":"2024-03-06T07:18:09.520658+0000","last_unstale":"2024-03-06T07:18:14.719443+0000","last_undegraded":"2024-03-06T07:18:14.719443+0000","last_fullsized":"2024-03-06T07:18:14.719443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551744+0000","last_change":"2024-03-06T07:18:08.519229+0000","last_active":"2024-03-06T07:18:14.551744+0000","last_peered":"2024-03-06T07:18:14.551744+0000","last_clean":"2024-03-06T07:18:14.551744+0000","last_became_active":"2024-03-06T07:18:08.518936+0000","last_became_peered":"2024-03-06T07:18:08.518936+0000","last_unstale":"2024-03-06T07:18:14.551744+0000","last_undegraded":"2024-03-06T07:18:14.551744+0000","last_fullsized":"2024-03-06T07:18:14.551744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552379+0000","last_change":"2024-03-06T07:18:13.787076+0000","last_active":"2024-03-06T07:18:14.552379+0000","last_peered":"2024-03-06T07:18:14.552379+0000","last_clean":"2024-03-06T07:18:14.552379+0000","last_became_active":"2024-03-06T07:18:13.786733+0000","last_became_peered":"2024-03-06T07:18:13.786733+0000","last_unstale":"2024-03-06T07:18:14.552379+0000","last_undegraded":"2024-03-06T07:18:14.552379+0000","last_fullsized":"2024-03-06T07:18:14.552379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720898+0000","last_change":"2024-03-06T07:18:09.534800+0000","last_active":"2024-03-06T07:18:14.720898+0000","last_peered":"2024-03-06T07:18:14.720898+0000","last_clean":"2024-03-06T07:18:14.720898+0000","last_became_active":"2024-03-06T07:18:09.534408+0000","last_became_peered":"2024-03-06T07:18:09.534408+0000","last_unstale":"2024-03-06T07:18:14.720898+0000","last_undegraded":"2024-03-06T07:18:14.720898+0000","last_fullsized":"2024-03-06T07:18:14.720898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719429+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.719429+0000","last_peered":"2024-03-06T07:18:14.719429+0000","last_clean":"2024-03-06T07:18:14.719429+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.719429+0000","last_undegraded":"2024-03-06T07:18:14.719429+0000","last_fullsized":"2024-03-06T07:18:14.719429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720202+0000","last_change":"2024-03-06T07:18:13.785863+0000","last_active":"2024-03-06T07:18:14.720202+0000","last_peered":"2024-03-06T07:18:14.720202+0000","last_clean":"2024-03-06T07:18:14.720202+0000","last_became_active":"2024-03-06T07:18:13.785533+0000","last_became_peered":"2024-03-06T07:18:13.785533+0000","last_unstale":"2024-03-06T07:18:14.720202+0000","last_undegraded":"2024-03-06T07:18:14.720202+0000","last_fullsized":"2024-03-06T07:18:14.720202+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:43.382716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718962+0000","last_change":"2024-03-06T07:18:09.523934+0000","last_active":"2024-03-06T07:18:14.718962+0000","last_peered":"2024-03-06T07:18:14.718962+0000","last_clean":"2024-03-06T07:18:14.718962+0000","last_became_active":"2024-03-06T07:18:09.523663+0000","last_became_peered":"2024-03-06T07:18:09.523663+0000","last_unstale":"2024-03-06T07:18:14.718962+0000","last_undegraded":"2024-03-06T07:18:14.718962+0000","last_fullsized":"2024-03-06T07:18:14.718962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718957+0000","last_change":"2024-03-06T07:18:08.520839+0000","last_active":"2024-03-06T07:18:14.718957+0000","last_peered":"2024-03-06T07:18:14.718957+0000","last_clean":"2024-03-06T07:18:14.718957+0000","last_became_active":"2024-03-06T07:18:08.520578+0000","last_became_peered":"2024-03-06T07:18:08.520578+0000","last_unstale":"2024-03-06T07:18:14.718957+0000","last_undegraded":"2024-03-06T07:18:14.718957+0000","last_fullsized":"2024-03-06T07:18:14.718957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:21.462756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718986+0000","last_change":"2024-03-06T07:18:13.787099+0000","last_active":"2024-03-06T07:18:14.718986+0000","last_peered":"2024-03-06T07:18:14.718986+0000","last_clean":"2024-03-06T07:18:14.718986+0000","last_became_active":"2024-03-06T07:18:13.786694+0000","last_became_peered":"2024-03-06T07:18:13.786694+0000","last_unstale":"2024-03-06T07:18:14.718986+0000","last_undegraded":"2024-03-06T07:18:14.718986+0000","last_fullsized":"2024-03-06T07:18:14.718986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719219+0000","last_change":"2024-03-06T07:18:09.531609+0000","last_active":"2024-03-06T07:18:14.719219+0000","last_peered":"2024-03-06T07:18:14.719219+0000","last_clean":"2024-03-06T07:18:14.719219+0000","last_became_active":"2024-03-06T07:18:09.531318+0000","last_became_peered":"2024-03-06T07:18:09.531318+0000","last_unstale":"2024-03-06T07:18:14.719219+0000","last_undegraded":"2024-03-06T07:18:14.719219+0000","last_fullsized":"2024-03-06T07:18:14.719219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719042+0000","last_change":"2024-03-06T07:18:08.516897+0000","last_active":"2024-03-06T07:18:14.719042+0000","last_peered":"2024-03-06T07:18:14.719042+0000","last_clean":"2024-03-06T07:18:14.719042+0000","last_became_active":"2024-03-06T07:18:08.516647+0000","last_became_peered":"2024-03-06T07:18:08.516647+0000","last_unstale":"2024-03-06T07:18:14.719042+0000","last_undegraded":"2024-03-06T07:18:14.719042+0000","last_fullsized":"2024-03-06T07:18:14.719042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718719+0000","last_change":"2024-03-06T07:18:13.564264+0000","last_active":"2024-03-06T07:18:14.718719+0000","last_peered":"2024-03-06T07:18:14.718719+0000","last_clean":"2024-03-06T07:18:14.718719+0000","last_became_active":"2024-03-06T07:18:13.563923+0000","last_became_peered":"2024-03-06T07:18:13.563923+0000","last_unstale":"2024-03-06T07:18:14.718719+0000","last_undegraded":"2024-03-06T07:18:14.718719+0000","last_fullsized":"2024-03-06T07:18:14.718719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552093+0000","last_change":"2024-03-06T07:18:09.519337+0000","last_active":"2024-03-06T07:18:14.552093+0000","last_peered":"2024-03-06T07:18:14.552093+0000","last_clean":"2024-03-06T07:18:14.552093+0000","last_became_active":"2024-03-06T07:18:09.519042+0000","last_became_peered":"2024-03-06T07:18:09.519042+0000","last_unstale":"2024-03-06T07:18:14.552093+0000","last_undegraded":"2024-03-06T07:18:14.552093+0000","last_fullsized":"2024-03-06T07:18:14.552093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:48.003076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718514+0000","last_change":"2024-03-06T07:18:08.521877+0000","last_active":"2024-03-06T07:18:14.718514+0000","last_peered":"2024-03-06T07:18:14.718514+0000","last_clean":"2024-03-06T07:18:14.718514+0000","last_became_active":"2024-03-06T07:18:08.521583+0000","last_became_peered":"2024-03-06T07:18:08.521583+0000","last_unstale":"2024-03-06T07:18:14.718514+0000","last_undegraded":"2024-03-06T07:18:14.718514+0000","last_fullsized":"2024-03-06T07:18:14.718514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718907+0000","last_change":"2024-03-06T07:18:13.566418+0000","last_active":"2024-03-06T07:18:14.718907+0000","last_peered":"2024-03-06T07:18:14.718907+0000","last_clean":"2024-03-06T07:18:14.718907+0000","last_became_active":"2024-03-06T07:18:13.566013+0000","last_became_peered":"2024-03-06T07:18:13.566013+0000","last_unstale":"2024-03-06T07:18:14.718907+0000","last_undegraded":"2024-03-06T07:18:14.718907+0000","last_fullsized":"2024-03-06T07:18:14.718907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719777+0000","last_change":"2024-03-06T07:18:09.522608+0000","last_active":"2024-03-06T07:18:14.719777+0000","last_peered":"2024-03-06T07:18:14.719777+0000","last_clean":"2024-03-06T07:18:14.719777+0000","last_became_active":"2024-03-06T07:18:09.522360+0000","last_became_peered":"2024-03-06T07:18:09.522360+0000","last_unstale":"2024-03-06T07:18:14.719777+0000","last_undegraded":"2024-03-06T07:18:14.719777+0000","last_fullsized":"2024-03-06T07:18:14.719777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548304+0000","last_change":"2024-03-06T07:18:08.512125+0000","last_active":"2024-03-06T07:18:14.548304+0000","last_peered":"2024-03-06T07:18:14.548304+0000","last_clean":"2024-03-06T07:18:14.548304+0000","last_became_active":"2024-03-06T07:18:08.511827+0000","last_became_peered":"2024-03-06T07:18:08.511827+0000","last_unstale":"2024-03-06T07:18:14.548304+0000","last_undegraded":"2024-03-06T07:18:14.548304+0000","last_fullsized":"2024-03-06T07:18:14.548304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719927+0000","last_change":"2024-03-06T07:18:13.573031+0000","last_active":"2024-03-06T07:18:14.719927+0000","last_peered":"2024-03-06T07:18:14.719927+0000","last_clean":"2024-03-06T07:18:14.719927+0000","last_became_active":"2024-03-06T07:18:13.572562+0000","last_became_peered":"2024-03-06T07:18:13.572562+0000","last_unstale":"2024-03-06T07:18:14.719927+0000","last_undegraded":"2024-03-06T07:18:14.719927+0000","last_fullsized":"2024-03-06T07:18:14.719927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:55.484210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719484+0000","last_change":"2024-03-06T07:18:09.518341+0000","last_active":"2024-03-06T07:18:14.719484+0000","last_peered":"2024-03-06T07:18:14.719484+0000","last_clean":"2024-03-06T07:18:14.719484+0000","last_became_active":"2024-03-06T07:18:09.518030+0000","last_became_peered":"2024-03-06T07:18:09.518030+0000","last_unstale":"2024-03-06T07:18:14.719484+0000","last_undegraded":"2024-03-06T07:18:14.719484+0000","last_fullsized":"2024-03-06T07:18:14.719484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:13.874406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718993+0000","last_change":"2024-03-06T07:18:08.522022+0000","last_active":"2024-03-06T07:18:14.718993+0000","last_peered":"2024-03-06T07:18:14.718993+0000","last_clean":"2024-03-06T07:18:14.718993+0000","last_became_active":"2024-03-06T07:18:08.521764+0000","last_became_peered":"2024-03-06T07:18:08.521764+0000","last_unstale":"2024-03-06T07:18:14.718993+0000","last_undegraded":"2024-03-06T07:18:14.718993+0000","last_fullsized":"2024-03-06T07:18:14.718993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719331+0000","last_change":"2024-03-06T07:18:13.785137+0000","last_active":"2024-03-06T07:18:14.719331+0000","last_peered":"2024-03-06T07:18:14.719331+0000","last_clean":"2024-03-06T07:18:14.719331+0000","last_became_active":"2024-03-06T07:18:13.784706+0000","last_became_peered":"2024-03-06T07:18:13.784706+0000","last_unstale":"2024-03-06T07:18:14.719331+0000","last_undegraded":"2024-03-06T07:18:14.719331+0000","last_fullsized":"2024-03-06T07:18:14.719331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719805+0000","last_change":"2024-03-06T07:18:09.532012+0000","last_active":"2024-03-06T07:18:14.719805+0000","last_peered":"2024-03-06T07:18:14.719805+0000","last_clean":"2024-03-06T07:18:14.719805+0000","last_became_active":"2024-03-06T07:18:09.531724+0000","last_became_peered":"2024-03-06T07:18:09.531724+0000","last_unstale":"2024-03-06T07:18:14.719805+0000","last_undegraded":"2024-03-06T07:18:14.719805+0000","last_fullsized":"2024-03-06T07:18:14.719805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718407+0000","last_change":"2024-03-06T07:18:08.515757+0000","last_active":"2024-03-06T07:18:14.718407+0000","last_peered":"2024-03-06T07:18:14.718407+0000","last_clean":"2024-03-06T07:18:14.718407+0000","last_became_active":"2024-03-06T07:18:08.515494+0000","last_became_peered":"2024-03-06T07:18:08.515494+0000","last_unstale":"2024-03-06T07:18:14.718407+0000","last_undegraded":"2024-03-06T07:18:14.718407+0000","last_fullsized":"2024-03-06T07:18:14.718407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719078+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:18:14.719078+0000","last_peered":"2024-03-06T07:18:14.719078+0000","last_clean":"2024-03-06T07:18:14.719078+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:18:14.719078+0000","last_undegraded":"2024-03-06T07:18:14.719078+0000","last_fullsized":"2024-03-06T07:18:14.719078+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551436+0000","last_change":"2024-03-06T07:18:13.786445+0000","last_active":"2024-03-06T07:18:14.551436+0000","last_peered":"2024-03-06T07:18:14.551436+0000","last_clean":"2024-03-06T07:18:14.551436+0000","last_became_active":"2024-03-06T07:18:13.786085+0000","last_became_peered":"2024-03-06T07:18:13.786085+0000","last_unstale":"2024-03-06T07:18:14.551436+0000","last_undegraded":"2024-03-06T07:18:14.551436+0000","last_fullsized":"2024-03-06T07:18:14.551436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718724+0000","last_change":"2024-03-06T07:18:09.532185+0000","last_active":"2024-03-06T07:18:14.718724+0000","last_peered":"2024-03-06T07:18:14.718724+0000","last_clean":"2024-03-06T07:18:14.718724+0000","last_became_active":"2024-03-06T07:18:09.531883+0000","last_became_peered":"2024-03-06T07:18:09.531883+0000","last_unstale":"2024-03-06T07:18:14.718724+0000","last_undegraded":"2024-03-06T07:18:14.718724+0000","last_fullsized":"2024-03-06T07:18:14.718724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548537+0000","last_change":"2024-03-06T07:18:08.518305+0000","last_active":"2024-03-06T07:18:14.548537+0000","last_peered":"2024-03-06T07:18:14.548537+0000","last_clean":"2024-03-06T07:18:14.548537+0000","last_became_active":"2024-03-06T07:18:08.517977+0000","last_became_peered":"2024-03-06T07:18:08.517977+0000","last_unstale":"2024-03-06T07:18:14.548537+0000","last_undegraded":"2024-03-06T07:18:14.548537+0000","last_fullsized":"2024-03-06T07:18:14.548537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550763+0000","last_change":"2024-03-06T07:18:13.785349+0000","last_active":"2024-03-06T07:18:14.550763+0000","last_peered":"2024-03-06T07:18:14.550763+0000","last_clean":"2024-03-06T07:18:14.550763+0000","last_became_active":"2024-03-06T07:18:13.784931+0000","last_became_peered":"2024-03-06T07:18:13.784931+0000","last_unstale":"2024-03-06T07:18:14.550763+0000","last_undegraded":"2024-03-06T07:18:14.550763+0000","last_fullsized":"2024-03-06T07:18:14.550763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718607+0000","last_change":"2024-03-06T07:18:09.532114+0000","last_active":"2024-03-06T07:18:14.718607+0000","last_peered":"2024-03-06T07:18:14.718607+0000","last_clean":"2024-03-06T07:18:14.718607+0000","last_became_active":"2024-03-06T07:18:09.531373+0000","last_became_peered":"2024-03-06T07:18:09.531373+0000","last_unstale":"2024-03-06T07:18:14.718607+0000","last_undegraded":"2024-03-06T07:18:14.718607+0000","last_fullsized":"2024-03-06T07:18:14.718607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718980+0000","last_change":"2024-03-06T07:18:08.520471+0000","last_active":"2024-03-06T07:18:14.718980+0000","last_peered":"2024-03-06T07:18:14.718980+0000","last_clean":"2024-03-06T07:18:14.718980+0000","last_became_active":"2024-03-06T07:18:08.520156+0000","last_became_peered":"2024-03-06T07:18:08.520156+0000","last_unstale":"2024-03-06T07:18:14.718980+0000","last_undegraded":"2024-03-06T07:18:14.718980+0000","last_fullsized":"2024-03-06T07:18:14.718980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719199+0000","last_change":"2024-03-06T07:18:13.569753+0000","last_active":"2024-03-06T07:18:14.719199+0000","last_peered":"2024-03-06T07:18:14.719199+0000","last_clean":"2024-03-06T07:18:14.719199+0000","last_became_active":"2024-03-06T07:18:13.569369+0000","last_became_peered":"2024-03-06T07:18:13.569369+0000","last_unstale":"2024-03-06T07:18:14.719199+0000","last_undegraded":"2024-03-06T07:18:14.719199+0000","last_fullsized":"2024-03-06T07:18:14.719199+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719153+0000","last_change":"2024-03-06T07:18:09.536431+0000","last_active":"2024-03-06T07:18:14.719153+0000","last_peered":"2024-03-06T07:18:14.719153+0000","last_clean":"2024-03-06T07:18:14.719153+0000","last_became_active":"2024-03-06T07:18:09.536167+0000","last_became_peered":"2024-03-06T07:18:09.536167+0000","last_unstale":"2024-03-06T07:18:14.719153+0000","last_undegraded":"2024-03-06T07:18:14.719153+0000","last_fullsized":"2024-03-06T07:18:14.719153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.441770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548593+0000","last_change":"2024-03-06T07:18:08.511825+0000","last_active":"2024-03-06T07:18:14.548593+0000","last_peered":"2024-03-06T07:18:14.548593+0000","last_clean":"2024-03-06T07:18:14.548593+0000","last_became_active":"2024-03-06T07:18:08.511363+0000","last_became_peered":"2024-03-06T07:18:08.511363+0000","last_unstale":"2024-03-06T07:18:14.548593+0000","last_undegraded":"2024-03-06T07:18:14.548593+0000","last_fullsized":"2024-03-06T07:18:14.548593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718008+0000","last_change":"2024-03-06T07:18:13.786468+0000","last_active":"2024-03-06T07:18:14.718008+0000","last_peered":"2024-03-06T07:18:14.718008+0000","last_clean":"2024-03-06T07:18:14.718008+0000","last_became_active":"2024-03-06T07:18:13.786129+0000","last_became_peered":"2024-03-06T07:18:13.786129+0000","last_unstale":"2024-03-06T07:18:14.718008+0000","last_undegraded":"2024-03-06T07:18:14.718008+0000","last_fullsized":"2024-03-06T07:18:14.718008+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:42.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718705+0000","last_change":"2024-03-06T07:18:09.534236+0000","last_active":"2024-03-06T07:18:14.718705+0000","last_peered":"2024-03-06T07:18:14.718705+0000","last_clean":"2024-03-06T07:18:14.718705+0000","last_became_active":"2024-03-06T07:18:09.533090+0000","last_became_peered":"2024-03-06T07:18:09.533090+0000","last_unstale":"2024-03-06T07:18:14.718705+0000","last_undegraded":"2024-03-06T07:18:14.718705+0000","last_fullsized":"2024-03-06T07:18:14.718705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552342+0000","last_change":"2024-03-06T07:18:08.519410+0000","last_active":"2024-03-06T07:18:14.552342+0000","last_peered":"2024-03-06T07:18:14.552342+0000","last_clean":"2024-03-06T07:18:14.552342+0000","last_became_active":"2024-03-06T07:18:08.519115+0000","last_became_peered":"2024-03-06T07:18:08.519115+0000","last_unstale":"2024-03-06T07:18:14.552342+0000","last_undegraded":"2024-03-06T07:18:14.552342+0000","last_fullsized":"2024-03-06T07:18:14.552342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:22.247661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552289+0000","last_change":"2024-03-06T07:18:13.563952+0000","last_active":"2024-03-06T07:18:14.552289+0000","last_peered":"2024-03-06T07:18:14.552289+0000","last_clean":"2024-03-06T07:18:14.552289+0000","last_became_active":"2024-03-06T07:18:13.563578+0000","last_became_peered":"2024-03-06T07:18:13.563578+0000","last_unstale":"2024-03-06T07:18:14.552289+0000","last_undegraded":"2024-03-06T07:18:14.552289+0000","last_fullsized":"2024-03-06T07:18:14.552289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718657+0000","last_change":"2024-03-06T07:18:09.526419+0000","last_active":"2024-03-06T07:18:14.718657+0000","last_peered":"2024-03-06T07:18:14.718657+0000","last_clean":"2024-03-06T07:18:14.718657+0000","last_became_active":"2024-03-06T07:18:09.526165+0000","last_became_peered":"2024-03-06T07:18:09.526165+0000","last_unstale":"2024-03-06T07:18:14.718657+0000","last_undegraded":"2024-03-06T07:18:14.718657+0000","last_fullsized":"2024-03-06T07:18:14.718657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548110+0000","last_change":"2024-03-06T07:18:08.508444+0000","last_active":"2024-03-06T07:18:14.548110+0000","last_peered":"2024-03-06T07:18:14.548110+0000","last_clean":"2024-03-06T07:18:14.548110+0000","last_became_active":"2024-03-06T07:18:08.508153+0000","last_became_peered":"2024-03-06T07:18:08.508153+0000","last_unstale":"2024-03-06T07:18:14.548110+0000","last_undegraded":"2024-03-06T07:18:14.548110+0000","last_fullsized":"2024-03-06T07:18:14.548110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719075+0000","last_change":"2024-03-06T07:18:13.567057+0000","last_active":"2024-03-06T07:18:14.719075+0000","last_peered":"2024-03-06T07:18:14.719075+0000","last_clean":"2024-03-06T07:18:14.719075+0000","last_became_active":"2024-03-06T07:18:13.566702+0000","last_became_peered":"2024-03-06T07:18:13.566702+0000","last_unstale":"2024-03-06T07:18:14.719075+0000","last_undegraded":"2024-03-06T07:18:14.719075+0000","last_fullsized":"2024-03-06T07:18:14.719075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550710+0000","last_change":"2024-03-06T07:18:09.520527+0000","last_active":"2024-03-06T07:18:14.550710+0000","last_peered":"2024-03-06T07:18:14.550710+0000","last_clean":"2024-03-06T07:18:14.550710+0000","last_became_active":"2024-03-06T07:18:09.520233+0000","last_became_peered":"2024-03-06T07:18:09.520233+0000","last_unstale":"2024-03-06T07:18:14.550710+0000","last_undegraded":"2024-03-06T07:18:14.550710+0000","last_fullsized":"2024-03-06T07:18:14.550710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550736+0000","last_change":"2024-03-06T07:18:08.521908+0000","last_active":"2024-03-06T07:18:14.550736+0000","last_peered":"2024-03-06T07:18:14.550736+0000","last_clean":"2024-03-06T07:18:14.550736+0000","last_became_active":"2024-03-06T07:18:08.521324+0000","last_became_peered":"2024-03-06T07:18:08.521324+0000","last_unstale":"2024-03-06T07:18:14.550736+0000","last_undegraded":"2024-03-06T07:18:14.550736+0000","last_fullsized":"2024-03-06T07:18:14.550736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720936+0000","last_change":"2024-03-06T07:18:13.788800+0000","last_active":"2024-03-06T07:18:14.720936+0000","last_peered":"2024-03-06T07:18:14.720936+0000","last_clean":"2024-03-06T07:18:14.720936+0000","last_became_active":"2024-03-06T07:18:13.788435+0000","last_became_peered":"2024-03-06T07:18:13.788435+0000","last_unstale":"2024-03-06T07:18:14.720936+0000","last_undegraded":"2024-03-06T07:18:14.720936+0000","last_fullsized":"2024-03-06T07:18:14.720936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:36.842720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719105+0000","last_change":"2024-03-06T07:18:09.527698+0000","last_active":"2024-03-06T07:18:14.719105+0000","last_peered":"2024-03-06T07:18:14.719105+0000","last_clean":"2024-03-06T07:18:14.719105+0000","last_became_active":"2024-03-06T07:18:09.527407+0000","last_became_peered":"2024-03-06T07:18:09.527407+0000","last_unstale":"2024-03-06T07:18:14.719105+0000","last_undegraded":"2024-03-06T07:18:14.719105+0000","last_fullsized":"2024-03-06T07:18:14.719105+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718447+0000","last_change":"2024-03-06T07:18:08.518380+0000","last_active":"2024-03-06T07:18:14.718447+0000","last_peered":"2024-03-06T07:18:14.718447+0000","last_clean":"2024-03-06T07:18:14.718447+0000","last_became_active":"2024-03-06T07:18:08.517571+0000","last_became_peered":"2024-03-06T07:18:08.517571+0000","last_unstale":"2024-03-06T07:18:14.718447+0000","last_undegraded":"2024-03-06T07:18:14.718447+0000","last_fullsized":"2024-03-06T07:18:14.718447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720838+0000","last_change":"2024-03-06T07:18:13.785111+0000","last_active":"2024-03-06T07:18:14.720838+0000","last_peered":"2024-03-06T07:18:14.720838+0000","last_clean":"2024-03-06T07:18:14.720838+0000","last_became_active":"2024-03-06T07:18:13.784751+0000","last_became_peered":"2024-03-06T07:18:13.784751+0000","last_unstale":"2024-03-06T07:18:14.720838+0000","last_undegraded":"2024-03-06T07:18:14.720838+0000","last_fullsized":"2024-03-06T07:18:14.720838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.733802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719140+0000","last_change":"2024-03-06T07:18:09.526243+0000","last_active":"2024-03-06T07:18:14.719140+0000","last_peered":"2024-03-06T07:18:14.719140+0000","last_clean":"2024-03-06T07:18:14.719140+0000","last_became_active":"2024-03-06T07:18:09.525985+0000","last_became_peered":"2024-03-06T07:18:09.525985+0000","last_unstale":"2024-03-06T07:18:14.719140+0000","last_undegraded":"2024-03-06T07:18:14.719140+0000","last_fullsized":"2024-03-06T07:18:14.719140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:20.656356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719309+0000","last_change":"2024-03-06T07:18:08.520844+0000","last_active":"2024-03-06T07:18:14.719309+0000","last_peered":"2024-03-06T07:18:14.719309+0000","last_clean":"2024-03-06T07:18:14.719309+0000","last_became_active":"2024-03-06T07:18:08.520318+0000","last_became_peered":"2024-03-06T07:18:08.520318+0000","last_unstale":"2024-03-06T07:18:14.719309+0000","last_undegraded":"2024-03-06T07:18:14.719309+0000","last_fullsized":"2024-03-06T07:18:14.719309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:28.274132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719091+0000","last_change":"2024-03-06T07:18:13.785334+0000","last_active":"2024-03-06T07:18:14.719091+0000","last_peered":"2024-03-06T07:18:14.719091+0000","last_clean":"2024-03-06T07:18:14.719091+0000","last_became_active":"2024-03-06T07:18:13.784953+0000","last_became_peered":"2024-03-06T07:18:13.784953+0000","last_unstale":"2024-03-06T07:18:14.719091+0000","last_undegraded":"2024-03-06T07:18:14.719091+0000","last_fullsized":"2024-03-06T07:18:14.719091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:18.038907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718773+0000","last_change":"2024-03-06T07:18:09.533771+0000","last_active":"2024-03-06T07:18:14.718773+0000","last_peered":"2024-03-06T07:18:14.718773+0000","last_clean":"2024-03-06T07:18:14.718773+0000","last_became_active":"2024-03-06T07:18:09.533520+0000","last_became_peered":"2024-03-06T07:18:09.533520+0000","last_unstale":"2024-03-06T07:18:14.718773+0000","last_undegraded":"2024-03-06T07:18:14.718773+0000","last_fullsized":"2024-03-06T07:18:14.718773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:18.912751+0000","last_change":"2024-03-06T07:18:08.511813+0000","last_active":"2024-03-06T07:18:18.912751+0000","last_peered":"2024-03-06T07:18:18.912751+0000","last_clean":"2024-03-06T07:18:18.912751+0000","last_became_active":"2024-03-06T07:18:08.511552+0000","last_became_peered":"2024-03-06T07:18:08.511552+0000","last_unstale":"2024-03-06T07:18:18.912751+0000","last_undegraded":"2024-03-06T07:18:18.912751+0000","last_fullsized":"2024-03-06T07:18:18.912751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718960+0000","last_change":"2024-03-06T07:18:13.567483+0000","last_active":"2024-03-06T07:18:14.718960+0000","last_peered":"2024-03-06T07:18:14.718960+0000","last_clean":"2024-03-06T07:18:14.718960+0000","last_became_active":"2024-03-06T07:18:13.566829+0000","last_became_peered":"2024-03-06T07:18:13.566829+0000","last_unstale":"2024-03-06T07:18:14.718960+0000","last_undegraded":"2024-03-06T07:18:14.718960+0000","last_fullsized":"2024-03-06T07:18:14.718960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551865+0000","last_change":"2024-03-06T07:18:09.525404+0000","last_active":"2024-03-06T07:18:14.551865+0000","last_peered":"2024-03-06T07:18:14.551865+0000","last_clean":"2024-03-06T07:18:14.551865+0000","last_became_active":"2024-03-06T07:18:09.525127+0000","last_became_peered":"2024-03-06T07:18:09.525127+0000","last_unstale":"2024-03-06T07:18:14.551865+0000","last_undegraded":"2024-03-06T07:18:14.551865+0000","last_fullsized":"2024-03-06T07:18:14.551865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718876+0000","last_change":"2024-03-06T07:18:08.518033+0000","last_active":"2024-03-06T07:18:14.718876+0000","last_peered":"2024-03-06T07:18:14.718876+0000","last_clean":"2024-03-06T07:18:14.718876+0000","last_became_active":"2024-03-06T07:18:08.517752+0000","last_became_peered":"2024-03-06T07:18:08.517752+0000","last_unstale":"2024-03-06T07:18:14.718876+0000","last_undegraded":"2024-03-06T07:18:14.718876+0000","last_fullsized":"2024-03-06T07:18:14.718876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551959+0000","last_change":"2024-03-06T07:18:13.561392+0000","last_active":"2024-03-06T07:18:14.551959+0000","last_peered":"2024-03-06T07:18:14.551959+0000","last_clean":"2024-03-06T07:18:14.551959+0000","last_became_active":"2024-03-06T07:18:13.561014+0000","last_became_peered":"2024-03-06T07:18:13.561014+0000","last_unstale":"2024-03-06T07:18:14.551959+0000","last_undegraded":"2024-03-06T07:18:14.551959+0000","last_fullsized":"2024-03-06T07:18:14.551959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:43.183715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719832+0000","last_change":"2024-03-06T07:18:09.531541+0000","last_active":"2024-03-06T07:18:14.719832+0000","last_peered":"2024-03-06T07:18:14.719832+0000","last_clean":"2024-03-06T07:18:14.719832+0000","last_became_active":"2024-03-06T07:18:09.531248+0000","last_became_peered":"2024-03-06T07:18:09.531248+0000","last_unstale":"2024-03-06T07:18:14.719832+0000","last_undegraded":"2024-03-06T07:18:14.719832+0000","last_fullsized":"2024-03-06T07:18:14.719832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719237+0000","last_change":"2024-03-06T07:18:08.519552+0000","last_active":"2024-03-06T07:18:14.719237+0000","last_peered":"2024-03-06T07:18:14.719237+0000","last_clean":"2024-03-06T07:18:14.719237+0000","last_became_active":"2024-03-06T07:18:08.519273+0000","last_became_peered":"2024-03-06T07:18:08.519273+0000","last_unstale":"2024-03-06T07:18:14.719237+0000","last_undegraded":"2024-03-06T07:18:14.719237+0000","last_fullsized":"2024-03-06T07:18:14.719237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720395+0000","last_change":"2024-03-06T07:18:13.564953+0000","last_active":"2024-03-06T07:18:14.720395+0000","last_peered":"2024-03-06T07:18:14.720395+0000","last_clean":"2024-03-06T07:18:14.720395+0000","last_became_active":"2024-03-06T07:18:13.564601+0000","last_became_peered":"2024-03-06T07:18:13.564601+0000","last_unstale":"2024-03-06T07:18:14.720395+0000","last_undegraded":"2024-03-06T07:18:14.720395+0000","last_fullsized":"2024-03-06T07:18:14.720395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:31.246872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718301+0000","last_change":"2024-03-06T07:18:09.535125+0000","last_active":"2024-03-06T07:18:14.718301+0000","last_peered":"2024-03-06T07:18:14.718301+0000","last_clean":"2024-03-06T07:18:14.718301+0000","last_became_active":"2024-03-06T07:18:09.534863+0000","last_became_peered":"2024-03-06T07:18:09.534863+0000","last_unstale":"2024-03-06T07:18:14.718301+0000","last_undegraded":"2024-03-06T07:18:14.718301+0000","last_fullsized":"2024-03-06T07:18:14.718301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551568+0000","last_change":"2024-03-06T07:18:08.517122+0000","last_active":"2024-03-06T07:18:14.551568+0000","last_peered":"2024-03-06T07:18:14.551568+0000","last_clean":"2024-03-06T07:18:14.551568+0000","last_became_active":"2024-03-06T07:18:08.516566+0000","last_became_peered":"2024-03-06T07:18:08.516566+0000","last_unstale":"2024-03-06T07:18:14.551568+0000","last_undegraded":"2024-03-06T07:18:14.551568+0000","last_fullsized":"2024-03-06T07:18:14.551568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720688+0000","last_change":"2024-03-06T07:18:13.789985+0000","last_active":"2024-03-06T07:18:14.720688+0000","last_peered":"2024-03-06T07:18:14.720688+0000","last_clean":"2024-03-06T07:18:14.720688+0000","last_became_active":"2024-03-06T07:18:13.789627+0000","last_became_peered":"2024-03-06T07:18:13.789627+0000","last_unstale":"2024-03-06T07:18:14.720688+0000","last_undegraded":"2024-03-06T07:18:14.720688+0000","last_fullsized":"2024-03-06T07:18:14.720688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:22:45.216688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718902+0000","last_change":"2024-03-06T07:18:13.786557+0000","last_active":"2024-03-06T07:18:14.718902+0000","last_peered":"2024-03-06T07:18:14.718902+0000","last_clean":"2024-03-06T07:18:14.718902+0000","last_became_active":"2024-03-06T07:18:13.786179+0000","last_became_peered":"2024-03-06T07:18:13.786179+0000","last_unstale":"2024-03-06T07:18:14.718902+0000","last_undegraded":"2024-03-06T07:18:14.718902+0000","last_fullsized":"2024-03-06T07:18:14.718902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719435+0000","last_change":"2024-03-06T07:18:09.528134+0000","last_active":"2024-03-06T07:18:14.719435+0000","last_peered":"2024-03-06T07:18:14.719435+0000","last_clean":"2024-03-06T07:18:14.719435+0000","last_became_active":"2024-03-06T07:18:09.527831+0000","last_became_peered":"2024-03-06T07:18:09.527831+0000","last_unstale":"2024-03-06T07:18:14.719435+0000","last_undegraded":"2024-03-06T07:18:14.719435+0000","last_fullsized":"2024-03-06T07:18:14.719435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718145+0000","last_change":"2024-03-06T07:18:08.520786+0000","last_active":"2024-03-06T07:18:14.718145+0000","last_peered":"2024-03-06T07:18:14.718145+0000","last_clean":"2024-03-06T07:18:14.718145+0000","last_became_active":"2024-03-06T07:18:08.520392+0000","last_became_peered":"2024-03-06T07:18:08.520392+0000","last_unstale":"2024-03-06T07:18:14.718145+0000","last_undegraded":"2024-03-06T07:18:14.718145+0000","last_fullsized":"2024-03-06T07:18:14.718145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551680+0000","last_change":"2024-03-06T07:18:13.566639+0000","last_active":"2024-03-06T07:18:14.551680+0000","last_peered":"2024-03-06T07:18:14.551680+0000","last_clean":"2024-03-06T07:18:14.551680+0000","last_became_active":"2024-03-06T07:18:13.566321+0000","last_became_peered":"2024-03-06T07:18:13.566321+0000","last_unstale":"2024-03-06T07:18:14.551680+0000","last_undegraded":"2024-03-06T07:18:14.551680+0000","last_fullsized":"2024-03-06T07:18:14.551680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720366+0000","last_change":"2024-03-06T07:18:09.523023+0000","last_active":"2024-03-06T07:18:14.720366+0000","last_peered":"2024-03-06T07:18:14.720366+0000","last_clean":"2024-03-06T07:18:14.720366+0000","last_became_active":"2024-03-06T07:18:09.522736+0000","last_became_peered":"2024-03-06T07:18:09.522736+0000","last_unstale":"2024-03-06T07:18:14.720366+0000","last_undegraded":"2024-03-06T07:18:14.720366+0000","last_fullsized":"2024-03-06T07:18:14.720366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720831+0000","last_change":"2024-03-06T07:18:08.516225+0000","last_active":"2024-03-06T07:18:14.720831+0000","last_peered":"2024-03-06T07:18:14.720831+0000","last_clean":"2024-03-06T07:18:14.720831+0000","last_became_active":"2024-03-06T07:18:08.515908+0000","last_became_peered":"2024-03-06T07:18:08.515908+0000","last_unstale":"2024-03-06T07:18:14.720831+0000","last_undegraded":"2024-03-06T07:18:14.720831+0000","last_fullsized":"2024-03-06T07:18:14.720831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550914+0000","last_change":"2024-03-06T07:18:08.513484+0000","last_active":"2024-03-06T07:18:14.550914+0000","last_peered":"2024-03-06T07:18:14.550914+0000","last_clean":"2024-03-06T07:18:14.550914+0000","last_became_active":"2024-03-06T07:18:08.513195+0000","last_became_peered":"2024-03-06T07:18:08.513195+0000","last_unstale":"2024-03-06T07:18:14.550914+0000","last_undegraded":"2024-03-06T07:18:14.550914+0000","last_fullsized":"2024-03-06T07:18:14.550914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719247+0000","last_change":"2024-03-06T07:18:09.518972+0000","last_active":"2024-03-06T07:18:14.719247+0000","last_peered":"2024-03-06T07:18:14.719247+0000","last_clean":"2024-03-06T07:18:14.719247+0000","last_became_active":"2024-03-06T07:18:09.518673+0000","last_became_peered":"2024-03-06T07:18:09.518673+0000","last_unstale":"2024-03-06T07:18:14.719247+0000","last_undegraded":"2024-03-06T07:18:14.719247+0000","last_fullsized":"2024-03-06T07:18:14.719247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548431+0000","last_change":"2024-03-06T07:18:13.569839+0000","last_active":"2024-03-06T07:18:14.548431+0000","last_peered":"2024-03-06T07:18:14.548431+0000","last_clean":"2024-03-06T07:18:14.548431+0000","last_became_active":"2024-03-06T07:18:13.569443+0000","last_became_peered":"2024-03-06T07:18:13.569443+0000","last_unstale":"2024-03-06T07:18:14.548431+0000","last_undegraded":"2024-03-06T07:18:14.548431+0000","last_fullsized":"2024-03-06T07:18:14.548431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:45.425994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720231+0000","last_change":"2024-03-06T07:18:08.519883+0000","last_active":"2024-03-06T07:18:14.720231+0000","last_peered":"2024-03-06T07:18:14.720231+0000","last_clean":"2024-03-06T07:18:14.720231+0000","last_became_active":"2024-03-06T07:18:08.519579+0000","last_became_peered":"2024-03-06T07:18:08.519579+0000","last_unstale":"2024-03-06T07:18:14.720231+0000","last_undegraded":"2024-03-06T07:18:14.720231+0000","last_fullsized":"2024-03-06T07:18:14.720231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551887+0000","last_change":"2024-03-06T07:18:09.520603+0000","last_active":"2024-03-06T07:18:14.551887+0000","last_peered":"2024-03-06T07:18:14.551887+0000","last_clean":"2024-03-06T07:18:14.551887+0000","last_became_active":"2024-03-06T07:18:09.520258+0000","last_became_peered":"2024-03-06T07:18:09.520258+0000","last_unstale":"2024-03-06T07:18:14.551887+0000","last_undegraded":"2024-03-06T07:18:14.551887+0000","last_fullsized":"2024-03-06T07:18:14.551887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720637+0000","last_change":"2024-03-06T07:18:13.564140+0000","last_active":"2024-03-06T07:18:14.720637+0000","last_peered":"2024-03-06T07:18:14.720637+0000","last_clean":"2024-03-06T07:18:14.720637+0000","last_became_active":"2024-03-06T07:18:13.563770+0000","last_became_peered":"2024-03-06T07:18:13.563770+0000","last_unstale":"2024-03-06T07:18:14.720637+0000","last_undegraded":"2024-03-06T07:18:14.720637+0000","last_fullsized":"2024-03-06T07:18:14.720637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:03.956217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550619+0000","last_change":"2024-03-06T07:18:09.520176+0000","last_active":"2024-03-06T07:18:14.550619+0000","last_peered":"2024-03-06T07:18:14.550619+0000","last_clean":"2024-03-06T07:18:14.550619+0000","last_became_active":"2024-03-06T07:18:09.519911+0000","last_became_peered":"2024-03-06T07:18:09.519911+0000","last_unstale":"2024-03-06T07:18:14.550619+0000","last_undegraded":"2024-03-06T07:18:14.550619+0000","last_fullsized":"2024-03-06T07:18:14.550619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548193+0000","last_change":"2024-03-06T07:18:09.524681+0000","last_active":"2024-03-06T07:18:14.548193+0000","last_peered":"2024-03-06T07:18:14.548193+0000","last_clean":"2024-03-06T07:18:14.548193+0000","last_became_active":"2024-03-06T07:18:09.524427+0000","last_became_peered":"2024-03-06T07:18:09.524427+0000","last_unstale":"2024-03-06T07:18:14.548193+0000","last_undegraded":"2024-03-06T07:18:14.548193+0000","last_fullsized":"2024-03-06T07:18:14.548193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.531928+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531670+0000","last_became_peered":"2024-03-06T07:18:09.531670+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719143+0000","last_change":"2024-03-06T07:18:09.523431+0000","last_active":"2024-03-06T07:18:14.719143+0000","last_peered":"2024-03-06T07:18:14.719143+0000","last_clean":"2024-03-06T07:18:14.719143+0000","last_became_active":"2024-03-06T07:18:09.523169+0000","last_became_peered":"2024-03-06T07:18:09.523169+0000","last_unstale":"2024-03-06T07:18:14.719143+0000","last_undegraded":"2024-03-06T07:18:14.719143+0000","last_fullsized":"2024-03-06T07:18:14.719143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720897+0000","last_change":"2024-03-06T07:18:09.525359+0000","last_active":"2024-03-06T07:18:14.720897+0000","last_peered":"2024-03-06T07:18:14.720897+0000","last_clean":"2024-03-06T07:18:14.720897+0000","last_became_active":"2024-03-06T07:18:09.525069+0000","last_became_peered":"2024-03-06T07:18:09.525069+0000","last_unstale":"2024-03-06T07:18:14.720897+0000","last_undegraded":"2024-03-06T07:18:14.720897+0000","last_fullsized":"2024-03-06T07:18:14.720897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718709+0000","last_change":"2024-03-06T07:18:09.528783+0000","last_active":"2024-03-06T07:18:14.718709+0000","last_peered":"2024-03-06T07:18:14.718709+0000","last_clean":"2024-03-06T07:18:14.718709+0000","last_became_active":"2024-03-06T07:18:09.528488+0000","last_became_peered":"2024-03-06T07:18:09.528488+0000","last_unstale":"2024-03-06T07:18:14.718709+0000","last_undegraded":"2024-03-06T07:18:14.718709+0000","last_fullsized":"2024-03-06T07:18:14.718709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551739+0000","last_change":"2024-03-06T07:18:09.526930+0000","last_active":"2024-03-06T07:18:14.551739+0000","last_peered":"2024-03-06T07:18:14.551739+0000","last_clean":"2024-03-06T07:18:14.551739+0000","last_became_active":"2024-03-06T07:18:09.526575+0000","last_became_peered":"2024-03-06T07:18:09.526575+0000","last_unstale":"2024-03-06T07:18:14.551739+0000","last_undegraded":"2024-03-06T07:18:14.551739+0000","last_fullsized":"2024-03-06T07:18:14.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720809+0000","last_change":"2024-03-06T07:18:09.532541+0000","last_active":"2024-03-06T07:18:14.720809+0000","last_peered":"2024-03-06T07:18:14.720809+0000","last_clean":"2024-03-06T07:18:14.720809+0000","last_became_active":"2024-03-06T07:18:09.532260+0000","last_became_peered":"2024-03-06T07:18:09.532260+0000","last_unstale":"2024-03-06T07:18:14.720809+0000","last_undegraded":"2024-03-06T07:18:14.720809+0000","last_fullsized":"2024-03-06T07:18:14.720809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718061+0000","last_change":"2024-03-06T07:18:09.530812+0000","last_active":"2024-03-06T07:18:14.718061+0000","last_peered":"2024-03-06T07:18:14.718061+0000","last_clean":"2024-03-06T07:18:14.718061+0000","last_became_active":"2024-03-06T07:18:09.530493+0000","last_became_peered":"2024-03-06T07:18:09.530493+0000","last_unstale":"2024-03-06T07:18:14.718061+0000","last_undegraded":"2024-03-06T07:18:14.718061+0000","last_fullsized":"2024-03-06T07:18:14.718061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720741+0000","last_change":"2024-03-06T07:18:09.525260+0000","last_active":"2024-03-06T07:18:14.720741+0000","last_peered":"2024-03-06T07:18:14.720741+0000","last_clean":"2024-03-06T07:18:14.720741+0000","last_became_active":"2024-03-06T07:18:09.524996+0000","last_became_peered":"2024-03-06T07:18:09.524996+0000","last_unstale":"2024-03-06T07:18:14.720741+0000","last_undegraded":"2024-03-06T07:18:14.720741+0000","last_fullsized":"2024-03-06T07:18:14.720741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718326+0000","last_change":"2024-03-06T07:18:09.539784+0000","last_active":"2024-03-06T07:18:14.718326+0000","last_peered":"2024-03-06T07:18:14.718326+0000","last_clean":"2024-03-06T07:18:14.718326+0000","last_became_active":"2024-03-06T07:18:09.538642+0000","last_became_peered":"2024-03-06T07:18:09.538642+0000","last_unstale":"2024-03-06T07:18:14.718326+0000","last_undegraded":"2024-03-06T07:18:14.718326+0000","last_fullsized":"2024-03-06T07:18:14.718326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719031+0000","last_change":"2024-03-06T07:18:09.526156+0000","last_active":"2024-03-06T07:18:14.719031+0000","last_peered":"2024-03-06T07:18:14.719031+0000","last_clean":"2024-03-06T07:18:14.719031+0000","last_became_active":"2024-03-06T07:18:09.525877+0000","last_became_peered":"2024-03-06T07:18:09.525877+0000","last_unstale":"2024-03-06T07:18:14.719031+0000","last_undegraded":"2024-03-06T07:18:14.719031+0000","last_fullsized":"2024-03-06T07:18:14.719031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551188+0000","last_change":"2024-03-06T07:18:09.516685+0000","last_active":"2024-03-06T07:18:14.551188+0000","last_peered":"2024-03-06T07:18:14.551188+0000","last_clean":"2024-03-06T07:18:14.551188+0000","last_became_active":"2024-03-06T07:18:09.516377+0000","last_became_peered":"2024-03-06T07:18:09.516377+0000","last_unstale":"2024-03-06T07:18:14.551188+0000","last_undegraded":"2024-03-06T07:18:14.551188+0000","last_fullsized":"2024-03-06T07:18:14.551188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718906+0000","last_change":"2024-03-06T07:18:09.524856+0000","last_active":"2024-03-06T07:18:14.718906+0000","last_peered":"2024-03-06T07:18:14.718906+0000","last_clean":"2024-03-06T07:18:14.718906+0000","last_became_active":"2024-03-06T07:18:09.523916+0000","last_became_peered":"2024-03-06T07:18:09.523916+0000","last_unstale":"2024-03-06T07:18:14.718906+0000","last_undegraded":"2024-03-06T07:18:14.718906+0000","last_fullsized":"2024-03-06T07:18:14.718906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548544+0000","last_change":"2024-03-06T07:18:09.518046+0000","last_active":"2024-03-06T07:18:14.548544+0000","last_peered":"2024-03-06T07:18:14.548544+0000","last_clean":"2024-03-06T07:18:14.548544+0000","last_became_active":"2024-03-06T07:18:09.517756+0000","last_became_peered":"2024-03-06T07:18:09.517756+0000","last_unstale":"2024-03-06T07:18:14.548544+0000","last_undegraded":"2024-03-06T07:18:14.548544+0000","last_fullsized":"2024-03-06T07:18:14.548544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719287+0000","last_change":"2024-03-06T07:18:09.537036+0000","last_active":"2024-03-06T07:18:14.719287+0000","last_peered":"2024-03-06T07:18:14.719287+0000","last_clean":"2024-03-06T07:18:14.719287+0000","last_became_active":"2024-03-06T07:18:09.536576+0000","last_became_peered":"2024-03-06T07:18:09.536576+0000","last_unstale":"2024-03-06T07:18:14.719287+0000","last_undegraded":"2024-03-06T07:18:14.719287+0000","last_fullsized":"2024-03-06T07:18:14.719287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.717960+0000","last_change":"2024-03-06T07:18:09.527339+0000","last_active":"2024-03-06T07:18:14.717960+0000","last_peered":"2024-03-06T07:18:14.717960+0000","last_clean":"2024-03-06T07:18:14.717960+0000","last_became_active":"2024-03-06T07:18:09.526873+0000","last_became_peered":"2024-03-06T07:18:09.526873+0000","last_unstale":"2024-03-06T07:18:14.717960+0000","last_undegraded":"2024-03-06T07:18:14.717960+0000","last_fullsized":"2024-03-06T07:18:14.717960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719530+0000","last_change":"2024-03-06T07:18:09.522712+0000","last_active":"2024-03-06T07:18:14.719530+0000","last_peered":"2024-03-06T07:18:14.719530+0000","last_clean":"2024-03-06T07:18:14.719530+0000","last_became_active":"2024-03-06T07:18:09.522433+0000","last_became_peered":"2024-03-06T07:18:09.522433+0000","last_unstale":"2024-03-06T07:18:14.719530+0000","last_undegraded":"2024-03-06T07:18:14.719530+0000","last_fullsized":"2024-03-06T07:18:14.719530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719097+0000","last_change":"2024-03-06T07:18:09.522057+0000","last_active":"2024-03-06T07:18:14.719097+0000","last_peered":"2024-03-06T07:18:14.719097+0000","last_clean":"2024-03-06T07:18:14.719097+0000","last_became_active":"2024-03-06T07:18:09.521270+0000","last_became_peered":"2024-03-06T07:18:09.521270+0000","last_unstale":"2024-03-06T07:18:14.719097+0000","last_undegraded":"2024-03-06T07:18:14.719097+0000","last_fullsized":"2024-03-06T07:18:14.719097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:57.186351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720044+0000","last_change":"2024-03-06T07:18:09.526002+0000","last_active":"2024-03-06T07:18:14.720044+0000","last_peered":"2024-03-06T07:18:14.720044+0000","last_clean":"2024-03-06T07:18:14.720044+0000","last_became_active":"2024-03-06T07:18:09.525717+0000","last_became_peered":"2024-03-06T07:18:09.525717+0000","last_unstale":"2024-03-06T07:18:14.720044+0000","last_undegraded":"2024-03-06T07:18:14.720044+0000","last_fullsized":"2024-03-06T07:18:14.720044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551860+0000","last_change":"2024-03-06T07:18:09.521766+0000","last_active":"2024-03-06T07:18:14.551860+0000","last_peered":"2024-03-06T07:18:14.551860+0000","last_clean":"2024-03-06T07:18:14.551860+0000","last_became_active":"2024-03-06T07:18:09.521159+0000","last_became_peered":"2024-03-06T07:18:09.521159+0000","last_unstale":"2024-03-06T07:18:14.551860+0000","last_undegraded":"2024-03-06T07:18:14.551860+0000","last_fullsized":"2024-03-06T07:18:14.551860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:57.265260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551251+0000","last_change":"2024-03-06T07:18:09.533157+0000","last_active":"2024-03-06T07:18:14.551251+0000","last_peered":"2024-03-06T07:18:14.551251+0000","last_clean":"2024-03-06T07:18:14.551251+0000","last_became_active":"2024-03-06T07:18:09.532868+0000","last_became_peered":"2024-03-06T07:18:09.532868+0000","last_unstale":"2024-03-06T07:18:14.551251+0000","last_undegraded":"2024-03-06T07:18:14.551251+0000","last_fullsized":"2024-03-06T07:18:14.551251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551648+0000","last_change":"2024-03-06T07:18:09.532473+0000","last_active":"2024-03-06T07:18:14.551648+0000","last_peered":"2024-03-06T07:18:14.551648+0000","last_clean":"2024-03-06T07:18:14.551648+0000","last_became_active":"2024-03-06T07:18:09.532191+0000","last_became_peered":"2024-03-06T07:18:09.532191+0000","last_unstale":"2024-03-06T07:18:14.551648+0000","last_undegraded":"2024-03-06T07:18:14.551648+0000","last_fullsized":"2024-03-06T07:18:14.551648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719062+0000","last_change":"2024-03-06T07:18:09.533061+0000","last_active":"2024-03-06T07:18:14.719062+0000","last_peered":"2024-03-06T07:18:14.719062+0000","last_clean":"2024-03-06T07:18:14.719062+0000","last_became_active":"2024-03-06T07:18:09.532742+0000","last_became_peered":"2024-03-06T07:18:09.532742+0000","last_unstale":"2024-03-06T07:18:14.719062+0000","last_undegraded":"2024-03-06T07:18:14.719062+0000","last_fullsized":"2024-03-06T07:18:14.719062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:25.345923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720283+0000","last_change":"2024-03-06T07:18:09.532342+0000","last_active":"2024-03-06T07:18:14.720283+0000","last_peered":"2024-03-06T07:18:14.720283+0000","last_clean":"2024-03-06T07:18:14.720283+0000","last_became_active":"2024-03-06T07:18:09.532062+0000","last_became_peered":"2024-03-06T07:18:09.532062+0000","last_unstale":"2024-03-06T07:18:14.720283+0000","last_undegraded":"2024-03-06T07:18:14.720283+0000","last_fullsized":"2024-03-06T07:18:14.720283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718610+0000","last_change":"2024-03-06T07:18:09.525768+0000","last_active":"2024-03-06T07:18:14.718610+0000","last_peered":"2024-03-06T07:18:14.718610+0000","last_clean":"2024-03-06T07:18:14.718610+0000","last_became_active":"2024-03-06T07:18:09.524604+0000","last_became_peered":"2024-03-06T07:18:09.524604+0000","last_unstale":"2024-03-06T07:18:14.718610+0000","last_undegraded":"2024-03-06T07:18:14.718610+0000","last_fullsized":"2024-03-06T07:18:14.718610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718219+0000","last_change":"2024-03-06T07:18:09.531193+0000","last_active":"2024-03-06T07:18:14.718219+0000","last_peered":"2024-03-06T07:18:14.718219+0000","last_clean":"2024-03-06T07:18:14.718219+0000","last_became_active":"2024-03-06T07:18:09.530931+0000","last_became_peered":"2024-03-06T07:18:09.530931+0000","last_unstale":"2024-03-06T07:18:14.718219+0000","last_undegraded":"2024-03-06T07:18:14.718219+0000","last_fullsized":"2024-03-06T07:18:14.718219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718236+0000","last_change":"2024-03-06T07:18:09.524492+0000","last_active":"2024-03-06T07:18:14.718236+0000","last_peered":"2024-03-06T07:18:14.718236+0000","last_clean":"2024-03-06T07:18:14.718236+0000","last_became_active":"2024-03-06T07:18:09.524202+0000","last_became_peered":"2024-03-06T07:18:09.524202+0000","last_unstale":"2024-03-06T07:18:14.718236+0000","last_undegraded":"2024-03-06T07:18:14.718236+0000","last_fullsized":"2024-03-06T07:18:14.718236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.532118+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531771+0000","last_became_peered":"2024-03-06T07:18:09.531771+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719327+0000","last_change":"2024-03-06T07:18:09.519755+0000","last_active":"2024-03-06T07:18:14.719327+0000","last_peered":"2024-03-06T07:18:14.719327+0000","last_clean":"2024-03-06T07:18:14.719327+0000","last_became_active":"2024-03-06T07:18:09.519475+0000","last_became_peered":"2024-03-06T07:18:09.519475+0000","last_unstale":"2024-03-06T07:18:14.719327+0000","last_undegraded":"2024-03-06T07:18:14.719327+0000","last_fullsized":"2024-03-06T07:18:14.719327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719345+0000","last_change":"2024-03-06T07:18:09.524863+0000","last_active":"2024-03-06T07:18:14.719345+0000","last_peered":"2024-03-06T07:18:14.719345+0000","last_clean":"2024-03-06T07:18:14.719345+0000","last_became_active":"2024-03-06T07:18:09.524588+0000","last_became_peered":"2024-03-06T07:18:09.524588+0000","last_unstale":"2024-03-06T07:18:14.719345+0000","last_undegraded":"2024-03-06T07:18:14.719345+0000","last_fullsized":"2024-03-06T07:18:14.719345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718889+0000","last_change":"2024-03-06T07:18:09.528727+0000","last_active":"2024-03-06T07:18:14.718889+0000","last_peered":"2024-03-06T07:18:14.718889+0000","last_clean":"2024-03-06T07:18:14.718889+0000","last_became_active":"2024-03-06T07:18:09.528474+0000","last_became_peered":"2024-03-06T07:18:09.528474+0000","last_unstale":"2024-03-06T07:18:14.718889+0000","last_undegraded":"2024-03-06T07:18:14.718889+0000","last_fullsized":"2024-03-06T07:18:14.718889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:18:33.022 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:18:33.023 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:18:33.023 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:18:33.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:18:33.023 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:18:33.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.156+0000 7f373369a640 1 Processor -- start 2024-03-06T07:18:33.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.156+0000 7f373369a640 1 -- start start 2024-03-06T07:18:33.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f373369a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f372c06da40 0x7f372c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f373369a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c7380 0x7f372c0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f373369a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372c156050 con 0x7f372c0c6780 2024-03-06T07:18:33.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f373369a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372c06de20 con 0x7f372c06da40 2024-03-06T07:18:33.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f373369a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c072630 con 0x7f372c0c7380 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731436640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f372c06da40 0x7f372c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731436640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f372c06da40 0x7f372c155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40784/0 (socket says 172.21.15.94:40784) 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731436640 1 -- 172.21.15.94:0/4057363407 learned_addr learned my addr 172.21.15.94:0/4057363407 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731c37640 1 --2- 172.21.15.94:0/4057363407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c7380 0x7f372c0c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731436640 1 -- 172.21.15.94:0/4057363407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c7380 msgr2=0x7f372c0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731436640 1 --2- 172.21.15.94:0/4057363407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c7380 0x7f372c0c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731436640 1 -- 172.21.15.94:0/4057363407 >> v1:172.21.15.145:6789/0 conn(0x7f372c0c6780 legacy=0x7f372c158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.157+0000 7f3731436640 1 -- 172.21.15.94:0/4057363407 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c072910 con 0x7f372c06da40 2024-03-06T07:18:33.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.158+0000 7f3731436640 1 --2- 172.21.15.94:0/4057363407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c155b10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3728009870 tx=0x7f372802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ed6f27ec85ed75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.158+0000 7f3723fff640 1 -- 172.21.15.94:0/4057363407 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728035020 con 0x7f372c06da40 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.158+0000 7f3723fff640 1 -- 172.21.15.94:0/4057363407 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3728031640 con 0x7f372c06da40 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.158+0000 7f3723fff640 1 -- 172.21.15.94:0/4057363407 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372802d040 con 0x7f372c06da40 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 -- 172.21.15.94:0/4057363407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 msgr2=0x7f372c155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 --2- 172.21.15.94:0/4057363407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c155b10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3728009870 tx=0x7f372802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 -- 172.21.15.94:0/4057363407 shutdown_connections 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 --2- 172.21.15.94:0/4057363407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c7380 0x7f372c0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 --2- 172.21.15.94:0/4057363407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c155b10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 -- 172.21.15.94:0/4057363407 >> 172.21.15.94:0/4057363407 conn(0x7f372c0aa630 msgr2=0x7f372c0aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 -- 172.21.15.94:0/4057363407 shutdown_connections 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.159+0000 7f373369a640 1 -- 172.21.15.94:0/4057363407 wait complete. 2024-03-06T07:18:33.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.160+0000 7f373369a640 1 Processor -- start 2024-03-06T07:18:33.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.160+0000 7f373369a640 1 -- start start 2024-03-06T07:18:33.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c6780 0x7f372c154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f372c0c7380 0x7f372c1eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c1594a0 con 0x7f372c06da40 2024-03-06T07:18:33.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c158890 con 0x7f372c0c7380 2024-03-06T07:18:33.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0c37c0 con 0x7f372c0c6780 2024-03-06T07:18:33.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40798/0 (socket says 172.21.15.94:40798) 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 -- 172.21.15.94:0/2403754637 learned_addr learned my addr 172.21.15.94:0/2403754637 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731c37640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f372c0c7380 0x7f372c1eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3730c35640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c6780 0x7f372c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 -- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c6780 msgr2=0x7f372c154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c6780 0x7f372c154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 -- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f372c0c7380 msgr2=0x7f372c1eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f372c0c7380 0x7f372c1eea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 -- 172.21.15.94:0/2403754637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c0adef0 con 0x7f372c06da40 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3730c35640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c6780 0x7f372c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3731436640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c156f70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f372802b930 tx=0x7f37280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3721ffb640 1 -- 172.21.15.94:0/2403754637 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728040070 con 0x7f372c06da40 2024-03-06T07:18:33.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3721ffb640 1 -- 172.21.15.94:0/2403754637 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372803e070 con 0x7f372c06da40 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f3721ffb640 1 -- 172.21.15.94:0/2403754637 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728035040 con 0x7f372c06da40 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c0ae440 con 0x7f372c06da40 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.161+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c1ee250 con 0x7f372c06da40 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.163+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372c04f330 con 0x7f372c06da40 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.163+0000 7f3721ffb640 1 -- 172.21.15.94:0/2403754637 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3728038020 con 0x7f372c06da40 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.163+0000 7f3721ffb640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f371006b4d0 0x7f371006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.163+0000 7f3721ffb640 1 -- 172.21.15.94:0/2403754637 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f37280abe70 con 0x7f372c06da40 2024-03-06T07:18:33.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.163+0000 7f3730c35640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f371006b4d0 0x7f371006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.164+0000 7f3730c35640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f371006b4d0 0x7f371006d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f372c0aba50 tx=0x7f371c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.168+0000 7f3721ffb640 1 -- 172.21.15.94:0/2403754637 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f372807aa40 con 0x7f372c06da40 2024-03-06T07:18:33.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.385+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f372c0aec10 con 0x7f372c06da40 2024-03-06T07:18:33.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.388+0000 7f3721ffb640 1 -- 172.21.15.94:0/2403754637 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171792 (secure 0 0 0) 0x7f372807a340 con 0x7f372c06da40 2024-03-06T07:18:33.393 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f371006b4d0 msgr2=0x7f371006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f371006b4d0 0x7f371006d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f372c0aba50 tx=0x7f371c009040 comp rx=0 tx=0).stop 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 msgr2=0x7f372c156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c156f70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f372802b930 tx=0x7f37280046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 shutdown_connections 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f371006b4d0 0x7f371006d990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f372c0c7380 0x7f372c1eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f372c0c6780 0x7f372c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 --2- 172.21.15.94:0/2403754637 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f372c06da40 0x7f372c156f70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 >> 172.21.15.94:0/2403754637 conn(0x7f372c0aa630 msgr2=0x7f372c0c2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:33.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 shutdown_connections 2024-03-06T07:18:33.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.391+0000 7f373369a640 1 -- 172.21.15.94:0/2403754637 wait complete. 2024-03-06T07:18:33.404 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2240241265},{"type":"v1","addr":"172.21.15.145:6833","nonce":2240241265}]},"active_addr":"172.21.15.145:6833/2240241265","active_change":"2024-03-06T07:17:56.626152+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":1836194904}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":2443300601}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":3861649436}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":2476066038}]}]} 2024-03-06T07:18:33.408 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:18:33.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:18:33.408 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:33.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.507+0000 7fe30e713640 1 Processor -- start 2024-03-06T07:18:33.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.507+0000 7fe30e713640 1 -- start start 2024-03-06T07:18:33.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.508+0000 7fe30e713640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308155c80 0x7fe30806ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.508+0000 7fe30e713640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080c1ae0 con 0x7fe3080720a0 2024-03-06T07:18:33.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.508+0000 7fe30e713640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30806e450 con 0x7fe308152520 2024-03-06T07:18:33.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.508+0000 7fe30e713640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080c1dc0 con 0x7fe308155c80 2024-03-06T07:18:33.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.508+0000 7fe30ccb0640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe3080720a0 0x7fe30806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41422/0 (socket says 172.21.15.94:41422) 2024-03-06T07:18:33.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.508+0000 7fe30ccb0640 1 -- 172.21.15.94:0/3912819180 learned_addr learned my addr 172.21.15.94:0/3912819180 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:33.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.508+0000 7fe3077fe640 1 --2- 172.21.15.94:0/3912819180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308155c80 0x7fe30806ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.509+0000 7fe306ffd640 1 -- 172.21.15.94:0/3912819180 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281745065 0 0) 0x7fe3080c1ae0 con 0x7fe3080720a0 2024-03-06T07:18:33.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.509+0000 7fe306ffd640 1 -- 172.21.15.94:0/3912819180 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f0003520 con 0x7fe3080720a0 2024-03-06T07:18:33.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.509+0000 7fe306ffd640 1 -- 172.21.15.94:0/3912819180 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970709859 0 0) 0x7fe30806e450 con 0x7fe308152520 2024-03-06T07:18:33.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.509+0000 7fe3077fe640 1 -- 172.21.15.94:0/3912819180 >> v1:172.21.15.145:6789/0 conn(0x7fe308152520 legacy=0x7fe3080c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.509+0000 7fe3077fe640 1 -- 172.21.15.94:0/3912819180 >> v1:172.21.15.94:6789/0 conn(0x7fe3080720a0 legacy=0x7fe30806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.509+0000 7fe3077fe640 1 -- 172.21.15.94:0/3912819180 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30806e730 con 0x7fe308155c80 2024-03-06T07:18:33.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.509+0000 7fe3077fe640 1 --2- 172.21.15.94:0/3912819180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308155c80 0x7fe30806ef30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0097d0 tx=0x7fe2fc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e2025532864e531 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe306ffd640 1 -- 172.21.15.94:0/3912819180 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc035020 con 0x7fe308155c80 2024-03-06T07:18:33.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe306ffd640 1 -- 172.21.15.94:0/3912819180 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc031890 con 0x7fe308155c80 2024-03-06T07:18:33.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe306ffd640 1 -- 172.21.15.94:0/3912819180 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc031c40 con 0x7fe308155c80 2024-03-06T07:18:33.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe30e713640 1 -- 172.21.15.94:0/3912819180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308155c80 msgr2=0x7fe30806ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe30e713640 1 --2- 172.21.15.94:0/3912819180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308155c80 0x7fe30806ef30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0097d0 tx=0x7fe2fc02b930 comp rx=0 tx=0).stop 2024-03-06T07:18:33.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe30e713640 1 -- 172.21.15.94:0/3912819180 shutdown_connections 2024-03-06T07:18:33.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe30e713640 1 --2- 172.21.15.94:0/3912819180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308155c80 0x7fe30806ef30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe30e713640 1 -- 172.21.15.94:0/3912819180 >> 172.21.15.94:0/3912819180 conn(0x7fe3080c76b0 msgr2=0x7fe3080c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:33.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe30e713640 1 -- 172.21.15.94:0/3912819180 shutdown_connections 2024-03-06T07:18:33.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.510+0000 7fe30e713640 1 -- 172.21.15.94:0/3912819180 wait complete. 2024-03-06T07:18:33.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.511+0000 7fe30e713640 1 Processor -- start 2024-03-06T07:18:33.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.511+0000 7fe30e713640 1 -- start start 2024-03-06T07:18:33.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30e713640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3080720a0 0x7fe3081ea910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30e713640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308152520 0x7fe3081588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30e713640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe308155c80 0x7fe3080a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30e713640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30806b850 con 0x7fe308155c80 2024-03-06T07:18:33.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30e713640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30806b6d0 con 0x7fe3080720a0 2024-03-06T07:18:33.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30e713640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe308152900 con 0x7fe308152520 2024-03-06T07:18:33.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30ccb0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe308155c80 0x7fe3080a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe307fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3080720a0 0x7fe3081ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30ccb0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe308155c80 0x7fe3080a6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40802/0 (socket says 172.21.15.94:40802) 2024-03-06T07:18:33.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe30ccb0640 1 -- 172.21.15.94:0/225744683 learned_addr learned my addr 172.21.15.94:0/225744683 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:33.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.512+0000 7fe3077fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308152520 0x7fe3081588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30ccb0640 1 -- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308152520 msgr2=0x7fe3081588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30ccb0640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308152520 0x7fe3081588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30ccb0640 1 -- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3080720a0 msgr2=0x7fe3081ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30ccb0640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3080720a0 0x7fe3081ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30ccb0640 1 -- 172.21.15.94:0/225744683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe308159320 con 0x7fe308155c80 2024-03-06T07:18:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30ccb0640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe308155c80 0x7fe3080a6f40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe30000caa0 tx=0x7fe30000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe304ff9640 1 -- 172.21.15.94:0/225744683 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300017070 con 0x7fe308155c80 2024-03-06T07:18:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe304ff9640 1 -- 172.21.15.94:0/225744683 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30000aba0 con 0x7fe308155c80 2024-03-06T07:18:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe308157800 con 0x7fe308155c80 2024-03-06T07:18:33.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe304ff9640 1 -- 172.21.15.94:0/225744683 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300012400 con 0x7fe308155c80 2024-03-06T07:18:33.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.513+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe308157dc0 con 0x7fe308155c80 2024-03-06T07:18:33.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.514+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30804f330 con 0x7fe308155c80 2024-03-06T07:18:33.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.514+0000 7fe304ff9640 1 -- 172.21.15.94:0/225744683 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe30001a070 con 0x7fe308155c80 2024-03-06T07:18:33.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.515+0000 7fe304ff9640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.515+0000 7fe304ff9640 1 -- 172.21.15.94:0/225744683 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe30008a170 con 0x7fe308155c80 2024-03-06T07:18:33.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.515+0000 7fe307fff640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.515+0000 7fe307fff640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8000f60 tx=0x7fe2f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.520+0000 7fe304ff9640 1 -- 172.21.15.94:0/225744683 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3000586f0 con 0x7fe308155c80 2024-03-06T07:18:33.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.696+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3080c0b70 con 0x7fe308155c80 2024-03-06T07:18:33.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.698+0000 7fe304ff9640 1 -- 172.21.15.94:0/225744683 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7fe30005c790 con 0x7fe308155c80 2024-03-06T07:18:33.701 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:33.701 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2f406b4d0 msgr2=0x7fe2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8000f60 tx=0x7fe2f8009040 comp rx=0 tx=0).stop 2024-03-06T07:18:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe308155c80 msgr2=0x7fe3080a6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe308155c80 0x7fe3080a6f40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe30000caa0 tx=0x7fe30000cf60 comp rx=0 tx=0).stop 2024-03-06T07:18:33.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 shutdown_connections 2024-03-06T07:18:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe308155c80 0x7fe3080a6f40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe308152520 0x7fe3081588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 --2- 172.21.15.94:0/225744683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3080720a0 0x7fe3081ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 >> 172.21.15.94:0/225744683 conn(0x7fe3080c76b0 msgr2=0x7fe3080c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 shutdown_connections 2024-03-06T07:18:33.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.700+0000 7fe30e713640 1 -- 172.21.15.94:0/225744683 wait complete. 2024-03-06T07:18:33.714 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:18:33.714 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.812+0000 7f59a7165640 1 Processor -- start 2024-03-06T07:18:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.812+0000 7f59a7165640 1 -- start start 2024-03-06T07:18:33.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f59a7165640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f59a006afc0 0x7f59a006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f59a7165640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a00a9530 con 0x7f59a00c20b0 2024-03-06T07:18:33.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f59a7165640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a00a9810 con 0x7f59a00c25c0 2024-03-06T07:18:33.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f59a7165640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a00a9af0 con 0x7f59a006afc0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f599ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f59a00c20b0 0x7f59a0153180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41432/0 (socket says 172.21.15.94:41432) 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f599ffff640 1 -- 172.21.15.94:0/2666021586 learned_addr learned my addr 172.21.15.94:0/2666021586 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f59a4f01640 1 --2- 172.21.15.94:0/2666021586 >> v2:172.21.15.145:3300/0 conn(0x7f59a006afc0 0x7f59a006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f599f7fe640 1 -- 172.21.15.94:0/2666021586 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544305106 0 0) 0x7f59a00a9530 con 0x7f59a00c20b0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.813+0000 7f599f7fe640 1 -- 172.21.15.94:0/2666021586 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5988003520 con 0x7f59a00c20b0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.814+0000 7f59a4f01640 1 -- 172.21.15.94:0/2666021586 >> v1:172.21.15.145:6789/0 conn(0x7f59a00c25c0 legacy=0x7f59a0153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.814+0000 7f59a4f01640 1 -- 172.21.15.94:0/2666021586 >> v1:172.21.15.94:6789/0 conn(0x7f59a00c20b0 legacy=0x7f59a0153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.814+0000 7f59a4f01640 1 -- 172.21.15.94:0/2666021586 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00a9dd0 con 0x7f59a006afc0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.814+0000 7f59a4f01640 1 --2- 172.21.15.94:0/2666021586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 0x7f59a006b3a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f59900097a0 tx=0x7f599002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9213f118ee25dd9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.814+0000 7f599f7fe640 1 -- 172.21.15.94:0/2666021586 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59900315a0 con 0x7f59a006afc0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.814+0000 7f599f7fe640 1 -- 172.21.15.94:0/2666021586 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5990037070 con 0x7f59a006afc0 2024-03-06T07:18:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.814+0000 7f599f7fe640 1 -- 172.21.15.94:0/2666021586 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990031940 con 0x7f59a006afc0 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.815+0000 7f59a7165640 1 -- 172.21.15.94:0/2666021586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 msgr2=0x7f59a006b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.815+0000 7f59a7165640 1 --2- 172.21.15.94:0/2666021586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 0x7f59a006b3a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f59900097a0 tx=0x7f599002b840 comp rx=0 tx=0).stop 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.815+0000 7f59a7165640 1 -- 172.21.15.94:0/2666021586 shutdown_connections 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.815+0000 7f59a7165640 1 --2- 172.21.15.94:0/2666021586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 0x7f59a006b3a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.815+0000 7f59a7165640 1 -- 172.21.15.94:0/2666021586 >> 172.21.15.94:0/2666021586 conn(0x7f59a00c76b0 msgr2=0x7f59a00c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.815+0000 7f59a7165640 1 -- 172.21.15.94:0/2666021586 shutdown_connections 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.815+0000 7f59a7165640 1 -- 172.21.15.94:0/2666021586 wait complete. 2024-03-06T07:18:33.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.816+0000 7f59a7165640 1 Processor -- start 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.816+0000 7f59a7165640 1 -- start start 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a7165640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 0x7f59a0156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a7165640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59a00c20b0 0x7f59a0143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a7165640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a00c25c0 0x7f59a0143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a7165640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0150b00 con 0x7f59a00c20b0 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a7165640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0150980 con 0x7f59a006afc0 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a7165640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00a7d50 con 0x7f59a00c25c0 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59a00c20b0 0x7f59a0143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59a00c20b0 0x7f59a0143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40814/0 (socket says 172.21.15.94:40814) 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 -- 172.21.15.94:0/336693104 learned_addr learned my addr 172.21.15.94:0/336693104 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a4f01640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 0x7f59a0156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f59a5702640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a00c25c0 0x7f59a0143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 -- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a00c25c0 msgr2=0x7f59a0143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a00c25c0 0x7f59a0143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 -- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 msgr2=0x7f59a0156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 0x7f59a0156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 -- 172.21.15.94:0/336693104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a01e8d00 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.817+0000 7f599ffff640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59a00c20b0 0x7f59a0143ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f599400c4d0 tx=0x7f599400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.818+0000 7f599d7fa640 1 -- 172.21.15.94:0/336693104 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994017070 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.818+0000 7f599d7fa640 1 -- 172.21.15.94:0/336693104 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5994005ce0 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.818+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a01e8fe0 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.818+0000 7f599d7fa640 1 -- 172.21.15.94:0/336693104 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994007270 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.818+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a01e95a0 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.819+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a004f330 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.819+0000 7f599d7fa640 1 -- 172.21.15.94:0/336693104 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f599401a070 con 0x7f59a00c20b0 2024-03-06T07:18:33.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.819+0000 7f599d7fa640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f598c06b4d0 0x7f598c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:33.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.820+0000 7f599d7fa640 1 -- 172.21.15.94:0/336693104 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5994089770 con 0x7f59a00c20b0 2024-03-06T07:18:33.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.820+0000 7f59a4f01640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f598c06b4d0 0x7f598c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:33.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.820+0000 7f59a4f01640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f598c06b4d0 0x7f598c06d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5990036000 tx=0x7f5990008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:33.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.824+0000 7f599d7fa640 1 -- 172.21.15.94:0/336693104 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5994057cf0 con 0x7f59a00c20b0 2024-03-06T07:18:34.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.997+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59a00ad810 con 0x7f59a00c20b0 2024-03-06T07:18:34.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:33.998+0000 7f599d7fa640 1 -- 172.21.15.94:0/336693104 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f599405bd90 con 0x7f59a00c20b0 2024-03-06T07:18:34.001 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:34.002 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f598c06b4d0 msgr2=0x7f598c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f598c06b4d0 0x7f598c06d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5990036000 tx=0x7f5990008030 comp rx=0 tx=0).stop 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59a00c20b0 msgr2=0x7f59a0143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59a00c20b0 0x7f59a0143ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f599400c4d0 tx=0x7f599400c990 comp rx=0 tx=0).stop 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 shutdown_connections 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f598c06b4d0 0x7f598c06d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a00c25c0 0x7f59a0143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59a00c20b0 0x7f59a0143ab0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 --2- 172.21.15.94:0/336693104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a006afc0 0x7f59a0156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 >> 172.21.15.94:0/336693104 conn(0x7f59a00c76b0 msgr2=0x7f59a014db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 shutdown_connections 2024-03-06T07:18:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.000+0000 7f59a7165640 1 -- 172.21.15.94:0/336693104 wait complete. 2024-03-06T07:18:34.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:18:34.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:18:34.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:18:34.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:18:34.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:18:34.018 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:18:34.018 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:18:34.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:18:34.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.185+0000 7f1207d3e640 1 Processor -- start 2024-03-06T07:18:34.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.185+0000 7f1207d3e640 1 -- start start 2024-03-06T07:18:34.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.186+0000 7f276a89a640 1 Processor -- start 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.186+0000 7f4b4d083640 1 Processor -- start 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f1207d3e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f12000acdd0 0x7f1200156910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f1207d3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f1200156e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f1207d3e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f12000c0320 0x7f120015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f1207d3e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12001598c0 con 0x7f12000c0320 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f1207d3e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1200159ba0 con 0x7f12000acdd0 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f1207d3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1200159e80 con 0x7f12000c6d30 2024-03-06T07:18:34.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f163cd81640 1 Processor -- start 2024-03-06T07:18:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f4b4d083640 1 -- start start 2024-03-06T07:18:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f12052d9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f1200156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f12052d9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f1200156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54294/0 (socket says 172.21.15.94:54294) 2024-03-06T07:18:34.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f12052d9640 1 -- 172.21.15.94:0/2304190201 learned_addr learned my addr 172.21.15.94:0/2304190201 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.186+0000 7f276a89a640 1 -- start start 2024-03-06T07:18:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f276a89a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2764156bf0 0x7f27640c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f276a89a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f276406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f276a89a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27640c6d80 0x7f276406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f276a89a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27640a9960 con 0x7f27640c6d80 2024-03-06T07:18:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f276a89a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f276415d080 con 0x7f2764156bf0 2024-03-06T07:18:34.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f276a89a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276415d360 con 0x7f276415cb80 2024-03-06T07:18:34.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f3057395640 1 Processor -- start 2024-03-06T07:18:34.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b4d083640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4b480c2220 0x7f4b480c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b4d083640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b4806c240 0x7f4b48066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b4d083640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b480c1b50 con 0x7f4b480bcd10 2024-03-06T07:18:34.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b4d083640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b481523b0 con 0x7f4b480c2220 2024-03-06T07:18:34.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b4d083640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480c1df0 con 0x7f4b4806c240 2024-03-06T07:18:34.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27640c6d80 0x7f276406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2763fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2764156bf0 0x7f27640c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27640c6d80 0x7f276406dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40842/0 (socket says 172.21.15.94:40842) 2024-03-06T07:18:34.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 -- 172.21.15.94:0/1615292556 learned_addr learned my addr 172.21.15.94:0/1615292556 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f3057395640 1 -- start start 2024-03-06T07:18:34.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b46d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f4b480bcd10 0x7f4b480c2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43796/0 (socket says 172.21.15.94:43796) 2024-03-06T07:18:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b46d9d640 1 -- 172.21.15.94:0/537276379 learned_addr learned my addr 172.21.15.94:0/537276379 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.186+0000 7f50bb2b8640 1 Processor -- start 2024-03-06T07:18:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f50bb2b8640 1 -- start start 2024-03-06T07:18:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f50bb2b8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f50b40c0320 0x7f50b415d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f50bb2b8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b41598c0 con 0x7f50b40c6d30 2024-03-06T07:18:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f50bb2b8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b4159ba0 con 0x7f50b40acdd0 2024-03-06T07:18:34.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f50bb2b8640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b415da70 con 0x7f50b40c0320 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f50b8853640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f50b40c6d30 0x7f50b415ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41442/0 (socket says 172.21.15.94:41442) 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f50b8853640 1 -- 172.21.15.94:0/2461951705 learned_addr learned my addr 172.21.15.94:0/2461951705 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b4659c640 1 --2- 172.21.15.94:0/537276379 >> v2:172.21.15.94:3300/0 conn(0x7f4b480c2220 0x7f4b480c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f4b4759e640 1 --2- 172.21.15.94:0/537276379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b4806c240 0x7f4b48066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f4b4659c640 1 -- 172.21.15.94:0/537276379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b4806c240 msgr2=0x7f4b48066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f4b4659c640 1 --2- 172.21.15.94:0/537276379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b4806c240 0x7f4b48066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f4b4659c640 1 -- 172.21.15.94:0/537276379 >> v1:172.21.15.145:6789/0 conn(0x7f4b480bcd10 legacy=0x7f4b480c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f4b4659c640 1 -- 172.21.15.94:0/537276379 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b48152690 con 0x7f4b480c2220 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f1205ada640 1 --2- 172.21.15.94:0/2304190201 >> v2:172.21.15.145:3300/0 conn(0x7f12000acdd0 0x7f1200156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f12062db640 1 --2- 172.21.15.94:0/2304190201 >> v2:172.21.15.94:3300/0 conn(0x7f12000c0320 0x7f120015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f1205ada640 1 -- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 msgr2=0x7f1200156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f1205ada640 1 --2- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f1200156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f1205ada640 1 -- 172.21.15.94:0/2304190201 >> v2:172.21.15.94:3300/0 conn(0x7f12000c0320 msgr2=0x7f120015ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f1205ada640 1 --2- 172.21.15.94:0/2304190201 >> v2:172.21.15.94:3300/0 conn(0x7f12000c0320 0x7f120015ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f1205ada640 1 -- 172.21.15.94:0/2304190201 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120015a160 con 0x7f12000acdd0 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f12052d9640 1 --2- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f1200156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-06T07:18:34.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f27637fe640 1 --2- 172.21.15.94:0/1615292556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f276406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 -- 172.21.15.94:0/1615292556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 msgr2=0x7f276406d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 --2- 172.21.15.94:0/1615292556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f276406d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 -- 172.21.15.94:0/1615292556 >> v2:172.21.15.145:3300/0 conn(0x7f2764156bf0 msgr2=0x7f27640c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 --2- 172.21.15.94:0/1615292556 >> v2:172.21.15.145:3300/0 conn(0x7f2764156bf0 0x7f27640c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f2768e37640 1 -- 172.21.15.94:0/1615292556 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276415d640 con 0x7f27640c6d80 2024-03-06T07:18:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106727926 0 0) 0x7f50b41598c0 con 0x7f50b40c6d30 2024-03-06T07:18:34.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a0003520 con 0x7f50b40c6d30 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.187+0000 7f163cd81640 1 -- start start 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f163cd81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300dbce0 0x7f16300474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f163cd81640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163000bb50 con 0x7f163000a8b0 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f163cd81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1630048060 con 0x7f16300df180 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.188+0000 7f163cd81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163000a420 con 0x7f16300dbce0 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f1635d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f16300df180 0x7f1630006ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43816/0 (socket says 172.21.15.94:43816) 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f1635d9b640 1 -- 172.21.15.94:0/1245476479 learned_addr learned my addr 172.21.15.94:0/1245476479 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:34.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f163659c640 1 --2- 172.21.15.94:0/1245476479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300dbce0 0x7f16300474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f50b9855640 1 --2- 172.21.15.94:0/2461951705 >> v2:172.21.15.145:3300/0 conn(0x7f50b40c0320 0x7f50b415d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740847543 0 0) 0x7f50a0003520 con 0x7f50b40c6d30 2024-03-06T07:18:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 >> v2:172.21.15.145:3300/0 conn(0x7f50b40c0320 msgr2=0x7f50b415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 --2- 172.21.15.94:0/2461951705 >> v2:172.21.15.145:3300/0 conn(0x7f50b40c0320 0x7f50b415d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3057395640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30480dbe80 0x7f30480df200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3057395640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f304800ea00 0x7f304800ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 >> v1:172.21.15.145:6789/0 conn(0x7f50b40acdd0 legacy=0x7f50b4156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3056393640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30480dbe80 0x7f30480df200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3056b94640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f304800ea00 0x7f304800ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3056393640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30480dbe80 0x7f30480df200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54348/0 (socket says 172.21.15.94:54348) 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3056393640 1 -- 172.21.15.94:0/1532139862 learned_addr learned my addr 172.21.15.94:0/1532139862 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30480df740 con 0x7f304804dae0 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304804a560 con 0x7f304800ea00 2024-03-06T07:18:34.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304804a840 con 0x7f30480dbe80 2024-03-06T07:18:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4159e80 con 0x7f50b40c6d30 2024-03-06T07:18:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f2768e37640 1 --2- 172.21.15.94:0/1615292556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 0x7f276406dda0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f276406bf30 tx=0x7f275c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2619dab03e61f141 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f3056393640 1 -- 172.21.15.94:0/1532139862 >> v2:172.21.15.94:3300/0 conn(0x7f304800ea00 msgr2=0x7f304800ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f3056393640 1 --2- 172.21.15.94:0/1532139862 >> v2:172.21.15.94:3300/0 conn(0x7f304800ea00 0x7f304800ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f3056393640 1 -- 172.21.15.94:0/1532139862 >> v1:172.21.15.94:6789/0 conn(0x7f304804dae0 legacy=0x7f30480071b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f3056393640 1 -- 172.21.15.94:0/1532139862 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304804ab20 con 0x7f30480dbe80 2024-03-06T07:18:34.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f50a4001670 con 0x7f50b40c6d30 2024-03-06T07:18:34.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 >> v1:172.21.15.94:6789/0 conn(0x7f50b40c6d30 legacy=0x7f50b415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f1205ada640 1 --2- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000acdd0 0x7f1200156910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f12000b0c60 tx=0x7f11fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2fd4526579c7c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f163659c640 1 -- 172.21.15.94:0/1245476479 >> v1:172.21.15.145:6789/0 conn(0x7f16300df180 legacy=0x7f1630006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f163659c640 1 -- 172.21.15.94:0/1245476479 >> v1:172.21.15.94:6789/0 conn(0x7f163000a8b0 legacy=0x7f163000ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7f163659c640 1 -- 172.21.15.94:0/1245476479 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163000a0f0 con 0x7f16300dbce0 2024-03-06T07:18:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50a0003a00 0x7f50a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50a0004800 0x7f50a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f4b4659c640 1 --2- 172.21.15.94:0/537276379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b480c2220 0x7f4b480c34f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009d90 tx=0x7f4b4002bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18770dcb7e797a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f1204ad8640 1 -- 172.21.15.94:0/2304190201 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0315a0 con 0x7f12000acdd0 2024-03-06T07:18:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f1204ad8640 1 -- 172.21.15.94:0/2304190201 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc037070 con 0x7f12000acdd0 2024-03-06T07:18:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f1204ad8640 1 -- 172.21.15.94:0/2304190201 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0315a0 con 0x7f12000acdd0 2024-03-06T07:18:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50b8853640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50a0004800 0x7f50a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a000c1b0 con 0x7f50a0004800 2024-03-06T07:18:34.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a000c360 con 0x7f50a0003a00 2024-03-06T07:18:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50b9054640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50a0003a00 0x7f50a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a000c560 con 0x7f50a0005290 2024-03-06T07:18:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f163659c640 1 --2- 172.21.15.94:0/1245476479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300dbce0 0x7f16300474a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f16380505a0 tx=0x7f1638068d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=782c8e9309d83301 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f50b9855640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f2762ffd640 1 -- 172.21.15.94:0/1615292556 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c02c9f0 con 0x7f27640c6d80 2024-03-06T07:18:34.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.190+0000 7f2762ffd640 1 -- 172.21.15.94:0/1615292556 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275c02cb90 con 0x7f27640c6d80 2024-03-06T07:18:34.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f2762ffd640 1 -- 172.21.15.94:0/1615292556 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275c0343f0 con 0x7f27640c6d80 2024-03-06T07:18:34.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7fa480bfa640 1 Processor -- start 2024-03-06T07:18:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.189+0000 7fa480bfa640 1 -- start start 2024-03-06T07:18:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa480bfa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa47404db50 0x7fa474007590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa480bfa640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47404a560 con 0x7fa4740e2730 2024-03-06T07:18:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa480bfa640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47400e200 con 0x7fa4740071b0 2024-03-06T07:18:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa480bfa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47400e530 con 0x7fa47404db50 2024-03-06T07:18:34.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f34a968f640 1 Processor -- start 2024-03-06T07:18:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f4b45d9b640 1 -- 172.21.15.94:0/537276379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40039070 con 0x7f4b480c2220 2024-03-06T07:18:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f4b45d9b640 1 -- 172.21.15.94:0/537276379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4002c980 con 0x7f4b480c2220 2024-03-06T07:18:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa47affd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa47404db50 0x7fa474007590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa47affd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa47404db50 0x7fa474007590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54376/0 (socket says 172.21.15.94:54376) 2024-03-06T07:18:34.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa47affd640 1 -- 172.21.15.94:0/2732626797 learned_addr learned my addr 172.21.15.94:0/2732626797 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f3056393640 1 --2- 172.21.15.94:0/1532139862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30480dbe80 0x7f30480df200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3040009870 tx=0x7f304002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=514e05e06d90ee64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f3055391640 1 -- 172.21.15.94:0/1532139862 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040038070 con 0x7f30480dbe80 2024-03-06T07:18:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f3055391640 1 -- 172.21.15.94:0/1532139862 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040031700 con 0x7f30480dbe80 2024-03-06T07:18:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f3055391640 1 -- 172.21.15.94:0/1532139862 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040038070 con 0x7f30480dbe80 2024-03-06T07:18:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b45d9b640 1 -- 172.21.15.94:0/537276379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4002cd50 con 0x7f4b480c2220 2024-03-06T07:18:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 -- 172.21.15.94:0/537276379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b480c2220 msgr2=0x7f4b480c34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 --2- 172.21.15.94:0/537276379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b480c2220 0x7f4b480c34f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009d90 tx=0x7f4b4002bc50 comp rx=0 tx=0).stop 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f1207d3e640 1 -- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000acdd0 msgr2=0x7f1200156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f1207d3e640 1 --2- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000acdd0 0x7f1200156910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f12000b0c60 tx=0x7f11fc02b840 comp rx=0 tx=0).stop 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f1207d3e640 1 -- 172.21.15.94:0/2304190201 shutdown_connections 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f1207d3e640 1 --2- 172.21.15.94:0/2304190201 >> v2:172.21.15.94:3300/0 conn(0x7f12000c0320 0x7f120015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f1207d3e640 1 --2- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f1200156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f1207d3e640 1 --2- 172.21.15.94:0/2304190201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000acdd0 0x7f1200156910 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f1207d3e640 1 -- 172.21.15.94:0/2304190201 >> 172.21.15.94:0/2304190201 conn(0x7f12000cae70 msgr2=0x7f12000cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f1207d3e640 1 -- 172.21.15.94:0/2304190201 shutdown_connections 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f276a89a640 1 -- 172.21.15.94:0/1615292556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 msgr2=0x7f276406dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f276a89a640 1 --2- 172.21.15.94:0/1615292556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 0x7f276406dda0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f276406bf30 tx=0x7f275c02be50 comp rx=0 tx=0).stop 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f276a89a640 1 -- 172.21.15.94:0/1615292556 shutdown_connections 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f276a89a640 1 --2- 172.21.15.94:0/1615292556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 0x7f276406dda0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f276a89a640 1 --2- 172.21.15.94:0/1615292556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f276406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f276a89a640 1 --2- 172.21.15.94:0/1615292556 >> v2:172.21.15.145:3300/0 conn(0x7f2764156bf0 0x7f27640c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f276a89a640 1 -- 172.21.15.94:0/1615292556 >> 172.21.15.94:0/1615292556 conn(0x7f276406b1a0 msgr2=0x7f276406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f50b9855640 1 -- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50a0003a00 msgr2=0x7f50a0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f50b9855640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50a0003a00 0x7f50a0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f50b9855640 1 -- 172.21.15.94:0/2461951705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50a0004800 msgr2=0x7f50a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f50b9855640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50a0004800 0x7f50a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f50b9855640 1 -- 172.21.15.94:0/2461951705 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0003520 con 0x7f50a0005290 2024-03-06T07:18:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f50b9855640 1 -- 172.21.15.94:0/2461951705 reap_dead start 2024-03-06T07:18:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f163559a640 1 -- 172.21.15.94:0/1245476479 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16380b4070 con 0x7f16300dbce0 2024-03-06T07:18:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f163559a640 1 -- 172.21.15.94:0/1245476479 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16380a7910 con 0x7f16300dbce0 2024-03-06T07:18:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 -- 172.21.15.94:0/537276379 shutdown_connections 2024-03-06T07:18:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 --2- 172.21.15.94:0/537276379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b4806c240 0x7f4b48066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 --2- 172.21.15.94:0/537276379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b480c2220 0x7f4b480c34f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 -- 172.21.15.94:0/537276379 >> 172.21.15.94:0/537276379 conn(0x7f4b480c3bb0 msgr2=0x7f4b480c3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7f34a968f640 1 -- start start 2024-03-06T07:18:34.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f34a968f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a4151ef0 0x7f34a40c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 -- 172.21.15.94:0/537276379 shutdown_connections 2024-03-06T07:18:34.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f34a968f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34a40aa680 0x7f34a40aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f50b9855640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50a000b9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f50ac007360 tx=0x7f50a0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a539393d0ad7794d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa47affd640 1 -- 172.21.15.94:0/2732626797 >> v1:172.21.15.145:6789/0 conn(0x7fa4740071b0 legacy=0x7fa47400e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f34a27fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34a40aa680 0x7f34a40aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f34a27fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34a40aa680 0x7f34a40aaae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40870/0 (socket says 172.21.15.94:40870) 2024-03-06T07:18:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f34a27fc640 1 -- 172.21.15.94:0/801824005 learned_addr learned my addr 172.21.15.94:0/801824005 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f276a89a640 1 -- 172.21.15.94:0/1615292556 shutdown_connections 2024-03-06T07:18:34.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa47affd640 1 -- 172.21.15.94:0/2732626797 >> v1:172.21.15.94:6789/0 conn(0x7fa4740e2730 legacy=0x7fa4740e9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac009be0 con 0x7f50a0005290 2024-03-06T07:18:34.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ac02cd50 con 0x7f50a0005290 2024-03-06T07:18:34.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f50abfff640 1 -- 172.21.15.94:0/2461951705 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac003bd0 con 0x7f50a0005290 2024-03-06T07:18:34.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f4b4d083640 1 -- 172.21.15.94:0/537276379 wait complete. 2024-03-06T07:18:34.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.191+0000 7fa47affd640 1 -- 172.21.15.94:0/2732626797 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47404a800 con 0x7fa47404db50 2024-03-06T07:18:34.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f34a2ffd640 1 --2- 172.21.15.94:0/801824005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a4151ef0 0x7f34a40c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7fa47affd640 1 --2- 172.21.15.94:0/2732626797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa47404db50 0x7fa474007590 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4700098a0 tx=0x7fa47002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b56d33ecd32610cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f163cd81640 1 -- 172.21.15.94:0/1245476479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300dbce0 msgr2=0x7f16300474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f163cd81640 1 --2- 172.21.15.94:0/1245476479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300dbce0 0x7f16300474a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f16380505a0 tx=0x7f1638068d50 comp rx=0 tx=0).stop 2024-03-06T07:18:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a41522d0 con 0x7f34a406d6f0 2024-03-06T07:18:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a4155860 con 0x7f34a40aa680 2024-03-06T07:18:34.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4154c50 con 0x7f34a4151ef0 2024-03-06T07:18:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7fa47a7fc640 1 -- 172.21.15.94:0/2732626797 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47002f830 con 0x7fa47404db50 2024-03-06T07:18:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7fa47a7fc640 1 -- 172.21.15.94:0/2732626797 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47002f9d0 con 0x7fa47404db50 2024-03-06T07:18:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30480dbe80 msgr2=0x7f30480df200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f3057395640 1 --2- 172.21.15.94:0/1532139862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30480dbe80 0x7f30480df200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3040009870 tx=0x7f304002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7fa47a7fc640 1 -- 172.21.15.94:0/2732626797 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47002fda0 con 0x7fa47404db50 2024-03-06T07:18:34.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 shutdown_connections 2024-03-06T07:18:34.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f1207d3e640 1 -- 172.21.15.94:0/2304190201 wait complete. 2024-03-06T07:18:34.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.192+0000 7f276a89a640 1 -- 172.21.15.94:0/1615292556 wait complete. 2024-03-06T07:18:34.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7fa480bfa640 1 -- 172.21.15.94:0/2732626797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa47404db50 msgr2=0x7fa474007590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7fa480bfa640 1 --2- 172.21.15.94:0/2732626797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa47404db50 0x7fa474007590 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa4700098a0 tx=0x7fa47002ba10 comp rx=0 tx=0).stop 2024-03-06T07:18:34.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f50bb2b8640 1 -- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 msgr2=0x7f50a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f50bb2b8640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50a000b9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f50ac007360 tx=0x7f50a0005830 comp rx=0 tx=0).stop 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7fa480bfa640 1 -- 172.21.15.94:0/2732626797 shutdown_connections 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7fa480bfa640 1 --2- 172.21.15.94:0/2732626797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa47404db50 0x7fa474007590 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7fa480bfa640 1 -- 172.21.15.94:0/2732626797 >> 172.21.15.94:0/2732626797 conn(0x7fa47404c660 msgr2=0x7fa47404ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f3057395640 1 --2- 172.21.15.94:0/1532139862 >> v2:172.21.15.94:3300/0 conn(0x7f304800ea00 0x7f304800ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f3057395640 1 --2- 172.21.15.94:0/1532139862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30480dbe80 0x7f30480df200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.193+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 >> 172.21.15.94:0/1532139862 conn(0x7f304804c660 msgr2=0x7f304804ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 shutdown_connections 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f3057395640 1 -- 172.21.15.94:0/1532139862 wait complete. 2024-03-06T07:18:34.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f1207d3e640 1 Processor -- start 2024-03-06T07:18:34.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f50bb2b8640 1 -- 172.21.15.94:0/2461951705 shutdown_connections 2024-03-06T07:18:34.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f50bb2b8640 1 --2- 172.21.15.94:0/2461951705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50a000b9c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f50bb2b8640 1 -- 172.21.15.94:0/2461951705 >> 172.21.15.94:0/2461951705 conn(0x7f50b40cae70 msgr2=0x7f50b40cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f163cd81640 1 -- 172.21.15.94:0/1245476479 shutdown_connections 2024-03-06T07:18:34.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f163cd81640 1 --2- 172.21.15.94:0/1245476479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300dbce0 0x7f16300474a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f163cd81640 1 -- 172.21.15.94:0/1245476479 >> 172.21.15.94:0/1245476479 conn(0x7f1630048250 msgr2=0x7f1630048660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f1207d3e640 1 -- start start 2024-03-06T07:18:34.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7fa480bfa640 1 -- 172.21.15.94:0/2732626797 shutdown_connections 2024-03-06T07:18:34.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7fa480bfa640 1 -- 172.21.15.94:0/2732626797 wait complete. 2024-03-06T07:18:34.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f34a27fc640 1 -- 172.21.15.94:0/801824005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a4151ef0 msgr2=0x7f34a40c1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f34a27fc640 1 --2- 172.21.15.94:0/801824005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a4151ef0 0x7f34a40c1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f34a27fc640 1 -- 172.21.15.94:0/801824005 >> v1:172.21.15.94:6789/0 conn(0x7f34a406d6f0 legacy=0x7f34a406db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f34a27fc640 1 -- 172.21.15.94:0/801824005 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4154f30 con 0x7f34a40aa680 2024-03-06T07:18:34.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f4b4d083640 1 Processor -- start 2024-03-06T07:18:34.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f50bb2b8640 1 -- 172.21.15.94:0/2461951705 shutdown_connections 2024-03-06T07:18:34.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f4b4d083640 1 -- start start 2024-03-06T07:18:34.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f276a89a640 1 Processor -- start 2024-03-06T07:18:34.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.194+0000 7f50bb2b8640 1 -- 172.21.15.94:0/2461951705 wait complete. 2024-03-06T07:18:34.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f276a89a640 1 -- start start 2024-03-06T07:18:34.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f163cd81640 1 -- 172.21.15.94:0/1245476479 shutdown_connections 2024-03-06T07:18:34.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f163cd81640 1 -- 172.21.15.94:0/1245476479 wait complete. 2024-03-06T07:18:34.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f1207d3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12000acdd0 0x7f120006c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f3057395640 1 Processor -- start 2024-03-06T07:18:34.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f3057395640 1 -- start start 2024-03-06T07:18:34.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f1207d3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000c0320 0x7f12000afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f34a27fc640 1 --2- 172.21.15.94:0/801824005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a40aa680 0x7f34a40aaae0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3494007940 tx=0x7f349402bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b20af6bf16ff2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f1207d3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f12000b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f1207d3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12000ad1b0 con 0x7f12000acdd0 2024-03-06T07:18:34.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f276a89a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 0x7f276413df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f1207d3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200159570 con 0x7f12000c0320 2024-03-06T07:18:34.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f1207d3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12000ad330 con 0x7f12000c6d30 2024-03-06T07:18:34.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f276a89a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2764156bf0 0x7f27641458c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7fa480bfa640 1 Processor -- start 2024-03-06T07:18:34.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f276a89a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f2764157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f276a89a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276406e2e0 con 0x7f27640c6d80 2024-03-06T07:18:34.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f276a89a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2764156fd0 con 0x7f2764156bf0 2024-03-06T07:18:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f12052d9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000c0320 0x7f12000afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f2768e37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f2764157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f2768e37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f2764157820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54408/0 (socket says 172.21.15.94:54408) 2024-03-06T07:18:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f2768e37640 1 -- 172.21.15.94:0/2037538477 learned_addr learned my addr 172.21.15.94:0/2037538477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f276a89a640 1 -- 172.21.15.94:0/2037538477 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27640c2220 con 0x7f276415cb80 2024-03-06T07:18:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7fa480bfa640 1 -- start start 2024-03-06T07:18:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f34a1ffb640 1 -- 172.21.15.94:0/801824005 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349402c5e0 con 0x7f34a40aa680 2024-03-06T07:18:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f34a1ffb640 1 -- 172.21.15.94:0/801824005 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349402c780 con 0x7f34a40aa680 2024-03-06T07:18:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f34a1ffb640 1 -- 172.21.15.94:0/801824005 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349402cb50 con 0x7f34a40aa680 2024-03-06T07:18:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f12052d9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000c0320 0x7f12000afbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41680/0 (socket says 172.21.15.94:41680) 2024-03-06T07:18:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f12052d9640 1 -- 172.21.15.94:0/4205690840 learned_addr learned my addr 172.21.15.94:0/4205690840 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f163cd81640 1 Processor -- start 2024-03-06T07:18:34.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f50bb2b8640 1 Processor -- start 2024-03-06T07:18:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f50bb2b8640 1 -- start start 2024-03-06T07:18:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.195+0000 7f4b4d083640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b4806c240 0x7f4b48150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b4806c240 0x7f4b48150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b4806c240 0x7f4b48150ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40886/0 (socket says 172.21.15.94:40886) 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b46d9d640 1 -- 172.21.15.94:0/1468984419 learned_addr learned my addr 172.21.15.94:0/1468984419 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b4d083640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b480bcd10 0x7f4b480c0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b4d083640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b480c2220 0x7f4b480c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b4659c640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b480bcd10 0x7f4b480c0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480673c0 con 0x7f4b4806c240 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48067240 con 0x7f4b480bcd10 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4806c650 con 0x7f4b480c2220 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.196+0000 7f12062db640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f12000b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f1205ada640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12000acdd0 0x7f120006c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f4b4759e640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b480c2220 0x7f4b480c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f12052d9640 1 -- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 msgr2=0x7f12000b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3057395640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f304800ea00 0x7f3048030e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f12052d9640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f12000b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f12052d9640 1 -- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12000acdd0 msgr2=0x7f120006c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f12052d9640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12000acdd0 0x7f120006c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f12052d9640 1 -- 172.21.15.94:0/4205690840 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120015c580 con 0x7f12000c0320 2024-03-06T07:18:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f2763fff640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 0x7f276413df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3057395640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f304804dae0 0x7f3048177b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f12062db640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12000c6d30 0x7f12000b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:18:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3056393640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f304800ea00 0x7f3048030e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f4b46d9d640 1 -- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b480c2220 msgr2=0x7f4b480c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f4b46d9d640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b480c2220 0x7f4b480c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f4b46d9d640 1 -- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b480bcd10 msgr2=0x7f4b480c0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f4b46d9d640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b480bcd10 0x7f4b480c0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f4b46d9d640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480acac0 con 0x7f4b4806c240 2024-03-06T07:18:34.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3056393640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f304800ea00 0x7f3048030e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40910/0 (socket says 172.21.15.94:40910) 2024-03-06T07:18:34.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3056393640 1 -- 172.21.15.94:0/1100452313 learned_addr learned my addr 172.21.15.94:0/1100452313 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f27637fe640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2764156bf0 0x7f27641458c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3055b92640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f304804dae0 0x7f3048177b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f12052d9640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000c0320 0x7f12000afbf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f11f800af20 tx=0x7f11f800c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3057395640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30480dbe80 0x7f3048178060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f50bb2b8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50b40c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f50bb2b8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50b40b07b0 0x7f50b41577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f50bb2b8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50b4157d10 0x7f50b420b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f50bb2b8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4159570 con 0x7f50b40b07b0 2024-03-06T07:18:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f50bb2b8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b415a440 con 0x7f50b4157d10 2024-03-06T07:18:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f50bb2b8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40ad1b0 con 0x7f50a0005290 2024-03-06T07:18:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f4b46d9d640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b4806c240 0x7f4b48150ae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c800 tx=0x7f4b3c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8013430 con 0x7f12000c0320 2024-03-06T07:18:34.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8018070 con 0x7f12000c0320 2024-03-06T07:18:34.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7fa480bfa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4740071b0 0x7fa4740c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f80137f0 con 0x7f12000c0320 2024-03-06T07:18:34.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304804ae00 con 0x7f304800ea00 2024-03-06T07:18:34.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304800e4e0 con 0x7f30480dbe80 2024-03-06T07:18:34.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f3056b94640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30480dbe80 0x7f3048178060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30480078c0 con 0x7f304804dae0 2024-03-06T07:18:34.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f1207d3e640 1 -- 172.21.15.94:0/4205690840 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120015c860 con 0x7f12000c0320 2024-03-06T07:18:34.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f1207d3e640 1 -- 172.21.15.94:0/4205690840 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12001ec5a0 con 0x7f12000c0320 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa480bfa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa47404db50 0x7fa4740353f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa47b7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4740071b0 0x7fa4740c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa47b7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4740071b0 0x7fa4740c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54444/0 (socket says 172.21.15.94:54444) 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa47affd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa47404db50 0x7fa4740353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa47affd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa47404db50 0x7fa4740353f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41730/0 (socket says 172.21.15.94:41730) 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa480bfa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4740e2730 0x7fa4740dae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa480bfa640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47404adc0 con 0x7fa4740e2730 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa480bfa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4740e2be0 con 0x7fa47404db50 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa480bfa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47404af40 con 0x7fa4740071b0 2024-03-06T07:18:34.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa47affd640 1 -- 172.21.15.94:0/207724818 learned_addr learned my addr 172.21.15.94:0/207724818 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f27637fe640 1 -- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 msgr2=0x7f2764157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f27637fe640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f2764157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f27637fe640 1 -- 172.21.15.94:0/2037538477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 msgr2=0x7f276413df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f27637fe640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27640c6d80 0x7f276413df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f27637fe640 1 -- 172.21.15.94:0/2037538477 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764150e90 con 0x7f2764156bf0 2024-03-06T07:18:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f2768e37640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f276415cb80 0x7f2764157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:34.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7fa47bfff640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4740e2730 0x7fa4740dae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c018070 con 0x7f4b4806c240 2024-03-06T07:18:34.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c0072d0 con 0x7f4b4806c240 2024-03-06T07:18:34.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c0047e0 con 0x7f4b4806c240 2024-03-06T07:18:34.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b480ac820 con 0x7f4b4806c240 2024-03-06T07:18:34.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f27637fe640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2764156bf0 0x7f27641458c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f275400d650 tx=0x7f275400db10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b480ab820 con 0x7f4b4806c240 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.197+0000 7f163cd81640 1 -- start start 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f163cd81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f163000a8b0 0x7f1630006030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f163cd81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16300dbce0 0x7f16300d4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f163cd81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300df180 0x7f16300d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f163cd81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16300dc0c0 con 0x7f16300dbce0 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f163cd81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630047d80 con 0x7f163000a8b0 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f163cd81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16300df5c0 con 0x7f16300df180 2024-03-06T07:18:34.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f1207d3e640 1 -- 172.21.15.94:0/4205690840 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f120015caa0 con 0x7f12000c0320 2024-03-06T07:18:34.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f3056393640 1 -- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f304804dae0 msgr2=0x7f3048177b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f3056393640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f304804dae0 0x7f3048177b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f3056393640 1 -- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30480dbe80 msgr2=0x7f3048178060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f3056393640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30480dbe80 0x7f3048178060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f3056393640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048174620 con 0x7f304800ea00 2024-03-06T07:18:34.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f3056393640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f304800ea00 0x7f3048030e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3040036040 tx=0x7f30400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f50b9855640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50b4157d10 0x7f50b420b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f50b8853640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50b40b07b0 0x7f50b41577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f50b9855640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50b4157d10 0x7f50b420b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41710/0 (socket says 172.21.15.94:41710) 2024-03-06T07:18:34.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f50b9855640 1 -- 172.21.15.94:0/3809476551 learned_addr learned my addr 172.21.15.94:0/3809476551 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f50b9054640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50b40c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a40aa680 msgr2=0x7f34a40aaae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.198+0000 7f34a968f640 1 --2- 172.21.15.94:0/801824005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a40aa680 0x7f34a40aaae0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3494007940 tx=0x7f349402bcc0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 shutdown_connections 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f34a968f640 1 --2- 172.21.15.94:0/801824005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a40aa680 0x7f34a40aaae0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f34a968f640 1 --2- 172.21.15.94:0/801824005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a4151ef0 0x7f34a40c1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 >> 172.21.15.94:0/801824005 conn(0x7f34a406b410 msgr2=0x7f34a406b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 shutdown_connections 2024-03-06T07:18:34.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f34a968f640 1 -- 172.21.15.94:0/801824005 wait complete. 2024-03-06T07:18:34.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b4804f330 con 0x7f4b4806c240 2024-03-06T07:18:34.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275400b6c0 con 0x7f2764156bf0 2024-03-06T07:18:34.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2754019070 con 0x7f2764156bf0 2024-03-06T07:18:34.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275400ba80 con 0x7f2764156bf0 2024-03-06T07:18:34.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f276a89a640 1 -- 172.21.15.94:0/2037538477 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2764151170 con 0x7f2764156bf0 2024-03-06T07:18:34.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f1635d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16300dbce0 0x7f16300d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f163000a8b0 0x7f1630006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f163000a8b0 0x7f1630006030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41740/0 (socket says 172.21.15.94:41740) 2024-03-06T07:18:34.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 -- 172.21.15.94:0/1910126561 learned_addr learned my addr 172.21.15.94:0/1910126561 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 -- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300df180 msgr2=0x7f16300d4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16300df180 0x7f16300d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 -- 172.21.15.94:0/1910126561 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16300dbce0 msgr2=0x7f16300d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16300dbce0 0x7f16300d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 -- 172.21.15.94:0/1910126561 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630044940 con 0x7f163000a8b0 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f163659c640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f163000a8b0 0x7f1630006030 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f163807ad80 tx=0x7f1638069930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.199+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16380bf020 con 0x7f163000a8b0 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16380b96d0 con 0x7f163000a8b0 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16380b4040 con 0x7f163000a8b0 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1630044c20 con 0x7f163000a8b0 2024-03-06T07:18:34.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f276a89a640 1 -- 172.21.15.94:0/2037538477 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2764151730 con 0x7f2764156bf0 2024-03-06T07:18:34.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f50b9054640 1 -- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50b4157d10 msgr2=0x7f50b420b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f50b9054640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50b4157d10 0x7f50b420b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f50b9054640 1 -- 172.21.15.94:0/3809476551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50b40b07b0 msgr2=0x7f50b41577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f50b9054640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50b40b07b0 0x7f50b41577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7fa47bfff640 1 -- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4740071b0 msgr2=0x7fa4740c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7fa47bfff640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4740071b0 0x7fa4740c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7fa47bfff640 1 -- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa47404db50 msgr2=0x7fa4740353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7fa47bfff640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa47404db50 0x7fa4740353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7fa47bfff640 1 -- 172.21.15.94:0/207724818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474032f30 con 0x7fa4740e2730 2024-03-06T07:18:34.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f50b9054640 1 -- 172.21.15.94:0/3809476551 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b415bf80 con 0x7f50a0005290 2024-03-06T07:18:34.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7fa47b7fe640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4740071b0 0x7fa4740c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:34.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f50b9054640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50b40c4090 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f50b000f540 tx=0x7f50b000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16300451e0 con 0x7f163000a8b0 2024-03-06T07:18:34.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.200+0000 7fa47bfff640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4740e2730 0x7fa4740dae00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa47c069cc0 tx=0x7fa47c0736a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0018030 con 0x7f50a0005290 2024-03-06T07:18:34.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b420c430 con 0x7f50a0005290 2024-03-06T07:18:34.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0003860 con 0x7f50a0005290 2024-03-06T07:18:34.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0003a50 con 0x7f50a0005290 2024-03-06T07:18:34.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2764074950 con 0x7f2764156bf0 2024-03-06T07:18:34.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0688e0 con 0x7fa4740e2730 2024-03-06T07:18:34.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c08f030 con 0x7fa4740e2730 2024-03-06T07:18:34.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0688e0 con 0x7fa4740e2730 2024-03-06T07:18:34.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa474033210 con 0x7fa4740e2730 2024-03-06T07:18:34.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa474034ed0 con 0x7fa4740e2730 2024-03-06T07:18:34.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b420c710 con 0x7f50a0005290 2024-03-06T07:18:34.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f34a968f640 1 Processor -- start 2024-03-06T07:18:34.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f34a968f640 1 -- start start 2024-03-06T07:18:34.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040038050 con 0x7f304800ea00 2024-03-06T07:18:34.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.201+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11f8005d10 con 0x7f12000c0320 2024-03-06T07:18:34.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040044070 con 0x7f304800ea00 2024-03-06T07:18:34.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f11f67fc640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11e406b500 0x7f11e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400044e0 con 0x7f304800ea00 2024-03-06T07:18:34.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1630005390 con 0x7f163000a8b0 2024-03-06T07:18:34.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16380c2070 con 0x7f163000a8b0 2024-03-06T07:18:34.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f11f808ba50 con 0x7f12000c0320 2024-03-06T07:18:34.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f11f67fc640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f11e4074bf0 0x7f11e4077030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 --> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11e4077700 con 0x7f11e4074bf0 2024-03-06T07:18:34.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f808be40 con 0x7f12000c0320 2024-03-06T07:18:34.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5094001030 con 0x7f50a0005290 2024-03-06T07:18:34.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3048174900 con 0x7f304800ea00 2024-03-06T07:18:34.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3048174ec0 con 0x7f304800ea00 2024-03-06T07:18:34.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f1626ffd640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f161807c5b0 0x7f161807ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f1205ada640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11e406b500 0x7f11e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1638126bf0 con 0x7f163000a8b0 2024-03-06T07:18:34.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f30480057e0 con 0x7f304800ea00 2024-03-06T07:18:34.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f12062db640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f11e4074bf0 0x7f11e4077030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f1626ffd640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f1618085d20 0x7f1618088160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 --> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16380a9ea0 con 0x7f1618085d20 2024-03-06T07:18:34.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f163812d050 con 0x7f163000a8b0 2024-03-06T07:18:34.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa474034d10 con 0x7fa4740e2730 2024-03-06T07:18:34.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.202+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b3c01b070 con 0x7f4b4806c240 2024-03-06T07:18:34.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f4b2f7fe640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4b3406b5a0 0x7f4b3406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4b3c089a70 con 0x7f4b4806c240 2024-03-06T07:18:34.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f4b2f7fe640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7f4b34074d20 0x7f4b34077160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f34a968f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34a406d6f0 0x7f34a40c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f34a968f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a40aa680 0x7f34a4147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b3c00ae50 con 0x7f4b34074d20 2024-03-06T07:18:34.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b3c08e2a0 con 0x7f4b4806c240 2024-03-06T07:18:34.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f34a2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34a406d6f0 0x7f34a40c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f34a2ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34a406d6f0 0x7f34a40c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41756/0 (socket says 172.21.15.94:41756) 2024-03-06T07:18:34.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f34a2ffd640 1 -- 172.21.15.94:0/2283338675 learned_addr learned my addr 172.21.15.94:0/2283338675 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f34a27fc640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a40aa680 0x7f34a4147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f34a968f640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a4151ef0 0x7f34a41447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f4b4659c640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4b3406b5a0 0x7f4b3406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f4b4759e640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7f4b34074d20 0x7f4b34077160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4154900 con 0x7f34a4151ef0 2024-03-06T07:18:34.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a41554f0 con 0x7f34a406d6f0 2024-03-06T07:18:34.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a406e290 con 0x7f34a40aa680 2024-03-06T07:18:34.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f4b4659c640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4b3406b5a0 0x7f4b3406da60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b40036000 tx=0x7f4b40030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f4b4759e640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7f4b34074d20 0x7f4b34077160 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f1205ada640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11e406b500 0x7f11e406d9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11fc036000 tx=0x7f11fc03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f12062db640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f11e4074bf0 0x7f11e4077030 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f34a37fe640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a4151ef0 0x7f34a41447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a37fe640 1 -- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a40aa680 msgr2=0x7f34a4147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a37fe640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a40aa680 0x7f34a4147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a37fe640 1 -- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34a406d6f0 msgr2=0x7f34a40c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a37fe640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34a406d6f0 0x7f34a40c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a37fe640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a41eb1d0 con 0x7f34a4151ef0 2024-03-06T07:18:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f275400bc20 con 0x7f2764156bf0 2024-03-06T07:18:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f2760ff9640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f275006b5a0 0x7f275006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f275408a9b0 con 0x7f2764156bf0 2024-03-06T07:18:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50b001b070 con 0x7f50a0005290 2024-03-06T07:18:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f50a9ffb640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f509806b5d0 0x7f509806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f50b008a3b0 con 0x7f50a0005290 2024-03-06T07:18:34.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f50b8853640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f509806b5d0 0x7f509806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.203+0000 7f1635d9b640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f161807c5b0 0x7f161807ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f1636d9d640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f1618085d20 0x7f1618088160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f1635d9b640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f161807c5b0 0x7f161807ea70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f16300dadc0 tx=0x7f1628007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.204+0000 7f1636d9d640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f1618085d20 0x7f1618088160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3040003a60 con 0x7f304800ea00 2024-03-06T07:18:34.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== osd.1 v2:172.21.15.94:6808/2571975958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4b3c00ae50 con 0x7f4b34074d20 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== osd.2 v2:172.21.15.94:6800/516902551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f11e4077700 con 0x7f11e4074bf0 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa47c094420 con 0x7fa4740e2730 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7fa467fff640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa46006b6d0 0x7fa46006db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa47c105220 con 0x7fa4740e2730 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7fa467fff640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7fa460074dc0 0x7fa460077200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 --> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa47c067b60 con 0x7fa460074dc0 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa47c10c050 con 0x7fa4740e2730 2024-03-06T07:18:34.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7fa47b7fe640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa46006b6d0 0x7fa46006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a2ffd640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34a406d6f0 0x7f34a40c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a27fc640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34a40aa680 0x7f34a4147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f34a37fe640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a4151ef0 0x7f34a41447c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c4b0 tx=0x7f349c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c017070 con 0x7f34a4151ef0 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c005480 con 0x7f34a4151ef0 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a41eb4b0 con 0x7f34a4151ef0 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f2760ff9640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f2750074d10 0x7f2750077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 --> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2754001d30 con 0x7f2750074d10 2024-03-06T07:18:34.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f275408f2a0 con 0x7f2764156bf0 2024-03-06T07:18:34.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f2768e37640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f2750074d10 0x7f2750077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f2763fff640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f275006b5a0 0x7f275006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f50a9ffb640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f5098074d40 0x7f5098077180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 --> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50b0022ec0 con 0x7f5098074d40 2024-03-06T07:18:34.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50b0091050 con 0x7f50a0005290 2024-03-06T07:18:34.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f50b8853640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f509806b5d0 0x7f509806da90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f50b4206150 tx=0x7f50a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.205+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== osd.7 v2:172.21.15.145:6824/1323110445 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f16380a9ea0 con 0x7f1618085d20 2024-03-06T07:18:34.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a41eba70 con 0x7f34a4151ef0 2024-03-06T07:18:34.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f3046ffd640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f303406b5a0 0x7f303406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f50b9855640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f5098074d40 0x7f5098077180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7fa47affd640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7fa460074dc0 0x7fa460077200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7fa47b7fe640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa46006b6d0 0x7fa46006db90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa4740dfd00 tx=0x7fa468009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f2768e37640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f2750074d10 0x7f2750077150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f30400acbf0 con 0x7f304800ea00 2024-03-06T07:18:34.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f3055b92640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f303406b5a0 0x7f303406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7f3046ffd640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7f3034074d20 0x7f3034077160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3034077830 con 0x7f3034074d20 2024-03-06T07:18:34.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30400acfe0 con 0x7f304800ea00 2024-03-06T07:18:34.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.206+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c003830 con 0x7f34a4151ef0 2024-03-06T07:18:34.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7f3056b94640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7f3034074d20 0x7f3034077160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== osd.5 v2:172.21.15.145:6808/2538796010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2754001d30 con 0x7f2750074d10 2024-03-06T07:18:34.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7f2763fff640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f275006b5a0 0x7f275006da60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2758009a20 tx=0x7f2758009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7f50b9855640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f5098074d40 0x7f5098077180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.207+0000 7fa47affd640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7fa460074dc0 0x7fa460077200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.208+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3484001030 con 0x7f34a4151ef0 2024-03-06T07:18:34.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.208+0000 7f3056b94640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7f3034074d20 0x7f3034077160 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.208+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== osd.6 v2:172.21.15.145:6816/1405026475 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f50b0022ec0 con 0x7f5098074d40 2024-03-06T07:18:34.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.209+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f349c01a070 con 0x7f34a4151ef0 2024-03-06T07:18:34.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.209+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== osd.3 v2:172.21.15.94:6824/3669301969 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa47c067b60 con 0x7fa460074dc0 2024-03-06T07:18:34.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.208+0000 7f3055b92640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f303406b5a0 0x7f303406da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f30480e0bc0 tx=0x7f304c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.209+0000 7f348b7fe640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f349006b520 0x7f349006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.209+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== osd.0 v2:172.21.15.94:6809/4051752422 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3034077830 con 0x7f3034074d20 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.209+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f349c0897f0 con 0x7f34a4151ef0 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.210+0000 7f34a2ffd640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f349006b520 0x7f349006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.210+0000 7f348b7fe640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f3490074c90 0x7f34900770d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.210+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34900777a0 con 0x7f3490074c90 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.210+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f349c090050 con 0x7f34a4151ef0 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.210+0000 7f34a27fc640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f3490074c90 0x7f34900770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.210+0000 7f34a2ffd640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f349006b520 0x7f349006d9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3498009b80 tx=0x7f3498008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.210+0000 7f34a27fc640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f3490074c90 0x7f34900770d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.211+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== osd.4 v2:172.21.15.145:6800/4137837373 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34900777a0 con 0x7f3490074c90 2024-03-06T07:18:34.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.236+0000 7f1207d3e640 1 -- 172.21.15.94:0/4205690840 --> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f120004f330 con 0x7f11e4074bf0 2024-03-06T07:18:34.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.236+0000 7f11f67fc640 1 -- 172.21.15.94:0/4205690840 <== osd.2 v2:172.21.15.94:6800/516902551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f120004f330 con 0x7f11e4074bf0 2024-03-06T07:18:34.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.237+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 --> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f163000b590 con 0x7f1618085d20 2024-03-06T07:18:34.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.237+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 --> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4b48154230 con 0x7f4b34074d20 2024-03-06T07:18:34.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.237+0000 7f4b2f7fe640 1 -- 172.21.15.94:0/1468984419 <== osd.1 v2:172.21.15.94:6808/2571975958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4b48154230 con 0x7f4b34074d20 2024-03-06T07:18:34.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.237+0000 7f1626ffd640 1 -- 172.21.15.94:0/1910126561 <== osd.7 v2:172.21.15.145:6824/1323110445 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f163000b590 con 0x7f1618085d20 2024-03-06T07:18:34.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.237+0000 7f11e3fff640 1 -- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f11e4074bf0 msgr2=0x7f11e4077030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f11e3fff640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f11e4074bf0 0x7f11e4077030 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7f4b34074d20 msgr2=0x7f4b34077160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f4b4d083640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7f4b34074d20 0x7f4b34077160 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f1618085d20 msgr2=0x7f1618088160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f1618085d20 0x7f1618088160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4b3406b5a0 msgr2=0x7f4b3406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f4b4d083640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4b3406b5a0 0x7f4b3406da60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b40036000 tx=0x7f4b40030000 comp rx=0 tx=0).stop 2024-03-06T07:18:34.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b4806c240 msgr2=0x7f4b48150ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f4b4d083640 1 --2- 172.21.15.94:0/1468984419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4b4806c240 0x7f4b48150ae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c800 tx=0x7f4b3c00ccc0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f4b4759e640 1 -- 172.21.15.94:0/1468984419 reap_dead start 2024-03-06T07:18:34.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f161807c5b0 msgr2=0x7f161807ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f161807c5b0 0x7f161807ea70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f16300dadc0 tx=0x7f1628007de0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f163000a8b0 msgr2=0x7f1630006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 --2- 172.21.15.94:0/1910126561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f163000a8b0 0x7f1630006030 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f163807ad80 tx=0x7f1638069930 comp rx=0 tx=0).stop 2024-03-06T07:18:34.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f1636d9d640 1 -- 172.21.15.94:0/1910126561 reap_dead start 2024-03-06T07:18:34.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 shutdown_connections 2024-03-06T07:18:34.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 >> 172.21.15.94:0/1910126561 conn(0x7f1630048250 msgr2=0x7f1630048630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f11e3fff640 1 -- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11e406b500 msgr2=0x7f11e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f11e3fff640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11e406b500 0x7f11e406d9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11fc036000 tx=0x7f11fc03d040 comp rx=0 tx=0).stop 2024-03-06T07:18:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f11e3fff640 1 -- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000c0320 msgr2=0x7f12000afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f11e3fff640 1 --2- 172.21.15.94:0/4205690840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12000c0320 0x7f12000afbf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f11f800af20 tx=0x7f11f800c4e0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f12062db640 1 -- 172.21.15.94:0/4205690840 reap_dead start 2024-03-06T07:18:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f11e3fff640 1 -- 172.21.15.94:0/4205690840 shutdown_connections 2024-03-06T07:18:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.238+0000 7f11e3fff640 1 -- 172.21.15.94:0/4205690840 >> 172.21.15.94:0/4205690840 conn(0x7f12000cae70 msgr2=0x7f1200073e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 shutdown_connections 2024-03-06T07:18:34.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 >> 172.21.15.94:0/1468984419 conn(0x7f4b480c3bb0 msgr2=0x7f4b4814feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 shutdown_connections 2024-03-06T07:18:34.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f4b4d083640 1 -- 172.21.15.94:0/1468984419 wait complete. 2024-03-06T07:18:34.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f11e3fff640 1 -- 172.21.15.94:0/4205690840 shutdown_connections 2024-03-06T07:18:34.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f11e3fff640 1 -- 172.21.15.94:0/4205690840 wait complete. 2024-03-06T07:18:34.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 shutdown_connections 2024-03-06T07:18:34.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.239+0000 7f163cd81640 1 -- 172.21.15.94:0/1910126561 wait complete. 2024-03-06T07:18:34.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.241+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 --> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f30481778e0 con 0x7f3034074d20 2024-03-06T07:18:34.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.241+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 --> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5094002dd0 con 0x7f5098074d40 2024-03-06T07:18:34.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.242+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 --> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa4740057e0 con 0x7fa460074dc0 2024-03-06T07:18:34.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.242+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 --> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2764146050 con 0x7f2750074d10 2024-03-06T07:18:34.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.242+0000 7f50a9ffb640 1 -- 172.21.15.94:0/3809476551 <== osd.6 v2:172.21.15.145:6816/1405026475 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5094002dd0 con 0x7f5098074d40 2024-03-06T07:18:34.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.242+0000 7f3046ffd640 1 -- 172.21.15.94:0/1100452313 <== osd.0 v2:172.21.15.94:6809/4051752422 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f30481778e0 con 0x7f3034074d20 2024-03-06T07:18:34.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.242+0000 7fa467fff640 1 -- 172.21.15.94:0/207724818 <== osd.3 v2:172.21.15.94:6824/3669301969 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa4740057e0 con 0x7fa460074dc0 2024-03-06T07:18:34.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.242+0000 7f2760ff9640 1 -- 172.21.15.94:0/2037538477 <== osd.5 v2:172.21.15.145:6808/2538796010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2764146050 con 0x7f2750074d10 2024-03-06T07:18:34.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7f3034074d20 msgr2=0x7f3034077160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7f3034074d20 0x7f3034077160 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7fa460074dc0 msgr2=0x7fa460077200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7fa480bfa640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7fa460074dc0 0x7fa460077200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 --> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3484002dd0 con 0x7f3490074c90 2024-03-06T07:18:34.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f5098074d40 msgr2=0x7f5098077180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f50bb2b8640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f5098074d40 0x7f5098077180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f2750074d10 msgr2=0x7f2750077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f274a7fc640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f2750074d10 0x7f2750077150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f303406b5a0 msgr2=0x7f303406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f303406b5a0 0x7f303406da60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f30480e0bc0 tx=0x7f304c007e30 comp rx=0 tx=0).stop 2024-03-06T07:18:34.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f348b7fe640 1 -- 172.21.15.94:0/2283338675 <== osd.4 v2:172.21.15.145:6800/4137837373 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3484002dd0 con 0x7f3490074c90 2024-03-06T07:18:34.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f304800ea00 msgr2=0x7f3048030e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 --2- 172.21.15.94:0/1100452313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f304800ea00 0x7f3048030e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3040036040 tx=0x7f30400046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3056b94640 1 -- 172.21.15.94:0/1100452313 reap_dead start 2024-03-06T07:18:34.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f509806b5d0 msgr2=0x7f509806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f50bb2b8640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f509806b5d0 0x7f509806da90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f50b4206150 tx=0x7f50a4009040 comp rx=0 tx=0).stop 2024-03-06T07:18:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 shutdown_connections 2024-03-06T07:18:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 >> 172.21.15.94:0/1100452313 conn(0x7f304804c660 msgr2=0x7f30480d75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 msgr2=0x7f50b40c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f50bb2b8640 1 --2- 172.21.15.94:0/3809476551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50a0005290 0x7f50b40c4090 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f50b000f540 tx=0x7f50b000fa00 comp rx=0 tx=0).stop 2024-03-06T07:18:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.243+0000 7f50b9855640 1 -- 172.21.15.94:0/3809476551 reap_dead start 2024-03-06T07:18:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa46006b6d0 msgr2=0x7fa46006db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f275006b5a0 msgr2=0x7f275006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa46006b6d0 0x7fa46006db90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa4740dfd00 tx=0x7fa468009040 comp rx=0 tx=0).stop 2024-03-06T07:18:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4740e2730 msgr2=0x7fa4740dae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 --2- 172.21.15.94:0/207724818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4740e2730 0x7fa4740dae00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa47c069cc0 tx=0x7fa47c0736a0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa47bfff640 1 -- 172.21.15.94:0/207724818 reap_dead start 2024-03-06T07:18:34.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f274a7fc640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f275006b5a0 0x7f275006da60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2758009a20 tx=0x7f2758009190 comp rx=0 tx=0).stop 2024-03-06T07:18:34.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 shutdown_connections 2024-03-06T07:18:34.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 shutdown_connections 2024-03-06T07:18:34.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 >> 172.21.15.94:0/207724818 conn(0x7fa47404c660 msgr2=0x7fa47404dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2764156bf0 msgr2=0x7f27641458c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f274a7fc640 1 --2- 172.21.15.94:0/2037538477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2764156bf0 0x7f27641458c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f275400d650 tx=0x7f275400db10 comp rx=0 tx=0).stop 2024-03-06T07:18:34.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f2768e37640 1 -- 172.21.15.94:0/2037538477 reap_dead start 2024-03-06T07:18:34.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 shutdown_connections 2024-03-06T07:18:34.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 >> 172.21.15.94:0/3809476551 conn(0x7f50b40cae70 msgr2=0x7f50b40bf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f3057395640 1 -- 172.21.15.94:0/1100452313 wait complete. 2024-03-06T07:18:34.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 shutdown_connections 2024-03-06T07:18:34.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f3490074c90 msgr2=0x7f34900770d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f34a968f640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f3490074c90 0x7f34900770d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 shutdown_connections 2024-03-06T07:18:34.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f50bb2b8640 1 -- 172.21.15.94:0/3809476551 wait complete. 2024-03-06T07:18:34.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7fa480bfa640 1 -- 172.21.15.94:0/207724818 wait complete. 2024-03-06T07:18:34.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 shutdown_connections 2024-03-06T07:18:34.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 >> 172.21.15.94:0/2037538477 conn(0x7f276406b1a0 msgr2=0x7f27640c2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f349006b520 msgr2=0x7f349006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f34a968f640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f349006b520 0x7f349006d9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3498009b80 tx=0x7f3498008040 comp rx=0 tx=0).stop 2024-03-06T07:18:34.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a4151ef0 msgr2=0x7f34a41447c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f34a968f640 1 --2- 172.21.15.94:0/2283338675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34a4151ef0 0x7f34a41447c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c4b0 tx=0x7f349c00c970 comp rx=0 tx=0).stop 2024-03-06T07:18:34.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.244+0000 7f34a37fe640 1 -- 172.21.15.94:0/2283338675 reap_dead start 2024-03-06T07:18:34.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.245+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 shutdown_connections 2024-03-06T07:18:34.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.245+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 >> 172.21.15.94:0/2283338675 conn(0x7f34a406b410 msgr2=0x7f34a4148f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.245+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 shutdown_connections 2024-03-06T07:18:34.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.245+0000 7f274a7fc640 1 -- 172.21.15.94:0/2037538477 wait complete. 2024-03-06T07:18:34.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.246+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 shutdown_connections 2024-03-06T07:18:34.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.246+0000 7f34a968f640 1 -- 172.21.15.94:0/2283338675 wait complete. 2024-03-06T07:18:34.306 INFO:teuthology.orchestra.run.smithi094.stdout:55834574857 2024-03-06T07:18:34.306 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:18:34.307 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:18:34.308 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.308 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:18:34.309 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:18:34.310 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:18:34.311 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:18:34.312 INFO:teuthology.orchestra.run.smithi094.stdout:55834574857 2024-03-06T07:18:34.312 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:18:34.313 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.314 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:18:34.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.512+0000 7fd63faed640 1 Processor -- start 2024-03-06T07:18:34.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.512+0000 7fd63faed640 1 -- start start 2024-03-06T07:18:34.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.513+0000 7fd63faed640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd638151cc0 0x7fd6380c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.513+0000 7fd63faed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6380c20d0 0x7fd6380c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.513+0000 7fd63faed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6380c3230 con 0x7fd6380c1a50 2024-03-06T07:18:34.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.513+0000 7fd63faed640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6380a95e0 con 0x7fd638151cc0 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.513+0000 7fd63faed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380a98c0 con 0x7fd6380c20d0 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.514+0000 7fd63d889640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd638151cc0 0x7fd6380c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.514+0000 7fd63d889640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd638151cc0 0x7fd6380c27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41762/0 (socket says 172.21.15.94:41762) 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.514+0000 7fd63d889640 1 -- 172.21.15.94:0/617997445 learned_addr learned my addr 172.21.15.94:0/617997445 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.514+0000 7fd63d088640 1 --2- 172.21.15.94:0/617997445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6380c20d0 0x7fd6380c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.514+0000 7fd63d088640 1 -- 172.21.15.94:0/617997445 >> v2:172.21.15.145:3300/0 conn(0x7fd638151cc0 msgr2=0x7fd6380c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.514+0000 7fd63d088640 1 --2- 172.21.15.94:0/617997445 >> v2:172.21.15.145:3300/0 conn(0x7fd638151cc0 0x7fd6380c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.514+0000 7fd63d088640 1 -- 172.21.15.94:0/617997445 >> v1:172.21.15.94:6789/0 conn(0x7fd6380c1a50 legacy=0x7fd63806d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.515+0000 7fd63d088640 1 -- 172.21.15.94:0/617997445 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380aad60 con 0x7fd6380c20d0 2024-03-06T07:18:34.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7fd63d088640 1 --2- 172.21.15.94:0/617997445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6380c20d0 0x7fd6380c2cf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6280097a0 tx=0x7fd62802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a46e13605bbe6f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7fd63c887640 1 -- 172.21.15.94:0/617997445 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628035020 con 0x7fd6380c20d0 2024-03-06T07:18:34.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7fd63c887640 1 -- 172.21.15.94:0/617997445 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628031810 con 0x7fd6380c20d0 2024-03-06T07:18:34.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7fd63c887640 1 -- 172.21.15.94:0/617997445 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628031be0 con 0x7fd6380c20d0 2024-03-06T07:18:34.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.515+0000 7f32b004e640 1 Processor -- start 2024-03-06T07:18:34.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.515+0000 7f32b004e640 1 -- start start 2024-03-06T07:18:34.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7efe79472640 1 Processor -- start 2024-03-06T07:18:34.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7efe79472640 1 -- start start 2024-03-06T07:18:34.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.515+0000 7fa1b3fc7640 1 Processor -- start 2024-03-06T07:18:34.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7f2704681640 1 Processor -- start 2024-03-06T07:18:34.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7f2704681640 1 -- start start 2024-03-06T07:18:34.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7f3b734a4640 1 Processor -- start 2024-03-06T07:18:34.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32b004e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f32a003e7d0 0x7f32a003ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32b004e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00dfdc0 0x7f32a00df350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32b004e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f32a00e2580 0x7f32a00e29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32b004e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a0046990 con 0x7f32a003e7d0 2024-03-06T07:18:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32b004e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a00df890 con 0x7f32a00e2580 2024-03-06T07:18:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32b004e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a0047510 con 0x7f32a00dfdc0 2024-03-06T07:18:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7fa1b3fc7640 1 -- start start 2024-03-06T07:18:34.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b3fc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac0c15f0 0x7fa1ac1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b3fc7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1ac0c2c10 0x7fa1ac159f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b3fc7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac0c39b0 con 0x7fa1ac156bf0 2024-03-06T07:18:34.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b3fc7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac0c3110 con 0x7fa1ac0c2c10 2024-03-06T07:18:34.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b3fc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0c33f0 con 0x7fa1ac0c15f0 2024-03-06T07:18:34.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b2564640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1ac0c2c10 0x7fa1ac159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fd63faed640 1 -- 172.21.15.94:0/617997445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6380c20d0 msgr2=0x7fd6380c2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fd63faed640 1 --2- 172.21.15.94:0/617997445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6380c20d0 0x7fd6380c2cf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6280097a0 tx=0x7fd62802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b2564640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1ac0c2c10 0x7fa1ac159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40970/0 (socket says 172.21.15.94:40970) 2024-03-06T07:18:34.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fa1b2564640 1 -- 172.21.15.94:0/3279469730 learned_addr learned my addr 172.21.15.94:0/3279469730 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fd63faed640 1 -- 172.21.15.94:0/617997445 shutdown_connections 2024-03-06T07:18:34.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fd63faed640 1 --2- 172.21.15.94:0/617997445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6380c20d0 0x7fd6380c2cf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fd63faed640 1 --2- 172.21.15.94:0/617997445 >> v2:172.21.15.145:3300/0 conn(0x7fd638151cc0 0x7fd6380c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fd63faed640 1 -- 172.21.15.94:0/617997445 >> 172.21.15.94:0/617997445 conn(0x7fd638067120 msgr2=0x7fd63806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7fe2cbfff640 1 Processor -- start 2024-03-06T07:18:34.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fd63faed640 1 -- 172.21.15.94:0/617997445 shutdown_connections 2024-03-06T07:18:34.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fd63faed640 1 -- 172.21.15.94:0/617997445 wait complete. 2024-03-06T07:18:34.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe79472640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe6c006b70 0x7efe6c0db790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe79472640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c0072c0 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe79472640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c030ad0 con 0x7efe6c04a980 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe79472640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6c0dc180 con 0x7efe6c006b70 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe737fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe6c006b70 0x7efe6c0db790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe737fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe6c006b70 0x7efe6c0db790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41786/0 (socket says 172.21.15.94:41786) 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe73fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7efe6c04a980 0x7efe6c0d5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43846/0 (socket says 172.21.15.94:43846) 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7efe73fff640 1 -- 172.21.15.94:0/2974853400 learned_addr learned my addr 172.21.15.94:0/2974853400 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fa1b1562640 1 --2- 172.21.15.94:0/3279469730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac0c15f0 0x7fa1ac1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fa1b2564640 1 -- 172.21.15.94:0/3279469730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac0c15f0 msgr2=0x7fa1ac1599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fa1b2564640 1 --2- 172.21.15.94:0/3279469730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac0c15f0 0x7fa1ac1599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fa1b2564640 1 -- 172.21.15.94:0/3279469730 >> v1:172.21.15.145:6789/0 conn(0x7fa1ac156bf0 legacy=0x7fa1ac0aa0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fa1b2564640 1 -- 172.21.15.94:0/3279469730 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0c36d0 con 0x7fa1ac0c2c10 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2704681640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40d3aa0 0x7f26f40d7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2704681640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f4007040 con 0x7f26f40d6d00 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f270241d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40d3aa0 0x7f26f40d7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2702c1e640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f26f40decd0 0x7f26f40e10c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43834/0 (socket says 172.21.15.94:43834) 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2702c1e640 1 -- 172.21.15.94:0/1178374461 learned_addr learned my addr 172.21.15.94:0/1178374461 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2704681640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f4007320 con 0x7f26f40decd0 2024-03-06T07:18:34.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2704681640 1 -- 172.21.15.94:0/1178374461 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4006df0 con 0x7f26f40d3aa0 2024-03-06T07:18:34.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7fe2cbfff640 1 -- start start 2024-03-06T07:18:34.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f3b734a4640 1 -- start start 2024-03-06T07:18:34.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7efe737fe640 1 -- 172.21.15.94:0/2974853400 >> v1:172.21.15.145:6789/0 conn(0x7efe6c04a980 legacy=0x7efe6c0d5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7efe737fe640 1 -- 172.21.15.94:0/2974853400 >> v1:172.21.15.94:6789/0 conn(0x7efe6c0dbcd0 legacy=0x7efe6c0e2b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:18:34.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7efe737fe640 1 -- 172.21.15.94:0/2974853400 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c030db0 con 0x7efe6c006b70 2024-03-06T07:18:34.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f270141b640 1 -- 172.21.15.94:0/1178374461 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122838879 0 0) 0x7f26f4007320 con 0x7f26f40decd0 2024-03-06T07:18:34.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f270141b640 1 -- 172.21.15.94:0/1178374461 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e8003440 con 0x7f26f40decd0 2024-03-06T07:18:34.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f270241d640 1 -- 172.21.15.94:0/1178374461 >> v1:172.21.15.145:6789/0 conn(0x7f26f40decd0 legacy=0x7f26f40e10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f270241d640 1 -- 172.21.15.94:0/1178374461 >> v1:172.21.15.94:6789/0 conn(0x7f26f40d6d00 legacy=0x7f26f4046590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f270241d640 1 -- 172.21.15.94:0/1178374461 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f40d6500 con 0x7f26f40d3aa0 2024-03-06T07:18:34.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc13b300 0x7fe2cc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7fe2cbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2cc0ac110 0x7fe2cc0ac570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32ad5e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00dfdc0 0x7f32a00df350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32ad5e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00dfdc0 0x7f32a00df350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54482/0 (socket says 172.21.15.94:54482) 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32ad5e9640 1 -- 172.21.15.94:0/130514603 learned_addr learned my addr 172.21.15.94:0/130514603 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f32ae5eb640 1 --2- 172.21.15.94:0/130514603 >> v2:172.21.15.145:3300/0 conn(0x7f32a00e2580 0x7f32a00e29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f32ad5e9640 1 -- 172.21.15.94:0/130514603 >> v2:172.21.15.145:3300/0 conn(0x7f32a00e2580 msgr2=0x7f32a00e29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f32ad5e9640 1 --2- 172.21.15.94:0/130514603 >> v2:172.21.15.145:3300/0 conn(0x7f32a00e2580 0x7f32a00e29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f32ad5e9640 1 -- 172.21.15.94:0/130514603 >> v2:172.21.15.94:3300/0 conn(0x7f32a003e7d0 msgr2=0x7f32a003ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f32ad5e9640 1 --2- 172.21.15.94:0/130514603 >> v2:172.21.15.94:3300/0 conn(0x7f32a003e7d0 0x7f32a003ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f32ad5e9640 1 -- 172.21.15.94:0/130514603 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00477f0 con 0x7f32a00dfdc0 2024-03-06T07:18:34.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f32ad5e9640 1 --2- 172.21.15.94:0/130514603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00dfdc0 0x7f32a00df350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32a0046c50 tx=0x7f329802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8826fcd34ea82a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7f2858afc640 1 Processor -- start 2024-03-06T07:18:34.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.516+0000 7f2858afc640 1 -- start start 2024-03-06T07:18:34.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7fd63faed640 1 Processor -- start 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f2858afc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f284c00b0d0 0x7f284c0069b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f2858afc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f2858afc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f284c031170 con 0x7f284c0dbfe0 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f2858afc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f284c04aa70 con 0x7f284c00b0d0 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f2858afc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284c031430 con 0x7f284c006ef0 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2852ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f284c00b0d0 0x7f284c0069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2853fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2852ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f284c00b0d0 0x7f284c0069b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41780/0 (socket says 172.21.15.94:41780) 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2852ffd640 1 -- 172.21.15.94:0/3220911466 learned_addr learned my addr 172.21.15.94:0/3220911466 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.517+0000 7f28537fe640 1 --1- 172.21.15.94:0/3220911466 >> v1:172.21.15.145:6789/0 conn(0x7f284c0dbfe0 0x7f284c03e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43826/0 (socket says 172.21.15.94:43826) 2024-03-06T07:18:34.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2853fff640 1 -- 172.21.15.94:0/3220911466 >> v2:172.21.15.145:3300/0 conn(0x7f284c00b0d0 msgr2=0x7f284c0069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2853fff640 1 --2- 172.21.15.94:0/3220911466 >> v2:172.21.15.145:3300/0 conn(0x7f284c00b0d0 0x7f284c0069b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2853fff640 1 -- 172.21.15.94:0/3220911466 >> v1:172.21.15.145:6789/0 conn(0x7f284c0dbfe0 legacy=0x7f284c03e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.518+0000 7f2853fff640 1 -- 172.21.15.94:0/3220911466 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c04b0f0 con 0x7f284c006ef0 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cbfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe2cc152390 0x7fe2cc152800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7fe2caffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc13b300 0x7fe2cc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cbfff640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc1562c0 con 0x7fe2cc0ac110 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cbfff640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc151f00 con 0x7fe2cc152390 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cbfff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc152d40 con 0x7fe2cc13b300 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7fe2ca7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2cc0ac110 0x7fe2cc0ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2ca7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2cc0ac110 0x7fe2cc0ac570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40976/0 (socket says 172.21.15.94:40976) 2024-03-06T07:18:34.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2caffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc13b300 0x7fe2cc155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54518/0 (socket says 172.21.15.94:54518) 2024-03-06T07:18:34.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fa1b2564640 1 --2- 172.21.15.94:0/3279469730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac159f30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009870 tx=0x7fa19c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec71caca93ced5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2caffd640 1 -- 172.21.15.94:0/3585033752 learned_addr learned my addr 172.21.15.94:0/3585033752 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f3b734a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b640dbe70 0x7f3b640df1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f3b734a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6400e9f0 0x7f3b6400ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f3b734a4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b640df730 con 0x7f3b6404dad0 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f3b734a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6404a550 con 0x7f3b640dbe70 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7f3b734a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6404a830 con 0x7f3b6400e9f0 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71240640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b640dbe70 0x7f3b640df1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f32acde8640 1 -- 172.21.15.94:0/130514603 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32980317c0 con 0x7f32a00dfdc0 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f32acde8640 1 -- 172.21.15.94:0/130514603 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3298037070 con 0x7f32a00dfdc0 2024-03-06T07:18:34.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f32acde8640 1 -- 172.21.15.94:0/130514603 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298031b60 con 0x7f32a00dfdc0 2024-03-06T07:18:34.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71240640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b640dbe70 0x7f3b640df1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40990/0 (socket says 172.21.15.94:40990) 2024-03-06T07:18:34.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71240640 1 -- 172.21.15.94:0/3804105872 learned_addr learned my addr 172.21.15.94:0/3804105872 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.519+0000 7fd63faed640 1 -- start start 2024-03-06T07:18:34.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/3585033752 >> v2:172.21.15.145:3300/0 conn(0x7fe2cc152390 0x7fe2cc152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71a41640 1 --2- 172.21.15.94:0/3804105872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6400e9f0 0x7f3b6400ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f2853fff640 1 --2- 172.21.15.94:0/3220911466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0e90d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28400097a0 tx=0x7f284002b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35de23303e38b813 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7efe737fe640 1 --2- 172.21.15.94:0/2974853400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c006b70 0x7efe6c0db790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe60009720 tx=0x7efe6002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c6068e1498d8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fa1b0d61640 1 -- 172.21.15.94:0/3279469730 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c038070 con 0x7fa1ac0c2c10 2024-03-06T07:18:34.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fa1b0d61640 1 -- 172.21.15.94:0/3279469730 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c031740 con 0x7fa1ac0c2c10 2024-03-06T07:18:34.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fa1b0d61640 1 -- 172.21.15.94:0/3279469730 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c02f050 con 0x7fa1ac0c2c10 2024-03-06T07:18:34.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f270241d640 1 --2- 172.21.15.94:0/1178374461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40d3aa0 0x7f26f40d7850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009750 tx=0x7f26f802b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce571c5c38c9b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f28527fc640 1 -- 172.21.15.94:0/3220911466 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840038070 con 0x7f284c006ef0 2024-03-06T07:18:34.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f28527fc640 1 -- 172.21.15.94:0/3220911466 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2840031710 con 0x7f284c006ef0 2024-03-06T07:18:34.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f28527fc640 1 -- 172.21.15.94:0/3220911466 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840031b20 con 0x7f284c006ef0 2024-03-06T07:18:34.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f270141b640 1 -- 172.21.15.94:0/1178374461 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8037070 con 0x7f26f40d3aa0 2024-03-06T07:18:34.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f270141b640 1 -- 172.21.15.94:0/1178374461 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f8030510 con 0x7f26f40d3aa0 2024-03-06T07:18:34.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f270141b640 1 -- 172.21.15.94:0/1178374461 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc074f80 con 0x7f26f40d3aa0 2024-03-06T07:18:34.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71240640 1 -- 172.21.15.94:0/3804105872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6400e9f0 msgr2=0x7f3b6400ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71240640 1 --2- 172.21.15.94:0/3804105872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6400e9f0 0x7f3b6400ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71240640 1 -- 172.21.15.94:0/3804105872 >> v1:172.21.15.145:6789/0 conn(0x7f3b6404dad0 legacy=0x7f3b640071a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7f3b71240640 1 -- 172.21.15.94:0/3804105872 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6404ab10 con 0x7f3b640dbe70 2024-03-06T07:18:34.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f32b004e640 1 -- 172.21.15.94:0/130514603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00dfdc0 msgr2=0x7f32a00df350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7efe72ffd640 1 -- 172.21.15.94:0/2974853400 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60037070 con 0x7efe6c006b70 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7efe72ffd640 1 -- 172.21.15.94:0/2974853400 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60030710 con 0x7efe6c006b70 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7efe72ffd640 1 -- 172.21.15.94:0/2974853400 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60030ae0 con 0x7efe6c006b70 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cb7fe640 1 -- 172.21.15.94:0/3585033752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc13b300 msgr2=0x7fe2cc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/3585033752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc13b300 0x7fe2cc155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cb7fe640 1 -- 172.21.15.94:0/3585033752 >> v2:172.21.15.94:3300/0 conn(0x7fe2cc0ac110 msgr2=0x7fe2cc0ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/3585033752 >> v2:172.21.15.94:3300/0 conn(0x7fe2cc0ac110 0x7fe2cc0ac570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.520+0000 7fe2cb7fe640 1 -- 172.21.15.94:0/3585033752 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc0aa9d0 con 0x7fe2cc152390 2024-03-06T07:18:34.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f32b004e640 1 --2- 172.21.15.94:0/130514603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00dfdc0 0x7f32a00df350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f32a0046c50 tx=0x7f329802b860 comp rx=0 tx=0).stop 2024-03-06T07:18:34.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f32b004e640 1 -- 172.21.15.94:0/130514603 shutdown_connections 2024-03-06T07:18:34.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f32b004e640 1 --2- 172.21.15.94:0/130514603 >> v2:172.21.15.145:3300/0 conn(0x7f32a00e2580 0x7f32a00e29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f32b004e640 1 --2- 172.21.15.94:0/130514603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00dfdc0 0x7f32a00df350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f32b004e640 1 --2- 172.21.15.94:0/130514603 >> v2:172.21.15.94:3300/0 conn(0x7f32a003e7d0 0x7f32a003ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f32b004e640 1 -- 172.21.15.94:0/130514603 >> 172.21.15.94:0/130514603 conn(0x7f32a0048280 msgr2=0x7f32a0048690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/3279469730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 msgr2=0x7fa1ac159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/3279469730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac159f30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009870 tx=0x7fa19c02b7e0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f2858afc640 1 -- 172.21.15.94:0/3220911466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 msgr2=0x7f284c0e90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.521+0000 7f2858afc640 1 --2- 172.21.15.94:0/3220911466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0e90d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28400097a0 tx=0x7f284002b7b0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2704681640 1 -- 172.21.15.94:0/1178374461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40d3aa0 msgr2=0x7f26f40d7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2704681640 1 --2- 172.21.15.94:0/1178374461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40d3aa0 0x7f26f40d7850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009750 tx=0x7f26f802b5a0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f32b004e640 1 -- 172.21.15.94:0/130514603 shutdown_connections 2024-03-06T07:18:34.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7efe79472640 1 -- 172.21.15.94:0/2974853400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c006b70 msgr2=0x7efe6c0db790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7efe79472640 1 --2- 172.21.15.94:0/2974853400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c006b70 0x7efe6c0db790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe60009720 tx=0x7efe6002b710 comp rx=0 tx=0).stop 2024-03-06T07:18:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2858afc640 1 -- 172.21.15.94:0/3220911466 shutdown_connections 2024-03-06T07:18:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2858afc640 1 --2- 172.21.15.94:0/3220911466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0e90d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2858afc640 1 --2- 172.21.15.94:0/3220911466 >> v2:172.21.15.145:3300/0 conn(0x7f284c00b0d0 0x7f284c0069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2858afc640 1 -- 172.21.15.94:0/3220911466 >> 172.21.15.94:0/3220911466 conn(0x7f284c005e90 msgr2=0x7f284c0062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/3279469730 shutdown_connections 2024-03-06T07:18:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/3279469730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac159f30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/3279469730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac0c15f0 0x7fa1ac1599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/3279469730 >> 172.21.15.94:0/3279469730 conn(0x7fa1ac06afc0 msgr2=0x7fa1ac06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f3b71240640 1 --2- 172.21.15.94:0/3804105872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640df1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009870 tx=0x7f3b6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf492e673de931d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7efe79472640 1 -- 172.21.15.94:0/2974853400 shutdown_connections 2024-03-06T07:18:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7efe79472640 1 --2- 172.21.15.94:0/2974853400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c006b70 0x7efe6c0db790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7efe79472640 1 -- 172.21.15.94:0/2974853400 >> 172.21.15.94:0/2974853400 conn(0x7efe6c005e90 msgr2=0x7efe6c0062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7efe79472640 1 -- 172.21.15.94:0/2974853400 shutdown_connections 2024-03-06T07:18:34.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2704681640 1 -- 172.21.15.94:0/1178374461 shutdown_connections 2024-03-06T07:18:34.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2704681640 1 --2- 172.21.15.94:0/1178374461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40d3aa0 0x7f26f40d7850 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2704681640 1 -- 172.21.15.94:0/1178374461 >> 172.21.15.94:0/1178374461 conn(0x7f26f40062d0 msgr2=0x7f26f40066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2858afc640 1 -- 172.21.15.94:0/3220911466 shutdown_connections 2024-03-06T07:18:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2858afc640 1 -- 172.21.15.94:0/3220911466 wait complete. 2024-03-06T07:18:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/3279469730 shutdown_connections 2024-03-06T07:18:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/3585033752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc152390 0x7fe2cc152800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc071020 tx=0x7fe2c4034630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9886f9d5a8df21dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f3b63fff640 1 -- 172.21.15.94:0/3804105872 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b680315a0 con 0x7f3b640dbe70 2024-03-06T07:18:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/3279469730 wait complete. 2024-03-06T07:18:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f3b63fff640 1 -- 172.21.15.94:0/3804105872 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b68037070 con 0x7f3b640dbe70 2024-03-06T07:18:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f3b63fff640 1 -- 172.21.15.94:0/3804105872 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b68031960 con 0x7f3b640dbe70 2024-03-06T07:18:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63faed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6380c1a50 0x7fd6381504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63faed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd638151cc0 0x7fd6380bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63faed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6380c0620 0x7fd6380bd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63faed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381554f0 con 0x7fd6380c0620 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63faed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638155210 con 0x7fd6380c1a50 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63faed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381520a0 con 0x7fd638151cc0 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63d088640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd638151cc0 0x7fd6380bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63d088640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd638151cc0 0x7fd6380bd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54536/0 (socket says 172.21.15.94:54536) 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63d088640 1 -- 172.21.15.94:0/2518475669 learned_addr learned my addr 172.21.15.94:0/2518475669 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2704681640 1 -- 172.21.15.94:0/1178374461 shutdown_connections 2024-03-06T07:18:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f2704681640 1 -- 172.21.15.94:0/1178374461 wait complete. 2024-03-06T07:18:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7f32b004e640 1 -- 172.21.15.94:0/130514603 wait complete. 2024-03-06T07:18:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63d889640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6380c1a50 0x7fd6381504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7fd63e08a640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6380c0620 0x7fd6380bd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.522+0000 7efe79472640 1 -- 172.21.15.94:0/2974853400 wait complete. 2024-03-06T07:18:34.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7f3b734a4640 1 -- 172.21.15.94:0/3804105872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 msgr2=0x7f3b640df1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7f3b734a4640 1 --2- 172.21.15.94:0/3804105872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640df1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009870 tx=0x7f3b6802b840 comp rx=0 tx=0).stop 2024-03-06T07:18:34.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fd63e08a640 1 -- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd638151cc0 msgr2=0x7fd6380bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fd63e08a640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd638151cc0 0x7fd6380bd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fd63e08a640 1 -- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6380c1a50 msgr2=0x7fd6381504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fd63e08a640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6380c1a50 0x7fd6381504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fe2c9ffb640 1 -- 172.21.15.94:0/3585033752 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4007740 con 0x7fe2cc152390 2024-03-06T07:18:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fe2c9ffb640 1 -- 172.21.15.94:0/3585033752 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c40078e0 con 0x7fe2cc152390 2024-03-06T07:18:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fd63e08a640 1 -- 172.21.15.94:0/2518475669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380bde50 con 0x7fd6380c0620 2024-03-06T07:18:34.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fe2c9ffb640 1 -- 172.21.15.94:0/3585033752 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4007cb0 con 0x7fe2cc152390 2024-03-06T07:18:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7f2858afc640 1 Processor -- start 2024-03-06T07:18:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f3b734a4640 1 -- 172.21.15.94:0/3804105872 shutdown_connections 2024-03-06T07:18:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f3b734a4640 1 --2- 172.21.15.94:0/3804105872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6400e9f0 0x7f3b6400ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f3b734a4640 1 --2- 172.21.15.94:0/3804105872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640df1f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f3b734a4640 1 -- 172.21.15.94:0/3804105872 >> 172.21.15.94:0/3804105872 conn(0x7f3b6404c650 msgr2=0x7f3b6404ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f2858afc640 1 -- start start 2024-03-06T07:18:34.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7fd63e08a640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6380c0620 0x7fd6380bd910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd63000c890 tx=0x7fd63000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7efe79472640 1 Processor -- start 2024-03-06T07:18:34.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fd62e7fc640 1 -- 172.21.15.94:0/2518475669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630016020 con 0x7fd6380c0620 2024-03-06T07:18:34.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f3b734a4640 1 -- 172.21.15.94:0/3804105872 shutdown_connections 2024-03-06T07:18:34.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fa1b3fc7640 1 Processor -- start 2024-03-06T07:18:34.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fa1b3fc7640 1 -- start start 2024-03-06T07:18:34.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f2704681640 1 Processor -- start 2024-03-06T07:18:34.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f2704681640 1 -- start start 2024-03-06T07:18:34.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fe2cbfff640 1 -- 172.21.15.94:0/3585033752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc152390 msgr2=0x7fe2cc152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/3585033752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc152390 0x7fe2cc152800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc071020 tx=0x7fe2c4034630 comp rx=0 tx=0).stop 2024-03-06T07:18:34.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fe2cbfff640 1 -- 172.21.15.94:0/3585033752 shutdown_connections 2024-03-06T07:18:34.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/3585033752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc152390 0x7fe2cc152800 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/3585033752 >> v2:172.21.15.94:3300/0 conn(0x7fe2cc0ac110 0x7fe2cc0ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/3585033752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc13b300 0x7fe2cc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fd63faed640 1 -- 172.21.15.94:0/2518475669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6381539d0 con 0x7fd6380c0620 2024-03-06T07:18:34.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7fd63faed640 1 -- 172.21.15.94:0/2518475669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6381545e0 con 0x7fd6380c0620 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7fe2cbfff640 1 -- 172.21.15.94:0/3585033752 >> 172.21.15.94:0/3585033752 conn(0x7fe2cc067330 msgr2=0x7fe2cc0a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.523+0000 7f32b004e640 1 Processor -- start 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f32b004e640 1 -- start start 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32b004e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a003e7d0 0x7f32a00dc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32b004e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32a00dfdc0 0x7f32a0044930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32b004e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00e2580 0x7f32a0044e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32b004e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00e2060 con 0x7f32a003e7d0 2024-03-06T07:18:34.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32b004e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a0047db0 con 0x7f32a00dfdc0 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32b004e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a000b9d0 con 0x7f32a00e2580 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32addea640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a003e7d0 0x7f32a00dc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ae5eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00e2580 0x7f32a0044e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ae5eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00e2580 0x7f32a0044e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54542/0 (socket says 172.21.15.94:54542) 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ae5eb640 1 -- 172.21.15.94:0/4070268639 learned_addr learned my addr 172.21.15.94:0/4070268639 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ad5e9640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32a00dfdc0 0x7f32a0044930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ae5eb640 1 -- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32a00dfdc0 msgr2=0x7f32a0044930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ae5eb640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32a00dfdc0 0x7f32a0044930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ae5eb640 1 -- 172.21.15.94:0/4070268639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a003e7d0 msgr2=0x7f32a00dc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f32ad5e9640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32a00dfdc0 0x7f32a0044930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2858afc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0df330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2858afc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f284c00b0d0 0x7f284c0dd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2858afc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f284c0dbfe0 0x7f284c0d9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2858afc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c0dc3c0 con 0x7f284c0dbfe0 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2858afc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c0096a0 con 0x7f284c00b0d0 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2852ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f284c00b0d0 0x7f284c0dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2852ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f284c00b0d0 0x7f284c0dd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41810/0 (socket says 172.21.15.94:41810) 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2852ffd640 1 -- 172.21.15.94:0/3162284763 learned_addr learned my addr 172.21.15.94:0/3162284763 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2853fff640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f284c0dbfe0 0x7f284c0d9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7fa1b3fc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ac0c15f0 0x7fa1ac1fad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2704681640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26f40d3aa0 0x7f26f416eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2704681640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26f40d6d00 0x7f26f4049fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2704681640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40decd0 0x7f26f416bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2704681640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4046e20 con 0x7f26f40d6d00 2024-03-06T07:18:34.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2704681640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4046ca0 con 0x7f26f40d3aa0 2024-03-06T07:18:34.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2704681640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4009a50 con 0x7f26f40decd0 2024-03-06T07:18:34.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7f3b734a4640 1 -- 172.21.15.94:0/3804105872 wait complete. 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.524+0000 7efe79472640 1 -- start start 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7efe79472640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe6c04a980 0x7efe6c0da800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7efe79472640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c0dbcd0 0x7efe6c043bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7efe79472640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe6c0c7070 0x7efe6c044100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7efe79472640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c009680 con 0x7efe6c0c7070 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7efe79472640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c006f90 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7efe79472640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c0315b0 con 0x7efe6c04a980 2024-03-06T07:18:34.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b3fc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac0a80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f270241d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26f40d3aa0 0x7f26f416eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c0dbcd0 0x7efe6c043bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c0dbcd0 0x7efe6c043bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41834/0 (socket says 172.21.15.94:41834) 2024-03-06T07:18:34.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 -- 172.21.15.94:0/2912471733 learned_addr learned my addr 172.21.15.94:0/2912471733 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b3fc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac070850 0x7fa1ac13ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b1562640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac0a80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f32ae5eb640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a003e7d0 0x7f32a00dc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f32ae5eb640 1 -- 172.21.15.94:0/4070268639 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a003e3f0 con 0x7f32a00e2580 2024-03-06T07:18:34.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7fd62e7fc640 1 -- 172.21.15.94:0/2518475669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630006d50 con 0x7fd6380c0620 2024-03-06T07:18:34.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7fd62e7fc640 1 -- 172.21.15.94:0/2518475669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630003b90 con 0x7fd6380c0620 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f32addea640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a003e7d0 0x7f32a00dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f32ae5eb640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00e2580 0x7f32a0044e70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f32a8050b10 tx=0x7f32a80736a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f329e7fc640 1 -- 172.21.15.94:0/4070268639 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a80686e0 con 0x7f32a00e2580 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a0042a70 con 0x7f32a00e2580 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a00dcef0 con 0x7f32a00e2580 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.525+0000 7f2858afc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284c03eeb0 con 0x7f284c006ef0 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2852ffd640 1 -- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 msgr2=0x7f284c0df330 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2852ffd640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0df330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2852ffd640 1 -- 172.21.15.94:0/3162284763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f284c0dbfe0 msgr2=0x7f284c0d9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2852ffd640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f284c0dbfe0 0x7f284c0d9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2852ffd640 1 -- 172.21.15.94:0/3162284763 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c0dd580 con 0x7f284c00b0d0 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2853fff640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f284c0dbfe0 0x7f284c0d9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2852ffd640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f284c00b0d0 0x7f284c0dd040 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f284800e840 tx=0x7f284800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f283ffff640 1 -- 172.21.15.94:0/3162284763 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848017070 con 0x7f284c00b0d0 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 -- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe6c04a980 msgr2=0x7efe6c0da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe6c04a980 0x7efe6c0da800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 -- 172.21.15.94:0/2912471733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe6c0c7070 msgr2=0x7efe6c044100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe6c0c7070 0x7efe6c044100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 -- 172.21.15.94:0/2912471733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c044640 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7efe737fe640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c0dbcd0 0x7efe6c043bc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe6002bca0 tx=0x7efe600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7efe70ff9640 1 -- 172.21.15.94:0/2912471733 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60037040 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b3fc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0aa980 con 0x7fa1ac0c2c10 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b1562640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac0a80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41044/0 (socket says 172.21.15.94:41044) 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b1562640 1 -- 172.21.15.94:0/1928965937 learned_addr learned my addr 172.21.15.94:0/1928965937 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac0aa800 con 0x7fa1ac0c15f0 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac15c9b0 con 0x7fa1ac070850 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7fa1b2564640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac070850 0x7fa1ac13ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7fa1b1d63640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ac0c15f0 0x7fa1ac1fad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fe2cbfff640 1 -- 172.21.15.94:0/3585033752 shutdown_connections 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7fe2cbfff640 1 -- 172.21.15.94:0/3585033752 wait complete. 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f3b734a4640 1 Processor -- start 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f3b734a4640 1 -- start start 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b734a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b6400e9f0 0x7f3b640490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b734a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6404dad0 0x7f3b640e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b734a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640e1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b734a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6404adf0 con 0x7f3b640dbe70 2024-03-06T07:18:34.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b734a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6400e4d0 con 0x7f3b6400e9f0 2024-03-06T07:18:34.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b734a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b640078b0 con 0x7f3b6404dad0 2024-03-06T07:18:34.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fd63faed640 1 -- 172.21.15.94:0/2518475669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd620005290 con 0x7fd6380c0620 2024-03-06T07:18:34.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f329e7fc640 1 -- 172.21.15.94:0/4070268639 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a8068880 con 0x7f32a00e2580 2024-03-06T07:18:34.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f329e7fc640 1 -- 172.21.15.94:0/4070268639 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8095420 con 0x7f32a00e2580 2024-03-06T07:18:34.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284c041d60 con 0x7f284c00b0d0 2024-03-06T07:18:34.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284c0deb50 con 0x7f284c00b0d0 2024-03-06T07:18:34.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f283ffff640 1 -- 172.21.15.94:0/3162284763 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284800c8e0 con 0x7f284c00b0d0 2024-03-06T07:18:34.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f283ffff640 1 -- 172.21.15.94:0/3162284763 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284800cc90 con 0x7f284c00b0d0 2024-03-06T07:18:34.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c0d06b0 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c0d2a60 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7efe70ff9640 1 -- 172.21.15.94:0/2912471733 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60032a40 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7efe70ff9640 1 -- 172.21.15.94:0/2912471733 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60032e10 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26f40d6d00 0x7f26f4049fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26f40d6d00 0x7f26f4049fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41018/0 (socket says 172.21.15.94:41018) 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 -- 172.21.15.94:0/1105306171 learned_addr learned my addr 172.21.15.94:0/1105306171 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 -- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40decd0 msgr2=0x7f26f416bcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40decd0 0x7f26f416bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 -- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26f40d3aa0 msgr2=0x7f26f416eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26f40d3aa0 0x7f26f416eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.526+0000 7f2701c1c640 1 -- 172.21.15.94:0/1105306171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f40cf380 con 0x7f26f40d6d00 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2701c1c640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26f40d6d00 0x7f26f4049fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00b420 tx=0x7f26ec00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f26f2ffd640 1 -- 172.21.15.94:0/1105306171 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec011830 con 0x7f26f40d6d00 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f416f210 con 0x7f26f40d6d00 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f416f7d0 con 0x7f26f40d6d00 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7f26f2ffd640 1 -- 172.21.15.94:0/1105306171 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec016070 con 0x7f26f40d6d00 2024-03-06T07:18:34.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7f26f2ffd640 1 -- 172.21.15.94:0/1105306171 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec011bf0 con 0x7f26f40d6d00 2024-03-06T07:18:34.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7fe2cbfff640 1 Processor -- start 2024-03-06T07:18:34.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7fe2cbfff640 1 -- start start 2024-03-06T07:18:34.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b71a41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640e1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7f3b71240640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b6400e9f0 0x7f3b640490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7f3b70a3f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6404dad0 0x7f3b640e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7f3b71a41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640e1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41058/0 (socket says 172.21.15.94:41058) 2024-03-06T07:18:34.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.528+0000 7f3b71a41640 1 -- 172.21.15.94:0/4266819533 learned_addr learned my addr 172.21.15.94:0/4266819533 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7f329e7fc640 1 -- 172.21.15.94:0/4070268639 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32a80955c0 con 0x7f32a00e2580 2024-03-06T07:18:34.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fd62e7fc640 1 -- 172.21.15.94:0/2518475669 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6300050d0 con 0x7fd6380c0620 2024-03-06T07:18:34.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe54005290 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc0ac110 0x7fe2cc1e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fd62e7fc640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd61c06b5a0 0x7fd61c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7efe70ff9640 1 -- 172.21.15.94:0/2912471733 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe6003a040 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f3b70a3f640 1 -- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b6400e9f0 msgr2=0x7f3b640490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f3b70a3f640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b6400e9f0 0x7f3b640490d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f3b70a3f640 1 -- 172.21.15.94:0/4266819533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 msgr2=0x7f3b640e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f3b70a3f640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640e1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f3b70a3f640 1 -- 172.21.15.94:0/4266819533 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6417d640 con 0x7f3b6404dad0 2024-03-06T07:18:34.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fd62e7fc640 1 -- 172.21.15.94:0/2518475669 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd630089b30 con 0x7fd6380c0620 2024-03-06T07:18:34.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc13b300 0x7fe2cc1dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f3b70a3f640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6404dad0 0x7f3b640e18f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00c3c0 tx=0x7f3b5c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f283ffff640 1 -- 172.21.15.94:0/3162284763 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f284801a070 con 0x7f284c00b0d0 2024-03-06T07:18:34.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7efe70ff9640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe5806b5d0 0x7efe5806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc152390 0x7fe2cc1dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fe2cbfff640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0aa660 con 0x7fe2cc152390 2024-03-06T07:18:34.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fe2cbfff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc06f190 con 0x7fe2cc13b300 2024-03-06T07:18:34.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fe2cbfff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c2140 con 0x7fe2cc0ac110 2024-03-06T07:18:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fe2caffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc0ac110 0x7fe2cc1e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7fa1b2564640 1 -- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ac0c15f0 msgr2=0x7fa1ac1fad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7fa1b2564640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ac0c15f0 0x7fa1ac1fad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7fa1b2564640 1 -- 172.21.15.94:0/1928965937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 msgr2=0x7fa1ac0a80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7fa1b2564640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac0a80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.527+0000 7fa1b2564640 1 -- 172.21.15.94:0/1928965937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac1fb7d0 con 0x7fa1ac070850 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fa1b2564640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac070850 0x7fa1ac13ef10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa19c033000 tx=0x7fa19c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fa1a27fc640 1 -- 172.21.15.94:0/1928965937 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c042070 con 0x7fa1ac070850 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fa1a27fc640 1 -- 172.21.15.94:0/1928965937 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c033a70 con 0x7fa1ac070850 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fa1a27fc640 1 -- 172.21.15.94:0/1928965937 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c038040 con 0x7fa1ac070850 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac145370 con 0x7fa1ac070850 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac145930 con 0x7fa1ac070850 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7efe70ff9640 1 -- 172.21.15.94:0/2912471733 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efe600abf30 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2caffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc0ac110 0x7fe2cc1e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:54616/0 (socket says 172.21.15.94:54616) 2024-03-06T07:18:34.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2caffd640 1 -- 172.21.15.94:0/462109007 learned_addr learned my addr 172.21.15.94:0/462109007 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:34.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7f329e7fc640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f328c06b5a0 0x7f328c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7efe73fff640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe5806b5d0 0x7efe5806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4005290 con 0x7f26f40d6d00 2024-03-06T07:18:34.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2ca7fc640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc13b300 0x7fe2cc1dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2cb7fe640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc152390 0x7fe2cc1dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7f283ffff640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f283806b5d0 0x7f283806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.529+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3290005290 con 0x7f32a00e2580 2024-03-06T07:18:34.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.530+0000 7f32addea640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f328c06b5a0 0x7f328c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7f329e7fc640 1 -- 172.21.15.94:0/4070268639 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f32a8105760 con 0x7f32a00e2580 2024-03-06T07:18:34.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7f283ffff640 1 -- 172.21.15.94:0/3162284763 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f284808a280 con 0x7f284c00b0d0 2024-03-06T07:18:34.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2834005290 con 0x7f284c00b0d0 2024-03-06T07:18:34.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fd63d889640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd61c06b5a0 0x7fd61c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2ca7fc640 1 -- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc0ac110 msgr2=0x7fe2cc1e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2ca7fc640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc0ac110 0x7fe2cc1e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2ca7fc640 1 -- 172.21.15.94:0/462109007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc152390 msgr2=0x7fe2cc1dc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2ca7fc640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc152390 0x7fe2cc1dc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7f32addea640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f328c06b5a0 0x7f328c06da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32a00dfc60 tx=0x7f32a4009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7fe2ca7fc640 1 -- 172.21.15.94:0/462109007 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc1dda20 con 0x7fe2cc13b300 2024-03-06T07:18:34.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.531+0000 7f28537fe640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f283806b5d0 0x7f283806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.532+0000 7efe73fff640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe5806b5d0 0x7efe5806da90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe68007fd0 tx=0x7efe68007980 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.532+0000 7fe2ca7fc640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc13b300 0x7fe2cc1dbc70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0008650 tx=0x7fe2c0008740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.532+0000 7fd63d889640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd61c06b5a0 0x7fd61c06da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd6340044d0 tx=0x7fd634012040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.532+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac0693c0 con 0x7fa1ac070850 2024-03-06T07:18:34.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.532+0000 7f28537fe640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f283806b5d0 0x7f283806da90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2854069e90 tx=0x7f2854068d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.532+0000 7f26f2ffd640 1 -- 172.21.15.94:0/1105306171 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26ec004020 con 0x7f26f40d6d00 2024-03-06T07:18:34.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.533+0000 7f26f2ffd640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26e006b520 0x7f26e006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.533+0000 7f270241d640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26e006b520 0x7f26e006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.533+0000 7f26f2ffd640 1 -- 172.21.15.94:0/1105306171 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f26ec08a8f0 con 0x7f26f40d6d00 2024-03-06T07:18:34.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.533+0000 7f270241d640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26e006b520 0x7f26e006d9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26f8034000 tx=0x7f26f8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.535+0000 7efe70ff9640 1 -- 172.21.15.94:0/2912471733 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe6c044640 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.535+0000 7f329e7fc640 1 -- 172.21.15.94:0/4070268639 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32a80d4310 con 0x7f32a00e2580 2024-03-06T07:18:34.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.535+0000 7f3b61ffb640 1 -- 172.21.15.94:0/4266819533 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c005220 con 0x7f3b6404dad0 2024-03-06T07:18:34.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.535+0000 7f3b61ffb640 1 -- 172.21.15.94:0/4266819533 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c0053c0 con 0x7f3b6404dad0 2024-03-06T07:18:34.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.536+0000 7fd62e7fc640 1 -- 172.21.15.94:0/2518475669 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6380c3230 con 0x7fd6380c0620 2024-03-06T07:18:34.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.535+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6417d920 con 0x7f3b6404dad0 2024-03-06T07:18:34.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.536+0000 7f3b61ffb640 1 -- 172.21.15.94:0/4266819533 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c007230 con 0x7f3b6404dad0 2024-03-06T07:18:34.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.536+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b640e10f0 con 0x7f3b6404dad0 2024-03-06T07:18:34.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.537+0000 7fa1a27fc640 1 -- 172.21.15.94:0/1928965937 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa19c045070 con 0x7fa1ac070850 2024-03-06T07:18:34.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.537+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b640057d0 con 0x7f3b6404dad0 2024-03-06T07:18:34.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.537+0000 7fa1a27fc640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa19006b5a0 0x7fa19006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.537+0000 7fa1a27fc640 1 -- 172.21.15.94:0/1928965937 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa19c0ac4d0 con 0x7fa1ac070850 2024-03-06T07:18:34.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.537+0000 7fa1b1d63640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa19006b5a0 0x7fa19006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.538+0000 7fa1b1d63640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa19006b5a0 0x7fa19006da60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8006fd0 tx=0x7fa1a8009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.540+0000 7fa1a27fc640 1 -- 172.21.15.94:0/1928965937 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa19c07aa30 con 0x7fa1ac070850 2024-03-06T07:18:34.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.541+0000 7f3b61ffb640 1 -- 172.21.15.94:0/4266819533 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b5c019020 con 0x7f3b6404dad0 2024-03-06T07:18:34.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.542+0000 7f26f2ffd640 1 -- 172.21.15.94:0/1105306171 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26ec018310 con 0x7f26f40d6d00 2024-03-06T07:18:34.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.542+0000 7f3b61ffb640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b5006b4d0 0x7f3b5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.542+0000 7f3b71240640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b5006b4d0 0x7f3b5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.543+0000 7f3b61ffb640 1 -- 172.21.15.94:0/4266819533 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3b5c089d40 con 0x7f3b6404dad0 2024-03-06T07:18:34.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.543+0000 7f3b71240640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b5006b4d0 0x7f3b5006d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3b68036000 tx=0x7f3b6803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.543+0000 7f283ffff640 1 -- 172.21.15.94:0/3162284763 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2848058820 con 0x7f284c00b0d0 2024-03-06T07:18:34.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.544+0000 7fe2b37fe640 1 -- 172.21.15.94:0/462109007 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c001b070 con 0x7fe2cc13b300 2024-03-06T07:18:34.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.545+0000 7f3b61ffb640 1 -- 172.21.15.94:0/4266819533 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b5c020070 con 0x7f3b6404dad0 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.544+0000 7fe2b37fe640 1 -- 172.21.15.94:0/462109007 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0005b90 con 0x7fe2cc13b300 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.544+0000 7fe2b37fe640 1 -- 172.21.15.94:0/462109007 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0009910 con 0x7fe2cc13b300 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.544+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc1ddc90 con 0x7fe2cc13b300 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.544+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc0c4610 con 0x7fe2cc13b300 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.547+0000 7fe2b37fe640 1 -- 172.21.15.94:0/462109007 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2c001e070 con 0x7fe2cc13b300 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.547+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc071ad0 con 0x7fe2cc13b300 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.548+0000 7fe2b37fe640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b550 0x7fe2b806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.548+0000 7fe2b37fe640 1 -- 172.21.15.94:0/462109007 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe2c00563b0 con 0x7fe2cc13b300 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.548+0000 7fe2caffd640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b550 0x7fe2b806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.549+0000 7fe2caffd640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b550 0x7fe2b806da10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0094a0 tx=0x7fe2bc009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:34.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.553+0000 7fe2b37fe640 1 -- 172.21.15.94:0/462109007 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2c00b5e10 con 0x7fe2cc13b300 2024-03-06T07:18:34.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.945+0000 7fd63faed640 1 -- 172.21.15.94:0/2518475669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd620005580 con 0x7fd6380c0620 2024-03-06T07:18:34.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.946+0000 7fd62e7fc640 1 -- 172.21.15.94:0/2518475669 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd630058010 con 0x7fd6380c0620 2024-03-06T07:18:34.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.946+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7efe54005580 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.949 INFO:teuthology.orchestra.run.smithi094.stdout:55834574857 2024-03-06T07:18:34.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.947+0000 7efe70ff9640 1 -- 172.21.15.94:0/2912471733 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efe60042020 con 0x7efe6c0dbcd0 2024-03-06T07:18:34.950 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.949+0000 7fd61bfff640 1 -- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd61c06b5a0 msgr2=0x7fd61c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.949+0000 7fd61bfff640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd61c06b5a0 0x7fd61c06da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd6340044d0 tx=0x7fd634012040 comp rx=0 tx=0).stop 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.949+0000 7fd61bfff640 1 -- 172.21.15.94:0/2518475669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6380c0620 msgr2=0x7fd6380bd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.949+0000 7fd61bfff640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6380c0620 0x7fd6380bd910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd63000c890 tx=0x7fd63000cd50 comp rx=0 tx=0).stop 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.949+0000 7fd61bfff640 1 -- 172.21.15.94:0/2518475669 shutdown_connections 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7fd61bfff640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd61c06b5a0 0x7fd61c06da60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7fd61bfff640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6380c0620 0x7fd6380bd910 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7fd61bfff640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd638151cc0 0x7fd6380bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7fd61bfff640 1 --2- 172.21.15.94:0/2518475669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6380c1a50 0x7fd6381504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7fd61bfff640 1 -- 172.21.15.94:0/2518475669 >> 172.21.15.94:0/2518475669 conn(0x7fd638067120 msgr2=0x7fd6380c3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7fd61bfff640 1 -- 172.21.15.94:0/2518475669 shutdown_connections 2024-03-06T07:18:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7fd61bfff640 1 -- 172.21.15.94:0/2518475669 wait complete. 2024-03-06T07:18:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe5806b5d0 msgr2=0x7efe5806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe5806b5d0 0x7efe5806da90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe68007fd0 tx=0x7efe68007980 comp rx=0 tx=0).stop 2024-03-06T07:18:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c0dbcd0 msgr2=0x7efe6c043bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c0dbcd0 0x7efe6c043bc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe6002bca0 tx=0x7efe600046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 shutdown_connections 2024-03-06T07:18:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3290005580 con 0x7f32a00e2580 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe5806b5d0 0x7efe5806da90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe6c0c7070 0x7efe6c044100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe6c0dbcd0 0x7efe6c043bc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 --2- 172.21.15.94:0/2912471733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe6c04a980 0x7efe6c0da800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.950+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 >> 172.21.15.94:0/2912471733 conn(0x7efe6c005e90 msgr2=0x7efe6c0d5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.951+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 shutdown_connections 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.951+0000 7efe79472640 1 -- 172.21.15.94:0/2912471733 wait complete. 2024-03-06T07:18:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.951+0000 7f329e7fc640 1 -- 172.21.15.94:0/4070268639 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f32a80d3c10 con 0x7f32a00e2580 2024-03-06T07:18:34.957 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.957 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.951+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa1ac0c1340 con 0x7fa1ac070850 2024-03-06T07:18:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.952+0000 7fa1a27fc640 1 -- 172.21.15.94:0/1928965937 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa19c07ead0 con 0x7fa1ac070850 2024-03-06T07:18:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.954+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f328c06b5a0 msgr2=0x7f328c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.954+0000 7f32b004e640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f328c06b5a0 0x7f328c06da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32a00dfc60 tx=0x7f32a4009110 comp rx=0 tx=0).stop 2024-03-06T07:18:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00e2580 msgr2=0x7f32a0044e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00e2580 0x7f32a0044e70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f32a8050b10 tx=0x7f32a80736a0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.954+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2834005580 con 0x7f284c00b0d0 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 shutdown_connections 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f328c06b5a0 0x7f328c06da60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32a00e2580 0x7f32a0044e70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32a00dfdc0 0x7f32a0044930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 --2- 172.21.15.94:0/4070268639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32a003e7d0 0x7f32a00dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 >> 172.21.15.94:0/4070268639 conn(0x7f32a0048280 msgr2=0x7f32a00068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 shutdown_connections 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f32b004e640 1 -- 172.21.15.94:0/4070268639 wait complete. 2024-03-06T07:18:34.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.955+0000 7f283ffff640 1 -- 172.21.15.94:0/3162284763 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f284805c8c0 con 0x7f284c00b0d0 2024-03-06T07:18:34.960 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa19006b5a0 msgr2=0x7fa19006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa19006b5a0 0x7fa19006da60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8006fd0 tx=0x7fa1a8009060 comp rx=0 tx=0).stop 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac070850 msgr2=0x7fa1ac13ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac070850 0x7fa1ac13ef10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa19c033000 tx=0x7fa19c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 shutdown_connections 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa19006b5a0 0x7fa19006da60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ac070850 0x7fa1ac13ef10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ac0c2c10 0x7fa1ac0a80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.956+0000 7fa1b3fc7640 1 --2- 172.21.15.94:0/1928965937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ac0c15f0 0x7fa1ac1fad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.957+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 >> 172.21.15.94:0/1928965937 conn(0x7fa1ac06afc0 msgr2=0x7fa1ac1643d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.957+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 shutdown_connections 2024-03-06T07:18:34.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.957+0000 7fa1b3fc7640 1 -- 172.21.15.94:0/1928965937 wait complete. 2024-03-06T07:18:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f283806b5d0 msgr2=0x7f283806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f283806b5d0 0x7f283806da90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2854069e90 tx=0x7f2854068d50 comp rx=0 tx=0).stop 2024-03-06T07:18:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f284c00b0d0 msgr2=0x7f284c0dd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f284c00b0d0 0x7f284c0dd040 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f284800e840 tx=0x7f284800ed00 comp rx=0 tx=0).stop 2024-03-06T07:18:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 shutdown_connections 2024-03-06T07:18:34.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f283806b5d0 0x7f283806da90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f284c0dbfe0 0x7f284c0d9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f284c00b0d0 0x7f284c0dd040 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 --2- 172.21.15.94:0/3162284763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f284c006ef0 0x7f284c0df330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 >> 172.21.15.94:0/3162284763 conn(0x7f284c005e90 msgr2=0x7f284c030770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.959+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3b6400fb90 con 0x7f3b6404dad0 2024-03-06T07:18:34.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.960+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f26e4005850 con 0x7f26f40d6d00 2024-03-06T07:18:34.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.960+0000 7f3b61ffb640 1 -- 172.21.15.94:0/4266819533 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b5c058320 con 0x7f3b6404dad0 2024-03-06T07:18:34.965 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.960+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 shutdown_connections 2024-03-06T07:18:34.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.960+0000 7f2858afc640 1 -- 172.21.15.94:0/3162284763 wait complete. 2024-03-06T07:18:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.960+0000 7f26f2ffd640 1 -- 172.21.15.94:0/1105306171 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f26ec0594a0 con 0x7f26f40d6d00 2024-03-06T07:18:34.966 INFO:teuthology.orchestra.run.smithi094.stdout:55834574856 2024-03-06T07:18:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b5006b4d0 msgr2=0x7f3b5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26e006b520 msgr2=0x7f26e006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26e006b520 0x7f26e006d9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26f8034000 tx=0x7f26f8008030 comp rx=0 tx=0).stop 2024-03-06T07:18:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26f40d6d00 msgr2=0x7f26f4049fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26f40d6d00 0x7f26f4049fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00b420 tx=0x7f26ec00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b5006b4d0 0x7f3b5006d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3b68036000 tx=0x7f3b6803d040 comp rx=0 tx=0).stop 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6404dad0 msgr2=0x7f3b640e18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6404dad0 0x7f3b640e18f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00c3c0 tx=0x7f3b5c00c880 comp rx=0 tx=0).stop 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 shutdown_connections 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b5006b4d0 0x7f3b5006d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b640dbe70 0x7f3b640e1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b6404dad0 0x7f3b640e18f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 --2- 172.21.15.94:0/4266819533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b6400e9f0 0x7f3b640490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 >> 172.21.15.94:0/4266819533 conn(0x7f3b6404c650 msgr2=0x7f3b64030de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 shutdown_connections 2024-03-06T07:18:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26e006b520 0x7f26e006d9e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26f40decd0 0x7f26f416bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26f40d6d00 0x7f26f4049fb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 --2- 172.21.15.94:0/1105306171 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26f40d3aa0 0x7f26f416eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 >> 172.21.15.94:0/1105306171 conn(0x7f26f40062d0 msgr2=0x7f26f4046490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 shutdown_connections 2024-03-06T07:18:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.964+0000 7f3b734a4640 1 -- 172.21.15.94:0/4266819533 wait complete. 2024-03-06T07:18:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.965+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 shutdown_connections 2024-03-06T07:18:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.965+0000 7f2704681640 1 -- 172.21.15.94:0/1105306171 wait complete. 2024-03-06T07:18:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.968+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe2cc0677c0 con 0x7fe2cc13b300 2024-03-06T07:18:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.971+0000 7fe2b37fe640 1 -- 172.21.15.94:0/462109007 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2cc0677c0 con 0x7fe2cc13b300 2024-03-06T07:18:34.973 INFO:teuthology.orchestra.run.smithi094.stdout:55834574857 2024-03-06T07:18:34.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-03-06T07:18:34.975 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.976 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-03-06T07:18:34.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.973+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b550 msgr2=0x7fe2b806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.973+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b550 0x7fe2b806da10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0094a0 tx=0x7fe2bc009240 comp rx=0 tx=0).stop 2024-03-06T07:18:34.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.973+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc13b300 msgr2=0x7fe2cc1dbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:34.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.973+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc13b300 0x7fe2cc1dbc70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0008650 tx=0x7fe2c0008740 comp rx=0 tx=0).stop 2024-03-06T07:18:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 shutdown_connections 2024-03-06T07:18:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b550 0x7fe2b806da10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2cc152390 0x7fe2cc1dc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2cc13b300 0x7fe2cc1dbc70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 --2- 172.21.15.94:0/462109007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2cc0ac110 0x7fe2cc1e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 >> 172.21.15.94:0/462109007 conn(0x7fe2cc067330 msgr2=0x7fe2cc0ac570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:34.977 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 shutdown_connections 2024-03-06T07:18:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:34.974+0000 7fe2cbfff640 1 -- 172.21.15.94:0/462109007 wait complete. 2024-03-06T07:18:34.980 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-03-06T07:18:34.980 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.983 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-03-06T07:18:34.983 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-03-06T07:18:34.984 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.987 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-03-06T07:18:34.987 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-03-06T07:18:34.989 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-03-06T07:18:34.991 DEBUG:teuthology.parallel:result is None 2024-03-06T07:18:34.991 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:18:34.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:18:35.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.127+0000 7f5ac3e39640 1 Processor -- start 2024-03-06T07:18:35.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.127+0000 7f5ac3e39640 1 -- start start 2024-03-06T07:18:35.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac3e39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac3e39640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5abc0c5860 0x7f5abc06f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac3e39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5abc1586f0 0x7f5abc158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac3e39640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5abc06df40 con 0x7f5abc0c5860 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac3e39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5abc0c5cc0 con 0x7f5abc1586f0 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac3e39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc1591c0 con 0x7f5abc0c5480 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac13d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5abc0c5860 0x7f5abc06f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac13d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5abc0c5860 0x7f5abc06f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41078/0 (socket says 172.21.15.94:41078) 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac23d6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5abc1586f0 0x7f5abc158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac1bd5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.128+0000 7f5ac13d4640 1 -- 172.21.15.94:0/4024721455 learned_addr learned my addr 172.21.15.94:0/4024721455 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.129+0000 7f5ac13d4640 1 -- 172.21.15.94:0/4024721455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 msgr2=0x7f5abc155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:35.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.129+0000 7f5ac13d4640 1 --2- 172.21.15.94:0/4024721455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.129+0000 7f5ac13d4640 1 -- 172.21.15.94:0/4024721455 >> v2:172.21.15.145:3300/0 conn(0x7f5abc1586f0 msgr2=0x7f5abc158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:35.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.129+0000 7f5ac13d4640 1 --2- 172.21.15.94:0/4024721455 >> v2:172.21.15.145:3300/0 conn(0x7f5abc1586f0 0x7f5abc158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.129+0000 7f5ac13d4640 1 -- 172.21.15.94:0/4024721455 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc1594a0 con 0x7f5abc0c5860 2024-03-06T07:18:35.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.129+0000 7f5ac13d4640 1 --2- 172.21.15.94:0/4024721455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc0c5860 0x7f5abc06f6a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5abc071060 tx=0x7f5aac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f4cb78c6e3d04e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac0bd3640 1 -- 172.21.15.94:0/4024721455 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac02e070 con 0x7f5abc0c5860 2024-03-06T07:18:35.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac0bd3640 1 -- 172.21.15.94:0/4024721455 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac037070 con 0x7f5abc0c5860 2024-03-06T07:18:35.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac0bd3640 1 -- 172.21.15.94:0/4024721455 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac031a00 con 0x7f5abc0c5860 2024-03-06T07:18:35.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 -- 172.21.15.94:0/4024721455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc0c5860 msgr2=0x7f5abc06f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/4024721455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc0c5860 0x7f5abc06f6a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5abc071060 tx=0x7f5aac02b880 comp rx=0 tx=0).stop 2024-03-06T07:18:35.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 -- 172.21.15.94:0/4024721455 shutdown_connections 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/4024721455 >> v2:172.21.15.145:3300/0 conn(0x7f5abc1586f0 0x7f5abc158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/4024721455 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc0c5860 0x7f5abc06f6a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/4024721455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 -- 172.21.15.94:0/4024721455 >> 172.21.15.94:0/4024721455 conn(0x7f5abc06b1a0 msgr2=0x7f5abc06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 -- 172.21.15.94:0/4024721455 shutdown_connections 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.130+0000 7f5ac3e39640 1 -- 172.21.15.94:0/4024721455 wait complete. 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.131+0000 7f5ac3e39640 1 Processor -- start 2024-03-06T07:18:35.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.131+0000 7f5ac3e39640 1 -- start start 2024-03-06T07:18:35.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac3e39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc0c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac3e39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5abc0c5860 0x7f5abc1eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac3e39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc1586f0 0x7f5abc1eb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac3e39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc152a20 con 0x7f5abc1586f0 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac3e39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc155680 con 0x7f5abc0c5860 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac3e39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc155050 con 0x7f5abc0c5480 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac23d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc1586f0 0x7f5abc1eb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac23d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc1586f0 0x7f5abc1eb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41092/0 (socket says 172.21.15.94:41092) 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac23d6640 1 -- 172.21.15.94:0/1467319878 learned_addr learned my addr 172.21.15.94:0/1467319878 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac13d4640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5abc0c5860 0x7f5abc1eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac1bd5640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc0c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac23d6640 1 -- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 msgr2=0x7f5abc0c47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.132+0000 7f5ac23d6640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc0c47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ac23d6640 1 -- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5abc0c5860 msgr2=0x7f5abc1eb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ac23d6640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5abc0c5860 0x7f5abc1eb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ac23d6640 1 -- 172.21.15.94:0/1467319878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc1576b0 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ac23d6640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc1586f0 0x7f5abc1eb9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400e7f0 tx=0x7f5ab400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ab27fc640 1 -- 172.21.15.94:0/1467319878 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab4017070 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ab27fc640 1 -- 172.21.15.94:0/1467319878 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab400c730 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5abc1dfa70 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ab27fc640 1 -- 172.21.15.94:0/1467319878 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab400cb20 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.133+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5abc1e0030 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.134+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5abc04f330 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.134+0000 7f5ab27fc640 1 -- 172.21.15.94:0/1467319878 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ab401a070 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.135+0000 7f5ab27fc640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5aa006b4d0 0x7f5aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.135+0000 7f5ab27fc640 1 -- 172.21.15.94:0/1467319878 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5ab408a5d0 con 0x7f5abc1586f0 2024-03-06T07:18:35.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.135+0000 7f5ac1bd5640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5aa006b4d0 0x7f5aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.135+0000 7f5ac1bd5640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5aa006b4d0 0x7f5aa006d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80113e0 tx=0x7f5ab800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.140+0000 7f5ab27fc640 1 -- 172.21.15.94:0/1467319878 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ab4058a40 con 0x7f5abc1586f0 2024-03-06T07:18:35.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.307+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5abc1eaff0 con 0x7f5aa006b4d0 2024-03-06T07:18:35.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.315+0000 7f5ab27fc640 1 -- 172.21.15.94:0/1467319878 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332610 (secure 0 0 0) 0x7f5ab8003990 con 0x7f5aa006b4d0 2024-03-06T07:18:35.321 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:35.323 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:18:35.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5aa006b4d0 msgr2=0x7f5aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5aa006b4d0 0x7f5aa006d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80113e0 tx=0x7f5ab800b040 comp rx=0 tx=0).stop 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc1586f0 msgr2=0x7f5abc1eb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc1586f0 0x7f5abc1eb9f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ab400e7f0 tx=0x7f5ab400ecb0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 shutdown_connections 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5aa006b4d0 0x7f5aa006d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5abc1586f0 0x7f5abc1eb9f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5abc0c5860 0x7f5abc1eb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 --2- 172.21.15.94:0/1467319878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5abc0c5480 0x7f5abc0c47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 >> 172.21.15.94:0/1467319878 conn(0x7f5abc06b1a0 msgr2=0x7f5abc06f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 shutdown_connections 2024-03-06T07:18:35.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.319+0000 7f5ac3e39640 1 -- 172.21.15.94:0/1467319878 wait complete. 2024-03-06T07:18:35.332 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-06T07:18:34.640338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226688,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754748032,"statfs":{"total":773094113280,"available":772861984768,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002489"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719950+0000","last_change":"2024-03-06T07:18:08.523161+0000","last_active":"2024-03-06T07:18:14.719950+0000","last_peered":"2024-03-06T07:18:14.719950+0000","last_clean":"2024-03-06T07:18:14.719950+0000","last_became_active":"2024-03-06T07:18:08.522883+0000","last_became_peered":"2024-03-06T07:18:08.522883+0000","last_unstale":"2024-03-06T07:18:14.719950+0000","last_undegraded":"2024-03-06T07:18:14.719950+0000","last_fullsized":"2024-03-06T07:18:14.719950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720669+0000","last_change":"2024-03-06T07:18:09.526639+0000","last_active":"2024-03-06T07:18:14.720669+0000","last_peered":"2024-03-06T07:18:14.720669+0000","last_clean":"2024-03-06T07:18:14.720669+0000","last_became_active":"2024-03-06T07:18:09.526380+0000","last_became_peered":"2024-03-06T07:18:09.526380+0000","last_unstale":"2024-03-06T07:18:14.720669+0000","last_undegraded":"2024-03-06T07:18:14.720669+0000","last_fullsized":"2024-03-06T07:18:14.720669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720585+0000","last_change":"2024-03-06T07:18:13.788926+0000","last_active":"2024-03-06T07:18:14.720585+0000","last_peered":"2024-03-06T07:18:14.720585+0000","last_clean":"2024-03-06T07:18:14.720585+0000","last_became_active":"2024-03-06T07:18:13.788572+0000","last_became_peered":"2024-03-06T07:18:13.788572+0000","last_unstale":"2024-03-06T07:18:14.720585+0000","last_undegraded":"2024-03-06T07:18:14.720585+0000","last_fullsized":"2024-03-06T07:18:14.720585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:00.875495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720880+0000","last_change":"2024-03-06T07:18:08.517732+0000","last_active":"2024-03-06T07:18:14.720880+0000","last_peered":"2024-03-06T07:18:14.720880+0000","last_clean":"2024-03-06T07:18:14.720880+0000","last_became_active":"2024-03-06T07:18:08.517469+0000","last_became_peered":"2024-03-06T07:18:08.517469+0000","last_unstale":"2024-03-06T07:18:14.720880+0000","last_undegraded":"2024-03-06T07:18:14.720880+0000","last_fullsized":"2024-03-06T07:18:14.720880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719676+0000","last_change":"2024-03-06T07:18:09.531502+0000","last_active":"2024-03-06T07:18:14.719676+0000","last_peered":"2024-03-06T07:18:14.719676+0000","last_clean":"2024-03-06T07:18:14.719676+0000","last_became_active":"2024-03-06T07:18:09.531231+0000","last_became_peered":"2024-03-06T07:18:09.531231+0000","last_unstale":"2024-03-06T07:18:14.719676+0000","last_undegraded":"2024-03-06T07:18:14.719676+0000","last_fullsized":"2024-03-06T07:18:14.719676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551608+0000","last_change":"2024-03-06T07:18:13.565825+0000","last_active":"2024-03-06T07:18:14.551608+0000","last_peered":"2024-03-06T07:18:14.551608+0000","last_clean":"2024-03-06T07:18:14.551608+0000","last_became_active":"2024-03-06T07:18:13.565250+0000","last_became_peered":"2024-03-06T07:18:13.565250+0000","last_unstale":"2024-03-06T07:18:14.551608+0000","last_undegraded":"2024-03-06T07:18:14.551608+0000","last_fullsized":"2024-03-06T07:18:14.551608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:15.265063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718863+0000","last_change":"2024-03-06T07:18:08.523848+0000","last_active":"2024-03-06T07:18:14.718863+0000","last_peered":"2024-03-06T07:18:14.718863+0000","last_clean":"2024-03-06T07:18:14.718863+0000","last_became_active":"2024-03-06T07:18:08.523455+0000","last_became_peered":"2024-03-06T07:18:08.523455+0000","last_unstale":"2024-03-06T07:18:14.718863+0000","last_undegraded":"2024-03-06T07:18:14.718863+0000","last_fullsized":"2024-03-06T07:18:14.718863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718515+0000","last_change":"2024-03-06T07:18:09.531072+0000","last_active":"2024-03-06T07:18:14.718515+0000","last_peered":"2024-03-06T07:18:14.718515+0000","last_clean":"2024-03-06T07:18:14.718515+0000","last_became_active":"2024-03-06T07:18:09.530815+0000","last_became_peered":"2024-03-06T07:18:09.530815+0000","last_unstale":"2024-03-06T07:18:14.718515+0000","last_undegraded":"2024-03-06T07:18:14.718515+0000","last_fullsized":"2024-03-06T07:18:14.718515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551688+0000","last_change":"2024-03-06T07:18:13.563305+0000","last_active":"2024-03-06T07:18:14.551688+0000","last_peered":"2024-03-06T07:18:14.551688+0000","last_clean":"2024-03-06T07:18:14.551688+0000","last_became_active":"2024-03-06T07:18:13.562936+0000","last_became_peered":"2024-03-06T07:18:13.562936+0000","last_unstale":"2024-03-06T07:18:14.551688+0000","last_undegraded":"2024-03-06T07:18:14.551688+0000","last_fullsized":"2024-03-06T07:18:14.551688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:02.828593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718486+0000","last_change":"2024-03-06T07:18:08.520455+0000","last_active":"2024-03-06T07:18:14.718486+0000","last_peered":"2024-03-06T07:18:14.718486+0000","last_clean":"2024-03-06T07:18:14.718486+0000","last_became_active":"2024-03-06T07:18:08.520170+0000","last_became_peered":"2024-03-06T07:18:08.520170+0000","last_unstale":"2024-03-06T07:18:14.718486+0000","last_undegraded":"2024-03-06T07:18:14.718486+0000","last_fullsized":"2024-03-06T07:18:14.718486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719134+0000","last_change":"2024-03-06T07:18:09.536183+0000","last_active":"2024-03-06T07:18:14.719134+0000","last_peered":"2024-03-06T07:18:14.719134+0000","last_clean":"2024-03-06T07:18:14.719134+0000","last_became_active":"2024-03-06T07:18:09.535516+0000","last_became_peered":"2024-03-06T07:18:09.535516+0000","last_unstale":"2024-03-06T07:18:14.719134+0000","last_undegraded":"2024-03-06T07:18:14.719134+0000","last_fullsized":"2024-03-06T07:18:14.719134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551978+0000","last_change":"2024-03-06T07:18:13.564931+0000","last_active":"2024-03-06T07:18:14.551978+0000","last_peered":"2024-03-06T07:18:14.551978+0000","last_clean":"2024-03-06T07:18:14.551978+0000","last_became_active":"2024-03-06T07:18:13.564588+0000","last_became_peered":"2024-03-06T07:18:13.564588+0000","last_unstale":"2024-03-06T07:18:14.551978+0000","last_undegraded":"2024-03-06T07:18:14.551978+0000","last_fullsized":"2024-03-06T07:18:14.551978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:41.700550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550818+0000","last_change":"2024-03-06T07:18:09.516785+0000","last_active":"2024-03-06T07:18:14.550818+0000","last_peered":"2024-03-06T07:18:14.550818+0000","last_clean":"2024-03-06T07:18:14.550818+0000","last_became_active":"2024-03-06T07:18:09.516494+0000","last_became_peered":"2024-03-06T07:18:09.516494+0000","last_unstale":"2024-03-06T07:18:14.550818+0000","last_undegraded":"2024-03-06T07:18:14.550818+0000","last_fullsized":"2024-03-06T07:18:14.550818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719130+0000","last_change":"2024-03-06T07:18:08.523549+0000","last_active":"2024-03-06T07:18:14.719130+0000","last_peered":"2024-03-06T07:18:14.719130+0000","last_clean":"2024-03-06T07:18:14.719130+0000","last_became_active":"2024-03-06T07:18:08.523264+0000","last_became_peered":"2024-03-06T07:18:08.523264+0000","last_unstale":"2024-03-06T07:18:14.719130+0000","last_undegraded":"2024-03-06T07:18:14.719130+0000","last_fullsized":"2024-03-06T07:18:14.719130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:02.351247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552173+0000","last_change":"2024-03-06T07:18:13.563114+0000","last_active":"2024-03-06T07:18:14.552173+0000","last_peered":"2024-03-06T07:18:14.552173+0000","last_clean":"2024-03-06T07:18:14.552173+0000","last_became_active":"2024-03-06T07:18:13.562736+0000","last_became_peered":"2024-03-06T07:18:13.562736+0000","last_unstale":"2024-03-06T07:18:14.552173+0000","last_undegraded":"2024-03-06T07:18:14.552173+0000","last_fullsized":"2024-03-06T07:18:14.552173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:20.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720134+0000","last_change":"2024-03-06T07:18:09.524088+0000","last_active":"2024-03-06T07:18:14.720134+0000","last_peered":"2024-03-06T07:18:14.720134+0000","last_clean":"2024-03-06T07:18:14.720134+0000","last_became_active":"2024-03-06T07:18:09.523821+0000","last_became_peered":"2024-03-06T07:18:09.523821+0000","last_unstale":"2024-03-06T07:18:14.720134+0000","last_undegraded":"2024-03-06T07:18:14.720134+0000","last_fullsized":"2024-03-06T07:18:14.720134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719053+0000","last_change":"2024-03-06T07:18:08.516137+0000","last_active":"2024-03-06T07:18:14.719053+0000","last_peered":"2024-03-06T07:18:14.719053+0000","last_clean":"2024-03-06T07:18:14.719053+0000","last_became_active":"2024-03-06T07:18:08.515841+0000","last_became_peered":"2024-03-06T07:18:08.515841+0000","last_unstale":"2024-03-06T07:18:14.719053+0000","last_undegraded":"2024-03-06T07:18:14.719053+0000","last_fullsized":"2024-03-06T07:18:14.719053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551452+0000","last_change":"2024-03-06T07:18:13.564851+0000","last_active":"2024-03-06T07:18:14.551452+0000","last_peered":"2024-03-06T07:18:14.551452+0000","last_clean":"2024-03-06T07:18:14.551452+0000","last_became_active":"2024-03-06T07:18:13.564494+0000","last_became_peered":"2024-03-06T07:18:13.564494+0000","last_unstale":"2024-03-06T07:18:14.551452+0000","last_undegraded":"2024-03-06T07:18:14.551452+0000","last_fullsized":"2024-03-06T07:18:14.551452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:27.757638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720452+0000","last_change":"2024-03-06T07:18:09.527318+0000","last_active":"2024-03-06T07:18:14.720452+0000","last_peered":"2024-03-06T07:18:14.720452+0000","last_clean":"2024-03-06T07:18:14.720452+0000","last_became_active":"2024-03-06T07:18:09.527061+0000","last_became_peered":"2024-03-06T07:18:09.527061+0000","last_unstale":"2024-03-06T07:18:14.720452+0000","last_undegraded":"2024-03-06T07:18:14.720452+0000","last_fullsized":"2024-03-06T07:18:14.720452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719621+0000","last_change":"2024-03-06T07:18:08.515303+0000","last_active":"2024-03-06T07:18:14.719621+0000","last_peered":"2024-03-06T07:18:14.719621+0000","last_clean":"2024-03-06T07:18:14.719621+0000","last_became_active":"2024-03-06T07:18:08.514178+0000","last_became_peered":"2024-03-06T07:18:08.514178+0000","last_unstale":"2024-03-06T07:18:14.719621+0000","last_undegraded":"2024-03-06T07:18:14.719621+0000","last_fullsized":"2024-03-06T07:18:14.719621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719788+0000","last_change":"2024-03-06T07:18:13.788851+0000","last_active":"2024-03-06T07:18:14.719788+0000","last_peered":"2024-03-06T07:18:14.719788+0000","last_clean":"2024-03-06T07:18:14.719788+0000","last_became_active":"2024-03-06T07:18:13.788492+0000","last_became_peered":"2024-03-06T07:18:13.788492+0000","last_unstale":"2024-03-06T07:18:14.719788+0000","last_undegraded":"2024-03-06T07:18:14.719788+0000","last_fullsized":"2024-03-06T07:18:14.719788+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552468+0000","last_change":"2024-03-06T07:18:09.517957+0000","last_active":"2024-03-06T07:18:14.552468+0000","last_peered":"2024-03-06T07:18:14.552468+0000","last_clean":"2024-03-06T07:18:14.552468+0000","last_became_active":"2024-03-06T07:18:09.517197+0000","last_became_peered":"2024-03-06T07:18:09.517197+0000","last_unstale":"2024-03-06T07:18:14.552468+0000","last_undegraded":"2024-03-06T07:18:14.552468+0000","last_fullsized":"2024-03-06T07:18:14.552468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:31.051748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718952+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.718952+0000","last_peered":"2024-03-06T07:18:14.718952+0000","last_clean":"2024-03-06T07:18:14.718952+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.718952+0000","last_undegraded":"2024-03-06T07:18:14.718952+0000","last_fullsized":"2024-03-06T07:18:14.718952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552149+0000","last_change":"2024-03-06T07:18:13.565073+0000","last_active":"2024-03-06T07:18:14.552149+0000","last_peered":"2024-03-06T07:18:14.552149+0000","last_clean":"2024-03-06T07:18:14.552149+0000","last_became_active":"2024-03-06T07:18:13.564708+0000","last_became_peered":"2024-03-06T07:18:13.564708+0000","last_unstale":"2024-03-06T07:18:14.552149+0000","last_undegraded":"2024-03-06T07:18:14.552149+0000","last_fullsized":"2024-03-06T07:18:14.552149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:20.796366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719443+0000","last_change":"2024-03-06T07:18:09.520925+0000","last_active":"2024-03-06T07:18:14.719443+0000","last_peered":"2024-03-06T07:18:14.719443+0000","last_clean":"2024-03-06T07:18:14.719443+0000","last_became_active":"2024-03-06T07:18:09.520658+0000","last_became_peered":"2024-03-06T07:18:09.520658+0000","last_unstale":"2024-03-06T07:18:14.719443+0000","last_undegraded":"2024-03-06T07:18:14.719443+0000","last_fullsized":"2024-03-06T07:18:14.719443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551744+0000","last_change":"2024-03-06T07:18:08.519229+0000","last_active":"2024-03-06T07:18:14.551744+0000","last_peered":"2024-03-06T07:18:14.551744+0000","last_clean":"2024-03-06T07:18:14.551744+0000","last_became_active":"2024-03-06T07:18:08.518936+0000","last_became_peered":"2024-03-06T07:18:08.518936+0000","last_unstale":"2024-03-06T07:18:14.551744+0000","last_undegraded":"2024-03-06T07:18:14.551744+0000","last_fullsized":"2024-03-06T07:18:14.551744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552379+0000","last_change":"2024-03-06T07:18:13.787076+0000","last_active":"2024-03-06T07:18:14.552379+0000","last_peered":"2024-03-06T07:18:14.552379+0000","last_clean":"2024-03-06T07:18:14.552379+0000","last_became_active":"2024-03-06T07:18:13.786733+0000","last_became_peered":"2024-03-06T07:18:13.786733+0000","last_unstale":"2024-03-06T07:18:14.552379+0000","last_undegraded":"2024-03-06T07:18:14.552379+0000","last_fullsized":"2024-03-06T07:18:14.552379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720898+0000","last_change":"2024-03-06T07:18:09.534800+0000","last_active":"2024-03-06T07:18:14.720898+0000","last_peered":"2024-03-06T07:18:14.720898+0000","last_clean":"2024-03-06T07:18:14.720898+0000","last_became_active":"2024-03-06T07:18:09.534408+0000","last_became_peered":"2024-03-06T07:18:09.534408+0000","last_unstale":"2024-03-06T07:18:14.720898+0000","last_undegraded":"2024-03-06T07:18:14.720898+0000","last_fullsized":"2024-03-06T07:18:14.720898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719429+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.719429+0000","last_peered":"2024-03-06T07:18:14.719429+0000","last_clean":"2024-03-06T07:18:14.719429+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.719429+0000","last_undegraded":"2024-03-06T07:18:14.719429+0000","last_fullsized":"2024-03-06T07:18:14.719429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720202+0000","last_change":"2024-03-06T07:18:13.785863+0000","last_active":"2024-03-06T07:18:14.720202+0000","last_peered":"2024-03-06T07:18:14.720202+0000","last_clean":"2024-03-06T07:18:14.720202+0000","last_became_active":"2024-03-06T07:18:13.785533+0000","last_became_peered":"2024-03-06T07:18:13.785533+0000","last_unstale":"2024-03-06T07:18:14.720202+0000","last_undegraded":"2024-03-06T07:18:14.720202+0000","last_fullsized":"2024-03-06T07:18:14.720202+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:43.382716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718962+0000","last_change":"2024-03-06T07:18:09.523934+0000","last_active":"2024-03-06T07:18:14.718962+0000","last_peered":"2024-03-06T07:18:14.718962+0000","last_clean":"2024-03-06T07:18:14.718962+0000","last_became_active":"2024-03-06T07:18:09.523663+0000","last_became_peered":"2024-03-06T07:18:09.523663+0000","last_unstale":"2024-03-06T07:18:14.718962+0000","last_undegraded":"2024-03-06T07:18:14.718962+0000","last_fullsized":"2024-03-06T07:18:14.718962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718957+0000","last_change":"2024-03-06T07:18:08.520839+0000","last_active":"2024-03-06T07:18:14.718957+0000","last_peered":"2024-03-06T07:18:14.718957+0000","last_clean":"2024-03-06T07:18:14.718957+0000","last_became_active":"2024-03-06T07:18:08.520578+0000","last_became_peered":"2024-03-06T07:18:08.520578+0000","last_unstale":"2024-03-06T07:18:14.718957+0000","last_undegraded":"2024-03-06T07:18:14.718957+0000","last_fullsized":"2024-03-06T07:18:14.718957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:21.462756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718986+0000","last_change":"2024-03-06T07:18:13.787099+0000","last_active":"2024-03-06T07:18:14.718986+0000","last_peered":"2024-03-06T07:18:14.718986+0000","last_clean":"2024-03-06T07:18:14.718986+0000","last_became_active":"2024-03-06T07:18:13.786694+0000","last_became_peered":"2024-03-06T07:18:13.786694+0000","last_unstale":"2024-03-06T07:18:14.718986+0000","last_undegraded":"2024-03-06T07:18:14.718986+0000","last_fullsized":"2024-03-06T07:18:14.718986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719219+0000","last_change":"2024-03-06T07:18:09.531609+0000","last_active":"2024-03-06T07:18:14.719219+0000","last_peered":"2024-03-06T07:18:14.719219+0000","last_clean":"2024-03-06T07:18:14.719219+0000","last_became_active":"2024-03-06T07:18:09.531318+0000","last_became_peered":"2024-03-06T07:18:09.531318+0000","last_unstale":"2024-03-06T07:18:14.719219+0000","last_undegraded":"2024-03-06T07:18:14.719219+0000","last_fullsized":"2024-03-06T07:18:14.719219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719042+0000","last_change":"2024-03-06T07:18:08.516897+0000","last_active":"2024-03-06T07:18:14.719042+0000","last_peered":"2024-03-06T07:18:14.719042+0000","last_clean":"2024-03-06T07:18:14.719042+0000","last_became_active":"2024-03-06T07:18:08.516647+0000","last_became_peered":"2024-03-06T07:18:08.516647+0000","last_unstale":"2024-03-06T07:18:14.719042+0000","last_undegraded":"2024-03-06T07:18:14.719042+0000","last_fullsized":"2024-03-06T07:18:14.719042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718719+0000","last_change":"2024-03-06T07:18:13.564264+0000","last_active":"2024-03-06T07:18:14.718719+0000","last_peered":"2024-03-06T07:18:14.718719+0000","last_clean":"2024-03-06T07:18:14.718719+0000","last_became_active":"2024-03-06T07:18:13.563923+0000","last_became_peered":"2024-03-06T07:18:13.563923+0000","last_unstale":"2024-03-06T07:18:14.718719+0000","last_undegraded":"2024-03-06T07:18:14.718719+0000","last_fullsized":"2024-03-06T07:18:14.718719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552093+0000","last_change":"2024-03-06T07:18:09.519337+0000","last_active":"2024-03-06T07:18:14.552093+0000","last_peered":"2024-03-06T07:18:14.552093+0000","last_clean":"2024-03-06T07:18:14.552093+0000","last_became_active":"2024-03-06T07:18:09.519042+0000","last_became_peered":"2024-03-06T07:18:09.519042+0000","last_unstale":"2024-03-06T07:18:14.552093+0000","last_undegraded":"2024-03-06T07:18:14.552093+0000","last_fullsized":"2024-03-06T07:18:14.552093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:48.003076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718514+0000","last_change":"2024-03-06T07:18:08.521877+0000","last_active":"2024-03-06T07:18:14.718514+0000","last_peered":"2024-03-06T07:18:14.718514+0000","last_clean":"2024-03-06T07:18:14.718514+0000","last_became_active":"2024-03-06T07:18:08.521583+0000","last_became_peered":"2024-03-06T07:18:08.521583+0000","last_unstale":"2024-03-06T07:18:14.718514+0000","last_undegraded":"2024-03-06T07:18:14.718514+0000","last_fullsized":"2024-03-06T07:18:14.718514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718907+0000","last_change":"2024-03-06T07:18:13.566418+0000","last_active":"2024-03-06T07:18:14.718907+0000","last_peered":"2024-03-06T07:18:14.718907+0000","last_clean":"2024-03-06T07:18:14.718907+0000","last_became_active":"2024-03-06T07:18:13.566013+0000","last_became_peered":"2024-03-06T07:18:13.566013+0000","last_unstale":"2024-03-06T07:18:14.718907+0000","last_undegraded":"2024-03-06T07:18:14.718907+0000","last_fullsized":"2024-03-06T07:18:14.718907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719777+0000","last_change":"2024-03-06T07:18:09.522608+0000","last_active":"2024-03-06T07:18:14.719777+0000","last_peered":"2024-03-06T07:18:14.719777+0000","last_clean":"2024-03-06T07:18:14.719777+0000","last_became_active":"2024-03-06T07:18:09.522360+0000","last_became_peered":"2024-03-06T07:18:09.522360+0000","last_unstale":"2024-03-06T07:18:14.719777+0000","last_undegraded":"2024-03-06T07:18:14.719777+0000","last_fullsized":"2024-03-06T07:18:14.719777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548304+0000","last_change":"2024-03-06T07:18:08.512125+0000","last_active":"2024-03-06T07:18:14.548304+0000","last_peered":"2024-03-06T07:18:14.548304+0000","last_clean":"2024-03-06T07:18:14.548304+0000","last_became_active":"2024-03-06T07:18:08.511827+0000","last_became_peered":"2024-03-06T07:18:08.511827+0000","last_unstale":"2024-03-06T07:18:14.548304+0000","last_undegraded":"2024-03-06T07:18:14.548304+0000","last_fullsized":"2024-03-06T07:18:14.548304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719927+0000","last_change":"2024-03-06T07:18:13.573031+0000","last_active":"2024-03-06T07:18:14.719927+0000","last_peered":"2024-03-06T07:18:14.719927+0000","last_clean":"2024-03-06T07:18:14.719927+0000","last_became_active":"2024-03-06T07:18:13.572562+0000","last_became_peered":"2024-03-06T07:18:13.572562+0000","last_unstale":"2024-03-06T07:18:14.719927+0000","last_undegraded":"2024-03-06T07:18:14.719927+0000","last_fullsized":"2024-03-06T07:18:14.719927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:55.484210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719484+0000","last_change":"2024-03-06T07:18:09.518341+0000","last_active":"2024-03-06T07:18:14.719484+0000","last_peered":"2024-03-06T07:18:14.719484+0000","last_clean":"2024-03-06T07:18:14.719484+0000","last_became_active":"2024-03-06T07:18:09.518030+0000","last_became_peered":"2024-03-06T07:18:09.518030+0000","last_unstale":"2024-03-06T07:18:14.719484+0000","last_undegraded":"2024-03-06T07:18:14.719484+0000","last_fullsized":"2024-03-06T07:18:14.719484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:13.874406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718993+0000","last_change":"2024-03-06T07:18:08.522022+0000","last_active":"2024-03-06T07:18:14.718993+0000","last_peered":"2024-03-06T07:18:14.718993+0000","last_clean":"2024-03-06T07:18:14.718993+0000","last_became_active":"2024-03-06T07:18:08.521764+0000","last_became_peered":"2024-03-06T07:18:08.521764+0000","last_unstale":"2024-03-06T07:18:14.718993+0000","last_undegraded":"2024-03-06T07:18:14.718993+0000","last_fullsized":"2024-03-06T07:18:14.718993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719331+0000","last_change":"2024-03-06T07:18:13.785137+0000","last_active":"2024-03-06T07:18:14.719331+0000","last_peered":"2024-03-06T07:18:14.719331+0000","last_clean":"2024-03-06T07:18:14.719331+0000","last_became_active":"2024-03-06T07:18:13.784706+0000","last_became_peered":"2024-03-06T07:18:13.784706+0000","last_unstale":"2024-03-06T07:18:14.719331+0000","last_undegraded":"2024-03-06T07:18:14.719331+0000","last_fullsized":"2024-03-06T07:18:14.719331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719805+0000","last_change":"2024-03-06T07:18:09.532012+0000","last_active":"2024-03-06T07:18:14.719805+0000","last_peered":"2024-03-06T07:18:14.719805+0000","last_clean":"2024-03-06T07:18:14.719805+0000","last_became_active":"2024-03-06T07:18:09.531724+0000","last_became_peered":"2024-03-06T07:18:09.531724+0000","last_unstale":"2024-03-06T07:18:14.719805+0000","last_undegraded":"2024-03-06T07:18:14.719805+0000","last_fullsized":"2024-03-06T07:18:14.719805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718407+0000","last_change":"2024-03-06T07:18:08.515757+0000","last_active":"2024-03-06T07:18:14.718407+0000","last_peered":"2024-03-06T07:18:14.718407+0000","last_clean":"2024-03-06T07:18:14.718407+0000","last_became_active":"2024-03-06T07:18:08.515494+0000","last_became_peered":"2024-03-06T07:18:08.515494+0000","last_unstale":"2024-03-06T07:18:14.718407+0000","last_undegraded":"2024-03-06T07:18:14.718407+0000","last_fullsized":"2024-03-06T07:18:14.718407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719078+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:18:14.719078+0000","last_peered":"2024-03-06T07:18:14.719078+0000","last_clean":"2024-03-06T07:18:14.719078+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:18:14.719078+0000","last_undegraded":"2024-03-06T07:18:14.719078+0000","last_fullsized":"2024-03-06T07:18:14.719078+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551436+0000","last_change":"2024-03-06T07:18:13.786445+0000","last_active":"2024-03-06T07:18:14.551436+0000","last_peered":"2024-03-06T07:18:14.551436+0000","last_clean":"2024-03-06T07:18:14.551436+0000","last_became_active":"2024-03-06T07:18:13.786085+0000","last_became_peered":"2024-03-06T07:18:13.786085+0000","last_unstale":"2024-03-06T07:18:14.551436+0000","last_undegraded":"2024-03-06T07:18:14.551436+0000","last_fullsized":"2024-03-06T07:18:14.551436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718724+0000","last_change":"2024-03-06T07:18:09.532185+0000","last_active":"2024-03-06T07:18:14.718724+0000","last_peered":"2024-03-06T07:18:14.718724+0000","last_clean":"2024-03-06T07:18:14.718724+0000","last_became_active":"2024-03-06T07:18:09.531883+0000","last_became_peered":"2024-03-06T07:18:09.531883+0000","last_unstale":"2024-03-06T07:18:14.718724+0000","last_undegraded":"2024-03-06T07:18:14.718724+0000","last_fullsized":"2024-03-06T07:18:14.718724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548537+0000","last_change":"2024-03-06T07:18:08.518305+0000","last_active":"2024-03-06T07:18:14.548537+0000","last_peered":"2024-03-06T07:18:14.548537+0000","last_clean":"2024-03-06T07:18:14.548537+0000","last_became_active":"2024-03-06T07:18:08.517977+0000","last_became_peered":"2024-03-06T07:18:08.517977+0000","last_unstale":"2024-03-06T07:18:14.548537+0000","last_undegraded":"2024-03-06T07:18:14.548537+0000","last_fullsized":"2024-03-06T07:18:14.548537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550763+0000","last_change":"2024-03-06T07:18:13.785349+0000","last_active":"2024-03-06T07:18:14.550763+0000","last_peered":"2024-03-06T07:18:14.550763+0000","last_clean":"2024-03-06T07:18:14.550763+0000","last_became_active":"2024-03-06T07:18:13.784931+0000","last_became_peered":"2024-03-06T07:18:13.784931+0000","last_unstale":"2024-03-06T07:18:14.550763+0000","last_undegraded":"2024-03-06T07:18:14.550763+0000","last_fullsized":"2024-03-06T07:18:14.550763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718607+0000","last_change":"2024-03-06T07:18:09.532114+0000","last_active":"2024-03-06T07:18:14.718607+0000","last_peered":"2024-03-06T07:18:14.718607+0000","last_clean":"2024-03-06T07:18:14.718607+0000","last_became_active":"2024-03-06T07:18:09.531373+0000","last_became_peered":"2024-03-06T07:18:09.531373+0000","last_unstale":"2024-03-06T07:18:14.718607+0000","last_undegraded":"2024-03-06T07:18:14.718607+0000","last_fullsized":"2024-03-06T07:18:14.718607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718980+0000","last_change":"2024-03-06T07:18:08.520471+0000","last_active":"2024-03-06T07:18:14.718980+0000","last_peered":"2024-03-06T07:18:14.718980+0000","last_clean":"2024-03-06T07:18:14.718980+0000","last_became_active":"2024-03-06T07:18:08.520156+0000","last_became_peered":"2024-03-06T07:18:08.520156+0000","last_unstale":"2024-03-06T07:18:14.718980+0000","last_undegraded":"2024-03-06T07:18:14.718980+0000","last_fullsized":"2024-03-06T07:18:14.718980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719199+0000","last_change":"2024-03-06T07:18:13.569753+0000","last_active":"2024-03-06T07:18:14.719199+0000","last_peered":"2024-03-06T07:18:14.719199+0000","last_clean":"2024-03-06T07:18:14.719199+0000","last_became_active":"2024-03-06T07:18:13.569369+0000","last_became_peered":"2024-03-06T07:18:13.569369+0000","last_unstale":"2024-03-06T07:18:14.719199+0000","last_undegraded":"2024-03-06T07:18:14.719199+0000","last_fullsized":"2024-03-06T07:18:14.719199+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719153+0000","last_change":"2024-03-06T07:18:09.536431+0000","last_active":"2024-03-06T07:18:14.719153+0000","last_peered":"2024-03-06T07:18:14.719153+0000","last_clean":"2024-03-06T07:18:14.719153+0000","last_became_active":"2024-03-06T07:18:09.536167+0000","last_became_peered":"2024-03-06T07:18:09.536167+0000","last_unstale":"2024-03-06T07:18:14.719153+0000","last_undegraded":"2024-03-06T07:18:14.719153+0000","last_fullsized":"2024-03-06T07:18:14.719153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.441770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548593+0000","last_change":"2024-03-06T07:18:08.511825+0000","last_active":"2024-03-06T07:18:14.548593+0000","last_peered":"2024-03-06T07:18:14.548593+0000","last_clean":"2024-03-06T07:18:14.548593+0000","last_became_active":"2024-03-06T07:18:08.511363+0000","last_became_peered":"2024-03-06T07:18:08.511363+0000","last_unstale":"2024-03-06T07:18:14.548593+0000","last_undegraded":"2024-03-06T07:18:14.548593+0000","last_fullsized":"2024-03-06T07:18:14.548593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718008+0000","last_change":"2024-03-06T07:18:13.786468+0000","last_active":"2024-03-06T07:18:14.718008+0000","last_peered":"2024-03-06T07:18:14.718008+0000","last_clean":"2024-03-06T07:18:14.718008+0000","last_became_active":"2024-03-06T07:18:13.786129+0000","last_became_peered":"2024-03-06T07:18:13.786129+0000","last_unstale":"2024-03-06T07:18:14.718008+0000","last_undegraded":"2024-03-06T07:18:14.718008+0000","last_fullsized":"2024-03-06T07:18:14.718008+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:42.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718705+0000","last_change":"2024-03-06T07:18:09.534236+0000","last_active":"2024-03-06T07:18:14.718705+0000","last_peered":"2024-03-06T07:18:14.718705+0000","last_clean":"2024-03-06T07:18:14.718705+0000","last_became_active":"2024-03-06T07:18:09.533090+0000","last_became_peered":"2024-03-06T07:18:09.533090+0000","last_unstale":"2024-03-06T07:18:14.718705+0000","last_undegraded":"2024-03-06T07:18:14.718705+0000","last_fullsized":"2024-03-06T07:18:14.718705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552342+0000","last_change":"2024-03-06T07:18:08.519410+0000","last_active":"2024-03-06T07:18:14.552342+0000","last_peered":"2024-03-06T07:18:14.552342+0000","last_clean":"2024-03-06T07:18:14.552342+0000","last_became_active":"2024-03-06T07:18:08.519115+0000","last_became_peered":"2024-03-06T07:18:08.519115+0000","last_unstale":"2024-03-06T07:18:14.552342+0000","last_undegraded":"2024-03-06T07:18:14.552342+0000","last_fullsized":"2024-03-06T07:18:14.552342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:22.247661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552289+0000","last_change":"2024-03-06T07:18:13.563952+0000","last_active":"2024-03-06T07:18:14.552289+0000","last_peered":"2024-03-06T07:18:14.552289+0000","last_clean":"2024-03-06T07:18:14.552289+0000","last_became_active":"2024-03-06T07:18:13.563578+0000","last_became_peered":"2024-03-06T07:18:13.563578+0000","last_unstale":"2024-03-06T07:18:14.552289+0000","last_undegraded":"2024-03-06T07:18:14.552289+0000","last_fullsized":"2024-03-06T07:18:14.552289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718657+0000","last_change":"2024-03-06T07:18:09.526419+0000","last_active":"2024-03-06T07:18:14.718657+0000","last_peered":"2024-03-06T07:18:14.718657+0000","last_clean":"2024-03-06T07:18:14.718657+0000","last_became_active":"2024-03-06T07:18:09.526165+0000","last_became_peered":"2024-03-06T07:18:09.526165+0000","last_unstale":"2024-03-06T07:18:14.718657+0000","last_undegraded":"2024-03-06T07:18:14.718657+0000","last_fullsized":"2024-03-06T07:18:14.718657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548110+0000","last_change":"2024-03-06T07:18:08.508444+0000","last_active":"2024-03-06T07:18:14.548110+0000","last_peered":"2024-03-06T07:18:14.548110+0000","last_clean":"2024-03-06T07:18:14.548110+0000","last_became_active":"2024-03-06T07:18:08.508153+0000","last_became_peered":"2024-03-06T07:18:08.508153+0000","last_unstale":"2024-03-06T07:18:14.548110+0000","last_undegraded":"2024-03-06T07:18:14.548110+0000","last_fullsized":"2024-03-06T07:18:14.548110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719075+0000","last_change":"2024-03-06T07:18:13.567057+0000","last_active":"2024-03-06T07:18:14.719075+0000","last_peered":"2024-03-06T07:18:14.719075+0000","last_clean":"2024-03-06T07:18:14.719075+0000","last_became_active":"2024-03-06T07:18:13.566702+0000","last_became_peered":"2024-03-06T07:18:13.566702+0000","last_unstale":"2024-03-06T07:18:14.719075+0000","last_undegraded":"2024-03-06T07:18:14.719075+0000","last_fullsized":"2024-03-06T07:18:14.719075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550710+0000","last_change":"2024-03-06T07:18:09.520527+0000","last_active":"2024-03-06T07:18:14.550710+0000","last_peered":"2024-03-06T07:18:14.550710+0000","last_clean":"2024-03-06T07:18:14.550710+0000","last_became_active":"2024-03-06T07:18:09.520233+0000","last_became_peered":"2024-03-06T07:18:09.520233+0000","last_unstale":"2024-03-06T07:18:14.550710+0000","last_undegraded":"2024-03-06T07:18:14.550710+0000","last_fullsized":"2024-03-06T07:18:14.550710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550736+0000","last_change":"2024-03-06T07:18:08.521908+0000","last_active":"2024-03-06T07:18:14.550736+0000","last_peered":"2024-03-06T07:18:14.550736+0000","last_clean":"2024-03-06T07:18:14.550736+0000","last_became_active":"2024-03-06T07:18:08.521324+0000","last_became_peered":"2024-03-06T07:18:08.521324+0000","last_unstale":"2024-03-06T07:18:14.550736+0000","last_undegraded":"2024-03-06T07:18:14.550736+0000","last_fullsized":"2024-03-06T07:18:14.550736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720936+0000","last_change":"2024-03-06T07:18:13.788800+0000","last_active":"2024-03-06T07:18:14.720936+0000","last_peered":"2024-03-06T07:18:14.720936+0000","last_clean":"2024-03-06T07:18:14.720936+0000","last_became_active":"2024-03-06T07:18:13.788435+0000","last_became_peered":"2024-03-06T07:18:13.788435+0000","last_unstale":"2024-03-06T07:18:14.720936+0000","last_undegraded":"2024-03-06T07:18:14.720936+0000","last_fullsized":"2024-03-06T07:18:14.720936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:36.842720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719105+0000","last_change":"2024-03-06T07:18:09.527698+0000","last_active":"2024-03-06T07:18:14.719105+0000","last_peered":"2024-03-06T07:18:14.719105+0000","last_clean":"2024-03-06T07:18:14.719105+0000","last_became_active":"2024-03-06T07:18:09.527407+0000","last_became_peered":"2024-03-06T07:18:09.527407+0000","last_unstale":"2024-03-06T07:18:14.719105+0000","last_undegraded":"2024-03-06T07:18:14.719105+0000","last_fullsized":"2024-03-06T07:18:14.719105+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718447+0000","last_change":"2024-03-06T07:18:08.518380+0000","last_active":"2024-03-06T07:18:14.718447+0000","last_peered":"2024-03-06T07:18:14.718447+0000","last_clean":"2024-03-06T07:18:14.718447+0000","last_became_active":"2024-03-06T07:18:08.517571+0000","last_became_peered":"2024-03-06T07:18:08.517571+0000","last_unstale":"2024-03-06T07:18:14.718447+0000","last_undegraded":"2024-03-06T07:18:14.718447+0000","last_fullsized":"2024-03-06T07:18:14.718447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720838+0000","last_change":"2024-03-06T07:18:13.785111+0000","last_active":"2024-03-06T07:18:14.720838+0000","last_peered":"2024-03-06T07:18:14.720838+0000","last_clean":"2024-03-06T07:18:14.720838+0000","last_became_active":"2024-03-06T07:18:13.784751+0000","last_became_peered":"2024-03-06T07:18:13.784751+0000","last_unstale":"2024-03-06T07:18:14.720838+0000","last_undegraded":"2024-03-06T07:18:14.720838+0000","last_fullsized":"2024-03-06T07:18:14.720838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.733802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719140+0000","last_change":"2024-03-06T07:18:09.526243+0000","last_active":"2024-03-06T07:18:14.719140+0000","last_peered":"2024-03-06T07:18:14.719140+0000","last_clean":"2024-03-06T07:18:14.719140+0000","last_became_active":"2024-03-06T07:18:09.525985+0000","last_became_peered":"2024-03-06T07:18:09.525985+0000","last_unstale":"2024-03-06T07:18:14.719140+0000","last_undegraded":"2024-03-06T07:18:14.719140+0000","last_fullsized":"2024-03-06T07:18:14.719140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:20.656356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719309+0000","last_change":"2024-03-06T07:18:08.520844+0000","last_active":"2024-03-06T07:18:14.719309+0000","last_peered":"2024-03-06T07:18:14.719309+0000","last_clean":"2024-03-06T07:18:14.719309+0000","last_became_active":"2024-03-06T07:18:08.520318+0000","last_became_peered":"2024-03-06T07:18:08.520318+0000","last_unstale":"2024-03-06T07:18:14.719309+0000","last_undegraded":"2024-03-06T07:18:14.719309+0000","last_fullsized":"2024-03-06T07:18:14.719309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:28.274132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719091+0000","last_change":"2024-03-06T07:18:13.785334+0000","last_active":"2024-03-06T07:18:14.719091+0000","last_peered":"2024-03-06T07:18:14.719091+0000","last_clean":"2024-03-06T07:18:14.719091+0000","last_became_active":"2024-03-06T07:18:13.784953+0000","last_became_peered":"2024-03-06T07:18:13.784953+0000","last_unstale":"2024-03-06T07:18:14.719091+0000","last_undegraded":"2024-03-06T07:18:14.719091+0000","last_fullsized":"2024-03-06T07:18:14.719091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:18.038907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718773+0000","last_change":"2024-03-06T07:18:09.533771+0000","last_active":"2024-03-06T07:18:14.718773+0000","last_peered":"2024-03-06T07:18:14.718773+0000","last_clean":"2024-03-06T07:18:14.718773+0000","last_became_active":"2024-03-06T07:18:09.533520+0000","last_became_peered":"2024-03-06T07:18:09.533520+0000","last_unstale":"2024-03-06T07:18:14.718773+0000","last_undegraded":"2024-03-06T07:18:14.718773+0000","last_fullsized":"2024-03-06T07:18:14.718773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:18.912751+0000","last_change":"2024-03-06T07:18:08.511813+0000","last_active":"2024-03-06T07:18:18.912751+0000","last_peered":"2024-03-06T07:18:18.912751+0000","last_clean":"2024-03-06T07:18:18.912751+0000","last_became_active":"2024-03-06T07:18:08.511552+0000","last_became_peered":"2024-03-06T07:18:08.511552+0000","last_unstale":"2024-03-06T07:18:18.912751+0000","last_undegraded":"2024-03-06T07:18:18.912751+0000","last_fullsized":"2024-03-06T07:18:18.912751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718960+0000","last_change":"2024-03-06T07:18:13.567483+0000","last_active":"2024-03-06T07:18:14.718960+0000","last_peered":"2024-03-06T07:18:14.718960+0000","last_clean":"2024-03-06T07:18:14.718960+0000","last_became_active":"2024-03-06T07:18:13.566829+0000","last_became_peered":"2024-03-06T07:18:13.566829+0000","last_unstale":"2024-03-06T07:18:14.718960+0000","last_undegraded":"2024-03-06T07:18:14.718960+0000","last_fullsized":"2024-03-06T07:18:14.718960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551865+0000","last_change":"2024-03-06T07:18:09.525404+0000","last_active":"2024-03-06T07:18:14.551865+0000","last_peered":"2024-03-06T07:18:14.551865+0000","last_clean":"2024-03-06T07:18:14.551865+0000","last_became_active":"2024-03-06T07:18:09.525127+0000","last_became_peered":"2024-03-06T07:18:09.525127+0000","last_unstale":"2024-03-06T07:18:14.551865+0000","last_undegraded":"2024-03-06T07:18:14.551865+0000","last_fullsized":"2024-03-06T07:18:14.551865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718876+0000","last_change":"2024-03-06T07:18:08.518033+0000","last_active":"2024-03-06T07:18:14.718876+0000","last_peered":"2024-03-06T07:18:14.718876+0000","last_clean":"2024-03-06T07:18:14.718876+0000","last_became_active":"2024-03-06T07:18:08.517752+0000","last_became_peered":"2024-03-06T07:18:08.517752+0000","last_unstale":"2024-03-06T07:18:14.718876+0000","last_undegraded":"2024-03-06T07:18:14.718876+0000","last_fullsized":"2024-03-06T07:18:14.718876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551959+0000","last_change":"2024-03-06T07:18:13.561392+0000","last_active":"2024-03-06T07:18:14.551959+0000","last_peered":"2024-03-06T07:18:14.551959+0000","last_clean":"2024-03-06T07:18:14.551959+0000","last_became_active":"2024-03-06T07:18:13.561014+0000","last_became_peered":"2024-03-06T07:18:13.561014+0000","last_unstale":"2024-03-06T07:18:14.551959+0000","last_undegraded":"2024-03-06T07:18:14.551959+0000","last_fullsized":"2024-03-06T07:18:14.551959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:43.183715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719832+0000","last_change":"2024-03-06T07:18:09.531541+0000","last_active":"2024-03-06T07:18:14.719832+0000","last_peered":"2024-03-06T07:18:14.719832+0000","last_clean":"2024-03-06T07:18:14.719832+0000","last_became_active":"2024-03-06T07:18:09.531248+0000","last_became_peered":"2024-03-06T07:18:09.531248+0000","last_unstale":"2024-03-06T07:18:14.719832+0000","last_undegraded":"2024-03-06T07:18:14.719832+0000","last_fullsized":"2024-03-06T07:18:14.719832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719237+0000","last_change":"2024-03-06T07:18:08.519552+0000","last_active":"2024-03-06T07:18:14.719237+0000","last_peered":"2024-03-06T07:18:14.719237+0000","last_clean":"2024-03-06T07:18:14.719237+0000","last_became_active":"2024-03-06T07:18:08.519273+0000","last_became_peered":"2024-03-06T07:18:08.519273+0000","last_unstale":"2024-03-06T07:18:14.719237+0000","last_undegraded":"2024-03-06T07:18:14.719237+0000","last_fullsized":"2024-03-06T07:18:14.719237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720395+0000","last_change":"2024-03-06T07:18:13.564953+0000","last_active":"2024-03-06T07:18:14.720395+0000","last_peered":"2024-03-06T07:18:14.720395+0000","last_clean":"2024-03-06T07:18:14.720395+0000","last_became_active":"2024-03-06T07:18:13.564601+0000","last_became_peered":"2024-03-06T07:18:13.564601+0000","last_unstale":"2024-03-06T07:18:14.720395+0000","last_undegraded":"2024-03-06T07:18:14.720395+0000","last_fullsized":"2024-03-06T07:18:14.720395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:31.246872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718301+0000","last_change":"2024-03-06T07:18:09.535125+0000","last_active":"2024-03-06T07:18:14.718301+0000","last_peered":"2024-03-06T07:18:14.718301+0000","last_clean":"2024-03-06T07:18:14.718301+0000","last_became_active":"2024-03-06T07:18:09.534863+0000","last_became_peered":"2024-03-06T07:18:09.534863+0000","last_unstale":"2024-03-06T07:18:14.718301+0000","last_undegraded":"2024-03-06T07:18:14.718301+0000","last_fullsized":"2024-03-06T07:18:14.718301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551568+0000","last_change":"2024-03-06T07:18:08.517122+0000","last_active":"2024-03-06T07:18:14.551568+0000","last_peered":"2024-03-06T07:18:14.551568+0000","last_clean":"2024-03-06T07:18:14.551568+0000","last_became_active":"2024-03-06T07:18:08.516566+0000","last_became_peered":"2024-03-06T07:18:08.516566+0000","last_unstale":"2024-03-06T07:18:14.551568+0000","last_undegraded":"2024-03-06T07:18:14.551568+0000","last_fullsized":"2024-03-06T07:18:14.551568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720688+0000","last_change":"2024-03-06T07:18:13.789985+0000","last_active":"2024-03-06T07:18:14.720688+0000","last_peered":"2024-03-06T07:18:14.720688+0000","last_clean":"2024-03-06T07:18:14.720688+0000","last_became_active":"2024-03-06T07:18:13.789627+0000","last_became_peered":"2024-03-06T07:18:13.789627+0000","last_unstale":"2024-03-06T07:18:14.720688+0000","last_undegraded":"2024-03-06T07:18:14.720688+0000","last_fullsized":"2024-03-06T07:18:14.720688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:22:45.216688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718902+0000","last_change":"2024-03-06T07:18:13.786557+0000","last_active":"2024-03-06T07:18:14.718902+0000","last_peered":"2024-03-06T07:18:14.718902+0000","last_clean":"2024-03-06T07:18:14.718902+0000","last_became_active":"2024-03-06T07:18:13.786179+0000","last_became_peered":"2024-03-06T07:18:13.786179+0000","last_unstale":"2024-03-06T07:18:14.718902+0000","last_undegraded":"2024-03-06T07:18:14.718902+0000","last_fullsized":"2024-03-06T07:18:14.718902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719435+0000","last_change":"2024-03-06T07:18:09.528134+0000","last_active":"2024-03-06T07:18:14.719435+0000","last_peered":"2024-03-06T07:18:14.719435+0000","last_clean":"2024-03-06T07:18:14.719435+0000","last_became_active":"2024-03-06T07:18:09.527831+0000","last_became_peered":"2024-03-06T07:18:09.527831+0000","last_unstale":"2024-03-06T07:18:14.719435+0000","last_undegraded":"2024-03-06T07:18:14.719435+0000","last_fullsized":"2024-03-06T07:18:14.719435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718145+0000","last_change":"2024-03-06T07:18:08.520786+0000","last_active":"2024-03-06T07:18:14.718145+0000","last_peered":"2024-03-06T07:18:14.718145+0000","last_clean":"2024-03-06T07:18:14.718145+0000","last_became_active":"2024-03-06T07:18:08.520392+0000","last_became_peered":"2024-03-06T07:18:08.520392+0000","last_unstale":"2024-03-06T07:18:14.718145+0000","last_undegraded":"2024-03-06T07:18:14.718145+0000","last_fullsized":"2024-03-06T07:18:14.718145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551680+0000","last_change":"2024-03-06T07:18:13.566639+0000","last_active":"2024-03-06T07:18:14.551680+0000","last_peered":"2024-03-06T07:18:14.551680+0000","last_clean":"2024-03-06T07:18:14.551680+0000","last_became_active":"2024-03-06T07:18:13.566321+0000","last_became_peered":"2024-03-06T07:18:13.566321+0000","last_unstale":"2024-03-06T07:18:14.551680+0000","last_undegraded":"2024-03-06T07:18:14.551680+0000","last_fullsized":"2024-03-06T07:18:14.551680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720366+0000","last_change":"2024-03-06T07:18:09.523023+0000","last_active":"2024-03-06T07:18:14.720366+0000","last_peered":"2024-03-06T07:18:14.720366+0000","last_clean":"2024-03-06T07:18:14.720366+0000","last_became_active":"2024-03-06T07:18:09.522736+0000","last_became_peered":"2024-03-06T07:18:09.522736+0000","last_unstale":"2024-03-06T07:18:14.720366+0000","last_undegraded":"2024-03-06T07:18:14.720366+0000","last_fullsized":"2024-03-06T07:18:14.720366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720831+0000","last_change":"2024-03-06T07:18:08.516225+0000","last_active":"2024-03-06T07:18:14.720831+0000","last_peered":"2024-03-06T07:18:14.720831+0000","last_clean":"2024-03-06T07:18:14.720831+0000","last_became_active":"2024-03-06T07:18:08.515908+0000","last_became_peered":"2024-03-06T07:18:08.515908+0000","last_unstale":"2024-03-06T07:18:14.720831+0000","last_undegraded":"2024-03-06T07:18:14.720831+0000","last_fullsized":"2024-03-06T07:18:14.720831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550914+0000","last_change":"2024-03-06T07:18:08.513484+0000","last_active":"2024-03-06T07:18:14.550914+0000","last_peered":"2024-03-06T07:18:14.550914+0000","last_clean":"2024-03-06T07:18:14.550914+0000","last_became_active":"2024-03-06T07:18:08.513195+0000","last_became_peered":"2024-03-06T07:18:08.513195+0000","last_unstale":"2024-03-06T07:18:14.550914+0000","last_undegraded":"2024-03-06T07:18:14.550914+0000","last_fullsized":"2024-03-06T07:18:14.550914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719247+0000","last_change":"2024-03-06T07:18:09.518972+0000","last_active":"2024-03-06T07:18:14.719247+0000","last_peered":"2024-03-06T07:18:14.719247+0000","last_clean":"2024-03-06T07:18:14.719247+0000","last_became_active":"2024-03-06T07:18:09.518673+0000","last_became_peered":"2024-03-06T07:18:09.518673+0000","last_unstale":"2024-03-06T07:18:14.719247+0000","last_undegraded":"2024-03-06T07:18:14.719247+0000","last_fullsized":"2024-03-06T07:18:14.719247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548431+0000","last_change":"2024-03-06T07:18:13.569839+0000","last_active":"2024-03-06T07:18:14.548431+0000","last_peered":"2024-03-06T07:18:14.548431+0000","last_clean":"2024-03-06T07:18:14.548431+0000","last_became_active":"2024-03-06T07:18:13.569443+0000","last_became_peered":"2024-03-06T07:18:13.569443+0000","last_unstale":"2024-03-06T07:18:14.548431+0000","last_undegraded":"2024-03-06T07:18:14.548431+0000","last_fullsized":"2024-03-06T07:18:14.548431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:45.425994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720231+0000","last_change":"2024-03-06T07:18:08.519883+0000","last_active":"2024-03-06T07:18:14.720231+0000","last_peered":"2024-03-06T07:18:14.720231+0000","last_clean":"2024-03-06T07:18:14.720231+0000","last_became_active":"2024-03-06T07:18:08.519579+0000","last_became_peered":"2024-03-06T07:18:08.519579+0000","last_unstale":"2024-03-06T07:18:14.720231+0000","last_undegraded":"2024-03-06T07:18:14.720231+0000","last_fullsized":"2024-03-06T07:18:14.720231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551887+0000","last_change":"2024-03-06T07:18:09.520603+0000","last_active":"2024-03-06T07:18:14.551887+0000","last_peered":"2024-03-06T07:18:14.551887+0000","last_clean":"2024-03-06T07:18:14.551887+0000","last_became_active":"2024-03-06T07:18:09.520258+0000","last_became_peered":"2024-03-06T07:18:09.520258+0000","last_unstale":"2024-03-06T07:18:14.551887+0000","last_undegraded":"2024-03-06T07:18:14.551887+0000","last_fullsized":"2024-03-06T07:18:14.551887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720637+0000","last_change":"2024-03-06T07:18:13.564140+0000","last_active":"2024-03-06T07:18:14.720637+0000","last_peered":"2024-03-06T07:18:14.720637+0000","last_clean":"2024-03-06T07:18:14.720637+0000","last_became_active":"2024-03-06T07:18:13.563770+0000","last_became_peered":"2024-03-06T07:18:13.563770+0000","last_unstale":"2024-03-06T07:18:14.720637+0000","last_undegraded":"2024-03-06T07:18:14.720637+0000","last_fullsized":"2024-03-06T07:18:14.720637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:03.956217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550619+0000","last_change":"2024-03-06T07:18:09.520176+0000","last_active":"2024-03-06T07:18:14.550619+0000","last_peered":"2024-03-06T07:18:14.550619+0000","last_clean":"2024-03-06T07:18:14.550619+0000","last_became_active":"2024-03-06T07:18:09.519911+0000","last_became_peered":"2024-03-06T07:18:09.519911+0000","last_unstale":"2024-03-06T07:18:14.550619+0000","last_undegraded":"2024-03-06T07:18:14.550619+0000","last_fullsized":"2024-03-06T07:18:14.550619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548193+0000","last_change":"2024-03-06T07:18:09.524681+0000","last_active":"2024-03-06T07:18:14.548193+0000","last_peered":"2024-03-06T07:18:14.548193+0000","last_clean":"2024-03-06T07:18:14.548193+0000","last_became_active":"2024-03-06T07:18:09.524427+0000","last_became_peered":"2024-03-06T07:18:09.524427+0000","last_unstale":"2024-03-06T07:18:14.548193+0000","last_undegraded":"2024-03-06T07:18:14.548193+0000","last_fullsized":"2024-03-06T07:18:14.548193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.531928+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531670+0000","last_became_peered":"2024-03-06T07:18:09.531670+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719143+0000","last_change":"2024-03-06T07:18:09.523431+0000","last_active":"2024-03-06T07:18:14.719143+0000","last_peered":"2024-03-06T07:18:14.719143+0000","last_clean":"2024-03-06T07:18:14.719143+0000","last_became_active":"2024-03-06T07:18:09.523169+0000","last_became_peered":"2024-03-06T07:18:09.523169+0000","last_unstale":"2024-03-06T07:18:14.719143+0000","last_undegraded":"2024-03-06T07:18:14.719143+0000","last_fullsized":"2024-03-06T07:18:14.719143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720897+0000","last_change":"2024-03-06T07:18:09.525359+0000","last_active":"2024-03-06T07:18:14.720897+0000","last_peered":"2024-03-06T07:18:14.720897+0000","last_clean":"2024-03-06T07:18:14.720897+0000","last_became_active":"2024-03-06T07:18:09.525069+0000","last_became_peered":"2024-03-06T07:18:09.525069+0000","last_unstale":"2024-03-06T07:18:14.720897+0000","last_undegraded":"2024-03-06T07:18:14.720897+0000","last_fullsized":"2024-03-06T07:18:14.720897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718709+0000","last_change":"2024-03-06T07:18:09.528783+0000","last_active":"2024-03-06T07:18:14.718709+0000","last_peered":"2024-03-06T07:18:14.718709+0000","last_clean":"2024-03-06T07:18:14.718709+0000","last_became_active":"2024-03-06T07:18:09.528488+0000","last_became_peered":"2024-03-06T07:18:09.528488+0000","last_unstale":"2024-03-06T07:18:14.718709+0000","last_undegraded":"2024-03-06T07:18:14.718709+0000","last_fullsized":"2024-03-06T07:18:14.718709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551739+0000","last_change":"2024-03-06T07:18:09.526930+0000","last_active":"2024-03-06T07:18:14.551739+0000","last_peered":"2024-03-06T07:18:14.551739+0000","last_clean":"2024-03-06T07:18:14.551739+0000","last_became_active":"2024-03-06T07:18:09.526575+0000","last_became_peered":"2024-03-06T07:18:09.526575+0000","last_unstale":"2024-03-06T07:18:14.551739+0000","last_undegraded":"2024-03-06T07:18:14.551739+0000","last_fullsized":"2024-03-06T07:18:14.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720809+0000","last_change":"2024-03-06T07:18:09.532541+0000","last_active":"2024-03-06T07:18:14.720809+0000","last_peered":"2024-03-06T07:18:14.720809+0000","last_clean":"2024-03-06T07:18:14.720809+0000","last_became_active":"2024-03-06T07:18:09.532260+0000","last_became_peered":"2024-03-06T07:18:09.532260+0000","last_unstale":"2024-03-06T07:18:14.720809+0000","last_undegraded":"2024-03-06T07:18:14.720809+0000","last_fullsized":"2024-03-06T07:18:14.720809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718061+0000","last_change":"2024-03-06T07:18:09.530812+0000","last_active":"2024-03-06T07:18:14.718061+0000","last_peered":"2024-03-06T07:18:14.718061+0000","last_clean":"2024-03-06T07:18:14.718061+0000","last_became_active":"2024-03-06T07:18:09.530493+0000","last_became_peered":"2024-03-06T07:18:09.530493+0000","last_unstale":"2024-03-06T07:18:14.718061+0000","last_undegraded":"2024-03-06T07:18:14.718061+0000","last_fullsized":"2024-03-06T07:18:14.718061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720741+0000","last_change":"2024-03-06T07:18:09.525260+0000","last_active":"2024-03-06T07:18:14.720741+0000","last_peered":"2024-03-06T07:18:14.720741+0000","last_clean":"2024-03-06T07:18:14.720741+0000","last_became_active":"2024-03-06T07:18:09.524996+0000","last_became_peered":"2024-03-06T07:18:09.524996+0000","last_unstale":"2024-03-06T07:18:14.720741+0000","last_undegraded":"2024-03-06T07:18:14.720741+0000","last_fullsized":"2024-03-06T07:18:14.720741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718326+0000","last_change":"2024-03-06T07:18:09.539784+0000","last_active":"2024-03-06T07:18:14.718326+0000","last_peered":"2024-03-06T07:18:14.718326+0000","last_clean":"2024-03-06T07:18:14.718326+0000","last_became_active":"2024-03-06T07:18:09.538642+0000","last_became_peered":"2024-03-06T07:18:09.538642+0000","last_unstale":"2024-03-06T07:18:14.718326+0000","last_undegraded":"2024-03-06T07:18:14.718326+0000","last_fullsized":"2024-03-06T07:18:14.718326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719031+0000","last_change":"2024-03-06T07:18:09.526156+0000","last_active":"2024-03-06T07:18:14.719031+0000","last_peered":"2024-03-06T07:18:14.719031+0000","last_clean":"2024-03-06T07:18:14.719031+0000","last_became_active":"2024-03-06T07:18:09.525877+0000","last_became_peered":"2024-03-06T07:18:09.525877+0000","last_unstale":"2024-03-06T07:18:14.719031+0000","last_undegraded":"2024-03-06T07:18:14.719031+0000","last_fullsized":"2024-03-06T07:18:14.719031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551188+0000","last_change":"2024-03-06T07:18:09.516685+0000","last_active":"2024-03-06T07:18:14.551188+0000","last_peered":"2024-03-06T07:18:14.551188+0000","last_clean":"2024-03-06T07:18:14.551188+0000","last_became_active":"2024-03-06T07:18:09.516377+0000","last_became_peered":"2024-03-06T07:18:09.516377+0000","last_unstale":"2024-03-06T07:18:14.551188+0000","last_undegraded":"2024-03-06T07:18:14.551188+0000","last_fullsized":"2024-03-06T07:18:14.551188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718906+0000","last_change":"2024-03-06T07:18:09.524856+0000","last_active":"2024-03-06T07:18:14.718906+0000","last_peered":"2024-03-06T07:18:14.718906+0000","last_clean":"2024-03-06T07:18:14.718906+0000","last_became_active":"2024-03-06T07:18:09.523916+0000","last_became_peered":"2024-03-06T07:18:09.523916+0000","last_unstale":"2024-03-06T07:18:14.718906+0000","last_undegraded":"2024-03-06T07:18:14.718906+0000","last_fullsized":"2024-03-06T07:18:14.718906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548544+0000","last_change":"2024-03-06T07:18:09.518046+0000","last_active":"2024-03-06T07:18:14.548544+0000","last_peered":"2024-03-06T07:18:14.548544+0000","last_clean":"2024-03-06T07:18:14.548544+0000","last_became_active":"2024-03-06T07:18:09.517756+0000","last_became_peered":"2024-03-06T07:18:09.517756+0000","last_unstale":"2024-03-06T07:18:14.548544+0000","last_undegraded":"2024-03-06T07:18:14.548544+0000","last_fullsized":"2024-03-06T07:18:14.548544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719287+0000","last_change":"2024-03-06T07:18:09.537036+0000","last_active":"2024-03-06T07:18:14.719287+0000","last_peered":"2024-03-06T07:18:14.719287+0000","last_clean":"2024-03-06T07:18:14.719287+0000","last_became_active":"2024-03-06T07:18:09.536576+0000","last_became_peered":"2024-03-06T07:18:09.536576+0000","last_unstale":"2024-03-06T07:18:14.719287+0000","last_undegraded":"2024-03-06T07:18:14.719287+0000","last_fullsized":"2024-03-06T07:18:14.719287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.717960+0000","last_change":"2024-03-06T07:18:09.527339+0000","last_active":"2024-03-06T07:18:14.717960+0000","last_peered":"2024-03-06T07:18:14.717960+0000","last_clean":"2024-03-06T07:18:14.717960+0000","last_became_active":"2024-03-06T07:18:09.526873+0000","last_became_peered":"2024-03-06T07:18:09.526873+0000","last_unstale":"2024-03-06T07:18:14.717960+0000","last_undegraded":"2024-03-06T07:18:14.717960+0000","last_fullsized":"2024-03-06T07:18:14.717960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719530+0000","last_change":"2024-03-06T07:18:09.522712+0000","last_active":"2024-03-06T07:18:14.719530+0000","last_peered":"2024-03-06T07:18:14.719530+0000","last_clean":"2024-03-06T07:18:14.719530+0000","last_became_active":"2024-03-06T07:18:09.522433+0000","last_became_peered":"2024-03-06T07:18:09.522433+0000","last_unstale":"2024-03-06T07:18:14.719530+0000","last_undegraded":"2024-03-06T07:18:14.719530+0000","last_fullsized":"2024-03-06T07:18:14.719530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719097+0000","last_change":"2024-03-06T07:18:09.522057+0000","last_active":"2024-03-06T07:18:14.719097+0000","last_peered":"2024-03-06T07:18:14.719097+0000","last_clean":"2024-03-06T07:18:14.719097+0000","last_became_active":"2024-03-06T07:18:09.521270+0000","last_became_peered":"2024-03-06T07:18:09.521270+0000","last_unstale":"2024-03-06T07:18:14.719097+0000","last_undegraded":"2024-03-06T07:18:14.719097+0000","last_fullsized":"2024-03-06T07:18:14.719097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:57.186351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720044+0000","last_change":"2024-03-06T07:18:09.526002+0000","last_active":"2024-03-06T07:18:14.720044+0000","last_peered":"2024-03-06T07:18:14.720044+0000","last_clean":"2024-03-06T07:18:14.720044+0000","last_became_active":"2024-03-06T07:18:09.525717+0000","last_became_peered":"2024-03-06T07:18:09.525717+0000","last_unstale":"2024-03-06T07:18:14.720044+0000","last_undegraded":"2024-03-06T07:18:14.720044+0000","last_fullsized":"2024-03-06T07:18:14.720044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551860+0000","last_change":"2024-03-06T07:18:09.521766+0000","last_active":"2024-03-06T07:18:14.551860+0000","last_peered":"2024-03-06T07:18:14.551860+0000","last_clean":"2024-03-06T07:18:14.551860+0000","last_became_active":"2024-03-06T07:18:09.521159+0000","last_became_peered":"2024-03-06T07:18:09.521159+0000","last_unstale":"2024-03-06T07:18:14.551860+0000","last_undegraded":"2024-03-06T07:18:14.551860+0000","last_fullsized":"2024-03-06T07:18:14.551860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:57.265260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551251+0000","last_change":"2024-03-06T07:18:09.533157+0000","last_active":"2024-03-06T07:18:14.551251+0000","last_peered":"2024-03-06T07:18:14.551251+0000","last_clean":"2024-03-06T07:18:14.551251+0000","last_became_active":"2024-03-06T07:18:09.532868+0000","last_became_peered":"2024-03-06T07:18:09.532868+0000","last_unstale":"2024-03-06T07:18:14.551251+0000","last_undegraded":"2024-03-06T07:18:14.551251+0000","last_fullsized":"2024-03-06T07:18:14.551251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551648+0000","last_change":"2024-03-06T07:18:09.532473+0000","last_active":"2024-03-06T07:18:14.551648+0000","last_peered":"2024-03-06T07:18:14.551648+0000","last_clean":"2024-03-06T07:18:14.551648+0000","last_became_active":"2024-03-06T07:18:09.532191+0000","last_became_peered":"2024-03-06T07:18:09.532191+0000","last_unstale":"2024-03-06T07:18:14.551648+0000","last_undegraded":"2024-03-06T07:18:14.551648+0000","last_fullsized":"2024-03-06T07:18:14.551648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719062+0000","last_change":"2024-03-06T07:18:09.533061+0000","last_active":"2024-03-06T07:18:14.719062+0000","last_peered":"2024-03-06T07:18:14.719062+0000","last_clean":"2024-03-06T07:18:14.719062+0000","last_became_active":"2024-03-06T07:18:09.532742+0000","last_became_peered":"2024-03-06T07:18:09.532742+0000","last_unstale":"2024-03-06T07:18:14.719062+0000","last_undegraded":"2024-03-06T07:18:14.719062+0000","last_fullsized":"2024-03-06T07:18:14.719062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:25.345923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720283+0000","last_change":"2024-03-06T07:18:09.532342+0000","last_active":"2024-03-06T07:18:14.720283+0000","last_peered":"2024-03-06T07:18:14.720283+0000","last_clean":"2024-03-06T07:18:14.720283+0000","last_became_active":"2024-03-06T07:18:09.532062+0000","last_became_peered":"2024-03-06T07:18:09.532062+0000","last_unstale":"2024-03-06T07:18:14.720283+0000","last_undegraded":"2024-03-06T07:18:14.720283+0000","last_fullsized":"2024-03-06T07:18:14.720283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718610+0000","last_change":"2024-03-06T07:18:09.525768+0000","last_active":"2024-03-06T07:18:14.718610+0000","last_peered":"2024-03-06T07:18:14.718610+0000","last_clean":"2024-03-06T07:18:14.718610+0000","last_became_active":"2024-03-06T07:18:09.524604+0000","last_became_peered":"2024-03-06T07:18:09.524604+0000","last_unstale":"2024-03-06T07:18:14.718610+0000","last_undegraded":"2024-03-06T07:18:14.718610+0000","last_fullsized":"2024-03-06T07:18:14.718610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718219+0000","last_change":"2024-03-06T07:18:09.531193+0000","last_active":"2024-03-06T07:18:14.718219+0000","last_peered":"2024-03-06T07:18:14.718219+0000","last_clean":"2024-03-06T07:18:14.718219+0000","last_became_active":"2024-03-06T07:18:09.530931+0000","last_became_peered":"2024-03-06T07:18:09.530931+0000","last_unstale":"2024-03-06T07:18:14.718219+0000","last_undegraded":"2024-03-06T07:18:14.718219+0000","last_fullsized":"2024-03-06T07:18:14.718219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718236+0000","last_change":"2024-03-06T07:18:09.524492+0000","last_active":"2024-03-06T07:18:14.718236+0000","last_peered":"2024-03-06T07:18:14.718236+0000","last_clean":"2024-03-06T07:18:14.718236+0000","last_became_active":"2024-03-06T07:18:09.524202+0000","last_became_peered":"2024-03-06T07:18:09.524202+0000","last_unstale":"2024-03-06T07:18:14.718236+0000","last_undegraded":"2024-03-06T07:18:14.718236+0000","last_fullsized":"2024-03-06T07:18:14.718236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.532118+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531771+0000","last_became_peered":"2024-03-06T07:18:09.531771+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719327+0000","last_change":"2024-03-06T07:18:09.519755+0000","last_active":"2024-03-06T07:18:14.719327+0000","last_peered":"2024-03-06T07:18:14.719327+0000","last_clean":"2024-03-06T07:18:14.719327+0000","last_became_active":"2024-03-06T07:18:09.519475+0000","last_became_peered":"2024-03-06T07:18:09.519475+0000","last_unstale":"2024-03-06T07:18:14.719327+0000","last_undegraded":"2024-03-06T07:18:14.719327+0000","last_fullsized":"2024-03-06T07:18:14.719327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719345+0000","last_change":"2024-03-06T07:18:09.524863+0000","last_active":"2024-03-06T07:18:14.719345+0000","last_peered":"2024-03-06T07:18:14.719345+0000","last_clean":"2024-03-06T07:18:14.719345+0000","last_became_active":"2024-03-06T07:18:09.524588+0000","last_became_peered":"2024-03-06T07:18:09.524588+0000","last_unstale":"2024-03-06T07:18:14.719345+0000","last_undegraded":"2024-03-06T07:18:14.719345+0000","last_fullsized":"2024-03-06T07:18:14.719345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718889+0000","last_change":"2024-03-06T07:18:09.528727+0000","last_active":"2024-03-06T07:18:14.718889+0000","last_peered":"2024-03-06T07:18:14.718889+0000","last_clean":"2024-03-06T07:18:14.718889+0000","last_became_active":"2024-03-06T07:18:09.528474+0000","last_became_peered":"2024-03-06T07:18:09.528474+0000","last_unstale":"2024-03-06T07:18:14.718889+0000","last_undegraded":"2024-03-06T07:18:14.718889+0000","last_fullsized":"2024-03-06T07:18:14.718889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:18:35.337 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:18:35.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.471+0000 7fa824803640 1 Processor -- start 2024-03-06T07:18:35.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.471+0000 7fa824803640 1 -- start start 2024-03-06T07:18:35.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa824803640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8200c8690 0x7fa8201559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa824803640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa820158a40 0x7fa820158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa824803640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82014be70 con 0x7fa82014ba90 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa824803640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8201593f0 con 0x7fa8200c8690 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa824803640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8200cad30 con 0x7fa820158a40 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa81e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8200c8690 0x7fa8201559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa81e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8200c8690 0x7fa8201559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41094/0 (socket says 172.21.15.94:41094) 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa81e59c640 1 -- 172.21.15.94:0/2860233981 learned_addr learned my addr 172.21.15.94:0/2860233981 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa81ed9d640 1 --1- 172.21.15.94:0/2860233981 >> v1:172.21.15.145:6789/0 conn(0x7fa82014ba90 0x7fa8200c5110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43868/0 (socket says 172.21.15.94:43868) 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.472+0000 7fa81f59e640 1 --2- 172.21.15.94:0/2860233981 >> v2:172.21.15.145:3300/0 conn(0x7fa820158a40 0x7fa820158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.473+0000 7fa81e59c640 1 -- 172.21.15.94:0/2860233981 >> v2:172.21.15.145:3300/0 conn(0x7fa820158a40 msgr2=0x7fa820158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.473+0000 7fa81e59c640 1 --2- 172.21.15.94:0/2860233981 >> v2:172.21.15.145:3300/0 conn(0x7fa820158a40 0x7fa820158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.473+0000 7fa81e59c640 1 -- 172.21.15.94:0/2860233981 >> v1:172.21.15.145:6789/0 conn(0x7fa82014ba90 legacy=0x7fa8200c5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.473+0000 7fa81e59c640 1 -- 172.21.15.94:0/2860233981 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8200cb010 con 0x7fa8200c8690 2024-03-06T07:18:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.473+0000 7fa81dd9b640 1 -- 172.21.15.94:0/2860233981 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463352323 0 0) 0x7fa814001110 con 0x7fa82014ba90 2024-03-06T07:18:35.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.473+0000 7fa81e59c640 1 --2- 172.21.15.94:0/2860233981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201559b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa820154f30 tx=0x7fa81002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a145e96964a8e5f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa81dd9b640 1 -- 172.21.15.94:0/2860233981 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810034070 con 0x7fa8200c8690 2024-03-06T07:18:35.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa81dd9b640 1 -- 172.21.15.94:0/2860233981 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810030860 con 0x7fa8200c8690 2024-03-06T07:18:35.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa81dd9b640 1 -- 172.21.15.94:0/2860233981 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810030c30 con 0x7fa8200c8690 2024-03-06T07:18:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 -- 172.21.15.94:0/2860233981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 msgr2=0x7fa8201559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 --2- 172.21.15.94:0/2860233981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201559b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa820154f30 tx=0x7fa81002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 -- 172.21.15.94:0/2860233981 shutdown_connections 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 --2- 172.21.15.94:0/2860233981 >> v2:172.21.15.145:3300/0 conn(0x7fa820158a40 0x7fa820158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 --2- 172.21.15.94:0/2860233981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201559b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 -- 172.21.15.94:0/2860233981 >> 172.21.15.94:0/2860233981 conn(0x7fa8200728d0 msgr2=0x7fa820072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 -- 172.21.15.94:0/2860233981 shutdown_connections 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.474+0000 7fa824803640 1 -- 172.21.15.94:0/2860233981 wait complete. 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 Processor -- start 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 -- start start 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82014ba90 0x7fa8201e1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa820158a40 0x7fa8200aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82006cf60 con 0x7fa8200c8690 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa820158520 con 0x7fa82014ba90 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa824803640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200c5820 con 0x7fa820158a40 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa81ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa81ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201eaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41110/0 (socket says 172.21.15.94:41110) 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa81ed9d640 1 -- 172.21.15.94:0/3220949696 learned_addr learned my addr 172.21.15.94:0/3220949696 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa81e59c640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82014ba90 0x7fa8201e1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.476+0000 7fa81f59e640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa820158a40 0x7fa8200aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa81ed9d640 1 -- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa820158a40 msgr2=0x7fa8200aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa81ed9d640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa820158a40 0x7fa8200aa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa81ed9d640 1 -- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82014ba90 msgr2=0x7fa8201e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa81ed9d640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82014ba90 0x7fa8201e1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa81ed9d640 1 -- 172.21.15.94:0/3220949696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8201eb3d0 con 0x7fa8200c8690 2024-03-06T07:18:35.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa81ed9d640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201eaca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c7b0 tx=0x7fa81400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa8077fe640 1 -- 172.21.15.94:0/3220949696 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814018070 con 0x7fa8200c8690 2024-03-06T07:18:35.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa8077fe640 1 -- 172.21.15.94:0/3220949696 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8140072d0 con 0x7fa8200c8690 2024-03-06T07:18:35.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8201eb6b0 con 0x7fa8200c8690 2024-03-06T07:18:35.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa8077fe640 1 -- 172.21.15.94:0/3220949696 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8140047e0 con 0x7fa8200c8690 2024-03-06T07:18:35.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.477+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8200aa3b0 con 0x7fa8200c8690 2024-03-06T07:18:35.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.478+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa820069430 con 0x7fa8200c8690 2024-03-06T07:18:35.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.478+0000 7fa8077fe640 1 -- 172.21.15.94:0/3220949696 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa81401b070 con 0x7fa8200c8690 2024-03-06T07:18:35.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.479+0000 7fa8077fe640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa80c06b4d0 0x7fa80c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.479+0000 7fa8077fe640 1 -- 172.21.15.94:0/3220949696 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa814089720 con 0x7fa8200c8690 2024-03-06T07:18:35.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.479+0000 7fa81e59c640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa80c06b4d0 0x7fa80c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.479+0000 7fa81e59c640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa80c06b4d0 0x7fa80c06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa810036000 tx=0x7fa810008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.483+0000 7fa8077fe640 1 -- 172.21.15.94:0/3220949696 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa82014be70 con 0x7fa8200c8690 2024-03-06T07:18:35.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.654+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa8200c15f0 con 0x7fa80c06b4d0 2024-03-06T07:18:35.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.662+0000 7fa8077fe640 1 -- 172.21.15.94:0/3220949696 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332610 (secure 0 0 0) 0x7fa8200c15f0 con 0x7fa80c06b4d0 2024-03-06T07:18:35.668 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:35.670 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:18:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa80c06b4d0 msgr2=0x7fa80c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa80c06b4d0 0x7fa80c06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa810036000 tx=0x7fa810008030 comp rx=0 tx=0).stop 2024-03-06T07:18:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 msgr2=0x7fa8201eaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201eaca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c7b0 tx=0x7fa81400cc70 comp rx=0 tx=0).stop 2024-03-06T07:18:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 shutdown_connections 2024-03-06T07:18:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa80c06b4d0 0x7fa80c06d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa820158a40 0x7fa8200aa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82014ba90 0x7fa8201e1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 --2- 172.21.15.94:0/3220949696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8200c8690 0x7fa8201eaca0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 >> 172.21.15.94:0/3220949696 conn(0x7fa8200728d0 msgr2=0x7fa8200c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 shutdown_connections 2024-03-06T07:18:35.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.665+0000 7fa824803640 1 -- 172.21.15.94:0/3220949696 wait complete. 2024-03-06T07:18:35.678 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-06T07:18:34.640338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226688,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754748032,"statfs":{"total":773094113280,"available":772861984768,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002489"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719950+0000","last_change":"2024-03-06T07:18:08.523161+0000","last_active":"2024-03-06T07:18:14.719950+0000","last_peered":"2024-03-06T07:18:14.719950+0000","last_clean":"2024-03-06T07:18:14.719950+0000","last_became_active":"2024-03-06T07:18:08.522883+0000","last_became_peered":"2024-03-06T07:18:08.522883+0000","last_unstale":"2024-03-06T07:18:14.719950+0000","last_undegraded":"2024-03-06T07:18:14.719950+0000","last_fullsized":"2024-03-06T07:18:14.719950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720669+0000","last_change":"2024-03-06T07:18:09.526639+0000","last_active":"2024-03-06T07:18:14.720669+0000","last_peered":"2024-03-06T07:18:14.720669+0000","last_clean":"2024-03-06T07:18:14.720669+0000","last_became_active":"2024-03-06T07:18:09.526380+0000","last_became_peered":"2024-03-06T07:18:09.526380+0000","last_unstale":"2024-03-06T07:18:14.720669+0000","last_undegraded":"2024-03-06T07:18:14.720669+0000","last_fullsized":"2024-03-06T07:18:14.720669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720585+0000","last_change":"2024-03-06T07:18:13.788926+0000","last_active":"2024-03-06T07:18:14.720585+0000","last_peered":"2024-03-06T07:18:14.720585+0000","last_clean":"2024-03-06T07:18:14.720585+0000","last_became_active":"2024-03-06T07:18:13.788572+0000","last_became_peered":"2024-03-06T07:18:13.788572+0000","last_unstale":"2024-03-06T07:18:14.720585+0000","last_undegraded":"2024-03-06T07:18:14.720585+0000","last_fullsized":"2024-03-06T07:18:14.720585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:00.875495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720880+0000","last_change":"2024-03-06T07:18:08.517732+0000","last_active":"2024-03-06T07:18:14.720880+0000","last_peered":"2024-03-06T07:18:14.720880+0000","last_clean":"2024-03-06T07:18:14.720880+0000","last_became_active":"2024-03-06T07:18:08.517469+0000","last_became_peered":"2024-03-06T07:18:08.517469+0000","last_unstale":"2024-03-06T07:18:14.720880+0000","last_undegraded":"2024-03-06T07:18:14.720880+0000","last_fullsized":"2024-03-06T07:18:14.720880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719676+0000","last_change":"2024-03-06T07:18:09.531502+0000","last_active":"2024-03-06T07:18:14.719676+0000","last_peered":"2024-03-06T07:18:14.719676+0000","last_clean":"2024-03-06T07:18:14.719676+0000","last_became_active":"2024-03-06T07:18:09.531231+0000","last_became_peered":"2024-03-06T07:18:09.531231+0000","last_unstale":"2024-03-06T07:18:14.719676+0000","last_undegraded":"2024-03-06T07:18:14.719676+0000","last_fullsized":"2024-03-06T07:18:14.719676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551608+0000","last_change":"2024-03-06T07:18:13.565825+0000","last_active":"2024-03-06T07:18:14.551608+0000","last_peered":"2024-03-06T07:18:14.551608+0000","last_clean":"2024-03-06T07:18:14.551608+0000","last_became_active":"2024-03-06T07:18:13.565250+0000","last_became_peered":"2024-03-06T07:18:13.565250+0000","last_unstale":"2024-03-06T07:18:14.551608+0000","last_undegraded":"2024-03-06T07:18:14.551608+0000","last_fullsized":"2024-03-06T07:18:14.551608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:15.265063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718863+0000","last_change":"2024-03-06T07:18:08.523848+0000","last_active":"2024-03-06T07:18:14.718863+0000","last_peered":"2024-03-06T07:18:14.718863+0000","last_clean":"2024-03-06T07:18:14.718863+0000","last_became_active":"2024-03-06T07:18:08.523455+0000","last_became_peered":"2024-03-06T07:18:08.523455+0000","last_unstale":"2024-03-06T07:18:14.718863+0000","last_undegraded":"2024-03-06T07:18:14.718863+0000","last_fullsized":"2024-03-06T07:18:14.718863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718515+0000","last_change":"2024-03-06T07:18:09.531072+0000","last_active":"2024-03-06T07:18:14.718515+0000","last_peered":"2024-03-06T07:18:14.718515+0000","last_clean":"2024-03-06T07:18:14.718515+0000","last_became_active":"2024-03-06T07:18:09.530815+0000","last_became_peered":"2024-03-06T07:18:09.530815+0000","last_unstale":"2024-03-06T07:18:14.718515+0000","last_undegraded":"2024-03-06T07:18:14.718515+0000","last_fullsized":"2024-03-06T07:18:14.718515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551688+0000","last_change":"2024-03-06T07:18:13.563305+0000","last_active":"2024-03-06T07:18:14.551688+0000","last_peered":"2024-03-06T07:18:14.551688+0000","last_clean":"2024-03-06T07:18:14.551688+0000","last_became_active":"2024-03-06T07:18:13.562936+0000","last_became_peered":"2024-03-06T07:18:13.562936+0000","last_unstale":"2024-03-06T07:18:14.551688+0000","last_undegraded":"2024-03-06T07:18:14.551688+0000","last_fullsized":"2024-03-06T07:18:14.551688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:02.828593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718486+0000","last_change":"2024-03-06T07:18:08.520455+0000","last_active":"2024-03-06T07:18:14.718486+0000","last_peered":"2024-03-06T07:18:14.718486+0000","last_clean":"2024-03-06T07:18:14.718486+0000","last_became_active":"2024-03-06T07:18:08.520170+0000","last_became_peered":"2024-03-06T07:18:08.520170+0000","last_unstale":"2024-03-06T07:18:14.718486+0000","last_undegraded":"2024-03-06T07:18:14.718486+0000","last_fullsized":"2024-03-06T07:18:14.718486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719134+0000","last_change":"2024-03-06T07:18:09.536183+0000","last_active":"2024-03-06T07:18:14.719134+0000","last_peered":"2024-03-06T07:18:14.719134+0000","last_clean":"2024-03-06T07:18:14.719134+0000","last_became_active":"2024-03-06T07:18:09.535516+0000","last_became_peered":"2024-03-06T07:18:09.535516+0000","last_unstale":"2024-03-06T07:18:14.719134+0000","last_undegraded":"2024-03-06T07:18:14.719134+0000","last_fullsized":"2024-03-06T07:18:14.719134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551978+0000","last_change":"2024-03-06T07:18:13.564931+0000","last_active":"2024-03-06T07:18:14.551978+0000","last_peered":"2024-03-06T07:18:14.551978+0000","last_clean":"2024-03-06T07:18:14.551978+0000","last_became_active":"2024-03-06T07:18:13.564588+0000","last_became_peered":"2024-03-06T07:18:13.564588+0000","last_unstale":"2024-03-06T07:18:14.551978+0000","last_undegraded":"2024-03-06T07:18:14.551978+0000","last_fullsized":"2024-03-06T07:18:14.551978+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:41.700550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550818+0000","last_change":"2024-03-06T07:18:09.516785+0000","last_active":"2024-03-06T07:18:14.550818+0000","last_peered":"2024-03-06T07:18:14.550818+0000","last_clean":"2024-03-06T07:18:14.550818+0000","last_became_active":"2024-03-06T07:18:09.516494+0000","last_became_peered":"2024-03-06T07:18:09.516494+0000","last_unstale":"2024-03-06T07:18:14.550818+0000","last_undegraded":"2024-03-06T07:18:14.550818+0000","last_fullsized":"2024-03-06T07:18:14.550818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719130+0000","last_change":"2024-03-06T07:18:08.523549+0000","last_active":"2024-03-06T07:18:14.719130+0000","last_peered":"2024-03-06T07:18:14.719130+0000","last_clean":"2024-03-06T07:18:14.719130+0000","last_became_active":"2024-03-06T07:18:08.523264+0000","last_became_peered":"2024-03-06T07:18:08.523264+0000","last_unstale":"2024-03-06T07:18:14.719130+0000","last_undegraded":"2024-03-06T07:18:14.719130+0000","last_fullsized":"2024-03-06T07:18:14.719130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:02.351247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552173+0000","last_change":"2024-03-06T07:18:13.563114+0000","last_active":"2024-03-06T07:18:14.552173+0000","last_peered":"2024-03-06T07:18:14.552173+0000","last_clean":"2024-03-06T07:18:14.552173+0000","last_became_active":"2024-03-06T07:18:13.562736+0000","last_became_peered":"2024-03-06T07:18:13.562736+0000","last_unstale":"2024-03-06T07:18:14.552173+0000","last_undegraded":"2024-03-06T07:18:14.552173+0000","last_fullsized":"2024-03-06T07:18:14.552173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:20.335594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720134+0000","last_change":"2024-03-06T07:18:09.524088+0000","last_active":"2024-03-06T07:18:14.720134+0000","last_peered":"2024-03-06T07:18:14.720134+0000","last_clean":"2024-03-06T07:18:14.720134+0000","last_became_active":"2024-03-06T07:18:09.523821+0000","last_became_peered":"2024-03-06T07:18:09.523821+0000","last_unstale":"2024-03-06T07:18:14.720134+0000","last_undegraded":"2024-03-06T07:18:14.720134+0000","last_fullsized":"2024-03-06T07:18:14.720134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719053+0000","last_change":"2024-03-06T07:18:08.516137+0000","last_active":"2024-03-06T07:18:14.719053+0000","last_peered":"2024-03-06T07:18:14.719053+0000","last_clean":"2024-03-06T07:18:14.719053+0000","last_became_active":"2024-03-06T07:18:08.515841+0000","last_became_peered":"2024-03-06T07:18:08.515841+0000","last_unstale":"2024-03-06T07:18:14.719053+0000","last_undegraded":"2024-03-06T07:18:14.719053+0000","last_fullsized":"2024-03-06T07:18:14.719053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551452+0000","last_change":"2024-03-06T07:18:13.564851+0000","last_active":"2024-03-06T07:18:14.551452+0000","last_peered":"2024-03-06T07:18:14.551452+0000","last_clean":"2024-03-06T07:18:14.551452+0000","last_became_active":"2024-03-06T07:18:13.564494+0000","last_became_peered":"2024-03-06T07:18:13.564494+0000","last_unstale":"2024-03-06T07:18:14.551452+0000","last_undegraded":"2024-03-06T07:18:14.551452+0000","last_fullsized":"2024-03-06T07:18:14.551452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:27.757638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720452+0000","last_change":"2024-03-06T07:18:09.527318+0000","last_active":"2024-03-06T07:18:14.720452+0000","last_peered":"2024-03-06T07:18:14.720452+0000","last_clean":"2024-03-06T07:18:14.720452+0000","last_became_active":"2024-03-06T07:18:09.527061+0000","last_became_peered":"2024-03-06T07:18:09.527061+0000","last_unstale":"2024-03-06T07:18:14.720452+0000","last_undegraded":"2024-03-06T07:18:14.720452+0000","last_fullsized":"2024-03-06T07:18:14.720452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719621+0000","last_change":"2024-03-06T07:18:08.515303+0000","last_active":"2024-03-06T07:18:14.719621+0000","last_peered":"2024-03-06T07:18:14.719621+0000","last_clean":"2024-03-06T07:18:14.719621+0000","last_became_active":"2024-03-06T07:18:08.514178+0000","last_became_peered":"2024-03-06T07:18:08.514178+0000","last_unstale":"2024-03-06T07:18:14.719621+0000","last_undegraded":"2024-03-06T07:18:14.719621+0000","last_fullsized":"2024-03-06T07:18:14.719621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719788+0000","last_change":"2024-03-06T07:18:13.788851+0000","last_active":"2024-03-06T07:18:14.719788+0000","last_peered":"2024-03-06T07:18:14.719788+0000","last_clean":"2024-03-06T07:18:14.719788+0000","last_became_active":"2024-03-06T07:18:13.788492+0000","last_became_peered":"2024-03-06T07:18:13.788492+0000","last_unstale":"2024-03-06T07:18:14.719788+0000","last_undegraded":"2024-03-06T07:18:14.719788+0000","last_fullsized":"2024-03-06T07:18:14.719788+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552468+0000","last_change":"2024-03-06T07:18:09.517957+0000","last_active":"2024-03-06T07:18:14.552468+0000","last_peered":"2024-03-06T07:18:14.552468+0000","last_clean":"2024-03-06T07:18:14.552468+0000","last_became_active":"2024-03-06T07:18:09.517197+0000","last_became_peered":"2024-03-06T07:18:09.517197+0000","last_unstale":"2024-03-06T07:18:14.552468+0000","last_undegraded":"2024-03-06T07:18:14.552468+0000","last_fullsized":"2024-03-06T07:18:14.552468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:31.051748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718952+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.718952+0000","last_peered":"2024-03-06T07:18:14.718952+0000","last_clean":"2024-03-06T07:18:14.718952+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.718952+0000","last_undegraded":"2024-03-06T07:18:14.718952+0000","last_fullsized":"2024-03-06T07:18:14.718952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552149+0000","last_change":"2024-03-06T07:18:13.565073+0000","last_active":"2024-03-06T07:18:14.552149+0000","last_peered":"2024-03-06T07:18:14.552149+0000","last_clean":"2024-03-06T07:18:14.552149+0000","last_became_active":"2024-03-06T07:18:13.564708+0000","last_became_peered":"2024-03-06T07:18:13.564708+0000","last_unstale":"2024-03-06T07:18:14.552149+0000","last_undegraded":"2024-03-06T07:18:14.552149+0000","last_fullsized":"2024-03-06T07:18:14.552149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:20.796366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719443+0000","last_change":"2024-03-06T07:18:09.520925+0000","last_active":"2024-03-06T07:18:14.719443+0000","last_peered":"2024-03-06T07:18:14.719443+0000","last_clean":"2024-03-06T07:18:14.719443+0000","last_became_active":"2024-03-06T07:18:09.520658+0000","last_became_peered":"2024-03-06T07:18:09.520658+0000","last_unstale":"2024-03-06T07:18:14.719443+0000","last_undegraded":"2024-03-06T07:18:14.719443+0000","last_fullsized":"2024-03-06T07:18:14.719443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551744+0000","last_change":"2024-03-06T07:18:08.519229+0000","last_active":"2024-03-06T07:18:14.551744+0000","last_peered":"2024-03-06T07:18:14.551744+0000","last_clean":"2024-03-06T07:18:14.551744+0000","last_became_active":"2024-03-06T07:18:08.518936+0000","last_became_peered":"2024-03-06T07:18:08.518936+0000","last_unstale":"2024-03-06T07:18:14.551744+0000","last_undegraded":"2024-03-06T07:18:14.551744+0000","last_fullsized":"2024-03-06T07:18:14.551744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552379+0000","last_change":"2024-03-06T07:18:13.787076+0000","last_active":"2024-03-06T07:18:14.552379+0000","last_peered":"2024-03-06T07:18:14.552379+0000","last_clean":"2024-03-06T07:18:14.552379+0000","last_became_active":"2024-03-06T07:18:13.786733+0000","last_became_peered":"2024-03-06T07:18:13.786733+0000","last_unstale":"2024-03-06T07:18:14.552379+0000","last_undegraded":"2024-03-06T07:18:14.552379+0000","last_fullsized":"2024-03-06T07:18:14.552379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720898+0000","last_change":"2024-03-06T07:18:09.534800+0000","last_active":"2024-03-06T07:18:14.720898+0000","last_peered":"2024-03-06T07:18:14.720898+0000","last_clean":"2024-03-06T07:18:14.720898+0000","last_became_active":"2024-03-06T07:18:09.534408+0000","last_became_peered":"2024-03-06T07:18:09.534408+0000","last_unstale":"2024-03-06T07:18:14.720898+0000","last_undegraded":"2024-03-06T07:18:14.720898+0000","last_fullsized":"2024-03-06T07:18:14.720898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719429+0000","last_change":"2024-03-06T07:18:08.517685+0000","last_active":"2024-03-06T07:18:14.719429+0000","last_peered":"2024-03-06T07:18:14.719429+0000","last_clean":"2024-03-06T07:18:14.719429+0000","last_became_active":"2024-03-06T07:18:08.517423+0000","last_became_peered":"2024-03-06T07:18:08.517423+0000","last_unstale":"2024-03-06T07:18:14.719429+0000","last_undegraded":"2024-03-06T07:18:14.719429+0000","last_fullsized":"2024-03-06T07:18:14.719429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.066441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720202+0000","last_change":"2024-03-06T07:18:13.785863+0000","last_active":"2024-03-06T07:18:14.720202+0000","last_peered":"2024-03-06T07:18:14.720202+0000","last_clean":"2024-03-06T07:18:14.720202+0000","last_became_active":"2024-03-06T07:18:13.785533+0000","last_became_peered":"2024-03-06T07:18:13.785533+0000","last_unstale":"2024-03-06T07:18:14.720202+0000","last_undegraded":"2024-03-06T07:18:14.720202+0000","last_fullsized":"2024-03-06T07:18:14.720202+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:43.382716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718962+0000","last_change":"2024-03-06T07:18:09.523934+0000","last_active":"2024-03-06T07:18:14.718962+0000","last_peered":"2024-03-06T07:18:14.718962+0000","last_clean":"2024-03-06T07:18:14.718962+0000","last_became_active":"2024-03-06T07:18:09.523663+0000","last_became_peered":"2024-03-06T07:18:09.523663+0000","last_unstale":"2024-03-06T07:18:14.718962+0000","last_undegraded":"2024-03-06T07:18:14.718962+0000","last_fullsized":"2024-03-06T07:18:14.718962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718957+0000","last_change":"2024-03-06T07:18:08.520839+0000","last_active":"2024-03-06T07:18:14.718957+0000","last_peered":"2024-03-06T07:18:14.718957+0000","last_clean":"2024-03-06T07:18:14.718957+0000","last_became_active":"2024-03-06T07:18:08.520578+0000","last_became_peered":"2024-03-06T07:18:08.520578+0000","last_unstale":"2024-03-06T07:18:14.718957+0000","last_undegraded":"2024-03-06T07:18:14.718957+0000","last_fullsized":"2024-03-06T07:18:14.718957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:21.462756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718986+0000","last_change":"2024-03-06T07:18:13.787099+0000","last_active":"2024-03-06T07:18:14.718986+0000","last_peered":"2024-03-06T07:18:14.718986+0000","last_clean":"2024-03-06T07:18:14.718986+0000","last_became_active":"2024-03-06T07:18:13.786694+0000","last_became_peered":"2024-03-06T07:18:13.786694+0000","last_unstale":"2024-03-06T07:18:14.718986+0000","last_undegraded":"2024-03-06T07:18:14.718986+0000","last_fullsized":"2024-03-06T07:18:14.718986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719219+0000","last_change":"2024-03-06T07:18:09.531609+0000","last_active":"2024-03-06T07:18:14.719219+0000","last_peered":"2024-03-06T07:18:14.719219+0000","last_clean":"2024-03-06T07:18:14.719219+0000","last_became_active":"2024-03-06T07:18:09.531318+0000","last_became_peered":"2024-03-06T07:18:09.531318+0000","last_unstale":"2024-03-06T07:18:14.719219+0000","last_undegraded":"2024-03-06T07:18:14.719219+0000","last_fullsized":"2024-03-06T07:18:14.719219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719042+0000","last_change":"2024-03-06T07:18:08.516897+0000","last_active":"2024-03-06T07:18:14.719042+0000","last_peered":"2024-03-06T07:18:14.719042+0000","last_clean":"2024-03-06T07:18:14.719042+0000","last_became_active":"2024-03-06T07:18:08.516647+0000","last_became_peered":"2024-03-06T07:18:08.516647+0000","last_unstale":"2024-03-06T07:18:14.719042+0000","last_undegraded":"2024-03-06T07:18:14.719042+0000","last_fullsized":"2024-03-06T07:18:14.719042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718719+0000","last_change":"2024-03-06T07:18:13.564264+0000","last_active":"2024-03-06T07:18:14.718719+0000","last_peered":"2024-03-06T07:18:14.718719+0000","last_clean":"2024-03-06T07:18:14.718719+0000","last_became_active":"2024-03-06T07:18:13.563923+0000","last_became_peered":"2024-03-06T07:18:13.563923+0000","last_unstale":"2024-03-06T07:18:14.718719+0000","last_undegraded":"2024-03-06T07:18:14.718719+0000","last_fullsized":"2024-03-06T07:18:14.718719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552093+0000","last_change":"2024-03-06T07:18:09.519337+0000","last_active":"2024-03-06T07:18:14.552093+0000","last_peered":"2024-03-06T07:18:14.552093+0000","last_clean":"2024-03-06T07:18:14.552093+0000","last_became_active":"2024-03-06T07:18:09.519042+0000","last_became_peered":"2024-03-06T07:18:09.519042+0000","last_unstale":"2024-03-06T07:18:14.552093+0000","last_undegraded":"2024-03-06T07:18:14.552093+0000","last_fullsized":"2024-03-06T07:18:14.552093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:48.003076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718514+0000","last_change":"2024-03-06T07:18:08.521877+0000","last_active":"2024-03-06T07:18:14.718514+0000","last_peered":"2024-03-06T07:18:14.718514+0000","last_clean":"2024-03-06T07:18:14.718514+0000","last_became_active":"2024-03-06T07:18:08.521583+0000","last_became_peered":"2024-03-06T07:18:08.521583+0000","last_unstale":"2024-03-06T07:18:14.718514+0000","last_undegraded":"2024-03-06T07:18:14.718514+0000","last_fullsized":"2024-03-06T07:18:14.718514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718907+0000","last_change":"2024-03-06T07:18:13.566418+0000","last_active":"2024-03-06T07:18:14.718907+0000","last_peered":"2024-03-06T07:18:14.718907+0000","last_clean":"2024-03-06T07:18:14.718907+0000","last_became_active":"2024-03-06T07:18:13.566013+0000","last_became_peered":"2024-03-06T07:18:13.566013+0000","last_unstale":"2024-03-06T07:18:14.718907+0000","last_undegraded":"2024-03-06T07:18:14.718907+0000","last_fullsized":"2024-03-06T07:18:14.718907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719777+0000","last_change":"2024-03-06T07:18:09.522608+0000","last_active":"2024-03-06T07:18:14.719777+0000","last_peered":"2024-03-06T07:18:14.719777+0000","last_clean":"2024-03-06T07:18:14.719777+0000","last_became_active":"2024-03-06T07:18:09.522360+0000","last_became_peered":"2024-03-06T07:18:09.522360+0000","last_unstale":"2024-03-06T07:18:14.719777+0000","last_undegraded":"2024-03-06T07:18:14.719777+0000","last_fullsized":"2024-03-06T07:18:14.719777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548304+0000","last_change":"2024-03-06T07:18:08.512125+0000","last_active":"2024-03-06T07:18:14.548304+0000","last_peered":"2024-03-06T07:18:14.548304+0000","last_clean":"2024-03-06T07:18:14.548304+0000","last_became_active":"2024-03-06T07:18:08.511827+0000","last_became_peered":"2024-03-06T07:18:08.511827+0000","last_unstale":"2024-03-06T07:18:14.548304+0000","last_undegraded":"2024-03-06T07:18:14.548304+0000","last_fullsized":"2024-03-06T07:18:14.548304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719927+0000","last_change":"2024-03-06T07:18:13.573031+0000","last_active":"2024-03-06T07:18:14.719927+0000","last_peered":"2024-03-06T07:18:14.719927+0000","last_clean":"2024-03-06T07:18:14.719927+0000","last_became_active":"2024-03-06T07:18:13.572562+0000","last_became_peered":"2024-03-06T07:18:13.572562+0000","last_unstale":"2024-03-06T07:18:14.719927+0000","last_undegraded":"2024-03-06T07:18:14.719927+0000","last_fullsized":"2024-03-06T07:18:14.719927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:55.484210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719484+0000","last_change":"2024-03-06T07:18:09.518341+0000","last_active":"2024-03-06T07:18:14.719484+0000","last_peered":"2024-03-06T07:18:14.719484+0000","last_clean":"2024-03-06T07:18:14.719484+0000","last_became_active":"2024-03-06T07:18:09.518030+0000","last_became_peered":"2024-03-06T07:18:09.518030+0000","last_unstale":"2024-03-06T07:18:14.719484+0000","last_undegraded":"2024-03-06T07:18:14.719484+0000","last_fullsized":"2024-03-06T07:18:14.719484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:13.874406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718993+0000","last_change":"2024-03-06T07:18:08.522022+0000","last_active":"2024-03-06T07:18:14.718993+0000","last_peered":"2024-03-06T07:18:14.718993+0000","last_clean":"2024-03-06T07:18:14.718993+0000","last_became_active":"2024-03-06T07:18:08.521764+0000","last_became_peered":"2024-03-06T07:18:08.521764+0000","last_unstale":"2024-03-06T07:18:14.718993+0000","last_undegraded":"2024-03-06T07:18:14.718993+0000","last_fullsized":"2024-03-06T07:18:14.718993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719331+0000","last_change":"2024-03-06T07:18:13.785137+0000","last_active":"2024-03-06T07:18:14.719331+0000","last_peered":"2024-03-06T07:18:14.719331+0000","last_clean":"2024-03-06T07:18:14.719331+0000","last_became_active":"2024-03-06T07:18:13.784706+0000","last_became_peered":"2024-03-06T07:18:13.784706+0000","last_unstale":"2024-03-06T07:18:14.719331+0000","last_undegraded":"2024-03-06T07:18:14.719331+0000","last_fullsized":"2024-03-06T07:18:14.719331+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:29.388363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719805+0000","last_change":"2024-03-06T07:18:09.532012+0000","last_active":"2024-03-06T07:18:14.719805+0000","last_peered":"2024-03-06T07:18:14.719805+0000","last_clean":"2024-03-06T07:18:14.719805+0000","last_became_active":"2024-03-06T07:18:09.531724+0000","last_became_peered":"2024-03-06T07:18:09.531724+0000","last_unstale":"2024-03-06T07:18:14.719805+0000","last_undegraded":"2024-03-06T07:18:14.719805+0000","last_fullsized":"2024-03-06T07:18:14.719805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718407+0000","last_change":"2024-03-06T07:18:08.515757+0000","last_active":"2024-03-06T07:18:14.718407+0000","last_peered":"2024-03-06T07:18:14.718407+0000","last_clean":"2024-03-06T07:18:14.718407+0000","last_became_active":"2024-03-06T07:18:08.515494+0000","last_became_peered":"2024-03-06T07:18:08.515494+0000","last_unstale":"2024-03-06T07:18:14.718407+0000","last_undegraded":"2024-03-06T07:18:14.718407+0000","last_fullsized":"2024-03-06T07:18:14.718407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719078+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:18:14.719078+0000","last_peered":"2024-03-06T07:18:14.719078+0000","last_clean":"2024-03-06T07:18:14.719078+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:18:14.719078+0000","last_undegraded":"2024-03-06T07:18:14.719078+0000","last_fullsized":"2024-03-06T07:18:14.719078+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551436+0000","last_change":"2024-03-06T07:18:13.786445+0000","last_active":"2024-03-06T07:18:14.551436+0000","last_peered":"2024-03-06T07:18:14.551436+0000","last_clean":"2024-03-06T07:18:14.551436+0000","last_became_active":"2024-03-06T07:18:13.786085+0000","last_became_peered":"2024-03-06T07:18:13.786085+0000","last_unstale":"2024-03-06T07:18:14.551436+0000","last_undegraded":"2024-03-06T07:18:14.551436+0000","last_fullsized":"2024-03-06T07:18:14.551436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:12.183624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718724+0000","last_change":"2024-03-06T07:18:09.532185+0000","last_active":"2024-03-06T07:18:14.718724+0000","last_peered":"2024-03-06T07:18:14.718724+0000","last_clean":"2024-03-06T07:18:14.718724+0000","last_became_active":"2024-03-06T07:18:09.531883+0000","last_became_peered":"2024-03-06T07:18:09.531883+0000","last_unstale":"2024-03-06T07:18:14.718724+0000","last_undegraded":"2024-03-06T07:18:14.718724+0000","last_fullsized":"2024-03-06T07:18:14.718724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.097215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548537+0000","last_change":"2024-03-06T07:18:08.518305+0000","last_active":"2024-03-06T07:18:14.548537+0000","last_peered":"2024-03-06T07:18:14.548537+0000","last_clean":"2024-03-06T07:18:14.548537+0000","last_became_active":"2024-03-06T07:18:08.517977+0000","last_became_peered":"2024-03-06T07:18:08.517977+0000","last_unstale":"2024-03-06T07:18:14.548537+0000","last_undegraded":"2024-03-06T07:18:14.548537+0000","last_fullsized":"2024-03-06T07:18:14.548537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550763+0000","last_change":"2024-03-06T07:18:13.785349+0000","last_active":"2024-03-06T07:18:14.550763+0000","last_peered":"2024-03-06T07:18:14.550763+0000","last_clean":"2024-03-06T07:18:14.550763+0000","last_became_active":"2024-03-06T07:18:13.784931+0000","last_became_peered":"2024-03-06T07:18:13.784931+0000","last_unstale":"2024-03-06T07:18:14.550763+0000","last_undegraded":"2024-03-06T07:18:14.550763+0000","last_fullsized":"2024-03-06T07:18:14.550763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718607+0000","last_change":"2024-03-06T07:18:09.532114+0000","last_active":"2024-03-06T07:18:14.718607+0000","last_peered":"2024-03-06T07:18:14.718607+0000","last_clean":"2024-03-06T07:18:14.718607+0000","last_became_active":"2024-03-06T07:18:09.531373+0000","last_became_peered":"2024-03-06T07:18:09.531373+0000","last_unstale":"2024-03-06T07:18:14.718607+0000","last_undegraded":"2024-03-06T07:18:14.718607+0000","last_fullsized":"2024-03-06T07:18:14.718607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718980+0000","last_change":"2024-03-06T07:18:08.520471+0000","last_active":"2024-03-06T07:18:14.718980+0000","last_peered":"2024-03-06T07:18:14.718980+0000","last_clean":"2024-03-06T07:18:14.718980+0000","last_became_active":"2024-03-06T07:18:08.520156+0000","last_became_peered":"2024-03-06T07:18:08.520156+0000","last_unstale":"2024-03-06T07:18:14.718980+0000","last_undegraded":"2024-03-06T07:18:14.718980+0000","last_fullsized":"2024-03-06T07:18:14.718980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:50.329508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719199+0000","last_change":"2024-03-06T07:18:13.569753+0000","last_active":"2024-03-06T07:18:14.719199+0000","last_peered":"2024-03-06T07:18:14.719199+0000","last_clean":"2024-03-06T07:18:14.719199+0000","last_became_active":"2024-03-06T07:18:13.569369+0000","last_became_peered":"2024-03-06T07:18:13.569369+0000","last_unstale":"2024-03-06T07:18:14.719199+0000","last_undegraded":"2024-03-06T07:18:14.719199+0000","last_fullsized":"2024-03-06T07:18:14.719199+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719153+0000","last_change":"2024-03-06T07:18:09.536431+0000","last_active":"2024-03-06T07:18:14.719153+0000","last_peered":"2024-03-06T07:18:14.719153+0000","last_clean":"2024-03-06T07:18:14.719153+0000","last_became_active":"2024-03-06T07:18:09.536167+0000","last_became_peered":"2024-03-06T07:18:09.536167+0000","last_unstale":"2024-03-06T07:18:14.719153+0000","last_undegraded":"2024-03-06T07:18:14.719153+0000","last_fullsized":"2024-03-06T07:18:14.719153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.441770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548593+0000","last_change":"2024-03-06T07:18:08.511825+0000","last_active":"2024-03-06T07:18:14.548593+0000","last_peered":"2024-03-06T07:18:14.548593+0000","last_clean":"2024-03-06T07:18:14.548593+0000","last_became_active":"2024-03-06T07:18:08.511363+0000","last_became_peered":"2024-03-06T07:18:08.511363+0000","last_unstale":"2024-03-06T07:18:14.548593+0000","last_undegraded":"2024-03-06T07:18:14.548593+0000","last_fullsized":"2024-03-06T07:18:14.548593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718008+0000","last_change":"2024-03-06T07:18:13.786468+0000","last_active":"2024-03-06T07:18:14.718008+0000","last_peered":"2024-03-06T07:18:14.718008+0000","last_clean":"2024-03-06T07:18:14.718008+0000","last_became_active":"2024-03-06T07:18:13.786129+0000","last_became_peered":"2024-03-06T07:18:13.786129+0000","last_unstale":"2024-03-06T07:18:14.718008+0000","last_undegraded":"2024-03-06T07:18:14.718008+0000","last_fullsized":"2024-03-06T07:18:14.718008+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:54:42.478643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718705+0000","last_change":"2024-03-06T07:18:09.534236+0000","last_active":"2024-03-06T07:18:14.718705+0000","last_peered":"2024-03-06T07:18:14.718705+0000","last_clean":"2024-03-06T07:18:14.718705+0000","last_became_active":"2024-03-06T07:18:09.533090+0000","last_became_peered":"2024-03-06T07:18:09.533090+0000","last_unstale":"2024-03-06T07:18:14.718705+0000","last_undegraded":"2024-03-06T07:18:14.718705+0000","last_fullsized":"2024-03-06T07:18:14.718705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552342+0000","last_change":"2024-03-06T07:18:08.519410+0000","last_active":"2024-03-06T07:18:14.552342+0000","last_peered":"2024-03-06T07:18:14.552342+0000","last_clean":"2024-03-06T07:18:14.552342+0000","last_became_active":"2024-03-06T07:18:08.519115+0000","last_became_peered":"2024-03-06T07:18:08.519115+0000","last_unstale":"2024-03-06T07:18:14.552342+0000","last_undegraded":"2024-03-06T07:18:14.552342+0000","last_fullsized":"2024-03-06T07:18:14.552342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:22.247661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.552289+0000","last_change":"2024-03-06T07:18:13.563952+0000","last_active":"2024-03-06T07:18:14.552289+0000","last_peered":"2024-03-06T07:18:14.552289+0000","last_clean":"2024-03-06T07:18:14.552289+0000","last_became_active":"2024-03-06T07:18:13.563578+0000","last_became_peered":"2024-03-06T07:18:13.563578+0000","last_unstale":"2024-03-06T07:18:14.552289+0000","last_undegraded":"2024-03-06T07:18:14.552289+0000","last_fullsized":"2024-03-06T07:18:14.552289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718657+0000","last_change":"2024-03-06T07:18:09.526419+0000","last_active":"2024-03-06T07:18:14.718657+0000","last_peered":"2024-03-06T07:18:14.718657+0000","last_clean":"2024-03-06T07:18:14.718657+0000","last_became_active":"2024-03-06T07:18:09.526165+0000","last_became_peered":"2024-03-06T07:18:09.526165+0000","last_unstale":"2024-03-06T07:18:14.718657+0000","last_undegraded":"2024-03-06T07:18:14.718657+0000","last_fullsized":"2024-03-06T07:18:14.718657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548110+0000","last_change":"2024-03-06T07:18:08.508444+0000","last_active":"2024-03-06T07:18:14.548110+0000","last_peered":"2024-03-06T07:18:14.548110+0000","last_clean":"2024-03-06T07:18:14.548110+0000","last_became_active":"2024-03-06T07:18:08.508153+0000","last_became_peered":"2024-03-06T07:18:08.508153+0000","last_unstale":"2024-03-06T07:18:14.548110+0000","last_undegraded":"2024-03-06T07:18:14.548110+0000","last_fullsized":"2024-03-06T07:18:14.548110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:27.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719075+0000","last_change":"2024-03-06T07:18:13.567057+0000","last_active":"2024-03-06T07:18:14.719075+0000","last_peered":"2024-03-06T07:18:14.719075+0000","last_clean":"2024-03-06T07:18:14.719075+0000","last_became_active":"2024-03-06T07:18:13.566702+0000","last_became_peered":"2024-03-06T07:18:13.566702+0000","last_unstale":"2024-03-06T07:18:14.719075+0000","last_undegraded":"2024-03-06T07:18:14.719075+0000","last_fullsized":"2024-03-06T07:18:14.719075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550710+0000","last_change":"2024-03-06T07:18:09.520527+0000","last_active":"2024-03-06T07:18:14.550710+0000","last_peered":"2024-03-06T07:18:14.550710+0000","last_clean":"2024-03-06T07:18:14.550710+0000","last_became_active":"2024-03-06T07:18:09.520233+0000","last_became_peered":"2024-03-06T07:18:09.520233+0000","last_unstale":"2024-03-06T07:18:14.550710+0000","last_undegraded":"2024-03-06T07:18:14.550710+0000","last_fullsized":"2024-03-06T07:18:14.550710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550736+0000","last_change":"2024-03-06T07:18:08.521908+0000","last_active":"2024-03-06T07:18:14.550736+0000","last_peered":"2024-03-06T07:18:14.550736+0000","last_clean":"2024-03-06T07:18:14.550736+0000","last_became_active":"2024-03-06T07:18:08.521324+0000","last_became_peered":"2024-03-06T07:18:08.521324+0000","last_unstale":"2024-03-06T07:18:14.550736+0000","last_undegraded":"2024-03-06T07:18:14.550736+0000","last_fullsized":"2024-03-06T07:18:14.550736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:30.047638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720936+0000","last_change":"2024-03-06T07:18:13.788800+0000","last_active":"2024-03-06T07:18:14.720936+0000","last_peered":"2024-03-06T07:18:14.720936+0000","last_clean":"2024-03-06T07:18:14.720936+0000","last_became_active":"2024-03-06T07:18:13.788435+0000","last_became_peered":"2024-03-06T07:18:13.788435+0000","last_unstale":"2024-03-06T07:18:14.720936+0000","last_undegraded":"2024-03-06T07:18:14.720936+0000","last_fullsized":"2024-03-06T07:18:14.720936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:36.842720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719105+0000","last_change":"2024-03-06T07:18:09.527698+0000","last_active":"2024-03-06T07:18:14.719105+0000","last_peered":"2024-03-06T07:18:14.719105+0000","last_clean":"2024-03-06T07:18:14.719105+0000","last_became_active":"2024-03-06T07:18:09.527407+0000","last_became_peered":"2024-03-06T07:18:09.527407+0000","last_unstale":"2024-03-06T07:18:14.719105+0000","last_undegraded":"2024-03-06T07:18:14.719105+0000","last_fullsized":"2024-03-06T07:18:14.719105+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718447+0000","last_change":"2024-03-06T07:18:08.518380+0000","last_active":"2024-03-06T07:18:14.718447+0000","last_peered":"2024-03-06T07:18:14.718447+0000","last_clean":"2024-03-06T07:18:14.718447+0000","last_became_active":"2024-03-06T07:18:08.517571+0000","last_became_peered":"2024-03-06T07:18:08.517571+0000","last_unstale":"2024-03-06T07:18:14.718447+0000","last_undegraded":"2024-03-06T07:18:14.718447+0000","last_fullsized":"2024-03-06T07:18:14.718447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720838+0000","last_change":"2024-03-06T07:18:13.785111+0000","last_active":"2024-03-06T07:18:14.720838+0000","last_peered":"2024-03-06T07:18:14.720838+0000","last_clean":"2024-03-06T07:18:14.720838+0000","last_became_active":"2024-03-06T07:18:13.784751+0000","last_became_peered":"2024-03-06T07:18:13.784751+0000","last_unstale":"2024-03-06T07:18:14.720838+0000","last_undegraded":"2024-03-06T07:18:14.720838+0000","last_fullsized":"2024-03-06T07:18:14.720838+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:55.733802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719140+0000","last_change":"2024-03-06T07:18:09.526243+0000","last_active":"2024-03-06T07:18:14.719140+0000","last_peered":"2024-03-06T07:18:14.719140+0000","last_clean":"2024-03-06T07:18:14.719140+0000","last_became_active":"2024-03-06T07:18:09.525985+0000","last_became_peered":"2024-03-06T07:18:09.525985+0000","last_unstale":"2024-03-06T07:18:14.719140+0000","last_undegraded":"2024-03-06T07:18:14.719140+0000","last_fullsized":"2024-03-06T07:18:14.719140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:20.656356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719309+0000","last_change":"2024-03-06T07:18:08.520844+0000","last_active":"2024-03-06T07:18:14.719309+0000","last_peered":"2024-03-06T07:18:14.719309+0000","last_clean":"2024-03-06T07:18:14.719309+0000","last_became_active":"2024-03-06T07:18:08.520318+0000","last_became_peered":"2024-03-06T07:18:08.520318+0000","last_unstale":"2024-03-06T07:18:14.719309+0000","last_undegraded":"2024-03-06T07:18:14.719309+0000","last_fullsized":"2024-03-06T07:18:14.719309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:28.274132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719091+0000","last_change":"2024-03-06T07:18:13.785334+0000","last_active":"2024-03-06T07:18:14.719091+0000","last_peered":"2024-03-06T07:18:14.719091+0000","last_clean":"2024-03-06T07:18:14.719091+0000","last_became_active":"2024-03-06T07:18:13.784953+0000","last_became_peered":"2024-03-06T07:18:13.784953+0000","last_unstale":"2024-03-06T07:18:14.719091+0000","last_undegraded":"2024-03-06T07:18:14.719091+0000","last_fullsized":"2024-03-06T07:18:14.719091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:18.038907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718773+0000","last_change":"2024-03-06T07:18:09.533771+0000","last_active":"2024-03-06T07:18:14.718773+0000","last_peered":"2024-03-06T07:18:14.718773+0000","last_clean":"2024-03-06T07:18:14.718773+0000","last_became_active":"2024-03-06T07:18:09.533520+0000","last_became_peered":"2024-03-06T07:18:09.533520+0000","last_unstale":"2024-03-06T07:18:14.718773+0000","last_undegraded":"2024-03-06T07:18:14.718773+0000","last_fullsized":"2024-03-06T07:18:14.718773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:18.912751+0000","last_change":"2024-03-06T07:18:08.511813+0000","last_active":"2024-03-06T07:18:18.912751+0000","last_peered":"2024-03-06T07:18:18.912751+0000","last_clean":"2024-03-06T07:18:18.912751+0000","last_became_active":"2024-03-06T07:18:08.511552+0000","last_became_peered":"2024-03-06T07:18:08.511552+0000","last_unstale":"2024-03-06T07:18:18.912751+0000","last_undegraded":"2024-03-06T07:18:18.912751+0000","last_fullsized":"2024-03-06T07:18:18.912751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718960+0000","last_change":"2024-03-06T07:18:13.567483+0000","last_active":"2024-03-06T07:18:14.718960+0000","last_peered":"2024-03-06T07:18:14.718960+0000","last_clean":"2024-03-06T07:18:14.718960+0000","last_became_active":"2024-03-06T07:18:13.566829+0000","last_became_peered":"2024-03-06T07:18:13.566829+0000","last_unstale":"2024-03-06T07:18:14.718960+0000","last_undegraded":"2024-03-06T07:18:14.718960+0000","last_fullsized":"2024-03-06T07:18:14.718960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:21.920867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551865+0000","last_change":"2024-03-06T07:18:09.525404+0000","last_active":"2024-03-06T07:18:14.551865+0000","last_peered":"2024-03-06T07:18:14.551865+0000","last_clean":"2024-03-06T07:18:14.551865+0000","last_became_active":"2024-03-06T07:18:09.525127+0000","last_became_peered":"2024-03-06T07:18:09.525127+0000","last_unstale":"2024-03-06T07:18:14.551865+0000","last_undegraded":"2024-03-06T07:18:14.551865+0000","last_fullsized":"2024-03-06T07:18:14.551865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718876+0000","last_change":"2024-03-06T07:18:08.518033+0000","last_active":"2024-03-06T07:18:14.718876+0000","last_peered":"2024-03-06T07:18:14.718876+0000","last_clean":"2024-03-06T07:18:14.718876+0000","last_became_active":"2024-03-06T07:18:08.517752+0000","last_became_peered":"2024-03-06T07:18:08.517752+0000","last_unstale":"2024-03-06T07:18:14.718876+0000","last_undegraded":"2024-03-06T07:18:14.718876+0000","last_fullsized":"2024-03-06T07:18:14.718876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551959+0000","last_change":"2024-03-06T07:18:13.561392+0000","last_active":"2024-03-06T07:18:14.551959+0000","last_peered":"2024-03-06T07:18:14.551959+0000","last_clean":"2024-03-06T07:18:14.551959+0000","last_became_active":"2024-03-06T07:18:13.561014+0000","last_became_peered":"2024-03-06T07:18:13.561014+0000","last_unstale":"2024-03-06T07:18:14.551959+0000","last_undegraded":"2024-03-06T07:18:14.551959+0000","last_fullsized":"2024-03-06T07:18:14.551959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:43.183715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719832+0000","last_change":"2024-03-06T07:18:09.531541+0000","last_active":"2024-03-06T07:18:14.719832+0000","last_peered":"2024-03-06T07:18:14.719832+0000","last_clean":"2024-03-06T07:18:14.719832+0000","last_became_active":"2024-03-06T07:18:09.531248+0000","last_became_peered":"2024-03-06T07:18:09.531248+0000","last_unstale":"2024-03-06T07:18:14.719832+0000","last_undegraded":"2024-03-06T07:18:14.719832+0000","last_fullsized":"2024-03-06T07:18:14.719832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719237+0000","last_change":"2024-03-06T07:18:08.519552+0000","last_active":"2024-03-06T07:18:14.719237+0000","last_peered":"2024-03-06T07:18:14.719237+0000","last_clean":"2024-03-06T07:18:14.719237+0000","last_became_active":"2024-03-06T07:18:08.519273+0000","last_became_peered":"2024-03-06T07:18:08.519273+0000","last_unstale":"2024-03-06T07:18:14.719237+0000","last_undegraded":"2024-03-06T07:18:14.719237+0000","last_fullsized":"2024-03-06T07:18:14.719237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:56.261150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720395+0000","last_change":"2024-03-06T07:18:13.564953+0000","last_active":"2024-03-06T07:18:14.720395+0000","last_peered":"2024-03-06T07:18:14.720395+0000","last_clean":"2024-03-06T07:18:14.720395+0000","last_became_active":"2024-03-06T07:18:13.564601+0000","last_became_peered":"2024-03-06T07:18:13.564601+0000","last_unstale":"2024-03-06T07:18:14.720395+0000","last_undegraded":"2024-03-06T07:18:14.720395+0000","last_fullsized":"2024-03-06T07:18:14.720395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:31.246872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718301+0000","last_change":"2024-03-06T07:18:09.535125+0000","last_active":"2024-03-06T07:18:14.718301+0000","last_peered":"2024-03-06T07:18:14.718301+0000","last_clean":"2024-03-06T07:18:14.718301+0000","last_became_active":"2024-03-06T07:18:09.534863+0000","last_became_peered":"2024-03-06T07:18:09.534863+0000","last_unstale":"2024-03-06T07:18:14.718301+0000","last_undegraded":"2024-03-06T07:18:14.718301+0000","last_fullsized":"2024-03-06T07:18:14.718301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551568+0000","last_change":"2024-03-06T07:18:08.517122+0000","last_active":"2024-03-06T07:18:14.551568+0000","last_peered":"2024-03-06T07:18:14.551568+0000","last_clean":"2024-03-06T07:18:14.551568+0000","last_became_active":"2024-03-06T07:18:08.516566+0000","last_became_peered":"2024-03-06T07:18:08.516566+0000","last_unstale":"2024-03-06T07:18:14.551568+0000","last_undegraded":"2024-03-06T07:18:14.551568+0000","last_fullsized":"2024-03-06T07:18:14.551568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:46.998965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720688+0000","last_change":"2024-03-06T07:18:13.789985+0000","last_active":"2024-03-06T07:18:14.720688+0000","last_peered":"2024-03-06T07:18:14.720688+0000","last_clean":"2024-03-06T07:18:14.720688+0000","last_became_active":"2024-03-06T07:18:13.789627+0000","last_became_peered":"2024-03-06T07:18:13.789627+0000","last_unstale":"2024-03-06T07:18:14.720688+0000","last_undegraded":"2024-03-06T07:18:14.720688+0000","last_fullsized":"2024-03-06T07:18:14.720688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:22:45.216688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718902+0000","last_change":"2024-03-06T07:18:13.786557+0000","last_active":"2024-03-06T07:18:14.718902+0000","last_peered":"2024-03-06T07:18:14.718902+0000","last_clean":"2024-03-06T07:18:14.718902+0000","last_became_active":"2024-03-06T07:18:13.786179+0000","last_became_peered":"2024-03-06T07:18:13.786179+0000","last_unstale":"2024-03-06T07:18:14.718902+0000","last_undegraded":"2024-03-06T07:18:14.718902+0000","last_fullsized":"2024-03-06T07:18:14.718902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:01.139655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719435+0000","last_change":"2024-03-06T07:18:09.528134+0000","last_active":"2024-03-06T07:18:14.719435+0000","last_peered":"2024-03-06T07:18:14.719435+0000","last_clean":"2024-03-06T07:18:14.719435+0000","last_became_active":"2024-03-06T07:18:09.527831+0000","last_became_peered":"2024-03-06T07:18:09.527831+0000","last_unstale":"2024-03-06T07:18:14.719435+0000","last_undegraded":"2024-03-06T07:18:14.719435+0000","last_fullsized":"2024-03-06T07:18:14.719435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718145+0000","last_change":"2024-03-06T07:18:08.520786+0000","last_active":"2024-03-06T07:18:14.718145+0000","last_peered":"2024-03-06T07:18:14.718145+0000","last_clean":"2024-03-06T07:18:14.718145+0000","last_became_active":"2024-03-06T07:18:08.520392+0000","last_became_peered":"2024-03-06T07:18:08.520392+0000","last_unstale":"2024-03-06T07:18:14.718145+0000","last_undegraded":"2024-03-06T07:18:14.718145+0000","last_fullsized":"2024-03-06T07:18:14.718145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551680+0000","last_change":"2024-03-06T07:18:13.566639+0000","last_active":"2024-03-06T07:18:14.551680+0000","last_peered":"2024-03-06T07:18:14.551680+0000","last_clean":"2024-03-06T07:18:14.551680+0000","last_became_active":"2024-03-06T07:18:13.566321+0000","last_became_peered":"2024-03-06T07:18:13.566321+0000","last_unstale":"2024-03-06T07:18:14.551680+0000","last_undegraded":"2024-03-06T07:18:14.551680+0000","last_fullsized":"2024-03-06T07:18:14.551680+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:35.090355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720366+0000","last_change":"2024-03-06T07:18:09.523023+0000","last_active":"2024-03-06T07:18:14.720366+0000","last_peered":"2024-03-06T07:18:14.720366+0000","last_clean":"2024-03-06T07:18:14.720366+0000","last_became_active":"2024-03-06T07:18:09.522736+0000","last_became_peered":"2024-03-06T07:18:09.522736+0000","last_unstale":"2024-03-06T07:18:14.720366+0000","last_undegraded":"2024-03-06T07:18:14.720366+0000","last_fullsized":"2024-03-06T07:18:14.720366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720831+0000","last_change":"2024-03-06T07:18:08.516225+0000","last_active":"2024-03-06T07:18:14.720831+0000","last_peered":"2024-03-06T07:18:14.720831+0000","last_clean":"2024-03-06T07:18:14.720831+0000","last_became_active":"2024-03-06T07:18:08.515908+0000","last_became_peered":"2024-03-06T07:18:08.515908+0000","last_unstale":"2024-03-06T07:18:14.720831+0000","last_undegraded":"2024-03-06T07:18:14.720831+0000","last_fullsized":"2024-03-06T07:18:14.720831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:24.697679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550914+0000","last_change":"2024-03-06T07:18:08.513484+0000","last_active":"2024-03-06T07:18:14.550914+0000","last_peered":"2024-03-06T07:18:14.550914+0000","last_clean":"2024-03-06T07:18:14.550914+0000","last_became_active":"2024-03-06T07:18:08.513195+0000","last_became_peered":"2024-03-06T07:18:08.513195+0000","last_unstale":"2024-03-06T07:18:14.550914+0000","last_undegraded":"2024-03-06T07:18:14.550914+0000","last_fullsized":"2024-03-06T07:18:14.550914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:20.583200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719247+0000","last_change":"2024-03-06T07:18:09.518972+0000","last_active":"2024-03-06T07:18:14.719247+0000","last_peered":"2024-03-06T07:18:14.719247+0000","last_clean":"2024-03-06T07:18:14.719247+0000","last_became_active":"2024-03-06T07:18:09.518673+0000","last_became_peered":"2024-03-06T07:18:09.518673+0000","last_unstale":"2024-03-06T07:18:14.719247+0000","last_undegraded":"2024-03-06T07:18:14.719247+0000","last_fullsized":"2024-03-06T07:18:14.719247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548431+0000","last_change":"2024-03-06T07:18:13.569839+0000","last_active":"2024-03-06T07:18:14.548431+0000","last_peered":"2024-03-06T07:18:14.548431+0000","last_clean":"2024-03-06T07:18:14.548431+0000","last_became_active":"2024-03-06T07:18:13.569443+0000","last_became_peered":"2024-03-06T07:18:13.569443+0000","last_unstale":"2024-03-06T07:18:14.548431+0000","last_undegraded":"2024-03-06T07:18:14.548431+0000","last_fullsized":"2024-03-06T07:18:14.548431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:45.425994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720231+0000","last_change":"2024-03-06T07:18:08.519883+0000","last_active":"2024-03-06T07:18:14.720231+0000","last_peered":"2024-03-06T07:18:14.720231+0000","last_clean":"2024-03-06T07:18:14.720231+0000","last_became_active":"2024-03-06T07:18:08.519579+0000","last_became_peered":"2024-03-06T07:18:08.519579+0000","last_unstale":"2024-03-06T07:18:14.720231+0000","last_undegraded":"2024-03-06T07:18:14.720231+0000","last_fullsized":"2024-03-06T07:18:14.720231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:07.492920+0000","last_clean_scrub_stamp":"2024-03-06T07:18:07.492920+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:49.448724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551887+0000","last_change":"2024-03-06T07:18:09.520603+0000","last_active":"2024-03-06T07:18:14.551887+0000","last_peered":"2024-03-06T07:18:14.551887+0000","last_clean":"2024-03-06T07:18:14.551887+0000","last_became_active":"2024-03-06T07:18:09.520258+0000","last_became_peered":"2024-03-06T07:18:09.520258+0000","last_unstale":"2024-03-06T07:18:14.551887+0000","last_undegraded":"2024-03-06T07:18:14.551887+0000","last_fullsized":"2024-03-06T07:18:14.551887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720637+0000","last_change":"2024-03-06T07:18:13.564140+0000","last_active":"2024-03-06T07:18:14.720637+0000","last_peered":"2024-03-06T07:18:14.720637+0000","last_clean":"2024-03-06T07:18:14.720637+0000","last_became_active":"2024-03-06T07:18:13.563770+0000","last_became_peered":"2024-03-06T07:18:13.563770+0000","last_unstale":"2024-03-06T07:18:14.720637+0000","last_undegraded":"2024-03-06T07:18:14.720637+0000","last_fullsized":"2024-03-06T07:18:14.720637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:12.539471+0000","last_clean_scrub_stamp":"2024-03-06T07:18:12.539471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:03.956217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.550619+0000","last_change":"2024-03-06T07:18:09.520176+0000","last_active":"2024-03-06T07:18:14.550619+0000","last_peered":"2024-03-06T07:18:14.550619+0000","last_clean":"2024-03-06T07:18:14.550619+0000","last_became_active":"2024-03-06T07:18:09.519911+0000","last_became_peered":"2024-03-06T07:18:09.519911+0000","last_unstale":"2024-03-06T07:18:14.550619+0000","last_undegraded":"2024-03-06T07:18:14.550619+0000","last_fullsized":"2024-03-06T07:18:14.550619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548193+0000","last_change":"2024-03-06T07:18:09.524681+0000","last_active":"2024-03-06T07:18:14.548193+0000","last_peered":"2024-03-06T07:18:14.548193+0000","last_clean":"2024-03-06T07:18:14.548193+0000","last_became_active":"2024-03-06T07:18:09.524427+0000","last_became_peered":"2024-03-06T07:18:09.524427+0000","last_unstale":"2024-03-06T07:18:14.548193+0000","last_undegraded":"2024-03-06T07:18:14.548193+0000","last_fullsized":"2024-03-06T07:18:14.548193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:51.333618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.531928+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531670+0000","last_became_peered":"2024-03-06T07:18:09.531670+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:39.141275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719143+0000","last_change":"2024-03-06T07:18:09.523431+0000","last_active":"2024-03-06T07:18:14.719143+0000","last_peered":"2024-03-06T07:18:14.719143+0000","last_clean":"2024-03-06T07:18:14.719143+0000","last_became_active":"2024-03-06T07:18:09.523169+0000","last_became_peered":"2024-03-06T07:18:09.523169+0000","last_unstale":"2024-03-06T07:18:14.719143+0000","last_undegraded":"2024-03-06T07:18:14.719143+0000","last_fullsized":"2024-03-06T07:18:14.719143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720897+0000","last_change":"2024-03-06T07:18:09.525359+0000","last_active":"2024-03-06T07:18:14.720897+0000","last_peered":"2024-03-06T07:18:14.720897+0000","last_clean":"2024-03-06T07:18:14.720897+0000","last_became_active":"2024-03-06T07:18:09.525069+0000","last_became_peered":"2024-03-06T07:18:09.525069+0000","last_unstale":"2024-03-06T07:18:14.720897+0000","last_undegraded":"2024-03-06T07:18:14.720897+0000","last_fullsized":"2024-03-06T07:18:14.720897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:37.658110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718709+0000","last_change":"2024-03-06T07:18:09.528783+0000","last_active":"2024-03-06T07:18:14.718709+0000","last_peered":"2024-03-06T07:18:14.718709+0000","last_clean":"2024-03-06T07:18:14.718709+0000","last_became_active":"2024-03-06T07:18:09.528488+0000","last_became_peered":"2024-03-06T07:18:09.528488+0000","last_unstale":"2024-03-06T07:18:14.718709+0000","last_undegraded":"2024-03-06T07:18:14.718709+0000","last_fullsized":"2024-03-06T07:18:14.718709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551739+0000","last_change":"2024-03-06T07:18:09.526930+0000","last_active":"2024-03-06T07:18:14.551739+0000","last_peered":"2024-03-06T07:18:14.551739+0000","last_clean":"2024-03-06T07:18:14.551739+0000","last_became_active":"2024-03-06T07:18:09.526575+0000","last_became_peered":"2024-03-06T07:18:09.526575+0000","last_unstale":"2024-03-06T07:18:14.551739+0000","last_undegraded":"2024-03-06T07:18:14.551739+0000","last_fullsized":"2024-03-06T07:18:14.551739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720809+0000","last_change":"2024-03-06T07:18:09.532541+0000","last_active":"2024-03-06T07:18:14.720809+0000","last_peered":"2024-03-06T07:18:14.720809+0000","last_clean":"2024-03-06T07:18:14.720809+0000","last_became_active":"2024-03-06T07:18:09.532260+0000","last_became_peered":"2024-03-06T07:18:09.532260+0000","last_unstale":"2024-03-06T07:18:14.720809+0000","last_undegraded":"2024-03-06T07:18:14.720809+0000","last_fullsized":"2024-03-06T07:18:14.720809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718061+0000","last_change":"2024-03-06T07:18:09.530812+0000","last_active":"2024-03-06T07:18:14.718061+0000","last_peered":"2024-03-06T07:18:14.718061+0000","last_clean":"2024-03-06T07:18:14.718061+0000","last_became_active":"2024-03-06T07:18:09.530493+0000","last_became_peered":"2024-03-06T07:18:09.530493+0000","last_unstale":"2024-03-06T07:18:14.718061+0000","last_undegraded":"2024-03-06T07:18:14.718061+0000","last_fullsized":"2024-03-06T07:18:14.718061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720741+0000","last_change":"2024-03-06T07:18:09.525260+0000","last_active":"2024-03-06T07:18:14.720741+0000","last_peered":"2024-03-06T07:18:14.720741+0000","last_clean":"2024-03-06T07:18:14.720741+0000","last_became_active":"2024-03-06T07:18:09.524996+0000","last_became_peered":"2024-03-06T07:18:09.524996+0000","last_unstale":"2024-03-06T07:18:14.720741+0000","last_undegraded":"2024-03-06T07:18:14.720741+0000","last_fullsized":"2024-03-06T07:18:14.720741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:54.685094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718326+0000","last_change":"2024-03-06T07:18:09.539784+0000","last_active":"2024-03-06T07:18:14.718326+0000","last_peered":"2024-03-06T07:18:14.718326+0000","last_clean":"2024-03-06T07:18:14.718326+0000","last_became_active":"2024-03-06T07:18:09.538642+0000","last_became_peered":"2024-03-06T07:18:09.538642+0000","last_unstale":"2024-03-06T07:18:14.718326+0000","last_undegraded":"2024-03-06T07:18:14.718326+0000","last_fullsized":"2024-03-06T07:18:14.718326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719031+0000","last_change":"2024-03-06T07:18:09.526156+0000","last_active":"2024-03-06T07:18:14.719031+0000","last_peered":"2024-03-06T07:18:14.719031+0000","last_clean":"2024-03-06T07:18:14.719031+0000","last_became_active":"2024-03-06T07:18:09.525877+0000","last_became_peered":"2024-03-06T07:18:09.525877+0000","last_unstale":"2024-03-06T07:18:14.719031+0000","last_undegraded":"2024-03-06T07:18:14.719031+0000","last_fullsized":"2024-03-06T07:18:14.719031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551188+0000","last_change":"2024-03-06T07:18:09.516685+0000","last_active":"2024-03-06T07:18:14.551188+0000","last_peered":"2024-03-06T07:18:14.551188+0000","last_clean":"2024-03-06T07:18:14.551188+0000","last_became_active":"2024-03-06T07:18:09.516377+0000","last_became_peered":"2024-03-06T07:18:09.516377+0000","last_unstale":"2024-03-06T07:18:14.551188+0000","last_undegraded":"2024-03-06T07:18:14.551188+0000","last_fullsized":"2024-03-06T07:18:14.551188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718906+0000","last_change":"2024-03-06T07:18:09.524856+0000","last_active":"2024-03-06T07:18:14.718906+0000","last_peered":"2024-03-06T07:18:14.718906+0000","last_clean":"2024-03-06T07:18:14.718906+0000","last_became_active":"2024-03-06T07:18:09.523916+0000","last_became_peered":"2024-03-06T07:18:09.523916+0000","last_unstale":"2024-03-06T07:18:14.718906+0000","last_undegraded":"2024-03-06T07:18:14.718906+0000","last_fullsized":"2024-03-06T07:18:14.718906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:08.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.548544+0000","last_change":"2024-03-06T07:18:09.518046+0000","last_active":"2024-03-06T07:18:14.548544+0000","last_peered":"2024-03-06T07:18:14.548544+0000","last_clean":"2024-03-06T07:18:14.548544+0000","last_became_active":"2024-03-06T07:18:09.517756+0000","last_became_peered":"2024-03-06T07:18:09.517756+0000","last_unstale":"2024-03-06T07:18:14.548544+0000","last_undegraded":"2024-03-06T07:18:14.548544+0000","last_fullsized":"2024-03-06T07:18:14.548544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:29.070551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719287+0000","last_change":"2024-03-06T07:18:09.537036+0000","last_active":"2024-03-06T07:18:14.719287+0000","last_peered":"2024-03-06T07:18:14.719287+0000","last_clean":"2024-03-06T07:18:14.719287+0000","last_became_active":"2024-03-06T07:18:09.536576+0000","last_became_peered":"2024-03-06T07:18:09.536576+0000","last_unstale":"2024-03-06T07:18:14.719287+0000","last_undegraded":"2024-03-06T07:18:14.719287+0000","last_fullsized":"2024-03-06T07:18:14.719287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:17.878427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.717960+0000","last_change":"2024-03-06T07:18:09.527339+0000","last_active":"2024-03-06T07:18:14.717960+0000","last_peered":"2024-03-06T07:18:14.717960+0000","last_clean":"2024-03-06T07:18:14.717960+0000","last_became_active":"2024-03-06T07:18:09.526873+0000","last_became_peered":"2024-03-06T07:18:09.526873+0000","last_unstale":"2024-03-06T07:18:14.717960+0000","last_undegraded":"2024-03-06T07:18:14.717960+0000","last_fullsized":"2024-03-06T07:18:14.717960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:44.629290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719530+0000","last_change":"2024-03-06T07:18:09.522712+0000","last_active":"2024-03-06T07:18:14.719530+0000","last_peered":"2024-03-06T07:18:14.719530+0000","last_clean":"2024-03-06T07:18:14.719530+0000","last_became_active":"2024-03-06T07:18:09.522433+0000","last_became_peered":"2024-03-06T07:18:09.522433+0000","last_unstale":"2024-03-06T07:18:14.719530+0000","last_undegraded":"2024-03-06T07:18:14.719530+0000","last_fullsized":"2024-03-06T07:18:14.719530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719097+0000","last_change":"2024-03-06T07:18:09.522057+0000","last_active":"2024-03-06T07:18:14.719097+0000","last_peered":"2024-03-06T07:18:14.719097+0000","last_clean":"2024-03-06T07:18:14.719097+0000","last_became_active":"2024-03-06T07:18:09.521270+0000","last_became_peered":"2024-03-06T07:18:09.521270+0000","last_unstale":"2024-03-06T07:18:14.719097+0000","last_undegraded":"2024-03-06T07:18:14.719097+0000","last_fullsized":"2024-03-06T07:18:14.719097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:57.186351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720044+0000","last_change":"2024-03-06T07:18:09.526002+0000","last_active":"2024-03-06T07:18:14.720044+0000","last_peered":"2024-03-06T07:18:14.720044+0000","last_clean":"2024-03-06T07:18:14.720044+0000","last_became_active":"2024-03-06T07:18:09.525717+0000","last_became_peered":"2024-03-06T07:18:09.525717+0000","last_unstale":"2024-03-06T07:18:14.720044+0000","last_undegraded":"2024-03-06T07:18:14.720044+0000","last_fullsized":"2024-03-06T07:18:14.720044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551860+0000","last_change":"2024-03-06T07:18:09.521766+0000","last_active":"2024-03-06T07:18:14.551860+0000","last_peered":"2024-03-06T07:18:14.551860+0000","last_clean":"2024-03-06T07:18:14.551860+0000","last_became_active":"2024-03-06T07:18:09.521159+0000","last_became_peered":"2024-03-06T07:18:09.521159+0000","last_unstale":"2024-03-06T07:18:14.551860+0000","last_undegraded":"2024-03-06T07:18:14.551860+0000","last_fullsized":"2024-03-06T07:18:14.551860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:57.265260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551251+0000","last_change":"2024-03-06T07:18:09.533157+0000","last_active":"2024-03-06T07:18:14.551251+0000","last_peered":"2024-03-06T07:18:14.551251+0000","last_clean":"2024-03-06T07:18:14.551251+0000","last_became_active":"2024-03-06T07:18:09.532868+0000","last_became_peered":"2024-03-06T07:18:09.532868+0000","last_unstale":"2024-03-06T07:18:14.551251+0000","last_undegraded":"2024-03-06T07:18:14.551251+0000","last_fullsized":"2024-03-06T07:18:14.551251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:12:05.128360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.551648+0000","last_change":"2024-03-06T07:18:09.532473+0000","last_active":"2024-03-06T07:18:14.551648+0000","last_peered":"2024-03-06T07:18:14.551648+0000","last_clean":"2024-03-06T07:18:14.551648+0000","last_became_active":"2024-03-06T07:18:09.532191+0000","last_became_peered":"2024-03-06T07:18:09.532191+0000","last_unstale":"2024-03-06T07:18:14.551648+0000","last_undegraded":"2024-03-06T07:18:14.551648+0000","last_fullsized":"2024-03-06T07:18:14.551648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:11.222623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719062+0000","last_change":"2024-03-06T07:18:09.533061+0000","last_active":"2024-03-06T07:18:14.719062+0000","last_peered":"2024-03-06T07:18:14.719062+0000","last_clean":"2024-03-06T07:18:14.719062+0000","last_became_active":"2024-03-06T07:18:09.532742+0000","last_became_peered":"2024-03-06T07:18:09.532742+0000","last_unstale":"2024-03-06T07:18:14.719062+0000","last_undegraded":"2024-03-06T07:18:14.719062+0000","last_fullsized":"2024-03-06T07:18:14.719062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:25.345923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.720283+0000","last_change":"2024-03-06T07:18:09.532342+0000","last_active":"2024-03-06T07:18:14.720283+0000","last_peered":"2024-03-06T07:18:14.720283+0000","last_clean":"2024-03-06T07:18:14.720283+0000","last_became_active":"2024-03-06T07:18:09.532062+0000","last_became_peered":"2024-03-06T07:18:09.532062+0000","last_unstale":"2024-03-06T07:18:14.720283+0000","last_undegraded":"2024-03-06T07:18:14.720283+0000","last_fullsized":"2024-03-06T07:18:14.720283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:57.235362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718610+0000","last_change":"2024-03-06T07:18:09.525768+0000","last_active":"2024-03-06T07:18:14.718610+0000","last_peered":"2024-03-06T07:18:14.718610+0000","last_clean":"2024-03-06T07:18:14.718610+0000","last_became_active":"2024-03-06T07:18:09.524604+0000","last_became_peered":"2024-03-06T07:18:09.524604+0000","last_unstale":"2024-03-06T07:18:14.718610+0000","last_undegraded":"2024-03-06T07:18:14.718610+0000","last_fullsized":"2024-03-06T07:18:14.718610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:58.786153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718219+0000","last_change":"2024-03-06T07:18:09.531193+0000","last_active":"2024-03-06T07:18:14.718219+0000","last_peered":"2024-03-06T07:18:14.718219+0000","last_clean":"2024-03-06T07:18:14.718219+0000","last_became_active":"2024-03-06T07:18:09.530931+0000","last_became_peered":"2024-03-06T07:18:09.530931+0000","last_unstale":"2024-03-06T07:18:14.718219+0000","last_undegraded":"2024-03-06T07:18:14.718219+0000","last_fullsized":"2024-03-06T07:18:14.718219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718236+0000","last_change":"2024-03-06T07:18:09.524492+0000","last_active":"2024-03-06T07:18:14.718236+0000","last_peered":"2024-03-06T07:18:14.718236+0000","last_clean":"2024-03-06T07:18:14.718236+0000","last_became_active":"2024-03-06T07:18:09.524202+0000","last_became_peered":"2024-03-06T07:18:09.524202+0000","last_unstale":"2024-03-06T07:18:14.718236+0000","last_undegraded":"2024-03-06T07:18:14.718236+0000","last_fullsized":"2024-03-06T07:18:14.718236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:16.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718408+0000","last_change":"2024-03-06T07:18:09.532118+0000","last_active":"2024-03-06T07:18:14.718408+0000","last_peered":"2024-03-06T07:18:14.718408+0000","last_clean":"2024-03-06T07:18:14.718408+0000","last_became_active":"2024-03-06T07:18:09.531771+0000","last_became_peered":"2024-03-06T07:18:09.531771+0000","last_unstale":"2024-03-06T07:18:14.718408+0000","last_undegraded":"2024-03-06T07:18:14.718408+0000","last_fullsized":"2024-03-06T07:18:14.718408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:16.293154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719327+0000","last_change":"2024-03-06T07:18:09.519755+0000","last_active":"2024-03-06T07:18:14.719327+0000","last_peered":"2024-03-06T07:18:14.719327+0000","last_clean":"2024-03-06T07:18:14.719327+0000","last_became_active":"2024-03-06T07:18:09.519475+0000","last_became_peered":"2024-03-06T07:18:09.519475+0000","last_unstale":"2024-03-06T07:18:14.719327+0000","last_undegraded":"2024-03-06T07:18:14.719327+0000","last_fullsized":"2024-03-06T07:18:14.719327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.759070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.719345+0000","last_change":"2024-03-06T07:18:09.524863+0000","last_active":"2024-03-06T07:18:14.719345+0000","last_peered":"2024-03-06T07:18:14.719345+0000","last_clean":"2024-03-06T07:18:14.719345+0000","last_became_active":"2024-03-06T07:18:09.524588+0000","last_became_peered":"2024-03-06T07:18:09.524588+0000","last_unstale":"2024-03-06T07:18:14.719345+0000","last_undegraded":"2024-03-06T07:18:14.719345+0000","last_fullsized":"2024-03-06T07:18:14.719345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:23.715198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-06T07:18:14.718889+0000","last_change":"2024-03-06T07:18:09.528727+0000","last_active":"2024-03-06T07:18:14.718889+0000","last_peered":"2024-03-06T07:18:14.718889+0000","last_clean":"2024-03-06T07:18:14.718889+0000","last_became_active":"2024-03-06T07:18:09.528474+0000","last_became_peered":"2024-03-06T07:18:09.528474+0000","last_unstale":"2024-03-06T07:18:14.718889+0000","last_undegraded":"2024-03-06T07:18:14.718889+0000","last_fullsized":"2024-03-06T07:18:14.718889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:08.497031+0000","last_clean_scrub_stamp":"2024-03-06T07:18:08.497031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.047915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:18:35.683 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:18:35.683 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:18:35.683 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:18:35.684 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:18:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.816+0000 7fbe75f77640 1 Processor -- start 2024-03-06T07:18:35.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.816+0000 7fbe75f77640 1 -- start start 2024-03-06T07:18:35.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe75f77640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbe7014f790 0x7fbe701529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe75f77640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe70155a40 0x7fbe70155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe75f77640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe701563d0 con 0x7fbe700c1aa0 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe75f77640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe700c2680 con 0x7fbe7014f790 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe75f77640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe700c28d0 con 0x7fbe70155a40 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbe7014f790 0x7fbe701529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbe700c1aa0 0x7fbe700c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41536/0 (socket says 172.21.15.94:41536) 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6effd640 1 -- 172.21.15.94:0/3585906809 learned_addr learned my addr 172.21.15.94:0/3585906809 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6ffff640 1 --2- 172.21.15.94:0/3585906809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe70155a40 0x7fbe70155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6f7fe640 1 -- 172.21.15.94:0/3585906809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe70155a40 msgr2=0x7fbe70155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6f7fe640 1 --2- 172.21.15.94:0/3585906809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe70155a40 0x7fbe70155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6f7fe640 1 -- 172.21.15.94:0/3585906809 >> v1:172.21.15.94:6789/0 conn(0x7fbe700c1aa0 legacy=0x7fbe700c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6f7fe640 1 -- 172.21.15.94:0/3585906809 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700c2bb0 con 0x7fbe7014f790 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.817+0000 7fbe6e7fc640 1 -- 172.21.15.94:0/3585906809 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573915105 0 0) 0x7fbe701563d0 con 0x7fbe700c1aa0 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.818+0000 7fbe6f7fe640 1 --2- 172.21.15.94:0/3585906809 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7014f790 0x7fbe701529d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbe600097a0 tx=0x7fbe6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70530259723435d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.818+0000 7fbe6e7fc640 1 -- 172.21.15.94:0/3585906809 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60035020 con 0x7fbe7014f790 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.818+0000 7fbe6e7fc640 1 -- 172.21.15.94:0/3585906809 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe60031830 con 0x7fbe7014f790 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.818+0000 7fbe6e7fc640 1 -- 172.21.15.94:0/3585906809 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60031c00 con 0x7fbe7014f790 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 -- 172.21.15.94:0/3585906809 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7014f790 msgr2=0x7fbe701529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 --2- 172.21.15.94:0/3585906809 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7014f790 0x7fbe701529d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbe600097a0 tx=0x7fbe6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 -- 172.21.15.94:0/3585906809 shutdown_connections 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 --2- 172.21.15.94:0/3585906809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe70155a40 0x7fbe70155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 --2- 172.21.15.94:0/3585906809 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe7014f790 0x7fbe701529d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 -- 172.21.15.94:0/3585906809 >> 172.21.15.94:0/3585906809 conn(0x7fbe700c6710 msgr2=0x7fbe700c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:35.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 -- 172.21.15.94:0/3585906809 shutdown_connections 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.819+0000 7fbe75f77640 1 -- 172.21.15.94:0/3585906809 wait complete. 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.820+0000 7fbe75f77640 1 Processor -- start 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.820+0000 7fbe75f77640 1 -- start start 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe75f77640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe700c1aa0 0x7fbe701dfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe75f77640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7014f790 0x7fbe70070d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe75f77640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe70155a40 0x7fbe70071260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe75f77640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe700c0d60 con 0x7fbe70155a40 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe75f77640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70155520 con 0x7fbe7014f790 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe75f77640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe700c5e30 con 0x7fbe700c1aa0 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe70155a40 0x7fbe70071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe70155a40 0x7fbe70071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41128/0 (socket says 172.21.15.94:41128) 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6f7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe700c1aa0 0x7fbe701dfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 -- 172.21.15.94:0/1007564456 learned_addr learned my addr 172.21.15.94:0/1007564456 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:35.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6effd640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7014f790 0x7fbe70070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 -- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe700c1aa0 msgr2=0x7fbe701dfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe700c1aa0 0x7fbe701dfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 -- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7014f790 msgr2=0x7fbe70070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7014f790 0x7fbe70070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 -- 172.21.15.94:0/1007564456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe701510d0 con 0x7fbe70155a40 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6f7fe640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe700c1aa0 0x7fbe701dfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe6ffff640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe70155a40 0x7fbe70071260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009d30 tx=0x7fbe6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe57fff640 1 -- 172.21.15.94:0/1007564456 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68018020 con 0x7fbe70155a40 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe57fff640 1 -- 172.21.15.94:0/1007564456 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe68014480 con 0x7fbe70155a40 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70072c50 con 0x7fbe70155a40 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.821+0000 7fbe57fff640 1 -- 172.21.15.94:0/1007564456 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68014870 con 0x7fbe70155a40 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.822+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe70073210 con 0x7fbe70155a40 2024-03-06T07:18:35.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.822+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe70073450 con 0x7fbe70155a40 2024-03-06T07:18:35.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.823+0000 7fbe57fff640 1 -- 172.21.15.94:0/1007564456 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe68005d10 con 0x7fbe70155a40 2024-03-06T07:18:35.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.823+0000 7fbe57fff640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe5c06b4d0 0x7fbe5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:35.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.823+0000 7fbe57fff640 1 -- 172.21.15.94:0/1007564456 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbe6808ae70 con 0x7fbe70155a40 2024-03-06T07:18:35.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.823+0000 7fbe6f7fe640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe5c06b4d0 0x7fbe5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:35.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.824+0000 7fbe6f7fe640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe5c06b4d0 0x7fbe5c06d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe60036000 tx=0x7fbe60008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:35.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:35.827+0000 7fbe57fff640 1 -- 172.21.15.94:0/1007564456 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe70073450 con 0x7fbe70155a40 2024-03-06T07:18:36.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.035+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe7004f330 con 0x7fbe70155a40 2024-03-06T07:18:36.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.036+0000 7fbe57fff640 1 -- 172.21.15.94:0/1007564456 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbe68022070 con 0x7fbe70155a40 2024-03-06T07:18:36.038 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:36.038 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:18:36.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe5c06b4d0 msgr2=0x7fbe5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe5c06b4d0 0x7fbe5c06d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe60036000 tx=0x7fbe60008030 comp rx=0 tx=0).stop 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe70155a40 msgr2=0x7fbe70071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe70155a40 0x7fbe70071260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009d30 tx=0x7fbe6800c490 comp rx=0 tx=0).stop 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 shutdown_connections 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe5c06b4d0 0x7fbe5c06d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe70155a40 0x7fbe70071260 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7014f790 0x7fbe70070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 --2- 172.21.15.94:0/1007564456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe700c1aa0 0x7fbe701dfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 >> 172.21.15.94:0/1007564456 conn(0x7fbe700c6710 msgr2=0x7fbe700a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:36.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 shutdown_connections 2024-03-06T07:18:36.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.038+0000 7fbe75f77640 1 -- 172.21.15.94:0/1007564456 wait complete. 2024-03-06T07:18:36.051 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:18:36.051 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:36.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.148+0000 7fc4653ae640 1 Processor -- start 2024-03-06T07:18:36.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc4653ae640 1 -- start start 2024-03-06T07:18:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc4653ae640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc46014f360 0x7fc460155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc4653ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc460158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc4653ae640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4600c73d0 con 0x7fc4600c88c0 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc4653ae640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4600c8dd0 con 0x7fc46014f360 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc4653ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600be2e0 con 0x7fc460158b60 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc45effd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc46014f360 0x7fc460155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc45f7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc460158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc45effd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc46014f360 0x7fc460155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:41900/0 (socket says 172.21.15.94:41900) 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc45effd640 1 -- 172.21.15.94:0/559923354 learned_addr learned my addr 172.21.15.94:0/559923354 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.149+0000 7fc45e7fc640 1 --1- 172.21.15.94:0/559923354 >> v1:172.21.15.145:6789/0 conn(0x7fc4600c88c0 0x7fc4600c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43884/0 (socket says 172.21.15.94:43884) 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.150+0000 7fc45effd640 1 -- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 msgr2=0x7fc460158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.150+0000 7fc45effd640 1 --2- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc460158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.150+0000 7fc45effd640 1 -- 172.21.15.94:0/559923354 >> v1:172.21.15.145:6789/0 conn(0x7fc4600c88c0 legacy=0x7fc4600c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.150+0000 7fc45effd640 1 -- 172.21.15.94:0/559923354 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600c7670 con 0x7fc46014f360 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.150+0000 7fc45f7fe640 1 --2- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc460158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.150+0000 7fc45effd640 1 --2- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc46014f360 0x7fc460155c10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc458009870 tx=0x7fc45802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f06f33a467a025c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc45dffb640 1 -- 172.21.15.94:0/559923354 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458035020 con 0x7fc46014f360 2024-03-06T07:18:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc45dffb640 1 -- 172.21.15.94:0/559923354 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458031640 con 0x7fc46014f360 2024-03-06T07:18:36.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc45dffb640 1 -- 172.21.15.94:0/559923354 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45802d040 con 0x7fc46014f360 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 -- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc46014f360 msgr2=0x7fc460155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 --2- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc46014f360 0x7fc460155c10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc458009870 tx=0x7fc45802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 -- 172.21.15.94:0/559923354 shutdown_connections 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 --2- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc460158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 --2- 172.21.15.94:0/559923354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc46014f360 0x7fc460155c10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 -- 172.21.15.94:0/559923354 >> 172.21.15.94:0/559923354 conn(0x7fc4600ac570 msgr2=0x7fc4600ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 -- 172.21.15.94:0/559923354 shutdown_connections 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.151+0000 7fc4653ae640 1 -- 172.21.15.94:0/559923354 wait complete. 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.152+0000 7fc4653ae640 1 Processor -- start 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc4653ae640 1 -- start start 2024-03-06T07:18:36.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc4653ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4600c88c0 0x7fc4600bf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc4653ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc46014f360 0x7fc4601ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc4653ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc4601eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc4653ae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600c5a50 con 0x7fc46014f360 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc4653ae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600c7c30 con 0x7fc4600c88c0 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc4653ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600c5bd0 con 0x7fc460158b60 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc45e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc46014f360 0x7fc4601ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc45e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc46014f360 0x7fc4601ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41132/0 (socket says 172.21.15.94:41132) 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc45e7fc640 1 -- 172.21.15.94:0/3324494365 learned_addr learned my addr 172.21.15.94:0/3324494365 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc45effd640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4600c88c0 0x7fc4600bf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.153+0000 7fc45f7fe640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc4601eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc45e7fc640 1 -- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 msgr2=0x7fc4601eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc45e7fc640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc4601eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc45e7fc640 1 -- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4600c88c0 msgr2=0x7fc4600bf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc45e7fc640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4600c88c0 0x7fc4600bf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc45e7fc640 1 -- 172.21.15.94:0/3324494365 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600bdfe0 con 0x7fc46014f360 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc45effd640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4600c88c0 0x7fc4600bf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc45e7fc640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc46014f360 0x7fc4601ee4e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc44800c470 tx=0x7fc44800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc4577fe640 1 -- 172.21.15.94:0/3324494365 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4480074e0 con 0x7fc46014f360 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc4577fe640 1 -- 172.21.15.94:0/3324494365 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448007680 con 0x7fc46014f360 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc4577fe640 1 -- 172.21.15.94:0/3324494365 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448004a00 con 0x7fc46014f360 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4601ef1b0 con 0x7fc46014f360 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.154+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4600c4d20 con 0x7fc46014f360 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.155+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc460068ea0 con 0x7fc46014f360 2024-03-06T07:18:36.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.155+0000 7fc4577fe640 1 -- 172.21.15.94:0/3324494365 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc44801a020 con 0x7fc46014f360 2024-03-06T07:18:36.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.156+0000 7fc4577fe640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc44006b4d0 0x7fc44006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.156+0000 7fc4577fe640 1 -- 172.21.15.94:0/3324494365 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc448089ac0 con 0x7fc46014f360 2024-03-06T07:18:36.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.156+0000 7fc45effd640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc44006b4d0 0x7fc44006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.156+0000 7fc45effd640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc44006b4d0 0x7fc44006d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc458036000 tx=0x7fc45803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.160+0000 7fc4577fe640 1 -- 172.21.15.94:0/3324494365 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc460068ea0 con 0x7fc46014f360 2024-03-06T07:18:36.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.330+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc46004f330 con 0x7fc46014f360 2024-03-06T07:18:36.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.331+0000 7fc4577fe640 1 -- 172.21.15.94:0/3324494365 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7fc448058690 con 0x7fc46014f360 2024-03-06T07:18:36.333 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:36.333 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc44006b4d0 msgr2=0x7fc44006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc44006b4d0 0x7fc44006d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc458036000 tx=0x7fc45803c040 comp rx=0 tx=0).stop 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc46014f360 msgr2=0x7fc4601ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc46014f360 0x7fc4601ee4e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc44800c470 tx=0x7fc44800c870 comp rx=0 tx=0).stop 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 shutdown_connections 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc44006b4d0 0x7fc44006d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc460158b60 0x7fc4601eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc46014f360 0x7fc4601ee4e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 --2- 172.21.15.94:0/3324494365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4600c88c0 0x7fc4600bf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 >> 172.21.15.94:0/3324494365 conn(0x7fc4600ac570 msgr2=0x7fc46014da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 shutdown_connections 2024-03-06T07:18:36.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.333+0000 7fc4653ae640 1 -- 172.21.15.94:0/3324494365 wait complete. 2024-03-06T07:18:36.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:36.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.448+0000 7fb9aeb7a640 1 Processor -- start 2024-03-06T07:18:36.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.448+0000 7fb9aeb7a640 1 -- start start 2024-03-06T07:18:36.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9aeb7a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb9a81529d0 0x7fb9a80c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9aeb7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a80c57e0 0x7fb9a8155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9aeb7a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a8152db0 con 0x7fb9a8072cd0 2024-03-06T07:18:36.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9aeb7a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a80c6dd0 con 0x7fb9a81529d0 2024-03-06T07:18:36.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9aeb7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a80c6940 con 0x7fb9a80c57e0 2024-03-06T07:18:36.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9ad117640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb9a8072cd0 0x7fb9a8155660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41540/0 (socket says 172.21.15.94:41540) 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9ad117640 1 -- 172.21.15.94:0/1327890662 learned_addr learned my addr 172.21.15.94:0/1327890662 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9a7fff640 1 --2- 172.21.15.94:0/1327890662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a80c57e0 0x7fb9a8155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9ac916640 1 --2- 172.21.15.94:0/1327890662 >> v2:172.21.15.145:3300/0 conn(0x7fb9a81529d0 0x7fb9a80c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9a77fe640 1 -- 172.21.15.94:0/1327890662 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155974618 0 0) 0x7fb9a0001490 con 0x7fb9a8072cd0 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9a7fff640 1 -- 172.21.15.94:0/1327890662 >> v2:172.21.15.145:3300/0 conn(0x7fb9a81529d0 msgr2=0x7fb9a80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9a7fff640 1 --2- 172.21.15.94:0/1327890662 >> v2:172.21.15.145:3300/0 conn(0x7fb9a81529d0 0x7fb9a80c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9a7fff640 1 -- 172.21.15.94:0/1327890662 >> v1:172.21.15.94:6789/0 conn(0x7fb9a8072cd0 legacy=0x7fb9a8155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.449+0000 7fb9a7fff640 1 -- 172.21.15.94:0/1327890662 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a81590f0 con 0x7fb9a80c57e0 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.450+0000 7fb9a7fff640 1 --2- 172.21.15.94:0/1327890662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a80c57e0 0x7fb9a8155120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0097a0 tx=0x7fb99c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad54cf32808d7b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.450+0000 7fb9a77fe640 1 -- 172.21.15.94:0/1327890662 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c038070 con 0x7fb9a80c57e0 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.450+0000 7fb9a77fe640 1 -- 172.21.15.94:0/1327890662 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c031700 con 0x7fb9a80c57e0 2024-03-06T07:18:36.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.450+0000 7fb9a77fe640 1 -- 172.21.15.94:0/1327890662 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c031ad0 con 0x7fb9a80c57e0 2024-03-06T07:18:36.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/1327890662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a80c57e0 msgr2=0x7fb9a8155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/1327890662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a80c57e0 0x7fb9a8155120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0097a0 tx=0x7fb99c02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/1327890662 shutdown_connections 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/1327890662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a80c57e0 0x7fb9a8155120 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/1327890662 >> v2:172.21.15.145:3300/0 conn(0x7fb9a81529d0 0x7fb9a80c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/1327890662 >> 172.21.15.94:0/1327890662 conn(0x7fb9a80aa630 msgr2=0x7fb9a80aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/1327890662 shutdown_connections 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.451+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/1327890662 wait complete. 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.452+0000 7fb9aeb7a640 1 Processor -- start 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.452+0000 7fb9aeb7a640 1 -- start start 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9aeb7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a8072cd0 0x7fb9a80ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9aeb7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9a80c57e0 0x7fb9a814f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9aeb7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a81529d0 0x7fb9a814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9aeb7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a8072720 con 0x7fb9a81529d0 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9aeb7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a8155d70 con 0x7fb9a80c57e0 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9aeb7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a8159370 con 0x7fb9a8072cd0 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ad117640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a81529d0 0x7fb9a814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ad117640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a81529d0 0x7fb9a814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41144/0 (socket says 172.21.15.94:41144) 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ad117640 1 -- 172.21.15.94:0/4271506732 learned_addr learned my addr 172.21.15.94:0/4271506732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ac916640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a8072cd0 0x7fb9a80ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9a7fff640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9a80c57e0 0x7fb9a814f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ac916640 1 -- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9a80c57e0 msgr2=0x7fb9a814f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ac916640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9a80c57e0 0x7fb9a814f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ac916640 1 -- 172.21.15.94:0/4271506732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a81529d0 msgr2=0x7fb9a814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ac916640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a81529d0 0x7fb9a814f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ac916640 1 -- 172.21.15.94:0/4271506732 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a80ae070 con 0x7fb9a8072cd0 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.453+0000 7fb9ad117640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a81529d0 0x7fb9a814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:36.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.454+0000 7fb9ac916640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a8072cd0 0x7fb9a80ac4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb99800e8a0 tx=0x7fb99800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.454+0000 7fb9a57fa640 1 -- 172.21.15.94:0/4271506732 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998017070 con 0x7fb9a8072cd0 2024-03-06T07:18:36.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.454+0000 7fb9a57fa640 1 -- 172.21.15.94:0/4271506732 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99800c940 con 0x7fb9a8072cd0 2024-03-06T07:18:36.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.454+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a81eb5d0 con 0x7fb9a8072cd0 2024-03-06T07:18:36.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.454+0000 7fb9a57fa640 1 -- 172.21.15.94:0/4271506732 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998009e60 con 0x7fb9a8072cd0 2024-03-06T07:18:36.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.454+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a8155010 con 0x7fb9a8072cd0 2024-03-06T07:18:36.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.455+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a804f330 con 0x7fb9a8072cd0 2024-03-06T07:18:36.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.455+0000 7fb9a57fa640 1 -- 172.21.15.94:0/4271506732 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb99801a070 con 0x7fb9a8072cd0 2024-03-06T07:18:36.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.456+0000 7fb9a57fa640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb99406b4d0 0x7fb99406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.456+0000 7fb9a57fa640 1 -- 172.21.15.94:0/4271506732 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb99808a1e0 con 0x7fb9a8072cd0 2024-03-06T07:18:36.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.456+0000 7fb9a7fff640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb99406b4d0 0x7fb99406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.456+0000 7fb9a7fff640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb99406b4d0 0x7fb99406d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb99c035000 tx=0x7fb99c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.465+0000 7fb9a57fa640 1 -- 172.21.15.94:0/4271506732 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb998058760 con 0x7fb9a8072cd0 2024-03-06T07:18:36.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.634+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9a80c7540 con 0x7fb9a8072cd0 2024-03-06T07:18:36.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.634+0000 7fb9a57fa640 1 -- 172.21.15.94:0/4271506732 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb99805c800 con 0x7fb9a8072cd0 2024-03-06T07:18:36.637 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:36.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb99406b4d0 msgr2=0x7fb99406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb99406b4d0 0x7fb99406d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb99c035000 tx=0x7fb99c008030 comp rx=0 tx=0).stop 2024-03-06T07:18:36.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a8072cd0 msgr2=0x7fb9a80ac4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a8072cd0 0x7fb9a80ac4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb99800e8a0 tx=0x7fb99800ed60 comp rx=0 tx=0).stop 2024-03-06T07:18:36.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 shutdown_connections 2024-03-06T07:18:36.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb99406b4d0 0x7fb99406d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9a81529d0 0x7fb9a814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9a80c57e0 0x7fb9a814f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 --2- 172.21.15.94:0/4271506732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9a8072cd0 0x7fb9a80ac4d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 >> 172.21.15.94:0/4271506732 conn(0x7fb9a80aa630 msgr2=0x7fb9a80a85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:36.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 shutdown_connections 2024-03-06T07:18:36.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.636+0000 7fb9aeb7a640 1 -- 172.21.15.94:0/4271506732 wait complete. 2024-03-06T07:18:36.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:36.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.752+0000 7fe85ad81640 1 Processor -- start 2024-03-06T07:18:36.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.752+0000 7fe85ad81640 1 -- start start 2024-03-06T07:18:36.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe85ad81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe854072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe85ad81640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8540c62a0 con 0x7fe8540c4740 2024-03-06T07:18:36.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe85ad81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8540c4c00 con 0x7fe8540a8290 2024-03-06T07:18:36.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe85ad81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540c4ee0 con 0x7fe8541539c0 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe853fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe8540c4740 0x7fe8540c69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41544/0 (socket says 172.21.15.94:41544) 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe853fff640 1 -- 172.21.15.94:0/568528331 learned_addr learned my addr 172.21.15.94:0/568528331 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe85931e640 1 --2- 172.21.15.94:0/568528331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe854072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe8537fe640 1 -- 172.21.15.94:0/568528331 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722197198 0 0) 0x7fe8540c62a0 con 0x7fe8540c4740 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.753+0000 7fe8537fe640 1 -- 172.21.15.94:0/568528331 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe83c003520 con 0x7fe8540c4740 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.754+0000 7fe8537fe640 1 -- 172.21.15.94:0/568528331 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086444133 0 0) 0x7fe8540c4c00 con 0x7fe8540a8290 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.754+0000 7fe8537fe640 1 -- 172.21.15.94:0/568528331 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8540c62a0 con 0x7fe8540a8290 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.754+0000 7fe85931e640 1 -- 172.21.15.94:0/568528331 >> v1:172.21.15.145:6789/0 conn(0x7fe8540a8290 legacy=0x7fe8540701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.754+0000 7fe85931e640 1 -- 172.21.15.94:0/568528331 >> v1:172.21.15.94:6789/0 conn(0x7fe8540c4740 legacy=0x7fe8540c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.754+0000 7fe85931e640 1 -- 172.21.15.94:0/568528331 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85406f9a0 con 0x7fe8541539c0 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.754+0000 7fe85931e640 1 --2- 172.21.15.94:0/568528331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe854072680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009dc0 tx=0x7fe84c02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=548bb12e6c0d6043 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe8537fe640 1 -- 172.21.15.94:0/568528331 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c039070 con 0x7fe8541539c0 2024-03-06T07:18:36.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe8537fe640 1 -- 172.21.15.94:0/568528331 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84c02cbb0 con 0x7fe8541539c0 2024-03-06T07:18:36.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe8537fe640 1 -- 172.21.15.94:0/568528331 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c034440 con 0x7fe8541539c0 2024-03-06T07:18:36.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe85ad81640 1 -- 172.21.15.94:0/568528331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 msgr2=0x7fe854072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe85ad81640 1 --2- 172.21.15.94:0/568528331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe854072680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009dc0 tx=0x7fe84c02be00 comp rx=0 tx=0).stop 2024-03-06T07:18:36.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe85ad81640 1 -- 172.21.15.94:0/568528331 shutdown_connections 2024-03-06T07:18:36.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe85ad81640 1 --2- 172.21.15.94:0/568528331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe854072680 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe85ad81640 1 -- 172.21.15.94:0/568528331 >> 172.21.15.94:0/568528331 conn(0x7fe8540c5560 msgr2=0x7fe8540c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe85ad81640 1 -- 172.21.15.94:0/568528331 shutdown_connections 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.755+0000 7fe85ad81640 1 -- 172.21.15.94:0/568528331 wait complete. 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.756+0000 7fe85ad81640 1 Processor -- start 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.756+0000 7fe85ad81640 1 -- start start 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe85ad81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8540a8290 0x7fe8541ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe85ad81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8540c4740 0x7fe8541ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe85ad81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe8541569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe85ad81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe854072bc0 con 0x7fe8540a8290 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe85ad81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe854153e50 con 0x7fe8540c4740 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe85ad81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe854072d40 con 0x7fe8541539c0 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8540a8290 0x7fe8541ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8540a8290 0x7fe8541ebdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41156/0 (socket says 172.21.15.94:41156) 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 -- 172.21.15.94:0/3508097374 learned_addr learned my addr 172.21.15.94:0/3508097374 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe853fff640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8540c4740 0x7fe8541ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe85931e640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe8541569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 -- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 msgr2=0x7fe8541569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe8541569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 -- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8540c4740 msgr2=0x7fe8541ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8540c4740 0x7fe8541ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.757+0000 7fe858b1d640 1 -- 172.21.15.94:0/3508097374 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8540ae960 con 0x7fe8540a8290 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.758+0000 7fe858b1d640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8540a8290 0x7fe8541ebdd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe84400c530 tx=0x7fe84400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.758+0000 7fe8517fa640 1 -- 172.21.15.94:0/3508097374 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8440052d0 con 0x7fe8540a8290 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.758+0000 7fe8517fa640 1 -- 172.21.15.94:0/3508097374 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe844005490 con 0x7fe8540a8290 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.758+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8540ae680 con 0x7fe8540a8290 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.758+0000 7fe8517fa640 1 -- 172.21.15.94:0/3508097374 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844012870 con 0x7fe8540a8290 2024-03-06T07:18:36.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.758+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8541eb5d0 con 0x7fe8540a8290 2024-03-06T07:18:36.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.759+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85404f330 con 0x7fe8540a8290 2024-03-06T07:18:36.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.759+0000 7fe8517fa640 1 -- 172.21.15.94:0/3508097374 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe844012a10 con 0x7fe8540a8290 2024-03-06T07:18:36.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.760+0000 7fe8517fa640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe84006b4d0 0x7fe84006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:36.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.760+0000 7fe8517fa640 1 -- 172.21.15.94:0/3508097374 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe84408a040 con 0x7fe8540a8290 2024-03-06T07:18:36.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.760+0000 7fe853fff640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe84006b4d0 0x7fe84006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:36.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.760+0000 7fe853fff640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe84006b4d0 0x7fe84006d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe85414a8c0 tx=0x7fe848007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:36.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.764+0000 7fe8517fa640 1 -- 172.21.15.94:0/3508097374 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe844058540 con 0x7fe8540a8290 2024-03-06T07:18:36.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.934+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8540c7330 con 0x7fe8540a8290 2024-03-06T07:18:36.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.935+0000 7fe8517fa640 1 -- 172.21.15.94:0/3508097374 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe84405c5e0 con 0x7fe8540a8290 2024-03-06T07:18:36.937 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.936+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe84006b4d0 msgr2=0x7fe84006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.936+0000 7fe85ad81640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe84006b4d0 0x7fe84006d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe85414a8c0 tx=0x7fe848007d30 comp rx=0 tx=0).stop 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.936+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8540a8290 msgr2=0x7fe8541ebdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.936+0000 7fe85ad81640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8540a8290 0x7fe8541ebdd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe84400c530 tx=0x7fe84400c9f0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 shutdown_connections 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe84006b4d0 0x7fe84006d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8541539c0 0x7fe8541569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8540c4740 0x7fe8541ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 --2- 172.21.15.94:0/3508097374 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8540a8290 0x7fe8541ebdd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:36.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 >> 172.21.15.94:0/3508097374 conn(0x7fe8540c5560 msgr2=0x7fe8540c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:36.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 shutdown_connections 2024-03-06T07:18:36.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:36.937+0000 7fe85ad81640 1 -- 172.21.15.94:0/3508097374 wait complete. 2024-03-06T07:18:36.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:37.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.050+0000 7ffad6143640 1 Processor -- start 2024-03-06T07:18:37.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffad6143640 1 -- start start 2024-03-06T07:18:37.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffad6143640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffad00b0070 0x7ffad0155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffad6143640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00b0450 con 0x7ffad006ca70 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffad6143640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad006cf20 con 0x7ffad00c5380 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffad6143640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad006c020 con 0x7ffad00b0070 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffac6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ffad006ca70 0x7ffad00c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41548/0 (socket says 172.21.15.94:41548) 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffac6ffd640 1 -- 172.21.15.94:0/3069808982 learned_addr learned my addr 172.21.15.94:0/3069808982 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.051+0000 7ffacf7fe640 1 --2- 172.21.15.94:0/3069808982 >> v2:172.21.15.145:3300/0 conn(0x7ffad00b0070 0x7ffad0155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.052+0000 7ffaceffd640 1 -- 172.21.15.94:0/3069808982 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128136133 0 0) 0x7ffab8001110 con 0x7ffad006ca70 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.052+0000 7ffaceffd640 1 -- 172.21.15.94:0/3069808982 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc003520 con 0x7ffad006ca70 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.052+0000 7ffacf7fe640 1 -- 172.21.15.94:0/3069808982 >> v1:172.21.15.145:6789/0 conn(0x7ffad00c5380 legacy=0x7ffad014e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.052+0000 7ffaceffd640 1 -- 172.21.15.94:0/3069808982 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939136154 0 0) 0x7ffad006cf20 con 0x7ffad00c5380 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.052+0000 7ffacf7fe640 1 -- 172.21.15.94:0/3069808982 >> v1:172.21.15.94:6789/0 conn(0x7ffad006ca70 legacy=0x7ffad00c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.052+0000 7ffacf7fe640 1 -- 172.21.15.94:0/3069808982 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00c77f0 con 0x7ffad00b0070 2024-03-06T07:18:37.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.052+0000 7ffacf7fe640 1 --2- 172.21.15.94:0/3069808982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 0x7ffad0155c30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffac00097a0 tx=0x7ffac002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7340e89b07cc4c45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffaceffd640 1 -- 172.21.15.94:0/3069808982 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0035020 con 0x7ffad00b0070 2024-03-06T07:18:37.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffaceffd640 1 -- 172.21.15.94:0/3069808982 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0031830 con 0x7ffad00b0070 2024-03-06T07:18:37.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffaceffd640 1 -- 172.21.15.94:0/3069808982 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0031be0 con 0x7ffad00b0070 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffad6143640 1 -- 172.21.15.94:0/3069808982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 msgr2=0x7ffad0155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffad6143640 1 --2- 172.21.15.94:0/3069808982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 0x7ffad0155c30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffac00097a0 tx=0x7ffac002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffad6143640 1 -- 172.21.15.94:0/3069808982 shutdown_connections 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffad6143640 1 --2- 172.21.15.94:0/3069808982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 0x7ffad0155c30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffad6143640 1 -- 172.21.15.94:0/3069808982 >> 172.21.15.94:0/3069808982 conn(0x7ffad00ac590 msgr2=0x7ffad00ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffad6143640 1 -- 172.21.15.94:0/3069808982 shutdown_connections 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.053+0000 7ffad6143640 1 -- 172.21.15.94:0/3069808982 wait complete. 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.054+0000 7ffad6143640 1 Processor -- start 2024-03-06T07:18:37.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffad6143640 1 -- start start 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffad6143640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffad006ca70 0x7ffad01511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffad6143640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 0x7ffad0158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffad6143640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffad00c5380 0x7ffad014dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffad6143640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad006c550 con 0x7ffad00c5380 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffad6143640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad006c270 con 0x7ffad00b0070 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffad6143640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad006da80 con 0x7ffad006ca70 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffacffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffad00c5380 0x7ffad014dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffacffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffad00c5380 0x7ffad014dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41164/0 (socket says 172.21.15.94:41164) 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffacffff640 1 -- 172.21.15.94:0/736248556 learned_addr learned my addr 172.21.15.94:0/736248556 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffac6ffd640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 0x7ffad0158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.055+0000 7ffacf7fe640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffad006ca70 0x7ffad01511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffacffff640 1 -- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffad006ca70 msgr2=0x7ffad01511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffacffff640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffad006ca70 0x7ffad01511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffacffff640 1 -- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 msgr2=0x7ffad0158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffacffff640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 0x7ffad0158ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffacffff640 1 -- 172.21.15.94:0/736248556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00c4680 con 0x7ffad00c5380 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffacffff640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffad00c5380 0x7ffad014dbd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffac800c790 tx=0x7ffac800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffaccff9640 1 -- 172.21.15.94:0/736248556 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8003da0 con 0x7ffad00c5380 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffaccff9640 1 -- 172.21.15.94:0/736248556 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac8005070 con 0x7ffad00c5380 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffaccff9640 1 -- 172.21.15.94:0/736248556 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8005460 con 0x7ffad00c5380 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad0157c40 con 0x7ffad00c5380 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.056+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad00c4920 con 0x7ffad00c5380 2024-03-06T07:18:37.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.057+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad00c4b20 con 0x7ffad00c5380 2024-03-06T07:18:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.057+0000 7ffaccff9640 1 -- 172.21.15.94:0/736248556 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffac8005600 con 0x7ffad00c5380 2024-03-06T07:18:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.058+0000 7ffaccff9640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.058+0000 7ffaccff9640 1 -- 172.21.15.94:0/736248556 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ffac808a290 con 0x7ffad00c5380 2024-03-06T07:18:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.058+0000 7ffacf7fe640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.058+0000 7ffacf7fe640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffab006b4d0 0x7ffab006d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffac0036000 tx=0x7ffac0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.062+0000 7ffaccff9640 1 -- 172.21.15.94:0/736248556 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffad00c4b20 con 0x7ffad00c5380 2024-03-06T07:18:37.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.235+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffad01521d0 con 0x7ffad00c5380 2024-03-06T07:18:37.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.236+0000 7ffaccff9640 1 -- 172.21.15.94:0/736248556 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffac8058810 con 0x7ffad00c5380 2024-03-06T07:18:37.238 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:37.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffab006b4d0 msgr2=0x7ffab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffab006b4d0 0x7ffab006d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffac0036000 tx=0x7ffac0008030 comp rx=0 tx=0).stop 2024-03-06T07:18:37.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffad00c5380 msgr2=0x7ffad014dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffad00c5380 0x7ffad014dbd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffac800c790 tx=0x7ffac800cc50 comp rx=0 tx=0).stop 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 shutdown_connections 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffad00c5380 0x7ffad014dbd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffad00b0070 0x7ffad0158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 --2- 172.21.15.94:0/736248556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffad006ca70 0x7ffad01511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 >> 172.21.15.94:0/736248556 conn(0x7ffad00ac590 msgr2=0x7ffad0154f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 shutdown_connections 2024-03-06T07:18:37.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.238+0000 7ffad6143640 1 -- 172.21.15.94:0/736248556 wait complete. 2024-03-06T07:18:37.251 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:37.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.352+0000 7f03bb3ac640 1 Processor -- start 2024-03-06T07:18:37.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.352+0000 7f03bb3ac640 1 -- start start 2024-03-06T07:18:37.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03bb3ac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f03b40c67f0 0x7f03b4155220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03bb3ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f03b40c4ef0 0x7f03b4155760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03bb3ac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b4152cb0 con 0x7f03b41528d0 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03bb3ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b4155ca0 con 0x7f03b40c4ef0 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03bb3ac640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b40c38b0 con 0x7f03b40c67f0 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03b9949640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f03b40c4ef0 0x7f03b4155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03b9148640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f03b41528d0 0x7f03b40724d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41564/0 (socket says 172.21.15.94:41564) 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03b9148640 1 -- 172.21.15.94:0/1592038912 learned_addr learned my addr 172.21.15.94:0/1592038912 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03b8947640 1 --2- 172.21.15.94:0/1592038912 >> v2:172.21.15.145:3300/0 conn(0x7f03b40c67f0 0x7f03b4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.353+0000 7f03b9949640 1 -- 172.21.15.94:0/1592038912 >> v2:172.21.15.145:3300/0 conn(0x7f03b40c67f0 msgr2=0x7f03b4155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.354+0000 7f03b9949640 1 --2- 172.21.15.94:0/1592038912 >> v2:172.21.15.145:3300/0 conn(0x7f03b40c67f0 0x7f03b4155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.354+0000 7f03b9949640 1 -- 172.21.15.94:0/1592038912 >> v1:172.21.15.94:6789/0 conn(0x7f03b41528d0 legacy=0x7f03b40724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.354+0000 7f03b9949640 1 -- 172.21.15.94:0/1592038912 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b41594f0 con 0x7f03b40c4ef0 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.354+0000 7f03b9949640 1 --2- 172.21.15.94:0/1592038912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c4ef0 0x7f03b4155760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f03b4074ff0 tx=0x7f03ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2e1f994bb632 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.354+0000 7f03abfff640 1 -- 172.21.15.94:0/1592038912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac034070 con 0x7f03b40c4ef0 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.354+0000 7f03abfff640 1 -- 172.21.15.94:0/1592038912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac030610 con 0x7f03b40c4ef0 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.354+0000 7f03abfff640 1 -- 172.21.15.94:0/1592038912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0309e0 con 0x7f03b40c4ef0 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 -- 172.21.15.94:0/1592038912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c4ef0 msgr2=0x7f03b4155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/1592038912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c4ef0 0x7f03b4155760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f03b4074ff0 tx=0x7f03ac02b960 comp rx=0 tx=0).stop 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 -- 172.21.15.94:0/1592038912 shutdown_connections 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/1592038912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c4ef0 0x7f03b4155760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/1592038912 >> v2:172.21.15.145:3300/0 conn(0x7f03b40c67f0 0x7f03b4155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 -- 172.21.15.94:0/1592038912 >> 172.21.15.94:0/1592038912 conn(0x7f03b40aa6a0 msgr2=0x7f03b40aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:37.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 -- 172.21.15.94:0/1592038912 shutdown_connections 2024-03-06T07:18:37.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.355+0000 7f03bb3ac640 1 -- 172.21.15.94:0/1592038912 wait complete. 2024-03-06T07:18:37.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.356+0000 7f03bb3ac640 1 Processor -- start 2024-03-06T07:18:37.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.356+0000 7f03bb3ac640 1 -- start start 2024-03-06T07:18:37.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03bb3ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03b40c4ef0 0x7f03b41517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03bb3ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c67f0 0x7f03b4149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03bb3ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03b41528d0 0x7f03b4158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03bb3ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40a8250 con 0x7f03b40c67f0 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03bb3ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40c3390 con 0x7f03b40c4ef0 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03bb3ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b41591a0 con 0x7f03b41528d0 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c67f0 0x7f03b4149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c67f0 0x7f03b4149f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41174/0 (socket says 172.21.15.94:41174) 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 -- 172.21.15.94:0/2632645230 learned_addr learned my addr 172.21.15.94:0/2632645230 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b9148640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03b40c4ef0 0x7f03b41517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b9949640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03b41528d0 0x7f03b4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 -- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03b41528d0 msgr2=0x7f03b4158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03b41528d0 0x7f03b4158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 -- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03b40c4ef0 msgr2=0x7f03b41517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03b40c4ef0 0x7f03b41517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 -- 172.21.15.94:0/2632645230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4155e80 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b9949640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03b41528d0 0x7f03b4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03b8947640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c67f0 0x7f03b4149f10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f03a400e9b0 tx=0x7f03a400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03a9ffb640 1 -- 172.21.15.94:0/2632645230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4017070 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03a9ffb640 1 -- 172.21.15.94:0/2632645230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a400c9a0 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b4156160 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.357+0000 7f03a9ffb640 1 -- 172.21.15.94:0/2632645230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a400cd90 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.358+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b4157d40 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.359+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b404f330 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.359+0000 7f03a9ffb640 1 -- 172.21.15.94:0/2632645230 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03a401a070 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.359+0000 7f03a9ffb640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f039806b4d0 0x7f039806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.359+0000 7f03a9ffb640 1 -- 172.21.15.94:0/2632645230 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f03a408a830 con 0x7f03b40c67f0 2024-03-06T07:18:37.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.359+0000 7f03b9148640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f039806b4d0 0x7f039806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.360+0000 7f03b9148640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f039806b4d0 0x7f039806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03b00012e0 tx=0x7f03b0001a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.364+0000 7f03a9ffb640 1 -- 172.21.15.94:0/2632645230 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03a4058d20 con 0x7f03b40c67f0 2024-03-06T07:18:37.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.533+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f03b4154f90 con 0x7f03b40c67f0 2024-03-06T07:18:37.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.534+0000 7f03a9ffb640 1 -- 172.21.15.94:0/2632645230 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f03a405cdc0 con 0x7f03b40c67f0 2024-03-06T07:18:37.536 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:37.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f039806b4d0 msgr2=0x7f039806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f039806b4d0 0x7f039806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03b00012e0 tx=0x7f03b0001a30 comp rx=0 tx=0).stop 2024-03-06T07:18:37.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c67f0 msgr2=0x7f03b4149f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c67f0 0x7f03b4149f10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f03a400e9b0 tx=0x7f03a400ee70 comp rx=0 tx=0).stop 2024-03-06T07:18:37.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 shutdown_connections 2024-03-06T07:18:37.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f039806b4d0 0x7f039806d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03b41528d0 0x7f03b4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03b40c67f0 0x7f03b4149f10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 --2- 172.21.15.94:0/2632645230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03b40c4ef0 0x7f03b41517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 >> 172.21.15.94:0/2632645230 conn(0x7f03b40aa6a0 msgr2=0x7f03b40c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:37.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 shutdown_connections 2024-03-06T07:18:37.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.536+0000 7f03bb3ac640 1 -- 172.21.15.94:0/2632645230 wait complete. 2024-03-06T07:18:37.549 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:37.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.647+0000 7fd4f04ed640 1 Processor -- start 2024-03-06T07:18:37.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.647+0000 7fd4f04ed640 1 -- start start 2024-03-06T07:18:37.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4f04ed640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd4e80c20b0 0x7fd4e806b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4f04ed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4e80c25c0 0x7fd4e8153890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4f04ed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e8153dd0 con 0x7fd4e806afc0 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4f04ed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e80a9530 con 0x7fd4e80c25c0 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4f04ed640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e80a9810 con 0x7fd4e80c20b0 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4eea8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4e80c25c0 0x7fd4e8153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4ee289640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd4e806afc0 0x7fd4e8153180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41566/0 (socket says 172.21.15.94:41566) 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4ee289640 1 -- 172.21.15.94:0/2052886189 learned_addr learned my addr 172.21.15.94:0/2052886189 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.648+0000 7fd4eda88640 1 --2- 172.21.15.94:0/2052886189 >> v2:172.21.15.145:3300/0 conn(0x7fd4e80c20b0 0x7fd4e806b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.649+0000 7fd4eea8a640 1 -- 172.21.15.94:0/2052886189 >> v2:172.21.15.145:3300/0 conn(0x7fd4e80c20b0 msgr2=0x7fd4e806b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.649+0000 7fd4eea8a640 1 --2- 172.21.15.94:0/2052886189 >> v2:172.21.15.145:3300/0 conn(0x7fd4e80c20b0 0x7fd4e806b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.649+0000 7fd4eea8a640 1 -- 172.21.15.94:0/2052886189 >> v1:172.21.15.94:6789/0 conn(0x7fd4e806afc0 legacy=0x7fd4e8153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.649+0000 7fd4ed287640 1 -- 172.21.15.94:0/2052886189 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817460311 0 0) 0x7fd4e8153dd0 con 0x7fd4e806afc0 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.649+0000 7fd4eea8a640 1 -- 172.21.15.94:0/2052886189 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80a9af0 con 0x7fd4e80c25c0 2024-03-06T07:18:37.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.649+0000 7fd4eea8a640 1 --2- 172.21.15.94:0/2052886189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c25c0 0x7fd4e8153890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8067830 tx=0x7fd4d802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ae4faad92ad04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4ed287640 1 -- 172.21.15.94:0/2052886189 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8038070 con 0x7fd4e80c25c0 2024-03-06T07:18:37.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4ed287640 1 -- 172.21.15.94:0/2052886189 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8031710 con 0x7fd4e80c25c0 2024-03-06T07:18:37.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4ed287640 1 -- 172.21.15.94:0/2052886189 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8031ae0 con 0x7fd4e80c25c0 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2052886189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c25c0 msgr2=0x7fd4e8153890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2052886189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c25c0 0x7fd4e8153890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8067830 tx=0x7fd4d802b7b0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2052886189 shutdown_connections 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2052886189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c25c0 0x7fd4e8153890 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2052886189 >> v2:172.21.15.145:3300/0 conn(0x7fd4e80c20b0 0x7fd4e806b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2052886189 >> 172.21.15.94:0/2052886189 conn(0x7fd4e80c76b0 msgr2=0x7fd4e80c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2052886189 shutdown_connections 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.650+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2052886189 wait complete. 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.651+0000 7fd4f04ed640 1 Processor -- start 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.651+0000 7fd4f04ed640 1 -- start start 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4f04ed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e806afc0 0x7fd4e8147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4f04ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c20b0 0x7fd4e81e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4f04ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e80c25c0 0x7fd4e80c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4f04ed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e8150980 con 0x7fd4e80c20b0 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4f04ed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80aa0b0 con 0x7fd4e806afc0 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4f04ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e8150b00 con 0x7fd4e80c25c0 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c20b0 0x7fd4e81e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c20b0 0x7fd4e81e96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41204/0 (socket says 172.21.15.94:41204) 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 -- 172.21.15.94:0/2807735925 learned_addr learned my addr 172.21.15.94:0/2807735925 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eea8a640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e80c25c0 0x7fd4e80c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4ee289640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e806afc0 0x7fd4e8147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 -- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e80c25c0 msgr2=0x7fd4e80c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e80c25c0 0x7fd4e80c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 -- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e806afc0 msgr2=0x7fd4e8147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e806afc0 0x7fd4e8147d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.652+0000 7fd4eda88640 1 -- 172.21.15.94:0/2807735925 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8149790 con 0x7fd4e80c20b0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.653+0000 7fd4eda88640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c20b0 0x7fd4e81e96a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00e840 tx=0x7fd4dc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.653+0000 7fd4e6ffd640 1 -- 172.21.15.94:0/2807735925 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc017070 con 0x7fd4e80c20b0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.653+0000 7fd4e6ffd640 1 -- 172.21.15.94:0/2807735925 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc00c940 con 0x7fd4e80c20b0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.653+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e8149a70 con 0x7fd4e80c20b0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.653+0000 7fd4e6ffd640 1 -- 172.21.15.94:0/2807735925 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00cd10 con 0x7fd4e80c20b0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.653+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e81e9180 con 0x7fd4e80c20b0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.654+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e804f330 con 0x7fd4e80c20b0 2024-03-06T07:18:37.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.654+0000 7fd4e6ffd640 1 -- 172.21.15.94:0/2807735925 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4dc01a070 con 0x7fd4e80c20b0 2024-03-06T07:18:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.654+0000 7fd4e6ffd640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4d406b4d0 0x7fd4d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.654+0000 7fd4e6ffd640 1 -- 172.21.15.94:0/2807735925 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd4dc08a6a0 con 0x7fd4e80c20b0 2024-03-06T07:18:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.655+0000 7fd4ee289640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4d406b4d0 0x7fd4d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.655+0000 7fd4ee289640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4d406b4d0 0x7fd4d406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00012e0 tx=0x7fd4e0001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.659+0000 7fd4e6ffd640 1 -- 172.21.15.94:0/2807735925 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4dc058c20 con 0x7fd4e80c20b0 2024-03-06T07:18:37.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.868+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e806bc50 con 0x7fd4e80c20b0 2024-03-06T07:18:37.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.869+0000 7fd4e6ffd640 1 -- 172.21.15.94:0/2807735925 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7fd4dc05ccc0 con 0x7fd4e80c20b0 2024-03-06T07:18:37.871 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:37.871 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:37.871 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:18:37.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4d406b4d0 msgr2=0x7fd4d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4d406b4d0 0x7fd4d406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00012e0 tx=0x7fd4e0001630 comp rx=0 tx=0).stop 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c20b0 msgr2=0x7fd4e81e96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c20b0 0x7fd4e81e96a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00e840 tx=0x7fd4dc00ed00 comp rx=0 tx=0).stop 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 shutdown_connections 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4d406b4d0 0x7fd4d406d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e80c25c0 0x7fd4e80c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4e80c20b0 0x7fd4e81e96a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 --2- 172.21.15.94:0/2807735925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e806afc0 0x7fd4e8147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.870+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 >> 172.21.15.94:0/2807735925 conn(0x7fd4e80c76b0 msgr2=0x7fd4e80c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.871+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 shutdown_connections 2024-03-06T07:18:37.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.871+0000 7fd4f04ed640 1 -- 172.21.15.94:0/2807735925 wait complete. 2024-03-06T07:18:37.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:37.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.981+0000 7fbf2e31e640 1 Processor -- start 2024-03-06T07:18:37.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.981+0000 7fbf2e31e640 1 -- start start 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2e31e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbf2806d9f0 0x7fbf280a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2e31e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbf280c3710 0x7fbf281558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2e31e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28158800 con 0x7fbf280c3280 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2e31e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf28158ae0 con 0x7fbf280c3710 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2e31e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf28158dc0 con 0x7fbf2806d9f0 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2c8bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbf280c3710 0x7fbf281558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2c8bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbf280c3710 0x7fbf281558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41214/0 (socket says 172.21.15.94:41214) 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf277fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbf280c3280 0x7fbf281551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41582/0 (socket says 172.21.15.94:41582) 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/1988015670 learned_addr learned my addr 172.21.15.94:0/1988015670 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:37.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf27fff640 1 --2- 172.21.15.94:0/1988015670 >> v2:172.21.15.145:3300/0 conn(0x7fbf2806d9f0 0x7fbf280a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/1988015670 >> v2:172.21.15.145:3300/0 conn(0x7fbf2806d9f0 msgr2=0x7fbf280a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2c8bb640 1 --2- 172.21.15.94:0/1988015670 >> v2:172.21.15.145:3300/0 conn(0x7fbf2806d9f0 0x7fbf280a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.982+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/1988015670 >> v1:172.21.15.94:6789/0 conn(0x7fbf280c3280 legacy=0x7fbf281551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.983+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/1988015670 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf281590a0 con 0x7fbf280c3710 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.983+0000 7fbf26ffd640 1 -- 172.21.15.94:0/1988015670 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236628618 0 0) 0x7fbf28158800 con 0x7fbf280c3280 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.983+0000 7fbf2c8bb640 1 --2- 172.21.15.94:0/1988015670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281558f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf280c9820 tx=0x7fbf2002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdeab5ee20f4c481 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.983+0000 7fbf26ffd640 1 -- 172.21.15.94:0/1988015670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2002ca80 con 0x7fbf280c3710 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.983+0000 7fbf26ffd640 1 -- 172.21.15.94:0/1988015670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2002cc20 con 0x7fbf280c3710 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.983+0000 7fbf26ffd640 1 -- 172.21.15.94:0/1988015670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf200343f0 con 0x7fbf280c3710 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 -- 172.21.15.94:0/1988015670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 msgr2=0x7fbf281558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/1988015670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281558f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf280c9820 tx=0x7fbf2002bee0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 -- 172.21.15.94:0/1988015670 shutdown_connections 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/1988015670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281558f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/1988015670 >> v2:172.21.15.145:3300/0 conn(0x7fbf2806d9f0 0x7fbf280a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 -- 172.21.15.94:0/1988015670 >> 172.21.15.94:0/1988015670 conn(0x7fbf280aa630 msgr2=0x7fbf280aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:37.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 -- 172.21.15.94:0/1988015670 shutdown_connections 2024-03-06T07:18:37.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.984+0000 7fbf2e31e640 1 -- 172.21.15.94:0/1988015670 wait complete. 2024-03-06T07:18:37.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.985+0000 7fbf2e31e640 1 Processor -- start 2024-03-06T07:18:37.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.985+0000 7fbf2e31e640 1 -- start start 2024-03-06T07:18:37.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2e31e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf2806d9f0 0x7fbf2806f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2e31e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280c3280 0x7fbf281e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2e31e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2e31e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2806ddd0 con 0x7fbf280c3710 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2e31e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28155e30 con 0x7fbf2806d9f0 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2e31e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2806df50 con 0x7fbf280c3280 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281e7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41218/0 (socket says 172.21.15.94:41218) 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/602692020 learned_addr learned my addr 172.21.15.94:0/602692020 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf27fff640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf2806d9f0 0x7fbf2806f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf277fe640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280c3280 0x7fbf281e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280c3280 msgr2=0x7fbf281e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280c3280 0x7fbf281e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf2806d9f0 msgr2=0x7fbf2806f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf2806d9f0 0x7fbf2806f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 -- 172.21.15.94:0/602692020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2814fef0 con 0x7fbf280c3710 2024-03-06T07:18:37.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.986+0000 7fbf2c8bb640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281e7e30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf20036040 tx=0x7fbf20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.987+0000 7fbf24ff9640 1 -- 172.21.15.94:0/602692020 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20043020 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.987+0000 7fbf24ff9640 1 -- 172.21.15.94:0/602692020 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2003e450 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.987+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf281501d0 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.987+0000 7fbf24ff9640 1 -- 172.21.15.94:0/602692020 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20009510 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.987+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf281ea6b0 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.988+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf28075300 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.988+0000 7fbf24ff9640 1 -- 172.21.15.94:0/602692020 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf20046020 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.988+0000 7fbf24ff9640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf1406b4d0 0x7fbf1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.988+0000 7fbf24ff9640 1 -- 172.21.15.94:0/602692020 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbf200ac9d0 con 0x7fbf280c3710 2024-03-06T07:18:37.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.988+0000 7fbf27fff640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf1406b4d0 0x7fbf1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:37.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.989+0000 7fbf27fff640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf1406b4d0 0x7fbf1406d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009b40 tx=0x7fbf18009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:37.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:37.993+0000 7fbf24ff9640 1 -- 172.21.15.94:0/602692020 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf28075300 con 0x7fbf280c3710 2024-03-06T07:18:38.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.166+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf2804f330 con 0x7fbf280c3710 2024-03-06T07:18:38.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.167+0000 7fbf24ff9640 1 -- 172.21.15.94:0/602692020 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7fbf2004d030 con 0x7fbf280c3710 2024-03-06T07:18:38.169 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:38.169 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf1406b4d0 msgr2=0x7fbf1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf1406b4d0 0x7fbf1406d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009b40 tx=0x7fbf18009190 comp rx=0 tx=0).stop 2024-03-06T07:18:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 msgr2=0x7fbf281e7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281e7e30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf20036040 tx=0x7fbf20004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 shutdown_connections 2024-03-06T07:18:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf1406b4d0 0x7fbf1406d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf280c3710 0x7fbf281e7e30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280c3280 0x7fbf281e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 --2- 172.21.15.94:0/602692020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf2806d9f0 0x7fbf2806f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 >> 172.21.15.94:0/602692020 conn(0x7fbf280aa630 msgr2=0x7fbf280aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 shutdown_connections 2024-03-06T07:18:38.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.169+0000 7fbf2e31e640 1 -- 172.21.15.94:0/602692020 wait complete. 2024-03-06T07:18:38.182 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:38.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.284+0000 7f84c8113640 1 Processor -- start 2024-03-06T07:18:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.284+0000 7f84c8113640 1 -- start start 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.284+0000 7f84c8113640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c00c25a0 0x7f84c00732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.284+0000 7f84c8113640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c0156060 con 0x7f84c006d750 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.284+0000 7f84c8113640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c00a8430 con 0x7f84c00c3a70 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.284+0000 7f84c8113640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0155b30 con 0x7f84c00c25a0 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c5eaf640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f84c006d750 0x7f84c0072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41598/0 (socket says 172.21.15.94:41598) 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c5eaf640 1 -- 172.21.15.94:0/3273952929 learned_addr learned my addr 172.21.15.94:0/3273952929 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c66b0640 1 --2- 172.21.15.94:0/3273952929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c00c25a0 0x7f84c00732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c4ead640 1 -- 172.21.15.94:0/3273952929 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955302264 0 0) 0x7f84c0156060 con 0x7f84c006d750 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c4ead640 1 -- 172.21.15.94:0/3273952929 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ac003520 con 0x7f84c006d750 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c4ead640 1 -- 172.21.15.94:0/3273952929 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185970910 0 0) 0x7f84c00a8430 con 0x7f84c00c3a70 2024-03-06T07:18:38.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c66b0640 1 -- 172.21.15.94:0/3273952929 >> v1:172.21.15.145:6789/0 conn(0x7f84c00c3a70 legacy=0x7f84c00c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c66b0640 1 -- 172.21.15.94:0/3273952929 >> v1:172.21.15.94:6789/0 conn(0x7f84c006d750 legacy=0x7f84c0072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.285+0000 7f84c66b0640 1 -- 172.21.15.94:0/3273952929 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0155d80 con 0x7f84c00c25a0 2024-03-06T07:18:38.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.286+0000 7f84c66b0640 1 --2- 172.21.15.94:0/3273952929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c00c25a0 0x7f84c00732a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009dc0 tx=0x7f84b802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c596ebf8d863ba7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.286+0000 7f84c4ead640 1 -- 172.21.15.94:0/3273952929 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8039070 con 0x7f84c00c25a0 2024-03-06T07:18:38.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.286+0000 7f84c4ead640 1 -- 172.21.15.94:0/3273952929 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b802cbb0 con 0x7f84c00c25a0 2024-03-06T07:18:38.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.286+0000 7f84c4ead640 1 -- 172.21.15.94:0/3273952929 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8034440 con 0x7f84c00c25a0 2024-03-06T07:18:38.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.286+0000 7f84c8113640 1 -- 172.21.15.94:0/3273952929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c00c25a0 msgr2=0x7f84c00732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.286+0000 7f84c8113640 1 --2- 172.21.15.94:0/3273952929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c00c25a0 0x7f84c00732a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009dc0 tx=0x7f84b802be00 comp rx=0 tx=0).stop 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.287+0000 7f84c8113640 1 -- 172.21.15.94:0/3273952929 shutdown_connections 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.287+0000 7f84c8113640 1 --2- 172.21.15.94:0/3273952929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c00c25a0 0x7f84c00732a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.287+0000 7f84c8113640 1 -- 172.21.15.94:0/3273952929 >> 172.21.15.94:0/3273952929 conn(0x7f84c00c5560 msgr2=0x7f84c00c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.287+0000 7f84c8113640 1 -- 172.21.15.94:0/3273952929 shutdown_connections 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.287+0000 7f84c8113640 1 -- 172.21.15.94:0/3273952929 wait complete. 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 Processor -- start 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 -- start start 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c006d750 0x7f84c01eaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84c00c3a70 0x7f84c00ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c01504f0 0x7f84c00acc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c006dcb0 con 0x7f84c01504f0 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c006db30 con 0x7f84c00c3a70 2024-03-06T07:18:38.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c8113640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c006de30 con 0x7f84c006d750 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c66b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c01504f0 0x7f84c00acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c5eaf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c006d750 0x7f84c01eaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c56ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84c00c3a70 0x7f84c00ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.288+0000 7f84c66b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c01504f0 0x7f84c00acc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41224/0 (socket says 172.21.15.94:41224) 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c66b0640 1 -- 172.21.15.94:0/1077181618 learned_addr learned my addr 172.21.15.94:0/1077181618 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c66b0640 1 -- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c006d750 msgr2=0x7f84c01eaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c66b0640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c006d750 0x7f84c01eaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c66b0640 1 -- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84c00c3a70 msgr2=0x7f84c00ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c66b0640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84c00c3a70 0x7f84c00ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c66b0640 1 -- 172.21.15.94:0/1077181618 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c00abbb0 con 0x7f84c01504f0 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c66b0640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c01504f0 0x7f84c00acc10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009ee0 tx=0x7f84b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84b67fc640 1 -- 172.21.15.94:0/1077181618 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8039050 con 0x7f84c01504f0 2024-03-06T07:18:38.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84b67fc640 1 -- 172.21.15.94:0/1077181618 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b8045070 con 0x7f84c01504f0 2024-03-06T07:18:38.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84b67fc640 1 -- 172.21.15.94:0/1077181618 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b803e850 con 0x7f84c01504f0 2024-03-06T07:18:38.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c00abe50 con 0x7f84c01504f0 2024-03-06T07:18:38.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.289+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c00ac6f0 con 0x7f84c01504f0 2024-03-06T07:18:38.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.290+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c00ac020 con 0x7f84c01504f0 2024-03-06T07:18:38.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.291+0000 7f84b67fc640 1 -- 172.21.15.94:0/1077181618 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84b803e9f0 con 0x7f84c01504f0 2024-03-06T07:18:38.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.291+0000 7f84b67fc640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f84a406b500 0x7f84a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.291+0000 7f84b67fc640 1 -- 172.21.15.94:0/1077181618 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f84b80ad240 con 0x7f84c01504f0 2024-03-06T07:18:38.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.291+0000 7f84c5eaf640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f84a406b500 0x7f84a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.292+0000 7f84c5eaf640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f84a406b500 0x7f84a406d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005880 tx=0x7f84bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.296+0000 7f84b67fc640 1 -- 172.21.15.94:0/1077181618 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84b807b8e0 con 0x7f84c01504f0 2024-03-06T07:18:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.466+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84c004f330 con 0x7f84c01504f0 2024-03-06T07:18:38.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.466+0000 7f84b67fc640 1 -- 172.21.15.94:0/1077181618 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f84b807f980 con 0x7f84c01504f0 2024-03-06T07:18:38.468 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:38.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f84a406b500 msgr2=0x7f84a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f84a406b500 0x7f84a406d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005880 tx=0x7f84bc009040 comp rx=0 tx=0).stop 2024-03-06T07:18:38.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c01504f0 msgr2=0x7f84c00acc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c01504f0 0x7f84c00acc10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009ee0 tx=0x7f84b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:38.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 shutdown_connections 2024-03-06T07:18:38.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f84a406b500 0x7f84a406d9c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84c01504f0 0x7f84c00acc10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84c00c3a70 0x7f84c00ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 --2- 172.21.15.94:0/1077181618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84c006d750 0x7f84c01eaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 >> 172.21.15.94:0/1077181618 conn(0x7f84c00c5560 msgr2=0x7f84c00701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 shutdown_connections 2024-03-06T07:18:38.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.468+0000 7f84c8113640 1 -- 172.21.15.94:0/1077181618 wait complete. 2024-03-06T07:18:38.481 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:38.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.581+0000 7fd8391c9640 1 Processor -- start 2024-03-06T07:18:38.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.581+0000 7fd8391c9640 1 -- start start 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd8391c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd83406c900 0x7fd8340c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd8391c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8340c3ef0 con 0x7fd8340ab170 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd8391c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8341556d0 con 0x7fd8340c5990 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd8391c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8341559b0 con 0x7fd83406c900 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd83359e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd8340ab170 0x7fd8340a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47264/0 (socket says 172.21.15.94:47264) 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd83359e640 1 -- 172.21.15.94:0/515246690 learned_addr learned my addr 172.21.15.94:0/515246690 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd83259c640 1 --2- 172.21.15.94:0/515246690 >> v2:172.21.15.145:3300/0 conn(0x7fd83406c900 0x7fd8340c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.582+0000 7fd831d9b640 1 -- 172.21.15.94:0/515246690 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324421440 0 0) 0x7fd8340c3ef0 con 0x7fd8340ab170 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd831d9b640 1 -- 172.21.15.94:0/515246690 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd81c003520 con 0x7fd8340ab170 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd831d9b640 1 -- 172.21.15.94:0/515246690 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241270167 0 0) 0x7fd8341556d0 con 0x7fd8340c5990 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd831d9b640 1 -- 172.21.15.94:0/515246690 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8340c3ef0 con 0x7fd8340c5990 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd83259c640 1 -- 172.21.15.94:0/515246690 >> v1:172.21.15.145:6789/0 conn(0x7fd8340c5990 legacy=0x7fd83406c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd83259c640 1 -- 172.21.15.94:0/515246690 >> v1:172.21.15.94:6789/0 conn(0x7fd8340ab170 legacy=0x7fd8340a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd83259c640 1 -- 172.21.15.94:0/515246690 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd834155c90 con 0x7fd83406c900 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd83259c640 1 --2- 172.21.15.94:0/515246690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd83406c900 0x7fd8340c0ab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8280097d0 tx=0x7fd82802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e04a5bfb6293bfc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.583+0000 7fd831d9b640 1 -- 172.21.15.94:0/515246690 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828034070 con 0x7fd83406c900 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd831d9b640 1 -- 172.21.15.94:0/515246690 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd828030610 con 0x7fd83406c900 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd831d9b640 1 -- 172.21.15.94:0/515246690 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8280309e0 con 0x7fd83406c900 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd8391c9640 1 -- 172.21.15.94:0/515246690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd83406c900 msgr2=0x7fd8340c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd8391c9640 1 --2- 172.21.15.94:0/515246690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd83406c900 0x7fd8340c0ab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8280097d0 tx=0x7fd82802b960 comp rx=0 tx=0).stop 2024-03-06T07:18:38.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd8391c9640 1 -- 172.21.15.94:0/515246690 shutdown_connections 2024-03-06T07:18:38.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd8391c9640 1 --2- 172.21.15.94:0/515246690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd83406c900 0x7fd8340c0ab0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd8391c9640 1 -- 172.21.15.94:0/515246690 >> 172.21.15.94:0/515246690 conn(0x7fd8340c6710 msgr2=0x7fd8340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:38.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd8391c9640 1 -- 172.21.15.94:0/515246690 shutdown_connections 2024-03-06T07:18:38.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.584+0000 7fd8391c9640 1 -- 172.21.15.94:0/515246690 wait complete. 2024-03-06T07:18:38.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.585+0000 7fd8391c9640 1 Processor -- start 2024-03-06T07:18:38.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.585+0000 7fd8391c9640 1 -- start start 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd8391c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd83406c900 0x7fd834154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd8391c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8340ab170 0x7fd8340c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd8391c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8340c5990 0x7fd8341ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd8391c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834156610 con 0x7fd8340ab170 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd8391c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834156490 con 0x7fd8340c5990 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd8391c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340c3af0 con 0x7fd83406c900 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd83259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8340ab170 0x7fd8340c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd83259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8340ab170 0x7fd8340c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37420/0 (socket says 172.21.15.94:37420) 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd83259c640 1 -- 172.21.15.94:0/2019833880 learned_addr learned my addr 172.21.15.94:0/2019833880 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd83359e640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8340c5990 0x7fd8341ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.586+0000 7fd832d9d640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd83406c900 0x7fd834154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd83259c640 1 -- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd83406c900 msgr2=0x7fd834154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd83259c640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd83406c900 0x7fd834154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd83259c640 1 -- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8340c5990 msgr2=0x7fd8341ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd83259c640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8340c5990 0x7fd8341ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd83259c640 1 -- 172.21.15.94:0/2019833880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd834155220 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd83259c640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8340ab170 0x7fd8340c4d00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8280098c0 tx=0x7fd8280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd81b7fe640 1 -- 172.21.15.94:0/2019833880 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828034040 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd81b7fe640 1 -- 172.21.15.94:0/2019833880 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd828030410 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8341557a0 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd81b7fe640 1 -- 172.21.15.94:0/2019833880 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd828006e10 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.587+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8341edfe0 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.588+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83404f330 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.588+0000 7fd81b7fe640 1 -- 172.21.15.94:0/2019833880 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd828037070 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.589+0000 7fd81b7fe640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82006b4d0 0x7fd82006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.589+0000 7fd81b7fe640 1 -- 172.21.15.94:0/2019833880 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd8280ac240 con 0x7fd8340ab170 2024-03-06T07:18:38.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.589+0000 7fd832d9d640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82006b4d0 0x7fd82006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.589+0000 7fd832d9d640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82006b4d0 0x7fd82006d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd824007fd0 tx=0x7fd824009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.593+0000 7fd81b7fe640 1 -- 172.21.15.94:0/2019833880 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd82807a7c0 con 0x7fd8340ab170 2024-03-06T07:18:38.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.765+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8340c6f00 con 0x7fd8340ab170 2024-03-06T07:18:38.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.765+0000 7fd81b7fe640 1 -- 172.21.15.94:0/2019833880 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd82807e860 con 0x7fd8340ab170 2024-03-06T07:18:38.767 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:38.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82006b4d0 msgr2=0x7fd82006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82006b4d0 0x7fd82006d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd824007fd0 tx=0x7fd824009040 comp rx=0 tx=0).stop 2024-03-06T07:18:38.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8340ab170 msgr2=0x7fd8340c4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8340ab170 0x7fd8340c4d00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8280098c0 tx=0x7fd8280046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 shutdown_connections 2024-03-06T07:18:38.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82006b4d0 0x7fd82006d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8340c5990 0x7fd8341ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8340ab170 0x7fd8340c4d00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 --2- 172.21.15.94:0/2019833880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd83406c900 0x7fd834154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 >> 172.21.15.94:0/2019833880 conn(0x7fd8340c6710 msgr2=0x7fd8341dbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:38.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 shutdown_connections 2024-03-06T07:18:38.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.767+0000 7fd8391c9640 1 -- 172.21.15.94:0/2019833880 wait complete. 2024-03-06T07:18:38.780 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:38.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.880+0000 7f92b87eb640 1 Processor -- start 2024-03-06T07:18:38.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.880+0000 7f92b87eb640 1 -- start start 2024-03-06T07:18:38.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b87eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b006dca0 0x7f92b0153420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b87eb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f92b0155520 0x7f92b01559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b87eb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00c2d80 con 0x7f92b014f580 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b87eb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b00c2b30 con 0x7f92b0155520 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b87eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00c3d70 con 0x7f92b006dca0 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b6d88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f92b0155520 0x7f92b01559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b5d86640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b006dca0 0x7f92b0153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b5d86640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b006dca0 0x7f92b0153420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:49556/0 (socket says 172.21.15.94:49556) 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b5d86640 1 -- 172.21.15.94:0/4087753004 learned_addr learned my addr 172.21.15.94:0/4087753004 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.881+0000 7f92b6d88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f92b0155520 0x7f92b01559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:35540/0 (socket says 172.21.15.94:35540) 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.882+0000 7f92b5d86640 1 -- 172.21.15.94:0/4087753004 >> v2:172.21.15.145:3300/0 conn(0x7f92b0155520 msgr2=0x7f92b01559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.882+0000 7f92b5d86640 1 --2- 172.21.15.94:0/4087753004 >> v2:172.21.15.145:3300/0 conn(0x7f92b0155520 0x7f92b01559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.882+0000 7f92b5d86640 1 -- 172.21.15.94:0/4087753004 >> v1:172.21.15.145:6789/0 conn(0x7f92b014f580 legacy=0x7f92b006f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.882+0000 7f92b5d86640 1 -- 172.21.15.94:0/4087753004 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00c3a00 con 0x7f92b006dca0 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.882+0000 7f92b5d86640 1 --2- 172.21.15.94:0/4087753004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b006dca0 0x7f92b0153420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009870 tx=0x7f92a402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2d4ef2047edd796 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b5585640 1 -- 172.21.15.94:0/4087753004 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a40317c0 con 0x7f92b006dca0 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b5585640 1 -- 172.21.15.94:0/4087753004 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a4037070 con 0x7f92b006dca0 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b5585640 1 -- 172.21.15.94:0/4087753004 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4031b80 con 0x7f92b006dca0 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 -- 172.21.15.94:0/4087753004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b006dca0 msgr2=0x7f92b0153420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 --2- 172.21.15.94:0/4087753004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b006dca0 0x7f92b0153420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009870 tx=0x7f92a402b860 comp rx=0 tx=0).stop 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 -- 172.21.15.94:0/4087753004 shutdown_connections 2024-03-06T07:18:38.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 --2- 172.21.15.94:0/4087753004 >> v2:172.21.15.145:3300/0 conn(0x7f92b0155520 0x7f92b01559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 --2- 172.21.15.94:0/4087753004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b006dca0 0x7f92b0153420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 -- 172.21.15.94:0/4087753004 >> 172.21.15.94:0/4087753004 conn(0x7f92b00a94d0 msgr2=0x7f92b00a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:38.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 -- 172.21.15.94:0/4087753004 shutdown_connections 2024-03-06T07:18:38.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.883+0000 7f92b87eb640 1 -- 172.21.15.94:0/4087753004 wait complete. 2024-03-06T07:18:38.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.884+0000 7f92b87eb640 1 Processor -- start 2024-03-06T07:18:38.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.884+0000 7f92b87eb640 1 -- start start 2024-03-06T07:18:38.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b87eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92b006dca0 0x7f92b01ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b87eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b014f580 0x7f92b014a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b87eb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92b0155520 0x7f92b014aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b87eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00c3060 con 0x7f92b006dca0 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b87eb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b014f960 con 0x7f92b0155520 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b87eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b0155ef0 con 0x7f92b014f580 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6587640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92b006dca0 0x7f92b01ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6587640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92b006dca0 0x7f92b01ee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37430/0 (socket says 172.21.15.94:37430) 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6587640 1 -- 172.21.15.94:0/707545199 learned_addr learned my addr 172.21.15.94:0/707545199 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b5d86640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b014f580 0x7f92b014a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6d88640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92b0155520 0x7f92b014aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6587640 1 -- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b014f580 msgr2=0x7f92b014a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6587640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b014f580 0x7f92b014a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6587640 1 -- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92b0155520 msgr2=0x7f92b014aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.885+0000 7f92b6587640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92b0155520 0x7f92b014aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92b6587640 1 -- 172.21.15.94:0/707545199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b01eed30 con 0x7f92b006dca0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92b5d86640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b014f580 0x7f92b014a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92b6587640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92b006dca0 0x7f92b01ee340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f92a800c7c0 tx=0x7f92a800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92a2ffd640 1 -- 172.21.15.94:0/707545199 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8016020 con 0x7f92b006dca0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92a2ffd640 1 -- 172.21.15.94:0/707545199 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a8006d50 con 0x7f92b006dca0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92a2ffd640 1 -- 172.21.15.94:0/707545199 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8003bb0 con 0x7f92b006dca0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b01ec070 con 0x7f92b006dca0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.886+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b01ec630 con 0x7f92b006dca0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.887+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b00735b0 con 0x7f92b006dca0 2024-03-06T07:18:38.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.887+0000 7f92a2ffd640 1 -- 172.21.15.94:0/707545199 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92a800a850 con 0x7f92b006dca0 2024-03-06T07:18:38.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.888+0000 7f92a2ffd640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f929406b4d0 0x7f929406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:38.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.888+0000 7f92a2ffd640 1 -- 172.21.15.94:0/707545199 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f92a808a070 con 0x7f92b006dca0 2024-03-06T07:18:38.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.888+0000 7f92b5d86640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f929406b4d0 0x7f929406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:38.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.888+0000 7f92b5d86640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f929406b4d0 0x7f929406d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92a4036000 tx=0x7f92a403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:38.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:38.892+0000 7f92a2ffd640 1 -- 172.21.15.94:0/707545199 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92b00735b0 con 0x7f92b006dca0 2024-03-06T07:18:39.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.061+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92b0067ad0 con 0x7f92b006dca0 2024-03-06T07:18:39.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.062+0000 7f92a2ffd640 1 -- 172.21.15.94:0/707545199 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f92b0067ad0 con 0x7f92b006dca0 2024-03-06T07:18:39.064 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f929406b4d0 msgr2=0x7f929406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f929406b4d0 0x7f929406d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92a4036000 tx=0x7f92a403d040 comp rx=0 tx=0).stop 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92b006dca0 msgr2=0x7f92b01ee340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92b006dca0 0x7f92b01ee340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f92a800c7c0 tx=0x7f92a800cc80 comp rx=0 tx=0).stop 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 shutdown_connections 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f929406b4d0 0x7f929406d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92b0155520 0x7f92b014aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92b014f580 0x7f92b014a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.063+0000 7f92b87eb640 1 --2- 172.21.15.94:0/707545199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92b006dca0 0x7f92b01ee340 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.064+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 >> 172.21.15.94:0/707545199 conn(0x7f92b00a94d0 msgr2=0x7f92b00a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.064+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 shutdown_connections 2024-03-06T07:18:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.064+0000 7f92b87eb640 1 -- 172.21.15.94:0/707545199 wait complete. 2024-03-06T07:18:39.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:39.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.180+0000 7f8d5d0c3640 1 Processor -- start 2024-03-06T07:18:39.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.180+0000 7f8d5d0c3640 1 -- start start 2024-03-06T07:18:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d5d0c3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8d58072cd0 0x7f8d58155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d5d0c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d58073290 0x7f8d58155660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d5d0c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d58152db0 con 0x7f8d581529d0 2024-03-06T07:18:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d5d0c3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d580c6dd0 con 0x7f8d58072cd0 2024-03-06T07:18:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d5d0c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580c6940 con 0x7f8d58073290 2024-03-06T07:18:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d4ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8d58072cd0 0x7f8d58155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d4ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8d58072cd0 0x7f8d58155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37444/0 (socket says 172.21.15.94:37444) 2024-03-06T07:18:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2789398369 learned_addr learned my addr 172.21.15.94:0/2789398369 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d5759e640 1 --2- 172.21.15.94:0/2789398369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d58073290 0x7f8d58155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2789398369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d58073290 msgr2=0x7f8d58155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d4ffff640 1 --2- 172.21.15.94:0/2789398369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d58073290 0x7f8d58155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2789398369 >> v1:172.21.15.145:6789/0 conn(0x7f8d581529d0 legacy=0x7f8d580c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.181+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2789398369 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d581590f0 con 0x7f8d58072cd0 2024-03-06T07:18:39.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.182+0000 7f8d4ffff640 1 --2- 172.21.15.94:0/2789398369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58072cd0 0x7f8d58155120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009870 tx=0x7f8d4002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af19522e9286f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.182+0000 7f8d5659c640 1 -- 172.21.15.94:0/2789398369 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d400317c0 con 0x7f8d58072cd0 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.182+0000 7f8d5659c640 1 -- 172.21.15.94:0/2789398369 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d40037070 con 0x7f8d58072cd0 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.182+0000 7f8d5659c640 1 -- 172.21.15.94:0/2789398369 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40031b80 con 0x7f8d58072cd0 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.182+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2789398369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58072cd0 msgr2=0x7f8d58155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.182+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2789398369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58072cd0 0x7f8d58155120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009870 tx=0x7f8d4002b860 comp rx=0 tx=0).stop 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.183+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2789398369 shutdown_connections 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.183+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2789398369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d58073290 0x7f8d58155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.183+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2789398369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58072cd0 0x7f8d58155120 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.183+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2789398369 >> 172.21.15.94:0/2789398369 conn(0x7f8d580aa630 msgr2=0x7f8d580aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.183+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2789398369 shutdown_connections 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.183+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2789398369 wait complete. 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.184+0000 7f8d5d0c3640 1 Processor -- start 2024-03-06T07:18:39.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.184+0000 7f8d5d0c3640 1 -- start start 2024-03-06T07:18:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.184+0000 7f8d5d0c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d58072cd0 0x7f8d581564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5d0c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58073290 0x7f8d58151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5d0c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d581529d0 0x7f8d58144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5d0c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d58159370 con 0x7f8d58073290 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5d0c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d58072720 con 0x7f8d58072cd0 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5d0c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d580723b0 con 0x7f8d581529d0 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58073290 0x7f8d58151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58073290 0x7f8d58151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37460/0 (socket says 172.21.15.94:37460) 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2301520971 learned_addr learned my addr 172.21.15.94:0/2301520971 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d56d9d640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d58072cd0 0x7f8d581564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5759e640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d581529d0 0x7f8d58144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d581529d0 msgr2=0x7f8d58144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d581529d0 0x7f8d58144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d58072cd0 msgr2=0x7f8d581564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d58072cd0 0x7f8d581564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 -- 172.21.15.94:0/2301520971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d580ad040 con 0x7f8d58073290 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4ffff640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58073290 0x7f8d58151a50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000c00 tx=0x7f8d400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4f7fe640 1 -- 172.21.15.94:0/2301520971 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40032650 con 0x7f8d58073290 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4f7fe640 1 -- 172.21.15.94:0/2301520971 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d400327f0 con 0x7f8d58073290 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d4f7fe640 1 -- 172.21.15.94:0/2301520971 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40032be0 con 0x7f8d58073290 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d581eefa0 con 0x7f8d58073290 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.185+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5806d470 con 0x7f8d58073290 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.186+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5804f330 con 0x7f8d58073290 2024-03-06T07:18:39.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.187+0000 7f8d4f7fe640 1 -- 172.21.15.94:0/2301520971 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d4004c060 con 0x7f8d58073290 2024-03-06T07:18:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.187+0000 7f8d4f7fe640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8d3806b4d0 0x7f8d3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.187+0000 7f8d4f7fe640 1 -- 172.21.15.94:0/2301520971 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8d400abf60 con 0x7f8d58073290 2024-03-06T07:18:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.187+0000 7f8d56d9d640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8d3806b4d0 0x7f8d3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.188+0000 7f8d56d9d640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8d3806b4d0 0x7f8d3806d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8d58152300 tx=0x7f8d48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.191+0000 7f8d4f7fe640 1 -- 172.21.15.94:0/2301520971 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d40040070 con 0x7f8d58073290 2024-03-06T07:18:39.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.361+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8d580c7010 con 0x7f8d58073290 2024-03-06T07:18:39.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.362+0000 7f8d4f7fe640 1 -- 172.21.15.94:0/2301520971 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8d581590f0 con 0x7f8d58073290 2024-03-06T07:18:39.364 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8d3806b4d0 msgr2=0x7f8d3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8d3806b4d0 0x7f8d3806d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8d58152300 tx=0x7f8d48009040 comp rx=0 tx=0).stop 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58073290 msgr2=0x7f8d58151a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58073290 0x7f8d58151a50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000c00 tx=0x7f8d400046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 shutdown_connections 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8d3806b4d0 0x7f8d3806d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d581529d0 0x7f8d58144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d58073290 0x7f8d58151a50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 --2- 172.21.15.94:0/2301520971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d58072cd0 0x7f8d581564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 >> 172.21.15.94:0/2301520971 conn(0x7f8d580aa630 msgr2=0x7f8d580c4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:39.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 shutdown_connections 2024-03-06T07:18:39.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.364+0000 7f8d5d0c3640 1 -- 172.21.15.94:0/2301520971 wait complete. 2024-03-06T07:18:39.377 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:39.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.475+0000 7fd7a1888640 1 Processor -- start 2024-03-06T07:18:39.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.475+0000 7fd7a1888640 1 -- start start 2024-03-06T07:18:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.475+0000 7fd7a1888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.475+0000 7fd7a1888640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd79c158b60 0x7fd79c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.475+0000 7fd7a1888640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c0c73d0 con 0x7fd79c0c88c0 2024-03-06T07:18:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.475+0000 7fd7a1888640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd79c0c8dd0 con 0x7fd79c158b60 2024-03-06T07:18:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.475+0000 7fd7a1888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0be2e0 con 0x7fd79c14f360 2024-03-06T07:18:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79a7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd79c0c88c0 0x7fd79c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47266/0 (socket says 172.21.15.94:47266) 2024-03-06T07:18:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79a7fc640 1 -- 172.21.15.94:0/1057498825 learned_addr learned my addr 172.21.15.94:0/1057498825 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79affd640 1 --2- 172.21.15.94:0/1057498825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79b7fe640 1 --2- 172.21.15.94:0/1057498825 >> v2:172.21.15.145:3300/0 conn(0x7fd79c158b60 0x7fd79c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd799ffb640 1 -- 172.21.15.94:0/1057498825 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230086279 0 0) 0x7fd79c0c73d0 con 0x7fd79c0c88c0 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd799ffb640 1 -- 172.21.15.94:0/1057498825 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd784003520 con 0x7fd79c0c88c0 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79affd640 1 -- 172.21.15.94:0/1057498825 >> v2:172.21.15.145:3300/0 conn(0x7fd79c158b60 msgr2=0x7fd79c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79affd640 1 --2- 172.21.15.94:0/1057498825 >> v2:172.21.15.145:3300/0 conn(0x7fd79c158b60 0x7fd79c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79affd640 1 -- 172.21.15.94:0/1057498825 >> v1:172.21.15.94:6789/0 conn(0x7fd79c0c88c0 legacy=0x7fd79c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.476+0000 7fd79affd640 1 -- 172.21.15.94:0/1057498825 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c0c7670 con 0x7fd79c14f360 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd79affd640 1 --2- 172.21.15.94:0/1057498825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c155c10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0097a0 tx=0x7fd78c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f46bec4de12a7bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd799ffb640 1 -- 172.21.15.94:0/1057498825 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0315a0 con 0x7fd79c14f360 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd799ffb640 1 -- 172.21.15.94:0/1057498825 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c037070 con 0x7fd79c14f360 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd799ffb640 1 -- 172.21.15.94:0/1057498825 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c031940 con 0x7fd79c14f360 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 -- 172.21.15.94:0/1057498825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 msgr2=0x7fd79c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 --2- 172.21.15.94:0/1057498825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c155c10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0097a0 tx=0x7fd78c02b840 comp rx=0 tx=0).stop 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 -- 172.21.15.94:0/1057498825 shutdown_connections 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 --2- 172.21.15.94:0/1057498825 >> v2:172.21.15.145:3300/0 conn(0x7fd79c158b60 0x7fd79c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 --2- 172.21.15.94:0/1057498825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c155c10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 -- 172.21.15.94:0/1057498825 >> 172.21.15.94:0/1057498825 conn(0x7fd79c0ac570 msgr2=0x7fd79c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 -- 172.21.15.94:0/1057498825 shutdown_connections 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.477+0000 7fd7a1888640 1 -- 172.21.15.94:0/1057498825 wait complete. 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.478+0000 7fd7a1888640 1 Processor -- start 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd7a1888640 1 -- start start 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd7a1888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd79c0c88c0 0x7fd79c1ea810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd7a1888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c0c2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd7a1888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd79c158b60 0x7fd79c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd7a1888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0c5a50 con 0x7fd79c0c88c0 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd7a1888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0c7c30 con 0x7fd79c158b60 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd7a1888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c0c5bd0 con 0x7fd79c14f360 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd79affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd79c0c88c0 0x7fd79c1ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd79affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd79c0c88c0 0x7fd79c1ea810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37468/0 (socket says 172.21.15.94:37468) 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd79affd640 1 -- 172.21.15.94:0/3499085677 learned_addr learned my addr 172.21.15.94:0/3499085677 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd79b7fe640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd79c158b60 0x7fd79c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.479+0000 7fd79a7fc640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c0c2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd79affd640 1 -- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 msgr2=0x7fd79c0c2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd79affd640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c0c2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd79affd640 1 -- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd79c158b60 msgr2=0x7fd79c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd79affd640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd79c158b60 0x7fd79c0afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd79affd640 1 -- 172.21.15.94:0/3499085677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c14f740 con 0x7fd79c0c88c0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd79affd640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd79c0c88c0 0x7fd79c1ea810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd78c034040 tx=0x7fd78c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd783fff640 1 -- 172.21.15.94:0/3499085677 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c032430 con 0x7fd79c0c88c0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd783fff640 1 -- 172.21.15.94:0/3499085677 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c0325d0 con 0x7fd79c0c88c0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79c0ad5e0 con 0x7fd79c0c88c0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd783fff640 1 -- 172.21.15.94:0/3499085677 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c032980 con 0x7fd79c0c88c0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.480+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79c0adba0 con 0x7fd79c0c88c0 2024-03-06T07:18:39.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.481+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79c04f330 con 0x7fd79c0c88c0 2024-03-06T07:18:39.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.481+0000 7fd783fff640 1 -- 172.21.15.94:0/3499085677 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd78c032b20 con 0x7fd79c0c88c0 2024-03-06T07:18:39.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.482+0000 7fd783fff640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd78806b4d0 0x7fd78806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.482+0000 7fd783fff640 1 -- 172.21.15.94:0/3499085677 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd78c02f030 con 0x7fd79c0c88c0 2024-03-06T07:18:39.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.482+0000 7fd79a7fc640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd78806b4d0 0x7fd78806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.482+0000 7fd79a7fc640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd78806b4d0 0x7fd78806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd790005880 tx=0x7fd790009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.486+0000 7fd783fff640 1 -- 172.21.15.94:0/3499085677 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd78c040070 con 0x7fd79c0c88c0 2024-03-06T07:18:39.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.693+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd79c06cda0 con 0x7fd79c0c88c0 2024-03-06T07:18:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.694+0000 7fd783fff640 1 -- 172.21.15.94:0/3499085677 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7fd78c07a680 con 0x7fd79c0c88c0 2024-03-06T07:18:39.696 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:39.696 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:18:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd78806b4d0 msgr2=0x7fd78806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd78806b4d0 0x7fd78806d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd790005880 tx=0x7fd790009040 comp rx=0 tx=0).stop 2024-03-06T07:18:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd79c0c88c0 msgr2=0x7fd79c1ea810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd79c0c88c0 0x7fd79c1ea810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd78c034040 tx=0x7fd78c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 shutdown_connections 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd78806b4d0 0x7fd78806d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd79c158b60 0x7fd79c0afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd79c14f360 0x7fd79c0c2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 --2- 172.21.15.94:0/3499085677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd79c0c88c0 0x7fd79c1ea810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 >> 172.21.15.94:0/3499085677 conn(0x7fd79c0ac570 msgr2=0x7fd79c153310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 shutdown_connections 2024-03-06T07:18:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.696+0000 7fd7a1888640 1 -- 172.21.15.94:0/3499085677 wait complete. 2024-03-06T07:18:39.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:39.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f702d69d640 1 Processor -- start 2024-03-06T07:18:39.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f702d69d640 1 -- start start 2024-03-06T07:18:39.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f702d69d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f7028155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f702d69d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70280c73d0 con 0x7f70280c88c0 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f702d69d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70280be2e0 con 0x7f702814f360 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f702d69d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7028159700 con 0x7f7028159270 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f70267fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f70280c88c0 0x7f7028158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47268/0 (socket says 172.21.15.94:47268) 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f70267fc640 1 -- 172.21.15.94:0/2236986064 learned_addr learned my addr 172.21.15.94:0/2236986064 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.807+0000 7f70277fe640 1 --2- 172.21.15.94:0/2236986064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f7028155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.808+0000 7f7025ffb640 1 -- 172.21.15.94:0/2236986064 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27265723 0 0) 0x7f70280c73d0 con 0x7f70280c88c0 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.808+0000 7f7025ffb640 1 -- 172.21.15.94:0/2236986064 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7010003520 con 0x7f70280c88c0 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.808+0000 7f7025ffb640 1 -- 172.21.15.94:0/2236986064 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061541684 0 0) 0x7f70280be2e0 con 0x7f702814f360 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.808+0000 7f7025ffb640 1 -- 172.21.15.94:0/2236986064 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70280c73d0 con 0x7f702814f360 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.808+0000 7f70277fe640 1 -- 172.21.15.94:0/2236986064 >> v1:172.21.15.145:6789/0 conn(0x7f702814f360 legacy=0x7f70280c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.808+0000 7f70277fe640 1 -- 172.21.15.94:0/2236986064 >> v1:172.21.15.94:6789/0 conn(0x7f70280c88c0 legacy=0x7f7028158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.808+0000 7f70277fe640 1 -- 172.21.15.94:0/2236986064 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70280c7670 con 0x7f7028159270 2024-03-06T07:18:39.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f70277fe640 1 --2- 172.21.15.94:0/2236986064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f7028155c10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7020009dc0 tx=0x7f702002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39aec6eb9c5102b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f7025ffb640 1 -- 172.21.15.94:0/2236986064 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020039070 con 0x7f7028159270 2024-03-06T07:18:39.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f7025ffb640 1 -- 172.21.15.94:0/2236986064 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702002cc40 con 0x7f7028159270 2024-03-06T07:18:39.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f7025ffb640 1 -- 172.21.15.94:0/2236986064 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020034440 con 0x7f7028159270 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f702d69d640 1 -- 172.21.15.94:0/2236986064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 msgr2=0x7f7028155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f702d69d640 1 --2- 172.21.15.94:0/2236986064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f7028155c10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7020009dc0 tx=0x7f702002be90 comp rx=0 tx=0).stop 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f702d69d640 1 -- 172.21.15.94:0/2236986064 shutdown_connections 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f702d69d640 1 --2- 172.21.15.94:0/2236986064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f7028155c10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f702d69d640 1 -- 172.21.15.94:0/2236986064 >> 172.21.15.94:0/2236986064 conn(0x7f70280ac570 msgr2=0x7f70280ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f702d69d640 1 -- 172.21.15.94:0/2236986064 shutdown_connections 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.809+0000 7f702d69d640 1 -- 172.21.15.94:0/2236986064 wait complete. 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 Processor -- start 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 -- start start 2024-03-06T07:18:39.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70280c88c0 0x7f70280af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f702814f360 0x7f702814b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f702814bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70280c7c30 con 0x7f70280c88c0 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70280c5a50 con 0x7f702814f360 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f702d69d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70280c5bd0 con 0x7f7028159270 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f7026ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70280c88c0 0x7f70280af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f7026ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70280c88c0 0x7f70280af4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37474/0 (socket says 172.21.15.94:37474) 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f7026ffd640 1 -- 172.21.15.94:0/2315988096 learned_addr learned my addr 172.21.15.94:0/2315988096 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f70267fc640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f702814f360 0x7f702814b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.811+0000 7f70277fe640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f702814bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f7026ffd640 1 -- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 msgr2=0x7f702814bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f7026ffd640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f702814bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f7026ffd640 1 -- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f702814f360 msgr2=0x7f702814b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f7026ffd640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f702814f360 0x7f702814b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f7026ffd640 1 -- 172.21.15.94:0/2315988096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70281eb8a0 con 0x7f70280c88c0 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f7026ffd640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70280c88c0 0x7f70280af4a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f701c00c530 tx=0x7f701c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f700f7fe640 1 -- 172.21.15.94:0/2315988096 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c0052d0 con 0x7f70280c88c0 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f700f7fe640 1 -- 172.21.15.94:0/2315988096 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c005490 con 0x7f70280c88c0 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f700f7fe640 1 -- 172.21.15.94:0/2315988096 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c012870 con 0x7f70280c88c0 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70280b0280 con 0x7f70280c88c0 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.812+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70280ad320 con 0x7f70280c88c0 2024-03-06T07:18:39.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.813+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702804f330 con 0x7f70280c88c0 2024-03-06T07:18:39.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.813+0000 7f700f7fe640 1 -- 172.21.15.94:0/2315988096 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f701c012a10 con 0x7f70280c88c0 2024-03-06T07:18:39.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.814+0000 7f700f7fe640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f701406b500 0x7f701406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:39.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.814+0000 7f700f7fe640 1 -- 172.21.15.94:0/2315988096 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f701c08a000 con 0x7f70280c88c0 2024-03-06T07:18:39.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.814+0000 7f70267fc640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f701406b500 0x7f701406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:39.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.814+0000 7f70267fc640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f701406b500 0x7f701406d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7018000f60 tx=0x7f7018007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:39.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.818+0000 7f700f7fe640 1 -- 172.21.15.94:0/2315988096 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f701c058500 con 0x7f70280c88c0 2024-03-06T07:18:39.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.989+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70280c8d50 con 0x7f70280c88c0 2024-03-06T07:18:39.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.990+0000 7f700f7fe640 1 -- 172.21.15.94:0/2315988096 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f701c05c5a0 con 0x7f70280c88c0 2024-03-06T07:18:39.992 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:39.992 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f701406b500 msgr2=0x7f701406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f701406b500 0x7f701406d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7018000f60 tx=0x7f7018007d30 comp rx=0 tx=0).stop 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70280c88c0 msgr2=0x7f70280af4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70280c88c0 0x7f70280af4a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f701c00c530 tx=0x7f701c00c9f0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 shutdown_connections 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f701406b500 0x7f701406d9c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028159270 0x7f702814bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f702814f360 0x7f702814b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 --2- 172.21.15.94:0/2315988096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70280c88c0 0x7f70280af4a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:39.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 >> 172.21.15.94:0/2315988096 conn(0x7f70280ac570 msgr2=0x7f7028154a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:39.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 shutdown_connections 2024-03-06T07:18:39.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:39.992+0000 7f702d69d640 1 -- 172.21.15.94:0/2315988096 wait complete. 2024-03-06T07:18:40.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:40.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.104+0000 7fcc95f81640 1 Processor -- start 2024-03-06T07:18:40.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.104+0000 7fcc95f81640 1 -- start start 2024-03-06T07:18:40.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.104+0000 7fcc95f81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcc900c7180 0x7fcc90159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.104+0000 7fcc95f81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcc900c88b0 0x7fcc900c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.104+0000 7fcc95f81640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc900c7c60 con 0x7fcc901527b0 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.104+0000 7fcc95f81640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc90159640 con 0x7fcc900c7180 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.104+0000 7fcc95f81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc900c8060 con 0x7fcc900c88b0 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc87fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcc900c7180 0x7fcc90159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc87fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcc900c7180 0x7fcc90159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37490/0 (socket says 172.21.15.94:37490) 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc87fff640 1 -- 172.21.15.94:0/3981215379 learned_addr learned my addr 172.21.15.94:0/3981215379 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc8f7fe640 1 --1- 172.21.15.94:0/3981215379 >> v1:172.21.15.94:6789/0 conn(0x7fcc901527b0 0x7fcc90155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47284/0 (socket says 172.21.15.94:47284) 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc8ffff640 1 --2- 172.21.15.94:0/3981215379 >> v2:172.21.15.145:3300/0 conn(0x7fcc900c88b0 0x7fcc900c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc87fff640 1 -- 172.21.15.94:0/3981215379 >> v2:172.21.15.145:3300/0 conn(0x7fcc900c88b0 msgr2=0x7fcc900c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc87fff640 1 --2- 172.21.15.94:0/3981215379 >> v2:172.21.15.145:3300/0 conn(0x7fcc900c88b0 0x7fcc900c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc87fff640 1 -- 172.21.15.94:0/3981215379 >> v1:172.21.15.94:6789/0 conn(0x7fcc901527b0 legacy=0x7fcc90155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc87fff640 1 -- 172.21.15.94:0/3981215379 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900c76a0 con 0x7fcc900c7180 2024-03-06T07:18:40.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.105+0000 7fcc8effd640 1 -- 172.21.15.94:0/3981215379 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548161206 0 0) 0x7fcc80001110 con 0x7fcc901527b0 2024-03-06T07:18:40.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc87fff640 1 --2- 172.21.15.94:0/3981215379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c7180 0x7fcc90159100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc900ad600 tx=0x7fcc7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be299723c05feca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc8effd640 1 -- 172.21.15.94:0/3981215379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78035020 con 0x7fcc900c7180 2024-03-06T07:18:40.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc8effd640 1 -- 172.21.15.94:0/3981215379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78031870 con 0x7fcc900c7180 2024-03-06T07:18:40.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc8effd640 1 -- 172.21.15.94:0/3981215379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78031c40 con 0x7fcc900c7180 2024-03-06T07:18:40.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 -- 172.21.15.94:0/3981215379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c7180 msgr2=0x7fcc90159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3981215379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c7180 0x7fcc90159100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc900ad600 tx=0x7fcc7802b910 comp rx=0 tx=0).stop 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 -- 172.21.15.94:0/3981215379 shutdown_connections 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3981215379 >> v2:172.21.15.145:3300/0 conn(0x7fcc900c88b0 0x7fcc900c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3981215379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c7180 0x7fcc90159100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 -- 172.21.15.94:0/3981215379 >> 172.21.15.94:0/3981215379 conn(0x7fcc900ac560 msgr2=0x7fcc900ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 -- 172.21.15.94:0/3981215379 shutdown_connections 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.106+0000 7fcc95f81640 1 -- 172.21.15.94:0/3981215379 wait complete. 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.107+0000 7fcc95f81640 1 Processor -- start 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.107+0000 7fcc95f81640 1 -- start start 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc95f81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc900c7180 0x7fcc90157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc95f81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c88b0 0x7fcc901eabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc95f81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901527b0 0x7fcc901eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc95f81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90155600 con 0x7fcc900c88b0 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc95f81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90155320 con 0x7fcc900c7180 2024-03-06T07:18:40.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc95f81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90152b90 con 0x7fcc901527b0 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c88b0 0x7fcc901eabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c88b0 0x7fcc901eabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37498/0 (socket says 172.21.15.94:37498) 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 -- 172.21.15.94:0/3037337631 learned_addr learned my addr 172.21.15.94:0/3037337631 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc8f7fe640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc900c7180 0x7fcc90157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc8ffff640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901527b0 0x7fcc901eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 -- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901527b0 msgr2=0x7fcc901eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901527b0 0x7fcc901eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 -- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc900c7180 msgr2=0x7fcc90157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc900c7180 0x7fcc90157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.108+0000 7fcc87fff640 1 -- 172.21.15.94:0/3037337631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc901ee100 con 0x7fcc900c88b0 2024-03-06T07:18:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.109+0000 7fcc87fff640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c88b0 0x7fcc901eabf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc7802b950 tx=0x7fcc780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.109+0000 7fcc8cff9640 1 -- 172.21.15.94:0/3037337631 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78040070 con 0x7fcc900c88b0 2024-03-06T07:18:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.109+0000 7fcc8cff9640 1 -- 172.21.15.94:0/3037337631 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78030d00 con 0x7fcc900c88b0 2024-03-06T07:18:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.109+0000 7fcc8cff9640 1 -- 172.21.15.94:0/3037337631 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78035040 con 0x7fcc900c88b0 2024-03-06T07:18:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.109+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc901ee3e0 con 0x7fcc900c88b0 2024-03-06T07:18:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.109+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc901ee9a0 con 0x7fcc900c88b0 2024-03-06T07:18:40.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.110+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9004f330 con 0x7fcc900c88b0 2024-03-06T07:18:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.110+0000 7fcc8cff9640 1 -- 172.21.15.94:0/3037337631 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc7804b060 con 0x7fcc900c88b0 2024-03-06T07:18:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.111+0000 7fcc8cff9640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7006b4d0 0x7fcc7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.111+0000 7fcc8cff9640 1 -- 172.21.15.94:0/3037337631 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcc780ac0a0 con 0x7fcc900c88b0 2024-03-06T07:18:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.111+0000 7fcc8f7fe640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7006b4d0 0x7fcc7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.111+0000 7fcc8f7fe640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7006b4d0 0x7fcc7006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc80007fd0 tx=0x7fcc80007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.115+0000 7fcc8cff9640 1 -- 172.21.15.94:0/3037337631 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc7807a620 con 0x7fcc900c88b0 2024-03-06T07:18:40.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.321+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc9006ce00 con 0x7fcc900c88b0 2024-03-06T07:18:40.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.322+0000 7fcc8cff9640 1 -- 172.21.15.94:0/3037337631 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7fcc7807e6c0 con 0x7fcc900c88b0 2024-03-06T07:18:40.324 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:40.324 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:40.324 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7006b4d0 msgr2=0x7fcc7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7006b4d0 0x7fcc7006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc80007fd0 tx=0x7fcc80007de0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c88b0 msgr2=0x7fcc901eabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c88b0 0x7fcc901eabf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc7802b950 tx=0x7fcc780046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 shutdown_connections 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7006b4d0 0x7fcc7006d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901527b0 0x7fcc901eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c88b0 0x7fcc901eabf0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 --2- 172.21.15.94:0/3037337631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc900c7180 0x7fcc90157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 >> 172.21.15.94:0/3037337631 conn(0x7fcc900ac560 msgr2=0x7fcc900c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 shutdown_connections 2024-03-06T07:18:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.324+0000 7fcc95f81640 1 -- 172.21.15.94:0/3037337631 wait complete. 2024-03-06T07:18:40.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:18:08.851499+0000', 'modified': '2024-03-06T07:18:31.055922+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34136': {'gid': 34136, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/2223795314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2223795314}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2223795314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34136, 'qdb_cluster': [34136]} 2024-03-06T07:18:40.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:18:40.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:18:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.840+0000 7fd7cb4ed640 1 Processor -- start 2024-03-06T07:18:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.841+0000 7fd7cb4ed640 1 -- start start 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.841+0000 7fd7cb4ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c40c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.841+0000 7fd7cb4ed640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd7c40c30b0 0x7fd7c414f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.841+0000 7fd7cb4ed640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c4152030 con 0x7fd7c414f660 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.841+0000 7fd7cb4ed640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c4152920 con 0x7fd7c40c30b0 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.841+0000 7fd7cb4ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c4152db0 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9289640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c40c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c8a88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd7c40c30b0 0x7fd7c414f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9289640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c40c3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:49614/0 (socket says 172.21.15.94:49614) 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c8a88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd7c40c30b0 0x7fd7c414f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:35618/0 (socket says 172.21.15.94:35618) 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9289640 1 -- 172.21.15.94:0/3354769788 learned_addr learned my addr 172.21.15.94:0/3354769788 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9a8a640 1 --1- 172.21.15.94:0/3354769788 >> v1:172.21.15.145:6789/0 conn(0x7fd7c414f660 0x7fd7c4155510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49202/0 (socket says 172.21.15.94:49202) 2024-03-06T07:18:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9289640 1 -- 172.21.15.94:0/3354769788 >> v2:172.21.15.145:3300/0 conn(0x7fd7c40c30b0 msgr2=0x7fd7c414f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3354769788 >> v2:172.21.15.145:3300/0 conn(0x7fd7c40c30b0 0x7fd7c414f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9289640 1 -- 172.21.15.94:0/3354769788 >> v1:172.21.15.145:6789/0 conn(0x7fd7c414f660 legacy=0x7fd7c4155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.842+0000 7fd7c9289640 1 -- 172.21.15.94:0/3354769788 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c41522d0 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3354769788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c40c3780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4152200 tx=0x7fd7c002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff623c92176851f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7bbfff640 1 -- 172.21.15.94:0/3354769788 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c002c9c0 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7bbfff640 1 -- 172.21.15.94:0/3354769788 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c002cb60 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7bbfff640 1 -- 172.21.15.94:0/3354769788 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00343f0 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/3354769788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 msgr2=0x7fd7c40c3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/3354769788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c40c3780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4152200 tx=0x7fd7c002be20 comp rx=0 tx=0).stop 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/3354769788 shutdown_connections 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/3354769788 >> v2:172.21.15.145:3300/0 conn(0x7fd7c40c30b0 0x7fd7c414f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/3354769788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c40c3780 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/3354769788 >> 172.21.15.94:0/3354769788 conn(0x7fd7c40c7de0 msgr2=0x7fd7c40c4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/3354769788 shutdown_connections 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.843+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/3354769788 wait complete. 2024-03-06T07:18:40.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.844+0000 7fd7cb4ed640 1 Processor -- start 2024-03-06T07:18:40.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7cb4ed640 1 -- start start 2024-03-06T07:18:40.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7cb4ed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7c40c30b0 0x7fd7c40ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7cb4ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c4148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7cb4ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7c414f660 0x7fd7c41eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7cb4ed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40c3cc0 con 0x7fd7c414f660 2024-03-06T07:18:40.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7cb4ed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40c3fd0 con 0x7fd7c40c30b0 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7cb4ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40c4f90 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7c9a8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7c414f660 0x7fd7c41eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7c8a88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c4148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7c9a8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7c414f660 0x7fd7c41eb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37512/0 (socket says 172.21.15.94:37512) 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/564071076 learned_addr learned my addr 172.21.15.94:0/564071076 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.845+0000 7fd7c9289640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7c40c30b0 0x7fd7c40ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c8a88640 1 -- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7c40c30b0 msgr2=0x7fd7c40ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7c40c30b0 0x7fd7c40ad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c8a88640 1 -- 172.21.15.94:0/564071076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7c414f660 msgr2=0x7fd7c41eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7c414f660 0x7fd7c41eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c8a88640 1 -- 172.21.15.94:0/564071076 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4151ae0 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7c414f660 0x7fd7c41eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c9289640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7c40c30b0 0x7fd7c40ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c4148880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400e820 tx=0x7fd7b400ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7b9ffb640 1 -- 172.21.15.94:0/564071076 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4017070 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7b9ffb640 1 -- 172.21.15.94:0/564071076 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b400c940 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c4150670 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7b9ffb640 1 -- 172.21.15.94:0/564071076 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400cd30 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.846+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c4150bf0 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.848+0000 7fd7b9ffb640 1 -- 172.21.15.94:0/564071076 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7b401a070 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.848+0000 7fd7b9ffb640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd7a806b500 0x7fd7a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.848+0000 7fd7b9ffb640 1 -- 172.21.15.94:0/564071076 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd7b408a300 con 0x7fd7c40c4bb0 2024-03-06T07:18:40.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.848+0000 7fd7c9289640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd7a806b500 0x7fd7a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.849+0000 7fd7c9289640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd7a806b500 0x7fd7a806d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0035000 tx=0x7fd7c0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.849+0000 7fd7ce1b3740 1 Processor -- start 2024-03-06T07:18:40.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.849+0000 7fd7ce1b3740 1 -- start start 2024-03-06T07:18:40.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb478f50 0x5562fb476a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb479330 0x5562fb476f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb4774b0 0x5562fb473680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562fb3dd6b0 con 0x5562fb4774b0 2024-03-06T07:18:40.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562fb386a60 con 0x5562fb479330 2024-03-06T07:18:40.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562fb3d9750 con 0x5562fb478f50 2024-03-06T07:18:40.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb4774b0 0x5562fb473680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb4774b0 0x5562fb473680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37514/0 (socket says 172.21.15.94:37514) 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/3190596587 learned_addr learned my addr 172.21.15.94:0/3190596587 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c8a88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb479330 0x5562fb476f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb478f50 0x5562fb476a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/3190596587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb478f50 msgr2=0x5562fb476a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb478f50 0x5562fb476a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/3190596587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb479330 msgr2=0x5562fb476f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb479330 0x5562fb476f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/3190596587 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562fb3a4f30 con 0x5562fb4774b0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb4774b0 0x5562fb473680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc007710 tx=0x7fd7bc007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac6bcf6302688f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7aeffd640 1 -- 172.21.15.94:0/3190596587 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc001d00 con 0x5562fb4774b0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.850+0000 7fd7aeffd640 1 -- 172.21.15.94:0/3190596587 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc018070 con 0x5562fb4774b0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7aeffd640 1 -- 172.21.15.94:0/3190596587 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0135f0 con 0x5562fb4774b0 2024-03-06T07:18:40.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3190596587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb4774b0 msgr2=0x5562fb473680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb4774b0 0x5562fb473680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc007710 tx=0x7fd7bc007800 comp rx=0 tx=0).stop 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3190596587 shutdown_connections 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb4774b0 0x5562fb473680 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb479330 0x5562fb476f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3190596587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb478f50 0x5562fb476a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3190596587 >> 172.21.15.94:0/3190596587 conn(0x5562fb4757d0 msgr2=0x5562fb49a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3190596587 shutdown_connections 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.851+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3190596587 wait complete. 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 -- start start 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 Processor -- start 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb4953b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb49bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb49d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562fb49b230 con 0x5562fb479330 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562fb399ea0 con 0x5562fb478f50 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562fb49ac70 con 0x5562fb49c960 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.852+0000 7fd7ce1b3740 10 client.0 _ll_register_callbacks cb 0x5562fb399870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb49bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c9289640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb4953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb49bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37522/0 (socket says 172.21.15.94:37522) 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3298607589 learned_addr learned my addr 172.21.15.94:0/3298607589 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb49d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 msgr2=0x5562fb49d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb49d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 msgr2=0x5562fb4953b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb4953b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3298607589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562fb39a270 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb49bcf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400cfd0 tx=0x7fd7b4058880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9151135c818df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3298607589 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4070020 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3298607589 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd7b40701c0 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7bc018070 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3298607589 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b4021070 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7ce1b3740 10 client.4770 fetch_fsmap learned FSMap version 9 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x5562fb0f22d0 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3298607589 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc018070 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3298607589 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7fd7b4024060 con 0x5562fb479330 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7ce1b3740 10 client.4770 fetch_fsmap finished waiting for FSMap version 9 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.853+0000 7fd7ce1b3740 10 client.4770 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.854+0000 7fd7ce1b3740 20 client.4770 populate_metadata read hostname 'smithi094' 2024-03-06T07:18:40.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.854+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb4d2510 0x5562fb4d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.854+0000 7fd7ce1b3740 4 client.4770 mds_command: new command op to 34136 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.854+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 --> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562fb3eabf0 con 0x5562fb4d2510 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.854+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb4d2510 0x5562fb4d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.854+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb4d2510 0x5562fb4d4900 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c44253792be665c7 server_cookie=5ab9e0edd7d63275 in_seq=0 out_seq=0 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.854+0000 7fd7a3fff640 10 client.4770 ms_handle_connect on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3298607589 <== mds.0 v2:172.21.15.145:6835/2223795314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5562fb3eabf0 con 0x5562fb4d2510 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7a3fff640 10 client.4770 handle_command_reply: tid=0 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 client.4770 shutdown 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 msgr2=0x5562fb49bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb49bcf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400cfd0 tx=0x7fd7b4058880 comp rx=0 tx=0).stop 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 shutdown_connections 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb4d2510 0x5562fb4d4900 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb49d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb49bcf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3298607589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb4953b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 >> 172.21.15.94:0/3298607589 conn(0x5562fb4774b0 msgr2=0x5562fb4778e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7a3fff640 0 client.4770 ms_handle_reset on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:40.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 shutdown_connections 2024-03-06T07:18:40.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3298607589 wait complete. 2024-03-06T07:18:40.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.855+0000 7fd7ce1b3740 20 client.4770 trim_cache size 0 max 16384 2024-03-06T07:18:40.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.871+0000 7fd7ce1b3740 1 Processor -- start 2024-03-06T07:18:40.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.871+0000 7fd7ce1b3740 1 -- start start 2024-03-06T07:18:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb499940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb479330 0x5562fb4c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb49c960 0x5562fb4c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c41f8990 con 0x5562fb49c960 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c41f7b60 con 0x5562fb478f50 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c4147670 con 0x5562fb479330 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb49c960 0x5562fb4c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb49c960 0x5562fb4c2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37534/0 (socket says 172.21.15.94:37534) 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/1128016299 learned_addr learned my addr 172.21.15.94:0/1128016299 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9289640 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb499940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb479330 0x5562fb4c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb479330 msgr2=0x5562fb4c2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb479330 0x5562fb4c2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 msgr2=0x5562fb499940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb499940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.872+0000 7fd7c9a8a640 1 -- 172.21.15.94:0/1128016299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562fb1bc6b0 con 0x5562fb49c960 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb49c960 0x5562fb4c2f70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc014000 tx=0x7fd7bc00c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67fe84714d83180 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ae7fc640 1 -- 172.21.15.94:0/1128016299 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc007130 con 0x5562fb49c960 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ae7fc640 1 -- 172.21.15.94:0/1128016299 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc023020 con 0x5562fb49c960 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ae7fc640 1 -- 172.21.15.94:0/1128016299 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc01c420 con 0x5562fb49c960 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/1128016299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb49c960 msgr2=0x5562fb4c2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb49c960 0x5562fb4c2f70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc014000 tx=0x7fd7bc00c000 comp rx=0 tx=0).stop 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/1128016299 shutdown_connections 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb49c960 0x5562fb4c2f70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb479330 0x5562fb4c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/1128016299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb499940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/1128016299 >> 172.21.15.94:0/1128016299 conn(0x5562fb4774b0 msgr2=0x5562fb4b7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/1128016299 shutdown_connections 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.873+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/1128016299 wait complete. 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.874+0000 7fd7ce1b3740 1 -- start start 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.874+0000 7fd7ce1b3740 1 Processor -- start 2024-03-06T07:18:40.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb472910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb5257c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb522360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562fb49a610 con 0x5562fb479330 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c41f7560 con 0x5562fb478f50 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562fb4c8f70 con 0x5562fb49c960 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 10 client.0 _ll_register_callbacks cb 0x7fd7b40583c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb5257c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb5257c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37540/0 (socket says 172.21.15.94:37540) 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3250242281 learned_addr learned my addr 172.21.15.94:0/3250242281 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c9a8a640 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb522360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb472910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 msgr2=0x5562fb522360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb522360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 msgr2=0x5562fb472910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb472910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 -- 172.21.15.94:0/3250242281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562fb4c1a20 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7c8a88640 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb5257c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4058810 tx=0x7fd7b4088450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897b39e04c3eb350 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3250242281 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4074070 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3250242281 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd7b4057cd0 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7bc023020 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3250242281 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b4070040 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 10 client.4776 fetch_fsmap learned FSMap version 9 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.875+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x5562fb477180 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3250242281 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc023020 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3250242281 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 9) v1 ==== 3322+0+0 (secure 0 0 0) 0x7fd7b4059b00 con 0x5562fb479330 2024-03-06T07:18:40.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 10 client.4776 fetch_fsmap finished waiting for FSMap version 9 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 10 client.4776 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 20 client.4776 populate_metadata read hostname 'smithi094' 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb5286f0 0x5562fb528ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 4 client.4776 mds_command: new command op to 34136 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 --> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5562fb3e4f40 con 0x5562fb5286f0 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb5286f0 0x5562fb528ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7a3fff640 10 client.4776 ms_handle_connect on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7c9289640 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb5286f0 0x5562fb528ad0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=649b6a721c66ca65 server_cookie=9ace8462ed7357f4 in_seq=0 out_seq=0 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7a3fff640 1 -- 172.21.15.94:0/3250242281 <== mds.0 v2:172.21.15.145:6835/2223795314 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5562fb3e4f40 con 0x5562fb5286f0 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7a3fff640 10 client.4776 handle_command_reply: tid=0 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 1 client.4776 shutdown 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.876+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 msgr2=0x5562fb5257c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb5257c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4058810 tx=0x7fd7b4088450 comp rx=0 tx=0).stop 2024-03-06T07:18:40.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 shutdown_connections 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:6835/2223795314,v1:172.21.15.145:6837/2223795314] conn(0x5562fb5286f0 0x5562fb528ad0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562fb49c960 0x5562fb522360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562fb479330 0x5562fb5257c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7a3fff640 0 client.4776 ms_handle_reset on v2:172.21.15.145:6835/2223795314 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 --2- 172.21.15.94:0/3250242281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562fb478f50 0x5562fb472910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 >> 172.21.15.94:0/3250242281 conn(0x5562fb4774b0 msgr2=0x5562fb49b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 shutdown_connections 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 1 -- 172.21.15.94:0/3250242281 wait complete. 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7ce1b3740 20 client.4776 trim_cache size 0 max 16384 2024-03-06T07:18:40.885 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 34136, 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 31.351832879, 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 9, 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 24, 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 24, 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 37.799622575000001 2024-03-06T07:18:40.886 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:18:40.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd7a806b500 msgr2=0x7fd7a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd7a806b500 0x7fd7a806d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0035000 tx=0x7fd7c0030000 comp rx=0 tx=0).stop 2024-03-06T07:18:40.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 msgr2=0x7fd7c4148880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:40.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c4148880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400e820 tx=0x7fd7b400ece0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 shutdown_connections 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd7a806b500 0x7fd7a806d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7c414f660 0x7fd7c41eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7c40c4bb0 0x7fd7c4148880 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 --2- 172.21.15.94:0/564071076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7c40c30b0 0x7fd7c40ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 >> 172.21.15.94:0/564071076 conn(0x7fd7c40c7de0 msgr2=0x7fd7c4073e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 shutdown_connections 2024-03-06T07:18:40.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:40.877+0000 7fd7cb4ed640 1 -- 172.21.15.94:0/564071076 wait complete. 2024-03-06T07:18:40.890 INFO:teuthology.run_tasks:Running task kclient... 2024-03-06T07:18:40.904 INFO:tasks.kclient:Mounting kernel clients... 2024-03-06T07:18:40.904 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-06T07:18:40.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:40.904 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-03-06T07:18:40.919 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-06T07:18:40.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:40.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-06T07:18:40.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:40.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2024-03-06T07:18:41.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.082 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-03-06T07:18:41.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link delete ceph-brx 2024-03-06T07:18:41.166 INFO:teuthology.orchestra.run.smithi145.stderr:Cannot find device "ceph-brx" 2024-03-06T07:18:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:18:41.167 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-06T07:18:41.168 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:18:41.168 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:18:41.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:18:41.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.268+0000 7f6a08e39640 1 Processor -- start 2024-03-06T07:18:41.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.268+0000 7f6a08e39640 1 -- start start 2024-03-06T07:18:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a08e39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6a04153990 0x7f6a04156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a08e39640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a04070aa0 con 0x7f6a040acc00 2024-03-06T07:18:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a08e39640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a04070d80 con 0x7f6a041506b0 2024-03-06T07:18:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a08e39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a04156450 con 0x7f6a04153990 2024-03-06T07:18:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a02d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6a040acc00 0x7f6a040c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47300/0 (socket says 172.21.15.94:47300) 2024-03-06T07:18:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a02d9d640 1 -- 172.21.15.94:0/2317276400 learned_addr learned my addr 172.21.15.94:0/2317276400 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f69fbfff640 1 --2- 172.21.15.94:0/2317276400 >> v2:172.21.15.145:3300/0 conn(0x7f6a04153990 0x7f6a04156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a01d9b640 1 -- 172.21.15.94:0/2317276400 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895828585 0 0) 0x7f6a04070aa0 con 0x7f6a040acc00 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a01d9b640 1 -- 172.21.15.94:0/2317276400 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f0003520 con 0x7f6a040acc00 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a01d9b640 1 -- 172.21.15.94:0/2317276400 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696007944 0 0) 0x7f6a04070d80 con 0x7f6a041506b0 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f6a01d9b640 1 -- 172.21.15.94:0/2317276400 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a04070aa0 con 0x7f6a041506b0 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f69fbfff640 1 -- 172.21.15.94:0/2317276400 >> v1:172.21.15.145:6789/0 conn(0x7f6a041506b0 legacy=0x7f6a040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f69fbfff640 1 -- 172.21.15.94:0/2317276400 >> v1:172.21.15.94:6789/0 conn(0x7f6a040acc00 legacy=0x7f6a040c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:41.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.269+0000 7f69fbfff640 1 -- 172.21.15.94:0/2317276400 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a041566a0 con 0x7f6a04153990 2024-03-06T07:18:41.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.270+0000 7f69fbfff640 1 --2- 172.21.15.94:0/2317276400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a04153990 0x7f6a04156ea0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0097a0 tx=0x7f69ec02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfd2ec3161d86cc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:41.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.270+0000 7f6a01d9b640 1 -- 172.21.15.94:0/2317276400 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec034070 con 0x7f6a04153990 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.270+0000 7f6a01d9b640 1 -- 172.21.15.94:0/2317276400 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec030610 con 0x7f6a04153990 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.270+0000 7f6a01d9b640 1 -- 172.21.15.94:0/2317276400 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0309e0 con 0x7f6a04153990 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.271+0000 7f6a08e39640 1 -- 172.21.15.94:0/2317276400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a04153990 msgr2=0x7f6a04156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.271+0000 7f6a08e39640 1 --2- 172.21.15.94:0/2317276400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a04153990 0x7f6a04156ea0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0097a0 tx=0x7f69ec02b960 comp rx=0 tx=0).stop 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.271+0000 7f6a08e39640 1 -- 172.21.15.94:0/2317276400 shutdown_connections 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.271+0000 7f6a08e39640 1 --2- 172.21.15.94:0/2317276400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a04153990 0x7f6a04156ea0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.271+0000 7f6a08e39640 1 -- 172.21.15.94:0/2317276400 >> 172.21.15.94:0/2317276400 conn(0x7f6a0406b1a0 msgr2=0x7f6a0406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.271+0000 7f6a08e39640 1 -- 172.21.15.94:0/2317276400 shutdown_connections 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.271+0000 7f6a08e39640 1 -- 172.21.15.94:0/2317276400 wait complete. 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 Processor -- start 2024-03-06T07:18:41.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 -- start start 2024-03-06T07:18:41.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a040acc00 0x7f6a041491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:41.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a041506b0 0x7f6a041ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:41.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a04153990 0x7f6a04142f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:41.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040ad000 con 0x7f6a041506b0 2024-03-06T07:18:41.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04156a10 con 0x7f6a040acc00 2024-03-06T07:18:41.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.272+0000 7f6a08e39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040ad180 con 0x7f6a04153990 2024-03-06T07:18:41.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a041506b0 0x7f6a041ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a041506b0 0x7f6a041ebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37544/0 (socket says 172.21.15.94:37544) 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 -- 172.21.15.94:0/1176197066 learned_addr learned my addr 172.21.15.94:0/1176197066 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f6a02d9d640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a04153990 0x7f6a04142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f6a0259c640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a040acc00 0x7f6a041491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 -- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a04153990 msgr2=0x7f6a04142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a04153990 0x7f6a04142f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 -- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a040acc00 msgr2=0x7f6a041491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a040acc00 0x7f6a041491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 -- 172.21.15.94:0/1176197066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a04143e10 con 0x7f6a041506b0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69fbfff640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a041506b0 0x7f6a041ebd70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0098c0 tx=0x7f69ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69faffd640 1 -- 172.21.15.94:0/1176197066 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec034040 con 0x7f6a041506b0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69faffd640 1 -- 172.21.15.94:0/1176197066 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec03aaa0 con 0x7f6a041506b0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a040c7b00 con 0x7f6a041506b0 2024-03-06T07:18:41.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f69faffd640 1 -- 172.21.15.94:0/1176197066 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec006e10 con 0x7f6a041506b0 2024-03-06T07:18:41.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.273+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a04149db0 con 0x7f6a041506b0 2024-03-06T07:18:41.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.274+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a04075300 con 0x7f6a041506b0 2024-03-06T07:18:41.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.275+0000 7f69faffd640 1 -- 172.21.15.94:0/1176197066 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69ec04b060 con 0x7f6a041506b0 2024-03-06T07:18:41.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.275+0000 7f69faffd640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f69e406b4d0 0x7f69e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:41.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.275+0000 7f69faffd640 1 -- 172.21.15.94:0/1176197066 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f69ec0ac0a0 con 0x7f6a041506b0 2024-03-06T07:18:41.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.275+0000 7f6a0259c640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f69e406b4d0 0x7f69e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:41.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.275+0000 7f6a0259c640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f69e406b4d0 0x7f69e406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f69f4007fd0 tx=0x7f69f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:41.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.279+0000 7f69faffd640 1 -- 172.21.15.94:0/1176197066 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a04075300 con 0x7f6a041506b0 2024-03-06T07:18:41.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.486+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6a0404f330 con 0x7f6a041506b0 2024-03-06T07:18:41.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.486+0000 7f69faffd640 1 -- 172.21.15.94:0/1176197066 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f69ec03f070 con 0x7f6a041506b0 2024-03-06T07:18:41.489 INFO:teuthology.orchestra.run.smithi094.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-06T07:18:41.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f69e406b4d0 msgr2=0x7f69e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:41.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f69e406b4d0 0x7f69e406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f69f4007fd0 tx=0x7f69f4009040 comp rx=0 tx=0).stop 2024-03-06T07:18:41.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a041506b0 msgr2=0x7f6a041ebd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:41.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a041506b0 0x7f6a041ebd70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0098c0 tx=0x7f69ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 shutdown_connections 2024-03-06T07:18:41.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f69e406b4d0 0x7f69e406d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a04153990 0x7f6a04142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a041506b0 0x7f6a041ebd70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 --2- 172.21.15.94:0/1176197066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a040acc00 0x7f6a041491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:41.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 >> 172.21.15.94:0/1176197066 conn(0x7f6a0406b1a0 msgr2=0x7f6a040c31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:41.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 shutdown_connections 2024-03-06T07:18:41.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:41.488+0000 7f6a08e39640 1 -- 172.21.15.94:0/1176197066 wait complete. 2024-03-06T07:18:41.501 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:18:41.501 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:18:41.502 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2024-03-06T07:18:41.502 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:18:41.502 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:18:41.502 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:18:41.502 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:18:41.502 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:18:41.502 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:18:41.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.503 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2024-03-06T07:18:41.518 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:18:41.518 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:18:41.518 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:18:41.518 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:41.518 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f0 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: altname enp7s0f1 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:41.519 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:18:41.520 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42704sec preferred_lft 42704sec 2024-03-06T07:18:41.520 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::ec4:7aff:fe8f:d763/64 scope link 2024-03-06T07:18:41.520 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:41.520 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:18:41.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.520 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:41.520 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:41.520 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:18:41.520 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr flush dev ceph-brx 2024-03-06T07:18:41.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set ceph-brx up 2024-03-06T07:18:41.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:18:41.521 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:41.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.627 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:18:41.653 INFO:teuthology.orchestra.run.smithi094.stdout:1 2024-03-06T07:18:41.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.656 DEBUG:teuthology.orchestra.run.smithi094:> route 2024-03-06T07:18:41.711 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2024-03-06T07:18:41.712 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:18:41.712 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:18:41.712 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:18:41.712 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:18:41.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.712 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:41.712 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:41.712 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:18:41.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:18:41.713 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:18:41.713 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:41.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.810 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-03-06T07:18:41.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.825 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2024-03-06T07:18:41.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.880 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:41.880 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:41.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:18:41.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:18:41.880 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:41.962 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:18:41.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:41.962 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:41.962 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:41.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:18:41.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:18:41.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:18:41.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:18:41.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:18:41.963 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:42.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:42.088 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2024-03-06T07:18:42.088 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2024-03-06T07:18:42.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2024-03-06T07:18:42.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:18:42.088 DEBUG:teuthology.orchestra.run.smithi094:> ') 2024-03-06T07:18:42.130 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:18:42.130 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:18:42.185 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:18:42.185 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:18:42.185 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:18:42.240 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:18:42.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:42.240 DEBUG:teuthology.orchestra.run.smithi094:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:18:42.325 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:18:42.325 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:18:42.423 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:18:42.423 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:18:42.423 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.94:3300,172.21.15.145:3300,172.21.15.145:3301" 2024-03-06T07:18:42.423 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 0@00dede11-ad16-411b-be77-3a57b1483d8f.cephfs=/ 2024-03-06T07:18:42.423 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.94:3300/172.21.15.145:3300/172.21.15.145:3301" will pass to kernel 2024-03-06T07:18:42.425 INFO:teuthology.orchestra.run.smithi094.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:18:42.425 INFO:teuthology.orchestra.run.smithi094.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:18:42.425 INFO:teuthology.orchestra.run.smithi094.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:18:42.425 INFO:teuthology.orchestra.run.smithi094.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:18:42.425 INFO:teuthology.orchestra.run.smithi094.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:18:42.426 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:18:42.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:42.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:18:42.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:42.462 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:18:42.518 INFO:teuthology.orchestra.run.smithi094.stdout:client4813 2024-03-06T07:18:42.518 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:18:42.519 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:42.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:42.574 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:18:42.631 INFO:teuthology.orchestra.run.smithi094.stdout:client4813 2024-03-06T07:18:42.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:42.632 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:18:42.687 INFO:teuthology.orchestra.run.smithi094.stdout:client4813 2024-03-06T07:18:42.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:42.687 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/00dede11-ad16-411b-be77-3a57b1483d8f.client4813/status') 2024-03-06T07:18:42.754 INFO:teuthology.orchestra.run.smithi094.stdout:instance: client.4813 (3)172.21.15.94:0/3510858175 2024-03-06T07:18:42.754 INFO:teuthology.orchestra.run.smithi094.stdout:blocklisted: false 2024-03-06T07:18:42.754 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records in 2024-03-06T07:18:42.754 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records out 2024-03-06T07:18:42.755 INFO:teuthology.orchestra.run.smithi094.stderr:70 bytes copied, 2.1772e-05 s, 3.2 MB/s 2024-03-06T07:18:42.756 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-06T07:18:42.756 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:18:42.756 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:18:42.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:18:42.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.894+0000 7f27c2039640 1 Processor -- start 2024-03-06T07:18:42.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.894+0000 7f27c2039640 1 -- start start 2024-03-06T07:18:42.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27c2039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:42.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27c2039640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc155bb0 con 0x7f27bc1550c0 2024-03-06T07:18:42.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27c2039640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0aa660 con 0x7f27bc0aa180 2024-03-06T07:18:42.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27c2039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0c4610 con 0x7f27bc14f310 2024-03-06T07:18:42.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27bb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f27bc1550c0 0x7f27bc1554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47304/0 (socket says 172.21.15.94:47304) 2024-03-06T07:18:42.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27bb7fe640 1 -- 172.21.15.94:0/2473427864 learned_addr learned my addr 172.21.15.94:0/2473427864 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:42.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27bbfff640 1 --2- 172.21.15.94:0/2473427864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27ba7fc640 1 -- 172.21.15.94:0/2473427864 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881693993 0 0) 0x7f27bc155bb0 con 0x7f27bc1550c0 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27ba7fc640 1 -- 172.21.15.94:0/2473427864 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a4003520 con 0x7f27bc1550c0 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27ba7fc640 1 -- 172.21.15.94:0/2473427864 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111038601 0 0) 0x7f27bc0aa660 con 0x7f27bc0aa180 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27ba7fc640 1 -- 172.21.15.94:0/2473427864 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27bc155bb0 con 0x7f27bc0aa180 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27bbfff640 1 -- 172.21.15.94:0/2473427864 >> v1:172.21.15.145:6789/0 conn(0x7f27bc0aa180 legacy=0x7f27bc14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27bbfff640 1 -- 172.21.15.94:0/2473427864 >> v1:172.21.15.94:6789/0 conn(0x7f27bc1550c0 legacy=0x7f27bc1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.895+0000 7f27bbfff640 1 -- 172.21.15.94:0/2473427864 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0aa940 con 0x7f27bc14f310 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.896+0000 7f27bbfff640 1 --2- 172.21.15.94:0/2473427864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc0c5060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009dc0 tx=0x7f27b402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ea0a5ff4f627de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.896+0000 7f27ba7fc640 1 -- 172.21.15.94:0/2473427864 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4039070 con 0x7f27bc14f310 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.896+0000 7f27ba7fc640 1 -- 172.21.15.94:0/2473427864 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b402cbb0 con 0x7f27bc14f310 2024-03-06T07:18:42.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.896+0000 7f27ba7fc640 1 -- 172.21.15.94:0/2473427864 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b4034440 con 0x7f27bc14f310 2024-03-06T07:18:42.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.897+0000 7f27c2039640 1 -- 172.21.15.94:0/2473427864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 msgr2=0x7f27bc0c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:42.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.897+0000 7f27c2039640 1 --2- 172.21.15.94:0/2473427864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc0c5060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009dc0 tx=0x7f27b402be00 comp rx=0 tx=0).stop 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.897+0000 7f27c2039640 1 -- 172.21.15.94:0/2473427864 shutdown_connections 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.897+0000 7f27c2039640 1 --2- 172.21.15.94:0/2473427864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc0c5060 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.897+0000 7f27c2039640 1 -- 172.21.15.94:0/2473427864 >> 172.21.15.94:0/2473427864 conn(0x7f27bc0c6710 msgr2=0x7f27bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.897+0000 7f27c2039640 1 -- 172.21.15.94:0/2473427864 shutdown_connections 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.897+0000 7f27c2039640 1 -- 172.21.15.94:0/2473427864 wait complete. 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.898+0000 7f27c2039640 1 Processor -- start 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.898+0000 7f27c2039640 1 -- start start 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27c2039640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27bc0aa180 0x7f27bc153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27c2039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27c2039640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27bc1550c0 0x7f27bc072a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27c2039640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0c1490 con 0x7f27bc0aa180 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27c2039640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0c1310 con 0x7f27bc1550c0 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27c2039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0c55a0 con 0x7f27bc14f310 2024-03-06T07:18:42.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27bc0aa180 0x7f27bc153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27bc0aa180 0x7f27bc153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37560/0 (socket says 172.21.15.94:37560) 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 -- 172.21.15.94:0/1272775300 learned_addr learned my addr 172.21.15.94:0/1272775300 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27baffd640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bbfff640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27bc1550c0 0x7f27bc072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 -- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 msgr2=0x7f27bc072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 -- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27bc1550c0 msgr2=0x7f27bc072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27bc1550c0 0x7f27bc072a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 -- 172.21.15.94:0/1272775300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc1e8950 con 0x7f27bc0aa180 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.899+0000 7f27bb7fe640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27bc0aa180 0x7f27bc153f60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00c410 tx=0x7f27ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.900+0000 7f27a3fff640 1 -- 172.21.15.94:0/1272775300 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac017070 con 0x7f27bc0aa180 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.900+0000 7f27a3fff640 1 -- 172.21.15.94:0/1272775300 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac005ce0 con 0x7f27bc0aa180 2024-03-06T07:18:42.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.900+0000 7f27a3fff640 1 -- 172.21.15.94:0/1272775300 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0072d0 con 0x7f27bc0aa180 2024-03-06T07:18:42.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.900+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc1e8c30 con 0x7f27bc0aa180 2024-03-06T07:18:42.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.900+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc1e91f0 con 0x7f27bc0aa180 2024-03-06T07:18:42.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.901+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc04f330 con 0x7f27bc0aa180 2024-03-06T07:18:42.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.901+0000 7f27a3fff640 1 -- 172.21.15.94:0/1272775300 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27ac01a070 con 0x7f27bc0aa180 2024-03-06T07:18:42.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.901+0000 7f27a3fff640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27a806b4d0 0x7f27a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:42.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.902+0000 7f27a3fff640 1 -- 172.21.15.94:0/1272775300 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f27ac089790 con 0x7f27bc0aa180 2024-03-06T07:18:42.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.902+0000 7f27baffd640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27a806b4d0 0x7f27a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:42.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.902+0000 7f27baffd640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27a806b4d0 0x7f27a806d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f27b0007fd0 tx=0x7f27b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:42.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:42.906+0000 7f27a3fff640 1 -- 172.21.15.94:0/1272775300 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27ac057dd0 con 0x7f27bc0aa180 2024-03-06T07:18:43.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.115+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f27bc0a5ef0 con 0x7f27bc0aa180 2024-03-06T07:18:43.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.115+0000 7f27a3fff640 1 -- 172.21.15.94:0/1272775300 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f27ac05be70 con 0x7f27bc0aa180 2024-03-06T07:18:43.117 INFO:teuthology.orchestra.run.smithi094.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-06T07:18:43.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27a806b4d0 msgr2=0x7f27a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:43.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27a806b4d0 0x7f27a806d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f27b0007fd0 tx=0x7f27b0009040 comp rx=0 tx=0).stop 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27bc0aa180 msgr2=0x7f27bc153f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27bc0aa180 0x7f27bc153f60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00c410 tx=0x7f27ac00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 shutdown_connections 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27a806b4d0 0x7f27a806d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27bc1550c0 0x7f27bc072a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27bc14f310 0x7f27bc072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 --2- 172.21.15.94:0/1272775300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27bc0aa180 0x7f27bc153f60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 >> 172.21.15.94:0/1272775300 conn(0x7f27bc0c6710 msgr2=0x7f27bc14b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 shutdown_connections 2024-03-06T07:18:43.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:43.117+0000 7f27c2039640 1 -- 172.21.15.94:0/1272775300 wait complete. 2024-03-06T07:18:43.130 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:18:43.130 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:18:43.130 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi145.front.sepia.ceph.com 2024-03-06T07:18:43.130 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:18:43.130 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:18:43.130 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:18:43.130 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:18:43.131 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:18:43.131 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:18:43.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.131 DEBUG:teuthology.orchestra.run.smithi145:> ip addr 2024-03-06T07:18:43.146 INFO:teuthology.orchestra.run.smithi145.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:18:43.146 INFO:teuthology.orchestra.run.smithi145.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:18:43.146 INFO:teuthology.orchestra.run.smithi145.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:18:43.146 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:43.146 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 ::1/128 scope host 2024-03-06T07:18:43.146 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:43.146 INFO:teuthology.orchestra.run.smithi145.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:66 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: altname enp7s0f0 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:67 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: altname enp7s0f1 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:90 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:91 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: inet 172.21.15.145/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft 42710sec preferred_lft 42710sec 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::ec4:7aff:fe88:7791/64 scope link 2024-03-06T07:18:43.147 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:18:43.148 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:18:43.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.148 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-06T07:18:43.148 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-06T07:18:43.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:18:43.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr flush dev ceph-brx 2024-03-06T07:18:43.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set ceph-brx up 2024-03-06T07:18:43.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:18:43.148 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-06T07:18:43.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.394 DEBUG:teuthology.orchestra.run.smithi145:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:18:43.423 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-03-06T07:18:43.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.424 DEBUG:teuthology.orchestra.run.smithi145:> route 2024-03-06T07:18:43.488 INFO:teuthology.orchestra.run.smithi145.stdout:Kernel IP routing table 2024-03-06T07:18:43.488 INFO:teuthology.orchestra.run.smithi145.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:18:43.488 INFO:teuthology.orchestra.run.smithi145.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:18:43.488 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:18:43.488 INFO:teuthology.orchestra.run.smithi145.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:18:43.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.489 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-06T07:18:43.489 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-06T07:18:43.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:18:43.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:18:43.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:18:43.489 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-06T07:18:43.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:18:43.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:18:43.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.739 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-03-06T07:18:43.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.821 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list-id 2024-03-06T07:18:43.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.875 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-06T07:18:43.875 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-06T07:18:43.875 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:18:43.875 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:18:43.876 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-06T07:18:43.959 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:18:43.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:43.959 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-06T07:18:43.959 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-06T07:18:43.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:18:43.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:18:43.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:18:43.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:18:43.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:18:43.960 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-06T07:18:44.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:44.103 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-03-06T07:18:44.103 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-03-06T07:18:44.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set brx.0 up 2024-03-06T07:18:44.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:18:44.103 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-03-06T07:18:44.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:18:44.146 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:18:44.202 INFO:teuthology.orchestra.run.smithi145.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:18:44.202 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:18:44.202 DEBUG:teuthology.orchestra.run.smithi145:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:18:44.256 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-06T07:18:44.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:44.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:18:44.340 INFO:teuthology.orchestra.run.smithi145.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:18:44.340 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:18:44.438 INFO:teuthology.orchestra.run.smithi145.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:18:44.438 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:18:44.438 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: resolved to: "172.21.15.94:3300,172.21.15.145:3300,172.21.15.145:3301" 2024-03-06T07:18:44.438 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: trying mount with new device syntax: 1@00dede11-ad16-411b-be77-3a57b1483d8f.cephfs=/ 2024-03-06T07:18:44.438 INFO:teuthology.orchestra.run.smithi145.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.94:3300/172.21.15.145:3300/172.21.15.145:3301" will pass to kernel 2024-03-06T07:18:44.439 INFO:teuthology.orchestra.run.smithi145.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-06T07:18:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:18:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:18:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:18:44.439 INFO:teuthology.orchestra.run.smithi145.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:18:44.440 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:18:44.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:44.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:18:44.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:44.507 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:18:44.563 INFO:teuthology.orchestra.run.smithi145.stdout:client34541 2024-03-06T07:18:44.563 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:18:44.563 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:44.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:44.620 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:18:44.676 INFO:teuthology.orchestra.run.smithi145.stdout:client34541 2024-03-06T07:18:44.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:44.677 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:18:44.732 INFO:teuthology.orchestra.run.smithi145.stdout:client34541 2024-03-06T07:18:44.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:44.733 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/00dede11-ad16-411b-be77-3a57b1483d8f.client34541/status') 2024-03-06T07:18:44.801 INFO:teuthology.orchestra.run.smithi145.stdout:instance: client.34541 (3)192.168.0.1:0/790387398 2024-03-06T07:18:44.801 INFO:teuthology.orchestra.run.smithi145.stdout:blocklisted: false 2024-03-06T07:18:44.801 INFO:teuthology.orchestra.run.smithi145.stderr:0+1 records in 2024-03-06T07:18:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:0+1 records out 2024-03-06T07:18:44.802 INFO:teuthology.orchestra.run.smithi145.stderr:69 bytes copied, 2.1859e-05 s, 3.2 MB/s 2024-03-06T07:18:44.803 INFO:teuthology.run_tasks:Running task exec... 2024-03-06T07:18:44.825 INFO:teuthology.task.exec:Executing custom commands... 2024-03-06T07:18:44.825 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:18:44.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/crash/warn_recent_interval 0' 2024-03-06T07:18:44.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.926+0000 7f91c7a15640 1 Processor -- start 2024-03-06T07:18:44.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.927+0000 7f91c7a15640 1 -- start start 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.927+0000 7f91c7a15640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91c00c25e0 0x7f91c00c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.927+0000 7f91c7a15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c014c2b0 con 0x7f91c00c3630 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.927+0000 7f91c7a15640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c0155460 con 0x7f91c014bed0 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.927+0000 7f91c7a15640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c0156070 con 0x7f91c00c25e0 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.928+0000 7f91c4fb0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91c00c25e0 0x7f91c00c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.928+0000 7f91c5fb2640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f91c00c3630 0x7f91c00a75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47320/0 (socket says 172.21.15.94:47320) 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.928+0000 7f91c5fb2640 1 -- 172.21.15.94:0/690595377 learned_addr learned my addr 172.21.15.94:0/690595377 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.928+0000 7f91c4fb0640 1 -- 172.21.15.94:0/690595377 >> v1:172.21.15.145:6789/0 conn(0x7f91c014bed0 legacy=0x7f91c0152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.928+0000 7f91c4fb0640 1 -- 172.21.15.94:0/690595377 >> v1:172.21.15.94:6789/0 conn(0x7f91c00c3630 legacy=0x7f91c00a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.928+0000 7f91c4fb0640 1 -- 172.21.15.94:0/690595377 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c0155740 con 0x7f91c00c25e0 2024-03-06T07:18:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.928+0000 7f91b7fff640 1 -- 172.21.15.94:0/690595377 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969460510 0 0) 0x7f91bc001110 con 0x7f91c00c3630 2024-03-06T07:18:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c4fb0640 1 --2- 172.21.15.94:0/690595377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c00c30f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f91b00097a0 tx=0x7f91b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b6cbb66515a03b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91b7fff640 1 -- 172.21.15.94:0/690595377 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0034070 con 0x7f91c00c25e0 2024-03-06T07:18:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91b7fff640 1 -- 172.21.15.94:0/690595377 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b0030610 con 0x7f91c00c25e0 2024-03-06T07:18:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91b7fff640 1 -- 172.21.15.94:0/690595377 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b00309e0 con 0x7f91c00c25e0 2024-03-06T07:18:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c7a15640 1 -- 172.21.15.94:0/690595377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 msgr2=0x7f91c00c30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c7a15640 1 --2- 172.21.15.94:0/690595377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c00c30f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f91b00097a0 tx=0x7f91b002b960 comp rx=0 tx=0).stop 2024-03-06T07:18:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c7a15640 1 -- 172.21.15.94:0/690595377 shutdown_connections 2024-03-06T07:18:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c7a15640 1 --2- 172.21.15.94:0/690595377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c00c30f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c7a15640 1 -- 172.21.15.94:0/690595377 >> 172.21.15.94:0/690595377 conn(0x7f91c00c66a0 msgr2=0x7f91c00c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c7a15640 1 -- 172.21.15.94:0/690595377 shutdown_connections 2024-03-06T07:18:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.929+0000 7f91c7a15640 1 -- 172.21.15.94:0/690595377 wait complete. 2024-03-06T07:18:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.930+0000 7f91c7a15640 1 Processor -- start 2024-03-06T07:18:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.930+0000 7f91c7a15640 1 -- start start 2024-03-06T07:18:44.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c7a15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c0070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:44.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c7a15640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91c00c3630 0x7f91c01dda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c7a15640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91c014bed0 0x7f91c014cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c7a15640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c0155d00 con 0x7f91c00c25e0 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c7a15640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00a6400 con 0x7f91c00c3630 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c7a15640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00a7cd0 con 0x7f91c014bed0 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c0070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c0070430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37574/0 (socket says 172.21.15.94:37574) 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 -- 172.21.15.94:0/1644995945 learned_addr learned my addr 172.21.15.94:0/1644995945 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c4fb0640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91c00c3630 0x7f91c01dda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c5fb2640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91c014bed0 0x7f91c014cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 -- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91c014bed0 msgr2=0x7f91c014cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91c014bed0 0x7f91c014cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 -- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91c00c3630 msgr2=0x7f91c01dda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:44.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91c00c3630 0x7f91c01dda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.931+0000 7f91c57b1640 1 -- 172.21.15.94:0/1644995945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c01dcca0 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.932+0000 7f91c57b1640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c0070430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f91b800c760 tx=0x7f91b800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.932+0000 7f91b5ffb640 1 -- 172.21.15.94:0/1644995945 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8003b70 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.932+0000 7f91b5ffb640 1 -- 172.21.15.94:0/1644995945 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b8003d30 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.932+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c0148bc0 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.932+0000 7f91b5ffb640 1 -- 172.21.15.94:0/1644995945 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b800aa90 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.932+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c01dc1c0 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.933+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c004f330 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.933+0000 7f91b5ffb640 1 -- 172.21.15.94:0/1644995945 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91b800ac30 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.933+0000 7f91b5ffb640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91a406b4d0 0x7f91a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.934+0000 7f91b5ffb640 1 -- 172.21.15.94:0/1644995945 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f91b808a2c0 con 0x7f91c00c25e0 2024-03-06T07:18:44.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.934+0000 7f91c4fb0640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91a406b4d0 0x7f91a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:44.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.934+0000 7f91c4fb0640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91a406b4d0 0x7f91a406d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f91b0036000 tx=0x7f91b003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:44.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:44.938+0000 7f91b5ffb640 1 -- 172.21.15.94:0/1644995945 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91b80587b0 con 0x7f91c00c25e0 2024-03-06T07:18:45.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.108+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mgr/crash/warn_recent_interval}] v 0) v1 -- 0x7f91c0152270 con 0x7f91c00c25e0 2024-03-06T07:18:45.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.111+0000 7f91b5ffb640 1 -- 172.21.15.94:0/1644995945 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/crash/warn_recent_interval}]=0 v10)=0 v10) v1 ==== 128+0+0 (secure 0 0 0) 0x7f91b805c850 con 0x7f91c00c25e0 2024-03-06T07:18:45.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91a406b4d0 msgr2=0x7f91a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91a406b4d0 0x7f91a406d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f91b0036000 tx=0x7f91b003b040 comp rx=0 tx=0).stop 2024-03-06T07:18:45.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 msgr2=0x7f91c0070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c0070430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f91b800c760 tx=0x7f91b800cc20 comp rx=0 tx=0).stop 2024-03-06T07:18:45.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 shutdown_connections 2024-03-06T07:18:45.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91a406b4d0 0x7f91a406d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91c014bed0 0x7f91c014cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91c00c3630 0x7f91c01dda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 --2- 172.21.15.94:0/1644995945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91c00c25e0 0x7f91c0070430 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 >> 172.21.15.94:0/1644995945 conn(0x7f91c00c66a0 msgr2=0x7f91c014a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:45.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 shutdown_connections 2024-03-06T07:18:45.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.113+0000 7f91c7a15640 1 -- 172.21.15.94:0/1644995945 wait complete. 2024-03-06T07:18:45.125 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:18:45.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:45.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.292+0000 7f592184e640 1 Processor -- start 2024-03-06T07:18:45.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.292+0000 7f592184e640 1 -- start start 2024-03-06T07:18:45.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f592184e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c159270 0x7f591c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f592184e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c0c73d0 con 0x7f591c0c88c0 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f592184e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c0be2e0 con 0x7f591c14f360 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f592184e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c159700 con 0x7f591c159270 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f5913fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f591c0c88c0 0x7f591c158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47322/0 (socket says 172.21.15.94:47322) 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f5913fff640 1 -- 172.21.15.94:0/2786738049 learned_addr learned my addr 172.21.15.94:0/2786738049 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591b7fe640 1 --2- 172.21.15.94:0/2786738049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c159270 0x7f591c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591a7fc640 1 -- 172.21.15.94:0/2786738049 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976207169 0 0) 0x7f591c0c73d0 con 0x7f591c0c88c0 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591a7fc640 1 -- 172.21.15.94:0/2786738049 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5908003520 con 0x7f591c0c88c0 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591a7fc640 1 -- 172.21.15.94:0/2786738049 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545639303 0 0) 0x7f591c0be2e0 con 0x7f591c14f360 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591a7fc640 1 -- 172.21.15.94:0/2786738049 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f591c0c73d0 con 0x7f591c14f360 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591b7fe640 1 -- 172.21.15.94:0/2786738049 >> v1:172.21.15.145:6789/0 conn(0x7f591c14f360 legacy=0x7f591c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591b7fe640 1 -- 172.21.15.94:0/2786738049 >> v1:172.21.15.94:6789/0 conn(0x7f591c0c88c0 legacy=0x7f591c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.293+0000 7f591b7fe640 1 -- 172.21.15.94:0/2786738049 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c0c7670 con 0x7f591c159270 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.294+0000 7f591b7fe640 1 --2- 172.21.15.94:0/2786738049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c159270 0x7f591c155c10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5914009dc0 tx=0x7f591402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac197f8fcba80bb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.294+0000 7f591a7fc640 1 -- 172.21.15.94:0/2786738049 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914039070 con 0x7f591c159270 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.294+0000 7f591a7fc640 1 -- 172.21.15.94:0/2786738049 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591402cc40 con 0x7f591c159270 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.294+0000 7f591a7fc640 1 -- 172.21.15.94:0/2786738049 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914034440 con 0x7f591c159270 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.295+0000 7f592184e640 1 -- 172.21.15.94:0/2786738049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c159270 msgr2=0x7f591c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.295+0000 7f592184e640 1 --2- 172.21.15.94:0/2786738049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c159270 0x7f591c155c10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5914009dc0 tx=0x7f591402be90 comp rx=0 tx=0).stop 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.295+0000 7f592184e640 1 -- 172.21.15.94:0/2786738049 shutdown_connections 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.295+0000 7f592184e640 1 --2- 172.21.15.94:0/2786738049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c159270 0x7f591c155c10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.295+0000 7f592184e640 1 -- 172.21.15.94:0/2786738049 >> 172.21.15.94:0/2786738049 conn(0x7f591c0ac570 msgr2=0x7f591c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.295+0000 7f592184e640 1 -- 172.21.15.94:0/2786738049 shutdown_connections 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.295+0000 7f592184e640 1 -- 172.21.15.94:0/2786738049 wait complete. 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 Processor -- start 2024-03-06T07:18:45.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 -- start start 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c0c88c0 0x7f591c0af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c14f360 0x7f591c14b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591c159270 0x7f591c14bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0c7c30 con 0x7f591c0c88c0 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0c5a50 con 0x7f591c159270 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.296+0000 7f592184e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0c5bd0 con 0x7f591c14f360 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c0c88c0 0x7f591c0af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c0c88c0 0x7f591c0af4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37590/0 (socket says 172.21.15.94:37590) 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 -- 172.21.15.94:0/688353179 learned_addr learned my addr 172.21.15.94:0/688353179 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f5913fff640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c14f360 0x7f591c14b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591b7fe640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591c159270 0x7f591c14bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 -- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c14f360 msgr2=0x7f591c14b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c14f360 0x7f591c14b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 -- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591c159270 msgr2=0x7f591c14bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591c159270 0x7f591c14bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 -- 172.21.15.94:0/688353179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c1eb8a0 con 0x7f591c0c88c0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f591affd640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c0c88c0 0x7f591c0af4a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f590c00c530 tx=0x7f590c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f5918ff9640 1 -- 172.21.15.94:0/688353179 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c0052d0 con 0x7f591c0c88c0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f5918ff9640 1 -- 172.21.15.94:0/688353179 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c005490 con 0x7f591c0c88c0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c0b0280 con 0x7f591c0c88c0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f5918ff9640 1 -- 172.21.15.94:0/688353179 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c012870 con 0x7f591c0c88c0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.297+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c0ad320 con 0x7f591c0c88c0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.298+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c04f330 con 0x7f591c0c88c0 2024-03-06T07:18:45.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.299+0000 7f5918ff9640 1 -- 172.21.15.94:0/688353179 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f590c012a10 con 0x7f591c0c88c0 2024-03-06T07:18:45.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.299+0000 7f5918ff9640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f58fc06b500 0x7f58fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.299+0000 7f5918ff9640 1 -- 172.21.15.94:0/688353179 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f590c08a080 con 0x7f591c0c88c0 2024-03-06T07:18:45.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.299+0000 7f5913fff640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f58fc06b500 0x7f58fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.300+0000 7f5913fff640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f58fc06b500 0x7f58fc06d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f59040011e0 tx=0x7f5904007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.303+0000 7f5918ff9640 1 -- 172.21.15.94:0/688353179 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f590c058580 con 0x7f591c0c88c0 2024-03-06T07:18:45.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.476+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f591c0c8d50 con 0x7f591c0c88c0 2024-03-06T07:18:45.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.477+0000 7f5918ff9640 1 -- 172.21.15.94:0/688353179 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f590c05c620 con 0x7f591c0c88c0 2024-03-06T07:18:45.479 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:45.480 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:45.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f58fc06b500 msgr2=0x7f58fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f58fc06b500 0x7f58fc06d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f59040011e0 tx=0x7f5904007d30 comp rx=0 tx=0).stop 2024-03-06T07:18:45.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c0c88c0 msgr2=0x7f591c0af4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c0c88c0 0x7f591c0af4a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f590c00c530 tx=0x7f590c00c9f0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 shutdown_connections 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f58fc06b500 0x7f58fc06d9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591c159270 0x7f591c14bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591c14f360 0x7f591c14b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 --2- 172.21.15.94:0/688353179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f591c0c88c0 0x7f591c0af4a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 >> 172.21.15.94:0/688353179 conn(0x7f591c0ac570 msgr2=0x7f591c154a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 shutdown_connections 2024-03-06T07:18:45.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.479+0000 7f592184e640 1 -- 172.21.15.94:0/688353179 wait complete. 2024-03-06T07:18:45.492 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:45.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:45.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.595+0000 7f4d64e81640 1 Processor -- start 2024-03-06T07:18:45.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.595+0000 7f4d64e81640 1 -- start start 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d64e81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60155c30 0x7f4d6006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d64e81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4d600c5380 0x7f4d600c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d64e81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600b0450 con 0x7f4d600b0070 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d64e81640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6006cfb0 con 0x7f4d600c5380 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d64e81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6006c020 con 0x7f4d60155c30 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d5ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4d600c5380 0x7f4d600c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d5ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4d600c5380 0x7f4d600c57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37592/0 (socket says 172.21.15.94:37592) 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2516160080 learned_addr learned my addr 172.21.15.94:0/2516160080 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.596+0000 7f4d5dd9b640 1 --2- 172.21.15.94:0/2516160080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60155c30 0x7f4d6006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.597+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2516160080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60155c30 msgr2=0x7f4d6006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.597+0000 7f4d5ed9d640 1 --2- 172.21.15.94:0/2516160080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60155c30 0x7f4d6006ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.597+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2516160080 >> v1:172.21.15.145:6789/0 conn(0x7f4d600b0070 legacy=0x7f4d600c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.597+0000 7f4d5d59a640 1 -- 172.21.15.94:0/2516160080 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053269220 0 0) 0x7f4d50001110 con 0x7f4d600b0070 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.597+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2516160080 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d601560e0 con 0x7f4d600c5380 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.597+0000 7f4d5ed9d640 1 --2- 172.21.15.94:0/2516160080 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c5380 0x7f4d600c57f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d5802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de67ddd5708dd4d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.597+0000 7f4d5d59a640 1 -- 172.21.15.94:0/2516160080 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5802ca70 con 0x7f4d600c5380 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d5d59a640 1 -- 172.21.15.94:0/2516160080 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5802cc10 con 0x7f4d600c5380 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d5d59a640 1 -- 172.21.15.94:0/2516160080 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d580343f0 con 0x7f4d600c5380 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 -- 172.21.15.94:0/2516160080 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c5380 msgr2=0x7f4d600c57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2516160080 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c5380 0x7f4d600c57f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d5802bed0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 -- 172.21.15.94:0/2516160080 shutdown_connections 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2516160080 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c5380 0x7f4d600c57f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2516160080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60155c30 0x7f4d6006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 -- 172.21.15.94:0/2516160080 >> 172.21.15.94:0/2516160080 conn(0x7f4d600ac590 msgr2=0x7f4d600ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 -- 172.21.15.94:0/2516160080 shutdown_connections 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.598+0000 7f4d64e81640 1 -- 172.21.15.94:0/2516160080 wait complete. 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.599+0000 7f4d64e81640 1 Processor -- start 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.599+0000 7f4d64e81640 1 -- start start 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d64e81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b0070 0x7f4d6014bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d64e81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c5380 0x7f4d600c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d64e81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d60155c30 0x7f4d6014b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d64e81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6006c270 con 0x7f4d60155c30 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d64e81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c7ad0 con 0x7f4d600b0070 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d64e81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6006c550 con 0x7f4d600c5380 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d60155c30 0x7f4d6014b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d60155c30 0x7f4d6014b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37606/0 (socket says 172.21.15.94:37606) 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2683584420 learned_addr learned my addr 172.21.15.94:0/2683584420 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5e59c640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b0070 0x7f4d6014bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5dd9b640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c5380 0x7f4d600c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c5380 msgr2=0x7f4d600c4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c5380 0x7f4d600c4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b0070 msgr2=0x7f4d6014bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b0070 0x7f4d6014bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.600+0000 7f4d5ed9d640 1 -- 172.21.15.94:0/2683584420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6014c140 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.601+0000 7f4d5ed9d640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d60155c30 0x7f4d6014b040 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.601+0000 7f4d4effd640 1 -- 172.21.15.94:0/2683584420 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58043020 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.601+0000 7f4d4effd640 1 -- 172.21.15.94:0/2683584420 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5803e450 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.601+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d600c29e0 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.601+0000 7f4d4effd640 1 -- 172.21.15.94:0/2683584420 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58009510 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.601+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d600c2fa0 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.602+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6004f330 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.602+0000 7f4d4effd640 1 -- 172.21.15.94:0/2683584420 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d58046020 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.603+0000 7f4d4effd640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.603+0000 7f4d4effd640 1 -- 172.21.15.94:0/2683584420 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4d580aca70 con 0x7f4d60155c30 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.603+0000 7f4d5e59c640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.603+0000 7f4d5e59c640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4d50000f60 tx=0x7f4d50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.607+0000 7f4d4effd640 1 -- 172.21.15.94:0/2683584420 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d5804d030 con 0x7f4d60155c30 2024-03-06T07:18:45.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.778+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d600c1890 con 0x7f4d60155c30 2024-03-06T07:18:45.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.778+0000 7f4d4effd640 1 -- 172.21.15.94:0/2683584420 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d5807aff0 con 0x7f4d60155c30 2024-03-06T07:18:45.781 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:45.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 msgr2=0x7f4d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4d50000f60 tx=0x7f4d50009040 comp rx=0 tx=0).stop 2024-03-06T07:18:45.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d60155c30 msgr2=0x7f4d6014b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d60155c30 0x7f4d6014b040 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009e50 tx=0x7f4d58004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 shutdown_connections 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d60155c30 0x7f4d6014b040 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c5380 0x7f4d600c4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 --2- 172.21.15.94:0/2683584420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b0070 0x7f4d6014bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 >> 172.21.15.94:0/2683584420 conn(0x7f4d600ac590 msgr2=0x7f4d600ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 shutdown_connections 2024-03-06T07:18:45.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.780+0000 7f4d64e81640 1 -- 172.21.15.94:0/2683584420 wait complete. 2024-03-06T07:18:45.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:45.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.895+0000 7fda639bf640 1 Processor -- start 2024-03-06T07:18:45.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.895+0000 7fda639bf640 1 -- start start 2024-03-06T07:18:45.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda639bf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fda5c0c1820 0x7fda5c06cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda639bf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5c0c2790 con 0x7fda5c06c1f0 2024-03-06T07:18:45.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda639bf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5c0c1540 con 0x7fda5c14ef70 2024-03-06T07:18:45.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda639bf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5c0c3090 con 0x7fda5c0c1820 2024-03-06T07:18:45.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda61f5c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fda5c0c1820 0x7fda5c06cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda61f5c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fda5c0c1820 0x7fda5c06cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37620/0 (socket says 172.21.15.94:37620) 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda60f5a640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fda5c06c1f0 0x7fda5c154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47324/0 (socket says 172.21.15.94:47324) 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda61f5c640 1 -- 172.21.15.94:0/2337753294 learned_addr learned my addr 172.21.15.94:0/2337753294 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda53fff640 1 -- 172.21.15.94:0/2337753294 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93698648 0 0) 0x7fda5c0c2790 con 0x7fda5c06c1f0 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda53fff640 1 -- 172.21.15.94:0/2337753294 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda48003520 con 0x7fda5c06c1f0 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda53fff640 1 -- 172.21.15.94:0/2337753294 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603470226 0 0) 0x7fda5c0c1540 con 0x7fda5c14ef70 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda53fff640 1 -- 172.21.15.94:0/2337753294 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda5c0c2790 con 0x7fda5c14ef70 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda61f5c640 1 -- 172.21.15.94:0/2337753294 >> v1:172.21.15.145:6789/0 conn(0x7fda5c14ef70 legacy=0x7fda5c0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda61f5c640 1 -- 172.21.15.94:0/2337753294 >> v1:172.21.15.94:6789/0 conn(0x7fda5c06c1f0 legacy=0x7fda5c154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.896+0000 7fda61f5c640 1 -- 172.21.15.94:0/2337753294 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c0c24b0 con 0x7fda5c0c1820 2024-03-06T07:18:45.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.897+0000 7fda61f5c640 1 --2- 172.21.15.94:0/2337753294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c1820 0x7fda5c06cad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fda5400d150 tx=0x7fda5402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3148506dc0103fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.897+0000 7fda53fff640 1 -- 172.21.15.94:0/2337753294 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54034070 con 0x7fda5c0c1820 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.897+0000 7fda53fff640 1 -- 172.21.15.94:0/2337753294 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda54038400 con 0x7fda5c0c1820 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.897+0000 7fda53fff640 1 -- 172.21.15.94:0/2337753294 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda540387d0 con 0x7fda5c0c1820 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.898+0000 7fda639bf640 1 -- 172.21.15.94:0/2337753294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c1820 msgr2=0x7fda5c06cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.898+0000 7fda639bf640 1 --2- 172.21.15.94:0/2337753294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c1820 0x7fda5c06cad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fda5400d150 tx=0x7fda5402e170 comp rx=0 tx=0).stop 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.898+0000 7fda639bf640 1 -- 172.21.15.94:0/2337753294 shutdown_connections 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.898+0000 7fda639bf640 1 --2- 172.21.15.94:0/2337753294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c1820 0x7fda5c06cad0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.898+0000 7fda639bf640 1 -- 172.21.15.94:0/2337753294 >> 172.21.15.94:0/2337753294 conn(0x7fda5c0c6710 msgr2=0x7fda5c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.898+0000 7fda639bf640 1 -- 172.21.15.94:0/2337753294 shutdown_connections 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.898+0000 7fda639bf640 1 -- 172.21.15.94:0/2337753294 wait complete. 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.899+0000 7fda639bf640 1 Processor -- start 2024-03-06T07:18:45.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.899+0000 7fda639bf640 1 -- start start 2024-03-06T07:18:45.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda639bf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c06c1f0 0x7fda5c154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda639bf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c0c1820 0x7fda5c1e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda639bf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c14ef70 0x7fda5c1e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda639bf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0c1c70 con 0x7fda5c14ef70 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda639bf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c14f350 con 0x7fda5c06c1f0 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda639bf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c155340 con 0x7fda5c0c1820 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c14ef70 0x7fda5c1e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c14ef70 0x7fda5c1e7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37624/0 (socket says 172.21.15.94:37624) 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 -- 172.21.15.94:0/2892436550 learned_addr learned my addr 172.21.15.94:0/2892436550 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda60f5a640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c0c1820 0x7fda5c1e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda6175b640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c06c1f0 0x7fda5c154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 -- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c0c1820 msgr2=0x7fda5c1e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c0c1820 0x7fda5c1e6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 -- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c06c1f0 msgr2=0x7fda5c154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c06c1f0 0x7fda5c154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 -- 172.21.15.94:0/2892436550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c1506e0 con 0x7fda5c14ef70 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda61f5c640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c14ef70 0x7fda5c1e7130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fda5400d270 tx=0x7fda54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda51ffb640 1 -- 172.21.15.94:0/2892436550 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54047070 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda51ffb640 1 -- 172.21.15.94:0/2892436550 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda54043850 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c14fad0 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.900+0000 7fda51ffb640 1 -- 172.21.15.94:0/2892436550 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54043c40 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.901+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c150090 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.901+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c04f330 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.902+0000 7fda51ffb640 1 -- 172.21.15.94:0/2892436550 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda5404a070 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.902+0000 7fda51ffb640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4006b4d0 0x7fda4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.902+0000 7fda51ffb640 1 -- 172.21.15.94:0/2892436550 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fda540aeeb0 con 0x7fda5c14ef70 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.902+0000 7fda6175b640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4006b4d0 0x7fda4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:45.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.903+0000 7fda6175b640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4006b4d0 0x7fda4006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda58007fd0 tx=0x7fda58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:45.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:45.906+0000 7fda51ffb640 1 -- 172.21.15.94:0/2892436550 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda5407d430 con 0x7fda5c14ef70 2024-03-06T07:18:46.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.079+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda5c14e4d0 con 0x7fda5c14ef70 2024-03-06T07:18:46.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.080+0000 7fda51ffb640 1 -- 172.21.15.94:0/2892436550 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda540814d0 con 0x7fda5c14ef70 2024-03-06T07:18:46.082 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4006b4d0 msgr2=0x7fda4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4006b4d0 0x7fda4006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda58007fd0 tx=0x7fda58009040 comp rx=0 tx=0).stop 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c14ef70 msgr2=0x7fda5c1e7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c14ef70 0x7fda5c1e7130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fda5400d270 tx=0x7fda54004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 shutdown_connections 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4006b4d0 0x7fda4006d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c14ef70 0x7fda5c1e7130 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c0c1820 0x7fda5c1e6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 --2- 172.21.15.94:0/2892436550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c06c1f0 0x7fda5c154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 >> 172.21.15.94:0/2892436550 conn(0x7fda5c0c6710 msgr2=0x7fda5c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:46.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 shutdown_connections 2024-03-06T07:18:46.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.082+0000 7fda639bf640 1 -- 172.21.15.94:0/2892436550 wait complete. 2024-03-06T07:18:46.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:46.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.195+0000 7f182ccfa640 1 Processor -- start 2024-03-06T07:18:46.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.195+0000 7f182ccfa640 1 -- start start 2024-03-06T07:18:46.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182ccfa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18280b0070 0x7f1828155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182ccfa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280c5380 0x7f18280c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182ccfa640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280b0450 con 0x7f182806ca70 2024-03-06T07:18:46.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182ccfa640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182806c020 con 0x7f18280b0070 2024-03-06T07:18:46.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182ccfa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182806cf20 con 0x7f18280c5380 2024-03-06T07:18:46.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18280b0070 0x7f1828155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18280b0070 0x7f1828155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37634/0 (socket says 172.21.15.94:37634) 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182659c640 1 -- 172.21.15.94:0/357804719 learned_addr learned my addr 172.21.15.94:0/357804719 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f1826d9d640 1 --2- 172.21.15.94:0/357804719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280c5380 0x7f18280c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182659c640 1 -- 172.21.15.94:0/357804719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280c5380 msgr2=0x7f18280c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182659c640 1 --2- 172.21.15.94:0/357804719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280c5380 0x7f18280c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182659c640 1 -- 172.21.15.94:0/357804719 >> v1:172.21.15.145:6789/0 conn(0x7f182806ca70 legacy=0x7f18280c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.196+0000 7f182659c640 1 -- 172.21.15.94:0/357804719 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280c77f0 con 0x7f18280b0070 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.197+0000 7f182659c640 1 --2- 172.21.15.94:0/357804719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18280b0070 0x7f1828155c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1820009870 tx=0x7f182002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a25d22d697e7350 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.197+0000 7f182559a640 1 -- 172.21.15.94:0/357804719 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820035020 con 0x7f18280b0070 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.197+0000 7f182559a640 1 -- 172.21.15.94:0/357804719 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1820031640 con 0x7f18280b0070 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.197+0000 7f182559a640 1 -- 172.21.15.94:0/357804719 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182002d040 con 0x7f18280b0070 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.197+0000 7f182ccfa640 1 -- 172.21.15.94:0/357804719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18280b0070 msgr2=0x7f1828155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.197+0000 7f182ccfa640 1 --2- 172.21.15.94:0/357804719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18280b0070 0x7f1828155c30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1820009870 tx=0x7f182002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.198+0000 7f182ccfa640 1 -- 172.21.15.94:0/357804719 shutdown_connections 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.198+0000 7f182ccfa640 1 --2- 172.21.15.94:0/357804719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280c5380 0x7f18280c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.198+0000 7f182ccfa640 1 --2- 172.21.15.94:0/357804719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18280b0070 0x7f1828155c30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.198+0000 7f182ccfa640 1 -- 172.21.15.94:0/357804719 >> 172.21.15.94:0/357804719 conn(0x7f18280ac590 msgr2=0x7f18280ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.198+0000 7f182ccfa640 1 -- 172.21.15.94:0/357804719 shutdown_connections 2024-03-06T07:18:46.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.198+0000 7f182ccfa640 1 -- 172.21.15.94:0/357804719 wait complete. 2024-03-06T07:18:46.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 Processor -- start 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 -- start start 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f182806ca70 0x7f18280c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b0070 0x7f18280c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280c5380 0x7f18281eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182806c550 con 0x7f182806ca70 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182806c270 con 0x7f18280c5380 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.199+0000 7f182ccfa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182806da80 con 0x7f18280b0070 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f182806ca70 0x7f18280c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f182806ca70 0x7f18280c2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37648/0 (socket says 172.21.15.94:37648) 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 -- 172.21.15.94:0/3160425234 learned_addr learned my addr 172.21.15.94:0/3160425234 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f1825d9b640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b0070 0x7f18280c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f1826d9d640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280c5380 0x7f18281eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 -- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b0070 msgr2=0x7f18280c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b0070 0x7f18280c1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 -- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280c5380 msgr2=0x7f18281eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280c5380 0x7f18281eacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 -- 172.21.15.94:0/3160425234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280c49b0 con 0x7f182806ca70 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182659c640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f182806ca70 0x7f18280c2d60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f182002b930 tx=0x7f18200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f1816ffd640 1 -- 172.21.15.94:0/3160425234 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820040070 con 0x7f182806ca70 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f1816ffd640 1 -- 172.21.15.94:0/3160425234 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182003e070 con 0x7f182806ca70 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1828158b20 con 0x7f182806ca70 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f1816ffd640 1 -- 172.21.15.94:0/3160425234 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820035040 con 0x7f182806ca70 2024-03-06T07:18:46.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.200+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18280c32a0 con 0x7f182806ca70 2024-03-06T07:18:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.201+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182804f330 con 0x7f182806ca70 2024-03-06T07:18:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.202+0000 7f1816ffd640 1 -- 172.21.15.94:0/3160425234 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1820038020 con 0x7f182806ca70 2024-03-06T07:18:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.202+0000 7f1816ffd640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f180806b500 0x7f180806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.202+0000 7f1816ffd640 1 -- 172.21.15.94:0/3160425234 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f18200abce0 con 0x7f182806ca70 2024-03-06T07:18:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.202+0000 7f1825d9b640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f180806b500 0x7f180806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.203+0000 7f1825d9b640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f180806b500 0x7f180806d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f181c0016e0 tx=0x7f181c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.206+0000 7f1816ffd640 1 -- 172.21.15.94:0/3160425234 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f182007a260 con 0x7f182806ca70 2024-03-06T07:18:46.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.374+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1828159040 con 0x7f182806ca70 2024-03-06T07:18:46.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.374+0000 7f1816ffd640 1 -- 172.21.15.94:0/3160425234 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f182007e300 con 0x7f182806ca70 2024-03-06T07:18:46.376 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f180806b500 msgr2=0x7f180806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f180806b500 0x7f180806d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f181c0016e0 tx=0x7f181c009040 comp rx=0 tx=0).stop 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f182806ca70 msgr2=0x7f18280c2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f182806ca70 0x7f18280c2d60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f182002b930 tx=0x7f18200046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 shutdown_connections 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f180806b500 0x7f180806d9c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280c5380 0x7f18281eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b0070 0x7f18280c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 --2- 172.21.15.94:0/3160425234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f182806ca70 0x7f18280c2d60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 >> 172.21.15.94:0/3160425234 conn(0x7f18280ac590 msgr2=0x7f18280ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 shutdown_connections 2024-03-06T07:18:46.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.376+0000 7f182ccfa640 1 -- 172.21.15.94:0/3160425234 wait complete. 2024-03-06T07:18:46.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:46.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.490+0000 7f0bc0eeb640 1 Processor -- start 2024-03-06T07:18:46.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.490+0000 7f0bc0eeb640 1 -- start start 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bc0eeb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0bbc14ef70 0x7f0bbc06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bc0eeb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc06cad0 0x7f0bbc154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bc0eeb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0c2790 con 0x7f0bbc0c1820 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bc0eeb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc0c3090 con 0x7f0bbc14ef70 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bc0eeb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc14f350 con 0x7f0bbc06cad0 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bba59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0bbc14ef70 0x7f0bbc06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bba59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0bbc14ef70 0x7f0bbc06c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37658/0 (socket says 172.21.15.94:37658) 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bba59c640 1 -- 172.21.15.94:0/2304791150 learned_addr learned my addr 172.21.15.94:0/2304791150 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:46.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bbad9d640 1 --2- 172.21.15.94:0/2304791150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc06cad0 0x7f0bbc154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bba59c640 1 -- 172.21.15.94:0/2304791150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc06cad0 msgr2=0x7f0bbc154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bba59c640 1 --2- 172.21.15.94:0/2304791150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc06cad0 0x7f0bbc154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bba59c640 1 -- 172.21.15.94:0/2304791150 >> v1:172.21.15.145:6789/0 conn(0x7f0bbc0c1820 legacy=0x7f0bbc0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bba59c640 1 -- 172.21.15.94:0/2304791150 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0c1540 con 0x7f0bbc14ef70 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.491+0000 7f0bb959a640 1 -- 172.21.15.94:0/2304791150 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138788457 0 0) 0x7f0bbc0c2790 con 0x7f0bbc0c1820 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.492+0000 7f0bba59c640 1 --2- 172.21.15.94:0/2304791150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc14ef70 0x7f0bbc06c1f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009870 tx=0x7f0bac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e9842dbc258498 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.492+0000 7f0bb959a640 1 -- 172.21.15.94:0/2304791150 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac035020 con 0x7f0bbc14ef70 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.492+0000 7f0bb959a640 1 -- 172.21.15.94:0/2304791150 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac031640 con 0x7f0bbc14ef70 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.492+0000 7f0bb959a640 1 -- 172.21.15.94:0/2304791150 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac02d040 con 0x7f0bbc14ef70 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/2304791150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc14ef70 msgr2=0x7f0bbc06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/2304791150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc14ef70 0x7f0bbc06c1f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009870 tx=0x7f0bac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/2304791150 shutdown_connections 2024-03-06T07:18:46.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/2304791150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc06cad0 0x7f0bbc154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/2304791150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc14ef70 0x7f0bbc06c1f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/2304791150 >> 172.21.15.94:0/2304791150 conn(0x7f0bbc0c6710 msgr2=0x7f0bbc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:46.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/2304791150 shutdown_connections 2024-03-06T07:18:46.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.493+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/2304791150 wait complete. 2024-03-06T07:18:46.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.494+0000 7f0bc0eeb640 1 Processor -- start 2024-03-06T07:18:46.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.494+0000 7f0bc0eeb640 1 -- start start 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bc0eeb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc06cad0 0x7f0bbc1430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bc0eeb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc0c1820 0x7f0bbc15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bc0eeb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bbc14ef70 0x7f0bbc0a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bc0eeb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc155750 con 0x7f0bbc06cad0 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bc0eeb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc06cf60 con 0x7f0bbc14ef70 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bc0eeb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc155170 con 0x7f0bbc0c1820 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc06cad0 0x7f0bbc1430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc06cad0 0x7f0bbc1430b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37672/0 (socket says 172.21.15.94:37672) 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 -- 172.21.15.94:0/4100698730 learned_addr learned my addr 172.21.15.94:0/4100698730 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bb9d9b640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc0c1820 0x7f0bbc15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bbad9d640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bbc14ef70 0x7f0bbc0a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 -- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc0c1820 msgr2=0x7f0bbc15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc0c1820 0x7f0bbc15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 -- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bbc14ef70 msgr2=0x7f0bbc0a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bbc14ef70 0x7f0bbc0a85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.495+0000 7f0bba59c640 1 -- 172.21.15.94:0/4100698730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc14d950 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.496+0000 7f0bba59c640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc06cad0 0x7f0bbc1430b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000c00 tx=0x7f0bac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.496+0000 7f0baaffd640 1 -- 172.21.15.94:0/4100698730 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac040070 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.496+0000 7f0baaffd640 1 -- 172.21.15.94:0/4100698730 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac0399f0 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.496+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc0a9800 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.496+0000 7f0baaffd640 1 -- 172.21.15.94:0/4100698730 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac035040 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.496+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc0a9dc0 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.497+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc04f330 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.497+0000 7f0baaffd640 1 -- 172.21.15.94:0/4100698730 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bac04b060 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.497+0000 7f0baaffd640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9c06b4d0 0x7f0b9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.498+0000 7f0baaffd640 1 -- 172.21.15.94:0/4100698730 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0bac0abba0 con 0x7f0bbc06cad0 2024-03-06T07:18:46.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.498+0000 7f0bb9d9b640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9c06b4d0 0x7f0b9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.498+0000 7f0bb9d9b640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9c06b4d0 0x7f0b9c06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc144b30 tx=0x7f0bb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.502+0000 7f0baaffd640 1 -- 172.21.15.94:0/4100698730 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bac07a120 con 0x7f0bbc06cad0 2024-03-06T07:18:46.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.670+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0bbc0735b0 con 0x7f0bbc06cad0 2024-03-06T07:18:46.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.671+0000 7f0baaffd640 1 -- 172.21.15.94:0/4100698730 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0bac0ab020 con 0x7f0bbc06cad0 2024-03-06T07:18:46.673 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9c06b4d0 msgr2=0x7f0b9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9c06b4d0 0x7f0b9c06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc144b30 tx=0x7f0bb0009040 comp rx=0 tx=0).stop 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc06cad0 msgr2=0x7f0bbc1430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc06cad0 0x7f0bbc1430b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000c00 tx=0x7f0bac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 shutdown_connections 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9c06b4d0 0x7f0b9c06d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bbc14ef70 0x7f0bbc0a85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bbc0c1820 0x7f0bbc15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 --2- 172.21.15.94:0/4100698730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bbc06cad0 0x7f0bbc1430b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 >> 172.21.15.94:0/4100698730 conn(0x7f0bbc0c6710 msgr2=0x7f0bbc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:46.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 shutdown_connections 2024-03-06T07:18:46.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.673+0000 7f0bc0eeb640 1 -- 172.21.15.94:0/4100698730 wait complete. 2024-03-06T07:18:46.686 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:46.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.784+0000 7f0ed6956640 1 Processor -- start 2024-03-06T07:18:46.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.784+0000 7f0ed6956640 1 -- start start 2024-03-06T07:18:46.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed6956640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed0072f70 0x7f0ed0154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed6956640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0ed00bbf80 0x7f0ed0154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed6956640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed00bdc30 con 0x7f0ed00c6560 2024-03-06T07:18:46.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed6956640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed00bd690 con 0x7f0ed00bbf80 2024-03-06T07:18:46.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed6956640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed00bd950 con 0x7f0ed0072f70 2024-03-06T07:18:46.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed4ef3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0ed00bbf80 0x7f0ed0154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ecffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed0072f70 0x7f0ed0154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ecf7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0ed00c6560 0x7f0ed0156670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49298/0 (socket says 172.21.15.94:49298) 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ecf7fe640 1 -- 172.21.15.94:0/602893680 learned_addr learned my addr 172.21.15.94:0/602893680 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed4ef3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0ed00bbf80 0x7f0ed0154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:35756/0 (socket says 172.21.15.94:35756) 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ecffff640 1 -- 172.21.15.94:0/602893680 >> v2:172.21.15.145:3300/0 conn(0x7f0ed00bbf80 msgr2=0x7f0ed0154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ecffff640 1 --2- 172.21.15.94:0/602893680 >> v2:172.21.15.145:3300/0 conn(0x7f0ed00bbf80 0x7f0ed0154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ecffff640 1 -- 172.21.15.94:0/602893680 >> v1:172.21.15.145:6789/0 conn(0x7f0ed00c6560 legacy=0x7f0ed0156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ecffff640 1 -- 172.21.15.94:0/602893680 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed00c5540 con 0x7f0ed0072f70 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.785+0000 7f0ed4ef3640 1 --2- 172.21.15.94:0/602893680 >> v2:172.21.15.145:3300/0 conn(0x7f0ed00bbf80 0x7f0ed0154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.786+0000 7f0ecffff640 1 --2- 172.21.15.94:0/602893680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed0072f70 0x7f0ed0154570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00097a0 tx=0x7f0ec002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2df6953e77ce0e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.786+0000 7f0eceffd640 1 -- 172.21.15.94:0/602893680 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00315a0 con 0x7f0ed0072f70 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.786+0000 7f0eceffd640 1 -- 172.21.15.94:0/602893680 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec0037070 con 0x7f0ed0072f70 2024-03-06T07:18:46.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.786+0000 7f0eceffd640 1 -- 172.21.15.94:0/602893680 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0031960 con 0x7f0ed0072f70 2024-03-06T07:18:46.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 -- 172.21.15.94:0/602893680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed0072f70 msgr2=0x7f0ed0154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 --2- 172.21.15.94:0/602893680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed0072f70 0x7f0ed0154570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00097a0 tx=0x7f0ec002b840 comp rx=0 tx=0).stop 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 -- 172.21.15.94:0/602893680 shutdown_connections 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 --2- 172.21.15.94:0/602893680 >> v2:172.21.15.145:3300/0 conn(0x7f0ed00bbf80 0x7f0ed0154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 --2- 172.21.15.94:0/602893680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed0072f70 0x7f0ed0154570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 -- 172.21.15.94:0/602893680 >> 172.21.15.94:0/602893680 conn(0x7f0ed006b1a0 msgr2=0x7f0ed006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 -- 172.21.15.94:0/602893680 shutdown_connections 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.787+0000 7f0ed6956640 1 -- 172.21.15.94:0/602893680 wait complete. 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.788+0000 7f0ed6956640 1 Processor -- start 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.788+0000 7f0ed6956640 1 -- start start 2024-03-06T07:18:46.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed6956640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed0072f70 0x7f0ed00ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed6956640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed00bbf80 0x7f0ed0156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed6956640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ed00c6560 0x7f0ed00a9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed6956640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0073350 con 0x7f0ed0072f70 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed6956640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed00bba60 con 0x7f0ed00c6560 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed6956640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed00734d0 con 0x7f0ed00bbf80 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed0072f70 0x7f0ed00ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed0072f70 0x7f0ed00ac500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37678/0 (socket says 172.21.15.94:37678) 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 -- 172.21.15.94:0/604134192 learned_addr learned my addr 172.21.15.94:0/604134192 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed4ef3640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ed00c6560 0x7f0ed00a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecf7fe640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed00bbf80 0x7f0ed0156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 -- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed00bbf80 msgr2=0x7f0ed0156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed00bbf80 0x7f0ed0156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 -- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ed00c6560 msgr2=0x7f0ed00a9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ed00c6560 0x7f0ed00a9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ecffff640 1 -- 172.21.15.94:0/604134192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed0155fa0 con 0x7f0ed0072f70 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.789+0000 7f0ed4ef3640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ed00c6560 0x7f0ed00a9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.790+0000 7f0ecffff640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed0072f70 0x7f0ed00ac500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00098c0 tx=0x7f0ec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.790+0000 7f0eccff9640 1 -- 172.21.15.94:0/604134192 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0032630 con 0x7f0ed0072f70 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.790+0000 7f0eccff9640 1 -- 172.21.15.94:0/604134192 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec00327f0 con 0x7f0ed0072f70 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.790+0000 7f0eccff9640 1 -- 172.21.15.94:0/604134192 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0032be0 con 0x7f0ed0072f70 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.790+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed006db10 con 0x7f0ed0072f70 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.790+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed006ddf0 con 0x7f0ed0072f70 2024-03-06T07:18:46.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.791+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed004f330 con 0x7f0ed0072f70 2024-03-06T07:18:46.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.791+0000 7f0eccff9640 1 -- 172.21.15.94:0/604134192 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ec004e430 con 0x7f0ed0072f70 2024-03-06T07:18:46.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.791+0000 7f0eccff9640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:46.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.791+0000 7f0eccff9640 1 -- 172.21.15.94:0/604134192 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0ec002f030 con 0x7f0ed0072f70 2024-03-06T07:18:46.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.792+0000 7f0ecf7fe640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:46.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.792+0000 7f0ecf7fe640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00c10c0 tx=0x7f0ec4007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:46.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.796+0000 7f0eccff9640 1 -- 172.21.15.94:0/604134192 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ec0040070 con 0x7f0ed0072f70 2024-03-06T07:18:46.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.968+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ed00c6260 con 0x7f0ed0072f70 2024-03-06T07:18:46.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.969+0000 7f0eccff9640 1 -- 172.21.15.94:0/604134192 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f0ec007add0 con 0x7f0ed0072f70 2024-03-06T07:18:46.971 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:46.971 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:46.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ebc06b4d0 msgr2=0x7f0ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00c10c0 tx=0x7f0ec4007e80 comp rx=0 tx=0).stop 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed0072f70 msgr2=0x7f0ed00ac500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed0072f70 0x7f0ed00ac500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00098c0 tx=0x7f0ec00046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 shutdown_connections 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ed00c6560 0x7f0ed00a9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ed00bbf80 0x7f0ed0156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 --2- 172.21.15.94:0/604134192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ed0072f70 0x7f0ed00ac500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 >> 172.21.15.94:0/604134192 conn(0x7f0ed006b1a0 msgr2=0x7f0ed00c3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 shutdown_connections 2024-03-06T07:18:46.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:46.971+0000 7f0ed6956640 1 -- 172.21.15.94:0/604134192 wait complete. 2024-03-06T07:18:46.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:46.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:47.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.086+0000 7f449fc84640 1 Processor -- start 2024-03-06T07:18:47.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.086+0000 7f449fc84640 1 -- start start 2024-03-06T07:18:47.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449fc84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44980c5370 0x7f44980c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449fc84640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4498155530 0x7f44981559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449fc84640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4498152dd0 con 0x7f44981529f0 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449fc84640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4498159710 con 0x7f4498155530 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449fc84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44980c68b0 con 0x7f44980c5370 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449e221640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4498155530 0x7f44981559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449e221640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4498155530 0x7f44981559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37688/0 (socket says 172.21.15.94:37688) 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449e221640 1 -- 172.21.15.94:0/2231770335 learned_addr learned my addr 172.21.15.94:0/2231770335 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.087+0000 7f449d21f640 1 --2- 172.21.15.94:0/2231770335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44980c5370 0x7f44980c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449e221640 1 -- 172.21.15.94:0/2231770335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44980c5370 msgr2=0x7f44980c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449e221640 1 --2- 172.21.15.94:0/2231770335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44980c5370 0x7f44980c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449e221640 1 -- 172.21.15.94:0/2231770335 >> v1:172.21.15.145:6789/0 conn(0x7f44981529f0 legacy=0x7f44980c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449e221640 1 -- 172.21.15.94:0/2231770335 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4498158b00 con 0x7f4498155530 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449e221640 1 --2- 172.21.15.94:0/2231770335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981559c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4490009870 tx=0x7f449002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49acdab2c015eff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449ca1e640 1 -- 172.21.15.94:0/2231770335 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490035020 con 0x7f4498155530 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449ca1e640 1 -- 172.21.15.94:0/2231770335 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4490031640 con 0x7f4498155530 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.088+0000 7f449ca1e640 1 -- 172.21.15.94:0/2231770335 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449002d040 con 0x7f4498155530 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 -- 172.21.15.94:0/2231770335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 msgr2=0x7f44981559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 --2- 172.21.15.94:0/2231770335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981559c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4490009870 tx=0x7f449002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 -- 172.21.15.94:0/2231770335 shutdown_connections 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 --2- 172.21.15.94:0/2231770335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981559c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 --2- 172.21.15.94:0/2231770335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44980c5370 0x7f44980c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 -- 172.21.15.94:0/2231770335 >> 172.21.15.94:0/2231770335 conn(0x7f44980ac580 msgr2=0x7f44980ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:47.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 -- 172.21.15.94:0/2231770335 shutdown_connections 2024-03-06T07:18:47.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.089+0000 7f449fc84640 1 -- 172.21.15.94:0/2231770335 wait complete. 2024-03-06T07:18:47.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.090+0000 7f449fc84640 1 Processor -- start 2024-03-06T07:18:47.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.090+0000 7f449fc84640 1 -- start start 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449fc84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44980c5370 0x7f44980bef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449fc84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44981529f0 0x7f44981548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449fc84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981edb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449fc84640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44981590c0 con 0x7f4498155530 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449fc84640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44980c9000 con 0x7f44980c5370 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449fc84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44981593a0 con 0x7f44981529f0 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981edb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981edb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37700/0 (socket says 172.21.15.94:37700) 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 -- 172.21.15.94:0/1141992291 learned_addr learned my addr 172.21.15.94:0/1141992291 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449d21f640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44981529f0 0x7f44981548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449da20640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44980c5370 0x7f44980bef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 -- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44981529f0 msgr2=0x7f44981548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44981529f0 0x7f44981548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 -- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44980c5370 msgr2=0x7f44980bef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44980c5370 0x7f44980bef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 -- 172.21.15.94:0/1141992291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44980b0680 con 0x7f4498155530 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.091+0000 7f449e221640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981edb20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4490009870 tx=0x7f44900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.092+0000 7f448e7fc640 1 -- 172.21.15.94:0/1141992291 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449003aa60 con 0x7f4498155530 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.092+0000 7f448e7fc640 1 -- 172.21.15.94:0/1141992291 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449003ac00 con 0x7f4498155530 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.092+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44980afcd0 con 0x7f4498155530 2024-03-06T07:18:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.092+0000 7f448e7fc640 1 -- 172.21.15.94:0/1141992291 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44900443f0 con 0x7f4498155530 2024-03-06T07:18:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.092+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44980c01b0 con 0x7f4498155530 2024-03-06T07:18:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.092+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449804f330 con 0x7f4498155530 2024-03-06T07:18:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.093+0000 7f448e7fc640 1 -- 172.21.15.94:0/1141992291 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4490038020 con 0x7f4498155530 2024-03-06T07:18:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.093+0000 7f448e7fc640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f447c06b500 0x7f447c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.093+0000 7f448e7fc640 1 -- 172.21.15.94:0/1141992291 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f44900ac110 con 0x7f4498155530 2024-03-06T07:18:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.093+0000 7f449da20640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f447c06b500 0x7f447c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.094+0000 7f449da20640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f447c06b500 0x7f447c06d9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4494001f10 tx=0x7f4494009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.097+0000 7f448e7fc640 1 -- 172.21.15.94:0/1141992291 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f449003f070 con 0x7f4498155530 2024-03-06T07:18:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.269+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4498154e10 con 0x7f4498155530 2024-03-06T07:18:47.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.269+0000 7f448e7fc640 1 -- 172.21.15.94:0/1141992291 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f449007a570 con 0x7f4498155530 2024-03-06T07:18:47.271 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f447c06b500 msgr2=0x7f447c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f447c06b500 0x7f447c06d9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4494001f10 tx=0x7f4494009040 comp rx=0 tx=0).stop 2024-03-06T07:18:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 msgr2=0x7f44981edb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981edb20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4490009870 tx=0x7f44900046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 shutdown_connections 2024-03-06T07:18:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f447c06b500 0x7f447c06d9c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4498155530 0x7f44981edb20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44981529f0 0x7f44981548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 --2- 172.21.15.94:0/1141992291 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44980c5370 0x7f44980bef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 >> 172.21.15.94:0/1141992291 conn(0x7f44980ac580 msgr2=0x7f44980c9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 shutdown_connections 2024-03-06T07:18:47.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.271+0000 7f449fc84640 1 -- 172.21.15.94:0/1141992291 wait complete. 2024-03-06T07:18:47.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:47.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.386+0000 7fd08d3b5640 1 Processor -- start 2024-03-06T07:18:47.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.386+0000 7fd08d3b5640 1 -- start start 2024-03-06T07:18:47.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd08d3b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd08814bcc0 0x7fd088155a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd08d3b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd0880be150 0x7fd0880c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd08d3b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd0880c5910 0x7fd088158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd08d3b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0880c7f10 con 0x7fd0880be150 2024-03-06T07:18:47.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd08d3b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0880c75e0 con 0x7fd0880c5910 2024-03-06T07:18:47.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd08d3b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880c78c0 con 0x7fd08814bcc0 2024-03-06T07:18:47.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd0880be150 0x7fd0880c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd0880be150 0x7fd0880c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37710/0 (socket says 172.21.15.94:37710) 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 -- 172.21.15.94:0/3076504986 learned_addr learned my addr 172.21.15.94:0/3076504986 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd086ffd640 1 --2- 172.21.15.94:0/3076504986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd08814bcc0 0x7fd088155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd0877fe640 1 --2- 172.21.15.94:0/3076504986 >> v2:172.21.15.145:3300/0 conn(0x7fd0880c5910 0x7fd088158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 -- 172.21.15.94:0/3076504986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd08814bcc0 msgr2=0x7fd088155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 --2- 172.21.15.94:0/3076504986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd08814bcc0 0x7fd088155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 -- 172.21.15.94:0/3076504986 >> v2:172.21.15.145:3300/0 conn(0x7fd0880c5910 msgr2=0x7fd088158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 --2- 172.21.15.94:0/3076504986 >> v2:172.21.15.145:3300/0 conn(0x7fd0880c5910 0x7fd088158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.387+0000 7fd07ffff640 1 -- 172.21.15.94:0/3076504986 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880c7ba0 con 0x7fd0880be150 2024-03-06T07:18:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd07ffff640 1 --2- 172.21.15.94:0/3076504986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0880be150 0x7fd0880c53d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0880c7390 tx=0x7fd07002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185b175ba7492f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd0867fc640 1 -- 172.21.15.94:0/3076504986 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070034070 con 0x7fd0880be150 2024-03-06T07:18:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd0867fc640 1 -- 172.21.15.94:0/3076504986 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070030860 con 0x7fd0880be150 2024-03-06T07:18:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd0867fc640 1 -- 172.21.15.94:0/3076504986 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070030c30 con 0x7fd0880be150 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 -- 172.21.15.94:0/3076504986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0880be150 msgr2=0x7fd0880c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/3076504986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0880be150 0x7fd0880c53d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd0880c7390 tx=0x7fd07002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 -- 172.21.15.94:0/3076504986 shutdown_connections 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/3076504986 >> v2:172.21.15.145:3300/0 conn(0x7fd0880c5910 0x7fd088158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/3076504986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0880be150 0x7fd0880c53d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/3076504986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd08814bcc0 0x7fd088155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 -- 172.21.15.94:0/3076504986 >> 172.21.15.94:0/3076504986 conn(0x7fd0880ac600 msgr2=0x7fd0880aca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.388+0000 7fd08d3b5640 1 -- 172.21.15.94:0/3076504986 shutdown_connections 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.389+0000 7fd08d3b5640 1 -- 172.21.15.94:0/3076504986 wait complete. 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 Processor -- start 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 -- start start 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0880be150 0x7fd0881eba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0880c5910 0x7fd0880c02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd08814bcc0 0x7fd08814b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0881590d0 con 0x7fd08814bcc0 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd088158f50 con 0x7fd0880c5910 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd08d3b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd088159250 con 0x7fd0880be150 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd0877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd08814bcc0 0x7fd08814b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd0877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd08814bcc0 0x7fd08814b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37716/0 (socket says 172.21.15.94:37716) 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd0877fe640 1 -- 172.21.15.94:0/1438697110 learned_addr learned my addr 172.21.15.94:0/1438697110 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd086ffd640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0880be150 0x7fd0881eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.390+0000 7fd07ffff640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0880c5910 0x7fd0880c02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd0877fe640 1 -- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0880be150 msgr2=0x7fd0881eba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd0877fe640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0880be150 0x7fd0881eba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd0877fe640 1 -- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0880c5910 msgr2=0x7fd0880c02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd0877fe640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0880c5910 0x7fd0880c02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd0877fe640 1 -- 172.21.15.94:0/1438697110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880c0830 con 0x7fd08814bcc0 2024-03-06T07:18:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd0877fe640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd08814bcc0 0x7fd08814b100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd08000eed0 tx=0x7fd08000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd07f7fe640 1 -- 172.21.15.94:0/1438697110 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd080018020 con 0x7fd08814bcc0 2024-03-06T07:18:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd07f7fe640 1 -- 172.21.15.94:0/1438697110 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd080014400 con 0x7fd08814bcc0 2024-03-06T07:18:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd07f7fe640 1 -- 172.21.15.94:0/1438697110 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800147f0 con 0x7fd08814bcc0 2024-03-06T07:18:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0880c31e0 con 0x7fd08814bcc0 2024-03-06T07:18:47.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.391+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0880c37a0 con 0x7fd08814bcc0 2024-03-06T07:18:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.392+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08804f330 con 0x7fd08814bcc0 2024-03-06T07:18:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.392+0000 7fd07f7fe640 1 -- 172.21.15.94:0/1438697110 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd080005d10 con 0x7fd08814bcc0 2024-03-06T07:18:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.393+0000 7fd07f7fe640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.393+0000 7fd07f7fe640 1 -- 172.21.15.94:0/1438697110 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd08008a9c0 con 0x7fd08814bcc0 2024-03-06T07:18:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.393+0000 7fd086ffd640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.393+0000 7fd086ffd640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06806b4d0 0x7fd06806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd078009aa0 tx=0x7fd078009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.397+0000 7fd07f7fe640 1 -- 172.21.15.94:0/1438697110 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd08008ce20 con 0x7fd08814bcc0 2024-03-06T07:18:47.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.566+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd08806cfb0 con 0x7fd08814bcc0 2024-03-06T07:18:47.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.567+0000 7fd07f7fe640 1 -- 172.21.15.94:0/1438697110 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd080022070 con 0x7fd08814bcc0 2024-03-06T07:18:47.569 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.568+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06806b4d0 msgr2=0x7fd06806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.568+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06806b4d0 0x7fd06806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd078009aa0 tx=0x7fd078009190 comp rx=0 tx=0).stop 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd08814bcc0 msgr2=0x7fd08814b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd08814bcc0 0x7fd08814b100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd08000eed0 tx=0x7fd08000b480 comp rx=0 tx=0).stop 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 shutdown_connections 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06806b4d0 0x7fd06806d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd08814bcc0 0x7fd08814b100 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0880c5910 0x7fd0880c02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 --2- 172.21.15.94:0/1438697110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0880be150 0x7fd0881eba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 >> 172.21.15.94:0/1438697110 conn(0x7fd0880ac600 msgr2=0x7fd0880cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:47.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 shutdown_connections 2024-03-06T07:18:47.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.569+0000 7fd08d3b5640 1 -- 172.21.15.94:0/1438697110 wait complete. 2024-03-06T07:18:47.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:47.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.683+0000 7f4ac30a4640 1 Processor -- start 2024-03-06T07:18:47.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.683+0000 7f4ac30a4640 1 -- start start 2024-03-06T07:18:47.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac30a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc0ab170 0x7f4abc0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac30a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc0c3ef0 con 0x7f4abc06c900 2024-03-06T07:18:47.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac30a4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc1556d0 con 0x7f4abc0c5990 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac30a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc1559b0 con 0x7f4abc0ab170 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4abbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4abc06c900 0x7f4abc0a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47340/0 (socket says 172.21.15.94:47340) 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4abbfff640 1 -- 172.21.15.94:0/4044884443 learned_addr learned my addr 172.21.15.94:0/4044884443 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac1641640 1 --2- 172.21.15.94:0/4044884443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc0ab170 0x7f4abc0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4abb7fe640 1 -- 172.21.15.94:0/4044884443 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126377555 0 0) 0x7f4abc0c3ef0 con 0x7f4abc06c900 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4abb7fe640 1 -- 172.21.15.94:0/4044884443 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa4003520 con 0x7f4abc06c900 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4abb7fe640 1 -- 172.21.15.94:0/4044884443 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302527556 0 0) 0x7f4abc1556d0 con 0x7f4abc0c5990 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4abb7fe640 1 -- 172.21.15.94:0/4044884443 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4abc0c3ef0 con 0x7f4abc0c5990 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac1641640 1 -- 172.21.15.94:0/4044884443 >> v1:172.21.15.145:6789/0 conn(0x7f4abc0c5990 legacy=0x7f4abc06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac1641640 1 -- 172.21.15.94:0/4044884443 >> v1:172.21.15.94:6789/0 conn(0x7f4abc06c900 legacy=0x7f4abc0a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.684+0000 7f4ac1641640 1 -- 172.21.15.94:0/4044884443 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc155c90 con 0x7f4abc0ab170 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.685+0000 7f4ac1641640 1 --2- 172.21.15.94:0/4044884443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc0ab170 0x7f4abc0c0ab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0097d0 tx=0x7f4aac02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=959e492a3f344acd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.685+0000 7f4abb7fe640 1 -- 172.21.15.94:0/4044884443 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac035020 con 0x7f4abc0ab170 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.685+0000 7f4abb7fe640 1 -- 172.21.15.94:0/4044884443 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac031890 con 0x7f4abc0ab170 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.685+0000 7f4abb7fe640 1 -- 172.21.15.94:0/4044884443 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac031c40 con 0x7f4abc0ab170 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.686+0000 7f4ac30a4640 1 -- 172.21.15.94:0/4044884443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc0ab170 msgr2=0x7f4abc0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.686+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/4044884443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc0ab170 0x7f4abc0c0ab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0097d0 tx=0x7f4aac02b930 comp rx=0 tx=0).stop 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.686+0000 7f4ac30a4640 1 -- 172.21.15.94:0/4044884443 shutdown_connections 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.686+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/4044884443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc0ab170 0x7f4abc0c0ab0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.686+0000 7f4ac30a4640 1 -- 172.21.15.94:0/4044884443 >> 172.21.15.94:0/4044884443 conn(0x7f4abc0c6710 msgr2=0x7f4abc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.686+0000 7f4ac30a4640 1 -- 172.21.15.94:0/4044884443 shutdown_connections 2024-03-06T07:18:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.686+0000 7f4ac30a4640 1 -- 172.21.15.94:0/4044884443 wait complete. 2024-03-06T07:18:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.687+0000 7f4ac30a4640 1 Processor -- start 2024-03-06T07:18:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.687+0000 7f4ac30a4640 1 -- start start 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc06c900 0x7f4abc151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4abc0ab170 0x7f4abc1499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4abc0c5990 0x7f4abc149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc156610 con 0x7f4abc0ab170 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc156490 con 0x7f4abc0c5990 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0c3af0 con 0x7f4abc06c900 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4abc0ab170 0x7f4abc1499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4abc0ab170 0x7f4abc1499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37724/0 (socket says 172.21.15.94:37724) 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 -- 172.21.15.94:0/3136641851 learned_addr learned my addr 172.21.15.94:0/3136641851 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac1641640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4abc0c5990 0x7f4abc149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac0e40640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc06c900 0x7f4abc151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 -- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc06c900 msgr2=0x7f4abc151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc06c900 0x7f4abc151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 -- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4abc0c5990 msgr2=0x7f4abc149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4abc0c5990 0x7f4abc149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 -- 172.21.15.94:0/3136641851 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc14d2e0 con 0x7f4abc0ab170 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4abbfff640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4abc0ab170 0x7f4abc1499f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000c440 tx=0x7f4ab000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ab97fa640 1 -- 172.21.15.94:0/3136641851 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0017070 con 0x7f4abc0ab170 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ab97fa640 1 -- 172.21.15.94:0/3136641851 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab0005ce0 con 0x7f4abc0ab170 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc14d780 con 0x7f4abc0ab170 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ab97fa640 1 -- 172.21.15.94:0/3136641851 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0007270 con 0x7f4abc0ab170 2024-03-06T07:18:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.688+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc14dd40 con 0x7f4abc0ab170 2024-03-06T07:18:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.689+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc04f330 con 0x7f4abc0ab170 2024-03-06T07:18:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.690+0000 7f4ab97fa640 1 -- 172.21.15.94:0/3136641851 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ab001a070 con 0x7f4abc0ab170 2024-03-06T07:18:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.690+0000 7f4ab97fa640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4aa806b500 0x7f4aa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.690+0000 7f4ab97fa640 1 -- 172.21.15.94:0/3136641851 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4ab0089710 con 0x7f4abc0ab170 2024-03-06T07:18:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.690+0000 7f4ac0e40640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4aa806b500 0x7f4aa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.691+0000 7f4ac0e40640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4aa806b500 0x7f4aa806d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4001ea0 tx=0x7f4ab4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.694+0000 7f4ab97fa640 1 -- 172.21.15.94:0/3136641851 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ab0057c10 con 0x7f4abc0ab170 2024-03-06T07:18:47.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.865+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4abc0c1740 con 0x7f4abc0ab170 2024-03-06T07:18:47.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.865+0000 7f4ab97fa640 1 -- 172.21.15.94:0/3136641851 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4ab005bcb0 con 0x7f4abc0ab170 2024-03-06T07:18:47.867 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4aa806b500 msgr2=0x7f4aa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4aa806b500 0x7f4aa806d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4001ea0 tx=0x7f4ab4009040 comp rx=0 tx=0).stop 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4abc0ab170 msgr2=0x7f4abc1499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4abc0ab170 0x7f4abc1499f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000c440 tx=0x7f4ab000c900 comp rx=0 tx=0).stop 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 shutdown_connections 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4aa806b500 0x7f4aa806d9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4abc0c5990 0x7f4abc149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4abc0ab170 0x7f4abc1499f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 --2- 172.21.15.94:0/3136641851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4abc06c900 0x7f4abc151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 >> 172.21.15.94:0/3136641851 conn(0x7f4abc0c6710 msgr2=0x7f4abc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:47.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 shutdown_connections 2024-03-06T07:18:47.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.867+0000 7f4ac30a4640 1 -- 172.21.15.94:0/3136641851 wait complete. 2024-03-06T07:18:47.879 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.980+0000 7f3efaa8f640 1 Processor -- start 2024-03-06T07:18:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.980+0000 7f3efaa8f640 1 -- start start 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.981+0000 7f3efaa8f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef406ca70 0x7f3ef414e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.981+0000 7f3efaa8f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef40b0450 con 0x7f3ef40b0070 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.981+0000 7f3efaa8f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef41560e0 con 0x7f3ef4155c30 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.981+0000 7f3efaa8f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef406c020 con 0x7f3ef406ca70 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.981+0000 7f3ef882b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3ef40b0070 0x7f3ef40c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47354/0 (socket says 172.21.15.94:47354) 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.981+0000 7f3ef882b640 1 -- 172.21.15.94:0/2442097573 learned_addr learned my addr 172.21.15.94:0/2442097573 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.981+0000 7f3ef902c640 1 --2- 172.21.15.94:0/2442097573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef406ca70 0x7f3ef414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.982+0000 7f3ef37fe640 1 -- 172.21.15.94:0/2442097573 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968141106 0 0) 0x7f3ee4001110 con 0x7f3ef40b0070 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.982+0000 7f3ef37fe640 1 -- 172.21.15.94:0/2442097573 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3edc003520 con 0x7f3ef40b0070 2024-03-06T07:18:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.982+0000 7f3ef37fe640 1 -- 172.21.15.94:0/2442097573 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102666616 0 0) 0x7f3ef41560e0 con 0x7f3ef4155c30 2024-03-06T07:18:47.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.982+0000 7f3ef902c640 1 -- 172.21.15.94:0/2442097573 >> v1:172.21.15.145:6789/0 conn(0x7f3ef4155c30 legacy=0x7f3ef40c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.982+0000 7f3ef902c640 1 -- 172.21.15.94:0/2442097573 >> v1:172.21.15.94:6789/0 conn(0x7f3ef40b0070 legacy=0x7f3ef40c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.982+0000 7f3ef902c640 1 -- 172.21.15.94:0/2442097573 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef40c77f0 con 0x7f3ef406ca70 2024-03-06T07:18:47.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.982+0000 7f3ef902c640 1 --2- 172.21.15.94:0/2442097573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef406ca70 0x7f3ef414e800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3ef41562b0 tx=0x7f3eec02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=499eff90a4be2362 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3ef37fe640 1 -- 172.21.15.94:0/2442097573 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec039070 con 0x7f3ef406ca70 2024-03-06T07:18:47.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3ef37fe640 1 -- 172.21.15.94:0/2442097573 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eec02cc40 con 0x7f3ef406ca70 2024-03-06T07:18:47.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3ef37fe640 1 -- 172.21.15.94:0/2442097573 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eec034440 con 0x7f3ef406ca70 2024-03-06T07:18:47.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3efaa8f640 1 -- 172.21.15.94:0/2442097573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef406ca70 msgr2=0x7f3ef414e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/2442097573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef406ca70 0x7f3ef414e800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3ef41562b0 tx=0x7f3eec02be90 comp rx=0 tx=0).stop 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3efaa8f640 1 -- 172.21.15.94:0/2442097573 shutdown_connections 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/2442097573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef406ca70 0x7f3ef414e800 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3efaa8f640 1 -- 172.21.15.94:0/2442097573 >> 172.21.15.94:0/2442097573 conn(0x7f3ef40ac590 msgr2=0x7f3ef40ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3efaa8f640 1 -- 172.21.15.94:0/2442097573 shutdown_connections 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.983+0000 7f3efaa8f640 1 -- 172.21.15.94:0/2442097573 wait complete. 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.984+0000 7f3efaa8f640 1 Processor -- start 2024-03-06T07:18:47.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.984+0000 7f3efaa8f640 1 -- start start 2024-03-06T07:18:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3efaa8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ef406ca70 0x7f3ef40c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3efaa8f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef40b0070 0x7f3ef41eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3efaa8f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ef4155c30 0x7f3ef41eefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3efaa8f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef406c550 con 0x7f3ef406ca70 2024-03-06T07:18:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3efaa8f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef406c270 con 0x7f3ef4155c30 2024-03-06T07:18:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3efaa8f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef406da80 con 0x7f3ef40b0070 2024-03-06T07:18:47.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ef406ca70 0x7f3ef40c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ef406ca70 0x7f3ef40c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37740/0 (socket says 172.21.15.94:37740) 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 -- 172.21.15.94:0/1932518758 learned_addr learned my addr 172.21.15.94:0/1932518758 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef3fff640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef40b0070 0x7f3ef41eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef902c640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ef4155c30 0x7f3ef41eefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 -- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef40b0070 msgr2=0x7f3ef41eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef40b0070 0x7f3ef41eeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 -- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ef4155c30 msgr2=0x7f3ef41eefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ef4155c30 0x7f3ef41eefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef882b640 1 -- 172.21.15.94:0/1932518758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef41ef520 con 0x7f3ef406ca70 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef902c640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ef4155c30 0x7f3ef41eefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.985+0000 7f3ef3fff640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef40b0070 0x7f3ef41eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.986+0000 7f3ef882b640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ef406ca70 0x7f3ef40c3e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c730 tx=0x7f3ee400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.986+0000 7f3ef17fa640 1 -- 172.21.15.94:0/1932518758 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4017070 con 0x7f3ef406ca70 2024-03-06T07:18:47.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.986+0000 7f3ef17fa640 1 -- 172.21.15.94:0/1932518758 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee4005ad0 con 0x7f3ef406ca70 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.986+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef414bdf0 con 0x7f3ef406ca70 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.986+0000 7f3ef17fa640 1 -- 172.21.15.94:0/1932518758 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4007e20 con 0x7f3ef406ca70 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.986+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef414c3b0 con 0x7f3ef406ca70 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.987+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef4075300 con 0x7f3ef406ca70 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.987+0000 7f3ef17fa640 1 -- 172.21.15.94:0/1932518758 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ee401a070 con 0x7f3ef406ca70 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.988+0000 7f3ef17fa640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ee006b500 0x7f3ee006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.988+0000 7f3ef17fa640 1 -- 172.21.15.94:0/1932518758 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3ee4089900 con 0x7f3ef406ca70 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.988+0000 7f3ef3fff640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ee006b500 0x7f3ee006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:47.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.988+0000 7f3ef3fff640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ee006b500 0x7f3ee006d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40c1a80 tx=0x7f3ee8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:47.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:47.992+0000 7f3ef17fa640 1 -- 172.21.15.94:0/1932518758 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ef4075300 con 0x7f3ef406ca70 2024-03-06T07:18:48.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.163+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3ef406ce80 con 0x7f3ef406ca70 2024-03-06T07:18:48.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.163+0000 7f3ef17fa640 1 -- 172.21.15.94:0/1932518758 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3ef406ce80 con 0x7f3ef406ca70 2024-03-06T07:18:48.165 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:48.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ee006b500 msgr2=0x7f3ee006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ee006b500 0x7f3ee006d9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40c1a80 tx=0x7f3ee8009040 comp rx=0 tx=0).stop 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ef406ca70 msgr2=0x7f3ef40c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ef406ca70 0x7f3ef40c3e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c730 tx=0x7f3ee400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 shutdown_connections 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ee006b500 0x7f3ee006d9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ef4155c30 0x7f3ef41eefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ef40b0070 0x7f3ef41eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 --2- 172.21.15.94:0/1932518758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ef406ca70 0x7f3ef40c3e30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 >> 172.21.15.94:0/1932518758 conn(0x7f3ef40ac590 msgr2=0x7f3ef40aee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.165+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 shutdown_connections 2024-03-06T07:18:48.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.166+0000 7f3efaa8f640 1 -- 172.21.15.94:0/1932518758 wait complete. 2024-03-06T07:18:48.178 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:48.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.276+0000 7f2165d5e640 1 Processor -- start 2024-03-06T07:18:48.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.276+0000 7f2165d5e640 1 -- start start 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f2165d5e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2160072f70 0x7f2160154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f2165d5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f2160154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f2165d5e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21600bdc30 con 0x7f21600c6560 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f2165d5e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21600bd690 con 0x7f2160072f70 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f2165d5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21600bd950 con 0x7f21600bbf80 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f215effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f21600c6560 0x7f2160156670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47356/0 (socket says 172.21.15.94:47356) 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f215effd640 1 -- 172.21.15.94:0/312040938 learned_addr learned my addr 172.21.15.94:0/312040938 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f215ffff640 1 --2- 172.21.15.94:0/312040938 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f2160154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f215f7fe640 1 --2- 172.21.15.94:0/312040938 >> v2:172.21.15.145:3300/0 conn(0x7f2160072f70 0x7f2160154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f215e7fc640 1 -- 172.21.15.94:0/312040938 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165848834 0 0) 0x7f21600bdc30 con 0x7f21600c6560 2024-03-06T07:18:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.277+0000 7f215e7fc640 1 -- 172.21.15.94:0/312040938 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2148003520 con 0x7f21600c6560 2024-03-06T07:18:48.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.278+0000 7f215ffff640 1 -- 172.21.15.94:0/312040938 >> v2:172.21.15.145:3300/0 conn(0x7f2160072f70 msgr2=0x7f2160154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:48.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.278+0000 7f215ffff640 1 --2- 172.21.15.94:0/312040938 >> v2:172.21.15.145:3300/0 conn(0x7f2160072f70 0x7f2160154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.278+0000 7f215ffff640 1 -- 172.21.15.94:0/312040938 >> v1:172.21.15.94:6789/0 conn(0x7f21600c6560 legacy=0x7f2160156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.278+0000 7f215ffff640 1 -- 172.21.15.94:0/312040938 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21600c5540 con 0x7f21600bbf80 2024-03-06T07:18:48.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.278+0000 7f215f7fe640 1 --2- 172.21.15.94:0/312040938 >> v2:172.21.15.145:3300/0 conn(0x7f2160072f70 0x7f2160154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:48.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.278+0000 7f215ffff640 1 --2- 172.21.15.94:0/312040938 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f2160154ab0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2158009d90 tx=0x7f215802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6159cdb4983a7fc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f215e7fc640 1 -- 172.21.15.94:0/312040938 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158039070 con 0x7f21600bbf80 2024-03-06T07:18:48.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f215e7fc640 1 -- 172.21.15.94:0/312040938 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215802cbc0 con 0x7f21600bbf80 2024-03-06T07:18:48.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f215e7fc640 1 -- 172.21.15.94:0/312040938 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158034440 con 0x7f21600bbf80 2024-03-06T07:18:48.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 -- 172.21.15.94:0/312040938 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 msgr2=0x7f2160154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 --2- 172.21.15.94:0/312040938 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f2160154ab0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2158009d90 tx=0x7f215802be10 comp rx=0 tx=0).stop 2024-03-06T07:18:48.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 -- 172.21.15.94:0/312040938 shutdown_connections 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 --2- 172.21.15.94:0/312040938 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f2160154ab0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 --2- 172.21.15.94:0/312040938 >> v2:172.21.15.145:3300/0 conn(0x7f2160072f70 0x7f2160154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 -- 172.21.15.94:0/312040938 >> 172.21.15.94:0/312040938 conn(0x7f216006b1a0 msgr2=0x7f216006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 -- 172.21.15.94:0/312040938 shutdown_connections 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.279+0000 7f2165d5e640 1 -- 172.21.15.94:0/312040938 wait complete. 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.280+0000 7f2165d5e640 1 Processor -- start 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.280+0000 7f2165d5e640 1 -- start start 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f2165d5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2160072f70 0x7f216014c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f2165d5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f21601ebe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f2165d5e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21600c6560 0x7f21600aa880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f2165d5e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2160073350 con 0x7f2160072f70 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f2165d5e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21600bba60 con 0x7f21600c6560 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f2165d5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21600734d0 con 0x7f21600bbf80 2024-03-06T07:18:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2160072f70 0x7f216014c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2160072f70 0x7f216014c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37748/0 (socket says 172.21.15.94:37748) 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 -- 172.21.15.94:0/3592012586 learned_addr learned my addr 172.21.15.94:0/3592012586 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215effd640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f21601ebe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215ffff640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21600c6560 0x7f21600aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 -- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 msgr2=0x7f21601ebe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f21601ebe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 -- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21600c6560 msgr2=0x7f21600aa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21600c6560 0x7f21600aa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.281+0000 7f215f7fe640 1 -- 172.21.15.94:0/3592012586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21601ec6d0 con 0x7f2160072f70 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.282+0000 7f215f7fe640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2160072f70 0x7f216014c7e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f215000ebe0 tx=0x7f215000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.282+0000 7f2147fff640 1 -- 172.21.15.94:0/3592012586 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150017070 con 0x7f2160072f70 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.282+0000 7f2147fff640 1 -- 172.21.15.94:0/3592012586 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215000cc30 con 0x7f2160072f70 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.282+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21601ec9b0 con 0x7f2160072f70 2024-03-06T07:18:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.282+0000 7f2147fff640 1 -- 172.21.15.94:0/3592012586 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150013470 con 0x7f2160072f70 2024-03-06T07:18:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.282+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216014ee50 con 0x7f2160072f70 2024-03-06T07:18:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.283+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21600688f0 con 0x7f2160072f70 2024-03-06T07:18:48.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.283+0000 7f2147fff640 1 -- 172.21.15.94:0/3592012586 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f215001a070 con 0x7f2160072f70 2024-03-06T07:18:48.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.283+0000 7f2147fff640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f214c06b4d0 0x7f214c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.284+0000 7f2147fff640 1 -- 172.21.15.94:0/3592012586 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f215008a7a0 con 0x7f2160072f70 2024-03-06T07:18:48.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.284+0000 7f215effd640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f214c06b4d0 0x7f214c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.284+0000 7f215effd640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f214c06b4d0 0x7f214c06d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2154005880 tx=0x7f2154009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.288+0000 7f2147fff640 1 -- 172.21.15.94:0/3592012586 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21600688f0 con 0x7f2160072f70 2024-03-06T07:18:48.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.459+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2160069460 con 0x7f2160072f70 2024-03-06T07:18:48.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.461+0000 7f2147fff640 1 -- 172.21.15.94:0/3592012586 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f2160069460 con 0x7f2160072f70 2024-03-06T07:18:48.463 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:48.463 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.462+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f214c06b4d0 msgr2=0x7f214c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.462+0000 7f2165d5e640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f214c06b4d0 0x7f214c06d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2154005880 tx=0x7f2154009040 comp rx=0 tx=0).stop 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.462+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2160072f70 msgr2=0x7f216014c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2160072f70 0x7f216014c7e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f215000ebe0 tx=0x7f215000c520 comp rx=0 tx=0).stop 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 shutdown_connections 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f214c06b4d0 0x7f214c06d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21600c6560 0x7f21600aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21600bbf80 0x7f21601ebe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 --2- 172.21.15.94:0/3592012586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2160072f70 0x7f216014c7e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 >> 172.21.15.94:0/3592012586 conn(0x7f216006b1a0 msgr2=0x7f2160157090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 shutdown_connections 2024-03-06T07:18:48.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.463+0000 7f2165d5e640 1 -- 172.21.15.94:0/3592012586 wait complete. 2024-03-06T07:18:48.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:48.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:48.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.578+0000 7f391f9d6640 1 Processor -- start 2024-03-06T07:18:48.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.578+0000 7f391f9d6640 1 -- start start 2024-03-06T07:18:48.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391f9d6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f39180c5990 0x7f39180c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391f9d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39180ab170 0x7f391806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391f9d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f391806c730 0x7f39181523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391f9d6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391806cba0 con 0x7f39180ab170 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391f9d6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39180c3ef0 con 0x7f39180c5990 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391f9d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39181556d0 con 0x7f391806c730 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39180ab170 0x7f391806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39180ab170 0x7f391806c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33724/0 (socket says 172.21.15.94:33724) 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 -- 172.21.15.94:0/764898344 learned_addr learned my addr 172.21.15.94:0/764898344 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391df73640 1 --2- 172.21.15.94:0/764898344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f391806c730 0x7f39181523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391d772640 1 --2- 172.21.15.94:0/764898344 >> v2:172.21.15.145:3300/0 conn(0x7f39180c5990 0x7f39180c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 -- 172.21.15.94:0/764898344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f391806c730 msgr2=0x7f39181523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 --2- 172.21.15.94:0/764898344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f391806c730 0x7f39181523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 -- 172.21.15.94:0/764898344 >> v2:172.21.15.145:3300/0 conn(0x7f39180c5990 msgr2=0x7f39180c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 --2- 172.21.15.94:0/764898344 >> v2:172.21.15.145:3300/0 conn(0x7f39180c5990 0x7f39180c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.579+0000 7f391cf71640 1 -- 172.21.15.94:0/764898344 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39181559b0 con 0x7f39180ab170 2024-03-06T07:18:48.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391cf71640 1 --2- 172.21.15.94:0/764898344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39180ab170 0x7f391806c1f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f39180c12a0 tx=0x7f390802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e997af4224a7248 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f390ffff640 1 -- 172.21.15.94:0/764898344 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390803b070 con 0x7f39180ab170 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f390ffff640 1 -- 172.21.15.94:0/764898344 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3908036440 con 0x7f39180ab170 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f390ffff640 1 -- 172.21.15.94:0/764898344 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908036810 con 0x7f39180ab170 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391f9d6640 1 -- 172.21.15.94:0/764898344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39180ab170 msgr2=0x7f391806c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391f9d6640 1 --2- 172.21.15.94:0/764898344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39180ab170 0x7f391806c1f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f39180c12a0 tx=0x7f390802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391f9d6640 1 -- 172.21.15.94:0/764898344 shutdown_connections 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391f9d6640 1 --2- 172.21.15.94:0/764898344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f391806c730 0x7f39181523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391f9d6640 1 --2- 172.21.15.94:0/764898344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39180ab170 0x7f391806c1f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391f9d6640 1 --2- 172.21.15.94:0/764898344 >> v2:172.21.15.145:3300/0 conn(0x7f39180c5990 0x7f39180c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.580+0000 7f391f9d6640 1 -- 172.21.15.94:0/764898344 >> 172.21.15.94:0/764898344 conn(0x7f39180c6710 msgr2=0x7f39180c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.581+0000 7f391f9d6640 1 -- 172.21.15.94:0/764898344 shutdown_connections 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.581+0000 7f391f9d6640 1 -- 172.21.15.94:0/764898344 wait complete. 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 Processor -- start 2024-03-06T07:18:48.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 -- start start 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f391806c730 0x7f3918154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39180ab170 0x7f391814c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39180c5990 0x7f39181488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918156490 con 0x7f391806c730 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918155f70 con 0x7f39180ab170 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391f9d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3918156610 con 0x7f39180c5990 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391d772640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f391806c730 0x7f3918154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391d772640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f391806c730 0x7f3918154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33728/0 (socket says 172.21.15.94:33728) 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391d772640 1 -- 172.21.15.94:0/329331049 learned_addr learned my addr 172.21.15.94:0/329331049 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391cf71640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39180ab170 0x7f391814c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.582+0000 7f391df73640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39180c5990 0x7f39181488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391d772640 1 -- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39180c5990 msgr2=0x7f39181488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391d772640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39180c5990 0x7f39181488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391d772640 1 -- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39180ab170 msgr2=0x7f391814c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391d772640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39180ab170 0x7f391814c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391d772640 1 -- 172.21.15.94:0/329331049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39181e6e20 con 0x7f391806c730 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391d772640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f391806c730 0x7f3918154590 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f391400e9b0 tx=0x7f391400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f390dffb640 1 -- 172.21.15.94:0/329331049 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914017070 con 0x7f391806c730 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f390dffb640 1 -- 172.21.15.94:0/329331049 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391400c9a0 con 0x7f391806c730 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39181e7100 con 0x7f391806c730 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f390dffb640 1 -- 172.21.15.94:0/329331049 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391400cd90 con 0x7f391806c730 2024-03-06T07:18:48.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.583+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3918155150 con 0x7f391806c730 2024-03-06T07:18:48.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.584+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391804f330 con 0x7f391806c730 2024-03-06T07:18:48.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.585+0000 7f390dffb640 1 -- 172.21.15.94:0/329331049 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f391401a070 con 0x7f391806c730 2024-03-06T07:18:48.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.585+0000 7f390dffb640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38fc06b4d0 0x7f38fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.585+0000 7f390dffb640 1 -- 172.21.15.94:0/329331049 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f391408a7c0 con 0x7f391806c730 2024-03-06T07:18:48.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.585+0000 7f391cf71640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38fc06b4d0 0x7f38fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.586+0000 7f391cf71640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38fc06b4d0 0x7f38fc06d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3908038000 tx=0x7f3908032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.589+0000 7f390dffb640 1 -- 172.21.15.94:0/329331049 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3914058cb0 con 0x7f391806c730 2024-03-06T07:18:48.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.759+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39180ab550 con 0x7f391806c730 2024-03-06T07:18:48.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.759+0000 7f390dffb640 1 -- 172.21.15.94:0/329331049 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f391405cd50 con 0x7f391806c730 2024-03-06T07:18:48.761 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38fc06b4d0 msgr2=0x7f38fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38fc06b4d0 0x7f38fc06d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3908038000 tx=0x7f3908032000 comp rx=0 tx=0).stop 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f391806c730 msgr2=0x7f3918154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f391806c730 0x7f3918154590 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f391400e9b0 tx=0x7f391400ee70 comp rx=0 tx=0).stop 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 shutdown_connections 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38fc06b4d0 0x7f38fc06d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39180c5990 0x7f39181488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39180ab170 0x7f391814c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 --2- 172.21.15.94:0/329331049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f391806c730 0x7f3918154590 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 >> 172.21.15.94:0/329331049 conn(0x7f39180c6710 msgr2=0x7f39180c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 shutdown_connections 2024-03-06T07:18:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.761+0000 7f391f9d6640 1 -- 172.21.15.94:0/329331049 wait complete. 2024-03-06T07:18:48.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:48.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.875+0000 7f16cfa84640 1 Processor -- start 2024-03-06T07:18:48.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.875+0000 7f16cfa84640 1 -- start start 2024-03-06T07:18:48.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cfa84640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f16c80c5990 0x7f16c80c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cfa84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cfa84640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c80c3ef0 con 0x7f16c806c730 2024-03-06T07:18:48.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cfa84640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c806cbe0 con 0x7f16c80c5990 2024-03-06T07:18:48.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cfa84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c81556d0 con 0x7f16c80ab170 2024-03-06T07:18:48.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd01f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd820640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f16c80c5990 0x7f16c80c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd820640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f16c80c5990 0x7f16c80c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:46244/0 (socket says 172.21.15.94:46244) 2024-03-06T07:18:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd01f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c806c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:59154/0 (socket says 172.21.15.94:59154) 2024-03-06T07:18:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd820640 1 -- 172.21.15.94:0/2294770510 learned_addr learned my addr 172.21.15.94:0/2294770510 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd01f640 1 -- 172.21.15.94:0/2294770510 >> v2:172.21.15.145:3300/0 conn(0x7f16c80c5990 msgr2=0x7f16c80c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd01f640 1 --2- 172.21.15.94:0/2294770510 >> v2:172.21.15.145:3300/0 conn(0x7f16c80c5990 0x7f16c80c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd01f640 1 -- 172.21.15.94:0/2294770510 >> v1:172.21.15.145:6789/0 conn(0x7f16c806c730 legacy=0x7f16c81523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.876+0000 7f16cd01f640 1 -- 172.21.15.94:0/2294770510 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c81559b0 con 0x7f16c80ab170 2024-03-06T07:18:48.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.877+0000 7f16cd01f640 1 --2- 172.21.15.94:0/2294770510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c806c1f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f16b80097a0 tx=0x7f16b802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8001fc6c04fd50c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.877+0000 7f16cc81e640 1 -- 172.21.15.94:0/2294770510 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b80315a0 con 0x7f16c80ab170 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.877+0000 7f16cc81e640 1 -- 172.21.15.94:0/2294770510 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8037070 con 0x7f16c80ab170 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.877+0000 7f16cc81e640 1 -- 172.21.15.94:0/2294770510 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8031960 con 0x7f16c80ab170 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 -- 172.21.15.94:0/2294770510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 msgr2=0x7f16c806c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 --2- 172.21.15.94:0/2294770510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c806c1f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f16b80097a0 tx=0x7f16b802b840 comp rx=0 tx=0).stop 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 -- 172.21.15.94:0/2294770510 shutdown_connections 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 --2- 172.21.15.94:0/2294770510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c806c1f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 --2- 172.21.15.94:0/2294770510 >> v2:172.21.15.145:3300/0 conn(0x7f16c80c5990 0x7f16c80c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 -- 172.21.15.94:0/2294770510 >> 172.21.15.94:0/2294770510 conn(0x7f16c80c6710 msgr2=0x7f16c80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 -- 172.21.15.94:0/2294770510 shutdown_connections 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.878+0000 7f16cfa84640 1 -- 172.21.15.94:0/2294770510 wait complete. 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 Processor -- start 2024-03-06T07:18:48.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 -- start start 2024-03-06T07:18:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16c806c730 0x7f16c8154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c8148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16c80c5990 0x7f16c8148c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8156490 con 0x7f16c806c730 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8155f70 con 0x7f16c80c5990 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.879+0000 7f16cfa84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8156610 con 0x7f16c80ab170 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16c806c730 0x7f16c8154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16c806c730 0x7f16c8154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33736/0 (socket says 172.21.15.94:33736) 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 -- 172.21.15.94:0/1674913282 learned_addr learned my addr 172.21.15.94:0/1674913282 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16ce021640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16c80c5990 0x7f16c8148c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd01f640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c8148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 -- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 msgr2=0x7f16c8148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c8148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 -- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16c80c5990 msgr2=0x7f16c8148c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16c80c5990 0x7f16c8148c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 -- 172.21.15.94:0/1674913282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c814da30 con 0x7f16c806c730 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cd820640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16c806c730 0x7f16c8154930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f16c400e840 tx=0x7f16c400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16be7fc640 1 -- 172.21.15.94:0/1674913282 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c4017070 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16be7fc640 1 -- 172.21.15.94:0/1674913282 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c400c8e0 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16be7fc640 1 -- 172.21.15.94:0/1674913282 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c400ccd0 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c81ede70 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.880+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c81ee430 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.881+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c804f330 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.882+0000 7f16be7fc640 1 -- 172.21.15.94:0/1674913282 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16c401a070 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.882+0000 7f16be7fc640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.882+0000 7f16be7fc640 1 -- 172.21.15.94:0/1674913282 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f16c408a6a0 con 0x7f16c806c730 2024-03-06T07:18:48.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.882+0000 7f16cd01f640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:48.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.883+0000 7f16cd01f640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16ac06b4d0 0x7f16ac06d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f16b8036000 tx=0x7f16b803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:48.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:48.886+0000 7f16be7fc640 1 -- 172.21.15.94:0/1674913282 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16c4058c20 con 0x7f16c806c730 2024-03-06T07:18:49.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.056+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f16c80c1740 con 0x7f16c806c730 2024-03-06T07:18:49.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.057+0000 7f16be7fc640 1 -- 172.21.15.94:0/1674913282 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f16c405ccc0 con 0x7f16c806c730 2024-03-06T07:18:49.059 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.058+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16ac06b4d0 msgr2=0x7f16ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.058+0000 7f16cfa84640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16ac06b4d0 0x7f16ac06d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f16b8036000 tx=0x7f16b803d040 comp rx=0 tx=0).stop 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.058+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16c806c730 msgr2=0x7f16c8154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.058+0000 7f16cfa84640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16c806c730 0x7f16c8154930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f16c400e840 tx=0x7f16c400ed00 comp rx=0 tx=0).stop 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 shutdown_connections 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16c80c5990 0x7f16c8148c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16c80ab170 0x7f16c8148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 --2- 172.21.15.94:0/1674913282 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16c806c730 0x7f16c8154930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 >> 172.21.15.94:0/1674913282 conn(0x7f16c80c6710 msgr2=0x7f16c80c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:49.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 shutdown_connections 2024-03-06T07:18:49.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.059+0000 7f16cfa84640 1 -- 172.21.15.94:0/1674913282 wait complete. 2024-03-06T07:18:49.071 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:49.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.174+0000 7f604c9b6640 1 Processor -- start 2024-03-06T07:18:49.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.174+0000 7f604c9b6640 1 -- start start 2024-03-06T07:18:49.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604c9b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f604815c8d0 0x7f604815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604c9b6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604806b440 con 0x7f604806afc0 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604c9b6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60480c1c90 con 0x7f604814d350 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604c9b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60480ac750 con 0x7f604815c8d0 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f6045d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f604806afc0 0x7f60480ab950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50624/0 (socket says 172.21.15.94:50624) 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f6045d9b640 1 -- 172.21.15.94:0/239660741 learned_addr learned my addr 172.21.15.94:0/239660741 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f6046d9d640 1 --2- 172.21.15.94:0/239660741 >> v2:172.21.15.145:3300/0 conn(0x7f604815c8d0 0x7f604815ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642224905 0 0) 0x7f603c001110 con 0x7f604806afc0 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030003520 con 0x7f604806afc0 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663955450 0 0) 0x7f60480c1c90 con 0x7f604814d350 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.175+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c001110 con 0x7f604814d350 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4173139087 0 0) 0x7f6030003520 con 0x7f604806afc0 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 >> v2:172.21.15.145:3300/0 conn(0x7f604815c8d0 msgr2=0x7f604815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:49.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 --2- 172.21.15.94:0/239660741 >> v2:172.21.15.145:3300/0 conn(0x7f604815c8d0 0x7f604815ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 >> v1:172.21.15.145:6789/0 conn(0x7f604814d350 legacy=0x7f60480acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60480ac470 con 0x7f604806afc0 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f603c001840 con 0x7f604806afc0 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 >> v1:172.21.15.94:6789/0 conn(0x7f604806afc0 legacy=0x7f60480ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6030003b40 0x7f6030003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030004800 0x7f6030004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60300051c0 0x7f603000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603000c0c0 con 0x7f6030004800 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603000c270 con 0x7f60300051c0 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603000c470 con 0x7f6030003b40 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6045d9b640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030004800 0x7f6030004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f604659c640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6030003b40 0x7f6030003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6046d9d640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60300051c0 0x7f603000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6045d9b640 1 -- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6030003b40 msgr2=0x7f6030003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6045d9b640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6030003b40 0x7f6030003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6045d9b640 1 -- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60300051c0 msgr2=0x7f603000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6045d9b640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60300051c0 0x7f603000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6046d9d640 1 -- 172.21.15.94:0/239660741 reap_dead start 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.176+0000 7f6045d9b640 1 -- 172.21.15.94:0/239660741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030003520 con 0x7f6030004800 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f6045d9b640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030004800 0x7f6030004c80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c4c0 tx=0x7f6030005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55eff1e42024f6a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c00c0e0 con 0x7f6030004800 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c030070 con 0x7f6030004800 2024-03-06T07:18:49.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604559a640 1 -- 172.21.15.94:0/239660741 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c0058d0 con 0x7f6030004800 2024-03-06T07:18:49.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604c9b6640 1 -- 172.21.15.94:0/239660741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030004800 msgr2=0x7f6030004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604c9b6640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030004800 0x7f6030004c80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c4c0 tx=0x7f6030005830 comp rx=0 tx=0).stop 2024-03-06T07:18:49.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604c9b6640 1 -- 172.21.15.94:0/239660741 shutdown_connections 2024-03-06T07:18:49.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604c9b6640 1 --2- 172.21.15.94:0/239660741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030004800 0x7f6030004c80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.177+0000 7f604c9b6640 1 -- 172.21.15.94:0/239660741 >> 172.21.15.94:0/239660741 conn(0x7f60480c3b40 msgr2=0x7f60480c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:49.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.178+0000 7f604c9b6640 1 -- 172.21.15.94:0/239660741 shutdown_connections 2024-03-06T07:18:49.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.178+0000 7f604c9b6640 1 -- 172.21.15.94:0/239660741 wait complete. 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 Processor -- start 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 -- start start 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 0x7f60480c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60481e8d40 0x7f604814e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60480c15c0 0x7f604814edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6048153c00 con 0x7f60480c15c0 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60480c1a40 con 0x7f6030004800 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.179+0000 7f604c9b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60480aa160 con 0x7f60481e8d40 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60480c15c0 0x7f604814edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60480c15c0 0x7f604814edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33742/0 (socket says 172.21.15.94:33742) 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 -- 172.21.15.94:0/330689825 learned_addr learned my addr 172.21.15.94:0/330689825 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:49.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f604659c640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 0x7f60480c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6045d9b640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60481e8d40 0x7f604814e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 -- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60481e8d40 msgr2=0x7f604814e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60481e8d40 0x7f604814e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 -- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 msgr2=0x7f60480c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 0x7f60480c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 -- 172.21.15.94:0/330689825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604814dab0 con 0x7f60480c15c0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6046d9d640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60480c15c0 0x7f604814edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60480688b0 tx=0x7f6040011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6036ffd640 1 -- 172.21.15.94:0/330689825 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040004030 con 0x7f60480c15c0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6036ffd640 1 -- 172.21.15.94:0/330689825 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604001b070 con 0x7f60480c15c0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f6036ffd640 1 -- 172.21.15.94:0/330689825 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60400043f0 con 0x7f60480c15c0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604814dd90 con 0x7f60480c15c0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.180+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604814e350 con 0x7f60480c15c0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.181+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604804f330 con 0x7f60480c15c0 2024-03-06T07:18:49.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.182+0000 7f6036ffd640 1 -- 172.21.15.94:0/330689825 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6040012040 con 0x7f60480c15c0 2024-03-06T07:18:49.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.182+0000 7f6036ffd640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b500 0x7f602806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.182+0000 7f6036ffd640 1 -- 172.21.15.94:0/330689825 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f604008a110 con 0x7f60480c15c0 2024-03-06T07:18:49.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.182+0000 7f604659c640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b500 0x7f602806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.182+0000 7f604659c640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b500 0x7f602806d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6038000f80 tx=0x7f6038009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.186+0000 7f6036ffd640 1 -- 172.21.15.94:0/330689825 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6040058ce0 con 0x7f60480c15c0 2024-03-06T07:18:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.357+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6048150890 con 0x7f60480c15c0 2024-03-06T07:18:49.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.358+0000 7f6036ffd640 1 -- 172.21.15.94:0/330689825 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f60400585e0 con 0x7f60480c15c0 2024-03-06T07:18:49.360 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b500 msgr2=0x7f602806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b500 0x7f602806d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6038000f80 tx=0x7f6038009040 comp rx=0 tx=0).stop 2024-03-06T07:18:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60480c15c0 msgr2=0x7f604814edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60480c15c0 0x7f604814edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60480688b0 tx=0x7f6040011c30 comp rx=0 tx=0).stop 2024-03-06T07:18:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 shutdown_connections 2024-03-06T07:18:49.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b500 0x7f602806d9c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60480c15c0 0x7f604814edb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60481e8d40 0x7f604814e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 --2- 172.21.15.94:0/330689825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 0x7f60480c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 >> 172.21.15.94:0/330689825 conn(0x7f60480c3b40 msgr2=0x7f60480ab8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 shutdown_connections 2024-03-06T07:18:49.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.360+0000 7f604c9b6640 1 -- 172.21.15.94:0/330689825 wait complete. 2024-03-06T07:18:49.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:49.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.474+0000 7ff4715a2640 1 Processor -- start 2024-03-06T07:18:49.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.474+0000 7ff4715a2640 1 -- start start 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff4715a2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff46c1503d0 0x7ff46c0ae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff4715a2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c0c3120 0x7ff46c0c3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff4715a2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c157170 con 0x7ff46c1563f0 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff4715a2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff46c0adbf0 con 0x7ff46c1503d0 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff4715a2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c0ab770 con 0x7ff46c0c3120 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff46bfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff46c1503d0 0x7ff46c0ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff46b7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c0c3120 0x7ff46c0c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff470da1640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff46c1563f0 0x7ff46c156880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:58446/0 (socket says 172.21.15.94:58446) 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff46bfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff46c1503d0 0x7ff46c0ae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:46294/0 (socket says 172.21.15.94:46294) 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.475+0000 7ff470da1640 1 -- 172.21.15.94:0/3048170484 learned_addr learned my addr 172.21.15.94:0/3048170484 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.476+0000 7ff46b7fe640 1 -- 172.21.15.94:0/3048170484 >> v2:172.21.15.145:3300/0 conn(0x7ff46c1503d0 msgr2=0x7ff46c0ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:49.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.476+0000 7ff46b7fe640 1 --2- 172.21.15.94:0/3048170484 >> v2:172.21.15.145:3300/0 conn(0x7ff46c1503d0 0x7ff46c0ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.476+0000 7ff46b7fe640 1 -- 172.21.15.94:0/3048170484 >> v1:172.21.15.145:6789/0 conn(0x7ff46c1563f0 legacy=0x7ff46c156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.476+0000 7ff46b7fe640 1 -- 172.21.15.94:0/3048170484 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0aba50 con 0x7ff46c0c3120 2024-03-06T07:18:49.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.476+0000 7ff46affd640 1 -- 172.21.15.94:0/3048170484 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295503747 0 0) 0x7ff46c157170 con 0x7ff46c1563f0 2024-03-06T07:18:49.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.476+0000 7ff46bfff640 1 --2- 172.21.15.94:0/3048170484 >> v2:172.21.15.145:3300/0 conn(0x7ff46c1503d0 0x7ff46c0ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:49.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.476+0000 7ff46b7fe640 1 --2- 172.21.15.94:0/3048170484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c0c3120 0x7ff46c0c3820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff4600097a0 tx=0x7ff46002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=873bad2920de5f1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff46affd640 1 -- 172.21.15.94:0/3048170484 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4600315a0 con 0x7ff46c0c3120 2024-03-06T07:18:49.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff46affd640 1 -- 172.21.15.94:0/3048170484 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff460037070 con 0x7ff46c0c3120 2024-03-06T07:18:49.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff46affd640 1 -- 172.21.15.94:0/3048170484 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460031960 con 0x7ff46c0c3120 2024-03-06T07:18:49.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 -- 172.21.15.94:0/3048170484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c0c3120 msgr2=0x7ff46c0c3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 --2- 172.21.15.94:0/3048170484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c0c3120 0x7ff46c0c3820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff4600097a0 tx=0x7ff46002b840 comp rx=0 tx=0).stop 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 -- 172.21.15.94:0/3048170484 shutdown_connections 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 --2- 172.21.15.94:0/3048170484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c0c3120 0x7ff46c0c3820 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 --2- 172.21.15.94:0/3048170484 >> v2:172.21.15.145:3300/0 conn(0x7ff46c1503d0 0x7ff46c0ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 -- 172.21.15.94:0/3048170484 >> 172.21.15.94:0/3048170484 conn(0x7ff46c0c5010 msgr2=0x7ff46c0c5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 -- 172.21.15.94:0/3048170484 shutdown_connections 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.477+0000 7ff4715a2640 1 -- 172.21.15.94:0/3048170484 wait complete. 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.478+0000 7ff4715a2640 1 Processor -- start 2024-03-06T07:18:49.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff4715a2640 1 -- start start 2024-03-06T07:18:49.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff4715a2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff46c0c3120 0x7ff46c1ea930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff4715a2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c1503d0 0x7ff46c0c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff4715a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46c1563f0 0x7ff46c152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff4715a2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c073330 con 0x7ff46c1563f0 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff4715a2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c1507b0 con 0x7ff46c0c3120 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff4715a2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c0734b0 con 0x7ff46c1503d0 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff46bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff46c0c3120 0x7ff46c1ea930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff46b7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c1503d0 0x7ff46c0c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff470da1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46c1563f0 0x7ff46c152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff46bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff46c0c3120 0x7ff46c1ea930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:46302/0 (socket says 172.21.15.94:46302) 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.479+0000 7ff46bfff640 1 -- 172.21.15.94:0/2913172879 learned_addr learned my addr 172.21.15.94:0/2913172879 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff46b7fe640 1 -- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff46c0c3120 msgr2=0x7ff46c1ea930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff46b7fe640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff46c0c3120 0x7ff46c1ea930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff46b7fe640 1 -- 172.21.15.94:0/2913172879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46c1563f0 msgr2=0x7ff46c152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff46b7fe640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46c1563f0 0x7ff46c152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff46b7fe640 1 -- 172.21.15.94:0/2913172879 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c1eafb0 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff46bfff640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff46c0c3120 0x7ff46c1ea930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff470da1640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46c1563f0 0x7ff46c152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff46b7fe640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c1503d0 0x7ff46c0c1b50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff460033000 tx=0x7ff4600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff468ff9640 1 -- 172.21.15.94:0/2913172879 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460004230 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff468ff9640 1 -- 172.21.15.94:0/2913172879 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4600043f0 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46c1eb290 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff468ff9640 1 -- 172.21.15.94:0/2913172879 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460003850 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.480+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46c0c2090 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.481+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46c0684b0 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.481+0000 7ff468ff9640 1 -- 172.21.15.94:0/2913172879 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4600039f0 con 0x7ff46c1503d0 2024-03-06T07:18:49.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.482+0000 7ff468ff9640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45806b4d0 0x7ff45806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.482+0000 7ff468ff9640 1 -- 172.21.15.94:0/2913172879 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff46002f030 con 0x7ff46c1503d0 2024-03-06T07:18:49.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.482+0000 7ff46bfff640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45806b4d0 0x7ff45806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.482+0000 7ff46bfff640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45806b4d0 0x7ff45806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff45c009560 tx=0x7ff45c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.487+0000 7ff468ff9640 1 -- 172.21.15.94:0/2913172879 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff460041070 con 0x7ff46c1503d0 2024-03-06T07:18:49.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.662+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff46c0693c0 con 0x7ff46c1503d0 2024-03-06T07:18:49.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.662+0000 7ff468ff9640 1 -- 172.21.15.94:0/2913172879 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff46007ab70 con 0x7ff46c1503d0 2024-03-06T07:18:49.664 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45806b4d0 msgr2=0x7ff45806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45806b4d0 0x7ff45806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff45c009560 tx=0x7ff45c009190 comp rx=0 tx=0).stop 2024-03-06T07:18:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c1503d0 msgr2=0x7ff46c0c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c1503d0 0x7ff46c0c1b50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff460033000 tx=0x7ff4600046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 shutdown_connections 2024-03-06T07:18:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45806b4d0 0x7ff45806d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46c1563f0 0x7ff46c152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff46c1503d0 0x7ff46c0c1b50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 --2- 172.21.15.94:0/2913172879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff46c0c3120 0x7ff46c1ea930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 >> 172.21.15.94:0/2913172879 conn(0x7ff46c0c5010 msgr2=0x7ff46c14fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 shutdown_connections 2024-03-06T07:18:49.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.665+0000 7ff4715a2640 1 -- 172.21.15.94:0/2913172879 wait complete. 2024-03-06T07:18:49.678 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:49.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.776+0000 7f16d4039640 1 Processor -- start 2024-03-06T07:18:49.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.777+0000 7f16d4039640 1 -- start start 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.777+0000 7f16d4039640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f16cc159270 0x7f16cc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.777+0000 7f16d4039640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16cc0c73d0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.777+0000 7f16d4039640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16cc0be2e0 con 0x7f16cc14f360 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.777+0000 7f16d4039640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16cc159700 con 0x7f16cc159270 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d15d4640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f16cc0c88c0 0x7f16cc158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50630/0 (socket says 172.21.15.94:50630) 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d15d4640 1 -- 172.21.15.94:0/4260822306 learned_addr learned my addr 172.21.15.94:0/4260822306 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d25d6640 1 --2- 172.21.15.94:0/4260822306 >> v2:172.21.15.145:3300/0 conn(0x7f16cc159270 0x7f16cc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d0dd3640 1 -- 172.21.15.94:0/4260822306 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067086088 0 0) 0x7f16cc0c73d0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d0dd3640 1 -- 172.21.15.94:0/4260822306 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b8003520 con 0x7f16cc0c88c0 2024-03-06T07:18:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d25d6640 1 -- 172.21.15.94:0/4260822306 >> v1:172.21.15.145:6789/0 conn(0x7f16cc14f360 legacy=0x7f16cc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d0dd3640 1 -- 172.21.15.94:0/4260822306 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359889968 0 0) 0x7f16cc0be2e0 con 0x7f16cc14f360 2024-03-06T07:18:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d25d6640 1 -- 172.21.15.94:0/4260822306 >> v1:172.21.15.94:6789/0 conn(0x7f16cc0c88c0 legacy=0x7f16cc158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.778+0000 7f16d25d6640 1 -- 172.21.15.94:0/4260822306 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc0c7670 con 0x7f16cc159270 2024-03-06T07:18:49.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.779+0000 7f16d25d6640 1 --2- 172.21.15.94:0/4260822306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc159270 0x7f16cc155c10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f16c80097a0 tx=0x7f16c802b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f12b27b4ce06e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.779+0000 7f16d0dd3640 1 -- 172.21.15.94:0/4260822306 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c8037070 con 0x7f16cc159270 2024-03-06T07:18:49.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.779+0000 7f16d0dd3640 1 -- 172.21.15.94:0/4260822306 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c8030510 con 0x7f16cc159270 2024-03-06T07:18:49.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.779+0000 7f16d0dd3640 1 -- 172.21.15.94:0/4260822306 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc001670 con 0x7f16cc159270 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.779+0000 7f16d4039640 1 -- 172.21.15.94:0/4260822306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc159270 msgr2=0x7f16cc155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.779+0000 7f16d4039640 1 --2- 172.21.15.94:0/4260822306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc159270 0x7f16cc155c10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f16c80097a0 tx=0x7f16c802b6b0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.780+0000 7f16d4039640 1 -- 172.21.15.94:0/4260822306 shutdown_connections 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.780+0000 7f16d4039640 1 --2- 172.21.15.94:0/4260822306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc159270 0x7f16cc155c10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.780+0000 7f16d4039640 1 -- 172.21.15.94:0/4260822306 >> 172.21.15.94:0/4260822306 conn(0x7f16cc0ac570 msgr2=0x7f16cc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.780+0000 7f16d4039640 1 -- 172.21.15.94:0/4260822306 shutdown_connections 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.780+0000 7f16d4039640 1 -- 172.21.15.94:0/4260822306 wait complete. 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 Processor -- start 2024-03-06T07:18:49.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 -- start start 2024-03-06T07:18:49.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16cc0c88c0 0x7f16cc0c0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc14f360 0x7f16cc0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16cc159270 0x7f16cc1edef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0c7c30 con 0x7f16cc0c88c0 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0c5a50 con 0x7f16cc14f360 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d4039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0c5bd0 con 0x7f16cc159270 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d1dd5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16cc0c88c0 0x7f16cc0c0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d1dd5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16cc0c88c0 0x7f16cc0c0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33752/0 (socket says 172.21.15.94:33752) 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d1dd5640 1 -- 172.21.15.94:0/683958884 learned_addr learned my addr 172.21.15.94:0/683958884 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d15d4640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc14f360 0x7f16cc0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.781+0000 7f16d25d6640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16cc159270 0x7f16cc1edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d1dd5640 1 -- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16cc159270 msgr2=0x7f16cc1edef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d1dd5640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16cc159270 0x7f16cc1edef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d1dd5640 1 -- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc14f360 msgr2=0x7f16cc0ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d1dd5640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc14f360 0x7f16cc0ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d1dd5640 1 -- 172.21.15.94:0/683958884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc1eb8f0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d1dd5640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16cc0c88c0 0x7f16cc0c0bc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f16c400c8c0 tx=0x7f16c400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16c27fc640 1 -- 172.21.15.94:0/683958884 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c4017070 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16c27fc640 1 -- 172.21.15.94:0/683958884 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c4003dc0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16c27fc640 1 -- 172.21.15.94:0/683958884 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c40052b0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16cc0aefd0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.782+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16cc0ade30 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.783+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16cc0af4f0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.783+0000 7f16c27fc640 1 -- 172.21.15.94:0/683958884 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16c401a070 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.784+0000 7f16c27fc640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16b006b4d0 0x7f16b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.784+0000 7f16c27fc640 1 -- 172.21.15.94:0/683958884 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f16c4089b50 con 0x7f16cc0c88c0 2024-03-06T07:18:49.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.784+0000 7f16d15d4640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16b006b4d0 0x7f16b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:49.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.784+0000 7f16d15d4640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16b006b4d0 0x7f16b006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0c1720 tx=0x7f16bc0015b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:49.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.788+0000 7f16c27fc640 1 -- 172.21.15.94:0/683958884 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16c40580d0 con 0x7f16cc0c88c0 2024-03-06T07:18:49.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.958+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16cc0c8d50 con 0x7f16cc0c88c0 2024-03-06T07:18:49.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.959+0000 7f16c27fc640 1 -- 172.21.15.94:0/683958884 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f16c405c170 con 0x7f16cc0c88c0 2024-03-06T07:18:49.961 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:49.962 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16b006b4d0 msgr2=0x7f16b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16b006b4d0 0x7f16b006d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0c1720 tx=0x7f16bc0015b0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16cc0c88c0 msgr2=0x7f16cc0c0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16cc0c88c0 0x7f16cc0c0bc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f16c400c8c0 tx=0x7f16c400cd80 comp rx=0 tx=0).stop 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 shutdown_connections 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f16b006b4d0 0x7f16b006d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16cc159270 0x7f16cc1edef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16cc14f360 0x7f16cc0ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 --2- 172.21.15.94:0/683958884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16cc0c88c0 0x7f16cc0c0bc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 >> 172.21.15.94:0/683958884 conn(0x7f16cc0ac570 msgr2=0x7f16cc0c9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 shutdown_connections 2024-03-06T07:18:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:49.961+0000 7f16d4039640 1 -- 172.21.15.94:0/683958884 wait complete. 2024-03-06T07:18:49.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:49.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.076+0000 7f1d10dbc640 1 Processor -- start 2024-03-06T07:18:50.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.076+0000 7f1d10dbc640 1 -- start start 2024-03-06T07:18:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d10dbc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1d0c0c5990 0x7f1d0c0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d10dbc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1d0c06c900 0x7f1d0c0a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d10dbc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c0c3ef0 con 0x7f1d0c0ab170 2024-03-06T07:18:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d10dbc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c1556d0 con 0x7f1d0c0c5990 2024-03-06T07:18:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d10dbc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c1559b0 con 0x7f1d0c06c900 2024-03-06T07:18:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1d0c0c5990 0x7f1d0c0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1d0c0c5990 0x7f1d0c0c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33760/0 (socket says 172.21.15.94:33760) 2024-03-06T07:18:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0a59c640 1 -- 172.21.15.94:0/1518183944 learned_addr learned my addr 172.21.15.94:0/1518183944 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0ad9d640 1 --2- 172.21.15.94:0/1518183944 >> v2:172.21.15.145:3300/0 conn(0x7f1d0c06c900 0x7f1d0c0a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0a59c640 1 -- 172.21.15.94:0/1518183944 >> v2:172.21.15.145:3300/0 conn(0x7f1d0c06c900 msgr2=0x7f1d0c0a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0a59c640 1 --2- 172.21.15.94:0/1518183944 >> v2:172.21.15.145:3300/0 conn(0x7f1d0c06c900 0x7f1d0c0a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0a59c640 1 -- 172.21.15.94:0/1518183944 >> v1:172.21.15.145:6789/0 conn(0x7f1d0c0ab170 legacy=0x7f1d0c06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.077+0000 7f1d0a59c640 1 -- 172.21.15.94:0/1518183944 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c155c90 con 0x7f1d0c0c5990 2024-03-06T07:18:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.078+0000 7f1d0a59c640 1 --2- 172.21.15.94:0/1518183944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c0c0ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00bb10 tx=0x7f1cfc02e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d79a3fda41ad88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.078+0000 7f1d0959a640 1 -- 172.21.15.94:0/1518183944 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc03b070 con 0x7f1d0c0c5990 2024-03-06T07:18:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.078+0000 7f1d0959a640 1 -- 172.21.15.94:0/1518183944 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cfc036440 con 0x7f1d0c0c5990 2024-03-06T07:18:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.078+0000 7f1d0959a640 1 -- 172.21.15.94:0/1518183944 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc036810 con 0x7f1d0c0c5990 2024-03-06T07:18:50.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1518183944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 msgr2=0x7f1d0c0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1518183944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c0c0ab0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00bb10 tx=0x7f1cfc02e170 comp rx=0 tx=0).stop 2024-03-06T07:18:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1518183944 shutdown_connections 2024-03-06T07:18:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1518183944 >> v2:172.21.15.145:3300/0 conn(0x7f1d0c06c900 0x7f1d0c0a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1518183944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c0c0ab0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1518183944 >> 172.21.15.94:0/1518183944 conn(0x7f1d0c0c6710 msgr2=0x7f1d0c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1518183944 shutdown_connections 2024-03-06T07:18:50.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.079+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1518183944 wait complete. 2024-03-06T07:18:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.080+0000 7f1d10dbc640 1 Processor -- start 2024-03-06T07:18:50.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.080+0000 7f1d10dbc640 1 -- start start 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d0c06c900 0x7f1d0c153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0c0ab170 0x7f1d0c146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c1e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c156610 con 0x7f1d0c0c5990 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c156490 con 0x7f1d0c0ab170 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0c3af0 con 0x7f1d0c06c900 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c1e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c1e9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33762/0 (socket says 172.21.15.94:33762) 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 -- 172.21.15.94:0/1816315343 learned_addr learned my addr 172.21.15.94:0/1816315343 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d09d9b640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0c0ab170 0x7f1d0c146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0a59c640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d0c06c900 0x7f1d0c153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 -- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d0c06c900 msgr2=0x7f1d0c153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d0c06c900 0x7f1d0c153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 -- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0c0ab170 msgr2=0x7f1d0c146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0c0ab170 0x7f1d0c146d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 -- 172.21.15.94:0/1816315343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c0a8260 con 0x7f1d0c0c5990 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d0ad9d640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c1e9a80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400efc0 tx=0x7f1d0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1cfaffd640 1 -- 172.21.15.94:0/1816315343 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04018020 con 0x7f1d0c0c5990 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1cfaffd640 1 -- 172.21.15.94:0/1816315343 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d04014480 con 0x7f1d0c0c5990 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c1ea8d0 con 0x7f1d0c0c5990 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1cfaffd640 1 -- 172.21.15.94:0/1816315343 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04014870 con 0x7f1d0c0c5990 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.081+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c0a7f80 con 0x7f1d0c0c5990 2024-03-06T07:18:50.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.082+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0c04f330 con 0x7f1d0c0c5990 2024-03-06T07:18:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.083+0000 7f1cfaffd640 1 -- 172.21.15.94:0/1816315343 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d04005d10 con 0x7f1d0c0c5990 2024-03-06T07:18:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.083+0000 7f1cfaffd640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cec06b4d0 0x7f1cec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.083+0000 7f1cfaffd640 1 -- 172.21.15.94:0/1816315343 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1d0408ae30 con 0x7f1d0c0c5990 2024-03-06T07:18:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.083+0000 7f1d0a59c640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cec06b4d0 0x7f1cec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.084+0000 7f1d0a59c640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cec06b4d0 0x7f1cec06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc038000 tx=0x7f1cfc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.087+0000 7f1cfaffd640 1 -- 172.21.15.94:0/1816315343 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d04022070 con 0x7f1d0c0c5990 2024-03-06T07:18:50.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.258+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1d0c146ac0 con 0x7f1d0c0c5990 2024-03-06T07:18:50.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.258+0000 7f1cfaffd640 1 -- 172.21.15.94:0/1816315343 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1d04010040 con 0x7f1d0c0c5990 2024-03-06T07:18:50.260 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:50.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cec06b4d0 msgr2=0x7f1cec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cec06b4d0 0x7f1cec06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc038000 tx=0x7f1cfc032000 comp rx=0 tx=0).stop 2024-03-06T07:18:50.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 msgr2=0x7f1d0c1e9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c1e9a80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400efc0 tx=0x7f1d0400c490 comp rx=0 tx=0).stop 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 shutdown_connections 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cec06b4d0 0x7f1cec06d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1d0c0c5990 0x7f1d0c1e9a80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0c0ab170 0x7f1d0c146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 --2- 172.21.15.94:0/1816315343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d0c06c900 0x7f1d0c153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 >> 172.21.15.94:0/1816315343 conn(0x7f1d0c0c6710 msgr2=0x7f1d0c0c0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 shutdown_connections 2024-03-06T07:18:50.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.260+0000 7f1d10dbc640 1 -- 172.21.15.94:0/1816315343 wait complete. 2024-03-06T07:18:50.273 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:50.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.374+0000 7f095bbd7640 1 Processor -- start 2024-03-06T07:18:50.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.374+0000 7f095bbd7640 1 -- start start 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f095bbd7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09540b0070 0x7f0954155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f095bbd7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f09540c5380 0x7f09540c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f095bbd7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09540b0450 con 0x7f095406ca70 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f095bbd7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095406c020 con 0x7f09540b0070 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f095bbd7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095406cf20 con 0x7f09540c5380 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0959973640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09540b0070 0x7f0954155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0959973640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09540b0070 0x7f0954155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33778/0 (socket says 172.21.15.94:33778) 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0959973640 1 -- 172.21.15.94:0/3474268052 learned_addr learned my addr 172.21.15.94:0/3474268052 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:50.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f095a174640 1 --2- 172.21.15.94:0/3474268052 >> v2:172.21.15.145:3300/0 conn(0x7f09540c5380 0x7f09540c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0959973640 1 -- 172.21.15.94:0/3474268052 >> v2:172.21.15.145:3300/0 conn(0x7f09540c5380 msgr2=0x7f09540c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0959973640 1 --2- 172.21.15.94:0/3474268052 >> v2:172.21.15.145:3300/0 conn(0x7f09540c5380 0x7f09540c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0959973640 1 -- 172.21.15.94:0/3474268052 >> v1:172.21.15.94:6789/0 conn(0x7f095406ca70 legacy=0x7f09540c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0958971640 1 -- 172.21.15.94:0/3474268052 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744252259 0 0) 0x7f0944001110 con 0x7f095406ca70 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.375+0000 7f0959973640 1 -- 172.21.15.94:0/3474268052 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09540c77f0 con 0x7f09540b0070 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.376+0000 7f0959973640 1 --2- 172.21.15.94:0/3474268052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954155c30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f09540be030 tx=0x7f095002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e72635235e502c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.376+0000 7f0958971640 1 -- 172.21.15.94:0/3474268052 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950035020 con 0x7f09540b0070 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.376+0000 7f0958971640 1 -- 172.21.15.94:0/3474268052 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0950031870 con 0x7f09540b0070 2024-03-06T07:18:50.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.376+0000 7f0958971640 1 -- 172.21.15.94:0/3474268052 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950031c40 con 0x7f09540b0070 2024-03-06T07:18:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 -- 172.21.15.94:0/3474268052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 msgr2=0x7f0954155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 --2- 172.21.15.94:0/3474268052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954155c30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f09540be030 tx=0x7f095002b910 comp rx=0 tx=0).stop 2024-03-06T07:18:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 -- 172.21.15.94:0/3474268052 shutdown_connections 2024-03-06T07:18:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 --2- 172.21.15.94:0/3474268052 >> v2:172.21.15.145:3300/0 conn(0x7f09540c5380 0x7f09540c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 --2- 172.21.15.94:0/3474268052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954155c30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 -- 172.21.15.94:0/3474268052 >> 172.21.15.94:0/3474268052 conn(0x7f09540ac590 msgr2=0x7f09540ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 -- 172.21.15.94:0/3474268052 shutdown_connections 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.377+0000 7f095bbd7640 1 -- 172.21.15.94:0/3474268052 wait complete. 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.378+0000 7f095bbd7640 1 Processor -- start 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.378+0000 7f095bbd7640 1 -- start start 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f095406ca70 0x7f09540c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09540c5380 0x7f095414a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095406c550 con 0x7f09540b0070 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095406c270 con 0x7f095406ca70 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095406da80 con 0x7f09540c5380 2024-03-06T07:18:50.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33780/0 (socket says 172.21.15.94:33780) 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 -- 172.21.15.94:0/622208742 learned_addr learned my addr 172.21.15.94:0/622208742 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959973640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f095406ca70 0x7f09540c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095a174640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09540c5380 0x7f095414a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 -- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09540c5380 msgr2=0x7f095414a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09540c5380 0x7f095414a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 -- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f095406ca70 msgr2=0x7f09540c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f095406ca70 0x7f09540c23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 -- 172.21.15.94:0/622208742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0954154c60 con 0x7f09540b0070 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f0959172640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954154100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f094400c410 tx=0x7f094400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f094a7fc640 1 -- 172.21.15.94:0/622208742 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944017070 con 0x7f09540b0070 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f094a7fc640 1 -- 172.21.15.94:0/622208742 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09440035f0 con 0x7f09540b0070 2024-03-06T07:18:50.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09541eedf0 con 0x7f09540b0070 2024-03-06T07:18:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f094a7fc640 1 -- 172.21.15.94:0/622208742 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09440039e0 con 0x7f09540b0070 2024-03-06T07:18:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.379+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09541ee310 con 0x7f09540b0070 2024-03-06T07:18:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.380+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095404f330 con 0x7f09540b0070 2024-03-06T07:18:50.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.381+0000 7f094a7fc640 1 -- 172.21.15.94:0/622208742 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f094401a070 con 0x7f09540b0070 2024-03-06T07:18:50.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.381+0000 7f094a7fc640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.381+0000 7f094a7fc640 1 -- 172.21.15.94:0/622208742 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0944089870 con 0x7f09540b0070 2024-03-06T07:18:50.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.381+0000 7f0959973640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.382+0000 7f0959973640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f093806b4d0 0x7f093806d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0950036000 tx=0x7f0950008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.385+0000 7f094a7fc640 1 -- 172.21.15.94:0/622208742 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09540b0450 con 0x7f09540b0070 2024-03-06T07:18:50.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.554+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f095406d860 con 0x7f09540b0070 2024-03-06T07:18:50.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.555+0000 7f094a7fc640 1 -- 172.21.15.94:0/622208742 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0944057df0 con 0x7f09540b0070 2024-03-06T07:18:50.557 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f093806b4d0 msgr2=0x7f093806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f093806b4d0 0x7f093806d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0950036000 tx=0x7f0950008030 comp rx=0 tx=0).stop 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 msgr2=0x7f0954154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954154100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f094400c410 tx=0x7f094400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 shutdown_connections 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f093806b4d0 0x7f093806d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09540c5380 0x7f095414a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09540b0070 0x7f0954154100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 --2- 172.21.15.94:0/622208742 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f095406ca70 0x7f09540c23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 >> 172.21.15.94:0/622208742 conn(0x7f09540ac590 msgr2=0x7f09540ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 shutdown_connections 2024-03-06T07:18:50.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.557+0000 7f095bbd7640 1 -- 172.21.15.94:0/622208742 wait complete. 2024-03-06T07:18:50.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:50.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.671+0000 7f48b4b13640 1 Processor -- start 2024-03-06T07:18:50.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.671+0000 7f48b4b13640 1 -- start start 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.672+0000 7f48b4b13640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f48b00ab170 0x7f48b00c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.672+0000 7f48b4b13640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b00c3ef0 con 0x7f48b00c5990 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.672+0000 7f48b4b13640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b01556d0 con 0x7f48b006c900 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.672+0000 7f48b4b13640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b01559b0 con 0x7f48b00ab170 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.672+0000 7f48ae59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f48b00c5990 0x7f48b006c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50652/0 (socket says 172.21.15.94:50652) 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.672+0000 7f48ae59c640 1 -- 172.21.15.94:0/2008721262 learned_addr learned my addr 172.21.15.94:0/2008721262 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.672+0000 7f48aed9d640 1 --2- 172.21.15.94:0/2008721262 >> v2:172.21.15.145:3300/0 conn(0x7f48b00ab170 0x7f48b00c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48add9b640 1 -- 172.21.15.94:0/2008721262 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209994327 0 0) 0x7f48b00c3ef0 con 0x7f48b00c5990 2024-03-06T07:18:50.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48add9b640 1 -- 172.21.15.94:0/2008721262 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f489c003520 con 0x7f48b00c5990 2024-03-06T07:18:50.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48add9b640 1 -- 172.21.15.94:0/2008721262 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942027013 0 0) 0x7f48b01556d0 con 0x7f48b006c900 2024-03-06T07:18:50.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48add9b640 1 -- 172.21.15.94:0/2008721262 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b00c3ef0 con 0x7f48b006c900 2024-03-06T07:18:50.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48aed9d640 1 -- 172.21.15.94:0/2008721262 >> v1:172.21.15.145:6789/0 conn(0x7f48b006c900 legacy=0x7f48b00a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48aed9d640 1 -- 172.21.15.94:0/2008721262 >> v1:172.21.15.94:6789/0 conn(0x7f48b00c5990 legacy=0x7f48b006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48aed9d640 1 -- 172.21.15.94:0/2008721262 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0155c90 con 0x7f48b00ab170 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.673+0000 7f48aed9d640 1 --2- 172.21.15.94:0/2008721262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 0x7f48b00c0ab0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009dc0 tx=0x7f48a802bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea9285a148bfc649 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48add9b640 1 -- 172.21.15.94:0/2008721262 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8039070 con 0x7f48b00ab170 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48add9b640 1 -- 172.21.15.94:0/2008721262 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a802ccd0 con 0x7f48b00ab170 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48add9b640 1 -- 172.21.15.94:0/2008721262 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a8034640 con 0x7f48b00ab170 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48b4b13640 1 -- 172.21.15.94:0/2008721262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 msgr2=0x7f48b00c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2008721262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 0x7f48b00c0ab0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009dc0 tx=0x7f48a802bf20 comp rx=0 tx=0).stop 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48b4b13640 1 -- 172.21.15.94:0/2008721262 shutdown_connections 2024-03-06T07:18:50.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2008721262 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 0x7f48b00c0ab0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48b4b13640 1 -- 172.21.15.94:0/2008721262 >> 172.21.15.94:0/2008721262 conn(0x7f48b00c6710 msgr2=0x7f48b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48b4b13640 1 -- 172.21.15.94:0/2008721262 shutdown_connections 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.674+0000 7f48b4b13640 1 -- 172.21.15.94:0/2008721262 wait complete. 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.675+0000 7f48b4b13640 1 Processor -- start 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.675+0000 7f48b4b13640 1 -- start start 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48b4b13640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b006c900 0x7f48b01e9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48b4b13640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 0x7f48b0154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48b4b13640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48b00c5990 0x7f48b01479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48b4b13640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b0156610 con 0x7f48b006c900 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48b4b13640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b0156490 con 0x7f48b00ab170 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48b4b13640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00c3af0 con 0x7f48b00c5990 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b006c900 0x7f48b01e9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b006c900 0x7f48b01e9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33792/0 (socket says 172.21.15.94:33792) 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 -- 172.21.15.94:0/2376636891 learned_addr learned my addr 172.21.15.94:0/2376636891 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48aed9d640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48b00c5990 0x7f48b01479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48a7fff640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 0x7f48b0154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 -- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48b00c5990 msgr2=0x7f48b01479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48b00c5990 0x7f48b01479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 -- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 msgr2=0x7f48b0154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 0x7f48b0154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.676+0000 7f48ae59c640 1 -- 172.21.15.94:0/2376636891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b014ac70 con 0x7f48b006c900 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.677+0000 7f48ae59c640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b006c900 0x7f48b01e9e20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c410 tx=0x7f48a000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.677+0000 7f48a6ffd640 1 -- 172.21.15.94:0/2376636891 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0017070 con 0x7f48b006c900 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.677+0000 7f48a6ffd640 1 -- 172.21.15.94:0/2376636891 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a0005ce0 con 0x7f48b006c900 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.677+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b014eaa0 con 0x7f48b006c900 2024-03-06T07:18:50.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.677+0000 7f48a6ffd640 1 -- 172.21.15.94:0/2376636891 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a00072d0 con 0x7f48b006c900 2024-03-06T07:18:50.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.677+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b01eaa40 con 0x7f48b006c900 2024-03-06T07:18:50.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.678+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b004f330 con 0x7f48b006c900 2024-03-06T07:18:50.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.678+0000 7f48a6ffd640 1 -- 172.21.15.94:0/2376636891 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48a001a070 con 0x7f48b006c900 2024-03-06T07:18:50.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.679+0000 7f48a6ffd640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f489006b4d0 0x7f489006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.679+0000 7f48a6ffd640 1 -- 172.21.15.94:0/2376636891 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f48a00897c0 con 0x7f48b006c900 2024-03-06T07:18:50.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.679+0000 7f48a7fff640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f489006b4d0 0x7f489006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.679+0000 7f48a7fff640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f489006b4d0 0x7f489006d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4898007fd0 tx=0x7f4898009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.683+0000 7f48a6ffd640 1 -- 172.21.15.94:0/2376636891 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48a0057d40 con 0x7f48b006c900 2024-03-06T07:18:50.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.855+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48b00c14a0 con 0x7f48b006c900 2024-03-06T07:18:50.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.855+0000 7f48a6ffd640 1 -- 172.21.15.94:0/2376636891 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48a005bde0 con 0x7f48b006c900 2024-03-06T07:18:50.858 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f489006b4d0 msgr2=0x7f489006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f489006b4d0 0x7f489006d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4898007fd0 tx=0x7f4898009040 comp rx=0 tx=0).stop 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b006c900 msgr2=0x7f48b01e9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b006c900 0x7f48b01e9e20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c410 tx=0x7f48a000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 shutdown_connections 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f489006b4d0 0x7f489006d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48b00c5990 0x7f48b01479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48b00ab170 0x7f48b0154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 --2- 172.21.15.94:0/2376636891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b006c900 0x7f48b01e9e20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 >> 172.21.15.94:0/2376636891 conn(0x7f48b00c6710 msgr2=0x7f48b0149d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 shutdown_connections 2024-03-06T07:18:50.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.858+0000 7f48b4b13640 1 -- 172.21.15.94:0/2376636891 wait complete. 2024-03-06T07:18:50.870 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:50.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.971+0000 7f7aaf7bc640 1 Processor -- start 2024-03-06T07:18:50.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aaf7bc640 1 -- start start 2024-03-06T07:18:50.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aaf7bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7aa806dca0 0x7f7aa8153420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aaf7bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7aa8155520 0x7f7aa8155950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aaf7bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa80c2d80 con 0x7f7aa814f580 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aaf7bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa80c2b30 con 0x7f7aa8155520 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aaf7bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa80c3d70 con 0x7f7aa806dca0 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aadd59640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7aa8155520 0x7f7aa8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aadd59640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7aa8155520 0x7f7aa8155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33796/0 (socket says 172.21.15.94:33796) 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aadd59640 1 -- 172.21.15.94:0/2235683018 learned_addr learned my addr 172.21.15.94:0/2235683018 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.972+0000 7f7aacd57640 1 --2- 172.21.15.94:0/2235683018 >> v2:172.21.15.145:3300/0 conn(0x7f7aa806dca0 0x7f7aa8153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.973+0000 7f7aadd59640 1 -- 172.21.15.94:0/2235683018 >> v2:172.21.15.145:3300/0 conn(0x7f7aa806dca0 msgr2=0x7f7aa8153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.973+0000 7f7aadd59640 1 --2- 172.21.15.94:0/2235683018 >> v2:172.21.15.145:3300/0 conn(0x7f7aa806dca0 0x7f7aa8153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.973+0000 7f7aadd59640 1 -- 172.21.15.94:0/2235683018 >> v1:172.21.15.145:6789/0 conn(0x7f7aa814f580 legacy=0x7f7aa806f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.973+0000 7f7aadd59640 1 -- 172.21.15.94:0/2235683018 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80c3a00 con 0x7f7aa8155520 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.973+0000 7f7aadd59640 1 --2- 172.21.15.94:0/2235683018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa8155520 0x7f7aa8155950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7aa806ed20 tx=0x7f7aa002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4538268dbe23383 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.973+0000 7f7a9ffff640 1 -- 172.21.15.94:0/2235683018 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa002c9f0 con 0x7f7aa8155520 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7a9ffff640 1 -- 172.21.15.94:0/2235683018 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa002cb90 con 0x7f7aa8155520 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7a9ffff640 1 -- 172.21.15.94:0/2235683018 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00343f0 con 0x7f7aa8155520 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/2235683018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa8155520 msgr2=0x7f7aa8155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/2235683018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa8155520 0x7f7aa8155950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7aa806ed20 tx=0x7f7aa002be50 comp rx=0 tx=0).stop 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/2235683018 shutdown_connections 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/2235683018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa8155520 0x7f7aa8155950 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/2235683018 >> v2:172.21.15.145:3300/0 conn(0x7f7aa806dca0 0x7f7aa8153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/2235683018 >> 172.21.15.94:0/2235683018 conn(0x7f7aa80a94d0 msgr2=0x7f7aa80a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:50.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/2235683018 shutdown_connections 2024-03-06T07:18:50.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.974+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/2235683018 wait complete. 2024-03-06T07:18:50.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.975+0000 7f7aaf7bc640 1 Processor -- start 2024-03-06T07:18:50.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.975+0000 7f7aaf7bc640 1 -- start start 2024-03-06T07:18:50.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aaf7bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aa806dca0 0x7f7aa806b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aaf7bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa814f580 0x7f7aa80a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aaf7bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aa8155520 0x7f7aa80a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aaf7bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80c3060 con 0x7f7aa814f580 2024-03-06T07:18:50.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aaf7bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa814f960 con 0x7f7aa806dca0 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aaf7bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8155e90 con 0x7f7aa8155520 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa814f580 0x7f7aa80a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa814f580 0x7f7aa80a5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33806/0 (socket says 172.21.15.94:33806) 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 -- 172.21.15.94:0/3930315396 learned_addr learned my addr 172.21.15.94:0/3930315396 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aad558640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aa806dca0 0x7f7aa806b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aadd59640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aa8155520 0x7f7aa80a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 -- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aa8155520 msgr2=0x7f7aa80a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aa8155520 0x7f7aa80a6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 -- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aa806dca0 msgr2=0x7f7aa806b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aa806dca0 0x7f7aa806b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 -- 172.21.15.94:0/3930315396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa81520b0 con 0x7f7aa814f580 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aad558640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aa806dca0 0x7f7aa806b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:50.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aacd57640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa814f580 0x7f7aa80a5f10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800e9e0 tx=0x7f7a9800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7a9dffb640 1 -- 172.21.15.94:0/3930315396 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98017070 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7a9dffb640 1 -- 172.21.15.94:0/3930315396 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9800c970 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa8152910 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.976+0000 7f7a9dffb640 1 -- 172.21.15.94:0/3930315396 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9800cd60 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.977+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa8152350 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.978+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa804f330 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.978+0000 7f7a9dffb640 1 -- 172.21.15.94:0/3930315396 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a9801a070 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.978+0000 7f7a9dffb640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a8c06b4d0 0x7f7a8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.979+0000 7f7a9dffb640 1 -- 172.21.15.94:0/3930315396 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7a9808a4a0 con 0x7f7aa814f580 2024-03-06T07:18:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.979+0000 7f7aad558640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a8c06b4d0 0x7f7a8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:50.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.979+0000 7f7aad558640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a8c06b4d0 0x7f7a8c06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4001700 tx=0x7f7aa4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:50.983+0000 7f7a9dffb640 1 -- 172.21.15.94:0/3930315396 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a98059070 con 0x7f7aa814f580 2024-03-06T07:18:51.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.153+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7aa81526f0 con 0x7f7aa814f580 2024-03-06T07:18:51.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.153+0000 7f7a9dffb640 1 -- 172.21.15.94:0/3930315396 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7a98058970 con 0x7f7aa814f580 2024-03-06T07:18:51.156 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:51.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a8c06b4d0 msgr2=0x7f7a8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a8c06b4d0 0x7f7a8c06d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4001700 tx=0x7f7aa4009040 comp rx=0 tx=0).stop 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa814f580 msgr2=0x7f7aa80a5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa814f580 0x7f7aa80a5f10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7a9800e9e0 tx=0x7f7a9800eea0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 shutdown_connections 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7a8c06b4d0 0x7f7a8c06d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aa8155520 0x7f7aa80a6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7aa814f580 0x7f7aa80a5f10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 --2- 172.21.15.94:0/3930315396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aa806dca0 0x7f7aa806b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.155+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 >> 172.21.15.94:0/3930315396 conn(0x7f7aa80a94d0 msgr2=0x7f7aa80a7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.156+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 shutdown_connections 2024-03-06T07:18:51.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.156+0000 7f7aaf7bc640 1 -- 172.21.15.94:0/3930315396 wait complete. 2024-03-06T07:18:51.180 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2024-03-06T07:18:51.180 INFO:tasks.cephfs_test_runner:test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:18:51.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:51.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.278+0000 7f18d764e640 1 Processor -- start 2024-03-06T07:18:51.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.278+0000 7f18d764e640 1 -- start start 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.279+0000 7f18d764e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18d01527b0 0x7f18d0159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.279+0000 7f18d764e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d00c7c60 con 0x7f18d00c88b0 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.279+0000 7f18d764e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d00c8060 con 0x7f18d00c7180 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.279+0000 7f18d764e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d0159640 con 0x7f18d01527b0 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.279+0000 7f18d5beb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f18d00c88b0 0x7f18d00c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50658/0 (socket says 172.21.15.94:50658) 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.279+0000 7f18d5beb640 1 -- 172.21.15.94:0/1754233219 learned_addr learned my addr 172.21.15.94:0/1754233219 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.279+0000 7f18d53ea640 1 --2- 172.21.15.94:0/1754233219 >> v2:172.21.15.145:3300/0 conn(0x7f18d01527b0 0x7f18d0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.280+0000 7f18c7fff640 1 -- 172.21.15.94:0/1754233219 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189259070 0 0) 0x7f18c8001110 con 0x7f18d00c88b0 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.280+0000 7f18c7fff640 1 -- 172.21.15.94:0/1754233219 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18bc003520 con 0x7f18d00c88b0 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.280+0000 7f18d53ea640 1 -- 172.21.15.94:0/1754233219 >> v1:172.21.15.145:6789/0 conn(0x7f18d00c7180 legacy=0x7f18d0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.280+0000 7f18c7fff640 1 -- 172.21.15.94:0/1754233219 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733033001 0 0) 0x7f18d00c8060 con 0x7f18d00c7180 2024-03-06T07:18:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.280+0000 7f18d53ea640 1 -- 172.21.15.94:0/1754233219 >> v1:172.21.15.94:6789/0 conn(0x7f18d00c88b0 legacy=0x7f18d00c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.280+0000 7f18d53ea640 1 -- 172.21.15.94:0/1754233219 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d00c76a0 con 0x7f18d01527b0 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d53ea640 1 --2- 172.21.15.94:0/1754233219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 0x7f18d0159100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d90 tx=0x7f18cc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1cef22d2bc2c1c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18c7fff640 1 -- 172.21.15.94:0/1754233219 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc039070 con 0x7f18d01527b0 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18c7fff640 1 -- 172.21.15.94:0/1754233219 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc02cc60 con 0x7f18d01527b0 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18c7fff640 1 -- 172.21.15.94:0/1754233219 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc034660 con 0x7f18d01527b0 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d764e640 1 -- 172.21.15.94:0/1754233219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 msgr2=0x7f18d0159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d764e640 1 --2- 172.21.15.94:0/1754233219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 0x7f18d0159100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d90 tx=0x7f18cc02beb0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d764e640 1 -- 172.21.15.94:0/1754233219 shutdown_connections 2024-03-06T07:18:51.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d764e640 1 --2- 172.21.15.94:0/1754233219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 0x7f18d0159100 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d764e640 1 -- 172.21.15.94:0/1754233219 >> 172.21.15.94:0/1754233219 conn(0x7f18d00ac560 msgr2=0x7f18d00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d764e640 1 -- 172.21.15.94:0/1754233219 shutdown_connections 2024-03-06T07:18:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.281+0000 7f18d764e640 1 -- 172.21.15.94:0/1754233219 wait complete. 2024-03-06T07:18:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.282+0000 7f18d764e640 1 Processor -- start 2024-03-06T07:18:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.282+0000 7f18d764e640 1 -- start start 2024-03-06T07:18:51.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d764e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d00c7180 0x7f18d00c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d764e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18d00c88b0 0x7f18d014ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d764e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 0x7f18d014bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d764e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d0155600 con 0x7f18d00c7180 2024-03-06T07:18:51.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d764e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d0155320 con 0x7f18d01527b0 2024-03-06T07:18:51.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d764e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d0152b90 con 0x7f18d00c88b0 2024-03-06T07:18:51.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d53ea640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d00c7180 0x7f18d00c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d53ea640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d00c7180 0x7f18d00c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33812/0 (socket says 172.21.15.94:33812) 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d53ea640 1 -- 172.21.15.94:0/817996544 learned_addr learned my addr 172.21.15.94:0/817996544 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d5beb640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 0x7f18d014bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d4be9640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18d00c88b0 0x7f18d014ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d53ea640 1 -- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18d00c88b0 msgr2=0x7f18d014ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d53ea640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18d00c88b0 0x7f18d014ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d53ea640 1 -- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 msgr2=0x7f18d014bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.283+0000 7f18d53ea640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 0x7f18d014bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.284+0000 7f18d53ea640 1 -- 172.21.15.94:0/817996544 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d01564f0 con 0x7f18d00c7180 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.284+0000 7f18d53ea640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d00c7180 0x7f18d00c9fe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d90 tx=0x7f18cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.284+0000 7f18c5ffb640 1 -- 172.21.15.94:0/817996544 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc039050 con 0x7f18d00c7180 2024-03-06T07:18:51.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.284+0000 7f18c5ffb640 1 -- 172.21.15.94:0/817996544 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc03a040 con 0x7f18d00c7180 2024-03-06T07:18:51.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.284+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d01eb7d0 con 0x7f18d00c7180 2024-03-06T07:18:51.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.284+0000 7f18c5ffb640 1 -- 172.21.15.94:0/817996544 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc003bd0 con 0x7f18d00c7180 2024-03-06T07:18:51.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.284+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d0157be0 con 0x7f18d00c7180 2024-03-06T07:18:51.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.285+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d01566c0 con 0x7f18d00c7180 2024-03-06T07:18:51.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.285+0000 7f18c5ffb640 1 -- 172.21.15.94:0/817996544 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18cc004030 con 0x7f18d00c7180 2024-03-06T07:18:51.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.286+0000 7f18c5ffb640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18b406b500 0x7f18b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.286+0000 7f18c5ffb640 1 -- 172.21.15.94:0/817996544 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f18cc0acd50 con 0x7f18d00c7180 2024-03-06T07:18:51.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.286+0000 7f18d4be9640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18b406b500 0x7f18b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.286+0000 7f18d4be9640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18b406b500 0x7f18b406d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f18d006b860 tx=0x7f18c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.290+0000 7f18c5ffb640 1 -- 172.21.15.94:0/817996544 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18cc0b0030 con 0x7f18d00c7180 2024-03-06T07:18:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.462+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18d004f330 con 0x7f18d00c7180 2024-03-06T07:18:51.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.463+0000 7f18c5ffb640 1 -- 172.21.15.94:0/817996544 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f18cc07b2d0 con 0x7f18d00c7180 2024-03-06T07:18:51.523 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:51.523 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:51.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18b406b500 msgr2=0x7f18b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18b406b500 0x7f18b406d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f18d006b860 tx=0x7f18c0009040 comp rx=0 tx=0).stop 2024-03-06T07:18:51.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d00c7180 msgr2=0x7f18d00c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d00c7180 0x7f18d00c9fe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d90 tx=0x7f18cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:51.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 shutdown_connections 2024-03-06T07:18:51.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18b406b500 0x7f18b406d9c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18d01527b0 0x7f18d014bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18d00c88b0 0x7f18d014ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 --2- 172.21.15.94:0/817996544 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d00c7180 0x7f18d00c9fe0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 >> 172.21.15.94:0/817996544 conn(0x7f18d00ac560 msgr2=0x7f18d014d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:51.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 shutdown_connections 2024-03-06T07:18:51.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.465+0000 7f18d764e640 1 -- 172.21.15.94:0/817996544 wait complete. 2024-03-06T07:18:51.526 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:51.526 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:51.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.628+0000 7f29b76f4640 1 Processor -- start 2024-03-06T07:18:51.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.628+0000 7f29b76f4640 1 -- start start 2024-03-06T07:18:51.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b76f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29b01529d0 0x7f29b00c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b76f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29b00c57e0 0x7f29b0155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b76f4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b0152db0 con 0x7f29b0072cd0 2024-03-06T07:18:51.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b76f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b00c6dd0 con 0x7f29b00c57e0 2024-03-06T07:18:51.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b76f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b00c6940 con 0x7f29b01529d0 2024-03-06T07:18:51.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b4c8f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29b00c57e0 0x7f29b0155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b5c91640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f29b0072cd0 0x7f29b0155660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50670/0 (socket says 172.21.15.94:50670) 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b4c8f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29b00c57e0 0x7f29b0155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33826/0 (socket says 172.21.15.94:33826) 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b4c8f640 1 -- 172.21.15.94:0/2564668279 learned_addr learned my addr 172.21.15.94:0/2564668279 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.629+0000 7f29b5490640 1 --2- 172.21.15.94:0/2564668279 >> v2:172.21.15.145:3300/0 conn(0x7f29b01529d0 0x7f29b00c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29b4c8f640 1 -- 172.21.15.94:0/2564668279 >> v2:172.21.15.145:3300/0 conn(0x7f29b01529d0 msgr2=0x7f29b00c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29b4c8f640 1 --2- 172.21.15.94:0/2564668279 >> v2:172.21.15.145:3300/0 conn(0x7f29b01529d0 0x7f29b00c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29b4c8f640 1 -- 172.21.15.94:0/2564668279 >> v1:172.21.15.94:6789/0 conn(0x7f29b0072cd0 legacy=0x7f29b0155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29a7fff640 1 -- 172.21.15.94:0/2564668279 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942968358 0 0) 0x7f29a8001490 con 0x7f29b0072cd0 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29b4c8f640 1 -- 172.21.15.94:0/2564668279 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b01590f0 con 0x7f29b00c57e0 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29b4c8f640 1 --2- 172.21.15.94:0/2564668279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b00c57e0 0x7f29b0155120 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f29b00c98d0 tx=0x7f29a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce732bc91cb5e75b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29a7fff640 1 -- 172.21.15.94:0/2564668279 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a002e070 con 0x7f29b00c57e0 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29a7fff640 1 -- 172.21.15.94:0/2564668279 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0037070 con 0x7f29b00c57e0 2024-03-06T07:18:51.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.630+0000 7f29a7fff640 1 -- 172.21.15.94:0/2564668279 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0031a00 con 0x7f29b00c57e0 2024-03-06T07:18:51.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 -- 172.21.15.94:0/2564668279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b00c57e0 msgr2=0x7f29b0155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2564668279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b00c57e0 0x7f29b0155120 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f29b00c98d0 tx=0x7f29a002b880 comp rx=0 tx=0).stop 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 -- 172.21.15.94:0/2564668279 shutdown_connections 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2564668279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b00c57e0 0x7f29b0155120 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2564668279 >> v2:172.21.15.145:3300/0 conn(0x7f29b01529d0 0x7f29b00c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 -- 172.21.15.94:0/2564668279 >> 172.21.15.94:0/2564668279 conn(0x7f29b00aa630 msgr2=0x7f29b00aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 -- 172.21.15.94:0/2564668279 shutdown_connections 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.631+0000 7f29b76f4640 1 -- 172.21.15.94:0/2564668279 wait complete. 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.632+0000 7f29b76f4640 1 Processor -- start 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.632+0000 7f29b76f4640 1 -- start start 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b76f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0072cd0 0x7f29b01e1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b76f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29b00c57e0 0x7f29b01def30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b76f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29b01529d0 0x7f29b014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b76f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0072720 con 0x7f29b0072cd0 2024-03-06T07:18:51.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b76f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0155d70 con 0x7f29b00c57e0 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b76f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0159370 con 0x7f29b01529d0 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0072cd0 0x7f29b01e1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0072cd0 0x7f29b01e1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33830/0 (socket says 172.21.15.94:33830) 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 -- 172.21.15.94:0/2646416669 learned_addr learned my addr 172.21.15.94:0/2646416669 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b4c8f640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29b00c57e0 0x7f29b01def30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5c91640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29b01529d0 0x7f29b014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 -- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29b01529d0 msgr2=0x7f29b014b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29b01529d0 0x7f29b014b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 -- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29b00c57e0 msgr2=0x7f29b01def30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29b00c57e0 0x7f29b01def30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 -- 172.21.15.94:0/2646416669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b006dbf0 con 0x7f29b0072cd0 2024-03-06T07:18:51.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b4c8f640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29b00c57e0 0x7f29b01def30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29b5490640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0072cd0 0x7f29b01e1100 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00e8a0 tx=0x7f29ac00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29a5ffb640 1 -- 172.21.15.94:0/2646416669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac017070 con 0x7f29b0072cd0 2024-03-06T07:18:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.633+0000 7f29a5ffb640 1 -- 172.21.15.94:0/2646416669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ac00c940 con 0x7f29b0072cd0 2024-03-06T07:18:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.634+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b006ded0 con 0x7f29b0072cd0 2024-03-06T07:18:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.634+0000 7f29a5ffb640 1 -- 172.21.15.94:0/2646416669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac00cd30 con 0x7f29b0072cd0 2024-03-06T07:18:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.634+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b014be20 con 0x7f29b0072cd0 2024-03-06T07:18:51.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.635+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b0067910 con 0x7f29b0072cd0 2024-03-06T07:18:51.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.635+0000 7f29a5ffb640 1 -- 172.21.15.94:0/2646416669 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29ac01a070 con 0x7f29b0072cd0 2024-03-06T07:18:51.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.635+0000 7f29a5ffb640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.635+0000 7f29a5ffb640 1 -- 172.21.15.94:0/2646416669 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f29ac08a770 con 0x7f29b0072cd0 2024-03-06T07:18:51.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.636+0000 7f29b4c8f640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.636+0000 7f29b4c8f640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f299406b4d0 0x7f299406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f29a0036000 tx=0x7f29a003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.640+0000 7f29a5ffb640 1 -- 172.21.15.94:0/2646416669 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29b0067910 con 0x7f29b0072cd0 2024-03-06T07:18:51.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.814+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29b00a8b70 con 0x7f29b0072cd0 2024-03-06T07:18:51.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.814+0000 7f29a5ffb640 1 -- 172.21.15.94:0/2646416669 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29b00a8b70 con 0x7f29b0072cd0 2024-03-06T07:18:51.816 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:51.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f299406b4d0 msgr2=0x7f299406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f299406b4d0 0x7f299406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f29a0036000 tx=0x7f29a003d040 comp rx=0 tx=0).stop 2024-03-06T07:18:51.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0072cd0 msgr2=0x7f29b01e1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0072cd0 0x7f29b01e1100 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00e8a0 tx=0x7f29ac00ed60 comp rx=0 tx=0).stop 2024-03-06T07:18:51.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 shutdown_connections 2024-03-06T07:18:51.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f299406b4d0 0x7f299406d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29b01529d0 0x7f29b014b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29b00c57e0 0x7f29b01def30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 --2- 172.21.15.94:0/2646416669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29b0072cd0 0x7f29b01e1100 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 >> 172.21.15.94:0/2646416669 conn(0x7f29b00aa630 msgr2=0x7f29b0152f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:51.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 shutdown_connections 2024-03-06T07:18:51.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.816+0000 7f29b76f4640 1 -- 172.21.15.94:0/2646416669 wait complete. 2024-03-06T07:18:51.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:51.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.930+0000 7f9eebb7a640 1 Processor -- start 2024-03-06T07:18:51.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eebb7a640 1 -- start start 2024-03-06T07:18:51.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eebb7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eebb7a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9ee40c1820 0x7f9ee406cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eebb7a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9ee40c2980 0x7f9ee4154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eebb7a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee40c1ca0 con 0x7f9ee40c2980 2024-03-06T07:18:51.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eebb7a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee414f350 con 0x7f9ee40c1820 2024-03-06T07:18:51.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eebb7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40c2790 con 0x7f9ee414ef70 2024-03-06T07:18:51.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eea117640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9ee40c2980 0x7f9ee4154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eea117640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9ee40c2980 0x7f9ee4154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33832/0 (socket says 172.21.15.94:33832) 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9eea117640 1 -- 172.21.15.94:0/2589467368 learned_addr learned my addr 172.21.15.94:0/2589467368 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9ee9916640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.931+0000 7f9ee9115640 1 --2- 172.21.15.94:0/2589467368 >> v2:172.21.15.145:3300/0 conn(0x7f9ee40c1820 0x7f9ee406cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.932+0000 7f9eea117640 1 -- 172.21.15.94:0/2589467368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 msgr2=0x7f9ee406c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.932+0000 7f9eea117640 1 --2- 172.21.15.94:0/2589467368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee406c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.932+0000 7f9eea117640 1 -- 172.21.15.94:0/2589467368 >> v2:172.21.15.145:3300/0 conn(0x7f9ee40c1820 msgr2=0x7f9ee406cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.932+0000 7f9eea117640 1 --2- 172.21.15.94:0/2589467368 >> v2:172.21.15.145:3300/0 conn(0x7f9ee40c1820 0x7f9ee406cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.932+0000 7f9eea117640 1 -- 172.21.15.94:0/2589467368 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee40c1540 con 0x7f9ee40c2980 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.932+0000 7f9eea117640 1 --2- 172.21.15.94:0/2589467368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c2980 0x7f9ee4154c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40c75e0 tx=0x7f9edc02e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77668749fb0988d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9ee8914640 1 -- 172.21.15.94:0/2589467368 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc02ea00 con 0x7f9ee40c2980 2024-03-06T07:18:51.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9ee8914640 1 -- 172.21.15.94:0/2589467368 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc02eba0 con 0x7f9ee40c2980 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9ee8914640 1 -- 172.21.15.94:0/2589467368 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc009600 con 0x7f9ee40c2980 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2589467368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c2980 msgr2=0x7f9ee4154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2589467368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c2980 0x7f9ee4154c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40c75e0 tx=0x7f9edc02e060 comp rx=0 tx=0).stop 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2589467368 shutdown_connections 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2589467368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c2980 0x7f9ee4154c30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2589467368 >> v2:172.21.15.145:3300/0 conn(0x7f9ee40c1820 0x7f9ee406cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2589467368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee406c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2589467368 >> 172.21.15.94:0/2589467368 conn(0x7f9ee40c6710 msgr2=0x7f9ee40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2589467368 shutdown_connections 2024-03-06T07:18:51.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.933+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2589467368 wait complete. 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.934+0000 7f9eebb7a640 1 Processor -- start 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.934+0000 7f9eebb7a640 1 -- start start 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9eebb7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c1820 0x7f9ee4151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9eebb7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ee40c2980 0x7f9ee40aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9eebb7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee4070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9eebb7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee4155170 con 0x7f9ee40c1820 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9eebb7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee4155750 con 0x7f9ee40c2980 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9eebb7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee41552f0 con 0x7f9ee414ef70 2024-03-06T07:18:51.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c1820 0x7f9ee4151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c1820 0x7f9ee4151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33842/0 (socket says 172.21.15.94:33842) 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 -- 172.21.15.94:0/2297146499 learned_addr learned my addr 172.21.15.94:0/2297146499 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9115640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ee40c2980 0x7f9ee40aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9eea117640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee4070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 -- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 msgr2=0x7f9ee4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee4070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 -- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ee40c2980 msgr2=0x7f9ee40aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ee40c2980 0x7f9ee40aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 -- 172.21.15.94:0/2297146499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee41e6550 con 0x7f9ee40c1820 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.935+0000 7f9ee9916640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c1820 0x7f9ee4151e50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800e870 tx=0x7f9ed800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.936+0000 7f9ed67fc640 1 -- 172.21.15.94:0/2297146499 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8017070 con 0x7f9ee40c1820 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.936+0000 7f9ed67fc640 1 -- 172.21.15.94:0/2297146499 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed800c940 con 0x7f9ee40c1820 2024-03-06T07:18:51.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.936+0000 7f9ed67fc640 1 -- 172.21.15.94:0/2297146499 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed800cd30 con 0x7f9ee40c1820 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.936+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee41e67f0 con 0x7f9ee40c1820 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.936+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee41e6db0 con 0x7f9ee40c1820 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.937+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee404f330 con 0x7f9ee40c1820 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.937+0000 7f9ed67fc640 1 -- 172.21.15.94:0/2297146499 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ed801a070 con 0x7f9ee40c1820 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.938+0000 7f9ed67fc640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec806b4d0 0x7f9ec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.938+0000 7f9ed67fc640 1 -- 172.21.15.94:0/2297146499 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9ed808a240 con 0x7f9ee40c1820 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.938+0000 7f9ee9115640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec806b4d0 0x7f9ec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:51.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.938+0000 7f9ee9115640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec806b4d0 0x7f9ec806d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40abab0 tx=0x7f9ee0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:51.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:51.942+0000 7f9ed67fc640 1 -- 172.21.15.94:0/2297146499 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ed80587c0 con 0x7f9ee40c1820 2024-03-06T07:18:52.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.114+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9ee40c1110 con 0x7f9ee40c1820 2024-03-06T07:18:52.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.114+0000 7f9ed67fc640 1 -- 172.21.15.94:0/2297146499 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9ed805c860 con 0x7f9ee40c1820 2024-03-06T07:18:52.116 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:52.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec806b4d0 msgr2=0x7f9ec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec806b4d0 0x7f9ec806d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40abab0 tx=0x7f9ee0009190 comp rx=0 tx=0).stop 2024-03-06T07:18:52.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c1820 msgr2=0x7f9ee4151e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c1820 0x7f9ee4151e50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800e870 tx=0x7f9ed800ed30 comp rx=0 tx=0).stop 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 shutdown_connections 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec806b4d0 0x7f9ec806d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ee414ef70 0x7f9ee4070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ee40c2980 0x7f9ee40aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 --2- 172.21.15.94:0/2297146499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ee40c1820 0x7f9ee4151e50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 >> 172.21.15.94:0/2297146499 conn(0x7f9ee40c6710 msgr2=0x7f9ee414e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 shutdown_connections 2024-03-06T07:18:52.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.116+0000 7f9eebb7a640 1 -- 172.21.15.94:0/2297146499 wait complete. 2024-03-06T07:18:52.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.229+0000 7fe94138a640 1 Processor -- start 2024-03-06T07:18:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.229+0000 7fe94138a640 1 -- start start 2024-03-06T07:18:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe94138a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0a8290 0x7fe93c1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe94138a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe93c0c69a0 0x7fe93c072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe94138a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c0c62a0 con 0x7fe93c0c4740 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe94138a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c153f00 con 0x7fe93c0c69a0 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe94138a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0c4c00 con 0x7fe93c0a8290 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe93a7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe93c0c4740 0x7fe93c0701a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50680/0 (socket says 172.21.15.94:50680) 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe93a7fc640 1 -- 172.21.15.94:0/4163477072 learned_addr learned my addr 172.21.15.94:0/4163477072 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe93b7fe640 1 --2- 172.21.15.94:0/4163477072 >> v2:172.21.15.145:3300/0 conn(0x7fe93c0c69a0 0x7fe93c072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.230+0000 7fe93affd640 1 --2- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0a8290 0x7fe93c1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe939ffb640 1 -- 172.21.15.94:0/4163477072 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376704080 0 0) 0x7fe93c0c62a0 con 0x7fe93c0c4740 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe939ffb640 1 -- 172.21.15.94:0/4163477072 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe924003520 con 0x7fe93c0c4740 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe93b7fe640 1 -- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0a8290 msgr2=0x7fe93c1539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe93b7fe640 1 --2- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0a8290 0x7fe93c1539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe93b7fe640 1 -- 172.21.15.94:0/4163477072 >> v1:172.21.15.94:6789/0 conn(0x7fe93c0c4740 legacy=0x7fe93c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe93b7fe640 1 -- 172.21.15.94:0/4163477072 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0c4ee0 con 0x7fe93c0c69a0 2024-03-06T07:18:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe93affd640 1 --2- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0a8290 0x7fe93c1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.231+0000 7fe93b7fe640 1 --2- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c69a0 0x7fe93c072680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe934009d90 tx=0x7fe93402bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea3cbe3acb210c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe939ffb640 1 -- 172.21.15.94:0/4163477072 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934039070 con 0x7fe93c0c69a0 2024-03-06T07:18:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe939ffb640 1 -- 172.21.15.94:0/4163477072 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93402cb40 con 0x7fe93c0c69a0 2024-03-06T07:18:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe939ffb640 1 -- 172.21.15.94:0/4163477072 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934034440 con 0x7fe93c0c69a0 2024-03-06T07:18:52.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 -- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c69a0 msgr2=0x7fe93c072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 --2- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c69a0 0x7fe93c072680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe934009d90 tx=0x7fe93402bd90 comp rx=0 tx=0).stop 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 -- 172.21.15.94:0/4163477072 shutdown_connections 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 --2- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c69a0 0x7fe93c072680 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 --2- 172.21.15.94:0/4163477072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0a8290 0x7fe93c1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 -- 172.21.15.94:0/4163477072 >> 172.21.15.94:0/4163477072 conn(0x7fe93c0c5560 msgr2=0x7fe93c0c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 -- 172.21.15.94:0/4163477072 shutdown_connections 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.232+0000 7fe94138a640 1 -- 172.21.15.94:0/4163477072 wait complete. 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.233+0000 7fe94138a640 1 Processor -- start 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.233+0000 7fe94138a640 1 -- start start 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe94138a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c0a8290 0x7fe93c156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe94138a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c4740 0x7fe93c1dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe94138a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0c69a0 0x7fe93c14cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe94138a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0c6e50 con 0x7fe93c0a8290 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe94138a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c06fc80 con 0x7fe93c0c4740 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe94138a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c072bc0 con 0x7fe93c0c69a0 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe93affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c0a8290 0x7fe93c156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe93affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c0a8290 0x7fe93c156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33850/0 (socket says 172.21.15.94:33850) 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe93affd640 1 -- 172.21.15.94:0/3422435011 learned_addr learned my addr 172.21.15.94:0/3422435011 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe93b7fe640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0c69a0 0x7fe93c14cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.234+0000 7fe93a7fc640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c4740 0x7fe93c1dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe93affd640 1 -- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0c69a0 msgr2=0x7fe93c14cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe93affd640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0c69a0 0x7fe93c14cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe93affd640 1 -- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c4740 msgr2=0x7fe93c1dfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe93affd640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c4740 0x7fe93c1dfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe93affd640 1 -- 172.21.15.94:0/3422435011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c06ddc0 con 0x7fe93c0a8290 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe93affd640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c0a8290 0x7fe93c156ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe93000ebe0 tx=0x7fe93000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe9237fe640 1 -- 172.21.15.94:0/3422435011 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930017070 con 0x7fe93c0a8290 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe9237fe640 1 -- 172.21.15.94:0/3422435011 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93000cc30 con 0x7fe93c0a8290 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c0c2940 con 0x7fe93c0a8290 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe9237fe640 1 -- 172.21.15.94:0/3422435011 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930013470 con 0x7fe93c0a8290 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.235+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c06d4c0 con 0x7fe93c0a8290 2024-03-06T07:18:52.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.236+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c0c7f30 con 0x7fe93c0a8290 2024-03-06T07:18:52.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.236+0000 7fe9237fe640 1 -- 172.21.15.94:0/3422435011 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe93001a070 con 0x7fe93c0a8290 2024-03-06T07:18:52.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.237+0000 7fe9237fe640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe92806b4d0 0x7fe92806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.237+0000 7fe9237fe640 1 -- 172.21.15.94:0/3422435011 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe93008a340 con 0x7fe93c0a8290 2024-03-06T07:18:52.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.237+0000 7fe93a7fc640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe92806b4d0 0x7fe92806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.237+0000 7fe93a7fc640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe92806b4d0 0x7fe92806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe92c005880 tx=0x7fe92c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.241+0000 7fe9237fe640 1 -- 172.21.15.94:0/3422435011 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe93c0c7f30 con 0x7fe93c0a8290 2024-03-06T07:18:52.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.411+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe93c04f330 con 0x7fe93c0a8290 2024-03-06T07:18:52.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.411+0000 7fe9237fe640 1 -- 172.21.15.94:0/3422435011 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe9300588c0 con 0x7fe93c0a8290 2024-03-06T07:18:52.413 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe92806b4d0 msgr2=0x7fe92806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe92806b4d0 0x7fe92806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe92c005880 tx=0x7fe92c009040 comp rx=0 tx=0).stop 2024-03-06T07:18:52.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c0a8290 msgr2=0x7fe93c156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c0a8290 0x7fe93c156ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe93000ebe0 tx=0x7fe93000c520 comp rx=0 tx=0).stop 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 shutdown_connections 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe92806b4d0 0x7fe92806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93c0c69a0 0x7fe93c14cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93c0c4740 0x7fe93c1dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 --2- 172.21.15.94:0/3422435011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe93c0a8290 0x7fe93c156ae0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 >> 172.21.15.94:0/3422435011 conn(0x7fe93c0c5560 msgr2=0x7fe93c0a9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 shutdown_connections 2024-03-06T07:18:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.413+0000 7fe94138a640 1 -- 172.21.15.94:0/3422435011 wait complete. 2024-03-06T07:18:52.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:52.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.527+0000 7f257f6a4640 1 Processor -- start 2024-03-06T07:18:52.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.528+0000 7f257f6a4640 1 -- start start 2024-03-06T07:18:52.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.528+0000 7f257f6a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25780c5990 0x7f25780c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.528+0000 7f257f6a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f25780a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.528+0000 7f257f6a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25780c3ef0 con 0x7f25780ab170 2024-03-06T07:18:52.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.528+0000 7f257f6a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25781556d0 con 0x7f25780c5990 2024-03-06T07:18:52.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.528+0000 7f257f6a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25781559b0 con 0x7f257806c900 2024-03-06T07:18:52.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.528+0000 7f257d440640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25780c5990 0x7f25780c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257d440640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25780c5990 0x7f25780c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33860/0 (socket says 172.21.15.94:33860) 2024-03-06T07:18:52.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257cc3f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f25780ab170 0x7f257806c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50686/0 (socket says 172.21.15.94:50686) 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257d440640 1 -- 172.21.15.94:0/1452749770 learned_addr learned my addr 172.21.15.94:0/1452749770 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257dc41640 1 --2- 172.21.15.94:0/1452749770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f25780a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257d440640 1 -- 172.21.15.94:0/1452749770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 msgr2=0x7f25780a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257d440640 1 --2- 172.21.15.94:0/1452749770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f25780a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257d440640 1 -- 172.21.15.94:0/1452749770 >> v1:172.21.15.94:6789/0 conn(0x7f25780ab170 legacy=0x7f257806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f257d440640 1 -- 172.21.15.94:0/1452749770 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2578155c90 con 0x7f25780c5990 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.529+0000 7f256ffff640 1 -- 172.21.15.94:0/1452749770 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437033725 0 0) 0x7f25780c3ef0 con 0x7f25780ab170 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257d440640 1 --2- 172.21.15.94:0/1452749770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780c5990 0x7f25780c0ab0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2574009870 tx=0x7f257402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e7ab5c7fa89be8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f256ffff640 1 -- 172.21.15.94:0/1452749770 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574034070 con 0x7f25780c5990 2024-03-06T07:18:52.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f256ffff640 1 -- 172.21.15.94:0/1452749770 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2574030840 con 0x7f25780c5990 2024-03-06T07:18:52.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f256ffff640 1 -- 172.21.15.94:0/1452749770 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574030c10 con 0x7f25780c5990 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 -- 172.21.15.94:0/1452749770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780c5990 msgr2=0x7f25780c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 --2- 172.21.15.94:0/1452749770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780c5990 0x7f25780c0ab0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2574009870 tx=0x7f257402b980 comp rx=0 tx=0).stop 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 -- 172.21.15.94:0/1452749770 shutdown_connections 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 --2- 172.21.15.94:0/1452749770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f25780a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 --2- 172.21.15.94:0/1452749770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780c5990 0x7f25780c0ab0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 -- 172.21.15.94:0/1452749770 >> 172.21.15.94:0/1452749770 conn(0x7f25780c6710 msgr2=0x7f25780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 -- 172.21.15.94:0/1452749770 shutdown_connections 2024-03-06T07:18:52.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.530+0000 7f257f6a4640 1 -- 172.21.15.94:0/1452749770 wait complete. 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.531+0000 7f257f6a4640 1 Processor -- start 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.531+0000 7f257f6a4640 1 -- start start 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257f6a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f257814d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257f6a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780ab170 0x7f257814e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257f6a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25780c5990 0x7f25781e8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257f6a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578156610 con 0x7f25780ab170 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257f6a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578156490 con 0x7f25780c5990 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257f6a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25780c3af0 con 0x7f257806c900 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780ab170 0x7f257814e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780ab170 0x7f257814e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33862/0 (socket says 172.21.15.94:33862) 2024-03-06T07:18:52.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 -- 172.21.15.94:0/3641124211 learned_addr learned my addr 172.21.15.94:0/3641124211 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257d440640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f257814d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257dc41640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25780c5990 0x7f25781e8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 -- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 msgr2=0x7f257814d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f257814d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 -- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25780c5990 msgr2=0x7f25781e8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25780c5990 0x7f25781e8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.532+0000 7f257cc3f640 1 -- 172.21.15.94:0/3641124211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257814eae0 con 0x7f25780ab170 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.533+0000 7f257cc3f640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780ab170 0x7f257814e490 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f256800c500 tx=0x7f256800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.533+0000 7f256dffb640 1 -- 172.21.15.94:0/3641124211 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568018070 con 0x7f25780ab170 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.533+0000 7f256dffb640 1 -- 172.21.15.94:0/3641124211 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2568007580 con 0x7f25780ab170 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.533+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257814cdd0 con 0x7f25780ab170 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.533+0000 7f256dffb640 1 -- 172.21.15.94:0/3641124211 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568004a20 con 0x7f25780ab170 2024-03-06T07:18:52.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.533+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257814aac0 con 0x7f25780ab170 2024-03-06T07:18:52.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.534+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257804f330 con 0x7f25780ab170 2024-03-06T07:18:52.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.534+0000 7f256dffb640 1 -- 172.21.15.94:0/3641124211 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f256801b070 con 0x7f25780ab170 2024-03-06T07:18:52.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.535+0000 7f256dffb640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f255c06b4d0 0x7f255c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.535+0000 7f256dffb640 1 -- 172.21.15.94:0/3641124211 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2568089730 con 0x7f25780ab170 2024-03-06T07:18:52.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.535+0000 7f257d440640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f255c06b4d0 0x7f255c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.535+0000 7f257d440640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f255c06b4d0 0x7f255c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2574036000 tx=0x7f2574008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.539+0000 7f256dffb640 1 -- 172.21.15.94:0/3641124211 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2568057cb0 con 0x7f25780ab170 2024-03-06T07:18:52.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.709+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f25780c0bb0 con 0x7f25780ab170 2024-03-06T07:18:52.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.709+0000 7f256dffb640 1 -- 172.21.15.94:0/3641124211 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f256805bd50 con 0x7f25780ab170 2024-03-06T07:18:52.712 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f255c06b4d0 msgr2=0x7f255c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f255c06b4d0 0x7f255c06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2574036000 tx=0x7f2574008030 comp rx=0 tx=0).stop 2024-03-06T07:18:52.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780ab170 msgr2=0x7f257814e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780ab170 0x7f257814e490 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f256800c500 tx=0x7f256800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 shutdown_connections 2024-03-06T07:18:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f255c06b4d0 0x7f255c06d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25780c5990 0x7f25781e8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25780ab170 0x7f257814e490 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 --2- 172.21.15.94:0/3641124211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f257806c900 0x7f257814d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 >> 172.21.15.94:0/3641124211 conn(0x7f25780c6710 msgr2=0x7f2578154db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 shutdown_connections 2024-03-06T07:18:52.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.711+0000 7f257f6a4640 1 -- 172.21.15.94:0/3641124211 wait complete. 2024-03-06T07:18:52.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-03-06T07:18:52.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.822+0000 7fa343ad4640 1 Processor -- start 2024-03-06T07:18:52.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.822+0000 7fa343ad4640 1 -- start start 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa343ad4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa33c0c5480 0x7fa33c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa343ad4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa343ad4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c0c5cc0 con 0x7fa33c0c5860 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa343ad4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33c06df40 con 0x7fa33c0c5480 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa343ad4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c06fdb0 con 0x7fa33c1586f0 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa341870640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa33c0c5480 0x7fa33c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa34106f640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa33c0c5860 0x7fa33c06f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:58490/0 (socket says 172.21.15.94:58490) 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa342071640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.823+0000 7fa34106f640 1 -- 172.21.15.94:0/3246583783 learned_addr learned my addr 172.21.15.94:0/3246583783 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.824+0000 7fa341870640 1 -- 172.21.15.94:0/3246583783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 msgr2=0x7fa33c158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:52.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.824+0000 7fa341870640 1 --2- 172.21.15.94:0/3246583783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.824+0000 7fa341870640 1 -- 172.21.15.94:0/3246583783 >> v1:172.21.15.145:6789/0 conn(0x7fa33c0c5860 legacy=0x7fa33c06f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.824+0000 7fa34086e640 1 -- 172.21.15.94:0/3246583783 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198158963 0 0) 0x7fa33c0c5cc0 con 0x7fa33c0c5860 2024-03-06T07:18:52.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.824+0000 7fa341870640 1 -- 172.21.15.94:0/3246583783 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c1591c0 con 0x7fa33c0c5480 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa341870640 1 --2- 172.21.15.94:0/3246583783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5480 0x7fa33c155ba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa3380097a0 tx=0x7fa33802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f85aaf83320ad766 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa34086e640 1 -- 172.21.15.94:0/3246583783 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3380315a0 con 0x7fa33c0c5480 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa34086e640 1 -- 172.21.15.94:0/3246583783 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa338037070 con 0x7fa33c0c5480 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa34086e640 1 -- 172.21.15.94:0/3246583783 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338031960 con 0x7fa33c0c5480 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 -- 172.21.15.94:0/3246583783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5480 msgr2=0x7fa33c155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 --2- 172.21.15.94:0/3246583783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5480 0x7fa33c155ba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa3380097a0 tx=0x7fa33802b840 comp rx=0 tx=0).stop 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 -- 172.21.15.94:0/3246583783 shutdown_connections 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 --2- 172.21.15.94:0/3246583783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 --2- 172.21.15.94:0/3246583783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5480 0x7fa33c155ba0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 -- 172.21.15.94:0/3246583783 >> 172.21.15.94:0/3246583783 conn(0x7fa33c06b1a0 msgr2=0x7fa33c06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 -- 172.21.15.94:0/3246583783 shutdown_connections 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.825+0000 7fa343ad4640 1 -- 172.21.15.94:0/3246583783 wait complete. 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.826+0000 7fa343ad4640 1 Processor -- start 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.826+0000 7fa343ad4640 1 -- start start 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa343ad4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0c5480 0x7fa33c1501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa343ad4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5860 0x7fa33c0c3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa343ad4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa343ad4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c155680 con 0x7fa33c0c5480 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa343ad4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c1553a0 con 0x7fa33c0c5860 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa343ad4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c152a20 con 0x7fa33c1586f0 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0c5480 0x7fa33c1501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa34106f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5860 0x7fa33c0c3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0c5480 0x7fa33c1501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33872/0 (socket says 172.21.15.94:33872) 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa342071640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 -- 172.21.15.94:0/1832346663 learned_addr learned my addr 172.21.15.94:0/1832346663 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 -- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 msgr2=0x7fa33c150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 -- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5860 msgr2=0x7fa33c0c3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5860 0x7fa33c0c3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.827+0000 7fa341870640 1 -- 172.21.15.94:0/1832346663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c14ddd0 con 0x7fa33c0c5480 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.828+0000 7fa341870640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0c5480 0x7fa33c1501a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa338034040 tx=0x7fa3380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.828+0000 7fa3327fc640 1 -- 172.21.15.94:0/1832346663 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338004230 con 0x7fa33c0c5480 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.828+0000 7fa3327fc640 1 -- 172.21.15.94:0/1832346663 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3380043f0 con 0x7fa33c0c5480 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.828+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c14d8f0 con 0x7fa33c0c5480 2024-03-06T07:18:52.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.828+0000 7fa3327fc640 1 -- 172.21.15.94:0/1832346663 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338003850 con 0x7fa33c0c5480 2024-03-06T07:18:52.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.828+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c151830 con 0x7fa33c0c5480 2024-03-06T07:18:52.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.829+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33c14fa70 con 0x7fa33c0c5480 2024-03-06T07:18:52.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.829+0000 7fa3327fc640 1 -- 172.21.15.94:0/1832346663 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3380039f0 con 0x7fa33c0c5480 2024-03-06T07:18:52.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.830+0000 7fa3327fc640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b500 0x7fa32006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:52.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.830+0000 7fa3327fc640 1 -- 172.21.15.94:0/1832346663 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa33802f030 con 0x7fa33c0c5480 2024-03-06T07:18:52.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.830+0000 7fa34106f640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b500 0x7fa32006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:52.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.830+0000 7fa34106f640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b500 0x7fa32006d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0c5160 tx=0x7fa32c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:52.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:52.834+0000 7fa3327fc640 1 -- 172.21.15.94:0/1832346663 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa338041070 con 0x7fa33c0c5480 2024-03-06T07:18:53.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.075+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7fa33c04f330 con 0x7fa33c0c5480 2024-03-06T07:18:53.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.129+0000 7fa3327fc640 1 -- 172.21.15.94:0/1832346663 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v55) v1 ==== 146+0+0 (secure 0 0 0) 0x7fa33807aa80 con 0x7fa33c0c5480 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b500 msgr2=0x7fa32006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b500 0x7fa32006d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0c5160 tx=0x7fa32c007d50 comp rx=0 tx=0).stop 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0c5480 msgr2=0x7fa33c1501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0c5480 0x7fa33c1501a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa338034040 tx=0x7fa3380046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 shutdown_connections 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa32006b500 0x7fa32006d9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa33c1586f0 0x7fa33c150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa33c0c5860 0x7fa33c0c3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 --2- 172.21.15.94:0/1832346663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa33c0c5480 0x7fa33c1501a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 >> 172.21.15.94:0/1832346663 conn(0x7fa33c06b1a0 msgr2=0x7fa33c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:53.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 shutdown_connections 2024-03-06T07:18:53.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.131+0000 7fa343ad4640 1 -- 172.21.15.94:0/1832346663 wait complete. 2024-03-06T07:18:53.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:18:53.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.245+0000 7feaed5c7640 1 Processor -- start 2024-03-06T07:18:53.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.245+0000 7feaed5c7640 1 -- start start 2024-03-06T07:18:53.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feaed5c7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7feae80b0070 0x7feae8155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feaed5c7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7feae806ca70 0x7feae80c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feaed5c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae80b0450 con 0x7feae80c5380 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feaed5c7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae806c020 con 0x7feae80b0070 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feaed5c7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae806cf20 con 0x7feae806ca70 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7feae80b0070 0x7feae8155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feae80c5380 0x7feae814e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50702/0 (socket says 172.21.15.94:50702) 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7feae80b0070 0x7feae8155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33876/0 (socket says 172.21.15.94:33876) 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae77fe640 1 -- 172.21.15.94:0/2173503515 learned_addr learned my addr 172.21.15.94:0/2173503515 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feade7fc640 1 --2- 172.21.15.94:0/2173503515 >> v2:172.21.15.145:3300/0 conn(0x7feae806ca70 0x7feae80c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae6ffd640 1 -- 172.21.15.94:0/2173503515 >> v2:172.21.15.145:3300/0 conn(0x7feae806ca70 msgr2=0x7feae80c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae6ffd640 1 --2- 172.21.15.94:0/2173503515 >> v2:172.21.15.145:3300/0 conn(0x7feae806ca70 0x7feae80c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae6ffd640 1 -- 172.21.15.94:0/2173503515 >> v1:172.21.15.94:6789/0 conn(0x7feae80c5380 legacy=0x7feae814e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae6ffd640 1 -- 172.21.15.94:0/2173503515 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80c77f0 con 0x7feae80b0070 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.246+0000 7feae67fc640 1 -- 172.21.15.94:0/2173503515 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163114713 0 0) 0x7feae0001490 con 0x7feae80c5380 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.247+0000 7feae6ffd640 1 --2- 172.21.15.94:0/2173503515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae80b0070 0x7feae8155c30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feae806d4c0 tx=0x7fead802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f9a4c6437e4fd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.247+0000 7feae67fc640 1 -- 172.21.15.94:0/2173503515 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80315a0 con 0x7feae80b0070 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.247+0000 7feae67fc640 1 -- 172.21.15.94:0/2173503515 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead8037070 con 0x7feae80b0070 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.247+0000 7feae67fc640 1 -- 172.21.15.94:0/2173503515 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8031960 con 0x7feae80b0070 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.247+0000 7feaed5c7640 1 -- 172.21.15.94:0/2173503515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae80b0070 msgr2=0x7feae8155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.247+0000 7feaed5c7640 1 --2- 172.21.15.94:0/2173503515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae80b0070 0x7feae8155c30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feae806d4c0 tx=0x7fead802b840 comp rx=0 tx=0).stop 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.248+0000 7feaed5c7640 1 -- 172.21.15.94:0/2173503515 shutdown_connections 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.248+0000 7feaed5c7640 1 --2- 172.21.15.94:0/2173503515 >> v2:172.21.15.145:3300/0 conn(0x7feae806ca70 0x7feae80c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.248+0000 7feaed5c7640 1 --2- 172.21.15.94:0/2173503515 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae80b0070 0x7feae8155c30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.248+0000 7feaed5c7640 1 -- 172.21.15.94:0/2173503515 >> 172.21.15.94:0/2173503515 conn(0x7feae80ac590 msgr2=0x7feae80ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.248+0000 7feaed5c7640 1 -- 172.21.15.94:0/2173503515 shutdown_connections 2024-03-06T07:18:53.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.248+0000 7feaed5c7640 1 -- 172.21.15.94:0/2173503515 wait complete. 2024-03-06T07:18:53.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 Processor -- start 2024-03-06T07:18:53.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 -- start start 2024-03-06T07:18:53.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae806ca70 0x7feae80c3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae80b0070 0x7feae80c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae80c5380 0x7feae8154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae806c550 con 0x7feae806ca70 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae806c270 con 0x7feae80c5380 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.249+0000 7feaed5c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae806da80 con 0x7feae80b0070 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae806ca70 0x7feae80c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae806ca70 0x7feae80c3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33892/0 (socket says 172.21.15.94:33892) 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 -- 172.21.15.94:0/1023912204 learned_addr learned my addr 172.21.15.94:0/1023912204 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feade7fc640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae80b0070 0x7feae80c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae77fe640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae80c5380 0x7feae8154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 -- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae80b0070 msgr2=0x7feae80c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae80b0070 0x7feae80c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 -- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae80c5380 msgr2=0x7feae8154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae80c5380 0x7feae8154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 -- 172.21.15.94:0/1023912204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae814e760 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feae6ffd640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae806ca70 0x7feae80c3460 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fead8009890 tx=0x7fead80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feadffff640 1 -- 172.21.15.94:0/1023912204 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8032630 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feadffff640 1 -- 172.21.15.94:0/1023912204 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead80327f0 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae80c4200 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feadffff640 1 -- 172.21.15.94:0/1023912204 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8032be0 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.250+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae80ca510 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.251+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae804f330 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.251+0000 7feadffff640 1 -- 172.21.15.94:0/1023912204 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fead804c020 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.252+0000 7feadffff640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feac806b4d0 0x7feac806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.252+0000 7feadffff640 1 -- 172.21.15.94:0/1023912204 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fead802f020 con 0x7feae806ca70 2024-03-06T07:18:53.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.252+0000 7feade7fc640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feac806b4d0 0x7feac806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.252+0000 7feade7fc640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feac806b4d0 0x7feac806d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7feae80c8d70 tx=0x7fead0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:53.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.256+0000 7feadffff640 1 -- 172.21.15.94:0/1023912204 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fead8040070 con 0x7feae806ca70 2024-03-06T07:18:53.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.429+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7feae806d860 con 0x7feae806ca70 2024-03-06T07:18:53.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.431+0000 7feadffff640 1 -- 172.21.15.94:0/1023912204 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v11)=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7fead807a5d0 con 0x7feae806ca70 2024-03-06T07:18:53.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feac806b4d0 msgr2=0x7feac806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feac806b4d0 0x7feac806d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7feae80c8d70 tx=0x7fead0009190 comp rx=0 tx=0).stop 2024-03-06T07:18:53.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae806ca70 msgr2=0x7feae80c3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae806ca70 0x7feae80c3460 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fead8009890 tx=0x7fead80046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 shutdown_connections 2024-03-06T07:18:53.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feac806b4d0 0x7feac806d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae80c5380 0x7feae8154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae80b0070 0x7feae80c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 --2- 172.21.15.94:0/1023912204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feae806ca70 0x7feae80c3460 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.433+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 >> 172.21.15.94:0/1023912204 conn(0x7feae80ac590 msgr2=0x7feae8151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:53.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.434+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 shutdown_connections 2024-03-06T07:18:53.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.434+0000 7feaed5c7640 1 -- 172.21.15.94:0/1023912204 wait complete. 2024-03-06T07:18:53.446 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:18:53.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:18:53.484 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:18:53.524 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:18:53.524 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:18:53.524 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:53.540 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:18:53.540 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:53.596 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:18:53.596 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:53.652 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:53.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.795+0000 7fbd3a81c640 1 Processor -- start 2024-03-06T07:18:53.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.795+0000 7fbd3a81c640 1 -- start start 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd3a81c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd340c88c0 0x7fbd34155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd3a81c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbd34158b60 0x7fbd34158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd3a81c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd340c73d0 con 0x7fbd3414f360 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd3a81c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd340c8dd0 con 0x7fbd34158b60 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd3a81c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340be2e0 con 0x7fbd340c88c0 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd33fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbd3414f360 0x7fbd340c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50718/0 (socket says 172.21.15.94:50718) 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd33fff640 1 -- 172.21.15.94:0/1269607636 learned_addr learned my addr 172.21.15.94:0/1269607636 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd337fe640 1 --2- 172.21.15.94:0/1269607636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd340c88c0 0x7fbd34155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.796+0000 7fbd38db9640 1 --2- 172.21.15.94:0/1269607636 >> v2:172.21.15.145:3300/0 conn(0x7fbd34158b60 0x7fbd34158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.797+0000 7fbd32ffd640 1 -- 172.21.15.94:0/1269607636 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235574478 0 0) 0x7fbd340c73d0 con 0x7fbd3414f360 2024-03-06T07:18:53.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.797+0000 7fbd32ffd640 1 -- 172.21.15.94:0/1269607636 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c003520 con 0x7fbd3414f360 2024-03-06T07:18:53.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.797+0000 7fbd337fe640 1 -- 172.21.15.94:0/1269607636 >> v2:172.21.15.145:3300/0 conn(0x7fbd34158b60 msgr2=0x7fbd34158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:53.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.797+0000 7fbd337fe640 1 --2- 172.21.15.94:0/1269607636 >> v2:172.21.15.145:3300/0 conn(0x7fbd34158b60 0x7fbd34158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.797+0000 7fbd337fe640 1 -- 172.21.15.94:0/1269607636 >> v1:172.21.15.94:6789/0 conn(0x7fbd3414f360 legacy=0x7fbd340c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.797+0000 7fbd337fe640 1 -- 172.21.15.94:0/1269607636 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340c7670 con 0x7fbd340c88c0 2024-03-06T07:18:53.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd337fe640 1 --2- 172.21.15.94:0/1269607636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd340c88c0 0x7fbd34155c10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd280097a0 tx=0x7fbd2802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4010d68da2a9b5b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:53.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd32ffd640 1 -- 172.21.15.94:0/1269607636 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd280315a0 con 0x7fbd340c88c0 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd32ffd640 1 -- 172.21.15.94:0/1269607636 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd28037070 con 0x7fbd340c88c0 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd32ffd640 1 -- 172.21.15.94:0/1269607636 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd28031960 con 0x7fbd340c88c0 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd3a81c640 1 -- 172.21.15.94:0/1269607636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd340c88c0 msgr2=0x7fbd34155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/1269607636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd340c88c0 0x7fbd34155c10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd280097a0 tx=0x7fbd2802b840 comp rx=0 tx=0).stop 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd3a81c640 1 -- 172.21.15.94:0/1269607636 shutdown_connections 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/1269607636 >> v2:172.21.15.145:3300/0 conn(0x7fbd34158b60 0x7fbd34158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/1269607636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd340c88c0 0x7fbd34155c10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd3a81c640 1 -- 172.21.15.94:0/1269607636 >> 172.21.15.94:0/1269607636 conn(0x7fbd340ac570 msgr2=0x7fbd340ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.798+0000 7fbd3a81c640 1 -- 172.21.15.94:0/1269607636 shutdown_connections 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.799+0000 7fbd3a81c640 1 -- 172.21.15.94:0/1269607636 wait complete. 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 Processor -- start 2024-03-06T07:18:53.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 -- start start 2024-03-06T07:18:53.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd340c88c0 0x7fbd340c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3414f360 0x7fbd341ed750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd34158b60 0x7fbd341edc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340c5a50 con 0x7fbd34158b60 2024-03-06T07:18:53.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340c7c30 con 0x7fbd340c88c0 2024-03-06T07:18:53.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.800+0000 7fbd3a81c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340c5bd0 con 0x7fbd3414f360 2024-03-06T07:18:53.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd34158b60 0x7fbd341edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd34158b60 0x7fbd341edc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33906/0 (socket says 172.21.15.94:33906) 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 -- 172.21.15.94:0/2229485910 learned_addr learned my addr 172.21.15.94:0/2229485910 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd33fff640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd340c88c0 0x7fbd340c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd337fe640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3414f360 0x7fbd341ed750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 -- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3414f360 msgr2=0x7fbd341ed750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3414f360 0x7fbd341ed750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 -- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd340c88c0 msgr2=0x7fbd340c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd340c88c0 0x7fbd340c0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 -- 172.21.15.94:0/2229485910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340c0120 con 0x7fbd34158b60 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd38db9640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd34158b60 0x7fbd341edc90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009dd0 tx=0x7fbd2c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd30ff9640 1 -- 172.21.15.94:0/2229485910 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c018020 con 0x7fbd34158b60 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd30ff9640 1 -- 172.21.15.94:0/2229485910 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c014480 con 0x7fbd34158b60 2024-03-06T07:18:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd341ebc50 con 0x7fbd34158b60 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd30ff9640 1 -- 172.21.15.94:0/2229485910 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c014870 con 0x7fbd34158b60 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.801+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd340be9e0 con 0x7fbd34158b60 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.802+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd340be4f0 con 0x7fbd34158b60 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.803+0000 7fbd30ff9640 1 -- 172.21.15.94:0/2229485910 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd2c005d10 con 0x7fbd34158b60 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.803+0000 7fbd30ff9640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd2006b4d0 0x7fbd2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.803+0000 7fbd30ff9640 1 -- 172.21.15.94:0/2229485910 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbd2c08ae00 con 0x7fbd34158b60 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.803+0000 7fbd33fff640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd2006b4d0 0x7fbd2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:53.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.804+0000 7fbd33fff640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd2006b4d0 0x7fbd2006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd24005880 tx=0x7fbd24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.811+0000 7fbd30ff9640 1 -- 172.21.15.94:0/2229485910 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd2c022070 con 0x7fbd34158b60 2024-03-06T07:18:53.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.987+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd3404f330 con 0x7fbd34158b60 2024-03-06T07:18:53.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.988+0000 7fbd30ff9640 1 -- 172.21.15.94:0/2229485910 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7fbd2c010040 con 0x7fbd34158b60 2024-03-06T07:18:53.991 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:53.991 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd2006b4d0 msgr2=0x7fbd2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd2006b4d0 0x7fbd2006d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd24005880 tx=0x7fbd24009040 comp rx=0 tx=0).stop 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd34158b60 msgr2=0x7fbd341edc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd34158b60 0x7fbd341edc90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009dd0 tx=0x7fbd2c00c490 comp rx=0 tx=0).stop 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 shutdown_connections 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd2006b4d0 0x7fbd2006d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd34158b60 0x7fbd341edc90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3414f360 0x7fbd341ed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 --2- 172.21.15.94:0/2229485910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd340c88c0 0x7fbd340c0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 >> 172.21.15.94:0/2229485910 conn(0x7fbd340ac570 msgr2=0x7fbd3406c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 shutdown_connections 2024-03-06T07:18:53.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:53.990+0000 7fbd3a81c640 1 -- 172.21.15.94:0/2229485910 wait complete. 2024-03-06T07:18:54.003 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:54.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:54.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.105+0000 7fd603a9a640 1 Processor -- start 2024-03-06T07:18:54.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.105+0000 7fd603a9a640 1 -- start start 2024-03-06T07:18:54.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd603a9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc14d350 0x7fd5fc06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd603a9a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd5fc15b8c0 0x7fd5fc15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd603a9a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5fc0c1c90 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd603a9a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc0ac750 con 0x7fd5fc15b8c0 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd603a9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0c1a40 con 0x7fd5fc14d350 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd601035640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd5fc0acbc0 0x7fd5fc0ab950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50732/0 (socket says 172.21.15.94:50732) 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd601035640 1 -- 172.21.15.94:0/623936280 learned_addr learned my addr 172.21.15.94:0/623936280 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd601836640 1 --2- 172.21.15.94:0/623936280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc14d350 0x7fd5fc06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd602037640 1 --2- 172.21.15.94:0/623936280 >> v2:172.21.15.145:3300/0 conn(0x7fd5fc15b8c0 0x7fd5fc15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd600834640 1 -- 172.21.15.94:0/623936280 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194374679 0 0) 0x7fd5fc0c1c90 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.106+0000 7fd600834640 1 -- 172.21.15.94:0/623936280 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e8003520 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.107+0000 7fd601836640 1 -- 172.21.15.94:0/623936280 >> v2:172.21.15.145:3300/0 conn(0x7fd5fc15b8c0 msgr2=0x7fd5fc15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:54.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.107+0000 7fd601836640 1 --2- 172.21.15.94:0/623936280 >> v2:172.21.15.145:3300/0 conn(0x7fd5fc15b8c0 0x7fd5fc15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.107+0000 7fd601836640 1 -- 172.21.15.94:0/623936280 >> v1:172.21.15.94:6789/0 conn(0x7fd5fc0acbc0 legacy=0x7fd5fc0ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.107+0000 7fd601836640 1 -- 172.21.15.94:0/623936280 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc0ac470 con 0x7fd5fc14d350 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.107+0000 7fd601836640 1 --2- 172.21.15.94:0/623936280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc14d350 0x7fd5fc06afc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0c1bc0 tx=0x7fd5f802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2005121859ba2714 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.107+0000 7fd600834640 1 -- 172.21.15.94:0/623936280 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8039070 con 0x7fd5fc14d350 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.107+0000 7fd600834640 1 -- 172.21.15.94:0/623936280 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f802cbb0 con 0x7fd5fc14d350 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd600834640 1 -- 172.21.15.94:0/623936280 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8034440 con 0x7fd5fc14d350 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 -- 172.21.15.94:0/623936280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc14d350 msgr2=0x7fd5fc06afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 --2- 172.21.15.94:0/623936280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc14d350 0x7fd5fc06afc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0c1bc0 tx=0x7fd5f802be00 comp rx=0 tx=0).stop 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 -- 172.21.15.94:0/623936280 shutdown_connections 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 --2- 172.21.15.94:0/623936280 >> v2:172.21.15.145:3300/0 conn(0x7fd5fc15b8c0 0x7fd5fc15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 --2- 172.21.15.94:0/623936280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc14d350 0x7fd5fc06afc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 -- 172.21.15.94:0/623936280 >> 172.21.15.94:0/623936280 conn(0x7fd5fc0c3b40 msgr2=0x7fd5fc0c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 -- 172.21.15.94:0/623936280 shutdown_connections 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.108+0000 7fd603a9a640 1 -- 172.21.15.94:0/623936280 wait complete. 2024-03-06T07:18:54.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.109+0000 7fd603a9a640 1 Processor -- start 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.109+0000 7fd603a9a640 1 -- start start 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd603a9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5fc0acbc0 0x7fd5fc143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd603a9a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5fc14d350 0x7fd5fc159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd603a9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc15b8c0 0x7fd5fc1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd603a9a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0aa160 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd603a9a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc153c00 con 0x7fd5fc14d350 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd603a9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0aa2e0 con 0x7fd5fc15b8c0 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5fc0acbc0 0x7fd5fc143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5fc0acbc0 0x7fd5fc143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33908/0 (socket says 172.21.15.94:33908) 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 -- 172.21.15.94:0/3775457438 learned_addr learned my addr 172.21.15.94:0/3775457438 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601035640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5fc14d350 0x7fd5fc159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd602037640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc15b8c0 0x7fd5fc1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 -- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc15b8c0 msgr2=0x7fd5fc1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc15b8c0 0x7fd5fc1598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 -- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5fc14d350 msgr2=0x7fd5fc159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5fc14d350 0x7fd5fc159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 -- 172.21.15.94:0/3775457438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc144a40 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd601836640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5fc0acbc0 0x7fd5fc143c30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009e80 tx=0x7fd5f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.110+0000 7fd5f27fc640 1 -- 172.21.15.94:0/3775457438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8039050 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.111+0000 7fd5f27fc640 1 -- 172.21.15.94:0/3775457438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f8045070 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.111+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc156d30 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.111+0000 7fd5f27fc640 1 -- 172.21.15.94:0/3775457438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8009450 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.111+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc1ed220 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.111+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc0694c0 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.112+0000 7fd5f27fc640 1 -- 172.21.15.94:0/3775457438 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5f803e850 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.112+0000 7fd5f27fc640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5e006b4d0 0x7fd5e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.112+0000 7fd5f27fc640 1 -- 172.21.15.94:0/3775457438 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd5f80ad270 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.112+0000 7fd601035640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5e006b4d0 0x7fd5e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.113+0000 7fd601035640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5e006b4d0 0x7fd5e006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc1e9f50 tx=0x7fd5ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.117+0000 7fd5f27fc640 1 -- 172.21.15.94:0/3775457438 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5fc0694c0 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.288+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5fc074ac0 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.288+0000 7fd5f27fc640 1 -- 172.21.15.94:0/3775457438 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5fc074ac0 con 0x7fd5fc0acbc0 2024-03-06T07:18:54.290 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:54.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5e006b4d0 msgr2=0x7fd5e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5e006b4d0 0x7fd5e006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc1e9f50 tx=0x7fd5ec009040 comp rx=0 tx=0).stop 2024-03-06T07:18:54.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5fc0acbc0 msgr2=0x7fd5fc143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5fc0acbc0 0x7fd5fc143c30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009e80 tx=0x7fd5f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:54.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 shutdown_connections 2024-03-06T07:18:54.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5e006b4d0 0x7fd5e006d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5fc15b8c0 0x7fd5fc1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5fc14d350 0x7fd5fc159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 --2- 172.21.15.94:0/3775457438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5fc0acbc0 0x7fd5fc143c30 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 >> 172.21.15.94:0/3775457438 conn(0x7fd5fc0c3b40 msgr2=0x7fd5fc06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:54.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 shutdown_connections 2024-03-06T07:18:54.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.290+0000 7fd603a9a640 1 -- 172.21.15.94:0/3775457438 wait complete. 2024-03-06T07:18:54.303 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:54.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.409+0000 7f86cd9d6640 1 Processor -- start 2024-03-06T07:18:54.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.409+0000 7f86cd9d6640 1 -- start start 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86cd9d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f86c8159330 0x7f86c8160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86cd9d6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c806b800 con 0x7f86c806afc0 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86cd9d6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c80c5840 con 0x7f86c806b3a0 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86cd9d6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c8158790 con 0x7f86c8159330 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86c77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f86c8159330 0x7f86c8160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86c77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f86c8159330 0x7f86c8160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33910/0 (socket says 172.21.15.94:33910) 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86c77fe640 1 -- 172.21.15.94:0/689898693 learned_addr learned my addr 172.21.15.94:0/689898693 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:54.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86c6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f86c806afc0 0x7f86c80c2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50744/0 (socket says 172.21.15.94:50744) 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86c77fe640 1 -- 172.21.15.94:0/689898693 >> v1:172.21.15.145:6789/0 conn(0x7f86c806b3a0 legacy=0x7f86c8158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86c77fe640 1 -- 172.21.15.94:0/689898693 >> v1:172.21.15.94:6789/0 conn(0x7f86c806afc0 legacy=0x7f86c80c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.410+0000 7f86c77fe640 1 -- 172.21.15.94:0/689898693 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c80c5b20 con 0x7f86c8159330 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.411+0000 7f86c77fe640 1 --2- 172.21.15.94:0/689898693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c8159330 0x7f86c8160ad0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6431eade819f6ed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.411+0000 7f86c5ffb640 1 -- 172.21.15.94:0/689898693 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c002cb20 con 0x7f86c8159330 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.411+0000 7f86c5ffb640 1 -- 172.21.15.94:0/689898693 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c002ccc0 con 0x7f86c8159330 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.411+0000 7f86c5ffb640 1 -- 172.21.15.94:0/689898693 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0034610 con 0x7f86c8159330 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.412+0000 7f86cd9d6640 1 -- 172.21.15.94:0/689898693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c8159330 msgr2=0x7f86c8160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.412+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/689898693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c8159330 0x7f86c8160ad0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009e50 tx=0x7f86c002bf80 comp rx=0 tx=0).stop 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.412+0000 7f86cd9d6640 1 -- 172.21.15.94:0/689898693 shutdown_connections 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.412+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/689898693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c8159330 0x7f86c8160ad0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.412+0000 7f86cd9d6640 1 -- 172.21.15.94:0/689898693 >> 172.21.15.94:0/689898693 conn(0x7f86c80c4220 msgr2=0x7f86c80c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.412+0000 7f86cd9d6640 1 -- 172.21.15.94:0/689898693 shutdown_connections 2024-03-06T07:18:54.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.412+0000 7f86cd9d6640 1 -- 172.21.15.94:0/689898693 wait complete. 2024-03-06T07:18:54.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.413+0000 7f86cd9d6640 1 Processor -- start 2024-03-06T07:18:54.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.413+0000 7f86cd9d6640 1 -- start start 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c806afc0 0x7f86c814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86c806b3a0 0x7f86c8157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86c8159330 0x7f86c8158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c8155220 con 0x7f86c806afc0 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80c60e0 con 0x7f86c806b3a0 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80c2b30 con 0x7f86c8159330 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c806afc0 0x7f86c814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c806afc0 0x7f86c814cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33914/0 (socket says 172.21.15.94:33914) 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 -- 172.21.15.94:0/967638937 learned_addr learned my addr 172.21.15.94:0/967638937 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:54.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c67fc640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86c806b3a0 0x7f86c8157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c77fe640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86c8159330 0x7f86c8158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 -- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86c8159330 msgr2=0x7f86c8158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86c8159330 0x7f86c8158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 -- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86c806b3a0 msgr2=0x7f86c8157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86c806b3a0 0x7f86c8157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 -- 172.21.15.94:0/967638937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c814e400 con 0x7f86c806afc0 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c77fe640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86c8159330 0x7f86c8158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c67fc640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86c806b3a0 0x7f86c8157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86c6ffd640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c806afc0 0x7f86c814cd00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f86b800c3e0 tx=0x7f86b800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86affff640 1 -- 172.21.15.94:0/967638937 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b80074e0 con 0x7f86c806afc0 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86affff640 1 -- 172.21.15.94:0/967638937 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b8007680 con 0x7f86c806afc0 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c81eb6b0 con 0x7f86c806afc0 2024-03-06T07:18:54.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86affff640 1 -- 172.21.15.94:0/967638937 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b80049e0 con 0x7f86c806afc0 2024-03-06T07:18:54.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.414+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c81ebc70 con 0x7f86c806afc0 2024-03-06T07:18:54.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.415+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c804f330 con 0x7f86c806afc0 2024-03-06T07:18:54.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.416+0000 7f86affff640 1 -- 172.21.15.94:0/967638937 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86b801a020 con 0x7f86c806afc0 2024-03-06T07:18:54.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.416+0000 7f86affff640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86b406b4d0 0x7f86b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.416+0000 7f86affff640 1 -- 172.21.15.94:0/967638937 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f86b8089c70 con 0x7f86c806afc0 2024-03-06T07:18:54.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.416+0000 7f86c67fc640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86b406b4d0 0x7f86b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.417+0000 7f86c67fc640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86b406b4d0 0x7f86b406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f86c814c900 tx=0x7f86bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.420+0000 7f86affff640 1 -- 172.21.15.94:0/967638937 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86b8010300 con 0x7f86c806afc0 2024-03-06T07:18:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.590+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f86c806be40 con 0x7f86c806afc0 2024-03-06T07:18:54.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.590+0000 7f86affff640 1 -- 172.21.15.94:0/967638937 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f86b805e020 con 0x7f86c806afc0 2024-03-06T07:18:54.592 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86b406b4d0 msgr2=0x7f86b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86b406b4d0 0x7f86b406d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f86c814c900 tx=0x7f86bc009040 comp rx=0 tx=0).stop 2024-03-06T07:18:54.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c806afc0 msgr2=0x7f86c814cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c806afc0 0x7f86c814cd00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f86b800c3e0 tx=0x7f86b800c8a0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 shutdown_connections 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86b406b4d0 0x7f86b406d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86c8159330 0x7f86c8158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86c806b3a0 0x7f86c8157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 --2- 172.21.15.94:0/967638937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86c806afc0 0x7f86c814cd00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 >> 172.21.15.94:0/967638937 conn(0x7f86c80c4220 msgr2=0x7f86c8151360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 shutdown_connections 2024-03-06T07:18:54.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.592+0000 7f86cd9d6640 1 -- 172.21.15.94:0/967638937 wait complete. 2024-03-06T07:18:54.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:54.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.707+0000 7f12c6aed640 1 Processor -- start 2024-03-06T07:18:54.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.707+0000 7f12c6aed640 1 -- start start 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.707+0000 7f12c6aed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c006afc0 0x7f12c006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c6aed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00c5840 con 0x7f12c00c2580 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c6aed640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00c5b20 con 0x7f12c00c2120 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c6aed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0158790 con 0x7f12c006afc0 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c508a640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f12c00c2580 0x7f12c015fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50746/0 (socket says 172.21.15.94:50746) 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c508a640 1 -- 172.21.15.94:0/3962063113 learned_addr learned my addr 172.21.15.94:0/3962063113 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c4889640 1 --2- 172.21.15.94:0/3962063113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c006afc0 0x7f12c006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12b77fe640 1 -- 172.21.15.94:0/3962063113 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124463092 0 0) 0x7f12c00c5840 con 0x7f12c00c2580 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12b77fe640 1 -- 172.21.15.94:0/3962063113 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a8003520 con 0x7f12c00c2580 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12b77fe640 1 -- 172.21.15.94:0/3962063113 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301862763 0 0) 0x7f12c00c5b20 con 0x7f12c00c2120 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12b77fe640 1 -- 172.21.15.94:0/3962063113 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c00c5840 con 0x7f12c00c2120 2024-03-06T07:18:54.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c4889640 1 -- 172.21.15.94:0/3962063113 >> v1:172.21.15.145:6789/0 conn(0x7f12c00c2120 legacy=0x7f12c0158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c4889640 1 -- 172.21.15.94:0/3962063113 >> v1:172.21.15.94:6789/0 conn(0x7f12c00c2580 legacy=0x7f12c015fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.708+0000 7f12c4889640 1 -- 172.21.15.94:0/3962063113 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c00c5e00 con 0x7f12c006afc0 2024-03-06T07:18:54.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.709+0000 7f12c4889640 1 --2- 172.21.15.94:0/3962063113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c006afc0 0x7f12c006b3a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0097d0 tx=0x7f12bc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7cad4912b6b5466 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.709+0000 7f12b77fe640 1 -- 172.21.15.94:0/3962063113 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc035020 con 0x7f12c006afc0 2024-03-06T07:18:54.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.709+0000 7f12b77fe640 1 -- 172.21.15.94:0/3962063113 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc031800 con 0x7f12c006afc0 2024-03-06T07:18:54.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.709+0000 7f12b77fe640 1 -- 172.21.15.94:0/3962063113 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc031bd0 con 0x7f12c006afc0 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.709+0000 7f12c6aed640 1 -- 172.21.15.94:0/3962063113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c006afc0 msgr2=0x7f12c006b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.709+0000 7f12c6aed640 1 --2- 172.21.15.94:0/3962063113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c006afc0 0x7f12c006b3a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0097d0 tx=0x7f12bc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.710+0000 7f12c6aed640 1 -- 172.21.15.94:0/3962063113 shutdown_connections 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.710+0000 7f12c6aed640 1 --2- 172.21.15.94:0/3962063113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c006afc0 0x7f12c006b3a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.710+0000 7f12c6aed640 1 -- 172.21.15.94:0/3962063113 >> 172.21.15.94:0/3962063113 conn(0x7f12c00c4220 msgr2=0x7f12c00c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.710+0000 7f12c6aed640 1 -- 172.21.15.94:0/3962063113 shutdown_connections 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.710+0000 7f12c6aed640 1 -- 172.21.15.94:0/3962063113 wait complete. 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 Processor -- start 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 -- start start 2024-03-06T07:18:54.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12c006afc0 0x7f12c01eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12c00c2120 0x7f12c014ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c00c2580 0x7f12c0072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00c2b30 con 0x7f12c006afc0 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0155220 con 0x7f12c00c2120 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.711+0000 7f12c6aed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00c2cb0 con 0x7f12c00c2580 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12c006afc0 0x7f12c01eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12c006afc0 0x7f12c01eed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33920/0 (socket says 172.21.15.94:33920) 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 -- 172.21.15.94:0/1940318798 learned_addr learned my addr 172.21.15.94:0/1940318798 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12b7fff640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12c00c2120 0x7f12c014ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c508a640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c00c2580 0x7f12c0072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 -- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c00c2580 msgr2=0x7f12c0072bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c00c2580 0x7f12c0072bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 -- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12c00c2120 msgr2=0x7f12c014ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12c00c2120 0x7f12c014ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 -- 172.21.15.94:0/1940318798 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c01ef2b0 con 0x7f12c006afc0 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c4889640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12c006afc0 0x7f12c01eed70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f12bc02beb0 tx=0x7f12bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12b57fa640 1 -- 172.21.15.94:0/1940318798 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc035040 con 0x7f12c006afc0 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12b57fa640 1 -- 172.21.15.94:0/1940318798 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc03ad00 con 0x7f12c006afc0 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c014e0b0 con 0x7f12c006afc0 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12b57fa640 1 -- 172.21.15.94:0/1940318798 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12bc0466c0 con 0x7f12c006afc0 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.712+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c014e670 con 0x7f12c006afc0 2024-03-06T07:18:54.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.713+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c004f330 con 0x7f12c006afc0 2024-03-06T07:18:54.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.713+0000 7f12b57fa640 1 -- 172.21.15.94:0/1940318798 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12bc038020 con 0x7f12c006afc0 2024-03-06T07:18:54.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.714+0000 7f12b57fa640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12ac06b4d0 0x7f12ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:54.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.714+0000 7f12b57fa640 1 -- 172.21.15.94:0/1940318798 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f12bc0ac460 con 0x7f12c006afc0 2024-03-06T07:18:54.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.714+0000 7f12b7fff640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12ac06b4d0 0x7f12ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:54.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.714+0000 7f12b7fff640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12ac06b4d0 0x7f12ac06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f12b0007fd0 tx=0x7f12b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:54.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.718+0000 7f12b57fa640 1 -- 172.21.15.94:0/1940318798 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12bc07a9e0 con 0x7f12c006afc0 2024-03-06T07:18:54.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.887+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12c014a5d0 con 0x7f12c006afc0 2024-03-06T07:18:54.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.887+0000 7f12b57fa640 1 -- 172.21.15.94:0/1940318798 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f12bc07ea80 con 0x7f12c006afc0 2024-03-06T07:18:54.889 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:54.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12ac06b4d0 msgr2=0x7f12ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12ac06b4d0 0x7f12ac06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f12b0007fd0 tx=0x7f12b0009040 comp rx=0 tx=0).stop 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12c006afc0 msgr2=0x7f12c01eed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12c006afc0 0x7f12c01eed70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f12bc02beb0 tx=0x7f12bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 shutdown_connections 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12ac06b4d0 0x7f12ac06d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12c00c2580 0x7f12c0072bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12c00c2120 0x7f12c014ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 --2- 172.21.15.94:0/1940318798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12c006afc0 0x7f12c01eed70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 >> 172.21.15.94:0/1940318798 conn(0x7f12c00c4220 msgr2=0x7f12c0073e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 shutdown_connections 2024-03-06T07:18:54.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:54.889+0000 7f12c6aed640 1 -- 172.21.15.94:0/1940318798 wait complete. 2024-03-06T07:18:54.902 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.004+0000 7f79e0b7a640 1 Processor -- start 2024-03-06T07:18:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.004+0000 7f79e0b7a640 1 -- start start 2024-03-06T07:18:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79e0b7a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f79dc1538c0 0x7f79dc1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79e0b7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79e0b7a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc154330 con 0x7f79dc0ab310 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79e0b7a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79dc0c46a0 con 0x7f79dc1538c0 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79e0b7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc153dc0 con 0x7f79dc0a97c0 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79d9d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f79dc1538c0 0x7f79dc1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79da59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f79dc0ab310 0x7f79dc0ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50750/0 (socket says 172.21.15.94:50750) 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79da59c640 1 -- 172.21.15.94:0/1057839289 learned_addr learned my addr 172.21.15.94:0/1057839289 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79dad9d640 1 --2- 172.21.15.94:0/1057839289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79d9d9b640 1 -- 172.21.15.94:0/1057839289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 msgr2=0x7f79dc1499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79d9d9b640 1 --2- 172.21.15.94:0/1057839289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc1499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79d9d9b640 1 -- 172.21.15.94:0/1057839289 >> v1:172.21.15.94:6789/0 conn(0x7f79dc0ab310 legacy=0x7f79dc0ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79d959a640 1 -- 172.21.15.94:0/1057839289 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132464057 0 0) 0x7f79dc154330 con 0x7f79dc0ab310 2024-03-06T07:18:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.005+0000 7f79d9d9b640 1 -- 172.21.15.94:0/1057839289 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0c5d30 con 0x7f79dc1538c0 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.006+0000 7f79d9d9b640 1 --2- 172.21.15.94:0/1057839289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc1538c0 0x7f79dc1494a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f79d00097a0 tx=0x7f79d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61866619f7444a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.006+0000 7f79d959a640 1 -- 172.21.15.94:0/1057839289 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d00315a0 con 0x7f79dc1538c0 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.006+0000 7f79d959a640 1 -- 172.21.15.94:0/1057839289 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0037070 con 0x7f79dc1538c0 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.006+0000 7f79d959a640 1 -- 172.21.15.94:0/1057839289 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0031960 con 0x7f79dc1538c0 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 -- 172.21.15.94:0/1057839289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc1538c0 msgr2=0x7f79dc1494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/1057839289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc1538c0 0x7f79dc1494a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f79d00097a0 tx=0x7f79d002b840 comp rx=0 tx=0).stop 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 -- 172.21.15.94:0/1057839289 shutdown_connections 2024-03-06T07:18:55.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/1057839289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/1057839289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc1538c0 0x7f79dc1494a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 -- 172.21.15.94:0/1057839289 >> 172.21.15.94:0/1057839289 conn(0x7f79dc0c6710 msgr2=0x7f79dc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:55.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 -- 172.21.15.94:0/1057839289 shutdown_connections 2024-03-06T07:18:55.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.007+0000 7f79e0b7a640 1 -- 172.21.15.94:0/1057839289 wait complete. 2024-03-06T07:18:55.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.008+0000 7f79e0b7a640 1 Processor -- start 2024-03-06T07:18:55.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.008+0000 7f79e0b7a640 1 -- start start 2024-03-06T07:18:55.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79e0b7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc0c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79e0b7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc0ab310 0x7f79dc1ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79e0b7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79dc1538c0 0x7f79dc1ec220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79e0b7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0c4b30 con 0x7f79dc0ab310 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79e0b7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0c62f0 con 0x7f79dc1538c0 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79e0b7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc0c4cb0 con 0x7f79dc0a97c0 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc0ab310 0x7f79dc1ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc0ab310 0x7f79dc1ebce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33944/0 (socket says 172.21.15.94:33944) 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 -- 172.21.15.94:0/610313187 learned_addr learned my addr 172.21.15.94:0/610313187 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79dad9d640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79dc1538c0 0x7f79dc1ec220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79da59c640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc0c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 -- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 msgr2=0x7f79dc0c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc0c24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 -- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79dc1538c0 msgr2=0x7f79dc1ec220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79dc1538c0 0x7f79dc1ec220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 -- 172.21.15.94:0/610313187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0718e0 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.009+0000 7f79d9d9b640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc0ab310 0x7f79dc1ebce0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f79d0034040 tx=0x7f79d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.010+0000 7f79caffd640 1 -- 172.21.15.94:0/610313187 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0004230 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.010+0000 7f79caffd640 1 -- 172.21.15.94:0/610313187 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d00043d0 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.010+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc142880 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.010+0000 7f79caffd640 1 -- 172.21.15.94:0/610313187 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0003850 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.010+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc0c2a10 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.011+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc04f330 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.011+0000 7f79caffd640 1 -- 172.21.15.94:0/610313187 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79d00039f0 con 0x7f79dc0ab310 2024-03-06T07:18:55.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.011+0000 7f79caffd640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79bc06b4d0 0x7f79bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.011+0000 7f79caffd640 1 -- 172.21.15.94:0/610313187 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f79d002f030 con 0x7f79dc0ab310 2024-03-06T07:18:55.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.012+0000 7f79da59c640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79bc06b4d0 0x7f79bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.012+0000 7f79da59c640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79bc06b4d0 0x7f79bc06d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000f60 tx=0x7f79cc007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.016+0000 7f79caffd640 1 -- 172.21.15.94:0/610313187 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79d0041070 con 0x7f79dc0ab310 2024-03-06T07:18:55.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.186+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f79dc0c5a70 con 0x7f79dc0ab310 2024-03-06T07:18:55.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.186+0000 7f79caffd640 1 -- 172.21.15.94:0/610313187 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f79d007a820 con 0x7f79dc0ab310 2024-03-06T07:18:55.188 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:55.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79bc06b4d0 msgr2=0x7f79bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79bc06b4d0 0x7f79bc06d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000f60 tx=0x7f79cc007d50 comp rx=0 tx=0).stop 2024-03-06T07:18:55.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc0ab310 msgr2=0x7f79dc1ebce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc0ab310 0x7f79dc1ebce0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f79d0034040 tx=0x7f79d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 shutdown_connections 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79bc06b4d0 0x7f79bc06d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79dc1538c0 0x7f79dc1ec220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79dc0ab310 0x7f79dc1ebce0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 --2- 172.21.15.94:0/610313187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79dc0a97c0 0x7f79dc0c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 >> 172.21.15.94:0/610313187 conn(0x7f79dc0c6710 msgr2=0x7f79dc0a6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 shutdown_connections 2024-03-06T07:18:55.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.188+0000 7f79e0b7a640 1 -- 172.21.15.94:0/610313187 wait complete. 2024-03-06T07:18:55.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:18:55.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.301+0000 7f50c306f640 1 Processor -- start 2024-03-06T07:18:55.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.301+0000 7f50c306f640 1 -- start start 2024-03-06T07:18:55.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.301+0000 7f50c306f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc158fb0 0x7f50bc0c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c306f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50bc1553c0 con 0x7f50bc06c240 2024-03-06T07:18:55.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c306f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50bc1556a0 con 0x7f50bc0ab860 2024-03-06T07:18:55.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c306f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc155980 con 0x7f50bc158fb0 2024-03-06T07:18:55.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c0e0b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f50bc06c240 0x7f50bc0ab150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50754/0 (socket says 172.21.15.94:50754) 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c0e0b640 1 -- 172.21.15.94:0/3186273144 learned_addr learned my addr 172.21.15.94:0/3186273144 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c160c640 1 --2- 172.21.15.94:0/3186273144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc158fb0 0x7f50bc0c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50bb7fe640 1 -- 172.21.15.94:0/3186273144 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074134004 0 0) 0x7f50bc1553c0 con 0x7f50bc06c240 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50bb7fe640 1 -- 172.21.15.94:0/3186273144 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ac003520 con 0x7f50bc06c240 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50bb7fe640 1 -- 172.21.15.94:0/3186273144 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581192386 0 0) 0x7f50bc1556a0 con 0x7f50bc0ab860 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50bb7fe640 1 -- 172.21.15.94:0/3186273144 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50bc1553c0 con 0x7f50bc0ab860 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c160c640 1 -- 172.21.15.94:0/3186273144 >> v1:172.21.15.145:6789/0 conn(0x7f50bc0ab860 legacy=0x7f50bc1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c160c640 1 -- 172.21.15.94:0/3186273144 >> v1:172.21.15.94:6789/0 conn(0x7f50bc06c240 legacy=0x7f50bc0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.302+0000 7f50c160c640 1 -- 172.21.15.94:0/3186273144 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc155c60 con 0x7f50bc158fb0 2024-03-06T07:18:55.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.303+0000 7f50c160c640 1 --2- 172.21.15.94:0/3186273144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc158fb0 0x7f50bc0c4250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009dc0 tx=0x7f50b402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3ef3eded057c3c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.303+0000 7f50bb7fe640 1 -- 172.21.15.94:0/3186273144 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4039070 con 0x7f50bc158fb0 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.303+0000 7f50bb7fe640 1 -- 172.21.15.94:0/3186273144 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b402cc40 con 0x7f50bc158fb0 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.303+0000 7f50bb7fe640 1 -- 172.21.15.94:0/3186273144 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4034440 con 0x7f50bc158fb0 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.304+0000 7f50c306f640 1 -- 172.21.15.94:0/3186273144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc158fb0 msgr2=0x7f50bc0c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.304+0000 7f50c306f640 1 --2- 172.21.15.94:0/3186273144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc158fb0 0x7f50bc0c4250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009dc0 tx=0x7f50b402be90 comp rx=0 tx=0).stop 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.304+0000 7f50c306f640 1 -- 172.21.15.94:0/3186273144 shutdown_connections 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.304+0000 7f50c306f640 1 --2- 172.21.15.94:0/3186273144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc158fb0 0x7f50bc0c4250 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.304+0000 7f50c306f640 1 -- 172.21.15.94:0/3186273144 >> 172.21.15.94:0/3186273144 conn(0x7f50bc0c7670 msgr2=0x7f50bc0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.304+0000 7f50c306f640 1 -- 172.21.15.94:0/3186273144 shutdown_connections 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.304+0000 7f50c306f640 1 -- 172.21.15.94:0/3186273144 wait complete. 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.305+0000 7f50c306f640 1 Processor -- start 2024-03-06T07:18:55.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.305+0000 7f50c306f640 1 -- start start 2024-03-06T07:18:55.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50bc06c240 0x7f50bc0c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc0ab860 0x7f50bc140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50bc158fb0 0x7f50bc140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0c4830 con 0x7f50bc06c240 2024-03-06T07:18:55.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc156180 con 0x7f50bc158fb0 2024-03-06T07:18:55.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc06c620 con 0x7f50bc0ab860 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50bc06c240 0x7f50bc0c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50bc06c240 0x7f50bc0c06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33952/0 (socket says 172.21.15.94:33952) 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 -- 172.21.15.94:0/1241155977 learned_addr learned my addr 172.21.15.94:0/1241155977 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c160c640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50bc158fb0 0x7f50bc140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50bbfff640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc0ab860 0x7f50bc140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 -- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc0ab860 msgr2=0x7f50bc140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc0ab860 0x7f50bc140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 -- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50bc158fb0 msgr2=0x7f50bc140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50bc158fb0 0x7f50bc140dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 -- 172.21.15.94:0/1241155977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc072b80 con 0x7f50bc06c240 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c160c640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50bc158fb0 0x7f50bc140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:55.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50bbfff640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc0ab860 0x7f50bc140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c0e0b640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50bc06c240 0x7f50bc0c06e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f50b000c440 tx=0x7f50b000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50b97fa640 1 -- 172.21.15.94:0/1241155977 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0017070 con 0x7f50bc06c240 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50b97fa640 1 -- 172.21.15.94:0/1241155977 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0005ce0 con 0x7f50bc06c240 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc1eb7c0 con 0x7f50bc06c240 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50b97fa640 1 -- 172.21.15.94:0/1241155977 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0007270 con 0x7f50bc06c240 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.306+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc0728a0 con 0x7f50bc06c240 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.307+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc04f330 con 0x7f50bc06c240 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.308+0000 7f50b97fa640 1 -- 172.21.15.94:0/1241155977 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50b001a070 con 0x7f50bc06c240 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.308+0000 7f50b97fa640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50a006b4d0 0x7f50a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.308+0000 7f50b97fa640 1 -- 172.21.15.94:0/1241155977 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f50b0089690 con 0x7f50bc06c240 2024-03-06T07:18:55.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.308+0000 7f50bbfff640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50a006b4d0 0x7f50a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.309+0000 7f50bbfff640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50a006b4d0 0x7f50a006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f50bc1eb6f0 tx=0x7f50a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.312+0000 7f50b97fa640 1 -- 172.21.15.94:0/1241155977 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50b0057c10 con 0x7f50bc06c240 2024-03-06T07:18:55.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.497+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f50bc070ce0 con 0x7f50bc06c240 2024-03-06T07:18:55.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.497+0000 7f50b97fa640 1 -- 172.21.15.94:0/1241155977 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f50b005bcb0 con 0x7f50bc06c240 2024-03-06T07:18:55.499 INFO:teuthology.orchestra.run.smithi094.stderr:listed 0 entries 2024-03-06T07:18:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50a006b4d0 msgr2=0x7f50a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50a006b4d0 0x7f50a006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f50bc1eb6f0 tx=0x7f50a8009040 comp rx=0 tx=0).stop 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50bc06c240 msgr2=0x7f50bc0c06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50bc06c240 0x7f50bc0c06e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f50b000c440 tx=0x7f50b000c900 comp rx=0 tx=0).stop 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 shutdown_connections 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50a006b4d0 0x7f50a006d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50bc158fb0 0x7f50bc140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50bc0ab860 0x7f50bc140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 --2- 172.21.15.94:0/1241155977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50bc06c240 0x7f50bc0c06e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.499+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 >> 172.21.15.94:0/1241155977 conn(0x7f50bc0c7670 msgr2=0x7f50bc0c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.500+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 shutdown_connections 2024-03-06T07:18:55.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.500+0000 7f50c306f640 1 -- 172.21.15.94:0/1241155977 wait complete. 2024-03-06T07:18:55.513 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:18:55.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:55.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-06T07:18:55.542 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:18:55.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:55.543 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:18:55.598 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:18:55.598 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:55.613 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:18:55.614 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:55.669 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:18:55.669 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:18:55.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:55.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.824+0000 7f308e037640 1 Processor -- start 2024-03-06T07:18:55.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.824+0000 7f308e037640 1 -- start start 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.824+0000 7f308e037640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3088150580 0x7f308806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.824+0000 7f308e037640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f30880c1320 0x7f308806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.824+0000 7f308e037640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3088153a60 con 0x7f30880c6000 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.824+0000 7f308e037640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3088153d40 con 0x7f3088150580 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.824+0000 7f308e037640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308806c2c0 con 0x7f30880c1320 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f30877fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3088150580 0x7f308806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f30877fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3088150580 0x7f308806dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33966/0 (socket says 172.21.15.94:33966) 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f30877fe640 1 -- 172.21.15.94:0/2971197041 learned_addr learned my addr 172.21.15.94:0/2971197041 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f307ffff640 1 --2- 172.21.15.94:0/2971197041 >> v2:172.21.15.145:3300/0 conn(0x7f30880c1320 0x7f308806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f30877fe640 1 -- 172.21.15.94:0/2971197041 >> v2:172.21.15.145:3300/0 conn(0x7f30880c1320 msgr2=0x7f308806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f30877fe640 1 --2- 172.21.15.94:0/2971197041 >> v2:172.21.15.145:3300/0 conn(0x7f30880c1320 0x7f308806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f30877fe640 1 -- 172.21.15.94:0/2971197041 >> v1:172.21.15.145:6789/0 conn(0x7f30880c6000 legacy=0x7f30881566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f30877fe640 1 -- 172.21.15.94:0/2971197041 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308806c5a0 con 0x7f3088150580 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.825+0000 7f307ffff640 1 --2- 172.21.15.94:0/2971197041 >> v2:172.21.15.145:3300/0 conn(0x7f30880c1320 0x7f308806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f30877fe640 1 --2- 172.21.15.94:0/2971197041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3088150580 0x7f308806dd10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f30880c7730 tx=0x7f307802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51842c3032d2b016 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f3086ffd640 1 -- 172.21.15.94:0/2971197041 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078035020 con 0x7f3088150580 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f3086ffd640 1 -- 172.21.15.94:0/2971197041 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078031870 con 0x7f3088150580 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f3086ffd640 1 -- 172.21.15.94:0/2971197041 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078031c40 con 0x7f3088150580 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 -- 172.21.15.94:0/2971197041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3088150580 msgr2=0x7f308806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 --2- 172.21.15.94:0/2971197041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3088150580 0x7f308806dd10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f30880c7730 tx=0x7f307802b910 comp rx=0 tx=0).stop 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 -- 172.21.15.94:0/2971197041 shutdown_connections 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 --2- 172.21.15.94:0/2971197041 >> v2:172.21.15.145:3300/0 conn(0x7f30880c1320 0x7f308806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 --2- 172.21.15.94:0/2971197041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3088150580 0x7f308806dd10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 -- 172.21.15.94:0/2971197041 >> 172.21.15.94:0/2971197041 conn(0x7f30880c6710 msgr2=0x7f30880c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 -- 172.21.15.94:0/2971197041 shutdown_connections 2024-03-06T07:18:55.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.826+0000 7f308e037640 1 -- 172.21.15.94:0/2971197041 wait complete. 2024-03-06T07:18:55.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.827+0000 7f308e037640 1 Processor -- start 2024-03-06T07:18:55.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f308e037640 1 -- start start 2024-03-06T07:18:55.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f308e037640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30880c1320 0x7f30880ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f308e037640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30880c6000 0x7f30880bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f308e037640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3088150580 0x7f3088145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f308e037640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30880c0dc0 con 0x7f30880c1320 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f308e037640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30880c0c40 con 0x7f3088150580 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f308e037640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3088156dc0 con 0x7f30880c6000 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f30877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30880c1320 0x7f30880ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f30877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30880c1320 0x7f30880ab4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33970/0 (socket says 172.21.15.94:33970) 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f30877fe640 1 -- 172.21.15.94:0/2583247564 learned_addr learned my addr 172.21.15.94:0/2583247564 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f3087fff640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3088150580 0x7f3088145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.828+0000 7f307ffff640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30880c6000 0x7f30880bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f30877fe640 1 -- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30880c6000 msgr2=0x7f30880bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f30877fe640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30880c6000 0x7f30880bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f30877fe640 1 -- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3088150580 msgr2=0x7f3088145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f30877fe640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3088150580 0x7f3088145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f30877fe640 1 -- 172.21.15.94:0/2583247564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30880a99f0 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f30877fe640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30880c1320 0x7f30880ab4f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f307802b950 tx=0x7f30780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f3084ff9640 1 -- 172.21.15.94:0/2583247564 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078040070 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f3084ff9640 1 -- 172.21.15.94:0/2583247564 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078030d00 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30880a9cd0 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f3084ff9640 1 -- 172.21.15.94:0/2583247564 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078035040 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.829+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30880c02f0 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.830+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30881eac30 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.830+0000 7f3084ff9640 1 -- 172.21.15.94:0/2583247564 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f307804b060 con 0x7f30880c1320 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.831+0000 7f3084ff9640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f306806b500 0x7f306806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:55.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.831+0000 7f3084ff9640 1 -- 172.21.15.94:0/2583247564 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f30780ac0b0 con 0x7f30880c1320 2024-03-06T07:18:55.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.831+0000 7f307ffff640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f306806b500 0x7f306806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:55.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.831+0000 7f307ffff640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f306806b500 0x7f306806d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3088073440 tx=0x7f3070008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:55.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:55.835+0000 7f3084ff9640 1 -- 172.21.15.94:0/2583247564 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f307807a630 con 0x7f30880c1320 2024-03-06T07:18:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.007+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3088153430 con 0x7f30880c1320 2024-03-06T07:18:56.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.008+0000 7f3084ff9640 1 -- 172.21.15.94:0/2583247564 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18514 (secure 0 0 0) 0x7f307807e6d0 con 0x7f30880c1320 2024-03-06T07:18:56.011 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:56.011 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":24,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:13.978356+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:56.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f306806b500 msgr2=0x7f306806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f306806b500 0x7f306806d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3088073440 tx=0x7f3070008040 comp rx=0 tx=0).stop 2024-03-06T07:18:56.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30880c1320 msgr2=0x7f30880ab4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30880c1320 0x7f30880ab4f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f307802b950 tx=0x7f30780046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 shutdown_connections 2024-03-06T07:18:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f306806b500 0x7f306806d9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3088150580 0x7f3088145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30880c6000 0x7f30880bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 --2- 172.21.15.94:0/2583247564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30880c1320 0x7f30880ab4f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 >> 172.21.15.94:0/2583247564 conn(0x7f30880c6710 msgr2=0x7f308806dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 shutdown_connections 2024-03-06T07:18:56.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.010+0000 7f308e037640 1 -- 172.21.15.94:0/2583247564 wait complete. 2024-03-06T07:18:56.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:56.024 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:56.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.125+0000 7fe5ab5e9640 1 Processor -- start 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.125+0000 7fe5ab5e9640 1 -- start start 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5ab5e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5a40ab170 0x7fe5a40c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5ab5e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40c3ef0 con 0x7fe5a406c900 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5ab5e9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a41556d0 con 0x7fe5a40c5990 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5ab5e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a41559b0 con 0x7fe5a40ab170 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5a9b86640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5a40ab170 0x7fe5a40c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5a9b86640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5a40ab170 0x7fe5a40c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33982/0 (socket says 172.21.15.94:33982) 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5a8b84640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe5a406c900 0x7fe5a40a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50760/0 (socket says 172.21.15.94:50760) 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5a9b86640 1 -- 172.21.15.94:0/2304502881 learned_addr learned my addr 172.21.15.94:0/2304502881 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5a9b86640 1 -- 172.21.15.94:0/2304502881 >> v1:172.21.15.145:6789/0 conn(0x7fe5a40c5990 legacy=0x7fe5a406c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5a9b86640 1 -- 172.21.15.94:0/2304502881 >> v1:172.21.15.94:6789/0 conn(0x7fe5a406c900 legacy=0x7fe5a40a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe59bfff640 1 -- 172.21.15.94:0/2304502881 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561597089 0 0) 0x7fe5a40c3ef0 con 0x7fe5a406c900 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.126+0000 7fe5a9b86640 1 -- 172.21.15.94:0/2304502881 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4155c90 con 0x7fe5a40ab170 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.127+0000 7fe5a9b86640 1 --2- 172.21.15.94:0/2304502881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a40ab170 0x7fe5a40c0ab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00d180 tx=0x7fe59c02e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b073140a6487ecb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.127+0000 7fe59bfff640 1 -- 172.21.15.94:0/2304502881 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00b0d0 con 0x7fe5a40ab170 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.127+0000 7fe59bfff640 1 -- 172.21.15.94:0/2304502881 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c00b270 con 0x7fe5a40ab170 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.127+0000 7fe59bfff640 1 -- 172.21.15.94:0/2304502881 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c009600 con 0x7fe5a40ab170 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.128+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2304502881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a40ab170 msgr2=0x7fe5a40c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.128+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2304502881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a40ab170 0x7fe5a40c0ab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00d180 tx=0x7fe59c02e1f0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.128+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2304502881 shutdown_connections 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.128+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2304502881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a40ab170 0x7fe5a40c0ab0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.128+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2304502881 >> 172.21.15.94:0/2304502881 conn(0x7fe5a40c6710 msgr2=0x7fe5a40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.128+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2304502881 shutdown_connections 2024-03-06T07:18:56.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.128+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2304502881 wait complete. 2024-03-06T07:18:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.129+0000 7fe5ab5e9640 1 Processor -- start 2024-03-06T07:18:56.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.129+0000 7fe5ab5e9640 1 -- start start 2024-03-06T07:18:56.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.129+0000 7fe5ab5e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a406c900 0x7fe5a40c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.129+0000 7fe5ab5e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a40ab170 0x7fe5a41e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.129+0000 7fe5ab5e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a40c5990 0x7fe5a41ea240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.129+0000 7fe5ab5e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4156610 con 0x7fe5a406c900 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5ab5e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4156490 con 0x7fe5a40ab170 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5ab5e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40c3af0 con 0x7fe5a40c5990 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a406c900 0x7fe5a40c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a406c900 0x7fe5a40c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33990/0 (socket says 172.21.15.94:33990) 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 -- 172.21.15.94:0/2732695027 learned_addr learned my addr 172.21.15.94:0/2732695027 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9b86640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a40c5990 0x7fe5a41ea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a8b84640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a40ab170 0x7fe5a41e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 -- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a40c5990 msgr2=0x7fe5a41ea240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a40c5990 0x7fe5a41ea240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 -- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a40ab170 msgr2=0x7fe5a41e9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a40ab170 0x7fe5a41e9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 -- 172.21.15.94:0/2732695027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a414a270 con 0x7fe5a406c900 2024-03-06T07:18:56.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9b86640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a40c5990 0x7fe5a41ea240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5a9385640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a406c900 0x7fe5a40c2c20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000c400 tx=0x7fe5a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe599ffb640 1 -- 172.21.15.94:0/2732695027 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0017070 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe599ffb640 1 -- 172.21.15.94:0/2732695027 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a00052d0 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a414fa40 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe599ffb640 1 -- 172.21.15.94:0/2732695027 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0003830 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.130+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a414be70 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.131+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a404f330 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.132+0000 7fe599ffb640 1 -- 172.21.15.94:0/2732695027 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5a001a070 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.132+0000 7fe599ffb640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe58806b500 0x7fe58806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.132+0000 7fe599ffb640 1 -- 172.21.15.94:0/2732695027 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe5a0089830 con 0x7fe5a406c900 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.132+0000 7fe5a8b84640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe58806b500 0x7fe58806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.133+0000 7fe5a8b84640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe58806b500 0x7fe58806d9c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5a41533a0 tx=0x7fe594007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.136+0000 7fe599ffb640 1 -- 172.21.15.94:0/2732695027 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5a0058400 con 0x7fe5a406c900 2024-03-06T07:18:56.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.307+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5a40c1740 con 0x7fe5a406c900 2024-03-06T07:18:56.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.308+0000 7fe599ffb640 1 -- 172.21.15.94:0/2732695027 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5a0057d00 con 0x7fe5a406c900 2024-03-06T07:18:56.310 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe58806b500 msgr2=0x7fe58806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe58806b500 0x7fe58806d9c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5a41533a0 tx=0x7fe594007b90 comp rx=0 tx=0).stop 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a406c900 msgr2=0x7fe5a40c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a406c900 0x7fe5a40c2c20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000c400 tx=0x7fe5a000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 shutdown_connections 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe58806b500 0x7fe58806d9c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a40c5990 0x7fe5a41ea240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a40ab170 0x7fe5a41e9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 --2- 172.21.15.94:0/2732695027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5a406c900 0x7fe5a40c2c20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 >> 172.21.15.94:0/2732695027 conn(0x7fe5a40c6710 msgr2=0x7fe5a406c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 shutdown_connections 2024-03-06T07:18:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.310+0000 7fe5ab5e9640 1 -- 172.21.15.94:0/2732695027 wait complete. 2024-03-06T07:18:56.322 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.424+0000 7f80474df640 1 Processor -- start 2024-03-06T07:18:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.424+0000 7f80474df640 1 -- start start 2024-03-06T07:18:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f80474df640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f804015c8d0 0x7f804015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f80474df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804006b440 con 0x7f804014d350 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f80474df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80400c1c90 con 0x7f804006afc0 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f80474df640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80400ac750 con 0x7f804015c8d0 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f804527b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f804014d350 0x7f80400acbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50774/0 (socket says 172.21.15.94:50774) 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f804527b640 1 -- 172.21.15.94:0/1641562970 learned_addr learned my addr 172.21.15.94:0/1641562970 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8045a7c640 1 --2- 172.21.15.94:0/1641562970 >> v2:172.21.15.145:3300/0 conn(0x7f804015c8d0 0x7f804015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8037fff640 1 -- 172.21.15.94:0/1641562970 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915692731 0 0) 0x7f803c001110 con 0x7f804014d350 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8037fff640 1 -- 172.21.15.94:0/1641562970 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f802c003520 con 0x7f804014d350 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8037fff640 1 -- 172.21.15.94:0/1641562970 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729953895 0 0) 0x7f80400c1c90 con 0x7f804006afc0 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8037fff640 1 -- 172.21.15.94:0/1641562970 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c001110 con 0x7f804006afc0 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8045a7c640 1 -- 172.21.15.94:0/1641562970 >> v1:172.21.15.145:6789/0 conn(0x7f804006afc0 legacy=0x7f80400ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8045a7c640 1 -- 172.21.15.94:0/1641562970 >> v1:172.21.15.94:6789/0 conn(0x7f804014d350 legacy=0x7f80400acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.425+0000 7f8045a7c640 1 -- 172.21.15.94:0/1641562970 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80400ac470 con 0x7f804015c8d0 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.426+0000 7f8045a7c640 1 --2- 172.21.15.94:0/1641562970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804015c8d0 0x7f804015ecc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8038009d90 tx=0x7f803802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e36c11b20a70039 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.426+0000 7f8037fff640 1 -- 172.21.15.94:0/1641562970 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038039070 con 0x7f804015c8d0 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.426+0000 7f8037fff640 1 -- 172.21.15.94:0/1641562970 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803802cc60 con 0x7f804015c8d0 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.426+0000 7f8037fff640 1 -- 172.21.15.94:0/1641562970 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038034640 con 0x7f804015c8d0 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.427+0000 7f80474df640 1 -- 172.21.15.94:0/1641562970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804015c8d0 msgr2=0x7f804015ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.427+0000 7f80474df640 1 --2- 172.21.15.94:0/1641562970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804015c8d0 0x7f804015ecc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8038009d90 tx=0x7f803802beb0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.427+0000 7f80474df640 1 -- 172.21.15.94:0/1641562970 shutdown_connections 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.427+0000 7f80474df640 1 --2- 172.21.15.94:0/1641562970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804015c8d0 0x7f804015ecc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.427+0000 7f80474df640 1 -- 172.21.15.94:0/1641562970 >> 172.21.15.94:0/1641562970 conn(0x7f80400c3b40 msgr2=0x7f80400c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.427+0000 7f80474df640 1 -- 172.21.15.94:0/1641562970 shutdown_connections 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.427+0000 7f80474df640 1 -- 172.21.15.94:0/1641562970 wait complete. 2024-03-06T07:18:56.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 Processor -- start 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 -- start start 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804006afc0 0x7f8040149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f804014d350 0x7f80401e8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f804015c8d0 0x7f80401e9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8040153c00 con 0x7f804014d350 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80400c1a40 con 0x7f804006afc0 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f80474df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80400aa160 con 0x7f804015c8d0 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.428+0000 7f8044a7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f804014d350 0x7f80401e8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f804014d350 0x7f80401e8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33996/0 (socket says 172.21.15.94:33996) 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 -- 172.21.15.94:0/2827492361 learned_addr learned my addr 172.21.15.94:0/2827492361 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f804527b640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804006afc0 0x7f8040149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8045a7c640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f804015c8d0 0x7f80401e9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 -- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f804015c8d0 msgr2=0x7f80401e9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f804015c8d0 0x7f80401e9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 -- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804006afc0 msgr2=0x7f8040149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804006afc0 0x7f8040149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 -- 172.21.15.94:0/2827492361 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8040150590 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8044a7a640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f804014d350 0x7f80401e8c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f803000c530 tx=0x7f803000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8035ffb640 1 -- 172.21.15.94:0/2827492361 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80300052d0 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8035ffb640 1 -- 172.21.15.94:0/2827492361 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8030005490 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f8035ffb640 1 -- 172.21.15.94:0/2827492361 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8030012870 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8040150870 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.429+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8040156db0 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.430+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804004f330 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.431+0000 7f8035ffb640 1 -- 172.21.15.94:0/2827492361 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8030012a10 con 0x7f804014d350 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.431+0000 7f8035ffb640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f802406b4d0 0x7f802406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.431+0000 7f8035ffb640 1 -- 172.21.15.94:0/2827492361 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f803008a0c0 con 0x7f804014d350 2024-03-06T07:18:56.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.431+0000 7f804527b640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f802406b4d0 0x7f802406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.431+0000 7f804527b640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f802406b4d0 0x7f802406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f803c0013b0 tx=0x7f803c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.435+0000 7f8035ffb640 1 -- 172.21.15.94:0/2827492361 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80300585c0 con 0x7f804014d350 2024-03-06T07:18:56.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.609+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80400c49e0 con 0x7f804014d350 2024-03-06T07:18:56.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.609+0000 7f8035ffb640 1 -- 172.21.15.94:0/2827492361 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f803005c660 con 0x7f804014d350 2024-03-06T07:18:56.611 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:56.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f802406b4d0 msgr2=0x7f802406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f802406b4d0 0x7f802406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f803c0013b0 tx=0x7f803c007d30 comp rx=0 tx=0).stop 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f804014d350 msgr2=0x7f80401e8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f804014d350 0x7f80401e8c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f803000c530 tx=0x7f803000c9f0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 shutdown_connections 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f802406b4d0 0x7f802406d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f804015c8d0 0x7f80401e9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f804014d350 0x7f80401e8c40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 --2- 172.21.15.94:0/2827492361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f804006afc0 0x7f8040149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 >> 172.21.15.94:0/2827492361 conn(0x7f80400c3b40 msgr2=0x7f80400c1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 shutdown_connections 2024-03-06T07:18:56.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.611+0000 7f80474df640 1 -- 172.21.15.94:0/2827492361 wait complete. 2024-03-06T07:18:56.624 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:56.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.728+0000 7fb58202a640 1 Processor -- start 2024-03-06T07:18:56.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.728+0000 7fb58202a640 1 -- start start 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.728+0000 7fb58202a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0b0070 0x7fb57c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.728+0000 7fb58202a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb57c0c5380 0x7fb57c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.728+0000 7fb58202a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c0b0450 con 0x7fb57c06ca70 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.728+0000 7fb58202a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57c06c020 con 0x7fb57c0c5380 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.728+0000 7fb58202a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0c77f0 con 0x7fb57c0b0070 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb57c06ca70 0x7fb57c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50784/0 (socket says 172.21.15.94:50784) 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57affd640 1 -- 172.21.15.94:0/3461288899 learned_addr learned my addr 172.21.15.94:0/3461288899 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57b7fe640 1 --2- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0b0070 0x7fb57c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57bfff640 1 --2- 172.21.15.94:0/3461288899 >> v2:172.21.15.145:3300/0 conn(0x7fb57c0c5380 0x7fb57c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57a7fc640 1 -- 172.21.15.94:0/3461288899 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866573231 0 0) 0x7fb570001110 con 0x7fb57c06ca70 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57a7fc640 1 -- 172.21.15.94:0/3461288899 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564003520 con 0x7fb57c06ca70 2024-03-06T07:18:56.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57bfff640 1 -- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0b0070 msgr2=0x7fb57c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57bfff640 1 --2- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0b0070 0x7fb57c155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57bfff640 1 -- 172.21.15.94:0/3461288899 >> v1:172.21.15.94:6789/0 conn(0x7fb57c06ca70 legacy=0x7fb57c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57bfff640 1 -- 172.21.15.94:0/3461288899 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c0c7ad0 con 0x7fb57c0c5380 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.729+0000 7fb57b7fe640 1 --2- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0b0070 0x7fb57c155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb57bfff640 1 --2- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0c5380 0x7fb57c0c57d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1bd38d734cc3d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb57a7fc640 1 -- 172.21.15.94:0/3461288899 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574039070 con 0x7fb57c0c5380 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb57a7fc640 1 -- 172.21.15.94:0/3461288899 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57402cbc0 con 0x7fb57c0c5380 2024-03-06T07:18:56.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb57a7fc640 1 -- 172.21.15.94:0/3461288899 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574034440 con 0x7fb57c0c5380 2024-03-06T07:18:56.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb58202a640 1 -- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0c5380 msgr2=0x7fb57c0c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb58202a640 1 --2- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0c5380 0x7fb57c0c57d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57402be10 comp rx=0 tx=0).stop 2024-03-06T07:18:56.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb58202a640 1 -- 172.21.15.94:0/3461288899 shutdown_connections 2024-03-06T07:18:56.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb58202a640 1 --2- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0c5380 0x7fb57c0c57d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb58202a640 1 --2- 172.21.15.94:0/3461288899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0b0070 0x7fb57c155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.730+0000 7fb58202a640 1 -- 172.21.15.94:0/3461288899 >> 172.21.15.94:0/3461288899 conn(0x7fb57c0ac590 msgr2=0x7fb57c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.731+0000 7fb58202a640 1 -- 172.21.15.94:0/3461288899 shutdown_connections 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.731+0000 7fb58202a640 1 -- 172.21.15.94:0/3461288899 wait complete. 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 Processor -- start 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 -- start start 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb57c06ca70 0x7fb57c157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0b0070 0x7fb57c1ee650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0c5380 0x7fb57c1eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c06cef0 con 0x7fb57c06ca70 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c06c550 con 0x7fb57c0b0070 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb58202a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c06da80 con 0x7fb57c0c5380 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb57c06ca70 0x7fb57c157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb57c06ca70 0x7fb57c157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34012/0 (socket says 172.21.15.94:34012) 2024-03-06T07:18:56.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.732+0000 7fb57b7fe640 1 -- 172.21.15.94:0/3017014787 learned_addr learned my addr 172.21.15.94:0/3017014787 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57bfff640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0c5380 0x7fb57c1eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57affd640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0b0070 0x7fb57c1ee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57b7fe640 1 -- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0c5380 msgr2=0x7fb57c1eaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57b7fe640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0c5380 0x7fb57c1eaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57b7fe640 1 -- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0b0070 msgr2=0x7fb57c1ee650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57b7fe640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0b0070 0x7fb57c1ee650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57b7fe640 1 -- 172.21.15.94:0/3017014787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c0c4dc0 con 0x7fb57c06ca70 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57affd640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0b0070 0x7fb57c1ee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb57b7fe640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb57c06ca70 0x7fb57c157c90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00ebe0 tx=0x7fb56c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb563fff640 1 -- 172.21.15.94:0/3017014787 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c017070 con 0x7fb57c06ca70 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb563fff640 1 -- 172.21.15.94:0/3017014787 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c00cc30 con 0x7fb57c06ca70 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c1550a0 con 0x7fb57c06ca70 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb563fff640 1 -- 172.21.15.94:0/3017014787 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c013470 con 0x7fb57c06ca70 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.733+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c0c4ae0 con 0x7fb57c06ca70 2024-03-06T07:18:56.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.734+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c04f330 con 0x7fb57c06ca70 2024-03-06T07:18:56.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.735+0000 7fb563fff640 1 -- 172.21.15.94:0/3017014787 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb56c01a070 con 0x7fb57c06ca70 2024-03-06T07:18:56.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.735+0000 7fb563fff640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb56806b4d0 0x7fb56806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:56.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.735+0000 7fb563fff640 1 -- 172.21.15.94:0/3017014787 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb56c08a390 con 0x7fb57c06ca70 2024-03-06T07:18:56.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.735+0000 7fb57affd640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb56806b4d0 0x7fb56806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:56.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.735+0000 7fb57affd640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb56806b4d0 0x7fb56806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb570007fd0 tx=0x7fb570009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:56.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.739+0000 7fb563fff640 1 -- 172.21.15.94:0/3017014787 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb57c0c4dc0 con 0x7fb57c06ca70 2024-03-06T07:18:56.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.909+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb57c06d860 con 0x7fb57c06ca70 2024-03-06T07:18:56.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.910+0000 7fb563fff640 1 -- 172.21.15.94:0/3017014787 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb56c058910 con 0x7fb57c06ca70 2024-03-06T07:18:56.912 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb56806b4d0 msgr2=0x7fb56806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb56806b4d0 0x7fb56806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb570007fd0 tx=0x7fb570009040 comp rx=0 tx=0).stop 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb57c06ca70 msgr2=0x7fb57c157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb57c06ca70 0x7fb57c157c90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00ebe0 tx=0x7fb56c00c520 comp rx=0 tx=0).stop 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 shutdown_connections 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb56806b4d0 0x7fb56806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb57c0c5380 0x7fb57c1eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb57c0b0070 0x7fb57c1ee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 --2- 172.21.15.94:0/3017014787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb57c06ca70 0x7fb57c157c90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.911+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 >> 172.21.15.94:0/3017014787 conn(0x7fb57c0ac590 msgr2=0x7fb57c0aa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.912+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 shutdown_connections 2024-03-06T07:18:56.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:56.912+0000 7fb58202a640 1 -- 172.21.15.94:0/3017014787 wait complete. 2024-03-06T07:18:56.924 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:57.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.026+0000 7fe068ce1640 1 Processor -- start 2024-03-06T07:18:57.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.026+0000 7fe068ce1640 1 -- start start 2024-03-06T07:18:57.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe068ce1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe06406d800 0x7fe064152d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe068ce1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe0640a5eb0 0x7fe0640c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe068ce1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064155f70 con 0x7fe064152920 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe068ce1640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe064156250 con 0x7fe06406d800 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe068ce1640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0640c49a0 con 0x7fe0640a5eb0 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe061d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe06406d800 0x7fe064152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe061d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe06406d800 0x7fe064152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34026/0 (socket says 172.21.15.94:34026) 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe061d9b640 1 -- 172.21.15.94:0/1681174428 learned_addr learned my addr 172.21.15.94:0/1681174428 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe06259c640 1 --1- 172.21.15.94:0/1681174428 >> v1:172.21.15.145:6789/0 conn(0x7fe064152920 0x7fe0640bfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:58560/0 (socket says 172.21.15.94:58560) 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe062d9d640 1 --2- 172.21.15.94:0/1681174428 >> v2:172.21.15.145:3300/0 conn(0x7fe0640a5eb0 0x7fe0640c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe061d9b640 1 -- 172.21.15.94:0/1681174428 >> v2:172.21.15.145:3300/0 conn(0x7fe0640a5eb0 msgr2=0x7fe0640c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe061d9b640 1 --2- 172.21.15.94:0/1681174428 >> v2:172.21.15.145:3300/0 conn(0x7fe0640a5eb0 0x7fe0640c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe061d9b640 1 -- 172.21.15.94:0/1681174428 >> v1:172.21.15.145:6789/0 conn(0x7fe064152920 legacy=0x7fe0640bfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.027+0000 7fe061d9b640 1 -- 172.21.15.94:0/1681174428 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640c4c80 con 0x7fe06406d800 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.028+0000 7fe061d9b640 1 --2- 172.21.15.94:0/1681174428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe06406d800 0x7fe064152d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe064068270 tx=0x7fe05802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=263ec66eadb091f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.028+0000 7fe06159a640 1 -- 172.21.15.94:0/1681174428 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058034070 con 0x7fe06406d800 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.028+0000 7fe06159a640 1 -- 172.21.15.94:0/1681174428 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe058030860 con 0x7fe06406d800 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.028+0000 7fe06159a640 1 -- 172.21.15.94:0/1681174428 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058030c30 con 0x7fe06406d800 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 -- 172.21.15.94:0/1681174428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe06406d800 msgr2=0x7fe064152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 --2- 172.21.15.94:0/1681174428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe06406d800 0x7fe064152d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe064068270 tx=0x7fe05802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 -- 172.21.15.94:0/1681174428 shutdown_connections 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 --2- 172.21.15.94:0/1681174428 >> v2:172.21.15.145:3300/0 conn(0x7fe0640a5eb0 0x7fe0640c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 --2- 172.21.15.94:0/1681174428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe06406d800 0x7fe064152d00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 -- 172.21.15.94:0/1681174428 >> 172.21.15.94:0/1681174428 conn(0x7fe06406cb60 msgr2=0x7fe0640a86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:57.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 -- 172.21.15.94:0/1681174428 shutdown_connections 2024-03-06T07:18:57.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.029+0000 7fe068ce1640 1 -- 172.21.15.94:0/1681174428 wait complete. 2024-03-06T07:18:57.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.030+0000 7fe068ce1640 1 Processor -- start 2024-03-06T07:18:57.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.030+0000 7fe068ce1640 1 -- start start 2024-03-06T07:18:57.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe068ce1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe06406d800 0x7fe0640ab050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe068ce1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0640a5eb0 0x7fe0641ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe068ce1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe064152920 0x7fe0641ecba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe068ce1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0640735b0 con 0x7fe064152920 2024-03-06T07:18:57.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe068ce1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe064073430 con 0x7fe0640a5eb0 2024-03-06T07:18:57.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe068ce1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06406e3f0 con 0x7fe06406d800 2024-03-06T07:18:57.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe064152920 0x7fe0641ecba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe064152920 0x7fe0641ecba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34040/0 (socket says 172.21.15.94:34040) 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 -- 172.21.15.94:0/895372621 learned_addr learned my addr 172.21.15.94:0/895372621 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe06259c640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe06406d800 0x7fe0640ab050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe061d9b640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0640a5eb0 0x7fe0641ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 -- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe06406d800 msgr2=0x7fe0640ab050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe06406d800 0x7fe0640ab050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 -- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0640a5eb0 msgr2=0x7fe0641ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0640a5eb0 0x7fe0641ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 -- 172.21.15.94:0/895372621 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640a6760 con 0x7fe064152920 2024-03-06T07:18:57.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe062d9d640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe064152920 0x7fe0641ecba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009dd0 tx=0x7fe05c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.031+0000 7fe052ffd640 1 -- 172.21.15.94:0/895372621 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c018020 con 0x7fe064152920 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.032+0000 7fe052ffd640 1 -- 172.21.15.94:0/895372621 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c014480 con 0x7fe064152920 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.032+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0641ee760 con 0x7fe064152920 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.032+0000 7fe052ffd640 1 -- 172.21.15.94:0/895372621 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c014870 con 0x7fe064152920 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.032+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0641eed20 con 0x7fe064152920 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.033+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06404f330 con 0x7fe064152920 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.033+0000 7fe052ffd640 1 -- 172.21.15.94:0/895372621 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe05c005d10 con 0x7fe064152920 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.034+0000 7fe052ffd640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe04406b4d0 0x7fe04406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.034+0000 7fe052ffd640 1 -- 172.21.15.94:0/895372621 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe05c08af10 con 0x7fe064152920 2024-03-06T07:18:57.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.034+0000 7fe06259c640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe04406b4d0 0x7fe04406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.034+0000 7fe06259c640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe04406b4d0 0x7fe04406d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe054007fd0 tx=0x7fe054007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.038+0000 7fe052ffd640 1 -- 172.21.15.94:0/895372621 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe05c022070 con 0x7fe064152920 2024-03-06T07:18:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.214+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe064071710 con 0x7fe064152920 2024-03-06T07:18:57.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.214+0000 7fe052ffd640 1 -- 172.21.15.94:0/895372621 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe05c010040 con 0x7fe064152920 2024-03-06T07:18:57.216 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe04406b4d0 msgr2=0x7fe04406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe04406b4d0 0x7fe04406d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe054007fd0 tx=0x7fe054007e80 comp rx=0 tx=0).stop 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe064152920 msgr2=0x7fe0641ecba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe064152920 0x7fe0641ecba0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009dd0 tx=0x7fe05c00c490 comp rx=0 tx=0).stop 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 shutdown_connections 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe04406b4d0 0x7fe04406d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe064152920 0x7fe0641ecba0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0640a5eb0 0x7fe0641ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 --2- 172.21.15.94:0/895372621 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe06406d800 0x7fe0640ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 >> 172.21.15.94:0/895372621 conn(0x7fe06406cb60 msgr2=0x7fe0640a7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 shutdown_connections 2024-03-06T07:18:57.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.216+0000 7fe068ce1640 1 -- 172.21.15.94:0/895372621 wait complete. 2024-03-06T07:18:57.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:18:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.328+0000 7f698ffd7640 1 Processor -- start 2024-03-06T07:18:57.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.329+0000 7f698ffd7640 1 -- start start 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.329+0000 7f698ffd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6988152740 0x7f69880c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.329+0000 7f698ffd7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6988158720 0x7f6988158b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.329+0000 7f698ffd7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6988155150 con 0x7f69880c27b0 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.329+0000 7f698ffd7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6988155d00 con 0x7f6988158720 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.329+0000 7f698ffd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69881553d0 con 0x7f6988152740 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698d572640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f69880c27b0 0x7f698806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50792/0 (socket says 172.21.15.94:50792) 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698d572640 1 -- 172.21.15.94:0/347518023 learned_addr learned my addr 172.21.15.94:0/347518023 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698dd73640 1 --2- 172.21.15.94:0/347518023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6988152740 0x7f69880c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698e574640 1 --2- 172.21.15.94:0/347518023 >> v2:172.21.15.145:3300/0 conn(0x7f6988158720 0x7f6988158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698cd71640 1 -- 172.21.15.94:0/347518023 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240392695 0 0) 0x7f6978001110 con 0x7f69880c27b0 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698cd71640 1 -- 172.21.15.94:0/347518023 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6974003520 con 0x7f69880c27b0 2024-03-06T07:18:57.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698dd73640 1 -- 172.21.15.94:0/347518023 >> v2:172.21.15.145:3300/0 conn(0x7f6988158720 msgr2=0x7f6988158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:57.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698dd73640 1 --2- 172.21.15.94:0/347518023 >> v2:172.21.15.145:3300/0 conn(0x7f6988158720 0x7f6988158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698dd73640 1 -- 172.21.15.94:0/347518023 >> v1:172.21.15.94:6789/0 conn(0x7f69880c27b0 legacy=0x7f698806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.330+0000 7f698dd73640 1 -- 172.21.15.94:0/347518023 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69881556b0 con 0x7f6988152740 2024-03-06T07:18:57.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698dd73640 1 --2- 172.21.15.94:0/347518023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6988152740 0x7f69880c20b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6980009d90 tx=0x7f698002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78a1ade59aba78e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698cd71640 1 -- 172.21.15.94:0/347518023 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6980039070 con 0x7f6988152740 2024-03-06T07:18:57.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698cd71640 1 -- 172.21.15.94:0/347518023 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698002cbc0 con 0x7f6988152740 2024-03-06T07:18:57.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698cd71640 1 -- 172.21.15.94:0/347518023 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6980034440 con 0x7f6988152740 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698ffd7640 1 -- 172.21.15.94:0/347518023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6988152740 msgr2=0x7f69880c20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698ffd7640 1 --2- 172.21.15.94:0/347518023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6988152740 0x7f69880c20b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6980009d90 tx=0x7f698002be10 comp rx=0 tx=0).stop 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698ffd7640 1 -- 172.21.15.94:0/347518023 shutdown_connections 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698ffd7640 1 --2- 172.21.15.94:0/347518023 >> v2:172.21.15.145:3300/0 conn(0x7f6988158720 0x7f6988158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698ffd7640 1 --2- 172.21.15.94:0/347518023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6988152740 0x7f69880c20b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.331+0000 7f698ffd7640 1 -- 172.21.15.94:0/347518023 >> 172.21.15.94:0/347518023 conn(0x7f69880c41b0 msgr2=0x7f69880c4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.332+0000 7f698ffd7640 1 -- 172.21.15.94:0/347518023 shutdown_connections 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.332+0000 7f698ffd7640 1 -- 172.21.15.94:0/347518023 wait complete. 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 Processor -- start 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 -- start start 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69880c27b0 0x7f69880a8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6988152740 0x7f698814b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6988158720 0x7f698814bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6988152b20 con 0x7f6988152740 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698806be80 con 0x7f6988158720 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698ffd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698806b6d0 con 0x7f69880c27b0 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698d572640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6988152740 0x7f698814b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698d572640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6988152740 0x7f698814b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34056/0 (socket says 172.21.15.94:34056) 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698d572640 1 -- 172.21.15.94:0/2108594110 learned_addr learned my addr 172.21.15.94:0/2108594110 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698dd73640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69880c27b0 0x7f69880a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.333+0000 7f698e574640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6988158720 0x7f698814bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698d572640 1 -- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69880c27b0 msgr2=0x7f69880a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698d572640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69880c27b0 0x7f69880a8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698d572640 1 -- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6988158720 msgr2=0x7f698814bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698d572640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6988158720 0x7f698814bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698d572640 1 -- 172.21.15.94:0/2108594110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6988151d30 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698d572640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6988152740 0x7f698814b9d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f697800c730 tx=0x7f697800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f697e7fc640 1 -- 172.21.15.94:0/2108594110 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6978017070 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f697e7fc640 1 -- 172.21.15.94:0/2108594110 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6978005ad0 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69880aa1f0 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f697e7fc640 1 -- 172.21.15.94:0/2108594110 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6978007e20 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.334+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69880a8850 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.335+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6988071960 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.335+0000 7f697e7fc640 1 -- 172.21.15.94:0/2108594110 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f697801a070 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.336+0000 7f697e7fc640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696c06b4d0 0x7f696c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.336+0000 7f697e7fc640 1 -- 172.21.15.94:0/2108594110 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6978089860 con 0x7f6988152740 2024-03-06T07:18:57.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.336+0000 7f698dd73640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696c06b4d0 0x7f696c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.336+0000 7f698dd73640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696c06b4d0 0x7f696c06d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6980036000 tx=0x7f6980030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.340+0000 7f697e7fc640 1 -- 172.21.15.94:0/2108594110 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6988071960 con 0x7f6988152740 2024-03-06T07:18:57.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.524+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f698806b990 con 0x7f6988152740 2024-03-06T07:18:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.525+0000 7f697e7fc640 1 -- 172.21.15.94:0/2108594110 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f698806b990 con 0x7f6988152740 2024-03-06T07:18:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:listed 0 entries 2024-03-06T07:18:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696c06b4d0 msgr2=0x7f696c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696c06b4d0 0x7f696c06d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6980036000 tx=0x7f6980030000 comp rx=0 tx=0).stop 2024-03-06T07:18:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6988152740 msgr2=0x7f698814b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6988152740 0x7f698814b9d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f697800c730 tx=0x7f697800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 shutdown_connections 2024-03-06T07:18:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696c06b4d0 0x7f696c06d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6988158720 0x7f698814bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6988152740 0x7f698814b9d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 --2- 172.21.15.94:0/2108594110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69880c27b0 0x7f69880a8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 >> 172.21.15.94:0/2108594110 conn(0x7f69880c41b0 msgr2=0x7f698806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 shutdown_connections 2024-03-06T07:18:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.527+0000 7f698ffd7640 1 -- 172.21.15.94:0/2108594110 wait complete. 2024-03-06T07:18:57.539 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-06T07:18:57.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:57.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-06T07:18:57.572 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:18:57.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:18:57.572 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:18:57.640 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:18:57.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:57.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.739+0000 7f94736a4640 1 Processor -- start 2024-03-06T07:18:57.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.739+0000 7f94736a4640 1 -- start start 2024-03-06T07:18:57.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.739+0000 7f94736a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f946c1550c0 0x7f946c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.739+0000 7f94736a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f946c0aa180 0x7f946c0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.739+0000 7f94736a4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0aa660 con 0x7f946c1559e0 2024-03-06T07:18:57.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.739+0000 7f94736a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946c0c4610 con 0x7f946c0aa180 2024-03-06T07:18:57.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.739+0000 7f94736a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946c0aa940 con 0x7f946c1550c0 2024-03-06T07:18:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471c41640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f946c0aa180 0x7f946c0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471c41640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f946c0aa180 0x7f946c0c5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34070/0 (socket says 172.21.15.94:34070) 2024-03-06T07:18:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471c41640 1 -- 172.21.15.94:0/2684555471 learned_addr learned my addr 172.21.15.94:0/2684555471 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471440640 1 --2- 172.21.15.94:0/2684555471 >> v2:172.21.15.145:3300/0 conn(0x7f946c1550c0 0x7f946c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471c41640 1 -- 172.21.15.94:0/2684555471 >> v2:172.21.15.145:3300/0 conn(0x7f946c1550c0 msgr2=0x7f946c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471c41640 1 --2- 172.21.15.94:0/2684555471 >> v2:172.21.15.145:3300/0 conn(0x7f946c1550c0 0x7f946c1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471c41640 1 -- 172.21.15.94:0/2684555471 >> v1:172.21.15.145:6789/0 conn(0x7f946c1559e0 legacy=0x7f946c14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.740+0000 7f9471c41640 1 -- 172.21.15.94:0/2684555471 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c0c4860 con 0x7f946c0aa180 2024-03-06T07:18:57.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f9471c41640 1 --2- 172.21.15.94:0/2684555471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c0aa180 0x7f946c0c5060 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f946c0c3fc0 tx=0x7f946402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71cbff83c8ba2521 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f9463fff640 1 -- 172.21.15.94:0/2684555471 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946402c9f0 con 0x7f946c0aa180 2024-03-06T07:18:57.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f9463fff640 1 -- 172.21.15.94:0/2684555471 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946402cb90 con 0x7f946c0aa180 2024-03-06T07:18:57.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f9463fff640 1 -- 172.21.15.94:0/2684555471 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640343f0 con 0x7f946c0aa180 2024-03-06T07:18:57.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 -- 172.21.15.94:0/2684555471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c0aa180 msgr2=0x7f946c0c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 --2- 172.21.15.94:0/2684555471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c0aa180 0x7f946c0c5060 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f946c0c3fc0 tx=0x7f946402be50 comp rx=0 tx=0).stop 2024-03-06T07:18:57.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 -- 172.21.15.94:0/2684555471 shutdown_connections 2024-03-06T07:18:57.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 --2- 172.21.15.94:0/2684555471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c0aa180 0x7f946c0c5060 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 --2- 172.21.15.94:0/2684555471 >> v2:172.21.15.145:3300/0 conn(0x7f946c1550c0 0x7f946c1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 -- 172.21.15.94:0/2684555471 >> 172.21.15.94:0/2684555471 conn(0x7f946c0c6710 msgr2=0x7f946c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:57.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 -- 172.21.15.94:0/2684555471 shutdown_connections 2024-03-06T07:18:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.741+0000 7f94736a4640 1 -- 172.21.15.94:0/2684555471 wait complete. 2024-03-06T07:18:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.742+0000 7f94736a4640 1 Processor -- start 2024-03-06T07:18:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.742+0000 7f94736a4640 1 -- start start 2024-03-06T07:18:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f94736a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c0aa180 0x7f946c1ec400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f94736a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c1550c0 0x7f946c0a8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f94736a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c1559e0 0x7f946c070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f94736a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c14f310 con 0x7f946c1550c0 2024-03-06T07:18:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f94736a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0c1560 con 0x7f946c1559e0 2024-03-06T07:18:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f94736a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c14f490 con 0x7f946c0aa180 2024-03-06T07:18:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c1550c0 0x7f946c0a8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c1550c0 0x7f946c0a8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34082/0 (socket says 172.21.15.94:34082) 2024-03-06T07:18:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 -- 172.21.15.94:0/2751601612 learned_addr learned my addr 172.21.15.94:0/2751601612 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:57.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9471c41640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c1559e0 0x7f946c070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9471440640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c0aa180 0x7f946c1ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 -- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c0aa180 msgr2=0x7f946c1ec400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c0aa180 0x7f946c1ec400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 -- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c1559e0 msgr2=0x7f946c070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c1559e0 0x7f946c070d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.743+0000 7f9470c3f640 1 -- 172.21.15.94:0/2751601612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c1ec940 con 0x7f946c1550c0 2024-03-06T07:18:57.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.744+0000 7f9470c3f640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c1550c0 0x7f946c0a8700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f946800c3c0 tx=0x7f946800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.744+0000 7f9461ffb640 1 -- 172.21.15.94:0/2751601612 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94680052d0 con 0x7f946c1550c0 2024-03-06T07:18:57.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.744+0000 7f9461ffb640 1 -- 172.21.15.94:0/2751601612 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9468005490 con 0x7f946c1550c0 2024-03-06T07:18:57.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.744+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c072f80 con 0x7f946c1550c0 2024-03-06T07:18:57.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.744+0000 7f9461ffb640 1 -- 172.21.15.94:0/2751601612 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468007260 con 0x7f946c1550c0 2024-03-06T07:18:57.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.744+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c070540 con 0x7f946c1550c0 2024-03-06T07:18:57.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.745+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c04f330 con 0x7f946c1550c0 2024-03-06T07:18:57.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.745+0000 7f9461ffb640 1 -- 172.21.15.94:0/2751601612 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9468007400 con 0x7f946c1550c0 2024-03-06T07:18:57.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.746+0000 7f9461ffb640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:57.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.746+0000 7f9461ffb640 1 -- 172.21.15.94:0/2751601612 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9468089960 con 0x7f946c1550c0 2024-03-06T07:18:57.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.746+0000 7f9471440640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:57.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.746+0000 7f9471440640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f945c009aa0 tx=0x7f945c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:57.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.750+0000 7f9461ffb640 1 -- 172.21.15.94:0/2751601612 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9468057e20 con 0x7f946c1550c0 2024-03-06T07:18:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.956+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f946c068450 con 0x7f946c1550c0 2024-03-06T07:18:57.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.956+0000 7f9461ffb640 1 -- 172.21.15.94:0/2751601612 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7f946805bec0 con 0x7f946c1550c0 2024-03-06T07:18:57.958 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:57.959 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:57.959 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:18:57.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f945006b4d0 msgr2=0x7f945006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f945c009aa0 tx=0x7f945c009190 comp rx=0 tx=0).stop 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c1550c0 msgr2=0x7f946c0a8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c1550c0 0x7f946c0a8700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f946800c3c0 tx=0x7f946800c880 comp rx=0 tx=0).stop 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 shutdown_connections 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c1559e0 0x7f946c070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f946c1550c0 0x7f946c0a8700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 --2- 172.21.15.94:0/2751601612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c0aa180 0x7f946c1ec400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 >> 172.21.15.94:0/2751601612 conn(0x7f946c0c6710 msgr2=0x7f946c152bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 shutdown_connections 2024-03-06T07:18:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:57.958+0000 7f94736a4640 1 -- 172.21.15.94:0/2751601612 wait complete. 2024-03-06T07:18:57.971 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:58.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.069+0000 7f00a15b5640 1 Processor -- start 2024-03-06T07:18:58.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.069+0000 7f00a15b5640 1 -- start start 2024-03-06T07:18:58.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.070+0000 7f00a15b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f009c1506b0 0x7f009c153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.070+0000 7f00a15b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c070aa0 con 0x7f009c0acc00 2024-03-06T07:18:58.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.070+0000 7f00a15b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c070d80 con 0x7f009c156ea0 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.070+0000 7f00a15b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009c156450 con 0x7f009c1506b0 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.070+0000 7f009b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f009c0acc00 0x7f009c0c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50796/0 (socket says 172.21.15.94:50796) 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.070+0000 7f009b7fe640 1 -- 172.21.15.94:0/547214467 learned_addr learned my addr 172.21.15.94:0/547214467 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.070+0000 7f009affd640 1 --2- 172.21.15.94:0/547214467 >> v2:172.21.15.145:3300/0 conn(0x7f009c1506b0 0x7f009c153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f0099ffb640 1 -- 172.21.15.94:0/547214467 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784194999 0 0) 0x7f009c070aa0 con 0x7f009c0acc00 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f0099ffb640 1 -- 172.21.15.94:0/547214467 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0084003520 con 0x7f009c0acc00 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f0099ffb640 1 -- 172.21.15.94:0/547214467 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455742303 0 0) 0x7f009c070d80 con 0x7f009c156ea0 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f0099ffb640 1 -- 172.21.15.94:0/547214467 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f009c070aa0 con 0x7f009c156ea0 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f009affd640 1 -- 172.21.15.94:0/547214467 >> v1:172.21.15.145:6789/0 conn(0x7f009c156ea0 legacy=0x7f009c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f009affd640 1 -- 172.21.15.94:0/547214467 >> v1:172.21.15.94:6789/0 conn(0x7f009c0acc00 legacy=0x7f009c0c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f009affd640 1 -- 172.21.15.94:0/547214467 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c1566a0 con 0x7f009c1506b0 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.071+0000 7f009affd640 1 --2- 172.21.15.94:0/547214467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 0x7f009c153990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f008c009d90 tx=0x7f008c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=780a73570c0c9f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f0099ffb640 1 -- 172.21.15.94:0/547214467 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c039070 con 0x7f009c1506b0 2024-03-06T07:18:58.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f0099ffb640 1 -- 172.21.15.94:0/547214467 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c02cd00 con 0x7f009c1506b0 2024-03-06T07:18:58.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f0099ffb640 1 -- 172.21.15.94:0/547214467 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c034640 con 0x7f009c1506b0 2024-03-06T07:18:58.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f00a15b5640 1 -- 172.21.15.94:0/547214467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 msgr2=0x7f009c153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f00a15b5640 1 --2- 172.21.15.94:0/547214467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 0x7f009c153990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f008c009d90 tx=0x7f008c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f00a15b5640 1 -- 172.21.15.94:0/547214467 shutdown_connections 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f00a15b5640 1 --2- 172.21.15.94:0/547214467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 0x7f009c153990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f00a15b5640 1 -- 172.21.15.94:0/547214467 >> 172.21.15.94:0/547214467 conn(0x7f009c06b1a0 msgr2=0x7f009c06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f00a15b5640 1 -- 172.21.15.94:0/547214467 shutdown_connections 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.072+0000 7f00a15b5640 1 -- 172.21.15.94:0/547214467 wait complete. 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.073+0000 7f00a15b5640 1 Processor -- start 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.073+0000 7f00a15b5640 1 -- start start 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f00a15b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f009c0acc00 0x7f009c152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f00a15b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 0x7f009c1498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f00a15b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f009c156ea0 0x7f009c149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f00a15b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c0ad050 con 0x7f009c0acc00 2024-03-06T07:18:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f00a15b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c156a10 con 0x7f009c1506b0 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f00a15b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c0ad1d0 con 0x7f009c156ea0 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f009c0acc00 0x7f009c152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f009c0acc00 0x7f009c152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34084/0 (socket says 172.21.15.94:34084) 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 -- 172.21.15.94:0/4062422862 learned_addr learned my addr 172.21.15.94:0/4062422862 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009a7fc640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 0x7f009c1498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009b7fe640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f009c156ea0 0x7f009c149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 -- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f009c156ea0 msgr2=0x7f009c149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f009c156ea0 0x7f009c149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 -- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 msgr2=0x7f009c1498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 0x7f009c1498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.074+0000 7f009affd640 1 -- 172.21.15.94:0/4062422862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c14b0a0 con 0x7f009c0acc00 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.075+0000 7f009affd640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f009c0acc00 0x7f009c152050 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f008c037040 tx=0x7f008c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.075+0000 7f00837fe640 1 -- 172.21.15.94:0/4062422862 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c039050 con 0x7f009c0acc00 2024-03-06T07:18:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.075+0000 7f00837fe640 1 -- 172.21.15.94:0/4062422862 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c045070 con 0x7f009c0acc00 2024-03-06T07:18:58.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.075+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c0a92d0 con 0x7f009c0acc00 2024-03-06T07:18:58.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.075+0000 7f00837fe640 1 -- 172.21.15.94:0/4062422862 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c009450 con 0x7f009c0acc00 2024-03-06T07:18:58.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.075+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c14b340 con 0x7f009c0acc00 2024-03-06T07:18:58.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.076+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009c04f330 con 0x7f009c0acc00 2024-03-06T07:18:58.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.076+0000 7f00837fe640 1 -- 172.21.15.94:0/4062422862 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f008c03e850 con 0x7f009c0acc00 2024-03-06T07:18:58.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.077+0000 7f00837fe640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f008806b4d0 0x7f008806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.077+0000 7f00837fe640 1 -- 172.21.15.94:0/4062422862 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f008c0ad3a0 con 0x7f009c0acc00 2024-03-06T07:18:58.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.077+0000 7f009a7fc640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f008806b4d0 0x7f008806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.077+0000 7f009a7fc640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f008806b4d0 0x7f008806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0090007fd0 tx=0x7f0090009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.081+0000 7f00837fe640 1 -- 172.21.15.94:0/4062422862 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f008c07b980 con 0x7f009c0acc00 2024-03-06T07:18:58.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.256+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f009c0700a0 con 0x7f009c0acc00 2024-03-06T07:18:58.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.257+0000 7f00837fe640 1 -- 172.21.15.94:0/4062422862 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18903 (secure 0 0 0) 0x7f008c07fa20 con 0x7f009c0acc00 2024-03-06T07:18:58.260 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:58.260 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":26,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:58.142419+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f008806b4d0 msgr2=0x7f008806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f008806b4d0 0x7f008806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0090007fd0 tx=0x7f0090009040 comp rx=0 tx=0).stop 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f009c0acc00 msgr2=0x7f009c152050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f009c0acc00 0x7f009c152050 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f008c037040 tx=0x7f008c004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 shutdown_connections 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f008806b4d0 0x7f008806d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f009c156ea0 0x7f009c149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f009c1506b0 0x7f009c1498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 --2- 172.21.15.94:0/4062422862 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f009c0acc00 0x7f009c152050 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.259+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 >> 172.21.15.94:0/4062422862 conn(0x7f009c06b1a0 msgr2=0x7f009c14f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:58.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.260+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 shutdown_connections 2024-03-06T07:18:58.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.260+0000 7f00a15b5640 1 -- 172.21.15.94:0/4062422862 wait complete. 2024-03-06T07:18:58.273 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:18:07.163676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:18:07.821715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-06T07:18:11.560181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-06T07:18:58.273 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:18:58.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.374+0000 7f29ff278640 1 Processor -- start 2024-03-06T07:18:58.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.374+0000 7f29ff278640 1 -- start start 2024-03-06T07:18:58.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29ff278640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29f806afc0 0x7f29f80ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29ff278640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f80abe90 0x7f29f815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29ff278640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f806b440 con 0x7f29f814d350 2024-03-06T07:18:58.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29ff278640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f80ac750 con 0x7f29f806afc0 2024-03-06T07:18:58.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29ff278640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80c1a40 con 0x7f29f80abe90 2024-03-06T07:18:58.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fc813640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29f806afc0 0x7f29f80ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fc813640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29f806afc0 0x7f29f80ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34092/0 (socket says 172.21.15.94:34092) 2024-03-06T07:18:58.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fc813640 1 -- 172.21.15.94:0/792922934 learned_addr learned my addr 172.21.15.94:0/792922934 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:58.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fd815640 1 --2- 172.21.15.94:0/792922934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f80abe90 0x7f29f815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fc813640 1 -- 172.21.15.94:0/792922934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f80abe90 msgr2=0x7f29f815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:58.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fc813640 1 --2- 172.21.15.94:0/792922934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f80abe90 0x7f29f815d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fc813640 1 -- 172.21.15.94:0/792922934 >> v1:172.21.15.145:6789/0 conn(0x7f29f814d350 legacy=0x7f29f80acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.375+0000 7f29fc813640 1 -- 172.21.15.94:0/792922934 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80c1c90 con 0x7f29f806afc0 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.376+0000 7f29fc813640 1 --2- 172.21.15.94:0/792922934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f806afc0 0x7f29f80ab950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f29e8009870 tx=0x7f29e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa9d4fdbd16ff6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.376+0000 7f29effff640 1 -- 172.21.15.94:0/792922934 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8035020 con 0x7f29f806afc0 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.376+0000 7f29effff640 1 -- 172.21.15.94:0/792922934 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e8031640 con 0x7f29f806afc0 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.376+0000 7f29effff640 1 -- 172.21.15.94:0/792922934 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e802d040 con 0x7f29f806afc0 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.376+0000 7f29ff278640 1 -- 172.21.15.94:0/792922934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f806afc0 msgr2=0x7f29f80ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.376+0000 7f29ff278640 1 --2- 172.21.15.94:0/792922934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f806afc0 0x7f29f80ab950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f29e8009870 tx=0x7f29e802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.377+0000 7f29ff278640 1 -- 172.21.15.94:0/792922934 shutdown_connections 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.377+0000 7f29ff278640 1 --2- 172.21.15.94:0/792922934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f80abe90 0x7f29f815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.377+0000 7f29ff278640 1 --2- 172.21.15.94:0/792922934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f806afc0 0x7f29f80ab950 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.377+0000 7f29ff278640 1 -- 172.21.15.94:0/792922934 >> 172.21.15.94:0/792922934 conn(0x7f29f80c3b40 msgr2=0x7f29f80c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.377+0000 7f29ff278640 1 -- 172.21.15.94:0/792922934 shutdown_connections 2024-03-06T07:18:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.377+0000 7f29ff278640 1 -- 172.21.15.94:0/792922934 wait complete. 2024-03-06T07:18:58.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 Processor -- start 2024-03-06T07:18:58.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 -- start start 2024-03-06T07:18:58.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29f806afc0 0x7f29f80c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f80abe90 0x7f29f8145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f814d350 0x7f29f81462f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f806ea70 con 0x7f29f80abe90 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8153c00 con 0x7f29f806afc0 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29ff278640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f8146830 con 0x7f29f814d350 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29fc813640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f80abe90 0x7f29f8145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29fc813640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f80abe90 0x7f29f8145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34096/0 (socket says 172.21.15.94:34096) 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.378+0000 7f29fc813640 1 -- 172.21.15.94:0/569718283 learned_addr learned my addr 172.21.15.94:0/569718283 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fd815640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f814d350 0x7f29f81462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fd014640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29f806afc0 0x7f29f80c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fc813640 1 -- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f814d350 msgr2=0x7f29f81462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fc813640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f814d350 0x7f29f81462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fc813640 1 -- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29f806afc0 msgr2=0x7f29f80c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fc813640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29f806afc0 0x7f29f80c06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fc813640 1 -- 172.21.15.94:0/569718283 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f81ef5a0 con 0x7f29f80abe90 2024-03-06T07:18:58.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29fc813640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f80abe90 0x7f29f8145db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f29e8000c00 tx=0x7f29e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29edffb640 1 -- 172.21.15.94:0/569718283 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8040070 con 0x7f29f80abe90 2024-03-06T07:18:58.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29edffb640 1 -- 172.21.15.94:0/569718283 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e80399f0 con 0x7f29f80abe90 2024-03-06T07:18:58.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f81ee7e0 con 0x7f29f80abe90 2024-03-06T07:18:58.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29edffb640 1 -- 172.21.15.94:0/569718283 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8035040 con 0x7f29f80abe90 2024-03-06T07:18:58.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.379+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f81eeda0 con 0x7f29f80abe90 2024-03-06T07:18:58.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.380+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f814cb20 con 0x7f29f80abe90 2024-03-06T07:18:58.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.381+0000 7f29edffb640 1 -- 172.21.15.94:0/569718283 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29e804b060 con 0x7f29f80abe90 2024-03-06T07:18:58.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.381+0000 7f29edffb640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29dc06b4d0 0x7f29dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.381+0000 7f29edffb640 1 -- 172.21.15.94:0/569718283 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f29e80abce0 con 0x7f29f80abe90 2024-03-06T07:18:58.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.381+0000 7f29fd014640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29dc06b4d0 0x7f29dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.382+0000 7f29fd014640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29dc06b4d0 0x7f29dc06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f29f8150a60 tx=0x7f29f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.385+0000 7f29edffb640 1 -- 172.21.15.94:0/569718283 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29f814cb20 con 0x7f29f80abe90 2024-03-06T07:18:58.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.559+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29f804f330 con 0x7f29f80abe90 2024-03-06T07:18:58.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.559+0000 7f29edffb640 1 -- 172.21.15.94:0/569718283 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29e807a1a0 con 0x7f29f80abe90 2024-03-06T07:18:58.562 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:18:58.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29dc06b4d0 msgr2=0x7f29dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29dc06b4d0 0x7f29dc06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f29f8150a60 tx=0x7f29f4009040 comp rx=0 tx=0).stop 2024-03-06T07:18:58.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f80abe90 msgr2=0x7f29f8145db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f80abe90 0x7f29f8145db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f29e8000c00 tx=0x7f29e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 shutdown_connections 2024-03-06T07:18:58.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29dc06b4d0 0x7f29dc06d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29f814d350 0x7f29f81462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29f80abe90 0x7f29f8145db0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 --2- 172.21.15.94:0/569718283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29f806afc0 0x7f29f80c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 >> 172.21.15.94:0/569718283 conn(0x7f29f80c3b40 msgr2=0x7f29f80c26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:58.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 shutdown_connections 2024-03-06T07:18:58.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.561+0000 7f29ff278640 1 -- 172.21.15.94:0/569718283 wait complete. 2024-03-06T07:18:58.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:18:58.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.675+0000 7f6330735640 1 Processor -- start 2024-03-06T07:18:58.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.675+0000 7f6330735640 1 -- start start 2024-03-06T07:18:58.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.676+0000 7f6330735640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f63280a8290 0x7f63281539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.676+0000 7f6330735640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f63280c69a0 0x7f6328072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.676+0000 7f6330735640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63280c62a0 con 0x7f63280c4740 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.676+0000 7f6330735640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6328153f00 con 0x7f63280c69a0 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.676+0000 7f6330735640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63280c4c00 con 0x7f63280a8290 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.676+0000 7f632ecd2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f63280c69a0 0x7f6328072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.677+0000 7f632ecd2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f63280c69a0 0x7f6328072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39116/0 (socket says 172.21.15.94:39116) 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.677+0000 7f632ecd2640 1 -- 172.21.15.94:0/1365313174 learned_addr learned my addr 172.21.15.94:0/1365313174 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.677+0000 7f632e4d1640 1 --2- 172.21.15.94:0/1365313174 >> v2:172.21.15.145:3300/0 conn(0x7f63280a8290 0x7f63281539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.677+0000 7f632ecd2640 1 -- 172.21.15.94:0/1365313174 >> v2:172.21.15.145:3300/0 conn(0x7f63280a8290 msgr2=0x7f63281539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.677+0000 7f632ecd2640 1 --2- 172.21.15.94:0/1365313174 >> v2:172.21.15.145:3300/0 conn(0x7f63280a8290 0x7f63281539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.677+0000 7f632ecd2640 1 -- 172.21.15.94:0/1365313174 >> v1:172.21.15.145:6789/0 conn(0x7f63280c4740 legacy=0x7f63280701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.677+0000 7f632ecd2640 1 -- 172.21.15.94:0/1365313174 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63280c4ee0 con 0x7f63280c69a0 2024-03-06T07:18:58.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f632ecd2640 1 --2- 172.21.15.94:0/1365313174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f6328072680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f63280735b0 tx=0x7f632002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e96914a4408777 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f632d4cf640 1 -- 172.21.15.94:0/1365313174 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632002c9f0 con 0x7f63280c69a0 2024-03-06T07:18:58.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f632d4cf640 1 -- 172.21.15.94:0/1365313174 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632002cb90 con 0x7f63280c69a0 2024-03-06T07:18:58.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f632d4cf640 1 -- 172.21.15.94:0/1365313174 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63200343f0 con 0x7f63280c69a0 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 -- 172.21.15.94:0/1365313174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 msgr2=0x7f6328072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 --2- 172.21.15.94:0/1365313174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f6328072680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f63280735b0 tx=0x7f632002be50 comp rx=0 tx=0).stop 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 -- 172.21.15.94:0/1365313174 shutdown_connections 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 --2- 172.21.15.94:0/1365313174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f6328072680 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 --2- 172.21.15.94:0/1365313174 >> v2:172.21.15.145:3300/0 conn(0x7f63280a8290 0x7f63281539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 -- 172.21.15.94:0/1365313174 >> 172.21.15.94:0/1365313174 conn(0x7f63280c5560 msgr2=0x7f63280c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 -- 172.21.15.94:0/1365313174 shutdown_connections 2024-03-06T07:18:58.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.678+0000 7f6330735640 1 -- 172.21.15.94:0/1365313174 wait complete. 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.679+0000 7f6330735640 1 Processor -- start 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.679+0000 7f6330735640 1 -- start start 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f6330735640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63280a8290 0x7f63280ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f6330735640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63280c4740 0x7f63280acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f6330735640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f632814cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f6330735640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63280c6df0 con 0x7f63280c69a0 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f6330735640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632806fc80 con 0x7f63280c4740 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f6330735640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63280c6f70 con 0x7f63280a8290 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f632814cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f632814cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39120/0 (socket says 172.21.15.94:39120) 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 -- 172.21.15.94:0/2633207554 learned_addr learned my addr 172.21.15.94:0/2633207554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:58.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632dcd0640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63280c4740 0x7f63280acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632e4d1640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63280a8290 0x7f63280ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 -- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63280a8290 msgr2=0x7f63280ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63280a8290 0x7f63280ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 -- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63280c4740 msgr2=0x7f63280acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63280c4740 0x7f63280acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.680+0000 7f632ecd2640 1 -- 172.21.15.94:0/2633207554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63280ad070 con 0x7f63280c69a0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.681+0000 7f632ecd2640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f632814cd40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6320036040 tx=0x7f6320004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.681+0000 7f631affd640 1 -- 172.21.15.94:0/2633207554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320044020 con 0x7f63280c69a0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.681+0000 7f631affd640 1 -- 172.21.15.94:0/2633207554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6320040470 con 0x7f63280c69a0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.681+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63280abf90 con 0x7f63280c69a0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.681+0000 7f631affd640 1 -- 172.21.15.94:0/2633207554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6320009510 con 0x7f63280c69a0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.681+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632814d4f0 con 0x7f63280c69a0 2024-03-06T07:18:58.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.682+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632804f330 con 0x7f63280c69a0 2024-03-06T07:18:58.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.682+0000 7f631affd640 1 -- 172.21.15.94:0/2633207554 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6320047020 con 0x7f63280c69a0 2024-03-06T07:18:58.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.683+0000 7f631affd640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f630c06b4d0 0x7f630c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.683+0000 7f631affd640 1 -- 172.21.15.94:0/2633207554 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f63200ace40 con 0x7f63280c69a0 2024-03-06T07:18:58.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.683+0000 7f632e4d1640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f630c06b4d0 0x7f630c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.683+0000 7f632e4d1640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f630c06b4d0 0x7f630c06d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f631c009b40 tx=0x7f631c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.687+0000 7f631affd640 1 -- 172.21.15.94:0/2633207554 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f632004e030 con 0x7f63280c69a0 2024-03-06T07:18:58.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.855+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f63280735e0 con 0x7f63280c69a0 2024-03-06T07:18:58.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.855+0000 7f631affd640 1 -- 172.21.15.94:0/2633207554 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63280ad070 con 0x7f63280c69a0 2024-03-06T07:18:58.858 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:58.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f630c06b4d0 msgr2=0x7f630c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f630c06b4d0 0x7f630c06d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f631c009b40 tx=0x7f631c009190 comp rx=0 tx=0).stop 2024-03-06T07:18:58.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 msgr2=0x7f632814cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f632814cd40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6320036040 tx=0x7f6320004a40 comp rx=0 tx=0).stop 2024-03-06T07:18:58.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 shutdown_connections 2024-03-06T07:18:58.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f630c06b4d0 0x7f630c06d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63280c69a0 0x7f632814cd40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63280c4740 0x7f63280acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 --2- 172.21.15.94:0/2633207554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63280a8290 0x7f63280ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 >> 172.21.15.94:0/2633207554 conn(0x7f63280c5560 msgr2=0x7f6328153370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 shutdown_connections 2024-03-06T07:18:58.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.857+0000 7f6330735640 1 -- 172.21.15.94:0/2633207554 wait complete. 2024-03-06T07:18:58.870 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:18:58.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.971+0000 7fe9944b5640 1 Processor -- start 2024-03-06T07:18:58.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.971+0000 7fe9944b5640 1 -- start start 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.972+0000 7fe9944b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe98c0c5990 0x7fe98c0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.972+0000 7fe9944b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0ab170 0x7fe98c06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.972+0000 7fe9944b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe98c06c730 0x7fe98c1523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.972+0000 7fe9944b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe98c06cbe0 con 0x7fe98c06c730 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.972+0000 7fe9944b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe98c0c3ef0 con 0x7fe98c0c5990 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.972+0000 7fe9944b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c1556d0 con 0x7fe98c0ab170 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.972+0000 7fe992a52640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe98c06c730 0x7fe98c1523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992a52640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe98c06c730 0x7fe98c1523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39126/0 (socket says 172.21.15.94:39126) 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992a52640 1 -- 172.21.15.94:0/2561894347 learned_addr learned my addr 172.21.15.94:0/2561894347 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe991a50640 1 --2- 172.21.15.94:0/2561894347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0ab170 0x7fe98c06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992251640 1 --2- 172.21.15.94:0/2561894347 >> v2:172.21.15.145:3300/0 conn(0x7fe98c0c5990 0x7fe98c0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992a52640 1 -- 172.21.15.94:0/2561894347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0ab170 msgr2=0x7fe98c06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992a52640 1 --2- 172.21.15.94:0/2561894347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0ab170 0x7fe98c06c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992a52640 1 -- 172.21.15.94:0/2561894347 >> v2:172.21.15.145:3300/0 conn(0x7fe98c0c5990 msgr2=0x7fe98c0c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992a52640 1 --2- 172.21.15.94:0/2561894347 >> v2:172.21.15.145:3300/0 conn(0x7fe98c0c5990 0x7fe98c0c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe992a52640 1 -- 172.21.15.94:0/2561894347 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c1559b0 con 0x7fe98c06c730 2024-03-06T07:18:58.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.973+0000 7fe991a50640 1 --2- 172.21.15.94:0/2561894347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0ab170 0x7fe98c06c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:58.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.974+0000 7fe992a52640 1 --2- 172.21.15.94:0/2561894347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c06c730 0x7fe98c1523f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0c4100 tx=0x7fe98402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3194d6bb7f270b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.974+0000 7fe99124f640 1 -- 172.21.15.94:0/2561894347 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98402ca70 con 0x7fe98c06c730 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.974+0000 7fe99124f640 1 -- 172.21.15.94:0/2561894347 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98402cc10 con 0x7fe98c06c730 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.974+0000 7fe99124f640 1 -- 172.21.15.94:0/2561894347 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9840343f0 con 0x7fe98c06c730 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.974+0000 7fe9944b5640 1 -- 172.21.15.94:0/2561894347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c06c730 msgr2=0x7fe98c1523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.974+0000 7fe9944b5640 1 --2- 172.21.15.94:0/2561894347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c06c730 0x7fe98c1523f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0c4100 tx=0x7fe98402bed0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.975+0000 7fe9944b5640 1 -- 172.21.15.94:0/2561894347 shutdown_connections 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.975+0000 7fe9944b5640 1 --2- 172.21.15.94:0/2561894347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c06c730 0x7fe98c1523f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.975+0000 7fe9944b5640 1 --2- 172.21.15.94:0/2561894347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0ab170 0x7fe98c06c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.975+0000 7fe9944b5640 1 --2- 172.21.15.94:0/2561894347 >> v2:172.21.15.145:3300/0 conn(0x7fe98c0c5990 0x7fe98c0c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.975+0000 7fe9944b5640 1 -- 172.21.15.94:0/2561894347 >> 172.21.15.94:0/2561894347 conn(0x7fe98c0c6710 msgr2=0x7fe98c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.975+0000 7fe9944b5640 1 -- 172.21.15.94:0/2561894347 shutdown_connections 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.975+0000 7fe9944b5640 1 -- 172.21.15.94:0/2561894347 wait complete. 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 Processor -- start 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 -- start start 2024-03-06T07:18:58.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98c06c730 0x7fe98c154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c0ab170 0x7fe98c154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0c5990 0x7fe98c14d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c156490 con 0x7fe98c0ab170 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c155f70 con 0x7fe98c06c730 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe9944b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98c156610 con 0x7fe98c0c5990 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe991a50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c0ab170 0x7fe98c154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe991a50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c0ab170 0x7fe98c154a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39138/0 (socket says 172.21.15.94:39138) 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.976+0000 7fe991a50640 1 -- 172.21.15.94:0/489096479 learned_addr learned my addr 172.21.15.94:0/489096479 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe992251640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98c06c730 0x7fe98c154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe992a52640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0c5990 0x7fe98c14d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe991a50640 1 -- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0c5990 msgr2=0x7fe98c14d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe991a50640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0c5990 0x7fe98c14d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe991a50640 1 -- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98c06c730 msgr2=0x7fe98c154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe991a50640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98c06c730 0x7fe98c154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe991a50640 1 -- 172.21.15.94:0/489096479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c1e83c0 con 0x7fe98c0ab170 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe991a50640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c0ab170 0x7fe98c154a90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe988010df0 tx=0x7fe98800e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe982ffd640 1 -- 172.21.15.94:0/489096479 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98801e070 con 0x7fe98c0ab170 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe982ffd640 1 -- 172.21.15.94:0/489096479 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe988011660 con 0x7fe98c0ab170 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98c1e86a0 con 0x7fe98c0ab170 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe982ffd640 1 -- 172.21.15.94:0/489096479 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988011830 con 0x7fe98c0ab170 2024-03-06T07:18:58.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.977+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98c14f620 con 0x7fe98c0ab170 2024-03-06T07:18:58.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.978+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98c04f330 con 0x7fe98c0ab170 2024-03-06T07:18:58.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.979+0000 7fe982ffd640 1 -- 172.21.15.94:0/489096479 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe988021070 con 0x7fe98c0ab170 2024-03-06T07:18:58.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.979+0000 7fe982ffd640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe97006b4d0 0x7fe97006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:58.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.979+0000 7fe982ffd640 1 -- 172.21.15.94:0/489096479 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe98808c630 con 0x7fe98c0ab170 2024-03-06T07:18:58.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.979+0000 7fe992251640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe97006b4d0 0x7fe97006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:58.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.980+0000 7fe992251640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe97006b4d0 0x7fe97006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009a20 tx=0x7fe97c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:58.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:58.983+0000 7fe982ffd640 1 -- 172.21.15.94:0/489096479 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe98805aaf0 con 0x7fe98c0ab170 2024-03-06T07:18:59.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.155+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe98c0c1740 con 0x7fe98c0ab170 2024-03-06T07:18:59.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.156+0000 7fe982ffd640 1 -- 172.21.15.94:0/489096479 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe98805eb90 con 0x7fe98c0ab170 2024-03-06T07:18:59.158 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:18:59.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe97006b4d0 msgr2=0x7fe97006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe97006b4d0 0x7fe97006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009a20 tx=0x7fe97c009190 comp rx=0 tx=0).stop 2024-03-06T07:18:59.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c0ab170 msgr2=0x7fe98c154a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c0ab170 0x7fe98c154a90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe988010df0 tx=0x7fe98800e730 comp rx=0 tx=0).stop 2024-03-06T07:18:59.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 shutdown_connections 2024-03-06T07:18:59.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe97006b4d0 0x7fe97006d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98c0c5990 0x7fe98c14d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe98c0ab170 0x7fe98c154a90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 --2- 172.21.15.94:0/489096479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98c06c730 0x7fe98c154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 >> 172.21.15.94:0/489096479 conn(0x7fe98c0c6710 msgr2=0x7fe98c0a9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:59.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 shutdown_connections 2024-03-06T07:18:59.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.158+0000 7fe9944b5640 1 -- 172.21.15.94:0/489096479 wait complete. 2024-03-06T07:18:59.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-06T07:18:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.271+0000 7f410ab8a640 1 Processor -- start 2024-03-06T07:18:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.271+0000 7f410ab8a640 1 -- start start 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f410ab8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41040c5880 0x7f41040c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f410ab8a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f41040c5c60 0x7f41040c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f410ab8a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4104072600 con 0x7f41040c7720 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f410ab8a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41040723b0 con 0x7f41040c5880 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f410ab8a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41040728e0 con 0x7f41040c5c60 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4108926640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41040c5880 0x7f41040c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4109127640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f41040c7720 0x7f410406f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60162/0 (socket says 172.21.15.94:60162) 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4108926640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41040c5880 0x7f41040c6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39148/0 (socket says 172.21.15.94:39148) 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4109127640 1 -- 172.21.15.94:0/578977226 learned_addr learned my addr 172.21.15.94:0/578977226 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4103fff640 1 --2- 172.21.15.94:0/578977226 >> v2:172.21.15.145:3300/0 conn(0x7f41040c5c60 0x7f41040c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4108926640 1 -- 172.21.15.94:0/578977226 >> v2:172.21.15.145:3300/0 conn(0x7f41040c5c60 msgr2=0x7f41040c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4108926640 1 --2- 172.21.15.94:0/578977226 >> v2:172.21.15.145:3300/0 conn(0x7f41040c5c60 0x7f41040c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4108926640 1 -- 172.21.15.94:0/578977226 >> v1:172.21.15.94:6789/0 conn(0x7f41040c7720 legacy=0x7f410406f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f4108926640 1 -- 172.21.15.94:0/578977226 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104153af0 con 0x7f41040c5880 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.272+0000 7f41037fe640 1 -- 172.21.15.94:0/578977226 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043170469 0 0) 0x7f4104072600 con 0x7f41040c7720 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.273+0000 7f4108926640 1 --2- 172.21.15.94:0/578977226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c5880 0x7f41040c6ca0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f41040720f0 tx=0x7f40f402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7492b4eb44fe8e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.273+0000 7f41037fe640 1 -- 172.21.15.94:0/578977226 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4038070 con 0x7f41040c5880 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.273+0000 7f41037fe640 1 -- 172.21.15.94:0/578977226 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f4031710 con 0x7f41040c5880 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.273+0000 7f41037fe640 1 -- 172.21.15.94:0/578977226 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4031ae0 con 0x7f41040c5880 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 -- 172.21.15.94:0/578977226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c5880 msgr2=0x7f41040c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 --2- 172.21.15.94:0/578977226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c5880 0x7f41040c6ca0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f41040720f0 tx=0x7f40f402b7b0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 -- 172.21.15.94:0/578977226 shutdown_connections 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 --2- 172.21.15.94:0/578977226 >> v2:172.21.15.145:3300/0 conn(0x7f41040c5c60 0x7f41040c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 --2- 172.21.15.94:0/578977226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c5880 0x7f41040c6ca0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 -- 172.21.15.94:0/578977226 >> 172.21.15.94:0/578977226 conn(0x7f41040aa630 msgr2=0x7f41040aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:59.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 -- 172.21.15.94:0/578977226 shutdown_connections 2024-03-06T07:18:59.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.274+0000 7f410ab8a640 1 -- 172.21.15.94:0/578977226 wait complete. 2024-03-06T07:18:59.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.275+0000 7f410ab8a640 1 Processor -- start 2024-03-06T07:18:59.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.275+0000 7f410ab8a640 1 -- start start 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f410ab8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41040c5880 0x7f4104148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f410ab8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41040c5c60 0x7f41040abf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f410ab8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c7720 0x7f41040ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f410ab8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4104073190 con 0x7f41040c7720 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f410ab8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040c6780 con 0x7f41040c5c60 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f410ab8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4104073310 con 0x7f41040c5880 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c7720 0x7f41040ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c7720 0x7f41040ac460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39162/0 (socket says 172.21.15.94:39162) 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 -- 172.21.15.94:0/227803604 learned_addr learned my addr 172.21.15.94:0/227803604 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4103fff640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41040c5c60 0x7f41040abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4108926640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41040c5880 0x7f4104148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 -- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41040c5880 msgr2=0x7f4104148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41040c5880 0x7f4104148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 -- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41040c5c60 msgr2=0x7f41040abf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41040c5c60 0x7f41040abf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 -- 172.21.15.94:0/227803604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4104155980 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4103fff640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41040c5c60 0x7f41040abf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.276+0000 7f4109127640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c7720 0x7f41040ac460 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c780 tx=0x7f40fc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.277+0000 7f41017fa640 1 -- 172.21.15.94:0/227803604 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc018070 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.277+0000 7f41017fa640 1 -- 172.21.15.94:0/227803604 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc004d60 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.277+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41040ac9a0 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.277+0000 7f41017fa640 1 -- 172.21.15.94:0/227803604 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc00a880 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.277+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41041e92f0 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.278+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410404f330 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.278+0000 7f41017fa640 1 -- 172.21.15.94:0/227803604 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40fc01b070 con 0x7f41040c7720 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.278+0000 7f41017fa640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f40f006b4d0 0x7f40f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.279+0000 7f41017fa640 1 -- 172.21.15.94:0/227803604 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f40fc089920 con 0x7f41040c7720 2024-03-06T07:18:59.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.279+0000 7f4108926640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f40f006b4d0 0x7f40f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.279+0000 7f4108926640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f40f006b4d0 0x7f40f006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f40f4035000 tx=0x7f40f4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.283+0000 7f41017fa640 1 -- 172.21.15.94:0/227803604 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41040ac9a0 con 0x7f41040c7720 2024-03-06T07:18:59.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.453+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f41040ad160 con 0x7f41040c7720 2024-03-06T07:18:59.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.453+0000 7f41017fa640 1 -- 172.21.15.94:0/227803604 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f40fc057de0 con 0x7f41040c7720 2024-03-06T07:18:59.455 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.455+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f40f006b4d0 msgr2=0x7f40f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.455+0000 7f410ab8a640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f40f006b4d0 0x7f40f006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f40f4035000 tx=0x7f40f4008030 comp rx=0 tx=0).stop 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.455+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c7720 msgr2=0x7f41040ac460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.455+0000 7f410ab8a640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c7720 0x7f41040ac460 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c780 tx=0x7f40fc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 shutdown_connections 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f40f006b4d0 0x7f40f006d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41040c7720 0x7f41040ac460 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41040c5c60 0x7f41040abf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 --2- 172.21.15.94:0/227803604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41040c5880 0x7f4104148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 >> 172.21.15.94:0/227803604 conn(0x7f41040aa630 msgr2=0x7f410406f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 shutdown_connections 2024-03-06T07:18:59.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.456+0000 7f410ab8a640 1 -- 172.21.15.94:0/227803604 wait complete. 2024-03-06T07:18:59.469 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:18:59.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.568+0000 7f0892b20640 1 Processor -- start 2024-03-06T07:18:59.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.568+0000 7f0892b20640 1 -- start start 2024-03-06T07:18:59.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f0892b20640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f0892b20640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f088c0c5860 0x7f088c06f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f0892b20640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088c0c5cc0 con 0x7f088c1586f0 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f0892b20640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088c06df40 con 0x7f088c0c5860 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f0892b20640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c1591c0 con 0x7f088c0c5480 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f088bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f088c0c5860 0x7f088c06f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f088bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f088c0c5860 0x7f088c06f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39174/0 (socket says 172.21.15.94:39174) 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f088bfff640 1 -- 172.21.15.94:0/1654021031 learned_addr learned my addr 172.21.15.94:0/1654021031 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.569+0000 7f08908bc640 1 --2- 172.21.15.94:0/1654021031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.570+0000 7f088bfff640 1 -- 172.21.15.94:0/1654021031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 msgr2=0x7f088c155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.570+0000 7f088bfff640 1 --2- 172.21.15.94:0/1654021031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.570+0000 7f088bfff640 1 -- 172.21.15.94:0/1654021031 >> v1:172.21.15.94:6789/0 conn(0x7f088c1586f0 legacy=0x7f088c0724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.570+0000 7f088bfff640 1 -- 172.21.15.94:0/1654021031 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c1594a0 con 0x7f088c0c5860 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.570+0000 7f088b7fe640 1 -- 172.21.15.94:0/1654021031 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494100656 0 0) 0x7f088c0c5cc0 con 0x7f088c1586f0 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.570+0000 7f088bfff640 1 --2- 172.21.15.94:0/1654021031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c06f6a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0880009870 tx=0x7f088002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882389ee89dd41a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.570+0000 7f088b7fe640 1 -- 172.21.15.94:0/1654021031 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880035020 con 0x7f088c0c5860 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f088b7fe640 1 -- 172.21.15.94:0/1654021031 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0880031640 con 0x7f088c0c5860 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f088b7fe640 1 -- 172.21.15.94:0/1654021031 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088002d040 con 0x7f088c0c5860 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 -- 172.21.15.94:0/1654021031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 msgr2=0x7f088c06f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 --2- 172.21.15.94:0/1654021031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c06f6a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0880009870 tx=0x7f088002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 -- 172.21.15.94:0/1654021031 shutdown_connections 2024-03-06T07:18:59.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 --2- 172.21.15.94:0/1654021031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c06f6a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 --2- 172.21.15.94:0/1654021031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 -- 172.21.15.94:0/1654021031 >> 172.21.15.94:0/1654021031 conn(0x7f088c06b1a0 msgr2=0x7f088c06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:59.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 -- 172.21.15.94:0/1654021031 shutdown_connections 2024-03-06T07:18:59.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.571+0000 7f0892b20640 1 -- 172.21.15.94:0/1654021031 wait complete. 2024-03-06T07:18:59.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.572+0000 7f0892b20640 1 Processor -- start 2024-03-06T07:18:59.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.572+0000 7f0892b20640 1 -- start start 2024-03-06T07:18:59.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f0892b20640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c1e1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f0892b20640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c0ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f0892b20640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f088c1586f0 0x7f088c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f0892b20640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c152a20 con 0x7f088c0c5860 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f0892b20640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088c155680 con 0x7f088c1586f0 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f0892b20640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088c155050 con 0x7f088c0c5480 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c0ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c0ac4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39180/0 (socket says 172.21.15.94:39180) 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 -- 172.21.15.94:0/3296184254 learned_addr learned my addr 172.21.15.94:0/3296184254 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f08908bc640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c1e1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f08910bd640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f088c1586f0 0x7f088c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 -- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 msgr2=0x7f088c1e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c1e1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 -- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f088c1586f0 msgr2=0x7f088c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f088c1586f0 0x7f088c0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.573+0000 7f088bfff640 1 -- 172.21.15.94:0/3296184254 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c0c3250 con 0x7f088c0c5860 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.574+0000 7f088bfff640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c0ac4e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f08800090b0 tx=0x7f08800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.574+0000 7f08897fa640 1 -- 172.21.15.94:0/3296184254 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880040070 con 0x7f088c0c5860 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.574+0000 7f08897fa640 1 -- 172.21.15.94:0/3296184254 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08800399f0 con 0x7f088c0c5860 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.574+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088c0c3de0 con 0x7f088c0c5860 2024-03-06T07:18:59.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.574+0000 7f08897fa640 1 -- 172.21.15.94:0/3296184254 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880035040 con 0x7f088c0c5860 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.574+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088c0c4aa0 con 0x7f088c0c5860 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.575+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088c0c3fb0 con 0x7f088c0c5860 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.575+0000 7f08897fa640 1 -- 172.21.15.94:0/3296184254 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0880038020 con 0x7f088c0c5860 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.576+0000 7f08897fa640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f087806b4d0 0x7f087806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.576+0000 7f08897fa640 1 -- 172.21.15.94:0/3296184254 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f08800abdd0 con 0x7f088c0c5860 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.576+0000 7f08908bc640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f087806b4d0 0x7f087806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.576+0000 7f08908bc640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f087806b4d0 0x7f087806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f087c009b40 tx=0x7f087c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.580+0000 7f08897fa640 1 -- 172.21.15.94:0/3296184254 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f088007a290 con 0x7f088c0c5860 2024-03-06T07:18:59.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.786+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f088c04f330 con 0x7f088c0c5860 2024-03-06T07:18:59.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.786+0000 7f08897fa640 1 -- 172.21.15.94:0/3296184254 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7f088007e330 con 0x7f088c0c5860 2024-03-06T07:18:59.789 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:18:59.789 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:18:59.789 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:18:59.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.788+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f087806b4d0 msgr2=0x7f087806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.788+0000 7f0892b20640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f087806b4d0 0x7f087806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f087c009b40 tx=0x7f087c009190 comp rx=0 tx=0).stop 2024-03-06T07:18:59.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.788+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 msgr2=0x7f088c0ac4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.788+0000 7f0892b20640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c0ac4e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f08800090b0 tx=0x7f08800046c0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.788+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 shutdown_connections 2024-03-06T07:18:59.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.788+0000 7f0892b20640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f087806b4d0 0x7f087806d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.789+0000 7f0892b20640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f088c1586f0 0x7f088c0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.789+0000 7f0892b20640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f088c0c5860 0x7f088c0ac4e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.789+0000 7f0892b20640 1 --2- 172.21.15.94:0/3296184254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f088c0c5480 0x7f088c1e1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.789+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 >> 172.21.15.94:0/3296184254 conn(0x7f088c06b1a0 msgr2=0x7f088c06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:59.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.789+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 shutdown_connections 2024-03-06T07:18:59.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.789+0000 7f0892b20640 1 -- 172.21.15.94:0/3296184254 wait complete. 2024-03-06T07:18:59.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:18:59.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.900+0000 7fd4bf2f4640 1 Processor -- start 2024-03-06T07:18:59.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.900+0000 7fd4bf2f4640 1 -- start start 2024-03-06T07:18:59.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bf2f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd4b8152520 0x7fd4b806ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bf2f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4b8155c80 0x7fd4b80c12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bf2f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bf2f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b80c1dc0 con 0x7fd4b8155c80 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bf2f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b80c1ae0 con 0x7fd4b8152520 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bf2f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b806e450 con 0x7fd4b80720a0 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4b8155c80 0x7fd4b80c12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4b8155c80 0x7fd4b80c12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39192/0 (socket says 172.21.15.94:39192) 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 -- 172.21.15.94:0/1116629037 learned_addr learned my addr 172.21.15.94:0/1116629037 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bd891640 1 --2- 172.21.15.94:0/1116629037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bd090640 1 --2- 172.21.15.94:0/1116629037 >> v2:172.21.15.145:3300/0 conn(0x7fd4b8152520 0x7fd4b806ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 -- 172.21.15.94:0/1116629037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 msgr2=0x7fd4b806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 --2- 172.21.15.94:0/1116629037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 -- 172.21.15.94:0/1116629037 >> v2:172.21.15.145:3300/0 conn(0x7fd4b8152520 msgr2=0x7fd4b806ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:18:59.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 --2- 172.21.15.94:0/1116629037 >> v2:172.21.15.145:3300/0 conn(0x7fd4b8152520 0x7fd4b806ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.901+0000 7fd4bc88f640 1 -- 172.21.15.94:0/1116629037 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b806e730 con 0x7fd4b8155c80 2024-03-06T07:18:59.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bc88f640 1 --2- 172.21.15.94:0/1116629037 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b80c12b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80c1f80 tx=0x7fd4a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec27b3aa946df328 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4affff640 1 -- 172.21.15.94:0/1116629037 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8035020 con 0x7fd4b8155c80 2024-03-06T07:18:59.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4affff640 1 -- 172.21.15.94:0/1116629037 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a8031870 con 0x7fd4b8155c80 2024-03-06T07:18:59.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4affff640 1 -- 172.21.15.94:0/1116629037 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8031c40 con 0x7fd4b8155c80 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/1116629037 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 msgr2=0x7fd4b80c12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/1116629037 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b80c12b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80c1f80 tx=0x7fd4a802b910 comp rx=0 tx=0).stop 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/1116629037 shutdown_connections 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/1116629037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/1116629037 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b80c12b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/1116629037 >> v2:172.21.15.145:3300/0 conn(0x7fd4b8152520 0x7fd4b806ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/1116629037 >> 172.21.15.94:0/1116629037 conn(0x7fd4b80c76b0 msgr2=0x7fd4b80c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/1116629037 shutdown_connections 2024-03-06T07:18:59.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.902+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/1116629037 wait complete. 2024-03-06T07:18:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.903+0000 7fd4bf2f4640 1 Processor -- start 2024-03-06T07:18:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bf2f4640 1 -- start start 2024-03-06T07:18:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bf2f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b8154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bf2f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4b8152520 0x7fd4b81eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bf2f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b81eb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bf2f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b8152900 con 0x7fd4b8155c80 2024-03-06T07:18:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bf2f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b80c17f0 con 0x7fd4b8152520 2024-03-06T07:18:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bf2f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b8152a80 con 0x7fd4b80720a0 2024-03-06T07:18:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bd891640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b81eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bd891640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b81eb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39196/0 (socket says 172.21.15.94:39196) 2024-03-06T07:18:59.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bd891640 1 -- 172.21.15.94:0/2116296665 learned_addr learned my addr 172.21.15.94:0/2116296665 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:18:59.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bd090640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b8154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.904+0000 7fd4bc88f640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4b8152520 0x7fd4b81eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bd891640 1 -- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 msgr2=0x7fd4b8154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bd891640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b8154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bd891640 1 -- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4b8152520 msgr2=0x7fd4b81eaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:18:59.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bd891640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4b8152520 0x7fd4b81eaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:18:59.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bd891640 1 -- 172.21.15.94:0/2116296665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b8155760 con 0x7fd4b8155c80 2024-03-06T07:18:59.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bd891640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b81eb030 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000efc0 tx=0x7fd4b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4adffb640 1 -- 172.21.15.94:0/2116296665 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0018020 con 0x7fd4b8155c80 2024-03-06T07:18:59.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4adffb640 1 -- 172.21.15.94:0/2116296665 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b0014480 con 0x7fd4b8155c80 2024-03-06T07:18:59.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b81dea00 con 0x7fd4b8155c80 2024-03-06T07:18:59.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4adffb640 1 -- 172.21.15.94:0/2116296665 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0014870 con 0x7fd4b8155c80 2024-03-06T07:18:59.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.905+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b81eecf0 con 0x7fd4b8155c80 2024-03-06T07:18:59.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.906+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b804f330 con 0x7fd4b8155c80 2024-03-06T07:18:59.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.907+0000 7fd4adffb640 1 -- 172.21.15.94:0/2116296665 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4b0005d10 con 0x7fd4b8155c80 2024-03-06T07:18:59.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.907+0000 7fd4adffb640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd49c06b4d0 0x7fd49c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:18:59.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.907+0000 7fd4adffb640 1 -- 172.21.15.94:0/2116296665 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd4b008b140 con 0x7fd4b8155c80 2024-03-06T07:18:59.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.907+0000 7fd4bd090640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd49c06b4d0 0x7fd49c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:18:59.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.908+0000 7fd4bd090640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd49c06b4d0 0x7fd49c06d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8158d80 tx=0x7fd4b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:18:59.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:18:59.911+0000 7fd4adffb640 1 -- 172.21.15.94:0/2116296665 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4b0022030 con 0x7fd4b8155c80 2024-03-06T07:19:00.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.083+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4b80c0b70 con 0x7fd4b8155c80 2024-03-06T07:19:00.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.085+0000 7fd4adffb640 1 -- 172.21.15.94:0/2116296665 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18903 (secure 0 0 0) 0x7fd4b0010040 con 0x7fd4b8155c80 2024-03-06T07:19:00.087 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:00.087 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":26,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:58.142419+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd49c06b4d0 msgr2=0x7fd49c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd49c06b4d0 0x7fd49c06d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8158d80 tx=0x7fd4b4009190 comp rx=0 tx=0).stop 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 msgr2=0x7fd4b81eb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b81eb030 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000efc0 tx=0x7fd4b000c490 comp rx=0 tx=0).stop 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 shutdown_connections 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd49c06b4d0 0x7fd49c06d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4b8155c80 0x7fd4b81eb030 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4b8152520 0x7fd4b81eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 --2- 172.21.15.94:0/2116296665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4b80720a0 0x7fd4b8154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 >> 172.21.15.94:0/2116296665 conn(0x7fd4b80c76b0 msgr2=0x7fd4b80c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:00.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 shutdown_connections 2024-03-06T07:19:00.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.087+0000 7fd4bf2f4640 1 -- 172.21.15.94:0/2116296665 wait complete. 2024-03-06T07:19:00.100 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:19:00.100 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:00.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.198+0000 7ff4134f4640 1 Processor -- start 2024-03-06T07:19:00.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.199+0000 7ff4134f4640 1 -- start start 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.199+0000 7ff4134f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff40c152960 0x7ff40c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.199+0000 7ff4134f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff40c1592d0 0x7ff40c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.199+0000 7ff4134f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c0ac260 0x7ff40c155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.199+0000 7ff4134f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c066e30 con 0x7ff40c1592d0 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.199+0000 7ff4134f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c152d40 con 0x7ff40c152960 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.199+0000 7ff4134f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c0c1b50 con 0x7ff40c0ac260 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff40c1592d0 0x7ff40c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff40c1592d0 0x7ff40c1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39202/0 (socket says 172.21.15.94:39202) 2024-03-06T07:19:00.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 -- 172.21.15.94:0/255897252 learned_addr learned my addr 172.21.15.94:0/255897252 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:00.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff411a91640 1 --2- 172.21.15.94:0/255897252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c0ac260 0x7ff40c155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff411290640 1 --2- 172.21.15.94:0/255897252 >> v2:172.21.15.145:3300/0 conn(0x7ff40c152960 0x7ff40c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 -- 172.21.15.94:0/255897252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c0ac260 msgr2=0x7ff40c155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:00.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 --2- 172.21.15.94:0/255897252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c0ac260 0x7ff40c155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 -- 172.21.15.94:0/255897252 >> v2:172.21.15.145:3300/0 conn(0x7ff40c152960 msgr2=0x7ff40c06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:00.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 --2- 172.21.15.94:0/255897252 >> v2:172.21.15.145:3300/0 conn(0x7ff40c152960 0x7ff40c06dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.200+0000 7ff410a8f640 1 -- 172.21.15.94:0/255897252 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c155d50 con 0x7ff40c1592d0 2024-03-06T07:19:00.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff410a8f640 1 --2- 172.21.15.94:0/255897252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c1592d0 0x7ff40c1552d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff40c06b1c0 tx=0x7ff40002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a809351748da6e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff40bfff640 1 -- 172.21.15.94:0/255897252 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400035020 con 0x7ff40c1592d0 2024-03-06T07:19:00.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff40bfff640 1 -- 172.21.15.94:0/255897252 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff400031870 con 0x7ff40c1592d0 2024-03-06T07:19:00.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff40bfff640 1 -- 172.21.15.94:0/255897252 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400031c40 con 0x7ff40c1592d0 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 -- 172.21.15.94:0/255897252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c1592d0 msgr2=0x7ff40c1552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 --2- 172.21.15.94:0/255897252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c1592d0 0x7ff40c1552d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff40c06b1c0 tx=0x7ff40002b910 comp rx=0 tx=0).stop 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 -- 172.21.15.94:0/255897252 shutdown_connections 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 --2- 172.21.15.94:0/255897252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c0ac260 0x7ff40c155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 --2- 172.21.15.94:0/255897252 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c1592d0 0x7ff40c1552d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 --2- 172.21.15.94:0/255897252 >> v2:172.21.15.145:3300/0 conn(0x7ff40c152960 0x7ff40c06dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 -- 172.21.15.94:0/255897252 >> 172.21.15.94:0/255897252 conn(0x7ff40c0c70b0 msgr2=0x7ff40c0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 -- 172.21.15.94:0/255897252 shutdown_connections 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.201+0000 7ff4134f4640 1 -- 172.21.15.94:0/255897252 wait complete. 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.202+0000 7ff4134f4640 1 Processor -- start 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.202+0000 7ff4134f4640 1 -- start start 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff4134f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0ac260 0x7ff40c06b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff4134f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c152960 0x7ff40c14af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff4134f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c14bea0 0x7ff40c1539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff4134f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c159760 con 0x7ff40c14bea0 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff4134f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c0672c0 con 0x7ff40c0ac260 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff4134f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c0c20a0 con 0x7ff40c152960 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff411a91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c14bea0 0x7ff40c1539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff411a91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c14bea0 0x7ff40c1539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39212/0 (socket says 172.21.15.94:39212) 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff411290640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0ac260 0x7ff40c06b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff411a91640 1 -- 172.21.15.94:0/75318631 learned_addr learned my addr 172.21.15.94:0/75318631 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff410a8f640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c152960 0x7ff40c14af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff411a91640 1 -- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c152960 msgr2=0x7ff40c14af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.203+0000 7ff411a91640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c152960 0x7ff40c14af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff411a91640 1 -- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0ac260 msgr2=0x7ff40c06b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff411a91640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0ac260 0x7ff40c06b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff411a91640 1 -- 172.21.15.94:0/75318631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c1df520 con 0x7ff40c14bea0 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff411a91640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c14bea0 0x7ff40c1539a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00e830 tx=0x7ff3fc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff409ffb640 1 -- 172.21.15.94:0/75318631 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc017070 con 0x7ff40c14bea0 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff409ffb640 1 -- 172.21.15.94:0/75318631 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc00c940 con 0x7ff40c14bea0 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff409ffb640 1 -- 172.21.15.94:0/75318631 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc00cd30 con 0x7ff40c14bea0 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40c1531e0 con 0x7ff40c14bea0 2024-03-06T07:19:00.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.204+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40c153480 con 0x7ff40c14bea0 2024-03-06T07:19:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.205+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40c0c2630 con 0x7ff40c14bea0 2024-03-06T07:19:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.206+0000 7ff409ffb640 1 -- 172.21.15.94:0/75318631 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3fc01a070 con 0x7ff40c14bea0 2024-03-06T07:19:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.206+0000 7ff409ffb640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b4d0 0x7ff3f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.206+0000 7ff409ffb640 1 -- 172.21.15.94:0/75318631 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff3fc08a4b0 con 0x7ff40c14bea0 2024-03-06T07:19:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.206+0000 7ff411290640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b4d0 0x7ff3f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.206+0000 7ff411290640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b4d0 0x7ff3f806d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff40c154a20 tx=0x7ff40400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.210+0000 7ff409ffb640 1 -- 172.21.15.94:0/75318631 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff40c0c2630 con 0x7ff40c14bea0 2024-03-06T07:19:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.417+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff40c04f330 con 0x7ff40c14bea0 2024-03-06T07:19:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.418+0000 7ff409ffb640 1 -- 172.21.15.94:0/75318631 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7ff3fc058970 con 0x7ff40c14bea0 2024-03-06T07:19:00.420 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:00.420 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:19:00.420 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b4d0 msgr2=0x7ff3f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b4d0 0x7ff3f806d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff40c154a20 tx=0x7ff40400b040 comp rx=0 tx=0).stop 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c14bea0 msgr2=0x7ff40c1539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c14bea0 0x7ff40c1539a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00e830 tx=0x7ff3fc00ecf0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 shutdown_connections 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b4d0 0x7ff3f806d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c14bea0 0x7ff40c1539a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c152960 0x7ff40c14af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 --2- 172.21.15.94:0/75318631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0ac260 0x7ff40c06b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 >> 172.21.15.94:0/75318631 conn(0x7ff40c0c70b0 msgr2=0x7ff40c0c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:00.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 shutdown_connections 2024-03-06T07:19:00.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.420+0000 7ff4134f4640 1 -- 172.21.15.94:0/75318631 wait complete. 2024-03-06T07:19:00.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:00.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.531+0000 7f46e7e88640 1 Processor -- start 2024-03-06T07:19:00.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.531+0000 7f46e7e88640 1 -- start start 2024-03-06T07:19:00.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e7e88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f46e01550c0 0x7f46e01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e7e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e7e88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e00aa660 con 0x7f46e01559e0 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e7e88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e00c4610 con 0x7f46e01550c0 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e7e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00aa940 con 0x7f46e00aa180 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e5423640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f46e01559e0 0x7f46e014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:41028/0 (socket says 172.21.15.94:41028) 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e5423640 1 -- 172.21.15.94:0/1713342488 learned_addr learned my addr 172.21.15.94:0/1713342488 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e5c24640 1 --2- 172.21.15.94:0/1713342488 >> v2:172.21.15.145:3300/0 conn(0x7f46e01550c0 0x7f46e01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e6425640 1 --2- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e4c22640 1 -- 172.21.15.94:0/1713342488 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331382113 0 0) 0x7f46e00aa660 con 0x7f46e01559e0 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e4c22640 1 -- 172.21.15.94:0/1713342488 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46cc003440 con 0x7f46e01559e0 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e5c24640 1 -- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 msgr2=0x7f46e00c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e5c24640 1 --2- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e5c24640 1 -- 172.21.15.94:0/1713342488 >> v1:172.21.15.145:6789/0 conn(0x7f46e01559e0 legacy=0x7f46e014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e5c24640 1 -- 172.21.15.94:0/1713342488 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e00c4860 con 0x7f46e01550c0 2024-03-06T07:19:00.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.532+0000 7f46e6425640 1 --2- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.533+0000 7f46e5c24640 1 --2- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 0x7f46e01554a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19754f37e372e9f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.533+0000 7f46e4c22640 1 -- 172.21.15.94:0/1713342488 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc038070 con 0x7f46e01550c0 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.533+0000 7f46e4c22640 1 -- 172.21.15.94:0/1713342488 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc031700 con 0x7f46e01550c0 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.533+0000 7f46e4c22640 1 -- 172.21.15.94:0/1713342488 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc031ad0 con 0x7f46e01550c0 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 -- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 msgr2=0x7f46e01554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 0x7f46e01554a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0097a0 tx=0x7f46dc02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 -- 172.21.15.94:0/1713342488 shutdown_connections 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1713342488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 0x7f46e01554a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 -- 172.21.15.94:0/1713342488 >> 172.21.15.94:0/1713342488 conn(0x7f46e00c6710 msgr2=0x7f46e00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:00.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 -- 172.21.15.94:0/1713342488 shutdown_connections 2024-03-06T07:19:00.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.534+0000 7f46e7e88640 1 -- 172.21.15.94:0/1713342488 wait complete. 2024-03-06T07:19:00.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.535+0000 7f46e7e88640 1 Processor -- start 2024-03-06T07:19:00.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.535+0000 7f46e7e88640 1 -- start start 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e7e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e7e88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 0x7f46e014b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e7e88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46e01559e0 0x7f46e014bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e7e88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e014f310 con 0x7f46e01559e0 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e7e88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00c1560 con 0x7f46e01550c0 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e7e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e014f490 con 0x7f46e00aa180 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46e01559e0 0x7f46e014bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e5c24640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46e01559e0 0x7f46e014bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39226/0 (socket says 172.21.15.94:39226) 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 -- 172.21.15.94:0/1788081752 learned_addr learned my addr 172.21.15.94:0/1788081752 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e5423640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 0x7f46e014b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 -- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 msgr2=0x7f46e00a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 -- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 msgr2=0x7f46e014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 0x7f46e014b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 -- 172.21.15.94:0/1788081752 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e01e9d30 con 0x7f46e01559e0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e6425640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46e01559e0 0x7f46e014bd10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f46d800efc0 tx=0x7f46d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46d67fc640 1 -- 172.21.15.94:0/1788081752 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8018030 con 0x7f46e01559e0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46d67fc640 1 -- 172.21.15.94:0/1788081752 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d800cd30 con 0x7f46e01559e0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e01ea010 con 0x7f46e01559e0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.536+0000 7f46d67fc640 1 -- 172.21.15.94:0/1788081752 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8003bb0 con 0x7f46e01559e0 2024-03-06T07:19:00.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.537+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01ea5d0 con 0x7f46e01559e0 2024-03-06T07:19:00.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.537+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e004f330 con 0x7f46e01559e0 2024-03-06T07:19:00.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.538+0000 7f46d67fc640 1 -- 172.21.15.94:0/1788081752 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46d8005d10 con 0x7f46e01559e0 2024-03-06T07:19:00.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.538+0000 7f46d67fc640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46c406b4d0 0x7f46c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.538+0000 7f46d67fc640 1 -- 172.21.15.94:0/1788081752 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f46d808ada0 con 0x7f46e01559e0 2024-03-06T07:19:00.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.538+0000 7f46e5c24640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46c406b4d0 0x7f46c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.539+0000 7f46e5c24640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46c406b4d0 0x7f46c406d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f46dc035000 tx=0x7f46dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.543+0000 7f46d67fc640 1 -- 172.21.15.94:0/1788081752 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46d8022070 con 0x7f46e01559e0 2024-03-06T07:19:00.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.749+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46e0068450 con 0x7f46e01559e0 2024-03-06T07:19:00.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.750+0000 7f46d67fc640 1 -- 172.21.15.94:0/1788081752 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4981 (secure 0 0 0) 0x7f46d8010040 con 0x7f46e01559e0 2024-03-06T07:19:00.752 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:00.752 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/1806515100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1806515100},{"type":"v1","addr":"172.21.15.94:6833","nonce":1806515100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/737576548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":737576548},{"type":"v1","addr":"172.21.15.94:6835","nonce":737576548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1510465691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1510465691},{"type":"v1","addr":"172.21.15.145:6836","nonce":1510465691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:18:08.851499+0000","modified":"2024-03-06T07:18:31.055922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34136":{"gid":34136,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2223795314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2223795314},{"type":"v1","addr":"172.21.15.145:6837","nonce":2223795314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34136,"qdb_cluster":[34136]},"id":1}]} 2024-03-06T07:19:00.752 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 9 2024-03-06T07:19:00.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46c406b4d0 msgr2=0x7f46c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46c406b4d0 0x7f46c406d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f46dc035000 tx=0x7f46dc008030 comp rx=0 tx=0).stop 2024-03-06T07:19:00.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46e01559e0 msgr2=0x7f46e014bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46e01559e0 0x7f46e014bd10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f46d800efc0 tx=0x7f46d800c490 comp rx=0 tx=0).stop 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 shutdown_connections 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46c406b4d0 0x7f46c406d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46e01559e0 0x7f46e014bd10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46e01550c0 0x7f46e014b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 --2- 172.21.15.94:0/1788081752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46e00aa180 0x7f46e00a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.751+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 >> 172.21.15.94:0/1788081752 conn(0x7f46e00c6710 msgr2=0x7f46e00c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.752+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 shutdown_connections 2024-03-06T07:19:00.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.752+0000 7f46e7e88640 1 -- 172.21.15.94:0/1788081752 wait complete. 2024-03-06T07:19:00.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:00.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.863+0000 7ff427bc7640 1 Processor -- start 2024-03-06T07:19:00.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.863+0000 7ff427bc7640 1 -- start start 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff427bc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff4201529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff427bc7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff420155a40 0x7ff420155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff427bc7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff420156410 con 0x7ff4200c1aa0 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff427bc7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4200c2680 con 0x7ff420155a40 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff427bc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200c28d0 con 0x7ff42014f790 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff425162640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff4200c1aa0 0x7ff4200c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60188/0 (socket says 172.21.15.94:60188) 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff425162640 1 -- 172.21.15.94:0/1384581356 learned_addr learned my addr 172.21.15.94:0/1384581356 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff426164640 1 --2- 172.21.15.94:0/1384581356 >> v2:172.21.15.145:3300/0 conn(0x7ff420155a40 0x7ff420155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff425963640 1 --2- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff4201529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff424961640 1 -- 172.21.15.94:0/1384581356 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732360654 0 0) 0x7ff420156410 con 0x7ff4200c1aa0 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff424961640 1 -- 172.21.15.94:0/1384581356 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff40c003520 con 0x7ff4200c1aa0 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff426164640 1 -- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 msgr2=0x7ff4201529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff426164640 1 --2- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff4201529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff426164640 1 -- 172.21.15.94:0/1384581356 >> v1:172.21.15.94:6789/0 conn(0x7ff4200c1aa0 legacy=0x7ff4200c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff426164640 1 -- 172.21.15.94:0/1384581356 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4200c2bb0 con 0x7ff420155a40 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.864+0000 7ff425963640 1 --2- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff4201529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.865+0000 7ff426164640 1 --2- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 0x7ff420155ed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff41802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7512bb8b6d645928 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.865+0000 7ff424961640 1 -- 172.21.15.94:0/1384581356 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418039070 con 0x7ff420155a40 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.865+0000 7ff424961640 1 -- 172.21.15.94:0/1384581356 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41802cc50 con 0x7ff420155a40 2024-03-06T07:19:00.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.865+0000 7ff424961640 1 -- 172.21.15.94:0/1384581356 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418034440 con 0x7ff420155a40 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.865+0000 7ff427bc7640 1 -- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 msgr2=0x7ff420155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.865+0000 7ff427bc7640 1 --2- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 0x7ff420155ed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff41802bea0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.866+0000 7ff427bc7640 1 -- 172.21.15.94:0/1384581356 shutdown_connections 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.866+0000 7ff427bc7640 1 --2- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 0x7ff420155ed0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.866+0000 7ff427bc7640 1 --2- 172.21.15.94:0/1384581356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff4201529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.866+0000 7ff427bc7640 1 -- 172.21.15.94:0/1384581356 >> 172.21.15.94:0/1384581356 conn(0x7ff4200c6710 msgr2=0x7ff4200c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.866+0000 7ff427bc7640 1 -- 172.21.15.94:0/1384581356 shutdown_connections 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.866+0000 7ff427bc7640 1 -- 172.21.15.94:0/1384581356 wait complete. 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 Processor -- start 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 -- start start 2024-03-06T07:19:00.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200c1aa0 0x7ff4201dfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff420149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 0x7ff42014a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200c0d60 con 0x7ff4200c1aa0 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff420155520 con 0x7ff420155a40 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff427bc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200c5e30 con 0x7ff42014f790 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff425963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200c1aa0 0x7ff4201dfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff425963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200c1aa0 0x7ff4201dfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39228/0 (socket says 172.21.15.94:39228) 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.867+0000 7ff425963640 1 -- 172.21.15.94:0/58285923 learned_addr learned my addr 172.21.15.94:0/58285923 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff425162640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff420149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff426164640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 0x7ff42014a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff425963640 1 -- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 msgr2=0x7ff420149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff425963640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff420149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff425963640 1 -- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 msgr2=0x7ff42014a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:00.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff425963640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 0x7ff42014a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff425963640 1 -- 172.21.15.94:0/58285923 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4201546f0 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff425963640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200c1aa0 0x7ff4201dfea0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00ebe0 tx=0x7ff41c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff4167fc640 1 -- 172.21.15.94:0/58285923 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c017070 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff4167fc640 1 -- 172.21.15.94:0/58285923 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c00cc30 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff4167fc640 1 -- 172.21.15.94:0/58285923 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c013470 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4201dec50 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.868+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff420154380 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.869+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff420069460 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.869+0000 7ff4167fc640 1 -- 172.21.15.94:0/58285923 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff41c01a070 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.870+0000 7ff4167fc640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff40406b4d0 0x7ff40406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.870+0000 7ff4167fc640 1 -- 172.21.15.94:0/58285923 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff41c08a790 con 0x7ff4200c1aa0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.870+0000 7ff425162640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff40406b4d0 0x7ff40406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:00.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.870+0000 7ff425162640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff40406b4d0 0x7ff40406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff4200a7f20 tx=0x7ff410009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:00.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:00.874+0000 7ff4167fc640 1 -- 172.21.15.94:0/58285923 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff420069460 con 0x7ff4200c1aa0 2024-03-06T07:19:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.046+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff42004f330 con 0x7ff4200c1aa0 2024-03-06T07:19:01.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.047+0000 7ff4167fc640 1 -- 172.21.15.94:0/58285923 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18903 (secure 0 0 0) 0x7ff41c058c50 con 0x7ff4200c1aa0 2024-03-06T07:19:01.049 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:01.049 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":26,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:18:58.142419+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:18:07.163676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:18:07.821715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-06T07:18:11.560181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":5,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff40406b4d0 msgr2=0x7ff40406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff40406b4d0 0x7ff40406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff4200a7f20 tx=0x7ff410009040 comp rx=0 tx=0).stop 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200c1aa0 msgr2=0x7ff4201dfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200c1aa0 0x7ff4201dfea0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00ebe0 tx=0x7ff41c00c520 comp rx=0 tx=0).stop 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 shutdown_connections 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff40406b4d0 0x7ff40406d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff420155a40 0x7ff42014a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff42014f790 0x7ff420149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 --2- 172.21.15.94:0/58285923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200c1aa0 0x7ff4201dfea0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 >> 172.21.15.94:0/58285923 conn(0x7ff4200c6710 msgr2=0x7ff4200a6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 shutdown_connections 2024-03-06T07:19:01.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.049+0000 7ff427bc7640 1 -- 172.21.15.94:0/58285923 wait complete. 2024-03-06T07:19:01.063 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.161+0000 7f9e2ad3e640 1 Processor -- start 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.161+0000 7f9e2ad3e640 1 -- start start 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e2ad3e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9e240c69f0 0x7f9e240c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e2ad3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 0x7f9e24155d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e2ad3e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e24155130 con 0x7f9e240c8950 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e2ad3e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e24155440 con 0x7f9e240c69f0 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e2ad3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240ad480 con 0x7f9e241558d0 2024-03-06T07:19:01.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e23fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9e240c8950 0x7f9e240acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60194/0 (socket says 172.21.15.94:60194) 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e23fff640 1 -- 172.21.15.94:0/2266693877 learned_addr learned my addr 172.21.15.94:0/2266693877 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2266693877 >> v2:172.21.15.145:3300/0 conn(0x7f9e240c69f0 0x7f9e240c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e292db640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 0x7f9e24155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.162+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206289272 0 0) 0x7f9e10001110 con 0x7f9e240c8950 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e14003520 con 0x7f9e240c8950 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 596710201 0 0) 0x7f9e14003520 con 0x7f9e240c8950 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 msgr2=0x7f9e24155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 0x7f9e24155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 >> v2:172.21.15.145:3300/0 conn(0x7f9e240c69f0 msgr2=0x7f9e240c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 --2- 172.21.15.94:0/2266693877 >> v2:172.21.15.145:3300/0 conn(0x7f9e240c69f0 0x7f9e240c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240c5d50 con 0x7f9e240c8950 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9e10001840 con 0x7f9e240c8950 2024-03-06T07:19:01.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e292db640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 0x7f9e24155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2266693877 >> v2:172.21.15.145:3300/0 conn(0x7f9e240c69f0 0x7f9e240c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 >> v1:172.21.15.94:6789/0 conn(0x7f9e240c8950 legacy=0x7f9e240acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e14004800 0x7f9e14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 0x7f9e140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24155440 con 0x7f9e140039b0 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240ad480 con 0x7f9e14004800 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1400b9c0 con 0x7f9e241558d0 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e292db640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 0x7f9e140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e23fff640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e14004800 0x7f9e14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e28ada640 1 -- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 msgr2=0x7f9e140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e241558d0 0x7f9e140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e28ada640 1 -- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e14004800 msgr2=0x7f9e14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e14004800 0x7f9e14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e28ada640 1 -- 172.21.15.94:0/2266693877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14003520 con 0x7f9e140039b0 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.163+0000 7f9e23fff640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e14004800 0x7f9e14004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.164+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e14003dd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009c90 tx=0x7f9e14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e241cb0312003af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.164+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18039070 con 0x7f9e140039b0 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.164+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1802c980 con 0x7f9e140039b0 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.164+0000 7f9e237fe640 1 -- 172.21.15.94:0/2266693877 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18009860 con 0x7f9e140039b0 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.165+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2266693877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 msgr2=0x7f9e14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.165+0000 7f9e2ad3e640 1 --2- 172.21.15.94:0/2266693877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e14003dd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009c90 tx=0x7f9e14005830 comp rx=0 tx=0).stop 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.165+0000 7f9e292db640 1 -- 172.21.15.94:0/2266693877 reap_dead start 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.165+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2266693877 shutdown_connections 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.165+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2266693877 >> 172.21.15.94:0/2266693877 conn(0x7f9e240cae70 msgr2=0x7f9e240cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:01.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.165+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2266693877 shutdown_connections 2024-03-06T07:19:01.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.165+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2266693877 wait complete. 2024-03-06T07:19:01.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.166+0000 7f9e2ad3e640 1 Processor -- start 2024-03-06T07:19:01.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.166+0000 7f9e2ad3e640 1 -- start start 2024-03-06T07:19:01.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e240c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e2414d030 0x7f9e2406c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e2414beb0 0x7f9e2414c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24152a20 con 0x7f9e140039b0 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240c6310 con 0x7f9e2414beb0 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24152ba0 con 0x7f9e2414d030 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e240c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e240c4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39246/0 (socket says 172.21.15.94:39246) 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 -- 172.21.15.94:0/2107278579 learned_addr learned my addr 172.21.15.94:0/2107278579 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e292db640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e2414beb0 0x7f9e2414c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e23fff640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e2414d030 0x7f9e2406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 -- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e2414d030 msgr2=0x7f9e2406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e2414d030 0x7f9e2406c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 -- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e2414beb0 msgr2=0x7f9e2414c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e2414beb0 0x7f9e2414c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 -- 172.21.15.94:0/2107278579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2414cc30 con 0x7f9e140039b0 2024-03-06T07:19:01.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e292db640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e2414beb0 0x7f9e2414c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e23fff640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e2414d030 0x7f9e2406c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e28ada640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e240c4a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009db0 tx=0x7f9e18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e217fa640 1 -- 172.21.15.94:0/2107278579 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18039050 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e217fa640 1 -- 172.21.15.94:0/2107278579 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18045070 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e241583a0 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e217fa640 1 -- 172.21.15.94:0/2107278579 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18009860 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.167+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2406ccd0 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.168+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e241581c0 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.169+0000 7f9e217fa640 1 -- 172.21.15.94:0/2107278579 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e1804a460 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.169+0000 7f9e217fa640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e0806b500 0x7f9e0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.169+0000 7f9e217fa640 1 -- 172.21.15.94:0/2107278579 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9e180ac860 con 0x7f9e140039b0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.169+0000 7f9e23fff640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e0806b500 0x7f9e0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.170+0000 7f9e23fff640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e0806b500 0x7f9e0806d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e100013d0 tx=0x7f9e10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:01.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.173+0000 7f9e217fa640 1 -- 172.21.15.94:0/2107278579 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e241581c0 con 0x7f9e140039b0 2024-03-06T07:19:01.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.381+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9e24075300 con 0x7f9e140039b0 2024-03-06T07:19:01.383 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:19:01.381+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:19:01.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.383+0000 7f9e217fa640 1 -- 172.21.15.94:0/2107278579 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9e24075300 con 0x7f9e140039b0 2024-03-06T07:19:01.385 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:19:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e0806b500 msgr2=0x7f9e0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e0806b500 0x7f9e0806d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e100013d0 tx=0x7f9e10009040 comp rx=0 tx=0).stop 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 msgr2=0x7f9e240c4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e240c4a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009db0 tx=0x7f9e18004730 comp rx=0 tx=0).stop 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 shutdown_connections 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e0806b500 0x7f9e0806d9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e2414beb0 0x7f9e2414c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e2414d030 0x7f9e2406c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 --2- 172.21.15.94:0/2107278579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e140039b0 0x7f9e240c4a20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.385+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 >> 172.21.15.94:0/2107278579 conn(0x7f9e240cae70 msgr2=0x7f9e241519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.386+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 shutdown_connections 2024-03-06T07:19:01.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.386+0000 7f9e2ad3e640 1 -- 172.21.15.94:0/2107278579 wait complete. 2024-03-06T07:19:01.389 INFO:tasks.ceph.mds.d.smithi145.stderr: -4787> 2024-03-06T07:18:03.075+0000 7f5727514640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:19:01.392 INFO:tasks.ceph.mds.d.smithi145.stderr: -3532> 2024-03-06T07:18:14.074+0000 7f5721508640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:19:01.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:19:01.420 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:19:01.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.499+0000 7fc75e2bb640 1 Processor -- start 2024-03-06T07:19:01.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.499+0000 7fc75e2bb640 1 -- start start 2024-03-06T07:19:01.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.500+0000 7fc75e2bb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc75806d840 0x7fc758073090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.500+0000 7fc75e2bb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc758159580 con 0x7fc7580c24e0 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.500+0000 7fc75e2bb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7580c36b0 con 0x7fc7580c69a0 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.500+0000 7fc75e2bb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7580c3460 con 0x7fc75806d840 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.500+0000 7fc75c858640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc7580c24e0 0x7fc7580c2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60198/0 (socket says 172.21.15.94:60198) 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.500+0000 7fc75c858640 1 -- 172.21.15.94:0/4212178522 learned_addr learned my addr 172.21.15.94:0/4212178522 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.501+0000 7fc757fff640 1 --2- 172.21.15.94:0/4212178522 >> v2:172.21.15.145:3300/0 conn(0x7fc75806d840 0x7fc758073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.501+0000 7fc756ffd640 1 -- 172.21.15.94:0/4212178522 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262498473 0 0) 0x7fc758159580 con 0x7fc7580c24e0 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.501+0000 7fc756ffd640 1 -- 172.21.15.94:0/4212178522 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc740003520 con 0x7fc7580c24e0 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.501+0000 7fc757fff640 1 -- 172.21.15.94:0/4212178522 >> v1:172.21.15.145:6789/0 conn(0x7fc7580c69a0 legacy=0x7fc758158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.501+0000 7fc756ffd640 1 -- 172.21.15.94:0/4212178522 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048245438 0 0) 0x7fc740003520 con 0x7fc7580c24e0 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.501+0000 7fc757fff640 1 -- 172.21.15.94:0/4212178522 >> v1:172.21.15.94:6789/0 conn(0x7fc7580c24e0 legacy=0x7fc7580c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.501+0000 7fc757fff640 1 -- 172.21.15.94:0/4212178522 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7580c3990 con 0x7fc75806d840 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc757fff640 1 --2- 172.21.15.94:0/4212178522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75806d840 0x7fc758073090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc7480097a0 tx=0x7fc74802b570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a26fcb3ba952e582 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc756ffd640 1 -- 172.21.15.94:0/4212178522 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748037070 con 0x7fc75806d840 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc756ffd640 1 -- 172.21.15.94:0/4212178522 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc748030510 con 0x7fc75806d840 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc756ffd640 1 -- 172.21.15.94:0/4212178522 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7500019f0 con 0x7fc75806d840 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4212178522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75806d840 msgr2=0x7fc758073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4212178522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75806d840 0x7fc758073090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc7480097a0 tx=0x7fc74802b570 comp rx=0 tx=0).stop 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4212178522 shutdown_connections 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4212178522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc75806d840 0x7fc758073090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4212178522 >> 172.21.15.94:0/4212178522 conn(0x7fc7580c5560 msgr2=0x7fc7580c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4212178522 shutdown_connections 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.502+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4212178522 wait complete. 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.503+0000 7fc75e2bb640 1 Processor -- start 2024-03-06T07:19:01.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75e2bb640 1 -- start start 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75e2bb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc75806d840 0x7fc75806c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75e2bb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7580c24e0 0x7fc7581ede70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75e2bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7580c69a0 0x7fc7581ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75e2bb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7580c3040 con 0x7fc7580c69a0 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75e2bb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7580c2050 con 0x7fc7580c24e0 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75e2bb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75806dc20 con 0x7fc75806d840 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7580c69a0 0x7fc7581ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7580c69a0 0x7fc7581ee3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39256/0 (socket says 172.21.15.94:39256) 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 -- 172.21.15.94:0/4130563242 learned_addr learned my addr 172.21.15.94:0/4130563242 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc757fff640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc75806d840 0x7fc75806c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc7577fe640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7580c24e0 0x7fc7581ede70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 -- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc75806d840 msgr2=0x7fc75806c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc75806d840 0x7fc75806c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 -- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7580c24e0 msgr2=0x7fc7581ede70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7580c24e0 0x7fc7581ede70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.504+0000 7fc75c858640 1 -- 172.21.15.94:0/4130563242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7580ac5b0 con 0x7fc7580c69a0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.505+0000 7fc75c858640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7580c69a0 0x7fc7581ee3b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc75000ca70 tx=0x7fc75000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.505+0000 7fc754ff9640 1 -- 172.21.15.94:0/4130563242 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750018070 con 0x7fc7580c69a0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.505+0000 7fc754ff9640 1 -- 172.21.15.94:0/4130563242 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75000a6a0 con 0x7fc7580c69a0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.505+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7580ac850 con 0x7fc7580c69a0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.505+0000 7fc754ff9640 1 -- 172.21.15.94:0/4130563242 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75000aa90 con 0x7fc7580c69a0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.505+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75806cb50 con 0x7fc7580c69a0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.506+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75804f330 con 0x7fc7580c69a0 2024-03-06T07:19:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.506+0000 7fc754ff9640 1 -- 172.21.15.94:0/4130563242 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc75001b070 con 0x7fc7580c69a0 2024-03-06T07:19:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.507+0000 7fc754ff9640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc74406b4d0 0x7fc74406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.507+0000 7fc754ff9640 1 -- 172.21.15.94:0/4130563242 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fc750089dc0 con 0x7fc7580c69a0 2024-03-06T07:19:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.507+0000 7fc757fff640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc74406b4d0 0x7fc74406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.508+0000 7fc757fff640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc74406b4d0 0x7fc74406d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc748034000 tx=0x7fc748008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:01.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.511+0000 7fc754ff9640 1 -- 172.21.15.94:0/4130563242 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc750058870 con 0x7fc7580c69a0 2024-03-06T07:19:01.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:01.724+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc7580c5940 con 0x7fc7580c69a0 2024-03-06T07:19:02.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.384+0000 7fc754ff9640 1 -- 172.21.15.94:0/4130563242 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc750058170 con 0x7fc7580c69a0 2024-03-06T07:19:02.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc74406b4d0 msgr2=0x7fc74406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:02.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc74406b4d0 0x7fc74406d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc748034000 tx=0x7fc748008030 comp rx=0 tx=0).stop 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7580c69a0 msgr2=0x7fc7581ee3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7580c69a0 0x7fc7581ee3b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc75000ca70 tx=0x7fc75000cf30 comp rx=0 tx=0).stop 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 shutdown_connections 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc74406b4d0 0x7fc74406d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7580c69a0 0x7fc7581ee3b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7580c24e0 0x7fc7581ede70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 --2- 172.21.15.94:0/4130563242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc75806d840 0x7fc75806c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 >> 172.21.15.94:0/4130563242 conn(0x7fc7580c5560 msgr2=0x7fc7580a9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 shutdown_connections 2024-03-06T07:19:02.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.386+0000 7fc75e2bb640 1 -- 172.21.15.94:0/4130563242 wait complete. 2024-03-06T07:19:02.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:19:02.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.504+0000 7f8772ec2640 1 Processor -- start 2024-03-06T07:19:02.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.504+0000 7f8772ec2640 1 -- start start 2024-03-06T07:19:02.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f8772ec2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f876c14f180 0x7f876c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:02.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f8772ec2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f876c1551d0 0x7f876c1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f8772ec2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c0ca000 0x7f876c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f8772ec2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876c156c00 con 0x7f876c1551d0 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f8772ec2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876c0c8f60 con 0x7f876c14f180 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f8772ec2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c0c9240 con 0x7f876c0ca000 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f876bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f876c1551d0 0x7f876c1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f876bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f876c1551d0 0x7f876c1566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39262/0 (socket says 172.21.15.94:39262) 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.505+0000 7f876bfff640 1 -- 172.21.15.94:0/106519217 learned_addr learned my addr 172.21.15.94:0/106519217 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f877145f640 1 --2- 172.21.15.94:0/106519217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c0ca000 0x7f876c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f8770c5e640 1 --2- 172.21.15.94:0/106519217 >> v2:172.21.15.145:3300/0 conn(0x7f876c14f180 0x7f876c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f876bfff640 1 -- 172.21.15.94:0/106519217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c0ca000 msgr2=0x7f876c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f876bfff640 1 --2- 172.21.15.94:0/106519217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c0ca000 0x7f876c06c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f876bfff640 1 -- 172.21.15.94:0/106519217 >> v2:172.21.15.145:3300/0 conn(0x7f876c14f180 msgr2=0x7f876c0c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:02.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f876bfff640 1 --2- 172.21.15.94:0/106519217 >> v2:172.21.15.145:3300/0 conn(0x7f876c14f180 0x7f876c0c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f876bfff640 1 -- 172.21.15.94:0/106519217 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c0c9520 con 0x7f876c1551d0 2024-03-06T07:19:02.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.506+0000 7f876bfff640 1 --2- 172.21.15.94:0/106519217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1566c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f876c073630 tx=0x7f875802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa5a86d0b163e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:02.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f876b7fe640 1 -- 172.21.15.94:0/106519217 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758035020 con 0x7f876c1551d0 2024-03-06T07:19:02.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f876b7fe640 1 -- 172.21.15.94:0/106519217 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758031870 con 0x7f876c1551d0 2024-03-06T07:19:02.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f876b7fe640 1 -- 172.21.15.94:0/106519217 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758031c40 con 0x7f876c1551d0 2024-03-06T07:19:02.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 -- 172.21.15.94:0/106519217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 msgr2=0x7f876c1566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:02.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 --2- 172.21.15.94:0/106519217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1566c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f876c073630 tx=0x7f875802b910 comp rx=0 tx=0).stop 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 -- 172.21.15.94:0/106519217 shutdown_connections 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 --2- 172.21.15.94:0/106519217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c0ca000 0x7f876c06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 --2- 172.21.15.94:0/106519217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1566c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 --2- 172.21.15.94:0/106519217 >> v2:172.21.15.145:3300/0 conn(0x7f876c14f180 0x7f876c0c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 -- 172.21.15.94:0/106519217 >> 172.21.15.94:0/106519217 conn(0x7f876c072d20 msgr2=0x7f876c0ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 -- 172.21.15.94:0/106519217 shutdown_connections 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.507+0000 7f8772ec2640 1 -- 172.21.15.94:0/106519217 wait complete. 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.508+0000 7f8772ec2640 1 Processor -- start 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.508+0000 7f8772ec2640 1 -- start start 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8772ec2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c0ca000 0x7f876c1484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8772ec2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c14f180 0x7f876c14d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:02.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8772ec2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1ec270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8772ec2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c14f560 con 0x7f876c1551d0 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8772ec2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876c0c9ae0 con 0x7f876c0ca000 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8772ec2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c14f6e0 con 0x7f876c14f180 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1ec270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39272/0 (socket says 172.21.15.94:39272) 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 -- 172.21.15.94:0/1080658982 learned_addr learned my addr 172.21.15.94:0/1080658982 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f876bfff640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c14f180 0x7f876c14d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8770c5e640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c0ca000 0x7f876c1484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 -- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c14f180 msgr2=0x7f876c14d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c14f180 0x7f876c14d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 -- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c0ca000 msgr2=0x7f876c1484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c0ca000 0x7f876c1484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:02.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f877145f640 1 -- 172.21.15.94:0/1080658982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c1535b0 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.509+0000 7f8770c5e640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c0ca000 0x7f876c1484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.510+0000 7f877145f640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1ec270 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8764007d20 tx=0x7f876400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.510+0000 7f87697fa640 1 -- 172.21.15.94:0/1080658982 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764014420 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.510+0000 7f87697fa640 1 -- 172.21.15.94:0/1080658982 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764019070 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.510+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876c153890 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.510+0000 7f87697fa640 1 -- 172.21.15.94:0/1080658982 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764014820 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.510+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876c153e50 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.511+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876c075300 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.511+0000 7f87697fa640 1 -- 172.21.15.94:0/1080658982 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87640149c0 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.512+0000 7f87697fa640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875006b4d0 0x7f875006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.512+0000 7f87697fa640 1 -- 172.21.15.94:0/1080658982 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f876408ae20 con 0x7f876c1551d0 2024-03-06T07:19:02.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.512+0000 7f8770c5e640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875006b4d0 0x7f875006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:02.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.512+0000 7f8770c5e640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875006b4d0 0x7f875006d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8760006fd0 tx=0x7f8760008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:02.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.516+0000 7f87697fa640 1 -- 172.21.15.94:0/1080658982 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f876c075300 con 0x7f876c1551d0 2024-03-06T07:19:02.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:02.687+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f876c04f330 con 0x7f876c1551d0 2024-03-06T07:19:03.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.385+0000 7f87697fa640 1 -- 172.21.15.94:0/1080658982 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7f876405f070 con 0x7f876c1551d0 2024-03-06T07:19:03.387 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:19:03.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875006b4d0 msgr2=0x7f875006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:03.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875006b4d0 0x7f875006d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8760006fd0 tx=0x7f8760008040 comp rx=0 tx=0).stop 2024-03-06T07:19:03.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 msgr2=0x7f876c1ec270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1ec270 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8764007d20 tx=0x7f876400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 shutdown_connections 2024-03-06T07:19:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875006b4d0 0x7f875006d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f876c1551d0 0x7f876c1ec270 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876c14f180 0x7f876c14d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 --2- 172.21.15.94:0/1080658982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c0ca000 0x7f876c1484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 >> 172.21.15.94:0/1080658982 conn(0x7f876c072d20 msgr2=0x7f876c0c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.389+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 shutdown_connections 2024-03-06T07:19:03.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.390+0000 7f8772ec2640 1 -- 172.21.15.94:0/1080658982 wait complete. 2024-03-06T07:19:03.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:19:03.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.512+0000 7f5e49956640 1 Processor -- start 2024-03-06T07:19:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.512+0000 7f5e49956640 1 -- start start 2024-03-06T07:19:03.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.512+0000 7f5e49956640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 0x7f5e440c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.512+0000 7f5e49956640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5e441566c0 0x7f5e4406c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.512+0000 7f5e49956640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e440c8f60 con 0x7f5e441551d0 2024-03-06T07:19:03.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.512+0000 7f5e49956640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e440c9240 con 0x7f5e441566c0 2024-03-06T07:19:03.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.512+0000 7f5e49956640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e440c9520 con 0x7f5e4414f180 2024-03-06T07:19:03.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e3a7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5e441551d0 0x7f5e440ca000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60210/0 (socket says 172.21.15.94:60210) 2024-03-06T07:19:03.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e3a7fc640 1 -- 172.21.15.94:0/3463534230 learned_addr learned my addr 172.21.15.94:0/3463534230 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:03.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e42ffd640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 0x7f5e440c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:03.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382160443 0 0) 0x7f5e440c8f60 con 0x7f5e441551d0 2024-03-06T07:19:03.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e437fe640 1 --2- 172.21.15.94:0/3463534230 >> v2:172.21.15.145:3300/0 conn(0x7f5e441566c0 0x7f5e4406c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:03.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e30003520 con 0x7f5e441551d0 2024-03-06T07:19:03.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2528355632 0 0) 0x7f5e30003520 con 0x7f5e441551d0 2024-03-06T07:19:03.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 msgr2=0x7f5e440c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:03.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 0x7f5e440c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 >> v2:172.21.15.145:3300/0 conn(0x7f5e441566c0 msgr2=0x7f5e4406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:03.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 --2- 172.21.15.94:0/3463534230 >> v2:172.21.15.145:3300/0 conn(0x7f5e441566c0 0x7f5e4406c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440c9800 con 0x7f5e441551d0 2024-03-06T07:19:03.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f5e2c001670 con 0x7f5e441551d0 2024-03-06T07:19:03.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 >> v1:172.21.15.94:6789/0 conn(0x7f5e441551d0 legacy=0x7f5e440ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:03.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e300039b0 0x7f5e30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e30004800 0x7f5e30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 0x7f5e300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e440c9240 con 0x7f5e300039b0 2024-03-06T07:19:03.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e440c9520 con 0x7f5e30004800 2024-03-06T07:19:03.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3000b9c0 con 0x7f5e4414f180 2024-03-06T07:19:03.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e42ffd640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e300039b0 0x7f5e30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:03.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e3a7fc640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e30004800 0x7f5e30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:03.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.513+0000 7f5e437fe640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 0x7f5e300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:03.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e42ffd640 1 -- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 msgr2=0x7f5e300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:03.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e42ffd640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e4414f180 0x7f5e300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e42ffd640 1 -- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e30004800 msgr2=0x7f5e30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:03.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e42ffd640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e30004800 0x7f5e30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e42ffd640 1 -- 172.21.15.94:0/3463534230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30003520 con 0x7f5e300039b0 2024-03-06T07:19:03.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e3a7fc640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e30004800 0x7f5e30004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:03.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e42ffd640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e300039b0 0x7f5e30003dd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5e34009ac0 tx=0x7f5e30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d2d40bc575d6bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:03.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34039070 con 0x7f5e300039b0 2024-03-06T07:19:03.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3402c6b0 con 0x7f5e300039b0 2024-03-06T07:19:03.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.514+0000 7f5e427fc640 1 -- 172.21.15.94:0/3463534230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e34009690 con 0x7f5e300039b0 2024-03-06T07:19:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.515+0000 7f5e49956640 1 -- 172.21.15.94:0/3463534230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e300039b0 msgr2=0x7f5e30003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.515+0000 7f5e49956640 1 --2- 172.21.15.94:0/3463534230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e300039b0 0x7f5e30003dd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5e34009ac0 tx=0x7f5e30005830 comp rx=0 tx=0).stop 2024-03-06T07:19:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.515+0000 7f5e437fe640 1 -- 172.21.15.94:0/3463534230 reap_dead start 2024-03-06T07:19:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.515+0000 7f5e49956640 1 -- 172.21.15.94:0/3463534230 shutdown_connections 2024-03-06T07:19:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.515+0000 7f5e49956640 1 -- 172.21.15.94:0/3463534230 >> 172.21.15.94:0/3463534230 conn(0x7f5e44072d20 msgr2=0x7f5e440ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:03.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.515+0000 7f5e49956640 1 -- 172.21.15.94:0/3463534230 shutdown_connections 2024-03-06T07:19:03.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.515+0000 7f5e49956640 1 -- 172.21.15.94:0/3463534230 wait complete. 2024-03-06T07:19:03.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.516+0000 7f5e49956640 1 Processor -- start 2024-03-06T07:19:03.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.516+0000 7f5e49956640 1 -- start start 2024-03-06T07:19:03.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e49956640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e300039b0 0x7f5e4414e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e49956640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e44151220 0x7f5e440af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e49956640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e441516a0 0x7f5e4414f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e49956640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4414f560 con 0x7f5e441516a0 2024-03-06T07:19:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e49956640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44156b50 con 0x7f5e44151220 2024-03-06T07:19:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e49956640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4414f6e0 con 0x7f5e300039b0 2024-03-06T07:19:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e441516a0 0x7f5e4414f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e441516a0 0x7f5e4414f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39290/0 (socket says 172.21.15.94:39290) 2024-03-06T07:19:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 -- 172.21.15.94:0/508144235 learned_addr learned my addr 172.21.15.94:0/508144235 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 -- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e300039b0 msgr2=0x7f5e4414e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e300039b0 0x7f5e4414e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 -- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e44151220 msgr2=0x7f5e440af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e44151220 0x7f5e440af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 -- 172.21.15.94:0/508144235 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440c1f60 con 0x7f5e441516a0 2024-03-06T07:19:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.517+0000 7f5e437fe640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e441516a0 0x7f5e4414f860 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5e44159860 tx=0x7f5e3c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.518+0000 7f5e40ff9640 1 -- 172.21.15.94:0/508144235 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c011e10 con 0x7f5e441516a0 2024-03-06T07:19:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.518+0000 7f5e40ff9640 1 -- 172.21.15.94:0/508144235 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c01b070 con 0x7f5e441516a0 2024-03-06T07:19:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.518+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4414ed30 con 0x7f5e441516a0 2024-03-06T07:19:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.518+0000 7f5e40ff9640 1 -- 172.21.15.94:0/508144235 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c003990 con 0x7f5e441516a0 2024-03-06T07:19:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.518+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e441eaa00 con 0x7f5e441516a0 2024-03-06T07:19:03.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.519+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4404f330 con 0x7f5e441516a0 2024-03-06T07:19:03.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.519+0000 7f5e40ff9640 1 -- 172.21.15.94:0/508144235 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e3c005d10 con 0x7f5e441516a0 2024-03-06T07:19:03.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.519+0000 7f5e40ff9640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e2406b4d0 0x7f5e2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:03.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.520+0000 7f5e40ff9640 1 -- 172.21.15.94:0/508144235 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f5e3c08a990 con 0x7f5e441516a0 2024-03-06T07:19:03.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.520+0000 7f5e42ffd640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e2406b4d0 0x7f5e2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:03.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.520+0000 7f5e42ffd640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e2406b4d0 0x7f5e2406d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5e34036000 tx=0x7f5e3402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:03.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.524+0000 7f5e40ff9640 1 -- 172.21.15.94:0/508144235 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e3c025020 con 0x7f5e441516a0 2024-03-06T07:19:03.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:03.694+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e440afe70 con 0x7f5e441516a0 2024-03-06T07:19:04.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.398+0000 7f5e40ff9640 1 -- 172.21.15.94:0/508144235 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5e4414ed30 con 0x7f5e441516a0 2024-03-06T07:19:04.400 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:19:04.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.401+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e2406b4d0 msgr2=0x7f5e2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:04.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.401+0000 7f5e49956640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e2406b4d0 0x7f5e2406d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5e34036000 tx=0x7f5e3402e000 comp rx=0 tx=0).stop 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.401+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e441516a0 msgr2=0x7f5e4414f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.401+0000 7f5e49956640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e441516a0 0x7f5e4414f860 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5e44159860 tx=0x7f5e3c0115a0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 shutdown_connections 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e2406b4d0 0x7f5e2406d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e441516a0 0x7f5e4414f860 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e44151220 0x7f5e440af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 --2- 172.21.15.94:0/508144235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e300039b0 0x7f5e4414e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 >> 172.21.15.94:0/508144235 conn(0x7f5e44072d20 msgr2=0x7f5e440c53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:04.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 shutdown_connections 2024-03-06T07:19:04.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.402+0000 7f5e49956640 1 -- 172.21.15.94:0/508144235 wait complete. 2024-03-06T07:19:04.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-06T07:19:04.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.522+0000 7f1ecdb20640 1 Processor -- start 2024-03-06T07:19:04.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.522+0000 7f1ecdb20640 1 -- start start 2024-03-06T07:19:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ecdb20640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1ec8150910 0x7f1ec80c5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ecdb20640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec8152fd0 con 0x7f1ec80c49a0 2024-03-06T07:19:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ecdb20640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec8153270 con 0x7f1ec8150530 2024-03-06T07:19:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ecdb20640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec8153ba0 con 0x7f1ec8150910 2024-03-06T07:19:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ec6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1ec8150910 0x7f1ec80c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ec7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1ec80c49a0 0x7f1ec80c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:60220/0 (socket says 172.21.15.94:60220) 2024-03-06T07:19:04.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ec7fff640 1 -- 172.21.15.94:0/67868656 learned_addr learned my addr 172.21.15.94:0/67868656 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ec6ffd640 1 -- 172.21.15.94:0/67868656 >> v1:172.21.15.145:6789/0 conn(0x7f1ec8150530 legacy=0x7f1ec806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ec6ffd640 1 -- 172.21.15.94:0/67868656 >> v1:172.21.15.94:6789/0 conn(0x7f1ec80c49a0 legacy=0x7f1ec80c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ec6ffd640 1 -- 172.21.15.94:0/67868656 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec8153550 con 0x7f1ec8150910 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.523+0000 7f1ec67fc640 1 -- 172.21.15.94:0/67868656 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608716571 0 0) 0x7f1ec8152fd0 con 0x7f1ec80c49a0 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.524+0000 7f1ec6ffd640 1 --2- 172.21.15.94:0/67868656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150910 0x7f1ec80c5620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0097a0 tx=0x7f1ebc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4446116b975edd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.524+0000 7f1ec67fc640 1 -- 172.21.15.94:0/67868656 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc035020 con 0x7f1ec8150910 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.524+0000 7f1ec67fc640 1 -- 172.21.15.94:0/67868656 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ebc031830 con 0x7f1ec8150910 2024-03-06T07:19:04.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.524+0000 7f1ec67fc640 1 -- 172.21.15.94:0/67868656 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc031c00 con 0x7f1ec8150910 2024-03-06T07:19:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.525+0000 7f1ecdb20640 1 -- 172.21.15.94:0/67868656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150910 msgr2=0x7f1ec80c5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:04.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.525+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/67868656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150910 0x7f1ec80c5620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0097a0 tx=0x7f1ebc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.525+0000 7f1ecdb20640 1 -- 172.21.15.94:0/67868656 shutdown_connections 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.525+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/67868656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150910 0x7f1ec80c5620 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.525+0000 7f1ecdb20640 1 -- 172.21.15.94:0/67868656 >> 172.21.15.94:0/67868656 conn(0x7f1ec80a9b00 msgr2=0x7f1ec80c4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.525+0000 7f1ecdb20640 1 -- 172.21.15.94:0/67868656 shutdown_connections 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.525+0000 7f1ecdb20640 1 -- 172.21.15.94:0/67868656 wait complete. 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.526+0000 7f1ecdb20640 1 Processor -- start 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.526+0000 7f1ecdb20640 1 -- start start 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ecdb20640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ec80c49a0 0x7f1ec8145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ecdb20640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150530 0x7f1ec806f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ecdb20640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ec8150910 0x7f1ec80a8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:04.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ecdb20640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec806b6d0 con 0x7f1ec8150530 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ecdb20640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80c6270 con 0x7f1ec8150910 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ecdb20640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec806b850 con 0x7f1ec80c49a0 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150530 0x7f1ec806f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150530 0x7f1ec806f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39308/0 (socket says 172.21.15.94:39308) 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 -- 172.21.15.94:0/1848886125 learned_addr learned my addr 172.21.15.94:0/1848886125 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 -- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ec80c49a0 msgr2=0x7f1ec8145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec77fe640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ec80c49a0 0x7f1ec8145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ec80c49a0 0x7f1ec8145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 -- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ec8150910 msgr2=0x7f1ec80a8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec7fff640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ec8150910 0x7f1ec80a8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:04.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ec8150910 0x7f1ec80a8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 -- 172.21.15.94:0/1848886125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec806f280 con 0x7f1ec8150530 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1ec6ffd640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150530 0x7f1ec806f560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0098c0 tx=0x7f1ebc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1eaffff640 1 -- 172.21.15.94:0/1848886125 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc040070 con 0x7f1ec8150530 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1eaffff640 1 -- 172.21.15.94:0/1848886125 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ebc039a00 con 0x7f1ec8150530 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.527+0000 7f1eaffff640 1 -- 172.21.15.94:0/1848886125 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc035040 con 0x7f1ec8150530 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.528+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec80a9180 con 0x7f1ec8150530 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.528+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec806de90 con 0x7f1ec8150530 2024-03-06T07:19:04.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.529+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec804f330 con 0x7f1ec8150530 2024-03-06T07:19:04.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.529+0000 7f1eaffff640 1 -- 172.21.15.94:0/1848886125 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ebc038020 con 0x7f1ec8150530 2024-03-06T07:19:04.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.529+0000 7f1eaffff640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:04.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.529+0000 7f1eaffff640 1 -- 172.21.15.94:0/1848886125 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1ebc0abc70 con 0x7f1ec8150530 2024-03-06T07:19:04.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.530+0000 7f1ec77fe640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:04.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.530+0000 7f1ec77fe640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1eb406b4d0 0x7f1eb406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80c75f0 tx=0x7f1eb8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:04.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.534+0000 7f1eaffff640 1 -- 172.21.15.94:0/1848886125 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ebc07aaf0 con 0x7f1ec8150530 2024-03-06T07:19:04.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:04.704+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1ec80729e0 con 0x7f1ec8150530 2024-03-06T07:19:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.399+0000 7f1eaffff640 1 -- 172.21.15.94:0/1848886125 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7f1ebc07a3f0 con 0x7f1ec8150530 2024-03-06T07:19:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data_ec' removed 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.402+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1eb406b4d0 msgr2=0x7f1eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.402+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1eb406b4d0 0x7f1eb406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80c75f0 tx=0x7f1eb8009040 comp rx=0 tx=0).stop 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.402+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150530 msgr2=0x7f1ec806f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.402+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150530 0x7f1ec806f560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc0098c0 tx=0x7f1ebc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 shutdown_connections 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ec8150910 0x7f1ec80a8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ec8150530 0x7f1ec806f560 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 --2- 172.21.15.94:0/1848886125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ec80c49a0 0x7f1ec8145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 >> 172.21.15.94:0/1848886125 conn(0x7f1ec80a9b00 msgr2=0x7f1ec806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 shutdown_connections 2024-03-06T07:19:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:05.403+0000 7f1ecdb20640 1 -- 172.21.15.94:0/1848886125 wait complete. 2024-03-06T07:19:05.419 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:19:05.419 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:19:05.419 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:19:05.419 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:05.419 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:19:05.420 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:19:05.420 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:19:05.420 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:05.420 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:19:05.420 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:19:05.420 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:19:05.420 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:05.421 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:19:05.421 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:19:05.421 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:19:05.421 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:05.429 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:19:05.429 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:19:05.430 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:19:05.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:19:05.432 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:19:05.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:19:05.472 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:19:05.473 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:05.474 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:19:05.474 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:05.475 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:19:05.475 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:05.550 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:19:05.551 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:19:05.554 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:19:08.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:19:11.431 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:19:11.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:19:11.435 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:19:11.435 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:11.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:19:11.518 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:19:11.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.547+0000 7f994a8fb640 1 Processor -- start 2024-03-06T07:19:11.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.547+0000 7f994a8fb640 1 -- start start 2024-03-06T07:19:11.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.547+0000 7f994a8fb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f99440c69f0 0x7f99440c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:11.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.547+0000 7f994a8fb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99440c8950 0x7f99440acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:11.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.547+0000 7f994a8fb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9944155130 con 0x7f99440ad2b0 2024-03-06T07:19:11.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.547+0000 7f994a8fb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9944155440 con 0x7f99440c8950 2024-03-06T07:19:11.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.547+0000 7f994a8fb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99440c5d50 con 0x7f99440c69f0 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f99437fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99440c8950 0x7f99440acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f99437fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99440c8950 0x7f99440acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59360/0 (socket says 172.21.15.94:59360) 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f99437fe640 1 -- 172.21.15.94:0/3399474254 learned_addr learned my addr 172.21.15.94:0/3399474254 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f9943fff640 1 --2- 172.21.15.94:0/3399474254 >> v2:172.21.15.145:3300/0 conn(0x7f99440c69f0 0x7f99440c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f99437fe640 1 -- 172.21.15.94:0/3399474254 >> v2:172.21.15.145:3300/0 conn(0x7f99440c69f0 msgr2=0x7f99440c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f99437fe640 1 --2- 172.21.15.94:0/3399474254 >> v2:172.21.15.145:3300/0 conn(0x7f99440c69f0 0x7f99440c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f99437fe640 1 -- 172.21.15.94:0/3399474254 >> v1:172.21.15.145:6789/0 conn(0x7f99440ad2b0 legacy=0x7f99441558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.548+0000 7f99437fe640 1 -- 172.21.15.94:0/3399474254 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440c6030 con 0x7f99440c8950 2024-03-06T07:19:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f99437fe640 1 --2- 172.21.15.94:0/3399474254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440c8950 0x7f99440acd70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f99440aa430 tx=0x7f993402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5a79e3348a17c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f9942ffd640 1 -- 172.21.15.94:0/3399474254 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934035020 con 0x7f99440c8950 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f9942ffd640 1 -- 172.21.15.94:0/3399474254 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9934031870 con 0x7f99440c8950 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f9942ffd640 1 -- 172.21.15.94:0/3399474254 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934031c40 con 0x7f99440c8950 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f994a8fb640 1 -- 172.21.15.94:0/3399474254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440c8950 msgr2=0x7f99440acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f994a8fb640 1 --2- 172.21.15.94:0/3399474254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440c8950 0x7f99440acd70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f99440aa430 tx=0x7f993402b910 comp rx=0 tx=0).stop 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f994a8fb640 1 -- 172.21.15.94:0/3399474254 shutdown_connections 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f994a8fb640 1 --2- 172.21.15.94:0/3399474254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440c8950 0x7f99440acd70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f994a8fb640 1 --2- 172.21.15.94:0/3399474254 >> v2:172.21.15.145:3300/0 conn(0x7f99440c69f0 0x7f99440c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.549+0000 7f994a8fb640 1 -- 172.21.15.94:0/3399474254 >> 172.21.15.94:0/3399474254 conn(0x7f99440cae70 msgr2=0x7f99440cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.550+0000 7f994a8fb640 1 -- 172.21.15.94:0/3399474254 shutdown_connections 2024-03-06T07:19:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.550+0000 7f994a8fb640 1 -- 172.21.15.94:0/3399474254 wait complete. 2024-03-06T07:19:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.551+0000 7f994a8fb640 1 Processor -- start 2024-03-06T07:19:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.551+0000 7f994a8fb640 1 -- start start 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.551+0000 7f994a8fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440ad2b0 0x7f99440afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f994a8fb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99440c69f0 0x7f994406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f994a8fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99440c8950 0x7f99441eb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f994a8fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9944152a20 con 0x7f99440ad2b0 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f994a8fb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9944155fe0 con 0x7f99440c69f0 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f994a8fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9944152ba0 con 0x7f99440c8950 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440ad2b0 0x7f99440afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440ad2b0 0x7f99440afb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59372/0 (socket says 172.21.15.94:59372) 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 -- 172.21.15.94:0/1547421442 learned_addr learned my addr 172.21.15.94:0/1547421442 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 -- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99440c8950 msgr2=0x7f99441eb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99440c8950 0x7f99441eb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 -- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99440c69f0 msgr2=0x7f994406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f99437fe640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99440c69f0 0x7f994406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99440c69f0 0x7f994406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 -- 172.21.15.94:0/1547421442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440b00a0 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f99437fe640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99440c69f0 0x7f994406bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9943fff640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440ad2b0 0x7f99440afb60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f993800e9b0 tx=0x7f993800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9940ff9640 1 -- 172.21.15.94:0/1547421442 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938017070 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9940ff9640 1 -- 172.21.15.94:0/1547421442 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993800c9a0 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99440aebb0 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f9940ff9640 1 -- 172.21.15.94:0/1547421442 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993800cd90 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.552+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99441eae30 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.553+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994404f330 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.554+0000 7f9940ff9640 1 -- 172.21.15.94:0/1547421442 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993801a070 con 0x7f99440ad2b0 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.554+0000 7f9940ff9640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f993006b4d0 0x7f993006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:11.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.554+0000 7f9940ff9640 1 -- 172.21.15.94:0/1547421442 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f993808a040 con 0x7f99440ad2b0 2024-03-06T07:19:11.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.554+0000 7f99437fe640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f993006b4d0 0x7f993006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:11.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.555+0000 7f99437fe640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f993006b4d0 0x7f993006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9934036000 tx=0x7f9934008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.559+0000 7f9940ff9640 1 -- 172.21.15.94:0/1547421442 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99440b00a0 con 0x7f99440ad2b0 2024-03-06T07:19:11.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:11.742+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f99440ca650 con 0x7f99440ad2b0 2024-03-06T07:19:13.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.422+0000 7f9940ff9640 1 -- 172.21.15.94:0/1547421442 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v35) v1 ==== 97+0+0 (secure 0 0 0) 0x7f99380590c0 con 0x7f99440ad2b0 2024-03-06T07:19:13.424 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.424+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f993006b4d0 msgr2=0x7f993006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f993006b4d0 0x7f993006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9934036000 tx=0x7f9934008030 comp rx=0 tx=0).stop 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440ad2b0 msgr2=0x7f99440afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440ad2b0 0x7f99440afb60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f993800e9b0 tx=0x7f993800ee70 comp rx=0 tx=0).stop 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 shutdown_connections 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f993006b4d0 0x7f993006d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99440c8950 0x7f99441eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99440c69f0 0x7f994406bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 --2- 172.21.15.94:0/1547421442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99440ad2b0 0x7f99440afb60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 >> 172.21.15.94:0/1547421442 conn(0x7f99440cae70 msgr2=0x7f99440c8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:13.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 shutdown_connections 2024-03-06T07:19:13.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.425+0000 7f994a8fb640 1 -- 172.21.15.94:0/1547421442 wait complete. 2024-03-06T07:19:13.439 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:19:13.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.544+0000 7f1361076640 1 Processor -- start 2024-03-06T07:19:13.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.544+0000 7f1361076640 1 -- start start 2024-03-06T07:19:13.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.544+0000 7f1361076640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f135c14f650 0x7f135c071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.544+0000 7f1361076640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 0x7f135c0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.544+0000 7f1361076640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135c072440 con 0x7f135c152c20 2024-03-06T07:19:13.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.544+0000 7f1361076640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c0c2b10 con 0x7f135c14f650 2024-03-06T07:19:13.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.544+0000 7f1361076640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c152140 con 0x7f135c0c3150 2024-03-06T07:19:13.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135b59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f135c152c20 0x7f135c1554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57208/0 (socket says 172.21.15.94:57208) 2024-03-06T07:19:13.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135b59e640 1 -- 172.21.15.94:0/3649969434 learned_addr learned my addr 172.21.15.94:0/3649969434 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:13.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f1353fff640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 0x7f135c0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937871209 0 0) 0x7f1354001490 con 0x7f135c152c20 2024-03-06T07:19:13.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135ad9d640 1 --2- 172.21.15.94:0/3649969434 >> v2:172.21.15.145:3300/0 conn(0x7f135c14f650 0x7f135c071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1348003520 con 0x7f135c152c20 2024-03-06T07:19:13.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3613077202 0 0) 0x7f1348003520 con 0x7f135c152c20 2024-03-06T07:19:13.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 msgr2=0x7f135c0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:13.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 0x7f135c0c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 >> v2:172.21.15.145:3300/0 conn(0x7f135c14f650 msgr2=0x7f135c071880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:13.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 --2- 172.21.15.94:0/3649969434 >> v2:172.21.15.145:3300/0 conn(0x7f135c14f650 0x7f135c071880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c152420 con 0x7f135c152c20 2024-03-06T07:19:13.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f1354001bc0 con 0x7f135c152c20 2024-03-06T07:19:13.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 >> v1:172.21.15.94:6789/0 conn(0x7f135c152c20 legacy=0x7f135c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f1348003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 0x7f1348004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1348004d40 0x7f13480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0c2b10 con 0x7f1348004d40 2024-03-06T07:19:13.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c152140 con 0x7f13480039b0 2024-03-06T07:19:13.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134800b9c0 con 0x7f135c0c3150 2024-03-06T07:19:13.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135b59e640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1348004d40 0x7f13480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f135ad9d640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f1348003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.545+0000 7f1353fff640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 0x7f1348004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.546+0000 7f135ad9d640 1 -- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 msgr2=0x7f1348004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:13.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.546+0000 7f135ad9d640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c3150 0x7f1348004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.546+0000 7f135ad9d640 1 -- 172.21.15.94:0/3649969434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1348004d40 msgr2=0x7f13480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:13.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.546+0000 7f135ad9d640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1348004d40 0x7f13480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.546+0000 7f135ad9d640 1 -- 172.21.15.94:0/3649969434 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348003520 con 0x7f13480039b0 2024-03-06T07:19:13.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.546+0000 7f135b59e640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1348004d40 0x7f13480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:13.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.546+0000 7f135ad9d640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f1348003dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f134c009ac0 tx=0x7f1348005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f044e5c8b584d4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:13.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c039070 con 0x7f13480039b0 2024-03-06T07:19:13.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c02c710 con 0x7f13480039b0 2024-03-06T07:19:13.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f135a59c640 1 -- 172.21.15.94:0/3649969434 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c009690 con 0x7f13480039b0 2024-03-06T07:19:13.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f1361076640 1 -- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 msgr2=0x7f1348003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f1361076640 1 --2- 172.21.15.94:0/3649969434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f1348003dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f134c009ac0 tx=0x7f1348005830 comp rx=0 tx=0).stop 2024-03-06T07:19:13.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f135b59e640 1 -- 172.21.15.94:0/3649969434 reap_dead start 2024-03-06T07:19:13.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f1361076640 1 -- 172.21.15.94:0/3649969434 shutdown_connections 2024-03-06T07:19:13.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f1361076640 1 -- 172.21.15.94:0/3649969434 >> 172.21.15.94:0/3649969434 conn(0x7f135c0c4290 msgr2=0x7f135c0c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:13.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f1361076640 1 -- 172.21.15.94:0/3649969434 shutdown_connections 2024-03-06T07:19:13.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.547+0000 7f1361076640 1 -- 172.21.15.94:0/3649969434 wait complete. 2024-03-06T07:19:13.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 Processor -- start 2024-03-06T07:19:13.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 -- start start 2024-03-06T07:19:13.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f135c06b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c151aa0 0x7f135c1427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c1451a0 0x7f135c142d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0c40c0 con 0x7f135c151aa0 2024-03-06T07:19:13.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0c3f40 con 0x7f13480039b0 2024-03-06T07:19:13.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1361076640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c071dc0 con 0x7f135c1451a0 2024-03-06T07:19:13.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1353fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c151aa0 0x7f135c1427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1353fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c151aa0 0x7f135c1427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59388/0 (socket says 172.21.15.94:59388) 2024-03-06T07:19:13.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.549+0000 7f1353fff640 1 -- 172.21.15.94:0/2485007474 learned_addr learned my addr 172.21.15.94:0/2485007474 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:13.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f135b59e640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c1451a0 0x7f135c142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f135ad9d640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f135c06b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1353fff640 1 -- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c1451a0 msgr2=0x7f135c142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1353fff640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c1451a0 0x7f135c142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1353fff640 1 -- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 msgr2=0x7f135c06b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1353fff640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f135c06b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1353fff640 1 -- 172.21.15.94:0/2485007474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c14d510 con 0x7f135c151aa0 2024-03-06T07:19:13.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1353fff640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c151aa0 0x7f135c1427f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1344011a00 tx=0x7f1344011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:13.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f13537fe640 1 -- 172.21.15.94:0/2485007474 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134400faf0 con 0x7f135c151aa0 2024-03-06T07:19:13.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f13537fe640 1 -- 172.21.15.94:0/2485007474 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134400fc90 con 0x7f135c151aa0 2024-03-06T07:19:13.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c1445d0 con 0x7f135c151aa0 2024-03-06T07:19:13.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f13537fe640 1 -- 172.21.15.94:0/2485007474 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344014440 con 0x7f135c151aa0 2024-03-06T07:19:13.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.550+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c144870 con 0x7f135c151aa0 2024-03-06T07:19:13.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.551+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c04f330 con 0x7f135c151aa0 2024-03-06T07:19:13.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.552+0000 7f13537fe640 1 -- 172.21.15.94:0/2485007474 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1344012040 con 0x7f135c151aa0 2024-03-06T07:19:13.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.552+0000 7f13537fe640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.552+0000 7f13537fe640 1 -- 172.21.15.94:0/2485007474 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f134408a4c0 con 0x7f135c151aa0 2024-03-06T07:19:13.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.552+0000 7f135ad9d640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.553+0000 7f135ad9d640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133c06b4d0 0x7f133c06d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f134c036000 tx=0x7f134c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:13.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.556+0000 7f13537fe640 1 -- 172.21.15.94:0/2485007474 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1344019030 con 0x7f135c151aa0 2024-03-06T07:19:13.770 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:13.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:13.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.802+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f135c068460 con 0x7f135c151aa0 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.803+0000 7f13537fe640 1 -- 172.21.15.94:0/2485007474 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v5) v1 ==== 80+0+6065 (secure 0 0 0) 0x7f1344058f40 con 0x7f135c151aa0 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:19:13.805 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:13.806 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.807 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.808 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.809 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.810 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.811 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.812 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.813 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:19:13.814 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.815 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCgGOhl6XQKGBAADcDndCTQoldSyQ/kyATGSw==", 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCgGOhlPn0KGBAA70D+RqTPHjIPhC0BoYPiVg==", 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.816 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:19:13.817 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCgGOhl+4MKGBAADgusUHPtf9Wsko2XkZ9dxQ==", 2024-03-06T07:19:13.817 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.817 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:19:13.817 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.817 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCgGOhlo5wKGBAARGXsGZ+iEh11ZjPoKXL/Rw==", 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCgGOhlc6QKGBAAkVZlAqW9KuCeJKx4/iJD3A==", 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.818 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCgGOhlaasKGBAADufU/mrSdI7xb/ARflSOiQ==", 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.819 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.820 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:13.821 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:19:13.821 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:19:13.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133c06b4d0 msgr2=0x7f133c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133c06b4d0 0x7f133c06d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f134c036000 tx=0x7f134c030000 comp rx=0 tx=0).stop 2024-03-06T07:19:13.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c151aa0 msgr2=0x7f135c1427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c151aa0 0x7f135c1427f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1344011a00 tx=0x7f1344011ec0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 shutdown_connections 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c1451a0 0x7f135c142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c151aa0 0x7f135c1427f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 --2- 172.21.15.94:0/2485007474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13480039b0 0x7f135c06b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 >> 172.21.15.94:0/2485007474 conn(0x7f135c0c4290 msgr2=0x7f135c147d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 shutdown_connections 2024-03-06T07:19:13.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.805+0000 7f1361076640 1 -- 172.21.15.94:0/2485007474 wait complete. 2024-03-06T07:19:13.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:19:13.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.965+0000 7f49d66fb640 1 Processor -- start 2024-03-06T07:19:13.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.965+0000 7f49d66fb640 1 -- start start 2024-03-06T07:19:13.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49d66fb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f49d006ff10 0x7f49d0153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49d66fb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f49d00c7170 0x7f49d0154160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49d66fb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00a8390 con 0x7f49d00c6c40 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49d66fb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d00c7620 con 0x7f49d00c7170 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49d66fb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d00730f0 con 0x7f49d006ff10 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f49d00c7170 0x7f49d0154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f49d00c7170 0x7f49d0154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59398/0 (socket says 172.21.15.94:59398) 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cf7fe640 1 -- 172.21.15.94:0/1535396777 learned_addr learned my addr 172.21.15.94:0/1535396777 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cf7fe640 1 -- 172.21.15.94:0/1535396777 >> v2:172.21.15.145:3300/0 conn(0x7f49d006ff10 msgr2=0x7f49d0153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cffff640 1 --2- 172.21.15.94:0/1535396777 >> v2:172.21.15.145:3300/0 conn(0x7f49d006ff10 0x7f49d0153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cf7fe640 1 --2- 172.21.15.94:0/1535396777 >> v2:172.21.15.145:3300/0 conn(0x7f49d006ff10 0x7f49d0153c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cf7fe640 1 -- 172.21.15.94:0/1535396777 >> v1:172.21.15.145:6789/0 conn(0x7f49d00c6c40 legacy=0x7f49d00c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:13.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.966+0000 7f49cf7fe640 1 -- 172.21.15.94:0/1535396777 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00733d0 con 0x7f49d00c7170 2024-03-06T07:19:13.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.967+0000 7f49cf7fe640 1 --2- 172.21.15.94:0/1535396777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d0154160 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f49d00ab220 tx=0x7f49c002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975237a0d03dde4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.967+0000 7f49ceffd640 1 -- 172.21.15.94:0/1535396777 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0038070 con 0x7f49d00c7170 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49ceffd640 1 -- 172.21.15.94:0/1535396777 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c002f050 con 0x7f49d00c7170 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49ceffd640 1 -- 172.21.15.94:0/1535396777 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0031970 con 0x7f49d00c7170 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 -- 172.21.15.94:0/1535396777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 msgr2=0x7f49d0154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 --2- 172.21.15.94:0/1535396777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d0154160 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f49d00ab220 tx=0x7f49c002b7f0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 -- 172.21.15.94:0/1535396777 shutdown_connections 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 --2- 172.21.15.94:0/1535396777 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d0154160 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 --2- 172.21.15.94:0/1535396777 >> v2:172.21.15.145:3300/0 conn(0x7f49d006ff10 0x7f49d0153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 -- 172.21.15.94:0/1535396777 >> 172.21.15.94:0/1535396777 conn(0x7f49d006b970 msgr2=0x7f49d00aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 -- 172.21.15.94:0/1535396777 shutdown_connections 2024-03-06T07:19:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.968+0000 7f49d66fb640 1 -- 172.21.15.94:0/1535396777 wait complete. 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.969+0000 7f49d66fb640 1 Processor -- start 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d66fb640 1 -- start start 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d66fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49d006ff10 0x7f49d0156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d66fb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49d00c6c40 0x7f49d01552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d66fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d01557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d66fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0070470 con 0x7f49d00c7170 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d66fb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00702f0 con 0x7f49d00c6c40 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d66fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00c6860 con 0x7f49d006ff10 2024-03-06T07:19:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d4c98640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d01557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d4c98640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d01557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59414/0 (socket says 172.21.15.94:59414) 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.970+0000 7f49d4c98640 1 -- 172.21.15.94:0/3969985708 learned_addr learned my addr 172.21.15.94:0/3969985708 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49cf7fe640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49d00c6c40 0x7f49d01552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49cffff640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49d006ff10 0x7f49d0156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d4c98640 1 -- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49d006ff10 msgr2=0x7f49d0156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d4c98640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49d006ff10 0x7f49d0156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d4c98640 1 -- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49d00c6c40 msgr2=0x7f49d01552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d4c98640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49d00c6c40 0x7f49d01552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d4c98640 1 -- 172.21.15.94:0/3969985708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d01e9990 con 0x7f49d00c7170 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49cf7fe640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49d00c6c40 0x7f49d01552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d4c98640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d01557f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c780 tx=0x7f49c800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49ccff9640 1 -- 172.21.15.94:0/3969985708 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8017070 con 0x7f49d00c7170 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49ccff9640 1 -- 172.21.15.94:0/3969985708 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c8003da0 con 0x7f49d00c7170 2024-03-06T07:19:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49ccff9640 1 -- 172.21.15.94:0/3969985708 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c80052b0 con 0x7f49d00c7170 2024-03-06T07:19:13.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d01547d0 con 0x7f49d00c7170 2024-03-06T07:19:13.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.971+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d0154d90 con 0x7f49d00c7170 2024-03-06T07:19:13.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.972+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d00c30a0 con 0x7f49d00c7170 2024-03-06T07:19:13.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.972+0000 7f49ccff9640 1 -- 172.21.15.94:0/3969985708 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49c801a070 con 0x7f49d00c7170 2024-03-06T07:19:13.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.973+0000 7f49ccff9640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49bc06b4d0 0x7f49bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:13.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.973+0000 7f49ccff9640 1 -- 172.21.15.94:0/3969985708 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f49c80895f0 con 0x7f49d00c7170 2024-03-06T07:19:13.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.973+0000 7f49cffff640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49bc06b4d0 0x7f49bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:13.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.974+0000 7f49cffff640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49bc06b4d0 0x7f49bc06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49c4006fd0 tx=0x7f49c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:13.977+0000 7f49ccff9640 1 -- 172.21.15.94:0/3969985708 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49d00c30a0 con 0x7f49d00c7170 2024-03-06T07:19:14.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.225+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f49d004f330 con 0x7f49d00c7170 2024-03-06T07:19:14.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.227+0000 7f49ccff9640 1 -- 172.21.15.94:0/3969985708 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f49c80586c0 con 0x7f49d00c7170 2024-03-06T07:19:14.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49bc06b4d0 msgr2=0x7f49bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49bc06b4d0 0x7f49bc06d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f49c4006fd0 tx=0x7f49c4008040 comp rx=0 tx=0).stop 2024-03-06T07:19:14.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 msgr2=0x7f49d01557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d01557f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c780 tx=0x7f49c800cc40 comp rx=0 tx=0).stop 2024-03-06T07:19:14.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 shutdown_connections 2024-03-06T07:19:14.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49bc06b4d0 0x7f49bc06d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49d00c7170 0x7f49d01557f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49d00c6c40 0x7f49d01552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 --2- 172.21.15.94:0/3969985708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49d006ff10 0x7f49d0156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 >> 172.21.15.94:0/3969985708 conn(0x7f49d006b970 msgr2=0x7f49d0151020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:14.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 shutdown_connections 2024-03-06T07:19:14.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.229+0000 7f49d66fb640 1 -- 172.21.15.94:0/3969985708 wait complete. 2024-03-06T07:19:14.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:19:14.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.342+0000 7fa839e1c640 1 Processor -- start 2024-03-06T07:19:14.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.342+0000 7fa839e1c640 1 -- start start 2024-03-06T07:19:14.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa839e1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa8340c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa839e1c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8340c8950 0x7fa8340acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa839e1c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa8340ad2b0 0x7fa8341558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa839e1c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa834155440 con 0x7fa8340c8950 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa839e1c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa834155130 con 0x7fa8340ad2b0 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa839e1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340c5d50 con 0x7fa8340c69f0 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8340c8950 0x7fa8340acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8340c8950 0x7fa8340acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59416/0 (socket says 172.21.15.94:59416) 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 -- 172.21.15.94:0/2825783631 learned_addr learned my addr 172.21.15.94:0/2825783631 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa8337fe640 1 --2- 172.21.15.94:0/2825783631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa8340c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa833fff640 1 --2- 172.21.15.94:0/2825783631 >> v2:172.21.15.145:3300/0 conn(0x7fa8340ad2b0 0x7fa8341558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 -- 172.21.15.94:0/2825783631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 msgr2=0x7fa8340c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 --2- 172.21.15.94:0/2825783631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa8340c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 -- 172.21.15.94:0/2825783631 >> v2:172.21.15.145:3300/0 conn(0x7fa8340ad2b0 msgr2=0x7fa8341558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 --2- 172.21.15.94:0/2825783631 >> v2:172.21.15.145:3300/0 conn(0x7fa8340ad2b0 0x7fa8341558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.343+0000 7fa82affd640 1 -- 172.21.15.94:0/2825783631 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8340c6030 con 0x7fa8340c8950 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.344+0000 7fa82affd640 1 --2- 172.21.15.94:0/2825783631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c8950 0x7fa8340acd70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa834159820 tx=0x7fa81c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a79eb2f6a99a3d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.344+0000 7fa832ffd640 1 -- 172.21.15.94:0/2825783631 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c02e070 con 0x7fa8340c8950 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.344+0000 7fa832ffd640 1 -- 172.21.15.94:0/2825783631 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c037070 con 0x7fa8340c8950 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.344+0000 7fa832ffd640 1 -- 172.21.15.94:0/2825783631 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c031a00 con 0x7fa8340c8950 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 -- 172.21.15.94:0/2825783631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c8950 msgr2=0x7fa8340acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 --2- 172.21.15.94:0/2825783631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c8950 0x7fa8340acd70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa834159820 tx=0x7fa81c02b880 comp rx=0 tx=0).stop 2024-03-06T07:19:14.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 -- 172.21.15.94:0/2825783631 shutdown_connections 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 --2- 172.21.15.94:0/2825783631 >> v2:172.21.15.145:3300/0 conn(0x7fa8340ad2b0 0x7fa8341558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 --2- 172.21.15.94:0/2825783631 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c8950 0x7fa8340acd70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 --2- 172.21.15.94:0/2825783631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa8340c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 -- 172.21.15.94:0/2825783631 >> 172.21.15.94:0/2825783631 conn(0x7fa8340cae70 msgr2=0x7fa8340cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 -- 172.21.15.94:0/2825783631 shutdown_connections 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.345+0000 7fa839e1c640 1 -- 172.21.15.94:0/2825783631 wait complete. 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.346+0000 7fa839e1c640 1 Processor -- start 2024-03-06T07:19:14.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.346+0000 7fa839e1c640 1 -- start start 2024-03-06T07:19:14.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340ad2b0 0x7fa8340ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa834147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c8950 0x7fa834152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa834152ba0 con 0x7fa8340ad2b0 2024-03-06T07:19:14.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa834152a20 con 0x7fa8340c8950 2024-03-06T07:19:14.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340c58e0 con 0x7fa8340c69f0 2024-03-06T07:19:14.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340ad2b0 0x7fa8340ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340ad2b0 0x7fa8340ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59424/0 (socket says 172.21.15.94:59424) 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 -- 172.21.15.94:0/1478306090 learned_addr learned my addr 172.21.15.94:0/1478306090 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa82affd640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa834147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa833fff640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c8950 0x7fa834152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 -- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 msgr2=0x7fa834147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa834147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 -- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c8950 msgr2=0x7fa834152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c8950 0x7fa834152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 -- 172.21.15.94:0/1478306090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8340c1b20 con 0x7fa8340ad2b0 2024-03-06T07:19:14.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa833fff640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c8950 0x7fa834152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:14.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa8337fe640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340ad2b0 0x7fa8340ab7d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa82400d800 tx=0x7fa82400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:14.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa830ff9640 1 -- 172.21.15.94:0/1478306090 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400b6c0 con 0x7fa8340ad2b0 2024-03-06T07:19:14.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa830ff9640 1 -- 172.21.15.94:0/1478306090 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa824019070 con 0x7fa8340ad2b0 2024-03-06T07:19:14.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa830ff9640 1 -- 172.21.15.94:0/1478306090 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400ba80 con 0x7fa8340ad2b0 2024-03-06T07:19:14.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8341eaec0 con 0x7fa8340ad2b0 2024-03-06T07:19:14.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.347+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8341eb480 con 0x7fa8340ad2b0 2024-03-06T07:19:14.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.349+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83404f330 con 0x7fa8340ad2b0 2024-03-06T07:19:14.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.349+0000 7fa830ff9640 1 -- 172.21.15.94:0/1478306090 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa82400bc20 con 0x7fa8340ad2b0 2024-03-06T07:19:14.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.350+0000 7fa830ff9640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.350+0000 7fa830ff9640 1 -- 172.21.15.94:0/1478306090 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa82408a3c0 con 0x7fa8340ad2b0 2024-03-06T07:19:14.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.350+0000 7fa82affd640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.350+0000 7fa82affd640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa81c036000 tx=0x7fa81c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:14.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.354+0000 7fa830ff9640 1 -- 172.21.15.94:0/1478306090 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8340c1b20 con 0x7fa8340ad2b0 2024-03-06T07:19:14.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.600+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fa8340c4ac0 con 0x7fa8340ad2b0 2024-03-06T07:19:14.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.601+0000 7fa830ff9640 1 -- 172.21.15.94:0/1478306090 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa824059490 con 0x7fa8340ad2b0 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 msgr2=0x7fa81806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa81c036000 tx=0x7fa81c03d040 comp rx=0 tx=0).stop 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340ad2b0 msgr2=0x7fa8340ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340ad2b0 0x7fa8340ab7d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa82400d800 tx=0x7fa82400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 shutdown_connections 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c8950 0x7fa834152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c69f0 0x7fa834147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 --2- 172.21.15.94:0/1478306090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340ad2b0 0x7fa8340ab7d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 >> 172.21.15.94:0/1478306090 conn(0x7fa8340cae70 msgr2=0x7fa834151c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 shutdown_connections 2024-03-06T07:19:14.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.604+0000 7fa839e1c640 1 -- 172.21.15.94:0/1478306090 wait complete. 2024-03-06T07:19:14.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:19:14.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.717+0000 7fce6ff3e640 1 Processor -- start 2024-03-06T07:19:14.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.717+0000 7fce6ff3e640 1 -- start start 2024-03-06T07:19:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6ff3e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fce68152b40 0x7fce680c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6ff3e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce680733f0 con 0x7fce680a7ff0 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6ff3e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce680c82c0 con 0x7fce680724d0 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6ff3e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce68072d70 con 0x7fce68152b40 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6dcda640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fce68152b40 0x7fce680c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6e4db640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fce680a7ff0 0x7fce6815fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57212/0 (socket says 172.21.15.94:57212) 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6e4db640 1 -- 172.21.15.94:0/4242876086 learned_addr learned my addr 172.21.15.94:0/4242876086 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6dcda640 1 -- 172.21.15.94:0/4242876086 >> v1:172.21.15.145:6789/0 conn(0x7fce680724d0 legacy=0x7fce681551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6dcda640 1 -- 172.21.15.94:0/4242876086 >> v1:172.21.15.94:6789/0 conn(0x7fce680a7ff0 legacy=0x7fce6815fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.718+0000 7fce6dcda640 1 -- 172.21.15.94:0/4242876086 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6806fd30 con 0x7fce68152b40 2024-03-06T07:19:14.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.719+0000 7fce6dcda640 1 --2- 172.21.15.94:0/4242876086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce680c8810 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fce580097a0 tx=0x7fce5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10513cbfc99221e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.719+0000 7fce6ccd8640 1 -- 172.21.15.94:0/4242876086 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58035020 con 0x7fce68152b40 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.719+0000 7fce6ccd8640 1 -- 172.21.15.94:0/4242876086 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce58031830 con 0x7fce68152b40 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.719+0000 7fce6ccd8640 1 -- 172.21.15.94:0/4242876086 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58031c00 con 0x7fce68152b40 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.720+0000 7fce6ff3e640 1 -- 172.21.15.94:0/4242876086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 msgr2=0x7fce680c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.720+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/4242876086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce680c8810 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fce580097a0 tx=0x7fce5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.720+0000 7fce6ff3e640 1 -- 172.21.15.94:0/4242876086 shutdown_connections 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.720+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/4242876086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce680c8810 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.720+0000 7fce6ff3e640 1 -- 172.21.15.94:0/4242876086 >> 172.21.15.94:0/4242876086 conn(0x7fce6806b970 msgr2=0x7fce680aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.720+0000 7fce6ff3e640 1 -- 172.21.15.94:0/4242876086 shutdown_connections 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.720+0000 7fce6ff3e640 1 -- 172.21.15.94:0/4242876086 wait complete. 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.721+0000 7fce6ff3e640 1 Processor -- start 2024-03-06T07:19:14.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.721+0000 7fce6ff3e640 1 -- start start 2024-03-06T07:19:14.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6ff3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce680724d0 0x7fce681ebb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6ff3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce680a7ff0 0x7fce680ab930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6ff3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce6814a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6ff3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce680c8d50 con 0x7fce68152b40 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6ff3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce680704d0 con 0x7fce680724d0 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6ff3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce680c8ed0 con 0x7fce680a7ff0 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce6814a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce6814a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59454/0 (socket says 172.21.15.94:59454) 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 -- 172.21.15.94:0/3568155057 learned_addr learned my addr 172.21.15.94:0/3568155057 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:14.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6dcda640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce680724d0 0x7fce681ebb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6d4d9640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce680a7ff0 0x7fce680ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 -- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce680a7ff0 msgr2=0x7fce680ab930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce680a7ff0 0x7fce680ab930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 -- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce680724d0 msgr2=0x7fce681ebb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce680724d0 0x7fce681ebb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 -- 172.21.15.94:0/3568155057 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce680adc70 con 0x7fce68152b40 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6e4db640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce6814a390 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fce6000ca80 tx=0x7fce6000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce5e7fc640 1 -- 172.21.15.94:0/3568155057 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60017070 con 0x7fce68152b40 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce5e7fc640 1 -- 172.21.15.94:0/3568155057 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6000ac30 con 0x7fce68152b40 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6814b070 con 0x7fce68152b40 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.722+0000 7fce5e7fc640 1 -- 172.21.15.94:0/3568155057 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60012600 con 0x7fce68152b40 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.723+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce680abe70 con 0x7fce68152b40 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.724+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce68075300 con 0x7fce68152b40 2024-03-06T07:19:14.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.724+0000 7fce5e7fc640 1 -- 172.21.15.94:0/3568155057 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce6001a070 con 0x7fce68152b40 2024-03-06T07:19:14.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.724+0000 7fce5e7fc640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce4c06b4d0 0x7fce4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:14.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.724+0000 7fce5e7fc640 1 -- 172.21.15.94:0/3568155057 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fce60089cb0 con 0x7fce68152b40 2024-03-06T07:19:14.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.725+0000 7fce6dcda640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce4c06b4d0 0x7fce4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:14.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.725+0000 7fce6dcda640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce4c06b4d0 0x7fce4c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fce58036000 tx=0x7fce5803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:14.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.729+0000 7fce5e7fc640 1 -- 172.21.15.94:0/3568155057 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce68075300 con 0x7fce68152b40 2024-03-06T07:19:14.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:19:14.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.967+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fce6804f330 con 0x7fce68152b40 2024-03-06T07:19:14.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.969+0000 7fce5e7fc640 1 -- 172.21.15.94:0/3568155057 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fce6000ae20 con 0x7fce68152b40 2024-03-06T07:19:14.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce4c06b4d0 msgr2=0x7fce4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce4c06b4d0 0x7fce4c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fce58036000 tx=0x7fce5803c040 comp rx=0 tx=0).stop 2024-03-06T07:19:14.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 msgr2=0x7fce6814a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:14.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce6814a390 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fce6000ca80 tx=0x7fce6000cf40 comp rx=0 tx=0).stop 2024-03-06T07:19:14.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 shutdown_connections 2024-03-06T07:19:14.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce4c06b4d0 0x7fce4c06d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce68152b40 0x7fce6814a390 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce680a7ff0 0x7fce680ab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 --2- 172.21.15.94:0/3568155057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce680724d0 0x7fce681ebb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:14.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 >> 172.21.15.94:0/3568155057 conn(0x7fce6806b970 msgr2=0x7fce68151630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:14.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 shutdown_connections 2024-03-06T07:19:14.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:14.972+0000 7fce6ff3e640 1 -- 172.21.15.94:0/3568155057 wait complete. 2024-03-06T07:19:14.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:19:15.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.083+0000 7ff54bac9640 1 Processor -- start 2024-03-06T07:19:15.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54bac9640 1 -- start start 2024-03-06T07:19:15.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54bac9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff544150710 0x7ff54406bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54bac9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff54406c510 0x7ff544153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54bac9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5440c3100 con 0x7ff5440c3600 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54bac9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5440c1a40 con 0x7ff54406c510 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54bac9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff544072ad0 con 0x7ff544150710 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54a066640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff54406c510 0x7ff544153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54a066640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff54406c510 0x7ff544153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59462/0 (socket says 172.21.15.94:59462) 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff549064640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff5440c3600 0x7ff5441535e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57222/0 (socket says 172.21.15.94:57222) 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff54a066640 1 -- 172.21.15.94:0/3743382776 learned_addr learned my addr 172.21.15.94:0/3743382776 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.084+0000 7ff549865640 1 --2- 172.21.15.94:0/3743382776 >> v2:172.21.15.145:3300/0 conn(0x7ff544150710 0x7ff54406bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.085+0000 7ff54a066640 1 -- 172.21.15.94:0/3743382776 >> v2:172.21.15.145:3300/0 conn(0x7ff544150710 msgr2=0x7ff54406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.085+0000 7ff54a066640 1 --2- 172.21.15.94:0/3743382776 >> v2:172.21.15.145:3300/0 conn(0x7ff544150710 0x7ff54406bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.085+0000 7ff54a066640 1 -- 172.21.15.94:0/3743382776 >> v1:172.21.15.94:6789/0 conn(0x7ff5440c3600 legacy=0x7ff5441535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.085+0000 7ff54a066640 1 -- 172.21.15.94:0/3743382776 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544072d20 con 0x7ff54406c510 2024-03-06T07:19:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.085+0000 7ff54a066640 1 --2- 172.21.15.94:0/3743382776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff544153cf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff5440671b0 tx=0x7ff53c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10032c244081c221 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff548863640 1 -- 172.21.15.94:0/3743382776 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c02e070 con 0x7ff54406c510 2024-03-06T07:19:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff548863640 1 -- 172.21.15.94:0/3743382776 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c037070 con 0x7ff54406c510 2024-03-06T07:19:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff548863640 1 -- 172.21.15.94:0/3743382776 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c031a00 con 0x7ff54406c510 2024-03-06T07:19:15.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 -- 172.21.15.94:0/3743382776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 msgr2=0x7ff544153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 --2- 172.21.15.94:0/3743382776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff544153cf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff5440671b0 tx=0x7ff53c02b880 comp rx=0 tx=0).stop 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 -- 172.21.15.94:0/3743382776 shutdown_connections 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 --2- 172.21.15.94:0/3743382776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff544153cf0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 --2- 172.21.15.94:0/3743382776 >> v2:172.21.15.145:3300/0 conn(0x7ff544150710 0x7ff54406bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 -- 172.21.15.94:0/3743382776 >> 172.21.15.94:0/3743382776 conn(0x7ff5440c7670 msgr2=0x7ff5440c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 -- 172.21.15.94:0/3743382776 shutdown_connections 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.086+0000 7ff54bac9640 1 -- 172.21.15.94:0/3743382776 wait complete. 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.087+0000 7ff54bac9640 1 Processor -- start 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.087+0000 7ff54bac9640 1 -- start start 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff54bac9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff5441435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff54bac9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5440c3600 0x7ff54414bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff54bac9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff544150710 0x7ff54414c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff54bac9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff544150af0 con 0x7ff54406c510 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff54bac9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440c49b0 con 0x7ff5440c3600 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff54bac9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff544150c70 con 0x7ff544150710 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff5441435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff5441435a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59478/0 (socket says 172.21.15.94:59478) 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 -- 172.21.15.94:0/2774553021 learned_addr learned my addr 172.21.15.94:0/2774553021 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549064640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5440c3600 0x7ff54414bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff54a066640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff544150710 0x7ff54414c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 -- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff544150710 msgr2=0x7ff54414c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff544150710 0x7ff54414c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 -- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5440c3600 msgr2=0x7ff54414bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5440c3600 0x7ff54414bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 -- 172.21.15.94:0/2774553021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54414ff80 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549064640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5440c3600 0x7ff54414bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.088+0000 7ff549865640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff5441435a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff54000efc0 tx=0x7ff54000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.089+0000 7ff53a7fc640 1 -- 172.21.15.94:0/2774553021 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540018020 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.089+0000 7ff53a7fc640 1 -- 172.21.15.94:0/2774553021 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff540014480 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.089+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54414f9c0 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.089+0000 7ff53a7fc640 1 -- 172.21.15.94:0/2774553021 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540014850 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.089+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54414ee10 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.089+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff544068e70 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.090+0000 7ff53a7fc640 1 -- 172.21.15.94:0/2774553021 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff540005d10 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.090+0000 7ff53a7fc640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff52806b4d0 0x7ff52806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.090+0000 7ff53a7fc640 1 -- 172.21.15.94:0/2774553021 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff54008adc0 con 0x7ff54406c510 2024-03-06T07:19:15.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.090+0000 7ff549064640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff52806b4d0 0x7ff52806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.091+0000 7ff549064640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff52806b4d0 0x7ff52806d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff544151be0 tx=0x7ff534007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.095+0000 7ff53a7fc640 1 -- 172.21.15.94:0/2774553021 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff544068e70 con 0x7ff54406c510 2024-03-06T07:19:15.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.337+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7ff54404f330 con 0x7ff54406c510 2024-03-06T07:19:15.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.339+0000 7ff53a7fc640 1 -- 172.21.15.94:0/2774553021 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff540022030 con 0x7ff54406c510 2024-03-06T07:19:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff52806b4d0 msgr2=0x7ff52806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff52806b4d0 0x7ff52806d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff544151be0 tx=0x7ff534007d80 comp rx=0 tx=0).stop 2024-03-06T07:19:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 msgr2=0x7ff5441435a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff5441435a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff54000efc0 tx=0x7ff54000c490 comp rx=0 tx=0).stop 2024-03-06T07:19:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 shutdown_connections 2024-03-06T07:19:15.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff52806b4d0 0x7ff52806d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff544150710 0x7ff54414c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5440c3600 0x7ff54414bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 --2- 172.21.15.94:0/2774553021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff54406c510 0x7ff5441435a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 >> 172.21.15.94:0/2774553021 conn(0x7ff5440c7670 msgr2=0x7ff544153430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:15.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 shutdown_connections 2024-03-06T07:19:15.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.341+0000 7ff54bac9640 1 -- 172.21.15.94:0/2774553021 wait complete. 2024-03-06T07:19:15.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:19:15.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.454+0000 7f0463d56640 1 Processor -- start 2024-03-06T07:19:15.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f0463d56640 1 -- start start 2024-03-06T07:19:15.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f0463d56640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f045c0c8950 0x7f045c0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f0463d56640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c155130 con 0x7f045c1558d0 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f0463d56640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c0ad480 con 0x7f045c0c69f0 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f0463d56640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045c155440 con 0x7f045c0c8950 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f04612f1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f045c0c8950 0x7f045c0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f04612f1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f045c0c8950 0x7f045c0c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59490/0 (socket says 172.21.15.94:59490) 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f04612f1640 1 -- 172.21.15.94:0/1426989229 learned_addr learned my addr 172.21.15.94:0/1426989229 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.455+0000 7f04622f3640 1 --1- 172.21.15.94:0/1426989229 >> v1:172.21.15.94:6789/0 conn(0x7f045c1558d0 0x7f045c15fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57232/0 (socket says 172.21.15.94:57232) 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.456+0000 7f04612f1640 1 -- 172.21.15.94:0/1426989229 >> v1:172.21.15.145:6789/0 conn(0x7f045c0c69f0 legacy=0x7f045c0acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.456+0000 7f04612f1640 1 -- 172.21.15.94:0/1426989229 >> v1:172.21.15.94:6789/0 conn(0x7f045c1558d0 legacy=0x7f045c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.456+0000 7f04612f1640 1 -- 172.21.15.94:0/1426989229 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c0c5d50 con 0x7f045c0c8950 2024-03-06T07:19:15.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.456+0000 7f0460af0640 1 -- 172.21.15.94:0/1426989229 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881635574 0 0) 0x7f0454001490 con 0x7f045c1558d0 2024-03-06T07:19:15.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.456+0000 7f04612f1640 1 --2- 172.21.15.94:0/1426989229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c0c6dd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0458009870 tx=0x7f045802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8a27cf705171bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0460af0640 1 -- 172.21.15.94:0/1426989229 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458034070 con 0x7f045c0c8950 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0460af0640 1 -- 172.21.15.94:0/1426989229 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0458030860 con 0x7f045c0c8950 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0460af0640 1 -- 172.21.15.94:0/1426989229 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458030c30 con 0x7f045c0c8950 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0463d56640 1 -- 172.21.15.94:0/1426989229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 msgr2=0x7f045c0c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0463d56640 1 --2- 172.21.15.94:0/1426989229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c0c6dd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0458009870 tx=0x7f045802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0463d56640 1 -- 172.21.15.94:0/1426989229 shutdown_connections 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0463d56640 1 --2- 172.21.15.94:0/1426989229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c0c6dd0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0463d56640 1 -- 172.21.15.94:0/1426989229 >> 172.21.15.94:0/1426989229 conn(0x7f045c0cae70 msgr2=0x7f045c0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0463d56640 1 -- 172.21.15.94:0/1426989229 shutdown_connections 2024-03-06T07:19:15.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.457+0000 7f0463d56640 1 -- 172.21.15.94:0/1426989229 wait complete. 2024-03-06T07:19:15.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.458+0000 7f0463d56640 1 Processor -- start 2024-03-06T07:19:15.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.458+0000 7f0463d56640 1 -- start start 2024-03-06T07:19:15.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f0463d56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f045c0c69f0 0x7f045c14efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f0463d56640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c1eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f0463d56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f045c1558d0 0x7f045c1eb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f0463d56640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c152a20 con 0x7f045c0c8950 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f0463d56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c0c6310 con 0x7f045c0c69f0 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f0463d56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045c152ba0 con 0x7f045c1558d0 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c1eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c1eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59502/0 (socket says 172.21.15.94:59502) 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 -- 172.21.15.94:0/1269879746 learned_addr learned my addr 172.21.15.94:0/1269879746 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f0461af2640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f045c0c69f0 0x7f045c14efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04622f3640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f045c1558d0 0x7f045c1eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 -- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f045c1558d0 msgr2=0x7f045c1eb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f045c1558d0 0x7f045c1eb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 -- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f045c0c69f0 msgr2=0x7f045c14efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f045c0c69f0 0x7f045c14efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 -- 172.21.15.94:0/1269879746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c14d4d0 con 0x7f045c0c8950 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.459+0000 7f04612f1640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c1eb210 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0458009870 tx=0x7f0458004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.460+0000 7f04527fc640 1 -- 172.21.15.94:0/1269879746 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458048070 con 0x7f045c0c8950 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.460+0000 7f04527fc640 1 -- 172.21.15.94:0/1269879746 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0458039d90 con 0x7f045c0c8950 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.460+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c158140 con 0x7f045c0c8950 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.460+0000 7f04527fc640 1 -- 172.21.15.94:0/1269879746 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458034040 con 0x7f045c0c8950 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.460+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045c0c2cd0 con 0x7f045c0c8950 2024-03-06T07:19:15.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.461+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045c067910 con 0x7f045c0c8950 2024-03-06T07:19:15.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.461+0000 7f04527fc640 1 -- 172.21.15.94:0/1269879746 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f045804b070 con 0x7f045c0c8950 2024-03-06T07:19:15.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.461+0000 7f04527fc640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f044006b4d0 0x7f044006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.462+0000 7f04527fc640 1 -- 172.21.15.94:0/1269879746 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f04580ac140 con 0x7f045c0c8950 2024-03-06T07:19:15.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.462+0000 7f0461af2640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f044006b4d0 0x7f044006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.462+0000 7f0461af2640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f044006b4d0 0x7f044006d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f044c000f60 tx=0x7f044c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.466+0000 7f04527fc640 1 -- 172.21.15.94:0/1269879746 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f045c067910 con 0x7f045c0c8950 2024-03-06T07:19:15.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.703+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f045c04f330 con 0x7f045c0c8950 2024-03-06T07:19:15.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.704+0000 7f04527fc640 1 -- 172.21.15.94:0/1269879746 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7f045807b210 con 0x7f045c0c8950 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f044006b4d0 msgr2=0x7f044006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f044006b4d0 0x7f044006d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f044c000f60 tx=0x7f044c009040 comp rx=0 tx=0).stop 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 msgr2=0x7f045c1eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c1eb210 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0458009870 tx=0x7f0458004730 comp rx=0 tx=0).stop 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 shutdown_connections 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f044006b4d0 0x7f044006d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f045c1558d0 0x7f045c1eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f045c0c8950 0x7f045c1eb210 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.706+0000 7f0463d56640 1 --2- 172.21.15.94:0/1269879746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f045c0c69f0 0x7f045c14efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.707+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 >> 172.21.15.94:0/1269879746 conn(0x7f045c0cae70 msgr2=0x7f045c154690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:15.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.707+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 shutdown_connections 2024-03-06T07:19:15.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.707+0000 7f0463d56640 1 -- 172.21.15.94:0/1269879746 wait complete. 2024-03-06T07:19:15.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:19:15.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.819+0000 7ff55d7f6640 1 Processor -- start 2024-03-06T07:19:15.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.819+0000 7ff55d7f6640 1 -- start start 2024-03-06T07:19:15.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff55d7f6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff558155120 0x7ff5580c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff55d7f6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5581522d0 0x7ff558152740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff55d7f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558159590 con 0x7ff5580bfe90 2024-03-06T07:19:15.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff55d7f6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff558158710 con 0x7ff558155120 2024-03-06T07:19:15.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff55d7f6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5580c59d0 con 0x7ff5581522d0 2024-03-06T07:19:15.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5567fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff558155120 0x7ff5580c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5567fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff558155120 0x7ff5580c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59504/0 (socket says 172.21.15.94:59504) 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5567fc640 1 -- 172.21.15.94:0/3965670728 learned_addr learned my addr 172.21.15.94:0/3965670728 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5577fe640 1 --2- 172.21.15.94:0/3965670728 >> v2:172.21.15.145:3300/0 conn(0x7ff5581522d0 0x7ff558152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5567fc640 1 -- 172.21.15.94:0/3965670728 >> v2:172.21.15.145:3300/0 conn(0x7ff5581522d0 msgr2=0x7ff558152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5567fc640 1 --2- 172.21.15.94:0/3965670728 >> v2:172.21.15.145:3300/0 conn(0x7ff5581522d0 0x7ff558152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5567fc640 1 -- 172.21.15.94:0/3965670728 >> v1:172.21.15.145:6789/0 conn(0x7ff5580bfe90 legacy=0x7ff558158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5567fc640 1 -- 172.21.15.94:0/3965670728 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580c5cb0 con 0x7ff558155120 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.820+0000 7ff5577fe640 1 --2- 172.21.15.94:0/3965670728 >> v2:172.21.15.145:3300/0 conn(0x7ff5581522d0 0x7ff558152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.821+0000 7ff5567fc640 1 --2- 172.21.15.94:0/3965670728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff558155120 0x7ff5580c6d70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009870 tx=0x7ff54c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f9029a86539e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.821+0000 7ff555ffb640 1 -- 172.21.15.94:0/3965670728 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c035020 con 0x7ff558155120 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.821+0000 7ff555ffb640 1 -- 172.21.15.94:0/3965670728 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c031870 con 0x7ff558155120 2024-03-06T07:19:15.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.821+0000 7ff555ffb640 1 -- 172.21.15.94:0/3965670728 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54c031c40 con 0x7ff558155120 2024-03-06T07:19:15.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 -- 172.21.15.94:0/3965670728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff558155120 msgr2=0x7ff5580c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/3965670728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff558155120 0x7ff5580c6d70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff54c009870 tx=0x7ff54c02b910 comp rx=0 tx=0).stop 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 -- 172.21.15.94:0/3965670728 shutdown_connections 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/3965670728 >> v2:172.21.15.145:3300/0 conn(0x7ff5581522d0 0x7ff558152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/3965670728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff558155120 0x7ff5580c6d70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 -- 172.21.15.94:0/3965670728 >> 172.21.15.94:0/3965670728 conn(0x7ff5580acaa0 msgr2=0x7ff5580aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 -- 172.21.15.94:0/3965670728 shutdown_connections 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.822+0000 7ff55d7f6640 1 -- 172.21.15.94:0/3965670728 wait complete. 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.823+0000 7ff55d7f6640 1 Processor -- start 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.823+0000 7ff55d7f6640 1 -- start start 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff55d7f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5580bfe90 0x7ff55814aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff55d7f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5581522d0 0x7ff5581eaf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff55d7f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff558155120 0x7ff5581eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff55d7f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580ca700 con 0x7ff5580bfe90 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff55d7f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff558158960 con 0x7ff558155120 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff55d7f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580ca880 con 0x7ff5581522d0 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5580bfe90 0x7ff55814aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5580bfe90 0x7ff55814aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59506/0 (socket says 172.21.15.94:59506) 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 -- 172.21.15.94:0/177356630 learned_addr learned my addr 172.21.15.94:0/177356630 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff5577fe640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff558155120 0x7ff5581eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff5567fc640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5581522d0 0x7ff5581eaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 -- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5581522d0 msgr2=0x7ff5581eaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5581522d0 0x7ff5581eaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 -- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff558155120 msgr2=0x7ff5581eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff558155120 0x7ff5581eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 -- 172.21.15.94:0/177356630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580ad750 con 0x7ff5580bfe90 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff5577fe640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff558155120 0x7ff5581eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff556ffd640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5580bfe90 0x7ff55814aea0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff54800c7c0 tx=0x7ff54800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff53f7fe640 1 -- 172.21.15.94:0/177356630 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548016020 con 0x7ff5580bfe90 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff53f7fe640 1 -- 172.21.15.94:0/177356630 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff548006d50 con 0x7ff5580bfe90 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5581ef040 con 0x7ff5580bfe90 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.824+0000 7ff53f7fe640 1 -- 172.21.15.94:0/177356630 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548003bb0 con 0x7ff5580bfe90 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.825+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5580b08d0 con 0x7ff5580bfe90 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.826+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55804f330 con 0x7ff5580bfe90 2024-03-06T07:19:15.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.826+0000 7ff53f7fe640 1 -- 172.21.15.94:0/177356630 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff54800a850 con 0x7ff5580bfe90 2024-03-06T07:19:15.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.826+0000 7ff53f7fe640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff54406b4d0 0x7ff54406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:15.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.826+0000 7ff53f7fe640 1 -- 172.21.15.94:0/177356630 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff548089cf0 con 0x7ff5580bfe90 2024-03-06T07:19:15.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.826+0000 7ff5567fc640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff54406b4d0 0x7ff54406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:15.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.827+0000 7ff5567fc640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff54406b4d0 0x7ff54406d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff54c036000 tx=0x7ff54c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:15.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:15.830+0000 7ff53f7fe640 1 -- 172.21.15.94:0/177356630 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff54808e070 con 0x7ff5580bfe90 2024-03-06T07:19:16.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.068+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff5580ca240 con 0x7ff5580bfe90 2024-03-06T07:19:16.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.070+0000 7ff53f7fe640 1 -- 172.21.15.94:0/177356630 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff548020020 con 0x7ff5580bfe90 2024-03-06T07:19:16.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff54406b4d0 msgr2=0x7ff54406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff54406b4d0 0x7ff54406d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff54c036000 tx=0x7ff54c03c040 comp rx=0 tx=0).stop 2024-03-06T07:19:16.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5580bfe90 msgr2=0x7ff55814aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5580bfe90 0x7ff55814aea0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff54800c7c0 tx=0x7ff54800cc80 comp rx=0 tx=0).stop 2024-03-06T07:19:16.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 shutdown_connections 2024-03-06T07:19:16.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff54406b4d0 0x7ff54406d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff558155120 0x7ff5581eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5581522d0 0x7ff5581eaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 --2- 172.21.15.94:0/177356630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5580bfe90 0x7ff55814aea0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 >> 172.21.15.94:0/177356630 conn(0x7ff5580acaa0 msgr2=0x7ff5580c65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:16.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 shutdown_connections 2024-03-06T07:19:16.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.072+0000 7ff55d7f6640 1 -- 172.21.15.94:0/177356630 wait complete. 2024-03-06T07:19:16.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:16.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.183+0000 7f957b9a0640 1 Processor -- start 2024-03-06T07:19:16.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.183+0000 7f957b9a0640 1 -- start start 2024-03-06T07:19:16.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.183+0000 7f957b9a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574152520 0x7f957406ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f957b9a0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95740720a0 0x7f957406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f957b9a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95740c1ae0 con 0x7f9574155c80 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f957b9a0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95740c1dc0 con 0x7f95740720a0 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f957b9a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957406e450 con 0x7f9574152520 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f9579f3d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95740720a0 0x7f957406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f9578f3b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9574155c80 0x7f95740c12b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57242/0 (socket says 172.21.15.94:57242) 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f9578f3b640 1 -- 172.21.15.94:0/2831930004 learned_addr learned my addr 172.21.15.94:0/2831930004 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f957973c640 1 --2- 172.21.15.94:0/2831930004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574152520 0x7f957406ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f9579f3d640 1 -- 172.21.15.94:0/2831930004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574152520 msgr2=0x7f957406ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f9579f3d640 1 --2- 172.21.15.94:0/2831930004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574152520 0x7f957406ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f9579f3d640 1 -- 172.21.15.94:0/2831930004 >> v1:172.21.15.94:6789/0 conn(0x7f9574155c80 legacy=0x7f95740c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f956bfff640 1 -- 172.21.15.94:0/2831930004 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758199349 0 0) 0x7f95740c1ae0 con 0x7f9574155c80 2024-03-06T07:19:16.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.184+0000 7f9579f3d640 1 -- 172.21.15.94:0/2831930004 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957406e730 con 0x7f95740720a0 2024-03-06T07:19:16.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f9579f3d640 1 --2- 172.21.15.94:0/2831930004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f957406afc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f956c009d90 tx=0x7f956c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567679c89e922cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f956bfff640 1 -- 172.21.15.94:0/2831930004 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c039070 con 0x7f95740720a0 2024-03-06T07:19:16.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f956bfff640 1 -- 172.21.15.94:0/2831930004 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c02cc50 con 0x7f95740720a0 2024-03-06T07:19:16.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f956bfff640 1 -- 172.21.15.94:0/2831930004 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c034440 con 0x7f95740720a0 2024-03-06T07:19:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f957b9a0640 1 -- 172.21.15.94:0/2831930004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 msgr2=0x7f957406afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f957b9a0640 1 --2- 172.21.15.94:0/2831930004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f957406afc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f956c009d90 tx=0x7f956c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f957b9a0640 1 -- 172.21.15.94:0/2831930004 shutdown_connections 2024-03-06T07:19:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f957b9a0640 1 --2- 172.21.15.94:0/2831930004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f957406afc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f957b9a0640 1 --2- 172.21.15.94:0/2831930004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574152520 0x7f957406ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.185+0000 7f957b9a0640 1 -- 172.21.15.94:0/2831930004 >> 172.21.15.94:0/2831930004 conn(0x7f95740c76b0 msgr2=0x7f95740c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.186+0000 7f957b9a0640 1 -- 172.21.15.94:0/2831930004 shutdown_connections 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.186+0000 7f957b9a0640 1 -- 172.21.15.94:0/2831930004 wait complete. 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 Processor -- start 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 -- start start 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f9574157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9574152520 0x7f95740a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574155c80 0x7f95740a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9574152a80 con 0x7f95740720a0 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9574152900 con 0x7f9574152520 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957b9a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957406b500 con 0x7f9574155c80 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957973c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f9574157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957973c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f9574157850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59538/0 (socket says 172.21.15.94:59538) 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.187+0000 7f957973c640 1 -- 172.21.15.94:0/3425217297 learned_addr learned my addr 172.21.15.94:0/3425217297 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:16.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f9579f3d640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574155c80 0x7f95740a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f9578f3b640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9574152520 0x7f95740a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957973c640 1 -- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574155c80 msgr2=0x7f95740a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957973c640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574155c80 0x7f95740a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957973c640 1 -- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9574152520 msgr2=0x7f95740a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957973c640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9574152520 0x7f95740a6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957973c640 1 -- 172.21.15.94:0/3425217297 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95741567b0 con 0x7f95740720a0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f9579f3d640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574155c80 0x7f95740a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f9578f3b640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9574152520 0x7f95740a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957973c640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f9574157850 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f957000e9b0 tx=0x7f957000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f9569ffb640 1 -- 172.21.15.94:0/3425217297 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570017070 con 0x7f95740720a0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f9569ffb640 1 -- 172.21.15.94:0/3425217297 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957000c9a0 con 0x7f95740720a0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9574156a90 con 0x7f95740720a0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f9569ffb640 1 -- 172.21.15.94:0/3425217297 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957000cd90 con 0x7f95740720a0 2024-03-06T07:19:16.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.188+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9574157050 con 0x7f95740720a0 2024-03-06T07:19:16.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.189+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957404f330 con 0x7f95740720a0 2024-03-06T07:19:16.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.190+0000 7f9569ffb640 1 -- 172.21.15.94:0/3425217297 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f957001a070 con 0x7f95740720a0 2024-03-06T07:19:16.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.190+0000 7f9569ffb640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955806b4d0 0x7f955806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.190+0000 7f9569ffb640 1 -- 172.21.15.94:0/3425217297 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f957008a390 con 0x7f95740720a0 2024-03-06T07:19:16.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.190+0000 7f9578f3b640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955806b4d0 0x7f955806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.191+0000 7f9578f3b640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955806b4d0 0x7f955806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9574155bc0 tx=0x7f9564009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.194+0000 7f9569ffb640 1 -- 172.21.15.94:0/3425217297 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9570059460 con 0x7f95740720a0 2024-03-06T07:19:16.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.366+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95741512f0 con 0x7f95740720a0 2024-03-06T07:19:16.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.367+0000 7f9569ffb640 1 -- 172.21.15.94:0/3425217297 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13164 (secure 0 0 0) 0x7f9570058d60 con 0x7f95740720a0 2024-03-06T07:19:16.369 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:16.369 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":35,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:13.421401+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955806b4d0 msgr2=0x7f955806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955806b4d0 0x7f955806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9574155bc0 tx=0x7f9564009040 comp rx=0 tx=0).stop 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 msgr2=0x7f9574157850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f9574157850 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f957000e9b0 tx=0x7f957000ee70 comp rx=0 tx=0).stop 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 shutdown_connections 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955806b4d0 0x7f955806d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9574155c80 0x7f95740a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9574152520 0x7f95740a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 --2- 172.21.15.94:0/3425217297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95740720a0 0x7f9574157850 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 >> 172.21.15.94:0/3425217297 conn(0x7f95740c76b0 msgr2=0x7f95740c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:16.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 shutdown_connections 2024-03-06T07:19:16.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.369+0000 7f957b9a0640 1 -- 172.21.15.94:0/3425217297 wait complete. 2024-03-06T07:19:16.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:19:16.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:19:16.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a55a0640 1 Processor -- start 2024-03-06T07:19:16.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a55a0640 1 -- start start 2024-03-06T07:19:16.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a55a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 0x7fc0a01592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a55a0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc0a006afc0 0x7fc0a006b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a55a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a01552c0 con 0x7fc0a00c8640 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a55a0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a00c6d30 con 0x7fc0a006afc0 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a55a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0155580 con 0x7fc0a0152a20 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a4d9f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc0a00c8640 0x7fc0a00c50c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57256/0 (socket says 172.21.15.94:57256) 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.484+0000 7fc0a4d9f640 1 -- 172.21.15.94:0/3616895372 learned_addr learned my addr 172.21.15.94:0/3616895372 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152867116 0 0) 0x7fc0a01552c0 con 0x7fc0a00c8640 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09ffff640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 0x7fc0a01592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088003520 con 0x7fc0a00c8640 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/3616895372 >> v2:172.21.15.145:3300/0 conn(0x7fc0a006afc0 0x7fc0a006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142425053 0 0) 0x7fc088003520 con 0x7fc0a00c8640 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 msgr2=0x7fc0a01592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 0x7fc0a01592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 >> v2:172.21.15.145:3300/0 conn(0x7fc0a006afc0 msgr2=0x7fc0a006b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 --2- 172.21.15.94:0/3616895372 >> v2:172.21.15.145:3300/0 conn(0x7fc0a006afc0 0x7fc0a006b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a0155860 con 0x7fc0a00c8640 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fc0980019f0 con 0x7fc0a00c8640 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 >> v1:172.21.15.94:6789/0 conn(0x7fc0a00c8640 legacy=0x7fc0a00c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 0x7fc0880039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc088004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc088004d40 0x7fc0880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a00c6d30 con 0x7fc088003f30 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0155580 con 0x7fc088004d40 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08800baa0 con 0x7fc0a0152a20 2024-03-06T07:19:16.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc088004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09ffff640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 0x7fc0880039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc0a4d9f640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc088004d40 0x7fc0880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09f7fe640 1 -- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 msgr2=0x7fc0880039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a0152a20 0x7fc0880039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09f7fe640 1 -- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc088004d40 msgr2=0x7fc0880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc088004d40 0x7fc0880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc09f7fe640 1 -- 172.21.15.94:0/3616895372 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088003520 con 0x7fc088003f30 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.485+0000 7fc0a4d9f640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc088004d40 0x7fc0880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.486+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc088004800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc090009ac0 tx=0x7fc088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607316e845ff023d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.486+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090039070 con 0x7fc088003f30 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.486+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09002c710 con 0x7fc088003f30 2024-03-06T07:19:16.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.486+0000 7fc09effd640 1 -- 172.21.15.94:0/3616895372 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090009690 con 0x7fc088003f30 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.487+0000 7fc0a55a0640 1 -- 172.21.15.94:0/3616895372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 msgr2=0x7fc088004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.487+0000 7fc0a55a0640 1 --2- 172.21.15.94:0/3616895372 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc088004800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc090009ac0 tx=0x7fc088005830 comp rx=0 tx=0).stop 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.487+0000 7fc0a4d9f640 1 -- 172.21.15.94:0/3616895372 reap_dead start 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.487+0000 7fc0a55a0640 1 -- 172.21.15.94:0/3616895372 shutdown_connections 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.487+0000 7fc0a55a0640 1 -- 172.21.15.94:0/3616895372 >> 172.21.15.94:0/3616895372 conn(0x7fc0a00ca740 msgr2=0x7fc0a00cab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.487+0000 7fc0a55a0640 1 -- 172.21.15.94:0/3616895372 shutdown_connections 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.487+0000 7fc0a55a0640 1 -- 172.21.15.94:0/3616895372 wait complete. 2024-03-06T07:19:16.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.488+0000 7fc0a55a0640 1 Processor -- start 2024-03-06T07:19:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.488+0000 7fc0a55a0640 1 -- start start 2024-03-06T07:19:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a55a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc0a014fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a55a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0a014d980 0x7fc0a014de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a55a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a00be160 0x7fc0a00be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a55a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0156060 con 0x7fc088003f30 2024-03-06T07:19:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a55a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0152e00 con 0x7fc0a014d980 2024-03-06T07:19:16.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a55a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01e1e60 con 0x7fc0a00be160 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc0a014fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc0a014fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59556/0 (socket says 172.21.15.94:59556) 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 -- 172.21.15.94:0/1848858832 learned_addr learned my addr 172.21.15.94:0/1848858832 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0a014d980 0x7fc0a014de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a4d9f640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a00be160 0x7fc0a00be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 -- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a00be160 msgr2=0x7fc0a00be5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a00be160 0x7fc0a00be5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 -- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0a014d980 msgr2=0x7fc0a014de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0a014d980 0x7fc0a014de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 -- 172.21.15.94:0/1848858832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a01e2140 con 0x7fc088003f30 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc0a4d9f640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a00be160 0x7fc0a00be5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09ffff640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc0a014fa90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc0940119a0 tx=0x7fc094011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.489+0000 7fc09cff9640 1 -- 172.21.15.94:0/1848858832 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09400fa90 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.490+0000 7fc09cff9640 1 -- 172.21.15.94:0/1848858832 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09400fc30 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.490+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a00c0c50 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.490+0000 7fc09cff9640 1 -- 172.21.15.94:0/1848858832 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094014440 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.490+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a00c1210 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.490+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a00684b0 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.491+0000 7fc09cff9640 1 -- 172.21.15.94:0/1848858832 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc094012040 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.491+0000 7fc09cff9640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc08c06b500 0x7fc08c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.492+0000 7fc09cff9640 1 -- 172.21.15.94:0/1848858832 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc09408a4a0 con 0x7fc088003f30 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.492+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc08c06b500 0x7fc08c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.492+0000 7fc09f7fe640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc08c06b500 0x7fc08c06d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc090036000 tx=0x7fc090030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.496+0000 7fc09cff9640 1 -- 172.21.15.94:0/1848858832 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc094019030 con 0x7fc088003f30 2024-03-06T07:19:16.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.665+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0a00735a0 con 0x7fc088003f30 2024-03-06T07:19:16.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.666+0000 7fc09cff9640 1 -- 172.21.15.94:0/1848858832 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0a00735a0 con 0x7fc088003f30 2024-03-06T07:19:16.668 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:19:16.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc08c06b500 msgr2=0x7fc08c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc08c06b500 0x7fc08c06d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc090036000 tx=0x7fc090030000 comp rx=0 tx=0).stop 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 msgr2=0x7fc0a014fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc0a014fa90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc0940119a0 tx=0x7fc094011e60 comp rx=0 tx=0).stop 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 shutdown_connections 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc08c06b500 0x7fc08c06d9c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0a00be160 0x7fc0a00be5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0a014d980 0x7fc0a014de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 --2- 172.21.15.94:0/1848858832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc088003f30 0x7fc0a014fa90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.667+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 >> 172.21.15.94:0/1848858832 conn(0x7fc0a00ca740 msgr2=0x7fc0a00ab140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.668+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 shutdown_connections 2024-03-06T07:19:16.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.668+0000 7fc0a55a0640 1 -- 172.21.15.94:0/1848858832 wait complete. 2024-03-06T07:19:16.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:19:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.780+0000 7f3d35b79640 1 Processor -- start 2024-03-06T07:19:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.780+0000 7f3d35b79640 1 -- start start 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d35b79640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3d3014f310 0x7f3d300c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d35b79640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d30155bb0 con 0x7f3d300aa180 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d35b79640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300aa660 con 0x7f3d301550c0 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d35b79640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d300c4610 con 0x7f3d3014f310 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3d300aa180 0x7f3d3014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57268/0 (socket says 172.21.15.94:57268) 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2effd640 1 -- 172.21.15.94:0/2200259204 learned_addr learned my addr 172.21.15.94:0/2200259204 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561249109 0 0) 0x7f3d30155bb0 con 0x7f3d300aa180 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d18004630 con 0x7f3d300aa180 2024-03-06T07:19:16.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2ffff640 1 --2- 172.21.15.94:0/2200259204 >> v2:172.21.15.145:3300/0 conn(0x7f3d3014f310 0x7f3d300c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1005317321 0 0) 0x7f3d18004630 con 0x7f3d300aa180 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 >> v2:172.21.15.145:3300/0 conn(0x7f3d3014f310 msgr2=0x7f3d300c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 --2- 172.21.15.94:0/2200259204 >> v2:172.21.15.145:3300/0 conn(0x7f3d3014f310 0x7f3d300c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 >> v1:172.21.15.145:6789/0 conn(0x7f3d301550c0 legacy=0x7f3d301554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d300aa940 con 0x7f3d300aa180 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f3d24001670 con 0x7f3d300aa180 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 >> v1:172.21.15.94:6789/0 conn(0x7f3d300aa180 legacy=0x7f3d3014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d18004ac0 0x7f3d18004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d18005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.781+0000 7f3d2e7fc640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d180063f0 0x7f3d1800cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300c4610 con 0x7f3d18005910 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1800d2c0 con 0x7f3d18004ac0 2024-03-06T07:19:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1800d4c0 con 0x7f3d180063f0 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2effd640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d18005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2f7fe640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d18004ac0 0x7f3d18004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2ffff640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d180063f0 0x7f3d1800cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2effd640 1 -- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d180063f0 msgr2=0x7f3d1800cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2effd640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d180063f0 0x7f3d1800cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2effd640 1 -- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d18004ac0 msgr2=0x7f3d18004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2effd640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d18004ac0 0x7f3d18004ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2effd640 1 -- 172.21.15.94:0/2200259204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d18004630 con 0x7f3d18005910 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2ffff640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d180063f0 0x7f3d1800cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.782+0000 7f3d2ffff640 1 -- 172.21.15.94:0/2200259204 reap_dead start 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d2effd640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d18005d90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3d300aa590 tx=0x7f3d18006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5facf1452775b754 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24007d10 con 0x7f3d18005910 2024-03-06T07:19:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d24030070 con 0x7f3d18005910 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d2e7fc640 1 -- 172.21.15.94:0/2200259204 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24005d90 con 0x7f3d18005910 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d35b79640 1 -- 172.21.15.94:0/2200259204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 msgr2=0x7f3d18005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d35b79640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d18005d90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3d300aa590 tx=0x7f3d18006940 comp rx=0 tx=0).stop 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d35b79640 1 -- 172.21.15.94:0/2200259204 shutdown_connections 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d35b79640 1 --2- 172.21.15.94:0/2200259204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d18005d90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d35b79640 1 -- 172.21.15.94:0/2200259204 >> 172.21.15.94:0/2200259204 conn(0x7f3d300c6710 msgr2=0x7f3d300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d35b79640 1 -- 172.21.15.94:0/2200259204 shutdown_connections 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.783+0000 7f3d35b79640 1 -- 172.21.15.94:0/2200259204 wait complete. 2024-03-06T07:19:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.784+0000 7f3d35b79640 1 Processor -- start 2024-03-06T07:19:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d35b79640 1 -- start start 2024-03-06T07:19:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d35b79640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d30071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d35b79640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d300a7480 0x7f3d30142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d35b79640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d30143110 0x7f3d300a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d35b79640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300c1490 con 0x7f3d18005910 2024-03-06T07:19:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d35b79640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300c1310 con 0x7f3d30143110 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d35b79640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3014ec00 con 0x7f3d300a7480 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d2f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d30071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d2f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d30071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59574/0 (socket says 172.21.15.94:59574) 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d2f7fe640 1 -- 172.21.15.94:0/1039793065 learned_addr learned my addr 172.21.15.94:0/1039793065 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d2effd640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d300a7480 0x7f3d30142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.785+0000 7f3d2ffff640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d30143110 0x7f3d300a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d2f7fe640 1 -- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d300a7480 msgr2=0x7f3d30142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d2f7fe640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d300a7480 0x7f3d30142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d2f7fe640 1 -- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d30143110 msgr2=0x7f3d300a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d2f7fe640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d30143110 0x7f3d300a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d2f7fe640 1 -- 172.21.15.94:0/1039793065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30072190 con 0x7f3d18005910 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d2f7fe640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d30071a90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000f620 tx=0x7f3d2000fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d17fff640 1 -- 172.21.15.94:0/1039793065 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d200052e0 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d17fff640 1 -- 172.21.15.94:0/1039793065 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d20018070 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d30072710 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d17fff640 1 -- 172.21.15.94:0/1039793065 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20003810 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.786+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d30072cd0 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.787+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3004f330 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.787+0000 7f3d17fff640 1 -- 172.21.15.94:0/1039793065 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d200039b0 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.788+0000 7f3d17fff640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.788+0000 7f3d17fff640 1 -- 172.21.15.94:0/1039793065 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3d20089bc0 con 0x7f3d18005910 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.788+0000 7f3d2effd640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.788+0000 7f3d2effd640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3d24034000 tx=0x7f3d24009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.792+0000 7f3d17fff640 1 -- 172.21.15.94:0/1039793065 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d200585b0 con 0x7f3d18005910 2024-03-06T07:19:16.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.965+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3d30068450 con 0x7f3d18005910 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.966+0000 7f3d17fff640 1 -- 172.21.15.94:0/1039793065 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3d2005c650 con 0x7f3d18005910 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:16.968 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.967+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d1c06b500 msgr2=0x7f3d1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.967+0000 7f3d35b79640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3d24034000 tx=0x7f3d24009030 comp rx=0 tx=0).stop 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 msgr2=0x7f3d30071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d30071a90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000f620 tx=0x7f3d2000fae0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 shutdown_connections 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d30143110 0x7f3d300a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d300a7480 0x7f3d30142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 --2- 172.21.15.94:0/1039793065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d18005910 0x7f3d30071a90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:16.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 >> 172.21.15.94:0/1039793065 conn(0x7f3d300c6710 msgr2=0x7f3d300a9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:16.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 shutdown_connections 2024-03-06T07:19:16.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:16.968+0000 7f3d35b79640 1 -- 172.21.15.94:0/1039793065 wait complete. 2024-03-06T07:19:16.980 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:19:16.980 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:19:17.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.081+0000 7f38467a0640 1 Processor -- start 2024-03-06T07:19:17.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f38467a0640 1 -- start start 2024-03-06T07:19:17.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f38467a0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f38401528e0 0x7f384006c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f38467a0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f384006cae0 0x7f38400c5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f38467a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400c5d10 con 0x7f38400c5670 2024-03-06T07:19:17.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f38467a0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384006be10 con 0x7f38401528e0 2024-03-06T07:19:17.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f38467a0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3840155e70 con 0x7f384006cae0 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f384579e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f38401528e0 0x7f384006c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f3845f9f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f38400c5670 0x7f38400c86b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57280/0 (socket says 172.21.15.94:57280) 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.082+0000 7f3845f9f640 1 -- 172.21.15.94:0/786858090 learned_addr learned my addr 172.21.15.94:0/786858090 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f3844f9d640 1 --2- 172.21.15.94:0/786858090 >> v2:172.21.15.145:3300/0 conn(0x7f384006cae0 0x7f38400c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f384579e640 1 -- 172.21.15.94:0/786858090 >> v2:172.21.15.145:3300/0 conn(0x7f384006cae0 msgr2=0x7f38400c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f384579e640 1 --2- 172.21.15.94:0/786858090 >> v2:172.21.15.145:3300/0 conn(0x7f384006cae0 0x7f38400c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f3837fff640 1 -- 172.21.15.94:0/786858090 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439788322 0 0) 0x7f383c001490 con 0x7f38400c5670 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f384579e640 1 -- 172.21.15.94:0/786858090 >> v1:172.21.15.94:6789/0 conn(0x7f38400c5670 legacy=0x7f38400c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f384579e640 1 -- 172.21.15.94:0/786858090 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3840155540 con 0x7f38401528e0 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f3844f9d640 1 --2- 172.21.15.94:0/786858090 >> v2:172.21.15.145:3300/0 conn(0x7f384006cae0 0x7f38400c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.083+0000 7f384579e640 1 --2- 172.21.15.94:0/786858090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38401528e0 0x7f384006c5a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3840068c00 tx=0x7f383802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a831d4281dfa8547 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f3837fff640 1 -- 172.21.15.94:0/786858090 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380315a0 con 0x7f38401528e0 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f3837fff640 1 -- 172.21.15.94:0/786858090 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3838037070 con 0x7f38401528e0 2024-03-06T07:19:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f3837fff640 1 -- 172.21.15.94:0/786858090 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838031960 con 0x7f38401528e0 2024-03-06T07:19:17.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 -- 172.21.15.94:0/786858090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38401528e0 msgr2=0x7f384006c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 --2- 172.21.15.94:0/786858090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38401528e0 0x7f384006c5a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3840068c00 tx=0x7f383802b840 comp rx=0 tx=0).stop 2024-03-06T07:19:17.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 -- 172.21.15.94:0/786858090 shutdown_connections 2024-03-06T07:19:17.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 --2- 172.21.15.94:0/786858090 >> v2:172.21.15.145:3300/0 conn(0x7f384006cae0 0x7f38400c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 --2- 172.21.15.94:0/786858090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38401528e0 0x7f384006c5a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 -- 172.21.15.94:0/786858090 >> 172.21.15.94:0/786858090 conn(0x7f38400c7230 msgr2=0x7f38400c7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 -- 172.21.15.94:0/786858090 shutdown_connections 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.084+0000 7f38467a0640 1 -- 172.21.15.94:0/786858090 wait complete. 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.085+0000 7f38467a0640 1 Processor -- start 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.085+0000 7f38467a0640 1 -- start start 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f38467a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f384006cae0 0x7f38400ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f38467a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38400c5670 0x7f38400ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f38467a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38401528e0 0x7f38400aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f38467a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840152cc0 con 0x7f38400c5670 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f38467a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840155b00 con 0x7f384006cae0 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f38467a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400c8dc0 con 0x7f38401528e0 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38400c5670 0x7f38400ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38400c5670 0x7f38400ae9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59588/0 (socket says 172.21.15.94:59588) 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 -- 172.21.15.94:0/81054503 learned_addr learned my addr 172.21.15.94:0/81054503 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:17.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f384579e640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f384006cae0 0x7f38400ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 -- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38401528e0 msgr2=0x7f38400aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38401528e0 0x7f38400aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 -- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f384006cae0 msgr2=0x7f38400ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f384006cae0 0x7f38400ac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.086+0000 7f3844f9d640 1 -- 172.21.15.94:0/81054503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400af440 con 0x7f38400c5670 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.087+0000 7f3844f9d640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38400c5670 0x7f38400ae9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f383000c820 tx=0x7f383000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.087+0000 7f3835ffb640 1 -- 172.21.15.94:0/81054503 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830015070 con 0x7f38400c5670 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.087+0000 7f3835ffb640 1 -- 172.21.15.94:0/81054503 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3830007a30 con 0x7f38400c5670 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.087+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38400adc00 con 0x7f38400c5670 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.087+0000 7f3835ffb640 1 -- 172.21.15.94:0/81054503 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830007e00 con 0x7f38400c5670 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.087+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38400ae1c0 con 0x7f38400c5670 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.088+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38400684b0 con 0x7f38400c5670 2024-03-06T07:19:17.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.088+0000 7f3835ffb640 1 -- 172.21.15.94:0/81054503 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3830018070 con 0x7f38400c5670 2024-03-06T07:19:17.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.088+0000 7f3835ffb640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382406b4d0 0x7f382406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.089+0000 7f3835ffb640 1 -- 172.21.15.94:0/81054503 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3830089f90 con 0x7f38400c5670 2024-03-06T07:19:17.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.089+0000 7f384579e640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382406b4d0 0x7f382406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.089+0000 7f384579e640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382406b4d0 0x7f382406d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3838036000 tx=0x7f3838008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:17.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.093+0000 7f3835ffb640 1 -- 172.21.15.94:0/81054503 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3830059060 con 0x7f38400c5670 2024-03-06T07:19:17.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.265+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f384004f330 con 0x7f38400c5670 2024-03-06T07:19:17.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.438+0000 7f3835ffb640 1 -- 172.21.15.94:0/81054503 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f384004f330 con 0x7f38400c5670 2024-03-06T07:19:17.440 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382406b4d0 msgr2=0x7f382406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382406b4d0 0x7f382406d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3838036000 tx=0x7f3838008030 comp rx=0 tx=0).stop 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38400c5670 msgr2=0x7f38400ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38400c5670 0x7f38400ae9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f383000c820 tx=0x7f383000cc20 comp rx=0 tx=0).stop 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 shutdown_connections 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382406b4d0 0x7f382406d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38401528e0 0x7f38400aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38400c5670 0x7f38400ae9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 --2- 172.21.15.94:0/81054503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f384006cae0 0x7f38400ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.440+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 >> 172.21.15.94:0/81054503 conn(0x7f38400c7230 msgr2=0x7f3840154b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:17.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.441+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 shutdown_connections 2024-03-06T07:19:17.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.441+0000 7f38467a0640 1 -- 172.21.15.94:0/81054503 wait complete. 2024-03-06T07:19:17.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:19:17.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.561+0000 7ff5c15bc640 1 Processor -- start 2024-03-06T07:19:17.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.561+0000 7ff5c15bc640 1 -- start start 2024-03-06T07:19:17.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5c15bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5bc0c4740 0x7ff5bc1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5c15bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5bc0c62a0 con 0x7ff5bc0a8290 2024-03-06T07:19:17.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5c15bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5bc0c4c00 con 0x7ff5bc0c69a0 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5c15bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5bc153f00 con 0x7ff5bc0c4740 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5baffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff5bc0a8290 0x7ff5bc0701a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57290/0 (socket says 172.21.15.94:57290) 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5baffd640 1 -- 172.21.15.94:0/1229724605 learned_addr learned my addr 172.21.15.94:0/1229724605 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5ba7fc640 1 --2- 172.21.15.94:0/1229724605 >> v2:172.21.15.94:3300/0 conn(0x7ff5bc0c4740 0x7ff5bc1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5ba7fc640 1 -- 172.21.15.94:0/1229724605 >> v1:172.21.15.145:6789/0 conn(0x7ff5bc0c69a0 legacy=0x7ff5bc072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5b9ffb640 1 -- 172.21.15.94:0/1229724605 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077155642 0 0) 0x7ff5bc0c62a0 con 0x7ff5bc0a8290 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5ba7fc640 1 -- 172.21.15.94:0/1229724605 >> v1:172.21.15.94:6789/0 conn(0x7ff5bc0a8290 legacy=0x7ff5bc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.562+0000 7ff5ba7fc640 1 -- 172.21.15.94:0/1229724605 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc0c4ee0 con 0x7ff5bc0c4740 2024-03-06T07:19:17.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.563+0000 7ff5ba7fc640 1 --2- 172.21.15.94:0/1229724605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0c4740 0x7ff5bc1539c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00097a0 tx=0x7ff5b002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96663973c92bfc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:17.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.563+0000 7ff5b9ffb640 1 -- 172.21.15.94:0/1229724605 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0038070 con 0x7ff5bc0c4740 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.563+0000 7ff5b9ffb640 1 -- 172.21.15.94:0/1229724605 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b0031700 con 0x7ff5bc0c4740 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.563+0000 7ff5b9ffb640 1 -- 172.21.15.94:0/1229724605 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0031ad0 con 0x7ff5bc0c4740 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.564+0000 7ff5c15bc640 1 -- 172.21.15.94:0/1229724605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0c4740 msgr2=0x7ff5bc1539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.564+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/1229724605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0c4740 0x7ff5bc1539c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00097a0 tx=0x7ff5b002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.564+0000 7ff5c15bc640 1 -- 172.21.15.94:0/1229724605 shutdown_connections 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.564+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/1229724605 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0c4740 0x7ff5bc1539c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.564+0000 7ff5c15bc640 1 -- 172.21.15.94:0/1229724605 >> 172.21.15.94:0/1229724605 conn(0x7ff5bc0c5560 msgr2=0x7ff5bc0c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.564+0000 7ff5c15bc640 1 -- 172.21.15.94:0/1229724605 shutdown_connections 2024-03-06T07:19:17.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.564+0000 7ff5c15bc640 1 -- 172.21.15.94:0/1229724605 wait complete. 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.565+0000 7ff5c15bc640 1 Processor -- start 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.565+0000 7ff5c15bc640 1 -- start start 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5c15bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0a8290 0x7ff5bc1e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5c15bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5bc0c69a0 0x7ff5bc156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5c15bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5bc14d790 0x7ff5bc14dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5c15bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc072d90 con 0x7ff5bc0a8290 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5c15bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc06fc80 con 0x7ff5bc14d790 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5c15bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc072f10 con 0x7ff5bc0c69a0 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0a8290 0x7ff5bc1e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0a8290 0x7ff5bc1e9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59600/0 (socket says 172.21.15.94:59600) 2024-03-06T07:19:17.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 -- 172.21.15.94:0/3948262584 learned_addr learned my addr 172.21.15.94:0/3948262584 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 -- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5bc0c69a0 msgr2=0x7ff5bc156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5bc0c69a0 0x7ff5bc156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 -- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5bc14d790 msgr2=0x7ff5bc14dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5bc14d790 0x7ff5bc14dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 -- 172.21.15.94:0/3948262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc14e150 con 0x7ff5bc0a8290 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5baffd640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0a8290 0x7ff5bc1e9a80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00c8c0 tx=0x7ff5ac00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5a37fe640 1 -- 172.21.15.94:0/3948262584 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac018070 con 0x7ff5bc0a8290 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.566+0000 7ff5a37fe640 1 -- 172.21.15.94:0/3948262584 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ac00a5e0 con 0x7ff5bc0a8290 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.567+0000 7ff5a37fe640 1 -- 172.21.15.94:0/3948262584 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00a9d0 con 0x7ff5bc0a8290 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.567+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5bc14b280 con 0x7ff5bc0a8290 2024-03-06T07:19:17.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.567+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5bc14b840 con 0x7ff5bc0a8290 2024-03-06T07:19:17.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.567+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5bc04f330 con 0x7ff5bc0a8290 2024-03-06T07:19:17.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.568+0000 7ff5a37fe640 1 -- 172.21.15.94:0/3948262584 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5ac00ab70 con 0x7ff5bc0a8290 2024-03-06T07:19:17.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.568+0000 7ff5a37fe640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5a806b4d0 0x7ff5a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:17.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.568+0000 7ff5a37fe640 1 -- 172.21.15.94:0/3948262584 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5619+0+0 (secure 0 0 0) 0x7ff5ac0897e0 con 0x7ff5bc0a8290 2024-03-06T07:19:17.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.569+0000 7ff5ba7fc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5a806b4d0 0x7ff5a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:17.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.569+0000 7ff5ba7fc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5a806b4d0 0x7ff5a806d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0035000 tx=0x7ff5b0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:17.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.573+0000 7ff5a37fe640 1 -- 172.21.15.94:0/3948262584 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5bc14e150 con 0x7ff5bc0a8290 2024-03-06T07:19:17.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:17.743+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff5bc0c30b0 con 0x7ff5bc0a8290 2024-03-06T07:19:18.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.440+0000 7ff5a37fe640 1 -- 172.21.15.94:0/3948262584 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff5ac058740 con 0x7ff5bc0a8290 2024-03-06T07:19:18.442 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:19:18.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.443+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5a806b4d0 msgr2=0x7ff5a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:18.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.443+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5a806b4d0 0x7ff5a806d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0035000 tx=0x7ff5b0030000 comp rx=0 tx=0).stop 2024-03-06T07:19:18.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.443+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0a8290 msgr2=0x7ff5bc1e9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:18.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.443+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0a8290 0x7ff5bc1e9a80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00c8c0 tx=0x7ff5ac00cd80 comp rx=0 tx=0).stop 2024-03-06T07:19:18.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.444+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 shutdown_connections 2024-03-06T07:19:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.444+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5a806b4d0 0x7ff5a806d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.444+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5bc14d790 0x7ff5bc14dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.444+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5bc0c69a0 0x7ff5bc156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.444+0000 7ff5c15bc640 1 --2- 172.21.15.94:0/3948262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5bc0a8290 0x7ff5bc1e9a80 secure :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00c8c0 tx=0x7ff5ac00cd80 comp rx=0 tx=0).stop 2024-03-06T07:19:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.444+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 >> 172.21.15.94:0/3948262584 conn(0x7ff5bc0c5560 msgr2=0x7ff5bc0c3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.445+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 shutdown_connections 2024-03-06T07:19:18.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.445+0000 7ff5c15bc640 1 -- 172.21.15.94:0/3948262584 wait complete. 2024-03-06T07:19:18.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:19:18.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.578+0000 7f272438a640 1 Processor -- start 2024-03-06T07:19:18.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.578+0000 7f272438a640 1 -- start start 2024-03-06T07:19:18.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.579+0000 7f272438a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f271c0a8ff0 0x7f271c0ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:18.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.579+0000 7f272438a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f271c0a6290 0x7f271c0a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:18.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.579+0000 7f272438a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c0ab730 0x7f271c0aa360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:18.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.579+0000 7f272438a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271c14f850 con 0x7f271c0a6290 2024-03-06T07:19:18.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.579+0000 7f272438a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271c156260 con 0x7f271c0a8ff0 2024-03-06T07:19:18.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.579+0000 7f272438a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0aa8a0 con 0x7f271c0ab730 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f271c0a6290 0x7f271c0a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f271c0a6290 0x7f271c0a9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50802/0 (socket says 172.21.15.94:50802) 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 -- 172.21.15.94:0/749464271 learned_addr learned my addr 172.21.15.94:0/749464271 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2722927640 1 --2- 172.21.15.94:0/749464271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c0ab730 0x7f271c0aa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 -- 172.21.15.94:0/749464271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c0ab730 msgr2=0x7f271c0aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2722126640 1 --2- 172.21.15.94:0/749464271 >> v2:172.21.15.145:3300/0 conn(0x7f271c0a8ff0 0x7f271c0ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 --2- 172.21.15.94:0/749464271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c0ab730 0x7f271c0aa360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 -- 172.21.15.94:0/749464271 >> v2:172.21.15.145:3300/0 conn(0x7f271c0a8ff0 msgr2=0x7f271c0ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 --2- 172.21.15.94:0/749464271 >> v2:172.21.15.145:3300/0 conn(0x7f271c0a8ff0 0x7f271c0ab1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.580+0000 7f2721925640 1 -- 172.21.15.94:0/749464271 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c0c1f90 con 0x7f271c0a6290 2024-03-06T07:19:18.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f2721925640 1 --2- 172.21.15.94:0/749464271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0a6290 0x7f271c0a9e20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f271c0c24b0 tx=0x7f270c02dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debc8cb0de82b320 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f2721124640 1 -- 172.21.15.94:0/749464271 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c03b070 con 0x7f271c0a6290 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f2721124640 1 -- 172.21.15.94:0/749464271 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c02ed70 con 0x7f271c0a6290 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f2721124640 1 -- 172.21.15.94:0/749464271 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c036660 con 0x7f271c0a6290 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 -- 172.21.15.94:0/749464271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0a6290 msgr2=0x7f271c0a9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 --2- 172.21.15.94:0/749464271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0a6290 0x7f271c0a9e20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f271c0c24b0 tx=0x7f270c02dfc0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 -- 172.21.15.94:0/749464271 shutdown_connections 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 --2- 172.21.15.94:0/749464271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c0ab730 0x7f271c0aa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 --2- 172.21.15.94:0/749464271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0a6290 0x7f271c0a9e20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 --2- 172.21.15.94:0/749464271 >> v2:172.21.15.145:3300/0 conn(0x7f271c0a8ff0 0x7f271c0ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 -- 172.21.15.94:0/749464271 >> 172.21.15.94:0/749464271 conn(0x7f271c06afc0 msgr2=0x7f271c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 -- 172.21.15.94:0/749464271 shutdown_connections 2024-03-06T07:19:18.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.581+0000 7f272438a640 1 -- 172.21.15.94:0/749464271 wait complete. 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.582+0000 7f272438a640 1 Processor -- start 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f272438a640 1 -- start start 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f272438a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f271c0a8ff0 0x7f271c154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f272438a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0ab730 0x7f271c1ea240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f272438a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c071170 0x7f271c1ea780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f272438a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0c1790 con 0x7f271c0ab730 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f272438a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0c14b0 con 0x7f271c0a8ff0 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f272438a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0c1a70 con 0x7f271c071170 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f2721925640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0ab730 0x7f271c1ea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f2721925640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0ab730 0x7f271c1ea240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50816/0 (socket says 172.21.15.94:50816) 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f2721925640 1 -- 172.21.15.94:0/824947664 learned_addr learned my addr 172.21.15.94:0/824947664 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:18.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f2722126640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f271c0a8ff0 0x7f271c154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.583+0000 7f2722927640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c071170 0x7f271c1ea780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2721925640 1 -- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c071170 msgr2=0x7f271c1ea780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2721925640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c071170 0x7f271c1ea780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2721925640 1 -- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f271c0a8ff0 msgr2=0x7f271c154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2721925640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f271c0a8ff0 0x7f271c154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2721925640 1 -- 172.21.15.94:0/824947664 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c153b90 con 0x7f271c0ab730 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2721925640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0ab730 0x7f271c1ea240 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f270c02e000 tx=0x7f270c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2712ffd640 1 -- 172.21.15.94:0/824947664 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c03b040 con 0x7f271c0ab730 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c1eadf0 con 0x7f271c0ab730 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2712ffd640 1 -- 172.21.15.94:0/824947664 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c046070 con 0x7f271c0ab730 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f2712ffd640 1 -- 172.21.15.94:0/824947664 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c03f890 con 0x7f271c0ab730 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.584+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c1eb0d0 con 0x7f271c0ab730 2024-03-06T07:19:18.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.585+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c04f330 con 0x7f271c0ab730 2024-03-06T07:19:18.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.585+0000 7f2712ffd640 1 -- 172.21.15.94:0/824947664 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f270c02ec10 con 0x7f271c0ab730 2024-03-06T07:19:18.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.586+0000 7f2712ffd640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f270006b4d0 0x7f270006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:18.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.586+0000 7f2712ffd640 1 -- 172.21.15.94:0/824947664 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f270c0aea40 con 0x7f271c0ab730 2024-03-06T07:19:18.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.586+0000 7f2722126640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f270006b4d0 0x7f270006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:18.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.587+0000 7f2722126640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f270006b4d0 0x7f270006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f271c0c3120 tx=0x7f2718008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:18.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.590+0000 7f2712ffd640 1 -- 172.21.15.94:0/824947664 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f270c07d150 con 0x7f271c0ab730 2024-03-06T07:19:18.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:18.799+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f271c0732b0 con 0x7f271c0ab730 2024-03-06T07:19:19.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.461+0000 7f2712ffd640 1 -- 172.21.15.94:0/824947664 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v15) v1 ==== 172+0+0 (secure 0 0 0) 0x7f270c0811f0 con 0x7f271c0ab730 2024-03-06T07:19:19.466 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 5 and data pool 6 2024-03-06T07:19:19.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f270006b4d0 msgr2=0x7f270006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f270006b4d0 0x7f270006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f271c0c3120 tx=0x7f2718008040 comp rx=0 tx=0).stop 2024-03-06T07:19:19.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0ab730 msgr2=0x7f271c1ea240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0ab730 0x7f271c1ea240 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f270c02e000 tx=0x7f270c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 shutdown_connections 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f270006b4d0 0x7f270006d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f271c071170 0x7f271c1ea780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f271c0ab730 0x7f271c1ea240 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 --2- 172.21.15.94:0/824947664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f271c0a8ff0 0x7f271c154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 >> 172.21.15.94:0/824947664 conn(0x7f271c06afc0 msgr2=0x7f271c14ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 shutdown_connections 2024-03-06T07:19:19.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.469+0000 7f272438a640 1 -- 172.21.15.94:0/824947664 wait complete. 2024-03-06T07:19:19.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:19.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.587+0000 7fd0da1d7640 1 Processor -- start 2024-03-06T07:19:19.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.587+0000 7fd0da1d7640 1 -- start start 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0da1d7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd0d414f790 0x7fd0d41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0da1d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 0x7fd0d4155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0da1d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d41563d0 con 0x7fd0d40c1aa0 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0da1d7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d40c2680 con 0x7fd0d414f790 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0da1d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40c28d0 con 0x7fd0d4155a40 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd0d40c1aa0 0x7fd0d40c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43590/0 (socket says 172.21.15.94:43590) 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d2ffd640 1 -- 172.21.15.94:0/2434539765 learned_addr learned my addr 172.21.15.94:0/2434539765 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d37fe640 1 --2- 172.21.15.94:0/2434539765 >> v2:172.21.15.145:3300/0 conn(0x7fd0d414f790 0x7fd0d41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174817246 0 0) 0x7fd0d41563d0 con 0x7fd0d40c1aa0 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0bc003520 con 0x7fd0d40c1aa0 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d3fff640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 0x7fd0d4155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3250127101 0 0) 0x7fd0bc003520 con 0x7fd0d40c1aa0 2024-03-06T07:19:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 msgr2=0x7fd0d4155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 0x7fd0d4155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 >> v2:172.21.15.145:3300/0 conn(0x7fd0d414f790 msgr2=0x7fd0d41529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 --2- 172.21.15.94:0/2434539765 >> v2:172.21.15.145:3300/0 conn(0x7fd0d414f790 0x7fd0d41529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40c2bb0 con 0x7fd0d40c1aa0 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fd0c8001670 con 0x7fd0d40c1aa0 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 >> v1:172.21.15.94:6789/0 conn(0x7fd0d40c1aa0 legacy=0x7fd0d40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0039b0 0x7fd0bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 0x7fd0bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40c2680 con 0x7fd0bc004800 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40c28d0 con 0x7fd0bc0039b0 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.588+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc00dbd0 con 0x7fd0d4155a40 2024-03-06T07:19:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d3fff640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 0x7fd0bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d37fe640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0039b0 0x7fd0bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d2ffd640 1 -- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 msgr2=0x7fd0bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d4155a40 0x7fd0bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d2ffd640 1 -- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0039b0 msgr2=0x7fd0bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0039b0 0x7fd0bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d2ffd640 1 -- 172.21.15.94:0/2434539765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc003520 con 0x7fd0bc004800 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.589+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0bc004c80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8001050 tx=0x7fd0bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d877a6b94cea8ca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8007d70 con 0x7fd0bc004800 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c8030070 con 0x7fd0bc004800 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0d27fc640 1 -- 172.21.15.94:0/2434539765 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c802cd40 con 0x7fd0bc004800 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0da1d7640 1 -- 172.21.15.94:0/2434539765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 msgr2=0x7fd0bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0da1d7640 1 --2- 172.21.15.94:0/2434539765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0bc004c80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8001050 tx=0x7fd0bc005830 comp rx=0 tx=0).stop 2024-03-06T07:19:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0d3fff640 1 -- 172.21.15.94:0/2434539765 reap_dead start 2024-03-06T07:19:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0da1d7640 1 -- 172.21.15.94:0/2434539765 shutdown_connections 2024-03-06T07:19:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0da1d7640 1 -- 172.21.15.94:0/2434539765 >> 172.21.15.94:0/2434539765 conn(0x7fd0d40c6710 msgr2=0x7fd0d40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0da1d7640 1 -- 172.21.15.94:0/2434539765 shutdown_connections 2024-03-06T07:19:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.590+0000 7fd0da1d7640 1 -- 172.21.15.94:0/2434539765 wait complete. 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.591+0000 7fd0da1d7640 1 Processor -- start 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.591+0000 7fd0da1d7640 1 -- start start 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0da1d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0d414bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0da1d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d41dd870 0x7fd0d41ddcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0da1d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d41522c0 0x7fd0d4152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0da1d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40c0d60 con 0x7fd0bc004800 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0da1d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4155520 con 0x7fd0d41522c0 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0da1d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d40c5e30 con 0x7fd0d41dd870 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0d414bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0d414bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50824/0 (socket says 172.21.15.94:50824) 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 -- 172.21.15.94:0/515843239 learned_addr learned my addr 172.21.15.94:0/515843239 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d41dd870 0x7fd0d41ddcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d3fff640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d41522c0 0x7fd0d4152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 -- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d41dd870 msgr2=0x7fd0d41ddcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d41dd870 0x7fd0d41ddcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 -- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d41522c0 msgr2=0x7fd0d4152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d41522c0 0x7fd0d4152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d37fe640 1 -- 172.21.15.94:0/515843239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d4153320 con 0x7fd0bc004800 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d41dd870 0x7fd0d41ddcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.592+0000 7fd0d3fff640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d41522c0 0x7fd0d4152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.593+0000 7fd0d37fe640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0d414bf40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4011910 tx=0x7fd0c4011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.593+0000 7fd0bbfff640 1 -- 172.21.15.94:0/515843239 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c400faf0 con 0x7fd0bc004800 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.593+0000 7fd0bbfff640 1 -- 172.21.15.94:0/515843239 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c400fc90 con 0x7fd0bc004800 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.593+0000 7fd0bbfff640 1 -- 172.21.15.94:0/515843239 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4014440 con 0x7fd0bc004800 2024-03-06T07:19:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.593+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d41db6c0 con 0x7fd0bc004800 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.593+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d41dbc80 con 0x7fd0bc004800 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.594+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d404f330 con 0x7fd0bc004800 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.594+0000 7fd0bbfff640 1 -- 172.21.15.94:0/515843239 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0c4012040 con 0x7fd0bc004800 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.595+0000 7fd0bbfff640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd0c006b4d0 0x7fd0c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.595+0000 7fd0bbfff640 1 -- 172.21.15.94:0/515843239 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd0c408a930 con 0x7fd0bc004800 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.595+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd0c006b4d0 0x7fd0c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.595+0000 7fd0d2ffd640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd0c006b4d0 0x7fd0c006d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8034000 tx=0x7fd0c8007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.599+0000 7fd0bbfff640 1 -- 172.21.15.94:0/515843239 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0c4019030 con 0x7fd0bc004800 2024-03-06T07:19:19.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.771+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0d40aab00 con 0x7fd0bc004800 2024-03-06T07:19:19.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.773+0000 7fd0bbfff640 1 -- 172.21.15.94:0/515843239 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16955 (secure 0 0 0) 0x7fd0c40596b0 con 0x7fd0bc004800 2024-03-06T07:19:19.775 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:19.775 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":38,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:18.800661+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:17.283570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:17.762319+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd0c006b4d0 msgr2=0x7fd0c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd0c006b4d0 0x7fd0c006d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8034000 tx=0x7fd0c8007000 comp rx=0 tx=0).stop 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 msgr2=0x7fd0d414bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0d414bf40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4011910 tx=0x7fd0c4011dd0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 shutdown_connections 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd0c006b4d0 0x7fd0c006d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0d41522c0 0x7fd0d4152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0d41dd870 0x7fd0d41ddcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 --2- 172.21.15.94:0/515843239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0bc004800 0x7fd0d414bf40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.775+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 >> 172.21.15.94:0/515843239 conn(0x7fd0d40c6710 msgr2=0x7fd0d414e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.776+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 shutdown_connections 2024-03-06T07:19:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.776+0000 7fd0da1d7640 1 -- 172.21.15.94:0/515843239 wait complete. 2024-03-06T07:19:19.788 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:19.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.886+0000 7f43ca77a640 1 Processor -- start 2024-03-06T07:19:19.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43ca77a640 1 -- start start 2024-03-06T07:19:19.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43ca77a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43ca77a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43c40c88c0 0x7f43c40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43ca77a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f43c40c5880 0x7f43c4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43ca77a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c40c8dd0 con 0x7f43c40c88c0 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43ca77a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c40c73d0 con 0x7f43c40c5880 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43ca77a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40be2e0 con 0x7f43c414f360 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43c37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43c40c88c0 0x7f43c40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43c37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43c40c88c0 0x7f43c40c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50830/0 (socket says 172.21.15.94:50830) 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.887+0000 7f43c37fe640 1 -- 172.21.15.94:0/1005421438 learned_addr learned my addr 172.21.15.94:0/1005421438 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:19.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c8d17640 1 --2- 172.21.15.94:0/1005421438 >> v2:172.21.15.145:3300/0 conn(0x7f43c40c5880 0x7f43c4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c3fff640 1 --2- 172.21.15.94:0/1005421438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c37fe640 1 -- 172.21.15.94:0/1005421438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 msgr2=0x7f43c4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c37fe640 1 --2- 172.21.15.94:0/1005421438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c37fe640 1 -- 172.21.15.94:0/1005421438 >> v2:172.21.15.145:3300/0 conn(0x7f43c40c5880 msgr2=0x7f43c4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c37fe640 1 --2- 172.21.15.94:0/1005421438 >> v2:172.21.15.145:3300/0 conn(0x7f43c40c5880 0x7f43c4158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c37fe640 1 -- 172.21.15.94:0/1005421438 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c40c7670 con 0x7f43c40c88c0 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.888+0000 7f43c37fe640 1 --2- 172.21.15.94:0/1005421438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c40c5340 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f43c408cc10 tx=0x7f43b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca7cd7226a359a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43c2ffd640 1 -- 172.21.15.94:0/1005421438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b802e070 con 0x7f43c40c88c0 2024-03-06T07:19:19.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43c2ffd640 1 -- 172.21.15.94:0/1005421438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b8037070 con 0x7f43c40c88c0 2024-03-06T07:19:19.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43c2ffd640 1 -- 172.21.15.94:0/1005421438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8031a00 con 0x7f43c40c88c0 2024-03-06T07:19:19.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 -- 172.21.15.94:0/1005421438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 msgr2=0x7f43c40c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 --2- 172.21.15.94:0/1005421438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c40c5340 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f43c408cc10 tx=0x7f43b802b880 comp rx=0 tx=0).stop 2024-03-06T07:19:19.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 -- 172.21.15.94:0/1005421438 shutdown_connections 2024-03-06T07:19:19.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 --2- 172.21.15.94:0/1005421438 >> v2:172.21.15.145:3300/0 conn(0x7f43c40c5880 0x7f43c4158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 --2- 172.21.15.94:0/1005421438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c40c5340 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 --2- 172.21.15.94:0/1005421438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 -- 172.21.15.94:0/1005421438 >> 172.21.15.94:0/1005421438 conn(0x7f43c40ac570 msgr2=0x7f43c40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 -- 172.21.15.94:0/1005421438 shutdown_connections 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.889+0000 7f43ca77a640 1 -- 172.21.15.94:0/1005421438 wait complete. 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.890+0000 7f43ca77a640 1 Processor -- start 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.890+0000 7f43ca77a640 1 -- start start 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43ca77a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43c40c5880 0x7f43c41eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43ca77a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c4152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43ca77a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c41ebf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43ca77a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c41590a0 con 0x7f43c40c88c0 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43ca77a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c40c7c30 con 0x7f43c40c5880 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43ca77a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c4159220 con 0x7f43c414f360 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c4152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c4152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50846/0 (socket says 172.21.15.94:50846) 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 -- 172.21.15.94:0/2722373170 learned_addr learned my addr 172.21.15.94:0/2722373170 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:19.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c3fff640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43c40c5880 0x7f43c41eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c8d17640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c41ebf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 -- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 msgr2=0x7f43c41ebf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c41ebf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 -- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43c40c5880 msgr2=0x7f43c41eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43c40c5880 0x7f43c41eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.891+0000 7f43c37fe640 1 -- 172.21.15.94:0/2722373170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c414fbd0 con 0x7f43c40c88c0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.892+0000 7f43c37fe640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c4152160 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f43b8000c00 tx=0x7f43b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.892+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2722373170 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8032670 con 0x7f43c40c88c0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.892+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2722373170 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b8032810 con 0x7f43c40c88c0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.892+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c414feb0 con 0x7f43c40c88c0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.892+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2722373170 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8032c00 con 0x7f43c40c88c0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.892+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c4150750 con 0x7f43c40c88c0 2024-03-06T07:19:19.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.893+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c4150080 con 0x7f43c40c88c0 2024-03-06T07:19:19.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.893+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2722373170 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43b804c060 con 0x7f43c40c88c0 2024-03-06T07:19:19.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.894+0000 7f43c0ff9640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43b006b4d0 0x7f43b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:19.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.894+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2722373170 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43b80abbd0 con 0x7f43c40c88c0 2024-03-06T07:19:19.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.894+0000 7f43c3fff640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43b006b4d0 0x7f43b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:19.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.894+0000 7f43c3fff640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43b006b4d0 0x7f43b006d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f43b4009a30 tx=0x7f43b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:19.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:19.898+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2722373170 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43b8040070 con 0x7f43c40c88c0 2024-03-06T07:19:20.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.070+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43c404f330 con 0x7f43c40c88c0 2024-03-06T07:19:20.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.071+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2722373170 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16955 (secure 0 0 0) 0x7f43b807aa10 con 0x7f43c40c88c0 2024-03-06T07:19:20.073 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:20.073 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":38,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:18.800661+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:17.283570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:17.762319+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43b006b4d0 msgr2=0x7f43b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43b006b4d0 0x7f43b006d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f43b4009a30 tx=0x7f43b4008040 comp rx=0 tx=0).stop 2024-03-06T07:19:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 msgr2=0x7f43c4152160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c4152160 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f43b8000c00 tx=0x7f43b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 shutdown_connections 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43b006b4d0 0x7f43b006d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43c414f360 0x7f43c41ebf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43c40c88c0 0x7f43c4152160 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 --2- 172.21.15.94:0/2722373170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43c40c5880 0x7f43c41eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 >> 172.21.15.94:0/2722373170 conn(0x7f43c40ac570 msgr2=0x7f43c40c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 shutdown_connections 2024-03-06T07:19:20.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.073+0000 7f43ca77a640 1 -- 172.21.15.94:0/2722373170 wait complete. 2024-03-06T07:19:20.086 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:19:20.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.187+0000 7fd57a7e9640 1 Processor -- start 2024-03-06T07:19:20.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.187+0000 7fd57a7e9640 1 -- start start 2024-03-06T07:19:20.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd57a7e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd57406afc0 0x7fd5740ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd57a7e9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd5740abe90 0x7fd57415d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd57a7e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57406b440 con 0x7fd57414d350 2024-03-06T07:19:20.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd57a7e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740ac750 con 0x7fd57406afc0 2024-03-06T07:19:20.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd57a7e9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740c1a40 con 0x7fd5740abe90 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd573fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd57414d350 0x7fd5740acbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43592/0 (socket says 172.21.15.94:43592) 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd573fff640 1 -- 172.21.15.94:0/472092074 learned_addr learned my addr 172.21.15.94:0/472092074 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd5737fe640 1 --2- 172.21.15.94:0/472092074 >> v2:172.21.15.94:3300/0 conn(0x7fd57406afc0 0x7fd5740ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd578d86640 1 --2- 172.21.15.94:0/472092074 >> v2:172.21.15.145:3300/0 conn(0x7fd5740abe90 0x7fd57415d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd5737fe640 1 -- 172.21.15.94:0/472092074 >> v2:172.21.15.145:3300/0 conn(0x7fd5740abe90 msgr2=0x7fd57415d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd5737fe640 1 --2- 172.21.15.94:0/472092074 >> v2:172.21.15.145:3300/0 conn(0x7fd5740abe90 0x7fd57415d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd5737fe640 1 -- 172.21.15.94:0/472092074 >> v1:172.21.15.94:6789/0 conn(0x7fd57414d350 legacy=0x7fd5740acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd572ffd640 1 -- 172.21.15.94:0/472092074 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477830819 0 0) 0x7fd564001110 con 0x7fd57414d350 2024-03-06T07:19:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd5737fe640 1 -- 172.21.15.94:0/472092074 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740c1c90 con 0x7fd57406afc0 2024-03-06T07:19:20.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.188+0000 7fd578d86640 1 --2- 172.21.15.94:0/472092074 >> v2:172.21.15.145:3300/0 conn(0x7fd5740abe90 0x7fd57415d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:20.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.189+0000 7fd5737fe640 1 --2- 172.21.15.94:0/472092074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd57406afc0 0x7fd5740ab950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd5740aa520 tx=0x7fd56802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffe06951cce8550 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:20.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.189+0000 7fd572ffd640 1 -- 172.21.15.94:0/472092074 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5680315a0 con 0x7fd57406afc0 2024-03-06T07:19:20.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.189+0000 7fd572ffd640 1 -- 172.21.15.94:0/472092074 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd568037070 con 0x7fd57406afc0 2024-03-06T07:19:20.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.189+0000 7fd572ffd640 1 -- 172.21.15.94:0/472092074 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568031960 con 0x7fd57406afc0 2024-03-06T07:19:20.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 -- 172.21.15.94:0/472092074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd57406afc0 msgr2=0x7fd5740ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/472092074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd57406afc0 0x7fd5740ab950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd5740aa520 tx=0x7fd56802b840 comp rx=0 tx=0).stop 2024-03-06T07:19:20.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 -- 172.21.15.94:0/472092074 shutdown_connections 2024-03-06T07:19:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/472092074 >> v2:172.21.15.145:3300/0 conn(0x7fd5740abe90 0x7fd57415d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/472092074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd57406afc0 0x7fd5740ab950 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 -- 172.21.15.94:0/472092074 >> 172.21.15.94:0/472092074 conn(0x7fd5740c3b40 msgr2=0x7fd5740c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 -- 172.21.15.94:0/472092074 shutdown_connections 2024-03-06T07:19:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.190+0000 7fd57a7e9640 1 -- 172.21.15.94:0/472092074 wait complete. 2024-03-06T07:19:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.191+0000 7fd57a7e9640 1 Processor -- start 2024-03-06T07:19:20.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.191+0000 7fd57a7e9640 1 -- start start 2024-03-06T07:19:20.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd57a7e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd57406afc0 0x7fd574155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd57a7e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5740abe90 0x7fd5740c13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd57a7e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57414d350 0x7fd5741ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd57a7e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57406ea70 con 0x7fd5740abe90 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd57a7e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd574153c00 con 0x7fd57414d350 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd57a7e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd574155860 con 0x7fd57406afc0 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5740abe90 0x7fd5740c13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5740abe90 0x7fd5740c13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50862/0 (socket says 172.21.15.94:50862) 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 -- 172.21.15.94:0/888361379 learned_addr learned my addr 172.21.15.94:0/888361379 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd578d86640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57414d350 0x7fd5741ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd573fff640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd57406afc0 0x7fd574155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 -- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd57406afc0 msgr2=0x7fd574155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd57406afc0 0x7fd574155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 -- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57414d350 msgr2=0x7fd5741ec4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57414d350 0x7fd5741ec4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 -- 172.21.15.94:0/888361379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd574156910 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd573fff640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd57406afc0 0x7fd574155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.192+0000 7fd5737fe640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5740abe90 0x7fd5740c13e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd568009890 tx=0x7fd5680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.193+0000 7fd570ff9640 1 -- 172.21.15.94:0/888361379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568032840 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.193+0000 7fd570ff9640 1 -- 172.21.15.94:0/888361379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5680329e0 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.193+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd574155270 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.193+0000 7fd570ff9640 1 -- 172.21.15.94:0/888361379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568032dd0 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.193+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5741ecfa0 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.193+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57404f330 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.194+0000 7fd570ff9640 1 -- 172.21.15.94:0/888361379 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd56804c020 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.194+0000 7fd570ff9640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd56006b4d0 0x7fd56006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.194+0000 7fd570ff9640 1 -- 172.21.15.94:0/888361379 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd56802f020 con 0x7fd5740abe90 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.195+0000 7fd573fff640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd56006b4d0 0x7fd56006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.195+0000 7fd573fff640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd56006b4d0 0x7fd56006d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd564000f60 tx=0x7fd564009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:20.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.199+0000 7fd570ff9640 1 -- 172.21.15.94:0/888361379 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd574155270 con 0x7fd5740abe90 2024-03-06T07:19:20.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.404+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd5740c49e0 con 0x7fd5740abe90 2024-03-06T07:19:20.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.472+0000 7fd570ff9640 1 -- 172.21.15.94:0/888361379 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd574156910 con 0x7fd5740abe90 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd56006b4d0 msgr2=0x7fd56006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd56006b4d0 0x7fd56006d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd564000f60 tx=0x7fd564009040 comp rx=0 tx=0).stop 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5740abe90 msgr2=0x7fd5740c13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5740abe90 0x7fd5740c13e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd568009890 tx=0x7fd5680046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 shutdown_connections 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd56006b4d0 0x7fd56006d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57414d350 0x7fd5741ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5740abe90 0x7fd5740c13e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 --2- 172.21.15.94:0/888361379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd57406afc0 0x7fd574155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 >> 172.21.15.94:0/888361379 conn(0x7fd5740c3b40 msgr2=0x7fd5740c3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:20.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 shutdown_connections 2024-03-06T07:19:20.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.475+0000 7fd57a7e9640 1 -- 172.21.15.94:0/888361379 wait complete. 2024-03-06T07:19:20.487 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:19:20.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:19:20.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.589+0000 7fd49d965640 1 Processor -- start 2024-03-06T07:19:20.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.589+0000 7fd49d965640 1 -- start start 2024-03-06T07:19:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd49d965640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd4980c5880 0x7fd4980c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd49d965640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd498072600 con 0x7fd4980c5c60 2024-03-06T07:19:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd49d965640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd498153af0 con 0x7fd498073190 2024-03-06T07:19:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd49d965640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4980723b0 con 0x7fd4980c5880 2024-03-06T07:19:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd4967fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd4980c5c60 0x7fd4980c71e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43608/0 (socket says 172.21.15.94:43608) 2024-03-06T07:19:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd4967fc640 1 -- 172.21.15.94:0/986387261 learned_addr learned my addr 172.21.15.94:0/986387261 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:20.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd496ffd640 1 --2- 172.21.15.94:0/986387261 >> v2:172.21.15.145:3300/0 conn(0x7fd4980c5880 0x7fd4980c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd495ffb640 1 -- 172.21.15.94:0/986387261 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611593394 0 0) 0x7fd498072600 con 0x7fd4980c5c60 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd495ffb640 1 -- 172.21.15.94:0/986387261 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd480003520 con 0x7fd4980c5c60 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd496ffd640 1 -- 172.21.15.94:0/986387261 >> v1:172.21.15.145:6789/0 conn(0x7fd498073190 legacy=0x7fd498156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd496ffd640 1 -- 172.21.15.94:0/986387261 >> v1:172.21.15.94:6789/0 conn(0x7fd4980c5c60 legacy=0x7fd4980c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.590+0000 7fd496ffd640 1 -- 172.21.15.94:0/986387261 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd498153dd0 con 0x7fd4980c5880 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.591+0000 7fd496ffd640 1 --2- 172.21.15.94:0/986387261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c6ca0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd488009d90 tx=0x7fd48802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1707e8d3b46e228f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.591+0000 7fd495ffb640 1 -- 172.21.15.94:0/986387261 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488039070 con 0x7fd4980c5880 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.591+0000 7fd495ffb640 1 -- 172.21.15.94:0/986387261 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48802cbd0 con 0x7fd4980c5880 2024-03-06T07:19:20.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.591+0000 7fd495ffb640 1 -- 172.21.15.94:0/986387261 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488034440 con 0x7fd4980c5880 2024-03-06T07:19:20.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.592+0000 7fd49d965640 1 -- 172.21.15.94:0/986387261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 msgr2=0x7fd4980c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.592+0000 7fd49d965640 1 --2- 172.21.15.94:0/986387261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c6ca0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd488009d90 tx=0x7fd48802be20 comp rx=0 tx=0).stop 2024-03-06T07:19:20.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.592+0000 7fd49d965640 1 -- 172.21.15.94:0/986387261 shutdown_connections 2024-03-06T07:19:20.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.592+0000 7fd49d965640 1 --2- 172.21.15.94:0/986387261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c6ca0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.592+0000 7fd49d965640 1 -- 172.21.15.94:0/986387261 >> 172.21.15.94:0/986387261 conn(0x7fd4980aa630 msgr2=0x7fd4980aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:20.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.592+0000 7fd49d965640 1 -- 172.21.15.94:0/986387261 shutdown_connections 2024-03-06T07:19:20.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.592+0000 7fd49d965640 1 -- 172.21.15.94:0/986387261 wait complete. 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.593+0000 7fd49d965640 1 Processor -- start 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.593+0000 7fd49d965640 1 -- start start 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd49d965640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd498073190 0x7fd49806ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd49d965640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd49d965640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c5c60 0x7fd498154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd49d965640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980c78f0 con 0x7fd498073190 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd49d965640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980728e0 con 0x7fd4980c5880 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd49d965640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd498156c40 con 0x7fd4980c5c60 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd498073190 0x7fd49806ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd498073190 0x7fd49806ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50874/0 (socket says 172.21.15.94:50874) 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 -- 172.21.15.94:0/3914651168 learned_addr learned my addr 172.21.15.94:0/3914651168 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd4977fe640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c5c60 0x7fd498154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd4967fc640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 -- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c5c60 msgr2=0x7fd498154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c5c60 0x7fd498154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 -- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 msgr2=0x7fd4980c1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 -- 172.21.15.94:0/3914651168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49806e010 con 0x7fd498073190 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd4977fe640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c5c60 0x7fd498154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd4967fc640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.594+0000 7fd496ffd640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd498073190 0x7fd49806ceb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd488009d90 tx=0x7fd488004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.595+0000 7fd47ffff640 1 -- 172.21.15.94:0/3914651168 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488039050 con 0x7fd498073190 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.595+0000 7fd47ffff640 1 -- 172.21.15.94:0/3914651168 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48803a040 con 0x7fd498073190 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.595+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49814bb50 con 0x7fd498073190 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.595+0000 7fd47ffff640 1 -- 172.21.15.94:0/3914651168 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48803e610 con 0x7fd498073190 2024-03-06T07:19:20.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.595+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49806dd30 con 0x7fd498073190 2024-03-06T07:19:20.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.596+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49804f330 con 0x7fd498073190 2024-03-06T07:19:20.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.596+0000 7fd47ffff640 1 -- 172.21.15.94:0/3914651168 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd488047020 con 0x7fd498073190 2024-03-06T07:19:20.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.596+0000 7fd47ffff640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd48406b4d0 0x7fd48406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.597+0000 7fd47ffff640 1 -- 172.21.15.94:0/3914651168 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4880ac680 con 0x7fd498073190 2024-03-06T07:19:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.597+0000 7fd4967fc640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd48406b4d0 0x7fd48406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:20.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.597+0000 7fd4967fc640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd48406b4d0 0x7fd48406d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd498156b00 tx=0x7fd48c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:20.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.601+0000 7fd47ffff640 1 -- 172.21.15.94:0/3914651168 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd48804e020 con 0x7fd498073190 2024-03-06T07:19:20.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:20.808+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd4980ab350 con 0x7fd498073190 2024-03-06T07:19:21.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.478+0000 7fd47ffff640 1 -- 172.21.15.94:0/3914651168 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd488032e20 con 0x7fd498073190 2024-03-06T07:19:21.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd48406b4d0 msgr2=0x7fd48406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd48406b4d0 0x7fd48406d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd498156b00 tx=0x7fd48c009040 comp rx=0 tx=0).stop 2024-03-06T07:19:21.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd498073190 msgr2=0x7fd49806ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd498073190 0x7fd49806ceb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd488009d90 tx=0x7fd488004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 shutdown_connections 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd48406b4d0 0x7fd48406d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4980c5c60 0x7fd498154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4980c5880 0x7fd4980c1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 --2- 172.21.15.94:0/3914651168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd498073190 0x7fd49806ceb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 >> 172.21.15.94:0/3914651168 conn(0x7fd4980aa630 msgr2=0x7fd4980a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 shutdown_connections 2024-03-06T07:19:21.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.480+0000 7fd49d965640 1 -- 172.21.15.94:0/3914651168 wait complete. 2024-03-06T07:19:21.493 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:19:21.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:21.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.592+0000 7f51fde64640 1 Processor -- start 2024-03-06T07:19:21.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.592+0000 7f51fde64640 1 -- start start 2024-03-06T07:19:21.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51fde64640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51f8159270 0x7f51f8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51fde64640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f80c73d0 con 0x7f51f80c88c0 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51fde64640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f80be2e0 con 0x7f51f814f360 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51fde64640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f8159700 con 0x7f51f8159270 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51f7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51f8159270 0x7f51f8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51f7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51f8159270 0x7f51f8155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50884/0 (socket says 172.21.15.94:50884) 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51f7fff640 1 -- 172.21.15.94:0/2426100670 learned_addr learned my addr 172.21.15.94:0/2426100670 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.593+0000 7f51eeffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f51f80c88c0 0x7f51f8158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43616/0 (socket says 172.21.15.94:43616) 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f7fff640 1 -- 172.21.15.94:0/2426100670 >> v1:172.21.15.145:6789/0 conn(0x7f51f814f360 legacy=0x7f51f80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f7fff640 1 -- 172.21.15.94:0/2426100670 >> v1:172.21.15.94:6789/0 conn(0x7f51f80c88c0 legacy=0x7f51f8158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f6ffd640 1 -- 172.21.15.94:0/2426100670 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056275664 0 0) 0x7f51f80be2e0 con 0x7f51f814f360 2024-03-06T07:19:21.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f7fff640 1 -- 172.21.15.94:0/2426100670 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f80c7670 con 0x7f51f8159270 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f7fff640 1 --2- 172.21.15.94:0/2426100670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f8159270 0x7f51f8155c10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009e50 tx=0x7f51f002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd2a551e98033ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f6ffd640 1 -- 172.21.15.94:0/2426100670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f002c8d0 con 0x7f51f8159270 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f6ffd640 1 -- 172.21.15.94:0/2426100670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f002ca70 con 0x7f51f8159270 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.594+0000 7f51f6ffd640 1 -- 172.21.15.94:0/2426100670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f002ce40 con 0x7f51f8159270 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.595+0000 7f51fde64640 1 -- 172.21.15.94:0/2426100670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f8159270 msgr2=0x7f51f8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.595+0000 7f51fde64640 1 --2- 172.21.15.94:0/2426100670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f8159270 0x7f51f8155c10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009e50 tx=0x7f51f002bdb0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.595+0000 7f51fde64640 1 -- 172.21.15.94:0/2426100670 shutdown_connections 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.595+0000 7f51fde64640 1 --2- 172.21.15.94:0/2426100670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f8159270 0x7f51f8155c10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.595+0000 7f51fde64640 1 -- 172.21.15.94:0/2426100670 >> 172.21.15.94:0/2426100670 conn(0x7f51f80ac570 msgr2=0x7f51f80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:21.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.595+0000 7f51fde64640 1 -- 172.21.15.94:0/2426100670 shutdown_connections 2024-03-06T07:19:21.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.595+0000 7f51fde64640 1 -- 172.21.15.94:0/2426100670 wait complete. 2024-03-06T07:19:21.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.596+0000 7f51fde64640 1 Processor -- start 2024-03-06T07:19:21.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.596+0000 7f51fde64640 1 -- start start 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51fde64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f80c88c0 0x7f51f80add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51fde64640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f814f360 0x7f51f8157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51fde64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51f8159270 0x7f51f814acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51fde64640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80c7c30 con 0x7f51f80c88c0 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51fde64640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80c5a50 con 0x7f51f814f360 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51fde64640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80c5bd0 con 0x7f51f8159270 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f80c88c0 0x7f51f80add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f80c88c0 0x7f51f80add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50900/0 (socket says 172.21.15.94:50900) 2024-03-06T07:19:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 -- 172.21.15.94:0/4126873754 learned_addr learned my addr 172.21.15.94:0/4126873754 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51eeffd640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f814f360 0x7f51f8157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f7fff640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51f8159270 0x7f51f814acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 -- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51f8159270 msgr2=0x7f51f814acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51f8159270 0x7f51f814acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 -- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f814f360 msgr2=0x7f51f8157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f814f360 0x7f51f8157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 -- 172.21.15.94:0/4126873754 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f81ea820 con 0x7f51f80c88c0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f7fff640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51f8159270 0x7f51f814acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.597+0000 7f51f77fe640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f80c88c0 0x7f51f80add30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f51e800c440 tx=0x7f51e800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.598+0000 7f51f4ff9640 1 -- 172.21.15.94:0/4126873754 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e8017070 con 0x7f51f80c88c0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.598+0000 7f51f4ff9640 1 -- 172.21.15.94:0/4126873754 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e8005480 con 0x7f51f80c88c0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.598+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f81eaac0 con 0x7f51f80c88c0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.598+0000 7f51f4ff9640 1 -- 172.21.15.94:0/4126873754 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e8003830 con 0x7f51f80c88c0 2024-03-06T07:19:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.598+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f81eb080 con 0x7f51f80c88c0 2024-03-06T07:19:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.599+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f804f330 con 0x7f51f80c88c0 2024-03-06T07:19:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.599+0000 7f51f4ff9640 1 -- 172.21.15.94:0/4126873754 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51e801a070 con 0x7f51f80c88c0 2024-03-06T07:19:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.599+0000 7f51f4ff9640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51d806b4d0 0x7f51d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.599+0000 7f51f4ff9640 1 -- 172.21.15.94:0/4126873754 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f51e80895c0 con 0x7f51f80c88c0 2024-03-06T07:19:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.600+0000 7f51eeffd640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51d806b4d0 0x7f51d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.600+0000 7f51eeffd640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51d806b4d0 0x7f51d806d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51f80c4fe0 tx=0x7f51e0007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:21.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.604+0000 7f51f4ff9640 1 -- 172.21.15.94:0/4126873754 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51e8057c60 con 0x7f51f80c88c0 2024-03-06T07:19:21.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.813+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f80c8d50 con 0x7f51f80c88c0 2024-03-06T07:19:21.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.813+0000 7f51f4ff9640 1 -- 172.21.15.94:0/4126873754 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f51e805bd00 con 0x7f51f80c88c0 2024-03-06T07:19:21.815 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:21.815 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:20.809451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34820":{"gid":34820,"name":"b","rank":0,"incarnation":15,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/1305657307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1305657307},{"type":"v1","addr":"172.21.15.145:6837","nonce":1305657307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34820,"qdb_cluster":[34820]},"id":2}]} 2024-03-06T07:19:21.816 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 17 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51d806b4d0 msgr2=0x7f51d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51d806b4d0 0x7f51d806d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51f80c4fe0 tx=0x7f51e0007e20 comp rx=0 tx=0).stop 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f80c88c0 msgr2=0x7f51f80add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f80c88c0 0x7f51f80add30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f51e800c440 tx=0x7f51e800c900 comp rx=0 tx=0).stop 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 shutdown_connections 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51d806b4d0 0x7f51d806d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51f8159270 0x7f51f814acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51f814f360 0x7f51f8157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 --2- 172.21.15.94:0/4126873754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51f80c88c0 0x7f51f80add30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 >> 172.21.15.94:0/4126873754 conn(0x7f51f80ac570 msgr2=0x7f51f80c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:21.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 shutdown_connections 2024-03-06T07:19:21.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.815+0000 7f51fde64640 1 -- 172.21.15.94:0/4126873754 wait complete. 2024-03-06T07:19:21.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:21.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.926+0000 7f82fdcaa640 1 Processor -- start 2024-03-06T07:19:21.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.926+0000 7f82fdcaa640 1 -- start start 2024-03-06T07:19:21.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.926+0000 7f82fdcaa640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f82f8150580 0x7f82f806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.926+0000 7f82fdcaa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f80c6000 0x7f82f806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.926+0000 7f82fdcaa640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f8153a60 con 0x7f82f80c1320 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.926+0000 7f82fdcaa640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f8153d40 con 0x7f82f8150580 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.926+0000 7f82fdcaa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f806c2c0 con 0x7f82f80c6000 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f82f80c1320 0x7f82f81566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43630/0 (socket says 172.21.15.94:43630) 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f6ffd640 1 -- 172.21.15.94:0/2242879737 learned_addr learned my addr 172.21.15.94:0/2242879737 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f7fff640 1 --2- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f80c6000 0x7f82f806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f77fe640 1 --2- 172.21.15.94:0/2242879737 >> v2:172.21.15.145:3300/0 conn(0x7f82f8150580 0x7f82f806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f67fc640 1 -- 172.21.15.94:0/2242879737 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660469158 0 0) 0x7f82f8153a60 con 0x7f82f80c1320 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f67fc640 1 -- 172.21.15.94:0/2242879737 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e0003520 con 0x7f82f80c1320 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f77fe640 1 -- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f80c6000 msgr2=0x7f82f806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:21.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f77fe640 1 --2- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f80c6000 0x7f82f806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f77fe640 1 -- 172.21.15.94:0/2242879737 >> v1:172.21.15.94:6789/0 conn(0x7f82f80c1320 legacy=0x7f82f81566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f77fe640 1 -- 172.21.15.94:0/2242879737 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f806c5a0 con 0x7f82f8150580 2024-03-06T07:19:21.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.927+0000 7f82f7fff640 1 --2- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f80c6000 0x7f82f806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:21.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82f77fe640 1 --2- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f8150580 0x7f82f806dd10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82e80097a0 tx=0x7f82e802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b8d667f3254f866 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:21.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82f67fc640 1 -- 172.21.15.94:0/2242879737 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e80315a0 con 0x7f82f8150580 2024-03-06T07:19:21.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82f67fc640 1 -- 172.21.15.94:0/2242879737 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8037070 con 0x7f82f8150580 2024-03-06T07:19:21.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82f67fc640 1 -- 172.21.15.94:0/2242879737 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8031960 con 0x7f82f8150580 2024-03-06T07:19:21.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 -- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f8150580 msgr2=0x7f82f806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f8150580 0x7f82f806dd10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82e80097a0 tx=0x7f82e802b840 comp rx=0 tx=0).stop 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 -- 172.21.15.94:0/2242879737 shutdown_connections 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f80c6000 0x7f82f806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/2242879737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f8150580 0x7f82f806dd10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 -- 172.21.15.94:0/2242879737 >> 172.21.15.94:0/2242879737 conn(0x7f82f80c6710 msgr2=0x7f82f80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 -- 172.21.15.94:0/2242879737 shutdown_connections 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.928+0000 7f82fdcaa640 1 -- 172.21.15.94:0/2242879737 wait complete. 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.929+0000 7f82fdcaa640 1 Processor -- start 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82fdcaa640 1 -- start start 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82fdcaa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f80c1320 0x7f82f80a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82fdcaa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f80c6000 0x7f82f8155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82fdcaa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f8150580 0x7f82f8150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82fdcaa640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f80c0dc0 con 0x7f82f80c6000 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82fdcaa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f80c0c40 con 0x7f82f80c1320 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82fdcaa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8156dc0 con 0x7f82f8150580 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f80c6000 0x7f82f8155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82f6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f80c6000 0x7f82f8155b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50912/0 (socket says 172.21.15.94:50912) 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.930+0000 7f82f6ffd640 1 -- 172.21.15.94:0/19193881 learned_addr learned my addr 172.21.15.94:0/19193881 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f77fe640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f80c1320 0x7f82f80a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f7fff640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f8150580 0x7f82f8150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f6ffd640 1 -- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f8150580 msgr2=0x7f82f8150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f6ffd640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f8150580 0x7f82f8150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f6ffd640 1 -- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f80c1320 msgr2=0x7f82f80a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f6ffd640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f80c1320 0x7f82f80a8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f6ffd640 1 -- 172.21.15.94:0/19193881 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f8153180 con 0x7f82f80c6000 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82f6ffd640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f80c6000 0x7f82f8155b30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00c430 tx=0x7f82ec00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82dffff640 1 -- 172.21.15.94:0/19193881 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec017070 con 0x7f82f80c6000 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82dffff640 1 -- 172.21.15.94:0/19193881 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec005ce0 con 0x7f82f80c6000 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f814b720 con 0x7f82f80c6000 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82dffff640 1 -- 172.21.15.94:0/19193881 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0072f0 con 0x7f82f80c6000 2024-03-06T07:19:21.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.931+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f81e8760 con 0x7f82f80c6000 2024-03-06T07:19:21.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.932+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f804f330 con 0x7f82f80c6000 2024-03-06T07:19:21.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.932+0000 7f82dffff640 1 -- 172.21.15.94:0/19193881 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82ec01a070 con 0x7f82f80c6000 2024-03-06T07:19:21.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.933+0000 7f82dffff640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82e406b4d0 0x7f82e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:21.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.933+0000 7f82dffff640 1 -- 172.21.15.94:0/19193881 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f82ec089640 con 0x7f82f80c6000 2024-03-06T07:19:21.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.933+0000 7f82f77fe640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82e406b4d0 0x7f82e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:21.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.933+0000 7f82f77fe640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82e406b4d0 0x7f82e406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f82e8036000 tx=0x7f82e803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:21.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:21.937+0000 7f82dffff640 1 -- 172.21.15.94:0/19193881 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82ec057d70 con 0x7f82f80c6000 2024-03-06T07:19:22.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.110+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f8068450 con 0x7f82f80c6000 2024-03-06T07:19:22.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.112+0000 7f82dffff640 1 -- 172.21.15.94:0/19193881 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16942 (secure 0 0 0) 0x7f82ec05e0c0 con 0x7f82f80c6000 2024-03-06T07:19:22.114 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:22.114 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":39,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:20.452388+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:17.283570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:17.762319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:22.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82e406b4d0 msgr2=0x7f82e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82e406b4d0 0x7f82e406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f82e8036000 tx=0x7f82e803d040 comp rx=0 tx=0).stop 2024-03-06T07:19:22.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f80c6000 msgr2=0x7f82f8155b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f80c6000 0x7f82f8155b30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00c430 tx=0x7f82ec00c830 comp rx=0 tx=0).stop 2024-03-06T07:19:22.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 shutdown_connections 2024-03-06T07:19:22.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82e406b4d0 0x7f82e406d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f8150580 0x7f82f8150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f80c6000 0x7f82f8155b30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 --2- 172.21.15.94:0/19193881 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f80c1320 0x7f82f80a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 >> 172.21.15.94:0/19193881 conn(0x7f82f80c6710 msgr2=0x7f82f814eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:22.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 shutdown_connections 2024-03-06T07:19:22.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.114+0000 7f82fdcaa640 1 -- 172.21.15.94:0/19193881 wait complete. 2024-03-06T07:19:22.127 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:19:22.127 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:19:22.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.225+0000 7ff0a3708640 1 Processor -- start 2024-03-06T07:19:22.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.225+0000 7ff0a3708640 1 -- start start 2024-03-06T07:19:22.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a3708640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff09c14f790 0x7ff09c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a3708640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c0c1aa0 0x7ff09c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a3708640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c0c28d0 con 0x7ff09c0c5c60 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a3708640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09c0c2680 con 0x7ff09c14f790 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a3708640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0c2bb0 con 0x7ff09c0c1aa0 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a14a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff09c14f790 0x7ff09c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a14a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff09c14f790 0x7ff09c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50920/0 (socket says 172.21.15.94:50920) 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a14a4640 1 -- 172.21.15.94:0/666106792 learned_addr learned my addr 172.21.15.94:0/666106792 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a0ca3640 1 --2- 172.21.15.94:0/666106792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c0c1aa0 0x7ff09c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a14a4640 1 -- 172.21.15.94:0/666106792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c0c1aa0 msgr2=0x7ff09c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a14a4640 1 --2- 172.21.15.94:0/666106792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c0c1aa0 0x7ff09c0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a14a4640 1 -- 172.21.15.94:0/666106792 >> v1:172.21.15.145:6789/0 conn(0x7ff09c0c5c60 legacy=0x7ff09c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.226+0000 7ff0a14a4640 1 -- 172.21.15.94:0/666106792 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c0c2e90 con 0x7ff09c14f790 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.227+0000 7ff0a14a4640 1 --2- 172.21.15.94:0/666106792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c14f790 0x7ff09c1529d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff098009870 tx=0x7ff09802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4659af523aa4f276 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.227+0000 7ff093fff640 1 -- 172.21.15.94:0/666106792 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098034070 con 0x7ff09c14f790 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.227+0000 7ff093fff640 1 -- 172.21.15.94:0/666106792 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff098030840 con 0x7ff09c14f790 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.227+0000 7ff093fff640 1 -- 172.21.15.94:0/666106792 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098030c10 con 0x7ff09c14f790 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 -- 172.21.15.94:0/666106792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c14f790 msgr2=0x7ff09c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 --2- 172.21.15.94:0/666106792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c14f790 0x7ff09c1529d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff098009870 tx=0x7ff09802b980 comp rx=0 tx=0).stop 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 -- 172.21.15.94:0/666106792 shutdown_connections 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 --2- 172.21.15.94:0/666106792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c0c1aa0 0x7ff09c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 --2- 172.21.15.94:0/666106792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c14f790 0x7ff09c1529d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 -- 172.21.15.94:0/666106792 >> 172.21.15.94:0/666106792 conn(0x7ff09c0c6710 msgr2=0x7ff09c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 -- 172.21.15.94:0/666106792 shutdown_connections 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.228+0000 7ff0a3708640 1 -- 172.21.15.94:0/666106792 wait complete. 2024-03-06T07:19:22.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.229+0000 7ff0a3708640 1 Processor -- start 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.229+0000 7ff0a3708640 1 -- start start 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a3708640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff09c0c1aa0 0x7ff09c1eee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a3708640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c0c5c60 0x7ff09c156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a3708640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c14f790 0x7ff09c1ead10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a3708640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c06cc00 con 0x7ff09c0c5c60 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a3708640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0c0d60 con 0x7ff09c0c1aa0 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a3708640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c06cd80 con 0x7ff09c14f790 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c0c5c60 0x7ff09c156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c0c5c60 0x7ff09c156eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50930/0 (socket says 172.21.15.94:50930) 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 -- 172.21.15.94:0/1972773013 learned_addr learned my addr 172.21.15.94:0/1972773013 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a1ca5640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c14f790 0x7ff09c1ead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a14a4640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff09c0c1aa0 0x7ff09c1eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 -- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c14f790 msgr2=0x7ff09c1ead10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c14f790 0x7ff09c1ead10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 -- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff09c0c1aa0 msgr2=0x7ff09c1eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff09c0c1aa0 0x7ff09c1eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 -- 172.21.15.94:0/1972773013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c15f440 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a0ca3640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c0c5c60 0x7ff09c156eb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00e9e0 tx=0x7ff08c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff091ffb640 1 -- 172.21.15.94:0/1972773013 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c017070 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff091ffb640 1 -- 172.21.15.94:0/1972773013 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c00c970 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09c15f160 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.230+0000 7ff091ffb640 1 -- 172.21.15.94:0/1972773013 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00cd60 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.231+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09c1ee600 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.232+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09c04f330 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.232+0000 7ff091ffb640 1 -- 172.21.15.94:0/1972773013 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff08c01a070 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.232+0000 7ff091ffb640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff08006b4d0 0x7ff08006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.232+0000 7ff091ffb640 1 -- 172.21.15.94:0/1972773013 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff08c08a2d0 con 0x7ff09c0c5c60 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.233+0000 7ff0a14a4640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff08006b4d0 0x7ff08006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.233+0000 7ff0a14a4640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff08006b4d0 0x7ff08006d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff098036000 tx=0x7ff09803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.237+0000 7ff091ffb640 1 -- 172.21.15.94:0/1972773013 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff08c058970 con 0x7ff09c0c5c60 2024-03-06T07:19:22.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.408+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff09c0c1f40 con 0x7ff08006b4d0 2024-03-06T07:19:22.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.414+0000 7ff091ffb640 1 -- 172.21.15.94:0/1972773013 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237198 (secure 0 0 0) 0x7ff09c0c1f40 con 0x7ff08006b4d0 2024-03-06T07:19:22.419 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:22.420 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:19:22.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff08006b4d0 msgr2=0x7ff08006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff08006b4d0 0x7ff08006d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff098036000 tx=0x7ff09803b040 comp rx=0 tx=0).stop 2024-03-06T07:19:22.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c0c5c60 msgr2=0x7ff09c156eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c0c5c60 0x7ff09c156eb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00e9e0 tx=0x7ff08c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 shutdown_connections 2024-03-06T07:19:22.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff08006b4d0 0x7ff08006d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff09c14f790 0x7ff09c1ead10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09c0c5c60 0x7ff09c156eb0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 --2- 172.21.15.94:0/1972773013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff09c0c1aa0 0x7ff09c1eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 >> 172.21.15.94:0/1972773013 conn(0x7ff09c0c6710 msgr2=0x7ff09c0a6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:22.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 shutdown_connections 2024-03-06T07:19:22.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.417+0000 7ff0a3708640 1 -- 172.21.15.94:0/1972773013 wait complete. 2024-03-06T07:19:22.429 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-03-06T07:19:20.649316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":227752,"kb_used_data":3264,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754746968,"statfs":{"total":773094113280,"available":772860895232,"internally_reserved":0,"allocated":3342336,"data_stored":2532672,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000749"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:18.439515+0000","last_change":"2024-03-06T07:19:18.439515+0000","last_active":"2024-03-06T07:19:18.439515+0000","last_peered":"2024-03-06T07:19:18.439515+0000","last_clean":"2024-03-06T07:19:18.439515+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:18.439515+0000","last_undegraded":"2024-03-06T07:19:18.439515+0000","last_fullsized":"2024-03-06T07:19:18.439515+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:17.435531+0000","last_change":"2024-03-06T07:19:17.435531+0000","last_active":"2024-03-06T07:19:17.435531+0000","last_peered":"2024-03-06T07:19:17.435531+0000","last_clean":"2024-03-06T07:19:17.435531+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:17.435531+0000","last_undegraded":"2024-03-06T07:19:17.435531+0000","last_fullsized":"2024-03-06T07:19:17.435531+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":198,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:19:13.623767+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:19:13.623767+0000","last_peered":"2024-03-06T07:19:13.623767+0000","last_clean":"2024-03-06T07:19:13.623767+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:19:13.623767+0000","last_undegraded":"2024-03-06T07:19:13.623767+0000","last_fullsized":"2024-03-06T07:19:13.623767+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":759360,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":168992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":168992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":168992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":168992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":168992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":168992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31640,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":466944,"data_stored":759360,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:19:22.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:19:22.434 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:19:22.430+0000 7f2fc752a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:19:22.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.568+0000 7f9823684640 1 Processor -- start 2024-03-06T07:19:22.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.568+0000 7f9823684640 1 -- start start 2024-03-06T07:19:22.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.568+0000 7f9823684640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f981c0c1aa0 0x7f981c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9823684640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9823684640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c1563d0 con 0x7f981c14f790 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9823684640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981c0c2680 con 0x7f981c0c1aa0 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9823684640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c28d0 con 0x7f981c155a40 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9821420640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f981c14f790 0x7f981c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43638/0 (socket says 172.21.15.94:43638) 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9821420640 1 -- 172.21.15.94:0/270595903 learned_addr learned my addr 172.21.15.94:0/270595903 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9820c1f640 1 --2- 172.21.15.94:0/270595903 >> v2:172.21.15.145:3300/0 conn(0x7f981c0c1aa0 0x7f981c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9821c21640 1 --2- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9813fff640 1 -- 172.21.15.94:0/270595903 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355214174 0 0) 0x7f981c1563d0 con 0x7f981c14f790 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9813fff640 1 -- 172.21.15.94:0/270595903 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808003520 con 0x7f981c14f790 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9820c1f640 1 -- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 msgr2=0x7f981c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9820c1f640 1 --2- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9820c1f640 1 -- 172.21.15.94:0/270595903 >> v1:172.21.15.94:6789/0 conn(0x7f981c14f790 legacy=0x7f981c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9813fff640 1 -- 172.21.15.94:0/270595903 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702113167 0 0) 0x7f9808003520 con 0x7f981c14f790 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9820c1f640 1 -- 172.21.15.94:0/270595903 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0c2bb0 con 0x7f981c0c1aa0 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.569+0000 7f9821c21640 1 --2- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.570+0000 7f9820c1f640 1 --2- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c0c1aa0 0x7f981c1529d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f980c0097a0 tx=0x7f980c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a5e9f15aac2ab79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.570+0000 7f9813fff640 1 -- 172.21.15.94:0/270595903 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c037070 con 0x7f981c0c1aa0 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.570+0000 7f9813fff640 1 -- 172.21.15.94:0/270595903 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c030510 con 0x7f981c0c1aa0 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.570+0000 7f9813fff640 1 -- 172.21.15.94:0/270595903 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818001670 con 0x7f981c0c1aa0 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 -- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c0c1aa0 msgr2=0x7f981c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 --2- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c0c1aa0 0x7f981c1529d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f980c0097a0 tx=0x7f980c02b790 comp rx=0 tx=0).stop 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 -- 172.21.15.94:0/270595903 shutdown_connections 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 --2- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 --2- 172.21.15.94:0/270595903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c0c1aa0 0x7f981c1529d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 -- 172.21.15.94:0/270595903 >> 172.21.15.94:0/270595903 conn(0x7f981c0c6710 msgr2=0x7f981c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:22.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 -- 172.21.15.94:0/270595903 shutdown_connections 2024-03-06T07:19:22.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.571+0000 7f9823684640 1 -- 172.21.15.94:0/270595903 wait complete. 2024-03-06T07:19:22.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 Processor -- start 2024-03-06T07:19:22.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 -- start start 2024-03-06T07:19:22.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f981c0c1aa0 0x7f981c153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c14f790 0x7f981c154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c1ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c0d60 con 0x7f981c0c1aa0 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c155520 con 0x7f981c14f790 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9823684640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c5e30 con 0x7f981c155a40 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.572+0000 7f9821420640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f981c0c1aa0 0x7f981c153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f981c0c1aa0 0x7f981c153dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50936/0 (socket says 172.21.15.94:50936) 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 -- 172.21.15.94:0/3635570303 learned_addr learned my addr 172.21.15.94:0/3635570303 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821c21640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c1ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9820c1f640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c14f790 0x7f981c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 -- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 msgr2=0x7f981c1ec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c1ec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 -- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c14f790 msgr2=0x7f981c154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c14f790 0x7f981c154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 -- 172.21.15.94:0/3635570303 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c1508c0 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9821420640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f981c0c1aa0 0x7f981c153dd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f981800c750 tx=0x7f981800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9811ffb640 1 -- 172.21.15.94:0/3635570303 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818005ad0 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9811ffb640 1 -- 172.21.15.94:0/3635570303 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818005c90 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c150ba0 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9811ffb640 1 -- 172.21.15.94:0/3635570303 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180072b0 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.573+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c1dd6b0 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.574+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c04f330 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.575+0000 7f9811ffb640 1 -- 172.21.15.94:0/3635570303 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9818007450 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.575+0000 7f9811ffb640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f980006b500 0x7f980006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.575+0000 7f9811ffb640 1 -- 172.21.15.94:0/3635570303 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f981808a010 con 0x7f981c0c1aa0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.575+0000 7f9820c1f640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f980006b500 0x7f980006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.576+0000 7f9820c1f640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f980006b500 0x7f980006d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f980c034000 tx=0x7f980c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.580+0000 7f9811ffb640 1 -- 172.21.15.94:0/3635570303 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9818058d90 con 0x7f981c0c1aa0 2024-03-06T07:19:22.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.747+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f981c1ed0b0 con 0x7f980006b500 2024-03-06T07:19:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.753+0000 7f9811ffb640 1 -- 172.21.15.94:0/3635570303 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252031 (secure 0 0 0) 0x7f981c1ed0b0 con 0x7f980006b500 2024-03-06T07:19:22.758 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:22.759 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.756+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f980006b500 msgr2=0x7f980006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.756+0000 7f9823684640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f980006b500 0x7f980006d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f980c034000 tx=0x7f980c02f000 comp rx=0 tx=0).stop 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.756+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f981c0c1aa0 msgr2=0x7f981c153dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.756+0000 7f9823684640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f981c0c1aa0 0x7f981c153dd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f981800c750 tx=0x7f981800cb50 comp rx=0 tx=0).stop 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 shutdown_connections 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f980006b500 0x7f980006d9c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f981c155a40 0x7f981c1ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f981c14f790 0x7f981c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 --2- 172.21.15.94:0/3635570303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f981c0c1aa0 0x7f981c153dd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 >> 172.21.15.94:0/3635570303 conn(0x7f981c0c6710 msgr2=0x7f981c06dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:22.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 shutdown_connections 2024-03-06T07:19:22.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.757+0000 7f9823684640 1 -- 172.21.15.94:0/3635570303 wait complete. 2024-03-06T07:19:22.769 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-03-06T07:19:22.649645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232392,"kb_used_data":3712,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742328,"statfs":{"total":773094113280,"available":772856143872,"internally_reserved":0,"allocated":3801088,"data_stored":2751372,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.209510"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455453+0000","last_change":"2024-03-06T07:19:19.464926+0000","last_active":"2024-03-06T07:19:20.455453+0000","last_peered":"2024-03-06T07:19:20.455453+0000","last_clean":"2024-03-06T07:19:20.455453+0000","last_became_active":"2024-03-06T07:19:19.464690+0000","last_became_peered":"2024-03-06T07:19:19.464690+0000","last_unstale":"2024-03-06T07:19:20.455453+0000","last_undegraded":"2024-03-06T07:19:20.455453+0000","last_fullsized":"2024-03-06T07:19:20.455453+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:15.942607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455269+0000","last_change":"2024-03-06T07:19:19.464135+0000","last_active":"2024-03-06T07:19:20.455269+0000","last_peered":"2024-03-06T07:19:20.455269+0000","last_clean":"2024-03-06T07:19:20.455269+0000","last_became_active":"2024-03-06T07:19:19.463886+0000","last_became_peered":"2024-03-06T07:19:19.463886+0000","last_unstale":"2024-03-06T07:19:20.455269+0000","last_undegraded":"2024-03-06T07:19:20.455269+0000","last_fullsized":"2024-03-06T07:19:20.455269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:51.732469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518052+0000","last_change":"2024-03-06T07:19:19.488817+0000","last_active":"2024-03-06T07:19:20.518052+0000","last_peered":"2024-03-06T07:19:20.518052+0000","last_clean":"2024-03-06T07:19:20.518052+0000","last_became_active":"2024-03-06T07:19:19.488564+0000","last_became_peered":"2024-03-06T07:19:19.488564+0000","last_unstale":"2024-03-06T07:19:20.518052+0000","last_undegraded":"2024-03-06T07:19:20.518052+0000","last_fullsized":"2024-03-06T07:19:20.518052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:25.329174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517855+0000","last_change":"2024-03-06T07:19:19.462736+0000","last_active":"2024-03-06T07:19:20.517855+0000","last_peered":"2024-03-06T07:19:20.517855+0000","last_clean":"2024-03-06T07:19:20.517855+0000","last_became_active":"2024-03-06T07:19:19.462486+0000","last_became_peered":"2024-03-06T07:19:19.462486+0000","last_unstale":"2024-03-06T07:19:20.517855+0000","last_undegraded":"2024-03-06T07:19:20.517855+0000","last_fullsized":"2024-03-06T07:19:20.517855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:11.189488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517433+0000","last_change":"2024-03-06T07:19:19.463215+0000","last_active":"2024-03-06T07:19:20.517433+0000","last_peered":"2024-03-06T07:19:20.517433+0000","last_clean":"2024-03-06T07:19:20.517433+0000","last_became_active":"2024-03-06T07:19:19.462936+0000","last_became_peered":"2024-03-06T07:19:19.462936+0000","last_unstale":"2024-03-06T07:19:20.517433+0000","last_undegraded":"2024-03-06T07:19:20.517433+0000","last_fullsized":"2024-03-06T07:19:20.517433+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:13.750481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517529+0000","last_change":"2024-03-06T07:19:19.467717+0000","last_active":"2024-03-06T07:19:20.517529+0000","last_peered":"2024-03-06T07:19:20.517529+0000","last_clean":"2024-03-06T07:19:20.517529+0000","last_became_active":"2024-03-06T07:19:19.466909+0000","last_became_peered":"2024-03-06T07:19:19.466909+0000","last_unstale":"2024-03-06T07:19:20.517529+0000","last_undegraded":"2024-03-06T07:19:20.517529+0000","last_fullsized":"2024-03-06T07:19:20.517529+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:33.912448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517586+0000","last_change":"2024-03-06T07:19:19.485794+0000","last_active":"2024-03-06T07:19:20.517586+0000","last_peered":"2024-03-06T07:19:20.517586+0000","last_clean":"2024-03-06T07:19:20.517586+0000","last_became_active":"2024-03-06T07:19:19.485302+0000","last_became_peered":"2024-03-06T07:19:19.485302+0000","last_unstale":"2024-03-06T07:19:20.517586+0000","last_undegraded":"2024-03-06T07:19:20.517586+0000","last_fullsized":"2024-03-06T07:19:20.517586+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.853863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455049+0000","last_change":"2024-03-06T07:19:19.487033+0000","last_active":"2024-03-06T07:19:20.455049+0000","last_peered":"2024-03-06T07:19:20.455049+0000","last_clean":"2024-03-06T07:19:20.455049+0000","last_became_active":"2024-03-06T07:19:19.486757+0000","last_became_peered":"2024-03-06T07:19:19.486757+0000","last_unstale":"2024-03-06T07:19:20.455049+0000","last_undegraded":"2024-03-06T07:19:20.455049+0000","last_fullsized":"2024-03-06T07:19:20.455049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:43.428024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517998+0000","last_change":"2024-03-06T07:19:19.488809+0000","last_active":"2024-03-06T07:19:20.517998+0000","last_peered":"2024-03-06T07:19:20.517998+0000","last_clean":"2024-03-06T07:19:20.517998+0000","last_became_active":"2024-03-06T07:19:19.488548+0000","last_became_peered":"2024-03-06T07:19:19.488548+0000","last_unstale":"2024-03-06T07:19:20.517998+0000","last_undegraded":"2024-03-06T07:19:20.517998+0000","last_fullsized":"2024-03-06T07:19:20.517998+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:49.153689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517883+0000","last_change":"2024-03-06T07:19:19.486346+0000","last_active":"2024-03-06T07:19:20.517883+0000","last_peered":"2024-03-06T07:19:20.517883+0000","last_clean":"2024-03-06T07:19:20.517883+0000","last_became_active":"2024-03-06T07:19:19.485289+0000","last_became_peered":"2024-03-06T07:19:19.485289+0000","last_unstale":"2024-03-06T07:19:20.517883+0000","last_undegraded":"2024-03-06T07:19:20.517883+0000","last_fullsized":"2024-03-06T07:19:20.517883+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:56.762353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518033+0000","last_change":"2024-03-06T07:19:19.486621+0000","last_active":"2024-03-06T07:19:20.518033+0000","last_peered":"2024-03-06T07:19:20.518033+0000","last_clean":"2024-03-06T07:19:20.518033+0000","last_became_active":"2024-03-06T07:19:19.486329+0000","last_became_peered":"2024-03-06T07:19:19.486329+0000","last_unstale":"2024-03-06T07:19:20.518033+0000","last_undegraded":"2024-03-06T07:19:20.518033+0000","last_fullsized":"2024-03-06T07:19:20.518033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:01.000401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518270+0000","last_change":"2024-03-06T07:19:19.465023+0000","last_active":"2024-03-06T07:19:20.518270+0000","last_peered":"2024-03-06T07:19:20.518270+0000","last_clean":"2024-03-06T07:19:20.518270+0000","last_became_active":"2024-03-06T07:19:19.464749+0000","last_became_peered":"2024-03-06T07:19:19.464749+0000","last_unstale":"2024-03-06T07:19:20.518270+0000","last_undegraded":"2024-03-06T07:19:20.518270+0000","last_fullsized":"2024-03-06T07:19:20.518270+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:49.006305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517561+0000","last_change":"2024-03-06T07:19:19.487109+0000","last_active":"2024-03-06T07:19:20.517561+0000","last_peered":"2024-03-06T07:19:20.517561+0000","last_clean":"2024-03-06T07:19:20.517561+0000","last_became_active":"2024-03-06T07:19:19.486707+0000","last_became_peered":"2024-03-06T07:19:19.486707+0000","last_unstale":"2024-03-06T07:19:20.517561+0000","last_undegraded":"2024-03-06T07:19:20.517561+0000","last_fullsized":"2024-03-06T07:19:20.517561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:26.094998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.458620+0000","last_change":"2024-03-06T07:19:19.466574+0000","last_active":"2024-03-06T07:19:20.458620+0000","last_peered":"2024-03-06T07:19:20.458620+0000","last_clean":"2024-03-06T07:19:20.458620+0000","last_became_active":"2024-03-06T07:19:19.466283+0000","last_became_peered":"2024-03-06T07:19:19.466283+0000","last_unstale":"2024-03-06T07:19:20.458620+0000","last_undegraded":"2024-03-06T07:19:20.458620+0000","last_fullsized":"2024-03-06T07:19:20.458620+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:03.069202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517679+0000","last_change":"2024-03-06T07:19:19.485581+0000","last_active":"2024-03-06T07:19:20.517679+0000","last_peered":"2024-03-06T07:19:20.517679+0000","last_clean":"2024-03-06T07:19:20.517679+0000","last_became_active":"2024-03-06T07:19:19.485272+0000","last_became_peered":"2024-03-06T07:19:19.485272+0000","last_unstale":"2024-03-06T07:19:20.517679+0000","last_undegraded":"2024-03-06T07:19:20.517679+0000","last_fullsized":"2024-03-06T07:19:20.517679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:22.303368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517435+0000","last_change":"2024-03-06T07:19:19.488788+0000","last_active":"2024-03-06T07:19:20.517435+0000","last_peered":"2024-03-06T07:19:20.517435+0000","last_clean":"2024-03-06T07:19:20.517435+0000","last_became_active":"2024-03-06T07:19:19.488477+0000","last_became_peered":"2024-03-06T07:19:19.488477+0000","last_unstale":"2024-03-06T07:19:20.517435+0000","last_undegraded":"2024-03-06T07:19:20.517435+0000","last_fullsized":"2024-03-06T07:19:20.517435+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:32.348021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518483+0000","last_change":"2024-03-06T07:19:19.489117+0000","last_active":"2024-03-06T07:19:20.518483+0000","last_peered":"2024-03-06T07:19:20.518483+0000","last_clean":"2024-03-06T07:19:20.518483+0000","last_became_active":"2024-03-06T07:19:19.488836+0000","last_became_peered":"2024-03-06T07:19:19.488836+0000","last_unstale":"2024-03-06T07:19:20.518483+0000","last_undegraded":"2024-03-06T07:19:20.518483+0000","last_fullsized":"2024-03-06T07:19:20.518483+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:17.278137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459350+0000","last_change":"2024-03-06T07:19:19.475902+0000","last_active":"2024-03-06T07:19:20.459350+0000","last_peered":"2024-03-06T07:19:20.459350+0000","last_clean":"2024-03-06T07:19:20.459350+0000","last_became_active":"2024-03-06T07:19:19.475622+0000","last_became_peered":"2024-03-06T07:19:19.475622+0000","last_unstale":"2024-03-06T07:19:20.459350+0000","last_undegraded":"2024-03-06T07:19:20.459350+0000","last_fullsized":"2024-03-06T07:19:20.459350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:03.069202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518746+0000","last_change":"2024-03-06T07:19:19.460785+0000","last_active":"2024-03-06T07:19:20.518746+0000","last_peered":"2024-03-06T07:19:20.518746+0000","last_clean":"2024-03-06T07:19:20.518746+0000","last_became_active":"2024-03-06T07:19:19.460519+0000","last_became_peered":"2024-03-06T07:19:19.460519+0000","last_unstale":"2024-03-06T07:19:20.518746+0000","last_undegraded":"2024-03-06T07:19:20.518746+0000","last_fullsized":"2024-03-06T07:19:20.518746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:10.833641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459776+0000","last_change":"2024-03-06T07:19:19.468233+0000","last_active":"2024-03-06T07:19:20.459776+0000","last_peered":"2024-03-06T07:19:20.459776+0000","last_clean":"2024-03-06T07:19:20.459776+0000","last_became_active":"2024-03-06T07:19:19.467928+0000","last_became_peered":"2024-03-06T07:19:19.467928+0000","last_unstale":"2024-03-06T07:19:20.459776+0000","last_undegraded":"2024-03-06T07:19:20.459776+0000","last_fullsized":"2024-03-06T07:19:20.459776+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:00.817685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517751+0000","last_change":"2024-03-06T07:19:19.486471+0000","last_active":"2024-03-06T07:19:20.517751+0000","last_peered":"2024-03-06T07:19:20.517751+0000","last_clean":"2024-03-06T07:19:20.517751+0000","last_became_active":"2024-03-06T07:19:19.485660+0000","last_became_peered":"2024-03-06T07:19:19.485660+0000","last_unstale":"2024-03-06T07:19:20.517751+0000","last_undegraded":"2024-03-06T07:19:20.517751+0000","last_fullsized":"2024-03-06T07:19:20.517751+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:03.819592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.458845+0000","last_change":"2024-03-06T07:19:19.462788+0000","last_active":"2024-03-06T07:19:20.458845+0000","last_peered":"2024-03-06T07:19:20.458845+0000","last_clean":"2024-03-06T07:19:20.458845+0000","last_became_active":"2024-03-06T07:19:19.462465+0000","last_became_peered":"2024-03-06T07:19:19.462465+0000","last_unstale":"2024-03-06T07:19:20.458845+0000","last_undegraded":"2024-03-06T07:19:20.458845+0000","last_fullsized":"2024-03-06T07:19:20.458845+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:57.621686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517925+0000","last_change":"2024-03-06T07:19:19.482045+0000","last_active":"2024-03-06T07:19:20.517925+0000","last_peered":"2024-03-06T07:19:20.517925+0000","last_clean":"2024-03-06T07:19:20.517925+0000","last_became_active":"2024-03-06T07:19:19.481761+0000","last_became_peered":"2024-03-06T07:19:19.481761+0000","last_unstale":"2024-03-06T07:19:20.517925+0000","last_undegraded":"2024-03-06T07:19:20.517925+0000","last_fullsized":"2024-03-06T07:19:20.517925+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:27.060918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518768+0000","last_change":"2024-03-06T07:19:19.480667+0000","last_active":"2024-03-06T07:19:20.518768+0000","last_peered":"2024-03-06T07:19:20.518768+0000","last_clean":"2024-03-06T07:19:20.518768+0000","last_became_active":"2024-03-06T07:19:19.480198+0000","last_became_peered":"2024-03-06T07:19:19.480198+0000","last_unstale":"2024-03-06T07:19:20.518768+0000","last_undegraded":"2024-03-06T07:19:20.518768+0000","last_fullsized":"2024-03-06T07:19:20.518768+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.373509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459891+0000","last_change":"2024-03-06T07:19:19.464770+0000","last_active":"2024-03-06T07:19:20.459891+0000","last_peered":"2024-03-06T07:19:20.459891+0000","last_clean":"2024-03-06T07:19:20.459891+0000","last_became_active":"2024-03-06T07:19:19.464463+0000","last_became_peered":"2024-03-06T07:19:19.464463+0000","last_unstale":"2024-03-06T07:19:20.459891+0000","last_undegraded":"2024-03-06T07:19:20.459891+0000","last_fullsized":"2024-03-06T07:19:20.459891+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:33.912448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518073+0000","last_change":"2024-03-06T07:19:19.485865+0000","last_active":"2024-03-06T07:19:20.518073+0000","last_peered":"2024-03-06T07:19:20.518073+0000","last_clean":"2024-03-06T07:19:20.518073+0000","last_became_active":"2024-03-06T07:19:19.485595+0000","last_became_peered":"2024-03-06T07:19:19.485595+0000","last_unstale":"2024-03-06T07:19:20.518073+0000","last_undegraded":"2024-03-06T07:19:20.518073+0000","last_fullsized":"2024-03-06T07:19:20.518073+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:25.329174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459094+0000","last_change":"2024-03-06T07:19:19.465160+0000","last_active":"2024-03-06T07:19:20.459094+0000","last_peered":"2024-03-06T07:19:20.459094+0000","last_clean":"2024-03-06T07:19:20.459094+0000","last_became_active":"2024-03-06T07:19:19.464888+0000","last_became_peered":"2024-03-06T07:19:19.464888+0000","last_unstale":"2024-03-06T07:19:20.459094+0000","last_undegraded":"2024-03-06T07:19:20.459094+0000","last_fullsized":"2024-03-06T07:19:20.459094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:55.318228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517780+0000","last_change":"2024-03-06T07:19:19.480122+0000","last_active":"2024-03-06T07:19:20.517780+0000","last_peered":"2024-03-06T07:19:20.517780+0000","last_clean":"2024-03-06T07:19:20.517780+0000","last_became_active":"2024-03-06T07:19:19.479783+0000","last_became_peered":"2024-03-06T07:19:19.479783+0000","last_unstale":"2024-03-06T07:19:20.517780+0000","last_undegraded":"2024-03-06T07:19:20.517780+0000","last_fullsized":"2024-03-06T07:19:20.517780+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:16.507323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459685+0000","last_change":"2024-03-06T07:19:19.462835+0000","last_active":"2024-03-06T07:19:20.459685+0000","last_peered":"2024-03-06T07:19:20.459685+0000","last_clean":"2024-03-06T07:19:20.459685+0000","last_became_active":"2024-03-06T07:19:19.462561+0000","last_became_peered":"2024-03-06T07:19:19.462561+0000","last_unstale":"2024-03-06T07:19:20.459685+0000","last_undegraded":"2024-03-06T07:19:20.459685+0000","last_fullsized":"2024-03-06T07:19:20.459685+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:57.621686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.458792+0000","last_change":"2024-03-06T07:19:19.465923+0000","last_active":"2024-03-06T07:19:20.458792+0000","last_peered":"2024-03-06T07:19:20.458792+0000","last_clean":"2024-03-06T07:19:20.458792+0000","last_became_active":"2024-03-06T07:19:19.465664+0000","last_became_peered":"2024-03-06T07:19:19.465664+0000","last_unstale":"2024-03-06T07:19:20.458792+0000","last_undegraded":"2024-03-06T07:19:20.458792+0000","last_fullsized":"2024-03-06T07:19:20.458792+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:27.682534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518118+0000","last_change":"2024-03-06T07:19:19.482287+0000","last_active":"2024-03-06T07:19:20.518118+0000","last_peered":"2024-03-06T07:19:20.518118+0000","last_clean":"2024-03-06T07:19:20.518118+0000","last_became_active":"2024-03-06T07:19:19.482011+0000","last_became_peered":"2024-03-06T07:19:19.482011+0000","last_unstale":"2024-03-06T07:19:20.518118+0000","last_undegraded":"2024-03-06T07:19:20.518118+0000","last_fullsized":"2024-03-06T07:19:20.518118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:04.231653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517319+0000","last_change":"2024-03-06T07:19:19.463865+0000","last_active":"2024-03-06T07:19:20.517319+0000","last_peered":"2024-03-06T07:19:20.517319+0000","last_clean":"2024-03-06T07:19:20.517319+0000","last_became_active":"2024-03-06T07:19:19.463606+0000","last_became_peered":"2024-03-06T07:19:19.463606+0000","last_unstale":"2024-03-06T07:19:20.517319+0000","last_undegraded":"2024-03-06T07:19:20.517319+0000","last_fullsized":"2024-03-06T07:19:20.517319+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:00.817685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518463+0000","last_change":"2024-03-06T07:19:19.488861+0000","last_active":"2024-03-06T07:19:20.518463+0000","last_peered":"2024-03-06T07:19:20.518463+0000","last_clean":"2024-03-06T07:19:20.518463+0000","last_became_active":"2024-03-06T07:19:19.488575+0000","last_became_peered":"2024-03-06T07:19:19.488575+0000","last_unstale":"2024-03-06T07:19:20.518463+0000","last_undegraded":"2024-03-06T07:19:20.518463+0000","last_fullsized":"2024-03-06T07:19:20.518463+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:36:03.819592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459897+0000","last_change":"2024-03-06T07:19:18.456175+0000","last_active":"2024-03-06T07:19:20.459897+0000","last_peered":"2024-03-06T07:19:20.459897+0000","last_clean":"2024-03-06T07:19:20.459897+0000","last_became_active":"2024-03-06T07:19:18.455898+0000","last_became_peered":"2024-03-06T07:19:18.455898+0000","last_unstale":"2024-03-06T07:19:20.459897+0000","last_undegraded":"2024-03-06T07:19:20.459897+0000","last_fullsized":"2024-03-06T07:19:20.459897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:41.022537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518366+0000","last_change":"2024-03-06T07:19:19.487258+0000","last_active":"2024-03-06T07:19:20.518366+0000","last_peered":"2024-03-06T07:19:20.518366+0000","last_clean":"2024-03-06T07:19:20.518366+0000","last_became_active":"2024-03-06T07:19:19.487014+0000","last_became_peered":"2024-03-06T07:19:19.487014+0000","last_unstale":"2024-03-06T07:19:20.518366+0000","last_undegraded":"2024-03-06T07:19:20.518366+0000","last_fullsized":"2024-03-06T07:19:20.518366+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:49.006305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517731+0000","last_change":"2024-03-06T07:19:18.455158+0000","last_active":"2024-03-06T07:19:20.517731+0000","last_peered":"2024-03-06T07:19:20.517731+0000","last_clean":"2024-03-06T07:19:20.517731+0000","last_became_active":"2024-03-06T07:19:18.454755+0000","last_became_peered":"2024-03-06T07:19:18.454755+0000","last_unstale":"2024-03-06T07:19:20.517731+0000","last_undegraded":"2024-03-06T07:19:20.517731+0000","last_fullsized":"2024-03-06T07:19:20.517731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:01.928629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459625+0000","last_change":"2024-03-06T07:19:19.466459+0000","last_active":"2024-03-06T07:19:20.459625+0000","last_peered":"2024-03-06T07:19:20.459625+0000","last_clean":"2024-03-06T07:19:20.459625+0000","last_became_active":"2024-03-06T07:19:19.466163+0000","last_became_peered":"2024-03-06T07:19:19.466163+0000","last_unstale":"2024-03-06T07:19:20.459625+0000","last_undegraded":"2024-03-06T07:19:20.459625+0000","last_fullsized":"2024-03-06T07:19:20.459625+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:25.329174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518534+0000","last_change":"2024-03-06T07:19:18.461396+0000","last_active":"2024-03-06T07:19:20.518534+0000","last_peered":"2024-03-06T07:19:20.518534+0000","last_clean":"2024-03-06T07:19:20.518534+0000","last_became_active":"2024-03-06T07:19:18.461072+0000","last_became_peered":"2024-03-06T07:19:18.461072+0000","last_unstale":"2024-03-06T07:19:20.518534+0000","last_undegraded":"2024-03-06T07:19:20.518534+0000","last_fullsized":"2024-03-06T07:19:20.518534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:09.829657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455448+0000","last_change":"2024-03-06T07:19:19.468608+0000","last_active":"2024-03-06T07:19:20.455448+0000","last_peered":"2024-03-06T07:19:20.455448+0000","last_clean":"2024-03-06T07:19:20.455448+0000","last_became_active":"2024-03-06T07:19:19.468356+0000","last_became_peered":"2024-03-06T07:19:19.468356+0000","last_unstale":"2024-03-06T07:19:20.455448+0000","last_undegraded":"2024-03-06T07:19:20.455448+0000","last_fullsized":"2024-03-06T07:19:20.455448+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:45.666442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459362+0000","last_change":"2024-03-06T07:19:18.458861+0000","last_active":"2024-03-06T07:19:20.459362+0000","last_peered":"2024-03-06T07:19:20.459362+0000","last_clean":"2024-03-06T07:19:20.459362+0000","last_became_active":"2024-03-06T07:19:18.458606+0000","last_became_peered":"2024-03-06T07:19:18.458606+0000","last_unstale":"2024-03-06T07:19:20.459362+0000","last_undegraded":"2024-03-06T07:19:20.459362+0000","last_fullsized":"2024-03-06T07:19:20.459362+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:41.022537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517835+0000","last_change":"2024-03-06T07:19:19.463039+0000","last_active":"2024-03-06T07:19:20.517835+0000","last_peered":"2024-03-06T07:19:20.517835+0000","last_clean":"2024-03-06T07:19:20.517835+0000","last_became_active":"2024-03-06T07:19:19.462766+0000","last_became_peered":"2024-03-06T07:19:19.462766+0000","last_unstale":"2024-03-06T07:19:20.517835+0000","last_undegraded":"2024-03-06T07:19:20.517835+0000","last_fullsized":"2024-03-06T07:19:20.517835+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:33.912448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517799+0000","last_change":"2024-03-06T07:19:18.462706+0000","last_active":"2024-03-06T07:19:20.517799+0000","last_peered":"2024-03-06T07:19:20.517799+0000","last_clean":"2024-03-06T07:19:20.517799+0000","last_became_active":"2024-03-06T07:19:18.462453+0000","last_became_peered":"2024-03-06T07:19:18.462453+0000","last_unstale":"2024-03-06T07:19:20.517799+0000","last_undegraded":"2024-03-06T07:19:20.517799+0000","last_fullsized":"2024-03-06T07:19:20.517799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:11.369524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517751+0000","last_change":"2024-03-06T07:19:19.485565+0000","last_active":"2024-03-06T07:19:20.517751+0000","last_peered":"2024-03-06T07:19:20.517751+0000","last_clean":"2024-03-06T07:19:20.517751+0000","last_became_active":"2024-03-06T07:19:19.484926+0000","last_became_peered":"2024-03-06T07:19:19.484926+0000","last_unstale":"2024-03-06T07:19:20.517751+0000","last_undegraded":"2024-03-06T07:19:20.517751+0000","last_fullsized":"2024-03-06T07:19:20.517751+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:17.278137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518224+0000","last_change":"2024-03-06T07:19:18.460251+0000","last_active":"2024-03-06T07:19:20.518224+0000","last_peered":"2024-03-06T07:19:20.518224+0000","last_clean":"2024-03-06T07:19:20.518224+0000","last_became_active":"2024-03-06T07:19:18.459966+0000","last_became_peered":"2024-03-06T07:19:18.459966+0000","last_unstale":"2024-03-06T07:19:20.518224+0000","last_undegraded":"2024-03-06T07:19:20.518224+0000","last_fullsized":"2024-03-06T07:19:20.518224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:09.829657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.458999+0000","last_change":"2024-03-06T07:19:19.489342+0000","last_active":"2024-03-06T07:19:20.458999+0000","last_peered":"2024-03-06T07:19:20.458999+0000","last_clean":"2024-03-06T07:19:20.458999+0000","last_became_active":"2024-03-06T07:19:19.489076+0000","last_became_peered":"2024-03-06T07:19:19.489076+0000","last_unstale":"2024-03-06T07:19:20.458999+0000","last_undegraded":"2024-03-06T07:19:20.458999+0000","last_fullsized":"2024-03-06T07:19:20.458999+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:00.817685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517829+0000","last_change":"2024-03-06T07:19:18.461588+0000","last_active":"2024-03-06T07:19:20.517829+0000","last_peered":"2024-03-06T07:19:20.517829+0000","last_clean":"2024-03-06T07:19:20.517829+0000","last_became_active":"2024-03-06T07:19:18.461054+0000","last_became_peered":"2024-03-06T07:19:18.461054+0000","last_unstale":"2024-03-06T07:19:20.517829+0000","last_undegraded":"2024-03-06T07:19:20.517829+0000","last_fullsized":"2024-03-06T07:19:20.517829+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:20:48.818640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455144+0000","last_change":"2024-03-06T07:19:19.474081+0000","last_active":"2024-03-06T07:19:20.455144+0000","last_peered":"2024-03-06T07:19:20.455144+0000","last_clean":"2024-03-06T07:19:20.455144+0000","last_became_active":"2024-03-06T07:19:19.473750+0000","last_became_peered":"2024-03-06T07:19:19.473750+0000","last_unstale":"2024-03-06T07:19:20.455144+0000","last_undegraded":"2024-03-06T07:19:20.455144+0000","last_fullsized":"2024-03-06T07:19:20.455144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:20.349340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459178+0000","last_change":"2024-03-06T07:19:18.454332+0000","last_active":"2024-03-06T07:19:20.459178+0000","last_peered":"2024-03-06T07:19:20.459178+0000","last_clean":"2024-03-06T07:19:20.459178+0000","last_became_active":"2024-03-06T07:19:18.454063+0000","last_became_peered":"2024-03-06T07:19:18.454063+0000","last_unstale":"2024-03-06T07:19:20.459178+0000","last_undegraded":"2024-03-06T07:19:20.459178+0000","last_fullsized":"2024-03-06T07:19:20.459178+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:01.928629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517985+0000","last_change":"2024-03-06T07:19:19.462806+0000","last_active":"2024-03-06T07:19:20.517985+0000","last_peered":"2024-03-06T07:19:20.517985+0000","last_clean":"2024-03-06T07:19:20.517985+0000","last_became_active":"2024-03-06T07:19:19.462526+0000","last_became_peered":"2024-03-06T07:19:19.462526+0000","last_unstale":"2024-03-06T07:19:20.517985+0000","last_undegraded":"2024-03-06T07:19:20.517985+0000","last_fullsized":"2024-03-06T07:19:20.517985+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:20:49.822624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455533+0000","last_change":"2024-03-06T07:19:18.455290+0000","last_active":"2024-03-06T07:19:20.455533+0000","last_peered":"2024-03-06T07:19:20.455533+0000","last_clean":"2024-03-06T07:19:20.455533+0000","last_became_active":"2024-03-06T07:19:18.455021+0000","last_became_peered":"2024-03-06T07:19:18.455021+0000","last_unstale":"2024-03-06T07:19:20.455533+0000","last_undegraded":"2024-03-06T07:19:20.455533+0000","last_fullsized":"2024-03-06T07:19:20.455533+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:31.344036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518460+0000","last_change":"2024-03-06T07:19:19.477819+0000","last_active":"2024-03-06T07:19:20.518460+0000","last_peered":"2024-03-06T07:19:20.518460+0000","last_clean":"2024-03-06T07:19:20.518460+0000","last_became_active":"2024-03-06T07:19:19.477563+0000","last_became_peered":"2024-03-06T07:19:19.477563+0000","last_unstale":"2024-03-06T07:19:20.518460+0000","last_undegraded":"2024-03-06T07:19:20.518460+0000","last_fullsized":"2024-03-06T07:19:20.518460+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:26.094998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518006+0000","last_change":"2024-03-06T07:19:18.459770+0000","last_active":"2024-03-06T07:19:20.518006+0000","last_peered":"2024-03-06T07:19:20.518006+0000","last_clean":"2024-03-06T07:19:20.518006+0000","last_became_active":"2024-03-06T07:19:18.459516+0000","last_became_peered":"2024-03-06T07:19:18.459516+0000","last_unstale":"2024-03-06T07:19:20.518006+0000","last_undegraded":"2024-03-06T07:19:20.518006+0000","last_fullsized":"2024-03-06T07:19:20.518006+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:54.314244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517741+0000","last_change":"2024-03-06T07:19:19.479784+0000","last_active":"2024-03-06T07:19:20.517741+0000","last_peered":"2024-03-06T07:19:20.517741+0000","last_clean":"2024-03-06T07:19:20.517741+0000","last_became_active":"2024-03-06T07:19:19.479502+0000","last_became_peered":"2024-03-06T07:19:19.479502+0000","last_unstale":"2024-03-06T07:19:20.517741+0000","last_undegraded":"2024-03-06T07:19:20.517741+0000","last_fullsized":"2024-03-06T07:19:20.517741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:00.817685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518135+0000","last_change":"2024-03-06T07:19:18.464013+0000","last_active":"2024-03-06T07:19:20.518135+0000","last_peered":"2024-03-06T07:19:20.518135+0000","last_clean":"2024-03-06T07:19:20.518135+0000","last_became_active":"2024-03-06T07:19:18.463735+0000","last_became_peered":"2024-03-06T07:19:18.463735+0000","last_unstale":"2024-03-06T07:19:20.518135+0000","last_undegraded":"2024-03-06T07:19:20.518135+0000","last_fullsized":"2024-03-06T07:19:20.518135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:02.065218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459112+0000","last_change":"2024-03-06T07:19:19.465471+0000","last_active":"2024-03-06T07:19:20.459112+0000","last_peered":"2024-03-06T07:19:20.459112+0000","last_clean":"2024-03-06T07:19:20.459112+0000","last_became_active":"2024-03-06T07:19:19.463944+0000","last_became_peered":"2024-03-06T07:19:19.463944+0000","last_unstale":"2024-03-06T07:19:20.459112+0000","last_undegraded":"2024-03-06T07:19:20.459112+0000","last_fullsized":"2024-03-06T07:19:20.459112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:33.912448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455220+0000","last_change":"2024-03-06T07:19:18.462118+0000","last_active":"2024-03-06T07:19:20.455220+0000","last_peered":"2024-03-06T07:19:20.455220+0000","last_clean":"2024-03-06T07:19:20.455220+0000","last_became_active":"2024-03-06T07:19:18.461849+0000","last_became_peered":"2024-03-06T07:19:18.461849+0000","last_unstale":"2024-03-06T07:19:20.455220+0000","last_undegraded":"2024-03-06T07:19:20.455220+0000","last_fullsized":"2024-03-06T07:19:20.455220+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:24.325190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517961+0000","last_change":"2024-03-06T07:19:19.485485+0000","last_active":"2024-03-06T07:19:20.517961+0000","last_peered":"2024-03-06T07:19:20.517961+0000","last_clean":"2024-03-06T07:19:20.517961+0000","last_became_active":"2024-03-06T07:19:19.485203+0000","last_became_peered":"2024-03-06T07:19:19.485203+0000","last_unstale":"2024-03-06T07:19:20.517961+0000","last_undegraded":"2024-03-06T07:19:20.517961+0000","last_fullsized":"2024-03-06T07:19:20.517961+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:16.329067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455330+0000","last_change":"2024-03-06T07:19:18.460399+0000","last_active":"2024-03-06T07:19:20.455330+0000","last_peered":"2024-03-06T07:19:20.455330+0000","last_clean":"2024-03-06T07:19:20.455330+0000","last_became_active":"2024-03-06T07:19:18.459643+0000","last_became_peered":"2024-03-06T07:19:18.459643+0000","last_unstale":"2024-03-06T07:19:20.455330+0000","last_undegraded":"2024-03-06T07:19:20.455330+0000","last_fullsized":"2024-03-06T07:19:20.455330+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:48.002321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455048+0000","last_change":"2024-03-06T07:19:19.464416+0000","last_active":"2024-03-06T07:19:20.455048+0000","last_peered":"2024-03-06T07:19:20.455048+0000","last_clean":"2024-03-06T07:19:20.455048+0000","last_became_active":"2024-03-06T07:19:19.464183+0000","last_became_peered":"2024-03-06T07:19:19.464183+0000","last_unstale":"2024-03-06T07:19:20.455048+0000","last_undegraded":"2024-03-06T07:19:20.455048+0000","last_fullsized":"2024-03-06T07:19:20.455048+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:01.000401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517628+0000","last_change":"2024-03-06T07:19:18.461487+0000","last_active":"2024-03-06T07:19:20.517628+0000","last_peered":"2024-03-06T07:19:20.517628+0000","last_clean":"2024-03-06T07:19:20.517628+0000","last_became_active":"2024-03-06T07:19:18.461083+0000","last_became_peered":"2024-03-06T07:19:18.461083+0000","last_unstale":"2024-03-06T07:19:20.517628+0000","last_undegraded":"2024-03-06T07:19:20.517628+0000","last_fullsized":"2024-03-06T07:19:20.517628+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:10.185504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518465+0000","last_change":"2024-03-06T07:19:19.473844+0000","last_active":"2024-03-06T07:19:20.518465+0000","last_peered":"2024-03-06T07:19:20.518465+0000","last_clean":"2024-03-06T07:19:20.518465+0000","last_became_active":"2024-03-06T07:19:19.473567+0000","last_became_peered":"2024-03-06T07:19:19.473567+0000","last_unstale":"2024-03-06T07:19:20.518465+0000","last_undegraded":"2024-03-06T07:19:20.518465+0000","last_fullsized":"2024-03-06T07:19:20.518465+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.853863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.458622+0000","last_change":"2024-03-06T07:19:18.456670+0000","last_active":"2024-03-06T07:19:20.458622+0000","last_peered":"2024-03-06T07:19:20.458622+0000","last_clean":"2024-03-06T07:19:20.458622+0000","last_became_active":"2024-03-06T07:19:18.456396+0000","last_became_peered":"2024-03-06T07:19:18.456396+0000","last_unstale":"2024-03-06T07:19:20.458622+0000","last_undegraded":"2024-03-06T07:19:20.458622+0000","last_fullsized":"2024-03-06T07:19:20.458622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:01.928629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455191+0000","last_change":"2024-03-06T07:19:19.465808+0000","last_active":"2024-03-06T07:19:20.455191+0000","last_peered":"2024-03-06T07:19:20.455191+0000","last_clean":"2024-03-06T07:19:20.455191+0000","last_became_active":"2024-03-06T07:19:19.465504+0000","last_became_peered":"2024-03-06T07:19:19.465504+0000","last_unstale":"2024-03-06T07:19:20.455191+0000","last_undegraded":"2024-03-06T07:19:20.455191+0000","last_fullsized":"2024-03-06T07:19:20.455191+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:38.738970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517467+0000","last_change":"2024-03-06T07:19:18.460740+0000","last_active":"2024-03-06T07:19:20.517467+0000","last_peered":"2024-03-06T07:19:20.517467+0000","last_clean":"2024-03-06T07:19:20.517467+0000","last_became_active":"2024-03-06T07:19:18.460480+0000","last_became_peered":"2024-03-06T07:19:18.460480+0000","last_unstale":"2024-03-06T07:19:20.517467+0000","last_undegraded":"2024-03-06T07:19:20.517467+0000","last_fullsized":"2024-03-06T07:19:20.517467+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:01.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455255+0000","last_change":"2024-03-06T07:19:19.468349+0000","last_active":"2024-03-06T07:19:20.455255+0000","last_peered":"2024-03-06T07:19:20.455255+0000","last_clean":"2024-03-06T07:19:20.455255+0000","last_became_active":"2024-03-06T07:19:19.468099+0000","last_became_peered":"2024-03-06T07:19:19.468099+0000","last_unstale":"2024-03-06T07:19:20.455255+0000","last_undegraded":"2024-03-06T07:19:20.455255+0000","last_fullsized":"2024-03-06T07:19:20.455255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:53.750394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517976+0000","last_change":"2024-03-06T07:19:18.461540+0000","last_active":"2024-03-06T07:19:20.517976+0000","last_peered":"2024-03-06T07:19:20.517976+0000","last_clean":"2024-03-06T07:19:20.517976+0000","last_became_active":"2024-03-06T07:19:18.461280+0000","last_became_peered":"2024-03-06T07:19:18.461280+0000","last_unstale":"2024-03-06T07:19:20.517976+0000","last_undegraded":"2024-03-06T07:19:20.517976+0000","last_fullsized":"2024-03-06T07:19:20.517976+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:10.728058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518118+0000","last_change":"2024-03-06T07:19:19.486563+0000","last_active":"2024-03-06T07:19:20.518118+0000","last_peered":"2024-03-06T07:19:20.518118+0000","last_clean":"2024-03-06T07:19:20.518118+0000","last_became_active":"2024-03-06T07:19:19.486031+0000","last_became_peered":"2024-03-06T07:19:19.486031+0000","last_unstale":"2024-03-06T07:19:20.518118+0000","last_undegraded":"2024-03-06T07:19:20.518118+0000","last_fullsized":"2024-03-06T07:19:20.518118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:20.349340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459587+0000","last_change":"2024-03-06T07:19:18.460778+0000","last_active":"2024-03-06T07:19:20.459587+0000","last_peered":"2024-03-06T07:19:20.459587+0000","last_clean":"2024-03-06T07:19:20.459587+0000","last_became_active":"2024-03-06T07:19:18.460477+0000","last_became_peered":"2024-03-06T07:19:18.460477+0000","last_unstale":"2024-03-06T07:19:20.459587+0000","last_undegraded":"2024-03-06T07:19:20.459587+0000","last_fullsized":"2024-03-06T07:19:20.459587+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:10.185504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459482+0000","last_change":"2024-03-06T07:19:19.465545+0000","last_active":"2024-03-06T07:19:20.459482+0000","last_peered":"2024-03-06T07:19:20.459482+0000","last_clean":"2024-03-06T07:19:20.459482+0000","last_became_active":"2024-03-06T07:19:19.464920+0000","last_became_peered":"2024-03-06T07:19:19.464920+0000","last_unstale":"2024-03-06T07:19:20.459482+0000","last_undegraded":"2024-03-06T07:19:20.459482+0000","last_fullsized":"2024-03-06T07:19:20.459482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:49.153689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518774+0000","last_change":"2024-03-06T07:19:18.454070+0000","last_active":"2024-03-06T07:19:20.518774+0000","last_peered":"2024-03-06T07:19:20.518774+0000","last_clean":"2024-03-06T07:19:20.518774+0000","last_became_active":"2024-03-06T07:19:18.453701+0000","last_became_peered":"2024-03-06T07:19:18.453701+0000","last_unstale":"2024-03-06T07:19:20.518774+0000","last_undegraded":"2024-03-06T07:19:20.518774+0000","last_fullsized":"2024-03-06T07:19:20.518774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:35.507247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518356+0000","last_change":"2024-03-06T07:19:19.462534+0000","last_active":"2024-03-06T07:19:20.518356+0000","last_peered":"2024-03-06T07:19:20.518356+0000","last_clean":"2024-03-06T07:19:20.518356+0000","last_became_active":"2024-03-06T07:19:19.462229+0000","last_became_peered":"2024-03-06T07:19:19.462229+0000","last_unstale":"2024-03-06T07:19:20.518356+0000","last_undegraded":"2024-03-06T07:19:20.518356+0000","last_fullsized":"2024-03-06T07:19:20.518356+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:02.350645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517820+0000","last_change":"2024-03-06T07:19:18.458305+0000","last_active":"2024-03-06T07:19:20.517820+0000","last_peered":"2024-03-06T07:19:20.517820+0000","last_clean":"2024-03-06T07:19:20.517820+0000","last_became_active":"2024-03-06T07:19:18.458021+0000","last_became_peered":"2024-03-06T07:19:18.458021+0000","last_unstale":"2024-03-06T07:19:20.517820+0000","last_undegraded":"2024-03-06T07:19:20.517820+0000","last_fullsized":"2024-03-06T07:19:20.517820+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:01.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517587+0000","last_change":"2024-03-06T07:19:19.471040+0000","last_active":"2024-03-06T07:19:20.517587+0000","last_peered":"2024-03-06T07:19:20.517587+0000","last_clean":"2024-03-06T07:19:20.517587+0000","last_became_active":"2024-03-06T07:19:19.470776+0000","last_became_peered":"2024-03-06T07:19:19.470776+0000","last_unstale":"2024-03-06T07:19:20.517587+0000","last_undegraded":"2024-03-06T07:19:20.517587+0000","last_fullsized":"2024-03-06T07:19:20.517587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:49.632443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517538+0000","last_change":"2024-03-06T07:19:18.456802+0000","last_active":"2024-03-06T07:19:20.517538+0000","last_peered":"2024-03-06T07:19:20.517538+0000","last_clean":"2024-03-06T07:19:20.517538+0000","last_became_active":"2024-03-06T07:19:18.456520+0000","last_became_peered":"2024-03-06T07:19:18.456520+0000","last_unstale":"2024-03-06T07:19:20.517538+0000","last_undegraded":"2024-03-06T07:19:20.517538+0000","last_fullsized":"2024-03-06T07:19:20.517538+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:53.898086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517580+0000","last_change":"2024-03-06T07:19:19.479806+0000","last_active":"2024-03-06T07:19:20.517580+0000","last_peered":"2024-03-06T07:19:20.517580+0000","last_clean":"2024-03-06T07:19:20.517580+0000","last_became_active":"2024-03-06T07:19:19.479545+0000","last_became_peered":"2024-03-06T07:19:19.479545+0000","last_unstale":"2024-03-06T07:19:20.517580+0000","last_undegraded":"2024-03-06T07:19:20.517580+0000","last_fullsized":"2024-03-06T07:19:20.517580+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:57.621686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517588+0000","last_change":"2024-03-06T07:19:18.454202+0000","last_active":"2024-03-06T07:19:20.517588+0000","last_peered":"2024-03-06T07:19:20.517588+0000","last_clean":"2024-03-06T07:19:20.517588+0000","last_became_active":"2024-03-06T07:19:18.453883+0000","last_became_peered":"2024-03-06T07:19:18.453883+0000","last_unstale":"2024-03-06T07:19:20.517588+0000","last_undegraded":"2024-03-06T07:19:20.517588+0000","last_fullsized":"2024-03-06T07:19:20.517588+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:49.319183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517976+0000","last_change":"2024-03-06T07:19:19.485139+0000","last_active":"2024-03-06T07:19:20.517976+0000","last_peered":"2024-03-06T07:19:20.517976+0000","last_clean":"2024-03-06T07:19:20.517976+0000","last_became_active":"2024-03-06T07:19:19.484849+0000","last_became_peered":"2024-03-06T07:19:19.484849+0000","last_unstale":"2024-03-06T07:19:20.517976+0000","last_undegraded":"2024-03-06T07:19:20.517976+0000","last_fullsized":"2024-03-06T07:19:20.517976+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:43.428024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518094+0000","last_change":"2024-03-06T07:19:18.461603+0000","last_active":"2024-03-06T07:19:20.518094+0000","last_peered":"2024-03-06T07:19:20.518094+0000","last_clean":"2024-03-06T07:19:20.518094+0000","last_became_active":"2024-03-06T07:19:18.461292+0000","last_became_peered":"2024-03-06T07:19:18.461292+0000","last_unstale":"2024-03-06T07:19:20.518094+0000","last_undegraded":"2024-03-06T07:19:20.518094+0000","last_fullsized":"2024-03-06T07:19:20.518094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:10.728058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455315+0000","last_change":"2024-03-06T07:19:19.472934+0000","last_active":"2024-03-06T07:19:20.455315+0000","last_peered":"2024-03-06T07:19:20.455315+0000","last_clean":"2024-03-06T07:19:20.455315+0000","last_became_active":"2024-03-06T07:19:19.472586+0000","last_became_peered":"2024-03-06T07:19:19.472586+0000","last_unstale":"2024-03-06T07:19:20.455315+0000","last_undegraded":"2024-03-06T07:19:20.455315+0000","last_fullsized":"2024-03-06T07:19:20.455315+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:22.650100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518778+0000","last_change":"2024-03-06T07:19:18.456792+0000","last_active":"2024-03-06T07:19:20.518778+0000","last_peered":"2024-03-06T07:19:20.518778+0000","last_clean":"2024-03-06T07:19:20.518778+0000","last_became_active":"2024-03-06T07:19:18.456501+0000","last_became_peered":"2024-03-06T07:19:18.456501+0000","last_unstale":"2024-03-06T07:19:20.518778+0000","last_undegraded":"2024-03-06T07:19:20.518778+0000","last_fullsized":"2024-03-06T07:19:20.518778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:55.160145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517741+0000","last_change":"2024-03-06T07:19:19.480521+0000","last_active":"2024-03-06T07:19:20.517741+0000","last_peered":"2024-03-06T07:19:20.517741+0000","last_clean":"2024-03-06T07:19:20.517741+0000","last_became_active":"2024-03-06T07:19:19.480268+0000","last_became_peered":"2024-03-06T07:19:19.480268+0000","last_unstale":"2024-03-06T07:19:20.517741+0000","last_undegraded":"2024-03-06T07:19:20.517741+0000","last_fullsized":"2024-03-06T07:19:20.517741+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:16.329067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518139+0000","last_change":"2024-03-06T07:19:18.464383+0000","last_active":"2024-03-06T07:19:20.518139+0000","last_peered":"2024-03-06T07:19:20.518139+0000","last_clean":"2024-03-06T07:19:20.518139+0000","last_became_active":"2024-03-06T07:19:18.464139+0000","last_became_peered":"2024-03-06T07:19:18.464139+0000","last_unstale":"2024-03-06T07:19:20.518139+0000","last_undegraded":"2024-03-06T07:19:20.518139+0000","last_fullsized":"2024-03-06T07:19:20.518139+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:26.678549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518631+0000","last_change":"2024-03-06T07:19:19.478404+0000","last_active":"2024-03-06T07:19:20.518631+0000","last_peered":"2024-03-06T07:19:20.518631+0000","last_clean":"2024-03-06T07:19:20.518631+0000","last_became_active":"2024-03-06T07:19:19.478170+0000","last_became_peered":"2024-03-06T07:19:19.478170+0000","last_unstale":"2024-03-06T07:19:20.518631+0000","last_undegraded":"2024-03-06T07:19:20.518631+0000","last_fullsized":"2024-03-06T07:19:20.518631+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:16.507323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:21.471464+0000","last_change":"2024-03-06T07:19:18.460459+0000","last_active":"2024-03-06T07:19:21.471464+0000","last_peered":"2024-03-06T07:19:21.471464+0000","last_clean":"2024-03-06T07:19:21.471464+0000","last_became_active":"2024-03-06T07:19:18.460161+0000","last_became_peered":"2024-03-06T07:19:18.460161+0000","last_unstale":"2024-03-06T07:19:21.471464+0000","last_undegraded":"2024-03-06T07:19:21.471464+0000","last_fullsized":"2024-03-06T07:19:21.471464+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:44.228476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455347+0000","last_change":"2024-03-06T07:19:19.475539+0000","last_active":"2024-03-06T07:19:20.455347+0000","last_peered":"2024-03-06T07:19:20.455347+0000","last_clean":"2024-03-06T07:19:20.455347+0000","last_became_active":"2024-03-06T07:19:19.475289+0000","last_became_peered":"2024-03-06T07:19:19.475289+0000","last_unstale":"2024-03-06T07:19:20.455347+0000","last_undegraded":"2024-03-06T07:19:20.455347+0000","last_fullsized":"2024-03-06T07:19:20.455347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:46.144500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518143+0000","last_change":"2024-03-06T07:19:18.459050+0000","last_active":"2024-03-06T07:19:20.518143+0000","last_peered":"2024-03-06T07:19:20.518143+0000","last_clean":"2024-03-06T07:19:20.518143+0000","last_became_active":"2024-03-06T07:19:18.458708+0000","last_became_peered":"2024-03-06T07:19:18.458708+0000","last_unstale":"2024-03-06T07:19:20.518143+0000","last_undegraded":"2024-03-06T07:19:20.518143+0000","last_fullsized":"2024-03-06T07:19:20.518143+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:41.022537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518431+0000","last_change":"2024-03-06T07:19:19.464821+0000","last_active":"2024-03-06T07:19:20.518431+0000","last_peered":"2024-03-06T07:19:20.518431+0000","last_clean":"2024-03-06T07:19:20.518431+0000","last_became_active":"2024-03-06T07:19:19.464562+0000","last_became_peered":"2024-03-06T07:19:19.464562+0000","last_unstale":"2024-03-06T07:19:20.518431+0000","last_undegraded":"2024-03-06T07:19:20.518431+0000","last_fullsized":"2024-03-06T07:19:20.518431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:32.348021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518325+0000","last_change":"2024-03-06T07:19:18.455189+0000","last_active":"2024-03-06T07:19:20.518325+0000","last_peered":"2024-03-06T07:19:20.518325+0000","last_clean":"2024-03-06T07:19:20.518325+0000","last_became_active":"2024-03-06T07:19:18.454881+0000","last_became_peered":"2024-03-06T07:19:18.454881+0000","last_unstale":"2024-03-06T07:19:20.518325+0000","last_undegraded":"2024-03-06T07:19:20.518325+0000","last_fullsized":"2024-03-06T07:19:20.518325+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:12.746497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517756+0000","last_change":"2024-03-06T07:19:19.464675+0000","last_active":"2024-03-06T07:19:20.517756+0000","last_peered":"2024-03-06T07:19:20.517756+0000","last_clean":"2024-03-06T07:19:20.517756+0000","last_became_active":"2024-03-06T07:19:19.464419+0000","last_became_peered":"2024-03-06T07:19:19.464419+0000","last_unstale":"2024-03-06T07:19:20.517756+0000","last_undegraded":"2024-03-06T07:19:20.517756+0000","last_fullsized":"2024-03-06T07:19:20.517756+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:49.153689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.458798+0000","last_change":"2024-03-06T07:19:18.453866+0000","last_active":"2024-03-06T07:19:20.458798+0000","last_peered":"2024-03-06T07:19:20.458798+0000","last_clean":"2024-03-06T07:19:20.458798+0000","last_became_active":"2024-03-06T07:19:18.453580+0000","last_became_peered":"2024-03-06T07:19:18.453580+0000","last_unstale":"2024-03-06T07:19:20.458798+0000","last_undegraded":"2024-03-06T07:19:20.458798+0000","last_fullsized":"2024-03-06T07:19:20.458798+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:12.746497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.459454+0000","last_change":"2024-03-06T07:19:19.469415+0000","last_active":"2024-03-06T07:19:20.459454+0000","last_peered":"2024-03-06T07:19:20.459454+0000","last_clean":"2024-03-06T07:19:20.459454+0000","last_became_active":"2024-03-06T07:19:19.469114+0000","last_became_peered":"2024-03-06T07:19:19.469114+0000","last_unstale":"2024-03-06T07:19:20.459454+0000","last_undegraded":"2024-03-06T07:19:20.459454+0000","last_fullsized":"2024-03-06T07:19:20.459454+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:16.329067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.458545+0000","last_change":"2024-03-06T07:19:18.458921+0000","last_active":"2024-03-06T07:19:20.458545+0000","last_peered":"2024-03-06T07:19:20.458545+0000","last_clean":"2024-03-06T07:19:20.458545+0000","last_became_active":"2024-03-06T07:19:18.458650+0000","last_became_peered":"2024-03-06T07:19:18.458650+0000","last_unstale":"2024-03-06T07:19:20.458545+0000","last_undegraded":"2024-03-06T07:19:20.458545+0000","last_fullsized":"2024-03-06T07:19:20.458545+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:12.746497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518464+0000","last_change":"2024-03-06T07:19:19.477145+0000","last_active":"2024-03-06T07:19:20.518464+0000","last_peered":"2024-03-06T07:19:20.518464+0000","last_clean":"2024-03-06T07:19:20.518464+0000","last_became_active":"2024-03-06T07:19:19.476890+0000","last_became_peered":"2024-03-06T07:19:19.476890+0000","last_unstale":"2024-03-06T07:19:20.518464+0000","last_undegraded":"2024-03-06T07:19:20.518464+0000","last_fullsized":"2024-03-06T07:19:20.518464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:22.303368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.455059+0000","last_change":"2024-03-06T07:19:18.455670+0000","last_active":"2024-03-06T07:19:20.455059+0000","last_peered":"2024-03-06T07:19:20.455059+0000","last_clean":"2024-03-06T07:19:20.455059+0000","last_became_active":"2024-03-06T07:19:18.455225+0000","last_became_peered":"2024-03-06T07:19:18.455225+0000","last_unstale":"2024-03-06T07:19:20.455059+0000","last_undegraded":"2024-03-06T07:19:20.455059+0000","last_fullsized":"2024-03-06T07:19:20.455059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:48.149705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518228+0000","last_change":"2024-03-06T07:19:19.486601+0000","last_active":"2024-03-06T07:19:20.518228+0000","last_peered":"2024-03-06T07:19:20.518228+0000","last_clean":"2024-03-06T07:19:20.518228+0000","last_became_active":"2024-03-06T07:19:19.486282+0000","last_became_peered":"2024-03-06T07:19:19.486282+0000","last_unstale":"2024-03-06T07:19:20.518228+0000","last_undegraded":"2024-03-06T07:19:20.518228+0000","last_fullsized":"2024-03-06T07:19:20.518228+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:18.439515+0000","last_clean_scrub_stamp":"2024-03-06T07:19:18.439515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:38.738970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.518581+0000","last_change":"2024-03-06T07:19:18.455321+0000","last_active":"2024-03-06T07:19:20.518581+0000","last_peered":"2024-03-06T07:19:20.518581+0000","last_clean":"2024-03-06T07:19:20.518581+0000","last_became_active":"2024-03-06T07:19:18.455065+0000","last_became_peered":"2024-03-06T07:19:18.455065+0000","last_unstale":"2024-03-06T07:19:20.518581+0000","last_undegraded":"2024-03-06T07:19:20.518581+0000","last_fullsized":"2024-03-06T07:19:20.518581+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:17.435531+0000","last_clean_scrub_stamp":"2024-03-06T07:19:17.435531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:11.369524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":206,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-06T07:19:20.517534+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:19:20.517534+0000","last_peered":"2024-03-06T07:19:20.517534+0000","last_clean":"2024-03-06T07:19:20.517534+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:19:20.517534+0000","last_undegraded":"2024-03-06T07:19:20.517534+0000","last_fullsized":"2024-03-06T07:19:20.517534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":787777,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":196309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":195839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":196331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":197373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":195797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":195817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":786129,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:19:22.773 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:19:22.773 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:19:22.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.915+0000 7ff1b5dc7640 1 Processor -- start 2024-03-06T07:19:22.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.915+0000 7ff1b5dc7640 1 -- start start 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.915+0000 7ff1b5dc7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1b00c5990 0x7ff1b00c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.915+0000 7ff1b5dc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1b5dc7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff1b006c730 0x7ff1b01523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1b5dc7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b006cbe0 con 0x7ff1b00c5990 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1b5dc7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b00c3ef0 con 0x7ff1b006c730 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1b5dc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01556d0 con 0x7ff1b00ab170 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1b00c5990 0x7ff1b00c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1b00c5990 0x7ff1b00c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50942/0 (socket says 172.21.15.94:50942) 2024-03-06T07:19:22.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 -- 172.21.15.94:0/3378208519 learned_addr learned my addr 172.21.15.94:0/3378208519 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1aeffd640 1 --2- 172.21.15.94:0/3378208519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1affff640 1 --2- 172.21.15.94:0/3378208519 >> v2:172.21.15.145:3300/0 conn(0x7ff1b006c730 0x7ff1b01523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 -- 172.21.15.94:0/3378208519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 msgr2=0x7ff1b006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 --2- 172.21.15.94:0/3378208519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b006c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 -- 172.21.15.94:0/3378208519 >> v2:172.21.15.145:3300/0 conn(0x7ff1b006c730 msgr2=0x7ff1b01523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 --2- 172.21.15.94:0/3378208519 >> v2:172.21.15.145:3300/0 conn(0x7ff1b006c730 0x7ff1b01523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.916+0000 7ff1af7fe640 1 -- 172.21.15.94:0/3378208519 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b01559b0 con 0x7ff1b00c5990 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1af7fe640 1 --2- 172.21.15.94:0/3378208519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b00c0ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00c4100 tx=0x7ff1a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22f03e4f097c119 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1ae7fc640 1 -- 172.21.15.94:0/3378208519 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0035020 con 0x7ff1b00c5990 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1ae7fc640 1 -- 172.21.15.94:0/3378208519 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a0031640 con 0x7ff1b00c5990 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1ae7fc640 1 -- 172.21.15.94:0/3378208519 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a002d040 con 0x7ff1b00c5990 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/3378208519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 msgr2=0x7ff1b00c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/3378208519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b00c0ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff1b00c4100 tx=0x7ff1a002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/3378208519 shutdown_connections 2024-03-06T07:19:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/3378208519 >> v2:172.21.15.145:3300/0 conn(0x7ff1b006c730 0x7ff1b01523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/3378208519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b006c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/3378208519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b00c0ab0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.917+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/3378208519 >> 172.21.15.94:0/3378208519 conn(0x7ff1b00c6710 msgr2=0x7ff1b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.918+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/3378208519 shutdown_connections 2024-03-06T07:19:22.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.918+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/3378208519 wait complete. 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 Processor -- start 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 -- start start 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1b006c730 0x7ff1b00c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b0144d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b01452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0156490 con 0x7ff1b00c5990 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0155f70 con 0x7ff1b006c730 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.919+0000 7ff1b5dc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0156610 con 0x7ff1b00ab170 2024-03-06T07:19:22.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b01452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1af7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1b006c730 0x7ff1b00c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b01452a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50944/0 (socket says 172.21.15.94:50944) 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 -- 172.21.15.94:0/4047894596 learned_addr learned my addr 172.21.15.94:0/4047894596 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1aeffd640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b0144d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 -- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 msgr2=0x7ff1b0144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b0144d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 -- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1b006c730 msgr2=0x7ff1b00c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1b006c730 0x7ff1b00c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 -- 172.21.15.94:0/4047894596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b0148be0 con 0x7ff1b00c5990 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1af7fe640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1b006c730 0x7ff1b00c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1affff640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b01452a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009dd0 tx=0x7ff1a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff197fff640 1 -- 172.21.15.94:0/4047894596 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8018020 con 0x7ff1b00c5990 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff197fff640 1 -- 172.21.15.94:0/4047894596 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a8014480 con 0x7ff1b00c5990 2024-03-06T07:19:22.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b00c2e40 con 0x7ff1b00c5990 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff197fff640 1 -- 172.21.15.94:0/4047894596 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8014870 con 0x7ff1b00c5990 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.920+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b00c3400 con 0x7ff1b00c5990 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.921+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b00c2c30 con 0x7ff1b00c5990 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.922+0000 7ff197fff640 1 -- 172.21.15.94:0/4047894596 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1a8005d10 con 0x7ff1b00c5990 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.922+0000 7ff197fff640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19c06b4d0 0x7ff19c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.922+0000 7ff197fff640 1 -- 172.21.15.94:0/4047894596 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff1a808adf0 con 0x7ff1b00c5990 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.922+0000 7ff1af7fe640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19c06b4d0 0x7ff19c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:22.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.923+0000 7ff1af7fe640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19c06b4d0 0x7ff19c06d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0036000 tx=0x7ff1a003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:22.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:22.926+0000 7ff197fff640 1 -- 172.21.15.94:0/4047894596 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1a8022070 con 0x7ff1b00c5990 2024-03-06T07:19:23.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.162+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff1b004f330 con 0x7ff1b00c5990 2024-03-06T07:19:23.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.164+0000 7ff197fff640 1 -- 172.21.15.94:0/4047894596 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff1a8010040 con 0x7ff1b00c5990 2024-03-06T07:19:23.166 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:19:23.166 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:19:23.166 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:19:23.166 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:19:23.167 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:19:23.167 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:19:23.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19c06b4d0 msgr2=0x7ff19c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19c06b4d0 0x7ff19c06d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0036000 tx=0x7ff1a003c040 comp rx=0 tx=0).stop 2024-03-06T07:19:23.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 msgr2=0x7ff1b01452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b01452a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009dd0 tx=0x7ff1a800c490 comp rx=0 tx=0).stop 2024-03-06T07:19:23.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 shutdown_connections 2024-03-06T07:19:23.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19c06b4d0 0x7ff19c06d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1b00c5990 0x7ff1b01452a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1b00ab170 0x7ff1b0144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 --2- 172.21.15.94:0/4047894596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1b006c730 0x7ff1b00c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 >> 172.21.15.94:0/4047894596 conn(0x7ff1b00c6710 msgr2=0x7ff1b014c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:23.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 shutdown_connections 2024-03-06T07:19:23.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.166+0000 7ff1b5dc7640 1 -- 172.21.15.94:0/4047894596 wait complete. 2024-03-06T07:19:23.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:23.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.277+0000 7fc65a113640 1 Processor -- start 2024-03-06T07:19:23.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.277+0000 7fc65a113640 1 -- start start 2024-03-06T07:19:23.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc65a113640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc6540c3370 0x7fc654155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc65a113640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc654156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc65a113640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6540c5490 con 0x7fc6540c3750 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc65a113640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6540c5770 con 0x7fc6540c3370 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc65a113640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6540c5a50 con 0x7fc6540a5e60 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6537fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc6540c3370 0x7fc654155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6537fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc6540c3370 0x7fc654155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50952/0 (socket says 172.21.15.94:50952) 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6537fe640 1 -- 172.21.15.94:0/449658731 learned_addr learned my addr 172.21.15.94:0/449658731 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc652ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc6540c3750 0x7fc6540c1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43648/0 (socket says 172.21.15.94:43648) 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc653fff640 1 --2- 172.21.15.94:0/449658731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc654156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6537fe640 1 -- 172.21.15.94:0/449658731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 msgr2=0x7fc654156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6537fe640 1 --2- 172.21.15.94:0/449658731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc654156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6537fe640 1 -- 172.21.15.94:0/449658731 >> v1:172.21.15.94:6789/0 conn(0x7fc6540c3750 legacy=0x7fc6540c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6537fe640 1 -- 172.21.15.94:0/449658731 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540c5d30 con 0x7fc6540c3370 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.278+0000 7fc6527fc640 1 -- 172.21.15.94:0/449658731 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894017820 0 0) 0x7fc6540c5490 con 0x7fc6540c3750 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.279+0000 7fc6537fe640 1 --2- 172.21.15.94:0/449658731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3370 0x7fc654155ad0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc644009870 tx=0x7fc64402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642462c391114ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.279+0000 7fc6527fc640 1 -- 172.21.15.94:0/449658731 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6440317c0 con 0x7fc6540c3370 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.279+0000 7fc6527fc640 1 -- 172.21.15.94:0/449658731 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc644037070 con 0x7fc6540c3370 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.279+0000 7fc6527fc640 1 -- 172.21.15.94:0/449658731 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644031b80 con 0x7fc6540c3370 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 -- 172.21.15.94:0/449658731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3370 msgr2=0x7fc654155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 --2- 172.21.15.94:0/449658731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3370 0x7fc654155ad0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc644009870 tx=0x7fc64402b860 comp rx=0 tx=0).stop 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 -- 172.21.15.94:0/449658731 shutdown_connections 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 --2- 172.21.15.94:0/449658731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc654156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 --2- 172.21.15.94:0/449658731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3370 0x7fc654155ad0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 -- 172.21.15.94:0/449658731 >> 172.21.15.94:0/449658731 conn(0x7fc6540c6710 msgr2=0x7fc6540c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 -- 172.21.15.94:0/449658731 shutdown_connections 2024-03-06T07:19:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.280+0000 7fc65a113640 1 -- 172.21.15.94:0/449658731 wait complete. 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.281+0000 7fc65a113640 1 Processor -- start 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.281+0000 7fc65a113640 1 -- start start 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc65414f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6540c3370 0x7fc65406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3750 0x7fc65414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc654159910 con 0x7fc6540c3750 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc654159790 con 0x7fc6540c3370 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6540c4ed0 con 0x7fc6540a5e60 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3750 0x7fc65414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3750 0x7fc65414ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50968/0 (socket says 172.21.15.94:50968) 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 -- 172.21.15.94:0/1678114260 learned_addr learned my addr 172.21.15.94:0/1678114260 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc652ffd640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6540c3370 0x7fc65406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc6537fe640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc65414f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 -- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 msgr2=0x7fc65414f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc65414f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 -- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6540c3370 msgr2=0x7fc65406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6540c3370 0x7fc65406cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 -- 172.21.15.94:0/1678114260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65406d530 con 0x7fc6540c3750 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc6537fe640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc65414f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc653fff640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3750 0x7fc65414ffd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00efc0 tx=0x7fc64c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc63bfff640 1 -- 172.21.15.94:0/1678114260 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c018020 con 0x7fc6540c3750 2024-03-06T07:19:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc63bfff640 1 -- 172.21.15.94:0/1678114260 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c014480 con 0x7fc6540c3750 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65406d7d0 con 0x7fc6540c3750 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc63bfff640 1 -- 172.21.15.94:0/1678114260 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c014870 con 0x7fc6540c3750 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.282+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6541534c0 con 0x7fc6540c3750 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.283+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65404f330 con 0x7fc6540c3750 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.284+0000 7fc63bfff640 1 -- 172.21.15.94:0/1678114260 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc64c01b020 con 0x7fc6540c3750 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.284+0000 7fc63bfff640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc64006b4d0 0x7fc64006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.284+0000 7fc63bfff640 1 -- 172.21.15.94:0/1678114260 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc64c08b160 con 0x7fc6540c3750 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.285+0000 7fc6537fe640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc64006b4d0 0x7fc64006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.285+0000 7fc6537fe640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc64006b4d0 0x7fc64006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc644036000 tx=0x7fc64403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.289+0000 7fc63bfff640 1 -- 172.21.15.94:0/1678114260 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc64c022020 con 0x7fc6540c3750 2024-03-06T07:19:23.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.498+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc654068450 con 0x7fc6540c3750 2024-03-06T07:19:23.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.498+0000 7fc63bfff640 1 -- 172.21.15.94:0/1678114260 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fc64c010040 con 0x7fc6540c3750 2024-03-06T07:19:23.501 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:23.501 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:20.809451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34820":{"gid":34820,"name":"b","rank":0,"incarnation":15,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/1305657307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1305657307},{"type":"v1","addr":"172.21.15.145:6837","nonce":1305657307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34820,"qdb_cluster":[34820]},"id":2}]} 2024-03-06T07:19:23.501 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 17 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc64006b4d0 msgr2=0x7fc64006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc64006b4d0 0x7fc64006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc644036000 tx=0x7fc64403d040 comp rx=0 tx=0).stop 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3750 msgr2=0x7fc65414ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3750 0x7fc65414ffd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00efc0 tx=0x7fc64c00c490 comp rx=0 tx=0).stop 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 shutdown_connections 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc64006b4d0 0x7fc64006d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6540c3750 0x7fc65414ffd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6540c3370 0x7fc65406cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 --2- 172.21.15.94:0/1678114260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6540a5e60 0x7fc65414f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.500+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 >> 172.21.15.94:0/1678114260 conn(0x7fc6540c6710 msgr2=0x7fc6540c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.501+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 shutdown_connections 2024-03-06T07:19:23.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:23.501+0000 7fc65a113640 1 -- 172.21.15.94:0/1678114260 wait complete. 2024-03-06T07:19:23.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:19:18.801003+0000', 'modified': '2024-03-06T07:19:20.809451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34820}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34820': {'gid': 34820, 'name': 'b', 'rank': 0, 'incarnation': 15, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/1305657307', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 1305657307}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1305657307}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34820, 'qdb_cluster': [34820]} 2024-03-06T07:19:23.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:19:23.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:19:24.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.016+0000 7fab49d93640 1 Processor -- start 2024-03-06T07:19:24.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.016+0000 7fab49d93640 1 -- start start 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab49d93640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab4406d680 0x7fab44152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab49d93640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fab440c28f0 0x7fab4414e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab49d93640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab440c1140 con 0x7fab44072f40 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab49d93640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab440c1d50 con 0x7fab440c28f0 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab49d93640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab441525a0 con 0x7fab4406d680 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab43fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fab440c28f0 0x7fab4414e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab42ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fab44072f40 0x7fab440c21e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43658/0 (socket says 172.21.15.94:43658) 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab42ffd640 1 -- 172.21.15.94:0/2221465193 learned_addr learned my addr 172.21.15.94:0/2221465193 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.017+0000 7fab437fe640 1 --2- 172.21.15.94:0/2221465193 >> v2:172.21.15.145:3300/0 conn(0x7fab4406d680 0x7fab44152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.018+0000 7fab43fff640 1 -- 172.21.15.94:0/2221465193 >> v2:172.21.15.145:3300/0 conn(0x7fab4406d680 msgr2=0x7fab44152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.018+0000 7fab43fff640 1 --2- 172.21.15.94:0/2221465193 >> v2:172.21.15.145:3300/0 conn(0x7fab4406d680 0x7fab44152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.018+0000 7fab43fff640 1 -- 172.21.15.94:0/2221465193 >> v1:172.21.15.94:6789/0 conn(0x7fab44072f40 legacy=0x7fab440c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.018+0000 7fab427fc640 1 -- 172.21.15.94:0/2221465193 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344000762 0 0) 0x7fab440c1140 con 0x7fab44072f40 2024-03-06T07:19:24.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.018+0000 7fab43fff640 1 -- 172.21.15.94:0/2221465193 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab440c1420 con 0x7fab440c28f0 2024-03-06T07:19:24.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.018+0000 7fab43fff640 1 --2- 172.21.15.94:0/2221465193 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414e440 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00aed0 tx=0x7fab3c02cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5dca173189e3d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab427fc640 1 -- 172.21.15.94:0/2221465193 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c02dab0 con 0x7fab440c28f0 2024-03-06T07:19:24.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab427fc640 1 -- 172.21.15.94:0/2221465193 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c02dc50 con 0x7fab440c28f0 2024-03-06T07:19:24.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab427fc640 1 -- 172.21.15.94:0/2221465193 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c008440 con 0x7fab440c28f0 2024-03-06T07:19:24.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 -- 172.21.15.94:0/2221465193 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 msgr2=0x7fab4414e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 --2- 172.21.15.94:0/2221465193 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414e440 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fab3c00aed0 tx=0x7fab3c02cf10 comp rx=0 tx=0).stop 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 -- 172.21.15.94:0/2221465193 shutdown_connections 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 --2- 172.21.15.94:0/2221465193 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414e440 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 --2- 172.21.15.94:0/2221465193 >> v2:172.21.15.145:3300/0 conn(0x7fab4406d680 0x7fab44152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 -- 172.21.15.94:0/2221465193 >> 172.21.15.94:0/2221465193 conn(0x7fab4406b1a0 msgr2=0x7fab4406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 -- 172.21.15.94:0/2221465193 shutdown_connections 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.019+0000 7fab49d93640 1 -- 172.21.15.94:0/2221465193 wait complete. 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.020+0000 7fab49d93640 1 Processor -- start 2024-03-06T07:19:24.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.020+0000 7fab49d93640 1 -- start start 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab49d93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab4406d680 0x7fab440bb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab49d93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab44072f40 0x7fab441e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab49d93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab49d93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440c2d80 con 0x7fab440c28f0 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab49d93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440c19e0 con 0x7fab4406d680 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab49d93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4406da60 con 0x7fab44072f40 2024-03-06T07:19:24.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50986/0 (socket says 172.21.15.94:50986) 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 -- 172.21.15.94:0/327557095 learned_addr learned my addr 172.21.15.94:0/327557095 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab437fe640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab4406d680 0x7fab440bb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab42ffd640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab44072f40 0x7fab441e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 -- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab44072f40 msgr2=0x7fab441e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab44072f40 0x7fab441e8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 -- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab4406d680 msgr2=0x7fab440bb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab4406d680 0x7fab440bb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.021+0000 7fab43fff640 1 -- 172.21.15.94:0/327557095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab441d9800 con 0x7fab440c28f0 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.022+0000 7fab43fff640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414daf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fab3c037040 tx=0x7fab3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.022+0000 7fab2bfff640 1 -- 172.21.15.94:0/327557095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c044020 con 0x7fab440c28f0 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.022+0000 7fab2bfff640 1 -- 172.21.15.94:0/327557095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c040470 con 0x7fab440c28f0 2024-03-06T07:19:24.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.022+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab441db9b0 con 0x7fab440c28f0 2024-03-06T07:19:24.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.022+0000 7fab2bfff640 1 -- 172.21.15.94:0/327557095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c00a560 con 0x7fab440c28f0 2024-03-06T07:19:24.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.022+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab441dbf70 con 0x7fab440c28f0 2024-03-06T07:19:24.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.024+0000 7fab2bfff640 1 -- 172.21.15.94:0/327557095 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab3c047020 con 0x7fab440c28f0 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.024+0000 7fab2bfff640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab3006b4d0 0x7fab3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.024+0000 7fab2bfff640 1 -- 172.21.15.94:0/327557095 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fab3c0acd20 con 0x7fab440c28f0 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.024+0000 7fab437fe640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab3006b4d0 0x7fab3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab437fe640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab3006b4d0 0x7fab3006d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab34009b40 tx=0x7fab34009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 Processor -- start 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 -- start start 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x56201970ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201970cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970d4b0 0x562019709680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620196736b0 con 0x56201970d4b0 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56201961ca60 con 0x56201970f330 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.025+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56201966f750 con 0x56201970ef50 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970d4b0 0x562019709680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab437fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x56201970ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab437fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x56201970ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:50146/0 (socket says 172.21.15.94:50146) 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970d4b0 0x562019709680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50992/0 (socket says 172.21.15.94:50992) 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab437fe640 1 -- 172.21.15.94:0/3288292420 learned_addr learned my addr 172.21.15.94:0/3288292420 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab42ffd640 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201970cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 -- 172.21.15.94:0/3288292420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 msgr2=0x56201970ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x56201970ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 -- 172.21.15.94:0/3288292420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 msgr2=0x56201970cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201970cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 -- 172.21.15.94:0/3288292420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56201963af30 con 0x56201970d4b0 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab43fff640 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970d4b0 0x562019709680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0ab1a0 tx=0x7fab3c07af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6665027f49a5b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab297fa640 1 -- 172.21.15.94:0/3288292420 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c07f4f0 con 0x56201970d4b0 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab297fa640 1 -- 172.21.15.94:0/3288292420 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c083ab0 con 0x56201970d4b0 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.026+0000 7fab297fa640 1 -- 172.21.15.94:0/3288292420 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c077ac0 con 0x56201970d4b0 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 -- 172.21.15.94:0/3288292420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970d4b0 msgr2=0x562019709680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970d4b0 0x562019709680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0ab1a0 tx=0x7fab3c07af90 comp rx=0 tx=0).stop 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 -- 172.21.15.94:0/3288292420 shutdown_connections 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970d4b0 0x562019709680 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201970cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 --2- 172.21.15.94:0/3288292420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x56201970ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 -- 172.21.15.94:0/3288292420 >> 172.21.15.94:0/3288292420 conn(0x56201970b7d0 msgr2=0x5620197303c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 -- 172.21.15.94:0/3288292420 shutdown_connections 2024-03-06T07:19:24.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 -- 172.21.15.94:0/3288292420 wait complete. 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 -- start start 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.027+0000 7fab4c97f740 1 Processor -- start 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970ef50 0x56201972f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970f330 0x56201975b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562019733980 0x56201972bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab4c97f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620196736b0 con 0x56201970f330 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56201962fea0 con 0x56201970ef50 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x562019731230 con 0x562019733980 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab4c97f740 10 client.0 _ll_register_callbacks cb 0x56201962f870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970f330 0x56201975b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970f330 0x56201975b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51002/0 (socket says 172.21.15.94:51002) 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 -- 172.21.15.94:0/4239937780 learned_addr learned my addr 172.21.15.94:0/4239937780 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab437fe640 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970ef50 0x56201972f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab43fff640 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562019733980 0x56201972bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 -- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562019733980 msgr2=0x56201972bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562019733980 0x56201972bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 -- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970ef50 msgr2=0x56201972f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970ef50 0x56201972f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.028+0000 7fab42ffd640 1 -- 172.21.15.94:0/4239937780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562019630270 con 0x56201970f330 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab42ffd640 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970f330 0x56201975b560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fab38004ee0 tx=0x7fab38001680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccdb69f77ebbdfc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab227fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab227fc640 1 -- 172.21.15.94:0/4239937780 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38015650 con 0x56201970f330 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab227fc640 1 -- 172.21.15.94:0/4239937780 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fab38010040 con 0x56201970f330 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab3c083ab0 con 0x56201970f330 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab227fc640 1 -- 172.21.15.94:0/4239937780 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab380157f0 con 0x56201970f330 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 10 client.5364 fetch_fsmap learned FSMap version 17 2024-03-06T07:19:24.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5620193882d0 con 0x56201970f330 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab227fc640 1 -- 172.21.15.94:0/4239937780 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c083ab0 con 0x56201970f330 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab227fc640 1 -- 172.21.15.94:0/4239937780 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fab38015990 con 0x56201970f330 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 10 client.5364 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 10 client.5364 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 20 client.5364 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x56201975f3e0 0x56201976a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 4 client.5364 mds_command: new command op to 34820 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 --> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562019680bf0 con 0x56201975f3e0 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.029+0000 7fab437fe640 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x56201975f3e0 0x56201976a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.030+0000 7fab437fe640 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x56201975f3e0 0x56201976a110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c260f705ce7c904 server_cookie=f99e10ea0a5c73d2 in_seq=0 out_seq=0 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.030+0000 7fab227fc640 10 client.5364 ms_handle_connect on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.030+0000 7fab227fc640 1 -- 172.21.15.94:0/4239937780 <== mds.0 v2:172.21.15.145:6836/1305657307 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562019680bf0 con 0x56201975f3e0 2024-03-06T07:19:24.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.030+0000 7fab227fc640 10 client.5364 handle_command_reply: tid=0 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.030+0000 7fab4c97f740 1 client.5364 shutdown 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970f330 msgr2=0x56201975b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970f330 0x56201975b560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fab38004ee0 tx=0x7fab38001680 comp rx=0 tx=0).stop 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 shutdown_connections 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x56201975f3e0 0x56201976a110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562019733980 0x56201972bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab227fc640 0 client.5364 ms_handle_reset on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970f330 0x56201975b560 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 --2- 172.21.15.94:0/4239937780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970ef50 0x56201972f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 >> 172.21.15.94:0/4239937780 conn(0x56201970d4b0 msgr2=0x56201970d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 shutdown_connections 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 1 -- 172.21.15.94:0/4239937780 wait complete. 2024-03-06T07:19:24.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.031+0000 7fab4c97f740 20 client.5364 trim_cache size 0 max 16384 2024-03-06T07:19:24.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.047+0000 7fab4c97f740 1 Processor -- start 2024-03-06T07:19:24.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.047+0000 7fab4c97f740 1 -- start start 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970ef50 0x562019709300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970f330 0x562019708510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562019733980 0x562019708a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab4c97f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab441f4a00 con 0x56201970ef50 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab441f3e30 con 0x562019733980 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab441f4400 con 0x56201970f330 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970ef50 0x562019709300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970ef50 0x562019709300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51018/0 (socket says 172.21.15.94:51018) 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 -- 172.21.15.94:0/2693178049 learned_addr learned my addr 172.21.15.94:0/2693178049 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab43fff640 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562019733980 0x562019708a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab42ffd640 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970f330 0x562019708510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 -- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970f330 msgr2=0x562019708510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970f330 0x562019708510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 -- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562019733980 msgr2=0x562019708a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562019733980 0x562019708a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 -- 172.21.15.94:0/2693178049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56201970a0b0 con 0x56201970ef50 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab43fff640 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562019733980 0x562019708a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.048+0000 7fab437fe640 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970ef50 0x562019709300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fab34011690 tx=0x7fab34018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eab0344601c6701 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab28ff9640 1 -- 172.21.15.94:0/2693178049 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34032070 con 0x56201970ef50 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab28ff9640 1 -- 172.21.15.94:0/2693178049 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34004410 con 0x56201970ef50 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab28ff9640 1 -- 172.21.15.94:0/2693178049 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34003d00 con 0x56201970ef50 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 -- 172.21.15.94:0/2693178049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970ef50 msgr2=0x562019709300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970ef50 0x562019709300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fab34011690 tx=0x7fab34018000 comp rx=0 tx=0).stop 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 -- 172.21.15.94:0/2693178049 shutdown_connections 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562019733980 0x562019708a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970f330 0x562019708510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 --2- 172.21.15.94:0/2693178049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56201970ef50 0x562019709300 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 -- 172.21.15.94:0/2693178049 >> 172.21.15.94:0/2693178049 conn(0x56201970d4b0 msgr2=0x562019704ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 -- 172.21.15.94:0/2693178049 shutdown_connections 2024-03-06T07:19:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.049+0000 7fab4c97f740 1 -- 172.21.15.94:0/2693178049 wait complete. 2024-03-06T07:19:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 -- start start 2024-03-06T07:19:24.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 Processor -- start 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x5620197bac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201974f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562019733980 0x562019735570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x562019730030 con 0x562019733980 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x562019734e40 con 0x56201970f330 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56201975bb70 con 0x56201970ef50 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.050+0000 7fab4c97f740 10 client.0 _ll_register_callbacks cb 0x7fab380047e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562019733980 0x562019735570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562019733980 0x562019735570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51034/0 (socket says 172.21.15.94:51034) 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 -- 172.21.15.94:0/1905210480 learned_addr learned my addr 172.21.15.94:0/1905210480 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab437fe640 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x5620197bac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab42ffd640 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201974f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 -- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 msgr2=0x5620197bac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x5620197bac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 -- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 msgr2=0x56201974f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201974f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 -- 172.21.15.94:0/1905210480 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620194526b0 con 0x562019733980 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab43fff640 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562019733980 0x562019735570 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fab3c02d520 tx=0x7fab3c04b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84bf10a024e84b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab227fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab227fc640 1 -- 172.21.15.94:0/1905210480 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c07f4f0 con 0x562019733980 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab227fc640 1 -- 172.21.15.94:0/1905210480 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fab3c077ac0 con 0x562019733980 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab34004410 con 0x562019733980 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab227fc640 1 -- 172.21.15.94:0/1905210480 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab3c083ab0 con 0x562019733980 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab4c97f740 10 client.5370 fetch_fsmap learned FSMap version 17 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x562019709940 con 0x562019733980 2024-03-06T07:19:24.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab227fc640 1 -- 172.21.15.94:0/1905210480 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34004410 con 0x562019733980 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab227fc640 1 -- 172.21.15.94:0/1905210480 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fab3c07f120 con 0x562019733980 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.051+0000 7fab4c97f740 10 client.5370 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 10 client.5370 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 20 client.5370 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x5620197b85b0 0x5620197be180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 4 client.5370 mds_command: new command op to 34820 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 --> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56201967af40 con 0x5620197b85b0 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab437fe640 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x5620197b85b0 0x5620197be180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab227fc640 10 client.5370 ms_handle_connect on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab437fe640 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x5620197b85b0 0x5620197be180 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4c612999cc8b37d server_cookie=722e82c4ce5e5950 in_seq=0 out_seq=0 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab227fc640 1 -- 172.21.15.94:0/1905210480 <== mds.0 v2:172.21.15.145:6836/1305657307 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x56201967af40 con 0x5620197b85b0 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab227fc640 10 client.5370 handle_command_reply: tid=0 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 client.5370 shutdown 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562019733980 msgr2=0x562019735570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562019733980 0x562019735570 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fab3c02d520 tx=0x7fab3c04b070 comp rx=0 tx=0).stop 2024-03-06T07:19:24.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 shutdown_connections 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x5620197b85b0 0x5620197be180 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562019733980 0x562019735570 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56201970f330 0x56201974f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab227fc640 0 client.5370 ms_handle_reset on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 --2- 172.21.15.94:0/1905210480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56201970ef50 0x5620197bac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.052+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 >> 172.21.15.94:0/1905210480 conn(0x56201970d4b0 msgr2=0x56201972ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 shutdown_connections 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab4c97f740 1 -- 172.21.15.94:0/1905210480 wait complete. 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab4c97f740 20 client.5370 trim_cache size 0 max 16384 2024-03-06T07:19:24.062 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 34820, 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.5871935050000001, 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 17, 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 39, 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 39, 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 12.534944292 2024-03-06T07:19:24.063 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:19:24.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab3006b4d0 msgr2=0x7fab3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab3006b4d0 0x7fab3006d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab34009b40 tx=0x7fab34009190 comp rx=0 tx=0).stop 2024-03-06T07:19:24.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 msgr2=0x7fab4414daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414daf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fab3c037040 tx=0x7fab3c004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:24.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 shutdown_connections 2024-03-06T07:19:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab3006b4d0 0x7fab3006d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab440c28f0 0x7fab4414daf0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab44072f40 0x7fab441e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 --2- 172.21.15.94:0/327557095 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab4406d680 0x7fab440bb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 >> 172.21.15.94:0/327557095 conn(0x7fab4406b1a0 msgr2=0x7fab4406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 shutdown_connections 2024-03-06T07:19:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.053+0000 7fab49d93640 1 -- 172.21.15.94:0/327557095 wait complete. 2024-03-06T07:19:24.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:24.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.164+0000 7f178066b640 1 Processor -- start 2024-03-06T07:19:24.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.164+0000 7f178066b640 1 -- start start 2024-03-06T07:19:24.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f178066b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17780aa180 0x7f177814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f178066b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f177814f140 0x7f17780c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f178066b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1778155bb0 con 0x7f17781550c0 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f178066b640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780c4610 con 0x7f177814f140 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f178066b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177814f5f0 con 0x7f17780aa180 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177e407640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f17781550c0 0x7f17781554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43660/0 (socket says 172.21.15.94:43660) 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177e407640 1 -- 172.21.15.94:0/3875303057 learned_addr learned my addr 172.21.15.94:0/3875303057 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177ec08640 1 --2- 172.21.15.94:0/3875303057 >> v2:172.21.15.145:3300/0 conn(0x7f177814f140 0x7f17780c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177dc06640 1 --2- 172.21.15.94:0/3875303057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17780aa180 0x7f177814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177d405640 1 -- 172.21.15.94:0/3875303057 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885443357 0 0) 0x7f1778155bb0 con 0x7f17781550c0 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177d405640 1 -- 172.21.15.94:0/3875303057 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1764003520 con 0x7f17781550c0 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177dc06640 1 -- 172.21.15.94:0/3875303057 >> v2:172.21.15.145:3300/0 conn(0x7f177814f140 msgr2=0x7f17780c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177dc06640 1 --2- 172.21.15.94:0/3875303057 >> v2:172.21.15.145:3300/0 conn(0x7f177814f140 0x7f17780c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177dc06640 1 -- 172.21.15.94:0/3875303057 >> v1:172.21.15.94:6789/0 conn(0x7f17781550c0 legacy=0x7f17781554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.165+0000 7f177dc06640 1 -- 172.21.15.94:0/3875303057 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780aa660 con 0x7f17780aa180 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.166+0000 7f177dc06640 1 --2- 172.21.15.94:0/3875303057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17780aa180 0x7f177814ec00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17680097a0 tx=0x7f176802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edc7382f2d5a109f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.166+0000 7f177d405640 1 -- 172.21.15.94:0/3875303057 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768038070 con 0x7f17780aa180 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.166+0000 7f177d405640 1 -- 172.21.15.94:0/3875303057 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1768031700 con 0x7f17780aa180 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.166+0000 7f177d405640 1 -- 172.21.15.94:0/3875303057 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768031ab0 con 0x7f17780aa180 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 -- 172.21.15.94:0/3875303057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17780aa180 msgr2=0x7f177814ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 --2- 172.21.15.94:0/3875303057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17780aa180 0x7f177814ec00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17680097a0 tx=0x7f176802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 -- 172.21.15.94:0/3875303057 shutdown_connections 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 --2- 172.21.15.94:0/3875303057 >> v2:172.21.15.145:3300/0 conn(0x7f177814f140 0x7f17780c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 --2- 172.21.15.94:0/3875303057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17780aa180 0x7f177814ec00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 -- 172.21.15.94:0/3875303057 >> 172.21.15.94:0/3875303057 conn(0x7f17780c6710 msgr2=0x7f17780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 -- 172.21.15.94:0/3875303057 shutdown_connections 2024-03-06T07:19:24.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.167+0000 7f178066b640 1 -- 172.21.15.94:0/3875303057 wait complete. 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 Processor -- start 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 -- start start 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17780aa180 0x7f177813b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f177814f140 0x7f17780726d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17781550c0 0x7f1778072c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780c1310 con 0x7f17780aa180 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780c4860 con 0x7f17781550c0 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.168+0000 7f178066b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780c1490 con 0x7f177814f140 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17780aa180 0x7f177813b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17780aa180 0x7f177813b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51046/0 (socket says 172.21.15.94:51046) 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 -- 172.21.15.94:0/4124496499 learned_addr learned my addr 172.21.15.94:0/4124496499 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177ec08640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17781550c0 0x7f1778072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177dc06640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f177814f140 0x7f17780726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 -- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f177814f140 msgr2=0x7f17780726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f177814f140 0x7f17780726d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 -- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17781550c0 msgr2=0x7f1778072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17781550c0 0x7f1778072c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 -- 172.21.15.94:0/4124496499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177806da20 con 0x7f17780aa180 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f177e407640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17780aa180 0x7f177813b110 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f177400c3c0 tx=0x7f177400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f176effd640 1 -- 172.21.15.94:0/4124496499 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1774017070 con 0x7f17780aa180 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177806dd00 con 0x7f17780aa180 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f176effd640 1 -- 172.21.15.94:0/4124496499 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1774005ce0 con 0x7f17780aa180 2024-03-06T07:19:24.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f176effd640 1 -- 172.21.15.94:0/4124496499 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17740072f0 con 0x7f17780aa180 2024-03-06T07:19:24.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.169+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177806e2c0 con 0x7f17780aa180 2024-03-06T07:19:24.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.170+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177804f330 con 0x7f17780aa180 2024-03-06T07:19:24.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.171+0000 7f176effd640 1 -- 172.21.15.94:0/4124496499 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f177401a070 con 0x7f17780aa180 2024-03-06T07:19:24.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.171+0000 7f176effd640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f175c06b500 0x7f175c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.171+0000 7f176effd640 1 -- 172.21.15.94:0/4124496499 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f17740896f0 con 0x7f17780aa180 2024-03-06T07:19:24.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.171+0000 7f177dc06640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f175c06b500 0x7f175c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.172+0000 7f177dc06640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f175c06b500 0x7f175c06d9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1768035000 tx=0x7f1768008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.175+0000 7f176effd640 1 -- 172.21.15.94:0/4124496499 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1774057da0 con 0x7f17780aa180 2024-03-06T07:19:24.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.386+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f177814e1a0 con 0x7f17780aa180 2024-03-06T07:19:24.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.387+0000 7f176effd640 1 -- 172.21.15.94:0/4124496499 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f177405be40 con 0x7f17780aa180 2024-03-06T07:19:24.389 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:24.389 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:20.809451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34820":{"gid":34820,"name":"b","rank":0,"incarnation":15,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/1305657307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1305657307},{"type":"v1","addr":"172.21.15.145:6837","nonce":1305657307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34820,"qdb_cluster":[34820]},"id":2}]} 2024-03-06T07:19:24.389 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 17 2024-03-06T07:19:24.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f175c06b500 msgr2=0x7f175c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f175c06b500 0x7f175c06d9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1768035000 tx=0x7f1768008030 comp rx=0 tx=0).stop 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17780aa180 msgr2=0x7f177813b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17780aa180 0x7f177813b110 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f177400c3c0 tx=0x7f177400c880 comp rx=0 tx=0).stop 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 shutdown_connections 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f175c06b500 0x7f175c06d9c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17781550c0 0x7f1778072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f177814f140 0x7f17780726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.388+0000 7f178066b640 1 --2- 172.21.15.94:0/4124496499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17780aa180 0x7f177813b110 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.389+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 >> 172.21.15.94:0/4124496499 conn(0x7f17780c6710 msgr2=0x7f17780c16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.389+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 shutdown_connections 2024-03-06T07:19:24.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.389+0000 7f178066b640 1 -- 172.21.15.94:0/4124496499 wait complete. 2024-03-06T07:19:24.402 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:19:18.801003+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34820": { "addr": "172.21.15.145:6837/1305657307", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6836", "nonce": 1305657307, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 1305657307, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34820, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:19:20.809451+0000", "qdb_cluster": [ 34820 ], "qdb_leader": 34820, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34820 } } } ], "standbys": [ { "addr": "172.21.15.94:6835/2367914435", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 2367914435, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 2367914435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5166, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6835/1902381441", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1902381441, "type": "v2" }, { "addr": "172.21.15.145:6835", "nonce": 1902381441, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5215, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/597543014", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 597543014, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 597543014, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34814, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:19:24.402 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:19:24.402 INFO:tasks.cephfs.test_failover:current = [{'gid': 34814, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/597543014', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 597543014}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 597543014}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 34820, 'name': 'b', 'rank': 0, 'incarnation': 15, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/1305657307', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 1305657307}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1305657307}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5166, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5215, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6835/1902381441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1902381441}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 1902381441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:19:24.402 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-03-06T07:19:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.505+0000 7f5a90629640 1 Processor -- start 2024-03-06T07:19:24.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.505+0000 7f5a90629640 1 -- start start 2024-03-06T07:19:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a90629640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5a88150290 0x7f5a88150710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a90629640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a880c3d60 0x7f5a880c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a90629640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a88156f90 con 0x7f5a88072720 2024-03-06T07:19:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a90629640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a880c38d0 con 0x7f5a88150290 2024-03-06T07:19:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a90629640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a88157270 con 0x7f5a880c3d60 2024-03-06T07:19:24.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8ebc6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a880c3d60 0x7f5a880c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8ebc6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a880c3d60 0x7f5a880c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:50188/0 (socket says 172.21.15.94:50188) 2024-03-06T07:19:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/2949652539 learned_addr learned my addr 172.21.15.94:0/2949652539 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8dbc4640 1 --2- 172.21.15.94:0/2949652539 >> v2:172.21.15.145:3300/0 conn(0x7f5a88150290 0x7f5a88150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/2949652539 >> v2:172.21.15.145:3300/0 conn(0x7f5a88150290 msgr2=0x7f5a88150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8ebc6640 1 --2- 172.21.15.94:0/2949652539 >> v2:172.21.15.145:3300/0 conn(0x7f5a88150290 0x7f5a88150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/2949652539 >> v1:172.21.15.145:6789/0 conn(0x7f5a88072720 legacy=0x7f5a881564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.506+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/2949652539 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a880c3280 con 0x7f5a880c3d60 2024-03-06T07:19:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.507+0000 7f5a8ebc6640 1 --2- 172.21.15.94:0/2949652539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a880c3d60 0x7f5a880c41f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009e50 tx=0x7f5a8002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3018ab0006a322a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.507+0000 7f5a8d3c3640 1 -- 172.21.15.94:0/2949652539 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8002ca70 con 0x7f5a880c3d60 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.507+0000 7f5a8d3c3640 1 -- 172.21.15.94:0/2949652539 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8002cc10 con 0x7f5a880c3d60 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.507+0000 7f5a8d3c3640 1 -- 172.21.15.94:0/2949652539 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a800343f0 con 0x7f5a880c3d60 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 -- 172.21.15.94:0/2949652539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a880c3d60 msgr2=0x7f5a880c41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 --2- 172.21.15.94:0/2949652539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a880c3d60 0x7f5a880c41f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009e50 tx=0x7f5a8002bed0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 -- 172.21.15.94:0/2949652539 shutdown_connections 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 --2- 172.21.15.94:0/2949652539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a880c3d60 0x7f5a880c41f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 --2- 172.21.15.94:0/2949652539 >> v2:172.21.15.145:3300/0 conn(0x7f5a88150290 0x7f5a88150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 -- 172.21.15.94:0/2949652539 >> 172.21.15.94:0/2949652539 conn(0x7f5a880aa630 msgr2=0x7f5a880aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 -- 172.21.15.94:0/2949652539 shutdown_connections 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.508+0000 7f5a90629640 1 -- 172.21.15.94:0/2949652539 wait complete. 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 Processor -- start 2024-03-06T07:19:24.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 -- start start 2024-03-06T07:19:24.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a88072720 0x7f5a881495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a880c3d60 0x7f5a880adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a88150290 0x7f5a881513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a88156bd0 con 0x7f5a88150290 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8806f8f0 con 0x7f5a880c3d60 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.509+0000 7f5a90629640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a88156d50 con 0x7f5a88072720 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a88150290 0x7f5a881513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a88150290 0x7f5a881513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51048/0 (socket says 172.21.15.94:51048) 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/3924906148 learned_addr learned my addr 172.21.15.94:0/3924906148 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8e3c5640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a88072720 0x7f5a881495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8dbc4640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a880c3d60 0x7f5a880adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a88072720 msgr2=0x7f5a881495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a88072720 0x7f5a881495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a880c3d60 msgr2=0x7f5a880adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a880c3d60 0x7f5a880adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 -- 172.21.15.94:0/3924906148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8806d8b0 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a8ebc6640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a88150290 0x7f5a881513e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009e50 tx=0x7f5a80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a7effd640 1 -- 172.21.15.94:0/3924906148 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80043020 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a7effd640 1 -- 172.21.15.94:0/3924906148 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8003e450 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8806db90 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a7effd640 1 -- 172.21.15.94:0/3924906148 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80009510 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.510+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a881ee100 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.511+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8804f330 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.512+0000 7f5a7effd640 1 -- 172.21.15.94:0/3924906148 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a80046020 con 0x7f5a88150290 2024-03-06T07:19:24.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.512+0000 7f5a7effd640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.512+0000 7f5a7effd640 1 -- 172.21.15.94:0/3924906148 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5a800ac8f0 con 0x7f5a88150290 2024-03-06T07:19:24.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.512+0000 7f5a8e3c5640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.513+0000 7f5a8e3c5640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a84001780 tx=0x7f5a84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.517+0000 7f5a7effd640 1 -- 172.21.15.94:0/3924906148 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a8004d030 con 0x7f5a88150290 2024-03-06T07:19:24.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.687+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f5a8806d690 con 0x7f5a88150290 2024-03-06T07:19:24.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.690+0000 7f5a7effd640 1 -- 172.21.15.94:0/3924906148 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v12)=0 v12) v1 ==== 116+0+0 (secure 0 0 0) 0x7f5a8007afa0 con 0x7f5a88150290 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a6c06b4d0 msgr2=0x7f5a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a84001780 tx=0x7f5a84009040 comp rx=0 tx=0).stop 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a88150290 msgr2=0x7f5a881513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a88150290 0x7f5a881513e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009e50 tx=0x7f5a80004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 shutdown_connections 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a6c06b4d0 0x7f5a6c06d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a88150290 0x7f5a881513e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a880c3d60 0x7f5a880adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 --2- 172.21.15.94:0/3924906148 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a88072720 0x7f5a881495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 >> 172.21.15.94:0/3924906148 conn(0x7f5a880aa630 msgr2=0x7f5a880a9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 shutdown_connections 2024-03-06T07:19:24.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.692+0000 7f5a90629640 1 -- 172.21.15.94:0/3924906148 wait complete. 2024-03-06T07:19:24.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:24.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.805+0000 7fa21e822640 1 Processor -- start 2024-03-06T07:19:24.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.805+0000 7fa21e822640 1 -- start start 2024-03-06T07:19:24.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa21e822640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa21814f250 0x7fa2180c8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa21e822640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806c150 con 0x7fa2180c7350 2024-03-06T07:19:24.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa21e822640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21814f630 con 0x7fa218155c20 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa21e822640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21806c480 con 0x7fa21814f250 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa21cdbf640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa2180c7350 0x7fa21815fb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43664/0 (socket says 172.21.15.94:43664) 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa21cdbf640 1 -- 172.21.15.94:0/3117930343 learned_addr learned my addr 172.21.15.94:0/3117930343 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa217fff640 1 --2- 172.21.15.94:0/3117930343 >> v2:172.21.15.145:3300/0 conn(0x7fa21814f250 0x7fa2180c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa216ffd640 1 -- 172.21.15.94:0/3117930343 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673651568 0 0) 0x7fa21806c150 con 0x7fa2180c7350 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.806+0000 7fa216ffd640 1 -- 172.21.15.94:0/3117930343 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa200003520 con 0x7fa2180c7350 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.807+0000 7fa217fff640 1 -- 172.21.15.94:0/3117930343 >> v1:172.21.15.145:6789/0 conn(0x7fa218155c20 legacy=0x7fa21806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.807+0000 7fa216ffd640 1 -- 172.21.15.94:0/3117930343 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1678980411 0 0) 0x7fa200003520 con 0x7fa2180c7350 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.807+0000 7fa217fff640 1 -- 172.21.15.94:0/3117930343 >> v1:172.21.15.94:6789/0 conn(0x7fa2180c7350 legacy=0x7fa21815fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.807+0000 7fa217fff640 1 -- 172.21.15.94:0/3117930343 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180c5660 con 0x7fa21814f250 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.807+0000 7fa217fff640 1 --2- 172.21.15.94:0/3117930343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa21814f250 0x7fa2180c8840 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0097a0 tx=0x7fa20c02b570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c921405f27909742 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa216ffd640 1 -- 172.21.15.94:0/3117930343 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c037070 con 0x7fa21814f250 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa216ffd640 1 -- 172.21.15.94:0/3117930343 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c030510 con 0x7fa21814f250 2024-03-06T07:19:24.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa216ffd640 1 -- 172.21.15.94:0/3117930343 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2100019f0 con 0x7fa21814f250 2024-03-06T07:19:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa21e822640 1 -- 172.21.15.94:0/3117930343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa21814f250 msgr2=0x7fa2180c8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa21e822640 1 --2- 172.21.15.94:0/3117930343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa21814f250 0x7fa2180c8840 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0097a0 tx=0x7fa20c02b570 comp rx=0 tx=0).stop 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa21e822640 1 -- 172.21.15.94:0/3117930343 shutdown_connections 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa21e822640 1 --2- 172.21.15.94:0/3117930343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa21814f250 0x7fa2180c8840 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa21e822640 1 -- 172.21.15.94:0/3117930343 >> 172.21.15.94:0/3117930343 conn(0x7fa2180ac4d0 msgr2=0x7fa2180ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa21e822640 1 -- 172.21.15.94:0/3117930343 shutdown_connections 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.808+0000 7fa21e822640 1 -- 172.21.15.94:0/3117930343 wait complete. 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 Processor -- start 2024-03-06T07:19:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 -- start start 2024-03-06T07:19:24.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2180c7350 0x7fa2181eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa21814f250 0x7fa2181544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa218155c20 0x7fa218157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180c5c20 con 0x7fa21814f250 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21806daf0 con 0x7fa218155c20 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa21e822640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2181560a0 con 0x7fa2180c7350 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.810+0000 7fa2177fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa21814f250 0x7fa2181544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa21814f250 0x7fa2181544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51062/0 (socket says 172.21.15.94:51062) 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 -- 172.21.15.94:0/294998749 learned_addr learned my addr 172.21.15.94:0/294998749 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa21cdbf640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa218155c20 0x7fa218157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa217fff640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2180c7350 0x7fa2181eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 -- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2180c7350 msgr2=0x7fa2181eb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2180c7350 0x7fa2181eb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 -- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa218155c20 msgr2=0x7fa218157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa218155c20 0x7fa218157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:24.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 -- 172.21.15.94:0/294998749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2181eea50 con 0x7fa21814f250 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa2177fe640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa21814f250 0x7fa2181544b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c550 tx=0x7fa20800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa214ff9640 1 -- 172.21.15.94:0/294998749 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208017070 con 0x7fa21814f250 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa214ff9640 1 -- 172.21.15.94:0/294998749 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2080035f0 con 0x7fa21814f250 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2181eaa40 con 0x7fa21814f250 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa214ff9640 1 -- 172.21.15.94:0/294998749 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2080039e0 con 0x7fa21814f250 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.811+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2181eecf0 con 0x7fa21814f250 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.812+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21804f330 con 0x7fa21814f250 2024-03-06T07:19:24.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.813+0000 7fa214ff9640 1 -- 172.21.15.94:0/294998749 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa20801a070 con 0x7fa21814f250 2024-03-06T07:19:24.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.813+0000 7fa214ff9640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:24.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.813+0000 7fa214ff9640 1 -- 172.21.15.94:0/294998749 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa208089780 con 0x7fa21814f250 2024-03-06T07:19:24.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.813+0000 7fa217fff640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:24.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.814+0000 7fa217fff640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa20406b4d0 0x7fa20406d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa20c034000 tx=0x7fa20c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:24.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:24.817+0000 7fa214ff9640 1 -- 172.21.15.94:0/294998749 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa208057e20 con 0x7fa21814f250 2024-03-06T07:19:25.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.027+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa21806d860 con 0x7fa21814f250 2024-03-06T07:19:25.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.028+0000 7fa214ff9640 1 -- 172.21.15.94:0/294998749 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fa20805bec0 con 0x7fa21814f250 2024-03-06T07:19:25.030 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:25.030 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5166,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:20.809451+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":34820},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34820":{"gid":34820,"name":"b","rank":0,"incarnation":15,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/1305657307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1305657307},{"type":"v1","addr":"172.21.15.145:6837","nonce":1305657307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34820,"qdb_cluster":[34820]},"id":2}]} 2024-03-06T07:19:25.030 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 17 2024-03-06T07:19:25.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa20406b4d0 msgr2=0x7fa20406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa20406b4d0 0x7fa20406d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa20c034000 tx=0x7fa20c008030 comp rx=0 tx=0).stop 2024-03-06T07:19:25.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa21814f250 msgr2=0x7fa2181544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa21814f250 0x7fa2181544b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa20800c550 tx=0x7fa20800ca10 comp rx=0 tx=0).stop 2024-03-06T07:19:25.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 shutdown_connections 2024-03-06T07:19:25.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa20406b4d0 0x7fa20406d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa218155c20 0x7fa218157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa21814f250 0x7fa2181544b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 --2- 172.21.15.94:0/294998749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2180c7350 0x7fa2181eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 >> 172.21.15.94:0/294998749 conn(0x7fa2180ac4d0 msgr2=0x7fa21814aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:25.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 shutdown_connections 2024-03-06T07:19:25.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.030+0000 7fa21e822640 1 -- 172.21.15.94:0/294998749 wait complete. 2024-03-06T07:19:25.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:19:18.801003+0000', 'modified': '2024-03-06T07:19:20.809451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 34820}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_34820': {'gid': 34820, 'name': 'b', 'rank': 0, 'incarnation': 15, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/1305657307', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 1305657307}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1305657307}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34820, 'qdb_cluster': [34820]} 2024-03-06T07:19:25.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:19:25.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:19:25.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.547+0000 7fa6bc842640 1 Processor -- start 2024-03-06T07:19:25.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.547+0000 7fa6bc842640 1 -- start start 2024-03-06T07:19:25.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6bc842640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa6b8158fb0 0x7fa6b80c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6bc842640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b81553c0 con 0x7fa6b806c240 2024-03-06T07:19:25.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6bc842640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b81556a0 con 0x7fa6b80ab860 2024-03-06T07:19:25.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6bc842640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b8155980 con 0x7fa6b8158fb0 2024-03-06T07:19:25.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa6b806c240 0x7fa6b80ab150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43678/0 (socket says 172.21.15.94:43678) 2024-03-06T07:19:25.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b659c640 1 -- 172.21.15.94:0/178049041 learned_addr learned my addr 172.21.15.94:0/178049041 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/178049041 >> v2:172.21.15.145:3300/0 conn(0x7fa6b8158fb0 0x7fa6b80c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b559a640 1 -- 172.21.15.94:0/178049041 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609526551 0 0) 0x7fa6b81553c0 con 0x7fa6b806c240 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b559a640 1 -- 172.21.15.94:0/178049041 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0003520 con 0x7fa6b806c240 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b559a640 1 -- 172.21.15.94:0/178049041 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383832828 0 0) 0x7fa6b81556a0 con 0x7fa6b80ab860 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b559a640 1 -- 172.21.15.94:0/178049041 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b81553c0 con 0x7fa6b80ab860 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/178049041 >> v1:172.21.15.145:6789/0 conn(0x7fa6b80ab860 legacy=0x7fa6b81588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/178049041 >> v1:172.21.15.94:6789/0 conn(0x7fa6b806c240 legacy=0x7fa6b80ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.548+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/178049041 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8155c60 con 0x7fa6b8158fb0 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.549+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/178049041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8158fb0 0x7fa6b80c4250 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8009d90 tx=0x7fa6a802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa8c4f1519095733 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.549+0000 7fa6b559a640 1 -- 172.21.15.94:0/178049041 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8039070 con 0x7fa6b8158fb0 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.549+0000 7fa6b559a640 1 -- 172.21.15.94:0/178049041 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a802cc60 con 0x7fa6b8158fb0 2024-03-06T07:19:25.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.549+0000 7fa6b559a640 1 -- 172.21.15.94:0/178049041 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8034660 con 0x7fa6b8158fb0 2024-03-06T07:19:25.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.550+0000 7fa6bc842640 1 -- 172.21.15.94:0/178049041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8158fb0 msgr2=0x7fa6b80c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.550+0000 7fa6bc842640 1 --2- 172.21.15.94:0/178049041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8158fb0 0x7fa6b80c4250 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8009d90 tx=0x7fa6a802beb0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.550+0000 7fa6bc842640 1 -- 172.21.15.94:0/178049041 shutdown_connections 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.550+0000 7fa6bc842640 1 --2- 172.21.15.94:0/178049041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8158fb0 0x7fa6b80c4250 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.550+0000 7fa6bc842640 1 -- 172.21.15.94:0/178049041 >> 172.21.15.94:0/178049041 conn(0x7fa6b80c7670 msgr2=0x7fa6b80c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.550+0000 7fa6bc842640 1 -- 172.21.15.94:0/178049041 shutdown_connections 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.550+0000 7fa6bc842640 1 -- 172.21.15.94:0/178049041 wait complete. 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.551+0000 7fa6bc842640 1 Processor -- start 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.551+0000 7fa6bc842640 1 -- start start 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6bc842640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b806c240 0x7fa6b81edab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6bc842640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6b80ab860 0x7fa6b8071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6bc842640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8158fb0 0x7fa6b81498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6bc842640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b80c4830 con 0x7fa6b8158fb0 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6bc842640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b8156180 con 0x7fa6b806c240 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6bc842640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b806c620 con 0x7fa6b80ab860 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8158fb0 0x7fa6b81498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8158fb0 0x7fa6b81498a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51066/0 (socket says 172.21.15.94:51066) 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/1031030606 learned_addr learned my addr 172.21.15.94:0/1031030606 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b659c640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b806c240 0x7fa6b81edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6b80ab860 0x7fa6b8071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6b80ab860 msgr2=0x7fa6b8071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6b80ab860 0x7fa6b8071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b806c240 msgr2=0x7fa6b81edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b806c240 0x7fa6b81edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.552+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/1031030606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8155040 con 0x7fa6b8158fb0 2024-03-06T07:19:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.553+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8158fb0 0x7fa6b81498a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8009eb0 tx=0x7fa6a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.553+0000 7fa6a6ffd640 1 -- 172.21.15.94:0/1031030606 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8039050 con 0x7fa6b8158fb0 2024-03-06T07:19:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.553+0000 7fa6a6ffd640 1 -- 172.21.15.94:0/1031030606 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8045070 con 0x7fa6b8158fb0 2024-03-06T07:19:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.553+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b8072da0 con 0x7fa6b8158fb0 2024-03-06T07:19:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.553+0000 7fa6a6ffd640 1 -- 172.21.15.94:0/1031030606 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8009450 con 0x7fa6b8158fb0 2024-03-06T07:19:25.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.553+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b80716a0 con 0x7fa6b8158fb0 2024-03-06T07:19:25.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.554+0000 7fa6a6ffd640 1 -- 172.21.15.94:0/1031030606 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6a803e830 con 0x7fa6b8158fb0 2024-03-06T07:19:25.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.554+0000 7fa6a6ffd640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa69806b4d0 0x7fa69806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.555+0000 7fa6a6ffd640 1 -- 172.21.15.94:0/1031030606 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6a80ad060 con 0x7fa6b8158fb0 2024-03-06T07:19:25.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.555+0000 7fa6b659c640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa69806b4d0 0x7fa69806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.555+0000 7fa6b659c640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa69806b4d0 0x7fa69806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0000f60 tx=0x7fa6b0007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.555+0000 7fa6bf420740 1 Processor -- start 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6bf420740 1 -- start start 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5c99a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5c99f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9a4b0 0x55ece5c96680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6bf420740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ece5c006b0 con 0x55ece5c9c330 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ece5ba9a60 con 0x55ece5c9bf50 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ece5bfc750 con 0x55ece5c9a4b0 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6b5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5c99f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.556+0000 7fa6b5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5c99f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51078/0 (socket says 172.21.15.94:51078) 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/87551387 learned_addr learned my addr 172.21.15.94:0/87551387 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9a4b0 0x55ece5c96680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b659c640 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5c99a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9a4b0 msgr2=0x55ece5c96680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9a4b0 0x55ece5c96680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 msgr2=0x55ece5c99a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5c99a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/87551387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ece5bc7f30 con 0x55ece5c9c330 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5c99f70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80abcb0 tx=0x7fa6ac007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb664a0b837858f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa697fff640 1 -- 172.21.15.94:0/87551387 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac0052e0 con 0x55ece5c9c330 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa697fff640 1 -- 172.21.15.94:0/87551387 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac018070 con 0x55ece5c9c330 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.557+0000 7fa697fff640 1 -- 172.21.15.94:0/87551387 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac00f9b0 con 0x55ece5c9c330 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 -- 172.21.15.94:0/87551387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 msgr2=0x55ece5c99f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5c99f70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80abcb0 tx=0x7fa6ac007d20 comp rx=0 tx=0).stop 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 -- 172.21.15.94:0/87551387 shutdown_connections 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9a4b0 0x55ece5c96680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5c99f70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 --2- 172.21.15.94:0/87551387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5c99a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 -- 172.21.15.94:0/87551387 >> 172.21.15.94:0/87551387 conn(0x55ece5c987d0 msgr2=0x55ece5cbd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 -- 172.21.15.94:0/87551387 shutdown_connections 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 -- 172.21.15.94:0/87551387 wait complete. 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 -- start start 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.558+0000 7fa6bf420740 1 Processor -- start 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cdba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5cbfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5cbff90 0x55ece5cb8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6bf420740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ece5cbe230 con 0x55ece5c9c330 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ece5bbcea0 con 0x55ece5c9bf50 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ece5cbdc70 con 0x55ece5cbff90 2024-03-06T07:19:25.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6bf420740 10 client.0 _ll_register_callbacks cb 0x55ece5bbc870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6b5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5cbfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6b5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5cbfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51094/0 (socket says 172.21.15.94:51094) 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/3111972489 learned_addr learned my addr 172.21.15.94:0/3111972489 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cdba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.559+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5cbff90 0x55ece5cb8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5cbff90 msgr2=0x55ece5cb8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5cbff90 0x55ece5cb8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 msgr2=0x55ece5cdba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cdba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/3111972489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ece5bbd270 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5cbfa50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac007fd0 tx=0x7fa6ac000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2863e574cb7901b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6957fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6957fa640 1 -- 172.21.15.94:0/3111972489 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac00fb40 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6957fa640 1 -- 172.21.15.94:0/3111972489 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6ac00fce0 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6ac018070 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6957fa640 1 -- 172.21.15.94:0/3111972489 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6ac02d050 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 10 client.5397 fetch_fsmap learned FSMap version 17 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55ece59152d0 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6957fa640 1 -- 172.21.15.94:0/3111972489 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac018070 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6957fa640 1 -- 172.21.15.94:0/3111972489 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa6ac030020 con 0x55ece5c9c330 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 10 client.5397 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 10 client.5397 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 20 client.5397 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:25.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5cec350 0x55ece5cf72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 4 client.5397 mds_command: new command op to 34820 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.560+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 --> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ece5c0dbf0 con 0x55ece5cec350 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.561+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5cec350 0x55ece5cf72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.561+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5cec350 0x55ece5cf72a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbbbb8a673c3f519 server_cookie=ac2e1fdb5fa545c1 in_seq=0 out_seq=0 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.561+0000 7fa6957fa640 10 client.5397 ms_handle_connect on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6957fa640 1 -- 172.21.15.94:0/3111972489 <== mds.0 v2:172.21.15.145:6836/1305657307 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ece5c0dbf0 con 0x55ece5cec350 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6957fa640 10 client.5397 handle_command_reply: tid=0 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 client.5397 shutdown 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 msgr2=0x55ece5cbfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5cbfa50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac007fd0 tx=0x7fa6ac000ed0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 shutdown_connections 2024-03-06T07:19:25.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5cec350 0x55ece5cf72a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5cbff90 0x55ece5cb8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5c9c330 0x55ece5cbfa50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3111972489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cdba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6957fa640 0 client.5397 ms_handle_reset on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 >> 172.21.15.94:0/3111972489 conn(0x55ece5c9a4b0 msgr2=0x55ece5c9a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 shutdown_connections 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 1 -- 172.21.15.94:0/3111972489 wait complete. 2024-03-06T07:19:25.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.562+0000 7fa6bf420740 20 client.5397 trim_cache size 0 max 16384 2024-03-06T07:19:25.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.578+0000 7fa6bf420740 1 Processor -- start 2024-03-06T07:19:25.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.578+0000 7fa6bf420740 1 -- start start 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cc1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5c95510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c95a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6bf420740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81f6c20 con 0x55ece5cbff90 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81f6020 con 0x55ece5c9bf50 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81f6620 con 0x55ece5c9c330 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c95a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c95a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51100/0 (socket says 172.21.15.94:51100) 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/3165380854 learned_addr learned my addr 172.21.15.94:0/3165380854 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cc1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5c95510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 msgr2=0x55ece5c95510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5c95510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 msgr2=0x55ece5cc1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cc1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/3165380854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ece59df6b0 con 0x55ece5cbff90 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cc1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.579+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c95a50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8007130 tx=0x7fa6a804c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb9293e61a3b864 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6977fe640 1 -- 172.21.15.94:0/3165380854 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a807f980 con 0x55ece5cbff90 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6977fe640 1 -- 172.21.15.94:0/3165380854 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a80a8070 con 0x55ece5cbff90 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6977fe640 1 -- 172.21.15.94:0/3165380854 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8083f40 con 0x55ece5cbff90 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 -- 172.21.15.94:0/3165380854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 msgr2=0x55ece5c95a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c95a50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8007130 tx=0x7fa6a804c040 comp rx=0 tx=0).stop 2024-03-06T07:19:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 -- 172.21.15.94:0/3165380854 shutdown_connections 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c95a50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5c95510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3165380854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cc1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 -- 172.21.15.94:0/3165380854 >> 172.21.15.94:0/3165380854 conn(0x55ece5c9a4b0 msgr2=0x55ece5c97b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 -- 172.21.15.94:0/3165380854 shutdown_connections 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.580+0000 7fa6bf420740 1 -- 172.21.15.94:0/3165380854 wait complete. 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.581+0000 7fa6bf420740 1 -- start start 2024-03-06T07:19:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.581+0000 7fa6bf420740 1 Processor -- start 2024-03-06T07:19:25.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cfb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5d53a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6bf420740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c962e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6bf420740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ece5cbed40 con 0x55ece5cbff90 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ece5cbe230 con 0x55ece5c9bf50 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6bf420740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ece5cc04b0 con 0x55ece5c9c330 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6bf420740 10 client.0 _ll_register_callbacks cb 0x7fa6ac0016e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b5d9b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5d53a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51106/0 (socket says 172.21.15.94:51106) 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b6d9d640 1 -- 172.21.15.94:0/3222527306 learned_addr learned my addr 172.21.15.94:0/3222527306 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cfb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 msgr2=0x55ece5cfb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cfb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/3222527306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 msgr2=0x55ece5c962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b5d9b640 1 -- 172.21.15.94:0/3222527306 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ece5cd5410 con 0x55ece5c9c330 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b6d9d640 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6b5d9b640 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5d53a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55ece5cd8ef0 tx=0x7fa6ac009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d55bf7dfbc78639 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6957fa640 10 client.0 ms_handle_connect on v2:172.21.15.145:3301/0 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.582+0000 7fa6957fa640 1 -- 172.21.15.94:0/3222527306 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac028800 con 0x55ece5c9c330 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6957fa640 1 -- 172.21.15.94:0/3222527306 <== mon.2 v2:172.21.15.145:3301/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6ac0289a0 con 0x55ece5c9c330 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6a80a8070 con 0x55ece5c9c330 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6957fa640 1 -- 172.21.15.94:0/3222527306 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6ac028400 con 0x55ece5c9c330 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 10 client.34949 fetch_fsmap learned FSMap version 17 2024-03-06T07:19:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55ece5cb8920 con 0x55ece5c9c330 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6957fa640 1 -- 172.21.15.94:0/3222527306 <== mon.2 v2:172.21.15.145:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a80a8070 con 0x55ece5c9c330 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6957fa640 1 -- 172.21.15.94:0/3222527306 <== mon.2 v2:172.21.15.145:3301/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa6ac030040 con 0x55ece5c9c330 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 10 client.34949 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 10 client.34949 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 20 client.34949 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5c96e20 0x55ece5d4d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 4 client.34949 mds_command: new command op to 34820 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 --> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ece5c07f40 con 0x55ece5c96e20 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5c96e20 0x55ece5d4d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6957fa640 10 client.34949 ms_handle_connect on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.583+0000 7fa6b659c640 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5c96e20 0x55ece5d4d370 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6776f769c93483b server_cookie=bbb7d569d55bbc62 in_seq=0 out_seq=0 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6957fa640 1 -- 172.21.15.94:0/3222527306 <== mds.0 v2:172.21.15.145:6836/1305657307 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55ece5c07f40 con 0x55ece5c96e20 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6957fa640 10 client.34949 handle_command_reply: tid=0 2024-03-06T07:19:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 client.34949 shutdown 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 msgr2=0x55ece5d53a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5d53a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55ece5cd8ef0 tx=0x7fa6ac009040 comp rx=0 tx=0).stop 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 shutdown_connections 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:6836/1305657307,v1:172.21.15.145:6837/1305657307] conn(0x55ece5c96e20 0x55ece5d4d370 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ece5cbff90 0x55ece5c962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ece5c9c330 0x55ece5d53a80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 --2- 172.21.15.94:0/3222527306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ece5c9bf50 0x55ece5cfb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 >> 172.21.15.94:0/3222527306 conn(0x55ece5c9a4b0 msgr2=0x55ece5cc1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6957fa640 0 client.34949 ms_handle_reset on v2:172.21.15.145:6836/1305657307 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 shutdown_connections 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 1 -- 172.21.15.94:0/3222527306 wait complete. 2024-03-06T07:19:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bf420740 20 client.34949 trim_cache size 0 max 16384 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 34820, 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 6.1185850569999998, 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 17, 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 39, 2024-03-06T07:19:25.594 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 39, 2024-03-06T07:19:25.595 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 14.066335280000001 2024-03-06T07:19:25.595 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:19:25.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa69806b4d0 msgr2=0x7fa69806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa69806b4d0 0x7fa69806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0000f60 tx=0x7fa6b0007d70 comp rx=0 tx=0).stop 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8158fb0 msgr2=0x7fa6b81498a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8158fb0 0x7fa6b81498a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8009eb0 tx=0x7fa6a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 shutdown_connections 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa69806b4d0 0x7fa69806d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8158fb0 0x7fa6b81498a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6b80ab860 0x7fa6b8071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.584+0000 7fa6bc842640 1 --2- 172.21.15.94:0/1031030606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b806c240 0x7fa6b81edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.585+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 >> 172.21.15.94:0/1031030606 conn(0x7fa6b80c7670 msgr2=0x7fa6b814d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.585+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 shutdown_connections 2024-03-06T07:19:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:25.585+0000 7fa6bc842640 1 -- 172.21.15.94:0/1031030606 wait complete. 2024-03-06T07:19:25.597 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:19:18.801003+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_34820": { "addr": "172.21.15.145:6837/1305657307", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6836", "nonce": 1305657307, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 1305657307, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34820, "incarnation": 15, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:19:20.809451+0000", "qdb_cluster": [ 34820 ], "qdb_leader": 34820, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 34820 } } } ], "standbys": [ { "addr": "172.21.15.94:6835/2367914435", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 2367914435, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 2367914435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5166, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6835/1902381441", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1902381441, "type": "v2" }, { "addr": "172.21.15.145:6835", "nonce": 1902381441, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5215, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/597543014", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 597543014, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 597543014, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34814, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:19:25.599 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 2, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:19:25.599 INFO:tasks.cephfs.test_failover:current = [{'gid': 34814, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/597543014', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 597543014}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 597543014}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 34820, 'name': 'b', 'rank': 0, 'incarnation': 15, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/1305657307', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 1305657307}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1305657307}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5166, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}, {'gid': 5215, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6835/1902381441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1902381441}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 1902381441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:19:25.599 DEBUG:tasks.cephfs.test_failover:2 != -1 2024-03-06T07:19:25.599 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:19:26.493 INFO:tasks.ceph.mds.b.smithi145.stderr: -220> 2024-03-06T07:19:22.430+0000 7f2fc752a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:19:26.518 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:19:30.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:30.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.699+0000 7f87e8a8d640 1 Processor -- start 2024-03-06T07:19:30.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.699+0000 7f87e8a8d640 1 -- start start 2024-03-06T07:19:30.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e8a8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e40c88c0 0x7f87e4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:30.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e8a8d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87e4158b60 0x7f87e4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e8a8d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e40c73d0 con 0x7f87e414f360 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e8a8d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e40c8dd0 con 0x7f87e4158b60 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e8a8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40be2e0 con 0x7f87e40c88c0 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e2d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87e4158b60 0x7f87e4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e2d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87e4158b60 0x7f87e4158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50374/0 (socket says 172.21.15.94:50374) 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e2d9d640 1 -- 172.21.15.94:0/990710194 learned_addr learned my addr 172.21.15.94:0/990710194 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e259c640 1 --1- 172.21.15.94:0/990710194 >> v1:172.21.15.94:6789/0 conn(0x7f87e414f360 0x7f87e40c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59478/0 (socket says 172.21.15.94:59478) 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87dbfff640 1 --2- 172.21.15.94:0/990710194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e40c88c0 0x7f87e4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e2d9d640 1 -- 172.21.15.94:0/990710194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e40c88c0 msgr2=0x7f87e4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e2d9d640 1 --2- 172.21.15.94:0/990710194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e40c88c0 0x7f87e4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e2d9d640 1 -- 172.21.15.94:0/990710194 >> v1:172.21.15.94:6789/0 conn(0x7f87e414f360 legacy=0x7f87e40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:30.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e1d9b640 1 -- 172.21.15.94:0/990710194 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161158359 0 0) 0x7f87e40c73d0 con 0x7f87e414f360 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.700+0000 7f87e2d9d640 1 -- 172.21.15.94:0/990710194 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e40c7670 con 0x7f87e4158b60 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e2d9d640 1 --2- 172.21.15.94:0/990710194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e4158ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009e50 tx=0x7f87dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8b5d0dc6e76df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e1d9b640 1 -- 172.21.15.94:0/990710194 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc02ca70 con 0x7f87e4158b60 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e1d9b640 1 -- 172.21.15.94:0/990710194 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc02cc10 con 0x7f87e4158b60 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e1d9b640 1 -- 172.21.15.94:0/990710194 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0343f0 con 0x7f87e4158b60 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e8a8d640 1 -- 172.21.15.94:0/990710194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 msgr2=0x7f87e4158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/990710194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e4158ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009e50 tx=0x7f87dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e8a8d640 1 -- 172.21.15.94:0/990710194 shutdown_connections 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/990710194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e4158ff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.701+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/990710194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e40c88c0 0x7f87e4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.702+0000 7f87e8a8d640 1 -- 172.21.15.94:0/990710194 >> 172.21.15.94:0/990710194 conn(0x7f87e40ac570 msgr2=0x7f87e40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.702+0000 7f87e8a8d640 1 -- 172.21.15.94:0/990710194 shutdown_connections 2024-03-06T07:19:30.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.702+0000 7f87e8a8d640 1 -- 172.21.15.94:0/990710194 wait complete. 2024-03-06T07:19:30.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 Processor -- start 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 -- start start 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87e40c88c0 0x7f87e40c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e414f360 0x7f87e414af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e414b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40c5a50 con 0x7f87e4158b60 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40c7c30 con 0x7f87e40c88c0 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.703+0000 7f87e8a8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40c5bd0 con 0x7f87e414f360 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e414b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e414b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50390/0 (socket says 172.21.15.94:50390) 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 -- 172.21.15.94:0/3304603632 learned_addr learned my addr 172.21.15.94:0/3304603632 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87dbfff640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e414f360 0x7f87e414af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e259c640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87e40c88c0 0x7f87e40c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:30.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 -- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e414f360 msgr2=0x7f87e414af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e414f360 0x7f87e414af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 -- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87e40c88c0 msgr2=0x7f87e40c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87e40c88c0 0x7f87e40c41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 -- 172.21.15.94:0/3304603632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e41eb790 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e2d9d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e414b4d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009e50 tx=0x7f87dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87daffd640 1 -- 172.21.15.94:0/3304603632 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc044030 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87daffd640 1 -- 172.21.15.94:0/3304603632 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc040470 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e40c1bc0 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87daffd640 1 -- 172.21.15.94:0/3304603632 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc009510 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.704+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e40c2180 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.705+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e404f330 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.706+0000 7f87daffd640 1 -- 172.21.15.94:0/3304603632 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87dc047070 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.706+0000 7f87daffd640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.706+0000 7f87daffd640 1 -- 172.21.15.94:0/3304603632 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f87dc0aca80 con 0x7f87e4158b60 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.706+0000 7f87e259c640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:30.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.707+0000 7f87e259c640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c406b4d0 0x7f87c406d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f87d4000f60 tx=0x7f87d4007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:30.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.710+0000 7f87daffd640 1 -- 172.21.15.94:0/3304603632 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87dc04e070 con 0x7f87e4158b60 2024-03-06T07:19:30.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.918+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e406cda0 con 0x7f87e4158b60 2024-03-06T07:19:30.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.918+0000 7f87daffd640 1 -- 172.21.15.94:0/3304603632 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f87dc07b7b0 con 0x7f87e4158b60 2024-03-06T07:19:30.920 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:30.920 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:29.497645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":2}]} 2024-03-06T07:19:30.921 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 22 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c406b4d0 msgr2=0x7f87c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c406b4d0 0x7f87c406d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f87d4000f60 tx=0x7f87d4007b50 comp rx=0 tx=0).stop 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 msgr2=0x7f87e414b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e414b4d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009e50 tx=0x7f87dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 shutdown_connections 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c406b4d0 0x7f87c406d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87e4158b60 0x7f87e414b4d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87e414f360 0x7f87e414af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 --2- 172.21.15.94:0/3304603632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87e40c88c0 0x7f87e40c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 >> 172.21.15.94:0/3304603632 conn(0x7f87e40ac570 msgr2=0x7f87e40ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 shutdown_connections 2024-03-06T07:19:30.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:30.920+0000 7f87e8a8d640 1 -- 172.21.15.94:0/3304603632 wait complete. 2024-03-06T07:19:31.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:19:18.801003+0000', 'modified': '2024-03-06T07:19:29.497645+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5166}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5166': {'gid': 5166, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5166, 'qdb_cluster': [5166]} 2024-03-06T07:19:31.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:19:31.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:19:31.421 INFO:tasks.ceph.mds.c.smithi094.stderr:2024-03-06T07:19:31.418+0000 7f75cb463640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:19:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.437+0000 7f3a80e8d640 1 Processor -- start 2024-03-06T07:19:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.437+0000 7f3a80e8d640 1 -- start start 2024-03-06T07:19:31.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.437+0000 7f3a80e8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 0x7f3a7c0ac280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.437+0000 7f3a80e8d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c1552a0 con 0x7f3a7c158fa0 2024-03-06T07:19:31.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.437+0000 7f3a80e8d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c0c4730 con 0x7f3a7c06c240 2024-03-06T07:19:31.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.437+0000 7f3a80e8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c14f7e0 con 0x7f3a7c0c4250 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a73fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3a7c158fa0 0x7f3a7c0ab150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59484/0 (socket says 172.21.15.94:59484) 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a73fff640 1 -- 172.21.15.94:0/2063886154 learned_addr learned my addr 172.21.15.94:0/2063886154 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 0x7f3a7c0ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222625357 0 0) 0x7f3a7c1552a0 con 0x7f3a7c158fa0 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68003520 con 0x7f3a7c158fa0 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441949964 0 0) 0x7f3a7c0c4730 con 0x7f3a7c06c240 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a7c1552a0 con 0x7f3a7c06c240 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069295311 0 0) 0x7f3a68003520 con 0x7f3a7c158fa0 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 msgr2=0x7f3a7c0ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 0x7f3a7c0ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 >> v1:172.21.15.145:6789/0 conn(0x7f3a7c06c240 legacy=0x7f3a7c158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c155560 con 0x7f3a7c158fa0 2024-03-06T07:19:31.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f3a64001670 con 0x7f3a7c158fa0 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 >> v1:172.21.15.94:6789/0 conn(0x7f3a7c158fa0 legacy=0x7f3a7c0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a68003b70 0x7f3a68003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 0x7f3a68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c14f7e0 con 0x7f3a68004d40 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6800bb80 con 0x7f3a68003b70 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6800bd80 con 0x7f3a7c0c4250 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a68003b70 0x7f3a68003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.438+0000 7f3a73fff640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 0x7f3a68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.439+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 msgr2=0x7f3a68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.439+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a7c0c4250 0x7f3a68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.439+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a68003b70 msgr2=0x7f3a68003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.439+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a68003b70 0x7f3a68003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.439+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2063886154 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68003520 con 0x7f3a68004d40 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.439+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0ac7c0 tx=0x7f3a68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d453a5a22dfefcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74039070 con 0x7f3a68004d40 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7402cb90 con 0x7f3a68004d40 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a79d9b640 1 -- 172.21.15.94:0/2063886154 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74009c10 con 0x7f3a68004d40 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a80e8d640 1 -- 172.21.15.94:0/2063886154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68004d40 msgr2=0x7f3a680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a80e8d640 1 --2- 172.21.15.94:0/2063886154 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a68004d40 0x7f3a680051c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0ac7c0 tx=0x7f3a68005830 comp rx=0 tx=0).stop 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2063886154 reap_dead start 2024-03-06T07:19:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a80e8d640 1 -- 172.21.15.94:0/2063886154 shutdown_connections 2024-03-06T07:19:31.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a80e8d640 1 -- 172.21.15.94:0/2063886154 >> 172.21.15.94:0/2063886154 conn(0x7f3a7c0c7670 msgr2=0x7f3a7c0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a80e8d640 1 -- 172.21.15.94:0/2063886154 shutdown_connections 2024-03-06T07:19:31.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.440+0000 7f3a80e8d640 1 -- 172.21.15.94:0/2063886154 wait complete. 2024-03-06T07:19:31.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.441+0000 7f3a80e8d640 1 Processor -- start 2024-03-06T07:19:31.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a80e8d640 1 -- start start 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a80e8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a68004d40 0x7f3a7c1ee270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a80e8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a7c072f90 0x7f3a7c14c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a80e8d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7c0ac130 0x7f3a7c14c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a80e8d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0c1c00 con 0x7f3a7c0ac130 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a80e8d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c155bb0 con 0x7f3a7c072f90 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a80e8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0c1d80 con 0x7f3a68004d40 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a7ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7c0ac130 0x7f3a7c14c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a7ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7c0ac130 0x7f3a7c14c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50418/0 (socket says 172.21.15.94:50418) 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/1741576479 learned_addr learned my addr 172.21.15.94:0/1741576479 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a68004d40 0x7f3a7c1ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.442+0000 7f3a73fff640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a7c072f90 0x7f3a7c14c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a68004d40 msgr2=0x7f3a7c1ee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a68004d40 0x7f3a7c1ee270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a7c072f90 msgr2=0x7f3a7c14c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a7c072f90 0x7f3a7c14c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/1741576479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c0ab830 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a73fff640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a7c072f90 0x7f3a7c14c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7c0ac130 0x7f3a7c14c8a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3a7402c600 tx=0x7f3a74009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a72ffd640 1 -- 172.21.15.94:0/1741576479 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74039070 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a72ffd640 1 -- 172.21.15.94:0/1741576479 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a74044070 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c1eed00 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a72ffd640 1 -- 172.21.15.94:0/1741576479 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7403e830 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.443+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c1ee7b0 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.445+0000 7f3a72ffd640 1 -- 172.21.15.94:0/1741576479 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a74005d10 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.445+0000 7f3a72ffd640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5c06b500 0x7f3a5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.445+0000 7f3a72ffd640 1 -- 172.21.15.94:0/1741576479 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3a740ac420 con 0x7f3a7c0ac130 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.445+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5c06b500 0x7f3a5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.446+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5c06b500 0x7f3a5c06d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c001830 tx=0x7f3a6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.446+0000 7f3a83a71740 1 Processor -- start 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.446+0000 7f3a83a71740 1 -- start start 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712be0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be3330 0x556712be0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be14b0 0x556712bdd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a83a71740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x556712b476b0 con 0x556712be14b0 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556712af0a60 con 0x556712be3330 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556712b43750 con 0x556712be2f50 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be14b0 0x556712bdd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be14b0 0x556712bdd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50420/0 (socket says 172.21.15.94:50420) 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2834708229 learned_addr learned my addr 172.21.15.94:0/2834708229 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a73fff640 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be3330 0x556712be0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712be0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 msgr2=0x556712be0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712be0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be3330 msgr2=0x556712be0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be3330 0x556712be0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 -- 172.21.15.94:0/2834708229 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556712b0ef30 con 0x556712be14b0 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be14b0 0x556712bdd680 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3a740095c0 tx=0x7f3a74009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9d4b952f28ca73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a5bfff640 1 -- 172.21.15.94:0/2834708229 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74083160 con 0x556712be14b0 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a5bfff640 1 -- 172.21.15.94:0/2834708229 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7403a040 con 0x556712be14b0 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.447+0000 7f3a5bfff640 1 -- 172.21.15.94:0/2834708229 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a74077170 con 0x556712be14b0 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 -- 172.21.15.94:0/2834708229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be14b0 msgr2=0x556712bdd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be14b0 0x556712bdd680 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3a740095c0 tx=0x7f3a74009cd0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 -- 172.21.15.94:0/2834708229 shutdown_connections 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be14b0 0x556712bdd680 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be3330 0x556712be0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2834708229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712be0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 -- 172.21.15.94:0/2834708229 >> 172.21.15.94:0/2834708229 conn(0x556712bdf7d0 msgr2=0x556712c043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 -- 172.21.15.94:0/2834708229 shutdown_connections 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.448+0000 7f3a83a71740 1 -- 172.21.15.94:0/2834708229 wait complete. 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.449+0000 7f3a83a71740 1 -- start start 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.449+0000 7f3a83a71740 1 Processor -- start 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be2f50 0x556712c1c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c01060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712c05690 0x556712c05b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x556712af0a60 con 0x556712be3330 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556712b03ea0 con 0x556712be2f50 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556712b476b0 con 0x556712c05690 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 10 client.0 _ll_register_callbacks cb 0x556712b03870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:31.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c01060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c01060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50428/0 (socket says 172.21.15.94:50428) 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 -- 172.21.15.94:0/2519928500 learned_addr learned my addr 172.21.15.94:0/2519928500 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712c05690 0x556712c05b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be2f50 0x556712c1c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 -- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712c05690 msgr2=0x556712c05b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712c05690 0x556712c05b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 -- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be2f50 msgr2=0x556712c1c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be2f50 0x556712c1c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 -- 172.21.15.94:0/2519928500 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556712b04270 con 0x556712be3330 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a73fff640 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c01060 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3a64004d10 tx=0x7f3a64007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919c8007acc33992 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a597fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a597fa640 1 -- 172.21.15.94:0/2519928500 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6401a070 con 0x556712be3330 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a597fa640 1 -- 172.21.15.94:0/2519928500 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3a64007560 con 0x556712be3330 2024-03-06T07:19:31.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.450+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a7403a040 con 0x556712be3330 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a597fa640 1 -- 172.21.15.94:0/2519928500 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a64026050 con 0x556712be3330 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 10 client.5424 fetch_fsmap learned FSMap version 22 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x55671285c2d0 con 0x556712be3330 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a597fa640 1 -- 172.21.15.94:0/2519928500 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7403a040 con 0x556712be3330 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a597fa640 1 -- 172.21.15.94:0/2519928500 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 22) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3a64029020 con 0x556712be3330 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 10 client.5424 fetch_fsmap finished waiting for FSMap version 22 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 10 client.5424 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 20 client.5424 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712c2f930 0x556712c2fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 4 client.5424 mds_command: new command op to 5166 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 --> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556712b54bf0 con 0x556712c2f930 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712c2f930 0x556712c2fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a597fa640 10 client.5424 ms_handle_connect on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:31.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.451+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712c2f930 0x556712c2fd10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4139ccfdee9c4347 server_cookie=f082cc7df4775f43 in_seq=0 out_seq=0 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a597fa640 1 -- 172.21.15.94:0/2519928500 <== mds.0 v2:172.21.15.94:6833/2367914435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556712b54bf0 con 0x556712c2f930 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a597fa640 10 client.5424 handle_command_reply: tid=0 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 client.5424 shutdown 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 msgr2=0x556712c01060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c01060 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3a64004d10 tx=0x7f3a64007430 comp rx=0 tx=0).stop 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 shutdown_connections 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712c2f930 0x556712c2fd10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712c05690 0x556712c05b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c01060 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a597fa640 0 client.5424 ms_handle_reset on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 --2- 172.21.15.94:0/2519928500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712be2f50 0x556712c1c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 >> 172.21.15.94:0/2519928500 conn(0x556712be14b0 msgr2=0x556712be18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 shutdown_connections 2024-03-06T07:19:31.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 1 -- 172.21.15.94:0/2519928500 wait complete. 2024-03-06T07:19:31.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.452+0000 7f3a83a71740 20 client.5424 trim_cache size 0 max 16384 2024-03-06T07:19:31.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 Processor -- start 2024-03-06T07:19:31.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 -- start start 2024-03-06T07:19:31.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712c22650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c076c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 0x556712c07c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1f84f0 con 0x556712be3330 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1f76b0 con 0x556712c05690 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c0ab3d0 con 0x556712be2f50 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.469+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c076c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c076c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50430/0 (socket says 172.21.15.94:50430) 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 -- 172.21.15.94:0/3070220673 learned_addr learned my addr 172.21.15.94:0/3070220673 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712c22650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a7ad9d640 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 0x556712c07c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 -- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 msgr2=0x556712c22650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712c22650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 -- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 msgr2=0x556712c07c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 0x556712c07c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 -- 172.21.15.94:0/3070220673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556712c05e80 con 0x556712be3330 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a73fff640 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c076c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3a64007940 tx=0x7f3a6400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b716ed1fa7fdc81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a5b7fe640 1 -- 172.21.15.94:0/3070220673 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6402f070 con 0x556712be3330 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a5b7fe640 1 -- 172.21.15.94:0/3070220673 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64014b30 con 0x556712be3330 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a5b7fe640 1 -- 172.21.15.94:0/3070220673 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6401a070 con 0x556712be3330 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.470+0000 7f3a83a71740 1 -- 172.21.15.94:0/3070220673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 msgr2=0x556712c076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c076c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3a64007940 tx=0x7f3a6400f000 comp rx=0 tx=0).stop 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 -- 172.21.15.94:0/3070220673 shutdown_connections 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 0x556712c07c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be3330 0x556712c076c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 --2- 172.21.15.94:0/3070220673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be2f50 0x556712c22650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 -- 172.21.15.94:0/3070220673 >> 172.21.15.94:0/3070220673 conn(0x556712be14b0 msgr2=0x556712c2fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 -- 172.21.15.94:0/3070220673 shutdown_connections 2024-03-06T07:19:31.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 -- 172.21.15.94:0/3070220673 wait complete. 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 -- start start 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.471+0000 7f3a83a71740 1 Processor -- start 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be2f50 0x556712c22a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be3330 0x556712c8c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 0x556712c8f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x556712c2f7b0 con 0x556712be2f50 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c1f70b0 con 0x556712c05690 2024-03-06T07:19:31.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556712c32ff0 con 0x556712be3330 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 10 client.0 _ll_register_callbacks cb 0x7f3a64007ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be2f50 0x556712c22a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be2f50 0x556712c22a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50436/0 (socket says 172.21.15.94:50436) 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 -- 172.21.15.94:0/1069311406 learned_addr learned my addr 172.21.15.94:0/1069311406 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 -- 172.21.15.94:0/1069311406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be3330 msgr2=0x556712c8c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be3330 0x556712c8c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 -- 172.21.15.94:0/1069311406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 msgr2=0x556712c8f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 0x556712c8f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 -- 172.21.15.94:0/1069311406 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556712c0b450 con 0x556712be2f50 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a7a59c640 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be2f50 0x556712c22a50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c012690 tx=0x7f3a6c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfedd9e4317fff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a597fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a597fa640 1 -- 172.21.15.94:0/1069311406 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c019040 con 0x556712be2f50 2024-03-06T07:19:31.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a597fa640 1 -- 172.21.15.94:0/1069311406 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3a6c02bca0 con 0x556712be2f50 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.473+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a64014b30 con 0x556712be2f50 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a597fa640 1 -- 172.21.15.94:0/1069311406 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a6c007050 con 0x556712be2f50 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 10 client.5430 fetch_fsmap learned FSMap version 22 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x5567129266b0 con 0x556712be2f50 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a597fa640 1 -- 172.21.15.94:0/1069311406 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64014b30 con 0x556712be2f50 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a597fa640 1 -- 172.21.15.94:0/1069311406 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 22) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3a6c0071f0 con 0x556712be2f50 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 10 client.5430 fetch_fsmap finished waiting for FSMap version 22 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 10 client.5430 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 20 client.5430 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712bdc510 0x556712bdc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 4 client.5430 mds_command: new command op to 5166 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 --> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556712b4ef40 con 0x556712bdc510 2024-03-06T07:19:31.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a73fff640 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712bdc510 0x556712bdc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a597fa640 10 client.5430 ms_handle_connect on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a73fff640 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712bdc510 0x556712bdc8f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d252c479283c6d server_cookie=b8036e9df29b9d9d in_seq=0 out_seq=0 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a597fa640 1 -- 172.21.15.94:0/1069311406 <== mds.0 v2:172.21.15.94:6833/2367914435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x556712b4ef40 con 0x556712bdc510 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a597fa640 10 client.5430 handle_command_reply: tid=0 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.474+0000 7f3a83a71740 1 client.5430 shutdown 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be2f50 msgr2=0x556712c22a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be2f50 0x556712c22a50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c012690 tx=0x7f3a6c0012d0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 shutdown_connections 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x556712bdc510 0x556712bdc8f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556712c05690 0x556712c8f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a597fa640 0 client.5430 ms_handle_reset on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556712be3330 0x556712c8c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 --2- 172.21.15.94:0/1069311406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x556712be2f50 0x556712c22a50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 >> 172.21.15.94:0/1069311406 conn(0x556712be14b0 msgr2=0x556712c04b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 shutdown_connections 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 1 -- 172.21.15.94:0/1069311406 wait complete. 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a83a71740 20 client.5430 trim_cache size 0 max 16384 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 5166, 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:19:31.483 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:19:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.9880734909999997, 2024-03-06T07:19:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 22, 2024-03-06T07:19:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 40, 2024-03-06T07:19:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 40, 2024-03-06T07:19:31.484 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 25.924685464 2024-03-06T07:19:31.484 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5c06b500 msgr2=0x7f3a5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5c06b500 0x7f3a5c06d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c001830 tx=0x7f3a6c009040 comp rx=0 tx=0).stop 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7c0ac130 msgr2=0x7f3a7c14c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7c0ac130 0x7f3a7c14c8a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3a7402c600 tx=0x7f3a74009750 comp rx=0 tx=0).stop 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 shutdown_connections 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a5c06b500 0x7f3a5c06d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7c0ac130 0x7f3a7c14c8a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a7c072f90 0x7f3a7c14c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 --2- 172.21.15.94:0/1741576479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a68004d40 0x7f3a7c1ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 >> 172.21.15.94:0/1741576479 conn(0x7f3a7c0c7670 msgr2=0x7f3a7c14db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 shutdown_connections 2024-03-06T07:19:31.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.475+0000 7f3a80e8d640 1 -- 172.21.15.94:0/1741576479 wait complete. 2024-03-06T07:19:31.488 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:19:18.801003+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5166": { "addr": "172.21.15.94:6835/2367914435", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 2367914435, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 2367914435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5166, "incarnation": 19, "join_fscid": 2, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:19:29.497645+0000", "qdb_cluster": [ 5166 ], "qdb_leader": 5166, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5166 } } } ], "standbys": [ { "addr": "172.21.15.145:6835/1902381441", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1902381441, "type": "v2" }, { "addr": "172.21.15.145:6835", "nonce": 1902381441, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5215, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/856474008", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6836", "nonce": 856474008, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 856474008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/597543014", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 597543014, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 597543014, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34814, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:19:31.489 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 2, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:19:31.489 INFO:tasks.cephfs.test_failover:current = [{'gid': 34814, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/597543014', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 597543014}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 597543014}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 5410, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/856474008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 856474008}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 856474008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 20}, {'gid': 5166, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5215, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6835/1902381441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1902381441}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 1902381441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:19:31.489 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:19:31.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:19:31.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.594+0000 7f6ee775e640 1 Processor -- start 2024-03-06T07:19:31.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.594+0000 7f6ee775e640 1 -- start start 2024-03-06T07:19:31.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ee775e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6ee00c36b0 0x7f6ee006e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ee775e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 0x7f6ee00c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ee775e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00c3e50 con 0x7f6ee0157070 2024-03-06T07:19:31.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ee775e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00c3c00 con 0x7f6ee00c36b0 2024-03-06T07:19:31.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ee775e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00c4130 con 0x7f6ee006ede0 2024-03-06T07:19:31.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ee54fa640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6ee0157070 0x7f6ee0072ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59500/0 (socket says 172.21.15.94:59500) 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ee54fa640 1 -- 172.21.15.94:0/1000521326 learned_addr learned my addr 172.21.15.94:0/1000521326 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794961212 0 0) 0x7f6ee00c3e50 con 0x7f6ee0157070 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.595+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ecc003520 con 0x7f6ee0157070 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee5cfb640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 0x7f6ee00c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/1000521326 >> v2:172.21.15.145:3300/0 conn(0x7f6ee00c36b0 0x7f6ee006e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579868507 0 0) 0x7f6ecc003520 con 0x7f6ee0157070 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 msgr2=0x7f6ee00c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 0x7f6ee00c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 >> v2:172.21.15.145:3300/0 conn(0x7f6ee00c36b0 msgr2=0x7f6ee006e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 --2- 172.21.15.94:0/1000521326 >> v2:172.21.15.145:3300/0 conn(0x7f6ee00c36b0 0x7f6ee006e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00c4830 con 0x7f6ee0157070 2024-03-06T07:19:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f6ed0001670 con 0x7f6ee0157070 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 >> v1:172.21.15.94:6789/0 conn(0x7f6ee0157070 legacy=0x7f6ee0072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ecc0039b0 0x7f6ecc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ecc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 0x7f6ecc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00c3c00 con 0x7f6ecc004800 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00c4130 con 0x7f6ecc0039b0 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc00b9c0 con 0x7f6ee006ede0 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ecc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee54fa640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ecc0039b0 0x7f6ecc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee5cfb640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 0x7f6ecc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee4cf9640 1 -- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 msgr2=0x7f6ecc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee006ede0 0x7f6ecc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee4cf9640 1 -- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ecc0039b0 msgr2=0x7f6ecc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ecc0039b0 0x7f6ecc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.596+0000 7f6ee4cf9640 1 -- 172.21.15.94:0/1000521326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc003520 con 0x7f6ecc004800 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.597+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ecc004c80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6edc009ac0 tx=0x7f6ecc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff7a0aa54cf1699 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc039070 con 0x7f6ecc004800 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc02c710 con 0x7f6ecc004800 2024-03-06T07:19:31.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ed7fff640 1 -- 172.21.15.94:0/1000521326 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc009690 con 0x7f6ecc004800 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ee775e640 1 -- 172.21.15.94:0/1000521326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 msgr2=0x7f6ecc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ee775e640 1 --2- 172.21.15.94:0/1000521326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ecc004c80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6edc009ac0 tx=0x7f6ecc005830 comp rx=0 tx=0).stop 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ee5cfb640 1 -- 172.21.15.94:0/1000521326 reap_dead start 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ee775e640 1 -- 172.21.15.94:0/1000521326 shutdown_connections 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ee775e640 1 -- 172.21.15.94:0/1000521326 >> 172.21.15.94:0/1000521326 conn(0x7f6ee00aa5c0 msgr2=0x7f6ee00aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ee775e640 1 -- 172.21.15.94:0/1000521326 shutdown_connections 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.598+0000 7f6ee775e640 1 -- 172.21.15.94:0/1000521326 wait complete. 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.599+0000 7f6ee775e640 1 Processor -- start 2024-03-06T07:19:31.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.599+0000 7f6ee775e640 1 -- start start 2024-03-06T07:19:31.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee775e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ee01dc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee775e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee014d410 0x7f6ee01ecab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee775e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ee01ecff0 0x7f6ee014bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee775e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee01575d0 con 0x7f6ecc004800 2024-03-06T07:19:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee775e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0157450 con 0x7f6ee01ecff0 2024-03-06T07:19:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee775e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00c4d50 con 0x7f6ee014d410 2024-03-06T07:19:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee54fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ee01dc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee54fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ee01dc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50442/0 (socket says 172.21.15.94:50442) 2024-03-06T07:19:31.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee54fa640 1 -- 172.21.15.94:0/776529976 learned_addr learned my addr 172.21.15.94:0/776529976 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee014d410 0x7f6ee01ecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee5cfb640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ee01ecff0 0x7f6ee014bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee54fa640 1 -- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee014d410 msgr2=0x7f6ee01ecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.600+0000 7f6ee54fa640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee014d410 0x7f6ee01ecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ee54fa640 1 -- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ee01ecff0 msgr2=0x7f6ee014bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ee54fa640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ee01ecff0 0x7f6ee014bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ee54fa640 1 -- 172.21.15.94:0/776529976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee01ea8e0 con 0x7f6ecc004800 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ee5cfb640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ee01ecff0 0x7f6ee014bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ee54fa640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ee01dc720 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0007370 tx=0x7f6ed0007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ed5ffb640 1 -- 172.21.15.94:0/776529976 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed001a070 con 0x7f6ecc004800 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ed5ffb640 1 -- 172.21.15.94:0/776529976 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed000f960 con 0x7f6ecc004800 2024-03-06T07:19:31.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee01ed530 con 0x7f6ecc004800 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ed5ffb640 1 -- 172.21.15.94:0/776529976 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000fd50 con 0x7f6ecc004800 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.601+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee014c290 con 0x7f6ecc004800 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.602+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee006e410 con 0x7f6ecc004800 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.603+0000 7f6ed5ffb640 1 -- 172.21.15.94:0/776529976 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ed001d070 con 0x7f6ecc004800 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.603+0000 7f6ed5ffb640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ec406b500 0x7f6ec406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.604+0000 7f6ed5ffb640 1 -- 172.21.15.94:0/776529976 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6ed0089c30 con 0x7f6ecc004800 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.604+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ec406b500 0x7f6ec406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:31.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.605+0000 7f6ee4cf9640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ec406b500 0x7f6ec406d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6edc036000 tx=0x7f6edc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:31.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.608+0000 7f6ed5ffb640 1 -- 172.21.15.94:0/776529976 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ee006e410 con 0x7f6ecc004800 2024-03-06T07:19:31.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.943+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f6ee00aa9a0 con 0x7f6ecc004800 2024-03-06T07:19:31.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.946+0000 7f6ed5ffb640 1 -- 172.21.15.94:0/776529976 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7f6ee00aa9a0 con 0x7f6ecc004800 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ec406b500 msgr2=0x7f6ec406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ec406b500 0x7f6ec406d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6edc036000 tx=0x7f6edc030000 comp rx=0 tx=0).stop 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 msgr2=0x7f6ee01dc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ee01dc720 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0007370 tx=0x7f6ed0007830 comp rx=0 tx=0).stop 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 shutdown_connections 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ec406b500 0x7f6ec406d9c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ee01ecff0 0x7f6ee014bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ee014d410 0x7f6ee01ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 --2- 172.21.15.94:0/776529976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ecc004800 0x7f6ee01dc720 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.948+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 >> 172.21.15.94:0/776529976 conn(0x7f6ee00aa5c0 msgr2=0x7f6ee0155ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:31.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.949+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 shutdown_connections 2024-03-06T07:19:31.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:31.949+0000 7f6ee775e640 1 -- 172.21.15.94:0/776529976 wait complete. 2024-03-06T07:19:31.961 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:32.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.063+0000 7fb322a4e640 1 Processor -- start 2024-03-06T07:19:32.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.063+0000 7fb322a4e640 1 -- start start 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb322a4e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 0x7fb31c0a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb322a4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c158800 con 0x7fb31c0c36e0 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb322a4e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c158ae0 con 0x7fb31c0c3280 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb322a4e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0c3b10 con 0x7fb31c06d9f0 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb320feb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb31c0c36e0 0x7fb31c1558f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59514/0 (socket says 172.21.15.94:59514) 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb320feb640 1 -- 172.21.15.94:0/326411635 learned_addr learned my addr 172.21.15.94:0/326411635 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb31bfff640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 0x7fb31c0a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338585806 0 0) 0x7fb31c158800 con 0x7fb31c0c36e0 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.064+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb308003520 con 0x7fb31c0c36e0 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4149953714 0 0) 0x7fb308003520 con 0x7fb31c0c36e0 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 msgr2=0x7fb31c0a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 0x7fb31c0a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 >> v1:172.21.15.145:6789/0 conn(0x7fb31c0c3280 legacy=0x7fb31c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c158dc0 con 0x7fb31c0c36e0 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fb3140019f0 con 0x7fb31c0c36e0 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 >> v1:172.21.15.94:6789/0 conn(0x7fb31c0c36e0 legacy=0x7fb31c1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3080039e0 0x7fb308003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb308004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 0x7fb308005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0c3b10 con 0x7fb308004800 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30800bb80 con 0x7fb3080039e0 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30800bd80 con 0x7fb31c06d9f0 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31b7fe640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb308004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb320feb640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 0x7fb308005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31bfff640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3080039e0 0x7fb308003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31b7fe640 1 -- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 msgr2=0x7fb308005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31b7fe640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c06d9f0 0x7fb308005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31b7fe640 1 -- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3080039e0 msgr2=0x7fb308003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31b7fe640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3080039e0 0x7fb308003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.065+0000 7fb31b7fe640 1 -- 172.21.15.94:0/326411635 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb308003520 con 0x7fb308004800 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.066+0000 7fb31b7fe640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb308004c80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb30c001750 tx=0x7fb308005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd8daf3ecbdff1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.066+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c001370 con 0x7fb308004800 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.066+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c0052d0 con 0x7fb308004800 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.066+0000 7fb31affd640 1 -- 172.21.15.94:0/326411635 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c003830 con 0x7fb308004800 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.067+0000 7fb322a4e640 1 -- 172.21.15.94:0/326411635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 msgr2=0x7fb308004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.067+0000 7fb322a4e640 1 --2- 172.21.15.94:0/326411635 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb308004c80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb30c001750 tx=0x7fb308005830 comp rx=0 tx=0).stop 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.067+0000 7fb320feb640 1 -- 172.21.15.94:0/326411635 reap_dead start 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.067+0000 7fb322a4e640 1 -- 172.21.15.94:0/326411635 shutdown_connections 2024-03-06T07:19:32.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.067+0000 7fb322a4e640 1 -- 172.21.15.94:0/326411635 >> 172.21.15.94:0/326411635 conn(0x7fb31c0aa630 msgr2=0x7fb31c0aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.067+0000 7fb322a4e640 1 -- 172.21.15.94:0/326411635 shutdown_connections 2024-03-06T07:19:32.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.067+0000 7fb322a4e640 1 -- 172.21.15.94:0/326411635 wait complete. 2024-03-06T07:19:32.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.068+0000 7fb322a4e640 1 Processor -- start 2024-03-06T07:19:32.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.068+0000 7fb322a4e640 1 -- start start 2024-03-06T07:19:32.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb322a4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb31c14f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb322a4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb31c0ab890 0x7fb31c0a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb322a4e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c0a9400 0x7fb31c157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb322a4e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c06ddd0 con 0x7fb308004800 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb322a4e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c159380 con 0x7fb31c0ab890 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb322a4e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c06df50 con 0x7fb31c0a9400 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb31c14f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb31c14f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50462/0 (socket says 172.21.15.94:50462) 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 -- 172.21.15.94:0/2407698417 learned_addr learned my addr 172.21.15.94:0/2407698417 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb320feb640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c0a9400 0x7fb31c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31b7fe640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb31c0ab890 0x7fb31c0a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 -- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c0a9400 msgr2=0x7fb31c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c0a9400 0x7fb31c157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 -- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb31c0ab890 msgr2=0x7fb31c0a8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb31c0ab890 0x7fb31c0a8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 -- 172.21.15.94:0/2407698417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c1562c0 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb320feb640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c0a9400 0x7fb31c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb31bfff640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb31c14f640 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb304011910 tx=0x7fb304011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb318ff9640 1 -- 172.21.15.94:0/2407698417 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30400fa90 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb318ff9640 1 -- 172.21.15.94:0/2407698417 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30400fc30 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c156530 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.069+0000 7fb318ff9640 1 -- 172.21.15.94:0/2407698417 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304014440 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.070+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c153f00 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.070+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c04f330 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.071+0000 7fb318ff9640 1 -- 172.21.15.94:0/2407698417 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb304012040 con 0x7fb308004800 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.071+0000 7fb318ff9640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.072+0000 7fb318ff9640 1 -- 172.21.15.94:0/2407698417 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb30408a8f0 con 0x7fb308004800 2024-03-06T07:19:32.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.072+0000 7fb31b7fe640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.072+0000 7fb31b7fe640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb30c033000 tx=0x7fb30c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.080+0000 7fb318ff9640 1 -- 172.21.15.94:0/2407698417 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb304019030 con 0x7fb308004800 2024-03-06T07:19:32.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.291+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb31c0701f0 con 0x7fb308004800 2024-03-06T07:19:32.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.291+0000 7fb318ff9640 1 -- 172.21.15.94:0/2407698417 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fb304059620 con 0x7fb308004800 2024-03-06T07:19:32.294 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:32.294 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:29.497645+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":2}]} 2024-03-06T07:19:32.294 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 22 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb2fc06b4d0 msgr2=0x7fb2fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb30c033000 tx=0x7fb30c009030 comp rx=0 tx=0).stop 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 msgr2=0x7fb31c14f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb31c14f640 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb304011910 tx=0x7fb304011dd0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 shutdown_connections 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb31c0a9400 0x7fb31c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb31c0ab890 0x7fb31c0a8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 --2- 172.21.15.94:0/2407698417 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb308004800 0x7fb31c14f640 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.293+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 >> 172.21.15.94:0/2407698417 conn(0x7fb31c0aa630 msgr2=0x7fb31c06c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.294+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 shutdown_connections 2024-03-06T07:19:32.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.294+0000 7fb322a4e640 1 -- 172.21.15.94:0/2407698417 wait complete. 2024-03-06T07:19:32.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:19:18.801003+0000', 'modified': '2024-03-06T07:19:29.497645+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5166}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5166': {'gid': 5166, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5166, 'qdb_cluster': [5166]} 2024-03-06T07:19:32.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:19:32.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:19:32.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.809+0000 7f7309735640 1 Processor -- start 2024-03-06T07:19:32.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.809+0000 7f7309735640 1 -- start start 2024-03-06T07:19:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f7309735640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f73040c69f0 0x7f73040c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f7309735640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73041558d0 0x7f7304155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f7309735640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7304155130 con 0x7f73040c8950 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f7309735640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7304155440 con 0x7f73041558d0 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f7309735640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73040ad480 con 0x7f73040c69f0 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f73037fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73041558d0 0x7f7304155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f73027fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f73040c8950 0x7f73040acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59520/0 (socket says 172.21.15.94:59520) 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f73027fc640 1 -- 172.21.15.94:0/1187982290 learned_addr learned my addr 172.21.15.94:0/1187982290 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.810+0000 7f73037fe640 1 --2- 172.21.15.94:0/1187982290 >> v2:172.21.15.94:3300/0 conn(0x7f73041558d0 0x7f7304155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50466/0 (socket says 172.21.15.94:50466) 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f7302ffd640 1 --2- 172.21.15.94:0/1187982290 >> v2:172.21.15.145:3300/0 conn(0x7f73040c69f0 0x7f73040c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f73037fe640 1 -- 172.21.15.94:0/1187982290 >> v2:172.21.15.145:3300/0 conn(0x7f73040c69f0 msgr2=0x7f73040c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f73037fe640 1 --2- 172.21.15.94:0/1187982290 >> v2:172.21.15.145:3300/0 conn(0x7f73040c69f0 0x7f73040c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f73037fe640 1 -- 172.21.15.94:0/1187982290 >> v1:172.21.15.94:6789/0 conn(0x7f73040c8950 legacy=0x7f73040acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f7301ffb640 1 -- 172.21.15.94:0/1187982290 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078658354 0 0) 0x7f72f8001110 con 0x7f73040c8950 2024-03-06T07:19:32.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f73037fe640 1 -- 172.21.15.94:0/1187982290 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73040c5d50 con 0x7f73041558d0 2024-03-06T07:19:32.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f7302ffd640 1 --2- 172.21.15.94:0/1187982290 >> v2:172.21.15.145:3300/0 conn(0x7f73040c69f0 0x7f73040c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:32.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f73037fe640 1 --2- 172.21.15.94:0/1187982290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304155d00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f73040aa430 tx=0x7f72fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333c5b0db20af241 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f7301ffb640 1 -- 172.21.15.94:0/1187982290 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc039070 con 0x7f73041558d0 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f7301ffb640 1 -- 172.21.15.94:0/1187982290 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc02cbf0 con 0x7f73041558d0 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.811+0000 7f7301ffb640 1 -- 172.21.15.94:0/1187982290 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc034440 con 0x7f73041558d0 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 -- 172.21.15.94:0/1187982290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 msgr2=0x7f7304155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 --2- 172.21.15.94:0/1187982290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304155d00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f73040aa430 tx=0x7f72fc02be40 comp rx=0 tx=0).stop 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 -- 172.21.15.94:0/1187982290 shutdown_connections 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 --2- 172.21.15.94:0/1187982290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304155d00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 --2- 172.21.15.94:0/1187982290 >> v2:172.21.15.145:3300/0 conn(0x7f73040c69f0 0x7f73040c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 -- 172.21.15.94:0/1187982290 >> 172.21.15.94:0/1187982290 conn(0x7f73040cae70 msgr2=0x7f73040cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 -- 172.21.15.94:0/1187982290 shutdown_connections 2024-03-06T07:19:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.812+0000 7f7309735640 1 -- 172.21.15.94:0/1187982290 wait complete. 2024-03-06T07:19:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.814+0000 7f7309735640 1 Processor -- start 2024-03-06T07:19:32.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.814+0000 7f7309735640 1 -- start start 2024-03-06T07:19:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f7309735640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73040c69f0 0x7f730406c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f7309735640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73040c8950 0x7f730414c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f7309735640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f7309735640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7304152a20 con 0x7f73041558d0 2024-03-06T07:19:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f7309735640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73040c6310 con 0x7f73040c8950 2024-03-06T07:19:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f7309735640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7304152ba0 con 0x7f73040c69f0 2024-03-06T07:19:32.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f73037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f73037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50470/0 (socket says 172.21.15.94:50470) 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f73037fe640 1 -- 172.21.15.94:0/1472711778 learned_addr learned my addr 172.21.15.94:0/1472711778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f73027fc640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73040c8950 0x7f730414c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.815+0000 7f7302ffd640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73040c69f0 0x7f730406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f73037fe640 1 -- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73040c69f0 msgr2=0x7f730406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f73037fe640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73040c69f0 0x7f730406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f73037fe640 1 -- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73040c8950 msgr2=0x7f730414c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f73037fe640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73040c8950 0x7f730414c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f73037fe640 1 -- 172.21.15.94:0/1472711778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73040c4640 con 0x7f73041558d0 2024-03-06T07:19:32.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f73037fe640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304157980 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009e80 tx=0x7f72fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f72eb7fe640 1 -- 172.21.15.94:0/1472711778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc039050 con 0x7f73041558d0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f72eb7fe640 1 -- 172.21.15.94:0/1472711778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc045070 con 0x7f73041558d0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f72eb7fe640 1 -- 172.21.15.94:0/1472711778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc03e830 con 0x7f73041558d0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73040c4920 con 0x7f73041558d0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.816+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730406c7c0 con 0x7f73041558d0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.817+0000 7f72eb7fe640 1 -- 172.21.15.94:0/1472711778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72fc03e9d0 con 0x7f73041558d0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.818+0000 7f72eb7fe640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72f006b4d0 0x7f72f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.818+0000 7f72eb7fe640 1 -- 172.21.15.94:0/1472711778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f72fc0ad220 con 0x7f73041558d0 2024-03-06T07:19:32.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.818+0000 7f7302ffd640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72f006b4d0 0x7f72f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.818+0000 7f7302ffd640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72f006b4d0 0x7f72f006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f72f400a950 tx=0x7f72f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 Processor -- start 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 -- start start 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b42fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 0x55716b42ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b4304b0 0x55716b42c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55716b3966b0 con 0x55716b4304b0 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55716b33fa60 con 0x55716b432330 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.819+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55716b392750 con 0x55716b431f50 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b4304b0 0x55716b42c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b4304b0 0x55716b42c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50484/0 (socket says 172.21.15.94:50484) 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 -- 172.21.15.94:0/3504286645 learned_addr learned my addr 172.21.15.94:0/3504286645 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73027fc640 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 0x55716b42ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f7302ffd640 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b42fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 -- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 msgr2=0x55716b42fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b42fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 -- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 msgr2=0x55716b42ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 0x55716b42ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 -- 172.21.15.94:0/3504286645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55716b35df30 con 0x55716b4304b0 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f73037fe640 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b4304b0 0x55716b42c680 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f72fc037040 tx=0x7f72fc02cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83408d98792049c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f72e8ff9640 1 -- 172.21.15.94:0/3504286645 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc093070 con 0x55716b4304b0 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f72e8ff9640 1 -- 172.21.15.94:0/3504286645 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc03a040 con 0x55716b4304b0 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.820+0000 7f72e8ff9640 1 -- 172.21.15.94:0/3504286645 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc07b7c0 con 0x55716b4304b0 2024-03-06T07:19:32.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 -- 172.21.15.94:0/3504286645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b4304b0 msgr2=0x55716b42c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b4304b0 0x55716b42c680 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f72fc037040 tx=0x7f72fc02cbf0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 -- 172.21.15.94:0/3504286645 shutdown_connections 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b4304b0 0x55716b42c680 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 0x55716b42ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3504286645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b42fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 -- 172.21.15.94:0/3504286645 >> 172.21.15.94:0/3504286645 conn(0x55716b42e7d0 msgr2=0x55716b4533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 -- 172.21.15.94:0/3504286645 shutdown_connections 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 -- 172.21.15.94:0/3504286645 wait complete. 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.821+0000 7f730c3f4740 1 -- start start 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 1 Processor -- start 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b431f50 0x55716b4527b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b456dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b454a70 0x55716b454ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55716b3966b0 con 0x55716b432330 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55716b352ea0 con 0x55716b431f50 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55716b454230 con 0x55716b454a70 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f730c3f4740 10 client.0 _ll_register_callbacks cb 0x55716b352870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f73027fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b456dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f73027fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b456dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50494/0 (socket says 172.21.15.94:50494) 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.822+0000 7f73027fc640 1 -- 172.21.15.94:0/2560146389 learned_addr learned my addr 172.21.15.94:0/2560146389 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f7302ffd640 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b431f50 0x55716b4527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f73037fe640 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b454a70 0x55716b454ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f73027fc640 1 -- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b454a70 msgr2=0x55716b454ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f73027fc640 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b454a70 0x55716b454ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f73027fc640 1 -- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b431f50 msgr2=0x55716b4527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f73027fc640 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b431f50 0x55716b4527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f73027fc640 1 -- 172.21.15.94:0/2560146389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55716b353270 con 0x55716b432330 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f73027fc640 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b456dc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f72f80076d0 tx=0x7f72f8001850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962771551667a722 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f72e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2560146389 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f80145e0 con 0x55716b432330 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2560146389 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f72f8014780 con 0x55716b432330 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72fc03a040 con 0x55716b432330 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2560146389 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72f8022050 con 0x55716b432330 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 10 client.5457 fetch_fsmap learned FSMap version 22 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x55716b0ab2d0 con 0x55716b432330 2024-03-06T07:19:32.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2560146389 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc03a040 con 0x55716b432330 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2560146389 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 22) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f72f8025020 con 0x55716b432330 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 10 client.5457 fetch_fsmap finished waiting for FSMap version 22 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 10 client.5457 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 20 client.5457 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b482480 0x55716b482880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 4 client.5457 mds_command: new command op to 5166 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.823+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 --> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55716b3a3bf0 con 0x55716b482480 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.824+0000 7f7302ffd640 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b482480 0x55716b482880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.824+0000 7f7302ffd640 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b482480 0x55716b482880 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32354cf749cb9cd8 server_cookie=bcdf48255a78da44 in_seq=0 out_seq=0 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.824+0000 7f72e1ffb640 10 client.5457 ms_handle_connect on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2560146389 <== mds.0 v2:172.21.15.94:6833/2367914435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55716b3a3bf0 con 0x55716b482480 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f72e1ffb640 10 client.5457 handle_command_reply: tid=0 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 client.5457 shutdown 2024-03-06T07:19:32.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 msgr2=0x55716b456dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b456dc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f72f80076d0 tx=0x7f72f8001850 comp rx=0 tx=0).stop 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 shutdown_connections 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b482480 0x55716b482880 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b454a70 0x55716b454ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b456dc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2560146389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b431f50 0x55716b4527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f72e1ffb640 0 client.5457 ms_handle_reset on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 >> 172.21.15.94:0/2560146389 conn(0x55716b4304b0 msgr2=0x55716b42b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 shutdown_connections 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 1 -- 172.21.15.94:0/2560146389 wait complete. 2024-03-06T07:19:32.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.825+0000 7f730c3f4740 20 client.5457 trim_cache size 0 max 16384 2024-03-06T07:19:32.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 Processor -- start 2024-03-06T07:19:32.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 -- start start 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b42b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 0x55716b4308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b454a70 0x55716b42f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041f7c70 con 0x55716b454a70 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041f6d90 con 0x55716b432330 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73041572e0 con 0x55716b431f50 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f73037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b454a70 0x55716b42f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f73037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b454a70 0x55716b42f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50506/0 (socket says 172.21.15.94:50506) 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.842+0000 7f73037fe640 1 -- 172.21.15.94:0/3849828377 learned_addr learned my addr 172.21.15.94:0/3849828377 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f73037fe640 1 -- 172.21.15.94:0/3849828377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 msgr2=0x55716b42b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f73037fe640 1 --2- 172.21.15.94:0/3849828377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b42b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f73037fe640 1 -- 172.21.15.94:0/3849828377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 msgr2=0x55716b4308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f73037fe640 1 --2- 172.21.15.94:0/3849828377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 0x55716b4308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f73037fe640 1 -- 172.21.15.94:0/3849828377 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55716b1756b0 con 0x55716b454a70 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f73037fe640 1 --2- 172.21.15.94:0/3849828377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b454a70 0x55716b42f5a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f72fc07b750 tx=0x7f72fc04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816aa033f8cef9ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f72e3fff640 1 -- 172.21.15.94:0/3849828377 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc093040 con 0x55716b454a70 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f72e3fff640 1 -- 172.21.15.94:0/3849828377 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc07f910 con 0x55716b454a70 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f72e3fff640 1 -- 172.21.15.94:0/3849828377 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc083ed0 con 0x55716b454a70 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 -- 172.21.15.94:0/3849828377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b454a70 msgr2=0x55716b42f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3849828377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b454a70 0x55716b42f5a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f72fc07b750 tx=0x7f72fc04c000 comp rx=0 tx=0).stop 2024-03-06T07:19:32.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 -- 172.21.15.94:0/3849828377 shutdown_connections 2024-03-06T07:19:32.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3849828377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b454a70 0x55716b42f5a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3849828377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b432330 0x55716b4308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 --2- 172.21.15.94:0/3849828377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b42b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 -- 172.21.15.94:0/3849828377 >> 172.21.15.94:0/3849828377 conn(0x55716b4304b0 msgr2=0x55716b456bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 -- 172.21.15.94:0/3849828377 shutdown_connections 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.843+0000 7f730c3f4740 1 -- 172.21.15.94:0/3849828377 wait complete. 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.844+0000 7f730c3f4740 1 -- start start 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.844+0000 7f730c3f4740 1 Processor -- start 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b44fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b471a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b454a70 0x55716b4327f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55716b482300 con 0x55716b432330 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730414c5a0 con 0x55716b454a70 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55716b47eff0 con 0x55716b431f50 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 10 client.0 _ll_register_callbacks cb 0x7f72f8007280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:32.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b471a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b471a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50512/0 (socket says 172.21.15.94:50512) 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 -- 172.21.15.94:0/2001134537 learned_addr learned my addr 172.21.15.94:0/2001134537 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73037fe640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b454a70 0x55716b4327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f7302ffd640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b44fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 -- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 msgr2=0x55716b44fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b44fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 -- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b454a70 msgr2=0x55716b4327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b454a70 0x55716b4327f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 -- 172.21.15.94:0/2001134537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55716b47f2c0 con 0x55716b432330 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73037fe640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b454a70 0x55716b4327f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f73027fc640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b471a50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f72f80076d0 tx=0x7f72f8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33840bb77655d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f72e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:32.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2001134537 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f80147f0 con 0x55716b432330 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2001134537 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f72f8014990 con 0x55716b432330 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.845+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72fc07f910 con 0x55716b432330 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2001134537 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72f8035050 con 0x55716b432330 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 10 client.5463 fetch_fsmap learned FSMap version 22 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x55716b432df0 con 0x55716b432330 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2001134537 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc07f910 con 0x55716b432330 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2001134537 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 22) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f72f8038020 con 0x55716b432330 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 10 client.5463 fetch_fsmap finished waiting for FSMap version 22 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 10 client.5463 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 20 client.5463 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b4db5b0 0x55716b4db990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 4 client.5463 mds_command: new command op to 5166 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 --> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55716b39df40 con 0x55716b4db5b0 2024-03-06T07:19:32.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f7302ffd640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b4db5b0 0x55716b4db990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f7302ffd640 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b4db5b0 0x55716b4db990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31670af6826604ba server_cookie=d221098ca5731024 in_seq=0 out_seq=0 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f72e1ffb640 10 client.5463 ms_handle_connect on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f72e1ffb640 1 -- 172.21.15.94:0/2001134537 <== mds.0 v2:172.21.15.94:6833/2367914435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55716b39df40 con 0x55716b4db5b0 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f72e1ffb640 10 client.5463 handle_command_reply: tid=0 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 client.5463 shutdown 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 msgr2=0x55716b471a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b471a50 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f72f80076d0 tx=0x7f72f8000ed0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 shutdown_connections 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x55716b4db5b0 0x55716b4db990 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55716b454a70 0x55716b4327f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55716b432330 0x55716b471a50 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 --2- 172.21.15.94:0/2001134537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55716b431f50 0x55716b44fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 >> 172.21.15.94:0/2001134537 conn(0x55716b4304b0 msgr2=0x55716b46b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.846+0000 7f72e1ffb640 0 client.5463 ms_handle_reset on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 shutdown_connections 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f730c3f4740 1 -- 172.21.15.94:0/2001134537 wait complete. 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f730c3f4740 20 client.5463 trim_cache size 0 max 16384 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 5166, 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:19:32.860 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:19:32.861 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:19:32.861 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 6.3599025630000003, 2024-03-06T07:19:32.861 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 22, 2024-03-06T07:19:32.861 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 40, 2024-03-06T07:19:32.861 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 40, 2024-03-06T07:19:32.861 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 27.296514480999999 2024-03-06T07:19:32.861 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:19:32.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72f006b4d0 msgr2=0x7f72f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72f006b4d0 0x7f72f006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f72f400a950 tx=0x7f72f4008040 comp rx=0 tx=0).stop 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 msgr2=0x7f7304157980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304157980 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009e80 tx=0x7f72fc004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 shutdown_connections 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72f006b4d0 0x7f72f006d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73041558d0 0x7f7304157980 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73040c8950 0x7f730414c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 --2- 172.21.15.94:0/1472711778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73040c69f0 0x7f730406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 >> 172.21.15.94:0/1472711778 conn(0x7f73040cae70 msgr2=0x7f7304150aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 shutdown_connections 2024-03-06T07:19:32.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:32.847+0000 7f7309735640 1 -- 172.21.15.94:0/1472711778 wait complete. 2024-03-06T07:19:32.863 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:19:18.801003+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5166": { "addr": "172.21.15.94:6835/2367914435", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 2367914435, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 2367914435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5166, "incarnation": 19, "join_fscid": 2, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:19:29.497645+0000", "qdb_cluster": [ 5166 ], "qdb_leader": 5166, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5166 } } } ], "standbys": [ { "addr": "172.21.15.145:6835/1902381441", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1902381441, "type": "v2" }, { "addr": "172.21.15.145:6835", "nonce": 1902381441, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5215, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/856474008", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6836", "nonce": 856474008, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 856474008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/597543014", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 597543014, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 597543014, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34814, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:19:32.864 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:19:32.865 INFO:tasks.cephfs.test_failover:current = [{'gid': 34814, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/597543014', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 597543014}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 597543014}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 5410, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/856474008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 856474008}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 856474008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 20}, {'gid': 5166, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5215, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6835/1902381441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1902381441}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 1902381441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:19:32.865 DEBUG:tasks.cephfs.test_failover:-1 != 2 2024-03-06T07:19:32.865 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:19:37.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:37.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.966+0000 7fb61f66b640 1 Processor -- start 2024-03-06T07:19:37.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.966+0000 7fb61f66b640 1 -- start start 2024-03-06T07:19:37.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61f66b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb618156ce0 0x7fb6180c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:37.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61f66b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6180c88c0 0x7fb61815d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:37.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61f66b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb61806c2e0 0x7fb6180c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61f66b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61815d9f0 con 0x7fb61806c2e0 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61f66b640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6180c8d20 con 0x7fb618156ce0 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61f66b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180c7790 con 0x7fb6180c88c0 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb61806c2e0 0x7fb6180c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb61806c2e0 0x7fb6180c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50528/0 (socket says 172.21.15.94:50528) 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 -- 172.21.15.94:0/4141062468 learned_addr learned my addr 172.21.15.94:0/4141062468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61cc06640 1 --2- 172.21.15.94:0/4141062468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6180c88c0 0x7fb61815d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61d407640 1 --2- 172.21.15.94:0/4141062468 >> v2:172.21.15.145:3300/0 conn(0x7fb618156ce0 0x7fb6180c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 -- 172.21.15.94:0/4141062468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6180c88c0 msgr2=0x7fb61815d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 --2- 172.21.15.94:0/4141062468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6180c88c0 0x7fb61815d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 -- 172.21.15.94:0/4141062468 >> v2:172.21.15.145:3300/0 conn(0x7fb618156ce0 msgr2=0x7fb6180c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 --2- 172.21.15.94:0/4141062468 >> v2:172.21.15.145:3300/0 conn(0x7fb618156ce0 0x7fb6180c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:37.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61dc08640 1 -- 172.21.15.94:0/4141062468 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6180c7a70 con 0x7fb61806c2e0 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.967+0000 7fb61cc06640 1 --2- 172.21.15.94:0/4141062468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6180c88c0 0x7fb61815d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.968+0000 7fb61dc08640 1 --2- 172.21.15.94:0/4141062468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb61806c2e0 0x7fb6180c5340 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb6180c8300 tx=0x7fb61002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bf361461f3e073 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.968+0000 7fb60ffff640 1 -- 172.21.15.94:0/4141062468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61002c9e0 con 0x7fb61806c2e0 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.968+0000 7fb60ffff640 1 -- 172.21.15.94:0/4141062468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61002cb80 con 0x7fb61806c2e0 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.968+0000 7fb60ffff640 1 -- 172.21.15.94:0/4141062468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6100343f0 con 0x7fb61806c2e0 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.968+0000 7fb61f66b640 1 -- 172.21.15.94:0/4141062468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb61806c2e0 msgr2=0x7fb6180c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.968+0000 7fb61f66b640 1 --2- 172.21.15.94:0/4141062468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb61806c2e0 0x7fb6180c5340 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb6180c8300 tx=0x7fb61002be40 comp rx=0 tx=0).stop 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.969+0000 7fb61f66b640 1 -- 172.21.15.94:0/4141062468 shutdown_connections 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.969+0000 7fb61f66b640 1 --2- 172.21.15.94:0/4141062468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb61806c2e0 0x7fb6180c5340 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.969+0000 7fb61f66b640 1 --2- 172.21.15.94:0/4141062468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6180c88c0 0x7fb61815d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.969+0000 7fb61f66b640 1 --2- 172.21.15.94:0/4141062468 >> v2:172.21.15.145:3300/0 conn(0x7fb618156ce0 0x7fb6180c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.969+0000 7fb61f66b640 1 -- 172.21.15.94:0/4141062468 >> 172.21.15.94:0/4141062468 conn(0x7fb6180ac570 msgr2=0x7fb6180ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:37.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.969+0000 7fb61f66b640 1 -- 172.21.15.94:0/4141062468 shutdown_connections 2024-03-06T07:19:37.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.969+0000 7fb61f66b640 1 -- 172.21.15.94:0/4141062468 wait complete. 2024-03-06T07:19:37.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 Processor -- start 2024-03-06T07:19:37.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 -- start start 2024-03-06T07:19:37.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb61806c2e0 0x7fb6181f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:37.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6180c88c0 0x7fb6181594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:37.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb618156ce0 0x7fb6180af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61806c710 con 0x7fb6180c88c0 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61806bdc0 con 0x7fb61806c2e0 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.970+0000 7fb61f66b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb61806c890 con 0x7fb618156ce0 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6180c88c0 0x7fb6181594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6180c88c0 0x7fb6181594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50532/0 (socket says 172.21.15.94:50532) 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 -- 172.21.15.94:0/110039058 learned_addr learned my addr 172.21.15.94:0/110039058 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61d407640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb61806c2e0 0x7fb6181f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 -- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb618156ce0 msgr2=0x7fb6180af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb618156ce0 0x7fb6180af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 -- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb61806c2e0 msgr2=0x7fb6181f30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb61806c2e0 0x7fb6181f30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:37.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 -- 172.21.15.94:0/110039058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6180c4d70 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61cc06640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6180c88c0 0x7fb6181594c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c7c0 tx=0x7fb60800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb60dffb640 1 -- 172.21.15.94:0/110039058 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608015070 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb60dffb640 1 -- 172.21.15.94:0/110039058 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb608007a30 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6181ef300 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb60dffb640 1 -- 172.21.15.94:0/110039058 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608007e00 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.971+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6180c4a90 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.972+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61804f330 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.973+0000 7fb60dffb640 1 -- 172.21.15.94:0/110039058 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb608018070 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.973+0000 7fb60dffb640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb5fc06b4d0 0x7fb5fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.973+0000 7fb60dffb640 1 -- 172.21.15.94:0/110039058 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb60808a300 con 0x7fb6180c88c0 2024-03-06T07:19:37.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.973+0000 7fb61d407640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb5fc06b4d0 0x7fb5fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:37.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.974+0000 7fb61d407640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb5fc06b4d0 0x7fb5fc06d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb61400b7c0 tx=0x7fb614008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:37.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:37.978+0000 7fb60dffb640 1 -- 172.21.15.94:0/110039058 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb608058fa0 con 0x7fb6180c88c0 2024-03-06T07:19:38.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.184+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6180b08f0 con 0x7fb6180c88c0 2024-03-06T07:19:38.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.185+0000 7fb60dffb640 1 -- 172.21.15.94:0/110039058 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fb6080588a0 con 0x7fb6180c88c0 2024-03-06T07:19:38.187 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:38.187 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:32.497990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":2}]} 2024-03-06T07:19:38.187 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb5fc06b4d0 msgr2=0x7fb5fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb5fc06b4d0 0x7fb5fc06d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb61400b7c0 tx=0x7fb614008040 comp rx=0 tx=0).stop 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6180c88c0 msgr2=0x7fb6181594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6180c88c0 0x7fb6181594c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c7c0 tx=0x7fb60800cc80 comp rx=0 tx=0).stop 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 shutdown_connections 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb5fc06b4d0 0x7fb5fc06d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb618156ce0 0x7fb6180af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6180c88c0 0x7fb6181594c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 --2- 172.21.15.94:0/110039058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb61806c2e0 0x7fb6181f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 >> 172.21.15.94:0/110039058 conn(0x7fb6180ac570 msgr2=0x7fb6180c04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:38.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 shutdown_connections 2024-03-06T07:19:38.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.187+0000 7fb61f66b640 1 -- 172.21.15.94:0/110039058 wait complete. 2024-03-06T07:19:38.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:19:18.801003+0000', 'modified': '2024-03-06T07:19:32.497990+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5166}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5166': {'gid': 5166, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5166, 'qdb_cluster': [5166]} 2024-03-06T07:19:38.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:19:38.603 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:19:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.725+0000 7f9804119640 1 Processor -- start 2024-03-06T07:19:38.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9804119640 1 -- start start 2024-03-06T07:19:38.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9804119640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f97fc0c69f0 0x7f97fc0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9804119640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f97fc0c8950 0x7f97fc0acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9804119640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9804119640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97fc155440 con 0x7f97fc0c69f0 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9804119640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97fc155130 con 0x7f97fc0c8950 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9804119640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc0c5d50 con 0x7f97fc0ad2b0 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9801eb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f97fc0c69f0 0x7f97fc0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9801eb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f97fc0c69f0 0x7f97fc0c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60338/0 (socket says 172.21.15.94:60338) 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.726+0000 7f9801eb5640 1 -- 172.21.15.94:0/707817940 learned_addr learned my addr 172.21.15.94:0/707817940 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f98026b6640 1 --2- 172.21.15.94:0/707817940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f98016b4640 1 --2- 172.21.15.94:0/707817940 >> v2:172.21.15.145:3300/0 conn(0x7f97fc0c8950 0x7f97fc0acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f9801eb5640 1 -- 172.21.15.94:0/707817940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 msgr2=0x7f97fc1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f9801eb5640 1 --2- 172.21.15.94:0/707817940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f9801eb5640 1 -- 172.21.15.94:0/707817940 >> v2:172.21.15.145:3300/0 conn(0x7f97fc0c8950 msgr2=0x7f97fc0acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f9801eb5640 1 --2- 172.21.15.94:0/707817940 >> v2:172.21.15.145:3300/0 conn(0x7f97fc0c8950 0x7f97fc0acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f9801eb5640 1 -- 172.21.15.94:0/707817940 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0c6030 con 0x7f97fc0c69f0 2024-03-06T07:19:38.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f98026b6640 1 --2- 172.21.15.94:0/707817940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:38.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.727+0000 7f9801eb5640 1 --2- 172.21.15.94:0/707817940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0c6dd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0c7430 tx=0x7f97f402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d560b672ab55019b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:38.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9800eb3640 1 -- 172.21.15.94:0/707817940 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f402e070 con 0x7f97fc0c69f0 2024-03-06T07:19:38.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9800eb3640 1 -- 172.21.15.94:0/707817940 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f4037070 con 0x7f97fc0c69f0 2024-03-06T07:19:38.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9800eb3640 1 -- 172.21.15.94:0/707817940 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4031a00 con 0x7f97fc0c69f0 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 -- 172.21.15.94:0/707817940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 msgr2=0x7f97fc0c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 --2- 172.21.15.94:0/707817940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0c6dd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0c7430 tx=0x7f97f402b880 comp rx=0 tx=0).stop 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 -- 172.21.15.94:0/707817940 shutdown_connections 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 --2- 172.21.15.94:0/707817940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 --2- 172.21.15.94:0/707817940 >> v2:172.21.15.145:3300/0 conn(0x7f97fc0c8950 0x7f97fc0acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 --2- 172.21.15.94:0/707817940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0c6dd0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 -- 172.21.15.94:0/707817940 >> 172.21.15.94:0/707817940 conn(0x7f97fc0cae70 msgr2=0x7f97fc0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 -- 172.21.15.94:0/707817940 shutdown_connections 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.728+0000 7f9804119640 1 -- 172.21.15.94:0/707817940 wait complete. 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.729+0000 7f9804119640 1 Processor -- start 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.729+0000 7f9804119640 1 -- start start 2024-03-06T07:19:38.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f9804119640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc06c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f9804119640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f9804119640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97fc0c8950 0x7f97fc0afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f9804119640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc152ba0 con 0x7f97fc0c69f0 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f9804119640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc152a20 con 0x7f97fc0c8950 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f9804119640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc0c58e0 con 0x7f97fc0ad2b0 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f98016b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f98016b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0af880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60340/0 (socket says 172.21.15.94:60340) 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f98016b4640 1 -- 172.21.15.94:0/3932253203 learned_addr learned my addr 172.21.15.94:0/3932253203 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f9801eb5640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.730+0000 7f98026b6640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97fc0c8950 0x7f97fc0afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f98016b4640 1 -- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 msgr2=0x7f97fc06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f98016b4640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f98016b4640 1 -- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97fc0c8950 msgr2=0x7f97fc0afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f98016b4640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97fc0c8950 0x7f97fc0afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f98016b4640 1 -- 172.21.15.94:0/3932253203 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0aee20 con 0x7f97fc0c69f0 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f98026b6640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97fc0c8950 0x7f97fc0afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f9801eb5640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f98016b4640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0af880 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c7f0 tx=0x7f97f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f97fa7fc640 1 -- 172.21.15.94:0/3932253203 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0015070 con 0x7f97fc0c69f0 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f97fa7fc640 1 -- 172.21.15.94:0/3932253203 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f0004220 con 0x7f97fc0c69f0 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f97fa7fc640 1 -- 172.21.15.94:0/3932253203 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0007620 con 0x7f97fc0c69f0 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc0af100 con 0x7f97fc0c69f0 2024-03-06T07:19:38.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.731+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc1565c0 con 0x7f97fc0c69f0 2024-03-06T07:19:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.732+0000 7f97fa7fc640 1 -- 172.21.15.94:0/3932253203 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97f00079d0 con 0x7f97fc0c69f0 2024-03-06T07:19:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.733+0000 7f97fa7fc640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97e806b500 0x7f97e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.733+0000 7f97fa7fc640 1 -- 172.21.15.94:0/3932253203 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f97f008a340 con 0x7f97fc0c69f0 2024-03-06T07:19:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.733+0000 7f9801eb5640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97e806b500 0x7f97e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.734+0000 7f9801eb5640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97e806b500 0x7f97e806d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f97f4036000 tx=0x7f97f403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:38.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.735+0000 7f9806dd6740 1 Processor -- start 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.735+0000 7f9806dd6740 1 -- start start 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a738a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a738f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a7394b0 0x56246a735680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f9806dd6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56246a69f6b0 con 0x56246a73b330 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56246a648a60 con 0x56246a7394b0 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56246a69b750 con 0x56246a73af50 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a738f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a738f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60354/0 (socket says 172.21.15.94:60354) 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 -- 172.21.15.94:0/3935580187 learned_addr learned my addr 172.21.15.94:0/3935580187 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f9801eb5640 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a738a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98026b6640 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a7394b0 0x56246a735680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 -- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 msgr2=0x56246a738a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a738a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 -- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a7394b0 msgr2=0x56246a735680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a7394b0 0x56246a735680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 -- 172.21.15.94:0/3935580187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56246a666f30 con 0x56246a73b330 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f98016b4640 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a738f70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f97f0054dd0 tx=0x7f97f00591b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12deeda9080e82dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.736+0000 7f97e37fe640 1 -- 172.21.15.94:0/3935580187 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f001e040 con 0x56246a73b330 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f97e37fe640 1 -- 172.21.15.94:0/3935580187 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f0056c70 con 0x56246a73b330 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f97e37fe640 1 -- 172.21.15.94:0/3935580187 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f0054fe0 con 0x56246a73b330 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 -- 172.21.15.94:0/3935580187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 msgr2=0x56246a738f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a738f70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f97f0054dd0 tx=0x7f97f00591b0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 -- 172.21.15.94:0/3935580187 shutdown_connections 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a7394b0 0x56246a735680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a738f70 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 --2- 172.21.15.94:0/3935580187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a738a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 -- 172.21.15.94:0/3935580187 >> 172.21.15.94:0/3935580187 conn(0x56246a7377d0 msgr2=0x56246a75c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:38.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 -- 172.21.15.94:0/3935580187 shutdown_connections 2024-03-06T07:19:38.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.737+0000 7f9806dd6740 1 -- 172.21.15.94:0/3935580187 wait complete. 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.738+0000 7f9806dd6740 1 -- start start 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.738+0000 7f9806dd6740 1 Processor -- start 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.739+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a75b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.739+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a78aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.739+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a75f270 0x56246a757fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.739+0000 7f9806dd6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56246a75d230 con 0x56246a73b330 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.739+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56246a65bea0 con 0x56246a75f270 2024-03-06T07:19:38.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.739+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56246a75cc70 con 0x56246a73af50 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f9806dd6740 10 client.0 _ll_register_callbacks cb 0x56246a65b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a78aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a78aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60370/0 (socket says 172.21.15.94:60370) 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 -- 172.21.15.94:0/4103552589 learned_addr learned my addr 172.21.15.94:0/4103552589 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f9801eb5640 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a75b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 -- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 msgr2=0x56246a75b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98026b6640 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a75f270 0x56246a757fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a75b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 -- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a75f270 msgr2=0x56246a757fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a75f270 0x56246a757fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 -- 172.21.15.94:0/4103552589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56246a65c270 con 0x56246a73b330 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f98016b4640 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a78aad0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f97f0058b30 tx=0x7f97f001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f9cfa24f79a8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f97e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:38.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f97e0ff9640 1 -- 172.21.15.94:0/4103552589 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f001e070 con 0x56246a73b330 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f97e0ff9640 1 -- 172.21.15.94:0/4103552589 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f97f0054fe0 con 0x56246a73b330 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97f0056c70 con 0x56246a73b330 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f97e0ff9640 1 -- 172.21.15.94:0/4103552589 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97f00701d0 con 0x56246a73b330 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f9806dd6740 10 client.5481 fetch_fsmap learned FSMap version 23 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x56246a3b42d0 con 0x56246a73b330 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.740+0000 7f97e0ff9640 1 -- 172.21.15.94:0/4103552589 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f0056c70 con 0x56246a73b330 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f97e0ff9640 1 -- 172.21.15.94:0/4103552589 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f97f0070370 con 0x56246a73b330 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9806dd6740 10 client.5481 fetch_fsmap finished waiting for FSMap version 23 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9806dd6740 10 client.5481 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9806dd6740 20 client.5481 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9806dd6740 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a75dad0 0x56246a75deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9806dd6740 4 client.5481 mds_command: new command op to 5166 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:19:38.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 --> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56246a6acbf0 con 0x56246a75dad0 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9801eb5640 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a75dad0 0x56246a75deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f97e0ff9640 10 client.5481 ms_handle_connect on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.741+0000 7f9801eb5640 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a75dad0 0x56246a75deb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea9604f493ea4ef0 server_cookie=1637ac7f695d73cc in_seq=0 out_seq=0 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f97e0ff9640 1 -- 172.21.15.94:0/4103552589 <== mds.0 v2:172.21.15.94:6833/2367914435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56246a6acbf0 con 0x56246a75dad0 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f97e0ff9640 10 client.5481 handle_command_reply: tid=0 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 client.5481 shutdown 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 msgr2=0x56246a78aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a78aad0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f97f0058b30 tx=0x7f97f001c000 comp rx=0 tx=0).stop 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 shutdown_connections 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a75dad0 0x56246a75deb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a75f270 0x56246a757fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a73b330 0x56246a78aad0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f97e0ff9640 0 client.5481 ms_handle_reset on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 --2- 172.21.15.94:0/4103552589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a75b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 >> 172.21.15.94:0/4103552589 conn(0x56246a7394b0 msgr2=0x56246a7398e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 shutdown_connections 2024-03-06T07:19:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 1 -- 172.21.15.94:0/4103552589 wait complete. 2024-03-06T07:19:38.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.742+0000 7f9806dd6740 20 client.5481 trim_cache size 0 max 16384 2024-03-06T07:19:38.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.759+0000 7f9806dd6740 1 Processor -- start 2024-03-06T07:19:38.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.759+0000 7f9806dd6740 1 -- start start 2024-03-06T07:19:38.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a739d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73b330 0x56246a734510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a784a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f9806dd6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1f7030 con 0x56246a75f270 2024-03-06T07:19:38.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc14ce90 con 0x56246a73b330 2024-03-06T07:19:38.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1ef560 con 0x56246a73af50 2024-03-06T07:19:38.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a784a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a784a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60380/0 (socket says 172.21.15.94:60380) 2024-03-06T07:19:38.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 -- 172.21.15.94:0/141567731 learned_addr learned my addr 172.21.15.94:0/141567731 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:38.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 -- 172.21.15.94:0/141567731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 msgr2=0x56246a739d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 --2- 172.21.15.94:0/141567731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a739d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 -- 172.21.15.94:0/141567731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73b330 msgr2=0x56246a734510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 --2- 172.21.15.94:0/141567731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73b330 0x56246a734510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 -- 172.21.15.94:0/141567731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56246a47e6b0 con 0x56246a75f270 2024-03-06T07:19:38.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f98026b6640 1 --2- 172.21.15.94:0/141567731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a784a30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f97ec007130 tx=0x7f97ec00f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30748278a0721555 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:38.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f97e2ffd640 1 -- 172.21.15.94:0/141567731 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec00d430 con 0x56246a75f270 2024-03-06T07:19:38.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f97e2ffd640 1 -- 172.21.15.94:0/141567731 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec00d5f0 con 0x56246a75f270 2024-03-06T07:19:38.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.760+0000 7f97e2ffd640 1 -- 172.21.15.94:0/141567731 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec00d9e0 con 0x56246a75f270 2024-03-06T07:19:38.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 -- 172.21.15.94:0/141567731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 msgr2=0x56246a784a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 --2- 172.21.15.94:0/141567731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a784a30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f97ec007130 tx=0x7f97ec00f820 comp rx=0 tx=0).stop 2024-03-06T07:19:38.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 -- 172.21.15.94:0/141567731 shutdown_connections 2024-03-06T07:19:38.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 --2- 172.21.15.94:0/141567731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a784a30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 --2- 172.21.15.94:0/141567731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73b330 0x56246a734510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 --2- 172.21.15.94:0/141567731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73af50 0x56246a739d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 -- 172.21.15.94:0/141567731 >> 172.21.15.94:0/141567731 conn(0x56246a7394b0 msgr2=0x56246a75b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:38.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 -- 172.21.15.94:0/141567731 shutdown_connections 2024-03-06T07:19:38.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.761+0000 7f9806dd6740 1 -- 172.21.15.94:0/141567731 wait complete. 2024-03-06T07:19:38.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.762+0000 7f9806dd6740 1 -- start start 2024-03-06T07:19:38.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.762+0000 7f9806dd6740 1 Processor -- start 2024-03-06T07:19:38.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73af50 0x56246a734910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73b330 0x56246a783a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9806dd6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a7e38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9806dd6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56246a65bea0 con 0x56246a75f270 2024-03-06T07:19:38.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56246a761b90 con 0x56246a73af50 2024-03-06T07:19:38.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9806dd6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56246a75d230 con 0x56246a73b330 2024-03-06T07:19:38.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9806dd6740 10 client.0 _ll_register_callbacks cb 0x7f97f005cc00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:19:38.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a7e38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a7e38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60394/0 (socket says 172.21.15.94:60394) 2024-03-06T07:19:38.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 -- 172.21.15.94:0/941932661 learned_addr learned my addr 172.21.15.94:0/941932661 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:38.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98016b4640 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73b330 0x56246a783a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f9801eb5640 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73af50 0x56246a734910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 -- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73b330 msgr2=0x56246a783a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73b330 0x56246a783a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 -- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73af50 msgr2=0x56246a734910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:38.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73af50 0x56246a734910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 -- 172.21.15.94:0/941932661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56246a777d10 con 0x56246a75f270 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f98026b6640 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a7e38c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f97fc074a80 tx=0x7f97ec004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c435e9e851be60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.763+0000 7f97e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f97e0ff9640 1 -- 172.21.15.94:0/941932661 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec027030 con 0x56246a75f270 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f97e0ff9640 1 -- 172.21.15.94:0/941932661 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f97ec01e050 con 0x56246a75f270 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97ec00d5f0 con 0x56246a75f270 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f97e0ff9640 1 -- 172.21.15.94:0/941932661 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97ec030050 con 0x56246a75f270 2024-03-06T07:19:38.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 10 client.5487 fetch_fsmap learned FSMap version 23 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x56246a738570 con 0x56246a75f270 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f97e0ff9640 1 -- 172.21.15.94:0/941932661 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec00d5f0 con 0x56246a75f270 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f97e0ff9640 1 -- 172.21.15.94:0/941932661 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 23) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f97ec033020 con 0x56246a75f270 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 10 client.5487 fetch_fsmap finished waiting for FSMap version 23 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 10 client.5487 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 20 client.5487 populate_metadata read hostname 'smithi094' 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a79a130 0x56246a79a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:38.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 4 client.5487 mds_command: new command op to 5166 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 --> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56246a73a9a0 con 0x56246a79a130 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9801eb5640 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a79a130 0x56246a79a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f97e0ff9640 10 client.5487 ms_handle_connect on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.764+0000 7f9801eb5640 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a79a130 0x56246a79a510 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76920d55030c1b70 server_cookie=24eb4dcd14d50d6d in_seq=0 out_seq=0 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f97e0ff9640 1 -- 172.21.15.94:0/941932661 <== mds.0 v2:172.21.15.94:6833/2367914435 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x56246a73a9a0 con 0x56246a79a130 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f97e0ff9640 10 client.5487 handle_command_reply: tid=0 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 client.5487 shutdown 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 msgr2=0x56246a7e38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a7e38c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f97fc074a80 tx=0x7f97ec004ab0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 shutdown_connections 2024-03-06T07:19:38.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:6833/2367914435,v1:172.21.15.94:6835/2367914435] conn(0x56246a79a130 0x56246a79a510 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56246a75f270 0x56246a7e38c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f97e0ff9640 0 client.5487 ms_handle_reset on v2:172.21.15.94:6833/2367914435 2024-03-06T07:19:38.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56246a73b330 0x56246a783a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 --2- 172.21.15.94:0/941932661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56246a73af50 0x56246a734910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 >> 172.21.15.94:0/941932661 conn(0x56246a7394b0 msgr2=0x56246a75e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:38.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 shutdown_connections 2024-03-06T07:19:38.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 1 -- 172.21.15.94:0/941932661 wait complete. 2024-03-06T07:19:38.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9806dd6740 20 client.5487 trim_cache size 0 max 16384 2024-03-06T07:19:38.786 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:19:38.786 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:19:38.786 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:19:38.786 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 5166, 2024-03-06T07:19:38.786 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:19:38.787 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:19:38.787 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:19:38.787 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 12.278271226999999, 2024-03-06T07:19:38.787 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 23, 2024-03-06T07:19:38.787 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 40, 2024-03-06T07:19:38.787 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 40, 2024-03-06T07:19:38.787 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 33.214883952999998 2024-03-06T07:19:38.788 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:19:38.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97e806b500 msgr2=0x7f97e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9804119640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97e806b500 0x7f97e806d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f97f4036000 tx=0x7f97f403d040 comp rx=0 tx=0).stop 2024-03-06T07:19:38.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 msgr2=0x7f97fc0af880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:38.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.765+0000 7f9804119640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0af880 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f97f000c7f0 tx=0x7f97f000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 shutdown_connections 2024-03-06T07:19:38.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97e806b500 0x7f97e806d9c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97fc0c8950 0x7f97fc0afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97fc0c69f0 0x7f97fc0af880 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 --2- 172.21.15.94:0/3932253203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97fc0ad2b0 0x7f97fc06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:38.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 >> 172.21.15.94:0/3932253203 conn(0x7f97fc0cae70 msgr2=0x7f97fc0c8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:38.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 shutdown_connections 2024-03-06T07:19:38.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:38.766+0000 7f9804119640 1 -- 172.21.15.94:0/3932253203 wait complete. 2024-03-06T07:19:38.795 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:19:18.801003+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5166": { "addr": "172.21.15.94:6835/2367914435", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 2367914435, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 2367914435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5166, "incarnation": 19, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-03-06T07:19:32.497990+0000", "qdb_cluster": [ 5166 ], "qdb_leader": 5166, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5166 } } } ], "standbys": [ { "addr": "172.21.15.145:6835/1902381441", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1902381441, "type": "v2" }, { "addr": "172.21.15.145:6835", "nonce": 1902381441, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5215, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/856474008", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6836", "nonce": 856474008, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 856474008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/597543014", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 597543014, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 597543014, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 34814, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:19:38.797 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:19:38.797 INFO:tasks.cephfs.test_failover:current = [{'gid': 34814, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/597543014', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 597543014}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 597543014}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 5410, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/856474008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 856474008}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 856474008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 20}, {'gid': 5166, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.94:6835/2367914435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2367914435}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2367914435}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5215, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6835/1902381441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1902381441}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 1902381441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}] 2024-03-06T07:19:38.797 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:19:38.797 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:19:38.798 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:19:38.849 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:19:38.900 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:19:38.900 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:19:38.900 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:19:38.917 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:19:38.917 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:19:38.917 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:19:38.934 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:19:38.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:39.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.058+0000 7fb3d84f4640 1 Processor -- start 2024-03-06T07:19:39.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.058+0000 7fb3d84f4640 1 -- start start 2024-03-06T07:19:39.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d84f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 0x7fb3d0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d84f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb3d00c88c0 0x7fb3d00c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d84f4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00c73d0 con 0x7fb3d00c5880 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d84f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d00c8dd0 con 0x7fb3d00c88c0 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d84f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00be2e0 con 0x7fb3d014f360 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d6a91640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb3d00c5880 0x7fb3d0158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35938/0 (socket says 172.21.15.94:35938) 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d6a91640 1 -- 172.21.15.94:0/2612456815 learned_addr learned my addr 172.21.15.94:0/2612456815 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534681328 0 0) 0x7fb3d00c73d0 con 0x7fb3d00c5880 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3bc003520 con 0x7fb3d00c5880 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d6290640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 0x7fb3d0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2612456815 >> v2:172.21.15.145:3300/0 conn(0x7fb3d00c88c0 0x7fb3d00c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4104097654 0 0) 0x7fb3bc003520 con 0x7fb3d00c5880 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 msgr2=0x7fb3d0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:39.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 0x7fb3d0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 >> v2:172.21.15.145:3300/0 conn(0x7fb3d00c88c0 msgr2=0x7fb3d00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 --2- 172.21.15.94:0/2612456815 >> v2:172.21.15.145:3300/0 conn(0x7fb3d00c88c0 0x7fb3d00c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00c7670 con 0x7fb3d00c5880 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fb3c80019f0 con 0x7fb3d00c5880 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 >> v1:172.21.15.94:6789/0 conn(0x7fb3d00c5880 legacy=0x7fb3d0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 0x7fb3bc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3bc003f30 0x7fb3bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc004d40 0x7fb3bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00be2e0 con 0x7fb3bc003f30 2024-03-06T07:19:39.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc00b9c0 con 0x7fb3bc004d40 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.059+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc00bb90 con 0x7fb3d014f360 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3bc003f30 0x7fb3bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d6290640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 0x7fb3bc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d6a91640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc004d40 0x7fb3bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d5a8f640 1 -- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 msgr2=0x7fb3bc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3d014f360 0x7fb3bc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d5a8f640 1 -- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc004d40 msgr2=0x7fb3bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc004d40 0x7fb3bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.060+0000 7fb3d5a8f640 1 -- 172.21.15.94:0/2612456815 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc003520 con 0x7fb3bc003f30 2024-03-06T07:19:39.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3bc003f30 0x7fb3bc004800 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009ac0 tx=0x7fb3bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d963a62bd4d9fc32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0039070 con 0x7fb3bc003f30 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c002c6b0 con 0x7fb3bc003f30 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d528e640 1 -- 172.21.15.94:0/2612456815 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0009690 con 0x7fb3bc003f30 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2612456815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3bc003f30 msgr2=0x7fb3bc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d84f4640 1 --2- 172.21.15.94:0/2612456815 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3bc003f30 0x7fb3bc004800 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0009ac0 tx=0x7fb3bc005830 comp rx=0 tx=0).stop 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d6a91640 1 -- 172.21.15.94:0/2612456815 reap_dead start 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2612456815 shutdown_connections 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2612456815 >> 172.21.15.94:0/2612456815 conn(0x7fb3d00ac570 msgr2=0x7fb3d00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2612456815 shutdown_connections 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.061+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2612456815 wait complete. 2024-03-06T07:19:39.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.062+0000 7fb3d84f4640 1 Processor -- start 2024-03-06T07:19:39.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.062+0000 7fb3d84f4640 1 -- start start 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d84f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc003f30 0x7fb3d01eb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d84f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d0156330 0x7fb3d00beee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d84f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3d00c1a90 0x7fb3d00c4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d84f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d0159270 con 0x7fb3d0156330 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d84f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00c7c30 con 0x7fb3d00c1a90 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d84f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01593f0 con 0x7fb3bc003f30 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d5a8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d0156330 0x7fb3d00beee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d5a8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d0156330 0x7fb3d00beee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60416/0 (socket says 172.21.15.94:60416) 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d5a8f640 1 -- 172.21.15.94:0/2902155466 learned_addr learned my addr 172.21.15.94:0/2902155466 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:39.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d6290640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc003f30 0x7fb3d01eb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.063+0000 7fb3d6a91640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3d00c1a90 0x7fb3d00c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d5a8f640 1 -- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc003f30 msgr2=0x7fb3d01eb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc003f30 0x7fb3d01eb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d5a8f640 1 -- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3d00c1a90 msgr2=0x7fb3d00c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3d00c1a90 0x7fb3d00c4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d5a8f640 1 -- 172.21.15.94:0/2902155466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00c3510 con 0x7fb3d0156330 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d5a8f640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d0156330 0x7fb3d00beee0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0000c00 tx=0x7fb3c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3c6ffd640 1 -- 172.21.15.94:0/2902155466 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0039050 con 0x7fb3d0156330 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3c6ffd640 1 -- 172.21.15.94:0/2902155466 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c0045070 con 0x7fb3d0156330 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d0158530 con 0x7fb3d0156330 2024-03-06T07:19:39.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3c6ffd640 1 -- 172.21.15.94:0/2902155466 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c00044e0 con 0x7fb3d0156330 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.064+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d00c3a90 con 0x7fb3d0156330 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.065+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d004f330 con 0x7fb3d0156330 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.066+0000 7fb3c6ffd640 1 -- 172.21.15.94:0/2902155466 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3c0003a50 con 0x7fb3d0156330 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.066+0000 7fb3c6ffd640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb3b406b4d0 0x7fb3b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.066+0000 7fb3c6ffd640 1 -- 172.21.15.94:0/2902155466 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb3c00ac670 con 0x7fb3d0156330 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.066+0000 7fb3d6290640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb3b406b4d0 0x7fb3b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.067+0000 7fb3d6290640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb3b406b4d0 0x7fb3b406d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009a20 tx=0x7fb3cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.070+0000 7fb3c6ffd640 1 -- 172.21.15.94:0/2902155466 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3c004e020 con 0x7fb3d0156330 2024-03-06T07:19:39.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.277+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3d006cda0 con 0x7fb3d0156330 2024-03-06T07:19:39.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.277+0000 7fb3c6ffd640 1 -- 172.21.15.94:0/2902155466 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fb3c007b5a0 con 0x7fb3d0156330 2024-03-06T07:19:39.280 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:39.280 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:32.497990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":2}]} 2024-03-06T07:19:39.280 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-03-06T07:19:39.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb3b406b4d0 msgr2=0x7fb3b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb3b406b4d0 0x7fb3b406d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009a20 tx=0x7fb3cc008040 comp rx=0 tx=0).stop 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d0156330 msgr2=0x7fb3d00beee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d0156330 0x7fb3d00beee0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0000c00 tx=0x7fb3c0004730 comp rx=0 tx=0).stop 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 shutdown_connections 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb3b406b4d0 0x7fb3b406d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3d00c1a90 0x7fb3d00c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3d0156330 0x7fb3d00beee0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 --2- 172.21.15.94:0/2902155466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc003f30 0x7fb3d01eb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 >> 172.21.15.94:0/2902155466 conn(0x7fb3d00ac570 msgr2=0x7fb3d006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 shutdown_connections 2024-03-06T07:19:39.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.279+0000 7fb3d84f4640 1 -- 172.21.15.94:0/2902155466 wait complete. 2024-03-06T07:19:39.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:39.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.393+0000 7f30c55f6640 1 Processor -- start 2024-03-06T07:19:39.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.393+0000 7f30c55f6640 1 -- start start 2024-03-06T07:19:39.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30c55f6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f30c00c88b0 0x7f30c00c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30c55f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c0155110 con 0x7f30c00c5330 2024-03-06T07:19:39.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30c55f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00c8f40 con 0x7f30c0158db0 2024-03-06T07:19:39.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30c55f6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c01559e0 con 0x7f30c00c88b0 2024-03-06T07:19:39.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30bf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f30c00c88b0 0x7f30c00c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30bf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f30c00c88b0 0x7f30c00c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60422/0 (socket says 172.21.15.94:60422) 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30beffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f30c00c5330 0x7f30c01586a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35944/0 (socket says 172.21.15.94:35944) 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30bf7fe640 1 -- 172.21.15.94:0/4249014892 learned_addr learned my addr 172.21.15.94:0/4249014892 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30bf7fe640 1 -- 172.21.15.94:0/4249014892 >> v1:172.21.15.145:6789/0 conn(0x7f30c0158db0 legacy=0x7f30c015da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30bf7fe640 1 -- 172.21.15.94:0/4249014892 >> v1:172.21.15.94:6789/0 conn(0x7f30c00c5330 legacy=0x7f30c01586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.394+0000 7f30bf7fe640 1 -- 172.21.15.94:0/4249014892 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0155390 con 0x7f30c00c88b0 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.395+0000 7f30bf7fe640 1 --2- 172.21.15.94:0/4249014892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c00c88b0 0x7f30c00c5710 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009e50 tx=0x7f30b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6453f5c9f7474fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.395+0000 7f30bdffb640 1 -- 172.21.15.94:0/4249014892 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b802ca80 con 0x7f30c00c88b0 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.395+0000 7f30bdffb640 1 -- 172.21.15.94:0/4249014892 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b802cc20 con 0x7f30c00c88b0 2024-03-06T07:19:39.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.395+0000 7f30bdffb640 1 -- 172.21.15.94:0/4249014892 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80343f0 con 0x7f30c00c88b0 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.396+0000 7f30c55f6640 1 -- 172.21.15.94:0/4249014892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c00c88b0 msgr2=0x7f30c00c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.396+0000 7f30c55f6640 1 --2- 172.21.15.94:0/4249014892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c00c88b0 0x7f30c00c5710 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009e50 tx=0x7f30b802bee0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.396+0000 7f30c55f6640 1 -- 172.21.15.94:0/4249014892 shutdown_connections 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.396+0000 7f30c55f6640 1 --2- 172.21.15.94:0/4249014892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c00c88b0 0x7f30c00c5710 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.396+0000 7f30c55f6640 1 -- 172.21.15.94:0/4249014892 >> 172.21.15.94:0/4249014892 conn(0x7f30c00ac560 msgr2=0x7f30c00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.396+0000 7f30c55f6640 1 -- 172.21.15.94:0/4249014892 shutdown_connections 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.396+0000 7f30c55f6640 1 -- 172.21.15.94:0/4249014892 wait complete. 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.397+0000 7f30c55f6640 1 Processor -- start 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.397+0000 7f30c55f6640 1 -- start start 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30c55f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c00c5330 0x7f30c0148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30c55f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c00c88b0 0x7f30c00ab510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30c55f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c0149160 0x7f30c0156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30c55f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01529d0 con 0x7f30c0149160 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30c55f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c006d090 con 0x7f30c00c88b0 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30c55f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0152b50 con 0x7f30c00c5330 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c0149160 0x7f30c0156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c0149160 0x7f30c0156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60438/0 (socket says 172.21.15.94:60438) 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 -- 172.21.15.94:0/2438907596 learned_addr learned my addr 172.21.15.94:0/2438907596 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30be7fc640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c00c88b0 0x7f30c00ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 -- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c00c5330 msgr2=0x7f30c0148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30beffd640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c00c5330 0x7f30c0148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c00c5330 0x7f30c0148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 -- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c00c88b0 msgr2=0x7f30c00ab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c00c88b0 0x7f30c00ab510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 -- 172.21.15.94:0/2438907596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0151fa0 con 0x7f30c0149160 2024-03-06T07:19:39.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30bf7fe640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c0149160 0x7f30c0156460 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009e50 tx=0x7f30b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.398+0000 7f30a77fe640 1 -- 172.21.15.94:0/2438907596 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8043020 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.399+0000 7f30a77fe640 1 -- 172.21.15.94:0/2438907596 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b803e450 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.399+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c0158520 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.399+0000 7f30a77fe640 1 -- 172.21.15.94:0/2438907596 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8009510 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.399+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c014eac0 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.399+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c004f330 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.400+0000 7f30a77fe640 1 -- 172.21.15.94:0/2438907596 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30b8046020 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.400+0000 7f30a77fe640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f30ac06b4d0 0x7f30ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.401+0000 7f30a77fe640 1 -- 172.21.15.94:0/2438907596 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f30b80acc20 con 0x7f30c0149160 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.401+0000 7f30beffd640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f30ac06b4d0 0x7f30ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.401+0000 7f30beffd640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f30ac06b4d0 0x7f30ac06d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f30c0154bb0 tx=0x7f30b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.409+0000 7f30a77fe640 1 -- 172.21.15.94:0/2438907596 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30b804d030 con 0x7f30c0149160 2024-03-06T07:19:39.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.587+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30c00ca470 con 0x7f30c0149160 2024-03-06T07:19:39.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.588+0000 7f30a77fe640 1 -- 172.21.15.94:0/2438907596 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17073 (secure 0 0 0) 0x7f30b807b950 con 0x7f30c0149160 2024-03-06T07:19:39.590 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:39.590 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":40,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:26.486055+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:17.283570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:17.762319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/1305657307":"2024-03-07T07:19:26.486020+0000","172.21.15.145:6836/1305657307":"2024-03-07T07:19:26.486020+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:39.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f30ac06b4d0 msgr2=0x7f30ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f30ac06b4d0 0x7f30ac06d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f30c0154bb0 tx=0x7f30b4009040 comp rx=0 tx=0).stop 2024-03-06T07:19:39.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c0149160 msgr2=0x7f30c0156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c0149160 0x7f30c0156460 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009e50 tx=0x7f30b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:39.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 shutdown_connections 2024-03-06T07:19:39.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f30ac06b4d0 0x7f30ac06d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30c0149160 0x7f30c0156460 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c00c88b0 0x7f30c00ab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 --2- 172.21.15.94:0/2438907596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c00c5330 0x7f30c0148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 >> 172.21.15.94:0/2438907596 conn(0x7f30c00ac560 msgr2=0x7f30c00ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:39.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 shutdown_connections 2024-03-06T07:19:39.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.590+0000 7f30c55f6640 1 -- 172.21.15.94:0/2438907596 wait complete. 2024-03-06T07:19:39.604 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:19:17.283570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:19:17.762319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:19:39.604 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:19:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.707+0000 7f710d77a640 1 Processor -- start 2024-03-06T07:19:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.707+0000 7f710d77a640 1 -- start start 2024-03-06T07:19:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.707+0000 7f710d77a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f710806bff0 0x7f71080c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f710d77a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71080c7210 0x7f71080c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f710d77a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71080b04b0 con 0x7f71080b00d0 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f710d77a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7108156160 con 0x7f710806bff0 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f710d77a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71080c5710 con 0x7f71080c7210 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f70fffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f710806bff0 0x7f71080c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f7106ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f71080b00d0 0x7f710806b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35958/0 (socket says 172.21.15.94:35958) 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f70fffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f710806bff0 0x7f71080c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60446/0 (socket says 172.21.15.94:60446) 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f7106ffd640 1 -- 172.21.15.94:0/3238481706 learned_addr learned my addr 172.21.15.94:0/3238481706 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f71077fe640 1 --2- 172.21.15.94:0/3238481706 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71080c7210 0x7f71080c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f70fffff640 1 -- 172.21.15.94:0/3238481706 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71080c7210 msgr2=0x7f71080c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f70fffff640 1 --2- 172.21.15.94:0/3238481706 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71080c7210 0x7f71080c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f70fffff640 1 -- 172.21.15.94:0/3238481706 >> v1:172.21.15.94:6789/0 conn(0x7f71080b00d0 legacy=0x7f710806b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f71067fc640 1 -- 172.21.15.94:0/3238481706 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541273486 0 0) 0x7f71080b04b0 con 0x7f71080b00d0 2024-03-06T07:19:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.708+0000 7f70fffff640 1 -- 172.21.15.94:0/3238481706 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710806d210 con 0x7f710806bff0 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.709+0000 7f70fffff640 1 --2- 172.21.15.94:0/3238481706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f710806bff0 0x7f71080c8ff0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f70f00097a0 tx=0x7f70f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49649d8f33b70a37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.709+0000 7f71067fc640 1 -- 172.21.15.94:0/3238481706 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0035020 con 0x7f710806bff0 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.709+0000 7f71067fc640 1 -- 172.21.15.94:0/3238481706 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f0031830 con 0x7f710806bff0 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.709+0000 7f71067fc640 1 -- 172.21.15.94:0/3238481706 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0031c00 con 0x7f710806bff0 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.709+0000 7f710d77a640 1 -- 172.21.15.94:0/3238481706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f710806bff0 msgr2=0x7f71080c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.709+0000 7f710d77a640 1 --2- 172.21.15.94:0/3238481706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f710806bff0 0x7f71080c8ff0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f70f00097a0 tx=0x7f70f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.710+0000 7f710d77a640 1 -- 172.21.15.94:0/3238481706 shutdown_connections 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.710+0000 7f710d77a640 1 --2- 172.21.15.94:0/3238481706 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71080c7210 0x7f71080c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.710+0000 7f710d77a640 1 --2- 172.21.15.94:0/3238481706 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f710806bff0 0x7f71080c8ff0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.710+0000 7f710d77a640 1 -- 172.21.15.94:0/3238481706 >> 172.21.15.94:0/3238481706 conn(0x7f71080ac810 msgr2=0x7f71080acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.710+0000 7f710d77a640 1 -- 172.21.15.94:0/3238481706 shutdown_connections 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.710+0000 7f710d77a640 1 -- 172.21.15.94:0/3238481706 wait complete. 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 Processor -- start 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 -- start start 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f710806bff0 0x7f710814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71080b00d0 0x7f71081ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71080c7210 0x7f71081ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71080c5ba0 con 0x7f71080b00d0 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108152c50 con 0x7f71080c7210 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f710d77a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71080c5d20 con 0x7f710806bff0 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f70fffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71080b00d0 0x7f71081ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f70fffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71080b00d0 0x7f71081ee380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60448/0 (socket says 172.21.15.94:60448) 2024-03-06T07:19:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f70fffff640 1 -- 172.21.15.94:0/3855089021 learned_addr learned my addr 172.21.15.94:0/3855089021 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.711+0000 7f7106ffd640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f710806bff0 0x7f710814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f71077fe640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71080c7210 0x7f71081ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70fffff640 1 -- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f710806bff0 msgr2=0x7f710814faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70fffff640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f710806bff0 0x7f710814faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70fffff640 1 -- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71080c7210 msgr2=0x7f71081ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70fffff640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71080c7210 0x7f71081ee8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70fffff640 1 -- 172.21.15.94:0/3855089021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71080c1aa0 con 0x7f71080b00d0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f7106ffd640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f710806bff0 0x7f710814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70fffff640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71080b00d0 0x7f71081ee380 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f70f0000c00 tx=0x7f70f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70ff7fe640 1 -- 172.21.15.94:0/3855089021 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0035070 con 0x7f71080b00d0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70ff7fe640 1 -- 172.21.15.94:0/3855089021 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f0004230 con 0x7f71080b00d0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71081ef280 con 0x7f71080b00d0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f70ff7fe640 1 -- 172.21.15.94:0/3855089021 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f003a6f0 con 0x7f71080b00d0 2024-03-06T07:19:39.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.712+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710814f7c0 con 0x7f71080b00d0 2024-03-06T07:19:39.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.713+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710804f330 con 0x7f71080b00d0 2024-03-06T07:19:39.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.714+0000 7f70ff7fe640 1 -- 172.21.15.94:0/3855089021 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70f0038020 con 0x7f71080b00d0 2024-03-06T07:19:39.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.714+0000 7f70ff7fe640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e806b4d0 0x7f70e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:39.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.714+0000 7f70ff7fe640 1 -- 172.21.15.94:0/3855089021 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f70f00ac040 con 0x7f71080b00d0 2024-03-06T07:19:39.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.714+0000 7f7106ffd640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e806b4d0 0x7f70e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:39.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.715+0000 7f7106ffd640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e806b4d0 0x7f70e806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f70f8000f60 tx=0x7f70f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:39.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.719+0000 7f70ff7fe640 1 -- 172.21.15.94:0/3855089021 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70f0040070 con 0x7f71080b00d0 2024-03-06T07:19:39.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.895+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f710806ce50 con 0x7f71080b00d0 2024-03-06T07:19:39.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.895+0000 7f70ff7fe640 1 -- 172.21.15.94:0/3855089021 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f70f007ad70 con 0x7f71080b00d0 2024-03-06T07:19:39.897 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e806b4d0 msgr2=0x7f70e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e806b4d0 0x7f70e806d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f70f8000f60 tx=0x7f70f8009040 comp rx=0 tx=0).stop 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71080b00d0 msgr2=0x7f71081ee380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71080b00d0 0x7f71081ee380 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f70f0000c00 tx=0x7f70f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 shutdown_connections 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e806b4d0 0x7f70e806d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71080c7210 0x7f71081ee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71080b00d0 0x7f71081ee380 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 --2- 172.21.15.94:0/3855089021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f710806bff0 0x7f710814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:39.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 >> 172.21.15.94:0/3855089021 conn(0x7f71080ac810 msgr2=0x7f710806b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:39.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 shutdown_connections 2024-03-06T07:19:39.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:39.897+0000 7f710d77a640 1 -- 172.21.15.94:0/3855089021 wait complete. 2024-03-06T07:19:39.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:19:40.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.013+0000 7f6f8631e640 1 Processor -- start 2024-03-06T07:19:40.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.013+0000 7f6f8631e640 1 -- start start 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.014+0000 7f6f8631e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800a8290 0x7f6f801539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.014+0000 7f6f8631e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f800c4740 0x7f6f800701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.014+0000 7f6f8631e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800c62a0 con 0x7f6f800c69a0 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.014+0000 7f6f8631e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f80153f00 con 0x7f6f800c4740 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.014+0000 7f6f8631e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f800706e0 con 0x7f6f800a8290 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800a8290 0x7f6f801539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7f7fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f800c4740 0x7f6f800701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f848bb640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6f800c69a0 0x7f6f80072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:48040/0 (socket says 172.21.15.94:48040) 2024-03-06T07:19:40.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f848bb640 1 -- 172.21.15.94:0/3243958199 learned_addr learned my addr 172.21.15.94:0/3243958199 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:40.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7ffff640 1 -- 172.21.15.94:0/3243958199 >> v2:172.21.15.145:3300/0 conn(0x7f6f800c4740 msgr2=0x7f6f800701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:40.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7ffff640 1 --2- 172.21.15.94:0/3243958199 >> v2:172.21.15.145:3300/0 conn(0x7f6f800c4740 0x7f6f800701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7ffff640 1 -- 172.21.15.94:0/3243958199 >> v1:172.21.15.145:6789/0 conn(0x7f6f800c69a0 legacy=0x7f6f80072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7effd640 1 -- 172.21.15.94:0/3243958199 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146327486 0 0) 0x7f6f800c62a0 con 0x7f6f800c69a0 2024-03-06T07:19:40.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7ffff640 1 -- 172.21.15.94:0/3243958199 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f800c4c00 con 0x7f6f800a8290 2024-03-06T07:19:40.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.015+0000 7f6f7f7fe640 1 --2- 172.21.15.94:0/3243958199 >> v2:172.21.15.145:3300/0 conn(0x7f6f800c4740 0x7f6f800701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:19:40.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.016+0000 7f6f7ffff640 1 --2- 172.21.15.94:0/3243958199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800a8290 0x7f6f801539c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f700097a0 tx=0x7f6f7002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e554b79ea476ecb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.016+0000 7f6f7effd640 1 -- 172.21.15.94:0/3243958199 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70038070 con 0x7f6f800a8290 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.016+0000 7f6f7effd640 1 -- 172.21.15.94:0/3243958199 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f70031700 con 0x7f6f800a8290 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.016+0000 7f6f7effd640 1 -- 172.21.15.94:0/3243958199 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70031ad0 con 0x7f6f800a8290 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.016+0000 7f6f8631e640 1 -- 172.21.15.94:0/3243958199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800a8290 msgr2=0x7f6f801539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.016+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3243958199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800a8290 0x7f6f801539c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f700097a0 tx=0x7f6f7002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.017+0000 7f6f8631e640 1 -- 172.21.15.94:0/3243958199 shutdown_connections 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.017+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3243958199 >> v2:172.21.15.145:3300/0 conn(0x7f6f800c4740 0x7f6f800701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.017+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3243958199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800a8290 0x7f6f801539c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.017+0000 7f6f8631e640 1 -- 172.21.15.94:0/3243958199 >> 172.21.15.94:0/3243958199 conn(0x7f6f800c5560 msgr2=0x7f6f800c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.017+0000 7f6f8631e640 1 -- 172.21.15.94:0/3243958199 shutdown_connections 2024-03-06T07:19:40.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.017+0000 7f6f8631e640 1 -- 172.21.15.94:0/3243958199 wait complete. 2024-03-06T07:19:40.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.018+0000 7f6f8631e640 1 Processor -- start 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.019+0000 7f6f8631e640 1 -- start start 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f8631e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f800a8290 0x7f6f800a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f8631e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f800c4740 0x7f6f8014b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f8631e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800c69a0 0x7f6f8014bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f8631e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8006fc80 con 0x7f6f800a8290 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f8631e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8006f9a0 con 0x7f6f800c4740 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f8631e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f80072d90 con 0x7f6f800c69a0 2024-03-06T07:19:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f800a8290 0x7f6f800a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f800a8290 0x7f6f800a8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60456/0 (socket says 172.21.15.94:60456) 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 -- 172.21.15.94:0/3503413565 learned_addr learned my addr 172.21.15.94:0/3503413565 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f848bb640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800c69a0 0x7f6f8014bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7f7fe640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f800c4740 0x7f6f8014b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 -- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800c69a0 msgr2=0x7f6f8014bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800c69a0 0x7f6f8014bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 -- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f800c4740 msgr2=0x7f6f8014b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f800c4740 0x7f6f8014b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 -- 172.21.15.94:0/3503413565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f801ecba0 con 0x7f6f800a8290 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7ffff640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f800a8290 0x7f6f800a8bc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6f70036040 tx=0x7f6f700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7cff9640 1 -- 172.21.15.94:0/3503413565 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70038040 con 0x7f6f800a8290 2024-03-06T07:19:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7cff9640 1 -- 172.21.15.94:0/3503413565 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f70004030 con 0x7f6f800a8290 2024-03-06T07:19:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8014df70 con 0x7f6f800a8290 2024-03-06T07:19:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.020+0000 7f6f7cff9640 1 -- 172.21.15.94:0/3503413565 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70004420 con 0x7f6f800a8290 2024-03-06T07:19:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.021+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f800a9100 con 0x7f6f800a8290 2024-03-06T07:19:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.022+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8014e490 con 0x7f6f800a8290 2024-03-06T07:19:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.022+0000 7f6f7cff9640 1 -- 172.21.15.94:0/3503413565 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f70033780 con 0x7f6f800a8290 2024-03-06T07:19:40.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.022+0000 7f6f7cff9640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.023+0000 7f6f7cff9640 1 -- 172.21.15.94:0/3503413565 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6f700ac4d0 con 0x7f6f800a8290 2024-03-06T07:19:40.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.023+0000 7f6f7f7fe640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.023+0000 7f6f7f7fe640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009560 tx=0x7f6f74009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.027+0000 7f6f7cff9640 1 -- 172.21.15.94:0/3503413565 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f70030070 con 0x7f6f800a8290 2024-03-06T07:19:40.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.203+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f801536c0 con 0x7f6f800a8290 2024-03-06T07:19:40.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.204+0000 7f6f7cff9640 1 -- 172.21.15.94:0/3503413565 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f7007abb0 con 0x7f6f800a8290 2024-03-06T07:19:40.206 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:19:40.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.205+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 msgr2=0x7f6f6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.205+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009560 tx=0x7f6f74009190 comp rx=0 tx=0).stop 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.205+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f800a8290 msgr2=0x7f6f800a8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.205+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f800a8290 0x7f6f800a8bc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6f70036040 tx=0x7f6f700046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 shutdown_connections 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f800c69a0 0x7f6f8014bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f800c4740 0x7f6f8014b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 --2- 172.21.15.94:0/3503413565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f800a8290 0x7f6f800a8bc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 >> 172.21.15.94:0/3503413565 conn(0x7f6f800c5560 msgr2=0x7f6f80152fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:40.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 shutdown_connections 2024-03-06T07:19:40.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.206+0000 7f6f8631e640 1 -- 172.21.15.94:0/3503413565 wait complete. 2024-03-06T07:19:40.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:19:40.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.323+0000 7fe3c6683640 1 Processor -- start 2024-03-06T07:19:40.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.323+0000 7fe3c6683640 1 -- start start 2024-03-06T07:19:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3c6683640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe3c00b0070 0x7fe3c0155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3c6683640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe3c006ca70 0x7fe3c00c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3c6683640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c00c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3c6683640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c006c020 con 0x7fe3c00b0070 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3c6683640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c00b0450 con 0x7fe3c006ca70 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3c6683640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c006cf20 con 0x7fe3c00c5380 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe3c00b0070 0x7fe3c0155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe3c00b0070 0x7fe3c0155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60458/0 (socket says 172.21.15.94:60458) 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 -- 172.21.15.94:0/3283859220 learned_addr learned my addr 172.21.15.94:0/3283859220 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3c4c20640 1 --2- 172.21.15.94:0/3283859220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c00c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bf7fe640 1 --2- 172.21.15.94:0/3283859220 >> v2:172.21.15.145:3300/0 conn(0x7fe3c006ca70 0x7fe3c00c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 -- 172.21.15.94:0/3283859220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 msgr2=0x7fe3c00c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 --2- 172.21.15.94:0/3283859220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c00c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 -- 172.21.15.94:0/3283859220 >> v2:172.21.15.145:3300/0 conn(0x7fe3c006ca70 msgr2=0x7fe3c00c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 --2- 172.21.15.94:0/3283859220 >> v2:172.21.15.145:3300/0 conn(0x7fe3c006ca70 0x7fe3c00c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.325+0000 7fe3bffff640 1 -- 172.21.15.94:0/3283859220 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c00c77f0 con 0x7fe3c00b0070 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.326+0000 7fe3bffff640 1 --2- 172.21.15.94:0/3283859220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c00b0070 0x7fe3c0155c30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe3c006d4c0 tx=0x7fe3b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd79cde13993094b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.326+0000 7fe3beffd640 1 -- 172.21.15.94:0/3283859220 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b002e070 con 0x7fe3c00b0070 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.326+0000 7fe3beffd640 1 -- 172.21.15.94:0/3283859220 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0037070 con 0x7fe3c00b0070 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.326+0000 7fe3beffd640 1 -- 172.21.15.94:0/3283859220 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0031a00 con 0x7fe3c00b0070 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 -- 172.21.15.94:0/3283859220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c00b0070 msgr2=0x7fe3c0155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3283859220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c00b0070 0x7fe3c0155c30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe3c006d4c0 tx=0x7fe3b002b880 comp rx=0 tx=0).stop 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 -- 172.21.15.94:0/3283859220 shutdown_connections 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3283859220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c00c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3283859220 >> v2:172.21.15.145:3300/0 conn(0x7fe3c006ca70 0x7fe3c00c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3283859220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c00b0070 0x7fe3c0155c30 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 -- 172.21.15.94:0/3283859220 >> 172.21.15.94:0/3283859220 conn(0x7fe3c00ac590 msgr2=0x7fe3c00ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:40.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 -- 172.21.15.94:0/3283859220 shutdown_connections 2024-03-06T07:19:40.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.327+0000 7fe3c6683640 1 -- 172.21.15.94:0/3283859220 wait complete. 2024-03-06T07:19:40.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.328+0000 7fe3c6683640 1 Processor -- start 2024-03-06T07:19:40.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.328+0000 7fe3c6683640 1 -- start start 2024-03-06T07:19:40.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3c6683640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c006ca70 0x7fe3c00c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3c6683640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3c00b0070 0x7fe3c01e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3c6683640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c01e8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3c6683640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c006c550 con 0x7fe3c006ca70 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3c6683640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c006c270 con 0x7fe3c00b0070 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3c6683640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c006da80 con 0x7fe3c00c5380 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c006ca70 0x7fe3c00c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c006ca70 0x7fe3c00c4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60470/0 (socket says 172.21.15.94:60470) 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 -- 172.21.15.94:0/3612052997 learned_addr learned my addr 172.21.15.94:0/3612052997 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3c4c20640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c01e8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bf7fe640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3c00b0070 0x7fe3c01e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 -- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 msgr2=0x7fe3c01e8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c01e8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 -- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3c00b0070 msgr2=0x7fe3c01e7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3c00b0070 0x7fe3c01e7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 -- 172.21.15.94:0/3612052997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c0158610 con 0x7fe3c006ca70 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bf7fe640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3c00b0070 0x7fe3c01e7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.329+0000 7fe3bffff640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c006ca70 0x7fe3c00c4810 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe3b002b8c0 tx=0x7fe3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.330+0000 7fe3bcff9640 1 -- 172.21.15.94:0/3612052997 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0032670 con 0x7fe3c006ca70 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.330+0000 7fe3bcff9640 1 -- 172.21.15.94:0/3612052997 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0032850 con 0x7fe3c006ca70 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.330+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c0154330 con 0x7fe3c006ca70 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.330+0000 7fe3bcff9640 1 -- 172.21.15.94:0/3612052997 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0032c40 con 0x7fe3c006ca70 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.330+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c01548b0 con 0x7fe3c006ca70 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.331+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c004f330 con 0x7fe3c006ca70 2024-03-06T07:19:40.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.331+0000 7fe3bcff9640 1 -- 172.21.15.94:0/3612052997 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3b00453f0 con 0x7fe3c006ca70 2024-03-06T07:19:40.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.331+0000 7fe3bcff9640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe3ac06b4d0 0x7fe3ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.331+0000 7fe3bcff9640 1 -- 172.21.15.94:0/3612052997 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe3b00ac690 con 0x7fe3c006ca70 2024-03-06T07:19:40.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.332+0000 7fe3bf7fe640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe3ac06b4d0 0x7fe3ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.332+0000 7fe3bf7fe640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe3ac06b4d0 0x7fe3ac06d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00acb50 tx=0x7fe3b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.336+0000 7fe3bcff9640 1 -- 172.21.15.94:0/3612052997 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3b0040070 con 0x7fe3c006ca70 2024-03-06T07:19:40.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.554+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3c006d860 con 0x7fe3c006ca70 2024-03-06T07:19:40.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.554+0000 7fe3bcff9640 1 -- 172.21.15.94:0/3612052997 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe3b007b4e0 con 0x7fe3c006ca70 2024-03-06T07:19:40.556 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe3ac06b4d0 msgr2=0x7fe3ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe3ac06b4d0 0x7fe3ac06d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00acb50 tx=0x7fe3b4008040 comp rx=0 tx=0).stop 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c006ca70 msgr2=0x7fe3c00c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c006ca70 0x7fe3c00c4810 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe3b002b8c0 tx=0x7fe3b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 shutdown_connections 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe3ac06b4d0 0x7fe3ac06d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3c00c5380 0x7fe3c01e8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3c00b0070 0x7fe3c01e7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 --2- 172.21.15.94:0/3612052997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe3c006ca70 0x7fe3c00c4810 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 >> 172.21.15.94:0/3612052997 conn(0x7fe3c00ac590 msgr2=0x7fe3c00ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:40.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 shutdown_connections 2024-03-06T07:19:40.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.556+0000 7fe3c6683640 1 -- 172.21.15.94:0/3612052997 wait complete. 2024-03-06T07:19:40.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:40.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.669+0000 7f079a84e640 1 Processor -- start 2024-03-06T07:19:40.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.669+0000 7f079a84e640 1 -- start start 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f079a84e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0794159270 0x7f0794155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f079a84e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940c73d0 con 0x7f079414f360 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f079a84e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940be2e0 con 0x7f07940c88c0 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f079a84e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0794159700 con 0x7f0794159270 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f0798deb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0794159270 0x7f0794155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f0793fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f079414f360 0x7f07940c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35970/0 (socket says 172.21.15.94:35970) 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f0793fff640 1 -- 172.21.15.94:0/2517226525 learned_addr learned my addr 172.21.15.94:0/2517226525 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f0798deb640 1 -- 172.21.15.94:0/2517226525 >> v1:172.21.15.145:6789/0 conn(0x7f07940c88c0 legacy=0x7f0794158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:40.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f0798deb640 1 -- 172.21.15.94:0/2517226525 >> v1:172.21.15.94:6789/0 conn(0x7f079414f360 legacy=0x7f07940c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f0798deb640 1 -- 172.21.15.94:0/2517226525 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07940c7670 con 0x7f0794159270 2024-03-06T07:19:40.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.670+0000 7f0792ffd640 1 -- 172.21.15.94:0/2517226525 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070137017 0 0) 0x7f07940c73d0 con 0x7f079414f360 2024-03-06T07:19:40.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.671+0000 7f0798deb640 1 --2- 172.21.15.94:0/2517226525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0794159270 0x7f0794155c10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f078c009d90 tx=0x7f078c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641079385acc6112 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.671+0000 7f0792ffd640 1 -- 172.21.15.94:0/2517226525 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c039070 con 0x7f0794159270 2024-03-06T07:19:40.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.671+0000 7f0792ffd640 1 -- 172.21.15.94:0/2517226525 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c02cbd0 con 0x7f0794159270 2024-03-06T07:19:40.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.671+0000 7f0792ffd640 1 -- 172.21.15.94:0/2517226525 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c034440 con 0x7f0794159270 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.672+0000 7f079a84e640 1 -- 172.21.15.94:0/2517226525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0794159270 msgr2=0x7f0794155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.672+0000 7f079a84e640 1 --2- 172.21.15.94:0/2517226525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0794159270 0x7f0794155c10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f078c009d90 tx=0x7f078c02be20 comp rx=0 tx=0).stop 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.672+0000 7f079a84e640 1 -- 172.21.15.94:0/2517226525 shutdown_connections 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.672+0000 7f079a84e640 1 --2- 172.21.15.94:0/2517226525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0794159270 0x7f0794155c10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.672+0000 7f079a84e640 1 -- 172.21.15.94:0/2517226525 >> 172.21.15.94:0/2517226525 conn(0x7f07940ac570 msgr2=0x7f07940ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.672+0000 7f079a84e640 1 -- 172.21.15.94:0/2517226525 shutdown_connections 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.672+0000 7f079a84e640 1 -- 172.21.15.94:0/2517226525 wait complete. 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.673+0000 7f079a84e640 1 Processor -- start 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.673+0000 7f079a84e640 1 -- start start 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f079a84e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07940c88c0 0x7f079414c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f079a84e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f079414f360 0x7f079414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f079a84e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0794159270 0x7f0794157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f079a84e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07940c7c30 con 0x7f079414f360 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f079a84e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07940c5a50 con 0x7f07940c88c0 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f079a84e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07940c5bd0 con 0x7f0794159270 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f079414f360 0x7f079414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f079414f360 0x7f079414b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60494/0 (socket says 172.21.15.94:60494) 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 -- 172.21.15.94:0/142985929 learned_addr learned my addr 172.21.15.94:0/142985929 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f0798deb640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0794159270 0x7f0794157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 -- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0794159270 msgr2=0x7f0794157c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0794159270 0x7f0794157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f0793fff640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07940c88c0 0x7f079414c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 -- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07940c88c0 msgr2=0x7f079414c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07940c88c0 0x7f079414c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 -- 172.21.15.94:0/142985929 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07940b06c0 con 0x7f079414f360 2024-03-06T07:19:40.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f07937fe640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f079414f360 0x7f079414b790 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f078400c400 tx=0x7f078400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.674+0000 7f0790ff9640 1 -- 172.21.15.94:0/142985929 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784011790 con 0x7f079414f360 2024-03-06T07:19:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.675+0000 7f0790ff9640 1 -- 172.21.15.94:0/142985929 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0784017070 con 0x7f079414f360 2024-03-06T07:19:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.675+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07940add30 con 0x7f079414f360 2024-03-06T07:19:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.675+0000 7f0790ff9640 1 -- 172.21.15.94:0/142985929 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784011bb0 con 0x7f079414f360 2024-03-06T07:19:40.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.675+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07941eb8e0 con 0x7f079414f360 2024-03-06T07:19:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.676+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07940694c0 con 0x7f079414f360 2024-03-06T07:19:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.677+0000 7f0790ff9640 1 -- 172.21.15.94:0/142985929 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f078401c460 con 0x7f079414f360 2024-03-06T07:19:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.677+0000 7f0790ff9640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078006b4d0 0x7f078006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.677+0000 7f0790ff9640 1 -- 172.21.15.94:0/142985929 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f078408a290 con 0x7f079414f360 2024-03-06T07:19:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.677+0000 7f0793fff640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078006b4d0 0x7f078006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.678+0000 7f0793fff640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078006b4d0 0x7f078006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0788000f60 tx=0x7f0788009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:40.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.681+0000 7f0790ff9640 1 -- 172.21.15.94:0/142985929 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07940694c0 con 0x7f079414f360 2024-03-06T07:19:40.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.889+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07940c8d50 con 0x7f079414f360 2024-03-06T07:19:40.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.890+0000 7f0790ff9640 1 -- 172.21.15.94:0/142985929 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f07940add30 con 0x7f079414f360 2024-03-06T07:19:40.892 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:40.892 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:32.497990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":2}]} 2024-03-06T07:19:40.892 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-03-06T07:19:40.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078006b4d0 msgr2=0x7f078006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078006b4d0 0x7f078006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0788000f60 tx=0x7f0788009040 comp rx=0 tx=0).stop 2024-03-06T07:19:40.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f079414f360 msgr2=0x7f079414b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:40.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f079414f360 0x7f079414b790 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f078400c400 tx=0x7f078400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 shutdown_connections 2024-03-06T07:19:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078006b4d0 0x7f078006d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0794159270 0x7f0794157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f079414f360 0x7f079414b790 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 --2- 172.21.15.94:0/142985929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07940c88c0 0x7f079414c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:40.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.891+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 >> 172.21.15.94:0/142985929 conn(0x7f07940ac570 msgr2=0x7f07940cb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:40.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.892+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 shutdown_connections 2024-03-06T07:19:40.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:40.892+0000 7f079a84e640 1 -- 172.21.15.94:0/142985929 wait complete. 2024-03-06T07:19:40.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:41.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.005+0000 7fa68555c640 1 Processor -- start 2024-03-06T07:19:41.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.005+0000 7fa68555c640 1 -- start start 2024-03-06T07:19:41.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa68555c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6801527b0 0x7fa680159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa68555c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6800c7180 0x7fa6800c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa68555c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800c7c60 con 0x7fa680155b20 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa68555c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa680159640 con 0x7fa6800c7180 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa68555c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800c8df0 con 0x7fa6801527b0 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67e7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6800c7180 0x7fa6800c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67e7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6800c7180 0x7fa6800c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60496/0 (socket says 172.21.15.94:60496) 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67e7fc640 1 -- 172.21.15.94:0/3748065067 learned_addr learned my addr 172.21.15.94:0/3748065067 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67effd640 1 --2- 172.21.15.94:0/3748065067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6801527b0 0x7fa680159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67e7fc640 1 -- 172.21.15.94:0/3748065067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6801527b0 msgr2=0x7fa680159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67e7fc640 1 --2- 172.21.15.94:0/3748065067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6801527b0 0x7fa680159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67e7fc640 1 -- 172.21.15.94:0/3748065067 >> v1:172.21.15.145:6789/0 conn(0x7fa680155b20 legacy=0x7fa6800c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:41.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.006+0000 7fa67e7fc640 1 -- 172.21.15.94:0/3748065067 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6800c8060 con 0x7fa6800c7180 2024-03-06T07:19:41.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.007+0000 7fa67e7fc640 1 --2- 172.21.15.94:0/3748065067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6800c7180 0x7fa6800c88b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa674009870 tx=0x7fa67402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3320ba6286d6ffb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.007+0000 7fa67dffb640 1 -- 172.21.15.94:0/3748065067 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6740317c0 con 0x7fa6800c7180 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.007+0000 7fa67dffb640 1 -- 172.21.15.94:0/3748065067 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa674037070 con 0x7fa6800c7180 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.007+0000 7fa67dffb640 1 -- 172.21.15.94:0/3748065067 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674031b80 con 0x7fa6800c7180 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 -- 172.21.15.94:0/3748065067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6800c7180 msgr2=0x7fa6800c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 --2- 172.21.15.94:0/3748065067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6800c7180 0x7fa6800c88b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa674009870 tx=0x7fa67402b860 comp rx=0 tx=0).stop 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 -- 172.21.15.94:0/3748065067 shutdown_connections 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 --2- 172.21.15.94:0/3748065067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6800c7180 0x7fa6800c88b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 --2- 172.21.15.94:0/3748065067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6801527b0 0x7fa680159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 -- 172.21.15.94:0/3748065067 >> 172.21.15.94:0/3748065067 conn(0x7fa6800ac560 msgr2=0x7fa6800ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:41.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 -- 172.21.15.94:0/3748065067 shutdown_connections 2024-03-06T07:19:41.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.008+0000 7fa68555c640 1 -- 172.21.15.94:0/3748065067 wait complete. 2024-03-06T07:19:41.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.009+0000 7fa68555c640 1 Processor -- start 2024-03-06T07:19:41.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.009+0000 7fa68555c640 1 -- start start 2024-03-06T07:19:41.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa68555c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6800c7180 0x7fa6800c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa68555c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6801527b0 0x7fa6800c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa68555c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa680155b20 0x7fa68014d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa68555c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa680155320 con 0x7fa6801527b0 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa68555c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800c7980 con 0x7fa680155b20 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa68555c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa680155600 con 0x7fa6800c7180 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6801527b0 0x7fa6800c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6801527b0 0x7fa6800c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60504/0 (socket says 172.21.15.94:60504) 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 -- 172.21.15.94:0/1006737433 learned_addr learned my addr 172.21.15.94:0/1006737433 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67effd640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6800c7180 0x7fa6800c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67f7fe640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa680155b20 0x7fa68014d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 -- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6800c7180 msgr2=0x7fa6800c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6800c7180 0x7fa6800c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 -- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa680155b20 msgr2=0x7fa68014d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa680155b20 0x7fa68014d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67e7fc640 1 -- 172.21.15.94:0/1006737433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6801eba50 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67f7fe640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa680155b20 0x7fa68014d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.010+0000 7fa67effd640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6800c7180 0x7fa6800c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.011+0000 7fa67e7fc640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6801527b0 0x7fa6800c3fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa674000c00 tx=0x7fa6740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.011+0000 7fa6677fe640 1 -- 172.21.15.94:0/1006737433 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674032650 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.011+0000 7fa6677fe640 1 -- 172.21.15.94:0/1006737433 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa674032810 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.011+0000 7fa6677fe640 1 -- 172.21.15.94:0/1006737433 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674032c00 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.011+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6801ebd30 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.011+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68014f820 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.012+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68004f330 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.012+0000 7fa6677fe640 1 -- 172.21.15.94:0/1006737433 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa67404c060 con 0x7fa6801527b0 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.013+0000 7fa6677fe640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa66c06b4d0 0x7fa66c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.013+0000 7fa6677fe640 1 -- 172.21.15.94:0/1006737433 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa6740abeb0 con 0x7fa6801527b0 2024-03-06T07:19:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.013+0000 7fa67effd640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa66c06b4d0 0x7fa66c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.014+0000 7fa67effd640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa66c06b4d0 0x7fa66c06d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa670006fd0 tx=0x7fa670008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.021+0000 7fa6677fe640 1 -- 172.21.15.94:0/1006737433 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa674040070 con 0x7fa6801527b0 2024-03-06T07:19:41.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.194+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa68006ce00 con 0x7fa6801527b0 2024-03-06T07:19:41.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.196+0000 7fa6677fe640 1 -- 172.21.15.94:0/1006737433 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17073 (secure 0 0 0) 0x7fa6800c7c60 con 0x7fa6801527b0 2024-03-06T07:19:41.198 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:41.198 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":40,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:26.486055+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:17.283570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:17.762319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/1305657307":"2024-03-07T07:19:26.486020+0000","172.21.15.145:6836/1305657307":"2024-03-07T07:19:26.486020+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:41.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa66c06b4d0 msgr2=0x7fa66c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa66c06b4d0 0x7fa66c06d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa670006fd0 tx=0x7fa670008040 comp rx=0 tx=0).stop 2024-03-06T07:19:41.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6801527b0 msgr2=0x7fa6800c3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6801527b0 0x7fa6800c3fe0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa674000c00 tx=0x7fa6740046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 shutdown_connections 2024-03-06T07:19:41.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa66c06b4d0 0x7fa66c06d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa680155b20 0x7fa68014d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6801527b0 0x7fa6800c3fe0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 --2- 172.21.15.94:0/1006737433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6800c7180 0x7fa6800c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 >> 172.21.15.94:0/1006737433 conn(0x7fa6800ac560 msgr2=0x7fa68006b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:41.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 shutdown_connections 2024-03-06T07:19:41.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.198+0000 7fa68555c640 1 -- 172.21.15.94:0/1006737433 wait complete. 2024-03-06T07:19:41.211 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:19:41.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:41.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.311+0000 7f2d075d0640 1 Processor -- start 2024-03-06T07:19:41.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d075d0640 1 -- start start 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d075d0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2d0014f790 0x7f2d001529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d075d0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 0x7f2d00155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d075d0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d001563d0 con 0x7f2d000c1aa0 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d075d0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d000c2680 con 0x7f2d0014f790 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d075d0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d000c28d0 con 0x7f2d00155a40 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d04b6b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2d000c1aa0 0x7f2d000c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35972/0 (socket says 172.21.15.94:35972) 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.312+0000 7f2d04b6b640 1 -- 172.21.15.94:0/3839303869 learned_addr learned my addr 172.21.15.94:0/3839303869 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2d05b6d640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 0x7f2d00155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301479260 0 0) 0x7f2d001563d0 con 0x7f2d000c1aa0 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cec003520 con 0x7f2d000c1aa0 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2d0536c640 1 --2- 172.21.15.94:0/3839303869 >> v2:172.21.15.145:3300/0 conn(0x7f2d0014f790 0x7f2d001529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2809278782 0 0) 0x7f2cec003520 con 0x7f2d000c1aa0 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 msgr2=0x7f2d00155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 0x7f2d00155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 >> v2:172.21.15.145:3300/0 conn(0x7f2d0014f790 msgr2=0x7f2d001529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/3839303869 >> v2:172.21.15.145:3300/0 conn(0x7f2d0014f790 0x7f2d001529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d000c2bb0 con 0x7f2d000c1aa0 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f2cf0001670 con 0x7f2d000c1aa0 2024-03-06T07:19:41.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 >> v1:172.21.15.94:6789/0 conn(0x7f2d000c1aa0 legacy=0x7f2d000c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 0x7f2cec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2cec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cec004d40 0x7f2cec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d000c2680 con 0x7f2cec003f30 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d000c28d0 con 0x7f2cec004d40 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec00b9c0 con 0x7f2d00155a40 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2d04b6b640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2cec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2d0536c640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 0x7f2cec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.313+0000 7f2d05b6d640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cec004d40 0x7f2cec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2d04b6b640 1 -- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 msgr2=0x7f2cec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2d04b6b640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00155a40 0x7f2cec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2d04b6b640 1 -- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cec004d40 msgr2=0x7f2cec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2d04b6b640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cec004d40 0x7f2cec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2d04b6b640 1 -- 172.21.15.94:0/3839303869 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec003520 con 0x7f2cec003f30 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2d04b6b640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2cec004800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0001050 tx=0x7f2cec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9e8b3a6515efbe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0007d70 con 0x7f2cec003f30 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0030070 con 0x7f2cec003f30 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.314+0000 7f2cf7fff640 1 -- 172.21.15.94:0/3839303869 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0005d30 con 0x7f2cec003f30 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.315+0000 7f2d075d0640 1 -- 172.21.15.94:0/3839303869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 msgr2=0x7f2cec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.315+0000 7f2d075d0640 1 --2- 172.21.15.94:0/3839303869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2cec004800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0001050 tx=0x7f2cec005830 comp rx=0 tx=0).stop 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.315+0000 7f2d05b6d640 1 -- 172.21.15.94:0/3839303869 reap_dead start 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.315+0000 7f2d075d0640 1 -- 172.21.15.94:0/3839303869 shutdown_connections 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.315+0000 7f2d075d0640 1 -- 172.21.15.94:0/3839303869 >> 172.21.15.94:0/3839303869 conn(0x7f2d000c6710 msgr2=0x7f2d000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.315+0000 7f2d075d0640 1 -- 172.21.15.94:0/3839303869 shutdown_connections 2024-03-06T07:19:41.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.315+0000 7f2d075d0640 1 -- 172.21.15.94:0/3839303869 wait complete. 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.316+0000 7f2d075d0640 1 Processor -- start 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.316+0000 7f2d075d0640 1 -- start start 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d075d0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2d000bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d075d0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d0014aa60 0x7f2d001edea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d075d0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00150110 0x7f2d001ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d075d0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d000c0d60 con 0x7f2cec003f30 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d075d0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d00155520 con 0x7f2d0014aa60 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d075d0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d000c5e30 con 0x7f2d00150110 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2d000bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2d000bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60518/0 (socket says 172.21.15.94:60518) 2024-03-06T07:19:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 -- 172.21.15.94:0/1971307004 learned_addr learned my addr 172.21.15.94:0/1971307004 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 -- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00150110 msgr2=0x7f2d001ee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00150110 0x7f2d001ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 -- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d0014aa60 msgr2=0x7f2d001edea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d0014aa60 0x7f2d001edea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 -- 172.21.15.94:0/1971307004 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d001eb880 con 0x7f2cec003f30 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2d0536c640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2d000bfdb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0119d0 tx=0x7f2cfc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.317+0000 7f2cf5ffb640 1 -- 172.21.15.94:0/1971307004 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc00fad0 con 0x7f2cec003f30 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.318+0000 7f2cf5ffb640 1 -- 172.21.15.94:0/1971307004 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc00fc70 con 0x7f2cec003f30 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.318+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d000a6890 con 0x7f2cec003f30 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.318+0000 7f2cf5ffb640 1 -- 172.21.15.94:0/1971307004 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc014440 con 0x7f2cec003f30 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.318+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d001eb5a0 con 0x7f2cec003f30 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.319+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0004f330 con 0x7f2cec003f30 2024-03-06T07:19:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.319+0000 7f2cf5ffb640 1 -- 172.21.15.94:0/1971307004 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cfc012040 con 0x7f2cec003f30 2024-03-06T07:19:41.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.319+0000 7f2cf5ffb640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ce406b4d0 0x7f2ce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.320+0000 7f2cf5ffb640 1 -- 172.21.15.94:0/1971307004 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2cfc08a9a0 con 0x7f2cec003f30 2024-03-06T07:19:41.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.320+0000 7f2d04b6b640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ce406b4d0 0x7f2ce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.320+0000 7f2d04b6b640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ce406b4d0 0x7f2ce406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0034000 tx=0x7f2cf002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.324+0000 7f2cf5ffb640 1 -- 172.21.15.94:0/1971307004 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2cfc019030 con 0x7f2cec003f30 2024-03-06T07:19:41.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.559+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d000c1f40 con 0x7f2cec003f30 2024-03-06T07:19:41.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.560+0000 7f2cf5ffb640 1 -- 172.21.15.94:0/1971307004 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f2cfc0596d0 con 0x7f2cec003f30 2024-03-06T07:19:41.562 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:41.562 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:32.497990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":2}]} 2024-03-06T07:19:41.562 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ce406b4d0 msgr2=0x7f2ce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ce406b4d0 0x7f2ce406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0034000 tx=0x7f2cf002e000 comp rx=0 tx=0).stop 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 msgr2=0x7f2d000bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2d000bfdb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0119d0 tx=0x7f2cfc011e90 comp rx=0 tx=0).stop 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 shutdown_connections 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ce406b4d0 0x7f2ce406d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d00150110 0x7f2d001ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d0014aa60 0x7f2d001edea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 --2- 172.21.15.94:0/1971307004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cec003f30 0x7f2d000bfdb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 >> 172.21.15.94:0/1971307004 conn(0x7f2d000c6710 msgr2=0x7f2d000a7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:41.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 shutdown_connections 2024-03-06T07:19:41.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.562+0000 7f2d075d0640 1 -- 172.21.15.94:0/1971307004 wait complete. 2024-03-06T07:19:41.575 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.674+0000 7f5fe11c9640 1 Processor -- start 2024-03-06T07:19:41.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.674+0000 7f5fe11c9640 1 -- start start 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fe11c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5fdc159270 0x7f5fdc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fe11c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc0c73d0 con 0x7f5fdc0c88c0 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fe11c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc0be2e0 con 0x7f5fdc14f360 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fe11c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc159700 con 0x7f5fdc159270 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fdb59e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5fdc159270 0x7f5fdc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fda59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5fdc0c88c0 0x7f5fdc158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35986/0 (socket says 172.21.15.94:35986) 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fda59c640 1 -- 172.21.15.94:0/3824040355 learned_addr learned my addr 172.21.15.94:0/3824040355 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fdb59e640 1 --2- 172.21.15.94:0/3824040355 >> v2:172.21.15.94:3300/0 conn(0x7f5fdc159270 0x7f5fdc155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60530/0 (socket says 172.21.15.94:60530) 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fdb59e640 1 -- 172.21.15.94:0/3824040355 >> v1:172.21.15.145:6789/0 conn(0x7f5fdc14f360 legacy=0x7f5fdc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fdb59e640 1 -- 172.21.15.94:0/3824040355 >> v1:172.21.15.94:6789/0 conn(0x7f5fdc0c88c0 legacy=0x7f5fdc158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fd9d9b640 1 -- 172.21.15.94:0/3824040355 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263333546 0 0) 0x7f5fdc0c73d0 con 0x7f5fdc0c88c0 2024-03-06T07:19:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.675+0000 7f5fdb59e640 1 -- 172.21.15.94:0/3824040355 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc0c7670 con 0x7f5fdc159270 2024-03-06T07:19:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fdb59e640 1 --2- 172.21.15.94:0/3824040355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc159270 0x7f5fdc155c10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009d90 tx=0x7f5fd402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62af204f2429714e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fd9d9b640 1 -- 172.21.15.94:0/3824040355 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4039070 con 0x7f5fdc159270 2024-03-06T07:19:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fd9d9b640 1 -- 172.21.15.94:0/3824040355 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd402cbd0 con 0x7f5fdc159270 2024-03-06T07:19:41.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fd9d9b640 1 -- 172.21.15.94:0/3824040355 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4034440 con 0x7f5fdc159270 2024-03-06T07:19:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fe11c9640 1 -- 172.21.15.94:0/3824040355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc159270 msgr2=0x7f5fdc155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/3824040355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc159270 0x7f5fdc155c10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009d90 tx=0x7f5fd402be20 comp rx=0 tx=0).stop 2024-03-06T07:19:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fe11c9640 1 -- 172.21.15.94:0/3824040355 shutdown_connections 2024-03-06T07:19:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/3824040355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc159270 0x7f5fdc155c10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.676+0000 7f5fe11c9640 1 -- 172.21.15.94:0/3824040355 >> 172.21.15.94:0/3824040355 conn(0x7f5fdc0ac570 msgr2=0x7f5fdc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.677+0000 7f5fe11c9640 1 -- 172.21.15.94:0/3824040355 shutdown_connections 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.677+0000 7f5fe11c9640 1 -- 172.21.15.94:0/3824040355 wait complete. 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.678+0000 7f5fe11c9640 1 Processor -- start 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.678+0000 7f5fe11c9640 1 -- start start 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fe11c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fdc0c88c0 0x7f5fdc1eb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fe11c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc14f360 0x7f5fdc1519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fe11c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fdc159270 0x7f5fdc151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fe11c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0c7c30 con 0x7f5fdc14f360 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fe11c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0c5a50 con 0x7f5fdc0c88c0 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fe11c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0c5bd0 con 0x7f5fdc159270 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc14f360 0x7f5fdc1519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc14f360 0x7f5fdc1519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60546/0 (socket says 172.21.15.94:60546) 2024-03-06T07:19:41.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 -- 172.21.15.94:0/1034532354 learned_addr learned my addr 172.21.15.94:0/1034532354 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fdad9d640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fdc0c88c0 0x7f5fdc1eb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fdb59e640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fdc159270 0x7f5fdc151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 -- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fdc159270 msgr2=0x7f5fdc151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fdc159270 0x7f5fdc151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 -- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fdc0c88c0 msgr2=0x7f5fdc1eb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fdc0c88c0 0x7f5fdc1eb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 -- 172.21.15.94:0/1034532354 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc1527a0 con 0x7f5fdc14f360 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fda59c640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc14f360 0x7f5fdc1519a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000c4b0 tx=0x7f5fd000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fc37fe640 1 -- 172.21.15.94:0/1034532354 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0018070 con 0x7f5fdc14f360 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fc37fe640 1 -- 172.21.15.94:0/1034532354 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd0007500 con 0x7f5fdc14f360 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc1eb8c0 con 0x7f5fdc14f360 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.679+0000 7f5fc37fe640 1 -- 172.21.15.94:0/1034532354 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0004a20 con 0x7f5fdc14f360 2024-03-06T07:19:41.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.680+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc14f870 con 0x7f5fdc14f360 2024-03-06T07:19:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.680+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc04f330 con 0x7f5fdc14f360 2024-03-06T07:19:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.681+0000 7f5fc37fe640 1 -- 172.21.15.94:0/1034532354 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fd001b070 con 0x7f5fdc14f360 2024-03-06T07:19:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.681+0000 7f5fc37fe640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fc806b4d0 0x7f5fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.681+0000 7f5fc37fe640 1 -- 172.21.15.94:0/1034532354 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5fd00895d0 con 0x7f5fdc14f360 2024-03-06T07:19:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.682+0000 7f5fdad9d640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fc806b4d0 0x7f5fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.682+0000 7f5fdad9d640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fc806b4d0 0x7f5fc806d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc004f60 tx=0x7f5fcc007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.686+0000 7f5fc37fe640 1 -- 172.21.15.94:0/1034532354 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fdc1eb8c0 con 0x7f5fdc14f360 2024-03-06T07:19:41.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.901+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fdc0c8d50 con 0x7f5fdc14f360 2024-03-06T07:19:41.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.901+0000 7f5fc37fe640 1 -- 172.21.15.94:0/1034532354 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f5fdc1527a0 con 0x7f5fdc14f360 2024-03-06T07:19:41.903 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:41.903 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:18.801003+0000","modified":"2024-03-06T07:19:32.497990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5166},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5166":{"gid":5166,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2367914435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2367914435},{"type":"v1","addr":"172.21.15.94:6835","nonce":2367914435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5166,"qdb_cluster":[5166]},"id":2}]} 2024-03-06T07:19:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 23 2024-03-06T07:19:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fc806b4d0 msgr2=0x7f5fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fc806b4d0 0x7f5fc806d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc004f60 tx=0x7f5fcc007e30 comp rx=0 tx=0).stop 2024-03-06T07:19:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc14f360 msgr2=0x7f5fdc1519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc14f360 0x7f5fdc1519a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000c4b0 tx=0x7f5fd000c970 comp rx=0 tx=0).stop 2024-03-06T07:19:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 shutdown_connections 2024-03-06T07:19:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fc806b4d0 0x7f5fc806d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fdc159270 0x7f5fdc151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fdc14f360 0x7f5fdc1519a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 --2- 172.21.15.94:0/1034532354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fdc0c88c0 0x7f5fdc1eb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:41.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 >> 172.21.15.94:0/1034532354 conn(0x7f5fdc0ac570 msgr2=0x7f5fdc0c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:41.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 shutdown_connections 2024-03-06T07:19:41.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:41.903+0000 7f5fe11c9640 1 -- 172.21.15.94:0/1034532354 wait complete. 2024-03-06T07:19:41.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.016+0000 7f2f80b08640 1 Processor -- start 2024-03-06T07:19:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.016+0000 7f2f80b08640 1 -- start start 2024-03-06T07:19:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f80b08640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f7c153990 0x7f2f7c156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f80b08640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c0acc00 0x7f2f7c0c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f80b08640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7c070aa0 con 0x7f2f7c1506b0 2024-03-06T07:19:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f80b08640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c156450 con 0x7f2f7c153990 2024-03-06T07:19:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f80b08640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c156a10 con 0x7f2f7c0acc00 2024-03-06T07:19:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f79d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f7c153990 0x7f2f7c156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f79d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f7c153990 0x7f2f7c156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60562/0 (socket says 172.21.15.94:60562) 2024-03-06T07:19:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f79d9b640 1 -- 172.21.15.94:0/96989629 learned_addr learned my addr 172.21.15.94:0/96989629 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f7a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2f7c1506b0 0x7f2f7c0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35988/0 (socket says 172.21.15.94:35988) 2024-03-06T07:19:42.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f7ad9d640 1 --2- 172.21.15.94:0/96989629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c0acc00 0x7f2f7c0c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f79d9b640 1 -- 172.21.15.94:0/96989629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c0acc00 msgr2=0x7f2f7c0c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:42.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f79d9b640 1 --2- 172.21.15.94:0/96989629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c0acc00 0x7f2f7c0c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f79d9b640 1 -- 172.21.15.94:0/96989629 >> v1:172.21.15.94:6789/0 conn(0x7f2f7c1506b0 legacy=0x7f2f7c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f7959a640 1 -- 172.21.15.94:0/96989629 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087861679 0 0) 0x7f2f7c070aa0 con 0x7f2f7c1506b0 2024-03-06T07:19:42.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f79d9b640 1 -- 172.21.15.94:0/96989629 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c070d80 con 0x7f2f7c153990 2024-03-06T07:19:42.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.017+0000 7f2f7ad9d640 1 --2- 172.21.15.94:0/96989629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c0acc00 0x7f2f7c0c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:42.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.018+0000 7f2f79d9b640 1 --2- 172.21.15.94:0/96989629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c156ea0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009870 tx=0x7f2f7002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ca937c1609cd18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.018+0000 7f2f7959a640 1 -- 172.21.15.94:0/96989629 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70034070 con 0x7f2f7c153990 2024-03-06T07:19:42.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.018+0000 7f2f7959a640 1 -- 172.21.15.94:0/96989629 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f70030840 con 0x7f2f7c153990 2024-03-06T07:19:42.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.018+0000 7f2f7959a640 1 -- 172.21.15.94:0/96989629 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70030c10 con 0x7f2f7c153990 2024-03-06T07:19:42.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 -- 172.21.15.94:0/96989629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 msgr2=0x7f2f7c156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 --2- 172.21.15.94:0/96989629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c156ea0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009870 tx=0x7f2f7002b980 comp rx=0 tx=0).stop 2024-03-06T07:19:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 -- 172.21.15.94:0/96989629 shutdown_connections 2024-03-06T07:19:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 --2- 172.21.15.94:0/96989629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c0acc00 0x7f2f7c0c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 --2- 172.21.15.94:0/96989629 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c156ea0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 -- 172.21.15.94:0/96989629 >> 172.21.15.94:0/96989629 conn(0x7f2f7c06b1a0 msgr2=0x7f2f7c06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:42.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 -- 172.21.15.94:0/96989629 shutdown_connections 2024-03-06T07:19:42.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.019+0000 7f2f80b08640 1 -- 172.21.15.94:0/96989629 wait complete. 2024-03-06T07:19:42.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.020+0000 7f2f80b08640 1 Processor -- start 2024-03-06T07:19:42.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.020+0000 7f2f80b08640 1 -- start start 2024-03-06T07:19:42.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f80b08640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f7c0acc00 0x7f2f7c1454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f80b08640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c1506b0 0x7f2f7c147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f80b08640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c0c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f80b08640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c0ad1d0 con 0x7f2f7c153990 2024-03-06T07:19:42.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f80b08640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c0ad050 con 0x7f2f7c0acc00 2024-03-06T07:19:42.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f80b08640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c0ad350 con 0x7f2f7c1506b0 2024-03-06T07:19:42.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c0c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c0c1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60566/0 (socket says 172.21.15.94:60566) 2024-03-06T07:19:42.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 -- 172.21.15.94:0/149623438 learned_addr learned my addr 172.21.15.94:0/149623438 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:42.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f79d9b640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c1506b0 0x7f2f7c147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7a59c640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f7c0acc00 0x7f2f7c1454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 -- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c1506b0 msgr2=0x7f2f7c147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c1506b0 0x7f2f7c147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 -- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f7c0acc00 msgr2=0x7f2f7c1454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f7c0acc00 0x7f2f7c1454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 -- 172.21.15.94:0/149623438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c155680 con 0x7f2f7c153990 2024-03-06T07:19:42.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f79d9b640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c1506b0 0x7f2f7c147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:42.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7a59c640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f7c0acc00 0x7f2f7c1454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:42.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.021+0000 7f2f7ad9d640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c0c1af0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2f74007de0 tx=0x7f2f7400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.022+0000 7f2f6affd640 1 -- 172.21.15.94:0/149623438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7401a070 con 0x7f2f7c153990 2024-03-06T07:19:42.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.022+0000 7f2f6affd640 1 -- 172.21.15.94:0/149623438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f74003990 con 0x7f2f7c153990 2024-03-06T07:19:42.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.022+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7c1ebb60 con 0x7f2f7c153990 2024-03-06T07:19:42.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.022+0000 7f2f6affd640 1 -- 172.21.15.94:0/149623438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74003d80 con 0x7f2f7c153990 2024-03-06T07:19:42.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.022+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7c0c2670 con 0x7f2f7c153990 2024-03-06T07:19:42.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.023+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7c04f330 con 0x7f2f7c153990 2024-03-06T07:19:42.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.023+0000 7f2f6affd640 1 -- 172.21.15.94:0/149623438 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f7401d070 con 0x7f2f7c153990 2024-03-06T07:19:42.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.023+0000 7f2f6affd640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f5c06b4d0 0x7f2f5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.024+0000 7f2f6affd640 1 -- 172.21.15.94:0/149623438 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2f7408a740 con 0x7f2f7c153990 2024-03-06T07:19:42.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.024+0000 7f2f7a59c640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f5c06b4d0 0x7f2f5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.024+0000 7f2f7a59c640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f5c06b4d0 0x7f2f5c06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.028+0000 7f2f6affd640 1 -- 172.21.15.94:0/149623438 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f74010070 con 0x7f2f7c153990 2024-03-06T07:19:42.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.206+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f7c0700a0 con 0x7f2f7c153990 2024-03-06T07:19:42.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.208+0000 7f2f6affd640 1 -- 172.21.15.94:0/149623438 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17073 (secure 0 0 0) 0x7f2f740593f0 con 0x7f2f7c153990 2024-03-06T07:19:42.210 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:42.210 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":40,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:26.486055+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:17.283570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:17.762319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/1305657307":"2024-03-07T07:19:26.486020+0000","172.21.15.145:6836/1305657307":"2024-03-07T07:19:26.486020+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f5c06b4d0 msgr2=0x7f2f5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f5c06b4d0 0x7f2f5c06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).stop 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 msgr2=0x7f2f7c0c1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c0c1af0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2f74007de0 tx=0x7f2f7400d4a0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 shutdown_connections 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f5c06b4d0 0x7f2f5c06d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f7c153990 0x7f2f7c0c1af0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f7c1506b0 0x7f2f7c147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 --2- 172.21.15.94:0/149623438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f7c0acc00 0x7f2f7c1454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 >> 172.21.15.94:0/149623438 conn(0x7f2f7c06b1a0 msgr2=0x7f2f7c0702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:42.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 shutdown_connections 2024-03-06T07:19:42.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.210+0000 7f2f80b08640 1 -- 172.21.15.94:0/149623438 wait complete. 2024-03-06T07:19:42.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:19:42.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.323+0000 7fabe43bc640 1 Processor -- start 2024-03-06T07:19:42.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.323+0000 7fabe43bc640 1 -- start start 2024-03-06T07:19:42.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe43bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fabdc0a7ff0 0x7fabdc155900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe43bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc0733f0 con 0x7fabdc152b40 2024-03-06T07:19:42.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe43bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc0c82c0 con 0x7fabdc0724d0 2024-03-06T07:19:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe43bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabdc072d70 con 0x7fabdc0a7ff0 2024-03-06T07:19:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe2959640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fabdc0a7ff0 0x7fabdc155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe2959640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fabdc0a7ff0 0x7fabdc155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60580/0 (socket says 172.21.15.94:60580) 2024-03-06T07:19:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe2959640 1 -- 172.21.15.94:0/3056090710 learned_addr learned my addr 172.21.15.94:0/3056090710 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.324+0000 7fabe2158640 1 --1- 172.21.15.94:0/3056090710 >> v1:172.21.15.94:6789/0 conn(0x7fabdc152b40 0x7fabdc0c8810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36002/0 (socket says 172.21.15.94:36002) 2024-03-06T07:19:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.325+0000 7fabe2959640 1 -- 172.21.15.94:0/3056090710 >> v1:172.21.15.145:6789/0 conn(0x7fabdc0724d0 legacy=0x7fabdc1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.325+0000 7fabe2959640 1 -- 172.21.15.94:0/3056090710 >> v1:172.21.15.94:6789/0 conn(0x7fabdc152b40 legacy=0x7fabdc0c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.325+0000 7fabe2959640 1 -- 172.21.15.94:0/3056090710 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc06fd30 con 0x7fabdc0a7ff0 2024-03-06T07:19:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.325+0000 7fabe2959640 1 --2- 172.21.15.94:0/3056090710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0a7ff0 0x7fabdc155900 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009e50 tx=0x7fabd402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b931c29dfefbe6d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe1156640 1 -- 172.21.15.94:0/3056090710 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd402ca80 con 0x7fabdc0a7ff0 2024-03-06T07:19:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe1156640 1 -- 172.21.15.94:0/3056090710 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd402cc20 con 0x7fabdc0a7ff0 2024-03-06T07:19:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe1156640 1 -- 172.21.15.94:0/3056090710 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd40343f0 con 0x7fabdc0a7ff0 2024-03-06T07:19:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe43bc640 1 -- 172.21.15.94:0/3056090710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0a7ff0 msgr2=0x7fabdc155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe43bc640 1 --2- 172.21.15.94:0/3056090710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0a7ff0 0x7fabdc155900 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009e50 tx=0x7fabd402bee0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe43bc640 1 -- 172.21.15.94:0/3056090710 shutdown_connections 2024-03-06T07:19:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe43bc640 1 --2- 172.21.15.94:0/3056090710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0a7ff0 0x7fabdc155900 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe43bc640 1 -- 172.21.15.94:0/3056090710 >> 172.21.15.94:0/3056090710 conn(0x7fabdc06b970 msgr2=0x7fabdc0aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe43bc640 1 -- 172.21.15.94:0/3056090710 shutdown_connections 2024-03-06T07:19:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.326+0000 7fabe43bc640 1 -- 172.21.15.94:0/3056090710 wait complete. 2024-03-06T07:19:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.327+0000 7fabe43bc640 1 Processor -- start 2024-03-06T07:19:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.327+0000 7fabe43bc640 1 -- start start 2024-03-06T07:19:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe43bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0724d0 0x7fabdc1eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe43bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabdc0a7ff0 0x7fabdc0c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe43bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabdc152b40 0x7fabdc14e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe43bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc070400 con 0x7fabdc0724d0 2024-03-06T07:19:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe43bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc070280 con 0x7fabdc152b40 2024-03-06T07:19:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe43bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc160b80 con 0x7fabdc0a7ff0 2024-03-06T07:19:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0724d0 0x7fabdc1eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0724d0 0x7fabdc1eb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60584/0 (socket says 172.21.15.94:60584) 2024-03-06T07:19:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 -- 172.21.15.94:0/254099156 learned_addr learned my addr 172.21.15.94:0/254099156 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 -- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabdc0a7ff0 msgr2=0x7fabdc0c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabdc0a7ff0 0x7fabdc0c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 -- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabdc152b40 msgr2=0x7fabdc14e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabdc152b40 0x7fabdc14e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 -- 172.21.15.94:0/254099156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc0c5ea0 con 0x7fabdc0724d0 2024-03-06T07:19:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.328+0000 7fabe2158640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0724d0 0x7fabdc1eb7c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00c400 tx=0x7fabcc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.329+0000 7fabd2ffd640 1 -- 172.21.15.94:0/254099156 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc0074e0 con 0x7fabdc0724d0 2024-03-06T07:19:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.329+0000 7fabd2ffd640 1 -- 172.21.15.94:0/254099156 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc007680 con 0x7fabdc0724d0 2024-03-06T07:19:42.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.329+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc1eace0 con 0x7fabdc0724d0 2024-03-06T07:19:42.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.329+0000 7fabd2ffd640 1 -- 172.21.15.94:0/254099156 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc004a00 con 0x7fabdc0724d0 2024-03-06T07:19:42.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.329+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc0c43f0 con 0x7fabdc0724d0 2024-03-06T07:19:42.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.329+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc0adad0 con 0x7fabdc0724d0 2024-03-06T07:19:42.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.330+0000 7fabd2ffd640 1 -- 172.21.15.94:0/254099156 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabcc01a020 con 0x7fabdc0724d0 2024-03-06T07:19:42.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.330+0000 7fabd2ffd640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fabc006b4d0 0x7fabc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.330+0000 7fabd2ffd640 1 -- 172.21.15.94:0/254099156 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fabcc089c00 con 0x7fabdc0724d0 2024-03-06T07:19:42.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.331+0000 7fabe1957640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fabc006b4d0 0x7fabc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.331+0000 7fabe1957640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fabc006b4d0 0x7fabc006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabd8006fd0 tx=0x7fabd8006bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.335+0000 7fabd2ffd640 1 -- 172.21.15.94:0/254099156 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabdc0adad0 con 0x7fabdc0724d0 2024-03-06T07:19:42.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.608+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fabdc04f330 con 0x7fabdc0724d0 2024-03-06T07:19:42.610 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:19:42.608+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:19:42.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.609+0000 7fabd2ffd640 1 -- 172.21.15.94:0/254099156 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) v1 ==== 161+0+0 (secure 0 0 0) 0x7fabcc090080 con 0x7fabdc0724d0 2024-03-06T07:19:42.611 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:19:42.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fabc006b4d0 msgr2=0x7fabc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fabc006b4d0 0x7fabc006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabd8006fd0 tx=0x7fabd8006bc0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0724d0 msgr2=0x7fabdc1eb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0724d0 0x7fabdc1eb7c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00c400 tx=0x7fabcc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 shutdown_connections 2024-03-06T07:19:42.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fabc006b4d0 0x7fabc006d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabdc152b40 0x7fabdc14e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabdc0a7ff0 0x7fabdc0c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 --2- 172.21.15.94:0/254099156 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fabdc0724d0 0x7fabdc1eb7c0 secure :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00c400 tx=0x7fabcc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.612+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 >> 172.21.15.94:0/254099156 conn(0x7fabdc06b970 msgr2=0x7fabdc0ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:42.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.613+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 shutdown_connections 2024-03-06T07:19:42.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.613+0000 7fabe43bc640 1 -- 172.21.15.94:0/254099156 wait complete. 2024-03-06T07:19:42.617 INFO:tasks.ceph.mds.c.smithi094.stderr: -496> 2024-03-06T07:19:31.418+0000 7f75cb463640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:19:42.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:19:42.639 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:19:42.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.727+0000 7fefac481640 1 Processor -- start 2024-03-06T07:19:42.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.727+0000 7fefac481640 1 -- start start 2024-03-06T07:19:42.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.727+0000 7fefac481640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fefa40c2aa0 0x7fefa4155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.727+0000 7fefac481640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa40670c0 con 0x7fefa40c0470 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefac481640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa406b0f0 con 0x7fefa40c26c0 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefac481640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa406b740 con 0x7fefa40c2aa0 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefaaa1e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fefa40c0470 0x7fefa4152350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36014/0 (socket says 172.21.15.94:36014) 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefaaa1e640 1 -- 172.21.15.94:0/1128336259 learned_addr learned my addr 172.21.15.94:0/1128336259 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/1128336259 >> v2:172.21.15.145:3300/0 conn(0x7fefa40c2aa0 0x7fefa4155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705636560 0 0) 0x7fefa40670c0 con 0x7fefa40c0470 2024-03-06T07:19:42.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef90003520 con 0x7fefa40c0470 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1717063139 0 0) 0x7fef90003520 con 0x7fefa40c0470 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 >> v2:172.21.15.145:3300/0 conn(0x7fefa40c2aa0 msgr2=0x7fefa4155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 --2- 172.21.15.94:0/1128336259 >> v2:172.21.15.145:3300/0 conn(0x7fefa40c2aa0 0x7fefa4155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 >> v1:172.21.15.145:6789/0 conn(0x7fefa40c26c0 legacy=0x7fefa406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa406b3d0 con 0x7fefa40c0470 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fef9c0019f0 con 0x7fefa40c0470 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 >> v1:172.21.15.94:6789/0 conn(0x7fefa40c0470 legacy=0x7fefa4152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef900039b0 0x7fef90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef90004800 0x7fef90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef900052e0 0x7fef9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa406b740 con 0x7fef90004800 2024-03-06T07:19:42.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9000c1b0 con 0x7fef900052e0 2024-03-06T07:19:42.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9000c3b0 con 0x7fef900039b0 2024-03-06T07:19:42.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef90004800 0x7fef90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.728+0000 7fefaa21d640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef900039b0 0x7fef90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefaaa1e640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef900052e0 0x7fef9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa9a1c640 1 -- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef900039b0 msgr2=0x7fef90003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef900039b0 0x7fef90003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa9a1c640 1 -- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef900052e0 msgr2=0x7fef9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef900052e0 0x7fef9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefaaa1e640 1 -- 172.21.15.94:0/1128336259 reap_dead start 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa9a1c640 1 -- 172.21.15.94:0/1128336259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90003520 con 0x7fef90004800 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef90004800 0x7fef90004c80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fef94009a90 tx=0x7fef90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2584fe7ae6aae8b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94039070 con 0x7fef90004800 2024-03-06T07:19:42.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9402c680 con 0x7fef90004800 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.729+0000 7fefa921b640 1 -- 172.21.15.94:0/1128336259 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94009690 con 0x7fef90004800 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.730+0000 7fefac481640 1 -- 172.21.15.94:0/1128336259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef90004800 msgr2=0x7fef90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.730+0000 7fefac481640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef90004800 0x7fef90004c80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fef94009a90 tx=0x7fef90005830 comp rx=0 tx=0).stop 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.730+0000 7fefac481640 1 -- 172.21.15.94:0/1128336259 shutdown_connections 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.730+0000 7fefac481640 1 --2- 172.21.15.94:0/1128336259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fef90004800 0x7fef90004c80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.730+0000 7fefac481640 1 -- 172.21.15.94:0/1128336259 >> 172.21.15.94:0/1128336259 conn(0x7fefa40c3b40 msgr2=0x7fefa40c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.730+0000 7fefac481640 1 -- 172.21.15.94:0/1128336259 shutdown_connections 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.730+0000 7fefac481640 1 -- 172.21.15.94:0/1128336259 wait complete. 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.731+0000 7fefac481640 1 Processor -- start 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.731+0000 7fefac481640 1 -- start start 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefac481640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef90004800 0x7fefa4149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefac481640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fefa40ab840 0x7fefa40abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefac481640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fefa40ac200 0x7fefa41edc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefac481640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c20b0 con 0x7fefa40ab840 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefac481640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40c1f30 con 0x7fef90004800 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefac481640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa4066d00 con 0x7fefa40ac200 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fefa40ab840 0x7fefa40abcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fefa40ab840 0x7fefa40abcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60624/0 (socket says 172.21.15.94:60624) 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 -- 172.21.15.94:0/3753816126 learned_addr learned my addr 172.21.15.94:0/3753816126 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 -- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fefa40ac200 msgr2=0x7fefa41edc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fefa40ac200 0x7fefa41edc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 -- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef90004800 msgr2=0x7fefa4149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef90004800 0x7fefa4149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 -- 172.21.15.94:0/3753816126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa41461e0 con 0x7fefa40ab840 2024-03-06T07:19:42.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefa9a1c640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fefa40ab840 0x7fefa40abcc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fef94037040 tx=0x7fef94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fef9affd640 1 -- 172.21.15.94:0/3753816126 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94042020 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fef9affd640 1 -- 172.21.15.94:0/3753816126 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9403a040 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa41464c0 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.732+0000 7fef9affd640 1 -- 172.21.15.94:0/3753816126 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94039070 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.733+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa4146a80 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.734+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa404f330 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.734+0000 7fef9affd640 1 -- 172.21.15.94:0/3753816126 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef94045020 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.734+0000 7fef9affd640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fef8806b4d0 0x7fef8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.734+0000 7fef9affd640 1 -- 172.21.15.94:0/3753816126 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fef940ab790 con 0x7fefa40ab840 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.735+0000 7fefaa21d640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fef8806b4d0 0x7fef8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:42.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.735+0000 7fefaa21d640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fef8806b4d0 0x7fef8806d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fefa41507d0 tx=0x7fefa0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:42.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.739+0000 7fef9affd640 1 -- 172.21.15.94:0/3753816126 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef9404c070 con 0x7fefa40ab840 2024-03-06T07:19:42.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:42.944+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fefa4149fe0 con 0x7fefa40ab840 2024-03-06T07:19:43.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.620+0000 7fef9affd640 1 -- 172.21.15.94:0/3753816126 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fef9407a460 con 0x7fefa40ab840 2024-03-06T07:19:43.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fef8806b4d0 msgr2=0x7fef8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fef8806b4d0 0x7fef8806d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fefa41507d0 tx=0x7fefa0009040 comp rx=0 tx=0).stop 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fefa40ab840 msgr2=0x7fefa40abcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fefa40ab840 0x7fefa40abcc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fef94037040 tx=0x7fef94004730 comp rx=0 tx=0).stop 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 shutdown_connections 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fef8806b4d0 0x7fef8806d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fefa40ac200 0x7fefa41edc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fefa40ab840 0x7fefa40abcc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 --2- 172.21.15.94:0/3753816126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef90004800 0x7fefa4149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 >> 172.21.15.94:0/3753816126 conn(0x7fefa40c3b40 msgr2=0x7fefa414c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 shutdown_connections 2024-03-06T07:19:43.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.623+0000 7fefac481640 1 -- 172.21.15.94:0/3753816126 wait complete. 2024-03-06T07:19:43.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:19:43.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.786+0000 7ffb603d6640 1 Processor -- start 2024-03-06T07:19:43.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.786+0000 7ffb603d6640 1 -- start start 2024-03-06T07:19:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb603d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb58151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb603d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb58153390 0x7ffb58153820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb603d6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5806e850 con 0x7ffb580a9830 2024-03-06T07:19:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb603d6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5806d2b0 con 0x7ffb58153390 2024-03-06T07:19:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb603d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5806d590 con 0x7ffb580c22b0 2024-03-06T07:19:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e973640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb58153390 0x7ffb58153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:43.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e973640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb58153390 0x7ffb58153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60626/0 (socket says 172.21.15.94:60626) 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e973640 1 -- 172.21.15.94:0/2714311388 learned_addr learned my addr 172.21.15.94:0/2714311388 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e973640 1 -- 172.21.15.94:0/2714311388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 msgr2=0x7ffb58151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e172640 1 --2- 172.21.15.94:0/2714311388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb58151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e973640 1 --2- 172.21.15.94:0/2714311388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb58151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e973640 1 -- 172.21.15.94:0/2714311388 >> v1:172.21.15.145:6789/0 conn(0x7ffb580a9830 legacy=0x7ffb5814ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e973640 1 -- 172.21.15.94:0/2714311388 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5806d870 con 0x7ffb58153390 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.787+0000 7ffb5e172640 1 --2- 172.21.15.94:0/2714311388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb58151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.788+0000 7ffb5e973640 1 --2- 172.21.15.94:0/2714311388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb58153390 0x7ffb58153820 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009e50 tx=0x7ffb5002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa995567c7096299 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.788+0000 7ffb5d170640 1 -- 172.21.15.94:0/2714311388 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5002ca70 con 0x7ffb58153390 2024-03-06T07:19:43.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.788+0000 7ffb5d170640 1 -- 172.21.15.94:0/2714311388 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5002cc10 con 0x7ffb58153390 2024-03-06T07:19:43.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.788+0000 7ffb5d170640 1 -- 172.21.15.94:0/2714311388 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb500343f0 con 0x7ffb58153390 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 -- 172.21.15.94:0/2714311388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb58153390 msgr2=0x7ffb58153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 --2- 172.21.15.94:0/2714311388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb58153390 0x7ffb58153820 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009e50 tx=0x7ffb5002bed0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 -- 172.21.15.94:0/2714311388 shutdown_connections 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 --2- 172.21.15.94:0/2714311388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb58153390 0x7ffb58153820 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 --2- 172.21.15.94:0/2714311388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb58151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 -- 172.21.15.94:0/2714311388 >> 172.21.15.94:0/2714311388 conn(0x7ffb580c65b0 msgr2=0x7ffb580c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 -- 172.21.15.94:0/2714311388 shutdown_connections 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.789+0000 7ffb603d6640 1 -- 172.21.15.94:0/2714311388 wait complete. 2024-03-06T07:19:43.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.790+0000 7ffb603d6640 1 Processor -- start 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.790+0000 7ffb603d6640 1 -- start start 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb580a9830 0x7ffb580bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb581ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb58153390 0x7ffb580c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5814f7e0 con 0x7ffb580a9830 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5814f660 con 0x7ffb58153390 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb580c2690 con 0x7ffb580c22b0 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb580a9830 0x7ffb580bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb580a9830 0x7ffb580bcd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60630/0 (socket says 172.21.15.94:60630) 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 -- 172.21.15.94:0/3585528903 learned_addr learned my addr 172.21.15.94:0/3585528903 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:43.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5d971640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb581ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e973640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb58153390 0x7ffb580c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 -- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 msgr2=0x7ffb581ecbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb581ecbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 -- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb58153390 msgr2=0x7ffb580c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb58153390 0x7ffb580c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 -- 172.21.15.94:0/3585528903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb580be040 con 0x7ffb580a9830 2024-03-06T07:19:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e973640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb58153390 0x7ffb580c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:43.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5d971640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb581ecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:43.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb5e172640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb580a9830 0x7ffb580bcd50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400c820 tx=0x7ffb5400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:43.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb4effd640 1 -- 172.21.15.94:0/3585528903 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54015070 con 0x7ffb580a9830 2024-03-06T07:19:43.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb4effd640 1 -- 172.21.15.94:0/3585528903 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb54007a30 con 0x7ffb580a9830 2024-03-06T07:19:43.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb580bd730 con 0x7ffb580a9830 2024-03-06T07:19:43.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb4effd640 1 -- 172.21.15.94:0/3585528903 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54007e00 con 0x7ffb580a9830 2024-03-06T07:19:43.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.791+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb580be2e0 con 0x7ffb580a9830 2024-03-06T07:19:43.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.792+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5804f330 con 0x7ffb580a9830 2024-03-06T07:19:43.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.793+0000 7ffb4effd640 1 -- 172.21.15.94:0/3585528903 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb54018070 con 0x7ffb580a9830 2024-03-06T07:19:43.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.793+0000 7ffb4effd640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3c06b4d0 0x7ffb3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:43.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.793+0000 7ffb4effd640 1 -- 172.21.15.94:0/3585528903 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6273+0+0 (secure 0 0 0) 0x7ffb5408a410 con 0x7ffb580a9830 2024-03-06T07:19:43.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.793+0000 7ffb5d971640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3c06b4d0 0x7ffb3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:43.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.794+0000 7ffb5d971640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3c06b4d0 0x7ffb3c06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb581e6a10 tx=0x7ffb48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:43.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.797+0000 7ffb4effd640 1 -- 172.21.15.94:0/3585528903 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb54058fc0 con 0x7ffb580a9830 2024-03-06T07:19:43.826 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:43.829 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:19:43.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:43.972+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffb5815d580 con 0x7ffb580a9830 2024-03-06T07:19:44.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.971+0000 7ffb4effd640 1 -- 172.21.15.94:0/3585528903 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v42) v1 ==== 181+0+0 (secure 0 0 0) 0x7ffb540588c0 con 0x7ffb580a9830 2024-03-06T07:19:44.973 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3c06b4d0 msgr2=0x7ffb3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3c06b4d0 0x7ffb3c06d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb581e6a10 tx=0x7ffb48009040 comp rx=0 tx=0).stop 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb580a9830 msgr2=0x7ffb580bcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb580a9830 0x7ffb580bcd50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400c820 tx=0x7ffb5400cc20 comp rx=0 tx=0).stop 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 shutdown_connections 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3c06b4d0 0x7ffb3c06d990 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb581e6a10 tx=0x7ffb48009040 comp rx=0 tx=0).stop 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb58153390 0x7ffb580c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:44.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb580c22b0 0x7ffb581ecbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:44.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 --2- 172.21.15.94:0/3585528903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb580a9830 0x7ffb580bcd50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:44.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.974+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 >> 172.21.15.94:0/3585528903 conn(0x7ffb580c65b0 msgr2=0x7ffb581500d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:44.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.975+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 shutdown_connections 2024-03-06T07:19:44.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:44.975+0000 7ffb603d6640 1 -- 172.21.15.94:0/3585528903 wait complete. 2024-03-06T07:19:44.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:19:45.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.093+0000 7f7b1493e640 1 Processor -- start 2024-03-06T07:19:45.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.093+0000 7f7b1493e640 1 -- start start 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.094+0000 7f7b1493e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b100c1860 0x7f7b1006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.094+0000 7f7b1493e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7b1006c510 0x7f7b10158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.094+0000 7f7b1493e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b10155820 con 0x7f7b101527a0 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.094+0000 7f7b1493e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b10158820 con 0x7f7b1006c510 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.094+0000 7f7b1493e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100c2b10 con 0x7f7b100c1860 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.094+0000 7f7b0ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7b1006c510 0x7f7b10158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0e59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7b101527a0 0x7f7b10155040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36024/0 (socket says 172.21.15.94:36024) 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0e59c640 1 -- 172.21.15.94:0/1016205966 learned_addr learned my addr 172.21.15.94:0/1016205966 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0dd9b640 1 --2- 172.21.15.94:0/1016205966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b100c1860 0x7f7b1006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:45.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0ed9d640 1 -- 172.21.15.94:0/1016205966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b100c1860 msgr2=0x7f7b1006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:45.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0ed9d640 1 --2- 172.21.15.94:0/1016205966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b100c1860 0x7f7b1006bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0ed9d640 1 -- 172.21.15.94:0/1016205966 >> v1:172.21.15.94:6789/0 conn(0x7f7b101527a0 legacy=0x7f7b10155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:45.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0ed9d640 1 -- 172.21.15.94:0/1016205966 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10155b00 con 0x7f7b1006c510 2024-03-06T07:19:45.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.095+0000 7f7b0d59a640 1 -- 172.21.15.94:0/1016205966 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169275043 0 0) 0x7f7b10155820 con 0x7f7b101527a0 2024-03-06T07:19:45.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b0ed9d640 1 --2- 172.21.15.94:0/1016205966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b1006c510 0x7f7b10158cb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b0802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d64c5f410bbac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b0d59a640 1 -- 172.21.15.94:0/1016205966 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08039070 con 0x7f7b1006c510 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b0d59a640 1 -- 172.21.15.94:0/1016205966 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0802cbc0 con 0x7f7b1006c510 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b0d59a640 1 -- 172.21.15.94:0/1016205966 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08034440 con 0x7f7b1006c510 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b1493e640 1 -- 172.21.15.94:0/1016205966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b1006c510 msgr2=0x7f7b10158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b1493e640 1 --2- 172.21.15.94:0/1016205966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b1006c510 0x7f7b10158cb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b0802be10 comp rx=0 tx=0).stop 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b1493e640 1 -- 172.21.15.94:0/1016205966 shutdown_connections 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b1493e640 1 --2- 172.21.15.94:0/1016205966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b1006c510 0x7f7b10158cb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b1493e640 1 --2- 172.21.15.94:0/1016205966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b100c1860 0x7f7b1006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.096+0000 7f7b1493e640 1 -- 172.21.15.94:0/1016205966 >> 172.21.15.94:0/1016205966 conn(0x7f7b100c3960 msgr2=0x7f7b100c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.097+0000 7f7b1493e640 1 -- 172.21.15.94:0/1016205966 shutdown_connections 2024-03-06T07:19:45.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.097+0000 7f7b1493e640 1 -- 172.21.15.94:0/1016205966 wait complete. 2024-03-06T07:19:45.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.098+0000 7f7b1493e640 1 Processor -- start 2024-03-06T07:19:45.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.098+0000 7f7b1493e640 1 -- start start 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.098+0000 7f7b1493e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b1006c510 0x7f7b1014a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.098+0000 7f7b1493e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b100c1860 0x7f7b1014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.098+0000 7f7b1493e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b101527a0 0x7f7b1014cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b1493e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10152d00 con 0x7f7b101527a0 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b1493e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10152b80 con 0x7f7b100c1860 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b1493e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1006db70 con 0x7f7b1006c510 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b101527a0 0x7f7b1014cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b101527a0 0x7f7b1014cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60646/0 (socket says 172.21.15.94:60646) 2024-03-06T07:19:45.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 -- 172.21.15.94:0/2161907253 learned_addr learned my addr 172.21.15.94:0/2161907253 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:45.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 -- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b1006c510 msgr2=0x7f7b1014a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:45.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b1006c510 0x7f7b1014a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 -- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b100c1860 msgr2=0x7f7b1014ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:45.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b100c1860 0x7f7b1014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 -- 172.21.15.94:0/2161907253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b101eb080 con 0x7f7b101527a0 2024-03-06T07:19:45.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b0ed9d640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b101527a0 0x7f7b1014cfa0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:45.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7afeffd640 1 -- 172.21.15.94:0/2161907253 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08039050 con 0x7f7b101527a0 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7afeffd640 1 -- 172.21.15.94:0/2161907253 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0803a040 con 0x7f7b101527a0 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1014ad80 con 0x7f7b101527a0 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7afeffd640 1 -- 172.21.15.94:0/2161907253 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08003990 con 0x7f7b101527a0 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.099+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b100c5c40 con 0x7f7b101527a0 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.100+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1004f330 con 0x7f7b101527a0 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.101+0000 7f7afeffd640 1 -- 172.21.15.94:0/2161907253 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b08047030 con 0x7f7b101527a0 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.101+0000 7f7afeffd640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7af006b4d0 0x7f7af006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:45.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.101+0000 7f7afeffd640 1 -- 172.21.15.94:0/2161907253 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f7b080ac440 con 0x7f7b101527a0 2024-03-06T07:19:45.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.101+0000 7f7b0e59c640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7af006b4d0 0x7f7af006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:45.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.102+0000 7f7b0e59c640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7af006b4d0 0x7f7af006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7b04007fd0 tx=0x7f7b04007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:45.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.105+0000 7f7afeffd640 1 -- 172.21.15.94:0/2161907253 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b0804e060 con 0x7f7b101527a0 2024-03-06T07:19:45.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.276+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7b10067590 con 0x7f7b101527a0 2024-03-06T07:19:45.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.972+0000 7f7afeffd640 1 -- 172.21.15.94:0/2161907253 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v43) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7b0807b2d0 con 0x7f7b101527a0 2024-03-06T07:19:45.975 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:19:45.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.976+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7af006b4d0 msgr2=0x7f7af006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:45.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.976+0000 7f7b1493e640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7af006b4d0 0x7f7af006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7b04007fd0 tx=0x7f7b04007960 comp rx=0 tx=0).stop 2024-03-06T07:19:45.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.976+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b101527a0 msgr2=0x7f7b1014cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:45.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.976+0000 7f7b1493e640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b101527a0 0x7f7b1014cfa0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b08004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:45.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 shutdown_connections 2024-03-06T07:19:45.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7af006b4d0 0x7f7af006d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b101527a0 0x7f7b1014cfa0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b100c1860 0x7f7b1014ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 --2- 172.21.15.94:0/2161907253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b1006c510 0x7f7b1014a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:45.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 >> 172.21.15.94:0/2161907253 conn(0x7f7b100c3960 msgr2=0x7f7b10071fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:45.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 shutdown_connections 2024-03-06T07:19:45.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:45.978+0000 7f7b1493e640 1 -- 172.21.15.94:0/2161907253 wait complete. 2024-03-06T07:19:45.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:19:46.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.102+0000 7f14f52c9640 1 Processor -- start 2024-03-06T07:19:46.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.102+0000 7f14f52c9640 1 -- start start 2024-03-06T07:19:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14f52c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14f01538c0 0x7f14f01494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14f52c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f0154330 con 0x7f14f00ab310 2024-03-06T07:19:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14f52c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f0153dc0 con 0x7f14f00a97c0 2024-03-06T07:19:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14f52c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f00c46a0 con 0x7f14f01538c0 2024-03-06T07:19:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14ee7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14f01538c0 0x7f14f01494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14eeffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f14f00ab310 0x7f14f00ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36030/0 (socket says 172.21.15.94:36030) 2024-03-06T07:19:46.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14eeffd640 1 -- 172.21.15.94:0/878082171 learned_addr learned my addr 172.21.15.94:0/878082171 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:46.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14ee7fc640 1 -- 172.21.15.94:0/878082171 >> v1:172.21.15.145:6789/0 conn(0x7f14f00a97c0 legacy=0x7f14f00c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14ee7fc640 1 -- 172.21.15.94:0/878082171 >> v1:172.21.15.94:6789/0 conn(0x7f14f00ab310 legacy=0x7f14f00ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14edffb640 1 -- 172.21.15.94:0/878082171 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736298307 0 0) 0x7f14f0154330 con 0x7f14f00ab310 2024-03-06T07:19:46.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.103+0000 7f14ee7fc640 1 -- 172.21.15.94:0/878082171 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f00c5d30 con 0x7f14f01538c0 2024-03-06T07:19:46.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.104+0000 7f14ee7fc640 1 --2- 172.21.15.94:0/878082171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f01538c0 0x7f14f01494a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f14e40097a0 tx=0x7f14e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6fc3e5099093db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.104+0000 7f14edffb640 1 -- 172.21.15.94:0/878082171 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e40315a0 con 0x7f14f01538c0 2024-03-06T07:19:46.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.104+0000 7f14edffb640 1 -- 172.21.15.94:0/878082171 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e4037070 con 0x7f14f01538c0 2024-03-06T07:19:46.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.104+0000 7f14edffb640 1 -- 172.21.15.94:0/878082171 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4031960 con 0x7f14f01538c0 2024-03-06T07:19:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.105+0000 7f14f52c9640 1 -- 172.21.15.94:0/878082171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f01538c0 msgr2=0x7f14f01494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.105+0000 7f14f52c9640 1 --2- 172.21.15.94:0/878082171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f01538c0 0x7f14f01494a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f14e40097a0 tx=0x7f14e402b840 comp rx=0 tx=0).stop 2024-03-06T07:19:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.105+0000 7f14f52c9640 1 -- 172.21.15.94:0/878082171 shutdown_connections 2024-03-06T07:19:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.105+0000 7f14f52c9640 1 --2- 172.21.15.94:0/878082171 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f01538c0 0x7f14f01494a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.105+0000 7f14f52c9640 1 -- 172.21.15.94:0/878082171 >> 172.21.15.94:0/878082171 conn(0x7f14f00c6710 msgr2=0x7f14f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.105+0000 7f14f52c9640 1 -- 172.21.15.94:0/878082171 shutdown_connections 2024-03-06T07:19:46.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.105+0000 7f14f52c9640 1 -- 172.21.15.94:0/878082171 wait complete. 2024-03-06T07:19:46.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.106+0000 7f14f52c9640 1 Processor -- start 2024-03-06T07:19:46.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.106+0000 7f14f52c9640 1 -- start start 2024-03-06T07:19:46.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f00a97c0 0x7f14f00bb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14f00ab310 0x7f14f006d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14f01538c0 0x7f14f0072790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f00c5240 con 0x7f14f00a97c0 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f00c62f0 con 0x7f14f01538c0 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f00c53c0 con 0x7f14f00ab310 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f00a97c0 0x7f14f00bb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f00a97c0 0x7f14f00bb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60664/0 (socket says 172.21.15.94:60664) 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 -- 172.21.15.94:0/378206321 learned_addr learned my addr 172.21.15.94:0/378206321 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 -- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14f00ab310 msgr2=0x7f14f006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14f00ab310 0x7f14f006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 -- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14f01538c0 msgr2=0x7f14f0072790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14f01538c0 0x7f14f0072790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 -- 172.21.15.94:0/378206321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f0073350 con 0x7f14f00a97c0 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14eeffd640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f00a97c0 0x7f14f00bb730 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f14e000c890 tx=0x7f14e000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14d77fe640 1 -- 172.21.15.94:0/378206321 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0018070 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14d77fe640 1 -- 172.21.15.94:0/378206321 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e000a640 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f006c490 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14d77fe640 1 -- 172.21.15.94:0/378206321 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e000aa30 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.107+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f006ca50 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.108+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f004f330 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.109+0000 7f14d77fe640 1 -- 172.21.15.94:0/378206321 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14e001b070 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.109+0000 7f14d77fe640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14dc06b4d0 0x7f14dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.109+0000 7f14d77fe640 1 -- 172.21.15.94:0/378206321 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f14e0089740 con 0x7f14f00a97c0 2024-03-06T07:19:46.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.110+0000 7f14ee7fc640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14dc06b4d0 0x7f14dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.110+0000 7f14ee7fc640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14dc06b4d0 0x7f14dc06d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f14e4036000 tx=0x7f14e4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.113+0000 7f14d77fe640 1 -- 172.21.15.94:0/378206321 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14e0058770 con 0x7f14f00a97c0 2024-03-06T07:19:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.285+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f14f00bbc70 con 0x7f14f00a97c0 2024-03-06T07:19:46.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.286+0000 7f14d77fe640 1 -- 172.21.15.94:0/378206321 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7f14e0089070 con 0x7f14f00a97c0 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14dc06b4d0 msgr2=0x7f14dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14dc06b4d0 0x7f14dc06d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f14e4036000 tx=0x7f14e4030000 comp rx=0 tx=0).stop 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f00a97c0 msgr2=0x7f14f00bb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f00a97c0 0x7f14f00bb730 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f14e000c890 tx=0x7f14e000cd50 comp rx=0 tx=0).stop 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 shutdown_connections 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14dc06b4d0 0x7f14dc06d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14f01538c0 0x7f14f0072790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14f00ab310 0x7f14f006d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 --2- 172.21.15.94:0/378206321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14f00a97c0 0x7f14f00bb730 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 >> 172.21.15.94:0/378206321 conn(0x7f14f00c6710 msgr2=0x7f14f00c4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 shutdown_connections 2024-03-06T07:19:46.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.288+0000 7f14f52c9640 1 -- 172.21.15.94:0/378206321 wait complete. 2024-03-06T07:19:46.301 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:19:46.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.401+0000 7fd4719c7640 1 Processor -- start 2024-03-06T07:19:46.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.401+0000 7fd4719c7640 1 -- start start 2024-03-06T07:19:46.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.401+0000 7fd4719c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 0x7fd46c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.401+0000 7fd4719c7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd46c0c5380 0x7fd46c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.401+0000 7fd4719c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c0b0450 con 0x7fd46c06ca70 2024-03-06T07:19:46.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.401+0000 7fd4719c7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c06c020 con 0x7fd46c0c5380 2024-03-06T07:19:46.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.401+0000 7fd4719c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0c77f0 con 0x7fd46c0b0070 2024-03-06T07:19:46.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd4627fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd46c06ca70 0x7fd46c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36038/0 (socket says 172.21.15.94:36038) 2024-03-06T07:19:46.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd4627fc640 1 -- 172.21.15.94:0/3999527129 learned_addr learned my addr 172.21.15.94:0/3999527129 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3999527129 >> v2:172.21.15.145:3300/0 conn(0x7fd46c0c5380 0x7fd46c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46affd640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 0x7fd46c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698306392 0 0) 0x7fd454001110 con 0x7fd46c06ca70 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd458003520 con 0x7fd46c06ca70 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3483052362 0 0) 0x7fd458003520 con 0x7fd46c06ca70 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 msgr2=0x7fd46c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 0x7fd46c155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 >> v2:172.21.15.145:3300/0 conn(0x7fd46c0c5380 msgr2=0x7fd46c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 --2- 172.21.15.94:0/3999527129 >> v2:172.21.15.145:3300/0 conn(0x7fd46c0c5380 0x7fd46c0c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0c7ad0 con 0x7fd46c06ca70 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fd454001840 con 0x7fd46c06ca70 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 >> v1:172.21.15.94:6789/0 conn(0x7fd46c06ca70 legacy=0x7fd46c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd458003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd458004800 0x7fd458004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 0x7fd4580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c06c020 con 0x7fd4580039b0 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0c77f0 con 0x7fd458004800 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45800b9c0 con 0x7fd46c0b0070 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46affd640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd458003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd4627fc640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd458004800 0x7fd458004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.402+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 0x7fd4580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46affd640 1 -- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 msgr2=0x7fd4580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46affd640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 0x7fd4580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46affd640 1 -- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd458004800 msgr2=0x7fd458004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46affd640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd458004800 0x7fd458004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46affd640 1 -- 172.21.15.94:0/3999527129 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd458003520 con 0x7fd4580039b0 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c0b0070 0x7fd4580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd4627fc640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd458004800 0x7fd458004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46affd640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd458003dd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009af0 tx=0x7fd458005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e5e88b04782892 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.403+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c039070 con 0x7fd4580039b0 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c02c6b0 con 0x7fd4580039b0 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd46a7fc640 1 -- 172.21.15.94:0/3999527129 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c009690 con 0x7fd4580039b0 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd4719c7640 1 -- 172.21.15.94:0/3999527129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 msgr2=0x7fd458003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd4719c7640 1 --2- 172.21.15.94:0/3999527129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd458003dd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009af0 tx=0x7fd458005830 comp rx=0 tx=0).stop 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd46b7fe640 1 -- 172.21.15.94:0/3999527129 reap_dead start 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd4719c7640 1 -- 172.21.15.94:0/3999527129 shutdown_connections 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd4719c7640 1 -- 172.21.15.94:0/3999527129 >> 172.21.15.94:0/3999527129 conn(0x7fd46c0ac590 msgr2=0x7fd46c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd4719c7640 1 -- 172.21.15.94:0/3999527129 shutdown_connections 2024-03-06T07:19:46.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.404+0000 7fd4719c7640 1 -- 172.21.15.94:0/3999527129 wait complete. 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.405+0000 7fd4719c7640 1 Processor -- start 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.405+0000 7fd4719c7640 1 -- start start 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4719c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd46c1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4719c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd46c14e100 0x7fd46c1ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4719c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c1542e0 0x7fd46c14cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4719c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c06cef0 con 0x7fd4580039b0 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4719c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c06c550 con 0x7fd46c14e100 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4719c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c06da80 con 0x7fd46c1542e0 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd46c1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd46c1591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60668/0 (socket says 172.21.15.94:60668) 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 -- 172.21.15.94:0/3964547015 learned_addr learned my addr 172.21.15.94:0/3964547015 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4627fc640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd46c14e100 0x7fd46c1ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46b7fe640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c1542e0 0x7fd46c14cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 -- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c1542e0 msgr2=0x7fd46c14cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c1542e0 0x7fd46c14cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 -- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd46c14e100 msgr2=0x7fd46c1ead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd46c14e100 0x7fd46c1ead60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 -- 172.21.15.94:0/3964547015 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c158460 con 0x7fd4580039b0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd4627fc640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd46c14e100 0x7fd46c1ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.406+0000 7fd46affd640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd46c1591b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd45c037040 tx=0x7fd45c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.407+0000 7fd468ff9640 1 -- 172.21.15.94:0/3964547015 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c042020 con 0x7fd4580039b0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.407+0000 7fd468ff9640 1 -- 172.21.15.94:0/3964547015 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c03a040 con 0x7fd4580039b0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.407+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c1596f0 con 0x7fd4580039b0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.407+0000 7fd468ff9640 1 -- 172.21.15.94:0/3964547015 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c039070 con 0x7fd4580039b0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.407+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c1589b0 con 0x7fd4580039b0 2024-03-06T07:19:46.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.408+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46c04f330 con 0x7fd4580039b0 2024-03-06T07:19:46.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.408+0000 7fd468ff9640 1 -- 172.21.15.94:0/3964547015 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd45c045020 con 0x7fd4580039b0 2024-03-06T07:19:46.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.408+0000 7fd468ff9640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.408+0000 7fd468ff9640 1 -- 172.21.15.94:0/3964547015 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd45c0ab530 con 0x7fd4580039b0 2024-03-06T07:19:46.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.409+0000 7fd4627fc640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.409+0000 7fd4627fc640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd44c06b4d0 0x7fd44c06d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd454000f60 tx=0x7fd454000ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.413+0000 7fd468ff9640 1 -- 172.21.15.94:0/3964547015 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd45c04c070 con 0x7fd4580039b0 2024-03-06T07:19:46.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.588+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd46c06d860 con 0x7fd4580039b0 2024-03-06T07:19:46.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.591+0000 7fd468ff9640 1 -- 172.21.15.94:0/3964547015 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd45c07a560 con 0x7fd4580039b0 2024-03-06T07:19:46.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd44c06b4d0 msgr2=0x7fd44c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd44c06b4d0 0x7fd44c06d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd454000f60 tx=0x7fd454000ef0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 msgr2=0x7fd46c1591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd46c1591b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd45c037040 tx=0x7fd45c004730 comp rx=0 tx=0).stop 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 shutdown_connections 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd46c1542e0 0x7fd46c14cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd46c14e100 0x7fd46c1ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 --2- 172.21.15.94:0/3964547015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4580039b0 0x7fd46c1591b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 >> 172.21.15.94:0/3964547015 conn(0x7fd46c0ac590 msgr2=0x7fd46c0ae630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:46.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 shutdown_connections 2024-03-06T07:19:46.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.593+0000 7fd4719c7640 1 -- 172.21.15.94:0/3964547015 wait complete. 2024-03-06T07:19:46.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-03-06T07:19:46.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.710+0000 7faa7939a640 1 Processor -- start 2024-03-06T07:19:46.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.710+0000 7faa7939a640 1 -- start start 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa7939a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 0x7faa74155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa7939a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faa74158b60 0x7faa74158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa7939a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740c73d0 con 0x7faa740c88c0 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa7939a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa740c8dd0 con 0x7faa74158b60 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa7939a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa740be2e0 con 0x7faa7414f360 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa6bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7faa740c88c0 0x7faa740c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36048/0 (socket says 172.21.15.94:36048) 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa6bfff640 1 -- 172.21.15.94:0/2589349593 learned_addr learned my addr 172.21.15.94:0/2589349593 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa72ffd640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 0x7faa74155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa737fe640 1 --2- 172.21.15.94:0/2589349593 >> v2:172.21.15.145:3300/0 conn(0x7faa74158b60 0x7faa74158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788359069 0 0) 0x7faa740c73d0 con 0x7faa740c88c0 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60003520 con 0x7faa740c88c0 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3583956604 0 0) 0x7faa60003520 con 0x7faa740c88c0 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 msgr2=0x7faa74155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 0x7faa74155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 >> v2:172.21.15.145:3300/0 conn(0x7faa74158b60 msgr2=0x7faa74158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 --2- 172.21.15.94:0/2589349593 >> v2:172.21.15.145:3300/0 conn(0x7faa74158b60 0x7faa74158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740c7670 con 0x7faa740c88c0 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7faa5c001670 con 0x7faa740c88c0 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 >> v1:172.21.15.94:6789/0 conn(0x7faa740c88c0 legacy=0x7faa740c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa60004800 0x7faa60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 0x7faa600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740be2e0 con 0x7faa600039b0 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6000b9c0 con 0x7faa60004800 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6000bb40 con 0x7faa7414f360 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.711+0000 7faa72ffd640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa6bfff640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa60004800 0x7faa60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa737fe640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 0x7faa600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa72ffd640 1 -- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 msgr2=0x7faa600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa72ffd640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 0x7faa600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa72ffd640 1 -- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa60004800 msgr2=0x7faa60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa72ffd640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa60004800 0x7faa60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa72ffd640 1 -- 172.21.15.94:0/2589349593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa60003520 con 0x7faa600039b0 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa737fe640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7414f360 0x7faa600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.712+0000 7faa72ffd640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa60003dd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa64009af0 tx=0x7faa60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff6355b3a7aa7fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64039070 con 0x7faa600039b0 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6402c6b0 con 0x7faa600039b0 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa727fc640 1 -- 172.21.15.94:0/2589349593 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64009690 con 0x7faa600039b0 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa7939a640 1 -- 172.21.15.94:0/2589349593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 msgr2=0x7faa60003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa7939a640 1 --2- 172.21.15.94:0/2589349593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa60003dd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa64009af0 tx=0x7faa60005830 comp rx=0 tx=0).stop 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa737fe640 1 -- 172.21.15.94:0/2589349593 reap_dead start 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa7939a640 1 -- 172.21.15.94:0/2589349593 shutdown_connections 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa7939a640 1 -- 172.21.15.94:0/2589349593 >> 172.21.15.94:0/2589349593 conn(0x7faa740ac570 msgr2=0x7faa740ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa7939a640 1 -- 172.21.15.94:0/2589349593 shutdown_connections 2024-03-06T07:19:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.713+0000 7faa7939a640 1 -- 172.21.15.94:0/2589349593 wait complete. 2024-03-06T07:19:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.714+0000 7faa7939a640 1 Processor -- start 2024-03-06T07:19:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.714+0000 7faa7939a640 1 -- start start 2024-03-06T07:19:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa7939a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa741ee070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa7939a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa740afc30 0x7faa741e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa7939a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa741e7ff0 0x7faa741ec7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa7939a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740c5a50 con 0x7faa600039b0 2024-03-06T07:19:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa7939a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa740c7c30 con 0x7faa740afc30 2024-03-06T07:19:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa7939a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa740c5bd0 con 0x7faa741e7ff0 2024-03-06T07:19:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa741ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa741ee070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60680/0 (socket says 172.21.15.94:60680) 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 -- 172.21.15.94:0/1066215189 learned_addr learned my addr 172.21.15.94:0/1066215189 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 -- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa741e7ff0 msgr2=0x7faa741ec7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa737fe640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa741e7ff0 0x7faa741ec7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa6bfff640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa740afc30 0x7faa741e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa741e7ff0 0x7faa741ec7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 -- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa740afc30 msgr2=0x7faa741e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa740afc30 0x7faa741e7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 -- 172.21.15.94:0/1066215189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa741e8470 con 0x7faa600039b0 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa6bfff640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa740afc30 0x7faa741e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.715+0000 7faa72ffd640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa741ee070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faa64009af0 tx=0x7faa64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.716+0000 7faa6b7fe640 1 -- 172.21.15.94:0/1066215189 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64042020 con 0x7faa600039b0 2024-03-06T07:19:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.716+0000 7faa6b7fe640 1 -- 172.21.15.94:0/1066215189 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6403a040 con 0x7faa600039b0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.716+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa741ee940 con 0x7faa600039b0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.716+0000 7faa6b7fe640 1 -- 172.21.15.94:0/1066215189 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64039070 con 0x7faa600039b0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.716+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa741ebd00 con 0x7faa600039b0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.717+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7404f330 con 0x7faa600039b0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.717+0000 7faa6b7fe640 1 -- 172.21.15.94:0/1066215189 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa64045020 con 0x7faa600039b0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.717+0000 7faa6b7fe640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.717+0000 7faa6b7fe640 1 -- 172.21.15.94:0/1066215189 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7faa640ab550 con 0x7faa600039b0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.718+0000 7faa6bfff640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.718+0000 7faa6bfff640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa74152a90 tx=0x7faa5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:46.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.722+0000 7faa6b7fe640 1 -- 172.21.15.94:0/1066215189 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa6404c070 con 0x7faa600039b0 2024-03-06T07:19:46.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.958+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7faa7406cda0 con 0x7faa600039b0 2024-03-06T07:19:46.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.978+0000 7faa6b7fe640 1 -- 172.21.15.94:0/1066215189 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v103) v1 ==== 143+0+0 (secure 0 0 0) 0x7faa6407a4f0 con 0x7faa600039b0 2024-03-06T07:19:46.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 msgr2=0x7faa5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa74152a90 tx=0x7faa5c009040 comp rx=0 tx=0).stop 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 msgr2=0x7faa741ee070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa741ee070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7faa64009af0 tx=0x7faa64004730 comp rx=0 tx=0).stop 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 shutdown_connections 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa741e7ff0 0x7faa741ec7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa740afc30 0x7faa741e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 --2- 172.21.15.94:0/1066215189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa600039b0 0x7faa741ee070 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.980+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 >> 172.21.15.94:0/1066215189 conn(0x7faa740ac570 msgr2=0x7faa740c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.981+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 shutdown_connections 2024-03-06T07:19:46.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:46.981+0000 7faa7939a640 1 -- 172.21.15.94:0/1066215189 wait complete. 2024-03-06T07:19:46.993 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. ... ok 2024-03-06T07:19:46.994 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is still used if necessary. 2024-03-06T07:19:46.994 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:19:46.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:47.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3d7181640 1 Processor -- start 2024-03-06T07:19:47.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3d7181640 1 -- start start 2024-03-06T07:19:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3d7181640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d014f790 0x7ff3d01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3d7181640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff3d00c1aa0 0x7ff3d00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3d7181640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d00c28d0 con 0x7ff3d00c5c60 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3d7181640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d00c2680 con 0x7ff3d00c1aa0 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3d7181640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00c2bb0 con 0x7ff3d014f790 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3cffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff3d00c1aa0 0x7ff3d00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3cffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff3d00c1aa0 0x7ff3d00c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60686/0 (socket says 172.21.15.94:60686) 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.092+0000 7ff3cffff640 1 -- 172.21.15.94:0/2362422916 learned_addr learned my addr 172.21.15.94:0/2362422916 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.093+0000 7ff3cffff640 1 -- 172.21.15.94:0/2362422916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d014f790 msgr2=0x7ff3d01529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.093+0000 7ff3cffff640 1 --2- 172.21.15.94:0/2362422916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d014f790 0x7ff3d01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.093+0000 7ff3cffff640 1 -- 172.21.15.94:0/2362422916 >> v1:172.21.15.145:6789/0 conn(0x7ff3d00c5c60 legacy=0x7ff3d0155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.093+0000 7ff3cffff640 1 -- 172.21.15.94:0/2362422916 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d00c2e90 con 0x7ff3d00c1aa0 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.093+0000 7ff3cffff640 1 --2- 172.21.15.94:0/2362422916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d00c5720 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009870 tx=0x7ff3bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf071e811ce08cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3cf7fe640 1 -- 172.21.15.94:0/2362422916 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc035020 con 0x7ff3d00c1aa0 2024-03-06T07:19:47.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3cf7fe640 1 -- 172.21.15.94:0/2362422916 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc031640 con 0x7ff3d00c1aa0 2024-03-06T07:19:47.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3cf7fe640 1 -- 172.21.15.94:0/2362422916 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc02d040 con 0x7ff3d00c1aa0 2024-03-06T07:19:47.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 -- 172.21.15.94:0/2362422916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d00c1aa0 msgr2=0x7ff3d00c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2362422916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d00c5720 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009870 tx=0x7ff3bc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 -- 172.21.15.94:0/2362422916 shutdown_connections 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2362422916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d00c5720 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2362422916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d014f790 0x7ff3d01529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 -- 172.21.15.94:0/2362422916 >> 172.21.15.94:0/2362422916 conn(0x7ff3d00c6710 msgr2=0x7ff3d00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 -- 172.21.15.94:0/2362422916 shutdown_connections 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.094+0000 7ff3d7181640 1 -- 172.21.15.94:0/2362422916 wait complete. 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.095+0000 7ff3d7181640 1 Processor -- start 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.095+0000 7ff3d7181640 1 -- start start 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d7181640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d0153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d7181640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d00c5c60 0x7ff3d014d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d7181640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d014f790 0x7ff3d014d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d7181640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d006cc00 con 0x7ff3d014f790 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d7181640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00c0d60 con 0x7ff3d00c1aa0 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d7181640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d006cd80 con 0x7ff3d00c5c60 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d014f790 0x7ff3d014d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d014f790 0x7ff3d014d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60692/0 (socket says 172.21.15.94:60692) 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 -- 172.21.15.94:0/2252832104 learned_addr learned my addr 172.21.15.94:0/2252832104 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d4f1d640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d0153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3cffff640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d00c5c60 0x7ff3d014d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 -- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d00c5c60 msgr2=0x7ff3d014d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d00c5c60 0x7ff3d014d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 -- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3d00c1aa0 msgr2=0x7ff3d0153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d0153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d571e640 1 -- 172.21.15.94:0/2252832104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0154600 con 0x7ff3d014f790 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.096+0000 7ff3d4f1d640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d0153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.097+0000 7ff3d571e640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d014f790 0x7ff3d014d890 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800c780 tx=0x7ff3c800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.097+0000 7ff3cd7fa640 1 -- 172.21.15.94:0/2252832104 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8017070 con 0x7ff3d014f790 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.097+0000 7ff3cd7fa640 1 -- 172.21.15.94:0/2252832104 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c8003da0 con 0x7ff3d014f790 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.097+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d01548d0 con 0x7ff3d014f790 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.097+0000 7ff3cd7fa640 1 -- 172.21.15.94:0/2252832104 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80052b0 con 0x7ff3d014f790 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.097+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d0150cd0 con 0x7ff3d014f790 2024-03-06T07:19:47.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.098+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d004f330 con 0x7ff3d014f790 2024-03-06T07:19:47.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.098+0000 7ff3cd7fa640 1 -- 172.21.15.94:0/2252832104 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3c801a070 con 0x7ff3d014f790 2024-03-06T07:19:47.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.098+0000 7ff3cd7fa640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3b406b500 0x7ff3b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.099+0000 7ff3cd7fa640 1 -- 172.21.15.94:0/2252832104 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7ff3c80896b0 con 0x7ff3d014f790 2024-03-06T07:19:47.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.099+0000 7ff3d4f1d640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3b406b500 0x7ff3b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.099+0000 7ff3d4f1d640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3b406b500 0x7ff3b406d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400a870 tx=0x7ff3c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.103+0000 7ff3cd7fa640 1 -- 172.21.15.94:0/2252832104 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3d01548d0 con 0x7ff3d014f790 2024-03-06T07:19:47.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.274+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3d00c0b40 con 0x7ff3d014f790 2024-03-06T07:19:47.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.275+0000 7ff3cd7fa640 1 -- 172.21.15.94:0/2252832104 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+13425 (secure 0 0 0) 0x7ff3c80586e0 con 0x7ff3d014f790 2024-03-06T07:19:47.277 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:47.277 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":43,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:45.972156+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/2367914435":"2024-03-07T07:19:42.609423+0000","172.21.15.94:6833/2367914435":"2024-03-07T07:19:42.609423+0000","172.21.15.145:6837/1305657307":"2024-03-07T07:19:26.486020+0000","172.21.15.145:6836/1305657307":"2024-03-07T07:19:26.486020+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3b406b500 msgr2=0x7ff3b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3b406b500 0x7ff3b406d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400a870 tx=0x7ff3c4008040 comp rx=0 tx=0).stop 2024-03-06T07:19:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d014f790 msgr2=0x7ff3d014d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d014f790 0x7ff3d014d890 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800c780 tx=0x7ff3c800cc40 comp rx=0 tx=0).stop 2024-03-06T07:19:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 shutdown_connections 2024-03-06T07:19:47.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3b406b500 0x7ff3b406d9c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff3d014f790 0x7ff3d014d890 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3d00c5c60 0x7ff3d014d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 --2- 172.21.15.94:0/2252832104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3d00c1aa0 0x7ff3d0153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 >> 172.21.15.94:0/2252832104 conn(0x7ff3d00c6710 msgr2=0x7ff3d00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:47.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 shutdown_connections 2024-03-06T07:19:47.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.277+0000 7ff3d7181640 1 -- 172.21.15.94:0/2252832104 wait complete. 2024-03-06T07:19:47.290 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:19:47.290 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:19:47.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.390+0000 7fb02f41c640 1 Processor -- start 2024-03-06T07:19:47.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.390+0000 7fb02f41c640 1 -- start start 2024-03-06T07:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.390+0000 7fb02f41c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb0280b00d0 0x7fb0280c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.390+0000 7fb02f41c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280c7210 0x7fb0280c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.390+0000 7fb02f41c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0280b04b0 con 0x7fb02806b8e0 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02f41c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb028156160 con 0x7fb0280b00d0 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02f41c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280c5710 con 0x7fb0280c7210 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d1b8640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb0280b00d0 0x7fb0280c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d1b8640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb0280b00d0 0x7fb0280c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60696/0 (socket says 172.21.15.94:60696) 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d1b8640 1 -- 172.21.15.94:0/164814177 learned_addr learned my addr 172.21.15.94:0/164814177 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d9b9640 1 --2- 172.21.15.94:0/164814177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280c7210 0x7fb0280c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d1b8640 1 -- 172.21.15.94:0/164814177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280c7210 msgr2=0x7fb0280c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d1b8640 1 --2- 172.21.15.94:0/164814177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280c7210 0x7fb0280c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d1b8640 1 -- 172.21.15.94:0/164814177 >> v1:172.21.15.145:6789/0 conn(0x7fb02806b8e0 legacy=0x7fb02806bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d1b8640 1 -- 172.21.15.94:0/164814177 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02806d210 con 0x7fb0280b00d0 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.391+0000 7fb02d9b9640 1 --2- 172.21.15.94:0/164814177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280c7210 0x7fb0280c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:47.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02d1b8640 1 --2- 172.21.15.94:0/164814177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280b00d0 0x7fb0280c8ff0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb0280ad3f0 tx=0x7fb02402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939fb919e7e6327c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb01ffff640 1 -- 172.21.15.94:0/164814177 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024035020 con 0x7fb0280b00d0 2024-03-06T07:19:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb01ffff640 1 -- 172.21.15.94:0/164814177 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb024031830 con 0x7fb0280b00d0 2024-03-06T07:19:47.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb01ffff640 1 -- 172.21.15.94:0/164814177 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024031c00 con 0x7fb0280b00d0 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 -- 172.21.15.94:0/164814177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280b00d0 msgr2=0x7fb0280c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 --2- 172.21.15.94:0/164814177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280b00d0 0x7fb0280c8ff0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb0280ad3f0 tx=0x7fb02402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 -- 172.21.15.94:0/164814177 shutdown_connections 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 --2- 172.21.15.94:0/164814177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280c7210 0x7fb0280c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 --2- 172.21.15.94:0/164814177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280b00d0 0x7fb0280c8ff0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 -- 172.21.15.94:0/164814177 >> 172.21.15.94:0/164814177 conn(0x7fb0280ac810 msgr2=0x7fb0280acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 -- 172.21.15.94:0/164814177 shutdown_connections 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.392+0000 7fb02f41c640 1 -- 172.21.15.94:0/164814177 wait complete. 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 Processor -- start 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 -- start start 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02806b8e0 0x7fb02814fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280b00d0 0x7fb028154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280c7210 0x7fb028158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280c5ba0 con 0x7fb0280c7210 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb028152c50 con 0x7fb02806b8e0 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02f41c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280c5d20 con 0x7fb0280b00d0 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02d9b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280c7210 0x7fb028158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02d9b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280c7210 0x7fb028158c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60704/0 (socket says 172.21.15.94:60704) 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02d9b9640 1 -- 172.21.15.94:0/502260022 learned_addr learned my addr 172.21.15.94:0/502260022 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02d1b8640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02806b8e0 0x7fb02814fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.394+0000 7fb02c9b7640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280b00d0 0x7fb028154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02d9b9640 1 -- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280b00d0 msgr2=0x7fb028154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02d9b9640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280b00d0 0x7fb028154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02d9b9640 1 -- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02806b8e0 msgr2=0x7fb02814fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02d9b9640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02806b8e0 0x7fb02814fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02d9b9640 1 -- 172.21.15.94:0/502260022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb028150300 con 0x7fb0280c7210 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02c9b7640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280b00d0 0x7fb028154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02d9b9640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280c7210 0x7fb028158c10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb02000cef0 tx=0x7fb02000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb01dffb640 1 -- 172.21.15.94:0/502260022 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02001a070 con 0x7fb0280c7210 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb01dffb640 1 -- 172.21.15.94:0/502260022 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02000d860 con 0x7fb0280c7210 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb028153e40 con 0x7fb0280c7210 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb01dffb640 1 -- 172.21.15.94:0/502260022 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02000dc50 con 0x7fb0280c7210 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.395+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0281540e0 con 0x7fb0280c7210 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.396+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02804f330 con 0x7fb0280c7210 2024-03-06T07:19:47.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.396+0000 7fb01dffb640 1 -- 172.21.15.94:0/502260022 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb02001d070 con 0x7fb0280c7210 2024-03-06T07:19:47.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.397+0000 7fb01dffb640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.397+0000 7fb01dffb640 1 -- 172.21.15.94:0/502260022 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb02008a8d0 con 0x7fb0280c7210 2024-03-06T07:19:47.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.397+0000 7fb02d1b8640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.397+0000 7fb02d1b8640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb00c06b4d0 0x7fb00c06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb024036000 tx=0x7fb02403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.401+0000 7fb01dffb640 1 -- 172.21.15.94:0/502260022 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb028150300 con 0x7fb0280c7210 2024-03-06T07:19:47.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.570+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb02806ce50 con 0x7fb0280c7210 2024-03-06T07:19:47.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.570+0000 7fb01dffb640 1 -- 172.21.15.94:0/502260022 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0200592d0 con 0x7fb0280c7210 2024-03-06T07:19:47.572 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:19:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb00c06b4d0 msgr2=0x7fb00c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb00c06b4d0 0x7fb00c06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb024036000 tx=0x7fb02403c040 comp rx=0 tx=0).stop 2024-03-06T07:19:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280c7210 msgr2=0x7fb028158c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280c7210 0x7fb028158c10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb02000cef0 tx=0x7fb02000a710 comp rx=0 tx=0).stop 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 shutdown_connections 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0280c7210 0x7fb028158c10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0280b00d0 0x7fb028154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 --2- 172.21.15.94:0/502260022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02806b8e0 0x7fb02814fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 >> 172.21.15.94:0/502260022 conn(0x7fb0280ac810 msgr2=0x7fb0280cac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 shutdown_connections 2024-03-06T07:19:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.572+0000 7fb02f41c640 1 -- 172.21.15.94:0/502260022 wait complete. 2024-03-06T07:19:47.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-03-06T07:19:47.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.684+0000 7fd978b7a640 1 Processor -- start 2024-03-06T07:19:47.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.684+0000 7fd978b7a640 1 -- start start 2024-03-06T07:19:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd978b7a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd97406c720 0x7fd974071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd978b7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9740c2410 0x7fd974071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd978b7a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd974067340 con 0x7fd974150920 2024-03-06T07:19:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd978b7a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd974072400 con 0x7fd97406c720 2024-03-06T07:19:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd978b7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd974150d00 con 0x7fd9740c2410 2024-03-06T07:19:47.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd96bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd97406c720 0x7fd974071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd96bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd97406c720 0x7fd974071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60712/0 (socket says 172.21.15.94:60712) 2024-03-06T07:19:47.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd96bfff640 1 -- 172.21.15.94:0/2895139794 learned_addr learned my addr 172.21.15.94:0/2895139794 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:47.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd96bfff640 1 -- 172.21.15.94:0/2895139794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9740c2410 msgr2=0x7fd974071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:47.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd96bfff640 1 --2- 172.21.15.94:0/2895139794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9740c2410 0x7fd974071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd96bfff640 1 -- 172.21.15.94:0/2895139794 >> v1:172.21.15.145:6789/0 conn(0x7fd974150920 legacy=0x7fd9740c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:47.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.685+0000 7fd96bfff640 1 -- 172.21.15.94:0/2895139794 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740c2010 con 0x7fd97406c720 2024-03-06T07:19:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd96bfff640 1 --2- 172.21.15.94:0/2895139794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97406c720 0x7fd974071980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009870 tx=0x7fd95c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159ab009e6cc58b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd971d9b640 1 -- 172.21.15.94:0/2895139794 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0317c0 con 0x7fd97406c720 2024-03-06T07:19:47.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd971d9b640 1 -- 172.21.15.94:0/2895139794 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c037070 con 0x7fd97406c720 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd971d9b640 1 -- 172.21.15.94:0/2895139794 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c031b80 con 0x7fd97406c720 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd978b7a640 1 -- 172.21.15.94:0/2895139794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97406c720 msgr2=0x7fd974071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd978b7a640 1 --2- 172.21.15.94:0/2895139794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97406c720 0x7fd974071980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009870 tx=0x7fd95c02b860 comp rx=0 tx=0).stop 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd978b7a640 1 -- 172.21.15.94:0/2895139794 shutdown_connections 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd978b7a640 1 --2- 172.21.15.94:0/2895139794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9740c2410 0x7fd974071ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd978b7a640 1 --2- 172.21.15.94:0/2895139794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd97406c720 0x7fd974071980 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd978b7a640 1 -- 172.21.15.94:0/2895139794 >> 172.21.15.94:0/2895139794 conn(0x7fd9740c70b0 msgr2=0x7fd9740c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.686+0000 7fd978b7a640 1 -- 172.21.15.94:0/2895139794 shutdown_connections 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.687+0000 7fd978b7a640 1 -- 172.21.15.94:0/2895139794 wait complete. 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 Processor -- start 2024-03-06T07:19:47.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 -- start start 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd97406c720 0x7fd9740c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9740c2410 0x7fd974151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974150920 0x7fd97414d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9740c3040 con 0x7fd974150920 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974072b70 con 0x7fd9740c2410 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd978b7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9740c31c0 con 0x7fd97406c720 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd972d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974150920 0x7fd97414d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd972d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974150920 0x7fd97414d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60720/0 (socket says 172.21.15.94:60720) 2024-03-06T07:19:47.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.688+0000 7fd972d9d640 1 -- 172.21.15.94:0/3292894698 learned_addr learned my addr 172.21.15.94:0/3292894698 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd97259c640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd97406c720 0x7fd9740c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd972d9d640 1 -- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd97406c720 msgr2=0x7fd9740c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd972d9d640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd97406c720 0x7fd9740c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd972d9d640 1 -- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9740c2410 msgr2=0x7fd974151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd96bfff640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9740c2410 0x7fd974151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd972d9d640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9740c2410 0x7fd974151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd972d9d640 1 -- 172.21.15.94:0/3292894698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97414cbb0 con 0x7fd974150920 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd96bfff640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9740c2410 0x7fd974151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd972d9d640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974150920 0x7fd97414d630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00b970 tx=0x7fd96c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd96affd640 1 -- 172.21.15.94:0/3292894698 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c019070 con 0x7fd974150920 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd96affd640 1 -- 172.21.15.94:0/3292894698 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c00cb60 con 0x7fd974150920 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97414b0e0 con 0x7fd974150920 2024-03-06T07:19:47.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd96affd640 1 -- 172.21.15.94:0/3292894698 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c014440 con 0x7fd974150920 2024-03-06T07:19:47.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.689+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97414ce30 con 0x7fd974150920 2024-03-06T07:19:47.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.690+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97404f330 con 0x7fd974150920 2024-03-06T07:19:47.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.690+0000 7fd96affd640 1 -- 172.21.15.94:0/3292894698 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd96c01c070 con 0x7fd974150920 2024-03-06T07:19:47.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.691+0000 7fd96affd640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd95406b500 0x7fd95406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:47.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.691+0000 7fd96affd640 1 -- 172.21.15.94:0/3292894698 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd96c089d80 con 0x7fd974150920 2024-03-06T07:19:47.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.691+0000 7fd97259c640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd95406b500 0x7fd95406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:47.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.692+0000 7fd97259c640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd95406b500 0x7fd95406d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd96400a730 tx=0x7fd964008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:47.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.695+0000 7fd96affd640 1 -- 172.21.15.94:0/3292894698 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd97414cbb0 con 0x7fd974150920 2024-03-06T07:19:47.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.933+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7fd974146400 con 0x7fd974150920 2024-03-06T07:19:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.980+0000 7fd96affd640 1 -- 172.21.15.94:0/3292894698 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v104) v1 ==== 153+0+0 (secure 0 0 0) 0x7fd96c058db0 con 0x7fd974150920 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd95406b500 msgr2=0x7fd95406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd95406b500 0x7fd95406d9c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd96400a730 tx=0x7fd964008040 comp rx=0 tx=0).stop 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974150920 msgr2=0x7fd97414d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974150920 0x7fd97414d630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00b970 tx=0x7fd96c00be30 comp rx=0 tx=0).stop 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 shutdown_connections 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd95406b500 0x7fd95406d9c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd974150920 0x7fd97414d630 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9740c2410 0x7fd974151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 --2- 172.21.15.94:0/3292894698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd97406c720 0x7fd9740c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:47.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 >> 172.21.15.94:0/3292894698 conn(0x7fd9740c70b0 msgr2=0x7fd97406b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:47.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 shutdown_connections 2024-03-06T07:19:47.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:47.982+0000 7fd978b7a640 1 -- 172.21.15.94:0/3292894698 wait complete. 2024-03-06T07:19:47.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:19:48.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effad77a640 1 Processor -- start 2024-03-06T07:19:48.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effad77a640 1 -- start start 2024-03-06T07:19:48.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effad77a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7effa80b0070 0x7effa8155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effad77a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa80c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effad77a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa80b0450 con 0x7effa80c5380 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effad77a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa806c020 con 0x7effa80b0070 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effad77a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa806cf20 con 0x7effa806ca70 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.096+0000 7effa6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7effa80b0070 0x7effa8155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7effa6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7effa80b0070 0x7effa8155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60732/0 (socket says 172.21.15.94:60732) 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7effa6ffd640 1 -- 172.21.15.94:0/3491819780 learned_addr learned my addr 172.21.15.94:0/3491819780 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7eff9ffff640 1 --2- 172.21.15.94:0/3491819780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa80c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7effa6ffd640 1 -- 172.21.15.94:0/3491819780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 msgr2=0x7effa80c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7effa6ffd640 1 --2- 172.21.15.94:0/3491819780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa80c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7effa6ffd640 1 -- 172.21.15.94:0/3491819780 >> v1:172.21.15.145:6789/0 conn(0x7effa80c5380 legacy=0x7effa814e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7effa6ffd640 1 -- 172.21.15.94:0/3491819780 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa80c77f0 con 0x7effa80b0070 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.097+0000 7eff9ffff640 1 --2- 172.21.15.94:0/3491819780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa80c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:48.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effa6ffd640 1 --2- 172.21.15.94:0/3491819780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8155c30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7eff98009870 tx=0x7eff9802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f25e9f927b2194a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effa67fc640 1 -- 172.21.15.94:0/3491819780 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98035020 con 0x7effa80b0070 2024-03-06T07:19:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effa67fc640 1 -- 172.21.15.94:0/3491819780 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff98031640 con 0x7effa80b0070 2024-03-06T07:19:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effa67fc640 1 -- 172.21.15.94:0/3491819780 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9802d040 con 0x7effa80b0070 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 -- 172.21.15.94:0/3491819780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 msgr2=0x7effa8155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 --2- 172.21.15.94:0/3491819780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8155c30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7eff98009870 tx=0x7eff9802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 -- 172.21.15.94:0/3491819780 shutdown_connections 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 --2- 172.21.15.94:0/3491819780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa80c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 --2- 172.21.15.94:0/3491819780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8155c30 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 -- 172.21.15.94:0/3491819780 >> 172.21.15.94:0/3491819780 conn(0x7effa80ac590 msgr2=0x7effa80ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 -- 172.21.15.94:0/3491819780 shutdown_connections 2024-03-06T07:19:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.098+0000 7effad77a640 1 -- 172.21.15.94:0/3491819780 wait complete. 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.099+0000 7effad77a640 1 Processor -- start 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.099+0000 7effad77a640 1 -- start start 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effad77a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa8157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effad77a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effad77a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effa80c5380 0x7effa814df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effad77a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa806c550 con 0x7effa80b0070 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effad77a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa806c270 con 0x7effa80c5380 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effad77a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa806da80 con 0x7effa806ca70 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60748/0 (socket says 172.21.15.94:60748) 2024-03-06T07:19:48.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7eff9ffff640 1 -- 172.21.15.94:0/1389581003 learned_addr learned my addr 172.21.15.94:0/1389581003 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effa6ffd640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa8157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7effa77fe640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effa80c5380 0x7effa814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7eff9ffff640 1 -- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 msgr2=0x7effa8157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7eff9ffff640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa8157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7eff9ffff640 1 -- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effa80c5380 msgr2=0x7effa814df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.100+0000 7eff9ffff640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effa80c5380 0x7effa814df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.101+0000 7eff9ffff640 1 -- 172.21.15.94:0/1389581003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa81eb4c0 con 0x7effa80b0070 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.101+0000 7eff9ffff640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8151410 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff9000d800 tx=0x7eff9000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.101+0000 7eff9f7fe640 1 -- 172.21.15.94:0/1389581003 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90004030 con 0x7effa80b0070 2024-03-06T07:19:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.101+0000 7eff9f7fe640 1 -- 172.21.15.94:0/1389581003 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff900041d0 con 0x7effa80b0070 2024-03-06T07:19:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.101+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa8152fa0 con 0x7effa80b0070 2024-03-06T07:19:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.101+0000 7eff9f7fe640 1 -- 172.21.15.94:0/1389581003 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900096a0 con 0x7effa80b0070 2024-03-06T07:19:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.101+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa80bddc0 con 0x7effa80b0070 2024-03-06T07:19:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.102+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa804f330 con 0x7effa80b0070 2024-03-06T07:19:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.102+0000 7eff9f7fe640 1 -- 172.21.15.94:0/1389581003 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff90009840 con 0x7effa80b0070 2024-03-06T07:19:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.103+0000 7eff9f7fe640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7eff8806b4d0 0x7eff8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.103+0000 7eff9f7fe640 1 -- 172.21.15.94:0/1389581003 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7eff9008a320 con 0x7effa80b0070 2024-03-06T07:19:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.103+0000 7effa6ffd640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7eff8806b4d0 0x7eff8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.104+0000 7effa6ffd640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7eff8806b4d0 0x7eff8806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7eff98036000 tx=0x7eff9803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.107+0000 7eff9f7fe640 1 -- 172.21.15.94:0/1389581003 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff90059350 con 0x7effa80b0070 2024-03-06T07:19:48.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.278+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7effa806d860 con 0x7effa80b0070 2024-03-06T07:19:48.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.281+0000 7eff9f7fe640 1 -- 172.21.15.94:0/1389581003 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v15)=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7eff90058c50 con 0x7effa80b0070 2024-03-06T07:19:48.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7eff8806b4d0 msgr2=0x7eff8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7eff8806b4d0 0x7eff8806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7eff98036000 tx=0x7eff9803c040 comp rx=0 tx=0).stop 2024-03-06T07:19:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 msgr2=0x7effa8151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8151410 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff9000d800 tx=0x7eff9000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 shutdown_connections 2024-03-06T07:19:48.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7eff8806b4d0 0x7eff8806d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effa80c5380 0x7effa814df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7effa80b0070 0x7effa8151410 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 --2- 172.21.15.94:0/1389581003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effa806ca70 0x7effa8157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 >> 172.21.15.94:0/1389581003 conn(0x7effa80ac590 msgr2=0x7effa80c8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 shutdown_connections 2024-03-06T07:19:48.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.283+0000 7effad77a640 1 -- 172.21.15.94:0/1389581003 wait complete. 2024-03-06T07:19:48.296 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:19:48.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:19:48.334 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:19:48.390 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:19:48.390 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:19:48.390 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:19:48.406 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:19:48.406 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:19:48.406 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:19:48.423 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:19:48.423 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:19:48.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:48.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.543+0000 7f5ebfec2640 1 Processor -- start 2024-03-06T07:19:48.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.543+0000 7f5ebfec2640 1 -- start start 2024-03-06T07:19:48.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebfec2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5eb814f360 0x7f5eb8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebfec2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb80c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebfec2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80c73d0 con 0x7f5eb80c5880 2024-03-06T07:19:48.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebfec2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb80c8dd0 con 0x7f5eb814f360 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebfec2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80be2e0 con 0x7f5eb80c88c0 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebdc5e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5eb814f360 0x7f5eb8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebdc5e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5eb814f360 0x7f5eb8155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60228/0 (socket says 172.21.15.94:60228) 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebdc5e640 1 -- 172.21.15.94:0/1898782139 learned_addr learned my addr 172.21.15.94:0/1898782139 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.544+0000 7f5ebd45d640 1 --2- 172.21.15.94:0/1898782139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb80c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.545+0000 7f5ebdc5e640 1 -- 172.21.15.94:0/1898782139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 msgr2=0x7f5eb80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.545+0000 7f5ebdc5e640 1 --2- 172.21.15.94:0/1898782139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb80c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.545+0000 7f5ebdc5e640 1 -- 172.21.15.94:0/1898782139 >> v1:172.21.15.145:6789/0 conn(0x7f5eb80c5880 legacy=0x7f5eb8158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.545+0000 7f5ebdc5e640 1 -- 172.21.15.94:0/1898782139 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80c7670 con 0x7f5eb814f360 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.545+0000 7f5ebdc5e640 1 --2- 172.21.15.94:0/1898782139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb8155c10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009870 tx=0x7f5eb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fcdff0dfb124319 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.545+0000 7f5ebcc5c640 1 -- 172.21.15.94:0/1898782139 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4035020 con 0x7f5eb814f360 2024-03-06T07:19:48.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebcc5c640 1 -- 172.21.15.94:0/1898782139 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4031640 con 0x7f5eb814f360 2024-03-06T07:19:48.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebcc5c640 1 -- 172.21.15.94:0/1898782139 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb402d040 con 0x7f5eb814f360 2024-03-06T07:19:48.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 -- 172.21.15.94:0/1898782139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 msgr2=0x7f5eb8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/1898782139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb8155c10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009870 tx=0x7f5eb402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 -- 172.21.15.94:0/1898782139 shutdown_connections 2024-03-06T07:19:48.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/1898782139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/1898782139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb8155c10 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 -- 172.21.15.94:0/1898782139 >> 172.21.15.94:0/1898782139 conn(0x7f5eb80ac570 msgr2=0x7f5eb80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 -- 172.21.15.94:0/1898782139 shutdown_connections 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.546+0000 7f5ebfec2640 1 -- 172.21.15.94:0/1898782139 wait complete. 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.547+0000 7f5ebfec2640 1 Processor -- start 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.547+0000 7f5ebfec2640 1 -- start start 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebfec2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5eb80c5880 0x7f5eb81eb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebfec2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb8157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebfec2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb80bfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebfec2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8159270 con 0x7f5eb814f360 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebfec2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80c7c30 con 0x7f5eb80c5880 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebfec2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb81593f0 con 0x7f5eb80c88c0 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb80bfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb80bfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60236/0 (socket says 172.21.15.94:60236) 2024-03-06T07:19:48.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 -- 172.21.15.94:0/823855115 learned_addr learned my addr 172.21.15.94:0/823855115 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebdc5e640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5eb80c5880 0x7f5eb81eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebd45d640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb8157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 -- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 msgr2=0x7f5eb8157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb8157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 -- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5eb80c5880 msgr2=0x7f5eb81eb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5eb80c5880 0x7f5eb81eb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 -- 172.21.15.94:0/823855115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80c0820 con 0x7f5eb814f360 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebdc5e640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5eb80c5880 0x7f5eb81eb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5ebe45f640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb80bfc00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000c7d0 tx=0x7f5eb000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.548+0000 7f5eae7fc640 1 -- 172.21.15.94:0/823855115 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0003b70 con 0x7f5eb814f360 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.549+0000 7f5eae7fc640 1 -- 172.21.15.94:0/823855115 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb0003d30 con 0x7f5eb814f360 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.549+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb80c0ae0 con 0x7f5eb814f360 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.549+0000 7f5eae7fc640 1 -- 172.21.15.94:0/823855115 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000aa90 con 0x7f5eb814f360 2024-03-06T07:19:48.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.549+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb80bf6e0 con 0x7f5eb814f360 2024-03-06T07:19:48.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.549+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb80bf8e0 con 0x7f5eb814f360 2024-03-06T07:19:48.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.550+0000 7f5eae7fc640 1 -- 172.21.15.94:0/823855115 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5eb000ac30 con 0x7f5eb814f360 2024-03-06T07:19:48.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.550+0000 7f5eae7fc640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e9c06b4d0 0x7f5e9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.550+0000 7f5eae7fc640 1 -- 172.21.15.94:0/823855115 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f5eb0089f80 con 0x7f5eb814f360 2024-03-06T07:19:48.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.551+0000 7f5ebdc5e640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e9c06b4d0 0x7f5e9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.551+0000 7f5ebdc5e640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e9c06b4d0 0x7f5e9c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4036000 tx=0x7f5eb4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.555+0000 7f5eae7fc640 1 -- 172.21.15.94:0/823855115 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5eb80bf8e0 con 0x7f5eb814f360 2024-03-06T07:19:48.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.760+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eb804f330 con 0x7f5eb814f360 2024-03-06T07:19:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.761+0000 7f5eae7fc640 1 -- 172.21.15.94:0/823855115 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+3697 (secure 0 0 0) 0x7f5eb0058fb0 con 0x7f5eb814f360 2024-03-06T07:19:48.763 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:48.763 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5215,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1902381441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1902381441},{"type":"v1","addr":"172.21.15.145:6835","nonce":1902381441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5410,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/856474008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":856474008},{"type":"v1","addr":"172.21.15.145:6837","nonce":856474008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5574,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/536970635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":536970635},{"type":"v1","addr":"172.21.15.94:6835","nonce":536970635}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":34814,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/597543014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":597543014},{"type":"v1","addr":"172.21.15.94:6834","nonce":597543014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-06T07:19:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 25 2024-03-06T07:19:48.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e9c06b4d0 msgr2=0x7f5e9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e9c06b4d0 0x7f5e9c06d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4036000 tx=0x7f5eb4008030 comp rx=0 tx=0).stop 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 msgr2=0x7f5eb80bfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb80bfc00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000c7d0 tx=0x7f5eb000cc90 comp rx=0 tx=0).stop 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 shutdown_connections 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5e9c06b4d0 0x7f5e9c06d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5eb814f360 0x7f5eb80bfc00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5eb80c88c0 0x7f5eb8157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 --2- 172.21.15.94:0/823855115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5eb80c5880 0x7f5eb81eb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 >> 172.21.15.94:0/823855115 conn(0x7f5eb80ac570 msgr2=0x7f5eb80ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 shutdown_connections 2024-03-06T07:19:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.763+0000 7f5ebfec2640 1 -- 172.21.15.94:0/823855115 wait complete. 2024-03-06T07:19:48.776 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:19:48.776 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:19:48.776 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:19:48.776 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:48.776 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:19:48.776 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:19:48.777 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:19:48.777 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:48.777 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:19:48.777 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:19:48.777 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:19:48.777 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:48.777 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:19:48.778 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:19:48.778 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:19:48.778 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:19:48.785 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:19:48.785 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:19:48.786 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:19:48.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:19:48.788 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:19:48.789 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:19:48.789 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:19:48.790 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:19:48.828 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:19:48.829 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:48.829 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:19:48.829 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:48.832 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:19:48.832 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:48.832 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:19:48.832 DEBUG:teuthology.parallel:result is None 2024-03-06T07:19:48.833 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:19:48.905 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:19:48.906 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:19:48.914 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:19:48.915 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:19:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.936+0000 7f570cf5e640 1 Processor -- start 2024-03-06T07:19:48.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.936+0000 7f570cf5e640 1 -- start start 2024-03-06T07:19:48.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f570cf5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c69f0 0x7f57080c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f570cf5e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f57080c8950 0x7f57080acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f570cf5e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5708155130 con 0x7f57080ad2b0 2024-03-06T07:19:48.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f570cf5e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5708155440 con 0x7f57080c8950 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f570cf5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57080c5d50 con 0x7f57080c69f0 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f5705d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f57080c8950 0x7f57080acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f5706d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f57080ad2b0 0x7f57081558d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47792/0 (socket says 172.21.15.94:47792) 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f5706d9d640 1 -- 172.21.15.94:0/914903664 learned_addr learned my addr 172.21.15.94:0/914903664 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f5705d9b640 1 -- 172.21.15.94:0/914903664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c69f0 msgr2=0x7f57080c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f570659c640 1 --2- 172.21.15.94:0/914903664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c69f0 0x7f57080c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f5705d9b640 1 --2- 172.21.15.94:0/914903664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c69f0 0x7f57080c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f570559a640 1 -- 172.21.15.94:0/914903664 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659663565 0 0) 0x7f5700001490 con 0x7f57080ad2b0 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f5705d9b640 1 -- 172.21.15.94:0/914903664 >> v1:172.21.15.94:6789/0 conn(0x7f57080ad2b0 legacy=0x7f57081558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.937+0000 7f5705d9b640 1 -- 172.21.15.94:0/914903664 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57080c6030 con 0x7f57080c8950 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.938+0000 7f5705d9b640 1 --2- 172.21.15.94:0/914903664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c8950 0x7f57080acd70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0097a0 tx=0x7f56fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8569e829c2a6a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.938+0000 7f570559a640 1 -- 172.21.15.94:0/914903664 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0315a0 con 0x7f57080c8950 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.938+0000 7f570559a640 1 -- 172.21.15.94:0/914903664 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc037070 con 0x7f57080c8950 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.938+0000 7f570559a640 1 -- 172.21.15.94:0/914903664 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc031960 con 0x7f57080c8950 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 -- 172.21.15.94:0/914903664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c8950 msgr2=0x7f57080acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 --2- 172.21.15.94:0/914903664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c8950 0x7f57080acd70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0097a0 tx=0x7f56fc02b840 comp rx=0 tx=0).stop 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 -- 172.21.15.94:0/914903664 shutdown_connections 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 --2- 172.21.15.94:0/914903664 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c8950 0x7f57080acd70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 --2- 172.21.15.94:0/914903664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c69f0 0x7f57080c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 -- 172.21.15.94:0/914903664 >> 172.21.15.94:0/914903664 conn(0x7f57080cae70 msgr2=0x7f57080cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 -- 172.21.15.94:0/914903664 shutdown_connections 2024-03-06T07:19:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.939+0000 7f570cf5e640 1 -- 172.21.15.94:0/914903664 wait complete. 2024-03-06T07:19:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.940+0000 7f570cf5e640 1 Processor -- start 2024-03-06T07:19:48.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.940+0000 7f570cf5e640 1 -- start start 2024-03-06T07:19:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57080ad2b0 0x7f570806c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c69f0 0x7f57080ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c8950 0x7f57080adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5708152a20 con 0x7f57080c69f0 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5708155fe0 con 0x7f57080ad2b0 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5708152ba0 con 0x7f57080c8950 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c69f0 0x7f57080ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c69f0 0x7f57080ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60290/0 (socket says 172.21.15.94:60290) 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 -- 172.21.15.94:0/2225028887 learned_addr learned my addr 172.21.15.94:0/2225028887 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5706d9d640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c8950 0x7f57080adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570659c640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57080ad2b0 0x7f570806c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 -- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c8950 msgr2=0x7f57080adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c8950 0x7f57080adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 -- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57080ad2b0 msgr2=0x7f570806c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:48.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57080ad2b0 0x7f570806c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 -- 172.21.15.94:0/2225028887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57081585a0 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5706d9d640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c8950 0x7f57080adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f5705d9b640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c69f0 0x7f57080ad770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0098c0 tx=0x7f56fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f56f6ffd640 1 -- 172.21.15.94:0/2225028887 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc032840 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f56f6ffd640 1 -- 172.21.15.94:0/2225028887 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc032a00 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57080afb30 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f56f6ffd640 1 -- 172.21.15.94:0/2225028887 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc032df0 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.941+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57080b00f0 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.942+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570804f330 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.943+0000 7f56f6ffd640 1 -- 172.21.15.94:0/2225028887 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56fc04e430 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.943+0000 7f56f6ffd640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f56e806b500 0x7f56e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.943+0000 7f56f6ffd640 1 -- 172.21.15.94:0/2225028887 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f56fc02f030 con 0x7f57080c69f0 2024-03-06T07:19:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.943+0000 7f570659c640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f56e806b500 0x7f56e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:48.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.944+0000 7f570659c640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f56e806b500 0x7f56e806d9c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f56f8006fd0 tx=0x7f56f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:48.947+0000 7f56f6ffd640 1 -- 172.21.15.94:0/2225028887 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56fc040070 con 0x7f57080c69f0 2024-03-06T07:19:49.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:49.135+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f57080ca650 con 0x7f57080c69f0 2024-03-06T07:19:50.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.010+0000 7f56f6ffd640 1 -- 172.21.15.94:0/2225028887 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7f56fc07b080 con 0x7f57080c69f0 2024-03-06T07:19:50.012 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:19:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f56e806b500 msgr2=0x7f56e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f56e806b500 0x7f56e806d9c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f56f8006fd0 tx=0x7f56f8008040 comp rx=0 tx=0).stop 2024-03-06T07:19:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c69f0 msgr2=0x7f57080ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c69f0 0x7f57080ad770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0098c0 tx=0x7f56fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 shutdown_connections 2024-03-06T07:19:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f56e806b500 0x7f56e806d9c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57080c8950 0x7f57080adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57080c69f0 0x7f57080ad770 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 --2- 172.21.15.94:0/2225028887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57080ad2b0 0x7f570806c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 >> 172.21.15.94:0/2225028887 conn(0x7f57080cae70 msgr2=0x7f57081558d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 shutdown_connections 2024-03-06T07:19:50.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.013+0000 7f570cf5e640 1 -- 172.21.15.94:0/2225028887 wait complete. 2024-03-06T07:19:50.028 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:19:50.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.130+0000 7f82f4ad4640 1 Processor -- start 2024-03-06T07:19:50.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.130+0000 7f82f4ad4640 1 -- start start 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82f4ad4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f82f0152960 0x7f82f006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82f4ad4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f82f00ac260 0x7f82f01559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82f4ad4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f0152d40 con 0x7f82f01592d0 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82f4ad4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f0066e30 con 0x7f82f00ac260 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82f4ad4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f00c1dc0 con 0x7f82f0152960 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82eed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f82f00ac260 0x7f82f01559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82e7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f82f01592d0 0x7f82f01552d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47802/0 (socket says 172.21.15.94:47802) 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82e7fff640 1 -- 172.21.15.94:0/909916966 learned_addr learned my addr 172.21.15.94:0/909916966 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82eed9d640 1 --2- 172.21.15.94:0/909916966 >> v2:172.21.15.94:3300/0 conn(0x7f82f00ac260 0x7f82f01559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60296/0 (socket says 172.21.15.94:60296) 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82eed9d640 1 -- 172.21.15.94:0/909916966 >> v2:172.21.15.145:3300/0 conn(0x7f82f0152960 msgr2=0x7f82f006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82ee59c640 1 --2- 172.21.15.94:0/909916966 >> v2:172.21.15.145:3300/0 conn(0x7f82f0152960 0x7f82f006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82eed9d640 1 --2- 172.21.15.94:0/909916966 >> v2:172.21.15.145:3300/0 conn(0x7f82f0152960 0x7f82f006dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82eed9d640 1 -- 172.21.15.94:0/909916966 >> v1:172.21.15.94:6789/0 conn(0x7f82f01592d0 legacy=0x7f82f01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82edd9b640 1 -- 172.21.15.94:0/909916966 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563859614 0 0) 0x7f82f0152d40 con 0x7f82f01592d0 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.131+0000 7f82eed9d640 1 -- 172.21.15.94:0/909916966 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00c1b50 con 0x7f82f00ac260 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.132+0000 7f82eed9d640 1 --2- 172.21.15.94:0/909916966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f01559e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82f006b1c0 tx=0x7f82e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d18a61ea65f189 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.132+0000 7f82edd9b640 1 -- 172.21.15.94:0/909916966 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8039070 con 0x7f82f00ac260 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.132+0000 7f82edd9b640 1 -- 172.21.15.94:0/909916966 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e802cc80 con 0x7f82f00ac260 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.132+0000 7f82edd9b640 1 -- 172.21.15.94:0/909916966 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8034660 con 0x7f82f00ac260 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.132+0000 7f82f4ad4640 1 -- 172.21.15.94:0/909916966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 msgr2=0x7f82f01559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.133+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/909916966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f01559e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82f006b1c0 tx=0x7f82e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.133+0000 7f82f4ad4640 1 -- 172.21.15.94:0/909916966 shutdown_connections 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.133+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/909916966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f01559e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.133+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/909916966 >> v2:172.21.15.145:3300/0 conn(0x7f82f0152960 0x7f82f006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.133+0000 7f82f4ad4640 1 -- 172.21.15.94:0/909916966 >> 172.21.15.94:0/909916966 conn(0x7f82f00c70b0 msgr2=0x7f82f00c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:50.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.133+0000 7f82f4ad4640 1 -- 172.21.15.94:0/909916966 shutdown_connections 2024-03-06T07:19:50.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.133+0000 7f82f4ad4640 1 -- 172.21.15.94:0/909916966 wait complete. 2024-03-06T07:19:50.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 Processor -- start 2024-03-06T07:19:50.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 -- start start 2024-03-06T07:19:50.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f0154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f0152960 0x7f82f0157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f0072a30 0x7f82f014e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00c20a0 con 0x7f82f00ac260 2024-03-06T07:19:50.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0159760 con 0x7f82f0152960 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.134+0000 7f82f4ad4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00c2220 con 0x7f82f0072a30 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f0154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f0154400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60302/0 (socket says 172.21.15.94:60302) 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 -- 172.21.15.94:0/3193588400 learned_addr learned my addr 172.21.15.94:0/3193588400 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 -- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f0072a30 msgr2=0x7f82f014e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f0072a30 0x7f82f014e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 -- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f0152960 msgr2=0x7f82f0157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f0152960 0x7f82f0157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 -- 172.21.15.94:0/3193588400 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0156ba0 con 0x7f82f00ac260 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82ee59c640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f0154400 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c7f0 tx=0x7f82e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82e6ffd640 1 -- 172.21.15.94:0/3193588400 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0015070 con 0x7f82f00ac260 2024-03-06T07:19:50.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82e6ffd640 1 -- 172.21.15.94:0/3193588400 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e0004240 con 0x7f82f00ac260 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82e6ffd640 1 -- 172.21.15.94:0/3193588400 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0007620 con 0x7f82f00ac260 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f01ee9d0 con 0x7f82f00ac260 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.135+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f01568c0 con 0x7f82f00ac260 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.136+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f004f330 con 0x7f82f00ac260 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.137+0000 7f82e6ffd640 1 -- 172.21.15.94:0/3193588400 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82e00079d0 con 0x7f82f00ac260 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.137+0000 7f82e6ffd640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82d006b4d0 0x7f82d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.137+0000 7f82e6ffd640 1 -- 172.21.15.94:0/3193588400 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f82e008a060 con 0x7f82f00ac260 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.137+0000 7f82e7fff640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82d006b4d0 0x7f82d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.138+0000 7f82e7fff640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82d006b4d0 0x7f82d006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007fd0 tx=0x7f82d8007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.141+0000 7f82e6ffd640 1 -- 172.21.15.94:0/3193588400 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82e0059130 con 0x7f82f00ac260 2024-03-06T07:19:50.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.379+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f82f0156270 con 0x7f82f00ac260 2024-03-06T07:19:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.380+0000 7f82e6ffd640 1 -- 172.21.15.94:0/3193588400 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f82e0058a30 con 0x7f82f00ac260 2024-03-06T07:19:50.382 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:50.382 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:19:50.382 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:19:50.382 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.382 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:19:50.382 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:50.383 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.384 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:19:50.385 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.386 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:19:50.387 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:19:50.389 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.390 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.391 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.392 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:19:50.393 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:19:50.394 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:19:50.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82d006b4d0 msgr2=0x7f82d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82d006b4d0 0x7f82d006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f82d8007fd0 tx=0x7f82d8007920 comp rx=0 tx=0).stop 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 msgr2=0x7f82f0154400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f0154400 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c7f0 tx=0x7f82e000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 shutdown_connections 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f82d006b4d0 0x7f82d006d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82f0072a30 0x7f82f014e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82f0152960 0x7f82f0157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 --2- 172.21.15.94:0/3193588400 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82f00ac260 0x7f82f0154400 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 >> 172.21.15.94:0/3193588400 conn(0x7f82f00c70b0 msgr2=0x7f82f00c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 shutdown_connections 2024-03-06T07:19:50.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.382+0000 7f82f4ad4640 1 -- 172.21.15.94:0/3193588400 wait complete. 2024-03-06T07:19:50.397 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:50.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.495+0000 7f5dcf90c640 1 Processor -- start 2024-03-06T07:19:50.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcf90c640 1 -- start start 2024-03-06T07:19:50.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcf90c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5dc81527b0 0x7f5dc8159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcf90c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5dc80c88b0 0x7f5dc80c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcf90c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc80c7c60 con 0x7f5dc80c7180 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcf90c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc8159640 con 0x7f5dc80c88b0 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcf90c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc80c8060 con 0x7f5dc81527b0 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcdea9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5dc80c88b0 0x7f5dc80c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcdea9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5dc80c88b0 0x7f5dc80c5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60306/0 (socket says 172.21.15.94:60306) 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.496+0000 7f5dcdea9640 1 -- 172.21.15.94:0/472750127 learned_addr learned my addr 172.21.15.94:0/472750127 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.497+0000 7f5dcd6a8640 1 --2- 172.21.15.94:0/472750127 >> v2:172.21.15.145:3300/0 conn(0x7f5dc81527b0 0x7f5dc8159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.497+0000 7f5dcdea9640 1 -- 172.21.15.94:0/472750127 >> v2:172.21.15.145:3300/0 conn(0x7f5dc81527b0 msgr2=0x7f5dc8159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.497+0000 7f5dcdea9640 1 --2- 172.21.15.94:0/472750127 >> v2:172.21.15.145:3300/0 conn(0x7f5dc81527b0 0x7f5dc8159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.497+0000 7f5dcdea9640 1 -- 172.21.15.94:0/472750127 >> v1:172.21.15.145:6789/0 conn(0x7f5dc80c7180 legacy=0x7f5dc8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.497+0000 7f5dcdea9640 1 -- 172.21.15.94:0/472750127 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc80c76a0 con 0x7f5dc80c88b0 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.497+0000 7f5dcd6a8640 1 --2- 172.21.15.94:0/472750127 >> v2:172.21.15.145:3300/0 conn(0x7f5dc81527b0 0x7f5dc8159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:50.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.497+0000 7f5dcdea9640 1 --2- 172.21.15.94:0/472750127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc80c88b0 0x7f5dc80c5330 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80cba80 tx=0x7f5dc002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b6c8c50f939257 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dbbfff640 1 -- 172.21.15.94:0/472750127 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc002c9f0 con 0x7f5dc80c88b0 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dbbfff640 1 -- 172.21.15.94:0/472750127 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc002cb90 con 0x7f5dc80c88b0 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dbbfff640 1 -- 172.21.15.94:0/472750127 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc00343f0 con 0x7f5dc80c88b0 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 -- 172.21.15.94:0/472750127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc80c88b0 msgr2=0x7f5dc80c5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/472750127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc80c88b0 0x7f5dc80c5330 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80cba80 tx=0x7f5dc002be50 comp rx=0 tx=0).stop 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 -- 172.21.15.94:0/472750127 shutdown_connections 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/472750127 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc80c88b0 0x7f5dc80c5330 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/472750127 >> v2:172.21.15.145:3300/0 conn(0x7f5dc81527b0 0x7f5dc8159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 -- 172.21.15.94:0/472750127 >> 172.21.15.94:0/472750127 conn(0x7f5dc80ac560 msgr2=0x7f5dc80ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 -- 172.21.15.94:0/472750127 shutdown_connections 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.498+0000 7f5dcf90c640 1 -- 172.21.15.94:0/472750127 wait complete. 2024-03-06T07:19:50.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.499+0000 7f5dcf90c640 1 Processor -- start 2024-03-06T07:19:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.499+0000 7f5dcf90c640 1 -- start start 2024-03-06T07:19:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcf90c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dc80c7180 0x7f5dc81542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcf90c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc80c88b0 0x7f5dc814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcf90c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc81527b0 0x7f5dc814bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcf90c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8155600 con 0x7f5dc81527b0 2024-03-06T07:19:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcf90c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8155320 con 0x7f5dc80c88b0 2024-03-06T07:19:50.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcf90c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8152b90 con 0x7f5dc80c7180 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc81527b0 0x7f5dc814bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc81527b0 0x7f5dc814bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60320/0 (socket says 172.21.15.94:60320) 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 -- 172.21.15.94:0/1638018735 learned_addr learned my addr 172.21.15.94:0/1638018735 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dccea7640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc80c88b0 0x7f5dc814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcd6a8640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dc80c7180 0x7f5dc81542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 -- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dc80c7180 msgr2=0x7f5dc81542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dc80c7180 0x7f5dc81542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 -- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc80c88b0 msgr2=0x7f5dc814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc80c88b0 0x7f5dc814b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dcdea9640 1 -- 172.21.15.94:0/1638018735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc814c200 con 0x7f5dc81527b0 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.500+0000 7f5dccea7640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc80c88b0 0x7f5dc814b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.501+0000 7f5dcdea9640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc81527b0 0x7f5dc814bcc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0036040 tx=0x7f5dc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.501+0000 7f5db9ffb640 1 -- 172.21.15.94:0/1638018735 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0043020 con 0x7f5dc81527b0 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.501+0000 7f5db9ffb640 1 -- 172.21.15.94:0/1638018735 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc003e450 con 0x7f5dc81527b0 2024-03-06T07:19:50.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.501+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc80c8ea0 con 0x7f5dc81527b0 2024-03-06T07:19:50.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.501+0000 7f5db9ffb640 1 -- 172.21.15.94:0/1638018735 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0009510 con 0x7f5dc81527b0 2024-03-06T07:19:50.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.501+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc80ca130 con 0x7f5dc81527b0 2024-03-06T07:19:50.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.502+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc80c9c40 con 0x7f5dc81527b0 2024-03-06T07:19:50.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.502+0000 7f5db9ffb640 1 -- 172.21.15.94:0/1638018735 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5dc0046020 con 0x7f5dc81527b0 2024-03-06T07:19:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.503+0000 7f5db9ffb640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dac06b500 0x7f5dac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.503+0000 7f5db9ffb640 1 -- 172.21.15.94:0/1638018735 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5dc00ac7e0 con 0x7f5dc81527b0 2024-03-06T07:19:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.503+0000 7f5dcd6a8640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dac06b500 0x7f5dac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.504+0000 7f5dcd6a8640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dac06b500 0x7f5dac06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4006fd0 tx=0x7f5dc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.507+0000 7f5db9ffb640 1 -- 172.21.15.94:0/1638018735 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dc004d030 con 0x7f5dc81527b0 2024-03-06T07:19:50.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.680+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dc806ce00 con 0x7f5dc81527b0 2024-03-06T07:19:50.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.681+0000 7f5db9ffb640 1 -- 172.21.15.94:0/1638018735 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13164 (secure 0 0 0) 0x7f5dc80c7c60 con 0x7f5dc81527b0 2024-03-06T07:19:50.683 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:50.683 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":48,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:50.009889+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:50.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dac06b500 msgr2=0x7f5dac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dac06b500 0x7f5dac06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4006fd0 tx=0x7f5dc4008040 comp rx=0 tx=0).stop 2024-03-06T07:19:50.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc81527b0 msgr2=0x7f5dc814bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc81527b0 0x7f5dc814bcc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0036040 tx=0x7f5dc0004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:50.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 shutdown_connections 2024-03-06T07:19:50.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dac06b500 0x7f5dac06d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dc81527b0 0x7f5dc814bcc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc80c88b0 0x7f5dc814b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 --2- 172.21.15.94:0/1638018735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dc80c7180 0x7f5dc81542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 >> 172.21.15.94:0/1638018735 conn(0x7f5dc80ac560 msgr2=0x7f5dc814e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:50.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 shutdown_connections 2024-03-06T07:19:50.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.683+0000 7f5dcf90c640 1 -- 172.21.15.94:0/1638018735 wait complete. 2024-03-06T07:19:50.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:19:50.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:19:50.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.798+0000 7fc0da5ac640 1 Processor -- start 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.798+0000 7fc0da5ac640 1 -- start start 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0da5ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 0x7fc0d40ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0da5ac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d4154330 con 0x7fc0d41538c0 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0da5ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d4153dc0 con 0x7fc0d40a97c0 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0da5ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40c46a0 con 0x7fc0d40ab310 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d37fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc0d41538c0 0x7fc0d41494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47804/0 (socket says 172.21.15.94:47804) 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d37fe640 1 -- 172.21.15.94:0/1758192530 learned_addr learned my addr 172.21.15.94:0/1758192530 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 0x7fc0d40ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876805972 0 0) 0x7fc0d4154330 con 0x7fc0d41538c0 2024-03-06T07:19:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc003520 con 0x7fc0d41538c0 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 262157400 0 0) 0x7fc0bc003520 con 0x7fc0d41538c0 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 msgr2=0x7fc0d40ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 0x7fc0d40ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 >> v1:172.21.15.145:6789/0 conn(0x7fc0d40a97c0 legacy=0x7fc0d40c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d40c5d30 con 0x7fc0d41538c0 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.799+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fc0c8001670 con 0x7fc0d41538c0 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 >> v1:172.21.15.94:6789/0 conn(0x7fc0d41538c0 legacy=0x7fc0d41494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d2ffd640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d2ffd640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0bc004800 0x7fc0bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d2ffd640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 0x7fc0bc005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40c46a0 con 0x7fc0bc0039e0 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc00bb80 con 0x7fc0bc004800 2024-03-06T07:19:50.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc00bd80 con 0x7fc0d40ab310 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d37fe640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0bc004800 0x7fc0bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d8b49640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 0x7fc0bc005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d3fff640 1 -- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 msgr2=0x7fc0bc005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 0x7fc0bc005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d3fff640 1 -- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0bc004800 msgr2=0x7fc0bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0bc004800 0x7fc0bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d3fff640 1 -- 172.21.15.94:0/1758192530 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc003520 con 0x7fc0bc0039e0 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.800+0000 7fc0d8b49640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d40ab310 0x7fc0bc005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0bc003e20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009ac0 tx=0x7fc0bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316a8a174d76a748 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4039070 con 0x7fc0bc0039e0 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c402c6b0 con 0x7fc0bc0039e0 2024-03-06T07:19:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0d2ffd640 1 -- 172.21.15.94:0/1758192530 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4009690 con 0x7fc0bc0039e0 2024-03-06T07:19:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1758192530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 msgr2=0x7fc0bc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0da5ac640 1 --2- 172.21.15.94:0/1758192530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0bc003e20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009ac0 tx=0x7fc0bc005830 comp rx=0 tx=0).stop 2024-03-06T07:19:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0d8b49640 1 -- 172.21.15.94:0/1758192530 reap_dead start 2024-03-06T07:19:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1758192530 shutdown_connections 2024-03-06T07:19:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1758192530 >> 172.21.15.94:0/1758192530 conn(0x7fc0d40c6710 msgr2=0x7fc0d40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1758192530 shutdown_connections 2024-03-06T07:19:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.801+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1758192530 wait complete. 2024-03-06T07:19:50.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.802+0000 7fc0da5ac640 1 Processor -- start 2024-03-06T07:19:50.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0da5ac640 1 -- start start 2024-03-06T07:19:50.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0da5ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0d40a7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0da5ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d41eb7e0 0x7fc0d40c2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0da5ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0d4148b80 0x7fc0d4149030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0da5ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d4149bb0 con 0x7fc0bc0039e0 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0da5ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40c62f0 con 0x7fc0d4148b80 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0da5ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d4149d30 con 0x7fc0d41eb7e0 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0d3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0d40a7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0d3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0d40a7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60334/0 (socket says 172.21.15.94:60334) 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.803+0000 7fc0d3fff640 1 -- 172.21.15.94:0/1669605818 learned_addr learned my addr 172.21.15.94:0/1669605818 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d8b49640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0d4148b80 0x7fc0d4149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d37fe640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d41eb7e0 0x7fc0d40c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d3fff640 1 -- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d41eb7e0 msgr2=0x7fc0d40c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d41eb7e0 0x7fc0d40c2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d3fff640 1 -- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0d4148b80 msgr2=0x7fc0d4149030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0d4148b80 0x7fc0d4149030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d3fff640 1 -- 172.21.15.94:0/1669605818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d40a86b0 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d8b49640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0d4148b80 0x7fc0d4149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d3fff640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0d40a7fb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4037040 tx=0x7fc0c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d0ff9640 1 -- 172.21.15.94:0/1669605818 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4042020 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d0ff9640 1 -- 172.21.15.94:0/1669605818 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c403a040 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d41ead00 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0d0ff9640 1 -- 172.21.15.94:0/1669605818 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4039070 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.804+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d40c2940 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.805+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d4069460 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.805+0000 7fc0d0ff9640 1 -- 172.21.15.94:0/1669605818 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0c4045020 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.806+0000 7fc0d0ff9640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c006b500 0x7fc0c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.806+0000 7fc0d0ff9640 1 -- 172.21.15.94:0/1669605818 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc0c40ab770 con 0x7fc0bc0039e0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.806+0000 7fc0d37fe640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c006b500 0x7fc0c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.807+0000 7fc0d37fe640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c006b500 0x7fc0c006d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0d41ebc20 tx=0x7fc0c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.810+0000 7fc0d0ff9640 1 -- 172.21.15.94:0/1669605818 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0d4069460 con 0x7fc0bc0039e0 2024-03-06T07:19:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.980+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0d41eaaf0 con 0x7fc0bc0039e0 2024-03-06T07:19:50.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.980+0000 7fc0d0ff9640 1 -- 172.21.15.94:0/1669605818 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0c404c030 con 0x7fc0bc0039e0 2024-03-06T07:19:50.982 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:19:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c006b500 msgr2=0x7fc0c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c006b500 0x7fc0c006d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc0d41ebc20 tx=0x7fc0c8009040 comp rx=0 tx=0).stop 2024-03-06T07:19:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 msgr2=0x7fc0d40a7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0d40a7fb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4037040 tx=0x7fc0c4004730 comp rx=0 tx=0).stop 2024-03-06T07:19:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 shutdown_connections 2024-03-06T07:19:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c006b500 0x7fc0c006d9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0d4148b80 0x7fc0d4149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0d41eb7e0 0x7fc0d40c2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 --2- 172.21.15.94:0/1669605818 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0bc0039e0 0x7fc0d40a7fb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:50.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 >> 172.21.15.94:0/1669605818 conn(0x7fc0d40c6710 msgr2=0x7fc0d4151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:50.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 shutdown_connections 2024-03-06T07:19:50.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:50.982+0000 7fc0da5ac640 1 -- 172.21.15.94:0/1669605818 wait complete. 2024-03-06T07:19:50.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:19:51.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.094+0000 7fba7594b640 1 Processor -- start 2024-03-06T07:19:51.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.094+0000 7fba7594b640 1 -- start start 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba7594b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fba70158db0 0x7fba700c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba7594b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c88b0 0x7fba7015fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba7594b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba70155110 con 0x7fba700c5330 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba7594b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba701559e0 con 0x7fba70158db0 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba7594b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba700c8f40 con 0x7fba700c88b0 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba67fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fba70158db0 0x7fba700c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba6effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fba700c5330 0x7fba701586a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47814/0 (socket says 172.21.15.94:47814) 2024-03-06T07:19:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba6effd640 1 -- 172.21.15.94:0/1742695093 learned_addr learned my addr 172.21.15.94:0/1742695093 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba67fff640 1 -- 172.21.15.94:0/1742695093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c88b0 msgr2=0x7fba7015fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba6f7fe640 1 --2- 172.21.15.94:0/1742695093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c88b0 0x7fba7015fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba6e7fc640 1 -- 172.21.15.94:0/1742695093 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356544636 0 0) 0x7fba60001110 con 0x7fba700c5330 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba67fff640 1 --2- 172.21.15.94:0/1742695093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c88b0 0x7fba7015fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba67fff640 1 -- 172.21.15.94:0/1742695093 >> v1:172.21.15.94:6789/0 conn(0x7fba700c5330 legacy=0x7fba701586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba67fff640 1 -- 172.21.15.94:0/1742695093 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70155390 con 0x7fba70158db0 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.095+0000 7fba6f7fe640 1 --2- 172.21.15.94:0/1742695093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c88b0 0x7fba7015fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.096+0000 7fba67fff640 1 --2- 172.21.15.94:0/1742695093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba70158db0 0x7fba700c5710 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fba580097a0 tx=0x7fba5802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de25a6477afc1b86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.096+0000 7fba6e7fc640 1 -- 172.21.15.94:0/1742695093 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba580315a0 con 0x7fba70158db0 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.096+0000 7fba6e7fc640 1 -- 172.21.15.94:0/1742695093 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58037070 con 0x7fba70158db0 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.096+0000 7fba6e7fc640 1 -- 172.21.15.94:0/1742695093 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58031960 con 0x7fba70158db0 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 -- 172.21.15.94:0/1742695093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba70158db0 msgr2=0x7fba700c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 --2- 172.21.15.94:0/1742695093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba70158db0 0x7fba700c5710 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fba580097a0 tx=0x7fba5802b840 comp rx=0 tx=0).stop 2024-03-06T07:19:51.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 -- 172.21.15.94:0/1742695093 shutdown_connections 2024-03-06T07:19:51.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 --2- 172.21.15.94:0/1742695093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c88b0 0x7fba7015fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 --2- 172.21.15.94:0/1742695093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba70158db0 0x7fba700c5710 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 -- 172.21.15.94:0/1742695093 >> 172.21.15.94:0/1742695093 conn(0x7fba700ac560 msgr2=0x7fba700ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:51.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 -- 172.21.15.94:0/1742695093 shutdown_connections 2024-03-06T07:19:51.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.097+0000 7fba7594b640 1 -- 172.21.15.94:0/1742695093 wait complete. 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.098+0000 7fba7594b640 1 Processor -- start 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.098+0000 7fba7594b640 1 -- start start 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba7594b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c5330 0x7fba7014c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba7594b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba700c88b0 0x7fba70151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba7594b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba700c11f0 0x7fba700c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba7594b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba701529d0 con 0x7fba700c88b0 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba7594b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7006d090 con 0x7fba700c11f0 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba7594b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba70152b50 con 0x7fba700c5330 2024-03-06T07:19:51.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba700c88b0 0x7fba70151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba700c88b0 0x7fba70151640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60350/0 (socket says 172.21.15.94:60350) 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 -- 172.21.15.94:0/3078461833 learned_addr learned my addr 172.21.15.94:0/3078461833 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba6effd640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c5330 0x7fba7014c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba6f7fe640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba700c11f0 0x7fba700c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 -- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c5330 msgr2=0x7fba7014c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c5330 0x7fba7014c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 -- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba700c11f0 msgr2=0x7fba700c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba700c11f0 0x7fba700c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 -- 172.21.15.94:0/3078461833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7014bfb0 con 0x7fba700c88b0 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba6f7fe640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba700c11f0 0x7fba700c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba67fff640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba700c88b0 0x7fba70151640 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba58034040 tx=0x7fba580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba6cff9640 1 -- 172.21.15.94:0/3078461833 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58032840 con 0x7fba700c88b0 2024-03-06T07:19:51.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba6cff9640 1 -- 172.21.15.94:0/3078461833 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58032a00 con 0x7fba700c88b0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba6cff9640 1 -- 172.21.15.94:0/3078461833 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58032df0 con 0x7fba700c88b0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.099+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba700c2610 con 0x7fba700c88b0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.100+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba70151b80 con 0x7fba700c88b0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.100+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba70067910 con 0x7fba700c88b0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.101+0000 7fba6cff9640 1 -- 172.21.15.94:0/3078461833 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba5804e430 con 0x7fba700c88b0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.101+0000 7fba6cff9640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba5006b4d0 0x7fba5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.101+0000 7fba6cff9640 1 -- 172.21.15.94:0/3078461833 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fba5802f030 con 0x7fba700c88b0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.101+0000 7fba6effd640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba5006b4d0 0x7fba5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.102+0000 7fba6effd640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba5006b4d0 0x7fba5006d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba60000f60 tx=0x7fba60007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:51.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.105+0000 7fba6cff9640 1 -- 172.21.15.94:0/3078461833 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba70067910 con 0x7fba700c88b0 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.277+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fba7004f330 con 0x7fba700c88b0 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.277+0000 7fba6cff9640 1 -- 172.21.15.94:0/3078461833 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) v1 ==== 84+0+65 (secure 0 0 0) 0x7fba58040070 con 0x7fba700c88b0 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:19:51.279 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:19:51.280 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:19:51.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba5006b4d0 msgr2=0x7fba5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba5006b4d0 0x7fba5006d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba60000f60 tx=0x7fba60007970 comp rx=0 tx=0).stop 2024-03-06T07:19:51.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba700c88b0 msgr2=0x7fba70151640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba700c88b0 0x7fba70151640 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba58034040 tx=0x7fba580046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 shutdown_connections 2024-03-06T07:19:51.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba5006b4d0 0x7fba5006d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba700c11f0 0x7fba700c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba700c88b0 0x7fba70151640 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 --2- 172.21.15.94:0/3078461833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba700c5330 0x7fba7014c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 >> 172.21.15.94:0/3078461833 conn(0x7fba700ac560 msgr2=0x7fba7014f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 shutdown_connections 2024-03-06T07:19:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.279+0000 7fba7594b640 1 -- 172.21.15.94:0/3078461833 wait complete. 2024-03-06T07:19:51.292 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:19:51.292 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:19:51.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.394+0000 7f2d70f3a640 1 Processor -- start 2024-03-06T07:19:51.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.394+0000 7f2d70f3a640 1 -- start start 2024-03-06T07:19:51.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d70f3a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 0x7f2d6c151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d70f3a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2d6c153390 0x7f2d6c153820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d70f3a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6c06e850 con 0x7f2d6c0a9830 2024-03-06T07:19:51.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d70f3a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6c06d2b0 con 0x7f2d6c153390 2024-03-06T07:19:51.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d70f3a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c06d590 con 0x7f2d6c0c22b0 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d6ad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2d6c153390 0x7f2d6c153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d69d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2d6c0a9830 0x7f2d6c14ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47830/0 (socket says 172.21.15.94:47830) 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d69d9b640 1 -- 172.21.15.94:0/1945824470 learned_addr learned my addr 172.21.15.94:0/1945824470 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d6a59c640 1 --2- 172.21.15.94:0/1945824470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 0x7f2d6c151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d6ad9d640 1 -- 172.21.15.94:0/1945824470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 msgr2=0x7f2d6c151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d6ad9d640 1 --2- 172.21.15.94:0/1945824470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 0x7f2d6c151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d6ad9d640 1 -- 172.21.15.94:0/1945824470 >> v1:172.21.15.94:6789/0 conn(0x7f2d6c0a9830 legacy=0x7f2d6c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d6959a640 1 -- 172.21.15.94:0/1945824470 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333343768 0 0) 0x7f2d60001110 con 0x7f2d6c0a9830 2024-03-06T07:19:51.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.395+0000 7f2d6ad9d640 1 -- 172.21.15.94:0/1945824470 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c06d870 con 0x7f2d6c153390 2024-03-06T07:19:51.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.396+0000 7f2d6ad9d640 1 --2- 172.21.15.94:0/1945824470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c153820 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009d90 tx=0x7f2d6402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6b68d6a1b575ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:51.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.396+0000 7f2d6959a640 1 -- 172.21.15.94:0/1945824470 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64039070 con 0x7f2d6c153390 2024-03-06T07:19:51.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.396+0000 7f2d6959a640 1 -- 172.21.15.94:0/1945824470 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6402cc50 con 0x7f2d6c153390 2024-03-06T07:19:51.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.396+0000 7f2d6959a640 1 -- 172.21.15.94:0/1945824470 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64034440 con 0x7f2d6c153390 2024-03-06T07:19:51.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 -- 172.21.15.94:0/1945824470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 msgr2=0x7f2d6c153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/1945824470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c153820 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009d90 tx=0x7f2d6402bea0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 -- 172.21.15.94:0/1945824470 shutdown_connections 2024-03-06T07:19:51.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/1945824470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c153820 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/1945824470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 0x7f2d6c151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 -- 172.21.15.94:0/1945824470 >> 172.21.15.94:0/1945824470 conn(0x7f2d6c0c65b0 msgr2=0x7f2d6c0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:51.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 -- 172.21.15.94:0/1945824470 shutdown_connections 2024-03-06T07:19:51.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.397+0000 7f2d70f3a640 1 -- 172.21.15.94:0/1945824470 wait complete. 2024-03-06T07:19:51.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.398+0000 7f2d70f3a640 1 Processor -- start 2024-03-06T07:19:51.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.398+0000 7f2d70f3a640 1 -- start start 2024-03-06T07:19:51.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d70f3a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d6c0a9830 0x7f2d6c0aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d70f3a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 0x7f2d6c1ebd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d70f3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c1ec260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d70f3a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c14f7e0 con 0x7f2d6c153390 2024-03-06T07:19:51.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d70f3a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c14f660 con 0x7f2d6c0a9830 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d70f3a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0c2690 con 0x7f2d6c0c22b0 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c1ec260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c1ec260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60372/0 (socket says 172.21.15.94:60372) 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 -- 172.21.15.94:0/2240310723 learned_addr learned my addr 172.21.15.94:0/2240310723 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 -- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 msgr2=0x7f2d6c1ebd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 0x7f2d6c1ebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 -- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d6c0a9830 msgr2=0x7f2d6c0aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6a59c640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d6c0a9830 0x7f2d6c0aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d6c0a9830 0x7f2d6c0aaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 -- 172.21.15.94:0/2240310723 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c1e92d0 con 0x7f2d6c153390 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6a59c640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d6c0a9830 0x7f2d6c0aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.399+0000 7f2d6ad9d640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c1ec260 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009eb0 tx=0x7f2d64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.400+0000 7f2d5affd640 1 -- 172.21.15.94:0/2240310723 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64039050 con 0x7f2d6c153390 2024-03-06T07:19:51.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.400+0000 7f2d5affd640 1 -- 172.21.15.94:0/2240310723 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d64045070 con 0x7f2d6c153390 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.400+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6c1dd610 con 0x7f2d6c153390 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.400+0000 7f2d5affd640 1 -- 172.21.15.94:0/2240310723 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6403e610 con 0x7f2d6c153390 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.400+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6c1dc730 con 0x7f2d6c153390 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.401+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6c04f330 con 0x7f2d6c153390 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.401+0000 7f2d5affd640 1 -- 172.21.15.94:0/2240310723 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d6403e7b0 con 0x7f2d6c153390 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.401+0000 7f2d5affd640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2d4c06b500 0x7f2d4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.402+0000 7f2d5affd640 1 -- 172.21.15.94:0/2240310723 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2d640acb30 con 0x7f2d6c153390 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.402+0000 7f2d6a59c640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2d4c06b500 0x7f2d4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:51.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.402+0000 7f2d6a59c640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2d4c06b500 0x7f2d4c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0a8ce0 tx=0x7f2d5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:51.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.406+0000 7f2d5affd640 1 -- 172.21.15.94:0/2240310723 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d64032e20 con 0x7f2d6c153390 2024-03-06T07:19:51.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:51.578+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2d6c1dd9f0 con 0x7f2d6c153390 2024-03-06T07:19:52.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.022+0000 7f2d5affd640 1 -- 172.21.15.94:0/2240310723 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v49) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2d6407bc00 con 0x7f2d6c153390 2024-03-06T07:19:52.024 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.024+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2d4c06b500 msgr2=0x7f2d4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.024+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2d4c06b500 0x7f2d4c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0a8ce0 tx=0x7f2d5c008040 comp rx=0 tx=0).stop 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.024+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 msgr2=0x7f2d6c1ec260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.024+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c1ec260 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009eb0 tx=0x7f2d64004a40 comp rx=0 tx=0).stop 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 shutdown_connections 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2d4c06b500 0x7f2d4c06d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d6c153390 0x7f2d6c1ec260 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d6c0c22b0 0x7f2d6c1ebd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 --2- 172.21.15.94:0/2240310723 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d6c0a9830 0x7f2d6c0aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 >> 172.21.15.94:0/2240310723 conn(0x7f2d6c0c65b0 msgr2=0x7f2d6c150ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:52.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 shutdown_connections 2024-03-06T07:19:52.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.025+0000 7f2d70f3a640 1 -- 172.21.15.94:0/2240310723 wait complete. 2024-03-06T07:19:52.043 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:19:52.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.149+0000 7f359f4df640 1 Processor -- start 2024-03-06T07:19:52.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.150+0000 7f359f4df640 1 -- start start 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.150+0000 7f359f4df640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35980a8290 0x7f35981539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.150+0000 7f359f4df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c69a0 0x7f3598072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.150+0000 7f359f4df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35980c62a0 con 0x7f35980c4740 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.150+0000 7f359f4df640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3598153f00 con 0x7f35980a8290 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.150+0000 7f359f4df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35980c4c00 con 0x7f35980c69a0 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359da7c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c69a0 0x7f3598072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359d27b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35980a8290 0x7f35981539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:52.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359da7c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c69a0 0x7f3598072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:45364/0 (socket says 172.21.15.94:45364) 2024-03-06T07:19:52.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359d27b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35980a8290 0x7f35981539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:48984/0 (socket says 172.21.15.94:48984) 2024-03-06T07:19:52.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359da7c640 1 -- 172.21.15.94:0/932712243 learned_addr learned my addr 172.21.15.94:0/932712243 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:52.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359da7c640 1 -- 172.21.15.94:0/932712243 >> v2:172.21.15.145:3300/0 conn(0x7f35980a8290 msgr2=0x7f35981539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:52.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359da7c640 1 --2- 172.21.15.94:0/932712243 >> v2:172.21.15.145:3300/0 conn(0x7f35980a8290 0x7f35981539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359da7c640 1 -- 172.21.15.94:0/932712243 >> v1:172.21.15.145:6789/0 conn(0x7f35980c4740 legacy=0x7f35980701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:52.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.151+0000 7f359da7c640 1 -- 172.21.15.94:0/932712243 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35980c4ee0 con 0x7f35980c69a0 2024-03-06T07:19:52.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.152+0000 7f359da7c640 1 --2- 172.21.15.94:0/932712243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c69a0 0x7f3598072680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3590009870 tx=0x7f359002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45ab04e326f0e0ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.152+0000 7f358ffff640 1 -- 172.21.15.94:0/932712243 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35900317c0 con 0x7f35980c69a0 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.152+0000 7f358ffff640 1 -- 172.21.15.94:0/932712243 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3590037070 con 0x7f35980c69a0 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.152+0000 7f358ffff640 1 -- 172.21.15.94:0/932712243 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590031b80 con 0x7f35980c69a0 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 -- 172.21.15.94:0/932712243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c69a0 msgr2=0x7f3598072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 --2- 172.21.15.94:0/932712243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c69a0 0x7f3598072680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3590009870 tx=0x7f359002b860 comp rx=0 tx=0).stop 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 -- 172.21.15.94:0/932712243 shutdown_connections 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 --2- 172.21.15.94:0/932712243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c69a0 0x7f3598072680 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 --2- 172.21.15.94:0/932712243 >> v2:172.21.15.145:3300/0 conn(0x7f35980a8290 0x7f35981539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 -- 172.21.15.94:0/932712243 >> 172.21.15.94:0/932712243 conn(0x7f35980c5560 msgr2=0x7f35980c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 -- 172.21.15.94:0/932712243 shutdown_connections 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.153+0000 7f359f4df640 1 -- 172.21.15.94:0/932712243 wait complete. 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 Processor -- start 2024-03-06T07:19:52.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 -- start start 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35980a8290 0x7f35980ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c4740 0x7f35981ecc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35980c69a0 0x7f35981523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35980c6e30 con 0x7f35980a8290 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359806fc80 con 0x7f35980c69a0 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.154+0000 7f359f4df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3598072bc0 con 0x7f35980c4740 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35980a8290 0x7f35980ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:52.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35980a8290 0x7f35980ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60384/0 (socket says 172.21.15.94:60384) 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 -- 172.21.15.94:0/3479892366 learned_addr learned my addr 172.21.15.94:0/3479892366 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359da7c640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35980c69a0 0x7f35981523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359ca7a640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c4740 0x7f35981ecc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 -- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c4740 msgr2=0x7f35981ecc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c4740 0x7f35981ecc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 -- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35980c69a0 msgr2=0x7f35981523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35980c69a0 0x7f35981523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 -- 172.21.15.94:0/3479892366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35980abc80 con 0x7f35980a8290 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359d27b640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35980a8290 0x7f35980ad860 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f359400efc0 tx=0x7f359400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f358dffb640 1 -- 172.21.15.94:0/3479892366 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594018020 con 0x7f35980a8290 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f358dffb640 1 -- 172.21.15.94:0/3479892366 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594014480 con 0x7f35980a8290 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35980abf60 con 0x7f35980a8290 2024-03-06T07:19:52.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f358dffb640 1 -- 172.21.15.94:0/3479892366 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594014850 con 0x7f35980a8290 2024-03-06T07:19:52.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.155+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35981e8e20 con 0x7f35980a8290 2024-03-06T07:19:52.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.156+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35980a7c70 con 0x7f35980a8290 2024-03-06T07:19:52.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.157+0000 7f358dffb640 1 -- 172.21.15.94:0/3479892366 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3594005d10 con 0x7f35980a8290 2024-03-06T07:19:52.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.157+0000 7f358dffb640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f357c06b4d0 0x7f357c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:52.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.157+0000 7f358dffb640 1 -- 172.21.15.94:0/3479892366 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f359408ae80 con 0x7f35980a8290 2024-03-06T07:19:52.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.157+0000 7f359ca7a640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f357c06b4d0 0x7f357c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:52.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.158+0000 7f359ca7a640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f357c06b4d0 0x7f357c06d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f359814dfa0 tx=0x7f3588009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:52.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.162+0000 7f358dffb640 1 -- 172.21.15.94:0/3479892366 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3594022030 con 0x7f35980a8290 2024-03-06T07:19:52.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:52.333+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f359804f330 con 0x7f35980a8290 2024-03-06T07:19:53.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.027+0000 7f358dffb640 1 -- 172.21.15.94:0/3479892366 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v50) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3594010040 con 0x7f35980a8290 2024-03-06T07:19:53.029 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:19:53.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f357c06b4d0 msgr2=0x7f357c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f357c06b4d0 0x7f357c06d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f359814dfa0 tx=0x7f3588009040 comp rx=0 tx=0).stop 2024-03-06T07:19:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35980a8290 msgr2=0x7f35980ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35980a8290 0x7f35980ad860 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f359400efc0 tx=0x7f359400c490 comp rx=0 tx=0).stop 2024-03-06T07:19:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 shutdown_connections 2024-03-06T07:19:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f357c06b4d0 0x7f357c06d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35980c69a0 0x7f35981523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35980c4740 0x7f35981ecc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 --2- 172.21.15.94:0/3479892366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35980a8290 0x7f35980ad860 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.030+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 >> 172.21.15.94:0/3479892366 conn(0x7f35980c5560 msgr2=0x7f35980c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:53.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.031+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 shutdown_connections 2024-03-06T07:19:53.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.031+0000 7f359f4df640 1 -- 172.21.15.94:0/3479892366 wait complete. 2024-03-06T07:19:53.061 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:19:53.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.164+0000 7f7742aeb640 1 Processor -- start 2024-03-06T07:19:53.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.164+0000 7f7742aeb640 1 -- start start 2024-03-06T07:19:53.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.165+0000 7f7742aeb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f773c0c7f60 0x7f773c156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:53.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.165+0000 7f7742aeb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0c7290 con 0x7f773c1505f0 2024-03-06T07:19:53.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.165+0000 7f7742aeb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c073580 con 0x7f773c0c6710 2024-03-06T07:19:53.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.165+0000 7f7742aeb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773c070b20 con 0x7f773c0c7f60 2024-03-06T07:19:53.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.165+0000 7f773bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f773c0c7f60 0x7f773c156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.165+0000 7f7740887640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f773c1505f0 0x7f773c072510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47836/0 (socket says 172.21.15.94:47836) 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.165+0000 7f7740887640 1 -- 172.21.15.94:0/2247113540 learned_addr learned my addr 172.21.15.94:0/2247113540 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.166+0000 7f773bfff640 1 -- 172.21.15.94:0/2247113540 >> v1:172.21.15.145:6789/0 conn(0x7f773c0c6710 legacy=0x7f773c0c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.166+0000 7f773bfff640 1 -- 172.21.15.94:0/2247113540 >> v1:172.21.15.94:6789/0 conn(0x7f773c1505f0 legacy=0x7f773c072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.166+0000 7f773bfff640 1 -- 172.21.15.94:0/2247113540 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0701f0 con 0x7f773c0c7f60 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.166+0000 7f773b7fe640 1 -- 172.21.15.94:0/2247113540 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644878386 0 0) 0x7f773c0c7290 con 0x7f773c1505f0 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.166+0000 7f773bfff640 1 --2- 172.21.15.94:0/2247113540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c7f60 0x7f773c156ef0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f77300097a0 tx=0x7f773002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dfd5ccdd1d12f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f773b7fe640 1 -- 172.21.15.94:0/2247113540 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7730035020 con 0x7f773c0c7f60 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f773b7fe640 1 -- 172.21.15.94:0/2247113540 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7730031830 con 0x7f773c0c7f60 2024-03-06T07:19:53.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f773b7fe640 1 -- 172.21.15.94:0/2247113540 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7730031c00 con 0x7f773c0c7f60 2024-03-06T07:19:53.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f7742aeb640 1 -- 172.21.15.94:0/2247113540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c7f60 msgr2=0x7f773c156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:53.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f7742aeb640 1 --2- 172.21.15.94:0/2247113540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c7f60 0x7f773c156ef0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f77300097a0 tx=0x7f773002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f7742aeb640 1 -- 172.21.15.94:0/2247113540 shutdown_connections 2024-03-06T07:19:53.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f7742aeb640 1 --2- 172.21.15.94:0/2247113540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c7f60 0x7f773c156ef0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f7742aeb640 1 -- 172.21.15.94:0/2247113540 >> 172.21.15.94:0/2247113540 conn(0x7f773c0aa5c0 msgr2=0x7f773c0aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:53.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f7742aeb640 1 -- 172.21.15.94:0/2247113540 shutdown_connections 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.167+0000 7f7742aeb640 1 -- 172.21.15.94:0/2247113540 wait complete. 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.168+0000 7f7742aeb640 1 Processor -- start 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.168+0000 7f7742aeb640 1 -- start start 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7742aeb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c6710 0x7f773c1e9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7742aeb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f773c0c7f60 0x7f773c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7742aeb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f773c1505f0 0x7f773c1df7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7742aeb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c06fec0 con 0x7f773c0c6710 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7742aeb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0707b0 con 0x7f773c0c7f60 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7742aeb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c072c20 con 0x7f773c1505f0 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c6710 0x7f773c1e9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c6710 0x7f773c1e9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60408/0 (socket says 172.21.15.94:60408) 2024-03-06T07:19:53.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 -- 172.21.15.94:0/4057264569 learned_addr learned my addr 172.21.15.94:0/4057264569 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f773bfff640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f773c0c7f60 0x7f773c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7741088640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f773c1505f0 0x7f773c1df7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 -- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f773c1505f0 msgr2=0x7f773c1df7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f773c1505f0 0x7f773c1df7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 -- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f773c0c7f60 msgr2=0x7f773c152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f773c0c7f60 0x7f773c152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7740887640 1 -- 172.21.15.94:0/4057264569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0aa170 con 0x7f773c0c6710 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.169+0000 7f7741088640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f773c1505f0 0x7f773c1df7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.170+0000 7f7740887640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c6710 0x7f773c1e9ea0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c430 tx=0x7f772c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.170+0000 7f77397fa640 1 -- 172.21.15.94:0/4057264569 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c0074e0 con 0x7f773c0c6710 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.170+0000 7f77397fa640 1 -- 172.21.15.94:0/4057264569 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c0047e0 con 0x7f773c0c6710 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.170+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c1527e0 con 0x7f773c0c6710 2024-03-06T07:19:53.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.170+0000 7f77397fa640 1 -- 172.21.15.94:0/4057264569 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c004bd0 con 0x7f773c0c6710 2024-03-06T07:19:53.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.170+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c154620 con 0x7f773c0c6710 2024-03-06T07:19:53.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.171+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c075300 con 0x7f773c0c6710 2024-03-06T07:19:53.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.171+0000 7f77397fa640 1 -- 172.21.15.94:0/4057264569 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f772c011070 con 0x7f773c0c6710 2024-03-06T07:19:53.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.172+0000 7f77397fa640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f772806b500 0x7f772806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:53.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.172+0000 7f77397fa640 1 -- 172.21.15.94:0/4057264569 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f772c089f10 con 0x7f773c0c6710 2024-03-06T07:19:53.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.172+0000 7f773bfff640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f772806b500 0x7f772806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:53.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.172+0000 7f773bfff640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f772806b500 0x7f772806d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7730036000 tx=0x7f7730008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:53.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.176+0000 7f77397fa640 1 -- 172.21.15.94:0/4057264569 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f773c075300 con 0x7f773c0c6710 2024-03-06T07:19:53.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:53.384+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f773c0ab4c0 con 0x7f773c0c6710 2024-03-06T07:19:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.028+0000 7f77397fa640 1 -- 172.21.15.94:0/4057264569 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v27) v1 ==== 172+0+0 (secure 0 0 0) 0x7f772c058cf0 con 0x7f773c0c6710 2024-03-06T07:19:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 7 and data pool 8 2024-03-06T07:19:54.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f772806b500 msgr2=0x7f772806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f772806b500 0x7f772806d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7730036000 tx=0x7f7730008030 comp rx=0 tx=0).stop 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c6710 msgr2=0x7f773c1e9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c6710 0x7f773c1e9ea0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c430 tx=0x7f772c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 shutdown_connections 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f772806b500 0x7f772806d9c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f773c1505f0 0x7f773c1df7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f773c0c7f60 0x7f773c152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 --2- 172.21.15.94:0/4057264569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f773c0c6710 0x7f773c1e9ea0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 >> 172.21.15.94:0/4057264569 conn(0x7f773c0aa5c0 msgr2=0x7f773c0c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.031+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 shutdown_connections 2024-03-06T07:19:54.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.032+0000 7f7742aeb640 1 -- 172.21.15.94:0/4057264569 wait complete. 2024-03-06T07:19:54.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:54.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f39156640 1 Processor -- start 2024-03-06T07:19:54.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f39156640 1 -- start start 2024-03-06T07:19:54.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f39156640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2f3414f790 0x7f2f341529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f39156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f34155a40 0x7f2f34155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f39156640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f341563d0 con 0x7f2f340c1aa0 2024-03-06T07:19:54.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f39156640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f340c2680 con 0x7f2f3414f790 2024-03-06T07:19:54.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f39156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c28d0 con 0x7f2f34155a40 2024-03-06T07:19:54.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f2bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2f340c1aa0 0x7f2f340c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47848/0 (socket says 172.21.15.94:47848) 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.165+0000 7f2f2bfff640 1 -- 172.21.15.94:0/2306295051 learned_addr learned my addr 172.21.15.94:0/2306295051 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891237019 0 0) 0x7f2f341563d0 con 0x7f2f340c1aa0 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f20003520 con 0x7f2f340c1aa0 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f32d9d640 1 --2- 172.21.15.94:0/2306295051 >> v2:172.21.15.145:3300/0 conn(0x7f2f3414f790 0x7f2f341529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3359e640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f34155a40 0x7f2f34155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230935198 0 0) 0x7f2f20003520 con 0x7f2f340c1aa0 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f34155a40 msgr2=0x7f2f34155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f34155a40 0x7f2f34155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 >> v2:172.21.15.145:3300/0 conn(0x7f2f3414f790 msgr2=0x7f2f341529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 --2- 172.21.15.94:0/2306295051 >> v2:172.21.15.145:3300/0 conn(0x7f2f3414f790 0x7f2f341529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340c2bb0 con 0x7f2f340c1aa0 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f2f1c001670 con 0x7f2f340c1aa0 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 >> v1:172.21.15.94:6789/0 conn(0x7f2f340c1aa0 legacy=0x7f2f340c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f200039b0 0x7f2f20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f20004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20004fa0 0x7f2f2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c2680 con 0x7f2f200043a0 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c28d0 con 0x7f2f20004fa0 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2000bf30 con 0x7f2f200039b0 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3359e640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f34155a40 0x7f2f34155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f2bfff640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f20004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f32d9d640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f200039b0 0x7f2f20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f3359e640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20004fa0 0x7f2f2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f2bfff640 1 -- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f200039b0 msgr2=0x7f2f20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f2bfff640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f200039b0 0x7f2f20003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f2bfff640 1 -- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20004fa0 msgr2=0x7f2f2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:54.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f2bfff640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20004fa0 0x7f2f2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.166+0000 7f2f2bfff640 1 -- 172.21.15.94:0/2306295051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20003520 con 0x7f2f200043a0 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.167+0000 7f2f3359e640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f20004fa0 0x7f2f2000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.167+0000 7f2f2bfff640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f20004880 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c007fd0 tx=0x7f2f20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313c76b6ac3a6a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.167+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c007d50 con 0x7f2f200043a0 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.167+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c030070 con 0x7f2f200043a0 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.167+0000 7f2f3259c640 1 -- 172.21.15.94:0/2306295051 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c02cdc0 con 0x7f2f200043a0 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.168+0000 7f2f39156640 1 -- 172.21.15.94:0/2306295051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 msgr2=0x7f2f20004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.168+0000 7f2f39156640 1 --2- 172.21.15.94:0/2306295051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f20004880 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c007fd0 tx=0x7f2f20005830 comp rx=0 tx=0).stop 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.168+0000 7f2f3359e640 1 -- 172.21.15.94:0/2306295051 reap_dead start 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.168+0000 7f2f39156640 1 -- 172.21.15.94:0/2306295051 shutdown_connections 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.168+0000 7f2f39156640 1 -- 172.21.15.94:0/2306295051 >> 172.21.15.94:0/2306295051 conn(0x7f2f340c6710 msgr2=0x7f2f340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.168+0000 7f2f39156640 1 -- 172.21.15.94:0/2306295051 shutdown_connections 2024-03-06T07:19:54.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.168+0000 7f2f39156640 1 -- 172.21.15.94:0/2306295051 wait complete. 2024-03-06T07:19:54.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.169+0000 7f2f39156640 1 Processor -- start 2024-03-06T07:19:54.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.169+0000 7f2f39156640 1 -- start start 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f39156640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f3414b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f39156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f341ed4f0 0x7f2f340c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f39156640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f34148f30 0x7f2f341ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f39156640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c0d60 con 0x7f2f200043a0 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f39156640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f34155520 con 0x7f2f34148f30 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f39156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c5e30 con 0x7f2f341ed4f0 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f3414b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f3414b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60426/0 (socket says 172.21.15.94:60426) 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 -- 172.21.15.94:0/3240963174 learned_addr learned my addr 172.21.15.94:0/3240963174 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 -- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f341ed4f0 msgr2=0x7f2f340c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f341ed4f0 0x7f2f340c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 -- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f34148f30 msgr2=0x7f2f341ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f34148f30 0x7f2f341ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 -- 172.21.15.94:0/3240963174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f34149460 con 0x7f2f200043a0 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f32d9d640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f3414b650 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2f240119c0 tx=0x7f2f24011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f2b7fe640 1 -- 172.21.15.94:0/3240963174 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400fa10 con 0x7f2f200043a0 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.170+0000 7f2f2b7fe640 1 -- 172.21.15.94:0/3240963174 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2400fbb0 con 0x7f2f200043a0 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.171+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f341eca10 con 0x7f2f200043a0 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.171+0000 7f2f2b7fe640 1 -- 172.21.15.94:0/3240963174 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f24014440 con 0x7f2f200043a0 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.171+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f340c4dc0 con 0x7f2f200043a0 2024-03-06T07:19:54.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.171+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f34154dd0 con 0x7f2f200043a0 2024-03-06T07:19:54.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.172+0000 7f2f2b7fe640 1 -- 172.21.15.94:0/3240963174 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f24012040 con 0x7f2f200043a0 2024-03-06T07:19:54.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.172+0000 7f2f2b7fe640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f1406b500 0x7f2f1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.172+0000 7f2f2b7fe640 1 -- 172.21.15.94:0/3240963174 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2f2408a920 con 0x7f2f200043a0 2024-03-06T07:19:54.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.172+0000 7f2f2bfff640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f1406b500 0x7f2f1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.173+0000 7f2f2bfff640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f1406b500 0x7f2f1406d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c034000 tx=0x7f2f1c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.177+0000 7f2f2b7fe640 1 -- 172.21.15.94:0/3240963174 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f24019030 con 0x7f2f200043a0 2024-03-06T07:19:54.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.347+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f3404f330 con 0x7f2f200043a0 2024-03-06T07:19:54.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.348+0000 7f2f2b7fe640 1 -- 172.21.15.94:0/3240963174 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16955 (secure 0 0 0) 0x7f2f24059610 con 0x7f2f200043a0 2024-03-06T07:19:54.350 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:54.350 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":51,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:53.385537+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:54.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f1406b500 msgr2=0x7f2f1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f1406b500 0x7f2f1406d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c034000 tx=0x7f2f1c02e000 comp rx=0 tx=0).stop 2024-03-06T07:19:54.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 msgr2=0x7f2f3414b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f3414b650 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2f240119c0 tx=0x7f2f24011e80 comp rx=0 tx=0).stop 2024-03-06T07:19:54.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 shutdown_connections 2024-03-06T07:19:54.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f1406b500 0x7f2f1406d9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f34148f30 0x7f2f341ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f341ed4f0 0x7f2f340c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 --2- 172.21.15.94:0/3240963174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f200043a0 0x7f2f3414b650 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 >> 172.21.15.94:0/3240963174 conn(0x7f2f340c6710 msgr2=0x7f2f340c30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:54.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 shutdown_connections 2024-03-06T07:19:54.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.350+0000 7f2f39156640 1 -- 172.21.15.94:0/3240963174 wait complete. 2024-03-06T07:19:54.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:54.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.461+0000 7f5c1a55c640 1 Processor -- start 2024-03-06T07:19:54.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.461+0000 7f5c1a55c640 1 -- start start 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c1a55c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 0x7f5c141529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c1a55c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c14155a40 0x7f5c14155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c1a55c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c14156410 con 0x7f5c140c1aa0 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c1a55c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c140c2680 con 0x7f5c14155a40 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c1a55c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140c28d0 con 0x7f5c1414f790 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c18af9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c14155a40 0x7f5c14155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c137fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5c140c1aa0 0x7f5c140c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47854/0 (socket says 172.21.15.94:47854) 2024-03-06T07:19:54.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c137fe640 1 -- 172.21.15.94:0/1857491104 learned_addr learned my addr 172.21.15.94:0/1857491104 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c13fff640 1 --2- 172.21.15.94:0/1857491104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 0x7f5c141529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c18af9640 1 -- 172.21.15.94:0/1857491104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 msgr2=0x7f5c141529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c18af9640 1 --2- 172.21.15.94:0/1857491104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 0x7f5c141529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c18af9640 1 -- 172.21.15.94:0/1857491104 >> v1:172.21.15.94:6789/0 conn(0x7f5c140c1aa0 legacy=0x7f5c140c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c12ffd640 1 -- 172.21.15.94:0/1857491104 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913228104 0 0) 0x7f5c14156410 con 0x7f5c140c1aa0 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.462+0000 7f5c18af9640 1 -- 172.21.15.94:0/1857491104 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c140c2bb0 con 0x7f5c14155a40 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.463+0000 7f5c18af9640 1 --2- 172.21.15.94:0/1857491104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c14155a40 0x7f5c14155ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009d90 tx=0x7f5c0c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03607e055a98e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.463+0000 7f5c12ffd640 1 -- 172.21.15.94:0/1857491104 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c039070 con 0x7f5c14155a40 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.463+0000 7f5c12ffd640 1 -- 172.21.15.94:0/1857491104 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c02cc50 con 0x7f5c14155a40 2024-03-06T07:19:54.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.463+0000 7f5c12ffd640 1 -- 172.21.15.94:0/1857491104 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c034440 con 0x7f5c14155a40 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1857491104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c14155a40 msgr2=0x7f5c14155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1857491104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c14155a40 0x7f5c14155ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009d90 tx=0x7f5c0c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1857491104 shutdown_connections 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1857491104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c14155a40 0x7f5c14155ed0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1857491104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 0x7f5c141529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1857491104 >> 172.21.15.94:0/1857491104 conn(0x7f5c140c6710 msgr2=0x7f5c140c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1857491104 shutdown_connections 2024-03-06T07:19:54.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.464+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1857491104 wait complete. 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.465+0000 7f5c1a55c640 1 Processor -- start 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.465+0000 7f5c1a55c640 1 -- start start 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c1a55c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c140c1aa0 0x7f5c14151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c1a55c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 0x7f5c141dd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c1a55c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c14155a40 0x7f5c141eb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c1a55c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140c0d60 con 0x7f5c140c1aa0 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c1a55c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14155520 con 0x7f5c14155a40 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c1a55c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140c5e30 con 0x7f5c1414f790 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c140c1aa0 0x7f5c14151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c140c1aa0 0x7f5c14151060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60450/0 (socket says 172.21.15.94:60450) 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 -- 172.21.15.94:0/1815723212 learned_addr learned my addr 172.21.15.94:0/1815723212 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 -- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 msgr2=0x7f5c141dd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 0x7f5c141dd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 -- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c14155a40 msgr2=0x7f5c141eb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c14155a40 0x7f5c141eb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 -- 172.21.15.94:0/1815723212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c141515a0 con 0x7f5c140c1aa0 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.466+0000 7f5c13fff640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c140c1aa0 0x7f5c14151060 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800fa70 tx=0x7f5c0800ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.467+0000 7f5c10ff9640 1 -- 172.21.15.94:0/1815723212 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08015070 con 0x7f5c140c1aa0 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.467+0000 7f5c10ff9640 1 -- 172.21.15.94:0/1815723212 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c08007c90 con 0x7f5c140c1aa0 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.467+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c141db730 con 0x7f5c140c1aa0 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.467+0000 7f5c10ff9640 1 -- 172.21.15.94:0/1815723212 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0801a680 con 0x7f5c140c1aa0 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.467+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c14150580 con 0x7f5c140c1aa0 2024-03-06T07:19:54.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.468+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1404f330 con 0x7f5c140c1aa0 2024-03-06T07:19:54.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.468+0000 7f5c10ff9640 1 -- 172.21.15.94:0/1815723212 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c0801a820 con 0x7f5c140c1aa0 2024-03-06T07:19:54.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.468+0000 7f5c10ff9640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c0006b4d0 0x7f5c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.469+0000 7f5c10ff9640 1 -- 172.21.15.94:0/1815723212 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c0808cca0 con 0x7f5c140c1aa0 2024-03-06T07:19:54.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.469+0000 7f5c137fe640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c0006b4d0 0x7f5c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.469+0000 7f5c137fe640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c0006b4d0 0x7f5c0006d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5c04007fd0 tx=0x7f5c04007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.473+0000 7f5c10ff9640 1 -- 172.21.15.94:0/1815723212 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c0805b990 con 0x7f5c140c1aa0 2024-03-06T07:19:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.647+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c140c39d0 con 0x7f5c140c1aa0 2024-03-06T07:19:54.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.648+0000 7f5c10ff9640 1 -- 172.21.15.94:0/1815723212 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16955 (secure 0 0 0) 0x7f5c0805b290 con 0x7f5c140c1aa0 2024-03-06T07:19:54.650 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:54.650 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":51,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:53.385537+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c0006b4d0 msgr2=0x7f5c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c0006b4d0 0x7f5c0006d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5c04007fd0 tx=0x7f5c04007920 comp rx=0 tx=0).stop 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c140c1aa0 msgr2=0x7f5c14151060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c140c1aa0 0x7f5c14151060 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800fa70 tx=0x7f5c0800ff30 comp rx=0 tx=0).stop 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 shutdown_connections 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c0006b4d0 0x7f5c0006d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c14155a40 0x7f5c141eb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c1414f790 0x7f5c141dd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 --2- 172.21.15.94:0/1815723212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c140c1aa0 0x7f5c14151060 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 >> 172.21.15.94:0/1815723212 conn(0x7f5c140c6710 msgr2=0x7f5c1406cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:54.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 shutdown_connections 2024-03-06T07:19:54.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.650+0000 7f5c1a55c640 1 -- 172.21.15.94:0/1815723212 wait complete. 2024-03-06T07:19:54.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:19:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.764+0000 7f5c00e9d640 1 Processor -- start 2024-03-06T07:19:54.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.764+0000 7f5c00e9d640 1 -- start start 2024-03-06T07:19:54.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5c00e9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc14ef70 0x7f5bfc06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5c00e9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5bfc06cad0 0x7f5bfc154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5c00e9d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0c2790 con 0x7f5bfc0c1820 2024-03-06T07:19:54.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5c00e9d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc0c1ca0 con 0x7f5bfc06cad0 2024-03-06T07:19:54.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5c00e9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0c3090 con 0x7f5bfc14ef70 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc14ef70 0x7f5bfc06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5bfad9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5bfc06cad0 0x7f5bfc154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc14ef70 0x7f5bfc06c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:45438/0 (socket says 172.21.15.94:45438) 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.765+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2794009721 learned_addr learned my addr 172.21.15.94:0/2794009721 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.766+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2794009721 >> v2:172.21.15.145:3300/0 conn(0x7f5bfc06cad0 msgr2=0x7f5bfc154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.766+0000 7f5bfa59c640 1 --2- 172.21.15.94:0/2794009721 >> v2:172.21.15.145:3300/0 conn(0x7f5bfc06cad0 0x7f5bfc154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.766+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2794009721 >> v1:172.21.15.145:6789/0 conn(0x7f5bfc0c1820 legacy=0x7f5bfc0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.766+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2794009721 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0c1540 con 0x7f5bfc14ef70 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.766+0000 7f5bfa59c640 1 --2- 172.21.15.94:0/2794009721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc14ef70 0x7f5bfc06c1f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00bb00 tx=0x7f5bec02dea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fdeec7bc9763f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5bf9d9b640 1 -- 172.21.15.94:0/2794009721 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec034070 con 0x7f5bfc14ef70 2024-03-06T07:19:54.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5bf9d9b640 1 -- 172.21.15.94:0/2794009721 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec02ec50 con 0x7f5bfc14ef70 2024-03-06T07:19:54.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5bf9d9b640 1 -- 172.21.15.94:0/2794009721 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec038400 con 0x7f5bfc14ef70 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2794009721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc14ef70 msgr2=0x7f5bfc06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2794009721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc14ef70 0x7f5bfc06c1f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00bb00 tx=0x7f5bec02dea0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2794009721 shutdown_connections 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2794009721 >> v2:172.21.15.145:3300/0 conn(0x7f5bfc06cad0 0x7f5bfc154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2794009721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc14ef70 0x7f5bfc06c1f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2794009721 >> 172.21.15.94:0/2794009721 conn(0x7f5bfc0c6710 msgr2=0x7f5bfc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2794009721 shutdown_connections 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.767+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2794009721 wait complete. 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.768+0000 7f5c00e9d640 1 Processor -- start 2024-03-06T07:19:54.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.768+0000 7f5c00e9d640 1 -- start start 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5c00e9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5bfc06cad0 0x7f5bfc1511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5c00e9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc0c1820 0x7f5bfc150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5c00e9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc14ef70 0x7f5bfc151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5c00e9d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc06cf60 con 0x7f5bfc06cad0 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5c00e9d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc14f350 con 0x7f5bfc14ef70 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5c00e9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc155750 con 0x7f5bfc0c1820 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5bfc06cad0 0x7f5bfc1511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5bfc06cad0 0x7f5bfc1511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60454/0 (socket says 172.21.15.94:60454) 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2449422751 learned_addr learned my addr 172.21.15.94:0/2449422751 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfad9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc14ef70 0x7f5bfc151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bf3fff640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc0c1820 0x7f5bfc150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc0c1820 msgr2=0x7f5bfc150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc0c1820 0x7f5bfc150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc14ef70 msgr2=0x7f5bfc151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc14ef70 0x7f5bfc151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfa59c640 1 -- 172.21.15.94:0/2449422751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0705a0 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.769+0000 7f5bfad9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc14ef70 0x7f5bfc151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.770+0000 7f5bfa59c640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5bfc06cad0 0x7f5bfc1511c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00bb00 tx=0x7f5bec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.770+0000 7f5bf2ffd640 1 -- 172.21.15.94:0/2449422751 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec047020 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.770+0000 7f5bf2ffd640 1 -- 172.21.15.94:0/2449422751 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec02ea50 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.770+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc070b60 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.770+0000 7f5bf2ffd640 1 -- 172.21.15.94:0/2449422751 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec0438c0 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.770+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc1523d0 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.771+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc070330 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.771+0000 7f5bf2ffd640 1 -- 172.21.15.94:0/2449422751 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bec04a020 con 0x7f5bfc06cad0 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.772+0000 7f5bf2ffd640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5bdc06b4d0 0x7f5bdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:54.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.772+0000 7f5bf2ffd640 1 -- 172.21.15.94:0/2449422751 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5bec0aec20 con 0x7f5bfc06cad0 2024-03-06T07:19:54.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.772+0000 7f5bf3fff640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5bdc06b4d0 0x7f5bdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:54.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.772+0000 7f5bf3fff640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5bdc06b4d0 0x7f5bdc06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc151700 tx=0x7f5be4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:54.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.776+0000 7f5bf2ffd640 1 -- 172.21.15.94:0/2449422751 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bec051030 con 0x7f5bfc06cad0 2024-03-06T07:19:54.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:54.986+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5bfc04f330 con 0x7f5bfc06cad0 2024-03-06T07:19:55.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.043+0000 7f5bf2ffd640 1 -- 172.21.15.94:0/2449422751 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v28) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5bec07d9a0 con 0x7f5bfc06cad0 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5bdc06b4d0 msgr2=0x7f5bdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5bdc06b4d0 0x7f5bdc06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc151700 tx=0x7f5be4009040 comp rx=0 tx=0).stop 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5bfc06cad0 msgr2=0x7f5bfc1511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5bfc06cad0 0x7f5bfc1511c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5bec00bb00 tx=0x7f5bec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 shutdown_connections 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5bdc06b4d0 0x7f5bdc06d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bfc14ef70 0x7f5bfc151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bfc0c1820 0x7f5bfc150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 --2- 172.21.15.94:0/2449422751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5bfc06cad0 0x7f5bfc1511c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.045+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 >> 172.21.15.94:0/2449422751 conn(0x7f5bfc0c6710 msgr2=0x7f5bfc0ab6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.046+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 shutdown_connections 2024-03-06T07:19:55.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.046+0000 7f5c00e9d640 1 -- 172.21.15.94:0/2449422751 wait complete. 2024-03-06T07:19:55.058 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:19:55.059 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:19:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.159+0000 7f1b5c577640 1 Processor -- start 2024-03-06T07:19:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.159+0000 7f1b5c577640 1 -- start start 2024-03-06T07:19:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5c577640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b54155c30 0x7f1b5406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5c577640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1b540c5380 0x7f1b540c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5c577640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b540b0450 con 0x7f1b540b0070 2024-03-06T07:19:55.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5c577640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5406cfb0 con 0x7f1b540c5380 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5c577640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5406c020 con 0x7f1b54155c30 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b59b12640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b54155c30 0x7f1b5406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5ab14640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1b540c5380 0x7f1b540c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5ab14640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1b540c5380 0x7f1b540c57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:49058/0 (socket says 172.21.15.94:49058) 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b59b12640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b54155c30 0x7f1b5406ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:45452/0 (socket says 172.21.15.94:45452) 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5ab14640 1 -- 172.21.15.94:0/493615274 learned_addr learned my addr 172.21.15.94:0/493615274 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5a313640 1 --1- 172.21.15.94:0/493615274 >> v1:172.21.15.145:6789/0 conn(0x7f1b540b0070 0x7f1b540c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:60082/0 (socket says 172.21.15.94:60082) 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5ab14640 1 -- 172.21.15.94:0/493615274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b54155c30 msgr2=0x7f1b5406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5ab14640 1 --2- 172.21.15.94:0/493615274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b54155c30 0x7f1b5406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5ab14640 1 -- 172.21.15.94:0/493615274 >> v1:172.21.15.145:6789/0 conn(0x7f1b540b0070 legacy=0x7f1b540c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:55.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.160+0000 7f1b5ab14640 1 -- 172.21.15.94:0/493615274 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b541560e0 con 0x7f1b540c5380 2024-03-06T07:19:55.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.161+0000 7f1b5ab14640 1 --2- 172.21.15.94:0/493615274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b540c5380 0x7f1b540c57f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009e50 tx=0x7f1b4c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae89fc211024795 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.161+0000 7f1b59311640 1 -- 172.21.15.94:0/493615274 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c02ca70 con 0x7f1b540c5380 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.161+0000 7f1b59311640 1 -- 172.21.15.94:0/493615274 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c02cc10 con 0x7f1b540c5380 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.161+0000 7f1b59311640 1 -- 172.21.15.94:0/493615274 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0343f0 con 0x7f1b540c5380 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 -- 172.21.15.94:0/493615274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b540c5380 msgr2=0x7f1b540c57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 --2- 172.21.15.94:0/493615274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b540c5380 0x7f1b540c57f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009e50 tx=0x7f1b4c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 -- 172.21.15.94:0/493615274 shutdown_connections 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 --2- 172.21.15.94:0/493615274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b540c5380 0x7f1b540c57f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 --2- 172.21.15.94:0/493615274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b54155c30 0x7f1b5406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 -- 172.21.15.94:0/493615274 >> 172.21.15.94:0/493615274 conn(0x7f1b540ac590 msgr2=0x7f1b540ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 -- 172.21.15.94:0/493615274 shutdown_connections 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.162+0000 7f1b5c577640 1 -- 172.21.15.94:0/493615274 wait complete. 2024-03-06T07:19:55.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.163+0000 7f1b5c577640 1 Processor -- start 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.163+0000 7f1b5c577640 1 -- start start 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.163+0000 7f1b5c577640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b540b0070 0x7f1b540c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.163+0000 7f1b5c577640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b540c5380 0x7f1b54154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.163+0000 7f1b5c577640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54155c30 0x7f1b540bfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b5c577640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5406c270 con 0x7f1b540c5380 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b5c577640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540c7ad0 con 0x7f1b54155c30 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b5c577640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5406c550 con 0x7f1b540b0070 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b540c5380 0x7f1b54154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b540c5380 0x7f1b54154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60460/0 (socket says 172.21.15.94:60460) 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 -- 172.21.15.94:0/3473614234 learned_addr learned my addr 172.21.15.94:0/3473614234 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b5a313640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b540b0070 0x7f1b540c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b5ab14640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54155c30 0x7f1b540bfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 -- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b540b0070 msgr2=0x7f1b540c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:55.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b540b0070 0x7f1b540c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 -- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54155c30 msgr2=0x7f1b540bfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54155c30 0x7f1b540bfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 -- 172.21.15.94:0/3473614234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b541549b0 con 0x7f1b540c5380 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b59b12640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b540c5380 0x7f1b54154470 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400e9b0 tx=0x7f1b4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b4affd640 1 -- 172.21.15.94:0/3473614234 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44017070 con 0x7f1b540c5380 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b4affd640 1 -- 172.21.15.94:0/3473614234 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4400c9a0 con 0x7f1b540c5380 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b541eb990 con 0x7f1b540c5380 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b4affd640 1 -- 172.21.15.94:0/3473614234 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4400cd90 con 0x7f1b540c5380 2024-03-06T07:19:55.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.164+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b540bf6e0 con 0x7f1b540c5380 2024-03-06T07:19:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.165+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5404f330 con 0x7f1b540c5380 2024-03-06T07:19:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.166+0000 7f1b4affd640 1 -- 172.21.15.94:0/3473614234 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b4401a070 con 0x7f1b540c5380 2024-03-06T07:19:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.166+0000 7f1b4affd640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b3806b4d0 0x7f1b3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.166+0000 7f1b4affd640 1 -- 172.21.15.94:0/3473614234 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b4408a290 con 0x7f1b540c5380 2024-03-06T07:19:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.166+0000 7f1b5a313640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b3806b4d0 0x7f1b3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:55.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.167+0000 7f1b5a313640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b3806b4d0 0x7f1b3806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:55.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.171+0000 7f1b4affd640 1 -- 172.21.15.94:0/3473614234 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b440589c0 con 0x7f1b540c5380 2024-03-06T07:19:55.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:55.377+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1b540ac000 con 0x7f1b540c5380 2024-03-06T07:19:56.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.048+0000 7f1b4affd640 1 -- 172.21.15.94:0/3473614234 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v29) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1b4405ca60 con 0x7f1b540c5380 2024-03-06T07:19:56.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b3806b4d0 msgr2=0x7f1b3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b3806b4d0 0x7f1b3806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50007e80 comp rx=0 tx=0).stop 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b540c5380 msgr2=0x7f1b54154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b540c5380 0x7f1b54154470 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400e9b0 tx=0x7f1b4400ee70 comp rx=0 tx=0).stop 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 shutdown_connections 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b3806b4d0 0x7f1b3806d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b54155c30 0x7f1b540bfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b540c5380 0x7f1b54154470 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 --2- 172.21.15.94:0/3473614234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b540b0070 0x7f1b540c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 >> 172.21.15.94:0/3473614234 conn(0x7f1b540ac590 msgr2=0x7f1b540ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.050+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 shutdown_connections 2024-03-06T07:19:56.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.051+0000 7f1b5c577640 1 -- 172.21.15.94:0/3473614234 wait complete. 2024-03-06T07:19:56.065 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:19:56.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:19:56.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.168+0000 7f7f6c3d7640 1 Processor -- start 2024-03-06T07:19:56.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.168+0000 7f7f6c3d7640 1 -- start start 2024-03-06T07:19:56.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6c3d7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7f6414f790 0x7f7f641529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6c3d7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7f640c1aa0 0x7f7f640c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6c3d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f640c5c60 0x7f7f64155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6c3d7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f640c2680 con 0x7f7f640c1aa0 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6c3d7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f640c28d0 con 0x7f7f6414f790 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6c3d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640c2bb0 con 0x7f7f640c5c60 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7f640c1aa0 0x7f7f640c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7f640c1aa0 0x7f7f640c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60472/0 (socket says 172.21.15.94:60472) 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 -- 172.21.15.94:0/186012593 learned_addr learned my addr 172.21.15.94:0/186012593 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6a974640 1 --2- 172.21.15.94:0/186012593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f640c5c60 0x7f7f64155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f6a173640 1 --2- 172.21.15.94:0/186012593 >> v2:172.21.15.145:3300/0 conn(0x7f7f6414f790 0x7f7f641529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 -- 172.21.15.94:0/186012593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f640c5c60 msgr2=0x7f7f64155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:56.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 --2- 172.21.15.94:0/186012593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f640c5c60 0x7f7f64155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 -- 172.21.15.94:0/186012593 >> v2:172.21.15.145:3300/0 conn(0x7f7f6414f790 msgr2=0x7f7f641529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:56.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 --2- 172.21.15.94:0/186012593 >> v2:172.21.15.145:3300/0 conn(0x7f7f6414f790 0x7f7f641529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.169+0000 7f7f69972640 1 -- 172.21.15.94:0/186012593 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f640c2e90 con 0x7f7f640c1aa0 2024-03-06T07:19:56.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.170+0000 7f7f69972640 1 --2- 172.21.15.94:0/186012593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c1aa0 0x7f7f640c5720 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f640c3170 tx=0x7f7f5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31342fa9eda50c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.170+0000 7f7f69171640 1 -- 172.21.15.94:0/186012593 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54035020 con 0x7f7f640c1aa0 2024-03-06T07:19:56.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.170+0000 7f7f69171640 1 -- 172.21.15.94:0/186012593 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54031870 con 0x7f7f640c1aa0 2024-03-06T07:19:56.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.170+0000 7f7f69171640 1 -- 172.21.15.94:0/186012593 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54031c40 con 0x7f7f640c1aa0 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/186012593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c1aa0 msgr2=0x7f7f640c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/186012593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c1aa0 0x7f7f640c5720 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f640c3170 tx=0x7f7f5402b910 comp rx=0 tx=0).stop 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/186012593 shutdown_connections 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/186012593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f640c5c60 0x7f7f64155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/186012593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c1aa0 0x7f7f640c5720 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/186012593 >> v2:172.21.15.145:3300/0 conn(0x7f7f6414f790 0x7f7f641529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/186012593 >> 172.21.15.94:0/186012593 conn(0x7f7f640c6710 msgr2=0x7f7f640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/186012593 shutdown_connections 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.171+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/186012593 wait complete. 2024-03-06T07:19:56.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.172+0000 7f7f6c3d7640 1 Processor -- start 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.172+0000 7f7f6c3d7640 1 -- start start 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6c3d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7f640c1aa0 0x7f7f6414acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6c3d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c5c60 0x7f7f64153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6c3d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f6414f790 0x7f7f64149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6c3d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6406cc00 con 0x7f7f640c5c60 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6c3d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640c0d60 con 0x7f7f640c1aa0 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6c3d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6406cd80 con 0x7f7f6414f790 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c5c60 0x7f7f64153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c5c60 0x7f7f64153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60484/0 (socket says 172.21.15.94:60484) 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 -- 172.21.15.94:0/3768593161 learned_addr learned my addr 172.21.15.94:0/3768593161 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6a974640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f6414f790 0x7f7f64149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6a173640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7f640c1aa0 0x7f7f6414acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 -- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f6414f790 msgr2=0x7f7f64149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f6414f790 0x7f7f64149980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 -- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7f640c1aa0 msgr2=0x7f7f6414acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7f640c1aa0 0x7f7f6414acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 -- 172.21.15.94:0/3768593161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f641df410 con 0x7f7f640c5c60 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f69972640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c5c60 0x7f7f64153590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7f5402b950 tx=0x7f7f540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f5affd640 1 -- 172.21.15.94:0/3768593161 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54040070 con 0x7f7f640c5c60 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f5affd640 1 -- 172.21.15.94:0/3768593161 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54031e10 con 0x7f7f640c5c60 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6414a700 con 0x7f7f640c5c60 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.173+0000 7f7f5affd640 1 -- 172.21.15.94:0/3768593161 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54035040 con 0x7f7f640c5c60 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.174+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f64153ad0 con 0x7f7f640c5c60 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.175+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6404f330 con 0x7f7f640c5c60 2024-03-06T07:19:56.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.175+0000 7f7f5affd640 1 -- 172.21.15.94:0/3768593161 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f5404b060 con 0x7f7f640c5c60 2024-03-06T07:19:56.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.175+0000 7f7f5affd640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7f4806b4d0 0x7f7f4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.175+0000 7f7f5affd640 1 -- 172.21.15.94:0/3768593161 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7f540abe50 con 0x7f7f640c5c60 2024-03-06T07:19:56.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.175+0000 7f7f6a173640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7f4806b4d0 0x7f7f4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.176+0000 7f7f6a173640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7f4806b4d0 0x7f7f4806d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7f640733f0 tx=0x7f7f60009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.179+0000 7f7f5affd640 1 -- 172.21.15.94:0/3768593161 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f5407a580 con 0x7f7f640c5c60 2024-03-06T07:19:56.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.383+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f6406c770 con 0x7f7f640c5c60 2024-03-06T07:19:56.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.384+0000 7f7f5affd640 1 -- 172.21.15.94:0/3768593161 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f7f5407e620 con 0x7f7f640c5c60 2024-03-06T07:19:56.386 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:56.386 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5521,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/480754107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":480754107},{"type":"v1","addr":"172.21.15.94:6835","nonce":480754107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:19:55.378348+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35084":{"gid":35084,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3125091562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3125091562},{"type":"v1","addr":"172.21.15.145:6837","nonce":3125091562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35084,"qdb_cluster":[35084]},"id":3}]} 2024-03-06T07:19:56.386 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 29 2024-03-06T07:19:56.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7f4806b4d0 msgr2=0x7f7f4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7f4806b4d0 0x7f7f4806d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7f640733f0 tx=0x7f7f60009190 comp rx=0 tx=0).stop 2024-03-06T07:19:56.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c5c60 msgr2=0x7f7f64153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c5c60 0x7f7f64153590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7f5402b950 tx=0x7f7f540046c0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 shutdown_connections 2024-03-06T07:19:56.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7f4806b4d0 0x7f7f4806d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7f6414f790 0x7f7f64149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f640c5c60 0x7f7f64153590 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 --2- 172.21.15.94:0/3768593161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7f640c1aa0 0x7f7f6414acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 >> 172.21.15.94:0/3768593161 conn(0x7f7f640c6710 msgr2=0x7f7f640c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:56.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 shutdown_connections 2024-03-06T07:19:56.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.386+0000 7f7f6c3d7640 1 -- 172.21.15.94:0/3768593161 wait complete. 2024-03-06T07:19:56.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:19:56.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.497+0000 7fa06a38a640 1 Processor -- start 2024-03-06T07:19:56.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.497+0000 7fa06a38a640 1 -- start start 2024-03-06T07:19:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa06a38a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa06414f790 0x7fa0641529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa06a38a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa064155a40 0x7fa064155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa06a38a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640c28d0 con 0x7fa0640c1aa0 2024-03-06T07:19:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa06a38a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640c2680 con 0x7fa06414f790 2024-03-06T07:19:56.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa06a38a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640c2bb0 con 0x7fa064155a40 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa063fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa06414f790 0x7fa0641529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa063fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa06414f790 0x7fa0641529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60498/0 (socket says 172.21.15.94:60498) 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa063fff640 1 -- 172.21.15.94:0/506316228 learned_addr learned my addr 172.21.15.94:0/506316228 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.498+0000 7fa068927640 1 --2- 172.21.15.94:0/506316228 >> v2:172.21.15.145:3300/0 conn(0x7fa064155a40 0x7fa064155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa063fff640 1 -- 172.21.15.94:0/506316228 >> v2:172.21.15.145:3300/0 conn(0x7fa064155a40 msgr2=0x7fa064155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa063fff640 1 --2- 172.21.15.94:0/506316228 >> v2:172.21.15.145:3300/0 conn(0x7fa064155a40 0x7fa064155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa063fff640 1 -- 172.21.15.94:0/506316228 >> v1:172.21.15.145:6789/0 conn(0x7fa0640c1aa0 legacy=0x7fa0640c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa063fff640 1 -- 172.21.15.94:0/506316228 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640c2e90 con 0x7fa06414f790 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa063fff640 1 --2- 172.21.15.94:0/506316228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641529d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa06406c960 tx=0x7fa05402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbdd8ed92e8648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa062ffd640 1 -- 172.21.15.94:0/506316228 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054034070 con 0x7fa06414f790 2024-03-06T07:19:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa062ffd640 1 -- 172.21.15.94:0/506316228 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa054030860 con 0x7fa06414f790 2024-03-06T07:19:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.499+0000 7fa062ffd640 1 -- 172.21.15.94:0/506316228 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054030c30 con 0x7fa06414f790 2024-03-06T07:19:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 -- 172.21.15.94:0/506316228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 msgr2=0x7fa0641529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 --2- 172.21.15.94:0/506316228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641529d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa06406c960 tx=0x7fa05402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 -- 172.21.15.94:0/506316228 shutdown_connections 2024-03-06T07:19:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 --2- 172.21.15.94:0/506316228 >> v2:172.21.15.145:3300/0 conn(0x7fa064155a40 0x7fa064155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 --2- 172.21.15.94:0/506316228 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641529d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 -- 172.21.15.94:0/506316228 >> 172.21.15.94:0/506316228 conn(0x7fa0640c6710 msgr2=0x7fa0640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 -- 172.21.15.94:0/506316228 shutdown_connections 2024-03-06T07:19:56.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.500+0000 7fa06a38a640 1 -- 172.21.15.94:0/506316228 wait complete. 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.501+0000 7fa06a38a640 1 Processor -- start 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.501+0000 7fa06a38a640 1 -- start start 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa06a38a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c1aa0 0x7fa064153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa06a38a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641df890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa06a38a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064155a40 0x7fa0641e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa06a38a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c0d60 con 0x7fa06414f790 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa06a38a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064156390 con 0x7fa064155a40 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa06a38a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640c5e30 con 0x7fa0640c1aa0 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641df890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641df890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60512/0 (socket says 172.21.15.94:60512) 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 -- 172.21.15.94:0/1744803185 learned_addr learned my addr 172.21.15.94:0/1744803185 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa063fff640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c1aa0 0x7fa064153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa068927640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064155a40 0x7fa0641e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 -- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c1aa0 msgr2=0x7fa064153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c1aa0 0x7fa064153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 -- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064155a40 msgr2=0x7fa0641e92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064155a40 0x7fa0641e92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 -- 172.21.15.94:0/1744803185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06414a440 con 0x7fa06414f790 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.502+0000 7fa0637fe640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641df890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa05800c440 tx=0x7fa058001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.503+0000 7fa060ff9640 1 -- 172.21.15.94:0/1744803185 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580052d0 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.503+0000 7fa060ff9640 1 -- 172.21.15.94:0/1744803185 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa058005470 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.503+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa064154da0 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.503+0000 7fa060ff9640 1 -- 172.21.15.94:0/1744803185 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058007260 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.503+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0641e8ad0 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.504+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06404f330 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.504+0000 7fa060ff9640 1 -- 172.21.15.94:0/1744803185 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa058007400 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.504+0000 7fa060ff9640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa05006b4d0 0x7fa05006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.504+0000 7fa060ff9640 1 -- 172.21.15.94:0/1744803185 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa058089760 con 0x7fa06414f790 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.505+0000 7fa063fff640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa05006b4d0 0x7fa05006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.505+0000 7fa063fff640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa05006b4d0 0x7fa05006d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa054036000 tx=0x7fa054008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.509+0000 7fa060ff9640 1 -- 172.21.15.94:0/1744803185 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa058057e90 con 0x7fa06414f790 2024-03-06T07:19:56.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.680+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0640c1f40 con 0x7fa06414f790 2024-03-06T07:19:56.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.682+0000 7fa060ff9640 1 -- 172.21.15.94:0/1744803185 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16942 (secure 0 0 0) 0x7fa05805bf30 con 0x7fa06414f790 2024-03-06T07:19:56.684 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:56.684 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":52,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:55.029727+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:19:56.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa05006b4d0 msgr2=0x7fa05006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa05006b4d0 0x7fa05006d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa054036000 tx=0x7fa054008030 comp rx=0 tx=0).stop 2024-03-06T07:19:56.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 msgr2=0x7fa0641df890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641df890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa05800c440 tx=0x7fa058001e70 comp rx=0 tx=0).stop 2024-03-06T07:19:56.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 shutdown_connections 2024-03-06T07:19:56.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa05006b4d0 0x7fa05006d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064155a40 0x7fa0641e92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa06414f790 0x7fa0641df890 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 --2- 172.21.15.94:0/1744803185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c1aa0 0x7fa064153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.683+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 >> 172.21.15.94:0/1744803185 conn(0x7fa0640c6710 msgr2=0x7fa0640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:56.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.684+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 shutdown_connections 2024-03-06T07:19:56.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.684+0000 7fa06a38a640 1 -- 172.21.15.94:0/1744803185 wait complete. 2024-03-06T07:19:56.697 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:19:56.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.795+0000 7fa35cd1e640 1 Processor -- start 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.795+0000 7fa35cd1e640 1 -- start start 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa35cd1e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa3580c1aa0 0x7fa3581529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa35cd1e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa358155a40 0x7fa358155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa35cd1e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3580c28d0 con 0x7fa35814f790 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa35cd1e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3580c2680 con 0x7fa358155a40 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa35cd1e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3580c2bb0 con 0x7fa3580c1aa0 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa35659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa35814f790 0x7fa3580c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47866/0 (socket says 172.21.15.94:47866) 2024-03-06T07:19:56.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa356d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa358155a40 0x7fa358155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa35659c640 1 -- 172.21.15.94:0/1037504513 learned_addr learned my addr 172.21.15.94:0/1037504513 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:56.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.796+0000 7fa355d9b640 1 --2- 172.21.15.94:0/1037504513 >> v2:172.21.15.145:3300/0 conn(0x7fa3580c1aa0 0x7fa3581529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.797+0000 7fa355d9b640 1 -- 172.21.15.94:0/1037504513 >> v2:172.21.15.94:3300/0 conn(0x7fa358155a40 msgr2=0x7fa358155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:56.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.797+0000 7fa355d9b640 1 --2- 172.21.15.94:0/1037504513 >> v2:172.21.15.94:3300/0 conn(0x7fa358155a40 0x7fa358155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.797+0000 7fa355d9b640 1 -- 172.21.15.94:0/1037504513 >> v1:172.21.15.94:6789/0 conn(0x7fa35814f790 legacy=0x7fa3580c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.797+0000 7fa35559a640 1 -- 172.21.15.94:0/1037504513 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988312444 0 0) 0x7fa3580c28d0 con 0x7fa35814f790 2024-03-06T07:19:56.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.797+0000 7fa355d9b640 1 -- 172.21.15.94:0/1037504513 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580c2e90 con 0x7fa3580c1aa0 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.797+0000 7fa355d9b640 1 --2- 172.21.15.94:0/1037504513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3580c1aa0 0x7fa3581529d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa358070280 tx=0x7fa34c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f76a1548d0749c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35559a640 1 -- 172.21.15.94:0/1037504513 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c035020 con 0x7fa3580c1aa0 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35559a640 1 -- 172.21.15.94:0/1037504513 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c031830 con 0x7fa3580c1aa0 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35559a640 1 -- 172.21.15.94:0/1037504513 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c031c00 con 0x7fa3580c1aa0 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 -- 172.21.15.94:0/1037504513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3580c1aa0 msgr2=0x7fa3581529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/1037504513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3580c1aa0 0x7fa3581529d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa358070280 tx=0x7fa34c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 -- 172.21.15.94:0/1037504513 shutdown_connections 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/1037504513 >> v2:172.21.15.94:3300/0 conn(0x7fa358155a40 0x7fa358155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/1037504513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3580c1aa0 0x7fa3581529d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 -- 172.21.15.94:0/1037504513 >> 172.21.15.94:0/1037504513 conn(0x7fa3580c6710 msgr2=0x7fa3580c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 -- 172.21.15.94:0/1037504513 shutdown_connections 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.798+0000 7fa35cd1e640 1 -- 172.21.15.94:0/1037504513 wait complete. 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 Processor -- start 2024-03-06T07:19:56.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 -- start start 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3580c1aa0 0x7fa358148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa35814f790 0x7fa358148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa358155a40 0x7fa358153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580c0d60 con 0x7fa3580c1aa0 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358156390 con 0x7fa35814f790 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35cd1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580c5e30 con 0x7fa358155a40 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3580c1aa0 0x7fa358148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3580c1aa0 0x7fa358148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60532/0 (socket says 172.21.15.94:60532) 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa35659c640 1 -- 172.21.15.94:0/2675102189 learned_addr learned my addr 172.21.15.94:0/2675102189 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa355d9b640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa35814f790 0x7fa358148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.800+0000 7fa356d9d640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa358155a40 0x7fa358153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35659c640 1 -- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa358155a40 msgr2=0x7fa358153b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35659c640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa358155a40 0x7fa358153b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35659c640 1 -- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa35814f790 msgr2=0x7fa358148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35659c640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa35814f790 0x7fa358148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35659c640 1 -- 172.21.15.94:0/2675102189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3581518f0 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35659c640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3580c1aa0 0x7fa358148690 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c4d0 tx=0x7fa35000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa346ffd640 1 -- 172.21.15.94:0/2675102189 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350018070 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa346ffd640 1 -- 172.21.15.94:0/2675102189 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa350007500 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa358151bd0 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa346ffd640 1 -- 172.21.15.94:0/2675102189 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa350004a20 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.801+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3581463d0 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.802+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa358069460 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.802+0000 7fa346ffd640 1 -- 172.21.15.94:0/2675102189 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa350004bc0 con 0x7fa3580c1aa0 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.803+0000 7fa346ffd640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa33806b4d0 0x7fa33806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:56.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.803+0000 7fa346ffd640 1 -- 172.21.15.94:0/2675102189 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa3500896e0 con 0x7fa3580c1aa0 2024-03-06T07:19:56.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.803+0000 7fa355d9b640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa33806b4d0 0x7fa33806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:56.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.803+0000 7fa355d9b640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa33806b4d0 0x7fa33806d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa34c036000 tx=0x7fa34c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:56.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.807+0000 7fa346ffd640 1 -- 172.21.15.94:0/2675102189 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa358069460 con 0x7fa3580c1aa0 2024-03-06T07:19:56.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.976+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa35804f330 con 0x7fa33806b4d0 2024-03-06T07:19:56.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.983+0000 7fa346ffd640 1 -- 172.21.15.94:0/2675102189 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240115 (secure 0 0 0) 0x7fa34c085a10 con 0x7fa33806b4d0 2024-03-06T07:19:56.987 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:19:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa33806b4d0 msgr2=0x7fa33806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa33806b4d0 0x7fa33806d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa34c036000 tx=0x7fa34c03d040 comp rx=0 tx=0).stop 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3580c1aa0 msgr2=0x7fa358148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3580c1aa0 0x7fa358148690 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c4d0 tx=0x7fa35000c990 comp rx=0 tx=0).stop 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 shutdown_connections 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa33806b4d0 0x7fa33806d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa358155a40 0x7fa358153b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa35814f790 0x7fa358148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 --2- 172.21.15.94:0/2675102189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3580c1aa0 0x7fa358148690 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.985+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 >> 172.21.15.94:0/2675102189 conn(0x7fa3580c6710 msgr2=0x7fa3580c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.986+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 shutdown_connections 2024-03-06T07:19:56.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:56.986+0000 7fa35cd1e640 1 -- 172.21.15.94:0/2675102189 wait complete. 2024-03-06T07:19:56.998 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-06T07:19:56.657592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233264,"kb_used_data":4612,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741456,"statfs":{"total":773094113280,"available":772855250944,"internally_reserved":0,"allocated":4722688,"data_stored":3194211,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.631411"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032038+0000","last_change":"2024-03-06T07:19:52.032038+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032038+0000","last_undegraded":"2024-03-06T07:19:52.032038+0000","last_fullsized":"2024-03-06T07:19:52.032038+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028473+0000","last_change":"2024-03-06T07:19:52.028473+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028473+0000","last_undegraded":"2024-03-06T07:19:52.028473+0000","last_fullsized":"2024-03-06T07:19:52.028473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028203+0000","last_change":"2024-03-06T07:19:52.028203+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028203+0000","last_undegraded":"2024-03-06T07:19:52.028203+0000","last_fullsized":"2024-03-06T07:19:52.028203+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030450+0000","last_change":"2024-03-06T07:19:52.030450+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030450+0000","last_undegraded":"2024-03-06T07:19:52.030450+0000","last_fullsized":"2024-03-06T07:19:52.030450+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026946+0000","last_change":"2024-03-06T07:19:52.026946+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026946+0000","last_undegraded":"2024-03-06T07:19:52.026946+0000","last_fullsized":"2024-03-06T07:19:52.026946+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031565+0000","last_change":"2024-03-06T07:19:52.031565+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031565+0000","last_undegraded":"2024-03-06T07:19:52.031565+0000","last_fullsized":"2024-03-06T07:19:52.031565+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030267+0000","last_change":"2024-03-06T07:19:52.030267+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030267+0000","last_undegraded":"2024-03-06T07:19:52.030267+0000","last_fullsized":"2024-03-06T07:19:52.030267+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026041+0000","last_change":"2024-03-06T07:19:52.026041+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026041+0000","last_undegraded":"2024-03-06T07:19:52.026041+0000","last_fullsized":"2024-03-06T07:19:52.026041+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031120+0000","last_change":"2024-03-06T07:19:52.031120+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031120+0000","last_undegraded":"2024-03-06T07:19:52.031120+0000","last_fullsized":"2024-03-06T07:19:52.031120+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031693+0000","last_change":"2024-03-06T07:19:52.031693+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031693+0000","last_undegraded":"2024-03-06T07:19:52.031693+0000","last_fullsized":"2024-03-06T07:19:52.031693+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.033049+0000","last_change":"2024-03-06T07:19:52.033049+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.033049+0000","last_undegraded":"2024-03-06T07:19:52.033049+0000","last_fullsized":"2024-03-06T07:19:52.033049+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028770+0000","last_change":"2024-03-06T07:19:52.028770+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028770+0000","last_undegraded":"2024-03-06T07:19:52.028770+0000","last_fullsized":"2024-03-06T07:19:52.028770+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026034+0000","last_change":"2024-03-06T07:19:52.026034+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026034+0000","last_undegraded":"2024-03-06T07:19:52.026034+0000","last_fullsized":"2024-03-06T07:19:52.026034+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031288+0000","last_change":"2024-03-06T07:19:52.031288+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031288+0000","last_undegraded":"2024-03-06T07:19:52.031288+0000","last_fullsized":"2024-03-06T07:19:52.031288+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029723+0000","last_change":"2024-03-06T07:19:52.029723+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029723+0000","last_undegraded":"2024-03-06T07:19:52.029723+0000","last_fullsized":"2024-03-06T07:19:52.029723+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029048+0000","last_change":"2024-03-06T07:19:52.029048+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029048+0000","last_undegraded":"2024-03-06T07:19:52.029048+0000","last_fullsized":"2024-03-06T07:19:52.029048+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026057+0000","last_change":"2024-03-06T07:19:52.026057+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026057+0000","last_undegraded":"2024-03-06T07:19:52.026057+0000","last_fullsized":"2024-03-06T07:19:52.026057+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029237+0000","last_change":"2024-03-06T07:19:52.029237+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029237+0000","last_undegraded":"2024-03-06T07:19:52.029237+0000","last_fullsized":"2024-03-06T07:19:52.029237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030769+0000","last_change":"2024-03-06T07:19:52.030769+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030769+0000","last_undegraded":"2024-03-06T07:19:52.030769+0000","last_fullsized":"2024-03-06T07:19:52.030769+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030293+0000","last_change":"2024-03-06T07:19:52.030293+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030293+0000","last_undegraded":"2024-03-06T07:19:52.030293+0000","last_fullsized":"2024-03-06T07:19:52.030293+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028184+0000","last_change":"2024-03-06T07:19:52.028184+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028184+0000","last_undegraded":"2024-03-06T07:19:52.028184+0000","last_fullsized":"2024-03-06T07:19:52.028184+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029572+0000","last_change":"2024-03-06T07:19:52.029572+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029572+0000","last_undegraded":"2024-03-06T07:19:52.029572+0000","last_fullsized":"2024-03-06T07:19:52.029572+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026130+0000","last_change":"2024-03-06T07:19:52.026130+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026130+0000","last_undegraded":"2024-03-06T07:19:52.026130+0000","last_fullsized":"2024-03-06T07:19:52.026130+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029231+0000","last_change":"2024-03-06T07:19:52.029231+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029231+0000","last_undegraded":"2024-03-06T07:19:52.029231+0000","last_fullsized":"2024-03-06T07:19:52.029231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030787+0000","last_change":"2024-03-06T07:19:52.030787+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030787+0000","last_undegraded":"2024-03-06T07:19:52.030787+0000","last_fullsized":"2024-03-06T07:19:52.030787+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031978+0000","last_change":"2024-03-06T07:19:52.031978+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031978+0000","last_undegraded":"2024-03-06T07:19:52.031978+0000","last_fullsized":"2024-03-06T07:19:52.031978+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.025813+0000","last_change":"2024-03-06T07:19:52.025813+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.025813+0000","last_undegraded":"2024-03-06T07:19:52.025813+0000","last_fullsized":"2024-03-06T07:19:52.025813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028643+0000","last_change":"2024-03-06T07:19:52.028643+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028643+0000","last_undegraded":"2024-03-06T07:19:52.028643+0000","last_fullsized":"2024-03-06T07:19:52.028643+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028228+0000","last_change":"2024-03-06T07:19:52.028228+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028228+0000","last_undegraded":"2024-03-06T07:19:52.028228+0000","last_fullsized":"2024-03-06T07:19:52.028228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032309+0000","last_change":"2024-03-06T07:19:52.032309+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032309+0000","last_undegraded":"2024-03-06T07:19:52.032309+0000","last_fullsized":"2024-03-06T07:19:52.032309+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030595+0000","last_change":"2024-03-06T07:19:52.030595+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030595+0000","last_undegraded":"2024-03-06T07:19:52.030595+0000","last_fullsized":"2024-03-06T07:19:52.030595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026017+0000","last_change":"2024-03-06T07:19:52.026017+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026017+0000","last_undegraded":"2024-03-06T07:19:52.026017+0000","last_fullsized":"2024-03-06T07:19:52.026017+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":223,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-06T07:19:52.033500+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:19:52.033500+0000","last_peered":"2024-03-06T07:19:52.033500+0000","last_clean":"2024-03-06T07:19:52.033500+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:19:52.033500+0000","last_undegraded":"2024-03-06T07:19:52.033500+0000","last_fullsized":"2024-03-06T07:19:52.033500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:19:57.003 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:19:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.137+0000 7f3021c47640 1 Processor -- start 2024-03-06T07:19:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.137+0000 7f3021c47640 1 -- start start 2024-03-06T07:19:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f3021c47640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f301c0c1320 0x7f301c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f3021c47640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c153a60 con 0x7f301c150580 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f3021c47640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c153d40 con 0x7f301c0c6000 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f3021c47640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f301c06c2c0 con 0x7f301c0c1320 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f301c150580 0x7f301c06dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47876/0 (socket says 172.21.15.94:47876) 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301b7fe640 1 -- 172.21.15.94:0/3862035910 learned_addr learned my addr 172.21.15.94:0/3862035910 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301affd640 1 --2- 172.21.15.94:0/3862035910 >> v2:172.21.15.145:3300/0 conn(0x7f301c0c1320 0x7f301c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301a7fc640 1 -- 172.21.15.94:0/3862035910 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502552880 0 0) 0x7f301c153a60 con 0x7f301c150580 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301a7fc640 1 -- 172.21.15.94:0/3862035910 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3004003520 con 0x7f301c150580 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301affd640 1 -- 172.21.15.94:0/3862035910 >> v1:172.21.15.145:6789/0 conn(0x7f301c0c6000 legacy=0x7f301c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301affd640 1 -- 172.21.15.94:0/3862035910 >> v1:172.21.15.94:6789/0 conn(0x7f301c150580 legacy=0x7f301c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.138+0000 7f301affd640 1 -- 172.21.15.94:0/3862035910 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c06c5a0 con 0x7f301c0c1320 2024-03-06T07:19:57.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.139+0000 7f301affd640 1 --2- 172.21.15.94:0/3862035910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c1320 0x7f301c1566b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f30100097a0 tx=0x7f301002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5f2b048285e8439 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.139+0000 7f301a7fc640 1 -- 172.21.15.94:0/3862035910 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010035020 con 0x7f301c0c1320 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.139+0000 7f301a7fc640 1 -- 172.21.15.94:0/3862035910 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3010031830 con 0x7f301c0c1320 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.139+0000 7f301a7fc640 1 -- 172.21.15.94:0/3862035910 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010031c00 con 0x7f301c0c1320 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.140+0000 7f3021c47640 1 -- 172.21.15.94:0/3862035910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c1320 msgr2=0x7f301c1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.140+0000 7f3021c47640 1 --2- 172.21.15.94:0/3862035910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c1320 0x7f301c1566b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f30100097a0 tx=0x7f301002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.140+0000 7f3021c47640 1 -- 172.21.15.94:0/3862035910 shutdown_connections 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.140+0000 7f3021c47640 1 --2- 172.21.15.94:0/3862035910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c1320 0x7f301c1566b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.140+0000 7f3021c47640 1 -- 172.21.15.94:0/3862035910 >> 172.21.15.94:0/3862035910 conn(0x7f301c0c6710 msgr2=0x7f301c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.140+0000 7f3021c47640 1 -- 172.21.15.94:0/3862035910 shutdown_connections 2024-03-06T07:19:57.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.140+0000 7f3021c47640 1 -- 172.21.15.94:0/3862035910 wait complete. 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.141+0000 7f3021c47640 1 Processor -- start 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.141+0000 7f3021c47640 1 -- start start 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3021c47640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f301c0c1320 0x7f301c0a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3021c47640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c6000 0x7f301c1ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3021c47640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f301c150580 0x7f301c1e66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3021c47640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c06e420 con 0x7f301c150580 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3021c47640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c150960 con 0x7f301c0c6000 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3021c47640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c06e5a0 con 0x7f301c0c1320 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f301c150580 0x7f301c1e66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f301c150580 0x7f301c1e66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60536/0 (socket says 172.21.15.94:60536) 2024-03-06T07:19:57.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 -- 172.21.15.94:0/2218387551 learned_addr learned my addr 172.21.15.94:0/2218387551 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301b7fe640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f301c0c1320 0x7f301c0a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301affd640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c6000 0x7f301c1ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 -- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f301c0c1320 msgr2=0x7f301c0a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f301c0c1320 0x7f301c0a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 -- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c6000 msgr2=0x7f301c1ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c6000 0x7f301c1ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 -- 172.21.15.94:0/2218387551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c155be0 con 0x7f301c150580 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f301bfff640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f301c150580 0x7f301c1e66e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f301400c7d0 tx=0x7f301400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3003fff640 1 -- 172.21.15.94:0/2218387551 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014003b70 con 0x7f301c150580 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3003fff640 1 -- 172.21.15.94:0/2218387551 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3014003d30 con 0x7f301c150580 2024-03-06T07:19:57.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3003fff640 1 -- 172.21.15.94:0/2218387551 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301400aa90 con 0x7f301c150580 2024-03-06T07:19:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.142+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301c1543e0 con 0x7f301c150580 2024-03-06T07:19:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.143+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301c1549a0 con 0x7f301c150580 2024-03-06T07:19:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.144+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301c04f330 con 0x7f301c150580 2024-03-06T07:19:57.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.144+0000 7f3003fff640 1 -- 172.21.15.94:0/2218387551 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f301400ac30 con 0x7f301c150580 2024-03-06T07:19:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.144+0000 7f3003fff640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f300806b4d0 0x7f300806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.144+0000 7f3003fff640 1 -- 172.21.15.94:0/2218387551 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f301408a0d0 con 0x7f301c150580 2024-03-06T07:19:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.144+0000 7f301b7fe640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f300806b4d0 0x7f300806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.145+0000 7f301b7fe640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f300806b4d0 0x7f300806d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f300c0011e0 tx=0x7f300c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.149+0000 7f3003fff640 1 -- 172.21.15.94:0/2218387551 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3014058800 con 0x7f301c150580 2024-03-06T07:19:57.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.317+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f301c068450 con 0x7f300806b4d0 2024-03-06T07:19:57.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.323+0000 7f3003fff640 1 -- 172.21.15.94:0/2218387551 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240115 (secure 0 0 0) 0x7f301c068450 con 0x7f300806b4d0 2024-03-06T07:19:57.327 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:57.328 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f300806b4d0 msgr2=0x7f300806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f300806b4d0 0x7f300806d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f300c0011e0 tx=0x7f300c007d30 comp rx=0 tx=0).stop 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f301c150580 msgr2=0x7f301c1e66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f301c150580 0x7f301c1e66e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f301400c7d0 tx=0x7f301400cc90 comp rx=0 tx=0).stop 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 shutdown_connections 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f300806b4d0 0x7f300806d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f301c150580 0x7f301c1e66e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f301c0c6000 0x7f301c1ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 --2- 172.21.15.94:0/2218387551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f301c0c1320 0x7f301c0a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 >> 172.21.15.94:0/2218387551 conn(0x7f301c0c6710 msgr2=0x7f301c14f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 shutdown_connections 2024-03-06T07:19:57.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.326+0000 7f3021c47640 1 -- 172.21.15.94:0/2218387551 wait complete. 2024-03-06T07:19:57.339 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-06T07:19:56.657592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233264,"kb_used_data":4612,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741456,"statfs":{"total":773094113280,"available":772855250944,"internally_reserved":0,"allocated":4722688,"data_stored":3194211,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.631411"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032038+0000","last_change":"2024-03-06T07:19:52.032038+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032038+0000","last_undegraded":"2024-03-06T07:19:52.032038+0000","last_fullsized":"2024-03-06T07:19:52.032038+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028473+0000","last_change":"2024-03-06T07:19:52.028473+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028473+0000","last_undegraded":"2024-03-06T07:19:52.028473+0000","last_fullsized":"2024-03-06T07:19:52.028473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028203+0000","last_change":"2024-03-06T07:19:52.028203+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028203+0000","last_undegraded":"2024-03-06T07:19:52.028203+0000","last_fullsized":"2024-03-06T07:19:52.028203+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030450+0000","last_change":"2024-03-06T07:19:52.030450+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030450+0000","last_undegraded":"2024-03-06T07:19:52.030450+0000","last_fullsized":"2024-03-06T07:19:52.030450+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026946+0000","last_change":"2024-03-06T07:19:52.026946+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026946+0000","last_undegraded":"2024-03-06T07:19:52.026946+0000","last_fullsized":"2024-03-06T07:19:52.026946+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031565+0000","last_change":"2024-03-06T07:19:52.031565+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031565+0000","last_undegraded":"2024-03-06T07:19:52.031565+0000","last_fullsized":"2024-03-06T07:19:52.031565+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030267+0000","last_change":"2024-03-06T07:19:52.030267+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030267+0000","last_undegraded":"2024-03-06T07:19:52.030267+0000","last_fullsized":"2024-03-06T07:19:52.030267+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026041+0000","last_change":"2024-03-06T07:19:52.026041+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026041+0000","last_undegraded":"2024-03-06T07:19:52.026041+0000","last_fullsized":"2024-03-06T07:19:52.026041+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031120+0000","last_change":"2024-03-06T07:19:52.031120+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031120+0000","last_undegraded":"2024-03-06T07:19:52.031120+0000","last_fullsized":"2024-03-06T07:19:52.031120+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031693+0000","last_change":"2024-03-06T07:19:52.031693+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031693+0000","last_undegraded":"2024-03-06T07:19:52.031693+0000","last_fullsized":"2024-03-06T07:19:52.031693+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.033049+0000","last_change":"2024-03-06T07:19:52.033049+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.033049+0000","last_undegraded":"2024-03-06T07:19:52.033049+0000","last_fullsized":"2024-03-06T07:19:52.033049+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028770+0000","last_change":"2024-03-06T07:19:52.028770+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028770+0000","last_undegraded":"2024-03-06T07:19:52.028770+0000","last_fullsized":"2024-03-06T07:19:52.028770+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026034+0000","last_change":"2024-03-06T07:19:52.026034+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026034+0000","last_undegraded":"2024-03-06T07:19:52.026034+0000","last_fullsized":"2024-03-06T07:19:52.026034+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031288+0000","last_change":"2024-03-06T07:19:52.031288+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031288+0000","last_undegraded":"2024-03-06T07:19:52.031288+0000","last_fullsized":"2024-03-06T07:19:52.031288+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029723+0000","last_change":"2024-03-06T07:19:52.029723+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029723+0000","last_undegraded":"2024-03-06T07:19:52.029723+0000","last_fullsized":"2024-03-06T07:19:52.029723+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029048+0000","last_change":"2024-03-06T07:19:52.029048+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029048+0000","last_undegraded":"2024-03-06T07:19:52.029048+0000","last_fullsized":"2024-03-06T07:19:52.029048+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026057+0000","last_change":"2024-03-06T07:19:52.026057+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026057+0000","last_undegraded":"2024-03-06T07:19:52.026057+0000","last_fullsized":"2024-03-06T07:19:52.026057+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029237+0000","last_change":"2024-03-06T07:19:52.029237+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029237+0000","last_undegraded":"2024-03-06T07:19:52.029237+0000","last_fullsized":"2024-03-06T07:19:52.029237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030769+0000","last_change":"2024-03-06T07:19:52.030769+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030769+0000","last_undegraded":"2024-03-06T07:19:52.030769+0000","last_fullsized":"2024-03-06T07:19:52.030769+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030293+0000","last_change":"2024-03-06T07:19:52.030293+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030293+0000","last_undegraded":"2024-03-06T07:19:52.030293+0000","last_fullsized":"2024-03-06T07:19:52.030293+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028184+0000","last_change":"2024-03-06T07:19:52.028184+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028184+0000","last_undegraded":"2024-03-06T07:19:52.028184+0000","last_fullsized":"2024-03-06T07:19:52.028184+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029572+0000","last_change":"2024-03-06T07:19:52.029572+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029572+0000","last_undegraded":"2024-03-06T07:19:52.029572+0000","last_fullsized":"2024-03-06T07:19:52.029572+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026130+0000","last_change":"2024-03-06T07:19:52.026130+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026130+0000","last_undegraded":"2024-03-06T07:19:52.026130+0000","last_fullsized":"2024-03-06T07:19:52.026130+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029231+0000","last_change":"2024-03-06T07:19:52.029231+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029231+0000","last_undegraded":"2024-03-06T07:19:52.029231+0000","last_fullsized":"2024-03-06T07:19:52.029231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030787+0000","last_change":"2024-03-06T07:19:52.030787+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030787+0000","last_undegraded":"2024-03-06T07:19:52.030787+0000","last_fullsized":"2024-03-06T07:19:52.030787+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031978+0000","last_change":"2024-03-06T07:19:52.031978+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031978+0000","last_undegraded":"2024-03-06T07:19:52.031978+0000","last_fullsized":"2024-03-06T07:19:52.031978+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.025813+0000","last_change":"2024-03-06T07:19:52.025813+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.025813+0000","last_undegraded":"2024-03-06T07:19:52.025813+0000","last_fullsized":"2024-03-06T07:19:52.025813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028643+0000","last_change":"2024-03-06T07:19:52.028643+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028643+0000","last_undegraded":"2024-03-06T07:19:52.028643+0000","last_fullsized":"2024-03-06T07:19:52.028643+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028228+0000","last_change":"2024-03-06T07:19:52.028228+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028228+0000","last_undegraded":"2024-03-06T07:19:52.028228+0000","last_fullsized":"2024-03-06T07:19:52.028228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032309+0000","last_change":"2024-03-06T07:19:52.032309+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032309+0000","last_undegraded":"2024-03-06T07:19:52.032309+0000","last_fullsized":"2024-03-06T07:19:52.032309+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030595+0000","last_change":"2024-03-06T07:19:52.030595+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030595+0000","last_undegraded":"2024-03-06T07:19:52.030595+0000","last_fullsized":"2024-03-06T07:19:52.030595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026017+0000","last_change":"2024-03-06T07:19:52.026017+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026017+0000","last_undegraded":"2024-03-06T07:19:52.026017+0000","last_fullsized":"2024-03-06T07:19:52.026017+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":223,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-06T07:19:52.033500+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:19:52.033500+0000","last_peered":"2024-03-06T07:19:52.033500+0000","last_clean":"2024-03-06T07:19:52.033500+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:19:52.033500+0000","last_undegraded":"2024-03-06T07:19:52.033500+0000","last_fullsized":"2024-03-06T07:19:52.033500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:19:57.344 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:19:57.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.478+0000 7fdd3859a640 1 Processor -- start 2024-03-06T07:19:57.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.478+0000 7fdd3859a640 1 -- start start 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd3859a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdd300c1aa0 0x7fdd301529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd3859a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd300c28d0 con 0x7fdd30155a40 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd3859a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd300c2bb0 con 0x7fdd3014f790 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd3859a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd300c2680 con 0x7fdd300c1aa0 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd36b37640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdd30155a40 0x7fdd30155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47884/0 (socket says 172.21.15.94:47884) 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd36b37640 1 -- 172.21.15.94:0/152042224 learned_addr learned my addr 172.21.15.94:0/152042224 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35b35640 1 --2- 172.21.15.94:0/152042224 >> v2:172.21.15.145:3300/0 conn(0x7fdd300c1aa0 0x7fdd301529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695810522 0 0) 0x7fdd300c28d0 con 0x7fdd30155a40 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd1c005730 con 0x7fdd30155a40 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606358273 0 0) 0x7fdd1c005730 con 0x7fdd30155a40 2024-03-06T07:19:57.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 >> v2:172.21.15.145:3300/0 conn(0x7fdd300c1aa0 msgr2=0x7fdd301529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 --2- 172.21.15.94:0/152042224 >> v2:172.21.15.145:3300/0 conn(0x7fdd300c1aa0 0x7fdd301529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 >> v1:172.21.15.145:6789/0 conn(0x7fdd3014f790 legacy=0x7fdd300c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd300c2e90 con 0x7fdd30155a40 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fdd280019f0 con 0x7fdd30155a40 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 >> v1:172.21.15.94:6789/0 conn(0x7fdd30155a40 legacy=0x7fdd30155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd1c006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c006a10 0x7fdd1c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd1c0073d0 0x7fdd1c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd300c2680 con 0x7fdd1c005bc0 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c00e2a0 con 0x7fdd1c0073d0 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c00e4a0 con 0x7fdd1c006a10 2024-03-06T07:19:57.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.479+0000 7fdd36336640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd1c006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36b37640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd1c0073d0 0x7fdd1c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd35b35640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c006a10 0x7fdd1c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36336640 1 -- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c006a10 msgr2=0x7fdd1c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36336640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c006a10 0x7fdd1c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36336640 1 -- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd1c0073d0 msgr2=0x7fdd1c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36336640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd1c0073d0 0x7fdd1c00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36336640 1 -- 172.21.15.94:0/152042224 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c005730 con 0x7fdd1c005bc0 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36b37640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd1c0073d0 0x7fdd1c00dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36b37640 1 -- 172.21.15.94:0/152042224 reap_dead start 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd36336640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd1c006000 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c001750 tx=0x7fdd1c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ad47c7ddd52c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.480+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c001370 con 0x7fdd1c005bc0 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2c0052d0 con 0x7fdd1c005bc0 2024-03-06T07:19:57.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd35334640 1 -- 172.21.15.94:0/152042224 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c003830 con 0x7fdd1c005bc0 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd3859a640 1 -- 172.21.15.94:0/152042224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 msgr2=0x7fdd1c006000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd3859a640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd1c006000 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c001750 tx=0x7fdd1c007a40 comp rx=0 tx=0).stop 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd3859a640 1 -- 172.21.15.94:0/152042224 shutdown_connections 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd3859a640 1 --2- 172.21.15.94:0/152042224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd1c006000 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd3859a640 1 -- 172.21.15.94:0/152042224 >> 172.21.15.94:0/152042224 conn(0x7fdd300c6710 msgr2=0x7fdd300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd3859a640 1 -- 172.21.15.94:0/152042224 shutdown_connections 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.481+0000 7fdd3859a640 1 -- 172.21.15.94:0/152042224 wait complete. 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.482+0000 7fdd3859a640 1 Processor -- start 2024-03-06T07:19:57.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.482+0000 7fdd3859a640 1 -- start start 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd3859a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd300c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd3859a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd30153320 0x7fdd30154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd3859a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd301e6b60 0x7fdd301e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd3859a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3006cc00 con 0x7fdd1c005bc0 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd3859a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd300c0d60 con 0x7fdd301e6b60 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd3859a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3006cd80 con 0x7fdd30153320 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd300c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd300c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60564/0 (socket says 172.21.15.94:60564) 2024-03-06T07:19:57.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 -- 172.21.15.94:0/1878495879 learned_addr learned my addr 172.21.15.94:0/1878495879 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36b37640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd301e6b60 0x7fdd301e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd35b35640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd30153320 0x7fdd30154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 -- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd30153320 msgr2=0x7fdd30154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd30153320 0x7fdd30154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 -- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd301e6b60 msgr2=0x7fdd301e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd301e6b60 0x7fdd301e6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 -- 172.21.15.94:0/1878495879 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd301ea0d0 con 0x7fdd1c005bc0 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd35b35640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd30153320 0x7fdd30154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.483+0000 7fdd36336640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd300c54b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c005040 tx=0x7fdd2c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.484+0000 7fdd26ffd640 1 -- 172.21.15.94:0/1878495879 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c041020 con 0x7fdd1c005bc0 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.484+0000 7fdd26ffd640 1 -- 172.21.15.94:0/1878495879 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2c005480 con 0x7fdd1c005bc0 2024-03-06T07:19:57.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.484+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd301ea3b0 con 0x7fdd1c005bc0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.484+0000 7fdd26ffd640 1 -- 172.21.15.94:0/1878495879 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c007a30 con 0x7fdd1c005bc0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.484+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd301ea970 con 0x7fdd1c005bc0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.485+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd300c0b20 con 0x7fdd1c005bc0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.485+0000 7fdd26ffd640 1 -- 172.21.15.94:0/1878495879 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd2c044020 con 0x7fdd1c005bc0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.485+0000 7fdd26ffd640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd1406b4d0 0x7fdd1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.485+0000 7fdd26ffd640 1 -- 172.21.15.94:0/1878495879 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdd2c0abd20 con 0x7fdd1c005bc0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.485+0000 7fdd35b35640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd1406b4d0 0x7fdd1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.486+0000 7fdd35b35640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd1406b4d0 0x7fdd1406d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd3006e460 tx=0x7fdd20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.490+0000 7fdd26ffd640 1 -- 172.21.15.94:0/1878495879 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd300c0b20 con 0x7fdd1c005bc0 2024-03-06T07:19:57.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.657+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd3004f330 con 0x7fdd1406b4d0 2024-03-06T07:19:57.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.668+0000 7fdd26ffd640 1 -- 172.21.15.94:0/1878495879 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240115 (secure 0 0 0) 0x7fdd3004f330 con 0x7fdd1406b4d0 2024-03-06T07:19:57.676 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:57.677 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:19:57.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd1406b4d0 msgr2=0x7fdd1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd1406b4d0 0x7fdd1406d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd3006e460 tx=0x7fdd20008040 comp rx=0 tx=0).stop 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 msgr2=0x7fdd300c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd300c54b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c005040 tx=0x7fdd2c001120 comp rx=0 tx=0).stop 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 shutdown_connections 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd1406b4d0 0x7fdd1406d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd301e6b60 0x7fdd301e6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd30153320 0x7fdd30154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 --2- 172.21.15.94:0/1878495879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c005bc0 0x7fdd300c54b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 >> 172.21.15.94:0/1878495879 conn(0x7fdd300c6710 msgr2=0x7fdd3006db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 shutdown_connections 2024-03-06T07:19:57.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.671+0000 7fdd3859a640 1 -- 172.21.15.94:0/1878495879 wait complete. 2024-03-06T07:19:57.684 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-06T07:19:56.657592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233264,"kb_used_data":4612,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741456,"statfs":{"total":773094113280,"available":772855250944,"internally_reserved":0,"allocated":4722688,"data_stored":3194211,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.631411"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032038+0000","last_change":"2024-03-06T07:19:52.032038+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032038+0000","last_undegraded":"2024-03-06T07:19:52.032038+0000","last_fullsized":"2024-03-06T07:19:52.032038+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028473+0000","last_change":"2024-03-06T07:19:52.028473+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028473+0000","last_undegraded":"2024-03-06T07:19:52.028473+0000","last_fullsized":"2024-03-06T07:19:52.028473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028203+0000","last_change":"2024-03-06T07:19:52.028203+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028203+0000","last_undegraded":"2024-03-06T07:19:52.028203+0000","last_fullsized":"2024-03-06T07:19:52.028203+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030450+0000","last_change":"2024-03-06T07:19:52.030450+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030450+0000","last_undegraded":"2024-03-06T07:19:52.030450+0000","last_fullsized":"2024-03-06T07:19:52.030450+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026946+0000","last_change":"2024-03-06T07:19:52.026946+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026946+0000","last_undegraded":"2024-03-06T07:19:52.026946+0000","last_fullsized":"2024-03-06T07:19:52.026946+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031565+0000","last_change":"2024-03-06T07:19:52.031565+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031565+0000","last_undegraded":"2024-03-06T07:19:52.031565+0000","last_fullsized":"2024-03-06T07:19:52.031565+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030267+0000","last_change":"2024-03-06T07:19:52.030267+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030267+0000","last_undegraded":"2024-03-06T07:19:52.030267+0000","last_fullsized":"2024-03-06T07:19:52.030267+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026041+0000","last_change":"2024-03-06T07:19:52.026041+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026041+0000","last_undegraded":"2024-03-06T07:19:52.026041+0000","last_fullsized":"2024-03-06T07:19:52.026041+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031120+0000","last_change":"2024-03-06T07:19:52.031120+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031120+0000","last_undegraded":"2024-03-06T07:19:52.031120+0000","last_fullsized":"2024-03-06T07:19:52.031120+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031693+0000","last_change":"2024-03-06T07:19:52.031693+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031693+0000","last_undegraded":"2024-03-06T07:19:52.031693+0000","last_fullsized":"2024-03-06T07:19:52.031693+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.033049+0000","last_change":"2024-03-06T07:19:52.033049+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.033049+0000","last_undegraded":"2024-03-06T07:19:52.033049+0000","last_fullsized":"2024-03-06T07:19:52.033049+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028770+0000","last_change":"2024-03-06T07:19:52.028770+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028770+0000","last_undegraded":"2024-03-06T07:19:52.028770+0000","last_fullsized":"2024-03-06T07:19:52.028770+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026034+0000","last_change":"2024-03-06T07:19:52.026034+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026034+0000","last_undegraded":"2024-03-06T07:19:52.026034+0000","last_fullsized":"2024-03-06T07:19:52.026034+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031288+0000","last_change":"2024-03-06T07:19:52.031288+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031288+0000","last_undegraded":"2024-03-06T07:19:52.031288+0000","last_fullsized":"2024-03-06T07:19:52.031288+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029723+0000","last_change":"2024-03-06T07:19:52.029723+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029723+0000","last_undegraded":"2024-03-06T07:19:52.029723+0000","last_fullsized":"2024-03-06T07:19:52.029723+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029048+0000","last_change":"2024-03-06T07:19:52.029048+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029048+0000","last_undegraded":"2024-03-06T07:19:52.029048+0000","last_fullsized":"2024-03-06T07:19:52.029048+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026057+0000","last_change":"2024-03-06T07:19:52.026057+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026057+0000","last_undegraded":"2024-03-06T07:19:52.026057+0000","last_fullsized":"2024-03-06T07:19:52.026057+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029237+0000","last_change":"2024-03-06T07:19:52.029237+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029237+0000","last_undegraded":"2024-03-06T07:19:52.029237+0000","last_fullsized":"2024-03-06T07:19:52.029237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030769+0000","last_change":"2024-03-06T07:19:52.030769+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030769+0000","last_undegraded":"2024-03-06T07:19:52.030769+0000","last_fullsized":"2024-03-06T07:19:52.030769+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030293+0000","last_change":"2024-03-06T07:19:52.030293+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030293+0000","last_undegraded":"2024-03-06T07:19:52.030293+0000","last_fullsized":"2024-03-06T07:19:52.030293+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028184+0000","last_change":"2024-03-06T07:19:52.028184+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028184+0000","last_undegraded":"2024-03-06T07:19:52.028184+0000","last_fullsized":"2024-03-06T07:19:52.028184+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029572+0000","last_change":"2024-03-06T07:19:52.029572+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029572+0000","last_undegraded":"2024-03-06T07:19:52.029572+0000","last_fullsized":"2024-03-06T07:19:52.029572+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026130+0000","last_change":"2024-03-06T07:19:52.026130+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026130+0000","last_undegraded":"2024-03-06T07:19:52.026130+0000","last_fullsized":"2024-03-06T07:19:52.026130+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029231+0000","last_change":"2024-03-06T07:19:52.029231+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029231+0000","last_undegraded":"2024-03-06T07:19:52.029231+0000","last_fullsized":"2024-03-06T07:19:52.029231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030787+0000","last_change":"2024-03-06T07:19:52.030787+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030787+0000","last_undegraded":"2024-03-06T07:19:52.030787+0000","last_fullsized":"2024-03-06T07:19:52.030787+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031978+0000","last_change":"2024-03-06T07:19:52.031978+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031978+0000","last_undegraded":"2024-03-06T07:19:52.031978+0000","last_fullsized":"2024-03-06T07:19:52.031978+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.025813+0000","last_change":"2024-03-06T07:19:52.025813+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.025813+0000","last_undegraded":"2024-03-06T07:19:52.025813+0000","last_fullsized":"2024-03-06T07:19:52.025813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028643+0000","last_change":"2024-03-06T07:19:52.028643+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028643+0000","last_undegraded":"2024-03-06T07:19:52.028643+0000","last_fullsized":"2024-03-06T07:19:52.028643+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028228+0000","last_change":"2024-03-06T07:19:52.028228+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028228+0000","last_undegraded":"2024-03-06T07:19:52.028228+0000","last_fullsized":"2024-03-06T07:19:52.028228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032309+0000","last_change":"2024-03-06T07:19:52.032309+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032309+0000","last_undegraded":"2024-03-06T07:19:52.032309+0000","last_fullsized":"2024-03-06T07:19:52.032309+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030595+0000","last_change":"2024-03-06T07:19:52.030595+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030595+0000","last_undegraded":"2024-03-06T07:19:52.030595+0000","last_fullsized":"2024-03-06T07:19:52.030595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026017+0000","last_change":"2024-03-06T07:19:52.026017+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026017+0000","last_undegraded":"2024-03-06T07:19:52.026017+0000","last_fullsized":"2024-03-06T07:19:52.026017+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":223,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-06T07:19:52.033500+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:19:52.033500+0000","last_peered":"2024-03-06T07:19:52.033500+0000","last_clean":"2024-03-06T07:19:52.033500+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:19:52.033500+0000","last_undegraded":"2024-03-06T07:19:52.033500+0000","last_fullsized":"2024-03-06T07:19:52.033500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-03-06T07:19:57.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-03-06T07:19:57.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-03-06T07:19:57.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-03-06T07:19:57.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-03-06T07:19:57.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.032038+0000', 'last_change': '2024-03-06T07:19:52.032038+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.032038+0000', 'last_undegraded': '2024-03-06T07:19:52.032038+0000', 'last_fullsized': '2024-03-06T07:19:52.032038+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.028473+0000', 'last_change': '2024-03-06T07:19:52.028473+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.028473+0000', 'last_undegraded': '2024-03-06T07:19:52.028473+0000', 'last_fullsized': '2024-03-06T07:19:52.028473+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.028203+0000', 'last_change': '2024-03-06T07:19:52.028203+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.028203+0000', 'last_undegraded': '2024-03-06T07:19:52.028203+0000', 'last_fullsized': '2024-03-06T07:19:52.028203+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.030450+0000', 'last_change': '2024-03-06T07:19:52.030450+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.030450+0000', 'last_undegraded': '2024-03-06T07:19:52.030450+0000', 'last_fullsized': '2024-03-06T07:19:52.030450+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.026946+0000', 'last_change': '2024-03-06T07:19:52.026946+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.026946+0000', 'last_undegraded': '2024-03-06T07:19:52.026946+0000', 'last_fullsized': '2024-03-06T07:19:52.026946+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-03-06T07:19:57.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.696 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-06T07:19:57.696 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.031565+0000', 'last_change': '2024-03-06T07:19:52.031565+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.031565+0000', 'last_undegraded': '2024-03-06T07:19:52.031565+0000', 'last_fullsized': '2024-03-06T07:19:52.031565+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.030267+0000', 'last_change': '2024-03-06T07:19:52.030267+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.030267+0000', 'last_undegraded': '2024-03-06T07:19:52.030267+0000', 'last_fullsized': '2024-03-06T07:19:52.030267+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.026041+0000', 'last_change': '2024-03-06T07:19:52.026041+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.026041+0000', 'last_undegraded': '2024-03-06T07:19:52.026041+0000', 'last_fullsized': '2024-03-06T07:19:52.026041+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:19:57.697 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-03-06T07:19:57.698 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.698 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-06T07:19:57.698 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.031120+0000', 'last_change': '2024-03-06T07:19:52.031120+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.031120+0000', 'last_undegraded': '2024-03-06T07:19:52.031120+0000', 'last_fullsized': '2024-03-06T07:19:52.031120+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:19:57.698 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-03-06T07:19:57.699 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.699 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-06T07:19:57.699 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.031693+0000', 'last_change': '2024-03-06T07:19:52.031693+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.031693+0000', 'last_undegraded': '2024-03-06T07:19:52.031693+0000', 'last_fullsized': '2024-03-06T07:19:52.031693+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:19:57.699 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-03-06T07:19:57.699 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.699 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-06T07:19:57.699 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.033049+0000', 'last_change': '2024-03-06T07:19:52.033049+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.033049+0000', 'last_undegraded': '2024-03-06T07:19:52.033049+0000', 'last_fullsized': '2024-03-06T07:19:52.033049+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.028770+0000', 'last_change': '2024-03-06T07:19:52.028770+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.028770+0000', 'last_undegraded': '2024-03-06T07:19:52.028770+0000', 'last_fullsized': '2024-03-06T07:19:52.028770+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.026034+0000', 'last_change': '2024-03-06T07:19:52.026034+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.026034+0000', 'last_undegraded': '2024-03-06T07:19:52.026034+0000', 'last_fullsized': '2024-03-06T07:19:52.026034+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-06T07:19:57.700 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.031288+0000', 'last_change': '2024-03-06T07:19:52.031288+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.031288+0000', 'last_undegraded': '2024-03-06T07:19:52.031288+0000', 'last_fullsized': '2024-03-06T07:19:52.031288+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.029723+0000', 'last_change': '2024-03-06T07:19:52.029723+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.029723+0000', 'last_undegraded': '2024-03-06T07:19:52.029723+0000', 'last_fullsized': '2024-03-06T07:19:52.029723+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-06T07:19:57.701 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.029048+0000', 'last_change': '2024-03-06T07:19:52.029048+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.029048+0000', 'last_undegraded': '2024-03-06T07:19:52.029048+0000', 'last_fullsized': '2024-03-06T07:19:52.029048+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.026057+0000', 'last_change': '2024-03-06T07:19:52.026057+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.026057+0000', 'last_undegraded': '2024-03-06T07:19:52.026057+0000', 'last_fullsized': '2024-03-06T07:19:52.026057+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.029237+0000', 'last_change': '2024-03-06T07:19:52.029237+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.029237+0000', 'last_undegraded': '2024-03-06T07:19:52.029237+0000', 'last_fullsized': '2024-03-06T07:19:52.029237+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:19:57.702 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.030769+0000', 'last_change': '2024-03-06T07:19:52.030769+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.030769+0000', 'last_undegraded': '2024-03-06T07:19:52.030769+0000', 'last_fullsized': '2024-03-06T07:19:52.030769+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.030293+0000', 'last_change': '2024-03-06T07:19:52.030293+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.030293+0000', 'last_undegraded': '2024-03-06T07:19:52.030293+0000', 'last_fullsized': '2024-03-06T07:19:52.030293+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-03-06T07:19:57.703 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.028184+0000', 'last_change': '2024-03-06T07:19:52.028184+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.028184+0000', 'last_undegraded': '2024-03-06T07:19:52.028184+0000', 'last_fullsized': '2024-03-06T07:19:52.028184+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.029572+0000', 'last_change': '2024-03-06T07:19:52.029572+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.029572+0000', 'last_undegraded': '2024-03-06T07:19:52.029572+0000', 'last_fullsized': '2024-03-06T07:19:52.029572+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.026130+0000', 'last_change': '2024-03-06T07:19:52.026130+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.026130+0000', 'last_undegraded': '2024-03-06T07:19:52.026130+0000', 'last_fullsized': '2024-03-06T07:19:52.026130+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:19:57.704 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.029231+0000', 'last_change': '2024-03-06T07:19:52.029231+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.029231+0000', 'last_undegraded': '2024-03-06T07:19:52.029231+0000', 'last_fullsized': '2024-03-06T07:19:52.029231+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.030787+0000', 'last_change': '2024-03-06T07:19:52.030787+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.030787+0000', 'last_undegraded': '2024-03-06T07:19:52.030787+0000', 'last_fullsized': '2024-03-06T07:19:52.030787+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-06T07:19:57.705 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.031978+0000', 'last_change': '2024-03-06T07:19:52.031978+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.031978+0000', 'last_undegraded': '2024-03-06T07:19:52.031978+0000', 'last_fullsized': '2024-03-06T07:19:52.031978+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.025813+0000', 'last_change': '2024-03-06T07:19:52.025813+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.025813+0000', 'last_undegraded': '2024-03-06T07:19:52.025813+0000', 'last_fullsized': '2024-03-06T07:19:52.025813+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.028643+0000', 'last_change': '2024-03-06T07:19:52.028643+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.028643+0000', 'last_undegraded': '2024-03-06T07:19:52.028643+0000', 'last_fullsized': '2024-03-06T07:19:52.028643+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.706 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.028228+0000', 'last_change': '2024-03-06T07:19:52.028228+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.028228+0000', 'last_undegraded': '2024-03-06T07:19:52.028228+0000', 'last_fullsized': '2024-03-06T07:19:52.028228+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.032309+0000', 'last_change': '2024-03-06T07:19:52.032309+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.032309+0000', 'last_undegraded': '2024-03-06T07:19:52.032309+0000', 'last_fullsized': '2024-03-06T07:19:52.032309+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.030595+0000', 'last_change': '2024-03-06T07:19:52.030595+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.030595+0000', 'last_undegraded': '2024-03-06T07:19:52.030595+0000', 'last_fullsized': '2024-03-06T07:19:52.030595+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-03-06T07:19:57.707 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:19:53.023356+0000', 'last_change': '2024-03-06T07:19:53.023356+0000', 'last_active': '2024-03-06T07:19:53.023356+0000', 'last_peered': '2024-03-06T07:19:53.023356+0000', 'last_clean': '2024-03-06T07:19:53.023356+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:53.023356+0000', 'last_undegraded': '2024-03-06T07:19:53.023356+0000', 'last_fullsized': '2024-03-06T07:19:53.023356+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:53.023356+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:19:57.708 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-06T07:19:57.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 49, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:19:52.026017+0000', 'last_change': '2024-03-06T07:19:52.026017+0000', 'last_active': '2024-03-06T07:19:52.016094+0000', 'last_peered': '2024-03-06T07:19:52.016094+0000', 'last_clean': '2024-03-06T07:19:52.016094+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:19:52.026017+0000', 'last_undegraded': '2024-03-06T07:19:52.026017+0000', 'last_fullsized': '2024-03-06T07:19:52.026017+0000', 'mapping_epoch': 49, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 49, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'last_clean_scrub_stamp': '2024-03-06T07:19:52.016094+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:19:57.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:19:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.821+0000 7f8b420eb640 1 Processor -- start 2024-03-06T07:19:57.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.821+0000 7f8b420eb640 1 -- start start 2024-03-06T07:19:57.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b420eb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8b3c14f790 0x7f8b3c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b420eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8b3c0c1aa0 0x7f8b3c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b420eb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c0c28d0 con 0x7f8b3c0c5c60 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b420eb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0c2680 con 0x7f8b3c0c1aa0 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b420eb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0c2bb0 con 0x7f8b3c14f790 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8b3c0c1aa0 0x7f8b3c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8b3c0c5c60 0x7f8b3c155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47896/0 (socket says 172.21.15.94:47896) 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3bfff640 1 -- 172.21.15.94:0/2009167819 learned_addr learned my addr 172.21.15.94:0/2009167819 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3b7fe640 1 --2- 172.21.15.94:0/2009167819 >> v2:172.21.15.145:3300/0 conn(0x7f8b3c14f790 0x7f8b3c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3affd640 1 -- 172.21.15.94:0/2009167819 >> v2:172.21.15.145:3300/0 conn(0x7f8b3c14f790 msgr2=0x7f8b3c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3affd640 1 --2- 172.21.15.94:0/2009167819 >> v2:172.21.15.145:3300/0 conn(0x7f8b3c14f790 0x7f8b3c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3affd640 1 -- 172.21.15.94:0/2009167819 >> v1:172.21.15.94:6789/0 conn(0x7f8b3c0c5c60 legacy=0x7f8b3c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3a7fc640 1 -- 172.21.15.94:0/2009167819 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425930620 0 0) 0x7f8b3c0c28d0 con 0x7f8b3c0c5c60 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.822+0000 7f8b3affd640 1 -- 172.21.15.94:0/2009167819 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0c2e90 con 0x7f8b3c0c1aa0 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.823+0000 7f8b3affd640 1 --2- 172.21.15.94:0/2009167819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c0c1aa0 0x7f8b3c0c5720 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0c3170 tx=0x7f8b3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166b3fa8dc943a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.823+0000 7f8b3a7fc640 1 -- 172.21.15.94:0/2009167819 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30035020 con 0x7f8b3c0c1aa0 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.823+0000 7f8b3a7fc640 1 -- 172.21.15.94:0/2009167819 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30031830 con 0x7f8b3c0c1aa0 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.823+0000 7f8b3a7fc640 1 -- 172.21.15.94:0/2009167819 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30031c00 con 0x7f8b3c0c1aa0 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.823+0000 7f8b420eb640 1 -- 172.21.15.94:0/2009167819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c0c1aa0 msgr2=0x7f8b3c0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.823+0000 7f8b420eb640 1 --2- 172.21.15.94:0/2009167819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c0c1aa0 0x7f8b3c0c5720 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0c3170 tx=0x7f8b3002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.823+0000 7f8b420eb640 1 -- 172.21.15.94:0/2009167819 shutdown_connections 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.824+0000 7f8b420eb640 1 --2- 172.21.15.94:0/2009167819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c0c1aa0 0x7f8b3c0c5720 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.824+0000 7f8b420eb640 1 --2- 172.21.15.94:0/2009167819 >> v2:172.21.15.145:3300/0 conn(0x7f8b3c14f790 0x7f8b3c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.824+0000 7f8b420eb640 1 -- 172.21.15.94:0/2009167819 >> 172.21.15.94:0/2009167819 conn(0x7f8b3c0c6710 msgr2=0x7f8b3c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.824+0000 7f8b420eb640 1 -- 172.21.15.94:0/2009167819 shutdown_connections 2024-03-06T07:19:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.824+0000 7f8b420eb640 1 -- 172.21.15.94:0/2009167819 wait complete. 2024-03-06T07:19:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 Processor -- start 2024-03-06T07:19:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 -- start start 2024-03-06T07:19:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b3c0c1aa0 0x7f8b3c0bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b3c0c5c60 0x7f8b3c14ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c14f790 0x7f8b3c150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c06cc00 con 0x7f8b3c14f790 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0c0d60 con 0x7f8b3c0c5c60 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b420eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c06cd80 con 0x7f8b3c0c1aa0 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.825+0000 7f8b3bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c14f790 0x7f8b3c150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c14f790 0x7f8b3c150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60586/0 (socket says 172.21.15.94:60586) 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 -- 172.21.15.94:0/1376349525 learned_addr learned my addr 172.21.15.94:0/1376349525 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3affd640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b3c0c5c60 0x7f8b3c14ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3b7fe640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b3c0c1aa0 0x7f8b3c0bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 -- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b3c0c1aa0 msgr2=0x7f8b3c0bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b3c0c1aa0 0x7f8b3c0bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 -- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b3c0c5c60 msgr2=0x7f8b3c14ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b3c0c5c60 0x7f8b3c14ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 -- 172.21.15.94:0/1376349525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c155220 con 0x7f8b3c14f790 2024-03-06T07:19:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b3bfff640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c14f790 0x7f8b3c150490 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c7a0 tx=0x7f8b3400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b23fff640 1 -- 172.21.15.94:0/1376349525 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34018070 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b23fff640 1 -- 172.21.15.94:0/1376349525 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3400a640 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c1eb8a0 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b23fff640 1 -- 172.21.15.94:0/1376349525 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3400aab0 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.826+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c154f40 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.827+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c04f330 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.828+0000 7f8b23fff640 1 -- 172.21.15.94:0/1376349525 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b3401b070 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.828+0000 7f8b23fff640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b2806b4d0 0x7f8b2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.828+0000 7f8b23fff640 1 -- 172.21.15.94:0/1376349525 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8b34089920 con 0x7f8b3c14f790 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.828+0000 7f8b3b7fe640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b2806b4d0 0x7f8b2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.829+0000 7f8b3b7fe640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b2806b4d0 0x7f8b2806d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009aa0 tx=0x7f8b2c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:57.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:57.832+0000 7f8b23fff640 1 -- 172.21.15.94:0/1376349525 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b34057fc0 con 0x7f8b3c14f790 2024-03-06T07:19:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.072+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f8b3c0c1f40 con 0x7f8b3c14f790 2024-03-06T07:19:58.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.072+0000 7f8b23fff640 1 -- 172.21.15.94:0/1376349525 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1529 (secure 0 0 0) 0x7f8b3405c060 con 0x7f8b3c14f790 2024-03-06T07:19:58.074 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:58.074 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":124,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709482,"num_in_osds":8,"osd_in_since":1709709474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":238862336,"bytes_avail":772855250944,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":29,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"b","status":"up:active","gid":35084}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-06T07:19:42.654329+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:19:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b2806b4d0 msgr2=0x7f8b2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b2806b4d0 0x7f8b2806d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009aa0 tx=0x7f8b2c009190 comp rx=0 tx=0).stop 2024-03-06T07:19:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c14f790 msgr2=0x7f8b3c150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c14f790 0x7f8b3c150490 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c7a0 tx=0x7f8b3400cc60 comp rx=0 tx=0).stop 2024-03-06T07:19:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 shutdown_connections 2024-03-06T07:19:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b2806b4d0 0x7f8b2806d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b3c14f790 0x7f8b3c150490 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b3c0c5c60 0x7f8b3c14ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 --2- 172.21.15.94:0/1376349525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b3c0c1aa0 0x7f8b3c0bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 >> 172.21.15.94:0/1376349525 conn(0x7f8b3c0c6710 msgr2=0x7f8b3c072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:58.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 shutdown_connections 2024-03-06T07:19:58.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.074+0000 7f8b420eb640 1 -- 172.21.15.94:0/1376349525 wait complete. 2024-03-06T07:19:58.088 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:19:58.089 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:19:58.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.187+0000 7f98a72df640 1 Processor -- start 2024-03-06T07:19:58.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a72df640 1 -- start start 2024-03-06T07:19:58.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a72df640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f98a01527b0 0x7f98a0159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:58.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a72df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a00c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:58.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a72df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a00c7c60 con 0x7f98a00c7180 2024-03-06T07:19:58.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a72df640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a0159640 con 0x7f98a01527b0 2024-03-06T07:19:58.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a72df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c8060 con 0x7f98a00c88b0 2024-03-06T07:19:58.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a507b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f98a01527b0 0x7f98a0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:58.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a487a640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f98a00c7180 0x7f98a0155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47910/0 (socket says 172.21.15.94:47910) 2024-03-06T07:19:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.188+0000 7f98a487a640 1 -- 172.21.15.94:0/612078662 learned_addr learned my addr 172.21.15.94:0/612078662 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:19:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.189+0000 7f98a587c640 1 --2- 172.21.15.94:0/612078662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a00c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.189+0000 7f98a507b640 1 -- 172.21.15.94:0/612078662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 msgr2=0x7f98a00c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:19:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.189+0000 7f98a507b640 1 --2- 172.21.15.94:0/612078662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a00c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.189+0000 7f98a507b640 1 -- 172.21.15.94:0/612078662 >> v1:172.21.15.94:6789/0 conn(0x7f98a00c7180 legacy=0x7f98a0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.189+0000 7f98a507b640 1 -- 172.21.15.94:0/612078662 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a00c76a0 con 0x7f98a01527b0 2024-03-06T07:19:58.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.189+0000 7f9897fff640 1 -- 172.21.15.94:0/612078662 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274439342 0 0) 0x7f989c001110 con 0x7f98a00c7180 2024-03-06T07:19:58.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a507b640 1 --2- 172.21.15.94:0/612078662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a0159100 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f98900097a0 tx=0x7f989002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8049ea7ae88f093 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:58.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f9897fff640 1 -- 172.21.15.94:0/612078662 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900315a0 con 0x7f98a01527b0 2024-03-06T07:19:58.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f9897fff640 1 -- 172.21.15.94:0/612078662 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9890037070 con 0x7f98a01527b0 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f9897fff640 1 -- 172.21.15.94:0/612078662 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890031960 con 0x7f98a01527b0 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 -- 172.21.15.94:0/612078662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 msgr2=0x7f98a0159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 --2- 172.21.15.94:0/612078662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a0159100 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f98900097a0 tx=0x7f989002b840 comp rx=0 tx=0).stop 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 -- 172.21.15.94:0/612078662 shutdown_connections 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 --2- 172.21.15.94:0/612078662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a00c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 --2- 172.21.15.94:0/612078662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a0159100 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 -- 172.21.15.94:0/612078662 >> 172.21.15.94:0/612078662 conn(0x7f98a00ac560 msgr2=0x7f98a00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 -- 172.21.15.94:0/612078662 shutdown_connections 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.190+0000 7f98a72df640 1 -- 172.21.15.94:0/612078662 wait complete. 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.191+0000 7f98a72df640 1 Processor -- start 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.191+0000 7f98a72df640 1 -- start start 2024-03-06T07:19:58.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a72df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98a00c7180 0x7f98a0154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a72df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a01eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a72df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a01564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a72df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0155600 con 0x7f98a01527b0 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a72df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a0155320 con 0x7f98a00c7180 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a72df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a00c8d60 con 0x7f98a00c88b0 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a01564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a01564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60606/0 (socket says 172.21.15.94:60606) 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 -- 172.21.15.94:0/644051052 learned_addr learned my addr 172.21.15.94:0/644051052 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a507b640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98a00c7180 0x7f98a0154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a487a640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a01eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 -- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 msgr2=0x7f98a01eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a01eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 -- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98a00c7180 msgr2=0x7f98a0154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98a00c7180 0x7f98a0154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.192+0000 7f98a587c640 1 -- 172.21.15.94:0/644051052 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a01eaba0 con 0x7f98a01527b0 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.193+0000 7f98a587c640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a01564f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f989800efc0 tx=0x7f989800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.193+0000 7f9895ffb640 1 -- 172.21.15.94:0/644051052 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898018020 con 0x7f98a01527b0 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.193+0000 7f9895ffb640 1 -- 172.21.15.94:0/644051052 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9898014480 con 0x7f98a01527b0 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.193+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a0157c60 con 0x7f98a01527b0 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.193+0000 7f9895ffb640 1 -- 172.21.15.94:0/644051052 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9898014870 con 0x7f98a01527b0 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.193+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a01eae10 con 0x7f98a01527b0 2024-03-06T07:19:58.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.194+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a0158180 con 0x7f98a01527b0 2024-03-06T07:19:58.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.194+0000 7f9895ffb640 1 -- 172.21.15.94:0/644051052 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9898005d10 con 0x7f98a01527b0 2024-03-06T07:19:58.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.195+0000 7f9895ffb640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988406b4d0 0x7f988406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:19:58.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.195+0000 7f9895ffb640 1 -- 172.21.15.94:0/644051052 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f989808ac80 con 0x7f98a01527b0 2024-03-06T07:19:58.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.195+0000 7f98a507b640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988406b4d0 0x7f988406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:19:58.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.195+0000 7f98a507b640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988406b4d0 0x7f988406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9890036000 tx=0x7f989003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:19:58.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.199+0000 7f9895ffb640 1 -- 172.21.15.94:0/644051052 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98a0158180 con 0x7f98a01527b0 2024-03-06T07:19:58.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.366+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98a01eb010 con 0x7f988406b4d0 2024-03-06T07:19:58.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.372+0000 7f9895ffb640 1 -- 172.21.15.94:0/644051052 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240115 (secure 0 0 0) 0x7f98a01eb010 con 0x7f988406b4d0 2024-03-06T07:19:58.377 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:19:58.378 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:19:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988406b4d0 msgr2=0x7f988406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988406b4d0 0x7f988406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9890036000 tx=0x7f989003d040 comp rx=0 tx=0).stop 2024-03-06T07:19:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 msgr2=0x7f98a01564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:19:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a01564f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f989800efc0 tx=0x7f989800c490 comp rx=0 tx=0).stop 2024-03-06T07:19:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 shutdown_connections 2024-03-06T07:19:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988406b4d0 0x7f988406d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98a01527b0 0x7f98a01564f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98a00c88b0 0x7f98a01eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 --2- 172.21.15.94:0/644051052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98a00c7180 0x7f98a0154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:19:58.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 >> 172.21.15.94:0/644051052 conn(0x7f98a00ac560 msgr2=0x7f98a0158e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:19:58.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 shutdown_connections 2024-03-06T07:19:58.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:19:58.375+0000 7f98a72df640 1 -- 172.21.15.94:0/644051052 wait complete. 2024-03-06T07:19:58.388 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-03-06T07:19:56.657592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233264,"kb_used_data":4612,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754741456,"statfs":{"total":773094113280,"available":772855250944,"internally_reserved":0,"allocated":4722688,"data_stored":3194211,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.631411"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032038+0000","last_change":"2024-03-06T07:19:52.032038+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032038+0000","last_undegraded":"2024-03-06T07:19:52.032038+0000","last_fullsized":"2024-03-06T07:19:52.032038+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028473+0000","last_change":"2024-03-06T07:19:52.028473+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028473+0000","last_undegraded":"2024-03-06T07:19:52.028473+0000","last_fullsized":"2024-03-06T07:19:52.028473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028203+0000","last_change":"2024-03-06T07:19:52.028203+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028203+0000","last_undegraded":"2024-03-06T07:19:52.028203+0000","last_fullsized":"2024-03-06T07:19:52.028203+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030450+0000","last_change":"2024-03-06T07:19:52.030450+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030450+0000","last_undegraded":"2024-03-06T07:19:52.030450+0000","last_fullsized":"2024-03-06T07:19:52.030450+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026946+0000","last_change":"2024-03-06T07:19:52.026946+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026946+0000","last_undegraded":"2024-03-06T07:19:52.026946+0000","last_fullsized":"2024-03-06T07:19:52.026946+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031565+0000","last_change":"2024-03-06T07:19:52.031565+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031565+0000","last_undegraded":"2024-03-06T07:19:52.031565+0000","last_fullsized":"2024-03-06T07:19:52.031565+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030267+0000","last_change":"2024-03-06T07:19:52.030267+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030267+0000","last_undegraded":"2024-03-06T07:19:52.030267+0000","last_fullsized":"2024-03-06T07:19:52.030267+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026041+0000","last_change":"2024-03-06T07:19:52.026041+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026041+0000","last_undegraded":"2024-03-06T07:19:52.026041+0000","last_fullsized":"2024-03-06T07:19:52.026041+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031120+0000","last_change":"2024-03-06T07:19:52.031120+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031120+0000","last_undegraded":"2024-03-06T07:19:52.031120+0000","last_fullsized":"2024-03-06T07:19:52.031120+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031693+0000","last_change":"2024-03-06T07:19:52.031693+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031693+0000","last_undegraded":"2024-03-06T07:19:52.031693+0000","last_fullsized":"2024-03-06T07:19:52.031693+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.033049+0000","last_change":"2024-03-06T07:19:52.033049+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.033049+0000","last_undegraded":"2024-03-06T07:19:52.033049+0000","last_fullsized":"2024-03-06T07:19:52.033049+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028770+0000","last_change":"2024-03-06T07:19:52.028770+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028770+0000","last_undegraded":"2024-03-06T07:19:52.028770+0000","last_fullsized":"2024-03-06T07:19:52.028770+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026034+0000","last_change":"2024-03-06T07:19:52.026034+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026034+0000","last_undegraded":"2024-03-06T07:19:52.026034+0000","last_fullsized":"2024-03-06T07:19:52.026034+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031288+0000","last_change":"2024-03-06T07:19:52.031288+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031288+0000","last_undegraded":"2024-03-06T07:19:52.031288+0000","last_fullsized":"2024-03-06T07:19:52.031288+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029723+0000","last_change":"2024-03-06T07:19:52.029723+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029723+0000","last_undegraded":"2024-03-06T07:19:52.029723+0000","last_fullsized":"2024-03-06T07:19:52.029723+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029048+0000","last_change":"2024-03-06T07:19:52.029048+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029048+0000","last_undegraded":"2024-03-06T07:19:52.029048+0000","last_fullsized":"2024-03-06T07:19:52.029048+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026057+0000","last_change":"2024-03-06T07:19:52.026057+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026057+0000","last_undegraded":"2024-03-06T07:19:52.026057+0000","last_fullsized":"2024-03-06T07:19:52.026057+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029237+0000","last_change":"2024-03-06T07:19:52.029237+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029237+0000","last_undegraded":"2024-03-06T07:19:52.029237+0000","last_fullsized":"2024-03-06T07:19:52.029237+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030769+0000","last_change":"2024-03-06T07:19:52.030769+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030769+0000","last_undegraded":"2024-03-06T07:19:52.030769+0000","last_fullsized":"2024-03-06T07:19:52.030769+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030293+0000","last_change":"2024-03-06T07:19:52.030293+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030293+0000","last_undegraded":"2024-03-06T07:19:52.030293+0000","last_fullsized":"2024-03-06T07:19:52.030293+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028184+0000","last_change":"2024-03-06T07:19:52.028184+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028184+0000","last_undegraded":"2024-03-06T07:19:52.028184+0000","last_fullsized":"2024-03-06T07:19:52.028184+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029572+0000","last_change":"2024-03-06T07:19:52.029572+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029572+0000","last_undegraded":"2024-03-06T07:19:52.029572+0000","last_fullsized":"2024-03-06T07:19:52.029572+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026130+0000","last_change":"2024-03-06T07:19:52.026130+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026130+0000","last_undegraded":"2024-03-06T07:19:52.026130+0000","last_fullsized":"2024-03-06T07:19:52.026130+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.029231+0000","last_change":"2024-03-06T07:19:52.029231+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.029231+0000","last_undegraded":"2024-03-06T07:19:52.029231+0000","last_fullsized":"2024-03-06T07:19:52.029231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030787+0000","last_change":"2024-03-06T07:19:52.030787+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030787+0000","last_undegraded":"2024-03-06T07:19:52.030787+0000","last_fullsized":"2024-03-06T07:19:52.030787+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.031978+0000","last_change":"2024-03-06T07:19:52.031978+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.031978+0000","last_undegraded":"2024-03-06T07:19:52.031978+0000","last_fullsized":"2024-03-06T07:19:52.031978+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.025813+0000","last_change":"2024-03-06T07:19:52.025813+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.025813+0000","last_undegraded":"2024-03-06T07:19:52.025813+0000","last_fullsized":"2024-03-06T07:19:52.025813+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028643+0000","last_change":"2024-03-06T07:19:52.028643+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028643+0000","last_undegraded":"2024-03-06T07:19:52.028643+0000","last_fullsized":"2024-03-06T07:19:52.028643+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.028228+0000","last_change":"2024-03-06T07:19:52.028228+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.028228+0000","last_undegraded":"2024-03-06T07:19:52.028228+0000","last_fullsized":"2024-03-06T07:19:52.028228+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.032309+0000","last_change":"2024-03-06T07:19:52.032309+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.032309+0000","last_undegraded":"2024-03-06T07:19:52.032309+0000","last_fullsized":"2024-03-06T07:19:52.032309+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.030595+0000","last_change":"2024-03-06T07:19:52.030595+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.030595+0000","last_undegraded":"2024-03-06T07:19:52.030595+0000","last_fullsized":"2024-03-06T07:19:52.030595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:19:53.023356+0000","last_change":"2024-03-06T07:19:53.023356+0000","last_active":"2024-03-06T07:19:53.023356+0000","last_peered":"2024-03-06T07:19:53.023356+0000","last_clean":"2024-03-06T07:19:53.023356+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:53.023356+0000","last_undegraded":"2024-03-06T07:19:53.023356+0000","last_fullsized":"2024-03-06T07:19:53.023356+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":49,"state":"creating+peering","last_fresh":"2024-03-06T07:19:52.026017+0000","last_change":"2024-03-06T07:19:52.026017+0000","last_active":"2024-03-06T07:19:52.016094+0000","last_peered":"2024-03-06T07:19:52.016094+0000","last_clean":"2024-03-06T07:19:52.016094+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:19:52.026017+0000","last_undegraded":"2024-03-06T07:19:52.026017+0000","last_fullsized":"2024-03-06T07:19:52.026017+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":223,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-06T07:19:52.033500+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:19:52.033500+0000","last_peered":"2024-03-06T07:19:52.033500+0000","last_clean":"2024-03-06T07:19:52.033500+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:19:52.033500+0000","last_undegraded":"2024-03-06T07:19:52.033500+0000","last_fullsized":"2024-03-06T07:19:52.033500+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":581632,"data_stored":255565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":569344,"data_stored":249356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":839724,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:19:59.294 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:19:59.293+0000 7f8d655e1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:20:01.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.511+0000 7f0753fe2640 1 Processor -- start 2024-03-06T07:20:01.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.512+0000 7f0753fe2640 1 -- start start 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.512+0000 7f0753fe2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f074c14f360 0x7f074c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.512+0000 7f0753fe2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f074c158b60 0x7f074c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.512+0000 7f0753fe2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074c0c73d0 con 0x7f074c0c88c0 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.512+0000 7f0753fe2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c0c8dd0 con 0x7f074c14f360 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.512+0000 7f0753fe2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c0be2e0 con 0x7f074c158b60 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.512+0000 7f0751d7e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f074c14f360 0x7f074c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f075157d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f074c0c88c0 0x7f074c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57324/0 (socket says 172.21.15.94:57324) 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f075157d640 1 -- 172.21.15.94:0/3406272670 learned_addr learned my addr 172.21.15.94:0/3406272670 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f075257f640 1 --2- 172.21.15.94:0/3406272670 >> v2:172.21.15.145:3300/0 conn(0x7f074c158b60 0x7f074c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f0751d7e640 1 -- 172.21.15.94:0/3406272670 >> v2:172.21.15.145:3300/0 conn(0x7f074c158b60 msgr2=0x7f074c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:01.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f0751d7e640 1 --2- 172.21.15.94:0/3406272670 >> v2:172.21.15.145:3300/0 conn(0x7f074c158b60 0x7f074c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f0751d7e640 1 -- 172.21.15.94:0/3406272670 >> v1:172.21.15.94:6789/0 conn(0x7f074c0c88c0 legacy=0x7f074c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f0750d7c640 1 -- 172.21.15.94:0/3406272670 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370633884 0 0) 0x7f074c0c73d0 con 0x7f074c0c88c0 2024-03-06T07:20:01.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.513+0000 7f0751d7e640 1 -- 172.21.15.94:0/3406272670 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c0c7670 con 0x7f074c14f360 2024-03-06T07:20:01.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.514+0000 7f0751d7e640 1 --2- 172.21.15.94:0/3406272670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c14f360 0x7f074c155c10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f074c152d60 tx=0x7f073c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a45aaae5e7c5732 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:01.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.514+0000 7f0750d7c640 1 -- 172.21.15.94:0/3406272670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c0315a0 con 0x7f074c14f360 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.514+0000 7f0750d7c640 1 -- 172.21.15.94:0/3406272670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c037070 con 0x7f074c14f360 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.514+0000 7f0750d7c640 1 -- 172.21.15.94:0/3406272670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c031960 con 0x7f074c14f360 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.514+0000 7f0753fe2640 1 -- 172.21.15.94:0/3406272670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c14f360 msgr2=0x7f074c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.514+0000 7f0753fe2640 1 --2- 172.21.15.94:0/3406272670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c14f360 0x7f074c155c10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f074c152d60 tx=0x7f073c02b840 comp rx=0 tx=0).stop 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.515+0000 7f0753fe2640 1 -- 172.21.15.94:0/3406272670 shutdown_connections 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.515+0000 7f0753fe2640 1 --2- 172.21.15.94:0/3406272670 >> v2:172.21.15.145:3300/0 conn(0x7f074c158b60 0x7f074c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.515+0000 7f0753fe2640 1 --2- 172.21.15.94:0/3406272670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c14f360 0x7f074c155c10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.515+0000 7f0753fe2640 1 -- 172.21.15.94:0/3406272670 >> 172.21.15.94:0/3406272670 conn(0x7f074c0ac570 msgr2=0x7f074c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.515+0000 7f0753fe2640 1 -- 172.21.15.94:0/3406272670 shutdown_connections 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.515+0000 7f0753fe2640 1 -- 172.21.15.94:0/3406272670 wait complete. 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.516+0000 7f0753fe2640 1 Processor -- start 2024-03-06T07:20:01.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.516+0000 7f0753fe2640 1 -- start start 2024-03-06T07:20:01.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.516+0000 7f0753fe2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c0c88c0 0x7f074c0c0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.516+0000 7f0753fe2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f074c14f360 0x7f074c152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.516+0000 7f0753fe2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074c158b60 0x7f074c152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.516+0000 7f0753fe2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0c5a50 con 0x7f074c0c88c0 2024-03-06T07:20:01.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.516+0000 7f0753fe2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0c7c30 con 0x7f074c14f360 2024-03-06T07:20:01.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0753fe2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0c5bd0 con 0x7f074c158b60 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c0c88c0 0x7f074c0c0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c0c88c0 0x7f074c0c0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51632/0 (socket says 172.21.15.94:51632) 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 -- 172.21.15.94:0/2073194159 learned_addr learned my addr 172.21.15.94:0/2073194159 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f075157d640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f074c14f360 0x7f074c152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f075257f640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074c158b60 0x7f074c152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 -- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074c158b60 msgr2=0x7f074c152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074c158b60 0x7f074c152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 -- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f074c14f360 msgr2=0x7f074c152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f074c14f360 0x7f074c152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 -- 172.21.15.94:0/2073194159 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c0bf4b0 con 0x7f074c0c88c0 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f075157d640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f074c14f360 0x7f074c152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0751d7e640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c0c88c0 0x7f074c0c0e30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f073c009890 tx=0x7f073c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:01.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f07427fc640 1 -- 172.21.15.94:0/2073194159 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c032630 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f07427fc640 1 -- 172.21.15.94:0/2073194159 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c0327d0 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074c0bf790 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f07427fc640 1 -- 172.21.15.94:0/2073194159 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c032bc0 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.517+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c0adab0 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.519+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c152db0 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.519+0000 7f07427fc640 1 -- 172.21.15.94:0/2073194159 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f073c03b3f0 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.519+0000 7f07427fc640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073006b4d0 0x7f073006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.519+0000 7f07427fc640 1 -- 172.21.15.94:0/2073194159 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f073c02f030 con 0x7f074c0c88c0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.519+0000 7f075157d640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073006b4d0 0x7f073006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.520+0000 7f075157d640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073006b4d0 0x7f073006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f074c0cb540 tx=0x7f0748007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:01.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.526+0000 7f07427fc640 1 -- 172.21.15.94:0/2073194159 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f074c152db0 con 0x7f074c0c88c0 2024-03-06T07:20:01.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.695+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f074c0af820 con 0x7f073006b4d0 2024-03-06T07:20:01.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.701+0000 7f07427fc640 1 -- 172.21.15.94:0/2073194159 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252029 (secure 0 0 0) 0x7f074c0af820 con 0x7f073006b4d0 2024-03-06T07:20:01.707 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:01.710 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073006b4d0 msgr2=0x7f073006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073006b4d0 0x7f073006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f074c0cb540 tx=0x7f0748007de0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c0c88c0 msgr2=0x7f074c0c0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c0c88c0 0x7f074c0c0e30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f073c009890 tx=0x7f073c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 shutdown_connections 2024-03-06T07:20:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073006b4d0 0x7f073006d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074c158b60 0x7f074c152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f074c14f360 0x7f074c152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 --2- 172.21.15.94:0/2073194159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f074c0c88c0 0x7f074c0c0e30 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 >> 172.21.15.94:0/2073194159 conn(0x7f074c0ac570 msgr2=0x7f074c14d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 shutdown_connections 2024-03-06T07:20:01.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.704+0000 7f0753fe2640 1 -- 172.21.15.94:0/2073194159 wait complete. 2024-03-06T07:20:01.717 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-03-06T07:20:00.658296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237852,"kb_used_data":5024,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736868,"statfs":{"total":773094113280,"available":772850552832,"internally_reserved":0,"allocated":5144576,"data_stored":3394284,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.629435"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.033126+0000","last_change":"2024-03-06T07:19:54.052304+0000","last_active":"2024-03-06T07:19:55.033126+0000","last_peered":"2024-03-06T07:19:55.033126+0000","last_clean":"2024-03-06T07:19:55.033126+0000","last_became_active":"2024-03-06T07:19:54.052048+0000","last_became_peered":"2024-03-06T07:19:54.052048+0000","last_unstale":"2024-03-06T07:19:55.033126+0000","last_undegraded":"2024-03-06T07:19:55.033126+0000","last_fullsized":"2024-03-06T07:19:55.033126+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:24.129824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406711+0000","last_change":"2024-03-06T07:19:54.062911+0000","last_active":"2024-03-06T07:19:55.406711+0000","last_peered":"2024-03-06T07:19:55.406711+0000","last_clean":"2024-03-06T07:19:55.406711+0000","last_became_active":"2024-03-06T07:19:54.062566+0000","last_became_peered":"2024-03-06T07:19:54.062566+0000","last_unstale":"2024-03-06T07:19:55.406711+0000","last_undegraded":"2024-03-06T07:19:55.406711+0000","last_fullsized":"2024-03-06T07:19:55.406711+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407142+0000","last_change":"2024-03-06T07:19:54.051868+0000","last_active":"2024-03-06T07:19:55.407142+0000","last_peered":"2024-03-06T07:19:55.407142+0000","last_clean":"2024-03-06T07:19:55.407142+0000","last_became_active":"2024-03-06T07:19:54.051611+0000","last_became_peered":"2024-03-06T07:19:54.051611+0000","last_unstale":"2024-03-06T07:19:55.407142+0000","last_undegraded":"2024-03-06T07:19:55.407142+0000","last_fullsized":"2024-03-06T07:19:55.407142+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406765+0000","last_change":"2024-03-06T07:19:54.055553+0000","last_active":"2024-03-06T07:19:55.406765+0000","last_peered":"2024-03-06T07:19:55.406765+0000","last_clean":"2024-03-06T07:19:55.406765+0000","last_became_active":"2024-03-06T07:19:54.055271+0000","last_became_peered":"2024-03-06T07:19:54.055271+0000","last_unstale":"2024-03-06T07:19:55.406765+0000","last_undegraded":"2024-03-06T07:19:55.406765+0000","last_fullsized":"2024-03-06T07:19:55.406765+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034818+0000","last_change":"2024-03-06T07:19:54.046161+0000","last_active":"2024-03-06T07:19:55.034818+0000","last_peered":"2024-03-06T07:19:55.034818+0000","last_clean":"2024-03-06T07:19:55.034818+0000","last_became_active":"2024-03-06T07:19:54.045875+0000","last_became_peered":"2024-03-06T07:19:54.045875+0000","last_unstale":"2024-03-06T07:19:55.034818+0000","last_undegraded":"2024-03-06T07:19:55.034818+0000","last_fullsized":"2024-03-06T07:19:55.034818+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:50.526448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032799+0000","last_change":"2024-03-06T07:19:54.047960+0000","last_active":"2024-03-06T07:19:55.032799+0000","last_peered":"2024-03-06T07:19:55.032799+0000","last_clean":"2024-03-06T07:19:55.032799+0000","last_became_active":"2024-03-06T07:19:54.047638+0000","last_became_peered":"2024-03-06T07:19:54.047638+0000","last_unstale":"2024-03-06T07:19:55.032799+0000","last_undegraded":"2024-03-06T07:19:55.032799+0000","last_fullsized":"2024-03-06T07:19:55.032799+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407650+0000","last_change":"2024-03-06T07:19:54.048726+0000","last_active":"2024-03-06T07:19:55.407650+0000","last_peered":"2024-03-06T07:19:55.407650+0000","last_clean":"2024-03-06T07:19:55.407650+0000","last_became_active":"2024-03-06T07:19:54.048377+0000","last_became_peered":"2024-03-06T07:19:54.048377+0000","last_unstale":"2024-03-06T07:19:55.407650+0000","last_undegraded":"2024-03-06T07:19:55.407650+0000","last_fullsized":"2024-03-06T07:19:55.407650+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:57.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406455+0000","last_change":"2024-03-06T07:19:54.059007+0000","last_active":"2024-03-06T07:19:55.406455+0000","last_peered":"2024-03-06T07:19:55.406455+0000","last_clean":"2024-03-06T07:19:55.406455+0000","last_became_active":"2024-03-06T07:19:54.058694+0000","last_became_peered":"2024-03-06T07:19:54.058694+0000","last_unstale":"2024-03-06T07:19:55.406455+0000","last_undegraded":"2024-03-06T07:19:55.406455+0000","last_fullsized":"2024-03-06T07:19:55.406455+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:50.508506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407095+0000","last_change":"2024-03-06T07:19:54.057769+0000","last_active":"2024-03-06T07:19:55.407095+0000","last_peered":"2024-03-06T07:19:55.407095+0000","last_clean":"2024-03-06T07:19:55.407095+0000","last_became_active":"2024-03-06T07:19:54.057496+0000","last_became_peered":"2024-03-06T07:19:54.057496+0000","last_unstale":"2024-03-06T07:19:55.407095+0000","last_undegraded":"2024-03-06T07:19:55.407095+0000","last_fullsized":"2024-03-06T07:19:55.407095+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:51.719701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.033058+0000","last_change":"2024-03-06T07:19:54.050005+0000","last_active":"2024-03-06T07:19:55.033058+0000","last_peered":"2024-03-06T07:19:55.033058+0000","last_clean":"2024-03-06T07:19:55.033058+0000","last_became_active":"2024-03-06T07:19:54.049750+0000","last_became_peered":"2024-03-06T07:19:54.049750+0000","last_unstale":"2024-03-06T07:19:55.033058+0000","last_undegraded":"2024-03-06T07:19:55.033058+0000","last_fullsized":"2024-03-06T07:19:55.033058+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:37.609871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406641+0000","last_change":"2024-03-06T07:19:54.052367+0000","last_active":"2024-03-06T07:19:55.406641+0000","last_peered":"2024-03-06T07:19:55.406641+0000","last_clean":"2024-03-06T07:19:55.406641+0000","last_became_active":"2024-03-06T07:19:54.052119+0000","last_became_peered":"2024-03-06T07:19:54.052119+0000","last_unstale":"2024-03-06T07:19:55.406641+0000","last_undegraded":"2024-03-06T07:19:55.406641+0000","last_fullsized":"2024-03-06T07:19:55.406641+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.678839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407356+0000","last_change":"2024-03-06T07:19:54.040294+0000","last_active":"2024-03-06T07:19:55.407356+0000","last_peered":"2024-03-06T07:19:55.407356+0000","last_clean":"2024-03-06T07:19:55.407356+0000","last_became_active":"2024-03-06T07:19:54.040048+0000","last_became_peered":"2024-03-06T07:19:54.040048+0000","last_unstale":"2024-03-06T07:19:55.407356+0000","last_undegraded":"2024-03-06T07:19:55.407356+0000","last_fullsized":"2024-03-06T07:19:55.407356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:41.393636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032898+0000","last_change":"2024-03-06T07:19:54.056367+0000","last_active":"2024-03-06T07:19:55.032898+0000","last_peered":"2024-03-06T07:19:55.032898+0000","last_clean":"2024-03-06T07:19:55.032898+0000","last_became_active":"2024-03-06T07:19:54.056125+0000","last_became_peered":"2024-03-06T07:19:54.056125+0000","last_unstale":"2024-03-06T07:19:55.032898+0000","last_undegraded":"2024-03-06T07:19:55.032898+0000","last_fullsized":"2024-03-06T07:19:55.032898+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406479+0000","last_change":"2024-03-06T07:19:54.051456+0000","last_active":"2024-03-06T07:19:55.406479+0000","last_peered":"2024-03-06T07:19:55.406479+0000","last_clean":"2024-03-06T07:19:55.406479+0000","last_became_active":"2024-03-06T07:19:54.051193+0000","last_became_peered":"2024-03-06T07:19:54.051193+0000","last_unstale":"2024-03-06T07:19:55.406479+0000","last_undegraded":"2024-03-06T07:19:55.406479+0000","last_fullsized":"2024-03-06T07:19:55.406479+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406559+0000","last_change":"2024-03-06T07:19:54.049011+0000","last_active":"2024-03-06T07:19:55.406559+0000","last_peered":"2024-03-06T07:19:55.406559+0000","last_clean":"2024-03-06T07:19:55.406559+0000","last_became_active":"2024-03-06T07:19:54.048738+0000","last_became_peered":"2024-03-06T07:19:54.048738+0000","last_unstale":"2024-03-06T07:19:55.406559+0000","last_undegraded":"2024-03-06T07:19:55.406559+0000","last_fullsized":"2024-03-06T07:19:55.406559+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:36.367136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032712+0000","last_change":"2024-03-06T07:19:54.049339+0000","last_active":"2024-03-06T07:19:55.032712+0000","last_peered":"2024-03-06T07:19:55.032712+0000","last_clean":"2024-03-06T07:19:55.032712+0000","last_became_active":"2024-03-06T07:19:54.049059+0000","last_became_peered":"2024-03-06T07:19:54.049059+0000","last_unstale":"2024-03-06T07:19:55.032712+0000","last_undegraded":"2024-03-06T07:19:55.032712+0000","last_fullsized":"2024-03-06T07:19:55.032712+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407377+0000","last_change":"2024-03-06T07:19:54.058699+0000","last_active":"2024-03-06T07:19:55.407377+0000","last_peered":"2024-03-06T07:19:55.407377+0000","last_clean":"2024-03-06T07:19:55.407377+0000","last_became_active":"2024-03-06T07:19:54.056798+0000","last_became_peered":"2024-03-06T07:19:54.056798+0000","last_unstale":"2024-03-06T07:19:55.407377+0000","last_undegraded":"2024-03-06T07:19:55.407377+0000","last_fullsized":"2024-03-06T07:19:55.407377+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406812+0000","last_change":"2024-03-06T07:19:54.062261+0000","last_active":"2024-03-06T07:19:55.406812+0000","last_peered":"2024-03-06T07:19:55.406812+0000","last_clean":"2024-03-06T07:19:55.406812+0000","last_became_active":"2024-03-06T07:19:54.061853+0000","last_became_peered":"2024-03-06T07:19:54.061853+0000","last_unstale":"2024-03-06T07:19:55.406812+0000","last_undegraded":"2024-03-06T07:19:55.406812+0000","last_fullsized":"2024-03-06T07:19:55.406812+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407269+0000","last_change":"2024-03-06T07:19:54.054672+0000","last_active":"2024-03-06T07:19:55.407269+0000","last_peered":"2024-03-06T07:19:55.407269+0000","last_clean":"2024-03-06T07:19:55.407269+0000","last_became_active":"2024-03-06T07:19:54.054301+0000","last_became_peered":"2024-03-06T07:19:54.054301+0000","last_unstale":"2024-03-06T07:19:55.407269+0000","last_undegraded":"2024-03-06T07:19:55.407269+0000","last_fullsized":"2024-03-06T07:19:55.407269+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:05.772166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034707+0000","last_change":"2024-03-06T07:19:54.051669+0000","last_active":"2024-03-06T07:19:55.034707+0000","last_peered":"2024-03-06T07:19:55.034707+0000","last_clean":"2024-03-06T07:19:55.034707+0000","last_became_active":"2024-03-06T07:19:54.051312+0000","last_became_peered":"2024-03-06T07:19:54.051312+0000","last_unstale":"2024-03-06T07:19:55.034707+0000","last_undegraded":"2024-03-06T07:19:55.034707+0000","last_fullsized":"2024-03-06T07:19:55.034707+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034794+0000","last_change":"2024-03-06T07:19:54.047677+0000","last_active":"2024-03-06T07:19:55.034794+0000","last_peered":"2024-03-06T07:19:55.034794+0000","last_clean":"2024-03-06T07:19:55.034794+0000","last_became_active":"2024-03-06T07:19:54.045393+0000","last_became_peered":"2024-03-06T07:19:54.045393+0000","last_unstale":"2024-03-06T07:19:55.034794+0000","last_undegraded":"2024-03-06T07:19:55.034794+0000","last_fullsized":"2024-03-06T07:19:55.034794+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406837+0000","last_change":"2024-03-06T07:19:54.058376+0000","last_active":"2024-03-06T07:19:55.406837+0000","last_peered":"2024-03-06T07:19:55.406837+0000","last_clean":"2024-03-06T07:19:55.406837+0000","last_became_active":"2024-03-06T07:19:54.058151+0000","last_became_peered":"2024-03-06T07:19:54.058151+0000","last_unstale":"2024-03-06T07:19:55.406837+0000","last_undegraded":"2024-03-06T07:19:55.406837+0000","last_fullsized":"2024-03-06T07:19:55.406837+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:10.287168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407313+0000","last_change":"2024-03-06T07:19:54.046473+0000","last_active":"2024-03-06T07:19:55.407313+0000","last_peered":"2024-03-06T07:19:55.407313+0000","last_clean":"2024-03-06T07:19:55.407313+0000","last_became_active":"2024-03-06T07:19:54.045977+0000","last_became_peered":"2024-03-06T07:19:54.045977+0000","last_unstale":"2024-03-06T07:19:55.407313+0000","last_undegraded":"2024-03-06T07:19:55.407313+0000","last_fullsized":"2024-03-06T07:19:55.407313+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:28.437704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406223+0000","last_change":"2024-03-06T07:19:54.050649+0000","last_active":"2024-03-06T07:19:55.406223+0000","last_peered":"2024-03-06T07:19:55.406223+0000","last_clean":"2024-03-06T07:19:55.406223+0000","last_became_active":"2024-03-06T07:19:54.050392+0000","last_became_peered":"2024-03-06T07:19:54.050392+0000","last_unstale":"2024-03-06T07:19:55.406223+0000","last_undegraded":"2024-03-06T07:19:55.406223+0000","last_fullsized":"2024-03-06T07:19:55.406223+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407519+0000","last_change":"2024-03-06T07:19:54.046865+0000","last_active":"2024-03-06T07:19:55.407519+0000","last_peered":"2024-03-06T07:19:55.407519+0000","last_clean":"2024-03-06T07:19:55.407519+0000","last_became_active":"2024-03-06T07:19:54.046545+0000","last_became_peered":"2024-03-06T07:19:54.046545+0000","last_unstale":"2024-03-06T07:19:55.407519+0000","last_undegraded":"2024-03-06T07:19:55.407519+0000","last_fullsized":"2024-03-06T07:19:55.407519+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:09.982751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407607+0000","last_change":"2024-03-06T07:19:54.042151+0000","last_active":"2024-03-06T07:19:55.407607+0000","last_peered":"2024-03-06T07:19:55.407607+0000","last_clean":"2024-03-06T07:19:55.407607+0000","last_became_active":"2024-03-06T07:19:54.041796+0000","last_became_peered":"2024-03-06T07:19:54.041796+0000","last_unstale":"2024-03-06T07:19:55.407607+0000","last_undegraded":"2024-03-06T07:19:55.407607+0000","last_fullsized":"2024-03-06T07:19:55.407607+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:27.407999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.033133+0000","last_change":"2024-03-06T07:19:54.051161+0000","last_active":"2024-03-06T07:19:55.033133+0000","last_peered":"2024-03-06T07:19:55.033133+0000","last_clean":"2024-03-06T07:19:55.033133+0000","last_became_active":"2024-03-06T07:19:54.050893+0000","last_became_peered":"2024-03-06T07:19:54.050893+0000","last_unstale":"2024-03-06T07:19:55.033133+0000","last_undegraded":"2024-03-06T07:19:55.033133+0000","last_fullsized":"2024-03-06T07:19:55.033133+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:28.711633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034729+0000","last_change":"2024-03-06T07:19:54.040041+0000","last_active":"2024-03-06T07:19:55.034729+0000","last_peered":"2024-03-06T07:19:55.034729+0000","last_clean":"2024-03-06T07:19:55.034729+0000","last_became_active":"2024-03-06T07:19:54.039730+0000","last_became_peered":"2024-03-06T07:19:54.039730+0000","last_unstale":"2024-03-06T07:19:55.034729+0000","last_undegraded":"2024-03-06T07:19:55.034729+0000","last_fullsized":"2024-03-06T07:19:55.034729+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407813+0000","last_change":"2024-03-06T07:19:54.055165+0000","last_active":"2024-03-06T07:19:55.407813+0000","last_peered":"2024-03-06T07:19:55.407813+0000","last_clean":"2024-03-06T07:19:55.407813+0000","last_became_active":"2024-03-06T07:19:54.054908+0000","last_became_peered":"2024-03-06T07:19:54.054908+0000","last_unstale":"2024-03-06T07:19:55.407813+0000","last_undegraded":"2024-03-06T07:19:55.407813+0000","last_fullsized":"2024-03-06T07:19:55.407813+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:02.499776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406590+0000","last_change":"2024-03-06T07:19:54.049805+0000","last_active":"2024-03-06T07:19:55.406590+0000","last_peered":"2024-03-06T07:19:55.406590+0000","last_clean":"2024-03-06T07:19:55.406590+0000","last_became_active":"2024-03-06T07:19:54.049562+0000","last_became_peered":"2024-03-06T07:19:54.049562+0000","last_unstale":"2024-03-06T07:19:55.406590+0000","last_undegraded":"2024-03-06T07:19:55.406590+0000","last_fullsized":"2024-03-06T07:19:55.406590+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:21.645621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407816+0000","last_change":"2024-03-06T07:19:54.052763+0000","last_active":"2024-03-06T07:19:55.407816+0000","last_peered":"2024-03-06T07:19:55.407816+0000","last_clean":"2024-03-06T07:19:55.407816+0000","last_became_active":"2024-03-06T07:19:54.052472+0000","last_became_peered":"2024-03-06T07:19:54.052472+0000","last_unstale":"2024-03-06T07:19:55.407816+0000","last_undegraded":"2024-03-06T07:19:55.407816+0000","last_fullsized":"2024-03-06T07:19:55.407816+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:07.485843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203088+0000","last_change":"2024-03-06T07:19:54.059383+0000","last_active":"2024-03-06T07:19:55.203088+0000","last_peered":"2024-03-06T07:19:55.203088+0000","last_clean":"2024-03-06T07:19:55.203088+0000","last_became_active":"2024-03-06T07:19:54.059123+0000","last_became_peered":"2024-03-06T07:19:54.059123+0000","last_unstale":"2024-03-06T07:19:55.203088+0000","last_undegraded":"2024-03-06T07:19:55.203088+0000","last_fullsized":"2024-03-06T07:19:55.203088+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406402+0000","last_change":"2024-03-06T07:19:54.060233+0000","last_active":"2024-03-06T07:19:55.406402+0000","last_peered":"2024-03-06T07:19:55.406402+0000","last_clean":"2024-03-06T07:19:55.406402+0000","last_became_active":"2024-03-06T07:19:54.059901+0000","last_became_peered":"2024-03-06T07:19:54.059901+0000","last_unstale":"2024-03-06T07:19:55.406402+0000","last_undegraded":"2024-03-06T07:19:55.406402+0000","last_fullsized":"2024-03-06T07:19:55.406402+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:09.921962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034768+0000","last_change":"2024-03-06T07:19:53.042382+0000","last_active":"2024-03-06T07:19:55.034768+0000","last_peered":"2024-03-06T07:19:55.034768+0000","last_clean":"2024-03-06T07:19:55.034768+0000","last_became_active":"2024-03-06T07:19:53.041334+0000","last_became_peered":"2024-03-06T07:19:53.041334+0000","last_unstale":"2024-03-06T07:19:55.034768+0000","last_undegraded":"2024-03-06T07:19:55.034768+0000","last_fullsized":"2024-03-06T07:19:55.034768+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203112+0000","last_change":"2024-03-06T07:19:54.057284+0000","last_active":"2024-03-06T07:19:55.203112+0000","last_peered":"2024-03-06T07:19:55.203112+0000","last_clean":"2024-03-06T07:19:55.203112+0000","last_became_active":"2024-03-06T07:19:54.056999+0000","last_became_peered":"2024-03-06T07:19:54.056999+0000","last_unstale":"2024-03-06T07:19:55.203112+0000","last_undegraded":"2024-03-06T07:19:55.203112+0000","last_fullsized":"2024-03-06T07:19:55.203112+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032964+0000","last_change":"2024-03-06T07:19:53.048675+0000","last_active":"2024-03-06T07:19:55.032964+0000","last_peered":"2024-03-06T07:19:55.032964+0000","last_clean":"2024-03-06T07:19:55.032964+0000","last_became_active":"2024-03-06T07:19:53.048416+0000","last_became_peered":"2024-03-06T07:19:53.048416+0000","last_unstale":"2024-03-06T07:19:55.032964+0000","last_undegraded":"2024-03-06T07:19:55.032964+0000","last_fullsized":"2024-03-06T07:19:55.032964+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034407+0000","last_change":"2024-03-06T07:19:54.052199+0000","last_active":"2024-03-06T07:19:55.034407+0000","last_peered":"2024-03-06T07:19:55.034407+0000","last_clean":"2024-03-06T07:19:55.034407+0000","last_became_active":"2024-03-06T07:19:54.051923+0000","last_became_peered":"2024-03-06T07:19:54.051923+0000","last_unstale":"2024-03-06T07:19:55.034407+0000","last_undegraded":"2024-03-06T07:19:55.034407+0000","last_fullsized":"2024-03-06T07:19:55.034407+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.033088+0000","last_change":"2024-03-06T07:19:53.044281+0000","last_active":"2024-03-06T07:19:55.033088+0000","last_peered":"2024-03-06T07:19:55.033088+0000","last_clean":"2024-03-06T07:19:55.033088+0000","last_became_active":"2024-03-06T07:19:53.043841+0000","last_became_peered":"2024-03-06T07:19:53.043841+0000","last_unstale":"2024-03-06T07:19:55.033088+0000","last_undegraded":"2024-03-06T07:19:55.033088+0000","last_fullsized":"2024-03-06T07:19:55.033088+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406499+0000","last_change":"2024-03-06T07:19:54.039974+0000","last_active":"2024-03-06T07:19:55.406499+0000","last_peered":"2024-03-06T07:19:55.406499+0000","last_clean":"2024-03-06T07:19:55.406499+0000","last_became_active":"2024-03-06T07:19:54.039675+0000","last_became_peered":"2024-03-06T07:19:54.039675+0000","last_unstale":"2024-03-06T07:19:55.406499+0000","last_undegraded":"2024-03-06T07:19:55.406499+0000","last_fullsized":"2024-03-06T07:19:55.406499+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:53.089205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203292+0000","last_change":"2024-03-06T07:19:53.051640+0000","last_active":"2024-03-06T07:19:55.203292+0000","last_peered":"2024-03-06T07:19:55.203292+0000","last_clean":"2024-03-06T07:19:55.203292+0000","last_became_active":"2024-03-06T07:19:53.051429+0000","last_became_peered":"2024-03-06T07:19:53.051429+0000","last_unstale":"2024-03-06T07:19:55.203292+0000","last_undegraded":"2024-03-06T07:19:55.203292+0000","last_fullsized":"2024-03-06T07:19:55.203292+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:09.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.035336+0000","last_change":"2024-03-06T07:19:54.054949+0000","last_active":"2024-03-06T07:19:55.035336+0000","last_peered":"2024-03-06T07:19:55.035336+0000","last_clean":"2024-03-06T07:19:55.035336+0000","last_became_active":"2024-03-06T07:19:54.054661+0000","last_became_peered":"2024-03-06T07:19:54.054661+0000","last_unstale":"2024-03-06T07:19:55.035336+0000","last_undegraded":"2024-03-06T07:19:55.035336+0000","last_fullsized":"2024-03-06T07:19:55.035336+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:57.129016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407773+0000","last_change":"2024-03-06T07:19:53.042003+0000","last_active":"2024-03-06T07:19:55.407773+0000","last_peered":"2024-03-06T07:19:55.407773+0000","last_clean":"2024-03-06T07:19:55.407773+0000","last_became_active":"2024-03-06T07:19:53.041728+0000","last_became_peered":"2024-03-06T07:19:53.041728+0000","last_unstale":"2024-03-06T07:19:55.407773+0000","last_undegraded":"2024-03-06T07:19:55.407773+0000","last_fullsized":"2024-03-06T07:19:55.407773+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.964097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406723+0000","last_change":"2024-03-06T07:19:54.061284+0000","last_active":"2024-03-06T07:19:55.406723+0000","last_peered":"2024-03-06T07:19:55.406723+0000","last_clean":"2024-03-06T07:19:55.406723+0000","last_became_active":"2024-03-06T07:19:54.061017+0000","last_became_peered":"2024-03-06T07:19:54.061017+0000","last_unstale":"2024-03-06T07:19:55.406723+0000","last_undegraded":"2024-03-06T07:19:55.406723+0000","last_fullsized":"2024-03-06T07:19:55.406723+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034561+0000","last_change":"2024-03-06T07:19:53.041819+0000","last_active":"2024-03-06T07:19:55.034561+0000","last_peered":"2024-03-06T07:19:55.034561+0000","last_clean":"2024-03-06T07:19:55.034561+0000","last_became_active":"2024-03-06T07:19:53.041542+0000","last_became_peered":"2024-03-06T07:19:53.041542+0000","last_unstale":"2024-03-06T07:19:55.034561+0000","last_undegraded":"2024-03-06T07:19:55.034561+0000","last_fullsized":"2024-03-06T07:19:55.034561+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407731+0000","last_change":"2024-03-06T07:19:54.039491+0000","last_active":"2024-03-06T07:19:55.407731+0000","last_peered":"2024-03-06T07:19:55.407731+0000","last_clean":"2024-03-06T07:19:55.407731+0000","last_became_active":"2024-03-06T07:19:54.039202+0000","last_became_peered":"2024-03-06T07:19:54.039202+0000","last_unstale":"2024-03-06T07:19:55.407731+0000","last_undegraded":"2024-03-06T07:19:55.407731+0000","last_fullsized":"2024-03-06T07:19:55.407731+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:58.292241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034362+0000","last_change":"2024-03-06T07:19:53.043210+0000","last_active":"2024-03-06T07:19:55.034362+0000","last_peered":"2024-03-06T07:19:55.034362+0000","last_clean":"2024-03-06T07:19:55.034362+0000","last_became_active":"2024-03-06T07:19:53.042951+0000","last_became_peered":"2024-03-06T07:19:53.042951+0000","last_unstale":"2024-03-06T07:19:55.034362+0000","last_undegraded":"2024-03-06T07:19:55.034362+0000","last_fullsized":"2024-03-06T07:19:55.034362+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406655+0000","last_change":"2024-03-06T07:19:54.049884+0000","last_active":"2024-03-06T07:19:55.406655+0000","last_peered":"2024-03-06T07:19:55.406655+0000","last_clean":"2024-03-06T07:19:55.406655+0000","last_became_active":"2024-03-06T07:19:54.049597+0000","last_became_peered":"2024-03-06T07:19:54.049597+0000","last_unstale":"2024-03-06T07:19:55.406655+0000","last_undegraded":"2024-03-06T07:19:55.406655+0000","last_fullsized":"2024-03-06T07:19:55.406655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406214+0000","last_change":"2024-03-06T07:19:53.043198+0000","last_active":"2024-03-06T07:19:55.406214+0000","last_peered":"2024-03-06T07:19:55.406214+0000","last_clean":"2024-03-06T07:19:55.406214+0000","last_became_active":"2024-03-06T07:19:53.042939+0000","last_became_peered":"2024-03-06T07:19:53.042939+0000","last_unstale":"2024-03-06T07:19:55.406214+0000","last_undegraded":"2024-03-06T07:19:55.406214+0000","last_fullsized":"2024-03-06T07:19:55.406214+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034310+0000","last_change":"2024-03-06T07:19:54.045809+0000","last_active":"2024-03-06T07:19:55.034310+0000","last_peered":"2024-03-06T07:19:55.034310+0000","last_clean":"2024-03-06T07:19:55.034310+0000","last_became_active":"2024-03-06T07:19:54.040263+0000","last_became_peered":"2024-03-06T07:19:54.040263+0000","last_unstale":"2024-03-06T07:19:55.034310+0000","last_undegraded":"2024-03-06T07:19:55.034310+0000","last_fullsized":"2024-03-06T07:19:55.034310+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:20.728340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406634+0000","last_change":"2024-03-06T07:19:53.047864+0000","last_active":"2024-03-06T07:19:55.406634+0000","last_peered":"2024-03-06T07:19:55.406634+0000","last_clean":"2024-03-06T07:19:55.406634+0000","last_became_active":"2024-03-06T07:19:53.047607+0000","last_became_peered":"2024-03-06T07:19:53.047607+0000","last_unstale":"2024-03-06T07:19:55.406634+0000","last_undegraded":"2024-03-06T07:19:55.406634+0000","last_fullsized":"2024-03-06T07:19:55.406634+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.148872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203084+0000","last_change":"2024-03-06T07:19:54.056898+0000","last_active":"2024-03-06T07:19:55.203084+0000","last_peered":"2024-03-06T07:19:55.203084+0000","last_clean":"2024-03-06T07:19:55.203084+0000","last_became_active":"2024-03-06T07:19:54.056522+0000","last_became_peered":"2024-03-06T07:19:54.056522+0000","last_unstale":"2024-03-06T07:19:55.203084+0000","last_undegraded":"2024-03-06T07:19:55.203084+0000","last_fullsized":"2024-03-06T07:19:55.203084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.035245+0000","last_change":"2024-03-06T07:19:53.043065+0000","last_active":"2024-03-06T07:19:55.035245+0000","last_peered":"2024-03-06T07:19:55.035245+0000","last_clean":"2024-03-06T07:19:55.035245+0000","last_became_active":"2024-03-06T07:19:53.042756+0000","last_became_peered":"2024-03-06T07:19:53.042756+0000","last_unstale":"2024-03-06T07:19:55.035245+0000","last_undegraded":"2024-03-06T07:19:55.035245+0000","last_fullsized":"2024-03-06T07:19:55.035245+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.338932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406614+0000","last_change":"2024-03-06T07:19:54.057647+0000","last_active":"2024-03-06T07:19:55.406614+0000","last_peered":"2024-03-06T07:19:55.406614+0000","last_clean":"2024-03-06T07:19:55.406614+0000","last_became_active":"2024-03-06T07:19:54.057396+0000","last_became_peered":"2024-03-06T07:19:54.057396+0000","last_unstale":"2024-03-06T07:19:55.406614+0000","last_undegraded":"2024-03-06T07:19:55.406614+0000","last_fullsized":"2024-03-06T07:19:55.406614+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406783+0000","last_change":"2024-03-06T07:19:53.046359+0000","last_active":"2024-03-06T07:19:55.406783+0000","last_peered":"2024-03-06T07:19:55.406783+0000","last_clean":"2024-03-06T07:19:55.406783+0000","last_became_active":"2024-03-06T07:19:53.046065+0000","last_became_peered":"2024-03-06T07:19:53.046065+0000","last_unstale":"2024-03-06T07:19:55.406783+0000","last_undegraded":"2024-03-06T07:19:55.406783+0000","last_fullsized":"2024-03-06T07:19:55.406783+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:08.146267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.035000+0000","last_change":"2024-03-06T07:19:54.041140+0000","last_active":"2024-03-06T07:19:55.035000+0000","last_peered":"2024-03-06T07:19:55.035000+0000","last_clean":"2024-03-06T07:19:55.035000+0000","last_became_active":"2024-03-06T07:19:54.040877+0000","last_became_peered":"2024-03-06T07:19:54.040877+0000","last_unstale":"2024-03-06T07:19:55.035000+0000","last_undegraded":"2024-03-06T07:19:55.035000+0000","last_fullsized":"2024-03-06T07:19:55.035000+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:29.971359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407406+0000","last_change":"2024-03-06T07:19:53.045210+0000","last_active":"2024-03-06T07:19:55.407406+0000","last_peered":"2024-03-06T07:19:55.407406+0000","last_clean":"2024-03-06T07:19:55.407406+0000","last_became_active":"2024-03-06T07:19:53.044950+0000","last_became_peered":"2024-03-06T07:19:53.044950+0000","last_unstale":"2024-03-06T07:19:55.407406+0000","last_undegraded":"2024-03-06T07:19:55.407406+0000","last_fullsized":"2024-03-06T07:19:55.407406+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:53.407226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203222+0000","last_change":"2024-03-06T07:19:54.058718+0000","last_active":"2024-03-06T07:19:55.203222+0000","last_peered":"2024-03-06T07:19:55.203222+0000","last_clean":"2024-03-06T07:19:55.203222+0000","last_became_active":"2024-03-06T07:19:54.058415+0000","last_became_peered":"2024-03-06T07:19:54.058415+0000","last_unstale":"2024-03-06T07:19:55.203222+0000","last_undegraded":"2024-03-06T07:19:55.203222+0000","last_fullsized":"2024-03-06T07:19:55.203222+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407845+0000","last_change":"2024-03-06T07:19:53.046555+0000","last_active":"2024-03-06T07:19:55.407845+0000","last_peered":"2024-03-06T07:19:55.407845+0000","last_clean":"2024-03-06T07:19:55.407845+0000","last_became_active":"2024-03-06T07:19:53.046298+0000","last_became_peered":"2024-03-06T07:19:53.046298+0000","last_unstale":"2024-03-06T07:19:55.407845+0000","last_undegraded":"2024-03-06T07:19:55.407845+0000","last_fullsized":"2024-03-06T07:19:55.407845+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.243021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407397+0000","last_change":"2024-03-06T07:19:54.047654+0000","last_active":"2024-03-06T07:19:55.407397+0000","last_peered":"2024-03-06T07:19:55.407397+0000","last_clean":"2024-03-06T07:19:55.407397+0000","last_became_active":"2024-03-06T07:19:54.047339+0000","last_became_peered":"2024-03-06T07:19:54.047339+0000","last_unstale":"2024-03-06T07:19:55.407397+0000","last_undegraded":"2024-03-06T07:19:55.407397+0000","last_fullsized":"2024-03-06T07:19:55.407397+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:35.272829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034455+0000","last_change":"2024-03-06T07:19:53.040132+0000","last_active":"2024-03-06T07:19:55.034455+0000","last_peered":"2024-03-06T07:19:55.034455+0000","last_clean":"2024-03-06T07:19:55.034455+0000","last_became_active":"2024-03-06T07:19:53.039809+0000","last_became_peered":"2024-03-06T07:19:53.039809+0000","last_unstale":"2024-03-06T07:19:55.034455+0000","last_undegraded":"2024-03-06T07:19:55.034455+0000","last_fullsized":"2024-03-06T07:19:55.034455+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407653+0000","last_change":"2024-03-06T07:19:54.050847+0000","last_active":"2024-03-06T07:19:55.407653+0000","last_peered":"2024-03-06T07:19:55.407653+0000","last_clean":"2024-03-06T07:19:55.407653+0000","last_became_active":"2024-03-06T07:19:54.050378+0000","last_became_peered":"2024-03-06T07:19:54.050378+0000","last_unstale":"2024-03-06T07:19:55.407653+0000","last_undegraded":"2024-03-06T07:19:55.407653+0000","last_fullsized":"2024-03-06T07:19:55.407653+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406622+0000","last_change":"2024-03-06T07:19:53.045139+0000","last_active":"2024-03-06T07:19:55.406622+0000","last_peered":"2024-03-06T07:19:55.406622+0000","last_clean":"2024-03-06T07:19:55.406622+0000","last_became_active":"2024-03-06T07:19:53.044903+0000","last_became_peered":"2024-03-06T07:19:53.044903+0000","last_unstale":"2024-03-06T07:19:55.406622+0000","last_undegraded":"2024-03-06T07:19:55.406622+0000","last_fullsized":"2024-03-06T07:19:55.406622+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:56.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407174+0000","last_change":"2024-03-06T07:19:54.059322+0000","last_active":"2024-03-06T07:19:55.407174+0000","last_peered":"2024-03-06T07:19:55.407174+0000","last_clean":"2024-03-06T07:19:55.407174+0000","last_became_active":"2024-03-06T07:19:54.059092+0000","last_became_peered":"2024-03-06T07:19:54.059092+0000","last_unstale":"2024-03-06T07:19:55.407174+0000","last_undegraded":"2024-03-06T07:19:55.407174+0000","last_fullsized":"2024-03-06T07:19:55.407174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":22,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406639+0000","last_change":"2024-03-06T07:19:53.046815+0000","last_active":"2024-03-06T07:19:55.406639+0000","last_peered":"2024-03-06T07:19:55.406639+0000","last_clean":"2024-03-06T07:19:55.406639+0000","last_became_active":"2024-03-06T07:19:53.043687+0000","last_became_peered":"2024-03-06T07:19:53.043687+0000","last_unstale":"2024-03-06T07:19:55.406639+0000","last_undegraded":"2024-03-06T07:19:55.406639+0000","last_fullsized":"2024-03-06T07:19:55.406639+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:44.465449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406590+0000","last_change":"2024-03-06T07:19:54.057521+0000","last_active":"2024-03-06T07:19:55.406590+0000","last_peered":"2024-03-06T07:19:55.406590+0000","last_clean":"2024-03-06T07:19:55.406590+0000","last_became_active":"2024-03-06T07:19:54.057264+0000","last_became_peered":"2024-03-06T07:19:54.057264+0000","last_unstale":"2024-03-06T07:19:55.406590+0000","last_undegraded":"2024-03-06T07:19:55.406590+0000","last_fullsized":"2024-03-06T07:19:55.406590+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407477+0000","last_change":"2024-03-06T07:19:53.051189+0000","last_active":"2024-03-06T07:19:55.407477+0000","last_peered":"2024-03-06T07:19:55.407477+0000","last_clean":"2024-03-06T07:19:55.407477+0000","last_became_active":"2024-03-06T07:19:53.050897+0000","last_became_peered":"2024-03-06T07:19:53.050897+0000","last_unstale":"2024-03-06T07:19:55.407477+0000","last_undegraded":"2024-03-06T07:19:55.407477+0000","last_fullsized":"2024-03-06T07:19:55.407477+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032696+0000","last_change":"2024-03-06T07:19:54.052438+0000","last_active":"2024-03-06T07:19:55.032696+0000","last_peered":"2024-03-06T07:19:55.032696+0000","last_clean":"2024-03-06T07:19:55.032696+0000","last_became_active":"2024-03-06T07:19:54.052185+0000","last_became_peered":"2024-03-06T07:19:54.052185+0000","last_unstale":"2024-03-06T07:19:55.032696+0000","last_undegraded":"2024-03-06T07:19:55.032696+0000","last_fullsized":"2024-03-06T07:19:55.032696+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406931+0000","last_change":"2024-03-06T07:19:53.046903+0000","last_active":"2024-03-06T07:19:55.406931+0000","last_peered":"2024-03-06T07:19:55.406931+0000","last_clean":"2024-03-06T07:19:55.406931+0000","last_became_active":"2024-03-06T07:19:53.046639+0000","last_became_peered":"2024-03-06T07:19:53.046639+0000","last_unstale":"2024-03-06T07:19:55.406931+0000","last_undegraded":"2024-03-06T07:19:55.406931+0000","last_fullsized":"2024-03-06T07:19:55.406931+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:23.209022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034491+0000","last_change":"2024-03-06T07:19:54.053795+0000","last_active":"2024-03-06T07:19:55.034491+0000","last_peered":"2024-03-06T07:19:55.034491+0000","last_clean":"2024-03-06T07:19:55.034491+0000","last_became_active":"2024-03-06T07:19:54.053519+0000","last_became_peered":"2024-03-06T07:19:54.053519+0000","last_unstale":"2024-03-06T07:19:55.034491+0000","last_undegraded":"2024-03-06T07:19:55.034491+0000","last_fullsized":"2024-03-06T07:19:55.034491+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:20.250283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034790+0000","last_change":"2024-03-06T07:19:53.046151+0000","last_active":"2024-03-06T07:19:55.034790+0000","last_peered":"2024-03-06T07:19:55.034790+0000","last_clean":"2024-03-06T07:19:55.034790+0000","last_became_active":"2024-03-06T07:19:53.045797+0000","last_became_peered":"2024-03-06T07:19:53.045797+0000","last_unstale":"2024-03-06T07:19:55.034790+0000","last_undegraded":"2024-03-06T07:19:55.034790+0000","last_fullsized":"2024-03-06T07:19:55.034790+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032931+0000","last_change":"2024-03-06T07:19:54.055324+0000","last_active":"2024-03-06T07:19:55.032931+0000","last_peered":"2024-03-06T07:19:55.032931+0000","last_clean":"2024-03-06T07:19:55.032931+0000","last_became_active":"2024-03-06T07:19:54.055081+0000","last_became_peered":"2024-03-06T07:19:54.055081+0000","last_unstale":"2024-03-06T07:19:55.032931+0000","last_undegraded":"2024-03-06T07:19:55.032931+0000","last_fullsized":"2024-03-06T07:19:55.032931+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:57.518725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034757+0000","last_change":"2024-03-06T07:19:53.048898+0000","last_active":"2024-03-06T07:19:55.034757+0000","last_peered":"2024-03-06T07:19:55.034757+0000","last_clean":"2024-03-06T07:19:55.034757+0000","last_became_active":"2024-03-06T07:19:53.048613+0000","last_became_peered":"2024-03-06T07:19:53.048613+0000","last_unstale":"2024-03-06T07:19:55.034757+0000","last_undegraded":"2024-03-06T07:19:55.034757+0000","last_fullsized":"2024-03-06T07:19:55.034757+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203264+0000","last_change":"2024-03-06T07:19:54.056310+0000","last_active":"2024-03-06T07:19:55.203264+0000","last_peered":"2024-03-06T07:19:55.203264+0000","last_clean":"2024-03-06T07:19:55.203264+0000","last_became_active":"2024-03-06T07:19:54.056039+0000","last_became_peered":"2024-03-06T07:19:54.056039+0000","last_unstale":"2024-03-06T07:19:55.203264+0000","last_undegraded":"2024-03-06T07:19:55.203264+0000","last_fullsized":"2024-03-06T07:19:55.203264+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032962+0000","last_change":"2024-03-06T07:19:53.043949+0000","last_active":"2024-03-06T07:19:55.032962+0000","last_peered":"2024-03-06T07:19:55.032962+0000","last_clean":"2024-03-06T07:19:55.032962+0000","last_became_active":"2024-03-06T07:19:53.043639+0000","last_became_peered":"2024-03-06T07:19:53.043639+0000","last_unstale":"2024-03-06T07:19:55.032962+0000","last_undegraded":"2024-03-06T07:19:55.032962+0000","last_fullsized":"2024-03-06T07:19:55.032962+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.035011+0000","last_change":"2024-03-06T07:19:54.040887+0000","last_active":"2024-03-06T07:19:55.035011+0000","last_peered":"2024-03-06T07:19:55.035011+0000","last_clean":"2024-03-06T07:19:55.035011+0000","last_became_active":"2024-03-06T07:19:54.040632+0000","last_became_peered":"2024-03-06T07:19:54.040632+0000","last_unstale":"2024-03-06T07:19:55.035011+0000","last_undegraded":"2024-03-06T07:19:55.035011+0000","last_fullsized":"2024-03-06T07:19:55.035011+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406885+0000","last_change":"2024-03-06T07:19:53.044330+0000","last_active":"2024-03-06T07:19:55.406885+0000","last_peered":"2024-03-06T07:19:55.406885+0000","last_clean":"2024-03-06T07:19:55.406885+0000","last_became_active":"2024-03-06T07:19:53.044066+0000","last_became_peered":"2024-03-06T07:19:53.044066+0000","last_unstale":"2024-03-06T07:19:55.406885+0000","last_undegraded":"2024-03-06T07:19:55.406885+0000","last_fullsized":"2024-03-06T07:19:55.406885+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:19.476961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406689+0000","last_change":"2024-03-06T07:19:54.049940+0000","last_active":"2024-03-06T07:19:55.406689+0000","last_peered":"2024-03-06T07:19:55.406689+0000","last_clean":"2024-03-06T07:19:55.406689+0000","last_became_active":"2024-03-06T07:19:54.049481+0000","last_became_peered":"2024-03-06T07:19:54.049481+0000","last_unstale":"2024-03-06T07:19:55.406689+0000","last_undegraded":"2024-03-06T07:19:55.406689+0000","last_fullsized":"2024-03-06T07:19:55.406689+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407693+0000","last_change":"2024-03-06T07:19:53.048472+0000","last_active":"2024-03-06T07:19:55.407693+0000","last_peered":"2024-03-06T07:19:55.407693+0000","last_clean":"2024-03-06T07:19:55.407693+0000","last_became_active":"2024-03-06T07:19:53.048213+0000","last_became_peered":"2024-03-06T07:19:53.048213+0000","last_unstale":"2024-03-06T07:19:55.407693+0000","last_undegraded":"2024-03-06T07:19:55.407693+0000","last_fullsized":"2024-03-06T07:19:55.407693+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:19.721078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.033186+0000","last_change":"2024-03-06T07:19:54.042432+0000","last_active":"2024-03-06T07:19:55.033186+0000","last_peered":"2024-03-06T07:19:55.033186+0000","last_clean":"2024-03-06T07:19:55.033186+0000","last_became_active":"2024-03-06T07:19:54.042166+0000","last_became_peered":"2024-03-06T07:19:54.042166+0000","last_unstale":"2024-03-06T07:19:55.033186+0000","last_undegraded":"2024-03-06T07:19:55.033186+0000","last_fullsized":"2024-03-06T07:19:55.033186+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.231274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406429+0000","last_change":"2024-03-06T07:19:53.044950+0000","last_active":"2024-03-06T07:19:55.406429+0000","last_peered":"2024-03-06T07:19:55.406429+0000","last_clean":"2024-03-06T07:19:55.406429+0000","last_became_active":"2024-03-06T07:19:53.044630+0000","last_became_peered":"2024-03-06T07:19:53.044630+0000","last_unstale":"2024-03-06T07:19:55.406429+0000","last_undegraded":"2024-03-06T07:19:55.406429+0000","last_fullsized":"2024-03-06T07:19:55.406429+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:05.924599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203102+0000","last_change":"2024-03-06T07:19:54.059591+0000","last_active":"2024-03-06T07:19:55.203102+0000","last_peered":"2024-03-06T07:19:55.203102+0000","last_clean":"2024-03-06T07:19:55.203102+0000","last_became_active":"2024-03-06T07:19:54.058814+0000","last_became_peered":"2024-03-06T07:19:54.058814+0000","last_unstale":"2024-03-06T07:19:55.203102+0000","last_undegraded":"2024-03-06T07:19:55.203102+0000","last_fullsized":"2024-03-06T07:19:55.203102+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034858+0000","last_change":"2024-03-06T07:19:53.040833+0000","last_active":"2024-03-06T07:19:55.034858+0000","last_peered":"2024-03-06T07:19:55.034858+0000","last_clean":"2024-03-06T07:19:55.034858+0000","last_became_active":"2024-03-06T07:19:53.040561+0000","last_became_peered":"2024-03-06T07:19:53.040561+0000","last_unstale":"2024-03-06T07:19:55.034858+0000","last_undegraded":"2024-03-06T07:19:55.034858+0000","last_fullsized":"2024-03-06T07:19:55.034858+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:12.315549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.032881+0000","last_change":"2024-03-06T07:19:54.042216+0000","last_active":"2024-03-06T07:19:55.032881+0000","last_peered":"2024-03-06T07:19:55.032881+0000","last_clean":"2024-03-06T07:19:55.032881+0000","last_became_active":"2024-03-06T07:19:54.041923+0000","last_became_peered":"2024-03-06T07:19:54.041923+0000","last_unstale":"2024-03-06T07:19:55.032881+0000","last_undegraded":"2024-03-06T07:19:55.032881+0000","last_fullsized":"2024-03-06T07:19:55.032881+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406797+0000","last_change":"2024-03-06T07:19:53.047959+0000","last_active":"2024-03-06T07:19:55.406797+0000","last_peered":"2024-03-06T07:19:55.406797+0000","last_clean":"2024-03-06T07:19:55.406797+0000","last_became_active":"2024-03-06T07:19:53.047680+0000","last_became_peered":"2024-03-06T07:19:53.047680+0000","last_unstale":"2024-03-06T07:19:55.406797+0000","last_undegraded":"2024-03-06T07:19:55.406797+0000","last_fullsized":"2024-03-06T07:19:55.406797+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:38.499903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406633+0000","last_change":"2024-03-06T07:19:54.043516+0000","last_active":"2024-03-06T07:19:55.406633+0000","last_peered":"2024-03-06T07:19:55.406633+0000","last_clean":"2024-03-06T07:19:55.406633+0000","last_became_active":"2024-03-06T07:19:54.041651+0000","last_became_peered":"2024-03-06T07:19:54.041651+0000","last_unstale":"2024-03-06T07:19:55.406633+0000","last_undegraded":"2024-03-06T07:19:55.406633+0000","last_fullsized":"2024-03-06T07:19:55.406633+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:04.528232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407048+0000","last_change":"2024-03-06T07:19:53.048966+0000","last_active":"2024-03-06T07:19:55.407048+0000","last_peered":"2024-03-06T07:19:55.407048+0000","last_clean":"2024-03-06T07:19:55.407048+0000","last_became_active":"2024-03-06T07:19:53.047988+0000","last_became_peered":"2024-03-06T07:19:53.047988+0000","last_unstale":"2024-03-06T07:19:55.407048+0000","last_undegraded":"2024-03-06T07:19:55.407048+0000","last_fullsized":"2024-03-06T07:19:55.407048+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:49.519186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407000+0000","last_change":"2024-03-06T07:19:54.062368+0000","last_active":"2024-03-06T07:19:55.407000+0000","last_peered":"2024-03-06T07:19:55.407000+0000","last_clean":"2024-03-06T07:19:55.407000+0000","last_became_active":"2024-03-06T07:19:54.060740+0000","last_became_peered":"2024-03-06T07:19:54.060740+0000","last_unstale":"2024-03-06T07:19:55.407000+0000","last_undegraded":"2024-03-06T07:19:55.407000+0000","last_fullsized":"2024-03-06T07:19:55.407000+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406682+0000","last_change":"2024-03-06T07:19:53.044944+0000","last_active":"2024-03-06T07:19:55.406682+0000","last_peered":"2024-03-06T07:19:55.406682+0000","last_clean":"2024-03-06T07:19:55.406682+0000","last_became_active":"2024-03-06T07:19:53.038900+0000","last_became_peered":"2024-03-06T07:19:53.038900+0000","last_unstale":"2024-03-06T07:19:55.406682+0000","last_undegraded":"2024-03-06T07:19:55.406682+0000","last_fullsized":"2024-03-06T07:19:55.406682+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:59.062789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203276+0000","last_change":"2024-03-06T07:19:54.059347+0000","last_active":"2024-03-06T07:19:55.203276+0000","last_peered":"2024-03-06T07:19:55.203276+0000","last_clean":"2024-03-06T07:19:55.203276+0000","last_became_active":"2024-03-06T07:19:54.059083+0000","last_became_peered":"2024-03-06T07:19:54.059083+0000","last_unstale":"2024-03-06T07:19:55.203276+0000","last_undegraded":"2024-03-06T07:19:55.203276+0000","last_fullsized":"2024-03-06T07:19:55.203276+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406981+0000","last_change":"2024-03-06T07:19:53.048645+0000","last_active":"2024-03-06T07:19:55.406981+0000","last_peered":"2024-03-06T07:19:55.406981+0000","last_clean":"2024-03-06T07:19:55.406981+0000","last_became_active":"2024-03-06T07:19:53.048406+0000","last_became_peered":"2024-03-06T07:19:53.048406+0000","last_unstale":"2024-03-06T07:19:55.406981+0000","last_undegraded":"2024-03-06T07:19:55.406981+0000","last_fullsized":"2024-03-06T07:19:55.406981+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:10.846361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406510+0000","last_change":"2024-03-06T07:19:54.049719+0000","last_active":"2024-03-06T07:19:55.406510+0000","last_peered":"2024-03-06T07:19:55.406510+0000","last_clean":"2024-03-06T07:19:55.406510+0000","last_became_active":"2024-03-06T07:19:54.049222+0000","last_became_peered":"2024-03-06T07:19:54.049222+0000","last_unstale":"2024-03-06T07:19:55.406510+0000","last_undegraded":"2024-03-06T07:19:55.406510+0000","last_fullsized":"2024-03-06T07:19:55.406510+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034599+0000","last_change":"2024-03-06T07:19:53.045046+0000","last_active":"2024-03-06T07:19:55.034599+0000","last_peered":"2024-03-06T07:19:55.034599+0000","last_clean":"2024-03-06T07:19:55.034599+0000","last_became_active":"2024-03-06T07:19:53.044751+0000","last_became_peered":"2024-03-06T07:19:53.044751+0000","last_unstale":"2024-03-06T07:19:55.034599+0000","last_undegraded":"2024-03-06T07:19:55.034599+0000","last_fullsized":"2024-03-06T07:19:55.034599+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.407231+0000","last_change":"2024-03-06T07:19:54.039796+0000","last_active":"2024-03-06T07:19:55.407231+0000","last_peered":"2024-03-06T07:19:55.407231+0000","last_clean":"2024-03-06T07:19:55.407231+0000","last_became_active":"2024-03-06T07:19:54.039527+0000","last_became_peered":"2024-03-06T07:19:54.039527+0000","last_unstale":"2024-03-06T07:19:55.407231+0000","last_undegraded":"2024-03-06T07:19:55.407231+0000","last_fullsized":"2024-03-06T07:19:55.407231+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:49.114747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.203132+0000","last_change":"2024-03-06T07:19:53.043599+0000","last_active":"2024-03-06T07:19:55.203132+0000","last_peered":"2024-03-06T07:19:55.203132+0000","last_clean":"2024-03-06T07:19:55.203132+0000","last_became_active":"2024-03-06T07:19:53.043294+0000","last_became_peered":"2024-03-06T07:19:53.043294+0000","last_unstale":"2024-03-06T07:19:55.203132+0000","last_undegraded":"2024-03-06T07:19:55.203132+0000","last_fullsized":"2024-03-06T07:19:55.203132+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:56.511464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.034309+0000","last_change":"2024-03-06T07:19:54.055814+0000","last_active":"2024-03-06T07:19:55.034309+0000","last_peered":"2024-03-06T07:19:55.034309+0000","last_clean":"2024-03-06T07:19:55.034309+0000","last_became_active":"2024-03-06T07:19:54.055480+0000","last_became_peered":"2024-03-06T07:19:54.055480+0000","last_unstale":"2024-03-06T07:19:55.034309+0000","last_undegraded":"2024-03-06T07:19:55.034309+0000","last_fullsized":"2024-03-06T07:19:55.034309+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:35.156134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406771+0000","last_change":"2024-03-06T07:19:53.046120+0000","last_active":"2024-03-06T07:19:55.406771+0000","last_peered":"2024-03-06T07:19:55.406771+0000","last_clean":"2024-03-06T07:19:55.406771+0000","last_became_active":"2024-03-06T07:19:53.045816+0000","last_became_peered":"2024-03-06T07:19:53.045816+0000","last_unstale":"2024-03-06T07:19:55.406771+0000","last_undegraded":"2024-03-06T07:19:55.406771+0000","last_fullsized":"2024-03-06T07:19:55.406771+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:56.226679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-06T07:19:55.406361+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:19:55.406361+0000","last_peered":"2024-03-06T07:19:55.406361+0000","last_clean":"2024-03-06T07:19:55.406361+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:19:55.406361+0000","last_undegraded":"2024-03-06T07:19:55.406361+0000","last_fullsized":"2024-03-06T07:19:55.406361+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":866583,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":626688,"data_stored":276743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":276091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":622592,"data_stored":276673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":277725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":630784,"data_stored":277703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":626688,"data_stored":276251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":866515,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:01.721 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:20:01.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:20:01.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.863+0000 7f5b47237640 1 Processor -- start 2024-03-06T07:20:01.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.863+0000 7f5b47237640 1 -- start start 2024-03-06T07:20:01.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b47237640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5b4006c1f0 0x7f5b400c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b47237640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5b4006cad0 0x7f5b40154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b47237640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b400c2790 con 0x7f5b4014ef70 2024-03-06T07:20:01.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b47237640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b400c3090 con 0x7f5b4006cad0 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b47237640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b4014f350 con 0x7f5b4006c1f0 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b457d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5b4006cad0 0x7f5b40154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b457d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5b4006cad0 0x7f5b40154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51640/0 (socket says 172.21.15.94:51640) 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b457d4640 1 -- 172.21.15.94:0/2758134759 learned_addr learned my addr 172.21.15.94:0/2758134759 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b3ffff640 1 --2- 172.21.15.94:0/2758134759 >> v2:172.21.15.145:3300/0 conn(0x7f5b4006c1f0 0x7f5b400c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b457d4640 1 -- 172.21.15.94:0/2758134759 >> v2:172.21.15.145:3300/0 conn(0x7f5b4006c1f0 msgr2=0x7f5b400c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b457d4640 1 --2- 172.21.15.94:0/2758134759 >> v2:172.21.15.145:3300/0 conn(0x7f5b4006c1f0 0x7f5b400c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b457d4640 1 -- 172.21.15.94:0/2758134759 >> v1:172.21.15.145:6789/0 conn(0x7f5b4014ef70 legacy=0x7f5b400c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.864+0000 7f5b457d4640 1 -- 172.21.15.94:0/2758134759 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4006cf20 con 0x7f5b4006cad0 2024-03-06T07:20:01.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.865+0000 7f5b457d4640 1 --2- 172.21.15.94:0/2758134759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b40154c30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5b4006e600 tx=0x7f5b3802cff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea53dc2d25ec6e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:01.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.865+0000 7f5b3f7fe640 1 -- 172.21.15.94:0/2758134759 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3802db90 con 0x7f5b4006cad0 2024-03-06T07:20:01.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.865+0000 7f5b3f7fe640 1 -- 172.21.15.94:0/2758134759 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3802dd30 con 0x7f5b4006cad0 2024-03-06T07:20:01.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.865+0000 7f5b3f7fe640 1 -- 172.21.15.94:0/2758134759 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38035610 con 0x7f5b4006cad0 2024-03-06T07:20:01.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.865+0000 7f5b47237640 1 -- 172.21.15.94:0/2758134759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 msgr2=0x7f5b40154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.865+0000 7f5b47237640 1 --2- 172.21.15.94:0/2758134759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b40154c30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5b4006e600 tx=0x7f5b3802cff0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.866+0000 7f5b47237640 1 -- 172.21.15.94:0/2758134759 shutdown_connections 2024-03-06T07:20:01.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.866+0000 7f5b47237640 1 --2- 172.21.15.94:0/2758134759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b40154c30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.866+0000 7f5b47237640 1 --2- 172.21.15.94:0/2758134759 >> v2:172.21.15.145:3300/0 conn(0x7f5b4006c1f0 0x7f5b400c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.866+0000 7f5b47237640 1 -- 172.21.15.94:0/2758134759 >> 172.21.15.94:0/2758134759 conn(0x7f5b400c6710 msgr2=0x7f5b400c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:01.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.866+0000 7f5b47237640 1 -- 172.21.15.94:0/2758134759 shutdown_connections 2024-03-06T07:20:01.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.866+0000 7f5b47237640 1 -- 172.21.15.94:0/2758134759 wait complete. 2024-03-06T07:20:01.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.867+0000 7f5b47237640 1 Processor -- start 2024-03-06T07:20:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.867+0000 7f5b47237640 1 -- start start 2024-03-06T07:20:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.867+0000 7f5b47237640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b4006c1f0 0x7f5b401e9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.867+0000 7f5b47237640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b400a8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.867+0000 7f5b47237640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b4014ef70 0x7f5b40147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.867+0000 7f5b47237640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b40155750 con 0x7f5b4006cad0 2024-03-06T07:20:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b47237640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b400c24b0 con 0x7f5b4014ef70 2024-03-06T07:20:01.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b47237640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b40155170 con 0x7f5b4006c1f0 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b400a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b400a8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51644/0 (socket says 172.21.15.94:51644) 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 -- 172.21.15.94:0/352727324 learned_addr learned my addr 172.21.15.94:0/352727324 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b44fd3640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b4006c1f0 0x7f5b401e9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b457d4640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b4014ef70 0x7f5b40147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 -- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b4006c1f0 msgr2=0x7f5b401e9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b4006c1f0 0x7f5b401e9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 -- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b4014ef70 msgr2=0x7f5b40147860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b4014ef70 0x7f5b40147860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 -- 172.21.15.94:0/352727324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b401e9fa0 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3ffff640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b400a8840 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00e9b0 tx=0x7f5b2c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3d7fa640 1 -- 172.21.15.94:0/352727324 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c017070 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3d7fa640 1 -- 172.21.15.94:0/352727324 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2c00c9a0 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b3d7fa640 1 -- 172.21.15.94:0/352727324 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c00cd90 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b40073270 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.868+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4014f9a0 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.869+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4006dc80 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.870+0000 7f5b3d7fa640 1 -- 172.21.15.94:0/352727324 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b2c01a070 con 0x7f5b4006cad0 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.870+0000 7f5b3d7fa640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5b2406b4d0 0x7f5b2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:01.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.870+0000 7f5b3d7fa640 1 -- 172.21.15.94:0/352727324 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5b2c08a210 con 0x7f5b4006cad0 2024-03-06T07:20:01.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.870+0000 7f5b44fd3640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5b2406b4d0 0x7f5b2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:01.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.871+0000 7f5b44fd3640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5b2406b4d0 0x7f5b2406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b40070670 tx=0x7f5b34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:01.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:01.874+0000 7f5b3d7fa640 1 -- 172.21.15.94:0/352727324 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b4006dc80 con 0x7f5b4006cad0 2024-03-06T07:20:02.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.115+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5b400c2e70 con 0x7f5b4006cad0 2024-03-06T07:20:02.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.117+0000 7f5b3d7fa640 1 -- 172.21.15.94:0/352727324 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5b400c2e70 con 0x7f5b4006cad0 2024-03-06T07:20:02.119 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:20:02.119 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:20:02.119 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:20:02.119 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:20:02.119 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:20:02.119 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5b2406b4d0 msgr2=0x7f5b2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5b2406b4d0 0x7f5b2406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b40070670 tx=0x7f5b34009040 comp rx=0 tx=0).stop 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 msgr2=0x7f5b400a8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b400a8840 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00e9b0 tx=0x7f5b2c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 shutdown_connections 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5b2406b4d0 0x7f5b2406d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5b4014ef70 0x7f5b40147860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b4006cad0 0x7f5b400a8840 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 --2- 172.21.15.94:0/352727324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5b4006c1f0 0x7f5b401e9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 >> 172.21.15.94:0/352727324 conn(0x7f5b400c6710 msgr2=0x7f5b400c1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:02.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 shutdown_connections 2024-03-06T07:20:02.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.119+0000 7f5b47237640 1 -- 172.21.15.94:0/352727324 wait complete. 2024-03-06T07:20:02.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:02.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.230+0000 7f1fea0c9640 1 Processor -- start 2024-03-06T07:20:02.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.230+0000 7f1fea0c9640 1 -- start start 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fea0c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fe40aa180 0x7f1fe414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fea0c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe414f5d0 con 0x7f1fe41550c0 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fea0c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe4155bb0 con 0x7f1fe414f140 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fea0c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe40c4610 con 0x7f1fe40aa180 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe2ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fe40aa180 0x7f1fe414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe37fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1fe41550c0 0x7f1fe41554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57338/0 (socket says 172.21.15.94:57338) 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe2ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fe40aa180 0x7f1fe414ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51648/0 (socket says 172.21.15.94:51648) 2024-03-06T07:20:02.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe37fe640 1 -- 172.21.15.94:0/3278933131 learned_addr learned my addr 172.21.15.94:0/3278933131 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe2ffd640 1 -- 172.21.15.94:0/3278933131 >> v1:172.21.15.145:6789/0 conn(0x7f1fe414f140 legacy=0x7f1fe40c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe2ffd640 1 -- 172.21.15.94:0/3278933131 >> v1:172.21.15.94:6789/0 conn(0x7f1fe41550c0 legacy=0x7f1fe41554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe27fc640 1 -- 172.21.15.94:0/3278933131 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147697647 0 0) 0x7f1fe414f5d0 con 0x7f1fe41550c0 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.231+0000 7f1fe2ffd640 1 -- 172.21.15.94:0/3278933131 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe40aa660 con 0x7f1fe40aa180 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.232+0000 7f1fe2ffd640 1 --2- 172.21.15.94:0/3278933131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe414ec00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800a940 tx=0x7f1fd802cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c937d7b53d895835 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.232+0000 7f1fe27fc640 1 -- 172.21.15.94:0/3278933131 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd803a070 con 0x7f1fe40aa180 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.232+0000 7f1fe27fc640 1 -- 172.21.15.94:0/3278933131 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd802dab0 con 0x7f1fe40aa180 2024-03-06T07:20:02.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.232+0000 7f1fe27fc640 1 -- 172.21.15.94:0/3278933131 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800b040 con 0x7f1fe40aa180 2024-03-06T07:20:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.233+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3278933131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 msgr2=0x7f1fe414ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.233+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3278933131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe414ec00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800a940 tx=0x7f1fd802cd00 comp rx=0 tx=0).stop 2024-03-06T07:20:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.233+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3278933131 shutdown_connections 2024-03-06T07:20:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.233+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3278933131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe414ec00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.233+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3278933131 >> 172.21.15.94:0/3278933131 conn(0x7f1fe40c6710 msgr2=0x7f1fe40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.233+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3278933131 shutdown_connections 2024-03-06T07:20:02.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.233+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3278933131 wait complete. 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.234+0000 7f1fea0c9640 1 Processor -- start 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.234+0000 7f1fea0c9640 1 -- start start 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe406c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe414f140 0x7f1fe40a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe41550c0 0x7f1fe40a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40c1310 con 0x7f1fe40aa180 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40c4860 con 0x7f1fe41550c0 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40c1490 con 0x7f1fe414f140 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe406c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe406c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51664/0 (socket says 172.21.15.94:51664) 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 -- 172.21.15.94:0/3518454667 learned_addr learned my addr 172.21.15.94:0/3518454667 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe2ffd640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe414f140 0x7f1fe40a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe3fff640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe41550c0 0x7f1fe40a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 -- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe414f140 msgr2=0x7f1fe40a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe414f140 0x7f1fe40a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 -- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe41550c0 msgr2=0x7f1fe40a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe41550c0 0x7f1fe40a8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 -- 172.21.15.94:0/3518454667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe406e5c0 con 0x7f1fe40aa180 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fe37fe640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe406c830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400c3f0 tx=0x7f1fd400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fcbfff640 1 -- 172.21.15.94:0/3518454667 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40074e0 con 0x7f1fe40aa180 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fcbfff640 1 -- 172.21.15.94:0/3518454667 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd40047e0 con 0x7f1fe40aa180 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe40a6b50 con 0x7f1fe40aa180 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fcbfff640 1 -- 172.21.15.94:0/3518454667 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4004bd0 con 0x7f1fe40aa180 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.235+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe4146b00 con 0x7f1fe40aa180 2024-03-06T07:20:02.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.236+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe404f330 con 0x7f1fe40aa180 2024-03-06T07:20:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.237+0000 7f1fcbfff640 1 -- 172.21.15.94:0/3518454667 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd4011070 con 0x7f1fe40aa180 2024-03-06T07:20:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.237+0000 7f1fcbfff640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.237+0000 7f1fcbfff640 1 -- 172.21.15.94:0/3518454667 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1fd4089e80 con 0x7f1fe40aa180 2024-03-06T07:20:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.237+0000 7f1fe2ffd640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.238+0000 7f1fe2ffd640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fd006b4d0 0x7f1fd006d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8037000 tx=0x7f1fd8031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.242+0000 7f1fcbfff640 1 -- 172.21.15.94:0/3518454667 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fd4058510 con 0x7f1fe40aa180 2024-03-06T07:20:02.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.450+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fe40c0df0 con 0x7f1fe40aa180 2024-03-06T07:20:02.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.451+0000 7f1fcbfff640 1 -- 172.21.15.94:0/3518454667 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f1fd405c5b0 con 0x7f1fe40aa180 2024-03-06T07:20:02.453 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:02.453 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5521,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/480754107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":480754107},{"type":"v1","addr":"172.21.15.94:6835","nonce":480754107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:19:55.378348+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35084":{"gid":35084,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3125091562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3125091562},{"type":"v1","addr":"172.21.15.145:6837","nonce":3125091562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35084,"qdb_cluster":[35084]},"id":3}]} 2024-03-06T07:20:02.454 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 29 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fd006b4d0 msgr2=0x7f1fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fd006b4d0 0x7f1fd006d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8037000 tx=0x7f1fd8031000 comp rx=0 tx=0).stop 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 msgr2=0x7f1fe406c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe406c830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400c3f0 tx=0x7f1fd400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 shutdown_connections 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe41550c0 0x7f1fe40a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe414f140 0x7f1fe40a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 --2- 172.21.15.94:0/3518454667 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe40aa180 0x7f1fe406c830 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 >> 172.21.15.94:0/3518454667 conn(0x7f1fe40c6710 msgr2=0x7f1fe40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 shutdown_connections 2024-03-06T07:20:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.453+0000 7f1fea0c9640 1 -- 172.21.15.94:0/3518454667 wait complete. 2024-03-06T07:20:02.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:19:53.385858+0000', 'modified': '2024-03-06T07:19:55.378348+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 35084}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_35084': {'gid': 35084, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/3125091562', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3125091562}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3125091562}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 35084, 'qdb_cluster': [35084]} 2024-03-06T07:20:02.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:20:02.869 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:20:02.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff49c8d4640 1 Processor -- start 2024-03-06T07:20:02.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff49c8d4640 1 -- start start 2024-03-06T07:20:02.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff49c8d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4980c8950 0x7ff4980c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff49c8d4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff4981558d0 0x7ff498155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff49c8d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff498155130 con 0x7ff4980c69f0 2024-03-06T07:20:02.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff49c8d4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff498155440 con 0x7ff4980c8950 2024-03-06T07:20:02.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff49c8d4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4980ad480 con 0x7ff4981558d0 2024-03-06T07:20:02.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff48ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4980c8950 0x7ff4980c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.971+0000 7ff48ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4980c8950 0x7ff4980c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51674/0 (socket says 172.21.15.94:51674) 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.972+0000 7ff48ffff640 1 -- 172.21.15.94:0/1332832745 learned_addr learned my addr 172.21.15.94:0/1332832745 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.972+0000 7ff496d9d640 1 --2- 172.21.15.94:0/1332832745 >> v2:172.21.15.145:3300/0 conn(0x7ff4981558d0 0x7ff498155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.972+0000 7ff48ffff640 1 -- 172.21.15.94:0/1332832745 >> v2:172.21.15.145:3300/0 conn(0x7ff4981558d0 msgr2=0x7ff498155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.972+0000 7ff48ffff640 1 --2- 172.21.15.94:0/1332832745 >> v2:172.21.15.145:3300/0 conn(0x7ff4981558d0 0x7ff498155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.972+0000 7ff48ffff640 1 -- 172.21.15.94:0/1332832745 >> v1:172.21.15.145:6789/0 conn(0x7ff4980c69f0 legacy=0x7ff4980acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.972+0000 7ff48ffff640 1 -- 172.21.15.94:0/1332832745 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4980c5d50 con 0x7ff4980c8950 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.972+0000 7ff48ffff640 1 --2- 172.21.15.94:0/1332832745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff4980c6dd0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff498074a60 tx=0x7ff48002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f36f07e031f71d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff495d9b640 1 -- 172.21.15.94:0/1332832745 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480035020 con 0x7ff4980c8950 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff495d9b640 1 -- 172.21.15.94:0/1332832745 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff480031870 con 0x7ff4980c8950 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff495d9b640 1 -- 172.21.15.94:0/1332832745 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480031c40 con 0x7ff4980c8950 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 -- 172.21.15.94:0/1332832745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 msgr2=0x7ff4980c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/1332832745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff4980c6dd0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff498074a60 tx=0x7ff48002b910 comp rx=0 tx=0).stop 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 -- 172.21.15.94:0/1332832745 shutdown_connections 2024-03-06T07:20:02.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/1332832745 >> v2:172.21.15.145:3300/0 conn(0x7ff4981558d0 0x7ff498155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/1332832745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff4980c6dd0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 -- 172.21.15.94:0/1332832745 >> 172.21.15.94:0/1332832745 conn(0x7ff4980cae70 msgr2=0x7ff4980cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:02.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 -- 172.21.15.94:0/1332832745 shutdown_connections 2024-03-06T07:20:02.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.973+0000 7ff49c8d4640 1 -- 172.21.15.94:0/1332832745 wait complete. 2024-03-06T07:20:02.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.974+0000 7ff49c8d4640 1 Processor -- start 2024-03-06T07:20:02.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.974+0000 7ff49c8d4640 1 -- start start 2024-03-06T07:20:02.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff49c8d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4980c69f0 0x7ff4981591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff49c8d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff49814ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff49c8d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4981558d0 0x7ff4980c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff49c8d4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff498152a20 con 0x7ff4980c8950 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff49c8d4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4980c6310 con 0x7ff4980c69f0 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff49c8d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff498152ba0 con 0x7ff4981558d0 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff49814ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff49814ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51684/0 (socket says 172.21.15.94:51684) 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff48ffff640 1 -- 172.21.15.94:0/824066209 learned_addr learned my addr 172.21.15.94:0/824066209 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff49659c640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4980c69f0 0x7ff4981591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.975+0000 7ff496d9d640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4981558d0 0x7ff4980c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48ffff640 1 -- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4981558d0 msgr2=0x7ff4980c1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48ffff640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4981558d0 0x7ff4980c1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48ffff640 1 -- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4980c69f0 msgr2=0x7ff4981591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48ffff640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4980c69f0 0x7ff4981591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48ffff640 1 -- 172.21.15.94:0/824066209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4980c36f0 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48ffff640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff49814ea00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff48002b950 tx=0x7ff4800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48effd640 1 -- 172.21.15.94:0/824066209 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480040070 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48effd640 1 -- 172.21.15.94:0/824066209 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff480030d00 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4980c4960 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff48effd640 1 -- 172.21.15.94:0/824066209 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480035040 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.976+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4981ef160 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.977+0000 7ff48effd640 1 -- 172.21.15.94:0/824066209 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff48004b060 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.978+0000 7ff48effd640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff47806b4d0 0x7ff47806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.978+0000 7ff48effd640 1 -- 172.21.15.94:0/824066209 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4800abf60 con 0x7ff4980c8950 2024-03-06T07:20:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.978+0000 7ff49659c640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff47806b4d0 0x7ff47806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.978+0000 7ff49659c640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff47806b4d0 0x7ff47806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff4980c2ce0 tx=0x7ff488009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 Processor -- start 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 -- start start 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b7eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ed330 0x55cf4b7eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7eb4b0 0x55cf4b7e7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b7516b0 con 0x55cf4b7ed330 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b6faa60 con 0x55cf4b7ecf50 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.979+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b74d750 con 0x55cf4b7eb4b0 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ed330 0x55cf4b7eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff49659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b7eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ed330 0x55cf4b7eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51696/0 (socket says 172.21.15.94:51696) 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 -- 172.21.15.94:0/207179485 learned_addr learned my addr 172.21.15.94:0/207179485 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:02.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff496d9d640 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7eb4b0 0x55cf4b7e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 -- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7eb4b0 msgr2=0x55cf4b7e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7eb4b0 0x55cf4b7e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 -- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 msgr2=0x55cf4b7eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b7eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 -- 172.21.15.94:0/207179485 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf4b718f30 con 0x55cf4b7ed330 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff49659c640 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b7eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff48ffff640 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ed330 0x55cf4b7eaf70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff480076a60 tx=0x7ff480009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e6312d69b6e5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff477fff640 1 -- 172.21.15.94:0/207179485 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800a8070 con 0x55cf4b7ed330 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff477fff640 1 -- 172.21.15.94:0/207179485 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48007a600 con 0x55cf4b7ed330 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.980+0000 7ff477fff640 1 -- 172.21.15.94:0/207179485 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48007e6a0 con 0x55cf4b7ed330 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 -- 172.21.15.94:0/207179485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ed330 msgr2=0x55cf4b7eaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ed330 0x55cf4b7eaf70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff480076a60 tx=0x7ff480009550 comp rx=0 tx=0).stop 2024-03-06T07:20:02.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 -- 172.21.15.94:0/207179485 shutdown_connections 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7eb4b0 0x55cf4b7e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ed330 0x55cf4b7eaf70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 --2- 172.21.15.94:0/207179485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b7eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 -- 172.21.15.94:0/207179485 >> 172.21.15.94:0/207179485 conn(0x55cf4b7e97d0 msgr2=0x55cf4b80e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 -- 172.21.15.94:0/207179485 shutdown_connections 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 -- 172.21.15.94:0/207179485 wait complete. 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 -- start start 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.981+0000 7ff49f596740 1 Processor -- start 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49f596740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b83a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b7e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b8117c0 0x55cf4b80b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49f596740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b7516b0 con 0x55cf4b7ecf50 2024-03-06T07:20:02.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b70dea0 con 0x55cf4b8117c0 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b80f230 con 0x55cf4b7ed330 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49f596740 10 client.0 _ll_register_callbacks cb 0x55cf4b70d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b83a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b83a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51698/0 (socket says 172.21.15.94:51698) 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff49659c640 1 -- 172.21.15.94:0/4178899824 learned_addr learned my addr 172.21.15.94:0/4178899824 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff48ffff640 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b7e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.982+0000 7ff496d9d640 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b8117c0 0x55cf4b80b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49659c640 1 -- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 msgr2=0x55cf4b7e78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49659c640 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b7e78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49659c640 1 -- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b8117c0 msgr2=0x55cf4b80b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49659c640 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b8117c0 0x55cf4b80b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49659c640 1 -- 172.21.15.94:0/4178899824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf4b70e270 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49659c640 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b83a120 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff488014700 tx=0x7ff488000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a7d0259809286e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff4757fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff4757fa640 1 -- 172.21.15.94:0/4178899824 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488025020 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff4757fa640 1 -- 172.21.15.94:0/4178899824 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff488013740 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff48007a600 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff4757fa640 1 -- 172.21.15.94:0/4178899824 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff48802f050 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 10 client.5838 fetch_fsmap learned FSMap version 29 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x55cf4b4662d0 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff4757fa640 1 -- 172.21.15.94:0/4178899824 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48007a600 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff4757fa640 1 -- 172.21.15.94:0/4178899824 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 29) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff488032020 con 0x55cf4b7ecf50 2024-03-06T07:20:02.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 10 client.5838 fetch_fsmap finished waiting for FSMap version 29 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 10 client.5838 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 20 client.5838 populate_metadata read hostname 'smithi094' 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b846170 0x55cf4b848560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 4 client.5838 mds_command: new command op to 35084 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.983+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 --> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf4b75ebf0 con 0x55cf4b846170 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.984+0000 7ff48ffff640 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b846170 0x55cf4b848560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.984+0000 7ff48ffff640 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b846170 0x55cf4b848560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad653441e901a021 server_cookie=10e6aa42ebd0224 in_seq=0 out_seq=0 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.984+0000 7ff4757fa640 10 client.5838 ms_handle_connect on v2:172.21.15.145:6835/3125091562 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff4757fa640 1 -- 172.21.15.94:0/4178899824 <== mds.0 v2:172.21.15.145:6835/3125091562 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cf4b75ebf0 con 0x55cf4b846170 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff4757fa640 10 client.5838 handle_command_reply: tid=0 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 client.5838 shutdown 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 msgr2=0x55cf4b83a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b83a120 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff488014700 tx=0x7ff488000ed0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 shutdown_connections 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b846170 0x55cf4b848560 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b8117c0 0x55cf4b80b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b7e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 --2- 172.21.15.94:0/4178899824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b83a120 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff4757fa640 0 client.5838 ms_handle_reset on v2:172.21.15.145:6835/3125091562 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 >> 172.21.15.94:0/4178899824 conn(0x55cf4b7eb4b0 msgr2=0x55cf4b7ec280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 shutdown_connections 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 1 -- 172.21.15.94:0/4178899824 wait complete. 2024-03-06T07:20:02.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:02.985+0000 7ff49f596740 20 client.5838 trim_cache size 0 max 16384 2024-03-06T07:20:03.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.001+0000 7ff49f596740 1 Processor -- start 2024-03-06T07:20:03.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49f596740 1 -- start start 2024-03-06T07:20:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b826490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b80dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49f596740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b8117c0 0x55cf4b80e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49f596740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4981f6f30 con 0x55cf4b8117c0 2024-03-06T07:20:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4981499f0 con 0x55cf4b7ecf50 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4981f6a10 con 0x55cf4b7ed330 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff496d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b8117c0 0x55cf4b80e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff496d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b8117c0 0x55cf4b80e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51704/0 (socket says 172.21.15.94:51704) 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff496d9d640 1 -- 172.21.15.94:0/3436420123 learned_addr learned my addr 172.21.15.94:0/3436420123 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff48ffff640 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b80dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.002+0000 7ff49659c640 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b826490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff496d9d640 1 -- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 msgr2=0x55cf4b80dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff496d9d640 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b80dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff496d9d640 1 -- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 msgr2=0x55cf4b826490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff496d9d640 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b826490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff496d9d640 1 -- 172.21.15.94:0/3436420123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf4b5306b0 con 0x55cf4b8117c0 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff48ffff640 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b80dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49659c640 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b826490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff496d9d640 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b8117c0 0x55cf4b80e290 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff4900157a0 tx=0x7ff490009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4a07dae56ed83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff4777fe640 1 -- 172.21.15.94:0/3436420123 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff490024070 con 0x55cf4b8117c0 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff4777fe640 1 -- 172.21.15.94:0/3436420123 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff490017d80 con 0x55cf4b8117c0 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff4777fe640 1 -- 172.21.15.94:0/3436420123 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49001f670 con 0x55cf4b8117c0 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 -- 172.21.15.94:0/3436420123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b8117c0 msgr2=0x55cf4b80e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b8117c0 0x55cf4b80e290 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff4900157a0 tx=0x7ff490009750 comp rx=0 tx=0).stop 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 -- 172.21.15.94:0/3436420123 shutdown_connections 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b8117c0 0x55cf4b80e290 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b7ed330 0x55cf4b80dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 --2- 172.21.15.94:0/3436420123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ecf50 0x55cf4b826490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 -- 172.21.15.94:0/3436420123 >> 172.21.15.94:0/3436420123 conn(0x55cf4b7eb4b0 msgr2=0x55cf4b7e2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 -- 172.21.15.94:0/3436420123 shutdown_connections 2024-03-06T07:20:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.003+0000 7ff49f596740 1 -- 172.21.15.94:0/3436420123 wait complete. 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.004+0000 7ff49f596740 1 -- start start 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.004+0000 7ff49f596740 1 Processor -- start 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b89d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ed330 0x55cf4b820770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b8117c0 0x55cf4b892f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b80fb90 con 0x55cf4b7ecf50 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b839c00 con 0x55cf4b7ed330 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf4b83cc10 con 0x55cf4b8117c0 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 10 client.0 _ll_register_callbacks cb 0x7ff48800fed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b89d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b89d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51718/0 (socket says 172.21.15.94:51718) 2024-03-06T07:20:03.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 -- 172.21.15.94:0/4134447555 learned_addr learned my addr 172.21.15.94:0/4134447555 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff496d9d640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b8117c0 0x55cf4b892f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff48ffff640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ed330 0x55cf4b820770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 -- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b8117c0 msgr2=0x55cf4b892f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b8117c0 0x55cf4b892f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 -- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ed330 msgr2=0x55cf4b820770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ed330 0x55cf4b820770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 -- 172.21.15.94:0/4134447555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf4b80e300 con 0x55cf4b7ecf50 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff48ffff640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ed330 0x55cf4b820770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49659c640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b89d170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff488001d30 tx=0x7ff488014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35f94d642cd8eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff4757fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff4757fa640 1 -- 172.21.15.94:0/4134447555 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488025040 con 0x55cf4b7ecf50 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff4757fa640 1 -- 172.21.15.94:0/4134447555 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff488021400 con 0x55cf4b7ecf50 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.005+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff490017d80 con 0x55cf4b7ecf50 2024-03-06T07:20:03.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff4757fa640 1 -- 172.21.15.94:0/4134447555 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff48803f050 con 0x55cf4b7ecf50 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 10 client.5844 fetch_fsmap learned FSMap version 29 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x55cf4b812110 con 0x55cf4b7ecf50 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff4757fa640 1 -- 172.21.15.94:0/4134447555 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff490017d80 con 0x55cf4b7ecf50 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff4757fa640 1 -- 172.21.15.94:0/4134447555 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 29) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff488042020 con 0x55cf4b7ecf50 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 10 client.5844 fetch_fsmap finished waiting for FSMap version 29 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 10 client.5844 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 20 client.5844 populate_metadata read hostname 'smithi094' 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b836050 0x55cf4b836430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 4 client.5844 mds_command: new command op to 35084 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 --> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cf4b758f40 con 0x55cf4b836050 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff48ffff640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b836050 0x55cf4b836430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff48ffff640 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b836050 0x55cf4b836430 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5af70f5f3849b403 server_cookie=c142bf6f054479d0 in_seq=0 out_seq=0 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.006+0000 7ff4757fa640 10 client.5844 ms_handle_connect on v2:172.21.15.145:6835/3125091562 2024-03-06T07:20:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff4757fa640 1 -- 172.21.15.94:0/4134447555 <== mds.0 v2:172.21.15.145:6835/3125091562 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55cf4b758f40 con 0x55cf4b836050 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff4757fa640 10 client.5844 handle_command_reply: tid=0 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 client.5844 shutdown 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 msgr2=0x55cf4b89d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b89d170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff488001d30 tx=0x7ff488014000 comp rx=0 tx=0).stop 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 shutdown_connections 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:6835/3125091562,v1:172.21.15.145:6837/3125091562] conn(0x55cf4b836050 0x55cf4b836430 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55cf4b8117c0 0x55cf4b892f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55cf4b7ed330 0x55cf4b820770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 --2- 172.21.15.94:0/4134447555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55cf4b7ecf50 0x55cf4b89d170 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff4757fa640 0 client.5844 ms_handle_reset on v2:172.21.15.145:6835/3125091562 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 >> 172.21.15.94:0/4134447555 conn(0x55cf4b7eb4b0 msgr2=0x55cf4b826490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 shutdown_connections 2024-03-06T07:20:03.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 1 -- 172.21.15.94:0/4134447555 wait complete. 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49f596740 20 client.5844 trim_cache size 0 max 16384 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 35084, 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 8.9772777779999995, 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 29, 2024-03-06T07:20:03.018 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 52, 2024-03-06T07:20:03.019 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 52, 2024-03-06T07:20:03.019 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 14.092602768000001 2024-03-06T07:20:03.019 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:20:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff47806b4d0 msgr2=0x7ff47806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff47806b4d0 0x7ff47806d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff4980c2ce0 tx=0x7ff488009040 comp rx=0 tx=0).stop 2024-03-06T07:20:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 msgr2=0x7ff49814ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff49814ea00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff48002b950 tx=0x7ff4800046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 shutdown_connections 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff47806b4d0 0x7ff47806d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4981558d0 0x7ff4980c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4980c8950 0x7ff49814ea00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 --2- 172.21.15.94:0/824066209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4980c69f0 0x7ff4981591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.007+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 >> 172.21.15.94:0/824066209 conn(0x7ff4980cae70 msgr2=0x7ff498150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.008+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 shutdown_connections 2024-03-06T07:20:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.008+0000 7ff49c8d4640 1 -- 172.21.15.94:0/824066209 wait complete. 2024-03-06T07:20:03.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:03.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.120+0000 7f7cd4039640 1 Processor -- start 2024-03-06T07:20:03.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.120+0000 7f7cd4039640 1 -- start start 2024-03-06T07:20:03.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd4039640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ccc1527b0 0x7f7ccc159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd4039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd4039640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0c7c60 con 0x7f7ccc155b20 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd4039640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc159640 con 0x7f7ccc1527b0 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd4039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c8df0 con 0x7f7ccc0c7180 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd1dd5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ccc1527b0 0x7f7ccc159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd1dd5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ccc1527b0 0x7f7ccc159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51728/0 (socket says 172.21.15.94:51728) 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd1dd5640 1 -- 172.21.15.94:0/4099853453 learned_addr learned my addr 172.21.15.94:0/4099853453 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd15d4640 1 --2- 172.21.15.94:0/4099853453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd1dd5640 1 -- 172.21.15.94:0/4099853453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 msgr2=0x7f7ccc0c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:03.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd1dd5640 1 --2- 172.21.15.94:0/4099853453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd1dd5640 1 -- 172.21.15.94:0/4099853453 >> v1:172.21.15.145:6789/0 conn(0x7f7ccc155b20 legacy=0x7f7ccc0c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.121+0000 7f7cd1dd5640 1 -- 172.21.15.94:0/4099853453 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0c8060 con 0x7f7ccc1527b0 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.122+0000 7f7cd1dd5640 1 --2- 172.21.15.94:0/4099853453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc159100 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009870 tx=0x7f7cc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75d886c5ddcc4bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.122+0000 7f7cd0dd3640 1 -- 172.21.15.94:0/4099853453 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4035020 con 0x7f7ccc1527b0 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.122+0000 7f7cd0dd3640 1 -- 172.21.15.94:0/4099853453 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4031640 con 0x7f7ccc1527b0 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.122+0000 7f7cd0dd3640 1 -- 172.21.15.94:0/4099853453 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc402d040 con 0x7f7ccc1527b0 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 -- 172.21.15.94:0/4099853453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 msgr2=0x7f7ccc159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 --2- 172.21.15.94:0/4099853453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc159100 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009870 tx=0x7f7cc402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 -- 172.21.15.94:0/4099853453 shutdown_connections 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 --2- 172.21.15.94:0/4099853453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 --2- 172.21.15.94:0/4099853453 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc159100 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 -- 172.21.15.94:0/4099853453 >> 172.21.15.94:0/4099853453 conn(0x7f7ccc0ac560 msgr2=0x7f7ccc0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 -- 172.21.15.94:0/4099853453 shutdown_connections 2024-03-06T07:20:03.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.123+0000 7f7cd4039640 1 -- 172.21.15.94:0/4099853453 wait complete. 2024-03-06T07:20:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.124+0000 7f7cd4039640 1 Processor -- start 2024-03-06T07:20:03.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.124+0000 7f7cd4039640 1 -- start start 2024-03-06T07:20:03.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0bde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc155b20 0x7f7ccc14af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc155320 con 0x7f7ccc1527b0 2024-03-06T07:20:03.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c7980 con 0x7f7ccc155b20 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc155600 con 0x7f7ccc0c7180 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51738/0 (socket says 172.21.15.94:51738) 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 -- 172.21.15.94:0/2136372513 learned_addr learned my addr 172.21.15.94:0/2136372513 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd25d6640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc155b20 0x7f7ccc14af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd1dd5640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 -- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 msgr2=0x7f7ccc0bde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0bde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 -- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc155b20 msgr2=0x7f7ccc14af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc155b20 0x7f7ccc14af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 -- 172.21.15.94:0/2136372513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc1eaf30 con 0x7f7ccc1527b0 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd15d4640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc154440 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00e9b0 tx=0x7f7cbc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cc27fc640 1 -- 172.21.15.94:0/2136372513 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc017070 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cc27fc640 1 -- 172.21.15.94:0/2136372513 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc00c9a0 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc1eb210 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cc27fc640 1 -- 172.21.15.94:0/2136372513 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc00cd90 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.125+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc154980 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.126+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc04f330 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.127+0000 7f7cc27fc640 1 -- 172.21.15.94:0/2136372513 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cbc01a070 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.127+0000 7f7cc27fc640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7cb006b500 0x7f7cb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.127+0000 7f7cc27fc640 1 -- 172.21.15.94:0/2136372513 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7cbc08a5e0 con 0x7f7ccc1527b0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.127+0000 7f7cd1dd5640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7cb006b500 0x7f7cb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.128+0000 7f7cd1dd5640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7cb006b500 0x7f7cb006d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4036000 tx=0x7f7cc403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.132+0000 7f7cc27fc640 1 -- 172.21.15.94:0/2136372513 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7cbc058d10 con 0x7f7ccc1527b0 2024-03-06T07:20:03.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.339+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ccc1560a0 con 0x7f7ccc1527b0 2024-03-06T07:20:03.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.339+0000 7f7cc27fc640 1 -- 172.21.15.94:0/2136372513 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f7cbc05cdb0 con 0x7f7ccc1527b0 2024-03-06T07:20:03.341 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:03.341 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5521,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/480754107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":480754107},{"type":"v1","addr":"172.21.15.94:6835","nonce":480754107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:19:55.378348+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35084":{"gid":35084,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3125091562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3125091562},{"type":"v1","addr":"172.21.15.145:6837","nonce":3125091562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35084,"qdb_cluster":[35084]},"id":3}]} 2024-03-06T07:20:03.342 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 29 2024-03-06T07:20:03.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7cb006b500 msgr2=0x7f7cb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7cb006b500 0x7f7cb006d9c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4036000 tx=0x7f7cc403c040 comp rx=0 tx=0).stop 2024-03-06T07:20:03.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 msgr2=0x7f7ccc154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc154440 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00e9b0 tx=0x7f7cbc00ee70 comp rx=0 tx=0).stop 2024-03-06T07:20:03.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 shutdown_connections 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7cb006b500 0x7f7cb006d9c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc155b20 0x7f7ccc14af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ccc1527b0 0x7f7ccc154440 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 --2- 172.21.15.94:0/2136372513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0c7180 0x7f7ccc0bde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 >> 172.21.15.94:0/2136372513 conn(0x7f7ccc0ac560 msgr2=0x7f7ccc0c91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 shutdown_connections 2024-03-06T07:20:03.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.341+0000 7f7cd4039640 1 -- 172.21.15.94:0/2136372513 wait complete. 2024-03-06T07:20:03.354 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 3, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:19:53.385858+0000", "damaged": [], "data_pools": [ 8 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_35084": { "addr": "172.21.15.145:6837/3125091562", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 3125091562, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 3125091562, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35084, "incarnation": 27, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 7, "modified": "2024-03-06T07:19:55.378348+0000", "qdb_cluster": [ 35084 ], "qdb_leader": 35084, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 35084 } } } ], "standbys": [ { "addr": "172.21.15.94:6835/480754107", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 480754107, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 480754107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 26, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5521, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6836/3216323598", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 3216323598, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 3216323598, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 26, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5530, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/1798838613", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 1798838613, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 1798838613, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 26, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5652, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:20:03.354 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:20:03.354 INFO:tasks.cephfs.test_failover:current = [{'gid': 5521, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/480754107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 480754107}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 480754107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 26}, {'gid': 35084, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/3125091562', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3125091562}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3125091562}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5652, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/1798838613', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 1798838613}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 1798838613}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 26}, {'gid': 5530, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/3216323598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3216323598}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3216323598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 26}] 2024-03-06T07:20:03.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs2 2024-03-06T07:20:03.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.456+0000 7fa90dda0640 1 Processor -- start 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.456+0000 7fa90dda0640 1 -- start start 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90dda0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa9081526a0 0x7fa9080c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90dda0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9081559c0 0x7fa9080c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90dda0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9080c5190 con 0x7fa9080c7ed0 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90dda0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90806cbc0 con 0x7fa9081559c0 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90dda0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9080c7220 con 0x7fa9081526a0 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa907fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9081559c0 0x7fa9080c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90d59f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa9080c7ed0 0x7fa9080cab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57354/0 (socket says 172.21.15.94:57354) 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90d59f640 1 -- 172.21.15.94:0/903055564 learned_addr learned my addr 172.21.15.94:0/903055564 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa90cd9e640 1 --2- 172.21.15.94:0/903055564 >> v2:172.21.15.145:3300/0 conn(0x7fa9081526a0 0x7fa9080c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa9077fe640 1 -- 172.21.15.94:0/903055564 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024542793 0 0) 0x7fa9080c5190 con 0x7fa9080c7ed0 2024-03-06T07:20:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa9077fe640 1 -- 172.21.15.94:0/903055564 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f4003520 con 0x7fa9080c7ed0 2024-03-06T07:20:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa907fff640 1 -- 172.21.15.94:0/903055564 >> v2:172.21.15.145:3300/0 conn(0x7fa9081526a0 msgr2=0x7fa9080c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa907fff640 1 --2- 172.21.15.94:0/903055564 >> v2:172.21.15.145:3300/0 conn(0x7fa9081526a0 0x7fa9080c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.457+0000 7fa907fff640 1 -- 172.21.15.94:0/903055564 >> v1:172.21.15.94:6789/0 conn(0x7fa9080c7ed0 legacy=0x7fa9080cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.458+0000 7fa907fff640 1 -- 172.21.15.94:0/903055564 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9080c5430 con 0x7fa9081559c0 2024-03-06T07:20:03.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.458+0000 7fa907fff640 1 --2- 172.21.15.94:0/903055564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081559c0 0x7fa9080c7990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa9080c73a0 tx=0x7fa8f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5fee66b8b9b6c0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.458+0000 7fa9077fe640 1 -- 172.21.15.94:0/903055564 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f00315a0 con 0x7fa9081559c0 2024-03-06T07:20:03.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.458+0000 7fa9077fe640 1 -- 172.21.15.94:0/903055564 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f0037070 con 0x7fa9081559c0 2024-03-06T07:20:03.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.458+0000 7fa9077fe640 1 -- 172.21.15.94:0/903055564 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0031960 con 0x7fa9081559c0 2024-03-06T07:20:03.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 -- 172.21.15.94:0/903055564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081559c0 msgr2=0x7fa9080c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 --2- 172.21.15.94:0/903055564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081559c0 0x7fa9080c7990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa9080c73a0 tx=0x7fa8f002b840 comp rx=0 tx=0).stop 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 -- 172.21.15.94:0/903055564 shutdown_connections 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 --2- 172.21.15.94:0/903055564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081559c0 0x7fa9080c7990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 --2- 172.21.15.94:0/903055564 >> v2:172.21.15.145:3300/0 conn(0x7fa9081526a0 0x7fa9080c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 -- 172.21.15.94:0/903055564 >> 172.21.15.94:0/903055564 conn(0x7fa9080728d0 msgr2=0x7fa908072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 -- 172.21.15.94:0/903055564 shutdown_connections 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.459+0000 7fa90dda0640 1 -- 172.21.15.94:0/903055564 wait complete. 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.460+0000 7fa90dda0640 1 Processor -- start 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.460+0000 7fa90dda0640 1 -- start start 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90dda0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081526a0 0x7fa9080acc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90dda0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9081559c0 0x7fa9081540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90dda0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9080c4180 0x7fa9080bea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90dda0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9080c7470 con 0x7fa9081526a0 2024-03-06T07:20:03.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90dda0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9080c59f0 con 0x7fa9081559c0 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90dda0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa908155e00 con 0x7fa9080c4180 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081526a0 0x7fa9080acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081526a0 0x7fa9080acc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51746/0 (socket says 172.21.15.94:51746) 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 -- 172.21.15.94:0/3378782624 learned_addr learned my addr 172.21.15.94:0/3378782624 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa907fff640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9081559c0 0x7fa9081540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90d59f640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9080c4180 0x7fa9080bea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 -- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9080c4180 msgr2=0x7fa9080bea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9080c4180 0x7fa9080bea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 -- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9081559c0 msgr2=0x7fa9081540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9081559c0 0x7fa9081540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 -- 172.21.15.94:0/3378782624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9080ad180 con 0x7fa9081526a0 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa907fff640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9081559c0 0x7fa9081540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:03.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90cd9e640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081526a0 0x7fa9080acc40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e9e0 tx=0x7fa8f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa9057fa640 1 -- 172.21.15.94:0/3378782624 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8017070 con 0x7fa9081526a0 2024-03-06T07:20:03.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa9057fa640 1 -- 172.21.15.94:0/3378782624 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f800c9d0 con 0x7fa9081526a0 2024-03-06T07:20:03.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa9057fa640 1 -- 172.21.15.94:0/3378782624 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f800cdc0 con 0x7fa9081526a0 2024-03-06T07:20:03.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.461+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9081eb410 con 0x7fa9081526a0 2024-03-06T07:20:03.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.462+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9080c0220 con 0x7fa9081526a0 2024-03-06T07:20:03.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.462+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9080693c0 con 0x7fa9081526a0 2024-03-06T07:20:03.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.463+0000 7fa9057fa640 1 -- 172.21.15.94:0/3378782624 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8f801a070 con 0x7fa9081526a0 2024-03-06T07:20:03.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.463+0000 7fa9057fa640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa8e806b4d0 0x7fa8e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.463+0000 7fa9057fa640 1 -- 172.21.15.94:0/3378782624 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa8f808a340 con 0x7fa9081526a0 2024-03-06T07:20:03.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.463+0000 7fa907fff640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa8e806b4d0 0x7fa8e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.464+0000 7fa907fff640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa8e806b4d0 0x7fa8e806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0036000 tx=0x7fa8f003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.467+0000 7fa9057fa640 1 -- 172.21.15.94:0/3378782624 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8f80590c0 con 0x7fa9081526a0 2024-03-06T07:20:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.638+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fa9080cb2b0 con 0x7fa9081526a0 2024-03-06T07:20:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.641+0000 7fa9057fa640 1 -- 172.21.15.94:0/3378782624 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v16)=0 v16) v1 ==== 117+0+0 (secure 0 0 0) 0x7fa8f80589c0 con 0x7fa9081526a0 2024-03-06T07:20:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa8e806b4d0 msgr2=0x7fa8e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa8e806b4d0 0x7fa8e806d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0036000 tx=0x7fa8f003d040 comp rx=0 tx=0).stop 2024-03-06T07:20:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081526a0 msgr2=0x7fa9080acc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081526a0 0x7fa9080acc40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e9e0 tx=0x7fa8f800eea0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 shutdown_connections 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa8e806b4d0 0x7fa8e806d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9080c4180 0x7fa9080bea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9081559c0 0x7fa9081540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 --2- 172.21.15.94:0/3378782624 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9081526a0 0x7fa9080acc40 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 >> 172.21.15.94:0/3378782624 conn(0x7fa9080728d0 msgr2=0x7fa90814fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 shutdown_connections 2024-03-06T07:20:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.643+0000 7fa90dda0640 1 -- 172.21.15.94:0/3378782624 wait complete. 2024-03-06T07:20:03.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-03-06T07:20:03.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.757+0000 7f93a8be9640 1 Processor -- start 2024-03-06T07:20:03.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a8be9640 1 -- start start 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a8be9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f93a40a6290 0x7f93a40a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a8be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40aa360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a8be9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a40aa8a0 con 0x7f93a40a8ff0 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a8be9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a414f850 con 0x7f93a40a6290 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a8be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a4156260 con 0x7f93a4155cf0 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f93a40a8ff0 0x7f93a40ab1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57356/0 (socket says 172.21.15.94:57356) 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a259c640 1 -- 172.21.15.94:0/506141842 learned_addr learned my addr 172.21.15.94:0/506141842 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f93a2d9d640 1 --2- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40aa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.758+0000 7f939bfff640 1 --2- 172.21.15.94:0/506141842 >> v2:172.21.15.145:3300/0 conn(0x7f93a40a6290 0x7f93a40a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.759+0000 7f93a1d9b640 1 -- 172.21.15.94:0/506141842 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113777408 0 0) 0x7f93a40aa8a0 con 0x7f93a40a8ff0 2024-03-06T07:20:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.759+0000 7f93a1d9b640 1 -- 172.21.15.94:0/506141842 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9390003520 con 0x7f93a40a8ff0 2024-03-06T07:20:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.759+0000 7f939bfff640 1 -- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 msgr2=0x7f93a40aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.759+0000 7f939bfff640 1 --2- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40aa360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.759+0000 7f939bfff640 1 -- 172.21.15.94:0/506141842 >> v1:172.21.15.94:6789/0 conn(0x7f93a40a8ff0 legacy=0x7f93a40ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.759+0000 7f939bfff640 1 -- 172.21.15.94:0/506141842 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a40c1f90 con 0x7f93a40a6290 2024-03-06T07:20:03.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.759+0000 7f93a2d9d640 1 --2- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40aa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f939bfff640 1 --2- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40a6290 0x7f93a40a9e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f93a41563e0 tx=0x7f938c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824239950a9e353a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a1d9b640 1 -- 172.21.15.94:0/506141842 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c039070 con 0x7f93a40a6290 2024-03-06T07:20:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a1d9b640 1 -- 172.21.15.94:0/506141842 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c0316f0 con 0x7f93a40a6290 2024-03-06T07:20:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a1d9b640 1 -- 172.21.15.94:0/506141842 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c031ac0 con 0x7f93a40a6290 2024-03-06T07:20:03.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 -- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40a6290 msgr2=0x7f93a40a9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 --2- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40a6290 0x7f93a40a9e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f93a41563e0 tx=0x7f938c02b790 comp rx=0 tx=0).stop 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 -- 172.21.15.94:0/506141842 shutdown_connections 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 --2- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40aa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 --2- 172.21.15.94:0/506141842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40a6290 0x7f93a40a9e20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 -- 172.21.15.94:0/506141842 >> 172.21.15.94:0/506141842 conn(0x7f93a406afc0 msgr2=0x7f93a406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 -- 172.21.15.94:0/506141842 shutdown_connections 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.760+0000 7f93a8be9640 1 -- 172.21.15.94:0/506141842 wait complete. 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 Processor -- start 2024-03-06T07:20:03.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 -- start start 2024-03-06T07:20:03.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93a40a8ff0 0x7f93a4147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40c0010 0x7f93a41ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40c1790 con 0x7f93a40a8ff0 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40c14b0 con 0x7f93a40c0010 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a8be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40c1a70 con 0x7f93a4155cf0 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93a40a8ff0 0x7f93a4147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93a40a8ff0 0x7f93a4147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51748/0 (socket says 172.21.15.94:51748) 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a259c640 1 -- 172.21.15.94:0/3941732379 learned_addr learned my addr 172.21.15.94:0/3941732379 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f939bfff640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.762+0000 7f93a2d9d640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40c0010 0x7f93a41ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a259c640 1 -- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 msgr2=0x7f93a40bfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a259c640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40bfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a259c640 1 -- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40c0010 msgr2=0x7f93a41ea6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a259c640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40c0010 0x7f93a41ea6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a259c640 1 -- 172.21.15.94:0/3941732379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a41ede60 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a259c640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93a40a8ff0 0x7f93a4147410 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f939400c4d0 tx=0x7f939400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f939affd640 1 -- 172.21.15.94:0/3941732379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394017070 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f939affd640 1 -- 172.21.15.94:0/3941732379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9394005ce0 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f939affd640 1 -- 172.21.15.94:0/3941732379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394007310 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a41ee140 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.763+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a41ee700 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.764+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a404f330 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.764+0000 7f939affd640 1 -- 172.21.15.94:0/3941732379 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f939401a070 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.765+0000 7f939affd640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b500 0x7f938406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.765+0000 7f939affd640 1 -- 172.21.15.94:0/3941732379 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f93940895f0 con 0x7f93a40a8ff0 2024-03-06T07:20:03.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.765+0000 7f939bfff640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b500 0x7f938406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:03.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.765+0000 7f939bfff640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b500 0x7f938406d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f938c036000 tx=0x7f938c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:03.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.769+0000 7f939affd640 1 -- 172.21.15.94:0/3941732379 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9394057d20 con 0x7f93a40a8ff0 2024-03-06T07:20:03.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.940+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f93a41ee8d0 con 0x7f93a40a8ff0 2024-03-06T07:20:03.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.943+0000 7f939affd640 1 -- 172.21.15.94:0/3941732379 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v17)=0 v17) v1 ==== 117+0+0 (secure 0 0 0) 0x7f939405bdc0 con 0x7f93a40a8ff0 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.944+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b500 msgr2=0x7f938406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b500 0x7f938406d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f938c036000 tx=0x7f938c008030 comp rx=0 tx=0).stop 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93a40a8ff0 msgr2=0x7f93a4147410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93a40a8ff0 0x7f93a4147410 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f939400c4d0 tx=0x7f939400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 shutdown_connections 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b500 0x7f938406d9c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93a40c0010 0x7f93a41ea6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93a4155cf0 0x7f93a40bfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 --2- 172.21.15.94:0/3941732379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93a40a8ff0 0x7f93a4147410 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:03.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 >> 172.21.15.94:0/3941732379 conn(0x7f93a406afc0 msgr2=0x7f93a406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:03.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 shutdown_connections 2024-03-06T07:20:03.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:03.945+0000 7f93a8be9640 1 -- 172.21.15.94:0/3941732379 wait complete. 2024-03-06T07:20:03.957 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-03-06T07:20:04.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.059+0000 7f0568df6640 1 Processor -- start 2024-03-06T07:20:04.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0568df6640 1 -- start start 2024-03-06T07:20:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0568df6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05640c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0568df6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f05640c77d0 0x7f056406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0568df6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0564155260 con 0x7f05640c7370 2024-03-06T07:20:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0568df6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05640c7c00 con 0x7f05640c77d0 2024-03-06T07:20:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0568df6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0564155540 con 0x7f05641527f0 2024-03-06T07:20:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0562d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f05640c77d0 0x7f056406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0562d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f05640c77d0 0x7f056406c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51754/0 (socket says 172.21.15.94:51754) 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f0562d9d640 1 -- 172.21.15.94:0/1698714202 learned_addr learned my addr 172.21.15.94:0/1698714202 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.060+0000 7f056259c640 1 --2- 172.21.15.94:0/1698714202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05640c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.061+0000 7f0562d9d640 1 -- 172.21.15.94:0/1698714202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 msgr2=0x7f05640c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.061+0000 7f0562d9d640 1 --2- 172.21.15.94:0/1698714202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05640c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.061+0000 7f0562d9d640 1 -- 172.21.15.94:0/1698714202 >> v1:172.21.15.145:6789/0 conn(0x7f05640c7370 legacy=0x7f056406c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.061+0000 7f0562d9d640 1 -- 172.21.15.94:0/1698714202 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0564155820 con 0x7f05640c77d0 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.061+0000 7f0562d9d640 1 --2- 172.21.15.94:0/1698714202 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c77d0 0x7f056406c7c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f055c009e50 tx=0x7f055c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbd43f2927b6dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0561d9b640 1 -- 172.21.15.94:0/1698714202 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c02c9e0 con 0x7f05640c77d0 2024-03-06T07:20:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0561d9b640 1 -- 172.21.15.94:0/1698714202 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055c02cb80 con 0x7f05640c77d0 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0561d9b640 1 -- 172.21.15.94:0/1698714202 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c0343f0 con 0x7f05640c77d0 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 -- 172.21.15.94:0/1698714202 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c77d0 msgr2=0x7f056406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 --2- 172.21.15.94:0/1698714202 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c77d0 0x7f056406c7c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f055c009e50 tx=0x7f055c02be40 comp rx=0 tx=0).stop 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 -- 172.21.15.94:0/1698714202 shutdown_connections 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 --2- 172.21.15.94:0/1698714202 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c77d0 0x7f056406c7c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 --2- 172.21.15.94:0/1698714202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05640c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 -- 172.21.15.94:0/1698714202 >> 172.21.15.94:0/1698714202 conn(0x7f05640ac4f0 msgr2=0x7f05640ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 -- 172.21.15.94:0/1698714202 shutdown_connections 2024-03-06T07:20:04.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.062+0000 7f0568df6640 1 -- 172.21.15.94:0/1698714202 wait complete. 2024-03-06T07:20:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.063+0000 7f0568df6640 1 Processor -- start 2024-03-06T07:20:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.063+0000 7f0568df6640 1 -- start start 2024-03-06T07:20:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f0568df6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c7370 0x7f05641eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f0568df6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05640c77d0 0x7f05641ebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f0568df6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05641eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f0568df6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05640be1a0 con 0x7f05640c7370 2024-03-06T07:20:04.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f0568df6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056406cd00 con 0x7f05640c77d0 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f0568df6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05640c8da0 con 0x7f05641527f0 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c7370 0x7f05641eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c7370 0x7f05641eeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51764/0 (socket says 172.21.15.94:51764) 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 -- 172.21.15.94:0/3024254225 learned_addr learned my addr 172.21.15.94:0/3024254225 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f055bfff640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05640c77d0 0x7f05641ebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f0562d9d640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05641eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 -- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 msgr2=0x7f05641eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05641eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 -- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05640c77d0 msgr2=0x7f05641ebfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05640c77d0 0x7f05641ebfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.064+0000 7f056259c640 1 -- 172.21.15.94:0/3024254225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05640c4720 con 0x7f05640c7370 2024-03-06T07:20:04.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.065+0000 7f056259c640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c7370 0x7f05641eeaa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f055400e870 tx=0x7f055400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.065+0000 7f055affd640 1 -- 172.21.15.94:0/3024254225 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0554017070 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.065+0000 7f055affd640 1 -- 172.21.15.94:0/3024254225 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f055400c940 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.065+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05640c4440 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.065+0000 7f055affd640 1 -- 172.21.15.94:0/3024254225 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055400cd30 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.065+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05641eaf50 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.066+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056404f330 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.066+0000 7f055affd640 1 -- 172.21.15.94:0/3024254225 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f055401a070 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.067+0000 7f055affd640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f054406b4d0 0x7f054406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.067+0000 7f055affd640 1 -- 172.21.15.94:0/3024254225 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f055408a470 con 0x7f05640c7370 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.067+0000 7f055bfff640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f054406b4d0 0x7f054406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.067+0000 7f055bfff640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f054406b4d0 0x7f054406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0564155000 tx=0x7f054c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.071+0000 7f055affd640 1 -- 172.21.15.94:0/3024254225 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0554058ba0 con 0x7f05640c7370 2024-03-06T07:20:04.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.243+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f056414c590 con 0x7f05640c7370 2024-03-06T07:20:04.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.246+0000 7f055affd640 1 -- 172.21.15.94:0/3024254225 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v18)=0 v18) v1 ==== 117+0+0 (secure 0 0 0) 0x7f055405cc40 con 0x7f05640c7370 2024-03-06T07:20:04.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f054406b4d0 msgr2=0x7f054406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f054406b4d0 0x7f054406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0564155000 tx=0x7f054c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:04.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c7370 msgr2=0x7f05641eeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c7370 0x7f05641eeaa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f055400e870 tx=0x7f055400ed30 comp rx=0 tx=0).stop 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 shutdown_connections 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f054406b4d0 0x7f054406d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05641527f0 0x7f05641eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05640c77d0 0x7f05641ebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 --2- 172.21.15.94:0/3024254225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05640c7370 0x7f05641eeaa0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 >> 172.21.15.94:0/3024254225 conn(0x7f05640ac4f0 msgr2=0x7f056406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 shutdown_connections 2024-03-06T07:20:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.248+0000 7f0568df6640 1 -- 172.21.15.94:0/3024254225 wait complete. 2024-03-06T07:20:04.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:04.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.359+0000 7f3fd875e640 1 Processor -- start 2024-03-06T07:20:04.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.359+0000 7f3fd875e640 1 -- start start 2024-03-06T07:20:04.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd875e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3fd0150580 0x7f3fd006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd875e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3fd00c1320 0x7f3fd006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd875e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd0153a60 con 0x7f3fd00c6000 2024-03-06T07:20:04.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd875e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd0153d40 con 0x7f3fd00c1320 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd875e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd006c2c0 con 0x7f3fd0150580 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd5cf9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3fd00c1320 0x7f3fd006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd6cfb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3fd00c6000 0x7f3fd01566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57360/0 (socket says 172.21.15.94:57360) 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd6cfb640 1 -- 172.21.15.94:0/187547108 learned_addr learned my addr 172.21.15.94:0/187547108 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd64fa640 1 --2- 172.21.15.94:0/187547108 >> v2:172.21.15.145:3300/0 conn(0x7f3fd0150580 0x7f3fd006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd54f8640 1 -- 172.21.15.94:0/187547108 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268357632 0 0) 0x7f3fd0153a60 con 0x7f3fd00c6000 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd54f8640 1 -- 172.21.15.94:0/187547108 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fbc003520 con 0x7f3fd00c6000 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd5cf9640 1 -- 172.21.15.94:0/187547108 >> v2:172.21.15.145:3300/0 conn(0x7f3fd0150580 msgr2=0x7f3fd006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd5cf9640 1 --2- 172.21.15.94:0/187547108 >> v2:172.21.15.145:3300/0 conn(0x7f3fd0150580 0x7f3fd006dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd5cf9640 1 -- 172.21.15.94:0/187547108 >> v1:172.21.15.94:6789/0 conn(0x7f3fd00c6000 legacy=0x7f3fd01566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.360+0000 7f3fd5cf9640 1 -- 172.21.15.94:0/187547108 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd006c5a0 con 0x7f3fd00c1320 2024-03-06T07:20:04.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.361+0000 7f3fd5cf9640 1 --2- 172.21.15.94:0/187547108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd006e250 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00a9550 tx=0x7f3fc002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0c33e25a165295 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.361+0000 7f3fd54f8640 1 -- 172.21.15.94:0/187547108 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc00315a0 con 0x7f3fd00c1320 2024-03-06T07:20:04.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.361+0000 7f3fd54f8640 1 -- 172.21.15.94:0/187547108 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc0037070 con 0x7f3fd00c1320 2024-03-06T07:20:04.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.361+0000 7f3fd54f8640 1 -- 172.21.15.94:0/187547108 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0031960 con 0x7f3fd00c1320 2024-03-06T07:20:04.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 -- 172.21.15.94:0/187547108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 msgr2=0x7f3fd006e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 --2- 172.21.15.94:0/187547108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd006e250 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00a9550 tx=0x7f3fc002b840 comp rx=0 tx=0).stop 2024-03-06T07:20:04.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 -- 172.21.15.94:0/187547108 shutdown_connections 2024-03-06T07:20:04.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 --2- 172.21.15.94:0/187547108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd006e250 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 --2- 172.21.15.94:0/187547108 >> v2:172.21.15.145:3300/0 conn(0x7f3fd0150580 0x7f3fd006dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 -- 172.21.15.94:0/187547108 >> 172.21.15.94:0/187547108 conn(0x7f3fd00c6710 msgr2=0x7f3fd00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 -- 172.21.15.94:0/187547108 shutdown_connections 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.362+0000 7f3fd875e640 1 -- 172.21.15.94:0/187547108 wait complete. 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.363+0000 7f3fd875e640 1 Processor -- start 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.363+0000 7f3fd875e640 1 -- start start 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd00ab570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd00c6000 0x7f3fd0152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd0150580 0x7f3fd01eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00c0dc0 con 0x7f3fd00c1320 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00c0c40 con 0x7f3fd00c6000 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd0156dc0 con 0x7f3fd0150580 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd00ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd00ab570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51782/0 (socket says 172.21.15.94:51782) 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 -- 172.21.15.94:0/2601636445 learned_addr learned my addr 172.21.15.94:0/2601636445 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd6cfb640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd0150580 0x7f3fd01eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd5cf9640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd00c6000 0x7f3fd0152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 -- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd0150580 msgr2=0x7f3fd01eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd0150580 0x7f3fd01eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 -- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd00c6000 msgr2=0x7f3fd0152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd00c6000 0x7f3fd0152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 -- 172.21.15.94:0/2601636445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd00a74b0 con 0x7f3fd00c1320 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd64fa640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd00ab570 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00e8a0 tx=0x7f3fcc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fc6ffd640 1 -- 172.21.15.94:0/2601636445 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc017070 con 0x7f3fd00c1320 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fc6ffd640 1 -- 172.21.15.94:0/2601636445 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fcc00c940 con 0x7f3fd00c1320 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd00a7790 con 0x7f3fd00c1320 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fc6ffd640 1 -- 172.21.15.94:0/2601636445 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc00cd30 con 0x7f3fd00c1320 2024-03-06T07:20:04.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.364+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd00a7d50 con 0x7f3fd00c1320 2024-03-06T07:20:04.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.365+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd004f330 con 0x7f3fd00c1320 2024-03-06T07:20:04.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.366+0000 7f3fc6ffd640 1 -- 172.21.15.94:0/2601636445 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fcc01a070 con 0x7f3fd00c1320 2024-03-06T07:20:04.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.366+0000 7f3fc6ffd640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fb406b4d0 0x7f3fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.366+0000 7f3fd5cf9640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fb406b4d0 0x7f3fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.366+0000 7f3fc6ffd640 1 -- 172.21.15.94:0/2601636445 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3fcc08a480 con 0x7f3fd00c1320 2024-03-06T07:20:04.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.367+0000 7f3fd5cf9640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fb406b4d0 0x7f3fb406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0036000 tx=0x7f3fc003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.371+0000 7f3fc6ffd640 1 -- 172.21.15.94:0/2601636445 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fcc058bb0 con 0x7f3fd00c1320 2024-03-06T07:20:04.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.543+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fd014b930 con 0x7f3fd00c1320 2024-03-06T07:20:04.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.544+0000 7f3fc6ffd640 1 -- 172.21.15.94:0/2601636445 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16942 (secure 0 0 0) 0x7f3fcc05cc50 con 0x7f3fd00c1320 2024-03-06T07:20:04.546 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:04.546 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":52,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:19:55.029727+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:04.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fb406b4d0 msgr2=0x7f3fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fb406b4d0 0x7f3fb406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0036000 tx=0x7f3fc003d040 comp rx=0 tx=0).stop 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 msgr2=0x7f3fd00ab570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd00ab570 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00e8a0 tx=0x7f3fcc00ed60 comp rx=0 tx=0).stop 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 shutdown_connections 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fb406b4d0 0x7f3fb406d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd0150580 0x7f3fd01eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd00c6000 0x7f3fd0152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 --2- 172.21.15.94:0/2601636445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fd00c1320 0x7f3fd00ab570 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 >> 172.21.15.94:0/2601636445 conn(0x7f3fd00c6710 msgr2=0x7f3fd00c4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 shutdown_connections 2024-03-06T07:20:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.546+0000 7f3fd875e640 1 -- 172.21.15.94:0/2601636445 wait complete. 2024-03-06T07:20:04.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:19:51.596832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:19:52.352004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:20:04.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:04.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.662+0000 7fd07a5b6640 1 Processor -- start 2024-03-06T07:20:04.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.662+0000 7fd07a5b6640 1 -- start start 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.662+0000 7fd07a5b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07406c1f0 0x7fd0740c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.662+0000 7fd07a5b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd07406cad0 0x7fd074154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.662+0000 7fd07a5b6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0740c2790 con 0x7fd07414ef70 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.662+0000 7fd07a5b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0740c3090 con 0x7fd07406cad0 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.662+0000 7fd07a5b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07414f350 con 0x7fd07406c1f0 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd073fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd07414ef70 0x7fd0740c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57376/0 (socket says 172.21.15.94:57376) 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd073fff640 1 -- 172.21.15.94:0/1522103945 learned_addr learned my addr 172.21.15.94:0/1522103945 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd078b53640 1 --2- 172.21.15.94:0/1522103945 >> v2:172.21.15.145:3300/0 conn(0x7fd07406cad0 0x7fd074154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd0737fe640 1 --2- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07406c1f0 0x7fd0740c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd072ffd640 1 -- 172.21.15.94:0/1522103945 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782096149 0 0) 0x7fd0740c2790 con 0x7fd07414ef70 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd072ffd640 1 -- 172.21.15.94:0/1522103945 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd05c004630 con 0x7fd07414ef70 2024-03-06T07:20:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd078b53640 1 -- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07406c1f0 msgr2=0x7fd0740c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd078b53640 1 --2- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07406c1f0 0x7fd0740c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd078b53640 1 -- 172.21.15.94:0/1522103945 >> v1:172.21.15.94:6789/0 conn(0x7fd07414ef70 legacy=0x7fd0740c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd078b53640 1 -- 172.21.15.94:0/1522103945 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0740c1540 con 0x7fd07406cad0 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.663+0000 7fd0737fe640 1 --2- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07406c1f0 0x7fd0740c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd078b53640 1 --2- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 0x7fd074154c30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009d90 tx=0x7fd06c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d5b5048b67aefde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd072ffd640 1 -- 172.21.15.94:0/1522103945 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c039070 con 0x7fd07406cad0 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd072ffd640 1 -- 172.21.15.94:0/1522103945 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c02cbc0 con 0x7fd07406cad0 2024-03-06T07:20:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd072ffd640 1 -- 172.21.15.94:0/1522103945 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c034440 con 0x7fd07406cad0 2024-03-06T07:20:04.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 msgr2=0x7fd074154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 0x7fd074154c30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009d90 tx=0x7fd06c02be10 comp rx=0 tx=0).stop 2024-03-06T07:20:04.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1522103945 shutdown_connections 2024-03-06T07:20:04.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 0x7fd074154c30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1522103945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07406c1f0 0x7fd0740c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1522103945 >> 172.21.15.94:0/1522103945 conn(0x7fd0740c6710 msgr2=0x7fd0740c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1522103945 shutdown_connections 2024-03-06T07:20:04.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.664+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1522103945 wait complete. 2024-03-06T07:20:04.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 Processor -- start 2024-03-06T07:20:04.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 -- start start 2024-03-06T07:20:04.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd07406c1f0 0x7fd07414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 0x7fd07414b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07414ef70 0x7fd07414b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd074155750 con 0x7fd07406c1f0 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07406cf60 con 0x7fd07406cad0 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd07a5b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd074155170 con 0x7fd07414ef70 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.666+0000 7fd073fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd07406c1f0 0x7fd07414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd07406c1f0 0x7fd07414ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51788/0 (socket says 172.21.15.94:51788) 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 -- 172.21.15.94:0/1364105115 learned_addr learned my addr 172.21.15.94:0/1364105115 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd0737fe640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 0x7fd07414b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd078b53640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07414ef70 0x7fd07414b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 -- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07414ef70 msgr2=0x7fd07414b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07414ef70 0x7fd07414b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 -- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 msgr2=0x7fd07414b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 0x7fd07414b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 -- 172.21.15.94:0/1364105115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0741dd2a0 con 0x7fd07406c1f0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd073fff640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd07406c1f0 0x7fd07414ce40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd06400c3c0 tx=0x7fd06400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd070ff9640 1 -- 172.21.15.94:0/1364105115 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd064017070 con 0x7fd07406c1f0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd070ff9640 1 -- 172.21.15.94:0/1364105115 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd064005ce0 con 0x7fd07406c1f0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd070ff9640 1 -- 172.21.15.94:0/1364105115 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0640072f0 con 0x7fd07406c1f0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0741528b0 con 0x7fd07406c1f0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.667+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07414d380 con 0x7fd07406c1f0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.668+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd074152dd0 con 0x7fd07406c1f0 2024-03-06T07:20:04.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.668+0000 7fd070ff9640 1 -- 172.21.15.94:0/1364105115 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd06401a070 con 0x7fd07406c1f0 2024-03-06T07:20:04.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.669+0000 7fd070ff9640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06006b4d0 0x7fd06006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.669+0000 7fd070ff9640 1 -- 172.21.15.94:0/1364105115 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd0640895f0 con 0x7fd07406c1f0 2024-03-06T07:20:04.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.669+0000 7fd0737fe640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06006b4d0 0x7fd06006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.669+0000 7fd0737fe640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06006b4d0 0x7fd06006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd068009560 tx=0x7fd068009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.673+0000 7fd070ff9640 1 -- 172.21.15.94:0/1364105115 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd074152dd0 con 0x7fd07406c1f0 2024-03-06T07:20:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.843+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd07404f330 con 0x7fd07406c1f0 2024-03-06T07:20:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.843+0000 7fd070ff9640 1 -- 172.21.15.94:0/1364105115 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd064057d20 con 0x7fd07406c1f0 2024-03-06T07:20:04.846 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:20:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06006b4d0 msgr2=0x7fd06006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06006b4d0 0x7fd06006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd068009560 tx=0x7fd068009190 comp rx=0 tx=0).stop 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd07406c1f0 msgr2=0x7fd07414ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd07406c1f0 0x7fd07414ce40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd06400c3c0 tx=0x7fd06400c880 comp rx=0 tx=0).stop 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 shutdown_connections 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd06006b4d0 0x7fd06006d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd07414ef70 0x7fd07414b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd07406cad0 0x7fd07414b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 --2- 172.21.15.94:0/1364105115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd07406c1f0 0x7fd07414ce40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 >> 172.21.15.94:0/1364105115 conn(0x7fd0740c6710 msgr2=0x7fd074154000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 shutdown_connections 2024-03-06T07:20:04.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.845+0000 7fd07a5b6640 1 -- 172.21.15.94:0/1364105115 wait complete. 2024-03-06T07:20:04.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:04.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.960+0000 7f44db4df640 1 Processor -- start 2024-03-06T07:20:04.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.960+0000 7f44db4df640 1 -- start start 2024-03-06T07:20:04.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.961+0000 7f44db4df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406d750 0x7f44d40732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.961+0000 7f44db4df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d4156060 con 0x7f44d406db30 2024-03-06T07:20:04.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.961+0000 7f44db4df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d40a8430 con 0x7f44d40c3a70 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.961+0000 7f44db4df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d4155b30 con 0x7f44d406d750 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.961+0000 7f44d9a7c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f44d406db30 0x7f44d40c1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57380/0 (socket says 172.21.15.94:57380) 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.961+0000 7f44d9a7c640 1 -- 172.21.15.94:0/3882676154 learned_addr learned my addr 172.21.15.94:0/3882676154 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.961+0000 7f44d927b640 1 --2- 172.21.15.94:0/3882676154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406d750 0x7f44d40732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44cbfff640 1 -- 172.21.15.94:0/3882676154 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757326676 0 0) 0x7f44d4156060 con 0x7f44d406db30 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44cbfff640 1 -- 172.21.15.94:0/3882676154 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c0003520 con 0x7f44d406db30 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44cbfff640 1 -- 172.21.15.94:0/3882676154 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038009063 0 0) 0x7f44d40a8430 con 0x7f44d40c3a70 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44cbfff640 1 -- 172.21.15.94:0/3882676154 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4156060 con 0x7f44d40c3a70 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44d927b640 1 -- 172.21.15.94:0/3882676154 >> v1:172.21.15.145:6789/0 conn(0x7f44d40c3a70 legacy=0x7f44d4072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44d927b640 1 -- 172.21.15.94:0/3882676154 >> v1:172.21.15.94:6789/0 conn(0x7f44d406db30 legacy=0x7f44d40c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44d927b640 1 -- 172.21.15.94:0/3882676154 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d4155d80 con 0x7f44d406d750 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.962+0000 7f44d927b640 1 --2- 172.21.15.94:0/3882676154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406d750 0x7f44d40732a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009dc0 tx=0x7f44d002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bdd97615285fb1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44cbfff640 1 -- 172.21.15.94:0/3882676154 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0039070 con 0x7f44d406d750 2024-03-06T07:20:04.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44cbfff640 1 -- 172.21.15.94:0/3882676154 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d002cbb0 con 0x7f44d406d750 2024-03-06T07:20:04.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44cbfff640 1 -- 172.21.15.94:0/3882676154 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0034440 con 0x7f44d406d750 2024-03-06T07:20:04.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44db4df640 1 -- 172.21.15.94:0/3882676154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406d750 msgr2=0x7f44d40732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44db4df640 1 --2- 172.21.15.94:0/3882676154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406d750 0x7f44d40732a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009dc0 tx=0x7f44d002be00 comp rx=0 tx=0).stop 2024-03-06T07:20:04.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44db4df640 1 -- 172.21.15.94:0/3882676154 shutdown_connections 2024-03-06T07:20:04.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44db4df640 1 --2- 172.21.15.94:0/3882676154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406d750 0x7f44d40732a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44db4df640 1 -- 172.21.15.94:0/3882676154 >> 172.21.15.94:0/3882676154 conn(0x7f44d40c5560 msgr2=0x7f44d40c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:04.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44db4df640 1 -- 172.21.15.94:0/3882676154 shutdown_connections 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.963+0000 7f44db4df640 1 -- 172.21.15.94:0/3882676154 wait complete. 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.964+0000 7f44db4df640 1 Processor -- start 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.964+0000 7f44db4df640 1 -- start start 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44db4df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d406d750 0x7f44d414d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44db4df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406db30 0x7f44d406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44db4df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d40c3a70 0x7f44d4153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44db4df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40c2720 con 0x7f44d406d750 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44db4df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40c25a0 con 0x7f44d40c3a70 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44db4df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40a7cd0 con 0x7f44d406db30 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d406d750 0x7f44d414d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d406d750 0x7f44d414d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51794/0 (socket says 172.21.15.94:51794) 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 -- 172.21.15.94:0/547347600 learned_addr learned my addr 172.21.15.94:0/547347600 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d8a7a640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406db30 0x7f44d406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d9a7c640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d40c3a70 0x7f44d4153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 -- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406db30 msgr2=0x7f44d406f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406db30 0x7f44d406f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 -- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d40c3a70 msgr2=0x7f44d4153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d40c3a70 0x7f44d4153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.965+0000 7f44d927b640 1 -- 172.21.15.94:0/547347600 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d41edde0 con 0x7f44d406d750 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.966+0000 7f44d927b640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d406d750 0x7f44d414d2d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f44d0000f80 tx=0x7f44d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.966+0000 7f44c9ffb640 1 -- 172.21.15.94:0/547347600 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0039050 con 0x7f44d406d750 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.966+0000 7f44c9ffb640 1 -- 172.21.15.94:0/547347600 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0045070 con 0x7f44d406d750 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.966+0000 7f44c9ffb640 1 -- 172.21.15.94:0/547347600 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0009450 con 0x7f44d406d750 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.966+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d41ee080 con 0x7f44d406d750 2024-03-06T07:20:04.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.966+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d414d810 con 0x7f44d406d750 2024-03-06T07:20:04.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.967+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d404f330 con 0x7f44d406d750 2024-03-06T07:20:04.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.967+0000 7f44c9ffb640 1 -- 172.21.15.94:0/547347600 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44d003e850 con 0x7f44d406d750 2024-03-06T07:20:04.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.968+0000 7f44c9ffb640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b806b4d0 0x7f44b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:04.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.968+0000 7f44c9ffb640 1 -- 172.21.15.94:0/547347600 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f44d00ad240 con 0x7f44d406d750 2024-03-06T07:20:04.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.968+0000 7f44d8a7a640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b806b4d0 0x7f44b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:04.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.968+0000 7f44d8a7a640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b806b4d0 0x7f44b806d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f44c4007fd0 tx=0x7f44c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:04.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:04.972+0000 7f44c9ffb640 1 -- 172.21.15.94:0/547347600 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44d007ba00 con 0x7f44d406d750 2024-03-06T07:20:05.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.143+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44d4155500 con 0x7f44d406d750 2024-03-06T07:20:05.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.144+0000 7f44c9ffb640 1 -- 172.21.15.94:0/547347600 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f44d007faa0 con 0x7f44d406d750 2024-03-06T07:20:05.146 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b806b4d0 msgr2=0x7f44b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b806b4d0 0x7f44b806d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f44c4007fd0 tx=0x7f44c4009040 comp rx=0 tx=0).stop 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d406d750 msgr2=0x7f44d414d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d406d750 0x7f44d414d2d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f44d0000f80 tx=0x7f44d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 shutdown_connections 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b806b4d0 0x7f44b806d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d40c3a70 0x7f44d4153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d406db30 0x7f44d406f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 --2- 172.21.15.94:0/547347600 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d406d750 0x7f44d414d2d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 >> 172.21.15.94:0/547347600 conn(0x7f44d40c5560 msgr2=0x7f44d40c1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 shutdown_connections 2024-03-06T07:20:05.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.146+0000 7f44db4df640 1 -- 172.21.15.94:0/547347600 wait complete. 2024-03-06T07:20:05.158 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:05.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.261+0000 7f3c9d064640 1 Processor -- start 2024-03-06T07:20:05.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.261+0000 7f3c9d064640 1 -- start start 2024-03-06T07:20:05.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9d064640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3c9806b3a0 0x7f3c98158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9d064640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3c98159160 0x7f3c9815fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9d064640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9806b800 con 0x7f3c9806afc0 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9d064640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c98158790 con 0x7f3c9806b3a0 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9d064640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980c5840 con 0x7f3c98159160 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3c9806b3a0 0x7f3c98158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3c9806b3a0 0x7f3c98158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51810/0 (socket says 172.21.15.94:51810) 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9659c640 1 -- 172.21.15.94:0/144342735 learned_addr learned my addr 172.21.15.94:0/144342735 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9759e640 1 --2- 172.21.15.94:0/144342735 >> v2:172.21.15.145:3300/0 conn(0x7f3c98159160 0x7f3c9815fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9659c640 1 -- 172.21.15.94:0/144342735 >> v2:172.21.15.145:3300/0 conn(0x7f3c98159160 msgr2=0x7f3c9815fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9659c640 1 --2- 172.21.15.94:0/144342735 >> v2:172.21.15.145:3300/0 conn(0x7f3c98159160 0x7f3c9815fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9659c640 1 -- 172.21.15.94:0/144342735 >> v1:172.21.15.145:6789/0 conn(0x7f3c9806afc0 legacy=0x7f3c980c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9659c640 1 -- 172.21.15.94:0/144342735 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c980c5b20 con 0x7f3c9806b3a0 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.262+0000 7f3c9759e640 1 --2- 172.21.15.94:0/144342735 >> v2:172.21.15.145:3300/0 conn(0x7f3c98159160 0x7f3c9815fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.263+0000 7f3c9659c640 1 --2- 172.21.15.94:0/144342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98158c20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3c980c1e10 tx=0x7f3c8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50af0d5c66718b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.263+0000 7f3c95d9b640 1 -- 172.21.15.94:0/144342735 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c035020 con 0x7f3c9806b3a0 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.263+0000 7f3c95d9b640 1 -- 172.21.15.94:0/144342735 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c031870 con 0x7f3c9806b3a0 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.263+0000 7f3c95d9b640 1 -- 172.21.15.94:0/144342735 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c031c40 con 0x7f3c9806b3a0 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 -- 172.21.15.94:0/144342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 msgr2=0x7f3c98158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 --2- 172.21.15.94:0/144342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98158c20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3c980c1e10 tx=0x7f3c8c02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 -- 172.21.15.94:0/144342735 shutdown_connections 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 --2- 172.21.15.94:0/144342735 >> v2:172.21.15.145:3300/0 conn(0x7f3c98159160 0x7f3c9815fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 --2- 172.21.15.94:0/144342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98158c20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 -- 172.21.15.94:0/144342735 >> 172.21.15.94:0/144342735 conn(0x7f3c980c4220 msgr2=0x7f3c980c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 -- 172.21.15.94:0/144342735 shutdown_connections 2024-03-06T07:20:05.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.264+0000 7f3c9d064640 1 -- 172.21.15.94:0/144342735 wait complete. 2024-03-06T07:20:05.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.265+0000 7f3c9d064640 1 Processor -- start 2024-03-06T07:20:05.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.265+0000 7f3c9d064640 1 -- start start 2024-03-06T07:20:05.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9806afc0 0x7f3c9806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c98159160 0x7f3c98072250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c981595f0 con 0x7f3c9806b3a0 2024-03-06T07:20:05.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980c60e0 con 0x7f3c9806afc0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c98155220 con 0x7f3c98159160 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51820/0 (socket says 172.21.15.94:51820) 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 -- 172.21.15.94:0/3508343166 learned_addr learned my addr 172.21.15.94:0/3508343166 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9806afc0 0x7f3c9806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9759e640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c98159160 0x7f3c98072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 -- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c98159160 msgr2=0x7f3c98072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c98159160 0x7f3c98072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 -- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9806afc0 msgr2=0x7f3c9806da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9806afc0 0x7f3c9806da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 -- 172.21.15.94:0/3508343166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c98157f00 con 0x7f3c9806b3a0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9659c640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98071d10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c02b950 tx=0x7f3c8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/3508343166 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c040070 con 0x7f3c9806b3a0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/3508343166 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c030d00 con 0x7f3c9806b3a0 2024-03-06T07:20:05.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c98071510 con 0x7f3c9806b3a0 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/3508343166 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c035040 con 0x7f3c9806b3a0 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.266+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9806fc00 con 0x7f3c9806b3a0 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.268+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c980716e0 con 0x7f3c9806b3a0 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.268+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/3508343166 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c8c04b060 con 0x7f3c9806b3a0 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.268+0000 7f3c7f7fe640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.268+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/3508343166 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3c8c0abef0 con 0x7f3c9806b3a0 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.268+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.269+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3c9814fea0 tx=0x7f3c88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.273+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/3508343166 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c8c07a620 con 0x7f3c9806b3a0 2024-03-06T07:20:05.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.442+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c9804f330 con 0x7f3c9806b3a0 2024-03-06T07:20:05.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.442+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/3508343166 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c8c07e6c0 con 0x7f3c9806b3a0 2024-03-06T07:20:05.444 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:20:05.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 msgr2=0x7f3c8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3c9814fea0 tx=0x7f3c88009040 comp rx=0 tx=0).stop 2024-03-06T07:20:05.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 msgr2=0x7f3c98071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98071d10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c02b950 tx=0x7f3c8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 shutdown_connections 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c98159160 0x7f3c98072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9806b3a0 0x7f3c98071d10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 --2- 172.21.15.94:0/3508343166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9806afc0 0x7f3c9806da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 >> 172.21.15.94:0/3508343166 conn(0x7f3c980c4220 msgr2=0x7f3c98158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 shutdown_connections 2024-03-06T07:20:05.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.444+0000 7f3c9d064640 1 -- 172.21.15.94:0/3508343166 wait complete. 2024-03-06T07:20:05.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:20:05.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.555+0000 7f527fa1c640 1 Processor -- start 2024-03-06T07:20:05.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.555+0000 7f527fa1c640 1 -- start start 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527fa1c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f52780c1aa0 0x7f52781529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527fa1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f5278155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527fa1c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52781563d0 con 0x7f527814f790 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527fa1c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52780c2680 con 0x7f52780c1aa0 2024-03-06T07:20:05.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527fa1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52780c28d0 con 0x7f5278155a40 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527cfb7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f52780c1aa0 0x7f52781529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527d7b8640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f527814f790 0x7f52780c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57382/0 (socket says 172.21.15.94:57382) 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527d7b8640 1 -- 172.21.15.94:0/3484132036 learned_addr learned my addr 172.21.15.94:0/3484132036 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527dfb9640 1 --2- 172.21.15.94:0/3484132036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f5278155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527cfb7640 1 -- 172.21.15.94:0/3484132036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 msgr2=0x7f5278155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527cfb7640 1 --2- 172.21.15.94:0/3484132036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f5278155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527cfb7640 1 -- 172.21.15.94:0/3484132036 >> v1:172.21.15.94:6789/0 conn(0x7f527814f790 legacy=0x7f52780c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f527cfb7640 1 -- 172.21.15.94:0/3484132036 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52780c2bb0 con 0x7f52780c1aa0 2024-03-06T07:20:05.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.556+0000 7f526ffff640 1 -- 172.21.15.94:0/3484132036 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067326907 0 0) 0x7f52781563d0 con 0x7f527814f790 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.557+0000 7f527cfb7640 1 --2- 172.21.15.94:0/3484132036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52780c1aa0 0x7f52781529d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f52680097a0 tx=0x7f526802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d43697f34a452d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.557+0000 7f526ffff640 1 -- 172.21.15.94:0/3484132036 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268035020 con 0x7f52780c1aa0 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.557+0000 7f526ffff640 1 -- 172.21.15.94:0/3484132036 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5268031830 con 0x7f52780c1aa0 2024-03-06T07:20:05.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.557+0000 7f526ffff640 1 -- 172.21.15.94:0/3484132036 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268031c00 con 0x7f52780c1aa0 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.557+0000 7f527fa1c640 1 -- 172.21.15.94:0/3484132036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52780c1aa0 msgr2=0x7f52781529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.557+0000 7f527fa1c640 1 --2- 172.21.15.94:0/3484132036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52780c1aa0 0x7f52781529d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f52680097a0 tx=0x7f526802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.558+0000 7f527fa1c640 1 -- 172.21.15.94:0/3484132036 shutdown_connections 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.558+0000 7f527fa1c640 1 --2- 172.21.15.94:0/3484132036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f5278155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.558+0000 7f527fa1c640 1 --2- 172.21.15.94:0/3484132036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52780c1aa0 0x7f52781529d0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.558+0000 7f527fa1c640 1 -- 172.21.15.94:0/3484132036 >> 172.21.15.94:0/3484132036 conn(0x7f52780c6710 msgr2=0x7f52780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.558+0000 7f527fa1c640 1 -- 172.21.15.94:0/3484132036 shutdown_connections 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.558+0000 7f527fa1c640 1 -- 172.21.15.94:0/3484132036 wait complete. 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 Processor -- start 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 -- start start 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52780c1aa0 0x7f52781e8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527814f790 0x7f5278150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f52780702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52780c0d60 con 0x7f527814f790 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5278155520 con 0x7f52780c1aa0 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527fa1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52780c5e30 con 0x7f5278155a40 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.559+0000 7f527cfb7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527814f790 0x7f5278150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527814f790 0x7f5278150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51846/0 (socket says 172.21.15.94:51846) 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 -- 172.21.15.94:0/1260518214 learned_addr learned my addr 172.21.15.94:0/1260518214 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527d7b8640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52780c1aa0 0x7f52781e8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527dfb9640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f52780702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 -- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 msgr2=0x7f52780702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f52780702f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 -- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52780c1aa0 msgr2=0x7f52781e8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52780c1aa0 0x7f52781e8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 -- 172.21.15.94:0/1260518214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52780c59c0 con 0x7f527814f790 2024-03-06T07:20:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527cfb7640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527814f790 0x7f5278150fc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5268009890 tx=0x7f52680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f526dffb640 1 -- 172.21.15.94:0/1260518214 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268035040 con 0x7f527814f790 2024-03-06T07:20:05.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f526dffb640 1 -- 172.21.15.94:0/1260518214 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526803b920 con 0x7f527814f790 2024-03-06T07:20:05.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52780bb980 con 0x7f527814f790 2024-03-06T07:20:05.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f526dffb640 1 -- 172.21.15.94:0/1260518214 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526803bd10 con 0x7f527814f790 2024-03-06T07:20:05.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.560+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52781e7f10 con 0x7f527814f790 2024-03-06T07:20:05.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.561+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527804f330 con 0x7f527814f790 2024-03-06T07:20:05.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.562+0000 7f526dffb640 1 -- 172.21.15.94:0/1260518214 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5268038020 con 0x7f527814f790 2024-03-06T07:20:05.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.562+0000 7f526dffb640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f525c06b4d0 0x7f525c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.562+0000 7f526dffb640 1 -- 172.21.15.94:0/1260518214 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f52680ac380 con 0x7f527814f790 2024-03-06T07:20:05.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.562+0000 7f527d7b8640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f525c06b4d0 0x7f525c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.563+0000 7f527d7b8640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f525c06b4d0 0x7f525c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5274000f60 tx=0x7f5274007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.566+0000 7f526dffb640 1 -- 172.21.15.94:0/1260518214 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5268031800 con 0x7f527814f790 2024-03-06T07:20:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.737+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f52780c1f40 con 0x7f527814f790 2024-03-06T07:20:05.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.738+0000 7f526dffb640 1 -- 172.21.15.94:0/1260518214 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) v1 ==== 84+0+207 (secure 0 0 0) 0x7f526807aa30 con 0x7f527814f790 2024-03-06T07:20:05.740 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:05.740 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:20:05.740 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:05.740 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:20:05.740 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 7, 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": "cephfs_metadata" 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 8, 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": "cephfs_data" 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:05.741 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f525c06b4d0 msgr2=0x7f525c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f525c06b4d0 0x7f525c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5274000f60 tx=0x7f5274007d50 comp rx=0 tx=0).stop 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527814f790 msgr2=0x7f5278150fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527814f790 0x7f5278150fc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5268009890 tx=0x7f52680046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 shutdown_connections 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f525c06b4d0 0x7f525c06d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5278155a40 0x7f52780702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f527814f790 0x7f5278150fc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 --2- 172.21.15.94:0/1260518214 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52780c1aa0 0x7f52781e8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 >> 172.21.15.94:0/1260518214 conn(0x7f52780c6710 msgr2=0x7f52780a6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 shutdown_connections 2024-03-06T07:20:05.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.740+0000 7f527fa1c640 1 -- 172.21.15.94:0/1260518214 wait complete. 2024-03-06T07:20:05.753 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-06T07:20:05.753 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-06T07:20:05.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 Processor -- start 2024-03-06T07:20:05.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 -- start start 2024-03-06T07:20:05.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd41527a0 0x7f5fd4155040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5fd40c1860 0x7f5fd406bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5fd406c510 0x7f5fd4158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd4158820 con 0x7f5fd40c1860 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd40c2b10 con 0x7f5fd406c510 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.855+0000 7f5fd8a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4155580 con 0x7f5fd41527a0 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5fd40c1860 0x7f5fd406bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5fd40c1860 0x7f5fd406bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51854/0 (socket says 172.21.15.94:51854) 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 -- 172.21.15.94:0/2144643354 learned_addr learned my addr 172.21.15.94:0/2144643354 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fd259c640 1 --2- 172.21.15.94:0/2144643354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd41527a0 0x7f5fd4155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fd2d9d640 1 --2- 172.21.15.94:0/2144643354 >> v2:172.21.15.145:3300/0 conn(0x7f5fd406c510 0x7f5fd4158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 -- 172.21.15.94:0/2144643354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd41527a0 msgr2=0x7f5fd4155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 --2- 172.21.15.94:0/2144643354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd41527a0 0x7f5fd4155040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 -- 172.21.15.94:0/2144643354 >> v2:172.21.15.145:3300/0 conn(0x7f5fd406c510 msgr2=0x7f5fd4158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:05.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 --2- 172.21.15.94:0/2144643354 >> v2:172.21.15.145:3300/0 conn(0x7f5fd406c510 0x7f5fd4158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.856+0000 7f5fcbfff640 1 -- 172.21.15.94:0/2144643354 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4155820 con 0x7f5fd40c1860 2024-03-06T07:20:05.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fcbfff640 1 --2- 172.21.15.94:0/2144643354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd406bfd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40ac5c0 tx=0x7f5fbc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed1adca148cf86e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd1d9b640 1 -- 172.21.15.94:0/2144643354 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc02e070 con 0x7f5fd40c1860 2024-03-06T07:20:05.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd1d9b640 1 -- 172.21.15.94:0/2144643354 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc037070 con 0x7f5fd40c1860 2024-03-06T07:20:05.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd1d9b640 1 -- 172.21.15.94:0/2144643354 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc031a00 con 0x7f5fd40c1860 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 -- 172.21.15.94:0/2144643354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 msgr2=0x7f5fd406bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/2144643354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd406bfd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40ac5c0 tx=0x7f5fbc02b880 comp rx=0 tx=0).stop 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 -- 172.21.15.94:0/2144643354 shutdown_connections 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/2144643354 >> v2:172.21.15.145:3300/0 conn(0x7f5fd406c510 0x7f5fd4158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/2144643354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd406bfd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/2144643354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd41527a0 0x7f5fd4155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 -- 172.21.15.94:0/2144643354 >> 172.21.15.94:0/2144643354 conn(0x7f5fd40c3960 msgr2=0x7f5fd40c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 -- 172.21.15.94:0/2144643354 shutdown_connections 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.857+0000 7f5fd8a39640 1 -- 172.21.15.94:0/2144643354 wait complete. 2024-03-06T07:20:05.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 Processor -- start 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 -- start start 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd406c510 0x7f5fd41e0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd4071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd41527a0 0x7f5fd4153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4152b80 con 0x7f5fd40c1860 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4155de0 con 0x7f5fd41527a0 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd8a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4152d00 con 0x7f5fd406c510 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fcbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd4071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fcbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd4071510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51866/0 (socket says 172.21.15.94:51866) 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fcbfff640 1 -- 172.21.15.94:0/3191606422 learned_addr learned my addr 172.21.15.94:0/3191606422 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd259c640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd406c510 0x7f5fd41e0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.859+0000 7f5fd2d9d640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd41527a0 0x7f5fd4153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcbfff640 1 -- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd406c510 msgr2=0x7f5fd41e0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcbfff640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd406c510 0x7f5fd41e0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcbfff640 1 -- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd41527a0 msgr2=0x7f5fd4153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcbfff640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd41527a0 0x7f5fd4153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcbfff640 1 -- 172.21.15.94:0/3191606422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd41e0570 con 0x7f5fd40c1860 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcbfff640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd4071510 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc000c00 tx=0x7f5fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcaffd640 1 -- 172.21.15.94:0/3191606422 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc032670 con 0x7f5fd40c1860 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcaffd640 1 -- 172.21.15.94:0/3191606422 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc032810 con 0x7f5fd40c1860 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd41ddbd0 con 0x7f5fd40c1860 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fcaffd640 1 -- 172.21.15.94:0/3191606422 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc032c00 con 0x7f5fd40c1860 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.860+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd4072890 con 0x7f5fd40c1860 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.861+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd404f330 con 0x7f5fd40c1860 2024-03-06T07:20:05.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.861+0000 7f5fcaffd640 1 -- 172.21.15.94:0/3191606422 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fbc0453f0 con 0x7f5fd40c1860 2024-03-06T07:20:05.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.862+0000 7f5fcaffd640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fb406b4d0 0x7f5fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:05.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.862+0000 7f5fcaffd640 1 -- 172.21.15.94:0/3191606422 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5fbc0abfa0 con 0x7f5fd40c1860 2024-03-06T07:20:05.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.862+0000 7f5fd259c640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fb406b4d0 0x7f5fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:05.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.862+0000 7f5fd259c640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fb406b4d0 0x7f5fb406d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009a20 tx=0x7f5fc4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:05.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:05.866+0000 7f5fcaffd640 1 -- 172.21.15.94:0/3191606422 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fd41e0570 con 0x7f5fd40c1860 2024-03-06T07:20:06.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.036+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5fd40c64b0 con 0x7f5fd40c1860 2024-03-06T07:20:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.256+0000 7f5fcaffd640 1 -- 172.21.15.94:0/3191606422 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v53) v1 ==== 140+0+0 (secure 0 0 0) 0x7f5fbc040070 con 0x7f5fd40c1860 2024-03-06T07:20:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_metadata' created 2024-03-06T07:20:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fb406b4d0 msgr2=0x7f5fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fb406b4d0 0x7f5fb406d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009a20 tx=0x7f5fc4009190 comp rx=0 tx=0).stop 2024-03-06T07:20:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 msgr2=0x7f5fd4071510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd4071510 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc000c00 tx=0x7f5fbc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 shutdown_connections 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5fb406b4d0 0x7f5fb406d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fd41527a0 0x7f5fd4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5fd40c1860 0x7f5fd4071510 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 --2- 172.21.15.94:0/3191606422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fd406c510 0x7f5fd41e0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 >> 172.21.15.94:0/3191606422 conn(0x7f5fd40c3960 msgr2=0x7f5fd41563d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 shutdown_connections 2024-03-06T07:20:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.259+0000 7f5fd8a39640 1 -- 172.21.15.94:0/3191606422 wait complete. 2024-03-06T07:20:06.274 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:20:06.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd43acb5640 1 Processor -- start 2024-03-06T07:20:06.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd43acb5640 1 -- start start 2024-03-06T07:20:06.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd43acb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd434155c30 0x7fd43406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd43acb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4340c5380 0x7fd4340c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd43acb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4340b0450 con 0x7fd4340b0070 2024-03-06T07:20:06.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd43acb5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd43406cfb0 con 0x7fd4340c5380 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd43acb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd43406c020 con 0x7fd434155c30 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd439252640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4340c5380 0x7fd4340c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd438a51640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd4340b0070 0x7fd4340c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57396/0 (socket says 172.21.15.94:57396) 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.380+0000 7fd438a51640 1 -- 172.21.15.94:0/557825395 learned_addr learned my addr 172.21.15.94:0/557825395 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.381+0000 7fd433fff640 1 --2- 172.21.15.94:0/557825395 >> v2:172.21.15.145:3300/0 conn(0x7fd434155c30 0x7fd43406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.381+0000 7fd439252640 1 -- 172.21.15.94:0/557825395 >> v2:172.21.15.145:3300/0 conn(0x7fd434155c30 msgr2=0x7fd43406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.381+0000 7fd439252640 1 --2- 172.21.15.94:0/557825395 >> v2:172.21.15.145:3300/0 conn(0x7fd434155c30 0x7fd43406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.381+0000 7fd439252640 1 -- 172.21.15.94:0/557825395 >> v1:172.21.15.94:6789/0 conn(0x7fd4340b0070 legacy=0x7fd4340c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.381+0000 7fd4337fe640 1 -- 172.21.15.94:0/557825395 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616898413 0 0) 0x7fd424001110 con 0x7fd4340b0070 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.381+0000 7fd439252640 1 -- 172.21.15.94:0/557825395 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4341560e0 con 0x7fd4340c5380 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.381+0000 7fd439252640 1 --2- 172.21.15.94:0/557825395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd4340c5790 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd43406d4c0 tx=0x7fd42c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c91452a787d762 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd4337fe640 1 -- 172.21.15.94:0/557825395 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c039070 con 0x7fd4340c5380 2024-03-06T07:20:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd4337fe640 1 -- 172.21.15.94:0/557825395 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42c02cbf0 con 0x7fd4340c5380 2024-03-06T07:20:06.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd4337fe640 1 -- 172.21.15.94:0/557825395 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c034440 con 0x7fd4340c5380 2024-03-06T07:20:06.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 -- 172.21.15.94:0/557825395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 msgr2=0x7fd4340c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 --2- 172.21.15.94:0/557825395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd4340c5790 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd43406d4c0 tx=0x7fd42c02be40 comp rx=0 tx=0).stop 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 -- 172.21.15.94:0/557825395 shutdown_connections 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 --2- 172.21.15.94:0/557825395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd4340c5790 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 --2- 172.21.15.94:0/557825395 >> v2:172.21.15.145:3300/0 conn(0x7fd434155c30 0x7fd43406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 -- 172.21.15.94:0/557825395 >> 172.21.15.94:0/557825395 conn(0x7fd4340ac590 msgr2=0x7fd4340ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 -- 172.21.15.94:0/557825395 shutdown_connections 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.382+0000 7fd43acb5640 1 -- 172.21.15.94:0/557825395 wait complete. 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.383+0000 7fd43acb5640 1 Processor -- start 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.383+0000 7fd43acb5640 1 -- start start 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd43acb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4340b0070 0x7fd4340c1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd43acb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd434157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd43acb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd434155c30 0x7fd4341ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd43acb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43406c270 con 0x7fd4340c5380 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd43acb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340c7ad0 con 0x7fd4340b0070 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd43acb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43406c550 con 0x7fd434155c30 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd434157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd434157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51886/0 (socket says 172.21.15.94:51886) 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 -- 172.21.15.94:0/3989159047 learned_addr learned my addr 172.21.15.94:0/3989159047 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd438a51640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4340b0070 0x7fd4340c1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd439252640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd434155c30 0x7fd4341ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 -- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd434155c30 msgr2=0x7fd4341ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd434155c30 0x7fd4341ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 -- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4340b0070 msgr2=0x7fd4340c1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4340b0070 0x7fd4340c1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd433fff640 1 -- 172.21.15.94:0/3989159047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd434158a70 con 0x7fd4340c5380 2024-03-06T07:20:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.384+0000 7fd438a51640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4340b0070 0x7fd4340c1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.385+0000 7fd433fff640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd434157770 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd42800e9b0 tx=0x7fd42800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.385+0000 7fd4317fa640 1 -- 172.21.15.94:0/3989159047 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd428017070 con 0x7fd4340c5380 2024-03-06T07:20:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.385+0000 7fd4317fa640 1 -- 172.21.15.94:0/3989159047 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42800c9a0 con 0x7fd4340c5380 2024-03-06T07:20:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.385+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd434158d50 con 0x7fd4340c5380 2024-03-06T07:20:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.385+0000 7fd4317fa640 1 -- 172.21.15.94:0/3989159047 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42800cd90 con 0x7fd4340c5380 2024-03-06T07:20:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.385+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd434156f70 con 0x7fd4340c5380 2024-03-06T07:20:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.386+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43404f330 con 0x7fd4340c5380 2024-03-06T07:20:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.386+0000 7fd4317fa640 1 -- 172.21.15.94:0/3989159047 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd42801a070 con 0x7fd4340c5380 2024-03-06T07:20:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.387+0000 7fd4317fa640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd42006b4d0 0x7fd42006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.387+0000 7fd4317fa640 1 -- 172.21.15.94:0/3989159047 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6472+0+0 (secure 0 0 0) 0x7fd42808a800 con 0x7fd4340c5380 2024-03-06T07:20:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.387+0000 7fd438a51640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd42006b4d0 0x7fd42006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.387+0000 7fd438a51640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd42006b4d0 0x7fd42006d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd424000f60 tx=0x7fd424007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:06.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.391+0000 7fd4317fa640 1 -- 172.21.15.94:0/3989159047 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd428059410 con 0x7fd4340c5380 2024-03-06T07:20:06.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:06.562+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd43406d860 con 0x7fd4340c5380 2024-03-06T07:20:07.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.263+0000 7fd4317fa640 1 -- 172.21.15.94:0/3989159047 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v54) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd428058d10 con 0x7fd4340c5380 2024-03-06T07:20:07.265 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_data' created 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.266+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd42006b4d0 msgr2=0x7fd42006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.266+0000 7fd43acb5640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd42006b4d0 0x7fd42006d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd424000f60 tx=0x7fd424007ca0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.266+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 msgr2=0x7fd434157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.266+0000 7fd43acb5640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd434157770 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd42800e9b0 tx=0x7fd42800ee70 comp rx=0 tx=0).stop 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 shutdown_connections 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd42006b4d0 0x7fd42006d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd434155c30 0x7fd4341ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4340c5380 0x7fd434157770 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 --2- 172.21.15.94:0/3989159047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4340b0070 0x7fd4340c1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 >> 172.21.15.94:0/3989159047 conn(0x7fd4340ac590 msgr2=0x7fd4340aef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:07.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 shutdown_connections 2024-03-06T07:20:07.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.267+0000 7fd43acb5640 1 -- 172.21.15.94:0/3989159047 wait complete. 2024-03-06T07:20:07.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-06T07:20:07.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.401+0000 7fa152288640 1 Processor -- start 2024-03-06T07:20:07.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.401+0000 7fa152288640 1 -- start start 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa152288640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c06c720 0x7fa14c071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa152288640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa14c0c2410 0x7fa14c071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa152288640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c067340 con 0x7fa14c150920 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa152288640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14c072400 con 0x7fa14c0c2410 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa152288640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c150d00 con 0x7fa14c06c720 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa150825640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa14c0c2410 0x7fa14c071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa150825640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa14c0c2410 0x7fa14c071ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51902/0 (socket says 172.21.15.94:51902) 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa150825640 1 -- 172.21.15.94:0/1246336039 learned_addr learned my addr 172.21.15.94:0/1246336039 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa14b7fe640 1 --2- 172.21.15.94:0/1246336039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c06c720 0x7fa14c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa150825640 1 -- 172.21.15.94:0/1246336039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c06c720 msgr2=0x7fa14c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa150825640 1 --2- 172.21.15.94:0/1246336039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c06c720 0x7fa14c071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.402+0000 7fa150825640 1 -- 172.21.15.94:0/1246336039 >> v1:172.21.15.145:6789/0 conn(0x7fa14c150920 legacy=0x7fa14c0c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.403+0000 7fa150825640 1 -- 172.21.15.94:0/1246336039 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c0c2010 con 0x7fa14c0c2410 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.403+0000 7fa150825640 1 --2- 172.21.15.94:0/1246336039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c071ec0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa144009e50 tx=0x7fa14402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22be651378c2d913 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.403+0000 7fa14affd640 1 -- 172.21.15.94:0/1246336039 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14402ca70 con 0x7fa14c0c2410 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.403+0000 7fa14affd640 1 -- 172.21.15.94:0/1246336039 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14402cc10 con 0x7fa14c0c2410 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.403+0000 7fa14affd640 1 -- 172.21.15.94:0/1246336039 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1440343f0 con 0x7fa14c0c2410 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 -- 172.21.15.94:0/1246336039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 msgr2=0x7fa14c071ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 --2- 172.21.15.94:0/1246336039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c071ec0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa144009e50 tx=0x7fa14402bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 -- 172.21.15.94:0/1246336039 shutdown_connections 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 --2- 172.21.15.94:0/1246336039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c071ec0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 --2- 172.21.15.94:0/1246336039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c06c720 0x7fa14c071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 -- 172.21.15.94:0/1246336039 >> 172.21.15.94:0/1246336039 conn(0x7fa14c0c70b0 msgr2=0x7fa14c0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:07.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 -- 172.21.15.94:0/1246336039 shutdown_connections 2024-03-06T07:20:07.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.404+0000 7fa152288640 1 -- 172.21.15.94:0/1246336039 wait complete. 2024-03-06T07:20:07.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.405+0000 7fa152288640 1 Processor -- start 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.405+0000 7fa152288640 1 -- start start 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa152288640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa14c06c720 0x7fa14c145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa152288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa152288640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c150920 0x7fa14c152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa152288640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0c3040 con 0x7fa14c0c2410 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa152288640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c072b70 con 0x7fa14c06c720 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa152288640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0c31c0 con 0x7fa14c150920 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51912/0 (socket says 172.21.15.94:51912) 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 -- 172.21.15.94:0/845565333 learned_addr learned my addr 172.21.15.94:0/845565333 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14bfff640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa14c06c720 0x7fa14c145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa150825640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c150920 0x7fa14c152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 -- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c150920 msgr2=0x7fa14c152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c150920 0x7fa14c152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 -- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa14c06c720 msgr2=0x7fa14c145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:07.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa14c06c720 0x7fa14c145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:07.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 -- 172.21.15.94:0/845565333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c1e8540 con 0x7fa14c0c2410 2024-03-06T07:20:07.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.406+0000 7fa14b7fe640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c152020 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa14000e870 tx=0x7fa14000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:07.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.407+0000 7fa148ff9640 1 -- 172.21.15.94:0/845565333 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140017070 con 0x7fa14c0c2410 2024-03-06T07:20:07.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.407+0000 7fa148ff9640 1 -- 172.21.15.94:0/845565333 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14000c940 con 0x7fa14c0c2410 2024-03-06T07:20:07.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.407+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c14aca0 con 0x7fa14c0c2410 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.407+0000 7fa148ff9640 1 -- 172.21.15.94:0/845565333 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000cd30 con 0x7fa14c0c2410 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.407+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c1e8800 con 0x7fa14c0c2410 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.408+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c04f330 con 0x7fa14c0c2410 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.408+0000 7fa148ff9640 1 -- 172.21.15.94:0/845565333 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa14001a070 con 0x7fa14c0c2410 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.408+0000 7fa148ff9640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa13806b500 0x7fa13806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.409+0000 7fa148ff9640 1 -- 172.21.15.94:0/845565333 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6855+0+0 (secure 0 0 0) 0x7fa14008a3b0 con 0x7fa14c0c2410 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.409+0000 7fa14bfff640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa13806b500 0x7fa13806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:07.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.409+0000 7fa14bfff640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa13806b500 0x7fa13806d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa13c001780 tx=0x7fa13c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:07.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.413+0000 7fa148ff9640 1 -- 172.21.15.94:0/845565333 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1400587f0 con 0x7fa14c0c2410 2024-03-06T07:20:07.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:07.619+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7fa14c068450 con 0x7fa14c0c2410 2024-03-06T07:20:08.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.264+0000 7fa148ff9640 1 -- 172.21.15.94:0/845565333 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 9 and data pool 10 v30) v1 ==== 176+0+0 (secure 0 0 0) 0x7fa14005c890 con 0x7fa14c0c2410 2024-03-06T07:20:08.266 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 9 and data pool 10 2024-03-06T07:20:08.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.269+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa13806b500 msgr2=0x7fa13806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.269+0000 7fa152288640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa13806b500 0x7fa13806d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa13c001780 tx=0x7fa13c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:08.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.269+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 msgr2=0x7fa14c152020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.269+0000 7fa152288640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c152020 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa14000e870 tx=0x7fa14000ed30 comp rx=0 tx=0).stop 2024-03-06T07:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.270+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 shutdown_connections 2024-03-06T07:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.270+0000 7fa152288640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa13806b500 0x7fa13806d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.270+0000 7fa152288640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa14c150920 0x7fa14c152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.270+0000 7fa152288640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa14c0c2410 0x7fa14c152020 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.270+0000 7fa152288640 1 --2- 172.21.15.94:0/845565333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa14c06c720 0x7fa14c145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.270+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 >> 172.21.15.94:0/845565333 conn(0x7fa14c0c70b0 msgr2=0x7fa14c0c7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.278+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 shutdown_connections 2024-03-06T07:20:08.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.280+0000 7fa152288640 1 -- 172.21.15.94:0/845565333 wait complete. 2024-03-06T07:20:08.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:08.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.406+0000 7f2543b20640 1 Processor -- start 2024-03-06T07:20:08.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.407+0000 7f2543b20640 1 -- start start 2024-03-06T07:20:08.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.407+0000 7f2543b20640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f253c0c1aa0 0x7f253c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.407+0000 7f2543b20640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.407+0000 7f2543b20640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253c1563d0 con 0x7f253c14f790 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.407+0000 7f2543b20640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c0c2680 con 0x7f253c0c1aa0 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.407+0000 7f2543b20640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0c28d0 con 0x7f253c155a40 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25410bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f253c0c1aa0 0x7f253c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25410bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f253c0c1aa0 0x7f253c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51928/0 (socket says 172.21.15.94:51928) 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25410bb640 1 -- 172.21.15.94:0/1676153014 learned_addr learned my addr 172.21.15.94:0/1676153014 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25420bd640 1 --2- 172.21.15.94:0/1676153014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25410bb640 1 -- 172.21.15.94:0/1676153014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 msgr2=0x7f253c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25410bb640 1 --2- 172.21.15.94:0/1676153014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25410bb640 1 -- 172.21.15.94:0/1676153014 >> v1:172.21.15.145:6789/0 conn(0x7f253c14f790 legacy=0x7f253c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.408+0000 7f25410bb640 1 -- 172.21.15.94:0/1676153014 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c0c2bb0 con 0x7f253c0c1aa0 2024-03-06T07:20:08.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f25410bb640 1 --2- 172.21.15.94:0/1676153014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c1529d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f252c009870 tx=0x7f252c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e3128be4246f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f25408ba640 1 -- 172.21.15.94:0/1676153014 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c034070 con 0x7f253c0c1aa0 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f25408ba640 1 -- 172.21.15.94:0/1676153014 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252c030840 con 0x7f253c0c1aa0 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f25408ba640 1 -- 172.21.15.94:0/1676153014 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c030c10 con 0x7f253c0c1aa0 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 -- 172.21.15.94:0/1676153014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 msgr2=0x7f253c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 --2- 172.21.15.94:0/1676153014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c1529d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f252c009870 tx=0x7f252c02b980 comp rx=0 tx=0).stop 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 -- 172.21.15.94:0/1676153014 shutdown_connections 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 --2- 172.21.15.94:0/1676153014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 --2- 172.21.15.94:0/1676153014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c1529d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 -- 172.21.15.94:0/1676153014 >> 172.21.15.94:0/1676153014 conn(0x7f253c0c6710 msgr2=0x7f253c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 -- 172.21.15.94:0/1676153014 shutdown_connections 2024-03-06T07:20:08.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.409+0000 7f2543b20640 1 -- 172.21.15.94:0/1676153014 wait complete. 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.410+0000 7f2543b20640 1 Processor -- start 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.410+0000 7f2543b20640 1 -- start start 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f2543b20640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c14e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f2543b20640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f253c14f790 0x7f253c1eb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f2543b20640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c1eb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f2543b20640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0c0d60 con 0x7f253c0c1aa0 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f2543b20640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c155520 con 0x7f253c14f790 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f2543b20640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0c5e30 con 0x7f253c155a40 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f25418bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c14e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f25418bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c14e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51938/0 (socket says 172.21.15.94:51938) 2024-03-06T07:20:08.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f25418bc640 1 -- 172.21.15.94:0/4133541116 learned_addr learned my addr 172.21.15.94:0/4133541116 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f25410bb640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f253c14f790 0x7f253c1eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f25420bd640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c1eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.411+0000 7f25418bc640 1 -- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 msgr2=0x7f253c1eb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25418bc640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c1eb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25418bc640 1 -- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f253c14f790 msgr2=0x7f253c1eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25418bc640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f253c14f790 0x7f253c1eb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25418bc640 1 -- 172.21.15.94:0/4133541116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c1ead60 con 0x7f253c0c1aa0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25418bc640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c14e6d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f253400c790 tx=0x7f253400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25327fc640 1 -- 172.21.15.94:0/4133541116 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534016020 con 0x7f253c0c1aa0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25327fc640 1 -- 172.21.15.94:0/4133541116 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2534006d50 con 0x7f253c0c1aa0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c1ee2f0 con 0x7f253c0c1aa0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f25327fc640 1 -- 172.21.15.94:0/4133541116 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534003bb0 con 0x7f253c0c1aa0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.412+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c1db870 con 0x7f253c0c1aa0 2024-03-06T07:20:08.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.413+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253c04f330 con 0x7f253c0c1aa0 2024-03-06T07:20:08.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.413+0000 7f25327fc640 1 -- 172.21.15.94:0/4133541116 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f253400a850 con 0x7f253c0c1aa0 2024-03-06T07:20:08.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.414+0000 7f25327fc640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f252006b4d0 0x7f252006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.414+0000 7f25327fc640 1 -- 172.21.15.94:0/4133541116 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f253408a230 con 0x7f253c0c1aa0 2024-03-06T07:20:08.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.414+0000 7f25410bb640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f252006b4d0 0x7f252006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.414+0000 7f25410bb640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f252006b4d0 0x7f252006d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f252c036000 tx=0x7f252c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:08.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.418+0000 7f25327fc640 1 -- 172.21.15.94:0/4133541116 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2534020020 con 0x7f253c0c1aa0 2024-03-06T07:20:08.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.591+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f253c1eaf40 con 0x7f253c0c1aa0 2024-03-06T07:20:08.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.592+0000 7f25327fc640 1 -- 172.21.15.94:0/4133541116 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+20679 (secure 0 0 0) 0x7f253c1ead60 con 0x7f253c0c1aa0 2024-03-06T07:20:08.595 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:08.595 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":55,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:07.620766+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f252006b4d0 msgr2=0x7f252006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f252006b4d0 0x7f252006d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f252c036000 tx=0x7f252c008030 comp rx=0 tx=0).stop 2024-03-06T07:20:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 msgr2=0x7f253c14e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c14e6d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f253400c790 tx=0x7f253400cc50 comp rx=0 tx=0).stop 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 shutdown_connections 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f252006b4d0 0x7f252006d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f253c155a40 0x7f253c1eb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f253c14f790 0x7f253c1eb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 --2- 172.21.15.94:0/4133541116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f253c0c1aa0 0x7f253c14e6d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 >> 172.21.15.94:0/4133541116 conn(0x7f253c0c6710 msgr2=0x7f253c0c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 shutdown_connections 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.594+0000 7f2543b20640 1 -- 172.21.15.94:0/4133541116 wait complete. 2024-03-06T07:20:08.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:08.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.705+0000 7fbf57381640 1 Processor -- start 2024-03-06T07:20:08.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.706+0000 7fbf57381640 1 -- start start 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.706+0000 7fbf57381640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbf500c88c0 0x7fbf50155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.706+0000 7fbf57381640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf50158b60 0x7fbf50158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.706+0000 7fbf57381640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf500c73d0 con 0x7fbf5014f360 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.706+0000 7fbf57381640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf500c8dd0 con 0x7fbf500c88c0 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.706+0000 7fbf57381640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500be2e0 con 0x7fbf50158b60 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.706+0000 7fbf5491c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbf500c88c0 0x7fbf50155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5591e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf50158b60 0x7fbf50158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5491c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbf500c88c0 0x7fbf50155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:36828/0 (socket says 172.21.15.94:36828) 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5511d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fbf5014f360 0x7fbf500c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:50608/0 (socket says 172.21.15.94:50608) 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5591e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf50158b60 0x7fbf50158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53378/0 (socket says 172.21.15.94:53378) 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5491c640 1 -- 172.21.15.94:0/885701922 learned_addr learned my addr 172.21.15.94:0/885701922 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5591e640 1 -- 172.21.15.94:0/885701922 >> v2:172.21.15.145:3300/0 conn(0x7fbf500c88c0 msgr2=0x7fbf50155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5591e640 1 --2- 172.21.15.94:0/885701922 >> v2:172.21.15.145:3300/0 conn(0x7fbf500c88c0 0x7fbf50155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5591e640 1 -- 172.21.15.94:0/885701922 >> v1:172.21.15.145:6789/0 conn(0x7fbf5014f360 legacy=0x7fbf500c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.707+0000 7fbf5591e640 1 -- 172.21.15.94:0/885701922 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf500c7670 con 0x7fbf50158b60 2024-03-06T07:20:08.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf5591e640 1 --2- 172.21.15.94:0/885701922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf50158b60 0x7fbf50158ff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009d90 tx=0x7fbf4802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6b53bbb8df7410 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:08.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf47fff640 1 -- 172.21.15.94:0/885701922 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48039070 con 0x7fbf50158b60 2024-03-06T07:20:08.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf47fff640 1 -- 172.21.15.94:0/885701922 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4802cbc0 con 0x7fbf50158b60 2024-03-06T07:20:08.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf47fff640 1 -- 172.21.15.94:0/885701922 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48034440 con 0x7fbf50158b60 2024-03-06T07:20:08.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 -- 172.21.15.94:0/885701922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf50158b60 msgr2=0x7fbf50158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 --2- 172.21.15.94:0/885701922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf50158b60 0x7fbf50158ff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009d90 tx=0x7fbf4802be10 comp rx=0 tx=0).stop 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 -- 172.21.15.94:0/885701922 shutdown_connections 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 --2- 172.21.15.94:0/885701922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf50158b60 0x7fbf50158ff0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 --2- 172.21.15.94:0/885701922 >> v2:172.21.15.145:3300/0 conn(0x7fbf500c88c0 0x7fbf50155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 -- 172.21.15.94:0/885701922 >> 172.21.15.94:0/885701922 conn(0x7fbf500ac570 msgr2=0x7fbf500ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 -- 172.21.15.94:0/885701922 shutdown_connections 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.708+0000 7fbf57381640 1 -- 172.21.15.94:0/885701922 wait complete. 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.709+0000 7fbf57381640 1 Processor -- start 2024-03-06T07:20:08.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.709+0000 7fbf57381640 1 -- start start 2024-03-06T07:20:08.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf57381640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf500c88c0 0x7fbf501ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf57381640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5014f360 0x7fbf501ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf57381640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf50158b60 0x7fbf5014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf57381640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500c5a50 con 0x7fbf500c88c0 2024-03-06T07:20:08.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf57381640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500c7c30 con 0x7fbf50158b60 2024-03-06T07:20:08.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf57381640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500c5bd0 con 0x7fbf5014f360 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf5511d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf500c88c0 0x7fbf501ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf5511d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf500c88c0 0x7fbf501ee030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37562/0 (socket says 172.21.15.94:37562) 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf5491c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5014f360 0x7fbf501ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf5511d640 1 -- 172.21.15.94:0/3977854697 learned_addr learned my addr 172.21.15.94:0/3977854697 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.710+0000 7fbf5591e640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf50158b60 0x7fbf5014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf5491c640 1 -- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf50158b60 msgr2=0x7fbf5014b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf5491c640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf50158b60 0x7fbf5014b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf5491c640 1 -- 172.21.15.94:0/3977854697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf500c88c0 msgr2=0x7fbf501ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf5491c640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf500c88c0 0x7fbf501ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf5491c640 1 -- 172.21.15.94:0/3977854697 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf501eeab0 con 0x7fbf5014f360 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf5511d640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf500c88c0 0x7fbf501ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf5491c640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5014f360 0x7fbf501ee570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000e840 tx=0x7fbf4000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:08.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf45ffb640 1 -- 172.21.15.94:0/3977854697 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40017070 con 0x7fbf5014f360 2024-03-06T07:20:08.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf45ffb640 1 -- 172.21.15.94:0/3977854697 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4000c730 con 0x7fbf5014f360 2024-03-06T07:20:08.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf501eed20 con 0x7fbf5014f360 2024-03-06T07:20:08.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf45ffb640 1 -- 172.21.15.94:0/3977854697 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000cb00 con 0x7fbf5014f360 2024-03-06T07:20:08.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.711+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf500c1450 con 0x7fbf5014f360 2024-03-06T07:20:08.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.712+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5004f330 con 0x7fbf5014f360 2024-03-06T07:20:08.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.713+0000 7fbf45ffb640 1 -- 172.21.15.94:0/3977854697 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf4001a070 con 0x7fbf5014f360 2024-03-06T07:20:08.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.713+0000 7fbf45ffb640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf3406b4d0 0x7fbf3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:08.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.713+0000 7fbf45ffb640 1 -- 172.21.15.94:0/3977854697 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fbf4008a730 con 0x7fbf5014f360 2024-03-06T07:20:08.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.713+0000 7fbf5511d640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf3406b4d0 0x7fbf3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:08.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.714+0000 7fbf5511d640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf3406b4d0 0x7fbf3406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbf500bdd80 tx=0x7fbf4c007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:08.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.718+0000 7fbf45ffb640 1 -- 172.21.15.94:0/3977854697 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf40058b10 con 0x7fbf5014f360 2024-03-06T07:20:08.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.888+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf5006cda0 con 0x7fbf5014f360 2024-03-06T07:20:08.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.889+0000 7fbf45ffb640 1 -- 172.21.15.94:0/3977854697 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+20679 (secure 0 0 0) 0x7fbf4005cbb0 con 0x7fbf5014f360 2024-03-06T07:20:08.892 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:08.892 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":55,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:07.620766+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf3406b4d0 msgr2=0x7fbf3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf3406b4d0 0x7fbf3406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbf500bdd80 tx=0x7fbf4c007f10 comp rx=0 tx=0).stop 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5014f360 msgr2=0x7fbf501ee570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5014f360 0x7fbf501ee570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000e840 tx=0x7fbf4000ed00 comp rx=0 tx=0).stop 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 shutdown_connections 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf3406b4d0 0x7fbf3406d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf50158b60 0x7fbf5014b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5014f360 0x7fbf501ee570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 --2- 172.21.15.94:0/3977854697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf500c88c0 0x7fbf501ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:08.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 >> 172.21.15.94:0/3977854697 conn(0x7fbf500ac570 msgr2=0x7fbf500ab310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:08.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 shutdown_connections 2024-03-06T07:20:08.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:08.891+0000 7fbf57381640 1 -- 172.21.15.94:0/3977854697 wait complete. 2024-03-06T07:20:08.905 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-06T07:20:09.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.006+0000 7f406b0c2640 1 Processor -- start 2024-03-06T07:20:09.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.006+0000 7f406b0c2640 1 -- start start 2024-03-06T07:20:09.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f406b0c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640b0070 0x7f4064155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f406b0c2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f40640c5380 0x7f40640c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f406b0c2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40640b0450 con 0x7f406406ca70 2024-03-06T07:20:09.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f406b0c2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406406c020 con 0x7f40640c5380 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f406b0c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40640c77f0 con 0x7f40640b0070 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f4063fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f406406ca70 0x7f40640c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38570/0 (socket says 172.21.15.94:38570) 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f4063fff640 1 -- 172.21.15.94:0/4155149023 learned_addr learned my addr 172.21.15.94:0/4155149023 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f4068e5e640 1 --2- 172.21.15.94:0/4155149023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640b0070 0x7f4064155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.007+0000 7f406965f640 1 --2- 172.21.15.94:0/4155149023 >> v2:172.21.15.145:3300/0 conn(0x7f40640c5380 0x7f40640c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.008+0000 7f40637fe640 1 -- 172.21.15.94:0/4155149023 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076008934 0 0) 0x7f4058001110 con 0x7f406406ca70 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.008+0000 7f40637fe640 1 -- 172.21.15.94:0/4155149023 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c003520 con 0x7f406406ca70 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.008+0000 7f4068e5e640 1 -- 172.21.15.94:0/4155149023 >> v2:172.21.15.145:3300/0 conn(0x7f40640c5380 msgr2=0x7f40640c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.008+0000 7f4068e5e640 1 --2- 172.21.15.94:0/4155149023 >> v2:172.21.15.145:3300/0 conn(0x7f40640c5380 0x7f40640c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.008+0000 7f4068e5e640 1 -- 172.21.15.94:0/4155149023 >> v1:172.21.15.94:6789/0 conn(0x7f406406ca70 legacy=0x7f40640c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.008+0000 7f4068e5e640 1 -- 172.21.15.94:0/4155149023 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40640c7ad0 con 0x7f40640b0070 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.008+0000 7f406965f640 1 --2- 172.21.15.94:0/4155149023 >> v2:172.21.15.145:3300/0 conn(0x7f40640c5380 0x7f40640c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f4068e5e640 1 --2- 172.21.15.94:0/4155149023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640b0070 0x7f4064155c30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40540097a0 tx=0x7f405402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c6df6a31d0278d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:09.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f40637fe640 1 -- 172.21.15.94:0/4155149023 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40540315a0 con 0x7f40640b0070 2024-03-06T07:20:09.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f40637fe640 1 -- 172.21.15.94:0/4155149023 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4054037070 con 0x7f40640b0070 2024-03-06T07:20:09.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f40637fe640 1 -- 172.21.15.94:0/4155149023 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054031940 con 0x7f40640b0070 2024-03-06T07:20:09.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 -- 172.21.15.94:0/4155149023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640b0070 msgr2=0x7f4064155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 --2- 172.21.15.94:0/4155149023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640b0070 0x7f4064155c30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40540097a0 tx=0x7f405402b840 comp rx=0 tx=0).stop 2024-03-06T07:20:09.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 -- 172.21.15.94:0/4155149023 shutdown_connections 2024-03-06T07:20:09.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 --2- 172.21.15.94:0/4155149023 >> v2:172.21.15.145:3300/0 conn(0x7f40640c5380 0x7f40640c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 --2- 172.21.15.94:0/4155149023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640b0070 0x7f4064155c30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 -- 172.21.15.94:0/4155149023 >> 172.21.15.94:0/4155149023 conn(0x7f40640ac590 msgr2=0x7f40640ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 -- 172.21.15.94:0/4155149023 shutdown_connections 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.009+0000 7f406b0c2640 1 -- 172.21.15.94:0/4155149023 wait complete. 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.010+0000 7f406b0c2640 1 Processor -- start 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.010+0000 7f406b0c2640 1 -- start start 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f406b0c2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406406ca70 0x7f4064154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f406b0c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40640b0070 0x7f40640c1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f406b0c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640c5380 0x7f40640c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f406b0c2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406406cef0 con 0x7f406406ca70 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f406b0c2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406406c550 con 0x7f40640b0070 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f406b0c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406406da80 con 0x7f40640c5380 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406406ca70 0x7f4064154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406406ca70 0x7f4064154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37576/0 (socket says 172.21.15.94:37576) 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 -- 172.21.15.94:0/2630729024 learned_addr learned my addr 172.21.15.94:0/2630729024 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4063fff640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40640b0070 0x7f40640c1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f406965f640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640c5380 0x7f40640c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 -- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640c5380 msgr2=0x7f40640c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640c5380 0x7f40640c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 -- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40640b0070 msgr2=0x7f40640c1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40640b0070 0x7f40640c1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.011+0000 7f4068e5e640 1 -- 172.21.15.94:0/2630729024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064158300 con 0x7f406406ca70 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.012+0000 7f4068e5e640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406406ca70 0x7f4064154550 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f405402b880 tx=0x7f40540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.012+0000 7f40617fa640 1 -- 172.21.15.94:0/2630729024 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054032630 con 0x7f406406ca70 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.012+0000 7f40617fa640 1 -- 172.21.15.94:0/2630729024 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40540327f0 con 0x7f406406ca70 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.012+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40640c48b0 con 0x7f406406ca70 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.012+0000 7f40617fa640 1 -- 172.21.15.94:0/2630729024 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054032be0 con 0x7f406406ca70 2024-03-06T07:20:09.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.012+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4064154a90 con 0x7f406406ca70 2024-03-06T07:20:09.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.013+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406404f330 con 0x7f406406ca70 2024-03-06T07:20:09.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.013+0000 7f40617fa640 1 -- 172.21.15.94:0/2630729024 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f405403b3f0 con 0x7f406406ca70 2024-03-06T07:20:09.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.014+0000 7f40617fa640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405006b4d0 0x7f405006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.014+0000 7f40617fa640 1 -- 172.21.15.94:0/2630729024 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f405402f030 con 0x7f406406ca70 2024-03-06T07:20:09.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.014+0000 7f4063fff640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405006b4d0 0x7f405006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.014+0000 7f4063fff640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405006b4d0 0x7f405006d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4064154ff0 tx=0x7f4058009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:09.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.018+0000 7f40617fa640 1 -- 172.21.15.94:0/2630729024 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4054040070 con 0x7f406406ca70 2024-03-06T07:20:09.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.230+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f40640c4a80 con 0x7f406406ca70 2024-03-06T07:20:09.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.276+0000 7f40617fa640 1 -- 172.21.15.94:0/2630729024 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 119+0+0 (secure 0 0 0) 0x7f405407ac80 con 0x7f406406ca70 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405006b4d0 msgr2=0x7f405006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405006b4d0 0x7f405006d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4064154ff0 tx=0x7f4058009040 comp rx=0 tx=0).stop 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406406ca70 msgr2=0x7f4064154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406406ca70 0x7f4064154550 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f405402b880 tx=0x7f40540046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 shutdown_connections 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405006b4d0 0x7f405006d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40640c5380 0x7f40640c28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40640b0070 0x7f40640c1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 --2- 172.21.15.94:0/2630729024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406406ca70 0x7f4064154550 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 >> 172.21.15.94:0/2630729024 conn(0x7f40640ac590 msgr2=0x7f40640ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 shutdown_connections 2024-03-06T07:20:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.279+0000 7f406b0c2640 1 -- 172.21.15.94:0/2630729024 wait complete. 2024-03-06T07:20:09.292 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:20:09.292 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-06T07:20:09.293 INFO:tasks.ceph.mds.c.smithi094.stderr:2024-03-06T07:20:09.291+0000 7f8e6b537640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:20:09.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.394+0000 7f97d97bc640 1 Processor -- start 2024-03-06T07:20:09.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.394+0000 7f97d97bc640 1 -- start start 2024-03-06T07:20:09.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d97bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f97d406d750 0x7f97d40732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d97bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d4156060 con 0x7f97d40c3a70 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d97bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d40a8430 con 0x7f97d406db30 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d97bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d4155b30 con 0x7f97d406d750 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d27fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f97d40c3a70 0x7f97d4072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38578/0 (socket says 172.21.15.94:38578) 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d27fc640 1 -- 172.21.15.94:0/2609910777 learned_addr learned my addr 172.21.15.94:0/2609910777 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d2ffd640 1 --2- 172.21.15.94:0/2609910777 >> v2:172.21.15.145:3300/0 conn(0x7f97d406d750 0x7f97d40732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d1ffb640 1 -- 172.21.15.94:0/2609910777 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933828130 0 0) 0x7f97d4156060 con 0x7f97d40c3a70 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.395+0000 7f97d1ffb640 1 -- 172.21.15.94:0/2609910777 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c0003520 con 0x7f97d40c3a70 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.396+0000 7f97d1ffb640 1 -- 172.21.15.94:0/2609910777 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881331677 0 0) 0x7f97c0003520 con 0x7f97d40c3a70 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.396+0000 7f97d2ffd640 1 -- 172.21.15.94:0/2609910777 >> v1:172.21.15.145:6789/0 conn(0x7f97d406db30 legacy=0x7f97d40c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.396+0000 7f97d2ffd640 1 -- 172.21.15.94:0/2609910777 >> v1:172.21.15.94:6789/0 conn(0x7f97d40c3a70 legacy=0x7f97d4072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.396+0000 7f97d2ffd640 1 -- 172.21.15.94:0/2609910777 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d4155d80 con 0x7f97d406d750 2024-03-06T07:20:09.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.396+0000 7f97d2ffd640 1 --2- 172.21.15.94:0/2609910777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d406d750 0x7f97d40732a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009d90 tx=0x7f97cc02bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a177abb8e99d64d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d1ffb640 1 -- 172.21.15.94:0/2609910777 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc039070 con 0x7f97d406d750 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d1ffb640 1 -- 172.21.15.94:0/2609910777 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc02c820 con 0x7f97d406d750 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d1ffb640 1 -- 172.21.15.94:0/2609910777 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4001670 con 0x7f97d406d750 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d97bc640 1 -- 172.21.15.94:0/2609910777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d406d750 msgr2=0x7f97d40732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2609910777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d406d750 0x7f97d40732a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009d90 tx=0x7f97cc02bcf0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d97bc640 1 -- 172.21.15.94:0/2609910777 shutdown_connections 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2609910777 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d406d750 0x7f97d40732a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d97bc640 1 -- 172.21.15.94:0/2609910777 >> 172.21.15.94:0/2609910777 conn(0x7f97d40c5560 msgr2=0x7f97d40c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d97bc640 1 -- 172.21.15.94:0/2609910777 shutdown_connections 2024-03-06T07:20:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.397+0000 7f97d97bc640 1 -- 172.21.15.94:0/2609910777 wait complete. 2024-03-06T07:20:09.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.399+0000 7f97d97bc640 1 Processor -- start 2024-03-06T07:20:09.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.399+0000 7f97d97bc640 1 -- start start 2024-03-06T07:20:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d97bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97d406d750 0x7f97d41eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d97bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97d406db30 0x7f97d41e1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d97bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d40c3a70 0x7f97d41eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d97bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d40c2720 con 0x7f97d406db30 2024-03-06T07:20:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d97bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d40c25a0 con 0x7f97d40c3a70 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d97bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d40a7cd0 con 0x7f97d406d750 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97d406db30 0x7f97d41e1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97d406db30 0x7f97d41e1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37586/0 (socket says 172.21.15.94:37586) 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.400+0000 7f97d27fc640 1 -- 172.21.15.94:0/2097876787 learned_addr learned my addr 172.21.15.94:0/2097876787 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d2ffd640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97d406d750 0x7f97d41eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d37fe640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d40c3a70 0x7f97d41eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d27fc640 1 -- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97d406d750 msgr2=0x7f97d41eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d27fc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97d406d750 0x7f97d41eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d27fc640 1 -- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d40c3a70 msgr2=0x7f97d41eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d27fc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d40c3a70 0x7f97d41eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d27fc640 1 -- 172.21.15.94:0/2097876787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d40ae300 con 0x7f97d406db30 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d2ffd640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97d406d750 0x7f97d41eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d27fc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97d406db30 0x7f97d41e1860 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f97c400c730 tx=0x7f97c400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97cb7fe640 1 -- 172.21.15.94:0/2097876787 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4017070 con 0x7f97d406db30 2024-03-06T07:20:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97cb7fe640 1 -- 172.21.15.94:0/2097876787 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c4005ad0 con 0x7f97d406db30 2024-03-06T07:20:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d41ee260 con 0x7f97d406db30 2024-03-06T07:20:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97cb7fe640 1 -- 172.21.15.94:0/2097876787 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4007070 con 0x7f97d406db30 2024-03-06T07:20:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.401+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d40abdd0 con 0x7f97d406db30 2024-03-06T07:20:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.403+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d404f330 con 0x7f97d406db30 2024-03-06T07:20:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.403+0000 7f97cb7fe640 1 -- 172.21.15.94:0/2097876787 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97c401a070 con 0x7f97d406db30 2024-03-06T07:20:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.403+0000 7f97cb7fe640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97b406b500 0x7f97b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.403+0000 7f97cb7fe640 1 -- 172.21.15.94:0/2097876787 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f97c40899d0 con 0x7f97d406db30 2024-03-06T07:20:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.403+0000 7f97d2ffd640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97b406b500 0x7f97b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.404+0000 7f97d2ffd640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97b406b500 0x7f97b406d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f97cc036000 tx=0x7f97cc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:09.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.407+0000 7f97cb7fe640 1 -- 172.21.15.94:0/2097876787 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97c4057db0 con 0x7f97d406db30 2024-03-06T07:20:09.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:09.623+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f97d4072b40 con 0x7f97d406db30 2024-03-06T07:20:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.285+0000 7f97cb7fe640 1 -- 172.21.15.94:0/2097876787 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 123+0+0 (secure 0 0 0) 0x7f97c405be50 con 0x7f97d406db30 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97b406b500 msgr2=0x7f97b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97b406b500 0x7f97b406d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f97cc036000 tx=0x7f97cc030000 comp rx=0 tx=0).stop 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97d406db30 msgr2=0x7f97d41e1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97d406db30 0x7f97d41e1860 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f97c400c730 tx=0x7f97c400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 shutdown_connections 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97b406b500 0x7f97b406d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97d40c3a70 0x7f97d41eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97d406db30 0x7f97d41e1860 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 --2- 172.21.15.94:0/2097876787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97d406d750 0x7f97d41eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 >> 172.21.15.94:0/2097876787 conn(0x7f97d40c5560 msgr2=0x7f97d40730d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 shutdown_connections 2024-03-06T07:20:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.289+0000 7f97d97bc640 1 -- 172.21.15.94:0/2097876787 wait complete. 2024-03-06T07:20:10.305 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-06T07:20:10.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:10.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.404+0000 7f8f1642a640 1 Processor -- start 2024-03-06T07:20:10.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.404+0000 7f8f1642a640 1 -- start start 2024-03-06T07:20:10.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f1642a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f101527b0 0x7f8f10159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f1642a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8f100c7180 0x7f8f100c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f1642a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8f10155b20 0x7f8f100c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f1642a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f10159640 con 0x7f8f100c7180 2024-03-06T07:20:10.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f1642a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f100c8df0 con 0x7f8f10155b20 2024-03-06T07:20:10.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f1642a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100c7c60 con 0x7f8f101527b0 2024-03-06T07:20:10.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8f100c7180 0x7f8f100c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8f100c7180 0x7f8f100c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37602/0 (socket says 172.21.15.94:37602) 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 -- 172.21.15.94:0/2379635725 learned_addr learned my addr 172.21.15.94:0/2379635725 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0ffff640 1 --2- 172.21.15.94:0/2379635725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f101527b0 0x7f8f10159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f149c7640 1 --2- 172.21.15.94:0/2379635725 >> v2:172.21.15.145:3300/0 conn(0x7f8f10155b20 0x7f8f100c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 -- 172.21.15.94:0/2379635725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f101527b0 msgr2=0x7f8f10159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 --2- 172.21.15.94:0/2379635725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f101527b0 0x7f8f10159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 -- 172.21.15.94:0/2379635725 >> v2:172.21.15.145:3300/0 conn(0x7f8f10155b20 msgr2=0x7f8f100c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 --2- 172.21.15.94:0/2379635725 >> v2:172.21.15.145:3300/0 conn(0x7f8f10155b20 0x7f8f100c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.405+0000 7f8f0f7fe640 1 -- 172.21.15.94:0/2379635725 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f100c8060 con 0x7f8f100c7180 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.406+0000 7f8f0f7fe640 1 --2- 172.21.15.94:0/2379635725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f100c7180 0x7f8f100c88b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8f1006b170 tx=0x7f8f0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e6ce08be28ec40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.406+0000 7f8f0effd640 1 -- 172.21.15.94:0/2379635725 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0402e070 con 0x7f8f100c7180 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.406+0000 7f8f0effd640 1 -- 172.21.15.94:0/2379635725 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f04037070 con 0x7f8f100c7180 2024-03-06T07:20:10.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.406+0000 7f8f0effd640 1 -- 172.21.15.94:0/2379635725 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04031a00 con 0x7f8f100c7180 2024-03-06T07:20:10.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 -- 172.21.15.94:0/2379635725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f100c7180 msgr2=0x7f8f100c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2379635725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f100c7180 0x7f8f100c88b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8f1006b170 tx=0x7f8f0402b880 comp rx=0 tx=0).stop 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 -- 172.21.15.94:0/2379635725 shutdown_connections 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2379635725 >> v2:172.21.15.145:3300/0 conn(0x7f8f10155b20 0x7f8f100c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2379635725 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f100c7180 0x7f8f100c88b0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2379635725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f101527b0 0x7f8f10159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 -- 172.21.15.94:0/2379635725 >> 172.21.15.94:0/2379635725 conn(0x7f8f100ac560 msgr2=0x7f8f100ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 -- 172.21.15.94:0/2379635725 shutdown_connections 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.407+0000 7f8f1642a640 1 -- 172.21.15.94:0/2379635725 wait complete. 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.408+0000 7f8f1642a640 1 Processor -- start 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.408+0000 7f8f1642a640 1 -- start start 2024-03-06T07:20:10.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f1642a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f100c7180 0x7f8f101ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f1642a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f101527b0 0x7f8f100c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f1642a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f10155b20 0x7f8f100c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f1642a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10155320 con 0x7f8f101527b0 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f1642a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f100c7980 con 0x7f8f10155b20 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f1642a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10155600 con 0x7f8f100c7180 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f0f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f101527b0 0x7f8f100c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f0ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f100c7180 0x7f8f101ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f0f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f101527b0 0x7f8f100c26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37606/0 (socket says 172.21.15.94:37606) 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f149c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f10155b20 0x7f8f100c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.409+0000 7f8f0f7fe640 1 -- 172.21.15.94:0/2971640510 learned_addr learned my addr 172.21.15.94:0/2971640510 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f149c7640 1 -- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f100c7180 msgr2=0x7f8f101ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f149c7640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f100c7180 0x7f8f101ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f149c7640 1 -- 172.21.15.94:0/2971640510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f101527b0 msgr2=0x7f8f100c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f149c7640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f101527b0 0x7f8f100c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f149c7640 1 -- 172.21.15.94:0/2971640510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f100c1cd0 con 0x7f8f10155b20 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f149c7640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f10155b20 0x7f8f100c2c20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800eba0 tx=0x7f8f0800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f0cff9640 1 -- 172.21.15.94:0/2971640510 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08018020 con 0x7f8f10155b20 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f0cff9640 1 -- 172.21.15.94:0/2971640510 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0800cd20 con 0x7f8f10155b20 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f101eeb80 con 0x7f8f10155b20 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f0cff9640 1 -- 172.21.15.94:0/2971640510 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f080146c0 con 0x7f8f10155b20 2024-03-06T07:20:10.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.410+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f101ee780 con 0x7f8f10155b20 2024-03-06T07:20:10.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.411+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1004f330 con 0x7f8f10155b20 2024-03-06T07:20:10.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.412+0000 7f8f0cff9640 1 -- 172.21.15.94:0/2971640510 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f08005d10 con 0x7f8f10155b20 2024-03-06T07:20:10.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.412+0000 7f8f0cff9640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8efc06b4d0 0x7f8efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.412+0000 7f8f0cff9640 1 -- 172.21.15.94:0/2971640510 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f8f0808b1d0 con 0x7f8f10155b20 2024-03-06T07:20:10.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.412+0000 7f8f0ffff640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8efc06b4d0 0x7f8efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.412+0000 7f8f0ffff640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8efc06b4d0 0x7f8efc06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cb350 tx=0x7f8f00009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:10.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.417+0000 7f8f0cff9640 1 -- 172.21.15.94:0/2971640510 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f08022030 con 0x7f8f10155b20 2024-03-06T07:20:10.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.623+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f10158e30 con 0x7f8f10155b20 2024-03-06T07:20:10.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.623+0000 7f8f0cff9640 1 -- 172.21.15.94:0/2971640510 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6264 (secure 0 0 0) 0x7f8f08010040 con 0x7f8f10155b20 2024-03-06T07:20:10.625 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:10.626 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":32,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5521,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.94:6835/480754107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":480754107},{"type":"v1","addr":"172.21.15.94:6835","nonce":480754107}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:19:55.378348+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35084},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35084":{"gid":35084,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3125091562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3125091562},{"type":"v1","addr":"172.21.15.145:6837","nonce":3125091562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35084,"qdb_cluster":[35084]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:10.626 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 32 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8efc06b4d0 msgr2=0x7f8efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8efc06b4d0 0x7f8efc06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8f100cb350 tx=0x7f8f00009190 comp rx=0 tx=0).stop 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f10155b20 msgr2=0x7f8f100c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f10155b20 0x7f8f100c2c20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800eba0 tx=0x7f8f0800c490 comp rx=0 tx=0).stop 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 shutdown_connections 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8efc06b4d0 0x7f8efc06d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f10155b20 0x7f8f100c2c20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f101527b0 0x7f8f100c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 --2- 172.21.15.94:0/2971640510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f100c7180 0x7f8f101ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 >> 172.21.15.94:0/2971640510 conn(0x7f8f100ac560 msgr2=0x7f8f100aa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:10.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 shutdown_connections 2024-03-06T07:20:10.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.626+0000 7f8f1642a640 1 -- 172.21.15.94:0/2971640510 wait complete. 2024-03-06T07:20:10.642 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.758+0000 7fb47dd72640 1 Processor -- start 2024-03-06T07:20:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.758+0000 7fb47dd72640 1 -- start start 2024-03-06T07:20:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb47dd72640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb4781550c0 0x7fb4781554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb47dd72640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb4781559e0 0x7fb47814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb47dd72640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780aa660 con 0x7fb4780aa180 2024-03-06T07:20:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb47dd72640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4780c4610 con 0x7fb4781550c0 2024-03-06T07:20:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb47dd72640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4780aa940 con 0x7fb4781559e0 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb4777fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb4781550c0 0x7fb4781554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb4777fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb4781550c0 0x7fb4781554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37620/0 (socket says 172.21.15.94:37620) 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb4777fe640 1 -- 172.21.15.94:0/3317650419 learned_addr learned my addr 172.21.15.94:0/3317650419 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb476ffd640 1 --2- 172.21.15.94:0/3317650419 >> v2:172.21.15.145:3300/0 conn(0x7fb4781559e0 0x7fb47814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb4777fe640 1 -- 172.21.15.94:0/3317650419 >> v2:172.21.15.145:3300/0 conn(0x7fb4781559e0 msgr2=0x7fb47814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb4777fe640 1 --2- 172.21.15.94:0/3317650419 >> v2:172.21.15.145:3300/0 conn(0x7fb4781559e0 0x7fb47814ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb4777fe640 1 -- 172.21.15.94:0/3317650419 >> v1:172.21.15.145:6789/0 conn(0x7fb4780aa180 legacy=0x7fb4780c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.759+0000 7fb4777fe640 1 -- 172.21.15.94:0/3317650419 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4780c4860 con 0x7fb4781550c0 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb4777fe640 1 --2- 172.21.15.94:0/3317650419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb4781554a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb4780abe90 tx=0x7fb46c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bcdf01c4b8f11a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb4767fc640 1 -- 172.21.15.94:0/3317650419 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c02e070 con 0x7fb4781550c0 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb4767fc640 1 -- 172.21.15.94:0/3317650419 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c037070 con 0x7fb4781550c0 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb4767fc640 1 -- 172.21.15.94:0/3317650419 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c031a00 con 0x7fb4781550c0 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb47dd72640 1 -- 172.21.15.94:0/3317650419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 msgr2=0x7fb4781554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3317650419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb4781554a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb4780abe90 tx=0x7fb46c02b880 comp rx=0 tx=0).stop 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb47dd72640 1 -- 172.21.15.94:0/3317650419 shutdown_connections 2024-03-06T07:20:10.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3317650419 >> v2:172.21.15.145:3300/0 conn(0x7fb4781559e0 0x7fb47814ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3317650419 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb4781554a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.760+0000 7fb47dd72640 1 -- 172.21.15.94:0/3317650419 >> 172.21.15.94:0/3317650419 conn(0x7fb4780c6710 msgr2=0x7fb4780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:10.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.761+0000 7fb47dd72640 1 -- 172.21.15.94:0/3317650419 shutdown_connections 2024-03-06T07:20:10.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.761+0000 7fb47dd72640 1 -- 172.21.15.94:0/3317650419 wait complete. 2024-03-06T07:20:10.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 Processor -- start 2024-03-06T07:20:10.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 -- start start 2024-03-06T07:20:10.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4780aa180 0x7fb4780bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb478147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4781559e0 0x7fb478146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47814f140 con 0x7fb4781550c0 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4780c1560 con 0x7fb4781559e0 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.762+0000 7fb47dd72640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47814f2c0 con 0x7fb4780aa180 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb478147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb478147b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37622/0 (socket says 172.21.15.94:37622) 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 -- 172.21.15.94:0/3675886673 learned_addr learned my addr 172.21.15.94:0/3675886673 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb4777fe640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4780aa180 0x7fb4780bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb477fff640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4781559e0 0x7fb478146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 -- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4780aa180 msgr2=0x7fb4780bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4780aa180 0x7fb4780bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 -- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4781559e0 msgr2=0x7fb478146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4781559e0 0x7fb478146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 -- 172.21.15.94:0/3675886673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478148b50 con 0x7fb4781550c0 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb476ffd640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb478147b00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb46800e870 tx=0x7fb46800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb45ffff640 1 -- 172.21.15.94:0/3675886673 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468017070 con 0x7fb4781550c0 2024-03-06T07:20:10.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb45ffff640 1 -- 172.21.15.94:0/3675886673 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46800c940 con 0x7fb4781550c0 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb45ffff640 1 -- 172.21.15.94:0/3675886673 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800cd30 con 0x7fb4781550c0 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb478148e30 con 0x7fb4781550c0 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.763+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4780a7fa0 con 0x7fb4781550c0 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.764+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47804f330 con 0x7fb4781550c0 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.765+0000 7fb45ffff640 1 -- 172.21.15.94:0/3675886673 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb46801a070 con 0x7fb4781550c0 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.765+0000 7fb45ffff640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b500 0x7fb46406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.765+0000 7fb45ffff640 1 -- 172.21.15.94:0/3675886673 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fb46808a850 con 0x7fb4781550c0 2024-03-06T07:20:10.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.765+0000 7fb4777fe640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b500 0x7fb46406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:10.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.766+0000 7fb4777fe640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b500 0x7fb46406d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb46c036000 tx=0x7fb46c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:10.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.770+0000 7fb45ffff640 1 -- 172.21.15.94:0/3675886673 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb468058ba0 con 0x7fb4781550c0 2024-03-06T07:20:10.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.948+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb478068450 con 0x7fb4781550c0 2024-03-06T07:20:10.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.949+0000 7fb45ffff640 1 -- 172.21.15.94:0/3675886673 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+20666 (secure 0 0 0) 0x7fb46805cc40 con 0x7fb4781550c0 2024-03-06T07:20:10.952 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:10.952 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":56,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:09.273378+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b500 msgr2=0x7fb46406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b500 0x7fb46406d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb46c036000 tx=0x7fb46c03d040 comp rx=0 tx=0).stop 2024-03-06T07:20:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 msgr2=0x7fb478147b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb478147b00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb46800e870 tx=0x7fb46800ed30 comp rx=0 tx=0).stop 2024-03-06T07:20:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 shutdown_connections 2024-03-06T07:20:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b500 0x7fb46406d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4781559e0 0x7fb478146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb4781550c0 0x7fb478147b00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.951+0000 7fb47dd72640 1 --2- 172.21.15.94:0/3675886673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4780aa180 0x7fb4780bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.952+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 >> 172.21.15.94:0/3675886673 conn(0x7fb4780c6710 msgr2=0x7fb47814d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.952+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 shutdown_connections 2024-03-06T07:20:10.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:10.952+0000 7fb47dd72640 1 -- 172.21.15.94:0/3675886673 wait complete. 2024-03-06T07:20:10.966 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:20:10.966 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:11.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.065+0000 7fc821f32640 1 Processor -- start 2024-03-06T07:20:11.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.065+0000 7fc821f32640 1 -- start start 2024-03-06T07:20:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc821f32640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc81c150580 0x7fc81c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc821f32640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc81c0c6000 0x7fc81c06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc821f32640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c153a60 con 0x7fc81c0c1320 2024-03-06T07:20:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc821f32640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c153d40 con 0x7fc81c150580 2024-03-06T07:20:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc821f32640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c06c2c0 con 0x7fc81c0c6000 2024-03-06T07:20:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc81c150580 0x7fc81c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc81c150580 0x7fc81c06dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37638/0 (socket says 172.21.15.94:37638) 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81b7fe640 1 -- 172.21.15.94:0/672969394 learned_addr learned my addr 172.21.15.94:0/672969394 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81bfff640 1 --2- 172.21.15.94:0/672969394 >> v2:172.21.15.145:3300/0 conn(0x7fc81c0c6000 0x7fc81c06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81b7fe640 1 -- 172.21.15.94:0/672969394 >> v2:172.21.15.145:3300/0 conn(0x7fc81c0c6000 msgr2=0x7fc81c06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81b7fe640 1 --2- 172.21.15.94:0/672969394 >> v2:172.21.15.145:3300/0 conn(0x7fc81c0c6000 0x7fc81c06e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81b7fe640 1 -- 172.21.15.94:0/672969394 >> v1:172.21.15.145:6789/0 conn(0x7fc81c0c1320 legacy=0x7fc81c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.066+0000 7fc81b7fe640 1 -- 172.21.15.94:0/672969394 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c06c5a0 con 0x7fc81c150580 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.067+0000 7fc81b7fe640 1 --2- 172.21.15.94:0/672969394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c150580 0x7fc81c06dd10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0c5960 tx=0x7fc80c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc04d38bd1a89885 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.067+0000 7fc81affd640 1 -- 172.21.15.94:0/672969394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c035020 con 0x7fc81c150580 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.067+0000 7fc81affd640 1 -- 172.21.15.94:0/672969394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80c031870 con 0x7fc81c150580 2024-03-06T07:20:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.067+0000 7fc81affd640 1 -- 172.21.15.94:0/672969394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80c031c40 con 0x7fc81c150580 2024-03-06T07:20:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 -- 172.21.15.94:0/672969394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c150580 msgr2=0x7fc81c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 --2- 172.21.15.94:0/672969394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c150580 0x7fc81c06dd10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0c5960 tx=0x7fc80c02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 -- 172.21.15.94:0/672969394 shutdown_connections 2024-03-06T07:20:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 --2- 172.21.15.94:0/672969394 >> v2:172.21.15.145:3300/0 conn(0x7fc81c0c6000 0x7fc81c06e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 --2- 172.21.15.94:0/672969394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c150580 0x7fc81c06dd10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 -- 172.21.15.94:0/672969394 >> 172.21.15.94:0/672969394 conn(0x7fc81c0c6710 msgr2=0x7fc81c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 -- 172.21.15.94:0/672969394 shutdown_connections 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.068+0000 7fc821f32640 1 -- 172.21.15.94:0/672969394 wait complete. 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.069+0000 7fc821f32640 1 Processor -- start 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.069+0000 7fc821f32640 1 -- start start 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc821f32640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc81c0c1320 0x7fc81c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc821f32640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c0c6000 0x7fc81c1ec940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc821f32640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc81c150580 0x7fc81c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc821f32640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0c0dc0 con 0x7fc81c0c6000 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc821f32640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0c0c40 con 0x7fc81c0c1320 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc821f32640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c156dc0 con 0x7fc81c150580 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c0c6000 0x7fc81c1ec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c0c6000 0x7fc81c1ec940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37650/0 (socket says 172.21.15.94:37650) 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 -- 172.21.15.94:0/482567585 learned_addr learned my addr 172.21.15.94:0/482567585 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:11.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc81b7fe640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc81c0c1320 0x7fc81c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc81bfff640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc81c150580 0x7fc81c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 -- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc81c150580 msgr2=0x7fc81c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc81c150580 0x7fc81c0707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 -- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc81c0c1320 msgr2=0x7fc81c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc81c0c1320 0x7fc81c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 -- 172.21.15.94:0/482567585 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c1ed180 con 0x7fc81c0c6000 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc81b7fe640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc81c0c1320 0x7fc81c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc813fff640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c0c6000 0x7fc81c1ec940 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc80400c400 tx=0x7fc80400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc818ff9640 1 -- 172.21.15.94:0/482567585 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8040052d0 con 0x7fc81c0c6000 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc818ff9640 1 -- 172.21.15.94:0/482567585 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc804005490 con 0x7fc81c0c6000 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c0a8150 con 0x7fc81c0c6000 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.070+0000 7fc818ff9640 1 -- 172.21.15.94:0/482567585 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804007260 con 0x7fc81c0c6000 2024-03-06T07:20:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.071+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c1ed420 con 0x7fc81c0c6000 2024-03-06T07:20:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.072+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c06bfa0 con 0x7fc81c0c6000 2024-03-06T07:20:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.072+0000 7fc818ff9640 1 -- 172.21.15.94:0/482567585 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc804007400 con 0x7fc81c0c6000 2024-03-06T07:20:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.072+0000 7fc818ff9640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc7fc06b4d0 0x7fc7fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.072+0000 7fc818ff9640 1 -- 172.21.15.94:0/482567585 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc804089af0 con 0x7fc81c0c6000 2024-03-06T07:20:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.073+0000 7fc81b7fe640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc7fc06b4d0 0x7fc7fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.073+0000 7fc81b7fe640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc7fc06b4d0 0x7fc7fc06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc80c036000 tx=0x7fc80c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.077+0000 7fc818ff9640 1 -- 172.21.15.94:0/482567585 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc81c06bfa0 con 0x7fc81c0c6000 2024-03-06T07:20:11.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.250+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc81c152fc0 con 0x7fc7fc06b4d0 2024-03-06T07:20:11.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.261+0000 7fc818ff9640 1 -- 172.21.15.94:0/482567585 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485250 (secure 0 0 0) 0x7fc80c0be7a0 con 0x7fc7fc06b4d0 2024-03-06T07:20:11.274 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:11.278 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc7fc06b4d0 msgr2=0x7fc7fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc7fc06b4d0 0x7fc7fc06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc80c036000 tx=0x7fc80c03c040 comp rx=0 tx=0).stop 2024-03-06T07:20:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c0c6000 msgr2=0x7fc81c1ec940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c0c6000 0x7fc81c1ec940 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc80400c400 tx=0x7fc80400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 shutdown_connections 2024-03-06T07:20:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc7fc06b4d0 0x7fc7fc06d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc81c150580 0x7fc81c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc81c0c6000 0x7fc81c1ec940 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 --2- 172.21.15.94:0/482567585 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc81c0c1320 0x7fc81c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 >> 172.21.15.94:0/482567585 conn(0x7fc81c0c6710 msgr2=0x7fc81c06dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:11.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 shutdown_connections 2024-03-06T07:20:11.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.265+0000 7fc821f32640 1 -- 172.21.15.94:0/482567585 wait complete. 2024-03-06T07:20:11.282 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-03-06T07:20:10.659980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":209,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237920,"kb_used_data":5120,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736800,"statfs":{"total":773094113280,"available":772850483200,"internally_reserved":0,"allocated":5242880,"data_stored":3450788,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.396926"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276976+0000","last_change":"2024-03-06T07:20:07.276976+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276976+0000","last_undegraded":"2024-03-06T07:20:07.276976+0000","last_fullsized":"2024-03-06T07:20:07.276976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266870+0000","last_change":"2024-03-06T07:19:54.052304+0000","last_active":"2024-03-06T07:20:07.266870+0000","last_peered":"2024-03-06T07:20:07.266870+0000","last_clean":"2024-03-06T07:20:07.266870+0000","last_became_active":"2024-03-06T07:19:54.052048+0000","last_became_peered":"2024-03-06T07:19:54.052048+0000","last_unstale":"2024-03-06T07:20:07.266870+0000","last_undegraded":"2024-03-06T07:20:07.266870+0000","last_fullsized":"2024-03-06T07:20:07.266870+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:24.129824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270987+0000","last_change":"2024-03-06T07:20:07.270987+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270987+0000","last_undegraded":"2024-03-06T07:20:07.270987+0000","last_fullsized":"2024-03-06T07:20:07.270987+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283665+0000","last_change":"2024-03-06T07:19:54.062911+0000","last_active":"2024-03-06T07:20:07.283665+0000","last_peered":"2024-03-06T07:20:07.283665+0000","last_clean":"2024-03-06T07:20:07.283665+0000","last_became_active":"2024-03-06T07:19:54.062566+0000","last_became_peered":"2024-03-06T07:19:54.062566+0000","last_unstale":"2024-03-06T07:20:07.283665+0000","last_undegraded":"2024-03-06T07:20:07.283665+0000","last_fullsized":"2024-03-06T07:20:07.283665+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271068+0000","last_change":"2024-03-06T07:20:07.271068+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271068+0000","last_undegraded":"2024-03-06T07:20:07.271068+0000","last_fullsized":"2024-03-06T07:20:07.271068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272063+0000","last_change":"2024-03-06T07:19:54.051868+0000","last_active":"2024-03-06T07:20:07.272063+0000","last_peered":"2024-03-06T07:20:07.272063+0000","last_clean":"2024-03-06T07:20:07.272063+0000","last_became_active":"2024-03-06T07:19:54.051611+0000","last_became_peered":"2024-03-06T07:19:54.051611+0000","last_unstale":"2024-03-06T07:20:07.272063+0000","last_undegraded":"2024-03-06T07:20:07.272063+0000","last_fullsized":"2024-03-06T07:20:07.272063+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270217+0000","last_change":"2024-03-06T07:20:07.270217+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270217+0000","last_undegraded":"2024-03-06T07:20:07.270217+0000","last_fullsized":"2024-03-06T07:20:07.270217+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279217+0000","last_change":"2024-03-06T07:19:54.055553+0000","last_active":"2024-03-06T07:20:07.279217+0000","last_peered":"2024-03-06T07:20:07.279217+0000","last_clean":"2024-03-06T07:20:07.279217+0000","last_became_active":"2024-03-06T07:19:54.055271+0000","last_became_peered":"2024-03-06T07:19:54.055271+0000","last_unstale":"2024-03-06T07:20:07.279217+0000","last_undegraded":"2024-03-06T07:20:07.279217+0000","last_fullsized":"2024-03-06T07:20:07.279217+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274065+0000","last_change":"2024-03-06T07:20:07.274065+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274065+0000","last_undegraded":"2024-03-06T07:20:07.274065+0000","last_fullsized":"2024-03-06T07:20:07.274065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274273+0000","last_change":"2024-03-06T07:19:54.046161+0000","last_active":"2024-03-06T07:20:07.274273+0000","last_peered":"2024-03-06T07:20:07.274273+0000","last_clean":"2024-03-06T07:20:07.274273+0000","last_became_active":"2024-03-06T07:19:54.045875+0000","last_became_peered":"2024-03-06T07:19:54.045875+0000","last_unstale":"2024-03-06T07:20:07.274273+0000","last_undegraded":"2024-03-06T07:20:07.274273+0000","last_fullsized":"2024-03-06T07:20:07.274273+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:50.526448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277826+0000","last_change":"2024-03-06T07:20:07.277826+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277826+0000","last_undegraded":"2024-03-06T07:20:07.277826+0000","last_fullsized":"2024-03-06T07:20:07.277826+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266355+0000","last_change":"2024-03-06T07:19:54.047960+0000","last_active":"2024-03-06T07:20:07.266355+0000","last_peered":"2024-03-06T07:20:07.266355+0000","last_clean":"2024-03-06T07:20:07.266355+0000","last_became_active":"2024-03-06T07:19:54.047638+0000","last_became_peered":"2024-03-06T07:19:54.047638+0000","last_unstale":"2024-03-06T07:20:07.266355+0000","last_undegraded":"2024-03-06T07:20:07.266355+0000","last_fullsized":"2024-03-06T07:20:07.266355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272580+0000","last_change":"2024-03-06T07:20:07.272580+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272580+0000","last_undegraded":"2024-03-06T07:20:07.272580+0000","last_fullsized":"2024-03-06T07:20:07.272580+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.269825+0000","last_change":"2024-03-06T07:19:54.048726+0000","last_active":"2024-03-06T07:20:07.269825+0000","last_peered":"2024-03-06T07:20:07.269825+0000","last_clean":"2024-03-06T07:20:07.269825+0000","last_became_active":"2024-03-06T07:19:54.048377+0000","last_became_peered":"2024-03-06T07:19:54.048377+0000","last_unstale":"2024-03-06T07:20:07.269825+0000","last_undegraded":"2024-03-06T07:20:07.269825+0000","last_fullsized":"2024-03-06T07:20:07.269825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:57.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271012+0000","last_change":"2024-03-06T07:20:07.271012+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271012+0000","last_undegraded":"2024-03-06T07:20:07.271012+0000","last_fullsized":"2024-03-06T07:20:07.271012+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273477+0000","last_change":"2024-03-06T07:19:54.059007+0000","last_active":"2024-03-06T07:20:07.273477+0000","last_peered":"2024-03-06T07:20:07.273477+0000","last_clean":"2024-03-06T07:20:07.273477+0000","last_became_active":"2024-03-06T07:19:54.058694+0000","last_became_peered":"2024-03-06T07:19:54.058694+0000","last_unstale":"2024-03-06T07:20:07.273477+0000","last_undegraded":"2024-03-06T07:20:07.273477+0000","last_fullsized":"2024-03-06T07:20:07.273477+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:50.508506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271093+0000","last_change":"2024-03-06T07:20:07.271093+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271093+0000","last_undegraded":"2024-03-06T07:20:07.271093+0000","last_fullsized":"2024-03-06T07:20:07.271093+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274539+0000","last_change":"2024-03-06T07:19:54.057769+0000","last_active":"2024-03-06T07:20:07.274539+0000","last_peered":"2024-03-06T07:20:07.274539+0000","last_clean":"2024-03-06T07:20:07.274539+0000","last_became_active":"2024-03-06T07:19:54.057496+0000","last_became_peered":"2024-03-06T07:19:54.057496+0000","last_unstale":"2024-03-06T07:20:07.274539+0000","last_undegraded":"2024-03-06T07:20:07.274539+0000","last_fullsized":"2024-03-06T07:20:07.274539+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:51.719701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278288+0000","last_change":"2024-03-06T07:20:07.278288+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278288+0000","last_undegraded":"2024-03-06T07:20:07.278288+0000","last_fullsized":"2024-03-06T07:20:07.278288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266695+0000","last_change":"2024-03-06T07:19:54.050005+0000","last_active":"2024-03-06T07:20:07.266695+0000","last_peered":"2024-03-06T07:20:07.266695+0000","last_clean":"2024-03-06T07:20:07.266695+0000","last_became_active":"2024-03-06T07:19:54.049750+0000","last_became_peered":"2024-03-06T07:19:54.049750+0000","last_unstale":"2024-03-06T07:20:07.266695+0000","last_undegraded":"2024-03-06T07:20:07.266695+0000","last_fullsized":"2024-03-06T07:20:07.266695+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:37.609871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.280462+0000","last_change":"2024-03-06T07:20:07.280462+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.280462+0000","last_undegraded":"2024-03-06T07:20:07.280462+0000","last_fullsized":"2024-03-06T07:20:07.280462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275862+0000","last_change":"2024-03-06T07:19:54.052367+0000","last_active":"2024-03-06T07:20:07.275862+0000","last_peered":"2024-03-06T07:20:07.275862+0000","last_clean":"2024-03-06T07:20:07.275862+0000","last_became_active":"2024-03-06T07:19:54.052119+0000","last_became_peered":"2024-03-06T07:19:54.052119+0000","last_unstale":"2024-03-06T07:20:07.275862+0000","last_undegraded":"2024-03-06T07:20:07.275862+0000","last_fullsized":"2024-03-06T07:20:07.275862+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.678839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278688+0000","last_change":"2024-03-06T07:20:07.278688+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278688+0000","last_undegraded":"2024-03-06T07:20:07.278688+0000","last_fullsized":"2024-03-06T07:20:07.278688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273827+0000","last_change":"2024-03-06T07:19:54.040294+0000","last_active":"2024-03-06T07:20:07.273827+0000","last_peered":"2024-03-06T07:20:07.273827+0000","last_clean":"2024-03-06T07:20:07.273827+0000","last_became_active":"2024-03-06T07:19:54.040048+0000","last_became_peered":"2024-03-06T07:19:54.040048+0000","last_unstale":"2024-03-06T07:20:07.273827+0000","last_undegraded":"2024-03-06T07:20:07.273827+0000","last_fullsized":"2024-03-06T07:20:07.273827+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:41.393636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277157+0000","last_change":"2024-03-06T07:20:07.277157+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277157+0000","last_undegraded":"2024-03-06T07:20:07.277157+0000","last_fullsized":"2024-03-06T07:20:07.277157+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266413+0000","last_change":"2024-03-06T07:19:54.056367+0000","last_active":"2024-03-06T07:20:07.266413+0000","last_peered":"2024-03-06T07:20:07.266413+0000","last_clean":"2024-03-06T07:20:07.266413+0000","last_became_active":"2024-03-06T07:19:54.056125+0000","last_became_peered":"2024-03-06T07:19:54.056125+0000","last_unstale":"2024-03-06T07:20:07.266413+0000","last_undegraded":"2024-03-06T07:20:07.266413+0000","last_fullsized":"2024-03-06T07:20:07.266413+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273689+0000","last_change":"2024-03-06T07:20:07.273689+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273689+0000","last_undegraded":"2024-03-06T07:20:07.273689+0000","last_fullsized":"2024-03-06T07:20:07.273689+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271282+0000","last_change":"2024-03-06T07:19:54.051456+0000","last_active":"2024-03-06T07:20:07.271282+0000","last_peered":"2024-03-06T07:20:07.271282+0000","last_clean":"2024-03-06T07:20:07.271282+0000","last_became_active":"2024-03-06T07:19:54.051193+0000","last_became_peered":"2024-03-06T07:19:54.051193+0000","last_unstale":"2024-03-06T07:20:07.271282+0000","last_undegraded":"2024-03-06T07:20:07.271282+0000","last_fullsized":"2024-03-06T07:20:07.271282+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276915+0000","last_change":"2024-03-06T07:20:07.276915+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276915+0000","last_undegraded":"2024-03-06T07:20:07.276915+0000","last_fullsized":"2024-03-06T07:20:07.276915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279180+0000","last_change":"2024-03-06T07:19:54.049011+0000","last_active":"2024-03-06T07:20:07.279180+0000","last_peered":"2024-03-06T07:20:07.279180+0000","last_clean":"2024-03-06T07:20:07.279180+0000","last_became_active":"2024-03-06T07:19:54.048738+0000","last_became_peered":"2024-03-06T07:19:54.048738+0000","last_unstale":"2024-03-06T07:20:07.279180+0000","last_undegraded":"2024-03-06T07:20:07.279180+0000","last_fullsized":"2024-03-06T07:20:07.279180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:36.367136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278431+0000","last_change":"2024-03-06T07:20:07.278431+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278431+0000","last_undegraded":"2024-03-06T07:20:07.278431+0000","last_fullsized":"2024-03-06T07:20:07.278431+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266230+0000","last_change":"2024-03-06T07:19:54.049339+0000","last_active":"2024-03-06T07:20:07.266230+0000","last_peered":"2024-03-06T07:20:07.266230+0000","last_clean":"2024-03-06T07:20:07.266230+0000","last_became_active":"2024-03-06T07:19:54.049059+0000","last_became_peered":"2024-03-06T07:19:54.049059+0000","last_unstale":"2024-03-06T07:20:07.266230+0000","last_undegraded":"2024-03-06T07:20:07.266230+0000","last_fullsized":"2024-03-06T07:20:07.266230+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272112+0000","last_change":"2024-03-06T07:20:07.272112+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272112+0000","last_undegraded":"2024-03-06T07:20:07.272112+0000","last_fullsized":"2024-03-06T07:20:07.272112+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272044+0000","last_change":"2024-03-06T07:19:54.058699+0000","last_active":"2024-03-06T07:20:07.272044+0000","last_peered":"2024-03-06T07:20:07.272044+0000","last_clean":"2024-03-06T07:20:07.272044+0000","last_became_active":"2024-03-06T07:19:54.056798+0000","last_became_peered":"2024-03-06T07:19:54.056798+0000","last_unstale":"2024-03-06T07:20:07.272044+0000","last_undegraded":"2024-03-06T07:20:07.272044+0000","last_fullsized":"2024-03-06T07:20:07.272044+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.279205+0000","last_change":"2024-03-06T07:20:07.279205+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.279205+0000","last_undegraded":"2024-03-06T07:20:07.279205+0000","last_fullsized":"2024-03-06T07:20:07.279205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283763+0000","last_change":"2024-03-06T07:19:54.062261+0000","last_active":"2024-03-06T07:20:07.283763+0000","last_peered":"2024-03-06T07:20:07.283763+0000","last_clean":"2024-03-06T07:20:07.283763+0000","last_became_active":"2024-03-06T07:19:54.061853+0000","last_became_peered":"2024-03-06T07:19:54.061853+0000","last_unstale":"2024-03-06T07:20:07.283763+0000","last_undegraded":"2024-03-06T07:20:07.283763+0000","last_fullsized":"2024-03-06T07:20:07.283763+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270647+0000","last_change":"2024-03-06T07:20:07.270647+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270647+0000","last_undegraded":"2024-03-06T07:20:07.270647+0000","last_fullsized":"2024-03-06T07:20:07.270647+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272164+0000","last_change":"2024-03-06T07:19:54.054672+0000","last_active":"2024-03-06T07:20:07.272164+0000","last_peered":"2024-03-06T07:20:07.272164+0000","last_clean":"2024-03-06T07:20:07.272164+0000","last_became_active":"2024-03-06T07:19:54.054301+0000","last_became_peered":"2024-03-06T07:19:54.054301+0000","last_unstale":"2024-03-06T07:20:07.272164+0000","last_undegraded":"2024-03-06T07:20:07.272164+0000","last_fullsized":"2024-03-06T07:20:07.272164+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:05.772166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272968+0000","last_change":"2024-03-06T07:20:07.272968+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272968+0000","last_undegraded":"2024-03-06T07:20:07.272968+0000","last_fullsized":"2024-03-06T07:20:07.272968+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275480+0000","last_change":"2024-03-06T07:19:54.051669+0000","last_active":"2024-03-06T07:20:07.275480+0000","last_peered":"2024-03-06T07:20:07.275480+0000","last_clean":"2024-03-06T07:20:07.275480+0000","last_became_active":"2024-03-06T07:19:54.051312+0000","last_became_peered":"2024-03-06T07:19:54.051312+0000","last_unstale":"2024-03-06T07:20:07.275480+0000","last_undegraded":"2024-03-06T07:20:07.275480+0000","last_fullsized":"2024-03-06T07:20:07.275480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277902+0000","last_change":"2024-03-06T07:20:07.277902+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277902+0000","last_undegraded":"2024-03-06T07:20:07.277902+0000","last_fullsized":"2024-03-06T07:20:07.277902+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275715+0000","last_change":"2024-03-06T07:19:54.047677+0000","last_active":"2024-03-06T07:20:07.275715+0000","last_peered":"2024-03-06T07:20:07.275715+0000","last_clean":"2024-03-06T07:20:07.275715+0000","last_became_active":"2024-03-06T07:19:54.045393+0000","last_became_peered":"2024-03-06T07:19:54.045393+0000","last_unstale":"2024-03-06T07:20:07.275715+0000","last_undegraded":"2024-03-06T07:20:07.275715+0000","last_fullsized":"2024-03-06T07:20:07.275715+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277366+0000","last_change":"2024-03-06T07:20:07.277366+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277366+0000","last_undegraded":"2024-03-06T07:20:07.277366+0000","last_fullsized":"2024-03-06T07:20:07.277366+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275908+0000","last_change":"2024-03-06T07:19:54.058376+0000","last_active":"2024-03-06T07:20:07.275908+0000","last_peered":"2024-03-06T07:20:07.275908+0000","last_clean":"2024-03-06T07:20:07.275908+0000","last_became_active":"2024-03-06T07:19:54.058151+0000","last_became_peered":"2024-03-06T07:19:54.058151+0000","last_unstale":"2024-03-06T07:20:07.275908+0000","last_undegraded":"2024-03-06T07:20:07.275908+0000","last_fullsized":"2024-03-06T07:20:07.275908+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:10.287168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276244+0000","last_change":"2024-03-06T07:20:07.276244+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276244+0000","last_undegraded":"2024-03-06T07:20:07.276244+0000","last_fullsized":"2024-03-06T07:20:07.276244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275078+0000","last_change":"2024-03-06T07:19:54.046473+0000","last_active":"2024-03-06T07:20:07.275078+0000","last_peered":"2024-03-06T07:20:07.275078+0000","last_clean":"2024-03-06T07:20:07.275078+0000","last_became_active":"2024-03-06T07:19:54.045977+0000","last_became_peered":"2024-03-06T07:19:54.045977+0000","last_unstale":"2024-03-06T07:20:07.275078+0000","last_undegraded":"2024-03-06T07:20:07.275078+0000","last_fullsized":"2024-03-06T07:20:07.275078+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:28.437704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278418+0000","last_change":"2024-03-06T07:20:07.278418+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278418+0000","last_undegraded":"2024-03-06T07:20:07.278418+0000","last_fullsized":"2024-03-06T07:20:07.278418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272271+0000","last_change":"2024-03-06T07:19:54.050649+0000","last_active":"2024-03-06T07:20:07.272271+0000","last_peered":"2024-03-06T07:20:07.272271+0000","last_clean":"2024-03-06T07:20:07.272271+0000","last_became_active":"2024-03-06T07:19:54.050392+0000","last_became_peered":"2024-03-06T07:19:54.050392+0000","last_unstale":"2024-03-06T07:20:07.272271+0000","last_undegraded":"2024-03-06T07:20:07.272271+0000","last_fullsized":"2024-03-06T07:20:07.272271+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272555+0000","last_change":"2024-03-06T07:20:07.272555+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272555+0000","last_undegraded":"2024-03-06T07:20:07.272555+0000","last_fullsized":"2024-03-06T07:20:07.272555+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272084+0000","last_change":"2024-03-06T07:19:54.046865+0000","last_active":"2024-03-06T07:20:07.272084+0000","last_peered":"2024-03-06T07:20:07.272084+0000","last_clean":"2024-03-06T07:20:07.272084+0000","last_became_active":"2024-03-06T07:19:54.046545+0000","last_became_peered":"2024-03-06T07:19:54.046545+0000","last_unstale":"2024-03-06T07:20:07.272084+0000","last_undegraded":"2024-03-06T07:20:07.272084+0000","last_fullsized":"2024-03-06T07:20:07.272084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:09.982751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270046+0000","last_change":"2024-03-06T07:20:07.270046+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270046+0000","last_undegraded":"2024-03-06T07:20:07.270046+0000","last_fullsized":"2024-03-06T07:20:07.270046+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273407+0000","last_change":"2024-03-06T07:19:54.042151+0000","last_active":"2024-03-06T07:20:07.273407+0000","last_peered":"2024-03-06T07:20:07.273407+0000","last_clean":"2024-03-06T07:20:07.273407+0000","last_became_active":"2024-03-06T07:19:54.041796+0000","last_became_peered":"2024-03-06T07:19:54.041796+0000","last_unstale":"2024-03-06T07:20:07.273407+0000","last_undegraded":"2024-03-06T07:20:07.273407+0000","last_fullsized":"2024-03-06T07:20:07.273407+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:27.407999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278821+0000","last_change":"2024-03-06T07:20:07.278821+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278821+0000","last_undegraded":"2024-03-06T07:20:07.278821+0000","last_fullsized":"2024-03-06T07:20:07.278821+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266779+0000","last_change":"2024-03-06T07:19:54.051161+0000","last_active":"2024-03-06T07:20:07.266779+0000","last_peered":"2024-03-06T07:20:07.266779+0000","last_clean":"2024-03-06T07:20:07.266779+0000","last_became_active":"2024-03-06T07:19:54.050893+0000","last_became_peered":"2024-03-06T07:19:54.050893+0000","last_unstale":"2024-03-06T07:20:07.266779+0000","last_undegraded":"2024-03-06T07:20:07.266779+0000","last_fullsized":"2024-03-06T07:20:07.266779+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:28.711633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277996+0000","last_change":"2024-03-06T07:20:07.277996+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277996+0000","last_undegraded":"2024-03-06T07:20:07.277996+0000","last_fullsized":"2024-03-06T07:20:07.277996+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275574+0000","last_change":"2024-03-06T07:19:54.040041+0000","last_active":"2024-03-06T07:20:07.275574+0000","last_peered":"2024-03-06T07:20:07.275574+0000","last_clean":"2024-03-06T07:20:07.275574+0000","last_became_active":"2024-03-06T07:19:54.039730+0000","last_became_peered":"2024-03-06T07:19:54.039730+0000","last_unstale":"2024-03-06T07:20:07.275574+0000","last_undegraded":"2024-03-06T07:20:07.275574+0000","last_fullsized":"2024-03-06T07:20:07.275574+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271067+0000","last_change":"2024-03-06T07:20:07.271067+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271067+0000","last_undegraded":"2024-03-06T07:20:07.271067+0000","last_fullsized":"2024-03-06T07:20:07.271067+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272272+0000","last_change":"2024-03-06T07:19:54.055165+0000","last_active":"2024-03-06T07:20:07.272272+0000","last_peered":"2024-03-06T07:20:07.272272+0000","last_clean":"2024-03-06T07:20:07.272272+0000","last_became_active":"2024-03-06T07:19:54.054908+0000","last_became_peered":"2024-03-06T07:19:54.054908+0000","last_unstale":"2024-03-06T07:20:07.272272+0000","last_undegraded":"2024-03-06T07:20:07.272272+0000","last_fullsized":"2024-03-06T07:20:07.272272+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:02.499776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270342+0000","last_change":"2024-03-06T07:20:07.270342+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270342+0000","last_undegraded":"2024-03-06T07:20:07.270342+0000","last_fullsized":"2024-03-06T07:20:07.270342+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278948+0000","last_change":"2024-03-06T07:19:54.049805+0000","last_active":"2024-03-06T07:20:07.278948+0000","last_peered":"2024-03-06T07:20:07.278948+0000","last_clean":"2024-03-06T07:20:07.278948+0000","last_became_active":"2024-03-06T07:19:54.049562+0000","last_became_peered":"2024-03-06T07:19:54.049562+0000","last_unstale":"2024-03-06T07:20:07.278948+0000","last_undegraded":"2024-03-06T07:20:07.278948+0000","last_fullsized":"2024-03-06T07:20:07.278948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:21.645621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276440+0000","last_change":"2024-03-06T07:20:07.276440+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276440+0000","last_undegraded":"2024-03-06T07:20:07.276440+0000","last_fullsized":"2024-03-06T07:20:07.276440+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.269928+0000","last_change":"2024-03-06T07:19:54.052763+0000","last_active":"2024-03-06T07:20:07.269928+0000","last_peered":"2024-03-06T07:20:07.269928+0000","last_clean":"2024-03-06T07:20:07.269928+0000","last_became_active":"2024-03-06T07:19:54.052472+0000","last_became_peered":"2024-03-06T07:19:54.052472+0000","last_unstale":"2024-03-06T07:20:07.269928+0000","last_undegraded":"2024-03-06T07:20:07.269928+0000","last_fullsized":"2024-03-06T07:20:07.269928+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:07.485843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277442+0000","last_change":"2024-03-06T07:20:07.277442+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277442+0000","last_undegraded":"2024-03-06T07:20:07.277442+0000","last_fullsized":"2024-03-06T07:20:07.277442+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279478+0000","last_change":"2024-03-06T07:19:54.059383+0000","last_active":"2024-03-06T07:20:07.279478+0000","last_peered":"2024-03-06T07:20:07.279478+0000","last_clean":"2024-03-06T07:20:07.279478+0000","last_became_active":"2024-03-06T07:19:54.059123+0000","last_became_peered":"2024-03-06T07:19:54.059123+0000","last_unstale":"2024-03-06T07:20:07.279478+0000","last_undegraded":"2024-03-06T07:20:07.279478+0000","last_fullsized":"2024-03-06T07:20:07.279478+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273940+0000","last_change":"2024-03-06T07:20:07.273940+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273940+0000","last_undegraded":"2024-03-06T07:20:07.273940+0000","last_fullsized":"2024-03-06T07:20:07.273940+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289071+0000","last_change":"2024-03-06T07:20:07.288076+0000","last_active":"2024-03-06T07:20:07.289071+0000","last_peered":"2024-03-06T07:20:07.289071+0000","last_clean":"2024-03-06T07:20:07.289071+0000","last_became_active":"2024-03-06T07:20:07.286803+0000","last_became_peered":"2024-03-06T07:20:07.286803+0000","last_unstale":"2024-03-06T07:20:07.289071+0000","last_undegraded":"2024-03-06T07:20:07.289071+0000","last_fullsized":"2024-03-06T07:20:07.289071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.832403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278691+0000","last_change":"2024-03-06T07:19:54.060233+0000","last_active":"2024-03-06T07:20:07.278691+0000","last_peered":"2024-03-06T07:20:07.278691+0000","last_clean":"2024-03-06T07:20:07.278691+0000","last_became_active":"2024-03-06T07:19:54.059901+0000","last_became_peered":"2024-03-06T07:19:54.059901+0000","last_unstale":"2024-03-06T07:20:07.278691+0000","last_undegraded":"2024-03-06T07:20:07.278691+0000","last_fullsized":"2024-03-06T07:20:07.278691+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:09.921962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275933+0000","last_change":"2024-03-06T07:19:53.042382+0000","last_active":"2024-03-06T07:20:07.275933+0000","last_peered":"2024-03-06T07:20:07.275933+0000","last_clean":"2024-03-06T07:20:07.275933+0000","last_became_active":"2024-03-06T07:19:53.041334+0000","last_became_peered":"2024-03-06T07:19:53.041334+0000","last_unstale":"2024-03-06T07:20:07.275933+0000","last_undegraded":"2024-03-06T07:20:07.275933+0000","last_fullsized":"2024-03-06T07:20:07.275933+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269519+0000","last_change":"2024-03-06T07:20:07.269519+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269519+0000","last_undegraded":"2024-03-06T07:20:07.269519+0000","last_fullsized":"2024-03-06T07:20:07.269519+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287328+0000","last_change":"2024-03-06T07:20:07.285089+0000","last_active":"2024-03-06T07:20:07.287328+0000","last_peered":"2024-03-06T07:20:07.287328+0000","last_clean":"2024-03-06T07:20:07.287328+0000","last_became_active":"2024-03-06T07:20:07.284839+0000","last_became_peered":"2024-03-06T07:20:07.284839+0000","last_unstale":"2024-03-06T07:20:07.287328+0000","last_undegraded":"2024-03-06T07:20:07.287328+0000","last_fullsized":"2024-03-06T07:20:07.287328+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:10.745713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279379+0000","last_change":"2024-03-06T07:19:54.057284+0000","last_active":"2024-03-06T07:20:07.279379+0000","last_peered":"2024-03-06T07:20:07.279379+0000","last_clean":"2024-03-06T07:20:07.279379+0000","last_became_active":"2024-03-06T07:19:54.056999+0000","last_became_peered":"2024-03-06T07:19:54.056999+0000","last_unstale":"2024-03-06T07:20:07.279379+0000","last_undegraded":"2024-03-06T07:20:07.279379+0000","last_fullsized":"2024-03-06T07:20:07.279379+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266885+0000","last_change":"2024-03-06T07:19:53.048675+0000","last_active":"2024-03-06T07:20:07.266885+0000","last_peered":"2024-03-06T07:20:07.266885+0000","last_clean":"2024-03-06T07:20:07.266885+0000","last_became_active":"2024-03-06T07:19:53.048416+0000","last_became_peered":"2024-03-06T07:19:53.048416+0000","last_unstale":"2024-03-06T07:20:07.266885+0000","last_undegraded":"2024-03-06T07:20:07.266885+0000","last_fullsized":"2024-03-06T07:20:07.266885+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271271+0000","last_change":"2024-03-06T07:20:07.271271+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271271+0000","last_undegraded":"2024-03-06T07:20:07.271271+0000","last_fullsized":"2024-03-06T07:20:07.271271+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287393+0000","last_change":"2024-03-06T07:20:07.285244+0000","last_active":"2024-03-06T07:20:07.287393+0000","last_peered":"2024-03-06T07:20:07.287393+0000","last_clean":"2024-03-06T07:20:07.287393+0000","last_became_active":"2024-03-06T07:20:07.285012+0000","last_became_peered":"2024-03-06T07:20:07.285012+0000","last_unstale":"2024-03-06T07:20:07.287393+0000","last_undegraded":"2024-03-06T07:20:07.287393+0000","last_fullsized":"2024-03-06T07:20:07.287393+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:10.657318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275244+0000","last_change":"2024-03-06T07:19:54.052199+0000","last_active":"2024-03-06T07:20:07.275244+0000","last_peered":"2024-03-06T07:20:07.275244+0000","last_clean":"2024-03-06T07:20:07.275244+0000","last_became_active":"2024-03-06T07:19:54.051923+0000","last_became_peered":"2024-03-06T07:19:54.051923+0000","last_unstale":"2024-03-06T07:20:07.275244+0000","last_undegraded":"2024-03-06T07:20:07.275244+0000","last_fullsized":"2024-03-06T07:20:07.275244+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266828+0000","last_change":"2024-03-06T07:19:53.044281+0000","last_active":"2024-03-06T07:20:07.266828+0000","last_peered":"2024-03-06T07:20:07.266828+0000","last_clean":"2024-03-06T07:20:07.266828+0000","last_became_active":"2024-03-06T07:19:53.043841+0000","last_became_peered":"2024-03-06T07:19:53.043841+0000","last_unstale":"2024-03-06T07:20:07.266828+0000","last_undegraded":"2024-03-06T07:20:07.266828+0000","last_fullsized":"2024-03-06T07:20:07.266828+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276210+0000","last_change":"2024-03-06T07:20:07.276210+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276210+0000","last_undegraded":"2024-03-06T07:20:07.276210+0000","last_fullsized":"2024-03-06T07:20:07.276210+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287847+0000","last_change":"2024-03-06T07:20:07.285494+0000","last_active":"2024-03-06T07:20:07.287847+0000","last_peered":"2024-03-06T07:20:07.287847+0000","last_clean":"2024-03-06T07:20:07.287847+0000","last_became_active":"2024-03-06T07:20:07.285238+0000","last_became_peered":"2024-03-06T07:20:07.285238+0000","last_unstale":"2024-03-06T07:20:07.287847+0000","last_undegraded":"2024-03-06T07:20:07.287847+0000","last_fullsized":"2024-03-06T07:20:07.287847+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.832403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274742+0000","last_change":"2024-03-06T07:19:54.039974+0000","last_active":"2024-03-06T07:20:07.274742+0000","last_peered":"2024-03-06T07:20:07.274742+0000","last_clean":"2024-03-06T07:20:07.274742+0000","last_became_active":"2024-03-06T07:19:54.039675+0000","last_became_peered":"2024-03-06T07:19:54.039675+0000","last_unstale":"2024-03-06T07:20:07.274742+0000","last_undegraded":"2024-03-06T07:20:07.274742+0000","last_fullsized":"2024-03-06T07:20:07.274742+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:53.089205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279819+0000","last_change":"2024-03-06T07:19:53.051640+0000","last_active":"2024-03-06T07:20:07.279819+0000","last_peered":"2024-03-06T07:20:07.279819+0000","last_clean":"2024-03-06T07:20:07.279819+0000","last_became_active":"2024-03-06T07:19:53.051429+0000","last_became_peered":"2024-03-06T07:19:53.051429+0000","last_unstale":"2024-03-06T07:20:07.279819+0000","last_undegraded":"2024-03-06T07:20:07.279819+0000","last_fullsized":"2024-03-06T07:20:07.279819+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:09.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271758+0000","last_change":"2024-03-06T07:20:07.271758+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271758+0000","last_undegraded":"2024-03-06T07:20:07.271758+0000","last_fullsized":"2024-03-06T07:20:07.271758+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288414+0000","last_change":"2024-03-06T07:20:07.287035+0000","last_active":"2024-03-06T07:20:07.288414+0000","last_peered":"2024-03-06T07:20:07.288414+0000","last_clean":"2024-03-06T07:20:07.288414+0000","last_became_active":"2024-03-06T07:20:07.286453+0000","last_became_peered":"2024-03-06T07:20:07.286453+0000","last_unstale":"2024-03-06T07:20:07.288414+0000","last_undegraded":"2024-03-06T07:20:07.288414+0000","last_fullsized":"2024-03-06T07:20:07.288414+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:15.726763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274553+0000","last_change":"2024-03-06T07:19:54.054949+0000","last_active":"2024-03-06T07:20:07.274553+0000","last_peered":"2024-03-06T07:20:07.274553+0000","last_clean":"2024-03-06T07:20:07.274553+0000","last_became_active":"2024-03-06T07:19:54.054661+0000","last_became_peered":"2024-03-06T07:19:54.054661+0000","last_unstale":"2024-03-06T07:20:07.274553+0000","last_undegraded":"2024-03-06T07:20:07.274553+0000","last_fullsized":"2024-03-06T07:20:07.274553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:57.129016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272386+0000","last_change":"2024-03-06T07:19:53.042003+0000","last_active":"2024-03-06T07:20:07.272386+0000","last_peered":"2024-03-06T07:20:07.272386+0000","last_clean":"2024-03-06T07:20:07.272386+0000","last_became_active":"2024-03-06T07:19:53.041728+0000","last_became_peered":"2024-03-06T07:19:53.041728+0000","last_unstale":"2024-03-06T07:20:07.272386+0000","last_undegraded":"2024-03-06T07:20:07.272386+0000","last_fullsized":"2024-03-06T07:20:07.272386+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.964097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271661+0000","last_change":"2024-03-06T07:20:07.271661+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271661+0000","last_undegraded":"2024-03-06T07:20:07.271661+0000","last_fullsized":"2024-03-06T07:20:07.271661+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284084+0000","last_change":"2024-03-06T07:20:07.278974+0000","last_active":"2024-03-06T07:20:07.284084+0000","last_peered":"2024-03-06T07:20:07.284084+0000","last_clean":"2024-03-06T07:20:07.284084+0000","last_became_active":"2024-03-06T07:20:07.278682+0000","last_became_peered":"2024-03-06T07:20:07.278682+0000","last_unstale":"2024-03-06T07:20:07.284084+0000","last_undegraded":"2024-03-06T07:20:07.284084+0000","last_fullsized":"2024-03-06T07:20:07.284084+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:27.012782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281002+0000","last_change":"2024-03-06T07:19:54.061284+0000","last_active":"2024-03-06T07:20:07.281002+0000","last_peered":"2024-03-06T07:20:07.281002+0000","last_clean":"2024-03-06T07:20:07.281002+0000","last_became_active":"2024-03-06T07:19:54.061017+0000","last_became_peered":"2024-03-06T07:19:54.061017+0000","last_unstale":"2024-03-06T07:20:07.281002+0000","last_undegraded":"2024-03-06T07:20:07.281002+0000","last_fullsized":"2024-03-06T07:20:07.281002+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274253+0000","last_change":"2024-03-06T07:19:53.041819+0000","last_active":"2024-03-06T07:20:07.274253+0000","last_peered":"2024-03-06T07:20:07.274253+0000","last_clean":"2024-03-06T07:20:07.274253+0000","last_became_active":"2024-03-06T07:19:53.041542+0000","last_became_peered":"2024-03-06T07:19:53.041542+0000","last_unstale":"2024-03-06T07:20:07.274253+0000","last_undegraded":"2024-03-06T07:20:07.274253+0000","last_fullsized":"2024-03-06T07:20:07.274253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269496+0000","last_change":"2024-03-06T07:20:07.269496+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269496+0000","last_undegraded":"2024-03-06T07:20:07.269496+0000","last_fullsized":"2024-03-06T07:20:07.269496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.290812+0000","last_change":"2024-03-06T07:20:07.289776+0000","last_active":"2024-03-06T07:20:07.290812+0000","last_peered":"2024-03-06T07:20:07.290812+0000","last_clean":"2024-03-06T07:20:07.290812+0000","last_became_active":"2024-03-06T07:20:07.289535+0000","last_became_peered":"2024-03-06T07:20:07.289535+0000","last_unstale":"2024-03-06T07:20:07.290812+0000","last_undegraded":"2024-03-06T07:20:07.290812+0000","last_fullsized":"2024-03-06T07:20:07.290812+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273691+0000","last_change":"2024-03-06T07:19:54.039491+0000","last_active":"2024-03-06T07:20:07.273691+0000","last_peered":"2024-03-06T07:20:07.273691+0000","last_clean":"2024-03-06T07:20:07.273691+0000","last_became_active":"2024-03-06T07:19:54.039202+0000","last_became_peered":"2024-03-06T07:19:54.039202+0000","last_unstale":"2024-03-06T07:20:07.273691+0000","last_undegraded":"2024-03-06T07:20:07.273691+0000","last_fullsized":"2024-03-06T07:20:07.273691+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:58.292241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275185+0000","last_change":"2024-03-06T07:19:53.043210+0000","last_active":"2024-03-06T07:20:07.275185+0000","last_peered":"2024-03-06T07:20:07.275185+0000","last_clean":"2024-03-06T07:20:07.275185+0000","last_became_active":"2024-03-06T07:19:53.042951+0000","last_became_peered":"2024-03-06T07:19:53.042951+0000","last_unstale":"2024-03-06T07:20:07.275185+0000","last_undegraded":"2024-03-06T07:20:07.275185+0000","last_fullsized":"2024-03-06T07:20:07.275185+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276607+0000","last_change":"2024-03-06T07:20:07.276607+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276607+0000","last_undegraded":"2024-03-06T07:20:07.276607+0000","last_fullsized":"2024-03-06T07:20:07.276607+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283717+0000","last_change":"2024-03-06T07:20:07.278534+0000","last_active":"2024-03-06T07:20:07.283717+0000","last_peered":"2024-03-06T07:20:07.283717+0000","last_clean":"2024-03-06T07:20:07.283717+0000","last_became_active":"2024-03-06T07:20:07.278261+0000","last_became_peered":"2024-03-06T07:20:07.278261+0000","last_unstale":"2024-03-06T07:20:07.283717+0000","last_undegraded":"2024-03-06T07:20:07.283717+0000","last_fullsized":"2024-03-06T07:20:07.283717+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:23.448306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285369+0000","last_change":"2024-03-06T07:19:54.049884+0000","last_active":"2024-03-06T07:20:07.285369+0000","last_peered":"2024-03-06T07:20:07.285369+0000","last_clean":"2024-03-06T07:20:07.285369+0000","last_became_active":"2024-03-06T07:19:54.049597+0000","last_became_peered":"2024-03-06T07:19:54.049597+0000","last_unstale":"2024-03-06T07:20:07.285369+0000","last_undegraded":"2024-03-06T07:20:07.285369+0000","last_fullsized":"2024-03-06T07:20:07.285369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.270618+0000","last_change":"2024-03-06T07:19:53.043198+0000","last_active":"2024-03-06T07:20:07.270618+0000","last_peered":"2024-03-06T07:20:07.270618+0000","last_clean":"2024-03-06T07:20:07.270618+0000","last_became_active":"2024-03-06T07:19:53.042939+0000","last_became_peered":"2024-03-06T07:19:53.042939+0000","last_unstale":"2024-03-06T07:20:07.270618+0000","last_undegraded":"2024-03-06T07:20:07.270618+0000","last_fullsized":"2024-03-06T07:20:07.270618+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273576+0000","last_change":"2024-03-06T07:20:07.273576+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273576+0000","last_undegraded":"2024-03-06T07:20:07.273576+0000","last_fullsized":"2024-03-06T07:20:07.273576+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285627+0000","last_change":"2024-03-06T07:20:07.283576+0000","last_active":"2024-03-06T07:20:07.285627+0000","last_peered":"2024-03-06T07:20:07.285627+0000","last_clean":"2024-03-06T07:20:07.285627+0000","last_became_active":"2024-03-06T07:20:07.283136+0000","last_became_peered":"2024-03-06T07:20:07.283136+0000","last_unstale":"2024-03-06T07:20:07.285627+0000","last_undegraded":"2024-03-06T07:20:07.285627+0000","last_fullsized":"2024-03-06T07:20:07.285627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:56.830929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274311+0000","last_change":"2024-03-06T07:19:54.045809+0000","last_active":"2024-03-06T07:20:07.274311+0000","last_peered":"2024-03-06T07:20:07.274311+0000","last_clean":"2024-03-06T07:20:07.274311+0000","last_became_active":"2024-03-06T07:19:54.040263+0000","last_became_peered":"2024-03-06T07:19:54.040263+0000","last_unstale":"2024-03-06T07:20:07.274311+0000","last_undegraded":"2024-03-06T07:20:07.274311+0000","last_fullsized":"2024-03-06T07:20:07.274311+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:20.728340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285454+0000","last_change":"2024-03-06T07:19:53.047864+0000","last_active":"2024-03-06T07:20:07.285454+0000","last_peered":"2024-03-06T07:20:07.285454+0000","last_clean":"2024-03-06T07:20:07.285454+0000","last_became_active":"2024-03-06T07:19:53.047607+0000","last_became_peered":"2024-03-06T07:19:53.047607+0000","last_unstale":"2024-03-06T07:20:07.285454+0000","last_undegraded":"2024-03-06T07:20:07.285454+0000","last_fullsized":"2024-03-06T07:20:07.285454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.148872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277242+0000","last_change":"2024-03-06T07:20:07.277242+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277242+0000","last_undegraded":"2024-03-06T07:20:07.277242+0000","last_fullsized":"2024-03-06T07:20:07.277242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288691+0000","last_change":"2024-03-06T07:20:07.285280+0000","last_active":"2024-03-06T07:20:07.288691+0000","last_peered":"2024-03-06T07:20:07.288691+0000","last_clean":"2024-03-06T07:20:07.288691+0000","last_became_active":"2024-03-06T07:20:07.284993+0000","last_became_peered":"2024-03-06T07:20:07.284993+0000","last_unstale":"2024-03-06T07:20:07.288691+0000","last_undegraded":"2024-03-06T07:20:07.288691+0000","last_fullsized":"2024-03-06T07:20:07.288691+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:59.458261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279291+0000","last_change":"2024-03-06T07:19:54.056898+0000","last_active":"2024-03-06T07:20:07.279291+0000","last_peered":"2024-03-06T07:20:07.279291+0000","last_clean":"2024-03-06T07:20:07.279291+0000","last_became_active":"2024-03-06T07:19:54.056522+0000","last_became_peered":"2024-03-06T07:19:54.056522+0000","last_unstale":"2024-03-06T07:20:07.279291+0000","last_undegraded":"2024-03-06T07:20:07.279291+0000","last_fullsized":"2024-03-06T07:20:07.279291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274521+0000","last_change":"2024-03-06T07:19:53.043065+0000","last_active":"2024-03-06T07:20:07.274521+0000","last_peered":"2024-03-06T07:20:07.274521+0000","last_clean":"2024-03-06T07:20:07.274521+0000","last_became_active":"2024-03-06T07:19:53.042756+0000","last_became_peered":"2024-03-06T07:19:53.042756+0000","last_unstale":"2024-03-06T07:20:07.274521+0000","last_undegraded":"2024-03-06T07:20:07.274521+0000","last_fullsized":"2024-03-06T07:20:07.274521+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.338932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274188+0000","last_change":"2024-03-06T07:20:07.274188+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274188+0000","last_undegraded":"2024-03-06T07:20:07.274188+0000","last_fullsized":"2024-03-06T07:20:07.274188+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285804+0000","last_change":"2024-03-06T07:20:07.282855+0000","last_active":"2024-03-06T07:20:07.285804+0000","last_peered":"2024-03-06T07:20:07.285804+0000","last_clean":"2024-03-06T07:20:07.285804+0000","last_became_active":"2024-03-06T07:20:07.282585+0000","last_became_peered":"2024-03-06T07:20:07.282585+0000","last_unstale":"2024-03-06T07:20:07.285804+0000","last_undegraded":"2024-03-06T07:20:07.285804+0000","last_fullsized":"2024-03-06T07:20:07.285804+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:52.503066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271456+0000","last_change":"2024-03-06T07:19:54.057647+0000","last_active":"2024-03-06T07:20:07.271456+0000","last_peered":"2024-03-06T07:20:07.271456+0000","last_clean":"2024-03-06T07:20:07.271456+0000","last_became_active":"2024-03-06T07:19:54.057396+0000","last_became_peered":"2024-03-06T07:19:54.057396+0000","last_unstale":"2024-03-06T07:20:07.271456+0000","last_undegraded":"2024-03-06T07:20:07.271456+0000","last_fullsized":"2024-03-06T07:20:07.271456+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":21,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281162+0000","last_change":"2024-03-06T07:19:53.046359+0000","last_active":"2024-03-06T07:20:07.281162+0000","last_peered":"2024-03-06T07:20:07.281162+0000","last_clean":"2024-03-06T07:20:07.281162+0000","last_became_active":"2024-03-06T07:19:53.046065+0000","last_became_peered":"2024-03-06T07:19:53.046065+0000","last_unstale":"2024-03-06T07:20:07.281162+0000","last_undegraded":"2024-03-06T07:20:07.281162+0000","last_fullsized":"2024-03-06T07:20:07.281162+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:08.146267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269510+0000","last_change":"2024-03-06T07:20:07.269510+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269510+0000","last_undegraded":"2024-03-06T07:20:07.269510+0000","last_fullsized":"2024-03-06T07:20:07.269510+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283148+0000","last_change":"2024-03-06T07:20:07.280915+0000","last_active":"2024-03-06T07:20:07.283148+0000","last_peered":"2024-03-06T07:20:07.283148+0000","last_clean":"2024-03-06T07:20:07.283148+0000","last_became_active":"2024-03-06T07:20:07.280659+0000","last_became_peered":"2024-03-06T07:20:07.280659+0000","last_unstale":"2024-03-06T07:20:07.283148+0000","last_undegraded":"2024-03-06T07:20:07.283148+0000","last_fullsized":"2024-03-06T07:20:07.283148+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:59.749257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274291+0000","last_change":"2024-03-06T07:19:54.041140+0000","last_active":"2024-03-06T07:20:07.274291+0000","last_peered":"2024-03-06T07:20:07.274291+0000","last_clean":"2024-03-06T07:20:07.274291+0000","last_became_active":"2024-03-06T07:19:54.040877+0000","last_became_peered":"2024-03-06T07:19:54.040877+0000","last_unstale":"2024-03-06T07:20:07.274291+0000","last_undegraded":"2024-03-06T07:20:07.274291+0000","last_fullsized":"2024-03-06T07:20:07.274291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:29.971359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":24,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273930+0000","last_change":"2024-03-06T07:19:53.045210+0000","last_active":"2024-03-06T07:20:07.273930+0000","last_peered":"2024-03-06T07:20:07.273930+0000","last_clean":"2024-03-06T07:20:07.273930+0000","last_became_active":"2024-03-06T07:19:53.044950+0000","last_became_peered":"2024-03-06T07:19:53.044950+0000","last_unstale":"2024-03-06T07:20:07.273930+0000","last_undegraded":"2024-03-06T07:20:07.273930+0000","last_fullsized":"2024-03-06T07:20:07.273930+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:53.407226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271405+0000","last_change":"2024-03-06T07:20:07.271405+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271405+0000","last_undegraded":"2024-03-06T07:20:07.271405+0000","last_fullsized":"2024-03-06T07:20:07.271405+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286730+0000","last_change":"2024-03-06T07:20:07.285841+0000","last_active":"2024-03-06T07:20:07.286730+0000","last_peered":"2024-03-06T07:20:07.286730+0000","last_clean":"2024-03-06T07:20:07.286730+0000","last_became_active":"2024-03-06T07:20:07.285505+0000","last_became_peered":"2024-03-06T07:20:07.285505+0000","last_unstale":"2024-03-06T07:20:07.286730+0000","last_undegraded":"2024-03-06T07:20:07.286730+0000","last_fullsized":"2024-03-06T07:20:07.286730+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:06.316192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279554+0000","last_change":"2024-03-06T07:19:54.058718+0000","last_active":"2024-03-06T07:20:07.279554+0000","last_peered":"2024-03-06T07:20:07.279554+0000","last_clean":"2024-03-06T07:20:07.279554+0000","last_became_active":"2024-03-06T07:19:54.058415+0000","last_became_peered":"2024-03-06T07:19:54.058415+0000","last_unstale":"2024-03-06T07:20:07.279554+0000","last_undegraded":"2024-03-06T07:20:07.279554+0000","last_fullsized":"2024-03-06T07:20:07.279554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272518+0000","last_change":"2024-03-06T07:19:53.046555+0000","last_active":"2024-03-06T07:20:07.272518+0000","last_peered":"2024-03-06T07:20:07.272518+0000","last_clean":"2024-03-06T07:20:07.272518+0000","last_became_active":"2024-03-06T07:19:53.046298+0000","last_became_peered":"2024-03-06T07:19:53.046298+0000","last_unstale":"2024-03-06T07:20:07.272518+0000","last_undegraded":"2024-03-06T07:20:07.272518+0000","last_fullsized":"2024-03-06T07:20:07.272518+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.243021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.280843+0000","last_change":"2024-03-06T07:20:07.280843+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.280843+0000","last_undegraded":"2024-03-06T07:20:07.280843+0000","last_fullsized":"2024-03-06T07:20:07.280843+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286818+0000","last_change":"2024-03-06T07:20:07.285895+0000","last_active":"2024-03-06T07:20:07.286818+0000","last_peered":"2024-03-06T07:20:07.286818+0000","last_clean":"2024-03-06T07:20:07.286818+0000","last_became_active":"2024-03-06T07:20:07.285614+0000","last_became_peered":"2024-03-06T07:20:07.285614+0000","last_unstale":"2024-03-06T07:20:07.286818+0000","last_undegraded":"2024-03-06T07:20:07.286818+0000","last_fullsized":"2024-03-06T07:20:07.286818+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:00.451319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272257+0000","last_change":"2024-03-06T07:19:54.047654+0000","last_active":"2024-03-06T07:20:07.272257+0000","last_peered":"2024-03-06T07:20:07.272257+0000","last_clean":"2024-03-06T07:20:07.272257+0000","last_became_active":"2024-03-06T07:19:54.047339+0000","last_became_peered":"2024-03-06T07:19:54.047339+0000","last_unstale":"2024-03-06T07:20:07.272257+0000","last_undegraded":"2024-03-06T07:20:07.272257+0000","last_fullsized":"2024-03-06T07:20:07.272257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:35.272829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275306+0000","last_change":"2024-03-06T07:19:53.040132+0000","last_active":"2024-03-06T07:20:07.275306+0000","last_peered":"2024-03-06T07:20:07.275306+0000","last_clean":"2024-03-06T07:20:07.275306+0000","last_became_active":"2024-03-06T07:19:53.039809+0000","last_became_peered":"2024-03-06T07:19:53.039809+0000","last_unstale":"2024-03-06T07:20:07.275306+0000","last_undegraded":"2024-03-06T07:20:07.275306+0000","last_fullsized":"2024-03-06T07:20:07.275306+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272239+0000","last_change":"2024-03-06T07:20:07.272239+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272239+0000","last_undegraded":"2024-03-06T07:20:07.272239+0000","last_fullsized":"2024-03-06T07:20:07.272239+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289344+0000","last_change":"2024-03-06T07:20:07.288210+0000","last_active":"2024-03-06T07:20:07.289344+0000","last_peered":"2024-03-06T07:20:07.289344+0000","last_clean":"2024-03-06T07:20:07.289344+0000","last_became_active":"2024-03-06T07:20:07.287885+0000","last_became_peered":"2024-03-06T07:20:07.287885+0000","last_unstale":"2024-03-06T07:20:07.289344+0000","last_undegraded":"2024-03-06T07:20:07.289344+0000","last_fullsized":"2024-03-06T07:20:07.289344+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:25.080610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273595+0000","last_change":"2024-03-06T07:19:54.050847+0000","last_active":"2024-03-06T07:20:07.273595+0000","last_peered":"2024-03-06T07:20:07.273595+0000","last_clean":"2024-03-06T07:20:07.273595+0000","last_became_active":"2024-03-06T07:19:54.050378+0000","last_became_peered":"2024-03-06T07:19:54.050378+0000","last_unstale":"2024-03-06T07:20:07.273595+0000","last_undegraded":"2024-03-06T07:20:07.273595+0000","last_fullsized":"2024-03-06T07:20:07.273595+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281018+0000","last_change":"2024-03-06T07:19:53.045139+0000","last_active":"2024-03-06T07:20:07.281018+0000","last_peered":"2024-03-06T07:20:07.281018+0000","last_clean":"2024-03-06T07:20:07.281018+0000","last_became_active":"2024-03-06T07:19:53.044903+0000","last_became_peered":"2024-03-06T07:19:53.044903+0000","last_unstale":"2024-03-06T07:20:07.281018+0000","last_undegraded":"2024-03-06T07:20:07.281018+0000","last_fullsized":"2024-03-06T07:20:07.281018+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:56.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272163+0000","last_change":"2024-03-06T07:20:07.272163+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272163+0000","last_undegraded":"2024-03-06T07:20:07.272163+0000","last_fullsized":"2024-03-06T07:20:07.272163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.290186+0000","last_change":"2024-03-06T07:20:07.289518+0000","last_active":"2024-03-06T07:20:07.290186+0000","last_peered":"2024-03-06T07:20:07.290186+0000","last_clean":"2024-03-06T07:20:07.290186+0000","last_became_active":"2024-03-06T07:20:07.289213+0000","last_became_peered":"2024-03-06T07:20:07.289213+0000","last_unstale":"2024-03-06T07:20:07.290186+0000","last_undegraded":"2024-03-06T07:20:07.290186+0000","last_fullsized":"2024-03-06T07:20:07.290186+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.514155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271904+0000","last_change":"2024-03-06T07:19:54.059322+0000","last_active":"2024-03-06T07:20:07.271904+0000","last_peered":"2024-03-06T07:20:07.271904+0000","last_clean":"2024-03-06T07:20:07.271904+0000","last_became_active":"2024-03-06T07:19:54.059092+0000","last_became_peered":"2024-03-06T07:19:54.059092+0000","last_unstale":"2024-03-06T07:20:07.271904+0000","last_undegraded":"2024-03-06T07:20:07.271904+0000","last_fullsized":"2024-03-06T07:20:07.271904+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":26,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275158+0000","last_change":"2024-03-06T07:19:53.046815+0000","last_active":"2024-03-06T07:20:07.275158+0000","last_peered":"2024-03-06T07:20:07.275158+0000","last_clean":"2024-03-06T07:20:07.275158+0000","last_became_active":"2024-03-06T07:19:53.043687+0000","last_became_peered":"2024-03-06T07:19:53.043687+0000","last_unstale":"2024-03-06T07:20:07.275158+0000","last_undegraded":"2024-03-06T07:20:07.275158+0000","last_fullsized":"2024-03-06T07:20:07.275158+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:44.465449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269497+0000","last_change":"2024-03-06T07:20:07.269497+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269497+0000","last_undegraded":"2024-03-06T07:20:07.269497+0000","last_fullsized":"2024-03-06T07:20:07.269497+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285213+0000","last_change":"2024-03-06T07:20:07.283273+0000","last_active":"2024-03-06T07:20:07.285213+0000","last_peered":"2024-03-06T07:20:07.285213+0000","last_clean":"2024-03-06T07:20:07.285213+0000","last_became_active":"2024-03-06T07:20:07.282989+0000","last_became_peered":"2024-03-06T07:20:07.282989+0000","last_unstale":"2024-03-06T07:20:07.285213+0000","last_undegraded":"2024-03-06T07:20:07.285213+0000","last_fullsized":"2024-03-06T07:20:07.285213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:08.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275055+0000","last_change":"2024-03-06T07:19:54.057521+0000","last_active":"2024-03-06T07:20:07.275055+0000","last_peered":"2024-03-06T07:20:07.275055+0000","last_clean":"2024-03-06T07:20:07.275055+0000","last_became_active":"2024-03-06T07:19:54.057264+0000","last_became_peered":"2024-03-06T07:19:54.057264+0000","last_unstale":"2024-03-06T07:20:07.275055+0000","last_undegraded":"2024-03-06T07:20:07.275055+0000","last_fullsized":"2024-03-06T07:20:07.275055+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272172+0000","last_change":"2024-03-06T07:19:53.051189+0000","last_active":"2024-03-06T07:20:07.272172+0000","last_peered":"2024-03-06T07:20:07.272172+0000","last_clean":"2024-03-06T07:20:07.272172+0000","last_became_active":"2024-03-06T07:19:53.050897+0000","last_became_peered":"2024-03-06T07:19:53.050897+0000","last_unstale":"2024-03-06T07:20:07.272172+0000","last_undegraded":"2024-03-06T07:20:07.272172+0000","last_fullsized":"2024-03-06T07:20:07.272172+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274169+0000","last_change":"2024-03-06T07:20:07.274169+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274169+0000","last_undegraded":"2024-03-06T07:20:07.274169+0000","last_fullsized":"2024-03-06T07:20:07.274169+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284678+0000","last_change":"2024-03-06T07:20:07.283920+0000","last_active":"2024-03-06T07:20:07.284678+0000","last_peered":"2024-03-06T07:20:07.284678+0000","last_clean":"2024-03-06T07:20:07.284678+0000","last_became_active":"2024-03-06T07:20:07.283631+0000","last_became_peered":"2024-03-06T07:20:07.283631+0000","last_unstale":"2024-03-06T07:20:07.284678+0000","last_undegraded":"2024-03-06T07:20:07.284678+0000","last_fullsized":"2024-03-06T07:20:07.284678+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:33.711210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266154+0000","last_change":"2024-03-06T07:19:54.052438+0000","last_active":"2024-03-06T07:20:07.266154+0000","last_peered":"2024-03-06T07:20:07.266154+0000","last_clean":"2024-03-06T07:20:07.266154+0000","last_became_active":"2024-03-06T07:19:54.052185+0000","last_became_peered":"2024-03-06T07:19:54.052185+0000","last_unstale":"2024-03-06T07:20:07.266154+0000","last_undegraded":"2024-03-06T07:20:07.266154+0000","last_fullsized":"2024-03-06T07:20:07.266154+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276246+0000","last_change":"2024-03-06T07:19:53.046903+0000","last_active":"2024-03-06T07:20:07.276246+0000","last_peered":"2024-03-06T07:20:07.276246+0000","last_clean":"2024-03-06T07:20:07.276246+0000","last_became_active":"2024-03-06T07:19:53.046639+0000","last_became_peered":"2024-03-06T07:19:53.046639+0000","last_unstale":"2024-03-06T07:20:07.276246+0000","last_undegraded":"2024-03-06T07:20:07.276246+0000","last_fullsized":"2024-03-06T07:20:07.276246+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:23.209022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273354+0000","last_change":"2024-03-06T07:20:07.273354+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273354+0000","last_undegraded":"2024-03-06T07:20:07.273354+0000","last_fullsized":"2024-03-06T07:20:07.273354+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289363+0000","last_change":"2024-03-06T07:20:07.288781+0000","last_active":"2024-03-06T07:20:07.289363+0000","last_peered":"2024-03-06T07:20:07.289363+0000","last_clean":"2024-03-06T07:20:07.289363+0000","last_became_active":"2024-03-06T07:20:07.288548+0000","last_became_peered":"2024-03-06T07:20:07.288548+0000","last_unstale":"2024-03-06T07:20:07.289363+0000","last_undegraded":"2024-03-06T07:20:07.289363+0000","last_fullsized":"2024-03-06T07:20:07.289363+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274167+0000","last_change":"2024-03-06T07:19:54.053795+0000","last_active":"2024-03-06T07:20:07.274167+0000","last_peered":"2024-03-06T07:20:07.274167+0000","last_clean":"2024-03-06T07:20:07.274167+0000","last_became_active":"2024-03-06T07:19:54.053519+0000","last_became_peered":"2024-03-06T07:19:54.053519+0000","last_unstale":"2024-03-06T07:20:07.274167+0000","last_undegraded":"2024-03-06T07:20:07.274167+0000","last_fullsized":"2024-03-06T07:20:07.274167+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:20.250283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275485+0000","last_change":"2024-03-06T07:19:53.046151+0000","last_active":"2024-03-06T07:20:07.275485+0000","last_peered":"2024-03-06T07:20:07.275485+0000","last_clean":"2024-03-06T07:20:07.275485+0000","last_became_active":"2024-03-06T07:19:53.045797+0000","last_became_peered":"2024-03-06T07:19:53.045797+0000","last_unstale":"2024-03-06T07:20:07.275485+0000","last_undegraded":"2024-03-06T07:20:07.275485+0000","last_fullsized":"2024-03-06T07:20:07.275485+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271460+0000","last_change":"2024-03-06T07:20:07.271460+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271460+0000","last_undegraded":"2024-03-06T07:20:07.271460+0000","last_fullsized":"2024-03-06T07:20:07.271460+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285658+0000","last_change":"2024-03-06T07:20:07.284617+0000","last_active":"2024-03-06T07:20:07.285658+0000","last_peered":"2024-03-06T07:20:07.285658+0000","last_clean":"2024-03-06T07:20:07.285658+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:07.285658+0000","last_undegraded":"2024-03-06T07:20:07.285658+0000","last_fullsized":"2024-03-06T07:20:07.285658+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:29.205056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266558+0000","last_change":"2024-03-06T07:19:54.055324+0000","last_active":"2024-03-06T07:20:07.266558+0000","last_peered":"2024-03-06T07:20:07.266558+0000","last_clean":"2024-03-06T07:20:07.266558+0000","last_became_active":"2024-03-06T07:19:54.055081+0000","last_became_peered":"2024-03-06T07:19:54.055081+0000","last_unstale":"2024-03-06T07:20:07.266558+0000","last_undegraded":"2024-03-06T07:20:07.266558+0000","last_fullsized":"2024-03-06T07:20:07.266558+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:57.518725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275654+0000","last_change":"2024-03-06T07:19:53.048898+0000","last_active":"2024-03-06T07:20:07.275654+0000","last_peered":"2024-03-06T07:20:07.275654+0000","last_clean":"2024-03-06T07:20:07.275654+0000","last_became_active":"2024-03-06T07:19:53.048613+0000","last_became_peered":"2024-03-06T07:19:53.048613+0000","last_unstale":"2024-03-06T07:20:07.275654+0000","last_undegraded":"2024-03-06T07:20:07.275654+0000","last_fullsized":"2024-03-06T07:20:07.275654+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277278+0000","last_change":"2024-03-06T07:20:07.277278+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277278+0000","last_undegraded":"2024-03-06T07:20:07.277278+0000","last_fullsized":"2024-03-06T07:20:07.277278+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286112+0000","last_change":"2024-03-06T07:20:07.285007+0000","last_active":"2024-03-06T07:20:07.286112+0000","last_peered":"2024-03-06T07:20:07.286112+0000","last_clean":"2024-03-06T07:20:07.286112+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:07.286112+0000","last_undegraded":"2024-03-06T07:20:07.286112+0000","last_fullsized":"2024-03-06T07:20:07.286112+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279669+0000","last_change":"2024-03-06T07:19:54.056310+0000","last_active":"2024-03-06T07:20:07.279669+0000","last_peered":"2024-03-06T07:20:07.279669+0000","last_clean":"2024-03-06T07:20:07.279669+0000","last_became_active":"2024-03-06T07:19:54.056039+0000","last_became_peered":"2024-03-06T07:19:54.056039+0000","last_unstale":"2024-03-06T07:20:07.279669+0000","last_undegraded":"2024-03-06T07:20:07.279669+0000","last_fullsized":"2024-03-06T07:20:07.279669+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266526+0000","last_change":"2024-03-06T07:19:53.043949+0000","last_active":"2024-03-06T07:20:07.266526+0000","last_peered":"2024-03-06T07:20:07.266526+0000","last_clean":"2024-03-06T07:20:07.266526+0000","last_became_active":"2024-03-06T07:19:53.043639+0000","last_became_peered":"2024-03-06T07:19:53.043639+0000","last_unstale":"2024-03-06T07:20:07.266526+0000","last_undegraded":"2024-03-06T07:20:07.266526+0000","last_fullsized":"2024-03-06T07:20:07.266526+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271451+0000","last_change":"2024-03-06T07:20:07.271451+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271451+0000","last_undegraded":"2024-03-06T07:20:07.271451+0000","last_fullsized":"2024-03-06T07:20:07.271451+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284102+0000","last_change":"2024-03-06T07:20:07.279146+0000","last_active":"2024-03-06T07:20:07.284102+0000","last_peered":"2024-03-06T07:20:07.284102+0000","last_clean":"2024-03-06T07:20:07.284102+0000","last_became_active":"2024-03-06T07:20:07.278872+0000","last_became_peered":"2024-03-06T07:20:07.278872+0000","last_unstale":"2024-03-06T07:20:07.284102+0000","last_undegraded":"2024-03-06T07:20:07.284102+0000","last_fullsized":"2024-03-06T07:20:07.284102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:34.974180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275593+0000","last_change":"2024-03-06T07:19:54.040887+0000","last_active":"2024-03-06T07:20:07.275593+0000","last_peered":"2024-03-06T07:20:07.275593+0000","last_clean":"2024-03-06T07:20:07.275593+0000","last_became_active":"2024-03-06T07:19:54.040632+0000","last_became_peered":"2024-03-06T07:19:54.040632+0000","last_unstale":"2024-03-06T07:20:07.275593+0000","last_undegraded":"2024-03-06T07:20:07.275593+0000","last_fullsized":"2024-03-06T07:20:07.275593+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276010+0000","last_change":"2024-03-06T07:19:53.044330+0000","last_active":"2024-03-06T07:20:07.276010+0000","last_peered":"2024-03-06T07:20:07.276010+0000","last_clean":"2024-03-06T07:20:07.276010+0000","last_became_active":"2024-03-06T07:19:53.044066+0000","last_became_peered":"2024-03-06T07:19:53.044066+0000","last_unstale":"2024-03-06T07:20:07.276010+0000","last_undegraded":"2024-03-06T07:20:07.276010+0000","last_fullsized":"2024-03-06T07:20:07.276010+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:19.476961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271863+0000","last_change":"2024-03-06T07:20:07.271863+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271863+0000","last_undegraded":"2024-03-06T07:20:07.271863+0000","last_fullsized":"2024-03-06T07:20:07.271863+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289018+0000","last_change":"2024-03-06T07:20:07.287646+0000","last_active":"2024-03-06T07:20:07.289018+0000","last_peered":"2024-03-06T07:20:07.289018+0000","last_clean":"2024-03-06T07:20:07.289018+0000","last_became_active":"2024-03-06T07:20:07.287378+0000","last_became_peered":"2024-03-06T07:20:07.287378+0000","last_unstale":"2024-03-06T07:20:07.289018+0000","last_undegraded":"2024-03-06T07:20:07.289018+0000","last_fullsized":"2024-03-06T07:20:07.289018+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272180+0000","last_change":"2024-03-06T07:19:54.049940+0000","last_active":"2024-03-06T07:20:07.272180+0000","last_peered":"2024-03-06T07:20:07.272180+0000","last_clean":"2024-03-06T07:20:07.272180+0000","last_became_active":"2024-03-06T07:19:54.049481+0000","last_became_peered":"2024-03-06T07:19:54.049481+0000","last_unstale":"2024-03-06T07:20:07.272180+0000","last_undegraded":"2024-03-06T07:20:07.272180+0000","last_fullsized":"2024-03-06T07:20:07.272180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.270036+0000","last_change":"2024-03-06T07:19:53.048472+0000","last_active":"2024-03-06T07:20:07.270036+0000","last_peered":"2024-03-06T07:20:07.270036+0000","last_clean":"2024-03-06T07:20:07.270036+0000","last_became_active":"2024-03-06T07:19:53.048213+0000","last_became_peered":"2024-03-06T07:19:53.048213+0000","last_unstale":"2024-03-06T07:20:07.270036+0000","last_undegraded":"2024-03-06T07:20:07.270036+0000","last_fullsized":"2024-03-06T07:20:07.270036+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:19.721078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274293+0000","last_change":"2024-03-06T07:20:07.274293+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274293+0000","last_undegraded":"2024-03-06T07:20:07.274293+0000","last_fullsized":"2024-03-06T07:20:07.274293+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286969+0000","last_change":"2024-03-06T07:20:07.283238+0000","last_active":"2024-03-06T07:20:07.286969+0000","last_peered":"2024-03-06T07:20:07.286969+0000","last_clean":"2024-03-06T07:20:07.286969+0000","last_became_active":"2024-03-06T07:20:07.282996+0000","last_became_peered":"2024-03-06T07:20:07.282996+0000","last_unstale":"2024-03-06T07:20:07.286969+0000","last_undegraded":"2024-03-06T07:20:07.286969+0000","last_fullsized":"2024-03-06T07:20:07.286969+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.712830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.267007+0000","last_change":"2024-03-06T07:19:54.042432+0000","last_active":"2024-03-06T07:20:07.267007+0000","last_peered":"2024-03-06T07:20:07.267007+0000","last_clean":"2024-03-06T07:20:07.267007+0000","last_became_active":"2024-03-06T07:19:54.042166+0000","last_became_peered":"2024-03-06T07:19:54.042166+0000","last_unstale":"2024-03-06T07:20:07.267007+0000","last_undegraded":"2024-03-06T07:20:07.267007+0000","last_fullsized":"2024-03-06T07:20:07.267007+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.231274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275074+0000","last_change":"2024-03-06T07:19:53.044950+0000","last_active":"2024-03-06T07:20:07.275074+0000","last_peered":"2024-03-06T07:20:07.275074+0000","last_clean":"2024-03-06T07:20:07.275074+0000","last_became_active":"2024-03-06T07:19:53.044630+0000","last_became_peered":"2024-03-06T07:19:53.044630+0000","last_unstale":"2024-03-06T07:20:07.275074+0000","last_undegraded":"2024-03-06T07:20:07.275074+0000","last_fullsized":"2024-03-06T07:20:07.275074+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:05.924599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272790+0000","last_change":"2024-03-06T07:20:07.272790+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272790+0000","last_undegraded":"2024-03-06T07:20:07.272790+0000","last_fullsized":"2024-03-06T07:20:07.272790+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287689+0000","last_change":"2024-03-06T07:20:07.287051+0000","last_active":"2024-03-06T07:20:07.287689+0000","last_peered":"2024-03-06T07:20:07.287689+0000","last_clean":"2024-03-06T07:20:07.287689+0000","last_became_active":"2024-03-06T07:20:07.286569+0000","last_became_peered":"2024-03-06T07:20:07.286569+0000","last_unstale":"2024-03-06T07:20:07.287689+0000","last_undegraded":"2024-03-06T07:20:07.287689+0000","last_fullsized":"2024-03-06T07:20:07.287689+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:17.755219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279576+0000","last_change":"2024-03-06T07:19:54.059591+0000","last_active":"2024-03-06T07:20:07.279576+0000","last_peered":"2024-03-06T07:20:07.279576+0000","last_clean":"2024-03-06T07:20:07.279576+0000","last_became_active":"2024-03-06T07:19:54.058814+0000","last_became_peered":"2024-03-06T07:19:54.058814+0000","last_unstale":"2024-03-06T07:20:07.279576+0000","last_undegraded":"2024-03-06T07:20:07.279576+0000","last_fullsized":"2024-03-06T07:20:07.279576+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275779+0000","last_change":"2024-03-06T07:19:53.040833+0000","last_active":"2024-03-06T07:20:07.275779+0000","last_peered":"2024-03-06T07:20:07.275779+0000","last_clean":"2024-03-06T07:20:07.275779+0000","last_became_active":"2024-03-06T07:19:53.040561+0000","last_became_peered":"2024-03-06T07:19:53.040561+0000","last_unstale":"2024-03-06T07:20:07.275779+0000","last_undegraded":"2024-03-06T07:20:07.275779+0000","last_fullsized":"2024-03-06T07:20:07.275779+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:12.315549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269859+0000","last_change":"2024-03-06T07:20:07.269859+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269859+0000","last_undegraded":"2024-03-06T07:20:07.269859+0000","last_fullsized":"2024-03-06T07:20:07.269859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281857+0000","last_change":"2024-03-06T07:20:07.280861+0000","last_active":"2024-03-06T07:20:07.281857+0000","last_peered":"2024-03-06T07:20:07.281857+0000","last_clean":"2024-03-06T07:20:07.281857+0000","last_became_active":"2024-03-06T07:20:07.280598+0000","last_became_peered":"2024-03-06T07:20:07.280598+0000","last_unstale":"2024-03-06T07:20:07.281857+0000","last_undegraded":"2024-03-06T07:20:07.281857+0000","last_fullsized":"2024-03-06T07:20:07.281857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266454+0000","last_change":"2024-03-06T07:19:54.042216+0000","last_active":"2024-03-06T07:20:07.266454+0000","last_peered":"2024-03-06T07:20:07.266454+0000","last_clean":"2024-03-06T07:20:07.266454+0000","last_became_active":"2024-03-06T07:19:54.041923+0000","last_became_peered":"2024-03-06T07:19:54.041923+0000","last_unstale":"2024-03-06T07:20:07.266454+0000","last_undegraded":"2024-03-06T07:20:07.266454+0000","last_fullsized":"2024-03-06T07:20:07.266454+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281579+0000","last_change":"2024-03-06T07:19:53.047959+0000","last_active":"2024-03-06T07:20:07.281579+0000","last_peered":"2024-03-06T07:20:07.281579+0000","last_clean":"2024-03-06T07:20:07.281579+0000","last_became_active":"2024-03-06T07:19:53.047680+0000","last_became_peered":"2024-03-06T07:19:53.047680+0000","last_unstale":"2024-03-06T07:20:07.281579+0000","last_undegraded":"2024-03-06T07:20:07.281579+0000","last_fullsized":"2024-03-06T07:20:07.281579+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:38.499903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276715+0000","last_change":"2024-03-06T07:20:07.276715+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276715+0000","last_undegraded":"2024-03-06T07:20:07.276715+0000","last_fullsized":"2024-03-06T07:20:07.276715+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287826+0000","last_change":"2024-03-06T07:20:07.284739+0000","last_active":"2024-03-06T07:20:07.287826+0000","last_peered":"2024-03-06T07:20:07.287826+0000","last_clean":"2024-03-06T07:20:07.287826+0000","last_became_active":"2024-03-06T07:20:07.284477+0000","last_became_peered":"2024-03-06T07:20:07.284477+0000","last_unstale":"2024-03-06T07:20:07.287826+0000","last_undegraded":"2024-03-06T07:20:07.287826+0000","last_fullsized":"2024-03-06T07:20:07.287826+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:47.605321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274844+0000","last_change":"2024-03-06T07:19:54.043516+0000","last_active":"2024-03-06T07:20:07.274844+0000","last_peered":"2024-03-06T07:20:07.274844+0000","last_clean":"2024-03-06T07:20:07.274844+0000","last_became_active":"2024-03-06T07:19:54.041651+0000","last_became_peered":"2024-03-06T07:19:54.041651+0000","last_unstale":"2024-03-06T07:20:07.274844+0000","last_undegraded":"2024-03-06T07:20:07.274844+0000","last_fullsized":"2024-03-06T07:20:07.274844+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:04.528232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272322+0000","last_change":"2024-03-06T07:19:53.048966+0000","last_active":"2024-03-06T07:20:07.272322+0000","last_peered":"2024-03-06T07:20:07.272322+0000","last_clean":"2024-03-06T07:20:07.272322+0000","last_became_active":"2024-03-06T07:19:53.047988+0000","last_became_peered":"2024-03-06T07:19:53.047988+0000","last_unstale":"2024-03-06T07:20:07.272322+0000","last_undegraded":"2024-03-06T07:20:07.272322+0000","last_fullsized":"2024-03-06T07:20:07.272322+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:49.519186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272529+0000","last_change":"2024-03-06T07:20:07.272529+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272529+0000","last_undegraded":"2024-03-06T07:20:07.272529+0000","last_fullsized":"2024-03-06T07:20:07.272529+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289051+0000","last_change":"2024-03-06T07:20:07.288004+0000","last_active":"2024-03-06T07:20:07.289051+0000","last_peered":"2024-03-06T07:20:07.289051+0000","last_clean":"2024-03-06T07:20:07.289051+0000","last_became_active":"2024-03-06T07:20:07.287745+0000","last_became_peered":"2024-03-06T07:20:07.287745+0000","last_unstale":"2024-03-06T07:20:07.289051+0000","last_undegraded":"2024-03-06T07:20:07.289051+0000","last_fullsized":"2024-03-06T07:20:07.289051+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:47.605321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272236+0000","last_change":"2024-03-06T07:19:54.062368+0000","last_active":"2024-03-06T07:20:07.272236+0000","last_peered":"2024-03-06T07:20:07.272236+0000","last_clean":"2024-03-06T07:20:07.272236+0000","last_became_active":"2024-03-06T07:19:54.060740+0000","last_became_peered":"2024-03-06T07:19:54.060740+0000","last_unstale":"2024-03-06T07:20:07.272236+0000","last_undegraded":"2024-03-06T07:20:07.272236+0000","last_fullsized":"2024-03-06T07:20:07.272236+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274942+0000","last_change":"2024-03-06T07:19:53.044944+0000","last_active":"2024-03-06T07:20:07.274942+0000","last_peered":"2024-03-06T07:20:07.274942+0000","last_clean":"2024-03-06T07:20:07.274942+0000","last_became_active":"2024-03-06T07:19:53.038900+0000","last_became_peered":"2024-03-06T07:19:53.038900+0000","last_unstale":"2024-03-06T07:20:07.274942+0000","last_undegraded":"2024-03-06T07:20:07.274942+0000","last_fullsized":"2024-03-06T07:20:07.274942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:59.062789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271846+0000","last_change":"2024-03-06T07:20:07.271846+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271846+0000","last_undegraded":"2024-03-06T07:20:07.271846+0000","last_fullsized":"2024-03-06T07:20:07.271846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284471+0000","last_change":"2024-03-06T07:20:07.281920+0000","last_active":"2024-03-06T07:20:07.284471+0000","last_peered":"2024-03-06T07:20:07.284471+0000","last_clean":"2024-03-06T07:20:07.284471+0000","last_became_active":"2024-03-06T07:20:07.281654+0000","last_became_peered":"2024-03-06T07:20:07.281654+0000","last_unstale":"2024-03-06T07:20:07.284471+0000","last_undegraded":"2024-03-06T07:20:07.284471+0000","last_fullsized":"2024-03-06T07:20:07.284471+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:44.082119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279797+0000","last_change":"2024-03-06T07:19:54.059347+0000","last_active":"2024-03-06T07:20:07.279797+0000","last_peered":"2024-03-06T07:20:07.279797+0000","last_clean":"2024-03-06T07:20:07.279797+0000","last_became_active":"2024-03-06T07:19:54.059083+0000","last_became_peered":"2024-03-06T07:19:54.059083+0000","last_unstale":"2024-03-06T07:20:07.279797+0000","last_undegraded":"2024-03-06T07:20:07.279797+0000","last_fullsized":"2024-03-06T07:20:07.279797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276107+0000","last_change":"2024-03-06T07:19:53.048645+0000","last_active":"2024-03-06T07:20:07.276107+0000","last_peered":"2024-03-06T07:20:07.276107+0000","last_clean":"2024-03-06T07:20:07.276107+0000","last_became_active":"2024-03-06T07:19:53.048406+0000","last_became_peered":"2024-03-06T07:19:53.048406+0000","last_unstale":"2024-03-06T07:20:07.276107+0000","last_undegraded":"2024-03-06T07:20:07.276107+0000","last_fullsized":"2024-03-06T07:20:07.276107+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:10.846361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278072+0000","last_change":"2024-03-06T07:20:07.278072+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278072+0000","last_undegraded":"2024-03-06T07:20:07.278072+0000","last_fullsized":"2024-03-06T07:20:07.278072+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288382+0000","last_change":"2024-03-06T07:20:07.287091+0000","last_active":"2024-03-06T07:20:07.288382+0000","last_peered":"2024-03-06T07:20:07.288382+0000","last_clean":"2024-03-06T07:20:07.288382+0000","last_became_active":"2024-03-06T07:20:07.286303+0000","last_became_peered":"2024-03-06T07:20:07.286303+0000","last_unstale":"2024-03-06T07:20:07.288382+0000","last_undegraded":"2024-03-06T07:20:07.288382+0000","last_fullsized":"2024-03-06T07:20:07.288382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271636+0000","last_change":"2024-03-06T07:19:54.049719+0000","last_active":"2024-03-06T07:20:07.271636+0000","last_peered":"2024-03-06T07:20:07.271636+0000","last_clean":"2024-03-06T07:20:07.271636+0000","last_became_active":"2024-03-06T07:19:54.049222+0000","last_became_peered":"2024-03-06T07:19:54.049222+0000","last_unstale":"2024-03-06T07:20:07.271636+0000","last_undegraded":"2024-03-06T07:20:07.271636+0000","last_fullsized":"2024-03-06T07:20:07.271636+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274213+0000","last_change":"2024-03-06T07:19:53.045046+0000","last_active":"2024-03-06T07:20:07.274213+0000","last_peered":"2024-03-06T07:20:07.274213+0000","last_clean":"2024-03-06T07:20:07.274213+0000","last_became_active":"2024-03-06T07:19:53.044751+0000","last_became_peered":"2024-03-06T07:19:53.044751+0000","last_unstale":"2024-03-06T07:20:07.274213+0000","last_undegraded":"2024-03-06T07:20:07.274213+0000","last_fullsized":"2024-03-06T07:20:07.274213+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272523+0000","last_change":"2024-03-06T07:20:07.272523+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272523+0000","last_undegraded":"2024-03-06T07:20:07.272523+0000","last_fullsized":"2024-03-06T07:20:07.272523+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289046+0000","last_change":"2024-03-06T07:20:07.287811+0000","last_active":"2024-03-06T07:20:07.289046+0000","last_peered":"2024-03-06T07:20:07.289046+0000","last_clean":"2024-03-06T07:20:07.289046+0000","last_became_active":"2024-03-06T07:20:07.287540+0000","last_became_peered":"2024-03-06T07:20:07.287540+0000","last_unstale":"2024-03-06T07:20:07.289046+0000","last_undegraded":"2024-03-06T07:20:07.289046+0000","last_fullsized":"2024-03-06T07:20:07.289046+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.712830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274628+0000","last_change":"2024-03-06T07:19:54.039796+0000","last_active":"2024-03-06T07:20:07.274628+0000","last_peered":"2024-03-06T07:20:07.274628+0000","last_clean":"2024-03-06T07:20:07.274628+0000","last_became_active":"2024-03-06T07:19:54.039527+0000","last_became_peered":"2024-03-06T07:19:54.039527+0000","last_unstale":"2024-03-06T07:20:07.274628+0000","last_undegraded":"2024-03-06T07:20:07.274628+0000","last_fullsized":"2024-03-06T07:20:07.274628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:49.114747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279603+0000","last_change":"2024-03-06T07:19:53.043599+0000","last_active":"2024-03-06T07:20:07.279603+0000","last_peered":"2024-03-06T07:20:07.279603+0000","last_clean":"2024-03-06T07:20:07.279603+0000","last_became_active":"2024-03-06T07:19:53.043294+0000","last_became_peered":"2024-03-06T07:19:53.043294+0000","last_unstale":"2024-03-06T07:20:07.279603+0000","last_undegraded":"2024-03-06T07:20:07.279603+0000","last_fullsized":"2024-03-06T07:20:07.279603+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:56.511464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271196+0000","last_change":"2024-03-06T07:20:07.271196+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271196+0000","last_undegraded":"2024-03-06T07:20:07.271196+0000","last_fullsized":"2024-03-06T07:20:07.271196+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278877+0000","last_change":"2024-03-06T07:20:07.277710+0000","last_active":"2024-03-06T07:20:07.278877+0000","last_peered":"2024-03-06T07:20:07.278877+0000","last_clean":"2024-03-06T07:20:07.278877+0000","last_became_active":"2024-03-06T07:20:07.277346+0000","last_became_peered":"2024-03-06T07:20:07.277346+0000","last_unstale":"2024-03-06T07:20:07.278877+0000","last_undegraded":"2024-03-06T07:20:07.278877+0000","last_fullsized":"2024-03-06T07:20:07.278877+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:51.111934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275167+0000","last_change":"2024-03-06T07:19:54.055814+0000","last_active":"2024-03-06T07:20:07.275167+0000","last_peered":"2024-03-06T07:20:07.275167+0000","last_clean":"2024-03-06T07:20:07.275167+0000","last_became_active":"2024-03-06T07:19:54.055480+0000","last_became_peered":"2024-03-06T07:19:54.055480+0000","last_unstale":"2024-03-06T07:20:07.275167+0000","last_undegraded":"2024-03-06T07:20:07.275167+0000","last_fullsized":"2024-03-06T07:20:07.275167+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:35.156134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271765+0000","last_change":"2024-03-06T07:19:53.046120+0000","last_active":"2024-03-06T07:20:07.271765+0000","last_peered":"2024-03-06T07:20:07.271765+0000","last_clean":"2024-03-06T07:20:07.271765+0000","last_became_active":"2024-03-06T07:19:53.045816+0000","last_became_peered":"2024-03-06T07:19:53.045816+0000","last_unstale":"2024-03-06T07:20:07.271765+0000","last_undegraded":"2024-03-06T07:20:07.271765+0000","last_fullsized":"2024-03-06T07:20:07.271765+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:56.226679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":234,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278863+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:20:07.278863+0000","last_peered":"2024-03-06T07:20:07.278863+0000","last_clean":"2024-03-06T07:20:07.278863+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:20:07.278863+0000","last_undegraded":"2024-03-06T07:20:07.278863+0000","last_fullsized":"2024-03-06T07:20:07.278863+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":873646,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":283806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":283154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":283736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":284788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":643072,"data_stored":284766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":283314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":873578,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:11.291 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:11.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.419+0000 7ffbef101640 1 Processor -- start 2024-03-06T07:20:11.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.419+0000 7ffbef101640 1 -- start start 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.419+0000 7ffbef101640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe814f650 0x7ffbe8071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.419+0000 7ffbef101640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe8072440 con 0x7ffbe80c3150 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.419+0000 7ffbef101640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe80c2b10 con 0x7ffbe8152c20 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.419+0000 7ffbef101640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe8152140 con 0x7ffbe814f650 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbe7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ffbe80c3150 0x7ffbe80c5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38580/0 (socket says 172.21.15.94:38580) 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbe7fff640 1 -- 172.21.15.94:0/3977345098 learned_addr learned my addr 172.21.15.94:0/3977345098 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbece9d640 1 --2- 172.21.15.94:0/3977345098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe814f650 0x7ffbe8071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbe77fe640 1 -- 172.21.15.94:0/3977345098 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593548571 0 0) 0x7ffbd4001110 con 0x7ffbe80c3150 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbe77fe640 1 -- 172.21.15.94:0/3977345098 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd8003520 con 0x7ffbe80c3150 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbe77fe640 1 -- 172.21.15.94:0/3977345098 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338540511 0 0) 0x7ffbe80c2b10 con 0x7ffbe8152c20 2024-03-06T07:20:11.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbece9d640 1 -- 172.21.15.94:0/3977345098 >> v1:172.21.15.145:6789/0 conn(0x7ffbe8152c20 legacy=0x7ffbe815c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbece9d640 1 -- 172.21.15.94:0/3977345098 >> v1:172.21.15.94:6789/0 conn(0x7ffbe80c3150 legacy=0x7ffbe80c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.420+0000 7ffbece9d640 1 -- 172.21.15.94:0/3977345098 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe8152420 con 0x7ffbe814f650 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbece9d640 1 --2- 172.21.15.94:0/3977345098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe814f650 0x7ffbe8071880 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0097d0 tx=0x7ffbdc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=628cc6848648dd22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbe77fe640 1 -- 172.21.15.94:0/3977345098 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc035020 con 0x7ffbe814f650 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbe77fe640 1 -- 172.21.15.94:0/3977345098 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbdc031890 con 0x7ffbe814f650 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbe77fe640 1 -- 172.21.15.94:0/3977345098 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc031c40 con 0x7ffbe814f650 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbef101640 1 -- 172.21.15.94:0/3977345098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe814f650 msgr2=0x7ffbe8071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbef101640 1 --2- 172.21.15.94:0/3977345098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe814f650 0x7ffbe8071880 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0097d0 tx=0x7ffbdc02b930 comp rx=0 tx=0).stop 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbef101640 1 -- 172.21.15.94:0/3977345098 shutdown_connections 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbef101640 1 --2- 172.21.15.94:0/3977345098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe814f650 0x7ffbe8071880 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbef101640 1 -- 172.21.15.94:0/3977345098 >> 172.21.15.94:0/3977345098 conn(0x7ffbe80c4290 msgr2=0x7ffbe80c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbef101640 1 -- 172.21.15.94:0/3977345098 shutdown_connections 2024-03-06T07:20:11.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.421+0000 7ffbef101640 1 -- 172.21.15.94:0/3977345098 wait complete. 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.422+0000 7ffbef101640 1 Processor -- start 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbef101640 1 -- start start 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbef101640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe80c3150 0x7ffbe8154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbef101640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe814f650 0x7ffbe814a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbef101640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbe8152c20 0x7ffbe81489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbef101640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80c40c0 con 0x7ffbe814f650 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbef101640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80c3f40 con 0x7ffbe8152c20 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbef101640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe80c5da0 con 0x7ffbe80c3150 2024-03-06T07:20:11.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbe7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe814f650 0x7ffbe814a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbe7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe814f650 0x7ffbe814a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37666/0 (socket says 172.21.15.94:37666) 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbe7fff640 1 -- 172.21.15.94:0/499729577 learned_addr learned my addr 172.21.15.94:0/499729577 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.423+0000 7ffbece9d640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe80c3150 0x7ffbe8154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbed69e640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbe8152c20 0x7ffbe81489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe7fff640 1 -- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe80c3150 msgr2=0x7ffbe8154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe7fff640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe80c3150 0x7ffbe8154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe7fff640 1 -- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbe8152c20 msgr2=0x7ffbe81489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe7fff640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbe8152c20 0x7ffbe81489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe7fff640 1 -- 172.21.15.94:0/499729577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe81edf40 con 0x7ffbe814f650 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe7fff640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe814f650 0x7ffbe814a370 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400c760 tx=0x7ffbd400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe57fa640 1 -- 172.21.15.94:0/499729577 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4017070 con 0x7ffbe814f650 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe57fa640 1 -- 172.21.15.94:0/499729577 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd4005ad0 con 0x7ffbe814f650 2024-03-06T07:20:11.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe81ee220 con 0x7ffbe814f650 2024-03-06T07:20:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbe57fa640 1 -- 172.21.15.94:0/499729577 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4007e20 con 0x7ffbe814f650 2024-03-06T07:20:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.424+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe81ee7e0 con 0x7ffbe814f650 2024-03-06T07:20:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.425+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe804f330 con 0x7ffbe814f650 2024-03-06T07:20:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.426+0000 7ffbe57fa640 1 -- 172.21.15.94:0/499729577 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbd401a070 con 0x7ffbe814f650 2024-03-06T07:20:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.426+0000 7ffbe57fa640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbcc06b4d0 0x7ffbcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.426+0000 7ffbe57fa640 1 -- 172.21.15.94:0/499729577 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ffbd4089a10 con 0x7ffbe814f650 2024-03-06T07:20:11.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.426+0000 7ffbece9d640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbcc06b4d0 0x7ffbcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.427+0000 7ffbece9d640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbcc06b4d0 0x7ffbcc06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc036000 tx=0x7ffbdc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.435+0000 7ffbe57fa640 1 -- 172.21.15.94:0/499729577 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbe8072440 con 0x7ffbe814f650 2024-03-06T07:20:11.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.637+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffbe8068460 con 0x7ffbcc06b4d0 2024-03-06T07:20:11.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.648+0000 7ffbe57fa640 1 -- 172.21.15.94:0/499729577 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485250 (secure 0 0 0) 0x7ffbdc0c07a0 con 0x7ffbcc06b4d0 2024-03-06T07:20:11.655 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:11.658 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbcc06b4d0 msgr2=0x7ffbcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbcc06b4d0 0x7ffbcc06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc036000 tx=0x7ffbdc008030 comp rx=0 tx=0).stop 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe814f650 msgr2=0x7ffbe814a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe814f650 0x7ffbe814a370 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400c760 tx=0x7ffbd400cc20 comp rx=0 tx=0).stop 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 shutdown_connections 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbcc06b4d0 0x7ffbcc06d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbe8152c20 0x7ffbe81489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbe814f650 0x7ffbe814a370 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 --2- 172.21.15.94:0/499729577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbe80c3150 0x7ffbe8154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 >> 172.21.15.94:0/499729577 conn(0x7ffbe80c4290 msgr2=0x7ffbe80c3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:11.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.652+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 shutdown_connections 2024-03-06T07:20:11.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.653+0000 7ffbef101640 1 -- 172.21.15.94:0/499729577 wait complete. 2024-03-06T07:20:11.669 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-03-06T07:20:10.659980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":209,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237920,"kb_used_data":5120,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736800,"statfs":{"total":773094113280,"available":772850483200,"internally_reserved":0,"allocated":5242880,"data_stored":3450788,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.396926"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276976+0000","last_change":"2024-03-06T07:20:07.276976+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276976+0000","last_undegraded":"2024-03-06T07:20:07.276976+0000","last_fullsized":"2024-03-06T07:20:07.276976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266870+0000","last_change":"2024-03-06T07:19:54.052304+0000","last_active":"2024-03-06T07:20:07.266870+0000","last_peered":"2024-03-06T07:20:07.266870+0000","last_clean":"2024-03-06T07:20:07.266870+0000","last_became_active":"2024-03-06T07:19:54.052048+0000","last_became_peered":"2024-03-06T07:19:54.052048+0000","last_unstale":"2024-03-06T07:20:07.266870+0000","last_undegraded":"2024-03-06T07:20:07.266870+0000","last_fullsized":"2024-03-06T07:20:07.266870+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:24.129824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270987+0000","last_change":"2024-03-06T07:20:07.270987+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270987+0000","last_undegraded":"2024-03-06T07:20:07.270987+0000","last_fullsized":"2024-03-06T07:20:07.270987+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283665+0000","last_change":"2024-03-06T07:19:54.062911+0000","last_active":"2024-03-06T07:20:07.283665+0000","last_peered":"2024-03-06T07:20:07.283665+0000","last_clean":"2024-03-06T07:20:07.283665+0000","last_became_active":"2024-03-06T07:19:54.062566+0000","last_became_peered":"2024-03-06T07:19:54.062566+0000","last_unstale":"2024-03-06T07:20:07.283665+0000","last_undegraded":"2024-03-06T07:20:07.283665+0000","last_fullsized":"2024-03-06T07:20:07.283665+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271068+0000","last_change":"2024-03-06T07:20:07.271068+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271068+0000","last_undegraded":"2024-03-06T07:20:07.271068+0000","last_fullsized":"2024-03-06T07:20:07.271068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272063+0000","last_change":"2024-03-06T07:19:54.051868+0000","last_active":"2024-03-06T07:20:07.272063+0000","last_peered":"2024-03-06T07:20:07.272063+0000","last_clean":"2024-03-06T07:20:07.272063+0000","last_became_active":"2024-03-06T07:19:54.051611+0000","last_became_peered":"2024-03-06T07:19:54.051611+0000","last_unstale":"2024-03-06T07:20:07.272063+0000","last_undegraded":"2024-03-06T07:20:07.272063+0000","last_fullsized":"2024-03-06T07:20:07.272063+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270217+0000","last_change":"2024-03-06T07:20:07.270217+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270217+0000","last_undegraded":"2024-03-06T07:20:07.270217+0000","last_fullsized":"2024-03-06T07:20:07.270217+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279217+0000","last_change":"2024-03-06T07:19:54.055553+0000","last_active":"2024-03-06T07:20:07.279217+0000","last_peered":"2024-03-06T07:20:07.279217+0000","last_clean":"2024-03-06T07:20:07.279217+0000","last_became_active":"2024-03-06T07:19:54.055271+0000","last_became_peered":"2024-03-06T07:19:54.055271+0000","last_unstale":"2024-03-06T07:20:07.279217+0000","last_undegraded":"2024-03-06T07:20:07.279217+0000","last_fullsized":"2024-03-06T07:20:07.279217+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274065+0000","last_change":"2024-03-06T07:20:07.274065+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274065+0000","last_undegraded":"2024-03-06T07:20:07.274065+0000","last_fullsized":"2024-03-06T07:20:07.274065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274273+0000","last_change":"2024-03-06T07:19:54.046161+0000","last_active":"2024-03-06T07:20:07.274273+0000","last_peered":"2024-03-06T07:20:07.274273+0000","last_clean":"2024-03-06T07:20:07.274273+0000","last_became_active":"2024-03-06T07:19:54.045875+0000","last_became_peered":"2024-03-06T07:19:54.045875+0000","last_unstale":"2024-03-06T07:20:07.274273+0000","last_undegraded":"2024-03-06T07:20:07.274273+0000","last_fullsized":"2024-03-06T07:20:07.274273+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:50.526448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277826+0000","last_change":"2024-03-06T07:20:07.277826+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277826+0000","last_undegraded":"2024-03-06T07:20:07.277826+0000","last_fullsized":"2024-03-06T07:20:07.277826+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266355+0000","last_change":"2024-03-06T07:19:54.047960+0000","last_active":"2024-03-06T07:20:07.266355+0000","last_peered":"2024-03-06T07:20:07.266355+0000","last_clean":"2024-03-06T07:20:07.266355+0000","last_became_active":"2024-03-06T07:19:54.047638+0000","last_became_peered":"2024-03-06T07:19:54.047638+0000","last_unstale":"2024-03-06T07:20:07.266355+0000","last_undegraded":"2024-03-06T07:20:07.266355+0000","last_fullsized":"2024-03-06T07:20:07.266355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272580+0000","last_change":"2024-03-06T07:20:07.272580+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272580+0000","last_undegraded":"2024-03-06T07:20:07.272580+0000","last_fullsized":"2024-03-06T07:20:07.272580+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.269825+0000","last_change":"2024-03-06T07:19:54.048726+0000","last_active":"2024-03-06T07:20:07.269825+0000","last_peered":"2024-03-06T07:20:07.269825+0000","last_clean":"2024-03-06T07:20:07.269825+0000","last_became_active":"2024-03-06T07:19:54.048377+0000","last_became_peered":"2024-03-06T07:19:54.048377+0000","last_unstale":"2024-03-06T07:20:07.269825+0000","last_undegraded":"2024-03-06T07:20:07.269825+0000","last_fullsized":"2024-03-06T07:20:07.269825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:57.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271012+0000","last_change":"2024-03-06T07:20:07.271012+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271012+0000","last_undegraded":"2024-03-06T07:20:07.271012+0000","last_fullsized":"2024-03-06T07:20:07.271012+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273477+0000","last_change":"2024-03-06T07:19:54.059007+0000","last_active":"2024-03-06T07:20:07.273477+0000","last_peered":"2024-03-06T07:20:07.273477+0000","last_clean":"2024-03-06T07:20:07.273477+0000","last_became_active":"2024-03-06T07:19:54.058694+0000","last_became_peered":"2024-03-06T07:19:54.058694+0000","last_unstale":"2024-03-06T07:20:07.273477+0000","last_undegraded":"2024-03-06T07:20:07.273477+0000","last_fullsized":"2024-03-06T07:20:07.273477+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:50.508506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271093+0000","last_change":"2024-03-06T07:20:07.271093+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271093+0000","last_undegraded":"2024-03-06T07:20:07.271093+0000","last_fullsized":"2024-03-06T07:20:07.271093+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274539+0000","last_change":"2024-03-06T07:19:54.057769+0000","last_active":"2024-03-06T07:20:07.274539+0000","last_peered":"2024-03-06T07:20:07.274539+0000","last_clean":"2024-03-06T07:20:07.274539+0000","last_became_active":"2024-03-06T07:19:54.057496+0000","last_became_peered":"2024-03-06T07:19:54.057496+0000","last_unstale":"2024-03-06T07:20:07.274539+0000","last_undegraded":"2024-03-06T07:20:07.274539+0000","last_fullsized":"2024-03-06T07:20:07.274539+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:51.719701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278288+0000","last_change":"2024-03-06T07:20:07.278288+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278288+0000","last_undegraded":"2024-03-06T07:20:07.278288+0000","last_fullsized":"2024-03-06T07:20:07.278288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266695+0000","last_change":"2024-03-06T07:19:54.050005+0000","last_active":"2024-03-06T07:20:07.266695+0000","last_peered":"2024-03-06T07:20:07.266695+0000","last_clean":"2024-03-06T07:20:07.266695+0000","last_became_active":"2024-03-06T07:19:54.049750+0000","last_became_peered":"2024-03-06T07:19:54.049750+0000","last_unstale":"2024-03-06T07:20:07.266695+0000","last_undegraded":"2024-03-06T07:20:07.266695+0000","last_fullsized":"2024-03-06T07:20:07.266695+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:37.609871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.280462+0000","last_change":"2024-03-06T07:20:07.280462+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.280462+0000","last_undegraded":"2024-03-06T07:20:07.280462+0000","last_fullsized":"2024-03-06T07:20:07.280462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275862+0000","last_change":"2024-03-06T07:19:54.052367+0000","last_active":"2024-03-06T07:20:07.275862+0000","last_peered":"2024-03-06T07:20:07.275862+0000","last_clean":"2024-03-06T07:20:07.275862+0000","last_became_active":"2024-03-06T07:19:54.052119+0000","last_became_peered":"2024-03-06T07:19:54.052119+0000","last_unstale":"2024-03-06T07:20:07.275862+0000","last_undegraded":"2024-03-06T07:20:07.275862+0000","last_fullsized":"2024-03-06T07:20:07.275862+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.678839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278688+0000","last_change":"2024-03-06T07:20:07.278688+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278688+0000","last_undegraded":"2024-03-06T07:20:07.278688+0000","last_fullsized":"2024-03-06T07:20:07.278688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273827+0000","last_change":"2024-03-06T07:19:54.040294+0000","last_active":"2024-03-06T07:20:07.273827+0000","last_peered":"2024-03-06T07:20:07.273827+0000","last_clean":"2024-03-06T07:20:07.273827+0000","last_became_active":"2024-03-06T07:19:54.040048+0000","last_became_peered":"2024-03-06T07:19:54.040048+0000","last_unstale":"2024-03-06T07:20:07.273827+0000","last_undegraded":"2024-03-06T07:20:07.273827+0000","last_fullsized":"2024-03-06T07:20:07.273827+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:41.393636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277157+0000","last_change":"2024-03-06T07:20:07.277157+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277157+0000","last_undegraded":"2024-03-06T07:20:07.277157+0000","last_fullsized":"2024-03-06T07:20:07.277157+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266413+0000","last_change":"2024-03-06T07:19:54.056367+0000","last_active":"2024-03-06T07:20:07.266413+0000","last_peered":"2024-03-06T07:20:07.266413+0000","last_clean":"2024-03-06T07:20:07.266413+0000","last_became_active":"2024-03-06T07:19:54.056125+0000","last_became_peered":"2024-03-06T07:19:54.056125+0000","last_unstale":"2024-03-06T07:20:07.266413+0000","last_undegraded":"2024-03-06T07:20:07.266413+0000","last_fullsized":"2024-03-06T07:20:07.266413+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273689+0000","last_change":"2024-03-06T07:20:07.273689+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273689+0000","last_undegraded":"2024-03-06T07:20:07.273689+0000","last_fullsized":"2024-03-06T07:20:07.273689+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271282+0000","last_change":"2024-03-06T07:19:54.051456+0000","last_active":"2024-03-06T07:20:07.271282+0000","last_peered":"2024-03-06T07:20:07.271282+0000","last_clean":"2024-03-06T07:20:07.271282+0000","last_became_active":"2024-03-06T07:19:54.051193+0000","last_became_peered":"2024-03-06T07:19:54.051193+0000","last_unstale":"2024-03-06T07:20:07.271282+0000","last_undegraded":"2024-03-06T07:20:07.271282+0000","last_fullsized":"2024-03-06T07:20:07.271282+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276915+0000","last_change":"2024-03-06T07:20:07.276915+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276915+0000","last_undegraded":"2024-03-06T07:20:07.276915+0000","last_fullsized":"2024-03-06T07:20:07.276915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279180+0000","last_change":"2024-03-06T07:19:54.049011+0000","last_active":"2024-03-06T07:20:07.279180+0000","last_peered":"2024-03-06T07:20:07.279180+0000","last_clean":"2024-03-06T07:20:07.279180+0000","last_became_active":"2024-03-06T07:19:54.048738+0000","last_became_peered":"2024-03-06T07:19:54.048738+0000","last_unstale":"2024-03-06T07:20:07.279180+0000","last_undegraded":"2024-03-06T07:20:07.279180+0000","last_fullsized":"2024-03-06T07:20:07.279180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:36.367136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278431+0000","last_change":"2024-03-06T07:20:07.278431+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278431+0000","last_undegraded":"2024-03-06T07:20:07.278431+0000","last_fullsized":"2024-03-06T07:20:07.278431+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266230+0000","last_change":"2024-03-06T07:19:54.049339+0000","last_active":"2024-03-06T07:20:07.266230+0000","last_peered":"2024-03-06T07:20:07.266230+0000","last_clean":"2024-03-06T07:20:07.266230+0000","last_became_active":"2024-03-06T07:19:54.049059+0000","last_became_peered":"2024-03-06T07:19:54.049059+0000","last_unstale":"2024-03-06T07:20:07.266230+0000","last_undegraded":"2024-03-06T07:20:07.266230+0000","last_fullsized":"2024-03-06T07:20:07.266230+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272112+0000","last_change":"2024-03-06T07:20:07.272112+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272112+0000","last_undegraded":"2024-03-06T07:20:07.272112+0000","last_fullsized":"2024-03-06T07:20:07.272112+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272044+0000","last_change":"2024-03-06T07:19:54.058699+0000","last_active":"2024-03-06T07:20:07.272044+0000","last_peered":"2024-03-06T07:20:07.272044+0000","last_clean":"2024-03-06T07:20:07.272044+0000","last_became_active":"2024-03-06T07:19:54.056798+0000","last_became_peered":"2024-03-06T07:19:54.056798+0000","last_unstale":"2024-03-06T07:20:07.272044+0000","last_undegraded":"2024-03-06T07:20:07.272044+0000","last_fullsized":"2024-03-06T07:20:07.272044+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.279205+0000","last_change":"2024-03-06T07:20:07.279205+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.279205+0000","last_undegraded":"2024-03-06T07:20:07.279205+0000","last_fullsized":"2024-03-06T07:20:07.279205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283763+0000","last_change":"2024-03-06T07:19:54.062261+0000","last_active":"2024-03-06T07:20:07.283763+0000","last_peered":"2024-03-06T07:20:07.283763+0000","last_clean":"2024-03-06T07:20:07.283763+0000","last_became_active":"2024-03-06T07:19:54.061853+0000","last_became_peered":"2024-03-06T07:19:54.061853+0000","last_unstale":"2024-03-06T07:20:07.283763+0000","last_undegraded":"2024-03-06T07:20:07.283763+0000","last_fullsized":"2024-03-06T07:20:07.283763+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270647+0000","last_change":"2024-03-06T07:20:07.270647+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270647+0000","last_undegraded":"2024-03-06T07:20:07.270647+0000","last_fullsized":"2024-03-06T07:20:07.270647+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272164+0000","last_change":"2024-03-06T07:19:54.054672+0000","last_active":"2024-03-06T07:20:07.272164+0000","last_peered":"2024-03-06T07:20:07.272164+0000","last_clean":"2024-03-06T07:20:07.272164+0000","last_became_active":"2024-03-06T07:19:54.054301+0000","last_became_peered":"2024-03-06T07:19:54.054301+0000","last_unstale":"2024-03-06T07:20:07.272164+0000","last_undegraded":"2024-03-06T07:20:07.272164+0000","last_fullsized":"2024-03-06T07:20:07.272164+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:05.772166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272968+0000","last_change":"2024-03-06T07:20:07.272968+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272968+0000","last_undegraded":"2024-03-06T07:20:07.272968+0000","last_fullsized":"2024-03-06T07:20:07.272968+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275480+0000","last_change":"2024-03-06T07:19:54.051669+0000","last_active":"2024-03-06T07:20:07.275480+0000","last_peered":"2024-03-06T07:20:07.275480+0000","last_clean":"2024-03-06T07:20:07.275480+0000","last_became_active":"2024-03-06T07:19:54.051312+0000","last_became_peered":"2024-03-06T07:19:54.051312+0000","last_unstale":"2024-03-06T07:20:07.275480+0000","last_undegraded":"2024-03-06T07:20:07.275480+0000","last_fullsized":"2024-03-06T07:20:07.275480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277902+0000","last_change":"2024-03-06T07:20:07.277902+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277902+0000","last_undegraded":"2024-03-06T07:20:07.277902+0000","last_fullsized":"2024-03-06T07:20:07.277902+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275715+0000","last_change":"2024-03-06T07:19:54.047677+0000","last_active":"2024-03-06T07:20:07.275715+0000","last_peered":"2024-03-06T07:20:07.275715+0000","last_clean":"2024-03-06T07:20:07.275715+0000","last_became_active":"2024-03-06T07:19:54.045393+0000","last_became_peered":"2024-03-06T07:19:54.045393+0000","last_unstale":"2024-03-06T07:20:07.275715+0000","last_undegraded":"2024-03-06T07:20:07.275715+0000","last_fullsized":"2024-03-06T07:20:07.275715+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277366+0000","last_change":"2024-03-06T07:20:07.277366+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277366+0000","last_undegraded":"2024-03-06T07:20:07.277366+0000","last_fullsized":"2024-03-06T07:20:07.277366+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275908+0000","last_change":"2024-03-06T07:19:54.058376+0000","last_active":"2024-03-06T07:20:07.275908+0000","last_peered":"2024-03-06T07:20:07.275908+0000","last_clean":"2024-03-06T07:20:07.275908+0000","last_became_active":"2024-03-06T07:19:54.058151+0000","last_became_peered":"2024-03-06T07:19:54.058151+0000","last_unstale":"2024-03-06T07:20:07.275908+0000","last_undegraded":"2024-03-06T07:20:07.275908+0000","last_fullsized":"2024-03-06T07:20:07.275908+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:10.287168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276244+0000","last_change":"2024-03-06T07:20:07.276244+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276244+0000","last_undegraded":"2024-03-06T07:20:07.276244+0000","last_fullsized":"2024-03-06T07:20:07.276244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275078+0000","last_change":"2024-03-06T07:19:54.046473+0000","last_active":"2024-03-06T07:20:07.275078+0000","last_peered":"2024-03-06T07:20:07.275078+0000","last_clean":"2024-03-06T07:20:07.275078+0000","last_became_active":"2024-03-06T07:19:54.045977+0000","last_became_peered":"2024-03-06T07:19:54.045977+0000","last_unstale":"2024-03-06T07:20:07.275078+0000","last_undegraded":"2024-03-06T07:20:07.275078+0000","last_fullsized":"2024-03-06T07:20:07.275078+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:28.437704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278418+0000","last_change":"2024-03-06T07:20:07.278418+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278418+0000","last_undegraded":"2024-03-06T07:20:07.278418+0000","last_fullsized":"2024-03-06T07:20:07.278418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272271+0000","last_change":"2024-03-06T07:19:54.050649+0000","last_active":"2024-03-06T07:20:07.272271+0000","last_peered":"2024-03-06T07:20:07.272271+0000","last_clean":"2024-03-06T07:20:07.272271+0000","last_became_active":"2024-03-06T07:19:54.050392+0000","last_became_peered":"2024-03-06T07:19:54.050392+0000","last_unstale":"2024-03-06T07:20:07.272271+0000","last_undegraded":"2024-03-06T07:20:07.272271+0000","last_fullsized":"2024-03-06T07:20:07.272271+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272555+0000","last_change":"2024-03-06T07:20:07.272555+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272555+0000","last_undegraded":"2024-03-06T07:20:07.272555+0000","last_fullsized":"2024-03-06T07:20:07.272555+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272084+0000","last_change":"2024-03-06T07:19:54.046865+0000","last_active":"2024-03-06T07:20:07.272084+0000","last_peered":"2024-03-06T07:20:07.272084+0000","last_clean":"2024-03-06T07:20:07.272084+0000","last_became_active":"2024-03-06T07:19:54.046545+0000","last_became_peered":"2024-03-06T07:19:54.046545+0000","last_unstale":"2024-03-06T07:20:07.272084+0000","last_undegraded":"2024-03-06T07:20:07.272084+0000","last_fullsized":"2024-03-06T07:20:07.272084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:09.982751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270046+0000","last_change":"2024-03-06T07:20:07.270046+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270046+0000","last_undegraded":"2024-03-06T07:20:07.270046+0000","last_fullsized":"2024-03-06T07:20:07.270046+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273407+0000","last_change":"2024-03-06T07:19:54.042151+0000","last_active":"2024-03-06T07:20:07.273407+0000","last_peered":"2024-03-06T07:20:07.273407+0000","last_clean":"2024-03-06T07:20:07.273407+0000","last_became_active":"2024-03-06T07:19:54.041796+0000","last_became_peered":"2024-03-06T07:19:54.041796+0000","last_unstale":"2024-03-06T07:20:07.273407+0000","last_undegraded":"2024-03-06T07:20:07.273407+0000","last_fullsized":"2024-03-06T07:20:07.273407+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:27.407999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278821+0000","last_change":"2024-03-06T07:20:07.278821+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278821+0000","last_undegraded":"2024-03-06T07:20:07.278821+0000","last_fullsized":"2024-03-06T07:20:07.278821+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266779+0000","last_change":"2024-03-06T07:19:54.051161+0000","last_active":"2024-03-06T07:20:07.266779+0000","last_peered":"2024-03-06T07:20:07.266779+0000","last_clean":"2024-03-06T07:20:07.266779+0000","last_became_active":"2024-03-06T07:19:54.050893+0000","last_became_peered":"2024-03-06T07:19:54.050893+0000","last_unstale":"2024-03-06T07:20:07.266779+0000","last_undegraded":"2024-03-06T07:20:07.266779+0000","last_fullsized":"2024-03-06T07:20:07.266779+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:28.711633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277996+0000","last_change":"2024-03-06T07:20:07.277996+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277996+0000","last_undegraded":"2024-03-06T07:20:07.277996+0000","last_fullsized":"2024-03-06T07:20:07.277996+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275574+0000","last_change":"2024-03-06T07:19:54.040041+0000","last_active":"2024-03-06T07:20:07.275574+0000","last_peered":"2024-03-06T07:20:07.275574+0000","last_clean":"2024-03-06T07:20:07.275574+0000","last_became_active":"2024-03-06T07:19:54.039730+0000","last_became_peered":"2024-03-06T07:19:54.039730+0000","last_unstale":"2024-03-06T07:20:07.275574+0000","last_undegraded":"2024-03-06T07:20:07.275574+0000","last_fullsized":"2024-03-06T07:20:07.275574+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271067+0000","last_change":"2024-03-06T07:20:07.271067+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271067+0000","last_undegraded":"2024-03-06T07:20:07.271067+0000","last_fullsized":"2024-03-06T07:20:07.271067+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272272+0000","last_change":"2024-03-06T07:19:54.055165+0000","last_active":"2024-03-06T07:20:07.272272+0000","last_peered":"2024-03-06T07:20:07.272272+0000","last_clean":"2024-03-06T07:20:07.272272+0000","last_became_active":"2024-03-06T07:19:54.054908+0000","last_became_peered":"2024-03-06T07:19:54.054908+0000","last_unstale":"2024-03-06T07:20:07.272272+0000","last_undegraded":"2024-03-06T07:20:07.272272+0000","last_fullsized":"2024-03-06T07:20:07.272272+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:02.499776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270342+0000","last_change":"2024-03-06T07:20:07.270342+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270342+0000","last_undegraded":"2024-03-06T07:20:07.270342+0000","last_fullsized":"2024-03-06T07:20:07.270342+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278948+0000","last_change":"2024-03-06T07:19:54.049805+0000","last_active":"2024-03-06T07:20:07.278948+0000","last_peered":"2024-03-06T07:20:07.278948+0000","last_clean":"2024-03-06T07:20:07.278948+0000","last_became_active":"2024-03-06T07:19:54.049562+0000","last_became_peered":"2024-03-06T07:19:54.049562+0000","last_unstale":"2024-03-06T07:20:07.278948+0000","last_undegraded":"2024-03-06T07:20:07.278948+0000","last_fullsized":"2024-03-06T07:20:07.278948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:21.645621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276440+0000","last_change":"2024-03-06T07:20:07.276440+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276440+0000","last_undegraded":"2024-03-06T07:20:07.276440+0000","last_fullsized":"2024-03-06T07:20:07.276440+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.269928+0000","last_change":"2024-03-06T07:19:54.052763+0000","last_active":"2024-03-06T07:20:07.269928+0000","last_peered":"2024-03-06T07:20:07.269928+0000","last_clean":"2024-03-06T07:20:07.269928+0000","last_became_active":"2024-03-06T07:19:54.052472+0000","last_became_peered":"2024-03-06T07:19:54.052472+0000","last_unstale":"2024-03-06T07:20:07.269928+0000","last_undegraded":"2024-03-06T07:20:07.269928+0000","last_fullsized":"2024-03-06T07:20:07.269928+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:07.485843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277442+0000","last_change":"2024-03-06T07:20:07.277442+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277442+0000","last_undegraded":"2024-03-06T07:20:07.277442+0000","last_fullsized":"2024-03-06T07:20:07.277442+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279478+0000","last_change":"2024-03-06T07:19:54.059383+0000","last_active":"2024-03-06T07:20:07.279478+0000","last_peered":"2024-03-06T07:20:07.279478+0000","last_clean":"2024-03-06T07:20:07.279478+0000","last_became_active":"2024-03-06T07:19:54.059123+0000","last_became_peered":"2024-03-06T07:19:54.059123+0000","last_unstale":"2024-03-06T07:20:07.279478+0000","last_undegraded":"2024-03-06T07:20:07.279478+0000","last_fullsized":"2024-03-06T07:20:07.279478+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273940+0000","last_change":"2024-03-06T07:20:07.273940+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273940+0000","last_undegraded":"2024-03-06T07:20:07.273940+0000","last_fullsized":"2024-03-06T07:20:07.273940+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289071+0000","last_change":"2024-03-06T07:20:07.288076+0000","last_active":"2024-03-06T07:20:07.289071+0000","last_peered":"2024-03-06T07:20:07.289071+0000","last_clean":"2024-03-06T07:20:07.289071+0000","last_became_active":"2024-03-06T07:20:07.286803+0000","last_became_peered":"2024-03-06T07:20:07.286803+0000","last_unstale":"2024-03-06T07:20:07.289071+0000","last_undegraded":"2024-03-06T07:20:07.289071+0000","last_fullsized":"2024-03-06T07:20:07.289071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.832403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278691+0000","last_change":"2024-03-06T07:19:54.060233+0000","last_active":"2024-03-06T07:20:07.278691+0000","last_peered":"2024-03-06T07:20:07.278691+0000","last_clean":"2024-03-06T07:20:07.278691+0000","last_became_active":"2024-03-06T07:19:54.059901+0000","last_became_peered":"2024-03-06T07:19:54.059901+0000","last_unstale":"2024-03-06T07:20:07.278691+0000","last_undegraded":"2024-03-06T07:20:07.278691+0000","last_fullsized":"2024-03-06T07:20:07.278691+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:09.921962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275933+0000","last_change":"2024-03-06T07:19:53.042382+0000","last_active":"2024-03-06T07:20:07.275933+0000","last_peered":"2024-03-06T07:20:07.275933+0000","last_clean":"2024-03-06T07:20:07.275933+0000","last_became_active":"2024-03-06T07:19:53.041334+0000","last_became_peered":"2024-03-06T07:19:53.041334+0000","last_unstale":"2024-03-06T07:20:07.275933+0000","last_undegraded":"2024-03-06T07:20:07.275933+0000","last_fullsized":"2024-03-06T07:20:07.275933+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269519+0000","last_change":"2024-03-06T07:20:07.269519+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269519+0000","last_undegraded":"2024-03-06T07:20:07.269519+0000","last_fullsized":"2024-03-06T07:20:07.269519+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287328+0000","last_change":"2024-03-06T07:20:07.285089+0000","last_active":"2024-03-06T07:20:07.287328+0000","last_peered":"2024-03-06T07:20:07.287328+0000","last_clean":"2024-03-06T07:20:07.287328+0000","last_became_active":"2024-03-06T07:20:07.284839+0000","last_became_peered":"2024-03-06T07:20:07.284839+0000","last_unstale":"2024-03-06T07:20:07.287328+0000","last_undegraded":"2024-03-06T07:20:07.287328+0000","last_fullsized":"2024-03-06T07:20:07.287328+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:10.745713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279379+0000","last_change":"2024-03-06T07:19:54.057284+0000","last_active":"2024-03-06T07:20:07.279379+0000","last_peered":"2024-03-06T07:20:07.279379+0000","last_clean":"2024-03-06T07:20:07.279379+0000","last_became_active":"2024-03-06T07:19:54.056999+0000","last_became_peered":"2024-03-06T07:19:54.056999+0000","last_unstale":"2024-03-06T07:20:07.279379+0000","last_undegraded":"2024-03-06T07:20:07.279379+0000","last_fullsized":"2024-03-06T07:20:07.279379+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266885+0000","last_change":"2024-03-06T07:19:53.048675+0000","last_active":"2024-03-06T07:20:07.266885+0000","last_peered":"2024-03-06T07:20:07.266885+0000","last_clean":"2024-03-06T07:20:07.266885+0000","last_became_active":"2024-03-06T07:19:53.048416+0000","last_became_peered":"2024-03-06T07:19:53.048416+0000","last_unstale":"2024-03-06T07:20:07.266885+0000","last_undegraded":"2024-03-06T07:20:07.266885+0000","last_fullsized":"2024-03-06T07:20:07.266885+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271271+0000","last_change":"2024-03-06T07:20:07.271271+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271271+0000","last_undegraded":"2024-03-06T07:20:07.271271+0000","last_fullsized":"2024-03-06T07:20:07.271271+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287393+0000","last_change":"2024-03-06T07:20:07.285244+0000","last_active":"2024-03-06T07:20:07.287393+0000","last_peered":"2024-03-06T07:20:07.287393+0000","last_clean":"2024-03-06T07:20:07.287393+0000","last_became_active":"2024-03-06T07:20:07.285012+0000","last_became_peered":"2024-03-06T07:20:07.285012+0000","last_unstale":"2024-03-06T07:20:07.287393+0000","last_undegraded":"2024-03-06T07:20:07.287393+0000","last_fullsized":"2024-03-06T07:20:07.287393+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:10.657318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275244+0000","last_change":"2024-03-06T07:19:54.052199+0000","last_active":"2024-03-06T07:20:07.275244+0000","last_peered":"2024-03-06T07:20:07.275244+0000","last_clean":"2024-03-06T07:20:07.275244+0000","last_became_active":"2024-03-06T07:19:54.051923+0000","last_became_peered":"2024-03-06T07:19:54.051923+0000","last_unstale":"2024-03-06T07:20:07.275244+0000","last_undegraded":"2024-03-06T07:20:07.275244+0000","last_fullsized":"2024-03-06T07:20:07.275244+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266828+0000","last_change":"2024-03-06T07:19:53.044281+0000","last_active":"2024-03-06T07:20:07.266828+0000","last_peered":"2024-03-06T07:20:07.266828+0000","last_clean":"2024-03-06T07:20:07.266828+0000","last_became_active":"2024-03-06T07:19:53.043841+0000","last_became_peered":"2024-03-06T07:19:53.043841+0000","last_unstale":"2024-03-06T07:20:07.266828+0000","last_undegraded":"2024-03-06T07:20:07.266828+0000","last_fullsized":"2024-03-06T07:20:07.266828+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276210+0000","last_change":"2024-03-06T07:20:07.276210+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276210+0000","last_undegraded":"2024-03-06T07:20:07.276210+0000","last_fullsized":"2024-03-06T07:20:07.276210+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287847+0000","last_change":"2024-03-06T07:20:07.285494+0000","last_active":"2024-03-06T07:20:07.287847+0000","last_peered":"2024-03-06T07:20:07.287847+0000","last_clean":"2024-03-06T07:20:07.287847+0000","last_became_active":"2024-03-06T07:20:07.285238+0000","last_became_peered":"2024-03-06T07:20:07.285238+0000","last_unstale":"2024-03-06T07:20:07.287847+0000","last_undegraded":"2024-03-06T07:20:07.287847+0000","last_fullsized":"2024-03-06T07:20:07.287847+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.832403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274742+0000","last_change":"2024-03-06T07:19:54.039974+0000","last_active":"2024-03-06T07:20:07.274742+0000","last_peered":"2024-03-06T07:20:07.274742+0000","last_clean":"2024-03-06T07:20:07.274742+0000","last_became_active":"2024-03-06T07:19:54.039675+0000","last_became_peered":"2024-03-06T07:19:54.039675+0000","last_unstale":"2024-03-06T07:20:07.274742+0000","last_undegraded":"2024-03-06T07:20:07.274742+0000","last_fullsized":"2024-03-06T07:20:07.274742+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:53.089205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279819+0000","last_change":"2024-03-06T07:19:53.051640+0000","last_active":"2024-03-06T07:20:07.279819+0000","last_peered":"2024-03-06T07:20:07.279819+0000","last_clean":"2024-03-06T07:20:07.279819+0000","last_became_active":"2024-03-06T07:19:53.051429+0000","last_became_peered":"2024-03-06T07:19:53.051429+0000","last_unstale":"2024-03-06T07:20:07.279819+0000","last_undegraded":"2024-03-06T07:20:07.279819+0000","last_fullsized":"2024-03-06T07:20:07.279819+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:09.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271758+0000","last_change":"2024-03-06T07:20:07.271758+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271758+0000","last_undegraded":"2024-03-06T07:20:07.271758+0000","last_fullsized":"2024-03-06T07:20:07.271758+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288414+0000","last_change":"2024-03-06T07:20:07.287035+0000","last_active":"2024-03-06T07:20:07.288414+0000","last_peered":"2024-03-06T07:20:07.288414+0000","last_clean":"2024-03-06T07:20:07.288414+0000","last_became_active":"2024-03-06T07:20:07.286453+0000","last_became_peered":"2024-03-06T07:20:07.286453+0000","last_unstale":"2024-03-06T07:20:07.288414+0000","last_undegraded":"2024-03-06T07:20:07.288414+0000","last_fullsized":"2024-03-06T07:20:07.288414+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:15.726763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274553+0000","last_change":"2024-03-06T07:19:54.054949+0000","last_active":"2024-03-06T07:20:07.274553+0000","last_peered":"2024-03-06T07:20:07.274553+0000","last_clean":"2024-03-06T07:20:07.274553+0000","last_became_active":"2024-03-06T07:19:54.054661+0000","last_became_peered":"2024-03-06T07:19:54.054661+0000","last_unstale":"2024-03-06T07:20:07.274553+0000","last_undegraded":"2024-03-06T07:20:07.274553+0000","last_fullsized":"2024-03-06T07:20:07.274553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:57.129016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272386+0000","last_change":"2024-03-06T07:19:53.042003+0000","last_active":"2024-03-06T07:20:07.272386+0000","last_peered":"2024-03-06T07:20:07.272386+0000","last_clean":"2024-03-06T07:20:07.272386+0000","last_became_active":"2024-03-06T07:19:53.041728+0000","last_became_peered":"2024-03-06T07:19:53.041728+0000","last_unstale":"2024-03-06T07:20:07.272386+0000","last_undegraded":"2024-03-06T07:20:07.272386+0000","last_fullsized":"2024-03-06T07:20:07.272386+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.964097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271661+0000","last_change":"2024-03-06T07:20:07.271661+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271661+0000","last_undegraded":"2024-03-06T07:20:07.271661+0000","last_fullsized":"2024-03-06T07:20:07.271661+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284084+0000","last_change":"2024-03-06T07:20:07.278974+0000","last_active":"2024-03-06T07:20:07.284084+0000","last_peered":"2024-03-06T07:20:07.284084+0000","last_clean":"2024-03-06T07:20:07.284084+0000","last_became_active":"2024-03-06T07:20:07.278682+0000","last_became_peered":"2024-03-06T07:20:07.278682+0000","last_unstale":"2024-03-06T07:20:07.284084+0000","last_undegraded":"2024-03-06T07:20:07.284084+0000","last_fullsized":"2024-03-06T07:20:07.284084+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:27.012782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281002+0000","last_change":"2024-03-06T07:19:54.061284+0000","last_active":"2024-03-06T07:20:07.281002+0000","last_peered":"2024-03-06T07:20:07.281002+0000","last_clean":"2024-03-06T07:20:07.281002+0000","last_became_active":"2024-03-06T07:19:54.061017+0000","last_became_peered":"2024-03-06T07:19:54.061017+0000","last_unstale":"2024-03-06T07:20:07.281002+0000","last_undegraded":"2024-03-06T07:20:07.281002+0000","last_fullsized":"2024-03-06T07:20:07.281002+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274253+0000","last_change":"2024-03-06T07:19:53.041819+0000","last_active":"2024-03-06T07:20:07.274253+0000","last_peered":"2024-03-06T07:20:07.274253+0000","last_clean":"2024-03-06T07:20:07.274253+0000","last_became_active":"2024-03-06T07:19:53.041542+0000","last_became_peered":"2024-03-06T07:19:53.041542+0000","last_unstale":"2024-03-06T07:20:07.274253+0000","last_undegraded":"2024-03-06T07:20:07.274253+0000","last_fullsized":"2024-03-06T07:20:07.274253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269496+0000","last_change":"2024-03-06T07:20:07.269496+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269496+0000","last_undegraded":"2024-03-06T07:20:07.269496+0000","last_fullsized":"2024-03-06T07:20:07.269496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.290812+0000","last_change":"2024-03-06T07:20:07.289776+0000","last_active":"2024-03-06T07:20:07.290812+0000","last_peered":"2024-03-06T07:20:07.290812+0000","last_clean":"2024-03-06T07:20:07.290812+0000","last_became_active":"2024-03-06T07:20:07.289535+0000","last_became_peered":"2024-03-06T07:20:07.289535+0000","last_unstale":"2024-03-06T07:20:07.290812+0000","last_undegraded":"2024-03-06T07:20:07.290812+0000","last_fullsized":"2024-03-06T07:20:07.290812+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273691+0000","last_change":"2024-03-06T07:19:54.039491+0000","last_active":"2024-03-06T07:20:07.273691+0000","last_peered":"2024-03-06T07:20:07.273691+0000","last_clean":"2024-03-06T07:20:07.273691+0000","last_became_active":"2024-03-06T07:19:54.039202+0000","last_became_peered":"2024-03-06T07:19:54.039202+0000","last_unstale":"2024-03-06T07:20:07.273691+0000","last_undegraded":"2024-03-06T07:20:07.273691+0000","last_fullsized":"2024-03-06T07:20:07.273691+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:58.292241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275185+0000","last_change":"2024-03-06T07:19:53.043210+0000","last_active":"2024-03-06T07:20:07.275185+0000","last_peered":"2024-03-06T07:20:07.275185+0000","last_clean":"2024-03-06T07:20:07.275185+0000","last_became_active":"2024-03-06T07:19:53.042951+0000","last_became_peered":"2024-03-06T07:19:53.042951+0000","last_unstale":"2024-03-06T07:20:07.275185+0000","last_undegraded":"2024-03-06T07:20:07.275185+0000","last_fullsized":"2024-03-06T07:20:07.275185+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276607+0000","last_change":"2024-03-06T07:20:07.276607+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276607+0000","last_undegraded":"2024-03-06T07:20:07.276607+0000","last_fullsized":"2024-03-06T07:20:07.276607+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283717+0000","last_change":"2024-03-06T07:20:07.278534+0000","last_active":"2024-03-06T07:20:07.283717+0000","last_peered":"2024-03-06T07:20:07.283717+0000","last_clean":"2024-03-06T07:20:07.283717+0000","last_became_active":"2024-03-06T07:20:07.278261+0000","last_became_peered":"2024-03-06T07:20:07.278261+0000","last_unstale":"2024-03-06T07:20:07.283717+0000","last_undegraded":"2024-03-06T07:20:07.283717+0000","last_fullsized":"2024-03-06T07:20:07.283717+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:23.448306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285369+0000","last_change":"2024-03-06T07:19:54.049884+0000","last_active":"2024-03-06T07:20:07.285369+0000","last_peered":"2024-03-06T07:20:07.285369+0000","last_clean":"2024-03-06T07:20:07.285369+0000","last_became_active":"2024-03-06T07:19:54.049597+0000","last_became_peered":"2024-03-06T07:19:54.049597+0000","last_unstale":"2024-03-06T07:20:07.285369+0000","last_undegraded":"2024-03-06T07:20:07.285369+0000","last_fullsized":"2024-03-06T07:20:07.285369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.270618+0000","last_change":"2024-03-06T07:19:53.043198+0000","last_active":"2024-03-06T07:20:07.270618+0000","last_peered":"2024-03-06T07:20:07.270618+0000","last_clean":"2024-03-06T07:20:07.270618+0000","last_became_active":"2024-03-06T07:19:53.042939+0000","last_became_peered":"2024-03-06T07:19:53.042939+0000","last_unstale":"2024-03-06T07:20:07.270618+0000","last_undegraded":"2024-03-06T07:20:07.270618+0000","last_fullsized":"2024-03-06T07:20:07.270618+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273576+0000","last_change":"2024-03-06T07:20:07.273576+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273576+0000","last_undegraded":"2024-03-06T07:20:07.273576+0000","last_fullsized":"2024-03-06T07:20:07.273576+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285627+0000","last_change":"2024-03-06T07:20:07.283576+0000","last_active":"2024-03-06T07:20:07.285627+0000","last_peered":"2024-03-06T07:20:07.285627+0000","last_clean":"2024-03-06T07:20:07.285627+0000","last_became_active":"2024-03-06T07:20:07.283136+0000","last_became_peered":"2024-03-06T07:20:07.283136+0000","last_unstale":"2024-03-06T07:20:07.285627+0000","last_undegraded":"2024-03-06T07:20:07.285627+0000","last_fullsized":"2024-03-06T07:20:07.285627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:56.830929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274311+0000","last_change":"2024-03-06T07:19:54.045809+0000","last_active":"2024-03-06T07:20:07.274311+0000","last_peered":"2024-03-06T07:20:07.274311+0000","last_clean":"2024-03-06T07:20:07.274311+0000","last_became_active":"2024-03-06T07:19:54.040263+0000","last_became_peered":"2024-03-06T07:19:54.040263+0000","last_unstale":"2024-03-06T07:20:07.274311+0000","last_undegraded":"2024-03-06T07:20:07.274311+0000","last_fullsized":"2024-03-06T07:20:07.274311+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:20.728340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285454+0000","last_change":"2024-03-06T07:19:53.047864+0000","last_active":"2024-03-06T07:20:07.285454+0000","last_peered":"2024-03-06T07:20:07.285454+0000","last_clean":"2024-03-06T07:20:07.285454+0000","last_became_active":"2024-03-06T07:19:53.047607+0000","last_became_peered":"2024-03-06T07:19:53.047607+0000","last_unstale":"2024-03-06T07:20:07.285454+0000","last_undegraded":"2024-03-06T07:20:07.285454+0000","last_fullsized":"2024-03-06T07:20:07.285454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.148872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277242+0000","last_change":"2024-03-06T07:20:07.277242+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277242+0000","last_undegraded":"2024-03-06T07:20:07.277242+0000","last_fullsized":"2024-03-06T07:20:07.277242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288691+0000","last_change":"2024-03-06T07:20:07.285280+0000","last_active":"2024-03-06T07:20:07.288691+0000","last_peered":"2024-03-06T07:20:07.288691+0000","last_clean":"2024-03-06T07:20:07.288691+0000","last_became_active":"2024-03-06T07:20:07.284993+0000","last_became_peered":"2024-03-06T07:20:07.284993+0000","last_unstale":"2024-03-06T07:20:07.288691+0000","last_undegraded":"2024-03-06T07:20:07.288691+0000","last_fullsized":"2024-03-06T07:20:07.288691+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:59.458261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279291+0000","last_change":"2024-03-06T07:19:54.056898+0000","last_active":"2024-03-06T07:20:07.279291+0000","last_peered":"2024-03-06T07:20:07.279291+0000","last_clean":"2024-03-06T07:20:07.279291+0000","last_became_active":"2024-03-06T07:19:54.056522+0000","last_became_peered":"2024-03-06T07:19:54.056522+0000","last_unstale":"2024-03-06T07:20:07.279291+0000","last_undegraded":"2024-03-06T07:20:07.279291+0000","last_fullsized":"2024-03-06T07:20:07.279291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274521+0000","last_change":"2024-03-06T07:19:53.043065+0000","last_active":"2024-03-06T07:20:07.274521+0000","last_peered":"2024-03-06T07:20:07.274521+0000","last_clean":"2024-03-06T07:20:07.274521+0000","last_became_active":"2024-03-06T07:19:53.042756+0000","last_became_peered":"2024-03-06T07:19:53.042756+0000","last_unstale":"2024-03-06T07:20:07.274521+0000","last_undegraded":"2024-03-06T07:20:07.274521+0000","last_fullsized":"2024-03-06T07:20:07.274521+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.338932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274188+0000","last_change":"2024-03-06T07:20:07.274188+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274188+0000","last_undegraded":"2024-03-06T07:20:07.274188+0000","last_fullsized":"2024-03-06T07:20:07.274188+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285804+0000","last_change":"2024-03-06T07:20:07.282855+0000","last_active":"2024-03-06T07:20:07.285804+0000","last_peered":"2024-03-06T07:20:07.285804+0000","last_clean":"2024-03-06T07:20:07.285804+0000","last_became_active":"2024-03-06T07:20:07.282585+0000","last_became_peered":"2024-03-06T07:20:07.282585+0000","last_unstale":"2024-03-06T07:20:07.285804+0000","last_undegraded":"2024-03-06T07:20:07.285804+0000","last_fullsized":"2024-03-06T07:20:07.285804+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:52.503066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271456+0000","last_change":"2024-03-06T07:19:54.057647+0000","last_active":"2024-03-06T07:20:07.271456+0000","last_peered":"2024-03-06T07:20:07.271456+0000","last_clean":"2024-03-06T07:20:07.271456+0000","last_became_active":"2024-03-06T07:19:54.057396+0000","last_became_peered":"2024-03-06T07:19:54.057396+0000","last_unstale":"2024-03-06T07:20:07.271456+0000","last_undegraded":"2024-03-06T07:20:07.271456+0000","last_fullsized":"2024-03-06T07:20:07.271456+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":21,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281162+0000","last_change":"2024-03-06T07:19:53.046359+0000","last_active":"2024-03-06T07:20:07.281162+0000","last_peered":"2024-03-06T07:20:07.281162+0000","last_clean":"2024-03-06T07:20:07.281162+0000","last_became_active":"2024-03-06T07:19:53.046065+0000","last_became_peered":"2024-03-06T07:19:53.046065+0000","last_unstale":"2024-03-06T07:20:07.281162+0000","last_undegraded":"2024-03-06T07:20:07.281162+0000","last_fullsized":"2024-03-06T07:20:07.281162+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:08.146267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269510+0000","last_change":"2024-03-06T07:20:07.269510+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269510+0000","last_undegraded":"2024-03-06T07:20:07.269510+0000","last_fullsized":"2024-03-06T07:20:07.269510+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283148+0000","last_change":"2024-03-06T07:20:07.280915+0000","last_active":"2024-03-06T07:20:07.283148+0000","last_peered":"2024-03-06T07:20:07.283148+0000","last_clean":"2024-03-06T07:20:07.283148+0000","last_became_active":"2024-03-06T07:20:07.280659+0000","last_became_peered":"2024-03-06T07:20:07.280659+0000","last_unstale":"2024-03-06T07:20:07.283148+0000","last_undegraded":"2024-03-06T07:20:07.283148+0000","last_fullsized":"2024-03-06T07:20:07.283148+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:59.749257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274291+0000","last_change":"2024-03-06T07:19:54.041140+0000","last_active":"2024-03-06T07:20:07.274291+0000","last_peered":"2024-03-06T07:20:07.274291+0000","last_clean":"2024-03-06T07:20:07.274291+0000","last_became_active":"2024-03-06T07:19:54.040877+0000","last_became_peered":"2024-03-06T07:19:54.040877+0000","last_unstale":"2024-03-06T07:20:07.274291+0000","last_undegraded":"2024-03-06T07:20:07.274291+0000","last_fullsized":"2024-03-06T07:20:07.274291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:29.971359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":24,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273930+0000","last_change":"2024-03-06T07:19:53.045210+0000","last_active":"2024-03-06T07:20:07.273930+0000","last_peered":"2024-03-06T07:20:07.273930+0000","last_clean":"2024-03-06T07:20:07.273930+0000","last_became_active":"2024-03-06T07:19:53.044950+0000","last_became_peered":"2024-03-06T07:19:53.044950+0000","last_unstale":"2024-03-06T07:20:07.273930+0000","last_undegraded":"2024-03-06T07:20:07.273930+0000","last_fullsized":"2024-03-06T07:20:07.273930+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:53.407226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271405+0000","last_change":"2024-03-06T07:20:07.271405+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271405+0000","last_undegraded":"2024-03-06T07:20:07.271405+0000","last_fullsized":"2024-03-06T07:20:07.271405+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286730+0000","last_change":"2024-03-06T07:20:07.285841+0000","last_active":"2024-03-06T07:20:07.286730+0000","last_peered":"2024-03-06T07:20:07.286730+0000","last_clean":"2024-03-06T07:20:07.286730+0000","last_became_active":"2024-03-06T07:20:07.285505+0000","last_became_peered":"2024-03-06T07:20:07.285505+0000","last_unstale":"2024-03-06T07:20:07.286730+0000","last_undegraded":"2024-03-06T07:20:07.286730+0000","last_fullsized":"2024-03-06T07:20:07.286730+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:06.316192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279554+0000","last_change":"2024-03-06T07:19:54.058718+0000","last_active":"2024-03-06T07:20:07.279554+0000","last_peered":"2024-03-06T07:20:07.279554+0000","last_clean":"2024-03-06T07:20:07.279554+0000","last_became_active":"2024-03-06T07:19:54.058415+0000","last_became_peered":"2024-03-06T07:19:54.058415+0000","last_unstale":"2024-03-06T07:20:07.279554+0000","last_undegraded":"2024-03-06T07:20:07.279554+0000","last_fullsized":"2024-03-06T07:20:07.279554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272518+0000","last_change":"2024-03-06T07:19:53.046555+0000","last_active":"2024-03-06T07:20:07.272518+0000","last_peered":"2024-03-06T07:20:07.272518+0000","last_clean":"2024-03-06T07:20:07.272518+0000","last_became_active":"2024-03-06T07:19:53.046298+0000","last_became_peered":"2024-03-06T07:19:53.046298+0000","last_unstale":"2024-03-06T07:20:07.272518+0000","last_undegraded":"2024-03-06T07:20:07.272518+0000","last_fullsized":"2024-03-06T07:20:07.272518+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.243021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.280843+0000","last_change":"2024-03-06T07:20:07.280843+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.280843+0000","last_undegraded":"2024-03-06T07:20:07.280843+0000","last_fullsized":"2024-03-06T07:20:07.280843+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286818+0000","last_change":"2024-03-06T07:20:07.285895+0000","last_active":"2024-03-06T07:20:07.286818+0000","last_peered":"2024-03-06T07:20:07.286818+0000","last_clean":"2024-03-06T07:20:07.286818+0000","last_became_active":"2024-03-06T07:20:07.285614+0000","last_became_peered":"2024-03-06T07:20:07.285614+0000","last_unstale":"2024-03-06T07:20:07.286818+0000","last_undegraded":"2024-03-06T07:20:07.286818+0000","last_fullsized":"2024-03-06T07:20:07.286818+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:00.451319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272257+0000","last_change":"2024-03-06T07:19:54.047654+0000","last_active":"2024-03-06T07:20:07.272257+0000","last_peered":"2024-03-06T07:20:07.272257+0000","last_clean":"2024-03-06T07:20:07.272257+0000","last_became_active":"2024-03-06T07:19:54.047339+0000","last_became_peered":"2024-03-06T07:19:54.047339+0000","last_unstale":"2024-03-06T07:20:07.272257+0000","last_undegraded":"2024-03-06T07:20:07.272257+0000","last_fullsized":"2024-03-06T07:20:07.272257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:35.272829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275306+0000","last_change":"2024-03-06T07:19:53.040132+0000","last_active":"2024-03-06T07:20:07.275306+0000","last_peered":"2024-03-06T07:20:07.275306+0000","last_clean":"2024-03-06T07:20:07.275306+0000","last_became_active":"2024-03-06T07:19:53.039809+0000","last_became_peered":"2024-03-06T07:19:53.039809+0000","last_unstale":"2024-03-06T07:20:07.275306+0000","last_undegraded":"2024-03-06T07:20:07.275306+0000","last_fullsized":"2024-03-06T07:20:07.275306+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272239+0000","last_change":"2024-03-06T07:20:07.272239+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272239+0000","last_undegraded":"2024-03-06T07:20:07.272239+0000","last_fullsized":"2024-03-06T07:20:07.272239+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289344+0000","last_change":"2024-03-06T07:20:07.288210+0000","last_active":"2024-03-06T07:20:07.289344+0000","last_peered":"2024-03-06T07:20:07.289344+0000","last_clean":"2024-03-06T07:20:07.289344+0000","last_became_active":"2024-03-06T07:20:07.287885+0000","last_became_peered":"2024-03-06T07:20:07.287885+0000","last_unstale":"2024-03-06T07:20:07.289344+0000","last_undegraded":"2024-03-06T07:20:07.289344+0000","last_fullsized":"2024-03-06T07:20:07.289344+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:25.080610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273595+0000","last_change":"2024-03-06T07:19:54.050847+0000","last_active":"2024-03-06T07:20:07.273595+0000","last_peered":"2024-03-06T07:20:07.273595+0000","last_clean":"2024-03-06T07:20:07.273595+0000","last_became_active":"2024-03-06T07:19:54.050378+0000","last_became_peered":"2024-03-06T07:19:54.050378+0000","last_unstale":"2024-03-06T07:20:07.273595+0000","last_undegraded":"2024-03-06T07:20:07.273595+0000","last_fullsized":"2024-03-06T07:20:07.273595+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281018+0000","last_change":"2024-03-06T07:19:53.045139+0000","last_active":"2024-03-06T07:20:07.281018+0000","last_peered":"2024-03-06T07:20:07.281018+0000","last_clean":"2024-03-06T07:20:07.281018+0000","last_became_active":"2024-03-06T07:19:53.044903+0000","last_became_peered":"2024-03-06T07:19:53.044903+0000","last_unstale":"2024-03-06T07:20:07.281018+0000","last_undegraded":"2024-03-06T07:20:07.281018+0000","last_fullsized":"2024-03-06T07:20:07.281018+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:56.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272163+0000","last_change":"2024-03-06T07:20:07.272163+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272163+0000","last_undegraded":"2024-03-06T07:20:07.272163+0000","last_fullsized":"2024-03-06T07:20:07.272163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.290186+0000","last_change":"2024-03-06T07:20:07.289518+0000","last_active":"2024-03-06T07:20:07.290186+0000","last_peered":"2024-03-06T07:20:07.290186+0000","last_clean":"2024-03-06T07:20:07.290186+0000","last_became_active":"2024-03-06T07:20:07.289213+0000","last_became_peered":"2024-03-06T07:20:07.289213+0000","last_unstale":"2024-03-06T07:20:07.290186+0000","last_undegraded":"2024-03-06T07:20:07.290186+0000","last_fullsized":"2024-03-06T07:20:07.290186+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.514155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271904+0000","last_change":"2024-03-06T07:19:54.059322+0000","last_active":"2024-03-06T07:20:07.271904+0000","last_peered":"2024-03-06T07:20:07.271904+0000","last_clean":"2024-03-06T07:20:07.271904+0000","last_became_active":"2024-03-06T07:19:54.059092+0000","last_became_peered":"2024-03-06T07:19:54.059092+0000","last_unstale":"2024-03-06T07:20:07.271904+0000","last_undegraded":"2024-03-06T07:20:07.271904+0000","last_fullsized":"2024-03-06T07:20:07.271904+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":26,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275158+0000","last_change":"2024-03-06T07:19:53.046815+0000","last_active":"2024-03-06T07:20:07.275158+0000","last_peered":"2024-03-06T07:20:07.275158+0000","last_clean":"2024-03-06T07:20:07.275158+0000","last_became_active":"2024-03-06T07:19:53.043687+0000","last_became_peered":"2024-03-06T07:19:53.043687+0000","last_unstale":"2024-03-06T07:20:07.275158+0000","last_undegraded":"2024-03-06T07:20:07.275158+0000","last_fullsized":"2024-03-06T07:20:07.275158+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:44.465449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269497+0000","last_change":"2024-03-06T07:20:07.269497+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269497+0000","last_undegraded":"2024-03-06T07:20:07.269497+0000","last_fullsized":"2024-03-06T07:20:07.269497+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285213+0000","last_change":"2024-03-06T07:20:07.283273+0000","last_active":"2024-03-06T07:20:07.285213+0000","last_peered":"2024-03-06T07:20:07.285213+0000","last_clean":"2024-03-06T07:20:07.285213+0000","last_became_active":"2024-03-06T07:20:07.282989+0000","last_became_peered":"2024-03-06T07:20:07.282989+0000","last_unstale":"2024-03-06T07:20:07.285213+0000","last_undegraded":"2024-03-06T07:20:07.285213+0000","last_fullsized":"2024-03-06T07:20:07.285213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:08.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275055+0000","last_change":"2024-03-06T07:19:54.057521+0000","last_active":"2024-03-06T07:20:07.275055+0000","last_peered":"2024-03-06T07:20:07.275055+0000","last_clean":"2024-03-06T07:20:07.275055+0000","last_became_active":"2024-03-06T07:19:54.057264+0000","last_became_peered":"2024-03-06T07:19:54.057264+0000","last_unstale":"2024-03-06T07:20:07.275055+0000","last_undegraded":"2024-03-06T07:20:07.275055+0000","last_fullsized":"2024-03-06T07:20:07.275055+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272172+0000","last_change":"2024-03-06T07:19:53.051189+0000","last_active":"2024-03-06T07:20:07.272172+0000","last_peered":"2024-03-06T07:20:07.272172+0000","last_clean":"2024-03-06T07:20:07.272172+0000","last_became_active":"2024-03-06T07:19:53.050897+0000","last_became_peered":"2024-03-06T07:19:53.050897+0000","last_unstale":"2024-03-06T07:20:07.272172+0000","last_undegraded":"2024-03-06T07:20:07.272172+0000","last_fullsized":"2024-03-06T07:20:07.272172+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274169+0000","last_change":"2024-03-06T07:20:07.274169+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274169+0000","last_undegraded":"2024-03-06T07:20:07.274169+0000","last_fullsized":"2024-03-06T07:20:07.274169+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284678+0000","last_change":"2024-03-06T07:20:07.283920+0000","last_active":"2024-03-06T07:20:07.284678+0000","last_peered":"2024-03-06T07:20:07.284678+0000","last_clean":"2024-03-06T07:20:07.284678+0000","last_became_active":"2024-03-06T07:20:07.283631+0000","last_became_peered":"2024-03-06T07:20:07.283631+0000","last_unstale":"2024-03-06T07:20:07.284678+0000","last_undegraded":"2024-03-06T07:20:07.284678+0000","last_fullsized":"2024-03-06T07:20:07.284678+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:33.711210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266154+0000","last_change":"2024-03-06T07:19:54.052438+0000","last_active":"2024-03-06T07:20:07.266154+0000","last_peered":"2024-03-06T07:20:07.266154+0000","last_clean":"2024-03-06T07:20:07.266154+0000","last_became_active":"2024-03-06T07:19:54.052185+0000","last_became_peered":"2024-03-06T07:19:54.052185+0000","last_unstale":"2024-03-06T07:20:07.266154+0000","last_undegraded":"2024-03-06T07:20:07.266154+0000","last_fullsized":"2024-03-06T07:20:07.266154+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276246+0000","last_change":"2024-03-06T07:19:53.046903+0000","last_active":"2024-03-06T07:20:07.276246+0000","last_peered":"2024-03-06T07:20:07.276246+0000","last_clean":"2024-03-06T07:20:07.276246+0000","last_became_active":"2024-03-06T07:19:53.046639+0000","last_became_peered":"2024-03-06T07:19:53.046639+0000","last_unstale":"2024-03-06T07:20:07.276246+0000","last_undegraded":"2024-03-06T07:20:07.276246+0000","last_fullsized":"2024-03-06T07:20:07.276246+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:23.209022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273354+0000","last_change":"2024-03-06T07:20:07.273354+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273354+0000","last_undegraded":"2024-03-06T07:20:07.273354+0000","last_fullsized":"2024-03-06T07:20:07.273354+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289363+0000","last_change":"2024-03-06T07:20:07.288781+0000","last_active":"2024-03-06T07:20:07.289363+0000","last_peered":"2024-03-06T07:20:07.289363+0000","last_clean":"2024-03-06T07:20:07.289363+0000","last_became_active":"2024-03-06T07:20:07.288548+0000","last_became_peered":"2024-03-06T07:20:07.288548+0000","last_unstale":"2024-03-06T07:20:07.289363+0000","last_undegraded":"2024-03-06T07:20:07.289363+0000","last_fullsized":"2024-03-06T07:20:07.289363+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274167+0000","last_change":"2024-03-06T07:19:54.053795+0000","last_active":"2024-03-06T07:20:07.274167+0000","last_peered":"2024-03-06T07:20:07.274167+0000","last_clean":"2024-03-06T07:20:07.274167+0000","last_became_active":"2024-03-06T07:19:54.053519+0000","last_became_peered":"2024-03-06T07:19:54.053519+0000","last_unstale":"2024-03-06T07:20:07.274167+0000","last_undegraded":"2024-03-06T07:20:07.274167+0000","last_fullsized":"2024-03-06T07:20:07.274167+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:20.250283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275485+0000","last_change":"2024-03-06T07:19:53.046151+0000","last_active":"2024-03-06T07:20:07.275485+0000","last_peered":"2024-03-06T07:20:07.275485+0000","last_clean":"2024-03-06T07:20:07.275485+0000","last_became_active":"2024-03-06T07:19:53.045797+0000","last_became_peered":"2024-03-06T07:19:53.045797+0000","last_unstale":"2024-03-06T07:20:07.275485+0000","last_undegraded":"2024-03-06T07:20:07.275485+0000","last_fullsized":"2024-03-06T07:20:07.275485+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271460+0000","last_change":"2024-03-06T07:20:07.271460+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271460+0000","last_undegraded":"2024-03-06T07:20:07.271460+0000","last_fullsized":"2024-03-06T07:20:07.271460+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285658+0000","last_change":"2024-03-06T07:20:07.284617+0000","last_active":"2024-03-06T07:20:07.285658+0000","last_peered":"2024-03-06T07:20:07.285658+0000","last_clean":"2024-03-06T07:20:07.285658+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:07.285658+0000","last_undegraded":"2024-03-06T07:20:07.285658+0000","last_fullsized":"2024-03-06T07:20:07.285658+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:29.205056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266558+0000","last_change":"2024-03-06T07:19:54.055324+0000","last_active":"2024-03-06T07:20:07.266558+0000","last_peered":"2024-03-06T07:20:07.266558+0000","last_clean":"2024-03-06T07:20:07.266558+0000","last_became_active":"2024-03-06T07:19:54.055081+0000","last_became_peered":"2024-03-06T07:19:54.055081+0000","last_unstale":"2024-03-06T07:20:07.266558+0000","last_undegraded":"2024-03-06T07:20:07.266558+0000","last_fullsized":"2024-03-06T07:20:07.266558+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:57.518725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275654+0000","last_change":"2024-03-06T07:19:53.048898+0000","last_active":"2024-03-06T07:20:07.275654+0000","last_peered":"2024-03-06T07:20:07.275654+0000","last_clean":"2024-03-06T07:20:07.275654+0000","last_became_active":"2024-03-06T07:19:53.048613+0000","last_became_peered":"2024-03-06T07:19:53.048613+0000","last_unstale":"2024-03-06T07:20:07.275654+0000","last_undegraded":"2024-03-06T07:20:07.275654+0000","last_fullsized":"2024-03-06T07:20:07.275654+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277278+0000","last_change":"2024-03-06T07:20:07.277278+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277278+0000","last_undegraded":"2024-03-06T07:20:07.277278+0000","last_fullsized":"2024-03-06T07:20:07.277278+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286112+0000","last_change":"2024-03-06T07:20:07.285007+0000","last_active":"2024-03-06T07:20:07.286112+0000","last_peered":"2024-03-06T07:20:07.286112+0000","last_clean":"2024-03-06T07:20:07.286112+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:07.286112+0000","last_undegraded":"2024-03-06T07:20:07.286112+0000","last_fullsized":"2024-03-06T07:20:07.286112+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279669+0000","last_change":"2024-03-06T07:19:54.056310+0000","last_active":"2024-03-06T07:20:07.279669+0000","last_peered":"2024-03-06T07:20:07.279669+0000","last_clean":"2024-03-06T07:20:07.279669+0000","last_became_active":"2024-03-06T07:19:54.056039+0000","last_became_peered":"2024-03-06T07:19:54.056039+0000","last_unstale":"2024-03-06T07:20:07.279669+0000","last_undegraded":"2024-03-06T07:20:07.279669+0000","last_fullsized":"2024-03-06T07:20:07.279669+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266526+0000","last_change":"2024-03-06T07:19:53.043949+0000","last_active":"2024-03-06T07:20:07.266526+0000","last_peered":"2024-03-06T07:20:07.266526+0000","last_clean":"2024-03-06T07:20:07.266526+0000","last_became_active":"2024-03-06T07:19:53.043639+0000","last_became_peered":"2024-03-06T07:19:53.043639+0000","last_unstale":"2024-03-06T07:20:07.266526+0000","last_undegraded":"2024-03-06T07:20:07.266526+0000","last_fullsized":"2024-03-06T07:20:07.266526+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271451+0000","last_change":"2024-03-06T07:20:07.271451+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271451+0000","last_undegraded":"2024-03-06T07:20:07.271451+0000","last_fullsized":"2024-03-06T07:20:07.271451+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284102+0000","last_change":"2024-03-06T07:20:07.279146+0000","last_active":"2024-03-06T07:20:07.284102+0000","last_peered":"2024-03-06T07:20:07.284102+0000","last_clean":"2024-03-06T07:20:07.284102+0000","last_became_active":"2024-03-06T07:20:07.278872+0000","last_became_peered":"2024-03-06T07:20:07.278872+0000","last_unstale":"2024-03-06T07:20:07.284102+0000","last_undegraded":"2024-03-06T07:20:07.284102+0000","last_fullsized":"2024-03-06T07:20:07.284102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:34.974180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275593+0000","last_change":"2024-03-06T07:19:54.040887+0000","last_active":"2024-03-06T07:20:07.275593+0000","last_peered":"2024-03-06T07:20:07.275593+0000","last_clean":"2024-03-06T07:20:07.275593+0000","last_became_active":"2024-03-06T07:19:54.040632+0000","last_became_peered":"2024-03-06T07:19:54.040632+0000","last_unstale":"2024-03-06T07:20:07.275593+0000","last_undegraded":"2024-03-06T07:20:07.275593+0000","last_fullsized":"2024-03-06T07:20:07.275593+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276010+0000","last_change":"2024-03-06T07:19:53.044330+0000","last_active":"2024-03-06T07:20:07.276010+0000","last_peered":"2024-03-06T07:20:07.276010+0000","last_clean":"2024-03-06T07:20:07.276010+0000","last_became_active":"2024-03-06T07:19:53.044066+0000","last_became_peered":"2024-03-06T07:19:53.044066+0000","last_unstale":"2024-03-06T07:20:07.276010+0000","last_undegraded":"2024-03-06T07:20:07.276010+0000","last_fullsized":"2024-03-06T07:20:07.276010+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:19.476961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271863+0000","last_change":"2024-03-06T07:20:07.271863+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271863+0000","last_undegraded":"2024-03-06T07:20:07.271863+0000","last_fullsized":"2024-03-06T07:20:07.271863+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289018+0000","last_change":"2024-03-06T07:20:07.287646+0000","last_active":"2024-03-06T07:20:07.289018+0000","last_peered":"2024-03-06T07:20:07.289018+0000","last_clean":"2024-03-06T07:20:07.289018+0000","last_became_active":"2024-03-06T07:20:07.287378+0000","last_became_peered":"2024-03-06T07:20:07.287378+0000","last_unstale":"2024-03-06T07:20:07.289018+0000","last_undegraded":"2024-03-06T07:20:07.289018+0000","last_fullsized":"2024-03-06T07:20:07.289018+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272180+0000","last_change":"2024-03-06T07:19:54.049940+0000","last_active":"2024-03-06T07:20:07.272180+0000","last_peered":"2024-03-06T07:20:07.272180+0000","last_clean":"2024-03-06T07:20:07.272180+0000","last_became_active":"2024-03-06T07:19:54.049481+0000","last_became_peered":"2024-03-06T07:19:54.049481+0000","last_unstale":"2024-03-06T07:20:07.272180+0000","last_undegraded":"2024-03-06T07:20:07.272180+0000","last_fullsized":"2024-03-06T07:20:07.272180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.270036+0000","last_change":"2024-03-06T07:19:53.048472+0000","last_active":"2024-03-06T07:20:07.270036+0000","last_peered":"2024-03-06T07:20:07.270036+0000","last_clean":"2024-03-06T07:20:07.270036+0000","last_became_active":"2024-03-06T07:19:53.048213+0000","last_became_peered":"2024-03-06T07:19:53.048213+0000","last_unstale":"2024-03-06T07:20:07.270036+0000","last_undegraded":"2024-03-06T07:20:07.270036+0000","last_fullsized":"2024-03-06T07:20:07.270036+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:19.721078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274293+0000","last_change":"2024-03-06T07:20:07.274293+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274293+0000","last_undegraded":"2024-03-06T07:20:07.274293+0000","last_fullsized":"2024-03-06T07:20:07.274293+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286969+0000","last_change":"2024-03-06T07:20:07.283238+0000","last_active":"2024-03-06T07:20:07.286969+0000","last_peered":"2024-03-06T07:20:07.286969+0000","last_clean":"2024-03-06T07:20:07.286969+0000","last_became_active":"2024-03-06T07:20:07.282996+0000","last_became_peered":"2024-03-06T07:20:07.282996+0000","last_unstale":"2024-03-06T07:20:07.286969+0000","last_undegraded":"2024-03-06T07:20:07.286969+0000","last_fullsized":"2024-03-06T07:20:07.286969+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.712830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.267007+0000","last_change":"2024-03-06T07:19:54.042432+0000","last_active":"2024-03-06T07:20:07.267007+0000","last_peered":"2024-03-06T07:20:07.267007+0000","last_clean":"2024-03-06T07:20:07.267007+0000","last_became_active":"2024-03-06T07:19:54.042166+0000","last_became_peered":"2024-03-06T07:19:54.042166+0000","last_unstale":"2024-03-06T07:20:07.267007+0000","last_undegraded":"2024-03-06T07:20:07.267007+0000","last_fullsized":"2024-03-06T07:20:07.267007+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.231274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275074+0000","last_change":"2024-03-06T07:19:53.044950+0000","last_active":"2024-03-06T07:20:07.275074+0000","last_peered":"2024-03-06T07:20:07.275074+0000","last_clean":"2024-03-06T07:20:07.275074+0000","last_became_active":"2024-03-06T07:19:53.044630+0000","last_became_peered":"2024-03-06T07:19:53.044630+0000","last_unstale":"2024-03-06T07:20:07.275074+0000","last_undegraded":"2024-03-06T07:20:07.275074+0000","last_fullsized":"2024-03-06T07:20:07.275074+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:05.924599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272790+0000","last_change":"2024-03-06T07:20:07.272790+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272790+0000","last_undegraded":"2024-03-06T07:20:07.272790+0000","last_fullsized":"2024-03-06T07:20:07.272790+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287689+0000","last_change":"2024-03-06T07:20:07.287051+0000","last_active":"2024-03-06T07:20:07.287689+0000","last_peered":"2024-03-06T07:20:07.287689+0000","last_clean":"2024-03-06T07:20:07.287689+0000","last_became_active":"2024-03-06T07:20:07.286569+0000","last_became_peered":"2024-03-06T07:20:07.286569+0000","last_unstale":"2024-03-06T07:20:07.287689+0000","last_undegraded":"2024-03-06T07:20:07.287689+0000","last_fullsized":"2024-03-06T07:20:07.287689+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:17.755219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279576+0000","last_change":"2024-03-06T07:19:54.059591+0000","last_active":"2024-03-06T07:20:07.279576+0000","last_peered":"2024-03-06T07:20:07.279576+0000","last_clean":"2024-03-06T07:20:07.279576+0000","last_became_active":"2024-03-06T07:19:54.058814+0000","last_became_peered":"2024-03-06T07:19:54.058814+0000","last_unstale":"2024-03-06T07:20:07.279576+0000","last_undegraded":"2024-03-06T07:20:07.279576+0000","last_fullsized":"2024-03-06T07:20:07.279576+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275779+0000","last_change":"2024-03-06T07:19:53.040833+0000","last_active":"2024-03-06T07:20:07.275779+0000","last_peered":"2024-03-06T07:20:07.275779+0000","last_clean":"2024-03-06T07:20:07.275779+0000","last_became_active":"2024-03-06T07:19:53.040561+0000","last_became_peered":"2024-03-06T07:19:53.040561+0000","last_unstale":"2024-03-06T07:20:07.275779+0000","last_undegraded":"2024-03-06T07:20:07.275779+0000","last_fullsized":"2024-03-06T07:20:07.275779+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:12.315549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269859+0000","last_change":"2024-03-06T07:20:07.269859+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269859+0000","last_undegraded":"2024-03-06T07:20:07.269859+0000","last_fullsized":"2024-03-06T07:20:07.269859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281857+0000","last_change":"2024-03-06T07:20:07.280861+0000","last_active":"2024-03-06T07:20:07.281857+0000","last_peered":"2024-03-06T07:20:07.281857+0000","last_clean":"2024-03-06T07:20:07.281857+0000","last_became_active":"2024-03-06T07:20:07.280598+0000","last_became_peered":"2024-03-06T07:20:07.280598+0000","last_unstale":"2024-03-06T07:20:07.281857+0000","last_undegraded":"2024-03-06T07:20:07.281857+0000","last_fullsized":"2024-03-06T07:20:07.281857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266454+0000","last_change":"2024-03-06T07:19:54.042216+0000","last_active":"2024-03-06T07:20:07.266454+0000","last_peered":"2024-03-06T07:20:07.266454+0000","last_clean":"2024-03-06T07:20:07.266454+0000","last_became_active":"2024-03-06T07:19:54.041923+0000","last_became_peered":"2024-03-06T07:19:54.041923+0000","last_unstale":"2024-03-06T07:20:07.266454+0000","last_undegraded":"2024-03-06T07:20:07.266454+0000","last_fullsized":"2024-03-06T07:20:07.266454+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281579+0000","last_change":"2024-03-06T07:19:53.047959+0000","last_active":"2024-03-06T07:20:07.281579+0000","last_peered":"2024-03-06T07:20:07.281579+0000","last_clean":"2024-03-06T07:20:07.281579+0000","last_became_active":"2024-03-06T07:19:53.047680+0000","last_became_peered":"2024-03-06T07:19:53.047680+0000","last_unstale":"2024-03-06T07:20:07.281579+0000","last_undegraded":"2024-03-06T07:20:07.281579+0000","last_fullsized":"2024-03-06T07:20:07.281579+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:38.499903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276715+0000","last_change":"2024-03-06T07:20:07.276715+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276715+0000","last_undegraded":"2024-03-06T07:20:07.276715+0000","last_fullsized":"2024-03-06T07:20:07.276715+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287826+0000","last_change":"2024-03-06T07:20:07.284739+0000","last_active":"2024-03-06T07:20:07.287826+0000","last_peered":"2024-03-06T07:20:07.287826+0000","last_clean":"2024-03-06T07:20:07.287826+0000","last_became_active":"2024-03-06T07:20:07.284477+0000","last_became_peered":"2024-03-06T07:20:07.284477+0000","last_unstale":"2024-03-06T07:20:07.287826+0000","last_undegraded":"2024-03-06T07:20:07.287826+0000","last_fullsized":"2024-03-06T07:20:07.287826+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:47.605321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274844+0000","last_change":"2024-03-06T07:19:54.043516+0000","last_active":"2024-03-06T07:20:07.274844+0000","last_peered":"2024-03-06T07:20:07.274844+0000","last_clean":"2024-03-06T07:20:07.274844+0000","last_became_active":"2024-03-06T07:19:54.041651+0000","last_became_peered":"2024-03-06T07:19:54.041651+0000","last_unstale":"2024-03-06T07:20:07.274844+0000","last_undegraded":"2024-03-06T07:20:07.274844+0000","last_fullsized":"2024-03-06T07:20:07.274844+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:04.528232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272322+0000","last_change":"2024-03-06T07:19:53.048966+0000","last_active":"2024-03-06T07:20:07.272322+0000","last_peered":"2024-03-06T07:20:07.272322+0000","last_clean":"2024-03-06T07:20:07.272322+0000","last_became_active":"2024-03-06T07:19:53.047988+0000","last_became_peered":"2024-03-06T07:19:53.047988+0000","last_unstale":"2024-03-06T07:20:07.272322+0000","last_undegraded":"2024-03-06T07:20:07.272322+0000","last_fullsized":"2024-03-06T07:20:07.272322+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:49.519186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272529+0000","last_change":"2024-03-06T07:20:07.272529+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272529+0000","last_undegraded":"2024-03-06T07:20:07.272529+0000","last_fullsized":"2024-03-06T07:20:07.272529+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289051+0000","last_change":"2024-03-06T07:20:07.288004+0000","last_active":"2024-03-06T07:20:07.289051+0000","last_peered":"2024-03-06T07:20:07.289051+0000","last_clean":"2024-03-06T07:20:07.289051+0000","last_became_active":"2024-03-06T07:20:07.287745+0000","last_became_peered":"2024-03-06T07:20:07.287745+0000","last_unstale":"2024-03-06T07:20:07.289051+0000","last_undegraded":"2024-03-06T07:20:07.289051+0000","last_fullsized":"2024-03-06T07:20:07.289051+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:47.605321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272236+0000","last_change":"2024-03-06T07:19:54.062368+0000","last_active":"2024-03-06T07:20:07.272236+0000","last_peered":"2024-03-06T07:20:07.272236+0000","last_clean":"2024-03-06T07:20:07.272236+0000","last_became_active":"2024-03-06T07:19:54.060740+0000","last_became_peered":"2024-03-06T07:19:54.060740+0000","last_unstale":"2024-03-06T07:20:07.272236+0000","last_undegraded":"2024-03-06T07:20:07.272236+0000","last_fullsized":"2024-03-06T07:20:07.272236+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274942+0000","last_change":"2024-03-06T07:19:53.044944+0000","last_active":"2024-03-06T07:20:07.274942+0000","last_peered":"2024-03-06T07:20:07.274942+0000","last_clean":"2024-03-06T07:20:07.274942+0000","last_became_active":"2024-03-06T07:19:53.038900+0000","last_became_peered":"2024-03-06T07:19:53.038900+0000","last_unstale":"2024-03-06T07:20:07.274942+0000","last_undegraded":"2024-03-06T07:20:07.274942+0000","last_fullsized":"2024-03-06T07:20:07.274942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:59.062789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271846+0000","last_change":"2024-03-06T07:20:07.271846+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271846+0000","last_undegraded":"2024-03-06T07:20:07.271846+0000","last_fullsized":"2024-03-06T07:20:07.271846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284471+0000","last_change":"2024-03-06T07:20:07.281920+0000","last_active":"2024-03-06T07:20:07.284471+0000","last_peered":"2024-03-06T07:20:07.284471+0000","last_clean":"2024-03-06T07:20:07.284471+0000","last_became_active":"2024-03-06T07:20:07.281654+0000","last_became_peered":"2024-03-06T07:20:07.281654+0000","last_unstale":"2024-03-06T07:20:07.284471+0000","last_undegraded":"2024-03-06T07:20:07.284471+0000","last_fullsized":"2024-03-06T07:20:07.284471+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:44.082119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279797+0000","last_change":"2024-03-06T07:19:54.059347+0000","last_active":"2024-03-06T07:20:07.279797+0000","last_peered":"2024-03-06T07:20:07.279797+0000","last_clean":"2024-03-06T07:20:07.279797+0000","last_became_active":"2024-03-06T07:19:54.059083+0000","last_became_peered":"2024-03-06T07:19:54.059083+0000","last_unstale":"2024-03-06T07:20:07.279797+0000","last_undegraded":"2024-03-06T07:20:07.279797+0000","last_fullsized":"2024-03-06T07:20:07.279797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276107+0000","last_change":"2024-03-06T07:19:53.048645+0000","last_active":"2024-03-06T07:20:07.276107+0000","last_peered":"2024-03-06T07:20:07.276107+0000","last_clean":"2024-03-06T07:20:07.276107+0000","last_became_active":"2024-03-06T07:19:53.048406+0000","last_became_peered":"2024-03-06T07:19:53.048406+0000","last_unstale":"2024-03-06T07:20:07.276107+0000","last_undegraded":"2024-03-06T07:20:07.276107+0000","last_fullsized":"2024-03-06T07:20:07.276107+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:10.846361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278072+0000","last_change":"2024-03-06T07:20:07.278072+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278072+0000","last_undegraded":"2024-03-06T07:20:07.278072+0000","last_fullsized":"2024-03-06T07:20:07.278072+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288382+0000","last_change":"2024-03-06T07:20:07.287091+0000","last_active":"2024-03-06T07:20:07.288382+0000","last_peered":"2024-03-06T07:20:07.288382+0000","last_clean":"2024-03-06T07:20:07.288382+0000","last_became_active":"2024-03-06T07:20:07.286303+0000","last_became_peered":"2024-03-06T07:20:07.286303+0000","last_unstale":"2024-03-06T07:20:07.288382+0000","last_undegraded":"2024-03-06T07:20:07.288382+0000","last_fullsized":"2024-03-06T07:20:07.288382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271636+0000","last_change":"2024-03-06T07:19:54.049719+0000","last_active":"2024-03-06T07:20:07.271636+0000","last_peered":"2024-03-06T07:20:07.271636+0000","last_clean":"2024-03-06T07:20:07.271636+0000","last_became_active":"2024-03-06T07:19:54.049222+0000","last_became_peered":"2024-03-06T07:19:54.049222+0000","last_unstale":"2024-03-06T07:20:07.271636+0000","last_undegraded":"2024-03-06T07:20:07.271636+0000","last_fullsized":"2024-03-06T07:20:07.271636+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274213+0000","last_change":"2024-03-06T07:19:53.045046+0000","last_active":"2024-03-06T07:20:07.274213+0000","last_peered":"2024-03-06T07:20:07.274213+0000","last_clean":"2024-03-06T07:20:07.274213+0000","last_became_active":"2024-03-06T07:19:53.044751+0000","last_became_peered":"2024-03-06T07:19:53.044751+0000","last_unstale":"2024-03-06T07:20:07.274213+0000","last_undegraded":"2024-03-06T07:20:07.274213+0000","last_fullsized":"2024-03-06T07:20:07.274213+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272523+0000","last_change":"2024-03-06T07:20:07.272523+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272523+0000","last_undegraded":"2024-03-06T07:20:07.272523+0000","last_fullsized":"2024-03-06T07:20:07.272523+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289046+0000","last_change":"2024-03-06T07:20:07.287811+0000","last_active":"2024-03-06T07:20:07.289046+0000","last_peered":"2024-03-06T07:20:07.289046+0000","last_clean":"2024-03-06T07:20:07.289046+0000","last_became_active":"2024-03-06T07:20:07.287540+0000","last_became_peered":"2024-03-06T07:20:07.287540+0000","last_unstale":"2024-03-06T07:20:07.289046+0000","last_undegraded":"2024-03-06T07:20:07.289046+0000","last_fullsized":"2024-03-06T07:20:07.289046+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.712830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274628+0000","last_change":"2024-03-06T07:19:54.039796+0000","last_active":"2024-03-06T07:20:07.274628+0000","last_peered":"2024-03-06T07:20:07.274628+0000","last_clean":"2024-03-06T07:20:07.274628+0000","last_became_active":"2024-03-06T07:19:54.039527+0000","last_became_peered":"2024-03-06T07:19:54.039527+0000","last_unstale":"2024-03-06T07:20:07.274628+0000","last_undegraded":"2024-03-06T07:20:07.274628+0000","last_fullsized":"2024-03-06T07:20:07.274628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:49.114747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279603+0000","last_change":"2024-03-06T07:19:53.043599+0000","last_active":"2024-03-06T07:20:07.279603+0000","last_peered":"2024-03-06T07:20:07.279603+0000","last_clean":"2024-03-06T07:20:07.279603+0000","last_became_active":"2024-03-06T07:19:53.043294+0000","last_became_peered":"2024-03-06T07:19:53.043294+0000","last_unstale":"2024-03-06T07:20:07.279603+0000","last_undegraded":"2024-03-06T07:20:07.279603+0000","last_fullsized":"2024-03-06T07:20:07.279603+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:56.511464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271196+0000","last_change":"2024-03-06T07:20:07.271196+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271196+0000","last_undegraded":"2024-03-06T07:20:07.271196+0000","last_fullsized":"2024-03-06T07:20:07.271196+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278877+0000","last_change":"2024-03-06T07:20:07.277710+0000","last_active":"2024-03-06T07:20:07.278877+0000","last_peered":"2024-03-06T07:20:07.278877+0000","last_clean":"2024-03-06T07:20:07.278877+0000","last_became_active":"2024-03-06T07:20:07.277346+0000","last_became_peered":"2024-03-06T07:20:07.277346+0000","last_unstale":"2024-03-06T07:20:07.278877+0000","last_undegraded":"2024-03-06T07:20:07.278877+0000","last_fullsized":"2024-03-06T07:20:07.278877+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:51.111934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275167+0000","last_change":"2024-03-06T07:19:54.055814+0000","last_active":"2024-03-06T07:20:07.275167+0000","last_peered":"2024-03-06T07:20:07.275167+0000","last_clean":"2024-03-06T07:20:07.275167+0000","last_became_active":"2024-03-06T07:19:54.055480+0000","last_became_peered":"2024-03-06T07:19:54.055480+0000","last_unstale":"2024-03-06T07:20:07.275167+0000","last_undegraded":"2024-03-06T07:20:07.275167+0000","last_fullsized":"2024-03-06T07:20:07.275167+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:35.156134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271765+0000","last_change":"2024-03-06T07:19:53.046120+0000","last_active":"2024-03-06T07:20:07.271765+0000","last_peered":"2024-03-06T07:20:07.271765+0000","last_clean":"2024-03-06T07:20:07.271765+0000","last_became_active":"2024-03-06T07:19:53.045816+0000","last_became_peered":"2024-03-06T07:19:53.045816+0000","last_unstale":"2024-03-06T07:20:07.271765+0000","last_undegraded":"2024-03-06T07:20:07.271765+0000","last_fullsized":"2024-03-06T07:20:07.271765+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:56.226679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":234,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278863+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:20:07.278863+0000","last_peered":"2024-03-06T07:20:07.278863+0000","last_clean":"2024-03-06T07:20:07.278863+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:20:07.278863+0000","last_undegraded":"2024-03-06T07:20:07.278863+0000","last_fullsized":"2024-03-06T07:20:07.278863+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":873646,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":283806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":283154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":283736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":284788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":643072,"data_stored":284766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":283314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":873578,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:11.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:11.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.808+0000 7f8876047640 1 Processor -- start 2024-03-06T07:20:11.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.808+0000 7f8876047640 1 -- start start 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.809+0000 7f8876047640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f88700c7180 0x7f8870159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.809+0000 7f8876047640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88700c7c60 con 0x7f88701527b0 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.809+0000 7f8876047640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88700c8060 con 0x7f88700c88b0 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.809+0000 7f8876047640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8870159640 con 0x7f88700c7180 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f88701527b0 0x7f8870155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38596/0 (socket says 172.21.15.94:38596) 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886f7fe640 1 -- 172.21.15.94:0/712997071 learned_addr learned my addr 172.21.15.94:0/712997071 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886effd640 1 --2- 172.21.15.94:0/712997071 >> v2:172.21.15.94:3300/0 conn(0x7f88700c7180 0x7f8870159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886e7fc640 1 -- 172.21.15.94:0/712997071 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347026531 0 0) 0x7f8860001110 con 0x7f88701527b0 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886e7fc640 1 -- 172.21.15.94:0/712997071 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8858003520 con 0x7f88701527b0 2024-03-06T07:20:11.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886e7fc640 1 -- 172.21.15.94:0/712997071 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33946894 0 0) 0x7f88700c8060 con 0x7f88700c88b0 2024-03-06T07:20:11.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886effd640 1 -- 172.21.15.94:0/712997071 >> v1:172.21.15.145:6789/0 conn(0x7f88700c88b0 legacy=0x7f88700c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886effd640 1 -- 172.21.15.94:0/712997071 >> v1:172.21.15.94:6789/0 conn(0x7f88701527b0 legacy=0x7f8870155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.810+0000 7f886effd640 1 -- 172.21.15.94:0/712997071 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700c76a0 con 0x7f88700c7180 2024-03-06T07:20:11.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f886effd640 1 --2- 172.21.15.94:0/712997071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88700c7180 0x7f8870159100 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f88640097a0 tx=0x7f886402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09908c2fbbb65c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f886e7fc640 1 -- 172.21.15.94:0/712997071 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864035020 con 0x7f88700c7180 2024-03-06T07:20:11.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f886e7fc640 1 -- 172.21.15.94:0/712997071 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8864031830 con 0x7f88700c7180 2024-03-06T07:20:11.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f886e7fc640 1 -- 172.21.15.94:0/712997071 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864031be0 con 0x7f88700c7180 2024-03-06T07:20:11.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f8876047640 1 -- 172.21.15.94:0/712997071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88700c7180 msgr2=0x7f8870159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f8876047640 1 --2- 172.21.15.94:0/712997071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88700c7180 0x7f8870159100 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f88640097a0 tx=0x7f886402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f8876047640 1 -- 172.21.15.94:0/712997071 shutdown_connections 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f8876047640 1 --2- 172.21.15.94:0/712997071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88700c7180 0x7f8870159100 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f8876047640 1 -- 172.21.15.94:0/712997071 >> 172.21.15.94:0/712997071 conn(0x7f88700ac560 msgr2=0x7f88700ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f8876047640 1 -- 172.21.15.94:0/712997071 shutdown_connections 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.811+0000 7f8876047640 1 -- 172.21.15.94:0/712997071 wait complete. 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.812+0000 7f8876047640 1 Processor -- start 2024-03-06T07:20:11.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f8876047640 1 -- start start 2024-03-06T07:20:11.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f8876047640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88700c7180 0x7f887014c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f8876047640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88700c88b0 0x7f88701ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f8876047640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88701527b0 0x7f887014d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f8876047640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8870155600 con 0x7f88701527b0 2024-03-06T07:20:11.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f8876047640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8870155320 con 0x7f88700c88b0 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f8876047640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8870152b90 con 0x7f88700c7180 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f886ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88701527b0 0x7f887014d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f886ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88701527b0 0x7f887014d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37682/0 (socket says 172.21.15.94:37682) 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f886ffff640 1 -- 172.21.15.94:0/2439116423 learned_addr learned my addr 172.21.15.94:0/2439116423 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f886f7fe640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88700c7180 0x7f887014c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.813+0000 7f886effd640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88700c88b0 0x7f88701ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f886ffff640 1 -- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88700c7180 msgr2=0x7f887014c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f886ffff640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88700c7180 0x7f887014c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f886ffff640 1 -- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88700c88b0 msgr2=0x7f88701ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f886ffff640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88700c88b0 0x7f88701ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f886ffff640 1 -- 172.21.15.94:0/2439116423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88701ef020 con 0x7f88701527b0 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f886ffff640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88701527b0 0x7f887014d740 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f886800c790 tx=0x7f886800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f8857fff640 1 -- 172.21.15.94:0/2439116423 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868003da0 con 0x7f88701527b0 2024-03-06T07:20:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f8857fff640 1 -- 172.21.15.94:0/2439116423 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8868005070 con 0x7f88701527b0 2024-03-06T07:20:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887014f640 con 0x7f88701527b0 2024-03-06T07:20:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f8857fff640 1 -- 172.21.15.94:0/2439116423 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868005460 con 0x7f88701527b0 2024-03-06T07:20:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.814+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88700b07d0 con 0x7f88701527b0 2024-03-06T07:20:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.815+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870069490 con 0x7f88701527b0 2024-03-06T07:20:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.815+0000 7f8857fff640 1 -- 172.21.15.94:0/2439116423 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8868005600 con 0x7f88701527b0 2024-03-06T07:20:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.816+0000 7f8857fff640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.816+0000 7f886f7fe640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.816+0000 7f8857fff640 1 -- 172.21.15.94:0/2439116423 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f886808a430 con 0x7f88701527b0 2024-03-06T07:20:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.817+0000 7f886f7fe640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f885c06b4d0 0x7f885c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8860005880 tx=0x7f8860009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:11.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:11.820+0000 7f8857fff640 1 -- 172.21.15.94:0/2439116423 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8870069490 con 0x7f88701527b0 2024-03-06T07:20:12.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.073+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f887004f330 con 0x7f885c06b4d0 2024-03-06T07:20:12.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.085+0000 7f8857fff640 1 -- 172.21.15.94:0/2439116423 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485250 (secure 0 0 0) 0x7f887004f330 con 0x7f885c06b4d0 2024-03-06T07:20:12.092 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:12.095 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:12.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.089+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f885c06b4d0 msgr2=0x7f885c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.089+0000 7f8876047640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f885c06b4d0 0x7f885c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8860005880 tx=0x7f8860009040 comp rx=0 tx=0).stop 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.089+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88701527b0 msgr2=0x7f887014d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.089+0000 7f8876047640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88701527b0 0x7f887014d740 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f886800c790 tx=0x7f886800cc50 comp rx=0 tx=0).stop 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 shutdown_connections 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88701527b0 0x7f887014d740 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88700c88b0 0x7f88701ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 --2- 172.21.15.94:0/2439116423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88700c7180 0x7f887014c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 >> 172.21.15.94:0/2439116423 conn(0x7f88700ac560 msgr2=0x7f88700c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 shutdown_connections 2024-03-06T07:20:12.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.090+0000 7f8876047640 1 -- 172.21.15.94:0/2439116423 wait complete. 2024-03-06T07:20:12.103 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-03-06T07:20:10.659980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":209,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237920,"kb_used_data":5120,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736800,"statfs":{"total":773094113280,"available":772850483200,"internally_reserved":0,"allocated":5242880,"data_stored":3450788,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.396926"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276976+0000","last_change":"2024-03-06T07:20:07.276976+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276976+0000","last_undegraded":"2024-03-06T07:20:07.276976+0000","last_fullsized":"2024-03-06T07:20:07.276976+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266870+0000","last_change":"2024-03-06T07:19:54.052304+0000","last_active":"2024-03-06T07:20:07.266870+0000","last_peered":"2024-03-06T07:20:07.266870+0000","last_clean":"2024-03-06T07:20:07.266870+0000","last_became_active":"2024-03-06T07:19:54.052048+0000","last_became_peered":"2024-03-06T07:19:54.052048+0000","last_unstale":"2024-03-06T07:20:07.266870+0000","last_undegraded":"2024-03-06T07:20:07.266870+0000","last_fullsized":"2024-03-06T07:20:07.266870+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:24.129824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270987+0000","last_change":"2024-03-06T07:20:07.270987+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270987+0000","last_undegraded":"2024-03-06T07:20:07.270987+0000","last_fullsized":"2024-03-06T07:20:07.270987+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283665+0000","last_change":"2024-03-06T07:19:54.062911+0000","last_active":"2024-03-06T07:20:07.283665+0000","last_peered":"2024-03-06T07:20:07.283665+0000","last_clean":"2024-03-06T07:20:07.283665+0000","last_became_active":"2024-03-06T07:19:54.062566+0000","last_became_peered":"2024-03-06T07:19:54.062566+0000","last_unstale":"2024-03-06T07:20:07.283665+0000","last_undegraded":"2024-03-06T07:20:07.283665+0000","last_fullsized":"2024-03-06T07:20:07.283665+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271068+0000","last_change":"2024-03-06T07:20:07.271068+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271068+0000","last_undegraded":"2024-03-06T07:20:07.271068+0000","last_fullsized":"2024-03-06T07:20:07.271068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272063+0000","last_change":"2024-03-06T07:19:54.051868+0000","last_active":"2024-03-06T07:20:07.272063+0000","last_peered":"2024-03-06T07:20:07.272063+0000","last_clean":"2024-03-06T07:20:07.272063+0000","last_became_active":"2024-03-06T07:19:54.051611+0000","last_became_peered":"2024-03-06T07:19:54.051611+0000","last_unstale":"2024-03-06T07:20:07.272063+0000","last_undegraded":"2024-03-06T07:20:07.272063+0000","last_fullsized":"2024-03-06T07:20:07.272063+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270217+0000","last_change":"2024-03-06T07:20:07.270217+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270217+0000","last_undegraded":"2024-03-06T07:20:07.270217+0000","last_fullsized":"2024-03-06T07:20:07.270217+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279217+0000","last_change":"2024-03-06T07:19:54.055553+0000","last_active":"2024-03-06T07:20:07.279217+0000","last_peered":"2024-03-06T07:20:07.279217+0000","last_clean":"2024-03-06T07:20:07.279217+0000","last_became_active":"2024-03-06T07:19:54.055271+0000","last_became_peered":"2024-03-06T07:19:54.055271+0000","last_unstale":"2024-03-06T07:20:07.279217+0000","last_undegraded":"2024-03-06T07:20:07.279217+0000","last_fullsized":"2024-03-06T07:20:07.279217+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274065+0000","last_change":"2024-03-06T07:20:07.274065+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274065+0000","last_undegraded":"2024-03-06T07:20:07.274065+0000","last_fullsized":"2024-03-06T07:20:07.274065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274273+0000","last_change":"2024-03-06T07:19:54.046161+0000","last_active":"2024-03-06T07:20:07.274273+0000","last_peered":"2024-03-06T07:20:07.274273+0000","last_clean":"2024-03-06T07:20:07.274273+0000","last_became_active":"2024-03-06T07:19:54.045875+0000","last_became_peered":"2024-03-06T07:19:54.045875+0000","last_unstale":"2024-03-06T07:20:07.274273+0000","last_undegraded":"2024-03-06T07:20:07.274273+0000","last_fullsized":"2024-03-06T07:20:07.274273+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:50.526448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277826+0000","last_change":"2024-03-06T07:20:07.277826+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277826+0000","last_undegraded":"2024-03-06T07:20:07.277826+0000","last_fullsized":"2024-03-06T07:20:07.277826+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266355+0000","last_change":"2024-03-06T07:19:54.047960+0000","last_active":"2024-03-06T07:20:07.266355+0000","last_peered":"2024-03-06T07:20:07.266355+0000","last_clean":"2024-03-06T07:20:07.266355+0000","last_became_active":"2024-03-06T07:19:54.047638+0000","last_became_peered":"2024-03-06T07:19:54.047638+0000","last_unstale":"2024-03-06T07:20:07.266355+0000","last_undegraded":"2024-03-06T07:20:07.266355+0000","last_fullsized":"2024-03-06T07:20:07.266355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272580+0000","last_change":"2024-03-06T07:20:07.272580+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272580+0000","last_undegraded":"2024-03-06T07:20:07.272580+0000","last_fullsized":"2024-03-06T07:20:07.272580+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.269825+0000","last_change":"2024-03-06T07:19:54.048726+0000","last_active":"2024-03-06T07:20:07.269825+0000","last_peered":"2024-03-06T07:20:07.269825+0000","last_clean":"2024-03-06T07:20:07.269825+0000","last_became_active":"2024-03-06T07:19:54.048377+0000","last_became_peered":"2024-03-06T07:19:54.048377+0000","last_unstale":"2024-03-06T07:20:07.269825+0000","last_undegraded":"2024-03-06T07:20:07.269825+0000","last_fullsized":"2024-03-06T07:20:07.269825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:57.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271012+0000","last_change":"2024-03-06T07:20:07.271012+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271012+0000","last_undegraded":"2024-03-06T07:20:07.271012+0000","last_fullsized":"2024-03-06T07:20:07.271012+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273477+0000","last_change":"2024-03-06T07:19:54.059007+0000","last_active":"2024-03-06T07:20:07.273477+0000","last_peered":"2024-03-06T07:20:07.273477+0000","last_clean":"2024-03-06T07:20:07.273477+0000","last_became_active":"2024-03-06T07:19:54.058694+0000","last_became_peered":"2024-03-06T07:19:54.058694+0000","last_unstale":"2024-03-06T07:20:07.273477+0000","last_undegraded":"2024-03-06T07:20:07.273477+0000","last_fullsized":"2024-03-06T07:20:07.273477+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:50.508506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271093+0000","last_change":"2024-03-06T07:20:07.271093+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271093+0000","last_undegraded":"2024-03-06T07:20:07.271093+0000","last_fullsized":"2024-03-06T07:20:07.271093+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274539+0000","last_change":"2024-03-06T07:19:54.057769+0000","last_active":"2024-03-06T07:20:07.274539+0000","last_peered":"2024-03-06T07:20:07.274539+0000","last_clean":"2024-03-06T07:20:07.274539+0000","last_became_active":"2024-03-06T07:19:54.057496+0000","last_became_peered":"2024-03-06T07:19:54.057496+0000","last_unstale":"2024-03-06T07:20:07.274539+0000","last_undegraded":"2024-03-06T07:20:07.274539+0000","last_fullsized":"2024-03-06T07:20:07.274539+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:51.719701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278288+0000","last_change":"2024-03-06T07:20:07.278288+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278288+0000","last_undegraded":"2024-03-06T07:20:07.278288+0000","last_fullsized":"2024-03-06T07:20:07.278288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266695+0000","last_change":"2024-03-06T07:19:54.050005+0000","last_active":"2024-03-06T07:20:07.266695+0000","last_peered":"2024-03-06T07:20:07.266695+0000","last_clean":"2024-03-06T07:20:07.266695+0000","last_became_active":"2024-03-06T07:19:54.049750+0000","last_became_peered":"2024-03-06T07:19:54.049750+0000","last_unstale":"2024-03-06T07:20:07.266695+0000","last_undegraded":"2024-03-06T07:20:07.266695+0000","last_fullsized":"2024-03-06T07:20:07.266695+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:37.609871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.280462+0000","last_change":"2024-03-06T07:20:07.280462+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.280462+0000","last_undegraded":"2024-03-06T07:20:07.280462+0000","last_fullsized":"2024-03-06T07:20:07.280462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275862+0000","last_change":"2024-03-06T07:19:54.052367+0000","last_active":"2024-03-06T07:20:07.275862+0000","last_peered":"2024-03-06T07:20:07.275862+0000","last_clean":"2024-03-06T07:20:07.275862+0000","last_became_active":"2024-03-06T07:19:54.052119+0000","last_became_peered":"2024-03-06T07:19:54.052119+0000","last_unstale":"2024-03-06T07:20:07.275862+0000","last_undegraded":"2024-03-06T07:20:07.275862+0000","last_fullsized":"2024-03-06T07:20:07.275862+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.678839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278688+0000","last_change":"2024-03-06T07:20:07.278688+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278688+0000","last_undegraded":"2024-03-06T07:20:07.278688+0000","last_fullsized":"2024-03-06T07:20:07.278688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273827+0000","last_change":"2024-03-06T07:19:54.040294+0000","last_active":"2024-03-06T07:20:07.273827+0000","last_peered":"2024-03-06T07:20:07.273827+0000","last_clean":"2024-03-06T07:20:07.273827+0000","last_became_active":"2024-03-06T07:19:54.040048+0000","last_became_peered":"2024-03-06T07:19:54.040048+0000","last_unstale":"2024-03-06T07:20:07.273827+0000","last_undegraded":"2024-03-06T07:20:07.273827+0000","last_fullsized":"2024-03-06T07:20:07.273827+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:41.393636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277157+0000","last_change":"2024-03-06T07:20:07.277157+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277157+0000","last_undegraded":"2024-03-06T07:20:07.277157+0000","last_fullsized":"2024-03-06T07:20:07.277157+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266413+0000","last_change":"2024-03-06T07:19:54.056367+0000","last_active":"2024-03-06T07:20:07.266413+0000","last_peered":"2024-03-06T07:20:07.266413+0000","last_clean":"2024-03-06T07:20:07.266413+0000","last_became_active":"2024-03-06T07:19:54.056125+0000","last_became_peered":"2024-03-06T07:19:54.056125+0000","last_unstale":"2024-03-06T07:20:07.266413+0000","last_undegraded":"2024-03-06T07:20:07.266413+0000","last_fullsized":"2024-03-06T07:20:07.266413+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273689+0000","last_change":"2024-03-06T07:20:07.273689+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273689+0000","last_undegraded":"2024-03-06T07:20:07.273689+0000","last_fullsized":"2024-03-06T07:20:07.273689+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271282+0000","last_change":"2024-03-06T07:19:54.051456+0000","last_active":"2024-03-06T07:20:07.271282+0000","last_peered":"2024-03-06T07:20:07.271282+0000","last_clean":"2024-03-06T07:20:07.271282+0000","last_became_active":"2024-03-06T07:19:54.051193+0000","last_became_peered":"2024-03-06T07:19:54.051193+0000","last_unstale":"2024-03-06T07:20:07.271282+0000","last_undegraded":"2024-03-06T07:20:07.271282+0000","last_fullsized":"2024-03-06T07:20:07.271282+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276915+0000","last_change":"2024-03-06T07:20:07.276915+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276915+0000","last_undegraded":"2024-03-06T07:20:07.276915+0000","last_fullsized":"2024-03-06T07:20:07.276915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279180+0000","last_change":"2024-03-06T07:19:54.049011+0000","last_active":"2024-03-06T07:20:07.279180+0000","last_peered":"2024-03-06T07:20:07.279180+0000","last_clean":"2024-03-06T07:20:07.279180+0000","last_became_active":"2024-03-06T07:19:54.048738+0000","last_became_peered":"2024-03-06T07:19:54.048738+0000","last_unstale":"2024-03-06T07:20:07.279180+0000","last_undegraded":"2024-03-06T07:20:07.279180+0000","last_fullsized":"2024-03-06T07:20:07.279180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:36.367136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278431+0000","last_change":"2024-03-06T07:20:07.278431+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278431+0000","last_undegraded":"2024-03-06T07:20:07.278431+0000","last_fullsized":"2024-03-06T07:20:07.278431+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266230+0000","last_change":"2024-03-06T07:19:54.049339+0000","last_active":"2024-03-06T07:20:07.266230+0000","last_peered":"2024-03-06T07:20:07.266230+0000","last_clean":"2024-03-06T07:20:07.266230+0000","last_became_active":"2024-03-06T07:19:54.049059+0000","last_became_peered":"2024-03-06T07:19:54.049059+0000","last_unstale":"2024-03-06T07:20:07.266230+0000","last_undegraded":"2024-03-06T07:20:07.266230+0000","last_fullsized":"2024-03-06T07:20:07.266230+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272112+0000","last_change":"2024-03-06T07:20:07.272112+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272112+0000","last_undegraded":"2024-03-06T07:20:07.272112+0000","last_fullsized":"2024-03-06T07:20:07.272112+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272044+0000","last_change":"2024-03-06T07:19:54.058699+0000","last_active":"2024-03-06T07:20:07.272044+0000","last_peered":"2024-03-06T07:20:07.272044+0000","last_clean":"2024-03-06T07:20:07.272044+0000","last_became_active":"2024-03-06T07:19:54.056798+0000","last_became_peered":"2024-03-06T07:19:54.056798+0000","last_unstale":"2024-03-06T07:20:07.272044+0000","last_undegraded":"2024-03-06T07:20:07.272044+0000","last_fullsized":"2024-03-06T07:20:07.272044+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.279205+0000","last_change":"2024-03-06T07:20:07.279205+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.279205+0000","last_undegraded":"2024-03-06T07:20:07.279205+0000","last_fullsized":"2024-03-06T07:20:07.279205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283763+0000","last_change":"2024-03-06T07:19:54.062261+0000","last_active":"2024-03-06T07:20:07.283763+0000","last_peered":"2024-03-06T07:20:07.283763+0000","last_clean":"2024-03-06T07:20:07.283763+0000","last_became_active":"2024-03-06T07:19:54.061853+0000","last_became_peered":"2024-03-06T07:19:54.061853+0000","last_unstale":"2024-03-06T07:20:07.283763+0000","last_undegraded":"2024-03-06T07:20:07.283763+0000","last_fullsized":"2024-03-06T07:20:07.283763+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270647+0000","last_change":"2024-03-06T07:20:07.270647+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270647+0000","last_undegraded":"2024-03-06T07:20:07.270647+0000","last_fullsized":"2024-03-06T07:20:07.270647+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272164+0000","last_change":"2024-03-06T07:19:54.054672+0000","last_active":"2024-03-06T07:20:07.272164+0000","last_peered":"2024-03-06T07:20:07.272164+0000","last_clean":"2024-03-06T07:20:07.272164+0000","last_became_active":"2024-03-06T07:19:54.054301+0000","last_became_peered":"2024-03-06T07:19:54.054301+0000","last_unstale":"2024-03-06T07:20:07.272164+0000","last_undegraded":"2024-03-06T07:20:07.272164+0000","last_fullsized":"2024-03-06T07:20:07.272164+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:05.772166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272968+0000","last_change":"2024-03-06T07:20:07.272968+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272968+0000","last_undegraded":"2024-03-06T07:20:07.272968+0000","last_fullsized":"2024-03-06T07:20:07.272968+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275480+0000","last_change":"2024-03-06T07:19:54.051669+0000","last_active":"2024-03-06T07:20:07.275480+0000","last_peered":"2024-03-06T07:20:07.275480+0000","last_clean":"2024-03-06T07:20:07.275480+0000","last_became_active":"2024-03-06T07:19:54.051312+0000","last_became_peered":"2024-03-06T07:19:54.051312+0000","last_unstale":"2024-03-06T07:20:07.275480+0000","last_undegraded":"2024-03-06T07:20:07.275480+0000","last_fullsized":"2024-03-06T07:20:07.275480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277902+0000","last_change":"2024-03-06T07:20:07.277902+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277902+0000","last_undegraded":"2024-03-06T07:20:07.277902+0000","last_fullsized":"2024-03-06T07:20:07.277902+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275715+0000","last_change":"2024-03-06T07:19:54.047677+0000","last_active":"2024-03-06T07:20:07.275715+0000","last_peered":"2024-03-06T07:20:07.275715+0000","last_clean":"2024-03-06T07:20:07.275715+0000","last_became_active":"2024-03-06T07:19:54.045393+0000","last_became_peered":"2024-03-06T07:19:54.045393+0000","last_unstale":"2024-03-06T07:20:07.275715+0000","last_undegraded":"2024-03-06T07:20:07.275715+0000","last_fullsized":"2024-03-06T07:20:07.275715+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277366+0000","last_change":"2024-03-06T07:20:07.277366+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277366+0000","last_undegraded":"2024-03-06T07:20:07.277366+0000","last_fullsized":"2024-03-06T07:20:07.277366+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275908+0000","last_change":"2024-03-06T07:19:54.058376+0000","last_active":"2024-03-06T07:20:07.275908+0000","last_peered":"2024-03-06T07:20:07.275908+0000","last_clean":"2024-03-06T07:20:07.275908+0000","last_became_active":"2024-03-06T07:19:54.058151+0000","last_became_peered":"2024-03-06T07:19:54.058151+0000","last_unstale":"2024-03-06T07:20:07.275908+0000","last_undegraded":"2024-03-06T07:20:07.275908+0000","last_fullsized":"2024-03-06T07:20:07.275908+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:10.287168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276244+0000","last_change":"2024-03-06T07:20:07.276244+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276244+0000","last_undegraded":"2024-03-06T07:20:07.276244+0000","last_fullsized":"2024-03-06T07:20:07.276244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275078+0000","last_change":"2024-03-06T07:19:54.046473+0000","last_active":"2024-03-06T07:20:07.275078+0000","last_peered":"2024-03-06T07:20:07.275078+0000","last_clean":"2024-03-06T07:20:07.275078+0000","last_became_active":"2024-03-06T07:19:54.045977+0000","last_became_peered":"2024-03-06T07:19:54.045977+0000","last_unstale":"2024-03-06T07:20:07.275078+0000","last_undegraded":"2024-03-06T07:20:07.275078+0000","last_fullsized":"2024-03-06T07:20:07.275078+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:28.437704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278418+0000","last_change":"2024-03-06T07:20:07.278418+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278418+0000","last_undegraded":"2024-03-06T07:20:07.278418+0000","last_fullsized":"2024-03-06T07:20:07.278418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272271+0000","last_change":"2024-03-06T07:19:54.050649+0000","last_active":"2024-03-06T07:20:07.272271+0000","last_peered":"2024-03-06T07:20:07.272271+0000","last_clean":"2024-03-06T07:20:07.272271+0000","last_became_active":"2024-03-06T07:19:54.050392+0000","last_became_peered":"2024-03-06T07:19:54.050392+0000","last_unstale":"2024-03-06T07:20:07.272271+0000","last_undegraded":"2024-03-06T07:20:07.272271+0000","last_fullsized":"2024-03-06T07:20:07.272271+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272555+0000","last_change":"2024-03-06T07:20:07.272555+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272555+0000","last_undegraded":"2024-03-06T07:20:07.272555+0000","last_fullsized":"2024-03-06T07:20:07.272555+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272084+0000","last_change":"2024-03-06T07:19:54.046865+0000","last_active":"2024-03-06T07:20:07.272084+0000","last_peered":"2024-03-06T07:20:07.272084+0000","last_clean":"2024-03-06T07:20:07.272084+0000","last_became_active":"2024-03-06T07:19:54.046545+0000","last_became_peered":"2024-03-06T07:19:54.046545+0000","last_unstale":"2024-03-06T07:20:07.272084+0000","last_undegraded":"2024-03-06T07:20:07.272084+0000","last_fullsized":"2024-03-06T07:20:07.272084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:09.982751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270046+0000","last_change":"2024-03-06T07:20:07.270046+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270046+0000","last_undegraded":"2024-03-06T07:20:07.270046+0000","last_fullsized":"2024-03-06T07:20:07.270046+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273407+0000","last_change":"2024-03-06T07:19:54.042151+0000","last_active":"2024-03-06T07:20:07.273407+0000","last_peered":"2024-03-06T07:20:07.273407+0000","last_clean":"2024-03-06T07:20:07.273407+0000","last_became_active":"2024-03-06T07:19:54.041796+0000","last_became_peered":"2024-03-06T07:19:54.041796+0000","last_unstale":"2024-03-06T07:20:07.273407+0000","last_undegraded":"2024-03-06T07:20:07.273407+0000","last_fullsized":"2024-03-06T07:20:07.273407+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:27.407999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278821+0000","last_change":"2024-03-06T07:20:07.278821+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278821+0000","last_undegraded":"2024-03-06T07:20:07.278821+0000","last_fullsized":"2024-03-06T07:20:07.278821+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266779+0000","last_change":"2024-03-06T07:19:54.051161+0000","last_active":"2024-03-06T07:20:07.266779+0000","last_peered":"2024-03-06T07:20:07.266779+0000","last_clean":"2024-03-06T07:20:07.266779+0000","last_became_active":"2024-03-06T07:19:54.050893+0000","last_became_peered":"2024-03-06T07:19:54.050893+0000","last_unstale":"2024-03-06T07:20:07.266779+0000","last_undegraded":"2024-03-06T07:20:07.266779+0000","last_fullsized":"2024-03-06T07:20:07.266779+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:28.711633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277996+0000","last_change":"2024-03-06T07:20:07.277996+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277996+0000","last_undegraded":"2024-03-06T07:20:07.277996+0000","last_fullsized":"2024-03-06T07:20:07.277996+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275574+0000","last_change":"2024-03-06T07:19:54.040041+0000","last_active":"2024-03-06T07:20:07.275574+0000","last_peered":"2024-03-06T07:20:07.275574+0000","last_clean":"2024-03-06T07:20:07.275574+0000","last_became_active":"2024-03-06T07:19:54.039730+0000","last_became_peered":"2024-03-06T07:19:54.039730+0000","last_unstale":"2024-03-06T07:20:07.275574+0000","last_undegraded":"2024-03-06T07:20:07.275574+0000","last_fullsized":"2024-03-06T07:20:07.275574+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271067+0000","last_change":"2024-03-06T07:20:07.271067+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271067+0000","last_undegraded":"2024-03-06T07:20:07.271067+0000","last_fullsized":"2024-03-06T07:20:07.271067+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272272+0000","last_change":"2024-03-06T07:19:54.055165+0000","last_active":"2024-03-06T07:20:07.272272+0000","last_peered":"2024-03-06T07:20:07.272272+0000","last_clean":"2024-03-06T07:20:07.272272+0000","last_became_active":"2024-03-06T07:19:54.054908+0000","last_became_peered":"2024-03-06T07:19:54.054908+0000","last_unstale":"2024-03-06T07:20:07.272272+0000","last_undegraded":"2024-03-06T07:20:07.272272+0000","last_fullsized":"2024-03-06T07:20:07.272272+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:02.499776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.270342+0000","last_change":"2024-03-06T07:20:07.270342+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.270342+0000","last_undegraded":"2024-03-06T07:20:07.270342+0000","last_fullsized":"2024-03-06T07:20:07.270342+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278948+0000","last_change":"2024-03-06T07:19:54.049805+0000","last_active":"2024-03-06T07:20:07.278948+0000","last_peered":"2024-03-06T07:20:07.278948+0000","last_clean":"2024-03-06T07:20:07.278948+0000","last_became_active":"2024-03-06T07:19:54.049562+0000","last_became_peered":"2024-03-06T07:19:54.049562+0000","last_unstale":"2024-03-06T07:20:07.278948+0000","last_undegraded":"2024-03-06T07:20:07.278948+0000","last_fullsized":"2024-03-06T07:20:07.278948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:21.645621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276440+0000","last_change":"2024-03-06T07:20:07.276440+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276440+0000","last_undegraded":"2024-03-06T07:20:07.276440+0000","last_fullsized":"2024-03-06T07:20:07.276440+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.269928+0000","last_change":"2024-03-06T07:19:54.052763+0000","last_active":"2024-03-06T07:20:07.269928+0000","last_peered":"2024-03-06T07:20:07.269928+0000","last_clean":"2024-03-06T07:20:07.269928+0000","last_became_active":"2024-03-06T07:19:54.052472+0000","last_became_peered":"2024-03-06T07:19:54.052472+0000","last_unstale":"2024-03-06T07:20:07.269928+0000","last_undegraded":"2024-03-06T07:20:07.269928+0000","last_fullsized":"2024-03-06T07:20:07.269928+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:07.485843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277442+0000","last_change":"2024-03-06T07:20:07.277442+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277442+0000","last_undegraded":"2024-03-06T07:20:07.277442+0000","last_fullsized":"2024-03-06T07:20:07.277442+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279478+0000","last_change":"2024-03-06T07:19:54.059383+0000","last_active":"2024-03-06T07:20:07.279478+0000","last_peered":"2024-03-06T07:20:07.279478+0000","last_clean":"2024-03-06T07:20:07.279478+0000","last_became_active":"2024-03-06T07:19:54.059123+0000","last_became_peered":"2024-03-06T07:19:54.059123+0000","last_unstale":"2024-03-06T07:20:07.279478+0000","last_undegraded":"2024-03-06T07:20:07.279478+0000","last_fullsized":"2024-03-06T07:20:07.279478+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273940+0000","last_change":"2024-03-06T07:20:07.273940+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273940+0000","last_undegraded":"2024-03-06T07:20:07.273940+0000","last_fullsized":"2024-03-06T07:20:07.273940+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289071+0000","last_change":"2024-03-06T07:20:07.288076+0000","last_active":"2024-03-06T07:20:07.289071+0000","last_peered":"2024-03-06T07:20:07.289071+0000","last_clean":"2024-03-06T07:20:07.289071+0000","last_became_active":"2024-03-06T07:20:07.286803+0000","last_became_peered":"2024-03-06T07:20:07.286803+0000","last_unstale":"2024-03-06T07:20:07.289071+0000","last_undegraded":"2024-03-06T07:20:07.289071+0000","last_fullsized":"2024-03-06T07:20:07.289071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.832403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278691+0000","last_change":"2024-03-06T07:19:54.060233+0000","last_active":"2024-03-06T07:20:07.278691+0000","last_peered":"2024-03-06T07:20:07.278691+0000","last_clean":"2024-03-06T07:20:07.278691+0000","last_became_active":"2024-03-06T07:19:54.059901+0000","last_became_peered":"2024-03-06T07:19:54.059901+0000","last_unstale":"2024-03-06T07:20:07.278691+0000","last_undegraded":"2024-03-06T07:20:07.278691+0000","last_fullsized":"2024-03-06T07:20:07.278691+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:09.921962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275933+0000","last_change":"2024-03-06T07:19:53.042382+0000","last_active":"2024-03-06T07:20:07.275933+0000","last_peered":"2024-03-06T07:20:07.275933+0000","last_clean":"2024-03-06T07:20:07.275933+0000","last_became_active":"2024-03-06T07:19:53.041334+0000","last_became_peered":"2024-03-06T07:19:53.041334+0000","last_unstale":"2024-03-06T07:20:07.275933+0000","last_undegraded":"2024-03-06T07:20:07.275933+0000","last_fullsized":"2024-03-06T07:20:07.275933+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269519+0000","last_change":"2024-03-06T07:20:07.269519+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269519+0000","last_undegraded":"2024-03-06T07:20:07.269519+0000","last_fullsized":"2024-03-06T07:20:07.269519+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287328+0000","last_change":"2024-03-06T07:20:07.285089+0000","last_active":"2024-03-06T07:20:07.287328+0000","last_peered":"2024-03-06T07:20:07.287328+0000","last_clean":"2024-03-06T07:20:07.287328+0000","last_became_active":"2024-03-06T07:20:07.284839+0000","last_became_peered":"2024-03-06T07:20:07.284839+0000","last_unstale":"2024-03-06T07:20:07.287328+0000","last_undegraded":"2024-03-06T07:20:07.287328+0000","last_fullsized":"2024-03-06T07:20:07.287328+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:10.745713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279379+0000","last_change":"2024-03-06T07:19:54.057284+0000","last_active":"2024-03-06T07:20:07.279379+0000","last_peered":"2024-03-06T07:20:07.279379+0000","last_clean":"2024-03-06T07:20:07.279379+0000","last_became_active":"2024-03-06T07:19:54.056999+0000","last_became_peered":"2024-03-06T07:19:54.056999+0000","last_unstale":"2024-03-06T07:20:07.279379+0000","last_undegraded":"2024-03-06T07:20:07.279379+0000","last_fullsized":"2024-03-06T07:20:07.279379+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266885+0000","last_change":"2024-03-06T07:19:53.048675+0000","last_active":"2024-03-06T07:20:07.266885+0000","last_peered":"2024-03-06T07:20:07.266885+0000","last_clean":"2024-03-06T07:20:07.266885+0000","last_became_active":"2024-03-06T07:19:53.048416+0000","last_became_peered":"2024-03-06T07:19:53.048416+0000","last_unstale":"2024-03-06T07:20:07.266885+0000","last_undegraded":"2024-03-06T07:20:07.266885+0000","last_fullsized":"2024-03-06T07:20:07.266885+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271271+0000","last_change":"2024-03-06T07:20:07.271271+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271271+0000","last_undegraded":"2024-03-06T07:20:07.271271+0000","last_fullsized":"2024-03-06T07:20:07.271271+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287393+0000","last_change":"2024-03-06T07:20:07.285244+0000","last_active":"2024-03-06T07:20:07.287393+0000","last_peered":"2024-03-06T07:20:07.287393+0000","last_clean":"2024-03-06T07:20:07.287393+0000","last_became_active":"2024-03-06T07:20:07.285012+0000","last_became_peered":"2024-03-06T07:20:07.285012+0000","last_unstale":"2024-03-06T07:20:07.287393+0000","last_undegraded":"2024-03-06T07:20:07.287393+0000","last_fullsized":"2024-03-06T07:20:07.287393+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:10.657318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275244+0000","last_change":"2024-03-06T07:19:54.052199+0000","last_active":"2024-03-06T07:20:07.275244+0000","last_peered":"2024-03-06T07:20:07.275244+0000","last_clean":"2024-03-06T07:20:07.275244+0000","last_became_active":"2024-03-06T07:19:54.051923+0000","last_became_peered":"2024-03-06T07:19:54.051923+0000","last_unstale":"2024-03-06T07:20:07.275244+0000","last_undegraded":"2024-03-06T07:20:07.275244+0000","last_fullsized":"2024-03-06T07:20:07.275244+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266828+0000","last_change":"2024-03-06T07:19:53.044281+0000","last_active":"2024-03-06T07:20:07.266828+0000","last_peered":"2024-03-06T07:20:07.266828+0000","last_clean":"2024-03-06T07:20:07.266828+0000","last_became_active":"2024-03-06T07:19:53.043841+0000","last_became_peered":"2024-03-06T07:19:53.043841+0000","last_unstale":"2024-03-06T07:20:07.266828+0000","last_undegraded":"2024-03-06T07:20:07.266828+0000","last_fullsized":"2024-03-06T07:20:07.266828+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276210+0000","last_change":"2024-03-06T07:20:07.276210+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276210+0000","last_undegraded":"2024-03-06T07:20:07.276210+0000","last_fullsized":"2024-03-06T07:20:07.276210+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287847+0000","last_change":"2024-03-06T07:20:07.285494+0000","last_active":"2024-03-06T07:20:07.287847+0000","last_peered":"2024-03-06T07:20:07.287847+0000","last_clean":"2024-03-06T07:20:07.287847+0000","last_became_active":"2024-03-06T07:20:07.285238+0000","last_became_peered":"2024-03-06T07:20:07.285238+0000","last_unstale":"2024-03-06T07:20:07.287847+0000","last_undegraded":"2024-03-06T07:20:07.287847+0000","last_fullsized":"2024-03-06T07:20:07.287847+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.832403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274742+0000","last_change":"2024-03-06T07:19:54.039974+0000","last_active":"2024-03-06T07:20:07.274742+0000","last_peered":"2024-03-06T07:20:07.274742+0000","last_clean":"2024-03-06T07:20:07.274742+0000","last_became_active":"2024-03-06T07:19:54.039675+0000","last_became_peered":"2024-03-06T07:19:54.039675+0000","last_unstale":"2024-03-06T07:20:07.274742+0000","last_undegraded":"2024-03-06T07:20:07.274742+0000","last_fullsized":"2024-03-06T07:20:07.274742+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:53.089205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279819+0000","last_change":"2024-03-06T07:19:53.051640+0000","last_active":"2024-03-06T07:20:07.279819+0000","last_peered":"2024-03-06T07:20:07.279819+0000","last_clean":"2024-03-06T07:20:07.279819+0000","last_became_active":"2024-03-06T07:19:53.051429+0000","last_became_peered":"2024-03-06T07:19:53.051429+0000","last_unstale":"2024-03-06T07:20:07.279819+0000","last_undegraded":"2024-03-06T07:20:07.279819+0000","last_fullsized":"2024-03-06T07:20:07.279819+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:09.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271758+0000","last_change":"2024-03-06T07:20:07.271758+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271758+0000","last_undegraded":"2024-03-06T07:20:07.271758+0000","last_fullsized":"2024-03-06T07:20:07.271758+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288414+0000","last_change":"2024-03-06T07:20:07.287035+0000","last_active":"2024-03-06T07:20:07.288414+0000","last_peered":"2024-03-06T07:20:07.288414+0000","last_clean":"2024-03-06T07:20:07.288414+0000","last_became_active":"2024-03-06T07:20:07.286453+0000","last_became_peered":"2024-03-06T07:20:07.286453+0000","last_unstale":"2024-03-06T07:20:07.288414+0000","last_undegraded":"2024-03-06T07:20:07.288414+0000","last_fullsized":"2024-03-06T07:20:07.288414+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:15.726763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274553+0000","last_change":"2024-03-06T07:19:54.054949+0000","last_active":"2024-03-06T07:20:07.274553+0000","last_peered":"2024-03-06T07:20:07.274553+0000","last_clean":"2024-03-06T07:20:07.274553+0000","last_became_active":"2024-03-06T07:19:54.054661+0000","last_became_peered":"2024-03-06T07:19:54.054661+0000","last_unstale":"2024-03-06T07:20:07.274553+0000","last_undegraded":"2024-03-06T07:20:07.274553+0000","last_fullsized":"2024-03-06T07:20:07.274553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:57.129016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272386+0000","last_change":"2024-03-06T07:19:53.042003+0000","last_active":"2024-03-06T07:20:07.272386+0000","last_peered":"2024-03-06T07:20:07.272386+0000","last_clean":"2024-03-06T07:20:07.272386+0000","last_became_active":"2024-03-06T07:19:53.041728+0000","last_became_peered":"2024-03-06T07:19:53.041728+0000","last_unstale":"2024-03-06T07:20:07.272386+0000","last_undegraded":"2024-03-06T07:20:07.272386+0000","last_fullsized":"2024-03-06T07:20:07.272386+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.964097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271661+0000","last_change":"2024-03-06T07:20:07.271661+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271661+0000","last_undegraded":"2024-03-06T07:20:07.271661+0000","last_fullsized":"2024-03-06T07:20:07.271661+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284084+0000","last_change":"2024-03-06T07:20:07.278974+0000","last_active":"2024-03-06T07:20:07.284084+0000","last_peered":"2024-03-06T07:20:07.284084+0000","last_clean":"2024-03-06T07:20:07.284084+0000","last_became_active":"2024-03-06T07:20:07.278682+0000","last_became_peered":"2024-03-06T07:20:07.278682+0000","last_unstale":"2024-03-06T07:20:07.284084+0000","last_undegraded":"2024-03-06T07:20:07.284084+0000","last_fullsized":"2024-03-06T07:20:07.284084+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:27.012782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281002+0000","last_change":"2024-03-06T07:19:54.061284+0000","last_active":"2024-03-06T07:20:07.281002+0000","last_peered":"2024-03-06T07:20:07.281002+0000","last_clean":"2024-03-06T07:20:07.281002+0000","last_became_active":"2024-03-06T07:19:54.061017+0000","last_became_peered":"2024-03-06T07:19:54.061017+0000","last_unstale":"2024-03-06T07:20:07.281002+0000","last_undegraded":"2024-03-06T07:20:07.281002+0000","last_fullsized":"2024-03-06T07:20:07.281002+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274253+0000","last_change":"2024-03-06T07:19:53.041819+0000","last_active":"2024-03-06T07:20:07.274253+0000","last_peered":"2024-03-06T07:20:07.274253+0000","last_clean":"2024-03-06T07:20:07.274253+0000","last_became_active":"2024-03-06T07:19:53.041542+0000","last_became_peered":"2024-03-06T07:19:53.041542+0000","last_unstale":"2024-03-06T07:20:07.274253+0000","last_undegraded":"2024-03-06T07:20:07.274253+0000","last_fullsized":"2024-03-06T07:20:07.274253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269496+0000","last_change":"2024-03-06T07:20:07.269496+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269496+0000","last_undegraded":"2024-03-06T07:20:07.269496+0000","last_fullsized":"2024-03-06T07:20:07.269496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.290812+0000","last_change":"2024-03-06T07:20:07.289776+0000","last_active":"2024-03-06T07:20:07.290812+0000","last_peered":"2024-03-06T07:20:07.290812+0000","last_clean":"2024-03-06T07:20:07.290812+0000","last_became_active":"2024-03-06T07:20:07.289535+0000","last_became_peered":"2024-03-06T07:20:07.289535+0000","last_unstale":"2024-03-06T07:20:07.290812+0000","last_undegraded":"2024-03-06T07:20:07.290812+0000","last_fullsized":"2024-03-06T07:20:07.290812+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273691+0000","last_change":"2024-03-06T07:19:54.039491+0000","last_active":"2024-03-06T07:20:07.273691+0000","last_peered":"2024-03-06T07:20:07.273691+0000","last_clean":"2024-03-06T07:20:07.273691+0000","last_became_active":"2024-03-06T07:19:54.039202+0000","last_became_peered":"2024-03-06T07:19:54.039202+0000","last_unstale":"2024-03-06T07:20:07.273691+0000","last_undegraded":"2024-03-06T07:20:07.273691+0000","last_fullsized":"2024-03-06T07:20:07.273691+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:58.292241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275185+0000","last_change":"2024-03-06T07:19:53.043210+0000","last_active":"2024-03-06T07:20:07.275185+0000","last_peered":"2024-03-06T07:20:07.275185+0000","last_clean":"2024-03-06T07:20:07.275185+0000","last_became_active":"2024-03-06T07:19:53.042951+0000","last_became_peered":"2024-03-06T07:19:53.042951+0000","last_unstale":"2024-03-06T07:20:07.275185+0000","last_undegraded":"2024-03-06T07:20:07.275185+0000","last_fullsized":"2024-03-06T07:20:07.275185+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276607+0000","last_change":"2024-03-06T07:20:07.276607+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276607+0000","last_undegraded":"2024-03-06T07:20:07.276607+0000","last_fullsized":"2024-03-06T07:20:07.276607+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283717+0000","last_change":"2024-03-06T07:20:07.278534+0000","last_active":"2024-03-06T07:20:07.283717+0000","last_peered":"2024-03-06T07:20:07.283717+0000","last_clean":"2024-03-06T07:20:07.283717+0000","last_became_active":"2024-03-06T07:20:07.278261+0000","last_became_peered":"2024-03-06T07:20:07.278261+0000","last_unstale":"2024-03-06T07:20:07.283717+0000","last_undegraded":"2024-03-06T07:20:07.283717+0000","last_fullsized":"2024-03-06T07:20:07.283717+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:23.448306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285369+0000","last_change":"2024-03-06T07:19:54.049884+0000","last_active":"2024-03-06T07:20:07.285369+0000","last_peered":"2024-03-06T07:20:07.285369+0000","last_clean":"2024-03-06T07:20:07.285369+0000","last_became_active":"2024-03-06T07:19:54.049597+0000","last_became_peered":"2024-03-06T07:19:54.049597+0000","last_unstale":"2024-03-06T07:20:07.285369+0000","last_undegraded":"2024-03-06T07:20:07.285369+0000","last_fullsized":"2024-03-06T07:20:07.285369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.270618+0000","last_change":"2024-03-06T07:19:53.043198+0000","last_active":"2024-03-06T07:20:07.270618+0000","last_peered":"2024-03-06T07:20:07.270618+0000","last_clean":"2024-03-06T07:20:07.270618+0000","last_became_active":"2024-03-06T07:19:53.042939+0000","last_became_peered":"2024-03-06T07:19:53.042939+0000","last_unstale":"2024-03-06T07:20:07.270618+0000","last_undegraded":"2024-03-06T07:20:07.270618+0000","last_fullsized":"2024-03-06T07:20:07.270618+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273576+0000","last_change":"2024-03-06T07:20:07.273576+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273576+0000","last_undegraded":"2024-03-06T07:20:07.273576+0000","last_fullsized":"2024-03-06T07:20:07.273576+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285627+0000","last_change":"2024-03-06T07:20:07.283576+0000","last_active":"2024-03-06T07:20:07.285627+0000","last_peered":"2024-03-06T07:20:07.285627+0000","last_clean":"2024-03-06T07:20:07.285627+0000","last_became_active":"2024-03-06T07:20:07.283136+0000","last_became_peered":"2024-03-06T07:20:07.283136+0000","last_unstale":"2024-03-06T07:20:07.285627+0000","last_undegraded":"2024-03-06T07:20:07.285627+0000","last_fullsized":"2024-03-06T07:20:07.285627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:56.830929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274311+0000","last_change":"2024-03-06T07:19:54.045809+0000","last_active":"2024-03-06T07:20:07.274311+0000","last_peered":"2024-03-06T07:20:07.274311+0000","last_clean":"2024-03-06T07:20:07.274311+0000","last_became_active":"2024-03-06T07:19:54.040263+0000","last_became_peered":"2024-03-06T07:19:54.040263+0000","last_unstale":"2024-03-06T07:20:07.274311+0000","last_undegraded":"2024-03-06T07:20:07.274311+0000","last_fullsized":"2024-03-06T07:20:07.274311+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:20.728340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285454+0000","last_change":"2024-03-06T07:19:53.047864+0000","last_active":"2024-03-06T07:20:07.285454+0000","last_peered":"2024-03-06T07:20:07.285454+0000","last_clean":"2024-03-06T07:20:07.285454+0000","last_became_active":"2024-03-06T07:19:53.047607+0000","last_became_peered":"2024-03-06T07:19:53.047607+0000","last_unstale":"2024-03-06T07:20:07.285454+0000","last_undegraded":"2024-03-06T07:20:07.285454+0000","last_fullsized":"2024-03-06T07:20:07.285454+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.148872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277242+0000","last_change":"2024-03-06T07:20:07.277242+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277242+0000","last_undegraded":"2024-03-06T07:20:07.277242+0000","last_fullsized":"2024-03-06T07:20:07.277242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288691+0000","last_change":"2024-03-06T07:20:07.285280+0000","last_active":"2024-03-06T07:20:07.288691+0000","last_peered":"2024-03-06T07:20:07.288691+0000","last_clean":"2024-03-06T07:20:07.288691+0000","last_became_active":"2024-03-06T07:20:07.284993+0000","last_became_peered":"2024-03-06T07:20:07.284993+0000","last_unstale":"2024-03-06T07:20:07.288691+0000","last_undegraded":"2024-03-06T07:20:07.288691+0000","last_fullsized":"2024-03-06T07:20:07.288691+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:59.458261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279291+0000","last_change":"2024-03-06T07:19:54.056898+0000","last_active":"2024-03-06T07:20:07.279291+0000","last_peered":"2024-03-06T07:20:07.279291+0000","last_clean":"2024-03-06T07:20:07.279291+0000","last_became_active":"2024-03-06T07:19:54.056522+0000","last_became_peered":"2024-03-06T07:19:54.056522+0000","last_unstale":"2024-03-06T07:20:07.279291+0000","last_undegraded":"2024-03-06T07:20:07.279291+0000","last_fullsized":"2024-03-06T07:20:07.279291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274521+0000","last_change":"2024-03-06T07:19:53.043065+0000","last_active":"2024-03-06T07:20:07.274521+0000","last_peered":"2024-03-06T07:20:07.274521+0000","last_clean":"2024-03-06T07:20:07.274521+0000","last_became_active":"2024-03-06T07:19:53.042756+0000","last_became_peered":"2024-03-06T07:19:53.042756+0000","last_unstale":"2024-03-06T07:20:07.274521+0000","last_undegraded":"2024-03-06T07:20:07.274521+0000","last_fullsized":"2024-03-06T07:20:07.274521+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.338932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274188+0000","last_change":"2024-03-06T07:20:07.274188+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274188+0000","last_undegraded":"2024-03-06T07:20:07.274188+0000","last_fullsized":"2024-03-06T07:20:07.274188+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285804+0000","last_change":"2024-03-06T07:20:07.282855+0000","last_active":"2024-03-06T07:20:07.285804+0000","last_peered":"2024-03-06T07:20:07.285804+0000","last_clean":"2024-03-06T07:20:07.285804+0000","last_became_active":"2024-03-06T07:20:07.282585+0000","last_became_peered":"2024-03-06T07:20:07.282585+0000","last_unstale":"2024-03-06T07:20:07.285804+0000","last_undegraded":"2024-03-06T07:20:07.285804+0000","last_fullsized":"2024-03-06T07:20:07.285804+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:52.503066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271456+0000","last_change":"2024-03-06T07:19:54.057647+0000","last_active":"2024-03-06T07:20:07.271456+0000","last_peered":"2024-03-06T07:20:07.271456+0000","last_clean":"2024-03-06T07:20:07.271456+0000","last_became_active":"2024-03-06T07:19:54.057396+0000","last_became_peered":"2024-03-06T07:19:54.057396+0000","last_unstale":"2024-03-06T07:20:07.271456+0000","last_undegraded":"2024-03-06T07:20:07.271456+0000","last_fullsized":"2024-03-06T07:20:07.271456+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":21,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281162+0000","last_change":"2024-03-06T07:19:53.046359+0000","last_active":"2024-03-06T07:20:07.281162+0000","last_peered":"2024-03-06T07:20:07.281162+0000","last_clean":"2024-03-06T07:20:07.281162+0000","last_became_active":"2024-03-06T07:19:53.046065+0000","last_became_peered":"2024-03-06T07:19:53.046065+0000","last_unstale":"2024-03-06T07:20:07.281162+0000","last_undegraded":"2024-03-06T07:20:07.281162+0000","last_fullsized":"2024-03-06T07:20:07.281162+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:08.146267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269510+0000","last_change":"2024-03-06T07:20:07.269510+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269510+0000","last_undegraded":"2024-03-06T07:20:07.269510+0000","last_fullsized":"2024-03-06T07:20:07.269510+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.283148+0000","last_change":"2024-03-06T07:20:07.280915+0000","last_active":"2024-03-06T07:20:07.283148+0000","last_peered":"2024-03-06T07:20:07.283148+0000","last_clean":"2024-03-06T07:20:07.283148+0000","last_became_active":"2024-03-06T07:20:07.280659+0000","last_became_peered":"2024-03-06T07:20:07.280659+0000","last_unstale":"2024-03-06T07:20:07.283148+0000","last_undegraded":"2024-03-06T07:20:07.283148+0000","last_fullsized":"2024-03-06T07:20:07.283148+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:59.749257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274291+0000","last_change":"2024-03-06T07:19:54.041140+0000","last_active":"2024-03-06T07:20:07.274291+0000","last_peered":"2024-03-06T07:20:07.274291+0000","last_clean":"2024-03-06T07:20:07.274291+0000","last_became_active":"2024-03-06T07:19:54.040877+0000","last_became_peered":"2024-03-06T07:19:54.040877+0000","last_unstale":"2024-03-06T07:20:07.274291+0000","last_undegraded":"2024-03-06T07:20:07.274291+0000","last_fullsized":"2024-03-06T07:20:07.274291+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:29.971359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":24,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273930+0000","last_change":"2024-03-06T07:19:53.045210+0000","last_active":"2024-03-06T07:20:07.273930+0000","last_peered":"2024-03-06T07:20:07.273930+0000","last_clean":"2024-03-06T07:20:07.273930+0000","last_became_active":"2024-03-06T07:19:53.044950+0000","last_became_peered":"2024-03-06T07:19:53.044950+0000","last_unstale":"2024-03-06T07:20:07.273930+0000","last_undegraded":"2024-03-06T07:20:07.273930+0000","last_fullsized":"2024-03-06T07:20:07.273930+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:53.407226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271405+0000","last_change":"2024-03-06T07:20:07.271405+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271405+0000","last_undegraded":"2024-03-06T07:20:07.271405+0000","last_fullsized":"2024-03-06T07:20:07.271405+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286730+0000","last_change":"2024-03-06T07:20:07.285841+0000","last_active":"2024-03-06T07:20:07.286730+0000","last_peered":"2024-03-06T07:20:07.286730+0000","last_clean":"2024-03-06T07:20:07.286730+0000","last_became_active":"2024-03-06T07:20:07.285505+0000","last_became_peered":"2024-03-06T07:20:07.285505+0000","last_unstale":"2024-03-06T07:20:07.286730+0000","last_undegraded":"2024-03-06T07:20:07.286730+0000","last_fullsized":"2024-03-06T07:20:07.286730+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:06.316192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279554+0000","last_change":"2024-03-06T07:19:54.058718+0000","last_active":"2024-03-06T07:20:07.279554+0000","last_peered":"2024-03-06T07:20:07.279554+0000","last_clean":"2024-03-06T07:20:07.279554+0000","last_became_active":"2024-03-06T07:19:54.058415+0000","last_became_peered":"2024-03-06T07:19:54.058415+0000","last_unstale":"2024-03-06T07:20:07.279554+0000","last_undegraded":"2024-03-06T07:20:07.279554+0000","last_fullsized":"2024-03-06T07:20:07.279554+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272518+0000","last_change":"2024-03-06T07:19:53.046555+0000","last_active":"2024-03-06T07:20:07.272518+0000","last_peered":"2024-03-06T07:20:07.272518+0000","last_clean":"2024-03-06T07:20:07.272518+0000","last_became_active":"2024-03-06T07:19:53.046298+0000","last_became_peered":"2024-03-06T07:19:53.046298+0000","last_unstale":"2024-03-06T07:20:07.272518+0000","last_undegraded":"2024-03-06T07:20:07.272518+0000","last_fullsized":"2024-03-06T07:20:07.272518+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.243021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.280843+0000","last_change":"2024-03-06T07:20:07.280843+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.280843+0000","last_undegraded":"2024-03-06T07:20:07.280843+0000","last_fullsized":"2024-03-06T07:20:07.280843+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286818+0000","last_change":"2024-03-06T07:20:07.285895+0000","last_active":"2024-03-06T07:20:07.286818+0000","last_peered":"2024-03-06T07:20:07.286818+0000","last_clean":"2024-03-06T07:20:07.286818+0000","last_became_active":"2024-03-06T07:20:07.285614+0000","last_became_peered":"2024-03-06T07:20:07.285614+0000","last_unstale":"2024-03-06T07:20:07.286818+0000","last_undegraded":"2024-03-06T07:20:07.286818+0000","last_fullsized":"2024-03-06T07:20:07.286818+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:00.451319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272257+0000","last_change":"2024-03-06T07:19:54.047654+0000","last_active":"2024-03-06T07:20:07.272257+0000","last_peered":"2024-03-06T07:20:07.272257+0000","last_clean":"2024-03-06T07:20:07.272257+0000","last_became_active":"2024-03-06T07:19:54.047339+0000","last_became_peered":"2024-03-06T07:19:54.047339+0000","last_unstale":"2024-03-06T07:20:07.272257+0000","last_undegraded":"2024-03-06T07:20:07.272257+0000","last_fullsized":"2024-03-06T07:20:07.272257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:35.272829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275306+0000","last_change":"2024-03-06T07:19:53.040132+0000","last_active":"2024-03-06T07:20:07.275306+0000","last_peered":"2024-03-06T07:20:07.275306+0000","last_clean":"2024-03-06T07:20:07.275306+0000","last_became_active":"2024-03-06T07:19:53.039809+0000","last_became_peered":"2024-03-06T07:19:53.039809+0000","last_unstale":"2024-03-06T07:20:07.275306+0000","last_undegraded":"2024-03-06T07:20:07.275306+0000","last_fullsized":"2024-03-06T07:20:07.275306+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272239+0000","last_change":"2024-03-06T07:20:07.272239+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272239+0000","last_undegraded":"2024-03-06T07:20:07.272239+0000","last_fullsized":"2024-03-06T07:20:07.272239+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289344+0000","last_change":"2024-03-06T07:20:07.288210+0000","last_active":"2024-03-06T07:20:07.289344+0000","last_peered":"2024-03-06T07:20:07.289344+0000","last_clean":"2024-03-06T07:20:07.289344+0000","last_became_active":"2024-03-06T07:20:07.287885+0000","last_became_peered":"2024-03-06T07:20:07.287885+0000","last_unstale":"2024-03-06T07:20:07.289344+0000","last_undegraded":"2024-03-06T07:20:07.289344+0000","last_fullsized":"2024-03-06T07:20:07.289344+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:25.080610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.273595+0000","last_change":"2024-03-06T07:19:54.050847+0000","last_active":"2024-03-06T07:20:07.273595+0000","last_peered":"2024-03-06T07:20:07.273595+0000","last_clean":"2024-03-06T07:20:07.273595+0000","last_became_active":"2024-03-06T07:19:54.050378+0000","last_became_peered":"2024-03-06T07:19:54.050378+0000","last_unstale":"2024-03-06T07:20:07.273595+0000","last_undegraded":"2024-03-06T07:20:07.273595+0000","last_fullsized":"2024-03-06T07:20:07.273595+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281018+0000","last_change":"2024-03-06T07:19:53.045139+0000","last_active":"2024-03-06T07:20:07.281018+0000","last_peered":"2024-03-06T07:20:07.281018+0000","last_clean":"2024-03-06T07:20:07.281018+0000","last_became_active":"2024-03-06T07:19:53.044903+0000","last_became_peered":"2024-03-06T07:19:53.044903+0000","last_unstale":"2024-03-06T07:20:07.281018+0000","last_undegraded":"2024-03-06T07:20:07.281018+0000","last_fullsized":"2024-03-06T07:20:07.281018+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:56.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272163+0000","last_change":"2024-03-06T07:20:07.272163+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272163+0000","last_undegraded":"2024-03-06T07:20:07.272163+0000","last_fullsized":"2024-03-06T07:20:07.272163+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.290186+0000","last_change":"2024-03-06T07:20:07.289518+0000","last_active":"2024-03-06T07:20:07.290186+0000","last_peered":"2024-03-06T07:20:07.290186+0000","last_clean":"2024-03-06T07:20:07.290186+0000","last_became_active":"2024-03-06T07:20:07.289213+0000","last_became_peered":"2024-03-06T07:20:07.289213+0000","last_unstale":"2024-03-06T07:20:07.290186+0000","last_undegraded":"2024-03-06T07:20:07.290186+0000","last_fullsized":"2024-03-06T07:20:07.290186+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.514155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271904+0000","last_change":"2024-03-06T07:19:54.059322+0000","last_active":"2024-03-06T07:20:07.271904+0000","last_peered":"2024-03-06T07:20:07.271904+0000","last_clean":"2024-03-06T07:20:07.271904+0000","last_became_active":"2024-03-06T07:19:54.059092+0000","last_became_peered":"2024-03-06T07:19:54.059092+0000","last_unstale":"2024-03-06T07:20:07.271904+0000","last_undegraded":"2024-03-06T07:20:07.271904+0000","last_fullsized":"2024-03-06T07:20:07.271904+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":26,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275158+0000","last_change":"2024-03-06T07:19:53.046815+0000","last_active":"2024-03-06T07:20:07.275158+0000","last_peered":"2024-03-06T07:20:07.275158+0000","last_clean":"2024-03-06T07:20:07.275158+0000","last_became_active":"2024-03-06T07:19:53.043687+0000","last_became_peered":"2024-03-06T07:19:53.043687+0000","last_unstale":"2024-03-06T07:20:07.275158+0000","last_undegraded":"2024-03-06T07:20:07.275158+0000","last_fullsized":"2024-03-06T07:20:07.275158+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:44.465449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269497+0000","last_change":"2024-03-06T07:20:07.269497+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269497+0000","last_undegraded":"2024-03-06T07:20:07.269497+0000","last_fullsized":"2024-03-06T07:20:07.269497+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285213+0000","last_change":"2024-03-06T07:20:07.283273+0000","last_active":"2024-03-06T07:20:07.285213+0000","last_peered":"2024-03-06T07:20:07.285213+0000","last_clean":"2024-03-06T07:20:07.285213+0000","last_became_active":"2024-03-06T07:20:07.282989+0000","last_became_peered":"2024-03-06T07:20:07.282989+0000","last_unstale":"2024-03-06T07:20:07.285213+0000","last_undegraded":"2024-03-06T07:20:07.285213+0000","last_fullsized":"2024-03-06T07:20:07.285213+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:08.705777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275055+0000","last_change":"2024-03-06T07:19:54.057521+0000","last_active":"2024-03-06T07:20:07.275055+0000","last_peered":"2024-03-06T07:20:07.275055+0000","last_clean":"2024-03-06T07:20:07.275055+0000","last_became_active":"2024-03-06T07:19:54.057264+0000","last_became_peered":"2024-03-06T07:19:54.057264+0000","last_unstale":"2024-03-06T07:20:07.275055+0000","last_undegraded":"2024-03-06T07:20:07.275055+0000","last_fullsized":"2024-03-06T07:20:07.275055+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272172+0000","last_change":"2024-03-06T07:19:53.051189+0000","last_active":"2024-03-06T07:20:07.272172+0000","last_peered":"2024-03-06T07:20:07.272172+0000","last_clean":"2024-03-06T07:20:07.272172+0000","last_became_active":"2024-03-06T07:19:53.050897+0000","last_became_peered":"2024-03-06T07:19:53.050897+0000","last_unstale":"2024-03-06T07:20:07.272172+0000","last_undegraded":"2024-03-06T07:20:07.272172+0000","last_fullsized":"2024-03-06T07:20:07.272172+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274169+0000","last_change":"2024-03-06T07:20:07.274169+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274169+0000","last_undegraded":"2024-03-06T07:20:07.274169+0000","last_fullsized":"2024-03-06T07:20:07.274169+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284678+0000","last_change":"2024-03-06T07:20:07.283920+0000","last_active":"2024-03-06T07:20:07.284678+0000","last_peered":"2024-03-06T07:20:07.284678+0000","last_clean":"2024-03-06T07:20:07.284678+0000","last_became_active":"2024-03-06T07:20:07.283631+0000","last_became_peered":"2024-03-06T07:20:07.283631+0000","last_unstale":"2024-03-06T07:20:07.284678+0000","last_undegraded":"2024-03-06T07:20:07.284678+0000","last_fullsized":"2024-03-06T07:20:07.284678+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:33.711210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266154+0000","last_change":"2024-03-06T07:19:54.052438+0000","last_active":"2024-03-06T07:20:07.266154+0000","last_peered":"2024-03-06T07:20:07.266154+0000","last_clean":"2024-03-06T07:20:07.266154+0000","last_became_active":"2024-03-06T07:19:54.052185+0000","last_became_peered":"2024-03-06T07:19:54.052185+0000","last_unstale":"2024-03-06T07:20:07.266154+0000","last_undegraded":"2024-03-06T07:20:07.266154+0000","last_fullsized":"2024-03-06T07:20:07.266154+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276246+0000","last_change":"2024-03-06T07:19:53.046903+0000","last_active":"2024-03-06T07:20:07.276246+0000","last_peered":"2024-03-06T07:20:07.276246+0000","last_clean":"2024-03-06T07:20:07.276246+0000","last_became_active":"2024-03-06T07:19:53.046639+0000","last_became_peered":"2024-03-06T07:19:53.046639+0000","last_unstale":"2024-03-06T07:20:07.276246+0000","last_undegraded":"2024-03-06T07:20:07.276246+0000","last_fullsized":"2024-03-06T07:20:07.276246+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:23.209022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.273354+0000","last_change":"2024-03-06T07:20:07.273354+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.273354+0000","last_undegraded":"2024-03-06T07:20:07.273354+0000","last_fullsized":"2024-03-06T07:20:07.273354+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289363+0000","last_change":"2024-03-06T07:20:07.288781+0000","last_active":"2024-03-06T07:20:07.289363+0000","last_peered":"2024-03-06T07:20:07.289363+0000","last_clean":"2024-03-06T07:20:07.289363+0000","last_became_active":"2024-03-06T07:20:07.288548+0000","last_became_peered":"2024-03-06T07:20:07.288548+0000","last_unstale":"2024-03-06T07:20:07.289363+0000","last_undegraded":"2024-03-06T07:20:07.289363+0000","last_fullsized":"2024-03-06T07:20:07.289363+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274167+0000","last_change":"2024-03-06T07:19:54.053795+0000","last_active":"2024-03-06T07:20:07.274167+0000","last_peered":"2024-03-06T07:20:07.274167+0000","last_clean":"2024-03-06T07:20:07.274167+0000","last_became_active":"2024-03-06T07:19:54.053519+0000","last_became_peered":"2024-03-06T07:19:54.053519+0000","last_unstale":"2024-03-06T07:20:07.274167+0000","last_undegraded":"2024-03-06T07:20:07.274167+0000","last_fullsized":"2024-03-06T07:20:07.274167+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:20.250283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275485+0000","last_change":"2024-03-06T07:19:53.046151+0000","last_active":"2024-03-06T07:20:07.275485+0000","last_peered":"2024-03-06T07:20:07.275485+0000","last_clean":"2024-03-06T07:20:07.275485+0000","last_became_active":"2024-03-06T07:19:53.045797+0000","last_became_peered":"2024-03-06T07:19:53.045797+0000","last_unstale":"2024-03-06T07:20:07.275485+0000","last_undegraded":"2024-03-06T07:20:07.275485+0000","last_fullsized":"2024-03-06T07:20:07.275485+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271460+0000","last_change":"2024-03-06T07:20:07.271460+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271460+0000","last_undegraded":"2024-03-06T07:20:07.271460+0000","last_fullsized":"2024-03-06T07:20:07.271460+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.285658+0000","last_change":"2024-03-06T07:20:07.284617+0000","last_active":"2024-03-06T07:20:07.285658+0000","last_peered":"2024-03-06T07:20:07.285658+0000","last_clean":"2024-03-06T07:20:07.285658+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:07.285658+0000","last_undegraded":"2024-03-06T07:20:07.285658+0000","last_fullsized":"2024-03-06T07:20:07.285658+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:29.205056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266558+0000","last_change":"2024-03-06T07:19:54.055324+0000","last_active":"2024-03-06T07:20:07.266558+0000","last_peered":"2024-03-06T07:20:07.266558+0000","last_clean":"2024-03-06T07:20:07.266558+0000","last_became_active":"2024-03-06T07:19:54.055081+0000","last_became_peered":"2024-03-06T07:19:54.055081+0000","last_unstale":"2024-03-06T07:20:07.266558+0000","last_undegraded":"2024-03-06T07:20:07.266558+0000","last_fullsized":"2024-03-06T07:20:07.266558+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:57.518725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275654+0000","last_change":"2024-03-06T07:19:53.048898+0000","last_active":"2024-03-06T07:20:07.275654+0000","last_peered":"2024-03-06T07:20:07.275654+0000","last_clean":"2024-03-06T07:20:07.275654+0000","last_became_active":"2024-03-06T07:19:53.048613+0000","last_became_peered":"2024-03-06T07:19:53.048613+0000","last_unstale":"2024-03-06T07:20:07.275654+0000","last_undegraded":"2024-03-06T07:20:07.275654+0000","last_fullsized":"2024-03-06T07:20:07.275654+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.277278+0000","last_change":"2024-03-06T07:20:07.277278+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.277278+0000","last_undegraded":"2024-03-06T07:20:07.277278+0000","last_fullsized":"2024-03-06T07:20:07.277278+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286112+0000","last_change":"2024-03-06T07:20:07.285007+0000","last_active":"2024-03-06T07:20:07.286112+0000","last_peered":"2024-03-06T07:20:07.286112+0000","last_clean":"2024-03-06T07:20:07.286112+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:07.286112+0000","last_undegraded":"2024-03-06T07:20:07.286112+0000","last_fullsized":"2024-03-06T07:20:07.286112+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279669+0000","last_change":"2024-03-06T07:19:54.056310+0000","last_active":"2024-03-06T07:20:07.279669+0000","last_peered":"2024-03-06T07:20:07.279669+0000","last_clean":"2024-03-06T07:20:07.279669+0000","last_became_active":"2024-03-06T07:19:54.056039+0000","last_became_peered":"2024-03-06T07:19:54.056039+0000","last_unstale":"2024-03-06T07:20:07.279669+0000","last_undegraded":"2024-03-06T07:20:07.279669+0000","last_fullsized":"2024-03-06T07:20:07.279669+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266526+0000","last_change":"2024-03-06T07:19:53.043949+0000","last_active":"2024-03-06T07:20:07.266526+0000","last_peered":"2024-03-06T07:20:07.266526+0000","last_clean":"2024-03-06T07:20:07.266526+0000","last_became_active":"2024-03-06T07:19:53.043639+0000","last_became_peered":"2024-03-06T07:19:53.043639+0000","last_unstale":"2024-03-06T07:20:07.266526+0000","last_undegraded":"2024-03-06T07:20:07.266526+0000","last_fullsized":"2024-03-06T07:20:07.266526+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271451+0000","last_change":"2024-03-06T07:20:07.271451+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271451+0000","last_undegraded":"2024-03-06T07:20:07.271451+0000","last_fullsized":"2024-03-06T07:20:07.271451+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284102+0000","last_change":"2024-03-06T07:20:07.279146+0000","last_active":"2024-03-06T07:20:07.284102+0000","last_peered":"2024-03-06T07:20:07.284102+0000","last_clean":"2024-03-06T07:20:07.284102+0000","last_became_active":"2024-03-06T07:20:07.278872+0000","last_became_peered":"2024-03-06T07:20:07.278872+0000","last_unstale":"2024-03-06T07:20:07.284102+0000","last_undegraded":"2024-03-06T07:20:07.284102+0000","last_fullsized":"2024-03-06T07:20:07.284102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:34.974180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275593+0000","last_change":"2024-03-06T07:19:54.040887+0000","last_active":"2024-03-06T07:20:07.275593+0000","last_peered":"2024-03-06T07:20:07.275593+0000","last_clean":"2024-03-06T07:20:07.275593+0000","last_became_active":"2024-03-06T07:19:54.040632+0000","last_became_peered":"2024-03-06T07:19:54.040632+0000","last_unstale":"2024-03-06T07:20:07.275593+0000","last_undegraded":"2024-03-06T07:20:07.275593+0000","last_fullsized":"2024-03-06T07:20:07.275593+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276010+0000","last_change":"2024-03-06T07:19:53.044330+0000","last_active":"2024-03-06T07:20:07.276010+0000","last_peered":"2024-03-06T07:20:07.276010+0000","last_clean":"2024-03-06T07:20:07.276010+0000","last_became_active":"2024-03-06T07:19:53.044066+0000","last_became_peered":"2024-03-06T07:19:53.044066+0000","last_unstale":"2024-03-06T07:20:07.276010+0000","last_undegraded":"2024-03-06T07:20:07.276010+0000","last_fullsized":"2024-03-06T07:20:07.276010+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:19.476961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271863+0000","last_change":"2024-03-06T07:20:07.271863+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271863+0000","last_undegraded":"2024-03-06T07:20:07.271863+0000","last_fullsized":"2024-03-06T07:20:07.271863+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289018+0000","last_change":"2024-03-06T07:20:07.287646+0000","last_active":"2024-03-06T07:20:07.289018+0000","last_peered":"2024-03-06T07:20:07.289018+0000","last_clean":"2024-03-06T07:20:07.289018+0000","last_became_active":"2024-03-06T07:20:07.287378+0000","last_became_peered":"2024-03-06T07:20:07.287378+0000","last_unstale":"2024-03-06T07:20:07.289018+0000","last_undegraded":"2024-03-06T07:20:07.289018+0000","last_fullsized":"2024-03-06T07:20:07.289018+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:03.735494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272180+0000","last_change":"2024-03-06T07:19:54.049940+0000","last_active":"2024-03-06T07:20:07.272180+0000","last_peered":"2024-03-06T07:20:07.272180+0000","last_clean":"2024-03-06T07:20:07.272180+0000","last_became_active":"2024-03-06T07:19:54.049481+0000","last_became_peered":"2024-03-06T07:19:54.049481+0000","last_unstale":"2024-03-06T07:20:07.272180+0000","last_undegraded":"2024-03-06T07:20:07.272180+0000","last_fullsized":"2024-03-06T07:20:07.272180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.270036+0000","last_change":"2024-03-06T07:19:53.048472+0000","last_active":"2024-03-06T07:20:07.270036+0000","last_peered":"2024-03-06T07:20:07.270036+0000","last_clean":"2024-03-06T07:20:07.270036+0000","last_became_active":"2024-03-06T07:19:53.048213+0000","last_became_peered":"2024-03-06T07:19:53.048213+0000","last_unstale":"2024-03-06T07:20:07.270036+0000","last_undegraded":"2024-03-06T07:20:07.270036+0000","last_fullsized":"2024-03-06T07:20:07.270036+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:19.721078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.274293+0000","last_change":"2024-03-06T07:20:07.274293+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.274293+0000","last_undegraded":"2024-03-06T07:20:07.274293+0000","last_fullsized":"2024-03-06T07:20:07.274293+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.286969+0000","last_change":"2024-03-06T07:20:07.283238+0000","last_active":"2024-03-06T07:20:07.286969+0000","last_peered":"2024-03-06T07:20:07.286969+0000","last_clean":"2024-03-06T07:20:07.286969+0000","last_became_active":"2024-03-06T07:20:07.282996+0000","last_became_peered":"2024-03-06T07:20:07.282996+0000","last_unstale":"2024-03-06T07:20:07.286969+0000","last_undegraded":"2024-03-06T07:20:07.286969+0000","last_fullsized":"2024-03-06T07:20:07.286969+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.712830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.267007+0000","last_change":"2024-03-06T07:19:54.042432+0000","last_active":"2024-03-06T07:20:07.267007+0000","last_peered":"2024-03-06T07:20:07.267007+0000","last_clean":"2024-03-06T07:20:07.267007+0000","last_became_active":"2024-03-06T07:19:54.042166+0000","last_became_peered":"2024-03-06T07:19:54.042166+0000","last_unstale":"2024-03-06T07:20:07.267007+0000","last_undegraded":"2024-03-06T07:20:07.267007+0000","last_fullsized":"2024-03-06T07:20:07.267007+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.231274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275074+0000","last_change":"2024-03-06T07:19:53.044950+0000","last_active":"2024-03-06T07:20:07.275074+0000","last_peered":"2024-03-06T07:20:07.275074+0000","last_clean":"2024-03-06T07:20:07.275074+0000","last_became_active":"2024-03-06T07:19:53.044630+0000","last_became_peered":"2024-03-06T07:19:53.044630+0000","last_unstale":"2024-03-06T07:20:07.275074+0000","last_undegraded":"2024-03-06T07:20:07.275074+0000","last_fullsized":"2024-03-06T07:20:07.275074+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:05.924599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272790+0000","last_change":"2024-03-06T07:20:07.272790+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272790+0000","last_undegraded":"2024-03-06T07:20:07.272790+0000","last_fullsized":"2024-03-06T07:20:07.272790+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287689+0000","last_change":"2024-03-06T07:20:07.287051+0000","last_active":"2024-03-06T07:20:07.287689+0000","last_peered":"2024-03-06T07:20:07.287689+0000","last_clean":"2024-03-06T07:20:07.287689+0000","last_became_active":"2024-03-06T07:20:07.286569+0000","last_became_peered":"2024-03-06T07:20:07.286569+0000","last_unstale":"2024-03-06T07:20:07.287689+0000","last_undegraded":"2024-03-06T07:20:07.287689+0000","last_fullsized":"2024-03-06T07:20:07.287689+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:17.755219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279576+0000","last_change":"2024-03-06T07:19:54.059591+0000","last_active":"2024-03-06T07:20:07.279576+0000","last_peered":"2024-03-06T07:20:07.279576+0000","last_clean":"2024-03-06T07:20:07.279576+0000","last_became_active":"2024-03-06T07:19:54.058814+0000","last_became_peered":"2024-03-06T07:19:54.058814+0000","last_unstale":"2024-03-06T07:20:07.279576+0000","last_undegraded":"2024-03-06T07:20:07.279576+0000","last_fullsized":"2024-03-06T07:20:07.279576+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275779+0000","last_change":"2024-03-06T07:19:53.040833+0000","last_active":"2024-03-06T07:20:07.275779+0000","last_peered":"2024-03-06T07:20:07.275779+0000","last_clean":"2024-03-06T07:20:07.275779+0000","last_became_active":"2024-03-06T07:19:53.040561+0000","last_became_peered":"2024-03-06T07:19:53.040561+0000","last_unstale":"2024-03-06T07:20:07.275779+0000","last_undegraded":"2024-03-06T07:20:07.275779+0000","last_fullsized":"2024-03-06T07:20:07.275779+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:12.315549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.269859+0000","last_change":"2024-03-06T07:20:07.269859+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.269859+0000","last_undegraded":"2024-03-06T07:20:07.269859+0000","last_fullsized":"2024-03-06T07:20:07.269859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281857+0000","last_change":"2024-03-06T07:20:07.280861+0000","last_active":"2024-03-06T07:20:07.281857+0000","last_peered":"2024-03-06T07:20:07.281857+0000","last_clean":"2024-03-06T07:20:07.281857+0000","last_became_active":"2024-03-06T07:20:07.280598+0000","last_became_peered":"2024-03-06T07:20:07.280598+0000","last_unstale":"2024-03-06T07:20:07.281857+0000","last_undegraded":"2024-03-06T07:20:07.281857+0000","last_fullsized":"2024-03-06T07:20:07.281857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.266454+0000","last_change":"2024-03-06T07:19:54.042216+0000","last_active":"2024-03-06T07:20:07.266454+0000","last_peered":"2024-03-06T07:20:07.266454+0000","last_clean":"2024-03-06T07:20:07.266454+0000","last_became_active":"2024-03-06T07:19:54.041923+0000","last_became_peered":"2024-03-06T07:19:54.041923+0000","last_unstale":"2024-03-06T07:20:07.266454+0000","last_undegraded":"2024-03-06T07:20:07.266454+0000","last_fullsized":"2024-03-06T07:20:07.266454+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.281579+0000","last_change":"2024-03-06T07:19:53.047959+0000","last_active":"2024-03-06T07:20:07.281579+0000","last_peered":"2024-03-06T07:20:07.281579+0000","last_clean":"2024-03-06T07:20:07.281579+0000","last_became_active":"2024-03-06T07:19:53.047680+0000","last_became_peered":"2024-03-06T07:19:53.047680+0000","last_unstale":"2024-03-06T07:20:07.281579+0000","last_undegraded":"2024-03-06T07:20:07.281579+0000","last_fullsized":"2024-03-06T07:20:07.281579+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:38.499903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.276715+0000","last_change":"2024-03-06T07:20:07.276715+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.276715+0000","last_undegraded":"2024-03-06T07:20:07.276715+0000","last_fullsized":"2024-03-06T07:20:07.276715+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.287826+0000","last_change":"2024-03-06T07:20:07.284739+0000","last_active":"2024-03-06T07:20:07.287826+0000","last_peered":"2024-03-06T07:20:07.287826+0000","last_clean":"2024-03-06T07:20:07.287826+0000","last_became_active":"2024-03-06T07:20:07.284477+0000","last_became_peered":"2024-03-06T07:20:07.284477+0000","last_unstale":"2024-03-06T07:20:07.287826+0000","last_undegraded":"2024-03-06T07:20:07.287826+0000","last_fullsized":"2024-03-06T07:20:07.287826+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:47.605321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274844+0000","last_change":"2024-03-06T07:19:54.043516+0000","last_active":"2024-03-06T07:20:07.274844+0000","last_peered":"2024-03-06T07:20:07.274844+0000","last_clean":"2024-03-06T07:20:07.274844+0000","last_became_active":"2024-03-06T07:19:54.041651+0000","last_became_peered":"2024-03-06T07:19:54.041651+0000","last_unstale":"2024-03-06T07:20:07.274844+0000","last_undegraded":"2024-03-06T07:20:07.274844+0000","last_fullsized":"2024-03-06T07:20:07.274844+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:04.528232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272322+0000","last_change":"2024-03-06T07:19:53.048966+0000","last_active":"2024-03-06T07:20:07.272322+0000","last_peered":"2024-03-06T07:20:07.272322+0000","last_clean":"2024-03-06T07:20:07.272322+0000","last_became_active":"2024-03-06T07:19:53.047988+0000","last_became_peered":"2024-03-06T07:19:53.047988+0000","last_unstale":"2024-03-06T07:20:07.272322+0000","last_undegraded":"2024-03-06T07:20:07.272322+0000","last_fullsized":"2024-03-06T07:20:07.272322+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:49.519186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272529+0000","last_change":"2024-03-06T07:20:07.272529+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272529+0000","last_undegraded":"2024-03-06T07:20:07.272529+0000","last_fullsized":"2024-03-06T07:20:07.272529+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289051+0000","last_change":"2024-03-06T07:20:07.288004+0000","last_active":"2024-03-06T07:20:07.289051+0000","last_peered":"2024-03-06T07:20:07.289051+0000","last_clean":"2024-03-06T07:20:07.289051+0000","last_became_active":"2024-03-06T07:20:07.287745+0000","last_became_peered":"2024-03-06T07:20:07.287745+0000","last_unstale":"2024-03-06T07:20:07.289051+0000","last_undegraded":"2024-03-06T07:20:07.289051+0000","last_fullsized":"2024-03-06T07:20:07.289051+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:47.605321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.272236+0000","last_change":"2024-03-06T07:19:54.062368+0000","last_active":"2024-03-06T07:20:07.272236+0000","last_peered":"2024-03-06T07:20:07.272236+0000","last_clean":"2024-03-06T07:20:07.272236+0000","last_became_active":"2024-03-06T07:19:54.060740+0000","last_became_peered":"2024-03-06T07:19:54.060740+0000","last_unstale":"2024-03-06T07:20:07.272236+0000","last_undegraded":"2024-03-06T07:20:07.272236+0000","last_fullsized":"2024-03-06T07:20:07.272236+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274942+0000","last_change":"2024-03-06T07:19:53.044944+0000","last_active":"2024-03-06T07:20:07.274942+0000","last_peered":"2024-03-06T07:20:07.274942+0000","last_clean":"2024-03-06T07:20:07.274942+0000","last_became_active":"2024-03-06T07:19:53.038900+0000","last_became_peered":"2024-03-06T07:19:53.038900+0000","last_unstale":"2024-03-06T07:20:07.274942+0000","last_undegraded":"2024-03-06T07:20:07.274942+0000","last_fullsized":"2024-03-06T07:20:07.274942+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:59.062789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271846+0000","last_change":"2024-03-06T07:20:07.271846+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271846+0000","last_undegraded":"2024-03-06T07:20:07.271846+0000","last_fullsized":"2024-03-06T07:20:07.271846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.284471+0000","last_change":"2024-03-06T07:20:07.281920+0000","last_active":"2024-03-06T07:20:07.284471+0000","last_peered":"2024-03-06T07:20:07.284471+0000","last_clean":"2024-03-06T07:20:07.284471+0000","last_became_active":"2024-03-06T07:20:07.281654+0000","last_became_peered":"2024-03-06T07:20:07.281654+0000","last_unstale":"2024-03-06T07:20:07.284471+0000","last_undegraded":"2024-03-06T07:20:07.284471+0000","last_fullsized":"2024-03-06T07:20:07.284471+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:44.082119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279797+0000","last_change":"2024-03-06T07:19:54.059347+0000","last_active":"2024-03-06T07:20:07.279797+0000","last_peered":"2024-03-06T07:20:07.279797+0000","last_clean":"2024-03-06T07:20:07.279797+0000","last_became_active":"2024-03-06T07:19:54.059083+0000","last_became_peered":"2024-03-06T07:19:54.059083+0000","last_unstale":"2024-03-06T07:20:07.279797+0000","last_undegraded":"2024-03-06T07:20:07.279797+0000","last_fullsized":"2024-03-06T07:20:07.279797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.276107+0000","last_change":"2024-03-06T07:19:53.048645+0000","last_active":"2024-03-06T07:20:07.276107+0000","last_peered":"2024-03-06T07:20:07.276107+0000","last_clean":"2024-03-06T07:20:07.276107+0000","last_became_active":"2024-03-06T07:19:53.048406+0000","last_became_peered":"2024-03-06T07:19:53.048406+0000","last_unstale":"2024-03-06T07:20:07.276107+0000","last_undegraded":"2024-03-06T07:20:07.276107+0000","last_fullsized":"2024-03-06T07:20:07.276107+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:10.846361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.278072+0000","last_change":"2024-03-06T07:20:07.278072+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.278072+0000","last_undegraded":"2024-03-06T07:20:07.278072+0000","last_fullsized":"2024-03-06T07:20:07.278072+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.288382+0000","last_change":"2024-03-06T07:20:07.287091+0000","last_active":"2024-03-06T07:20:07.288382+0000","last_peered":"2024-03-06T07:20:07.288382+0000","last_clean":"2024-03-06T07:20:07.288382+0000","last_became_active":"2024-03-06T07:20:07.286303+0000","last_became_peered":"2024-03-06T07:20:07.286303+0000","last_unstale":"2024-03-06T07:20:07.288382+0000","last_undegraded":"2024-03-06T07:20:07.288382+0000","last_fullsized":"2024-03-06T07:20:07.288382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:02.845608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271636+0000","last_change":"2024-03-06T07:19:54.049719+0000","last_active":"2024-03-06T07:20:07.271636+0000","last_peered":"2024-03-06T07:20:07.271636+0000","last_clean":"2024-03-06T07:20:07.271636+0000","last_became_active":"2024-03-06T07:19:54.049222+0000","last_became_peered":"2024-03-06T07:19:54.049222+0000","last_unstale":"2024-03-06T07:20:07.271636+0000","last_undegraded":"2024-03-06T07:20:07.271636+0000","last_fullsized":"2024-03-06T07:20:07.271636+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274213+0000","last_change":"2024-03-06T07:19:53.045046+0000","last_active":"2024-03-06T07:20:07.274213+0000","last_peered":"2024-03-06T07:20:07.274213+0000","last_clean":"2024-03-06T07:20:07.274213+0000","last_became_active":"2024-03-06T07:19:53.044751+0000","last_became_peered":"2024-03-06T07:19:53.044751+0000","last_unstale":"2024-03-06T07:20:07.274213+0000","last_undegraded":"2024-03-06T07:20:07.274213+0000","last_fullsized":"2024-03-06T07:20:07.274213+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.272523+0000","last_change":"2024-03-06T07:20:07.272523+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.272523+0000","last_undegraded":"2024-03-06T07:20:07.272523+0000","last_fullsized":"2024-03-06T07:20:07.272523+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.289046+0000","last_change":"2024-03-06T07:20:07.287811+0000","last_active":"2024-03-06T07:20:07.289046+0000","last_peered":"2024-03-06T07:20:07.289046+0000","last_clean":"2024-03-06T07:20:07.289046+0000","last_became_active":"2024-03-06T07:20:07.287540+0000","last_became_peered":"2024-03-06T07:20:07.287540+0000","last_unstale":"2024-03-06T07:20:07.289046+0000","last_undegraded":"2024-03-06T07:20:07.289046+0000","last_fullsized":"2024-03-06T07:20:07.289046+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.712830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.274628+0000","last_change":"2024-03-06T07:19:54.039796+0000","last_active":"2024-03-06T07:20:07.274628+0000","last_peered":"2024-03-06T07:20:07.274628+0000","last_clean":"2024-03-06T07:20:07.274628+0000","last_became_active":"2024-03-06T07:19:54.039527+0000","last_became_peered":"2024-03-06T07:19:54.039527+0000","last_unstale":"2024-03-06T07:20:07.274628+0000","last_undegraded":"2024-03-06T07:20:07.274628+0000","last_fullsized":"2024-03-06T07:20:07.274628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:49.114747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.279603+0000","last_change":"2024-03-06T07:19:53.043599+0000","last_active":"2024-03-06T07:20:07.279603+0000","last_peered":"2024-03-06T07:20:07.279603+0000","last_clean":"2024-03-06T07:20:07.279603+0000","last_became_active":"2024-03-06T07:19:53.043294+0000","last_became_peered":"2024-03-06T07:19:53.043294+0000","last_unstale":"2024-03-06T07:20:07.279603+0000","last_undegraded":"2024-03-06T07:20:07.279603+0000","last_fullsized":"2024-03-06T07:20:07.279603+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:56.511464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":54,"state":"creating+peering","last_fresh":"2024-03-06T07:20:07.271196+0000","last_change":"2024-03-06T07:20:07.271196+0000","last_active":"2024-03-06T07:20:07.257688+0000","last_peered":"2024-03-06T07:20:07.257688+0000","last_clean":"2024-03-06T07:20:07.257688+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:20:07.271196+0000","last_undegraded":"2024-03-06T07:20:07.271196+0000","last_fullsized":"2024-03-06T07:20:07.271196+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278877+0000","last_change":"2024-03-06T07:20:07.277710+0000","last_active":"2024-03-06T07:20:07.278877+0000","last_peered":"2024-03-06T07:20:07.278877+0000","last_clean":"2024-03-06T07:20:07.278877+0000","last_became_active":"2024-03-06T07:20:07.277346+0000","last_became_peered":"2024-03-06T07:20:07.277346+0000","last_unstale":"2024-03-06T07:20:07.278877+0000","last_undegraded":"2024-03-06T07:20:07.278877+0000","last_fullsized":"2024-03-06T07:20:07.278877+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:51.111934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.275167+0000","last_change":"2024-03-06T07:19:54.055814+0000","last_active":"2024-03-06T07:20:07.275167+0000","last_peered":"2024-03-06T07:20:07.275167+0000","last_clean":"2024-03-06T07:20:07.275167+0000","last_became_active":"2024-03-06T07:19:54.055480+0000","last_became_peered":"2024-03-06T07:19:54.055480+0000","last_unstale":"2024-03-06T07:20:07.275167+0000","last_undegraded":"2024-03-06T07:20:07.275167+0000","last_fullsized":"2024-03-06T07:20:07.275167+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:35.156134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.271765+0000","last_change":"2024-03-06T07:19:53.046120+0000","last_active":"2024-03-06T07:20:07.271765+0000","last_peered":"2024-03-06T07:20:07.271765+0000","last_clean":"2024-03-06T07:20:07.271765+0000","last_became_active":"2024-03-06T07:19:53.045816+0000","last_became_peered":"2024-03-06T07:19:53.045816+0000","last_unstale":"2024-03-06T07:20:07.271765+0000","last_undegraded":"2024-03-06T07:20:07.271765+0000","last_fullsized":"2024-03-06T07:20:07.271765+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:56.226679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":234,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-06T07:20:07.278863+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:20:07.278863+0000","last_peered":"2024-03-06T07:20:07.278863+0000","last_clean":"2024-03-06T07:20:07.278863+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:20:07.278863+0000","last_undegraded":"2024-03-06T07:20:07.278863+0000","last_fullsized":"2024-03-06T07:20:07.278863+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":704512,"data_stored":873646,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":283806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":283154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":283736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":647168,"data_stored":284788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":643072,"data_stored":284766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":283314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":873578,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:12.111 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3d is not active+clean 2024-03-06T07:20:12.111 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.276976+0000', 'last_change': '2024-03-06T07:20:07.276976+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.276976+0000', 'last_undegraded': '2024-03-06T07:20:07.276976+0000', 'last_fullsized': '2024-03-06T07:20:07.276976+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3c is not active+clean 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.270987+0000', 'last_change': '2024-03-06T07:20:07.270987+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.270987+0000', 'last_undegraded': '2024-03-06T07:20:07.270987+0000', 'last_fullsized': '2024-03-06T07:20:07.270987+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3f is not active+clean 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271068+0000', 'last_change': '2024-03-06T07:20:07.271068+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271068+0000', 'last_undegraded': '2024-03-06T07:20:07.271068+0000', 'last_fullsized': '2024-03-06T07:20:07.271068+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3e is not active+clean 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.270217+0000', 'last_change': '2024-03-06T07:20:07.270217+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.270217+0000', 'last_undegraded': '2024-03-06T07:20:07.270217+0000', 'last_fullsized': '2024-03-06T07:20:07.270217+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.39 is not active+clean 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.274065+0000', 'last_change': '2024-03-06T07:20:07.274065+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.274065+0000', 'last_undegraded': '2024-03-06T07:20:07.274065+0000', 'last_fullsized': '2024-03-06T07:20:07.274065+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.38 is not active+clean 2024-03-06T07:20:12.112 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277826+0000', 'last_change': '2024-03-06T07:20:07.277826+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277826+0000', 'last_undegraded': '2024-03-06T07:20:07.277826+0000', 'last_fullsized': '2024-03-06T07:20:07.277826+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3b is not active+clean 2024-03-06T07:20:12.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272580+0000', 'last_change': '2024-03-06T07:20:07.272580+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272580+0000', 'last_undegraded': '2024-03-06T07:20:07.272580+0000', 'last_fullsized': '2024-03-06T07:20:07.272580+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3a is not active+clean 2024-03-06T07:20:12.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271012+0000', 'last_change': '2024-03-06T07:20:07.271012+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271012+0000', 'last_undegraded': '2024-03-06T07:20:07.271012+0000', 'last_fullsized': '2024-03-06T07:20:07.271012+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:20:12.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.35 is not active+clean 2024-03-06T07:20:12.113 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271093+0000', 'last_change': '2024-03-06T07:20:07.271093+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271093+0000', 'last_undegraded': '2024-03-06T07:20:07.271093+0000', 'last_fullsized': '2024-03-06T07:20:07.271093+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.113 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.34 is not active+clean 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.278288+0000', 'last_change': '2024-03-06T07:20:07.278288+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.278288+0000', 'last_undegraded': '2024-03-06T07:20:07.278288+0000', 'last_fullsized': '2024-03-06T07:20:07.278288+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.37 is not active+clean 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.280462+0000', 'last_change': '2024-03-06T07:20:07.280462+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.280462+0000', 'last_undegraded': '2024-03-06T07:20:07.280462+0000', 'last_fullsized': '2024-03-06T07:20:07.280462+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.36 is not active+clean 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.278688+0000', 'last_change': '2024-03-06T07:20:07.278688+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.278688+0000', 'last_undegraded': '2024-03-06T07:20:07.278688+0000', 'last_fullsized': '2024-03-06T07:20:07.278688+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.31 is not active+clean 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277157+0000', 'last_change': '2024-03-06T07:20:07.277157+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277157+0000', 'last_undegraded': '2024-03-06T07:20:07.277157+0000', 'last_fullsized': '2024-03-06T07:20:07.277157+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.30 is not active+clean 2024-03-06T07:20:12.114 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.273689+0000', 'last_change': '2024-03-06T07:20:07.273689+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.273689+0000', 'last_undegraded': '2024-03-06T07:20:07.273689+0000', 'last_fullsized': '2024-03-06T07:20:07.273689+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.33 is not active+clean 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.276915+0000', 'last_change': '2024-03-06T07:20:07.276915+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.276915+0000', 'last_undegraded': '2024-03-06T07:20:07.276915+0000', 'last_fullsized': '2024-03-06T07:20:07.276915+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.32 is not active+clean 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.278431+0000', 'last_change': '2024-03-06T07:20:07.278431+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.278431+0000', 'last_undegraded': '2024-03-06T07:20:07.278431+0000', 'last_fullsized': '2024-03-06T07:20:07.278431+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2d is not active+clean 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272112+0000', 'last_change': '2024-03-06T07:20:07.272112+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272112+0000', 'last_undegraded': '2024-03-06T07:20:07.272112+0000', 'last_fullsized': '2024-03-06T07:20:07.272112+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2c is not active+clean 2024-03-06T07:20:12.115 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.279205+0000', 'last_change': '2024-03-06T07:20:07.279205+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.279205+0000', 'last_undegraded': '2024-03-06T07:20:07.279205+0000', 'last_fullsized': '2024-03-06T07:20:07.279205+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2f is not active+clean 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.270647+0000', 'last_change': '2024-03-06T07:20:07.270647+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.270647+0000', 'last_undegraded': '2024-03-06T07:20:07.270647+0000', 'last_fullsized': '2024-03-06T07:20:07.270647+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2e is not active+clean 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272968+0000', 'last_change': '2024-03-06T07:20:07.272968+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272968+0000', 'last_undegraded': '2024-03-06T07:20:07.272968+0000', 'last_fullsized': '2024-03-06T07:20:07.272968+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.29 is not active+clean 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277902+0000', 'last_change': '2024-03-06T07:20:07.277902+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277902+0000', 'last_undegraded': '2024-03-06T07:20:07.277902+0000', 'last_fullsized': '2024-03-06T07:20:07.277902+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.28 is not active+clean 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277366+0000', 'last_change': '2024-03-06T07:20:07.277366+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277366+0000', 'last_undegraded': '2024-03-06T07:20:07.277366+0000', 'last_fullsized': '2024-03-06T07:20:07.277366+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2b is not active+clean 2024-03-06T07:20:12.116 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.276244+0000', 'last_change': '2024-03-06T07:20:07.276244+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.276244+0000', 'last_undegraded': '2024-03-06T07:20:07.276244+0000', 'last_fullsized': '2024-03-06T07:20:07.276244+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.117 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2a is not active+clean 2024-03-06T07:20:12.117 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.278418+0000', 'last_change': '2024-03-06T07:20:07.278418+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.278418+0000', 'last_undegraded': '2024-03-06T07:20:07.278418+0000', 'last_fullsized': '2024-03-06T07:20:07.278418+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.25 is not active+clean 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272555+0000', 'last_change': '2024-03-06T07:20:07.272555+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272555+0000', 'last_undegraded': '2024-03-06T07:20:07.272555+0000', 'last_fullsized': '2024-03-06T07:20:07.272555+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.24 is not active+clean 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.270046+0000', 'last_change': '2024-03-06T07:20:07.270046+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.270046+0000', 'last_undegraded': '2024-03-06T07:20:07.270046+0000', 'last_fullsized': '2024-03-06T07:20:07.270046+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.27 is not active+clean 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.278821+0000', 'last_change': '2024-03-06T07:20:07.278821+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.278821+0000', 'last_undegraded': '2024-03-06T07:20:07.278821+0000', 'last_fullsized': '2024-03-06T07:20:07.278821+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.26 is not active+clean 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277996+0000', 'last_change': '2024-03-06T07:20:07.277996+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277996+0000', 'last_undegraded': '2024-03-06T07:20:07.277996+0000', 'last_fullsized': '2024-03-06T07:20:07.277996+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.21 is not active+clean 2024-03-06T07:20:12.118 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271067+0000', 'last_change': '2024-03-06T07:20:07.271067+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271067+0000', 'last_undegraded': '2024-03-06T07:20:07.271067+0000', 'last_fullsized': '2024-03-06T07:20:07.271067+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.20 is not active+clean 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.270342+0000', 'last_change': '2024-03-06T07:20:07.270342+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.270342+0000', 'last_undegraded': '2024-03-06T07:20:07.270342+0000', 'last_fullsized': '2024-03-06T07:20:07.270342+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.23 is not active+clean 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.276440+0000', 'last_change': '2024-03-06T07:20:07.276440+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.276440+0000', 'last_undegraded': '2024-03-06T07:20:07.276440+0000', 'last_fullsized': '2024-03-06T07:20:07.276440+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.22 is not active+clean 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277442+0000', 'last_change': '2024-03-06T07:20:07.277442+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277442+0000', 'last_undegraded': '2024-03-06T07:20:07.277442+0000', 'last_fullsized': '2024-03-06T07:20:07.277442+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.12 is not active+clean 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.273940+0000', 'last_change': '2024-03-06T07:20:07.273940+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.273940+0000', 'last_undegraded': '2024-03-06T07:20:07.273940+0000', 'last_fullsized': '2024-03-06T07:20:07.273940+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.13 is not active+clean 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.269519+0000', 'last_change': '2024-03-06T07:20:07.269519+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.269519+0000', 'last_undegraded': '2024-03-06T07:20:07.269519+0000', 'last_fullsized': '2024-03-06T07:20:07.269519+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.10 is not active+clean 2024-03-06T07:20:12.119 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271271+0000', 'last_change': '2024-03-06T07:20:07.271271+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271271+0000', 'last_undegraded': '2024-03-06T07:20:07.271271+0000', 'last_fullsized': '2024-03-06T07:20:07.271271+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.11 is not active+clean 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.276210+0000', 'last_change': '2024-03-06T07:20:07.276210+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.276210+0000', 'last_undegraded': '2024-03-06T07:20:07.276210+0000', 'last_fullsized': '2024-03-06T07:20:07.276210+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.16 is not active+clean 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271758+0000', 'last_change': '2024-03-06T07:20:07.271758+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271758+0000', 'last_undegraded': '2024-03-06T07:20:07.271758+0000', 'last_fullsized': '2024-03-06T07:20:07.271758+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.17 is not active+clean 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271661+0000', 'last_change': '2024-03-06T07:20:07.271661+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271661+0000', 'last_undegraded': '2024-03-06T07:20:07.271661+0000', 'last_fullsized': '2024-03-06T07:20:07.271661+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.14 is not active+clean 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.269496+0000', 'last_change': '2024-03-06T07:20:07.269496+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.269496+0000', 'last_undegraded': '2024-03-06T07:20:07.269496+0000', 'last_fullsized': '2024-03-06T07:20:07.269496+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.15 is not active+clean 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.276607+0000', 'last_change': '2024-03-06T07:20:07.276607+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.276607+0000', 'last_undegraded': '2024-03-06T07:20:07.276607+0000', 'last_fullsized': '2024-03-06T07:20:07.276607+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1a is not active+clean 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.273576+0000', 'last_change': '2024-03-06T07:20:07.273576+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.273576+0000', 'last_undegraded': '2024-03-06T07:20:07.273576+0000', 'last_fullsized': '2024-03-06T07:20:07.273576+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1b is not active+clean 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277242+0000', 'last_change': '2024-03-06T07:20:07.277242+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277242+0000', 'last_undegraded': '2024-03-06T07:20:07.277242+0000', 'last_fullsized': '2024-03-06T07:20:07.277242+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.18 is not active+clean 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.274188+0000', 'last_change': '2024-03-06T07:20:07.274188+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.274188+0000', 'last_undegraded': '2024-03-06T07:20:07.274188+0000', 'last_fullsized': '2024-03-06T07:20:07.274188+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.19 is not active+clean 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.269510+0000', 'last_change': '2024-03-06T07:20:07.269510+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.269510+0000', 'last_undegraded': '2024-03-06T07:20:07.269510+0000', 'last_fullsized': '2024-03-06T07:20:07.269510+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1e is not active+clean 2024-03-06T07:20:12.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271405+0000', 'last_change': '2024-03-06T07:20:07.271405+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271405+0000', 'last_undegraded': '2024-03-06T07:20:07.271405+0000', 'last_fullsized': '2024-03-06T07:20:07.271405+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.280843+0000', 'last_change': '2024-03-06T07:20:07.280843+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.280843+0000', 'last_undegraded': '2024-03-06T07:20:07.280843+0000', 'last_fullsized': '2024-03-06T07:20:07.280843+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1c is not active+clean 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272239+0000', 'last_change': '2024-03-06T07:20:07.272239+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272239+0000', 'last_undegraded': '2024-03-06T07:20:07.272239+0000', 'last_fullsized': '2024-03-06T07:20:07.272239+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1d is not active+clean 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272163+0000', 'last_change': '2024-03-06T07:20:07.272163+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272163+0000', 'last_undegraded': '2024-03-06T07:20:07.272163+0000', 'last_fullsized': '2024-03-06T07:20:07.272163+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2 is not active+clean 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.269497+0000', 'last_change': '2024-03-06T07:20:07.269497+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.269497+0000', 'last_undegraded': '2024-03-06T07:20:07.269497+0000', 'last_fullsized': '2024-03-06T07:20:07.269497+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3 is not active+clean 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.274169+0000', 'last_change': '2024-03-06T07:20:07.274169+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.274169+0000', 'last_undegraded': '2024-03-06T07:20:07.274169+0000', 'last_fullsized': '2024-03-06T07:20:07.274169+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.0 is not active+clean 2024-03-06T07:20:12.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.273354+0000', 'last_change': '2024-03-06T07:20:07.273354+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.273354+0000', 'last_undegraded': '2024-03-06T07:20:07.273354+0000', 'last_fullsized': '2024-03-06T07:20:07.273354+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1 is not active+clean 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271460+0000', 'last_change': '2024-03-06T07:20:07.271460+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271460+0000', 'last_undegraded': '2024-03-06T07:20:07.271460+0000', 'last_fullsized': '2024-03-06T07:20:07.271460+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.6 is not active+clean 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.277278+0000', 'last_change': '2024-03-06T07:20:07.277278+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.277278+0000', 'last_undegraded': '2024-03-06T07:20:07.277278+0000', 'last_fullsized': '2024-03-06T07:20:07.277278+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.7 is not active+clean 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271451+0000', 'last_change': '2024-03-06T07:20:07.271451+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271451+0000', 'last_undegraded': '2024-03-06T07:20:07.271451+0000', 'last_fullsized': '2024-03-06T07:20:07.271451+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.4 is not active+clean 2024-03-06T07:20:12.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271863+0000', 'last_change': '2024-03-06T07:20:07.271863+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271863+0000', 'last_undegraded': '2024-03-06T07:20:07.271863+0000', 'last_fullsized': '2024-03-06T07:20:07.271863+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.5 is not active+clean 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.274293+0000', 'last_change': '2024-03-06T07:20:07.274293+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.274293+0000', 'last_undegraded': '2024-03-06T07:20:07.274293+0000', 'last_fullsized': '2024-03-06T07:20:07.274293+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.a is not active+clean 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272790+0000', 'last_change': '2024-03-06T07:20:07.272790+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272790+0000', 'last_undegraded': '2024-03-06T07:20:07.272790+0000', 'last_fullsized': '2024-03-06T07:20:07.272790+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.8 is not active+clean 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.269859+0000', 'last_change': '2024-03-06T07:20:07.269859+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.269859+0000', 'last_undegraded': '2024-03-06T07:20:07.269859+0000', 'last_fullsized': '2024-03-06T07:20:07.269859+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.9 is not active+clean 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.276715+0000', 'last_change': '2024-03-06T07:20:07.276715+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.276715+0000', 'last_undegraded': '2024-03-06T07:20:07.276715+0000', 'last_fullsized': '2024-03-06T07:20:07.276715+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.e is not active+clean 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272529+0000', 'last_change': '2024-03-06T07:20:07.272529+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272529+0000', 'last_undegraded': '2024-03-06T07:20:07.272529+0000', 'last_fullsized': '2024-03-06T07:20:07.272529+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:20:12.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.f is not active+clean 2024-03-06T07:20:12.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271846+0000', 'last_change': '2024-03-06T07:20:07.271846+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271846+0000', 'last_undegraded': '2024-03-06T07:20:07.271846+0000', 'last_fullsized': '2024-03-06T07:20:07.271846+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2024-03-06T07:20:12.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.278072+0000', 'last_change': '2024-03-06T07:20:07.278072+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.278072+0000', 'last_undegraded': '2024-03-06T07:20:07.278072+0000', 'last_fullsized': '2024-03-06T07:20:07.278072+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:20:12.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.d is not active+clean 2024-03-06T07:20:12.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.272523+0000', 'last_change': '2024-03-06T07:20:07.272523+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.272523+0000', 'last_undegraded': '2024-03-06T07:20:07.272523+0000', 'last_fullsized': '2024-03-06T07:20:07.272523+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.b is not active+clean 2024-03-06T07:20:12.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 54, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:20:07.271196+0000', 'last_change': '2024-03-06T07:20:07.271196+0000', 'last_active': '2024-03-06T07:20:07.257688+0000', 'last_peered': '2024-03-06T07:20:07.257688+0000', 'last_clean': '2024-03-06T07:20:07.257688+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:20:07.271196+0000', 'last_undegraded': '2024-03-06T07:20:07.271196+0000', 'last_fullsized': '2024-03-06T07:20:07.271196+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 54, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'last_clean_scrub_stamp': '2024-03-06T07:20:07.257688+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:20:12.126 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:20:12.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.243+0000 7fb071e4e640 1 Processor -- start 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.243+0000 7fb071e4e640 1 -- start start 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb071e4e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb06c14f360 0x7fb06c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb071e4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0c73d0 con 0x7fb06c158b60 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb071e4e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0be2e0 con 0x7fb06c0c88c0 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb071e4e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c159700 con 0x7fb06c14f360 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb06c14f360 0x7fb06c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb06c158b60 0x7fb06c158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38608/0 (socket says 172.21.15.94:38608) 2024-03-06T07:20:12.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06bfff640 1 -- 172.21.15.94:0/1099507173 learned_addr learned my addr 172.21.15.94:0/1099507173 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:12.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06affd640 1 -- 172.21.15.94:0/1099507173 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254854398 0 0) 0x7fb06c0c73d0 con 0x7fb06c158b60 2024-03-06T07:20:12.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06affd640 1 -- 172.21.15.94:0/1099507173 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb058003520 con 0x7fb06c158b60 2024-03-06T07:20:12.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06affd640 1 -- 172.21.15.94:0/1099507173 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028595075 0 0) 0x7fb06c0be2e0 con 0x7fb06c0c88c0 2024-03-06T07:20:12.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06b7fe640 1 -- 172.21.15.94:0/1099507173 >> v1:172.21.15.145:6789/0 conn(0x7fb06c0c88c0 legacy=0x7fb06c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06b7fe640 1 -- 172.21.15.94:0/1099507173 >> v1:172.21.15.94:6789/0 conn(0x7fb06c158b60 legacy=0x7fb06c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.244+0000 7fb06b7fe640 1 -- 172.21.15.94:0/1099507173 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c0c7670 con 0x7fb06c14f360 2024-03-06T07:20:12.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.245+0000 7fb06b7fe640 1 --2- 172.21.15.94:0/1099507173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c155c10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0097a0 tx=0x7fb05c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa2a6172004db0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.245+0000 7fb06affd640 1 -- 172.21.15.94:0/1099507173 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c035020 con 0x7fb06c14f360 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.245+0000 7fb06affd640 1 -- 172.21.15.94:0/1099507173 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c031830 con 0x7fb06c14f360 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.245+0000 7fb06affd640 1 -- 172.21.15.94:0/1099507173 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c031be0 con 0x7fb06c14f360 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.246+0000 7fb071e4e640 1 -- 172.21.15.94:0/1099507173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 msgr2=0x7fb06c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.246+0000 7fb071e4e640 1 --2- 172.21.15.94:0/1099507173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c155c10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0097a0 tx=0x7fb05c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.246+0000 7fb071e4e640 1 -- 172.21.15.94:0/1099507173 shutdown_connections 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.246+0000 7fb071e4e640 1 --2- 172.21.15.94:0/1099507173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c155c10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.246+0000 7fb071e4e640 1 -- 172.21.15.94:0/1099507173 >> 172.21.15.94:0/1099507173 conn(0x7fb06c0ac570 msgr2=0x7fb06c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.246+0000 7fb071e4e640 1 -- 172.21.15.94:0/1099507173 shutdown_connections 2024-03-06T07:20:12.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.246+0000 7fb071e4e640 1 -- 172.21.15.94:0/1099507173 wait complete. 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.247+0000 7fb071e4e640 1 Processor -- start 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.247+0000 7fb071e4e640 1 -- start start 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb071e4e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb06c0c88c0 0x7fb06c1ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb071e4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c1e7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb071e4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb06c158b60 0x7fb06c0ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb071e4e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0c7c30 con 0x7fb06c14f360 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb071e4e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0c5a50 con 0x7fb06c158b60 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb071e4e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0c5bd0 con 0x7fb06c0c88c0 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c1e7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c1e7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37700/0 (socket says 172.21.15.94:37700) 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 -- 172.21.15.94:0/318042186 learned_addr learned my addr 172.21.15.94:0/318042186 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb06bfff640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb06c158b60 0x7fb06c0ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb06b7fe640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb06c0c88c0 0x7fb06c1ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 -- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb06c0c88c0 msgr2=0x7fb06c1ea6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb06c0c88c0 0x7fb06c1ea6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 -- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb06c158b60 msgr2=0x7fb06c0ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb06c158b60 0x7fb06c0ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 -- 172.21.15.94:0/318042186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c1edf10 con 0x7fb06c14f360 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb062ffd640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c1e7d90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb05400c520 tx=0x7fb05400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.248+0000 7fb068ff9640 1 -- 172.21.15.94:0/318042186 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054017070 con 0x7fb06c14f360 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.249+0000 7fb068ff9640 1 -- 172.21.15.94:0/318042186 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0540035f0 con 0x7fb06c14f360 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.249+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c1ee1f0 con 0x7fb06c14f360 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.249+0000 7fb068ff9640 1 -- 172.21.15.94:0/318042186 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0540039e0 con 0x7fb06c14f360 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.249+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c1ee7b0 con 0x7fb06c14f360 2024-03-06T07:20:12.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.249+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c1eeb90 con 0x7fb06c14f360 2024-03-06T07:20:12.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.250+0000 7fb068ff9640 1 -- 172.21.15.94:0/318042186 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb05401a070 con 0x7fb06c14f360 2024-03-06T07:20:12.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.250+0000 7fb068ff9640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb04c06b4d0 0x7fb04c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.251+0000 7fb068ff9640 1 -- 172.21.15.94:0/318042186 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fb054089a30 con 0x7fb06c14f360 2024-03-06T07:20:12.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.251+0000 7fb06b7fe640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb04c06b4d0 0x7fb04c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.251+0000 7fb06b7fe640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb04c06b4d0 0x7fb04c06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb05c036000 tx=0x7fb05c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:12.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.259+0000 7fb068ff9640 1 -- 172.21.15.94:0/318042186 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb054057e10 con 0x7fb06c14f360 2024-03-06T07:20:12.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.506+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb06c04f330 con 0x7fb06c14f360 2024-03-06T07:20:12.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.507+0000 7fb068ff9640 1 -- 172.21.15.94:0/318042186 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1558 (secure 0 0 0) 0x7fb05405beb0 con 0x7fb06c14f360 2024-03-06T07:20:12.509 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:12.509 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":139,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709482,"num_in_osds":8,"osd_in_since":1709709474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129},{"state_name":"creating+peering","count":64}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":592778,"bytes_used":243630080,"bytes_avail":772850483200,"bytes_total":773094113280,"inactive_pgs_ratio":0.33160620927810669},"fsmap":{"epoch":32,"id":3,"up":1,"in":1,"max":1,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":5652},{"filesystem_id":3,"rank":0,"name":"b","status":"up:active","gid":35084}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-06T07:19:42.654329+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:20:12.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb04c06b4d0 msgr2=0x7fb04c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb04c06b4d0 0x7fb04c06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb05c036000 tx=0x7fb05c008030 comp rx=0 tx=0).stop 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 msgr2=0x7fb06c1e7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c1e7d90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb05400c520 tx=0x7fb05400c9e0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 shutdown_connections 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb04c06b4d0 0x7fb04c06d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb06c158b60 0x7fb06c0ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb06c14f360 0x7fb06c1e7d90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 --2- 172.21.15.94:0/318042186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb06c0c88c0 0x7fb06c1ea6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.508+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 >> 172.21.15.94:0/318042186 conn(0x7fb06c0ac570 msgr2=0x7fb06c14cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.509+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 shutdown_connections 2024-03-06T07:20:12.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.509+0000 7fb071e4e640 1 -- 172.21.15.94:0/318042186 wait complete. 2024-03-06T07:20:12.521 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:20:12.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:12.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.621+0000 7ff1b1866640 1 Processor -- start 2024-03-06T07:20:12.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.621+0000 7ff1b1866640 1 -- start start 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1b1866640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff1ac0c7180 0x7ff1ac159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1b1866640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1ac0c88b0 0x7ff1ac0c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1b1866640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac0c7c60 con 0x7ff1ac1527b0 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1b1866640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac159640 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1b1866640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac0c8060 con 0x7ff1ac0c7180 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1ab7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1ac0c88b0 0x7ff1ac0c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1ab7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1ac0c88b0 0x7ff1ac0c5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37714/0 (socket says 172.21.15.94:37714) 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1ab7fe640 1 -- 172.21.15.94:0/1354676899 learned_addr learned my addr 172.21.15.94:0/1354676899 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:12.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1aa7fc640 1 --2- 172.21.15.94:0/1354676899 >> v2:172.21.15.145:3300/0 conn(0x7ff1ac0c7180 0x7ff1ac159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1ab7fe640 1 -- 172.21.15.94:0/1354676899 >> v2:172.21.15.145:3300/0 conn(0x7ff1ac0c7180 msgr2=0x7ff1ac159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:12.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1ab7fe640 1 --2- 172.21.15.94:0/1354676899 >> v2:172.21.15.145:3300/0 conn(0x7ff1ac0c7180 0x7ff1ac159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1ab7fe640 1 -- 172.21.15.94:0/1354676899 >> v1:172.21.15.145:6789/0 conn(0x7ff1ac1527b0 legacy=0x7ff1ac155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.622+0000 7ff1ab7fe640 1 -- 172.21.15.94:0/1354676899 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0c76a0 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.623+0000 7ff1ab7fe640 1 --2- 172.21.15.94:0/1354676899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac0c5330 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac08cac0 tx=0x7ff1a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7545172440d21e96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:12.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.623+0000 7ff1a9ffb640 1 -- 172.21.15.94:0/1354676899 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a402ca80 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.623+0000 7ff1a9ffb640 1 -- 172.21.15.94:0/1354676899 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a402cc20 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.623+0000 7ff1a9ffb640 1 -- 172.21.15.94:0/1354676899 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40343f0 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 -- 172.21.15.94:0/1354676899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 msgr2=0x7ff1ac0c5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 --2- 172.21.15.94:0/1354676899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac0c5330 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac08cac0 tx=0x7ff1a402bee0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 -- 172.21.15.94:0/1354676899 shutdown_connections 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 --2- 172.21.15.94:0/1354676899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac0c5330 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 --2- 172.21.15.94:0/1354676899 >> v2:172.21.15.145:3300/0 conn(0x7ff1ac0c7180 0x7ff1ac159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 -- 172.21.15.94:0/1354676899 >> 172.21.15.94:0/1354676899 conn(0x7ff1ac0ac560 msgr2=0x7ff1ac0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 -- 172.21.15.94:0/1354676899 shutdown_connections 2024-03-06T07:20:12.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.624+0000 7ff1b1866640 1 -- 172.21.15.94:0/1354676899 wait complete. 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.625+0000 7ff1b1866640 1 Processor -- start 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.625+0000 7ff1b1866640 1 -- start start 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1b1866640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ac0c7180 0x7ff1ac0c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1b1866640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac1e0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1b1866640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1ac1527b0 0x7ff1ac0ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1b1866640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac155600 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1b1866640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac155320 con 0x7ff1ac0c7180 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1b1866640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac152b90 con 0x7ff1ac1527b0 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac1e0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac1e0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37722/0 (socket says 172.21.15.94:37722) 2024-03-06T07:20:12.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 -- 172.21.15.94:0/344862206 learned_addr learned my addr 172.21.15.94:0/344862206 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aaffd640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ac0c7180 0x7ff1ac0c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1ab7fe640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1ac1527b0 0x7ff1ac0ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 -- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1ac1527b0 msgr2=0x7ff1ac0ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1ac1527b0 0x7ff1ac0ab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 -- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ac0c7180 msgr2=0x7ff1ac0c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ac0c7180 0x7ff1ac0c4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 -- 172.21.15.94:0/344862206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac1e1220 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.626+0000 7ff1aa7fc640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac1e0c20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000e9b0 tx=0x7ff1a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.627+0000 7ff193fff640 1 -- 172.21.15.94:0/344862206 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0017070 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.627+0000 7ff193fff640 1 -- 172.21.15.94:0/344862206 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a000c9a0 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.627+0000 7ff193fff640 1 -- 172.21.15.94:0/344862206 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a000cd90 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.627+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac1e1500 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.627+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac158220 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.628+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac075300 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.628+0000 7ff193fff640 1 -- 172.21.15.94:0/344862206 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1a001a070 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.628+0000 7ff193fff640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19806b4d0 0x7ff19806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:12.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.629+0000 7ff193fff640 1 -- 172.21.15.94:0/344862206 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff1a008a980 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.629+0000 7ff1aaffd640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19806b4d0 0x7ff19806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:12.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.629+0000 7ff1aaffd640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19806b4d0 0x7ff19806d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0cb1c0 tx=0x7ff19c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:12.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.633+0000 7ff193fff640 1 -- 172.21.15.94:0/344862206 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1ac075300 con 0x7ff1ac0c88b0 2024-03-06T07:20:12.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.842+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1ac06ce00 con 0x7ff19806b4d0 2024-03-06T07:20:12.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.853+0000 7ff193fff640 1 -- 172.21.15.94:0/344862206 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490559 (secure 0 0 0) 0x7ff1ac06ce00 con 0x7ff19806b4d0 2024-03-06T07:20:12.860 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:12.864 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19806b4d0 msgr2=0x7ff19806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19806b4d0 0x7ff19806d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0cb1c0 tx=0x7ff19c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 msgr2=0x7ff1ac1e0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac1e0c20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000e9b0 tx=0x7ff1a000ee70 comp rx=0 tx=0).stop 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 shutdown_connections 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff19806b4d0 0x7ff19806d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1ac1527b0 0x7ff1ac0ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1ac0c88b0 0x7ff1ac1e0c20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 --2- 172.21.15.94:0/344862206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1ac0c7180 0x7ff1ac0c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:12.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 >> 172.21.15.94:0/344862206 conn(0x7ff1ac0ac560 msgr2=0x7ff1ac151740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:12.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 shutdown_connections 2024-03-06T07:20:12.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:12.857+0000 7ff1b1866640 1 -- 172.21.15.94:0/344862206 wait complete. 2024-03-06T07:20:12.870 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-03-06T07:20:12.660306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238288,"kb_used_data":5472,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736432,"statfs":{"total":773094113280,"available":772850106368,"internally_reserved":0,"allocated":5603328,"data_stored":3640312,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000941"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290558+0000","last_change":"2024-03-06T07:20:08.289892+0000","last_active":"2024-03-06T07:20:10.290558+0000","last_peered":"2024-03-06T07:20:10.290558+0000","last_clean":"2024-03-06T07:20:10.290558+0000","last_became_active":"2024-03-06T07:20:08.289621+0000","last_became_peered":"2024-03-06T07:20:08.289621+0000","last_unstale":"2024-03-06T07:20:10.290558+0000","last_undegraded":"2024-03-06T07:20:10.290558+0000","last_fullsized":"2024-03-06T07:20:10.290558+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:51.962575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289797+0000","last_change":"2024-03-06T07:19:54.052304+0000","last_active":"2024-03-06T07:20:10.289797+0000","last_peered":"2024-03-06T07:20:10.289797+0000","last_clean":"2024-03-06T07:20:10.289797+0000","last_became_active":"2024-03-06T07:19:54.052048+0000","last_became_peered":"2024-03-06T07:19:54.052048+0000","last_unstale":"2024-03-06T07:20:10.289797+0000","last_undegraded":"2024-03-06T07:20:10.289797+0000","last_fullsized":"2024-03-06T07:20:10.289797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:24.129824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281131+0000","last_change":"2024-03-06T07:20:08.288626+0000","last_active":"2024-03-06T07:20:09.281131+0000","last_peered":"2024-03-06T07:20:09.281131+0000","last_clean":"2024-03-06T07:20:09.281131+0000","last_became_active":"2024-03-06T07:20:08.288363+0000","last_became_peered":"2024-03-06T07:20:08.288363+0000","last_unstale":"2024-03-06T07:20:09.281131+0000","last_undegraded":"2024-03-06T07:20:09.281131+0000","last_fullsized":"2024-03-06T07:20:09.281131+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:16.734109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290756+0000","last_change":"2024-03-06T07:19:54.062911+0000","last_active":"2024-03-06T07:20:10.290756+0000","last_peered":"2024-03-06T07:20:10.290756+0000","last_clean":"2024-03-06T07:20:10.290756+0000","last_became_active":"2024-03-06T07:19:54.062566+0000","last_became_peered":"2024-03-06T07:19:54.062566+0000","last_unstale":"2024-03-06T07:20:10.290756+0000","last_undegraded":"2024-03-06T07:20:10.290756+0000","last_fullsized":"2024-03-06T07:20:10.290756+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289681+0000","last_change":"2024-03-06T07:20:08.298070+0000","last_active":"2024-03-06T07:20:10.289681+0000","last_peered":"2024-03-06T07:20:10.289681+0000","last_clean":"2024-03-06T07:20:10.289681+0000","last_became_active":"2024-03-06T07:20:08.297509+0000","last_became_peered":"2024-03-06T07:20:08.297509+0000","last_unstale":"2024-03-06T07:20:10.289681+0000","last_undegraded":"2024-03-06T07:20:10.289681+0000","last_fullsized":"2024-03-06T07:20:10.289681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:18.762564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279967+0000","last_change":"2024-03-06T07:19:54.051868+0000","last_active":"2024-03-06T07:20:09.279967+0000","last_peered":"2024-03-06T07:20:09.279967+0000","last_clean":"2024-03-06T07:20:09.279967+0000","last_became_active":"2024-03-06T07:19:54.051611+0000","last_became_peered":"2024-03-06T07:19:54.051611+0000","last_unstale":"2024-03-06T07:20:09.279967+0000","last_undegraded":"2024-03-06T07:20:09.279967+0000","last_fullsized":"2024-03-06T07:20:09.279967+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289614+0000","last_change":"2024-03-06T07:20:08.295096+0000","last_active":"2024-03-06T07:20:10.289614+0000","last_peered":"2024-03-06T07:20:10.289614+0000","last_clean":"2024-03-06T07:20:10.289614+0000","last_became_active":"2024-03-06T07:20:08.294783+0000","last_became_peered":"2024-03-06T07:20:08.294783+0000","last_unstale":"2024-03-06T07:20:10.289614+0000","last_undegraded":"2024-03-06T07:20:10.289614+0000","last_fullsized":"2024-03-06T07:20:10.289614+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.458664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290585+0000","last_change":"2024-03-06T07:19:54.055553+0000","last_active":"2024-03-06T07:20:10.290585+0000","last_peered":"2024-03-06T07:20:10.290585+0000","last_clean":"2024-03-06T07:20:10.290585+0000","last_became_active":"2024-03-06T07:19:54.055271+0000","last_became_peered":"2024-03-06T07:19:54.055271+0000","last_unstale":"2024-03-06T07:20:10.290585+0000","last_undegraded":"2024-03-06T07:20:10.290585+0000","last_fullsized":"2024-03-06T07:20:10.290585+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274653+0000","last_change":"2024-03-06T07:20:08.287946+0000","last_active":"2024-03-06T07:20:10.274653+0000","last_peered":"2024-03-06T07:20:10.274653+0000","last_clean":"2024-03-06T07:20:10.274653+0000","last_became_active":"2024-03-06T07:20:08.287660+0000","last_became_peered":"2024-03-06T07:20:08.287660+0000","last_unstale":"2024-03-06T07:20:10.274653+0000","last_undegraded":"2024-03-06T07:20:10.274653+0000","last_fullsized":"2024-03-06T07:20:10.274653+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:11.468274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280063+0000","last_change":"2024-03-06T07:19:54.046161+0000","last_active":"2024-03-06T07:20:09.280063+0000","last_peered":"2024-03-06T07:20:09.280063+0000","last_clean":"2024-03-06T07:20:09.280063+0000","last_became_active":"2024-03-06T07:19:54.045875+0000","last_became_peered":"2024-03-06T07:19:54.045875+0000","last_unstale":"2024-03-06T07:20:09.280063+0000","last_undegraded":"2024-03-06T07:20:09.280063+0000","last_fullsized":"2024-03-06T07:20:09.280063+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:50.526448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272026+0000","last_change":"2024-03-06T07:20:08.274700+0000","last_active":"2024-03-06T07:20:10.272026+0000","last_peered":"2024-03-06T07:20:10.272026+0000","last_clean":"2024-03-06T07:20:10.272026+0000","last_became_active":"2024-03-06T07:20:08.274419+0000","last_became_peered":"2024-03-06T07:20:08.274419+0000","last_unstale":"2024-03-06T07:20:10.272026+0000","last_undegraded":"2024-03-06T07:20:10.272026+0000","last_fullsized":"2024-03-06T07:20:10.272026+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:00.756603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289147+0000","last_change":"2024-03-06T07:19:54.047960+0000","last_active":"2024-03-06T07:20:10.289147+0000","last_peered":"2024-03-06T07:20:10.289147+0000","last_clean":"2024-03-06T07:20:10.289147+0000","last_became_active":"2024-03-06T07:19:54.047638+0000","last_became_peered":"2024-03-06T07:19:54.047638+0000","last_unstale":"2024-03-06T07:20:10.289147+0000","last_undegraded":"2024-03-06T07:20:10.289147+0000","last_fullsized":"2024-03-06T07:20:10.289147+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281001+0000","last_change":"2024-03-06T07:20:08.286960+0000","last_active":"2024-03-06T07:20:09.281001+0000","last_peered":"2024-03-06T07:20:09.281001+0000","last_clean":"2024-03-06T07:20:09.281001+0000","last_became_active":"2024-03-06T07:20:08.286724+0000","last_became_peered":"2024-03-06T07:20:08.286724+0000","last_unstale":"2024-03-06T07:20:09.281001+0000","last_undegraded":"2024-03-06T07:20:09.281001+0000","last_fullsized":"2024-03-06T07:20:09.281001+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:03.852954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279203+0000","last_change":"2024-03-06T07:19:54.048726+0000","last_active":"2024-03-06T07:20:09.279203+0000","last_peered":"2024-03-06T07:20:09.279203+0000","last_clean":"2024-03-06T07:20:09.279203+0000","last_became_active":"2024-03-06T07:19:54.048377+0000","last_became_peered":"2024-03-06T07:19:54.048377+0000","last_unstale":"2024-03-06T07:20:09.279203+0000","last_undegraded":"2024-03-06T07:20:09.279203+0000","last_fullsized":"2024-03-06T07:20:09.279203+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:57.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279232+0000","last_change":"2024-03-06T07:20:08.286854+0000","last_active":"2024-03-06T07:20:09.279232+0000","last_peered":"2024-03-06T07:20:09.279232+0000","last_clean":"2024-03-06T07:20:09.279232+0000","last_became_active":"2024-03-06T07:20:08.286383+0000","last_became_peered":"2024-03-06T07:20:08.286383+0000","last_unstale":"2024-03-06T07:20:09.279232+0000","last_undegraded":"2024-03-06T07:20:09.279232+0000","last_fullsized":"2024-03-06T07:20:09.279232+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:53.510411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287708+0000","last_change":"2024-03-06T07:19:54.059007+0000","last_active":"2024-03-06T07:20:10.287708+0000","last_peered":"2024-03-06T07:20:10.287708+0000","last_clean":"2024-03-06T07:20:10.287708+0000","last_became_active":"2024-03-06T07:19:54.058694+0000","last_became_peered":"2024-03-06T07:19:54.058694+0000","last_unstale":"2024-03-06T07:20:10.287708+0000","last_undegraded":"2024-03-06T07:20:10.287708+0000","last_fullsized":"2024-03-06T07:20:10.287708+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:50.508506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280082+0000","last_change":"2024-03-06T07:20:08.296591+0000","last_active":"2024-03-06T07:20:09.280082+0000","last_peered":"2024-03-06T07:20:09.280082+0000","last_clean":"2024-03-06T07:20:09.280082+0000","last_became_active":"2024-03-06T07:20:08.295965+0000","last_became_peered":"2024-03-06T07:20:08.295965+0000","last_unstale":"2024-03-06T07:20:09.280082+0000","last_undegraded":"2024-03-06T07:20:09.280082+0000","last_fullsized":"2024-03-06T07:20:09.280082+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:43.959679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287761+0000","last_change":"2024-03-06T07:19:54.057769+0000","last_active":"2024-03-06T07:20:10.287761+0000","last_peered":"2024-03-06T07:20:10.287761+0000","last_clean":"2024-03-06T07:20:10.287761+0000","last_became_active":"2024-03-06T07:19:54.057496+0000","last_became_peered":"2024-03-06T07:19:54.057496+0000","last_unstale":"2024-03-06T07:20:10.287761+0000","last_undegraded":"2024-03-06T07:20:10.287761+0000","last_fullsized":"2024-03-06T07:20:10.287761+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:51.719701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272401+0000","last_change":"2024-03-06T07:20:08.281429+0000","last_active":"2024-03-06T07:20:10.272401+0000","last_peered":"2024-03-06T07:20:10.272401+0000","last_clean":"2024-03-06T07:20:10.272401+0000","last_became_active":"2024-03-06T07:20:08.281116+0000","last_became_peered":"2024-03-06T07:20:08.281116+0000","last_unstale":"2024-03-06T07:20:10.272401+0000","last_undegraded":"2024-03-06T07:20:10.272401+0000","last_fullsized":"2024-03-06T07:20:10.272401+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:51.962575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289359+0000","last_change":"2024-03-06T07:19:54.050005+0000","last_active":"2024-03-06T07:20:10.289359+0000","last_peered":"2024-03-06T07:20:10.289359+0000","last_clean":"2024-03-06T07:20:10.289359+0000","last_became_active":"2024-03-06T07:19:54.049750+0000","last_became_peered":"2024-03-06T07:19:54.049750+0000","last_unstale":"2024-03-06T07:20:10.289359+0000","last_undegraded":"2024-03-06T07:20:10.289359+0000","last_fullsized":"2024-03-06T07:20:10.289359+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:37.609871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290810+0000","last_change":"2024-03-06T07:20:08.284734+0000","last_active":"2024-03-06T07:20:10.290810+0000","last_peered":"2024-03-06T07:20:10.290810+0000","last_clean":"2024-03-06T07:20:10.290810+0000","last_became_active":"2024-03-06T07:20:08.284289+0000","last_became_peered":"2024-03-06T07:20:08.284289+0000","last_unstale":"2024-03-06T07:20:10.290810+0000","last_undegraded":"2024-03-06T07:20:10.290810+0000","last_fullsized":"2024-03-06T07:20:10.290810+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:01.438975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.275106+0000","last_change":"2024-03-06T07:19:54.052367+0000","last_active":"2024-03-06T07:20:10.275106+0000","last_peered":"2024-03-06T07:20:10.275106+0000","last_clean":"2024-03-06T07:20:10.275106+0000","last_became_active":"2024-03-06T07:19:54.052119+0000","last_became_peered":"2024-03-06T07:19:54.052119+0000","last_unstale":"2024-03-06T07:20:10.275106+0000","last_undegraded":"2024-03-06T07:20:10.275106+0000","last_fullsized":"2024-03-06T07:20:10.275106+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.678839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290604+0000","last_change":"2024-03-06T07:20:08.288011+0000","last_active":"2024-03-06T07:20:10.290604+0000","last_peered":"2024-03-06T07:20:10.290604+0000","last_clean":"2024-03-06T07:20:10.290604+0000","last_became_active":"2024-03-06T07:20:08.286929+0000","last_became_peered":"2024-03-06T07:20:08.286929+0000","last_unstale":"2024-03-06T07:20:10.290604+0000","last_undegraded":"2024-03-06T07:20:10.290604+0000","last_fullsized":"2024-03-06T07:20:10.290604+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:14.797649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287884+0000","last_change":"2024-03-06T07:19:54.040294+0000","last_active":"2024-03-06T07:20:10.287884+0000","last_peered":"2024-03-06T07:20:10.287884+0000","last_clean":"2024-03-06T07:20:10.287884+0000","last_became_active":"2024-03-06T07:19:54.040048+0000","last_became_peered":"2024-03-06T07:19:54.040048+0000","last_unstale":"2024-03-06T07:20:10.287884+0000","last_undegraded":"2024-03-06T07:20:10.287884+0000","last_fullsized":"2024-03-06T07:20:10.287884+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:41.393636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289930+0000","last_change":"2024-03-06T07:20:08.280889+0000","last_active":"2024-03-06T07:20:10.289930+0000","last_peered":"2024-03-06T07:20:10.289930+0000","last_clean":"2024-03-06T07:20:10.289930+0000","last_became_active":"2024-03-06T07:20:08.280619+0000","last_became_peered":"2024-03-06T07:20:08.280619+0000","last_unstale":"2024-03-06T07:20:10.289930+0000","last_undegraded":"2024-03-06T07:20:10.289930+0000","last_fullsized":"2024-03-06T07:20:10.289930+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:53.510411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289088+0000","last_change":"2024-03-06T07:19:54.056367+0000","last_active":"2024-03-06T07:20:10.289088+0000","last_peered":"2024-03-06T07:20:10.289088+0000","last_clean":"2024-03-06T07:20:10.289088+0000","last_became_active":"2024-03-06T07:19:54.056125+0000","last_became_peered":"2024-03-06T07:19:54.056125+0000","last_unstale":"2024-03-06T07:20:10.289088+0000","last_undegraded":"2024-03-06T07:20:10.289088+0000","last_fullsized":"2024-03-06T07:20:10.289088+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274585+0000","last_change":"2024-03-06T07:20:08.291928+0000","last_active":"2024-03-06T07:20:10.274585+0000","last_peered":"2024-03-06T07:20:10.274585+0000","last_clean":"2024-03-06T07:20:10.274585+0000","last_became_active":"2024-03-06T07:20:08.291627+0000","last_became_peered":"2024-03-06T07:20:08.291627+0000","last_unstale":"2024-03-06T07:20:10.274585+0000","last_undegraded":"2024-03-06T07:20:10.274585+0000","last_fullsized":"2024-03-06T07:20:10.274585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:40.550642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279266+0000","last_change":"2024-03-06T07:19:54.051456+0000","last_active":"2024-03-06T07:20:09.279266+0000","last_peered":"2024-03-06T07:20:09.279266+0000","last_clean":"2024-03-06T07:20:09.279266+0000","last_became_active":"2024-03-06T07:19:54.051193+0000","last_became_peered":"2024-03-06T07:19:54.051193+0000","last_unstale":"2024-03-06T07:20:09.279266+0000","last_undegraded":"2024-03-06T07:20:09.279266+0000","last_fullsized":"2024-03-06T07:20:09.279266+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272338+0000","last_change":"2024-03-06T07:20:08.284233+0000","last_active":"2024-03-06T07:20:10.272338+0000","last_peered":"2024-03-06T07:20:10.272338+0000","last_clean":"2024-03-06T07:20:10.272338+0000","last_became_active":"2024-03-06T07:20:08.284044+0000","last_became_peered":"2024-03-06T07:20:08.284044+0000","last_unstale":"2024-03-06T07:20:10.272338+0000","last_undegraded":"2024-03-06T07:20:10.272338+0000","last_fullsized":"2024-03-06T07:20:10.272338+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.838274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289350+0000","last_change":"2024-03-06T07:19:54.049011+0000","last_active":"2024-03-06T07:20:10.289350+0000","last_peered":"2024-03-06T07:20:10.289350+0000","last_clean":"2024-03-06T07:20:10.289350+0000","last_became_active":"2024-03-06T07:19:54.048738+0000","last_became_peered":"2024-03-06T07:19:54.048738+0000","last_unstale":"2024-03-06T07:20:10.289350+0000","last_undegraded":"2024-03-06T07:20:10.289350+0000","last_fullsized":"2024-03-06T07:20:10.289350+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:36.367136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290293+0000","last_change":"2024-03-06T07:20:08.290724+0000","last_active":"2024-03-06T07:20:10.290293+0000","last_peered":"2024-03-06T07:20:10.290293+0000","last_clean":"2024-03-06T07:20:10.290293+0000","last_became_active":"2024-03-06T07:20:08.290411+0000","last_became_peered":"2024-03-06T07:20:08.290411+0000","last_unstale":"2024-03-06T07:20:10.290293+0000","last_undegraded":"2024-03-06T07:20:10.290293+0000","last_fullsized":"2024-03-06T07:20:10.290293+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.838274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288991+0000","last_change":"2024-03-06T07:19:54.049339+0000","last_active":"2024-03-06T07:20:10.288991+0000","last_peered":"2024-03-06T07:20:10.288991+0000","last_clean":"2024-03-06T07:20:10.288991+0000","last_became_active":"2024-03-06T07:19:54.049059+0000","last_became_peered":"2024-03-06T07:19:54.049059+0000","last_unstale":"2024-03-06T07:20:10.288991+0000","last_undegraded":"2024-03-06T07:20:10.288991+0000","last_fullsized":"2024-03-06T07:20:10.288991+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280625+0000","last_change":"2024-03-06T07:20:08.288942+0000","last_active":"2024-03-06T07:20:09.280625+0000","last_peered":"2024-03-06T07:20:09.280625+0000","last_clean":"2024-03-06T07:20:09.280625+0000","last_became_active":"2024-03-06T07:20:08.288543+0000","last_became_peered":"2024-03-06T07:20:08.288543+0000","last_unstale":"2024-03-06T07:20:09.280625+0000","last_undegraded":"2024-03-06T07:20:09.280625+0000","last_fullsized":"2024-03-06T07:20:09.280625+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:48.612666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280007+0000","last_change":"2024-03-06T07:19:54.058699+0000","last_active":"2024-03-06T07:20:09.280007+0000","last_peered":"2024-03-06T07:20:09.280007+0000","last_clean":"2024-03-06T07:20:09.280007+0000","last_became_active":"2024-03-06T07:19:54.056798+0000","last_became_peered":"2024-03-06T07:19:54.056798+0000","last_unstale":"2024-03-06T07:20:09.280007+0000","last_undegraded":"2024-03-06T07:20:09.280007+0000","last_fullsized":"2024-03-06T07:20:09.280007+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288570+0000","last_change":"2024-03-06T07:20:08.285547+0000","last_active":"2024-03-06T07:20:10.288570+0000","last_peered":"2024-03-06T07:20:10.288570+0000","last_clean":"2024-03-06T07:20:10.288570+0000","last_became_active":"2024-03-06T07:20:08.282256+0000","last_became_peered":"2024-03-06T07:20:08.282256+0000","last_unstale":"2024-03-06T07:20:10.288570+0000","last_undegraded":"2024-03-06T07:20:10.288570+0000","last_fullsized":"2024-03-06T07:20:10.288570+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:53.137345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290830+0000","last_change":"2024-03-06T07:19:54.062261+0000","last_active":"2024-03-06T07:20:10.290830+0000","last_peered":"2024-03-06T07:20:10.290830+0000","last_clean":"2024-03-06T07:20:10.290830+0000","last_became_active":"2024-03-06T07:19:54.061853+0000","last_became_peered":"2024-03-06T07:19:54.061853+0000","last_unstale":"2024-03-06T07:20:10.290830+0000","last_undegraded":"2024-03-06T07:20:10.290830+0000","last_fullsized":"2024-03-06T07:20:10.290830+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289718+0000","last_change":"2024-03-06T07:20:08.288551+0000","last_active":"2024-03-06T07:20:10.289718+0000","last_peered":"2024-03-06T07:20:10.289718+0000","last_clean":"2024-03-06T07:20:10.289718+0000","last_became_active":"2024-03-06T07:20:08.288232+0000","last_became_peered":"2024-03-06T07:20:08.288232+0000","last_unstale":"2024-03-06T07:20:10.289718+0000","last_undegraded":"2024-03-06T07:20:10.289718+0000","last_fullsized":"2024-03-06T07:20:10.289718+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.521500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280103+0000","last_change":"2024-03-06T07:19:54.054672+0000","last_active":"2024-03-06T07:20:09.280103+0000","last_peered":"2024-03-06T07:20:09.280103+0000","last_clean":"2024-03-06T07:20:09.280103+0000","last_became_active":"2024-03-06T07:19:54.054301+0000","last_became_peered":"2024-03-06T07:19:54.054301+0000","last_unstale":"2024-03-06T07:20:09.280103+0000","last_undegraded":"2024-03-06T07:20:09.280103+0000","last_fullsized":"2024-03-06T07:20:09.280103+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:05.772166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281220+0000","last_change":"2024-03-06T07:20:08.286513+0000","last_active":"2024-03-06T07:20:09.281220+0000","last_peered":"2024-03-06T07:20:09.281220+0000","last_clean":"2024-03-06T07:20:09.281220+0000","last_became_active":"2024-03-06T07:20:08.286189+0000","last_became_peered":"2024-03-06T07:20:08.286189+0000","last_unstale":"2024-03-06T07:20:09.281220+0000","last_undegraded":"2024-03-06T07:20:09.281220+0000","last_fullsized":"2024-03-06T07:20:09.281220+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:21.720175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279896+0000","last_change":"2024-03-06T07:19:54.051669+0000","last_active":"2024-03-06T07:20:09.279896+0000","last_peered":"2024-03-06T07:20:09.279896+0000","last_clean":"2024-03-06T07:20:09.279896+0000","last_became_active":"2024-03-06T07:19:54.051312+0000","last_became_peered":"2024-03-06T07:19:54.051312+0000","last_unstale":"2024-03-06T07:20:09.279896+0000","last_undegraded":"2024-03-06T07:20:09.279896+0000","last_fullsized":"2024-03-06T07:20:09.279896+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288123+0000","last_change":"2024-03-06T07:20:08.281934+0000","last_active":"2024-03-06T07:20:10.288123+0000","last_peered":"2024-03-06T07:20:10.288123+0000","last_clean":"2024-03-06T07:20:10.288123+0000","last_became_active":"2024-03-06T07:20:08.281674+0000","last_became_peered":"2024-03-06T07:20:08.281674+0000","last_unstale":"2024-03-06T07:20:10.288123+0000","last_undegraded":"2024-03-06T07:20:10.288123+0000","last_fullsized":"2024-03-06T07:20:10.288123+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:32.350695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281124+0000","last_change":"2024-03-06T07:19:54.047677+0000","last_active":"2024-03-06T07:20:09.281124+0000","last_peered":"2024-03-06T07:20:09.281124+0000","last_clean":"2024-03-06T07:20:09.281124+0000","last_became_active":"2024-03-06T07:19:54.045393+0000","last_became_peered":"2024-03-06T07:19:54.045393+0000","last_unstale":"2024-03-06T07:20:09.281124+0000","last_undegraded":"2024-03-06T07:20:09.281124+0000","last_fullsized":"2024-03-06T07:20:09.281124+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289563+0000","last_change":"2024-03-06T07:20:08.288100+0000","last_active":"2024-03-06T07:20:10.289563+0000","last_peered":"2024-03-06T07:20:10.289563+0000","last_clean":"2024-03-06T07:20:10.289563+0000","last_became_active":"2024-03-06T07:20:08.287387+0000","last_became_peered":"2024-03-06T07:20:08.287387+0000","last_unstale":"2024-03-06T07:20:10.289563+0000","last_undegraded":"2024-03-06T07:20:10.289563+0000","last_fullsized":"2024-03-06T07:20:10.289563+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:45.089464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288401+0000","last_change":"2024-03-06T07:19:54.058376+0000","last_active":"2024-03-06T07:20:10.288401+0000","last_peered":"2024-03-06T07:20:10.288401+0000","last_clean":"2024-03-06T07:20:10.288401+0000","last_became_active":"2024-03-06T07:19:54.058151+0000","last_became_peered":"2024-03-06T07:19:54.058151+0000","last_unstale":"2024-03-06T07:20:10.288401+0000","last_undegraded":"2024-03-06T07:20:10.288401+0000","last_fullsized":"2024-03-06T07:20:10.288401+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:10.287168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272359+0000","last_change":"2024-03-06T07:20:08.280882+0000","last_active":"2024-03-06T07:20:10.272359+0000","last_peered":"2024-03-06T07:20:10.272359+0000","last_clean":"2024-03-06T07:20:10.272359+0000","last_became_active":"2024-03-06T07:20:08.280648+0000","last_became_peered":"2024-03-06T07:20:08.280648+0000","last_unstale":"2024-03-06T07:20:10.272359+0000","last_undegraded":"2024-03-06T07:20:10.272359+0000","last_fullsized":"2024-03-06T07:20:10.272359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:30.212401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274623+0000","last_change":"2024-03-06T07:19:54.046473+0000","last_active":"2024-03-06T07:20:10.274623+0000","last_peered":"2024-03-06T07:20:10.274623+0000","last_clean":"2024-03-06T07:20:10.274623+0000","last_became_active":"2024-03-06T07:19:54.045977+0000","last_became_peered":"2024-03-06T07:19:54.045977+0000","last_unstale":"2024-03-06T07:20:10.274623+0000","last_undegraded":"2024-03-06T07:20:10.274623+0000","last_fullsized":"2024-03-06T07:20:10.274623+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:28.437704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288551+0000","last_change":"2024-03-06T07:20:08.279693+0000","last_active":"2024-03-06T07:20:10.288551+0000","last_peered":"2024-03-06T07:20:10.288551+0000","last_clean":"2024-03-06T07:20:10.288551+0000","last_became_active":"2024-03-06T07:20:08.279409+0000","last_became_peered":"2024-03-06T07:20:08.279409+0000","last_unstale":"2024-03-06T07:20:10.288551+0000","last_undegraded":"2024-03-06T07:20:10.288551+0000","last_fullsized":"2024-03-06T07:20:10.288551+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:20.720975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279102+0000","last_change":"2024-03-06T07:19:54.050649+0000","last_active":"2024-03-06T07:20:09.279102+0000","last_peered":"2024-03-06T07:20:09.279102+0000","last_clean":"2024-03-06T07:20:09.279102+0000","last_became_active":"2024-03-06T07:19:54.050392+0000","last_became_peered":"2024-03-06T07:19:54.050392+0000","last_unstale":"2024-03-06T07:20:09.279102+0000","last_undegraded":"2024-03-06T07:20:09.279102+0000","last_fullsized":"2024-03-06T07:20:09.279102+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279855+0000","last_change":"2024-03-06T07:20:08.291858+0000","last_active":"2024-03-06T07:20:09.279855+0000","last_peered":"2024-03-06T07:20:09.279855+0000","last_clean":"2024-03-06T07:20:09.279855+0000","last_became_active":"2024-03-06T07:20:08.291629+0000","last_became_peered":"2024-03-06T07:20:08.291629+0000","last_unstale":"2024-03-06T07:20:09.279855+0000","last_undegraded":"2024-03-06T07:20:09.279855+0000","last_fullsized":"2024-03-06T07:20:09.279855+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:16.214208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280048+0000","last_change":"2024-03-06T07:19:54.046865+0000","last_active":"2024-03-06T07:20:09.280048+0000","last_peered":"2024-03-06T07:20:09.280048+0000","last_clean":"2024-03-06T07:20:09.280048+0000","last_became_active":"2024-03-06T07:19:54.046545+0000","last_became_peered":"2024-03-06T07:19:54.046545+0000","last_unstale":"2024-03-06T07:20:09.280048+0000","last_undegraded":"2024-03-06T07:20:09.280048+0000","last_fullsized":"2024-03-06T07:20:09.280048+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:09.982751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288312+0000","last_change":"2024-03-06T07:20:08.278501+0000","last_active":"2024-03-06T07:20:10.288312+0000","last_peered":"2024-03-06T07:20:10.288312+0000","last_clean":"2024-03-06T07:20:10.288312+0000","last_became_active":"2024-03-06T07:20:08.278246+0000","last_became_peered":"2024-03-06T07:20:08.278246+0000","last_unstale":"2024-03-06T07:20:10.288312+0000","last_undegraded":"2024-03-06T07:20:10.288312+0000","last_fullsized":"2024-03-06T07:20:10.288312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:20.839748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288935+0000","last_change":"2024-03-06T07:19:54.042151+0000","last_active":"2024-03-06T07:20:10.288935+0000","last_peered":"2024-03-06T07:20:10.288935+0000","last_clean":"2024-03-06T07:20:10.288935+0000","last_became_active":"2024-03-06T07:19:54.041796+0000","last_became_peered":"2024-03-06T07:19:54.041796+0000","last_unstale":"2024-03-06T07:20:10.288935+0000","last_undegraded":"2024-03-06T07:20:10.288935+0000","last_fullsized":"2024-03-06T07:20:10.288935+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:27.407999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272612+0000","last_change":"2024-03-06T07:20:08.289732+0000","last_active":"2024-03-06T07:20:10.272612+0000","last_peered":"2024-03-06T07:20:10.272612+0000","last_clean":"2024-03-06T07:20:10.272612+0000","last_became_active":"2024-03-06T07:20:08.289476+0000","last_became_peered":"2024-03-06T07:20:08.289476+0000","last_unstale":"2024-03-06T07:20:10.272612+0000","last_undegraded":"2024-03-06T07:20:10.272612+0000","last_fullsized":"2024-03-06T07:20:10.272612+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:34.619396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288148+0000","last_change":"2024-03-06T07:19:54.051161+0000","last_active":"2024-03-06T07:20:10.288148+0000","last_peered":"2024-03-06T07:20:10.288148+0000","last_clean":"2024-03-06T07:20:10.288148+0000","last_became_active":"2024-03-06T07:19:54.050893+0000","last_became_peered":"2024-03-06T07:19:54.050893+0000","last_unstale":"2024-03-06T07:20:10.288148+0000","last_undegraded":"2024-03-06T07:20:10.288148+0000","last_fullsized":"2024-03-06T07:20:10.288148+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:28.711633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272490+0000","last_change":"2024-03-06T07:20:08.291381+0000","last_active":"2024-03-06T07:20:10.272490+0000","last_peered":"2024-03-06T07:20:10.272490+0000","last_clean":"2024-03-06T07:20:10.272490+0000","last_became_active":"2024-03-06T07:20:08.291115+0000","last_became_peered":"2024-03-06T07:20:08.291115+0000","last_unstale":"2024-03-06T07:20:10.272490+0000","last_undegraded":"2024-03-06T07:20:10.272490+0000","last_fullsized":"2024-03-06T07:20:10.272490+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:14.797649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279973+0000","last_change":"2024-03-06T07:19:54.040041+0000","last_active":"2024-03-06T07:20:09.279973+0000","last_peered":"2024-03-06T07:20:09.279973+0000","last_clean":"2024-03-06T07:20:09.279973+0000","last_became_active":"2024-03-06T07:19:54.039730+0000","last_became_peered":"2024-03-06T07:19:54.039730+0000","last_unstale":"2024-03-06T07:20:09.279973+0000","last_undegraded":"2024-03-06T07:20:09.279973+0000","last_fullsized":"2024-03-06T07:20:09.279973+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280738+0000","last_change":"2024-03-06T07:20:08.289711+0000","last_active":"2024-03-06T07:20:09.280738+0000","last_peered":"2024-03-06T07:20:09.280738+0000","last_clean":"2024-03-06T07:20:09.280738+0000","last_became_active":"2024-03-06T07:20:08.289485+0000","last_became_peered":"2024-03-06T07:20:08.289485+0000","last_unstale":"2024-03-06T07:20:09.280738+0000","last_undegraded":"2024-03-06T07:20:09.280738+0000","last_fullsized":"2024-03-06T07:20:09.280738+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:11.664663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279226+0000","last_change":"2024-03-06T07:19:54.055165+0000","last_active":"2024-03-06T07:20:09.279226+0000","last_peered":"2024-03-06T07:20:09.279226+0000","last_clean":"2024-03-06T07:20:09.279226+0000","last_became_active":"2024-03-06T07:19:54.054908+0000","last_became_peered":"2024-03-06T07:19:54.054908+0000","last_unstale":"2024-03-06T07:20:09.279226+0000","last_undegraded":"2024-03-06T07:20:09.279226+0000","last_fullsized":"2024-03-06T07:20:09.279226+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:02.499776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.278929+0000","last_change":"2024-03-06T07:20:08.291670+0000","last_active":"2024-03-06T07:20:09.278929+0000","last_peered":"2024-03-06T07:20:09.278929+0000","last_clean":"2024-03-06T07:20:09.278929+0000","last_became_active":"2024-03-06T07:20:08.290863+0000","last_became_peered":"2024-03-06T07:20:08.290863+0000","last_unstale":"2024-03-06T07:20:09.278929+0000","last_undegraded":"2024-03-06T07:20:09.278929+0000","last_fullsized":"2024-03-06T07:20:09.278929+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:00.756603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290339+0000","last_change":"2024-03-06T07:19:54.049805+0000","last_active":"2024-03-06T07:20:10.290339+0000","last_peered":"2024-03-06T07:20:10.290339+0000","last_clean":"2024-03-06T07:20:10.290339+0000","last_became_active":"2024-03-06T07:19:54.049562+0000","last_became_peered":"2024-03-06T07:19:54.049562+0000","last_unstale":"2024-03-06T07:20:10.290339+0000","last_undegraded":"2024-03-06T07:20:10.290339+0000","last_fullsized":"2024-03-06T07:20:10.290339+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:21.645621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290274+0000","last_change":"2024-03-06T07:20:08.283946+0000","last_active":"2024-03-06T07:20:10.290274+0000","last_peered":"2024-03-06T07:20:10.290274+0000","last_clean":"2024-03-06T07:20:10.290274+0000","last_became_active":"2024-03-06T07:20:08.283701+0000","last_became_peered":"2024-03-06T07:20:08.283701+0000","last_unstale":"2024-03-06T07:20:10.290274+0000","last_undegraded":"2024-03-06T07:20:10.290274+0000","last_fullsized":"2024-03-06T07:20:10.290274+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:09.713122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279055+0000","last_change":"2024-03-06T07:19:54.052763+0000","last_active":"2024-03-06T07:20:09.279055+0000","last_peered":"2024-03-06T07:20:09.279055+0000","last_clean":"2024-03-06T07:20:09.279055+0000","last_became_active":"2024-03-06T07:19:54.052472+0000","last_became_peered":"2024-03-06T07:19:54.052472+0000","last_unstale":"2024-03-06T07:20:09.279055+0000","last_undegraded":"2024-03-06T07:20:09.279055+0000","last_fullsized":"2024-03-06T07:20:09.279055+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:07.485843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272194+0000","last_change":"2024-03-06T07:20:08.274735+0000","last_active":"2024-03-06T07:20:10.272194+0000","last_peered":"2024-03-06T07:20:10.272194+0000","last_clean":"2024-03-06T07:20:10.272194+0000","last_became_active":"2024-03-06T07:20:08.274463+0000","last_became_peered":"2024-03-06T07:20:08.274463+0000","last_unstale":"2024-03-06T07:20:10.272194+0000","last_undegraded":"2024-03-06T07:20:10.272194+0000","last_fullsized":"2024-03-06T07:20:10.272194+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:09.713122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272121+0000","last_change":"2024-03-06T07:19:54.059383+0000","last_active":"2024-03-06T07:20:10.272121+0000","last_peered":"2024-03-06T07:20:10.272121+0000","last_clean":"2024-03-06T07:20:10.272121+0000","last_became_active":"2024-03-06T07:19:54.059123+0000","last_became_peered":"2024-03-06T07:19:54.059123+0000","last_unstale":"2024-03-06T07:20:10.272121+0000","last_undegraded":"2024-03-06T07:20:10.272121+0000","last_fullsized":"2024-03-06T07:20:10.272121+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280311+0000","last_change":"2024-03-06T07:20:08.289078+0000","last_active":"2024-03-06T07:20:09.280311+0000","last_peered":"2024-03-06T07:20:09.280311+0000","last_clean":"2024-03-06T07:20:09.280311+0000","last_became_active":"2024-03-06T07:20:08.288498+0000","last_became_peered":"2024-03-06T07:20:08.288498+0000","last_unstale":"2024-03-06T07:20:09.280311+0000","last_undegraded":"2024-03-06T07:20:09.280311+0000","last_fullsized":"2024-03-06T07:20:09.280311+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:11.664663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289729+0000","last_change":"2024-03-06T07:20:07.288076+0000","last_active":"2024-03-06T07:20:10.289729+0000","last_peered":"2024-03-06T07:20:10.289729+0000","last_clean":"2024-03-06T07:20:10.289729+0000","last_became_active":"2024-03-06T07:20:07.286803+0000","last_became_peered":"2024-03-06T07:20:07.286803+0000","last_unstale":"2024-03-06T07:20:10.289729+0000","last_undegraded":"2024-03-06T07:20:10.289729+0000","last_fullsized":"2024-03-06T07:20:10.289729+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:00.451319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289744+0000","last_change":"2024-03-06T07:19:54.060233+0000","last_active":"2024-03-06T07:20:10.289744+0000","last_peered":"2024-03-06T07:20:10.289744+0000","last_clean":"2024-03-06T07:20:10.289744+0000","last_became_active":"2024-03-06T07:19:54.059901+0000","last_became_peered":"2024-03-06T07:19:54.059901+0000","last_unstale":"2024-03-06T07:20:10.289744+0000","last_undegraded":"2024-03-06T07:20:10.289744+0000","last_fullsized":"2024-03-06T07:20:10.289744+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:09.921962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":26,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281406+0000","last_change":"2024-03-06T07:19:53.042382+0000","last_active":"2024-03-06T07:20:09.281406+0000","last_peered":"2024-03-06T07:20:09.281406+0000","last_clean":"2024-03-06T07:20:09.281406+0000","last_became_active":"2024-03-06T07:19:53.041334+0000","last_became_peered":"2024-03-06T07:19:53.041334+0000","last_unstale":"2024-03-06T07:20:09.281406+0000","last_undegraded":"2024-03-06T07:20:09.281406+0000","last_fullsized":"2024-03-06T07:20:09.281406+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289462+0000","last_change":"2024-03-06T07:20:08.285257+0000","last_active":"2024-03-06T07:20:10.289462+0000","last_peered":"2024-03-06T07:20:10.289462+0000","last_clean":"2024-03-06T07:20:10.289462+0000","last_became_active":"2024-03-06T07:20:08.285019+0000","last_became_peered":"2024-03-06T07:20:08.285019+0000","last_unstale":"2024-03-06T07:20:10.289462+0000","last_undegraded":"2024-03-06T07:20:10.289462+0000","last_fullsized":"2024-03-06T07:20:10.289462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.323537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279402+0000","last_change":"2024-03-06T07:20:07.285089+0000","last_active":"2024-03-06T07:20:09.279402+0000","last_peered":"2024-03-06T07:20:09.279402+0000","last_clean":"2024-03-06T07:20:09.279402+0000","last_became_active":"2024-03-06T07:20:07.284839+0000","last_became_peered":"2024-03-06T07:20:07.284839+0000","last_unstale":"2024-03-06T07:20:09.279402+0000","last_undegraded":"2024-03-06T07:20:09.279402+0000","last_fullsized":"2024-03-06T07:20:09.279402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:50.836858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272278+0000","last_change":"2024-03-06T07:19:54.057284+0000","last_active":"2024-03-06T07:20:10.272278+0000","last_peered":"2024-03-06T07:20:10.272278+0000","last_clean":"2024-03-06T07:20:10.272278+0000","last_became_active":"2024-03-06T07:19:54.056999+0000","last_became_peered":"2024-03-06T07:19:54.056999+0000","last_unstale":"2024-03-06T07:20:10.272278+0000","last_undegraded":"2024-03-06T07:20:10.272278+0000","last_fullsized":"2024-03-06T07:20:10.272278+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289709+0000","last_change":"2024-03-06T07:19:53.048675+0000","last_active":"2024-03-06T07:20:10.289709+0000","last_peered":"2024-03-06T07:20:10.289709+0000","last_clean":"2024-03-06T07:20:10.289709+0000","last_became_active":"2024-03-06T07:19:53.048416+0000","last_became_peered":"2024-03-06T07:19:53.048416+0000","last_unstale":"2024-03-06T07:20:10.289709+0000","last_undegraded":"2024-03-06T07:20:10.289709+0000","last_fullsized":"2024-03-06T07:20:10.289709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279419+0000","last_change":"2024-03-06T07:20:08.287669+0000","last_active":"2024-03-06T07:20:09.279419+0000","last_peered":"2024-03-06T07:20:09.279419+0000","last_clean":"2024-03-06T07:20:09.279419+0000","last_became_active":"2024-03-06T07:20:08.286498+0000","last_became_peered":"2024-03-06T07:20:08.286498+0000","last_unstale":"2024-03-06T07:20:09.279419+0000","last_undegraded":"2024-03-06T07:20:09.279419+0000","last_fullsized":"2024-03-06T07:20:09.279419+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:08.648820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290235+0000","last_change":"2024-03-06T07:20:07.285244+0000","last_active":"2024-03-06T07:20:10.290235+0000","last_peered":"2024-03-06T07:20:10.290235+0000","last_clean":"2024-03-06T07:20:10.290235+0000","last_became_active":"2024-03-06T07:20:07.285012+0000","last_became_peered":"2024-03-06T07:20:07.285012+0000","last_unstale":"2024-03-06T07:20:10.290235+0000","last_undegraded":"2024-03-06T07:20:10.290235+0000","last_fullsized":"2024-03-06T07:20:10.290235+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:17.755219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280744+0000","last_change":"2024-03-06T07:19:54.052199+0000","last_active":"2024-03-06T07:20:09.280744+0000","last_peered":"2024-03-06T07:20:09.280744+0000","last_clean":"2024-03-06T07:20:09.280744+0000","last_became_active":"2024-03-06T07:19:54.051923+0000","last_became_peered":"2024-03-06T07:19:54.051923+0000","last_unstale":"2024-03-06T07:20:09.280744+0000","last_undegraded":"2024-03-06T07:20:09.280744+0000","last_fullsized":"2024-03-06T07:20:09.280744+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288720+0000","last_change":"2024-03-06T07:19:53.044281+0000","last_active":"2024-03-06T07:20:10.288720+0000","last_peered":"2024-03-06T07:20:10.288720+0000","last_clean":"2024-03-06T07:20:10.288720+0000","last_became_active":"2024-03-06T07:19:53.043841+0000","last_became_peered":"2024-03-06T07:19:53.043841+0000","last_unstale":"2024-03-06T07:20:10.288720+0000","last_undegraded":"2024-03-06T07:20:10.288720+0000","last_fullsized":"2024-03-06T07:20:10.288720+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272172+0000","last_change":"2024-03-06T07:20:08.287358+0000","last_active":"2024-03-06T07:20:10.272172+0000","last_peered":"2024-03-06T07:20:10.272172+0000","last_clean":"2024-03-06T07:20:10.272172+0000","last_became_active":"2024-03-06T07:20:08.287072+0000","last_became_peered":"2024-03-06T07:20:08.287072+0000","last_unstale":"2024-03-06T07:20:10.272172+0000","last_undegraded":"2024-03-06T07:20:10.272172+0000","last_fullsized":"2024-03-06T07:20:10.272172+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:45.089464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272133+0000","last_change":"2024-03-06T07:20:07.285494+0000","last_active":"2024-03-06T07:20:10.272133+0000","last_peered":"2024-03-06T07:20:10.272133+0000","last_clean":"2024-03-06T07:20:10.272133+0000","last_became_active":"2024-03-06T07:20:07.285238+0000","last_became_peered":"2024-03-06T07:20:07.285238+0000","last_unstale":"2024-03-06T07:20:10.272133+0000","last_undegraded":"2024-03-06T07:20:10.272133+0000","last_fullsized":"2024-03-06T07:20:10.272133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:06.316192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288331+0000","last_change":"2024-03-06T07:19:54.039974+0000","last_active":"2024-03-06T07:20:10.288331+0000","last_peered":"2024-03-06T07:20:10.288331+0000","last_clean":"2024-03-06T07:20:10.288331+0000","last_became_active":"2024-03-06T07:19:54.039675+0000","last_became_peered":"2024-03-06T07:19:54.039675+0000","last_unstale":"2024-03-06T07:20:10.288331+0000","last_undegraded":"2024-03-06T07:20:10.288331+0000","last_fullsized":"2024-03-06T07:20:10.288331+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:53.089205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272507+0000","last_change":"2024-03-06T07:19:53.051640+0000","last_active":"2024-03-06T07:20:10.272507+0000","last_peered":"2024-03-06T07:20:10.272507+0000","last_clean":"2024-03-06T07:20:10.272507+0000","last_became_active":"2024-03-06T07:19:53.051429+0000","last_became_peered":"2024-03-06T07:19:53.051429+0000","last_unstale":"2024-03-06T07:20:10.272507+0000","last_undegraded":"2024-03-06T07:20:10.272507+0000","last_fullsized":"2024-03-06T07:20:10.272507+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:09.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281249+0000","last_change":"2024-03-06T07:20:08.281703+0000","last_active":"2024-03-06T07:20:09.281249+0000","last_peered":"2024-03-06T07:20:09.281249+0000","last_clean":"2024-03-06T07:20:09.281249+0000","last_became_active":"2024-03-06T07:20:08.281437+0000","last_became_peered":"2024-03-06T07:20:08.281437+0000","last_unstale":"2024-03-06T07:20:09.281249+0000","last_undegraded":"2024-03-06T07:20:09.281249+0000","last_fullsized":"2024-03-06T07:20:09.281249+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.323537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290789+0000","last_change":"2024-03-06T07:20:07.287035+0000","last_active":"2024-03-06T07:20:10.290789+0000","last_peered":"2024-03-06T07:20:10.290789+0000","last_clean":"2024-03-06T07:20:10.290789+0000","last_became_active":"2024-03-06T07:20:07.286453+0000","last_became_peered":"2024-03-06T07:20:07.286453+0000","last_unstale":"2024-03-06T07:20:10.290789+0000","last_undegraded":"2024-03-06T07:20:10.290789+0000","last_fullsized":"2024-03-06T07:20:10.290789+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:40.634986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281098+0000","last_change":"2024-03-06T07:19:54.054949+0000","last_active":"2024-03-06T07:20:09.281098+0000","last_peered":"2024-03-06T07:20:09.281098+0000","last_clean":"2024-03-06T07:20:09.281098+0000","last_became_active":"2024-03-06T07:19:54.054661+0000","last_became_peered":"2024-03-06T07:19:54.054661+0000","last_unstale":"2024-03-06T07:20:09.281098+0000","last_undegraded":"2024-03-06T07:20:09.281098+0000","last_fullsized":"2024-03-06T07:20:09.281098+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:57.129016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279423+0000","last_change":"2024-03-06T07:19:53.042003+0000","last_active":"2024-03-06T07:20:09.279423+0000","last_peered":"2024-03-06T07:20:09.279423+0000","last_clean":"2024-03-06T07:20:09.279423+0000","last_became_active":"2024-03-06T07:19:53.041728+0000","last_became_peered":"2024-03-06T07:19:53.041728+0000","last_unstale":"2024-03-06T07:20:09.279423+0000","last_undegraded":"2024-03-06T07:20:09.279423+0000","last_fullsized":"2024-03-06T07:20:09.279423+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.964097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280369+0000","last_change":"2024-03-06T07:20:08.289513+0000","last_active":"2024-03-06T07:20:09.280369+0000","last_peered":"2024-03-06T07:20:09.280369+0000","last_clean":"2024-03-06T07:20:09.280369+0000","last_became_active":"2024-03-06T07:20:08.288775+0000","last_became_peered":"2024-03-06T07:20:08.288775+0000","last_unstale":"2024-03-06T07:20:09.280369+0000","last_undegraded":"2024-03-06T07:20:09.280369+0000","last_fullsized":"2024-03-06T07:20:09.280369+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:49.507162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288560+0000","last_change":"2024-03-06T07:20:07.278974+0000","last_active":"2024-03-06T07:20:10.288560+0000","last_peered":"2024-03-06T07:20:10.288560+0000","last_clean":"2024-03-06T07:20:10.288560+0000","last_became_active":"2024-03-06T07:20:07.278682+0000","last_became_peered":"2024-03-06T07:20:07.278682+0000","last_unstale":"2024-03-06T07:20:10.288560+0000","last_undegraded":"2024-03-06T07:20:10.288560+0000","last_fullsized":"2024-03-06T07:20:10.288560+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:58.699699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290654+0000","last_change":"2024-03-06T07:19:54.061284+0000","last_active":"2024-03-06T07:20:10.290654+0000","last_peered":"2024-03-06T07:20:10.290654+0000","last_clean":"2024-03-06T07:20:10.290654+0000","last_became_active":"2024-03-06T07:19:54.061017+0000","last_became_peered":"2024-03-06T07:19:54.061017+0000","last_unstale":"2024-03-06T07:20:10.290654+0000","last_undegraded":"2024-03-06T07:20:10.290654+0000","last_fullsized":"2024-03-06T07:20:10.290654+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279587+0000","last_change":"2024-03-06T07:19:53.041819+0000","last_active":"2024-03-06T07:20:09.279587+0000","last_peered":"2024-03-06T07:20:09.279587+0000","last_clean":"2024-03-06T07:20:09.279587+0000","last_became_active":"2024-03-06T07:19:53.041542+0000","last_became_peered":"2024-03-06T07:19:53.041542+0000","last_unstale":"2024-03-06T07:20:09.279587+0000","last_undegraded":"2024-03-06T07:20:09.279587+0000","last_fullsized":"2024-03-06T07:20:09.279587+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288468+0000","last_change":"2024-03-06T07:20:08.291722+0000","last_active":"2024-03-06T07:20:10.288468+0000","last_peered":"2024-03-06T07:20:10.288468+0000","last_clean":"2024-03-06T07:20:10.288468+0000","last_became_active":"2024-03-06T07:20:08.291460+0000","last_became_peered":"2024-03-06T07:20:08.291460+0000","last_unstale":"2024-03-06T07:20:10.288468+0000","last_undegraded":"2024-03-06T07:20:10.288468+0000","last_fullsized":"2024-03-06T07:20:10.288468+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:03.852954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290654+0000","last_change":"2024-03-06T07:20:07.289776+0000","last_active":"2024-03-06T07:20:10.290654+0000","last_peered":"2024-03-06T07:20:10.290654+0000","last_clean":"2024-03-06T07:20:10.290654+0000","last_became_active":"2024-03-06T07:20:07.289535+0000","last_became_peered":"2024-03-06T07:20:07.289535+0000","last_unstale":"2024-03-06T07:20:10.290654+0000","last_undegraded":"2024-03-06T07:20:10.290654+0000","last_fullsized":"2024-03-06T07:20:10.290654+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:54.620623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289125+0000","last_change":"2024-03-06T07:19:54.039491+0000","last_active":"2024-03-06T07:20:10.289125+0000","last_peered":"2024-03-06T07:20:10.289125+0000","last_clean":"2024-03-06T07:20:10.289125+0000","last_became_active":"2024-03-06T07:19:54.039202+0000","last_became_peered":"2024-03-06T07:19:54.039202+0000","last_unstale":"2024-03-06T07:20:10.289125+0000","last_undegraded":"2024-03-06T07:20:10.289125+0000","last_fullsized":"2024-03-06T07:20:10.289125+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:58.292241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280219+0000","last_change":"2024-03-06T07:19:53.043210+0000","last_active":"2024-03-06T07:20:09.280219+0000","last_peered":"2024-03-06T07:20:09.280219+0000","last_clean":"2024-03-06T07:20:09.280219+0000","last_became_active":"2024-03-06T07:19:53.042951+0000","last_became_peered":"2024-03-06T07:19:53.042951+0000","last_unstale":"2024-03-06T07:20:09.280219+0000","last_undegraded":"2024-03-06T07:20:09.280219+0000","last_fullsized":"2024-03-06T07:20:09.280219+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288386+0000","last_change":"2024-03-06T07:20:08.276876+0000","last_active":"2024-03-06T07:20:10.288386+0000","last_peered":"2024-03-06T07:20:10.288386+0000","last_clean":"2024-03-06T07:20:10.288386+0000","last_became_active":"2024-03-06T07:20:08.276574+0000","last_became_peered":"2024-03-06T07:20:08.276574+0000","last_unstale":"2024-03-06T07:20:10.288386+0000","last_undegraded":"2024-03-06T07:20:10.288386+0000","last_fullsized":"2024-03-06T07:20:10.288386+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:18.752703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281073+0000","last_change":"2024-03-06T07:20:07.278534+0000","last_active":"2024-03-06T07:20:09.281073+0000","last_peered":"2024-03-06T07:20:09.281073+0000","last_clean":"2024-03-06T07:20:09.281073+0000","last_became_active":"2024-03-06T07:20:07.278261+0000","last_became_peered":"2024-03-06T07:20:07.278261+0000","last_unstale":"2024-03-06T07:20:09.281073+0000","last_undegraded":"2024-03-06T07:20:09.281073+0000","last_fullsized":"2024-03-06T07:20:09.281073+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:37.356811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289390+0000","last_change":"2024-03-06T07:19:54.049884+0000","last_active":"2024-03-06T07:20:10.289390+0000","last_peered":"2024-03-06T07:20:10.289390+0000","last_clean":"2024-03-06T07:20:10.289390+0000","last_became_active":"2024-03-06T07:19:54.049597+0000","last_became_peered":"2024-03-06T07:19:54.049597+0000","last_unstale":"2024-03-06T07:20:10.289390+0000","last_undegraded":"2024-03-06T07:20:10.289390+0000","last_fullsized":"2024-03-06T07:20:10.289390+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279196+0000","last_change":"2024-03-06T07:19:53.043198+0000","last_active":"2024-03-06T07:20:09.279196+0000","last_peered":"2024-03-06T07:20:09.279196+0000","last_clean":"2024-03-06T07:20:09.279196+0000","last_became_active":"2024-03-06T07:19:53.042939+0000","last_became_peered":"2024-03-06T07:19:53.042939+0000","last_unstale":"2024-03-06T07:20:09.279196+0000","last_undegraded":"2024-03-06T07:20:09.279196+0000","last_fullsized":"2024-03-06T07:20:09.279196+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280484+0000","last_change":"2024-03-06T07:20:08.288199+0000","last_active":"2024-03-06T07:20:09.280484+0000","last_peered":"2024-03-06T07:20:09.280484+0000","last_clean":"2024-03-06T07:20:09.280484+0000","last_became_active":"2024-03-06T07:20:08.287695+0000","last_became_peered":"2024-03-06T07:20:08.287695+0000","last_unstale":"2024-03-06T07:20:09.280484+0000","last_undegraded":"2024-03-06T07:20:09.280484+0000","last_fullsized":"2024-03-06T07:20:09.280484+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:21.720175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288165+0000","last_change":"2024-03-06T07:20:07.283576+0000","last_active":"2024-03-06T07:20:10.288165+0000","last_peered":"2024-03-06T07:20:10.288165+0000","last_clean":"2024-03-06T07:20:10.288165+0000","last_became_active":"2024-03-06T07:20:07.283136+0000","last_became_peered":"2024-03-06T07:20:07.283136+0000","last_unstale":"2024-03-06T07:20:10.288165+0000","last_undegraded":"2024-03-06T07:20:10.288165+0000","last_fullsized":"2024-03-06T07:20:10.288165+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:33.612051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279526+0000","last_change":"2024-03-06T07:19:54.045809+0000","last_active":"2024-03-06T07:20:09.279526+0000","last_peered":"2024-03-06T07:20:09.279526+0000","last_clean":"2024-03-06T07:20:09.279526+0000","last_became_active":"2024-03-06T07:19:54.040263+0000","last_became_peered":"2024-03-06T07:19:54.040263+0000","last_unstale":"2024-03-06T07:20:09.279526+0000","last_undegraded":"2024-03-06T07:20:09.279526+0000","last_fullsized":"2024-03-06T07:20:09.279526+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:20.728340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290867+0000","last_change":"2024-03-06T07:19:53.047864+0000","last_active":"2024-03-06T07:20:10.290867+0000","last_peered":"2024-03-06T07:20:10.290867+0000","last_clean":"2024-03-06T07:20:10.290867+0000","last_became_active":"2024-03-06T07:19:53.047607+0000","last_became_peered":"2024-03-06T07:19:53.047607+0000","last_unstale":"2024-03-06T07:20:10.290867+0000","last_undegraded":"2024-03-06T07:20:10.290867+0000","last_fullsized":"2024-03-06T07:20:10.290867+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.148872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288473+0000","last_change":"2024-03-06T07:20:08.280295+0000","last_active":"2024-03-06T07:20:10.288473+0000","last_peered":"2024-03-06T07:20:10.288473+0000","last_clean":"2024-03-06T07:20:10.288473+0000","last_became_active":"2024-03-06T07:20:08.280012+0000","last_became_peered":"2024-03-06T07:20:08.280012+0000","last_unstale":"2024-03-06T07:20:10.288473+0000","last_undegraded":"2024-03-06T07:20:10.288473+0000","last_fullsized":"2024-03-06T07:20:10.288473+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:21.055750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279812+0000","last_change":"2024-03-06T07:20:07.285280+0000","last_active":"2024-03-06T07:20:09.279812+0000","last_peered":"2024-03-06T07:20:09.279812+0000","last_clean":"2024-03-06T07:20:09.279812+0000","last_became_active":"2024-03-06T07:20:07.284993+0000","last_became_peered":"2024-03-06T07:20:07.284993+0000","last_unstale":"2024-03-06T07:20:09.279812+0000","last_undegraded":"2024-03-06T07:20:09.279812+0000","last_fullsized":"2024-03-06T07:20:09.279812+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:23.148949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.271838+0000","last_change":"2024-03-06T07:19:54.056898+0000","last_active":"2024-03-06T07:20:10.271838+0000","last_peered":"2024-03-06T07:20:10.271838+0000","last_clean":"2024-03-06T07:20:10.271838+0000","last_became_active":"2024-03-06T07:19:54.056522+0000","last_became_peered":"2024-03-06T07:19:54.056522+0000","last_unstale":"2024-03-06T07:20:10.271838+0000","last_undegraded":"2024-03-06T07:20:10.271838+0000","last_fullsized":"2024-03-06T07:20:10.271838+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281374+0000","last_change":"2024-03-06T07:19:53.043065+0000","last_active":"2024-03-06T07:20:09.281374+0000","last_peered":"2024-03-06T07:20:09.281374+0000","last_clean":"2024-03-06T07:20:09.281374+0000","last_became_active":"2024-03-06T07:19:53.042756+0000","last_became_peered":"2024-03-06T07:19:53.042756+0000","last_unstale":"2024-03-06T07:20:09.281374+0000","last_undegraded":"2024-03-06T07:20:09.281374+0000","last_fullsized":"2024-03-06T07:20:09.281374+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.338932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274550+0000","last_change":"2024-03-06T07:20:08.283743+0000","last_active":"2024-03-06T07:20:10.274550+0000","last_peered":"2024-03-06T07:20:10.274550+0000","last_clean":"2024-03-06T07:20:10.274550+0000","last_became_active":"2024-03-06T07:20:08.283434+0000","last_became_peered":"2024-03-06T07:20:08.283434+0000","last_unstale":"2024-03-06T07:20:10.274550+0000","last_undegraded":"2024-03-06T07:20:10.274550+0000","last_fullsized":"2024-03-06T07:20:10.274550+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:42.568567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288241+0000","last_change":"2024-03-06T07:20:07.282855+0000","last_active":"2024-03-06T07:20:10.288241+0000","last_peered":"2024-03-06T07:20:10.288241+0000","last_clean":"2024-03-06T07:20:10.288241+0000","last_became_active":"2024-03-06T07:20:07.282585+0000","last_became_peered":"2024-03-06T07:20:07.282585+0000","last_unstale":"2024-03-06T07:20:10.288241+0000","last_undegraded":"2024-03-06T07:20:10.288241+0000","last_fullsized":"2024-03-06T07:20:10.288241+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:13.790304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279317+0000","last_change":"2024-03-06T07:19:54.057647+0000","last_active":"2024-03-06T07:20:09.279317+0000","last_peered":"2024-03-06T07:20:09.279317+0000","last_clean":"2024-03-06T07:20:09.279317+0000","last_became_active":"2024-03-06T07:19:54.057396+0000","last_became_peered":"2024-03-06T07:19:54.057396+0000","last_unstale":"2024-03-06T07:20:09.279317+0000","last_undegraded":"2024-03-06T07:20:09.279317+0000","last_fullsized":"2024-03-06T07:20:09.279317+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":25,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290748+0000","last_change":"2024-03-06T07:19:53.046359+0000","last_active":"2024-03-06T07:20:10.290748+0000","last_peered":"2024-03-06T07:20:10.290748+0000","last_clean":"2024-03-06T07:20:10.290748+0000","last_became_active":"2024-03-06T07:19:53.046065+0000","last_became_peered":"2024-03-06T07:19:53.046065+0000","last_unstale":"2024-03-06T07:20:10.290748+0000","last_undegraded":"2024-03-06T07:20:10.290748+0000","last_fullsized":"2024-03-06T07:20:10.290748+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:08.146267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289992+0000","last_change":"2024-03-06T07:20:08.284659+0000","last_active":"2024-03-06T07:20:10.289992+0000","last_peered":"2024-03-06T07:20:10.289992+0000","last_clean":"2024-03-06T07:20:10.289992+0000","last_became_active":"2024-03-06T07:20:08.284415+0000","last_became_peered":"2024-03-06T07:20:08.284415+0000","last_unstale":"2024-03-06T07:20:10.289992+0000","last_undegraded":"2024-03-06T07:20:10.289992+0000","last_fullsized":"2024-03-06T07:20:10.289992+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:05.954034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288642+0000","last_change":"2024-03-06T07:20:07.280915+0000","last_active":"2024-03-06T07:20:10.288642+0000","last_peered":"2024-03-06T07:20:10.288642+0000","last_clean":"2024-03-06T07:20:10.288642+0000","last_became_active":"2024-03-06T07:20:07.280659+0000","last_became_peered":"2024-03-06T07:20:07.280659+0000","last_unstale":"2024-03-06T07:20:10.288642+0000","last_undegraded":"2024-03-06T07:20:10.288642+0000","last_fullsized":"2024-03-06T07:20:10.288642+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:00.431630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280146+0000","last_change":"2024-03-06T07:19:54.041140+0000","last_active":"2024-03-06T07:20:09.280146+0000","last_peered":"2024-03-06T07:20:09.280146+0000","last_clean":"2024-03-06T07:20:09.280146+0000","last_became_active":"2024-03-06T07:19:54.040877+0000","last_became_peered":"2024-03-06T07:19:54.040877+0000","last_unstale":"2024-03-06T07:20:09.280146+0000","last_undegraded":"2024-03-06T07:20:09.280146+0000","last_fullsized":"2024-03-06T07:20:09.280146+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:29.971359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":28,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288877+0000","last_change":"2024-03-06T07:19:53.045210+0000","last_active":"2024-03-06T07:20:10.288877+0000","last_peered":"2024-03-06T07:20:10.288877+0000","last_clean":"2024-03-06T07:20:10.288877+0000","last_became_active":"2024-03-06T07:19:53.044950+0000","last_became_peered":"2024-03-06T07:19:53.044950+0000","last_unstale":"2024-03-06T07:20:10.288877+0000","last_undegraded":"2024-03-06T07:20:10.288877+0000","last_fullsized":"2024-03-06T07:20:10.288877+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:53.407226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279877+0000","last_change":"2024-03-06T07:20:08.290423+0000","last_active":"2024-03-06T07:20:09.279877+0000","last_peered":"2024-03-06T07:20:09.279877+0000","last_clean":"2024-03-06T07:20:09.279877+0000","last_became_active":"2024-03-06T07:20:08.290112+0000","last_became_peered":"2024-03-06T07:20:08.290112+0000","last_unstale":"2024-03-06T07:20:09.279877+0000","last_undegraded":"2024-03-06T07:20:09.279877+0000","last_fullsized":"2024-03-06T07:20:09.279877+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.838274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279743+0000","last_change":"2024-03-06T07:20:07.285841+0000","last_active":"2024-03-06T07:20:09.279743+0000","last_peered":"2024-03-06T07:20:09.279743+0000","last_clean":"2024-03-06T07:20:09.279743+0000","last_became_active":"2024-03-06T07:20:07.285505+0000","last_became_peered":"2024-03-06T07:20:07.285505+0000","last_unstale":"2024-03-06T07:20:09.279743+0000","last_undegraded":"2024-03-06T07:20:09.279743+0000","last_fullsized":"2024-03-06T07:20:09.279743+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:40.634986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272369+0000","last_change":"2024-03-06T07:19:54.058718+0000","last_active":"2024-03-06T07:20:10.272369+0000","last_peered":"2024-03-06T07:20:10.272369+0000","last_clean":"2024-03-06T07:20:10.272369+0000","last_became_active":"2024-03-06T07:19:54.058415+0000","last_became_peered":"2024-03-06T07:19:54.058415+0000","last_unstale":"2024-03-06T07:20:10.272369+0000","last_undegraded":"2024-03-06T07:20:10.272369+0000","last_fullsized":"2024-03-06T07:20:10.272369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279408+0000","last_change":"2024-03-06T07:19:53.046555+0000","last_active":"2024-03-06T07:20:09.279408+0000","last_peered":"2024-03-06T07:20:09.279408+0000","last_clean":"2024-03-06T07:20:09.279408+0000","last_became_active":"2024-03-06T07:19:53.046298+0000","last_became_peered":"2024-03-06T07:19:53.046298+0000","last_unstale":"2024-03-06T07:20:09.279408+0000","last_undegraded":"2024-03-06T07:20:09.279408+0000","last_fullsized":"2024-03-06T07:20:09.279408+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.243021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290624+0000","last_change":"2024-03-06T07:20:08.290149+0000","last_active":"2024-03-06T07:20:10.290624+0000","last_peered":"2024-03-06T07:20:10.290624+0000","last_clean":"2024-03-06T07:20:10.290624+0000","last_became_active":"2024-03-06T07:20:08.289623+0000","last_became_peered":"2024-03-06T07:20:08.289623+0000","last_unstale":"2024-03-06T07:20:10.290624+0000","last_undegraded":"2024-03-06T07:20:10.290624+0000","last_fullsized":"2024-03-06T07:20:10.290624+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:34.619396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279506+0000","last_change":"2024-03-06T07:20:07.285895+0000","last_active":"2024-03-06T07:20:09.279506+0000","last_peered":"2024-03-06T07:20:09.279506+0000","last_clean":"2024-03-06T07:20:09.279506+0000","last_became_active":"2024-03-06T07:20:07.285614+0000","last_became_peered":"2024-03-06T07:20:07.285614+0000","last_unstale":"2024-03-06T07:20:09.279506+0000","last_undegraded":"2024-03-06T07:20:09.279506+0000","last_fullsized":"2024-03-06T07:20:09.279506+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280187+0000","last_change":"2024-03-06T07:19:54.047654+0000","last_active":"2024-03-06T07:20:09.280187+0000","last_peered":"2024-03-06T07:20:09.280187+0000","last_clean":"2024-03-06T07:20:09.280187+0000","last_became_active":"2024-03-06T07:19:54.047339+0000","last_became_peered":"2024-03-06T07:19:54.047339+0000","last_unstale":"2024-03-06T07:20:09.280187+0000","last_undegraded":"2024-03-06T07:20:09.280187+0000","last_fullsized":"2024-03-06T07:20:09.280187+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:35.272829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280818+0000","last_change":"2024-03-06T07:19:53.040132+0000","last_active":"2024-03-06T07:20:09.280818+0000","last_peered":"2024-03-06T07:20:09.280818+0000","last_clean":"2024-03-06T07:20:09.280818+0000","last_became_active":"2024-03-06T07:19:53.039809+0000","last_became_peered":"2024-03-06T07:19:53.039809+0000","last_unstale":"2024-03-06T07:20:09.280818+0000","last_undegraded":"2024-03-06T07:20:09.280818+0000","last_fullsized":"2024-03-06T07:20:09.280818+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279774+0000","last_change":"2024-03-06T07:20:08.285155+0000","last_active":"2024-03-06T07:20:09.279774+0000","last_peered":"2024-03-06T07:20:09.279774+0000","last_clean":"2024-03-06T07:20:09.279774+0000","last_became_active":"2024-03-06T07:20:08.284896+0000","last_became_peered":"2024-03-06T07:20:08.284896+0000","last_unstale":"2024-03-06T07:20:09.279774+0000","last_undegraded":"2024-03-06T07:20:09.279774+0000","last_fullsized":"2024-03-06T07:20:09.279774+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:24.217084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289760+0000","last_change":"2024-03-06T07:20:07.288210+0000","last_active":"2024-03-06T07:20:10.289760+0000","last_peered":"2024-03-06T07:20:10.289760+0000","last_clean":"2024-03-06T07:20:10.289760+0000","last_became_active":"2024-03-06T07:20:07.287885+0000","last_became_peered":"2024-03-06T07:20:07.287885+0000","last_unstale":"2024-03-06T07:20:10.289760+0000","last_undegraded":"2024-03-06T07:20:10.289760+0000","last_fullsized":"2024-03-06T07:20:10.289760+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:15.206863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289029+0000","last_change":"2024-03-06T07:19:54.050847+0000","last_active":"2024-03-06T07:20:10.289029+0000","last_peered":"2024-03-06T07:20:10.289029+0000","last_clean":"2024-03-06T07:20:10.289029+0000","last_became_active":"2024-03-06T07:19:54.050378+0000","last_became_peered":"2024-03-06T07:19:54.050378+0000","last_unstale":"2024-03-06T07:20:10.289029+0000","last_undegraded":"2024-03-06T07:20:10.289029+0000","last_fullsized":"2024-03-06T07:20:10.289029+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290068+0000","last_change":"2024-03-06T07:19:53.045139+0000","last_active":"2024-03-06T07:20:10.290068+0000","last_peered":"2024-03-06T07:20:10.290068+0000","last_clean":"2024-03-06T07:20:10.290068+0000","last_became_active":"2024-03-06T07:19:53.044903+0000","last_became_peered":"2024-03-06T07:19:53.044903+0000","last_unstale":"2024-03-06T07:20:10.290068+0000","last_undegraded":"2024-03-06T07:20:10.290068+0000","last_fullsized":"2024-03-06T07:20:10.290068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:56.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279970+0000","last_change":"2024-03-06T07:20:08.296792+0000","last_active":"2024-03-06T07:20:09.279970+0000","last_peered":"2024-03-06T07:20:09.279970+0000","last_clean":"2024-03-06T07:20:09.279970+0000","last_became_active":"2024-03-06T07:20:08.296332+0000","last_became_peered":"2024-03-06T07:20:08.296332+0000","last_unstale":"2024-03-06T07:20:09.279970+0000","last_undegraded":"2024-03-06T07:20:09.279970+0000","last_fullsized":"2024-03-06T07:20:09.279970+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:20.006499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279921+0000","last_change":"2024-03-06T07:20:07.289518+0000","last_active":"2024-03-06T07:20:09.279921+0000","last_peered":"2024-03-06T07:20:09.279921+0000","last_clean":"2024-03-06T07:20:09.279921+0000","last_became_active":"2024-03-06T07:20:07.289213+0000","last_became_peered":"2024-03-06T07:20:07.289213+0000","last_unstale":"2024-03-06T07:20:09.279921+0000","last_undegraded":"2024-03-06T07:20:09.279921+0000","last_fullsized":"2024-03-06T07:20:09.279921+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:11.519229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279864+0000","last_change":"2024-03-06T07:19:54.059322+0000","last_active":"2024-03-06T07:20:09.279864+0000","last_peered":"2024-03-06T07:20:09.279864+0000","last_clean":"2024-03-06T07:20:09.279864+0000","last_became_active":"2024-03-06T07:19:54.059092+0000","last_became_peered":"2024-03-06T07:19:54.059092+0000","last_unstale":"2024-03-06T07:20:09.279864+0000","last_undegraded":"2024-03-06T07:20:09.279864+0000","last_fullsized":"2024-03-06T07:20:09.279864+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":30,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288081+0000","last_change":"2024-03-06T07:19:53.046815+0000","last_active":"2024-03-06T07:20:10.288081+0000","last_peered":"2024-03-06T07:20:10.288081+0000","last_clean":"2024-03-06T07:20:10.288081+0000","last_became_active":"2024-03-06T07:19:53.043687+0000","last_became_peered":"2024-03-06T07:19:53.043687+0000","last_unstale":"2024-03-06T07:20:10.288081+0000","last_undegraded":"2024-03-06T07:20:10.288081+0000","last_fullsized":"2024-03-06T07:20:10.288081+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:44.465449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288910+0000","last_change":"2024-03-06T07:20:08.284036+0000","last_active":"2024-03-06T07:20:10.288910+0000","last_peered":"2024-03-06T07:20:10.288910+0000","last_clean":"2024-03-06T07:20:10.288910+0000","last_became_active":"2024-03-06T07:20:08.283771+0000","last_became_peered":"2024-03-06T07:20:08.283771+0000","last_unstale":"2024-03-06T07:20:10.288910+0000","last_undegraded":"2024-03-06T07:20:10.288910+0000","last_fullsized":"2024-03-06T07:20:10.288910+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:04.742839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288216+0000","last_change":"2024-03-06T07:20:07.283273+0000","last_active":"2024-03-06T07:20:10.288216+0000","last_peered":"2024-03-06T07:20:10.288216+0000","last_clean":"2024-03-06T07:20:10.288216+0000","last_became_active":"2024-03-06T07:20:07.282989+0000","last_became_peered":"2024-03-06T07:20:07.282989+0000","last_unstale":"2024-03-06T07:20:10.288216+0000","last_undegraded":"2024-03-06T07:20:10.288216+0000","last_fullsized":"2024-03-06T07:20:10.288216+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:14.967028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288048+0000","last_change":"2024-03-06T07:19:54.057521+0000","last_active":"2024-03-06T07:20:10.288048+0000","last_peered":"2024-03-06T07:20:10.288048+0000","last_clean":"2024-03-06T07:20:10.288048+0000","last_became_active":"2024-03-06T07:19:54.057264+0000","last_became_peered":"2024-03-06T07:19:54.057264+0000","last_unstale":"2024-03-06T07:20:10.288048+0000","last_undegraded":"2024-03-06T07:20:10.288048+0000","last_fullsized":"2024-03-06T07:20:10.288048+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280115+0000","last_change":"2024-03-06T07:19:53.051189+0000","last_active":"2024-03-06T07:20:09.280115+0000","last_peered":"2024-03-06T07:20:09.280115+0000","last_clean":"2024-03-06T07:20:09.280115+0000","last_became_active":"2024-03-06T07:19:53.050897+0000","last_became_peered":"2024-03-06T07:19:53.050897+0000","last_unstale":"2024-03-06T07:20:09.280115+0000","last_undegraded":"2024-03-06T07:20:09.280115+0000","last_fullsized":"2024-03-06T07:20:09.280115+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274871+0000","last_change":"2024-03-06T07:20:08.292612+0000","last_active":"2024-03-06T07:20:10.274871+0000","last_peered":"2024-03-06T07:20:10.274871+0000","last_clean":"2024-03-06T07:20:10.274871+0000","last_became_active":"2024-03-06T07:20:08.292312+0000","last_became_peered":"2024-03-06T07:20:08.292312+0000","last_unstale":"2024-03-06T07:20:10.274871+0000","last_undegraded":"2024-03-06T07:20:10.274871+0000","last_fullsized":"2024-03-06T07:20:10.274871+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:34.962673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289212+0000","last_change":"2024-03-06T07:20:07.283920+0000","last_active":"2024-03-06T07:20:10.289212+0000","last_peered":"2024-03-06T07:20:10.289212+0000","last_clean":"2024-03-06T07:20:10.289212+0000","last_became_active":"2024-03-06T07:20:07.283631+0000","last_became_peered":"2024-03-06T07:20:07.283631+0000","last_unstale":"2024-03-06T07:20:10.289212+0000","last_undegraded":"2024-03-06T07:20:10.289212+0000","last_fullsized":"2024-03-06T07:20:10.289212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:13.297038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289789+0000","last_change":"2024-03-06T07:19:54.052438+0000","last_active":"2024-03-06T07:20:10.289789+0000","last_peered":"2024-03-06T07:20:10.289789+0000","last_clean":"2024-03-06T07:20:10.289789+0000","last_became_active":"2024-03-06T07:19:54.052185+0000","last_became_peered":"2024-03-06T07:19:54.052185+0000","last_unstale":"2024-03-06T07:20:10.289789+0000","last_undegraded":"2024-03-06T07:20:10.289789+0000","last_fullsized":"2024-03-06T07:20:10.289789+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.275194+0000","last_change":"2024-03-06T07:19:53.046903+0000","last_active":"2024-03-06T07:20:10.275194+0000","last_peered":"2024-03-06T07:20:10.275194+0000","last_clean":"2024-03-06T07:20:10.275194+0000","last_became_active":"2024-03-06T07:19:53.046639+0000","last_became_peered":"2024-03-06T07:19:53.046639+0000","last_unstale":"2024-03-06T07:20:10.275194+0000","last_undegraded":"2024-03-06T07:20:10.275194+0000","last_fullsized":"2024-03-06T07:20:10.275194+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:23.209022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279730+0000","last_change":"2024-03-06T07:20:08.285798+0000","last_active":"2024-03-06T07:20:09.279730+0000","last_peered":"2024-03-06T07:20:09.279730+0000","last_clean":"2024-03-06T07:20:09.279730+0000","last_became_active":"2024-03-06T07:20:08.285524+0000","last_became_peered":"2024-03-06T07:20:08.285524+0000","last_unstale":"2024-03-06T07:20:09.279730+0000","last_undegraded":"2024-03-06T07:20:09.279730+0000","last_fullsized":"2024-03-06T07:20:09.279730+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:49.507162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290343+0000","last_change":"2024-03-06T07:20:07.288781+0000","last_active":"2024-03-06T07:20:10.290343+0000","last_peered":"2024-03-06T07:20:10.290343+0000","last_clean":"2024-03-06T07:20:10.290343+0000","last_became_active":"2024-03-06T07:20:07.288548+0000","last_became_peered":"2024-03-06T07:20:07.288548+0000","last_unstale":"2024-03-06T07:20:10.290343+0000","last_undegraded":"2024-03-06T07:20:10.290343+0000","last_fullsized":"2024-03-06T07:20:10.290343+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.514155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279481+0000","last_change":"2024-03-06T07:19:54.053795+0000","last_active":"2024-03-06T07:20:09.279481+0000","last_peered":"2024-03-06T07:20:09.279481+0000","last_clean":"2024-03-06T07:20:09.279481+0000","last_became_active":"2024-03-06T07:19:54.053519+0000","last_became_peered":"2024-03-06T07:19:54.053519+0000","last_unstale":"2024-03-06T07:20:09.279481+0000","last_undegraded":"2024-03-06T07:20:09.279481+0000","last_fullsized":"2024-03-06T07:20:09.279481+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:20.250283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280490+0000","last_change":"2024-03-06T07:19:53.046151+0000","last_active":"2024-03-06T07:20:09.280490+0000","last_peered":"2024-03-06T07:20:09.280490+0000","last_clean":"2024-03-06T07:20:09.280490+0000","last_became_active":"2024-03-06T07:19:53.045797+0000","last_became_peered":"2024-03-06T07:19:53.045797+0000","last_unstale":"2024-03-06T07:20:09.280490+0000","last_undegraded":"2024-03-06T07:20:09.280490+0000","last_fullsized":"2024-03-06T07:20:09.280490+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280601+0000","last_change":"2024-03-06T07:20:08.293922+0000","last_active":"2024-03-06T07:20:09.280601+0000","last_peered":"2024-03-06T07:20:09.280601+0000","last_clean":"2024-03-06T07:20:09.280601+0000","last_became_active":"2024-03-06T07:20:08.293668+0000","last_became_peered":"2024-03-06T07:20:08.293668+0000","last_unstale":"2024-03-06T07:20:09.280601+0000","last_undegraded":"2024-03-06T07:20:09.280601+0000","last_fullsized":"2024-03-06T07:20:09.280601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:24.217084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288735+0000","last_change":"2024-03-06T07:20:07.284617+0000","last_active":"2024-03-06T07:20:10.288735+0000","last_peered":"2024-03-06T07:20:10.288735+0000","last_clean":"2024-03-06T07:20:10.288735+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:10.288735+0000","last_undegraded":"2024-03-06T07:20:10.288735+0000","last_fullsized":"2024-03-06T07:20:10.288735+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:28.252791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289335+0000","last_change":"2024-03-06T07:19:54.055324+0000","last_active":"2024-03-06T07:20:10.289335+0000","last_peered":"2024-03-06T07:20:10.289335+0000","last_clean":"2024-03-06T07:20:10.289335+0000","last_became_active":"2024-03-06T07:19:54.055081+0000","last_became_peered":"2024-03-06T07:19:54.055081+0000","last_unstale":"2024-03-06T07:20:10.289335+0000","last_undegraded":"2024-03-06T07:20:10.289335+0000","last_fullsized":"2024-03-06T07:20:10.289335+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:57.518725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280051+0000","last_change":"2024-03-06T07:19:53.048898+0000","last_active":"2024-03-06T07:20:09.280051+0000","last_peered":"2024-03-06T07:20:09.280051+0000","last_clean":"2024-03-06T07:20:09.280051+0000","last_became_active":"2024-03-06T07:19:53.048613+0000","last_became_peered":"2024-03-06T07:19:53.048613+0000","last_unstale":"2024-03-06T07:20:09.280051+0000","last_undegraded":"2024-03-06T07:20:09.280051+0000","last_fullsized":"2024-03-06T07:20:09.280051+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272659+0000","last_change":"2024-03-06T07:20:08.287444+0000","last_active":"2024-03-06T07:20:10.272659+0000","last_peered":"2024-03-06T07:20:10.272659+0000","last_clean":"2024-03-06T07:20:10.272659+0000","last_became_active":"2024-03-06T07:20:08.287125+0000","last_became_peered":"2024-03-06T07:20:08.287125+0000","last_unstale":"2024-03-06T07:20:10.272659+0000","last_undegraded":"2024-03-06T07:20:10.272659+0000","last_fullsized":"2024-03-06T07:20:10.272659+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:01.438975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279826+0000","last_change":"2024-03-06T07:20:07.285007+0000","last_active":"2024-03-06T07:20:09.279826+0000","last_peered":"2024-03-06T07:20:09.279826+0000","last_clean":"2024-03-06T07:20:09.279826+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:09.279826+0000","last_undegraded":"2024-03-06T07:20:09.279826+0000","last_fullsized":"2024-03-06T07:20:09.279826+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:55.549631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.271883+0000","last_change":"2024-03-06T07:19:54.056310+0000","last_active":"2024-03-06T07:20:10.271883+0000","last_peered":"2024-03-06T07:20:10.271883+0000","last_clean":"2024-03-06T07:20:10.271883+0000","last_became_active":"2024-03-06T07:19:54.056039+0000","last_became_peered":"2024-03-06T07:19:54.056039+0000","last_unstale":"2024-03-06T07:20:10.271883+0000","last_undegraded":"2024-03-06T07:20:10.271883+0000","last_fullsized":"2024-03-06T07:20:10.271883+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289547+0000","last_change":"2024-03-06T07:19:53.043949+0000","last_active":"2024-03-06T07:20:10.289547+0000","last_peered":"2024-03-06T07:20:10.289547+0000","last_clean":"2024-03-06T07:20:10.289547+0000","last_became_active":"2024-03-06T07:19:53.043639+0000","last_became_peered":"2024-03-06T07:19:53.043639+0000","last_unstale":"2024-03-06T07:20:10.289547+0000","last_undegraded":"2024-03-06T07:20:10.289547+0000","last_fullsized":"2024-03-06T07:20:10.289547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281525+0000","last_change":"2024-03-06T07:20:08.282466+0000","last_active":"2024-03-06T07:20:09.281525+0000","last_peered":"2024-03-06T07:20:09.281525+0000","last_clean":"2024-03-06T07:20:09.281525+0000","last_became_active":"2024-03-06T07:20:08.282002+0000","last_became_peered":"2024-03-06T07:20:08.282002+0000","last_unstale":"2024-03-06T07:20:09.281525+0000","last_undegraded":"2024-03-06T07:20:09.281525+0000","last_fullsized":"2024-03-06T07:20:09.281525+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:18.762564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288667+0000","last_change":"2024-03-06T07:20:07.279146+0000","last_active":"2024-03-06T07:20:10.288667+0000","last_peered":"2024-03-06T07:20:10.288667+0000","last_clean":"2024-03-06T07:20:10.288667+0000","last_became_active":"2024-03-06T07:20:07.278872+0000","last_became_peered":"2024-03-06T07:20:07.278872+0000","last_unstale":"2024-03-06T07:20:10.288667+0000","last_undegraded":"2024-03-06T07:20:10.288667+0000","last_fullsized":"2024-03-06T07:20:10.288667+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:07.641475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280754+0000","last_change":"2024-03-06T07:19:54.040887+0000","last_active":"2024-03-06T07:20:09.280754+0000","last_peered":"2024-03-06T07:20:09.280754+0000","last_clean":"2024-03-06T07:20:09.280754+0000","last_became_active":"2024-03-06T07:19:54.040632+0000","last_became_peered":"2024-03-06T07:19:54.040632+0000","last_unstale":"2024-03-06T07:20:09.280754+0000","last_undegraded":"2024-03-06T07:20:09.280754+0000","last_fullsized":"2024-03-06T07:20:09.280754+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288832+0000","last_change":"2024-03-06T07:19:53.044330+0000","last_active":"2024-03-06T07:20:10.288832+0000","last_peered":"2024-03-06T07:20:10.288832+0000","last_clean":"2024-03-06T07:20:10.288832+0000","last_became_active":"2024-03-06T07:19:53.044066+0000","last_became_peered":"2024-03-06T07:19:53.044066+0000","last_unstale":"2024-03-06T07:20:10.288832+0000","last_undegraded":"2024-03-06T07:20:10.288832+0000","last_fullsized":"2024-03-06T07:20:10.288832+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:19.476961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279133+0000","last_change":"2024-03-06T07:20:08.284792+0000","last_active":"2024-03-06T07:20:09.279133+0000","last_peered":"2024-03-06T07:20:09.279133+0000","last_clean":"2024-03-06T07:20:09.279133+0000","last_became_active":"2024-03-06T07:20:08.284533+0000","last_became_peered":"2024-03-06T07:20:08.284533+0000","last_unstale":"2024-03-06T07:20:09.279133+0000","last_undegraded":"2024-03-06T07:20:09.279133+0000","last_fullsized":"2024-03-06T07:20:09.279133+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:09.713122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272685+0000","last_change":"2024-03-06T07:20:07.287646+0000","last_active":"2024-03-06T07:20:10.272685+0000","last_peered":"2024-03-06T07:20:10.272685+0000","last_clean":"2024-03-06T07:20:10.272685+0000","last_became_active":"2024-03-06T07:20:07.287378+0000","last_became_peered":"2024-03-06T07:20:07.287378+0000","last_unstale":"2024-03-06T07:20:10.272685+0000","last_undegraded":"2024-03-06T07:20:10.272685+0000","last_fullsized":"2024-03-06T07:20:10.272685+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.514155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280084+0000","last_change":"2024-03-06T07:19:54.049940+0000","last_active":"2024-03-06T07:20:09.280084+0000","last_peered":"2024-03-06T07:20:09.280084+0000","last_clean":"2024-03-06T07:20:09.280084+0000","last_became_active":"2024-03-06T07:19:54.049481+0000","last_became_peered":"2024-03-06T07:19:54.049481+0000","last_unstale":"2024-03-06T07:20:09.280084+0000","last_undegraded":"2024-03-06T07:20:09.280084+0000","last_fullsized":"2024-03-06T07:20:09.280084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279297+0000","last_change":"2024-03-06T07:19:53.048472+0000","last_active":"2024-03-06T07:20:09.279297+0000","last_peered":"2024-03-06T07:20:09.279297+0000","last_clean":"2024-03-06T07:20:09.279297+0000","last_became_active":"2024-03-06T07:19:53.048213+0000","last_became_peered":"2024-03-06T07:19:53.048213+0000","last_unstale":"2024-03-06T07:20:09.279297+0000","last_undegraded":"2024-03-06T07:20:09.279297+0000","last_fullsized":"2024-03-06T07:20:09.279297+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:19.721078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.275107+0000","last_change":"2024-03-06T07:20:08.285541+0000","last_active":"2024-03-06T07:20:10.275107+0000","last_peered":"2024-03-06T07:20:10.275107+0000","last_clean":"2024-03-06T07:20:10.275107+0000","last_became_active":"2024-03-06T07:20:08.285278+0000","last_became_peered":"2024-03-06T07:20:08.285278+0000","last_unstale":"2024-03-06T07:20:10.275107+0000","last_undegraded":"2024-03-06T07:20:10.275107+0000","last_fullsized":"2024-03-06T07:20:10.275107+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:34.484615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272590+0000","last_change":"2024-03-06T07:20:07.283238+0000","last_active":"2024-03-06T07:20:10.272590+0000","last_peered":"2024-03-06T07:20:10.272590+0000","last_clean":"2024-03-06T07:20:10.272590+0000","last_became_active":"2024-03-06T07:20:07.282996+0000","last_became_peered":"2024-03-06T07:20:07.282996+0000","last_unstale":"2024-03-06T07:20:10.272590+0000","last_undegraded":"2024-03-06T07:20:10.272590+0000","last_fullsized":"2024-03-06T07:20:10.272590+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:17.755219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289845+0000","last_change":"2024-03-06T07:19:54.042432+0000","last_active":"2024-03-06T07:20:10.289845+0000","last_peered":"2024-03-06T07:20:10.289845+0000","last_clean":"2024-03-06T07:20:10.289845+0000","last_became_active":"2024-03-06T07:19:54.042166+0000","last_became_peered":"2024-03-06T07:19:54.042166+0000","last_unstale":"2024-03-06T07:20:10.289845+0000","last_undegraded":"2024-03-06T07:20:10.289845+0000","last_fullsized":"2024-03-06T07:20:10.289845+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.231274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274773+0000","last_change":"2024-03-06T07:19:53.044950+0000","last_active":"2024-03-06T07:20:10.274773+0000","last_peered":"2024-03-06T07:20:10.274773+0000","last_clean":"2024-03-06T07:20:10.274773+0000","last_became_active":"2024-03-06T07:19:53.044630+0000","last_became_peered":"2024-03-06T07:19:53.044630+0000","last_unstale":"2024-03-06T07:20:10.274773+0000","last_undegraded":"2024-03-06T07:20:10.274773+0000","last_fullsized":"2024-03-06T07:20:10.274773+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:05.924599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280669+0000","last_change":"2024-03-06T07:20:08.287941+0000","last_active":"2024-03-06T07:20:09.280669+0000","last_peered":"2024-03-06T07:20:09.280669+0000","last_clean":"2024-03-06T07:20:09.280669+0000","last_became_active":"2024-03-06T07:20:08.287332+0000","last_became_peered":"2024-03-06T07:20:08.287332+0000","last_unstale":"2024-03-06T07:20:09.280669+0000","last_undegraded":"2024-03-06T07:20:09.280669+0000","last_fullsized":"2024-03-06T07:20:09.280669+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:20.839748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.278917+0000","last_change":"2024-03-06T07:20:07.287051+0000","last_active":"2024-03-06T07:20:09.278917+0000","last_peered":"2024-03-06T07:20:09.278917+0000","last_clean":"2024-03-06T07:20:09.278917+0000","last_became_active":"2024-03-06T07:20:07.286569+0000","last_became_peered":"2024-03-06T07:20:07.286569+0000","last_unstale":"2024-03-06T07:20:09.278917+0000","last_undegraded":"2024-03-06T07:20:09.278917+0000","last_fullsized":"2024-03-06T07:20:09.278917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:02.341734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272137+0000","last_change":"2024-03-06T07:19:54.059591+0000","last_active":"2024-03-06T07:20:10.272137+0000","last_peered":"2024-03-06T07:20:10.272137+0000","last_clean":"2024-03-06T07:20:10.272137+0000","last_became_active":"2024-03-06T07:19:54.058814+0000","last_became_peered":"2024-03-06T07:19:54.058814+0000","last_unstale":"2024-03-06T07:20:10.272137+0000","last_undegraded":"2024-03-06T07:20:10.272137+0000","last_fullsized":"2024-03-06T07:20:10.272137+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281499+0000","last_change":"2024-03-06T07:19:53.040833+0000","last_active":"2024-03-06T07:20:09.281499+0000","last_peered":"2024-03-06T07:20:09.281499+0000","last_clean":"2024-03-06T07:20:09.281499+0000","last_became_active":"2024-03-06T07:19:53.040561+0000","last_became_peered":"2024-03-06T07:19:53.040561+0000","last_unstale":"2024-03-06T07:20:09.281499+0000","last_undegraded":"2024-03-06T07:20:09.281499+0000","last_fullsized":"2024-03-06T07:20:09.281499+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:12.315549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279011+0000","last_change":"2024-03-06T07:20:08.291312+0000","last_active":"2024-03-06T07:20:09.279011+0000","last_peered":"2024-03-06T07:20:09.279011+0000","last_clean":"2024-03-06T07:20:09.279011+0000","last_became_active":"2024-03-06T07:20:08.290482+0000","last_became_peered":"2024-03-06T07:20:08.290482+0000","last_unstale":"2024-03-06T07:20:09.279011+0000","last_undegraded":"2024-03-06T07:20:09.279011+0000","last_fullsized":"2024-03-06T07:20:09.279011+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:45.089464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279311+0000","last_change":"2024-03-06T07:20:07.280861+0000","last_active":"2024-03-06T07:20:09.279311+0000","last_peered":"2024-03-06T07:20:09.279311+0000","last_clean":"2024-03-06T07:20:09.279311+0000","last_became_active":"2024-03-06T07:20:07.280598+0000","last_became_peered":"2024-03-06T07:20:07.280598+0000","last_unstale":"2024-03-06T07:20:09.279311+0000","last_undegraded":"2024-03-06T07:20:09.279311+0000","last_fullsized":"2024-03-06T07:20:09.279311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:54.620623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289224+0000","last_change":"2024-03-06T07:19:54.042216+0000","last_active":"2024-03-06T07:20:10.289224+0000","last_peered":"2024-03-06T07:20:10.289224+0000","last_clean":"2024-03-06T07:20:10.289224+0000","last_became_active":"2024-03-06T07:19:54.041923+0000","last_became_peered":"2024-03-06T07:19:54.041923+0000","last_unstale":"2024-03-06T07:20:10.289224+0000","last_undegraded":"2024-03-06T07:20:10.289224+0000","last_fullsized":"2024-03-06T07:20:10.289224+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290843+0000","last_change":"2024-03-06T07:19:53.047959+0000","last_active":"2024-03-06T07:20:10.290843+0000","last_peered":"2024-03-06T07:20:10.290843+0000","last_clean":"2024-03-06T07:20:10.290843+0000","last_became_active":"2024-03-06T07:19:53.047680+0000","last_became_peered":"2024-03-06T07:19:53.047680+0000","last_unstale":"2024-03-06T07:20:10.290843+0000","last_undegraded":"2024-03-06T07:20:10.290843+0000","last_fullsized":"2024-03-06T07:20:10.290843+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:38.499903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272182+0000","last_change":"2024-03-06T07:20:08.279705+0000","last_active":"2024-03-06T07:20:10.272182+0000","last_peered":"2024-03-06T07:20:10.272182+0000","last_clean":"2024-03-06T07:20:10.272182+0000","last_became_active":"2024-03-06T07:20:08.279448+0000","last_became_peered":"2024-03-06T07:20:08.279448+0000","last_unstale":"2024-03-06T07:20:10.272182+0000","last_undegraded":"2024-03-06T07:20:10.272182+0000","last_fullsized":"2024-03-06T07:20:10.272182+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:53.510411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272225+0000","last_change":"2024-03-06T07:20:07.284739+0000","last_active":"2024-03-06T07:20:10.272225+0000","last_peered":"2024-03-06T07:20:10.272225+0000","last_clean":"2024-03-06T07:20:10.272225+0000","last_became_active":"2024-03-06T07:20:07.284477+0000","last_became_peered":"2024-03-06T07:20:07.284477+0000","last_unstale":"2024-03-06T07:20:10.272225+0000","last_undegraded":"2024-03-06T07:20:10.272225+0000","last_fullsized":"2024-03-06T07:20:10.272225+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:00.451319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287761+0000","last_change":"2024-03-06T07:19:54.043516+0000","last_active":"2024-03-06T07:20:10.287761+0000","last_peered":"2024-03-06T07:20:10.287761+0000","last_clean":"2024-03-06T07:20:10.287761+0000","last_became_active":"2024-03-06T07:19:54.041651+0000","last_became_peered":"2024-03-06T07:19:54.041651+0000","last_unstale":"2024-03-06T07:20:10.287761+0000","last_undegraded":"2024-03-06T07:20:10.287761+0000","last_fullsized":"2024-03-06T07:20:10.287761+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:04.528232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279699+0000","last_change":"2024-03-06T07:19:53.048966+0000","last_active":"2024-03-06T07:20:09.279699+0000","last_peered":"2024-03-06T07:20:09.279699+0000","last_clean":"2024-03-06T07:20:09.279699+0000","last_became_active":"2024-03-06T07:19:53.047988+0000","last_became_peered":"2024-03-06T07:19:53.047988+0000","last_unstale":"2024-03-06T07:20:09.279699+0000","last_undegraded":"2024-03-06T07:20:09.279699+0000","last_fullsized":"2024-03-06T07:20:09.279699+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:49.519186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281429+0000","last_change":"2024-03-06T07:20:08.296538+0000","last_active":"2024-03-06T07:20:09.281429+0000","last_peered":"2024-03-06T07:20:09.281429+0000","last_clean":"2024-03-06T07:20:09.281429+0000","last_became_active":"2024-03-06T07:20:08.296263+0000","last_became_peered":"2024-03-06T07:20:08.296263+0000","last_unstale":"2024-03-06T07:20:09.281429+0000","last_undegraded":"2024-03-06T07:20:09.281429+0000","last_fullsized":"2024-03-06T07:20:09.281429+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:48.612666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290324+0000","last_change":"2024-03-06T07:20:07.288004+0000","last_active":"2024-03-06T07:20:10.290324+0000","last_peered":"2024-03-06T07:20:10.290324+0000","last_clean":"2024-03-06T07:20:10.290324+0000","last_became_active":"2024-03-06T07:20:07.287745+0000","last_became_peered":"2024-03-06T07:20:07.287745+0000","last_unstale":"2024-03-06T07:20:10.290324+0000","last_undegraded":"2024-03-06T07:20:10.290324+0000","last_fullsized":"2024-03-06T07:20:10.290324+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:02.341734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279694+0000","last_change":"2024-03-06T07:19:54.062368+0000","last_active":"2024-03-06T07:20:09.279694+0000","last_peered":"2024-03-06T07:20:09.279694+0000","last_clean":"2024-03-06T07:20:09.279694+0000","last_became_active":"2024-03-06T07:19:54.060740+0000","last_became_peered":"2024-03-06T07:19:54.060740+0000","last_unstale":"2024-03-06T07:20:09.279694+0000","last_undegraded":"2024-03-06T07:20:09.279694+0000","last_fullsized":"2024-03-06T07:20:09.279694+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288557+0000","last_change":"2024-03-06T07:19:53.044944+0000","last_active":"2024-03-06T07:20:10.288557+0000","last_peered":"2024-03-06T07:20:10.288557+0000","last_clean":"2024-03-06T07:20:10.288557+0000","last_became_active":"2024-03-06T07:19:53.038900+0000","last_became_peered":"2024-03-06T07:19:53.038900+0000","last_unstale":"2024-03-06T07:20:10.288557+0000","last_undegraded":"2024-03-06T07:20:10.288557+0000","last_fullsized":"2024-03-06T07:20:10.288557+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:59.062789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281418+0000","last_change":"2024-03-06T07:20:08.282575+0000","last_active":"2024-03-06T07:20:09.281418+0000","last_peered":"2024-03-06T07:20:09.281418+0000","last_clean":"2024-03-06T07:20:09.281418+0000","last_became_active":"2024-03-06T07:20:08.282325+0000","last_became_peered":"2024-03-06T07:20:08.282325+0000","last_unstale":"2024-03-06T07:20:09.281418+0000","last_undegraded":"2024-03-06T07:20:09.281418+0000","last_fullsized":"2024-03-06T07:20:09.281418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.458664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288696+0000","last_change":"2024-03-06T07:20:07.281920+0000","last_active":"2024-03-06T07:20:10.288696+0000","last_peered":"2024-03-06T07:20:10.288696+0000","last_clean":"2024-03-06T07:20:10.288696+0000","last_became_active":"2024-03-06T07:20:07.281654+0000","last_became_peered":"2024-03-06T07:20:07.281654+0000","last_unstale":"2024-03-06T07:20:10.288696+0000","last_undegraded":"2024-03-06T07:20:10.288696+0000","last_fullsized":"2024-03-06T07:20:10.288696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:14.013690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272500+0000","last_change":"2024-03-06T07:19:54.059347+0000","last_active":"2024-03-06T07:20:10.272500+0000","last_peered":"2024-03-06T07:20:10.272500+0000","last_clean":"2024-03-06T07:20:10.272500+0000","last_became_active":"2024-03-06T07:19:54.059083+0000","last_became_peered":"2024-03-06T07:19:54.059083+0000","last_unstale":"2024-03-06T07:20:10.272500+0000","last_undegraded":"2024-03-06T07:20:10.272500+0000","last_fullsized":"2024-03-06T07:20:10.272500+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288915+0000","last_change":"2024-03-06T07:19:53.048645+0000","last_active":"2024-03-06T07:20:10.288915+0000","last_peered":"2024-03-06T07:20:10.288915+0000","last_clean":"2024-03-06T07:20:10.288915+0000","last_became_active":"2024-03-06T07:19:53.048406+0000","last_became_peered":"2024-03-06T07:19:53.048406+0000","last_unstale":"2024-03-06T07:20:10.288915+0000","last_undegraded":"2024-03-06T07:20:10.288915+0000","last_fullsized":"2024-03-06T07:20:10.288915+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:10.846361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290445+0000","last_change":"2024-03-06T07:20:08.284610+0000","last_active":"2024-03-06T07:20:10.290445+0000","last_peered":"2024-03-06T07:20:10.290445+0000","last_clean":"2024-03-06T07:20:10.290445+0000","last_became_active":"2024-03-06T07:20:08.284343+0000","last_became_peered":"2024-03-06T07:20:08.284343+0000","last_unstale":"2024-03-06T07:20:10.290445+0000","last_undegraded":"2024-03-06T07:20:10.290445+0000","last_fullsized":"2024-03-06T07:20:10.290445+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:30.212401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272571+0000","last_change":"2024-03-06T07:20:07.287091+0000","last_active":"2024-03-06T07:20:10.272571+0000","last_peered":"2024-03-06T07:20:10.272571+0000","last_clean":"2024-03-06T07:20:10.272571+0000","last_became_active":"2024-03-06T07:20:07.286303+0000","last_became_peered":"2024-03-06T07:20:07.286303+0000","last_unstale":"2024-03-06T07:20:10.272571+0000","last_undegraded":"2024-03-06T07:20:10.272571+0000","last_fullsized":"2024-03-06T07:20:10.272571+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279975+0000","last_change":"2024-03-06T07:19:54.049719+0000","last_active":"2024-03-06T07:20:09.279975+0000","last_peered":"2024-03-06T07:20:09.279975+0000","last_clean":"2024-03-06T07:20:09.279975+0000","last_became_active":"2024-03-06T07:19:54.049222+0000","last_became_peered":"2024-03-06T07:19:54.049222+0000","last_unstale":"2024-03-06T07:20:09.279975+0000","last_undegraded":"2024-03-06T07:20:09.279975+0000","last_fullsized":"2024-03-06T07:20:09.279975+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280893+0000","last_change":"2024-03-06T07:19:53.045046+0000","last_active":"2024-03-06T07:20:09.280893+0000","last_peered":"2024-03-06T07:20:09.280893+0000","last_clean":"2024-03-06T07:20:09.280893+0000","last_became_active":"2024-03-06T07:19:53.044751+0000","last_became_peered":"2024-03-06T07:19:53.044751+0000","last_unstale":"2024-03-06T07:20:09.280893+0000","last_undegraded":"2024-03-06T07:20:09.280893+0000","last_fullsized":"2024-03-06T07:20:09.280893+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280351+0000","last_change":"2024-03-06T07:20:08.289713+0000","last_active":"2024-03-06T07:20:09.280351+0000","last_peered":"2024-03-06T07:20:09.280351+0000","last_clean":"2024-03-06T07:20:09.280351+0000","last_became_active":"2024-03-06T07:20:08.289464+0000","last_became_peered":"2024-03-06T07:20:08.289464+0000","last_unstale":"2024-03-06T07:20:09.280351+0000","last_undegraded":"2024-03-06T07:20:09.280351+0000","last_fullsized":"2024-03-06T07:20:09.280351+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:16.734109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290664+0000","last_change":"2024-03-06T07:20:07.287811+0000","last_active":"2024-03-06T07:20:10.290664+0000","last_peered":"2024-03-06T07:20:10.290664+0000","last_clean":"2024-03-06T07:20:10.290664+0000","last_became_active":"2024-03-06T07:20:07.287540+0000","last_became_peered":"2024-03-06T07:20:07.287540+0000","last_unstale":"2024-03-06T07:20:10.290664+0000","last_undegraded":"2024-03-06T07:20:10.290664+0000","last_fullsized":"2024-03-06T07:20:10.290664+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289255+0000","last_change":"2024-03-06T07:19:54.039796+0000","last_active":"2024-03-06T07:20:10.289255+0000","last_peered":"2024-03-06T07:20:10.289255+0000","last_clean":"2024-03-06T07:20:10.289255+0000","last_became_active":"2024-03-06T07:19:54.039527+0000","last_became_peered":"2024-03-06T07:19:54.039527+0000","last_unstale":"2024-03-06T07:20:10.289255+0000","last_undegraded":"2024-03-06T07:20:10.289255+0000","last_fullsized":"2024-03-06T07:20:10.289255+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:49.114747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272213+0000","last_change":"2024-03-06T07:19:53.043599+0000","last_active":"2024-03-06T07:20:10.272213+0000","last_peered":"2024-03-06T07:20:10.272213+0000","last_clean":"2024-03-06T07:20:10.272213+0000","last_became_active":"2024-03-06T07:19:53.043294+0000","last_became_peered":"2024-03-06T07:19:53.043294+0000","last_unstale":"2024-03-06T07:20:10.272213+0000","last_undegraded":"2024-03-06T07:20:10.272213+0000","last_fullsized":"2024-03-06T07:20:10.272213+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:56.511464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281081+0000","last_change":"2024-03-06T07:20:08.284337+0000","last_active":"2024-03-06T07:20:09.281081+0000","last_peered":"2024-03-06T07:20:09.281081+0000","last_clean":"2024-03-06T07:20:09.281081+0000","last_became_active":"2024-03-06T07:20:08.284098+0000","last_became_peered":"2024-03-06T07:20:08.284098+0000","last_unstale":"2024-03-06T07:20:09.281081+0000","last_undegraded":"2024-03-06T07:20:09.281081+0000","last_fullsized":"2024-03-06T07:20:09.281081+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:04.742839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287921+0000","last_change":"2024-03-06T07:20:07.277710+0000","last_active":"2024-03-06T07:20:10.287921+0000","last_peered":"2024-03-06T07:20:10.287921+0000","last_clean":"2024-03-06T07:20:10.287921+0000","last_became_active":"2024-03-06T07:20:07.277346+0000","last_became_peered":"2024-03-06T07:20:07.277346+0000","last_unstale":"2024-03-06T07:20:10.287921+0000","last_undegraded":"2024-03-06T07:20:10.287921+0000","last_fullsized":"2024-03-06T07:20:10.287921+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:50.955230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280174+0000","last_change":"2024-03-06T07:19:54.055814+0000","last_active":"2024-03-06T07:20:09.280174+0000","last_peered":"2024-03-06T07:20:09.280174+0000","last_clean":"2024-03-06T07:20:09.280174+0000","last_became_active":"2024-03-06T07:19:54.055480+0000","last_became_peered":"2024-03-06T07:19:54.055480+0000","last_unstale":"2024-03-06T07:20:09.280174+0000","last_undegraded":"2024-03-06T07:20:09.280174+0000","last_fullsized":"2024-03-06T07:20:09.280174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:35.156134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280086+0000","last_change":"2024-03-06T07:19:53.046120+0000","last_active":"2024-03-06T07:20:09.280086+0000","last_peered":"2024-03-06T07:20:09.280086+0000","last_clean":"2024-03-06T07:20:09.280086+0000","last_became_active":"2024-03-06T07:19:53.045816+0000","last_became_peered":"2024-03-06T07:19:53.045816+0000","last_unstale":"2024-03-06T07:20:09.280086+0000","last_undegraded":"2024-03-06T07:20:09.280086+0000","last_fullsized":"2024-03-06T07:20:09.280086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:56.226679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":238,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289815+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:20:10.289815+0000","last_peered":"2024-03-06T07:20:10.289815+0000","last_clean":"2024-03-06T07:20:10.289815+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:20:10.289815+0000","last_undegraded":"2024-03-06T07:20:10.289815+0000","last_fullsized":"2024-03-06T07:20:10.289815+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":757760,"data_stored":897510,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":679936,"data_stored":306928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":306264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":308436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":307876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":679936,"data_stored":307854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":308778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":741376,"data_stored":896666,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:13.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:13.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:15.878 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:15.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.979+0000 7f6f9026f640 1 Processor -- start 2024-03-06T07:20:15.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.979+0000 7f6f9026f640 1 -- start start 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f9026f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f8814f360 0x7f6f88155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f9026f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f88158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f9026f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f880c73d0 con 0x7f6f880c88c0 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f9026f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f880c8dd0 con 0x7f6f8814f360 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f9026f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880be2e0 con 0x7f6f88158b60 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f8d80a640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6f880c88c0 0x7f6f880c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38616/0 (socket says 172.21.15.94:38616) 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f8d80a640 1 -- 172.21.15.94:0/1737387653 learned_addr learned my addr 172.21.15.94:0/1737387653 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f8e00b640 1 --2- 172.21.15.94:0/1737387653 >> v2:172.21.15.145:3300/0 conn(0x7f6f8814f360 0x7f6f88155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f8e80c640 1 --2- 172.21.15.94:0/1737387653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f88158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f8d009640 1 -- 172.21.15.94:0/1737387653 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962260733 0 0) 0x7f6f880c73d0 con 0x7f6f880c88c0 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.980+0000 7f6f8d009640 1 -- 172.21.15.94:0/1737387653 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f74003520 con 0x7f6f880c88c0 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.981+0000 7f6f8e80c640 1 -- 172.21.15.94:0/1737387653 >> v2:172.21.15.145:3300/0 conn(0x7f6f8814f360 msgr2=0x7f6f88155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:15.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.981+0000 7f6f8e80c640 1 --2- 172.21.15.94:0/1737387653 >> v2:172.21.15.145:3300/0 conn(0x7f6f8814f360 0x7f6f88155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.981+0000 7f6f8e80c640 1 -- 172.21.15.94:0/1737387653 >> v1:172.21.15.94:6789/0 conn(0x7f6f880c88c0 legacy=0x7f6f880c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.981+0000 7f6f8e80c640 1 -- 172.21.15.94:0/1737387653 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f880c7670 con 0x7f6f88158b60 2024-03-06T07:20:15.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.981+0000 7f6f8e00b640 1 --2- 172.21.15.94:0/1737387653 >> v2:172.21.15.145:3300/0 conn(0x7f6f8814f360 0x7f6f88155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:15.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.981+0000 7f6f8e80c640 1 --2- 172.21.15.94:0/1737387653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f88158ff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009d90 tx=0x7f6f8002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1feacc21e054ca38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f8d009640 1 -- 172.21.15.94:0/1737387653 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80039070 con 0x7f6f88158b60 2024-03-06T07:20:15.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f8d009640 1 -- 172.21.15.94:0/1737387653 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8002cbd0 con 0x7f6f88158b60 2024-03-06T07:20:15.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f8d009640 1 -- 172.21.15.94:0/1737387653 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80034440 con 0x7f6f88158b60 2024-03-06T07:20:15.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 -- 172.21.15.94:0/1737387653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 msgr2=0x7f6f88158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 --2- 172.21.15.94:0/1737387653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f88158ff0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009d90 tx=0x7f6f8002be20 comp rx=0 tx=0).stop 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 -- 172.21.15.94:0/1737387653 shutdown_connections 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 --2- 172.21.15.94:0/1737387653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f88158ff0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 --2- 172.21.15.94:0/1737387653 >> v2:172.21.15.145:3300/0 conn(0x7f6f8814f360 0x7f6f88155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 -- 172.21.15.94:0/1737387653 >> 172.21.15.94:0/1737387653 conn(0x7f6f880ac570 msgr2=0x7f6f880ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 -- 172.21.15.94:0/1737387653 shutdown_connections 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.982+0000 7f6f9026f640 1 -- 172.21.15.94:0/1737387653 wait complete. 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.983+0000 7f6f9026f640 1 Processor -- start 2024-03-06T07:20:15.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.983+0000 7f6f9026f640 1 -- start start 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f9026f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f880c88c0 0x7f6f881ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f9026f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f8814f360 0x7f6f881ee770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f9026f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f8814b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f9026f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880c5a50 con 0x7f6f8814f360 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f9026f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880c7c30 con 0x7f6f880c88c0 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f9026f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880c5bd0 con 0x7f6f88158b60 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f8814f360 0x7f6f881ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f8814f360 0x7f6f881ee770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37734/0 (socket says 172.21.15.94:37734) 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 -- 172.21.15.94:0/294823747 learned_addr learned my addr 172.21.15.94:0/294823747 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8e00b640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f880c88c0 0x7f6f881ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8e80c640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f8814b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 -- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 msgr2=0x7f6f8814b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f8814b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 -- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f880c88c0 msgr2=0x7f6f881ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f880c88c0 0x7f6f881ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.984+0000 7f6f8d80a640 1 -- 172.21.15.94:0/294823747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f881ebf60 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.985+0000 7f6f8d80a640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f8814f360 0x7f6f881ee770 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6f78001eb0 tx=0x7f6f7800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.985+0000 7f6f7effd640 1 -- 172.21.15.94:0/294823747 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78017070 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.985+0000 7f6f7effd640 1 -- 172.21.15.94:0/294823747 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f78005d00 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.985+0000 7f6f7effd640 1 -- 172.21.15.94:0/294823747 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78007310 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.985+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f880c0fc0 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.985+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f880c12a0 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.986+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8804f330 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.986+0000 7f6f7effd640 1 -- 172.21.15.94:0/294823747 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f7801a070 con 0x7f6f8814f360 2024-03-06T07:20:15.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.987+0000 7f6f7effd640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:15.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.987+0000 7f6f7effd640 1 -- 172.21.15.94:0/294823747 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f6f78089970 con 0x7f6f8814f360 2024-03-06T07:20:15.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.987+0000 7f6f8e00b640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:15.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.987+0000 7f6f8e00b640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6f881edf20 tx=0x7f6f84009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:15.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:15.991+0000 7f6f7effd640 1 -- 172.21.15.94:0/294823747 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f78057d50 con 0x7f6f8814f360 2024-03-06T07:20:16.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.165+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6f8806cda0 con 0x7f6f6c06b4d0 2024-03-06T07:20:16.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.177+0000 7f6f7effd640 1 -- 172.21.15.94:0/294823747 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490559 (secure 0 0 0) 0x7f6f8806cda0 con 0x7f6f6c06b4d0 2024-03-06T07:20:16.184 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:16.188 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 msgr2=0x7f6f6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6f881edf20 tx=0x7f6f84009190 comp rx=0 tx=0).stop 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f8814f360 msgr2=0x7f6f881ee770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f8814f360 0x7f6f881ee770 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6f78001eb0 tx=0x7f6f7800c820 comp rx=0 tx=0).stop 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 shutdown_connections 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f6c06b4d0 0x7f6f6c06d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f88158b60 0x7f6f8814b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f8814f360 0x7f6f881ee770 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 --2- 172.21.15.94:0/294823747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f880c88c0 0x7f6f881ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 >> 172.21.15.94:0/294823747 conn(0x7f6f880ac570 msgr2=0x7f6f88153000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 shutdown_connections 2024-03-06T07:20:16.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.182+0000 7f6f9026f640 1 -- 172.21.15.94:0/294823747 wait complete. 2024-03-06T07:20:16.194 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-03-06T07:20:14.660922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238288,"kb_used_data":5472,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736432,"statfs":{"total":773094113280,"available":772850106368,"internally_reserved":0,"allocated":5603328,"data_stored":3640312,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.392092"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290558+0000","last_change":"2024-03-06T07:20:08.289892+0000","last_active":"2024-03-06T07:20:10.290558+0000","last_peered":"2024-03-06T07:20:10.290558+0000","last_clean":"2024-03-06T07:20:10.290558+0000","last_became_active":"2024-03-06T07:20:08.289621+0000","last_became_peered":"2024-03-06T07:20:08.289621+0000","last_unstale":"2024-03-06T07:20:10.290558+0000","last_undegraded":"2024-03-06T07:20:10.290558+0000","last_fullsized":"2024-03-06T07:20:10.290558+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:51.962575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289797+0000","last_change":"2024-03-06T07:19:54.052304+0000","last_active":"2024-03-06T07:20:10.289797+0000","last_peered":"2024-03-06T07:20:10.289797+0000","last_clean":"2024-03-06T07:20:10.289797+0000","last_became_active":"2024-03-06T07:19:54.052048+0000","last_became_peered":"2024-03-06T07:19:54.052048+0000","last_unstale":"2024-03-06T07:20:10.289797+0000","last_undegraded":"2024-03-06T07:20:10.289797+0000","last_fullsized":"2024-03-06T07:20:10.289797+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:24.129824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281131+0000","last_change":"2024-03-06T07:20:08.288626+0000","last_active":"2024-03-06T07:20:09.281131+0000","last_peered":"2024-03-06T07:20:09.281131+0000","last_clean":"2024-03-06T07:20:09.281131+0000","last_became_active":"2024-03-06T07:20:08.288363+0000","last_became_peered":"2024-03-06T07:20:08.288363+0000","last_unstale":"2024-03-06T07:20:09.281131+0000","last_undegraded":"2024-03-06T07:20:09.281131+0000","last_fullsized":"2024-03-06T07:20:09.281131+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:16.734109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290756+0000","last_change":"2024-03-06T07:19:54.062911+0000","last_active":"2024-03-06T07:20:10.290756+0000","last_peered":"2024-03-06T07:20:10.290756+0000","last_clean":"2024-03-06T07:20:10.290756+0000","last_became_active":"2024-03-06T07:19:54.062566+0000","last_became_peered":"2024-03-06T07:19:54.062566+0000","last_unstale":"2024-03-06T07:20:10.290756+0000","last_undegraded":"2024-03-06T07:20:10.290756+0000","last_fullsized":"2024-03-06T07:20:10.290756+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289681+0000","last_change":"2024-03-06T07:20:08.298070+0000","last_active":"2024-03-06T07:20:10.289681+0000","last_peered":"2024-03-06T07:20:10.289681+0000","last_clean":"2024-03-06T07:20:10.289681+0000","last_became_active":"2024-03-06T07:20:08.297509+0000","last_became_peered":"2024-03-06T07:20:08.297509+0000","last_unstale":"2024-03-06T07:20:10.289681+0000","last_undegraded":"2024-03-06T07:20:10.289681+0000","last_fullsized":"2024-03-06T07:20:10.289681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:18.762564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279967+0000","last_change":"2024-03-06T07:19:54.051868+0000","last_active":"2024-03-06T07:20:09.279967+0000","last_peered":"2024-03-06T07:20:09.279967+0000","last_clean":"2024-03-06T07:20:09.279967+0000","last_became_active":"2024-03-06T07:19:54.051611+0000","last_became_peered":"2024-03-06T07:19:54.051611+0000","last_unstale":"2024-03-06T07:20:09.279967+0000","last_undegraded":"2024-03-06T07:20:09.279967+0000","last_fullsized":"2024-03-06T07:20:09.279967+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289614+0000","last_change":"2024-03-06T07:20:08.295096+0000","last_active":"2024-03-06T07:20:10.289614+0000","last_peered":"2024-03-06T07:20:10.289614+0000","last_clean":"2024-03-06T07:20:10.289614+0000","last_became_active":"2024-03-06T07:20:08.294783+0000","last_became_peered":"2024-03-06T07:20:08.294783+0000","last_unstale":"2024-03-06T07:20:10.289614+0000","last_undegraded":"2024-03-06T07:20:10.289614+0000","last_fullsized":"2024-03-06T07:20:10.289614+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.458664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290585+0000","last_change":"2024-03-06T07:19:54.055553+0000","last_active":"2024-03-06T07:20:10.290585+0000","last_peered":"2024-03-06T07:20:10.290585+0000","last_clean":"2024-03-06T07:20:10.290585+0000","last_became_active":"2024-03-06T07:19:54.055271+0000","last_became_peered":"2024-03-06T07:19:54.055271+0000","last_unstale":"2024-03-06T07:20:10.290585+0000","last_undegraded":"2024-03-06T07:20:10.290585+0000","last_fullsized":"2024-03-06T07:20:10.290585+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274653+0000","last_change":"2024-03-06T07:20:08.287946+0000","last_active":"2024-03-06T07:20:10.274653+0000","last_peered":"2024-03-06T07:20:10.274653+0000","last_clean":"2024-03-06T07:20:10.274653+0000","last_became_active":"2024-03-06T07:20:08.287660+0000","last_became_peered":"2024-03-06T07:20:08.287660+0000","last_unstale":"2024-03-06T07:20:10.274653+0000","last_undegraded":"2024-03-06T07:20:10.274653+0000","last_fullsized":"2024-03-06T07:20:10.274653+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:11.468274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280063+0000","last_change":"2024-03-06T07:19:54.046161+0000","last_active":"2024-03-06T07:20:09.280063+0000","last_peered":"2024-03-06T07:20:09.280063+0000","last_clean":"2024-03-06T07:20:09.280063+0000","last_became_active":"2024-03-06T07:19:54.045875+0000","last_became_peered":"2024-03-06T07:19:54.045875+0000","last_unstale":"2024-03-06T07:20:09.280063+0000","last_undegraded":"2024-03-06T07:20:09.280063+0000","last_fullsized":"2024-03-06T07:20:09.280063+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:50.526448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272026+0000","last_change":"2024-03-06T07:20:08.274700+0000","last_active":"2024-03-06T07:20:10.272026+0000","last_peered":"2024-03-06T07:20:10.272026+0000","last_clean":"2024-03-06T07:20:10.272026+0000","last_became_active":"2024-03-06T07:20:08.274419+0000","last_became_peered":"2024-03-06T07:20:08.274419+0000","last_unstale":"2024-03-06T07:20:10.272026+0000","last_undegraded":"2024-03-06T07:20:10.272026+0000","last_fullsized":"2024-03-06T07:20:10.272026+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:00.756603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289147+0000","last_change":"2024-03-06T07:19:54.047960+0000","last_active":"2024-03-06T07:20:10.289147+0000","last_peered":"2024-03-06T07:20:10.289147+0000","last_clean":"2024-03-06T07:20:10.289147+0000","last_became_active":"2024-03-06T07:19:54.047638+0000","last_became_peered":"2024-03-06T07:19:54.047638+0000","last_unstale":"2024-03-06T07:20:10.289147+0000","last_undegraded":"2024-03-06T07:20:10.289147+0000","last_fullsized":"2024-03-06T07:20:10.289147+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281001+0000","last_change":"2024-03-06T07:20:08.286960+0000","last_active":"2024-03-06T07:20:09.281001+0000","last_peered":"2024-03-06T07:20:09.281001+0000","last_clean":"2024-03-06T07:20:09.281001+0000","last_became_active":"2024-03-06T07:20:08.286724+0000","last_became_peered":"2024-03-06T07:20:08.286724+0000","last_unstale":"2024-03-06T07:20:09.281001+0000","last_undegraded":"2024-03-06T07:20:09.281001+0000","last_fullsized":"2024-03-06T07:20:09.281001+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:03.852954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279203+0000","last_change":"2024-03-06T07:19:54.048726+0000","last_active":"2024-03-06T07:20:09.279203+0000","last_peered":"2024-03-06T07:20:09.279203+0000","last_clean":"2024-03-06T07:20:09.279203+0000","last_became_active":"2024-03-06T07:19:54.048377+0000","last_became_peered":"2024-03-06T07:19:54.048377+0000","last_unstale":"2024-03-06T07:20:09.279203+0000","last_undegraded":"2024-03-06T07:20:09.279203+0000","last_fullsized":"2024-03-06T07:20:09.279203+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:57.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279232+0000","last_change":"2024-03-06T07:20:08.286854+0000","last_active":"2024-03-06T07:20:09.279232+0000","last_peered":"2024-03-06T07:20:09.279232+0000","last_clean":"2024-03-06T07:20:09.279232+0000","last_became_active":"2024-03-06T07:20:08.286383+0000","last_became_peered":"2024-03-06T07:20:08.286383+0000","last_unstale":"2024-03-06T07:20:09.279232+0000","last_undegraded":"2024-03-06T07:20:09.279232+0000","last_fullsized":"2024-03-06T07:20:09.279232+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:53.510411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287708+0000","last_change":"2024-03-06T07:19:54.059007+0000","last_active":"2024-03-06T07:20:10.287708+0000","last_peered":"2024-03-06T07:20:10.287708+0000","last_clean":"2024-03-06T07:20:10.287708+0000","last_became_active":"2024-03-06T07:19:54.058694+0000","last_became_peered":"2024-03-06T07:19:54.058694+0000","last_unstale":"2024-03-06T07:20:10.287708+0000","last_undegraded":"2024-03-06T07:20:10.287708+0000","last_fullsized":"2024-03-06T07:20:10.287708+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:50.508506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280082+0000","last_change":"2024-03-06T07:20:08.296591+0000","last_active":"2024-03-06T07:20:09.280082+0000","last_peered":"2024-03-06T07:20:09.280082+0000","last_clean":"2024-03-06T07:20:09.280082+0000","last_became_active":"2024-03-06T07:20:08.295965+0000","last_became_peered":"2024-03-06T07:20:08.295965+0000","last_unstale":"2024-03-06T07:20:09.280082+0000","last_undegraded":"2024-03-06T07:20:09.280082+0000","last_fullsized":"2024-03-06T07:20:09.280082+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:43.959679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287761+0000","last_change":"2024-03-06T07:19:54.057769+0000","last_active":"2024-03-06T07:20:10.287761+0000","last_peered":"2024-03-06T07:20:10.287761+0000","last_clean":"2024-03-06T07:20:10.287761+0000","last_became_active":"2024-03-06T07:19:54.057496+0000","last_became_peered":"2024-03-06T07:19:54.057496+0000","last_unstale":"2024-03-06T07:20:10.287761+0000","last_undegraded":"2024-03-06T07:20:10.287761+0000","last_fullsized":"2024-03-06T07:20:10.287761+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:51.719701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272401+0000","last_change":"2024-03-06T07:20:08.281429+0000","last_active":"2024-03-06T07:20:10.272401+0000","last_peered":"2024-03-06T07:20:10.272401+0000","last_clean":"2024-03-06T07:20:10.272401+0000","last_became_active":"2024-03-06T07:20:08.281116+0000","last_became_peered":"2024-03-06T07:20:08.281116+0000","last_unstale":"2024-03-06T07:20:10.272401+0000","last_undegraded":"2024-03-06T07:20:10.272401+0000","last_fullsized":"2024-03-06T07:20:10.272401+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:51.962575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289359+0000","last_change":"2024-03-06T07:19:54.050005+0000","last_active":"2024-03-06T07:20:10.289359+0000","last_peered":"2024-03-06T07:20:10.289359+0000","last_clean":"2024-03-06T07:20:10.289359+0000","last_became_active":"2024-03-06T07:19:54.049750+0000","last_became_peered":"2024-03-06T07:19:54.049750+0000","last_unstale":"2024-03-06T07:20:10.289359+0000","last_undegraded":"2024-03-06T07:20:10.289359+0000","last_fullsized":"2024-03-06T07:20:10.289359+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:37.609871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290810+0000","last_change":"2024-03-06T07:20:08.284734+0000","last_active":"2024-03-06T07:20:10.290810+0000","last_peered":"2024-03-06T07:20:10.290810+0000","last_clean":"2024-03-06T07:20:10.290810+0000","last_became_active":"2024-03-06T07:20:08.284289+0000","last_became_peered":"2024-03-06T07:20:08.284289+0000","last_unstale":"2024-03-06T07:20:10.290810+0000","last_undegraded":"2024-03-06T07:20:10.290810+0000","last_fullsized":"2024-03-06T07:20:10.290810+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:01.438975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.275106+0000","last_change":"2024-03-06T07:19:54.052367+0000","last_active":"2024-03-06T07:20:10.275106+0000","last_peered":"2024-03-06T07:20:10.275106+0000","last_clean":"2024-03-06T07:20:10.275106+0000","last_became_active":"2024-03-06T07:19:54.052119+0000","last_became_peered":"2024-03-06T07:19:54.052119+0000","last_unstale":"2024-03-06T07:20:10.275106+0000","last_undegraded":"2024-03-06T07:20:10.275106+0000","last_fullsized":"2024-03-06T07:20:10.275106+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.678839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290604+0000","last_change":"2024-03-06T07:20:08.288011+0000","last_active":"2024-03-06T07:20:10.290604+0000","last_peered":"2024-03-06T07:20:10.290604+0000","last_clean":"2024-03-06T07:20:10.290604+0000","last_became_active":"2024-03-06T07:20:08.286929+0000","last_became_peered":"2024-03-06T07:20:08.286929+0000","last_unstale":"2024-03-06T07:20:10.290604+0000","last_undegraded":"2024-03-06T07:20:10.290604+0000","last_fullsized":"2024-03-06T07:20:10.290604+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:14.797649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287884+0000","last_change":"2024-03-06T07:19:54.040294+0000","last_active":"2024-03-06T07:20:10.287884+0000","last_peered":"2024-03-06T07:20:10.287884+0000","last_clean":"2024-03-06T07:20:10.287884+0000","last_became_active":"2024-03-06T07:19:54.040048+0000","last_became_peered":"2024-03-06T07:19:54.040048+0000","last_unstale":"2024-03-06T07:20:10.287884+0000","last_undegraded":"2024-03-06T07:20:10.287884+0000","last_fullsized":"2024-03-06T07:20:10.287884+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:41.393636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289930+0000","last_change":"2024-03-06T07:20:08.280889+0000","last_active":"2024-03-06T07:20:10.289930+0000","last_peered":"2024-03-06T07:20:10.289930+0000","last_clean":"2024-03-06T07:20:10.289930+0000","last_became_active":"2024-03-06T07:20:08.280619+0000","last_became_peered":"2024-03-06T07:20:08.280619+0000","last_unstale":"2024-03-06T07:20:10.289930+0000","last_undegraded":"2024-03-06T07:20:10.289930+0000","last_fullsized":"2024-03-06T07:20:10.289930+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:53.510411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289088+0000","last_change":"2024-03-06T07:19:54.056367+0000","last_active":"2024-03-06T07:20:10.289088+0000","last_peered":"2024-03-06T07:20:10.289088+0000","last_clean":"2024-03-06T07:20:10.289088+0000","last_became_active":"2024-03-06T07:19:54.056125+0000","last_became_peered":"2024-03-06T07:19:54.056125+0000","last_unstale":"2024-03-06T07:20:10.289088+0000","last_undegraded":"2024-03-06T07:20:10.289088+0000","last_fullsized":"2024-03-06T07:20:10.289088+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274585+0000","last_change":"2024-03-06T07:20:08.291928+0000","last_active":"2024-03-06T07:20:10.274585+0000","last_peered":"2024-03-06T07:20:10.274585+0000","last_clean":"2024-03-06T07:20:10.274585+0000","last_became_active":"2024-03-06T07:20:08.291627+0000","last_became_peered":"2024-03-06T07:20:08.291627+0000","last_unstale":"2024-03-06T07:20:10.274585+0000","last_undegraded":"2024-03-06T07:20:10.274585+0000","last_fullsized":"2024-03-06T07:20:10.274585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:40.550642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279266+0000","last_change":"2024-03-06T07:19:54.051456+0000","last_active":"2024-03-06T07:20:09.279266+0000","last_peered":"2024-03-06T07:20:09.279266+0000","last_clean":"2024-03-06T07:20:09.279266+0000","last_became_active":"2024-03-06T07:19:54.051193+0000","last_became_peered":"2024-03-06T07:19:54.051193+0000","last_unstale":"2024-03-06T07:20:09.279266+0000","last_undegraded":"2024-03-06T07:20:09.279266+0000","last_fullsized":"2024-03-06T07:20:09.279266+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272338+0000","last_change":"2024-03-06T07:20:08.284233+0000","last_active":"2024-03-06T07:20:10.272338+0000","last_peered":"2024-03-06T07:20:10.272338+0000","last_clean":"2024-03-06T07:20:10.272338+0000","last_became_active":"2024-03-06T07:20:08.284044+0000","last_became_peered":"2024-03-06T07:20:08.284044+0000","last_unstale":"2024-03-06T07:20:10.272338+0000","last_undegraded":"2024-03-06T07:20:10.272338+0000","last_fullsized":"2024-03-06T07:20:10.272338+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.838274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289350+0000","last_change":"2024-03-06T07:19:54.049011+0000","last_active":"2024-03-06T07:20:10.289350+0000","last_peered":"2024-03-06T07:20:10.289350+0000","last_clean":"2024-03-06T07:20:10.289350+0000","last_became_active":"2024-03-06T07:19:54.048738+0000","last_became_peered":"2024-03-06T07:19:54.048738+0000","last_unstale":"2024-03-06T07:20:10.289350+0000","last_undegraded":"2024-03-06T07:20:10.289350+0000","last_fullsized":"2024-03-06T07:20:10.289350+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:38:36.367136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290293+0000","last_change":"2024-03-06T07:20:08.290724+0000","last_active":"2024-03-06T07:20:10.290293+0000","last_peered":"2024-03-06T07:20:10.290293+0000","last_clean":"2024-03-06T07:20:10.290293+0000","last_became_active":"2024-03-06T07:20:08.290411+0000","last_became_peered":"2024-03-06T07:20:08.290411+0000","last_unstale":"2024-03-06T07:20:10.290293+0000","last_undegraded":"2024-03-06T07:20:10.290293+0000","last_fullsized":"2024-03-06T07:20:10.290293+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.838274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288991+0000","last_change":"2024-03-06T07:19:54.049339+0000","last_active":"2024-03-06T07:20:10.288991+0000","last_peered":"2024-03-06T07:20:10.288991+0000","last_clean":"2024-03-06T07:20:10.288991+0000","last_became_active":"2024-03-06T07:19:54.049059+0000","last_became_peered":"2024-03-06T07:19:54.049059+0000","last_unstale":"2024-03-06T07:20:10.288991+0000","last_undegraded":"2024-03-06T07:20:10.288991+0000","last_fullsized":"2024-03-06T07:20:10.288991+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280625+0000","last_change":"2024-03-06T07:20:08.288942+0000","last_active":"2024-03-06T07:20:09.280625+0000","last_peered":"2024-03-06T07:20:09.280625+0000","last_clean":"2024-03-06T07:20:09.280625+0000","last_became_active":"2024-03-06T07:20:08.288543+0000","last_became_peered":"2024-03-06T07:20:08.288543+0000","last_unstale":"2024-03-06T07:20:09.280625+0000","last_undegraded":"2024-03-06T07:20:09.280625+0000","last_fullsized":"2024-03-06T07:20:09.280625+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:48.612666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280007+0000","last_change":"2024-03-06T07:19:54.058699+0000","last_active":"2024-03-06T07:20:09.280007+0000","last_peered":"2024-03-06T07:20:09.280007+0000","last_clean":"2024-03-06T07:20:09.280007+0000","last_became_active":"2024-03-06T07:19:54.056798+0000","last_became_peered":"2024-03-06T07:19:54.056798+0000","last_unstale":"2024-03-06T07:20:09.280007+0000","last_undegraded":"2024-03-06T07:20:09.280007+0000","last_fullsized":"2024-03-06T07:20:09.280007+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.725346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288570+0000","last_change":"2024-03-06T07:20:08.285547+0000","last_active":"2024-03-06T07:20:10.288570+0000","last_peered":"2024-03-06T07:20:10.288570+0000","last_clean":"2024-03-06T07:20:10.288570+0000","last_became_active":"2024-03-06T07:20:08.282256+0000","last_became_peered":"2024-03-06T07:20:08.282256+0000","last_unstale":"2024-03-06T07:20:10.288570+0000","last_undegraded":"2024-03-06T07:20:10.288570+0000","last_fullsized":"2024-03-06T07:20:10.288570+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:53.137345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290830+0000","last_change":"2024-03-06T07:19:54.062261+0000","last_active":"2024-03-06T07:20:10.290830+0000","last_peered":"2024-03-06T07:20:10.290830+0000","last_clean":"2024-03-06T07:20:10.290830+0000","last_became_active":"2024-03-06T07:19:54.061853+0000","last_became_peered":"2024-03-06T07:19:54.061853+0000","last_unstale":"2024-03-06T07:20:10.290830+0000","last_undegraded":"2024-03-06T07:20:10.290830+0000","last_fullsized":"2024-03-06T07:20:10.290830+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289718+0000","last_change":"2024-03-06T07:20:08.288551+0000","last_active":"2024-03-06T07:20:10.289718+0000","last_peered":"2024-03-06T07:20:10.289718+0000","last_clean":"2024-03-06T07:20:10.289718+0000","last_became_active":"2024-03-06T07:20:08.288232+0000","last_became_peered":"2024-03-06T07:20:08.288232+0000","last_unstale":"2024-03-06T07:20:10.289718+0000","last_undegraded":"2024-03-06T07:20:10.289718+0000","last_fullsized":"2024-03-06T07:20:10.289718+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:24.521500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280103+0000","last_change":"2024-03-06T07:19:54.054672+0000","last_active":"2024-03-06T07:20:09.280103+0000","last_peered":"2024-03-06T07:20:09.280103+0000","last_clean":"2024-03-06T07:20:09.280103+0000","last_became_active":"2024-03-06T07:19:54.054301+0000","last_became_peered":"2024-03-06T07:19:54.054301+0000","last_unstale":"2024-03-06T07:20:09.280103+0000","last_undegraded":"2024-03-06T07:20:09.280103+0000","last_fullsized":"2024-03-06T07:20:09.280103+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:05.772166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281220+0000","last_change":"2024-03-06T07:20:08.286513+0000","last_active":"2024-03-06T07:20:09.281220+0000","last_peered":"2024-03-06T07:20:09.281220+0000","last_clean":"2024-03-06T07:20:09.281220+0000","last_became_active":"2024-03-06T07:20:08.286189+0000","last_became_peered":"2024-03-06T07:20:08.286189+0000","last_unstale":"2024-03-06T07:20:09.281220+0000","last_undegraded":"2024-03-06T07:20:09.281220+0000","last_fullsized":"2024-03-06T07:20:09.281220+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:21.720175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279896+0000","last_change":"2024-03-06T07:19:54.051669+0000","last_active":"2024-03-06T07:20:09.279896+0000","last_peered":"2024-03-06T07:20:09.279896+0000","last_clean":"2024-03-06T07:20:09.279896+0000","last_became_active":"2024-03-06T07:19:54.051312+0000","last_became_peered":"2024-03-06T07:19:54.051312+0000","last_unstale":"2024-03-06T07:20:09.279896+0000","last_undegraded":"2024-03-06T07:20:09.279896+0000","last_fullsized":"2024-03-06T07:20:09.279896+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:39.507165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288123+0000","last_change":"2024-03-06T07:20:08.281934+0000","last_active":"2024-03-06T07:20:10.288123+0000","last_peered":"2024-03-06T07:20:10.288123+0000","last_clean":"2024-03-06T07:20:10.288123+0000","last_became_active":"2024-03-06T07:20:08.281674+0000","last_became_peered":"2024-03-06T07:20:08.281674+0000","last_unstale":"2024-03-06T07:20:10.288123+0000","last_undegraded":"2024-03-06T07:20:10.288123+0000","last_fullsized":"2024-03-06T07:20:10.288123+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:32.350695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281124+0000","last_change":"2024-03-06T07:19:54.047677+0000","last_active":"2024-03-06T07:20:09.281124+0000","last_peered":"2024-03-06T07:20:09.281124+0000","last_clean":"2024-03-06T07:20:09.281124+0000","last_became_active":"2024-03-06T07:19:54.045393+0000","last_became_peered":"2024-03-06T07:19:54.045393+0000","last_unstale":"2024-03-06T07:20:09.281124+0000","last_undegraded":"2024-03-06T07:20:09.281124+0000","last_fullsized":"2024-03-06T07:20:09.281124+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289563+0000","last_change":"2024-03-06T07:20:08.288100+0000","last_active":"2024-03-06T07:20:10.289563+0000","last_peered":"2024-03-06T07:20:10.289563+0000","last_clean":"2024-03-06T07:20:10.289563+0000","last_became_active":"2024-03-06T07:20:08.287387+0000","last_became_peered":"2024-03-06T07:20:08.287387+0000","last_unstale":"2024-03-06T07:20:10.289563+0000","last_undegraded":"2024-03-06T07:20:10.289563+0000","last_fullsized":"2024-03-06T07:20:10.289563+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:45.089464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288401+0000","last_change":"2024-03-06T07:19:54.058376+0000","last_active":"2024-03-06T07:20:10.288401+0000","last_peered":"2024-03-06T07:20:10.288401+0000","last_clean":"2024-03-06T07:20:10.288401+0000","last_became_active":"2024-03-06T07:19:54.058151+0000","last_became_peered":"2024-03-06T07:19:54.058151+0000","last_unstale":"2024-03-06T07:20:10.288401+0000","last_undegraded":"2024-03-06T07:20:10.288401+0000","last_fullsized":"2024-03-06T07:20:10.288401+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:10.287168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272359+0000","last_change":"2024-03-06T07:20:08.280882+0000","last_active":"2024-03-06T07:20:10.272359+0000","last_peered":"2024-03-06T07:20:10.272359+0000","last_clean":"2024-03-06T07:20:10.272359+0000","last_became_active":"2024-03-06T07:20:08.280648+0000","last_became_peered":"2024-03-06T07:20:08.280648+0000","last_unstale":"2024-03-06T07:20:10.272359+0000","last_undegraded":"2024-03-06T07:20:10.272359+0000","last_fullsized":"2024-03-06T07:20:10.272359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:30.212401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274623+0000","last_change":"2024-03-06T07:19:54.046473+0000","last_active":"2024-03-06T07:20:10.274623+0000","last_peered":"2024-03-06T07:20:10.274623+0000","last_clean":"2024-03-06T07:20:10.274623+0000","last_became_active":"2024-03-06T07:19:54.045977+0000","last_became_peered":"2024-03-06T07:19:54.045977+0000","last_unstale":"2024-03-06T07:20:10.274623+0000","last_undegraded":"2024-03-06T07:20:10.274623+0000","last_fullsized":"2024-03-06T07:20:10.274623+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:28.437704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288551+0000","last_change":"2024-03-06T07:20:08.279693+0000","last_active":"2024-03-06T07:20:10.288551+0000","last_peered":"2024-03-06T07:20:10.288551+0000","last_clean":"2024-03-06T07:20:10.288551+0000","last_became_active":"2024-03-06T07:20:08.279409+0000","last_became_peered":"2024-03-06T07:20:08.279409+0000","last_unstale":"2024-03-06T07:20:10.288551+0000","last_undegraded":"2024-03-06T07:20:10.288551+0000","last_fullsized":"2024-03-06T07:20:10.288551+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:20.720975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279102+0000","last_change":"2024-03-06T07:19:54.050649+0000","last_active":"2024-03-06T07:20:09.279102+0000","last_peered":"2024-03-06T07:20:09.279102+0000","last_clean":"2024-03-06T07:20:09.279102+0000","last_became_active":"2024-03-06T07:19:54.050392+0000","last_became_peered":"2024-03-06T07:19:54.050392+0000","last_unstale":"2024-03-06T07:20:09.279102+0000","last_undegraded":"2024-03-06T07:20:09.279102+0000","last_fullsized":"2024-03-06T07:20:09.279102+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279855+0000","last_change":"2024-03-06T07:20:08.291858+0000","last_active":"2024-03-06T07:20:09.279855+0000","last_peered":"2024-03-06T07:20:09.279855+0000","last_clean":"2024-03-06T07:20:09.279855+0000","last_became_active":"2024-03-06T07:20:08.291629+0000","last_became_peered":"2024-03-06T07:20:08.291629+0000","last_unstale":"2024-03-06T07:20:09.279855+0000","last_undegraded":"2024-03-06T07:20:09.279855+0000","last_fullsized":"2024-03-06T07:20:09.279855+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:16.214208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280048+0000","last_change":"2024-03-06T07:19:54.046865+0000","last_active":"2024-03-06T07:20:09.280048+0000","last_peered":"2024-03-06T07:20:09.280048+0000","last_clean":"2024-03-06T07:20:09.280048+0000","last_became_active":"2024-03-06T07:19:54.046545+0000","last_became_peered":"2024-03-06T07:19:54.046545+0000","last_unstale":"2024-03-06T07:20:09.280048+0000","last_undegraded":"2024-03-06T07:20:09.280048+0000","last_fullsized":"2024-03-06T07:20:09.280048+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:09.982751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288312+0000","last_change":"2024-03-06T07:20:08.278501+0000","last_active":"2024-03-06T07:20:10.288312+0000","last_peered":"2024-03-06T07:20:10.288312+0000","last_clean":"2024-03-06T07:20:10.288312+0000","last_became_active":"2024-03-06T07:20:08.278246+0000","last_became_peered":"2024-03-06T07:20:08.278246+0000","last_unstale":"2024-03-06T07:20:10.288312+0000","last_undegraded":"2024-03-06T07:20:10.288312+0000","last_fullsized":"2024-03-06T07:20:10.288312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:20.839748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288935+0000","last_change":"2024-03-06T07:19:54.042151+0000","last_active":"2024-03-06T07:20:10.288935+0000","last_peered":"2024-03-06T07:20:10.288935+0000","last_clean":"2024-03-06T07:20:10.288935+0000","last_became_active":"2024-03-06T07:19:54.041796+0000","last_became_peered":"2024-03-06T07:19:54.041796+0000","last_unstale":"2024-03-06T07:20:10.288935+0000","last_undegraded":"2024-03-06T07:20:10.288935+0000","last_fullsized":"2024-03-06T07:20:10.288935+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:27.407999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272612+0000","last_change":"2024-03-06T07:20:08.289732+0000","last_active":"2024-03-06T07:20:10.272612+0000","last_peered":"2024-03-06T07:20:10.272612+0000","last_clean":"2024-03-06T07:20:10.272612+0000","last_became_active":"2024-03-06T07:20:08.289476+0000","last_became_peered":"2024-03-06T07:20:08.289476+0000","last_unstale":"2024-03-06T07:20:10.272612+0000","last_undegraded":"2024-03-06T07:20:10.272612+0000","last_fullsized":"2024-03-06T07:20:10.272612+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:34.619396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288148+0000","last_change":"2024-03-06T07:19:54.051161+0000","last_active":"2024-03-06T07:20:10.288148+0000","last_peered":"2024-03-06T07:20:10.288148+0000","last_clean":"2024-03-06T07:20:10.288148+0000","last_became_active":"2024-03-06T07:19:54.050893+0000","last_became_peered":"2024-03-06T07:19:54.050893+0000","last_unstale":"2024-03-06T07:20:10.288148+0000","last_undegraded":"2024-03-06T07:20:10.288148+0000","last_fullsized":"2024-03-06T07:20:10.288148+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:28.711633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272490+0000","last_change":"2024-03-06T07:20:08.291381+0000","last_active":"2024-03-06T07:20:10.272490+0000","last_peered":"2024-03-06T07:20:10.272490+0000","last_clean":"2024-03-06T07:20:10.272490+0000","last_became_active":"2024-03-06T07:20:08.291115+0000","last_became_peered":"2024-03-06T07:20:08.291115+0000","last_unstale":"2024-03-06T07:20:10.272490+0000","last_undegraded":"2024-03-06T07:20:10.272490+0000","last_fullsized":"2024-03-06T07:20:10.272490+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:14.797649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279973+0000","last_change":"2024-03-06T07:19:54.040041+0000","last_active":"2024-03-06T07:20:09.279973+0000","last_peered":"2024-03-06T07:20:09.279973+0000","last_clean":"2024-03-06T07:20:09.279973+0000","last_became_active":"2024-03-06T07:19:54.039730+0000","last_became_peered":"2024-03-06T07:19:54.039730+0000","last_unstale":"2024-03-06T07:20:09.279973+0000","last_undegraded":"2024-03-06T07:20:09.279973+0000","last_fullsized":"2024-03-06T07:20:09.279973+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.629067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280738+0000","last_change":"2024-03-06T07:20:08.289711+0000","last_active":"2024-03-06T07:20:09.280738+0000","last_peered":"2024-03-06T07:20:09.280738+0000","last_clean":"2024-03-06T07:20:09.280738+0000","last_became_active":"2024-03-06T07:20:08.289485+0000","last_became_peered":"2024-03-06T07:20:08.289485+0000","last_unstale":"2024-03-06T07:20:09.280738+0000","last_undegraded":"2024-03-06T07:20:09.280738+0000","last_fullsized":"2024-03-06T07:20:09.280738+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:11.664663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279226+0000","last_change":"2024-03-06T07:19:54.055165+0000","last_active":"2024-03-06T07:20:09.279226+0000","last_peered":"2024-03-06T07:20:09.279226+0000","last_clean":"2024-03-06T07:20:09.279226+0000","last_became_active":"2024-03-06T07:19:54.054908+0000","last_became_peered":"2024-03-06T07:19:54.054908+0000","last_unstale":"2024-03-06T07:20:09.279226+0000","last_undegraded":"2024-03-06T07:20:09.279226+0000","last_fullsized":"2024-03-06T07:20:09.279226+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:02.499776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.278929+0000","last_change":"2024-03-06T07:20:08.291670+0000","last_active":"2024-03-06T07:20:09.278929+0000","last_peered":"2024-03-06T07:20:09.278929+0000","last_clean":"2024-03-06T07:20:09.278929+0000","last_became_active":"2024-03-06T07:20:08.290863+0000","last_became_peered":"2024-03-06T07:20:08.290863+0000","last_unstale":"2024-03-06T07:20:09.278929+0000","last_undegraded":"2024-03-06T07:20:09.278929+0000","last_fullsized":"2024-03-06T07:20:09.278929+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:00.756603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290339+0000","last_change":"2024-03-06T07:19:54.049805+0000","last_active":"2024-03-06T07:20:10.290339+0000","last_peered":"2024-03-06T07:20:10.290339+0000","last_clean":"2024-03-06T07:20:10.290339+0000","last_became_active":"2024-03-06T07:19:54.049562+0000","last_became_peered":"2024-03-06T07:19:54.049562+0000","last_unstale":"2024-03-06T07:20:10.290339+0000","last_undegraded":"2024-03-06T07:20:10.290339+0000","last_fullsized":"2024-03-06T07:20:10.290339+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:21.645621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290274+0000","last_change":"2024-03-06T07:20:08.283946+0000","last_active":"2024-03-06T07:20:10.290274+0000","last_peered":"2024-03-06T07:20:10.290274+0000","last_clean":"2024-03-06T07:20:10.290274+0000","last_became_active":"2024-03-06T07:20:08.283701+0000","last_became_peered":"2024-03-06T07:20:08.283701+0000","last_unstale":"2024-03-06T07:20:10.290274+0000","last_undegraded":"2024-03-06T07:20:10.290274+0000","last_fullsized":"2024-03-06T07:20:10.290274+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:09.713122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279055+0000","last_change":"2024-03-06T07:19:54.052763+0000","last_active":"2024-03-06T07:20:09.279055+0000","last_peered":"2024-03-06T07:20:09.279055+0000","last_clean":"2024-03-06T07:20:09.279055+0000","last_became_active":"2024-03-06T07:19:54.052472+0000","last_became_peered":"2024-03-06T07:19:54.052472+0000","last_unstale":"2024-03-06T07:20:09.279055+0000","last_undegraded":"2024-03-06T07:20:09.279055+0000","last_fullsized":"2024-03-06T07:20:09.279055+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:07.485843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272194+0000","last_change":"2024-03-06T07:20:08.274735+0000","last_active":"2024-03-06T07:20:10.272194+0000","last_peered":"2024-03-06T07:20:10.272194+0000","last_clean":"2024-03-06T07:20:10.272194+0000","last_became_active":"2024-03-06T07:20:08.274463+0000","last_became_peered":"2024-03-06T07:20:08.274463+0000","last_unstale":"2024-03-06T07:20:10.272194+0000","last_undegraded":"2024-03-06T07:20:10.272194+0000","last_fullsized":"2024-03-06T07:20:10.272194+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:09.713122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272121+0000","last_change":"2024-03-06T07:19:54.059383+0000","last_active":"2024-03-06T07:20:10.272121+0000","last_peered":"2024-03-06T07:20:10.272121+0000","last_clean":"2024-03-06T07:20:10.272121+0000","last_became_active":"2024-03-06T07:19:54.059123+0000","last_became_peered":"2024-03-06T07:19:54.059123+0000","last_unstale":"2024-03-06T07:20:10.272121+0000","last_undegraded":"2024-03-06T07:20:10.272121+0000","last_fullsized":"2024-03-06T07:20:10.272121+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280311+0000","last_change":"2024-03-06T07:20:08.289078+0000","last_active":"2024-03-06T07:20:09.280311+0000","last_peered":"2024-03-06T07:20:09.280311+0000","last_clean":"2024-03-06T07:20:09.280311+0000","last_became_active":"2024-03-06T07:20:08.288498+0000","last_became_peered":"2024-03-06T07:20:08.288498+0000","last_unstale":"2024-03-06T07:20:09.280311+0000","last_undegraded":"2024-03-06T07:20:09.280311+0000","last_fullsized":"2024-03-06T07:20:09.280311+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:11.664663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289729+0000","last_change":"2024-03-06T07:20:07.288076+0000","last_active":"2024-03-06T07:20:10.289729+0000","last_peered":"2024-03-06T07:20:10.289729+0000","last_clean":"2024-03-06T07:20:10.289729+0000","last_became_active":"2024-03-06T07:20:07.286803+0000","last_became_peered":"2024-03-06T07:20:07.286803+0000","last_unstale":"2024-03-06T07:20:10.289729+0000","last_undegraded":"2024-03-06T07:20:10.289729+0000","last_fullsized":"2024-03-06T07:20:10.289729+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:00.451319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289744+0000","last_change":"2024-03-06T07:19:54.060233+0000","last_active":"2024-03-06T07:20:10.289744+0000","last_peered":"2024-03-06T07:20:10.289744+0000","last_clean":"2024-03-06T07:20:10.289744+0000","last_became_active":"2024-03-06T07:19:54.059901+0000","last_became_peered":"2024-03-06T07:19:54.059901+0000","last_unstale":"2024-03-06T07:20:10.289744+0000","last_undegraded":"2024-03-06T07:20:10.289744+0000","last_fullsized":"2024-03-06T07:20:10.289744+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:09.921962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":26,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281406+0000","last_change":"2024-03-06T07:19:53.042382+0000","last_active":"2024-03-06T07:20:09.281406+0000","last_peered":"2024-03-06T07:20:09.281406+0000","last_clean":"2024-03-06T07:20:09.281406+0000","last_became_active":"2024-03-06T07:19:53.041334+0000","last_became_peered":"2024-03-06T07:19:53.041334+0000","last_unstale":"2024-03-06T07:20:09.281406+0000","last_undegraded":"2024-03-06T07:20:09.281406+0000","last_fullsized":"2024-03-06T07:20:09.281406+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289462+0000","last_change":"2024-03-06T07:20:08.285257+0000","last_active":"2024-03-06T07:20:10.289462+0000","last_peered":"2024-03-06T07:20:10.289462+0000","last_clean":"2024-03-06T07:20:10.289462+0000","last_became_active":"2024-03-06T07:20:08.285019+0000","last_became_peered":"2024-03-06T07:20:08.285019+0000","last_unstale":"2024-03-06T07:20:10.289462+0000","last_undegraded":"2024-03-06T07:20:10.289462+0000","last_fullsized":"2024-03-06T07:20:10.289462+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.323537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279402+0000","last_change":"2024-03-06T07:20:07.285089+0000","last_active":"2024-03-06T07:20:09.279402+0000","last_peered":"2024-03-06T07:20:09.279402+0000","last_clean":"2024-03-06T07:20:09.279402+0000","last_became_active":"2024-03-06T07:20:07.284839+0000","last_became_peered":"2024-03-06T07:20:07.284839+0000","last_unstale":"2024-03-06T07:20:09.279402+0000","last_undegraded":"2024-03-06T07:20:09.279402+0000","last_fullsized":"2024-03-06T07:20:09.279402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:50.836858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272278+0000","last_change":"2024-03-06T07:19:54.057284+0000","last_active":"2024-03-06T07:20:10.272278+0000","last_peered":"2024-03-06T07:20:10.272278+0000","last_clean":"2024-03-06T07:20:10.272278+0000","last_became_active":"2024-03-06T07:19:54.056999+0000","last_became_peered":"2024-03-06T07:19:54.056999+0000","last_unstale":"2024-03-06T07:20:10.272278+0000","last_undegraded":"2024-03-06T07:20:10.272278+0000","last_fullsized":"2024-03-06T07:20:10.272278+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289709+0000","last_change":"2024-03-06T07:19:53.048675+0000","last_active":"2024-03-06T07:20:10.289709+0000","last_peered":"2024-03-06T07:20:10.289709+0000","last_clean":"2024-03-06T07:20:10.289709+0000","last_became_active":"2024-03-06T07:19:53.048416+0000","last_became_peered":"2024-03-06T07:19:53.048416+0000","last_unstale":"2024-03-06T07:20:10.289709+0000","last_undegraded":"2024-03-06T07:20:10.289709+0000","last_fullsized":"2024-03-06T07:20:10.289709+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279419+0000","last_change":"2024-03-06T07:20:08.287669+0000","last_active":"2024-03-06T07:20:09.279419+0000","last_peered":"2024-03-06T07:20:09.279419+0000","last_clean":"2024-03-06T07:20:09.279419+0000","last_became_active":"2024-03-06T07:20:08.286498+0000","last_became_peered":"2024-03-06T07:20:08.286498+0000","last_unstale":"2024-03-06T07:20:09.279419+0000","last_undegraded":"2024-03-06T07:20:09.279419+0000","last_fullsized":"2024-03-06T07:20:09.279419+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:08.648820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290235+0000","last_change":"2024-03-06T07:20:07.285244+0000","last_active":"2024-03-06T07:20:10.290235+0000","last_peered":"2024-03-06T07:20:10.290235+0000","last_clean":"2024-03-06T07:20:10.290235+0000","last_became_active":"2024-03-06T07:20:07.285012+0000","last_became_peered":"2024-03-06T07:20:07.285012+0000","last_unstale":"2024-03-06T07:20:10.290235+0000","last_undegraded":"2024-03-06T07:20:10.290235+0000","last_fullsized":"2024-03-06T07:20:10.290235+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:17.755219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280744+0000","last_change":"2024-03-06T07:19:54.052199+0000","last_active":"2024-03-06T07:20:09.280744+0000","last_peered":"2024-03-06T07:20:09.280744+0000","last_clean":"2024-03-06T07:20:09.280744+0000","last_became_active":"2024-03-06T07:19:54.051923+0000","last_became_peered":"2024-03-06T07:19:54.051923+0000","last_unstale":"2024-03-06T07:20:09.280744+0000","last_undegraded":"2024-03-06T07:20:09.280744+0000","last_fullsized":"2024-03-06T07:20:09.280744+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288720+0000","last_change":"2024-03-06T07:19:53.044281+0000","last_active":"2024-03-06T07:20:10.288720+0000","last_peered":"2024-03-06T07:20:10.288720+0000","last_clean":"2024-03-06T07:20:10.288720+0000","last_became_active":"2024-03-06T07:19:53.043841+0000","last_became_peered":"2024-03-06T07:19:53.043841+0000","last_unstale":"2024-03-06T07:20:10.288720+0000","last_undegraded":"2024-03-06T07:20:10.288720+0000","last_fullsized":"2024-03-06T07:20:10.288720+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:53.925918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272172+0000","last_change":"2024-03-06T07:20:08.287358+0000","last_active":"2024-03-06T07:20:10.272172+0000","last_peered":"2024-03-06T07:20:10.272172+0000","last_clean":"2024-03-06T07:20:10.272172+0000","last_became_active":"2024-03-06T07:20:08.287072+0000","last_became_peered":"2024-03-06T07:20:08.287072+0000","last_unstale":"2024-03-06T07:20:10.272172+0000","last_undegraded":"2024-03-06T07:20:10.272172+0000","last_fullsized":"2024-03-06T07:20:10.272172+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:45.089464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272133+0000","last_change":"2024-03-06T07:20:07.285494+0000","last_active":"2024-03-06T07:20:10.272133+0000","last_peered":"2024-03-06T07:20:10.272133+0000","last_clean":"2024-03-06T07:20:10.272133+0000","last_became_active":"2024-03-06T07:20:07.285238+0000","last_became_peered":"2024-03-06T07:20:07.285238+0000","last_unstale":"2024-03-06T07:20:10.272133+0000","last_undegraded":"2024-03-06T07:20:10.272133+0000","last_fullsized":"2024-03-06T07:20:10.272133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:06.316192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288331+0000","last_change":"2024-03-06T07:19:54.039974+0000","last_active":"2024-03-06T07:20:10.288331+0000","last_peered":"2024-03-06T07:20:10.288331+0000","last_clean":"2024-03-06T07:20:10.288331+0000","last_became_active":"2024-03-06T07:19:54.039675+0000","last_became_peered":"2024-03-06T07:19:54.039675+0000","last_unstale":"2024-03-06T07:20:10.288331+0000","last_undegraded":"2024-03-06T07:20:10.288331+0000","last_fullsized":"2024-03-06T07:20:10.288331+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:53.089205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272507+0000","last_change":"2024-03-06T07:19:53.051640+0000","last_active":"2024-03-06T07:20:10.272507+0000","last_peered":"2024-03-06T07:20:10.272507+0000","last_clean":"2024-03-06T07:20:10.272507+0000","last_became_active":"2024-03-06T07:19:53.051429+0000","last_became_peered":"2024-03-06T07:19:53.051429+0000","last_unstale":"2024-03-06T07:20:10.272507+0000","last_undegraded":"2024-03-06T07:20:10.272507+0000","last_fullsized":"2024-03-06T07:20:10.272507+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:09.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281249+0000","last_change":"2024-03-06T07:20:08.281703+0000","last_active":"2024-03-06T07:20:09.281249+0000","last_peered":"2024-03-06T07:20:09.281249+0000","last_clean":"2024-03-06T07:20:09.281249+0000","last_became_active":"2024-03-06T07:20:08.281437+0000","last_became_peered":"2024-03-06T07:20:08.281437+0000","last_unstale":"2024-03-06T07:20:09.281249+0000","last_undegraded":"2024-03-06T07:20:09.281249+0000","last_fullsized":"2024-03-06T07:20:09.281249+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:07.323537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290789+0000","last_change":"2024-03-06T07:20:07.287035+0000","last_active":"2024-03-06T07:20:10.290789+0000","last_peered":"2024-03-06T07:20:10.290789+0000","last_clean":"2024-03-06T07:20:10.290789+0000","last_became_active":"2024-03-06T07:20:07.286453+0000","last_became_peered":"2024-03-06T07:20:07.286453+0000","last_unstale":"2024-03-06T07:20:10.290789+0000","last_undegraded":"2024-03-06T07:20:10.290789+0000","last_fullsized":"2024-03-06T07:20:10.290789+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:40.634986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281098+0000","last_change":"2024-03-06T07:19:54.054949+0000","last_active":"2024-03-06T07:20:09.281098+0000","last_peered":"2024-03-06T07:20:09.281098+0000","last_clean":"2024-03-06T07:20:09.281098+0000","last_became_active":"2024-03-06T07:19:54.054661+0000","last_became_peered":"2024-03-06T07:19:54.054661+0000","last_unstale":"2024-03-06T07:20:09.281098+0000","last_undegraded":"2024-03-06T07:20:09.281098+0000","last_fullsized":"2024-03-06T07:20:09.281098+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:57.129016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279423+0000","last_change":"2024-03-06T07:19:53.042003+0000","last_active":"2024-03-06T07:20:09.279423+0000","last_peered":"2024-03-06T07:20:09.279423+0000","last_clean":"2024-03-06T07:20:09.279423+0000","last_became_active":"2024-03-06T07:19:53.041728+0000","last_became_peered":"2024-03-06T07:19:53.041728+0000","last_unstale":"2024-03-06T07:20:09.279423+0000","last_undegraded":"2024-03-06T07:20:09.279423+0000","last_fullsized":"2024-03-06T07:20:09.279423+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:28.964097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280369+0000","last_change":"2024-03-06T07:20:08.289513+0000","last_active":"2024-03-06T07:20:09.280369+0000","last_peered":"2024-03-06T07:20:09.280369+0000","last_clean":"2024-03-06T07:20:09.280369+0000","last_became_active":"2024-03-06T07:20:08.288775+0000","last_became_peered":"2024-03-06T07:20:08.288775+0000","last_unstale":"2024-03-06T07:20:09.280369+0000","last_undegraded":"2024-03-06T07:20:09.280369+0000","last_fullsized":"2024-03-06T07:20:09.280369+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:49.507162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288560+0000","last_change":"2024-03-06T07:20:07.278974+0000","last_active":"2024-03-06T07:20:10.288560+0000","last_peered":"2024-03-06T07:20:10.288560+0000","last_clean":"2024-03-06T07:20:10.288560+0000","last_became_active":"2024-03-06T07:20:07.278682+0000","last_became_peered":"2024-03-06T07:20:07.278682+0000","last_unstale":"2024-03-06T07:20:10.288560+0000","last_undegraded":"2024-03-06T07:20:10.288560+0000","last_fullsized":"2024-03-06T07:20:10.288560+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:58.699699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290654+0000","last_change":"2024-03-06T07:19:54.061284+0000","last_active":"2024-03-06T07:20:10.290654+0000","last_peered":"2024-03-06T07:20:10.290654+0000","last_clean":"2024-03-06T07:20:10.290654+0000","last_became_active":"2024-03-06T07:19:54.061017+0000","last_became_peered":"2024-03-06T07:19:54.061017+0000","last_unstale":"2024-03-06T07:20:10.290654+0000","last_undegraded":"2024-03-06T07:20:10.290654+0000","last_fullsized":"2024-03-06T07:20:10.290654+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279587+0000","last_change":"2024-03-06T07:19:53.041819+0000","last_active":"2024-03-06T07:20:09.279587+0000","last_peered":"2024-03-06T07:20:09.279587+0000","last_clean":"2024-03-06T07:20:09.279587+0000","last_became_active":"2024-03-06T07:19:53.041542+0000","last_became_peered":"2024-03-06T07:19:53.041542+0000","last_unstale":"2024-03-06T07:20:09.279587+0000","last_undegraded":"2024-03-06T07:20:09.279587+0000","last_fullsized":"2024-03-06T07:20:09.279587+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288468+0000","last_change":"2024-03-06T07:20:08.291722+0000","last_active":"2024-03-06T07:20:10.288468+0000","last_peered":"2024-03-06T07:20:10.288468+0000","last_clean":"2024-03-06T07:20:10.288468+0000","last_became_active":"2024-03-06T07:20:08.291460+0000","last_became_peered":"2024-03-06T07:20:08.291460+0000","last_unstale":"2024-03-06T07:20:10.288468+0000","last_undegraded":"2024-03-06T07:20:10.288468+0000","last_fullsized":"2024-03-06T07:20:10.288468+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:03.852954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290654+0000","last_change":"2024-03-06T07:20:07.289776+0000","last_active":"2024-03-06T07:20:10.290654+0000","last_peered":"2024-03-06T07:20:10.290654+0000","last_clean":"2024-03-06T07:20:10.290654+0000","last_became_active":"2024-03-06T07:20:07.289535+0000","last_became_peered":"2024-03-06T07:20:07.289535+0000","last_unstale":"2024-03-06T07:20:10.290654+0000","last_undegraded":"2024-03-06T07:20:10.290654+0000","last_fullsized":"2024-03-06T07:20:10.290654+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:54.620623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289125+0000","last_change":"2024-03-06T07:19:54.039491+0000","last_active":"2024-03-06T07:20:10.289125+0000","last_peered":"2024-03-06T07:20:10.289125+0000","last_clean":"2024-03-06T07:20:10.289125+0000","last_became_active":"2024-03-06T07:19:54.039202+0000","last_became_peered":"2024-03-06T07:19:54.039202+0000","last_unstale":"2024-03-06T07:20:10.289125+0000","last_undegraded":"2024-03-06T07:20:10.289125+0000","last_fullsized":"2024-03-06T07:20:10.289125+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:58.292241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280219+0000","last_change":"2024-03-06T07:19:53.043210+0000","last_active":"2024-03-06T07:20:09.280219+0000","last_peered":"2024-03-06T07:20:09.280219+0000","last_clean":"2024-03-06T07:20:09.280219+0000","last_became_active":"2024-03-06T07:19:53.042951+0000","last_became_peered":"2024-03-06T07:19:53.042951+0000","last_unstale":"2024-03-06T07:20:09.280219+0000","last_undegraded":"2024-03-06T07:20:09.280219+0000","last_fullsized":"2024-03-06T07:20:09.280219+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288386+0000","last_change":"2024-03-06T07:20:08.276876+0000","last_active":"2024-03-06T07:20:10.288386+0000","last_peered":"2024-03-06T07:20:10.288386+0000","last_clean":"2024-03-06T07:20:10.288386+0000","last_became_active":"2024-03-06T07:20:08.276574+0000","last_became_peered":"2024-03-06T07:20:08.276574+0000","last_unstale":"2024-03-06T07:20:10.288386+0000","last_undegraded":"2024-03-06T07:20:10.288386+0000","last_fullsized":"2024-03-06T07:20:10.288386+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:18.752703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281073+0000","last_change":"2024-03-06T07:20:07.278534+0000","last_active":"2024-03-06T07:20:09.281073+0000","last_peered":"2024-03-06T07:20:09.281073+0000","last_clean":"2024-03-06T07:20:09.281073+0000","last_became_active":"2024-03-06T07:20:07.278261+0000","last_became_peered":"2024-03-06T07:20:07.278261+0000","last_unstale":"2024-03-06T07:20:09.281073+0000","last_undegraded":"2024-03-06T07:20:09.281073+0000","last_fullsized":"2024-03-06T07:20:09.281073+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:37.356811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289390+0000","last_change":"2024-03-06T07:19:54.049884+0000","last_active":"2024-03-06T07:20:10.289390+0000","last_peered":"2024-03-06T07:20:10.289390+0000","last_clean":"2024-03-06T07:20:10.289390+0000","last_became_active":"2024-03-06T07:19:54.049597+0000","last_became_peered":"2024-03-06T07:19:54.049597+0000","last_unstale":"2024-03-06T07:20:10.289390+0000","last_undegraded":"2024-03-06T07:20:10.289390+0000","last_fullsized":"2024-03-06T07:20:10.289390+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:13.785795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279196+0000","last_change":"2024-03-06T07:19:53.043198+0000","last_active":"2024-03-06T07:20:09.279196+0000","last_peered":"2024-03-06T07:20:09.279196+0000","last_clean":"2024-03-06T07:20:09.279196+0000","last_became_active":"2024-03-06T07:19:53.042939+0000","last_became_peered":"2024-03-06T07:19:53.042939+0000","last_unstale":"2024-03-06T07:20:09.279196+0000","last_undegraded":"2024-03-06T07:20:09.279196+0000","last_fullsized":"2024-03-06T07:20:09.279196+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280484+0000","last_change":"2024-03-06T07:20:08.288199+0000","last_active":"2024-03-06T07:20:09.280484+0000","last_peered":"2024-03-06T07:20:09.280484+0000","last_clean":"2024-03-06T07:20:09.280484+0000","last_became_active":"2024-03-06T07:20:08.287695+0000","last_became_peered":"2024-03-06T07:20:08.287695+0000","last_unstale":"2024-03-06T07:20:09.280484+0000","last_undegraded":"2024-03-06T07:20:09.280484+0000","last_fullsized":"2024-03-06T07:20:09.280484+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:21.720175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288165+0000","last_change":"2024-03-06T07:20:07.283576+0000","last_active":"2024-03-06T07:20:10.288165+0000","last_peered":"2024-03-06T07:20:10.288165+0000","last_clean":"2024-03-06T07:20:10.288165+0000","last_became_active":"2024-03-06T07:20:07.283136+0000","last_became_peered":"2024-03-06T07:20:07.283136+0000","last_unstale":"2024-03-06T07:20:10.288165+0000","last_undegraded":"2024-03-06T07:20:10.288165+0000","last_fullsized":"2024-03-06T07:20:10.288165+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:33.612051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279526+0000","last_change":"2024-03-06T07:19:54.045809+0000","last_active":"2024-03-06T07:20:09.279526+0000","last_peered":"2024-03-06T07:20:09.279526+0000","last_clean":"2024-03-06T07:20:09.279526+0000","last_became_active":"2024-03-06T07:19:54.040263+0000","last_became_peered":"2024-03-06T07:19:54.040263+0000","last_unstale":"2024-03-06T07:20:09.279526+0000","last_undegraded":"2024-03-06T07:20:09.279526+0000","last_fullsized":"2024-03-06T07:20:09.279526+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:20.728340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290867+0000","last_change":"2024-03-06T07:19:53.047864+0000","last_active":"2024-03-06T07:20:10.290867+0000","last_peered":"2024-03-06T07:20:10.290867+0000","last_clean":"2024-03-06T07:20:10.290867+0000","last_became_active":"2024-03-06T07:19:53.047607+0000","last_became_peered":"2024-03-06T07:19:53.047607+0000","last_unstale":"2024-03-06T07:20:10.290867+0000","last_undegraded":"2024-03-06T07:20:10.290867+0000","last_fullsized":"2024-03-06T07:20:10.290867+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:34.148872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288473+0000","last_change":"2024-03-06T07:20:08.280295+0000","last_active":"2024-03-06T07:20:10.288473+0000","last_peered":"2024-03-06T07:20:10.288473+0000","last_clean":"2024-03-06T07:20:10.288473+0000","last_became_active":"2024-03-06T07:20:08.280012+0000","last_became_peered":"2024-03-06T07:20:08.280012+0000","last_unstale":"2024-03-06T07:20:10.288473+0000","last_undegraded":"2024-03-06T07:20:10.288473+0000","last_fullsized":"2024-03-06T07:20:10.288473+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:21.055750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279812+0000","last_change":"2024-03-06T07:20:07.285280+0000","last_active":"2024-03-06T07:20:09.279812+0000","last_peered":"2024-03-06T07:20:09.279812+0000","last_clean":"2024-03-06T07:20:09.279812+0000","last_became_active":"2024-03-06T07:20:07.284993+0000","last_became_peered":"2024-03-06T07:20:07.284993+0000","last_unstale":"2024-03-06T07:20:09.279812+0000","last_undegraded":"2024-03-06T07:20:09.279812+0000","last_fullsized":"2024-03-06T07:20:09.279812+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:23.148949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.271838+0000","last_change":"2024-03-06T07:19:54.056898+0000","last_active":"2024-03-06T07:20:10.271838+0000","last_peered":"2024-03-06T07:20:10.271838+0000","last_clean":"2024-03-06T07:20:10.271838+0000","last_became_active":"2024-03-06T07:19:54.056522+0000","last_became_peered":"2024-03-06T07:19:54.056522+0000","last_unstale":"2024-03-06T07:20:10.271838+0000","last_undegraded":"2024-03-06T07:20:10.271838+0000","last_fullsized":"2024-03-06T07:20:10.271838+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:21.747193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281374+0000","last_change":"2024-03-06T07:19:53.043065+0000","last_active":"2024-03-06T07:20:09.281374+0000","last_peered":"2024-03-06T07:20:09.281374+0000","last_clean":"2024-03-06T07:20:09.281374+0000","last_became_active":"2024-03-06T07:19:53.042756+0000","last_became_peered":"2024-03-06T07:19:53.042756+0000","last_unstale":"2024-03-06T07:20:09.281374+0000","last_undegraded":"2024-03-06T07:20:09.281374+0000","last_fullsized":"2024-03-06T07:20:09.281374+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.338932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274550+0000","last_change":"2024-03-06T07:20:08.283743+0000","last_active":"2024-03-06T07:20:10.274550+0000","last_peered":"2024-03-06T07:20:10.274550+0000","last_clean":"2024-03-06T07:20:10.274550+0000","last_became_active":"2024-03-06T07:20:08.283434+0000","last_became_peered":"2024-03-06T07:20:08.283434+0000","last_unstale":"2024-03-06T07:20:10.274550+0000","last_undegraded":"2024-03-06T07:20:10.274550+0000","last_fullsized":"2024-03-06T07:20:10.274550+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:42.568567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288241+0000","last_change":"2024-03-06T07:20:07.282855+0000","last_active":"2024-03-06T07:20:10.288241+0000","last_peered":"2024-03-06T07:20:10.288241+0000","last_clean":"2024-03-06T07:20:10.288241+0000","last_became_active":"2024-03-06T07:20:07.282585+0000","last_became_peered":"2024-03-06T07:20:07.282585+0000","last_unstale":"2024-03-06T07:20:10.288241+0000","last_undegraded":"2024-03-06T07:20:10.288241+0000","last_fullsized":"2024-03-06T07:20:10.288241+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:13.790304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279317+0000","last_change":"2024-03-06T07:19:54.057647+0000","last_active":"2024-03-06T07:20:09.279317+0000","last_peered":"2024-03-06T07:20:09.279317+0000","last_clean":"2024-03-06T07:20:09.279317+0000","last_became_active":"2024-03-06T07:19:54.057396+0000","last_became_peered":"2024-03-06T07:19:54.057396+0000","last_unstale":"2024-03-06T07:20:09.279317+0000","last_undegraded":"2024-03-06T07:20:09.279317+0000","last_fullsized":"2024-03-06T07:20:09.279317+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":25,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290748+0000","last_change":"2024-03-06T07:19:53.046359+0000","last_active":"2024-03-06T07:20:10.290748+0000","last_peered":"2024-03-06T07:20:10.290748+0000","last_clean":"2024-03-06T07:20:10.290748+0000","last_became_active":"2024-03-06T07:19:53.046065+0000","last_became_peered":"2024-03-06T07:19:53.046065+0000","last_unstale":"2024-03-06T07:20:10.290748+0000","last_undegraded":"2024-03-06T07:20:10.290748+0000","last_fullsized":"2024-03-06T07:20:10.290748+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:08.146267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289992+0000","last_change":"2024-03-06T07:20:08.284659+0000","last_active":"2024-03-06T07:20:10.289992+0000","last_peered":"2024-03-06T07:20:10.289992+0000","last_clean":"2024-03-06T07:20:10.289992+0000","last_became_active":"2024-03-06T07:20:08.284415+0000","last_became_peered":"2024-03-06T07:20:08.284415+0000","last_unstale":"2024-03-06T07:20:10.289992+0000","last_undegraded":"2024-03-06T07:20:10.289992+0000","last_fullsized":"2024-03-06T07:20:10.289992+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:05.954034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288642+0000","last_change":"2024-03-06T07:20:07.280915+0000","last_active":"2024-03-06T07:20:10.288642+0000","last_peered":"2024-03-06T07:20:10.288642+0000","last_clean":"2024-03-06T07:20:10.288642+0000","last_became_active":"2024-03-06T07:20:07.280659+0000","last_became_peered":"2024-03-06T07:20:07.280659+0000","last_unstale":"2024-03-06T07:20:10.288642+0000","last_undegraded":"2024-03-06T07:20:10.288642+0000","last_fullsized":"2024-03-06T07:20:10.288642+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:00.431630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280146+0000","last_change":"2024-03-06T07:19:54.041140+0000","last_active":"2024-03-06T07:20:09.280146+0000","last_peered":"2024-03-06T07:20:09.280146+0000","last_clean":"2024-03-06T07:20:09.280146+0000","last_became_active":"2024-03-06T07:19:54.040877+0000","last_became_peered":"2024-03-06T07:19:54.040877+0000","last_unstale":"2024-03-06T07:20:09.280146+0000","last_undegraded":"2024-03-06T07:20:09.280146+0000","last_fullsized":"2024-03-06T07:20:09.280146+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:29.971359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":28,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288877+0000","last_change":"2024-03-06T07:19:53.045210+0000","last_active":"2024-03-06T07:20:10.288877+0000","last_peered":"2024-03-06T07:20:10.288877+0000","last_clean":"2024-03-06T07:20:10.288877+0000","last_became_active":"2024-03-06T07:19:53.044950+0000","last_became_peered":"2024-03-06T07:19:53.044950+0000","last_unstale":"2024-03-06T07:20:10.288877+0000","last_undegraded":"2024-03-06T07:20:10.288877+0000","last_fullsized":"2024-03-06T07:20:10.288877+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:53.407226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279877+0000","last_change":"2024-03-06T07:20:08.290423+0000","last_active":"2024-03-06T07:20:09.279877+0000","last_peered":"2024-03-06T07:20:09.279877+0000","last_clean":"2024-03-06T07:20:09.279877+0000","last_became_active":"2024-03-06T07:20:08.290112+0000","last_became_peered":"2024-03-06T07:20:08.290112+0000","last_unstale":"2024-03-06T07:20:09.279877+0000","last_undegraded":"2024-03-06T07:20:09.279877+0000","last_fullsized":"2024-03-06T07:20:09.279877+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:57.838274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279743+0000","last_change":"2024-03-06T07:20:07.285841+0000","last_active":"2024-03-06T07:20:09.279743+0000","last_peered":"2024-03-06T07:20:09.279743+0000","last_clean":"2024-03-06T07:20:09.279743+0000","last_became_active":"2024-03-06T07:20:07.285505+0000","last_became_peered":"2024-03-06T07:20:07.285505+0000","last_unstale":"2024-03-06T07:20:09.279743+0000","last_undegraded":"2024-03-06T07:20:09.279743+0000","last_fullsized":"2024-03-06T07:20:09.279743+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:40.634986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272369+0000","last_change":"2024-03-06T07:19:54.058718+0000","last_active":"2024-03-06T07:20:10.272369+0000","last_peered":"2024-03-06T07:20:10.272369+0000","last_clean":"2024-03-06T07:20:10.272369+0000","last_became_active":"2024-03-06T07:19:54.058415+0000","last_became_peered":"2024-03-06T07:19:54.058415+0000","last_unstale":"2024-03-06T07:20:10.272369+0000","last_undegraded":"2024-03-06T07:20:10.272369+0000","last_fullsized":"2024-03-06T07:20:10.272369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279408+0000","last_change":"2024-03-06T07:19:53.046555+0000","last_active":"2024-03-06T07:20:09.279408+0000","last_peered":"2024-03-06T07:20:09.279408+0000","last_clean":"2024-03-06T07:20:09.279408+0000","last_became_active":"2024-03-06T07:19:53.046298+0000","last_became_peered":"2024-03-06T07:19:53.046298+0000","last_unstale":"2024-03-06T07:20:09.279408+0000","last_undegraded":"2024-03-06T07:20:09.279408+0000","last_fullsized":"2024-03-06T07:20:09.279408+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:19.243021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290624+0000","last_change":"2024-03-06T07:20:08.290149+0000","last_active":"2024-03-06T07:20:10.290624+0000","last_peered":"2024-03-06T07:20:10.290624+0000","last_clean":"2024-03-06T07:20:10.290624+0000","last_became_active":"2024-03-06T07:20:08.289623+0000","last_became_peered":"2024-03-06T07:20:08.289623+0000","last_unstale":"2024-03-06T07:20:10.290624+0000","last_undegraded":"2024-03-06T07:20:10.290624+0000","last_fullsized":"2024-03-06T07:20:10.290624+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:34.619396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279506+0000","last_change":"2024-03-06T07:20:07.285895+0000","last_active":"2024-03-06T07:20:09.279506+0000","last_peered":"2024-03-06T07:20:09.279506+0000","last_clean":"2024-03-06T07:20:09.279506+0000","last_became_active":"2024-03-06T07:20:07.285614+0000","last_became_peered":"2024-03-06T07:20:07.285614+0000","last_unstale":"2024-03-06T07:20:09.279506+0000","last_undegraded":"2024-03-06T07:20:09.279506+0000","last_fullsized":"2024-03-06T07:20:09.279506+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280187+0000","last_change":"2024-03-06T07:19:54.047654+0000","last_active":"2024-03-06T07:20:09.280187+0000","last_peered":"2024-03-06T07:20:09.280187+0000","last_clean":"2024-03-06T07:20:09.280187+0000","last_became_active":"2024-03-06T07:19:54.047339+0000","last_became_peered":"2024-03-06T07:19:54.047339+0000","last_unstale":"2024-03-06T07:20:09.280187+0000","last_undegraded":"2024-03-06T07:20:09.280187+0000","last_fullsized":"2024-03-06T07:20:09.280187+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:35.272829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280818+0000","last_change":"2024-03-06T07:19:53.040132+0000","last_active":"2024-03-06T07:20:09.280818+0000","last_peered":"2024-03-06T07:20:09.280818+0000","last_clean":"2024-03-06T07:20:09.280818+0000","last_became_active":"2024-03-06T07:19:53.039809+0000","last_became_peered":"2024-03-06T07:19:53.039809+0000","last_unstale":"2024-03-06T07:20:09.280818+0000","last_undegraded":"2024-03-06T07:20:09.280818+0000","last_fullsized":"2024-03-06T07:20:09.280818+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279774+0000","last_change":"2024-03-06T07:20:08.285155+0000","last_active":"2024-03-06T07:20:09.279774+0000","last_peered":"2024-03-06T07:20:09.279774+0000","last_clean":"2024-03-06T07:20:09.279774+0000","last_became_active":"2024-03-06T07:20:08.284896+0000","last_became_peered":"2024-03-06T07:20:08.284896+0000","last_unstale":"2024-03-06T07:20:09.279774+0000","last_undegraded":"2024-03-06T07:20:09.279774+0000","last_fullsized":"2024-03-06T07:20:09.279774+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:24.217084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289760+0000","last_change":"2024-03-06T07:20:07.288210+0000","last_active":"2024-03-06T07:20:10.289760+0000","last_peered":"2024-03-06T07:20:10.289760+0000","last_clean":"2024-03-06T07:20:10.289760+0000","last_became_active":"2024-03-06T07:20:07.287885+0000","last_became_peered":"2024-03-06T07:20:07.287885+0000","last_unstale":"2024-03-06T07:20:10.289760+0000","last_undegraded":"2024-03-06T07:20:10.289760+0000","last_fullsized":"2024-03-06T07:20:10.289760+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:15.206863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289029+0000","last_change":"2024-03-06T07:19:54.050847+0000","last_active":"2024-03-06T07:20:10.289029+0000","last_peered":"2024-03-06T07:20:10.289029+0000","last_clean":"2024-03-06T07:20:10.289029+0000","last_became_active":"2024-03-06T07:19:54.050378+0000","last_became_peered":"2024-03-06T07:19:54.050378+0000","last_unstale":"2024-03-06T07:20:10.289029+0000","last_undegraded":"2024-03-06T07:20:10.289029+0000","last_fullsized":"2024-03-06T07:20:10.289029+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:42.322644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290068+0000","last_change":"2024-03-06T07:19:53.045139+0000","last_active":"2024-03-06T07:20:10.290068+0000","last_peered":"2024-03-06T07:20:10.290068+0000","last_clean":"2024-03-06T07:20:10.290068+0000","last_became_active":"2024-03-06T07:19:53.044903+0000","last_became_peered":"2024-03-06T07:19:53.044903+0000","last_unstale":"2024-03-06T07:20:10.290068+0000","last_undegraded":"2024-03-06T07:20:10.290068+0000","last_fullsized":"2024-03-06T07:20:10.290068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:56.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279970+0000","last_change":"2024-03-06T07:20:08.296792+0000","last_active":"2024-03-06T07:20:09.279970+0000","last_peered":"2024-03-06T07:20:09.279970+0000","last_clean":"2024-03-06T07:20:09.279970+0000","last_became_active":"2024-03-06T07:20:08.296332+0000","last_became_peered":"2024-03-06T07:20:08.296332+0000","last_unstale":"2024-03-06T07:20:09.279970+0000","last_undegraded":"2024-03-06T07:20:09.279970+0000","last_fullsized":"2024-03-06T07:20:09.279970+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:20.006499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279921+0000","last_change":"2024-03-06T07:20:07.289518+0000","last_active":"2024-03-06T07:20:09.279921+0000","last_peered":"2024-03-06T07:20:09.279921+0000","last_clean":"2024-03-06T07:20:09.279921+0000","last_became_active":"2024-03-06T07:20:07.289213+0000","last_became_peered":"2024-03-06T07:20:07.289213+0000","last_unstale":"2024-03-06T07:20:09.279921+0000","last_undegraded":"2024-03-06T07:20:09.279921+0000","last_fullsized":"2024-03-06T07:20:09.279921+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:11.519229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279864+0000","last_change":"2024-03-06T07:19:54.059322+0000","last_active":"2024-03-06T07:20:09.279864+0000","last_peered":"2024-03-06T07:20:09.279864+0000","last_clean":"2024-03-06T07:20:09.279864+0000","last_became_active":"2024-03-06T07:19:54.059092+0000","last_became_peered":"2024-03-06T07:19:54.059092+0000","last_unstale":"2024-03-06T07:20:09.279864+0000","last_undegraded":"2024-03-06T07:20:09.279864+0000","last_fullsized":"2024-03-06T07:20:09.279864+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":30,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288081+0000","last_change":"2024-03-06T07:19:53.046815+0000","last_active":"2024-03-06T07:20:10.288081+0000","last_peered":"2024-03-06T07:20:10.288081+0000","last_clean":"2024-03-06T07:20:10.288081+0000","last_became_active":"2024-03-06T07:19:53.043687+0000","last_became_peered":"2024-03-06T07:19:53.043687+0000","last_unstale":"2024-03-06T07:20:10.288081+0000","last_undegraded":"2024-03-06T07:20:10.288081+0000","last_fullsized":"2024-03-06T07:20:10.288081+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:44.465449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288910+0000","last_change":"2024-03-06T07:20:08.284036+0000","last_active":"2024-03-06T07:20:10.288910+0000","last_peered":"2024-03-06T07:20:10.288910+0000","last_clean":"2024-03-06T07:20:10.288910+0000","last_became_active":"2024-03-06T07:20:08.283771+0000","last_became_peered":"2024-03-06T07:20:08.283771+0000","last_unstale":"2024-03-06T07:20:10.288910+0000","last_undegraded":"2024-03-06T07:20:10.288910+0000","last_fullsized":"2024-03-06T07:20:10.288910+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:04.742839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288216+0000","last_change":"2024-03-06T07:20:07.283273+0000","last_active":"2024-03-06T07:20:10.288216+0000","last_peered":"2024-03-06T07:20:10.288216+0000","last_clean":"2024-03-06T07:20:10.288216+0000","last_became_active":"2024-03-06T07:20:07.282989+0000","last_became_peered":"2024-03-06T07:20:07.282989+0000","last_unstale":"2024-03-06T07:20:10.288216+0000","last_undegraded":"2024-03-06T07:20:10.288216+0000","last_fullsized":"2024-03-06T07:20:10.288216+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:14.967028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288048+0000","last_change":"2024-03-06T07:19:54.057521+0000","last_active":"2024-03-06T07:20:10.288048+0000","last_peered":"2024-03-06T07:20:10.288048+0000","last_clean":"2024-03-06T07:20:10.288048+0000","last_became_active":"2024-03-06T07:19:54.057264+0000","last_became_peered":"2024-03-06T07:19:54.057264+0000","last_unstale":"2024-03-06T07:20:10.288048+0000","last_undegraded":"2024-03-06T07:20:10.288048+0000","last_fullsized":"2024-03-06T07:20:10.288048+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:47.224332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280115+0000","last_change":"2024-03-06T07:19:53.051189+0000","last_active":"2024-03-06T07:20:09.280115+0000","last_peered":"2024-03-06T07:20:09.280115+0000","last_clean":"2024-03-06T07:20:09.280115+0000","last_became_active":"2024-03-06T07:19:53.050897+0000","last_became_peered":"2024-03-06T07:19:53.050897+0000","last_unstale":"2024-03-06T07:20:09.280115+0000","last_undegraded":"2024-03-06T07:20:09.280115+0000","last_fullsized":"2024-03-06T07:20:09.280115+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:27.326973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274871+0000","last_change":"2024-03-06T07:20:08.292612+0000","last_active":"2024-03-06T07:20:10.274871+0000","last_peered":"2024-03-06T07:20:10.274871+0000","last_clean":"2024-03-06T07:20:10.274871+0000","last_became_active":"2024-03-06T07:20:08.292312+0000","last_became_peered":"2024-03-06T07:20:08.292312+0000","last_unstale":"2024-03-06T07:20:10.274871+0000","last_undegraded":"2024-03-06T07:20:10.274871+0000","last_fullsized":"2024-03-06T07:20:10.274871+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:34.962673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289212+0000","last_change":"2024-03-06T07:20:07.283920+0000","last_active":"2024-03-06T07:20:10.289212+0000","last_peered":"2024-03-06T07:20:10.289212+0000","last_clean":"2024-03-06T07:20:10.289212+0000","last_became_active":"2024-03-06T07:20:07.283631+0000","last_became_peered":"2024-03-06T07:20:07.283631+0000","last_unstale":"2024-03-06T07:20:10.289212+0000","last_undegraded":"2024-03-06T07:20:10.289212+0000","last_fullsized":"2024-03-06T07:20:10.289212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:13.297038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289789+0000","last_change":"2024-03-06T07:19:54.052438+0000","last_active":"2024-03-06T07:20:10.289789+0000","last_peered":"2024-03-06T07:20:10.289789+0000","last_clean":"2024-03-06T07:20:10.289789+0000","last_became_active":"2024-03-06T07:19:54.052185+0000","last_became_peered":"2024-03-06T07:19:54.052185+0000","last_unstale":"2024-03-06T07:20:10.289789+0000","last_undegraded":"2024-03-06T07:20:10.289789+0000","last_fullsized":"2024-03-06T07:20:10.289789+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:09.153529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.275194+0000","last_change":"2024-03-06T07:19:53.046903+0000","last_active":"2024-03-06T07:20:10.275194+0000","last_peered":"2024-03-06T07:20:10.275194+0000","last_clean":"2024-03-06T07:20:10.275194+0000","last_became_active":"2024-03-06T07:19:53.046639+0000","last_became_peered":"2024-03-06T07:19:53.046639+0000","last_unstale":"2024-03-06T07:20:10.275194+0000","last_undegraded":"2024-03-06T07:20:10.275194+0000","last_fullsized":"2024-03-06T07:20:10.275194+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:23.209022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279730+0000","last_change":"2024-03-06T07:20:08.285798+0000","last_active":"2024-03-06T07:20:09.279730+0000","last_peered":"2024-03-06T07:20:09.279730+0000","last_clean":"2024-03-06T07:20:09.279730+0000","last_became_active":"2024-03-06T07:20:08.285524+0000","last_became_peered":"2024-03-06T07:20:08.285524+0000","last_unstale":"2024-03-06T07:20:09.279730+0000","last_undegraded":"2024-03-06T07:20:09.279730+0000","last_fullsized":"2024-03-06T07:20:09.279730+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:49.507162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290343+0000","last_change":"2024-03-06T07:20:07.288781+0000","last_active":"2024-03-06T07:20:10.290343+0000","last_peered":"2024-03-06T07:20:10.290343+0000","last_clean":"2024-03-06T07:20:10.290343+0000","last_became_active":"2024-03-06T07:20:07.288548+0000","last_became_peered":"2024-03-06T07:20:07.288548+0000","last_unstale":"2024-03-06T07:20:10.290343+0000","last_undegraded":"2024-03-06T07:20:10.290343+0000","last_fullsized":"2024-03-06T07:20:10.290343+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.514155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279481+0000","last_change":"2024-03-06T07:19:54.053795+0000","last_active":"2024-03-06T07:20:09.279481+0000","last_peered":"2024-03-06T07:20:09.279481+0000","last_clean":"2024-03-06T07:20:09.279481+0000","last_became_active":"2024-03-06T07:19:54.053519+0000","last_became_peered":"2024-03-06T07:19:54.053519+0000","last_unstale":"2024-03-06T07:20:09.279481+0000","last_undegraded":"2024-03-06T07:20:09.279481+0000","last_fullsized":"2024-03-06T07:20:09.279481+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:20.250283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280490+0000","last_change":"2024-03-06T07:19:53.046151+0000","last_active":"2024-03-06T07:20:09.280490+0000","last_peered":"2024-03-06T07:20:09.280490+0000","last_clean":"2024-03-06T07:20:09.280490+0000","last_became_active":"2024-03-06T07:19:53.045797+0000","last_became_peered":"2024-03-06T07:19:53.045797+0000","last_unstale":"2024-03-06T07:20:09.280490+0000","last_undegraded":"2024-03-06T07:20:09.280490+0000","last_fullsized":"2024-03-06T07:20:09.280490+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280601+0000","last_change":"2024-03-06T07:20:08.293922+0000","last_active":"2024-03-06T07:20:09.280601+0000","last_peered":"2024-03-06T07:20:09.280601+0000","last_clean":"2024-03-06T07:20:09.280601+0000","last_became_active":"2024-03-06T07:20:08.293668+0000","last_became_peered":"2024-03-06T07:20:08.293668+0000","last_unstale":"2024-03-06T07:20:09.280601+0000","last_undegraded":"2024-03-06T07:20:09.280601+0000","last_fullsized":"2024-03-06T07:20:09.280601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:24.217084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288735+0000","last_change":"2024-03-06T07:20:07.284617+0000","last_active":"2024-03-06T07:20:10.288735+0000","last_peered":"2024-03-06T07:20:10.288735+0000","last_clean":"2024-03-06T07:20:10.288735+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:10.288735+0000","last_undegraded":"2024-03-06T07:20:10.288735+0000","last_fullsized":"2024-03-06T07:20:10.288735+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:28.252791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289335+0000","last_change":"2024-03-06T07:19:54.055324+0000","last_active":"2024-03-06T07:20:10.289335+0000","last_peered":"2024-03-06T07:20:10.289335+0000","last_clean":"2024-03-06T07:20:10.289335+0000","last_became_active":"2024-03-06T07:19:54.055081+0000","last_became_peered":"2024-03-06T07:19:54.055081+0000","last_unstale":"2024-03-06T07:20:10.289335+0000","last_undegraded":"2024-03-06T07:20:10.289335+0000","last_fullsized":"2024-03-06T07:20:10.289335+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:57.518725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280051+0000","last_change":"2024-03-06T07:19:53.048898+0000","last_active":"2024-03-06T07:20:09.280051+0000","last_peered":"2024-03-06T07:20:09.280051+0000","last_clean":"2024-03-06T07:20:09.280051+0000","last_became_active":"2024-03-06T07:19:53.048613+0000","last_became_peered":"2024-03-06T07:19:53.048613+0000","last_unstale":"2024-03-06T07:20:09.280051+0000","last_undegraded":"2024-03-06T07:20:09.280051+0000","last_fullsized":"2024-03-06T07:20:09.280051+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:25.309048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272659+0000","last_change":"2024-03-06T07:20:08.287444+0000","last_active":"2024-03-06T07:20:10.272659+0000","last_peered":"2024-03-06T07:20:10.272659+0000","last_clean":"2024-03-06T07:20:10.272659+0000","last_became_active":"2024-03-06T07:20:08.287125+0000","last_became_peered":"2024-03-06T07:20:08.287125+0000","last_unstale":"2024-03-06T07:20:10.272659+0000","last_undegraded":"2024-03-06T07:20:10.272659+0000","last_fullsized":"2024-03-06T07:20:10.272659+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:01.438975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279826+0000","last_change":"2024-03-06T07:20:07.285007+0000","last_active":"2024-03-06T07:20:09.279826+0000","last_peered":"2024-03-06T07:20:09.279826+0000","last_clean":"2024-03-06T07:20:09.279826+0000","last_became_active":"2024-03-06T07:20:07.284332+0000","last_became_peered":"2024-03-06T07:20:07.284332+0000","last_unstale":"2024-03-06T07:20:09.279826+0000","last_undegraded":"2024-03-06T07:20:09.279826+0000","last_fullsized":"2024-03-06T07:20:09.279826+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:55.549631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.271883+0000","last_change":"2024-03-06T07:19:54.056310+0000","last_active":"2024-03-06T07:20:10.271883+0000","last_peered":"2024-03-06T07:20:10.271883+0000","last_clean":"2024-03-06T07:20:10.271883+0000","last_became_active":"2024-03-06T07:19:54.056039+0000","last_became_peered":"2024-03-06T07:19:54.056039+0000","last_unstale":"2024-03-06T07:20:10.271883+0000","last_undegraded":"2024-03-06T07:20:10.271883+0000","last_fullsized":"2024-03-06T07:20:10.271883+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:54.414488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289547+0000","last_change":"2024-03-06T07:19:53.043949+0000","last_active":"2024-03-06T07:20:10.289547+0000","last_peered":"2024-03-06T07:20:10.289547+0000","last_clean":"2024-03-06T07:20:10.289547+0000","last_became_active":"2024-03-06T07:19:53.043639+0000","last_became_peered":"2024-03-06T07:19:53.043639+0000","last_unstale":"2024-03-06T07:20:10.289547+0000","last_undegraded":"2024-03-06T07:20:10.289547+0000","last_fullsized":"2024-03-06T07:20:10.289547+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:34.576980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281525+0000","last_change":"2024-03-06T07:20:08.282466+0000","last_active":"2024-03-06T07:20:09.281525+0000","last_peered":"2024-03-06T07:20:09.281525+0000","last_clean":"2024-03-06T07:20:09.281525+0000","last_became_active":"2024-03-06T07:20:08.282002+0000","last_became_peered":"2024-03-06T07:20:08.282002+0000","last_unstale":"2024-03-06T07:20:09.281525+0000","last_undegraded":"2024-03-06T07:20:09.281525+0000","last_fullsized":"2024-03-06T07:20:09.281525+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:18.762564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288667+0000","last_change":"2024-03-06T07:20:07.279146+0000","last_active":"2024-03-06T07:20:10.288667+0000","last_peered":"2024-03-06T07:20:10.288667+0000","last_clean":"2024-03-06T07:20:10.288667+0000","last_became_active":"2024-03-06T07:20:07.278872+0000","last_became_peered":"2024-03-06T07:20:07.278872+0000","last_unstale":"2024-03-06T07:20:10.288667+0000","last_undegraded":"2024-03-06T07:20:10.288667+0000","last_fullsized":"2024-03-06T07:20:10.288667+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:07.641475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280754+0000","last_change":"2024-03-06T07:19:54.040887+0000","last_active":"2024-03-06T07:20:09.280754+0000","last_peered":"2024-03-06T07:20:09.280754+0000","last_clean":"2024-03-06T07:20:09.280754+0000","last_became_active":"2024-03-06T07:19:54.040632+0000","last_became_peered":"2024-03-06T07:19:54.040632+0000","last_unstale":"2024-03-06T07:20:09.280754+0000","last_undegraded":"2024-03-06T07:20:09.280754+0000","last_fullsized":"2024-03-06T07:20:09.280754+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:10.221319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288832+0000","last_change":"2024-03-06T07:19:53.044330+0000","last_active":"2024-03-06T07:20:10.288832+0000","last_peered":"2024-03-06T07:20:10.288832+0000","last_clean":"2024-03-06T07:20:10.288832+0000","last_became_active":"2024-03-06T07:19:53.044066+0000","last_became_peered":"2024-03-06T07:19:53.044066+0000","last_unstale":"2024-03-06T07:20:10.288832+0000","last_undegraded":"2024-03-06T07:20:10.288832+0000","last_fullsized":"2024-03-06T07:20:10.288832+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:19.476961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279133+0000","last_change":"2024-03-06T07:20:08.284792+0000","last_active":"2024-03-06T07:20:09.279133+0000","last_peered":"2024-03-06T07:20:09.279133+0000","last_clean":"2024-03-06T07:20:09.279133+0000","last_became_active":"2024-03-06T07:20:08.284533+0000","last_became_peered":"2024-03-06T07:20:08.284533+0000","last_unstale":"2024-03-06T07:20:09.279133+0000","last_undegraded":"2024-03-06T07:20:09.279133+0000","last_fullsized":"2024-03-06T07:20:09.279133+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:09.713122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272685+0000","last_change":"2024-03-06T07:20:07.287646+0000","last_active":"2024-03-06T07:20:10.272685+0000","last_peered":"2024-03-06T07:20:10.272685+0000","last_clean":"2024-03-06T07:20:10.272685+0000","last_became_active":"2024-03-06T07:20:07.287378+0000","last_became_peered":"2024-03-06T07:20:07.287378+0000","last_unstale":"2024-03-06T07:20:10.272685+0000","last_undegraded":"2024-03-06T07:20:10.272685+0000","last_fullsized":"2024-03-06T07:20:10.272685+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.514155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280084+0000","last_change":"2024-03-06T07:19:54.049940+0000","last_active":"2024-03-06T07:20:09.280084+0000","last_peered":"2024-03-06T07:20:09.280084+0000","last_clean":"2024-03-06T07:20:09.280084+0000","last_became_active":"2024-03-06T07:19:54.049481+0000","last_became_peered":"2024-03-06T07:19:54.049481+0000","last_unstale":"2024-03-06T07:20:09.280084+0000","last_undegraded":"2024-03-06T07:20:09.280084+0000","last_fullsized":"2024-03-06T07:20:09.280084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:00.070051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279297+0000","last_change":"2024-03-06T07:19:53.048472+0000","last_active":"2024-03-06T07:20:09.279297+0000","last_peered":"2024-03-06T07:20:09.279297+0000","last_clean":"2024-03-06T07:20:09.279297+0000","last_became_active":"2024-03-06T07:19:53.048213+0000","last_became_peered":"2024-03-06T07:19:53.048213+0000","last_unstale":"2024-03-06T07:20:09.279297+0000","last_undegraded":"2024-03-06T07:20:09.279297+0000","last_fullsized":"2024-03-06T07:20:09.279297+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:19.721078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.275107+0000","last_change":"2024-03-06T07:20:08.285541+0000","last_active":"2024-03-06T07:20:10.275107+0000","last_peered":"2024-03-06T07:20:10.275107+0000","last_clean":"2024-03-06T07:20:10.275107+0000","last_became_active":"2024-03-06T07:20:08.285278+0000","last_became_peered":"2024-03-06T07:20:08.285278+0000","last_unstale":"2024-03-06T07:20:10.275107+0000","last_undegraded":"2024-03-06T07:20:10.275107+0000","last_fullsized":"2024-03-06T07:20:10.275107+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:34.484615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272590+0000","last_change":"2024-03-06T07:20:07.283238+0000","last_active":"2024-03-06T07:20:10.272590+0000","last_peered":"2024-03-06T07:20:10.272590+0000","last_clean":"2024-03-06T07:20:10.272590+0000","last_became_active":"2024-03-06T07:20:07.282996+0000","last_became_peered":"2024-03-06T07:20:07.282996+0000","last_unstale":"2024-03-06T07:20:10.272590+0000","last_undegraded":"2024-03-06T07:20:10.272590+0000","last_fullsized":"2024-03-06T07:20:10.272590+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:17.755219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289845+0000","last_change":"2024-03-06T07:19:54.042432+0000","last_active":"2024-03-06T07:20:10.289845+0000","last_peered":"2024-03-06T07:20:10.289845+0000","last_clean":"2024-03-06T07:20:10.289845+0000","last_became_active":"2024-03-06T07:19:54.042166+0000","last_became_peered":"2024-03-06T07:19:54.042166+0000","last_unstale":"2024-03-06T07:20:10.289845+0000","last_undegraded":"2024-03-06T07:20:10.289845+0000","last_fullsized":"2024-03-06T07:20:10.289845+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.231274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.274773+0000","last_change":"2024-03-06T07:19:53.044950+0000","last_active":"2024-03-06T07:20:10.274773+0000","last_peered":"2024-03-06T07:20:10.274773+0000","last_clean":"2024-03-06T07:20:10.274773+0000","last_became_active":"2024-03-06T07:19:53.044630+0000","last_became_peered":"2024-03-06T07:19:53.044630+0000","last_unstale":"2024-03-06T07:20:10.274773+0000","last_undegraded":"2024-03-06T07:20:10.274773+0000","last_fullsized":"2024-03-06T07:20:10.274773+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:05.924599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280669+0000","last_change":"2024-03-06T07:20:08.287941+0000","last_active":"2024-03-06T07:20:09.280669+0000","last_peered":"2024-03-06T07:20:09.280669+0000","last_clean":"2024-03-06T07:20:09.280669+0000","last_became_active":"2024-03-06T07:20:08.287332+0000","last_became_peered":"2024-03-06T07:20:08.287332+0000","last_unstale":"2024-03-06T07:20:09.280669+0000","last_undegraded":"2024-03-06T07:20:09.280669+0000","last_fullsized":"2024-03-06T07:20:09.280669+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:20.839748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.278917+0000","last_change":"2024-03-06T07:20:07.287051+0000","last_active":"2024-03-06T07:20:09.278917+0000","last_peered":"2024-03-06T07:20:09.278917+0000","last_clean":"2024-03-06T07:20:09.278917+0000","last_became_active":"2024-03-06T07:20:07.286569+0000","last_became_peered":"2024-03-06T07:20:07.286569+0000","last_unstale":"2024-03-06T07:20:09.278917+0000","last_undegraded":"2024-03-06T07:20:09.278917+0000","last_fullsized":"2024-03-06T07:20:09.278917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:02.341734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272137+0000","last_change":"2024-03-06T07:19:54.059591+0000","last_active":"2024-03-06T07:20:10.272137+0000","last_peered":"2024-03-06T07:20:10.272137+0000","last_clean":"2024-03-06T07:20:10.272137+0000","last_became_active":"2024-03-06T07:19:54.058814+0000","last_became_peered":"2024-03-06T07:19:54.058814+0000","last_unstale":"2024-03-06T07:20:10.272137+0000","last_undegraded":"2024-03-06T07:20:10.272137+0000","last_fullsized":"2024-03-06T07:20:10.272137+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:11.853623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":24,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281499+0000","last_change":"2024-03-06T07:19:53.040833+0000","last_active":"2024-03-06T07:20:09.281499+0000","last_peered":"2024-03-06T07:20:09.281499+0000","last_clean":"2024-03-06T07:20:09.281499+0000","last_became_active":"2024-03-06T07:19:53.040561+0000","last_became_peered":"2024-03-06T07:19:53.040561+0000","last_unstale":"2024-03-06T07:20:09.281499+0000","last_undegraded":"2024-03-06T07:20:09.281499+0000","last_fullsized":"2024-03-06T07:20:09.281499+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:12.315549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279011+0000","last_change":"2024-03-06T07:20:08.291312+0000","last_active":"2024-03-06T07:20:09.279011+0000","last_peered":"2024-03-06T07:20:09.279011+0000","last_clean":"2024-03-06T07:20:09.279011+0000","last_became_active":"2024-03-06T07:20:08.290482+0000","last_became_peered":"2024-03-06T07:20:08.290482+0000","last_unstale":"2024-03-06T07:20:09.279011+0000","last_undegraded":"2024-03-06T07:20:09.279011+0000","last_fullsized":"2024-03-06T07:20:09.279011+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:45.089464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279311+0000","last_change":"2024-03-06T07:20:07.280861+0000","last_active":"2024-03-06T07:20:09.279311+0000","last_peered":"2024-03-06T07:20:09.279311+0000","last_clean":"2024-03-06T07:20:09.279311+0000","last_became_active":"2024-03-06T07:20:07.280598+0000","last_became_peered":"2024-03-06T07:20:07.280598+0000","last_unstale":"2024-03-06T07:20:09.279311+0000","last_undegraded":"2024-03-06T07:20:09.279311+0000","last_fullsized":"2024-03-06T07:20:09.279311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:54.620623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289224+0000","last_change":"2024-03-06T07:19:54.042216+0000","last_active":"2024-03-06T07:20:10.289224+0000","last_peered":"2024-03-06T07:20:10.289224+0000","last_clean":"2024-03-06T07:20:10.289224+0000","last_became_active":"2024-03-06T07:19:54.041923+0000","last_became_peered":"2024-03-06T07:19:54.041923+0000","last_unstale":"2024-03-06T07:20:10.289224+0000","last_undegraded":"2024-03-06T07:20:10.289224+0000","last_fullsized":"2024-03-06T07:20:10.289224+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:07.043081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290843+0000","last_change":"2024-03-06T07:19:53.047959+0000","last_active":"2024-03-06T07:20:10.290843+0000","last_peered":"2024-03-06T07:20:10.290843+0000","last_clean":"2024-03-06T07:20:10.290843+0000","last_became_active":"2024-03-06T07:19:53.047680+0000","last_became_peered":"2024-03-06T07:19:53.047680+0000","last_unstale":"2024-03-06T07:20:10.290843+0000","last_undegraded":"2024-03-06T07:20:10.290843+0000","last_fullsized":"2024-03-06T07:20:10.290843+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:38.499903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272182+0000","last_change":"2024-03-06T07:20:08.279705+0000","last_active":"2024-03-06T07:20:10.272182+0000","last_peered":"2024-03-06T07:20:10.272182+0000","last_clean":"2024-03-06T07:20:10.272182+0000","last_became_active":"2024-03-06T07:20:08.279448+0000","last_became_peered":"2024-03-06T07:20:08.279448+0000","last_unstale":"2024-03-06T07:20:10.272182+0000","last_undegraded":"2024-03-06T07:20:10.272182+0000","last_fullsized":"2024-03-06T07:20:10.272182+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:53.510411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272225+0000","last_change":"2024-03-06T07:20:07.284739+0000","last_active":"2024-03-06T07:20:10.272225+0000","last_peered":"2024-03-06T07:20:10.272225+0000","last_clean":"2024-03-06T07:20:10.272225+0000","last_became_active":"2024-03-06T07:20:07.284477+0000","last_became_peered":"2024-03-06T07:20:07.284477+0000","last_unstale":"2024-03-06T07:20:10.272225+0000","last_undegraded":"2024-03-06T07:20:10.272225+0000","last_fullsized":"2024-03-06T07:20:10.272225+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:00.451319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287761+0000","last_change":"2024-03-06T07:19:54.043516+0000","last_active":"2024-03-06T07:20:10.287761+0000","last_peered":"2024-03-06T07:20:10.287761+0000","last_clean":"2024-03-06T07:20:10.287761+0000","last_became_active":"2024-03-06T07:19:54.041651+0000","last_became_peered":"2024-03-06T07:19:54.041651+0000","last_unstale":"2024-03-06T07:20:10.287761+0000","last_undegraded":"2024-03-06T07:20:10.287761+0000","last_fullsized":"2024-03-06T07:20:10.287761+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:04.528232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279699+0000","last_change":"2024-03-06T07:19:53.048966+0000","last_active":"2024-03-06T07:20:09.279699+0000","last_peered":"2024-03-06T07:20:09.279699+0000","last_clean":"2024-03-06T07:20:09.279699+0000","last_became_active":"2024-03-06T07:19:53.047988+0000","last_became_peered":"2024-03-06T07:19:53.047988+0000","last_unstale":"2024-03-06T07:20:09.279699+0000","last_undegraded":"2024-03-06T07:20:09.279699+0000","last_fullsized":"2024-03-06T07:20:09.279699+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:49.519186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281429+0000","last_change":"2024-03-06T07:20:08.296538+0000","last_active":"2024-03-06T07:20:09.281429+0000","last_peered":"2024-03-06T07:20:09.281429+0000","last_clean":"2024-03-06T07:20:09.281429+0000","last_became_active":"2024-03-06T07:20:08.296263+0000","last_became_peered":"2024-03-06T07:20:08.296263+0000","last_unstale":"2024-03-06T07:20:09.281429+0000","last_undegraded":"2024-03-06T07:20:09.281429+0000","last_fullsized":"2024-03-06T07:20:09.281429+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:48.612666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290324+0000","last_change":"2024-03-06T07:20:07.288004+0000","last_active":"2024-03-06T07:20:10.290324+0000","last_peered":"2024-03-06T07:20:10.290324+0000","last_clean":"2024-03-06T07:20:10.290324+0000","last_became_active":"2024-03-06T07:20:07.287745+0000","last_became_peered":"2024-03-06T07:20:07.287745+0000","last_unstale":"2024-03-06T07:20:10.290324+0000","last_undegraded":"2024-03-06T07:20:10.290324+0000","last_fullsized":"2024-03-06T07:20:10.290324+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:02.341734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279694+0000","last_change":"2024-03-06T07:19:54.062368+0000","last_active":"2024-03-06T07:20:09.279694+0000","last_peered":"2024-03-06T07:20:09.279694+0000","last_clean":"2024-03-06T07:20:09.279694+0000","last_became_active":"2024-03-06T07:19:54.060740+0000","last_became_peered":"2024-03-06T07:19:54.060740+0000","last_unstale":"2024-03-06T07:20:09.279694+0000","last_undegraded":"2024-03-06T07:20:09.279694+0000","last_fullsized":"2024-03-06T07:20:09.279694+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:45.472711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288557+0000","last_change":"2024-03-06T07:19:53.044944+0000","last_active":"2024-03-06T07:20:10.288557+0000","last_peered":"2024-03-06T07:20:10.288557+0000","last_clean":"2024-03-06T07:20:10.288557+0000","last_became_active":"2024-03-06T07:19:53.038900+0000","last_became_peered":"2024-03-06T07:19:53.038900+0000","last_unstale":"2024-03-06T07:20:10.288557+0000","last_undegraded":"2024-03-06T07:20:10.288557+0000","last_fullsized":"2024-03-06T07:20:10.288557+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:59.062789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281418+0000","last_change":"2024-03-06T07:20:08.282575+0000","last_active":"2024-03-06T07:20:09.281418+0000","last_peered":"2024-03-06T07:20:09.281418+0000","last_clean":"2024-03-06T07:20:09.281418+0000","last_became_active":"2024-03-06T07:20:08.282325+0000","last_became_peered":"2024-03-06T07:20:08.282325+0000","last_unstale":"2024-03-06T07:20:09.281418+0000","last_undegraded":"2024-03-06T07:20:09.281418+0000","last_fullsized":"2024-03-06T07:20:09.281418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:01.458664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288696+0000","last_change":"2024-03-06T07:20:07.281920+0000","last_active":"2024-03-06T07:20:10.288696+0000","last_peered":"2024-03-06T07:20:10.288696+0000","last_clean":"2024-03-06T07:20:10.288696+0000","last_became_active":"2024-03-06T07:20:07.281654+0000","last_became_peered":"2024-03-06T07:20:07.281654+0000","last_unstale":"2024-03-06T07:20:10.288696+0000","last_undegraded":"2024-03-06T07:20:10.288696+0000","last_fullsized":"2024-03-06T07:20:10.288696+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:14.013690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272500+0000","last_change":"2024-03-06T07:19:54.059347+0000","last_active":"2024-03-06T07:20:10.272500+0000","last_peered":"2024-03-06T07:20:10.272500+0000","last_clean":"2024-03-06T07:20:10.272500+0000","last_became_active":"2024-03-06T07:19:54.059083+0000","last_became_peered":"2024-03-06T07:19:54.059083+0000","last_unstale":"2024-03-06T07:20:10.272500+0000","last_undegraded":"2024-03-06T07:20:10.272500+0000","last_fullsized":"2024-03-06T07:20:10.272500+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:01.979875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.288915+0000","last_change":"2024-03-06T07:19:53.048645+0000","last_active":"2024-03-06T07:20:10.288915+0000","last_peered":"2024-03-06T07:20:10.288915+0000","last_clean":"2024-03-06T07:20:10.288915+0000","last_became_active":"2024-03-06T07:19:53.048406+0000","last_became_peered":"2024-03-06T07:19:53.048406+0000","last_unstale":"2024-03-06T07:20:10.288915+0000","last_undegraded":"2024-03-06T07:20:10.288915+0000","last_fullsized":"2024-03-06T07:20:10.288915+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:10.846361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290445+0000","last_change":"2024-03-06T07:20:08.284610+0000","last_active":"2024-03-06T07:20:10.290445+0000","last_peered":"2024-03-06T07:20:10.290445+0000","last_clean":"2024-03-06T07:20:10.290445+0000","last_became_active":"2024-03-06T07:20:08.284343+0000","last_became_peered":"2024-03-06T07:20:08.284343+0000","last_unstale":"2024-03-06T07:20:10.290445+0000","last_undegraded":"2024-03-06T07:20:10.290445+0000","last_fullsized":"2024-03-06T07:20:10.290445+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:30.212401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272571+0000","last_change":"2024-03-06T07:20:07.287091+0000","last_active":"2024-03-06T07:20:10.272571+0000","last_peered":"2024-03-06T07:20:10.272571+0000","last_clean":"2024-03-06T07:20:10.272571+0000","last_became_active":"2024-03-06T07:20:07.286303+0000","last_became_peered":"2024-03-06T07:20:07.286303+0000","last_unstale":"2024-03-06T07:20:10.272571+0000","last_undegraded":"2024-03-06T07:20:10.272571+0000","last_fullsized":"2024-03-06T07:20:10.272571+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.279975+0000","last_change":"2024-03-06T07:19:54.049719+0000","last_active":"2024-03-06T07:20:09.279975+0000","last_peered":"2024-03-06T07:20:09.279975+0000","last_clean":"2024-03-06T07:20:09.279975+0000","last_became_active":"2024-03-06T07:19:54.049222+0000","last_became_peered":"2024-03-06T07:19:54.049222+0000","last_unstale":"2024-03-06T07:20:09.279975+0000","last_undegraded":"2024-03-06T07:20:09.279975+0000","last_fullsized":"2024-03-06T07:20:09.279975+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:20.484223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280893+0000","last_change":"2024-03-06T07:19:53.045046+0000","last_active":"2024-03-06T07:20:09.280893+0000","last_peered":"2024-03-06T07:20:09.280893+0000","last_clean":"2024-03-06T07:20:09.280893+0000","last_became_active":"2024-03-06T07:19:53.044751+0000","last_became_peered":"2024-03-06T07:19:53.044751+0000","last_unstale":"2024-03-06T07:20:09.280893+0000","last_undegraded":"2024-03-06T07:20:09.280893+0000","last_fullsized":"2024-03-06T07:20:09.280893+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.004603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280351+0000","last_change":"2024-03-06T07:20:08.289713+0000","last_active":"2024-03-06T07:20:09.280351+0000","last_peered":"2024-03-06T07:20:09.280351+0000","last_clean":"2024-03-06T07:20:09.280351+0000","last_became_active":"2024-03-06T07:20:08.289464+0000","last_became_peered":"2024-03-06T07:20:08.289464+0000","last_unstale":"2024-03-06T07:20:09.280351+0000","last_undegraded":"2024-03-06T07:20:09.280351+0000","last_fullsized":"2024-03-06T07:20:09.280351+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:16.734109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.290664+0000","last_change":"2024-03-06T07:20:07.287811+0000","last_active":"2024-03-06T07:20:10.290664+0000","last_peered":"2024-03-06T07:20:10.290664+0000","last_clean":"2024-03-06T07:20:10.290664+0000","last_became_active":"2024-03-06T07:20:07.287540+0000","last_became_peered":"2024-03-06T07:20:07.287540+0000","last_unstale":"2024-03-06T07:20:10.290664+0000","last_undegraded":"2024-03-06T07:20:10.290664+0000","last_fullsized":"2024-03-06T07:20:10.290664+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.946689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289255+0000","last_change":"2024-03-06T07:19:54.039796+0000","last_active":"2024-03-06T07:20:10.289255+0000","last_peered":"2024-03-06T07:20:10.289255+0000","last_clean":"2024-03-06T07:20:10.289255+0000","last_became_active":"2024-03-06T07:19:54.039527+0000","last_became_peered":"2024-03-06T07:19:54.039527+0000","last_unstale":"2024-03-06T07:20:10.289255+0000","last_undegraded":"2024-03-06T07:20:10.289255+0000","last_fullsized":"2024-03-06T07:20:10.289255+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:49.114747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.272213+0000","last_change":"2024-03-06T07:19:53.043599+0000","last_active":"2024-03-06T07:20:10.272213+0000","last_peered":"2024-03-06T07:20:10.272213+0000","last_clean":"2024-03-06T07:20:10.272213+0000","last_became_active":"2024-03-06T07:19:53.043294+0000","last_became_peered":"2024-03-06T07:19:53.043294+0000","last_unstale":"2024-03-06T07:20:10.272213+0000","last_undegraded":"2024-03-06T07:20:10.272213+0000","last_fullsized":"2024-03-06T07:20:10.272213+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:56.511464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.281081+0000","last_change":"2024-03-06T07:20:08.284337+0000","last_active":"2024-03-06T07:20:09.281081+0000","last_peered":"2024-03-06T07:20:09.281081+0000","last_clean":"2024-03-06T07:20:09.281081+0000","last_became_active":"2024-03-06T07:20:08.284098+0000","last_became_peered":"2024-03-06T07:20:08.284098+0000","last_unstale":"2024-03-06T07:20:09.281081+0000","last_undegraded":"2024-03-06T07:20:09.281081+0000","last_fullsized":"2024-03-06T07:20:09.281081+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:07.257688+0000","last_clean_scrub_stamp":"2024-03-06T07:20:07.257688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:04.742839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.287921+0000","last_change":"2024-03-06T07:20:07.277710+0000","last_active":"2024-03-06T07:20:10.287921+0000","last_peered":"2024-03-06T07:20:10.287921+0000","last_clean":"2024-03-06T07:20:10.287921+0000","last_became_active":"2024-03-06T07:20:07.277346+0000","last_became_peered":"2024-03-06T07:20:07.277346+0000","last_unstale":"2024-03-06T07:20:10.287921+0000","last_undegraded":"2024-03-06T07:20:10.287921+0000","last_fullsized":"2024-03-06T07:20:10.287921+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:06.250343+0000","last_clean_scrub_stamp":"2024-03-06T07:20:06.250343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:50.955230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280174+0000","last_change":"2024-03-06T07:19:54.055814+0000","last_active":"2024-03-06T07:20:09.280174+0000","last_peered":"2024-03-06T07:20:09.280174+0000","last_clean":"2024-03-06T07:20:09.280174+0000","last_became_active":"2024-03-06T07:19:54.055480+0000","last_became_peered":"2024-03-06T07:19:54.055480+0000","last_unstale":"2024-03-06T07:20:09.280174+0000","last_undegraded":"2024-03-06T07:20:09.280174+0000","last_fullsized":"2024-03-06T07:20:09.280174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:53.023356+0000","last_clean_scrub_stamp":"2024-03-06T07:19:53.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:35.156134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:09.280086+0000","last_change":"2024-03-06T07:19:53.046120+0000","last_active":"2024-03-06T07:20:09.280086+0000","last_peered":"2024-03-06T07:20:09.280086+0000","last_clean":"2024-03-06T07:20:09.280086+0000","last_became_active":"2024-03-06T07:19:53.045816+0000","last_became_peered":"2024-03-06T07:19:53.045816+0000","last_unstale":"2024-03-06T07:20:09.280086+0000","last_undegraded":"2024-03-06T07:20:09.280086+0000","last_fullsized":"2024-03-06T07:20:09.280086+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:19:52.016094+0000","last_clean_scrub_stamp":"2024-03-06T07:19:52.016094+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:56.226679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":238,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:20:10.289815+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:20:10.289815+0000","last_peered":"2024-03-06T07:20:10.289815+0000","last_clean":"2024-03-06T07:20:10.289815+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:20:10.289815+0000","last_undegraded":"2024-03-06T07:20:10.289815+0000","last_fullsized":"2024-03-06T07:20:10.289815+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":757760,"data_stored":897510,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":679936,"data_stored":306928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":306264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":308436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":307876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":679936,"data_stored":307854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":308778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":741376,"data_stored":896666,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:16.202 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:20:16.202 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 3:0 2024-03-06T07:20:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.339+0000 7fa27a17a640 1 Processor -- start 2024-03-06T07:20:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.339+0000 7fa27a17a640 1 -- start start 2024-03-06T07:20:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.339+0000 7fa27a17a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa2740c8950 0x7fa2740c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.339+0000 7fa27a17a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa2741558d0 0x7fa274155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.339+0000 7fa27a17a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa274155130 con 0x7fa2740c69f0 2024-03-06T07:20:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.339+0000 7fa27a17a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa274155440 con 0x7fa2740c8950 2024-03-06T07:20:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.339+0000 7fa27a17a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2740ad480 con 0x7fa2741558d0 2024-03-06T07:20:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa272ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa2740c8950 0x7fa2740c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa2737fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa2740c69f0 0x7fa2740acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38628/0 (socket says 172.21.15.94:38628) 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa2737fe640 1 -- 172.21.15.94:0/3694814986 learned_addr learned my addr 172.21.15.94:0/3694814986 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa273fff640 1 --2- 172.21.15.94:0/3694814986 >> v2:172.21.15.145:3300/0 conn(0x7fa2741558d0 0x7fa274155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa272ffd640 1 -- 172.21.15.94:0/3694814986 >> v2:172.21.15.145:3300/0 conn(0x7fa2741558d0 msgr2=0x7fa274155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa272ffd640 1 --2- 172.21.15.94:0/3694814986 >> v2:172.21.15.145:3300/0 conn(0x7fa2741558d0 0x7fa274155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa272ffd640 1 -- 172.21.15.94:0/3694814986 >> v1:172.21.15.94:6789/0 conn(0x7fa2740c69f0 legacy=0x7fa2740acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.340+0000 7fa272ffd640 1 -- 172.21.15.94:0/3694814986 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2740c5d50 con 0x7fa2740c8950 2024-03-06T07:20:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa272ffd640 1 --2- 172.21.15.94:0/3694814986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa2740c6dd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa274074a60 tx=0x7fa26802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d63c1930af2665 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa2727fc640 1 -- 172.21.15.94:0/3694814986 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2680315a0 con 0x7fa2740c8950 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa2727fc640 1 -- 172.21.15.94:0/3694814986 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa268037070 con 0x7fa2740c8950 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa2727fc640 1 -- 172.21.15.94:0/3694814986 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268031960 con 0x7fa2740c8950 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 -- 172.21.15.94:0/3694814986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 msgr2=0x7fa2740c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 --2- 172.21.15.94:0/3694814986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa2740c6dd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa274074a60 tx=0x7fa26802b840 comp rx=0 tx=0).stop 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 -- 172.21.15.94:0/3694814986 shutdown_connections 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 --2- 172.21.15.94:0/3694814986 >> v2:172.21.15.145:3300/0 conn(0x7fa2741558d0 0x7fa274155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 --2- 172.21.15.94:0/3694814986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa2740c6dd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 -- 172.21.15.94:0/3694814986 >> 172.21.15.94:0/3694814986 conn(0x7fa2740cae70 msgr2=0x7fa2740cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 -- 172.21.15.94:0/3694814986 shutdown_connections 2024-03-06T07:20:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.341+0000 7fa27a17a640 1 -- 172.21.15.94:0/3694814986 wait complete. 2024-03-06T07:20:16.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.342+0000 7fa27a17a640 1 Processor -- start 2024-03-06T07:20:16.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa27a17a640 1 -- start start 2024-03-06T07:20:16.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa27a17a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2740c69f0 0x7fa2741591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa27a17a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa27414ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa27a17a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2741558d0 0x7fa2740c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa27a17a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274152a20 con 0x7fa2740c8950 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa27a17a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2740c6310 con 0x7fa2741558d0 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa27a17a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa274152ba0 con 0x7fa2740c69f0 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa272ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa27414ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa272ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa27414ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37748/0 (socket says 172.21.15.94:37748) 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa272ffd640 1 -- 172.21.15.94:0/888919192 learned_addr learned my addr 172.21.15.94:0/888919192 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa273fff640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2741558d0 0x7fa2740c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.343+0000 7fa2737fe640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2740c69f0 0x7fa2741591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa272ffd640 1 -- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2740c69f0 msgr2=0x7fa2741591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa272ffd640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2740c69f0 0x7fa2741591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa272ffd640 1 -- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2741558d0 msgr2=0x7fa2740c1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa272ffd640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2741558d0 0x7fa2740c1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa272ffd640 1 -- 172.21.15.94:0/888919192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2740c36f0 con 0x7fa2740c8950 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa272ffd640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa27414ea00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa268000c00 tx=0x7fa2680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa25bfff640 1 -- 172.21.15.94:0/888919192 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268032630 con 0x7fa2740c8950 2024-03-06T07:20:16.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa25bfff640 1 -- 172.21.15.94:0/888919192 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2680327d0 con 0x7fa2740c8950 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2740c4960 con 0x7fa2740c8950 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa25bfff640 1 -- 172.21.15.94:0/888919192 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268032ba0 con 0x7fa2740c8950 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.344+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2741ef160 con 0x7fa2740c8950 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.345+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2740c38c0 con 0x7fa2740c8950 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.346+0000 7fa25bfff640 1 -- 172.21.15.94:0/888919192 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa26804d3f0 con 0x7fa2740c8950 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.346+0000 7fa25bfff640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26006b4d0 0x7fa26006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.346+0000 7fa25bfff640 1 -- 172.21.15.94:0/888919192 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa26802f030 con 0x7fa2740c8950 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.346+0000 7fa2737fe640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26006b4d0 0x7fa26006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.347+0000 7fa2737fe640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26006b4d0 0x7fa26006d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa2740c2ce0 tx=0x7fa264007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:16.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.350+0000 7fa25bfff640 1 -- 172.21.15.94:0/888919192 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa268040070 con 0x7fa2740c8950 2024-03-06T07:20:16.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.599+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "3:0"} v 0) v1 -- 0x7fa274067910 con 0x7fa2740c8950 2024-03-06T07:20:16.601 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:20:16.600+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:20:16.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.601+0000 7fa25bfff640 1 -- 172.21.15.94:0/888919192 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "3:0"}]=0 failed mds gid 35084 v33) v1 ==== 98+0+0 (secure 0 0 0) 0x7fa274067910 con 0x7fa2740c8950 2024-03-06T07:20:16.603 INFO:teuthology.orchestra.run.smithi094.stderr:failed mds gid 35084 2024-03-06T07:20:16.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26006b4d0 msgr2=0x7fa26006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26006b4d0 0x7fa26006d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa2740c2ce0 tx=0x7fa264007e80 comp rx=0 tx=0).stop 2024-03-06T07:20:16.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 msgr2=0x7fa27414ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa27414ea00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa268000c00 tx=0x7fa2680046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 shutdown_connections 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26006b4d0 0x7fa26006d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2741558d0 0x7fa2740c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2740c8950 0x7fa27414ea00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 --2- 172.21.15.94:0/888919192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2740c69f0 0x7fa2741591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 >> 172.21.15.94:0/888919192 conn(0x7fa2740cae70 msgr2=0x7fa274150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 shutdown_connections 2024-03-06T07:20:16.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.604+0000 7fa27a17a640 1 -- 172.21.15.94:0/888919192 wait complete. 2024-03-06T07:20:16.608 INFO:tasks.ceph.mds.b.smithi145.stderr: -578> 2024-03-06T07:19:59.293+0000 7f8d655e1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:20:16.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:16.635 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:20:16.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.720+0000 7f46c68c9640 1 Processor -- start 2024-03-06T07:20:16.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.720+0000 7f46c68c9640 1 -- start start 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46c68c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f46c0155ba0 0x7f46c00c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46c68c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f46c01586f0 0x7f46c0158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46c68c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c01591c0 con 0x7f46c00c5480 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46c68c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c006df40 con 0x7f46c0155ba0 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46c68c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c006fdb0 con 0x7f46c01586f0 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46bf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f46c0155ba0 0x7f46c00c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46bf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f46c0155ba0 0x7f46c00c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37752/0 (socket says 172.21.15.94:37752) 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46bf7fe640 1 -- 172.21.15.94:0/3276804689 learned_addr learned my addr 172.21.15.94:0/3276804689 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:16.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46bf7fe640 1 -- 172.21.15.94:0/3276804689 >> v2:172.21.15.145:3300/0 conn(0x7f46c01586f0 msgr2=0x7f46c0158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46c4e66640 1 --2- 172.21.15.94:0/3276804689 >> v2:172.21.15.145:3300/0 conn(0x7f46c01586f0 0x7f46c0158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46bf7fe640 1 --2- 172.21.15.94:0/3276804689 >> v2:172.21.15.145:3300/0 conn(0x7f46c01586f0 0x7f46c0158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46bf7fe640 1 -- 172.21.15.94:0/3276804689 >> v1:172.21.15.145:6789/0 conn(0x7f46c00c5480 legacy=0x7f46c006f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.721+0000 7f46bf7fe640 1 -- 172.21.15.94:0/3276804689 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c01594a0 con 0x7f46c0155ba0 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.722+0000 7f46bf7fe640 1 --2- 172.21.15.94:0/3276804689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c0155ba0 0x7f46c00c5860 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f46c00ab500 tx=0x7f46b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18195813ef660c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.722+0000 7f46beffd640 1 -- 172.21.15.94:0/3276804689 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b002e070 con 0x7f46c0155ba0 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.722+0000 7f46beffd640 1 -- 172.21.15.94:0/3276804689 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b0037070 con 0x7f46c0155ba0 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.722+0000 7f46beffd640 1 -- 172.21.15.94:0/3276804689 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0031a00 con 0x7f46c0155ba0 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 -- 172.21.15.94:0/3276804689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c0155ba0 msgr2=0x7f46c00c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3276804689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c0155ba0 0x7f46c00c5860 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f46c00ab500 tx=0x7f46b002b880 comp rx=0 tx=0).stop 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 -- 172.21.15.94:0/3276804689 shutdown_connections 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3276804689 >> v2:172.21.15.145:3300/0 conn(0x7f46c01586f0 0x7f46c0158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3276804689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c0155ba0 0x7f46c00c5860 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 -- 172.21.15.94:0/3276804689 >> 172.21.15.94:0/3276804689 conn(0x7f46c006b1a0 msgr2=0x7f46c006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:16.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 -- 172.21.15.94:0/3276804689 shutdown_connections 2024-03-06T07:20:16.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.723+0000 7f46c68c9640 1 -- 172.21.15.94:0/3276804689 wait complete. 2024-03-06T07:20:16.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.725+0000 7f46c68c9640 1 Processor -- start 2024-03-06T07:20:16.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.725+0000 7f46c68c9640 1 -- start start 2024-03-06T07:20:16.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c68c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c00c5480 0x7f46c014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c68c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c0155ba0 0x7f46c0156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c68c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01586f0 0x7f46c01571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c68c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0155680 con 0x7f46c01586f0 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c68c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0156000 con 0x7f46c0155ba0 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c68c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0152a20 con 0x7f46c00c5480 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01586f0 0x7f46c01571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01586f0 0x7f46c01571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37768/0 (socket says 172.21.15.94:37768) 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 -- 172.21.15.94:0/3248720566 learned_addr learned my addr 172.21.15.94:0/3248720566 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46bf7fe640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c0155ba0 0x7f46c0156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46bffff640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c00c5480 0x7f46c014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 -- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c00c5480 msgr2=0x7f46c014d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c00c5480 0x7f46c014d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 -- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c0155ba0 msgr2=0x7f46c0156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c0155ba0 0x7f46c0156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.726+0000 7f46c4e66640 1 -- 172.21.15.94:0/3248720566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c01e1100 con 0x7f46c01586f0 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.727+0000 7f46c4e66640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01586f0 0x7f46c01571d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f46b800ccb0 tx=0x7f46b800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:16.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.727+0000 7f46bcff9640 1 -- 172.21.15.94:0/3248720566 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b800ce60 con 0x7f46c01586f0 2024-03-06T07:20:16.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.727+0000 7f46bcff9640 1 -- 172.21.15.94:0/3248720566 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b8003990 con 0x7f46c01586f0 2024-03-06T07:20:16.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.727+0000 7f46bcff9640 1 -- 172.21.15.94:0/3248720566 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8003d60 con 0x7f46c01586f0 2024-03-06T07:20:16.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.727+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c01e13e0 con 0x7f46c01586f0 2024-03-06T07:20:16.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.727+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c014d180 con 0x7f46c01586f0 2024-03-06T07:20:16.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.728+0000 7f46bcff9640 1 -- 172.21.15.94:0/3248720566 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46b8003f00 con 0x7f46c01586f0 2024-03-06T07:20:16.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.728+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c0075300 con 0x7f46c01586f0 2024-03-06T07:20:16.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.728+0000 7f46bcff9640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46ac06b500 0x7f46ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:16.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.729+0000 7f46bcff9640 1 -- 172.21.15.94:0/3248720566 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f46b808aac0 con 0x7f46c01586f0 2024-03-06T07:20:16.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.729+0000 7f46bffff640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46ac06b500 0x7f46ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:16.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.729+0000 7f46bffff640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46ac06b500 0x7f46ac06d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f46b4004ea0 tx=0x7f46b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:16.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.733+0000 7f46bcff9640 1 -- 172.21.15.94:0/3248720566 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46c0075300 con 0x7f46c01586f0 2024-03-06T07:20:16.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.955+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46c004f330 con 0x7f46c01586f0 2024-03-06T07:20:16.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.955+0000 7f46bcff9640 1 -- 172.21.15.94:0/3248720566 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5459 (secure 0 0 0) 0x7f46b8014030 con 0x7f46c01586f0 2024-03-06T07:20:16.957 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:16.957 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:20:16.604943+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5521":{"gid":5521,"name":"a","rank":0,"incarnation":34,"state":"up:replay","state_seq":6,"addr":"172.21.15.94:6835/480754107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":480754107},{"type":"v1","addr":"172.21.15.94:6835","nonce":480754107}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:16.958 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 34 2024-03-06T07:20:16.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46ac06b500 msgr2=0x7f46ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46ac06b500 0x7f46ac06d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f46b4004ea0 tx=0x7f46b4009040 comp rx=0 tx=0).stop 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01586f0 msgr2=0x7f46c01571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01586f0 0x7f46c01571d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f46b800ccb0 tx=0x7f46b800a5c0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 shutdown_connections 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46ac06b500 0x7f46ac06d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01586f0 0x7f46c01571d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c0155ba0 0x7f46c0156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 --2- 172.21.15.94:0/3248720566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c00c5480 0x7f46c014d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 >> 172.21.15.94:0/3248720566 conn(0x7f46c006b1a0 msgr2=0x7f46c00707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 shutdown_connections 2024-03-06T07:20:16.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:16.957+0000 7f46c68c9640 1 -- 172.21.15.94:0/3248720566 wait complete. 2024-03-06T07:20:16.974 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-03-06T07:20:16.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-06T07:20:17.107 INFO:teuthology.orchestra.run.smithi094.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-06T07:20:17.107 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-06T07:20:17.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:17.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.206+0000 7fd902f1e640 1 Processor -- start 2024-03-06T07:20:17.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.206+0000 7fd902f1e640 1 -- start start 2024-03-06T07:20:17.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.206+0000 7fd902f1e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd8fc14f360 0x7fd8fc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.206+0000 7fd902f1e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd8fc158b60 0x7fd8fc158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.206+0000 7fd902f1e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0c73d0 con 0x7fd8fc0c88c0 2024-03-06T07:20:17.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.206+0000 7fd902f1e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc0c8dd0 con 0x7fd8fc14f360 2024-03-06T07:20:17.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.206+0000 7fd902f1e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc0be2e0 con 0x7fd8fc158b60 2024-03-06T07:20:17.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd900cba640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd8fc14f360 0x7fd8fc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd8fbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd8fc0c88c0 0x7fd8fc0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38636/0 (socket says 172.21.15.94:38636) 2024-03-06T07:20:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd8fbfff640 1 -- 172.21.15.94:0/1558685786 learned_addr learned my addr 172.21.15.94:0/1558685786 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd9014bb640 1 --2- 172.21.15.94:0/1558685786 >> v2:172.21.15.145:3300/0 conn(0x7fd8fc158b60 0x7fd8fc158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd900cba640 1 -- 172.21.15.94:0/1558685786 >> v2:172.21.15.145:3300/0 conn(0x7fd8fc158b60 msgr2=0x7fd8fc158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd900cba640 1 --2- 172.21.15.94:0/1558685786 >> v2:172.21.15.145:3300/0 conn(0x7fd8fc158b60 0x7fd8fc158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd900cba640 1 -- 172.21.15.94:0/1558685786 >> v1:172.21.15.94:6789/0 conn(0x7fd8fc0c88c0 legacy=0x7fd8fc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd900cba640 1 -- 172.21.15.94:0/1558685786 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc0c7670 con 0x7fd8fc14f360 2024-03-06T07:20:17.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.207+0000 7fd8fb7fe640 1 -- 172.21.15.94:0/1558685786 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582156014 0 0) 0x7fd8fc0c73d0 con 0x7fd8fc0c88c0 2024-03-06T07:20:17.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.208+0000 7fd900cba640 1 --2- 172.21.15.94:0/1558685786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc155c10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc152d60 tx=0x7fd8f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7667420c90a73ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:17.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.208+0000 7fd8fb7fe640 1 -- 172.21.15.94:0/1558685786 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f00315a0 con 0x7fd8fc14f360 2024-03-06T07:20:17.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.208+0000 7fd8fb7fe640 1 -- 172.21.15.94:0/1558685786 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0037070 con 0x7fd8fc14f360 2024-03-06T07:20:17.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.208+0000 7fd8fb7fe640 1 -- 172.21.15.94:0/1558685786 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0031960 con 0x7fd8fc14f360 2024-03-06T07:20:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.208+0000 7fd902f1e640 1 -- 172.21.15.94:0/1558685786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 msgr2=0x7fd8fc155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.208+0000 7fd902f1e640 1 --2- 172.21.15.94:0/1558685786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc155c10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc152d60 tx=0x7fd8f002b840 comp rx=0 tx=0).stop 2024-03-06T07:20:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.209+0000 7fd902f1e640 1 -- 172.21.15.94:0/1558685786 shutdown_connections 2024-03-06T07:20:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.209+0000 7fd902f1e640 1 --2- 172.21.15.94:0/1558685786 >> v2:172.21.15.145:3300/0 conn(0x7fd8fc158b60 0x7fd8fc158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.209+0000 7fd902f1e640 1 --2- 172.21.15.94:0/1558685786 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc155c10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.209+0000 7fd902f1e640 1 -- 172.21.15.94:0/1558685786 >> 172.21.15.94:0/1558685786 conn(0x7fd8fc0ac570 msgr2=0x7fd8fc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.209+0000 7fd902f1e640 1 -- 172.21.15.94:0/1558685786 shutdown_connections 2024-03-06T07:20:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.209+0000 7fd902f1e640 1 -- 172.21.15.94:0/1558685786 wait complete. 2024-03-06T07:20:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.210+0000 7fd902f1e640 1 Processor -- start 2024-03-06T07:20:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.210+0000 7fd902f1e640 1 -- start start 2024-03-06T07:20:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd902f1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc0c88c0 0x7fd8fc0c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd902f1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc14f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd902f1e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc158b60 0x7fd8fc0bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd902f1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0c5a50 con 0x7fd8fc14f360 2024-03-06T07:20:17.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd902f1e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0c7c30 con 0x7fd8fc158b60 2024-03-06T07:20:17.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd902f1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0c5bd0 con 0x7fd8fc0c88c0 2024-03-06T07:20:17.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc14f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc14f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37778/0 (socket says 172.21.15.94:37778) 2024-03-06T07:20:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 -- 172.21.15.94:0/43245963 learned_addr learned my addr 172.21.15.94:0/43245963 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd9014bb640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc158b60 0x7fd8fc0bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd900cba640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc0c88c0 0x7fd8fc0c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 -- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc0c88c0 msgr2=0x7fd8fc0c3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc0c88c0 0x7fd8fc0c3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 -- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc158b60 msgr2=0x7fd8fc0bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc158b60 0x7fd8fc0bf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 -- 172.21.15.94:0/43245963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc0bfd60 con 0x7fd8fc14f360 2024-03-06T07:20:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.211+0000 7fd8fbfff640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc14f740 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c440 tx=0x7fd8e800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.212+0000 7fd8f97fa640 1 -- 172.21.15.94:0/43245963 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8018070 con 0x7fd8fc14f360 2024-03-06T07:20:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.212+0000 7fd8f97fa640 1 -- 172.21.15.94:0/43245963 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e80074e0 con 0x7fd8fc14f360 2024-03-06T07:20:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.212+0000 7fd8f97fa640 1 -- 172.21.15.94:0/43245963 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8004a20 con 0x7fd8fc14f360 2024-03-06T07:20:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.212+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc1ebc60 con 0x7fd8fc14f360 2024-03-06T07:20:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.212+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc0be8d0 con 0x7fd8fc14f360 2024-03-06T07:20:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.213+0000 7fd8f97fa640 1 -- 172.21.15.94:0/43245963 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8e801b070 con 0x7fd8fc14f360 2024-03-06T07:20:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.213+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc0c9f60 con 0x7fd8fc14f360 2024-03-06T07:20:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.213+0000 7fd8f97fa640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.213+0000 7fd8f97fa640 1 -- 172.21.15.94:0/43245963 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fd8e80898f0 con 0x7fd8fc14f360 2024-03-06T07:20:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.214+0000 7fd900cba640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.214+0000 7fd900cba640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd8e006b4d0 0x7fd8e006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0036000 tx=0x7fd8f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.218+0000 7fd8f97fa640 1 -- 172.21.15.94:0/43245963 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8fc0c9f60 con 0x7fd8fc14f360 2024-03-06T07:20:17.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.456+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8fc04f330 con 0x7fd8fc14f360 2024-03-06T07:20:17.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.457+0000 7fd8f97fa640 1 -- 172.21.15.94:0/43245963 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5459 (secure 0 0 0) 0x7fd8e80582c0 con 0x7fd8fc14f360 2024-03-06T07:20:17.459 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:17.459 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:20:16.604943+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5521":{"gid":5521,"name":"a","rank":0,"incarnation":34,"state":"up:replay","state_seq":6,"addr":"172.21.15.94:6835/480754107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":480754107},{"type":"v1","addr":"172.21.15.94:6835","nonce":480754107}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:17.459 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 34 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd8e006b4d0 msgr2=0x7fd8e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd8e006b4d0 0x7fd8e006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0036000 tx=0x7fd8f0008030 comp rx=0 tx=0).stop 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 msgr2=0x7fd8fc14f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc14f740 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c440 tx=0x7fd8e800c900 comp rx=0 tx=0).stop 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 shutdown_connections 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc158b60 0x7fd8fc0bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8fc14f360 0x7fd8fc14f740 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 --2- 172.21.15.94:0/43245963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc0c88c0 0x7fd8fc0c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 >> 172.21.15.94:0/43245963 conn(0x7fd8fc0ac570 msgr2=0x7fd8fc14e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:17.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 shutdown_connections 2024-03-06T07:20:17.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:17.459+0000 7fd902f1e640 1 -- 172.21.15.94:0/43245963 wait complete. 2024-03-06T07:20:17.472 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-06T07:20:19.660 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:20:22.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:22.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.584+0000 7f6045f25640 1 Processor -- start 2024-03-06T07:20:22.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.584+0000 7f6045f25640 1 -- start start 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.584+0000 7f6045f25640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 0x7f6040155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.584+0000 7f6045f25640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60400c73d0 con 0x7f6040158b60 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.584+0000 7f6045f25640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60400be2e0 con 0x7f604014f360 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.584+0000 7f6045f25640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6040159700 con 0x7f60400c88c0 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6040158b60 0x7f6040158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35884/0 (socket says 172.21.15.94:35884) 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603ffff640 1 -- 172.21.15.94:0/2930338490 learned_addr learned my addr 172.21.15.94:0/2930338490 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603effd640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 0x7f6040155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302009333 0 0) 0x7f60400c73d0 con 0x7f6040158b60 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028003520 con 0x7f6040158b60 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110755597 0 0) 0x7f60400be2e0 con 0x7f604014f360 2024-03-06T07:20:22.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60400c73d0 con 0x7f604014f360 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3502357318 0 0) 0x7f6028003520 con 0x7f6040158b60 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 msgr2=0x7f6040155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 0x7f6040155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 >> v1:172.21.15.145:6789/0 conn(0x7f604014f360 legacy=0x7f60400c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400c7670 con 0x7f6040158b60 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f60380019f0 con 0x7f6040158b60 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 >> v1:172.21.15.94:6789/0 conn(0x7f6040158b60 legacy=0x7f6040158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6028003b70 0x7f6028003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f6028004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 0x7f60280051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6040159700 con 0x7f6028004800 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602800bad0 con 0x7f6028003b70 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.585+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602800bcd0 con 0x7f60400c88c0 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603f7fe640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6028003b70 0x7f6028003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603ffff640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 0x7f60280051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603effd640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f6028004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603effd640 1 -- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 msgr2=0x7f60280051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603effd640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60400c88c0 0x7f60280051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603effd640 1 -- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6028003b70 msgr2=0x7f6028003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603effd640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6028003b70 0x7f6028003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603effd640 1 -- 172.21.15.94:0/2930338490 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6028003520 con 0x7f6028004800 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.586+0000 7f603f7fe640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6028003b70 0x7f6028003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.587+0000 7f603effd640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f6028004c80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6034009a90 tx=0x7f6028005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982d63cf3734701f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.587+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034039070 con 0x7f6028004800 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.587+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603402c680 con 0x7f6028004800 2024-03-06T07:20:22.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.587+0000 7f603e7fc640 1 -- 172.21.15.94:0/2930338490 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034009690 con 0x7f6028004800 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.587+0000 7f6045f25640 1 -- 172.21.15.94:0/2930338490 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 msgr2=0x7f6028004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.587+0000 7f6045f25640 1 --2- 172.21.15.94:0/2930338490 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f6028004c80 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6034009a90 tx=0x7f6028005830 comp rx=0 tx=0).stop 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.587+0000 7f603ffff640 1 -- 172.21.15.94:0/2930338490 reap_dead start 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.588+0000 7f6045f25640 1 -- 172.21.15.94:0/2930338490 shutdown_connections 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.588+0000 7f6045f25640 1 -- 172.21.15.94:0/2930338490 >> 172.21.15.94:0/2930338490 conn(0x7f60400ac570 msgr2=0x7f60400ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.588+0000 7f6045f25640 1 -- 172.21.15.94:0/2930338490 shutdown_connections 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.588+0000 7f6045f25640 1 -- 172.21.15.94:0/2930338490 wait complete. 2024-03-06T07:20:22.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 Processor -- start 2024-03-06T07:20:22.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 -- start start 2024-03-06T07:20:22.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f60401ee770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f604014c350 0x7f6040147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60400c1050 0x7f6040148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400c7c30 con 0x7f6028004800 2024-03-06T07:20:22.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400c5a50 con 0x7f60400c1050 2024-03-06T07:20:22.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.589+0000 7f6045f25640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60400c5bd0 con 0x7f604014c350 2024-03-06T07:20:22.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f60401ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f60401ee770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56136/0 (socket says 172.21.15.94:56136) 2024-03-06T07:20:22.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 -- 172.21.15.94:0/3982585450 learned_addr learned my addr 172.21.15.94:0/3982585450 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:22.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 -- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f604014c350 msgr2=0x7f6040147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f604014c350 0x7f6040147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 -- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60400c1050 msgr2=0x7f6040148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603ffff640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60400c1050 0x7f6040148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60400c1050 0x7f6040148170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 -- 172.21.15.94:0/3982585450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400c1ea0 con 0x7f6028004800 2024-03-06T07:20:22.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f603f7fe640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f60401ee770 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f603000fb40 tx=0x7f6030001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:22.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f6027fff640 1 -- 172.21.15.94:0/3982585450 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60300052e0 con 0x7f6028004800 2024-03-06T07:20:22.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f6027fff640 1 -- 172.21.15.94:0/3982585450 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6030018070 con 0x7f6028004800 2024-03-06T07:20:22.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60400c1bc0 con 0x7f6028004800 2024-03-06T07:20:22.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f6027fff640 1 -- 172.21.15.94:0/3982585450 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603000d990 con 0x7f6028004800 2024-03-06T07:20:22.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.590+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60401eecb0 con 0x7f6028004800 2024-03-06T07:20:22.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.592+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604004f330 con 0x7f6028004800 2024-03-06T07:20:22.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.592+0000 7f6027fff640 1 -- 172.21.15.94:0/3982585450 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f603000db30 con 0x7f6028004800 2024-03-06T07:20:22.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.592+0000 7f6027fff640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602c06b4d0 0x7f602c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:22.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.592+0000 7f6027fff640 1 -- 172.21.15.94:0/3982585450 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f603008a7c0 con 0x7f6028004800 2024-03-06T07:20:22.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.592+0000 7f603effd640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602c06b4d0 0x7f602c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:22.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.593+0000 7f603effd640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602c06b4d0 0x7f602c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6034036000 tx=0x7f603402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:22.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.596+0000 7f6027fff640 1 -- 172.21.15.94:0/3982585450 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6030059140 con 0x7f6028004800 2024-03-06T07:20:22.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.807+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60400c8d50 con 0x7f6028004800 2024-03-06T07:20:22.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.807+0000 7f6027fff640 1 -- 172.21.15.94:0/3982585450 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6260 (secure 0 0 0) 0x7f6030058a40 con 0x7f6028004800 2024-03-06T07:20:22.809 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:22.810 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:20:22.636873+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/279249774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":279249774},{"type":"v1","addr":"172.21.15.145:6837","nonce":279249774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:22.810 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 41 2024-03-06T07:20:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.809+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602c06b4d0 msgr2=0x7f602c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.809+0000 7f6045f25640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602c06b4d0 0x7f602c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6034036000 tx=0x7f603402e000 comp rx=0 tx=0).stop 2024-03-06T07:20:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.809+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 msgr2=0x7f60401ee770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.809+0000 7f6045f25640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f60401ee770 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f603000fb40 tx=0x7f6030001cf0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 shutdown_connections 2024-03-06T07:20:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602c06b4d0 0x7f602c06d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60400c1050 0x7f6040148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f604014c350 0x7f6040147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 --2- 172.21.15.94:0/3982585450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6028004800 0x7f60401ee770 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:22.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 >> 172.21.15.94:0/3982585450 conn(0x7f60400ac570 msgr2=0x7f604014e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:22.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 shutdown_connections 2024-03-06T07:20:22.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:22.810+0000 7f6045f25640 1 -- 172.21.15.94:0/3982585450 wait complete. 2024-03-06T07:20:22.823 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:20:22.823 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:20:22.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:20:22.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:20:22.909 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:20:22.909 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:20:22.910 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:22.926 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:20:22.926 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:20:22.926 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:22.965 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:20:22.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.064+0000 7fd388e8d640 1 Processor -- start 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.065+0000 7fd388e8d640 1 -- start start 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd388e8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd384155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd388e8d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd384158b60 0x7fd384158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd388e8d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3840c73d0 con 0x7fd38414f360 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd388e8d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3840c8dd0 con 0x7fd384158b60 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd388e8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3840be2e0 con 0x7fd3840c88c0 2024-03-06T07:20:23.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd382d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd384158b60 0x7fd384158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd382d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd384158b60 0x7fd384158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56146/0 (socket says 172.21.15.94:56146) 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd382d9d640 1 -- 172.21.15.94:0/20051177 learned_addr learned my addr 172.21.15.94:0/20051177 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd38259c640 1 --1- 172.21.15.94:0/20051177 >> v1:172.21.15.94:6789/0 conn(0x7fd38414f360 0x7fd3840c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35888/0 (socket says 172.21.15.94:35888) 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd381d9b640 1 --2- 172.21.15.94:0/20051177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd384155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd382d9d640 1 -- 172.21.15.94:0/20051177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 msgr2=0x7fd384155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd382d9d640 1 --2- 172.21.15.94:0/20051177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd384155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd382d9d640 1 -- 172.21.15.94:0/20051177 >> v1:172.21.15.94:6789/0 conn(0x7fd38414f360 legacy=0x7fd3840c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd382d9d640 1 -- 172.21.15.94:0/20051177 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3840c7670 con 0x7fd384158b60 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.066+0000 7fd38159a640 1 -- 172.21.15.94:0/20051177 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731638089 0 0) 0x7fd3840c73d0 con 0x7fd38414f360 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.067+0000 7fd382d9d640 1 --2- 172.21.15.94:0/20051177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd384158b60 0x7fd384158ff0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd374009870 tx=0x7fd37402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf13fcf5165066a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.067+0000 7fd38159a640 1 -- 172.21.15.94:0/20051177 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd374035020 con 0x7fd384158b60 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.067+0000 7fd38159a640 1 -- 172.21.15.94:0/20051177 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd374031640 con 0x7fd384158b60 2024-03-06T07:20:23.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.067+0000 7fd38159a640 1 -- 172.21.15.94:0/20051177 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37402d040 con 0x7fd384158b60 2024-03-06T07:20:23.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 -- 172.21.15.94:0/20051177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd384158b60 msgr2=0x7fd384158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 --2- 172.21.15.94:0/20051177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd384158b60 0x7fd384158ff0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd374009870 tx=0x7fd37402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 -- 172.21.15.94:0/20051177 shutdown_connections 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 --2- 172.21.15.94:0/20051177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd384158b60 0x7fd384158ff0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 --2- 172.21.15.94:0/20051177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd384155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 -- 172.21.15.94:0/20051177 >> 172.21.15.94:0/20051177 conn(0x7fd3840ac570 msgr2=0x7fd3840ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 -- 172.21.15.94:0/20051177 shutdown_connections 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.068+0000 7fd388e8d640 1 -- 172.21.15.94:0/20051177 wait complete. 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.069+0000 7fd388e8d640 1 Processor -- start 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.069+0000 7fd388e8d640 1 -- start start 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd388e8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd3841540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd388e8d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd38414f360 0x7fd3841edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd388e8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd384158b60 0x7fd3840ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd388e8d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3840c5a50 con 0x7fd38414f360 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd388e8d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3840c7c30 con 0x7fd384158b60 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd388e8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3840c5bd0 con 0x7fd3840c88c0 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd38414f360 0x7fd3841edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd38414f360 0x7fd3841edbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56162/0 (socket says 172.21.15.94:56162) 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 -- 172.21.15.94:0/1028644949 learned_addr learned my addr 172.21.15.94:0/1028644949 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd38259c640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd3841540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd382d9d640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd384158b60 0x7fd3840ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 -- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 msgr2=0x7fd3841540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd3841540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 -- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd384158b60 msgr2=0x7fd3840ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd384158b60 0x7fd3840ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 -- 172.21.15.94:0/1028644949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd384154630 con 0x7fd38414f360 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.070+0000 7fd381d9b640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd38414f360 0x7fd3841edbe0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd37800e870 tx=0x7fd37800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.071+0000 7fd372ffd640 1 -- 172.21.15.94:0/1028644949 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378017070 con 0x7fd38414f360 2024-03-06T07:20:23.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.071+0000 7fd372ffd640 1 -- 172.21.15.94:0/1028644949 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37800c940 con 0x7fd38414f360 2024-03-06T07:20:23.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.071+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3841eb390 con 0x7fd38414f360 2024-03-06T07:20:23.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.071+0000 7fd372ffd640 1 -- 172.21.15.94:0/1028644949 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37800cd10 con 0x7fd38414f360 2024-03-06T07:20:23.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.071+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3840aeb00 con 0x7fd38414f360 2024-03-06T07:20:23.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.072+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38404f330 con 0x7fd38414f360 2024-03-06T07:20:23.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.072+0000 7fd372ffd640 1 -- 172.21.15.94:0/1028644949 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd37801a070 con 0x7fd38414f360 2024-03-06T07:20:23.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.072+0000 7fd372ffd640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd36406b4d0 0x7fd36406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.073+0000 7fd372ffd640 1 -- 172.21.15.94:0/1028644949 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fd37808a860 con 0x7fd38414f360 2024-03-06T07:20:23.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.073+0000 7fd38259c640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd36406b4d0 0x7fd36406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.073+0000 7fd38259c640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd36406b4d0 0x7fd36406d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0012e0 tx=0x7fd37c001700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.077+0000 7fd372ffd640 1 -- 172.21.15.94:0/1028644949 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd378058b90 con 0x7fd38414f360 2024-03-06T07:20:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.291+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd38406cda0 con 0x7fd38414f360 2024-03-06T07:20:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.291+0000 7fd372ffd640 1 -- 172.21.15.94:0/1028644949 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6260 (secure 0 0 0) 0x7fd37805cc30 con 0x7fd38414f360 2024-03-06T07:20:23.295 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:23.295 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:20:22.636873+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/279249774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":279249774},{"type":"v1","addr":"172.21.15.145:6837","nonce":279249774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:23.296 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 41 2024-03-06T07:20:23.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd36406b4d0 msgr2=0x7fd36406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd36406b4d0 0x7fd36406d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0012e0 tx=0x7fd37c001700 comp rx=0 tx=0).stop 2024-03-06T07:20:23.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd38414f360 msgr2=0x7fd3841edbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd38414f360 0x7fd3841edbe0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd37800e870 tx=0x7fd37800ed30 comp rx=0 tx=0).stop 2024-03-06T07:20:23.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 shutdown_connections 2024-03-06T07:20:23.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd36406b4d0 0x7fd36406d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd384158b60 0x7fd3840ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd38414f360 0x7fd3841edbe0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 --2- 172.21.15.94:0/1028644949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3840c88c0 0x7fd3841540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 >> 172.21.15.94:0/1028644949 conn(0x7fd3840ac570 msgr2=0x7fd3840ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:23.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 shutdown_connections 2024-03-06T07:20:23.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.294+0000 7fd388e8d640 1 -- 172.21.15.94:0/1028644949 wait complete. 2024-03-06T07:20:23.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:23.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.409+0000 7fe83a9a0640 1 Processor -- start 2024-03-06T07:20:23.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.409+0000 7fe83a9a0640 1 -- start start 2024-03-06T07:20:23.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a9a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe834159150 0x7fe8341559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a9a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83414be70 con 0x7fe83414ba90 2024-03-06T07:20:23.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a9a0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8340cad30 con 0x7fe8340c8690 2024-03-06T07:20:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a9a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340cb010 con 0x7fe834159150 2024-03-06T07:20:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83999e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe83414ba90 0x7fe8340c5110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35894/0 (socket says 172.21.15.94:35894) 2024-03-06T07:20:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83999e640 1 -- 172.21.15.94:0/1819341930 learned_addr learned my addr 172.21.15.94:0/1819341930 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a19f640 1 --2- 172.21.15.94:0/1819341930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe834159150 0x7fe8341559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83899c640 1 -- 172.21.15.94:0/1819341930 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569843764 0 0) 0x7fe82c001110 con 0x7fe83414ba90 2024-03-06T07:20:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83899c640 1 -- 172.21.15.94:0/1819341930 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe81c003520 con 0x7fe83414ba90 2024-03-06T07:20:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83899c640 1 -- 172.21.15.94:0/1819341930 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142453030 0 0) 0x7fe8340cad30 con 0x7fe8340c8690 2024-03-06T07:20:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a19f640 1 -- 172.21.15.94:0/1819341930 >> v1:172.21.15.145:6789/0 conn(0x7fe8340c8690 legacy=0x7fe834158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a19f640 1 -- 172.21.15.94:0/1819341930 >> v1:172.21.15.94:6789/0 conn(0x7fe83414ba90 legacy=0x7fe8340c5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.410+0000 7fe83a19f640 1 -- 172.21.15.94:0/1819341930 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8340cb2f0 con 0x7fe834159150 2024-03-06T07:20:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.411+0000 7fe83a19f640 1 --2- 172.21.15.94:0/1819341930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe834159150 0x7fe8341559b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe824009dc0 tx=0x7fe82402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=784bb056811f0024 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.411+0000 7fe83899c640 1 -- 172.21.15.94:0/1819341930 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824039070 con 0x7fe834159150 2024-03-06T07:20:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.411+0000 7fe83899c640 1 -- 172.21.15.94:0/1819341930 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82402ccd0 con 0x7fe834159150 2024-03-06T07:20:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.411+0000 7fe83899c640 1 -- 172.21.15.94:0/1819341930 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824034660 con 0x7fe834159150 2024-03-06T07:20:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.412+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1819341930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe834159150 msgr2=0x7fe8341559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.412+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1819341930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe834159150 0x7fe8341559b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe824009dc0 tx=0x7fe82402bf20 comp rx=0 tx=0).stop 2024-03-06T07:20:23.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.412+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1819341930 shutdown_connections 2024-03-06T07:20:23.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.412+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1819341930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe834159150 0x7fe8341559b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.412+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1819341930 >> 172.21.15.94:0/1819341930 conn(0x7fe8340728d0 msgr2=0x7fe834072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:23.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.412+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1819341930 shutdown_connections 2024-03-06T07:20:23.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.412+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1819341930 wait complete. 2024-03-06T07:20:23.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.413+0000 7fe83a9a0640 1 Processor -- start 2024-03-06T07:20:23.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.413+0000 7fe83a9a0640 1 -- start start 2024-03-06T07:20:23.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8340c8690 0x7fe834144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83414ba90 0x7fe8341492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe834159150 0x7fe8340c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340c5820 con 0x7fe834159150 2024-03-06T07:20:23.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83406cf60 con 0x7fe83414ba90 2024-03-06T07:20:23.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340c59a0 con 0x7fe8340c8690 2024-03-06T07:20:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe834159150 0x7fe8340c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83999e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8340c8690 0x7fe834144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe834159150 0x7fe8340c3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56176/0 (socket says 172.21.15.94:56176) 2024-03-06T07:20:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 -- 172.21.15.94:0/1335969073 learned_addr learned my addr 172.21.15.94:0/1335969073 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83919d640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83414ba90 0x7fe8341492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 -- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8340c8690 msgr2=0x7fe834144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8340c8690 0x7fe834144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 -- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83414ba90 msgr2=0x7fe8341492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83414ba90 0x7fe8341492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 -- 172.21.15.94:0/1335969073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8341453b0 con 0x7fe834159150 2024-03-06T07:20:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83919d640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83414ba90 0x7fe8341492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a19f640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe834159150 0x7fe8340c3d70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe824009ee0 tx=0x7fe824004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe8327fc640 1 -- 172.21.15.94:0/1335969073 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824039050 con 0x7fe834159150 2024-03-06T07:20:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe8327fc640 1 -- 172.21.15.94:0/1335969073 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824045070 con 0x7fe834159150 2024-03-06T07:20:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83414b0e0 con 0x7fe834159150 2024-03-06T07:20:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe8327fc640 1 -- 172.21.15.94:0/1335969073 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824009450 con 0x7fe834159150 2024-03-06T07:20:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.414+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe834144c70 con 0x7fe834159150 2024-03-06T07:20:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.415+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe834069430 con 0x7fe834159150 2024-03-06T07:20:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.416+0000 7fe8327fc640 1 -- 172.21.15.94:0/1335969073 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe82403e850 con 0x7fe834159150 2024-03-06T07:20:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.416+0000 7fe8327fc640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe82006b4d0 0x7fe82006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.417+0000 7fe8327fc640 1 -- 172.21.15.94:0/1335969073 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fe8240ad4a0 con 0x7fe834159150 2024-03-06T07:20:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.417+0000 7fe83999e640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe82006b4d0 0x7fe82006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.417+0000 7fe83999e640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe82006b4d0 0x7fe82006d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005880 tx=0x7fe82c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.421+0000 7fe8327fc640 1 -- 172.21.15.94:0/1335969073 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8341453b0 con 0x7fe834159150 2024-03-06T07:20:23.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.600+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe83406cd40 con 0x7fe834159150 2024-03-06T07:20:23.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.601+0000 7fe8327fc640 1 -- 172.21.15.94:0/1335969073 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+20925 (secure 0 0 0) 0x7fe82407b8f0 con 0x7fe834159150 2024-03-06T07:20:23.603 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:23.603 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":58,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:19.628932+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.94:6833/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.145:6837/3125091562":"2024-03-07T07:20:16.601064+0000","172.21.15.145:6835/3125091562":"2024-03-07T07:20:16.601064+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:23.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe82006b4d0 msgr2=0x7fe82006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe82006b4d0 0x7fe82006d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005880 tx=0x7fe82c009040 comp rx=0 tx=0).stop 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe834159150 msgr2=0x7fe8340c3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe834159150 0x7fe8340c3d70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe824009ee0 tx=0x7fe824004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 shutdown_connections 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe82006b4d0 0x7fe82006d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe834159150 0x7fe8340c3d70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83414ba90 0x7fe8341492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 --2- 172.21.15.94:0/1335969073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8340c8690 0x7fe834144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 >> 172.21.15.94:0/1335969073 conn(0x7fe8340728d0 msgr2=0x7fe834154820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 shutdown_connections 2024-03-06T07:20:23.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.603+0000 7fe83a9a0640 1 -- 172.21.15.94:0/1335969073 wait complete. 2024-03-06T07:20:23.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:19:51.596832+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:19:52.352004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:20:06.055145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:20:06.580638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:20:23.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:23.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.820+0000 7fb6a9ac9640 1 Processor -- start 2024-03-06T07:20:23.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.820+0000 7fb6a9ac9640 1 -- start start 2024-03-06T07:20:23.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.821+0000 7fb6a9ac9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a414ef70 0x7fb6a406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.821+0000 7fb6a9ac9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a40c2790 con 0x7fb6a406cad0 2024-03-06T07:20:23.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.821+0000 7fb6a9ac9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a40c1540 con 0x7fb6a40c1820 2024-03-06T07:20:23.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.821+0000 7fb6a9ac9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a40c1ca0 con 0x7fb6a414ef70 2024-03-06T07:20:23.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.821+0000 7fb6a3fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb6a406cad0 0x7fb6a4154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35902/0 (socket says 172.21.15.94:35902) 2024-03-06T07:20:23.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.821+0000 7fb6a3fff640 1 -- 172.21.15.94:0/2657952486 learned_addr learned my addr 172.21.15.94:0/2657952486 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:23.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.821+0000 7fb6a37fe640 1 --2- 172.21.15.94:0/2657952486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a414ef70 0x7fb6a406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.822+0000 7fb6a27fc640 1 -- 172.21.15.94:0/2657952486 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365685940 0 0) 0x7fb6a40c2790 con 0x7fb6a406cad0 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.822+0000 7fb6a27fc640 1 -- 172.21.15.94:0/2657952486 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68c003440 con 0x7fb6a406cad0 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.822+0000 7fb6a27fc640 1 -- 172.21.15.94:0/2657952486 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7595095 0 0) 0x7fb6a40c1540 con 0x7fb6a40c1820 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.822+0000 7fb6a37fe640 1 -- 172.21.15.94:0/2657952486 >> v1:172.21.15.145:6789/0 conn(0x7fb6a40c1820 legacy=0x7fb6a40c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.822+0000 7fb6a37fe640 1 -- 172.21.15.94:0/2657952486 >> v1:172.21.15.94:6789/0 conn(0x7fb6a406cad0 legacy=0x7fb6a4154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.822+0000 7fb6a37fe640 1 -- 172.21.15.94:0/2657952486 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a40c24b0 con 0x7fb6a414ef70 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.822+0000 7fb6a37fe640 1 --2- 172.21.15.94:0/2657952486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a414ef70 0x7fb6a406c1f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb6940097d0 tx=0x7fb69402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a70a173bf2d123e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a27fc640 1 -- 172.21.15.94:0/2657952486 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694035020 con 0x7fb6a414ef70 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a27fc640 1 -- 172.21.15.94:0/2657952486 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb694031800 con 0x7fb6a414ef70 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a27fc640 1 -- 172.21.15.94:0/2657952486 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694031bd0 con 0x7fb6a414ef70 2024-03-06T07:20:23.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/2657952486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a414ef70 msgr2=0x7fb6a406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/2657952486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a414ef70 0x7fb6a406c1f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb6940097d0 tx=0x7fb69402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/2657952486 shutdown_connections 2024-03-06T07:20:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/2657952486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a414ef70 0x7fb6a406c1f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/2657952486 >> 172.21.15.94:0/2657952486 conn(0x7fb6a40c6710 msgr2=0x7fb6a40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/2657952486 shutdown_connections 2024-03-06T07:20:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.823+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/2657952486 wait complete. 2024-03-06T07:20:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.824+0000 7fb6a9ac9640 1 Processor -- start 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.824+0000 7fb6a9ac9640 1 -- start start 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a9ac9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb6a406cad0 0x7fb6a4149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a9ac9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a40c1820 0x7fb6a414c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a9ac9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6a414ef70 0x7fb6a414cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a9ac9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a414f350 con 0x7fb6a414ef70 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a9ac9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a40c3090 con 0x7fb6a406cad0 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a9ac9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a406cf60 con 0x7fb6a40c1820 2024-03-06T07:20:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6a414ef70 0x7fb6a414cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6a414ef70 0x7fb6a414cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56184/0 (socket says 172.21.15.94:56184) 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a3fff640 1 -- 172.21.15.94:0/3308624313 learned_addr learned my addr 172.21.15.94:0/3308624313 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a2ffd640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a40c1820 0x7fb6a414c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.825+0000 7fb6a37fe640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb6a406cad0 0x7fb6a4149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a2ffd640 1 -- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb6a406cad0 msgr2=0x7fb6a4149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a2ffd640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb6a406cad0 0x7fb6a4149150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a2ffd640 1 -- 172.21.15.94:0/3308624313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6a414ef70 msgr2=0x7fb6a414cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a2ffd640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6a414ef70 0x7fb6a414cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a2ffd640 1 -- 172.21.15.94:0/3308624313 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a40a6930 con 0x7fb6a40c1820 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a3fff640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6a414ef70 0x7fb6a414cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a2ffd640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a40c1820 0x7fb6a414c640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb69800c480 tx=0x7fb69800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb68bfff640 1 -- 172.21.15.94:0/3308624313 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698017070 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb68bfff640 1 -- 172.21.15.94:0/3308624313 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb698005480 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a41ea020 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb68bfff640 1 -- 172.21.15.94:0/3308624313 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698003830 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.826+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a41ea5e0 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.827+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a404f330 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.828+0000 7fb68bfff640 1 -- 172.21.15.94:0/3308624313 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb69801a070 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.828+0000 7fb68bfff640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb69006b4d0 0x7fb69006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.828+0000 7fb68bfff640 1 -- 172.21.15.94:0/3308624313 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fb6980899a0 con 0x7fb6a40c1820 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.828+0000 7fb6a37fe640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb69006b4d0 0x7fb69006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:23.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.829+0000 7fb6a37fe640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb69006b4d0 0x7fb69006d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb694036000 tx=0x7fb69403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:23.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:23.833+0000 7fb68bfff640 1 -- 172.21.15.94:0/3308624313 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb698057cd0 con 0x7fb6a40c1820 2024-03-06T07:20:24.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.005+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6a40c3f90 con 0x7fb6a40c1820 2024-03-06T07:20:24.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.005+0000 7fb68bfff640 1 -- 172.21.15.94:0/3308624313 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb69805bd70 con 0x7fb6a40c1820 2024-03-06T07:20:24.007 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb69006b4d0 msgr2=0x7fb69006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb69006b4d0 0x7fb69006d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb694036000 tx=0x7fb69403c040 comp rx=0 tx=0).stop 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a40c1820 msgr2=0x7fb6a414c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a40c1820 0x7fb6a414c640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb69800c480 tx=0x7fb69800c940 comp rx=0 tx=0).stop 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 shutdown_connections 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb69006b4d0 0x7fb69006d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb6a414ef70 0x7fb6a414cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb6a40c1820 0x7fb6a414c640 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 --2- 172.21.15.94:0/3308624313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb6a406cad0 0x7fb6a4149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 >> 172.21.15.94:0/3308624313 conn(0x7fb6a40c6710 msgr2=0x7fb6a414a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 shutdown_connections 2024-03-06T07:20:24.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.008+0000 7fb6a9ac9640 1 -- 172.21.15.94:0/3308624313 wait complete. 2024-03-06T07:20:24.020 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:24.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.122+0000 7f44d2f7a640 1 Processor -- start 2024-03-06T07:20:24.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.122+0000 7f44d2f7a640 1 -- start start 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.122+0000 7f44d2f7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0a8290 0x7f44cc1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d2f7a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f44cc0c4740 0x7f44cc0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d2f7a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44cc0c62a0 con 0x7f44cc0c69a0 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d2f7a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44cc153f00 con 0x7f44cc0c4740 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d2f7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc0706e0 con 0x7f44cc0a8290 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d0d16640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0a8290 0x7f44cc1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44cbfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f44cc0c4740 0x7f44cc0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d0d16640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0a8290 0x7f44cc1539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:40248/0 (socket says 172.21.15.94:40248) 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d0d16640 1 -- 172.21.15.94:0/3565763222 learned_addr learned my addr 172.21.15.94:0/3565763222 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d0d16640 1 -- 172.21.15.94:0/3565763222 >> v2:172.21.15.145:3300/0 conn(0x7f44cc0c4740 msgr2=0x7f44cc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d0d16640 1 --2- 172.21.15.94:0/3565763222 >> v2:172.21.15.145:3300/0 conn(0x7f44cc0c4740 0x7f44cc0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d0d16640 1 -- 172.21.15.94:0/3565763222 >> v1:172.21.15.145:6789/0 conn(0x7f44cc0c69a0 legacy=0x7f44cc072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.123+0000 7f44d0d16640 1 -- 172.21.15.94:0/3565763222 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc0c4c00 con 0x7f44cc0a8290 2024-03-06T07:20:24.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.124+0000 7f44d0d16640 1 --2- 172.21.15.94:0/3565763222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0a8290 0x7f44cc1539c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f44c0009870 tx=0x7f44c002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f1c6252648701b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.124+0000 7f44cb7fe640 1 -- 172.21.15.94:0/3565763222 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c00317c0 con 0x7f44cc0a8290 2024-03-06T07:20:24.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.124+0000 7f44cb7fe640 1 -- 172.21.15.94:0/3565763222 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c0037070 con 0x7f44cc0a8290 2024-03-06T07:20:24.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.124+0000 7f44cb7fe640 1 -- 172.21.15.94:0/3565763222 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0031b80 con 0x7f44cc0a8290 2024-03-06T07:20:24.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.124+0000 7f44d2f7a640 1 -- 172.21.15.94:0/3565763222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0a8290 msgr2=0x7f44cc1539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.124+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/3565763222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0a8290 0x7f44cc1539c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f44c0009870 tx=0x7f44c002b860 comp rx=0 tx=0).stop 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.125+0000 7f44d2f7a640 1 -- 172.21.15.94:0/3565763222 shutdown_connections 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.125+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/3565763222 >> v2:172.21.15.145:3300/0 conn(0x7f44cc0c4740 0x7f44cc0701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.125+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/3565763222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0a8290 0x7f44cc1539c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.125+0000 7f44d2f7a640 1 -- 172.21.15.94:0/3565763222 >> 172.21.15.94:0/3565763222 conn(0x7f44cc0c5560 msgr2=0x7f44cc0c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.125+0000 7f44d2f7a640 1 -- 172.21.15.94:0/3565763222 shutdown_connections 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.125+0000 7f44d2f7a640 1 -- 172.21.15.94:0/3565763222 wait complete. 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.126+0000 7f44d2f7a640 1 Processor -- start 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.126+0000 7f44d2f7a640 1 -- start start 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.126+0000 7f44d2f7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44cc0a8290 0x7f44cc0adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d2f7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0c4740 0x7f44cc0bda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d2f7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44cc0c69a0 0x7f44cc1dcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d2f7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc06fc80 con 0x7f44cc0a8290 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d2f7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc06f9a0 con 0x7f44cc0c69a0 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d2f7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc072d90 con 0x7f44cc0c4740 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44cc0a8290 0x7f44cc0adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44cc0a8290 0x7f44cc0adce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56194/0 (socket says 172.21.15.94:56194) 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 -- 172.21.15.94:0/302418620 learned_addr learned my addr 172.21.15.94:0/302418620 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44cbfff640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0c4740 0x7f44cc0bda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d1517640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44cc0c69a0 0x7f44cc1dcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 -- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0c4740 msgr2=0x7f44cc0bda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0c4740 0x7f44cc0bda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 -- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44cc0c69a0 msgr2=0x7f44cc1dcaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44cc0c69a0 0x7f44cc1dcaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 -- 172.21.15.94:0/302418620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc1de520 con 0x7f44cc0a8290 2024-03-06T07:20:24.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d0d16640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44cc0a8290 0x7f44cc0adce0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f44c0000c00 tx=0x7f44c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44c97fa640 1 -- 172.21.15.94:0/302418620 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0032650 con 0x7f44cc0a8290 2024-03-06T07:20:24.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44c97fa640 1 -- 172.21.15.94:0/302418620 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c0032810 con 0x7f44cc0a8290 2024-03-06T07:20:24.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44c97fa640 1 -- 172.21.15.94:0/302418620 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0032c00 con 0x7f44cc0a8290 2024-03-06T07:20:24.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44cc1ddc80 con 0x7f44cc0a8290 2024-03-06T07:20:24.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.127+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44cc1de240 con 0x7f44cc0a8290 2024-03-06T07:20:24.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.128+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44cc04f330 con 0x7f44cc0a8290 2024-03-06T07:20:24.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.129+0000 7f44c97fa640 1 -- 172.21.15.94:0/302418620 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44c004c060 con 0x7f44cc0a8290 2024-03-06T07:20:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.129+0000 7f44c97fa640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b006b4d0 0x7f44b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.129+0000 7f44c97fa640 1 -- 172.21.15.94:0/302418620 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f44c00ac1b0 con 0x7f44cc0a8290 2024-03-06T07:20:24.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.129+0000 7f44cbfff640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b006b4d0 0x7f44b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.130+0000 7f44cbfff640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b006b4d0 0x7f44b006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f44cc1ed6a0 tx=0x7f44b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.137+0000 7f44c97fa640 1 -- 172.21.15.94:0/302418620 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44c0040070 con 0x7f44cc0a8290 2024-03-06T07:20:24.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.319+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44cc074470 con 0x7f44cc0a8290 2024-03-06T07:20:24.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.319+0000 7f44c97fa640 1 -- 172.21.15.94:0/302418620 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f44c007a450 con 0x7f44cc0a8290 2024-03-06T07:20:24.321 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:20:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b006b4d0 msgr2=0x7f44b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b006b4d0 0x7f44b006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f44cc1ed6a0 tx=0x7f44b8009190 comp rx=0 tx=0).stop 2024-03-06T07:20:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44cc0a8290 msgr2=0x7f44cc0adce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44cc0a8290 0x7f44cc0adce0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f44c0000c00 tx=0x7f44c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 shutdown_connections 2024-03-06T07:20:24.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44b006b4d0 0x7f44b006d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44cc0c69a0 0x7f44cc1dcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44cc0c4740 0x7f44cc0bda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 --2- 172.21.15.94:0/302418620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44cc0a8290 0x7f44cc0adce0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 >> 172.21.15.94:0/302418620 conn(0x7f44cc0c5560 msgr2=0x7f44cc1562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 shutdown_connections 2024-03-06T07:20:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.322+0000 7f44d2f7a640 1 -- 172.21.15.94:0/302418620 wait complete. 2024-03-06T07:20:24.335 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:24.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.436+0000 7f5f24fd7640 1 Processor -- start 2024-03-06T07:20:24.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.436+0000 7f5f24fd7640 1 -- start start 2024-03-06T07:20:24.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.437+0000 7f5f24fd7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5f200b0070 0x7f5f20155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f24fd7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5f200c5380 0x7f5f200c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f24fd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f200b0450 con 0x7f5f2006ca70 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f24fd7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f2006c020 con 0x7f5f200b0070 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f24fd7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f2006cf20 con 0x7f5f200c5380 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5f200b0070 0x7f5f20155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5f200b0070 0x7f5f20155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56210/0 (socket says 172.21.15.94:56210) 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1ed9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5f200c5380 0x7f5f200c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1554822655 learned_addr learned my addr 172.21.15.94:0/1554822655 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f17fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5f2006ca70 0x7f5f200c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52876/0 (socket says 172.21.15.94:52876) 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1554822655 >> v2:172.21.15.145:3300/0 conn(0x7f5f200c5380 msgr2=0x7f5f200c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1e59c640 1 --2- 172.21.15.94:0/1554822655 >> v2:172.21.15.145:3300/0 conn(0x7f5f200c5380 0x7f5f200c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1554822655 >> v1:172.21.15.145:6789/0 conn(0x7f5f2006ca70 legacy=0x7f5f200c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1554822655 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f200c77f0 con 0x7f5f200b0070 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.438+0000 7f5f1dd9b640 1 -- 172.21.15.94:0/1554822655 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586536955 0 0) 0x7f5f08001110 con 0x7f5f2006ca70 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f1e59c640 1 --2- 172.21.15.94:0/1554822655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f200b0070 0x7f5f20155c30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5f200be030 tx=0x7f5f1002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7afe19bf972f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f1dd9b640 1 -- 172.21.15.94:0/1554822655 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10035020 con 0x7f5f200b0070 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f1dd9b640 1 -- 172.21.15.94:0/1554822655 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f10031870 con 0x7f5f200b0070 2024-03-06T07:20:24.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f1dd9b640 1 -- 172.21.15.94:0/1554822655 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10031c40 con 0x7f5f200b0070 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1554822655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f200b0070 msgr2=0x7f5f20155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1554822655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f200b0070 0x7f5f20155c30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5f200be030 tx=0x7f5f1002b910 comp rx=0 tx=0).stop 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1554822655 shutdown_connections 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1554822655 >> v2:172.21.15.145:3300/0 conn(0x7f5f200c5380 0x7f5f200c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1554822655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f200b0070 0x7f5f20155c30 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1554822655 >> 172.21.15.94:0/1554822655 conn(0x7f5f200ac590 msgr2=0x7f5f200ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1554822655 shutdown_connections 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.439+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1554822655 wait complete. 2024-03-06T07:20:24.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 Processor -- start 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 -- start start 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f2006ca70 0x7f5f200c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f200b0070 0x7f5f20154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f200c5380 0x7f5f2014a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2006c550 con 0x7f5f2006ca70 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2006c270 con 0x7f5f200c5380 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f24fd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2006da80 con 0x7f5f200b0070 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f1e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f2006ca70 0x7f5f200c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f1e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f2006ca70 0x7f5f200c23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56216/0 (socket says 172.21.15.94:56216) 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.441+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1880666210 learned_addr learned my addr 172.21.15.94:0/1880666210 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f1ed9d640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f200c5380 0x7f5f2014a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f17fff640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f200b0070 0x7f5f20154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f200b0070 msgr2=0x7f5f20154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f1e59c640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f200b0070 0x7f5f20154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f200c5380 msgr2=0x7f5f2014a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f1e59c640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f200c5380 0x7f5f2014a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f1e59c640 1 -- 172.21.15.94:0/1880666210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f20154c60 con 0x7f5f2006ca70 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f1e59c640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f2006ca70 0x7f5f200c23c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5f10000c00 tx=0x7f5f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f16ffd640 1 -- 172.21.15.94:0/1880666210 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10041070 con 0x7f5f2006ca70 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f16ffd640 1 -- 172.21.15.94:0/1880666210 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1003ad00 con 0x7f5f2006ca70 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f201eedf0 con 0x7f5f2006ca70 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f16ffd640 1 -- 172.21.15.94:0/1880666210 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10035040 con 0x7f5f2006ca70 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.442+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f201ee310 con 0x7f5f2006ca70 2024-03-06T07:20:24.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.443+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2004f330 con 0x7f5f2006ca70 2024-03-06T07:20:24.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.444+0000 7f5f16ffd640 1 -- 172.21.15.94:0/1880666210 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f10038020 con 0x7f5f2006ca70 2024-03-06T07:20:24.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.444+0000 7f5f16ffd640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5f0006b4d0 0x7f5f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.444+0000 7f5f16ffd640 1 -- 172.21.15.94:0/1880666210 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f5f100ac400 con 0x7f5f2006ca70 2024-03-06T07:20:24.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.444+0000 7f5f17fff640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5f0006b4d0 0x7f5f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.445+0000 7f5f17fff640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5f0006b4d0 0x7f5f0006d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f200c3cf0 tx=0x7f5f08007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.448+0000 7f5f16ffd640 1 -- 172.21.15.94:0/1880666210 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f1007a730 con 0x7f5f2006ca70 2024-03-06T07:20:24.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.625+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f2006d860 con 0x7f5f2006ca70 2024-03-06T07:20:24.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.625+0000 7f5f16ffd640 1 -- 172.21.15.94:0/1880666210 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5f1007e7d0 con 0x7f5f2006ca70 2024-03-06T07:20:24.627 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:20:24.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5f0006b4d0 msgr2=0x7f5f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5f0006b4d0 0x7f5f0006d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f200c3cf0 tx=0x7f5f08007de0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f2006ca70 msgr2=0x7f5f200c23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f2006ca70 0x7f5f200c23c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5f10000c00 tx=0x7f5f100046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 shutdown_connections 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5f0006b4d0 0x7f5f0006d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f200c5380 0x7f5f2014a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f200b0070 0x7f5f20154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 --2- 172.21.15.94:0/1880666210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5f2006ca70 0x7f5f200c23c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 >> 172.21.15.94:0/1880666210 conn(0x7f5f200ac590 msgr2=0x7f5f200ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 shutdown_connections 2024-03-06T07:20:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.627+0000 7f5f24fd7640 1 -- 172.21.15.94:0/1880666210 wait complete. 2024-03-06T07:20:24.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:20:24.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.743+0000 7f3844a84640 1 Processor -- start 2024-03-06T07:20:24.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.743+0000 7f3844a84640 1 -- start start 2024-03-06T07:20:24.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f3844a84640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f384006ca70 0x7f384014e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f3844a84640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400b0450 con 0x7f38400b0070 2024-03-06T07:20:24.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f3844a84640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38401560e0 con 0x7f3840155c30 2024-03-06T07:20:24.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f3844a84640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384006c020 con 0x7f384006ca70 2024-03-06T07:20:24.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f384006ca70 0x7f384014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383e59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f38400b0070 0x7f38400c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35910/0 (socket says 172.21.15.94:35910) 2024-03-06T07:20:24.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f384006ca70 0x7f384014e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56220/0 (socket says 172.21.15.94:56220) 2024-03-06T07:20:24.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383e59c640 1 -- 172.21.15.94:0/3985817636 learned_addr learned my addr 172.21.15.94:0/3985817636 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:24.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383ed9d640 1 -- 172.21.15.94:0/3985817636 >> v1:172.21.15.145:6789/0 conn(0x7f3840155c30 legacy=0x7f38400c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383ed9d640 1 -- 172.21.15.94:0/3985817636 >> v1:172.21.15.94:6789/0 conn(0x7f38400b0070 legacy=0x7f38400c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383d59a640 1 -- 172.21.15.94:0/3985817636 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048263867 0 0) 0x7f38401560e0 con 0x7f3840155c30 2024-03-06T07:20:24.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.744+0000 7f383ed9d640 1 -- 172.21.15.94:0/3985817636 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400c77f0 con 0x7f384006ca70 2024-03-06T07:20:24.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.745+0000 7f383ed9d640 1 --2- 172.21.15.94:0/3985817636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384006ca70 0x7f384014e800 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f38300097a0 tx=0x7f383002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0be9e4202193dbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.745+0000 7f383d59a640 1 -- 172.21.15.94:0/3985817636 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830037070 con 0x7f384006ca70 2024-03-06T07:20:24.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.745+0000 7f383d59a640 1 -- 172.21.15.94:0/3985817636 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3830030710 con 0x7f384006ca70 2024-03-06T07:20:24.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.745+0000 7f383d59a640 1 -- 172.21.15.94:0/3985817636 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830030ae0 con 0x7f384006ca70 2024-03-06T07:20:24.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.746+0000 7f3844a84640 1 -- 172.21.15.94:0/3985817636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384006ca70 msgr2=0x7f384014e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.746+0000 7f3844a84640 1 --2- 172.21.15.94:0/3985817636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384006ca70 0x7f384014e800 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f38300097a0 tx=0x7f383002b710 comp rx=0 tx=0).stop 2024-03-06T07:20:24.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.746+0000 7f3844a84640 1 -- 172.21.15.94:0/3985817636 shutdown_connections 2024-03-06T07:20:24.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.746+0000 7f3844a84640 1 --2- 172.21.15.94:0/3985817636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f384006ca70 0x7f384014e800 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.746+0000 7f3844a84640 1 -- 172.21.15.94:0/3985817636 >> 172.21.15.94:0/3985817636 conn(0x7f38400ac590 msgr2=0x7f38400ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.746+0000 7f3844a84640 1 -- 172.21.15.94:0/3985817636 shutdown_connections 2024-03-06T07:20:24.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.746+0000 7f3844a84640 1 -- 172.21.15.94:0/3985817636 wait complete. 2024-03-06T07:20:24.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.747+0000 7f3844a84640 1 Processor -- start 2024-03-06T07:20:24.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.747+0000 7f3844a84640 1 -- start start 2024-03-06T07:20:24.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f3844a84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f384006ca70 0x7f38401eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f3844a84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38400b0070 0x7f3840154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f3844a84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3840155c30 0x7f3840156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f3844a84640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384006c550 con 0x7f3840155c30 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f3844a84640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384006c270 con 0x7f38400b0070 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f3844a84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384006da80 con 0x7f384006ca70 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3840155c30 0x7f3840156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3840155c30 0x7f3840156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56236/0 (socket says 172.21.15.94:56236) 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 -- 172.21.15.94:0/4232133040 learned_addr learned my addr 172.21.15.94:0/4232133040 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383e59c640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f384006ca70 0x7f38401eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383dd9b640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38400b0070 0x7f3840154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 -- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f384006ca70 msgr2=0x7f38401eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f384006ca70 0x7f38401eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 -- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38400b0070 msgr2=0x7f3840154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38400b0070 0x7f3840154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.748+0000 7f383ed9d640 1 -- 172.21.15.94:0/4232133040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400c2710 con 0x7f3840155c30 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.749+0000 7f383ed9d640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3840155c30 0x7f3840156560 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f38300097a0 tx=0x7f38300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.749+0000 7f382effd640 1 -- 172.21.15.94:0/4232133040 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830042070 con 0x7f3840155c30 2024-03-06T07:20:24.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.749+0000 7f382effd640 1 -- 172.21.15.94:0/4232133040 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38300329c0 con 0x7f3840155c30 2024-03-06T07:20:24.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.749+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38401ea6f0 con 0x7f3840155c30 2024-03-06T07:20:24.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.749+0000 7f382effd640 1 -- 172.21.15.94:0/4232133040 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830037040 con 0x7f3840155c30 2024-03-06T07:20:24.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.749+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38401eacb0 con 0x7f3840155c30 2024-03-06T07:20:24.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.750+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384004f330 con 0x7f3840155c30 2024-03-06T07:20:24.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.751+0000 7f382effd640 1 -- 172.21.15.94:0/4232133040 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3830045070 con 0x7f3840155c30 2024-03-06T07:20:24.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.751+0000 7f382effd640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:24.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.751+0000 7f382effd640 1 -- 172.21.15.94:0/4232133040 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f38300ac400 con 0x7f3840155c30 2024-03-06T07:20:24.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.751+0000 7f383e59c640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:24.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.751+0000 7f383e59c640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382006b4d0 0x7f382006d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3838000f60 tx=0x7f3838007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:24.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.755+0000 7f382effd640 1 -- 172.21.15.94:0/4232133040 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f383007a730 con 0x7f3840155c30 2024-03-06T07:20:24.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.926+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f384006d860 con 0x7f3840155c30 2024-03-06T07:20:24.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.926+0000 7f382effd640 1 -- 172.21.15.94:0/4232133040 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v58) v1 ==== 105+0+11 (secure 0 0 0) 0x7f383007e7d0 con 0x7f3840155c30 2024-03-06T07:20:24.928 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:20:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382006b4d0 msgr2=0x7f382006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382006b4d0 0x7f382006d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3838000f60 tx=0x7f3838007e20 comp rx=0 tx=0).stop 2024-03-06T07:20:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3840155c30 msgr2=0x7f3840156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3840155c30 0x7f3840156560 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f38300097a0 tx=0x7f38300046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 shutdown_connections 2024-03-06T07:20:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f382006b4d0 0x7f382006d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3840155c30 0x7f3840156560 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38400b0070 0x7f3840154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 --2- 172.21.15.94:0/4232133040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f384006ca70 0x7f38401eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:24.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 >> 172.21.15.94:0/4232133040 conn(0x7f38400ac590 msgr2=0x7f38400ae570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:24.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 shutdown_connections 2024-03-06T07:20:24.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:24.928+0000 7f3844a84640 1 -- 172.21.15.94:0/4232133040 wait complete. 2024-03-06T07:20:24.940 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:20:25.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.043+0000 7fd69909a640 1 Processor -- start 2024-03-06T07:20:25.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.043+0000 7fd69909a640 1 -- start start 2024-03-06T07:20:25.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd69909a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd6940b0070 0x7fd694155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd69909a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6940c5380 0x7fd6940c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd69909a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6940b0450 con 0x7fd69406ca70 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd69909a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69406c020 con 0x7fd6940b0070 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd69909a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69406cf20 con 0x7fd6940c5380 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd692d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd6940b0070 0x7fd694155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd692d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd6940b0070 0x7fd694155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56242/0 (socket says 172.21.15.94:56242) 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd68bfff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd69406ca70 0x7fd6940c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52900/0 (socket says 172.21.15.94:52900) 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd692d9d640 1 -- 172.21.15.94:0/4199711151 learned_addr learned my addr 172.21.15.94:0/4199711151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd69359e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6940c5380 0x7fd6940c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd692d9d640 1 -- 172.21.15.94:0/4199711151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6940c5380 msgr2=0x7fd6940c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd692d9d640 1 --2- 172.21.15.94:0/4199711151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6940c5380 0x7fd6940c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd692d9d640 1 -- 172.21.15.94:0/4199711151 >> v1:172.21.15.145:6789/0 conn(0x7fd69406ca70 legacy=0x7fd6940c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.045+0000 7fd692d9d640 1 -- 172.21.15.94:0/4199711151 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940c77f0 con 0x7fd6940b0070 2024-03-06T07:20:25.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.044+0000 7fd69259c640 1 -- 172.21.15.94:0/4199711151 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518334862 0 0) 0x7fd67c001110 con 0x7fd69406ca70 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.045+0000 7fd692d9d640 1 --2- 172.21.15.94:0/4199711151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940b0070 0x7fd694155c30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd684009870 tx=0x7fd68402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558d676896cbbc0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.045+0000 7fd69259c640 1 -- 172.21.15.94:0/4199711151 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684035020 con 0x7fd6940b0070 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.045+0000 7fd69259c640 1 -- 172.21.15.94:0/4199711151 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd684031640 con 0x7fd6940b0070 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.045+0000 7fd69259c640 1 -- 172.21.15.94:0/4199711151 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68402d040 con 0x7fd6940b0070 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 -- 172.21.15.94:0/4199711151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940b0070 msgr2=0x7fd694155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 --2- 172.21.15.94:0/4199711151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940b0070 0x7fd694155c30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd684009870 tx=0x7fd68402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 -- 172.21.15.94:0/4199711151 shutdown_connections 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 --2- 172.21.15.94:0/4199711151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6940c5380 0x7fd6940c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 --2- 172.21.15.94:0/4199711151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940b0070 0x7fd694155c30 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 -- 172.21.15.94:0/4199711151 >> 172.21.15.94:0/4199711151 conn(0x7fd6940ac590 msgr2=0x7fd6940ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 -- 172.21.15.94:0/4199711151 shutdown_connections 2024-03-06T07:20:25.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.046+0000 7fd69909a640 1 -- 172.21.15.94:0/4199711151 wait complete. 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 Processor -- start 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 -- start start 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd69406ca70 0x7fd694154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6940b0070 0x7fd6940befb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940c5380 0x7fd6940bf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69406c550 con 0x7fd6940c5380 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69406c270 con 0x7fd6940b0070 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69909a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69406da80 con 0x7fd69406ca70 2024-03-06T07:20:25.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940c5380 0x7fd6940bf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd692d9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd69406ca70 0x7fd694154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940c5380 0x7fd6940bf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56252/0 (socket says 172.21.15.94:56252) 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd69359e640 1 -- 172.21.15.94:0/1256736552 learned_addr learned my addr 172.21.15.94:0/1256736552 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.048+0000 7fd68bfff640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6940b0070 0x7fd6940befb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd692d9d640 1 -- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6940b0070 msgr2=0x7fd6940befb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd692d9d640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6940b0070 0x7fd6940befb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd692d9d640 1 -- 172.21.15.94:0/1256736552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940c5380 msgr2=0x7fd6940bf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd692d9d640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940c5380 0x7fd6940bf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd692d9d640 1 -- 172.21.15.94:0/1256736552 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940be670 con 0x7fd69406ca70 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd69359e640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940c5380 0x7fd6940bf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd692d9d640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd69406ca70 0x7fd694154400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd684033000 tx=0x7fd6840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd68b7fe640 1 -- 172.21.15.94:0/1256736552 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684040070 con 0x7fd69406ca70 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd68b7fe640 1 -- 172.21.15.94:0/1256736552 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6840399f0 con 0x7fd69406ca70 2024-03-06T07:20:25.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd68b7fe640 1 -- 172.21.15.94:0/1256736552 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684035040 con 0x7fd69406ca70 2024-03-06T07:20:25.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6940c3250 con 0x7fd69406ca70 2024-03-06T07:20:25.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.049+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6940c3810 con 0x7fd69406ca70 2024-03-06T07:20:25.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.050+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69404f330 con 0x7fd69406ca70 2024-03-06T07:20:25.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.051+0000 7fd68b7fe640 1 -- 172.21.15.94:0/1256736552 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd68404b060 con 0x7fd69406ca70 2024-03-06T07:20:25.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.051+0000 7fd68b7fe640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd67406b4d0 0x7fd67406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.051+0000 7fd68b7fe640 1 -- 172.21.15.94:0/1256736552 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fd6840abff0 con 0x7fd69406ca70 2024-03-06T07:20:25.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.051+0000 7fd68bfff640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd67406b4d0 0x7fd67406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.052+0000 7fd68bfff640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd67406b4d0 0x7fd67406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd67c007fd0 tx=0x7fd67c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.056+0000 7fd68b7fe640 1 -- 172.21.15.94:0/1256736552 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd68407a2a0 con 0x7fd69406ca70 2024-03-06T07:20:25.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.263+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7fd6940bec60 con 0x7fd69406ca70 2024-03-06T07:20:25.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.263+0000 7fd68b7fe640 1 -- 172.21.15.94:0/1256736552 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v58) v1 ==== 101+0+11 (secure 0 0 0) 0x7fd68407e340 con 0x7fd69406ca70 2024-03-06T07:20:25.265 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:20:25.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd67406b4d0 msgr2=0x7fd67406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd67406b4d0 0x7fd67406d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd67c007fd0 tx=0x7fd67c007de0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd69406ca70 msgr2=0x7fd694154400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd69406ca70 0x7fd694154400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd684033000 tx=0x7fd6840046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 shutdown_connections 2024-03-06T07:20:25.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd67406b4d0 0x7fd67406d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd6940c5380 0x7fd6940bf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6940b0070 0x7fd6940befb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 --2- 172.21.15.94:0/1256736552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd69406ca70 0x7fd694154400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 >> 172.21.15.94:0/1256736552 conn(0x7fd6940ac590 msgr2=0x7fd6940ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 shutdown_connections 2024-03-06T07:20:25.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.265+0000 7fd69909a640 1 -- 172.21.15.94:0/1256736552 wait complete. 2024-03-06T07:20:25.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:25.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.376+0000 7f24073bc640 1 Processor -- start 2024-03-06T07:20:25.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.376+0000 7f24073bc640 1 -- start start 2024-03-06T07:20:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f24073bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f2400155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f24073bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f24000c88c0 0x7f24000c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f24073bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24000c73d0 con 0x7f24000c5880 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f24073bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24000c8dd0 con 0x7f24000c88c0 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f24073bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000be2e0 con 0x7f240014f360 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2405158640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f2400155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2404957640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f24000c88c0 0x7f24000c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2404957640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f24000c88c0 0x7f24000c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:34122/0 (socket says 172.21.15.94:34122) 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2405158640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f2400155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:40270/0 (socket says 172.21.15.94:40270) 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2404957640 1 -- 172.21.15.94:0/4029133097 learned_addr learned my addr 172.21.15.94:0/4029133097 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2405959640 1 --1- 172.21.15.94:0/4029133097 >> v1:172.21.15.145:6789/0 conn(0x7f24000c5880 0x7f2400158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52902/0 (socket says 172.21.15.94:52902) 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2404957640 1 -- 172.21.15.94:0/4029133097 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 msgr2=0x7f2400155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2404957640 1 --2- 172.21.15.94:0/4029133097 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f2400155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2404957640 1 -- 172.21.15.94:0/4029133097 >> v1:172.21.15.145:6789/0 conn(0x7f24000c5880 legacy=0x7f2400158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f2404957640 1 -- 172.21.15.94:0/4029133097 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000c7670 con 0x7f24000c88c0 2024-03-06T07:20:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.377+0000 7f23f7fff640 1 -- 172.21.15.94:0/4029133097 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632859634 0 0) 0x7f24000c73d0 con 0x7f24000c5880 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.378+0000 7f2404957640 1 --2- 172.21.15.94:0/4029133097 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f24000c5340 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009870 tx=0x7f23f002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=500ae524b66942c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.378+0000 7f23f7fff640 1 -- 172.21.15.94:0/4029133097 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0035020 con 0x7f24000c88c0 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.378+0000 7f23f7fff640 1 -- 172.21.15.94:0/4029133097 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f0031640 con 0x7f24000c88c0 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.378+0000 7f23f7fff640 1 -- 172.21.15.94:0/4029133097 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f002d040 con 0x7f24000c88c0 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 -- 172.21.15.94:0/4029133097 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 msgr2=0x7f24000c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 --2- 172.21.15.94:0/4029133097 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f24000c5340 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009870 tx=0x7f23f002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 -- 172.21.15.94:0/4029133097 shutdown_connections 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 --2- 172.21.15.94:0/4029133097 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f24000c5340 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 --2- 172.21.15.94:0/4029133097 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f2400155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 -- 172.21.15.94:0/4029133097 >> 172.21.15.94:0/4029133097 conn(0x7f24000ac570 msgr2=0x7f24000ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 -- 172.21.15.94:0/4029133097 shutdown_connections 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.379+0000 7f24073bc640 1 -- 172.21.15.94:0/4029133097 wait complete. 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.380+0000 7f24073bc640 1 Processor -- start 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.380+0000 7f24073bc640 1 -- start start 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f24073bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24000c5880 0x7f240014df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f24073bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f240014bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f24073bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f240014c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f24073bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2400159270 con 0x7f24000c5880 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f24073bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000c7c30 con 0x7f24000c88c0 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f24073bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24001593f0 con 0x7f240014f360 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24000c5880 0x7f240014df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24000c5880 0x7f240014df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56264/0 (socket says 172.21.15.94:56264) 2024-03-06T07:20:25.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 -- 172.21.15.94:0/1794591775 learned_addr learned my addr 172.21.15.94:0/1794591775 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405959640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f240014c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2404957640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f240014bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 -- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 msgr2=0x7f240014c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f240014c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 -- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 msgr2=0x7f240014bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f240014bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2405158640 1 -- 172.21.15.94:0/1794591775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240014c5f0 con 0x7f24000c5880 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.381+0000 7f2404957640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f240014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.382+0000 7f2405158640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24000c5880 0x7f240014df70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00e870 tx=0x7f23fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.382+0000 7f23f5ffb640 1 -- 172.21.15.94:0/1794591775 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc017070 con 0x7f24000c5880 2024-03-06T07:20:25.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.382+0000 7f23f5ffb640 1 -- 172.21.15.94:0/1794591775 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc00c940 con 0x7f24000c5880 2024-03-06T07:20:25.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.382+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24001edf70 con 0x7f24000c5880 2024-03-06T07:20:25.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.382+0000 7f23f5ffb640 1 -- 172.21.15.94:0/1794591775 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00cd30 con 0x7f24000c5880 2024-03-06T07:20:25.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.382+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24001ee530 con 0x7f24000c5880 2024-03-06T07:20:25.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.383+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240004f330 con 0x7f24000c5880 2024-03-06T07:20:25.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.383+0000 7f23f5ffb640 1 -- 172.21.15.94:0/1794591775 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23fc01a070 con 0x7f24000c5880 2024-03-06T07:20:25.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.384+0000 7f23f5ffb640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e406b500 0x7f23e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.384+0000 7f23f5ffb640 1 -- 172.21.15.94:0/1794591775 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f23fc08a900 con 0x7f24000c5880 2024-03-06T07:20:25.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.384+0000 7f2404957640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e406b500 0x7f23e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.384+0000 7f2404957640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e406b500 0x7f23e406d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f23f0036000 tx=0x7f23f003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.392+0000 7f23f5ffb640 1 -- 172.21.15.94:0/1794591775 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23fc058bb0 con 0x7f24000c5880 2024-03-06T07:20:25.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.601+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f240006cda0 con 0x7f24000c5880 2024-03-06T07:20:25.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.602+0000 7f23f5ffb640 1 -- 172.21.15.94:0/1794591775 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6260 (secure 0 0 0) 0x7f23fc05cc50 con 0x7f24000c5880 2024-03-06T07:20:25.604 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:25.604 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:20:22.636873+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/279249774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":279249774},{"type":"v1","addr":"172.21.15.145:6837","nonce":279249774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:25.604 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 41 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e406b500 msgr2=0x7f23e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e406b500 0x7f23e406d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f23f0036000 tx=0x7f23f003c040 comp rx=0 tx=0).stop 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24000c5880 msgr2=0x7f240014df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24000c5880 0x7f240014df70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00e870 tx=0x7f23fc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 shutdown_connections 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e406b500 0x7f23e406d9c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f240014f360 0x7f240014c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24000c88c0 0x7f240014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 --2- 172.21.15.94:0/1794591775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24000c5880 0x7f240014df70 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 >> 172.21.15.94:0/1794591775 conn(0x7f24000ac570 msgr2=0x7f24000ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 shutdown_connections 2024-03-06T07:20:25.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.604+0000 7f24073bc640 1 -- 172.21.15.94:0/1794591775 wait complete. 2024-03-06T07:20:25.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:25.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.718+0000 7fc93acfb640 1 Processor -- start 2024-03-06T07:20:25.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.718+0000 7fc93acfb640 1 -- start start 2024-03-06T07:20:25.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc93acfb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc934155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc93acfb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc934158b60 0x7fc934158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc93acfb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9340c73d0 con 0x7fc93414f360 2024-03-06T07:20:25.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc93acfb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9340c8dd0 con 0x7fc934158b60 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc93acfb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340be2e0 con 0x7fc9340c88c0 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc939298640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc934158b60 0x7fc934158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc933fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc934155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc938a97640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc93414f360 0x7fc9340c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52906/0 (socket says 172.21.15.94:52906) 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc938a97640 1 -- 172.21.15.94:0/3415003611 learned_addr learned my addr 172.21.15.94:0/3415003611 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc9337fe640 1 -- 172.21.15.94:0/3415003611 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433512091 0 0) 0x7fc9340c73d0 con 0x7fc93414f360 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc9337fe640 1 -- 172.21.15.94:0/3415003611 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc924003520 con 0x7fc93414f360 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc933fff640 1 -- 172.21.15.94:0/3415003611 >> v2:172.21.15.145:3300/0 conn(0x7fc934158b60 msgr2=0x7fc934158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc933fff640 1 --2- 172.21.15.94:0/3415003611 >> v2:172.21.15.145:3300/0 conn(0x7fc934158b60 0x7fc934158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc933fff640 1 -- 172.21.15.94:0/3415003611 >> v1:172.21.15.145:6789/0 conn(0x7fc93414f360 legacy=0x7fc9340c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.719+0000 7fc933fff640 1 -- 172.21.15.94:0/3415003611 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9340c7670 con 0x7fc9340c88c0 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.720+0000 7fc933fff640 1 --2- 172.21.15.94:0/3415003611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc934155c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc9200097a0 tx=0x7fc92002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=599a951300f93300 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.720+0000 7fc9337fe640 1 -- 172.21.15.94:0/3415003611 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9200315a0 con 0x7fc9340c88c0 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.720+0000 7fc9337fe640 1 -- 172.21.15.94:0/3415003611 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920037070 con 0x7fc9340c88c0 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.720+0000 7fc9337fe640 1 -- 172.21.15.94:0/3415003611 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920031960 con 0x7fc9340c88c0 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 -- 172.21.15.94:0/3415003611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 msgr2=0x7fc934155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 --2- 172.21.15.94:0/3415003611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc934155c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc9200097a0 tx=0x7fc92002b840 comp rx=0 tx=0).stop 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 -- 172.21.15.94:0/3415003611 shutdown_connections 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 --2- 172.21.15.94:0/3415003611 >> v2:172.21.15.145:3300/0 conn(0x7fc934158b60 0x7fc934158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 --2- 172.21.15.94:0/3415003611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc934155c10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 -- 172.21.15.94:0/3415003611 >> 172.21.15.94:0/3415003611 conn(0x7fc9340ac570 msgr2=0x7fc9340ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 -- 172.21.15.94:0/3415003611 shutdown_connections 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.721+0000 7fc93acfb640 1 -- 172.21.15.94:0/3415003611 wait complete. 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.722+0000 7fc93acfb640 1 Processor -- start 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.722+0000 7fc93acfb640 1 -- start start 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc9340c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93414f360 0x7fc9340c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc934158b60 0x7fc9341ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340c5a50 con 0x7fc93414f360 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340c7c30 con 0x7fc934158b60 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9340c5bd0 con 0x7fc9340c88c0 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93414f360 0x7fc9340c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93414f360 0x7fc9340c4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56276/0 (socket says 172.21.15.94:56276) 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 -- 172.21.15.94:0/1556093652 learned_addr learned my addr 172.21.15.94:0/1556093652 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc938a97640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc9340c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc939298640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc934158b60 0x7fc9341ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 -- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 msgr2=0x7fc9340c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc9340c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 -- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc934158b60 msgr2=0x7fc9341ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc934158b60 0x7fc9341ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 -- 172.21.15.94:0/1556093652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9340c0e80 con 0x7fc93414f360 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc933fff640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93414f360 0x7fc9340c4970 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc9200098c0 tx=0x7fc9200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc9317fa640 1 -- 172.21.15.94:0/1556093652 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920032630 con 0x7fc93414f360 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc9317fa640 1 -- 172.21.15.94:0/1556093652 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9200327d0 con 0x7fc93414f360 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9340c1160 con 0x7fc93414f360 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc9317fa640 1 -- 172.21.15.94:0/1556093652 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920032bc0 con 0x7fc93414f360 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.723+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9341ec230 con 0x7fc93414f360 2024-03-06T07:20:25.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.724+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc934069460 con 0x7fc93414f360 2024-03-06T07:20:25.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.725+0000 7fc9317fa640 1 -- 172.21.15.94:0/1556093652 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc92004e430 con 0x7fc93414f360 2024-03-06T07:20:25.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.725+0000 7fc9317fa640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc91806b4d0 0x7fc91806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:25.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.726+0000 7fc9317fa640 1 -- 172.21.15.94:0/1556093652 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fc92002f030 con 0x7fc93414f360 2024-03-06T07:20:25.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.726+0000 7fc938a97640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc91806b4d0 0x7fc91806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:25.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.726+0000 7fc938a97640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc91806b4d0 0x7fc91806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc928005880 tx=0x7fc928009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:25.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.730+0000 7fc9317fa640 1 -- 172.21.15.94:0/1556093652 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc934069460 con 0x7fc93414f360 2024-03-06T07:20:25.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.919+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc93404f330 con 0x7fc93414f360 2024-03-06T07:20:25.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.920+0000 7fc9317fa640 1 -- 172.21.15.94:0/1556093652 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+20925 (secure 0 0 0) 0x7fc920040070 con 0x7fc93414f360 2024-03-06T07:20:25.922 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:25.922 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":58,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:19.628932+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.94:6833/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.145:6837/3125091562":"2024-03-07T07:20:16.601064+0000","172.21.15.145:6835/3125091562":"2024-03-07T07:20:16.601064+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:25.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc91806b4d0 msgr2=0x7fc91806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc91806b4d0 0x7fc91806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc928005880 tx=0x7fc928009040 comp rx=0 tx=0).stop 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93414f360 msgr2=0x7fc9340c4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93414f360 0x7fc9340c4970 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc9200098c0 tx=0x7fc9200046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 shutdown_connections 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc91806b4d0 0x7fc91806d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc934158b60 0x7fc9341ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93414f360 0x7fc9340c4970 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 --2- 172.21.15.94:0/1556093652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9340c88c0 0x7fc9340c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 >> 172.21.15.94:0/1556093652 conn(0x7fc9340ac570 msgr2=0x7fc9340c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 shutdown_connections 2024-03-06T07:20:25.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:25.922+0000 7fc93acfb640 1 -- 172.21.15.94:0/1556093652 wait complete. 2024-03-06T07:20:25.935 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:20:25.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.033+0000 7fee48415640 1 Processor -- start 2024-03-06T07:20:26.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.033+0000 7fee48415640 1 -- start start 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.034+0000 7fee48415640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee400c3280 0x7fee400c3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.034+0000 7fee48415640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fee401551e0 0x7fee401555f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.034+0000 7fee48415640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee40158800 con 0x7fee4006d9f0 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.034+0000 7fee48415640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee40158ae0 con 0x7fee400c3280 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.034+0000 7fee48415640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee40158dc0 con 0x7fee401551e0 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.034+0000 7fee459b0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee400c3280 0x7fee400c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee459b0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee400c3280 0x7fee400c3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56288/0 (socket says 172.21.15.94:56288) 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee459b0640 1 -- 172.21.15.94:0/1037059989 learned_addr learned my addr 172.21.15.94:0/1037059989 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee461b1640 1 --1- 172.21.15.94:0/1037059989 >> v1:172.21.15.94:6789/0 conn(0x7fee4006d9f0 0x7fee400a7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35916/0 (socket says 172.21.15.94:35916) 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee469b2640 1 --2- 172.21.15.94:0/1037059989 >> v2:172.21.15.145:3300/0 conn(0x7fee401551e0 0x7fee401555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee459b0640 1 -- 172.21.15.94:0/1037059989 >> v2:172.21.15.145:3300/0 conn(0x7fee401551e0 msgr2=0x7fee401555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee459b0640 1 --2- 172.21.15.94:0/1037059989 >> v2:172.21.15.145:3300/0 conn(0x7fee401551e0 0x7fee401555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee459b0640 1 -- 172.21.15.94:0/1037059989 >> v1:172.21.15.94:6789/0 conn(0x7fee4006d9f0 legacy=0x7fee400a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee459b0640 1 -- 172.21.15.94:0/1037059989 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee401590a0 con 0x7fee400c3280 2024-03-06T07:20:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.035+0000 7fee451af640 1 -- 172.21.15.94:0/1037059989 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758208125 0 0) 0x7fee40158800 con 0x7fee4006d9f0 2024-03-06T07:20:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.036+0000 7fee459b0640 1 --2- 172.21.15.94:0/1037059989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee400c3280 0x7fee400c3710 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fee40072f50 tx=0x7fee3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c94ef6e59941698 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.036+0000 7fee451af640 1 -- 172.21.15.94:0/1037059989 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee30035020 con 0x7fee400c3280 2024-03-06T07:20:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.036+0000 7fee451af640 1 -- 172.21.15.94:0/1037059989 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee30031870 con 0x7fee400c3280 2024-03-06T07:20:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.036+0000 7fee451af640 1 -- 172.21.15.94:0/1037059989 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee30031c40 con 0x7fee400c3280 2024-03-06T07:20:26.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 -- 172.21.15.94:0/1037059989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee400c3280 msgr2=0x7fee400c3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 --2- 172.21.15.94:0/1037059989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee400c3280 0x7fee400c3710 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fee40072f50 tx=0x7fee3002b910 comp rx=0 tx=0).stop 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 -- 172.21.15.94:0/1037059989 shutdown_connections 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 --2- 172.21.15.94:0/1037059989 >> v2:172.21.15.145:3300/0 conn(0x7fee401551e0 0x7fee401555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 --2- 172.21.15.94:0/1037059989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee400c3280 0x7fee400c3710 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 -- 172.21.15.94:0/1037059989 >> 172.21.15.94:0/1037059989 conn(0x7fee400aa630 msgr2=0x7fee400aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 -- 172.21.15.94:0/1037059989 shutdown_connections 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.037+0000 7fee48415640 1 -- 172.21.15.94:0/1037059989 wait complete. 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.038+0000 7fee48415640 1 Processor -- start 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.038+0000 7fee48415640 1 -- start start 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee48415640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee4006d9f0 0x7fee4006ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee48415640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee400c3280 0x7fee400ad460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee48415640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee401551e0 0x7fee400ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee48415640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4006df50 con 0x7fee401551e0 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee48415640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4006ddd0 con 0x7fee400c3280 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee48415640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee400c6910 con 0x7fee4006d9f0 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee401551e0 0x7fee400ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee401551e0 0x7fee400ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56290/0 (socket says 172.21.15.94:56290) 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 -- 172.21.15.94:0/3355065669 learned_addr learned my addr 172.21.15.94:0/3355065669 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee461b1640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee4006d9f0 0x7fee4006ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee459b0640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee400c3280 0x7fee400ad460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 -- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee4006d9f0 msgr2=0x7fee4006ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee4006d9f0 0x7fee4006ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 -- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee400c3280 msgr2=0x7fee400ad460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee400c3280 0x7fee400ad460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 -- 172.21.15.94:0/3355065669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee401ee6b0 con 0x7fee401551e0 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee461b1640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee4006d9f0 0x7fee4006ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee469b2640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee401551e0 0x7fee400ad9a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fee3800ec10 tx=0x7fee3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.039+0000 7fee36ffd640 1 -- 172.21.15.94:0/3355065669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee38018020 con 0x7fee401551e0 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.040+0000 7fee36ffd640 1 -- 172.21.15.94:0/3355065669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3800cd20 con 0x7fee401551e0 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.040+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee401ee990 con 0x7fee401551e0 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.040+0000 7fee36ffd640 1 -- 172.21.15.94:0/3355065669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee380146c0 con 0x7fee401551e0 2024-03-06T07:20:26.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.040+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee40070150 con 0x7fee401551e0 2024-03-06T07:20:26.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.041+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4004f330 con 0x7fee401551e0 2024-03-06T07:20:26.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.041+0000 7fee36ffd640 1 -- 172.21.15.94:0/3355065669 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee38005d10 con 0x7fee401551e0 2024-03-06T07:20:26.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.042+0000 7fee36ffd640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.042+0000 7fee36ffd640 1 -- 172.21.15.94:0/3355065669 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fee3808b590 con 0x7fee401551e0 2024-03-06T07:20:26.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.042+0000 7fee461b1640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.042+0000 7fee461b1640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee2406b4d0 0x7fee2406d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000f60 tx=0x7fee3c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.046+0000 7fee36ffd640 1 -- 172.21.15.94:0/3355065669 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee38022030 con 0x7fee401551e0 2024-03-06T07:20:26.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.412+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee40073240 con 0x7fee401551e0 2024-03-06T07:20:26.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.413+0000 7fee36ffd640 1 -- 172.21.15.94:0/3355065669 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6260 (secure 0 0 0) 0x7fee38010040 con 0x7fee401551e0 2024-03-06T07:20:26.415 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:26.415 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:20:22.636873+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/279249774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":279249774},{"type":"v1","addr":"172.21.15.145:6837","nonce":279249774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:26.416 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 41 2024-03-06T07:20:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee2406b4d0 msgr2=0x7fee2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee2406b4d0 0x7fee2406d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000f60 tx=0x7fee3c007c10 comp rx=0 tx=0).stop 2024-03-06T07:20:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee401551e0 msgr2=0x7fee400ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee401551e0 0x7fee400ad9a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fee3800ec10 tx=0x7fee3800c490 comp rx=0 tx=0).stop 2024-03-06T07:20:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 shutdown_connections 2024-03-06T07:20:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee2406b4d0 0x7fee2406d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee401551e0 0x7fee400ad9a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee400c3280 0x7fee400ad460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 --2- 172.21.15.94:0/3355065669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee4006d9f0 0x7fee4006ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 >> 172.21.15.94:0/3355065669 conn(0x7fee400aa630 msgr2=0x7fee40154960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 shutdown_connections 2024-03-06T07:20:26.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.415+0000 7fee48415640 1 -- 172.21.15.94:0/3355065669 wait complete. 2024-03-06T07:20:26.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:26.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.531+0000 7fe9ef247640 1 Processor -- start 2024-03-06T07:20:26.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.531+0000 7fe9ef247640 1 -- start start 2024-03-06T07:20:26.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ef247640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e8155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ef247640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe9e80c5860 0x7fe9e806f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ef247640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe9e81586f0 0x7fe9e8158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ef247640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e806df40 con 0x7fe9e81586f0 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ef247640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e80c5cc0 con 0x7fe9e80c5860 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ef247640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e81591c0 con 0x7fe9e80c5480 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ed7e4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe9e81586f0 0x7fe9e8158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9e7fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe9e80c5860 0x7fe9e806f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.532+0000 7fe9ecfe3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e8155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ed7e4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe9e81586f0 0x7fe9e8158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56300/0 (socket says 172.21.15.94:56300) 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2236953338 learned_addr learned my addr 172.21.15.94:0/2236953338 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:26.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2236953338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 msgr2=0x7fe9e8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ed7e4640 1 --2- 172.21.15.94:0/2236953338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e8155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2236953338 >> v2:172.21.15.145:3300/0 conn(0x7fe9e80c5860 msgr2=0x7fe9e806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ed7e4640 1 --2- 172.21.15.94:0/2236953338 >> v2:172.21.15.145:3300/0 conn(0x7fe9e80c5860 0x7fe9e806f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2236953338 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e81594a0 con 0x7fe9e81586f0 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9ecfe3640 1 --2- 172.21.15.94:0/2236953338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.533+0000 7fe9e7fff640 1 --2- 172.21.15.94:0/2236953338 >> v2:172.21.15.145:3300/0 conn(0x7fe9e80c5860 0x7fe9e806f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.534+0000 7fe9ed7e4640 1 --2- 172.21.15.94:0/2236953338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e8158b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80735c0 tx=0x7fe9e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1fa2e5918396a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.534+0000 7fe9e77fe640 1 -- 172.21.15.94:0/2236953338 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e002c9e0 con 0x7fe9e81586f0 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.534+0000 7fe9e77fe640 1 -- 172.21.15.94:0/2236953338 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e002cb80 con 0x7fe9e81586f0 2024-03-06T07:20:26.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.534+0000 7fe9e77fe640 1 -- 172.21.15.94:0/2236953338 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e00343f0 con 0x7fe9e81586f0 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.534+0000 7fe9ef247640 1 -- 172.21.15.94:0/2236953338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 msgr2=0x7fe9e8158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.534+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2236953338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e8158b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80735c0 tx=0x7fe9e002be40 comp rx=0 tx=0).stop 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.535+0000 7fe9ef247640 1 -- 172.21.15.94:0/2236953338 shutdown_connections 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.535+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2236953338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e8158b80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.535+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2236953338 >> v2:172.21.15.145:3300/0 conn(0x7fe9e80c5860 0x7fe9e806f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.535+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2236953338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.535+0000 7fe9ef247640 1 -- 172.21.15.94:0/2236953338 >> 172.21.15.94:0/2236953338 conn(0x7fe9e806b1a0 msgr2=0x7fe9e806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:26.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.535+0000 7fe9ef247640 1 -- 172.21.15.94:0/2236953338 shutdown_connections 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.535+0000 7fe9ef247640 1 -- 172.21.15.94:0/2236953338 wait complete. 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 Processor -- start 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 -- start start 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e80a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9e80c5860 0x7fe9e81e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e80aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e8152a20 con 0x7fe9e81586f0 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e8155680 con 0x7fe9e80c5860 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.536+0000 7fe9ef247640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e8155050 con 0x7fe9e80c5480 2024-03-06T07:20:26.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e80aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ecfe3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e80a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e80aa270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56314/0 (socket says 172.21.15.94:56314) 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2493471468 learned_addr learned my addr 172.21.15.94:0/2493471468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9e7fff640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9e80c5860 0x7fe9e81e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 msgr2=0x7fe9e80a96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e80a96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9e80c5860 msgr2=0x7fe9e81e1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9e80c5860 0x7fe9e81e1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 -- 172.21.15.94:0/2493471468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80a9c10 con 0x7fe9e81586f0 2024-03-06T07:20:26.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ed7e4640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e80aa270 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0036040 tx=0x7fe9e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9e57fa640 1 -- 172.21.15.94:0/2493471468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0044020 con 0x7fe9e81586f0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9e57fa640 1 -- 172.21.15.94:0/2493471468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e0034e50 con 0x7fe9e81586f0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9e57fa640 1 -- 172.21.15.94:0/2493471468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0009510 con 0x7fe9e81586f0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e814fb80 con 0x7fe9e81586f0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.537+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e80ad0f0 con 0x7fe9e81586f0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.538+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e804f330 con 0x7fe9e81586f0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.539+0000 7fe9e57fa640 1 -- 172.21.15.94:0/2493471468 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9e0047020 con 0x7fe9e81586f0 2024-03-06T07:20:26.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.539+0000 7fe9e57fa640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe9cc06b500 0x7fe9cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.539+0000 7fe9e57fa640 1 -- 172.21.15.94:0/2493471468 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fe9e00ad210 con 0x7fe9e81586f0 2024-03-06T07:20:26.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.539+0000 7fe9ecfe3640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe9cc06b500 0x7fe9cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.540+0000 7fe9ecfe3640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe9cc06b500 0x7fe9cc06d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009560 tx=0x7fe9dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.543+0000 7fe9e57fa640 1 -- 172.21.15.94:0/2493471468 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9e004e020 con 0x7fe9e81586f0 2024-03-06T07:20:26.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.835+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e806e0c0 con 0x7fe9e81586f0 2024-03-06T07:20:26.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.835+0000 7fe9e57fa640 1 -- 172.21.15.94:0/2493471468 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6260 (secure 0 0 0) 0x7fe9e814fb80 con 0x7fe9e81586f0 2024-03-06T07:20:26.837 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:26.837 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5530,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":6,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:19:53.385858+0000","modified":"2024-03-06T07:20:22.636873+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5827":{"gid":5827,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/279249774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":279249774},{"type":"v1","addr":"172.21.15.145:6837","nonce":279249774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5827,"qdb_cluster":[5827]},"id":3},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:09.624864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5652":{"gid":5652,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1798838613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1798838613},{"type":"v1","addr":"172.21.15.94:6834","nonce":1798838613}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5652,"qdb_cluster":[5652]},"id":4}]} 2024-03-06T07:20:26.838 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 41 2024-03-06T07:20:26.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe9cc06b500 msgr2=0x7fe9cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe9cc06b500 0x7fe9cc06d9c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009560 tx=0x7fe9dc009190 comp rx=0 tx=0).stop 2024-03-06T07:20:26.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 msgr2=0x7fe9e80aa270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e80aa270 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0036040 tx=0x7fe9e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 shutdown_connections 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe9cc06b500 0x7fe9cc06d9c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9e81586f0 0x7fe9e80aa270 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9e80c5860 0x7fe9e81e1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 --2- 172.21.15.94:0/2493471468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9e80c5480 0x7fe9e80a96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 >> 172.21.15.94:0/2493471468 conn(0x7fe9e806b1a0 msgr2=0x7fe9e806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 shutdown_connections 2024-03-06T07:20:26.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.837+0000 7fe9ef247640 1 -- 172.21.15.94:0/2493471468 wait complete. 2024-03-06T07:20:26.850 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:26.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.949+0000 7f67c47e9640 1 Processor -- start 2024-03-06T07:20:26.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.949+0000 7f67c47e9640 1 -- start start 2024-03-06T07:20:26.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c47e9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67bc0c88c0 0x7f67bc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c47e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc158b60 0x7f67bc158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c47e9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67bc0c73d0 con 0x7f67bc14f360 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c47e9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67bc0c8dd0 con 0x7f67bc0c88c0 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c47e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0be2e0 con 0x7f67bc158b60 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c2d86640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc158b60 0x7f67bc158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c1d84640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67bc0c88c0 0x7f67bc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c2585640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f67bc14f360 0x7f67bc0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52914/0 (socket says 172.21.15.94:52914) 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.950+0000 7f67c2585640 1 -- 172.21.15.94:0/2660727874 learned_addr learned my addr 172.21.15.94:0/2660727874 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.951+0000 7f67c2d86640 1 -- 172.21.15.94:0/2660727874 >> v2:172.21.15.145:3300/0 conn(0x7f67bc0c88c0 msgr2=0x7f67bc155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.951+0000 7f67c2d86640 1 --2- 172.21.15.94:0/2660727874 >> v2:172.21.15.145:3300/0 conn(0x7f67bc0c88c0 0x7f67bc155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.951+0000 7f67c2d86640 1 -- 172.21.15.94:0/2660727874 >> v1:172.21.15.145:6789/0 conn(0x7f67bc14f360 legacy=0x7f67bc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.951+0000 7f67c2d86640 1 -- 172.21.15.94:0/2660727874 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc0c7670 con 0x7f67bc158b60 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.951+0000 7f67c2d86640 1 --2- 172.21.15.94:0/2660727874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc158b60 0x7f67bc158ff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009d90 tx=0x7f67b402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0c2845c21f81918 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c1583640 1 -- 172.21.15.94:0/2660727874 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4039070 con 0x7f67bc158b60 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c1583640 1 -- 172.21.15.94:0/2660727874 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b402cbc0 con 0x7f67bc158b60 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c1583640 1 -- 172.21.15.94:0/2660727874 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4034440 con 0x7f67bc158b60 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 -- 172.21.15.94:0/2660727874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc158b60 msgr2=0x7f67bc158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 --2- 172.21.15.94:0/2660727874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc158b60 0x7f67bc158ff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009d90 tx=0x7f67b402be10 comp rx=0 tx=0).stop 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 -- 172.21.15.94:0/2660727874 shutdown_connections 2024-03-06T07:20:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 --2- 172.21.15.94:0/2660727874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc158b60 0x7f67bc158ff0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 --2- 172.21.15.94:0/2660727874 >> v2:172.21.15.145:3300/0 conn(0x7f67bc0c88c0 0x7f67bc155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 -- 172.21.15.94:0/2660727874 >> 172.21.15.94:0/2660727874 conn(0x7f67bc0ac570 msgr2=0x7f67bc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:26.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 -- 172.21.15.94:0/2660727874 shutdown_connections 2024-03-06T07:20:26.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.952+0000 7f67c47e9640 1 -- 172.21.15.94:0/2660727874 wait complete. 2024-03-06T07:20:26.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.953+0000 7f67c47e9640 1 Processor -- start 2024-03-06T07:20:26.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.953+0000 7f67c47e9640 1 -- start start 2024-03-06T07:20:26.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c47e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67bc0c88c0 0x7f67bc0c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c47e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc14f360 0x7f67bc1ec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c47e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67bc158b60 0x7f67bc1ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c47e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0c5a50 con 0x7f67bc0c88c0 2024-03-06T07:20:26.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c47e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0c7c30 con 0x7f67bc158b60 2024-03-06T07:20:26.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c47e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc0c5bd0 con 0x7f67bc14f360 2024-03-06T07:20:26.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67bc0c88c0 0x7f67bc0c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67bc0c88c0 0x7f67bc0c2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56330/0 (socket says 172.21.15.94:56330) 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 -- 172.21.15.94:0/4141639686 learned_addr learned my addr 172.21.15.94:0/4141639686 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c1d84640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc14f360 0x7f67bc1ec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2d86640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67bc158b60 0x7f67bc1ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 -- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc14f360 msgr2=0x7f67bc1ec0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc14f360 0x7f67bc1ec0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 -- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67bc158b60 msgr2=0x7f67bc1ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67bc158b60 0x7f67bc1ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c2585640 1 -- 172.21.15.94:0/4141639686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc0c3430 con 0x7f67bc0c88c0 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.954+0000 7f67c1d84640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc14f360 0x7f67bc1ec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.955+0000 7f67c2585640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67bc0c88c0 0x7f67bc0c2ef0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f67b800c3c0 tx=0x7f67b800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.955+0000 7f67b2ffd640 1 -- 172.21.15.94:0/4141639686 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8017070 con 0x7f67bc0c88c0 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.955+0000 7f67b2ffd640 1 -- 172.21.15.94:0/4141639686 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b8005ce0 con 0x7f67bc0c88c0 2024-03-06T07:20:26.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.955+0000 7f67b2ffd640 1 -- 172.21.15.94:0/4141639686 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b80072f0 con 0x7f67bc0c88c0 2024-03-06T07:20:26.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.955+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67bc1eec50 con 0x7f67bc0c88c0 2024-03-06T07:20:26.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.955+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67bc0c26f0 con 0x7f67bc0c88c0 2024-03-06T07:20:26.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.956+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67bc0c7f10 con 0x7f67bc0c88c0 2024-03-06T07:20:26.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.956+0000 7f67b2ffd640 1 -- 172.21.15.94:0/4141639686 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67b801a070 con 0x7f67bc0c88c0 2024-03-06T07:20:26.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.957+0000 7f67b2ffd640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67a006b500 0x7f67a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:26.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.957+0000 7f67b2ffd640 1 -- 172.21.15.94:0/4141639686 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f67b80899f0 con 0x7f67bc0c88c0 2024-03-06T07:20:26.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.957+0000 7f67c1d84640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67a006b500 0x7f67a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:26.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.957+0000 7f67c1d84640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67a006b500 0x7f67a006d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0c9260 tx=0x7f67ac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:26.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:26.961+0000 7f67b2ffd640 1 -- 172.21.15.94:0/4141639686 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67bc0c7f10 con 0x7f67bc0c88c0 2024-03-06T07:20:27.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.134+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67bc0c2500 con 0x7f67bc0c88c0 2024-03-06T07:20:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.135+0000 7f67b2ffd640 1 -- 172.21.15.94:0/4141639686 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+20925 (secure 0 0 0) 0x7f67b8057d20 con 0x7f67bc0c88c0 2024-03-06T07:20:27.137 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:27.138 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":58,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:19.628932+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:19:51.596832+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-06T07:19:52.352004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.94:6833/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.145:6837/3125091562":"2024-03-07T07:20:16.601064+0000","172.21.15.145:6835/3125091562":"2024-03-07T07:20:16.601064+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:27.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67a006b500 msgr2=0x7f67a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67a006b500 0x7f67a006d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0c9260 tx=0x7f67ac009190 comp rx=0 tx=0).stop 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67bc0c88c0 msgr2=0x7f67bc0c2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67bc0c88c0 0x7f67bc0c2ef0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f67b800c3c0 tx=0x7f67b800c880 comp rx=0 tx=0).stop 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 shutdown_connections 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67a006b500 0x7f67a006d9c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67bc158b60 0x7f67bc1ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67bc14f360 0x7f67bc1ec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 --2- 172.21.15.94:0/4141639686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67bc0c88c0 0x7f67bc0c2ef0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 >> 172.21.15.94:0/4141639686 conn(0x7f67bc0ac570 msgr2=0x7f67bc06c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 shutdown_connections 2024-03-06T07:20:27.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.137+0000 7f67c47e9640 1 -- 172.21.15.94:0/4141639686 wait complete. 2024-03-06T07:20:27.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:20:27.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.256+0000 7f1c10eec640 1 Processor -- start 2024-03-06T07:20:27.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.256+0000 7f1c10eec640 1 -- start start 2024-03-06T07:20:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.257+0000 7f1c10eec640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c8950 0x7f1c0c0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.257+0000 7f1c10eec640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c155130 con 0x7f1c0c0c69f0 2024-03-06T07:20:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.257+0000 7f1c10eec640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c0ad480 con 0x7f1c0c1558d0 2024-03-06T07:20:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.257+0000 7f1c10eec640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c155440 con 0x7f1c0c0c8950 2024-03-06T07:20:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.257+0000 7f1c0a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1c0c0c69f0 0x7f1c0c0acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35930/0 (socket says 172.21.15.94:35930) 2024-03-06T07:20:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.257+0000 7f1c0a59c640 1 -- 172.21.15.94:0/347798588 learned_addr learned my addr 172.21.15.94:0/347798588 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.257+0000 7f1c09d9b640 1 --2- 172.21.15.94:0/347798588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c8950 0x7f1c0c0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.258+0000 7f1c0959a640 1 -- 172.21.15.94:0/347798588 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890661431 0 0) 0x7f1bfc001110 con 0x7f1c0c0c69f0 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.258+0000 7f1c0959a640 1 -- 172.21.15.94:0/347798588 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf4003520 con 0x7f1c0c0c69f0 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.258+0000 7f1c0959a640 1 -- 172.21.15.94:0/347798588 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161098280 0 0) 0x7f1c0c0ad480 con 0x7f1c0c1558d0 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.258+0000 7f1c09d9b640 1 -- 172.21.15.94:0/347798588 >> v1:172.21.15.145:6789/0 conn(0x7f1c0c1558d0 legacy=0x7f1c0c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.258+0000 7f1c09d9b640 1 -- 172.21.15.94:0/347798588 >> v1:172.21.15.94:6789/0 conn(0x7f1c0c0c69f0 legacy=0x7f1c0c0acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.258+0000 7f1c09d9b640 1 -- 172.21.15.94:0/347798588 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c0c5d50 con 0x7f1c0c0c8950 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.258+0000 7f1c09d9b640 1 --2- 172.21.15.94:0/347798588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c8950 0x7f1c0c0c6dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c000097d0 tx=0x7f1c0002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e628aaeb5761198 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c0959a640 1 -- 172.21.15.94:0/347798588 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00035020 con 0x7f1c0c0c8950 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c0959a640 1 -- 172.21.15.94:0/347798588 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00031800 con 0x7f1c0c0c8950 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c0959a640 1 -- 172.21.15.94:0/347798588 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00031bd0 con 0x7f1c0c0c8950 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c10eec640 1 -- 172.21.15.94:0/347798588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c8950 msgr2=0x7f1c0c0c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c10eec640 1 --2- 172.21.15.94:0/347798588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c8950 0x7f1c0c0c6dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c000097d0 tx=0x7f1c0002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c10eec640 1 -- 172.21.15.94:0/347798588 shutdown_connections 2024-03-06T07:20:27.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c10eec640 1 --2- 172.21.15.94:0/347798588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c8950 0x7f1c0c0c6dd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c10eec640 1 -- 172.21.15.94:0/347798588 >> 172.21.15.94:0/347798588 conn(0x7f1c0c0cae70 msgr2=0x7f1c0c0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c10eec640 1 -- 172.21.15.94:0/347798588 shutdown_connections 2024-03-06T07:20:27.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.259+0000 7f1c10eec640 1 -- 172.21.15.94:0/347798588 wait complete. 2024-03-06T07:20:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.260+0000 7f1c10eec640 1 Processor -- start 2024-03-06T07:20:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c10eec640 1 -- start start 2024-03-06T07:20:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c10eec640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c69f0 0x7f1c0c157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c10eec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c0c0c8950 0x7f1c0c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c10eec640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c0c1558d0 0x7f1c0c0c22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c10eec640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c152a20 con 0x7f1c0c0c8950 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c10eec640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c0c6310 con 0x7f1c0c1558d0 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c10eec640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c152ba0 con 0x7f1c0c0c69f0 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c09d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c0c0c8950 0x7f1c0c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c09d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c0c0c8950 0x7f1c0c06c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56332/0 (socket says 172.21.15.94:56332) 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c09d9b640 1 -- 172.21.15.94:0/2101611570 learned_addr learned my addr 172.21.15.94:0/2101611570 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c0a59c640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c69f0 0x7f1c0c157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.261+0000 7f1c0ad9d640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c0c1558d0 0x7f1c0c0c22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c09d9b640 1 -- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c69f0 msgr2=0x7f1c0c157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c09d9b640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c69f0 0x7f1c0c157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c09d9b640 1 -- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c0c1558d0 msgr2=0x7f1c0c0c22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c09d9b640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c0c1558d0 0x7f1c0c0c22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c09d9b640 1 -- 172.21.15.94:0/2101611570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c157c30 con 0x7f1c0c0c8950 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c09d9b640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c0c0c8950 0x7f1c0c06c280 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1c000097d0 tx=0x7f1c000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1bfaffd640 1 -- 172.21.15.94:0/2101611570 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00041070 con 0x7f1c0c0c8950 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1bfaffd640 1 -- 172.21.15.94:0/2101611570 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0003ad00 con 0x7f1c0c0c8950 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0c0c43e0 con 0x7f1c0c0c8950 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1bfaffd640 1 -- 172.21.15.94:0/2101611570 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00035040 con 0x7f1c0c0c8950 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.262+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c06c7c0 con 0x7f1c0c0c8950 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.263+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c04f330 con 0x7f1c0c0c8950 2024-03-06T07:20:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.264+0000 7f1bfaffd640 1 -- 172.21.15.94:0/2101611570 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c00038020 con 0x7f1c0c0c8950 2024-03-06T07:20:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.264+0000 7f1bfaffd640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bec06b4d0 0x7f1bec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.264+0000 7f1bfaffd640 1 -- 172.21.15.94:0/2101611570 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f1c000ac250 con 0x7f1c0c0c8950 2024-03-06T07:20:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.264+0000 7f1c0a59c640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bec06b4d0 0x7f1bec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.265+0000 7f1c0a59c640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bec06b4d0 0x7f1bec06d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc005880 tx=0x7f1bfc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.268+0000 7f1bfaffd640 1 -- 172.21.15.94:0/2101611570 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c0007a4f0 con 0x7f1c0c0c8950 2024-03-06T07:20:27.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.483+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1c0c159550 con 0x7f1c0c0c8950 2024-03-06T07:20:27.486 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:20:27.484+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:20:27.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.485+0000 7f1bfaffd640 1 -- 172.21.15.94:0/2101611570 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1c0007e590 con 0x7f1c0c0c8950 2024-03-06T07:20:27.487 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:20:27.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bec06b4d0 msgr2=0x7f1bec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bec06b4d0 0x7f1bec06d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc005880 tx=0x7f1bfc009040 comp rx=0 tx=0).stop 2024-03-06T07:20:27.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c0c0c8950 msgr2=0x7f1c0c06c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c0c0c8950 0x7f1c0c06c280 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1c000097d0 tx=0x7f1c000046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 shutdown_connections 2024-03-06T07:20:27.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bec06b4d0 0x7f1bec06d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c0c1558d0 0x7f1c0c0c22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1c0c0c8950 0x7f1c0c06c280 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 --2- 172.21.15.94:0/2101611570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c0c0c69f0 0x7f1c0c157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 >> 172.21.15.94:0/2101611570 conn(0x7f1c0c0cae70 msgr2=0x7f1c0c0c7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 shutdown_connections 2024-03-06T07:20:27.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.488+0000 7f1c10eec640 1 -- 172.21.15.94:0/2101611570 wait complete. 2024-03-06T07:20:27.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:20:27.515 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:20:27.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa4016c3640 1 Processor -- start 2024-03-06T07:20:27.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa4016c3640 1 -- start start 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa4016c3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa3fc0ab310 0x7fa3fc0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa4016c3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa3fc1538c0 0x7fa3fc1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa4016c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc154330 con 0x7fa3fc0a97c0 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa4016c3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c46a0 con 0x7fa3fc0ab310 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa4016c3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc153dc0 con 0x7fa3fc1538c0 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.603+0000 7fa3faffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa3fc0ab310 0x7fa3fc0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.604+0000 7fa3faffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa3fc0ab310 0x7fa3fc0ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56340/0 (socket says 172.21.15.94:56340) 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.604+0000 7fa3faffd640 1 -- 172.21.15.94:0/1189295704 learned_addr learned my addr 172.21.15.94:0/1189295704 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.604+0000 7fa3fa7fc640 1 --2- 172.21.15.94:0/1189295704 >> v2:172.21.15.145:3300/0 conn(0x7fa3fc1538c0 0x7fa3fc1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.604+0000 7fa3faffd640 1 -- 172.21.15.94:0/1189295704 >> v2:172.21.15.145:3300/0 conn(0x7fa3fc1538c0 msgr2=0x7fa3fc1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:27.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.604+0000 7fa3faffd640 1 --2- 172.21.15.94:0/1189295704 >> v2:172.21.15.145:3300/0 conn(0x7fa3fc1538c0 0x7fa3fc1494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.604+0000 7fa3faffd640 1 -- 172.21.15.94:0/1189295704 >> v1:172.21.15.145:6789/0 conn(0x7fa3fc0a97c0 legacy=0x7fa3fc0c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.604+0000 7fa3faffd640 1 -- 172.21.15.94:0/1189295704 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0c5d30 con 0x7fa3fc0ab310 2024-03-06T07:20:27.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa3faffd640 1 --2- 172.21.15.94:0/1189295704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc0ab310 0x7fa3fc0ab6f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0c75b0 tx=0x7fa3ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936c8a632833d16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa3f9ffb640 1 -- 172.21.15.94:0/1189295704 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec035020 con 0x7fa3fc0ab310 2024-03-06T07:20:27.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa3f9ffb640 1 -- 172.21.15.94:0/1189295704 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec031870 con 0x7fa3fc0ab310 2024-03-06T07:20:27.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa3f9ffb640 1 -- 172.21.15.94:0/1189295704 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec031c40 con 0x7fa3fc0ab310 2024-03-06T07:20:27.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 -- 172.21.15.94:0/1189295704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc0ab310 msgr2=0x7fa3fc0ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 --2- 172.21.15.94:0/1189295704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc0ab310 0x7fa3fc0ab6f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0c75b0 tx=0x7fa3ec02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 -- 172.21.15.94:0/1189295704 shutdown_connections 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 --2- 172.21.15.94:0/1189295704 >> v2:172.21.15.145:3300/0 conn(0x7fa3fc1538c0 0x7fa3fc1494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 --2- 172.21.15.94:0/1189295704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc0ab310 0x7fa3fc0ab6f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 -- 172.21.15.94:0/1189295704 >> 172.21.15.94:0/1189295704 conn(0x7fa3fc0c6710 msgr2=0x7fa3fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 -- 172.21.15.94:0/1189295704 shutdown_connections 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.605+0000 7fa4016c3640 1 -- 172.21.15.94:0/1189295704 wait complete. 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 Processor -- start 2024-03-06T07:20:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 -- start start 2024-03-06T07:20:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3fc0a97c0 0x7fa3fc156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3fc0ab310 0x7fa3fc14f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc1538c0 0x7fa3fc14fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0abc30 con 0x7fa3fc1538c0 2024-03-06T07:20:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c62f0 con 0x7fa3fc0ab310 2024-03-06T07:20:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa4016c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c5240 con 0x7fa3fc0a97c0 2024-03-06T07:20:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc1538c0 0x7fa3fc14fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc1538c0 0x7fa3fc14fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:56356/0 (socket says 172.21.15.94:56356) 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/2303083731 learned_addr learned my addr 172.21.15.94:0/2303083731 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa3fa7fc640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3fc0ab310 0x7fa3fc14f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.607+0000 7fa3faffd640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3fc0a97c0 0x7fa3fc156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3fc0a97c0 msgr2=0x7fa3fc156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3fb7fe640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3fc0a97c0 0x7fa3fc156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3fc0ab310 msgr2=0x7fa3fc14f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3fb7fe640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3fc0ab310 0x7fa3fc14f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3fb7fe640 1 -- 172.21.15.94:0/2303083731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc1eabb0 con 0x7fa3fc1538c0 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3faffd640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3fc0a97c0 0x7fa3fc156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3fb7fe640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc1538c0 0x7fa3fc14fd70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c7d0 tx=0x7fa3f400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3e37fe640 1 -- 172.21.15.94:0/2303083731 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4003b70 con 0x7fa3fc1538c0 2024-03-06T07:20:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3e37fe640 1 -- 172.21.15.94:0/2303083731 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4003d30 con 0x7fa3fc1538c0 2024-03-06T07:20:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc1eae50 con 0x7fa3fc1538c0 2024-03-06T07:20:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa3e37fe640 1 -- 172.21.15.94:0/2303083731 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f400aa90 con 0x7fa3fc1538c0 2024-03-06T07:20:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.608+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc1eb410 con 0x7fa3fc1538c0 2024-03-06T07:20:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.609+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc04f330 con 0x7fa3fc1538c0 2024-03-06T07:20:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.610+0000 7fa3e37fe640 1 -- 172.21.15.94:0/2303083731 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3f400ac30 con 0x7fa3fc1538c0 2024-03-06T07:20:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.610+0000 7fa3e37fe640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:27.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.610+0000 7fa3e37fe640 1 -- 172.21.15.94:0/2303083731 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7215+0+0 (secure 0 0 0) 0x7fa3f408a5d0 con 0x7fa3fc1538c0 2024-03-06T07:20:27.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.610+0000 7fa3faffd640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:27.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.611+0000 7fa3faffd640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa3e806b4d0 0x7fa3e806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec036000 tx=0x7fa3ec03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.614+0000 7fa3e37fe640 1 -- 172.21.15.94:0/2303083731 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3f4058f00 con 0x7fa3fc1538c0 2024-03-06T07:20:27.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:27.872+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa3fc0c5a70 con 0x7fa3fc1538c0 2024-03-06T07:20:28.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.486+0000 7fa3e37fe640 1 -- 172.21.15.94:0/2303083731 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3f4058800 con 0x7fa3fc1538c0 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa3e806b4d0 msgr2=0x7fa3e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa3e806b4d0 0x7fa3e806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec036000 tx=0x7fa3ec03c040 comp rx=0 tx=0).stop 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc1538c0 msgr2=0x7fa3fc14fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc1538c0 0x7fa3fc14fd70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c7d0 tx=0x7fa3f400cc90 comp rx=0 tx=0).stop 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 shutdown_connections 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa3e806b4d0 0x7fa3e806d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3fc1538c0 0x7fa3fc14fd70 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3fc0ab310 0x7fa3fc14f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 --2- 172.21.15.94:0/2303083731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3fc0a97c0 0x7fa3fc156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 >> 172.21.15.94:0/2303083731 conn(0x7fa3fc0c6710 msgr2=0x7fa3fc0c1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:28.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 shutdown_connections 2024-03-06T07:20:28.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.489+0000 7fa4016c3640 1 -- 172.21.15.94:0/2303083731 wait complete. 2024-03-06T07:20:28.495 INFO:tasks.ceph.mds.c.smithi094.stderr: -633> 2024-03-06T07:20:09.291+0000 7f8e6b537640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:20:28.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:20:28.522 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:20:28.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.609+0000 7f41dacc9640 1 Processor -- start 2024-03-06T07:20:28.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.609+0000 7f41dacc9640 1 -- start start 2024-03-06T07:20:28.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.609+0000 7f41dacc9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f41d40c2200 0x7f41d40c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.609+0000 7f41dacc9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41d4155b70 0x7f41d415fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.609+0000 7f41dacc9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d41596c0 con 0x7f41d40a6cc0 2024-03-06T07:20:28.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.609+0000 7f41dacc9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d4158ab0 con 0x7f41d4155b70 2024-03-06T07:20:28.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.609+0000 7f41dacc9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d4158d90 con 0x7f41d40c2200 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d9266640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41d4155b70 0x7f41d415fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d9266640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41d4155b70 0x7f41d415fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53100/0 (socket says 172.21.15.94:53100) 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d9266640 1 -- 172.21.15.94:0/4194220946 learned_addr learned my addr 172.21.15.94:0/4194220946 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d8a65640 1 --2- 172.21.15.94:0/4194220946 >> v2:172.21.15.145:3300/0 conn(0x7f41d40c2200 0x7f41d40c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d9266640 1 -- 172.21.15.94:0/4194220946 >> v2:172.21.15.145:3300/0 conn(0x7f41d40c2200 msgr2=0x7f41d40c25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d9266640 1 --2- 172.21.15.94:0/4194220946 >> v2:172.21.15.145:3300/0 conn(0x7f41d40c2200 0x7f41d40c25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d9266640 1 -- 172.21.15.94:0/4194220946 >> v1:172.21.15.145:6789/0 conn(0x7f41d40a6cc0 legacy=0x7f41d4155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.610+0000 7f41d9266640 1 -- 172.21.15.94:0/4194220946 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d4159070 con 0x7f41d4155b70 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41d9266640 1 --2- 172.21.15.94:0/4194220946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d4155b70 0x7f41d415fad0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f41d406d190 tx=0x7f41cc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd25bbd23c0875ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41d37fe640 1 -- 172.21.15.94:0/4194220946 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc02c9f0 con 0x7f41d4155b70 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41d37fe640 1 -- 172.21.15.94:0/4194220946 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc02cb90 con 0x7f41d4155b70 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41d37fe640 1 -- 172.21.15.94:0/4194220946 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0343f0 con 0x7f41d4155b70 2024-03-06T07:20:28.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 -- 172.21.15.94:0/4194220946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d4155b70 msgr2=0x7f41d415fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 --2- 172.21.15.94:0/4194220946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d4155b70 0x7f41d415fad0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f41d406d190 tx=0x7f41cc02be50 comp rx=0 tx=0).stop 2024-03-06T07:20:28.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 -- 172.21.15.94:0/4194220946 shutdown_connections 2024-03-06T07:20:28.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 --2- 172.21.15.94:0/4194220946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d4155b70 0x7f41d415fad0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 --2- 172.21.15.94:0/4194220946 >> v2:172.21.15.145:3300/0 conn(0x7f41d40c2200 0x7f41d40c25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 -- 172.21.15.94:0/4194220946 >> 172.21.15.94:0/4194220946 conn(0x7f41d40c4300 msgr2=0x7f41d40c46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:28.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 -- 172.21.15.94:0/4194220946 shutdown_connections 2024-03-06T07:20:28.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.611+0000 7f41dacc9640 1 -- 172.21.15.94:0/4194220946 wait complete. 2024-03-06T07:20:28.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.612+0000 7f41dacc9640 1 Processor -- start 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.612+0000 7f41dacc9640 1 -- start start 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41dacc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41d40a6cc0 0x7f41d406f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41dacc9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d40c2200 0x7f41d4072010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41dacc9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41d4155b70 0x7f41d4072550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41dacc9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4073580 con 0x7f41d40c2200 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41dacc9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d406be10 con 0x7f41d4155b70 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41dacc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4158740 con 0x7f41d40a6cc0 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d40c2200 0x7f41d4072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d40c2200 0x7f41d4072010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53102/0 (socket says 172.21.15.94:53102) 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41d3fff640 1 -- 172.21.15.94:0/1084836638 learned_addr learned my addr 172.21.15.94:0/1084836638 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41d8a65640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41d40a6cc0 0x7f41d406f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.613+0000 7f41d9266640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41d4155b70 0x7f41d4072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d3fff640 1 -- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41d40a6cc0 msgr2=0x7f41d406f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d3fff640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41d40a6cc0 0x7f41d406f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d3fff640 1 -- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41d4155b70 msgr2=0x7f41d4072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d3fff640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41d4155b70 0x7f41d4072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d3fff640 1 -- 172.21.15.94:0/1084836638 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d41ee250 con 0x7f41d40c2200 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d9266640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41d4155b70 0x7f41d4072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d3fff640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d40c2200 0x7f41d4072010 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c460 tx=0x7f41c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d17fa640 1 -- 172.21.15.94:0/1084836638 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8016020 con 0x7f41d40c2200 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d17fa640 1 -- 172.21.15.94:0/1084836638 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c8012870 con 0x7f41d40c2200 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d41ee530 con 0x7f41d40c2200 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41d17fa640 1 -- 172.21.15.94:0/1084836638 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8012c60 con 0x7f41d40c2200 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.614+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d41eeaf0 con 0x7f41d40c2200 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.615+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d404f330 con 0x7f41d40c2200 2024-03-06T07:20:28.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.615+0000 7f41d17fa640 1 -- 172.21.15.94:0/1084836638 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41c8019020 con 0x7f41d40c2200 2024-03-06T07:20:28.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.616+0000 7f41d17fa640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41c006b4d0 0x7f41c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:28.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.616+0000 7f41d17fa640 1 -- 172.21.15.94:0/1084836638 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7301+0+0 (secure 0 0 0) 0x7f41c808a430 con 0x7f41d40c2200 2024-03-06T07:20:28.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.616+0000 7f41d8a65640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41c006b4d0 0x7f41c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:28.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.617+0000 7f41d8a65640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41c006b4d0 0x7f41c006d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006fd0 tx=0x7f41c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:28.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.620+0000 7f41d17fa640 1 -- 172.21.15.94:0/1084836638 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41c8020030 con 0x7f41d40c2200 2024-03-06T07:20:28.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:28.792+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f41d406bbf0 con 0x7f41d40c2200 2024-03-06T07:20:29.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.666+0000 7f41d17fa640 1 -- 172.21.15.94:0/1084836638 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7f41c8058d00 con 0x7f41d40c2200 2024-03-06T07:20:29.668 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:20:29.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41c006b4d0 msgr2=0x7f41c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41c006b4d0 0x7f41c006d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006fd0 tx=0x7f41c4008040 comp rx=0 tx=0).stop 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d40c2200 msgr2=0x7f41d4072010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d40c2200 0x7f41d4072010 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c460 tx=0x7f41c800c920 comp rx=0 tx=0).stop 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 shutdown_connections 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41c006b4d0 0x7f41c006d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41d4155b70 0x7f41d4072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41d40c2200 0x7f41d4072010 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 --2- 172.21.15.94:0/1084836638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41d40a6cc0 0x7f41d406f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.670+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 >> 172.21.15.94:0/1084836638 conn(0x7f41d40c4300 msgr2=0x7f41d40c0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.671+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 shutdown_connections 2024-03-06T07:20:29.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.671+0000 7f41dacc9640 1 -- 172.21.15.94:0/1084836638 wait complete. 2024-03-06T07:20:29.686 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:20:29.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.836+0000 7fc4b33ae640 1 Processor -- start 2024-03-06T07:20:29.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b33ae640 1 -- start start 2024-03-06T07:20:29.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b33ae640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc4ac0c5e40 0x7fc4ac153940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b33ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac0c46d0 0x7fc4ac0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b33ae640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac153e80 con 0x7fc4ac072fc0 2024-03-06T07:20:29.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b33ae640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac0c6220 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b33ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac070210 con 0x7fc4ac0c46d0 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b114a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc4ac0c5e40 0x7fc4ac153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b114a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc4ac0c5e40 0x7fc4ac153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53118/0 (socket says 172.21.15.94:53118) 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.837+0000 7fc4b114a640 1 -- 172.21.15.94:0/4197954461 learned_addr learned my addr 172.21.15.94:0/4197954461 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.838+0000 7fc4b194b640 1 --2- 172.21.15.94:0/4197954461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac0c46d0 0x7fc4ac0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.838+0000 7fc4b114a640 1 -- 172.21.15.94:0/4197954461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac0c46d0 msgr2=0x7fc4ac0c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.838+0000 7fc4b114a640 1 --2- 172.21.15.94:0/4197954461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac0c46d0 0x7fc4ac0c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.838+0000 7fc4b114a640 1 -- 172.21.15.94:0/4197954461 >> v1:172.21.15.145:6789/0 conn(0x7fc4ac072fc0 legacy=0x7fc4ac0724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.838+0000 7fc4b114a640 1 -- 172.21.15.94:0/4197954461 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0704d0 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b114a640 1 --2- 172.21.15.94:0/4197954461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac153940 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009e50 tx=0x7fc4a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160bb219db8521c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4a3fff640 1 -- 172.21.15.94:0/4197954461 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a802ca70 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4a3fff640 1 -- 172.21.15.94:0/4197954461 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a802cc10 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4a3fff640 1 -- 172.21.15.94:0/4197954461 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a80343f0 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 -- 172.21.15.94:0/4197954461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 msgr2=0x7fc4ac153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/4197954461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac153940 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009e50 tx=0x7fc4a802bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 -- 172.21.15.94:0/4197954461 shutdown_connections 2024-03-06T07:20:29.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/4197954461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac0c46d0 0x7fc4ac0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/4197954461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac153940 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 -- 172.21.15.94:0/4197954461 >> 172.21.15.94:0/4197954461 conn(0x7fc4ac0c88f0 msgr2=0x7fc4ac0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 -- 172.21.15.94:0/4197954461 shutdown_connections 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.839+0000 7fc4b33ae640 1 -- 172.21.15.94:0/4197954461 wait complete. 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.840+0000 7fc4b33ae640 1 Processor -- start 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.840+0000 7fc4b33ae640 1 -- start start 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b33ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac072fc0 0x7fc4ac0a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b33ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4ac0c46d0 0x7fc4ac1ebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b33ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac1ec510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b33ae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac06fbd0 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b33ae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac06f8f0 con 0x7fc4ac0c46d0 2024-03-06T07:20:29.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b33ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac072be0 con 0x7fc4ac072fc0 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b194b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac1ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b194b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac1ec510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53120/0 (socket says 172.21.15.94:53120) 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b194b640 1 -- 172.21.15.94:0/2140436011 learned_addr learned my addr 172.21.15.94:0/2140436011 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b114a640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac072fc0 0x7fc4ac0a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.841+0000 7fc4b0949640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4ac0c46d0 0x7fc4ac1ebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b194b640 1 -- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac072fc0 msgr2=0x7fc4ac0a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b194b640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac072fc0 0x7fc4ac0a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b194b640 1 -- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4ac0c46d0 msgr2=0x7fc4ac1ebfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b194b640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4ac0c46d0 0x7fc4ac1ebfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b194b640 1 -- 172.21.15.94:0/2140436011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac156810 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b194b640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac1ec510 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400d600 tx=0x7fc4a400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4a1ffb640 1 -- 172.21.15.94:0/2140436011 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40125f0 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4a1ffb640 1 -- 172.21.15.94:0/2140436011 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a4019070 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac156250 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4a1ffb640 1 -- 172.21.15.94:0/2140436011 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40129f0 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.842+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac1ebab0 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.843+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac04f330 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.844+0000 7fc4a1ffb640 1 -- 172.21.15.94:0/2140436011 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4a4012b90 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.844+0000 7fc4a1ffb640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49006b4d0 0x7fc49006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:29.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.844+0000 7fc4a1ffb640 1 -- 172.21.15.94:0/2140436011 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6863+0+0 (secure 0 0 0) 0x7fc4a408a8f0 con 0x7fc4ac0c5e40 2024-03-06T07:20:29.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.844+0000 7fc4b114a640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49006b4d0 0x7fc49006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:29.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.845+0000 7fc4b114a640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49006b4d0 0x7fc49006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8035000 tx=0x7fc4a8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:29.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:29.848+0000 7fc4a1ffb640 1 -- 172.21.15.94:0/2140436011 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4a4058cb0 con 0x7fc4ac0c5e40 2024-03-06T07:20:30.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.024+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc4ac06f3f0 con 0x7fc4ac0c5e40 2024-03-06T07:20:30.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.673+0000 7fc4a1ffb640 1 -- 172.21.15.94:0/2140436011 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc4a405cd50 con 0x7fc4ac0c5e40 2024-03-06T07:20:30.675 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:20:30.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.676+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49006b4d0 msgr2=0x7fc49006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.676+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49006b4d0 0x7fc49006d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8035000 tx=0x7fc4a8030000 comp rx=0 tx=0).stop 2024-03-06T07:20:30.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.676+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 msgr2=0x7fc4ac1ec510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.676+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac1ec510 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400d600 tx=0x7fc4a400dac0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.676+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 shutdown_connections 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.676+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49006b4d0 0x7fc49006d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.677+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4ac0c5e40 0x7fc4ac1ec510 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.677+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4ac0c46d0 0x7fc4ac1ebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.677+0000 7fc4b33ae640 1 --2- 172.21.15.94:0/2140436011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4ac072fc0 0x7fc4ac0a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.677+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 >> 172.21.15.94:0/2140436011 conn(0x7fc4ac0c88f0 msgr2=0x7fc4ac0c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.677+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 shutdown_connections 2024-03-06T07:20:30.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.677+0000 7fc4b33ae640 1 -- 172.21.15.94:0/2140436011 wait complete. 2024-03-06T07:20:30.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:30.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.793+0000 7f5deeb81640 1 Processor -- start 2024-03-06T07:20:30.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.793+0000 7f5deeb81640 1 -- start start 2024-03-06T07:20:30.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.793+0000 7f5deeb81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5de814f790 0x7f5de81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:30.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.793+0000 7f5deeb81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5de8155a40 0x7f5de8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:30.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.793+0000 7f5deeb81640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de80c28d0 con 0x7f5de80c1aa0 2024-03-06T07:20:30.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.793+0000 7f5deeb81640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de80c2680 con 0x7f5de814f790 2024-03-06T07:20:30.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.793+0000 7f5deeb81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de80c2bb0 con 0x7f5de8155a40 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5dec91d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5de814f790 0x7f5de81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5dec91d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5de814f790 0x7f5de81529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53132/0 (socket says 172.21.15.94:53132) 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5dec91d640 1 -- 172.21.15.94:0/1897960428 learned_addr learned my addr 172.21.15.94:0/1897960428 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5de7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5de80c1aa0 0x7f5de80c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45934/0 (socket says 172.21.15.94:45934) 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5ded11e640 1 --2- 172.21.15.94:0/1897960428 >> v2:172.21.15.145:3300/0 conn(0x7f5de8155a40 0x7f5de8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5dec91d640 1 -- 172.21.15.94:0/1897960428 >> v2:172.21.15.145:3300/0 conn(0x7f5de8155a40 msgr2=0x7f5de8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5dec91d640 1 --2- 172.21.15.94:0/1897960428 >> v2:172.21.15.145:3300/0 conn(0x7f5de8155a40 0x7f5de8155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5dec91d640 1 -- 172.21.15.94:0/1897960428 >> v1:172.21.15.94:6789/0 conn(0x7f5de80c1aa0 legacy=0x7f5de80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5dec91d640 1 -- 172.21.15.94:0/1897960428 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de80c2e90 con 0x7f5de814f790 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.794+0000 7f5de77fe640 1 -- 172.21.15.94:0/1897960428 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225668730 0 0) 0x7f5de80c28d0 con 0x7f5de80c1aa0 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5dec91d640 1 --2- 172.21.15.94:0/1897960428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de814f790 0x7f5de81529d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5de806c960 tx=0x7f5ddc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72124ca8d686611b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5de77fe640 1 -- 172.21.15.94:0/1897960428 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc034070 con 0x7f5de814f790 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5de77fe640 1 -- 172.21.15.94:0/1897960428 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc030860 con 0x7f5de814f790 2024-03-06T07:20:30.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5de77fe640 1 -- 172.21.15.94:0/1897960428 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc030c30 con 0x7f5de814f790 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 -- 172.21.15.94:0/1897960428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de814f790 msgr2=0x7f5de81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 --2- 172.21.15.94:0/1897960428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de814f790 0x7f5de81529d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5de806c960 tx=0x7f5ddc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 -- 172.21.15.94:0/1897960428 shutdown_connections 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 --2- 172.21.15.94:0/1897960428 >> v2:172.21.15.145:3300/0 conn(0x7f5de8155a40 0x7f5de8155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 --2- 172.21.15.94:0/1897960428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de814f790 0x7f5de81529d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 -- 172.21.15.94:0/1897960428 >> 172.21.15.94:0/1897960428 conn(0x7f5de80c6710 msgr2=0x7f5de80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 -- 172.21.15.94:0/1897960428 shutdown_connections 2024-03-06T07:20:30.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.795+0000 7f5deeb81640 1 -- 172.21.15.94:0/1897960428 wait complete. 2024-03-06T07:20:30.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.796+0000 7f5deeb81640 1 Processor -- start 2024-03-06T07:20:30.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.796+0000 7f5deeb81640 1 -- start start 2024-03-06T07:20:30.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5deeb81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de80c1aa0 0x7f5de814a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5deeb81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5de814f790 0x7f5de81e89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5deeb81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5de8155a40 0x7f5de814dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5deeb81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de80c0d60 con 0x7f5de80c1aa0 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5deeb81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de8156390 con 0x7f5de814f790 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5deeb81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de80c5e30 con 0x7f5de8155a40 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5dec91d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de80c1aa0 0x7f5de814a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5dec91d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de80c1aa0 0x7f5de814a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53142/0 (socket says 172.21.15.94:53142) 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5dec91d640 1 -- 172.21.15.94:0/956805326 learned_addr learned my addr 172.21.15.94:0/956805326 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5de7fff640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5de814f790 0x7f5de81e89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.797+0000 7f5ded11e640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5de8155a40 0x7f5de814dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5dec91d640 1 -- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5de8155a40 msgr2=0x7f5de814dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5dec91d640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5de8155a40 0x7f5de814dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5dec91d640 1 -- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5de814f790 msgr2=0x7f5de81e89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5dec91d640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5de814f790 0x7f5de81e89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5dec91d640 1 -- 172.21.15.94:0/956805326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8151d40 con 0x7f5de80c1aa0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5dec91d640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de80c1aa0 0x7f5de814a200 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc000c00 tx=0x7f5ddc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5de57fa640 1 -- 172.21.15.94:0/956805326 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc048070 con 0x7f5de80c1aa0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5de57fa640 1 -- 172.21.15.94:0/956805326 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc030660 con 0x7f5de80c1aa0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de806de20 con 0x7f5de80c1aa0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5de57fa640 1 -- 172.21.15.94:0/956805326 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc034040 con 0x7f5de80c1aa0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.798+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de806e3e0 con 0x7f5de80c1aa0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.799+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de804f330 con 0x7f5de80c1aa0 2024-03-06T07:20:30.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.799+0000 7f5de57fa640 1 -- 172.21.15.94:0/956805326 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ddc04b070 con 0x7f5de80c1aa0 2024-03-06T07:20:30.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.800+0000 7f5de57fa640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dcc06b4d0 0x7f5dcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:30.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.800+0000 7f5de57fa640 1 -- 172.21.15.94:0/956805326 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f5ddc0ac320 con 0x7f5de80c1aa0 2024-03-06T07:20:30.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.800+0000 7f5de7fff640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dcc06b4d0 0x7f5dcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:30.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.801+0000 7f5de7fff640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dcc06b4d0 0x7f5dcc06d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5de80ab4e0 tx=0x7f5dd4007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:30.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.804+0000 7f5de57fa640 1 -- 172.21.15.94:0/956805326 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ddc07aa10 con 0x7f5de80c1aa0 2024-03-06T07:20:30.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.979+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5de8156170 con 0x7f5de80c1aa0 2024-03-06T07:20:30.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.980+0000 7f5de57fa640 1 -- 172.21.15.94:0/956805326 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f5ddc07eab0 con 0x7f5de80c1aa0 2024-03-06T07:20:30.982 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:30.982 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":62,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:30.667810+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6834/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6835/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.94:6833/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.145:6837/3125091562":"2024-03-07T07:20:16.601064+0000","172.21.15.94:6832/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/3125091562":"2024-03-07T07:20:16.601064+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:30.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dcc06b4d0 msgr2=0x7f5dcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dcc06b4d0 0x7f5dcc06d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5de80ab4e0 tx=0x7f5dd4007d50 comp rx=0 tx=0).stop 2024-03-06T07:20:30.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de80c1aa0 msgr2=0x7f5de814a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:30.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de80c1aa0 0x7f5de814a200 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc000c00 tx=0x7f5ddc004730 comp rx=0 tx=0).stop 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 shutdown_connections 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5dcc06b4d0 0x7f5dcc06d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5de8155a40 0x7f5de814dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5de814f790 0x7f5de81e89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 --2- 172.21.15.94:0/956805326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5de80c1aa0 0x7f5de814a200 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 >> 172.21.15.94:0/956805326 conn(0x7f5de80c6710 msgr2=0x7f5de80c30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 shutdown_connections 2024-03-06T07:20:30.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:30.982+0000 7f5deeb81640 1 -- 172.21.15.94:0/956805326 wait complete. 2024-03-06T07:20:30.995 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:20:06.055145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:20:06.580638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:20:30.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:31.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.097+0000 7f333093e640 1 Processor -- start 2024-03-06T07:20:31.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.097+0000 7f333093e640 1 -- start start 2024-03-06T07:20:31.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.097+0000 7f333093e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f332c14ef70 0x7f332c06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.097+0000 7f333093e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f332c06cad0 0x7f332c154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.097+0000 7f333093e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c0c2790 con 0x7f332c0c1820 2024-03-06T07:20:31.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.097+0000 7f333093e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332c0c3090 con 0x7f332c14ef70 2024-03-06T07:20:31.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.097+0000 7f333093e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332c14f350 con 0x7f332c06cad0 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f332c14ef70 0x7f332c06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f332c14ef70 0x7f332c06c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53152/0 (socket says 172.21.15.94:53152) 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332a59c640 1 -- 172.21.15.94:0/1272172674 learned_addr learned my addr 172.21.15.94:0/1272172674 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f3323fff640 1 --1- 172.21.15.94:0/1272172674 >> v1:172.21.15.94:6789/0 conn(0x7f332c0c1820 0x7f332c0c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45948/0 (socket says 172.21.15.94:45948) 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332ad9d640 1 --2- 172.21.15.94:0/1272172674 >> v2:172.21.15.145:3300/0 conn(0x7f332c06cad0 0x7f332c154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332a59c640 1 -- 172.21.15.94:0/1272172674 >> v2:172.21.15.145:3300/0 conn(0x7f332c06cad0 msgr2=0x7f332c154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332a59c640 1 --2- 172.21.15.94:0/1272172674 >> v2:172.21.15.145:3300/0 conn(0x7f332c06cad0 0x7f332c154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f3329d9b640 1 -- 172.21.15.94:0/1272172674 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090443872 0 0) 0x7f332c0c2790 con 0x7f332c0c1820 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332a59c640 1 -- 172.21.15.94:0/1272172674 >> v1:172.21.15.94:6789/0 conn(0x7f332c0c1820 legacy=0x7f332c0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332a59c640 1 -- 172.21.15.94:0/1272172674 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c06cf40 con 0x7f332c14ef70 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.098+0000 7f332ad9d640 1 --2- 172.21.15.94:0/1272172674 >> v2:172.21.15.145:3300/0 conn(0x7f332c06cad0 0x7f332c154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f332a59c640 1 --2- 172.21.15.94:0/1272172674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c14ef70 0x7f332c06c1f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f332c0c76b0 tx=0x7f331c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08ad0d7f1025d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f3329d9b640 1 -- 172.21.15.94:0/1272172674 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c03b070 con 0x7f332c14ef70 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f3329d9b640 1 -- 172.21.15.94:0/1272172674 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c036440 con 0x7f332c14ef70 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f3329d9b640 1 -- 172.21.15.94:0/1272172674 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c036810 con 0x7f332c14ef70 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 -- 172.21.15.94:0/1272172674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c14ef70 msgr2=0x7f332c06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 --2- 172.21.15.94:0/1272172674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c14ef70 0x7f332c06c1f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f332c0c76b0 tx=0x7f331c02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 -- 172.21.15.94:0/1272172674 shutdown_connections 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 --2- 172.21.15.94:0/1272172674 >> v2:172.21.15.145:3300/0 conn(0x7f332c06cad0 0x7f332c154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 --2- 172.21.15.94:0/1272172674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c14ef70 0x7f332c06c1f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 -- 172.21.15.94:0/1272172674 >> 172.21.15.94:0/1272172674 conn(0x7f332c0c6710 msgr2=0x7f332c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 -- 172.21.15.94:0/1272172674 shutdown_connections 2024-03-06T07:20:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.099+0000 7f333093e640 1 -- 172.21.15.94:0/1272172674 wait complete. 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 Processor -- start 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 -- start start 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f332c06cad0 0x7f332c1eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c0c1820 0x7f332c1ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f332c14ef70 0x7f332c14d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c155750 con 0x7f332c0c1820 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0c24b0 con 0x7f332c06cad0 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f333093e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c155170 con 0x7f332c14ef70 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f3323fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c0c1820 0x7f332c1ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f3323fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c0c1820 0x7f332c1ee8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53160/0 (socket says 172.21.15.94:53160) 2024-03-06T07:20:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.101+0000 7f3323fff640 1 -- 172.21.15.94:0/864615103 learned_addr learned my addr 172.21.15.94:0/864615103 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f332a59c640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f332c06cad0 0x7f332c1eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f332ad9d640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f332c14ef70 0x7f332c14d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3323fff640 1 -- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f332c14ef70 msgr2=0x7f332c14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3323fff640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f332c14ef70 0x7f332c14d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3323fff640 1 -- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f332c06cad0 msgr2=0x7f332c1eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3323fff640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f332c06cad0 0x7f332c1eb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3323fff640 1 -- 172.21.15.94:0/864615103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c0c03c0 con 0x7f332c0c1820 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3323fff640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c0c1820 0x7f332c1ee8d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f331400c500 tx=0x7f331400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3322ffd640 1 -- 172.21.15.94:0/864615103 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314018070 con 0x7f332c0c1820 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3322ffd640 1 -- 172.21.15.94:0/864615103 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3314007580 con 0x7f332c0c1820 2024-03-06T07:20:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f3322ffd640 1 -- 172.21.15.94:0/864615103 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314004a20 con 0x7f332c0c1820 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332c0c06a0 con 0x7f332c0c1820 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.102+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c1ead40 con 0x7f332c0c1820 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.103+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c04f330 con 0x7f332c0c1820 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.103+0000 7f3322ffd640 1 -- 172.21.15.94:0/864615103 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f331401b070 con 0x7f332c0c1820 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.104+0000 7f3322ffd640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f330c06b4d0 0x7f330c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.104+0000 7f3322ffd640 1 -- 172.21.15.94:0/864615103 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f3314089820 con 0x7f332c0c1820 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.104+0000 7f332a59c640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f330c06b4d0 0x7f330c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.105+0000 7f332a59c640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f330c06b4d0 0x7f330c06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f331c038000 tx=0x7f331c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.108+0000 7f3322ffd640 1 -- 172.21.15.94:0/864615103 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33140583b0 con 0x7f332c0c1820 2024-03-06T07:20:31.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.281+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f332c0c1110 con 0x7f332c0c1820 2024-03-06T07:20:31.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.281+0000 7f3322ffd640 1 -- 172.21.15.94:0/864615103 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3314057cb0 con 0x7f332c0c1820 2024-03-06T07:20:31.283 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f330c06b4d0 msgr2=0x7f330c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f330c06b4d0 0x7f330c06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f331c038000 tx=0x7f331c008040 comp rx=0 tx=0).stop 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c0c1820 msgr2=0x7f332c1ee8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c0c1820 0x7f332c1ee8d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f331400c500 tx=0x7f331400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 shutdown_connections 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f330c06b4d0 0x7f330c06d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f332c14ef70 0x7f332c14d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f332c0c1820 0x7f332c1ee8d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 --2- 172.21.15.94:0/864615103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f332c06cad0 0x7f332c1eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.283+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 >> 172.21.15.94:0/864615103 conn(0x7f332c0c6710 msgr2=0x7f332c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.284+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 shutdown_connections 2024-03-06T07:20:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.284+0000 7f333093e640 1 -- 172.21.15.94:0/864615103 wait complete. 2024-03-06T07:20:31.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:20:31.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.397+0000 7f98676c9640 1 Processor -- start 2024-03-06T07:20:31.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.397+0000 7f98676c9640 1 -- start start 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f98676c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f98600c5990 0x7f98600c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f98676c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98600c3ef0 con 0x7f98600ab170 2024-03-06T07:20:31.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f98676c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98601556d0 con 0x7f986006c900 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f98676c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98601559b0 con 0x7f98600c5990 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f9864c64640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f98600ab170 0x7f986006c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45950/0 (socket says 172.21.15.94:45950) 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f9864c64640 1 -- 172.21.15.94:0/252082670 learned_addr learned my addr 172.21.15.94:0/252082670 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f9865465640 1 --2- 172.21.15.94:0/252082670 >> v2:172.21.15.145:3300/0 conn(0x7f98600c5990 0x7f98600c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933963592 0 0) 0x7f98600c3ef0 con 0x7f98600ab170 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.398+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f984c003520 con 0x7f98600ab170 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 800295905 0 0) 0x7f984c003520 con 0x7f98600ab170 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 >> v2:172.21.15.145:3300/0 conn(0x7f98600c5990 msgr2=0x7f98600c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 --2- 172.21.15.94:0/252082670 >> v2:172.21.15.145:3300/0 conn(0x7f98600c5990 0x7f98600c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 >> v1:172.21.15.145:6789/0 conn(0x7f986006c900 legacy=0x7f98600a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9860155c90 con 0x7f98600ab170 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9850001670 con 0x7f98600ab170 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 >> v1:172.21.15.94:6789/0 conn(0x7f98600ab170 legacy=0x7f986006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f984c004800 0x7f984c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f984c0051c0 0x7f984c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98601559b0 con 0x7f984c0039b0 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c00c060 con 0x7f984c004800 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c00c260 con 0x7f984c0051c0 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865465640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9864c64640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f984c004800 0x7f984c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865c66640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f984c0051c0 0x7f984c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865465640 1 -- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f984c0051c0 msgr2=0x7f984c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:31.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865465640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f984c0051c0 0x7f984c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865465640 1 -- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f984c004800 msgr2=0x7f984c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865465640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f984c004800 0x7f984c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865465640 1 -- 172.21.15.94:0/252082670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c003520 con 0x7f984c0039b0 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.399+0000 7f9865c66640 1 -- 172.21.15.94:0/252082670 reap_dead start 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.400+0000 7f9865465640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f985c009ac0 tx=0x7f984c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8cb80c60dd6431 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.400+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c039070 con 0x7f984c0039b0 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.400+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c02c6d0 con 0x7f984c0039b0 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.400+0000 7f9857fff640 1 -- 172.21.15.94:0/252082670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c009690 con 0x7f984c0039b0 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.401+0000 7f98676c9640 1 -- 172.21.15.94:0/252082670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 msgr2=0x7f984c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.401+0000 7f98676c9640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f985c009ac0 tx=0x7f984c005830 comp rx=0 tx=0).stop 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.401+0000 7f98676c9640 1 -- 172.21.15.94:0/252082670 shutdown_connections 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.401+0000 7f98676c9640 1 --2- 172.21.15.94:0/252082670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.401+0000 7f98676c9640 1 -- 172.21.15.94:0/252082670 >> 172.21.15.94:0/252082670 conn(0x7f98600c6710 msgr2=0x7f98600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.401+0000 7f98676c9640 1 -- 172.21.15.94:0/252082670 shutdown_connections 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.401+0000 7f98676c9640 1 -- 172.21.15.94:0/252082670 wait complete. 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 Processor -- start 2024-03-06T07:20:31.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 -- start start 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f9860154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98600c0990 0x7f986014d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98601eaa80 0x7f986014d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9860156610 con 0x7f984c0039b0 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9860156490 con 0x7f98601eaa80 2024-03-06T07:20:31.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.402+0000 7f98676c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98600c3af0 con 0x7f98600c0990 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f9860154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f9860154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53174/0 (socket says 172.21.15.94:53174) 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 -- 172.21.15.94:0/1088043477 learned_addr learned my addr 172.21.15.94:0/1088043477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 -- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98600c0990 msgr2=0x7f986014d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98600c0990 0x7f986014d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 -- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98601eaa80 msgr2=0x7f986014d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98601eaa80 0x7f986014d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 -- 172.21.15.94:0/1088043477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986014db60 con 0x7f984c0039b0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9865465640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f9860154da0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f985c009ac0 tx=0x7f985c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9855ffb640 1 -- 172.21.15.94:0/1088043477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c039050 con 0x7f984c0039b0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9855ffb640 1 -- 172.21.15.94:0/1088043477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c045070 con 0x7f984c0039b0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98601eb020 con 0x7f984c0039b0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f9855ffb640 1 -- 172.21.15.94:0/1088043477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c0044e0 con 0x7f984c0039b0 2024-03-06T07:20:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.403+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98601eb5e0 con 0x7f984c0039b0 2024-03-06T07:20:31.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.404+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986004f330 con 0x7f984c0039b0 2024-03-06T07:20:31.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.405+0000 7f9855ffb640 1 -- 172.21.15.94:0/1088043477 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f985c003a30 con 0x7f984c0039b0 2024-03-06T07:20:31.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.405+0000 7f9855ffb640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f984406b500 0x7f984406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.405+0000 7f9855ffb640 1 -- 172.21.15.94:0/1088043477 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f985c0ac020 con 0x7f984c0039b0 2024-03-06T07:20:31.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.405+0000 7f9864c64640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f984406b500 0x7f984406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.406+0000 7f9864c64640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f984406b500 0x7f984406d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9850001b60 tx=0x7f9850001ad0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.410+0000 7f9855ffb640 1 -- 172.21.15.94:0/1088043477 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f985c04e020 con 0x7f984c0039b0 2024-03-06T07:20:31.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.587+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f986014f920 con 0x7f984c0039b0 2024-03-06T07:20:31.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.587+0000 7f9855ffb640 1 -- 172.21.15.94:0/1088043477 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v62) v1 ==== 105+0+11 (secure 0 0 0) 0x7f985c07ac40 con 0x7f984c0039b0 2024-03-06T07:20:31.589 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:20:31.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f984406b500 msgr2=0x7f984406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f984406b500 0x7f984406d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9850001b60 tx=0x7f9850001ad0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 msgr2=0x7f9860154da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f9860154da0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f985c009ac0 tx=0x7f985c004730 comp rx=0 tx=0).stop 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 shutdown_connections 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f984406b500 0x7f984406d9c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98601eaa80 0x7f986014d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98600c0990 0x7f986014d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 --2- 172.21.15.94:0/1088043477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f984c0039b0 0x7f9860154da0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 >> 172.21.15.94:0/1088043477 conn(0x7f98600c6710 msgr2=0x7f986006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 shutdown_connections 2024-03-06T07:20:31.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.589+0000 7f98676c9640 1 -- 172.21.15.94:0/1088043477 wait complete. 2024-03-06T07:20:31.602 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:20:31.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.704+0000 7f841d365640 1 Processor -- start 2024-03-06T07:20:31.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.704+0000 7f841d365640 1 -- start start 2024-03-06T07:20:31.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f841d365640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84181529d0 0x7f84180c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f841d365640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8418152db0 con 0x7f84180c57e0 2024-03-06T07:20:31.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f841d365640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84181590f0 con 0x7f8418072cd0 2024-03-06T07:20:31.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f841d365640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84180c6dd0 con 0x7f84181529d0 2024-03-06T07:20:31.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f84167fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f84180c57e0 0x7f8418155120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45954/0 (socket says 172.21.15.94:45954) 2024-03-06T07:20:31.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f84167fc640 1 -- 172.21.15.94:0/2055182654 learned_addr learned my addr 172.21.15.94:0/2055182654 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8416ffd640 1 --2- 172.21.15.94:0/2055182654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84181529d0 0x7f84180c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8415ffb640 1 -- 172.21.15.94:0/2055182654 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176542326 0 0) 0x7f840c001110 con 0x7f84180c57e0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8415ffb640 1 -- 172.21.15.94:0/2055182654 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400003520 con 0x7f84180c57e0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8415ffb640 1 -- 172.21.15.94:0/2055182654 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158060877 0 0) 0x7f8410001490 con 0x7f8418072cd0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8415ffb640 1 -- 172.21.15.94:0/2055182654 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c001110 con 0x7f8418072cd0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8416ffd640 1 -- 172.21.15.94:0/2055182654 >> v1:172.21.15.145:6789/0 conn(0x7f8418072cd0 legacy=0x7f841815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8416ffd640 1 -- 172.21.15.94:0/2055182654 >> v1:172.21.15.94:6789/0 conn(0x7f84180c57e0 legacy=0x7f8418155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.705+0000 7f8416ffd640 1 -- 172.21.15.94:0/2055182654 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418159650 con 0x7f84181529d0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.706+0000 7f8416ffd640 1 --2- 172.21.15.94:0/2055182654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84181529d0 0x7f84180c52a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84180c6f50 tx=0x7f840802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=484d9f151ed90bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.706+0000 7f8415ffb640 1 -- 172.21.15.94:0/2055182654 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408031760 con 0x7f84181529d0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.706+0000 7f8415ffb640 1 -- 172.21.15.94:0/2055182654 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8408037070 con 0x7f84181529d0 2024-03-06T07:20:31.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.706+0000 7f8415ffb640 1 -- 172.21.15.94:0/2055182654 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408031b20 con 0x7f84181529d0 2024-03-06T07:20:31.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.707+0000 7f841d365640 1 -- 172.21.15.94:0/2055182654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84181529d0 msgr2=0x7f84180c52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.707+0000 7f841d365640 1 --2- 172.21.15.94:0/2055182654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84181529d0 0x7f84180c52a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84180c6f50 tx=0x7f840802b800 comp rx=0 tx=0).stop 2024-03-06T07:20:31.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.707+0000 7f841d365640 1 -- 172.21.15.94:0/2055182654 shutdown_connections 2024-03-06T07:20:31.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.707+0000 7f841d365640 1 --2- 172.21.15.94:0/2055182654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84181529d0 0x7f84180c52a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.707+0000 7f841d365640 1 -- 172.21.15.94:0/2055182654 >> 172.21.15.94:0/2055182654 conn(0x7f84180aa630 msgr2=0x7f84180aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.707+0000 7f841d365640 1 -- 172.21.15.94:0/2055182654 shutdown_connections 2024-03-06T07:20:31.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.707+0000 7f841d365640 1 -- 172.21.15.94:0/2055182654 wait complete. 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.708+0000 7f841d365640 1 Processor -- start 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.708+0000 7f841d365640 1 -- start start 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f841d365640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8418072cd0 0x7f84181eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f841d365640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84180c57e0 0x7f841814a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f841d365640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84181529d0 0x7f84181501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f841d365640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418159370 con 0x7f84181529d0 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f841d365640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84180c6940 con 0x7f8418072cd0 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f841d365640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84180723b0 con 0x7f84180c57e0 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84181529d0 0x7f84181501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84181529d0 0x7f84181501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53190/0 (socket says 172.21.15.94:53190) 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 -- 172.21.15.94:0/287021824 learned_addr learned my addr 172.21.15.94:0/287021824 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84167fc640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84180c57e0 0x7f841814a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f8416ffd640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8418072cd0 0x7f84181eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 -- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84180c57e0 msgr2=0x7f841814a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84180c57e0 0x7f841814a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 -- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8418072cd0 msgr2=0x7f84181eb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8418072cd0 0x7f84181eb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 -- 172.21.15.94:0/287021824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418151a20 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f84177fe640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84181529d0 0x7f84181501f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f841000c700 tx=0x7f841000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.709+0000 7f83ff7fe640 1 -- 172.21.15.94:0/287021824 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410003b70 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.710+0000 7f83ff7fe640 1 -- 172.21.15.94:0/287021824 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410003d10 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.710+0000 7f83ff7fe640 1 -- 172.21.15.94:0/287021824 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410005310 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.710+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8418151ce0 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.710+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84181eacd0 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.711+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841804f330 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.711+0000 7f83ff7fe640 1 -- 172.21.15.94:0/287021824 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84100054b0 con 0x7f84181529d0 2024-03-06T07:20:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.711+0000 7f83ff7fe640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f840406b4d0 0x7f840406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:31.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.711+0000 7f83ff7fe640 1 -- 172.21.15.94:0/287021824 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f841008a1b0 con 0x7f84181529d0 2024-03-06T07:20:31.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.712+0000 7f8416ffd640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f840406b4d0 0x7f840406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:31.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.712+0000 7f8416ffd640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f840406b4d0 0x7f840406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8408036000 tx=0x7f840803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:31.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.716+0000 7f83ff7fe640 1 -- 172.21.15.94:0/287021824 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84181590f0 con 0x7f84181529d0 2024-03-06T07:20:31.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.885+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f84180aacf0 con 0x7f84181529d0 2024-03-06T07:20:31.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.885+0000 7f83ff7fe640 1 -- 172.21.15.94:0/287021824 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v62) v1 ==== 101+0+11 (secure 0 0 0) 0x7f84100586f0 con 0x7f84181529d0 2024-03-06T07:20:31.887 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f840406b4d0 msgr2=0x7f840406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f840406b4d0 0x7f840406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8408036000 tx=0x7f840803d040 comp rx=0 tx=0).stop 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84181529d0 msgr2=0x7f84181501f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84181529d0 0x7f84181501f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f841000c700 tx=0x7f841000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 shutdown_connections 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f840406b4d0 0x7f840406d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84181529d0 0x7f84181501f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84180c57e0 0x7f841814a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 --2- 172.21.15.94:0/287021824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8418072cd0 0x7f84181eb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 >> 172.21.15.94:0/287021824 conn(0x7f84180aa630 msgr2=0x7f84181588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 shutdown_connections 2024-03-06T07:20:31.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.887+0000 7f841d365640 1 -- 172.21.15.94:0/287021824 wait complete. 2024-03-06T07:20:31.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:32.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.998+0000 7fc7a3ced640 1 Processor -- start 2024-03-06T07:20:32.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a3ced640 1 -- start start 2024-03-06T07:20:32.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a3ced640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc79c14f360 0x7fc79c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a3ced640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc79c0c88c0 0x7fc79c0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a3ced640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a3ced640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c0c8dd0 con 0x7fc79c0c88c0 2024-03-06T07:20:32.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a3ced640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79c0c73d0 con 0x7fc79c14f360 2024-03-06T07:20:32.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a3ced640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0be2e0 con 0x7fc79c0c5880 2024-03-06T07:20:32.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a1288640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc79c0c88c0 0x7fc79c0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a1288640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc79c0c88c0 0x7fc79c0c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53200/0 (socket says 172.21.15.94:53200) 2024-03-06T07:20:32.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:31.999+0000 7fc7a1288640 1 -- 172.21.15.94:0/99564458 learned_addr learned my addr 172.21.15.94:0/99564458 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:32.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a1a89640 1 --2- 172.21.15.94:0/99564458 >> v2:172.21.15.145:3300/0 conn(0x7fc79c14f360 0x7fc79c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a228a640 1 --2- 172.21.15.94:0/99564458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a1288640 1 -- 172.21.15.94:0/99564458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 msgr2=0x7fc79c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a1288640 1 --2- 172.21.15.94:0/99564458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a1288640 1 -- 172.21.15.94:0/99564458 >> v2:172.21.15.145:3300/0 conn(0x7fc79c14f360 msgr2=0x7fc79c155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a1288640 1 --2- 172.21.15.94:0/99564458 >> v2:172.21.15.145:3300/0 conn(0x7fc79c14f360 0x7fc79c155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a1288640 1 -- 172.21.15.94:0/99564458 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c0c7670 con 0x7fc79c0c88c0 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.000+0000 7fc7a1288640 1 --2- 172.21.15.94:0/99564458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c5340 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc79c08c840 tx=0x7fc79002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3604335fc381b475 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a0a87640 1 -- 172.21.15.94:0/99564458 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79002e070 con 0x7fc79c0c88c0 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a0a87640 1 -- 172.21.15.94:0/99564458 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc790037070 con 0x7fc79c0c88c0 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a0a87640 1 -- 172.21.15.94:0/99564458 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790031a00 con 0x7fc79c0c88c0 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 -- 172.21.15.94:0/99564458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 msgr2=0x7fc79c0c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/99564458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c5340 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc79c08c840 tx=0x7fc79002b880 comp rx=0 tx=0).stop 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 -- 172.21.15.94:0/99564458 shutdown_connections 2024-03-06T07:20:32.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/99564458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/99564458 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c5340 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/99564458 >> v2:172.21.15.145:3300/0 conn(0x7fc79c14f360 0x7fc79c155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 -- 172.21.15.94:0/99564458 >> 172.21.15.94:0/99564458 conn(0x7fc79c0ac570 msgr2=0x7fc79c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 -- 172.21.15.94:0/99564458 shutdown_connections 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.001+0000 7fc7a3ced640 1 -- 172.21.15.94:0/99564458 wait complete. 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.002+0000 7fc7a3ced640 1 Processor -- start 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.002+0000 7fc7a3ced640 1 -- start start 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a3ced640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c0c10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a3ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a3ced640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc79c14f360 0x7fc79c0c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a3ced640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c1590a0 con 0x7fc79c0c88c0 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a3ced640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c0c7c30 con 0x7fc79c14f360 2024-03-06T07:20:32.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a3ced640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c159220 con 0x7fc79c0c5880 2024-03-06T07:20:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53212/0 (socket says 172.21.15.94:53212) 2024-03-06T07:20:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 -- 172.21.15.94:0/2269006949 learned_addr learned my addr 172.21.15.94:0/2269006949 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a228a640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc79c14f360 0x7fc79c0c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1a89640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c0c10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 -- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 msgr2=0x7fc79c0c10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c0c10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 -- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc79c14f360 msgr2=0x7fc79c0c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc79c14f360 0x7fc79c0c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.003+0000 7fc7a1288640 1 -- 172.21.15.94:0/2269006949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c0be740 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.004+0000 7fc7a1288640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c06e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc790000c00 tx=0x7fc7900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.004+0000 7fc78e7fc640 1 -- 172.21.15.94:0/2269006949 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790032670 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.004+0000 7fc78e7fc640 1 -- 172.21.15.94:0/2269006949 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc790032810 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.004+0000 7fc78e7fc640 1 -- 172.21.15.94:0/2269006949 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790032c00 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.004+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c1ee880 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.004+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c1eeb60 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.005+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c1ee6b0 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.005+0000 7fc78e7fc640 1 -- 172.21.15.94:0/2269006949 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc79004c060 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.006+0000 7fc78e7fc640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc78006b500 0x7fc78006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.006+0000 7fc78e7fc640 1 -- 172.21.15.94:0/2269006949 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fc7900abbe0 con 0x7fc79c0c88c0 2024-03-06T07:20:32.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.006+0000 7fc7a1a89640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc78006b500 0x7fc78006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.006+0000 7fc7a1a89640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc78006b500 0x7fc78006d9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc798009aa0 tx=0x7fc798009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.010+0000 7fc78e7fc640 1 -- 172.21.15.94:0/2269006949 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc790040070 con 0x7fc79c0c88c0 2024-03-06T07:20:32.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.215+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc79c04f330 con 0x7fc79c0c88c0 2024-03-06T07:20:32.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.215+0000 7fc78e7fc640 1 -- 172.21.15.94:0/2269006949 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fc79007a270 con 0x7fc79c0c88c0 2024-03-06T07:20:32.217 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:32.217 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6138,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1223829390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1223829390},{"type":"v1","addr":"172.21.15.94:6834","nonce":1223829390}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":35378,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2852856592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2852856592},{"type":"v1","addr":"172.21.15.145:6837","nonce":2852856592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:31.495001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5530":{"gid":5530,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5530,"qdb_cluster":[5530]},"id":4}]} 2024-03-06T07:20:32.218 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 47 2024-03-06T07:20:32.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc78006b500 msgr2=0x7fc78006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc78006b500 0x7fc78006d9c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc798009aa0 tx=0x7fc798009190 comp rx=0 tx=0).stop 2024-03-06T07:20:32.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 msgr2=0x7fc79c0c06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c06e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc790000c00 tx=0x7fc7900046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 shutdown_connections 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc78006b500 0x7fc78006d9c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc79c14f360 0x7fc79c0c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79c0c88c0 0x7fc79c0c06e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 --2- 172.21.15.94:0/2269006949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc79c0c5880 0x7fc79c0c10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 >> 172.21.15.94:0/2269006949 conn(0x7fc79c0ac570 msgr2=0x7fc79c0cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 shutdown_connections 2024-03-06T07:20:32.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.217+0000 7fc7a3ced640 1 -- 172.21.15.94:0/2269006949 wait complete. 2024-03-06T07:20:32.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:32.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.331+0000 7faaa0c04640 1 Processor -- start 2024-03-06T07:20:32.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.331+0000 7faaa0c04640 1 -- start start 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faaa0c04640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa9c150920 0x7faa9c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faaa0c04640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faa9c06c720 0x7faa9c071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faaa0c04640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c067340 con 0x7faa9c0c2e70 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faaa0c04640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9c150d00 con 0x7faa9c150920 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faaa0c04640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9c0c2010 con 0x7faa9c06c720 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa9a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa9c150920 0x7faa9c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa9a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa9c150920 0x7faa9c0c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53224/0 (socket says 172.21.15.94:53224) 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa9a59c640 1 -- 172.21.15.94:0/4064676546 learned_addr learned my addr 172.21.15.94:0/4064676546 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa99d9b640 1 --2- 172.21.15.94:0/4064676546 >> v2:172.21.15.145:3300/0 conn(0x7faa9c06c720 0x7faa9c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa9a59c640 1 -- 172.21.15.94:0/4064676546 >> v2:172.21.15.145:3300/0 conn(0x7faa9c06c720 msgr2=0x7faa9c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa9a59c640 1 --2- 172.21.15.94:0/4064676546 >> v2:172.21.15.145:3300/0 conn(0x7faa9c06c720 0x7faa9c071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa9a59c640 1 -- 172.21.15.94:0/4064676546 >> v1:172.21.15.145:6789/0 conn(0x7faa9c0c2e70 legacy=0x7faa9c1532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.332+0000 7faa9a59c640 1 -- 172.21.15.94:0/4064676546 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c067060 con 0x7faa9c150920 2024-03-06T07:20:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.333+0000 7faa9a59c640 1 --2- 172.21.15.94:0/4064676546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c150920 0x7faa9c0c2930 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7faa9c06ef00 tx=0x7faa8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671b523fc8fdf6dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.333+0000 7faa9959a640 1 -- 172.21.15.94:0/4064676546 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c035020 con 0x7faa9c150920 2024-03-06T07:20:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.333+0000 7faa9959a640 1 -- 172.21.15.94:0/4064676546 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c031870 con 0x7faa9c150920 2024-03-06T07:20:32.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.333+0000 7faa9959a640 1 -- 172.21.15.94:0/4064676546 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c031c40 con 0x7faa9c150920 2024-03-06T07:20:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.333+0000 7faaa0c04640 1 -- 172.21.15.94:0/4064676546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c150920 msgr2=0x7faa9c0c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.334+0000 7faaa0c04640 1 --2- 172.21.15.94:0/4064676546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c150920 0x7faa9c0c2930 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7faa9c06ef00 tx=0x7faa8c02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.334+0000 7faaa0c04640 1 -- 172.21.15.94:0/4064676546 shutdown_connections 2024-03-06T07:20:32.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.334+0000 7faaa0c04640 1 --2- 172.21.15.94:0/4064676546 >> v2:172.21.15.145:3300/0 conn(0x7faa9c06c720 0x7faa9c071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.334+0000 7faaa0c04640 1 --2- 172.21.15.94:0/4064676546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c150920 0x7faa9c0c2930 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.334+0000 7faaa0c04640 1 -- 172.21.15.94:0/4064676546 >> 172.21.15.94:0/4064676546 conn(0x7faa9c0c70b0 msgr2=0x7faa9c0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.334+0000 7faaa0c04640 1 -- 172.21.15.94:0/4064676546 shutdown_connections 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.334+0000 7faaa0c04640 1 -- 172.21.15.94:0/4064676546 wait complete. 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.335+0000 7faaa0c04640 1 Processor -- start 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.335+0000 7faaa0c04640 1 -- start start 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c06c720 0x7faa9c148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c0c2e70 0x7faa9c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c150920 0x7faa9c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c073210 con 0x7faa9c0c2e70 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c073090 con 0x7faa9c150920 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c0c1b40 con 0x7faa9c06c720 2024-03-06T07:20:32.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c0c2e70 0x7faa9c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c0c2e70 0x7faa9c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53240/0 (socket says 172.21.15.94:53240) 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 -- 172.21.15.94:0/548443745 learned_addr learned my addr 172.21.15.94:0/548443745 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa9ad9d640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c150920 0x7faa9c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa9a59c640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c06c720 0x7faa9c148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 -- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c06c720 msgr2=0x7faa9c148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c06c720 0x7faa9c148450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 -- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c150920 msgr2=0x7faa9c152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c150920 0x7faa9c152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 -- 172.21.15.94:0/548443745 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c1512d0 con 0x7faa9c0c2e70 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa99d9b640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c0c2e70 0x7faa9c1529d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7faa9000e840 tx=0x7faa9000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa8affd640 1 -- 172.21.15.94:0/548443745 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90017070 con 0x7faa9c0c2e70 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa8affd640 1 -- 172.21.15.94:0/548443745 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9000c940 con 0x7faa9c0c2e70 2024-03-06T07:20:32.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faa8affd640 1 -- 172.21.15.94:0/548443745 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9000cd10 con 0x7faa9c0c2e70 2024-03-06T07:20:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c151550 con 0x7faa9c0c2e70 2024-03-06T07:20:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.336+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c148990 con 0x7faa9c0c2e70 2024-03-06T07:20:32.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.337+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c04f330 con 0x7faa9c0c2e70 2024-03-06T07:20:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.338+0000 7faa8affd640 1 -- 172.21.15.94:0/548443745 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa9001a070 con 0x7faa9c0c2e70 2024-03-06T07:20:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.338+0000 7faa8affd640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa7c06b4d0 0x7faa7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.338+0000 7faa8affd640 1 -- 172.21.15.94:0/548443745 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7faa9008a710 con 0x7faa9c0c2e70 2024-03-06T07:20:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.338+0000 7faa9a59c640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa7c06b4d0 0x7faa7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.339+0000 7faa9a59c640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa7c06b4d0 0x7faa7c06d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faa8c036000 tx=0x7faa8c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.343+0000 7faa8affd640 1 -- 172.21.15.94:0/548443745 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa9c1512d0 con 0x7faa9c0c2e70 2024-03-06T07:20:32.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.515+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa9c06eca0 con 0x7faa9c0c2e70 2024-03-06T07:20:32.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.516+0000 7faa8affd640 1 -- 172.21.15.94:0/548443745 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+17407 (secure 0 0 0) 0x7faa90058bc0 con 0x7faa9c0c2e70 2024-03-06T07:20:32.518 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:32.518 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":62,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:30.667810+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6834/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6835/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.94:6833/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.145:6837/3125091562":"2024-03-07T07:20:16.601064+0000","172.21.15.94:6832/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/3125091562":"2024-03-07T07:20:16.601064+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.517+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa7c06b4d0 msgr2=0x7faa7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.517+0000 7faaa0c04640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa7c06b4d0 0x7faa7c06d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faa8c036000 tx=0x7faa8c03c040 comp rx=0 tx=0).stop 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c0c2e70 msgr2=0x7faa9c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c0c2e70 0x7faa9c1529d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7faa9000e840 tx=0x7faa9000ed00 comp rx=0 tx=0).stop 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 shutdown_connections 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa7c06b4d0 0x7faa7c06d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c150920 0x7faa9c152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa9c0c2e70 0x7faa9c1529d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 --2- 172.21.15.94:0/548443745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c06c720 0x7faa9c148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 >> 172.21.15.94:0/548443745 conn(0x7faa9c0c70b0 msgr2=0x7faa9c14f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 shutdown_connections 2024-03-06T07:20:32.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.518+0000 7faaa0c04640 1 -- 172.21.15.94:0/548443745 wait complete. 2024-03-06T07:20:32.530 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-06T07:20:32.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.629+0000 7f88c55d0640 1 Processor -- start 2024-03-06T07:20:32.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.629+0000 7f88c55d0640 1 -- start start 2024-03-06T07:20:32.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.629+0000 7f88c55d0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f88c0150580 0x7f88c006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.629+0000 7f88c55d0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c6000 0x7f88c006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.629+0000 7f88c55d0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c0153a60 con 0x7f88c00c1320 2024-03-06T07:20:32.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.629+0000 7f88c55d0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c0153d40 con 0x7f88c0150580 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.629+0000 7f88c55d0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c006c2c0 con 0x7f88c00c6000 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88beffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f88c0150580 0x7f88c006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88be7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f88c00c1320 0x7f88c01566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45958/0 (socket says 172.21.15.94:45958) 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88be7fc640 1 -- 172.21.15.94:0/2594764168 learned_addr learned my addr 172.21.15.94:0/2594764168 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88bf7fe640 1 --2- 172.21.15.94:0/2594764168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c6000 0x7f88c006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88beffd640 1 -- 172.21.15.94:0/2594764168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c6000 msgr2=0x7f88c006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88beffd640 1 --2- 172.21.15.94:0/2594764168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c6000 0x7f88c006e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88beffd640 1 -- 172.21.15.94:0/2594764168 >> v1:172.21.15.94:6789/0 conn(0x7f88c00c1320 legacy=0x7f88c01566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88beffd640 1 -- 172.21.15.94:0/2594764168 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c006c5a0 con 0x7f88c0150580 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.630+0000 7f88bdffb640 1 -- 172.21.15.94:0/2594764168 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114428919 0 0) 0x7f88c0153a60 con 0x7f88c00c1320 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88beffd640 1 --2- 172.21.15.94:0/2594764168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c0150580 0x7f88c006dd10 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f88b00097a0 tx=0x7f88b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fe167feef967dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88bdffb640 1 -- 172.21.15.94:0/2594764168 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b00315a0 con 0x7f88c0150580 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88bdffb640 1 -- 172.21.15.94:0/2594764168 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b0037070 con 0x7f88c0150580 2024-03-06T07:20:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88bdffb640 1 -- 172.21.15.94:0/2594764168 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0031960 con 0x7f88c0150580 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 -- 172.21.15.94:0/2594764168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c0150580 msgr2=0x7f88c006dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 --2- 172.21.15.94:0/2594764168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c0150580 0x7f88c006dd10 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f88b00097a0 tx=0x7f88b002b840 comp rx=0 tx=0).stop 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 -- 172.21.15.94:0/2594764168 shutdown_connections 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 --2- 172.21.15.94:0/2594764168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c6000 0x7f88c006e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 --2- 172.21.15.94:0/2594764168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c0150580 0x7f88c006dd10 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 -- 172.21.15.94:0/2594764168 >> 172.21.15.94:0/2594764168 conn(0x7f88c00c6710 msgr2=0x7f88c00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 -- 172.21.15.94:0/2594764168 shutdown_connections 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.631+0000 7f88c55d0640 1 -- 172.21.15.94:0/2594764168 wait complete. 2024-03-06T07:20:32.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.632+0000 7f88c55d0640 1 Processor -- start 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88c55d0640 1 -- start start 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88c55d0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c1320 0x7f88c01edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88c55d0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c00c6000 0x7f88c00bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88c55d0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c0150580 0x7f88c00ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88c55d0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00c0dc0 con 0x7f88c00c6000 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88c55d0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00c0c40 con 0x7f88c0150580 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88c55d0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c0156dc0 con 0x7f88c00c1320 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88be7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c00c6000 0x7f88c00bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88be7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c00c6000 0x7f88c00bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53252/0 (socket says 172.21.15.94:53252) 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88be7fc640 1 -- 172.21.15.94:0/1700968290 learned_addr learned my addr 172.21.15.94:0/1700968290 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88beffd640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c1320 0x7f88c01edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.633+0000 7f88bf7fe640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c0150580 0x7f88c00ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88be7fc640 1 -- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c1320 msgr2=0x7f88c01edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88be7fc640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c1320 0x7f88c01edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88be7fc640 1 -- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c0150580 msgr2=0x7f88c00ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88be7fc640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c0150580 0x7f88c00ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88be7fc640 1 -- 172.21.15.94:0/1700968290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c00aa3b0 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88be7fc640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c00c6000 0x7f88c00bfdb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f88b400c500 tx=0x7f88b400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88a77fe640 1 -- 172.21.15.94:0/1700968290 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4018070 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88a77fe640 1 -- 172.21.15.94:0/1700968290 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b4007580 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c00a9b10 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88a77fe640 1 -- 172.21.15.94:0/1700968290 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4004a20 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.634+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c00aa0d0 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.635+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c00a95b0 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.635+0000 7f88a77fe640 1 -- 172.21.15.94:0/1700968290 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88b401b070 con 0x7f88c00c6000 2024-03-06T07:20:32.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.636+0000 7f88a77fe640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88ac06b4d0 0x7f88ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.636+0000 7f88a77fe640 1 -- 172.21.15.94:0/1700968290 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f88b40896c0 con 0x7f88c00c6000 2024-03-06T07:20:32.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.636+0000 7f88beffd640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88ac06b4d0 0x7f88ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.636+0000 7f88beffd640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88ac06b4d0 0x7f88ac06d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f88b0036000 tx=0x7f88b0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.640+0000 7f88a77fe640 1 -- 172.21.15.94:0/1700968290 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88c00a95b0 con 0x7f88c00c6000 2024-03-06T07:20:32.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.847+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88c004f330 con 0x7f88c00c6000 2024-03-06T07:20:32.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.847+0000 7f88a77fe640 1 -- 172.21.15.94:0/1700968290 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4986 (secure 0 0 0) 0x7f88b4057c90 con 0x7f88c00c6000 2024-03-06T07:20:32.849 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:32.850 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6138,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1223829390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1223829390},{"type":"v1","addr":"172.21.15.94:6834","nonce":1223829390}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":35378,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2852856592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2852856592},{"type":"v1","addr":"172.21.15.145:6837","nonce":2852856592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:31.495001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5530":{"gid":5530,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5530,"qdb_cluster":[5530]},"id":4}]} 2024-03-06T07:20:32.850 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 47 2024-03-06T07:20:32.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88ac06b4d0 msgr2=0x7f88ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88ac06b4d0 0x7f88ac06d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f88b0036000 tx=0x7f88b0008030 comp rx=0 tx=0).stop 2024-03-06T07:20:32.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c00c6000 msgr2=0x7f88c00bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c00c6000 0x7f88c00bfdb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f88b400c500 tx=0x7f88b400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 shutdown_connections 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88ac06b4d0 0x7f88ac06d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c0150580 0x7f88c00ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88c00c6000 0x7f88c00bfdb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 --2- 172.21.15.94:0/1700968290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c00c1320 0x7f88c01edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 >> 172.21.15.94:0/1700968290 conn(0x7f88c00c6710 msgr2=0x7f88c0153630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 shutdown_connections 2024-03-06T07:20:32.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.849+0000 7f88c55d0640 1 -- 172.21.15.94:0/1700968290 wait complete. 2024-03-06T07:20:32.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:32.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.961+0000 7ffa5d5bc640 1 Processor -- start 2024-03-06T07:20:32.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.961+0000 7ffa5d5bc640 1 -- start start 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa5d5bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa581506b0 0x7ffa58153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa5d5bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffa58156ea0 0x7ffa580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa5d5bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffa580acc00 0x7ffa580c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa5d5bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa58157320 con 0x7ffa58156ea0 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa5d5bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa58156450 con 0x7ffa580acc00 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa5d5bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa58156a10 con 0x7ffa581506b0 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa567fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffa58156ea0 0x7ffa580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa567fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffa58156ea0 0x7ffa580702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53264/0 (socket says 172.21.15.94:53264) 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa567fc640 1 -- 172.21.15.94:0/2949798955 learned_addr learned my addr 172.21.15.94:0/2949798955 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:32.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa56ffd640 1 --2- 172.21.15.94:0/2949798955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa581506b0 0x7ffa58153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.962+0000 7ffa577fe640 1 --2- 172.21.15.94:0/2949798955 >> v2:172.21.15.145:3300/0 conn(0x7ffa580acc00 0x7ffa580c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa567fc640 1 -- 172.21.15.94:0/2949798955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa581506b0 msgr2=0x7ffa58153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa567fc640 1 --2- 172.21.15.94:0/2949798955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa581506b0 0x7ffa58153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa567fc640 1 -- 172.21.15.94:0/2949798955 >> v2:172.21.15.145:3300/0 conn(0x7ffa580acc00 msgr2=0x7ffa580c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa567fc640 1 --2- 172.21.15.94:0/2949798955 >> v2:172.21.15.145:3300/0 conn(0x7ffa580acc00 0x7ffa580c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa567fc640 1 -- 172.21.15.94:0/2949798955 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa58070800 con 0x7ffa58156ea0 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa577fe640 1 --2- 172.21.15.94:0/2949798955 >> v2:172.21.15.145:3300/0 conn(0x7ffa580acc00 0x7ffa580c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa56ffd640 1 --2- 172.21.15.94:0/2949798955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa581506b0 0x7ffa58153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa567fc640 1 --2- 172.21.15.94:0/2949798955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa58156ea0 0x7ffa580702c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ffa58150a90 tx=0x7ffa4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4a6a6237f6fbdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.963+0000 7ffa55ffb640 1 -- 172.21.15.94:0/2949798955 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c035020 con 0x7ffa58156ea0 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa55ffb640 1 -- 172.21.15.94:0/2949798955 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c031870 con 0x7ffa58156ea0 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa55ffb640 1 -- 172.21.15.94:0/2949798955 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c031c40 con 0x7ffa58156ea0 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/2949798955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa58156ea0 msgr2=0x7ffa580702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/2949798955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa58156ea0 0x7ffa580702c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ffa58150a90 tx=0x7ffa4c02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:32.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/2949798955 shutdown_connections 2024-03-06T07:20:32.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/2949798955 >> v2:172.21.15.145:3300/0 conn(0x7ffa580acc00 0x7ffa580c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/2949798955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa58156ea0 0x7ffa580702c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/2949798955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa581506b0 0x7ffa58153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/2949798955 >> 172.21.15.94:0/2949798955 conn(0x7ffa5806b1a0 msgr2=0x7ffa5806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:32.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/2949798955 shutdown_connections 2024-03-06T07:20:32.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.964+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/2949798955 wait complete. 2024-03-06T07:20:32.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.965+0000 7ffa5d5bc640 1 Processor -- start 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.965+0000 7ffa5d5bc640 1 -- start start 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa5d5bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa580acc00 0x7ffa580a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa5d5bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa581506b0 0x7ffa581511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa5d5bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa58156ea0 0x7ffa58151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa5d5bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa581566a0 con 0x7ffa581506b0 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa5d5bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa58070d80 con 0x7ffa580acc00 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa5d5bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa580ad050 con 0x7ffa58156ea0 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa581506b0 0x7ffa581511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa581506b0 0x7ffa581511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53280/0 (socket says 172.21.15.94:53280) 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa56ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa580acc00 0x7ffa580a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 -- 172.21.15.94:0/1152192310 learned_addr learned my addr 172.21.15.94:0/1152192310 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa577fe640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa58156ea0 0x7ffa58151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 -- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa58156ea0 msgr2=0x7ffa58151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa58156ea0 0x7ffa58151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 -- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa580acc00 msgr2=0x7ffa580a8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa580acc00 0x7ffa580a8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.966+0000 7ffa567fc640 1 -- 172.21.15.94:0/1152192310 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa581e03a0 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.967+0000 7ffa567fc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa581506b0 0x7ffa581511c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c000c00 tx=0x7ffa4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.967+0000 7ffa3f7fe640 1 -- 172.21.15.94:0/1152192310 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c035040 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.967+0000 7ffa3f7fe640 1 -- 172.21.15.94:0/1152192310 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c030c80 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.967+0000 7ffa3f7fe640 1 -- 172.21.15.94:0/1152192310 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0456c0 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.967+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa580a86c0 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.967+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5814c580 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.968+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5804f330 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.968+0000 7ffa3f7fe640 1 -- 172.21.15.94:0/1152192310 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa4c038020 con 0x7ffa581506b0 2024-03-06T07:20:32.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.969+0000 7ffa3f7fe640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4406b500 0x7ffa4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:32.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.969+0000 7ffa3f7fe640 1 -- 172.21.15.94:0/1152192310 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7ffa4c0ac6b0 con 0x7ffa581506b0 2024-03-06T07:20:32.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.969+0000 7ffa56ffd640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4406b500 0x7ffa4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:32.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.969+0000 7ffa56ffd640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4406b500 0x7ffa4406d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009bd0 tx=0x7ffa4800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:32.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:32.973+0000 7ffa3f7fe640 1 -- 172.21.15.94:0/1152192310 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa4c07abf0 con 0x7ffa581506b0 2024-03-06T07:20:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.183+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa580700a0 con 0x7ffa581506b0 2024-03-06T07:20:33.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.183+0000 7ffa3f7fe640 1 -- 172.21.15.94:0/1152192310 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4986 (secure 0 0 0) 0x7ffa4c07ec90 con 0x7ffa581506b0 2024-03-06T07:20:33.185 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:33.186 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6138,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1223829390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1223829390},{"type":"v1","addr":"172.21.15.94:6834","nonce":1223829390}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":35378,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2852856592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2852856592},{"type":"v1","addr":"172.21.15.145:6837","nonce":2852856592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:07.621086+0000","modified":"2024-03-06T07:20:31.495001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5530":{"gid":5530,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.145:6836/3216323598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3216323598},{"type":"v1","addr":"172.21.15.145:6836","nonce":3216323598}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5530,"qdb_cluster":[5530]},"id":4}]} 2024-03-06T07:20:33.186 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 47 2024-03-06T07:20:33.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4406b500 msgr2=0x7ffa4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4406b500 0x7ffa4406d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009bd0 tx=0x7ffa4800b040 comp rx=0 tx=0).stop 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa581506b0 msgr2=0x7ffa581511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa581506b0 0x7ffa581511c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c000c00 tx=0x7ffa4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 shutdown_connections 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4406b500 0x7ffa4406d9c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa58156ea0 0x7ffa58151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa581506b0 0x7ffa581511c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 --2- 172.21.15.94:0/1152192310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa580acc00 0x7ffa580a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 >> 172.21.15.94:0/1152192310 conn(0x7ffa5806b1a0 msgr2=0x7ffa580c5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 shutdown_connections 2024-03-06T07:20:33.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.185+0000 7ffa5d5bc640 1 -- 172.21.15.94:0/1152192310 wait complete. 2024-03-06T07:20:33.198 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:33.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.295+0000 7f6970f0c640 1 Processor -- start 2024-03-06T07:20:33.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.295+0000 7f6970f0c640 1 -- start start 2024-03-06T07:20:33.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f6970f0c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f696c1550c0 0x7f696c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f6970f0c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f696c0aa180 0x7f696c0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f6970f0c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c0aa660 con 0x7f696c1559e0 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f6970f0c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c0c4610 con 0x7f696c1550c0 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f6970f0c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c0aa940 con 0x7f696c0aa180 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f696c1550c0 0x7f696c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f696c1550c0 0x7f696c1554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53288/0 (socket says 172.21.15.94:53288) 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696a59c640 1 -- 172.21.15.94:0/2525785784 learned_addr learned my addr 172.21.15.94:0/2525785784 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696ad9d640 1 --2- 172.21.15.94:0/2525785784 >> v2:172.21.15.145:3300/0 conn(0x7f696c0aa180 0x7f696c0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696a59c640 1 -- 172.21.15.94:0/2525785784 >> v2:172.21.15.145:3300/0 conn(0x7f696c0aa180 msgr2=0x7f696c0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696a59c640 1 --2- 172.21.15.94:0/2525785784 >> v2:172.21.15.145:3300/0 conn(0x7f696c0aa180 0x7f696c0c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696a59c640 1 -- 172.21.15.94:0/2525785784 >> v1:172.21.15.145:6789/0 conn(0x7f696c1559e0 legacy=0x7f696c14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.296+0000 7f696a59c640 1 -- 172.21.15.94:0/2525785784 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c0c4860 con 0x7f696c1550c0 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.297+0000 7f696a59c640 1 --2- 172.21.15.94:0/2525785784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c1550c0 0x7f696c1554a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f696c0abd80 tx=0x7f695c02e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c673878cfbc5b1cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.297+0000 7f696959a640 1 -- 172.21.15.94:0/2525785784 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c02ea00 con 0x7f696c1550c0 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.297+0000 7f696959a640 1 -- 172.21.15.94:0/2525785784 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c02eba0 con 0x7f696c1550c0 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.297+0000 7f696959a640 1 -- 172.21.15.94:0/2525785784 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c009600 con 0x7f696c1550c0 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 -- 172.21.15.94:0/2525785784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c1550c0 msgr2=0x7f696c1554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 --2- 172.21.15.94:0/2525785784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c1550c0 0x7f696c1554a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f696c0abd80 tx=0x7f695c02e060 comp rx=0 tx=0).stop 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 -- 172.21.15.94:0/2525785784 shutdown_connections 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 --2- 172.21.15.94:0/2525785784 >> v2:172.21.15.145:3300/0 conn(0x7f696c0aa180 0x7f696c0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 --2- 172.21.15.94:0/2525785784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c1550c0 0x7f696c1554a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 -- 172.21.15.94:0/2525785784 >> 172.21.15.94:0/2525785784 conn(0x7f696c0c6710 msgr2=0x7f696c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 -- 172.21.15.94:0/2525785784 shutdown_connections 2024-03-06T07:20:33.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.298+0000 7f6970f0c640 1 -- 172.21.15.94:0/2525785784 wait complete. 2024-03-06T07:20:33.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.299+0000 7f6970f0c640 1 Processor -- start 2024-03-06T07:20:33.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.299+0000 7f6970f0c640 1 -- start start 2024-03-06T07:20:33.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f6970f0c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c0aa180 0x7f696c1523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f6970f0c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f696c1550c0 0x7f696c150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f6970f0c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f696c1559e0 0x7f696c1e6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f6970f0c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c14f310 con 0x7f696c0aa180 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f6970f0c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c1560 con 0x7f696c1550c0 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f6970f0c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c14f490 con 0x7f696c1559e0 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c0aa180 0x7f696c1523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c0aa180 0x7f696c1523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53304/0 (socket says 172.21.15.94:53304) 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 -- 172.21.15.94:0/1773214506 learned_addr learned my addr 172.21.15.94:0/1773214506 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f6969d9b640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f696c1550c0 0x7f696c150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696ad9d640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f696c1559e0 0x7f696c1e6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 -- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f696c1559e0 msgr2=0x7f696c1e6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f696c1559e0 0x7f696c1e6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 -- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f696c1550c0 msgr2=0x7f696c150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f696c1550c0 0x7f696c150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 -- 172.21.15.94:0/1773214506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c1e67b0 con 0x7f696c0aa180 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.300+0000 7f696a59c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c0aa180 0x7f696c1523e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f695c000f80 tx=0x7f695c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.301+0000 7f695affd640 1 -- 172.21.15.94:0/1773214506 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c045020 con 0x7f696c0aa180 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.301+0000 7f695affd640 1 -- 172.21.15.94:0/1773214506 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c009e50 con 0x7f696c0aa180 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.301+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c14f9b0 con 0x7f696c0aa180 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.301+0000 7f695affd640 1 -- 172.21.15.94:0/1773214506 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c00b720 con 0x7f696c0aa180 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.301+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c14ff70 con 0x7f696c0aa180 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.302+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c04f330 con 0x7f696c0aa180 2024-03-06T07:20:33.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.302+0000 7f695affd640 1 -- 172.21.15.94:0/1773214506 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f695c048020 con 0x7f696c0aa180 2024-03-06T07:20:33.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.302+0000 7f695affd640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f694c06b4d0 0x7f694c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.302+0000 7f695affd640 1 -- 172.21.15.94:0/1773214506 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f695c0aed20 con 0x7f696c0aa180 2024-03-06T07:20:33.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.302+0000 7f6969d9b640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f694c06b4d0 0x7f694c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.303+0000 7f6969d9b640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f694c06b4d0 0x7f694c06d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f696c152160 tx=0x7f6960009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.307+0000 7f695affd640 1 -- 172.21.15.94:0/1773214506 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f695c04f030 con 0x7f696c0aa180 2024-03-06T07:20:33.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.479+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f696c1e69d0 con 0x7f696c0aa180 2024-03-06T07:20:33.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.481+0000 7f695affd640 1 -- 172.21.15.94:0/1773214506 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f695c07d270 con 0x7f696c0aa180 2024-03-06T07:20:33.483 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:33.483 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":62,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:30.667810+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:06.055145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:20:06.580638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6834/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6835/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.94:6833/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.145:6837/3125091562":"2024-03-07T07:20:16.601064+0000","172.21.15.94:6832/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/3125091562":"2024-03-07T07:20:16.601064+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:33.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.482+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f694c06b4d0 msgr2=0x7f694c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.482+0000 7f6970f0c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f694c06b4d0 0x7f694c06d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f696c152160 tx=0x7f6960009040 comp rx=0 tx=0).stop 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.482+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c0aa180 msgr2=0x7f696c1523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.482+0000 7f6970f0c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c0aa180 0x7f696c1523e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f695c000f80 tx=0x7f695c004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 shutdown_connections 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f694c06b4d0 0x7f694c06d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f696c1559e0 0x7f696c1e6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f696c1550c0 0x7f696c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 --2- 172.21.15.94:0/1773214506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f696c0aa180 0x7f696c1523e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 >> 172.21.15.94:0/1773214506 conn(0x7f696c0c6710 msgr2=0x7f696c1539b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 shutdown_connections 2024-03-06T07:20:33.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.483+0000 7f6970f0c640 1 -- 172.21.15.94:0/1773214506 wait complete. 2024-03-06T07:20:33.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-06T07:20:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.595+0000 7fb0792e1640 1 Processor -- start 2024-03-06T07:20:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.595+0000 7fb0792e1640 1 -- start start 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0792e1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb074152760 0x7fb074152b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0792e1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb07406f570 0x7fb0740c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0792e1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0741591e0 con 0x7fb0740c9620 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0792e1640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07406feb0 con 0x7fb07406f570 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0792e1640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0740c2840 con 0x7fb074152760 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb07406f570 0x7fb0740c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb07406f570 0x7fb0740c1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53318/0 (socket says 172.21.15.94:53318) 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0737fe640 1 -- 172.21.15.94:0/1300921025 learned_addr learned my addr 172.21.15.94:0/1300921025 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb072ffd640 1 --2- 172.21.15.94:0/1300921025 >> v2:172.21.15.145:3300/0 conn(0x7fb074152760 0x7fb074152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0737fe640 1 -- 172.21.15.94:0/1300921025 >> v2:172.21.15.145:3300/0 conn(0x7fb074152760 msgr2=0x7fb074152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0737fe640 1 --2- 172.21.15.94:0/1300921025 >> v2:172.21.15.145:3300/0 conn(0x7fb074152760 0x7fb074152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0737fe640 1 -- 172.21.15.94:0/1300921025 >> v1:172.21.15.145:6789/0 conn(0x7fb0740c9620 legacy=0x7fb0740c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.596+0000 7fb0737fe640 1 -- 172.21.15.94:0/1300921025 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074070150 con 0x7fb07406f570 2024-03-06T07:20:33.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.597+0000 7fb0737fe640 1 --2- 172.21.15.94:0/1300921025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb07406f570 0x7fb0740c1d90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb0740c6270 tx=0x7fb06c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1545234687363c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.597+0000 7fb071ffb640 1 -- 172.21.15.94:0/1300921025 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c035020 con 0x7fb07406f570 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.597+0000 7fb071ffb640 1 -- 172.21.15.94:0/1300921025 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c031870 con 0x7fb07406f570 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.597+0000 7fb071ffb640 1 -- 172.21.15.94:0/1300921025 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c031c40 con 0x7fb07406f570 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 -- 172.21.15.94:0/1300921025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb07406f570 msgr2=0x7fb0740c1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 --2- 172.21.15.94:0/1300921025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb07406f570 0x7fb0740c1d90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb0740c6270 tx=0x7fb06c02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 -- 172.21.15.94:0/1300921025 shutdown_connections 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 --2- 172.21.15.94:0/1300921025 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb07406f570 0x7fb0740c1d90 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 --2- 172.21.15.94:0/1300921025 >> v2:172.21.15.145:3300/0 conn(0x7fb074152760 0x7fb074152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 -- 172.21.15.94:0/1300921025 >> 172.21.15.94:0/1300921025 conn(0x7fb07406b5f0 msgr2=0x7fb0740c8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 -- 172.21.15.94:0/1300921025 shutdown_connections 2024-03-06T07:20:33.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.598+0000 7fb0792e1640 1 -- 172.21.15.94:0/1300921025 wait complete. 2024-03-06T07:20:33.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.599+0000 7fb0792e1640 1 Processor -- start 2024-03-06T07:20:33.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.599+0000 7fb0792e1640 1 -- start start 2024-03-06T07:20:33.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0792e1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb07406f570 0x7fb0741567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0792e1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0740c9620 0x7fb0741eb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0792e1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb074152760 0x7fb0740abe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0792e1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0740c83c0 con 0x7fb0740c9620 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0792e1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0741596a0 con 0x7fb07406f570 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0792e1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0740c8540 con 0x7fb074152760 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0740c9620 0x7fb0741eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0740c9620 0x7fb0741eb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53324/0 (socket says 172.21.15.94:53324) 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 -- 172.21.15.94:0/3553405648 learned_addr learned my addr 172.21.15.94:0/3553405648 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb072ffd640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb07406f570 0x7fb0741567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0737fe640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb074152760 0x7fb0740abe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 -- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb074152760 msgr2=0x7fb0740abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb074152760 0x7fb0740abe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 -- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb07406f570 msgr2=0x7fb0741567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb07406f570 0x7fb0741567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 -- 172.21.15.94:0/3553405648 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0741583b0 con 0x7fb0740c9620 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.600+0000 7fb0727fc640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0740c9620 0x7fb0741eb200 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb06800c480 tx=0x7fb06800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.601+0000 7fb05b7fe640 1 -- 172.21.15.94:0/3553405648 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0680052d0 con 0x7fb0740c9620 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.601+0000 7fb05b7fe640 1 -- 172.21.15.94:0/3553405648 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb068005470 con 0x7fb0740c9620 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.601+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0741eb740 con 0x7fb0740c9620 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.601+0000 7fb05b7fe640 1 -- 172.21.15.94:0/3553405648 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb068012870 con 0x7fb0740c9620 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.601+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb074158950 con 0x7fb0740c9620 2024-03-06T07:20:33.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.602+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07404f330 con 0x7fb0740c9620 2024-03-06T07:20:33.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.602+0000 7fb05b7fe640 1 -- 172.21.15.94:0/3553405648 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb068012a10 con 0x7fb0740c9620 2024-03-06T07:20:33.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.602+0000 7fb05b7fe640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb06006b4d0 0x7fb06006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.603+0000 7fb05b7fe640 1 -- 172.21.15.94:0/3553405648 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fb068089f90 con 0x7fb0740c9620 2024-03-06T07:20:33.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.603+0000 7fb072ffd640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb06006b4d0 0x7fb06006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.603+0000 7fb072ffd640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb06006b4d0 0x7fb06006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb074155360 tx=0x7fb06400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.607+0000 7fb05b7fe640 1 -- 172.21.15.94:0/3553405648 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0741591e0 con 0x7fb0740c9620 2024-03-06T07:20:33.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.815+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7fb0740c41c0 con 0x7fb0740c9620 2024-03-06T07:20:33.817 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:20:33.815+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:20:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.816+0000 7fb05b7fe640 1 -- 172.21.15.94:0/3553405648 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v48) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb068058560 con 0x7fb0740c9620 2024-03-06T07:20:33.818 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:20:33.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb06006b4d0 msgr2=0x7fb06006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb06006b4d0 0x7fb06006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb074155360 tx=0x7fb06400b040 comp rx=0 tx=0).stop 2024-03-06T07:20:33.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0740c9620 msgr2=0x7fb0741eb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0740c9620 0x7fb0741eb200 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb06800c480 tx=0x7fb06800c940 comp rx=0 tx=0).stop 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 shutdown_connections 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb06006b4d0 0x7fb06006d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb074152760 0x7fb0740abe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0740c9620 0x7fb0741eb200 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 --2- 172.21.15.94:0/3553405648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb07406f570 0x7fb0741567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 >> 172.21.15.94:0/3553405648 conn(0x7fb07406b5f0 msgr2=0x7fb0740a9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 shutdown_connections 2024-03-06T07:20:33.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.819+0000 7fb0792e1640 1 -- 172.21.15.94:0/3553405648 wait complete. 2024-03-06T07:20:33.832 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-06T07:20:33.847 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:20:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.934+0000 7f588a5d7640 1 Processor -- start 2024-03-06T07:20:33.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.934+0000 7f588a5d7640 1 -- start start 2024-03-06T07:20:33.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f588a5d7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f58840c26c0 0x7f5884155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f588a5d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58840670c0 con 0x7f58840c2aa0 2024-03-06T07:20:33.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f588a5d7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588406b0f0 con 0x7f58840c0470 2024-03-06T07:20:33.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f588a5d7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588406b740 con 0x7f58840c26c0 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f58837fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f58840c2aa0 0x7f588406bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45970/0 (socket says 172.21.15.94:45970) 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f58837fe640 1 -- 172.21.15.94:0/2745937631 learned_addr learned my addr 172.21.15.94:0/2745937631 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f5883fff640 1 --2- 172.21.15.94:0/2745937631 >> v2:172.21.15.145:3300/0 conn(0x7f58840c26c0 0x7f5884155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072093848 0 0) 0x7f58840670c0 con 0x7f58840c2aa0 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.935+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586c003520 con 0x7f58840c2aa0 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.936+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752534678 0 0) 0x7f588406b0f0 con 0x7f58840c0470 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.936+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58840670c0 con 0x7f58840c0470 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.936+0000 7f5883fff640 1 -- 172.21.15.94:0/2745937631 >> v1:172.21.15.145:6789/0 conn(0x7f58840c0470 legacy=0x7f5884152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.936+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 645941355 0 0) 0x7f586c003520 con 0x7f58840c2aa0 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.936+0000 7f5883fff640 1 -- 172.21.15.94:0/2745937631 >> v1:172.21.15.94:6789/0 conn(0x7f58840c2aa0 legacy=0x7f588406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.936+0000 7f5883fff640 1 -- 172.21.15.94:0/2745937631 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588406b3d0 con 0x7f58840c26c0 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.936+0000 7f5883fff640 1 --2- 172.21.15.94:0/2745937631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 0x7f5884155bd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f58740097a0 tx=0x7f587402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5587a9c7251946da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874037070 con 0x7f58840c26c0 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5874030510 con 0x7f58840c26c0 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f5882ffd640 1 -- 172.21.15.94:0/2745937631 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5878001670 con 0x7f58840c26c0 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f588a5d7640 1 -- 172.21.15.94:0/2745937631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 msgr2=0x7f5884155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2745937631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 0x7f5884155bd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f58740097a0 tx=0x7f587402b710 comp rx=0 tx=0).stop 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f588a5d7640 1 -- 172.21.15.94:0/2745937631 shutdown_connections 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2745937631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 0x7f5884155bd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f588a5d7640 1 -- 172.21.15.94:0/2745937631 >> 172.21.15.94:0/2745937631 conn(0x7f58840c3b40 msgr2=0x7f58840c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f588a5d7640 1 -- 172.21.15.94:0/2745937631 shutdown_connections 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.937+0000 7f588a5d7640 1 -- 172.21.15.94:0/2745937631 wait complete. 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.938+0000 7f588a5d7640 1 Processor -- start 2024-03-06T07:20:33.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.938+0000 7f588a5d7640 1 -- start start 2024-03-06T07:20:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f588a5d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58840c0470 0x7f58840ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f588a5d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 0x7f588414ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f588a5d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58840c2aa0 0x7f58840aa560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f588a5d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58840c20b0 con 0x7f58840c0470 2024-03-06T07:20:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f588a5d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58840c1f30 con 0x7f58840c26c0 2024-03-06T07:20:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f588a5d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5884066d00 con 0x7f58840c2aa0 2024-03-06T07:20:33.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f5883fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58840c0470 0x7f58840ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f5883fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58840c0470 0x7f58840ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53332/0 (socket says 172.21.15.94:53332) 2024-03-06T07:20:33.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f5883fff640 1 -- 172.21.15.94:0/2812207377 learned_addr learned my addr 172.21.15.94:0/2812207377 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:33.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f58837fe640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 0x7f588414ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.939+0000 7f5888b74640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58840c2aa0 0x7f58840aa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5883fff640 1 -- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58840c2aa0 msgr2=0x7f58840aa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5883fff640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58840c2aa0 0x7f58840aa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5883fff640 1 -- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 msgr2=0x7f588414ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5883fff640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 0x7f588414ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5883fff640 1 -- 172.21.15.94:0/2812207377 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58841eb310 con 0x7f58840c0470 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5883fff640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58840c0470 0x7f58840ab7e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f587402b750 tx=0x7f58740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5880ff9640 1 -- 172.21.15.94:0/2812207377 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874041020 con 0x7f58840c0470 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5880ff9640 1 -- 172.21.15.94:0/2812207377 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587403c870 con 0x7f58840c0470 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f5880ff9640 1 -- 172.21.15.94:0/2812207377 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874037040 con 0x7f58840c0470 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588414e150 con 0x7f58840c0470 2024-03-06T07:20:33.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.940+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588414f170 con 0x7f58840c0470 2024-03-06T07:20:33.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.941+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588404f330 con 0x7f58840c0470 2024-03-06T07:20:33.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.941+0000 7f5880ff9640 1 -- 172.21.15.94:0/2812207377 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5874044020 con 0x7f58840c0470 2024-03-06T07:20:33.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.942+0000 7f5880ff9640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f587006b500 0x7f587006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:33.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.942+0000 7f5880ff9640 1 -- 172.21.15.94:0/2812207377 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6535+0+0 (secure 0 0 0) 0x7f58740abb30 con 0x7f58840c0470 2024-03-06T07:20:33.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.942+0000 7f58837fe640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f587006b500 0x7f587006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:33.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.942+0000 7f58837fe640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f587006b500 0x7f587006d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f58841e85a0 tx=0x7f58780016b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:33.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:33.946+0000 7f5880ff9640 1 -- 172.21.15.94:0/2812207377 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f587404b070 con 0x7f58840c0470 2024-03-06T07:20:34.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.156+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f58840c47b0 con 0x7f58840c0470 2024-03-06T07:20:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.820+0000 7f5880ff9640 1 -- 172.21.15.94:0/2812207377 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v49) v1 ==== 105+0+0 (secure 0 0 0) 0x7f587407a0b0 con 0x7f58840c0470 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.822+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f587006b500 msgr2=0x7f587006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.822+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f587006b500 0x7f587006d9c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f58841e85a0 tx=0x7f58780016b0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.822+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58840c0470 msgr2=0x7f58840ab7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.822+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58840c0470 0x7f58840ab7e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f587402b750 tx=0x7f58740046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 shutdown_connections 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f587006b500 0x7f587006d9c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58840c2aa0 0x7f58840aa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58840c26c0 0x7f588414ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 --2- 172.21.15.94:0/2812207377 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58840c0470 0x7f58840ab7e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 >> 172.21.15.94:0/2812207377 conn(0x7f58840c3b40 msgr2=0x7f5884155790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 shutdown_connections 2024-03-06T07:20:34.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.823+0000 7f588a5d7640 1 -- 172.21.15.94:0/2812207377 wait complete. 2024-03-06T07:20:34.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-06T07:20:34.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.943+0000 7fd591b63640 1 Processor -- start 2024-03-06T07:20:34.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.943+0000 7fd591b63640 1 -- start start 2024-03-06T07:20:34.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd591b63640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd58c0abb60 0x7fd58c1502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd591b63640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd58c0c2920 0x7fd58c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd591b63640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd58c070430 con 0x7fd58c0c1d10 2024-03-06T07:20:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd591b63640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c0711f0 con 0x7fd58c0c2920 2024-03-06T07:20:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd591b63640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c070d60 con 0x7fd58c0abb60 2024-03-06T07:20:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd58c0c2920 0x7fd58c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd58c0c2920 0x7fd58c156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53342/0 (socket says 172.21.15.94:53342) 2024-03-06T07:20:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58bfff640 1 -- 172.21.15.94:0/3791051657 learned_addr learned my addr 172.21.15.94:0/3791051657 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58b7fe640 1 --2- 172.21.15.94:0/3791051657 >> v2:172.21.15.145:3300/0 conn(0x7fd58c0abb60 0x7fd58c1502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58bfff640 1 -- 172.21.15.94:0/3791051657 >> v2:172.21.15.145:3300/0 conn(0x7fd58c0abb60 msgr2=0x7fd58c1502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58bfff640 1 --2- 172.21.15.94:0/3791051657 >> v2:172.21.15.145:3300/0 conn(0x7fd58c0abb60 0x7fd58c1502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58bfff640 1 -- 172.21.15.94:0/3791051657 >> v1:172.21.15.145:6789/0 conn(0x7fd58c0c1d10 legacy=0x7fd58c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.944+0000 7fd58bfff640 1 -- 172.21.15.94:0/3791051657 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c070710 con 0x7fd58c0c2920 2024-03-06T07:20:34.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.945+0000 7fd58bfff640 1 --2- 172.21.15.94:0/3791051657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c0c2920 0x7fd58c156b50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd58c06d150 tx=0x7fd58402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c85bf61af98d73d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.945+0000 7fd58a7fc640 1 -- 172.21.15.94:0/3791051657 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58402c960 con 0x7fd58c0c2920 2024-03-06T07:20:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.945+0000 7fd58a7fc640 1 -- 172.21.15.94:0/3791051657 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58402cb00 con 0x7fd58c0c2920 2024-03-06T07:20:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.945+0000 7fd58a7fc640 1 -- 172.21.15.94:0/3791051657 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5840343f0 con 0x7fd58c0c2920 2024-03-06T07:20:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 -- 172.21.15.94:0/3791051657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c0c2920 msgr2=0x7fd58c156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 --2- 172.21.15.94:0/3791051657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c0c2920 0x7fd58c156b50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd58c06d150 tx=0x7fd58402bdc0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 -- 172.21.15.94:0/3791051657 shutdown_connections 2024-03-06T07:20:34.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 --2- 172.21.15.94:0/3791051657 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c0c2920 0x7fd58c156b50 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 --2- 172.21.15.94:0/3791051657 >> v2:172.21.15.145:3300/0 conn(0x7fd58c0abb60 0x7fd58c1502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 -- 172.21.15.94:0/3791051657 >> 172.21.15.94:0/3791051657 conn(0x7fd58c0c65b0 msgr2=0x7fd58c0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 -- 172.21.15.94:0/3791051657 shutdown_connections 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.946+0000 7fd591b63640 1 -- 172.21.15.94:0/3791051657 wait complete. 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.947+0000 7fd591b63640 1 Processor -- start 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.947+0000 7fd591b63640 1 -- start start 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd58c0abb60 0x7fd58c1e8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd58c0c2920 0x7fd58c153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c1dfde0 0x7fd58c1ec710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c0abf40 con 0x7fd58c1dfde0 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c157090 con 0x7fd58c0c2920 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c0ac0c0 con 0x7fd58c0abb60 2024-03-06T07:20:34.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c1dfde0 0x7fd58c1ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c1dfde0 0x7fd58c1ec710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53352/0 (socket says 172.21.15.94:53352) 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 -- 172.21.15.94:0/982142749 learned_addr learned my addr 172.21.15.94:0/982142749 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 -- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd58c0abb60 msgr2=0x7fd58c1e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd58c0abb60 0x7fd58c1e8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 -- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd58c0c2920 msgr2=0x7fd58c153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd58c0c2920 0x7fd58c153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 -- 172.21.15.94:0/982142749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c1dcc60 con 0x7fd58c1dfde0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd58bfff640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c1dfde0 0x7fd58c1ec710 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd584036040 tx=0x7fd584004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd573fff640 1 -- 172.21.15.94:0/982142749 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584043020 con 0x7fd58c1dfde0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd573fff640 1 -- 172.21.15.94:0/982142749 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58403e450 con 0x7fd58c1dfde0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c1dcf40 con 0x7fd58c1dfde0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd573fff640 1 -- 172.21.15.94:0/982142749 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd584009510 con 0x7fd58c1dfde0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.948+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c1dc2b0 con 0x7fd58c1dfde0 2024-03-06T07:20:34.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.949+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58c04f330 con 0x7fd58c1dfde0 2024-03-06T07:20:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.950+0000 7fd573fff640 1 -- 172.21.15.94:0/982142749 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd584046020 con 0x7fd58c1dfde0 2024-03-06T07:20:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.950+0000 7fd573fff640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.950+0000 7fd573fff640 1 -- 172.21.15.94:0/982142749 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6535+0+0 (secure 0 0 0) 0x7fd5840acd50 con 0x7fd58c1dfde0 2024-03-06T07:20:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.950+0000 7fd58b7fe640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.951+0000 7fd58b7fe640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0098f0 tx=0x7fd57c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:34.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:34.954+0000 7fd573fff640 1 -- 172.21.15.94:0/982142749 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5840b0030 con 0x7fd58c1dfde0 2024-03-06T07:20:35.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.129+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd58c0a8f80 con 0x7fd58c1dfde0 2024-03-06T07:20:35.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.821+0000 7fd573fff640 1 -- 172.21.15.94:0/982142749 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v64) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd58404d030 con 0x7fd58c1dfde0 2024-03-06T07:20:35.824 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_metadata' removed 2024-03-06T07:20:35.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd57806b4d0 msgr2=0x7fd57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0098f0 tx=0x7fd57c008040 comp rx=0 tx=0).stop 2024-03-06T07:20:35.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c1dfde0 msgr2=0x7fd58c1ec710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c1dfde0 0x7fd58c1ec710 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd584036040 tx=0x7fd584004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 shutdown_connections 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd58c1dfde0 0x7fd58c1ec710 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd58c0c2920 0x7fd58c153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 --2- 172.21.15.94:0/982142749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd58c0abb60 0x7fd58c1e8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 >> 172.21.15.94:0/982142749 conn(0x7fd58c0c65b0 msgr2=0x7fd58c06d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 shutdown_connections 2024-03-06T07:20:35.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.824+0000 7fd591b63640 1 -- 172.21.15.94:0/982142749 wait complete. 2024-03-06T07:20:35.839 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.942+0000 7f0b15633640 1 Processor -- start 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.942+0000 7f0b15633640 1 -- start start 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b15633640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b10153940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b15633640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0b10072fc0 0x7f0b100724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b15633640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0b10072a10 0x7f0b100c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b15633640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b10153e80 con 0x7f0b10072a10 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b15633640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b100c6220 con 0x7f0b10072fc0 2024-03-06T07:20:35.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b15633640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b10070210 con 0x7f0b100c5e40 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0b10072a10 0x7f0b100c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0b10072a10 0x7f0b100c46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53358/0 (socket says 172.21.15.94:53358) 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 -- 172.21.15.94:0/1643716526 learned_addr learned my addr 172.21.15.94:0/1643716526 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0e7fc640 1 --2- 172.21.15.94:0/1643716526 >> v2:172.21.15.145:3300/0 conn(0x7f0b10072fc0 0x7f0b100724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0effd640 1 --2- 172.21.15.94:0/1643716526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b10153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 -- 172.21.15.94:0/1643716526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 msgr2=0x7f0b10153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 --2- 172.21.15.94:0/1643716526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b10153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 -- 172.21.15.94:0/1643716526 >> v2:172.21.15.145:3300/0 conn(0x7f0b10072fc0 msgr2=0x7f0b100724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 --2- 172.21.15.94:0/1643716526 >> v2:172.21.15.145:3300/0 conn(0x7f0b10072fc0 0x7f0b100724d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.943+0000 7f0b0f7fe640 1 -- 172.21.15.94:0/1643716526 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100704d0 con 0x7f0b10072a10 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.944+0000 7f0b0f7fe640 1 --2- 172.21.15.94:0/1643716526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072a10 0x7f0b100c46d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0b10073690 tx=0x7f0b0802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e679fa7f8a7565e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.944+0000 7f0b0dffb640 1 -- 172.21.15.94:0/1643716526 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0802c9e0 con 0x7f0b10072a10 2024-03-06T07:20:35.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.944+0000 7f0b0dffb640 1 -- 172.21.15.94:0/1643716526 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0802cb80 con 0x7f0b10072a10 2024-03-06T07:20:35.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.944+0000 7f0b0dffb640 1 -- 172.21.15.94:0/1643716526 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b080343f0 con 0x7f0b10072a10 2024-03-06T07:20:35.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 -- 172.21.15.94:0/1643716526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072a10 msgr2=0x7f0b100c46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 --2- 172.21.15.94:0/1643716526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072a10 0x7f0b100c46d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0b10073690 tx=0x7f0b0802be40 comp rx=0 tx=0).stop 2024-03-06T07:20:35.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 -- 172.21.15.94:0/1643716526 shutdown_connections 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 --2- 172.21.15.94:0/1643716526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072a10 0x7f0b100c46d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 --2- 172.21.15.94:0/1643716526 >> v2:172.21.15.145:3300/0 conn(0x7f0b10072fc0 0x7f0b100724d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 --2- 172.21.15.94:0/1643716526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b10153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 -- 172.21.15.94:0/1643716526 >> 172.21.15.94:0/1643716526 conn(0x7f0b100c88f0 msgr2=0x7f0b100c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 -- 172.21.15.94:0/1643716526 shutdown_connections 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.945+0000 7f0b15633640 1 -- 172.21.15.94:0/1643716526 wait complete. 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.946+0000 7f0b15633640 1 Processor -- start 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.946+0000 7f0b15633640 1 -- start start 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b15633640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b10072a10 0x7f0b10156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b15633640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072fc0 0x7f0b101dc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b15633640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b101dca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b15633640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1006fbd0 con 0x7f0b10072fc0 2024-03-06T07:20:35.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b15633640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1006f8f0 con 0x7f0b10072a10 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b15633640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b100c4c10 con 0x7f0b100c5e40 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072fc0 0x7f0b101dc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072fc0 0x7f0b101dc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53360/0 (socket says 172.21.15.94:53360) 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 -- 172.21.15.94:0/3681474323 learned_addr learned my addr 172.21.15.94:0/3681474323 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 -- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 msgr2=0x7f0b101dca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0f7fe640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b101dca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0effd640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b10072a10 0x7f0b10156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b101dca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 -- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b10072a10 msgr2=0x7f0b10156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b10072a10 0x7f0b10156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 -- 172.21.15.94:0/3681474323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b101de2f0 con 0x7f0b10072fc0 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0b0e7fc640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072fc0 0x7f0b101dc520 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400d800 tx=0x7f0b0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0af77fe640 1 -- 172.21.15.94:0/3681474323 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0400b6c0 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.947+0000 7f0af77fe640 1 -- 172.21.15.94:0/3681474323 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b04019070 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.948+0000 7f0af77fe640 1 -- 172.21.15.94:0/3681474323 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0400baa0 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.948+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b101de5d0 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.948+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b101df950 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.948+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1004f330 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.949+0000 7f0af77fe640 1 -- 172.21.15.94:0/3681474323 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b0400bc40 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.949+0000 7f0af77fe640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0afc06b4d0 0x7f0afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.949+0000 7f0af77fe640 1 -- 172.21.15.94:0/3681474323 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6095+0+0 (secure 0 0 0) 0x7f0b0408a6e0 con 0x7f0b10072fc0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.949+0000 7f0b0effd640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0afc06b4d0 0x7f0afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:35.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.950+0000 7f0b0effd640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0afc06b4d0 0x7f0afc06d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0b00006fd0 tx=0x7f0b00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:35.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:35.953+0000 7f0af77fe640 1 -- 172.21.15.94:0/3681474323 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b04059470 con 0x7f0b10072fc0 2024-03-06T07:20:36.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.125+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0b1006f3f0 con 0x7f0b10072fc0 2024-03-06T07:20:36.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.825+0000 7f0af77fe640 1 -- 172.21.15.94:0/3681474323 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v65) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0b04058d70 con 0x7f0b10072fc0 2024-03-06T07:20:36.828 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_data' removed 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.828+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0afc06b4d0 msgr2=0x7f0afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.828+0000 7f0b15633640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0afc06b4d0 0x7f0afc06d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0b00006fd0 tx=0x7f0b00008040 comp rx=0 tx=0).stop 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.828+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072fc0 msgr2=0x7f0b101dc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.828+0000 7f0b15633640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072fc0 0x7f0b101dc520 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400d800 tx=0x7f0b0400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 shutdown_connections 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0afc06b4d0 0x7f0afc06d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b100c5e40 0x7f0b101dca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b10072fc0 0x7f0b101dc520 secure :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400d800 tx=0x7f0b0400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 --2- 172.21.15.94:0/3681474323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b10072a10 0x7f0b10156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 >> 172.21.15.94:0/3681474323 conn(0x7f0b100c88f0 msgr2=0x7f0b100ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:36.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 shutdown_connections 2024-03-06T07:20:36.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.829+0000 7f0b15633640 1 -- 172.21.15.94:0/3681474323 wait complete. 2024-03-06T07:20:36.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:20:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.948+0000 7fde083e9640 1 Processor -- start 2024-03-06T07:20:36.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.948+0000 7fde083e9640 1 -- start start 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde083e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde000c5990 0x7fde000c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde083e9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fde0006c900 0x7fde000a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde083e9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde000c3ef0 con 0x7fde000ab170 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde083e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde001556d0 con 0x7fde000c5990 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde083e9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde001559b0 con 0x7fde0006c900 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde06185640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde000c5990 0x7fde000c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde06185640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde000c5990 0x7fde000c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53362/0 (socket says 172.21.15.94:53362) 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde06185640 1 -- 172.21.15.94:0/1270413316 learned_addr learned my addr 172.21.15.94:0/1270413316 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde06185640 1 -- 172.21.15.94:0/1270413316 >> v2:172.21.15.145:3300/0 conn(0x7fde0006c900 msgr2=0x7fde000a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde06185640 1 --2- 172.21.15.94:0/1270413316 >> v2:172.21.15.145:3300/0 conn(0x7fde0006c900 0x7fde000a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde06185640 1 -- 172.21.15.94:0/1270413316 >> v1:172.21.15.145:6789/0 conn(0x7fde000ab170 legacy=0x7fde0006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.949+0000 7fde06185640 1 -- 172.21.15.94:0/1270413316 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00155c90 con 0x7fde000c5990 2024-03-06T07:20:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.950+0000 7fde06185640 1 --2- 172.21.15.94:0/1270413316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000c5990 0x7fde000c0ab0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009870 tx=0x7fddfc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a370eba6873d0678 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.950+0000 7fde05183640 1 -- 172.21.15.94:0/1270413316 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc035020 con 0x7fde000c5990 2024-03-06T07:20:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.950+0000 7fde05183640 1 -- 172.21.15.94:0/1270413316 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc031870 con 0x7fde000c5990 2024-03-06T07:20:36.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.950+0000 7fde05183640 1 -- 172.21.15.94:0/1270413316 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc031c40 con 0x7fde000c5990 2024-03-06T07:20:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 -- 172.21.15.94:0/1270413316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000c5990 msgr2=0x7fde000c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 --2- 172.21.15.94:0/1270413316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000c5990 0x7fde000c0ab0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009870 tx=0x7fddfc02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 -- 172.21.15.94:0/1270413316 shutdown_connections 2024-03-06T07:20:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 --2- 172.21.15.94:0/1270413316 >> v2:172.21.15.145:3300/0 conn(0x7fde0006c900 0x7fde000a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 --2- 172.21.15.94:0/1270413316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000c5990 0x7fde000c0ab0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 -- 172.21.15.94:0/1270413316 >> 172.21.15.94:0/1270413316 conn(0x7fde000c6710 msgr2=0x7fde000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 -- 172.21.15.94:0/1270413316 shutdown_connections 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.951+0000 7fde083e9640 1 -- 172.21.15.94:0/1270413316 wait complete. 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.952+0000 7fde083e9640 1 Processor -- start 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.952+0000 7fde083e9640 1 -- start start 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.952+0000 7fde083e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006c900 0x7fde0014f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde083e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000ab170 0x7fde001ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde083e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde000c5990 0x7fde001ec800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde083e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde00156610 con 0x7fde000ab170 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde083e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde00156490 con 0x7fde0006c900 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde083e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde000c3af0 con 0x7fde000c5990 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000ab170 0x7fde001ec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000ab170 0x7fde001ec2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53370/0 (socket says 172.21.15.94:53370) 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 -- 172.21.15.94:0/2531913183 learned_addr learned my addr 172.21.15.94:0/2531913183 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 -- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde000c5990 msgr2=0x7fde001ec800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde000c5990 0x7fde001ec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 -- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006c900 msgr2=0x7fde0014f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde06185640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006c900 0x7fde0014f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006c900 0x7fde0014f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 -- 172.21.15.94:0/2531913183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00151ee0 con 0x7fde000ab170 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde05984640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000ab170 0x7fde001ec2c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c410 tx=0x7fddf000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fddf6ffd640 1 -- 172.21.15.94:0/2531913183 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0017070 con 0x7fde000ab170 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fddf6ffd640 1 -- 172.21.15.94:0/2531913183 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf00035f0 con 0x7fde000ab170 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fddf6ffd640 1 -- 172.21.15.94:0/2531913183 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf00039e0 con 0x7fde000ab170 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde001521c0 con 0x7fde000ab170 2024-03-06T07:20:36.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.953+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde00152780 con 0x7fde000ab170 2024-03-06T07:20:36.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.954+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0004f330 con 0x7fde000ab170 2024-03-06T07:20:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.955+0000 7fddf6ffd640 1 -- 172.21.15.94:0/2531913183 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddf0003b80 con 0x7fde000ab170 2024-03-06T07:20:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.955+0000 7fddf6ffd640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.955+0000 7fddf6ffd640 1 -- 172.21.15.94:0/2531913183 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fddf0089530 con 0x7fde000ab170 2024-03-06T07:20:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.955+0000 7fde06185640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.956+0000 7fde06185640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fddfc036000 tx=0x7fddfc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:36.959+0000 7fddf6ffd640 1 -- 172.21.15.94:0/2531913183 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddf00583d0 con 0x7fde000ab170 2024-03-06T07:20:37.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.129+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fde000c1740 con 0x7fde000ab170 2024-03-06T07:20:37.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.133+0000 7fddf6ffd640 1 -- 172.21.15.94:0/2531913183 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v19) v1 ==== 96+0+0 (secure 0 0 0) 0x7fddf0057cd0 con 0x7fde000ab170 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 msgr2=0x7fdde406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fddfc036000 tx=0x7fddfc03c040 comp rx=0 tx=0).stop 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000ab170 msgr2=0x7fde001ec2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000ab170 0x7fde001ec2c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c410 tx=0x7fddf000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 shutdown_connections 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde000c5990 0x7fde001ec800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde000ab170 0x7fde001ec2c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 --2- 172.21.15.94:0/2531913183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde0006c900 0x7fde0014f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 >> 172.21.15.94:0/2531913183 conn(0x7fde000c6710 msgr2=0x7fde0006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 shutdown_connections 2024-03-06T07:20:37.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.135+0000 7fde083e9640 1 -- 172.21.15.94:0/2531913183 wait complete. 2024-03-06T07:20:37.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:20:37.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.248+0000 7f1737be9640 1 Processor -- start 2024-03-06T07:20:37.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.248+0000 7f1737be9640 1 -- start start 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.248+0000 7f1737be9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f173006ca70 0x7f173014e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.248+0000 7f1737be9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17300b0450 con 0x7f1730155c30 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.248+0000 7f1737be9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17301560e0 con 0x7f17300b0070 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.248+0000 7f1737be9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173006c020 con 0x7f173006ca70 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.248+0000 7f1736186640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f173006ca70 0x7f173014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1735184640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1730155c30 0x7f17300c5380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45980/0 (socket says 172.21.15.94:45980) 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1736186640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f173006ca70 0x7f173014e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53372/0 (socket says 172.21.15.94:53372) 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1735184640 1 -- 172.21.15.94:0/839756688 learned_addr learned my addr 172.21.15.94:0/839756688 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1736186640 1 -- 172.21.15.94:0/839756688 >> v1:172.21.15.145:6789/0 conn(0x7f17300b0070 legacy=0x7f17300c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1736186640 1 -- 172.21.15.94:0/839756688 >> v1:172.21.15.94:6789/0 conn(0x7f1730155c30 legacy=0x7f17300c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1736186640 1 -- 172.21.15.94:0/839756688 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17300c77f0 con 0x7f173006ca70 2024-03-06T07:20:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1734983640 1 -- 172.21.15.94:0/839756688 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145089236 0 0) 0x7f1724001110 con 0x7f1730155c30 2024-03-06T07:20:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.249+0000 7f1736186640 1 --2- 172.21.15.94:0/839756688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f173006ca70 0x7f173014e800 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f17200097a0 tx=0x7f172002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f926fb20f7ca4c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1734983640 1 -- 172.21.15.94:0/839756688 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17200315a0 con 0x7f173006ca70 2024-03-06T07:20:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1734983640 1 -- 172.21.15.94:0/839756688 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1720037070 con 0x7f173006ca70 2024-03-06T07:20:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1734983640 1 -- 172.21.15.94:0/839756688 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720031960 con 0x7f173006ca70 2024-03-06T07:20:37.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1737be9640 1 -- 172.21.15.94:0/839756688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f173006ca70 msgr2=0x7f173014e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1737be9640 1 --2- 172.21.15.94:0/839756688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f173006ca70 0x7f173014e800 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f17200097a0 tx=0x7f172002b840 comp rx=0 tx=0).stop 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1737be9640 1 -- 172.21.15.94:0/839756688 shutdown_connections 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1737be9640 1 --2- 172.21.15.94:0/839756688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f173006ca70 0x7f173014e800 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1737be9640 1 -- 172.21.15.94:0/839756688 >> 172.21.15.94:0/839756688 conn(0x7f17300ac590 msgr2=0x7f17300ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1737be9640 1 -- 172.21.15.94:0/839756688 shutdown_connections 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.250+0000 7f1737be9640 1 -- 172.21.15.94:0/839756688 wait complete. 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.251+0000 7f1737be9640 1 Processor -- start 2024-03-06T07:20:37.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.251+0000 7f1737be9640 1 -- start start 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1737be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f173006ca70 0x7f173014aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1737be9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17300b0070 0x7f1730154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1737be9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1730155c30 0x7f1730157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1737be9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173006c550 con 0x7f1730155c30 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1737be9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173006c270 con 0x7f17300b0070 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1737be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173006da80 con 0x7f173006ca70 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1730155c30 0x7f1730157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1730155c30 0x7f1730157640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53386/0 (socket says 172.21.15.94:53386) 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 -- 172.21.15.94:0/2246415379 learned_addr learned my addr 172.21.15.94:0/2246415379 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 -- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f173006ca70 msgr2=0x7f173014aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f173006ca70 0x7f173014aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 -- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17300b0070 msgr2=0x7f1730154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17300b0070 0x7f1730154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 -- 172.21.15.94:0/2246415379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17300c2c60 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.252+0000 7f1736186640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1730155c30 0x7f1730157640 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f17200097a0 tx=0x7f17200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.253+0000 7f172e7fc640 1 -- 172.21.15.94:0/2246415379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720032840 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.253+0000 7f172e7fc640 1 -- 172.21.15.94:0/2246415379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17200329e0 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.253+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17301eef80 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.253+0000 7f172e7fc640 1 -- 172.21.15.94:0/2246415379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1720032db0 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.253+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173014e7f0 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.254+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f173004f330 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.254+0000 7f172e7fc640 1 -- 172.21.15.94:0/2246415379 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f172004c020 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.254+0000 7f172e7fc640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f171c06b4d0 0x7f171c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.254+0000 7f172e7fc640 1 -- 172.21.15.94:0/2246415379 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f172002f020 con 0x7f1730155c30 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.255+0000 7f1735985640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f171c06b4d0 0x7f171c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.255+0000 7f1735985640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f171c06b4d0 0x7f171c06d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1728001c60 tx=0x7f1728007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.259+0000 7f172e7fc640 1 -- 172.21.15.94:0/2246415379 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1720040070 con 0x7f1730155c30 2024-03-06T07:20:37.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.437+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f17300af690 con 0x7f1730155c30 2024-03-06T07:20:37.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.440+0000 7f172e7fc640 1 -- 172.21.15.94:0/2246415379 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f172007aab0 con 0x7f1730155c30 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f171c06b4d0 msgr2=0x7f171c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f171c06b4d0 0x7f171c06d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1728001c60 tx=0x7f1728007d10 comp rx=0 tx=0).stop 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1730155c30 msgr2=0x7f1730157640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1730155c30 0x7f1730157640 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f17200097a0 tx=0x7f17200046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 shutdown_connections 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f171c06b4d0 0x7f171c06d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1730155c30 0x7f1730157640 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17300b0070 0x7f1730154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 --2- 172.21.15.94:0/2246415379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f173006ca70 0x7f173014aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 >> 172.21.15.94:0/2246415379 conn(0x7f17300ac590 msgr2=0x7f17300ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 shutdown_connections 2024-03-06T07:20:37.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.443+0000 7f1737be9640 1 -- 172.21.15.94:0/2246415379 wait complete. 2024-03-06T07:20:37.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:20:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.556+0000 7fdb05a6f640 1 Processor -- start 2024-03-06T07:20:37.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.556+0000 7fdb05a6f640 1 -- start start 2024-03-06T07:20:37.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.556+0000 7fdb05a6f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 0x7fdb000c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.556+0000 7fdb05a6f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb0006c900 0x7fdb000a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.556+0000 7fdb05a6f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb000c3ef0 con 0x7fdb000ab170 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdb05a6f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb001556d0 con 0x7fdb0006c900 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdb05a6f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb001559b0 con 0x7fdb000c5990 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafeffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdb000ab170 0x7fdb0006c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45982/0 (socket says 172.21.15.94:45982) 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafeffd640 1 -- 172.21.15.94:0/3040379349 learned_addr learned my addr 172.21.15.94:0/3040379349 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdaff7fe640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 0x7fdb000c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafffff640 1 --2- 172.21.15.94:0/3040379349 >> v2:172.21.15.145:3300/0 conn(0x7fdb0006c900 0x7fdb000a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974401486 0 0) 0x7fdb000c3ef0 con 0x7fdb000ab170 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae8003520 con 0x7fdb000ab170 2024-03-06T07:20:37.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3878542090 0 0) 0x7fdae8003520 con 0x7fdb000ab170 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 msgr2=0x7fdb000c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 0x7fdb000c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 >> v2:172.21.15.145:3300/0 conn(0x7fdb0006c900 msgr2=0x7fdb000a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 --2- 172.21.15.94:0/3040379349 >> v2:172.21.15.145:3300/0 conn(0x7fdb0006c900 0x7fdb000a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb00155c90 con 0x7fdb000ab170 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fdaf4001670 con 0x7fdb000ab170 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 >> v1:172.21.15.94:6789/0 conn(0x7fdb000ab170 legacy=0x7fdb0006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 0x7fdae80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdae8003f30 0x7fdae8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdae8004d40 0x7fdae80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb001556d0 con 0x7fdae8004d40 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb001559b0 con 0x7fdae8003f30 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae800b9c0 con 0x7fdb000c5990 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafffff640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdae8004d40 0x7fdae80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdaff7fe640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 0x7fdae80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.557+0000 7fdafeffd640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdae8003f30 0x7fdae8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafffff640 1 -- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 msgr2=0x7fdae80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafffff640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb000c5990 0x7fdae80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafffff640 1 -- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdae8003f30 msgr2=0x7fdae8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafffff640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdae8003f30 0x7fdae8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafffff640 1 -- 172.21.15.94:0/3040379349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae8003520 con 0x7fdae8004d40 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafffff640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdae8004d40 0x7fdae80051c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009f90 tx=0x7fdae8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876f4af7198a66dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8039070 con 0x7fdae8004d40 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf802cc00 con 0x7fdae8004d40 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.558+0000 7fdafe7fc640 1 -- 172.21.15.94:0/3040379349 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8009c10 con 0x7fdae8004d40 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.559+0000 7fdb05a6f640 1 -- 172.21.15.94:0/3040379349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdae8004d40 msgr2=0x7fdae80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.559+0000 7fdb05a6f640 1 --2- 172.21.15.94:0/3040379349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdae8004d40 0x7fdae80051c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009f90 tx=0x7fdae8005830 comp rx=0 tx=0).stop 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.559+0000 7fdafffff640 1 -- 172.21.15.94:0/3040379349 reap_dead start 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.559+0000 7fdb05a6f640 1 -- 172.21.15.94:0/3040379349 shutdown_connections 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.559+0000 7fdb05a6f640 1 -- 172.21.15.94:0/3040379349 >> 172.21.15.94:0/3040379349 conn(0x7fdb000c6710 msgr2=0x7fdb000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.559+0000 7fdb05a6f640 1 -- 172.21.15.94:0/3040379349 shutdown_connections 2024-03-06T07:20:37.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.559+0000 7fdb05a6f640 1 -- 172.21.15.94:0/3040379349 wait complete. 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.560+0000 7fdb05a6f640 1 Processor -- start 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.560+0000 7fdb05a6f640 1 -- start start 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdb05a6f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdae8004d40 0x7fdb001ec770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdb05a6f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb00154470 0x7fdb0014bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdb05a6f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb001e8680 0x7fdb00148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdb05a6f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb00156610 con 0x7fdb001e8680 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdb05a6f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb00156490 con 0x7fdb00154470 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdb05a6f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb000c3af0 con 0x7fdae8004d40 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb001e8680 0x7fdb00148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb001e8680 0x7fdb00148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53402/0 (socket says 172.21.15.94:53402) 2024-03-06T07:20:37.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 -- 172.21.15.94:0/659348608 learned_addr learned my addr 172.21.15.94:0/659348608 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafeffd640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb00154470 0x7fdb0014bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdaff7fe640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdae8004d40 0x7fdb001ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 -- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdae8004d40 msgr2=0x7fdb001ec770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdae8004d40 0x7fdb001ec770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 -- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb00154470 msgr2=0x7fdb0014bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb00154470 0x7fdb0014bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 -- 172.21.15.94:0/659348608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb000c0e20 con 0x7fdb001e8680 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafeffd640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb00154470 0x7fdb0014bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.561+0000 7fdafffff640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb001e8680 0x7fdb00148950 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8037040 tx=0x7fdaf8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.562+0000 7fdae7fff640 1 -- 172.21.15.94:0/659348608 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8039030 con 0x7fdb001e8680 2024-03-06T07:20:37.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.562+0000 7fdae7fff640 1 -- 172.21.15.94:0/659348608 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf803a040 con 0x7fdb001e8680 2024-03-06T07:20:37.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.562+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb001551b0 con 0x7fdb001e8680 2024-03-06T07:20:37.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.562+0000 7fdae7fff640 1 -- 172.21.15.94:0/659348608 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8003bd0 con 0x7fdb001e8680 2024-03-06T07:20:37.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.562+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb001ebc90 con 0x7fdb001e8680 2024-03-06T07:20:37.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.563+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0004f330 con 0x7fdb001e8680 2024-03-06T07:20:37.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.563+0000 7fdae7fff640 1 -- 172.21.15.94:0/659348608 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdaf8004030 con 0x7fdb001e8680 2024-03-06T07:20:37.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.564+0000 7fdae7fff640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaec06b4d0 0x7fdaec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.564+0000 7fdae7fff640 1 -- 172.21.15.94:0/659348608 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fdaf80ac4d0 con 0x7fdb001e8680 2024-03-06T07:20:37.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.564+0000 7fdaff7fe640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaec06b4d0 0x7fdaec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.564+0000 7fdaff7fe640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaec06b4d0 0x7fdaec06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdb001e97b0 tx=0x7fdaf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.568+0000 7fdae7fff640 1 -- 172.21.15.94:0/659348608 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdaf807b400 con 0x7fdb001e8680 2024-03-06T07:20:37.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.737+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fdb000c1740 con 0x7fdb001e8680 2024-03-06T07:20:37.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.740+0000 7fdae7fff640 1 -- 172.21.15.94:0/659348608 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v21) v1 ==== 96+0+0 (secure 0 0 0) 0x7fdaf807ad00 con 0x7fdb001e8680 2024-03-06T07:20:37.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaec06b4d0 msgr2=0x7fdaec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaec06b4d0 0x7fdaec06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdb001e97b0 tx=0x7fdaf0008040 comp rx=0 tx=0).stop 2024-03-06T07:20:37.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb001e8680 msgr2=0x7fdb00148950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb001e8680 0x7fdb00148950 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8037040 tx=0x7fdaf8009750 comp rx=0 tx=0).stop 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 shutdown_connections 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaec06b4d0 0x7fdaec06d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb001e8680 0x7fdb00148950 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb00154470 0x7fdb0014bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 --2- 172.21.15.94:0/659348608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdae8004d40 0x7fdb001ec770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 >> 172.21.15.94:0/659348608 conn(0x7fdb000c6710 msgr2=0x7fdb001ddfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 shutdown_connections 2024-03-06T07:20:37.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.742+0000 7fdb05a6f640 1 -- 172.21.15.94:0/659348608 wait complete. 2024-03-06T07:20:37.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-06T07:20:37.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.855+0000 7f6e8e97a640 1 Processor -- start 2024-03-06T07:20:37.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.855+0000 7f6e8e97a640 1 -- start start 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e8e97a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e880b0070 0x7f6e88155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e8e97a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e880b0450 con 0x7f6e8806ca70 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e8e97a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8806cf20 con 0x7f6e880c5380 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e8e97a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8806c020 con 0x7f6e880b0070 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e87fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e880b0070 0x7f6e88155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e87fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e880b0070 0x7f6e88155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53410/0 (socket says 172.21.15.94:53410) 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e87fff640 1 -- 172.21.15.94:0/3722315000 learned_addr learned my addr 172.21.15.94:0/3722315000 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e877fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6e8806ca70 0x7f6e880c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45986/0 (socket says 172.21.15.94:45986) 2024-03-06T07:20:37.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e87fff640 1 -- 172.21.15.94:0/3722315000 >> v1:172.21.15.145:6789/0 conn(0x7f6e880c5380 legacy=0x7f6e8814e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:37.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e87fff640 1 -- 172.21.15.94:0/3722315000 >> v1:172.21.15.94:6789/0 conn(0x7f6e8806ca70 legacy=0x7f6e880c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e87fff640 1 -- 172.21.15.94:0/3722315000 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e880c77f0 con 0x7f6e880b0070 2024-03-06T07:20:37.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e86ffd640 1 -- 172.21.15.94:0/3722315000 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007732571 0 0) 0x7f6e7c001110 con 0x7f6e8806ca70 2024-03-06T07:20:37.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e87fff640 1 --2- 172.21.15.94:0/3722315000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e88155c30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009870 tx=0x7f6e7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0bd789f6b837d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e86ffd640 1 -- 172.21.15.94:0/3722315000 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78035020 con 0x7f6e880b0070 2024-03-06T07:20:37.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e86ffd640 1 -- 172.21.15.94:0/3722315000 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78031870 con 0x7f6e880b0070 2024-03-06T07:20:37.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.857+0000 7f6e86ffd640 1 -- 172.21.15.94:0/3722315000 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78031c40 con 0x7f6e880b0070 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.858+0000 7f6e8e97a640 1 -- 172.21.15.94:0/3722315000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 msgr2=0x7f6e88155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.858+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/3722315000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e88155c30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009870 tx=0x7f6e7802b910 comp rx=0 tx=0).stop 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.858+0000 7f6e8e97a640 1 -- 172.21.15.94:0/3722315000 shutdown_connections 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.858+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/3722315000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e88155c30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.858+0000 7f6e8e97a640 1 -- 172.21.15.94:0/3722315000 >> 172.21.15.94:0/3722315000 conn(0x7f6e880ac590 msgr2=0x7f6e880ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.858+0000 7f6e8e97a640 1 -- 172.21.15.94:0/3722315000 shutdown_connections 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.858+0000 7f6e8e97a640 1 -- 172.21.15.94:0/3722315000 wait complete. 2024-03-06T07:20:37.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.859+0000 7f6e8e97a640 1 Processor -- start 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.859+0000 7f6e8e97a640 1 -- start start 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8806ca70 0x7f6e881546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e881e2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e880c5380 0x7f6e881584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8806c550 con 0x7f6e880b0070 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8806c270 con 0x7f6e880c5380 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8806da80 con 0x7f6e8806ca70 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e881e2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e881e2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53422/0 (socket says 172.21.15.94:53422) 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 -- 172.21.15.94:0/1807241784 learned_addr learned my addr 172.21.15.94:0/1807241784 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 -- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8806ca70 msgr2=0x7f6e881546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8806ca70 0x7f6e881546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8cf17640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e880c5380 0x7f6e881584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 -- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e880c5380 msgr2=0x7f6e881584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e880c5380 0x7f6e881584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 -- 172.21.15.94:0/1807241784 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88155580 con 0x7f6e880b0070 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e877fe640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e881e2030 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00c430 tx=0x7f6e7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e84ff9640 1 -- 172.21.15.94:0/1807241784 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0074e0 con 0x7f6e880b0070 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e84ff9640 1 -- 172.21.15.94:0/1807241784 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7c0047e0 con 0x7f6e880b0070 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8814ad20 con 0x7f6e880b0070 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e84ff9640 1 -- 172.21.15.94:0/1807241784 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c004bd0 con 0x7f6e880b0070 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.860+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e88157cf0 con 0x7f6e880b0070 2024-03-06T07:20:37.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.861+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8804f330 con 0x7f6e880b0070 2024-03-06T07:20:37.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.862+0000 7f6e84ff9640 1 -- 172.21.15.94:0/1807241784 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e7c011070 con 0x7f6e880b0070 2024-03-06T07:20:37.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.862+0000 7f6e84ff9640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e7406b4d0 0x7f6e7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:37.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.862+0000 7f6e84ff9640 1 -- 172.21.15.94:0/1807241784 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f6e7c089df0 con 0x7f6e880b0070 2024-03-06T07:20:37.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.862+0000 7f6e87fff640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e7406b4d0 0x7f6e7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:37.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.863+0000 7f6e87fff640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e7406b4d0 0x7f6e7406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6e78036000 tx=0x7f6e78030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:37.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:37.866+0000 7f6e84ff9640 1 -- 172.21.15.94:0/1807241784 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e880b0450 con 0x7f6e880b0070 2024-03-06T07:20:38.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.037+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7f6e88154c30 con 0x7f6e880b0070 2024-03-06T07:20:38.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.040+0000 7f6e84ff9640 1 -- 172.21.15.94:0/1807241784 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v22) v1 ==== 96+0+0 (secure 0 0 0) 0x7f6e7c00e280 con 0x7f6e880b0070 2024-03-06T07:20:38.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e7406b4d0 msgr2=0x7f6e7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e7406b4d0 0x7f6e7406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6e78036000 tx=0x7f6e78030000 comp rx=0 tx=0).stop 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 msgr2=0x7f6e881e2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e881e2030 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00c430 tx=0x7f6e7c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 shutdown_connections 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e7406b4d0 0x7f6e7406d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e880c5380 0x7f6e881584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e880b0070 0x7f6e881e2030 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 --2- 172.21.15.94:0/1807241784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8806ca70 0x7f6e881546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 >> 172.21.15.94:0/1807241784 conn(0x7f6e880ac590 msgr2=0x7f6e880ada40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 shutdown_connections 2024-03-06T07:20:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.042+0000 7f6e8e97a640 1 -- 172.21.15.94:0/1807241784 wait complete. 2024-03-06T07:20:38.056 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-03-06T07:20:38.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.153+0000 7fb559039640 1 Processor -- start 2024-03-06T07:20:38.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.153+0000 7fb559039640 1 -- start start 2024-03-06T07:20:38.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb559039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb554072e70 0x7fb5540bd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb559039640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb55406e490 0x7fb55415fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb559039640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55406c8e0 con 0x7fb554073250 2024-03-06T07:20:38.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb559039640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55406ce80 con 0x7fb55406e490 2024-03-06T07:20:38.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb559039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540c1b00 con 0x7fb554072e70 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb54a59c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fb554073250 0x7fb5540c3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:42894/0 (socket says 172.21.15.94:42894) 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb54a59c640 1 -- 172.21.15.94:0/3362982262 learned_addr learned my addr 172.21.15.94:0/3362982262 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb552d9d640 1 --2- 172.21.15.94:0/3362982262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb554072e70 0x7fb5540bd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb55359e640 1 --2- 172.21.15.94:0/3362982262 >> v2:172.21.15.145:3300/0 conn(0x7fb55406e490 0x7fb55415fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb552d9d640 1 -- 172.21.15.94:0/3362982262 >> v2:172.21.15.145:3300/0 conn(0x7fb55406e490 msgr2=0x7fb55415fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb552d9d640 1 --2- 172.21.15.94:0/3362982262 >> v2:172.21.15.145:3300/0 conn(0x7fb55406e490 0x7fb55415fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb552d9d640 1 -- 172.21.15.94:0/3362982262 >> v1:172.21.15.145:6789/0 conn(0x7fb554073250 legacy=0x7fb5540c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.154+0000 7fb552d9d640 1 -- 172.21.15.94:0/3362982262 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55406cba0 con 0x7fb554072e70 2024-03-06T07:20:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.155+0000 7fb552d9d640 1 --2- 172.21.15.94:0/3362982262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb554072e70 0x7fb5540bd510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5440097a0 tx=0x7fb54402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=113904b727e55537 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.155+0000 7fb55259c640 1 -- 172.21.15.94:0/3362982262 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440315a0 con 0x7fb554072e70 2024-03-06T07:20:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.155+0000 7fb55259c640 1 -- 172.21.15.94:0/3362982262 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb544037070 con 0x7fb554072e70 2024-03-06T07:20:38.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.155+0000 7fb55259c640 1 -- 172.21.15.94:0/3362982262 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544031960 con 0x7fb554072e70 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 -- 172.21.15.94:0/3362982262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb554072e70 msgr2=0x7fb5540bd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 --2- 172.21.15.94:0/3362982262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb554072e70 0x7fb5540bd510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5440097a0 tx=0x7fb54402b840 comp rx=0 tx=0).stop 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 -- 172.21.15.94:0/3362982262 shutdown_connections 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 --2- 172.21.15.94:0/3362982262 >> v2:172.21.15.145:3300/0 conn(0x7fb55406e490 0x7fb55415fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 --2- 172.21.15.94:0/3362982262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb554072e70 0x7fb5540bd510 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 -- 172.21.15.94:0/3362982262 >> 172.21.15.94:0/3362982262 conn(0x7fb55406b1a0 msgr2=0x7fb55406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 -- 172.21.15.94:0/3362982262 shutdown_connections 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.156+0000 7fb559039640 1 -- 172.21.15.94:0/3362982262 wait complete. 2024-03-06T07:20:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.157+0000 7fb559039640 1 Processor -- start 2024-03-06T07:20:38.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.157+0000 7fb559039640 1 -- start start 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb559039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb55406e490 0x7fb554159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb559039640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb554072e70 0x7fb5541ebe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb559039640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb554073250 0x7fb5541eac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb559039640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540c6760 con 0x7fb554072e70 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb559039640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540c4a00 con 0x7fb554073250 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb559039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540bdeb0 con 0x7fb55406e490 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb54a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb554072e70 0x7fb5541ebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb54a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb554072e70 0x7fb5541ebe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53434/0 (socket says 172.21.15.94:53434) 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb54a59c640 1 -- 172.21.15.94:0/993205206 learned_addr learned my addr 172.21.15.94:0/993205206 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb55359e640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb554073250 0x7fb5541eac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.158+0000 7fb552d9d640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb55406e490 0x7fb554159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54a59c640 1 -- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb55406e490 msgr2=0x7fb554159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54a59c640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb55406e490 0x7fb554159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54a59c640 1 -- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb554073250 msgr2=0x7fb5541eac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54a59c640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb554073250 0x7fb5541eac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54a59c640 1 -- 172.21.15.94:0/993205206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540be190 con 0x7fb554072e70 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb552d9d640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb55406e490 0x7fb554159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb55359e640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb554073250 0x7fb5541eac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54a59c640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb554072e70 0x7fb5541ebe20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c3c0 tx=0x7fb53c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54bfff640 1 -- 172.21.15.94:0/993205206 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c017070 con 0x7fb554072e70 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54bfff640 1 -- 172.21.15.94:0/993205206 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c005ce0 con 0x7fb554072e70 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb54bfff640 1 -- 172.21.15.94:0/993205206 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c0072f0 con 0x7fb554072e70 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5540be470 con 0x7fb554072e70 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.159+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55406d470 con 0x7fb554072e70 2024-03-06T07:20:38.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.160+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5541586f0 con 0x7fb554072e70 2024-03-06T07:20:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.160+0000 7fb54bfff640 1 -- 172.21.15.94:0/993205206 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb53c01a070 con 0x7fb554072e70 2024-03-06T07:20:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.161+0000 7fb54bfff640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb53406b4d0 0x7fb53406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.161+0000 7fb54bfff640 1 -- 172.21.15.94:0/993205206 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fb53c0895a0 con 0x7fb554072e70 2024-03-06T07:20:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.161+0000 7fb552d9d640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb53406b4d0 0x7fb53406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.162+0000 7fb552d9d640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb53406b4d0 0x7fb53406d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb544036000 tx=0x7fb54403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.165+0000 7fb54bfff640 1 -- 172.21.15.94:0/993205206 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb53c058450 con 0x7fb554072e70 2024-03-06T07:20:38.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.400+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7fb55404f330 con 0x7fb554072e70 2024-03-06T07:20:38.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.828+0000 7fb54bfff640 1 -- 172.21.15.94:0/993205206 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v152) v1 ==== 150+0+0 (secure 0 0 0) 0x7fb53c057d50 con 0x7fb554072e70 2024-03-06T07:20:38.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb53406b4d0 msgr2=0x7fb53406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb53406b4d0 0x7fb53406d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb544036000 tx=0x7fb54403d040 comp rx=0 tx=0).stop 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb554072e70 msgr2=0x7fb5541ebe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb554072e70 0x7fb5541ebe20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c3c0 tx=0x7fb53c00c880 comp rx=0 tx=0).stop 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 shutdown_connections 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb53406b4d0 0x7fb53406d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb554073250 0x7fb5541eac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb554072e70 0x7fb5541ebe20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 --2- 172.21.15.94:0/993205206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb55406e490 0x7fb554159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.830+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 >> 172.21.15.94:0/993205206 conn(0x7fb55406b1a0 msgr2=0x7fb5540c2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.831+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 shutdown_connections 2024-03-06T07:20:38.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.831+0000 7fb559039640 1 -- 172.21.15.94:0/993205206 wait complete. 2024-03-06T07:20:38.844 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is still used if necessary. ... ok 2024-03-06T07:20:38.844 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. 2024-03-06T07:20:38.844 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:20:38.845 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:38.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 Processor -- start 2024-03-06T07:20:38.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 -- start start 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5194150580 0x7f519406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 0x7f519406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51940c6000 0x7f51941566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5194153a60 con 0x7f51940c6000 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5194153d40 con 0x7f5194150580 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.943+0000 7f519a9e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519406c2c0 con 0x7f51940c1320 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51940c6000 0x7f51941566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51940c6000 0x7f51941566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48582/0 (socket says 172.21.15.94:48582) 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 -- 172.21.15.94:0/364471122 learned_addr learned my addr 172.21.15.94:0/364471122 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:38.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5193fff640 1 --2- 172.21.15.94:0/364471122 >> v2:172.21.15.145:3300/0 conn(0x7f5194150580 0x7f519406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 -- 172.21.15.94:0/364471122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 msgr2=0x7f519406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:38.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 --2- 172.21.15.94:0/364471122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 0x7f519406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 -- 172.21.15.94:0/364471122 >> v2:172.21.15.145:3300/0 conn(0x7f5194150580 msgr2=0x7f519406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:38.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 --2- 172.21.15.94:0/364471122 >> v2:172.21.15.145:3300/0 conn(0x7f5194150580 0x7f519406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.944+0000 7f5198f7f640 1 -- 172.21.15.94:0/364471122 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519406c5a0 con 0x7f51940c6000 2024-03-06T07:20:38.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f5198f7f640 1 --2- 172.21.15.94:0/364471122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51941566b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f51940a9550 tx=0x7f518c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a74229cf3764a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f5192ffd640 1 -- 172.21.15.94:0/364471122 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c02c9e0 con 0x7f51940c6000 2024-03-06T07:20:38.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f5192ffd640 1 -- 172.21.15.94:0/364471122 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c02cb80 con 0x7f51940c6000 2024-03-06T07:20:38.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f5192ffd640 1 -- 172.21.15.94:0/364471122 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c0343f0 con 0x7f51940c6000 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 -- 172.21.15.94:0/364471122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 msgr2=0x7f51941566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 --2- 172.21.15.94:0/364471122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51941566b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f51940a9550 tx=0x7f518c02be40 comp rx=0 tx=0).stop 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 -- 172.21.15.94:0/364471122 shutdown_connections 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 --2- 172.21.15.94:0/364471122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51941566b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 --2- 172.21.15.94:0/364471122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 0x7f519406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 --2- 172.21.15.94:0/364471122 >> v2:172.21.15.145:3300/0 conn(0x7f5194150580 0x7f519406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 -- 172.21.15.94:0/364471122 >> 172.21.15.94:0/364471122 conn(0x7f51940c6710 msgr2=0x7f51940c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 -- 172.21.15.94:0/364471122 shutdown_connections 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.945+0000 7f519a9e2640 1 -- 172.21.15.94:0/364471122 wait complete. 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.946+0000 7f519a9e2640 1 Processor -- start 2024-03-06T07:20:38.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f519a9e2640 1 -- start start 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f519a9e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 0x7f51941e9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f519a9e2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51940a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f519a9e2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5194150580 0x7f5194140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f519a9e2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940c0dc0 con 0x7f51940c6000 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f519a9e2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940c0c40 con 0x7f5194150580 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f519a9e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5194156bf0 con 0x7f51940c1320 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f51937fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51940a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f51937fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51940a75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48584/0 (socket says 172.21.15.94:48584) 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f51937fe640 1 -- 172.21.15.94:0/3847364398 learned_addr learned my addr 172.21.15.94:0/3847364398 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f5193fff640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 0x7f51941e9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.947+0000 7f5198f7f640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5194150580 0x7f5194140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f51937fe640 1 -- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 msgr2=0x7f51941e9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f51937fe640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 0x7f51941e9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f51937fe640 1 -- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5194150580 msgr2=0x7f5194140430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f51937fe640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5194150580 0x7f5194140430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f51937fe640 1 -- 172.21.15.94:0/3847364398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194155b90 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f51937fe640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51940a75f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f518800b4d0 tx=0x7f518800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f5190ff9640 1 -- 172.21.15.94:0/3847364398 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188014070 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f5190ff9640 1 -- 172.21.15.94:0/3847364398 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5188010870 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51940c0a00 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f5190ff9640 1 -- 172.21.15.94:0/3847364398 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188010c60 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.948+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51941558b0 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.949+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5194067ad0 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.949+0000 7f5190ff9640 1 -- 172.21.15.94:0/3847364398 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5188017070 con 0x7f51940c6000 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.950+0000 7f5190ff9640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f518006b4d0 0x7f518006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:38.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.950+0000 7f5190ff9640 1 -- 172.21.15.94:0/3847364398 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f5188089e00 con 0x7f51940c6000 2024-03-06T07:20:38.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.950+0000 7f5193fff640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f518006b4d0 0x7f518006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:38.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.950+0000 7f5193fff640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f518006b4d0 0x7f518006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f518400aa20 tx=0x7f5184008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:38.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:38.954+0000 7f5190ff9640 1 -- 172.21.15.94:0/3847364398 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5194067ad0 con 0x7f51940c6000 2024-03-06T07:20:39.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.127+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f519404f330 con 0x7f51940c6000 2024-03-06T07:20:39.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.128+0000 7f5190ff9640 1 -- 172.21.15.94:0/3847364398 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+13816 (secure 0 0 0) 0x7f51880586e0 con 0x7f51940c6000 2024-03-06T07:20:39.130 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:39.130 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":65,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:36.823553+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6834/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/279249774":"2024-03-07T07:20:27.485101+0000","172.21.15.94:6835/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.94:6833/480754107":"2024-03-07T07:20:19.628911+0000","172.21.15.145:6836/3216323598":"2024-03-07T07:20:33.816463+0000","172.21.15.145:6834/3216323598":"2024-03-07T07:20:33.816463+0000","172.21.15.145:6837/3125091562":"2024-03-07T07:20:16.601064+0000","172.21.15.94:6832/1798838613":"2024-03-07T07:20:28.489384+0000","172.21.15.145:6835/3125091562":"2024-03-07T07:20:16.601064+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:39.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f518006b4d0 msgr2=0x7f518006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f518006b4d0 0x7f518006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f518400aa20 tx=0x7f5184008040 comp rx=0 tx=0).stop 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 msgr2=0x7f51940a75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51940a75f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f518800b4d0 tx=0x7f518800b990 comp rx=0 tx=0).stop 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 shutdown_connections 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f518006b4d0 0x7f518006d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5194150580 0x7f5194140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51940c6000 0x7f51940a75f0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 --2- 172.21.15.94:0/3847364398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51940c1320 0x7f51941e9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.130+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 >> 172.21.15.94:0/3847364398 conn(0x7f51940c6710 msgr2=0x7f51940c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.131+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 shutdown_connections 2024-03-06T07:20:39.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.131+0000 7f519a9e2640 1 -- 172.21.15.94:0/3847364398 wait complete. 2024-03-06T07:20:39.143 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:20:39.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:39.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.245+0000 7fa0bda29640 1 Processor -- start 2024-03-06T07:20:39.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.245+0000 7fa0bda29640 1 -- start start 2024-03-06T07:20:39.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0bda29640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa0b8072cd0 0x7fa0b8155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0bda29640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b8152db0 con 0x7fa0b8073290 2024-03-06T07:20:39.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0bda29640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b81590f0 con 0x7fa0b81529d0 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0bda29640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b80c6dd0 con 0x7fa0b8072cd0 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0b77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa0b8073290 0x7fa0b8155660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48568/0 (socket says 172.21.15.94:48568) 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0b77fe640 1 -- 172.21.15.94:0/2168347545 learned_addr learned my addr 172.21.15.94:0/2168347545 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145314415 0 0) 0x7fa0b0001110 con 0x7fa0b8073290 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0b67fc640 1 --2- 172.21.15.94:0/2168347545 >> v2:172.21.15.145:3300/0 conn(0x7fa0b8072cd0 0x7fa0b8155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.246+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a0003520 con 0x7fa0b8073290 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525596246 0 0) 0x7fa0a0003520 con 0x7fa0b8073290 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 >> v2:172.21.15.145:3300/0 conn(0x7fa0b8072cd0 msgr2=0x7fa0b8155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 --2- 172.21.15.94:0/2168347545 >> v2:172.21.15.145:3300/0 conn(0x7fa0b8072cd0 0x7fa0b8155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 >> v1:172.21.15.145:6789/0 conn(0x7fa0b81529d0 legacy=0x7fa0b80c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8159650 con 0x7fa0b8073290 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fa0b0001840 con 0x7fa0b8073290 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 >> v1:172.21.15.94:6789/0 conn(0x7fa0b8073290 legacy=0x7fa0b8155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0a00039b0 0x7fa0a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a0004800 0x7fa0a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0a00052e0 0x7fa0a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80c6dd0 con 0x7fa0a00052e0 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a000c1b0 con 0x7fa0a00039b0 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a000c3b0 con 0x7fa0a0004800 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b77fe640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0a00052e0 0x7fa0a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b6ffd640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0a00039b0 0x7fa0a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b67fc640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a0004800 0x7fa0a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b77fe640 1 -- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a0004800 msgr2=0x7fa0a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b77fe640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a0004800 0x7fa0a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b77fe640 1 -- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0a00039b0 msgr2=0x7fa0a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b77fe640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0a00039b0 0x7fa0a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b77fe640 1 -- 172.21.15.94:0/2168347545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a0003520 con 0x7fa0a00052e0 2024-03-06T07:20:39.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.247+0000 7fa0b67fc640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a0004800 0x7fa0a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0b77fe640 1 -- 172.21.15.94:0/2168347545 reap_dead start 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0b77fe640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0a00052e0 0x7fa0a000b9c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000c490 tx=0x7fa0a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e223dfc139e22ae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b000c0e0 con 0x7fa0a00052e0 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b0030070 con 0x7fa0a00052e0 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0b5ffb640 1 -- 172.21.15.94:0/2168347545 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b00058d0 con 0x7fa0a00052e0 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0bda29640 1 -- 172.21.15.94:0/2168347545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0a00052e0 msgr2=0x7fa0a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0bda29640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0a00052e0 0x7fa0a000b9c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000c490 tx=0x7fa0a0005830 comp rx=0 tx=0).stop 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0bda29640 1 -- 172.21.15.94:0/2168347545 shutdown_connections 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0bda29640 1 --2- 172.21.15.94:0/2168347545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0a00052e0 0x7fa0a000b9c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.248+0000 7fa0bda29640 1 -- 172.21.15.94:0/2168347545 >> 172.21.15.94:0/2168347545 conn(0x7fa0b80aa630 msgr2=0x7fa0b80aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.249+0000 7fa0bda29640 1 -- 172.21.15.94:0/2168347545 shutdown_connections 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.249+0000 7fa0bda29640 1 -- 172.21.15.94:0/2168347545 wait complete. 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 Processor -- start 2024-03-06T07:20:39.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 -- start start 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a00052e0 0x7fa0b8150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0b80ac320 0x7fa0b80ab930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0b80abe70 0x7fa0b814bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8159370 con 0x7fa0b80ac320 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80c6940 con 0x7fa0b80abe70 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.250+0000 7fa0bda29640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80723b0 con 0x7fa0a00052e0 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0b80ac320 0x7fa0b80ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0b80ac320 0x7fa0b80ab930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48602/0 (socket says 172.21.15.94:48602) 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 -- 172.21.15.94:0/3896568475 learned_addr learned my addr 172.21.15.94:0/3896568475 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b77fe640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0b80abe70 0x7fa0b814bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b6ffd640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a00052e0 0x7fa0b8150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 -- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a00052e0 msgr2=0x7fa0b8150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a00052e0 0x7fa0b8150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 -- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0b80abe70 msgr2=0x7fa0b814bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0b80abe70 0x7fa0b814bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 -- 172.21.15.94:0/3896568475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80ae550 con 0x7fa0b80ac320 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0b67fc640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0b80ac320 0x7fa0b80ab930 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac011570 tx=0x7fa0ac011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa09ffff640 1 -- 172.21.15.94:0/3896568475 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac015870 con 0x7fa0b80ac320 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa09ffff640 1 -- 172.21.15.94:0/3896568475 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac015a10 con 0x7fa0b80ac320 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b814f6f0 con 0x7fa0b80ac320 2024-03-06T07:20:39.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa09ffff640 1 -- 172.21.15.94:0/3896568475 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac015dc0 con 0x7fa0b80ac320 2024-03-06T07:20:39.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.251+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b8151360 con 0x7fa0b80ac320 2024-03-06T07:20:39.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.252+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b814f470 con 0x7fa0b80ac320 2024-03-06T07:20:39.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.253+0000 7fa09ffff640 1 -- 172.21.15.94:0/3896568475 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0ac00f460 con 0x7fa0b80ac320 2024-03-06T07:20:39.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.253+0000 7fa09ffff640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a406b500 0x7fa0a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.253+0000 7fa09ffff640 1 -- 172.21.15.94:0/3896568475 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fa0ac08a9b0 con 0x7fa0b80ac320 2024-03-06T07:20:39.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.253+0000 7fa0b6ffd640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a406b500 0x7fa0a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.254+0000 7fa0b6ffd640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a406b500 0x7fa0a406d9c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8000f80 tx=0x7fa0a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.257+0000 7fa09ffff640 1 -- 172.21.15.94:0/3896568475 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0b814f470 con 0x7fa0b80ac320 2024-03-06T07:20:39.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.428+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0b80c2f70 con 0x7fa0b80ac320 2024-03-06T07:20:39.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.429+0000 7fa09ffff640 1 -- 172.21.15.94:0/3896568475 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0b80c2f70 con 0x7fa0b80ac320 2024-03-06T07:20:39.431 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.430+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a406b500 msgr2=0x7fa0a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.430+0000 7fa0bda29640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a406b500 0x7fa0a406d9c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8000f80 tx=0x7fa0a8009040 comp rx=0 tx=0).stop 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0b80ac320 msgr2=0x7fa0b80ab930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0b80ac320 0x7fa0b80ab930 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac011570 tx=0x7fa0ac011a30 comp rx=0 tx=0).stop 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 shutdown_connections 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a406b500 0x7fa0a406d9c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0b80abe70 0x7fa0b814bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0b80ac320 0x7fa0b80ab930 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 --2- 172.21.15.94:0/3896568475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0a00052e0 0x7fa0b8150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 >> 172.21.15.94:0/3896568475 conn(0x7fa0b80aa630 msgr2=0x7fa0b80c22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 shutdown_connections 2024-03-06T07:20:39.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.431+0000 7fa0bda29640 1 -- 172.21.15.94:0/3896568475 wait complete. 2024-03-06T07:20:39.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-03-06T07:20:39.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.542+0000 7fe6af0c3640 1 Processor -- start 2024-03-06T07:20:39.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6af0c3640 1 -- start start 2024-03-06T07:20:39.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6af0c3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe6a814f790 0x7fe6a81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6af0c3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6a80c1aa0 0x7fe6a80c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6af0c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a80c28d0 con 0x7fe6a80c5c60 2024-03-06T07:20:39.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6af0c3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a80c2680 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6af0c3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a80c2bb0 con 0x7fe6a814f790 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6a7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6a80c1aa0 0x7fe6a80c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6a7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6a80c1aa0 0x7fe6a80c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48604/0 (socket says 172.21.15.94:48604) 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.543+0000 7fe6a7fff640 1 -- 172.21.15.94:0/1070603261 learned_addr learned my addr 172.21.15.94:0/1070603261 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.544+0000 7fe6a7fff640 1 -- 172.21.15.94:0/1070603261 >> v2:172.21.15.145:3300/0 conn(0x7fe6a814f790 msgr2=0x7fe6a81529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.544+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/1070603261 >> v2:172.21.15.145:3300/0 conn(0x7fe6a814f790 0x7fe6a81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.544+0000 7fe6a7fff640 1 -- 172.21.15.94:0/1070603261 >> v1:172.21.15.145:6789/0 conn(0x7fe6a80c5c60 legacy=0x7fe6a8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.544+0000 7fe6a7fff640 1 -- 172.21.15.94:0/1070603261 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a80c2e90 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.544+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/1070603261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a80c5720 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe6a80c3170 tx=0x7fe69402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2cc823bb5d0561 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.544+0000 7fe6a77fe640 1 -- 172.21.15.94:0/1070603261 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694035020 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6a77fe640 1 -- 172.21.15.94:0/1070603261 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe694031870 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6a77fe640 1 -- 172.21.15.94:0/1070603261 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe694031c40 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 -- 172.21.15.94:0/1070603261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 msgr2=0x7fe6a80c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/1070603261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a80c5720 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe6a80c3170 tx=0x7fe69402b910 comp rx=0 tx=0).stop 2024-03-06T07:20:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 -- 172.21.15.94:0/1070603261 shutdown_connections 2024-03-06T07:20:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/1070603261 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a80c5720 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/1070603261 >> v2:172.21.15.145:3300/0 conn(0x7fe6a814f790 0x7fe6a81529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 -- 172.21.15.94:0/1070603261 >> 172.21.15.94:0/1070603261 conn(0x7fe6a80c6710 msgr2=0x7fe6a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 -- 172.21.15.94:0/1070603261 shutdown_connections 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.545+0000 7fe6af0c3640 1 -- 172.21.15.94:0/1070603261 wait complete. 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.546+0000 7fe6af0c3640 1 Processor -- start 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.546+0000 7fe6af0c3640 1 -- start start 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6af0c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a815f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6af0c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a80c5c60 0x7fe6a81549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6af0c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a814f790 0x7fe6a815dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6af0c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a806cc00 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6af0c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a80c0d60 con 0x7fe6a80c5c60 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6af0c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a806cd80 con 0x7fe6a814f790 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a815f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a815f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48612/0 (socket says 172.21.15.94:48612) 2024-03-06T07:20:39.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 -- 172.21.15.94:0/2465502497 learned_addr learned my addr 172.21.15.94:0/2465502497 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a80c5c60 0x7fe6a81549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ad660640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a814f790 0x7fe6a815dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 -- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a814f790 msgr2=0x7fe6a815dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a814f790 0x7fe6a815dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 -- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a80c5c60 msgr2=0x7fe6a81549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a80c5c60 0x7fe6a81549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.547+0000 7fe6ace5f640 1 -- 172.21.15.94:0/2465502497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a81502e0 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.548+0000 7fe6ace5f640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a815f540 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00b440 tx=0x7fe69c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.548+0000 7fe6a57fa640 1 -- 172.21.15.94:0/2465502497 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c015020 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.548+0000 7fe6a57fa640 1 -- 172.21.15.94:0/2465502497 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c011890 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.548+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a81505c0 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.548+0000 7fe6a57fa640 1 -- 172.21.15.94:0/2465502497 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c011c80 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.548+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a8150b80 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.549+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a80c5830 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.549+0000 7fe6a57fa640 1 -- 172.21.15.94:0/2465502497 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe69c018020 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.549+0000 7fe6a57fa640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe68c06b4d0 0x7fe68c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.550+0000 7fe6a57fa640 1 -- 172.21.15.94:0/2465502497 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe69c08a240 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.550+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe68c06b4d0 0x7fe68c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.550+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe68c06b4d0 0x7fe68c06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe694036000 tx=0x7fe69403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.554+0000 7fe6a57fa640 1 -- 172.21.15.94:0/2465502497 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6a80c5830 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.792+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7fe6a8156200 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.830+0000 7fe6a57fa640 1 -- 172.21.15.94:0/2465502497 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v153) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe6a8156200 con 0x7fe6a80c1aa0 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe68c06b4d0 msgr2=0x7fe68c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe68c06b4d0 0x7fe68c06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe694036000 tx=0x7fe69403c040 comp rx=0 tx=0).stop 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 msgr2=0x7fe6a815f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a815f540 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00b440 tx=0x7fe69c00b900 comp rx=0 tx=0).stop 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 shutdown_connections 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe68c06b4d0 0x7fe68c06d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a814f790 0x7fe6a815dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a80c5c60 0x7fe6a81549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 --2- 172.21.15.94:0/2465502497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80c1aa0 0x7fe6a815f540 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.832+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 >> 172.21.15.94:0/2465502497 conn(0x7fe6a80c6710 msgr2=0x7fe6a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.833+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 shutdown_connections 2024-03-06T07:20:39.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.833+0000 7fe6af0c3640 1 -- 172.21.15.94:0/2465502497 wait complete. 2024-03-06T07:20:39.845 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:20:39.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.946+0000 7f998dd5c640 1 Processor -- start 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.946+0000 7f998dd5c640 1 -- start start 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f998dd5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 0x7f99880ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f998dd5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f99881538c0 0x7f99881494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f998dd5c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9988154330 con 0x7f99880a97c0 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f998dd5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99880c46a0 con 0x7f99881538c0 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f998dd5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9988153dc0 con 0x7f99880ab310 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f9987fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f99880a97c0 0x7f99880c4b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48572/0 (socket says 172.21.15.94:48572) 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f9987fff640 1 -- 172.21.15.94:0/488251301 learned_addr learned my addr 172.21.15.94:0/488251301 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:39.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f9986ffd640 1 --2- 172.21.15.94:0/488251301 >> v2:172.21.15.145:3300/0 conn(0x7f99881538c0 0x7f99881494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99877fe640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 0x7f99880ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546402324 0 0) 0x7f9988154330 con 0x7f99880a97c0 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9970003520 con 0x7f99880a97c0 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2262038735 0 0) 0x7f9970003520 con 0x7f99880a97c0 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 msgr2=0x7f99880ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 0x7f99880ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 >> v2:172.21.15.145:3300/0 conn(0x7f99881538c0 msgr2=0x7f99881494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 --2- 172.21.15.94:0/488251301 >> v2:172.21.15.145:3300/0 conn(0x7f99881538c0 0x7f99881494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99880c5d30 con 0x7f99880a97c0 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f99800019f0 con 0x7f99880a97c0 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 >> v1:172.21.15.94:6789/0 conn(0x7f99880a97c0 legacy=0x7f99880c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99700039b0 0x7f9970003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9970004800 0x7f9970004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.947+0000 7f99867fc640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 0x7f99700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99880c46a0 con 0x7f9970004800 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9988153dc0 con 0x7f99700039b0 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997000fe00 con 0x7f99880ab310 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f9986ffd640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9970004800 0x7f9970004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f99877fe640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99700039b0 0x7f9970003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f9987fff640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 0x7f99700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f9986ffd640 1 -- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 msgr2=0x7f99700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f9986ffd640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99880ab310 0x7f99700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f9986ffd640 1 -- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99700039b0 msgr2=0x7f9970003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f9986ffd640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99700039b0 0x7f9970003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f9986ffd640 1 -- 172.21.15.94:0/488251301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9970003520 con 0x7f9970004800 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.948+0000 7f99877fe640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99700039b0 0x7f9970003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f9986ffd640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9970004800 0x7f9970004c80 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f997c009ac0 tx=0x7f9970005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29a236dc0b2f050 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c039070 con 0x7f9970004800 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c02c710 con 0x7f9970004800 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f99867fc640 1 -- 172.21.15.94:0/488251301 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c009690 con 0x7f9970004800 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f998dd5c640 1 -- 172.21.15.94:0/488251301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9970004800 msgr2=0x7f9970004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f998dd5c640 1 --2- 172.21.15.94:0/488251301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9970004800 0x7f9970004c80 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f997c009ac0 tx=0x7f9970005830 comp rx=0 tx=0).stop 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f9987fff640 1 -- 172.21.15.94:0/488251301 reap_dead start 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f998dd5c640 1 -- 172.21.15.94:0/488251301 shutdown_connections 2024-03-06T07:20:39.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f998dd5c640 1 -- 172.21.15.94:0/488251301 >> 172.21.15.94:0/488251301 conn(0x7f99880c6710 msgr2=0x7f99880c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:39.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f998dd5c640 1 -- 172.21.15.94:0/488251301 shutdown_connections 2024-03-06T07:20:39.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.949+0000 7f998dd5c640 1 -- 172.21.15.94:0/488251301 wait complete. 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.950+0000 7f998dd5c640 1 Processor -- start 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f998dd5c640 1 -- start start 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f998dd5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9970004800 0x7f99880a7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f998dd5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9988070ab0 0x7f998814f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f998dd5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9988070350 0x7f998814fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f998dd5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99880c5240 con 0x7f9988070350 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f998dd5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99880c62f0 con 0x7f9988070ab0 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f998dd5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99880c53c0 con 0x7f9970004800 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f9987fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9988070350 0x7f998814fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f9987fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9988070350 0x7f998814fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48634/0 (socket says 172.21.15.94:48634) 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.951+0000 7f9987fff640 1 -- 172.21.15.94:0/1726150049 learned_addr learned my addr 172.21.15.94:0/1726150049 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9986ffd640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9988070ab0 0x7f998814f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f99877fe640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9970004800 0x7f99880a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9987fff640 1 -- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9970004800 msgr2=0x7f99880a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9987fff640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9970004800 0x7f99880a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9987fff640 1 -- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9988070ab0 msgr2=0x7f998814f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9987fff640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9988070ab0 0x7f998814f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9987fff640 1 -- 172.21.15.94:0/1726150049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99880c04b0 con 0x7f9988070350 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9986ffd640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9988070ab0 0x7f998814f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f9987fff640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9988070350 0x7f998814fc70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f99880c75b0 tx=0x7f9980007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f996ffff640 1 -- 172.21.15.94:0/1726150049 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978009690 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f996ffff640 1 -- 172.21.15.94:0/1726150049 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980019070 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f996ffff640 1 -- 172.21.15.94:0/1726150049 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99800133f0 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9988156390 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.952+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99881ea5b0 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.953+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998804f330 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.953+0000 7f996ffff640 1 -- 172.21.15.94:0/1726150049 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9980013590 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.954+0000 7f996ffff640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f997406b500 0x7f997406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.954+0000 7f996ffff640 1 -- 172.21.15.94:0/1726150049 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f998008a670 con 0x7f9988070350 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.954+0000 7f99877fe640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f997406b500 0x7f997406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:39.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.954+0000 7f99877fe640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f997406b500 0x7f997406d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9978009b40 tx=0x7f9978008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:39.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:39.958+0000 7f996ffff640 1 -- 172.21.15.94:0/1726150049 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99880c04b0 con 0x7f9988070350 2024-03-06T07:20:40.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.128+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f99880c5a70 con 0x7f9988070350 2024-03-06T07:20:40.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.131+0000 7f996ffff640 1 -- 172.21.15.94:0/1726150049 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v23)=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9980059510 con 0x7f9988070350 2024-03-06T07:20:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f997406b500 msgr2=0x7f997406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f997406b500 0x7f997406d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9978009b40 tx=0x7f9978008040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9988070350 msgr2=0x7f998814fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9988070350 0x7f998814fc70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f99880c75b0 tx=0x7f9980007bd0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 shutdown_connections 2024-03-06T07:20:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f997406b500 0x7f997406d9c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9988070350 0x7f998814fc70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9988070ab0 0x7f998814f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 --2- 172.21.15.94:0/1726150049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9970004800 0x7f99880a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 >> 172.21.15.94:0/1726150049 conn(0x7f99880c6710 msgr2=0x7f9988150a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 shutdown_connections 2024-03-06T07:20:40.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.133+0000 7f998dd5c640 1 -- 172.21.15.94:0/1726150049 wait complete. 2024-03-06T07:20:40.146 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:20:40.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:20:40.191 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:20:40.237 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:20:40.238 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:20:40.238 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:40.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:20:40.253 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:20:40.253 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:20:40.270 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:20:40.270 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:20:40.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:40.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 Processor -- start 2024-03-06T07:20:40.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 -- start start 2024-03-06T07:20:40.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53781550c0 0x7f53781554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 0x7f537814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53780aa180 0x7f537814f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53780c4610 con 0x7f53780aa180 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53780aa660 con 0x7f53781550c0 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f537e0c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780aa940 con 0x7f53781559e0 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.392+0000 7f5377fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53780aa180 0x7f537814f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53780aa180 0x7f537814f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48636/0 (socket says 172.21.15.94:48636) 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 -- 172.21.15.94:0/3011698653 learned_addr learned my addr 172.21.15.94:0/3011698653 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 -- 172.21.15.94:0/3011698653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 msgr2=0x7f537814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 --2- 172.21.15.94:0/3011698653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 0x7f537814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 -- 172.21.15.94:0/3011698653 >> v2:172.21.15.145:3300/0 conn(0x7f53781550c0 msgr2=0x7f53781554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 --2- 172.21.15.94:0/3011698653 >> v2:172.21.15.145:3300/0 conn(0x7f53781550c0 0x7f53781554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 -- 172.21.15.94:0/3011698653 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780c4860 con 0x7f53780aa180 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.393+0000 7f5377fff640 1 --2- 172.21.15.94:0/3011698653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f537814f140 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f53780c3fc0 tx=0x7f537002ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b537c001e977dec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f53767fc640 1 -- 172.21.15.94:0/3011698653 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537002d870 con 0x7f53780aa180 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f53767fc640 1 -- 172.21.15.94:0/3011698653 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537002da10 con 0x7f53780aa180 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f53767fc640 1 -- 172.21.15.94:0/3011698653 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537002dde0 con 0x7f53780aa180 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 -- 172.21.15.94:0/3011698653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 msgr2=0x7f537814f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3011698653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f537814f140 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f53780c3fc0 tx=0x7f537002ced0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 -- 172.21.15.94:0/3011698653 shutdown_connections 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3011698653 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f537814f140 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3011698653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 0x7f537814ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3011698653 >> v2:172.21.15.145:3300/0 conn(0x7f53781550c0 0x7f53781554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 -- 172.21.15.94:0/3011698653 >> 172.21.15.94:0/3011698653 conn(0x7f53780c6710 msgr2=0x7f53780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 -- 172.21.15.94:0/3011698653 shutdown_connections 2024-03-06T07:20:40.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.394+0000 7f537e0c3640 1 -- 172.21.15.94:0/3011698653 wait complete. 2024-03-06T07:20:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.395+0000 7f537e0c3640 1 Processor -- start 2024-03-06T07:20:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.395+0000 7f537e0c3640 1 -- start start 2024-03-06T07:20:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f537e0c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f53780bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f537e0c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53781550c0 0x7f53780a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f537e0c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 0x7f53781ead70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f537e0c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53780c3210 con 0x7f53780aa180 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f537e0c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53780c1520 con 0x7f53781550c0 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f537e0c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780c3390 con 0x7f53781559e0 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f53780bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f53780bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48640/0 (socket says 172.21.15.94:48640) 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 -- 172.21.15.94:0/3790685134 learned_addr learned my addr 172.21.15.94:0/3790685134 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f5377fff640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 0x7f53781ead70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f5376ffd640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53781550c0 0x7f53780a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 -- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 msgr2=0x7f53781ead70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 0x7f53781ead70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 -- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53781550c0 msgr2=0x7f53780a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53781550c0 0x7f53780a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f53777fe640 1 -- 172.21.15.94:0/3790685134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53781ea8b0 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.396+0000 7f5376ffd640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53781550c0 0x7f53780a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.397+0000 7f53777fe640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f53780bfdb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f536c00b4a0 tx=0x7f536c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.397+0000 7f535ffff640 1 -- 172.21.15.94:0/3790685134 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c014070 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.397+0000 7f535ffff640 1 -- 172.21.15.94:0/3790685134 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c010610 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.397+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5378149610 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.397+0000 7f535ffff640 1 -- 172.21.15.94:0/3790685134 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c010a00 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.397+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53781ee5c0 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.398+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537804f330 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.398+0000 7f535ffff640 1 -- 172.21.15.94:0/3790685134 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f536c017070 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.399+0000 7f535ffff640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536406b4d0 0x7f536406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.399+0000 7f535ffff640 1 -- 172.21.15.94:0/3790685134 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f536c089b70 con 0x7f53780aa180 2024-03-06T07:20:40.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.399+0000 7f5376ffd640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536406b4d0 0x7f536406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:40.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.399+0000 7f5376ffd640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536406b4d0 0x7f536406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f537806d900 tx=0x7f5368008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:40.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.403+0000 7f535ffff640 1 -- 172.21.15.94:0/3790685134 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53781ea8b0 con 0x7f53780aa180 2024-03-06T07:20:40.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.613+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5378068450 con 0x7f53780aa180 2024-03-06T07:20:40.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.613+0000 7f535ffff640 1 -- 172.21.15.94:0/3790685134 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3707 (secure 0 0 0) 0x7f536c058aa0 con 0x7f53780aa180 2024-03-06T07:20:40.615 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:40.615 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1623600203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1623600203},{"type":"v1","addr":"172.21.15.94:6835","nonce":1623600203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":6138,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1223829390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1223829390},{"type":"v1","addr":"172.21.15.94:6834","nonce":1223829390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49},{"gid":35378,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2852856592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2852856592},{"type":"v1","addr":"172.21.15.145:6837","nonce":2852856592}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":35435,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1565796998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1565796998},{"type":"v1","addr":"172.21.15.145:6836","nonce":1565796998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-03-06T07:20:40.616 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 49 2024-03-06T07:20:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536406b4d0 msgr2=0x7f536406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536406b4d0 0x7f536406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f537806d900 tx=0x7f5368008040 comp rx=0 tx=0).stop 2024-03-06T07:20:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 msgr2=0x7f53780bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f53780bfdb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f536c00b4a0 tx=0x7f536c00b960 comp rx=0 tx=0).stop 2024-03-06T07:20:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 shutdown_connections 2024-03-06T07:20:40.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536406b4d0 0x7f536406d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53781559e0 0x7f53781ead70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53781550c0 0x7f53780a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 --2- 172.21.15.94:0/3790685134 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53780aa180 0x7f53780bfdb0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:40.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 >> 172.21.15.94:0/3790685134 conn(0x7f53780c6710 msgr2=0x7f53781dcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:40.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 shutdown_connections 2024-03-06T07:20:40.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:40.615+0000 7f537e0c3640 1 -- 172.21.15.94:0/3790685134 wait complete. 2024-03-06T07:20:40.628 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:20:40.628 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:20:40.628 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:20:40.628 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:20:40.628 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:20:40.629 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:20:40.629 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:20:40.629 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:20:40.629 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:20:40.629 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:20:40.629 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:20:40.629 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:20:40.629 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:20:40.630 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:20:40.630 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:20:40.630 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:20:40.637 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:20:40.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:20:40.639 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:20:40.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:20:40.642 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:20:40.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:20:40.680 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:20:40.683 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:40.684 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:20:40.684 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:40.685 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:20:40.686 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:40.755 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:20:40.755 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:20:40.767 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:20:42.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:20:43.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:43.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:46.642 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:20:46.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:20:46.645 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:20:46.646 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:46.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:20:46.729 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:20:46.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.758+0000 7ffb98fa0640 1 Processor -- start 2024-03-06T07:20:46.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.758+0000 7ffb98fa0640 1 -- start start 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb98fa0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb940a8080 0x7ffb941608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb98fa0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940c2b50 con 0x7ffb940c5510 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb98fa0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940c2e30 con 0x7ffb940c4e10 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb98fa0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb940c4610 con 0x7ffb940a8080 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb93fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb940a8080 0x7ffb941608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb93fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb940a8080 0x7ffb941608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48672/0 (socket says 172.21.15.94:48672) 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb93fff640 1 -- 172.21.15.94:0/4214394596 learned_addr learned my addr 172.21.15.94:0/4214394596 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb937fe640 1 --1- 172.21.15.94:0/4214394596 >> v1:172.21.15.94:6789/0 conn(0x7ffb940c5510 0x7ffb941520a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48598/0 (socket says 172.21.15.94:48598) 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb93fff640 1 -- 172.21.15.94:0/4214394596 >> v1:172.21.15.145:6789/0 conn(0x7ffb940c4e10 legacy=0x7ffb94154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb93fff640 1 -- 172.21.15.94:0/4214394596 >> v1:172.21.15.94:6789/0 conn(0x7ffb940c5510 legacy=0x7ffb941520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb93fff640 1 -- 172.21.15.94:0/4214394596 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940c42c0 con 0x7ffb940a8080 2024-03-06T07:20:46.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.759+0000 7ffb927fc640 1 -- 172.21.15.94:0/4214394596 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621660435 0 0) 0x7ffb940c2b50 con 0x7ffb940c5510 2024-03-06T07:20:46.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.760+0000 7ffb93fff640 1 --2- 172.21.15.94:0/4214394596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940a8080 0x7ffb941608b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009e50 tx=0x7ffb8c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9c60b968b4ad43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.760+0000 7ffb927fc640 1 -- 172.21.15.94:0/4214394596 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c02cbb0 con 0x7ffb940a8080 2024-03-06T07:20:46.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.760+0000 7ffb927fc640 1 -- 172.21.15.94:0/4214394596 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8c02cd50 con 0x7ffb940a8080 2024-03-06T07:20:46.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.760+0000 7ffb927fc640 1 -- 172.21.15.94:0/4214394596 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c034610 con 0x7ffb940a8080 2024-03-06T07:20:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.760+0000 7ffb98fa0640 1 -- 172.21.15.94:0/4214394596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940a8080 msgr2=0x7ffb941608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.761+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/4214394596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940a8080 0x7ffb941608b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009e50 tx=0x7ffb8c02c010 comp rx=0 tx=0).stop 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.761+0000 7ffb98fa0640 1 -- 172.21.15.94:0/4214394596 shutdown_connections 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.761+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/4214394596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940a8080 0x7ffb941608b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.761+0000 7ffb98fa0640 1 -- 172.21.15.94:0/4214394596 >> 172.21.15.94:0/4214394596 conn(0x7ffb9406b730 msgr2=0x7ffb940c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.761+0000 7ffb98fa0640 1 -- 172.21.15.94:0/4214394596 shutdown_connections 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.761+0000 7ffb98fa0640 1 -- 172.21.15.94:0/4214394596 wait complete. 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.762+0000 7ffb98fa0640 1 Processor -- start 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.762+0000 7ffb98fa0640 1 -- start start 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb940a8080 0x7ffb9414b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940c4e10 0x7ffb940abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c5510 0x7ffb941ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940c28f0 con 0x7ffb940c4e10 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940c2770 con 0x7ffb940c5510 2024-03-06T07:20:46.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94155630 con 0x7ffb940a8080 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940c4e10 0x7ffb940abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940c4e10 0x7ffb940abfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48682/0 (socket says 172.21.15.94:48682) 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 -- 172.21.15.94:0/2176597275 learned_addr learned my addr 172.21.15.94:0/2176597275 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb93fff640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c5510 0x7ffb941ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 -- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb940a8080 msgr2=0x7ffb9414b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb937fe640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb940a8080 0x7ffb9414b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb940a8080 0x7ffb9414b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 -- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c5510 msgr2=0x7ffb941ee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c5510 0x7ffb941ee7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:46.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 -- 172.21.15.94:0/2176597275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9406dcf0 con 0x7ffb940c4e10 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb93fff640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c5510 0x7ffb941ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb937fe640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb940a8080 0x7ffb9414b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb92ffd640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940c4e10 0x7ffb940abfd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c400 tx=0x7ffb8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb7bfff640 1 -- 172.21.15.94:0/2176597275 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88017070 con 0x7ffb940c4e10 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb7bfff640 1 -- 172.21.15.94:0/2176597275 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb880052d0 con 0x7ffb940c4e10 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb7bfff640 1 -- 172.21.15.94:0/2176597275 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88003830 con 0x7ffb940c4e10 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9406dfd0 con 0x7ffb940c4e10 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.763+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb941547a0 con 0x7ffb940c4e10 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.764+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb940693c0 con 0x7ffb940c4e10 2024-03-06T07:20:46.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.765+0000 7ffb7bfff640 1 -- 172.21.15.94:0/2176597275 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb8801a070 con 0x7ffb940c4e10 2024-03-06T07:20:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.765+0000 7ffb7bfff640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb8006b4d0 0x7ffb8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.765+0000 7ffb7bfff640 1 -- 172.21.15.94:0/2176597275 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7ffb88089670 con 0x7ffb940c4e10 2024-03-06T07:20:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.766+0000 7ffb937fe640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb8006b4d0 0x7ffb8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:46.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.766+0000 7ffb937fe640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb8006b4d0 0x7ffb8006d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffb84000f60 tx=0x7ffb84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:46.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.770+0000 7ffb7bfff640 1 -- 172.21.15.94:0/2176597275 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb88058480 con 0x7ffb940c4e10 2024-03-06T07:20:46.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:46.952+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffb9414f910 con 0x7ffb940c4e10 2024-03-06T07:20:47.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.847+0000 7ffb7bfff640 1 -- 172.21.15.94:0/2176597275 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v70) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffb88057d80 con 0x7ffb940c4e10 2024-03-06T07:20:47.850 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:20:47.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb8006b4d0 msgr2=0x7ffb8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb8006b4d0 0x7ffb8006d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffb84000f60 tx=0x7ffb84009040 comp rx=0 tx=0).stop 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940c4e10 msgr2=0x7ffb940abfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940c4e10 0x7ffb940abfd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c400 tx=0x7ffb8800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 shutdown_connections 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb8006b4d0 0x7ffb8006d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c5510 0x7ffb941ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb940c4e10 0x7ffb940abfd0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 --2- 172.21.15.94:0/2176597275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb940a8080 0x7ffb9414b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 >> 172.21.15.94:0/2176597275 conn(0x7ffb9406b730 msgr2=0x7ffb94151e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 shutdown_connections 2024-03-06T07:20:47.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.850+0000 7ffb98fa0640 1 -- 172.21.15.94:0/2176597275 wait complete. 2024-03-06T07:20:47.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:20:47.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.969+0000 7f933b13e640 1 Processor -- start 2024-03-06T07:20:47.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.969+0000 7f933b13e640 1 -- start start 2024-03-06T07:20:47.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f933b13e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f93340c1aa0 0x7f93341529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f933b13e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340c28d0 con 0x7f933414f790 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f933b13e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340c2bb0 con 0x7f9334155a40 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f933b13e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93340c2680 con 0x7f93340c1aa0 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f9338eda640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f933414f790 0x7f93340c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48610/0 (socket says 172.21.15.94:48610) 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f9338eda640 1 -- 172.21.15.94:0/422007920 learned_addr learned my addr 172.21.15.94:0/422007920 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f9333fff640 1 --2- 172.21.15.94:0/422007920 >> v2:172.21.15.145:3300/0 conn(0x7f93340c1aa0 0x7f93341529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831601953 0 0) 0x7f93340c28d0 con 0x7f933414f790 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9324003520 con 0x7f933414f790 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289085497 0 0) 0x7f9324003520 con 0x7f933414f790 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 >> v2:172.21.15.145:3300/0 conn(0x7f93340c1aa0 msgr2=0x7f93341529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 --2- 172.21.15.94:0/422007920 >> v2:172.21.15.145:3300/0 conn(0x7f93340c1aa0 0x7f93341529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 >> v1:172.21.15.145:6789/0 conn(0x7f9334155a40 legacy=0x7f9334155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340c2e90 con 0x7f933414f790 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9328001670 con 0x7f933414f790 2024-03-06T07:20:47.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 >> v1:172.21.15.94:6789/0 conn(0x7f933414f790 legacy=0x7f93340c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93240039b0 0x7f9324003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93240051c0 0x7f932400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c2680 con 0x7f9324004800 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932400c090 con 0x7f93240039b0 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.970+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932400c290 con 0x7f93240051c0 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f9333fff640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f9338eda640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93240039b0 0x7f9324003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f93396db640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93240051c0 0x7f932400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f9333fff640 1 -- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93240051c0 msgr2=0x7f932400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f9333fff640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93240051c0 0x7f932400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f9333fff640 1 -- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93240039b0 msgr2=0x7f9324003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:47.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f9333fff640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93240039b0 0x7f9324003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f9333fff640 1 -- 172.21.15.94:0/422007920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324003520 con 0x7f9324004800 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f93396db640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93240051c0 0x7f932400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.971+0000 7f93396db640 1 -- 172.21.15.94:0/422007920 reap_dead start 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f9333fff640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f9324004c80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9320009a90 tx=0x7f9324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f963f846bef5eb79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320039070 con 0x7f9324004800 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932002c680 con 0x7f9324004800 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f93337fe640 1 -- 172.21.15.94:0/422007920 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320009690 con 0x7f9324004800 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f933b13e640 1 -- 172.21.15.94:0/422007920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 msgr2=0x7f9324004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f933b13e640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f9324004c80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9320009a90 tx=0x7f9324005830 comp rx=0 tx=0).stop 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f933b13e640 1 -- 172.21.15.94:0/422007920 shutdown_connections 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f933b13e640 1 --2- 172.21.15.94:0/422007920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f933b13e640 1 -- 172.21.15.94:0/422007920 >> 172.21.15.94:0/422007920 conn(0x7f93340c6710 msgr2=0x7f93340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f933b13e640 1 -- 172.21.15.94:0/422007920 shutdown_connections 2024-03-06T07:20:47.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.972+0000 7f933b13e640 1 -- 172.21.15.94:0/422007920 wait complete. 2024-03-06T07:20:47.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.973+0000 7f933b13e640 1 Processor -- start 2024-03-06T07:20:47.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.973+0000 7f933b13e640 1 -- start start 2024-03-06T07:20:47.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f933b13e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f93341e96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f933b13e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f933414cbd0 0x7f9334143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f933b13e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9334143a40 0x7f93341ec450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f933b13e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933406cc00 con 0x7f9324004800 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f933b13e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c0d60 con 0x7f933414cbd0 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f933b13e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933406cd80 con 0x7f9334143a40 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f9338eda640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f93341e96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f9338eda640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f93341e96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48700/0 (socket says 172.21.15.94:48700) 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f9338eda640 1 -- 172.21.15.94:0/3826775696 learned_addr learned my addr 172.21.15.94:0/3826775696 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f9333fff640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f933414cbd0 0x7f9334143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.974+0000 7f93396db640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9334143a40 0x7f93341ec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f9338eda640 1 -- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9334143a40 msgr2=0x7f93341ec450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f9338eda640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9334143a40 0x7f93341ec450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f9338eda640 1 -- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f933414cbd0 msgr2=0x7f9334143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f9338eda640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f933414cbd0 0x7f9334143500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f9338eda640 1 -- 172.21.15.94:0/3826775696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334072900 con 0x7f9324004800 2024-03-06T07:20:47.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f9333fff640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f933414cbd0 0x7f9334143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f9338eda640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f93341e96f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9328007370 tx=0x7f9328007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f93317fa640 1 -- 172.21.15.94:0/3826775696 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800f9c0 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f93317fa640 1 -- 172.21.15.94:0/3826775696 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9328018070 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93341e9c30 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f93317fa640 1 -- 172.21.15.94:0/3826775696 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800fde0 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.975+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9334072e50 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.976+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93340acab0 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.976+0000 7f93317fa640 1 -- 172.21.15.94:0/3826775696 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f932801d460 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.977+0000 7f93317fa640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f931806b4d0 0x7f931806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.977+0000 7f93317fa640 1 -- 172.21.15.94:0/3826775696 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f932808a150 con 0x7f9324004800 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.977+0000 7f9333fff640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f931806b4d0 0x7f931806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:47.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.977+0000 7f9333fff640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f931806b4d0 0x7f931806d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9320009a90 tx=0x7f9320030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:47.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:47.981+0000 7f93317fa640 1 -- 172.21.15.94:0/3826775696 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93340acab0 con 0x7f9324004800 2024-03-06T07:20:48.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.219+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f933404f330 con 0x7f9324004800 2024-03-06T07:20:48.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.219+0000 7f93317fa640 1 -- 172.21.15.94:0/3826775696 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f9328059220 con 0x7f9324004800 2024-03-06T07:20:48.221 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:48.221 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:20:48.221 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:20:48.221 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:20:48.223 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:20:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.225 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.226 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:20:48.227 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:20:48.228 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:20:48.229 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.229 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.229 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.229 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.229 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.229 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.229 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:20:48.230 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:20:48.231 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:20:48.233 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.234 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:20:48.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:48.236 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:20:48.236 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:20:48.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f931806b4d0 msgr2=0x7f931806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f931806b4d0 0x7f931806d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9320009a90 tx=0x7f9320030000 comp rx=0 tx=0).stop 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 msgr2=0x7f93341e96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f93341e96f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9328007370 tx=0x7f9328007770 comp rx=0 tx=0).stop 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 shutdown_connections 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f931806b4d0 0x7f931806d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9334143a40 0x7f93341ec450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f933414cbd0 0x7f9334143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 --2- 172.21.15.94:0/3826775696 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9324004800 0x7f93341e96f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 >> 172.21.15.94:0/3826775696 conn(0x7f93340c6710 msgr2=0x7f93340c30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 shutdown_connections 2024-03-06T07:20:48.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.221+0000 7f933b13e640 1 -- 172.21.15.94:0/3826775696 wait complete. 2024-03-06T07:20:48.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:48.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.338+0000 7f25b3f8c640 1 Processor -- start 2024-03-06T07:20:48.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.338+0000 7f25b3f8c640 1 -- start start 2024-03-06T07:20:48.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.338+0000 7f25b3f8c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25ac0c1aa0 0x7f25ac1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b3f8c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0c28d0 con 0x7f25ac155a40 2024-03-06T07:20:48.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b3f8c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0c2bb0 con 0x7f25ac14f790 2024-03-06T07:20:48.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b3f8c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac0c2680 con 0x7f25ac0c1aa0 2024-03-06T07:20:48.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b1527640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25ac0c1aa0 0x7f25ac1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b1527640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25ac0c1aa0 0x7f25ac1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48708/0 (socket says 172.21.15.94:48708) 2024-03-06T07:20:48.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b2529640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f25ac155a40 0x7f25ac155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48612/0 (socket says 172.21.15.94:48612) 2024-03-06T07:20:48.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b1527640 1 -- 172.21.15.94:0/2760147270 learned_addr learned my addr 172.21.15.94:0/2760147270 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:48.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b1527640 1 -- 172.21.15.94:0/2760147270 >> v1:172.21.15.145:6789/0 conn(0x7f25ac14f790 legacy=0x7f25ac0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b1527640 1 -- 172.21.15.94:0/2760147270 >> v1:172.21.15.94:6789/0 conn(0x7f25ac155a40 legacy=0x7f25ac155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b0d26640 1 -- 172.21.15.94:0/2760147270 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134086762 0 0) 0x7f25ac0c28d0 con 0x7f25ac155a40 2024-03-06T07:20:48.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.339+0000 7f25b1527640 1 -- 172.21.15.94:0/2760147270 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac0c2e90 con 0x7f25ac0c1aa0 2024-03-06T07:20:48.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b1527640 1 --2- 172.21.15.94:0/2760147270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac0c1aa0 0x7f25ac1529d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0c2800 tx=0x7f259c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4605bbd2acc650c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b0d26640 1 -- 172.21.15.94:0/2760147270 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c02f840 con 0x7f25ac0c1aa0 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b0d26640 1 -- 172.21.15.94:0/2760147270 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c02f9e0 con 0x7f25ac0c1aa0 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b0d26640 1 -- 172.21.15.94:0/2760147270 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c02fdb0 con 0x7f25ac0c1aa0 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b3f8c640 1 -- 172.21.15.94:0/2760147270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac0c1aa0 msgr2=0x7f25ac1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/2760147270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac0c1aa0 0x7f25ac1529d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0c2800 tx=0x7f259c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b3f8c640 1 -- 172.21.15.94:0/2760147270 shutdown_connections 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/2760147270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac0c1aa0 0x7f25ac1529d0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.340+0000 7f25b3f8c640 1 -- 172.21.15.94:0/2760147270 >> 172.21.15.94:0/2760147270 conn(0x7f25ac0c6710 msgr2=0x7f25ac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.341+0000 7f25b3f8c640 1 -- 172.21.15.94:0/2760147270 shutdown_connections 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.341+0000 7f25b3f8c640 1 -- 172.21.15.94:0/2760147270 wait complete. 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 Processor -- start 2024-03-06T07:20:48.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 -- start start 2024-03-06T07:20:48.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25ac0c1aa0 0x7f25ac154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac14f790 0x7f25ac1ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25ac155a40 0x7f25ac1ec1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0c5e30 con 0x7f25ac14f790 2024-03-06T07:20:48.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0c0d60 con 0x7f25ac155a40 2024-03-06T07:20:48.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b3f8c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0c5fb0 con 0x7f25ac0c1aa0 2024-03-06T07:20:48.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b1527640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac14f790 0x7f25ac1ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b1527640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac14f790 0x7f25ac1ebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48716/0 (socket says 172.21.15.94:48716) 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.342+0000 7f25b1527640 1 -- 172.21.15.94:0/4171342735 learned_addr learned my addr 172.21.15.94:0/4171342735 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b2529640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25ac155a40 0x7f25ac1ec1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1527640 1 -- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25ac0c1aa0 msgr2=0x7f25ac154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1d28640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25ac0c1aa0 0x7f25ac154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1527640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25ac0c1aa0 0x7f25ac154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1527640 1 -- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25ac155a40 msgr2=0x7f25ac1ec1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1527640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25ac155a40 0x7f25ac1ec1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1527640 1 -- 172.21.15.94:0/4171342735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac1db590 con 0x7f25ac14f790 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b2529640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25ac155a40 0x7f25ac1ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1d28640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25ac0c1aa0 0x7f25ac154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:48.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b1527640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac14f790 0x7f25ac1ebc90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f259c0090b0 tx=0x7f259c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25a27fc640 1 -- 172.21.15.94:0/4171342735 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c047070 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25a27fc640 1 -- 172.21.15.94:0/4171342735 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c037cc0 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25a27fc640 1 -- 172.21.15.94:0/4171342735 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c004270 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac14dab0 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.343+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac155030 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.344+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac14d830 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.345+0000 7f25a27fc640 1 -- 172.21.15.94:0/4171342735 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f259c04a070 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.345+0000 7f25a27fc640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f259006b4d0 0x7f259006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.345+0000 7f25a27fc640 1 -- 172.21.15.94:0/4171342735 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f259c0ac320 con 0x7f25ac14f790 2024-03-06T07:20:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.345+0000 7f25b1d28640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f259006b4d0 0x7f259006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.346+0000 7f25b1d28640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f259006b4d0 0x7f259006d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f25a8000f60 tx=0x7f25a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.349+0000 7f25a27fc640 1 -- 172.21.15.94:0/4171342735 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25ac14d830 con 0x7f25ac14f790 2024-03-06T07:20:48.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.523+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25ac14d3e0 con 0x7f25ac14f790 2024-03-06T07:20:48.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.523+0000 7f25a27fc640 1 -- 172.21.15.94:0/4171342735 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+13165 (secure 0 0 0) 0x7f259c033030 con 0x7f25ac14f790 2024-03-06T07:20:48.525 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:48.526 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":70,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:47.845012+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f259006b4d0 msgr2=0x7f259006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f259006b4d0 0x7f259006d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f25a8000f60 tx=0x7f25a8009040 comp rx=0 tx=0).stop 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac14f790 msgr2=0x7f25ac1ebc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac14f790 0x7f25ac1ebc90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f259c0090b0 tx=0x7f259c004730 comp rx=0 tx=0).stop 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 shutdown_connections 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f259006b4d0 0x7f259006d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25ac155a40 0x7f25ac1ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25ac14f790 0x7f25ac1ebc90 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 --2- 172.21.15.94:0/4171342735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25ac0c1aa0 0x7f25ac154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 >> 172.21.15.94:0/4171342735 conn(0x7f25ac0c6710 msgr2=0x7f25ac0c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 shutdown_connections 2024-03-06T07:20:48.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.526+0000 7f25b3f8c640 1 -- 172.21.15.94:0/4171342735 wait complete. 2024-03-06T07:20:48.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:20:48.539 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.640+0000 7f9cc2ced640 1 Processor -- start 2024-03-06T07:20:48.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.640+0000 7f9cc2ced640 1 -- start start 2024-03-06T07:20:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc2ced640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9cbc06c900 0x7f9cbc0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc2ced640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc0a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc2ced640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc0c3ef0 con 0x7f9cbc0c5990 2024-03-06T07:20:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc2ced640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cbc1556d0 con 0x7f9cbc06c900 2024-03-06T07:20:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc2ced640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc1559b0 con 0x7f9cbc0ab170 2024-03-06T07:20:48.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc128a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc0a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cbbfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9cbc06c900 0x7f9cbc0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc0a89640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f9cbc0c5990 0x7f9cbc06c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38948/0 (socket says 172.21.15.94:38948) 2024-03-06T07:20:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cc0a89640 1 -- 172.21.15.94:0/3917126101 learned_addr learned my addr 172.21.15.94:0/3917126101 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cbbfff640 1 -- 172.21.15.94:0/3917126101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 msgr2=0x7f9cbc0a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cbbfff640 1 --2- 172.21.15.94:0/3917126101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc0a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cbbfff640 1 -- 172.21.15.94:0/3917126101 >> v1:172.21.15.145:6789/0 conn(0x7f9cbc0c5990 legacy=0x7f9cbc06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.641+0000 7f9cbbfff640 1 -- 172.21.15.94:0/3917126101 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc155c90 con 0x7f9cbc06c900 2024-03-06T07:20:48.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.642+0000 7f9cbbfff640 1 --2- 172.21.15.94:0/3917126101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 0x7f9cbc0c0ab0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9cb00097a0 tx=0x7f9cb002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74e2312388c0830 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.642+0000 7f9cbb7fe640 1 -- 172.21.15.94:0/3917126101 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0035020 con 0x7f9cbc06c900 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.642+0000 7f9cbb7fe640 1 -- 172.21.15.94:0/3917126101 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0031830 con 0x7f9cbc06c900 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cbb7fe640 1 -- 172.21.15.94:0/3917126101 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0031c00 con 0x7f9cbc06c900 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3917126101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 msgr2=0x7f9cbc0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3917126101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 0x7f9cbc0c0ab0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9cb00097a0 tx=0x7f9cb002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3917126101 shutdown_connections 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3917126101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc0a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3917126101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 0x7f9cbc0c0ab0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3917126101 >> 172.21.15.94:0/3917126101 conn(0x7f9cbc0c6710 msgr2=0x7f9cbc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3917126101 shutdown_connections 2024-03-06T07:20:48.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.643+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3917126101 wait complete. 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.644+0000 7f9cc2ced640 1 Processor -- start 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.644+0000 7f9cc2ced640 1 -- start start 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc2ced640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 0x7f9cbc0c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc2ced640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc1e8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc2ced640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9cbc0c5990 0x7f9cbc1e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc2ced640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc156610 con 0x7f9cbc0c5990 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc2ced640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc156490 con 0x7f9cbc06c900 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc2ced640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc0c3af0 con 0x7f9cbc0ab170 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc128a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9cbc0c5990 0x7f9cbc1e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc128a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9cbc0c5990 0x7f9cbc1e8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52034/0 (socket says 172.21.15.94:52034) 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc128a640 1 -- 172.21.15.94:0/3344524115 learned_addr learned my addr 172.21.15.94:0/3344524115 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cc0a89640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 0x7f9cbc0c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.645+0000 7f9cbbfff640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc1e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc128a640 1 -- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 msgr2=0x7f9cbc1e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc128a640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc1e8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc128a640 1 -- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 msgr2=0x7f9cbc0c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc128a640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 0x7f9cbc0c2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc128a640 1 -- 172.21.15.94:0/3344524115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc0c0aa0 con 0x7f9cbc0c5990 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc128a640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9cbc0c5990 0x7f9cbc1e8e60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00e930 tx=0x7f9cac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cb97fa640 1 -- 172.21.15.94:0/3344524115 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac017070 con 0x7f9cbc0c5990 2024-03-06T07:20:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cb97fa640 1 -- 172.21.15.94:0/3344524115 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac00c980 con 0x7f9cbc0c5990 2024-03-06T07:20:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc0c0d40 con 0x7f9cbc0c5990 2024-03-06T07:20:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cb97fa640 1 -- 172.21.15.94:0/3344524115 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00cd70 con 0x7f9cbc0c5990 2024-03-06T07:20:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.646+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc0c3000 con 0x7f9cbc0c5990 2024-03-06T07:20:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.647+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cbc067ad0 con 0x7f9cbc0c5990 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.647+0000 7f9cb97fa640 1 -- 172.21.15.94:0/3344524115 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cac01a070 con 0x7f9cbc0c5990 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.648+0000 7f9cb97fa640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ca806b4d0 0x7f9ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.648+0000 7f9cb97fa640 1 -- 172.21.15.94:0/3344524115 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9cac089eb0 con 0x7f9cbc0c5990 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.648+0000 7f9cc0a89640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ca806b4d0 0x7f9ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.649+0000 7f9cc0a89640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ca806b4d0 0x7f9ca806d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40012e0 tx=0x7f9cb4001990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.652+0000 7f9cb97fa640 1 -- 172.21.15.94:0/3344524115 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cbc067ad0 con 0x7f9cbc0c5990 2024-03-06T07:20:48.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:20:48.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.823+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9cbc04f330 con 0x7f9cbc0c5990 2024-03-06T07:20:48.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.823+0000 7f9cb97fa640 1 -- 172.21.15.94:0/3344524115 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9cac0589f0 con 0x7f9cbc0c5990 2024-03-06T07:20:48.825 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:20:48.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ca806b4d0 msgr2=0x7f9ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ca806b4d0 0x7f9ca806d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40012e0 tx=0x7f9cb4001990 comp rx=0 tx=0).stop 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9cbc0c5990 msgr2=0x7f9cbc1e8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9cbc0c5990 0x7f9cbc1e8e60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00e930 tx=0x7f9cac00edf0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 shutdown_connections 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ca806b4d0 0x7f9ca806d990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9cbc0c5990 0x7f9cbc1e8e60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cbc0ab170 0x7f9cbc1e8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 --2- 172.21.15.94:0/3344524115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cbc06c900 0x7f9cbc0c2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 >> 172.21.15.94:0/3344524115 conn(0x7f9cbc0c6710 msgr2=0x7f9cbc14f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 shutdown_connections 2024-03-06T07:20:48.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.825+0000 7f9cc2ced640 1 -- 172.21.15.94:0/3344524115 wait complete. 2024-03-06T07:20:48.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:20:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.938+0000 7f8c70765640 1 Processor -- start 2024-03-06T07:20:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.938+0000 7f8c70765640 1 -- start start 2024-03-06T07:20:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.938+0000 7f8c70765640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8c680c1aa0 0x7f8c681529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.938+0000 7f8c70765640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8c68155a40 0x7f8c68155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.938+0000 7f8c70765640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c680c28d0 con 0x7f8c6814f790 2024-03-06T07:20:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.938+0000 7f8c70765640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c680c2680 con 0x7f8c680c1aa0 2024-03-06T07:20:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.938+0000 7f8c70765640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c680c2bb0 con 0x7f8c68155a40 2024-03-06T07:20:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6dd00640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8c680c1aa0 0x7f8c681529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6e501640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8c6814f790 0x7f8c680c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59652/0 (socket says 172.21.15.94:59652) 2024-03-06T07:20:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6e501640 1 -- 172.21.15.94:0/3769859649 learned_addr learned my addr 172.21.15.94:0/3769859649 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6dd00640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8c680c1aa0 0x7f8c681529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52038/0 (socket says 172.21.15.94:52038) 2024-03-06T07:20:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6dd00640 1 -- 172.21.15.94:0/3769859649 >> v2:172.21.15.145:3300/0 conn(0x7f8c68155a40 msgr2=0x7f8c68155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6ed02640 1 --2- 172.21.15.94:0/3769859649 >> v2:172.21.15.145:3300/0 conn(0x7f8c68155a40 0x7f8c68155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6dd00640 1 --2- 172.21.15.94:0/3769859649 >> v2:172.21.15.145:3300/0 conn(0x7f8c68155a40 0x7f8c68155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6dd00640 1 -- 172.21.15.94:0/3769859649 >> v1:172.21.15.94:6789/0 conn(0x7f8c6814f790 legacy=0x7f8c680c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6dd00640 1 -- 172.21.15.94:0/3769859649 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680c2e90 con 0x7f8c680c1aa0 2024-03-06T07:20:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.939+0000 7f8c6d4ff640 1 -- 172.21.15.94:0/3769859649 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307621546 0 0) 0x7f8c680c28d0 con 0x7f8c6814f790 2024-03-06T07:20:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c6dd00640 1 --2- 172.21.15.94:0/3769859649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c680c1aa0 0x7f8c681529d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8c6806c960 tx=0x7f8c5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcb425ab3b82ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c6d4ff640 1 -- 172.21.15.94:0/3769859649 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58035020 con 0x7f8c680c1aa0 2024-03-06T07:20:48.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c6d4ff640 1 -- 172.21.15.94:0/3769859649 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c58031830 con 0x7f8c680c1aa0 2024-03-06T07:20:48.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c6d4ff640 1 -- 172.21.15.94:0/3769859649 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58031c00 con 0x7f8c680c1aa0 2024-03-06T07:20:48.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 -- 172.21.15.94:0/3769859649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c680c1aa0 msgr2=0x7f8c681529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 --2- 172.21.15.94:0/3769859649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c680c1aa0 0x7f8c681529d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8c6806c960 tx=0x7f8c5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 -- 172.21.15.94:0/3769859649 shutdown_connections 2024-03-06T07:20:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 --2- 172.21.15.94:0/3769859649 >> v2:172.21.15.145:3300/0 conn(0x7f8c68155a40 0x7f8c68155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 --2- 172.21.15.94:0/3769859649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c680c1aa0 0x7f8c681529d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 -- 172.21.15.94:0/3769859649 >> 172.21.15.94:0/3769859649 conn(0x7f8c680c6710 msgr2=0x7f8c680c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 -- 172.21.15.94:0/3769859649 shutdown_connections 2024-03-06T07:20:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.940+0000 7f8c70765640 1 -- 172.21.15.94:0/3769859649 wait complete. 2024-03-06T07:20:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 Processor -- start 2024-03-06T07:20:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 -- start start 2024-03-06T07:20:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c680c1aa0 0x7f8c681484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c6814f790 0x7f8c681df9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c68155a40 0x7f8c681ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c680c0d60 con 0x7f8c68155a40 2024-03-06T07:20:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c68156390 con 0x7f8c680c1aa0 2024-03-06T07:20:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c70765640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c680c5e30 con 0x7f8c6814f790 2024-03-06T07:20:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c6ed02640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c68155a40 0x7f8c681ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c6ed02640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c68155a40 0x7f8c681ee5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52052/0 (socket says 172.21.15.94:52052) 2024-03-06T07:20:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c6ed02640 1 -- 172.21.15.94:0/2483879768 learned_addr learned my addr 172.21.15.94:0/2483879768 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c6e501640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c680c1aa0 0x7f8c681484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.942+0000 7f8c6dd00640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c6814f790 0x7f8c681df9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c6ed02640 1 -- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c6814f790 msgr2=0x7f8c681df9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c6ed02640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c6814f790 0x7f8c681df9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c6ed02640 1 -- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c680c1aa0 msgr2=0x7f8c681484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c6ed02640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c680c1aa0 0x7f8c681484b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:48.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c6ed02640 1 -- 172.21.15.94:0/2483879768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68151820 con 0x7f8c68155a40 2024-03-06T07:20:48.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c6ed02640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c68155a40 0x7f8c681ee5c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000cc60 tx=0x7f8c6000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c5effd640 1 -- 172.21.15.94:0/2483879768 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60003990 con 0x7f8c68155a40 2024-03-06T07:20:48.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c5effd640 1 -- 172.21.15.94:0/2483879768 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c60003b30 con 0x7f8c68155a40 2024-03-06T07:20:48.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c681500e0 con 0x7f8c68155a40 2024-03-06T07:20:48.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c5effd640 1 -- 172.21.15.94:0/2483879768 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60004020 con 0x7f8c68155a40 2024-03-06T07:20:48.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.943+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c680c6b80 con 0x7f8c68155a40 2024-03-06T07:20:48.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.944+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6804f330 con 0x7f8c68155a40 2024-03-06T07:20:48.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.944+0000 7f8c5effd640 1 -- 172.21.15.94:0/2483879768 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c600041c0 con 0x7f8c68155a40 2024-03-06T07:20:48.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.945+0000 7f8c5effd640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c4c06b4d0 0x7f8c4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:48.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.945+0000 7f8c5effd640 1 -- 172.21.15.94:0/2483879768 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8c6008a3f0 con 0x7f8c68155a40 2024-03-06T07:20:48.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.945+0000 7f8c6e501640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c4c06b4d0 0x7f8c4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.946+0000 7f8c6e501640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c4c06b4d0 0x7f8c4c06d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8c64000f60 tx=0x7f8c64007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:48.949+0000 7f8c5effd640 1 -- 172.21.15.94:0/2483879768 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c60014030 con 0x7f8c68155a40 2024-03-06T07:20:49.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.123+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8c680c1f40 con 0x7f8c68155a40 2024-03-06T07:20:49.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.123+0000 7f8c5effd640 1 -- 172.21.15.94:0/2483879768 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v70) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8c60058e70 con 0x7f8c68155a40 2024-03-06T07:20:49.125 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:49.125 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:20:49.125 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:49.126 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:20:49.126 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:20:49.126 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:49.126 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:20:49.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c4c06b4d0 msgr2=0x7f8c4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c4c06b4d0 0x7f8c4c06d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8c64000f60 tx=0x7f8c64007d50 comp rx=0 tx=0).stop 2024-03-06T07:20:49.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c68155a40 msgr2=0x7f8c681ee5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c68155a40 0x7f8c681ee5c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000cc60 tx=0x7f8c6000a5c0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 shutdown_connections 2024-03-06T07:20:49.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c4c06b4d0 0x7f8c4c06d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c68155a40 0x7f8c681ee5c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c6814f790 0x7f8c681df9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 --2- 172.21.15.94:0/2483879768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c680c1aa0 0x7f8c681484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 >> 172.21.15.94:0/2483879768 conn(0x7f8c680c6710 msgr2=0x7f8c680a8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 shutdown_connections 2024-03-06T07:20:49.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.125+0000 7f8c70765640 1 -- 172.21.15.94:0/2483879768 wait complete. 2024-03-06T07:20:49.137 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:20:49.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:20:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.240+0000 7f35d646f640 1 Processor -- start 2024-03-06T07:20:49.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.240+0000 7f35d646f640 1 -- start start 2024-03-06T07:20:49.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35d646f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35d646f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35d00c2920 0x7f35d0156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35d646f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d0070430 con 0x7f35d00c1d10 2024-03-06T07:20:49.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35d646f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d00711f0 con 0x7f35d00c2920 2024-03-06T07:20:49.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35d646f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0070d60 con 0x7f35d00abb60 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35cffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35d4a0c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35d00c2920 0x7f35d0156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35cffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01502f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53326/0 (socket says 172.21.15.94:53326) 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35cf7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f35d00c1d10 0x7f35d0156440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38952/0 (socket says 172.21.15.94:38952) 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35cffff640 1 -- 172.21.15.94:0/3833911530 learned_addr learned my addr 172.21.15.94:0/3833911530 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35cffff640 1 -- 172.21.15.94:0/3833911530 >> v2:172.21.15.145:3300/0 conn(0x7f35d00c2920 msgr2=0x7f35d0156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.241+0000 7f35cffff640 1 --2- 172.21.15.94:0/3833911530 >> v2:172.21.15.145:3300/0 conn(0x7f35d00c2920 0x7f35d0156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.242+0000 7f35cffff640 1 -- 172.21.15.94:0/3833911530 >> v1:172.21.15.145:6789/0 conn(0x7f35d00c1d10 legacy=0x7f35d0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.242+0000 7f35cffff640 1 -- 172.21.15.94:0/3833911530 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d0070710 con 0x7f35d00abb60 2024-03-06T07:20:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.242+0000 7f35cffff640 1 --2- 172.21.15.94:0/3833911530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01502f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009870 tx=0x7f35c002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bde3386e2991ab69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.242+0000 7f35ceffd640 1 -- 172.21.15.94:0/3833911530 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c0035020 con 0x7f35d00abb60 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35ceffd640 1 -- 172.21.15.94:0/3833911530 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c0031640 con 0x7f35d00abb60 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35ceffd640 1 -- 172.21.15.94:0/3833911530 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c002d040 con 0x7f35d00abb60 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 -- 172.21.15.94:0/3833911530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 msgr2=0x7f35d01502f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 --2- 172.21.15.94:0/3833911530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01502f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009870 tx=0x7f35c002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 -- 172.21.15.94:0/3833911530 shutdown_connections 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 --2- 172.21.15.94:0/3833911530 >> v2:172.21.15.145:3300/0 conn(0x7f35d00c2920 0x7f35d0156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 --2- 172.21.15.94:0/3833911530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01502f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 -- 172.21.15.94:0/3833911530 >> 172.21.15.94:0/3833911530 conn(0x7f35d00c65b0 msgr2=0x7f35d00c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 -- 172.21.15.94:0/3833911530 shutdown_connections 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.243+0000 7f35d646f640 1 -- 172.21.15.94:0/3833911530 wait complete. 2024-03-06T07:20:49.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.244+0000 7f35d646f640 1 Processor -- start 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.244+0000 7f35d646f640 1 -- start start 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d646f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d646f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00c2920 0x7f35d0149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d646f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d00c0880 0x7f35d00bf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d646f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00abf40 con 0x7f35d00c2920 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d646f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0157090 con 0x7f35d00c0880 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d646f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00ac0c0 con 0x7f35d00abb60 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00c2920 0x7f35d0149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00c2920 0x7f35d0149c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52056/0 (socket says 172.21.15.94:52056) 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 -- 172.21.15.94:0/1344511240 learned_addr learned my addr 172.21.15.94:0/1344511240 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d4a0c640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d00c0880 0x7f35d00bf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cffff640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 -- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 msgr2=0x7f35d01496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 -- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d00c0880 msgr2=0x7f35d00bf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d00c0880 0x7f35d00bf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cf7fe640 1 -- 172.21.15.94:0/1344511240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d01ed180 con 0x7f35d00c2920 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35cffff640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.245+0000 7f35d4a0c640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d00c0880 0x7f35d00bf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.246+0000 7f35cf7fe640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00c2920 0x7f35d0149c30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c400 tx=0x7f35b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.246+0000 7f35ccff9640 1 -- 172.21.15.94:0/1344511240 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b80074e0 con 0x7f35d00c2920 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.246+0000 7f35ccff9640 1 -- 172.21.15.94:0/1344511240 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b8007680 con 0x7f35d00c2920 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.246+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d01dfe40 con 0x7f35d00c2920 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.246+0000 7f35ccff9640 1 -- 172.21.15.94:0/1344511240 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8004a00 con 0x7f35d00c2920 2024-03-06T07:20:49.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.246+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d01df080 con 0x7f35d00c2920 2024-03-06T07:20:49.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.247+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d004f330 con 0x7f35d00c2920 2024-03-06T07:20:49.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.247+0000 7f35ccff9640 1 -- 172.21.15.94:0/1344511240 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35b801a020 con 0x7f35d00c2920 2024-03-06T07:20:49.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.248+0000 7f35ccff9640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35b006b4d0 0x7f35b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.248+0000 7f35ccff9640 1 -- 172.21.15.94:0/1344511240 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f35b80897d0 con 0x7f35d00c2920 2024-03-06T07:20:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.248+0000 7f35cffff640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35b006b4d0 0x7f35b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.248+0000 7f35cffff640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35b006b4d0 0x7f35b006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35c0036000 tx=0x7f35c0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.252+0000 7f35ccff9640 1 -- 172.21.15.94:0/1344511240 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35b8010300 con 0x7f35d00c2920 2024-03-06T07:20:49.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.422+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f35d0068e00 con 0x7f35d00c2920 2024-03-06T07:20:49.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.857+0000 7f35ccff9640 1 -- 172.21.15.94:0/1344511240 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f35b80588a0 con 0x7f35d00c2920 2024-03-06T07:20:49.860 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:20:49.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35b006b4d0 msgr2=0x7f35b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35b006b4d0 0x7f35b006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35c0036000 tx=0x7f35c0008030 comp rx=0 tx=0).stop 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00c2920 msgr2=0x7f35d0149c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00c2920 0x7f35d0149c30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c400 tx=0x7f35b800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 shutdown_connections 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35b006b4d0 0x7f35b006d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d00c0880 0x7f35d00bf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d00c2920 0x7f35d0149c30 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 --2- 172.21.15.94:0/1344511240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d00abb60 0x7f35d01496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.860+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 >> 172.21.15.94:0/1344511240 conn(0x7f35d00c65b0 msgr2=0x7f35d01500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.861+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 shutdown_connections 2024-03-06T07:20:49.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.861+0000 7f35d646f640 1 -- 172.21.15.94:0/1344511240 wait complete. 2024-03-06T07:20:49.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:20:49.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.982+0000 7fd92ade4640 1 Processor -- start 2024-03-06T07:20:49.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.982+0000 7fd92ade4640 1 -- start start 2024-03-06T07:20:49.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd92ade4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240b0070 0x7fd924155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd92ade4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9240c5380 0x7fd9240c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd92ade4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240b0450 con 0x7fd92406ca70 2024-03-06T07:20:49.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd92ade4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92406c020 con 0x7fd9240c5380 2024-03-06T07:20:49.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd92ade4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240c77f0 con 0x7fd9240b0070 2024-03-06T07:20:49.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd929381640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9240c5380 0x7fd9240c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd923fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd92406ca70 0x7fd9240c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59654/0 (socket says 172.21.15.94:59654) 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd923fff640 1 -- 172.21.15.94:0/1579725176 learned_addr learned my addr 172.21.15.94:0/1579725176 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.983+0000 7fd929381640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9240c5380 0x7fd9240c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52072/0 (socket says 172.21.15.94:52072) 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd929381640 1 -- 172.21.15.94:0/1579725176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240b0070 msgr2=0x7fd924155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd928b80640 1 --2- 172.21.15.94:0/1579725176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240b0070 0x7fd924155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd9237fe640 1 -- 172.21.15.94:0/1579725176 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18323137 0 0) 0x7fd918001110 con 0x7fd92406ca70 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd929381640 1 --2- 172.21.15.94:0/1579725176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240b0070 0x7fd924155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd929381640 1 -- 172.21.15.94:0/1579725176 >> v1:172.21.15.94:6789/0 conn(0x7fd92406ca70 legacy=0x7fd9240c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd929381640 1 -- 172.21.15.94:0/1579725176 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240c7ad0 con 0x7fd9240c5380 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd929381640 1 --2- 172.21.15.94:0/1579725176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9240c5380 0x7fd9240c57d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009d90 tx=0x7fd91c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82ae91f31b6524d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.984+0000 7fd9237fe640 1 -- 172.21.15.94:0/1579725176 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c039070 con 0x7fd9240c5380 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd9237fe640 1 -- 172.21.15.94:0/1579725176 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91c02cbc0 con 0x7fd9240c5380 2024-03-06T07:20:49.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd9237fe640 1 -- 172.21.15.94:0/1579725176 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c034440 con 0x7fd9240c5380 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 -- 172.21.15.94:0/1579725176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9240c5380 msgr2=0x7fd9240c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 --2- 172.21.15.94:0/1579725176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9240c5380 0x7fd9240c57d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009d90 tx=0x7fd91c02be10 comp rx=0 tx=0).stop 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 -- 172.21.15.94:0/1579725176 shutdown_connections 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 --2- 172.21.15.94:0/1579725176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9240c5380 0x7fd9240c57d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 --2- 172.21.15.94:0/1579725176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240b0070 0x7fd924155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 -- 172.21.15.94:0/1579725176 >> 172.21.15.94:0/1579725176 conn(0x7fd9240ac590 msgr2=0x7fd9240ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 -- 172.21.15.94:0/1579725176 shutdown_connections 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.985+0000 7fd92ade4640 1 -- 172.21.15.94:0/1579725176 wait complete. 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.986+0000 7fd92ade4640 1 Processor -- start 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.986+0000 7fd92ade4640 1 -- start start 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd92ade4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd92406ca70 0x7fd924151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd92ade4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9240b0070 0x7fd92414ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd92ade4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240c5380 0x7fd92414b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd92ade4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92406cef0 con 0x7fd92406ca70 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd92ade4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92406c550 con 0x7fd9240b0070 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd92ade4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92406da80 con 0x7fd9240c5380 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd92406ca70 0x7fd924151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd92406ca70 0x7fd924151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52086/0 (socket says 172.21.15.94:52086) 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 -- 172.21.15.94:0/2319013756 learned_addr learned my addr 172.21.15.94:0/2319013756 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd923fff640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9240b0070 0x7fd92414ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd929381640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240c5380 0x7fd92414b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 -- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240c5380 msgr2=0x7fd92414b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240c5380 0x7fd92414b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 -- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9240b0070 msgr2=0x7fd92414ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9240b0070 0x7fd92414ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.987+0000 7fd928b80640 1 -- 172.21.15.94:0/2319013756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd924151eb0 con 0x7fd92406ca70 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.988+0000 7fd928b80640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd92406ca70 0x7fd924151550 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c7f0 tx=0x7fd91400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.988+0000 7fd9217fa640 1 -- 172.21.15.94:0/2319013756 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914015070 con 0x7fd92406ca70 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.988+0000 7fd9217fa640 1 -- 172.21.15.94:0/2319013756 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd914004240 con 0x7fd92406ca70 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.988+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd924157280 con 0x7fd92406ca70 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.988+0000 7fd9217fa640 1 -- 172.21.15.94:0/2319013756 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914007620 con 0x7fd92406ca70 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.988+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd924159550 con 0x7fd92406ca70 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.989+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92404f330 con 0x7fd92406ca70 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.989+0000 7fd9217fa640 1 -- 172.21.15.94:0/2319013756 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9140077c0 con 0x7fd92406ca70 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.989+0000 7fd9217fa640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd91006b4d0 0x7fd91006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:49.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.990+0000 7fd9217fa640 1 -- 172.21.15.94:0/2319013756 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fd91408a320 con 0x7fd92406ca70 2024-03-06T07:20:49.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.990+0000 7fd923fff640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd91006b4d0 0x7fd91006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:49.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.990+0000 7fd923fff640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd91006b4d0 0x7fd91006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd9241570c0 tx=0x7fd918007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:49.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:49.994+0000 7fd9217fa640 1 -- 172.21.15.94:0/2319013756 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd924151eb0 con 0x7fd92406ca70 2024-03-06T07:20:50.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.166+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd92406d860 con 0x7fd92406ca70 2024-03-06T07:20:50.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.865+0000 7fd9217fa640 1 -- 172.21.15.94:0/2319013756 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v72) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd914059280 con 0x7fd92406ca70 2024-03-06T07:20:50.868 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:20:50.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd91006b4d0 msgr2=0x7fd91006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd91006b4d0 0x7fd91006d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd9241570c0 tx=0x7fd918007970 comp rx=0 tx=0).stop 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd92406ca70 msgr2=0x7fd924151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd92406ca70 0x7fd924151550 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c7f0 tx=0x7fd91400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 shutdown_connections 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd91006b4d0 0x7fd91006d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9240c5380 0x7fd92414b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9240b0070 0x7fd92414ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 --2- 172.21.15.94:0/2319013756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd92406ca70 0x7fd924151550 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 >> 172.21.15.94:0/2319013756 conn(0x7fd9240ac590 msgr2=0x7fd9240ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:50.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 shutdown_connections 2024-03-06T07:20:50.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:50.868+0000 7fd92ade4640 1 -- 172.21.15.94:0/2319013756 wait complete. 2024-03-06T07:20:50.899 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:20:51.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.003+0000 7f52b5d65640 1 Processor -- start 2024-03-06T07:20:51.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.003+0000 7f52b5d65640 1 -- start start 2024-03-06T07:20:51.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52b5d65640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b00a6290 0x7f52b00a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52b5d65640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f52b0155cf0 0x7f52b00aa360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52b5d65640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b00aa8a0 con 0x7f52b00a8ff0 2024-03-06T07:20:51.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52b5d65640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b014f850 con 0x7f52b0155cf0 2024-03-06T07:20:51.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52b5d65640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b0156260 con 0x7f52b00a6290 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52af7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f52b00a8ff0 0x7f52b00ab1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59666/0 (socket says 172.21.15.94:59666) 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52af7fe640 1 -- 172.21.15.94:0/4293284183 learned_addr learned my addr 172.21.15.94:0/4293284183 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051599229 0 0) 0x7f52b00aa8a0 con 0x7f52b00a8ff0 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5298003520 con 0x7f52b00a8ff0 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52affff640 1 --2- 172.21.15.94:0/4293284183 >> v2:172.21.15.145:3300/0 conn(0x7f52b0155cf0 0x7f52b00aa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52aeffd640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b00a6290 0x7f52b00a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212909200 0 0) 0x7f5298003520 con 0x7f52b00a8ff0 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b00a6290 msgr2=0x7f52b00a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b00a6290 0x7f52b00a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 >> v2:172.21.15.145:3300/0 conn(0x7f52b0155cf0 msgr2=0x7f52b00aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 --2- 172.21.15.94:0/4293284183 >> v2:172.21.15.145:3300/0 conn(0x7f52b0155cf0 0x7f52b00aa360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b00c1f90 con 0x7f52b00a8ff0 2024-03-06T07:20:51.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f52a4001690 con 0x7f52b00a8ff0 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 >> v1:172.21.15.94:6789/0 conn(0x7f52b00a8ff0 legacy=0x7f52b00ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f5298003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5298004800 0x7f5298004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52980051c0 0x7f529800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b014f850 con 0x7f52980039b0 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b0156260 con 0x7f5298004800 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529800bf00 con 0x7f52980051c0 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52affff640 1 --2- 172.21.15.94:0/4293284183 >> v2:172.21.15.145:3300/0 conn(0x7f52b0155cf0 0x7f52b00aa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52af7fe640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f5298003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.004+0000 7f52aeffd640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5298004800 0x7f5298004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.005+0000 7f52affff640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52980051c0 0x7f529800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.005+0000 7f52af7fe640 1 -- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52980051c0 msgr2=0x7f529800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:51.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.005+0000 7f52af7fe640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52980051c0 0x7f529800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.005+0000 7f52af7fe640 1 -- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5298004800 msgr2=0x7f5298004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.005+0000 7f52af7fe640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5298004800 0x7f5298004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.005+0000 7f52af7fe640 1 -- 172.21.15.94:0/4293284183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5298003520 con 0x7f52980039b0 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.005+0000 7f52af7fe640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f5298003df0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f52a400c210 tx=0x7f5298005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf7f5cb4e76aa3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4007d70 con 0x7f52980039b0 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a4030070 con 0x7f52980039b0 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52ae7fc640 1 -- 172.21.15.94:0/4293284183 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a402cd60 con 0x7f52980039b0 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52b5d65640 1 -- 172.21.15.94:0/4293284183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 msgr2=0x7f5298003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52b5d65640 1 --2- 172.21.15.94:0/4293284183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f5298003df0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f52a400c210 tx=0x7f5298005830 comp rx=0 tx=0).stop 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52affff640 1 -- 172.21.15.94:0/4293284183 reap_dead start 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52b5d65640 1 -- 172.21.15.94:0/4293284183 shutdown_connections 2024-03-06T07:20:51.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52b5d65640 1 -- 172.21.15.94:0/4293284183 >> 172.21.15.94:0/4293284183 conn(0x7f52b006afc0 msgr2=0x7f52b006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52b5d65640 1 -- 172.21.15.94:0/4293284183 shutdown_connections 2024-03-06T07:20:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.006+0000 7f52b5d65640 1 -- 172.21.15.94:0/4293284183 wait complete. 2024-03-06T07:20:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.007+0000 7f52b5d65640 1 Processor -- start 2024-03-06T07:20:51.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.007+0000 7f52b5d65640 1 -- start start 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52b5d65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f52b00ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52b5d65640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b0150a40 0x7f52b0150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52b5d65640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52b0141620 0x7f52b0141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52b5d65640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00c1790 con 0x7f52980039b0 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52b5d65640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00c14b0 con 0x7f52b0141620 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52b5d65640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00c1a70 con 0x7f52b0150a40 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f52b00ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f52b00ab420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52092/0 (socket says 172.21.15.94:52092) 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 -- 172.21.15.94:0/651312741 learned_addr learned my addr 172.21.15.94:0/651312741 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:51.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52aeffd640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b0150a40 0x7f52b0150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52affff640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52b0141620 0x7f52b0141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 -- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b0150a40 msgr2=0x7f52b0150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b0150a40 0x7f52b0150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 -- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52b0141620 msgr2=0x7f52b0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52b0141620 0x7f52b0141aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52af7fe640 1 -- 172.21.15.94:0/651312741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0154040 con 0x7f52980039b0 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.008+0000 7f52affff640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52b0141620 0x7f52b0141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.009+0000 7f52af7fe640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f52b00ab420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f52a400c300 tx=0x7f52a4001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.009+0000 7f5297fff640 1 -- 172.21.15.94:0/651312741 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4045070 con 0x7f52980039b0 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.009+0000 7f5297fff640 1 -- 172.21.15.94:0/651312741 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a4004bf0 con 0x7f52980039b0 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.009+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b0154320 con 0x7f52980039b0 2024-03-06T07:20:51.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.009+0000 7f5297fff640 1 -- 172.21.15.94:0/651312741 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a4001cb0 con 0x7f52980039b0 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.009+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b00ab960 con 0x7f52980039b0 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.010+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b004f330 con 0x7f52980039b0 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.010+0000 7f5297fff640 1 -- 172.21.15.94:0/651312741 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52a4007070 con 0x7f52980039b0 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.011+0000 7f5297fff640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f529c06b500 0x7f529c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.011+0000 7f5297fff640 1 -- 172.21.15.94:0/651312741 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f52a40ab450 con 0x7f52980039b0 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.011+0000 7f52aeffd640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f529c06b500 0x7f529c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.012+0000 7f52aeffd640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f529c06b500 0x7f529c06d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f52b0147890 tx=0x7f52a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:51.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.015+0000 7f5297fff640 1 -- 172.21.15.94:0/651312741 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52a407a280 con 0x7f52980039b0 2024-03-06T07:20:51.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.221+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f52b00c24e0 con 0x7f52980039b0 2024-03-06T07:20:51.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.866+0000 7f5297fff640 1 -- 172.21.15.94:0/651312741 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v52) v1 ==== 174+0+0 (secure 0 0 0) 0x7f52a4079b80 con 0x7f52980039b0 2024-03-06T07:20:51.868 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 11 and data pool 12 2024-03-06T07:20:51.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f529c06b500 msgr2=0x7f529c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f529c06b500 0x7f529c06d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f52b0147890 tx=0x7f52a0008040 comp rx=0 tx=0).stop 2024-03-06T07:20:51.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 msgr2=0x7f52b00ab420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:51.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f52b00ab420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f52a400c300 tx=0x7f52a4001a60 comp rx=0 tx=0).stop 2024-03-06T07:20:51.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 shutdown_connections 2024-03-06T07:20:51.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f529c06b500 0x7f529c06d9c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52b0141620 0x7f52b0141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52b0150a40 0x7f52b0150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 --2- 172.21.15.94:0/651312741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52980039b0 0x7f52b00ab420 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:51.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.869+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 >> 172.21.15.94:0/651312741 conn(0x7f52b006afc0 msgr2=0x7f52b00a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:51.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.870+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 shutdown_connections 2024-03-06T07:20:51.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.870+0000 7f52b5d65640 1 -- 172.21.15.94:0/651312741 wait complete. 2024-03-06T07:20:51.897 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:51.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.998+0000 7fd9232df640 1 Processor -- start 2024-03-06T07:20:52.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.998+0000 7fd9232df640 1 -- start start 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.998+0000 7fd9232df640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd91c14f790 0x7fd91c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.998+0000 7fd9232df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c155a40 0x7fd91c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.998+0000 7fd9232df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c1563d0 con 0x7fd91c0c1aa0 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.998+0000 7fd9232df640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91c0c2680 con 0x7fd91c14f790 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.998+0000 7fd9232df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c0c28d0 con 0x7fd91c155a40 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92107b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd91c14f790 0x7fd91c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92187c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c155a40 0x7fd91c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92087a640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd91c0c1aa0 0x7fd91c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38966/0 (socket says 172.21.15.94:38966) 2024-03-06T07:20:52.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92087a640 1 -- 172.21.15.94:0/1919250198 learned_addr learned my addr 172.21.15.94:0/1919250198 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92107b640 1 -- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c155a40 msgr2=0x7fd91c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92107b640 1 --2- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c155a40 0x7fd91c155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92107b640 1 -- 172.21.15.94:0/1919250198 >> v1:172.21.15.145:6789/0 conn(0x7fd91c0c1aa0 legacy=0x7fd91c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92107b640 1 -- 172.21.15.94:0/1919250198 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0c2bb0 con 0x7fd91c14f790 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd913fff640 1 -- 172.21.15.94:0/1919250198 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791059705 0 0) 0x7fd91c1563d0 con 0x7fd91c0c1aa0 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:51.999+0000 7fd92187c640 1 --2- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c155a40 0x7fd91c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.000+0000 7fd92107b640 1 --2- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 0x7fd91c1529d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd9180097a0 tx=0x7fd91802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc7f878bdf3d34b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.000+0000 7fd913fff640 1 -- 172.21.15.94:0/1919250198 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918035020 con 0x7fd91c14f790 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.000+0000 7fd913fff640 1 -- 172.21.15.94:0/1919250198 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd918031830 con 0x7fd91c14f790 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.000+0000 7fd913fff640 1 -- 172.21.15.94:0/1919250198 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918031c00 con 0x7fd91c14f790 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 -- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 msgr2=0x7fd91c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 --2- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 0x7fd91c1529d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd9180097a0 tx=0x7fd91802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 -- 172.21.15.94:0/1919250198 shutdown_connections 2024-03-06T07:20:52.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 --2- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c155a40 0x7fd91c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 --2- 172.21.15.94:0/1919250198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 0x7fd91c1529d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 -- 172.21.15.94:0/1919250198 >> 172.21.15.94:0/1919250198 conn(0x7fd91c0c6710 msgr2=0x7fd91c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:52.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 -- 172.21.15.94:0/1919250198 shutdown_connections 2024-03-06T07:20:52.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.001+0000 7fd9232df640 1 -- 172.21.15.94:0/1919250198 wait complete. 2024-03-06T07:20:52.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.002+0000 7fd9232df640 1 Processor -- start 2024-03-06T07:20:52.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.002+0000 7fd9232df640 1 -- start start 2024-03-06T07:20:52.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.002+0000 7fd9232df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c0c1aa0 0x7fd91c1df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.002+0000 7fd9232df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 0x7fd91c1dc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.002+0000 7fd9232df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd91c155a40 0x7fd91c14e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.002+0000 7fd9232df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c0c0d60 con 0x7fd91c155a40 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.002+0000 7fd9232df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c155520 con 0x7fd91c14f790 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd9232df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c0c5e30 con 0x7fd91c0c1aa0 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd91c155a40 0x7fd91c14e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd91c155a40 0x7fd91c14e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52102/0 (socket says 172.21.15.94:52102) 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 -- 172.21.15.94:0/4125343516 learned_addr learned my addr 172.21.15.94:0/4125343516 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92087a640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 0x7fd91c1dc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92107b640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c0c1aa0 0x7fd91c1df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 -- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c0c1aa0 msgr2=0x7fd91c1df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c0c1aa0 0x7fd91c1df0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 -- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 msgr2=0x7fd91c1dc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 0x7fd91c1dc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 -- 172.21.15.94:0/4125343516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c149af0 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92107b640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c0c1aa0 0x7fd91c1df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd92187c640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd91c155a40 0x7fd91c14e490 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd91400efc0 tx=0x7fd91400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd911ffb640 1 -- 172.21.15.94:0/4125343516 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914018030 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd911ffb640 1 -- 172.21.15.94:0/4125343516 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91400cdf0 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd911ffb640 1 -- 172.21.15.94:0/4125343516 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914003bb0 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.003+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c150cd0 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.004+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c1517e0 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.004+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c068e00 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.005+0000 7fd911ffb640 1 -- 172.21.15.94:0/4125343516 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd914005d10 con 0x7fd91c155a40 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.005+0000 7fd911ffb640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd90006b4d0 0x7fd90006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.005+0000 7fd911ffb640 1 -- 172.21.15.94:0/4125343516 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd91408ac30 con 0x7fd91c155a40 2024-03-06T07:20:52.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.005+0000 7fd92107b640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd90006b4d0 0x7fd90006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.006+0000 7fd92107b640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd90006b4d0 0x7fd90006d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd918036000 tx=0x7fd918008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.009+0000 7fd911ffb640 1 -- 172.21.15.94:0/4125343516 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd91c068e00 con 0x7fd91c155a40 2024-03-06T07:20:52.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.182+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd91c04f330 con 0x7fd91c155a40 2024-03-06T07:20:52.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.183+0000 7fd911ffb640 1 -- 172.21.15.94:0/4125343516 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16958 (secure 0 0 0) 0x7fd914022070 con 0x7fd91c155a40 2024-03-06T07:20:52.185 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:52.185 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":73,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:51.223002+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:52.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd90006b4d0 msgr2=0x7fd90006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd90006b4d0 0x7fd90006d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd918036000 tx=0x7fd918008030 comp rx=0 tx=0).stop 2024-03-06T07:20:52.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd91c155a40 msgr2=0x7fd91c14e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd91c155a40 0x7fd91c14e490 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd91400efc0 tx=0x7fd91400c490 comp rx=0 tx=0).stop 2024-03-06T07:20:52.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 shutdown_connections 2024-03-06T07:20:52.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd90006b4d0 0x7fd90006d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd91c155a40 0x7fd91c14e490 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd91c14f790 0x7fd91c1dc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 --2- 172.21.15.94:0/4125343516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd91c0c1aa0 0x7fd91c1df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 >> 172.21.15.94:0/4125343516 conn(0x7fd91c0c6710 msgr2=0x7fd91c0c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:52.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 shutdown_connections 2024-03-06T07:20:52.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.185+0000 7fd9232df640 1 -- 172.21.15.94:0/4125343516 wait complete. 2024-03-06T07:20:52.199 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:52.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.298+0000 7f3f75681640 1 Processor -- start 2024-03-06T07:20:52.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.298+0000 7f3f75681640 1 -- start start 2024-03-06T07:20:52.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.298+0000 7f3f75681640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f7014f360 0x7f3f70155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.298+0000 7f3f75681640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3f70158b60 0x7f3f70158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.298+0000 7f3f75681640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700c73d0 con 0x7f3f700c88c0 2024-03-06T07:20:52.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.298+0000 7f3f75681640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f700c8dd0 con 0x7f3f70158b60 2024-03-06T07:20:52.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.298+0000 7f3f75681640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700be2e0 con 0x7f3f7014f360 2024-03-06T07:20:52.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6f7fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3f70158b60 0x7f3f70158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6effd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f7014f360 0x7f3f70155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6e7fc640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f3f700c88c0 0x7f3f700c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38978/0 (socket says 172.21.15.94:38978) 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6e7fc640 1 -- 172.21.15.94:0/3946741566 learned_addr learned my addr 172.21.15.94:0/3946741566 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6effd640 1 -- 172.21.15.94:0/3946741566 >> v2:172.21.15.145:3300/0 conn(0x7f3f70158b60 msgr2=0x7f3f70158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6effd640 1 --2- 172.21.15.94:0/3946741566 >> v2:172.21.15.145:3300/0 conn(0x7f3f70158b60 0x7f3f70158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6effd640 1 -- 172.21.15.94:0/3946741566 >> v1:172.21.15.145:6789/0 conn(0x7f3f700c88c0 legacy=0x7f3f700c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6effd640 1 -- 172.21.15.94:0/3946741566 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700c7670 con 0x7f3f7014f360 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6dffb640 1 -- 172.21.15.94:0/3946741566 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548987796 0 0) 0x7f3f700c73d0 con 0x7f3f700c88c0 2024-03-06T07:20:52.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.299+0000 7f3f6f7fe640 1 --2- 172.21.15.94:0/3946741566 >> v2:172.21.15.145:3300/0 conn(0x7f3f70158b60 0x7f3f70158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:52.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f6effd640 1 --2- 172.21.15.94:0/3946741566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f7014f360 0x7f3f70155c10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3f600097a0 tx=0x7f3f6002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ba91d582911c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f6dffb640 1 -- 172.21.15.94:0/3946741566 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60031780 con 0x7f3f7014f360 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f6dffb640 1 -- 172.21.15.94:0/3946741566 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f60037070 con 0x7f3f7014f360 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f6dffb640 1 -- 172.21.15.94:0/3946741566 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60031b40 con 0x7f3f7014f360 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f75681640 1 -- 172.21.15.94:0/3946741566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f7014f360 msgr2=0x7f3f70155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f75681640 1 --2- 172.21.15.94:0/3946741566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f7014f360 0x7f3f70155c10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3f600097a0 tx=0x7f3f6002b820 comp rx=0 tx=0).stop 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f75681640 1 -- 172.21.15.94:0/3946741566 shutdown_connections 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f75681640 1 --2- 172.21.15.94:0/3946741566 >> v2:172.21.15.145:3300/0 conn(0x7f3f70158b60 0x7f3f70158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f75681640 1 --2- 172.21.15.94:0/3946741566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f7014f360 0x7f3f70155c10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f75681640 1 -- 172.21.15.94:0/3946741566 >> 172.21.15.94:0/3946741566 conn(0x7f3f700ac570 msgr2=0x7f3f700ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:52.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.300+0000 7f3f75681640 1 -- 172.21.15.94:0/3946741566 shutdown_connections 2024-03-06T07:20:52.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.301+0000 7f3f75681640 1 -- 172.21.15.94:0/3946741566 wait complete. 2024-03-06T07:20:52.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 Processor -- start 2024-03-06T07:20:52.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 -- start start 2024-03-06T07:20:52.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f700c88c0 0x7f3f70151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f7014f360 0x7f3f70152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f70158b60 0x7f3f7014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700c5a50 con 0x7f3f7014f360 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700c7c30 con 0x7f3f700c88c0 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.302+0000 7f3f75681640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700c5bd0 con 0x7f3f70158b60 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f7014f360 0x7f3f70152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f7014f360 0x7f3f70152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52110/0 (socket says 172.21.15.94:52110) 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 -- 172.21.15.94:0/711126391 learned_addr learned my addr 172.21.15.94:0/711126391 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6effd640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f700c88c0 0x7f3f70151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6f7fe640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f70158b60 0x7f3f7014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 -- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f70158b60 msgr2=0x7f3f7014b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f70158b60 0x7f3f7014b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 -- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f700c88c0 msgr2=0x7f3f70151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f700c88c0 0x7f3f70151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 -- 172.21.15.94:0/711126391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f701502f0 con 0x7f3f7014f360 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f6e7fc640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f7014f360 0x7f3f70152170 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400c410 tx=0x7f3f6400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f577fe640 1 -- 172.21.15.94:0/711126391 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f640074e0 con 0x7f3f7014f360 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f577fe640 1 -- 172.21.15.94:0/711126391 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f640047e0 con 0x7f3f7014f360 2024-03-06T07:20:52.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f577fe640 1 -- 172.21.15.94:0/711126391 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64004bd0 con 0x7f3f7014f360 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f701505d0 con 0x7f3f7014f360 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.303+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f70150b90 con 0x7f3f7014f360 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.304+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7004f330 con 0x7f3f7014f360 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.305+0000 7f3f577fe640 1 -- 172.21.15.94:0/711126391 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f64011070 con 0x7f3f7014f360 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.305+0000 7f3f577fe640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f5c06b4d0 0x7f3f5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.305+0000 7f3f577fe640 1 -- 172.21.15.94:0/711126391 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3f64089d90 con 0x7f3f7014f360 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.305+0000 7f3f6effd640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f5c06b4d0 0x7f3f5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.306+0000 7f3f6effd640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f5c06b4d0 0x7f3f5c06d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3f60036000 tx=0x7f3f6003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.309+0000 7f3f577fe640 1 -- 172.21.15.94:0/711126391 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f64058b10 con 0x7f3f7014f360 2024-03-06T07:20:52.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.484+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f7006cda0 con 0x7f3f7014f360 2024-03-06T07:20:52.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.486+0000 7f3f577fe640 1 -- 172.21.15.94:0/711126391 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16958 (secure 0 0 0) 0x7f3f64058410 con 0x7f3f7014f360 2024-03-06T07:20:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:52.488 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":73,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:51.223002+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f5c06b4d0 msgr2=0x7f3f5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f5c06b4d0 0x7f3f5c06d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3f60036000 tx=0x7f3f6003d040 comp rx=0 tx=0).stop 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f7014f360 msgr2=0x7f3f70152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f7014f360 0x7f3f70152170 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400c410 tx=0x7f3f6400c810 comp rx=0 tx=0).stop 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 shutdown_connections 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f5c06b4d0 0x7f3f5c06d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f70158b60 0x7f3f7014b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f7014f360 0x7f3f70152170 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 --2- 172.21.15.94:0/711126391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f700c88c0 0x7f3f70151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 >> 172.21.15.94:0/711126391 conn(0x7f3f700ac570 msgr2=0x7f3f700ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 shutdown_connections 2024-03-06T07:20:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.488+0000 7f3f75681640 1 -- 172.21.15.94:0/711126391 wait complete. 2024-03-06T07:20:52.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:20:52.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.601+0000 7fe8979c7640 1 Processor -- start 2024-03-06T07:20:52.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe8979c7640 1 -- start start 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe8979c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8900b0070 0x7fe890155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe8979c7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe8900c5380 0x7fe8900c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe8979c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8900b0450 con 0x7fe89006ca70 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe8979c7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89006c020 con 0x7fe8900c5380 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe8979c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900c77f0 con 0x7fe8900b0070 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe895f64640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe8900c5380 0x7fe8900c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe895f64640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe8900c5380 0x7fe8900c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52116/0 (socket says 172.21.15.94:52116) 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.602+0000 7fe895f64640 1 -- 172.21.15.94:0/1342283343 learned_addr learned my addr 172.21.15.94:0/1342283343 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.603+0000 7fe895763640 1 --2- 172.21.15.94:0/1342283343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8900b0070 0x7fe890155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.603+0000 7fe895f64640 1 -- 172.21.15.94:0/1342283343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8900b0070 msgr2=0x7fe890155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:52.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.603+0000 7fe895f64640 1 --2- 172.21.15.94:0/1342283343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8900b0070 0x7fe890155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.603+0000 7fe895f64640 1 -- 172.21.15.94:0/1342283343 >> v1:172.21.15.145:6789/0 conn(0x7fe89006ca70 legacy=0x7fe8900c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.603+0000 7fe895f64640 1 -- 172.21.15.94:0/1342283343 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900c7ad0 con 0x7fe8900c5380 2024-03-06T07:20:52.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.603+0000 7fe895f64640 1 --2- 172.21.15.94:0/1342283343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900c5380 0x7fe8900c57d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe888009e50 tx=0x7fe88802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb944ba25bc1d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe887fff640 1 -- 172.21.15.94:0/1342283343 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88802ca70 con 0x7fe8900c5380 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe887fff640 1 -- 172.21.15.94:0/1342283343 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88802cc10 con 0x7fe8900c5380 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe887fff640 1 -- 172.21.15.94:0/1342283343 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880343f0 con 0x7fe8900c5380 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 -- 172.21.15.94:0/1342283343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900c5380 msgr2=0x7fe8900c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 --2- 172.21.15.94:0/1342283343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900c5380 0x7fe8900c57d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe888009e50 tx=0x7fe88802bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 -- 172.21.15.94:0/1342283343 shutdown_connections 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 --2- 172.21.15.94:0/1342283343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900c5380 0x7fe8900c57d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 --2- 172.21.15.94:0/1342283343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8900b0070 0x7fe890155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 -- 172.21.15.94:0/1342283343 >> 172.21.15.94:0/1342283343 conn(0x7fe8900ac590 msgr2=0x7fe8900ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:52.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 -- 172.21.15.94:0/1342283343 shutdown_connections 2024-03-06T07:20:52.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.604+0000 7fe8979c7640 1 -- 172.21.15.94:0/1342283343 wait complete. 2024-03-06T07:20:52.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.605+0000 7fe8979c7640 1 Processor -- start 2024-03-06T07:20:52.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.605+0000 7fe8979c7640 1 -- start start 2024-03-06T07:20:52.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe8979c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89006ca70 0x7fe890156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe8979c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900b0070 0x7fe89014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe8979c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8900c5380 0x7fe8901546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe8979c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89006cef0 con 0x7fe8900b0070 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe8979c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89006c550 con 0x7fe8900c5380 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe8979c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89006da80 con 0x7fe89006ca70 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900b0070 0x7fe89014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900b0070 0x7fe89014d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52118/0 (socket says 172.21.15.94:52118) 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 -- 172.21.15.94:0/2906255459 learned_addr learned my addr 172.21.15.94:0/2906255459 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe895f64640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8900c5380 0x7fe8901546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe895763640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89006ca70 0x7fe890156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 -- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89006ca70 msgr2=0x7fe890156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89006ca70 0x7fe890156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 -- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8900c5380 msgr2=0x7fe8901546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8900c5380 0x7fe8901546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.606+0000 7fe894f62640 1 -- 172.21.15.94:0/2906255459 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8901ebad0 con 0x7fe8900b0070 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.607+0000 7fe894f62640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900b0070 0x7fe89014d7f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe88000c430 tx=0x7fe88000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.607+0000 7fe885ffb640 1 -- 172.21.15.94:0/2906255459 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8800052d0 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.607+0000 7fe885ffb640 1 -- 172.21.15.94:0/2906255459 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880005490 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.607+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8901ebd70 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.607+0000 7fe885ffb640 1 -- 172.21.15.94:0/2906255459 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880007260 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.607+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe890156ad0 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.608+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8900adfe0 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.608+0000 7fe885ffb640 1 -- 172.21.15.94:0/2906255459 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe880007400 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.608+0000 7fe885ffb640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87406b500 0x7fe87406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.609+0000 7fe885ffb640 1 -- 172.21.15.94:0/2906255459 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe880089740 con 0x7fe8900b0070 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.609+0000 7fe895763640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87406b500 0x7fe87406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:52.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.609+0000 7fe895763640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87406b500 0x7fe87406d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009b40 tx=0x7fe88c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:52.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.613+0000 7fe885ffb640 1 -- 172.21.15.94:0/2906255459 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8900adfe0 con 0x7fe8900b0070 2024-03-06T07:20:52.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:52.821+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe89004f330 con 0x7fe8900b0070 2024-03-06T07:20:53.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.191+0000 7fe885ffb640 1 -- 172.21.15.94:0/2906255459 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe8900b0450 con 0x7fe8900b0070 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87406b500 msgr2=0x7fe87406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87406b500 0x7fe87406d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009b40 tx=0x7fe88c009190 comp rx=0 tx=0).stop 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900b0070 msgr2=0x7fe89014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900b0070 0x7fe89014d7f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe88000c430 tx=0x7fe88000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 shutdown_connections 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87406b500 0x7fe87406d9c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8900c5380 0x7fe8901546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8900b0070 0x7fe89014d7f0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 --2- 172.21.15.94:0/2906255459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89006ca70 0x7fe890156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.193+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 >> 172.21.15.94:0/2906255459 conn(0x7fe8900ac590 msgr2=0x7fe8900c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.194+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 shutdown_connections 2024-03-06T07:20:53.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.194+0000 7fe8979c7640 1 -- 172.21.15.94:0/2906255459 wait complete. 2024-03-06T07:20:53.206 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:20:53.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:20:53.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.308+0000 7fb96d239640 1 Processor -- start 2024-03-06T07:20:53.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.308+0000 7fb96d239640 1 -- start start 2024-03-06T07:20:53.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96d239640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9680c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96d239640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb9680ab170 0x7fb96806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96d239640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb96806c730 0x7fb9681523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96d239640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96806cbe0 con 0x7fb96806c730 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96d239640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9680c3ef0 con 0x7fb9680ab170 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96d239640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9681556d0 con 0x7fb9680c5990 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb96806c730 0x7fb9681523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb96806c730 0x7fb9681523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52120/0 (socket says 172.21.15.94:52120) 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 -- 172.21.15.94:0/3030939542 learned_addr learned my addr 172.21.15.94:0/3030939542 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96659c640 1 --2- 172.21.15.94:0/3030939542 >> v2:172.21.15.145:3300/0 conn(0x7fb9680ab170 0x7fb96806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb966d9d640 1 --2- 172.21.15.94:0/3030939542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9680c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 -- 172.21.15.94:0/3030939542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 msgr2=0x7fb9680c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 --2- 172.21.15.94:0/3030939542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9680c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 -- 172.21.15.94:0/3030939542 >> v2:172.21.15.145:3300/0 conn(0x7fb9680ab170 msgr2=0x7fb96806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 --2- 172.21.15.94:0/3030939542 >> v2:172.21.15.145:3300/0 conn(0x7fb9680ab170 0x7fb96806c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.309+0000 7fb96759e640 1 -- 172.21.15.94:0/3030939542 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9681559b0 con 0x7fb96806c730 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.310+0000 7fb96759e640 1 --2- 172.21.15.94:0/3030939542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681523f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb9680c4100 tx=0x7fb96002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c712b6b21eff4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.310+0000 7fb965d9b640 1 -- 172.21.15.94:0/3030939542 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96002ca70 con 0x7fb96806c730 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.310+0000 7fb965d9b640 1 -- 172.21.15.94:0/3030939542 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96002cc10 con 0x7fb96806c730 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.310+0000 7fb965d9b640 1 -- 172.21.15.94:0/3030939542 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9600343f0 con 0x7fb96806c730 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 -- 172.21.15.94:0/3030939542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 msgr2=0x7fb9681523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 --2- 172.21.15.94:0/3030939542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681523f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb9680c4100 tx=0x7fb96002bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 -- 172.21.15.94:0/3030939542 shutdown_connections 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 --2- 172.21.15.94:0/3030939542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681523f0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 --2- 172.21.15.94:0/3030939542 >> v2:172.21.15.145:3300/0 conn(0x7fb9680ab170 0x7fb96806c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 --2- 172.21.15.94:0/3030939542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9680c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 -- 172.21.15.94:0/3030939542 >> 172.21.15.94:0/3030939542 conn(0x7fb9680c6710 msgr2=0x7fb9680c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:53.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 -- 172.21.15.94:0/3030939542 shutdown_connections 2024-03-06T07:20:53.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.311+0000 7fb96d239640 1 -- 172.21.15.94:0/3030939542 wait complete. 2024-03-06T07:20:53.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.312+0000 7fb96d239640 1 Processor -- start 2024-03-06T07:20:53.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.312+0000 7fb96d239640 1 -- start start 2024-03-06T07:20:53.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96d239640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96d239640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9680ab170 0x7fb9680727d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96d239640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9681ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96d239640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb968156490 con 0x7fb96806c730 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96d239640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb968155f70 con 0x7fb9680ab170 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96d239640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb968156610 con 0x7fb9680c5990 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681e8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52134/0 (socket says 172.21.15.94:52134) 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 -- 172.21.15.94:0/3549671311 learned_addr learned my addr 172.21.15.94:0/3549671311 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96659c640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9680ab170 0x7fb9680727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb96759e640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9681ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 -- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 msgr2=0x7fb9681ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9681ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 -- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9680ab170 msgr2=0x7fb9680727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9680ab170 0x7fb9680727d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 -- 172.21.15.94:0/3549671311 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9680c5050 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb966d9d640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681e8dd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb95800e870 tx=0x7fb95800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb94f7fe640 1 -- 172.21.15.94:0/3549671311 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb958017070 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb94f7fe640 1 -- 172.21.15.94:0/3549671311 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95800c940 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.313+0000 7fb94f7fe640 1 -- 172.21.15.94:0/3549671311 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95800cd30 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.314+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9680c5330 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.314+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb968072d10 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.314+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96804f330 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.315+0000 7fb94f7fe640 1 -- 172.21.15.94:0/3549671311 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb95801a070 con 0x7fb96806c730 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.315+0000 7fb94f7fe640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb95406b4d0 0x7fb95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:53.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.316+0000 7fb94f7fe640 1 -- 172.21.15.94:0/3549671311 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb95808a430 con 0x7fb96806c730 2024-03-06T07:20:53.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.316+0000 7fb96659c640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb95406b4d0 0x7fb95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:53.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.316+0000 7fb96659c640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb95406b4d0 0x7fb95406d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb9680c2000 tx=0x7fb95c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:53.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.320+0000 7fb94f7fe640 1 -- 172.21.15.94:0/3549671311 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb958058b60 con 0x7fb96806c730 2024-03-06T07:20:53.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:53.530+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb9680c1740 con 0x7fb96806c730 2024-03-06T07:20:54.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.192+0000 7fb94f7fe640 1 -- 172.21.15.94:0/3549671311 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb95805cc00 con 0x7fb96806c730 2024-03-06T07:20:54.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb95406b4d0 msgr2=0x7fb95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb95406b4d0 0x7fb95406d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb9680c2000 tx=0x7fb95c009190 comp rx=0 tx=0).stop 2024-03-06T07:20:54.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 msgr2=0x7fb9681e8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681e8dd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb95800e870 tx=0x7fb95800ed30 comp rx=0 tx=0).stop 2024-03-06T07:20:54.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 shutdown_connections 2024-03-06T07:20:54.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb95406b4d0 0x7fb95406d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9680c5990 0x7fb9681ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9680ab170 0x7fb9680727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 --2- 172.21.15.94:0/3549671311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb96806c730 0x7fb9681e8dd0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 >> 172.21.15.94:0/3549671311 conn(0x7fb9680c6710 msgr2=0x7fb96814f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 shutdown_connections 2024-03-06T07:20:54.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.195+0000 7fb96d239640 1 -- 172.21.15.94:0/3549671311 wait complete. 2024-03-06T07:20:54.210 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:20:54.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:54.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.314+0000 7f90b0b13640 1 Processor -- start 2024-03-06T07:20:54.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.314+0000 7f90b0b13640 1 -- start start 2024-03-06T07:20:54.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90b0b13640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f90ac152960 0x7f90ac06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90b0b13640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f90ac1592d0 0x7f90ac1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90b0b13640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac0ac260 0x7f90ac155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90b0b13640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac066e30 con 0x7f90ac1592d0 2024-03-06T07:20:54.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90b0b13640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ac152d40 con 0x7f90ac152960 2024-03-06T07:20:54.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90b0b13640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0c1b50 con 0x7f90ac0ac260 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f90ac1592d0 0x7f90ac1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f90ac1592d0 0x7f90ac1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52138/0 (socket says 172.21.15.94:52138) 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 -- 172.21.15.94:0/998505707 learned_addr learned my addr 172.21.15.94:0/998505707 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90aad9d640 1 --2- 172.21.15.94:0/998505707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac0ac260 0x7f90ac155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90aa59c640 1 --2- 172.21.15.94:0/998505707 >> v2:172.21.15.145:3300/0 conn(0x7f90ac152960 0x7f90ac06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 -- 172.21.15.94:0/998505707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac0ac260 msgr2=0x7f90ac155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 --2- 172.21.15.94:0/998505707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac0ac260 0x7f90ac155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 -- 172.21.15.94:0/998505707 >> v2:172.21.15.145:3300/0 conn(0x7f90ac152960 msgr2=0x7f90ac06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 --2- 172.21.15.94:0/998505707 >> v2:172.21.15.145:3300/0 conn(0x7f90ac152960 0x7f90ac06dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.315+0000 7f90a9d9b640 1 -- 172.21.15.94:0/998505707 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac155d50 con 0x7f90ac1592d0 2024-03-06T07:20:54.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.316+0000 7f90a9d9b640 1 --2- 172.21.15.94:0/998505707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac1592d0 0x7f90ac1552d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f90ac06b1c0 tx=0x7f90a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f77fb331e3b7053 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.316+0000 7f90a959a640 1 -- 172.21.15.94:0/998505707 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0035020 con 0x7f90ac1592d0 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.316+0000 7f90a959a640 1 -- 172.21.15.94:0/998505707 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a0031870 con 0x7f90ac1592d0 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.316+0000 7f90a959a640 1 -- 172.21.15.94:0/998505707 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0031c40 con 0x7f90ac1592d0 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 -- 172.21.15.94:0/998505707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac1592d0 msgr2=0x7f90ac1552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 --2- 172.21.15.94:0/998505707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac1592d0 0x7f90ac1552d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f90ac06b1c0 tx=0x7f90a002b910 comp rx=0 tx=0).stop 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 -- 172.21.15.94:0/998505707 shutdown_connections 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 --2- 172.21.15.94:0/998505707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac0ac260 0x7f90ac155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 --2- 172.21.15.94:0/998505707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac1592d0 0x7f90ac1552d0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 --2- 172.21.15.94:0/998505707 >> v2:172.21.15.145:3300/0 conn(0x7f90ac152960 0x7f90ac06dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 -- 172.21.15.94:0/998505707 >> 172.21.15.94:0/998505707 conn(0x7f90ac0c70b0 msgr2=0x7f90ac0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 -- 172.21.15.94:0/998505707 shutdown_connections 2024-03-06T07:20:54.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.317+0000 7f90b0b13640 1 -- 172.21.15.94:0/998505707 wait complete. 2024-03-06T07:20:54.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.318+0000 7f90b0b13640 1 Processor -- start 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.318+0000 7f90b0b13640 1 -- start start 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90b0b13640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90ac0ac260 0x7f90ac06b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90b0b13640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac152960 0x7f90ac14af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90b0b13640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac14bea0 0x7f90ac1539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90b0b13640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac159760 con 0x7f90ac14bea0 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90b0b13640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0672c0 con 0x7f90ac0ac260 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90b0b13640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac0c20a0 con 0x7f90ac152960 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac14bea0 0x7f90ac1539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac14bea0 0x7f90ac1539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52148/0 (socket says 172.21.15.94:52148) 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 -- 172.21.15.94:0/3545208915 learned_addr learned my addr 172.21.15.94:0/3545208915 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aa59c640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90ac0ac260 0x7f90ac06b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90a9d9b640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac152960 0x7f90ac14af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 -- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac152960 msgr2=0x7f90ac14af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac152960 0x7f90ac14af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 -- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90ac0ac260 msgr2=0x7f90ac06b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90ac0ac260 0x7f90ac06b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aad9d640 1 -- 172.21.15.94:0/3545208915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac1df520 con 0x7f90ac14bea0 2024-03-06T07:20:54.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90aa59c640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90ac0ac260 0x7f90ac06b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:54.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.319+0000 7f90a9d9b640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac152960 0x7f90ac14af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.320+0000 7f90aad9d640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac14bea0 0x7f90ac1539a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f90a400eed0 tx=0x7f90a400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.320+0000 7f909affd640 1 -- 172.21.15.94:0/3545208915 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4018020 con 0x7f90ac14bea0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.320+0000 7f909affd640 1 -- 172.21.15.94:0/3545208915 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4014400 con 0x7f90ac14bea0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.320+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac1531e0 con 0x7f90ac14bea0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.320+0000 7f909affd640 1 -- 172.21.15.94:0/3545208915 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40147f0 con 0x7f90ac14bea0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.320+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac153480 con 0x7f90ac14bea0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.321+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ac0c2630 con 0x7f90ac14bea0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.321+0000 7f909affd640 1 -- 172.21.15.94:0/3545208915 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90a4005d10 con 0x7f90ac14bea0 2024-03-06T07:20:54.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.321+0000 7f909affd640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f908c06b4d0 0x7f908c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.322+0000 7f909affd640 1 -- 172.21.15.94:0/3545208915 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90a408a8c0 con 0x7f90ac14bea0 2024-03-06T07:20:54.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.322+0000 7f90aa59c640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f908c06b4d0 0x7f908c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.322+0000 7f90aa59c640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f908c06b4d0 0x7f908c06d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f90ac154a20 tx=0x7f909c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.326+0000 7f909affd640 1 -- 172.21.15.94:0/3545208915 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90ac0c2630 con 0x7f90ac14bea0 2024-03-06T07:20:54.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.533+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90ac04f330 con 0x7f90ac14bea0 2024-03-06T07:20:54.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.534+0000 7f909affd640 1 -- 172.21.15.94:0/3545208915 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f90a4022070 con 0x7f90ac14bea0 2024-03-06T07:20:54.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:54.537 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":35486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:20:53.531385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35492":{"gid":35492,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3537750458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3537750458},{"type":"v1","addr":"172.21.15.145:6837","nonce":3537750458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35492,"qdb_cluster":[35492]},"id":5}]} 2024-03-06T07:20:54.539 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 55 2024-03-06T07:20:54.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f908c06b4d0 msgr2=0x7f908c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f908c06b4d0 0x7f908c06d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f90ac154a20 tx=0x7f909c009190 comp rx=0 tx=0).stop 2024-03-06T07:20:54.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac14bea0 msgr2=0x7f90ac1539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac14bea0 0x7f90ac1539a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f90a400eed0 tx=0x7f90a400b480 comp rx=0 tx=0).stop 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 shutdown_connections 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f908c06b4d0 0x7f908c06d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90ac14bea0 0x7f90ac1539a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90ac152960 0x7f90ac14af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 --2- 172.21.15.94:0/3545208915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90ac0ac260 0x7f90ac06b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 >> 172.21.15.94:0/3545208915 conn(0x7f90ac0c70b0 msgr2=0x7f90ac0c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 shutdown_connections 2024-03-06T07:20:54.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.536+0000 7f90b0b13640 1 -- 172.21.15.94:0/3545208915 wait complete. 2024-03-06T07:20:54.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:54.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.645+0000 7f73b8c78640 1 Processor -- start 2024-03-06T07:20:54.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.646+0000 7f73b8c78640 1 -- start start 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.646+0000 7f73b8c78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b414f360 0x7f73b4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.646+0000 7f73b8c78640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73b4158b60 0x7f73b4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.646+0000 7f73b8c78640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b40c73d0 con 0x7f73b40c88c0 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.646+0000 7f73b8c78640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b40c8dd0 con 0x7f73b4158b60 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.646+0000 7f73b8c78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40be2e0 con 0x7f73b414f360 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b2d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73b4158b60 0x7f73b4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b2d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73b4158b60 0x7f73b4158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52164/0 (socket says 172.21.15.94:52164) 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b2d9d640 1 -- 172.21.15.94:0/255668874 learned_addr learned my addr 172.21.15.94:0/255668874 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b259c640 1 --2- 172.21.15.94:0/255668874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b414f360 0x7f73b4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b2d9d640 1 -- 172.21.15.94:0/255668874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b414f360 msgr2=0x7f73b4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b2d9d640 1 --2- 172.21.15.94:0/255668874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b414f360 0x7f73b4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b2d9d640 1 -- 172.21.15.94:0/255668874 >> v1:172.21.15.145:6789/0 conn(0x7f73b40c88c0 legacy=0x7f73b40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.647+0000 7f73b2d9d640 1 -- 172.21.15.94:0/255668874 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b40c7670 con 0x7f73b4158b60 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b2d9d640 1 --2- 172.21.15.94:0/255668874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b4158ff0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009e50 tx=0x7f73ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbb6b69a025c1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b159a640 1 -- 172.21.15.94:0/255668874 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac02ca70 con 0x7f73b4158b60 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b159a640 1 -- 172.21.15.94:0/255668874 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac02cc10 con 0x7f73b4158b60 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b159a640 1 -- 172.21.15.94:0/255668874 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0343f0 con 0x7f73b4158b60 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 -- 172.21.15.94:0/255668874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 msgr2=0x7f73b4158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 --2- 172.21.15.94:0/255668874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b4158ff0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009e50 tx=0x7f73ac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 -- 172.21.15.94:0/255668874 shutdown_connections 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 --2- 172.21.15.94:0/255668874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b4158ff0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 --2- 172.21.15.94:0/255668874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b414f360 0x7f73b4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 -- 172.21.15.94:0/255668874 >> 172.21.15.94:0/255668874 conn(0x7f73b40ac570 msgr2=0x7f73b40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 -- 172.21.15.94:0/255668874 shutdown_connections 2024-03-06T07:20:54.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.648+0000 7f73b8c78640 1 -- 172.21.15.94:0/255668874 wait complete. 2024-03-06T07:20:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 Processor -- start 2024-03-06T07:20:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 -- start start 2024-03-06T07:20:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b40c88c0 0x7f73b40c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73b414f360 0x7f73b4157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b40c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40c5a50 con 0x7f73b4158b60 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40c7c30 con 0x7f73b414f360 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b8c78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40c5bd0 con 0x7f73b40c88c0 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b40c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b40c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52172/0 (socket says 172.21.15.94:52172) 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b2d9d640 1 -- 172.21.15.94:0/865007946 learned_addr learned my addr 172.21.15.94:0/865007946 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b259c640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b40c88c0 0x7f73b40c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.650+0000 7f73b1d9b640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73b414f360 0x7f73b4157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b2d9d640 1 -- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b40c88c0 msgr2=0x7f73b40c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b2d9d640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b40c88c0 0x7f73b40c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b2d9d640 1 -- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73b414f360 msgr2=0x7f73b4157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b2d9d640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73b414f360 0x7f73b4157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b2d9d640 1 -- 172.21.15.94:0/865007946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b41587a0 con 0x7f73b4158b60 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b2d9d640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b40c3ab0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f73ac036040 tx=0x7f73ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73a2ffd640 1 -- 172.21.15.94:0/865007946 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac044020 con 0x7f73b4158b60 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73a2ffd640 1 -- 172.21.15.94:0/865007946 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac0406b0 con 0x7f73b4158b60 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b4157380 con 0x7f73b4158b60 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73a2ffd640 1 -- 172.21.15.94:0/865007946 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac009510 con 0x7f73b4158b60 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.651+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b41584c0 con 0x7f73b4158b60 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.652+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b404f330 con 0x7f73b4158b60 2024-03-06T07:20:54.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.653+0000 7f73a2ffd640 1 -- 172.21.15.94:0/865007946 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73ac047020 con 0x7f73b4158b60 2024-03-06T07:20:54.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.653+0000 7f73a2ffd640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f739406b4d0 0x7f739406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.653+0000 7f73a2ffd640 1 -- 172.21.15.94:0/865007946 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f73ac0acbe0 con 0x7f73b4158b60 2024-03-06T07:20:54.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.653+0000 7f73b259c640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f739406b4d0 0x7f739406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.654+0000 7f73b259c640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f739406b4d0 0x7f739406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f73b4155b50 tx=0x7f73a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.657+0000 7f73a2ffd640 1 -- 172.21.15.94:0/865007946 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73ac04e030 con 0x7f73b4158b60 2024-03-06T07:20:54.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.831+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73b40b0680 con 0x7f73b4158b60 2024-03-06T07:20:54.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.832+0000 7f73a2ffd640 1 -- 172.21.15.94:0/865007946 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16945 (secure 0 0 0) 0x7f73ac07b430 con 0x7f73b4158b60 2024-03-06T07:20:54.834 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:54.834 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":74,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:52.881152+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f739406b4d0 msgr2=0x7f739406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f739406b4d0 0x7f739406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f73b4155b50 tx=0x7f73a4009190 comp rx=0 tx=0).stop 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 msgr2=0x7f73b40c3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b40c3ab0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f73ac036040 tx=0x7f73ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 shutdown_connections 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f739406b4d0 0x7f739406d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73b4158b60 0x7f73b40c3ab0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73b414f360 0x7f73b4157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 --2- 172.21.15.94:0/865007946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73b40c88c0 0x7f73b40c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 >> 172.21.15.94:0/865007946 conn(0x7f73b40ac570 msgr2=0x7f73b414cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 shutdown_connections 2024-03-06T07:20:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.834+0000 7f73b8c78640 1 -- 172.21.15.94:0/865007946 wait complete. 2024-03-06T07:20:54.847 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:20:54.847 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:54.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.945+0000 7f0f9e015640 1 Processor -- start 2024-03-06T07:20:54.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.945+0000 7f0f9e015640 1 -- start start 2024-03-06T07:20:54.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f9e015640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f98155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f9e015640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0f98158b60 0x7f0f98158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f9e015640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c73d0 con 0x7f0f9814f360 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f9e015640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c8dd0 con 0x7f0f98158b60 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f9e015640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980be2e0 con 0x7f0f980c88c0 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f977fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0f9814f360 0x7f0f980c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59676/0 (socket says 172.21.15.94:59676) 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f977fe640 1 -- 172.21.15.94:0/2482435626 learned_addr learned my addr 172.21.15.94:0/2482435626 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f96ffd640 1 --2- 172.21.15.94:0/2482435626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f98155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.946+0000 7f0f97fff640 1 --2- 172.21.15.94:0/2482435626 >> v2:172.21.15.145:3300/0 conn(0x7f0f98158b60 0x7f0f98158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f967fc640 1 -- 172.21.15.94:0/2482435626 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353761368 0 0) 0x7f0f980c73d0 con 0x7f0f9814f360 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f967fc640 1 -- 172.21.15.94:0/2482435626 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80003520 con 0x7f0f9814f360 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f96ffd640 1 -- 172.21.15.94:0/2482435626 >> v2:172.21.15.145:3300/0 conn(0x7f0f98158b60 msgr2=0x7f0f98158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f96ffd640 1 --2- 172.21.15.94:0/2482435626 >> v2:172.21.15.145:3300/0 conn(0x7f0f98158b60 0x7f0f98158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f96ffd640 1 -- 172.21.15.94:0/2482435626 >> v1:172.21.15.94:6789/0 conn(0x7f0f9814f360 legacy=0x7f0f980c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f96ffd640 1 -- 172.21.15.94:0/2482435626 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980c7670 con 0x7f0f980c88c0 2024-03-06T07:20:54.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f97fff640 1 --2- 172.21.15.94:0/2482435626 >> v2:172.21.15.145:3300/0 conn(0x7f0f98158b60 0x7f0f98158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:54.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.947+0000 7f0f96ffd640 1 --2- 172.21.15.94:0/2482435626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f98155c10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0097a0 tx=0x7f0f8c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1114569d5833f533 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f967fc640 1 -- 172.21.15.94:0/2482435626 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0315a0 con 0x7f0f980c88c0 2024-03-06T07:20:54.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f967fc640 1 -- 172.21.15.94:0/2482435626 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8c037070 con 0x7f0f980c88c0 2024-03-06T07:20:54.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f967fc640 1 -- 172.21.15.94:0/2482435626 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c031960 con 0x7f0f980c88c0 2024-03-06T07:20:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 -- 172.21.15.94:0/2482435626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 msgr2=0x7f0f98155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 --2- 172.21.15.94:0/2482435626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f98155c10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c0097a0 tx=0x7f0f8c02b840 comp rx=0 tx=0).stop 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 -- 172.21.15.94:0/2482435626 shutdown_connections 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 --2- 172.21.15.94:0/2482435626 >> v2:172.21.15.145:3300/0 conn(0x7f0f98158b60 0x7f0f98158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 --2- 172.21.15.94:0/2482435626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f98155c10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 -- 172.21.15.94:0/2482435626 >> 172.21.15.94:0/2482435626 conn(0x7f0f980ac570 msgr2=0x7f0f980ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 -- 172.21.15.94:0/2482435626 shutdown_connections 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.948+0000 7f0f9e015640 1 -- 172.21.15.94:0/2482435626 wait complete. 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.949+0000 7f0f9e015640 1 Processor -- start 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.949+0000 7f0f9e015640 1 -- start start 2024-03-06T07:20:54.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f9e015640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f9814b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f9e015640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9814f360 0x7f0f981eab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f9e015640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f98158b60 0x7f0f981eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f9e015640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c5a50 con 0x7f0f9814f360 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f9e015640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c7c30 con 0x7f0f98158b60 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f9e015640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c5bd0 con 0x7f0f980c88c0 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9814f360 0x7f0f981eab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f96ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9814f360 0x7f0f981eab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52186/0 (socket says 172.21.15.94:52186) 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f96ffd640 1 -- 172.21.15.94:0/3699946285 learned_addr learned my addr 172.21.15.94:0/3699946285 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f977fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f9814b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.950+0000 7f0f97fff640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f98158b60 0x7f0f981eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f96ffd640 1 -- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 msgr2=0x7f0f9814b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f96ffd640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f9814b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f96ffd640 1 -- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f98158b60 msgr2=0x7f0f981eb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f96ffd640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f98158b60 0x7f0f981eb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f96ffd640 1 -- 172.21.15.94:0/3699946285 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980c1290 con 0x7f0f9814f360 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f97fff640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f98158b60 0x7f0f981eb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f96ffd640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9814f360 0x7f0f981eab30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c034040 tx=0x7f0f8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f7ffff640 1 -- 172.21.15.94:0/3699946285 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c032430 con 0x7f0f9814f360 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f7ffff640 1 -- 172.21.15.94:0/3699946285 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8c0325d0 con 0x7f0f9814f360 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9814c130 con 0x7f0f9814f360 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f7ffff640 1 -- 172.21.15.94:0/3699946285 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c0329c0 con 0x7f0f9814f360 2024-03-06T07:20:54.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.951+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f981eb650 con 0x7f0f9814f360 2024-03-06T07:20:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.952+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9804f330 con 0x7f0f9814f360 2024-03-06T07:20:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.953+0000 7f0f7ffff640 1 -- 172.21.15.94:0/3699946285 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f8c032b60 con 0x7f0f9814f360 2024-03-06T07:20:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.953+0000 7f0f7ffff640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f8406b4d0 0x7f0f8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.953+0000 7f0f7ffff640 1 -- 172.21.15.94:0/3699946285 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0f8c02f030 con 0x7f0f9814f360 2024-03-06T07:20:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.953+0000 7f0f977fe640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f8406b4d0 0x7f0f8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.953+0000 7f0f977fe640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f8406b4d0 0x7f0f8406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0f88005880 tx=0x7f0f88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:54.957+0000 7f0f7ffff640 1 -- 172.21.15.94:0/3699946285 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f8c040070 con 0x7f0f9814f360 2024-03-06T07:20:55.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.126+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f981eb820 con 0x7f0f8406b4d0 2024-03-06T07:20:55.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.132+0000 7f0f7ffff640 1 -- 172.21.15.94:0/3699946285 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252075 (secure 0 0 0) 0x7f0f981eb820 con 0x7f0f8406b4d0 2024-03-06T07:20:55.136 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:55.138 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f8406b4d0 msgr2=0x7f0f8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f8406b4d0 0x7f0f8406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0f88005880 tx=0x7f0f88009040 comp rx=0 tx=0).stop 2024-03-06T07:20:55.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9814f360 msgr2=0x7f0f981eab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9814f360 0x7f0f981eab30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c034040 tx=0x7f0f8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 shutdown_connections 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f8406b4d0 0x7f0f8406d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f98158b60 0x7f0f981eb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9814f360 0x7f0f981eab30 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 --2- 172.21.15.94:0/3699946285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f980c88c0 0x7f0f9814b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 >> 172.21.15.94:0/3699946285 conn(0x7f0f980ac570 msgr2=0x7f0f9814daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 shutdown_connections 2024-03-06T07:20:55.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.135+0000 7f0f9e015640 1 -- 172.21.15.94:0/3699946285 wait complete. 2024-03-06T07:20:55.148 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-03-06T07:20:54.669911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244100,"kb_used_data":7104,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730620,"statfs":{"total":773094113280,"available":772844154880,"internally_reserved":0,"allocated":7274496,"data_stored":4520389,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.810536"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898885+0000","last_change":"2024-03-06T07:20:51.896309+0000","last_active":"2024-03-06T07:20:51.898885+0000","last_peered":"2024-03-06T07:20:51.898885+0000","last_clean":"2024-03-06T07:20:51.898885+0000","last_became_active":"2024-03-06T07:20:51.896009+0000","last_became_peered":"2024-03-06T07:20:51.896009+0000","last_unstale":"2024-03-06T07:20:51.898885+0000","last_undegraded":"2024-03-06T07:20:51.898885+0000","last_fullsized":"2024-03-06T07:20:51.898885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896823+0000","last_change":"2024-03-06T07:20:51.895412+0000","last_active":"2024-03-06T07:20:51.896823+0000","last_peered":"2024-03-06T07:20:51.896823+0000","last_clean":"2024-03-06T07:20:51.896823+0000","last_became_active":"2024-03-06T07:20:51.895169+0000","last_became_peered":"2024-03-06T07:20:51.895169+0000","last_unstale":"2024-03-06T07:20:51.896823+0000","last_undegraded":"2024-03-06T07:20:51.896823+0000","last_fullsized":"2024-03-06T07:20:51.896823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:20.484391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879871+0000","last_change":"2024-03-06T07:20:51.878718+0000","last_active":"2024-03-06T07:20:51.879871+0000","last_peered":"2024-03-06T07:20:51.879871+0000","last_clean":"2024-03-06T07:20:51.879871+0000","last_became_active":"2024-03-06T07:20:51.878467+0000","last_became_peered":"2024-03-06T07:20:51.878467+0000","last_unstale":"2024-03-06T07:20:51.879871+0000","last_undegraded":"2024-03-06T07:20:51.879871+0000","last_fullsized":"2024-03-06T07:20:51.879871+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896201+0000","last_change":"2024-03-06T07:20:51.892550+0000","last_active":"2024-03-06T07:20:51.896201+0000","last_peered":"2024-03-06T07:20:51.896201+0000","last_clean":"2024-03-06T07:20:51.896201+0000","last_became_active":"2024-03-06T07:20:51.892264+0000","last_became_peered":"2024-03-06T07:20:51.892264+0000","last_unstale":"2024-03-06T07:20:51.896201+0000","last_undegraded":"2024-03-06T07:20:51.896201+0000","last_fullsized":"2024-03-06T07:20:51.896201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:29.691807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896287+0000","last_change":"2024-03-06T07:20:51.893305+0000","last_active":"2024-03-06T07:20:51.896287+0000","last_peered":"2024-03-06T07:20:51.896287+0000","last_clean":"2024-03-06T07:20:51.896287+0000","last_became_active":"2024-03-06T07:20:51.893041+0000","last_became_peered":"2024-03-06T07:20:51.893041+0000","last_unstale":"2024-03-06T07:20:51.896287+0000","last_undegraded":"2024-03-06T07:20:51.896287+0000","last_fullsized":"2024-03-06T07:20:51.896287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:04.441365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894594+0000","last_change":"2024-03-06T07:20:51.893812+0000","last_active":"2024-03-06T07:20:51.894594+0000","last_peered":"2024-03-06T07:20:51.894594+0000","last_clean":"2024-03-06T07:20:51.894594+0000","last_became_active":"2024-03-06T07:20:51.891649+0000","last_became_peered":"2024-03-06T07:20:51.891649+0000","last_unstale":"2024-03-06T07:20:51.894594+0000","last_undegraded":"2024-03-06T07:20:51.894594+0000","last_fullsized":"2024-03-06T07:20:51.894594+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:38.106753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896991+0000","last_change":"2024-03-06T07:20:51.894141+0000","last_active":"2024-03-06T07:20:51.896991+0000","last_peered":"2024-03-06T07:20:51.896991+0000","last_clean":"2024-03-06T07:20:51.896991+0000","last_became_active":"2024-03-06T07:20:51.893793+0000","last_became_peered":"2024-03-06T07:20:51.893793+0000","last_unstale":"2024-03-06T07:20:51.896991+0000","last_undegraded":"2024-03-06T07:20:51.896991+0000","last_fullsized":"2024-03-06T07:20:51.896991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:57.061120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896737+0000","last_change":"2024-03-06T07:20:51.894037+0000","last_active":"2024-03-06T07:20:51.896737+0000","last_peered":"2024-03-06T07:20:51.896737+0000","last_clean":"2024-03-06T07:20:51.896737+0000","last_became_active":"2024-03-06T07:20:51.893773+0000","last_became_peered":"2024-03-06T07:20:51.893773+0000","last_unstale":"2024-03-06T07:20:51.896737+0000","last_undegraded":"2024-03-06T07:20:51.896737+0000","last_fullsized":"2024-03-06T07:20:51.896737+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:47.454570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890036+0000","last_change":"2024-03-06T07:20:51.886223+0000","last_active":"2024-03-06T07:20:51.890036+0000","last_peered":"2024-03-06T07:20:51.890036+0000","last_clean":"2024-03-06T07:20:51.890036+0000","last_became_active":"2024-03-06T07:20:51.885985+0000","last_became_peered":"2024-03-06T07:20:51.885985+0000","last_unstale":"2024-03-06T07:20:51.890036+0000","last_undegraded":"2024-03-06T07:20:51.890036+0000","last_fullsized":"2024-03-06T07:20:51.890036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:44.358219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893550+0000","last_change":"2024-03-06T07:20:51.890465+0000","last_active":"2024-03-06T07:20:51.893550+0000","last_peered":"2024-03-06T07:20:51.893550+0000","last_clean":"2024-03-06T07:20:51.893550+0000","last_became_active":"2024-03-06T07:20:51.890183+0000","last_became_peered":"2024-03-06T07:20:51.890183+0000","last_unstale":"2024-03-06T07:20:51.893550+0000","last_undegraded":"2024-03-06T07:20:51.893550+0000","last_fullsized":"2024-03-06T07:20:51.893550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:36.873911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898264+0000","last_change":"2024-03-06T07:20:51.897836+0000","last_active":"2024-03-06T07:20:51.898264+0000","last_peered":"2024-03-06T07:20:51.898264+0000","last_clean":"2024-03-06T07:20:51.898264+0000","last_became_active":"2024-03-06T07:20:51.897551+0000","last_became_peered":"2024-03-06T07:20:51.897551+0000","last_unstale":"2024-03-06T07:20:51.898264+0000","last_undegraded":"2024-03-06T07:20:51.898264+0000","last_fullsized":"2024-03-06T07:20:51.898264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:06.737114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889290+0000","last_change":"2024-03-06T07:20:51.886475+0000","last_active":"2024-03-06T07:20:51.889290+0000","last_peered":"2024-03-06T07:20:51.889290+0000","last_clean":"2024-03-06T07:20:51.889290+0000","last_became_active":"2024-03-06T07:20:51.886218+0000","last_became_peered":"2024-03-06T07:20:51.886218+0000","last_unstale":"2024-03-06T07:20:51.889290+0000","last_undegraded":"2024-03-06T07:20:51.889290+0000","last_fullsized":"2024-03-06T07:20:51.889290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:35.092363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890597+0000","last_change":"2024-03-06T07:20:51.887999+0000","last_active":"2024-03-06T07:20:51.890597+0000","last_peered":"2024-03-06T07:20:51.890597+0000","last_clean":"2024-03-06T07:20:51.890597+0000","last_became_active":"2024-03-06T07:20:51.887749+0000","last_became_peered":"2024-03-06T07:20:51.887749+0000","last_unstale":"2024-03-06T07:20:51.890597+0000","last_undegraded":"2024-03-06T07:20:51.890597+0000","last_fullsized":"2024-03-06T07:20:51.890597+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:33.108778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893833+0000","last_change":"2024-03-06T07:20:51.892140+0000","last_active":"2024-03-06T07:20:51.893833+0000","last_peered":"2024-03-06T07:20:51.893833+0000","last_clean":"2024-03-06T07:20:51.893833+0000","last_became_active":"2024-03-06T07:20:51.891195+0000","last_became_peered":"2024-03-06T07:20:51.891195+0000","last_unstale":"2024-03-06T07:20:51.893833+0000","last_undegraded":"2024-03-06T07:20:51.893833+0000","last_fullsized":"2024-03-06T07:20:51.893833+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896553+0000","last_change":"2024-03-06T07:20:51.894735+0000","last_active":"2024-03-06T07:20:51.896553+0000","last_peered":"2024-03-06T07:20:51.896553+0000","last_clean":"2024-03-06T07:20:51.896553+0000","last_became_active":"2024-03-06T07:20:51.894479+0000","last_became_peered":"2024-03-06T07:20:51.894479+0000","last_unstale":"2024-03-06T07:20:51.896553+0000","last_undegraded":"2024-03-06T07:20:51.896553+0000","last_fullsized":"2024-03-06T07:20:51.896553+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:57.260587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896658+0000","last_change":"2024-03-06T07:20:51.892450+0000","last_active":"2024-03-06T07:20:51.896658+0000","last_peered":"2024-03-06T07:20:51.896658+0000","last_clean":"2024-03-06T07:20:51.896658+0000","last_became_active":"2024-03-06T07:20:51.892184+0000","last_became_peered":"2024-03-06T07:20:51.892184+0000","last_unstale":"2024-03-06T07:20:51.896658+0000","last_undegraded":"2024-03-06T07:20:51.896658+0000","last_fullsized":"2024-03-06T07:20:51.896658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893014+0000","last_change":"2024-03-06T07:20:51.891492+0000","last_active":"2024-03-06T07:20:51.893014+0000","last_peered":"2024-03-06T07:20:51.893014+0000","last_clean":"2024-03-06T07:20:51.893014+0000","last_became_active":"2024-03-06T07:20:51.891212+0000","last_became_peered":"2024-03-06T07:20:51.891212+0000","last_unstale":"2024-03-06T07:20:51.893014+0000","last_undegraded":"2024-03-06T07:20:51.893014+0000","last_fullsized":"2024-03-06T07:20:51.893014+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:05.321792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879250+0000","last_change":"2024-03-06T07:20:51.876410+0000","last_active":"2024-03-06T07:20:51.879250+0000","last_peered":"2024-03-06T07:20:51.879250+0000","last_clean":"2024-03-06T07:20:51.879250+0000","last_became_active":"2024-03-06T07:20:51.876155+0000","last_became_peered":"2024-03-06T07:20:51.876155+0000","last_unstale":"2024-03-06T07:20:51.879250+0000","last_undegraded":"2024-03-06T07:20:51.879250+0000","last_fullsized":"2024-03-06T07:20:51.879250+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.885514+0000","last_change":"2024-03-06T07:20:51.880019+0000","last_active":"2024-03-06T07:20:51.885514+0000","last_peered":"2024-03-06T07:20:51.885514+0000","last_clean":"2024-03-06T07:20:51.885514+0000","last_became_active":"2024-03-06T07:20:51.879756+0000","last_became_peered":"2024-03-06T07:20:51.879756+0000","last_unstale":"2024-03-06T07:20:51.885514+0000","last_undegraded":"2024-03-06T07:20:51.885514+0000","last_fullsized":"2024-03-06T07:20:51.885514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.892658+0000","last_change":"2024-03-06T07:20:51.890815+0000","last_active":"2024-03-06T07:20:51.892658+0000","last_peered":"2024-03-06T07:20:51.892658+0000","last_clean":"2024-03-06T07:20:51.892658+0000","last_became_active":"2024-03-06T07:20:51.890558+0000","last_became_peered":"2024-03-06T07:20:51.890558+0000","last_unstale":"2024-03-06T07:20:51.892658+0000","last_undegraded":"2024-03-06T07:20:51.892658+0000","last_fullsized":"2024-03-06T07:20:51.892658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:55.266280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890439+0000","last_change":"2024-03-06T07:20:51.884508+0000","last_active":"2024-03-06T07:20:51.890439+0000","last_peered":"2024-03-06T07:20:51.890439+0000","last_clean":"2024-03-06T07:20:51.890439+0000","last_became_active":"2024-03-06T07:20:51.884237+0000","last_became_peered":"2024-03-06T07:20:51.884237+0000","last_unstale":"2024-03-06T07:20:51.890439+0000","last_undegraded":"2024-03-06T07:20:51.890439+0000","last_fullsized":"2024-03-06T07:20:51.890439+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:36.241725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890945+0000","last_change":"2024-03-06T07:20:51.890118+0000","last_active":"2024-03-06T07:20:51.890945+0000","last_peered":"2024-03-06T07:20:51.890945+0000","last_clean":"2024-03-06T07:20:51.890945+0000","last_became_active":"2024-03-06T07:20:51.888426+0000","last_became_peered":"2024-03-06T07:20:51.888426+0000","last_unstale":"2024-03-06T07:20:51.890945+0000","last_undegraded":"2024-03-06T07:20:51.890945+0000","last_fullsized":"2024-03-06T07:20:51.890945+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:07.818700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890241+0000","last_change":"2024-03-06T07:20:51.885627+0000","last_active":"2024-03-06T07:20:51.890241+0000","last_peered":"2024-03-06T07:20:51.890241+0000","last_clean":"2024-03-06T07:20:51.890241+0000","last_became_active":"2024-03-06T07:20:51.885338+0000","last_became_peered":"2024-03-06T07:20:51.885338+0000","last_unstale":"2024-03-06T07:20:51.890241+0000","last_undegraded":"2024-03-06T07:20:51.890241+0000","last_fullsized":"2024-03-06T07:20:51.890241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:35.720895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886307+0000","last_change":"2024-03-06T07:20:51.884448+0000","last_active":"2024-03-06T07:20:51.886307+0000","last_peered":"2024-03-06T07:20:51.886307+0000","last_clean":"2024-03-06T07:20:51.886307+0000","last_became_active":"2024-03-06T07:20:51.884176+0000","last_became_peered":"2024-03-06T07:20:51.884176+0000","last_unstale":"2024-03-06T07:20:51.886307+0000","last_undegraded":"2024-03-06T07:20:51.886307+0000","last_fullsized":"2024-03-06T07:20:51.886307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:09.591477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894742+0000","last_change":"2024-03-06T07:20:51.886062+0000","last_active":"2024-03-06T07:20:51.894742+0000","last_peered":"2024-03-06T07:20:51.894742+0000","last_clean":"2024-03-06T07:20:51.894742+0000","last_became_active":"2024-03-06T07:20:51.885799+0000","last_became_peered":"2024-03-06T07:20:51.885799+0000","last_unstale":"2024-03-06T07:20:51.894742+0000","last_undegraded":"2024-03-06T07:20:51.894742+0000","last_fullsized":"2024-03-06T07:20:51.894742+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:11.878187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890256+0000","last_change":"2024-03-06T07:20:51.883614+0000","last_active":"2024-03-06T07:20:51.890256+0000","last_peered":"2024-03-06T07:20:51.890256+0000","last_clean":"2024-03-06T07:20:51.890256+0000","last_became_active":"2024-03-06T07:20:51.883266+0000","last_became_peered":"2024-03-06T07:20:51.883266+0000","last_unstale":"2024-03-06T07:20:51.890256+0000","last_undegraded":"2024-03-06T07:20:51.890256+0000","last_fullsized":"2024-03-06T07:20:51.890256+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.418498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.897131+0000","last_change":"2024-03-06T07:20:51.893301+0000","last_active":"2024-03-06T07:20:51.897131+0000","last_peered":"2024-03-06T07:20:51.897131+0000","last_clean":"2024-03-06T07:20:51.897131+0000","last_became_active":"2024-03-06T07:20:51.892946+0000","last_became_peered":"2024-03-06T07:20:51.892946+0000","last_unstale":"2024-03-06T07:20:51.897131+0000","last_undegraded":"2024-03-06T07:20:51.897131+0000","last_fullsized":"2024-03-06T07:20:51.897131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893448+0000","last_change":"2024-03-06T07:20:51.891689+0000","last_active":"2024-03-06T07:20:51.893448+0000","last_peered":"2024-03-06T07:20:51.893448+0000","last_clean":"2024-03-06T07:20:51.893448+0000","last_became_active":"2024-03-06T07:20:51.891341+0000","last_became_peered":"2024-03-06T07:20:51.891341+0000","last_unstale":"2024-03-06T07:20:51.893448+0000","last_undegraded":"2024-03-06T07:20:51.893448+0000","last_fullsized":"2024-03-06T07:20:51.893448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:32.214282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.895035+0000","last_change":"2024-03-06T07:20:51.887454+0000","last_active":"2024-03-06T07:20:51.895035+0000","last_peered":"2024-03-06T07:20:51.895035+0000","last_clean":"2024-03-06T07:20:51.895035+0000","last_became_active":"2024-03-06T07:20:51.887188+0000","last_became_peered":"2024-03-06T07:20:51.887188+0000","last_unstale":"2024-03-06T07:20:51.895035+0000","last_undegraded":"2024-03-06T07:20:51.895035+0000","last_fullsized":"2024-03-06T07:20:51.895035+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:35.622801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.900513+0000","last_change":"2024-03-06T07:20:51.899932+0000","last_active":"2024-03-06T07:20:51.900513+0000","last_peered":"2024-03-06T07:20:51.900513+0000","last_clean":"2024-03-06T07:20:51.900513+0000","last_became_active":"2024-03-06T07:20:51.899569+0000","last_became_peered":"2024-03-06T07:20:51.899569+0000","last_unstale":"2024-03-06T07:20:51.900513+0000","last_undegraded":"2024-03-06T07:20:51.900513+0000","last_fullsized":"2024-03-06T07:20:51.900513+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889559+0000","last_change":"2024-03-06T07:20:51.888537+0000","last_active":"2024-03-06T07:20:51.889559+0000","last_peered":"2024-03-06T07:20:51.889559+0000","last_clean":"2024-03-06T07:20:51.889559+0000","last_became_active":"2024-03-06T07:20:51.888281+0000","last_became_peered":"2024-03-06T07:20:51.888281+0000","last_unstale":"2024-03-06T07:20:51.889559+0000","last_undegraded":"2024-03-06T07:20:51.889559+0000","last_fullsized":"2024-03-06T07:20:51.889559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:57.061120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898696+0000","last_change":"2024-03-06T07:20:51.896158+0000","last_active":"2024-03-06T07:20:51.898696+0000","last_peered":"2024-03-06T07:20:51.898696+0000","last_clean":"2024-03-06T07:20:51.898696+0000","last_became_active":"2024-03-06T07:20:51.895889+0000","last_became_peered":"2024-03-06T07:20:51.895889+0000","last_unstale":"2024-03-06T07:20:51.898696+0000","last_undegraded":"2024-03-06T07:20:51.898696+0000","last_fullsized":"2024-03-06T07:20:51.898696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879095+0000","last_change":"2024-03-06T07:20:51.876393+0000","last_active":"2024-03-06T07:20:51.879095+0000","last_peered":"2024-03-06T07:20:51.879095+0000","last_clean":"2024-03-06T07:20:51.879095+0000","last_became_active":"2024-03-06T07:20:51.876059+0000","last_became_peered":"2024-03-06T07:20:51.876059+0000","last_unstale":"2024-03-06T07:20:51.879095+0000","last_undegraded":"2024-03-06T07:20:51.879095+0000","last_fullsized":"2024-03-06T07:20:51.879095+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.897856+0000","last_change":"2024-03-06T07:20:50.879586+0000","last_active":"2024-03-06T07:20:51.897856+0000","last_peered":"2024-03-06T07:20:51.897856+0000","last_clean":"2024-03-06T07:20:51.897856+0000","last_became_active":"2024-03-06T07:20:50.879319+0000","last_became_peered":"2024-03-06T07:20:50.879319+0000","last_unstale":"2024-03-06T07:20:51.897856+0000","last_undegraded":"2024-03-06T07:20:51.897856+0000","last_fullsized":"2024-03-06T07:20:51.897856+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:58.810754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886209+0000","last_change":"2024-03-06T07:20:51.884046+0000","last_active":"2024-03-06T07:20:51.886209+0000","last_peered":"2024-03-06T07:20:51.886209+0000","last_clean":"2024-03-06T07:20:51.886209+0000","last_became_active":"2024-03-06T07:20:51.883593+0000","last_became_peered":"2024-03-06T07:20:51.883593+0000","last_unstale":"2024-03-06T07:20:51.886209+0000","last_undegraded":"2024-03-06T07:20:51.886209+0000","last_fullsized":"2024-03-06T07:20:51.886209+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:53.956654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896404+0000","last_change":"2024-03-06T07:20:50.888715+0000","last_active":"2024-03-06T07:20:51.896404+0000","last_peered":"2024-03-06T07:20:51.896404+0000","last_clean":"2024-03-06T07:20:51.896404+0000","last_became_active":"2024-03-06T07:20:50.888140+0000","last_became_peered":"2024-03-06T07:20:50.888140+0000","last_unstale":"2024-03-06T07:20:51.896404+0000","last_undegraded":"2024-03-06T07:20:51.896404+0000","last_fullsized":"2024-03-06T07:20:51.896404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:34.087293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889183+0000","last_change":"2024-03-06T07:20:51.886764+0000","last_active":"2024-03-06T07:20:51.889183+0000","last_peered":"2024-03-06T07:20:51.889183+0000","last_clean":"2024-03-06T07:20:51.889183+0000","last_became_active":"2024-03-06T07:20:51.886187+0000","last_became_peered":"2024-03-06T07:20:51.886187+0000","last_unstale":"2024-03-06T07:20:51.889183+0000","last_undegraded":"2024-03-06T07:20:51.889183+0000","last_fullsized":"2024-03-06T07:20:51.889183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:28.691080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899922+0000","last_change":"2024-03-06T07:20:50.885147+0000","last_active":"2024-03-06T07:20:51.899922+0000","last_peered":"2024-03-06T07:20:51.899922+0000","last_clean":"2024-03-06T07:20:51.899922+0000","last_became_active":"2024-03-06T07:20:50.884811+0000","last_became_peered":"2024-03-06T07:20:50.884811+0000","last_unstale":"2024-03-06T07:20:51.899922+0000","last_undegraded":"2024-03-06T07:20:51.899922+0000","last_fullsized":"2024-03-06T07:20:51.899922+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:24.667879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890491+0000","last_change":"2024-03-06T07:20:51.887534+0000","last_active":"2024-03-06T07:20:51.890491+0000","last_peered":"2024-03-06T07:20:51.890491+0000","last_clean":"2024-03-06T07:20:51.890491+0000","last_became_active":"2024-03-06T07:20:51.887239+0000","last_became_peered":"2024-03-06T07:20:51.887239+0000","last_unstale":"2024-03-06T07:20:51.890491+0000","last_undegraded":"2024-03-06T07:20:51.890491+0000","last_fullsized":"2024-03-06T07:20:51.890491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:53.314738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890865+0000","last_change":"2024-03-06T07:20:50.892916+0000","last_active":"2024-03-06T07:20:51.890865+0000","last_peered":"2024-03-06T07:20:51.890865+0000","last_clean":"2024-03-06T07:20:51.890865+0000","last_became_active":"2024-03-06T07:20:50.892629+0000","last_became_peered":"2024-03-06T07:20:50.892629+0000","last_unstale":"2024-03-06T07:20:51.890865+0000","last_undegraded":"2024-03-06T07:20:51.890865+0000","last_fullsized":"2024-03-06T07:20:51.890865+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:12.808948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896846+0000","last_change":"2024-03-06T07:20:51.895988+0000","last_active":"2024-03-06T07:20:51.896846+0000","last_peered":"2024-03-06T07:20:51.896846+0000","last_clean":"2024-03-06T07:20:51.896846+0000","last_became_active":"2024-03-06T07:20:51.895747+0000","last_became_peered":"2024-03-06T07:20:51.895747+0000","last_unstale":"2024-03-06T07:20:51.896846+0000","last_undegraded":"2024-03-06T07:20:51.896846+0000","last_fullsized":"2024-03-06T07:20:51.896846+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.418498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891812+0000","last_change":"2024-03-06T07:20:50.893482+0000","last_active":"2024-03-06T07:20:51.891812+0000","last_peered":"2024-03-06T07:20:51.891812+0000","last_clean":"2024-03-06T07:20:51.891812+0000","last_became_active":"2024-03-06T07:20:50.893153+0000","last_became_peered":"2024-03-06T07:20:50.893153+0000","last_unstale":"2024-03-06T07:20:51.891812+0000","last_undegraded":"2024-03-06T07:20:51.891812+0000","last_fullsized":"2024-03-06T07:20:51.891812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:07.775768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898098+0000","last_change":"2024-03-06T07:20:51.897529+0000","last_active":"2024-03-06T07:20:51.898098+0000","last_peered":"2024-03-06T07:20:51.898098+0000","last_clean":"2024-03-06T07:20:51.898098+0000","last_became_active":"2024-03-06T07:20:51.897181+0000","last_became_peered":"2024-03-06T07:20:51.897181+0000","last_unstale":"2024-03-06T07:20:51.898098+0000","last_undegraded":"2024-03-06T07:20:51.898098+0000","last_fullsized":"2024-03-06T07:20:51.898098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:35.092363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890796+0000","last_change":"2024-03-06T07:20:50.886303+0000","last_active":"2024-03-06T07:20:51.890796+0000","last_peered":"2024-03-06T07:20:51.890796+0000","last_clean":"2024-03-06T07:20:51.890796+0000","last_became_active":"2024-03-06T07:20:50.886013+0000","last_became_peered":"2024-03-06T07:20:50.886013+0000","last_unstale":"2024-03-06T07:20:51.890796+0000","last_undegraded":"2024-03-06T07:20:51.890796+0000","last_fullsized":"2024-03-06T07:20:51.890796+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:34.617731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889833+0000","last_change":"2024-03-06T07:20:51.883834+0000","last_active":"2024-03-06T07:20:51.889833+0000","last_peered":"2024-03-06T07:20:51.889833+0000","last_clean":"2024-03-06T07:20:51.889833+0000","last_became_active":"2024-03-06T07:20:51.883553+0000","last_became_peered":"2024-03-06T07:20:51.883553+0000","last_unstale":"2024-03-06T07:20:51.889833+0000","last_undegraded":"2024-03-06T07:20:51.889833+0000","last_fullsized":"2024-03-06T07:20:51.889833+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:37.311544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.875471+0000","last_change":"2024-03-06T07:20:50.881449+0000","last_active":"2024-03-06T07:20:51.875471+0000","last_peered":"2024-03-06T07:20:51.875471+0000","last_clean":"2024-03-06T07:20:51.875471+0000","last_became_active":"2024-03-06T07:20:50.881172+0000","last_became_peered":"2024-03-06T07:20:50.881172+0000","last_unstale":"2024-03-06T07:20:51.875471+0000","last_undegraded":"2024-03-06T07:20:51.875471+0000","last_fullsized":"2024-03-06T07:20:51.875471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.317522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.895950+0000","last_change":"2024-03-06T07:20:51.893411+0000","last_active":"2024-03-06T07:20:51.895950+0000","last_peered":"2024-03-06T07:20:51.895950+0000","last_clean":"2024-03-06T07:20:51.895950+0000","last_became_active":"2024-03-06T07:20:51.893149+0000","last_became_peered":"2024-03-06T07:20:51.893149+0000","last_unstale":"2024-03-06T07:20:51.895950+0000","last_undegraded":"2024-03-06T07:20:51.895950+0000","last_fullsized":"2024-03-06T07:20:51.895950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:09.457018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.878563+0000","last_change":"2024-03-06T07:20:50.890159+0000","last_active":"2024-03-06T07:20:51.878563+0000","last_peered":"2024-03-06T07:20:51.878563+0000","last_clean":"2024-03-06T07:20:51.878563+0000","last_became_active":"2024-03-06T07:20:50.889863+0000","last_became_peered":"2024-03-06T07:20:50.889863+0000","last_unstale":"2024-03-06T07:20:51.878563+0000","last_undegraded":"2024-03-06T07:20:51.878563+0000","last_fullsized":"2024-03-06T07:20:51.878563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.303590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899855+0000","last_change":"2024-03-06T07:20:51.899199+0000","last_active":"2024-03-06T07:20:51.899855+0000","last_peered":"2024-03-06T07:20:51.899855+0000","last_clean":"2024-03-06T07:20:51.899855+0000","last_became_active":"2024-03-06T07:20:51.898622+0000","last_became_peered":"2024-03-06T07:20:51.898622+0000","last_unstale":"2024-03-06T07:20:51.899855+0000","last_undegraded":"2024-03-06T07:20:51.899855+0000","last_fullsized":"2024-03-06T07:20:51.899855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:36.873911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.883407+0000","last_change":"2024-03-06T07:20:50.891619+0000","last_active":"2024-03-06T07:20:51.883407+0000","last_peered":"2024-03-06T07:20:51.883407+0000","last_clean":"2024-03-06T07:20:51.883407+0000","last_became_active":"2024-03-06T07:20:50.891305+0000","last_became_peered":"2024-03-06T07:20:50.891305+0000","last_unstale":"2024-03-06T07:20:51.883407+0000","last_undegraded":"2024-03-06T07:20:51.883407+0000","last_fullsized":"2024-03-06T07:20:51.883407+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.732044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890297+0000","last_change":"2024-03-06T07:20:51.887972+0000","last_active":"2024-03-06T07:20:51.890297+0000","last_peered":"2024-03-06T07:20:51.890297+0000","last_clean":"2024-03-06T07:20:51.890297+0000","last_became_active":"2024-03-06T07:20:51.887725+0000","last_became_peered":"2024-03-06T07:20:51.887725+0000","last_unstale":"2024-03-06T07:20:51.890297+0000","last_undegraded":"2024-03-06T07:20:51.890297+0000","last_fullsized":"2024-03-06T07:20:51.890297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:06.737114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.877638+0000","last_change":"2024-03-06T07:20:50.890731+0000","last_active":"2024-03-06T07:20:51.877638+0000","last_peered":"2024-03-06T07:20:51.877638+0000","last_clean":"2024-03-06T07:20:51.877638+0000","last_became_active":"2024-03-06T07:20:50.890429+0000","last_became_peered":"2024-03-06T07:20:50.890429+0000","last_unstale":"2024-03-06T07:20:51.877638+0000","last_undegraded":"2024-03-06T07:20:51.877638+0000","last_fullsized":"2024-03-06T07:20:51.877638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891008+0000","last_change":"2024-03-06T07:20:51.885412+0000","last_active":"2024-03-06T07:20:51.891008+0000","last_peered":"2024-03-06T07:20:51.891008+0000","last_clean":"2024-03-06T07:20:51.891008+0000","last_became_active":"2024-03-06T07:20:51.885153+0000","last_became_peered":"2024-03-06T07:20:51.885153+0000","last_unstale":"2024-03-06T07:20:51.891008+0000","last_undegraded":"2024-03-06T07:20:51.891008+0000","last_fullsized":"2024-03-06T07:20:51.891008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.892651+0000","last_change":"2024-03-06T07:20:50.880143+0000","last_active":"2024-03-06T07:20:51.892651+0000","last_peered":"2024-03-06T07:20:51.892651+0000","last_clean":"2024-03-06T07:20:51.892651+0000","last_became_active":"2024-03-06T07:20:50.879859+0000","last_became_peered":"2024-03-06T07:20:50.879859+0000","last_unstale":"2024-03-06T07:20:51.892651+0000","last_undegraded":"2024-03-06T07:20:51.892651+0000","last_fullsized":"2024-03-06T07:20:51.892651+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:08.684501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879810+0000","last_change":"2024-03-06T07:20:51.878936+0000","last_active":"2024-03-06T07:20:51.879810+0000","last_peered":"2024-03-06T07:20:51.879810+0000","last_clean":"2024-03-06T07:20:51.879810+0000","last_became_active":"2024-03-06T07:20:51.878613+0000","last_became_peered":"2024-03-06T07:20:51.878613+0000","last_unstale":"2024-03-06T07:20:51.879810+0000","last_undegraded":"2024-03-06T07:20:51.879810+0000","last_fullsized":"2024-03-06T07:20:51.879810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899162+0000","last_change":"2024-03-06T07:20:50.888110+0000","last_active":"2024-03-06T07:20:51.899162+0000","last_peered":"2024-03-06T07:20:51.899162+0000","last_clean":"2024-03-06T07:20:51.899162+0000","last_became_active":"2024-03-06T07:20:50.887804+0000","last_became_peered":"2024-03-06T07:20:50.887804+0000","last_unstale":"2024-03-06T07:20:51.899162+0000","last_undegraded":"2024-03-06T07:20:51.899162+0000","last_fullsized":"2024-03-06T07:20:51.899162+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896181+0000","last_change":"2024-03-06T07:20:51.891149+0000","last_active":"2024-03-06T07:20:51.896181+0000","last_peered":"2024-03-06T07:20:51.896181+0000","last_clean":"2024-03-06T07:20:51.896181+0000","last_became_active":"2024-03-06T07:20:51.890877+0000","last_became_peered":"2024-03-06T07:20:51.890877+0000","last_unstale":"2024-03-06T07:20:51.896181+0000","last_undegraded":"2024-03-06T07:20:51.896181+0000","last_fullsized":"2024-03-06T07:20:51.896181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:25.672949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894759+0000","last_change":"2024-03-06T07:20:50.888227+0000","last_active":"2024-03-06T07:20:51.894759+0000","last_peered":"2024-03-06T07:20:51.894759+0000","last_clean":"2024-03-06T07:20:51.894759+0000","last_became_active":"2024-03-06T07:20:50.887926+0000","last_became_peered":"2024-03-06T07:20:50.887926+0000","last_unstale":"2024-03-06T07:20:51.894759+0000","last_undegraded":"2024-03-06T07:20:51.894759+0000","last_fullsized":"2024-03-06T07:20:51.894759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:56.255517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891054+0000","last_change":"2024-03-06T07:20:51.888556+0000","last_active":"2024-03-06T07:20:51.891054+0000","last_peered":"2024-03-06T07:20:51.891054+0000","last_clean":"2024-03-06T07:20:51.891054+0000","last_became_active":"2024-03-06T07:20:51.888278+0000","last_became_peered":"2024-03-06T07:20:51.888278+0000","last_unstale":"2024-03-06T07:20:51.891054+0000","last_undegraded":"2024-03-06T07:20:51.891054+0000","last_fullsized":"2024-03-06T07:20:51.891054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.880871+0000","last_change":"2024-03-06T07:20:50.881110+0000","last_active":"2024-03-06T07:20:51.880871+0000","last_peered":"2024-03-06T07:20:51.880871+0000","last_clean":"2024-03-06T07:20:51.880871+0000","last_became_active":"2024-03-06T07:20:50.880752+0000","last_became_peered":"2024-03-06T07:20:50.880752+0000","last_unstale":"2024-03-06T07:20:51.880871+0000","last_undegraded":"2024-03-06T07:20:51.880871+0000","last_fullsized":"2024-03-06T07:20:51.880871+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886530+0000","last_change":"2024-03-06T07:20:51.879914+0000","last_active":"2024-03-06T07:20:51.886530+0000","last_peered":"2024-03-06T07:20:51.886530+0000","last_clean":"2024-03-06T07:20:51.886530+0000","last_became_active":"2024-03-06T07:20:51.879633+0000","last_became_peered":"2024-03-06T07:20:51.879633+0000","last_unstale":"2024-03-06T07:20:51.886530+0000","last_undegraded":"2024-03-06T07:20:51.886530+0000","last_fullsized":"2024-03-06T07:20:51.886530+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:58.399266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.878494+0000","last_change":"2024-03-06T07:20:50.880559+0000","last_active":"2024-03-06T07:20:51.878494+0000","last_peered":"2024-03-06T07:20:51.878494+0000","last_clean":"2024-03-06T07:20:51.878494+0000","last_became_active":"2024-03-06T07:20:50.880160+0000","last_became_peered":"2024-03-06T07:20:50.880160+0000","last_unstale":"2024-03-06T07:20:51.878494+0000","last_undegraded":"2024-03-06T07:20:51.878494+0000","last_fullsized":"2024-03-06T07:20:51.878494+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:56.900930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891957+0000","last_change":"2024-03-06T07:20:51.890440+0000","last_active":"2024-03-06T07:20:51.891957+0000","last_peered":"2024-03-06T07:20:51.891957+0000","last_clean":"2024-03-06T07:20:51.891957+0000","last_became_active":"2024-03-06T07:20:51.890167+0000","last_became_peered":"2024-03-06T07:20:51.890167+0000","last_unstale":"2024-03-06T07:20:51.891957+0000","last_undegraded":"2024-03-06T07:20:51.891957+0000","last_fullsized":"2024-03-06T07:20:51.891957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889125+0000","last_change":"2024-03-06T07:20:50.885352+0000","last_active":"2024-03-06T07:20:51.889125+0000","last_peered":"2024-03-06T07:20:51.889125+0000","last_clean":"2024-03-06T07:20:51.889125+0000","last_became_active":"2024-03-06T07:20:50.884991+0000","last_became_peered":"2024-03-06T07:20:50.884991+0000","last_unstale":"2024-03-06T07:20:51.889125+0000","last_undegraded":"2024-03-06T07:20:51.889125+0000","last_fullsized":"2024-03-06T07:20:51.889125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:03.652296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.892091+0000","last_change":"2024-03-06T07:20:51.889501+0000","last_active":"2024-03-06T07:20:51.892091+0000","last_peered":"2024-03-06T07:20:51.892091+0000","last_clean":"2024-03-06T07:20:51.892091+0000","last_became_active":"2024-03-06T07:20:51.889206+0000","last_became_peered":"2024-03-06T07:20:51.889206+0000","last_unstale":"2024-03-06T07:20:51.892091+0000","last_undegraded":"2024-03-06T07:20:51.892091+0000","last_fullsized":"2024-03-06T07:20:51.892091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891700+0000","last_change":"2024-03-06T07:20:50.883039+0000","last_active":"2024-03-06T07:20:51.891700+0000","last_peered":"2024-03-06T07:20:51.891700+0000","last_clean":"2024-03-06T07:20:51.891700+0000","last_became_active":"2024-03-06T07:20:50.882748+0000","last_became_peered":"2024-03-06T07:20:50.882748+0000","last_unstale":"2024-03-06T07:20:51.891700+0000","last_undegraded":"2024-03-06T07:20:51.891700+0000","last_fullsized":"2024-03-06T07:20:51.891700+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:35.868841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893043+0000","last_change":"2024-03-06T07:20:51.890860+0000","last_active":"2024-03-06T07:20:51.893043+0000","last_peered":"2024-03-06T07:20:51.893043+0000","last_clean":"2024-03-06T07:20:51.893043+0000","last_became_active":"2024-03-06T07:20:51.890576+0000","last_became_peered":"2024-03-06T07:20:51.890576+0000","last_unstale":"2024-03-06T07:20:51.893043+0000","last_undegraded":"2024-03-06T07:20:51.893043+0000","last_fullsized":"2024-03-06T07:20:51.893043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:00.335725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.887822+0000","last_change":"2024-03-06T07:20:50.888701+0000","last_active":"2024-03-06T07:20:51.887822+0000","last_peered":"2024-03-06T07:20:51.887822+0000","last_clean":"2024-03-06T07:20:51.887822+0000","last_became_active":"2024-03-06T07:20:50.888392+0000","last_became_peered":"2024-03-06T07:20:50.888392+0000","last_unstale":"2024-03-06T07:20:51.887822+0000","last_undegraded":"2024-03-06T07:20:51.887822+0000","last_fullsized":"2024-03-06T07:20:51.887822+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.582638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890650+0000","last_change":"2024-03-06T07:20:51.887239+0000","last_active":"2024-03-06T07:20:51.890650+0000","last_peered":"2024-03-06T07:20:51.890650+0000","last_clean":"2024-03-06T07:20:51.890650+0000","last_became_active":"2024-03-06T07:20:51.886940+0000","last_became_peered":"2024-03-06T07:20:51.886940+0000","last_unstale":"2024-03-06T07:20:51.890650+0000","last_undegraded":"2024-03-06T07:20:51.890650+0000","last_fullsized":"2024-03-06T07:20:51.890650+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:27.561295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.897360+0000","last_change":"2024-03-06T07:20:50.886879+0000","last_active":"2024-03-06T07:20:51.897360+0000","last_peered":"2024-03-06T07:20:51.897360+0000","last_clean":"2024-03-06T07:20:51.897360+0000","last_became_active":"2024-03-06T07:20:50.886589+0000","last_became_peered":"2024-03-06T07:20:50.886589+0000","last_unstale":"2024-03-06T07:20:51.897360+0000","last_undegraded":"2024-03-06T07:20:51.897360+0000","last_fullsized":"2024-03-06T07:20:51.897360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:51.245367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893777+0000","last_change":"2024-03-06T07:20:51.892549+0000","last_active":"2024-03-06T07:20:51.893777+0000","last_peered":"2024-03-06T07:20:51.893777+0000","last_clean":"2024-03-06T07:20:51.893777+0000","last_became_active":"2024-03-06T07:20:51.889936+0000","last_became_peered":"2024-03-06T07:20:51.889936+0000","last_unstale":"2024-03-06T07:20:51.893777+0000","last_undegraded":"2024-03-06T07:20:51.893777+0000","last_fullsized":"2024-03-06T07:20:51.893777+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:56.445613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.869561+0000","last_change":"2024-03-06T07:20:50.883071+0000","last_active":"2024-03-06T07:20:51.869561+0000","last_peered":"2024-03-06T07:20:51.869561+0000","last_clean":"2024-03-06T07:20:51.869561+0000","last_became_active":"2024-03-06T07:20:50.882465+0000","last_became_peered":"2024-03-06T07:20:50.882465+0000","last_unstale":"2024-03-06T07:20:51.869561+0000","last_undegraded":"2024-03-06T07:20:51.869561+0000","last_fullsized":"2024-03-06T07:20:51.869561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886693+0000","last_change":"2024-03-06T07:20:51.884694+0000","last_active":"2024-03-06T07:20:51.886693+0000","last_peered":"2024-03-06T07:20:51.886693+0000","last_clean":"2024-03-06T07:20:51.886693+0000","last_became_active":"2024-03-06T07:20:51.883905+0000","last_became_peered":"2024-03-06T07:20:51.883905+0000","last_unstale":"2024-03-06T07:20:51.886693+0000","last_undegraded":"2024-03-06T07:20:51.886693+0000","last_fullsized":"2024-03-06T07:20:51.886693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:54.467344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890631+0000","last_change":"2024-03-06T07:20:50.881489+0000","last_active":"2024-03-06T07:20:51.890631+0000","last_peered":"2024-03-06T07:20:51.890631+0000","last_clean":"2024-03-06T07:20:51.890631+0000","last_became_active":"2024-03-06T07:20:50.881182+0000","last_became_peered":"2024-03-06T07:20:50.881182+0000","last_unstale":"2024-03-06T07:20:51.890631+0000","last_undegraded":"2024-03-06T07:20:51.890631+0000","last_fullsized":"2024-03-06T07:20:51.890631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:22.518381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894864+0000","last_change":"2024-03-06T07:20:51.889352+0000","last_active":"2024-03-06T07:20:51.894864+0000","last_peered":"2024-03-06T07:20:51.894864+0000","last_clean":"2024-03-06T07:20:51.894864+0000","last_became_active":"2024-03-06T07:20:51.888464+0000","last_became_peered":"2024-03-06T07:20:51.888464+0000","last_unstale":"2024-03-06T07:20:51.894864+0000","last_undegraded":"2024-03-06T07:20:51.894864+0000","last_fullsized":"2024-03-06T07:20:51.894864+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:40.919584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.873223+0000","last_change":"2024-03-06T07:20:50.887415+0000","last_active":"2024-03-06T07:20:51.873223+0000","last_peered":"2024-03-06T07:20:51.873223+0000","last_clean":"2024-03-06T07:20:51.873223+0000","last_became_active":"2024-03-06T07:20:50.887137+0000","last_became_peered":"2024-03-06T07:20:50.887137+0000","last_unstale":"2024-03-06T07:20:51.873223+0000","last_undegraded":"2024-03-06T07:20:51.873223+0000","last_fullsized":"2024-03-06T07:20:51.873223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:54.210267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.887614+0000","last_change":"2024-03-06T07:20:51.885672+0000","last_active":"2024-03-06T07:20:51.887614+0000","last_peered":"2024-03-06T07:20:51.887614+0000","last_clean":"2024-03-06T07:20:51.887614+0000","last_became_active":"2024-03-06T07:20:51.885412+0000","last_became_peered":"2024-03-06T07:20:51.885412+0000","last_unstale":"2024-03-06T07:20:51.887614+0000","last_undegraded":"2024-03-06T07:20:51.887614+0000","last_fullsized":"2024-03-06T07:20:51.887614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.873284+0000","last_change":"2024-03-06T07:20:50.889223+0000","last_active":"2024-03-06T07:20:51.873284+0000","last_peered":"2024-03-06T07:20:51.873284+0000","last_clean":"2024-03-06T07:20:51.873284+0000","last_became_active":"2024-03-06T07:20:50.888934+0000","last_became_peered":"2024-03-06T07:20:50.888934+0000","last_unstale":"2024-03-06T07:20:51.873284+0000","last_undegraded":"2024-03-06T07:20:51.873284+0000","last_fullsized":"2024-03-06T07:20:51.873284+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:04.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.884730+0000","last_change":"2024-03-06T07:20:51.880937+0000","last_active":"2024-03-06T07:20:51.884730+0000","last_peered":"2024-03-06T07:20:51.884730+0000","last_clean":"2024-03-06T07:20:51.884730+0000","last_became_active":"2024-03-06T07:20:51.880648+0000","last_became_peered":"2024-03-06T07:20:51.880648+0000","last_unstale":"2024-03-06T07:20:51.884730+0000","last_undegraded":"2024-03-06T07:20:51.884730+0000","last_fullsized":"2024-03-06T07:20:51.884730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:16.890629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886046+0000","last_change":"2024-03-06T07:20:50.886345+0000","last_active":"2024-03-06T07:20:51.886046+0000","last_peered":"2024-03-06T07:20:51.886046+0000","last_clean":"2024-03-06T07:20:51.886046+0000","last_became_active":"2024-03-06T07:20:50.886074+0000","last_became_peered":"2024-03-06T07:20:50.886074+0000","last_unstale":"2024-03-06T07:20:51.886046+0000","last_undegraded":"2024-03-06T07:20:51.886046+0000","last_fullsized":"2024-03-06T07:20:51.886046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:39.914514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893225+0000","last_change":"2024-03-06T07:20:51.889471+0000","last_active":"2024-03-06T07:20:51.893225+0000","last_peered":"2024-03-06T07:20:51.893225+0000","last_clean":"2024-03-06T07:20:51.893225+0000","last_became_active":"2024-03-06T07:20:51.889218+0000","last_became_peered":"2024-03-06T07:20:51.889218+0000","last_unstale":"2024-03-06T07:20:51.893225+0000","last_undegraded":"2024-03-06T07:20:51.893225+0000","last_fullsized":"2024-03-06T07:20:51.893225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.877169+0000","last_change":"2024-03-06T07:20:50.881740+0000","last_active":"2024-03-06T07:20:51.877169+0000","last_peered":"2024-03-06T07:20:51.877169+0000","last_clean":"2024-03-06T07:20:51.877169+0000","last_became_active":"2024-03-06T07:20:50.881306+0000","last_became_peered":"2024-03-06T07:20:50.881306+0000","last_unstale":"2024-03-06T07:20:51.877169+0000","last_undegraded":"2024-03-06T07:20:51.877169+0000","last_fullsized":"2024-03-06T07:20:51.877169+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:35.733892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889206+0000","last_change":"2024-03-06T07:20:51.886967+0000","last_active":"2024-03-06T07:20:51.889206+0000","last_peered":"2024-03-06T07:20:51.889206+0000","last_clean":"2024-03-06T07:20:51.889206+0000","last_became_active":"2024-03-06T07:20:51.886521+0000","last_became_peered":"2024-03-06T07:20:51.886521+0000","last_unstale":"2024-03-06T07:20:51.889206+0000","last_undegraded":"2024-03-06T07:20:51.889206+0000","last_fullsized":"2024-03-06T07:20:51.889206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:37.112027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.884762+0000","last_change":"2024-03-06T07:20:50.885378+0000","last_active":"2024-03-06T07:20:51.884762+0000","last_peered":"2024-03-06T07:20:51.884762+0000","last_clean":"2024-03-06T07:20:51.884762+0000","last_became_active":"2024-03-06T07:20:50.884859+0000","last_became_peered":"2024-03-06T07:20:50.884859+0000","last_unstale":"2024-03-06T07:20:51.884762+0000","last_undegraded":"2024-03-06T07:20:51.884762+0000","last_fullsized":"2024-03-06T07:20:51.884762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:28.686737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899861+0000","last_change":"2024-03-06T07:20:51.899213+0000","last_active":"2024-03-06T07:20:51.899861+0000","last_peered":"2024-03-06T07:20:51.899861+0000","last_clean":"2024-03-06T07:20:51.899861+0000","last_became_active":"2024-03-06T07:20:51.898882+0000","last_became_peered":"2024-03-06T07:20:51.898882+0000","last_unstale":"2024-03-06T07:20:51.899861+0000","last_undegraded":"2024-03-06T07:20:51.899861+0000","last_fullsized":"2024-03-06T07:20:51.899861+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:25.672949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.881839+0000","last_change":"2024-03-06T07:20:50.888878+0000","last_active":"2024-03-06T07:20:51.881839+0000","last_peered":"2024-03-06T07:20:51.881839+0000","last_clean":"2024-03-06T07:20:51.881839+0000","last_became_active":"2024-03-06T07:20:50.888604+0000","last_became_peered":"2024-03-06T07:20:50.888604+0000","last_unstale":"2024-03-06T07:20:51.881839+0000","last_undegraded":"2024-03-06T07:20:51.881839+0000","last_fullsized":"2024-03-06T07:20:51.881839+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:33.413428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894115+0000","last_change":"2024-03-06T07:20:51.893070+0000","last_active":"2024-03-06T07:20:51.894115+0000","last_peered":"2024-03-06T07:20:51.894115+0000","last_clean":"2024-03-06T07:20:51.894115+0000","last_became_active":"2024-03-06T07:20:51.892785+0000","last_became_peered":"2024-03-06T07:20:51.892785+0000","last_unstale":"2024-03-06T07:20:51.894115+0000","last_undegraded":"2024-03-06T07:20:51.894115+0000","last_fullsized":"2024-03-06T07:20:51.894115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:58.622652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.875874+0000","last_change":"2024-03-06T07:20:50.892942+0000","last_active":"2024-03-06T07:20:51.875874+0000","last_peered":"2024-03-06T07:20:51.875874+0000","last_clean":"2024-03-06T07:20:51.875874+0000","last_became_active":"2024-03-06T07:20:50.892638+0000","last_became_peered":"2024-03-06T07:20:50.892638+0000","last_unstale":"2024-03-06T07:20:51.875874+0000","last_undegraded":"2024-03-06T07:20:51.875874+0000","last_fullsized":"2024-03-06T07:20:51.875874+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:40.434821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898581+0000","last_change":"2024-03-06T07:20:51.895826+0000","last_active":"2024-03-06T07:20:51.898581+0000","last_peered":"2024-03-06T07:20:51.898581+0000","last_clean":"2024-03-06T07:20:51.898581+0000","last_became_active":"2024-03-06T07:20:51.895483+0000","last_became_peered":"2024-03-06T07:20:51.895483+0000","last_unstale":"2024-03-06T07:20:51.898581+0000","last_undegraded":"2024-03-06T07:20:51.898581+0000","last_fullsized":"2024-03-06T07:20:51.898581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879788+0000","last_change":"2024-03-06T07:20:50.879754+0000","last_active":"2024-03-06T07:20:51.879788+0000","last_peered":"2024-03-06T07:20:51.879788+0000","last_clean":"2024-03-06T07:20:51.879788+0000","last_became_active":"2024-03-06T07:20:50.879488+0000","last_became_peered":"2024-03-06T07:20:50.879488+0000","last_unstale":"2024-03-06T07:20:51.879788+0000","last_undegraded":"2024-03-06T07:20:51.879788+0000","last_fullsized":"2024-03-06T07:20:51.879788+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.603045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879838+0000","last_change":"2024-03-06T07:20:51.879085+0000","last_active":"2024-03-06T07:20:51.879838+0000","last_peered":"2024-03-06T07:20:51.879838+0000","last_clean":"2024-03-06T07:20:51.879838+0000","last_became_active":"2024-03-06T07:20:51.878776+0000","last_became_peered":"2024-03-06T07:20:51.878776+0000","last_unstale":"2024-03-06T07:20:51.879838+0000","last_undegraded":"2024-03-06T07:20:51.879838+0000","last_fullsized":"2024-03-06T07:20:51.879838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:58.622652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.875749+0000","last_change":"2024-03-06T07:20:50.887928+0000","last_active":"2024-03-06T07:20:51.875749+0000","last_peered":"2024-03-06T07:20:51.875749+0000","last_clean":"2024-03-06T07:20:51.875749+0000","last_became_active":"2024-03-06T07:20:50.887638+0000","last_became_peered":"2024-03-06T07:20:50.887638+0000","last_unstale":"2024-03-06T07:20:51.875749+0000","last_undegraded":"2024-03-06T07:20:51.875749+0000","last_fullsized":"2024-03-06T07:20:51.875749+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:52.309668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.884344+0000","last_change":"2024-03-06T07:20:51.883253+0000","last_active":"2024-03-06T07:20:51.884344+0000","last_peered":"2024-03-06T07:20:51.884344+0000","last_clean":"2024-03-06T07:20:51.884344+0000","last_became_active":"2024-03-06T07:20:51.882993+0000","last_became_peered":"2024-03-06T07:20:51.882993+0000","last_unstale":"2024-03-06T07:20:51.884344+0000","last_undegraded":"2024-03-06T07:20:51.884344+0000","last_fullsized":"2024-03-06T07:20:51.884344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:09.457018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.869503+0000","last_change":"2024-03-06T07:20:50.879149+0000","last_active":"2024-03-06T07:20:51.869503+0000","last_peered":"2024-03-06T07:20:51.869503+0000","last_clean":"2024-03-06T07:20:51.869503+0000","last_became_active":"2024-03-06T07:20:50.878867+0000","last_became_peered":"2024-03-06T07:20:50.878867+0000","last_unstale":"2024-03-06T07:20:51.869503+0000","last_undegraded":"2024-03-06T07:20:51.869503+0000","last_fullsized":"2024-03-06T07:20:51.869503+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:57.394196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894575+0000","last_change":"2024-03-06T07:20:51.891707+0000","last_active":"2024-03-06T07:20:51.894575+0000","last_peered":"2024-03-06T07:20:51.894575+0000","last_clean":"2024-03-06T07:20:51.894575+0000","last_became_active":"2024-03-06T07:20:51.891427+0000","last_became_peered":"2024-03-06T07:20:51.891427+0000","last_unstale":"2024-03-06T07:20:51.894575+0000","last_undegraded":"2024-03-06T07:20:51.894575+0000","last_fullsized":"2024-03-06T07:20:51.894575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:21.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.873811+0000","last_change":"2024-03-06T07:20:50.885372+0000","last_active":"2024-03-06T07:20:51.873811+0000","last_peered":"2024-03-06T07:20:51.873811+0000","last_clean":"2024-03-06T07:20:51.873811+0000","last_became_active":"2024-03-06T07:20:50.885093+0000","last_became_peered":"2024-03-06T07:20:50.885093+0000","last_unstale":"2024-03-06T07:20:51.873811+0000","last_undegraded":"2024-03-06T07:20:51.873811+0000","last_fullsized":"2024-03-06T07:20:51.873811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:12.807980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":270,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.881344+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:20:51.881344+0000","last_peered":"2024-03-06T07:20:51.881344+0000","last_clean":"2024-03-06T07:20:51.881344+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:20:51.881344+0000","last_undegraded":"2024-03-06T07:20:51.881344+0000","last_fullsized":"2024-03-06T07:20:51.881344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":1010269,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":412712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":412712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32076,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339764,"statfs":{"total":96636764160,"available":96603918336,"internally_reserved":0,"allocated":909312,"data_stored":420655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":897024,"data_stored":420461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":905216,"data_stored":420621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":880640,"data_stored":412712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":1010247,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:55.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:20:55.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.286+0000 7fdeb56e1640 1 Processor -- start 2024-03-06T07:20:55.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.286+0000 7fdeb56e1640 1 -- start start 2024-03-06T07:20:55.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.286+0000 7fdeb56e1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdeb014f360 0x7fdeb0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.286+0000 7fdeb56e1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb00c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.286+0000 7fdeb56e1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb00c73d0 con 0x7fdeb00c5880 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.286+0000 7fdeb56e1640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb00c8dd0 con 0x7fdeb014f360 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.286+0000 7fdeb56e1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00be2e0 con 0x7fdeb00c88c0 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeae7fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb00c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeaeffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdeb014f360 0x7fdeb0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeaf7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fdeb00c5880 0x7fdeb0158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38998/0 (socket says 172.21.15.94:38998) 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeae7fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb00c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53516/0 (socket says 172.21.15.94:53516) 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeaf7fe640 1 -- 172.21.15.94:0/1675423135 learned_addr learned my addr 172.21.15.94:0/1675423135 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeadffb640 1 -- 172.21.15.94:0/1675423135 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829699515 0 0) 0x7fdeb00c73d0 con 0x7fdeb00c5880 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeadffb640 1 -- 172.21.15.94:0/1675423135 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde98003520 con 0x7fdeb00c5880 2024-03-06T07:20:55.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeae7fc640 1 -- 172.21.15.94:0/1675423135 >> v2:172.21.15.145:3300/0 conn(0x7fdeb014f360 msgr2=0x7fdeb0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeae7fc640 1 --2- 172.21.15.94:0/1675423135 >> v2:172.21.15.145:3300/0 conn(0x7fdeb014f360 0x7fdeb0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeae7fc640 1 -- 172.21.15.94:0/1675423135 >> v1:172.21.15.145:6789/0 conn(0x7fdeb00c5880 legacy=0x7fdeb0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeae7fc640 1 -- 172.21.15.94:0/1675423135 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb00c7670 con 0x7fdeb00c88c0 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.287+0000 7fdeaeffd640 1 --2- 172.21.15.94:0/1675423135 >> v2:172.21.15.145:3300/0 conn(0x7fdeb014f360 0x7fdeb0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.288+0000 7fdeae7fc640 1 --2- 172.21.15.94:0/1675423135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb00c5340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdea40097a0 tx=0x7fdea402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=621a85556de6b671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.288+0000 7fdeadffb640 1 -- 172.21.15.94:0/1675423135 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea40315a0 con 0x7fdeb00c88c0 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.288+0000 7fdeadffb640 1 -- 172.21.15.94:0/1675423135 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea4037070 con 0x7fdeb00c88c0 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.288+0000 7fdeadffb640 1 -- 172.21.15.94:0/1675423135 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4031940 con 0x7fdeb00c88c0 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.288+0000 7fdeb56e1640 1 -- 172.21.15.94:0/1675423135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 msgr2=0x7fdeb00c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.288+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/1675423135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb00c5340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdea40097a0 tx=0x7fdea402b840 comp rx=0 tx=0).stop 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.289+0000 7fdeb56e1640 1 -- 172.21.15.94:0/1675423135 shutdown_connections 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.289+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/1675423135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb00c5340 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.289+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/1675423135 >> v2:172.21.15.145:3300/0 conn(0x7fdeb014f360 0x7fdeb0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.289+0000 7fdeb56e1640 1 -- 172.21.15.94:0/1675423135 >> 172.21.15.94:0/1675423135 conn(0x7fdeb00ac570 msgr2=0x7fdeb00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.289+0000 7fdeb56e1640 1 -- 172.21.15.94:0/1675423135 shutdown_connections 2024-03-06T07:20:55.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.289+0000 7fdeb56e1640 1 -- 172.21.15.94:0/1675423135 wait complete. 2024-03-06T07:20:55.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 Processor -- start 2024-03-06T07:20:55.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 -- start start 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb00c5880 0x7fdeb01ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb014bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdeb014f360 0x7fdeb014c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0159270 con 0x7fdeb014f360 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00c7c30 con 0x7fdeb00c5880 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.290+0000 7fdeb56e1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb01593f0 con 0x7fdeb00c88c0 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdeb014f360 0x7fdeb014c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaeffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb00c5880 0x7fdeb01ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdeb014f360 0x7fdeb014c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52196/0 (socket says 172.21.15.94:52196) 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaf7fe640 1 -- 172.21.15.94:0/479913226 learned_addr learned my addr 172.21.15.94:0/479913226 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeae7fc640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb014bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaeffd640 1 -- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 msgr2=0x7fdeb014bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaeffd640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb014bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaeffd640 1 -- 172.21.15.94:0/479913226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdeb014f360 msgr2=0x7fdeb014c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaeffd640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdeb014f360 0x7fdeb014c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaeffd640 1 -- 172.21.15.94:0/479913226 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb01eedc0 con 0x7fdeb00c5880 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaf7fe640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdeb014f360 0x7fdeb014c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeaeffd640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb00c5880 0x7fdeb01ee4f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdea000e870 tx=0x7fdea000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fde977fe640 1 -- 172.21.15.94:0/479913226 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0017070 con 0x7fdeb00c5880 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fde977fe640 1 -- 172.21.15.94:0/479913226 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea000c940 con 0x7fdeb00c5880 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fde977fe640 1 -- 172.21.15.94:0/479913226 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000cd10 con 0x7fdeb00c5880 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb00addb0 con 0x7fdeb00c5880 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.291+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb00ae370 con 0x7fdeb00c5880 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.292+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb004f330 con 0x7fdeb00c5880 2024-03-06T07:20:55.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.293+0000 7fde977fe640 1 -- 172.21.15.94:0/479913226 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdea001a070 con 0x7fdeb00c5880 2024-03-06T07:20:55.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.293+0000 7fde977fe640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde9c06b500 0x7fde9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.293+0000 7fde977fe640 1 -- 172.21.15.94:0/479913226 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdea008a3f0 con 0x7fdeb00c5880 2024-03-06T07:20:55.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.293+0000 7fdeae7fc640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde9c06b500 0x7fde9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.294+0000 7fdeae7fc640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde9c06b500 0x7fde9c06d9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdea4036000 tx=0x7fdea403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.298+0000 7fde977fe640 1 -- 172.21.15.94:0/479913226 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdea0058b20 con 0x7fdeb00c5880 2024-03-06T07:20:55.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.466+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdeb0158770 con 0x7fde9c06b500 2024-03-06T07:20:55.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.472+0000 7fde977fe640 1 -- 172.21.15.94:0/479913226 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252075 (secure 0 0 0) 0x7fdeb0158770 con 0x7fde9c06b500 2024-03-06T07:20:55.477 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:55.478 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde9c06b500 msgr2=0x7fde9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde9c06b500 0x7fde9c06d9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdea4036000 tx=0x7fdea403d040 comp rx=0 tx=0).stop 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb00c5880 msgr2=0x7fdeb01ee4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb00c5880 0x7fdeb01ee4f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdea000e870 tx=0x7fdea000ed30 comp rx=0 tx=0).stop 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 shutdown_connections 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde9c06b500 0x7fde9c06d9c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdeb014f360 0x7fdeb014c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00c88c0 0x7fdeb014bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 --2- 172.21.15.94:0/479913226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb00c5880 0x7fdeb01ee4f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 >> 172.21.15.94:0/479913226 conn(0x7fdeb00ac570 msgr2=0x7fdeb00ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 shutdown_connections 2024-03-06T07:20:55.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.475+0000 7fdeb56e1640 1 -- 172.21.15.94:0/479913226 wait complete. 2024-03-06T07:20:55.488 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-03-06T07:20:54.669911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244100,"kb_used_data":7104,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730620,"statfs":{"total":773094113280,"available":772844154880,"internally_reserved":0,"allocated":7274496,"data_stored":4520389,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.810536"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898885+0000","last_change":"2024-03-06T07:20:51.896309+0000","last_active":"2024-03-06T07:20:51.898885+0000","last_peered":"2024-03-06T07:20:51.898885+0000","last_clean":"2024-03-06T07:20:51.898885+0000","last_became_active":"2024-03-06T07:20:51.896009+0000","last_became_peered":"2024-03-06T07:20:51.896009+0000","last_unstale":"2024-03-06T07:20:51.898885+0000","last_undegraded":"2024-03-06T07:20:51.898885+0000","last_fullsized":"2024-03-06T07:20:51.898885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896823+0000","last_change":"2024-03-06T07:20:51.895412+0000","last_active":"2024-03-06T07:20:51.896823+0000","last_peered":"2024-03-06T07:20:51.896823+0000","last_clean":"2024-03-06T07:20:51.896823+0000","last_became_active":"2024-03-06T07:20:51.895169+0000","last_became_peered":"2024-03-06T07:20:51.895169+0000","last_unstale":"2024-03-06T07:20:51.896823+0000","last_undegraded":"2024-03-06T07:20:51.896823+0000","last_fullsized":"2024-03-06T07:20:51.896823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:20.484391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879871+0000","last_change":"2024-03-06T07:20:51.878718+0000","last_active":"2024-03-06T07:20:51.879871+0000","last_peered":"2024-03-06T07:20:51.879871+0000","last_clean":"2024-03-06T07:20:51.879871+0000","last_became_active":"2024-03-06T07:20:51.878467+0000","last_became_peered":"2024-03-06T07:20:51.878467+0000","last_unstale":"2024-03-06T07:20:51.879871+0000","last_undegraded":"2024-03-06T07:20:51.879871+0000","last_fullsized":"2024-03-06T07:20:51.879871+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896201+0000","last_change":"2024-03-06T07:20:51.892550+0000","last_active":"2024-03-06T07:20:51.896201+0000","last_peered":"2024-03-06T07:20:51.896201+0000","last_clean":"2024-03-06T07:20:51.896201+0000","last_became_active":"2024-03-06T07:20:51.892264+0000","last_became_peered":"2024-03-06T07:20:51.892264+0000","last_unstale":"2024-03-06T07:20:51.896201+0000","last_undegraded":"2024-03-06T07:20:51.896201+0000","last_fullsized":"2024-03-06T07:20:51.896201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:29.691807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896287+0000","last_change":"2024-03-06T07:20:51.893305+0000","last_active":"2024-03-06T07:20:51.896287+0000","last_peered":"2024-03-06T07:20:51.896287+0000","last_clean":"2024-03-06T07:20:51.896287+0000","last_became_active":"2024-03-06T07:20:51.893041+0000","last_became_peered":"2024-03-06T07:20:51.893041+0000","last_unstale":"2024-03-06T07:20:51.896287+0000","last_undegraded":"2024-03-06T07:20:51.896287+0000","last_fullsized":"2024-03-06T07:20:51.896287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:04.441365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894594+0000","last_change":"2024-03-06T07:20:51.893812+0000","last_active":"2024-03-06T07:20:51.894594+0000","last_peered":"2024-03-06T07:20:51.894594+0000","last_clean":"2024-03-06T07:20:51.894594+0000","last_became_active":"2024-03-06T07:20:51.891649+0000","last_became_peered":"2024-03-06T07:20:51.891649+0000","last_unstale":"2024-03-06T07:20:51.894594+0000","last_undegraded":"2024-03-06T07:20:51.894594+0000","last_fullsized":"2024-03-06T07:20:51.894594+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:38.106753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896991+0000","last_change":"2024-03-06T07:20:51.894141+0000","last_active":"2024-03-06T07:20:51.896991+0000","last_peered":"2024-03-06T07:20:51.896991+0000","last_clean":"2024-03-06T07:20:51.896991+0000","last_became_active":"2024-03-06T07:20:51.893793+0000","last_became_peered":"2024-03-06T07:20:51.893793+0000","last_unstale":"2024-03-06T07:20:51.896991+0000","last_undegraded":"2024-03-06T07:20:51.896991+0000","last_fullsized":"2024-03-06T07:20:51.896991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:57.061120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896737+0000","last_change":"2024-03-06T07:20:51.894037+0000","last_active":"2024-03-06T07:20:51.896737+0000","last_peered":"2024-03-06T07:20:51.896737+0000","last_clean":"2024-03-06T07:20:51.896737+0000","last_became_active":"2024-03-06T07:20:51.893773+0000","last_became_peered":"2024-03-06T07:20:51.893773+0000","last_unstale":"2024-03-06T07:20:51.896737+0000","last_undegraded":"2024-03-06T07:20:51.896737+0000","last_fullsized":"2024-03-06T07:20:51.896737+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:47.454570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890036+0000","last_change":"2024-03-06T07:20:51.886223+0000","last_active":"2024-03-06T07:20:51.890036+0000","last_peered":"2024-03-06T07:20:51.890036+0000","last_clean":"2024-03-06T07:20:51.890036+0000","last_became_active":"2024-03-06T07:20:51.885985+0000","last_became_peered":"2024-03-06T07:20:51.885985+0000","last_unstale":"2024-03-06T07:20:51.890036+0000","last_undegraded":"2024-03-06T07:20:51.890036+0000","last_fullsized":"2024-03-06T07:20:51.890036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:44.358219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893550+0000","last_change":"2024-03-06T07:20:51.890465+0000","last_active":"2024-03-06T07:20:51.893550+0000","last_peered":"2024-03-06T07:20:51.893550+0000","last_clean":"2024-03-06T07:20:51.893550+0000","last_became_active":"2024-03-06T07:20:51.890183+0000","last_became_peered":"2024-03-06T07:20:51.890183+0000","last_unstale":"2024-03-06T07:20:51.893550+0000","last_undegraded":"2024-03-06T07:20:51.893550+0000","last_fullsized":"2024-03-06T07:20:51.893550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:36.873911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898264+0000","last_change":"2024-03-06T07:20:51.897836+0000","last_active":"2024-03-06T07:20:51.898264+0000","last_peered":"2024-03-06T07:20:51.898264+0000","last_clean":"2024-03-06T07:20:51.898264+0000","last_became_active":"2024-03-06T07:20:51.897551+0000","last_became_peered":"2024-03-06T07:20:51.897551+0000","last_unstale":"2024-03-06T07:20:51.898264+0000","last_undegraded":"2024-03-06T07:20:51.898264+0000","last_fullsized":"2024-03-06T07:20:51.898264+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:06.737114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889290+0000","last_change":"2024-03-06T07:20:51.886475+0000","last_active":"2024-03-06T07:20:51.889290+0000","last_peered":"2024-03-06T07:20:51.889290+0000","last_clean":"2024-03-06T07:20:51.889290+0000","last_became_active":"2024-03-06T07:20:51.886218+0000","last_became_peered":"2024-03-06T07:20:51.886218+0000","last_unstale":"2024-03-06T07:20:51.889290+0000","last_undegraded":"2024-03-06T07:20:51.889290+0000","last_fullsized":"2024-03-06T07:20:51.889290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:35.092363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890597+0000","last_change":"2024-03-06T07:20:51.887999+0000","last_active":"2024-03-06T07:20:51.890597+0000","last_peered":"2024-03-06T07:20:51.890597+0000","last_clean":"2024-03-06T07:20:51.890597+0000","last_became_active":"2024-03-06T07:20:51.887749+0000","last_became_peered":"2024-03-06T07:20:51.887749+0000","last_unstale":"2024-03-06T07:20:51.890597+0000","last_undegraded":"2024-03-06T07:20:51.890597+0000","last_fullsized":"2024-03-06T07:20:51.890597+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:33.108778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893833+0000","last_change":"2024-03-06T07:20:51.892140+0000","last_active":"2024-03-06T07:20:51.893833+0000","last_peered":"2024-03-06T07:20:51.893833+0000","last_clean":"2024-03-06T07:20:51.893833+0000","last_became_active":"2024-03-06T07:20:51.891195+0000","last_became_peered":"2024-03-06T07:20:51.891195+0000","last_unstale":"2024-03-06T07:20:51.893833+0000","last_undegraded":"2024-03-06T07:20:51.893833+0000","last_fullsized":"2024-03-06T07:20:51.893833+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896553+0000","last_change":"2024-03-06T07:20:51.894735+0000","last_active":"2024-03-06T07:20:51.896553+0000","last_peered":"2024-03-06T07:20:51.896553+0000","last_clean":"2024-03-06T07:20:51.896553+0000","last_became_active":"2024-03-06T07:20:51.894479+0000","last_became_peered":"2024-03-06T07:20:51.894479+0000","last_unstale":"2024-03-06T07:20:51.896553+0000","last_undegraded":"2024-03-06T07:20:51.896553+0000","last_fullsized":"2024-03-06T07:20:51.896553+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:57.260587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896658+0000","last_change":"2024-03-06T07:20:51.892450+0000","last_active":"2024-03-06T07:20:51.896658+0000","last_peered":"2024-03-06T07:20:51.896658+0000","last_clean":"2024-03-06T07:20:51.896658+0000","last_became_active":"2024-03-06T07:20:51.892184+0000","last_became_peered":"2024-03-06T07:20:51.892184+0000","last_unstale":"2024-03-06T07:20:51.896658+0000","last_undegraded":"2024-03-06T07:20:51.896658+0000","last_fullsized":"2024-03-06T07:20:51.896658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893014+0000","last_change":"2024-03-06T07:20:51.891492+0000","last_active":"2024-03-06T07:20:51.893014+0000","last_peered":"2024-03-06T07:20:51.893014+0000","last_clean":"2024-03-06T07:20:51.893014+0000","last_became_active":"2024-03-06T07:20:51.891212+0000","last_became_peered":"2024-03-06T07:20:51.891212+0000","last_unstale":"2024-03-06T07:20:51.893014+0000","last_undegraded":"2024-03-06T07:20:51.893014+0000","last_fullsized":"2024-03-06T07:20:51.893014+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:05.321792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879250+0000","last_change":"2024-03-06T07:20:51.876410+0000","last_active":"2024-03-06T07:20:51.879250+0000","last_peered":"2024-03-06T07:20:51.879250+0000","last_clean":"2024-03-06T07:20:51.879250+0000","last_became_active":"2024-03-06T07:20:51.876155+0000","last_became_peered":"2024-03-06T07:20:51.876155+0000","last_unstale":"2024-03-06T07:20:51.879250+0000","last_undegraded":"2024-03-06T07:20:51.879250+0000","last_fullsized":"2024-03-06T07:20:51.879250+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.885514+0000","last_change":"2024-03-06T07:20:51.880019+0000","last_active":"2024-03-06T07:20:51.885514+0000","last_peered":"2024-03-06T07:20:51.885514+0000","last_clean":"2024-03-06T07:20:51.885514+0000","last_became_active":"2024-03-06T07:20:51.879756+0000","last_became_peered":"2024-03-06T07:20:51.879756+0000","last_unstale":"2024-03-06T07:20:51.885514+0000","last_undegraded":"2024-03-06T07:20:51.885514+0000","last_fullsized":"2024-03-06T07:20:51.885514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.892658+0000","last_change":"2024-03-06T07:20:51.890815+0000","last_active":"2024-03-06T07:20:51.892658+0000","last_peered":"2024-03-06T07:20:51.892658+0000","last_clean":"2024-03-06T07:20:51.892658+0000","last_became_active":"2024-03-06T07:20:51.890558+0000","last_became_peered":"2024-03-06T07:20:51.890558+0000","last_unstale":"2024-03-06T07:20:51.892658+0000","last_undegraded":"2024-03-06T07:20:51.892658+0000","last_fullsized":"2024-03-06T07:20:51.892658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:55.266280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890439+0000","last_change":"2024-03-06T07:20:51.884508+0000","last_active":"2024-03-06T07:20:51.890439+0000","last_peered":"2024-03-06T07:20:51.890439+0000","last_clean":"2024-03-06T07:20:51.890439+0000","last_became_active":"2024-03-06T07:20:51.884237+0000","last_became_peered":"2024-03-06T07:20:51.884237+0000","last_unstale":"2024-03-06T07:20:51.890439+0000","last_undegraded":"2024-03-06T07:20:51.890439+0000","last_fullsized":"2024-03-06T07:20:51.890439+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:36.241725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890945+0000","last_change":"2024-03-06T07:20:51.890118+0000","last_active":"2024-03-06T07:20:51.890945+0000","last_peered":"2024-03-06T07:20:51.890945+0000","last_clean":"2024-03-06T07:20:51.890945+0000","last_became_active":"2024-03-06T07:20:51.888426+0000","last_became_peered":"2024-03-06T07:20:51.888426+0000","last_unstale":"2024-03-06T07:20:51.890945+0000","last_undegraded":"2024-03-06T07:20:51.890945+0000","last_fullsized":"2024-03-06T07:20:51.890945+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:07.818700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890241+0000","last_change":"2024-03-06T07:20:51.885627+0000","last_active":"2024-03-06T07:20:51.890241+0000","last_peered":"2024-03-06T07:20:51.890241+0000","last_clean":"2024-03-06T07:20:51.890241+0000","last_became_active":"2024-03-06T07:20:51.885338+0000","last_became_peered":"2024-03-06T07:20:51.885338+0000","last_unstale":"2024-03-06T07:20:51.890241+0000","last_undegraded":"2024-03-06T07:20:51.890241+0000","last_fullsized":"2024-03-06T07:20:51.890241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:35.720895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886307+0000","last_change":"2024-03-06T07:20:51.884448+0000","last_active":"2024-03-06T07:20:51.886307+0000","last_peered":"2024-03-06T07:20:51.886307+0000","last_clean":"2024-03-06T07:20:51.886307+0000","last_became_active":"2024-03-06T07:20:51.884176+0000","last_became_peered":"2024-03-06T07:20:51.884176+0000","last_unstale":"2024-03-06T07:20:51.886307+0000","last_undegraded":"2024-03-06T07:20:51.886307+0000","last_fullsized":"2024-03-06T07:20:51.886307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:09.591477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894742+0000","last_change":"2024-03-06T07:20:51.886062+0000","last_active":"2024-03-06T07:20:51.894742+0000","last_peered":"2024-03-06T07:20:51.894742+0000","last_clean":"2024-03-06T07:20:51.894742+0000","last_became_active":"2024-03-06T07:20:51.885799+0000","last_became_peered":"2024-03-06T07:20:51.885799+0000","last_unstale":"2024-03-06T07:20:51.894742+0000","last_undegraded":"2024-03-06T07:20:51.894742+0000","last_fullsized":"2024-03-06T07:20:51.894742+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:11.878187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890256+0000","last_change":"2024-03-06T07:20:51.883614+0000","last_active":"2024-03-06T07:20:51.890256+0000","last_peered":"2024-03-06T07:20:51.890256+0000","last_clean":"2024-03-06T07:20:51.890256+0000","last_became_active":"2024-03-06T07:20:51.883266+0000","last_became_peered":"2024-03-06T07:20:51.883266+0000","last_unstale":"2024-03-06T07:20:51.890256+0000","last_undegraded":"2024-03-06T07:20:51.890256+0000","last_fullsized":"2024-03-06T07:20:51.890256+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.418498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.897131+0000","last_change":"2024-03-06T07:20:51.893301+0000","last_active":"2024-03-06T07:20:51.897131+0000","last_peered":"2024-03-06T07:20:51.897131+0000","last_clean":"2024-03-06T07:20:51.897131+0000","last_became_active":"2024-03-06T07:20:51.892946+0000","last_became_peered":"2024-03-06T07:20:51.892946+0000","last_unstale":"2024-03-06T07:20:51.897131+0000","last_undegraded":"2024-03-06T07:20:51.897131+0000","last_fullsized":"2024-03-06T07:20:51.897131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893448+0000","last_change":"2024-03-06T07:20:51.891689+0000","last_active":"2024-03-06T07:20:51.893448+0000","last_peered":"2024-03-06T07:20:51.893448+0000","last_clean":"2024-03-06T07:20:51.893448+0000","last_became_active":"2024-03-06T07:20:51.891341+0000","last_became_peered":"2024-03-06T07:20:51.891341+0000","last_unstale":"2024-03-06T07:20:51.893448+0000","last_undegraded":"2024-03-06T07:20:51.893448+0000","last_fullsized":"2024-03-06T07:20:51.893448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:32.214282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.895035+0000","last_change":"2024-03-06T07:20:51.887454+0000","last_active":"2024-03-06T07:20:51.895035+0000","last_peered":"2024-03-06T07:20:51.895035+0000","last_clean":"2024-03-06T07:20:51.895035+0000","last_became_active":"2024-03-06T07:20:51.887188+0000","last_became_peered":"2024-03-06T07:20:51.887188+0000","last_unstale":"2024-03-06T07:20:51.895035+0000","last_undegraded":"2024-03-06T07:20:51.895035+0000","last_fullsized":"2024-03-06T07:20:51.895035+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:35.622801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.900513+0000","last_change":"2024-03-06T07:20:51.899932+0000","last_active":"2024-03-06T07:20:51.900513+0000","last_peered":"2024-03-06T07:20:51.900513+0000","last_clean":"2024-03-06T07:20:51.900513+0000","last_became_active":"2024-03-06T07:20:51.899569+0000","last_became_peered":"2024-03-06T07:20:51.899569+0000","last_unstale":"2024-03-06T07:20:51.900513+0000","last_undegraded":"2024-03-06T07:20:51.900513+0000","last_fullsized":"2024-03-06T07:20:51.900513+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889559+0000","last_change":"2024-03-06T07:20:51.888537+0000","last_active":"2024-03-06T07:20:51.889559+0000","last_peered":"2024-03-06T07:20:51.889559+0000","last_clean":"2024-03-06T07:20:51.889559+0000","last_became_active":"2024-03-06T07:20:51.888281+0000","last_became_peered":"2024-03-06T07:20:51.888281+0000","last_unstale":"2024-03-06T07:20:51.889559+0000","last_undegraded":"2024-03-06T07:20:51.889559+0000","last_fullsized":"2024-03-06T07:20:51.889559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:57.061120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898696+0000","last_change":"2024-03-06T07:20:51.896158+0000","last_active":"2024-03-06T07:20:51.898696+0000","last_peered":"2024-03-06T07:20:51.898696+0000","last_clean":"2024-03-06T07:20:51.898696+0000","last_became_active":"2024-03-06T07:20:51.895889+0000","last_became_peered":"2024-03-06T07:20:51.895889+0000","last_unstale":"2024-03-06T07:20:51.898696+0000","last_undegraded":"2024-03-06T07:20:51.898696+0000","last_fullsized":"2024-03-06T07:20:51.898696+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879095+0000","last_change":"2024-03-06T07:20:51.876393+0000","last_active":"2024-03-06T07:20:51.879095+0000","last_peered":"2024-03-06T07:20:51.879095+0000","last_clean":"2024-03-06T07:20:51.879095+0000","last_became_active":"2024-03-06T07:20:51.876059+0000","last_became_peered":"2024-03-06T07:20:51.876059+0000","last_unstale":"2024-03-06T07:20:51.879095+0000","last_undegraded":"2024-03-06T07:20:51.879095+0000","last_fullsized":"2024-03-06T07:20:51.879095+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":16,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.897856+0000","last_change":"2024-03-06T07:20:50.879586+0000","last_active":"2024-03-06T07:20:51.897856+0000","last_peered":"2024-03-06T07:20:51.897856+0000","last_clean":"2024-03-06T07:20:51.897856+0000","last_became_active":"2024-03-06T07:20:50.879319+0000","last_became_peered":"2024-03-06T07:20:50.879319+0000","last_unstale":"2024-03-06T07:20:51.897856+0000","last_undegraded":"2024-03-06T07:20:51.897856+0000","last_fullsized":"2024-03-06T07:20:51.897856+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:58.810754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886209+0000","last_change":"2024-03-06T07:20:51.884046+0000","last_active":"2024-03-06T07:20:51.886209+0000","last_peered":"2024-03-06T07:20:51.886209+0000","last_clean":"2024-03-06T07:20:51.886209+0000","last_became_active":"2024-03-06T07:20:51.883593+0000","last_became_peered":"2024-03-06T07:20:51.883593+0000","last_unstale":"2024-03-06T07:20:51.886209+0000","last_undegraded":"2024-03-06T07:20:51.886209+0000","last_fullsized":"2024-03-06T07:20:51.886209+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:53.956654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896404+0000","last_change":"2024-03-06T07:20:50.888715+0000","last_active":"2024-03-06T07:20:51.896404+0000","last_peered":"2024-03-06T07:20:51.896404+0000","last_clean":"2024-03-06T07:20:51.896404+0000","last_became_active":"2024-03-06T07:20:50.888140+0000","last_became_peered":"2024-03-06T07:20:50.888140+0000","last_unstale":"2024-03-06T07:20:51.896404+0000","last_undegraded":"2024-03-06T07:20:51.896404+0000","last_fullsized":"2024-03-06T07:20:51.896404+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:34.087293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889183+0000","last_change":"2024-03-06T07:20:51.886764+0000","last_active":"2024-03-06T07:20:51.889183+0000","last_peered":"2024-03-06T07:20:51.889183+0000","last_clean":"2024-03-06T07:20:51.889183+0000","last_became_active":"2024-03-06T07:20:51.886187+0000","last_became_peered":"2024-03-06T07:20:51.886187+0000","last_unstale":"2024-03-06T07:20:51.889183+0000","last_undegraded":"2024-03-06T07:20:51.889183+0000","last_fullsized":"2024-03-06T07:20:51.889183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:28.691080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899922+0000","last_change":"2024-03-06T07:20:50.885147+0000","last_active":"2024-03-06T07:20:51.899922+0000","last_peered":"2024-03-06T07:20:51.899922+0000","last_clean":"2024-03-06T07:20:51.899922+0000","last_became_active":"2024-03-06T07:20:50.884811+0000","last_became_peered":"2024-03-06T07:20:50.884811+0000","last_unstale":"2024-03-06T07:20:51.899922+0000","last_undegraded":"2024-03-06T07:20:51.899922+0000","last_fullsized":"2024-03-06T07:20:51.899922+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:24.667879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890491+0000","last_change":"2024-03-06T07:20:51.887534+0000","last_active":"2024-03-06T07:20:51.890491+0000","last_peered":"2024-03-06T07:20:51.890491+0000","last_clean":"2024-03-06T07:20:51.890491+0000","last_became_active":"2024-03-06T07:20:51.887239+0000","last_became_peered":"2024-03-06T07:20:51.887239+0000","last_unstale":"2024-03-06T07:20:51.890491+0000","last_undegraded":"2024-03-06T07:20:51.890491+0000","last_fullsized":"2024-03-06T07:20:51.890491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:53.314738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890865+0000","last_change":"2024-03-06T07:20:50.892916+0000","last_active":"2024-03-06T07:20:51.890865+0000","last_peered":"2024-03-06T07:20:51.890865+0000","last_clean":"2024-03-06T07:20:51.890865+0000","last_became_active":"2024-03-06T07:20:50.892629+0000","last_became_peered":"2024-03-06T07:20:50.892629+0000","last_unstale":"2024-03-06T07:20:51.890865+0000","last_undegraded":"2024-03-06T07:20:51.890865+0000","last_fullsized":"2024-03-06T07:20:51.890865+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:12.808948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896846+0000","last_change":"2024-03-06T07:20:51.895988+0000","last_active":"2024-03-06T07:20:51.896846+0000","last_peered":"2024-03-06T07:20:51.896846+0000","last_clean":"2024-03-06T07:20:51.896846+0000","last_became_active":"2024-03-06T07:20:51.895747+0000","last_became_peered":"2024-03-06T07:20:51.895747+0000","last_unstale":"2024-03-06T07:20:51.896846+0000","last_undegraded":"2024-03-06T07:20:51.896846+0000","last_fullsized":"2024-03-06T07:20:51.896846+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:34.418498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891812+0000","last_change":"2024-03-06T07:20:50.893482+0000","last_active":"2024-03-06T07:20:51.891812+0000","last_peered":"2024-03-06T07:20:51.891812+0000","last_clean":"2024-03-06T07:20:51.891812+0000","last_became_active":"2024-03-06T07:20:50.893153+0000","last_became_peered":"2024-03-06T07:20:50.893153+0000","last_unstale":"2024-03-06T07:20:51.891812+0000","last_undegraded":"2024-03-06T07:20:51.891812+0000","last_fullsized":"2024-03-06T07:20:51.891812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:07.775768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898098+0000","last_change":"2024-03-06T07:20:51.897529+0000","last_active":"2024-03-06T07:20:51.898098+0000","last_peered":"2024-03-06T07:20:51.898098+0000","last_clean":"2024-03-06T07:20:51.898098+0000","last_became_active":"2024-03-06T07:20:51.897181+0000","last_became_peered":"2024-03-06T07:20:51.897181+0000","last_unstale":"2024-03-06T07:20:51.898098+0000","last_undegraded":"2024-03-06T07:20:51.898098+0000","last_fullsized":"2024-03-06T07:20:51.898098+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:35.092363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890796+0000","last_change":"2024-03-06T07:20:50.886303+0000","last_active":"2024-03-06T07:20:51.890796+0000","last_peered":"2024-03-06T07:20:51.890796+0000","last_clean":"2024-03-06T07:20:51.890796+0000","last_became_active":"2024-03-06T07:20:50.886013+0000","last_became_peered":"2024-03-06T07:20:50.886013+0000","last_unstale":"2024-03-06T07:20:51.890796+0000","last_undegraded":"2024-03-06T07:20:51.890796+0000","last_fullsized":"2024-03-06T07:20:51.890796+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:34.617731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889833+0000","last_change":"2024-03-06T07:20:51.883834+0000","last_active":"2024-03-06T07:20:51.889833+0000","last_peered":"2024-03-06T07:20:51.889833+0000","last_clean":"2024-03-06T07:20:51.889833+0000","last_became_active":"2024-03-06T07:20:51.883553+0000","last_became_peered":"2024-03-06T07:20:51.883553+0000","last_unstale":"2024-03-06T07:20:51.889833+0000","last_undegraded":"2024-03-06T07:20:51.889833+0000","last_fullsized":"2024-03-06T07:20:51.889833+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:37.311544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.875471+0000","last_change":"2024-03-06T07:20:50.881449+0000","last_active":"2024-03-06T07:20:51.875471+0000","last_peered":"2024-03-06T07:20:51.875471+0000","last_clean":"2024-03-06T07:20:51.875471+0000","last_became_active":"2024-03-06T07:20:50.881172+0000","last_became_peered":"2024-03-06T07:20:50.881172+0000","last_unstale":"2024-03-06T07:20:51.875471+0000","last_undegraded":"2024-03-06T07:20:51.875471+0000","last_fullsized":"2024-03-06T07:20:51.875471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.317522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.895950+0000","last_change":"2024-03-06T07:20:51.893411+0000","last_active":"2024-03-06T07:20:51.895950+0000","last_peered":"2024-03-06T07:20:51.895950+0000","last_clean":"2024-03-06T07:20:51.895950+0000","last_became_active":"2024-03-06T07:20:51.893149+0000","last_became_peered":"2024-03-06T07:20:51.893149+0000","last_unstale":"2024-03-06T07:20:51.895950+0000","last_undegraded":"2024-03-06T07:20:51.895950+0000","last_fullsized":"2024-03-06T07:20:51.895950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:09.457018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.878563+0000","last_change":"2024-03-06T07:20:50.890159+0000","last_active":"2024-03-06T07:20:51.878563+0000","last_peered":"2024-03-06T07:20:51.878563+0000","last_clean":"2024-03-06T07:20:51.878563+0000","last_became_active":"2024-03-06T07:20:50.889863+0000","last_became_peered":"2024-03-06T07:20:50.889863+0000","last_unstale":"2024-03-06T07:20:51.878563+0000","last_undegraded":"2024-03-06T07:20:51.878563+0000","last_fullsized":"2024-03-06T07:20:51.878563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.303590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899855+0000","last_change":"2024-03-06T07:20:51.899199+0000","last_active":"2024-03-06T07:20:51.899855+0000","last_peered":"2024-03-06T07:20:51.899855+0000","last_clean":"2024-03-06T07:20:51.899855+0000","last_became_active":"2024-03-06T07:20:51.898622+0000","last_became_peered":"2024-03-06T07:20:51.898622+0000","last_unstale":"2024-03-06T07:20:51.899855+0000","last_undegraded":"2024-03-06T07:20:51.899855+0000","last_fullsized":"2024-03-06T07:20:51.899855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:36.873911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.883407+0000","last_change":"2024-03-06T07:20:50.891619+0000","last_active":"2024-03-06T07:20:51.883407+0000","last_peered":"2024-03-06T07:20:51.883407+0000","last_clean":"2024-03-06T07:20:51.883407+0000","last_became_active":"2024-03-06T07:20:50.891305+0000","last_became_peered":"2024-03-06T07:20:50.891305+0000","last_unstale":"2024-03-06T07:20:51.883407+0000","last_undegraded":"2024-03-06T07:20:51.883407+0000","last_fullsized":"2024-03-06T07:20:51.883407+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.732044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890297+0000","last_change":"2024-03-06T07:20:51.887972+0000","last_active":"2024-03-06T07:20:51.890297+0000","last_peered":"2024-03-06T07:20:51.890297+0000","last_clean":"2024-03-06T07:20:51.890297+0000","last_became_active":"2024-03-06T07:20:51.887725+0000","last_became_peered":"2024-03-06T07:20:51.887725+0000","last_unstale":"2024-03-06T07:20:51.890297+0000","last_undegraded":"2024-03-06T07:20:51.890297+0000","last_fullsized":"2024-03-06T07:20:51.890297+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:06.737114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.877638+0000","last_change":"2024-03-06T07:20:50.890731+0000","last_active":"2024-03-06T07:20:51.877638+0000","last_peered":"2024-03-06T07:20:51.877638+0000","last_clean":"2024-03-06T07:20:51.877638+0000","last_became_active":"2024-03-06T07:20:50.890429+0000","last_became_peered":"2024-03-06T07:20:50.890429+0000","last_unstale":"2024-03-06T07:20:51.877638+0000","last_undegraded":"2024-03-06T07:20:51.877638+0000","last_fullsized":"2024-03-06T07:20:51.877638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891008+0000","last_change":"2024-03-06T07:20:51.885412+0000","last_active":"2024-03-06T07:20:51.891008+0000","last_peered":"2024-03-06T07:20:51.891008+0000","last_clean":"2024-03-06T07:20:51.891008+0000","last_became_active":"2024-03-06T07:20:51.885153+0000","last_became_peered":"2024-03-06T07:20:51.885153+0000","last_unstale":"2024-03-06T07:20:51.891008+0000","last_undegraded":"2024-03-06T07:20:51.891008+0000","last_fullsized":"2024-03-06T07:20:51.891008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":15,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.892651+0000","last_change":"2024-03-06T07:20:50.880143+0000","last_active":"2024-03-06T07:20:51.892651+0000","last_peered":"2024-03-06T07:20:51.892651+0000","last_clean":"2024-03-06T07:20:51.892651+0000","last_became_active":"2024-03-06T07:20:50.879859+0000","last_became_peered":"2024-03-06T07:20:50.879859+0000","last_unstale":"2024-03-06T07:20:51.892651+0000","last_undegraded":"2024-03-06T07:20:51.892651+0000","last_fullsized":"2024-03-06T07:20:51.892651+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:08.684501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879810+0000","last_change":"2024-03-06T07:20:51.878936+0000","last_active":"2024-03-06T07:20:51.879810+0000","last_peered":"2024-03-06T07:20:51.879810+0000","last_clean":"2024-03-06T07:20:51.879810+0000","last_became_active":"2024-03-06T07:20:51.878613+0000","last_became_peered":"2024-03-06T07:20:51.878613+0000","last_unstale":"2024-03-06T07:20:51.879810+0000","last_undegraded":"2024-03-06T07:20:51.879810+0000","last_fullsized":"2024-03-06T07:20:51.879810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":18,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899162+0000","last_change":"2024-03-06T07:20:50.888110+0000","last_active":"2024-03-06T07:20:51.899162+0000","last_peered":"2024-03-06T07:20:51.899162+0000","last_clean":"2024-03-06T07:20:51.899162+0000","last_became_active":"2024-03-06T07:20:50.887804+0000","last_became_peered":"2024-03-06T07:20:50.887804+0000","last_unstale":"2024-03-06T07:20:51.899162+0000","last_undegraded":"2024-03-06T07:20:51.899162+0000","last_fullsized":"2024-03-06T07:20:51.899162+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.896181+0000","last_change":"2024-03-06T07:20:51.891149+0000","last_active":"2024-03-06T07:20:51.896181+0000","last_peered":"2024-03-06T07:20:51.896181+0000","last_clean":"2024-03-06T07:20:51.896181+0000","last_became_active":"2024-03-06T07:20:51.890877+0000","last_became_peered":"2024-03-06T07:20:51.890877+0000","last_unstale":"2024-03-06T07:20:51.896181+0000","last_undegraded":"2024-03-06T07:20:51.896181+0000","last_fullsized":"2024-03-06T07:20:51.896181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:25.672949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894759+0000","last_change":"2024-03-06T07:20:50.888227+0000","last_active":"2024-03-06T07:20:51.894759+0000","last_peered":"2024-03-06T07:20:51.894759+0000","last_clean":"2024-03-06T07:20:51.894759+0000","last_became_active":"2024-03-06T07:20:50.887926+0000","last_became_peered":"2024-03-06T07:20:50.887926+0000","last_unstale":"2024-03-06T07:20:51.894759+0000","last_undegraded":"2024-03-06T07:20:51.894759+0000","last_fullsized":"2024-03-06T07:20:51.894759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:56.255517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891054+0000","last_change":"2024-03-06T07:20:51.888556+0000","last_active":"2024-03-06T07:20:51.891054+0000","last_peered":"2024-03-06T07:20:51.891054+0000","last_clean":"2024-03-06T07:20:51.891054+0000","last_became_active":"2024-03-06T07:20:51.888278+0000","last_became_peered":"2024-03-06T07:20:51.888278+0000","last_unstale":"2024-03-06T07:20:51.891054+0000","last_undegraded":"2024-03-06T07:20:51.891054+0000","last_fullsized":"2024-03-06T07:20:51.891054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.880871+0000","last_change":"2024-03-06T07:20:50.881110+0000","last_active":"2024-03-06T07:20:51.880871+0000","last_peered":"2024-03-06T07:20:51.880871+0000","last_clean":"2024-03-06T07:20:51.880871+0000","last_became_active":"2024-03-06T07:20:50.880752+0000","last_became_peered":"2024-03-06T07:20:50.880752+0000","last_unstale":"2024-03-06T07:20:51.880871+0000","last_undegraded":"2024-03-06T07:20:51.880871+0000","last_fullsized":"2024-03-06T07:20:51.880871+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886530+0000","last_change":"2024-03-06T07:20:51.879914+0000","last_active":"2024-03-06T07:20:51.886530+0000","last_peered":"2024-03-06T07:20:51.886530+0000","last_clean":"2024-03-06T07:20:51.886530+0000","last_became_active":"2024-03-06T07:20:51.879633+0000","last_became_peered":"2024-03-06T07:20:51.879633+0000","last_unstale":"2024-03-06T07:20:51.886530+0000","last_undegraded":"2024-03-06T07:20:51.886530+0000","last_fullsized":"2024-03-06T07:20:51.886530+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:58.399266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.878494+0000","last_change":"2024-03-06T07:20:50.880559+0000","last_active":"2024-03-06T07:20:51.878494+0000","last_peered":"2024-03-06T07:20:51.878494+0000","last_clean":"2024-03-06T07:20:51.878494+0000","last_became_active":"2024-03-06T07:20:50.880160+0000","last_became_peered":"2024-03-06T07:20:50.880160+0000","last_unstale":"2024-03-06T07:20:51.878494+0000","last_undegraded":"2024-03-06T07:20:51.878494+0000","last_fullsized":"2024-03-06T07:20:51.878494+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:56.900930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891957+0000","last_change":"2024-03-06T07:20:51.890440+0000","last_active":"2024-03-06T07:20:51.891957+0000","last_peered":"2024-03-06T07:20:51.891957+0000","last_clean":"2024-03-06T07:20:51.891957+0000","last_became_active":"2024-03-06T07:20:51.890167+0000","last_became_peered":"2024-03-06T07:20:51.890167+0000","last_unstale":"2024-03-06T07:20:51.891957+0000","last_undegraded":"2024-03-06T07:20:51.891957+0000","last_fullsized":"2024-03-06T07:20:51.891957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":20,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889125+0000","last_change":"2024-03-06T07:20:50.885352+0000","last_active":"2024-03-06T07:20:51.889125+0000","last_peered":"2024-03-06T07:20:51.889125+0000","last_clean":"2024-03-06T07:20:51.889125+0000","last_became_active":"2024-03-06T07:20:50.884991+0000","last_became_peered":"2024-03-06T07:20:50.884991+0000","last_unstale":"2024-03-06T07:20:51.889125+0000","last_undegraded":"2024-03-06T07:20:51.889125+0000","last_fullsized":"2024-03-06T07:20:51.889125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:03.652296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.892091+0000","last_change":"2024-03-06T07:20:51.889501+0000","last_active":"2024-03-06T07:20:51.892091+0000","last_peered":"2024-03-06T07:20:51.892091+0000","last_clean":"2024-03-06T07:20:51.892091+0000","last_became_active":"2024-03-06T07:20:51.889206+0000","last_became_peered":"2024-03-06T07:20:51.889206+0000","last_unstale":"2024-03-06T07:20:51.892091+0000","last_undegraded":"2024-03-06T07:20:51.892091+0000","last_fullsized":"2024-03-06T07:20:51.892091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.891700+0000","last_change":"2024-03-06T07:20:50.883039+0000","last_active":"2024-03-06T07:20:51.891700+0000","last_peered":"2024-03-06T07:20:51.891700+0000","last_clean":"2024-03-06T07:20:51.891700+0000","last_became_active":"2024-03-06T07:20:50.882748+0000","last_became_peered":"2024-03-06T07:20:50.882748+0000","last_unstale":"2024-03-06T07:20:51.891700+0000","last_undegraded":"2024-03-06T07:20:51.891700+0000","last_fullsized":"2024-03-06T07:20:51.891700+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:35.868841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893043+0000","last_change":"2024-03-06T07:20:51.890860+0000","last_active":"2024-03-06T07:20:51.893043+0000","last_peered":"2024-03-06T07:20:51.893043+0000","last_clean":"2024-03-06T07:20:51.893043+0000","last_became_active":"2024-03-06T07:20:51.890576+0000","last_became_peered":"2024-03-06T07:20:51.890576+0000","last_unstale":"2024-03-06T07:20:51.893043+0000","last_undegraded":"2024-03-06T07:20:51.893043+0000","last_fullsized":"2024-03-06T07:20:51.893043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:00.335725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.887822+0000","last_change":"2024-03-06T07:20:50.888701+0000","last_active":"2024-03-06T07:20:51.887822+0000","last_peered":"2024-03-06T07:20:51.887822+0000","last_clean":"2024-03-06T07:20:51.887822+0000","last_became_active":"2024-03-06T07:20:50.888392+0000","last_became_peered":"2024-03-06T07:20:50.888392+0000","last_unstale":"2024-03-06T07:20:51.887822+0000","last_undegraded":"2024-03-06T07:20:51.887822+0000","last_fullsized":"2024-03-06T07:20:51.887822+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.582638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890650+0000","last_change":"2024-03-06T07:20:51.887239+0000","last_active":"2024-03-06T07:20:51.890650+0000","last_peered":"2024-03-06T07:20:51.890650+0000","last_clean":"2024-03-06T07:20:51.890650+0000","last_became_active":"2024-03-06T07:20:51.886940+0000","last_became_peered":"2024-03-06T07:20:51.886940+0000","last_unstale":"2024-03-06T07:20:51.890650+0000","last_undegraded":"2024-03-06T07:20:51.890650+0000","last_fullsized":"2024-03-06T07:20:51.890650+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:27.561295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":13,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.897360+0000","last_change":"2024-03-06T07:20:50.886879+0000","last_active":"2024-03-06T07:20:51.897360+0000","last_peered":"2024-03-06T07:20:51.897360+0000","last_clean":"2024-03-06T07:20:51.897360+0000","last_became_active":"2024-03-06T07:20:50.886589+0000","last_became_peered":"2024-03-06T07:20:50.886589+0000","last_unstale":"2024-03-06T07:20:51.897360+0000","last_undegraded":"2024-03-06T07:20:51.897360+0000","last_fullsized":"2024-03-06T07:20:51.897360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:51.245367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893777+0000","last_change":"2024-03-06T07:20:51.892549+0000","last_active":"2024-03-06T07:20:51.893777+0000","last_peered":"2024-03-06T07:20:51.893777+0000","last_clean":"2024-03-06T07:20:51.893777+0000","last_became_active":"2024-03-06T07:20:51.889936+0000","last_became_peered":"2024-03-06T07:20:51.889936+0000","last_unstale":"2024-03-06T07:20:51.893777+0000","last_undegraded":"2024-03-06T07:20:51.893777+0000","last_fullsized":"2024-03-06T07:20:51.893777+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:56.445613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.869561+0000","last_change":"2024-03-06T07:20:50.883071+0000","last_active":"2024-03-06T07:20:51.869561+0000","last_peered":"2024-03-06T07:20:51.869561+0000","last_clean":"2024-03-06T07:20:51.869561+0000","last_became_active":"2024-03-06T07:20:50.882465+0000","last_became_peered":"2024-03-06T07:20:50.882465+0000","last_unstale":"2024-03-06T07:20:51.869561+0000","last_undegraded":"2024-03-06T07:20:51.869561+0000","last_fullsized":"2024-03-06T07:20:51.869561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886693+0000","last_change":"2024-03-06T07:20:51.884694+0000","last_active":"2024-03-06T07:20:51.886693+0000","last_peered":"2024-03-06T07:20:51.886693+0000","last_clean":"2024-03-06T07:20:51.886693+0000","last_became_active":"2024-03-06T07:20:51.883905+0000","last_became_peered":"2024-03-06T07:20:51.883905+0000","last_unstale":"2024-03-06T07:20:51.886693+0000","last_undegraded":"2024-03-06T07:20:51.886693+0000","last_fullsized":"2024-03-06T07:20:51.886693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:54.467344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.890631+0000","last_change":"2024-03-06T07:20:50.881489+0000","last_active":"2024-03-06T07:20:51.890631+0000","last_peered":"2024-03-06T07:20:51.890631+0000","last_clean":"2024-03-06T07:20:51.890631+0000","last_became_active":"2024-03-06T07:20:50.881182+0000","last_became_peered":"2024-03-06T07:20:50.881182+0000","last_unstale":"2024-03-06T07:20:51.890631+0000","last_undegraded":"2024-03-06T07:20:51.890631+0000","last_fullsized":"2024-03-06T07:20:51.890631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:22.518381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894864+0000","last_change":"2024-03-06T07:20:51.889352+0000","last_active":"2024-03-06T07:20:51.894864+0000","last_peered":"2024-03-06T07:20:51.894864+0000","last_clean":"2024-03-06T07:20:51.894864+0000","last_became_active":"2024-03-06T07:20:51.888464+0000","last_became_peered":"2024-03-06T07:20:51.888464+0000","last_unstale":"2024-03-06T07:20:51.894864+0000","last_undegraded":"2024-03-06T07:20:51.894864+0000","last_fullsized":"2024-03-06T07:20:51.894864+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:40.919584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.873223+0000","last_change":"2024-03-06T07:20:50.887415+0000","last_active":"2024-03-06T07:20:51.873223+0000","last_peered":"2024-03-06T07:20:51.873223+0000","last_clean":"2024-03-06T07:20:51.873223+0000","last_became_active":"2024-03-06T07:20:50.887137+0000","last_became_peered":"2024-03-06T07:20:50.887137+0000","last_unstale":"2024-03-06T07:20:51.873223+0000","last_undegraded":"2024-03-06T07:20:51.873223+0000","last_fullsized":"2024-03-06T07:20:51.873223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:54.210267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.887614+0000","last_change":"2024-03-06T07:20:51.885672+0000","last_active":"2024-03-06T07:20:51.887614+0000","last_peered":"2024-03-06T07:20:51.887614+0000","last_clean":"2024-03-06T07:20:51.887614+0000","last_became_active":"2024-03-06T07:20:51.885412+0000","last_became_peered":"2024-03-06T07:20:51.885412+0000","last_unstale":"2024-03-06T07:20:51.887614+0000","last_undegraded":"2024-03-06T07:20:51.887614+0000","last_fullsized":"2024-03-06T07:20:51.887614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.873284+0000","last_change":"2024-03-06T07:20:50.889223+0000","last_active":"2024-03-06T07:20:51.873284+0000","last_peered":"2024-03-06T07:20:51.873284+0000","last_clean":"2024-03-06T07:20:51.873284+0000","last_became_active":"2024-03-06T07:20:50.888934+0000","last_became_peered":"2024-03-06T07:20:50.888934+0000","last_unstale":"2024-03-06T07:20:51.873284+0000","last_undegraded":"2024-03-06T07:20:51.873284+0000","last_fullsized":"2024-03-06T07:20:51.873284+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:04.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.884730+0000","last_change":"2024-03-06T07:20:51.880937+0000","last_active":"2024-03-06T07:20:51.884730+0000","last_peered":"2024-03-06T07:20:51.884730+0000","last_clean":"2024-03-06T07:20:51.884730+0000","last_became_active":"2024-03-06T07:20:51.880648+0000","last_became_peered":"2024-03-06T07:20:51.880648+0000","last_unstale":"2024-03-06T07:20:51.884730+0000","last_undegraded":"2024-03-06T07:20:51.884730+0000","last_fullsized":"2024-03-06T07:20:51.884730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:16.890629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":14,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.886046+0000","last_change":"2024-03-06T07:20:50.886345+0000","last_active":"2024-03-06T07:20:51.886046+0000","last_peered":"2024-03-06T07:20:51.886046+0000","last_clean":"2024-03-06T07:20:51.886046+0000","last_became_active":"2024-03-06T07:20:50.886074+0000","last_became_peered":"2024-03-06T07:20:50.886074+0000","last_unstale":"2024-03-06T07:20:51.886046+0000","last_undegraded":"2024-03-06T07:20:51.886046+0000","last_fullsized":"2024-03-06T07:20:51.886046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:39.914514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.893225+0000","last_change":"2024-03-06T07:20:51.889471+0000","last_active":"2024-03-06T07:20:51.893225+0000","last_peered":"2024-03-06T07:20:51.893225+0000","last_clean":"2024-03-06T07:20:51.893225+0000","last_became_active":"2024-03-06T07:20:51.889218+0000","last_became_peered":"2024-03-06T07:20:51.889218+0000","last_unstale":"2024-03-06T07:20:51.893225+0000","last_undegraded":"2024-03-06T07:20:51.893225+0000","last_fullsized":"2024-03-06T07:20:51.893225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.877169+0000","last_change":"2024-03-06T07:20:50.881740+0000","last_active":"2024-03-06T07:20:51.877169+0000","last_peered":"2024-03-06T07:20:51.877169+0000","last_clean":"2024-03-06T07:20:51.877169+0000","last_became_active":"2024-03-06T07:20:50.881306+0000","last_became_peered":"2024-03-06T07:20:50.881306+0000","last_unstale":"2024-03-06T07:20:51.877169+0000","last_undegraded":"2024-03-06T07:20:51.877169+0000","last_fullsized":"2024-03-06T07:20:51.877169+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:35.733892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.889206+0000","last_change":"2024-03-06T07:20:51.886967+0000","last_active":"2024-03-06T07:20:51.889206+0000","last_peered":"2024-03-06T07:20:51.889206+0000","last_clean":"2024-03-06T07:20:51.889206+0000","last_became_active":"2024-03-06T07:20:51.886521+0000","last_became_peered":"2024-03-06T07:20:51.886521+0000","last_unstale":"2024-03-06T07:20:51.889206+0000","last_undegraded":"2024-03-06T07:20:51.889206+0000","last_fullsized":"2024-03-06T07:20:51.889206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:37.112027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.884762+0000","last_change":"2024-03-06T07:20:50.885378+0000","last_active":"2024-03-06T07:20:51.884762+0000","last_peered":"2024-03-06T07:20:51.884762+0000","last_clean":"2024-03-06T07:20:51.884762+0000","last_became_active":"2024-03-06T07:20:50.884859+0000","last_became_peered":"2024-03-06T07:20:50.884859+0000","last_unstale":"2024-03-06T07:20:51.884762+0000","last_undegraded":"2024-03-06T07:20:51.884762+0000","last_fullsized":"2024-03-06T07:20:51.884762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:28.686737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.899861+0000","last_change":"2024-03-06T07:20:51.899213+0000","last_active":"2024-03-06T07:20:51.899861+0000","last_peered":"2024-03-06T07:20:51.899861+0000","last_clean":"2024-03-06T07:20:51.899861+0000","last_became_active":"2024-03-06T07:20:51.898882+0000","last_became_peered":"2024-03-06T07:20:51.898882+0000","last_unstale":"2024-03-06T07:20:51.899861+0000","last_undegraded":"2024-03-06T07:20:51.899861+0000","last_fullsized":"2024-03-06T07:20:51.899861+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:25.672949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.881839+0000","last_change":"2024-03-06T07:20:50.888878+0000","last_active":"2024-03-06T07:20:51.881839+0000","last_peered":"2024-03-06T07:20:51.881839+0000","last_clean":"2024-03-06T07:20:51.881839+0000","last_became_active":"2024-03-06T07:20:50.888604+0000","last_became_peered":"2024-03-06T07:20:50.888604+0000","last_unstale":"2024-03-06T07:20:51.881839+0000","last_undegraded":"2024-03-06T07:20:51.881839+0000","last_fullsized":"2024-03-06T07:20:51.881839+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:33.413428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894115+0000","last_change":"2024-03-06T07:20:51.893070+0000","last_active":"2024-03-06T07:20:51.894115+0000","last_peered":"2024-03-06T07:20:51.894115+0000","last_clean":"2024-03-06T07:20:51.894115+0000","last_became_active":"2024-03-06T07:20:51.892785+0000","last_became_peered":"2024-03-06T07:20:51.892785+0000","last_unstale":"2024-03-06T07:20:51.894115+0000","last_undegraded":"2024-03-06T07:20:51.894115+0000","last_fullsized":"2024-03-06T07:20:51.894115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:58.622652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.875874+0000","last_change":"2024-03-06T07:20:50.892942+0000","last_active":"2024-03-06T07:20:51.875874+0000","last_peered":"2024-03-06T07:20:51.875874+0000","last_clean":"2024-03-06T07:20:51.875874+0000","last_became_active":"2024-03-06T07:20:50.892638+0000","last_became_peered":"2024-03-06T07:20:50.892638+0000","last_unstale":"2024-03-06T07:20:51.875874+0000","last_undegraded":"2024-03-06T07:20:51.875874+0000","last_fullsized":"2024-03-06T07:20:51.875874+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:40.434821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.898581+0000","last_change":"2024-03-06T07:20:51.895826+0000","last_active":"2024-03-06T07:20:51.898581+0000","last_peered":"2024-03-06T07:20:51.898581+0000","last_clean":"2024-03-06T07:20:51.898581+0000","last_became_active":"2024-03-06T07:20:51.895483+0000","last_became_peered":"2024-03-06T07:20:51.895483+0000","last_unstale":"2024-03-06T07:20:51.898581+0000","last_undegraded":"2024-03-06T07:20:51.898581+0000","last_fullsized":"2024-03-06T07:20:51.898581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879788+0000","last_change":"2024-03-06T07:20:50.879754+0000","last_active":"2024-03-06T07:20:51.879788+0000","last_peered":"2024-03-06T07:20:51.879788+0000","last_clean":"2024-03-06T07:20:51.879788+0000","last_became_active":"2024-03-06T07:20:50.879488+0000","last_became_peered":"2024-03-06T07:20:50.879488+0000","last_unstale":"2024-03-06T07:20:51.879788+0000","last_undegraded":"2024-03-06T07:20:51.879788+0000","last_fullsized":"2024-03-06T07:20:51.879788+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.603045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.879838+0000","last_change":"2024-03-06T07:20:51.879085+0000","last_active":"2024-03-06T07:20:51.879838+0000","last_peered":"2024-03-06T07:20:51.879838+0000","last_clean":"2024-03-06T07:20:51.879838+0000","last_became_active":"2024-03-06T07:20:51.878776+0000","last_became_peered":"2024-03-06T07:20:51.878776+0000","last_unstale":"2024-03-06T07:20:51.879838+0000","last_undegraded":"2024-03-06T07:20:51.879838+0000","last_fullsized":"2024-03-06T07:20:51.879838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:58.622652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.875749+0000","last_change":"2024-03-06T07:20:50.887928+0000","last_active":"2024-03-06T07:20:51.875749+0000","last_peered":"2024-03-06T07:20:51.875749+0000","last_clean":"2024-03-06T07:20:51.875749+0000","last_became_active":"2024-03-06T07:20:50.887638+0000","last_became_peered":"2024-03-06T07:20:50.887638+0000","last_unstale":"2024-03-06T07:20:51.875749+0000","last_undegraded":"2024-03-06T07:20:51.875749+0000","last_fullsized":"2024-03-06T07:20:51.875749+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:52.309668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.884344+0000","last_change":"2024-03-06T07:20:51.883253+0000","last_active":"2024-03-06T07:20:51.884344+0000","last_peered":"2024-03-06T07:20:51.884344+0000","last_clean":"2024-03-06T07:20:51.884344+0000","last_became_active":"2024-03-06T07:20:51.882993+0000","last_became_peered":"2024-03-06T07:20:51.882993+0000","last_unstale":"2024-03-06T07:20:51.884344+0000","last_undegraded":"2024-03-06T07:20:51.884344+0000","last_fullsized":"2024-03-06T07:20:51.884344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:09.457018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.869503+0000","last_change":"2024-03-06T07:20:50.879149+0000","last_active":"2024-03-06T07:20:51.869503+0000","last_peered":"2024-03-06T07:20:51.869503+0000","last_clean":"2024-03-06T07:20:51.869503+0000","last_became_active":"2024-03-06T07:20:50.878867+0000","last_became_peered":"2024-03-06T07:20:50.878867+0000","last_unstale":"2024-03-06T07:20:51.869503+0000","last_undegraded":"2024-03-06T07:20:51.869503+0000","last_fullsized":"2024-03-06T07:20:51.869503+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:57.394196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.894575+0000","last_change":"2024-03-06T07:20:51.891707+0000","last_active":"2024-03-06T07:20:51.894575+0000","last_peered":"2024-03-06T07:20:51.894575+0000","last_clean":"2024-03-06T07:20:51.894575+0000","last_became_active":"2024-03-06T07:20:51.891427+0000","last_became_peered":"2024-03-06T07:20:51.891427+0000","last_unstale":"2024-03-06T07:20:51.894575+0000","last_undegraded":"2024-03-06T07:20:51.894575+0000","last_fullsized":"2024-03-06T07:20:51.894575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:21.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.873811+0000","last_change":"2024-03-06T07:20:50.885372+0000","last_active":"2024-03-06T07:20:51.873811+0000","last_peered":"2024-03-06T07:20:51.873811+0000","last_clean":"2024-03-06T07:20:51.873811+0000","last_became_active":"2024-03-06T07:20:50.885093+0000","last_became_peered":"2024-03-06T07:20:50.885093+0000","last_unstale":"2024-03-06T07:20:51.873811+0000","last_undegraded":"2024-03-06T07:20:51.873811+0000","last_fullsized":"2024-03-06T07:20:51.873811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:12.807980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":270,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-06T07:20:51.881344+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:20:51.881344+0000","last_peered":"2024-03-06T07:20:51.881344+0000","last_clean":"2024-03-06T07:20:51.881344+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:20:51.881344+0000","last_undegraded":"2024-03-06T07:20:51.881344+0000","last_fullsized":"2024-03-06T07:20:51.881344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":962560,"data_stored":1010269,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":412712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":880640,"data_stored":412712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32076,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339764,"statfs":{"total":96636764160,"available":96603918336,"internally_reserved":0,"allocated":909312,"data_stored":420655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":897024,"data_stored":420461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":905216,"data_stored":420621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":880640,"data_stored":412712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":1010247,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:20:55.492 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:20:55.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:20:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.634+0000 7f5594ec3640 1 Processor -- start 2024-03-06T07:20:55.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.634+0000 7f5594ec3640 1 -- start start 2024-03-06T07:20:55.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f5594ec3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f55900c12a0 0x7f55900c1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f5594ec3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55900c7710 0x7f55900c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f5594ec3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5590155ed0 con 0x7f5590152770 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f5594ec3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55900c6a00 con 0x7f55900c7710 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f5594ec3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55900c6c30 con 0x7f55900c12a0 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f558ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55900c7710 0x7f55900c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f558ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55900c7710 0x7f55900c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52200/0 (socket says 172.21.15.94:52200) 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f558ed9d640 1 -- 172.21.15.94:0/696474047 learned_addr learned my addr 172.21.15.94:0/696474047 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f558dd9b640 1 --2- 172.21.15.94:0/696474047 >> v2:172.21.15.145:3300/0 conn(0x7f55900c12a0 0x7f55900c1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f558ed9d640 1 -- 172.21.15.94:0/696474047 >> v2:172.21.15.145:3300/0 conn(0x7f55900c12a0 msgr2=0x7f55900c1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.635+0000 7f558ed9d640 1 --2- 172.21.15.94:0/696474047 >> v2:172.21.15.145:3300/0 conn(0x7f55900c12a0 0x7f55900c1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.636+0000 7f558ed9d640 1 -- 172.21.15.94:0/696474047 >> v1:172.21.15.145:6789/0 conn(0x7f5590152770 legacy=0x7f559006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.636+0000 7f558ed9d640 1 -- 172.21.15.94:0/696474047 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55900c6f10 con 0x7f55900c7710 2024-03-06T07:20:55.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.636+0000 7f558ed9d640 1 --2- 172.21.15.94:0/696474047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f55900c7b40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f55900c41d0 tx=0x7f558802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab07a48059660b45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.636+0000 7f558d59a640 1 -- 172.21.15.94:0/696474047 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5588034070 con 0x7f55900c7710 2024-03-06T07:20:55.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.636+0000 7f558d59a640 1 -- 172.21.15.94:0/696474047 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5588030860 con 0x7f55900c7710 2024-03-06T07:20:55.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.636+0000 7f558d59a640 1 -- 172.21.15.94:0/696474047 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5588030c30 con 0x7f55900c7710 2024-03-06T07:20:55.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 -- 172.21.15.94:0/696474047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 msgr2=0x7f55900c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 --2- 172.21.15.94:0/696474047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f55900c7b40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f55900c41d0 tx=0x7f558802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 -- 172.21.15.94:0/696474047 shutdown_connections 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 --2- 172.21.15.94:0/696474047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f55900c7b40 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 --2- 172.21.15.94:0/696474047 >> v2:172.21.15.145:3300/0 conn(0x7f55900c12a0 0x7f55900c1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 -- 172.21.15.94:0/696474047 >> 172.21.15.94:0/696474047 conn(0x7f55900a94d0 msgr2=0x7f55900a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 -- 172.21.15.94:0/696474047 shutdown_connections 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.637+0000 7f5594ec3640 1 -- 172.21.15.94:0/696474047 wait complete. 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.638+0000 7f5594ec3640 1 Processor -- start 2024-03-06T07:20:55.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.638+0000 7f5594ec3640 1 -- start start 2024-03-06T07:20:55.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f5594ec3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55900c12a0 0x7f55900aa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f5594ec3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f559014a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f5594ec3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5590152770 0x7f559014af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f5594ec3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590155ba0 con 0x7f55900c7710 2024-03-06T07:20:55.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f5594ec3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590152b50 con 0x7f5590152770 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f5594ec3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559006b6d0 con 0x7f55900c12a0 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f559014a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f559014a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52206/0 (socket says 172.21.15.94:52206) 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 -- 172.21.15.94:0/1849079046 learned_addr learned my addr 172.21.15.94:0/1849079046 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558ed9d640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5590152770 0x7f559014af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558e59c640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55900c12a0 0x7f55900aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 -- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55900c12a0 msgr2=0x7f55900aa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55900c12a0 0x7f55900aa830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 -- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5590152770 msgr2=0x7f559014af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5590152770 0x7f559014af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.639+0000 7f558dd9b640 1 -- 172.21.15.94:0/1849079046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55900c4070 con 0x7f55900c7710 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.640+0000 7f558dd9b640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f559014a9f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f558400e8a0 tx=0x7f558400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.640+0000 7f557effd640 1 -- 172.21.15.94:0/1849079046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584017070 con 0x7f55900c7710 2024-03-06T07:20:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.640+0000 7f557effd640 1 -- 172.21.15.94:0/1849079046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558400c940 con 0x7f55900c7710 2024-03-06T07:20:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.640+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55900c4350 con 0x7f55900c7710 2024-03-06T07:20:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.640+0000 7f557effd640 1 -- 172.21.15.94:0/1849079046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558400cd30 con 0x7f55900c7710 2024-03-06T07:20:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.640+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559014b470 con 0x7f55900c7710 2024-03-06T07:20:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.641+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559004f330 con 0x7f55900c7710 2024-03-06T07:20:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.641+0000 7f557effd640 1 -- 172.21.15.94:0/1849079046 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f558401a070 con 0x7f55900c7710 2024-03-06T07:20:55.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.642+0000 7f557effd640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f557006b500 0x7f557006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.642+0000 7f557effd640 1 -- 172.21.15.94:0/1849079046 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f558408a460 con 0x7f55900c7710 2024-03-06T07:20:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.642+0000 7f558e59c640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f557006b500 0x7f557006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:55.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.642+0000 7f558e59c640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f557006b500 0x7f557006d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5590151920 tx=0x7f5580009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:55.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.646+0000 7f557effd640 1 -- 172.21.15.94:0/1849079046 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5584058b90 con 0x7f55900c7710 2024-03-06T07:20:55.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.885+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f55900c1010 con 0x7f55900c7710 2024-03-06T07:20:55.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.886+0000 7f557effd640 1 -- 172.21.15.94:0/1849079046 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f558405cc30 con 0x7f55900c7710 2024-03-06T07:20:55.888 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:20:55.889 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:20:55.889 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:20:55.889 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:20:55.889 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:20:55.889 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f557006b500 msgr2=0x7f557006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f557006b500 0x7f557006d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5590151920 tx=0x7f5580009040 comp rx=0 tx=0).stop 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 msgr2=0x7f559014a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f559014a9f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f558400e8a0 tx=0x7f558400ed60 comp rx=0 tx=0).stop 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 shutdown_connections 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f557006b500 0x7f557006d9c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5590152770 0x7f559014af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55900c7710 0x7f559014a9f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 --2- 172.21.15.94:0/1849079046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55900c12a0 0x7f55900aa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 >> 172.21.15.94:0/1849079046 conn(0x7f55900a94d0 msgr2=0x7f5590150920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 shutdown_connections 2024-03-06T07:20:55.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:55.889+0000 7f5594ec3640 1 -- 172.21.15.94:0/1849079046 wait complete. 2024-03-06T07:20:55.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:56.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.000+0000 7f0cbd1ac640 1 Processor -- start 2024-03-06T07:20:56.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.000+0000 7f0cbd1ac640 1 -- start start 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0cbd1ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0cbd1ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0cb80c1aa0 0x7f0cb80c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0cbd1ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb80c28d0 con 0x7f0cb80c5c60 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0cbd1ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb80c2680 con 0x7f0cb80c1aa0 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0cbd1ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb80c2bb0 con 0x7f0cb814f790 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0caffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0cb80c1aa0 0x7f0cb80c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0caffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0cb80c1aa0 0x7f0cb80c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52220/0 (socket says 172.21.15.94:52220) 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0caffff640 1 -- 172.21.15.94:0/76524707 learned_addr learned my addr 172.21.15.94:0/76524707 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0cb6d9d640 1 --2- 172.21.15.94:0/76524707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0caffff640 1 -- 172.21.15.94:0/76524707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 msgr2=0x7f0cb81529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0caffff640 1 --2- 172.21.15.94:0/76524707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0caffff640 1 -- 172.21.15.94:0/76524707 >> v1:172.21.15.145:6789/0 conn(0x7f0cb80c5c60 legacy=0x7f0cb8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.001+0000 7f0caffff640 1 -- 172.21.15.94:0/76524707 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb80c2e90 con 0x7f0cb80c1aa0 2024-03-06T07:20:56.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.002+0000 7f0caffff640 1 --2- 172.21.15.94:0/76524707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c1aa0 0x7f0cb80c5720 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009870 tx=0x7f0ca002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec6eef551ca6d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.002+0000 7f0cb659c640 1 -- 172.21.15.94:0/76524707 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0034070 con 0x7f0cb80c1aa0 2024-03-06T07:20:56.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.002+0000 7f0cb659c640 1 -- 172.21.15.94:0/76524707 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0030840 con 0x7f0cb80c1aa0 2024-03-06T07:20:56.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.002+0000 7f0cb659c640 1 -- 172.21.15.94:0/76524707 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0030c10 con 0x7f0cb80c1aa0 2024-03-06T07:20:56.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.002+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/76524707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c1aa0 msgr2=0x7f0cb80c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.002+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/76524707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c1aa0 0x7f0cb80c5720 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009870 tx=0x7f0ca002b980 comp rx=0 tx=0).stop 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.003+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/76524707 shutdown_connections 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.003+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/76524707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c1aa0 0x7f0cb80c5720 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.003+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/76524707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.003+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/76524707 >> 172.21.15.94:0/76524707 conn(0x7f0cb80c6710 msgr2=0x7f0cb80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.003+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/76524707 shutdown_connections 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.003+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/76524707 wait complete. 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.004+0000 7f0cbd1ac640 1 Processor -- start 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.004+0000 7f0cbd1ac640 1 -- start start 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.004+0000 7f0cbd1ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cb80c1aa0 0x7f0cb8152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cbd1ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c5c60 0x7f0cb80720e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cbd1ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cbd1ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb806cc00 con 0x7f0cb80c5c60 2024-03-06T07:20:56.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cbd1ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb80c0d60 con 0x7f0cb80c1aa0 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cbd1ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb806cd80 con 0x7f0cb814f790 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c5c60 0x7f0cb80720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c5c60 0x7f0cb80720e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52228/0 (socket says 172.21.15.94:52228) 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 -- 172.21.15.94:0/2957954437 learned_addr learned my addr 172.21.15.94:0/2957954437 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cb6d9d640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cb80c1aa0 0x7f0cb8152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cb759e640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 -- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 msgr2=0x7f0cb81471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 -- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cb80c1aa0 msgr2=0x7f0cb8152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cb80c1aa0 0x7f0cb8152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 -- 172.21.15.94:0/2957954437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb80711d0 con 0x7f0cb80c5c60 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caffff640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c5c60 0x7f0cb80720e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009870 tx=0x7f0ca0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caf7fe640 1 -- 172.21.15.94:0/2957954437 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca004a070 con 0x7f0cb80c5c60 2024-03-06T07:20:56.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caf7fe640 1 -- 172.21.15.94:0/2957954437 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0030640 con 0x7f0cb80c5c60 2024-03-06T07:20:56.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb80714b0 con 0x7f0cb80c5c60 2024-03-06T07:20:56.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0caf7fe640 1 -- 172.21.15.94:0/2957954437 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0034040 con 0x7f0cb80c5c60 2024-03-06T07:20:56.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.005+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb8072620 con 0x7f0cb80c5c60 2024-03-06T07:20:56.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.006+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb804f330 con 0x7f0cb80c5c60 2024-03-06T07:20:56.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.007+0000 7f0caf7fe640 1 -- 172.21.15.94:0/2957954437 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ca004d070 con 0x7f0cb80c5c60 2024-03-06T07:20:56.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.007+0000 7f0caf7fe640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c9806b500 0x7f0c9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.007+0000 7f0caf7fe640 1 -- 172.21.15.94:0/2957954437 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0ca00ae2f0 con 0x7f0cb80c5c60 2024-03-06T07:20:56.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.007+0000 7f0cb6d9d640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c9806b500 0x7f0c9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.008+0000 7f0cb6d9d640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c9806b500 0x7f0c9806d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009aa0 tx=0x7f0ca8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.011+0000 7f0caf7fe640 1 -- 172.21.15.94:0/2957954437 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ca007cb20 con 0x7f0cb80c5c60 2024-03-06T07:20:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.218+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb8068450 con 0x7f0cb80c5c60 2024-03-06T07:20:56.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.219+0000 7f0caf7fe640 1 -- 172.21.15.94:0/2957954437 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f0ca0080bc0 con 0x7f0cb80c5c60 2024-03-06T07:20:56.221 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:56.221 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":35486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:20:53.531385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35492":{"gid":35492,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3537750458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3537750458},{"type":"v1","addr":"172.21.15.145:6837","nonce":3537750458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35492,"qdb_cluster":[35492]},"id":5}]} 2024-03-06T07:20:56.221 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 55 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c9806b500 msgr2=0x7f0c9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c9806b500 0x7f0c9806d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8009aa0 tx=0x7f0ca8009190 comp rx=0 tx=0).stop 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c5c60 msgr2=0x7f0cb80720e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c5c60 0x7f0cb80720e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009870 tx=0x7f0ca0004730 comp rx=0 tx=0).stop 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 shutdown_connections 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c9806b500 0x7f0c9806d9c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cb814f790 0x7f0cb81471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb80c5c60 0x7f0cb80720e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 --2- 172.21.15.94:0/2957954437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cb80c1aa0 0x7f0cb8152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 >> 172.21.15.94:0/2957954437 conn(0x7f0cb80c6710 msgr2=0x7f0cb80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 shutdown_connections 2024-03-06T07:20:56.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.221+0000 7f0cbd1ac640 1 -- 172.21.15.94:0/2957954437 wait complete. 2024-03-06T07:20:56.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:20:51.223307+0000', 'modified': '2024-03-06T07:20:53.531385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 35492}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_35492': {'gid': 35492, 'name': 'b', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/3537750458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 3537750458}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3537750458}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 35492, 'qdb_cluster': [35492]} 2024-03-06T07:20:56.637 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:20:56.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.739+0000 7f672f35c640 1 Processor -- start 2024-03-06T07:20:56.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.739+0000 7f672f35c640 1 -- start start 2024-03-06T07:20:56.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672f35c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67280c6ab0 0x7f67281529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672f35c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f67281561a0 0x7f67280a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672f35c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67280a9990 con 0x7f67280aa430 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672f35c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6728155d10 con 0x7f67281561a0 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672f35c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67281556c0 con 0x7f67280c6ab0 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672c8f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f67281561a0 0x7f67280a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672c8f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f67281561a0 0x7f67280a9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52242/0 (socket says 172.21.15.94:52242) 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672c8f7640 1 -- 172.21.15.94:0/3771202519 learned_addr learned my addr 172.21.15.94:0/3771202519 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3771202519 >> v2:172.21.15.145:3300/0 conn(0x7f67280c6ab0 0x7f67281529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672c8f7640 1 -- 172.21.15.94:0/3771202519 >> v2:172.21.15.145:3300/0 conn(0x7f67280c6ab0 msgr2=0x7f67281529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3771202519 >> v2:172.21.15.145:3300/0 conn(0x7f67280c6ab0 0x7f67281529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672c8f7640 1 -- 172.21.15.94:0/3771202519 >> v1:172.21.15.145:6789/0 conn(0x7f67280aa430 legacy=0x7f672814f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.740+0000 7f672c8f7640 1 -- 172.21.15.94:0/3771202519 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67281559a0 con 0x7f67281561a0 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3771202519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281561a0 0x7f67280a9ef0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f67280c74f0 tx=0x7f671802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f921cdfe508bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f671ffff640 1 -- 172.21.15.94:0/3771202519 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718035020 con 0x7f67281561a0 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f671ffff640 1 -- 172.21.15.94:0/3771202519 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6718031870 con 0x7f67281561a0 2024-03-06T07:20:56.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f671ffff640 1 -- 172.21.15.94:0/3771202519 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718031c40 con 0x7f67281561a0 2024-03-06T07:20:56.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f672f35c640 1 -- 172.21.15.94:0/3771202519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281561a0 msgr2=0x7f67280a9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f672f35c640 1 --2- 172.21.15.94:0/3771202519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281561a0 0x7f67280a9ef0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f67280c74f0 tx=0x7f671802b910 comp rx=0 tx=0).stop 2024-03-06T07:20:56.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f672f35c640 1 -- 172.21.15.94:0/3771202519 shutdown_connections 2024-03-06T07:20:56.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f672f35c640 1 --2- 172.21.15.94:0/3771202519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67281561a0 0x7f67280a9ef0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f672f35c640 1 --2- 172.21.15.94:0/3771202519 >> v2:172.21.15.145:3300/0 conn(0x7f67280c6ab0 0x7f67281529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.741+0000 7f672f35c640 1 -- 172.21.15.94:0/3771202519 >> 172.21.15.94:0/3771202519 conn(0x7f672806b1a0 msgr2=0x7f672806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.742+0000 7f672f35c640 1 -- 172.21.15.94:0/3771202519 shutdown_connections 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.742+0000 7f672f35c640 1 -- 172.21.15.94:0/3771202519 wait complete. 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 Processor -- start 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 -- start start 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67280aa430 0x7f672814c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67280c6ab0 0x7f672814b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67281561a0 0x7f67281544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672814f820 con 0x7f67280c6ab0 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67280c2a90 con 0x7f67281561a0 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672f35c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672814ebb0 con 0x7f67280aa430 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672c8f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67280c6ab0 0x7f672814b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672c8f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67280c6ab0 0x7f672814b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52254/0 (socket says 172.21.15.94:52254) 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.743+0000 7f672c8f7640 1 -- 172.21.15.94:0/3859245863 learned_addr learned my addr 172.21.15.94:0/3859245863 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67280aa430 0x7f672814c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672d8f9640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67281561a0 0x7f67281544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672c8f7640 1 -- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67280aa430 msgr2=0x7f672814c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67280aa430 0x7f672814c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672c8f7640 1 -- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67281561a0 msgr2=0x7f67281544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67281561a0 0x7f67281544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672c8f7640 1 -- 172.21.15.94:0/3859245863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67281e7480 con 0x7f67280c6ab0 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67280c6ab0 0x7f672814b470 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f671802b950 tx=0x7f67180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f671dffb640 1 -- 172.21.15.94:0/3859245863 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718042070 con 0x7f67280c6ab0 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f671dffb640 1 -- 172.21.15.94:0/3859245863 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6718030b80 con 0x7f67280c6ab0 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67281e7760 con 0x7f67280c6ab0 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f671dffb640 1 -- 172.21.15.94:0/3859245863 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718035040 con 0x7f67280c6ab0 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.744+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67281e7d20 con 0x7f67280c6ab0 2024-03-06T07:20:56.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.746+0000 7f671dffb640 1 -- 172.21.15.94:0/3859245863 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f671804d060 con 0x7f67280c6ab0 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.746+0000 7f671dffb640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f670c06b4d0 0x7f670c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.746+0000 7f671dffb640 1 -- 172.21.15.94:0/3859245863 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f67180ae170 con 0x7f67280c6ab0 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.746+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f670c06b4d0 0x7f670c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.747+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f670c06b4d0 0x7f670c06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6724009b40 tx=0x7f6724009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.747+0000 7f6732015740 1 Processor -- start 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.747+0000 7f6732015740 1 -- start start 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896ce8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896ce8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f6732015740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ce94b0 0x561896ce5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f6732015740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x561896c4f6b0 con 0x561896ce94b0 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561896bf8a60 con 0x561896ceb330 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561896c4b750 con 0x561896ceaf50 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ce94b0 0x561896ce5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ce94b0 0x561896ce5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52262/0 (socket says 172.21.15.94:52262) 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 -- 172.21.15.94:0/1353523295 learned_addr learned my addr 172.21.15.94:0/1353523295 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d0f8640 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896ce8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672c8f7640 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896ce8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 -- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 msgr2=0x561896ce8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896ce8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 -- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 msgr2=0x561896ce8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896ce8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 -- 172.21.15.94:0/1353523295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561896c16f30 con 0x561896ce94b0 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.748+0000 7f672d8f9640 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ce94b0 0x561896ce5680 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6720001e30 tx=0x7f67200015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6043becf7e3abfd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6712ffd640 1 -- 172.21.15.94:0/1353523295 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672000fc40 con 0x561896ce94b0 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6712ffd640 1 -- 172.21.15.94:0/1353523295 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720018070 con 0x561896ce94b0 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6712ffd640 1 -- 172.21.15.94:0/1353523295 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720003970 con 0x561896ce94b0 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 -- 172.21.15.94:0/1353523295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ce94b0 msgr2=0x561896ce5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ce94b0 0x561896ce5680 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6720001e30 tx=0x7f67200015d0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 -- 172.21.15.94:0/1353523295 shutdown_connections 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ce94b0 0x561896ce5680 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896ce8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 --2- 172.21.15.94:0/1353523295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896ce8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 -- 172.21.15.94:0/1353523295 >> 172.21.15.94:0/1353523295 conn(0x561896ce77d0 msgr2=0x561896d0c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 -- 172.21.15.94:0/1353523295 shutdown_connections 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.749+0000 7f6732015740 1 -- 172.21.15.94:0/1353523295 wait complete. 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.750+0000 7f6732015740 1 -- start start 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.750+0000 7f6732015740 1 Processor -- start 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d0e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896d0b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d06fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x561896d0d230 con 0x561896d11e20 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561896c0bea0 con 0x561896ceb330 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561896d0cc70 con 0x561896ceaf50 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 10 client.0 _ll_register_callbacks cb 0x561896c0b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d06fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d06fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52268/0 (socket says 172.21.15.94:52268) 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 -- 172.21.15.94:0/1848778948 learned_addr learned my addr 172.21.15.94:0/1848778948 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672c8f7640 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896d0b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d0f8640 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d0e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 -- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 msgr2=0x561896d0e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d0e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 -- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 msgr2=0x561896d0b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896d0b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 -- 172.21.15.94:0/1848778948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561896c0c270 con 0x561896d11e20 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f672d8f9640 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d06fb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6720014000 tx=0x7f6720001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd687419f39e511d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6707fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6707fff640 1 -- 172.21.15.94:0/1848778948 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67200055a0 con 0x561896d11e20 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6707fff640 1 -- 172.21.15.94:0/1848778948 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6720005740 con 0x561896d11e20 2024-03-06T07:20:56.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6720018070 con 0x561896d11e20 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6707fff640 1 -- 172.21.15.94:0/1848778948 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f672002f050 con 0x561896d11e20 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 10 client.6450 fetch_fsmap learned FSMap version 55 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.751+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x5618969642d0 con 0x561896d11e20 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6707fff640 1 -- 172.21.15.94:0/1848778948 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720018070 con 0x561896d11e20 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6707fff640 1 -- 172.21.15.94:0/1848778948 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 55) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6720032020 con 0x561896d11e20 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6732015740 10 client.6450 fetch_fsmap finished waiting for FSMap version 55 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6732015740 10 client.6450 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6732015740 20 client.6450 populate_metadata read hostname 'smithi094' 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6732015740 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d380b0 0x561896d464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6732015740 4 client.6450 mds_command: new command op to 35492 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 --> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561896c5cbf0 con 0x561896d380b0 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f672d0f8640 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d380b0 0x561896d464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f6707fff640 10 client.6450 ms_handle_connect on v2:172.21.15.145:6836/3537750458 2024-03-06T07:20:56.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.752+0000 7f672d0f8640 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d380b0 0x561896d464f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aef37e242bf1a30d server_cookie=b1fc23a17920b4f1 in_seq=0 out_seq=0 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6707fff640 1 -- 172.21.15.94:0/1848778948 <== mds.0 v2:172.21.15.145:6836/3537750458 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561896c5cbf0 con 0x561896d380b0 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6707fff640 10 client.6450 handle_command_reply: tid=0 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 client.6450 shutdown 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 msgr2=0x561896d06fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d06fb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6720014000 tx=0x7f6720001250 comp rx=0 tx=0).stop 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 shutdown_connections 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d380b0 0x561896d464f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d06fb0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceb330 0x561896d0b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6707fff640 0 client.6450 ms_handle_reset on v2:172.21.15.145:6836/3537750458 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 --2- 172.21.15.94:0/1848778948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d0e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 >> 172.21.15.94:0/1848778948 conn(0x561896ce94b0 msgr2=0x561896ce98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 shutdown_connections 2024-03-06T07:20:56.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 1 -- 172.21.15.94:0/1848778948 wait complete. 2024-03-06T07:20:56.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.753+0000 7f6732015740 20 client.6450 trim_cache size 0 max 16384 2024-03-06T07:20:56.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.770+0000 7f6732015740 1 Processor -- start 2024-03-06T07:20:56.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.770+0000 7f6732015740 1 -- start start 2024-03-06T07:20:56.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceaf50 0x561896d4b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceb330 0x561896d3ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f6732015740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d33a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f6732015740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67281eb520 con 0x561896d11e20 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67281ea920 con 0x561896ceaf50 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67281eaf20 con 0x561896ceb330 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d33a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d33a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52278/0 (socket says 172.21.15.94:52278) 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 -- 172.21.15.94:0/3535191833 learned_addr learned my addr 172.21.15.94:0/3535191833 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceaf50 0x561896d4b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceb330 0x561896d3ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 -- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceb330 msgr2=0x561896d3ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceb330 0x561896d3ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 -- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceaf50 msgr2=0x561896d4b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceaf50 0x561896d4b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 -- 172.21.15.94:0/3535191833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561896a2e6b0 con 0x561896d11e20 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceb330 0x561896d3ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f672d8f9640 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d33a20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f6720000f80 tx=0x7f6720020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2489871ef01cf24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f67127fc640 1 -- 172.21.15.94:0/3535191833 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720038070 con 0x561896d11e20 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f67127fc640 1 -- 172.21.15.94:0/3535191833 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720018070 con 0x561896d11e20 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.771+0000 7f67127fc640 1 -- 172.21.15.94:0/3535191833 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720027690 con 0x561896d11e20 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 -- 172.21.15.94:0/3535191833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 msgr2=0x561896d33a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d33a20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f6720000f80 tx=0x7f6720020070 comp rx=0 tx=0).stop 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 -- 172.21.15.94:0/3535191833 shutdown_connections 2024-03-06T07:20:56.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896d11e20 0x561896d33a20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceb330 0x561896d3ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 --2- 172.21.15.94:0/3535191833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896ceaf50 0x561896d4b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 -- 172.21.15.94:0/3535191833 >> 172.21.15.94:0/3535191833 conn(0x561896ce94b0 msgr2=0x561896d0af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 -- 172.21.15.94:0/3535191833 shutdown_connections 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 -- 172.21.15.94:0/3535191833 wait complete. 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.772+0000 7f6732015740 1 -- start start 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 1 Processor -- start 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d09850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ceb330 0x561896ce57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896d11e20 0x561896d9a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x561896d0bbd0 con 0x561896ceb330 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561896d0a970 con 0x561896d11e20 2024-03-06T07:20:56.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561896d077a0 con 0x561896ceaf50 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.773+0000 7f6732015740 10 client.0 _ll_register_callbacks cb 0x7f6720001cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ceb330 0x561896ce57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ceb330 0x561896ce57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52288/0 (socket says 172.21.15.94:52288) 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 -- 172.21.15.94:0/3711708872 learned_addr learned my addr 172.21.15.94:0/3711708872 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d09850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672d8f9640 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896d11e20 0x561896d9a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 -- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 msgr2=0x561896d09850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d09850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 -- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896d11e20 msgr2=0x561896d9a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896d11e20 0x561896d9a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 -- 172.21.15.94:0/3711708872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561896d085c0 con 0x561896ceb330 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f672c8f7640 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ceb330 0x561896ce57d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f67180ac500 tx=0x7f67180ae040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fb939eb9456b46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6707fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:20:56.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6707fff640 1 -- 172.21.15.94:0/3711708872 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67180a5e60 con 0x561896ceb330 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6707fff640 1 -- 172.21.15.94:0/3711708872 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6718094250 con 0x561896ceb330 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6720018070 con 0x561896ceb330 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6707fff640 1 -- 172.21.15.94:0/3711708872 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6718084e60 con 0x561896ceb330 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 10 client.6456 fetch_fsmap learned FSMap version 55 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x561896d3b700 con 0x561896ceb330 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6707fff640 1 -- 172.21.15.94:0/3711708872 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720018070 con 0x561896ceb330 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6707fff640 1 -- 172.21.15.94:0/3711708872 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 55) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f671807bc50 con 0x561896ceb330 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 10 client.6456 fetch_fsmap finished waiting for FSMap version 55 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 10 client.6456 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 20 client.6456 populate_metadata read hostname 'smithi094' 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d4a510 0x561896d4a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.774+0000 7f6732015740 4 client.6456 mds_command: new command op to 35492 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:20:56.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 --> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561896c56f40 con 0x561896d4a510 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d4a510 0x561896d4a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6707fff640 10 client.6456 ms_handle_connect on v2:172.21.15.145:6836/3537750458 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f672d0f8640 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d4a510 0x561896d4a8f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4b3581161eade0b server_cookie=d8786ad3317c8a2c in_seq=0 out_seq=0 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6707fff640 1 -- 172.21.15.94:0/3711708872 <== mds.0 v2:172.21.15.145:6836/3537750458 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x561896c56f40 con 0x561896d4a510 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6707fff640 10 client.6456 handle_command_reply: tid=0 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 client.6456 shutdown 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ceb330 msgr2=0x561896ce57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ceb330 0x561896ce57d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f67180ac500 tx=0x7f67180ae040 comp rx=0 tx=0).stop 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 shutdown_connections 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:6836/3537750458,v1:172.21.15.145:6837/3537750458] conn(0x561896d4a510 0x561896d4a8f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561896d11e20 0x561896d9a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561896ceb330 0x561896ce57d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 --2- 172.21.15.94:0/3711708872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561896ceaf50 0x561896d09850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6707fff640 0 client.6456 ms_handle_reset on v2:172.21.15.145:6836/3537750458 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 >> 172.21.15.94:0/3711708872 conn(0x561896ce94b0 msgr2=0x561896d06d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.775+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 shutdown_connections 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f6732015740 1 -- 172.21.15.94:0/3711708872 wait complete. 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f6732015740 20 client.6456 trim_cache size 0 max 16384 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 35492, 2024-03-06T07:20:56.785 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.9079201000000001, 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 55, 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 74, 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 74, 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 10.046882120999999 2024-03-06T07:20:56.786 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f670c06b4d0 msgr2=0x7f670c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f670c06b4d0 0x7f670c06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6724009b40 tx=0x7f6724009190 comp rx=0 tx=0).stop 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67280c6ab0 msgr2=0x7f672814b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67280c6ab0 0x7f672814b470 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f671802b950 tx=0x7f67180046c0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 shutdown_connections 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f670c06b4d0 0x7f670c06d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67281561a0 0x7f67281544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67280c6ab0 0x7f672814b470 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 --2- 172.21.15.94:0/3859245863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67280aa430 0x7f672814c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 >> 172.21.15.94:0/3859245863 conn(0x7f672806b1a0 msgr2=0x7f67280ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 shutdown_connections 2024-03-06T07:20:56.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.776+0000 7f672f35c640 1 -- 172.21.15.94:0/3859245863 wait complete. 2024-03-06T07:20:56.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:20:56.850 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:20:56.848+0000 7f66a1e14640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:20:56.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.887+0000 7f00f6c6f640 1 Processor -- start 2024-03-06T07:20:56.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.887+0000 7f00f6c6f640 1 -- start start 2024-03-06T07:20:56.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f6c6f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f00f014f790 0x7f00f01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f6c6f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f6c6f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f00f00c5c60 0x7f00f0155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f6c6f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f00c2680 con 0x7f00f00c5c60 2024-03-06T07:20:56.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f6c6f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f00c28d0 con 0x7f00f014f790 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f6c6f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f00c2bb0 con 0x7f00f00c1aa0 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f00f00c5c60 0x7f00f0155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f00f00c5c60 0x7f00f0155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52306/0 (socket says 172.21.15.94:52306) 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 -- 172.21.15.94:0/2273530651 learned_addr learned my addr 172.21.15.94:0/2273530651 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f4a0b640 1 --2- 172.21.15.94:0/2273530651 >> v2:172.21.15.145:3300/0 conn(0x7f00f014f790 0x7f00f01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00e7fff640 1 --2- 172.21.15.94:0/2273530651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 -- 172.21.15.94:0/2273530651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 msgr2=0x7f00f00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 --2- 172.21.15.94:0/2273530651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f00c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 -- 172.21.15.94:0/2273530651 >> v2:172.21.15.145:3300/0 conn(0x7f00f014f790 msgr2=0x7f00f01529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 --2- 172.21.15.94:0/2273530651 >> v2:172.21.15.145:3300/0 conn(0x7f00f014f790 0x7f00f01529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.888+0000 7f00f520c640 1 -- 172.21.15.94:0/2273530651 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f00c2e90 con 0x7f00f00c5c60 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f520c640 1 --2- 172.21.15.94:0/2273530651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f0155a40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f00f00c3170 tx=0x7f00e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ad402c41edc6d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00e77fe640 1 -- 172.21.15.94:0/2273530651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e802ca70 con 0x7f00f00c5c60 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00e77fe640 1 -- 172.21.15.94:0/2273530651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e802cc10 con 0x7f00f00c5c60 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00e77fe640 1 -- 172.21.15.94:0/2273530651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e80343f0 con 0x7f00f00c5c60 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f6c6f640 1 -- 172.21.15.94:0/2273530651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 msgr2=0x7f00f0155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/2273530651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f0155a40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f00f00c3170 tx=0x7f00e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f6c6f640 1 -- 172.21.15.94:0/2273530651 shutdown_connections 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/2273530651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f0155a40 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/2273530651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f00c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/2273530651 >> v2:172.21.15.145:3300/0 conn(0x7f00f014f790 0x7f00f01529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.889+0000 7f00f6c6f640 1 -- 172.21.15.94:0/2273530651 >> 172.21.15.94:0/2273530651 conn(0x7f00f00c6710 msgr2=0x7f00f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.890+0000 7f00f6c6f640 1 -- 172.21.15.94:0/2273530651 shutdown_connections 2024-03-06T07:20:56.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.890+0000 7f00f6c6f640 1 -- 172.21.15.94:0/2273530651 wait complete. 2024-03-06T07:20:56.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 Processor -- start 2024-03-06T07:20:56.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 -- start start 2024-03-06T07:20:56.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f01eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00f014f790 0x7f00f01eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f006cc00 con 0x7f00f00c5c60 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f00c0d60 con 0x7f00f014f790 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.891+0000 7f00f6c6f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f006cd80 con 0x7f00f00c1aa0 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f01eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f01eacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52314/0 (socket says 172.21.15.94:52314) 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 -- 172.21.15.94:0/3345624913 learned_addr learned my addr 172.21.15.94:0/3345624913 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00f520c640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00f014f790 0x7f00f01eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00f4a0b640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 -- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 msgr2=0x7f00f006dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f006dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 -- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00f014f790 msgr2=0x7f00f01eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00f014f790 0x7f00f01eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:56.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 -- 172.21.15.94:0/3345624913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f00bc500 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e7fff640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f01eacf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f00e000e890 tx=0x7f00e000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e57fa640 1 -- 172.21.15.94:0/3345624913 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e0017070 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e57fa640 1 -- 172.21.15.94:0/3345624913 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e000c940 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f01ee400 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00e57fa640 1 -- 172.21.15.94:0/3345624913 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e000cd10 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.892+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f006e2e0 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.893+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f004f330 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.894+0000 7f00e57fa640 1 -- 172.21.15.94:0/3345624913 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00e001a070 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.894+0000 7f00e57fa640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f00dc06b4d0 0x7f00dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.894+0000 7f00e57fa640 1 -- 172.21.15.94:0/3345624913 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f00e008a570 con 0x7f00f00c5c60 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.894+0000 7f00f4a0b640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f00dc06b4d0 0x7f00dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:56.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.895+0000 7f00f4a0b640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f00dc06b4d0 0x7f00dc06d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009aa0 tx=0x7f00ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:56.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:56.898+0000 7f00e57fa640 1 -- 172.21.15.94:0/3345624913 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00e0058ca0 con 0x7f00f00c5c60 2024-03-06T07:20:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.104+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00f00c1f40 con 0x7f00f00c5c60 2024-03-06T07:20:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.104+0000 7f00e57fa640 1 -- 172.21.15.94:0/3345624913 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f00e005cd40 con 0x7f00f00c5c60 2024-03-06T07:20:57.108 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:57.108 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":35486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:20:53.531385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35492":{"gid":35492,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3537750458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3537750458},{"type":"v1","addr":"172.21.15.145:6837","nonce":3537750458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35492,"qdb_cluster":[35492]},"id":5}]} 2024-03-06T07:20:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 55 2024-03-06T07:20:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f00dc06b4d0 msgr2=0x7f00dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f00dc06b4d0 0x7f00dc06d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009aa0 tx=0x7f00ec009190 comp rx=0 tx=0).stop 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 msgr2=0x7f00f01eacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f01eacf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f00e000e890 tx=0x7f00e000ed50 comp rx=0 tx=0).stop 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 shutdown_connections 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f00dc06b4d0 0x7f00dc06d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00f014f790 0x7f00f01eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00f00c5c60 0x7f00f01eacf0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 --2- 172.21.15.94:0/3345624913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00f00c1aa0 0x7f00f006dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 >> 172.21.15.94:0/3345624913 conn(0x7f00f00c6710 msgr2=0x7f00f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.106+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 shutdown_connections 2024-03-06T07:20:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.107+0000 7f00f6c6f640 1 -- 172.21.15.94:0/3345624913 wait complete. 2024-03-06T07:20:57.119 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 5, "epoch": 55, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 5, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:20:51.223307+0000", "damaged": [], "data_pools": [ 12 ], "enabled": true, "epoch": 55, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_35492": { "addr": "172.21.15.145:6837/3537750458", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6836", "nonce": 3537750458, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 3537750458, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35492, "incarnation": 52, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 11, "modified": "2024-03-06T07:20:53.531385+0000", "qdb_cluster": [ 35492 ], "qdb_leader": 35492, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 35492 } } } ], "standbys": [ { "addr": "172.21.15.94:6834/3391508984", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3391508984, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 3391508984, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6318, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3556888574", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3556888574, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3556888574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6321, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6835/3330183565", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 3330183565, "type": "v2" }, { "addr": "172.21.15.145:6835", "nonce": 3330183565, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35486, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:20:57.120 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:20:57.120 INFO:tasks.cephfs.test_failover:current = [{'gid': 6318, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/3391508984', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3391508984}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 3391508984}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 50}, {'gid': 35492, 'name': 'b', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/3537750458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 3537750458}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3537750458}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6321, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3556888574', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3556888574}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3556888574}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 50}, {'gid': 35486, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6835/3330183565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3330183565}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 3330183565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 50}] 2024-03-06T07:20:57.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs2 2024-03-06T07:20:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.223+0000 7f5a4dd7a640 1 Processor -- start 2024-03-06T07:20:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.223+0000 7f5a4dd7a640 1 -- start start 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a4dd7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a4806b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a4dd7a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a480a9530 con 0x7f5a480c25c0 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a4dd7a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a480a9810 con 0x7f5a480c20b0 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a4dd7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480a9af0 con 0x7f5a4806afc0 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a47fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5a480c25c0 0x7f5a48153890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59680/0 (socket says 172.21.15.94:59680) 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a47fff640 1 -- 172.21.15.94:0/1307601060 learned_addr learned my addr 172.21.15.94:0/1307601060 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a477fe640 1 --2- 172.21.15.94:0/1307601060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a4806b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a467fc640 1 -- 172.21.15.94:0/1307601060 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908942198 0 0) 0x7f5a480a9530 con 0x7f5a480c25c0 2024-03-06T07:20:57.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a467fc640 1 -- 172.21.15.94:0/1307601060 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a30003520 con 0x7f5a480c25c0 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a467fc640 1 -- 172.21.15.94:0/1307601060 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862843268 0 0) 0x7f5a480a9810 con 0x7f5a480c20b0 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a467fc640 1 -- 172.21.15.94:0/1307601060 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a480a9530 con 0x7f5a480c20b0 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a477fe640 1 -- 172.21.15.94:0/1307601060 >> v1:172.21.15.145:6789/0 conn(0x7f5a480c20b0 legacy=0x7f5a48153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a477fe640 1 -- 172.21.15.94:0/1307601060 >> v1:172.21.15.94:6789/0 conn(0x7f5a480c25c0 legacy=0x7f5a48153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.224+0000 7f5a477fe640 1 -- 172.21.15.94:0/1307601060 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a480a9dd0 con 0x7f5a4806afc0 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.225+0000 7f5a477fe640 1 --2- 172.21.15.94:0/1307601060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a4806b3a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5a380097d0 tx=0x7f5a3802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e365be96cbf3d664 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.225+0000 7f5a467fc640 1 -- 172.21.15.94:0/1307601060 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38031770 con 0x7f5a4806afc0 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.225+0000 7f5a467fc640 1 -- 172.21.15.94:0/1307601060 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a38037070 con 0x7f5a4806afc0 2024-03-06T07:20:57.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.225+0000 7f5a467fc640 1 -- 172.21.15.94:0/1307601060 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38031b30 con 0x7f5a4806afc0 2024-03-06T07:20:57.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.226+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1307601060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 msgr2=0x7f5a4806b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.226+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1307601060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a4806b3a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5a380097d0 tx=0x7f5a3802b810 comp rx=0 tx=0).stop 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.226+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1307601060 shutdown_connections 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.226+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1307601060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a4806b3a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.226+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1307601060 >> 172.21.15.94:0/1307601060 conn(0x7f5a480c76b0 msgr2=0x7f5a480c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.226+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1307601060 shutdown_connections 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.226+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1307601060 wait complete. 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.227+0000 7f5a4dd7a640 1 Processor -- start 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.227+0000 7f5a4dd7a640 1 -- start start 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a4dd7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a480c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a4dd7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a480c20b0 0x7f5a48147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a4dd7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a480c25c0 0x7f5a48147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a4dd7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48150b00 con 0x7f5a480c20b0 2024-03-06T07:20:57.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a4dd7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a48150980 con 0x7f5a480c25c0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a4dd7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a480a7d50 con 0x7f5a4806afc0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a480c20b0 0x7f5a48147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a480c20b0 0x7f5a48147950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52328/0 (socket says 172.21.15.94:52328) 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 -- 172.21.15.94:0/1573367495 learned_addr learned my addr 172.21.15.94:0/1573367495 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a47fff640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a480c25c0 0x7f5a48147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a477fe640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a480c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 -- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 msgr2=0x7f5a480c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a480c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 -- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a480c25c0 msgr2=0x7f5a48147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a480c25c0 0x7f5a48147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 -- 172.21.15.94:0/1573367495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a48072860 con 0x7f5a480c20b0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a46ffd640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a480c20b0 0x7f5a48147950 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00c400 tx=0x7f5a3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a2ffff640 1 -- 172.21.15.94:0/1573367495 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c017070 con 0x7f5a480c20b0 2024-03-06T07:20:57.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a2ffff640 1 -- 172.21.15.94:0/1573367495 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c0052d0 con 0x7f5a480c20b0 2024-03-06T07:20:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.228+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a48072ad0 con 0x7f5a480c20b0 2024-03-06T07:20:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.229+0000 7f5a2ffff640 1 -- 172.21.15.94:0/1573367495 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c003810 con 0x7f5a480c20b0 2024-03-06T07:20:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.229+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a48073090 con 0x7f5a480c20b0 2024-03-06T07:20:57.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.230+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4804f330 con 0x7f5a480c20b0 2024-03-06T07:20:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.230+0000 7f5a2ffff640 1 -- 172.21.15.94:0/1573367495 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a3c01a070 con 0x7f5a480c20b0 2024-03-06T07:20:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.230+0000 7f5a2ffff640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a3406b4d0 0x7f5a3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.230+0000 7f5a2ffff640 1 -- 172.21.15.94:0/1573367495 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5a3c089690 con 0x7f5a480c20b0 2024-03-06T07:20:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.230+0000 7f5a477fe640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a3406b4d0 0x7f5a3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.231+0000 7f5a477fe640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a3406b4d0 0x7f5a3406d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5a38036000 tx=0x7f5a3803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.235+0000 7f5a2ffff640 1 -- 172.21.15.94:0/1573367495 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a3c057dc0 con 0x7f5a480c20b0 2024-03-06T07:20:57.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.403+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f5a4806bc50 con 0x7f5a480c20b0 2024-03-06T07:20:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.406+0000 7f5a2ffff640 1 -- 172.21.15.94:0/1573367495 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v24)=0 v24) v1 ==== 117+0+0 (secure 0 0 0) 0x7f5a3c05be60 con 0x7f5a480c20b0 2024-03-06T07:20:57.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a3406b4d0 msgr2=0x7f5a3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a3406b4d0 0x7f5a3406d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5a38036000 tx=0x7f5a3803d040 comp rx=0 tx=0).stop 2024-03-06T07:20:57.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a480c20b0 msgr2=0x7f5a48147950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a480c20b0 0x7f5a48147950 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00c400 tx=0x7f5a3c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 shutdown_connections 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a3406b4d0 0x7f5a3406d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a480c25c0 0x7f5a48147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a480c20b0 0x7f5a48147950 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 --2- 172.21.15.94:0/1573367495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a4806afc0 0x7f5a480c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 >> 172.21.15.94:0/1573367495 conn(0x7f5a480c76b0 msgr2=0x7f5a4814d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 shutdown_connections 2024-03-06T07:20:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.408+0000 7f5a4dd7a640 1 -- 172.21.15.94:0/1573367495 wait complete. 2024-03-06T07:20:57.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-03-06T07:20:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.522+0000 7f08c5d81640 1 Processor -- start 2024-03-06T07:20:57.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.522+0000 7f08c5d81640 1 -- start start 2024-03-06T07:20:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08c5d81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c00c7370 0x7f08c00c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08c5d81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f08c006c0b0 0x7f08c006c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08c5d81640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c0155260 con 0x7f08c01527f0 2024-03-06T07:20:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08c5d81640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08c00c8f70 con 0x7f08c006c0b0 2024-03-06T07:20:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08c5d81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c0155540 con 0x7f08c00c7370 2024-03-06T07:20:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08bffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f08c006c0b0 0x7f08c006c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08bf7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f08c01527f0 0x7f08c00c8860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59692/0 (socket says 172.21.15.94:59692) 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08bffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f08c006c0b0 0x7f08c006c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52340/0 (socket says 172.21.15.94:52340) 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08bf7fe640 1 -- 172.21.15.94:0/1382439264 learned_addr learned my addr 172.21.15.94:0/1382439264 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.523+0000 7f08b6ffd640 1 --2- 172.21.15.94:0/1382439264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c00c7370 0x7f08c00c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.524+0000 7f08bffff640 1 -- 172.21.15.94:0/1382439264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c00c7370 msgr2=0x7f08c00c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.524+0000 7f08bffff640 1 --2- 172.21.15.94:0/1382439264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c00c7370 0x7f08c00c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.524+0000 7f08bffff640 1 -- 172.21.15.94:0/1382439264 >> v1:172.21.15.94:6789/0 conn(0x7f08c01527f0 legacy=0x7f08c00c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.524+0000 7f08bffff640 1 -- 172.21.15.94:0/1382439264 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0155820 con 0x7f08c006c0b0 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.524+0000 7f08beffd640 1 -- 172.21.15.94:0/1382439264 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439691167 0 0) 0x7f08c0155260 con 0x7f08c01527f0 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.524+0000 7f08bffff640 1 --2- 172.21.15.94:0/1382439264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c006c0b0 0x7f08c006c4e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f08c00c90f0 tx=0x7f08b802bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a543218a1f940699 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.524+0000 7f08beffd640 1 -- 172.21.15.94:0/1382439264 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8039070 con 0x7f08c006c0b0 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08beffd640 1 -- 172.21.15.94:0/1382439264 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b802cb20 con 0x7f08c006c0b0 2024-03-06T07:20:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08beffd640 1 -- 172.21.15.94:0/1382439264 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8034440 con 0x7f08c006c0b0 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 -- 172.21.15.94:0/1382439264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c006c0b0 msgr2=0x7f08c006c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 --2- 172.21.15.94:0/1382439264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c006c0b0 0x7f08c006c4e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f08c00c90f0 tx=0x7f08b802bd70 comp rx=0 tx=0).stop 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 -- 172.21.15.94:0/1382439264 shutdown_connections 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 --2- 172.21.15.94:0/1382439264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c006c0b0 0x7f08c006c4e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 --2- 172.21.15.94:0/1382439264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c00c7370 0x7f08c00c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 -- 172.21.15.94:0/1382439264 >> 172.21.15.94:0/1382439264 conn(0x7f08c00ac4f0 msgr2=0x7f08c00ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 -- 172.21.15.94:0/1382439264 shutdown_connections 2024-03-06T07:20:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.525+0000 7f08c5d81640 1 -- 172.21.15.94:0/1382439264 wait complete. 2024-03-06T07:20:57.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.526+0000 7f08c5d81640 1 Processor -- start 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.526+0000 7f08c5d81640 1 -- start start 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08c5d81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c006c0b0 0x7f08c014ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08c5d81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08c00c7370 0x7f08c014d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08c5d81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c01527f0 0x7f08c01ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08c5d81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c006ca20 con 0x7f08c01527f0 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08c5d81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c00be1a0 con 0x7f08c00c7370 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08c5d81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c006cba0 con 0x7f08c006c0b0 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c01527f0 0x7f08c01ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c01527f0 0x7f08c01ee4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52352/0 (socket says 172.21.15.94:52352) 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 -- 172.21.15.94:0/713928735 learned_addr learned my addr 172.21.15.94:0/713928735 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bf7fe640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c006c0b0 0x7f08c014ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08b6ffd640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08c00c7370 0x7f08c014d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 -- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c006c0b0 msgr2=0x7f08c014ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c006c0b0 0x7f08c014ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 -- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08c00c7370 msgr2=0x7f08c014d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08c00c7370 0x7f08c014d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.527+0000 7f08bffff640 1 -- 172.21.15.94:0/713928735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c014fab0 con 0x7f08c01527f0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.528+0000 7f08bffff640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c01527f0 0x7f08c01ee4a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009e80 tx=0x7f08b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.528+0000 7f08bcff9640 1 -- 172.21.15.94:0/713928735 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8039050 con 0x7f08c01527f0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.528+0000 7f08bcff9640 1 -- 172.21.15.94:0/713928735 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b8045070 con 0x7f08c01527f0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.528+0000 7f08bcff9640 1 -- 172.21.15.94:0/713928735 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8009450 con 0x7f08c01527f0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.528+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c014d0f0 con 0x7f08c01527f0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.528+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c014f7d0 con 0x7f08c01527f0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.529+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c004f330 con 0x7f08c01527f0 2024-03-06T07:20:57.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.529+0000 7f08bcff9640 1 -- 172.21.15.94:0/713928735 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08b803e850 con 0x7f08c01527f0 2024-03-06T07:20:57.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.530+0000 7f08bcff9640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.530+0000 7f08bcff9640 1 -- 172.21.15.94:0/713928735 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f08b80ad190 con 0x7f08c01527f0 2024-03-06T07:20:57.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.530+0000 7f08bf7fe640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.530+0000 7f08bf7fe640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f08b0000f60 tx=0x7f08b0007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.534+0000 7f08bcff9640 1 -- 172.21.15.94:0/713928735 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08b807b9e0 con 0x7f08c01527f0 2024-03-06T07:20:57.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.704+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f08c014c350 con 0x7f08c01527f0 2024-03-06T07:20:57.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.706+0000 7f08bcff9640 1 -- 172.21.15.94:0/713928735 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v25)=0 v25) v1 ==== 117+0+0 (secure 0 0 0) 0x7f08b807fa80 con 0x7f08c01527f0 2024-03-06T07:20:57.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f08a006b4d0 msgr2=0x7f08a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f08a006b4d0 0x7f08a006d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f08b0000f60 tx=0x7f08b0007d50 comp rx=0 tx=0).stop 2024-03-06T07:20:57.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c01527f0 msgr2=0x7f08c01ee4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c01527f0 0x7f08c01ee4a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009e80 tx=0x7f08b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:57.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 shutdown_connections 2024-03-06T07:20:57.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f08a006b4d0 0x7f08a006d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08c01527f0 0x7f08c01ee4a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08c00c7370 0x7f08c014d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 --2- 172.21.15.94:0/713928735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08c006c0b0 0x7f08c014ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 >> 172.21.15.94:0/713928735 conn(0x7f08c00ac4f0 msgr2=0x7f08c00c9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 shutdown_connections 2024-03-06T07:20:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.709+0000 7f08c5d81640 1 -- 172.21.15.94:0/713928735 wait complete. 2024-03-06T07:20:57.721 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-03-06T07:20:57.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.822+0000 7fcd04d8a640 1 Processor -- start 2024-03-06T07:20:57.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.822+0000 7fcd04d8a640 1 -- start start 2024-03-06T07:20:57.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fcd04d8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd00072e70 0x7fcd000bd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fcd04d8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcd0006e490 0x7fcd0015fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fcd04d8a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0006c8e0 con 0x7fcd00073250 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fcd04d8a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0006ce80 con 0x7fcd0006e490 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fcd04d8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000c1b00 con 0x7fcd00072e70 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcd0006e490 0x7fcd0015fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcd0006e490 0x7fcd0015fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52360/0 (socket says 172.21.15.94:52360) 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfed9d640 1 -- 172.21.15.94:0/1976681848 learned_addr learned my addr 172.21.15.94:0/1976681848 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfe59c640 1 --2- 172.21.15.94:0/1976681848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd00072e70 0x7fcd000bd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfed9d640 1 -- 172.21.15.94:0/1976681848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd00072e70 msgr2=0x7fcd000bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfed9d640 1 --2- 172.21.15.94:0/1976681848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd00072e70 0x7fcd000bd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfed9d640 1 -- 172.21.15.94:0/1976681848 >> v1:172.21.15.145:6789/0 conn(0x7fcd00073250 legacy=0x7fcd000c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfdd9b640 1 -- 172.21.15.94:0/1976681848 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497982296 0 0) 0x7fcd0006c8e0 con 0x7fcd00073250 2024-03-06T07:20:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.823+0000 7fccfed9d640 1 -- 172.21.15.94:0/1976681848 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0006cba0 con 0x7fcd0006e490 2024-03-06T07:20:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fccfed9d640 1 --2- 172.21.15.94:0/1976681848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0006e490 0x7fcd0015fce0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009e50 tx=0x7fccf802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e87744fe3ad5290 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fccfdd9b640 1 -- 172.21.15.94:0/1976681848 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf802ca70 con 0x7fcd0006e490 2024-03-06T07:20:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fccfdd9b640 1 -- 172.21.15.94:0/1976681848 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf802cc10 con 0x7fcd0006e490 2024-03-06T07:20:57.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fccfdd9b640 1 -- 172.21.15.94:0/1976681848 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf80343f0 con 0x7fcd0006e490 2024-03-06T07:20:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fcd04d8a640 1 -- 172.21.15.94:0/1976681848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0006e490 msgr2=0x7fcd0015fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/1976681848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0006e490 0x7fcd0015fce0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009e50 tx=0x7fccf802bed0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fcd04d8a640 1 -- 172.21.15.94:0/1976681848 shutdown_connections 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/1976681848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd0006e490 0x7fcd0015fce0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.824+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/1976681848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd00072e70 0x7fcd000bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.825+0000 7fcd04d8a640 1 -- 172.21.15.94:0/1976681848 >> 172.21.15.94:0/1976681848 conn(0x7fcd0006b1a0 msgr2=0x7fcd0006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.825+0000 7fcd04d8a640 1 -- 172.21.15.94:0/1976681848 shutdown_connections 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.825+0000 7fcd04d8a640 1 -- 172.21.15.94:0/1976681848 wait complete. 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 Processor -- start 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 -- start start 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd0006e490 0x7fcd001eb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd00072e70 0x7fcd000c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd00073250 0x7fcd000bee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000c6760 con 0x7fcd00073250 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000c4a00 con 0x7fcd00072e70 2024-03-06T07:20:57.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.826+0000 7fcd04d8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd00151b60 con 0x7fcd0006e490 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd00073250 0x7fcd000bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd00073250 0x7fcd000bee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52368/0 (socket says 172.21.15.94:52368) 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 -- 172.21.15.94:0/159062016 learned_addr learned my addr 172.21.15.94:0/159062016 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfe59c640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd0006e490 0x7fcd001eb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccf7fff640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd00072e70 0x7fcd000c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 -- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd0006e490 msgr2=0x7fcd001eb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd0006e490 0x7fcd001eb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 -- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd00072e70 msgr2=0x7fcd000c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd00072e70 0x7fcd000c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 -- 172.21.15.94:0/159062016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd001ee060 con 0x7fcd00073250 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccfed9d640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd00073250 0x7fcd000bee10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009f70 tx=0x7fccf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccf6ffd640 1 -- 172.21.15.94:0/159062016 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8043020 con 0x7fcd00073250 2024-03-06T07:20:57.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccf6ffd640 1 -- 172.21.15.94:0/159062016 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf803e450 con 0x7fcd00073250 2024-03-06T07:20:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fccf6ffd640 1 -- 172.21.15.94:0/159062016 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8009510 con 0x7fcd00073250 2024-03-06T07:20:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd001ee300 con 0x7fcd00073250 2024-03-06T07:20:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.827+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd001eaee0 con 0x7fcd00073250 2024-03-06T07:20:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.828+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0004f330 con 0x7fcd00073250 2024-03-06T07:20:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.829+0000 7fccf6ffd640 1 -- 172.21.15.94:0/159062016 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccf8046020 con 0x7fcd00073250 2024-03-06T07:20:57.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.829+0000 7fccf6ffd640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcce006b500 0x7fcce006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:57.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.829+0000 7fccf6ffd640 1 -- 172.21.15.94:0/159062016 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fccf80ac900 con 0x7fcd00073250 2024-03-06T07:20:57.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.829+0000 7fccfe59c640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcce006b500 0x7fcce006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:57.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.830+0000 7fccfe59c640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcce006b500 0x7fcce006d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009b40 tx=0x7fccf0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:57.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:57.834+0000 7fccf6ffd640 1 -- 172.21.15.94:0/159062016 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccf804d030 con 0x7fcd00073250 2024-03-06T07:20:58.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.003+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fcd00068e90 con 0x7fcd00073250 2024-03-06T07:20:58.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.006+0000 7fccf6ffd640 1 -- 172.21.15.94:0/159062016 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v26)=0 v26) v1 ==== 117+0+0 (secure 0 0 0) 0x7fccf807afa0 con 0x7fcd00073250 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcce006b500 msgr2=0x7fcce006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcce006b500 0x7fcce006d9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009b40 tx=0x7fccf0009190 comp rx=0 tx=0).stop 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd00073250 msgr2=0x7fcd000bee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd00073250 0x7fcd000bee10 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009f70 tx=0x7fccf8004a40 comp rx=0 tx=0).stop 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 shutdown_connections 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcce006b500 0x7fcce006d9c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd00073250 0x7fcd000bee10 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd00072e70 0x7fcd000c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 --2- 172.21.15.94:0/159062016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd0006e490 0x7fcd001eb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 >> 172.21.15.94:0/159062016 conn(0x7fcd0006b1a0 msgr2=0x7fcd000bcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 shutdown_connections 2024-03-06T07:20:58.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.008+0000 7fcd04d8a640 1 -- 172.21.15.94:0/159062016 wait complete. 2024-03-06T07:20:58.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:20:58.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.119+0000 7f3ee33a0640 1 Processor -- start 2024-03-06T07:20:58.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.119+0000 7f3ee33a0640 1 -- start start 2024-03-06T07:20:58.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee33a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc1592d0 0x7f3edc1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee33a0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3edc0ac260 0x7f3edc155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee33a0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc152d40 con 0x7f3edc152960 2024-03-06T07:20:58.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee33a0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc066e30 con 0x7f3edc0ac260 2024-03-06T07:20:58.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee33a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc155d50 con 0x7f3edc1592d0 2024-03-06T07:20:58.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee093b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc1592d0 0x7f3edc1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee193d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3edc0ac260 0x7f3edc155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee113c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f3edc152960 0x7f3edc06dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:39050/0 (socket says 172.21.15.94:39050) 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee093b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc1592d0 0x7f3edc1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53678/0 (socket says 172.21.15.94:53678) 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee113c640 1 -- 172.21.15.94:0/205350862 learned_addr learned my addr 172.21.15.94:0/205350862 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee093b640 1 -- 172.21.15.94:0/205350862 >> v2:172.21.15.145:3300/0 conn(0x7f3edc0ac260 msgr2=0x7f3edc155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee093b640 1 --2- 172.21.15.94:0/205350862 >> v2:172.21.15.145:3300/0 conn(0x7f3edc0ac260 0x7f3edc155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee093b640 1 -- 172.21.15.94:0/205350862 >> v1:172.21.15.145:6789/0 conn(0x7f3edc152960 legacy=0x7f3edc06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.120+0000 7f3ee093b640 1 -- 172.21.15.94:0/205350862 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc0c1b50 con 0x7f3edc1592d0 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.121+0000 7f3ee093b640 1 --2- 172.21.15.94:0/205350862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc1592d0 0x7f3edc1552d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0097a0 tx=0x7f3ecc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d028bf9e85d8e553 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.121+0000 7f3ed3fff640 1 -- 172.21.15.94:0/205350862 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc035020 con 0x7f3edc1592d0 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.121+0000 7f3ed3fff640 1 -- 172.21.15.94:0/205350862 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc031830 con 0x7f3edc1592d0 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.121+0000 7f3ed3fff640 1 -- 172.21.15.94:0/205350862 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc031c00 con 0x7f3edc1592d0 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 -- 172.21.15.94:0/205350862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc1592d0 msgr2=0x7f3edc1552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/205350862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc1592d0 0x7f3edc1552d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0097a0 tx=0x7f3ecc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 -- 172.21.15.94:0/205350862 shutdown_connections 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/205350862 >> v2:172.21.15.145:3300/0 conn(0x7f3edc0ac260 0x7f3edc155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/205350862 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc1592d0 0x7f3edc1552d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 -- 172.21.15.94:0/205350862 >> 172.21.15.94:0/205350862 conn(0x7f3edc0c70b0 msgr2=0x7f3edc0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 -- 172.21.15.94:0/205350862 shutdown_connections 2024-03-06T07:20:58.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.122+0000 7f3ee33a0640 1 -- 172.21.15.94:0/205350862 wait complete. 2024-03-06T07:20:58.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.123+0000 7f3ee33a0640 1 Processor -- start 2024-03-06T07:20:58.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.123+0000 7f3ee33a0640 1 -- start start 2024-03-06T07:20:58.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee33a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3edc0ac260 0x7f3edc0727d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee33a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3edc152960 0x7f3edc1e2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee33a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc156620 0x7f3edc147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee33a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc159760 con 0x7f3edc0ac260 2024-03-06T07:20:58.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee33a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0672c0 con 0x7f3edc152960 2024-03-06T07:20:58.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee33a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc06e3a0 con 0x7f3edc156620 2024-03-06T07:20:58.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3edc0ac260 0x7f3edc0727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3edc0ac260 0x7f3edc0727d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52384/0 (socket says 172.21.15.94:52384) 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 -- 172.21.15.94:0/1430360048 learned_addr learned my addr 172.21.15.94:0/1430360048 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee093b640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3edc152960 0x7f3edc1e2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee193d640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc156620 0x7f3edc147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 -- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc156620 msgr2=0x7f3edc147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc156620 0x7f3edc147b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 -- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3edc152960 msgr2=0x7f3edc1e2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3edc152960 0x7f3edc1e2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 -- 172.21.15.94:0/1430360048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc1eeed0 con 0x7f3edc0ac260 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.124+0000 7f3ee113c640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3edc0ac260 0x7f3edc0727d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800cb10 tx=0x7f3ed800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.125+0000 7f3ed1ffb640 1 -- 172.21.15.94:0/1430360048 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8017070 con 0x7f3edc0ac260 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.125+0000 7f3ed1ffb640 1 -- 172.21.15.94:0/1430360048 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed800ac10 con 0x7f3edc0ac260 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.125+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc071cf0 con 0x7f3edc0ac260 2024-03-06T07:20:58.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.125+0000 7f3ed1ffb640 1 -- 172.21.15.94:0/1430360048 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8012400 con 0x7f3edc0ac260 2024-03-06T07:20:58.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.125+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc0722b0 con 0x7f3edc0ac260 2024-03-06T07:20:58.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.126+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc069460 con 0x7f3edc0ac260 2024-03-06T07:20:58.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.126+0000 7f3ed1ffb640 1 -- 172.21.15.94:0/1430360048 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ed801a070 con 0x7f3edc0ac260 2024-03-06T07:20:58.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.126+0000 7f3ed1ffb640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.126+0000 7f3ed1ffb640 1 -- 172.21.15.94:0/1430360048 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ed808a130 con 0x7f3edc0ac260 2024-03-06T07:20:58.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.126+0000 7f3ee093b640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.127+0000 7f3ee093b640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc036000 tx=0x7f3ecc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.131+0000 7f3ed1ffb640 1 -- 172.21.15.94:0/1430360048 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3edc069460 con 0x7f3edc0ac260 2024-03-06T07:20:58.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.302+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3edc071ae0 con 0x7f3edc0ac260 2024-03-06T07:20:58.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.303+0000 7f3ed1ffb640 1 -- 172.21.15.94:0/1430360048 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+16945 (secure 0 0 0) 0x7f3ed8058750 con 0x7f3edc0ac260 2024-03-06T07:20:58.305 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:58.306 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":74,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:20:52.881152+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:20:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 msgr2=0x7f3ec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc036000 tx=0x7f3ecc03c040 comp rx=0 tx=0).stop 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3edc0ac260 msgr2=0x7f3edc0727d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3edc0ac260 0x7f3edc0727d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800cb10 tx=0x7f3ed800a060 comp rx=0 tx=0).stop 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 shutdown_connections 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3edc156620 0x7f3edc147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3edc152960 0x7f3edc1e2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 --2- 172.21.15.94:0/1430360048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3edc0ac260 0x7f3edc0727d0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 >> 172.21.15.94:0/1430360048 conn(0x7f3edc0c70b0 msgr2=0x7f3edc0c3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 shutdown_connections 2024-03-06T07:20:58.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.305+0000 7f3ee33a0640 1 -- 172.21.15.94:0/1430360048 wait complete. 2024-03-06T07:20:58.318 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:49.440763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:50.185600+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:20:58.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:20:58.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.419+0000 7f0169247640 1 Processor -- start 2024-03-06T07:20:58.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.419+0000 7f0169247640 1 -- start start 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f0169247640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01641594f0 0x7f01640bc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f0169247640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f016406e400 0x7f01640c1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f0169247640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016406e970 con 0x7f01641527b0 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f0169247640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0164158a30 con 0x7f01641594f0 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f0169247640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0164158d10 con 0x7f016406e400 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f015bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01641594f0 0x7f01640bc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f015bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01641594f0 0x7f01640bc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38696/0 (socket says 172.21.15.94:38696) 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f015bfff640 1 -- 172.21.15.94:0/3244142468 learned_addr learned my addr 172.21.15.94:0/3244142468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f016359e640 1 --2- 172.21.15.94:0/3244142468 >> v2:172.21.15.145:3300/0 conn(0x7f016406e400 0x7f01640c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f016259c640 1 -- 172.21.15.94:0/3244142468 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068417685 0 0) 0x7f016406e970 con 0x7f01641527b0 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f016259c640 1 -- 172.21.15.94:0/3244142468 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0150003520 con 0x7f01641527b0 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f015bfff640 1 -- 172.21.15.94:0/3244142468 >> v2:172.21.15.145:3300/0 conn(0x7f016406e400 msgr2=0x7f01640c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f015bfff640 1 --2- 172.21.15.94:0/3244142468 >> v2:172.21.15.145:3300/0 conn(0x7f016406e400 0x7f01640c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f015bfff640 1 -- 172.21.15.94:0/3244142468 >> v1:172.21.15.145:6789/0 conn(0x7f01641527b0 legacy=0x7f016406cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.420+0000 7f015bfff640 1 -- 172.21.15.94:0/3244142468 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016406da70 con 0x7f01641594f0 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.421+0000 7f015bfff640 1 --2- 172.21.15.94:0/3244142468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640bc9e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f01640ac660 tx=0x7f014c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93aeaa02117e8d9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.421+0000 7f016259c640 1 -- 172.21.15.94:0/3244142468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c035020 con 0x7f01641594f0 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.421+0000 7f016259c640 1 -- 172.21.15.94:0/3244142468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c031870 con 0x7f01641594f0 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.421+0000 7f016259c640 1 -- 172.21.15.94:0/3244142468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c031c40 con 0x7f01641594f0 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 -- 172.21.15.94:0/3244142468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 msgr2=0x7f01640bc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 --2- 172.21.15.94:0/3244142468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640bc9e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f01640ac660 tx=0x7f014c02b910 comp rx=0 tx=0).stop 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 -- 172.21.15.94:0/3244142468 shutdown_connections 2024-03-06T07:20:58.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 --2- 172.21.15.94:0/3244142468 >> v2:172.21.15.145:3300/0 conn(0x7f016406e400 0x7f01640c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 --2- 172.21.15.94:0/3244142468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640bc9e0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 -- 172.21.15.94:0/3244142468 >> 172.21.15.94:0/3244142468 conn(0x7f016406afc0 msgr2=0x7f016406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 -- 172.21.15.94:0/3244142468 shutdown_connections 2024-03-06T07:20:58.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.422+0000 7f0169247640 1 -- 172.21.15.94:0/3244142468 wait complete. 2024-03-06T07:20:58.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.423+0000 7f0169247640 1 Processor -- start 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.423+0000 7f0169247640 1 -- start start 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0169247640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016406e400 0x7f0164149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0169247640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01641527b0 0x7f01640aa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0169247640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0169247640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0164152d10 con 0x7f01641594f0 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0169247640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0164152b90 con 0x7f016406e400 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0169247640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01640c1e80 con 0x7f01641527b0 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640a96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38698/0 (socket says 172.21.15.94:38698) 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 -- 172.21.15.94:0/3200809388 learned_addr learned my addr 172.21.15.94:0/3200809388 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:58.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f015bfff640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01641527b0 0x7f01640aa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0162d9d640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016406e400 0x7f0164149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 -- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01641527b0 msgr2=0x7f01640aa9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01641527b0 0x7f01640aa9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 -- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016406e400 msgr2=0x7f0164149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016406e400 0x7f0164149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 -- 172.21.15.94:0/3200809388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01640aaf00 con 0x7f01641594f0 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f016359e640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640a96c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f015c00efc0 tx=0x7f015c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f015b7fe640 1 -- 172.21.15.94:0/3200809388 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c018020 con 0x7f01641594f0 2024-03-06T07:20:58.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f015b7fe640 1 -- 172.21.15.94:0/3200809388 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015c014480 con 0x7f01641594f0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0164072730 con 0x7f01641594f0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.424+0000 7f015b7fe640 1 -- 172.21.15.94:0/3200809388 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c014870 con 0x7f01641594f0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.425+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01641ee710 con 0x7f01641594f0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.425+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0164152e90 con 0x7f01641594f0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.426+0000 7f015b7fe640 1 -- 172.21.15.94:0/3200809388 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f015c005d10 con 0x7f01641594f0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.426+0000 7f015b7fe640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f014406b4d0 0x7f014406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.426+0000 7f015b7fe640 1 -- 172.21.15.94:0/3200809388 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f015c08ac70 con 0x7f01641594f0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.426+0000 7f0162d9d640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f014406b4d0 0x7f014406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.427+0000 7f0162d9d640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f014406b4d0 0x7f014406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0154007fd0 tx=0x7f0154009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.431+0000 7f015b7fe640 1 -- 172.21.15.94:0/3200809388 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0164152e90 con 0x7f01641594f0 2024-03-06T07:20:58.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.603+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0164074ac0 con 0x7f01641594f0 2024-03-06T07:20:58.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.603+0000 7f015b7fe640 1 -- 172.21.15.94:0/3200809388 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0164074ac0 con 0x7f01641594f0 2024-03-06T07:20:58.605 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:20:58.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f014406b4d0 msgr2=0x7f014406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f014406b4d0 0x7f014406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0154007fd0 tx=0x7f0154009040 comp rx=0 tx=0).stop 2024-03-06T07:20:58.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 msgr2=0x7f01640a96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640a96c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f015c00efc0 tx=0x7f015c00c490 comp rx=0 tx=0).stop 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 shutdown_connections 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f014406b4d0 0x7f014406d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01641594f0 0x7f01640a96c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01641527b0 0x7f01640aa9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 --2- 172.21.15.94:0/3200809388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f016406e400 0x7f0164149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 >> 172.21.15.94:0/3200809388 conn(0x7f016406afc0 msgr2=0x7f01640c66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 shutdown_connections 2024-03-06T07:20:58.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.605+0000 7f0169247640 1 -- 172.21.15.94:0/3200809388 wait complete. 2024-03-06T07:20:58.618 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:20:58.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.719+0000 7f83674fa640 1 Processor -- start 2024-03-06T07:20:58.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.719+0000 7f83674fa640 1 -- start start 2024-03-06T07:20:58.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f83674fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600ab310 0x7f83600ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f83674fa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f83601538c0 0x7f83601494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f83674fa640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8360154330 con 0x7f83600a97c0 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f83674fa640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83600c46a0 con 0x7f83601538c0 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f83674fa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8360153dc0 con 0x7f83600ab310 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f8364a95640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f83601538c0 0x7f83601494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f8364a95640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f83601538c0 0x7f83601494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38714/0 (socket says 172.21.15.94:38714) 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f8364a95640 1 -- 172.21.15.94:0/4030795581 learned_addr learned my addr 172.21.15.94:0/4030795581 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.720+0000 7f8365296640 1 --2- 172.21.15.94:0/4030795581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600ab310 0x7f83600ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8364a95640 1 -- 172.21.15.94:0/4030795581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600ab310 msgr2=0x7f83600ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8364a95640 1 --2- 172.21.15.94:0/4030795581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600ab310 0x7f83600ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8364a95640 1 -- 172.21.15.94:0/4030795581 >> v1:172.21.15.145:6789/0 conn(0x7f83600a97c0 legacy=0x7f83600c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8357fff640 1 -- 172.21.15.94:0/4030795581 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914829194 0 0) 0x7f8360154330 con 0x7f83600a97c0 2024-03-06T07:20:58.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8364a95640 1 -- 172.21.15.94:0/4030795581 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83600c5d30 con 0x7f83601538c0 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8364a95640 1 --2- 172.21.15.94:0/4030795581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601494a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f83600c4820 tx=0x7f835002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ca34401d3134fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8357fff640 1 -- 172.21.15.94:0/4030795581 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350035020 con 0x7f83601538c0 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8357fff640 1 -- 172.21.15.94:0/4030795581 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8350031640 con 0x7f83601538c0 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.721+0000 7f8357fff640 1 -- 172.21.15.94:0/4030795581 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835002d040 con 0x7f83601538c0 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 -- 172.21.15.94:0/4030795581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 msgr2=0x7f83601494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 --2- 172.21.15.94:0/4030795581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601494a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f83600c4820 tx=0x7f835002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 -- 172.21.15.94:0/4030795581 shutdown_connections 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 --2- 172.21.15.94:0/4030795581 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601494a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 --2- 172.21.15.94:0/4030795581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600ab310 0x7f83600ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 -- 172.21.15.94:0/4030795581 >> 172.21.15.94:0/4030795581 conn(0x7f83600c6710 msgr2=0x7f83600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 -- 172.21.15.94:0/4030795581 shutdown_connections 2024-03-06T07:20:58.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.722+0000 7f83674fa640 1 -- 172.21.15.94:0/4030795581 wait complete. 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.723+0000 7f83674fa640 1 Processor -- start 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.723+0000 7f83674fa640 1 -- start start 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f83674fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600a97c0 0x7f83601ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f83674fa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83600ab310 0x7f83601dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f83674fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601df650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f83674fa640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83600abc30 con 0x7f83601538c0 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f83674fa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83600c62f0 con 0x7f83600ab310 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f83674fa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83600c5240 con 0x7f83600a97c0 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601df650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601df650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38730/0 (socket says 172.21.15.94:38730) 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 -- 172.21.15.94:0/751063778 learned_addr learned my addr 172.21.15.94:0/751063778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8364a95640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83600ab310 0x7f83601dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365296640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600a97c0 0x7f83601ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 -- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600a97c0 msgr2=0x7f83601ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600a97c0 0x7f83601ebfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 -- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83600ab310 msgr2=0x7f83601dceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83600ab310 0x7f83601dceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 -- 172.21.15.94:0/751063778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83601ec9b0 con 0x7f83601538c0 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8365a97640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601df650 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f835800c760 tx=0x7f835800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.724+0000 7f8355ffb640 1 -- 172.21.15.94:0/751063778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358003da0 con 0x7f83601538c0 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.725+0000 7f8355ffb640 1 -- 172.21.15.94:0/751063778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8358005070 con 0x7f83601538c0 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.725+0000 7f8355ffb640 1 -- 172.21.15.94:0/751063778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358005460 con 0x7f83601538c0 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.725+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8360072bf0 con 0x7f83601538c0 2024-03-06T07:20:58.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.725+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836014f700 con 0x7f83601538c0 2024-03-06T07:20:58.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.726+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836004f330 con 0x7f83601538c0 2024-03-06T07:20:58.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.726+0000 7f8355ffb640 1 -- 172.21.15.94:0/751063778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8358005600 con 0x7f83601538c0 2024-03-06T07:20:58.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.726+0000 7f8355ffb640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:58.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.726+0000 7f8355ffb640 1 -- 172.21.15.94:0/751063778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f835808a0c0 con 0x7f83601538c0 2024-03-06T07:20:58.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.726+0000 7f8365296640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:58.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.727+0000 7f8365296640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f835c009b40 tx=0x7f835c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:58.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.731+0000 7f8355ffb640 1 -- 172.21.15.94:0/751063778 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83580587f0 con 0x7f83601538c0 2024-03-06T07:20:58.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.905+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83600c5a70 con 0x7f83601538c0 2024-03-06T07:20:58.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.905+0000 7f8355ffb640 1 -- 172.21.15.94:0/751063778 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v74) v1 ==== 104+0+11 (secure 0 0 0) 0x7f835805c890 con 0x7f83601538c0 2024-03-06T07:20:58.907 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:20:58.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f834406b4d0 msgr2=0x7f834406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f835c009b40 tx=0x7f835c009190 comp rx=0 tx=0).stop 2024-03-06T07:20:58.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 msgr2=0x7f83601df650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:58.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601df650 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f835800c760 tx=0x7f835800cc20 comp rx=0 tx=0).stop 2024-03-06T07:20:58.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 shutdown_connections 2024-03-06T07:20:58.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83601538c0 0x7f83601df650 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83600ab310 0x7f83601dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 --2- 172.21.15.94:0/751063778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83600a97c0 0x7f83601ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:58.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 >> 172.21.15.94:0/751063778 conn(0x7f83600c6710 msgr2=0x7f83600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:58.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 shutdown_connections 2024-03-06T07:20:58.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:58.907+0000 7f83674fa640 1 -- 172.21.15.94:0/751063778 wait complete. 2024-03-06T07:20:58.920 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:20:59.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.022+0000 7fde4ca34640 1 Processor -- start 2024-03-06T07:20:59.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.022+0000 7fde4ca34640 1 -- start start 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde4ca34640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde480a8fd0 0x7fde480c5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde4ca34640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fde48071760 0x7fde48071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde4ca34640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde480c5d30 con 0x7fde481571e0 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde4ca34640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde480c0f00 con 0x7fde48071760 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde4ca34640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde480c6940 con 0x7fde480a8fd0 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde4659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fde481571e0 0x7fde4806deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41816/0 (socket says 172.21.15.94:41816) 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde4659c640 1 -- 172.21.15.94:0/1134852670 learned_addr learned my addr 172.21.15.94:0/1134852670 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde45d9b640 1 --2- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde480a8fd0 0x7fde480c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.023+0000 7fde46d9d640 1 --2- 172.21.15.94:0/1134852670 >> v2:172.21.15.145:3300/0 conn(0x7fde48071760 0x7fde48071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde4559a640 1 -- 172.21.15.94:0/1134852670 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868274291 0 0) 0x7fde480c5d30 con 0x7fde481571e0 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde4559a640 1 -- 172.21.15.94:0/1134852670 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde30003520 con 0x7fde481571e0 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde46d9d640 1 -- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde480a8fd0 msgr2=0x7fde480c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:59.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde46d9d640 1 --2- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde480a8fd0 0x7fde480c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde46d9d640 1 -- 172.21.15.94:0/1134852670 >> v1:172.21.15.94:6789/0 conn(0x7fde481571e0 legacy=0x7fde4806deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde46d9d640 1 -- 172.21.15.94:0/1134852670 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde480c6c20 con 0x7fde48071760 2024-03-06T07:20:59.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde45d9b640 1 --2- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde480a8fd0 0x7fde480c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:59.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.024+0000 7fde46d9d640 1 --2- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde48071760 0x7fde48071bb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fde40009d90 tx=0x7fde4002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62acfecc8c65fa13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4559a640 1 -- 172.21.15.94:0/1134852670 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40039070 con 0x7fde48071760 2024-03-06T07:20:59.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4559a640 1 -- 172.21.15.94:0/1134852670 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4002cb30 con 0x7fde48071760 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4559a640 1 -- 172.21.15.94:0/1134852670 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40034440 con 0x7fde48071760 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 -- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde48071760 msgr2=0x7fde48071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 --2- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde48071760 0x7fde48071bb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fde40009d90 tx=0x7fde4002bd80 comp rx=0 tx=0).stop 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 -- 172.21.15.94:0/1134852670 shutdown_connections 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 --2- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde48071760 0x7fde48071bb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 --2- 172.21.15.94:0/1134852670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde480a8fd0 0x7fde480c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 -- 172.21.15.94:0/1134852670 >> 172.21.15.94:0/1134852670 conn(0x7fde4806cb60 msgr2=0x7fde480a86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 -- 172.21.15.94:0/1134852670 shutdown_connections 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.025+0000 7fde4ca34640 1 -- 172.21.15.94:0/1134852670 wait complete. 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 Processor -- start 2024-03-06T07:20:59.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 -- start start 2024-03-06T07:20:59.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde48071760 0x7fde481e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde480a8fd0 0x7fde481ec8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde481571e0 0x7fde481e5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde480c7420 con 0x7fde480a8fd0 2024-03-06T07:20:59.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde480720f0 con 0x7fde481571e0 2024-03-06T07:20:59.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4ca34640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde480c6010 con 0x7fde48071760 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde45d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde480a8fd0 0x7fde481ec8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde45d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde480a8fd0 0x7fde481ec8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38738/0 (socket says 172.21.15.94:38738) 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde45d9b640 1 -- 172.21.15.94:0/2297821394 learned_addr learned my addr 172.21.15.94:0/2297821394 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde4659c640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde48071760 0x7fde481e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.027+0000 7fde46d9d640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde481571e0 0x7fde481e5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde45d9b640 1 -- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde48071760 msgr2=0x7fde481e93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde45d9b640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde48071760 0x7fde481e93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde45d9b640 1 -- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde481571e0 msgr2=0x7fde481e5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde45d9b640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde481571e0 0x7fde481e5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde45d9b640 1 -- 172.21.15.94:0/2297821394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde481ece30 con 0x7fde480a8fd0 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde45d9b640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde480a8fd0 0x7fde481ec8f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e960 tx=0x7fde3c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde36ffd640 1 -- 172.21.15.94:0/2297821394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c017070 con 0x7fde480a8fd0 2024-03-06T07:20:59.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde36ffd640 1 -- 172.21.15.94:0/2297821394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c00ca00 con 0x7fde480a8fd0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde481e88f0 con 0x7fde480a8fd0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde36ffd640 1 -- 172.21.15.94:0/2297821394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c00cdd0 con 0x7fde480a8fd0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.028+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde480bdfa0 con 0x7fde480a8fd0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.029+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4804f330 con 0x7fde480a8fd0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.030+0000 7fde36ffd640 1 -- 172.21.15.94:0/2297821394 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde3c01a070 con 0x7fde480a8fd0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.030+0000 7fde36ffd640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde2806b4d0 0x7fde2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.030+0000 7fde36ffd640 1 -- 172.21.15.94:0/2297821394 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde3c08a130 con 0x7fde480a8fd0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.030+0000 7fde4659c640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde2806b4d0 0x7fde2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.031+0000 7fde4659c640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde2806b4d0 0x7fde2806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fde38005880 tx=0x7fde38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.034+0000 7fde36ffd640 1 -- 172.21.15.94:0/2297821394 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde481ece30 con 0x7fde480a8fd0 2024-03-06T07:20:59.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.206+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde481e6520 con 0x7fde480a8fd0 2024-03-06T07:20:59.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.206+0000 7fde36ffd640 1 -- 172.21.15.94:0/2297821394 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v74) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde3c058860 con 0x7fde480a8fd0 2024-03-06T07:20:59.208 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:20:59.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde2806b4d0 msgr2=0x7fde2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde2806b4d0 0x7fde2806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fde38005880 tx=0x7fde38009040 comp rx=0 tx=0).stop 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde480a8fd0 msgr2=0x7fde481ec8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde480a8fd0 0x7fde481ec8f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e960 tx=0x7fde3c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 shutdown_connections 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde2806b4d0 0x7fde2806d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde481571e0 0x7fde481e5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde480a8fd0 0x7fde481ec8f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 --2- 172.21.15.94:0/2297821394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde48071760 0x7fde481e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 >> 172.21.15.94:0/2297821394 conn(0x7fde4806cb60 msgr2=0x7fde4806dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 shutdown_connections 2024-03-06T07:20:59.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.208+0000 7fde4ca34640 1 -- 172.21.15.94:0/2297821394 wait complete. 2024-03-06T07:20:59.221 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:20:59.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12f4837640 1 Processor -- start 2024-03-06T07:20:59.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12f4837640 1 -- start start 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12f4837640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f12f00c1aa0 0x7f12f01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12f4837640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f12f0155a40 0x7f12f0155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12f4837640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f00c28d0 con 0x7f12f014f790 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12f4837640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f00c2680 con 0x7f12f00c1aa0 2024-03-06T07:20:59.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12f4837640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f00c2bb0 con 0x7f12f0155a40 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.319+0000 7f12e7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f12f00c1aa0 0x7f12f01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12e7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f12f00c1aa0 0x7f12f01529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38754/0 (socket says 172.21.15.94:38754) 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12e7fff640 1 -- 172.21.15.94:0/2805163000 learned_addr learned my addr 172.21.15.94:0/2805163000 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12eed9d640 1 --2- 172.21.15.94:0/2805163000 >> v2:172.21.15.145:3300/0 conn(0x7f12f0155a40 0x7f12f0155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12e7fff640 1 -- 172.21.15.94:0/2805163000 >> v2:172.21.15.145:3300/0 conn(0x7f12f0155a40 msgr2=0x7f12f0155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12e7fff640 1 --2- 172.21.15.94:0/2805163000 >> v2:172.21.15.145:3300/0 conn(0x7f12f0155a40 0x7f12f0155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12e7fff640 1 -- 172.21.15.94:0/2805163000 >> v1:172.21.15.145:6789/0 conn(0x7f12f014f790 legacy=0x7f12f00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12e7fff640 1 -- 172.21.15.94:0/2805163000 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f00c2e90 con 0x7f12f00c1aa0 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.320+0000 7f12e7fff640 1 --2- 172.21.15.94:0/2805163000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f00c1aa0 0x7f12f01529d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f12f006c960 tx=0x7f12d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871349a16a434348 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12edd9b640 1 -- 172.21.15.94:0/2805163000 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8034070 con 0x7f12f00c1aa0 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12edd9b640 1 -- 172.21.15.94:0/2805163000 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d8030860 con 0x7f12f00c1aa0 2024-03-06T07:20:59.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12edd9b640 1 -- 172.21.15.94:0/2805163000 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8030c30 con 0x7f12f00c1aa0 2024-03-06T07:20:59.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 -- 172.21.15.94:0/2805163000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f00c1aa0 msgr2=0x7f12f01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 --2- 172.21.15.94:0/2805163000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f00c1aa0 0x7f12f01529d0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f12f006c960 tx=0x7f12d802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 -- 172.21.15.94:0/2805163000 shutdown_connections 2024-03-06T07:20:59.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 --2- 172.21.15.94:0/2805163000 >> v2:172.21.15.145:3300/0 conn(0x7f12f0155a40 0x7f12f0155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 --2- 172.21.15.94:0/2805163000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f00c1aa0 0x7f12f01529d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 -- 172.21.15.94:0/2805163000 >> 172.21.15.94:0/2805163000 conn(0x7f12f00c6710 msgr2=0x7f12f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 -- 172.21.15.94:0/2805163000 shutdown_connections 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.321+0000 7f12f4837640 1 -- 172.21.15.94:0/2805163000 wait complete. 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.322+0000 7f12f4837640 1 Processor -- start 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.322+0000 7f12f4837640 1 -- start start 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12f4837640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12f00c1aa0 0x7f12f00c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12f4837640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12f014f790 0x7f12f0151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12f4837640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f0155a40 0x7f12f01ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12f4837640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00c0d60 con 0x7f12f0155a40 2024-03-06T07:20:59.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12f4837640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0156390 con 0x7f12f00c1aa0 2024-03-06T07:20:59.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12f4837640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00c5e30 con 0x7f12f014f790 2024-03-06T07:20:59.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f0155a40 0x7f12f01ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12ee59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12f00c1aa0 0x7f12f00c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f0155a40 0x7f12f01ee5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38766/0 (socket says 172.21.15.94:38766) 2024-03-06T07:20:59.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12eed9d640 1 -- 172.21.15.94:0/2341844151 learned_addr learned my addr 172.21.15.94:0/2341844151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:59.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.323+0000 7f12e7fff640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12f014f790 0x7f12f0151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12ee59c640 1 -- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12f014f790 msgr2=0x7f12f0151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12ee59c640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12f014f790 0x7f12f0151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12ee59c640 1 -- 172.21.15.94:0/2341844151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f0155a40 msgr2=0x7f12f01ee5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12ee59c640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f0155a40 0x7f12f01ee5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12ee59c640 1 -- 172.21.15.94:0/2341844151 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0072e30 con 0x7f12f00c1aa0 2024-03-06T07:20:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12eed9d640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f0155a40 0x7f12f01ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:20:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12ee59c640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12f00c1aa0 0x7f12f00c4880 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c3a0 tx=0x7f12e000c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12e6ffd640 1 -- 172.21.15.94:0/2341844151 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e00052d0 con 0x7f12f00c1aa0 2024-03-06T07:20:59.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12e6ffd640 1 -- 172.21.15.94:0/2341844151 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0005490 con 0x7f12f00c1aa0 2024-03-06T07:20:59.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f0073110 con 0x7f12f00c1aa0 2024-03-06T07:20:59.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12e6ffd640 1 -- 172.21.15.94:0/2341844151 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0007260 con 0x7f12f00c1aa0 2024-03-06T07:20:59.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.324+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f00c4dc0 con 0x7f12f00c1aa0 2024-03-06T07:20:59.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.325+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f004f330 con 0x7f12f00c1aa0 2024-03-06T07:20:59.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.326+0000 7f12e6ffd640 1 -- 172.21.15.94:0/2341844151 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12e0007400 con 0x7f12f00c1aa0 2024-03-06T07:20:59.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.326+0000 7f12e6ffd640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12d006b4d0 0x7f12d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.326+0000 7f12e6ffd640 1 -- 172.21.15.94:0/2341844151 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f12e00897d0 con 0x7f12f00c1aa0 2024-03-06T07:20:59.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.326+0000 7f12e7fff640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12d006b4d0 0x7f12d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.327+0000 7f12e7fff640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12d006b4d0 0x7f12d006d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f12d8036000 tx=0x7f12d8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.335+0000 7f12e6ffd640 1 -- 172.21.15.94:0/2341844151 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12e0057df0 con 0x7f12f00c1aa0 2024-03-06T07:20:59.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.509+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f12f00c1f40 con 0x7f12f00c1aa0 2024-03-06T07:20:59.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.509+0000 7f12e6ffd640 1 -- 172.21.15.94:0/2341844151 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) v1 ==== 84+0+209 (secure 0 0 0) 0x7f12e005be90 con 0x7f12f00c1aa0 2024-03-06T07:20:59.511 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:20:59.512 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:20:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:20:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:20:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:59.512 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 11, 2024-03-06T07:20:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": "cephfs_metadata" 2024-03-06T07:20:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:20:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:20:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 12, 2024-03-06T07:20:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": "cephfs_data" 2024-03-06T07:20:59.513 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:20:59.513 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12d006b4d0 msgr2=0x7f12d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12d006b4d0 0x7f12d006d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f12d8036000 tx=0x7f12d8008030 comp rx=0 tx=0).stop 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12f00c1aa0 msgr2=0x7f12f00c4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12f00c1aa0 0x7f12f00c4880 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c3a0 tx=0x7f12e000c860 comp rx=0 tx=0).stop 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 shutdown_connections 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f12d006b4d0 0x7f12d006d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12f0155a40 0x7f12f01ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12f014f790 0x7f12f0151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 --2- 172.21.15.94:0/2341844151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12f00c1aa0 0x7f12f00c4880 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 >> 172.21.15.94:0/2341844151 conn(0x7f12f00c6710 msgr2=0x7f12f00a65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 shutdown_connections 2024-03-06T07:20:59.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.511+0000 7f12f4837640 1 -- 172.21.15.94:0/2341844151 wait complete. 2024-03-06T07:20:59.523 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-06T07:20:59.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-06T07:20:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.626+0000 7f1ef5747640 1 Processor -- start 2024-03-06T07:20:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.626+0000 7f1ef5747640 1 -- start start 2024-03-06T07:20:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.626+0000 7f1ef5747640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef00c7470 0x7f1ef0155d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1ef5747640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef0156260 con 0x7f1ef0159230 2024-03-06T07:20:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1ef5747640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef00c7a10 con 0x7f1ef0152860 2024-03-06T07:20:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1ef5747640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef0158da0 con 0x7f1ef00c7470 2024-03-06T07:20:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1eef7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1ef0159230 0x7f1ef00c53e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41828/0 (socket says 172.21.15.94:41828) 2024-03-06T07:20:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1eef7fe640 1 -- 172.21.15.94:0/2124988485 learned_addr learned my addr 172.21.15.94:0/2124988485 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:20:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1ee7fff640 1 --2- 172.21.15.94:0/2124988485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef00c7470 0x7f1ef0155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1eee7fc640 1 -- 172.21.15.94:0/2124988485 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976850496 0 0) 0x7f1ee8001490 con 0x7f1ef0159230 2024-03-06T07:20:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1eee7fc640 1 -- 172.21.15.94:0/2124988485 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc003520 con 0x7f1ef0159230 2024-03-06T07:20:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1eee7fc640 1 -- 172.21.15.94:0/2124988485 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686414777 0 0) 0x7f1ef00c7a10 con 0x7f1ef0152860 2024-03-06T07:20:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1ee7fff640 1 -- 172.21.15.94:0/2124988485 >> v1:172.21.15.145:6789/0 conn(0x7f1ef0152860 legacy=0x7f1ef00c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1ee7fff640 1 -- 172.21.15.94:0/2124988485 >> v1:172.21.15.94:6789/0 conn(0x7f1ef0159230 legacy=0x7f1ef00c53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.627+0000 7f1ee7fff640 1 -- 172.21.15.94:0/2124988485 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00c7cf0 con 0x7f1ef00c7470 2024-03-06T07:20:59.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.628+0000 7f1ee7fff640 1 --2- 172.21.15.94:0/2124988485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef00c7470 0x7f1ef0155d20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80097d0 tx=0x7f1ed802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b7e88f432bb472 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.628+0000 7f1eee7fc640 1 -- 172.21.15.94:0/2124988485 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8035020 con 0x7f1ef00c7470 2024-03-06T07:20:59.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.628+0000 7f1eee7fc640 1 -- 172.21.15.94:0/2124988485 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed8031800 con 0x7f1ef00c7470 2024-03-06T07:20:59.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.628+0000 7f1eee7fc640 1 -- 172.21.15.94:0/2124988485 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8031bd0 con 0x7f1ef00c7470 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.629+0000 7f1ef5747640 1 -- 172.21.15.94:0/2124988485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef00c7470 msgr2=0x7f1ef0155d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.629+0000 7f1ef5747640 1 --2- 172.21.15.94:0/2124988485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef00c7470 0x7f1ef0155d20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80097d0 tx=0x7f1ed802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.629+0000 7f1ef5747640 1 -- 172.21.15.94:0/2124988485 shutdown_connections 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.629+0000 7f1ef5747640 1 --2- 172.21.15.94:0/2124988485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef00c7470 0x7f1ef0155d20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.629+0000 7f1ef5747640 1 -- 172.21.15.94:0/2124988485 >> 172.21.15.94:0/2124988485 conn(0x7f1ef00ac630 msgr2=0x7f1ef00aca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.629+0000 7f1ef5747640 1 -- 172.21.15.94:0/2124988485 shutdown_connections 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.629+0000 7f1ef5747640 1 -- 172.21.15.94:0/2124988485 wait complete. 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 Processor -- start 2024-03-06T07:20:59.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 -- start start 2024-03-06T07:20:59.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef00c7470 0x7f1ef01eb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef0152860 0x7f1ef01e8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef0159230 0x7f1ef0157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef00c5af0 con 0x7f1ef0152860 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef0152c40 con 0x7f1ef00c7470 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ef5747640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef00c5c70 con 0x7f1ef0159230 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ee7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef0152860 0x7f1ef01e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ee7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef0152860 0x7f1ef01e8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38768/0 (socket says 172.21.15.94:38768) 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.630+0000 7f1ee7fff640 1 -- 172.21.15.94:0/3296560701 learned_addr learned my addr 172.21.15.94:0/3296560701 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1eef7fe640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef0159230 0x7f1ef0157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1eeeffd640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef00c7470 0x7f1ef01eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee7fff640 1 -- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef0159230 msgr2=0x7f1ef0157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee7fff640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef0159230 0x7f1ef0157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee7fff640 1 -- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef00c7470 msgr2=0x7f1ef01eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee7fff640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef00c7470 0x7f1ef01eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee7fff640 1 -- 172.21.15.94:0/3296560701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef01e86c0 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee7fff640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef0152860 0x7f1ef01e8180 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80097d0 tx=0x7f1ed80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee77fe640 1 -- 172.21.15.94:0/3296560701 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8041070 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee77fe640 1 -- 172.21.15.94:0/3296560701 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed803ad00 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef00c46c0 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ee77fe640 1 -- 172.21.15.94:0/3296560701 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8035040 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.631+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef01eae20 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.632+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef004f330 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.632+0000 7f1ee77fe640 1 -- 172.21.15.94:0/3296560701 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ed8038020 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.633+0000 7f1ee77fe640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed006b4d0 0x7f1ed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.633+0000 7f1ee77fe640 1 -- 172.21.15.94:0/3296560701 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1ed80abcc0 con 0x7f1ef0152860 2024-03-06T07:20:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.633+0000 7f1eeeffd640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed006b4d0 0x7f1ed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:20:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.633+0000 7f1eeeffd640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed006b4d0 0x7f1ed006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0000f60 tx=0x7f1ee0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:20:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.637+0000 7f1ee77fe640 1 -- 172.21.15.94:0/3296560701 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ef01e86c0 con 0x7f1ef0152860 2024-03-06T07:20:59.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:20:59.809+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1ef00c8110 con 0x7f1ef0152860 2024-03-06T07:21:00.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.414+0000 7f1ee77fe640 1 -- 172.21.15.94:0/3296560701 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v75) v1 ==== 140+0+0 (secure 0 0 0) 0x7f1ed807a3f0 con 0x7f1ef0152860 2024-03-06T07:21:00.416 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_metadata' created 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed006b4d0 msgr2=0x7f1ed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed006b4d0 0x7f1ed006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0000f60 tx=0x7f1ee0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef0152860 msgr2=0x7f1ef01e8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef0152860 0x7f1ef01e8180 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80097d0 tx=0x7f1ed80046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 shutdown_connections 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed006b4d0 0x7f1ed006d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef0159230 0x7f1ef0157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef0152860 0x7f1ef01e8180 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 --2- 172.21.15.94:0/3296560701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef00c7470 0x7f1ef01eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 >> 172.21.15.94:0/3296560701 conn(0x7f1ef00ac630 msgr2=0x7f1ef0148b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 shutdown_connections 2024-03-06T07:21:00.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.417+0000 7f1ef5747640 1 -- 172.21.15.94:0/3296560701 wait complete. 2024-03-06T07:21:00.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:00.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.543+0000 7f563784e640 1 Processor -- start 2024-03-06T07:21:00.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.543+0000 7f563784e640 1 -- start start 2024-03-06T07:21:00.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.543+0000 7f563784e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f563014ef70 0x7f563006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.543+0000 7f563784e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f563006cad0 0x7f5630154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.543+0000 7f563784e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56300c2790 con 0x7f56300c1820 2024-03-06T07:21:00.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.543+0000 7f563784e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56300c3090 con 0x7f563006cad0 2024-03-06T07:21:00.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.543+0000 7f563784e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563014f350 con 0x7f563014ef70 2024-03-06T07:21:00.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5635deb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f563006cad0 0x7f5630154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5635deb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f563006cad0 0x7f5630154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38772/0 (socket says 172.21.15.94:38772) 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5634de9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f56300c1820 0x7f56300c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41838/0 (socket says 172.21.15.94:41838) 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5635deb640 1 -- 172.21.15.94:0/3579200651 learned_addr learned my addr 172.21.15.94:0/3579200651 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f56355ea640 1 --2- 172.21.15.94:0/3579200651 >> v2:172.21.15.145:3300/0 conn(0x7f563014ef70 0x7f563006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5635deb640 1 -- 172.21.15.94:0/3579200651 >> v2:172.21.15.145:3300/0 conn(0x7f563014ef70 msgr2=0x7f563006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5635deb640 1 --2- 172.21.15.94:0/3579200651 >> v2:172.21.15.145:3300/0 conn(0x7f563014ef70 0x7f563006c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5635deb640 1 -- 172.21.15.94:0/3579200651 >> v1:172.21.15.94:6789/0 conn(0x7f56300c1820 legacy=0x7f56300c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5627fff640 1 -- 172.21.15.94:0/3579200651 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769738481 0 0) 0x7f56300c2790 con 0x7f56300c1820 2024-03-06T07:21:00.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.544+0000 7f5635deb640 1 -- 172.21.15.94:0/3579200651 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563006cf40 con 0x7f563006cad0 2024-03-06T07:21:00.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f5635deb640 1 --2- 172.21.15.94:0/3579200651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f563006cad0 0x7f5630154c30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f563006e5d0 tx=0x7f562802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a0bff4a2b45142 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f5627fff640 1 -- 172.21.15.94:0/3579200651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562802ca80 con 0x7f563006cad0 2024-03-06T07:21:00.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f5627fff640 1 -- 172.21.15.94:0/3579200651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562802cc20 con 0x7f563006cad0 2024-03-06T07:21:00.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f5627fff640 1 -- 172.21.15.94:0/3579200651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56280343f0 con 0x7f563006cad0 2024-03-06T07:21:00.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f563784e640 1 -- 172.21.15.94:0/3579200651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f563006cad0 msgr2=0x7f5630154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f563784e640 1 --2- 172.21.15.94:0/3579200651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f563006cad0 0x7f5630154c30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f563006e5d0 tx=0x7f562802bee0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f563784e640 1 -- 172.21.15.94:0/3579200651 shutdown_connections 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f563784e640 1 --2- 172.21.15.94:0/3579200651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f563006cad0 0x7f5630154c30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f563784e640 1 --2- 172.21.15.94:0/3579200651 >> v2:172.21.15.145:3300/0 conn(0x7f563014ef70 0x7f563006c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.545+0000 7f563784e640 1 -- 172.21.15.94:0/3579200651 >> 172.21.15.94:0/3579200651 conn(0x7f56300c6710 msgr2=0x7f56300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.546+0000 7f563784e640 1 -- 172.21.15.94:0/3579200651 shutdown_connections 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.546+0000 7f563784e640 1 -- 172.21.15.94:0/3579200651 wait complete. 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 Processor -- start 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 -- start start 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563006cad0 0x7f5630148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56300c1820 0x7f56300c3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f563014ef70 0x7f5630151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5630155750 con 0x7f56300c1820 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56300c24b0 con 0x7f563006cad0 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f563784e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5630155170 con 0x7f563014ef70 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f5634de9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56300c1820 0x7f56300c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f5634de9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56300c1820 0x7f56300c3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38780/0 (socket says 172.21.15.94:38780) 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f5634de9640 1 -- 172.21.15.94:0/21136662 learned_addr learned my addr 172.21.15.94:0/21136662 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f56355ea640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563006cad0 0x7f5630148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.547+0000 7f5635deb640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f563014ef70 0x7f5630151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5634de9640 1 -- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f563014ef70 msgr2=0x7f5630151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5634de9640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f563014ef70 0x7f5630151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5634de9640 1 -- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563006cad0 msgr2=0x7f5630148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5634de9640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563006cad0 0x7f5630148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5634de9640 1 -- 172.21.15.94:0/21136662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56301ec960 con 0x7f56300c1820 2024-03-06T07:21:00.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5634de9640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56300c1820 0x7f56300c3480 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f562000c4a0 tx=0x7f562000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5625ffb640 1 -- 172.21.15.94:0/21136662 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620018070 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5625ffb640 1 -- 172.21.15.94:0/21136662 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56200074e0 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5630150a90 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f5625ffb640 1 -- 172.21.15.94:0/21136662 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620004a20 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.548+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5630151050 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.549+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563004f330 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.550+0000 7f5625ffb640 1 -- 172.21.15.94:0/21136662 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f562001b070 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.550+0000 7f5625ffb640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f561406b4d0 0x7f561406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.550+0000 7f5625ffb640 1 -- 172.21.15.94:0/21136662 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6472+0+0 (secure 0 0 0) 0x7f56200897f0 con 0x7f56300c1820 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.550+0000 7f56355ea640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f561406b4d0 0x7f561406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:00.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.551+0000 7f56355ea640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f561406b4d0 0x7f561406d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f562c009aa0 tx=0x7f562c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:00.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.555+0000 7f5625ffb640 1 -- 172.21.15.94:0/21136662 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5620057c90 con 0x7f56300c1820 2024-03-06T07:21:00.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:00.725+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f56300c39c0 con 0x7f56300c1820 2024-03-06T07:21:01.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.416+0000 7f5625ffb640 1 -- 172.21.15.94:0/21136662 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v76) v1 ==== 172+0+0 (secure 0 0 0) 0x7f562005bd30 con 0x7f56300c1820 2024-03-06T07:21:01.418 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_data' created 2024-03-06T07:21:01.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f561406b4d0 msgr2=0x7f561406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f561406b4d0 0x7f561406d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f562c009aa0 tx=0x7f562c009190 comp rx=0 tx=0).stop 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56300c1820 msgr2=0x7f56300c3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56300c1820 0x7f56300c3480 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f562000c4a0 tx=0x7f562000c960 comp rx=0 tx=0).stop 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 shutdown_connections 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f561406b4d0 0x7f561406d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f563014ef70 0x7f5630151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56300c1820 0x7f56300c3480 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 --2- 172.21.15.94:0/21136662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f563006cad0 0x7f5630148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 >> 172.21.15.94:0/21136662 conn(0x7f56300c6710 msgr2=0x7f5630152a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 shutdown_connections 2024-03-06T07:21:01.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.419+0000 7f563784e640 1 -- 172.21.15.94:0/21136662 wait complete. 2024-03-06T07:21:01.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-06T07:21:01.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.546+0000 7f21b4ca4640 1 Processor -- start 2024-03-06T07:21:01.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.546+0000 7f21b4ca4640 1 -- start start 2024-03-06T07:21:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.546+0000 7f21b4ca4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b0150680 0x7f21b00c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.546+0000 7f21b4ca4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f21b00c2b10 0x7f21b00a87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21b4ca4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b0153b70 con 0x7f21b00c26b0 2024-03-06T07:21:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21b4ca4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b00c1140 con 0x7f21b00c2b10 2024-03-06T07:21:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21b4ca4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b00c1420 con 0x7f21b0150680 2024-03-06T07:21:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21ae59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b0150680 0x7f21b00c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21aed9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f21b00c2b10 0x7f21b00a87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21ae59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b0150680 0x7f21b00c1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53104/0 (socket says 172.21.15.94:53104) 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21aed9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f21b00c2b10 0x7f21b00a87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:57304/0 (socket says 172.21.15.94:57304) 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21add9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f21b00c26b0 0x7f21b01530f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38132/0 (socket says 172.21.15.94:38132) 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21ae59c640 1 -- 172.21.15.94:0/3413268284 learned_addr learned my addr 172.21.15.94:0/3413268284 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21aed9d640 1 -- 172.21.15.94:0/3413268284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b0150680 msgr2=0x7f21b00c1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21aed9d640 1 --2- 172.21.15.94:0/3413268284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b0150680 0x7f21b00c1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21aed9d640 1 -- 172.21.15.94:0/3413268284 >> v1:172.21.15.145:6789/0 conn(0x7f21b00c26b0 legacy=0x7f21b01530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21aed9d640 1 -- 172.21.15.94:0/3413268284 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b00c1700 con 0x7f21b00c2b10 2024-03-06T07:21:01.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.547+0000 7f21ad59a640 1 -- 172.21.15.94:0/3413268284 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439731084 0 0) 0x7f21b0153b70 con 0x7f21b00c26b0 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.548+0000 7f21aed9d640 1 --2- 172.21.15.94:0/3413268284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b00c2b10 0x7f21b00a87f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009d90 tx=0x7f21a802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cade5c146d6cd81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.548+0000 7f21ad59a640 1 -- 172.21.15.94:0/3413268284 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8039070 con 0x7f21b00c2b10 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.548+0000 7f21ad59a640 1 -- 172.21.15.94:0/3413268284 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a802cc50 con 0x7f21b00c2b10 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.548+0000 7f21ad59a640 1 -- 172.21.15.94:0/3413268284 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8034440 con 0x7f21b00c2b10 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 -- 172.21.15.94:0/3413268284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b00c2b10 msgr2=0x7f21b00a87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/3413268284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b00c2b10 0x7f21b00a87f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009d90 tx=0x7f21a802bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 -- 172.21.15.94:0/3413268284 shutdown_connections 2024-03-06T07:21:01.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/3413268284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b00c2b10 0x7f21b00a87f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/3413268284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b0150680 0x7f21b00c1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 -- 172.21.15.94:0/3413268284 >> 172.21.15.94:0/3413268284 conn(0x7f21b006afc0 msgr2=0x7f21b006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:01.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 -- 172.21.15.94:0/3413268284 shutdown_connections 2024-03-06T07:21:01.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.549+0000 7f21b4ca4640 1 -- 172.21.15.94:0/3413268284 wait complete. 2024-03-06T07:21:01.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.550+0000 7f21b4ca4640 1 Processor -- start 2024-03-06T07:21:01.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.550+0000 7f21b4ca4640 1 -- start start 2024-03-06T07:21:01.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21b4ca4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b00c26b0 0x7f21b00aa950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21b4ca4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b0150680 0x7f21b01ea380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21b4ca4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21b014d210 0x7f21b0151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21b4ca4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0156680 con 0x7f21b014d210 2024-03-06T07:21:01.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21b4ca4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0153800 con 0x7f21b0150680 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21b4ca4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0150a60 con 0x7f21b00c26b0 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21b014d210 0x7f21b0151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21b014d210 0x7f21b0151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38790/0 (socket says 172.21.15.94:38790) 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 -- 172.21.15.94:0/387987854 learned_addr learned my addr 172.21.15.94:0/387987854 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21add9b640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b0150680 0x7f21b01ea380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21ae59c640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b00c26b0 0x7f21b00aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 -- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b00c26b0 msgr2=0x7f21b00aa950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b00c26b0 0x7f21b00aa950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 -- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b0150680 msgr2=0x7f21b01ea380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:01.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b0150680 0x7f21b01ea380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:01.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 -- 172.21.15.94:0/387987854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b00a98f0 con 0x7f21b014d210 2024-03-06T07:21:01.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21aed9d640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21b014d210 0x7f21b0151da0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009d90 tx=0x7f21a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f219effd640 1 -- 172.21.15.94:0/387987854 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8044020 con 0x7f21b014d210 2024-03-06T07:21:01.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f219effd640 1 -- 172.21.15.94:0/387987854 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a803a040 con 0x7f21b014d210 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b00a9b90 con 0x7f21b014d210 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.551+0000 7f219effd640 1 -- 172.21.15.94:0/387987854 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8039050 con 0x7f21b014d210 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.552+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b00aa430 con 0x7f21b014d210 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.552+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b00a9d60 con 0x7f21b014d210 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.553+0000 7f219effd640 1 -- 172.21.15.94:0/387987854 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21a8003990 con 0x7f21b014d210 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.553+0000 7f219effd640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f219006b500 0x7f219006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.554+0000 7f219effd640 1 -- 172.21.15.94:0/387987854 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f21a80ac8a0 con 0x7f21b014d210 2024-03-06T07:21:01.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.554+0000 7f21ae59c640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f219006b500 0x7f219006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:01.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.554+0000 7f21ae59c640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f219006b500 0x7f219006d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009bd0 tx=0x7f21a4009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:01.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.558+0000 7f219effd640 1 -- 172.21.15.94:0/387987854 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21a804e060 con 0x7f21b014d210 2024-03-06T07:21:01.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:01.764+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f21b004f330 con 0x7f21b014d210 2024-03-06T07:21:02.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.442+0000 7f219effd640 1 -- 172.21.15.94:0/387987854 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 13 and data pool 14 v56) v1 ==== 177+0+0 (secure 0 0 0) 0x7f21a807ace0 con 0x7f21b014d210 2024-03-06T07:21:02.444 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 13 and data pool 14 2024-03-06T07:21:02.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.452+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f219006b500 msgr2=0x7f219006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.452+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f219006b500 0x7f219006d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009bd0 tx=0x7f21a4009240 comp rx=0 tx=0).stop 2024-03-06T07:21:02.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.452+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21b014d210 msgr2=0x7f21b0151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.452+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21b014d210 0x7f21b0151da0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009d90 tx=0x7f21a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.454+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 shutdown_connections 2024-03-06T07:21:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.454+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f219006b500 0x7f219006d9c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.454+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f21b014d210 0x7f21b0151da0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.454+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f21b0150680 0x7f21b01ea380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.454+0000 7f21b4ca4640 1 --2- 172.21.15.94:0/387987854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f21b00c26b0 0x7f21b00aa950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.454+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 >> 172.21.15.94:0/387987854 conn(0x7f21b006afc0 msgr2=0x7f21b014f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.468+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 shutdown_connections 2024-03-06T07:21:02.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.469+0000 7f21b4ca4640 1 -- 172.21.15.94:0/387987854 wait complete. 2024-03-06T07:21:02.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:02.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.588+0000 7fc0d36df640 1 Processor -- start 2024-03-06T07:21:02.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.589+0000 7fc0d36df640 1 -- start start 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.589+0000 7fc0d36df640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc0cc152960 0x7fc0cc06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d36df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0cc152d40 con 0x7fc0cc0ac260 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d36df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0cc0c1b50 con 0x7fc0cc1592d0 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d36df640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0cc066e30 con 0x7fc0cc152960 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d1c7c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc0cc0ac260 0x7fc0cc15fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41854/0 (socket says 172.21.15.94:41854) 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d1c7c640 1 -- 172.21.15.94:0/2611532878 learned_addr learned my addr 172.21.15.94:0/2611532878 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d147b640 1 --2- 172.21.15.94:0/2611532878 >> v2:172.21.15.145:3300/0 conn(0x7fc0cc152960 0x7fc0cc06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050480633 0 0) 0x7fc0cc152d40 con 0x7fc0cc0ac260 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b8003520 con 0x7fc0cc0ac260 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777500661 0 0) 0x7fc0cc0c1b50 con 0x7fc0cc1592d0 2024-03-06T07:21:02.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0cc152d40 con 0x7fc0cc1592d0 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737114856 0 0) 0x7fc0b8003520 con 0x7fc0cc0ac260 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d147b640 1 -- 172.21.15.94:0/2611532878 >> v1:172.21.15.145:6789/0 conn(0x7fc0cc1592d0 legacy=0x7fc0cc1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d147b640 1 -- 172.21.15.94:0/2611532878 >> v1:172.21.15.94:6789/0 conn(0x7fc0cc0ac260 legacy=0x7fc0cc15fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.590+0000 7fc0d147b640 1 -- 172.21.15.94:0/2611532878 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc0c1dc0 con 0x7fc0cc152960 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.591+0000 7fc0d147b640 1 --2- 172.21.15.94:0/2611532878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 0x7fc0cc06dc90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0097a0 tx=0x7fc0bc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de9e9fe385aed58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.591+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc038070 con 0x7fc0cc152960 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.591+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0bc031500 con 0x7fc0cc152960 2024-03-06T07:21:02.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.591+0000 7fc0c3fff640 1 -- 172.21.15.94:0/2611532878 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c40019f0 con 0x7fc0cc152960 2024-03-06T07:21:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.592+0000 7fc0d36df640 1 -- 172.21.15.94:0/2611532878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 msgr2=0x7fc0cc06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.592+0000 7fc0d36df640 1 --2- 172.21.15.94:0/2611532878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 0x7fc0cc06dc90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0097a0 tx=0x7fc0bc02b820 comp rx=0 tx=0).stop 2024-03-06T07:21:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.592+0000 7fc0d36df640 1 -- 172.21.15.94:0/2611532878 shutdown_connections 2024-03-06T07:21:02.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.592+0000 7fc0d36df640 1 --2- 172.21.15.94:0/2611532878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 0x7fc0cc06dc90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.592+0000 7fc0d36df640 1 -- 172.21.15.94:0/2611532878 >> 172.21.15.94:0/2611532878 conn(0x7fc0cc0c70b0 msgr2=0x7fc0cc0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.592+0000 7fc0d36df640 1 -- 172.21.15.94:0/2611532878 shutdown_connections 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.592+0000 7fc0d36df640 1 -- 172.21.15.94:0/2611532878 wait complete. 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 Processor -- start 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 -- start start 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0cc0ac260 0x7fc0cc14d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 0x7fc0cc1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0cc1eb2e0 0x7fc0cc1eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0c20a0 con 0x7fc0cc0ac260 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc159760 con 0x7fc0cc152960 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.593+0000 7fc0d36df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0c2220 con 0x7fc0cc1eb2e0 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0cc0ac260 0x7fc0cc14d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0cc0ac260 0x7fc0cc14d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38806/0 (socket says 172.21.15.94:38806) 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 -- 172.21.15.94:0/794638903 learned_addr learned my addr 172.21.15.94:0/794638903 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d0c7a640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 0x7fc0cc1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d1c7c640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0cc1eb2e0 0x7fc0cc1eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 -- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0cc1eb2e0 msgr2=0x7fc0cc1eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0cc1eb2e0 0x7fc0cc1eb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 -- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 msgr2=0x7fc0cc1558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 0x7fc0cc1558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 -- 172.21.15.94:0/794638903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc1eeaa0 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d147b640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0cc0ac260 0x7fc0cc14d050 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc02b860 tx=0x7fc0bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0c1ffb640 1 -- 172.21.15.94:0/794638903 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc042070 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0c1ffb640 1 -- 172.21.15.94:0/794638903 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0bc03d870 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0c1ffb640 1 -- 172.21.15.94:0/794638903 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc038040 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0cc1eed80 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.594+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0cc1ef340 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.595+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0cc04f330 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.596+0000 7fc0c1ffb640 1 -- 172.21.15.94:0/794638903 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0bc045070 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.596+0000 7fc0c1ffb640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0b006b4d0 0x7fc0b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.596+0000 7fc0c1ffb640 1 -- 172.21.15.94:0/794638903 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc0bc0ac3c0 con 0x7fc0cc0ac260 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.596+0000 7fc0d0c7a640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0b006b4d0 0x7fc0b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.597+0000 7fc0d0c7a640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0b006b4d0 0x7fc0b006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0c42d0 tx=0x7fc0c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.601+0000 7fc0c1ffb640 1 -- 172.21.15.94:0/794638903 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0bc030070 con 0x7fc0cc0ac260 2024-03-06T07:21:02.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.775+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0cc06ebc0 con 0x7fc0cc0ac260 2024-03-06T07:21:02.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.776+0000 7fc0c1ffb640 1 -- 172.21.15.94:0/794638903 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+20682 (secure 0 0 0) 0x7fc0bc07a830 con 0x7fc0cc0ac260 2024-03-06T07:21:02.779 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:02.779 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":77,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:01.765912+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0b006b4d0 msgr2=0x7fc0b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0b006b4d0 0x7fc0b006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0c42d0 tx=0x7fc0c8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0cc0ac260 msgr2=0x7fc0cc14d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0cc0ac260 0x7fc0cc14d050 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc02b860 tx=0x7fc0bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 shutdown_connections 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0b006b4d0 0x7fc0b006d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0cc1eb2e0 0x7fc0cc1eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0cc152960 0x7fc0cc1558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 --2- 172.21.15.94:0/794638903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0cc0ac260 0x7fc0cc14d050 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 >> 172.21.15.94:0/794638903 conn(0x7fc0cc0c70b0 msgr2=0x7fc0cc0c4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 shutdown_connections 2024-03-06T07:21:02.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.778+0000 7fc0d36df640 1 -- 172.21.15.94:0/794638903 wait complete. 2024-03-06T07:21:02.791 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:02.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.888+0000 7f84977d7640 1 Processor -- start 2024-03-06T07:21:02.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f84977d7640 1 -- start start 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f84977d7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84900c88c0 0x7f8490155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f84977d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84900c73d0 con 0x7f849014f360 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f84977d7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84900be2e0 con 0x7f8490158b60 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f84977d7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8490159700 con 0x7f84900c88c0 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f8494d72640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84900c88c0 0x7f8490155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f8494d72640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84900c88c0 0x7f8490155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38818/0 (socket says 172.21.15.94:38818) 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.889+0000 7f8495573640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f849014f360 0x7f84900c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41870/0 (socket says 172.21.15.94:41870) 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.890+0000 7f8494d72640 1 -- 172.21.15.94:0/2797975947 learned_addr learned my addr 172.21.15.94:0/2797975947 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.890+0000 7f8494d72640 1 -- 172.21.15.94:0/2797975947 >> v1:172.21.15.145:6789/0 conn(0x7f8490158b60 legacy=0x7f8490158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.890+0000 7f8494d72640 1 -- 172.21.15.94:0/2797975947 >> v1:172.21.15.94:6789/0 conn(0x7f849014f360 legacy=0x7f84900c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.890+0000 7f8487fff640 1 -- 172.21.15.94:0/2797975947 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601683887 0 0) 0x7f84900be2e0 con 0x7f8490158b60 2024-03-06T07:21:02.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.890+0000 7f8494d72640 1 -- 172.21.15.94:0/2797975947 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84900c7670 con 0x7f84900c88c0 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.890+0000 7f8494d72640 1 --2- 172.21.15.94:0/2797975947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84900c88c0 0x7f8490155c10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f848c009870 tx=0x7f848c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b4d563b17a5dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f8487fff640 1 -- 172.21.15.94:0/2797975947 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c037070 con 0x7f84900c88c0 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f8487fff640 1 -- 172.21.15.94:0/2797975947 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c004230 con 0x7f84900c88c0 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f8487fff640 1 -- 172.21.15.94:0/2797975947 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c003610 con 0x7f84900c88c0 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f84977d7640 1 -- 172.21.15.94:0/2797975947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84900c88c0 msgr2=0x7f8490155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f84977d7640 1 --2- 172.21.15.94:0/2797975947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84900c88c0 0x7f8490155c10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f848c009870 tx=0x7f848c02b7e0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f84977d7640 1 -- 172.21.15.94:0/2797975947 shutdown_connections 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f84977d7640 1 --2- 172.21.15.94:0/2797975947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84900c88c0 0x7f8490155c10 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f84977d7640 1 -- 172.21.15.94:0/2797975947 >> 172.21.15.94:0/2797975947 conn(0x7f84900ac570 msgr2=0x7f84900ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:02.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f84977d7640 1 -- 172.21.15.94:0/2797975947 shutdown_connections 2024-03-06T07:21:02.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.891+0000 7f84977d7640 1 -- 172.21.15.94:0/2797975947 wait complete. 2024-03-06T07:21:02.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.892+0000 7f84977d7640 1 Processor -- start 2024-03-06T07:21:02.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.892+0000 7f84977d7640 1 -- start start 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f84977d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84900c88c0 0x7f8490152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f84977d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f849014f360 0x7f8490157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f84977d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8490158b60 0x7f8490146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f84977d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84900c7c30 con 0x7f8490158b60 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f84977d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84900c5a50 con 0x7f849014f360 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f84977d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84900c5bd0 con 0x7f84900c88c0 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8490158b60 0x7f8490146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8490158b60 0x7f8490146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38822/0 (socket says 172.21.15.94:38822) 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 -- 172.21.15.94:0/2943845732 learned_addr learned my addr 172.21.15.94:0/2943845732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495573640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84900c88c0 0x7f8490152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8494d72640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f849014f360 0x7f8490157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 -- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84900c88c0 msgr2=0x7f8490152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84900c88c0 0x7f8490152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 -- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f849014f360 msgr2=0x7f8490157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f849014f360 0x7f8490157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.893+0000 7f8495d74640 1 -- 172.21.15.94:0/2943845732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84900c00f0 con 0x7f8490158b60 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.894+0000 7f8495d74640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8490158b60 0x7f8490146c00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f848800c7b0 tx=0x7f848800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.894+0000 7f8485ffb640 1 -- 172.21.15.94:0/2943845732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488003ba0 con 0x7f8490158b60 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.894+0000 7f8485ffb640 1 -- 172.21.15.94:0/2943845732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488003d40 con 0x7f8490158b60 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.894+0000 7f8485ffb640 1 -- 172.21.15.94:0/2943845732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488005270 con 0x7f8490158b60 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.894+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84900c03d0 con 0x7f8490158b60 2024-03-06T07:21:02.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.894+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84900c1210 con 0x7f8490158b60 2024-03-06T07:21:02.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.895+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849004f330 con 0x7f8490158b60 2024-03-06T07:21:02.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.895+0000 7f8485ffb640 1 -- 172.21.15.94:0/2943845732 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8488005410 con 0x7f8490158b60 2024-03-06T07:21:02.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.896+0000 7f8485ffb640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f847406b500 0x7f847406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:02.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.896+0000 7f8485ffb640 1 -- 172.21.15.94:0/2943845732 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f848808a330 con 0x7f8490158b60 2024-03-06T07:21:02.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.896+0000 7f8495573640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f847406b500 0x7f847406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:02.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.896+0000 7f8495573640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f847406b500 0x7f847406d9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8480000f60 tx=0x7f8480007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:02.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:02.900+0000 7f8485ffb640 1 -- 172.21.15.94:0/2943845732 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8488058710 con 0x7f8490158b60 2024-03-06T07:21:03.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.071+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84900c8d50 con 0x7f8490158b60 2024-03-06T07:21:03.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.072+0000 7f8485ffb640 1 -- 172.21.15.94:0/2943845732 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+20682 (secure 0 0 0) 0x7f848805c7b0 con 0x7f8490158b60 2024-03-06T07:21:03.074 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:03.074 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":77,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:01.765912+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:03.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f847406b500 msgr2=0x7f847406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f847406b500 0x7f847406d9c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8480000f60 tx=0x7f8480007e20 comp rx=0 tx=0).stop 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8490158b60 msgr2=0x7f8490146c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8490158b60 0x7f8490146c00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f848800c7b0 tx=0x7f848800cbb0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 shutdown_connections 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f847406b500 0x7f847406d9c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8490158b60 0x7f8490146c00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f849014f360 0x7f8490157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 --2- 172.21.15.94:0/2943845732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84900c88c0 0x7f8490152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 >> 172.21.15.94:0/2943845732 conn(0x7f84900ac570 msgr2=0x7f84900c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 shutdown_connections 2024-03-06T07:21:03.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.074+0000 7f84977d7640 1 -- 172.21.15.94:0/2943845732 wait complete. 2024-03-06T07:21:03.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-06T07:21:03.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.188+0000 7f6683d5e640 1 Processor -- start 2024-03-06T07:21:03.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f6683d5e640 1 -- start start 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f6683d5e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f667c155c30 0x7f667c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f6683d5e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c0b0450 con 0x7f667c0c5380 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f6683d5e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c1560e0 con 0x7f667c0b0070 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f6683d5e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667c06cfb0 con 0x7f667c155c30 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f66822fb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f667c0c5380 0x7f667c14e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41886/0 (socket says 172.21.15.94:41886) 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f6681afa640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f667c0b0070 0x7f667c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38168/0 (socket says 172.21.15.94:38168) 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.189+0000 7f66822fb640 1 -- 172.21.15.94:0/3703521869 learned_addr learned my addr 172.21.15.94:0/3703521869 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f66812f9640 1 --2- 172.21.15.94:0/3703521869 >> v2:172.21.15.145:3300/0 conn(0x7f667c155c30 0x7f667c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f6680af8640 1 -- 172.21.15.94:0/3703521869 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382688349 0 0) 0x7f667c1560e0 con 0x7f667c0b0070 2024-03-06T07:21:03.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f6680af8640 1 -- 172.21.15.94:0/3703521869 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6668003520 con 0x7f667c0b0070 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f6680af8640 1 -- 172.21.15.94:0/3703521869 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070499030 0 0) 0x7f6674001110 con 0x7f667c0c5380 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f6680af8640 1 -- 172.21.15.94:0/3703521869 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f667c1560e0 con 0x7f667c0c5380 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f66812f9640 1 -- 172.21.15.94:0/3703521869 >> v1:172.21.15.145:6789/0 conn(0x7f667c0b0070 legacy=0x7f667c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f66812f9640 1 -- 172.21.15.94:0/3703521869 >> v1:172.21.15.94:6789/0 conn(0x7f667c0c5380 legacy=0x7f667c14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.190+0000 7f66812f9640 1 -- 172.21.15.94:0/3703521869 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0c77f0 con 0x7f667c155c30 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f66812f9640 1 --2- 172.21.15.94:0/3703521869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c155c30 0x7f667c06ca70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f666c0097a0 tx=0x7f666c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d68b46a34971684 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6680af8640 1 -- 172.21.15.94:0/3703521869 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c035020 con 0x7f667c155c30 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6680af8640 1 -- 172.21.15.94:0/3703521869 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c031830 con 0x7f667c155c30 2024-03-06T07:21:03.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6680af8640 1 -- 172.21.15.94:0/3703521869 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c031be0 con 0x7f667c155c30 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6683d5e640 1 -- 172.21.15.94:0/3703521869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c155c30 msgr2=0x7f667c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6683d5e640 1 --2- 172.21.15.94:0/3703521869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c155c30 0x7f667c06ca70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f666c0097a0 tx=0x7f666c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6683d5e640 1 -- 172.21.15.94:0/3703521869 shutdown_connections 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6683d5e640 1 --2- 172.21.15.94:0/3703521869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c155c30 0x7f667c06ca70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6683d5e640 1 -- 172.21.15.94:0/3703521869 >> 172.21.15.94:0/3703521869 conn(0x7f667c0ac590 msgr2=0x7f667c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6683d5e640 1 -- 172.21.15.94:0/3703521869 shutdown_connections 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.191+0000 7f6683d5e640 1 -- 172.21.15.94:0/3703521869 wait complete. 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.192+0000 7f6683d5e640 1 Processor -- start 2024-03-06T07:21:03.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.192+0000 7f6683d5e640 1 -- start start 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f6683d5e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c0b0070 0x7f667c1eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f6683d5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f667c0c5380 0x7f667c0c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f6683d5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f667c155c30 0x7f667c14dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f6683d5e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c06c270 con 0x7f667c0c5380 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f6683d5e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c06c020 con 0x7f667c0b0070 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f6683d5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c06c550 con 0x7f667c155c30 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f667c0c5380 0x7f667c0c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f667c0c5380 0x7f667c0c1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38836/0 (socket says 172.21.15.94:38836) 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 -- 172.21.15.94:0/1654503215 learned_addr learned my addr 172.21.15.94:0/1654503215 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f6681afa640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c0b0070 0x7f667c1eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66822fb640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f667c155c30 0x7f667c14dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 -- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f667c155c30 msgr2=0x7f667c14dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f667c155c30 0x7f667c14dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 -- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c0b0070 msgr2=0x7f667c1eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c0b0070 0x7f667c1eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66812f9640 1 -- 172.21.15.94:0/1654503215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c14ccb0 con 0x7f667c0c5380 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.193+0000 7f66822fb640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f667c155c30 0x7f667c14dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.194+0000 7f66812f9640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f667c0c5380 0x7f667c0c1f00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f666c0098c0 tx=0x7f666c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.194+0000 7f66727fc640 1 -- 172.21.15.94:0/1654503215 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c035040 con 0x7f667c0c5380 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.194+0000 7f66727fc640 1 -- 172.21.15.94:0/1654503215 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c03f070 con 0x7f667c0c5380 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.194+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c14cf50 con 0x7f667c0c5380 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.194+0000 7f66727fc640 1 -- 172.21.15.94:0/1654503215 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c03bb60 con 0x7f667c0c5380 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.194+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c14d510 con 0x7f667c0c5380 2024-03-06T07:21:03.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.195+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667c04f330 con 0x7f667c0c5380 2024-03-06T07:21:03.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.195+0000 7f66727fc640 1 -- 172.21.15.94:0/1654503215 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f666c038020 con 0x7f667c0c5380 2024-03-06T07:21:03.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.196+0000 7f66727fc640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f666006b4d0 0x7f666006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.196+0000 7f66727fc640 1 -- 172.21.15.94:0/1654503215 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f666c0ac070 con 0x7f667c0c5380 2024-03-06T07:21:03.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.196+0000 7f6681afa640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f666006b4d0 0x7f666006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.196+0000 7f6681afa640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f666006b4d0 0x7f666006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6678001490 tx=0x7f6678001640 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.200+0000 7f66727fc640 1 -- 172.21.15.94:0/1654503215 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f666c07a450 con 0x7f667c0c5380 2024-03-06T07:21:03.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.404+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f667c06d860 con 0x7f667c0c5380 2024-03-06T07:21:03.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.455+0000 7f66727fc640 1 -- 172.21.15.94:0/1654503215 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v57) v1 ==== 119+0+0 (secure 0 0 0) 0x7f666c07e4f0 con 0x7f667c0c5380 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.457+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f666006b4d0 msgr2=0x7f666006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.457+0000 7f6683d5e640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f666006b4d0 0x7f666006d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6678001490 tx=0x7f6678001640 comp rx=0 tx=0).stop 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.457+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f667c0c5380 msgr2=0x7f667c0c1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.457+0000 7f6683d5e640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f667c0c5380 0x7f667c0c1f00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f666c0098c0 tx=0x7f666c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 shutdown_connections 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f666006b4d0 0x7f666006d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f667c155c30 0x7f667c14dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f667c0c5380 0x7f667c0c1f00 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 --2- 172.21.15.94:0/1654503215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f667c0b0070 0x7f667c1eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 >> 172.21.15.94:0/1654503215 conn(0x7f667c0ac590 msgr2=0x7f667c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 shutdown_connections 2024-03-06T07:21:03.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.458+0000 7f6683d5e640 1 -- 172.21.15.94:0/1654503215 wait complete. 2024-03-06T07:21:03.470 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:03.470 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-06T07:21:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.571+0000 7f7e80876640 1 Processor -- start 2024-03-06T07:21:03.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.571+0000 7f7e80876640 1 -- start start 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e80876640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7e7c1538c0 0x7f7e7c1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e80876640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7e7c0a97c0 0x7f7e7c1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e80876640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7c154330 con 0x7f7e7c0ab310 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e80876640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c0c46a0 con 0x7f7e7c0a97c0 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e80876640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c153dc0 con 0x7f7e7c1538c0 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7ad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7e7c0a97c0 0x7f7e7c1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7ad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7e7c0a97c0 0x7f7e7c1499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38842/0 (socket says 172.21.15.94:38842) 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7a59c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f7e7c0ab310 0x7f7e7c0ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38182/0 (socket says 172.21.15.94:38182) 2024-03-06T07:21:03.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7ad9d640 1 -- 172.21.15.94:0/2550108276 learned_addr learned my addr 172.21.15.94:0/2550108276 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e79d9b640 1 --2- 172.21.15.94:0/2550108276 >> v2:172.21.15.145:3300/0 conn(0x7f7e7c1538c0 0x7f7e7c1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7ad9d640 1 -- 172.21.15.94:0/2550108276 >> v2:172.21.15.145:3300/0 conn(0x7f7e7c1538c0 msgr2=0x7f7e7c1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7ad9d640 1 --2- 172.21.15.94:0/2550108276 >> v2:172.21.15.145:3300/0 conn(0x7f7e7c1538c0 0x7f7e7c1494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7ad9d640 1 -- 172.21.15.94:0/2550108276 >> v1:172.21.15.145:6789/0 conn(0x7f7e7c0ab310 legacy=0x7f7e7c0ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7959a640 1 -- 172.21.15.94:0/2550108276 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194314000 0 0) 0x7f7e7c154330 con 0x7f7e7c0ab310 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.572+0000 7f7e7ad9d640 1 -- 172.21.15.94:0/2550108276 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0c5d30 con 0x7f7e7c0a97c0 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.573+0000 7f7e7ad9d640 1 --2- 172.21.15.94:0/2550108276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0a97c0 0x7f7e7c1499e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0c75b0 tx=0x7f7e7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b48c63fe0a48f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.573+0000 7f7e7959a640 1 -- 172.21.15.94:0/2550108276 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7402ca80 con 0x7f7e7c0a97c0 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.573+0000 7f7e7959a640 1 -- 172.21.15.94:0/2550108276 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7402cc20 con 0x7f7e7c0a97c0 2024-03-06T07:21:03.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.573+0000 7f7e7959a640 1 -- 172.21.15.94:0/2550108276 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e740343f0 con 0x7f7e7c0a97c0 2024-03-06T07:21:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 -- 172.21.15.94:0/2550108276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0a97c0 msgr2=0x7f7e7c1499e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 --2- 172.21.15.94:0/2550108276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0a97c0 0x7f7e7c1499e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0c75b0 tx=0x7f7e7402bee0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 -- 172.21.15.94:0/2550108276 shutdown_connections 2024-03-06T07:21:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 --2- 172.21.15.94:0/2550108276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0a97c0 0x7f7e7c1499e0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 --2- 172.21.15.94:0/2550108276 >> v2:172.21.15.145:3300/0 conn(0x7f7e7c1538c0 0x7f7e7c1494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 -- 172.21.15.94:0/2550108276 >> 172.21.15.94:0/2550108276 conn(0x7f7e7c0c6710 msgr2=0x7f7e7c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 -- 172.21.15.94:0/2550108276 shutdown_connections 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.574+0000 7f7e80876640 1 -- 172.21.15.94:0/2550108276 wait complete. 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.575+0000 7f7e80876640 1 Processor -- start 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.575+0000 7f7e80876640 1 -- start start 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e80876640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e7c0a97c0 0x7f7e7c0720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e80876640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0ab310 0x7f7e7c14f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e80876640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e7c1538c0 0x7f7e7c14fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e80876640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0c4b30 con 0x7f7e7c0ab310 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e80876640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0c62f0 con 0x7f7e7c0a97c0 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e80876640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0c4cb0 con 0x7f7e7c1538c0 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0ab310 0x7f7e7c14f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0ab310 0x7f7e7c14f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38854/0 (socket says 172.21.15.94:38854) 2024-03-06T07:21:03.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 -- 172.21.15.94:0/2020194163 learned_addr learned my addr 172.21.15.94:0/2020194163 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e7a59c640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e7c0a97c0 0x7f7e7c0720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e7ad9d640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e7c1538c0 0x7f7e7c14fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 -- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e7c1538c0 msgr2=0x7f7e7c14fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e7c1538c0 0x7f7e7c14fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 -- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e7c0a97c0 msgr2=0x7f7e7c0720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e7c0a97c0 0x7f7e7c0720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 -- 172.21.15.94:0/2020194163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0c0a10 con 0x7f7e7c0ab310 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.576+0000 7f7e79d9b640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0ab310 0x7f7e7c14f730 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7e70012fd0 tx=0x7f7e7000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.577+0000 7f7e6affd640 1 -- 172.21.15.94:0/2020194163 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7001f070 con 0x7f7e7c0ab310 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.577+0000 7f7e6affd640 1 -- 172.21.15.94:0/2020194163 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7000ce30 con 0x7f7e7c0ab310 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.577+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c1e8da0 con 0x7f7e7c0ab310 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.577+0000 7f7e6affd640 1 -- 172.21.15.94:0/2020194163 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e70012250 con 0x7f7e7c0ab310 2024-03-06T07:21:03.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.577+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c072600 con 0x7f7e7c0ab310 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.578+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c0bb730 con 0x7f7e7c0ab310 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.578+0000 7f7e6affd640 1 -- 172.21.15.94:0/2020194163 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e70022070 con 0x7f7e7c0ab310 2024-03-06T07:21:03.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.578+0000 7f7e6affd640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.578+0000 7f7e6affd640 1 -- 172.21.15.94:0/2020194163 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7e7008e250 con 0x7f7e7c0ab310 2024-03-06T07:21:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.578+0000 7f7e7a59c640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:03.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.579+0000 7f7e7a59c640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c000f60 tx=0x7f7e6c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:03.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.583+0000 7f7e6affd640 1 -- 172.21.15.94:0/2020194163 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e7005c630 con 0x7f7e7c0ab310 2024-03-06T07:21:03.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:03.791+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7e7c04f330 con 0x7f7e7c0ab310 2024-03-06T07:21:04.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.459+0000 7f7e6affd640 1 -- 172.21.15.94:0/2020194163 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v58) v1 ==== 123+0+0 (secure 0 0 0) 0x7f7e700606d0 con 0x7f7e7c0ab310 2024-03-06T07:21:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e5c06b4d0 msgr2=0x7f7e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c000f60 tx=0x7f7e6c007d50 comp rx=0 tx=0).stop 2024-03-06T07:21:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0ab310 msgr2=0x7f7e7c14f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0ab310 0x7f7e7c14f730 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7e70012fd0 tx=0x7f7e7000c520 comp rx=0 tx=0).stop 2024-03-06T07:21:04.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 shutdown_connections 2024-03-06T07:21:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e7c1538c0 0x7f7e7c14fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e7c0ab310 0x7f7e7c14f730 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 --2- 172.21.15.94:0/2020194163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e7c0a97c0 0x7f7e7c0720c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 >> 172.21.15.94:0/2020194163 conn(0x7f7e7c0c6710 msgr2=0x7f7e7c0ab6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 shutdown_connections 2024-03-06T07:21:04.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.462+0000 7f7e80876640 1 -- 172.21.15.94:0/2020194163 wait complete. 2024-03-06T07:21:04.477 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-06T07:21:04.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:04.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.581+0000 7fc709565640 1 Processor -- start 2024-03-06T07:21:04.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.581+0000 7fc709565640 1 -- start start 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc709565640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc70414f790 0x7fc7041529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc709565640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc704155a40 0x7fc704155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc709565640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc704156410 con 0x7fc7040c1aa0 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc709565640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040c2680 con 0x7fc704155a40 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc709565640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040c28d0 con 0x7fc70414f790 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc7037fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc704155a40 0x7fc704155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc7037fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc704155a40 0x7fc704155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38868/0 (socket says 172.21.15.94:38868) 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc7037fe640 1 -- 172.21.15.94:0/3872867917 learned_addr learned my addr 172.21.15.94:0/3872867917 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc702ffd640 1 --2- 172.21.15.94:0/3872867917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc70414f790 0x7fc7041529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc7037fe640 1 -- 172.21.15.94:0/3872867917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc70414f790 msgr2=0x7fc7041529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc7037fe640 1 --2- 172.21.15.94:0/3872867917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc70414f790 0x7fc7041529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc7037fe640 1 -- 172.21.15.94:0/3872867917 >> v1:172.21.15.145:6789/0 conn(0x7fc7040c1aa0 legacy=0x7fc7040c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.582+0000 7fc7037fe640 1 -- 172.21.15.94:0/3872867917 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040c2bb0 con 0x7fc704155a40 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.583+0000 7fc7037fe640 1 --2- 172.21.15.94:0/3872867917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704155ed0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00d150 tx=0x7fc6fc02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c6c9ed22ccab49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.583+0000 7fc701ffb640 1 -- 172.21.15.94:0/3872867917 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00b0d0 con 0x7fc704155a40 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.583+0000 7fc701ffb640 1 -- 172.21.15.94:0/3872867917 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc00b270 con 0x7fc704155a40 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.583+0000 7fc701ffb640 1 -- 172.21.15.94:0/3872867917 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc009600 con 0x7fc704155a40 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 -- 172.21.15.94:0/3872867917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 msgr2=0x7fc704155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 --2- 172.21.15.94:0/3872867917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704155ed0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00d150 tx=0x7fc6fc02e160 comp rx=0 tx=0).stop 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 -- 172.21.15.94:0/3872867917 shutdown_connections 2024-03-06T07:21:04.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 --2- 172.21.15.94:0/3872867917 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704155ed0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 --2- 172.21.15.94:0/3872867917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc70414f790 0x7fc7041529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 -- 172.21.15.94:0/3872867917 >> 172.21.15.94:0/3872867917 conn(0x7fc7040c6710 msgr2=0x7fc7040c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 -- 172.21.15.94:0/3872867917 shutdown_connections 2024-03-06T07:21:04.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.584+0000 7fc709565640 1 -- 172.21.15.94:0/3872867917 wait complete. 2024-03-06T07:21:04.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.585+0000 7fc709565640 1 Processor -- start 2024-03-06T07:21:04.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.585+0000 7fc709565640 1 -- start start 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7040c1aa0 0x7fc7041e7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc70414f790 0x7fc704151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040c0d60 con 0x7fc704155a40 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc704155520 con 0x7fc70414f790 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040c5e30 con 0x7fc7040c1aa0 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38882/0 (socket says 172.21.15.94:38882) 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 -- 172.21.15.94:0/2477996525 learned_addr learned my addr 172.21.15.94:0/2477996525 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc702ffd640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7040c1aa0 0x7fc7041e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7027fc640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc70414f790 0x7fc704151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 -- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7040c1aa0 msgr2=0x7fc7041e7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7040c1aa0 0x7fc7041e7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 -- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc70414f790 msgr2=0x7fc704151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc70414f790 0x7fc704151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 -- 172.21.15.94:0/2477996525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704150cf0 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc702ffd640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7040c1aa0 0x7fc7041e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc7037fe640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704071300 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc038040 tx=0x7fc6fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc6eb7fe640 1 -- 172.21.15.94:0/2477996525 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc045020 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc6eb7fe640 1 -- 172.21.15.94:0/2477996525 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc009e30 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc704150fd0 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc6eb7fe640 1 -- 172.21.15.94:0/2477996525 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00b0d0 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.586+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc704151f90 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.587+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70404f330 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.588+0000 7fc6eb7fe640 1 -- 172.21.15.94:0/2477996525 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6fc048020 con 0x7fc704155a40 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.588+0000 7fc6eb7fe640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6f006b500 0x7fc6f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.588+0000 7fc6eb7fe640 1 -- 172.21.15.94:0/2477996525 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc6fc0aef60 con 0x7fc704155a40 2024-03-06T07:21:04.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.588+0000 7fc702ffd640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6f006b500 0x7fc6f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.589+0000 7fc702ffd640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6f006b500 0x7fc6f006d9c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009f30 tx=0x7fc6f4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.593+0000 7fc6eb7fe640 1 -- 172.21.15.94:0/2477996525 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6fc04f030 con 0x7fc704155a40 2024-03-06T07:21:04.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.804+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc70406d550 con 0x7fc704155a40 2024-03-06T07:21:04.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.805+0000 7fc6eb7fe640 1 -- 172.21.15.94:0/2477996525 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6277 (secure 0 0 0) 0x7fc6fc07d340 con 0x7fc704155a40 2024-03-06T07:21:04.807 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:04.807 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":58,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:20:53.531385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35492":{"gid":35492,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3537750458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3537750458},{"type":"v1","addr":"172.21.15.145:6837","nonce":3537750458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35492,"qdb_cluster":[35492]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:04.807 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 58 2024-03-06T07:21:04.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6f006b500 msgr2=0x7fc6f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6f006b500 0x7fc6f006d9c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009f30 tx=0x7fc6f4009190 comp rx=0 tx=0).stop 2024-03-06T07:21:04.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 msgr2=0x7fc704071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704071300 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc038040 tx=0x7fc6fc004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:04.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 shutdown_connections 2024-03-06T07:21:04.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6f006b500 0x7fc6f006d9c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc704155a40 0x7fc704071300 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc70414f790 0x7fc704151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 --2- 172.21.15.94:0/2477996525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7040c1aa0 0x7fc7041e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 >> 172.21.15.94:0/2477996525 conn(0x7fc7040c6710 msgr2=0x7fc7040c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 shutdown_connections 2024-03-06T07:21:04.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.807+0000 7fc709565640 1 -- 172.21.15.94:0/2477996525 wait complete. 2024-03-06T07:21:04.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:04.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.918+0000 7f1efbbc7640 1 Processor -- start 2024-03-06T07:21:04.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.918+0000 7f1efbbc7640 1 -- start start 2024-03-06T07:21:04.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1efbbc7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ef414f790 0x7f1ef41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1efbbc7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef40c28d0 con 0x7f1ef4155a40 2024-03-06T07:21:04.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1efbbc7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef40c2bb0 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1efbbc7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef40c2680 con 0x7f1ef414f790 2024-03-06T07:21:04.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1efa164640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1ef4155a40 0x7f1ef4155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41888/0 (socket says 172.21.15.94:41888) 2024-03-06T07:21:04.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1efa164640 1 -- 172.21.15.94:0/2450039144 learned_addr learned my addr 172.21.15.94:0/2450039144 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:04.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1ef9963640 1 --2- 172.21.15.94:0/2450039144 >> v2:172.21.15.145:3300/0 conn(0x7f1ef414f790 0x7f1ef41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1ef8961640 1 -- 172.21.15.94:0/2450039144 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586053406 0 0) 0x7f1ef40c28d0 con 0x7f1ef4155a40 2024-03-06T07:21:04.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1ef8961640 1 -- 172.21.15.94:0/2450039144 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee0003520 con 0x7f1ef4155a40 2024-03-06T07:21:04.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1ef9963640 1 -- 172.21.15.94:0/2450039144 >> v1:172.21.15.145:6789/0 conn(0x7f1ef40c1aa0 legacy=0x7f1ef40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1ef9963640 1 -- 172.21.15.94:0/2450039144 >> v1:172.21.15.94:6789/0 conn(0x7f1ef4155a40 legacy=0x7f1ef4155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1ef8961640 1 -- 172.21.15.94:0/2450039144 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990016206 0 0) 0x7f1ef40c2bb0 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.919+0000 7f1ef9963640 1 -- 172.21.15.94:0/2450039144 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef40c2e90 con 0x7f1ef414f790 2024-03-06T07:21:04.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.920+0000 7f1ef9963640 1 --2- 172.21.15.94:0/2450039144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef414f790 0x7f1ef41529d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40c2800 tx=0x7f1ee402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3070301ee90e7af2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.920+0000 7f1ef8961640 1 -- 172.21.15.94:0/2450039144 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4034070 con 0x7f1ef414f790 2024-03-06T07:21:04.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.920+0000 7f1ef8961640 1 -- 172.21.15.94:0/2450039144 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee4030610 con 0x7f1ef414f790 2024-03-06T07:21:04.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.920+0000 7f1ef8961640 1 -- 172.21.15.94:0/2450039144 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee40309e0 con 0x7f1ef414f790 2024-03-06T07:21:04.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.921+0000 7f1efbbc7640 1 -- 172.21.15.94:0/2450039144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef414f790 msgr2=0x7f1ef41529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.921+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/2450039144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef414f790 0x7f1ef41529d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40c2800 tx=0x7f1ee402b960 comp rx=0 tx=0).stop 2024-03-06T07:21:04.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.921+0000 7f1efbbc7640 1 -- 172.21.15.94:0/2450039144 shutdown_connections 2024-03-06T07:21:04.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.921+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/2450039144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef414f790 0x7f1ef41529d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.921+0000 7f1efbbc7640 1 -- 172.21.15.94:0/2450039144 >> 172.21.15.94:0/2450039144 conn(0x7f1ef40c6710 msgr2=0x7f1ef40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:04.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.921+0000 7f1efbbc7640 1 -- 172.21.15.94:0/2450039144 shutdown_connections 2024-03-06T07:21:04.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.921+0000 7f1efbbc7640 1 -- 172.21.15.94:0/2450039144 wait complete. 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 Processor -- start 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 -- start start 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef40c1aa0 0x7f1ef41ec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef414f790 0x7f1ef4148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef4155a40 0x7f1ef4154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef40c5e30 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef40c0d60 con 0x7f1ef4155a40 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.922+0000 7f1efbbc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef40c5fb0 con 0x7f1ef414f790 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef40c1aa0 0x7f1ef41ec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef40c1aa0 0x7f1ef41ec5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38892/0 (socket says 172.21.15.94:38892) 2024-03-06T07:21:04.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 -- 172.21.15.94:0/604200471 learned_addr learned my addr 172.21.15.94:0/604200471 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1efa164640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef4155a40 0x7f1ef4154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9162640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef414f790 0x7f1ef4148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 -- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef414f790 msgr2=0x7f1ef4148ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef414f790 0x7f1ef4148ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 -- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef4155a40 msgr2=0x7f1ef4154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef4155a40 0x7f1ef4154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 -- 172.21.15.94:0/604200471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef41ecb00 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9162640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef414f790 0x7f1ef4148ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1efa164640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef4155a40 0x7f1ef4154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1ef9963640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef40c1aa0 0x7f1ef41ec5c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1ee402b9a0 tx=0x7f1ee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1eea7fc640 1 -- 172.21.15.94:0/604200471 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4034040 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1eea7fc640 1 -- 172.21.15.94:0/604200471 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee4030410 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1eea7fc640 1 -- 172.21.15.94:0/604200471 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4039c30 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef41490e0 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.923+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef414d180 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.924+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef404f330 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.925+0000 7f1eea7fc640 1 -- 172.21.15.94:0/604200471 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ee4037070 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.925+0000 7f1eea7fc640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed806b4d0 0x7f1ed806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.925+0000 7f1eea7fc640 1 -- 172.21.15.94:0/604200471 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f1ee40ac690 con 0x7f1ef40c1aa0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.925+0000 7f1ef9162640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed806b4d0 0x7f1ed806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:04.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.925+0000 7f1ef9162640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed806b4d0 0x7f1ed806d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1ef41552f0 tx=0x7f1ef0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:04.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:04.929+0000 7f1eea7fc640 1 -- 172.21.15.94:0/604200471 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ee4038e90 con 0x7f1ef40c1aa0 2024-03-06T07:21:05.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.100+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ef40c1f40 con 0x7f1ef40c1aa0 2024-03-06T07:21:05.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.102+0000 7f1eea7fc640 1 -- 172.21.15.94:0/604200471 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+20669 (secure 0 0 0) 0x7f1ee407b040 con 0x7f1ef40c1aa0 2024-03-06T07:21:05.104 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:05.104 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":78,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:03.452806+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:05.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed806b4d0 msgr2=0x7f1ed806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed806b4d0 0x7f1ed806d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1ef41552f0 tx=0x7f1ef0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:05.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef40c1aa0 msgr2=0x7f1ef41ec5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef40c1aa0 0x7f1ef41ec5c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1ee402b9a0 tx=0x7f1ee40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 shutdown_connections 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ed806b4d0 0x7f1ed806d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ef4155a40 0x7f1ef4154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ef414f790 0x7f1ef4148ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 --2- 172.21.15.94:0/604200471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ef40c1aa0 0x7f1ef41ec5c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 >> 172.21.15.94:0/604200471 conn(0x7f1ef40c6710 msgr2=0x7f1ef40c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 shutdown_connections 2024-03-06T07:21:05.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.104+0000 7f1efbbc7640 1 -- 172.21.15.94:0/604200471 wait complete. 2024-03-06T07:21:05.117 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:21:05.117 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:05.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 Processor -- start 2024-03-06T07:21:05.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 -- start start 2024-03-06T07:21:05.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d8014f790 0x7f0d801529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0d800c1aa0 0x7f0d800c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d800c5c60 0x7f0d80155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d800c2680 con 0x7f0d8014f790 2024-03-06T07:21:05.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d800c28d0 con 0x7f0d800c1aa0 2024-03-06T07:21:05.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.216+0000 7f0d8484e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d800c2bb0 con 0x7f0d800c5c60 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d8014f790 0x7f0d801529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d8014f790 0x7f0d801529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38906/0 (socket says 172.21.15.94:38906) 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 -- 172.21.15.94:0/522678341 learned_addr learned my addr 172.21.15.94:0/522678341 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7ed9d640 1 --2- 172.21.15.94:0/522678341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d800c5c60 0x7f0d80155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7dd9b640 1 --2- 172.21.15.94:0/522678341 >> v2:172.21.15.145:3300/0 conn(0x7f0d800c1aa0 0x7f0d800c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 -- 172.21.15.94:0/522678341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d800c5c60 msgr2=0x7f0d80155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 --2- 172.21.15.94:0/522678341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d800c5c60 0x7f0d80155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 -- 172.21.15.94:0/522678341 >> v2:172.21.15.145:3300/0 conn(0x7f0d800c1aa0 msgr2=0x7f0d800c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 --2- 172.21.15.94:0/522678341 >> v2:172.21.15.145:3300/0 conn(0x7f0d800c1aa0 0x7f0d800c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.217+0000 7f0d7e59c640 1 -- 172.21.15.94:0/522678341 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d800c2e90 con 0x7f0d8014f790 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d7e59c640 1 --2- 172.21.15.94:0/522678341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d8014f790 0x7f0d801529d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0d80070280 tx=0x7f0d7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5a7a8688de1464 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d7d59a640 1 -- 172.21.15.94:0/522678341 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d74035020 con 0x7f0d8014f790 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d7d59a640 1 -- 172.21.15.94:0/522678341 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d74031870 con 0x7f0d8014f790 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d7d59a640 1 -- 172.21.15.94:0/522678341 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d74031c40 con 0x7f0d8014f790 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 -- 172.21.15.94:0/522678341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d8014f790 msgr2=0x7f0d801529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 --2- 172.21.15.94:0/522678341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d8014f790 0x7f0d801529d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0d80070280 tx=0x7f0d7402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 -- 172.21.15.94:0/522678341 shutdown_connections 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 --2- 172.21.15.94:0/522678341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d800c5c60 0x7f0d80155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 --2- 172.21.15.94:0/522678341 >> v2:172.21.15.145:3300/0 conn(0x7f0d800c1aa0 0x7f0d800c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 --2- 172.21.15.94:0/522678341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d8014f790 0x7f0d801529d0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 -- 172.21.15.94:0/522678341 >> 172.21.15.94:0/522678341 conn(0x7f0d800c6710 msgr2=0x7f0d800c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.218+0000 7f0d8484e640 1 -- 172.21.15.94:0/522678341 shutdown_connections 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.219+0000 7f0d8484e640 1 -- 172.21.15.94:0/522678341 wait complete. 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 Processor -- start 2024-03-06T07:21:05.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 -- start start 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d800c1aa0 0x7f0d801e2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d800c5c60 0x7f0d8014be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d8014f790 0x7f0d800a74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8006cc00 con 0x7f0d800c5c60 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d800c0d60 con 0x7f0d800c1aa0 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d8484e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8006cd80 con 0x7f0d8014f790 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d800c5c60 0x7f0d8014be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d7dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d800c5c60 0x7f0d8014be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38910/0 (socket says 172.21.15.94:38910) 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.220+0000 7f0d7dd9b640 1 -- 172.21.15.94:0/3125066623 learned_addr learned my addr 172.21.15.94:0/3125066623 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:05.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7ed9d640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d8014f790 0x7f0d800a74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7e59c640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d800c1aa0 0x7f0d801e2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7dd9b640 1 -- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d8014f790 msgr2=0x7f0d800a74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7dd9b640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d8014f790 0x7f0d800a74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7dd9b640 1 -- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d800c1aa0 msgr2=0x7f0d801e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7dd9b640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d800c1aa0 0x7f0d801e2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7dd9b640 1 -- 172.21.15.94:0/3125066623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8006d9d0 con 0x7f0d800c5c60 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d7dd9b640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d800c5c60 0x7f0d8014be40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000e890 tx=0x7f0d7000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d6effd640 1 -- 172.21.15.94:0/3125066623 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d70017070 con 0x7f0d800c5c60 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d6effd640 1 -- 172.21.15.94:0/3125066623 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7000c940 con 0x7f0d800c5c60 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d6effd640 1 -- 172.21.15.94:0/3125066623 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000cd10 con 0x7f0d800c5c60 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d80070b70 con 0x7f0d800c5c60 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.221+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8014da60 con 0x7f0d800c5c60 2024-03-06T07:21:05.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.222+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8004f330 con 0x7f0d800c5c60 2024-03-06T07:21:05.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.223+0000 7f0d6effd640 1 -- 172.21.15.94:0/3125066623 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d7001a070 con 0x7f0d800c5c60 2024-03-06T07:21:05.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.223+0000 7f0d6effd640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d6006b4d0 0x7f0d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.223+0000 7f0d6effd640 1 -- 172.21.15.94:0/3125066623 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f0d7008a570 con 0x7f0d800c5c60 2024-03-06T07:21:05.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.223+0000 7f0d7e59c640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d6006b4d0 0x7f0d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.224+0000 7f0d7e59c640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d6006b4d0 0x7f0d6006d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0d74036000 tx=0x7f0d7403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.227+0000 7f0d6effd640 1 -- 172.21.15.94:0/3125066623 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d70058950 con 0x7f0d800c5c60 2024-03-06T07:21:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.394+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d80073320 con 0x7f0d6006b4d0 2024-03-06T07:21:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.405+0000 7f0d6effd640 1 -- 172.21.15.94:0/3125066623 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485419 (secure 0 0 0) 0x7f0d740be840 con 0x7f0d6006b4d0 2024-03-06T07:21:05.413 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:05.416 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:21:05.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d6006b4d0 msgr2=0x7f0d6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d6006b4d0 0x7f0d6006d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0d74036000 tx=0x7f0d7403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:05.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d800c5c60 msgr2=0x7f0d8014be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d800c5c60 0x7f0d8014be40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000e890 tx=0x7f0d7000ed50 comp rx=0 tx=0).stop 2024-03-06T07:21:05.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 shutdown_connections 2024-03-06T07:21:05.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d6006b4d0 0x7f0d6006d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d8014f790 0x7f0d800a74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d800c5c60 0x7f0d8014be40 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 --2- 172.21.15.94:0/3125066623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d800c1aa0 0x7f0d801e2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 >> 172.21.15.94:0/3125066623 conn(0x7f0d800c6710 msgr2=0x7f0d8006cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 shutdown_connections 2024-03-06T07:21:05.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.409+0000 7f0d8484e640 1 -- 172.21.15.94:0/3125066623 wait complete. 2024-03-06T07:21:05.422 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-03-06T07:21:04.671494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244424,"kb_used_data":7456,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730296,"statfs":{"total":773094113280,"available":772843823104,"internally_reserved":0,"allocated":7634944,"data_stored":4714804,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.252754"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427479+0000","last_change":"2024-03-06T07:21:01.427479+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427479+0000","last_undegraded":"2024-03-06T07:21:01.427479+0000","last_fullsized":"2024-03-06T07:21:01.427479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431835+0000","last_change":"2024-03-06T07:20:51.896309+0000","last_active":"2024-03-06T07:21:01.431835+0000","last_peered":"2024-03-06T07:21:01.431835+0000","last_clean":"2024-03-06T07:21:01.431835+0000","last_became_active":"2024-03-06T07:20:51.896009+0000","last_became_peered":"2024-03-06T07:20:51.896009+0000","last_unstale":"2024-03-06T07:21:01.431835+0000","last_undegraded":"2024-03-06T07:21:01.431835+0000","last_fullsized":"2024-03-06T07:21:01.431835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426920+0000","last_change":"2024-03-06T07:21:01.426920+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426920+0000","last_undegraded":"2024-03-06T07:21:01.426920+0000","last_fullsized":"2024-03-06T07:21:01.426920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419624+0000","last_change":"2024-03-06T07:20:51.895412+0000","last_active":"2024-03-06T07:21:01.419624+0000","last_peered":"2024-03-06T07:21:01.419624+0000","last_clean":"2024-03-06T07:21:01.419624+0000","last_became_active":"2024-03-06T07:20:51.895169+0000","last_became_peered":"2024-03-06T07:20:51.895169+0000","last_unstale":"2024-03-06T07:21:01.419624+0000","last_undegraded":"2024-03-06T07:21:01.419624+0000","last_fullsized":"2024-03-06T07:21:01.419624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427931+0000","last_change":"2024-03-06T07:21:01.427931+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427931+0000","last_undegraded":"2024-03-06T07:21:01.427931+0000","last_fullsized":"2024-03-06T07:21:01.427931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428431+0000","last_change":"2024-03-06T07:20:51.878718+0000","last_active":"2024-03-06T07:21:01.428431+0000","last_peered":"2024-03-06T07:21:01.428431+0000","last_clean":"2024-03-06T07:21:01.428431+0000","last_became_active":"2024-03-06T07:20:51.878467+0000","last_became_peered":"2024-03-06T07:20:51.878467+0000","last_unstale":"2024-03-06T07:21:01.428431+0000","last_undegraded":"2024-03-06T07:21:01.428431+0000","last_fullsized":"2024-03-06T07:21:01.428431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428790+0000","last_change":"2024-03-06T07:21:01.428790+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428790+0000","last_undegraded":"2024-03-06T07:21:01.428790+0000","last_fullsized":"2024-03-06T07:21:01.428790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431362+0000","last_change":"2024-03-06T07:20:51.892550+0000","last_active":"2024-03-06T07:21:01.431362+0000","last_peered":"2024-03-06T07:21:01.431362+0000","last_clean":"2024-03-06T07:21:01.431362+0000","last_became_active":"2024-03-06T07:20:51.892264+0000","last_became_peered":"2024-03-06T07:20:51.892264+0000","last_unstale":"2024-03-06T07:21:01.431362+0000","last_undegraded":"2024-03-06T07:21:01.431362+0000","last_fullsized":"2024-03-06T07:21:01.431362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429413+0000","last_change":"2024-03-06T07:21:01.429413+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429413+0000","last_undegraded":"2024-03-06T07:21:01.429413+0000","last_fullsized":"2024-03-06T07:21:01.429413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428191+0000","last_change":"2024-03-06T07:20:51.893305+0000","last_active":"2024-03-06T07:21:01.428191+0000","last_peered":"2024-03-06T07:21:01.428191+0000","last_clean":"2024-03-06T07:21:01.428191+0000","last_became_active":"2024-03-06T07:20:51.893041+0000","last_became_peered":"2024-03-06T07:20:51.893041+0000","last_unstale":"2024-03-06T07:21:01.428191+0000","last_undegraded":"2024-03-06T07:21:01.428191+0000","last_fullsized":"2024-03-06T07:21:01.428191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:39.229585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424683+0000","last_change":"2024-03-06T07:21:01.424683+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424683+0000","last_undegraded":"2024-03-06T07:21:01.424683+0000","last_fullsized":"2024-03-06T07:21:01.424683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425180+0000","last_change":"2024-03-06T07:20:51.893812+0000","last_active":"2024-03-06T07:21:01.425180+0000","last_peered":"2024-03-06T07:21:01.425180+0000","last_clean":"2024-03-06T07:21:01.425180+0000","last_became_active":"2024-03-06T07:20:51.891649+0000","last_became_peered":"2024-03-06T07:20:51.891649+0000","last_unstale":"2024-03-06T07:21:01.425180+0000","last_undegraded":"2024-03-06T07:21:01.425180+0000","last_fullsized":"2024-03-06T07:21:01.425180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:46.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424861+0000","last_change":"2024-03-06T07:21:01.424861+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424861+0000","last_undegraded":"2024-03-06T07:21:01.424861+0000","last_fullsized":"2024-03-06T07:21:01.424861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426172+0000","last_change":"2024-03-06T07:20:51.894141+0000","last_active":"2024-03-06T07:21:01.426172+0000","last_peered":"2024-03-06T07:21:01.426172+0000","last_clean":"2024-03-06T07:21:01.426172+0000","last_became_active":"2024-03-06T07:20:51.893793+0000","last_became_peered":"2024-03-06T07:20:51.893793+0000","last_unstale":"2024-03-06T07:21:01.426172+0000","last_undegraded":"2024-03-06T07:21:01.426172+0000","last_fullsized":"2024-03-06T07:21:01.426172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430080+0000","last_change":"2024-03-06T07:21:01.430080+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430080+0000","last_undegraded":"2024-03-06T07:21:01.430080+0000","last_fullsized":"2024-03-06T07:21:01.430080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427332+0000","last_change":"2024-03-06T07:20:51.894037+0000","last_active":"2024-03-06T07:21:01.427332+0000","last_peered":"2024-03-06T07:21:01.427332+0000","last_clean":"2024-03-06T07:21:01.427332+0000","last_became_active":"2024-03-06T07:20:51.893773+0000","last_became_peered":"2024-03-06T07:20:51.893773+0000","last_unstale":"2024-03-06T07:21:01.427332+0000","last_undegraded":"2024-03-06T07:21:01.427332+0000","last_fullsized":"2024-03-06T07:21:01.427332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:48.344455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424697+0000","last_change":"2024-03-06T07:21:01.424697+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424697+0000","last_undegraded":"2024-03-06T07:21:01.424697+0000","last_fullsized":"2024-03-06T07:21:01.424697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427134+0000","last_change":"2024-03-06T07:20:51.886223+0000","last_active":"2024-03-06T07:21:01.427134+0000","last_peered":"2024-03-06T07:21:01.427134+0000","last_clean":"2024-03-06T07:21:01.427134+0000","last_became_active":"2024-03-06T07:20:51.885985+0000","last_became_peered":"2024-03-06T07:20:51.885985+0000","last_unstale":"2024-03-06T07:21:01.427134+0000","last_undegraded":"2024-03-06T07:21:01.427134+0000","last_fullsized":"2024-03-06T07:21:01.427134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:13.814018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426060+0000","last_change":"2024-03-06T07:21:01.426060+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426060+0000","last_undegraded":"2024-03-06T07:21:01.426060+0000","last_fullsized":"2024-03-06T07:21:01.426060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431693+0000","last_change":"2024-03-06T07:20:51.890465+0000","last_active":"2024-03-06T07:21:01.431693+0000","last_peered":"2024-03-06T07:21:01.431693+0000","last_clean":"2024-03-06T07:21:01.431693+0000","last_became_active":"2024-03-06T07:20:51.890183+0000","last_became_peered":"2024-03-06T07:20:51.890183+0000","last_unstale":"2024-03-06T07:21:01.431693+0000","last_undegraded":"2024-03-06T07:21:01.431693+0000","last_fullsized":"2024-03-06T07:21:01.431693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425029+0000","last_change":"2024-03-06T07:21:01.425029+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425029+0000","last_undegraded":"2024-03-06T07:21:01.425029+0000","last_fullsized":"2024-03-06T07:21:01.425029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419915+0000","last_change":"2024-03-06T07:20:51.897836+0000","last_active":"2024-03-06T07:21:01.419915+0000","last_peered":"2024-03-06T07:21:01.419915+0000","last_clean":"2024-03-06T07:21:01.419915+0000","last_became_active":"2024-03-06T07:20:51.897551+0000","last_became_peered":"2024-03-06T07:20:51.897551+0000","last_unstale":"2024-03-06T07:21:01.419915+0000","last_undegraded":"2024-03-06T07:21:01.419915+0000","last_fullsized":"2024-03-06T07:21:01.419915+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424341+0000","last_change":"2024-03-06T07:21:01.424341+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424341+0000","last_undegraded":"2024-03-06T07:21:01.424341+0000","last_fullsized":"2024-03-06T07:21:01.424341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431445+0000","last_change":"2024-03-06T07:20:51.886475+0000","last_active":"2024-03-06T07:21:01.431445+0000","last_peered":"2024-03-06T07:21:01.431445+0000","last_clean":"2024-03-06T07:21:01.431445+0000","last_became_active":"2024-03-06T07:20:51.886218+0000","last_became_peered":"2024-03-06T07:20:51.886218+0000","last_unstale":"2024-03-06T07:21:01.431445+0000","last_undegraded":"2024-03-06T07:21:01.431445+0000","last_fullsized":"2024-03-06T07:21:01.431445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426115+0000","last_change":"2024-03-06T07:21:01.426115+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426115+0000","last_undegraded":"2024-03-06T07:21:01.426115+0000","last_fullsized":"2024-03-06T07:21:01.426115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428289+0000","last_change":"2024-03-06T07:20:51.887999+0000","last_active":"2024-03-06T07:21:01.428289+0000","last_peered":"2024-03-06T07:21:01.428289+0000","last_clean":"2024-03-06T07:21:01.428289+0000","last_became_active":"2024-03-06T07:20:51.887749+0000","last_became_peered":"2024-03-06T07:20:51.887749+0000","last_unstale":"2024-03-06T07:21:01.428289+0000","last_undegraded":"2024-03-06T07:21:01.428289+0000","last_fullsized":"2024-03-06T07:21:01.428289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:25.243948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.431576+0000","last_change":"2024-03-06T07:21:01.431576+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.431576+0000","last_undegraded":"2024-03-06T07:21:01.431576+0000","last_fullsized":"2024-03-06T07:21:01.431576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425276+0000","last_change":"2024-03-06T07:20:51.892140+0000","last_active":"2024-03-06T07:21:01.425276+0000","last_peered":"2024-03-06T07:21:01.425276+0000","last_clean":"2024-03-06T07:21:01.425276+0000","last_became_active":"2024-03-06T07:20:51.891195+0000","last_became_peered":"2024-03-06T07:20:51.891195+0000","last_unstale":"2024-03-06T07:21:01.425276+0000","last_undegraded":"2024-03-06T07:21:01.425276+0000","last_fullsized":"2024-03-06T07:21:01.425276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:28.641052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424813+0000","last_change":"2024-03-06T07:21:01.424813+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424813+0000","last_undegraded":"2024-03-06T07:21:01.424813+0000","last_fullsized":"2024-03-06T07:21:01.424813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419327+0000","last_change":"2024-03-06T07:20:51.894735+0000","last_active":"2024-03-06T07:21:01.419327+0000","last_peered":"2024-03-06T07:21:01.419327+0000","last_clean":"2024-03-06T07:21:01.419327+0000","last_became_active":"2024-03-06T07:20:51.894479+0000","last_became_peered":"2024-03-06T07:20:51.894479+0000","last_unstale":"2024-03-06T07:21:01.419327+0000","last_undegraded":"2024-03-06T07:21:01.419327+0000","last_fullsized":"2024-03-06T07:21:01.419327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425999+0000","last_change":"2024-03-06T07:21:01.425999+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425999+0000","last_undegraded":"2024-03-06T07:21:01.425999+0000","last_fullsized":"2024-03-06T07:21:01.425999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430687+0000","last_change":"2024-03-06T07:20:51.892450+0000","last_active":"2024-03-06T07:21:01.430687+0000","last_peered":"2024-03-06T07:21:01.430687+0000","last_clean":"2024-03-06T07:21:01.430687+0000","last_became_active":"2024-03-06T07:20:51.892184+0000","last_became_peered":"2024-03-06T07:20:51.892184+0000","last_unstale":"2024-03-06T07:21:01.430687+0000","last_undegraded":"2024-03-06T07:21:01.430687+0000","last_fullsized":"2024-03-06T07:21:01.430687+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424318+0000","last_change":"2024-03-06T07:21:01.424318+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424318+0000","last_undegraded":"2024-03-06T07:21:01.424318+0000","last_fullsized":"2024-03-06T07:21:01.424318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426749+0000","last_change":"2024-03-06T07:20:51.891492+0000","last_active":"2024-03-06T07:21:01.426749+0000","last_peered":"2024-03-06T07:21:01.426749+0000","last_clean":"2024-03-06T07:21:01.426749+0000","last_became_active":"2024-03-06T07:20:51.891212+0000","last_became_peered":"2024-03-06T07:20:51.891212+0000","last_unstale":"2024-03-06T07:21:01.426749+0000","last_undegraded":"2024-03-06T07:21:01.426749+0000","last_fullsized":"2024-03-06T07:21:01.426749+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:49.555650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426582+0000","last_change":"2024-03-06T07:21:01.426582+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426582+0000","last_undegraded":"2024-03-06T07:21:01.426582+0000","last_fullsized":"2024-03-06T07:21:01.426582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428589+0000","last_change":"2024-03-06T07:20:51.876410+0000","last_active":"2024-03-06T07:21:01.428589+0000","last_peered":"2024-03-06T07:21:01.428589+0000","last_clean":"2024-03-06T07:21:01.428589+0000","last_became_active":"2024-03-06T07:20:51.876155+0000","last_became_peered":"2024-03-06T07:20:51.876155+0000","last_unstale":"2024-03-06T07:21:01.428589+0000","last_undegraded":"2024-03-06T07:21:01.428589+0000","last_fullsized":"2024-03-06T07:21:01.428589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429516+0000","last_change":"2024-03-06T07:21:01.429516+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429516+0000","last_undegraded":"2024-03-06T07:21:01.429516+0000","last_fullsized":"2024-03-06T07:21:01.429516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425366+0000","last_change":"2024-03-06T07:20:51.880019+0000","last_active":"2024-03-06T07:21:01.425366+0000","last_peered":"2024-03-06T07:21:01.425366+0000","last_clean":"2024-03-06T07:21:01.425366+0000","last_became_active":"2024-03-06T07:20:51.879756+0000","last_became_peered":"2024-03-06T07:20:51.879756+0000","last_unstale":"2024-03-06T07:21:01.425366+0000","last_undegraded":"2024-03-06T07:21:01.425366+0000","last_fullsized":"2024-03-06T07:21:01.425366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.594449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426508+0000","last_change":"2024-03-06T07:21:01.426508+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426508+0000","last_undegraded":"2024-03-06T07:21:01.426508+0000","last_fullsized":"2024-03-06T07:21:01.426508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426322+0000","last_change":"2024-03-06T07:20:51.890815+0000","last_active":"2024-03-06T07:21:01.426322+0000","last_peered":"2024-03-06T07:21:01.426322+0000","last_clean":"2024-03-06T07:21:01.426322+0000","last_became_active":"2024-03-06T07:20:51.890558+0000","last_became_peered":"2024-03-06T07:20:51.890558+0000","last_unstale":"2024-03-06T07:21:01.426322+0000","last_undegraded":"2024-03-06T07:21:01.426322+0000","last_fullsized":"2024-03-06T07:21:01.426322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:50.925154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423254+0000","last_change":"2024-03-06T07:21:01.423254+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423254+0000","last_undegraded":"2024-03-06T07:21:01.423254+0000","last_fullsized":"2024-03-06T07:21:01.423254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426944+0000","last_change":"2024-03-06T07:20:51.884508+0000","last_active":"2024-03-06T07:21:01.426944+0000","last_peered":"2024-03-06T07:21:01.426944+0000","last_clean":"2024-03-06T07:21:01.426944+0000","last_became_active":"2024-03-06T07:20:51.884237+0000","last_became_peered":"2024-03-06T07:20:51.884237+0000","last_unstale":"2024-03-06T07:21:01.426944+0000","last_undegraded":"2024-03-06T07:21:01.426944+0000","last_fullsized":"2024-03-06T07:21:01.426944+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422935+0000","last_change":"2024-03-06T07:21:01.422935+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422935+0000","last_undegraded":"2024-03-06T07:21:01.422935+0000","last_fullsized":"2024-03-06T07:21:01.422935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425637+0000","last_change":"2024-03-06T07:20:51.890118+0000","last_active":"2024-03-06T07:21:01.425637+0000","last_peered":"2024-03-06T07:21:01.425637+0000","last_clean":"2024-03-06T07:21:01.425637+0000","last_became_active":"2024-03-06T07:20:51.888426+0000","last_became_peered":"2024-03-06T07:20:51.888426+0000","last_unstale":"2024-03-06T07:21:01.425637+0000","last_undegraded":"2024-03-06T07:21:01.425637+0000","last_fullsized":"2024-03-06T07:21:01.425637+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:46.950695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423598+0000","last_change":"2024-03-06T07:21:01.423598+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423598+0000","last_undegraded":"2024-03-06T07:21:01.423598+0000","last_fullsized":"2024-03-06T07:21:01.423598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426626+0000","last_change":"2024-03-06T07:20:51.885627+0000","last_active":"2024-03-06T07:21:01.426626+0000","last_peered":"2024-03-06T07:21:01.426626+0000","last_clean":"2024-03-06T07:21:01.426626+0000","last_became_active":"2024-03-06T07:20:51.885338+0000","last_became_peered":"2024-03-06T07:20:51.885338+0000","last_unstale":"2024-03-06T07:21:01.426626+0000","last_undegraded":"2024-03-06T07:21:01.426626+0000","last_fullsized":"2024-03-06T07:21:01.426626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:41.439891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424033+0000","last_change":"2024-03-06T07:21:01.424033+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424033+0000","last_undegraded":"2024-03-06T07:21:01.424033+0000","last_fullsized":"2024-03-06T07:21:01.424033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419243+0000","last_change":"2024-03-06T07:20:51.884448+0000","last_active":"2024-03-06T07:21:01.419243+0000","last_peered":"2024-03-06T07:21:01.419243+0000","last_clean":"2024-03-06T07:21:01.419243+0000","last_became_active":"2024-03-06T07:20:51.884176+0000","last_became_peered":"2024-03-06T07:20:51.884176+0000","last_unstale":"2024-03-06T07:21:01.419243+0000","last_undegraded":"2024-03-06T07:21:01.419243+0000","last_fullsized":"2024-03-06T07:21:01.419243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:29.691807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427151+0000","last_change":"2024-03-06T07:21:01.427151+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427151+0000","last_undegraded":"2024-03-06T07:21:01.427151+0000","last_fullsized":"2024-03-06T07:21:01.427151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419496+0000","last_change":"2024-03-06T07:20:51.886062+0000","last_active":"2024-03-06T07:21:01.419496+0000","last_peered":"2024-03-06T07:21:01.419496+0000","last_clean":"2024-03-06T07:21:01.419496+0000","last_became_active":"2024-03-06T07:20:51.885799+0000","last_became_peered":"2024-03-06T07:20:51.885799+0000","last_unstale":"2024-03-06T07:21:01.419496+0000","last_undegraded":"2024-03-06T07:21:01.419496+0000","last_fullsized":"2024-03-06T07:21:01.419496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425922+0000","last_change":"2024-03-06T07:21:01.425922+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425922+0000","last_undegraded":"2024-03-06T07:21:01.425922+0000","last_fullsized":"2024-03-06T07:21:01.425922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431786+0000","last_change":"2024-03-06T07:20:51.883614+0000","last_active":"2024-03-06T07:21:01.431786+0000","last_peered":"2024-03-06T07:21:01.431786+0000","last_clean":"2024-03-06T07:21:01.431786+0000","last_became_active":"2024-03-06T07:20:51.883266+0000","last_became_peered":"2024-03-06T07:20:51.883266+0000","last_unstale":"2024-03-06T07:21:01.431786+0000","last_undegraded":"2024-03-06T07:21:01.431786+0000","last_fullsized":"2024-03-06T07:21:01.431786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428340+0000","last_change":"2024-03-06T07:21:01.428340+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428340+0000","last_undegraded":"2024-03-06T07:21:01.428340+0000","last_fullsized":"2024-03-06T07:21:01.428340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432598+0000","last_change":"2024-03-06T07:20:51.893301+0000","last_active":"2024-03-06T07:21:01.432598+0000","last_peered":"2024-03-06T07:21:01.432598+0000","last_clean":"2024-03-06T07:21:01.432598+0000","last_became_active":"2024-03-06T07:20:51.892946+0000","last_became_peered":"2024-03-06T07:20:51.892946+0000","last_unstale":"2024-03-06T07:21:01.432598+0000","last_undegraded":"2024-03-06T07:21:01.432598+0000","last_fullsized":"2024-03-06T07:21:01.432598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426198+0000","last_change":"2024-03-06T07:21:01.426198+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426198+0000","last_undegraded":"2024-03-06T07:21:01.426198+0000","last_fullsized":"2024-03-06T07:21:01.426198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426417+0000","last_change":"2024-03-06T07:20:51.891689+0000","last_active":"2024-03-06T07:21:01.426417+0000","last_peered":"2024-03-06T07:21:01.426417+0000","last_clean":"2024-03-06T07:21:01.426417+0000","last_became_active":"2024-03-06T07:20:51.891341+0000","last_became_peered":"2024-03-06T07:20:51.891341+0000","last_unstale":"2024-03-06T07:21:01.426417+0000","last_undegraded":"2024-03-06T07:21:01.426417+0000","last_fullsized":"2024-03-06T07:21:01.426417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:45.060281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428167+0000","last_change":"2024-03-06T07:21:01.428167+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428167+0000","last_undegraded":"2024-03-06T07:21:01.428167+0000","last_fullsized":"2024-03-06T07:21:01.428167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419570+0000","last_change":"2024-03-06T07:20:51.887454+0000","last_active":"2024-03-06T07:21:01.419570+0000","last_peered":"2024-03-06T07:21:01.419570+0000","last_clean":"2024-03-06T07:21:01.419570+0000","last_became_active":"2024-03-06T07:20:51.887188+0000","last_became_peered":"2024-03-06T07:20:51.887188+0000","last_unstale":"2024-03-06T07:21:01.419570+0000","last_undegraded":"2024-03-06T07:21:01.419570+0000","last_fullsized":"2024-03-06T07:21:01.419570+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425332+0000","last_change":"2024-03-06T07:21:01.425332+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425332+0000","last_undegraded":"2024-03-06T07:21:01.425332+0000","last_fullsized":"2024-03-06T07:21:01.425332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430984+0000","last_change":"2024-03-06T07:20:51.899932+0000","last_active":"2024-03-06T07:21:01.430984+0000","last_peered":"2024-03-06T07:21:01.430984+0000","last_clean":"2024-03-06T07:21:01.430984+0000","last_became_active":"2024-03-06T07:20:51.899569+0000","last_became_peered":"2024-03-06T07:20:51.899569+0000","last_unstale":"2024-03-06T07:21:01.430984+0000","last_undegraded":"2024-03-06T07:21:01.430984+0000","last_fullsized":"2024-03-06T07:21:01.430984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429450+0000","last_change":"2024-03-06T07:21:01.429450+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429450+0000","last_undegraded":"2024-03-06T07:21:01.429450+0000","last_fullsized":"2024-03-06T07:21:01.429450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432122+0000","last_change":"2024-03-06T07:20:51.888537+0000","last_active":"2024-03-06T07:21:01.432122+0000","last_peered":"2024-03-06T07:21:01.432122+0000","last_clean":"2024-03-06T07:21:01.432122+0000","last_became_active":"2024-03-06T07:20:51.888281+0000","last_became_peered":"2024-03-06T07:20:51.888281+0000","last_unstale":"2024-03-06T07:21:01.432122+0000","last_undegraded":"2024-03-06T07:21:01.432122+0000","last_fullsized":"2024-03-06T07:21:01.432122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:03.657994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424979+0000","last_change":"2024-03-06T07:21:01.424979+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424979+0000","last_undegraded":"2024-03-06T07:21:01.424979+0000","last_fullsized":"2024-03-06T07:21:01.424979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430775+0000","last_change":"2024-03-06T07:20:51.896158+0000","last_active":"2024-03-06T07:21:01.430775+0000","last_peered":"2024-03-06T07:21:01.430775+0000","last_clean":"2024-03-06T07:21:01.430775+0000","last_became_active":"2024-03-06T07:20:51.895889+0000","last_became_peered":"2024-03-06T07:20:51.895889+0000","last_unstale":"2024-03-06T07:21:01.430775+0000","last_undegraded":"2024-03-06T07:21:01.430775+0000","last_fullsized":"2024-03-06T07:21:01.430775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424138+0000","last_change":"2024-03-06T07:21:01.424138+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424138+0000","last_undegraded":"2024-03-06T07:21:01.424138+0000","last_fullsized":"2024-03-06T07:21:01.424138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440853+0000","last_change":"2024-03-06T07:21:01.440204+0000","last_active":"2024-03-06T07:21:01.440853+0000","last_peered":"2024-03-06T07:21:01.440853+0000","last_clean":"2024-03-06T07:21:01.440853+0000","last_became_active":"2024-03-06T07:21:01.439934+0000","last_became_peered":"2024-03-06T07:21:01.439934+0000","last_unstale":"2024-03-06T07:21:01.440853+0000","last_undegraded":"2024-03-06T07:21:01.440853+0000","last_fullsized":"2024-03-06T07:21:01.440853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:52.584141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424952+0000","last_change":"2024-03-06T07:20:51.876393+0000","last_active":"2024-03-06T07:21:01.424952+0000","last_peered":"2024-03-06T07:21:01.424952+0000","last_clean":"2024-03-06T07:21:01.424952+0000","last_became_active":"2024-03-06T07:20:51.876059+0000","last_became_peered":"2024-03-06T07:20:51.876059+0000","last_unstale":"2024-03-06T07:21:01.424952+0000","last_undegraded":"2024-03-06T07:21:01.424952+0000","last_fullsized":"2024-03-06T07:21:01.424952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426852+0000","last_change":"2024-03-06T07:20:50.879586+0000","last_active":"2024-03-06T07:21:01.426852+0000","last_peered":"2024-03-06T07:21:01.426852+0000","last_clean":"2024-03-06T07:21:01.426852+0000","last_became_active":"2024-03-06T07:20:50.879319+0000","last_became_peered":"2024-03-06T07:20:50.879319+0000","last_unstale":"2024-03-06T07:21:01.426852+0000","last_undegraded":"2024-03-06T07:21:01.426852+0000","last_fullsized":"2024-03-06T07:21:01.426852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:58.810754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424424+0000","last_change":"2024-03-06T07:21:01.424424+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424424+0000","last_undegraded":"2024-03-06T07:21:01.424424+0000","last_fullsized":"2024-03-06T07:21:01.424424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438871+0000","last_change":"2024-03-06T07:21:01.436192+0000","last_active":"2024-03-06T07:21:01.438871+0000","last_peered":"2024-03-06T07:21:01.438871+0000","last_clean":"2024-03-06T07:21:01.438871+0000","last_became_active":"2024-03-06T07:21:01.435924+0000","last_became_peered":"2024-03-06T07:21:01.435924+0000","last_unstale":"2024-03-06T07:21:01.438871+0000","last_undegraded":"2024-03-06T07:21:01.438871+0000","last_fullsized":"2024-03-06T07:21:01.438871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419269+0000","last_change":"2024-03-06T07:20:51.884046+0000","last_active":"2024-03-06T07:21:01.419269+0000","last_peered":"2024-03-06T07:21:01.419269+0000","last_clean":"2024-03-06T07:21:01.419269+0000","last_became_active":"2024-03-06T07:20:51.883593+0000","last_became_peered":"2024-03-06T07:20:51.883593+0000","last_unstale":"2024-03-06T07:21:01.419269+0000","last_undegraded":"2024-03-06T07:21:01.419269+0000","last_fullsized":"2024-03-06T07:21:01.419269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:21.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431733+0000","last_change":"2024-03-06T07:20:50.888715+0000","last_active":"2024-03-06T07:21:01.431733+0000","last_peered":"2024-03-06T07:21:01.431733+0000","last_clean":"2024-03-06T07:21:01.431733+0000","last_became_active":"2024-03-06T07:20:50.888140+0000","last_became_peered":"2024-03-06T07:20:50.888140+0000","last_unstale":"2024-03-06T07:21:01.431733+0000","last_undegraded":"2024-03-06T07:21:01.431733+0000","last_fullsized":"2024-03-06T07:21:01.431733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:34.087293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424260+0000","last_change":"2024-03-06T07:21:01.424260+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424260+0000","last_undegraded":"2024-03-06T07:21:01.424260+0000","last_fullsized":"2024-03-06T07:21:01.424260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438971+0000","last_change":"2024-03-06T07:21:01.435195+0000","last_active":"2024-03-06T07:21:01.438971+0000","last_peered":"2024-03-06T07:21:01.438971+0000","last_clean":"2024-03-06T07:21:01.438971+0000","last_became_active":"2024-03-06T07:21:01.434964+0000","last_became_peered":"2024-03-06T07:21:01.434964+0000","last_unstale":"2024-03-06T07:21:01.438971+0000","last_undegraded":"2024-03-06T07:21:01.438971+0000","last_fullsized":"2024-03-06T07:21:01.438971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:25.506818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426735+0000","last_change":"2024-03-06T07:20:51.886764+0000","last_active":"2024-03-06T07:21:01.426735+0000","last_peered":"2024-03-06T07:21:01.426735+0000","last_clean":"2024-03-06T07:21:01.426735+0000","last_became_active":"2024-03-06T07:20:51.886187+0000","last_became_peered":"2024-03-06T07:20:51.886187+0000","last_unstale":"2024-03-06T07:21:01.426735+0000","last_undegraded":"2024-03-06T07:21:01.426735+0000","last_fullsized":"2024-03-06T07:21:01.426735+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:35.564191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432695+0000","last_change":"2024-03-06T07:20:50.885147+0000","last_active":"2024-03-06T07:21:01.432695+0000","last_peered":"2024-03-06T07:21:01.432695+0000","last_clean":"2024-03-06T07:21:01.432695+0000","last_became_active":"2024-03-06T07:20:50.884811+0000","last_became_peered":"2024-03-06T07:20:50.884811+0000","last_unstale":"2024-03-06T07:21:01.432695+0000","last_undegraded":"2024-03-06T07:21:01.432695+0000","last_fullsized":"2024-03-06T07:21:01.432695+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:24.667879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422765+0000","last_change":"2024-03-06T07:21:01.422765+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422765+0000","last_undegraded":"2024-03-06T07:21:01.422765+0000","last_fullsized":"2024-03-06T07:21:01.422765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442369+0000","last_change":"2024-03-06T07:21:01.439913+0000","last_active":"2024-03-06T07:21:01.442369+0000","last_peered":"2024-03-06T07:21:01.442369+0000","last_clean":"2024-03-06T07:21:01.442369+0000","last_became_active":"2024-03-06T07:21:01.438882+0000","last_became_peered":"2024-03-06T07:21:01.438882+0000","last_unstale":"2024-03-06T07:21:01.442369+0000","last_undegraded":"2024-03-06T07:21:01.442369+0000","last_fullsized":"2024-03-06T07:21:01.442369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:45.177307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427034+0000","last_change":"2024-03-06T07:20:51.887534+0000","last_active":"2024-03-06T07:21:01.427034+0000","last_peered":"2024-03-06T07:21:01.427034+0000","last_clean":"2024-03-06T07:21:01.427034+0000","last_became_active":"2024-03-06T07:20:51.887239+0000","last_became_peered":"2024-03-06T07:20:51.887239+0000","last_unstale":"2024-03-06T07:21:01.427034+0000","last_undegraded":"2024-03-06T07:21:01.427034+0000","last_fullsized":"2024-03-06T07:21:01.427034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425754+0000","last_change":"2024-03-06T07:20:50.892916+0000","last_active":"2024-03-06T07:21:01.425754+0000","last_peered":"2024-03-06T07:21:01.425754+0000","last_clean":"2024-03-06T07:21:01.425754+0000","last_became_active":"2024-03-06T07:20:50.892629+0000","last_became_peered":"2024-03-06T07:20:50.892629+0000","last_unstale":"2024-03-06T07:21:01.425754+0000","last_undegraded":"2024-03-06T07:21:01.425754+0000","last_fullsized":"2024-03-06T07:21:01.425754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:12.808948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429826+0000","last_change":"2024-03-06T07:21:01.429826+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429826+0000","last_undegraded":"2024-03-06T07:21:01.429826+0000","last_fullsized":"2024-03-06T07:21:01.429826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.439996+0000","last_change":"2024-03-06T07:21:01.439106+0000","last_active":"2024-03-06T07:21:01.439996+0000","last_peered":"2024-03-06T07:21:01.439996+0000","last_clean":"2024-03-06T07:21:01.439996+0000","last_became_active":"2024-03-06T07:21:01.438835+0000","last_became_peered":"2024-03-06T07:21:01.438835+0000","last_unstale":"2024-03-06T07:21:01.439996+0000","last_undegraded":"2024-03-06T07:21:01.439996+0000","last_fullsized":"2024-03-06T07:21:01.439996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419852+0000","last_change":"2024-03-06T07:20:51.895988+0000","last_active":"2024-03-06T07:21:01.419852+0000","last_peered":"2024-03-06T07:21:01.419852+0000","last_clean":"2024-03-06T07:21:01.419852+0000","last_became_active":"2024-03-06T07:20:51.895747+0000","last_became_peered":"2024-03-06T07:20:51.895747+0000","last_unstale":"2024-03-06T07:21:01.419852+0000","last_undegraded":"2024-03-06T07:21:01.419852+0000","last_fullsized":"2024-03-06T07:21:01.419852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427081+0000","last_change":"2024-03-06T07:20:50.893482+0000","last_active":"2024-03-06T07:21:01.427081+0000","last_peered":"2024-03-06T07:21:01.427081+0000","last_clean":"2024-03-06T07:21:01.427081+0000","last_became_active":"2024-03-06T07:20:50.893153+0000","last_became_peered":"2024-03-06T07:20:50.893153+0000","last_unstale":"2024-03-06T07:21:01.427081+0000","last_undegraded":"2024-03-06T07:21:01.427081+0000","last_fullsized":"2024-03-06T07:21:01.427081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:07.775768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424875+0000","last_change":"2024-03-06T07:21:01.424875+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424875+0000","last_undegraded":"2024-03-06T07:21:01.424875+0000","last_fullsized":"2024-03-06T07:21:01.424875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446459+0000","last_change":"2024-03-06T07:21:01.442216+0000","last_active":"2024-03-06T07:21:01.446459+0000","last_peered":"2024-03-06T07:21:01.446459+0000","last_clean":"2024-03-06T07:21:01.446459+0000","last_became_active":"2024-03-06T07:21:01.441772+0000","last_became_peered":"2024-03-06T07:21:01.441772+0000","last_unstale":"2024-03-06T07:21:01.446459+0000","last_undegraded":"2024-03-06T07:21:01.446459+0000","last_fullsized":"2024-03-06T07:21:01.446459+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:44.646869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.420032+0000","last_change":"2024-03-06T07:20:51.897529+0000","last_active":"2024-03-06T07:21:01.420032+0000","last_peered":"2024-03-06T07:21:01.420032+0000","last_clean":"2024-03-06T07:21:01.420032+0000","last_became_active":"2024-03-06T07:20:51.897181+0000","last_became_peered":"2024-03-06T07:20:51.897181+0000","last_unstale":"2024-03-06T07:21:01.420032+0000","last_undegraded":"2024-03-06T07:21:01.420032+0000","last_fullsized":"2024-03-06T07:21:01.420032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431244+0000","last_change":"2024-03-06T07:20:50.886303+0000","last_active":"2024-03-06T07:21:01.431244+0000","last_peered":"2024-03-06T07:21:01.431244+0000","last_clean":"2024-03-06T07:21:01.431244+0000","last_became_active":"2024-03-06T07:20:50.886013+0000","last_became_peered":"2024-03-06T07:20:50.886013+0000","last_unstale":"2024-03-06T07:21:01.431244+0000","last_undegraded":"2024-03-06T07:21:01.431244+0000","last_fullsized":"2024-03-06T07:21:01.431244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:34.617731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424723+0000","last_change":"2024-03-06T07:21:01.424723+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424723+0000","last_undegraded":"2024-03-06T07:21:01.424723+0000","last_fullsized":"2024-03-06T07:21:01.424723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438261+0000","last_change":"2024-03-06T07:21:01.434722+0000","last_active":"2024-03-06T07:21:01.438261+0000","last_peered":"2024-03-06T07:21:01.438261+0000","last_clean":"2024-03-06T07:21:01.438261+0000","last_became_active":"2024-03-06T07:21:01.434380+0000","last_became_peered":"2024-03-06T07:21:01.434380+0000","last_unstale":"2024-03-06T07:21:01.438261+0000","last_undegraded":"2024-03-06T07:21:01.438261+0000","last_fullsized":"2024-03-06T07:21:01.438261+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427622+0000","last_change":"2024-03-06T07:20:51.883834+0000","last_active":"2024-03-06T07:21:01.427622+0000","last_peered":"2024-03-06T07:21:01.427622+0000","last_clean":"2024-03-06T07:21:01.427622+0000","last_became_active":"2024-03-06T07:20:51.883553+0000","last_became_peered":"2024-03-06T07:20:51.883553+0000","last_unstale":"2024-03-06T07:21:01.427622+0000","last_undegraded":"2024-03-06T07:21:01.427622+0000","last_fullsized":"2024-03-06T07:21:01.427622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419381+0000","last_change":"2024-03-06T07:20:50.881449+0000","last_active":"2024-03-06T07:21:01.419381+0000","last_peered":"2024-03-06T07:21:01.419381+0000","last_clean":"2024-03-06T07:21:01.419381+0000","last_became_active":"2024-03-06T07:20:50.881172+0000","last_became_peered":"2024-03-06T07:20:50.881172+0000","last_unstale":"2024-03-06T07:21:01.419381+0000","last_undegraded":"2024-03-06T07:21:01.419381+0000","last_fullsized":"2024-03-06T07:21:01.419381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.317522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424583+0000","last_change":"2024-03-06T07:21:01.424583+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424583+0000","last_undegraded":"2024-03-06T07:21:01.424583+0000","last_fullsized":"2024-03-06T07:21:01.424583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.437982+0000","last_change":"2024-03-06T07:21:01.431035+0000","last_active":"2024-03-06T07:21:01.437982+0000","last_peered":"2024-03-06T07:21:01.437982+0000","last_clean":"2024-03-06T07:21:01.437982+0000","last_became_active":"2024-03-06T07:21:01.430787+0000","last_became_peered":"2024-03-06T07:21:01.430787+0000","last_unstale":"2024-03-06T07:21:01.437982+0000","last_undegraded":"2024-03-06T07:21:01.437982+0000","last_fullsized":"2024-03-06T07:21:01.437982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:08.242986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426138+0000","last_change":"2024-03-06T07:20:51.893411+0000","last_active":"2024-03-06T07:21:01.426138+0000","last_peered":"2024-03-06T07:21:01.426138+0000","last_clean":"2024-03-06T07:21:01.426138+0000","last_became_active":"2024-03-06T07:20:51.893149+0000","last_became_peered":"2024-03-06T07:20:51.893149+0000","last_unstale":"2024-03-06T07:21:01.426138+0000","last_undegraded":"2024-03-06T07:21:01.426138+0000","last_fullsized":"2024-03-06T07:21:01.426138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427928+0000","last_change":"2024-03-06T07:20:50.890159+0000","last_active":"2024-03-06T07:21:01.427928+0000","last_peered":"2024-03-06T07:21:01.427928+0000","last_clean":"2024-03-06T07:21:01.427928+0000","last_became_active":"2024-03-06T07:20:50.889863+0000","last_became_peered":"2024-03-06T07:20:50.889863+0000","last_unstale":"2024-03-06T07:21:01.427928+0000","last_undegraded":"2024-03-06T07:21:01.427928+0000","last_fullsized":"2024-03-06T07:21:01.427928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.303590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425274+0000","last_change":"2024-03-06T07:21:01.425274+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425274+0000","last_undegraded":"2024-03-06T07:21:01.425274+0000","last_fullsized":"2024-03-06T07:21:01.425274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443004+0000","last_change":"2024-03-06T07:21:01.440880+0000","last_active":"2024-03-06T07:21:01.443004+0000","last_peered":"2024-03-06T07:21:01.443004+0000","last_clean":"2024-03-06T07:21:01.443004+0000","last_became_active":"2024-03-06T07:21:01.440583+0000","last_became_peered":"2024-03-06T07:21:01.440583+0000","last_unstale":"2024-03-06T07:21:01.443004+0000","last_undegraded":"2024-03-06T07:21:01.443004+0000","last_fullsized":"2024-03-06T07:21:01.443004+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419246+0000","last_change":"2024-03-06T07:20:51.899199+0000","last_active":"2024-03-06T07:21:01.419246+0000","last_peered":"2024-03-06T07:21:01.419246+0000","last_clean":"2024-03-06T07:21:01.419246+0000","last_became_active":"2024-03-06T07:20:51.898622+0000","last_became_peered":"2024-03-06T07:20:51.898622+0000","last_unstale":"2024-03-06T07:21:01.419246+0000","last_undegraded":"2024-03-06T07:21:01.419246+0000","last_fullsized":"2024-03-06T07:21:01.419246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431988+0000","last_change":"2024-03-06T07:20:50.891619+0000","last_active":"2024-03-06T07:21:01.431988+0000","last_peered":"2024-03-06T07:21:01.431988+0000","last_clean":"2024-03-06T07:21:01.431988+0000","last_became_active":"2024-03-06T07:20:50.891305+0000","last_became_peered":"2024-03-06T07:20:50.891305+0000","last_unstale":"2024-03-06T07:21:01.431988+0000","last_undegraded":"2024-03-06T07:21:01.431988+0000","last_fullsized":"2024-03-06T07:21:01.431988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.732044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427634+0000","last_change":"2024-03-06T07:21:01.427634+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427634+0000","last_undegraded":"2024-03-06T07:21:01.427634+0000","last_fullsized":"2024-03-06T07:21:01.427634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442394+0000","last_change":"2024-03-06T07:21:01.439974+0000","last_active":"2024-03-06T07:21:01.442394+0000","last_peered":"2024-03-06T07:21:01.442394+0000","last_clean":"2024-03-06T07:21:01.442394+0000","last_became_active":"2024-03-06T07:21:01.439682+0000","last_became_peered":"2024-03-06T07:21:01.439682+0000","last_unstale":"2024-03-06T07:21:01.442394+0000","last_undegraded":"2024-03-06T07:21:01.442394+0000","last_fullsized":"2024-03-06T07:21:01.442394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431422+0000","last_change":"2024-03-06T07:20:51.887972+0000","last_active":"2024-03-06T07:21:01.431422+0000","last_peered":"2024-03-06T07:21:01.431422+0000","last_clean":"2024-03-06T07:21:01.431422+0000","last_became_active":"2024-03-06T07:20:51.887725+0000","last_became_peered":"2024-03-06T07:20:51.887725+0000","last_unstale":"2024-03-06T07:21:01.431422+0000","last_undegraded":"2024-03-06T07:21:01.431422+0000","last_fullsized":"2024-03-06T07:21:01.431422+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431885+0000","last_change":"2024-03-06T07:20:50.890731+0000","last_active":"2024-03-06T07:21:01.431885+0000","last_peered":"2024-03-06T07:21:01.431885+0000","last_clean":"2024-03-06T07:21:01.431885+0000","last_became_active":"2024-03-06T07:20:50.890429+0000","last_became_peered":"2024-03-06T07:20:50.890429+0000","last_unstale":"2024-03-06T07:21:01.431885+0000","last_undegraded":"2024-03-06T07:21:01.431885+0000","last_fullsized":"2024-03-06T07:21:01.431885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423808+0000","last_change":"2024-03-06T07:21:01.423808+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423808+0000","last_undegraded":"2024-03-06T07:21:01.423808+0000","last_fullsized":"2024-03-06T07:21:01.423808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438843+0000","last_change":"2024-03-06T07:21:01.436794+0000","last_active":"2024-03-06T07:21:01.438843+0000","last_peered":"2024-03-06T07:21:01.438843+0000","last_clean":"2024-03-06T07:21:01.438843+0000","last_became_active":"2024-03-06T07:21:01.436556+0000","last_became_peered":"2024-03-06T07:21:01.436556+0000","last_unstale":"2024-03-06T07:21:01.438843+0000","last_undegraded":"2024-03-06T07:21:01.438843+0000","last_fullsized":"2024-03-06T07:21:01.438843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425550+0000","last_change":"2024-03-06T07:20:51.885412+0000","last_active":"2024-03-06T07:21:01.425550+0000","last_peered":"2024-03-06T07:21:01.425550+0000","last_clean":"2024-03-06T07:21:01.425550+0000","last_became_active":"2024-03-06T07:20:51.885153+0000","last_became_peered":"2024-03-06T07:20:51.885153+0000","last_unstale":"2024-03-06T07:21:01.425550+0000","last_undegraded":"2024-03-06T07:21:01.425550+0000","last_fullsized":"2024-03-06T07:21:01.425550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:07.592958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427582+0000","last_change":"2024-03-06T07:20:50.880143+0000","last_active":"2024-03-06T07:21:01.427582+0000","last_peered":"2024-03-06T07:21:01.427582+0000","last_clean":"2024-03-06T07:21:01.427582+0000","last_became_active":"2024-03-06T07:20:50.879859+0000","last_became_peered":"2024-03-06T07:20:50.879859+0000","last_unstale":"2024-03-06T07:21:01.427582+0000","last_undegraded":"2024-03-06T07:21:01.427582+0000","last_fullsized":"2024-03-06T07:21:01.427582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:08.684501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423462+0000","last_change":"2024-03-06T07:21:01.423462+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423462+0000","last_undegraded":"2024-03-06T07:21:01.423462+0000","last_fullsized":"2024-03-06T07:21:01.423462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433111+0000","last_change":"2024-03-06T07:21:01.432460+0000","last_active":"2024-03-06T07:21:01.433111+0000","last_peered":"2024-03-06T07:21:01.433111+0000","last_clean":"2024-03-06T07:21:01.433111+0000","last_became_active":"2024-03-06T07:21:01.432169+0000","last_became_peered":"2024-03-06T07:21:01.432169+0000","last_unstale":"2024-03-06T07:21:01.433111+0000","last_undegraded":"2024-03-06T07:21:01.433111+0000","last_fullsized":"2024-03-06T07:21:01.433111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427116+0000","last_change":"2024-03-06T07:20:51.878936+0000","last_active":"2024-03-06T07:21:01.427116+0000","last_peered":"2024-03-06T07:21:01.427116+0000","last_clean":"2024-03-06T07:21:01.427116+0000","last_became_active":"2024-03-06T07:20:51.878613+0000","last_became_peered":"2024-03-06T07:20:51.878613+0000","last_unstale":"2024-03-06T07:21:01.427116+0000","last_undegraded":"2024-03-06T07:21:01.427116+0000","last_fullsized":"2024-03-06T07:21:01.427116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":24,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425845+0000","last_change":"2024-03-06T07:20:50.888110+0000","last_active":"2024-03-06T07:21:01.425845+0000","last_peered":"2024-03-06T07:21:01.425845+0000","last_clean":"2024-03-06T07:21:01.425845+0000","last_became_active":"2024-03-06T07:20:50.887804+0000","last_became_peered":"2024-03-06T07:20:50.887804+0000","last_unstale":"2024-03-06T07:21:01.425845+0000","last_undegraded":"2024-03-06T07:21:01.425845+0000","last_fullsized":"2024-03-06T07:21:01.425845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423329+0000","last_change":"2024-03-06T07:21:01.423329+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423329+0000","last_undegraded":"2024-03-06T07:21:01.423329+0000","last_fullsized":"2024-03-06T07:21:01.423329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438955+0000","last_change":"2024-03-06T07:21:01.433077+0000","last_active":"2024-03-06T07:21:01.438955+0000","last_peered":"2024-03-06T07:21:01.438955+0000","last_clean":"2024-03-06T07:21:01.438955+0000","last_became_active":"2024-03-06T07:21:01.432684+0000","last_became_peered":"2024-03-06T07:21:01.432684+0000","last_unstale":"2024-03-06T07:21:01.438955+0000","last_undegraded":"2024-03-06T07:21:01.438955+0000","last_fullsized":"2024-03-06T07:21:01.438955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:13.212500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431469+0000","last_change":"2024-03-06T07:20:51.891149+0000","last_active":"2024-03-06T07:21:01.431469+0000","last_peered":"2024-03-06T07:21:01.431469+0000","last_clean":"2024-03-06T07:21:01.431469+0000","last_became_active":"2024-03-06T07:20:51.890877+0000","last_became_peered":"2024-03-06T07:20:51.890877+0000","last_unstale":"2024-03-06T07:21:01.431469+0000","last_undegraded":"2024-03-06T07:21:01.431469+0000","last_fullsized":"2024-03-06T07:21:01.431469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431369+0000","last_change":"2024-03-06T07:20:50.888227+0000","last_active":"2024-03-06T07:21:01.431369+0000","last_peered":"2024-03-06T07:21:01.431369+0000","last_clean":"2024-03-06T07:21:01.431369+0000","last_became_active":"2024-03-06T07:20:50.887926+0000","last_became_peered":"2024-03-06T07:20:50.887926+0000","last_unstale":"2024-03-06T07:21:01.431369+0000","last_undegraded":"2024-03-06T07:21:01.431369+0000","last_fullsized":"2024-03-06T07:21:01.431369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:56.255517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425722+0000","last_change":"2024-03-06T07:21:01.425722+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425722+0000","last_undegraded":"2024-03-06T07:21:01.425722+0000","last_fullsized":"2024-03-06T07:21:01.425722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433380+0000","last_change":"2024-03-06T07:21:01.432947+0000","last_active":"2024-03-06T07:21:01.433380+0000","last_peered":"2024-03-06T07:21:01.433380+0000","last_clean":"2024-03-06T07:21:01.433380+0000","last_became_active":"2024-03-06T07:21:01.432263+0000","last_became_peered":"2024-03-06T07:21:01.432263+0000","last_unstale":"2024-03-06T07:21:01.433380+0000","last_undegraded":"2024-03-06T07:21:01.433380+0000","last_fullsized":"2024-03-06T07:21:01.433380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425616+0000","last_change":"2024-03-06T07:20:51.888556+0000","last_active":"2024-03-06T07:21:01.425616+0000","last_peered":"2024-03-06T07:21:01.425616+0000","last_clean":"2024-03-06T07:21:01.425616+0000","last_became_active":"2024-03-06T07:20:51.888278+0000","last_became_peered":"2024-03-06T07:20:51.888278+0000","last_unstale":"2024-03-06T07:21:01.425616+0000","last_undegraded":"2024-03-06T07:21:01.425616+0000","last_fullsized":"2024-03-06T07:21:01.425616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431076+0000","last_change":"2024-03-06T07:20:50.881110+0000","last_active":"2024-03-06T07:21:01.431076+0000","last_peered":"2024-03-06T07:21:01.431076+0000","last_clean":"2024-03-06T07:21:01.431076+0000","last_became_active":"2024-03-06T07:20:50.880752+0000","last_became_peered":"2024-03-06T07:20:50.880752+0000","last_unstale":"2024-03-06T07:21:01.431076+0000","last_undegraded":"2024-03-06T07:21:01.431076+0000","last_fullsized":"2024-03-06T07:21:01.431076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423587+0000","last_change":"2024-03-06T07:21:01.423587+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423587+0000","last_undegraded":"2024-03-06T07:21:01.423587+0000","last_fullsized":"2024-03-06T07:21:01.423587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438705+0000","last_change":"2024-03-06T07:21:01.435628+0000","last_active":"2024-03-06T07:21:01.438705+0000","last_peered":"2024-03-06T07:21:01.438705+0000","last_clean":"2024-03-06T07:21:01.438705+0000","last_became_active":"2024-03-06T07:21:01.434996+0000","last_became_peered":"2024-03-06T07:21:01.434996+0000","last_unstale":"2024-03-06T07:21:01.438705+0000","last_undegraded":"2024-03-06T07:21:01.438705+0000","last_fullsized":"2024-03-06T07:21:01.438705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:21.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425994+0000","last_change":"2024-03-06T07:20:51.879914+0000","last_active":"2024-03-06T07:21:01.425994+0000","last_peered":"2024-03-06T07:21:01.425994+0000","last_clean":"2024-03-06T07:21:01.425994+0000","last_became_active":"2024-03-06T07:20:51.879633+0000","last_became_peered":"2024-03-06T07:20:51.879633+0000","last_unstale":"2024-03-06T07:21:01.425994+0000","last_undegraded":"2024-03-06T07:21:01.425994+0000","last_fullsized":"2024-03-06T07:21:01.425994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:36.738962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424764+0000","last_change":"2024-03-06T07:20:50.880559+0000","last_active":"2024-03-06T07:21:01.424764+0000","last_peered":"2024-03-06T07:21:01.424764+0000","last_clean":"2024-03-06T07:21:01.424764+0000","last_became_active":"2024-03-06T07:20:50.880160+0000","last_became_peered":"2024-03-06T07:20:50.880160+0000","last_unstale":"2024-03-06T07:21:01.424764+0000","last_undegraded":"2024-03-06T07:21:01.424764+0000","last_fullsized":"2024-03-06T07:21:01.424764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:56.900930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430090+0000","last_change":"2024-03-06T07:21:01.430090+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430090+0000","last_undegraded":"2024-03-06T07:21:01.430090+0000","last_fullsized":"2024-03-06T07:21:01.430090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442699+0000","last_change":"2024-03-06T07:21:01.441770+0000","last_active":"2024-03-06T07:21:01.442699+0000","last_peered":"2024-03-06T07:21:01.442699+0000","last_clean":"2024-03-06T07:21:01.442699+0000","last_became_active":"2024-03-06T07:21:01.441478+0000","last_became_peered":"2024-03-06T07:21:01.441478+0000","last_unstale":"2024-03-06T07:21:01.442699+0000","last_undegraded":"2024-03-06T07:21:01.442699+0000","last_fullsized":"2024-03-06T07:21:01.442699+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:10.768407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426237+0000","last_change":"2024-03-06T07:20:51.890440+0000","last_active":"2024-03-06T07:21:01.426237+0000","last_peered":"2024-03-06T07:21:01.426237+0000","last_clean":"2024-03-06T07:21:01.426237+0000","last_became_active":"2024-03-06T07:20:51.890167+0000","last_became_peered":"2024-03-06T07:20:51.890167+0000","last_unstale":"2024-03-06T07:21:01.426237+0000","last_undegraded":"2024-03-06T07:21:01.426237+0000","last_fullsized":"2024-03-06T07:21:01.426237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:53.956654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":26,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419383+0000","last_change":"2024-03-06T07:20:50.885352+0000","last_active":"2024-03-06T07:21:01.419383+0000","last_peered":"2024-03-06T07:21:01.419383+0000","last_clean":"2024-03-06T07:21:01.419383+0000","last_became_active":"2024-03-06T07:20:50.884991+0000","last_became_peered":"2024-03-06T07:20:50.884991+0000","last_unstale":"2024-03-06T07:21:01.419383+0000","last_undegraded":"2024-03-06T07:21:01.419383+0000","last_fullsized":"2024-03-06T07:21:01.419383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:03.652296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427525+0000","last_change":"2024-03-06T07:21:01.427525+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427525+0000","last_undegraded":"2024-03-06T07:21:01.427525+0000","last_fullsized":"2024-03-06T07:21:01.427525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438733+0000","last_change":"2024-03-06T07:21:01.436096+0000","last_active":"2024-03-06T07:21:01.438733+0000","last_peered":"2024-03-06T07:21:01.438733+0000","last_clean":"2024-03-06T07:21:01.438733+0000","last_became_active":"2024-03-06T07:21:01.435767+0000","last_became_peered":"2024-03-06T07:21:01.435767+0000","last_unstale":"2024-03-06T07:21:01.438733+0000","last_undegraded":"2024-03-06T07:21:01.438733+0000","last_fullsized":"2024-03-06T07:21:01.438733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:40.907133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427258+0000","last_change":"2024-03-06T07:20:51.889501+0000","last_active":"2024-03-06T07:21:01.427258+0000","last_peered":"2024-03-06T07:21:01.427258+0000","last_clean":"2024-03-06T07:21:01.427258+0000","last_became_active":"2024-03-06T07:20:51.889206+0000","last_became_peered":"2024-03-06T07:20:51.889206+0000","last_unstale":"2024-03-06T07:21:01.427258+0000","last_undegraded":"2024-03-06T07:21:01.427258+0000","last_fullsized":"2024-03-06T07:21:01.427258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:09.908900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432085+0000","last_change":"2024-03-06T07:20:50.883039+0000","last_active":"2024-03-06T07:21:01.432085+0000","last_peered":"2024-03-06T07:21:01.432085+0000","last_clean":"2024-03-06T07:21:01.432085+0000","last_became_active":"2024-03-06T07:20:50.882748+0000","last_became_peered":"2024-03-06T07:20:50.882748+0000","last_unstale":"2024-03-06T07:21:01.432085+0000","last_undegraded":"2024-03-06T07:21:01.432085+0000","last_fullsized":"2024-03-06T07:21:01.432085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:35.868841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423288+0000","last_change":"2024-03-06T07:21:01.423288+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423288+0000","last_undegraded":"2024-03-06T07:21:01.423288+0000","last_fullsized":"2024-03-06T07:21:01.423288+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440065+0000","last_change":"2024-03-06T07:21:01.438861+0000","last_active":"2024-03-06T07:21:01.440065+0000","last_peered":"2024-03-06T07:21:01.440065+0000","last_clean":"2024-03-06T07:21:01.440065+0000","last_became_active":"2024-03-06T07:21:01.438577+0000","last_became_peered":"2024-03-06T07:21:01.438577+0000","last_unstale":"2024-03-06T07:21:01.440065+0000","last_undegraded":"2024-03-06T07:21:01.440065+0000","last_fullsized":"2024-03-06T07:21:01.440065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.335344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424200+0000","last_change":"2024-03-06T07:20:51.890860+0000","last_active":"2024-03-06T07:21:01.424200+0000","last_peered":"2024-03-06T07:21:01.424200+0000","last_clean":"2024-03-06T07:21:01.424200+0000","last_became_active":"2024-03-06T07:20:51.890576+0000","last_became_peered":"2024-03-06T07:20:51.890576+0000","last_unstale":"2024-03-06T07:21:01.424200+0000","last_undegraded":"2024-03-06T07:21:01.424200+0000","last_fullsized":"2024-03-06T07:21:01.424200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:02.364180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427717+0000","last_change":"2024-03-06T07:20:50.888701+0000","last_active":"2024-03-06T07:21:01.427717+0000","last_peered":"2024-03-06T07:21:01.427717+0000","last_clean":"2024-03-06T07:21:01.427717+0000","last_became_active":"2024-03-06T07:20:50.888392+0000","last_became_peered":"2024-03-06T07:20:50.888392+0000","last_unstale":"2024-03-06T07:21:01.427717+0000","last_undegraded":"2024-03-06T07:21:01.427717+0000","last_fullsized":"2024-03-06T07:21:01.427717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.582638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422966+0000","last_change":"2024-03-06T07:21:01.422966+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422966+0000","last_undegraded":"2024-03-06T07:21:01.422966+0000","last_fullsized":"2024-03-06T07:21:01.422966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438788+0000","last_change":"2024-03-06T07:21:01.436497+0000","last_active":"2024-03-06T07:21:01.438788+0000","last_peered":"2024-03-06T07:21:01.438788+0000","last_clean":"2024-03-06T07:21:01.438788+0000","last_became_active":"2024-03-06T07:21:01.436029+0000","last_became_peered":"2024-03-06T07:21:01.436029+0000","last_unstale":"2024-03-06T07:21:01.438788+0000","last_undegraded":"2024-03-06T07:21:01.438788+0000","last_fullsized":"2024-03-06T07:21:01.438788+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:36.166225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425488+0000","last_change":"2024-03-06T07:20:51.887239+0000","last_active":"2024-03-06T07:21:01.425488+0000","last_peered":"2024-03-06T07:21:01.425488+0000","last_clean":"2024-03-06T07:21:01.425488+0000","last_became_active":"2024-03-06T07:20:51.886940+0000","last_became_peered":"2024-03-06T07:20:51.886940+0000","last_unstale":"2024-03-06T07:21:01.425488+0000","last_undegraded":"2024-03-06T07:21:01.425488+0000","last_fullsized":"2024-03-06T07:21:01.425488+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:08.123116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427698+0000","last_change":"2024-03-06T07:20:50.886879+0000","last_active":"2024-03-06T07:21:01.427698+0000","last_peered":"2024-03-06T07:21:01.427698+0000","last_clean":"2024-03-06T07:21:01.427698+0000","last_became_active":"2024-03-06T07:20:50.886589+0000","last_became_peered":"2024-03-06T07:20:50.886589+0000","last_unstale":"2024-03-06T07:21:01.427698+0000","last_undegraded":"2024-03-06T07:21:01.427698+0000","last_fullsized":"2024-03-06T07:21:01.427698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:51.245367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428740+0000","last_change":"2024-03-06T07:21:01.428740+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428740+0000","last_undegraded":"2024-03-06T07:21:01.428740+0000","last_fullsized":"2024-03-06T07:21:01.428740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443067+0000","last_change":"2024-03-06T07:21:01.442218+0000","last_active":"2024-03-06T07:21:01.443067+0000","last_peered":"2024-03-06T07:21:01.443067+0000","last_clean":"2024-03-06T07:21:01.443067+0000","last_became_active":"2024-03-06T07:21:01.441776+0000","last_became_peered":"2024-03-06T07:21:01.441776+0000","last_unstale":"2024-03-06T07:21:01.443067+0000","last_undegraded":"2024-03-06T07:21:01.443067+0000","last_fullsized":"2024-03-06T07:21:01.443067+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:26.445135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424527+0000","last_change":"2024-03-06T07:20:51.892549+0000","last_active":"2024-03-06T07:21:01.424527+0000","last_peered":"2024-03-06T07:21:01.424527+0000","last_clean":"2024-03-06T07:21:01.424527+0000","last_became_active":"2024-03-06T07:20:51.889936+0000","last_became_peered":"2024-03-06T07:20:51.889936+0000","last_unstale":"2024-03-06T07:21:01.424527+0000","last_undegraded":"2024-03-06T07:21:01.424527+0000","last_fullsized":"2024-03-06T07:21:01.424527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:23.986390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430588+0000","last_change":"2024-03-06T07:20:50.883071+0000","last_active":"2024-03-06T07:21:01.430588+0000","last_peered":"2024-03-06T07:21:01.430588+0000","last_clean":"2024-03-06T07:21:01.430588+0000","last_became_active":"2024-03-06T07:20:50.882465+0000","last_became_peered":"2024-03-06T07:20:50.882465+0000","last_unstale":"2024-03-06T07:21:01.430588+0000","last_undegraded":"2024-03-06T07:21:01.430588+0000","last_fullsized":"2024-03-06T07:21:01.430588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423784+0000","last_change":"2024-03-06T07:21:01.423784+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423784+0000","last_undegraded":"2024-03-06T07:21:01.423784+0000","last_fullsized":"2024-03-06T07:21:01.423784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.444053+0000","last_change":"2024-03-06T07:21:01.441755+0000","last_active":"2024-03-06T07:21:01.444053+0000","last_peered":"2024-03-06T07:21:01.444053+0000","last_clean":"2024-03-06T07:21:01.444053+0000","last_became_active":"2024-03-06T07:21:01.441485+0000","last_became_peered":"2024-03-06T07:21:01.441485+0000","last_unstale":"2024-03-06T07:21:01.444053+0000","last_undegraded":"2024-03-06T07:21:01.444053+0000","last_fullsized":"2024-03-06T07:21:01.444053+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:14.211872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419859+0000","last_change":"2024-03-06T07:20:51.884694+0000","last_active":"2024-03-06T07:21:01.419859+0000","last_peered":"2024-03-06T07:21:01.419859+0000","last_clean":"2024-03-06T07:21:01.419859+0000","last_became_active":"2024-03-06T07:20:51.883905+0000","last_became_peered":"2024-03-06T07:20:51.883905+0000","last_unstale":"2024-03-06T07:21:01.419859+0000","last_undegraded":"2024-03-06T07:21:01.419859+0000","last_fullsized":"2024-03-06T07:21:01.419859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424620+0000","last_change":"2024-03-06T07:20:50.881489+0000","last_active":"2024-03-06T07:21:01.424620+0000","last_peered":"2024-03-06T07:21:01.424620+0000","last_clean":"2024-03-06T07:21:01.424620+0000","last_became_active":"2024-03-06T07:20:50.881182+0000","last_became_peered":"2024-03-06T07:20:50.881182+0000","last_unstale":"2024-03-06T07:21:01.424620+0000","last_undegraded":"2024-03-06T07:21:01.424620+0000","last_fullsized":"2024-03-06T07:21:01.424620+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:22.518381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422551+0000","last_change":"2024-03-06T07:21:01.422551+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422551+0000","last_undegraded":"2024-03-06T07:21:01.422551+0000","last_fullsized":"2024-03-06T07:21:01.422551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438240+0000","last_change":"2024-03-06T07:21:01.434477+0000","last_active":"2024-03-06T07:21:01.438240+0000","last_peered":"2024-03-06T07:21:01.438240+0000","last_clean":"2024-03-06T07:21:01.438240+0000","last_became_active":"2024-03-06T07:21:01.434205+0000","last_became_peered":"2024-03-06T07:21:01.434205+0000","last_unstale":"2024-03-06T07:21:01.438240+0000","last_undegraded":"2024-03-06T07:21:01.438240+0000","last_fullsized":"2024-03-06T07:21:01.438240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419708+0000","last_change":"2024-03-06T07:20:51.889352+0000","last_active":"2024-03-06T07:21:01.419708+0000","last_peered":"2024-03-06T07:21:01.419708+0000","last_clean":"2024-03-06T07:21:01.419708+0000","last_became_active":"2024-03-06T07:20:51.888464+0000","last_became_peered":"2024-03-06T07:20:51.888464+0000","last_unstale":"2024-03-06T07:21:01.419708+0000","last_undegraded":"2024-03-06T07:21:01.419708+0000","last_fullsized":"2024-03-06T07:21:01.419708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425015+0000","last_change":"2024-03-06T07:20:50.887415+0000","last_active":"2024-03-06T07:21:01.425015+0000","last_peered":"2024-03-06T07:21:01.425015+0000","last_clean":"2024-03-06T07:21:01.425015+0000","last_became_active":"2024-03-06T07:20:50.887137+0000","last_became_peered":"2024-03-06T07:20:50.887137+0000","last_unstale":"2024-03-06T07:21:01.425015+0000","last_undegraded":"2024-03-06T07:21:01.425015+0000","last_fullsized":"2024-03-06T07:21:01.425015+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:54.210267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422909+0000","last_change":"2024-03-06T07:21:01.422909+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422909+0000","last_undegraded":"2024-03-06T07:21:01.422909+0000","last_fullsized":"2024-03-06T07:21:01.422909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441013+0000","last_change":"2024-03-06T07:21:01.439736+0000","last_active":"2024-03-06T07:21:01.441013+0000","last_peered":"2024-03-06T07:21:01.441013+0000","last_clean":"2024-03-06T07:21:01.441013+0000","last_became_active":"2024-03-06T07:21:01.439473+0000","last_became_peered":"2024-03-06T07:21:01.439473+0000","last_unstale":"2024-03-06T07:21:01.441013+0000","last_undegraded":"2024-03-06T07:21:01.441013+0000","last_fullsized":"2024-03-06T07:21:01.441013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:38.195559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426346+0000","last_change":"2024-03-06T07:20:51.885672+0000","last_active":"2024-03-06T07:21:01.426346+0000","last_peered":"2024-03-06T07:21:01.426346+0000","last_clean":"2024-03-06T07:21:01.426346+0000","last_became_active":"2024-03-06T07:20:51.885412+0000","last_became_peered":"2024-03-06T07:20:51.885412+0000","last_unstale":"2024-03-06T07:21:01.426346+0000","last_undegraded":"2024-03-06T07:21:01.426346+0000","last_fullsized":"2024-03-06T07:21:01.426346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:54.467344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427413+0000","last_change":"2024-03-06T07:20:50.889223+0000","last_active":"2024-03-06T07:21:01.427413+0000","last_peered":"2024-03-06T07:21:01.427413+0000","last_clean":"2024-03-06T07:21:01.427413+0000","last_became_active":"2024-03-06T07:20:50.888934+0000","last_became_peered":"2024-03-06T07:20:50.888934+0000","last_unstale":"2024-03-06T07:21:01.427413+0000","last_undegraded":"2024-03-06T07:21:01.427413+0000","last_fullsized":"2024-03-06T07:21:01.427413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:04.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423927+0000","last_change":"2024-03-06T07:21:01.423927+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423927+0000","last_undegraded":"2024-03-06T07:21:01.423927+0000","last_fullsized":"2024-03-06T07:21:01.423927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438700+0000","last_change":"2024-03-06T07:21:01.435463+0000","last_active":"2024-03-06T07:21:01.438700+0000","last_peered":"2024-03-06T07:21:01.438700+0000","last_clean":"2024-03-06T07:21:01.438700+0000","last_became_active":"2024-03-06T07:21:01.435086+0000","last_became_peered":"2024-03-06T07:21:01.435086+0000","last_unstale":"2024-03-06T07:21:01.438700+0000","last_undegraded":"2024-03-06T07:21:01.438700+0000","last_fullsized":"2024-03-06T07:21:01.438700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:43.973004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426480+0000","last_change":"2024-03-06T07:20:51.880937+0000","last_active":"2024-03-06T07:21:01.426480+0000","last_peered":"2024-03-06T07:21:01.426480+0000","last_clean":"2024-03-06T07:21:01.426480+0000","last_became_active":"2024-03-06T07:20:51.880648+0000","last_became_peered":"2024-03-06T07:20:51.880648+0000","last_unstale":"2024-03-06T07:21:01.426480+0000","last_undegraded":"2024-03-06T07:21:01.426480+0000","last_fullsized":"2024-03-06T07:21:01.426480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432004+0000","last_change":"2024-03-06T07:20:50.886345+0000","last_active":"2024-03-06T07:21:01.432004+0000","last_peered":"2024-03-06T07:21:01.432004+0000","last_clean":"2024-03-06T07:21:01.432004+0000","last_became_active":"2024-03-06T07:20:50.886074+0000","last_became_peered":"2024-03-06T07:20:50.886074+0000","last_unstale":"2024-03-06T07:21:01.432004+0000","last_undegraded":"2024-03-06T07:21:01.432004+0000","last_fullsized":"2024-03-06T07:21:01.432004+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:39.914514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422540+0000","last_change":"2024-03-06T07:21:01.422540+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422540+0000","last_undegraded":"2024-03-06T07:21:01.422540+0000","last_fullsized":"2024-03-06T07:21:01.422540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446264+0000","last_change":"2024-03-06T07:21:01.443325+0000","last_active":"2024-03-06T07:21:01.446264+0000","last_peered":"2024-03-06T07:21:01.446264+0000","last_clean":"2024-03-06T07:21:01.446264+0000","last_became_active":"2024-03-06T07:21:01.443143+0000","last_became_peered":"2024-03-06T07:21:01.443143+0000","last_unstale":"2024-03-06T07:21:01.446264+0000","last_undegraded":"2024-03-06T07:21:01.446264+0000","last_fullsized":"2024-03-06T07:21:01.446264+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:17.147464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426249+0000","last_change":"2024-03-06T07:20:51.889471+0000","last_active":"2024-03-06T07:21:01.426249+0000","last_peered":"2024-03-06T07:21:01.426249+0000","last_clean":"2024-03-06T07:21:01.426249+0000","last_became_active":"2024-03-06T07:20:51.889218+0000","last_became_peered":"2024-03-06T07:20:51.889218+0000","last_unstale":"2024-03-06T07:21:01.426249+0000","last_undegraded":"2024-03-06T07:21:01.426249+0000","last_fullsized":"2024-03-06T07:21:01.426249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419799+0000","last_change":"2024-03-06T07:20:50.881740+0000","last_active":"2024-03-06T07:21:01.419799+0000","last_peered":"2024-03-06T07:21:01.419799+0000","last_clean":"2024-03-06T07:21:01.419799+0000","last_became_active":"2024-03-06T07:20:50.881306+0000","last_became_peered":"2024-03-06T07:20:50.881306+0000","last_unstale":"2024-03-06T07:21:01.419799+0000","last_undegraded":"2024-03-06T07:21:01.419799+0000","last_fullsized":"2024-03-06T07:21:01.419799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:35.733892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427082+0000","last_change":"2024-03-06T07:21:01.427082+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427082+0000","last_undegraded":"2024-03-06T07:21:01.427082+0000","last_fullsized":"2024-03-06T07:21:01.427082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.436013+0000","last_change":"2024-03-06T07:21:01.435258+0000","last_active":"2024-03-06T07:21:01.436013+0000","last_peered":"2024-03-06T07:21:01.436013+0000","last_clean":"2024-03-06T07:21:01.436013+0000","last_became_active":"2024-03-06T07:21:01.435001+0000","last_became_peered":"2024-03-06T07:21:01.435001+0000","last_unstale":"2024-03-06T07:21:01.436013+0000","last_undegraded":"2024-03-06T07:21:01.436013+0000","last_fullsized":"2024-03-06T07:21:01.436013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426836+0000","last_change":"2024-03-06T07:20:51.886967+0000","last_active":"2024-03-06T07:21:01.426836+0000","last_peered":"2024-03-06T07:21:01.426836+0000","last_clean":"2024-03-06T07:21:01.426836+0000","last_became_active":"2024-03-06T07:20:51.886521+0000","last_became_peered":"2024-03-06T07:20:51.886521+0000","last_unstale":"2024-03-06T07:21:01.426836+0000","last_undegraded":"2024-03-06T07:21:01.426836+0000","last_fullsized":"2024-03-06T07:21:01.426836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:58.399266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432888+0000","last_change":"2024-03-06T07:20:50.885378+0000","last_active":"2024-03-06T07:21:01.432888+0000","last_peered":"2024-03-06T07:21:01.432888+0000","last_clean":"2024-03-06T07:21:01.432888+0000","last_became_active":"2024-03-06T07:20:50.884859+0000","last_became_peered":"2024-03-06T07:20:50.884859+0000","last_unstale":"2024-03-06T07:21:01.432888+0000","last_undegraded":"2024-03-06T07:21:01.432888+0000","last_fullsized":"2024-03-06T07:21:01.432888+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:28.686737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423705+0000","last_change":"2024-03-06T07:21:01.423705+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423705+0000","last_undegraded":"2024-03-06T07:21:01.423705+0000","last_fullsized":"2024-03-06T07:21:01.423705+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438756+0000","last_change":"2024-03-06T07:21:01.437079+0000","last_active":"2024-03-06T07:21:01.438756+0000","last_peered":"2024-03-06T07:21:01.438756+0000","last_clean":"2024-03-06T07:21:01.438756+0000","last_became_active":"2024-03-06T07:21:01.436848+0000","last_became_peered":"2024-03-06T07:21:01.436848+0000","last_unstale":"2024-03-06T07:21:01.438756+0000","last_undegraded":"2024-03-06T07:21:01.438756+0000","last_fullsized":"2024-03-06T07:21:01.438756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.233171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419658+0000","last_change":"2024-03-06T07:20:51.899213+0000","last_active":"2024-03-06T07:21:01.419658+0000","last_peered":"2024-03-06T07:21:01.419658+0000","last_clean":"2024-03-06T07:21:01.419658+0000","last_became_active":"2024-03-06T07:20:51.898882+0000","last_became_peered":"2024-03-06T07:20:51.898882+0000","last_unstale":"2024-03-06T07:21:01.419658+0000","last_undegraded":"2024-03-06T07:21:01.419658+0000","last_fullsized":"2024-03-06T07:21:01.419658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431260+0000","last_change":"2024-03-06T07:20:50.888878+0000","last_active":"2024-03-06T07:21:01.431260+0000","last_peered":"2024-03-06T07:21:01.431260+0000","last_clean":"2024-03-06T07:21:01.431260+0000","last_became_active":"2024-03-06T07:20:50.888604+0000","last_became_peered":"2024-03-06T07:20:50.888604+0000","last_unstale":"2024-03-06T07:21:01.431260+0000","last_undegraded":"2024-03-06T07:21:01.431260+0000","last_fullsized":"2024-03-06T07:21:01.431260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:33.413428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422649+0000","last_change":"2024-03-06T07:21:01.422649+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422649+0000","last_undegraded":"2024-03-06T07:21:01.422649+0000","last_fullsized":"2024-03-06T07:21:01.422649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438997+0000","last_change":"2024-03-06T07:21:01.433417+0000","last_active":"2024-03-06T07:21:01.438997+0000","last_peered":"2024-03-06T07:21:01.438997+0000","last_clean":"2024-03-06T07:21:01.438997+0000","last_became_active":"2024-03-06T07:21:01.432105+0000","last_became_peered":"2024-03-06T07:21:01.432105+0000","last_unstale":"2024-03-06T07:21:01.438997+0000","last_undegraded":"2024-03-06T07:21:01.438997+0000","last_fullsized":"2024-03-06T07:21:01.438997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:18.120482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425648+0000","last_change":"2024-03-06T07:20:51.893070+0000","last_active":"2024-03-06T07:21:01.425648+0000","last_peered":"2024-03-06T07:21:01.425648+0000","last_clean":"2024-03-06T07:21:01.425648+0000","last_became_active":"2024-03-06T07:20:51.892785+0000","last_became_peered":"2024-03-06T07:20:51.892785+0000","last_unstale":"2024-03-06T07:21:01.425648+0000","last_undegraded":"2024-03-06T07:21:01.425648+0000","last_fullsized":"2024-03-06T07:21:01.425648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:09.591477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425357+0000","last_change":"2024-03-06T07:20:50.892942+0000","last_active":"2024-03-06T07:21:01.425357+0000","last_peered":"2024-03-06T07:21:01.425357+0000","last_clean":"2024-03-06T07:21:01.425357+0000","last_became_active":"2024-03-06T07:20:50.892638+0000","last_became_peered":"2024-03-06T07:20:50.892638+0000","last_unstale":"2024-03-06T07:21:01.425357+0000","last_undegraded":"2024-03-06T07:21:01.425357+0000","last_fullsized":"2024-03-06T07:21:01.425357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:40.434821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427886+0000","last_change":"2024-03-06T07:21:01.427886+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427886+0000","last_undegraded":"2024-03-06T07:21:01.427886+0000","last_fullsized":"2024-03-06T07:21:01.427886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438316+0000","last_change":"2024-03-06T07:21:01.435337+0000","last_active":"2024-03-06T07:21:01.438316+0000","last_peered":"2024-03-06T07:21:01.438316+0000","last_clean":"2024-03-06T07:21:01.438316+0000","last_became_active":"2024-03-06T07:21:01.435098+0000","last_became_peered":"2024-03-06T07:21:01.435098+0000","last_unstale":"2024-03-06T07:21:01.438316+0000","last_undegraded":"2024-03-06T07:21:01.438316+0000","last_fullsized":"2024-03-06T07:21:01.438316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425885+0000","last_change":"2024-03-06T07:20:51.895826+0000","last_active":"2024-03-06T07:21:01.425885+0000","last_peered":"2024-03-06T07:21:01.425885+0000","last_clean":"2024-03-06T07:21:01.425885+0000","last_became_active":"2024-03-06T07:20:51.895483+0000","last_became_peered":"2024-03-06T07:20:51.895483+0000","last_unstale":"2024-03-06T07:21:01.425885+0000","last_undegraded":"2024-03-06T07:21:01.425885+0000","last_fullsized":"2024-03-06T07:21:01.425885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425724+0000","last_change":"2024-03-06T07:20:50.879754+0000","last_active":"2024-03-06T07:21:01.425724+0000","last_peered":"2024-03-06T07:21:01.425724+0000","last_clean":"2024-03-06T07:21:01.425724+0000","last_became_active":"2024-03-06T07:20:50.879488+0000","last_became_peered":"2024-03-06T07:20:50.879488+0000","last_unstale":"2024-03-06T07:21:01.425724+0000","last_undegraded":"2024-03-06T07:21:01.425724+0000","last_fullsized":"2024-03-06T07:21:01.425724+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.603045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422772+0000","last_change":"2024-03-06T07:21:01.422772+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422772+0000","last_undegraded":"2024-03-06T07:21:01.422772+0000","last_fullsized":"2024-03-06T07:21:01.422772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441007+0000","last_change":"2024-03-06T07:21:01.439920+0000","last_active":"2024-03-06T07:21:01.441007+0000","last_peered":"2024-03-06T07:21:01.441007+0000","last_clean":"2024-03-06T07:21:01.441007+0000","last_became_active":"2024-03-06T07:21:01.439672+0000","last_became_peered":"2024-03-06T07:21:01.439672+0000","last_unstale":"2024-03-06T07:21:01.441007+0000","last_undegraded":"2024-03-06T07:21:01.441007+0000","last_fullsized":"2024-03-06T07:21:01.441007+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:37.148955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428677+0000","last_change":"2024-03-06T07:20:51.879085+0000","last_active":"2024-03-06T07:21:01.428677+0000","last_peered":"2024-03-06T07:21:01.428677+0000","last_clean":"2024-03-06T07:21:01.428677+0000","last_became_active":"2024-03-06T07:20:51.878776+0000","last_became_peered":"2024-03-06T07:20:51.878776+0000","last_unstale":"2024-03-06T07:21:01.428677+0000","last_undegraded":"2024-03-06T07:21:01.428677+0000","last_fullsized":"2024-03-06T07:21:01.428677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426282+0000","last_change":"2024-03-06T07:20:50.887928+0000","last_active":"2024-03-06T07:21:01.426282+0000","last_peered":"2024-03-06T07:21:01.426282+0000","last_clean":"2024-03-06T07:21:01.426282+0000","last_became_active":"2024-03-06T07:20:50.887638+0000","last_became_peered":"2024-03-06T07:20:50.887638+0000","last_unstale":"2024-03-06T07:21:01.426282+0000","last_undegraded":"2024-03-06T07:21:01.426282+0000","last_fullsized":"2024-03-06T07:21:01.426282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:52.309668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422497+0000","last_change":"2024-03-06T07:21:01.422497+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422497+0000","last_undegraded":"2024-03-06T07:21:01.422497+0000","last_fullsized":"2024-03-06T07:21:01.422497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440089+0000","last_change":"2024-03-06T07:21:01.436756+0000","last_active":"2024-03-06T07:21:01.440089+0000","last_peered":"2024-03-06T07:21:01.440089+0000","last_clean":"2024-03-06T07:21:01.440089+0000","last_became_active":"2024-03-06T07:21:01.436158+0000","last_became_peered":"2024-03-06T07:21:01.436158+0000","last_unstale":"2024-03-06T07:21:01.440089+0000","last_undegraded":"2024-03-06T07:21:01.440089+0000","last_fullsized":"2024-03-06T07:21:01.440089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:11.908825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431875+0000","last_change":"2024-03-06T07:20:51.883253+0000","last_active":"2024-03-06T07:21:01.431875+0000","last_peered":"2024-03-06T07:21:01.431875+0000","last_clean":"2024-03-06T07:21:01.431875+0000","last_became_active":"2024-03-06T07:20:51.882993+0000","last_became_peered":"2024-03-06T07:20:51.882993+0000","last_unstale":"2024-03-06T07:21:01.431875+0000","last_undegraded":"2024-03-06T07:21:01.431875+0000","last_fullsized":"2024-03-06T07:21:01.431875+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426104+0000","last_change":"2024-03-06T07:20:50.879149+0000","last_active":"2024-03-06T07:21:01.426104+0000","last_peered":"2024-03-06T07:21:01.426104+0000","last_clean":"2024-03-06T07:21:01.426104+0000","last_became_active":"2024-03-06T07:20:50.878867+0000","last_became_peered":"2024-03-06T07:20:50.878867+0000","last_unstale":"2024-03-06T07:21:01.426104+0000","last_undegraded":"2024-03-06T07:21:01.426104+0000","last_fullsized":"2024-03-06T07:21:01.426104+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:57.394196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429819+0000","last_change":"2024-03-06T07:21:01.429819+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429819+0000","last_undegraded":"2024-03-06T07:21:01.429819+0000","last_fullsized":"2024-03-06T07:21:01.429819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442400+0000","last_change":"2024-03-06T07:21:01.440630+0000","last_active":"2024-03-06T07:21:01.442400+0000","last_peered":"2024-03-06T07:21:01.442400+0000","last_clean":"2024-03-06T07:21:01.442400+0000","last_became_active":"2024-03-06T07:21:01.440375+0000","last_became_peered":"2024-03-06T07:21:01.440375+0000","last_unstale":"2024-03-06T07:21:01.442400+0000","last_undegraded":"2024-03-06T07:21:01.442400+0000","last_fullsized":"2024-03-06T07:21:01.442400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432805+0000","last_change":"2024-03-06T07:20:51.891707+0000","last_active":"2024-03-06T07:21:01.432805+0000","last_peered":"2024-03-06T07:21:01.432805+0000","last_clean":"2024-03-06T07:21:01.432805+0000","last_became_active":"2024-03-06T07:20:51.891427+0000","last_became_peered":"2024-03-06T07:20:51.891427+0000","last_unstale":"2024-03-06T07:21:01.432805+0000","last_undegraded":"2024-03-06T07:21:01.432805+0000","last_fullsized":"2024-03-06T07:21:01.432805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425699+0000","last_change":"2024-03-06T07:20:50.885372+0000","last_active":"2024-03-06T07:21:01.425699+0000","last_peered":"2024-03-06T07:21:01.425699+0000","last_clean":"2024-03-06T07:21:01.425699+0000","last_became_active":"2024-03-06T07:20:50.885093+0000","last_became_peered":"2024-03-06T07:20:50.885093+0000","last_unstale":"2024-03-06T07:21:01.425699+0000","last_undegraded":"2024-03-06T07:21:01.425699+0000","last_fullsized":"2024-03-06T07:21:01.425699+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:12.807980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":276,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430887+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:21:01.430887+0000","last_peered":"2024-03-06T07:21:01.430887+0000","last_clean":"2024-03-06T07:21:01.430887+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:21:01.430887+0000","last_undegraded":"2024-03-06T07:21:01.430887+0000","last_fullsized":"2024-03-06T07:21:01.430887+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":1031546,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":442768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":441190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":950272,"data_stored":442872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":933888,"data_stored":441738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":441898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":441268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":1031524,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:05.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:05.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.559+0000 7f3fee1b5640 1 Processor -- start 2024-03-06T07:21:05.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.559+0000 7f3fee1b5640 1 -- start start 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fee1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c6000 0x7f3fe8156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fee1b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe8153a60 con 0x7f3fe8150580 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fee1b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe8153d40 con 0x7f3fe80c1320 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fee1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe806c2c0 con 0x7f3fe80c6000 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3fe8150580 0x7f3fe806dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41900/0 (socket says 172.21.15.94:41900) 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe77fe640 1 -- 172.21.15.94:0/2240824429 learned_addr learned my addr 172.21.15.94:0/2240824429 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe7fff640 1 --2- 172.21.15.94:0/2240824429 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c6000 0x7f3fe8156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe67fc640 1 -- 172.21.15.94:0/2240824429 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586653182 0 0) 0x7f3fe8153a60 con 0x7f3fe8150580 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe67fc640 1 -- 172.21.15.94:0/2240824429 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd0003520 con 0x7f3fe8150580 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe67fc640 1 -- 172.21.15.94:0/2240824429 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155503122 0 0) 0x7f3fe8153d40 con 0x7f3fe80c1320 2024-03-06T07:21:05.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe67fc640 1 -- 172.21.15.94:0/2240824429 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe8153a60 con 0x7f3fe80c1320 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe7fff640 1 -- 172.21.15.94:0/2240824429 >> v1:172.21.15.145:6789/0 conn(0x7f3fe80c1320 legacy=0x7f3fe81566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe7fff640 1 -- 172.21.15.94:0/2240824429 >> v1:172.21.15.94:6789/0 conn(0x7f3fe8150580 legacy=0x7f3fe806dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.560+0000 7f3fe7fff640 1 -- 172.21.15.94:0/2240824429 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe806c5a0 con 0x7f3fe80c6000 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.561+0000 7f3fe7fff640 1 --2- 172.21.15.94:0/2240824429 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c6000 0x7f3fe8156dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009dc0 tx=0x7f3fe002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b493e1afe06eb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.561+0000 7f3fe67fc640 1 -- 172.21.15.94:0/2240824429 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0039070 con 0x7f3fe80c6000 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.561+0000 7f3fe67fc640 1 -- 172.21.15.94:0/2240824429 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe002cc40 con 0x7f3fe80c6000 2024-03-06T07:21:05.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.561+0000 7f3fe67fc640 1 -- 172.21.15.94:0/2240824429 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0034440 con 0x7f3fe80c6000 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.562+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2240824429 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c6000 msgr2=0x7f3fe8156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.562+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2240824429 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c6000 0x7f3fe8156dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009dc0 tx=0x7f3fe002be90 comp rx=0 tx=0).stop 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.562+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2240824429 shutdown_connections 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.562+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2240824429 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c6000 0x7f3fe8156dc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.562+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2240824429 >> 172.21.15.94:0/2240824429 conn(0x7f3fe80c6710 msgr2=0x7f3fe80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.562+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2240824429 shutdown_connections 2024-03-06T07:21:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.562+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2240824429 wait complete. 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.563+0000 7f3fee1b5640 1 Processor -- start 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.563+0000 7f3fee1b5640 1 -- start start 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c1320 0x7f3fe8155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fe80c6000 0x7f3fe81e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe8150580 0x7f3fe81ea370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe806e420 con 0x7f3fe80c6000 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe8150960 con 0x7f3fe8150580 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe806e5a0 con 0x7f3fe80c1320 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fe80c6000 0x7f3fe81e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fe80c6000 0x7f3fe81e9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38922/0 (socket says 172.21.15.94:38922) 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 -- 172.21.15.94:0/2841048985 learned_addr learned my addr 172.21.15.94:0/2841048985 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe77fe640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c1320 0x7f3fe8155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe7fff640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe8150580 0x7f3fe81ea370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 -- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c1320 msgr2=0x7f3fe8155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c1320 0x7f3fe8155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 -- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe8150580 msgr2=0x7f3fe81ea370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe8150580 0x7f3fe81ea370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 -- 172.21.15.94:0/2841048985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe80723d0 con 0x7f3fe80c6000 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fe6ffd640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fe80c6000 0x7f3fe81e9e30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00c430 tx=0x7f3fdc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fcffff640 1 -- 172.21.15.94:0/2841048985 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc017070 con 0x7f3fe80c6000 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fcffff640 1 -- 172.21.15.94:0/2841048985 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc0052f0 con 0x7f3fe80c6000 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe81496f0 con 0x7f3fe80c6000 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fcffff640 1 -- 172.21.15.94:0/2841048985 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc003830 con 0x7f3fe80c6000 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.564+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe8155bd0 con 0x7f3fe80c6000 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.565+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe8070be0 con 0x7f3fe80c6000 2024-03-06T07:21:05.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.566+0000 7f3fcffff640 1 -- 172.21.15.94:0/2841048985 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fdc01a070 con 0x7f3fe80c6000 2024-03-06T07:21:05.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.566+0000 7f3fcffff640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fd406b500 0x7f3fd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.566+0000 7f3fcffff640 1 -- 172.21.15.94:0/2841048985 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3fdc089910 con 0x7f3fe80c6000 2024-03-06T07:21:05.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.566+0000 7f3fe77fe640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fd406b500 0x7f3fd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.567+0000 7f3fe77fe640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fd406b500 0x7f3fd406d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80011e0 tx=0x7f3fd8007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.571+0000 7f3fcffff640 1 -- 172.21.15.94:0/2841048985 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fdc057cf0 con 0x7f3fe80c6000 2024-03-06T07:21:05.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.737+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fe80c5de0 con 0x7f3fd406b500 2024-03-06T07:21:05.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.748+0000 7f3fcffff640 1 -- 172.21.15.94:0/2841048985 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485419 (secure 0 0 0) 0x7f3fe80c5de0 con 0x7f3fd406b500 2024-03-06T07:21:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:05.759 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:21:05.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.752+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fd406b500 msgr2=0x7f3fd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.752+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fd406b500 0x7f3fd406d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80011e0 tx=0x7f3fd8007d30 comp rx=0 tx=0).stop 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.752+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fe80c6000 msgr2=0x7f3fe81e9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.752+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fe80c6000 0x7f3fe81e9e30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00c430 tx=0x7f3fdc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 shutdown_connections 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3fd406b500 0x7f3fd406d9c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe8150580 0x7f3fe81ea370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3fe80c6000 0x7f3fe81e9e30 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 --2- 172.21.15.94:0/2841048985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe80c1320 0x7f3fe8155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 >> 172.21.15.94:0/2841048985 conn(0x7f3fe80c6710 msgr2=0x7f3fe80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 shutdown_connections 2024-03-06T07:21:05.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.753+0000 7f3fee1b5640 1 -- 172.21.15.94:0/2841048985 wait complete. 2024-03-06T07:21:05.765 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-03-06T07:21:04.671494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244424,"kb_used_data":7456,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730296,"statfs":{"total":773094113280,"available":772843823104,"internally_reserved":0,"allocated":7634944,"data_stored":4714804,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.252754"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427479+0000","last_change":"2024-03-06T07:21:01.427479+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427479+0000","last_undegraded":"2024-03-06T07:21:01.427479+0000","last_fullsized":"2024-03-06T07:21:01.427479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431835+0000","last_change":"2024-03-06T07:20:51.896309+0000","last_active":"2024-03-06T07:21:01.431835+0000","last_peered":"2024-03-06T07:21:01.431835+0000","last_clean":"2024-03-06T07:21:01.431835+0000","last_became_active":"2024-03-06T07:20:51.896009+0000","last_became_peered":"2024-03-06T07:20:51.896009+0000","last_unstale":"2024-03-06T07:21:01.431835+0000","last_undegraded":"2024-03-06T07:21:01.431835+0000","last_fullsized":"2024-03-06T07:21:01.431835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426920+0000","last_change":"2024-03-06T07:21:01.426920+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426920+0000","last_undegraded":"2024-03-06T07:21:01.426920+0000","last_fullsized":"2024-03-06T07:21:01.426920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419624+0000","last_change":"2024-03-06T07:20:51.895412+0000","last_active":"2024-03-06T07:21:01.419624+0000","last_peered":"2024-03-06T07:21:01.419624+0000","last_clean":"2024-03-06T07:21:01.419624+0000","last_became_active":"2024-03-06T07:20:51.895169+0000","last_became_peered":"2024-03-06T07:20:51.895169+0000","last_unstale":"2024-03-06T07:21:01.419624+0000","last_undegraded":"2024-03-06T07:21:01.419624+0000","last_fullsized":"2024-03-06T07:21:01.419624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427931+0000","last_change":"2024-03-06T07:21:01.427931+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427931+0000","last_undegraded":"2024-03-06T07:21:01.427931+0000","last_fullsized":"2024-03-06T07:21:01.427931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428431+0000","last_change":"2024-03-06T07:20:51.878718+0000","last_active":"2024-03-06T07:21:01.428431+0000","last_peered":"2024-03-06T07:21:01.428431+0000","last_clean":"2024-03-06T07:21:01.428431+0000","last_became_active":"2024-03-06T07:20:51.878467+0000","last_became_peered":"2024-03-06T07:20:51.878467+0000","last_unstale":"2024-03-06T07:21:01.428431+0000","last_undegraded":"2024-03-06T07:21:01.428431+0000","last_fullsized":"2024-03-06T07:21:01.428431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428790+0000","last_change":"2024-03-06T07:21:01.428790+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428790+0000","last_undegraded":"2024-03-06T07:21:01.428790+0000","last_fullsized":"2024-03-06T07:21:01.428790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431362+0000","last_change":"2024-03-06T07:20:51.892550+0000","last_active":"2024-03-06T07:21:01.431362+0000","last_peered":"2024-03-06T07:21:01.431362+0000","last_clean":"2024-03-06T07:21:01.431362+0000","last_became_active":"2024-03-06T07:20:51.892264+0000","last_became_peered":"2024-03-06T07:20:51.892264+0000","last_unstale":"2024-03-06T07:21:01.431362+0000","last_undegraded":"2024-03-06T07:21:01.431362+0000","last_fullsized":"2024-03-06T07:21:01.431362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429413+0000","last_change":"2024-03-06T07:21:01.429413+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429413+0000","last_undegraded":"2024-03-06T07:21:01.429413+0000","last_fullsized":"2024-03-06T07:21:01.429413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428191+0000","last_change":"2024-03-06T07:20:51.893305+0000","last_active":"2024-03-06T07:21:01.428191+0000","last_peered":"2024-03-06T07:21:01.428191+0000","last_clean":"2024-03-06T07:21:01.428191+0000","last_became_active":"2024-03-06T07:20:51.893041+0000","last_became_peered":"2024-03-06T07:20:51.893041+0000","last_unstale":"2024-03-06T07:21:01.428191+0000","last_undegraded":"2024-03-06T07:21:01.428191+0000","last_fullsized":"2024-03-06T07:21:01.428191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:39.229585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424683+0000","last_change":"2024-03-06T07:21:01.424683+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424683+0000","last_undegraded":"2024-03-06T07:21:01.424683+0000","last_fullsized":"2024-03-06T07:21:01.424683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425180+0000","last_change":"2024-03-06T07:20:51.893812+0000","last_active":"2024-03-06T07:21:01.425180+0000","last_peered":"2024-03-06T07:21:01.425180+0000","last_clean":"2024-03-06T07:21:01.425180+0000","last_became_active":"2024-03-06T07:20:51.891649+0000","last_became_peered":"2024-03-06T07:20:51.891649+0000","last_unstale":"2024-03-06T07:21:01.425180+0000","last_undegraded":"2024-03-06T07:21:01.425180+0000","last_fullsized":"2024-03-06T07:21:01.425180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:46.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424861+0000","last_change":"2024-03-06T07:21:01.424861+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424861+0000","last_undegraded":"2024-03-06T07:21:01.424861+0000","last_fullsized":"2024-03-06T07:21:01.424861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426172+0000","last_change":"2024-03-06T07:20:51.894141+0000","last_active":"2024-03-06T07:21:01.426172+0000","last_peered":"2024-03-06T07:21:01.426172+0000","last_clean":"2024-03-06T07:21:01.426172+0000","last_became_active":"2024-03-06T07:20:51.893793+0000","last_became_peered":"2024-03-06T07:20:51.893793+0000","last_unstale":"2024-03-06T07:21:01.426172+0000","last_undegraded":"2024-03-06T07:21:01.426172+0000","last_fullsized":"2024-03-06T07:21:01.426172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430080+0000","last_change":"2024-03-06T07:21:01.430080+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430080+0000","last_undegraded":"2024-03-06T07:21:01.430080+0000","last_fullsized":"2024-03-06T07:21:01.430080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427332+0000","last_change":"2024-03-06T07:20:51.894037+0000","last_active":"2024-03-06T07:21:01.427332+0000","last_peered":"2024-03-06T07:21:01.427332+0000","last_clean":"2024-03-06T07:21:01.427332+0000","last_became_active":"2024-03-06T07:20:51.893773+0000","last_became_peered":"2024-03-06T07:20:51.893773+0000","last_unstale":"2024-03-06T07:21:01.427332+0000","last_undegraded":"2024-03-06T07:21:01.427332+0000","last_fullsized":"2024-03-06T07:21:01.427332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:48.344455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424697+0000","last_change":"2024-03-06T07:21:01.424697+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424697+0000","last_undegraded":"2024-03-06T07:21:01.424697+0000","last_fullsized":"2024-03-06T07:21:01.424697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427134+0000","last_change":"2024-03-06T07:20:51.886223+0000","last_active":"2024-03-06T07:21:01.427134+0000","last_peered":"2024-03-06T07:21:01.427134+0000","last_clean":"2024-03-06T07:21:01.427134+0000","last_became_active":"2024-03-06T07:20:51.885985+0000","last_became_peered":"2024-03-06T07:20:51.885985+0000","last_unstale":"2024-03-06T07:21:01.427134+0000","last_undegraded":"2024-03-06T07:21:01.427134+0000","last_fullsized":"2024-03-06T07:21:01.427134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:13.814018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426060+0000","last_change":"2024-03-06T07:21:01.426060+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426060+0000","last_undegraded":"2024-03-06T07:21:01.426060+0000","last_fullsized":"2024-03-06T07:21:01.426060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431693+0000","last_change":"2024-03-06T07:20:51.890465+0000","last_active":"2024-03-06T07:21:01.431693+0000","last_peered":"2024-03-06T07:21:01.431693+0000","last_clean":"2024-03-06T07:21:01.431693+0000","last_became_active":"2024-03-06T07:20:51.890183+0000","last_became_peered":"2024-03-06T07:20:51.890183+0000","last_unstale":"2024-03-06T07:21:01.431693+0000","last_undegraded":"2024-03-06T07:21:01.431693+0000","last_fullsized":"2024-03-06T07:21:01.431693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425029+0000","last_change":"2024-03-06T07:21:01.425029+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425029+0000","last_undegraded":"2024-03-06T07:21:01.425029+0000","last_fullsized":"2024-03-06T07:21:01.425029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419915+0000","last_change":"2024-03-06T07:20:51.897836+0000","last_active":"2024-03-06T07:21:01.419915+0000","last_peered":"2024-03-06T07:21:01.419915+0000","last_clean":"2024-03-06T07:21:01.419915+0000","last_became_active":"2024-03-06T07:20:51.897551+0000","last_became_peered":"2024-03-06T07:20:51.897551+0000","last_unstale":"2024-03-06T07:21:01.419915+0000","last_undegraded":"2024-03-06T07:21:01.419915+0000","last_fullsized":"2024-03-06T07:21:01.419915+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424341+0000","last_change":"2024-03-06T07:21:01.424341+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424341+0000","last_undegraded":"2024-03-06T07:21:01.424341+0000","last_fullsized":"2024-03-06T07:21:01.424341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431445+0000","last_change":"2024-03-06T07:20:51.886475+0000","last_active":"2024-03-06T07:21:01.431445+0000","last_peered":"2024-03-06T07:21:01.431445+0000","last_clean":"2024-03-06T07:21:01.431445+0000","last_became_active":"2024-03-06T07:20:51.886218+0000","last_became_peered":"2024-03-06T07:20:51.886218+0000","last_unstale":"2024-03-06T07:21:01.431445+0000","last_undegraded":"2024-03-06T07:21:01.431445+0000","last_fullsized":"2024-03-06T07:21:01.431445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426115+0000","last_change":"2024-03-06T07:21:01.426115+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426115+0000","last_undegraded":"2024-03-06T07:21:01.426115+0000","last_fullsized":"2024-03-06T07:21:01.426115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428289+0000","last_change":"2024-03-06T07:20:51.887999+0000","last_active":"2024-03-06T07:21:01.428289+0000","last_peered":"2024-03-06T07:21:01.428289+0000","last_clean":"2024-03-06T07:21:01.428289+0000","last_became_active":"2024-03-06T07:20:51.887749+0000","last_became_peered":"2024-03-06T07:20:51.887749+0000","last_unstale":"2024-03-06T07:21:01.428289+0000","last_undegraded":"2024-03-06T07:21:01.428289+0000","last_fullsized":"2024-03-06T07:21:01.428289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:25.243948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.431576+0000","last_change":"2024-03-06T07:21:01.431576+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.431576+0000","last_undegraded":"2024-03-06T07:21:01.431576+0000","last_fullsized":"2024-03-06T07:21:01.431576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425276+0000","last_change":"2024-03-06T07:20:51.892140+0000","last_active":"2024-03-06T07:21:01.425276+0000","last_peered":"2024-03-06T07:21:01.425276+0000","last_clean":"2024-03-06T07:21:01.425276+0000","last_became_active":"2024-03-06T07:20:51.891195+0000","last_became_peered":"2024-03-06T07:20:51.891195+0000","last_unstale":"2024-03-06T07:21:01.425276+0000","last_undegraded":"2024-03-06T07:21:01.425276+0000","last_fullsized":"2024-03-06T07:21:01.425276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:28.641052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424813+0000","last_change":"2024-03-06T07:21:01.424813+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424813+0000","last_undegraded":"2024-03-06T07:21:01.424813+0000","last_fullsized":"2024-03-06T07:21:01.424813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419327+0000","last_change":"2024-03-06T07:20:51.894735+0000","last_active":"2024-03-06T07:21:01.419327+0000","last_peered":"2024-03-06T07:21:01.419327+0000","last_clean":"2024-03-06T07:21:01.419327+0000","last_became_active":"2024-03-06T07:20:51.894479+0000","last_became_peered":"2024-03-06T07:20:51.894479+0000","last_unstale":"2024-03-06T07:21:01.419327+0000","last_undegraded":"2024-03-06T07:21:01.419327+0000","last_fullsized":"2024-03-06T07:21:01.419327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425999+0000","last_change":"2024-03-06T07:21:01.425999+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425999+0000","last_undegraded":"2024-03-06T07:21:01.425999+0000","last_fullsized":"2024-03-06T07:21:01.425999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430687+0000","last_change":"2024-03-06T07:20:51.892450+0000","last_active":"2024-03-06T07:21:01.430687+0000","last_peered":"2024-03-06T07:21:01.430687+0000","last_clean":"2024-03-06T07:21:01.430687+0000","last_became_active":"2024-03-06T07:20:51.892184+0000","last_became_peered":"2024-03-06T07:20:51.892184+0000","last_unstale":"2024-03-06T07:21:01.430687+0000","last_undegraded":"2024-03-06T07:21:01.430687+0000","last_fullsized":"2024-03-06T07:21:01.430687+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424318+0000","last_change":"2024-03-06T07:21:01.424318+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424318+0000","last_undegraded":"2024-03-06T07:21:01.424318+0000","last_fullsized":"2024-03-06T07:21:01.424318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426749+0000","last_change":"2024-03-06T07:20:51.891492+0000","last_active":"2024-03-06T07:21:01.426749+0000","last_peered":"2024-03-06T07:21:01.426749+0000","last_clean":"2024-03-06T07:21:01.426749+0000","last_became_active":"2024-03-06T07:20:51.891212+0000","last_became_peered":"2024-03-06T07:20:51.891212+0000","last_unstale":"2024-03-06T07:21:01.426749+0000","last_undegraded":"2024-03-06T07:21:01.426749+0000","last_fullsized":"2024-03-06T07:21:01.426749+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:49.555650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426582+0000","last_change":"2024-03-06T07:21:01.426582+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426582+0000","last_undegraded":"2024-03-06T07:21:01.426582+0000","last_fullsized":"2024-03-06T07:21:01.426582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428589+0000","last_change":"2024-03-06T07:20:51.876410+0000","last_active":"2024-03-06T07:21:01.428589+0000","last_peered":"2024-03-06T07:21:01.428589+0000","last_clean":"2024-03-06T07:21:01.428589+0000","last_became_active":"2024-03-06T07:20:51.876155+0000","last_became_peered":"2024-03-06T07:20:51.876155+0000","last_unstale":"2024-03-06T07:21:01.428589+0000","last_undegraded":"2024-03-06T07:21:01.428589+0000","last_fullsized":"2024-03-06T07:21:01.428589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429516+0000","last_change":"2024-03-06T07:21:01.429516+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429516+0000","last_undegraded":"2024-03-06T07:21:01.429516+0000","last_fullsized":"2024-03-06T07:21:01.429516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425366+0000","last_change":"2024-03-06T07:20:51.880019+0000","last_active":"2024-03-06T07:21:01.425366+0000","last_peered":"2024-03-06T07:21:01.425366+0000","last_clean":"2024-03-06T07:21:01.425366+0000","last_became_active":"2024-03-06T07:20:51.879756+0000","last_became_peered":"2024-03-06T07:20:51.879756+0000","last_unstale":"2024-03-06T07:21:01.425366+0000","last_undegraded":"2024-03-06T07:21:01.425366+0000","last_fullsized":"2024-03-06T07:21:01.425366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.594449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426508+0000","last_change":"2024-03-06T07:21:01.426508+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426508+0000","last_undegraded":"2024-03-06T07:21:01.426508+0000","last_fullsized":"2024-03-06T07:21:01.426508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426322+0000","last_change":"2024-03-06T07:20:51.890815+0000","last_active":"2024-03-06T07:21:01.426322+0000","last_peered":"2024-03-06T07:21:01.426322+0000","last_clean":"2024-03-06T07:21:01.426322+0000","last_became_active":"2024-03-06T07:20:51.890558+0000","last_became_peered":"2024-03-06T07:20:51.890558+0000","last_unstale":"2024-03-06T07:21:01.426322+0000","last_undegraded":"2024-03-06T07:21:01.426322+0000","last_fullsized":"2024-03-06T07:21:01.426322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:50.925154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423254+0000","last_change":"2024-03-06T07:21:01.423254+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423254+0000","last_undegraded":"2024-03-06T07:21:01.423254+0000","last_fullsized":"2024-03-06T07:21:01.423254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426944+0000","last_change":"2024-03-06T07:20:51.884508+0000","last_active":"2024-03-06T07:21:01.426944+0000","last_peered":"2024-03-06T07:21:01.426944+0000","last_clean":"2024-03-06T07:21:01.426944+0000","last_became_active":"2024-03-06T07:20:51.884237+0000","last_became_peered":"2024-03-06T07:20:51.884237+0000","last_unstale":"2024-03-06T07:21:01.426944+0000","last_undegraded":"2024-03-06T07:21:01.426944+0000","last_fullsized":"2024-03-06T07:21:01.426944+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422935+0000","last_change":"2024-03-06T07:21:01.422935+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422935+0000","last_undegraded":"2024-03-06T07:21:01.422935+0000","last_fullsized":"2024-03-06T07:21:01.422935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425637+0000","last_change":"2024-03-06T07:20:51.890118+0000","last_active":"2024-03-06T07:21:01.425637+0000","last_peered":"2024-03-06T07:21:01.425637+0000","last_clean":"2024-03-06T07:21:01.425637+0000","last_became_active":"2024-03-06T07:20:51.888426+0000","last_became_peered":"2024-03-06T07:20:51.888426+0000","last_unstale":"2024-03-06T07:21:01.425637+0000","last_undegraded":"2024-03-06T07:21:01.425637+0000","last_fullsized":"2024-03-06T07:21:01.425637+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:46.950695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423598+0000","last_change":"2024-03-06T07:21:01.423598+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423598+0000","last_undegraded":"2024-03-06T07:21:01.423598+0000","last_fullsized":"2024-03-06T07:21:01.423598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426626+0000","last_change":"2024-03-06T07:20:51.885627+0000","last_active":"2024-03-06T07:21:01.426626+0000","last_peered":"2024-03-06T07:21:01.426626+0000","last_clean":"2024-03-06T07:21:01.426626+0000","last_became_active":"2024-03-06T07:20:51.885338+0000","last_became_peered":"2024-03-06T07:20:51.885338+0000","last_unstale":"2024-03-06T07:21:01.426626+0000","last_undegraded":"2024-03-06T07:21:01.426626+0000","last_fullsized":"2024-03-06T07:21:01.426626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:41.439891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424033+0000","last_change":"2024-03-06T07:21:01.424033+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424033+0000","last_undegraded":"2024-03-06T07:21:01.424033+0000","last_fullsized":"2024-03-06T07:21:01.424033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419243+0000","last_change":"2024-03-06T07:20:51.884448+0000","last_active":"2024-03-06T07:21:01.419243+0000","last_peered":"2024-03-06T07:21:01.419243+0000","last_clean":"2024-03-06T07:21:01.419243+0000","last_became_active":"2024-03-06T07:20:51.884176+0000","last_became_peered":"2024-03-06T07:20:51.884176+0000","last_unstale":"2024-03-06T07:21:01.419243+0000","last_undegraded":"2024-03-06T07:21:01.419243+0000","last_fullsized":"2024-03-06T07:21:01.419243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:29.691807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427151+0000","last_change":"2024-03-06T07:21:01.427151+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427151+0000","last_undegraded":"2024-03-06T07:21:01.427151+0000","last_fullsized":"2024-03-06T07:21:01.427151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419496+0000","last_change":"2024-03-06T07:20:51.886062+0000","last_active":"2024-03-06T07:21:01.419496+0000","last_peered":"2024-03-06T07:21:01.419496+0000","last_clean":"2024-03-06T07:21:01.419496+0000","last_became_active":"2024-03-06T07:20:51.885799+0000","last_became_peered":"2024-03-06T07:20:51.885799+0000","last_unstale":"2024-03-06T07:21:01.419496+0000","last_undegraded":"2024-03-06T07:21:01.419496+0000","last_fullsized":"2024-03-06T07:21:01.419496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425922+0000","last_change":"2024-03-06T07:21:01.425922+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425922+0000","last_undegraded":"2024-03-06T07:21:01.425922+0000","last_fullsized":"2024-03-06T07:21:01.425922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431786+0000","last_change":"2024-03-06T07:20:51.883614+0000","last_active":"2024-03-06T07:21:01.431786+0000","last_peered":"2024-03-06T07:21:01.431786+0000","last_clean":"2024-03-06T07:21:01.431786+0000","last_became_active":"2024-03-06T07:20:51.883266+0000","last_became_peered":"2024-03-06T07:20:51.883266+0000","last_unstale":"2024-03-06T07:21:01.431786+0000","last_undegraded":"2024-03-06T07:21:01.431786+0000","last_fullsized":"2024-03-06T07:21:01.431786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428340+0000","last_change":"2024-03-06T07:21:01.428340+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428340+0000","last_undegraded":"2024-03-06T07:21:01.428340+0000","last_fullsized":"2024-03-06T07:21:01.428340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432598+0000","last_change":"2024-03-06T07:20:51.893301+0000","last_active":"2024-03-06T07:21:01.432598+0000","last_peered":"2024-03-06T07:21:01.432598+0000","last_clean":"2024-03-06T07:21:01.432598+0000","last_became_active":"2024-03-06T07:20:51.892946+0000","last_became_peered":"2024-03-06T07:20:51.892946+0000","last_unstale":"2024-03-06T07:21:01.432598+0000","last_undegraded":"2024-03-06T07:21:01.432598+0000","last_fullsized":"2024-03-06T07:21:01.432598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426198+0000","last_change":"2024-03-06T07:21:01.426198+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426198+0000","last_undegraded":"2024-03-06T07:21:01.426198+0000","last_fullsized":"2024-03-06T07:21:01.426198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426417+0000","last_change":"2024-03-06T07:20:51.891689+0000","last_active":"2024-03-06T07:21:01.426417+0000","last_peered":"2024-03-06T07:21:01.426417+0000","last_clean":"2024-03-06T07:21:01.426417+0000","last_became_active":"2024-03-06T07:20:51.891341+0000","last_became_peered":"2024-03-06T07:20:51.891341+0000","last_unstale":"2024-03-06T07:21:01.426417+0000","last_undegraded":"2024-03-06T07:21:01.426417+0000","last_fullsized":"2024-03-06T07:21:01.426417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:45.060281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428167+0000","last_change":"2024-03-06T07:21:01.428167+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428167+0000","last_undegraded":"2024-03-06T07:21:01.428167+0000","last_fullsized":"2024-03-06T07:21:01.428167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419570+0000","last_change":"2024-03-06T07:20:51.887454+0000","last_active":"2024-03-06T07:21:01.419570+0000","last_peered":"2024-03-06T07:21:01.419570+0000","last_clean":"2024-03-06T07:21:01.419570+0000","last_became_active":"2024-03-06T07:20:51.887188+0000","last_became_peered":"2024-03-06T07:20:51.887188+0000","last_unstale":"2024-03-06T07:21:01.419570+0000","last_undegraded":"2024-03-06T07:21:01.419570+0000","last_fullsized":"2024-03-06T07:21:01.419570+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425332+0000","last_change":"2024-03-06T07:21:01.425332+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425332+0000","last_undegraded":"2024-03-06T07:21:01.425332+0000","last_fullsized":"2024-03-06T07:21:01.425332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430984+0000","last_change":"2024-03-06T07:20:51.899932+0000","last_active":"2024-03-06T07:21:01.430984+0000","last_peered":"2024-03-06T07:21:01.430984+0000","last_clean":"2024-03-06T07:21:01.430984+0000","last_became_active":"2024-03-06T07:20:51.899569+0000","last_became_peered":"2024-03-06T07:20:51.899569+0000","last_unstale":"2024-03-06T07:21:01.430984+0000","last_undegraded":"2024-03-06T07:21:01.430984+0000","last_fullsized":"2024-03-06T07:21:01.430984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429450+0000","last_change":"2024-03-06T07:21:01.429450+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429450+0000","last_undegraded":"2024-03-06T07:21:01.429450+0000","last_fullsized":"2024-03-06T07:21:01.429450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432122+0000","last_change":"2024-03-06T07:20:51.888537+0000","last_active":"2024-03-06T07:21:01.432122+0000","last_peered":"2024-03-06T07:21:01.432122+0000","last_clean":"2024-03-06T07:21:01.432122+0000","last_became_active":"2024-03-06T07:20:51.888281+0000","last_became_peered":"2024-03-06T07:20:51.888281+0000","last_unstale":"2024-03-06T07:21:01.432122+0000","last_undegraded":"2024-03-06T07:21:01.432122+0000","last_fullsized":"2024-03-06T07:21:01.432122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:03.657994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424979+0000","last_change":"2024-03-06T07:21:01.424979+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424979+0000","last_undegraded":"2024-03-06T07:21:01.424979+0000","last_fullsized":"2024-03-06T07:21:01.424979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430775+0000","last_change":"2024-03-06T07:20:51.896158+0000","last_active":"2024-03-06T07:21:01.430775+0000","last_peered":"2024-03-06T07:21:01.430775+0000","last_clean":"2024-03-06T07:21:01.430775+0000","last_became_active":"2024-03-06T07:20:51.895889+0000","last_became_peered":"2024-03-06T07:20:51.895889+0000","last_unstale":"2024-03-06T07:21:01.430775+0000","last_undegraded":"2024-03-06T07:21:01.430775+0000","last_fullsized":"2024-03-06T07:21:01.430775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424138+0000","last_change":"2024-03-06T07:21:01.424138+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424138+0000","last_undegraded":"2024-03-06T07:21:01.424138+0000","last_fullsized":"2024-03-06T07:21:01.424138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440853+0000","last_change":"2024-03-06T07:21:01.440204+0000","last_active":"2024-03-06T07:21:01.440853+0000","last_peered":"2024-03-06T07:21:01.440853+0000","last_clean":"2024-03-06T07:21:01.440853+0000","last_became_active":"2024-03-06T07:21:01.439934+0000","last_became_peered":"2024-03-06T07:21:01.439934+0000","last_unstale":"2024-03-06T07:21:01.440853+0000","last_undegraded":"2024-03-06T07:21:01.440853+0000","last_fullsized":"2024-03-06T07:21:01.440853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:52.584141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424952+0000","last_change":"2024-03-06T07:20:51.876393+0000","last_active":"2024-03-06T07:21:01.424952+0000","last_peered":"2024-03-06T07:21:01.424952+0000","last_clean":"2024-03-06T07:21:01.424952+0000","last_became_active":"2024-03-06T07:20:51.876059+0000","last_became_peered":"2024-03-06T07:20:51.876059+0000","last_unstale":"2024-03-06T07:21:01.424952+0000","last_undegraded":"2024-03-06T07:21:01.424952+0000","last_fullsized":"2024-03-06T07:21:01.424952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426852+0000","last_change":"2024-03-06T07:20:50.879586+0000","last_active":"2024-03-06T07:21:01.426852+0000","last_peered":"2024-03-06T07:21:01.426852+0000","last_clean":"2024-03-06T07:21:01.426852+0000","last_became_active":"2024-03-06T07:20:50.879319+0000","last_became_peered":"2024-03-06T07:20:50.879319+0000","last_unstale":"2024-03-06T07:21:01.426852+0000","last_undegraded":"2024-03-06T07:21:01.426852+0000","last_fullsized":"2024-03-06T07:21:01.426852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:58.810754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424424+0000","last_change":"2024-03-06T07:21:01.424424+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424424+0000","last_undegraded":"2024-03-06T07:21:01.424424+0000","last_fullsized":"2024-03-06T07:21:01.424424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438871+0000","last_change":"2024-03-06T07:21:01.436192+0000","last_active":"2024-03-06T07:21:01.438871+0000","last_peered":"2024-03-06T07:21:01.438871+0000","last_clean":"2024-03-06T07:21:01.438871+0000","last_became_active":"2024-03-06T07:21:01.435924+0000","last_became_peered":"2024-03-06T07:21:01.435924+0000","last_unstale":"2024-03-06T07:21:01.438871+0000","last_undegraded":"2024-03-06T07:21:01.438871+0000","last_fullsized":"2024-03-06T07:21:01.438871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419269+0000","last_change":"2024-03-06T07:20:51.884046+0000","last_active":"2024-03-06T07:21:01.419269+0000","last_peered":"2024-03-06T07:21:01.419269+0000","last_clean":"2024-03-06T07:21:01.419269+0000","last_became_active":"2024-03-06T07:20:51.883593+0000","last_became_peered":"2024-03-06T07:20:51.883593+0000","last_unstale":"2024-03-06T07:21:01.419269+0000","last_undegraded":"2024-03-06T07:21:01.419269+0000","last_fullsized":"2024-03-06T07:21:01.419269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:21.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431733+0000","last_change":"2024-03-06T07:20:50.888715+0000","last_active":"2024-03-06T07:21:01.431733+0000","last_peered":"2024-03-06T07:21:01.431733+0000","last_clean":"2024-03-06T07:21:01.431733+0000","last_became_active":"2024-03-06T07:20:50.888140+0000","last_became_peered":"2024-03-06T07:20:50.888140+0000","last_unstale":"2024-03-06T07:21:01.431733+0000","last_undegraded":"2024-03-06T07:21:01.431733+0000","last_fullsized":"2024-03-06T07:21:01.431733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:34.087293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424260+0000","last_change":"2024-03-06T07:21:01.424260+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424260+0000","last_undegraded":"2024-03-06T07:21:01.424260+0000","last_fullsized":"2024-03-06T07:21:01.424260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438971+0000","last_change":"2024-03-06T07:21:01.435195+0000","last_active":"2024-03-06T07:21:01.438971+0000","last_peered":"2024-03-06T07:21:01.438971+0000","last_clean":"2024-03-06T07:21:01.438971+0000","last_became_active":"2024-03-06T07:21:01.434964+0000","last_became_peered":"2024-03-06T07:21:01.434964+0000","last_unstale":"2024-03-06T07:21:01.438971+0000","last_undegraded":"2024-03-06T07:21:01.438971+0000","last_fullsized":"2024-03-06T07:21:01.438971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:25.506818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426735+0000","last_change":"2024-03-06T07:20:51.886764+0000","last_active":"2024-03-06T07:21:01.426735+0000","last_peered":"2024-03-06T07:21:01.426735+0000","last_clean":"2024-03-06T07:21:01.426735+0000","last_became_active":"2024-03-06T07:20:51.886187+0000","last_became_peered":"2024-03-06T07:20:51.886187+0000","last_unstale":"2024-03-06T07:21:01.426735+0000","last_undegraded":"2024-03-06T07:21:01.426735+0000","last_fullsized":"2024-03-06T07:21:01.426735+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:35.564191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432695+0000","last_change":"2024-03-06T07:20:50.885147+0000","last_active":"2024-03-06T07:21:01.432695+0000","last_peered":"2024-03-06T07:21:01.432695+0000","last_clean":"2024-03-06T07:21:01.432695+0000","last_became_active":"2024-03-06T07:20:50.884811+0000","last_became_peered":"2024-03-06T07:20:50.884811+0000","last_unstale":"2024-03-06T07:21:01.432695+0000","last_undegraded":"2024-03-06T07:21:01.432695+0000","last_fullsized":"2024-03-06T07:21:01.432695+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:24.667879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422765+0000","last_change":"2024-03-06T07:21:01.422765+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422765+0000","last_undegraded":"2024-03-06T07:21:01.422765+0000","last_fullsized":"2024-03-06T07:21:01.422765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442369+0000","last_change":"2024-03-06T07:21:01.439913+0000","last_active":"2024-03-06T07:21:01.442369+0000","last_peered":"2024-03-06T07:21:01.442369+0000","last_clean":"2024-03-06T07:21:01.442369+0000","last_became_active":"2024-03-06T07:21:01.438882+0000","last_became_peered":"2024-03-06T07:21:01.438882+0000","last_unstale":"2024-03-06T07:21:01.442369+0000","last_undegraded":"2024-03-06T07:21:01.442369+0000","last_fullsized":"2024-03-06T07:21:01.442369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:45.177307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427034+0000","last_change":"2024-03-06T07:20:51.887534+0000","last_active":"2024-03-06T07:21:01.427034+0000","last_peered":"2024-03-06T07:21:01.427034+0000","last_clean":"2024-03-06T07:21:01.427034+0000","last_became_active":"2024-03-06T07:20:51.887239+0000","last_became_peered":"2024-03-06T07:20:51.887239+0000","last_unstale":"2024-03-06T07:21:01.427034+0000","last_undegraded":"2024-03-06T07:21:01.427034+0000","last_fullsized":"2024-03-06T07:21:01.427034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425754+0000","last_change":"2024-03-06T07:20:50.892916+0000","last_active":"2024-03-06T07:21:01.425754+0000","last_peered":"2024-03-06T07:21:01.425754+0000","last_clean":"2024-03-06T07:21:01.425754+0000","last_became_active":"2024-03-06T07:20:50.892629+0000","last_became_peered":"2024-03-06T07:20:50.892629+0000","last_unstale":"2024-03-06T07:21:01.425754+0000","last_undegraded":"2024-03-06T07:21:01.425754+0000","last_fullsized":"2024-03-06T07:21:01.425754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:12.808948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429826+0000","last_change":"2024-03-06T07:21:01.429826+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429826+0000","last_undegraded":"2024-03-06T07:21:01.429826+0000","last_fullsized":"2024-03-06T07:21:01.429826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.439996+0000","last_change":"2024-03-06T07:21:01.439106+0000","last_active":"2024-03-06T07:21:01.439996+0000","last_peered":"2024-03-06T07:21:01.439996+0000","last_clean":"2024-03-06T07:21:01.439996+0000","last_became_active":"2024-03-06T07:21:01.438835+0000","last_became_peered":"2024-03-06T07:21:01.438835+0000","last_unstale":"2024-03-06T07:21:01.439996+0000","last_undegraded":"2024-03-06T07:21:01.439996+0000","last_fullsized":"2024-03-06T07:21:01.439996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419852+0000","last_change":"2024-03-06T07:20:51.895988+0000","last_active":"2024-03-06T07:21:01.419852+0000","last_peered":"2024-03-06T07:21:01.419852+0000","last_clean":"2024-03-06T07:21:01.419852+0000","last_became_active":"2024-03-06T07:20:51.895747+0000","last_became_peered":"2024-03-06T07:20:51.895747+0000","last_unstale":"2024-03-06T07:21:01.419852+0000","last_undegraded":"2024-03-06T07:21:01.419852+0000","last_fullsized":"2024-03-06T07:21:01.419852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427081+0000","last_change":"2024-03-06T07:20:50.893482+0000","last_active":"2024-03-06T07:21:01.427081+0000","last_peered":"2024-03-06T07:21:01.427081+0000","last_clean":"2024-03-06T07:21:01.427081+0000","last_became_active":"2024-03-06T07:20:50.893153+0000","last_became_peered":"2024-03-06T07:20:50.893153+0000","last_unstale":"2024-03-06T07:21:01.427081+0000","last_undegraded":"2024-03-06T07:21:01.427081+0000","last_fullsized":"2024-03-06T07:21:01.427081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:07.775768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424875+0000","last_change":"2024-03-06T07:21:01.424875+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424875+0000","last_undegraded":"2024-03-06T07:21:01.424875+0000","last_fullsized":"2024-03-06T07:21:01.424875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446459+0000","last_change":"2024-03-06T07:21:01.442216+0000","last_active":"2024-03-06T07:21:01.446459+0000","last_peered":"2024-03-06T07:21:01.446459+0000","last_clean":"2024-03-06T07:21:01.446459+0000","last_became_active":"2024-03-06T07:21:01.441772+0000","last_became_peered":"2024-03-06T07:21:01.441772+0000","last_unstale":"2024-03-06T07:21:01.446459+0000","last_undegraded":"2024-03-06T07:21:01.446459+0000","last_fullsized":"2024-03-06T07:21:01.446459+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:44.646869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.420032+0000","last_change":"2024-03-06T07:20:51.897529+0000","last_active":"2024-03-06T07:21:01.420032+0000","last_peered":"2024-03-06T07:21:01.420032+0000","last_clean":"2024-03-06T07:21:01.420032+0000","last_became_active":"2024-03-06T07:20:51.897181+0000","last_became_peered":"2024-03-06T07:20:51.897181+0000","last_unstale":"2024-03-06T07:21:01.420032+0000","last_undegraded":"2024-03-06T07:21:01.420032+0000","last_fullsized":"2024-03-06T07:21:01.420032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431244+0000","last_change":"2024-03-06T07:20:50.886303+0000","last_active":"2024-03-06T07:21:01.431244+0000","last_peered":"2024-03-06T07:21:01.431244+0000","last_clean":"2024-03-06T07:21:01.431244+0000","last_became_active":"2024-03-06T07:20:50.886013+0000","last_became_peered":"2024-03-06T07:20:50.886013+0000","last_unstale":"2024-03-06T07:21:01.431244+0000","last_undegraded":"2024-03-06T07:21:01.431244+0000","last_fullsized":"2024-03-06T07:21:01.431244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:34.617731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424723+0000","last_change":"2024-03-06T07:21:01.424723+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424723+0000","last_undegraded":"2024-03-06T07:21:01.424723+0000","last_fullsized":"2024-03-06T07:21:01.424723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438261+0000","last_change":"2024-03-06T07:21:01.434722+0000","last_active":"2024-03-06T07:21:01.438261+0000","last_peered":"2024-03-06T07:21:01.438261+0000","last_clean":"2024-03-06T07:21:01.438261+0000","last_became_active":"2024-03-06T07:21:01.434380+0000","last_became_peered":"2024-03-06T07:21:01.434380+0000","last_unstale":"2024-03-06T07:21:01.438261+0000","last_undegraded":"2024-03-06T07:21:01.438261+0000","last_fullsized":"2024-03-06T07:21:01.438261+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427622+0000","last_change":"2024-03-06T07:20:51.883834+0000","last_active":"2024-03-06T07:21:01.427622+0000","last_peered":"2024-03-06T07:21:01.427622+0000","last_clean":"2024-03-06T07:21:01.427622+0000","last_became_active":"2024-03-06T07:20:51.883553+0000","last_became_peered":"2024-03-06T07:20:51.883553+0000","last_unstale":"2024-03-06T07:21:01.427622+0000","last_undegraded":"2024-03-06T07:21:01.427622+0000","last_fullsized":"2024-03-06T07:21:01.427622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419381+0000","last_change":"2024-03-06T07:20:50.881449+0000","last_active":"2024-03-06T07:21:01.419381+0000","last_peered":"2024-03-06T07:21:01.419381+0000","last_clean":"2024-03-06T07:21:01.419381+0000","last_became_active":"2024-03-06T07:20:50.881172+0000","last_became_peered":"2024-03-06T07:20:50.881172+0000","last_unstale":"2024-03-06T07:21:01.419381+0000","last_undegraded":"2024-03-06T07:21:01.419381+0000","last_fullsized":"2024-03-06T07:21:01.419381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.317522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424583+0000","last_change":"2024-03-06T07:21:01.424583+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424583+0000","last_undegraded":"2024-03-06T07:21:01.424583+0000","last_fullsized":"2024-03-06T07:21:01.424583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.437982+0000","last_change":"2024-03-06T07:21:01.431035+0000","last_active":"2024-03-06T07:21:01.437982+0000","last_peered":"2024-03-06T07:21:01.437982+0000","last_clean":"2024-03-06T07:21:01.437982+0000","last_became_active":"2024-03-06T07:21:01.430787+0000","last_became_peered":"2024-03-06T07:21:01.430787+0000","last_unstale":"2024-03-06T07:21:01.437982+0000","last_undegraded":"2024-03-06T07:21:01.437982+0000","last_fullsized":"2024-03-06T07:21:01.437982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:08.242986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426138+0000","last_change":"2024-03-06T07:20:51.893411+0000","last_active":"2024-03-06T07:21:01.426138+0000","last_peered":"2024-03-06T07:21:01.426138+0000","last_clean":"2024-03-06T07:21:01.426138+0000","last_became_active":"2024-03-06T07:20:51.893149+0000","last_became_peered":"2024-03-06T07:20:51.893149+0000","last_unstale":"2024-03-06T07:21:01.426138+0000","last_undegraded":"2024-03-06T07:21:01.426138+0000","last_fullsized":"2024-03-06T07:21:01.426138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427928+0000","last_change":"2024-03-06T07:20:50.890159+0000","last_active":"2024-03-06T07:21:01.427928+0000","last_peered":"2024-03-06T07:21:01.427928+0000","last_clean":"2024-03-06T07:21:01.427928+0000","last_became_active":"2024-03-06T07:20:50.889863+0000","last_became_peered":"2024-03-06T07:20:50.889863+0000","last_unstale":"2024-03-06T07:21:01.427928+0000","last_undegraded":"2024-03-06T07:21:01.427928+0000","last_fullsized":"2024-03-06T07:21:01.427928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.303590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425274+0000","last_change":"2024-03-06T07:21:01.425274+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425274+0000","last_undegraded":"2024-03-06T07:21:01.425274+0000","last_fullsized":"2024-03-06T07:21:01.425274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443004+0000","last_change":"2024-03-06T07:21:01.440880+0000","last_active":"2024-03-06T07:21:01.443004+0000","last_peered":"2024-03-06T07:21:01.443004+0000","last_clean":"2024-03-06T07:21:01.443004+0000","last_became_active":"2024-03-06T07:21:01.440583+0000","last_became_peered":"2024-03-06T07:21:01.440583+0000","last_unstale":"2024-03-06T07:21:01.443004+0000","last_undegraded":"2024-03-06T07:21:01.443004+0000","last_fullsized":"2024-03-06T07:21:01.443004+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419246+0000","last_change":"2024-03-06T07:20:51.899199+0000","last_active":"2024-03-06T07:21:01.419246+0000","last_peered":"2024-03-06T07:21:01.419246+0000","last_clean":"2024-03-06T07:21:01.419246+0000","last_became_active":"2024-03-06T07:20:51.898622+0000","last_became_peered":"2024-03-06T07:20:51.898622+0000","last_unstale":"2024-03-06T07:21:01.419246+0000","last_undegraded":"2024-03-06T07:21:01.419246+0000","last_fullsized":"2024-03-06T07:21:01.419246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431988+0000","last_change":"2024-03-06T07:20:50.891619+0000","last_active":"2024-03-06T07:21:01.431988+0000","last_peered":"2024-03-06T07:21:01.431988+0000","last_clean":"2024-03-06T07:21:01.431988+0000","last_became_active":"2024-03-06T07:20:50.891305+0000","last_became_peered":"2024-03-06T07:20:50.891305+0000","last_unstale":"2024-03-06T07:21:01.431988+0000","last_undegraded":"2024-03-06T07:21:01.431988+0000","last_fullsized":"2024-03-06T07:21:01.431988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.732044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427634+0000","last_change":"2024-03-06T07:21:01.427634+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427634+0000","last_undegraded":"2024-03-06T07:21:01.427634+0000","last_fullsized":"2024-03-06T07:21:01.427634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442394+0000","last_change":"2024-03-06T07:21:01.439974+0000","last_active":"2024-03-06T07:21:01.442394+0000","last_peered":"2024-03-06T07:21:01.442394+0000","last_clean":"2024-03-06T07:21:01.442394+0000","last_became_active":"2024-03-06T07:21:01.439682+0000","last_became_peered":"2024-03-06T07:21:01.439682+0000","last_unstale":"2024-03-06T07:21:01.442394+0000","last_undegraded":"2024-03-06T07:21:01.442394+0000","last_fullsized":"2024-03-06T07:21:01.442394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431422+0000","last_change":"2024-03-06T07:20:51.887972+0000","last_active":"2024-03-06T07:21:01.431422+0000","last_peered":"2024-03-06T07:21:01.431422+0000","last_clean":"2024-03-06T07:21:01.431422+0000","last_became_active":"2024-03-06T07:20:51.887725+0000","last_became_peered":"2024-03-06T07:20:51.887725+0000","last_unstale":"2024-03-06T07:21:01.431422+0000","last_undegraded":"2024-03-06T07:21:01.431422+0000","last_fullsized":"2024-03-06T07:21:01.431422+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431885+0000","last_change":"2024-03-06T07:20:50.890731+0000","last_active":"2024-03-06T07:21:01.431885+0000","last_peered":"2024-03-06T07:21:01.431885+0000","last_clean":"2024-03-06T07:21:01.431885+0000","last_became_active":"2024-03-06T07:20:50.890429+0000","last_became_peered":"2024-03-06T07:20:50.890429+0000","last_unstale":"2024-03-06T07:21:01.431885+0000","last_undegraded":"2024-03-06T07:21:01.431885+0000","last_fullsized":"2024-03-06T07:21:01.431885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423808+0000","last_change":"2024-03-06T07:21:01.423808+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423808+0000","last_undegraded":"2024-03-06T07:21:01.423808+0000","last_fullsized":"2024-03-06T07:21:01.423808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438843+0000","last_change":"2024-03-06T07:21:01.436794+0000","last_active":"2024-03-06T07:21:01.438843+0000","last_peered":"2024-03-06T07:21:01.438843+0000","last_clean":"2024-03-06T07:21:01.438843+0000","last_became_active":"2024-03-06T07:21:01.436556+0000","last_became_peered":"2024-03-06T07:21:01.436556+0000","last_unstale":"2024-03-06T07:21:01.438843+0000","last_undegraded":"2024-03-06T07:21:01.438843+0000","last_fullsized":"2024-03-06T07:21:01.438843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425550+0000","last_change":"2024-03-06T07:20:51.885412+0000","last_active":"2024-03-06T07:21:01.425550+0000","last_peered":"2024-03-06T07:21:01.425550+0000","last_clean":"2024-03-06T07:21:01.425550+0000","last_became_active":"2024-03-06T07:20:51.885153+0000","last_became_peered":"2024-03-06T07:20:51.885153+0000","last_unstale":"2024-03-06T07:21:01.425550+0000","last_undegraded":"2024-03-06T07:21:01.425550+0000","last_fullsized":"2024-03-06T07:21:01.425550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:07.592958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427582+0000","last_change":"2024-03-06T07:20:50.880143+0000","last_active":"2024-03-06T07:21:01.427582+0000","last_peered":"2024-03-06T07:21:01.427582+0000","last_clean":"2024-03-06T07:21:01.427582+0000","last_became_active":"2024-03-06T07:20:50.879859+0000","last_became_peered":"2024-03-06T07:20:50.879859+0000","last_unstale":"2024-03-06T07:21:01.427582+0000","last_undegraded":"2024-03-06T07:21:01.427582+0000","last_fullsized":"2024-03-06T07:21:01.427582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:08.684501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423462+0000","last_change":"2024-03-06T07:21:01.423462+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423462+0000","last_undegraded":"2024-03-06T07:21:01.423462+0000","last_fullsized":"2024-03-06T07:21:01.423462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433111+0000","last_change":"2024-03-06T07:21:01.432460+0000","last_active":"2024-03-06T07:21:01.433111+0000","last_peered":"2024-03-06T07:21:01.433111+0000","last_clean":"2024-03-06T07:21:01.433111+0000","last_became_active":"2024-03-06T07:21:01.432169+0000","last_became_peered":"2024-03-06T07:21:01.432169+0000","last_unstale":"2024-03-06T07:21:01.433111+0000","last_undegraded":"2024-03-06T07:21:01.433111+0000","last_fullsized":"2024-03-06T07:21:01.433111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427116+0000","last_change":"2024-03-06T07:20:51.878936+0000","last_active":"2024-03-06T07:21:01.427116+0000","last_peered":"2024-03-06T07:21:01.427116+0000","last_clean":"2024-03-06T07:21:01.427116+0000","last_became_active":"2024-03-06T07:20:51.878613+0000","last_became_peered":"2024-03-06T07:20:51.878613+0000","last_unstale":"2024-03-06T07:21:01.427116+0000","last_undegraded":"2024-03-06T07:21:01.427116+0000","last_fullsized":"2024-03-06T07:21:01.427116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":24,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425845+0000","last_change":"2024-03-06T07:20:50.888110+0000","last_active":"2024-03-06T07:21:01.425845+0000","last_peered":"2024-03-06T07:21:01.425845+0000","last_clean":"2024-03-06T07:21:01.425845+0000","last_became_active":"2024-03-06T07:20:50.887804+0000","last_became_peered":"2024-03-06T07:20:50.887804+0000","last_unstale":"2024-03-06T07:21:01.425845+0000","last_undegraded":"2024-03-06T07:21:01.425845+0000","last_fullsized":"2024-03-06T07:21:01.425845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423329+0000","last_change":"2024-03-06T07:21:01.423329+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423329+0000","last_undegraded":"2024-03-06T07:21:01.423329+0000","last_fullsized":"2024-03-06T07:21:01.423329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438955+0000","last_change":"2024-03-06T07:21:01.433077+0000","last_active":"2024-03-06T07:21:01.438955+0000","last_peered":"2024-03-06T07:21:01.438955+0000","last_clean":"2024-03-06T07:21:01.438955+0000","last_became_active":"2024-03-06T07:21:01.432684+0000","last_became_peered":"2024-03-06T07:21:01.432684+0000","last_unstale":"2024-03-06T07:21:01.438955+0000","last_undegraded":"2024-03-06T07:21:01.438955+0000","last_fullsized":"2024-03-06T07:21:01.438955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:13.212500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431469+0000","last_change":"2024-03-06T07:20:51.891149+0000","last_active":"2024-03-06T07:21:01.431469+0000","last_peered":"2024-03-06T07:21:01.431469+0000","last_clean":"2024-03-06T07:21:01.431469+0000","last_became_active":"2024-03-06T07:20:51.890877+0000","last_became_peered":"2024-03-06T07:20:51.890877+0000","last_unstale":"2024-03-06T07:21:01.431469+0000","last_undegraded":"2024-03-06T07:21:01.431469+0000","last_fullsized":"2024-03-06T07:21:01.431469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431369+0000","last_change":"2024-03-06T07:20:50.888227+0000","last_active":"2024-03-06T07:21:01.431369+0000","last_peered":"2024-03-06T07:21:01.431369+0000","last_clean":"2024-03-06T07:21:01.431369+0000","last_became_active":"2024-03-06T07:20:50.887926+0000","last_became_peered":"2024-03-06T07:20:50.887926+0000","last_unstale":"2024-03-06T07:21:01.431369+0000","last_undegraded":"2024-03-06T07:21:01.431369+0000","last_fullsized":"2024-03-06T07:21:01.431369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:56.255517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425722+0000","last_change":"2024-03-06T07:21:01.425722+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425722+0000","last_undegraded":"2024-03-06T07:21:01.425722+0000","last_fullsized":"2024-03-06T07:21:01.425722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433380+0000","last_change":"2024-03-06T07:21:01.432947+0000","last_active":"2024-03-06T07:21:01.433380+0000","last_peered":"2024-03-06T07:21:01.433380+0000","last_clean":"2024-03-06T07:21:01.433380+0000","last_became_active":"2024-03-06T07:21:01.432263+0000","last_became_peered":"2024-03-06T07:21:01.432263+0000","last_unstale":"2024-03-06T07:21:01.433380+0000","last_undegraded":"2024-03-06T07:21:01.433380+0000","last_fullsized":"2024-03-06T07:21:01.433380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425616+0000","last_change":"2024-03-06T07:20:51.888556+0000","last_active":"2024-03-06T07:21:01.425616+0000","last_peered":"2024-03-06T07:21:01.425616+0000","last_clean":"2024-03-06T07:21:01.425616+0000","last_became_active":"2024-03-06T07:20:51.888278+0000","last_became_peered":"2024-03-06T07:20:51.888278+0000","last_unstale":"2024-03-06T07:21:01.425616+0000","last_undegraded":"2024-03-06T07:21:01.425616+0000","last_fullsized":"2024-03-06T07:21:01.425616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431076+0000","last_change":"2024-03-06T07:20:50.881110+0000","last_active":"2024-03-06T07:21:01.431076+0000","last_peered":"2024-03-06T07:21:01.431076+0000","last_clean":"2024-03-06T07:21:01.431076+0000","last_became_active":"2024-03-06T07:20:50.880752+0000","last_became_peered":"2024-03-06T07:20:50.880752+0000","last_unstale":"2024-03-06T07:21:01.431076+0000","last_undegraded":"2024-03-06T07:21:01.431076+0000","last_fullsized":"2024-03-06T07:21:01.431076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423587+0000","last_change":"2024-03-06T07:21:01.423587+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423587+0000","last_undegraded":"2024-03-06T07:21:01.423587+0000","last_fullsized":"2024-03-06T07:21:01.423587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438705+0000","last_change":"2024-03-06T07:21:01.435628+0000","last_active":"2024-03-06T07:21:01.438705+0000","last_peered":"2024-03-06T07:21:01.438705+0000","last_clean":"2024-03-06T07:21:01.438705+0000","last_became_active":"2024-03-06T07:21:01.434996+0000","last_became_peered":"2024-03-06T07:21:01.434996+0000","last_unstale":"2024-03-06T07:21:01.438705+0000","last_undegraded":"2024-03-06T07:21:01.438705+0000","last_fullsized":"2024-03-06T07:21:01.438705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:21.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425994+0000","last_change":"2024-03-06T07:20:51.879914+0000","last_active":"2024-03-06T07:21:01.425994+0000","last_peered":"2024-03-06T07:21:01.425994+0000","last_clean":"2024-03-06T07:21:01.425994+0000","last_became_active":"2024-03-06T07:20:51.879633+0000","last_became_peered":"2024-03-06T07:20:51.879633+0000","last_unstale":"2024-03-06T07:21:01.425994+0000","last_undegraded":"2024-03-06T07:21:01.425994+0000","last_fullsized":"2024-03-06T07:21:01.425994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:36.738962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424764+0000","last_change":"2024-03-06T07:20:50.880559+0000","last_active":"2024-03-06T07:21:01.424764+0000","last_peered":"2024-03-06T07:21:01.424764+0000","last_clean":"2024-03-06T07:21:01.424764+0000","last_became_active":"2024-03-06T07:20:50.880160+0000","last_became_peered":"2024-03-06T07:20:50.880160+0000","last_unstale":"2024-03-06T07:21:01.424764+0000","last_undegraded":"2024-03-06T07:21:01.424764+0000","last_fullsized":"2024-03-06T07:21:01.424764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:56.900930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430090+0000","last_change":"2024-03-06T07:21:01.430090+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430090+0000","last_undegraded":"2024-03-06T07:21:01.430090+0000","last_fullsized":"2024-03-06T07:21:01.430090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442699+0000","last_change":"2024-03-06T07:21:01.441770+0000","last_active":"2024-03-06T07:21:01.442699+0000","last_peered":"2024-03-06T07:21:01.442699+0000","last_clean":"2024-03-06T07:21:01.442699+0000","last_became_active":"2024-03-06T07:21:01.441478+0000","last_became_peered":"2024-03-06T07:21:01.441478+0000","last_unstale":"2024-03-06T07:21:01.442699+0000","last_undegraded":"2024-03-06T07:21:01.442699+0000","last_fullsized":"2024-03-06T07:21:01.442699+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:10.768407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426237+0000","last_change":"2024-03-06T07:20:51.890440+0000","last_active":"2024-03-06T07:21:01.426237+0000","last_peered":"2024-03-06T07:21:01.426237+0000","last_clean":"2024-03-06T07:21:01.426237+0000","last_became_active":"2024-03-06T07:20:51.890167+0000","last_became_peered":"2024-03-06T07:20:51.890167+0000","last_unstale":"2024-03-06T07:21:01.426237+0000","last_undegraded":"2024-03-06T07:21:01.426237+0000","last_fullsized":"2024-03-06T07:21:01.426237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:53.956654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":26,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419383+0000","last_change":"2024-03-06T07:20:50.885352+0000","last_active":"2024-03-06T07:21:01.419383+0000","last_peered":"2024-03-06T07:21:01.419383+0000","last_clean":"2024-03-06T07:21:01.419383+0000","last_became_active":"2024-03-06T07:20:50.884991+0000","last_became_peered":"2024-03-06T07:20:50.884991+0000","last_unstale":"2024-03-06T07:21:01.419383+0000","last_undegraded":"2024-03-06T07:21:01.419383+0000","last_fullsized":"2024-03-06T07:21:01.419383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:03.652296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427525+0000","last_change":"2024-03-06T07:21:01.427525+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427525+0000","last_undegraded":"2024-03-06T07:21:01.427525+0000","last_fullsized":"2024-03-06T07:21:01.427525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438733+0000","last_change":"2024-03-06T07:21:01.436096+0000","last_active":"2024-03-06T07:21:01.438733+0000","last_peered":"2024-03-06T07:21:01.438733+0000","last_clean":"2024-03-06T07:21:01.438733+0000","last_became_active":"2024-03-06T07:21:01.435767+0000","last_became_peered":"2024-03-06T07:21:01.435767+0000","last_unstale":"2024-03-06T07:21:01.438733+0000","last_undegraded":"2024-03-06T07:21:01.438733+0000","last_fullsized":"2024-03-06T07:21:01.438733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:40.907133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427258+0000","last_change":"2024-03-06T07:20:51.889501+0000","last_active":"2024-03-06T07:21:01.427258+0000","last_peered":"2024-03-06T07:21:01.427258+0000","last_clean":"2024-03-06T07:21:01.427258+0000","last_became_active":"2024-03-06T07:20:51.889206+0000","last_became_peered":"2024-03-06T07:20:51.889206+0000","last_unstale":"2024-03-06T07:21:01.427258+0000","last_undegraded":"2024-03-06T07:21:01.427258+0000","last_fullsized":"2024-03-06T07:21:01.427258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:09.908900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432085+0000","last_change":"2024-03-06T07:20:50.883039+0000","last_active":"2024-03-06T07:21:01.432085+0000","last_peered":"2024-03-06T07:21:01.432085+0000","last_clean":"2024-03-06T07:21:01.432085+0000","last_became_active":"2024-03-06T07:20:50.882748+0000","last_became_peered":"2024-03-06T07:20:50.882748+0000","last_unstale":"2024-03-06T07:21:01.432085+0000","last_undegraded":"2024-03-06T07:21:01.432085+0000","last_fullsized":"2024-03-06T07:21:01.432085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:35.868841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423288+0000","last_change":"2024-03-06T07:21:01.423288+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423288+0000","last_undegraded":"2024-03-06T07:21:01.423288+0000","last_fullsized":"2024-03-06T07:21:01.423288+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440065+0000","last_change":"2024-03-06T07:21:01.438861+0000","last_active":"2024-03-06T07:21:01.440065+0000","last_peered":"2024-03-06T07:21:01.440065+0000","last_clean":"2024-03-06T07:21:01.440065+0000","last_became_active":"2024-03-06T07:21:01.438577+0000","last_became_peered":"2024-03-06T07:21:01.438577+0000","last_unstale":"2024-03-06T07:21:01.440065+0000","last_undegraded":"2024-03-06T07:21:01.440065+0000","last_fullsized":"2024-03-06T07:21:01.440065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.335344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424200+0000","last_change":"2024-03-06T07:20:51.890860+0000","last_active":"2024-03-06T07:21:01.424200+0000","last_peered":"2024-03-06T07:21:01.424200+0000","last_clean":"2024-03-06T07:21:01.424200+0000","last_became_active":"2024-03-06T07:20:51.890576+0000","last_became_peered":"2024-03-06T07:20:51.890576+0000","last_unstale":"2024-03-06T07:21:01.424200+0000","last_undegraded":"2024-03-06T07:21:01.424200+0000","last_fullsized":"2024-03-06T07:21:01.424200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:02.364180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427717+0000","last_change":"2024-03-06T07:20:50.888701+0000","last_active":"2024-03-06T07:21:01.427717+0000","last_peered":"2024-03-06T07:21:01.427717+0000","last_clean":"2024-03-06T07:21:01.427717+0000","last_became_active":"2024-03-06T07:20:50.888392+0000","last_became_peered":"2024-03-06T07:20:50.888392+0000","last_unstale":"2024-03-06T07:21:01.427717+0000","last_undegraded":"2024-03-06T07:21:01.427717+0000","last_fullsized":"2024-03-06T07:21:01.427717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.582638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422966+0000","last_change":"2024-03-06T07:21:01.422966+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422966+0000","last_undegraded":"2024-03-06T07:21:01.422966+0000","last_fullsized":"2024-03-06T07:21:01.422966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438788+0000","last_change":"2024-03-06T07:21:01.436497+0000","last_active":"2024-03-06T07:21:01.438788+0000","last_peered":"2024-03-06T07:21:01.438788+0000","last_clean":"2024-03-06T07:21:01.438788+0000","last_became_active":"2024-03-06T07:21:01.436029+0000","last_became_peered":"2024-03-06T07:21:01.436029+0000","last_unstale":"2024-03-06T07:21:01.438788+0000","last_undegraded":"2024-03-06T07:21:01.438788+0000","last_fullsized":"2024-03-06T07:21:01.438788+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:36.166225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425488+0000","last_change":"2024-03-06T07:20:51.887239+0000","last_active":"2024-03-06T07:21:01.425488+0000","last_peered":"2024-03-06T07:21:01.425488+0000","last_clean":"2024-03-06T07:21:01.425488+0000","last_became_active":"2024-03-06T07:20:51.886940+0000","last_became_peered":"2024-03-06T07:20:51.886940+0000","last_unstale":"2024-03-06T07:21:01.425488+0000","last_undegraded":"2024-03-06T07:21:01.425488+0000","last_fullsized":"2024-03-06T07:21:01.425488+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:08.123116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427698+0000","last_change":"2024-03-06T07:20:50.886879+0000","last_active":"2024-03-06T07:21:01.427698+0000","last_peered":"2024-03-06T07:21:01.427698+0000","last_clean":"2024-03-06T07:21:01.427698+0000","last_became_active":"2024-03-06T07:20:50.886589+0000","last_became_peered":"2024-03-06T07:20:50.886589+0000","last_unstale":"2024-03-06T07:21:01.427698+0000","last_undegraded":"2024-03-06T07:21:01.427698+0000","last_fullsized":"2024-03-06T07:21:01.427698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:51.245367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428740+0000","last_change":"2024-03-06T07:21:01.428740+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428740+0000","last_undegraded":"2024-03-06T07:21:01.428740+0000","last_fullsized":"2024-03-06T07:21:01.428740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443067+0000","last_change":"2024-03-06T07:21:01.442218+0000","last_active":"2024-03-06T07:21:01.443067+0000","last_peered":"2024-03-06T07:21:01.443067+0000","last_clean":"2024-03-06T07:21:01.443067+0000","last_became_active":"2024-03-06T07:21:01.441776+0000","last_became_peered":"2024-03-06T07:21:01.441776+0000","last_unstale":"2024-03-06T07:21:01.443067+0000","last_undegraded":"2024-03-06T07:21:01.443067+0000","last_fullsized":"2024-03-06T07:21:01.443067+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:26.445135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424527+0000","last_change":"2024-03-06T07:20:51.892549+0000","last_active":"2024-03-06T07:21:01.424527+0000","last_peered":"2024-03-06T07:21:01.424527+0000","last_clean":"2024-03-06T07:21:01.424527+0000","last_became_active":"2024-03-06T07:20:51.889936+0000","last_became_peered":"2024-03-06T07:20:51.889936+0000","last_unstale":"2024-03-06T07:21:01.424527+0000","last_undegraded":"2024-03-06T07:21:01.424527+0000","last_fullsized":"2024-03-06T07:21:01.424527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:23.986390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430588+0000","last_change":"2024-03-06T07:20:50.883071+0000","last_active":"2024-03-06T07:21:01.430588+0000","last_peered":"2024-03-06T07:21:01.430588+0000","last_clean":"2024-03-06T07:21:01.430588+0000","last_became_active":"2024-03-06T07:20:50.882465+0000","last_became_peered":"2024-03-06T07:20:50.882465+0000","last_unstale":"2024-03-06T07:21:01.430588+0000","last_undegraded":"2024-03-06T07:21:01.430588+0000","last_fullsized":"2024-03-06T07:21:01.430588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423784+0000","last_change":"2024-03-06T07:21:01.423784+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423784+0000","last_undegraded":"2024-03-06T07:21:01.423784+0000","last_fullsized":"2024-03-06T07:21:01.423784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.444053+0000","last_change":"2024-03-06T07:21:01.441755+0000","last_active":"2024-03-06T07:21:01.444053+0000","last_peered":"2024-03-06T07:21:01.444053+0000","last_clean":"2024-03-06T07:21:01.444053+0000","last_became_active":"2024-03-06T07:21:01.441485+0000","last_became_peered":"2024-03-06T07:21:01.441485+0000","last_unstale":"2024-03-06T07:21:01.444053+0000","last_undegraded":"2024-03-06T07:21:01.444053+0000","last_fullsized":"2024-03-06T07:21:01.444053+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:14.211872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419859+0000","last_change":"2024-03-06T07:20:51.884694+0000","last_active":"2024-03-06T07:21:01.419859+0000","last_peered":"2024-03-06T07:21:01.419859+0000","last_clean":"2024-03-06T07:21:01.419859+0000","last_became_active":"2024-03-06T07:20:51.883905+0000","last_became_peered":"2024-03-06T07:20:51.883905+0000","last_unstale":"2024-03-06T07:21:01.419859+0000","last_undegraded":"2024-03-06T07:21:01.419859+0000","last_fullsized":"2024-03-06T07:21:01.419859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424620+0000","last_change":"2024-03-06T07:20:50.881489+0000","last_active":"2024-03-06T07:21:01.424620+0000","last_peered":"2024-03-06T07:21:01.424620+0000","last_clean":"2024-03-06T07:21:01.424620+0000","last_became_active":"2024-03-06T07:20:50.881182+0000","last_became_peered":"2024-03-06T07:20:50.881182+0000","last_unstale":"2024-03-06T07:21:01.424620+0000","last_undegraded":"2024-03-06T07:21:01.424620+0000","last_fullsized":"2024-03-06T07:21:01.424620+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:22.518381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422551+0000","last_change":"2024-03-06T07:21:01.422551+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422551+0000","last_undegraded":"2024-03-06T07:21:01.422551+0000","last_fullsized":"2024-03-06T07:21:01.422551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438240+0000","last_change":"2024-03-06T07:21:01.434477+0000","last_active":"2024-03-06T07:21:01.438240+0000","last_peered":"2024-03-06T07:21:01.438240+0000","last_clean":"2024-03-06T07:21:01.438240+0000","last_became_active":"2024-03-06T07:21:01.434205+0000","last_became_peered":"2024-03-06T07:21:01.434205+0000","last_unstale":"2024-03-06T07:21:01.438240+0000","last_undegraded":"2024-03-06T07:21:01.438240+0000","last_fullsized":"2024-03-06T07:21:01.438240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419708+0000","last_change":"2024-03-06T07:20:51.889352+0000","last_active":"2024-03-06T07:21:01.419708+0000","last_peered":"2024-03-06T07:21:01.419708+0000","last_clean":"2024-03-06T07:21:01.419708+0000","last_became_active":"2024-03-06T07:20:51.888464+0000","last_became_peered":"2024-03-06T07:20:51.888464+0000","last_unstale":"2024-03-06T07:21:01.419708+0000","last_undegraded":"2024-03-06T07:21:01.419708+0000","last_fullsized":"2024-03-06T07:21:01.419708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425015+0000","last_change":"2024-03-06T07:20:50.887415+0000","last_active":"2024-03-06T07:21:01.425015+0000","last_peered":"2024-03-06T07:21:01.425015+0000","last_clean":"2024-03-06T07:21:01.425015+0000","last_became_active":"2024-03-06T07:20:50.887137+0000","last_became_peered":"2024-03-06T07:20:50.887137+0000","last_unstale":"2024-03-06T07:21:01.425015+0000","last_undegraded":"2024-03-06T07:21:01.425015+0000","last_fullsized":"2024-03-06T07:21:01.425015+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:54.210267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422909+0000","last_change":"2024-03-06T07:21:01.422909+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422909+0000","last_undegraded":"2024-03-06T07:21:01.422909+0000","last_fullsized":"2024-03-06T07:21:01.422909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441013+0000","last_change":"2024-03-06T07:21:01.439736+0000","last_active":"2024-03-06T07:21:01.441013+0000","last_peered":"2024-03-06T07:21:01.441013+0000","last_clean":"2024-03-06T07:21:01.441013+0000","last_became_active":"2024-03-06T07:21:01.439473+0000","last_became_peered":"2024-03-06T07:21:01.439473+0000","last_unstale":"2024-03-06T07:21:01.441013+0000","last_undegraded":"2024-03-06T07:21:01.441013+0000","last_fullsized":"2024-03-06T07:21:01.441013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:38.195559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426346+0000","last_change":"2024-03-06T07:20:51.885672+0000","last_active":"2024-03-06T07:21:01.426346+0000","last_peered":"2024-03-06T07:21:01.426346+0000","last_clean":"2024-03-06T07:21:01.426346+0000","last_became_active":"2024-03-06T07:20:51.885412+0000","last_became_peered":"2024-03-06T07:20:51.885412+0000","last_unstale":"2024-03-06T07:21:01.426346+0000","last_undegraded":"2024-03-06T07:21:01.426346+0000","last_fullsized":"2024-03-06T07:21:01.426346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:54.467344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427413+0000","last_change":"2024-03-06T07:20:50.889223+0000","last_active":"2024-03-06T07:21:01.427413+0000","last_peered":"2024-03-06T07:21:01.427413+0000","last_clean":"2024-03-06T07:21:01.427413+0000","last_became_active":"2024-03-06T07:20:50.888934+0000","last_became_peered":"2024-03-06T07:20:50.888934+0000","last_unstale":"2024-03-06T07:21:01.427413+0000","last_undegraded":"2024-03-06T07:21:01.427413+0000","last_fullsized":"2024-03-06T07:21:01.427413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:04.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423927+0000","last_change":"2024-03-06T07:21:01.423927+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423927+0000","last_undegraded":"2024-03-06T07:21:01.423927+0000","last_fullsized":"2024-03-06T07:21:01.423927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438700+0000","last_change":"2024-03-06T07:21:01.435463+0000","last_active":"2024-03-06T07:21:01.438700+0000","last_peered":"2024-03-06T07:21:01.438700+0000","last_clean":"2024-03-06T07:21:01.438700+0000","last_became_active":"2024-03-06T07:21:01.435086+0000","last_became_peered":"2024-03-06T07:21:01.435086+0000","last_unstale":"2024-03-06T07:21:01.438700+0000","last_undegraded":"2024-03-06T07:21:01.438700+0000","last_fullsized":"2024-03-06T07:21:01.438700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:43.973004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426480+0000","last_change":"2024-03-06T07:20:51.880937+0000","last_active":"2024-03-06T07:21:01.426480+0000","last_peered":"2024-03-06T07:21:01.426480+0000","last_clean":"2024-03-06T07:21:01.426480+0000","last_became_active":"2024-03-06T07:20:51.880648+0000","last_became_peered":"2024-03-06T07:20:51.880648+0000","last_unstale":"2024-03-06T07:21:01.426480+0000","last_undegraded":"2024-03-06T07:21:01.426480+0000","last_fullsized":"2024-03-06T07:21:01.426480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432004+0000","last_change":"2024-03-06T07:20:50.886345+0000","last_active":"2024-03-06T07:21:01.432004+0000","last_peered":"2024-03-06T07:21:01.432004+0000","last_clean":"2024-03-06T07:21:01.432004+0000","last_became_active":"2024-03-06T07:20:50.886074+0000","last_became_peered":"2024-03-06T07:20:50.886074+0000","last_unstale":"2024-03-06T07:21:01.432004+0000","last_undegraded":"2024-03-06T07:21:01.432004+0000","last_fullsized":"2024-03-06T07:21:01.432004+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:39.914514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422540+0000","last_change":"2024-03-06T07:21:01.422540+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422540+0000","last_undegraded":"2024-03-06T07:21:01.422540+0000","last_fullsized":"2024-03-06T07:21:01.422540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446264+0000","last_change":"2024-03-06T07:21:01.443325+0000","last_active":"2024-03-06T07:21:01.446264+0000","last_peered":"2024-03-06T07:21:01.446264+0000","last_clean":"2024-03-06T07:21:01.446264+0000","last_became_active":"2024-03-06T07:21:01.443143+0000","last_became_peered":"2024-03-06T07:21:01.443143+0000","last_unstale":"2024-03-06T07:21:01.446264+0000","last_undegraded":"2024-03-06T07:21:01.446264+0000","last_fullsized":"2024-03-06T07:21:01.446264+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:17.147464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426249+0000","last_change":"2024-03-06T07:20:51.889471+0000","last_active":"2024-03-06T07:21:01.426249+0000","last_peered":"2024-03-06T07:21:01.426249+0000","last_clean":"2024-03-06T07:21:01.426249+0000","last_became_active":"2024-03-06T07:20:51.889218+0000","last_became_peered":"2024-03-06T07:20:51.889218+0000","last_unstale":"2024-03-06T07:21:01.426249+0000","last_undegraded":"2024-03-06T07:21:01.426249+0000","last_fullsized":"2024-03-06T07:21:01.426249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419799+0000","last_change":"2024-03-06T07:20:50.881740+0000","last_active":"2024-03-06T07:21:01.419799+0000","last_peered":"2024-03-06T07:21:01.419799+0000","last_clean":"2024-03-06T07:21:01.419799+0000","last_became_active":"2024-03-06T07:20:50.881306+0000","last_became_peered":"2024-03-06T07:20:50.881306+0000","last_unstale":"2024-03-06T07:21:01.419799+0000","last_undegraded":"2024-03-06T07:21:01.419799+0000","last_fullsized":"2024-03-06T07:21:01.419799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:35.733892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427082+0000","last_change":"2024-03-06T07:21:01.427082+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427082+0000","last_undegraded":"2024-03-06T07:21:01.427082+0000","last_fullsized":"2024-03-06T07:21:01.427082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.436013+0000","last_change":"2024-03-06T07:21:01.435258+0000","last_active":"2024-03-06T07:21:01.436013+0000","last_peered":"2024-03-06T07:21:01.436013+0000","last_clean":"2024-03-06T07:21:01.436013+0000","last_became_active":"2024-03-06T07:21:01.435001+0000","last_became_peered":"2024-03-06T07:21:01.435001+0000","last_unstale":"2024-03-06T07:21:01.436013+0000","last_undegraded":"2024-03-06T07:21:01.436013+0000","last_fullsized":"2024-03-06T07:21:01.436013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426836+0000","last_change":"2024-03-06T07:20:51.886967+0000","last_active":"2024-03-06T07:21:01.426836+0000","last_peered":"2024-03-06T07:21:01.426836+0000","last_clean":"2024-03-06T07:21:01.426836+0000","last_became_active":"2024-03-06T07:20:51.886521+0000","last_became_peered":"2024-03-06T07:20:51.886521+0000","last_unstale":"2024-03-06T07:21:01.426836+0000","last_undegraded":"2024-03-06T07:21:01.426836+0000","last_fullsized":"2024-03-06T07:21:01.426836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:58.399266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432888+0000","last_change":"2024-03-06T07:20:50.885378+0000","last_active":"2024-03-06T07:21:01.432888+0000","last_peered":"2024-03-06T07:21:01.432888+0000","last_clean":"2024-03-06T07:21:01.432888+0000","last_became_active":"2024-03-06T07:20:50.884859+0000","last_became_peered":"2024-03-06T07:20:50.884859+0000","last_unstale":"2024-03-06T07:21:01.432888+0000","last_undegraded":"2024-03-06T07:21:01.432888+0000","last_fullsized":"2024-03-06T07:21:01.432888+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:28.686737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423705+0000","last_change":"2024-03-06T07:21:01.423705+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423705+0000","last_undegraded":"2024-03-06T07:21:01.423705+0000","last_fullsized":"2024-03-06T07:21:01.423705+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438756+0000","last_change":"2024-03-06T07:21:01.437079+0000","last_active":"2024-03-06T07:21:01.438756+0000","last_peered":"2024-03-06T07:21:01.438756+0000","last_clean":"2024-03-06T07:21:01.438756+0000","last_became_active":"2024-03-06T07:21:01.436848+0000","last_became_peered":"2024-03-06T07:21:01.436848+0000","last_unstale":"2024-03-06T07:21:01.438756+0000","last_undegraded":"2024-03-06T07:21:01.438756+0000","last_fullsized":"2024-03-06T07:21:01.438756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.233171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419658+0000","last_change":"2024-03-06T07:20:51.899213+0000","last_active":"2024-03-06T07:21:01.419658+0000","last_peered":"2024-03-06T07:21:01.419658+0000","last_clean":"2024-03-06T07:21:01.419658+0000","last_became_active":"2024-03-06T07:20:51.898882+0000","last_became_peered":"2024-03-06T07:20:51.898882+0000","last_unstale":"2024-03-06T07:21:01.419658+0000","last_undegraded":"2024-03-06T07:21:01.419658+0000","last_fullsized":"2024-03-06T07:21:01.419658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431260+0000","last_change":"2024-03-06T07:20:50.888878+0000","last_active":"2024-03-06T07:21:01.431260+0000","last_peered":"2024-03-06T07:21:01.431260+0000","last_clean":"2024-03-06T07:21:01.431260+0000","last_became_active":"2024-03-06T07:20:50.888604+0000","last_became_peered":"2024-03-06T07:20:50.888604+0000","last_unstale":"2024-03-06T07:21:01.431260+0000","last_undegraded":"2024-03-06T07:21:01.431260+0000","last_fullsized":"2024-03-06T07:21:01.431260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:33.413428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422649+0000","last_change":"2024-03-06T07:21:01.422649+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422649+0000","last_undegraded":"2024-03-06T07:21:01.422649+0000","last_fullsized":"2024-03-06T07:21:01.422649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438997+0000","last_change":"2024-03-06T07:21:01.433417+0000","last_active":"2024-03-06T07:21:01.438997+0000","last_peered":"2024-03-06T07:21:01.438997+0000","last_clean":"2024-03-06T07:21:01.438997+0000","last_became_active":"2024-03-06T07:21:01.432105+0000","last_became_peered":"2024-03-06T07:21:01.432105+0000","last_unstale":"2024-03-06T07:21:01.438997+0000","last_undegraded":"2024-03-06T07:21:01.438997+0000","last_fullsized":"2024-03-06T07:21:01.438997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:18.120482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425648+0000","last_change":"2024-03-06T07:20:51.893070+0000","last_active":"2024-03-06T07:21:01.425648+0000","last_peered":"2024-03-06T07:21:01.425648+0000","last_clean":"2024-03-06T07:21:01.425648+0000","last_became_active":"2024-03-06T07:20:51.892785+0000","last_became_peered":"2024-03-06T07:20:51.892785+0000","last_unstale":"2024-03-06T07:21:01.425648+0000","last_undegraded":"2024-03-06T07:21:01.425648+0000","last_fullsized":"2024-03-06T07:21:01.425648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:09.591477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425357+0000","last_change":"2024-03-06T07:20:50.892942+0000","last_active":"2024-03-06T07:21:01.425357+0000","last_peered":"2024-03-06T07:21:01.425357+0000","last_clean":"2024-03-06T07:21:01.425357+0000","last_became_active":"2024-03-06T07:20:50.892638+0000","last_became_peered":"2024-03-06T07:20:50.892638+0000","last_unstale":"2024-03-06T07:21:01.425357+0000","last_undegraded":"2024-03-06T07:21:01.425357+0000","last_fullsized":"2024-03-06T07:21:01.425357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:40.434821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427886+0000","last_change":"2024-03-06T07:21:01.427886+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427886+0000","last_undegraded":"2024-03-06T07:21:01.427886+0000","last_fullsized":"2024-03-06T07:21:01.427886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438316+0000","last_change":"2024-03-06T07:21:01.435337+0000","last_active":"2024-03-06T07:21:01.438316+0000","last_peered":"2024-03-06T07:21:01.438316+0000","last_clean":"2024-03-06T07:21:01.438316+0000","last_became_active":"2024-03-06T07:21:01.435098+0000","last_became_peered":"2024-03-06T07:21:01.435098+0000","last_unstale":"2024-03-06T07:21:01.438316+0000","last_undegraded":"2024-03-06T07:21:01.438316+0000","last_fullsized":"2024-03-06T07:21:01.438316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425885+0000","last_change":"2024-03-06T07:20:51.895826+0000","last_active":"2024-03-06T07:21:01.425885+0000","last_peered":"2024-03-06T07:21:01.425885+0000","last_clean":"2024-03-06T07:21:01.425885+0000","last_became_active":"2024-03-06T07:20:51.895483+0000","last_became_peered":"2024-03-06T07:20:51.895483+0000","last_unstale":"2024-03-06T07:21:01.425885+0000","last_undegraded":"2024-03-06T07:21:01.425885+0000","last_fullsized":"2024-03-06T07:21:01.425885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425724+0000","last_change":"2024-03-06T07:20:50.879754+0000","last_active":"2024-03-06T07:21:01.425724+0000","last_peered":"2024-03-06T07:21:01.425724+0000","last_clean":"2024-03-06T07:21:01.425724+0000","last_became_active":"2024-03-06T07:20:50.879488+0000","last_became_peered":"2024-03-06T07:20:50.879488+0000","last_unstale":"2024-03-06T07:21:01.425724+0000","last_undegraded":"2024-03-06T07:21:01.425724+0000","last_fullsized":"2024-03-06T07:21:01.425724+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.603045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422772+0000","last_change":"2024-03-06T07:21:01.422772+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422772+0000","last_undegraded":"2024-03-06T07:21:01.422772+0000","last_fullsized":"2024-03-06T07:21:01.422772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441007+0000","last_change":"2024-03-06T07:21:01.439920+0000","last_active":"2024-03-06T07:21:01.441007+0000","last_peered":"2024-03-06T07:21:01.441007+0000","last_clean":"2024-03-06T07:21:01.441007+0000","last_became_active":"2024-03-06T07:21:01.439672+0000","last_became_peered":"2024-03-06T07:21:01.439672+0000","last_unstale":"2024-03-06T07:21:01.441007+0000","last_undegraded":"2024-03-06T07:21:01.441007+0000","last_fullsized":"2024-03-06T07:21:01.441007+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:37.148955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428677+0000","last_change":"2024-03-06T07:20:51.879085+0000","last_active":"2024-03-06T07:21:01.428677+0000","last_peered":"2024-03-06T07:21:01.428677+0000","last_clean":"2024-03-06T07:21:01.428677+0000","last_became_active":"2024-03-06T07:20:51.878776+0000","last_became_peered":"2024-03-06T07:20:51.878776+0000","last_unstale":"2024-03-06T07:21:01.428677+0000","last_undegraded":"2024-03-06T07:21:01.428677+0000","last_fullsized":"2024-03-06T07:21:01.428677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426282+0000","last_change":"2024-03-06T07:20:50.887928+0000","last_active":"2024-03-06T07:21:01.426282+0000","last_peered":"2024-03-06T07:21:01.426282+0000","last_clean":"2024-03-06T07:21:01.426282+0000","last_became_active":"2024-03-06T07:20:50.887638+0000","last_became_peered":"2024-03-06T07:20:50.887638+0000","last_unstale":"2024-03-06T07:21:01.426282+0000","last_undegraded":"2024-03-06T07:21:01.426282+0000","last_fullsized":"2024-03-06T07:21:01.426282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:52.309668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422497+0000","last_change":"2024-03-06T07:21:01.422497+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422497+0000","last_undegraded":"2024-03-06T07:21:01.422497+0000","last_fullsized":"2024-03-06T07:21:01.422497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440089+0000","last_change":"2024-03-06T07:21:01.436756+0000","last_active":"2024-03-06T07:21:01.440089+0000","last_peered":"2024-03-06T07:21:01.440089+0000","last_clean":"2024-03-06T07:21:01.440089+0000","last_became_active":"2024-03-06T07:21:01.436158+0000","last_became_peered":"2024-03-06T07:21:01.436158+0000","last_unstale":"2024-03-06T07:21:01.440089+0000","last_undegraded":"2024-03-06T07:21:01.440089+0000","last_fullsized":"2024-03-06T07:21:01.440089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:11.908825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431875+0000","last_change":"2024-03-06T07:20:51.883253+0000","last_active":"2024-03-06T07:21:01.431875+0000","last_peered":"2024-03-06T07:21:01.431875+0000","last_clean":"2024-03-06T07:21:01.431875+0000","last_became_active":"2024-03-06T07:20:51.882993+0000","last_became_peered":"2024-03-06T07:20:51.882993+0000","last_unstale":"2024-03-06T07:21:01.431875+0000","last_undegraded":"2024-03-06T07:21:01.431875+0000","last_fullsized":"2024-03-06T07:21:01.431875+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426104+0000","last_change":"2024-03-06T07:20:50.879149+0000","last_active":"2024-03-06T07:21:01.426104+0000","last_peered":"2024-03-06T07:21:01.426104+0000","last_clean":"2024-03-06T07:21:01.426104+0000","last_became_active":"2024-03-06T07:20:50.878867+0000","last_became_peered":"2024-03-06T07:20:50.878867+0000","last_unstale":"2024-03-06T07:21:01.426104+0000","last_undegraded":"2024-03-06T07:21:01.426104+0000","last_fullsized":"2024-03-06T07:21:01.426104+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:57.394196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429819+0000","last_change":"2024-03-06T07:21:01.429819+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429819+0000","last_undegraded":"2024-03-06T07:21:01.429819+0000","last_fullsized":"2024-03-06T07:21:01.429819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442400+0000","last_change":"2024-03-06T07:21:01.440630+0000","last_active":"2024-03-06T07:21:01.442400+0000","last_peered":"2024-03-06T07:21:01.442400+0000","last_clean":"2024-03-06T07:21:01.442400+0000","last_became_active":"2024-03-06T07:21:01.440375+0000","last_became_peered":"2024-03-06T07:21:01.440375+0000","last_unstale":"2024-03-06T07:21:01.442400+0000","last_undegraded":"2024-03-06T07:21:01.442400+0000","last_fullsized":"2024-03-06T07:21:01.442400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432805+0000","last_change":"2024-03-06T07:20:51.891707+0000","last_active":"2024-03-06T07:21:01.432805+0000","last_peered":"2024-03-06T07:21:01.432805+0000","last_clean":"2024-03-06T07:21:01.432805+0000","last_became_active":"2024-03-06T07:20:51.891427+0000","last_became_peered":"2024-03-06T07:20:51.891427+0000","last_unstale":"2024-03-06T07:21:01.432805+0000","last_undegraded":"2024-03-06T07:21:01.432805+0000","last_fullsized":"2024-03-06T07:21:01.432805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425699+0000","last_change":"2024-03-06T07:20:50.885372+0000","last_active":"2024-03-06T07:21:01.425699+0000","last_peered":"2024-03-06T07:21:01.425699+0000","last_clean":"2024-03-06T07:21:01.425699+0000","last_became_active":"2024-03-06T07:20:50.885093+0000","last_became_peered":"2024-03-06T07:20:50.885093+0000","last_unstale":"2024-03-06T07:21:01.425699+0000","last_undegraded":"2024-03-06T07:21:01.425699+0000","last_fullsized":"2024-03-06T07:21:01.425699+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:12.807980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":276,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430887+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:21:01.430887+0000","last_peered":"2024-03-06T07:21:01.430887+0000","last_clean":"2024-03-06T07:21:01.430887+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:21:01.430887+0000","last_undegraded":"2024-03-06T07:21:01.430887+0000","last_fullsized":"2024-03-06T07:21:01.430887+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":1031546,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":442768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":441190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":950272,"data_stored":442872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":933888,"data_stored":441738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":441898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":441268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":1031524,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:05.809 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:05.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.907+0000 7f63435d7640 1 Processor -- start 2024-03-06T07:21:05.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.907+0000 7f63435d7640 1 -- start start 2024-03-06T07:21:05.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f63435d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c156150 0x7f633c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f63435d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c0c28d0 con 0x7f633c14f790 2024-03-06T07:21:05.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f63435d7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c0c2bb0 con 0x7f633c0c1aa0 2024-03-06T07:21:05.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f63435d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0c2680 con 0x7f633c156150 2024-03-06T07:21:05.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6341373640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f633c14f790 0x7f633c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41904/0 (socket says 172.21.15.94:41904) 2024-03-06T07:21:05.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6341373640 1 -- 172.21.15.94:0/3675101710 learned_addr learned my addr 172.21.15.94:0/3675101710 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6341b74640 1 --2- 172.21.15.94:0/3675101710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c156150 0x7f633c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6333fff640 1 -- 172.21.15.94:0/3675101710 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454303160 0 0) 0x7f633c0c28d0 con 0x7f633c14f790 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6333fff640 1 -- 172.21.15.94:0/3675101710 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6328003520 con 0x7f633c14f790 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6333fff640 1 -- 172.21.15.94:0/3675101710 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163267637 0 0) 0x7f633c0c2bb0 con 0x7f633c0c1aa0 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6333fff640 1 -- 172.21.15.94:0/3675101710 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f633c0c28d0 con 0x7f633c0c1aa0 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6341b74640 1 -- 172.21.15.94:0/3675101710 >> v1:172.21.15.145:6789/0 conn(0x7f633c0c1aa0 legacy=0x7f633c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6341b74640 1 -- 172.21.15.94:0/3675101710 >> v1:172.21.15.94:6789/0 conn(0x7f633c14f790 legacy=0x7f633c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.908+0000 7f6341b74640 1 -- 172.21.15.94:0/3675101710 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c0c2e90 con 0x7f633c156150 2024-03-06T07:21:05.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.909+0000 7f6341b74640 1 --2- 172.21.15.94:0/3675101710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c156150 0x7f633c1529d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f63380097d0 tx=0x7f633802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16faab3eea17166f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.909+0000 7f6333fff640 1 -- 172.21.15.94:0/3675101710 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338035020 con 0x7f633c156150 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.909+0000 7f6333fff640 1 -- 172.21.15.94:0/3675101710 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6338031890 con 0x7f633c156150 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.909+0000 7f6333fff640 1 -- 172.21.15.94:0/3675101710 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338031c40 con 0x7f633c156150 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.910+0000 7f63435d7640 1 -- 172.21.15.94:0/3675101710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c156150 msgr2=0x7f633c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.910+0000 7f63435d7640 1 --2- 172.21.15.94:0/3675101710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c156150 0x7f633c1529d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f63380097d0 tx=0x7f633802b930 comp rx=0 tx=0).stop 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.910+0000 7f63435d7640 1 -- 172.21.15.94:0/3675101710 shutdown_connections 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.910+0000 7f63435d7640 1 --2- 172.21.15.94:0/3675101710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c156150 0x7f633c1529d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.910+0000 7f63435d7640 1 -- 172.21.15.94:0/3675101710 >> 172.21.15.94:0/3675101710 conn(0x7f633c0c6710 msgr2=0x7f633c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.910+0000 7f63435d7640 1 -- 172.21.15.94:0/3675101710 shutdown_connections 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.910+0000 7f63435d7640 1 -- 172.21.15.94:0/3675101710 wait complete. 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.911+0000 7f63435d7640 1 Processor -- start 2024-03-06T07:21:05.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.911+0000 7f63435d7640 1 -- start start 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.911+0000 7f63435d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c0c1aa0 0x7f633c1e8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f63435d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f633c14f790 0x7f633c0715d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f63435d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f633c156150 0x7f633c14bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f63435d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0c5e30 con 0x7f633c156150 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f63435d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0c0d60 con 0x7f633c14f790 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f63435d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0c5fb0 con 0x7f633c0c1aa0 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f633c156150 0x7f633c14bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f633c156150 0x7f633c14bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38924/0 (socket says 172.21.15.94:38924) 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 -- 172.21.15.94:0/3043524646 learned_addr learned my addr 172.21.15.94:0/3043524646 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6340b72640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f633c14f790 0x7f633c0715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341373640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c0c1aa0 0x7f633c1e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 -- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c0c1aa0 msgr2=0x7f633c1e8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c0c1aa0 0x7f633c1e8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 -- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f633c14f790 msgr2=0x7f633c0715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f633c14f790 0x7f633c0715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 -- 172.21.15.94:0/3043524646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c1db630 con 0x7f633c156150 2024-03-06T07:21:05.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6341b74640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f633c156150 0x7f633c14bf80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f633802b970 tx=0x7f63380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6331ffb640 1 -- 172.21.15.94:0/3043524646 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338042070 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6331ffb640 1 -- 172.21.15.94:0/3043524646 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6338030ba0 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c1db910 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f6331ffb640 1 -- 172.21.15.94:0/3043524646 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338035040 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.912+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c071b10 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.914+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633c04f330 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.914+0000 7f6331ffb640 1 -- 172.21.15.94:0/3043524646 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6338038020 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.914+0000 7f6331ffb640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f632006b500 0x7f632006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.914+0000 7f6331ffb640 1 -- 172.21.15.94:0/3043524646 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f63380ae5c0 con 0x7f633c156150 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.914+0000 7f6341373640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f632006b500 0x7f632006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:05.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.915+0000 7f6341373640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f632006b500 0x7f632006d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f63340012e0 tx=0x7f6334001ad0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:05.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:05.918+0000 7f6331ffb640 1 -- 172.21.15.94:0/3043524646 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f633802d040 con 0x7f633c156150 2024-03-06T07:21:06.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.086+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f633c0703d0 con 0x7f632006b500 2024-03-06T07:21:06.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.097+0000 7f6331ffb640 1 -- 172.21.15.94:0/3043524646 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485419 (secure 0 0 0) 0x7f6334001f60 con 0x7f632006b500 2024-03-06T07:21:06.104 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:06.107 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:21:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f632006b500 msgr2=0x7f632006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f632006b500 0x7f632006d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f63340012e0 tx=0x7f6334001ad0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f633c156150 msgr2=0x7f633c14bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f633c156150 0x7f633c14bf80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f633802b970 tx=0x7f63380046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 shutdown_connections 2024-03-06T07:21:06.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f632006b500 0x7f632006d9c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f633c156150 0x7f633c14bf80 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f633c14f790 0x7f633c0715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 --2- 172.21.15.94:0/3043524646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f633c0c1aa0 0x7f633c1e8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 >> 172.21.15.94:0/3043524646 conn(0x7f633c0c6710 msgr2=0x7f633c0c3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 shutdown_connections 2024-03-06T07:21:06.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.101+0000 7f63435d7640 1 -- 172.21.15.94:0/3043524646 wait complete. 2024-03-06T07:21:06.114 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-03-06T07:21:04.671494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244424,"kb_used_data":7456,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730296,"statfs":{"total":773094113280,"available":772843823104,"internally_reserved":0,"allocated":7634944,"data_stored":4714804,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.252754"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427479+0000","last_change":"2024-03-06T07:21:01.427479+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427479+0000","last_undegraded":"2024-03-06T07:21:01.427479+0000","last_fullsized":"2024-03-06T07:21:01.427479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431835+0000","last_change":"2024-03-06T07:20:51.896309+0000","last_active":"2024-03-06T07:21:01.431835+0000","last_peered":"2024-03-06T07:21:01.431835+0000","last_clean":"2024-03-06T07:21:01.431835+0000","last_became_active":"2024-03-06T07:20:51.896009+0000","last_became_peered":"2024-03-06T07:20:51.896009+0000","last_unstale":"2024-03-06T07:21:01.431835+0000","last_undegraded":"2024-03-06T07:21:01.431835+0000","last_fullsized":"2024-03-06T07:21:01.431835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426920+0000","last_change":"2024-03-06T07:21:01.426920+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426920+0000","last_undegraded":"2024-03-06T07:21:01.426920+0000","last_fullsized":"2024-03-06T07:21:01.426920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419624+0000","last_change":"2024-03-06T07:20:51.895412+0000","last_active":"2024-03-06T07:21:01.419624+0000","last_peered":"2024-03-06T07:21:01.419624+0000","last_clean":"2024-03-06T07:21:01.419624+0000","last_became_active":"2024-03-06T07:20:51.895169+0000","last_became_peered":"2024-03-06T07:20:51.895169+0000","last_unstale":"2024-03-06T07:21:01.419624+0000","last_undegraded":"2024-03-06T07:21:01.419624+0000","last_fullsized":"2024-03-06T07:21:01.419624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427931+0000","last_change":"2024-03-06T07:21:01.427931+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427931+0000","last_undegraded":"2024-03-06T07:21:01.427931+0000","last_fullsized":"2024-03-06T07:21:01.427931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428431+0000","last_change":"2024-03-06T07:20:51.878718+0000","last_active":"2024-03-06T07:21:01.428431+0000","last_peered":"2024-03-06T07:21:01.428431+0000","last_clean":"2024-03-06T07:21:01.428431+0000","last_became_active":"2024-03-06T07:20:51.878467+0000","last_became_peered":"2024-03-06T07:20:51.878467+0000","last_unstale":"2024-03-06T07:21:01.428431+0000","last_undegraded":"2024-03-06T07:21:01.428431+0000","last_fullsized":"2024-03-06T07:21:01.428431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428790+0000","last_change":"2024-03-06T07:21:01.428790+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428790+0000","last_undegraded":"2024-03-06T07:21:01.428790+0000","last_fullsized":"2024-03-06T07:21:01.428790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431362+0000","last_change":"2024-03-06T07:20:51.892550+0000","last_active":"2024-03-06T07:21:01.431362+0000","last_peered":"2024-03-06T07:21:01.431362+0000","last_clean":"2024-03-06T07:21:01.431362+0000","last_became_active":"2024-03-06T07:20:51.892264+0000","last_became_peered":"2024-03-06T07:20:51.892264+0000","last_unstale":"2024-03-06T07:21:01.431362+0000","last_undegraded":"2024-03-06T07:21:01.431362+0000","last_fullsized":"2024-03-06T07:21:01.431362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429413+0000","last_change":"2024-03-06T07:21:01.429413+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429413+0000","last_undegraded":"2024-03-06T07:21:01.429413+0000","last_fullsized":"2024-03-06T07:21:01.429413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428191+0000","last_change":"2024-03-06T07:20:51.893305+0000","last_active":"2024-03-06T07:21:01.428191+0000","last_peered":"2024-03-06T07:21:01.428191+0000","last_clean":"2024-03-06T07:21:01.428191+0000","last_became_active":"2024-03-06T07:20:51.893041+0000","last_became_peered":"2024-03-06T07:20:51.893041+0000","last_unstale":"2024-03-06T07:21:01.428191+0000","last_undegraded":"2024-03-06T07:21:01.428191+0000","last_fullsized":"2024-03-06T07:21:01.428191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:39.229585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424683+0000","last_change":"2024-03-06T07:21:01.424683+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424683+0000","last_undegraded":"2024-03-06T07:21:01.424683+0000","last_fullsized":"2024-03-06T07:21:01.424683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425180+0000","last_change":"2024-03-06T07:20:51.893812+0000","last_active":"2024-03-06T07:21:01.425180+0000","last_peered":"2024-03-06T07:21:01.425180+0000","last_clean":"2024-03-06T07:21:01.425180+0000","last_became_active":"2024-03-06T07:20:51.891649+0000","last_became_peered":"2024-03-06T07:20:51.891649+0000","last_unstale":"2024-03-06T07:21:01.425180+0000","last_undegraded":"2024-03-06T07:21:01.425180+0000","last_fullsized":"2024-03-06T07:21:01.425180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:46.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424861+0000","last_change":"2024-03-06T07:21:01.424861+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424861+0000","last_undegraded":"2024-03-06T07:21:01.424861+0000","last_fullsized":"2024-03-06T07:21:01.424861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426172+0000","last_change":"2024-03-06T07:20:51.894141+0000","last_active":"2024-03-06T07:21:01.426172+0000","last_peered":"2024-03-06T07:21:01.426172+0000","last_clean":"2024-03-06T07:21:01.426172+0000","last_became_active":"2024-03-06T07:20:51.893793+0000","last_became_peered":"2024-03-06T07:20:51.893793+0000","last_unstale":"2024-03-06T07:21:01.426172+0000","last_undegraded":"2024-03-06T07:21:01.426172+0000","last_fullsized":"2024-03-06T07:21:01.426172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430080+0000","last_change":"2024-03-06T07:21:01.430080+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430080+0000","last_undegraded":"2024-03-06T07:21:01.430080+0000","last_fullsized":"2024-03-06T07:21:01.430080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427332+0000","last_change":"2024-03-06T07:20:51.894037+0000","last_active":"2024-03-06T07:21:01.427332+0000","last_peered":"2024-03-06T07:21:01.427332+0000","last_clean":"2024-03-06T07:21:01.427332+0000","last_became_active":"2024-03-06T07:20:51.893773+0000","last_became_peered":"2024-03-06T07:20:51.893773+0000","last_unstale":"2024-03-06T07:21:01.427332+0000","last_undegraded":"2024-03-06T07:21:01.427332+0000","last_fullsized":"2024-03-06T07:21:01.427332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:48.344455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424697+0000","last_change":"2024-03-06T07:21:01.424697+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424697+0000","last_undegraded":"2024-03-06T07:21:01.424697+0000","last_fullsized":"2024-03-06T07:21:01.424697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427134+0000","last_change":"2024-03-06T07:20:51.886223+0000","last_active":"2024-03-06T07:21:01.427134+0000","last_peered":"2024-03-06T07:21:01.427134+0000","last_clean":"2024-03-06T07:21:01.427134+0000","last_became_active":"2024-03-06T07:20:51.885985+0000","last_became_peered":"2024-03-06T07:20:51.885985+0000","last_unstale":"2024-03-06T07:21:01.427134+0000","last_undegraded":"2024-03-06T07:21:01.427134+0000","last_fullsized":"2024-03-06T07:21:01.427134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:13.814018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426060+0000","last_change":"2024-03-06T07:21:01.426060+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426060+0000","last_undegraded":"2024-03-06T07:21:01.426060+0000","last_fullsized":"2024-03-06T07:21:01.426060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431693+0000","last_change":"2024-03-06T07:20:51.890465+0000","last_active":"2024-03-06T07:21:01.431693+0000","last_peered":"2024-03-06T07:21:01.431693+0000","last_clean":"2024-03-06T07:21:01.431693+0000","last_became_active":"2024-03-06T07:20:51.890183+0000","last_became_peered":"2024-03-06T07:20:51.890183+0000","last_unstale":"2024-03-06T07:21:01.431693+0000","last_undegraded":"2024-03-06T07:21:01.431693+0000","last_fullsized":"2024-03-06T07:21:01.431693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425029+0000","last_change":"2024-03-06T07:21:01.425029+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425029+0000","last_undegraded":"2024-03-06T07:21:01.425029+0000","last_fullsized":"2024-03-06T07:21:01.425029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419915+0000","last_change":"2024-03-06T07:20:51.897836+0000","last_active":"2024-03-06T07:21:01.419915+0000","last_peered":"2024-03-06T07:21:01.419915+0000","last_clean":"2024-03-06T07:21:01.419915+0000","last_became_active":"2024-03-06T07:20:51.897551+0000","last_became_peered":"2024-03-06T07:20:51.897551+0000","last_unstale":"2024-03-06T07:21:01.419915+0000","last_undegraded":"2024-03-06T07:21:01.419915+0000","last_fullsized":"2024-03-06T07:21:01.419915+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424341+0000","last_change":"2024-03-06T07:21:01.424341+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424341+0000","last_undegraded":"2024-03-06T07:21:01.424341+0000","last_fullsized":"2024-03-06T07:21:01.424341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431445+0000","last_change":"2024-03-06T07:20:51.886475+0000","last_active":"2024-03-06T07:21:01.431445+0000","last_peered":"2024-03-06T07:21:01.431445+0000","last_clean":"2024-03-06T07:21:01.431445+0000","last_became_active":"2024-03-06T07:20:51.886218+0000","last_became_peered":"2024-03-06T07:20:51.886218+0000","last_unstale":"2024-03-06T07:21:01.431445+0000","last_undegraded":"2024-03-06T07:21:01.431445+0000","last_fullsized":"2024-03-06T07:21:01.431445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426115+0000","last_change":"2024-03-06T07:21:01.426115+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426115+0000","last_undegraded":"2024-03-06T07:21:01.426115+0000","last_fullsized":"2024-03-06T07:21:01.426115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428289+0000","last_change":"2024-03-06T07:20:51.887999+0000","last_active":"2024-03-06T07:21:01.428289+0000","last_peered":"2024-03-06T07:21:01.428289+0000","last_clean":"2024-03-06T07:21:01.428289+0000","last_became_active":"2024-03-06T07:20:51.887749+0000","last_became_peered":"2024-03-06T07:20:51.887749+0000","last_unstale":"2024-03-06T07:21:01.428289+0000","last_undegraded":"2024-03-06T07:21:01.428289+0000","last_fullsized":"2024-03-06T07:21:01.428289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:25.243948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.431576+0000","last_change":"2024-03-06T07:21:01.431576+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.431576+0000","last_undegraded":"2024-03-06T07:21:01.431576+0000","last_fullsized":"2024-03-06T07:21:01.431576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425276+0000","last_change":"2024-03-06T07:20:51.892140+0000","last_active":"2024-03-06T07:21:01.425276+0000","last_peered":"2024-03-06T07:21:01.425276+0000","last_clean":"2024-03-06T07:21:01.425276+0000","last_became_active":"2024-03-06T07:20:51.891195+0000","last_became_peered":"2024-03-06T07:20:51.891195+0000","last_unstale":"2024-03-06T07:21:01.425276+0000","last_undegraded":"2024-03-06T07:21:01.425276+0000","last_fullsized":"2024-03-06T07:21:01.425276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:28.641052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424813+0000","last_change":"2024-03-06T07:21:01.424813+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424813+0000","last_undegraded":"2024-03-06T07:21:01.424813+0000","last_fullsized":"2024-03-06T07:21:01.424813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419327+0000","last_change":"2024-03-06T07:20:51.894735+0000","last_active":"2024-03-06T07:21:01.419327+0000","last_peered":"2024-03-06T07:21:01.419327+0000","last_clean":"2024-03-06T07:21:01.419327+0000","last_became_active":"2024-03-06T07:20:51.894479+0000","last_became_peered":"2024-03-06T07:20:51.894479+0000","last_unstale":"2024-03-06T07:21:01.419327+0000","last_undegraded":"2024-03-06T07:21:01.419327+0000","last_fullsized":"2024-03-06T07:21:01.419327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425999+0000","last_change":"2024-03-06T07:21:01.425999+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425999+0000","last_undegraded":"2024-03-06T07:21:01.425999+0000","last_fullsized":"2024-03-06T07:21:01.425999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430687+0000","last_change":"2024-03-06T07:20:51.892450+0000","last_active":"2024-03-06T07:21:01.430687+0000","last_peered":"2024-03-06T07:21:01.430687+0000","last_clean":"2024-03-06T07:21:01.430687+0000","last_became_active":"2024-03-06T07:20:51.892184+0000","last_became_peered":"2024-03-06T07:20:51.892184+0000","last_unstale":"2024-03-06T07:21:01.430687+0000","last_undegraded":"2024-03-06T07:21:01.430687+0000","last_fullsized":"2024-03-06T07:21:01.430687+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424318+0000","last_change":"2024-03-06T07:21:01.424318+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424318+0000","last_undegraded":"2024-03-06T07:21:01.424318+0000","last_fullsized":"2024-03-06T07:21:01.424318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426749+0000","last_change":"2024-03-06T07:20:51.891492+0000","last_active":"2024-03-06T07:21:01.426749+0000","last_peered":"2024-03-06T07:21:01.426749+0000","last_clean":"2024-03-06T07:21:01.426749+0000","last_became_active":"2024-03-06T07:20:51.891212+0000","last_became_peered":"2024-03-06T07:20:51.891212+0000","last_unstale":"2024-03-06T07:21:01.426749+0000","last_undegraded":"2024-03-06T07:21:01.426749+0000","last_fullsized":"2024-03-06T07:21:01.426749+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:49.555650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426582+0000","last_change":"2024-03-06T07:21:01.426582+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426582+0000","last_undegraded":"2024-03-06T07:21:01.426582+0000","last_fullsized":"2024-03-06T07:21:01.426582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428589+0000","last_change":"2024-03-06T07:20:51.876410+0000","last_active":"2024-03-06T07:21:01.428589+0000","last_peered":"2024-03-06T07:21:01.428589+0000","last_clean":"2024-03-06T07:21:01.428589+0000","last_became_active":"2024-03-06T07:20:51.876155+0000","last_became_peered":"2024-03-06T07:20:51.876155+0000","last_unstale":"2024-03-06T07:21:01.428589+0000","last_undegraded":"2024-03-06T07:21:01.428589+0000","last_fullsized":"2024-03-06T07:21:01.428589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429516+0000","last_change":"2024-03-06T07:21:01.429516+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429516+0000","last_undegraded":"2024-03-06T07:21:01.429516+0000","last_fullsized":"2024-03-06T07:21:01.429516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425366+0000","last_change":"2024-03-06T07:20:51.880019+0000","last_active":"2024-03-06T07:21:01.425366+0000","last_peered":"2024-03-06T07:21:01.425366+0000","last_clean":"2024-03-06T07:21:01.425366+0000","last_became_active":"2024-03-06T07:20:51.879756+0000","last_became_peered":"2024-03-06T07:20:51.879756+0000","last_unstale":"2024-03-06T07:21:01.425366+0000","last_undegraded":"2024-03-06T07:21:01.425366+0000","last_fullsized":"2024-03-06T07:21:01.425366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.594449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426508+0000","last_change":"2024-03-06T07:21:01.426508+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426508+0000","last_undegraded":"2024-03-06T07:21:01.426508+0000","last_fullsized":"2024-03-06T07:21:01.426508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426322+0000","last_change":"2024-03-06T07:20:51.890815+0000","last_active":"2024-03-06T07:21:01.426322+0000","last_peered":"2024-03-06T07:21:01.426322+0000","last_clean":"2024-03-06T07:21:01.426322+0000","last_became_active":"2024-03-06T07:20:51.890558+0000","last_became_peered":"2024-03-06T07:20:51.890558+0000","last_unstale":"2024-03-06T07:21:01.426322+0000","last_undegraded":"2024-03-06T07:21:01.426322+0000","last_fullsized":"2024-03-06T07:21:01.426322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:50.925154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423254+0000","last_change":"2024-03-06T07:21:01.423254+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423254+0000","last_undegraded":"2024-03-06T07:21:01.423254+0000","last_fullsized":"2024-03-06T07:21:01.423254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426944+0000","last_change":"2024-03-06T07:20:51.884508+0000","last_active":"2024-03-06T07:21:01.426944+0000","last_peered":"2024-03-06T07:21:01.426944+0000","last_clean":"2024-03-06T07:21:01.426944+0000","last_became_active":"2024-03-06T07:20:51.884237+0000","last_became_peered":"2024-03-06T07:20:51.884237+0000","last_unstale":"2024-03-06T07:21:01.426944+0000","last_undegraded":"2024-03-06T07:21:01.426944+0000","last_fullsized":"2024-03-06T07:21:01.426944+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422935+0000","last_change":"2024-03-06T07:21:01.422935+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422935+0000","last_undegraded":"2024-03-06T07:21:01.422935+0000","last_fullsized":"2024-03-06T07:21:01.422935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425637+0000","last_change":"2024-03-06T07:20:51.890118+0000","last_active":"2024-03-06T07:21:01.425637+0000","last_peered":"2024-03-06T07:21:01.425637+0000","last_clean":"2024-03-06T07:21:01.425637+0000","last_became_active":"2024-03-06T07:20:51.888426+0000","last_became_peered":"2024-03-06T07:20:51.888426+0000","last_unstale":"2024-03-06T07:21:01.425637+0000","last_undegraded":"2024-03-06T07:21:01.425637+0000","last_fullsized":"2024-03-06T07:21:01.425637+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:46.950695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423598+0000","last_change":"2024-03-06T07:21:01.423598+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423598+0000","last_undegraded":"2024-03-06T07:21:01.423598+0000","last_fullsized":"2024-03-06T07:21:01.423598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426626+0000","last_change":"2024-03-06T07:20:51.885627+0000","last_active":"2024-03-06T07:21:01.426626+0000","last_peered":"2024-03-06T07:21:01.426626+0000","last_clean":"2024-03-06T07:21:01.426626+0000","last_became_active":"2024-03-06T07:20:51.885338+0000","last_became_peered":"2024-03-06T07:20:51.885338+0000","last_unstale":"2024-03-06T07:21:01.426626+0000","last_undegraded":"2024-03-06T07:21:01.426626+0000","last_fullsized":"2024-03-06T07:21:01.426626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:41.439891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424033+0000","last_change":"2024-03-06T07:21:01.424033+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424033+0000","last_undegraded":"2024-03-06T07:21:01.424033+0000","last_fullsized":"2024-03-06T07:21:01.424033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419243+0000","last_change":"2024-03-06T07:20:51.884448+0000","last_active":"2024-03-06T07:21:01.419243+0000","last_peered":"2024-03-06T07:21:01.419243+0000","last_clean":"2024-03-06T07:21:01.419243+0000","last_became_active":"2024-03-06T07:20:51.884176+0000","last_became_peered":"2024-03-06T07:20:51.884176+0000","last_unstale":"2024-03-06T07:21:01.419243+0000","last_undegraded":"2024-03-06T07:21:01.419243+0000","last_fullsized":"2024-03-06T07:21:01.419243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:29.691807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427151+0000","last_change":"2024-03-06T07:21:01.427151+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427151+0000","last_undegraded":"2024-03-06T07:21:01.427151+0000","last_fullsized":"2024-03-06T07:21:01.427151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419496+0000","last_change":"2024-03-06T07:20:51.886062+0000","last_active":"2024-03-06T07:21:01.419496+0000","last_peered":"2024-03-06T07:21:01.419496+0000","last_clean":"2024-03-06T07:21:01.419496+0000","last_became_active":"2024-03-06T07:20:51.885799+0000","last_became_peered":"2024-03-06T07:20:51.885799+0000","last_unstale":"2024-03-06T07:21:01.419496+0000","last_undegraded":"2024-03-06T07:21:01.419496+0000","last_fullsized":"2024-03-06T07:21:01.419496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425922+0000","last_change":"2024-03-06T07:21:01.425922+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425922+0000","last_undegraded":"2024-03-06T07:21:01.425922+0000","last_fullsized":"2024-03-06T07:21:01.425922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431786+0000","last_change":"2024-03-06T07:20:51.883614+0000","last_active":"2024-03-06T07:21:01.431786+0000","last_peered":"2024-03-06T07:21:01.431786+0000","last_clean":"2024-03-06T07:21:01.431786+0000","last_became_active":"2024-03-06T07:20:51.883266+0000","last_became_peered":"2024-03-06T07:20:51.883266+0000","last_unstale":"2024-03-06T07:21:01.431786+0000","last_undegraded":"2024-03-06T07:21:01.431786+0000","last_fullsized":"2024-03-06T07:21:01.431786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428340+0000","last_change":"2024-03-06T07:21:01.428340+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428340+0000","last_undegraded":"2024-03-06T07:21:01.428340+0000","last_fullsized":"2024-03-06T07:21:01.428340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432598+0000","last_change":"2024-03-06T07:20:51.893301+0000","last_active":"2024-03-06T07:21:01.432598+0000","last_peered":"2024-03-06T07:21:01.432598+0000","last_clean":"2024-03-06T07:21:01.432598+0000","last_became_active":"2024-03-06T07:20:51.892946+0000","last_became_peered":"2024-03-06T07:20:51.892946+0000","last_unstale":"2024-03-06T07:21:01.432598+0000","last_undegraded":"2024-03-06T07:21:01.432598+0000","last_fullsized":"2024-03-06T07:21:01.432598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426198+0000","last_change":"2024-03-06T07:21:01.426198+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426198+0000","last_undegraded":"2024-03-06T07:21:01.426198+0000","last_fullsized":"2024-03-06T07:21:01.426198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426417+0000","last_change":"2024-03-06T07:20:51.891689+0000","last_active":"2024-03-06T07:21:01.426417+0000","last_peered":"2024-03-06T07:21:01.426417+0000","last_clean":"2024-03-06T07:21:01.426417+0000","last_became_active":"2024-03-06T07:20:51.891341+0000","last_became_peered":"2024-03-06T07:20:51.891341+0000","last_unstale":"2024-03-06T07:21:01.426417+0000","last_undegraded":"2024-03-06T07:21:01.426417+0000","last_fullsized":"2024-03-06T07:21:01.426417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:45.060281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428167+0000","last_change":"2024-03-06T07:21:01.428167+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428167+0000","last_undegraded":"2024-03-06T07:21:01.428167+0000","last_fullsized":"2024-03-06T07:21:01.428167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419570+0000","last_change":"2024-03-06T07:20:51.887454+0000","last_active":"2024-03-06T07:21:01.419570+0000","last_peered":"2024-03-06T07:21:01.419570+0000","last_clean":"2024-03-06T07:21:01.419570+0000","last_became_active":"2024-03-06T07:20:51.887188+0000","last_became_peered":"2024-03-06T07:20:51.887188+0000","last_unstale":"2024-03-06T07:21:01.419570+0000","last_undegraded":"2024-03-06T07:21:01.419570+0000","last_fullsized":"2024-03-06T07:21:01.419570+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425332+0000","last_change":"2024-03-06T07:21:01.425332+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425332+0000","last_undegraded":"2024-03-06T07:21:01.425332+0000","last_fullsized":"2024-03-06T07:21:01.425332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430984+0000","last_change":"2024-03-06T07:20:51.899932+0000","last_active":"2024-03-06T07:21:01.430984+0000","last_peered":"2024-03-06T07:21:01.430984+0000","last_clean":"2024-03-06T07:21:01.430984+0000","last_became_active":"2024-03-06T07:20:51.899569+0000","last_became_peered":"2024-03-06T07:20:51.899569+0000","last_unstale":"2024-03-06T07:21:01.430984+0000","last_undegraded":"2024-03-06T07:21:01.430984+0000","last_fullsized":"2024-03-06T07:21:01.430984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429450+0000","last_change":"2024-03-06T07:21:01.429450+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429450+0000","last_undegraded":"2024-03-06T07:21:01.429450+0000","last_fullsized":"2024-03-06T07:21:01.429450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432122+0000","last_change":"2024-03-06T07:20:51.888537+0000","last_active":"2024-03-06T07:21:01.432122+0000","last_peered":"2024-03-06T07:21:01.432122+0000","last_clean":"2024-03-06T07:21:01.432122+0000","last_became_active":"2024-03-06T07:20:51.888281+0000","last_became_peered":"2024-03-06T07:20:51.888281+0000","last_unstale":"2024-03-06T07:21:01.432122+0000","last_undegraded":"2024-03-06T07:21:01.432122+0000","last_fullsized":"2024-03-06T07:21:01.432122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:03.657994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424979+0000","last_change":"2024-03-06T07:21:01.424979+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424979+0000","last_undegraded":"2024-03-06T07:21:01.424979+0000","last_fullsized":"2024-03-06T07:21:01.424979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430775+0000","last_change":"2024-03-06T07:20:51.896158+0000","last_active":"2024-03-06T07:21:01.430775+0000","last_peered":"2024-03-06T07:21:01.430775+0000","last_clean":"2024-03-06T07:21:01.430775+0000","last_became_active":"2024-03-06T07:20:51.895889+0000","last_became_peered":"2024-03-06T07:20:51.895889+0000","last_unstale":"2024-03-06T07:21:01.430775+0000","last_undegraded":"2024-03-06T07:21:01.430775+0000","last_fullsized":"2024-03-06T07:21:01.430775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424138+0000","last_change":"2024-03-06T07:21:01.424138+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424138+0000","last_undegraded":"2024-03-06T07:21:01.424138+0000","last_fullsized":"2024-03-06T07:21:01.424138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440853+0000","last_change":"2024-03-06T07:21:01.440204+0000","last_active":"2024-03-06T07:21:01.440853+0000","last_peered":"2024-03-06T07:21:01.440853+0000","last_clean":"2024-03-06T07:21:01.440853+0000","last_became_active":"2024-03-06T07:21:01.439934+0000","last_became_peered":"2024-03-06T07:21:01.439934+0000","last_unstale":"2024-03-06T07:21:01.440853+0000","last_undegraded":"2024-03-06T07:21:01.440853+0000","last_fullsized":"2024-03-06T07:21:01.440853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:52.584141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424952+0000","last_change":"2024-03-06T07:20:51.876393+0000","last_active":"2024-03-06T07:21:01.424952+0000","last_peered":"2024-03-06T07:21:01.424952+0000","last_clean":"2024-03-06T07:21:01.424952+0000","last_became_active":"2024-03-06T07:20:51.876059+0000","last_became_peered":"2024-03-06T07:20:51.876059+0000","last_unstale":"2024-03-06T07:21:01.424952+0000","last_undegraded":"2024-03-06T07:21:01.424952+0000","last_fullsized":"2024-03-06T07:21:01.424952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426852+0000","last_change":"2024-03-06T07:20:50.879586+0000","last_active":"2024-03-06T07:21:01.426852+0000","last_peered":"2024-03-06T07:21:01.426852+0000","last_clean":"2024-03-06T07:21:01.426852+0000","last_became_active":"2024-03-06T07:20:50.879319+0000","last_became_peered":"2024-03-06T07:20:50.879319+0000","last_unstale":"2024-03-06T07:21:01.426852+0000","last_undegraded":"2024-03-06T07:21:01.426852+0000","last_fullsized":"2024-03-06T07:21:01.426852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:58.810754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424424+0000","last_change":"2024-03-06T07:21:01.424424+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424424+0000","last_undegraded":"2024-03-06T07:21:01.424424+0000","last_fullsized":"2024-03-06T07:21:01.424424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438871+0000","last_change":"2024-03-06T07:21:01.436192+0000","last_active":"2024-03-06T07:21:01.438871+0000","last_peered":"2024-03-06T07:21:01.438871+0000","last_clean":"2024-03-06T07:21:01.438871+0000","last_became_active":"2024-03-06T07:21:01.435924+0000","last_became_peered":"2024-03-06T07:21:01.435924+0000","last_unstale":"2024-03-06T07:21:01.438871+0000","last_undegraded":"2024-03-06T07:21:01.438871+0000","last_fullsized":"2024-03-06T07:21:01.438871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419269+0000","last_change":"2024-03-06T07:20:51.884046+0000","last_active":"2024-03-06T07:21:01.419269+0000","last_peered":"2024-03-06T07:21:01.419269+0000","last_clean":"2024-03-06T07:21:01.419269+0000","last_became_active":"2024-03-06T07:20:51.883593+0000","last_became_peered":"2024-03-06T07:20:51.883593+0000","last_unstale":"2024-03-06T07:21:01.419269+0000","last_undegraded":"2024-03-06T07:21:01.419269+0000","last_fullsized":"2024-03-06T07:21:01.419269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:21.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431733+0000","last_change":"2024-03-06T07:20:50.888715+0000","last_active":"2024-03-06T07:21:01.431733+0000","last_peered":"2024-03-06T07:21:01.431733+0000","last_clean":"2024-03-06T07:21:01.431733+0000","last_became_active":"2024-03-06T07:20:50.888140+0000","last_became_peered":"2024-03-06T07:20:50.888140+0000","last_unstale":"2024-03-06T07:21:01.431733+0000","last_undegraded":"2024-03-06T07:21:01.431733+0000","last_fullsized":"2024-03-06T07:21:01.431733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:34.087293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424260+0000","last_change":"2024-03-06T07:21:01.424260+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424260+0000","last_undegraded":"2024-03-06T07:21:01.424260+0000","last_fullsized":"2024-03-06T07:21:01.424260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438971+0000","last_change":"2024-03-06T07:21:01.435195+0000","last_active":"2024-03-06T07:21:01.438971+0000","last_peered":"2024-03-06T07:21:01.438971+0000","last_clean":"2024-03-06T07:21:01.438971+0000","last_became_active":"2024-03-06T07:21:01.434964+0000","last_became_peered":"2024-03-06T07:21:01.434964+0000","last_unstale":"2024-03-06T07:21:01.438971+0000","last_undegraded":"2024-03-06T07:21:01.438971+0000","last_fullsized":"2024-03-06T07:21:01.438971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:25.506818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426735+0000","last_change":"2024-03-06T07:20:51.886764+0000","last_active":"2024-03-06T07:21:01.426735+0000","last_peered":"2024-03-06T07:21:01.426735+0000","last_clean":"2024-03-06T07:21:01.426735+0000","last_became_active":"2024-03-06T07:20:51.886187+0000","last_became_peered":"2024-03-06T07:20:51.886187+0000","last_unstale":"2024-03-06T07:21:01.426735+0000","last_undegraded":"2024-03-06T07:21:01.426735+0000","last_fullsized":"2024-03-06T07:21:01.426735+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:35.564191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432695+0000","last_change":"2024-03-06T07:20:50.885147+0000","last_active":"2024-03-06T07:21:01.432695+0000","last_peered":"2024-03-06T07:21:01.432695+0000","last_clean":"2024-03-06T07:21:01.432695+0000","last_became_active":"2024-03-06T07:20:50.884811+0000","last_became_peered":"2024-03-06T07:20:50.884811+0000","last_unstale":"2024-03-06T07:21:01.432695+0000","last_undegraded":"2024-03-06T07:21:01.432695+0000","last_fullsized":"2024-03-06T07:21:01.432695+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:24.667879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422765+0000","last_change":"2024-03-06T07:21:01.422765+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422765+0000","last_undegraded":"2024-03-06T07:21:01.422765+0000","last_fullsized":"2024-03-06T07:21:01.422765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442369+0000","last_change":"2024-03-06T07:21:01.439913+0000","last_active":"2024-03-06T07:21:01.442369+0000","last_peered":"2024-03-06T07:21:01.442369+0000","last_clean":"2024-03-06T07:21:01.442369+0000","last_became_active":"2024-03-06T07:21:01.438882+0000","last_became_peered":"2024-03-06T07:21:01.438882+0000","last_unstale":"2024-03-06T07:21:01.442369+0000","last_undegraded":"2024-03-06T07:21:01.442369+0000","last_fullsized":"2024-03-06T07:21:01.442369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:45.177307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427034+0000","last_change":"2024-03-06T07:20:51.887534+0000","last_active":"2024-03-06T07:21:01.427034+0000","last_peered":"2024-03-06T07:21:01.427034+0000","last_clean":"2024-03-06T07:21:01.427034+0000","last_became_active":"2024-03-06T07:20:51.887239+0000","last_became_peered":"2024-03-06T07:20:51.887239+0000","last_unstale":"2024-03-06T07:21:01.427034+0000","last_undegraded":"2024-03-06T07:21:01.427034+0000","last_fullsized":"2024-03-06T07:21:01.427034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425754+0000","last_change":"2024-03-06T07:20:50.892916+0000","last_active":"2024-03-06T07:21:01.425754+0000","last_peered":"2024-03-06T07:21:01.425754+0000","last_clean":"2024-03-06T07:21:01.425754+0000","last_became_active":"2024-03-06T07:20:50.892629+0000","last_became_peered":"2024-03-06T07:20:50.892629+0000","last_unstale":"2024-03-06T07:21:01.425754+0000","last_undegraded":"2024-03-06T07:21:01.425754+0000","last_fullsized":"2024-03-06T07:21:01.425754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:12.808948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429826+0000","last_change":"2024-03-06T07:21:01.429826+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429826+0000","last_undegraded":"2024-03-06T07:21:01.429826+0000","last_fullsized":"2024-03-06T07:21:01.429826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.439996+0000","last_change":"2024-03-06T07:21:01.439106+0000","last_active":"2024-03-06T07:21:01.439996+0000","last_peered":"2024-03-06T07:21:01.439996+0000","last_clean":"2024-03-06T07:21:01.439996+0000","last_became_active":"2024-03-06T07:21:01.438835+0000","last_became_peered":"2024-03-06T07:21:01.438835+0000","last_unstale":"2024-03-06T07:21:01.439996+0000","last_undegraded":"2024-03-06T07:21:01.439996+0000","last_fullsized":"2024-03-06T07:21:01.439996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419852+0000","last_change":"2024-03-06T07:20:51.895988+0000","last_active":"2024-03-06T07:21:01.419852+0000","last_peered":"2024-03-06T07:21:01.419852+0000","last_clean":"2024-03-06T07:21:01.419852+0000","last_became_active":"2024-03-06T07:20:51.895747+0000","last_became_peered":"2024-03-06T07:20:51.895747+0000","last_unstale":"2024-03-06T07:21:01.419852+0000","last_undegraded":"2024-03-06T07:21:01.419852+0000","last_fullsized":"2024-03-06T07:21:01.419852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427081+0000","last_change":"2024-03-06T07:20:50.893482+0000","last_active":"2024-03-06T07:21:01.427081+0000","last_peered":"2024-03-06T07:21:01.427081+0000","last_clean":"2024-03-06T07:21:01.427081+0000","last_became_active":"2024-03-06T07:20:50.893153+0000","last_became_peered":"2024-03-06T07:20:50.893153+0000","last_unstale":"2024-03-06T07:21:01.427081+0000","last_undegraded":"2024-03-06T07:21:01.427081+0000","last_fullsized":"2024-03-06T07:21:01.427081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:07.775768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424875+0000","last_change":"2024-03-06T07:21:01.424875+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424875+0000","last_undegraded":"2024-03-06T07:21:01.424875+0000","last_fullsized":"2024-03-06T07:21:01.424875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446459+0000","last_change":"2024-03-06T07:21:01.442216+0000","last_active":"2024-03-06T07:21:01.446459+0000","last_peered":"2024-03-06T07:21:01.446459+0000","last_clean":"2024-03-06T07:21:01.446459+0000","last_became_active":"2024-03-06T07:21:01.441772+0000","last_became_peered":"2024-03-06T07:21:01.441772+0000","last_unstale":"2024-03-06T07:21:01.446459+0000","last_undegraded":"2024-03-06T07:21:01.446459+0000","last_fullsized":"2024-03-06T07:21:01.446459+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:44.646869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.420032+0000","last_change":"2024-03-06T07:20:51.897529+0000","last_active":"2024-03-06T07:21:01.420032+0000","last_peered":"2024-03-06T07:21:01.420032+0000","last_clean":"2024-03-06T07:21:01.420032+0000","last_became_active":"2024-03-06T07:20:51.897181+0000","last_became_peered":"2024-03-06T07:20:51.897181+0000","last_unstale":"2024-03-06T07:21:01.420032+0000","last_undegraded":"2024-03-06T07:21:01.420032+0000","last_fullsized":"2024-03-06T07:21:01.420032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431244+0000","last_change":"2024-03-06T07:20:50.886303+0000","last_active":"2024-03-06T07:21:01.431244+0000","last_peered":"2024-03-06T07:21:01.431244+0000","last_clean":"2024-03-06T07:21:01.431244+0000","last_became_active":"2024-03-06T07:20:50.886013+0000","last_became_peered":"2024-03-06T07:20:50.886013+0000","last_unstale":"2024-03-06T07:21:01.431244+0000","last_undegraded":"2024-03-06T07:21:01.431244+0000","last_fullsized":"2024-03-06T07:21:01.431244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:34.617731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424723+0000","last_change":"2024-03-06T07:21:01.424723+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424723+0000","last_undegraded":"2024-03-06T07:21:01.424723+0000","last_fullsized":"2024-03-06T07:21:01.424723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438261+0000","last_change":"2024-03-06T07:21:01.434722+0000","last_active":"2024-03-06T07:21:01.438261+0000","last_peered":"2024-03-06T07:21:01.438261+0000","last_clean":"2024-03-06T07:21:01.438261+0000","last_became_active":"2024-03-06T07:21:01.434380+0000","last_became_peered":"2024-03-06T07:21:01.434380+0000","last_unstale":"2024-03-06T07:21:01.438261+0000","last_undegraded":"2024-03-06T07:21:01.438261+0000","last_fullsized":"2024-03-06T07:21:01.438261+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427622+0000","last_change":"2024-03-06T07:20:51.883834+0000","last_active":"2024-03-06T07:21:01.427622+0000","last_peered":"2024-03-06T07:21:01.427622+0000","last_clean":"2024-03-06T07:21:01.427622+0000","last_became_active":"2024-03-06T07:20:51.883553+0000","last_became_peered":"2024-03-06T07:20:51.883553+0000","last_unstale":"2024-03-06T07:21:01.427622+0000","last_undegraded":"2024-03-06T07:21:01.427622+0000","last_fullsized":"2024-03-06T07:21:01.427622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419381+0000","last_change":"2024-03-06T07:20:50.881449+0000","last_active":"2024-03-06T07:21:01.419381+0000","last_peered":"2024-03-06T07:21:01.419381+0000","last_clean":"2024-03-06T07:21:01.419381+0000","last_became_active":"2024-03-06T07:20:50.881172+0000","last_became_peered":"2024-03-06T07:20:50.881172+0000","last_unstale":"2024-03-06T07:21:01.419381+0000","last_undegraded":"2024-03-06T07:21:01.419381+0000","last_fullsized":"2024-03-06T07:21:01.419381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.317522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424583+0000","last_change":"2024-03-06T07:21:01.424583+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424583+0000","last_undegraded":"2024-03-06T07:21:01.424583+0000","last_fullsized":"2024-03-06T07:21:01.424583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.437982+0000","last_change":"2024-03-06T07:21:01.431035+0000","last_active":"2024-03-06T07:21:01.437982+0000","last_peered":"2024-03-06T07:21:01.437982+0000","last_clean":"2024-03-06T07:21:01.437982+0000","last_became_active":"2024-03-06T07:21:01.430787+0000","last_became_peered":"2024-03-06T07:21:01.430787+0000","last_unstale":"2024-03-06T07:21:01.437982+0000","last_undegraded":"2024-03-06T07:21:01.437982+0000","last_fullsized":"2024-03-06T07:21:01.437982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:08.242986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426138+0000","last_change":"2024-03-06T07:20:51.893411+0000","last_active":"2024-03-06T07:21:01.426138+0000","last_peered":"2024-03-06T07:21:01.426138+0000","last_clean":"2024-03-06T07:21:01.426138+0000","last_became_active":"2024-03-06T07:20:51.893149+0000","last_became_peered":"2024-03-06T07:20:51.893149+0000","last_unstale":"2024-03-06T07:21:01.426138+0000","last_undegraded":"2024-03-06T07:21:01.426138+0000","last_fullsized":"2024-03-06T07:21:01.426138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427928+0000","last_change":"2024-03-06T07:20:50.890159+0000","last_active":"2024-03-06T07:21:01.427928+0000","last_peered":"2024-03-06T07:21:01.427928+0000","last_clean":"2024-03-06T07:21:01.427928+0000","last_became_active":"2024-03-06T07:20:50.889863+0000","last_became_peered":"2024-03-06T07:20:50.889863+0000","last_unstale":"2024-03-06T07:21:01.427928+0000","last_undegraded":"2024-03-06T07:21:01.427928+0000","last_fullsized":"2024-03-06T07:21:01.427928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.303590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425274+0000","last_change":"2024-03-06T07:21:01.425274+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425274+0000","last_undegraded":"2024-03-06T07:21:01.425274+0000","last_fullsized":"2024-03-06T07:21:01.425274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443004+0000","last_change":"2024-03-06T07:21:01.440880+0000","last_active":"2024-03-06T07:21:01.443004+0000","last_peered":"2024-03-06T07:21:01.443004+0000","last_clean":"2024-03-06T07:21:01.443004+0000","last_became_active":"2024-03-06T07:21:01.440583+0000","last_became_peered":"2024-03-06T07:21:01.440583+0000","last_unstale":"2024-03-06T07:21:01.443004+0000","last_undegraded":"2024-03-06T07:21:01.443004+0000","last_fullsized":"2024-03-06T07:21:01.443004+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419246+0000","last_change":"2024-03-06T07:20:51.899199+0000","last_active":"2024-03-06T07:21:01.419246+0000","last_peered":"2024-03-06T07:21:01.419246+0000","last_clean":"2024-03-06T07:21:01.419246+0000","last_became_active":"2024-03-06T07:20:51.898622+0000","last_became_peered":"2024-03-06T07:20:51.898622+0000","last_unstale":"2024-03-06T07:21:01.419246+0000","last_undegraded":"2024-03-06T07:21:01.419246+0000","last_fullsized":"2024-03-06T07:21:01.419246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431988+0000","last_change":"2024-03-06T07:20:50.891619+0000","last_active":"2024-03-06T07:21:01.431988+0000","last_peered":"2024-03-06T07:21:01.431988+0000","last_clean":"2024-03-06T07:21:01.431988+0000","last_became_active":"2024-03-06T07:20:50.891305+0000","last_became_peered":"2024-03-06T07:20:50.891305+0000","last_unstale":"2024-03-06T07:21:01.431988+0000","last_undegraded":"2024-03-06T07:21:01.431988+0000","last_fullsized":"2024-03-06T07:21:01.431988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.732044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427634+0000","last_change":"2024-03-06T07:21:01.427634+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427634+0000","last_undegraded":"2024-03-06T07:21:01.427634+0000","last_fullsized":"2024-03-06T07:21:01.427634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442394+0000","last_change":"2024-03-06T07:21:01.439974+0000","last_active":"2024-03-06T07:21:01.442394+0000","last_peered":"2024-03-06T07:21:01.442394+0000","last_clean":"2024-03-06T07:21:01.442394+0000","last_became_active":"2024-03-06T07:21:01.439682+0000","last_became_peered":"2024-03-06T07:21:01.439682+0000","last_unstale":"2024-03-06T07:21:01.442394+0000","last_undegraded":"2024-03-06T07:21:01.442394+0000","last_fullsized":"2024-03-06T07:21:01.442394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431422+0000","last_change":"2024-03-06T07:20:51.887972+0000","last_active":"2024-03-06T07:21:01.431422+0000","last_peered":"2024-03-06T07:21:01.431422+0000","last_clean":"2024-03-06T07:21:01.431422+0000","last_became_active":"2024-03-06T07:20:51.887725+0000","last_became_peered":"2024-03-06T07:20:51.887725+0000","last_unstale":"2024-03-06T07:21:01.431422+0000","last_undegraded":"2024-03-06T07:21:01.431422+0000","last_fullsized":"2024-03-06T07:21:01.431422+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431885+0000","last_change":"2024-03-06T07:20:50.890731+0000","last_active":"2024-03-06T07:21:01.431885+0000","last_peered":"2024-03-06T07:21:01.431885+0000","last_clean":"2024-03-06T07:21:01.431885+0000","last_became_active":"2024-03-06T07:20:50.890429+0000","last_became_peered":"2024-03-06T07:20:50.890429+0000","last_unstale":"2024-03-06T07:21:01.431885+0000","last_undegraded":"2024-03-06T07:21:01.431885+0000","last_fullsized":"2024-03-06T07:21:01.431885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423808+0000","last_change":"2024-03-06T07:21:01.423808+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423808+0000","last_undegraded":"2024-03-06T07:21:01.423808+0000","last_fullsized":"2024-03-06T07:21:01.423808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438843+0000","last_change":"2024-03-06T07:21:01.436794+0000","last_active":"2024-03-06T07:21:01.438843+0000","last_peered":"2024-03-06T07:21:01.438843+0000","last_clean":"2024-03-06T07:21:01.438843+0000","last_became_active":"2024-03-06T07:21:01.436556+0000","last_became_peered":"2024-03-06T07:21:01.436556+0000","last_unstale":"2024-03-06T07:21:01.438843+0000","last_undegraded":"2024-03-06T07:21:01.438843+0000","last_fullsized":"2024-03-06T07:21:01.438843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425550+0000","last_change":"2024-03-06T07:20:51.885412+0000","last_active":"2024-03-06T07:21:01.425550+0000","last_peered":"2024-03-06T07:21:01.425550+0000","last_clean":"2024-03-06T07:21:01.425550+0000","last_became_active":"2024-03-06T07:20:51.885153+0000","last_became_peered":"2024-03-06T07:20:51.885153+0000","last_unstale":"2024-03-06T07:21:01.425550+0000","last_undegraded":"2024-03-06T07:21:01.425550+0000","last_fullsized":"2024-03-06T07:21:01.425550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:07.592958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427582+0000","last_change":"2024-03-06T07:20:50.880143+0000","last_active":"2024-03-06T07:21:01.427582+0000","last_peered":"2024-03-06T07:21:01.427582+0000","last_clean":"2024-03-06T07:21:01.427582+0000","last_became_active":"2024-03-06T07:20:50.879859+0000","last_became_peered":"2024-03-06T07:20:50.879859+0000","last_unstale":"2024-03-06T07:21:01.427582+0000","last_undegraded":"2024-03-06T07:21:01.427582+0000","last_fullsized":"2024-03-06T07:21:01.427582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:08.684501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423462+0000","last_change":"2024-03-06T07:21:01.423462+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423462+0000","last_undegraded":"2024-03-06T07:21:01.423462+0000","last_fullsized":"2024-03-06T07:21:01.423462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433111+0000","last_change":"2024-03-06T07:21:01.432460+0000","last_active":"2024-03-06T07:21:01.433111+0000","last_peered":"2024-03-06T07:21:01.433111+0000","last_clean":"2024-03-06T07:21:01.433111+0000","last_became_active":"2024-03-06T07:21:01.432169+0000","last_became_peered":"2024-03-06T07:21:01.432169+0000","last_unstale":"2024-03-06T07:21:01.433111+0000","last_undegraded":"2024-03-06T07:21:01.433111+0000","last_fullsized":"2024-03-06T07:21:01.433111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427116+0000","last_change":"2024-03-06T07:20:51.878936+0000","last_active":"2024-03-06T07:21:01.427116+0000","last_peered":"2024-03-06T07:21:01.427116+0000","last_clean":"2024-03-06T07:21:01.427116+0000","last_became_active":"2024-03-06T07:20:51.878613+0000","last_became_peered":"2024-03-06T07:20:51.878613+0000","last_unstale":"2024-03-06T07:21:01.427116+0000","last_undegraded":"2024-03-06T07:21:01.427116+0000","last_fullsized":"2024-03-06T07:21:01.427116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":24,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425845+0000","last_change":"2024-03-06T07:20:50.888110+0000","last_active":"2024-03-06T07:21:01.425845+0000","last_peered":"2024-03-06T07:21:01.425845+0000","last_clean":"2024-03-06T07:21:01.425845+0000","last_became_active":"2024-03-06T07:20:50.887804+0000","last_became_peered":"2024-03-06T07:20:50.887804+0000","last_unstale":"2024-03-06T07:21:01.425845+0000","last_undegraded":"2024-03-06T07:21:01.425845+0000","last_fullsized":"2024-03-06T07:21:01.425845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423329+0000","last_change":"2024-03-06T07:21:01.423329+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423329+0000","last_undegraded":"2024-03-06T07:21:01.423329+0000","last_fullsized":"2024-03-06T07:21:01.423329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438955+0000","last_change":"2024-03-06T07:21:01.433077+0000","last_active":"2024-03-06T07:21:01.438955+0000","last_peered":"2024-03-06T07:21:01.438955+0000","last_clean":"2024-03-06T07:21:01.438955+0000","last_became_active":"2024-03-06T07:21:01.432684+0000","last_became_peered":"2024-03-06T07:21:01.432684+0000","last_unstale":"2024-03-06T07:21:01.438955+0000","last_undegraded":"2024-03-06T07:21:01.438955+0000","last_fullsized":"2024-03-06T07:21:01.438955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:13.212500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431469+0000","last_change":"2024-03-06T07:20:51.891149+0000","last_active":"2024-03-06T07:21:01.431469+0000","last_peered":"2024-03-06T07:21:01.431469+0000","last_clean":"2024-03-06T07:21:01.431469+0000","last_became_active":"2024-03-06T07:20:51.890877+0000","last_became_peered":"2024-03-06T07:20:51.890877+0000","last_unstale":"2024-03-06T07:21:01.431469+0000","last_undegraded":"2024-03-06T07:21:01.431469+0000","last_fullsized":"2024-03-06T07:21:01.431469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431369+0000","last_change":"2024-03-06T07:20:50.888227+0000","last_active":"2024-03-06T07:21:01.431369+0000","last_peered":"2024-03-06T07:21:01.431369+0000","last_clean":"2024-03-06T07:21:01.431369+0000","last_became_active":"2024-03-06T07:20:50.887926+0000","last_became_peered":"2024-03-06T07:20:50.887926+0000","last_unstale":"2024-03-06T07:21:01.431369+0000","last_undegraded":"2024-03-06T07:21:01.431369+0000","last_fullsized":"2024-03-06T07:21:01.431369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:56.255517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425722+0000","last_change":"2024-03-06T07:21:01.425722+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425722+0000","last_undegraded":"2024-03-06T07:21:01.425722+0000","last_fullsized":"2024-03-06T07:21:01.425722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433380+0000","last_change":"2024-03-06T07:21:01.432947+0000","last_active":"2024-03-06T07:21:01.433380+0000","last_peered":"2024-03-06T07:21:01.433380+0000","last_clean":"2024-03-06T07:21:01.433380+0000","last_became_active":"2024-03-06T07:21:01.432263+0000","last_became_peered":"2024-03-06T07:21:01.432263+0000","last_unstale":"2024-03-06T07:21:01.433380+0000","last_undegraded":"2024-03-06T07:21:01.433380+0000","last_fullsized":"2024-03-06T07:21:01.433380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425616+0000","last_change":"2024-03-06T07:20:51.888556+0000","last_active":"2024-03-06T07:21:01.425616+0000","last_peered":"2024-03-06T07:21:01.425616+0000","last_clean":"2024-03-06T07:21:01.425616+0000","last_became_active":"2024-03-06T07:20:51.888278+0000","last_became_peered":"2024-03-06T07:20:51.888278+0000","last_unstale":"2024-03-06T07:21:01.425616+0000","last_undegraded":"2024-03-06T07:21:01.425616+0000","last_fullsized":"2024-03-06T07:21:01.425616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431076+0000","last_change":"2024-03-06T07:20:50.881110+0000","last_active":"2024-03-06T07:21:01.431076+0000","last_peered":"2024-03-06T07:21:01.431076+0000","last_clean":"2024-03-06T07:21:01.431076+0000","last_became_active":"2024-03-06T07:20:50.880752+0000","last_became_peered":"2024-03-06T07:20:50.880752+0000","last_unstale":"2024-03-06T07:21:01.431076+0000","last_undegraded":"2024-03-06T07:21:01.431076+0000","last_fullsized":"2024-03-06T07:21:01.431076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423587+0000","last_change":"2024-03-06T07:21:01.423587+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423587+0000","last_undegraded":"2024-03-06T07:21:01.423587+0000","last_fullsized":"2024-03-06T07:21:01.423587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438705+0000","last_change":"2024-03-06T07:21:01.435628+0000","last_active":"2024-03-06T07:21:01.438705+0000","last_peered":"2024-03-06T07:21:01.438705+0000","last_clean":"2024-03-06T07:21:01.438705+0000","last_became_active":"2024-03-06T07:21:01.434996+0000","last_became_peered":"2024-03-06T07:21:01.434996+0000","last_unstale":"2024-03-06T07:21:01.438705+0000","last_undegraded":"2024-03-06T07:21:01.438705+0000","last_fullsized":"2024-03-06T07:21:01.438705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:21.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425994+0000","last_change":"2024-03-06T07:20:51.879914+0000","last_active":"2024-03-06T07:21:01.425994+0000","last_peered":"2024-03-06T07:21:01.425994+0000","last_clean":"2024-03-06T07:21:01.425994+0000","last_became_active":"2024-03-06T07:20:51.879633+0000","last_became_peered":"2024-03-06T07:20:51.879633+0000","last_unstale":"2024-03-06T07:21:01.425994+0000","last_undegraded":"2024-03-06T07:21:01.425994+0000","last_fullsized":"2024-03-06T07:21:01.425994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:36.738962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424764+0000","last_change":"2024-03-06T07:20:50.880559+0000","last_active":"2024-03-06T07:21:01.424764+0000","last_peered":"2024-03-06T07:21:01.424764+0000","last_clean":"2024-03-06T07:21:01.424764+0000","last_became_active":"2024-03-06T07:20:50.880160+0000","last_became_peered":"2024-03-06T07:20:50.880160+0000","last_unstale":"2024-03-06T07:21:01.424764+0000","last_undegraded":"2024-03-06T07:21:01.424764+0000","last_fullsized":"2024-03-06T07:21:01.424764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:56.900930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430090+0000","last_change":"2024-03-06T07:21:01.430090+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430090+0000","last_undegraded":"2024-03-06T07:21:01.430090+0000","last_fullsized":"2024-03-06T07:21:01.430090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442699+0000","last_change":"2024-03-06T07:21:01.441770+0000","last_active":"2024-03-06T07:21:01.442699+0000","last_peered":"2024-03-06T07:21:01.442699+0000","last_clean":"2024-03-06T07:21:01.442699+0000","last_became_active":"2024-03-06T07:21:01.441478+0000","last_became_peered":"2024-03-06T07:21:01.441478+0000","last_unstale":"2024-03-06T07:21:01.442699+0000","last_undegraded":"2024-03-06T07:21:01.442699+0000","last_fullsized":"2024-03-06T07:21:01.442699+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:10.768407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426237+0000","last_change":"2024-03-06T07:20:51.890440+0000","last_active":"2024-03-06T07:21:01.426237+0000","last_peered":"2024-03-06T07:21:01.426237+0000","last_clean":"2024-03-06T07:21:01.426237+0000","last_became_active":"2024-03-06T07:20:51.890167+0000","last_became_peered":"2024-03-06T07:20:51.890167+0000","last_unstale":"2024-03-06T07:21:01.426237+0000","last_undegraded":"2024-03-06T07:21:01.426237+0000","last_fullsized":"2024-03-06T07:21:01.426237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:53.956654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":26,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419383+0000","last_change":"2024-03-06T07:20:50.885352+0000","last_active":"2024-03-06T07:21:01.419383+0000","last_peered":"2024-03-06T07:21:01.419383+0000","last_clean":"2024-03-06T07:21:01.419383+0000","last_became_active":"2024-03-06T07:20:50.884991+0000","last_became_peered":"2024-03-06T07:20:50.884991+0000","last_unstale":"2024-03-06T07:21:01.419383+0000","last_undegraded":"2024-03-06T07:21:01.419383+0000","last_fullsized":"2024-03-06T07:21:01.419383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:03.652296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427525+0000","last_change":"2024-03-06T07:21:01.427525+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427525+0000","last_undegraded":"2024-03-06T07:21:01.427525+0000","last_fullsized":"2024-03-06T07:21:01.427525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438733+0000","last_change":"2024-03-06T07:21:01.436096+0000","last_active":"2024-03-06T07:21:01.438733+0000","last_peered":"2024-03-06T07:21:01.438733+0000","last_clean":"2024-03-06T07:21:01.438733+0000","last_became_active":"2024-03-06T07:21:01.435767+0000","last_became_peered":"2024-03-06T07:21:01.435767+0000","last_unstale":"2024-03-06T07:21:01.438733+0000","last_undegraded":"2024-03-06T07:21:01.438733+0000","last_fullsized":"2024-03-06T07:21:01.438733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:40.907133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427258+0000","last_change":"2024-03-06T07:20:51.889501+0000","last_active":"2024-03-06T07:21:01.427258+0000","last_peered":"2024-03-06T07:21:01.427258+0000","last_clean":"2024-03-06T07:21:01.427258+0000","last_became_active":"2024-03-06T07:20:51.889206+0000","last_became_peered":"2024-03-06T07:20:51.889206+0000","last_unstale":"2024-03-06T07:21:01.427258+0000","last_undegraded":"2024-03-06T07:21:01.427258+0000","last_fullsized":"2024-03-06T07:21:01.427258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:09.908900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432085+0000","last_change":"2024-03-06T07:20:50.883039+0000","last_active":"2024-03-06T07:21:01.432085+0000","last_peered":"2024-03-06T07:21:01.432085+0000","last_clean":"2024-03-06T07:21:01.432085+0000","last_became_active":"2024-03-06T07:20:50.882748+0000","last_became_peered":"2024-03-06T07:20:50.882748+0000","last_unstale":"2024-03-06T07:21:01.432085+0000","last_undegraded":"2024-03-06T07:21:01.432085+0000","last_fullsized":"2024-03-06T07:21:01.432085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:35.868841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423288+0000","last_change":"2024-03-06T07:21:01.423288+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423288+0000","last_undegraded":"2024-03-06T07:21:01.423288+0000","last_fullsized":"2024-03-06T07:21:01.423288+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440065+0000","last_change":"2024-03-06T07:21:01.438861+0000","last_active":"2024-03-06T07:21:01.440065+0000","last_peered":"2024-03-06T07:21:01.440065+0000","last_clean":"2024-03-06T07:21:01.440065+0000","last_became_active":"2024-03-06T07:21:01.438577+0000","last_became_peered":"2024-03-06T07:21:01.438577+0000","last_unstale":"2024-03-06T07:21:01.440065+0000","last_undegraded":"2024-03-06T07:21:01.440065+0000","last_fullsized":"2024-03-06T07:21:01.440065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.335344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424200+0000","last_change":"2024-03-06T07:20:51.890860+0000","last_active":"2024-03-06T07:21:01.424200+0000","last_peered":"2024-03-06T07:21:01.424200+0000","last_clean":"2024-03-06T07:21:01.424200+0000","last_became_active":"2024-03-06T07:20:51.890576+0000","last_became_peered":"2024-03-06T07:20:51.890576+0000","last_unstale":"2024-03-06T07:21:01.424200+0000","last_undegraded":"2024-03-06T07:21:01.424200+0000","last_fullsized":"2024-03-06T07:21:01.424200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:02.364180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427717+0000","last_change":"2024-03-06T07:20:50.888701+0000","last_active":"2024-03-06T07:21:01.427717+0000","last_peered":"2024-03-06T07:21:01.427717+0000","last_clean":"2024-03-06T07:21:01.427717+0000","last_became_active":"2024-03-06T07:20:50.888392+0000","last_became_peered":"2024-03-06T07:20:50.888392+0000","last_unstale":"2024-03-06T07:21:01.427717+0000","last_undegraded":"2024-03-06T07:21:01.427717+0000","last_fullsized":"2024-03-06T07:21:01.427717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.582638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422966+0000","last_change":"2024-03-06T07:21:01.422966+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422966+0000","last_undegraded":"2024-03-06T07:21:01.422966+0000","last_fullsized":"2024-03-06T07:21:01.422966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438788+0000","last_change":"2024-03-06T07:21:01.436497+0000","last_active":"2024-03-06T07:21:01.438788+0000","last_peered":"2024-03-06T07:21:01.438788+0000","last_clean":"2024-03-06T07:21:01.438788+0000","last_became_active":"2024-03-06T07:21:01.436029+0000","last_became_peered":"2024-03-06T07:21:01.436029+0000","last_unstale":"2024-03-06T07:21:01.438788+0000","last_undegraded":"2024-03-06T07:21:01.438788+0000","last_fullsized":"2024-03-06T07:21:01.438788+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:36.166225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425488+0000","last_change":"2024-03-06T07:20:51.887239+0000","last_active":"2024-03-06T07:21:01.425488+0000","last_peered":"2024-03-06T07:21:01.425488+0000","last_clean":"2024-03-06T07:21:01.425488+0000","last_became_active":"2024-03-06T07:20:51.886940+0000","last_became_peered":"2024-03-06T07:20:51.886940+0000","last_unstale":"2024-03-06T07:21:01.425488+0000","last_undegraded":"2024-03-06T07:21:01.425488+0000","last_fullsized":"2024-03-06T07:21:01.425488+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:08.123116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427698+0000","last_change":"2024-03-06T07:20:50.886879+0000","last_active":"2024-03-06T07:21:01.427698+0000","last_peered":"2024-03-06T07:21:01.427698+0000","last_clean":"2024-03-06T07:21:01.427698+0000","last_became_active":"2024-03-06T07:20:50.886589+0000","last_became_peered":"2024-03-06T07:20:50.886589+0000","last_unstale":"2024-03-06T07:21:01.427698+0000","last_undegraded":"2024-03-06T07:21:01.427698+0000","last_fullsized":"2024-03-06T07:21:01.427698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:51.245367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428740+0000","last_change":"2024-03-06T07:21:01.428740+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428740+0000","last_undegraded":"2024-03-06T07:21:01.428740+0000","last_fullsized":"2024-03-06T07:21:01.428740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443067+0000","last_change":"2024-03-06T07:21:01.442218+0000","last_active":"2024-03-06T07:21:01.443067+0000","last_peered":"2024-03-06T07:21:01.443067+0000","last_clean":"2024-03-06T07:21:01.443067+0000","last_became_active":"2024-03-06T07:21:01.441776+0000","last_became_peered":"2024-03-06T07:21:01.441776+0000","last_unstale":"2024-03-06T07:21:01.443067+0000","last_undegraded":"2024-03-06T07:21:01.443067+0000","last_fullsized":"2024-03-06T07:21:01.443067+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:26.445135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424527+0000","last_change":"2024-03-06T07:20:51.892549+0000","last_active":"2024-03-06T07:21:01.424527+0000","last_peered":"2024-03-06T07:21:01.424527+0000","last_clean":"2024-03-06T07:21:01.424527+0000","last_became_active":"2024-03-06T07:20:51.889936+0000","last_became_peered":"2024-03-06T07:20:51.889936+0000","last_unstale":"2024-03-06T07:21:01.424527+0000","last_undegraded":"2024-03-06T07:21:01.424527+0000","last_fullsized":"2024-03-06T07:21:01.424527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:23.986390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430588+0000","last_change":"2024-03-06T07:20:50.883071+0000","last_active":"2024-03-06T07:21:01.430588+0000","last_peered":"2024-03-06T07:21:01.430588+0000","last_clean":"2024-03-06T07:21:01.430588+0000","last_became_active":"2024-03-06T07:20:50.882465+0000","last_became_peered":"2024-03-06T07:20:50.882465+0000","last_unstale":"2024-03-06T07:21:01.430588+0000","last_undegraded":"2024-03-06T07:21:01.430588+0000","last_fullsized":"2024-03-06T07:21:01.430588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423784+0000","last_change":"2024-03-06T07:21:01.423784+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423784+0000","last_undegraded":"2024-03-06T07:21:01.423784+0000","last_fullsized":"2024-03-06T07:21:01.423784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.444053+0000","last_change":"2024-03-06T07:21:01.441755+0000","last_active":"2024-03-06T07:21:01.444053+0000","last_peered":"2024-03-06T07:21:01.444053+0000","last_clean":"2024-03-06T07:21:01.444053+0000","last_became_active":"2024-03-06T07:21:01.441485+0000","last_became_peered":"2024-03-06T07:21:01.441485+0000","last_unstale":"2024-03-06T07:21:01.444053+0000","last_undegraded":"2024-03-06T07:21:01.444053+0000","last_fullsized":"2024-03-06T07:21:01.444053+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:14.211872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419859+0000","last_change":"2024-03-06T07:20:51.884694+0000","last_active":"2024-03-06T07:21:01.419859+0000","last_peered":"2024-03-06T07:21:01.419859+0000","last_clean":"2024-03-06T07:21:01.419859+0000","last_became_active":"2024-03-06T07:20:51.883905+0000","last_became_peered":"2024-03-06T07:20:51.883905+0000","last_unstale":"2024-03-06T07:21:01.419859+0000","last_undegraded":"2024-03-06T07:21:01.419859+0000","last_fullsized":"2024-03-06T07:21:01.419859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424620+0000","last_change":"2024-03-06T07:20:50.881489+0000","last_active":"2024-03-06T07:21:01.424620+0000","last_peered":"2024-03-06T07:21:01.424620+0000","last_clean":"2024-03-06T07:21:01.424620+0000","last_became_active":"2024-03-06T07:20:50.881182+0000","last_became_peered":"2024-03-06T07:20:50.881182+0000","last_unstale":"2024-03-06T07:21:01.424620+0000","last_undegraded":"2024-03-06T07:21:01.424620+0000","last_fullsized":"2024-03-06T07:21:01.424620+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:22.518381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422551+0000","last_change":"2024-03-06T07:21:01.422551+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422551+0000","last_undegraded":"2024-03-06T07:21:01.422551+0000","last_fullsized":"2024-03-06T07:21:01.422551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438240+0000","last_change":"2024-03-06T07:21:01.434477+0000","last_active":"2024-03-06T07:21:01.438240+0000","last_peered":"2024-03-06T07:21:01.438240+0000","last_clean":"2024-03-06T07:21:01.438240+0000","last_became_active":"2024-03-06T07:21:01.434205+0000","last_became_peered":"2024-03-06T07:21:01.434205+0000","last_unstale":"2024-03-06T07:21:01.438240+0000","last_undegraded":"2024-03-06T07:21:01.438240+0000","last_fullsized":"2024-03-06T07:21:01.438240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419708+0000","last_change":"2024-03-06T07:20:51.889352+0000","last_active":"2024-03-06T07:21:01.419708+0000","last_peered":"2024-03-06T07:21:01.419708+0000","last_clean":"2024-03-06T07:21:01.419708+0000","last_became_active":"2024-03-06T07:20:51.888464+0000","last_became_peered":"2024-03-06T07:20:51.888464+0000","last_unstale":"2024-03-06T07:21:01.419708+0000","last_undegraded":"2024-03-06T07:21:01.419708+0000","last_fullsized":"2024-03-06T07:21:01.419708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425015+0000","last_change":"2024-03-06T07:20:50.887415+0000","last_active":"2024-03-06T07:21:01.425015+0000","last_peered":"2024-03-06T07:21:01.425015+0000","last_clean":"2024-03-06T07:21:01.425015+0000","last_became_active":"2024-03-06T07:20:50.887137+0000","last_became_peered":"2024-03-06T07:20:50.887137+0000","last_unstale":"2024-03-06T07:21:01.425015+0000","last_undegraded":"2024-03-06T07:21:01.425015+0000","last_fullsized":"2024-03-06T07:21:01.425015+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:54.210267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422909+0000","last_change":"2024-03-06T07:21:01.422909+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422909+0000","last_undegraded":"2024-03-06T07:21:01.422909+0000","last_fullsized":"2024-03-06T07:21:01.422909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441013+0000","last_change":"2024-03-06T07:21:01.439736+0000","last_active":"2024-03-06T07:21:01.441013+0000","last_peered":"2024-03-06T07:21:01.441013+0000","last_clean":"2024-03-06T07:21:01.441013+0000","last_became_active":"2024-03-06T07:21:01.439473+0000","last_became_peered":"2024-03-06T07:21:01.439473+0000","last_unstale":"2024-03-06T07:21:01.441013+0000","last_undegraded":"2024-03-06T07:21:01.441013+0000","last_fullsized":"2024-03-06T07:21:01.441013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:38.195559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426346+0000","last_change":"2024-03-06T07:20:51.885672+0000","last_active":"2024-03-06T07:21:01.426346+0000","last_peered":"2024-03-06T07:21:01.426346+0000","last_clean":"2024-03-06T07:21:01.426346+0000","last_became_active":"2024-03-06T07:20:51.885412+0000","last_became_peered":"2024-03-06T07:20:51.885412+0000","last_unstale":"2024-03-06T07:21:01.426346+0000","last_undegraded":"2024-03-06T07:21:01.426346+0000","last_fullsized":"2024-03-06T07:21:01.426346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:54.467344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427413+0000","last_change":"2024-03-06T07:20:50.889223+0000","last_active":"2024-03-06T07:21:01.427413+0000","last_peered":"2024-03-06T07:21:01.427413+0000","last_clean":"2024-03-06T07:21:01.427413+0000","last_became_active":"2024-03-06T07:20:50.888934+0000","last_became_peered":"2024-03-06T07:20:50.888934+0000","last_unstale":"2024-03-06T07:21:01.427413+0000","last_undegraded":"2024-03-06T07:21:01.427413+0000","last_fullsized":"2024-03-06T07:21:01.427413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:04.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423927+0000","last_change":"2024-03-06T07:21:01.423927+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423927+0000","last_undegraded":"2024-03-06T07:21:01.423927+0000","last_fullsized":"2024-03-06T07:21:01.423927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438700+0000","last_change":"2024-03-06T07:21:01.435463+0000","last_active":"2024-03-06T07:21:01.438700+0000","last_peered":"2024-03-06T07:21:01.438700+0000","last_clean":"2024-03-06T07:21:01.438700+0000","last_became_active":"2024-03-06T07:21:01.435086+0000","last_became_peered":"2024-03-06T07:21:01.435086+0000","last_unstale":"2024-03-06T07:21:01.438700+0000","last_undegraded":"2024-03-06T07:21:01.438700+0000","last_fullsized":"2024-03-06T07:21:01.438700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:43.973004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426480+0000","last_change":"2024-03-06T07:20:51.880937+0000","last_active":"2024-03-06T07:21:01.426480+0000","last_peered":"2024-03-06T07:21:01.426480+0000","last_clean":"2024-03-06T07:21:01.426480+0000","last_became_active":"2024-03-06T07:20:51.880648+0000","last_became_peered":"2024-03-06T07:20:51.880648+0000","last_unstale":"2024-03-06T07:21:01.426480+0000","last_undegraded":"2024-03-06T07:21:01.426480+0000","last_fullsized":"2024-03-06T07:21:01.426480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432004+0000","last_change":"2024-03-06T07:20:50.886345+0000","last_active":"2024-03-06T07:21:01.432004+0000","last_peered":"2024-03-06T07:21:01.432004+0000","last_clean":"2024-03-06T07:21:01.432004+0000","last_became_active":"2024-03-06T07:20:50.886074+0000","last_became_peered":"2024-03-06T07:20:50.886074+0000","last_unstale":"2024-03-06T07:21:01.432004+0000","last_undegraded":"2024-03-06T07:21:01.432004+0000","last_fullsized":"2024-03-06T07:21:01.432004+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:39.914514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422540+0000","last_change":"2024-03-06T07:21:01.422540+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422540+0000","last_undegraded":"2024-03-06T07:21:01.422540+0000","last_fullsized":"2024-03-06T07:21:01.422540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446264+0000","last_change":"2024-03-06T07:21:01.443325+0000","last_active":"2024-03-06T07:21:01.446264+0000","last_peered":"2024-03-06T07:21:01.446264+0000","last_clean":"2024-03-06T07:21:01.446264+0000","last_became_active":"2024-03-06T07:21:01.443143+0000","last_became_peered":"2024-03-06T07:21:01.443143+0000","last_unstale":"2024-03-06T07:21:01.446264+0000","last_undegraded":"2024-03-06T07:21:01.446264+0000","last_fullsized":"2024-03-06T07:21:01.446264+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:17.147464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426249+0000","last_change":"2024-03-06T07:20:51.889471+0000","last_active":"2024-03-06T07:21:01.426249+0000","last_peered":"2024-03-06T07:21:01.426249+0000","last_clean":"2024-03-06T07:21:01.426249+0000","last_became_active":"2024-03-06T07:20:51.889218+0000","last_became_peered":"2024-03-06T07:20:51.889218+0000","last_unstale":"2024-03-06T07:21:01.426249+0000","last_undegraded":"2024-03-06T07:21:01.426249+0000","last_fullsized":"2024-03-06T07:21:01.426249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419799+0000","last_change":"2024-03-06T07:20:50.881740+0000","last_active":"2024-03-06T07:21:01.419799+0000","last_peered":"2024-03-06T07:21:01.419799+0000","last_clean":"2024-03-06T07:21:01.419799+0000","last_became_active":"2024-03-06T07:20:50.881306+0000","last_became_peered":"2024-03-06T07:20:50.881306+0000","last_unstale":"2024-03-06T07:21:01.419799+0000","last_undegraded":"2024-03-06T07:21:01.419799+0000","last_fullsized":"2024-03-06T07:21:01.419799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:35.733892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427082+0000","last_change":"2024-03-06T07:21:01.427082+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427082+0000","last_undegraded":"2024-03-06T07:21:01.427082+0000","last_fullsized":"2024-03-06T07:21:01.427082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.436013+0000","last_change":"2024-03-06T07:21:01.435258+0000","last_active":"2024-03-06T07:21:01.436013+0000","last_peered":"2024-03-06T07:21:01.436013+0000","last_clean":"2024-03-06T07:21:01.436013+0000","last_became_active":"2024-03-06T07:21:01.435001+0000","last_became_peered":"2024-03-06T07:21:01.435001+0000","last_unstale":"2024-03-06T07:21:01.436013+0000","last_undegraded":"2024-03-06T07:21:01.436013+0000","last_fullsized":"2024-03-06T07:21:01.436013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426836+0000","last_change":"2024-03-06T07:20:51.886967+0000","last_active":"2024-03-06T07:21:01.426836+0000","last_peered":"2024-03-06T07:21:01.426836+0000","last_clean":"2024-03-06T07:21:01.426836+0000","last_became_active":"2024-03-06T07:20:51.886521+0000","last_became_peered":"2024-03-06T07:20:51.886521+0000","last_unstale":"2024-03-06T07:21:01.426836+0000","last_undegraded":"2024-03-06T07:21:01.426836+0000","last_fullsized":"2024-03-06T07:21:01.426836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:58.399266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432888+0000","last_change":"2024-03-06T07:20:50.885378+0000","last_active":"2024-03-06T07:21:01.432888+0000","last_peered":"2024-03-06T07:21:01.432888+0000","last_clean":"2024-03-06T07:21:01.432888+0000","last_became_active":"2024-03-06T07:20:50.884859+0000","last_became_peered":"2024-03-06T07:20:50.884859+0000","last_unstale":"2024-03-06T07:21:01.432888+0000","last_undegraded":"2024-03-06T07:21:01.432888+0000","last_fullsized":"2024-03-06T07:21:01.432888+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:28.686737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423705+0000","last_change":"2024-03-06T07:21:01.423705+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423705+0000","last_undegraded":"2024-03-06T07:21:01.423705+0000","last_fullsized":"2024-03-06T07:21:01.423705+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438756+0000","last_change":"2024-03-06T07:21:01.437079+0000","last_active":"2024-03-06T07:21:01.438756+0000","last_peered":"2024-03-06T07:21:01.438756+0000","last_clean":"2024-03-06T07:21:01.438756+0000","last_became_active":"2024-03-06T07:21:01.436848+0000","last_became_peered":"2024-03-06T07:21:01.436848+0000","last_unstale":"2024-03-06T07:21:01.438756+0000","last_undegraded":"2024-03-06T07:21:01.438756+0000","last_fullsized":"2024-03-06T07:21:01.438756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.233171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419658+0000","last_change":"2024-03-06T07:20:51.899213+0000","last_active":"2024-03-06T07:21:01.419658+0000","last_peered":"2024-03-06T07:21:01.419658+0000","last_clean":"2024-03-06T07:21:01.419658+0000","last_became_active":"2024-03-06T07:20:51.898882+0000","last_became_peered":"2024-03-06T07:20:51.898882+0000","last_unstale":"2024-03-06T07:21:01.419658+0000","last_undegraded":"2024-03-06T07:21:01.419658+0000","last_fullsized":"2024-03-06T07:21:01.419658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431260+0000","last_change":"2024-03-06T07:20:50.888878+0000","last_active":"2024-03-06T07:21:01.431260+0000","last_peered":"2024-03-06T07:21:01.431260+0000","last_clean":"2024-03-06T07:21:01.431260+0000","last_became_active":"2024-03-06T07:20:50.888604+0000","last_became_peered":"2024-03-06T07:20:50.888604+0000","last_unstale":"2024-03-06T07:21:01.431260+0000","last_undegraded":"2024-03-06T07:21:01.431260+0000","last_fullsized":"2024-03-06T07:21:01.431260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:33.413428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422649+0000","last_change":"2024-03-06T07:21:01.422649+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422649+0000","last_undegraded":"2024-03-06T07:21:01.422649+0000","last_fullsized":"2024-03-06T07:21:01.422649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438997+0000","last_change":"2024-03-06T07:21:01.433417+0000","last_active":"2024-03-06T07:21:01.438997+0000","last_peered":"2024-03-06T07:21:01.438997+0000","last_clean":"2024-03-06T07:21:01.438997+0000","last_became_active":"2024-03-06T07:21:01.432105+0000","last_became_peered":"2024-03-06T07:21:01.432105+0000","last_unstale":"2024-03-06T07:21:01.438997+0000","last_undegraded":"2024-03-06T07:21:01.438997+0000","last_fullsized":"2024-03-06T07:21:01.438997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:18.120482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425648+0000","last_change":"2024-03-06T07:20:51.893070+0000","last_active":"2024-03-06T07:21:01.425648+0000","last_peered":"2024-03-06T07:21:01.425648+0000","last_clean":"2024-03-06T07:21:01.425648+0000","last_became_active":"2024-03-06T07:20:51.892785+0000","last_became_peered":"2024-03-06T07:20:51.892785+0000","last_unstale":"2024-03-06T07:21:01.425648+0000","last_undegraded":"2024-03-06T07:21:01.425648+0000","last_fullsized":"2024-03-06T07:21:01.425648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:09.591477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425357+0000","last_change":"2024-03-06T07:20:50.892942+0000","last_active":"2024-03-06T07:21:01.425357+0000","last_peered":"2024-03-06T07:21:01.425357+0000","last_clean":"2024-03-06T07:21:01.425357+0000","last_became_active":"2024-03-06T07:20:50.892638+0000","last_became_peered":"2024-03-06T07:20:50.892638+0000","last_unstale":"2024-03-06T07:21:01.425357+0000","last_undegraded":"2024-03-06T07:21:01.425357+0000","last_fullsized":"2024-03-06T07:21:01.425357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:40.434821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427886+0000","last_change":"2024-03-06T07:21:01.427886+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427886+0000","last_undegraded":"2024-03-06T07:21:01.427886+0000","last_fullsized":"2024-03-06T07:21:01.427886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438316+0000","last_change":"2024-03-06T07:21:01.435337+0000","last_active":"2024-03-06T07:21:01.438316+0000","last_peered":"2024-03-06T07:21:01.438316+0000","last_clean":"2024-03-06T07:21:01.438316+0000","last_became_active":"2024-03-06T07:21:01.435098+0000","last_became_peered":"2024-03-06T07:21:01.435098+0000","last_unstale":"2024-03-06T07:21:01.438316+0000","last_undegraded":"2024-03-06T07:21:01.438316+0000","last_fullsized":"2024-03-06T07:21:01.438316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425885+0000","last_change":"2024-03-06T07:20:51.895826+0000","last_active":"2024-03-06T07:21:01.425885+0000","last_peered":"2024-03-06T07:21:01.425885+0000","last_clean":"2024-03-06T07:21:01.425885+0000","last_became_active":"2024-03-06T07:20:51.895483+0000","last_became_peered":"2024-03-06T07:20:51.895483+0000","last_unstale":"2024-03-06T07:21:01.425885+0000","last_undegraded":"2024-03-06T07:21:01.425885+0000","last_fullsized":"2024-03-06T07:21:01.425885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425724+0000","last_change":"2024-03-06T07:20:50.879754+0000","last_active":"2024-03-06T07:21:01.425724+0000","last_peered":"2024-03-06T07:21:01.425724+0000","last_clean":"2024-03-06T07:21:01.425724+0000","last_became_active":"2024-03-06T07:20:50.879488+0000","last_became_peered":"2024-03-06T07:20:50.879488+0000","last_unstale":"2024-03-06T07:21:01.425724+0000","last_undegraded":"2024-03-06T07:21:01.425724+0000","last_fullsized":"2024-03-06T07:21:01.425724+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.603045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422772+0000","last_change":"2024-03-06T07:21:01.422772+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422772+0000","last_undegraded":"2024-03-06T07:21:01.422772+0000","last_fullsized":"2024-03-06T07:21:01.422772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441007+0000","last_change":"2024-03-06T07:21:01.439920+0000","last_active":"2024-03-06T07:21:01.441007+0000","last_peered":"2024-03-06T07:21:01.441007+0000","last_clean":"2024-03-06T07:21:01.441007+0000","last_became_active":"2024-03-06T07:21:01.439672+0000","last_became_peered":"2024-03-06T07:21:01.439672+0000","last_unstale":"2024-03-06T07:21:01.441007+0000","last_undegraded":"2024-03-06T07:21:01.441007+0000","last_fullsized":"2024-03-06T07:21:01.441007+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:37.148955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428677+0000","last_change":"2024-03-06T07:20:51.879085+0000","last_active":"2024-03-06T07:21:01.428677+0000","last_peered":"2024-03-06T07:21:01.428677+0000","last_clean":"2024-03-06T07:21:01.428677+0000","last_became_active":"2024-03-06T07:20:51.878776+0000","last_became_peered":"2024-03-06T07:20:51.878776+0000","last_unstale":"2024-03-06T07:21:01.428677+0000","last_undegraded":"2024-03-06T07:21:01.428677+0000","last_fullsized":"2024-03-06T07:21:01.428677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426282+0000","last_change":"2024-03-06T07:20:50.887928+0000","last_active":"2024-03-06T07:21:01.426282+0000","last_peered":"2024-03-06T07:21:01.426282+0000","last_clean":"2024-03-06T07:21:01.426282+0000","last_became_active":"2024-03-06T07:20:50.887638+0000","last_became_peered":"2024-03-06T07:20:50.887638+0000","last_unstale":"2024-03-06T07:21:01.426282+0000","last_undegraded":"2024-03-06T07:21:01.426282+0000","last_fullsized":"2024-03-06T07:21:01.426282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:52.309668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422497+0000","last_change":"2024-03-06T07:21:01.422497+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422497+0000","last_undegraded":"2024-03-06T07:21:01.422497+0000","last_fullsized":"2024-03-06T07:21:01.422497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440089+0000","last_change":"2024-03-06T07:21:01.436756+0000","last_active":"2024-03-06T07:21:01.440089+0000","last_peered":"2024-03-06T07:21:01.440089+0000","last_clean":"2024-03-06T07:21:01.440089+0000","last_became_active":"2024-03-06T07:21:01.436158+0000","last_became_peered":"2024-03-06T07:21:01.436158+0000","last_unstale":"2024-03-06T07:21:01.440089+0000","last_undegraded":"2024-03-06T07:21:01.440089+0000","last_fullsized":"2024-03-06T07:21:01.440089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:11.908825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431875+0000","last_change":"2024-03-06T07:20:51.883253+0000","last_active":"2024-03-06T07:21:01.431875+0000","last_peered":"2024-03-06T07:21:01.431875+0000","last_clean":"2024-03-06T07:21:01.431875+0000","last_became_active":"2024-03-06T07:20:51.882993+0000","last_became_peered":"2024-03-06T07:20:51.882993+0000","last_unstale":"2024-03-06T07:21:01.431875+0000","last_undegraded":"2024-03-06T07:21:01.431875+0000","last_fullsized":"2024-03-06T07:21:01.431875+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426104+0000","last_change":"2024-03-06T07:20:50.879149+0000","last_active":"2024-03-06T07:21:01.426104+0000","last_peered":"2024-03-06T07:21:01.426104+0000","last_clean":"2024-03-06T07:21:01.426104+0000","last_became_active":"2024-03-06T07:20:50.878867+0000","last_became_peered":"2024-03-06T07:20:50.878867+0000","last_unstale":"2024-03-06T07:21:01.426104+0000","last_undegraded":"2024-03-06T07:21:01.426104+0000","last_fullsized":"2024-03-06T07:21:01.426104+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:57.394196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429819+0000","last_change":"2024-03-06T07:21:01.429819+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429819+0000","last_undegraded":"2024-03-06T07:21:01.429819+0000","last_fullsized":"2024-03-06T07:21:01.429819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442400+0000","last_change":"2024-03-06T07:21:01.440630+0000","last_active":"2024-03-06T07:21:01.442400+0000","last_peered":"2024-03-06T07:21:01.442400+0000","last_clean":"2024-03-06T07:21:01.442400+0000","last_became_active":"2024-03-06T07:21:01.440375+0000","last_became_peered":"2024-03-06T07:21:01.440375+0000","last_unstale":"2024-03-06T07:21:01.442400+0000","last_undegraded":"2024-03-06T07:21:01.442400+0000","last_fullsized":"2024-03-06T07:21:01.442400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432805+0000","last_change":"2024-03-06T07:20:51.891707+0000","last_active":"2024-03-06T07:21:01.432805+0000","last_peered":"2024-03-06T07:21:01.432805+0000","last_clean":"2024-03-06T07:21:01.432805+0000","last_became_active":"2024-03-06T07:20:51.891427+0000","last_became_peered":"2024-03-06T07:20:51.891427+0000","last_unstale":"2024-03-06T07:21:01.432805+0000","last_undegraded":"2024-03-06T07:21:01.432805+0000","last_fullsized":"2024-03-06T07:21:01.432805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425699+0000","last_change":"2024-03-06T07:20:50.885372+0000","last_active":"2024-03-06T07:21:01.425699+0000","last_peered":"2024-03-06T07:21:01.425699+0000","last_clean":"2024-03-06T07:21:01.425699+0000","last_became_active":"2024-03-06T07:20:50.885093+0000","last_became_peered":"2024-03-06T07:20:50.885093+0000","last_unstale":"2024-03-06T07:21:01.425699+0000","last_undegraded":"2024-03-06T07:21:01.425699+0000","last_fullsized":"2024-03-06T07:21:01.425699+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:12.807980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":276,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430887+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:21:01.430887+0000","last_peered":"2024-03-06T07:21:01.430887+0000","last_clean":"2024-03-06T07:21:01.430887+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:21:01.430887+0000","last_undegraded":"2024-03-06T07:21:01.430887+0000","last_fullsized":"2024-03-06T07:21:01.430887+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":1031546,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":442768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":441190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":950272,"data_stored":442872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":933888,"data_stored":441738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":441898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":441268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":1031524,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:06.120 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3d is not active+clean 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.427479+0000', 'last_change': '2024-03-06T07:21:01.427479+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.427479+0000', 'last_undegraded': '2024-03-06T07:21:01.427479+0000', 'last_fullsized': '2024-03-06T07:21:01.427479+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3c is not active+clean 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.426920+0000', 'last_change': '2024-03-06T07:21:01.426920+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.426920+0000', 'last_undegraded': '2024-03-06T07:21:01.426920+0000', 'last_fullsized': '2024-03-06T07:21:01.426920+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3f is not active+clean 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.427931+0000', 'last_change': '2024-03-06T07:21:01.427931+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.427931+0000', 'last_undegraded': '2024-03-06T07:21:01.427931+0000', 'last_fullsized': '2024-03-06T07:21:01.427931+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3e is not active+clean 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.428790+0000', 'last_change': '2024-03-06T07:21:01.428790+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.428790+0000', 'last_undegraded': '2024-03-06T07:21:01.428790+0000', 'last_fullsized': '2024-03-06T07:21:01.428790+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.39 is not active+clean 2024-03-06T07:21:06.121 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.429413+0000', 'last_change': '2024-03-06T07:21:01.429413+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.429413+0000', 'last_undegraded': '2024-03-06T07:21:01.429413+0000', 'last_fullsized': '2024-03-06T07:21:01.429413+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.38 is not active+clean 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424683+0000', 'last_change': '2024-03-06T07:21:01.424683+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424683+0000', 'last_undegraded': '2024-03-06T07:21:01.424683+0000', 'last_fullsized': '2024-03-06T07:21:01.424683+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3b is not active+clean 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424861+0000', 'last_change': '2024-03-06T07:21:01.424861+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424861+0000', 'last_undegraded': '2024-03-06T07:21:01.424861+0000', 'last_fullsized': '2024-03-06T07:21:01.424861+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3a is not active+clean 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.430080+0000', 'last_change': '2024-03-06T07:21:01.430080+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.430080+0000', 'last_undegraded': '2024-03-06T07:21:01.430080+0000', 'last_fullsized': '2024-03-06T07:21:01.430080+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.35 is not active+clean 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424697+0000', 'last_change': '2024-03-06T07:21:01.424697+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424697+0000', 'last_undegraded': '2024-03-06T07:21:01.424697+0000', 'last_fullsized': '2024-03-06T07:21:01.424697+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.34 is not active+clean 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.426060+0000', 'last_change': '2024-03-06T07:21:01.426060+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.426060+0000', 'last_undegraded': '2024-03-06T07:21:01.426060+0000', 'last_fullsized': '2024-03-06T07:21:01.426060+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.37 is not active+clean 2024-03-06T07:21:06.122 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.425029+0000', 'last_change': '2024-03-06T07:21:01.425029+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.425029+0000', 'last_undegraded': '2024-03-06T07:21:01.425029+0000', 'last_fullsized': '2024-03-06T07:21:01.425029+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.36 is not active+clean 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424341+0000', 'last_change': '2024-03-06T07:21:01.424341+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424341+0000', 'last_undegraded': '2024-03-06T07:21:01.424341+0000', 'last_fullsized': '2024-03-06T07:21:01.424341+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.31 is not active+clean 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.426115+0000', 'last_change': '2024-03-06T07:21:01.426115+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.426115+0000', 'last_undegraded': '2024-03-06T07:21:01.426115+0000', 'last_fullsized': '2024-03-06T07:21:01.426115+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.30 is not active+clean 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.431576+0000', 'last_change': '2024-03-06T07:21:01.431576+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.431576+0000', 'last_undegraded': '2024-03-06T07:21:01.431576+0000', 'last_fullsized': '2024-03-06T07:21:01.431576+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.33 is not active+clean 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424813+0000', 'last_change': '2024-03-06T07:21:01.424813+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424813+0000', 'last_undegraded': '2024-03-06T07:21:01.424813+0000', 'last_fullsized': '2024-03-06T07:21:01.424813+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.32 is not active+clean 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.425999+0000', 'last_change': '2024-03-06T07:21:01.425999+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.425999+0000', 'last_undegraded': '2024-03-06T07:21:01.425999+0000', 'last_fullsized': '2024-03-06T07:21:01.425999+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.123 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2d is not active+clean 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424318+0000', 'last_change': '2024-03-06T07:21:01.424318+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424318+0000', 'last_undegraded': '2024-03-06T07:21:01.424318+0000', 'last_fullsized': '2024-03-06T07:21:01.424318+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2c is not active+clean 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.426582+0000', 'last_change': '2024-03-06T07:21:01.426582+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.426582+0000', 'last_undegraded': '2024-03-06T07:21:01.426582+0000', 'last_fullsized': '2024-03-06T07:21:01.426582+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2f is not active+clean 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.429516+0000', 'last_change': '2024-03-06T07:21:01.429516+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.429516+0000', 'last_undegraded': '2024-03-06T07:21:01.429516+0000', 'last_fullsized': '2024-03-06T07:21:01.429516+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2e is not active+clean 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.426508+0000', 'last_change': '2024-03-06T07:21:01.426508+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.426508+0000', 'last_undegraded': '2024-03-06T07:21:01.426508+0000', 'last_fullsized': '2024-03-06T07:21:01.426508+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.29 is not active+clean 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423254+0000', 'last_change': '2024-03-06T07:21:01.423254+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423254+0000', 'last_undegraded': '2024-03-06T07:21:01.423254+0000', 'last_fullsized': '2024-03-06T07:21:01.423254+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.28 is not active+clean 2024-03-06T07:21:06.124 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422935+0000', 'last_change': '2024-03-06T07:21:01.422935+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422935+0000', 'last_undegraded': '2024-03-06T07:21:01.422935+0000', 'last_fullsized': '2024-03-06T07:21:01.422935+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2b is not active+clean 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423598+0000', 'last_change': '2024-03-06T07:21:01.423598+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423598+0000', 'last_undegraded': '2024-03-06T07:21:01.423598+0000', 'last_fullsized': '2024-03-06T07:21:01.423598+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2a is not active+clean 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424033+0000', 'last_change': '2024-03-06T07:21:01.424033+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424033+0000', 'last_undegraded': '2024-03-06T07:21:01.424033+0000', 'last_fullsized': '2024-03-06T07:21:01.424033+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.25 is not active+clean 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.427151+0000', 'last_change': '2024-03-06T07:21:01.427151+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.427151+0000', 'last_undegraded': '2024-03-06T07:21:01.427151+0000', 'last_fullsized': '2024-03-06T07:21:01.427151+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.24 is not active+clean 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.425922+0000', 'last_change': '2024-03-06T07:21:01.425922+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.425922+0000', 'last_undegraded': '2024-03-06T07:21:01.425922+0000', 'last_fullsized': '2024-03-06T07:21:01.425922+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.27 is not active+clean 2024-03-06T07:21:06.125 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.428340+0000', 'last_change': '2024-03-06T07:21:01.428340+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.428340+0000', 'last_undegraded': '2024-03-06T07:21:01.428340+0000', 'last_fullsized': '2024-03-06T07:21:01.428340+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.26 is not active+clean 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.426198+0000', 'last_change': '2024-03-06T07:21:01.426198+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.426198+0000', 'last_undegraded': '2024-03-06T07:21:01.426198+0000', 'last_fullsized': '2024-03-06T07:21:01.426198+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.21 is not active+clean 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.428167+0000', 'last_change': '2024-03-06T07:21:01.428167+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.428167+0000', 'last_undegraded': '2024-03-06T07:21:01.428167+0000', 'last_fullsized': '2024-03-06T07:21:01.428167+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.20 is not active+clean 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.425332+0000', 'last_change': '2024-03-06T07:21:01.425332+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.425332+0000', 'last_undegraded': '2024-03-06T07:21:01.425332+0000', 'last_fullsized': '2024-03-06T07:21:01.425332+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.23 is not active+clean 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.429450+0000', 'last_change': '2024-03-06T07:21:01.429450+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.429450+0000', 'last_undegraded': '2024-03-06T07:21:01.429450+0000', 'last_fullsized': '2024-03-06T07:21:01.429450+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.22 is not active+clean 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424979+0000', 'last_change': '2024-03-06T07:21:01.424979+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424979+0000', 'last_undegraded': '2024-03-06T07:21:01.424979+0000', 'last_fullsized': '2024-03-06T07:21:01.424979+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.126 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1a is not active+clean 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424138+0000', 'last_change': '2024-03-06T07:21:01.424138+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424138+0000', 'last_undegraded': '2024-03-06T07:21:01.424138+0000', 'last_fullsized': '2024-03-06T07:21:01.424138+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424424+0000', 'last_change': '2024-03-06T07:21:01.424424+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424424+0000', 'last_undegraded': '2024-03-06T07:21:01.424424+0000', 'last_fullsized': '2024-03-06T07:21:01.424424+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424260+0000', 'last_change': '2024-03-06T07:21:01.424260+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424260+0000', 'last_undegraded': '2024-03-06T07:21:01.424260+0000', 'last_fullsized': '2024-03-06T07:21:01.424260+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422765+0000', 'last_change': '2024-03-06T07:21:01.422765+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422765+0000', 'last_undegraded': '2024-03-06T07:21:01.422765+0000', 'last_fullsized': '2024-03-06T07:21:01.422765+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.429826+0000', 'last_change': '2024-03-06T07:21:01.429826+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.429826+0000', 'last_undegraded': '2024-03-06T07:21:01.429826+0000', 'last_fullsized': '2024-03-06T07:21:01.429826+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1f is not active+clean 2024-03-06T07:21:06.127 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424875+0000', 'last_change': '2024-03-06T07:21:01.424875+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424875+0000', 'last_undegraded': '2024-03-06T07:21:01.424875+0000', 'last_fullsized': '2024-03-06T07:21:01.424875+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1c is not active+clean 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424723+0000', 'last_change': '2024-03-06T07:21:01.424723+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424723+0000', 'last_undegraded': '2024-03-06T07:21:01.424723+0000', 'last_fullsized': '2024-03-06T07:21:01.424723+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1d is not active+clean 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.424583+0000', 'last_change': '2024-03-06T07:21:01.424583+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.424583+0000', 'last_undegraded': '2024-03-06T07:21:01.424583+0000', 'last_fullsized': '2024-03-06T07:21:01.424583+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.425274+0000', 'last_change': '2024-03-06T07:21:01.425274+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.425274+0000', 'last_undegraded': '2024-03-06T07:21:01.425274+0000', 'last_fullsized': '2024-03-06T07:21:01.425274+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.427634+0000', 'last_change': '2024-03-06T07:21:01.427634+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.427634+0000', 'last_undegraded': '2024-03-06T07:21:01.427634+0000', 'last_fullsized': '2024-03-06T07:21:01.427634+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-03-06T07:21:06.128 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423808+0000', 'last_change': '2024-03-06T07:21:01.423808+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423808+0000', 'last_undegraded': '2024-03-06T07:21:01.423808+0000', 'last_fullsized': '2024-03-06T07:21:01.423808+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423462+0000', 'last_change': '2024-03-06T07:21:01.423462+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423462+0000', 'last_undegraded': '2024-03-06T07:21:01.423462+0000', 'last_fullsized': '2024-03-06T07:21:01.423462+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423329+0000', 'last_change': '2024-03-06T07:21:01.423329+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423329+0000', 'last_undegraded': '2024-03-06T07:21:01.423329+0000', 'last_fullsized': '2024-03-06T07:21:01.423329+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.425722+0000', 'last_change': '2024-03-06T07:21:01.425722+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.425722+0000', 'last_undegraded': '2024-03-06T07:21:01.425722+0000', 'last_fullsized': '2024-03-06T07:21:01.425722+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423587+0000', 'last_change': '2024-03-06T07:21:01.423587+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423587+0000', 'last_undegraded': '2024-03-06T07:21:01.423587+0000', 'last_fullsized': '2024-03-06T07:21:01.423587+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.430090+0000', 'last_change': '2024-03-06T07:21:01.430090+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.430090+0000', 'last_undegraded': '2024-03-06T07:21:01.430090+0000', 'last_fullsized': '2024-03-06T07:21:01.430090+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:21:06.129 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.427525+0000', 'last_change': '2024-03-06T07:21:01.427525+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.427525+0000', 'last_undegraded': '2024-03-06T07:21:01.427525+0000', 'last_fullsized': '2024-03-06T07:21:01.427525+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.b is not active+clean 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423288+0000', 'last_change': '2024-03-06T07:21:01.423288+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423288+0000', 'last_undegraded': '2024-03-06T07:21:01.423288+0000', 'last_fullsized': '2024-03-06T07:21:01.423288+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.8 is not active+clean 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422966+0000', 'last_change': '2024-03-06T07:21:01.422966+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422966+0000', 'last_undegraded': '2024-03-06T07:21:01.422966+0000', 'last_fullsized': '2024-03-06T07:21:01.422966+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.428740+0000', 'last_change': '2024-03-06T07:21:01.428740+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.428740+0000', 'last_undegraded': '2024-03-06T07:21:01.428740+0000', 'last_fullsized': '2024-03-06T07:21:01.428740+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423784+0000', 'last_change': '2024-03-06T07:21:01.423784+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423784+0000', 'last_undegraded': '2024-03-06T07:21:01.423784+0000', 'last_fullsized': '2024-03-06T07:21:01.423784+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:21:06.130 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.c is not active+clean 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422551+0000', 'last_change': '2024-03-06T07:21:01.422551+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422551+0000', 'last_undegraded': '2024-03-06T07:21:01.422551+0000', 'last_fullsized': '2024-03-06T07:21:01.422551+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422909+0000', 'last_change': '2024-03-06T07:21:01.422909+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422909+0000', 'last_undegraded': '2024-03-06T07:21:01.422909+0000', 'last_fullsized': '2024-03-06T07:21:01.422909+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423927+0000', 'last_change': '2024-03-06T07:21:01.423927+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423927+0000', 'last_undegraded': '2024-03-06T07:21:01.423927+0000', 'last_fullsized': '2024-03-06T07:21:01.423927+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3 is not active+clean 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422540+0000', 'last_change': '2024-03-06T07:21:01.422540+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422540+0000', 'last_undegraded': '2024-03-06T07:21:01.422540+0000', 'last_fullsized': '2024-03-06T07:21:01.422540+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.427082+0000', 'last_change': '2024-03-06T07:21:01.427082+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.427082+0000', 'last_undegraded': '2024-03-06T07:21:01.427082+0000', 'last_fullsized': '2024-03-06T07:21:01.427082+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-03-06T07:21:06.131 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.423705+0000', 'last_change': '2024-03-06T07:21:01.423705+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.423705+0000', 'last_undegraded': '2024-03-06T07:21:01.423705+0000', 'last_fullsized': '2024-03-06T07:21:01.423705+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.6 is not active+clean 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422649+0000', 'last_change': '2024-03-06T07:21:01.422649+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422649+0000', 'last_undegraded': '2024-03-06T07:21:01.422649+0000', 'last_fullsized': '2024-03-06T07:21:01.422649+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.427886+0000', 'last_change': '2024-03-06T07:21:01.427886+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.427886+0000', 'last_undegraded': '2024-03-06T07:21:01.427886+0000', 'last_fullsized': '2024-03-06T07:21:01.427886+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422772+0000', 'last_change': '2024-03-06T07:21:01.422772+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422772+0000', 'last_undegraded': '2024-03-06T07:21:01.422772+0000', 'last_fullsized': '2024-03-06T07:21:01.422772+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.5 is not active+clean 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.422497+0000', 'last_change': '2024-03-06T07:21:01.422497+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.422497+0000', 'last_undegraded': '2024-03-06T07:21:01.422497+0000', 'last_fullsized': '2024-03-06T07:21:01.422497+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-03-06T07:21:06.132 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 76, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:21:01.429819+0000', 'last_change': '2024-03-06T07:21:01.429819+0000', 'last_active': '2024-03-06T07:21:01.415411+0000', 'last_peered': '2024-03-06T07:21:01.415411+0000', 'last_clean': '2024-03-06T07:21:01.415411+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:21:01.429819+0000', 'last_undegraded': '2024-03-06T07:21:01.429819+0000', 'last_fullsized': '2024-03-06T07:21:01.429819+0000', 'mapping_epoch': 76, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 76, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'last_clean_scrub_stamp': '2024-03-06T07:21:01.415411+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:21:06.133 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:21:06.141 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:21:06.139+0000 7ff1a967d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:06.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.251+0000 7fa119e78640 1 Processor -- start 2024-03-06T07:21:06.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.251+0000 7fa119e78640 1 -- start start 2024-03-06T07:21:06.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa119e78640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa114150580 0x7fa11406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa119e78640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa114153a60 con 0x7fa1140c6000 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa119e78640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa114153d40 con 0x7fa1140c1320 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa119e78640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11406c2c0 con 0x7fa114150580 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa113fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa1140c6000 0x7fa114156dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41920/0 (socket says 172.21.15.94:41920) 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa113fff640 1 -- 172.21.15.94:0/1540148454 learned_addr learned my addr 172.21.15.94:0/1540148454 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa1137fe640 1 --2- 172.21.15.94:0/1540148454 >> v2:172.21.15.145:3300/0 conn(0x7fa114150580 0x7fa11406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa1127fc640 1 -- 172.21.15.94:0/1540148454 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700675149 0 0) 0x7fa114153a60 con 0x7fa1140c6000 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.252+0000 7fa1127fc640 1 -- 172.21.15.94:0/1540148454 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc003520 con 0x7fa1140c6000 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1137fe640 1 -- 172.21.15.94:0/1540148454 >> v1:172.21.15.145:6789/0 conn(0x7fa1140c1320 legacy=0x7fa1141566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1137fe640 1 -- 172.21.15.94:0/1540148454 >> v1:172.21.15.94:6789/0 conn(0x7fa1140c6000 legacy=0x7fa114156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1127fc640 1 -- 172.21.15.94:0/1540148454 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355702378 0 0) 0x7fa114153d40 con 0x7fa1140c1320 2024-03-06T07:21:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1137fe640 1 -- 172.21.15.94:0/1540148454 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11406c5a0 con 0x7fa114150580 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1137fe640 1 --2- 172.21.15.94:0/1540148454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 0x7fa11406dd10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097a0 tx=0x7fa10402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2142b236be7a4e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1127fc640 1 -- 172.21.15.94:0/1540148454 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104035020 con 0x7fa114150580 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1127fc640 1 -- 172.21.15.94:0/1540148454 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104031830 con 0x7fa114150580 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.253+0000 7fa1127fc640 1 -- 172.21.15.94:0/1540148454 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104031c00 con 0x7fa114150580 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.254+0000 7fa119e78640 1 -- 172.21.15.94:0/1540148454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 msgr2=0x7fa11406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.254+0000 7fa119e78640 1 --2- 172.21.15.94:0/1540148454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 0x7fa11406dd10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097a0 tx=0x7fa10402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.254+0000 7fa119e78640 1 -- 172.21.15.94:0/1540148454 shutdown_connections 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.254+0000 7fa119e78640 1 --2- 172.21.15.94:0/1540148454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 0x7fa11406dd10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.254+0000 7fa119e78640 1 -- 172.21.15.94:0/1540148454 >> 172.21.15.94:0/1540148454 conn(0x7fa1140c6710 msgr2=0x7fa1140c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.254+0000 7fa119e78640 1 -- 172.21.15.94:0/1540148454 shutdown_connections 2024-03-06T07:21:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.254+0000 7fa119e78640 1 -- 172.21.15.94:0/1540148454 wait complete. 2024-03-06T07:21:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.255+0000 7fa119e78640 1 Processor -- start 2024-03-06T07:21:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.255+0000 7fa119e78640 1 -- start start 2024-03-06T07:21:06.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa119e78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c1320 0x7fa114151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa119e78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1140c6000 0x7fa114072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa119e78640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 0x7fa114149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa119e78640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140c0dc0 con 0x7fa1140c1320 2024-03-06T07:21:06.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa119e78640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140c0c40 con 0x7fa114150580 2024-03-06T07:21:06.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa119e78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11406e250 con 0x7fa1140c6000 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c1320 0x7fa114151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa113fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 0x7fa114149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c1320 0x7fa114151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38936/0 (socket says 172.21.15.94:38936) 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 -- 172.21.15.94:0/3963395740 learned_addr learned my addr 172.21.15.94:0/3963395740 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa112ffd640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1140c6000 0x7fa114072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 -- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1140c6000 msgr2=0x7fa114072d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1140c6000 0x7fa114072d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 -- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 msgr2=0x7fa114149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 0x7fa114149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.256+0000 7fa1137fe640 1 -- 172.21.15.94:0/3963395740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140c2f10 con 0x7fa1140c1320 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.257+0000 7fa1137fe640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c1320 0x7fa114151c70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa1040098c0 tx=0x7fa1040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.257+0000 7fa0fbfff640 1 -- 172.21.15.94:0/3963395740 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104040070 con 0x7fa1140c1320 2024-03-06T07:21:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.257+0000 7fa0fbfff640 1 -- 172.21.15.94:0/3963395740 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10403a830 con 0x7fa1140c1320 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.257+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1140c31f0 con 0x7fa1140c1320 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.257+0000 7fa0fbfff640 1 -- 172.21.15.94:0/3963395740 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104035040 con 0x7fa1140c1320 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.257+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1141521b0 con 0x7fa1140c1320 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.258+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11404f330 con 0x7fa1140c1320 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.258+0000 7fa0fbfff640 1 -- 172.21.15.94:0/3963395740 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa10404b060 con 0x7fa1140c1320 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.258+0000 7fa0fbfff640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b500 0x7fa10006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.259+0000 7fa0fbfff640 1 -- 172.21.15.94:0/3963395740 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa1040abcf0 con 0x7fa1140c1320 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.259+0000 7fa112ffd640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b500 0x7fa10006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.259+0000 7fa112ffd640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b500 0x7fa10006d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa108000f60 tx=0x7fa108009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.263+0000 7fa0fbfff640 1 -- 172.21.15.94:0/3963395740 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa1040b0070 con 0x7fa1140c1320 2024-03-06T07:21:06.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.500+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa11414db80 con 0x7fa1140c1320 2024-03-06T07:21:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.501+0000 7fa0fbfff640 1 -- 172.21.15.94:0/3963395740 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1559 (secure 0 0 0) 0x7fa10407a0d0 con 0x7fa1140c1320 2024-03-06T07:21:06.503 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:06.503 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":193,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":78,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709482,"num_in_osds":8,"osd_in_since":1709709474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129},{"state_name":"creating+peering","count":64}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":592778,"bytes_used":250290176,"bytes_avail":772843823104,"bytes_total":773094113280,"inactive_pgs_ratio":0.33160620927810669},"fsmap":{"epoch":59,"id":5,"up":1,"in":1,"max":1,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"d","status":"up:active","gid":35486},{"filesystem_id":5,"rank":0,"name":"b","status":"up:active","gid":35492}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-03-06T07:20:34.665817+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:21:06.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b500 msgr2=0x7fa10006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b500 0x7fa10006d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa108000f60 tx=0x7fa108009040 comp rx=0 tx=0).stop 2024-03-06T07:21:06.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c1320 msgr2=0x7fa114151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c1320 0x7fa114151c70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa1040098c0 tx=0x7fa1040046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 shutdown_connections 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b500 0x7fa10006d9c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114150580 0x7fa114149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1140c6000 0x7fa114072d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 --2- 172.21.15.94:0/3963395740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c1320 0x7fa114151c70 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 >> 172.21.15.94:0/3963395740 conn(0x7fa1140c6710 msgr2=0x7fa114156d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 shutdown_connections 2024-03-06T07:21:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.503+0000 7fa119e78640 1 -- 172.21.15.94:0/3963395740 wait complete. 2024-03-06T07:21:06.515 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:21:06.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.614+0000 7f876e283640 1 Processor -- start 2024-03-06T07:21:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.614+0000 7f876e283640 1 -- start start 2024-03-06T07:21:06.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876e283640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876e283640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f87680c88c0 0x7f87680c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876e283640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87680c5880 0x7f8768158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876e283640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87680c8dd0 con 0x7f87680c5880 2024-03-06T07:21:06.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876e283640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87680c73d0 con 0x7f87680c88c0 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876e283640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680be2e0 con 0x7f876814f360 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876c820640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87680c5880 0x7f8768158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876c820640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87680c5880 0x7f8768158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38952/0 (socket says 172.21.15.94:38952) 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f876c820640 1 -- 172.21.15.94:0/3093415395 learned_addr learned my addr 172.21.15.94:0/3093415395 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f87677fe640 1 --2- 172.21.15.94:0/3093415395 >> v2:172.21.15.145:3300/0 conn(0x7f87680c88c0 0x7f87680c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.615+0000 7f8767fff640 1 --2- 172.21.15.94:0/3093415395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f876c820640 1 -- 172.21.15.94:0/3093415395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 msgr2=0x7f8768155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f876c820640 1 --2- 172.21.15.94:0/3093415395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f876c820640 1 -- 172.21.15.94:0/3093415395 >> v2:172.21.15.145:3300/0 conn(0x7f87680c88c0 msgr2=0x7f87680c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f876c820640 1 --2- 172.21.15.94:0/3093415395 >> v2:172.21.15.145:3300/0 conn(0x7f87680c88c0 0x7f87680c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f876c820640 1 -- 172.21.15.94:0/3093415395 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680c7670 con 0x7f87680c5880 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f876c820640 1 --2- 172.21.15.94:0/3093415395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c5880 0x7f8768158b60 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f87680cbac0 tx=0x7f876002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81233efa5d6c7245 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f8766ffd640 1 -- 172.21.15.94:0/3093415395 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876002c9e0 con 0x7f87680c5880 2024-03-06T07:21:06.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f8766ffd640 1 -- 172.21.15.94:0/3093415395 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876002cb80 con 0x7f87680c5880 2024-03-06T07:21:06.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.616+0000 7f8766ffd640 1 -- 172.21.15.94:0/3093415395 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87600343f0 con 0x7f87680c5880 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 -- 172.21.15.94:0/3093415395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c5880 msgr2=0x7f8768158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 --2- 172.21.15.94:0/3093415395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c5880 0x7f8768158b60 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f87680cbac0 tx=0x7f876002be40 comp rx=0 tx=0).stop 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 -- 172.21.15.94:0/3093415395 shutdown_connections 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 --2- 172.21.15.94:0/3093415395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c5880 0x7f8768158b60 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 --2- 172.21.15.94:0/3093415395 >> v2:172.21.15.145:3300/0 conn(0x7f87680c88c0 0x7f87680c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 --2- 172.21.15.94:0/3093415395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 -- 172.21.15.94:0/3093415395 >> 172.21.15.94:0/3093415395 conn(0x7f87680ac570 msgr2=0x7f87680ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 -- 172.21.15.94:0/3093415395 shutdown_connections 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.617+0000 7f876e283640 1 -- 172.21.15.94:0/3093415395 wait complete. 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.618+0000 7f876e283640 1 Processor -- start 2024-03-06T07:21:06.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.618+0000 7f876e283640 1 -- start start 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f876e283640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87680c5880 0x7f87681ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f876e283640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c88c0 0x7f87680ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f876e283640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f876e283640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87681590a0 con 0x7f87680c88c0 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f876e283640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680c7c30 con 0x7f87680c5880 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f876e283640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8768159220 con 0x7f876814f360 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c88c0 0x7f87680ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c88c0 0x7f87680ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38956/0 (socket says 172.21.15.94:38956) 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 -- 172.21.15.94:0/4054387166 learned_addr learned my addr 172.21.15.94:0/4054387166 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f8767fff640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87680c5880 0x7f87681ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f876c820640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 -- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 msgr2=0x7f8768156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 -- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87680c5880 msgr2=0x7f87681ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87680c5880 0x7f87681ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.619+0000 7f87677fe640 1 -- 172.21.15.94:0/4054387166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680b0560 con 0x7f87680c88c0 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.620+0000 7f87677fe640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c88c0 0x7f87680ab7d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f875c00e860 tx=0x7f875c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.620+0000 7f8764ff9640 1 -- 172.21.15.94:0/4054387166 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c017070 con 0x7f87680c88c0 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.620+0000 7f8764ff9640 1 -- 172.21.15.94:0/4054387166 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c00c940 con 0x7f87680c88c0 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.620+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87680ac2d0 con 0x7f87680c88c0 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.620+0000 7f8764ff9640 1 -- 172.21.15.94:0/4054387166 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c00cd10 con 0x7f87680c88c0 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.620+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8768157d60 con 0x7f87680c88c0 2024-03-06T07:21:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.621+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876804f330 con 0x7f87680c88c0 2024-03-06T07:21:06.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.621+0000 7f8764ff9640 1 -- 172.21.15.94:0/4054387166 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f875c01a070 con 0x7f87680c88c0 2024-03-06T07:21:06.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.622+0000 7f8764ff9640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875406b4d0 0x7f875406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:06.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.622+0000 7f8764ff9640 1 -- 172.21.15.94:0/4054387166 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f875c08a5c0 con 0x7f87680c88c0 2024-03-06T07:21:06.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.622+0000 7f8767fff640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875406b4d0 0x7f875406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:06.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.622+0000 7f8767fff640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875406b4d0 0x7f875406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f87680ac540 tx=0x7f8758009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.626+0000 7f8764ff9640 1 -- 172.21.15.94:0/4054387166 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f875c0589a0 con 0x7f87680c88c0 2024-03-06T07:21:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.793+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8768158b00 con 0x7f875406b4d0 2024-03-06T07:21:06.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.805+0000 7f8764ff9640 1 -- 172.21.15.94:0/4054387166 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485419 (secure 0 0 0) 0x7f8768158b00 con 0x7f875406b4d0 2024-03-06T07:21:06.812 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:06.815 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:21:06.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875406b4d0 msgr2=0x7f875406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875406b4d0 0x7f875406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f87680ac540 tx=0x7f8758009190 comp rx=0 tx=0).stop 2024-03-06T07:21:06.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c88c0 msgr2=0x7f87680ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:06.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c88c0 0x7f87680ab7d0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f875c00e860 tx=0x7f875c00ed20 comp rx=0 tx=0).stop 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 shutdown_connections 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f875406b4d0 0x7f875406d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876814f360 0x7f8768156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.808+0000 7f876e283640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87680c88c0 0x7f87680ab7d0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.809+0000 7f876e283640 1 --2- 172.21.15.94:0/4054387166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87680c5880 0x7f87681ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.809+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 >> 172.21.15.94:0/4054387166 conn(0x7f87680ac570 msgr2=0x7f87680ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.809+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 shutdown_connections 2024-03-06T07:21:06.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:06.809+0000 7f876e283640 1 -- 172.21.15.94:0/4054387166 wait complete. 2024-03-06T07:21:06.821 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-06T07:21:06.671784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244424,"kb_used_data":7456,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730296,"statfs":{"total":773094113280,"available":772843823104,"internally_reserved":0,"allocated":7634944,"data_stored":4714804,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000883"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427479+0000","last_change":"2024-03-06T07:21:01.427479+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427479+0000","last_undegraded":"2024-03-06T07:21:01.427479+0000","last_fullsized":"2024-03-06T07:21:01.427479+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431835+0000","last_change":"2024-03-06T07:20:51.896309+0000","last_active":"2024-03-06T07:21:01.431835+0000","last_peered":"2024-03-06T07:21:01.431835+0000","last_clean":"2024-03-06T07:21:01.431835+0000","last_became_active":"2024-03-06T07:20:51.896009+0000","last_became_peered":"2024-03-06T07:20:51.896009+0000","last_unstale":"2024-03-06T07:21:01.431835+0000","last_undegraded":"2024-03-06T07:21:01.431835+0000","last_fullsized":"2024-03-06T07:21:01.431835+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426920+0000","last_change":"2024-03-06T07:21:01.426920+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426920+0000","last_undegraded":"2024-03-06T07:21:01.426920+0000","last_fullsized":"2024-03-06T07:21:01.426920+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419624+0000","last_change":"2024-03-06T07:20:51.895412+0000","last_active":"2024-03-06T07:21:01.419624+0000","last_peered":"2024-03-06T07:21:01.419624+0000","last_clean":"2024-03-06T07:21:01.419624+0000","last_became_active":"2024-03-06T07:20:51.895169+0000","last_became_peered":"2024-03-06T07:20:51.895169+0000","last_unstale":"2024-03-06T07:21:01.419624+0000","last_undegraded":"2024-03-06T07:21:01.419624+0000","last_fullsized":"2024-03-06T07:21:01.419624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427931+0000","last_change":"2024-03-06T07:21:01.427931+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427931+0000","last_undegraded":"2024-03-06T07:21:01.427931+0000","last_fullsized":"2024-03-06T07:21:01.427931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428431+0000","last_change":"2024-03-06T07:20:51.878718+0000","last_active":"2024-03-06T07:21:01.428431+0000","last_peered":"2024-03-06T07:21:01.428431+0000","last_clean":"2024-03-06T07:21:01.428431+0000","last_became_active":"2024-03-06T07:20:51.878467+0000","last_became_peered":"2024-03-06T07:20:51.878467+0000","last_unstale":"2024-03-06T07:21:01.428431+0000","last_undegraded":"2024-03-06T07:21:01.428431+0000","last_fullsized":"2024-03-06T07:21:01.428431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428790+0000","last_change":"2024-03-06T07:21:01.428790+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428790+0000","last_undegraded":"2024-03-06T07:21:01.428790+0000","last_fullsized":"2024-03-06T07:21:01.428790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431362+0000","last_change":"2024-03-06T07:20:51.892550+0000","last_active":"2024-03-06T07:21:01.431362+0000","last_peered":"2024-03-06T07:21:01.431362+0000","last_clean":"2024-03-06T07:21:01.431362+0000","last_became_active":"2024-03-06T07:20:51.892264+0000","last_became_peered":"2024-03-06T07:20:51.892264+0000","last_unstale":"2024-03-06T07:21:01.431362+0000","last_undegraded":"2024-03-06T07:21:01.431362+0000","last_fullsized":"2024-03-06T07:21:01.431362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429413+0000","last_change":"2024-03-06T07:21:01.429413+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429413+0000","last_undegraded":"2024-03-06T07:21:01.429413+0000","last_fullsized":"2024-03-06T07:21:01.429413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428191+0000","last_change":"2024-03-06T07:20:51.893305+0000","last_active":"2024-03-06T07:21:01.428191+0000","last_peered":"2024-03-06T07:21:01.428191+0000","last_clean":"2024-03-06T07:21:01.428191+0000","last_became_active":"2024-03-06T07:20:51.893041+0000","last_became_peered":"2024-03-06T07:20:51.893041+0000","last_unstale":"2024-03-06T07:21:01.428191+0000","last_undegraded":"2024-03-06T07:21:01.428191+0000","last_fullsized":"2024-03-06T07:21:01.428191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:39.229585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424683+0000","last_change":"2024-03-06T07:21:01.424683+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424683+0000","last_undegraded":"2024-03-06T07:21:01.424683+0000","last_fullsized":"2024-03-06T07:21:01.424683+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425180+0000","last_change":"2024-03-06T07:20:51.893812+0000","last_active":"2024-03-06T07:21:01.425180+0000","last_peered":"2024-03-06T07:21:01.425180+0000","last_clean":"2024-03-06T07:21:01.425180+0000","last_became_active":"2024-03-06T07:20:51.891649+0000","last_became_peered":"2024-03-06T07:20:51.891649+0000","last_unstale":"2024-03-06T07:21:01.425180+0000","last_undegraded":"2024-03-06T07:21:01.425180+0000","last_fullsized":"2024-03-06T07:21:01.425180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:46.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424861+0000","last_change":"2024-03-06T07:21:01.424861+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424861+0000","last_undegraded":"2024-03-06T07:21:01.424861+0000","last_fullsized":"2024-03-06T07:21:01.424861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426172+0000","last_change":"2024-03-06T07:20:51.894141+0000","last_active":"2024-03-06T07:21:01.426172+0000","last_peered":"2024-03-06T07:21:01.426172+0000","last_clean":"2024-03-06T07:21:01.426172+0000","last_became_active":"2024-03-06T07:20:51.893793+0000","last_became_peered":"2024-03-06T07:20:51.893793+0000","last_unstale":"2024-03-06T07:21:01.426172+0000","last_undegraded":"2024-03-06T07:21:01.426172+0000","last_fullsized":"2024-03-06T07:21:01.426172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430080+0000","last_change":"2024-03-06T07:21:01.430080+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430080+0000","last_undegraded":"2024-03-06T07:21:01.430080+0000","last_fullsized":"2024-03-06T07:21:01.430080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427332+0000","last_change":"2024-03-06T07:20:51.894037+0000","last_active":"2024-03-06T07:21:01.427332+0000","last_peered":"2024-03-06T07:21:01.427332+0000","last_clean":"2024-03-06T07:21:01.427332+0000","last_became_active":"2024-03-06T07:20:51.893773+0000","last_became_peered":"2024-03-06T07:20:51.893773+0000","last_unstale":"2024-03-06T07:21:01.427332+0000","last_undegraded":"2024-03-06T07:21:01.427332+0000","last_fullsized":"2024-03-06T07:21:01.427332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:48.344455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424697+0000","last_change":"2024-03-06T07:21:01.424697+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424697+0000","last_undegraded":"2024-03-06T07:21:01.424697+0000","last_fullsized":"2024-03-06T07:21:01.424697+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427134+0000","last_change":"2024-03-06T07:20:51.886223+0000","last_active":"2024-03-06T07:21:01.427134+0000","last_peered":"2024-03-06T07:21:01.427134+0000","last_clean":"2024-03-06T07:21:01.427134+0000","last_became_active":"2024-03-06T07:20:51.885985+0000","last_became_peered":"2024-03-06T07:20:51.885985+0000","last_unstale":"2024-03-06T07:21:01.427134+0000","last_undegraded":"2024-03-06T07:21:01.427134+0000","last_fullsized":"2024-03-06T07:21:01.427134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:13.814018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426060+0000","last_change":"2024-03-06T07:21:01.426060+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426060+0000","last_undegraded":"2024-03-06T07:21:01.426060+0000","last_fullsized":"2024-03-06T07:21:01.426060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431693+0000","last_change":"2024-03-06T07:20:51.890465+0000","last_active":"2024-03-06T07:21:01.431693+0000","last_peered":"2024-03-06T07:21:01.431693+0000","last_clean":"2024-03-06T07:21:01.431693+0000","last_became_active":"2024-03-06T07:20:51.890183+0000","last_became_peered":"2024-03-06T07:20:51.890183+0000","last_unstale":"2024-03-06T07:21:01.431693+0000","last_undegraded":"2024-03-06T07:21:01.431693+0000","last_fullsized":"2024-03-06T07:21:01.431693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425029+0000","last_change":"2024-03-06T07:21:01.425029+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425029+0000","last_undegraded":"2024-03-06T07:21:01.425029+0000","last_fullsized":"2024-03-06T07:21:01.425029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419915+0000","last_change":"2024-03-06T07:20:51.897836+0000","last_active":"2024-03-06T07:21:01.419915+0000","last_peered":"2024-03-06T07:21:01.419915+0000","last_clean":"2024-03-06T07:21:01.419915+0000","last_became_active":"2024-03-06T07:20:51.897551+0000","last_became_peered":"2024-03-06T07:20:51.897551+0000","last_unstale":"2024-03-06T07:21:01.419915+0000","last_undegraded":"2024-03-06T07:21:01.419915+0000","last_fullsized":"2024-03-06T07:21:01.419915+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424341+0000","last_change":"2024-03-06T07:21:01.424341+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424341+0000","last_undegraded":"2024-03-06T07:21:01.424341+0000","last_fullsized":"2024-03-06T07:21:01.424341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431445+0000","last_change":"2024-03-06T07:20:51.886475+0000","last_active":"2024-03-06T07:21:01.431445+0000","last_peered":"2024-03-06T07:21:01.431445+0000","last_clean":"2024-03-06T07:21:01.431445+0000","last_became_active":"2024-03-06T07:20:51.886218+0000","last_became_peered":"2024-03-06T07:20:51.886218+0000","last_unstale":"2024-03-06T07:21:01.431445+0000","last_undegraded":"2024-03-06T07:21:01.431445+0000","last_fullsized":"2024-03-06T07:21:01.431445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426115+0000","last_change":"2024-03-06T07:21:01.426115+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426115+0000","last_undegraded":"2024-03-06T07:21:01.426115+0000","last_fullsized":"2024-03-06T07:21:01.426115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428289+0000","last_change":"2024-03-06T07:20:51.887999+0000","last_active":"2024-03-06T07:21:01.428289+0000","last_peered":"2024-03-06T07:21:01.428289+0000","last_clean":"2024-03-06T07:21:01.428289+0000","last_became_active":"2024-03-06T07:20:51.887749+0000","last_became_peered":"2024-03-06T07:20:51.887749+0000","last_unstale":"2024-03-06T07:21:01.428289+0000","last_undegraded":"2024-03-06T07:21:01.428289+0000","last_fullsized":"2024-03-06T07:21:01.428289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:25.243948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.431576+0000","last_change":"2024-03-06T07:21:01.431576+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.431576+0000","last_undegraded":"2024-03-06T07:21:01.431576+0000","last_fullsized":"2024-03-06T07:21:01.431576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425276+0000","last_change":"2024-03-06T07:20:51.892140+0000","last_active":"2024-03-06T07:21:01.425276+0000","last_peered":"2024-03-06T07:21:01.425276+0000","last_clean":"2024-03-06T07:21:01.425276+0000","last_became_active":"2024-03-06T07:20:51.891195+0000","last_became_peered":"2024-03-06T07:20:51.891195+0000","last_unstale":"2024-03-06T07:21:01.425276+0000","last_undegraded":"2024-03-06T07:21:01.425276+0000","last_fullsized":"2024-03-06T07:21:01.425276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:28.641052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424813+0000","last_change":"2024-03-06T07:21:01.424813+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424813+0000","last_undegraded":"2024-03-06T07:21:01.424813+0000","last_fullsized":"2024-03-06T07:21:01.424813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419327+0000","last_change":"2024-03-06T07:20:51.894735+0000","last_active":"2024-03-06T07:21:01.419327+0000","last_peered":"2024-03-06T07:21:01.419327+0000","last_clean":"2024-03-06T07:21:01.419327+0000","last_became_active":"2024-03-06T07:20:51.894479+0000","last_became_peered":"2024-03-06T07:20:51.894479+0000","last_unstale":"2024-03-06T07:21:01.419327+0000","last_undegraded":"2024-03-06T07:21:01.419327+0000","last_fullsized":"2024-03-06T07:21:01.419327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425999+0000","last_change":"2024-03-06T07:21:01.425999+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425999+0000","last_undegraded":"2024-03-06T07:21:01.425999+0000","last_fullsized":"2024-03-06T07:21:01.425999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430687+0000","last_change":"2024-03-06T07:20:51.892450+0000","last_active":"2024-03-06T07:21:01.430687+0000","last_peered":"2024-03-06T07:21:01.430687+0000","last_clean":"2024-03-06T07:21:01.430687+0000","last_became_active":"2024-03-06T07:20:51.892184+0000","last_became_peered":"2024-03-06T07:20:51.892184+0000","last_unstale":"2024-03-06T07:21:01.430687+0000","last_undegraded":"2024-03-06T07:21:01.430687+0000","last_fullsized":"2024-03-06T07:21:01.430687+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424318+0000","last_change":"2024-03-06T07:21:01.424318+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424318+0000","last_undegraded":"2024-03-06T07:21:01.424318+0000","last_fullsized":"2024-03-06T07:21:01.424318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426749+0000","last_change":"2024-03-06T07:20:51.891492+0000","last_active":"2024-03-06T07:21:01.426749+0000","last_peered":"2024-03-06T07:21:01.426749+0000","last_clean":"2024-03-06T07:21:01.426749+0000","last_became_active":"2024-03-06T07:20:51.891212+0000","last_became_peered":"2024-03-06T07:20:51.891212+0000","last_unstale":"2024-03-06T07:21:01.426749+0000","last_undegraded":"2024-03-06T07:21:01.426749+0000","last_fullsized":"2024-03-06T07:21:01.426749+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:49.555650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426582+0000","last_change":"2024-03-06T07:21:01.426582+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426582+0000","last_undegraded":"2024-03-06T07:21:01.426582+0000","last_fullsized":"2024-03-06T07:21:01.426582+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428589+0000","last_change":"2024-03-06T07:20:51.876410+0000","last_active":"2024-03-06T07:21:01.428589+0000","last_peered":"2024-03-06T07:21:01.428589+0000","last_clean":"2024-03-06T07:21:01.428589+0000","last_became_active":"2024-03-06T07:20:51.876155+0000","last_became_peered":"2024-03-06T07:20:51.876155+0000","last_unstale":"2024-03-06T07:21:01.428589+0000","last_undegraded":"2024-03-06T07:21:01.428589+0000","last_fullsized":"2024-03-06T07:21:01.428589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429516+0000","last_change":"2024-03-06T07:21:01.429516+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429516+0000","last_undegraded":"2024-03-06T07:21:01.429516+0000","last_fullsized":"2024-03-06T07:21:01.429516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425366+0000","last_change":"2024-03-06T07:20:51.880019+0000","last_active":"2024-03-06T07:21:01.425366+0000","last_peered":"2024-03-06T07:21:01.425366+0000","last_clean":"2024-03-06T07:21:01.425366+0000","last_became_active":"2024-03-06T07:20:51.879756+0000","last_became_peered":"2024-03-06T07:20:51.879756+0000","last_unstale":"2024-03-06T07:21:01.425366+0000","last_undegraded":"2024-03-06T07:21:01.425366+0000","last_fullsized":"2024-03-06T07:21:01.425366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.594449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426508+0000","last_change":"2024-03-06T07:21:01.426508+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426508+0000","last_undegraded":"2024-03-06T07:21:01.426508+0000","last_fullsized":"2024-03-06T07:21:01.426508+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426322+0000","last_change":"2024-03-06T07:20:51.890815+0000","last_active":"2024-03-06T07:21:01.426322+0000","last_peered":"2024-03-06T07:21:01.426322+0000","last_clean":"2024-03-06T07:21:01.426322+0000","last_became_active":"2024-03-06T07:20:51.890558+0000","last_became_peered":"2024-03-06T07:20:51.890558+0000","last_unstale":"2024-03-06T07:21:01.426322+0000","last_undegraded":"2024-03-06T07:21:01.426322+0000","last_fullsized":"2024-03-06T07:21:01.426322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:50.925154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423254+0000","last_change":"2024-03-06T07:21:01.423254+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423254+0000","last_undegraded":"2024-03-06T07:21:01.423254+0000","last_fullsized":"2024-03-06T07:21:01.423254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426944+0000","last_change":"2024-03-06T07:20:51.884508+0000","last_active":"2024-03-06T07:21:01.426944+0000","last_peered":"2024-03-06T07:21:01.426944+0000","last_clean":"2024-03-06T07:21:01.426944+0000","last_became_active":"2024-03-06T07:20:51.884237+0000","last_became_peered":"2024-03-06T07:20:51.884237+0000","last_unstale":"2024-03-06T07:21:01.426944+0000","last_undegraded":"2024-03-06T07:21:01.426944+0000","last_fullsized":"2024-03-06T07:21:01.426944+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422935+0000","last_change":"2024-03-06T07:21:01.422935+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422935+0000","last_undegraded":"2024-03-06T07:21:01.422935+0000","last_fullsized":"2024-03-06T07:21:01.422935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425637+0000","last_change":"2024-03-06T07:20:51.890118+0000","last_active":"2024-03-06T07:21:01.425637+0000","last_peered":"2024-03-06T07:21:01.425637+0000","last_clean":"2024-03-06T07:21:01.425637+0000","last_became_active":"2024-03-06T07:20:51.888426+0000","last_became_peered":"2024-03-06T07:20:51.888426+0000","last_unstale":"2024-03-06T07:21:01.425637+0000","last_undegraded":"2024-03-06T07:21:01.425637+0000","last_fullsized":"2024-03-06T07:21:01.425637+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:46.950695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423598+0000","last_change":"2024-03-06T07:21:01.423598+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423598+0000","last_undegraded":"2024-03-06T07:21:01.423598+0000","last_fullsized":"2024-03-06T07:21:01.423598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426626+0000","last_change":"2024-03-06T07:20:51.885627+0000","last_active":"2024-03-06T07:21:01.426626+0000","last_peered":"2024-03-06T07:21:01.426626+0000","last_clean":"2024-03-06T07:21:01.426626+0000","last_became_active":"2024-03-06T07:20:51.885338+0000","last_became_peered":"2024-03-06T07:20:51.885338+0000","last_unstale":"2024-03-06T07:21:01.426626+0000","last_undegraded":"2024-03-06T07:21:01.426626+0000","last_fullsized":"2024-03-06T07:21:01.426626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:41.439891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424033+0000","last_change":"2024-03-06T07:21:01.424033+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424033+0000","last_undegraded":"2024-03-06T07:21:01.424033+0000","last_fullsized":"2024-03-06T07:21:01.424033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419243+0000","last_change":"2024-03-06T07:20:51.884448+0000","last_active":"2024-03-06T07:21:01.419243+0000","last_peered":"2024-03-06T07:21:01.419243+0000","last_clean":"2024-03-06T07:21:01.419243+0000","last_became_active":"2024-03-06T07:20:51.884176+0000","last_became_peered":"2024-03-06T07:20:51.884176+0000","last_unstale":"2024-03-06T07:21:01.419243+0000","last_undegraded":"2024-03-06T07:21:01.419243+0000","last_fullsized":"2024-03-06T07:21:01.419243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:29.691807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427151+0000","last_change":"2024-03-06T07:21:01.427151+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427151+0000","last_undegraded":"2024-03-06T07:21:01.427151+0000","last_fullsized":"2024-03-06T07:21:01.427151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419496+0000","last_change":"2024-03-06T07:20:51.886062+0000","last_active":"2024-03-06T07:21:01.419496+0000","last_peered":"2024-03-06T07:21:01.419496+0000","last_clean":"2024-03-06T07:21:01.419496+0000","last_became_active":"2024-03-06T07:20:51.885799+0000","last_became_peered":"2024-03-06T07:20:51.885799+0000","last_unstale":"2024-03-06T07:21:01.419496+0000","last_undegraded":"2024-03-06T07:21:01.419496+0000","last_fullsized":"2024-03-06T07:21:01.419496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425922+0000","last_change":"2024-03-06T07:21:01.425922+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425922+0000","last_undegraded":"2024-03-06T07:21:01.425922+0000","last_fullsized":"2024-03-06T07:21:01.425922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431786+0000","last_change":"2024-03-06T07:20:51.883614+0000","last_active":"2024-03-06T07:21:01.431786+0000","last_peered":"2024-03-06T07:21:01.431786+0000","last_clean":"2024-03-06T07:21:01.431786+0000","last_became_active":"2024-03-06T07:20:51.883266+0000","last_became_peered":"2024-03-06T07:20:51.883266+0000","last_unstale":"2024-03-06T07:21:01.431786+0000","last_undegraded":"2024-03-06T07:21:01.431786+0000","last_fullsized":"2024-03-06T07:21:01.431786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428340+0000","last_change":"2024-03-06T07:21:01.428340+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428340+0000","last_undegraded":"2024-03-06T07:21:01.428340+0000","last_fullsized":"2024-03-06T07:21:01.428340+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432598+0000","last_change":"2024-03-06T07:20:51.893301+0000","last_active":"2024-03-06T07:21:01.432598+0000","last_peered":"2024-03-06T07:21:01.432598+0000","last_clean":"2024-03-06T07:21:01.432598+0000","last_became_active":"2024-03-06T07:20:51.892946+0000","last_became_peered":"2024-03-06T07:20:51.892946+0000","last_unstale":"2024-03-06T07:21:01.432598+0000","last_undegraded":"2024-03-06T07:21:01.432598+0000","last_fullsized":"2024-03-06T07:21:01.432598+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.426198+0000","last_change":"2024-03-06T07:21:01.426198+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.426198+0000","last_undegraded":"2024-03-06T07:21:01.426198+0000","last_fullsized":"2024-03-06T07:21:01.426198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426417+0000","last_change":"2024-03-06T07:20:51.891689+0000","last_active":"2024-03-06T07:21:01.426417+0000","last_peered":"2024-03-06T07:21:01.426417+0000","last_clean":"2024-03-06T07:21:01.426417+0000","last_became_active":"2024-03-06T07:20:51.891341+0000","last_became_peered":"2024-03-06T07:20:51.891341+0000","last_unstale":"2024-03-06T07:21:01.426417+0000","last_undegraded":"2024-03-06T07:21:01.426417+0000","last_fullsized":"2024-03-06T07:21:01.426417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:45.060281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428167+0000","last_change":"2024-03-06T07:21:01.428167+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428167+0000","last_undegraded":"2024-03-06T07:21:01.428167+0000","last_fullsized":"2024-03-06T07:21:01.428167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419570+0000","last_change":"2024-03-06T07:20:51.887454+0000","last_active":"2024-03-06T07:21:01.419570+0000","last_peered":"2024-03-06T07:21:01.419570+0000","last_clean":"2024-03-06T07:21:01.419570+0000","last_became_active":"2024-03-06T07:20:51.887188+0000","last_became_peered":"2024-03-06T07:20:51.887188+0000","last_unstale":"2024-03-06T07:21:01.419570+0000","last_undegraded":"2024-03-06T07:21:01.419570+0000","last_fullsized":"2024-03-06T07:21:01.419570+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425332+0000","last_change":"2024-03-06T07:21:01.425332+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425332+0000","last_undegraded":"2024-03-06T07:21:01.425332+0000","last_fullsized":"2024-03-06T07:21:01.425332+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430984+0000","last_change":"2024-03-06T07:20:51.899932+0000","last_active":"2024-03-06T07:21:01.430984+0000","last_peered":"2024-03-06T07:21:01.430984+0000","last_clean":"2024-03-06T07:21:01.430984+0000","last_became_active":"2024-03-06T07:20:51.899569+0000","last_became_peered":"2024-03-06T07:20:51.899569+0000","last_unstale":"2024-03-06T07:21:01.430984+0000","last_undegraded":"2024-03-06T07:21:01.430984+0000","last_fullsized":"2024-03-06T07:21:01.430984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429450+0000","last_change":"2024-03-06T07:21:01.429450+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429450+0000","last_undegraded":"2024-03-06T07:21:01.429450+0000","last_fullsized":"2024-03-06T07:21:01.429450+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432122+0000","last_change":"2024-03-06T07:20:51.888537+0000","last_active":"2024-03-06T07:21:01.432122+0000","last_peered":"2024-03-06T07:21:01.432122+0000","last_clean":"2024-03-06T07:21:01.432122+0000","last_became_active":"2024-03-06T07:20:51.888281+0000","last_became_peered":"2024-03-06T07:20:51.888281+0000","last_unstale":"2024-03-06T07:21:01.432122+0000","last_undegraded":"2024-03-06T07:21:01.432122+0000","last_fullsized":"2024-03-06T07:21:01.432122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:03.657994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424979+0000","last_change":"2024-03-06T07:21:01.424979+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424979+0000","last_undegraded":"2024-03-06T07:21:01.424979+0000","last_fullsized":"2024-03-06T07:21:01.424979+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430775+0000","last_change":"2024-03-06T07:20:51.896158+0000","last_active":"2024-03-06T07:21:01.430775+0000","last_peered":"2024-03-06T07:21:01.430775+0000","last_clean":"2024-03-06T07:21:01.430775+0000","last_became_active":"2024-03-06T07:20:51.895889+0000","last_became_peered":"2024-03-06T07:20:51.895889+0000","last_unstale":"2024-03-06T07:21:01.430775+0000","last_undegraded":"2024-03-06T07:21:01.430775+0000","last_fullsized":"2024-03-06T07:21:01.430775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424138+0000","last_change":"2024-03-06T07:21:01.424138+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424138+0000","last_undegraded":"2024-03-06T07:21:01.424138+0000","last_fullsized":"2024-03-06T07:21:01.424138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440853+0000","last_change":"2024-03-06T07:21:01.440204+0000","last_active":"2024-03-06T07:21:01.440853+0000","last_peered":"2024-03-06T07:21:01.440853+0000","last_clean":"2024-03-06T07:21:01.440853+0000","last_became_active":"2024-03-06T07:21:01.439934+0000","last_became_peered":"2024-03-06T07:21:01.439934+0000","last_unstale":"2024-03-06T07:21:01.440853+0000","last_undegraded":"2024-03-06T07:21:01.440853+0000","last_fullsized":"2024-03-06T07:21:01.440853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:52.584141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424952+0000","last_change":"2024-03-06T07:20:51.876393+0000","last_active":"2024-03-06T07:21:01.424952+0000","last_peered":"2024-03-06T07:21:01.424952+0000","last_clean":"2024-03-06T07:21:01.424952+0000","last_became_active":"2024-03-06T07:20:51.876059+0000","last_became_peered":"2024-03-06T07:20:51.876059+0000","last_unstale":"2024-03-06T07:21:01.424952+0000","last_undegraded":"2024-03-06T07:21:01.424952+0000","last_fullsized":"2024-03-06T07:21:01.424952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426852+0000","last_change":"2024-03-06T07:20:50.879586+0000","last_active":"2024-03-06T07:21:01.426852+0000","last_peered":"2024-03-06T07:21:01.426852+0000","last_clean":"2024-03-06T07:21:01.426852+0000","last_became_active":"2024-03-06T07:20:50.879319+0000","last_became_peered":"2024-03-06T07:20:50.879319+0000","last_unstale":"2024-03-06T07:21:01.426852+0000","last_undegraded":"2024-03-06T07:21:01.426852+0000","last_fullsized":"2024-03-06T07:21:01.426852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:58.810754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424424+0000","last_change":"2024-03-06T07:21:01.424424+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424424+0000","last_undegraded":"2024-03-06T07:21:01.424424+0000","last_fullsized":"2024-03-06T07:21:01.424424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438871+0000","last_change":"2024-03-06T07:21:01.436192+0000","last_active":"2024-03-06T07:21:01.438871+0000","last_peered":"2024-03-06T07:21:01.438871+0000","last_clean":"2024-03-06T07:21:01.438871+0000","last_became_active":"2024-03-06T07:21:01.435924+0000","last_became_peered":"2024-03-06T07:21:01.435924+0000","last_unstale":"2024-03-06T07:21:01.438871+0000","last_undegraded":"2024-03-06T07:21:01.438871+0000","last_fullsized":"2024-03-06T07:21:01.438871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419269+0000","last_change":"2024-03-06T07:20:51.884046+0000","last_active":"2024-03-06T07:21:01.419269+0000","last_peered":"2024-03-06T07:21:01.419269+0000","last_clean":"2024-03-06T07:21:01.419269+0000","last_became_active":"2024-03-06T07:20:51.883593+0000","last_became_peered":"2024-03-06T07:20:51.883593+0000","last_unstale":"2024-03-06T07:21:01.419269+0000","last_undegraded":"2024-03-06T07:21:01.419269+0000","last_fullsized":"2024-03-06T07:21:01.419269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:21.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431733+0000","last_change":"2024-03-06T07:20:50.888715+0000","last_active":"2024-03-06T07:21:01.431733+0000","last_peered":"2024-03-06T07:21:01.431733+0000","last_clean":"2024-03-06T07:21:01.431733+0000","last_became_active":"2024-03-06T07:20:50.888140+0000","last_became_peered":"2024-03-06T07:20:50.888140+0000","last_unstale":"2024-03-06T07:21:01.431733+0000","last_undegraded":"2024-03-06T07:21:01.431733+0000","last_fullsized":"2024-03-06T07:21:01.431733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:34.087293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424260+0000","last_change":"2024-03-06T07:21:01.424260+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424260+0000","last_undegraded":"2024-03-06T07:21:01.424260+0000","last_fullsized":"2024-03-06T07:21:01.424260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438971+0000","last_change":"2024-03-06T07:21:01.435195+0000","last_active":"2024-03-06T07:21:01.438971+0000","last_peered":"2024-03-06T07:21:01.438971+0000","last_clean":"2024-03-06T07:21:01.438971+0000","last_became_active":"2024-03-06T07:21:01.434964+0000","last_became_peered":"2024-03-06T07:21:01.434964+0000","last_unstale":"2024-03-06T07:21:01.438971+0000","last_undegraded":"2024-03-06T07:21:01.438971+0000","last_fullsized":"2024-03-06T07:21:01.438971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:25.506818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426735+0000","last_change":"2024-03-06T07:20:51.886764+0000","last_active":"2024-03-06T07:21:01.426735+0000","last_peered":"2024-03-06T07:21:01.426735+0000","last_clean":"2024-03-06T07:21:01.426735+0000","last_became_active":"2024-03-06T07:20:51.886187+0000","last_became_peered":"2024-03-06T07:20:51.886187+0000","last_unstale":"2024-03-06T07:21:01.426735+0000","last_undegraded":"2024-03-06T07:21:01.426735+0000","last_fullsized":"2024-03-06T07:21:01.426735+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:35.564191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432695+0000","last_change":"2024-03-06T07:20:50.885147+0000","last_active":"2024-03-06T07:21:01.432695+0000","last_peered":"2024-03-06T07:21:01.432695+0000","last_clean":"2024-03-06T07:21:01.432695+0000","last_became_active":"2024-03-06T07:20:50.884811+0000","last_became_peered":"2024-03-06T07:20:50.884811+0000","last_unstale":"2024-03-06T07:21:01.432695+0000","last_undegraded":"2024-03-06T07:21:01.432695+0000","last_fullsized":"2024-03-06T07:21:01.432695+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:24.667879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422765+0000","last_change":"2024-03-06T07:21:01.422765+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422765+0000","last_undegraded":"2024-03-06T07:21:01.422765+0000","last_fullsized":"2024-03-06T07:21:01.422765+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442369+0000","last_change":"2024-03-06T07:21:01.439913+0000","last_active":"2024-03-06T07:21:01.442369+0000","last_peered":"2024-03-06T07:21:01.442369+0000","last_clean":"2024-03-06T07:21:01.442369+0000","last_became_active":"2024-03-06T07:21:01.438882+0000","last_became_peered":"2024-03-06T07:21:01.438882+0000","last_unstale":"2024-03-06T07:21:01.442369+0000","last_undegraded":"2024-03-06T07:21:01.442369+0000","last_fullsized":"2024-03-06T07:21:01.442369+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:45.177307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427034+0000","last_change":"2024-03-06T07:20:51.887534+0000","last_active":"2024-03-06T07:21:01.427034+0000","last_peered":"2024-03-06T07:21:01.427034+0000","last_clean":"2024-03-06T07:21:01.427034+0000","last_became_active":"2024-03-06T07:20:51.887239+0000","last_became_peered":"2024-03-06T07:20:51.887239+0000","last_unstale":"2024-03-06T07:21:01.427034+0000","last_undegraded":"2024-03-06T07:21:01.427034+0000","last_fullsized":"2024-03-06T07:21:01.427034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425754+0000","last_change":"2024-03-06T07:20:50.892916+0000","last_active":"2024-03-06T07:21:01.425754+0000","last_peered":"2024-03-06T07:21:01.425754+0000","last_clean":"2024-03-06T07:21:01.425754+0000","last_became_active":"2024-03-06T07:20:50.892629+0000","last_became_peered":"2024-03-06T07:20:50.892629+0000","last_unstale":"2024-03-06T07:21:01.425754+0000","last_undegraded":"2024-03-06T07:21:01.425754+0000","last_fullsized":"2024-03-06T07:21:01.425754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:12.808948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429826+0000","last_change":"2024-03-06T07:21:01.429826+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429826+0000","last_undegraded":"2024-03-06T07:21:01.429826+0000","last_fullsized":"2024-03-06T07:21:01.429826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.439996+0000","last_change":"2024-03-06T07:21:01.439106+0000","last_active":"2024-03-06T07:21:01.439996+0000","last_peered":"2024-03-06T07:21:01.439996+0000","last_clean":"2024-03-06T07:21:01.439996+0000","last_became_active":"2024-03-06T07:21:01.438835+0000","last_became_peered":"2024-03-06T07:21:01.438835+0000","last_unstale":"2024-03-06T07:21:01.439996+0000","last_undegraded":"2024-03-06T07:21:01.439996+0000","last_fullsized":"2024-03-06T07:21:01.439996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419852+0000","last_change":"2024-03-06T07:20:51.895988+0000","last_active":"2024-03-06T07:21:01.419852+0000","last_peered":"2024-03-06T07:21:01.419852+0000","last_clean":"2024-03-06T07:21:01.419852+0000","last_became_active":"2024-03-06T07:20:51.895747+0000","last_became_peered":"2024-03-06T07:20:51.895747+0000","last_unstale":"2024-03-06T07:21:01.419852+0000","last_undegraded":"2024-03-06T07:21:01.419852+0000","last_fullsized":"2024-03-06T07:21:01.419852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427081+0000","last_change":"2024-03-06T07:20:50.893482+0000","last_active":"2024-03-06T07:21:01.427081+0000","last_peered":"2024-03-06T07:21:01.427081+0000","last_clean":"2024-03-06T07:21:01.427081+0000","last_became_active":"2024-03-06T07:20:50.893153+0000","last_became_peered":"2024-03-06T07:20:50.893153+0000","last_unstale":"2024-03-06T07:21:01.427081+0000","last_undegraded":"2024-03-06T07:21:01.427081+0000","last_fullsized":"2024-03-06T07:21:01.427081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:07.775768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424875+0000","last_change":"2024-03-06T07:21:01.424875+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424875+0000","last_undegraded":"2024-03-06T07:21:01.424875+0000","last_fullsized":"2024-03-06T07:21:01.424875+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446459+0000","last_change":"2024-03-06T07:21:01.442216+0000","last_active":"2024-03-06T07:21:01.446459+0000","last_peered":"2024-03-06T07:21:01.446459+0000","last_clean":"2024-03-06T07:21:01.446459+0000","last_became_active":"2024-03-06T07:21:01.441772+0000","last_became_peered":"2024-03-06T07:21:01.441772+0000","last_unstale":"2024-03-06T07:21:01.446459+0000","last_undegraded":"2024-03-06T07:21:01.446459+0000","last_fullsized":"2024-03-06T07:21:01.446459+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:44.646869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.420032+0000","last_change":"2024-03-06T07:20:51.897529+0000","last_active":"2024-03-06T07:21:01.420032+0000","last_peered":"2024-03-06T07:21:01.420032+0000","last_clean":"2024-03-06T07:21:01.420032+0000","last_became_active":"2024-03-06T07:20:51.897181+0000","last_became_peered":"2024-03-06T07:20:51.897181+0000","last_unstale":"2024-03-06T07:21:01.420032+0000","last_undegraded":"2024-03-06T07:21:01.420032+0000","last_fullsized":"2024-03-06T07:21:01.420032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431244+0000","last_change":"2024-03-06T07:20:50.886303+0000","last_active":"2024-03-06T07:21:01.431244+0000","last_peered":"2024-03-06T07:21:01.431244+0000","last_clean":"2024-03-06T07:21:01.431244+0000","last_became_active":"2024-03-06T07:20:50.886013+0000","last_became_peered":"2024-03-06T07:20:50.886013+0000","last_unstale":"2024-03-06T07:21:01.431244+0000","last_undegraded":"2024-03-06T07:21:01.431244+0000","last_fullsized":"2024-03-06T07:21:01.431244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:34.617731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424723+0000","last_change":"2024-03-06T07:21:01.424723+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424723+0000","last_undegraded":"2024-03-06T07:21:01.424723+0000","last_fullsized":"2024-03-06T07:21:01.424723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438261+0000","last_change":"2024-03-06T07:21:01.434722+0000","last_active":"2024-03-06T07:21:01.438261+0000","last_peered":"2024-03-06T07:21:01.438261+0000","last_clean":"2024-03-06T07:21:01.438261+0000","last_became_active":"2024-03-06T07:21:01.434380+0000","last_became_peered":"2024-03-06T07:21:01.434380+0000","last_unstale":"2024-03-06T07:21:01.438261+0000","last_undegraded":"2024-03-06T07:21:01.438261+0000","last_fullsized":"2024-03-06T07:21:01.438261+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427622+0000","last_change":"2024-03-06T07:20:51.883834+0000","last_active":"2024-03-06T07:21:01.427622+0000","last_peered":"2024-03-06T07:21:01.427622+0000","last_clean":"2024-03-06T07:21:01.427622+0000","last_became_active":"2024-03-06T07:20:51.883553+0000","last_became_peered":"2024-03-06T07:20:51.883553+0000","last_unstale":"2024-03-06T07:21:01.427622+0000","last_undegraded":"2024-03-06T07:21:01.427622+0000","last_fullsized":"2024-03-06T07:21:01.427622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419381+0000","last_change":"2024-03-06T07:20:50.881449+0000","last_active":"2024-03-06T07:21:01.419381+0000","last_peered":"2024-03-06T07:21:01.419381+0000","last_clean":"2024-03-06T07:21:01.419381+0000","last_became_active":"2024-03-06T07:20:50.881172+0000","last_became_peered":"2024-03-06T07:20:50.881172+0000","last_unstale":"2024-03-06T07:21:01.419381+0000","last_undegraded":"2024-03-06T07:21:01.419381+0000","last_fullsized":"2024-03-06T07:21:01.419381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.317522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.424583+0000","last_change":"2024-03-06T07:21:01.424583+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.424583+0000","last_undegraded":"2024-03-06T07:21:01.424583+0000","last_fullsized":"2024-03-06T07:21:01.424583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.437982+0000","last_change":"2024-03-06T07:21:01.431035+0000","last_active":"2024-03-06T07:21:01.437982+0000","last_peered":"2024-03-06T07:21:01.437982+0000","last_clean":"2024-03-06T07:21:01.437982+0000","last_became_active":"2024-03-06T07:21:01.430787+0000","last_became_peered":"2024-03-06T07:21:01.430787+0000","last_unstale":"2024-03-06T07:21:01.437982+0000","last_undegraded":"2024-03-06T07:21:01.437982+0000","last_fullsized":"2024-03-06T07:21:01.437982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:08.242986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426138+0000","last_change":"2024-03-06T07:20:51.893411+0000","last_active":"2024-03-06T07:21:01.426138+0000","last_peered":"2024-03-06T07:21:01.426138+0000","last_clean":"2024-03-06T07:21:01.426138+0000","last_became_active":"2024-03-06T07:20:51.893149+0000","last_became_peered":"2024-03-06T07:20:51.893149+0000","last_unstale":"2024-03-06T07:21:01.426138+0000","last_undegraded":"2024-03-06T07:21:01.426138+0000","last_fullsized":"2024-03-06T07:21:01.426138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427928+0000","last_change":"2024-03-06T07:20:50.890159+0000","last_active":"2024-03-06T07:21:01.427928+0000","last_peered":"2024-03-06T07:21:01.427928+0000","last_clean":"2024-03-06T07:21:01.427928+0000","last_became_active":"2024-03-06T07:20:50.889863+0000","last_became_peered":"2024-03-06T07:20:50.889863+0000","last_unstale":"2024-03-06T07:21:01.427928+0000","last_undegraded":"2024-03-06T07:21:01.427928+0000","last_fullsized":"2024-03-06T07:21:01.427928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.303590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425274+0000","last_change":"2024-03-06T07:21:01.425274+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425274+0000","last_undegraded":"2024-03-06T07:21:01.425274+0000","last_fullsized":"2024-03-06T07:21:01.425274+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443004+0000","last_change":"2024-03-06T07:21:01.440880+0000","last_active":"2024-03-06T07:21:01.443004+0000","last_peered":"2024-03-06T07:21:01.443004+0000","last_clean":"2024-03-06T07:21:01.443004+0000","last_became_active":"2024-03-06T07:21:01.440583+0000","last_became_peered":"2024-03-06T07:21:01.440583+0000","last_unstale":"2024-03-06T07:21:01.443004+0000","last_undegraded":"2024-03-06T07:21:01.443004+0000","last_fullsized":"2024-03-06T07:21:01.443004+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419246+0000","last_change":"2024-03-06T07:20:51.899199+0000","last_active":"2024-03-06T07:21:01.419246+0000","last_peered":"2024-03-06T07:21:01.419246+0000","last_clean":"2024-03-06T07:21:01.419246+0000","last_became_active":"2024-03-06T07:20:51.898622+0000","last_became_peered":"2024-03-06T07:20:51.898622+0000","last_unstale":"2024-03-06T07:21:01.419246+0000","last_undegraded":"2024-03-06T07:21:01.419246+0000","last_fullsized":"2024-03-06T07:21:01.419246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431988+0000","last_change":"2024-03-06T07:20:50.891619+0000","last_active":"2024-03-06T07:21:01.431988+0000","last_peered":"2024-03-06T07:21:01.431988+0000","last_clean":"2024-03-06T07:21:01.431988+0000","last_became_active":"2024-03-06T07:20:50.891305+0000","last_became_peered":"2024-03-06T07:20:50.891305+0000","last_unstale":"2024-03-06T07:21:01.431988+0000","last_undegraded":"2024-03-06T07:21:01.431988+0000","last_fullsized":"2024-03-06T07:21:01.431988+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.732044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427634+0000","last_change":"2024-03-06T07:21:01.427634+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427634+0000","last_undegraded":"2024-03-06T07:21:01.427634+0000","last_fullsized":"2024-03-06T07:21:01.427634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442394+0000","last_change":"2024-03-06T07:21:01.439974+0000","last_active":"2024-03-06T07:21:01.442394+0000","last_peered":"2024-03-06T07:21:01.442394+0000","last_clean":"2024-03-06T07:21:01.442394+0000","last_became_active":"2024-03-06T07:21:01.439682+0000","last_became_peered":"2024-03-06T07:21:01.439682+0000","last_unstale":"2024-03-06T07:21:01.442394+0000","last_undegraded":"2024-03-06T07:21:01.442394+0000","last_fullsized":"2024-03-06T07:21:01.442394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431422+0000","last_change":"2024-03-06T07:20:51.887972+0000","last_active":"2024-03-06T07:21:01.431422+0000","last_peered":"2024-03-06T07:21:01.431422+0000","last_clean":"2024-03-06T07:21:01.431422+0000","last_became_active":"2024-03-06T07:20:51.887725+0000","last_became_peered":"2024-03-06T07:20:51.887725+0000","last_unstale":"2024-03-06T07:21:01.431422+0000","last_undegraded":"2024-03-06T07:21:01.431422+0000","last_fullsized":"2024-03-06T07:21:01.431422+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431885+0000","last_change":"2024-03-06T07:20:50.890731+0000","last_active":"2024-03-06T07:21:01.431885+0000","last_peered":"2024-03-06T07:21:01.431885+0000","last_clean":"2024-03-06T07:21:01.431885+0000","last_became_active":"2024-03-06T07:20:50.890429+0000","last_became_peered":"2024-03-06T07:20:50.890429+0000","last_unstale":"2024-03-06T07:21:01.431885+0000","last_undegraded":"2024-03-06T07:21:01.431885+0000","last_fullsized":"2024-03-06T07:21:01.431885+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423808+0000","last_change":"2024-03-06T07:21:01.423808+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423808+0000","last_undegraded":"2024-03-06T07:21:01.423808+0000","last_fullsized":"2024-03-06T07:21:01.423808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438843+0000","last_change":"2024-03-06T07:21:01.436794+0000","last_active":"2024-03-06T07:21:01.438843+0000","last_peered":"2024-03-06T07:21:01.438843+0000","last_clean":"2024-03-06T07:21:01.438843+0000","last_became_active":"2024-03-06T07:21:01.436556+0000","last_became_peered":"2024-03-06T07:21:01.436556+0000","last_unstale":"2024-03-06T07:21:01.438843+0000","last_undegraded":"2024-03-06T07:21:01.438843+0000","last_fullsized":"2024-03-06T07:21:01.438843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425550+0000","last_change":"2024-03-06T07:20:51.885412+0000","last_active":"2024-03-06T07:21:01.425550+0000","last_peered":"2024-03-06T07:21:01.425550+0000","last_clean":"2024-03-06T07:21:01.425550+0000","last_became_active":"2024-03-06T07:20:51.885153+0000","last_became_peered":"2024-03-06T07:20:51.885153+0000","last_unstale":"2024-03-06T07:21:01.425550+0000","last_undegraded":"2024-03-06T07:21:01.425550+0000","last_fullsized":"2024-03-06T07:21:01.425550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:07.592958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427582+0000","last_change":"2024-03-06T07:20:50.880143+0000","last_active":"2024-03-06T07:21:01.427582+0000","last_peered":"2024-03-06T07:21:01.427582+0000","last_clean":"2024-03-06T07:21:01.427582+0000","last_became_active":"2024-03-06T07:20:50.879859+0000","last_became_peered":"2024-03-06T07:20:50.879859+0000","last_unstale":"2024-03-06T07:21:01.427582+0000","last_undegraded":"2024-03-06T07:21:01.427582+0000","last_fullsized":"2024-03-06T07:21:01.427582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:08.684501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423462+0000","last_change":"2024-03-06T07:21:01.423462+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423462+0000","last_undegraded":"2024-03-06T07:21:01.423462+0000","last_fullsized":"2024-03-06T07:21:01.423462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433111+0000","last_change":"2024-03-06T07:21:01.432460+0000","last_active":"2024-03-06T07:21:01.433111+0000","last_peered":"2024-03-06T07:21:01.433111+0000","last_clean":"2024-03-06T07:21:01.433111+0000","last_became_active":"2024-03-06T07:21:01.432169+0000","last_became_peered":"2024-03-06T07:21:01.432169+0000","last_unstale":"2024-03-06T07:21:01.433111+0000","last_undegraded":"2024-03-06T07:21:01.433111+0000","last_fullsized":"2024-03-06T07:21:01.433111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427116+0000","last_change":"2024-03-06T07:20:51.878936+0000","last_active":"2024-03-06T07:21:01.427116+0000","last_peered":"2024-03-06T07:21:01.427116+0000","last_clean":"2024-03-06T07:21:01.427116+0000","last_became_active":"2024-03-06T07:20:51.878613+0000","last_became_peered":"2024-03-06T07:20:51.878613+0000","last_unstale":"2024-03-06T07:21:01.427116+0000","last_undegraded":"2024-03-06T07:21:01.427116+0000","last_fullsized":"2024-03-06T07:21:01.427116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":24,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425845+0000","last_change":"2024-03-06T07:20:50.888110+0000","last_active":"2024-03-06T07:21:01.425845+0000","last_peered":"2024-03-06T07:21:01.425845+0000","last_clean":"2024-03-06T07:21:01.425845+0000","last_became_active":"2024-03-06T07:20:50.887804+0000","last_became_peered":"2024-03-06T07:20:50.887804+0000","last_unstale":"2024-03-06T07:21:01.425845+0000","last_undegraded":"2024-03-06T07:21:01.425845+0000","last_fullsized":"2024-03-06T07:21:01.425845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423329+0000","last_change":"2024-03-06T07:21:01.423329+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423329+0000","last_undegraded":"2024-03-06T07:21:01.423329+0000","last_fullsized":"2024-03-06T07:21:01.423329+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438955+0000","last_change":"2024-03-06T07:21:01.433077+0000","last_active":"2024-03-06T07:21:01.438955+0000","last_peered":"2024-03-06T07:21:01.438955+0000","last_clean":"2024-03-06T07:21:01.438955+0000","last_became_active":"2024-03-06T07:21:01.432684+0000","last_became_peered":"2024-03-06T07:21:01.432684+0000","last_unstale":"2024-03-06T07:21:01.438955+0000","last_undegraded":"2024-03-06T07:21:01.438955+0000","last_fullsized":"2024-03-06T07:21:01.438955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:13.212500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431469+0000","last_change":"2024-03-06T07:20:51.891149+0000","last_active":"2024-03-06T07:21:01.431469+0000","last_peered":"2024-03-06T07:21:01.431469+0000","last_clean":"2024-03-06T07:21:01.431469+0000","last_became_active":"2024-03-06T07:20:51.890877+0000","last_became_peered":"2024-03-06T07:20:51.890877+0000","last_unstale":"2024-03-06T07:21:01.431469+0000","last_undegraded":"2024-03-06T07:21:01.431469+0000","last_fullsized":"2024-03-06T07:21:01.431469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431369+0000","last_change":"2024-03-06T07:20:50.888227+0000","last_active":"2024-03-06T07:21:01.431369+0000","last_peered":"2024-03-06T07:21:01.431369+0000","last_clean":"2024-03-06T07:21:01.431369+0000","last_became_active":"2024-03-06T07:20:50.887926+0000","last_became_peered":"2024-03-06T07:20:50.887926+0000","last_unstale":"2024-03-06T07:21:01.431369+0000","last_undegraded":"2024-03-06T07:21:01.431369+0000","last_fullsized":"2024-03-06T07:21:01.431369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:56.255517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.425722+0000","last_change":"2024-03-06T07:21:01.425722+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.425722+0000","last_undegraded":"2024-03-06T07:21:01.425722+0000","last_fullsized":"2024-03-06T07:21:01.425722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.433380+0000","last_change":"2024-03-06T07:21:01.432947+0000","last_active":"2024-03-06T07:21:01.433380+0000","last_peered":"2024-03-06T07:21:01.433380+0000","last_clean":"2024-03-06T07:21:01.433380+0000","last_became_active":"2024-03-06T07:21:01.432263+0000","last_became_peered":"2024-03-06T07:21:01.432263+0000","last_unstale":"2024-03-06T07:21:01.433380+0000","last_undegraded":"2024-03-06T07:21:01.433380+0000","last_fullsized":"2024-03-06T07:21:01.433380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425616+0000","last_change":"2024-03-06T07:20:51.888556+0000","last_active":"2024-03-06T07:21:01.425616+0000","last_peered":"2024-03-06T07:21:01.425616+0000","last_clean":"2024-03-06T07:21:01.425616+0000","last_became_active":"2024-03-06T07:20:51.888278+0000","last_became_peered":"2024-03-06T07:20:51.888278+0000","last_unstale":"2024-03-06T07:21:01.425616+0000","last_undegraded":"2024-03-06T07:21:01.425616+0000","last_fullsized":"2024-03-06T07:21:01.425616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431076+0000","last_change":"2024-03-06T07:20:50.881110+0000","last_active":"2024-03-06T07:21:01.431076+0000","last_peered":"2024-03-06T07:21:01.431076+0000","last_clean":"2024-03-06T07:21:01.431076+0000","last_became_active":"2024-03-06T07:20:50.880752+0000","last_became_peered":"2024-03-06T07:20:50.880752+0000","last_unstale":"2024-03-06T07:21:01.431076+0000","last_undegraded":"2024-03-06T07:21:01.431076+0000","last_fullsized":"2024-03-06T07:21:01.431076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423587+0000","last_change":"2024-03-06T07:21:01.423587+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423587+0000","last_undegraded":"2024-03-06T07:21:01.423587+0000","last_fullsized":"2024-03-06T07:21:01.423587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438705+0000","last_change":"2024-03-06T07:21:01.435628+0000","last_active":"2024-03-06T07:21:01.438705+0000","last_peered":"2024-03-06T07:21:01.438705+0000","last_clean":"2024-03-06T07:21:01.438705+0000","last_became_active":"2024-03-06T07:21:01.434996+0000","last_became_peered":"2024-03-06T07:21:01.434996+0000","last_unstale":"2024-03-06T07:21:01.438705+0000","last_undegraded":"2024-03-06T07:21:01.438705+0000","last_fullsized":"2024-03-06T07:21:01.438705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:21.526480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425994+0000","last_change":"2024-03-06T07:20:51.879914+0000","last_active":"2024-03-06T07:21:01.425994+0000","last_peered":"2024-03-06T07:21:01.425994+0000","last_clean":"2024-03-06T07:21:01.425994+0000","last_became_active":"2024-03-06T07:20:51.879633+0000","last_became_peered":"2024-03-06T07:20:51.879633+0000","last_unstale":"2024-03-06T07:21:01.425994+0000","last_undegraded":"2024-03-06T07:21:01.425994+0000","last_fullsized":"2024-03-06T07:21:01.425994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:36.738962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424764+0000","last_change":"2024-03-06T07:20:50.880559+0000","last_active":"2024-03-06T07:21:01.424764+0000","last_peered":"2024-03-06T07:21:01.424764+0000","last_clean":"2024-03-06T07:21:01.424764+0000","last_became_active":"2024-03-06T07:20:50.880160+0000","last_became_peered":"2024-03-06T07:20:50.880160+0000","last_unstale":"2024-03-06T07:21:01.424764+0000","last_undegraded":"2024-03-06T07:21:01.424764+0000","last_fullsized":"2024-03-06T07:21:01.424764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:56.900930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.430090+0000","last_change":"2024-03-06T07:21:01.430090+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.430090+0000","last_undegraded":"2024-03-06T07:21:01.430090+0000","last_fullsized":"2024-03-06T07:21:01.430090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442699+0000","last_change":"2024-03-06T07:21:01.441770+0000","last_active":"2024-03-06T07:21:01.442699+0000","last_peered":"2024-03-06T07:21:01.442699+0000","last_clean":"2024-03-06T07:21:01.442699+0000","last_became_active":"2024-03-06T07:21:01.441478+0000","last_became_peered":"2024-03-06T07:21:01.441478+0000","last_unstale":"2024-03-06T07:21:01.442699+0000","last_undegraded":"2024-03-06T07:21:01.442699+0000","last_fullsized":"2024-03-06T07:21:01.442699+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:10.768407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426237+0000","last_change":"2024-03-06T07:20:51.890440+0000","last_active":"2024-03-06T07:21:01.426237+0000","last_peered":"2024-03-06T07:21:01.426237+0000","last_clean":"2024-03-06T07:21:01.426237+0000","last_became_active":"2024-03-06T07:20:51.890167+0000","last_became_peered":"2024-03-06T07:20:51.890167+0000","last_unstale":"2024-03-06T07:21:01.426237+0000","last_undegraded":"2024-03-06T07:21:01.426237+0000","last_fullsized":"2024-03-06T07:21:01.426237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:53.956654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":26,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419383+0000","last_change":"2024-03-06T07:20:50.885352+0000","last_active":"2024-03-06T07:21:01.419383+0000","last_peered":"2024-03-06T07:21:01.419383+0000","last_clean":"2024-03-06T07:21:01.419383+0000","last_became_active":"2024-03-06T07:20:50.884991+0000","last_became_peered":"2024-03-06T07:20:50.884991+0000","last_unstale":"2024-03-06T07:21:01.419383+0000","last_undegraded":"2024-03-06T07:21:01.419383+0000","last_fullsized":"2024-03-06T07:21:01.419383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:03.652296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427525+0000","last_change":"2024-03-06T07:21:01.427525+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427525+0000","last_undegraded":"2024-03-06T07:21:01.427525+0000","last_fullsized":"2024-03-06T07:21:01.427525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438733+0000","last_change":"2024-03-06T07:21:01.436096+0000","last_active":"2024-03-06T07:21:01.438733+0000","last_peered":"2024-03-06T07:21:01.438733+0000","last_clean":"2024-03-06T07:21:01.438733+0000","last_became_active":"2024-03-06T07:21:01.435767+0000","last_became_peered":"2024-03-06T07:21:01.435767+0000","last_unstale":"2024-03-06T07:21:01.438733+0000","last_undegraded":"2024-03-06T07:21:01.438733+0000","last_fullsized":"2024-03-06T07:21:01.438733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:40.907133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427258+0000","last_change":"2024-03-06T07:20:51.889501+0000","last_active":"2024-03-06T07:21:01.427258+0000","last_peered":"2024-03-06T07:21:01.427258+0000","last_clean":"2024-03-06T07:21:01.427258+0000","last_became_active":"2024-03-06T07:20:51.889206+0000","last_became_peered":"2024-03-06T07:20:51.889206+0000","last_unstale":"2024-03-06T07:21:01.427258+0000","last_undegraded":"2024-03-06T07:21:01.427258+0000","last_fullsized":"2024-03-06T07:21:01.427258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:09.908900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432085+0000","last_change":"2024-03-06T07:20:50.883039+0000","last_active":"2024-03-06T07:21:01.432085+0000","last_peered":"2024-03-06T07:21:01.432085+0000","last_clean":"2024-03-06T07:21:01.432085+0000","last_became_active":"2024-03-06T07:20:50.882748+0000","last_became_peered":"2024-03-06T07:20:50.882748+0000","last_unstale":"2024-03-06T07:21:01.432085+0000","last_undegraded":"2024-03-06T07:21:01.432085+0000","last_fullsized":"2024-03-06T07:21:01.432085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:35.868841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423288+0000","last_change":"2024-03-06T07:21:01.423288+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423288+0000","last_undegraded":"2024-03-06T07:21:01.423288+0000","last_fullsized":"2024-03-06T07:21:01.423288+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440065+0000","last_change":"2024-03-06T07:21:01.438861+0000","last_active":"2024-03-06T07:21:01.440065+0000","last_peered":"2024-03-06T07:21:01.440065+0000","last_clean":"2024-03-06T07:21:01.440065+0000","last_became_active":"2024-03-06T07:21:01.438577+0000","last_became_peered":"2024-03-06T07:21:01.438577+0000","last_unstale":"2024-03-06T07:21:01.440065+0000","last_undegraded":"2024-03-06T07:21:01.440065+0000","last_fullsized":"2024-03-06T07:21:01.440065+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.335344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424200+0000","last_change":"2024-03-06T07:20:51.890860+0000","last_active":"2024-03-06T07:21:01.424200+0000","last_peered":"2024-03-06T07:21:01.424200+0000","last_clean":"2024-03-06T07:21:01.424200+0000","last_became_active":"2024-03-06T07:20:51.890576+0000","last_became_peered":"2024-03-06T07:20:51.890576+0000","last_unstale":"2024-03-06T07:21:01.424200+0000","last_undegraded":"2024-03-06T07:21:01.424200+0000","last_fullsized":"2024-03-06T07:21:01.424200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:02.364180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427717+0000","last_change":"2024-03-06T07:20:50.888701+0000","last_active":"2024-03-06T07:21:01.427717+0000","last_peered":"2024-03-06T07:21:01.427717+0000","last_clean":"2024-03-06T07:21:01.427717+0000","last_became_active":"2024-03-06T07:20:50.888392+0000","last_became_peered":"2024-03-06T07:20:50.888392+0000","last_unstale":"2024-03-06T07:21:01.427717+0000","last_undegraded":"2024-03-06T07:21:01.427717+0000","last_fullsized":"2024-03-06T07:21:01.427717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.582638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422966+0000","last_change":"2024-03-06T07:21:01.422966+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422966+0000","last_undegraded":"2024-03-06T07:21:01.422966+0000","last_fullsized":"2024-03-06T07:21:01.422966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438788+0000","last_change":"2024-03-06T07:21:01.436497+0000","last_active":"2024-03-06T07:21:01.438788+0000","last_peered":"2024-03-06T07:21:01.438788+0000","last_clean":"2024-03-06T07:21:01.438788+0000","last_became_active":"2024-03-06T07:21:01.436029+0000","last_became_peered":"2024-03-06T07:21:01.436029+0000","last_unstale":"2024-03-06T07:21:01.438788+0000","last_undegraded":"2024-03-06T07:21:01.438788+0000","last_fullsized":"2024-03-06T07:21:01.438788+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:36.166225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425488+0000","last_change":"2024-03-06T07:20:51.887239+0000","last_active":"2024-03-06T07:21:01.425488+0000","last_peered":"2024-03-06T07:21:01.425488+0000","last_clean":"2024-03-06T07:21:01.425488+0000","last_became_active":"2024-03-06T07:20:51.886940+0000","last_became_peered":"2024-03-06T07:20:51.886940+0000","last_unstale":"2024-03-06T07:21:01.425488+0000","last_undegraded":"2024-03-06T07:21:01.425488+0000","last_fullsized":"2024-03-06T07:21:01.425488+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:08.123116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427698+0000","last_change":"2024-03-06T07:20:50.886879+0000","last_active":"2024-03-06T07:21:01.427698+0000","last_peered":"2024-03-06T07:21:01.427698+0000","last_clean":"2024-03-06T07:21:01.427698+0000","last_became_active":"2024-03-06T07:20:50.886589+0000","last_became_peered":"2024-03-06T07:20:50.886589+0000","last_unstale":"2024-03-06T07:21:01.427698+0000","last_undegraded":"2024-03-06T07:21:01.427698+0000","last_fullsized":"2024-03-06T07:21:01.427698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:51.245367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.428740+0000","last_change":"2024-03-06T07:21:01.428740+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.428740+0000","last_undegraded":"2024-03-06T07:21:01.428740+0000","last_fullsized":"2024-03-06T07:21:01.428740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.443067+0000","last_change":"2024-03-06T07:21:01.442218+0000","last_active":"2024-03-06T07:21:01.443067+0000","last_peered":"2024-03-06T07:21:01.443067+0000","last_clean":"2024-03-06T07:21:01.443067+0000","last_became_active":"2024-03-06T07:21:01.441776+0000","last_became_peered":"2024-03-06T07:21:01.441776+0000","last_unstale":"2024-03-06T07:21:01.443067+0000","last_undegraded":"2024-03-06T07:21:01.443067+0000","last_fullsized":"2024-03-06T07:21:01.443067+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:26.445135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424527+0000","last_change":"2024-03-06T07:20:51.892549+0000","last_active":"2024-03-06T07:21:01.424527+0000","last_peered":"2024-03-06T07:21:01.424527+0000","last_clean":"2024-03-06T07:21:01.424527+0000","last_became_active":"2024-03-06T07:20:51.889936+0000","last_became_peered":"2024-03-06T07:20:51.889936+0000","last_unstale":"2024-03-06T07:21:01.424527+0000","last_undegraded":"2024-03-06T07:21:01.424527+0000","last_fullsized":"2024-03-06T07:21:01.424527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:23.986390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430588+0000","last_change":"2024-03-06T07:20:50.883071+0000","last_active":"2024-03-06T07:21:01.430588+0000","last_peered":"2024-03-06T07:21:01.430588+0000","last_clean":"2024-03-06T07:21:01.430588+0000","last_became_active":"2024-03-06T07:20:50.882465+0000","last_became_peered":"2024-03-06T07:20:50.882465+0000","last_unstale":"2024-03-06T07:21:01.430588+0000","last_undegraded":"2024-03-06T07:21:01.430588+0000","last_fullsized":"2024-03-06T07:21:01.430588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423784+0000","last_change":"2024-03-06T07:21:01.423784+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423784+0000","last_undegraded":"2024-03-06T07:21:01.423784+0000","last_fullsized":"2024-03-06T07:21:01.423784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.444053+0000","last_change":"2024-03-06T07:21:01.441755+0000","last_active":"2024-03-06T07:21:01.444053+0000","last_peered":"2024-03-06T07:21:01.444053+0000","last_clean":"2024-03-06T07:21:01.444053+0000","last_became_active":"2024-03-06T07:21:01.441485+0000","last_became_peered":"2024-03-06T07:21:01.441485+0000","last_unstale":"2024-03-06T07:21:01.444053+0000","last_undegraded":"2024-03-06T07:21:01.444053+0000","last_fullsized":"2024-03-06T07:21:01.444053+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:14.211872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419859+0000","last_change":"2024-03-06T07:20:51.884694+0000","last_active":"2024-03-06T07:21:01.419859+0000","last_peered":"2024-03-06T07:21:01.419859+0000","last_clean":"2024-03-06T07:21:01.419859+0000","last_became_active":"2024-03-06T07:20:51.883905+0000","last_became_peered":"2024-03-06T07:20:51.883905+0000","last_unstale":"2024-03-06T07:21:01.419859+0000","last_undegraded":"2024-03-06T07:21:01.419859+0000","last_fullsized":"2024-03-06T07:21:01.419859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.424620+0000","last_change":"2024-03-06T07:20:50.881489+0000","last_active":"2024-03-06T07:21:01.424620+0000","last_peered":"2024-03-06T07:21:01.424620+0000","last_clean":"2024-03-06T07:21:01.424620+0000","last_became_active":"2024-03-06T07:20:50.881182+0000","last_became_peered":"2024-03-06T07:20:50.881182+0000","last_unstale":"2024-03-06T07:21:01.424620+0000","last_undegraded":"2024-03-06T07:21:01.424620+0000","last_fullsized":"2024-03-06T07:21:01.424620+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:22.518381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422551+0000","last_change":"2024-03-06T07:21:01.422551+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422551+0000","last_undegraded":"2024-03-06T07:21:01.422551+0000","last_fullsized":"2024-03-06T07:21:01.422551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438240+0000","last_change":"2024-03-06T07:21:01.434477+0000","last_active":"2024-03-06T07:21:01.438240+0000","last_peered":"2024-03-06T07:21:01.438240+0000","last_clean":"2024-03-06T07:21:01.438240+0000","last_became_active":"2024-03-06T07:21:01.434205+0000","last_became_peered":"2024-03-06T07:21:01.434205+0000","last_unstale":"2024-03-06T07:21:01.438240+0000","last_undegraded":"2024-03-06T07:21:01.438240+0000","last_fullsized":"2024-03-06T07:21:01.438240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:50.474090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419708+0000","last_change":"2024-03-06T07:20:51.889352+0000","last_active":"2024-03-06T07:21:01.419708+0000","last_peered":"2024-03-06T07:21:01.419708+0000","last_clean":"2024-03-06T07:21:01.419708+0000","last_became_active":"2024-03-06T07:20:51.888464+0000","last_became_peered":"2024-03-06T07:20:51.888464+0000","last_unstale":"2024-03-06T07:21:01.419708+0000","last_undegraded":"2024-03-06T07:21:01.419708+0000","last_fullsized":"2024-03-06T07:21:01.419708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425015+0000","last_change":"2024-03-06T07:20:50.887415+0000","last_active":"2024-03-06T07:21:01.425015+0000","last_peered":"2024-03-06T07:21:01.425015+0000","last_clean":"2024-03-06T07:21:01.425015+0000","last_became_active":"2024-03-06T07:20:50.887137+0000","last_became_peered":"2024-03-06T07:20:50.887137+0000","last_unstale":"2024-03-06T07:21:01.425015+0000","last_undegraded":"2024-03-06T07:21:01.425015+0000","last_fullsized":"2024-03-06T07:21:01.425015+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:54.210267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422909+0000","last_change":"2024-03-06T07:21:01.422909+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422909+0000","last_undegraded":"2024-03-06T07:21:01.422909+0000","last_fullsized":"2024-03-06T07:21:01.422909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441013+0000","last_change":"2024-03-06T07:21:01.439736+0000","last_active":"2024-03-06T07:21:01.441013+0000","last_peered":"2024-03-06T07:21:01.441013+0000","last_clean":"2024-03-06T07:21:01.441013+0000","last_became_active":"2024-03-06T07:21:01.439473+0000","last_became_peered":"2024-03-06T07:21:01.439473+0000","last_unstale":"2024-03-06T07:21:01.441013+0000","last_undegraded":"2024-03-06T07:21:01.441013+0000","last_fullsized":"2024-03-06T07:21:01.441013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:38.195559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426346+0000","last_change":"2024-03-06T07:20:51.885672+0000","last_active":"2024-03-06T07:21:01.426346+0000","last_peered":"2024-03-06T07:21:01.426346+0000","last_clean":"2024-03-06T07:21:01.426346+0000","last_became_active":"2024-03-06T07:20:51.885412+0000","last_became_peered":"2024-03-06T07:20:51.885412+0000","last_unstale":"2024-03-06T07:21:01.426346+0000","last_undegraded":"2024-03-06T07:21:01.426346+0000","last_fullsized":"2024-03-06T07:21:01.426346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:54.467344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.427413+0000","last_change":"2024-03-06T07:20:50.889223+0000","last_active":"2024-03-06T07:21:01.427413+0000","last_peered":"2024-03-06T07:21:01.427413+0000","last_clean":"2024-03-06T07:21:01.427413+0000","last_became_active":"2024-03-06T07:20:50.888934+0000","last_became_peered":"2024-03-06T07:20:50.888934+0000","last_unstale":"2024-03-06T07:21:01.427413+0000","last_undegraded":"2024-03-06T07:21:01.427413+0000","last_fullsized":"2024-03-06T07:21:01.427413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:04.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423927+0000","last_change":"2024-03-06T07:21:01.423927+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423927+0000","last_undegraded":"2024-03-06T07:21:01.423927+0000","last_fullsized":"2024-03-06T07:21:01.423927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438700+0000","last_change":"2024-03-06T07:21:01.435463+0000","last_active":"2024-03-06T07:21:01.438700+0000","last_peered":"2024-03-06T07:21:01.438700+0000","last_clean":"2024-03-06T07:21:01.438700+0000","last_became_active":"2024-03-06T07:21:01.435086+0000","last_became_peered":"2024-03-06T07:21:01.435086+0000","last_unstale":"2024-03-06T07:21:01.438700+0000","last_undegraded":"2024-03-06T07:21:01.438700+0000","last_fullsized":"2024-03-06T07:21:01.438700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:43.973004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426480+0000","last_change":"2024-03-06T07:20:51.880937+0000","last_active":"2024-03-06T07:21:01.426480+0000","last_peered":"2024-03-06T07:21:01.426480+0000","last_clean":"2024-03-06T07:21:01.426480+0000","last_became_active":"2024-03-06T07:20:51.880648+0000","last_became_peered":"2024-03-06T07:20:51.880648+0000","last_unstale":"2024-03-06T07:21:01.426480+0000","last_undegraded":"2024-03-06T07:21:01.426480+0000","last_fullsized":"2024-03-06T07:21:01.426480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432004+0000","last_change":"2024-03-06T07:20:50.886345+0000","last_active":"2024-03-06T07:21:01.432004+0000","last_peered":"2024-03-06T07:21:01.432004+0000","last_clean":"2024-03-06T07:21:01.432004+0000","last_became_active":"2024-03-06T07:20:50.886074+0000","last_became_peered":"2024-03-06T07:20:50.886074+0000","last_unstale":"2024-03-06T07:21:01.432004+0000","last_undegraded":"2024-03-06T07:21:01.432004+0000","last_fullsized":"2024-03-06T07:21:01.432004+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:39.914514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422540+0000","last_change":"2024-03-06T07:21:01.422540+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422540+0000","last_undegraded":"2024-03-06T07:21:01.422540+0000","last_fullsized":"2024-03-06T07:21:01.422540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.446264+0000","last_change":"2024-03-06T07:21:01.443325+0000","last_active":"2024-03-06T07:21:01.446264+0000","last_peered":"2024-03-06T07:21:01.446264+0000","last_clean":"2024-03-06T07:21:01.446264+0000","last_became_active":"2024-03-06T07:21:01.443143+0000","last_became_peered":"2024-03-06T07:21:01.443143+0000","last_unstale":"2024-03-06T07:21:01.446264+0000","last_undegraded":"2024-03-06T07:21:01.446264+0000","last_fullsized":"2024-03-06T07:21:01.446264+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:17.147464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426249+0000","last_change":"2024-03-06T07:20:51.889471+0000","last_active":"2024-03-06T07:21:01.426249+0000","last_peered":"2024-03-06T07:21:01.426249+0000","last_clean":"2024-03-06T07:21:01.426249+0000","last_became_active":"2024-03-06T07:20:51.889218+0000","last_became_peered":"2024-03-06T07:20:51.889218+0000","last_unstale":"2024-03-06T07:21:01.426249+0000","last_undegraded":"2024-03-06T07:21:01.426249+0000","last_fullsized":"2024-03-06T07:21:01.426249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419799+0000","last_change":"2024-03-06T07:20:50.881740+0000","last_active":"2024-03-06T07:21:01.419799+0000","last_peered":"2024-03-06T07:21:01.419799+0000","last_clean":"2024-03-06T07:21:01.419799+0000","last_became_active":"2024-03-06T07:20:50.881306+0000","last_became_peered":"2024-03-06T07:20:50.881306+0000","last_unstale":"2024-03-06T07:21:01.419799+0000","last_undegraded":"2024-03-06T07:21:01.419799+0000","last_fullsized":"2024-03-06T07:21:01.419799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:35.733892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427082+0000","last_change":"2024-03-06T07:21:01.427082+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427082+0000","last_undegraded":"2024-03-06T07:21:01.427082+0000","last_fullsized":"2024-03-06T07:21:01.427082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.436013+0000","last_change":"2024-03-06T07:21:01.435258+0000","last_active":"2024-03-06T07:21:01.436013+0000","last_peered":"2024-03-06T07:21:01.436013+0000","last_clean":"2024-03-06T07:21:01.436013+0000","last_became_active":"2024-03-06T07:21:01.435001+0000","last_became_peered":"2024-03-06T07:21:01.435001+0000","last_unstale":"2024-03-06T07:21:01.436013+0000","last_undegraded":"2024-03-06T07:21:01.436013+0000","last_fullsized":"2024-03-06T07:21:01.436013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:56.128562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426836+0000","last_change":"2024-03-06T07:20:51.886967+0000","last_active":"2024-03-06T07:21:01.426836+0000","last_peered":"2024-03-06T07:21:01.426836+0000","last_clean":"2024-03-06T07:21:01.426836+0000","last_became_active":"2024-03-06T07:20:51.886521+0000","last_became_peered":"2024-03-06T07:20:51.886521+0000","last_unstale":"2024-03-06T07:21:01.426836+0000","last_undegraded":"2024-03-06T07:21:01.426836+0000","last_fullsized":"2024-03-06T07:21:01.426836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:58.399266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432888+0000","last_change":"2024-03-06T07:20:50.885378+0000","last_active":"2024-03-06T07:21:01.432888+0000","last_peered":"2024-03-06T07:21:01.432888+0000","last_clean":"2024-03-06T07:21:01.432888+0000","last_became_active":"2024-03-06T07:20:50.884859+0000","last_became_peered":"2024-03-06T07:20:50.884859+0000","last_unstale":"2024-03-06T07:21:01.432888+0000","last_undegraded":"2024-03-06T07:21:01.432888+0000","last_fullsized":"2024-03-06T07:21:01.432888+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:28.686737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.423705+0000","last_change":"2024-03-06T07:21:01.423705+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.423705+0000","last_undegraded":"2024-03-06T07:21:01.423705+0000","last_fullsized":"2024-03-06T07:21:01.423705+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438756+0000","last_change":"2024-03-06T07:21:01.437079+0000","last_active":"2024-03-06T07:21:01.438756+0000","last_peered":"2024-03-06T07:21:01.438756+0000","last_clean":"2024-03-06T07:21:01.438756+0000","last_became_active":"2024-03-06T07:21:01.436848+0000","last_became_peered":"2024-03-06T07:21:01.436848+0000","last_unstale":"2024-03-06T07:21:01.438756+0000","last_undegraded":"2024-03-06T07:21:01.438756+0000","last_fullsized":"2024-03-06T07:21:01.438756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.233171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.419658+0000","last_change":"2024-03-06T07:20:51.899213+0000","last_active":"2024-03-06T07:21:01.419658+0000","last_peered":"2024-03-06T07:21:01.419658+0000","last_clean":"2024-03-06T07:21:01.419658+0000","last_became_active":"2024-03-06T07:20:51.898882+0000","last_became_peered":"2024-03-06T07:20:51.898882+0000","last_unstale":"2024-03-06T07:21:01.419658+0000","last_undegraded":"2024-03-06T07:21:01.419658+0000","last_fullsized":"2024-03-06T07:21:01.419658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431260+0000","last_change":"2024-03-06T07:20:50.888878+0000","last_active":"2024-03-06T07:21:01.431260+0000","last_peered":"2024-03-06T07:21:01.431260+0000","last_clean":"2024-03-06T07:21:01.431260+0000","last_became_active":"2024-03-06T07:20:50.888604+0000","last_became_peered":"2024-03-06T07:20:50.888604+0000","last_unstale":"2024-03-06T07:21:01.431260+0000","last_undegraded":"2024-03-06T07:21:01.431260+0000","last_fullsized":"2024-03-06T07:21:01.431260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:33.413428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422649+0000","last_change":"2024-03-06T07:21:01.422649+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422649+0000","last_undegraded":"2024-03-06T07:21:01.422649+0000","last_fullsized":"2024-03-06T07:21:01.422649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438997+0000","last_change":"2024-03-06T07:21:01.433417+0000","last_active":"2024-03-06T07:21:01.438997+0000","last_peered":"2024-03-06T07:21:01.438997+0000","last_clean":"2024-03-06T07:21:01.438997+0000","last_became_active":"2024-03-06T07:21:01.432105+0000","last_became_peered":"2024-03-06T07:21:01.432105+0000","last_unstale":"2024-03-06T07:21:01.438997+0000","last_undegraded":"2024-03-06T07:21:01.438997+0000","last_fullsized":"2024-03-06T07:21:01.438997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:18.120482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425648+0000","last_change":"2024-03-06T07:20:51.893070+0000","last_active":"2024-03-06T07:21:01.425648+0000","last_peered":"2024-03-06T07:21:01.425648+0000","last_clean":"2024-03-06T07:21:01.425648+0000","last_became_active":"2024-03-06T07:20:51.892785+0000","last_became_peered":"2024-03-06T07:20:51.892785+0000","last_unstale":"2024-03-06T07:21:01.425648+0000","last_undegraded":"2024-03-06T07:21:01.425648+0000","last_fullsized":"2024-03-06T07:21:01.425648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:09.591477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425357+0000","last_change":"2024-03-06T07:20:50.892942+0000","last_active":"2024-03-06T07:21:01.425357+0000","last_peered":"2024-03-06T07:21:01.425357+0000","last_clean":"2024-03-06T07:21:01.425357+0000","last_became_active":"2024-03-06T07:20:50.892638+0000","last_became_peered":"2024-03-06T07:20:50.892638+0000","last_unstale":"2024-03-06T07:21:01.425357+0000","last_undegraded":"2024-03-06T07:21:01.425357+0000","last_fullsized":"2024-03-06T07:21:01.425357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:40.434821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.427886+0000","last_change":"2024-03-06T07:21:01.427886+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.427886+0000","last_undegraded":"2024-03-06T07:21:01.427886+0000","last_fullsized":"2024-03-06T07:21:01.427886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.438316+0000","last_change":"2024-03-06T07:21:01.435337+0000","last_active":"2024-03-06T07:21:01.438316+0000","last_peered":"2024-03-06T07:21:01.438316+0000","last_clean":"2024-03-06T07:21:01.438316+0000","last_became_active":"2024-03-06T07:21:01.435098+0000","last_became_peered":"2024-03-06T07:21:01.435098+0000","last_unstale":"2024-03-06T07:21:01.438316+0000","last_undegraded":"2024-03-06T07:21:01.438316+0000","last_fullsized":"2024-03-06T07:21:01.438316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:30.038898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425885+0000","last_change":"2024-03-06T07:20:51.895826+0000","last_active":"2024-03-06T07:21:01.425885+0000","last_peered":"2024-03-06T07:21:01.425885+0000","last_clean":"2024-03-06T07:21:01.425885+0000","last_became_active":"2024-03-06T07:20:51.895483+0000","last_became_peered":"2024-03-06T07:20:51.895483+0000","last_unstale":"2024-03-06T07:21:01.425885+0000","last_undegraded":"2024-03-06T07:21:01.425885+0000","last_fullsized":"2024-03-06T07:21:01.425885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425724+0000","last_change":"2024-03-06T07:20:50.879754+0000","last_active":"2024-03-06T07:21:01.425724+0000","last_peered":"2024-03-06T07:21:01.425724+0000","last_clean":"2024-03-06T07:21:01.425724+0000","last_became_active":"2024-03-06T07:20:50.879488+0000","last_became_peered":"2024-03-06T07:20:50.879488+0000","last_unstale":"2024-03-06T07:21:01.425724+0000","last_undegraded":"2024-03-06T07:21:01.425724+0000","last_fullsized":"2024-03-06T07:21:01.425724+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.603045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422772+0000","last_change":"2024-03-06T07:21:01.422772+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422772+0000","last_undegraded":"2024-03-06T07:21:01.422772+0000","last_fullsized":"2024-03-06T07:21:01.422772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.441007+0000","last_change":"2024-03-06T07:21:01.439920+0000","last_active":"2024-03-06T07:21:01.441007+0000","last_peered":"2024-03-06T07:21:01.441007+0000","last_clean":"2024-03-06T07:21:01.441007+0000","last_became_active":"2024-03-06T07:21:01.439672+0000","last_became_peered":"2024-03-06T07:21:01.439672+0000","last_unstale":"2024-03-06T07:21:01.441007+0000","last_undegraded":"2024-03-06T07:21:01.441007+0000","last_fullsized":"2024-03-06T07:21:01.441007+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:37.148955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.428677+0000","last_change":"2024-03-06T07:20:51.879085+0000","last_active":"2024-03-06T07:21:01.428677+0000","last_peered":"2024-03-06T07:21:01.428677+0000","last_clean":"2024-03-06T07:21:01.428677+0000","last_became_active":"2024-03-06T07:20:51.878776+0000","last_became_peered":"2024-03-06T07:20:51.878776+0000","last_unstale":"2024-03-06T07:21:01.428677+0000","last_undegraded":"2024-03-06T07:21:01.428677+0000","last_fullsized":"2024-03-06T07:21:01.428677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426282+0000","last_change":"2024-03-06T07:20:50.887928+0000","last_active":"2024-03-06T07:21:01.426282+0000","last_peered":"2024-03-06T07:21:01.426282+0000","last_clean":"2024-03-06T07:21:01.426282+0000","last_became_active":"2024-03-06T07:20:50.887638+0000","last_became_peered":"2024-03-06T07:20:50.887638+0000","last_unstale":"2024-03-06T07:21:01.426282+0000","last_undegraded":"2024-03-06T07:21:01.426282+0000","last_fullsized":"2024-03-06T07:21:01.426282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:52.309668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.422497+0000","last_change":"2024-03-06T07:21:01.422497+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.422497+0000","last_undegraded":"2024-03-06T07:21:01.422497+0000","last_fullsized":"2024-03-06T07:21:01.422497+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.440089+0000","last_change":"2024-03-06T07:21:01.436756+0000","last_active":"2024-03-06T07:21:01.440089+0000","last_peered":"2024-03-06T07:21:01.440089+0000","last_clean":"2024-03-06T07:21:01.440089+0000","last_became_active":"2024-03-06T07:21:01.436158+0000","last_became_peered":"2024-03-06T07:21:01.436158+0000","last_unstale":"2024-03-06T07:21:01.440089+0000","last_undegraded":"2024-03-06T07:21:01.440089+0000","last_fullsized":"2024-03-06T07:21:01.440089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:11.908825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.431875+0000","last_change":"2024-03-06T07:20:51.883253+0000","last_active":"2024-03-06T07:21:01.431875+0000","last_peered":"2024-03-06T07:21:01.431875+0000","last_clean":"2024-03-06T07:21:01.431875+0000","last_became_active":"2024-03-06T07:20:51.882993+0000","last_became_peered":"2024-03-06T07:20:51.882993+0000","last_unstale":"2024-03-06T07:21:01.431875+0000","last_undegraded":"2024-03-06T07:21:01.431875+0000","last_fullsized":"2024-03-06T07:21:01.431875+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.426104+0000","last_change":"2024-03-06T07:20:50.879149+0000","last_active":"2024-03-06T07:21:01.426104+0000","last_peered":"2024-03-06T07:21:01.426104+0000","last_clean":"2024-03-06T07:21:01.426104+0000","last_became_active":"2024-03-06T07:20:50.878867+0000","last_became_peered":"2024-03-06T07:20:50.878867+0000","last_unstale":"2024-03-06T07:21:01.426104+0000","last_undegraded":"2024-03-06T07:21:01.426104+0000","last_fullsized":"2024-03-06T07:21:01.426104+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:57.394196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":76,"state":"creating+peering","last_fresh":"2024-03-06T07:21:01.429819+0000","last_change":"2024-03-06T07:21:01.429819+0000","last_active":"2024-03-06T07:21:01.415411+0000","last_peered":"2024-03-06T07:21:01.415411+0000","last_clean":"2024-03-06T07:21:01.415411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:01.429819+0000","last_undegraded":"2024-03-06T07:21:01.429819+0000","last_fullsized":"2024-03-06T07:21:01.429819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.442400+0000","last_change":"2024-03-06T07:21:01.440630+0000","last_active":"2024-03-06T07:21:01.442400+0000","last_peered":"2024-03-06T07:21:01.442400+0000","last_clean":"2024-03-06T07:21:01.442400+0000","last_became_active":"2024-03-06T07:21:01.440375+0000","last_became_peered":"2024-03-06T07:21:01.440375+0000","last_unstale":"2024-03-06T07:21:01.442400+0000","last_undegraded":"2024-03-06T07:21:01.442400+0000","last_fullsized":"2024-03-06T07:21:01.442400+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:06.815094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.432805+0000","last_change":"2024-03-06T07:20:51.891707+0000","last_active":"2024-03-06T07:21:01.432805+0000","last_peered":"2024-03-06T07:21:01.432805+0000","last_clean":"2024-03-06T07:21:01.432805+0000","last_became_active":"2024-03-06T07:20:51.891427+0000","last_became_peered":"2024-03-06T07:20:51.891427+0000","last_unstale":"2024-03-06T07:21:01.432805+0000","last_undegraded":"2024-03-06T07:21:01.432805+0000","last_fullsized":"2024-03-06T07:21:01.432805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.425699+0000","last_change":"2024-03-06T07:20:50.885372+0000","last_active":"2024-03-06T07:21:01.425699+0000","last_peered":"2024-03-06T07:21:01.425699+0000","last_clean":"2024-03-06T07:21:01.425699+0000","last_became_active":"2024-03-06T07:20:50.885093+0000","last_became_peered":"2024-03-06T07:20:50.885093+0000","last_unstale":"2024-03-06T07:21:01.425699+0000","last_undegraded":"2024-03-06T07:21:01.425699+0000","last_fullsized":"2024-03-06T07:21:01.425699+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:12.807980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":276,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-06T07:21:01.430887+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:21:01.430887+0000","last_peered":"2024-03-06T07:21:01.430887+0000","last_clean":"2024-03-06T07:21:01.430887+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:21:01.430887+0000","last_undegraded":"2024-03-06T07:21:01.430887+0000","last_fullsized":"2024-03-06T07:21:01.430887+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":1031546,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":442768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":441190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574886,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":950272,"data_stored":442872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":933888,"data_stored":441738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":441898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":441268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":1031524,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:09.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:09.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.939+0000 7ff516ceb640 1 Processor -- start 2024-03-06T07:21:09.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.939+0000 7ff516ceb640 1 -- start start 2024-03-06T07:21:09.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff516ceb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff51006afc0 0x7ff51006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff516ceb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5100c25c0 0x7ff510153890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff516ceb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff510153dd0 con 0x7ff5100c20b0 2024-03-06T07:21:09.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff516ceb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5100a9530 con 0x7ff51006afc0 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff516ceb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5100a9810 con 0x7ff5100c25c0 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff514a87640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff51006afc0 0x7ff51006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff514a87640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff51006afc0 0x7ff51006b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40692/0 (socket says 172.21.15.94:40692) 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff514a87640 1 -- 172.21.15.94:0/174440068 learned_addr learned my addr 172.21.15.94:0/174440068 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff515288640 1 --2- 172.21.15.94:0/174440068 >> v2:172.21.15.145:3300/0 conn(0x7ff5100c25c0 0x7ff510153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff514a87640 1 -- 172.21.15.94:0/174440068 >> v2:172.21.15.145:3300/0 conn(0x7ff5100c25c0 msgr2=0x7ff510153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff514a87640 1 --2- 172.21.15.94:0/174440068 >> v2:172.21.15.145:3300/0 conn(0x7ff5100c25c0 0x7ff510153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff514a87640 1 -- 172.21.15.94:0/174440068 >> v1:172.21.15.145:6789/0 conn(0x7ff5100c20b0 legacy=0x7ff510153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.940+0000 7ff514a87640 1 -- 172.21.15.94:0/174440068 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100a9af0 con 0x7ff51006afc0 2024-03-06T07:21:09.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.941+0000 7ff514a87640 1 --2- 172.21.15.94:0/174440068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51006afc0 0x7ff51006b3a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff5100c4010 tx=0x7ff50402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1536dd86b5387570 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.941+0000 7ff50f7fe640 1 -- 172.21.15.94:0/174440068 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50402e070 con 0x7ff51006afc0 2024-03-06T07:21:09.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.941+0000 7ff50f7fe640 1 -- 172.21.15.94:0/174440068 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff504037070 con 0x7ff51006afc0 2024-03-06T07:21:09.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.941+0000 7ff50f7fe640 1 -- 172.21.15.94:0/174440068 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504031a00 con 0x7ff51006afc0 2024-03-06T07:21:09.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.941+0000 7ff516ceb640 1 -- 172.21.15.94:0/174440068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51006afc0 msgr2=0x7ff51006b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.941+0000 7ff516ceb640 1 --2- 172.21.15.94:0/174440068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51006afc0 0x7ff51006b3a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff5100c4010 tx=0x7ff50402b880 comp rx=0 tx=0).stop 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.942+0000 7ff516ceb640 1 -- 172.21.15.94:0/174440068 shutdown_connections 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.942+0000 7ff516ceb640 1 --2- 172.21.15.94:0/174440068 >> v2:172.21.15.145:3300/0 conn(0x7ff5100c25c0 0x7ff510153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.942+0000 7ff516ceb640 1 --2- 172.21.15.94:0/174440068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51006afc0 0x7ff51006b3a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.942+0000 7ff516ceb640 1 -- 172.21.15.94:0/174440068 >> 172.21.15.94:0/174440068 conn(0x7ff5100c76b0 msgr2=0x7ff5100c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.942+0000 7ff516ceb640 1 -- 172.21.15.94:0/174440068 shutdown_connections 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.942+0000 7ff516ceb640 1 -- 172.21.15.94:0/174440068 wait complete. 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.943+0000 7ff516ceb640 1 Processor -- start 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.943+0000 7ff516ceb640 1 -- start start 2024-03-06T07:21:09.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006afc0 0x7ff5101556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5100c20b0 0x7ff5101ec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5100c25c0 0x7ff5101e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510150980 con 0x7ff5100c25c0 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100aa0b0 con 0x7ff51006afc0 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff510150b00 con 0x7ff5100c20b0 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5100c25c0 0x7ff5101e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5100c25c0 0x7ff5101e9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40706/0 (socket says 172.21.15.94:40706) 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 -- 172.21.15.94:0/2114016426 learned_addr learned my addr 172.21.15.94:0/2114016426 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff514a87640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006afc0 0x7ff5101556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff50ffff640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5100c20b0 0x7ff5101ec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 -- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5100c20b0 msgr2=0x7ff5101ec4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5100c20b0 0x7ff5101ec4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 -- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006afc0 msgr2=0x7ff5101556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006afc0 0x7ff5101556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 -- 172.21.15.94:0/2114016426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5101ecce0 con 0x7ff5100c25c0 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff515288640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5100c25c0 0x7ff5101e9790 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff50800efc0 tx=0x7ff50800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff50d7fa640 1 -- 172.21.15.94:0/2114016426 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508018020 con 0x7ff5100c25c0 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff50d7fa640 1 -- 172.21.15.94:0/2114016426 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff508014480 con 0x7ff5100c25c0 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5101ecfc0 con 0x7ff5100c25c0 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff50d7fa640 1 -- 172.21.15.94:0/2114016426 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508014870 con 0x7ff5100c25c0 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.944+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5101ec9f0 con 0x7ff5100c25c0 2024-03-06T07:21:09.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.945+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51004f330 con 0x7ff5100c25c0 2024-03-06T07:21:09.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.946+0000 7ff50d7fa640 1 -- 172.21.15.94:0/2114016426 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff508005d10 con 0x7ff5100c25c0 2024-03-06T07:21:09.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.946+0000 7ff50d7fa640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4f406b500 0x7ff4f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:09.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.946+0000 7ff50d7fa640 1 -- 172.21.15.94:0/2114016426 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff50808b080 con 0x7ff5100c25c0 2024-03-06T07:21:09.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.946+0000 7ff514a87640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4f406b500 0x7ff4f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:09.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.947+0000 7ff514a87640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4f406b500 0x7ff4f406d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff504036000 tx=0x7ff50403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:09.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:09.951+0000 7ff50d7fa640 1 -- 172.21.15.94:0/2114016426 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff508022070 con 0x7ff5100c25c0 2024-03-06T07:21:10.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.117+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5100a6e70 con 0x7ff4f406b500 2024-03-06T07:21:10.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.129+0000 7ff50d7fa640 1 -- 172.21.15.94:0/2114016426 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490719 (secure 0 0 0) 0x7ff5100a6e70 con 0x7ff4f406b500 2024-03-06T07:21:10.136 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:10.139 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:21:10.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4f406b500 msgr2=0x7ff4f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4f406b500 0x7ff4f406d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff504036000 tx=0x7ff50403d040 comp rx=0 tx=0).stop 2024-03-06T07:21:10.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5100c25c0 msgr2=0x7ff5101e9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5100c25c0 0x7ff5101e9790 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff50800efc0 tx=0x7ff50800c490 comp rx=0 tx=0).stop 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 shutdown_connections 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4f406b500 0x7ff4f406d9c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5100c25c0 0x7ff5101e9790 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5100c20b0 0x7ff5101ec4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 --2- 172.21.15.94:0/2114016426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006afc0 0x7ff5101556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 >> 172.21.15.94:0/2114016426 conn(0x7ff5100c76b0 msgr2=0x7ff5100c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 shutdown_connections 2024-03-06T07:21:10.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.133+0000 7ff516ceb640 1 -- 172.21.15.94:0/2114016426 wait complete. 2024-03-06T07:21:10.145 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-03-06T07:21:08.672114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":377,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":244704,"kb_used_data":7712,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730016,"statfs":{"total":773094113280,"available":772843536384,"internally_reserved":0,"allocated":7897088,"data_stored":4844216,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.251436"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415842+0000","last_change":"2024-03-06T07:21:02.468913+0000","last_active":"2024-03-06T07:21:04.415842+0000","last_peered":"2024-03-06T07:21:04.415842+0000","last_clean":"2024-03-06T07:21:04.415842+0000","last_became_active":"2024-03-06T07:21:02.468601+0000","last_became_peered":"2024-03-06T07:21:02.468601+0000","last_unstale":"2024-03-06T07:21:04.415842+0000","last_undegraded":"2024-03-06T07:21:04.415842+0000","last_fullsized":"2024-03-06T07:21:04.415842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:09.401096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416266+0000","last_change":"2024-03-06T07:20:51.896309+0000","last_active":"2024-03-06T07:21:04.416266+0000","last_peered":"2024-03-06T07:21:04.416266+0000","last_clean":"2024-03-06T07:21:04.416266+0000","last_became_active":"2024-03-06T07:20:51.896009+0000","last_became_peered":"2024-03-06T07:20:51.896009+0000","last_unstale":"2024-03-06T07:21:04.416266+0000","last_undegraded":"2024-03-06T07:21:04.416266+0000","last_fullsized":"2024-03-06T07:21:04.416266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914604+0000","last_change":"2024-03-06T07:21:02.457687+0000","last_active":"2024-03-06T07:21:03.914604+0000","last_peered":"2024-03-06T07:21:03.914604+0000","last_clean":"2024-03-06T07:21:03.914604+0000","last_became_active":"2024-03-06T07:21:02.456998+0000","last_became_peered":"2024-03-06T07:21:02.456998+0000","last_unstale":"2024-03-06T07:21:03.914604+0000","last_undegraded":"2024-03-06T07:21:03.914604+0000","last_fullsized":"2024-03-06T07:21:03.914604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:34.542496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416726+0000","last_change":"2024-03-06T07:20:51.895412+0000","last_active":"2024-03-06T07:21:04.416726+0000","last_peered":"2024-03-06T07:21:04.416726+0000","last_clean":"2024-03-06T07:21:04.416726+0000","last_became_active":"2024-03-06T07:20:51.895169+0000","last_became_peered":"2024-03-06T07:20:51.895169+0000","last_unstale":"2024-03-06T07:21:04.416726+0000","last_undegraded":"2024-03-06T07:21:04.416726+0000","last_fullsized":"2024-03-06T07:21:04.416726+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:31.352627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416251+0000","last_change":"2024-03-06T07:21:02.469530+0000","last_active":"2024-03-06T07:21:04.416251+0000","last_peered":"2024-03-06T07:21:04.416251+0000","last_clean":"2024-03-06T07:21:04.416251+0000","last_became_active":"2024-03-06T07:21:02.469274+0000","last_became_peered":"2024-03-06T07:21:02.469274+0000","last_unstale":"2024-03-06T07:21:04.416251+0000","last_undegraded":"2024-03-06T07:21:04.416251+0000","last_fullsized":"2024-03-06T07:21:04.416251+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:53.260411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914774+0000","last_change":"2024-03-06T07:20:51.878718+0000","last_active":"2024-03-06T07:21:03.914774+0000","last_peered":"2024-03-06T07:21:03.914774+0000","last_clean":"2024-03-06T07:21:03.914774+0000","last_became_active":"2024-03-06T07:20:51.878467+0000","last_became_peered":"2024-03-06T07:20:51.878467+0000","last_unstale":"2024-03-06T07:21:03.914774+0000","last_undegraded":"2024-03-06T07:21:03.914774+0000","last_fullsized":"2024-03-06T07:21:03.914774+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415359+0000","last_change":"2024-03-06T07:21:02.472348+0000","last_active":"2024-03-06T07:21:04.415359+0000","last_peered":"2024-03-06T07:21:04.415359+0000","last_clean":"2024-03-06T07:21:04.415359+0000","last_became_active":"2024-03-06T07:21:02.472089+0000","last_became_peered":"2024-03-06T07:21:02.472089+0000","last_unstale":"2024-03-06T07:21:04.415359+0000","last_undegraded":"2024-03-06T07:21:04.415359+0000","last_fullsized":"2024-03-06T07:21:04.415359+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:09.463530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415869+0000","last_change":"2024-03-06T07:20:51.892550+0000","last_active":"2024-03-06T07:21:04.415869+0000","last_peered":"2024-03-06T07:21:04.415869+0000","last_clean":"2024-03-06T07:21:04.415869+0000","last_became_active":"2024-03-06T07:20:51.892264+0000","last_became_peered":"2024-03-06T07:20:51.892264+0000","last_unstale":"2024-03-06T07:21:04.415869+0000","last_undegraded":"2024-03-06T07:21:04.415869+0000","last_fullsized":"2024-03-06T07:21:04.415869+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416087+0000","last_change":"2024-03-06T07:21:02.471117+0000","last_active":"2024-03-06T07:21:04.416087+0000","last_peered":"2024-03-06T07:21:04.416087+0000","last_clean":"2024-03-06T07:21:04.416087+0000","last_became_active":"2024-03-06T07:21:02.470842+0000","last_became_peered":"2024-03-06T07:21:02.470842+0000","last_unstale":"2024-03-06T07:21:04.416087+0000","last_undegraded":"2024-03-06T07:21:04.416087+0000","last_fullsized":"2024-03-06T07:21:04.416087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:09.401096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457995+0000","last_change":"2024-03-06T07:20:51.893305+0000","last_active":"2024-03-06T07:21:03.457995+0000","last_peered":"2024-03-06T07:21:03.457995+0000","last_clean":"2024-03-06T07:21:03.457995+0000","last_became_active":"2024-03-06T07:20:51.893041+0000","last_became_peered":"2024-03-06T07:20:51.893041+0000","last_unstale":"2024-03-06T07:21:03.457995+0000","last_undegraded":"2024-03-06T07:21:03.457995+0000","last_fullsized":"2024-03-06T07:21:03.457995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:39.229585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914192+0000","last_change":"2024-03-06T07:21:02.462818+0000","last_active":"2024-03-06T07:21:03.914192+0000","last_peered":"2024-03-06T07:21:03.914192+0000","last_clean":"2024-03-06T07:21:03.914192+0000","last_became_active":"2024-03-06T07:21:02.462564+0000","last_became_peered":"2024-03-06T07:21:02.462564+0000","last_unstale":"2024-03-06T07:21:03.914192+0000","last_undegraded":"2024-03-06T07:21:03.914192+0000","last_fullsized":"2024-03-06T07:21:03.914192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:41.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415557+0000","last_change":"2024-03-06T07:20:51.893812+0000","last_active":"2024-03-06T07:21:04.415557+0000","last_peered":"2024-03-06T07:21:04.415557+0000","last_clean":"2024-03-06T07:21:04.415557+0000","last_became_active":"2024-03-06T07:20:51.891649+0000","last_became_peered":"2024-03-06T07:20:51.891649+0000","last_unstale":"2024-03-06T07:21:04.415557+0000","last_undegraded":"2024-03-06T07:21:04.415557+0000","last_fullsized":"2024-03-06T07:21:04.415557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:46.574055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460847+0000","last_change":"2024-03-06T07:21:02.466520+0000","last_active":"2024-03-06T07:21:03.460847+0000","last_peered":"2024-03-06T07:21:03.460847+0000","last_clean":"2024-03-06T07:21:03.460847+0000","last_became_active":"2024-03-06T07:21:02.466289+0000","last_became_peered":"2024-03-06T07:21:02.466289+0000","last_unstale":"2024-03-06T07:21:03.460847+0000","last_undegraded":"2024-03-06T07:21:03.460847+0000","last_fullsized":"2024-03-06T07:21:03.460847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.079558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459336+0000","last_change":"2024-03-06T07:20:51.894141+0000","last_active":"2024-03-06T07:21:03.459336+0000","last_peered":"2024-03-06T07:21:03.459336+0000","last_clean":"2024-03-06T07:21:03.459336+0000","last_became_active":"2024-03-06T07:20:51.893793+0000","last_became_peered":"2024-03-06T07:20:51.893793+0000","last_unstale":"2024-03-06T07:21:03.459336+0000","last_undegraded":"2024-03-06T07:21:03.459336+0000","last_fullsized":"2024-03-06T07:21:03.459336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415347+0000","last_change":"2024-03-06T07:21:02.461065+0000","last_active":"2024-03-06T07:21:04.415347+0000","last_peered":"2024-03-06T07:21:04.415347+0000","last_clean":"2024-03-06T07:21:04.415347+0000","last_became_active":"2024-03-06T07:21:02.459736+0000","last_became_peered":"2024-03-06T07:21:02.459736+0000","last_unstale":"2024-03-06T07:21:04.415347+0000","last_undegraded":"2024-03-06T07:21:04.415347+0000","last_fullsized":"2024-03-06T07:21:04.415347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:53.260411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457731+0000","last_change":"2024-03-06T07:20:51.894037+0000","last_active":"2024-03-06T07:21:03.457731+0000","last_peered":"2024-03-06T07:21:03.457731+0000","last_clean":"2024-03-06T07:21:03.457731+0000","last_became_active":"2024-03-06T07:20:51.893773+0000","last_became_peered":"2024-03-06T07:20:51.893773+0000","last_unstale":"2024-03-06T07:21:03.457731+0000","last_undegraded":"2024-03-06T07:21:03.457731+0000","last_fullsized":"2024-03-06T07:21:03.457731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:48.344455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461246+0000","last_change":"2024-03-06T07:21:02.473686+0000","last_active":"2024-03-06T07:21:03.461246+0000","last_peered":"2024-03-06T07:21:03.461246+0000","last_clean":"2024-03-06T07:21:03.461246+0000","last_became_active":"2024-03-06T07:21:02.473325+0000","last_became_peered":"2024-03-06T07:21:02.473325+0000","last_unstale":"2024-03-06T07:21:03.461246+0000","last_undegraded":"2024-03-06T07:21:03.461246+0000","last_fullsized":"2024-03-06T07:21:03.461246+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.234772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461347+0000","last_change":"2024-03-06T07:20:51.886223+0000","last_active":"2024-03-06T07:21:03.461347+0000","last_peered":"2024-03-06T07:21:03.461347+0000","last_clean":"2024-03-06T07:21:03.461347+0000","last_became_active":"2024-03-06T07:20:51.885985+0000","last_became_peered":"2024-03-06T07:20:51.885985+0000","last_unstale":"2024-03-06T07:21:03.461347+0000","last_undegraded":"2024-03-06T07:21:03.461347+0000","last_fullsized":"2024-03-06T07:21:03.461347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:13.814018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458028+0000","last_change":"2024-03-06T07:21:02.470822+0000","last_active":"2024-03-06T07:21:03.458028+0000","last_peered":"2024-03-06T07:21:03.458028+0000","last_clean":"2024-03-06T07:21:03.458028+0000","last_became_active":"2024-03-06T07:21:02.470562+0000","last_became_peered":"2024-03-06T07:21:02.470562+0000","last_unstale":"2024-03-06T07:21:03.458028+0000","last_undegraded":"2024-03-06T07:21:03.458028+0000","last_fullsized":"2024-03-06T07:21:03.458028+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:28.777119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415583+0000","last_change":"2024-03-06T07:20:51.890465+0000","last_active":"2024-03-06T07:21:04.415583+0000","last_peered":"2024-03-06T07:21:04.415583+0000","last_clean":"2024-03-06T07:21:04.415583+0000","last_became_active":"2024-03-06T07:20:51.890183+0000","last_became_peered":"2024-03-06T07:20:51.890183+0000","last_unstale":"2024-03-06T07:21:04.415583+0000","last_undegraded":"2024-03-06T07:21:04.415583+0000","last_fullsized":"2024-03-06T07:21:04.415583+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461334+0000","last_change":"2024-03-06T07:21:02.464767+0000","last_active":"2024-03-06T07:21:03.461334+0000","last_peered":"2024-03-06T07:21:03.461334+0000","last_clean":"2024-03-06T07:21:03.461334+0000","last_became_active":"2024-03-06T07:21:02.464533+0000","last_became_peered":"2024-03-06T07:21:02.464533+0000","last_unstale":"2024-03-06T07:21:03.461334+0000","last_undegraded":"2024-03-06T07:21:03.461334+0000","last_fullsized":"2024-03-06T07:21:03.461334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:07.001720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416054+0000","last_change":"2024-03-06T07:20:51.897836+0000","last_active":"2024-03-06T07:21:04.416054+0000","last_peered":"2024-03-06T07:21:04.416054+0000","last_clean":"2024-03-06T07:21:04.416054+0000","last_became_active":"2024-03-06T07:20:51.897551+0000","last_became_peered":"2024-03-06T07:20:51.897551+0000","last_unstale":"2024-03-06T07:21:04.416054+0000","last_undegraded":"2024-03-06T07:21:04.416054+0000","last_fullsized":"2024-03-06T07:21:04.416054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458245+0000","last_change":"2024-03-06T07:21:02.463426+0000","last_active":"2024-03-06T07:21:03.458245+0000","last_peered":"2024-03-06T07:21:03.458245+0000","last_clean":"2024-03-06T07:21:03.458245+0000","last_became_active":"2024-03-06T07:21:02.463058+0000","last_became_peered":"2024-03-06T07:21:02.463058+0000","last_unstale":"2024-03-06T07:21:03.458245+0000","last_undegraded":"2024-03-06T07:21:03.458245+0000","last_fullsized":"2024-03-06T07:21:03.458245+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:55.596698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415915+0000","last_change":"2024-03-06T07:20:51.886475+0000","last_active":"2024-03-06T07:21:04.415915+0000","last_peered":"2024-03-06T07:21:04.415915+0000","last_clean":"2024-03-06T07:21:04.415915+0000","last_became_active":"2024-03-06T07:20:51.886218+0000","last_became_peered":"2024-03-06T07:20:51.886218+0000","last_unstale":"2024-03-06T07:21:04.415915+0000","last_undegraded":"2024-03-06T07:21:04.415915+0000","last_fullsized":"2024-03-06T07:21:04.415915+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914305+0000","last_change":"2024-03-06T07:21:02.474547+0000","last_active":"2024-03-06T07:21:03.914305+0000","last_peered":"2024-03-06T07:21:03.914305+0000","last_clean":"2024-03-06T07:21:03.914305+0000","last_became_active":"2024-03-06T07:21:02.469965+0000","last_became_peered":"2024-03-06T07:21:02.469965+0000","last_unstale":"2024-03-06T07:21:03.914305+0000","last_undegraded":"2024-03-06T07:21:03.914305+0000","last_fullsized":"2024-03-06T07:21:03.914305+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.234772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458075+0000","last_change":"2024-03-06T07:20:51.887999+0000","last_active":"2024-03-06T07:21:03.458075+0000","last_peered":"2024-03-06T07:21:03.458075+0000","last_clean":"2024-03-06T07:21:03.458075+0000","last_became_active":"2024-03-06T07:20:51.887749+0000","last_became_peered":"2024-03-06T07:20:51.887749+0000","last_unstale":"2024-03-06T07:21:03.458075+0000","last_undegraded":"2024-03-06T07:21:03.458075+0000","last_fullsized":"2024-03-06T07:21:03.458075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:25.243948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415222+0000","last_change":"2024-03-06T07:21:02.477819+0000","last_active":"2024-03-06T07:21:04.415222+0000","last_peered":"2024-03-06T07:21:04.415222+0000","last_clean":"2024-03-06T07:21:04.415222+0000","last_became_active":"2024-03-06T07:21:02.477576+0000","last_became_peered":"2024-03-06T07:21:02.477576+0000","last_unstale":"2024-03-06T07:21:04.415222+0000","last_undegraded":"2024-03-06T07:21:04.415222+0000","last_fullsized":"2024-03-06T07:21:04.415222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:29.012824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415658+0000","last_change":"2024-03-06T07:20:51.892140+0000","last_active":"2024-03-06T07:21:04.415658+0000","last_peered":"2024-03-06T07:21:04.415658+0000","last_clean":"2024-03-06T07:21:04.415658+0000","last_became_active":"2024-03-06T07:20:51.891195+0000","last_became_peered":"2024-03-06T07:20:51.891195+0000","last_unstale":"2024-03-06T07:21:04.415658+0000","last_undegraded":"2024-03-06T07:21:04.415658+0000","last_fullsized":"2024-03-06T07:21:04.415658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:28.641052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914407+0000","last_change":"2024-03-06T07:21:02.460211+0000","last_active":"2024-03-06T07:21:03.914407+0000","last_peered":"2024-03-06T07:21:03.914407+0000","last_clean":"2024-03-06T07:21:03.914407+0000","last_became_active":"2024-03-06T07:21:02.459858+0000","last_became_peered":"2024-03-06T07:21:02.459858+0000","last_unstale":"2024-03-06T07:21:03.914407+0000","last_undegraded":"2024-03-06T07:21:03.914407+0000","last_fullsized":"2024-03-06T07:21:03.914407+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:48.662859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415156+0000","last_change":"2024-03-06T07:20:51.894735+0000","last_active":"2024-03-06T07:21:04.415156+0000","last_peered":"2024-03-06T07:21:04.415156+0000","last_clean":"2024-03-06T07:21:04.415156+0000","last_became_active":"2024-03-06T07:20:51.894479+0000","last_became_peered":"2024-03-06T07:20:51.894479+0000","last_unstale":"2024-03-06T07:21:04.415156+0000","last_undegraded":"2024-03-06T07:21:04.415156+0000","last_fullsized":"2024-03-06T07:21:04.415156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460459+0000","last_change":"2024-03-06T07:21:02.470796+0000","last_active":"2024-03-06T07:21:03.460459+0000","last_peered":"2024-03-06T07:21:03.460459+0000","last_clean":"2024-03-06T07:21:03.460459+0000","last_became_active":"2024-03-06T07:21:02.470427+0000","last_became_peered":"2024-03-06T07:21:02.470427+0000","last_unstale":"2024-03-06T07:21:03.460459+0000","last_undegraded":"2024-03-06T07:21:03.460459+0000","last_fullsized":"2024-03-06T07:21:03.460459+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:05.771444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415446+0000","last_change":"2024-03-06T07:20:51.892450+0000","last_active":"2024-03-06T07:21:04.415446+0000","last_peered":"2024-03-06T07:21:04.415446+0000","last_clean":"2024-03-06T07:21:04.415446+0000","last_became_active":"2024-03-06T07:20:51.892184+0000","last_became_peered":"2024-03-06T07:20:51.892184+0000","last_unstale":"2024-03-06T07:21:04.415446+0000","last_undegraded":"2024-03-06T07:21:04.415446+0000","last_fullsized":"2024-03-06T07:21:04.415446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461373+0000","last_change":"2024-03-06T07:21:02.470788+0000","last_active":"2024-03-06T07:21:03.461373+0000","last_peered":"2024-03-06T07:21:03.461373+0000","last_clean":"2024-03-06T07:21:03.461373+0000","last_became_active":"2024-03-06T07:21:02.470451+0000","last_became_peered":"2024-03-06T07:21:02.470451+0000","last_unstale":"2024-03-06T07:21:03.461373+0000","last_undegraded":"2024-03-06T07:21:03.461373+0000","last_fullsized":"2024-03-06T07:21:03.461373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:48.662859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458475+0000","last_change":"2024-03-06T07:20:51.891492+0000","last_active":"2024-03-06T07:21:03.458475+0000","last_peered":"2024-03-06T07:21:03.458475+0000","last_clean":"2024-03-06T07:21:03.458475+0000","last_became_active":"2024-03-06T07:20:51.891212+0000","last_became_peered":"2024-03-06T07:20:51.891212+0000","last_unstale":"2024-03-06T07:21:03.458475+0000","last_undegraded":"2024-03-06T07:21:03.458475+0000","last_fullsized":"2024-03-06T07:21:03.458475+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:49.555650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914557+0000","last_change":"2024-03-06T07:21:02.469469+0000","last_active":"2024-03-06T07:21:03.914557+0000","last_peered":"2024-03-06T07:21:03.914557+0000","last_clean":"2024-03-06T07:21:03.914557+0000","last_became_active":"2024-03-06T07:21:02.469019+0000","last_became_peered":"2024-03-06T07:21:02.469019+0000","last_unstale":"2024-03-06T07:21:03.914557+0000","last_undegraded":"2024-03-06T07:21:03.914557+0000","last_fullsized":"2024-03-06T07:21:03.914557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:20.465007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914930+0000","last_change":"2024-03-06T07:20:51.876410+0000","last_active":"2024-03-06T07:21:03.914930+0000","last_peered":"2024-03-06T07:21:03.914930+0000","last_clean":"2024-03-06T07:21:03.914930+0000","last_became_active":"2024-03-06T07:20:51.876155+0000","last_became_peered":"2024-03-06T07:20:51.876155+0000","last_unstale":"2024-03-06T07:21:03.914930+0000","last_undegraded":"2024-03-06T07:21:03.914930+0000","last_fullsized":"2024-03-06T07:21:03.914930+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415865+0000","last_change":"2024-03-06T07:21:02.477094+0000","last_active":"2024-03-06T07:21:04.415865+0000","last_peered":"2024-03-06T07:21:04.415865+0000","last_clean":"2024-03-06T07:21:04.415865+0000","last_became_active":"2024-03-06T07:21:02.476834+0000","last_became_peered":"2024-03-06T07:21:02.476834+0000","last_unstale":"2024-03-06T07:21:04.415865+0000","last_undegraded":"2024-03-06T07:21:04.415865+0000","last_fullsized":"2024-03-06T07:21:04.415865+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:16.260272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415937+0000","last_change":"2024-03-06T07:20:51.880019+0000","last_active":"2024-03-06T07:21:04.415937+0000","last_peered":"2024-03-06T07:21:04.415937+0000","last_clean":"2024-03-06T07:21:04.415937+0000","last_became_active":"2024-03-06T07:20:51.879756+0000","last_became_peered":"2024-03-06T07:20:51.879756+0000","last_unstale":"2024-03-06T07:21:04.415937+0000","last_undegraded":"2024-03-06T07:21:04.415937+0000","last_fullsized":"2024-03-06T07:21:04.415937+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:27.594449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416338+0000","last_change":"2024-03-06T07:21:02.470456+0000","last_active":"2024-03-06T07:21:04.416338+0000","last_peered":"2024-03-06T07:21:04.416338+0000","last_clean":"2024-03-06T07:21:04.416338+0000","last_became_active":"2024-03-06T07:21:02.470200+0000","last_became_peered":"2024-03-06T07:21:02.470200+0000","last_unstale":"2024-03-06T07:21:04.416338+0000","last_undegraded":"2024-03-06T07:21:04.416338+0000","last_fullsized":"2024-03-06T07:21:04.416338+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:30.513765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457989+0000","last_change":"2024-03-06T07:20:51.890815+0000","last_active":"2024-03-06T07:21:03.457989+0000","last_peered":"2024-03-06T07:21:03.457989+0000","last_clean":"2024-03-06T07:21:03.457989+0000","last_became_active":"2024-03-06T07:20:51.890558+0000","last_became_peered":"2024-03-06T07:20:51.890558+0000","last_unstale":"2024-03-06T07:21:03.457989+0000","last_undegraded":"2024-03-06T07:21:03.457989+0000","last_fullsized":"2024-03-06T07:21:03.457989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:50.925154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459456+0000","last_change":"2024-03-06T07:21:02.463384+0000","last_active":"2024-03-06T07:21:03.459456+0000","last_peered":"2024-03-06T07:21:03.459456+0000","last_clean":"2024-03-06T07:21:03.459456+0000","last_became_active":"2024-03-06T07:21:02.463029+0000","last_became_peered":"2024-03-06T07:21:02.463029+0000","last_unstale":"2024-03-06T07:21:03.459456+0000","last_undegraded":"2024-03-06T07:21:03.459456+0000","last_fullsized":"2024-03-06T07:21:03.459456+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:41.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461039+0000","last_change":"2024-03-06T07:20:51.884508+0000","last_active":"2024-03-06T07:21:03.461039+0000","last_peered":"2024-03-06T07:21:03.461039+0000","last_clean":"2024-03-06T07:21:03.461039+0000","last_became_active":"2024-03-06T07:20:51.884237+0000","last_became_peered":"2024-03-06T07:20:51.884237+0000","last_unstale":"2024-03-06T07:21:03.461039+0000","last_undegraded":"2024-03-06T07:21:03.461039+0000","last_fullsized":"2024-03-06T07:21:03.461039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:18:18.221013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415354+0000","last_change":"2024-03-06T07:21:02.468937+0000","last_active":"2024-03-06T07:21:04.415354+0000","last_peered":"2024-03-06T07:21:04.415354+0000","last_clean":"2024-03-06T07:21:04.415354+0000","last_became_active":"2024-03-06T07:21:02.468664+0000","last_became_peered":"2024-03-06T07:21:02.468664+0000","last_unstale":"2024-03-06T07:21:04.415354+0000","last_undegraded":"2024-03-06T07:21:04.415354+0000","last_fullsized":"2024-03-06T07:21:04.415354+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:53.404906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458076+0000","last_change":"2024-03-06T07:20:51.890118+0000","last_active":"2024-03-06T07:21:03.458076+0000","last_peered":"2024-03-06T07:21:03.458076+0000","last_clean":"2024-03-06T07:21:03.458076+0000","last_became_active":"2024-03-06T07:20:51.888426+0000","last_became_peered":"2024-03-06T07:20:51.888426+0000","last_unstale":"2024-03-06T07:21:03.458076+0000","last_undegraded":"2024-03-06T07:21:03.458076+0000","last_fullsized":"2024-03-06T07:21:03.458076+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:46.950695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460994+0000","last_change":"2024-03-06T07:21:02.465367+0000","last_active":"2024-03-06T07:21:03.460994+0000","last_peered":"2024-03-06T07:21:03.460994+0000","last_clean":"2024-03-06T07:21:03.460994+0000","last_became_active":"2024-03-06T07:21:02.465048+0000","last_became_peered":"2024-03-06T07:21:02.465048+0000","last_unstale":"2024-03-06T07:21:03.460994+0000","last_undegraded":"2024-03-06T07:21:03.460994+0000","last_fullsized":"2024-03-06T07:21:03.460994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:22.528081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460353+0000","last_change":"2024-03-06T07:20:51.885627+0000","last_active":"2024-03-06T07:21:03.460353+0000","last_peered":"2024-03-06T07:21:03.460353+0000","last_clean":"2024-03-06T07:21:03.460353+0000","last_became_active":"2024-03-06T07:20:51.885338+0000","last_became_peered":"2024-03-06T07:20:51.885338+0000","last_unstale":"2024-03-06T07:21:03.460353+0000","last_undegraded":"2024-03-06T07:21:03.460353+0000","last_fullsized":"2024-03-06T07:21:03.460353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:41.439891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460912+0000","last_change":"2024-03-06T07:21:02.470577+0000","last_active":"2024-03-06T07:21:03.460912+0000","last_peered":"2024-03-06T07:21:03.460912+0000","last_clean":"2024-03-06T07:21:03.460912+0000","last_became_active":"2024-03-06T07:21:02.470292+0000","last_became_peered":"2024-03-06T07:21:02.470292+0000","last_unstale":"2024-03-06T07:21:03.460912+0000","last_undegraded":"2024-03-06T07:21:03.460912+0000","last_fullsized":"2024-03-06T07:21:03.460912+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:24.369156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415326+0000","last_change":"2024-03-06T07:20:51.884448+0000","last_active":"2024-03-06T07:21:04.415326+0000","last_peered":"2024-03-06T07:21:04.415326+0000","last_clean":"2024-03-06T07:21:04.415326+0000","last_became_active":"2024-03-06T07:20:51.884176+0000","last_became_peered":"2024-03-06T07:20:51.884176+0000","last_unstale":"2024-03-06T07:21:04.415326+0000","last_undegraded":"2024-03-06T07:21:04.415326+0000","last_fullsized":"2024-03-06T07:21:04.415326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:29.691807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458298+0000","last_change":"2024-03-06T07:21:02.470206+0000","last_active":"2024-03-06T07:21:03.458298+0000","last_peered":"2024-03-06T07:21:03.458298+0000","last_clean":"2024-03-06T07:21:03.458298+0000","last_became_active":"2024-03-06T07:21:02.469925+0000","last_became_peered":"2024-03-06T07:21:02.469925+0000","last_unstale":"2024-03-06T07:21:03.458298+0000","last_undegraded":"2024-03-06T07:21:03.458298+0000","last_fullsized":"2024-03-06T07:21:03.458298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:10.132096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416126+0000","last_change":"2024-03-06T07:20:51.886062+0000","last_active":"2024-03-06T07:21:04.416126+0000","last_peered":"2024-03-06T07:21:04.416126+0000","last_clean":"2024-03-06T07:21:04.416126+0000","last_became_active":"2024-03-06T07:20:51.885799+0000","last_became_peered":"2024-03-06T07:20:51.885799+0000","last_unstale":"2024-03-06T07:21:04.416126+0000","last_undegraded":"2024-03-06T07:21:04.416126+0000","last_fullsized":"2024-03-06T07:21:04.416126+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914716+0000","last_change":"2024-03-06T07:21:02.469255+0000","last_active":"2024-03-06T07:21:03.914716+0000","last_peered":"2024-03-06T07:21:03.914716+0000","last_clean":"2024-03-06T07:21:03.914716+0000","last_became_active":"2024-03-06T07:21:02.468963+0000","last_became_peered":"2024-03-06T07:21:02.468963+0000","last_unstale":"2024-03-06T07:21:03.914716+0000","last_undegraded":"2024-03-06T07:21:03.914716+0000","last_fullsized":"2024-03-06T07:21:03.914716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:57.130162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415728+0000","last_change":"2024-03-06T07:20:51.883614+0000","last_active":"2024-03-06T07:21:04.415728+0000","last_peered":"2024-03-06T07:21:04.415728+0000","last_clean":"2024-03-06T07:21:04.415728+0000","last_became_active":"2024-03-06T07:20:51.883266+0000","last_became_peered":"2024-03-06T07:20:51.883266+0000","last_unstale":"2024-03-06T07:21:04.415728+0000","last_undegraded":"2024-03-06T07:21:04.415728+0000","last_fullsized":"2024-03-06T07:21:04.415728+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415466+0000","last_change":"2024-03-06T07:21:02.451939+0000","last_active":"2024-03-06T07:21:04.415466+0000","last_peered":"2024-03-06T07:21:04.415466+0000","last_clean":"2024-03-06T07:21:04.415466+0000","last_became_active":"2024-03-06T07:21:02.451486+0000","last_became_peered":"2024-03-06T07:21:02.451486+0000","last_unstale":"2024-03-06T07:21:04.415466+0000","last_undegraded":"2024-03-06T07:21:04.415466+0000","last_fullsized":"2024-03-06T07:21:04.415466+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:57.784614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415667+0000","last_change":"2024-03-06T07:20:51.893301+0000","last_active":"2024-03-06T07:21:04.415667+0000","last_peered":"2024-03-06T07:21:04.415667+0000","last_clean":"2024-03-06T07:21:04.415667+0000","last_became_active":"2024-03-06T07:20:51.892946+0000","last_became_peered":"2024-03-06T07:20:51.892946+0000","last_unstale":"2024-03-06T07:21:04.415667+0000","last_undegraded":"2024-03-06T07:21:04.415667+0000","last_fullsized":"2024-03-06T07:21:04.415667+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461138+0000","last_change":"2024-03-06T07:21:02.476949+0000","last_active":"2024-03-06T07:21:03.461138+0000","last_peered":"2024-03-06T07:21:03.461138+0000","last_clean":"2024-03-06T07:21:03.461138+0000","last_became_active":"2024-03-06T07:21:02.476650+0000","last_became_peered":"2024-03-06T07:21:02.476650+0000","last_unstale":"2024-03-06T07:21:03.461138+0000","last_undegraded":"2024-03-06T07:21:03.461138+0000","last_fullsized":"2024-03-06T07:21:03.461138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:41.908733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458097+0000","last_change":"2024-03-06T07:20:51.891689+0000","last_active":"2024-03-06T07:21:03.458097+0000","last_peered":"2024-03-06T07:21:03.458097+0000","last_clean":"2024-03-06T07:21:03.458097+0000","last_became_active":"2024-03-06T07:20:51.891341+0000","last_became_peered":"2024-03-06T07:20:51.891341+0000","last_unstale":"2024-03-06T07:21:03.458097+0000","last_undegraded":"2024-03-06T07:21:03.458097+0000","last_fullsized":"2024-03-06T07:21:03.458097+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:59:45.060281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415233+0000","last_change":"2024-03-06T07:21:02.462080+0000","last_active":"2024-03-06T07:21:04.415233+0000","last_peered":"2024-03-06T07:21:04.415233+0000","last_clean":"2024-03-06T07:21:04.415233+0000","last_became_active":"2024-03-06T07:21:02.461815+0000","last_became_peered":"2024-03-06T07:21:02.461815+0000","last_unstale":"2024-03-06T07:21:04.415233+0000","last_undegraded":"2024-03-06T07:21:04.415233+0000","last_fullsized":"2024-03-06T07:21:04.415233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:11.770007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416436+0000","last_change":"2024-03-06T07:20:51.887454+0000","last_active":"2024-03-06T07:21:04.416436+0000","last_peered":"2024-03-06T07:21:04.416436+0000","last_clean":"2024-03-06T07:21:04.416436+0000","last_became_active":"2024-03-06T07:20:51.887188+0000","last_became_peered":"2024-03-06T07:20:51.887188+0000","last_unstale":"2024-03-06T07:21:04.416436+0000","last_undegraded":"2024-03-06T07:21:04.416436+0000","last_fullsized":"2024-03-06T07:21:04.416436+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:05.770179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415070+0000","last_change":"2024-03-06T07:21:02.478126+0000","last_active":"2024-03-06T07:21:04.415070+0000","last_peered":"2024-03-06T07:21:04.415070+0000","last_clean":"2024-03-06T07:21:04.415070+0000","last_became_active":"2024-03-06T07:21:02.477872+0000","last_became_peered":"2024-03-06T07:21:02.477872+0000","last_unstale":"2024-03-06T07:21:04.415070+0000","last_undegraded":"2024-03-06T07:21:04.415070+0000","last_fullsized":"2024-03-06T07:21:04.415070+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:09.401096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415266+0000","last_change":"2024-03-06T07:20:51.899932+0000","last_active":"2024-03-06T07:21:04.415266+0000","last_peered":"2024-03-06T07:21:04.415266+0000","last_clean":"2024-03-06T07:21:04.415266+0000","last_became_active":"2024-03-06T07:20:51.899569+0000","last_became_peered":"2024-03-06T07:20:51.899569+0000","last_unstale":"2024-03-06T07:21:04.415266+0000","last_undegraded":"2024-03-06T07:21:04.415266+0000","last_fullsized":"2024-03-06T07:21:04.415266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:13.813050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415576+0000","last_change":"2024-03-06T07:21:02.459662+0000","last_active":"2024-03-06T07:21:04.415576+0000","last_peered":"2024-03-06T07:21:04.415576+0000","last_clean":"2024-03-06T07:21:04.415576+0000","last_became_active":"2024-03-06T07:21:02.459414+0000","last_became_peered":"2024-03-06T07:21:02.459414+0000","last_unstale":"2024-03-06T07:21:04.415576+0000","last_undegraded":"2024-03-06T07:21:04.415576+0000","last_fullsized":"2024-03-06T07:21:04.415576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:54.490222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914483+0000","last_change":"2024-03-06T07:20:51.888537+0000","last_active":"2024-03-06T07:21:03.914483+0000","last_peered":"2024-03-06T07:21:03.914483+0000","last_clean":"2024-03-06T07:21:03.914483+0000","last_became_active":"2024-03-06T07:20:51.888281+0000","last_became_peered":"2024-03-06T07:20:51.888281+0000","last_unstale":"2024-03-06T07:21:03.914483+0000","last_undegraded":"2024-03-06T07:21:03.914483+0000","last_fullsized":"2024-03-06T07:21:03.914483+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:03.657994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460786+0000","last_change":"2024-03-06T07:21:02.464751+0000","last_active":"2024-03-06T07:21:03.460786+0000","last_peered":"2024-03-06T07:21:03.460786+0000","last_clean":"2024-03-06T07:21:03.460786+0000","last_became_active":"2024-03-06T07:21:02.464460+0000","last_became_peered":"2024-03-06T07:21:02.464460+0000","last_unstale":"2024-03-06T07:21:03.460786+0000","last_undegraded":"2024-03-06T07:21:03.460786+0000","last_fullsized":"2024-03-06T07:21:03.460786+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:16.326286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415493+0000","last_change":"2024-03-06T07:20:51.896158+0000","last_active":"2024-03-06T07:21:04.415493+0000","last_peered":"2024-03-06T07:21:04.415493+0000","last_clean":"2024-03-06T07:21:04.415493+0000","last_became_active":"2024-03-06T07:20:51.895889+0000","last_became_peered":"2024-03-06T07:20:51.895889+0000","last_unstale":"2024-03-06T07:21:04.415493+0000","last_undegraded":"2024-03-06T07:21:04.415493+0000","last_fullsized":"2024-03-06T07:21:04.415493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:11.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415608+0000","last_change":"2024-03-06T07:21:02.460716+0000","last_active":"2024-03-06T07:21:04.415608+0000","last_peered":"2024-03-06T07:21:04.415608+0000","last_clean":"2024-03-06T07:21:04.415608+0000","last_became_active":"2024-03-06T07:21:02.459749+0000","last_became_peered":"2024-03-06T07:21:02.459749+0000","last_unstale":"2024-03-06T07:21:04.415608+0000","last_undegraded":"2024-03-06T07:21:04.415608+0000","last_fullsized":"2024-03-06T07:21:04.415608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:30.513765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415440+0000","last_change":"2024-03-06T07:21:01.440204+0000","last_active":"2024-03-06T07:21:04.415440+0000","last_peered":"2024-03-06T07:21:04.415440+0000","last_clean":"2024-03-06T07:21:04.415440+0000","last_became_active":"2024-03-06T07:21:01.439934+0000","last_became_peered":"2024-03-06T07:21:01.439934+0000","last_unstale":"2024-03-06T07:21:04.415440+0000","last_undegraded":"2024-03-06T07:21:04.415440+0000","last_fullsized":"2024-03-06T07:21:04.415440+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:07.781785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914422+0000","last_change":"2024-03-06T07:20:51.876393+0000","last_active":"2024-03-06T07:21:03.914422+0000","last_peered":"2024-03-06T07:21:03.914422+0000","last_clean":"2024-03-06T07:21:03.914422+0000","last_became_active":"2024-03-06T07:20:51.876059+0000","last_became_peered":"2024-03-06T07:20:51.876059+0000","last_unstale":"2024-03-06T07:21:03.914422+0000","last_undegraded":"2024-03-06T07:21:03.914422+0000","last_fullsized":"2024-03-06T07:21:03.914422+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"73'4","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458477+0000","last_change":"2024-03-06T07:20:50.879586+0000","last_active":"2024-03-06T07:21:03.458477+0000","last_peered":"2024-03-06T07:21:03.458477+0000","last_clean":"2024-03-06T07:21:03.458477+0000","last_became_active":"2024-03-06T07:20:50.879319+0000","last_became_peered":"2024-03-06T07:20:50.879319+0000","last_unstale":"2024-03-06T07:21:03.458477+0000","last_undegraded":"2024-03-06T07:21:03.458477+0000","last_fullsized":"2024-03-06T07:21:03.458477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:58.810754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914373+0000","last_change":"2024-03-06T07:21:02.457546+0000","last_active":"2024-03-06T07:21:03.914373+0000","last_peered":"2024-03-06T07:21:03.914373+0000","last_clean":"2024-03-06T07:21:03.914373+0000","last_became_active":"2024-03-06T07:21:02.457282+0000","last_became_peered":"2024-03-06T07:21:02.457282+0000","last_unstale":"2024-03-06T07:21:03.914373+0000","last_undegraded":"2024-03-06T07:21:03.914373+0000","last_fullsized":"2024-03-06T07:21:03.914373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:54.032813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415396+0000","last_change":"2024-03-06T07:21:01.436192+0000","last_active":"2024-03-06T07:21:04.415396+0000","last_peered":"2024-03-06T07:21:04.415396+0000","last_clean":"2024-03-06T07:21:04.415396+0000","last_became_active":"2024-03-06T07:21:01.435924+0000","last_became_peered":"2024-03-06T07:21:01.435924+0000","last_unstale":"2024-03-06T07:21:04.415396+0000","last_undegraded":"2024-03-06T07:21:04.415396+0000","last_fullsized":"2024-03-06T07:21:04.415396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:08.461930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415169+0000","last_change":"2024-03-06T07:20:51.884046+0000","last_active":"2024-03-06T07:21:04.415169+0000","last_peered":"2024-03-06T07:21:04.415169+0000","last_clean":"2024-03-06T07:21:04.415169+0000","last_became_active":"2024-03-06T07:20:51.883593+0000","last_became_peered":"2024-03-06T07:20:51.883593+0000","last_unstale":"2024-03-06T07:21:04.415169+0000","last_undegraded":"2024-03-06T07:21:04.415169+0000","last_fullsized":"2024-03-06T07:21:04.415169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:21.578818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415939+0000","last_change":"2024-03-06T07:20:50.888715+0000","last_active":"2024-03-06T07:21:04.415939+0000","last_peered":"2024-03-06T07:21:04.415939+0000","last_clean":"2024-03-06T07:21:04.415939+0000","last_became_active":"2024-03-06T07:20:50.888140+0000","last_became_peered":"2024-03-06T07:20:50.888140+0000","last_unstale":"2024-03-06T07:21:04.415939+0000","last_undegraded":"2024-03-06T07:21:04.415939+0000","last_fullsized":"2024-03-06T07:21:04.415939+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:34.087293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914220+0000","last_change":"2024-03-06T07:21:02.469857+0000","last_active":"2024-03-06T07:21:03.914220+0000","last_peered":"2024-03-06T07:21:03.914220+0000","last_clean":"2024-03-06T07:21:03.914220+0000","last_became_active":"2024-03-06T07:21:02.468597+0000","last_became_peered":"2024-03-06T07:21:02.468597+0000","last_unstale":"2024-03-06T07:21:03.914220+0000","last_undegraded":"2024-03-06T07:21:03.914220+0000","last_fullsized":"2024-03-06T07:21:03.914220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:37.167825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460867+0000","last_change":"2024-03-06T07:21:01.435195+0000","last_active":"2024-03-06T07:21:03.460867+0000","last_peered":"2024-03-06T07:21:03.460867+0000","last_clean":"2024-03-06T07:21:03.460867+0000","last_became_active":"2024-03-06T07:21:01.434964+0000","last_became_peered":"2024-03-06T07:21:01.434964+0000","last_unstale":"2024-03-06T07:21:03.460867+0000","last_undegraded":"2024-03-06T07:21:03.460867+0000","last_fullsized":"2024-03-06T07:21:03.460867+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:19.145983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460938+0000","last_change":"2024-03-06T07:20:51.886764+0000","last_active":"2024-03-06T07:21:03.460938+0000","last_peered":"2024-03-06T07:21:03.460938+0000","last_clean":"2024-03-06T07:21:03.460938+0000","last_became_active":"2024-03-06T07:20:51.886187+0000","last_became_peered":"2024-03-06T07:20:51.886187+0000","last_unstale":"2024-03-06T07:21:03.460938+0000","last_undegraded":"2024-03-06T07:21:03.460938+0000","last_fullsized":"2024-03-06T07:21:03.460938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:35.564191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415759+0000","last_change":"2024-03-06T07:20:50.885147+0000","last_active":"2024-03-06T07:21:04.415759+0000","last_peered":"2024-03-06T07:21:04.415759+0000","last_clean":"2024-03-06T07:21:04.415759+0000","last_became_active":"2024-03-06T07:20:50.884811+0000","last_became_peered":"2024-03-06T07:20:50.884811+0000","last_unstale":"2024-03-06T07:21:04.415759+0000","last_undegraded":"2024-03-06T07:21:04.415759+0000","last_fullsized":"2024-03-06T07:21:04.415759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:24.667879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459599+0000","last_change":"2024-03-06T07:21:02.462461+0000","last_active":"2024-03-06T07:21:03.459599+0000","last_peered":"2024-03-06T07:21:03.459599+0000","last_clean":"2024-03-06T07:21:03.459599+0000","last_became_active":"2024-03-06T07:21:02.462183+0000","last_became_peered":"2024-03-06T07:21:02.462183+0000","last_unstale":"2024-03-06T07:21:03.459599+0000","last_undegraded":"2024-03-06T07:21:03.459599+0000","last_fullsized":"2024-03-06T07:21:03.459599+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:37.167825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914641+0000","last_change":"2024-03-06T07:21:01.439913+0000","last_active":"2024-03-06T07:21:03.914641+0000","last_peered":"2024-03-06T07:21:03.914641+0000","last_clean":"2024-03-06T07:21:03.914641+0000","last_became_active":"2024-03-06T07:21:01.438882+0000","last_became_peered":"2024-03-06T07:21:01.438882+0000","last_unstale":"2024-03-06T07:21:03.914641+0000","last_undegraded":"2024-03-06T07:21:03.914641+0000","last_fullsized":"2024-03-06T07:21:03.914641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:16.291620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461093+0000","last_change":"2024-03-06T07:20:51.887534+0000","last_active":"2024-03-06T07:21:03.461093+0000","last_peered":"2024-03-06T07:21:03.461093+0000","last_clean":"2024-03-06T07:21:03.461093+0000","last_became_active":"2024-03-06T07:20:51.887239+0000","last_became_peered":"2024-03-06T07:20:51.887239+0000","last_unstale":"2024-03-06T07:21:03.461093+0000","last_undegraded":"2024-03-06T07:21:03.461093+0000","last_fullsized":"2024-03-06T07:21:03.461093+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:12.861753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460060+0000","last_change":"2024-03-06T07:20:50.892916+0000","last_active":"2024-03-06T07:21:03.460060+0000","last_peered":"2024-03-06T07:21:03.460060+0000","last_clean":"2024-03-06T07:21:03.460060+0000","last_became_active":"2024-03-06T07:20:50.892629+0000","last_became_peered":"2024-03-06T07:20:50.892629+0000","last_unstale":"2024-03-06T07:21:03.460060+0000","last_undegraded":"2024-03-06T07:21:03.460060+0000","last_fullsized":"2024-03-06T07:21:03.460060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:12.808948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415689+0000","last_change":"2024-03-06T07:21:02.465553+0000","last_active":"2024-03-06T07:21:04.415689+0000","last_peered":"2024-03-06T07:21:04.415689+0000","last_clean":"2024-03-06T07:21:04.415689+0000","last_became_active":"2024-03-06T07:21:02.465294+0000","last_became_peered":"2024-03-06T07:21:02.465294+0000","last_unstale":"2024-03-06T07:21:04.415689+0000","last_undegraded":"2024-03-06T07:21:04.415689+0000","last_fullsized":"2024-03-06T07:21:04.415689+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:53.404906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415641+0000","last_change":"2024-03-06T07:21:01.439106+0000","last_active":"2024-03-06T07:21:04.415641+0000","last_peered":"2024-03-06T07:21:04.415641+0000","last_clean":"2024-03-06T07:21:04.415641+0000","last_became_active":"2024-03-06T07:21:01.438835+0000","last_became_peered":"2024-03-06T07:21:01.438835+0000","last_unstale":"2024-03-06T07:21:04.415641+0000","last_undegraded":"2024-03-06T07:21:04.415641+0000","last_fullsized":"2024-03-06T07:21:04.415641+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:23.548727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416692+0000","last_change":"2024-03-06T07:20:51.895988+0000","last_active":"2024-03-06T07:21:04.416692+0000","last_peered":"2024-03-06T07:21:04.416692+0000","last_clean":"2024-03-06T07:21:04.416692+0000","last_became_active":"2024-03-06T07:20:51.895747+0000","last_became_peered":"2024-03-06T07:20:51.895747+0000","last_unstale":"2024-03-06T07:21:04.416692+0000","last_undegraded":"2024-03-06T07:21:04.416692+0000","last_fullsized":"2024-03-06T07:21:04.416692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:55.215337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461193+0000","last_change":"2024-03-06T07:20:50.893482+0000","last_active":"2024-03-06T07:21:03.461193+0000","last_peered":"2024-03-06T07:21:03.461193+0000","last_clean":"2024-03-06T07:21:03.461193+0000","last_became_active":"2024-03-06T07:20:50.893153+0000","last_became_peered":"2024-03-06T07:20:50.893153+0000","last_unstale":"2024-03-06T07:21:03.461193+0000","last_undegraded":"2024-03-06T07:21:03.461193+0000","last_fullsized":"2024-03-06T07:21:03.461193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:07.775768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416039+0000","last_change":"2024-03-06T07:21:02.467363+0000","last_active":"2024-03-06T07:21:04.416039+0000","last_peered":"2024-03-06T07:21:04.416039+0000","last_clean":"2024-03-06T07:21:04.416039+0000","last_became_active":"2024-03-06T07:21:02.466919+0000","last_became_peered":"2024-03-06T07:21:02.466919+0000","last_unstale":"2024-03-06T07:21:04.416039+0000","last_undegraded":"2024-03-06T07:21:04.416039+0000","last_fullsized":"2024-03-06T07:21:04.416039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:01.950506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459685+0000","last_change":"2024-03-06T07:21:01.442216+0000","last_active":"2024-03-06T07:21:03.459685+0000","last_peered":"2024-03-06T07:21:03.459685+0000","last_clean":"2024-03-06T07:21:03.459685+0000","last_became_active":"2024-03-06T07:21:01.441772+0000","last_became_peered":"2024-03-06T07:21:01.441772+0000","last_unstale":"2024-03-06T07:21:03.459685+0000","last_undegraded":"2024-03-06T07:21:03.459685+0000","last_fullsized":"2024-03-06T07:21:03.459685+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:35.227455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416206+0000","last_change":"2024-03-06T07:20:51.897529+0000","last_active":"2024-03-06T07:21:04.416206+0000","last_peered":"2024-03-06T07:21:04.416206+0000","last_clean":"2024-03-06T07:21:04.416206+0000","last_became_active":"2024-03-06T07:20:51.897181+0000","last_became_peered":"2024-03-06T07:20:51.897181+0000","last_unstale":"2024-03-06T07:21:04.416206+0000","last_undegraded":"2024-03-06T07:21:04.416206+0000","last_fullsized":"2024-03-06T07:21:04.416206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415393+0000","last_change":"2024-03-06T07:20:50.886303+0000","last_active":"2024-03-06T07:21:04.415393+0000","last_peered":"2024-03-06T07:21:04.415393+0000","last_clean":"2024-03-06T07:21:04.415393+0000","last_became_active":"2024-03-06T07:20:50.886013+0000","last_became_peered":"2024-03-06T07:20:50.886013+0000","last_unstale":"2024-03-06T07:21:04.415393+0000","last_undegraded":"2024-03-06T07:21:04.415393+0000","last_fullsized":"2024-03-06T07:21:04.415393+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:34.617731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416618+0000","last_change":"2024-03-06T07:21:02.474796+0000","last_active":"2024-03-06T07:21:04.416618+0000","last_peered":"2024-03-06T07:21:04.416618+0000","last_clean":"2024-03-06T07:21:04.416618+0000","last_became_active":"2024-03-06T07:21:02.474530+0000","last_became_peered":"2024-03-06T07:21:02.474530+0000","last_unstale":"2024-03-06T07:21:04.416618+0000","last_undegraded":"2024-03-06T07:21:04.416618+0000","last_fullsized":"2024-03-06T07:21:04.416618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:08.979985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"77'4","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459611+0000","last_change":"2024-03-06T07:21:01.434722+0000","last_active":"2024-03-06T07:21:03.459611+0000","last_peered":"2024-03-06T07:21:03.459611+0000","last_clean":"2024-03-06T07:21:03.459611+0000","last_became_active":"2024-03-06T07:21:01.434380+0000","last_became_peered":"2024-03-06T07:21:01.434380+0000","last_unstale":"2024-03-06T07:21:03.459611+0000","last_undegraded":"2024-03-06T07:21:03.459611+0000","last_fullsized":"2024-03-06T07:21:03.459611+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:46.428417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461624+0000","last_change":"2024-03-06T07:20:51.883834+0000","last_active":"2024-03-06T07:21:03.461624+0000","last_peered":"2024-03-06T07:21:03.461624+0000","last_clean":"2024-03-06T07:21:03.461624+0000","last_became_active":"2024-03-06T07:20:51.883553+0000","last_became_peered":"2024-03-06T07:20:51.883553+0000","last_unstale":"2024-03-06T07:21:03.461624+0000","last_undegraded":"2024-03-06T07:21:03.461624+0000","last_fullsized":"2024-03-06T07:21:03.461624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:59.575990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415299+0000","last_change":"2024-03-06T07:20:50.881449+0000","last_active":"2024-03-06T07:21:04.415299+0000","last_peered":"2024-03-06T07:21:04.415299+0000","last_clean":"2024-03-06T07:21:04.415299+0000","last_became_active":"2024-03-06T07:20:50.881172+0000","last_became_peered":"2024-03-06T07:20:50.881172+0000","last_unstale":"2024-03-06T07:21:04.415299+0000","last_undegraded":"2024-03-06T07:21:04.415299+0000","last_fullsized":"2024-03-06T07:21:04.415299+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.317522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415646+0000","last_change":"2024-03-06T07:21:02.460587+0000","last_active":"2024-03-06T07:21:04.415646+0000","last_peered":"2024-03-06T07:21:04.415646+0000","last_clean":"2024-03-06T07:21:04.415646+0000","last_became_active":"2024-03-06T07:21:02.460313+0000","last_became_peered":"2024-03-06T07:21:02.460313+0000","last_unstale":"2024-03-06T07:21:04.415646+0000","last_undegraded":"2024-03-06T07:21:04.415646+0000","last_fullsized":"2024-03-06T07:21:04.415646+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:06.562354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461240+0000","last_change":"2024-03-06T07:21:01.431035+0000","last_active":"2024-03-06T07:21:03.461240+0000","last_peered":"2024-03-06T07:21:03.461240+0000","last_clean":"2024-03-06T07:21:03.461240+0000","last_became_active":"2024-03-06T07:21:01.430787+0000","last_became_peered":"2024-03-06T07:21:01.430787+0000","last_unstale":"2024-03-06T07:21:03.461240+0000","last_undegraded":"2024-03-06T07:21:03.461240+0000","last_fullsized":"2024-03-06T07:21:03.461240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:04.021851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459936+0000","last_change":"2024-03-06T07:20:51.893411+0000","last_active":"2024-03-06T07:21:03.459936+0000","last_peered":"2024-03-06T07:21:03.459936+0000","last_clean":"2024-03-06T07:21:03.459936+0000","last_became_active":"2024-03-06T07:20:51.893149+0000","last_became_peered":"2024-03-06T07:20:51.893149+0000","last_unstale":"2024-03-06T07:21:03.459936+0000","last_undegraded":"2024-03-06T07:21:03.459936+0000","last_fullsized":"2024-03-06T07:21:03.459936+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457736+0000","last_change":"2024-03-06T07:20:50.890159+0000","last_active":"2024-03-06T07:21:03.457736+0000","last_peered":"2024-03-06T07:21:03.457736+0000","last_clean":"2024-03-06T07:21:03.457736+0000","last_became_active":"2024-03-06T07:20:50.889863+0000","last_became_peered":"2024-03-06T07:20:50.889863+0000","last_unstale":"2024-03-06T07:21:03.457736+0000","last_undegraded":"2024-03-06T07:21:03.457736+0000","last_fullsized":"2024-03-06T07:21:03.457736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:42.303590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457848+0000","last_change":"2024-03-06T07:21:02.463592+0000","last_active":"2024-03-06T07:21:03.457848+0000","last_peered":"2024-03-06T07:21:03.457848+0000","last_clean":"2024-03-06T07:21:03.457848+0000","last_became_active":"2024-03-06T07:21:02.463201+0000","last_became_peered":"2024-03-06T07:21:02.463201+0000","last_unstale":"2024-03-06T07:21:03.457848+0000","last_undegraded":"2024-03-06T07:21:03.457848+0000","last_fullsized":"2024-03-06T07:21:03.457848+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:08.955372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914318+0000","last_change":"2024-03-06T07:21:01.440880+0000","last_active":"2024-03-06T07:21:03.914318+0000","last_peered":"2024-03-06T07:21:03.914318+0000","last_clean":"2024-03-06T07:21:03.914318+0000","last_became_active":"2024-03-06T07:21:01.440583+0000","last_became_peered":"2024-03-06T07:21:01.440583+0000","last_unstale":"2024-03-06T07:21:03.914318+0000","last_undegraded":"2024-03-06T07:21:03.914318+0000","last_fullsized":"2024-03-06T07:21:03.914318+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:43.973004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415423+0000","last_change":"2024-03-06T07:20:51.899199+0000","last_active":"2024-03-06T07:21:04.415423+0000","last_peered":"2024-03-06T07:21:04.415423+0000","last_clean":"2024-03-06T07:21:04.415423+0000","last_became_active":"2024-03-06T07:20:51.898622+0000","last_became_peered":"2024-03-06T07:20:51.898622+0000","last_unstale":"2024-03-06T07:21:04.415423+0000","last_undegraded":"2024-03-06T07:21:04.415423+0000","last_fullsized":"2024-03-06T07:21:04.415423+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:57.053514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416352+0000","last_change":"2024-03-06T07:20:50.891619+0000","last_active":"2024-03-06T07:21:04.416352+0000","last_peered":"2024-03-06T07:21:04.416352+0000","last_clean":"2024-03-06T07:21:04.416352+0000","last_became_active":"2024-03-06T07:20:50.891305+0000","last_became_peered":"2024-03-06T07:20:50.891305+0000","last_unstale":"2024-03-06T07:21:04.416352+0000","last_undegraded":"2024-03-06T07:21:04.416352+0000","last_fullsized":"2024-03-06T07:21:04.416352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:05.732044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458281+0000","last_change":"2024-03-06T07:21:02.469888+0000","last_active":"2024-03-06T07:21:03.458281+0000","last_peered":"2024-03-06T07:21:03.458281+0000","last_clean":"2024-03-06T07:21:03.458281+0000","last_became_active":"2024-03-06T07:21:02.469626+0000","last_became_peered":"2024-03-06T07:21:02.469626+0000","last_unstale":"2024-03-06T07:21:03.458281+0000","last_undegraded":"2024-03-06T07:21:03.458281+0000","last_fullsized":"2024-03-06T07:21:03.458281+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:23.417859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"77'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914325+0000","last_change":"2024-03-06T07:21:01.439974+0000","last_active":"2024-03-06T07:21:03.914325+0000","last_peered":"2024-03-06T07:21:03.914325+0000","last_clean":"2024-03-06T07:21:03.914325+0000","last_became_active":"2024-03-06T07:21:01.439682+0000","last_became_peered":"2024-03-06T07:21:01.439682+0000","last_unstale":"2024-03-06T07:21:03.914325+0000","last_undegraded":"2024-03-06T07:21:03.914325+0000","last_fullsized":"2024-03-06T07:21:03.914325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:44.646869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415260+0000","last_change":"2024-03-06T07:20:51.887972+0000","last_active":"2024-03-06T07:21:04.415260+0000","last_peered":"2024-03-06T07:21:04.415260+0000","last_clean":"2024-03-06T07:21:04.415260+0000","last_became_active":"2024-03-06T07:20:51.887725+0000","last_became_peered":"2024-03-06T07:20:51.887725+0000","last_unstale":"2024-03-06T07:21:04.415260+0000","last_undegraded":"2024-03-06T07:21:04.415260+0000","last_fullsized":"2024-03-06T07:21:04.415260+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:33.212910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415817+0000","last_change":"2024-03-06T07:20:50.890731+0000","last_active":"2024-03-06T07:21:04.415817+0000","last_peered":"2024-03-06T07:21:04.415817+0000","last_clean":"2024-03-06T07:21:04.415817+0000","last_became_active":"2024-03-06T07:20:50.890429+0000","last_became_peered":"2024-03-06T07:20:50.890429+0000","last_unstale":"2024-03-06T07:21:04.415817+0000","last_undegraded":"2024-03-06T07:21:04.415817+0000","last_fullsized":"2024-03-06T07:21:04.415817+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914444+0000","last_change":"2024-03-06T07:21:02.464657+0000","last_active":"2024-03-06T07:21:03.914444+0000","last_peered":"2024-03-06T07:21:03.914444+0000","last_clean":"2024-03-06T07:21:03.914444+0000","last_became_active":"2024-03-06T07:21:02.460767+0000","last_became_peered":"2024-03-06T07:21:02.460767+0000","last_unstale":"2024-03-06T07:21:03.914444+0000","last_undegraded":"2024-03-06T07:21:03.914444+0000","last_fullsized":"2024-03-06T07:21:03.914444+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:07.001720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415728+0000","last_change":"2024-03-06T07:21:01.436794+0000","last_active":"2024-03-06T07:21:04.415728+0000","last_peered":"2024-03-06T07:21:04.415728+0000","last_clean":"2024-03-06T07:21:04.415728+0000","last_became_active":"2024-03-06T07:21:01.436556+0000","last_became_peered":"2024-03-06T07:21:01.436556+0000","last_unstale":"2024-03-06T07:21:04.415728+0000","last_undegraded":"2024-03-06T07:21:04.415728+0000","last_fullsized":"2024-03-06T07:21:04.415728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:27.209234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415938+0000","last_change":"2024-03-06T07:20:51.885412+0000","last_active":"2024-03-06T07:21:04.415938+0000","last_peered":"2024-03-06T07:21:04.415938+0000","last_clean":"2024-03-06T07:21:04.415938+0000","last_became_active":"2024-03-06T07:20:51.885153+0000","last_became_peered":"2024-03-06T07:20:51.885153+0000","last_unstale":"2024-03-06T07:21:04.415938+0000","last_undegraded":"2024-03-06T07:21:04.415938+0000","last_fullsized":"2024-03-06T07:21:04.415938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:07.592958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"73'3","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458312+0000","last_change":"2024-03-06T07:20:50.880143+0000","last_active":"2024-03-06T07:21:03.458312+0000","last_peered":"2024-03-06T07:21:03.458312+0000","last_clean":"2024-03-06T07:21:03.458312+0000","last_became_active":"2024-03-06T07:20:50.879859+0000","last_became_peered":"2024-03-06T07:20:50.879859+0000","last_unstale":"2024-03-06T07:21:03.458312+0000","last_undegraded":"2024-03-06T07:21:03.458312+0000","last_fullsized":"2024-03-06T07:21:03.458312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:42:08.684501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460242+0000","last_change":"2024-03-06T07:21:02.465387+0000","last_active":"2024-03-06T07:21:03.460242+0000","last_peered":"2024-03-06T07:21:03.460242+0000","last_clean":"2024-03-06T07:21:03.460242+0000","last_became_active":"2024-03-06T07:21:02.465131+0000","last_became_peered":"2024-03-06T07:21:02.465131+0000","last_unstale":"2024-03-06T07:21:03.460242+0000","last_undegraded":"2024-03-06T07:21:03.460242+0000","last_fullsized":"2024-03-06T07:21:03.460242+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:26.316211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415450+0000","last_change":"2024-03-06T07:21:01.432460+0000","last_active":"2024-03-06T07:21:04.415450+0000","last_peered":"2024-03-06T07:21:04.415450+0000","last_clean":"2024-03-06T07:21:04.415450+0000","last_became_active":"2024-03-06T07:21:01.432169+0000","last_became_peered":"2024-03-06T07:21:01.432169+0000","last_unstale":"2024-03-06T07:21:04.415450+0000","last_undegraded":"2024-03-06T07:21:04.415450+0000","last_fullsized":"2024-03-06T07:21:04.415450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:38.195559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914690+0000","last_change":"2024-03-06T07:20:51.878936+0000","last_active":"2024-03-06T07:21:03.914690+0000","last_peered":"2024-03-06T07:21:03.914690+0000","last_clean":"2024-03-06T07:21:03.914690+0000","last_became_active":"2024-03-06T07:20:51.878613+0000","last_became_peered":"2024-03-06T07:20:51.878613+0000","last_unstale":"2024-03-06T07:21:03.914690+0000","last_undegraded":"2024-03-06T07:21:03.914690+0000","last_fullsized":"2024-03-06T07:21:03.914690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"73'6","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460154+0000","last_change":"2024-03-06T07:20:50.888110+0000","last_active":"2024-03-06T07:21:03.460154+0000","last_peered":"2024-03-06T07:21:03.460154+0000","last_clean":"2024-03-06T07:21:03.460154+0000","last_became_active":"2024-03-06T07:20:50.887804+0000","last_became_peered":"2024-03-06T07:20:50.887804+0000","last_unstale":"2024-03-06T07:21:03.460154+0000","last_undegraded":"2024-03-06T07:21:03.460154+0000","last_fullsized":"2024-03-06T07:21:03.460154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416106+0000","last_change":"2024-03-06T07:21:02.470412+0000","last_active":"2024-03-06T07:21:04.416106+0000","last_peered":"2024-03-06T07:21:04.416106+0000","last_clean":"2024-03-06T07:21:04.416106+0000","last_became_active":"2024-03-06T07:21:02.469904+0000","last_became_peered":"2024-03-06T07:21:02.469904+0000","last_unstale":"2024-03-06T07:21:04.416106+0000","last_undegraded":"2024-03-06T07:21:04.416106+0000","last_fullsized":"2024-03-06T07:21:04.416106+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:26.069032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"77'3","reported_seq":17,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461424+0000","last_change":"2024-03-06T07:21:01.433077+0000","last_active":"2024-03-06T07:21:03.461424+0000","last_peered":"2024-03-06T07:21:03.461424+0000","last_clean":"2024-03-06T07:21:03.461424+0000","last_became_active":"2024-03-06T07:21:01.432684+0000","last_became_peered":"2024-03-06T07:21:01.432684+0000","last_unstale":"2024-03-06T07:21:03.461424+0000","last_undegraded":"2024-03-06T07:21:03.461424+0000","last_fullsized":"2024-03-06T07:21:03.461424+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:46.293468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416009+0000","last_change":"2024-03-06T07:20:51.891149+0000","last_active":"2024-03-06T07:21:04.416009+0000","last_peered":"2024-03-06T07:21:04.416009+0000","last_clean":"2024-03-06T07:21:04.416009+0000","last_became_active":"2024-03-06T07:20:51.890877+0000","last_became_peered":"2024-03-06T07:20:51.890877+0000","last_unstale":"2024-03-06T07:21:04.416009+0000","last_undegraded":"2024-03-06T07:21:04.416009+0000","last_fullsized":"2024-03-06T07:21:04.416009+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415203+0000","last_change":"2024-03-06T07:20:50.888227+0000","last_active":"2024-03-06T07:21:04.415203+0000","last_peered":"2024-03-06T07:21:04.415203+0000","last_clean":"2024-03-06T07:21:04.415203+0000","last_became_active":"2024-03-06T07:20:50.887926+0000","last_became_peered":"2024-03-06T07:20:50.887926+0000","last_unstale":"2024-03-06T07:21:04.415203+0000","last_undegraded":"2024-03-06T07:21:04.415203+0000","last_fullsized":"2024-03-06T07:21:04.415203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:56.255517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914918+0000","last_change":"2024-03-06T07:21:02.458615+0000","last_active":"2024-03-06T07:21:03.914918+0000","last_peered":"2024-03-06T07:21:03.914918+0000","last_clean":"2024-03-06T07:21:03.914918+0000","last_became_active":"2024-03-06T07:21:02.458342+0000","last_became_peered":"2024-03-06T07:21:02.458342+0000","last_unstale":"2024-03-06T07:21:03.914918+0000","last_undegraded":"2024-03-06T07:21:03.914918+0000","last_fullsized":"2024-03-06T07:21:03.914918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:39.197159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"77'6","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416539+0000","last_change":"2024-03-06T07:21:01.432947+0000","last_active":"2024-03-06T07:21:04.416539+0000","last_peered":"2024-03-06T07:21:04.416539+0000","last_clean":"2024-03-06T07:21:04.416539+0000","last_became_active":"2024-03-06T07:21:01.432263+0000","last_became_peered":"2024-03-06T07:21:01.432263+0000","last_unstale":"2024-03-06T07:21:04.416539+0000","last_undegraded":"2024-03-06T07:21:04.416539+0000","last_fullsized":"2024-03-06T07:21:04.416539+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:37.148955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459327+0000","last_change":"2024-03-06T07:20:51.888556+0000","last_active":"2024-03-06T07:21:03.459327+0000","last_peered":"2024-03-06T07:21:03.459327+0000","last_clean":"2024-03-06T07:21:03.459327+0000","last_became_active":"2024-03-06T07:20:51.888278+0000","last_became_peered":"2024-03-06T07:20:51.888278+0000","last_unstale":"2024-03-06T07:21:03.459327+0000","last_undegraded":"2024-03-06T07:21:03.459327+0000","last_fullsized":"2024-03-06T07:21:03.459327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:38:02.354319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415369+0000","last_change":"2024-03-06T07:20:50.881110+0000","last_active":"2024-03-06T07:21:04.415369+0000","last_peered":"2024-03-06T07:21:04.415369+0000","last_clean":"2024-03-06T07:21:04.415369+0000","last_became_active":"2024-03-06T07:20:50.880752+0000","last_became_peered":"2024-03-06T07:20:50.880752+0000","last_unstale":"2024-03-06T07:21:04.415369+0000","last_undegraded":"2024-03-06T07:21:04.415369+0000","last_fullsized":"2024-03-06T07:21:04.415369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.479321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461110+0000","last_change":"2024-03-06T07:21:02.468850+0000","last_active":"2024-03-06T07:21:03.461110+0000","last_peered":"2024-03-06T07:21:03.461110+0000","last_clean":"2024-03-06T07:21:03.461110+0000","last_became_active":"2024-03-06T07:21:02.468611+0000","last_became_peered":"2024-03-06T07:21:02.468611+0000","last_unstale":"2024-03-06T07:21:03.461110+0000","last_undegraded":"2024-03-06T07:21:03.461110+0000","last_fullsized":"2024-03-06T07:21:03.461110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:37.167825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415697+0000","last_change":"2024-03-06T07:21:01.435628+0000","last_active":"2024-03-06T07:21:04.415697+0000","last_peered":"2024-03-06T07:21:04.415697+0000","last_clean":"2024-03-06T07:21:04.415697+0000","last_became_active":"2024-03-06T07:21:01.434996+0000","last_became_peered":"2024-03-06T07:21:01.434996+0000","last_unstale":"2024-03-06T07:21:04.415697+0000","last_undegraded":"2024-03-06T07:21:04.415697+0000","last_fullsized":"2024-03-06T07:21:04.415697+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:47.661259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459576+0000","last_change":"2024-03-06T07:20:51.879914+0000","last_active":"2024-03-06T07:21:03.459576+0000","last_peered":"2024-03-06T07:21:03.459576+0000","last_clean":"2024-03-06T07:21:03.459576+0000","last_became_active":"2024-03-06T07:20:51.879633+0000","last_became_peered":"2024-03-06T07:20:51.879633+0000","last_unstale":"2024-03-06T07:21:03.459576+0000","last_undegraded":"2024-03-06T07:21:03.459576+0000","last_fullsized":"2024-03-06T07:21:03.459576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:36.738962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457680+0000","last_change":"2024-03-06T07:20:50.880559+0000","last_active":"2024-03-06T07:21:03.457680+0000","last_peered":"2024-03-06T07:21:03.457680+0000","last_clean":"2024-03-06T07:21:03.457680+0000","last_became_active":"2024-03-06T07:20:50.880160+0000","last_became_peered":"2024-03-06T07:20:50.880160+0000","last_unstale":"2024-03-06T07:21:03.457680+0000","last_undegraded":"2024-03-06T07:21:03.457680+0000","last_fullsized":"2024-03-06T07:21:03.457680+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:56.900930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415914+0000","last_change":"2024-03-06T07:21:02.467558+0000","last_active":"2024-03-06T07:21:04.415914+0000","last_peered":"2024-03-06T07:21:04.415914+0000","last_clean":"2024-03-06T07:21:04.415914+0000","last_became_active":"2024-03-06T07:21:02.467291+0000","last_became_peered":"2024-03-06T07:21:02.467291+0000","last_unstale":"2024-03-06T07:21:04.415914+0000","last_undegraded":"2024-03-06T07:21:04.415914+0000","last_fullsized":"2024-03-06T07:21:04.415914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:30.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416196+0000","last_change":"2024-03-06T07:21:01.441770+0000","last_active":"2024-03-06T07:21:04.416196+0000","last_peered":"2024-03-06T07:21:04.416196+0000","last_clean":"2024-03-06T07:21:04.416196+0000","last_became_active":"2024-03-06T07:21:01.441478+0000","last_became_peered":"2024-03-06T07:21:01.441478+0000","last_unstale":"2024-03-06T07:21:04.416196+0000","last_undegraded":"2024-03-06T07:21:04.416196+0000","last_fullsized":"2024-03-06T07:21:04.416196+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:27.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460006+0000","last_change":"2024-03-06T07:20:51.890440+0000","last_active":"2024-03-06T07:21:03.460006+0000","last_peered":"2024-03-06T07:21:03.460006+0000","last_clean":"2024-03-06T07:21:03.460006+0000","last_became_active":"2024-03-06T07:20:51.890167+0000","last_became_peered":"2024-03-06T07:20:51.890167+0000","last_unstale":"2024-03-06T07:21:03.460006+0000","last_undegraded":"2024-03-06T07:21:03.460006+0000","last_fullsized":"2024-03-06T07:21:03.460006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:53.956654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"73'8","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415597+0000","last_change":"2024-03-06T07:20:50.885352+0000","last_active":"2024-03-06T07:21:04.415597+0000","last_peered":"2024-03-06T07:21:04.415597+0000","last_clean":"2024-03-06T07:21:04.415597+0000","last_became_active":"2024-03-06T07:20:50.884991+0000","last_became_peered":"2024-03-06T07:20:50.884991+0000","last_unstale":"2024-03-06T07:21:04.415597+0000","last_undegraded":"2024-03-06T07:21:04.415597+0000","last_fullsized":"2024-03-06T07:21:04.415597+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:03.652296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415302+0000","last_change":"2024-03-06T07:21:02.451731+0000","last_active":"2024-03-06T07:21:04.415302+0000","last_peered":"2024-03-06T07:21:04.415302+0000","last_clean":"2024-03-06T07:21:04.415302+0000","last_became_active":"2024-03-06T07:21:02.451439+0000","last_became_peered":"2024-03-06T07:21:02.451439+0000","last_unstale":"2024-03-06T07:21:04.415302+0000","last_undegraded":"2024-03-06T07:21:04.415302+0000","last_fullsized":"2024-03-06T07:21:04.415302+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:08.783385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415054+0000","last_change":"2024-03-06T07:21:01.436096+0000","last_active":"2024-03-06T07:21:04.415054+0000","last_peered":"2024-03-06T07:21:04.415054+0000","last_clean":"2024-03-06T07:21:04.415054+0000","last_became_active":"2024-03-06T07:21:01.435767+0000","last_became_peered":"2024-03-06T07:21:01.435767+0000","last_unstale":"2024-03-06T07:21:04.415054+0000","last_undegraded":"2024-03-06T07:21:04.415054+0000","last_fullsized":"2024-03-06T07:21:04.415054+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:19.463406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415939+0000","last_change":"2024-03-06T07:20:51.889501+0000","last_active":"2024-03-06T07:21:04.415939+0000","last_peered":"2024-03-06T07:21:04.415939+0000","last_clean":"2024-03-06T07:21:04.415939+0000","last_became_active":"2024-03-06T07:20:51.889206+0000","last_became_peered":"2024-03-06T07:20:51.889206+0000","last_unstale":"2024-03-06T07:21:04.415939+0000","last_undegraded":"2024-03-06T07:21:04.415939+0000","last_fullsized":"2024-03-06T07:21:04.415939+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:09.908900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416410+0000","last_change":"2024-03-06T07:20:50.883039+0000","last_active":"2024-03-06T07:21:04.416410+0000","last_peered":"2024-03-06T07:21:04.416410+0000","last_clean":"2024-03-06T07:21:04.416410+0000","last_became_active":"2024-03-06T07:20:50.882748+0000","last_became_peered":"2024-03-06T07:20:50.882748+0000","last_unstale":"2024-03-06T07:21:04.416410+0000","last_undegraded":"2024-03-06T07:21:04.416410+0000","last_fullsized":"2024-03-06T07:21:04.416410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:35.868841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416098+0000","last_change":"2024-03-06T07:21:02.457670+0000","last_active":"2024-03-06T07:21:04.416098+0000","last_peered":"2024-03-06T07:21:04.416098+0000","last_clean":"2024-03-06T07:21:04.416098+0000","last_became_active":"2024-03-06T07:21:02.457228+0000","last_became_peered":"2024-03-06T07:21:02.457228+0000","last_unstale":"2024-03-06T07:21:04.416098+0000","last_undegraded":"2024-03-06T07:21:04.416098+0000","last_fullsized":"2024-03-06T07:21:04.416098+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:31.198258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457595+0000","last_change":"2024-03-06T07:21:01.438861+0000","last_active":"2024-03-06T07:21:03.457595+0000","last_peered":"2024-03-06T07:21:03.457595+0000","last_clean":"2024-03-06T07:21:03.457595+0000","last_became_active":"2024-03-06T07:21:01.438577+0000","last_became_peered":"2024-03-06T07:21:01.438577+0000","last_unstale":"2024-03-06T07:21:03.457595+0000","last_undegraded":"2024-03-06T07:21:03.457595+0000","last_fullsized":"2024-03-06T07:21:03.457595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:45.275401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457646+0000","last_change":"2024-03-06T07:20:51.890860+0000","last_active":"2024-03-06T07:21:03.457646+0000","last_peered":"2024-03-06T07:21:03.457646+0000","last_clean":"2024-03-06T07:21:03.457646+0000","last_became_active":"2024-03-06T07:20:51.890576+0000","last_became_peered":"2024-03-06T07:20:51.890576+0000","last_unstale":"2024-03-06T07:21:03.457646+0000","last_undegraded":"2024-03-06T07:21:03.457646+0000","last_fullsized":"2024-03-06T07:21:03.457646+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:02.364180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461655+0000","last_change":"2024-03-06T07:20:50.888701+0000","last_active":"2024-03-06T07:21:03.461655+0000","last_peered":"2024-03-06T07:21:03.461655+0000","last_clean":"2024-03-06T07:21:03.461655+0000","last_became_active":"2024-03-06T07:20:50.888392+0000","last_became_peered":"2024-03-06T07:20:50.888392+0000","last_unstale":"2024-03-06T07:21:03.461655+0000","last_undegraded":"2024-03-06T07:21:03.461655+0000","last_fullsized":"2024-03-06T07:21:03.461655+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:07.582638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415153+0000","last_change":"2024-03-06T07:21:02.468531+0000","last_active":"2024-03-06T07:21:04.415153+0000","last_peered":"2024-03-06T07:21:04.415153+0000","last_clean":"2024-03-06T07:21:04.415153+0000","last_became_active":"2024-03-06T07:21:02.468282+0000","last_became_peered":"2024-03-06T07:21:02.468282+0000","last_unstale":"2024-03-06T07:21:04.415153+0000","last_undegraded":"2024-03-06T07:21:04.415153+0000","last_fullsized":"2024-03-06T07:21:04.415153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:26.069032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415093+0000","last_change":"2024-03-06T07:21:01.436497+0000","last_active":"2024-03-06T07:21:04.415093+0000","last_peered":"2024-03-06T07:21:04.415093+0000","last_clean":"2024-03-06T07:21:04.415093+0000","last_became_active":"2024-03-06T07:21:01.436029+0000","last_became_peered":"2024-03-06T07:21:01.436029+0000","last_unstale":"2024-03-06T07:21:04.415093+0000","last_undegraded":"2024-03-06T07:21:04.415093+0000","last_fullsized":"2024-03-06T07:21:04.415093+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:53.031212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457891+0000","last_change":"2024-03-06T07:20:51.887239+0000","last_active":"2024-03-06T07:21:03.457891+0000","last_peered":"2024-03-06T07:21:03.457891+0000","last_clean":"2024-03-06T07:21:03.457891+0000","last_became_active":"2024-03-06T07:20:51.886940+0000","last_became_peered":"2024-03-06T07:20:51.886940+0000","last_unstale":"2024-03-06T07:21:03.457891+0000","last_undegraded":"2024-03-06T07:21:03.457891+0000","last_fullsized":"2024-03-06T07:21:03.457891+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:08.123116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458390+0000","last_change":"2024-03-06T07:20:50.886879+0000","last_active":"2024-03-06T07:21:03.458390+0000","last_peered":"2024-03-06T07:21:03.458390+0000","last_clean":"2024-03-06T07:21:03.458390+0000","last_became_active":"2024-03-06T07:20:50.886589+0000","last_became_peered":"2024-03-06T07:20:50.886589+0000","last_unstale":"2024-03-06T07:21:03.458390+0000","last_undegraded":"2024-03-06T07:21:03.458390+0000","last_fullsized":"2024-03-06T07:21:03.458390+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:51.245367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415077+0000","last_change":"2024-03-06T07:21:02.467756+0000","last_active":"2024-03-06T07:21:04.415077+0000","last_peered":"2024-03-06T07:21:04.415077+0000","last_clean":"2024-03-06T07:21:04.415077+0000","last_became_active":"2024-03-06T07:21:02.467466+0000","last_became_peered":"2024-03-06T07:21:02.467466+0000","last_unstale":"2024-03-06T07:21:04.415077+0000","last_undegraded":"2024-03-06T07:21:04.415077+0000","last_fullsized":"2024-03-06T07:21:04.415077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:53.585742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457795+0000","last_change":"2024-03-06T07:21:01.442218+0000","last_active":"2024-03-06T07:21:03.457795+0000","last_peered":"2024-03-06T07:21:03.457795+0000","last_clean":"2024-03-06T07:21:03.457795+0000","last_became_active":"2024-03-06T07:21:01.441776+0000","last_became_peered":"2024-03-06T07:21:01.441776+0000","last_unstale":"2024-03-06T07:21:03.457795+0000","last_undegraded":"2024-03-06T07:21:03.457795+0000","last_fullsized":"2024-03-06T07:21:03.457795+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:45.796231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416182+0000","last_change":"2024-03-06T07:20:51.892549+0000","last_active":"2024-03-06T07:21:04.416182+0000","last_peered":"2024-03-06T07:21:04.416182+0000","last_clean":"2024-03-06T07:21:04.416182+0000","last_became_active":"2024-03-06T07:20:51.889936+0000","last_became_peered":"2024-03-06T07:20:51.889936+0000","last_unstale":"2024-03-06T07:21:04.416182+0000","last_undegraded":"2024-03-06T07:21:04.416182+0000","last_fullsized":"2024-03-06T07:21:04.416182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:23.986390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914865+0000","last_change":"2024-03-06T07:20:50.883071+0000","last_active":"2024-03-06T07:21:03.914865+0000","last_peered":"2024-03-06T07:21:03.914865+0000","last_clean":"2024-03-06T07:21:03.914865+0000","last_became_active":"2024-03-06T07:20:50.882465+0000","last_became_peered":"2024-03-06T07:20:50.882465+0000","last_unstale":"2024-03-06T07:21:03.914865+0000","last_undegraded":"2024-03-06T07:21:03.914865+0000","last_fullsized":"2024-03-06T07:21:03.914865+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:34.559121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461455+0000","last_change":"2024-03-06T07:21:02.471704+0000","last_active":"2024-03-06T07:21:03.461455+0000","last_peered":"2024-03-06T07:21:03.461455+0000","last_clean":"2024-03-06T07:21:03.461455+0000","last_became_active":"2024-03-06T07:21:02.471331+0000","last_became_peered":"2024-03-06T07:21:02.471331+0000","last_unstale":"2024-03-06T07:21:03.461455+0000","last_undegraded":"2024-03-06T07:21:03.461455+0000","last_fullsized":"2024-03-06T07:21:03.461455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:54.032813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461491+0000","last_change":"2024-03-06T07:21:01.441755+0000","last_active":"2024-03-06T07:21:03.461491+0000","last_peered":"2024-03-06T07:21:03.461491+0000","last_clean":"2024-03-06T07:21:03.461491+0000","last_became_active":"2024-03-06T07:21:01.441485+0000","last_became_peered":"2024-03-06T07:21:01.441485+0000","last_unstale":"2024-03-06T07:21:03.461491+0000","last_undegraded":"2024-03-06T07:21:03.461491+0000","last_fullsized":"2024-03-06T07:21:03.461491+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:03.511160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416703+0000","last_change":"2024-03-06T07:20:51.884694+0000","last_active":"2024-03-06T07:21:04.416703+0000","last_peered":"2024-03-06T07:21:04.416703+0000","last_clean":"2024-03-06T07:21:04.416703+0000","last_became_active":"2024-03-06T07:20:51.883905+0000","last_became_peered":"2024-03-06T07:20:51.883905+0000","last_unstale":"2024-03-06T07:21:04.416703+0000","last_undegraded":"2024-03-06T07:21:04.416703+0000","last_fullsized":"2024-03-06T07:21:04.416703+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:15.760105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416337+0000","last_change":"2024-03-06T07:20:50.881489+0000","last_active":"2024-03-06T07:21:04.416337+0000","last_peered":"2024-03-06T07:21:04.416337+0000","last_clean":"2024-03-06T07:21:04.416337+0000","last_became_active":"2024-03-06T07:20:50.881182+0000","last_became_peered":"2024-03-06T07:20:50.881182+0000","last_unstale":"2024-03-06T07:21:04.416337+0000","last_undegraded":"2024-03-06T07:21:04.416337+0000","last_fullsized":"2024-03-06T07:21:04.416337+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:22.518381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416472+0000","last_change":"2024-03-06T07:21:02.474020+0000","last_active":"2024-03-06T07:21:04.416472+0000","last_peered":"2024-03-06T07:21:04.416472+0000","last_clean":"2024-03-06T07:21:04.416472+0000","last_became_active":"2024-03-06T07:21:02.473363+0000","last_became_peered":"2024-03-06T07:21:02.473363+0000","last_unstale":"2024-03-06T07:21:04.416472+0000","last_undegraded":"2024-03-06T07:21:04.416472+0000","last_fullsized":"2024-03-06T07:21:04.416472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:57.665277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459813+0000","last_change":"2024-03-06T07:21:01.434477+0000","last_active":"2024-03-06T07:21:03.459813+0000","last_peered":"2024-03-06T07:21:03.459813+0000","last_clean":"2024-03-06T07:21:03.459813+0000","last_became_active":"2024-03-06T07:21:01.434205+0000","last_became_peered":"2024-03-06T07:21:01.434205+0000","last_unstale":"2024-03-06T07:21:03.459813+0000","last_undegraded":"2024-03-06T07:21:03.459813+0000","last_fullsized":"2024-03-06T07:21:03.459813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:31.133324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416730+0000","last_change":"2024-03-06T07:20:51.889352+0000","last_active":"2024-03-06T07:21:04.416730+0000","last_peered":"2024-03-06T07:21:04.416730+0000","last_clean":"2024-03-06T07:21:04.416730+0000","last_became_active":"2024-03-06T07:20:51.888464+0000","last_became_peered":"2024-03-06T07:20:51.888464+0000","last_unstale":"2024-03-06T07:21:04.416730+0000","last_undegraded":"2024-03-06T07:21:04.416730+0000","last_fullsized":"2024-03-06T07:21:04.416730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:26.611719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415860+0000","last_change":"2024-03-06T07:20:50.887415+0000","last_active":"2024-03-06T07:21:04.415860+0000","last_peered":"2024-03-06T07:21:04.415860+0000","last_clean":"2024-03-06T07:21:04.415860+0000","last_became_active":"2024-03-06T07:20:50.887137+0000","last_became_peered":"2024-03-06T07:20:50.887137+0000","last_unstale":"2024-03-06T07:21:04.415860+0000","last_undegraded":"2024-03-06T07:21:04.415860+0000","last_fullsized":"2024-03-06T07:21:04.415860+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:54.210267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459347+0000","last_change":"2024-03-06T07:21:02.459161+0000","last_active":"2024-03-06T07:21:03.459347+0000","last_peered":"2024-03-06T07:21:03.459347+0000","last_clean":"2024-03-06T07:21:03.459347+0000","last_became_active":"2024-03-06T07:21:02.458917+0000","last_became_peered":"2024-03-06T07:21:02.458917+0000","last_unstale":"2024-03-06T07:21:03.459347+0000","last_undegraded":"2024-03-06T07:21:03.459347+0000","last_fullsized":"2024-03-06T07:21:03.459347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:22.528081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415782+0000","last_change":"2024-03-06T07:21:01.439736+0000","last_active":"2024-03-06T07:21:04.415782+0000","last_peered":"2024-03-06T07:21:04.415782+0000","last_clean":"2024-03-06T07:21:04.415782+0000","last_became_active":"2024-03-06T07:21:01.439473+0000","last_became_peered":"2024-03-06T07:21:01.439473+0000","last_unstale":"2024-03-06T07:21:04.415782+0000","last_undegraded":"2024-03-06T07:21:04.415782+0000","last_fullsized":"2024-03-06T07:21:04.415782+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:52.584141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460082+0000","last_change":"2024-03-06T07:20:51.885672+0000","last_active":"2024-03-06T07:21:03.460082+0000","last_peered":"2024-03-06T07:21:03.460082+0000","last_clean":"2024-03-06T07:21:03.460082+0000","last_became_active":"2024-03-06T07:20:51.885412+0000","last_became_peered":"2024-03-06T07:20:51.885412+0000","last_unstale":"2024-03-06T07:21:03.460082+0000","last_undegraded":"2024-03-06T07:21:03.460082+0000","last_fullsized":"2024-03-06T07:21:03.460082+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:54.467344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416123+0000","last_change":"2024-03-06T07:20:50.889223+0000","last_active":"2024-03-06T07:21:04.416123+0000","last_peered":"2024-03-06T07:21:04.416123+0000","last_clean":"2024-03-06T07:21:04.416123+0000","last_became_active":"2024-03-06T07:20:50.888934+0000","last_became_peered":"2024-03-06T07:20:50.888934+0000","last_unstale":"2024-03-06T07:21:04.416123+0000","last_undegraded":"2024-03-06T07:21:04.416123+0000","last_fullsized":"2024-03-06T07:21:04.416123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:04.765109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459191+0000","last_change":"2024-03-06T07:21:02.463313+0000","last_active":"2024-03-06T07:21:03.459191+0000","last_peered":"2024-03-06T07:21:03.459191+0000","last_clean":"2024-03-06T07:21:03.459191+0000","last_became_active":"2024-03-06T07:21:02.463053+0000","last_became_peered":"2024-03-06T07:21:02.463053+0000","last_unstale":"2024-03-06T07:21:03.459191+0000","last_undegraded":"2024-03-06T07:21:03.459191+0000","last_fullsized":"2024-03-06T07:21:03.459191+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:24.369156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458989+0000","last_change":"2024-03-06T07:21:01.435463+0000","last_active":"2024-03-06T07:21:03.458989+0000","last_peered":"2024-03-06T07:21:03.458989+0000","last_clean":"2024-03-06T07:21:03.458989+0000","last_became_active":"2024-03-06T07:21:01.435086+0000","last_became_peered":"2024-03-06T07:21:01.435086+0000","last_unstale":"2024-03-06T07:21:03.458989+0000","last_undegraded":"2024-03-06T07:21:03.458989+0000","last_fullsized":"2024-03-06T07:21:03.458989+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:42.767416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.460491+0000","last_change":"2024-03-06T07:20:51.880937+0000","last_active":"2024-03-06T07:21:03.460491+0000","last_peered":"2024-03-06T07:21:03.460491+0000","last_clean":"2024-03-06T07:21:03.460491+0000","last_became_active":"2024-03-06T07:20:51.880648+0000","last_became_peered":"2024-03-06T07:20:51.880648+0000","last_unstale":"2024-03-06T07:21:03.460491+0000","last_undegraded":"2024-03-06T07:21:03.460491+0000","last_fullsized":"2024-03-06T07:21:03.460491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:45.040592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"73'2","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415915+0000","last_change":"2024-03-06T07:20:50.886345+0000","last_active":"2024-03-06T07:21:04.415915+0000","last_peered":"2024-03-06T07:21:04.415915+0000","last_clean":"2024-03-06T07:21:04.415915+0000","last_became_active":"2024-03-06T07:20:50.886074+0000","last_became_peered":"2024-03-06T07:20:50.886074+0000","last_unstale":"2024-03-06T07:21:04.415915+0000","last_undegraded":"2024-03-06T07:21:04.415915+0000","last_fullsized":"2024-03-06T07:21:04.415915+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:39.914514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415825+0000","last_change":"2024-03-06T07:21:02.476440+0000","last_active":"2024-03-06T07:21:04.415825+0000","last_peered":"2024-03-06T07:21:04.415825+0000","last_clean":"2024-03-06T07:21:04.415825+0000","last_became_active":"2024-03-06T07:21:02.476180+0000","last_became_peered":"2024-03-06T07:21:02.476180+0000","last_unstale":"2024-03-06T07:21:04.415825+0000","last_undegraded":"2024-03-06T07:21:04.415825+0000","last_fullsized":"2024-03-06T07:21:04.415825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:09.333125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415158+0000","last_change":"2024-03-06T07:21:01.443325+0000","last_active":"2024-03-06T07:21:04.415158+0000","last_peered":"2024-03-06T07:21:04.415158+0000","last_clean":"2024-03-06T07:21:04.415158+0000","last_became_active":"2024-03-06T07:21:01.443143+0000","last_became_peered":"2024-03-06T07:21:01.443143+0000","last_unstale":"2024-03-06T07:21:04.415158+0000","last_undegraded":"2024-03-06T07:21:04.415158+0000","last_fullsized":"2024-03-06T07:21:04.415158+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:07.781785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459447+0000","last_change":"2024-03-06T07:20:51.889471+0000","last_active":"2024-03-06T07:21:03.459447+0000","last_peered":"2024-03-06T07:21:03.459447+0000","last_clean":"2024-03-06T07:21:03.459447+0000","last_became_active":"2024-03-06T07:20:51.889218+0000","last_became_peered":"2024-03-06T07:20:51.889218+0000","last_unstale":"2024-03-06T07:21:03.459447+0000","last_undegraded":"2024-03-06T07:21:03.459447+0000","last_fullsized":"2024-03-06T07:21:03.459447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:15.952312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"73'1","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416800+0000","last_change":"2024-03-06T07:20:50.881740+0000","last_active":"2024-03-06T07:21:04.416800+0000","last_peered":"2024-03-06T07:21:04.416800+0000","last_clean":"2024-03-06T07:21:04.416800+0000","last_became_active":"2024-03-06T07:20:50.881306+0000","last_became_peered":"2024-03-06T07:20:50.881306+0000","last_unstale":"2024-03-06T07:21:04.416800+0000","last_undegraded":"2024-03-06T07:21:04.416800+0000","last_fullsized":"2024-03-06T07:21:04.416800+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:35:35.733892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.457624+0000","last_change":"2024-03-06T07:21:02.462216+0000","last_active":"2024-03-06T07:21:03.457624+0000","last_peered":"2024-03-06T07:21:03.457624+0000","last_clean":"2024-03-06T07:21:03.457624+0000","last_became_active":"2024-03-06T07:21:02.461922+0000","last_became_peered":"2024-03-06T07:21:02.461922+0000","last_unstale":"2024-03-06T07:21:03.457624+0000","last_undegraded":"2024-03-06T07:21:03.457624+0000","last_fullsized":"2024-03-06T07:21:03.457624+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:46.120298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415941+0000","last_change":"2024-03-06T07:21:01.435258+0000","last_active":"2024-03-06T07:21:04.415941+0000","last_peered":"2024-03-06T07:21:04.415941+0000","last_clean":"2024-03-06T07:21:04.415941+0000","last_became_active":"2024-03-06T07:21:01.435001+0000","last_became_peered":"2024-03-06T07:21:01.435001+0000","last_unstale":"2024-03-06T07:21:04.415941+0000","last_undegraded":"2024-03-06T07:21:04.415941+0000","last_fullsized":"2024-03-06T07:21:04.415941+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:17.147464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461009+0000","last_change":"2024-03-06T07:20:51.886967+0000","last_active":"2024-03-06T07:21:03.461009+0000","last_peered":"2024-03-06T07:21:03.461009+0000","last_clean":"2024-03-06T07:21:03.461009+0000","last_became_active":"2024-03-06T07:20:51.886521+0000","last_became_peered":"2024-03-06T07:20:51.886521+0000","last_unstale":"2024-03-06T07:21:03.461009+0000","last_undegraded":"2024-03-06T07:21:03.461009+0000","last_fullsized":"2024-03-06T07:21:03.461009+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:58.399266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416024+0000","last_change":"2024-03-06T07:20:50.885378+0000","last_active":"2024-03-06T07:21:04.416024+0000","last_peered":"2024-03-06T07:21:04.416024+0000","last_clean":"2024-03-06T07:21:04.416024+0000","last_became_active":"2024-03-06T07:20:50.884859+0000","last_became_peered":"2024-03-06T07:20:50.884859+0000","last_unstale":"2024-03-06T07:21:04.416024+0000","last_undegraded":"2024-03-06T07:21:04.416024+0000","last_fullsized":"2024-03-06T07:21:04.416024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:28.686737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415764+0000","last_change":"2024-03-06T07:21:02.466894+0000","last_active":"2024-03-06T07:21:04.415764+0000","last_peered":"2024-03-06T07:21:04.415764+0000","last_clean":"2024-03-06T07:21:04.415764+0000","last_became_active":"2024-03-06T07:21:02.466561+0000","last_became_peered":"2024-03-06T07:21:02.466561+0000","last_unstale":"2024-03-06T07:21:04.415764+0000","last_undegraded":"2024-03-06T07:21:04.415764+0000","last_fullsized":"2024-03-06T07:21:04.415764+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:30.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415137+0000","last_change":"2024-03-06T07:21:01.437079+0000","last_active":"2024-03-06T07:21:04.415137+0000","last_peered":"2024-03-06T07:21:04.415137+0000","last_clean":"2024-03-06T07:21:04.415137+0000","last_became_active":"2024-03-06T07:21:01.436848+0000","last_became_peered":"2024-03-06T07:21:01.436848+0000","last_unstale":"2024-03-06T07:21:04.415137+0000","last_undegraded":"2024-03-06T07:21:04.415137+0000","last_fullsized":"2024-03-06T07:21:04.415137+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:33.540896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416438+0000","last_change":"2024-03-06T07:20:51.899213+0000","last_active":"2024-03-06T07:21:04.416438+0000","last_peered":"2024-03-06T07:21:04.416438+0000","last_clean":"2024-03-06T07:21:04.416438+0000","last_became_active":"2024-03-06T07:20:51.898882+0000","last_became_peered":"2024-03-06T07:20:51.898882+0000","last_unstale":"2024-03-06T07:21:04.416438+0000","last_undegraded":"2024-03-06T07:21:04.416438+0000","last_fullsized":"2024-03-06T07:21:04.416438+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:23.523451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415619+0000","last_change":"2024-03-06T07:20:50.888878+0000","last_active":"2024-03-06T07:21:04.415619+0000","last_peered":"2024-03-06T07:21:04.415619+0000","last_clean":"2024-03-06T07:21:04.415619+0000","last_became_active":"2024-03-06T07:20:50.888604+0000","last_became_peered":"2024-03-06T07:20:50.888604+0000","last_unstale":"2024-03-06T07:21:04.415619+0000","last_undegraded":"2024-03-06T07:21:04.415619+0000","last_fullsized":"2024-03-06T07:21:04.415619+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:33.413428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416369+0000","last_change":"2024-03-06T07:21:02.460204+0000","last_active":"2024-03-06T07:21:04.416369+0000","last_peered":"2024-03-06T07:21:04.416369+0000","last_clean":"2024-03-06T07:21:04.416369+0000","last_became_active":"2024-03-06T07:21:02.459602+0000","last_became_peered":"2024-03-06T07:21:02.459602+0000","last_unstale":"2024-03-06T07:21:04.416369+0000","last_undegraded":"2024-03-06T07:21:04.416369+0000","last_fullsized":"2024-03-06T07:21:04.416369+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:49.387377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461507+0000","last_change":"2024-03-06T07:21:01.433417+0000","last_active":"2024-03-06T07:21:03.461507+0000","last_peered":"2024-03-06T07:21:03.461507+0000","last_clean":"2024-03-06T07:21:03.461507+0000","last_became_active":"2024-03-06T07:21:01.432105+0000","last_became_peered":"2024-03-06T07:21:01.432105+0000","last_unstale":"2024-03-06T07:21:03.461507+0000","last_undegraded":"2024-03-06T07:21:03.461507+0000","last_fullsized":"2024-03-06T07:21:03.461507+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:45.177307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459833+0000","last_change":"2024-03-06T07:20:51.893070+0000","last_active":"2024-03-06T07:21:03.459833+0000","last_peered":"2024-03-06T07:21:03.459833+0000","last_clean":"2024-03-06T07:21:03.459833+0000","last_became_active":"2024-03-06T07:20:51.892785+0000","last_became_peered":"2024-03-06T07:20:51.892785+0000","last_unstale":"2024-03-06T07:21:03.459833+0000","last_undegraded":"2024-03-06T07:21:03.459833+0000","last_fullsized":"2024-03-06T07:21:03.459833+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:09.591477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459302+0000","last_change":"2024-03-06T07:20:50.892942+0000","last_active":"2024-03-06T07:21:03.459302+0000","last_peered":"2024-03-06T07:21:03.459302+0000","last_clean":"2024-03-06T07:21:03.459302+0000","last_became_active":"2024-03-06T07:20:50.892638+0000","last_became_peered":"2024-03-06T07:20:50.892638+0000","last_unstale":"2024-03-06T07:21:03.459302+0000","last_undegraded":"2024-03-06T07:21:03.459302+0000","last_fullsized":"2024-03-06T07:21:03.459302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:40.434821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415614+0000","last_change":"2024-03-06T07:21:02.468449+0000","last_active":"2024-03-06T07:21:04.415614+0000","last_peered":"2024-03-06T07:21:04.415614+0000","last_clean":"2024-03-06T07:21:04.415614+0000","last_became_active":"2024-03-06T07:21:02.468117+0000","last_became_peered":"2024-03-06T07:21:02.468117+0000","last_unstale":"2024-03-06T07:21:04.415614+0000","last_undegraded":"2024-03-06T07:21:04.415614+0000","last_fullsized":"2024-03-06T07:21:04.415614+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:43.597029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459686+0000","last_change":"2024-03-06T07:21:01.435337+0000","last_active":"2024-03-06T07:21:03.459686+0000","last_peered":"2024-03-06T07:21:03.459686+0000","last_clean":"2024-03-06T07:21:03.459686+0000","last_became_active":"2024-03-06T07:21:01.435098+0000","last_became_peered":"2024-03-06T07:21:01.435098+0000","last_unstale":"2024-03-06T07:21:03.459686+0000","last_undegraded":"2024-03-06T07:21:03.459686+0000","last_fullsized":"2024-03-06T07:21:03.459686+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:33:39.246313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459419+0000","last_change":"2024-03-06T07:20:51.895826+0000","last_active":"2024-03-06T07:21:03.459419+0000","last_peered":"2024-03-06T07:21:03.459419+0000","last_clean":"2024-03-06T07:21:03.459419+0000","last_became_active":"2024-03-06T07:20:51.895483+0000","last_became_peered":"2024-03-06T07:20:51.895483+0000","last_unstale":"2024-03-06T07:21:03.459419+0000","last_undegraded":"2024-03-06T07:21:03.459419+0000","last_fullsized":"2024-03-06T07:21:03.459419+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:04.657366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.458003+0000","last_change":"2024-03-06T07:20:50.879754+0000","last_active":"2024-03-06T07:21:03.458003+0000","last_peered":"2024-03-06T07:21:03.458003+0000","last_clean":"2024-03-06T07:21:03.458003+0000","last_became_active":"2024-03-06T07:20:50.879488+0000","last_became_peered":"2024-03-06T07:20:50.879488+0000","last_unstale":"2024-03-06T07:21:03.458003+0000","last_undegraded":"2024-03-06T07:21:03.458003+0000","last_fullsized":"2024-03-06T07:21:03.458003+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:02.603045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459971+0000","last_change":"2024-03-06T07:21:02.461439+0000","last_active":"2024-03-06T07:21:03.459971+0000","last_peered":"2024-03-06T07:21:03.459971+0000","last_clean":"2024-03-06T07:21:03.459971+0000","last_became_active":"2024-03-06T07:21:02.460141+0000","last_became_peered":"2024-03-06T07:21:02.460141+0000","last_unstale":"2024-03-06T07:21:03.459971+0000","last_undegraded":"2024-03-06T07:21:03.459971+0000","last_fullsized":"2024-03-06T07:21:03.459971+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.234772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416184+0000","last_change":"2024-03-06T07:21:01.439920+0000","last_active":"2024-03-06T07:21:04.416184+0000","last_peered":"2024-03-06T07:21:04.416184+0000","last_clean":"2024-03-06T07:21:04.416184+0000","last_became_active":"2024-03-06T07:21:01.439672+0000","last_became_peered":"2024-03-06T07:21:01.439672+0000","last_unstale":"2024-03-06T07:21:04.416184+0000","last_undegraded":"2024-03-06T07:21:04.416184+0000","last_fullsized":"2024-03-06T07:21:04.416184+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:10.768407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914995+0000","last_change":"2024-03-06T07:20:51.879085+0000","last_active":"2024-03-06T07:21:03.914995+0000","last_peered":"2024-03-06T07:21:03.914995+0000","last_clean":"2024-03-06T07:21:03.914995+0000","last_became_active":"2024-03-06T07:20:51.878776+0000","last_became_peered":"2024-03-06T07:20:51.878776+0000","last_unstale":"2024-03-06T07:21:03.914995+0000","last_undegraded":"2024-03-06T07:21:03.914995+0000","last_fullsized":"2024-03-06T07:21:03.914995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:04.322592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.459542+0000","last_change":"2024-03-06T07:20:50.887928+0000","last_active":"2024-03-06T07:21:03.459542+0000","last_peered":"2024-03-06T07:21:03.459542+0000","last_clean":"2024-03-06T07:21:03.459542+0000","last_became_active":"2024-03-06T07:20:50.887638+0000","last_became_peered":"2024-03-06T07:20:50.887638+0000","last_unstale":"2024-03-06T07:21:03.459542+0000","last_undegraded":"2024-03-06T07:21:03.459542+0000","last_fullsized":"2024-03-06T07:21:03.459542+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:52.309668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415877+0000","last_change":"2024-03-06T07:21:02.476706+0000","last_active":"2024-03-06T07:21:04.415877+0000","last_peered":"2024-03-06T07:21:04.415877+0000","last_clean":"2024-03-06T07:21:04.415877+0000","last_became_active":"2024-03-06T07:21:02.476115+0000","last_became_peered":"2024-03-06T07:21:02.476115+0000","last_unstale":"2024-03-06T07:21:04.415877+0000","last_undegraded":"2024-03-06T07:21:04.415877+0000","last_fullsized":"2024-03-06T07:21:04.415877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:12.148663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"78'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.461290+0000","last_change":"2024-03-06T07:21:01.436756+0000","last_active":"2024-03-06T07:21:03.461290+0000","last_peered":"2024-03-06T07:21:03.461290+0000","last_clean":"2024-03-06T07:21:03.461290+0000","last_became_active":"2024-03-06T07:21:01.436158+0000","last_became_peered":"2024-03-06T07:21:01.436158+0000","last_unstale":"2024-03-06T07:21:03.461290+0000","last_undegraded":"2024-03-06T07:21:03.461290+0000","last_fullsized":"2024-03-06T07:21:03.461290+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:21.432694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914997+0000","last_change":"2024-03-06T07:20:51.883253+0000","last_active":"2024-03-06T07:21:03.914997+0000","last_peered":"2024-03-06T07:21:03.914997+0000","last_clean":"2024-03-06T07:21:03.914997+0000","last_became_active":"2024-03-06T07:20:51.882993+0000","last_became_peered":"2024-03-06T07:20:51.882993+0000","last_unstale":"2024-03-06T07:21:03.914997+0000","last_undegraded":"2024-03-06T07:21:03.914997+0000","last_fullsized":"2024-03-06T07:21:03.914997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:08.565976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914551+0000","last_change":"2024-03-06T07:20:50.879149+0000","last_active":"2024-03-06T07:21:03.914551+0000","last_peered":"2024-03-06T07:21:03.914551+0000","last_clean":"2024-03-06T07:21:03.914551+0000","last_became_active":"2024-03-06T07:20:50.878867+0000","last_became_peered":"2024-03-06T07:20:50.878867+0000","last_unstale":"2024-03-06T07:21:03.914551+0000","last_undegraded":"2024-03-06T07:21:03.914551+0000","last_fullsized":"2024-03-06T07:21:03.914551+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:57.394196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416180+0000","last_change":"2024-03-06T07:21:02.469513+0000","last_active":"2024-03-06T07:21:04.416180+0000","last_peered":"2024-03-06T07:21:04.416180+0000","last_clean":"2024-03-06T07:21:04.416180+0000","last_became_active":"2024-03-06T07:21:02.469225+0000","last_became_peered":"2024-03-06T07:21:02.469225+0000","last_unstale":"2024-03-06T07:21:04.416180+0000","last_undegraded":"2024-03-06T07:21:04.416180+0000","last_fullsized":"2024-03-06T07:21:04.416180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:01.415411+0000","last_clean_scrub_stamp":"2024-03-06T07:21:01.415411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:26.069032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:03.914698+0000","last_change":"2024-03-06T07:21:01.440630+0000","last_active":"2024-03-06T07:21:03.914698+0000","last_peered":"2024-03-06T07:21:03.914698+0000","last_clean":"2024-03-06T07:21:03.914698+0000","last_became_active":"2024-03-06T07:21:01.440375+0000","last_became_peered":"2024-03-06T07:21:01.440375+0000","last_unstale":"2024-03-06T07:21:03.914698+0000","last_undegraded":"2024-03-06T07:21:03.914698+0000","last_fullsized":"2024-03-06T07:21:03.914698+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:00.413811+0000","last_clean_scrub_stamp":"2024-03-06T07:21:00.413811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:46.428417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416000+0000","last_change":"2024-03-06T07:20:51.891707+0000","last_active":"2024-03-06T07:21:04.416000+0000","last_peered":"2024-03-06T07:21:04.416000+0000","last_clean":"2024-03-06T07:21:04.416000+0000","last_became_active":"2024-03-06T07:20:51.891427+0000","last_became_peered":"2024-03-06T07:20:51.891427+0000","last_unstale":"2024-03-06T07:21:04.416000+0000","last_undegraded":"2024-03-06T07:21:04.416000+0000","last_fullsized":"2024-03-06T07:21:04.416000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:50.859305+0000","last_clean_scrub_stamp":"2024-03-06T07:20:50.859305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:29.678665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.416130+0000","last_change":"2024-03-06T07:20:50.885372+0000","last_active":"2024-03-06T07:21:04.416130+0000","last_peered":"2024-03-06T07:21:04.416130+0000","last_clean":"2024-03-06T07:21:04.416130+0000","last_became_active":"2024-03-06T07:20:50.885093+0000","last_became_peered":"2024-03-06T07:20:50.885093+0000","last_unstale":"2024-03-06T07:21:04.416130+0000","last_undegraded":"2024-03-06T07:21:04.416130+0000","last_fullsized":"2024-03-06T07:21:04.416130+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:20:49.854235+0000","last_clean_scrub_stamp":"2024-03-06T07:20:49.854235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:12.807980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":280,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:21:04.415548+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:21:04.415548+0000","last_peered":"2024-03-06T07:21:04.415548+0000","last_clean":"2024-03-06T07:21:04.415548+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:21:04.415548+0000","last_undegraded":"2024-03-06T07:21:04.415548+0000","last_fullsized":"2024-03-06T07:21:04.415548+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1032192,"data_stored":1047176,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":987136,"data_stored":460558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":958464,"data_stored":456764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":983040,"data_stored":458606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":457312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":966656,"data_stored":457472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":966656,"data_stored":456912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1044480,"data_stored":1049416,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:10.153 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:21:10.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs refuse_standby_for_another_fs true 2024-03-06T07:21:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.286+0000 7fafb611e640 1 Processor -- start 2024-03-06T07:21:10.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafb611e640 1 -- start start 2024-03-06T07:21:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafb611e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb014ef70 0x7fafb006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafb611e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fafb00c1820 0x7fafb006cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafb611e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb00c2790 con 0x7fafb00c2980 2024-03-06T07:21:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafb611e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb00c1ca0 con 0x7fafb00c1820 2024-03-06T07:21:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafb611e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb014f350 con 0x7fafb014ef70 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafaeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fafb00c1820 0x7fafb006cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafaeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fafb00c1820 0x7fafb006cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40708/0 (socket says 172.21.15.94:40708) 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafaeffd640 1 -- 172.21.15.94:0/2324088250 learned_addr learned my addr 172.21.15.94:0/2324088250 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.287+0000 7fafaf7fe640 1 --2- 172.21.15.94:0/2324088250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb014ef70 0x7fafb006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.288+0000 7fafaeffd640 1 -- 172.21.15.94:0/2324088250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb014ef70 msgr2=0x7fafb006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.288+0000 7fafaeffd640 1 --2- 172.21.15.94:0/2324088250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb014ef70 0x7fafb006c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.288+0000 7fafae7fc640 1 -- 172.21.15.94:0/2324088250 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693085038 0 0) 0x7fafb00c2790 con 0x7fafb00c2980 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.288+0000 7fafaeffd640 1 -- 172.21.15.94:0/2324088250 >> v1:172.21.15.145:6789/0 conn(0x7fafb00c2980 legacy=0x7fafb0154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.288+0000 7fafaeffd640 1 -- 172.21.15.94:0/2324088250 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb00c1540 con 0x7fafb00c1820 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.288+0000 7fafaeffd640 1 --2- 172.21.15.94:0/2324088250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb006cad0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fafa400bb00 tx=0x7fafa402dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168cbc0088e2f26d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafae7fc640 1 -- 172.21.15.94:0/2324088250 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4034070 con 0x7fafb00c1820 2024-03-06T07:21:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafae7fc640 1 -- 172.21.15.94:0/2324088250 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa402ec50 con 0x7fafb00c1820 2024-03-06T07:21:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafae7fc640 1 -- 172.21.15.94:0/2324088250 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4038400 con 0x7fafb00c1820 2024-03-06T07:21:10.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 -- 172.21.15.94:0/2324088250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 msgr2=0x7fafb006cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 --2- 172.21.15.94:0/2324088250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb006cad0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fafa400bb00 tx=0x7fafa402dea0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 -- 172.21.15.94:0/2324088250 shutdown_connections 2024-03-06T07:21:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 --2- 172.21.15.94:0/2324088250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb006cad0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 --2- 172.21.15.94:0/2324088250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb014ef70 0x7fafb006c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 -- 172.21.15.94:0/2324088250 >> 172.21.15.94:0/2324088250 conn(0x7fafb00c6710 msgr2=0x7fafb00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 -- 172.21.15.94:0/2324088250 shutdown_connections 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.289+0000 7fafb611e640 1 -- 172.21.15.94:0/2324088250 wait complete. 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.290+0000 7fafb611e640 1 Processor -- start 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.290+0000 7fafb611e640 1 -- start start 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafb611e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb01e67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafb611e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb00c2980 0x7fafb0141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafb611e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb014ef70 0x7fafb014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafb611e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb0155340 con 0x7fafb00c1820 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafb611e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb0155750 con 0x7fafb014ef70 2024-03-06T07:21:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafb611e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb01554c0 con 0x7fafb00c2980 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb01e67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb01e67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40724/0 (socket says 172.21.15.94:40724) 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 -- 172.21.15.94:0/799865550 learned_addr learned my addr 172.21.15.94:0/799865550 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaeffd640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb00c2980 0x7fafb0141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaffff640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb014ef70 0x7fafb014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 -- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb00c2980 msgr2=0x7fafb0141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb00c2980 0x7fafb0141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 -- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb014ef70 msgr2=0x7fafb014d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb014ef70 0x7fafb014d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaf7fe640 1 -- 172.21.15.94:0/799865550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb00a6460 con 0x7fafb00c1820 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.291+0000 7fafaeffd640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb00c2980 0x7fafb0141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.292+0000 7fafaf7fe640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb01e67c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fafa000e870 tx=0x7fafa000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.292+0000 7faf97fff640 1 -- 172.21.15.94:0/799865550 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0017070 con 0x7fafb00c1820 2024-03-06T07:21:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.292+0000 7faf97fff640 1 -- 172.21.15.94:0/799865550 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa000c8e0 con 0x7fafb00c1820 2024-03-06T07:21:10.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.292+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb00a6740 con 0x7fafb00c1820 2024-03-06T07:21:10.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.292+0000 7faf97fff640 1 -- 172.21.15.94:0/799865550 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa000ccd0 con 0x7fafb00c1820 2024-03-06T07:21:10.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.292+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb00a6d00 con 0x7fafb00c1820 2024-03-06T07:21:10.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.293+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb004f330 con 0x7fafb00c1820 2024-03-06T07:21:10.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.293+0000 7faf97fff640 1 -- 172.21.15.94:0/799865550 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafa001a070 con 0x7fafb00c1820 2024-03-06T07:21:10.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.294+0000 7faf97fff640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf9c06b4d0 0x7faf9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.294+0000 7faf97fff640 1 -- 172.21.15.94:0/799865550 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fafa008a780 con 0x7fafb00c1820 2024-03-06T07:21:10.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.294+0000 7fafaeffd640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf9c06b4d0 0x7faf9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.294+0000 7fafaeffd640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf9c06b4d0 0x7faf9c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fafa4039000 tx=0x7fafa4034000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.298+0000 7faf97fff640 1 -- 172.21.15.94:0/799865550 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafa0058b60 con 0x7fafb00c1820 2024-03-06T07:21:10.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.509+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"} v 0) v1 -- 0x7fafb006d900 con 0x7fafb00c1820 2024-03-06T07:21:10.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.731+0000 7faf97fff640 1 -- 172.21.15.94:0/799865550 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"}]=0 set to refuse standby for another fs v60) v1 ==== 166+0+0 (secure 0 0 0) 0x7fafa005cc00 con 0x7fafb00c1820 2024-03-06T07:21:10.733 INFO:teuthology.orchestra.run.smithi094.stderr:set to refuse standby for another fs 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.733+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf9c06b4d0 msgr2=0x7faf9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.733+0000 7fafb611e640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf9c06b4d0 0x7faf9c06d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fafa4039000 tx=0x7fafa4034000 comp rx=0 tx=0).stop 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.733+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 msgr2=0x7fafb01e67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.733+0000 7fafb611e640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb01e67c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fafa000e870 tx=0x7fafa000ed30 comp rx=0 tx=0).stop 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 shutdown_connections 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf9c06b4d0 0x7faf9c06d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb014ef70 0x7fafb014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb00c2980 0x7fafb0141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 --2- 172.21.15.94:0/799865550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fafb00c1820 0x7fafb01e67c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 >> 172.21.15.94:0/799865550 conn(0x7fafb00c6710 msgr2=0x7fafb014c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 shutdown_connections 2024-03-06T07:21:10.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.734+0000 7fafb611e640 1 -- 172.21.15.94:0/799865550 wait complete. 2024-03-06T07:21:10.746 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 5:0 2024-03-06T07:21:10.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.846+0000 7f621dd5c640 1 Processor -- start 2024-03-06T07:21:10.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.846+0000 7f621dd5c640 1 -- start start 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f621dd5c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6218150920 0x7f6218153a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f621dd5c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180c2f50 con 0x7f621806c240 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f621dd5c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180c2c70 con 0x7f62180c4740 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f621dd5c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6218073130 con 0x7f6218150920 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f6217fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6218150920 0x7f6218153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f62177fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f621806c240 0x7f6218150210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38748/0 (socket says 172.21.15.94:38748) 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f6217fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6218150920 0x7f6218153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40736/0 (socket says 172.21.15.94:40736) 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f62177fe640 1 -- 172.21.15.94:0/2019379368 learned_addr learned my addr 172.21.15.94:0/2019379368 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:10.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f6217fff640 1 -- 172.21.15.94:0/2019379368 >> v1:172.21.15.145:6789/0 conn(0x7f62180c4740 legacy=0x7f6218153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f6217fff640 1 -- 172.21.15.94:0/2019379368 >> v1:172.21.15.94:6789/0 conn(0x7f621806c240 legacy=0x7f6218150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.847+0000 7f6217fff640 1 -- 172.21.15.94:0/2019379368 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6218072ae0 con 0x7f6218150920 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f6217fff640 1 --2- 172.21.15.94:0/2019379368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f6218153a30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6210009d90 tx=0x7f621002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126ff23c1d510de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f62167fc640 1 -- 172.21.15.94:0/2019379368 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210039070 con 0x7f6218150920 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f62167fc640 1 -- 172.21.15.94:0/2019379368 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621002cc60 con 0x7f6218150920 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f62167fc640 1 -- 172.21.15.94:0/2019379368 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210034660 con 0x7f6218150920 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f621dd5c640 1 -- 172.21.15.94:0/2019379368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 msgr2=0x7f6218153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f621dd5c640 1 --2- 172.21.15.94:0/2019379368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f6218153a30 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6210009d90 tx=0x7f621002beb0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f621dd5c640 1 -- 172.21.15.94:0/2019379368 shutdown_connections 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f621dd5c640 1 --2- 172.21.15.94:0/2019379368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f6218153a30 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f621dd5c640 1 -- 172.21.15.94:0/2019379368 >> 172.21.15.94:0/2019379368 conn(0x7f62180c7670 msgr2=0x7f62180c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:10.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f621dd5c640 1 -- 172.21.15.94:0/2019379368 shutdown_connections 2024-03-06T07:21:10.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.848+0000 7f621dd5c640 1 -- 172.21.15.94:0/2019379368 wait complete. 2024-03-06T07:21:10.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.849+0000 7f621dd5c640 1 Processor -- start 2024-03-06T07:21:10.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f621dd5c640 1 -- start start 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f621dd5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621806c240 0x7f62180c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f621dd5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f621814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f621dd5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f621814b360 0x7f62181eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f621dd5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621806c620 con 0x7f6218150920 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f621dd5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218153f70 con 0x7f621814b360 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f621dd5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621806c7a0 con 0x7f621806c240 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f6216ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f621814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f6216ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f621814e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40742/0 (socket says 172.21.15.94:40742) 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f6216ffd640 1 -- 172.21.15.94:0/1031719713 learned_addr learned my addr 172.21.15.94:0/1031719713 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.850+0000 7f62177fe640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621806c240 0x7f62180c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f6217fff640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f621814b360 0x7f62181eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f6216ffd640 1 -- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621806c240 msgr2=0x7f62180c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f6216ffd640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621806c240 0x7f62180c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f6216ffd640 1 -- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f621814b360 msgr2=0x7f62181eccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f6216ffd640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f621814b360 0x7f62181eccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f6216ffd640 1 -- 172.21.15.94:0/1031719713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621814b880 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f6216ffd640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f621814e700 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f620800c400 tx=0x7f620800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f61fffff640 1 -- 172.21.15.94:0/1031719713 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62080052d0 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f61fffff640 1 -- 172.21.15.94:0/1031719713 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208005490 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621814bb60 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f61fffff640 1 -- 172.21.15.94:0/1031719713 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6208007260 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.851+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621814be40 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.852+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621804f330 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.852+0000 7f61fffff640 1 -- 172.21.15.94:0/1031719713 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6208007400 con 0x7f6218150920 2024-03-06T07:21:10.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.853+0000 7f61fffff640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f620406b4d0 0x7f620406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.853+0000 7f61fffff640 1 -- 172.21.15.94:0/1031719713 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f6208089b20 con 0x7f6218150920 2024-03-06T07:21:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.853+0000 7f62177fe640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f620406b4d0 0x7f620406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:10.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.853+0000 7f62177fe640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f620406b4d0 0x7f620406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f620c005880 tx=0x7f620c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:10.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:10.857+0000 7f61fffff640 1 -- 172.21.15.94:0/1031719713 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6208057f00 con 0x7f6218150920 2024-03-06T07:21:11.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.095+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "5:0"} v 0) v1 -- 0x7f6218146200 con 0x7f6218150920 2024-03-06T07:21:11.097 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:21:11.096+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:21:11.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.097+0000 7f61fffff640 1 -- 172.21.15.94:0/1031719713 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "5:0"}]=0 failed mds gid 35492 v61) v1 ==== 98+0+0 (secure 0 0 0) 0x7f620805bfa0 con 0x7f6218150920 2024-03-06T07:21:11.098 INFO:teuthology.orchestra.run.smithi094.stderr:failed mds gid 35492 2024-03-06T07:21:11.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f620406b4d0 msgr2=0x7f620406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f620406b4d0 0x7f620406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f620c005880 tx=0x7f620c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 msgr2=0x7f621814e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f621814e700 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f620800c400 tx=0x7f620800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 shutdown_connections 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f620406b4d0 0x7f620406d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f621814b360 0x7f62181eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6218150920 0x7f621814e700 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 --2- 172.21.15.94:0/1031719713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621806c240 0x7f62180c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.099+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 >> 172.21.15.94:0/1031719713 conn(0x7f62180c7670 msgr2=0x7f6218153320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.100+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 shutdown_connections 2024-03-06T07:21:11.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.100+0000 7f621dd5c640 1 -- 172.21.15.94:0/1031719713 wait complete. 2024-03-06T07:21:11.103 INFO:tasks.ceph.mds.b.smithi145.stderr: -482> 2024-03-06T07:20:56.848+0000 7f66a1e14640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:11.112 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:11.126 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:21:11.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.211+0000 7f81bf4fb640 1 Processor -- start 2024-03-06T07:21:11.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.211+0000 7f81bf4fb640 1 -- start start 2024-03-06T07:21:11.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.211+0000 7f81bf4fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b814f790 0x7f81b81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bf4fb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f81b8155a40 0x7f81b8155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bf4fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b8156410 con 0x7f81b80c1aa0 2024-03-06T07:21:11.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bf4fb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b80c2680 con 0x7f81b8155a40 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bf4fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80c28d0 con 0x7f81b814f790 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bca96640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f81b80c1aa0 0x7f81b80c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38760/0 (socket says 172.21.15.94:38760) 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bca96640 1 -- 172.21.15.94:0/1099805195 learned_addr learned my addr 172.21.15.94:0/1099805195 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bd297640 1 --2- 172.21.15.94:0/1099805195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b814f790 0x7f81b81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bda98640 1 --2- 172.21.15.94:0/1099805195 >> v2:172.21.15.145:3300/0 conn(0x7f81b8155a40 0x7f81b8155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81affff640 1 -- 172.21.15.94:0/1099805195 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274629852 0 0) 0x7f81b8156410 con 0x7f81b80c1aa0 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81affff640 1 -- 172.21.15.94:0/1099805195 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a4003520 con 0x7f81b80c1aa0 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bd297640 1 -- 172.21.15.94:0/1099805195 >> v2:172.21.15.145:3300/0 conn(0x7f81b8155a40 msgr2=0x7f81b8155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bd297640 1 --2- 172.21.15.94:0/1099805195 >> v2:172.21.15.145:3300/0 conn(0x7f81b8155a40 0x7f81b8155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bd297640 1 -- 172.21.15.94:0/1099805195 >> v1:172.21.15.94:6789/0 conn(0x7f81b80c1aa0 legacy=0x7f81b80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bd297640 1 -- 172.21.15.94:0/1099805195 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b80c2bb0 con 0x7f81b814f790 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.212+0000 7f81bda98640 1 --2- 172.21.15.94:0/1099805195 >> v2:172.21.15.145:3300/0 conn(0x7f81b8155a40 0x7f81b8155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:11.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.213+0000 7f81bd297640 1 --2- 172.21.15.94:0/1099805195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b814f790 0x7f81b81529d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81b40097a0 tx=0x7f81b402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78832b17f4f980cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.213+0000 7f81affff640 1 -- 172.21.15.94:0/1099805195 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4035020 con 0x7f81b814f790 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.213+0000 7f81affff640 1 -- 172.21.15.94:0/1099805195 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b4031830 con 0x7f81b814f790 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.213+0000 7f81affff640 1 -- 172.21.15.94:0/1099805195 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4031c00 con 0x7f81b814f790 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.213+0000 7f81bf4fb640 1 -- 172.21.15.94:0/1099805195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b814f790 msgr2=0x7f81b81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.213+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/1099805195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b814f790 0x7f81b81529d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81b40097a0 tx=0x7f81b402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.214+0000 7f81bf4fb640 1 -- 172.21.15.94:0/1099805195 shutdown_connections 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.214+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/1099805195 >> v2:172.21.15.145:3300/0 conn(0x7f81b8155a40 0x7f81b8155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.214+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/1099805195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b814f790 0x7f81b81529d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.214+0000 7f81bf4fb640 1 -- 172.21.15.94:0/1099805195 >> 172.21.15.94:0/1099805195 conn(0x7f81b80c6710 msgr2=0x7f81b80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.214+0000 7f81bf4fb640 1 -- 172.21.15.94:0/1099805195 shutdown_connections 2024-03-06T07:21:11.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.214+0000 7f81bf4fb640 1 -- 172.21.15.94:0/1099805195 wait complete. 2024-03-06T07:21:11.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 Processor -- start 2024-03-06T07:21:11.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 -- start start 2024-03-06T07:21:11.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b80c1aa0 0x7f81b80c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81b814f790 0x7f81b814bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81b8155a40 0x7f81b814c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80c0d60 con 0x7f81b814f790 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b8155520 con 0x7f81b8155a40 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bf4fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b80c5e30 con 0x7f81b80c1aa0 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bca96640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81b814f790 0x7f81b814bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bd297640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b80c1aa0 0x7f81b80c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bca96640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81b814f790 0x7f81b814bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40756/0 (socket says 172.21.15.94:40756) 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.215+0000 7f81bca96640 1 -- 172.21.15.94:0/351528317 learned_addr learned my addr 172.21.15.94:0/351528317 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bda98640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81b8155a40 0x7f81b814c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bd297640 1 -- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81b8155a40 msgr2=0x7f81b814c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bd297640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81b8155a40 0x7f81b814c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bd297640 1 -- 172.21.15.94:0/351528317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81b814f790 msgr2=0x7f81b814bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bd297640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81b814f790 0x7f81b814bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bd297640 1 -- 172.21.15.94:0/351528317 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b81dc170 con 0x7f81b80c1aa0 2024-03-06T07:21:11.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bca96640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81b814f790 0x7f81b814bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bd297640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b80c1aa0 0x7f81b80c4880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81b4033000 tx=0x7f81b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81adffb640 1 -- 172.21.15.94:0/351528317 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4035040 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81adffb640 1 -- 172.21.15.94:0/351528317 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b403b860 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b81ecba0 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81adffb640 1 -- 172.21.15.94:0/351528317 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b403bc50 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.216+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b81dc6f0 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.217+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b804f330 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.218+0000 7f81adffb640 1 -- 172.21.15.94:0/351528317 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81b4038020 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.218+0000 7f81adffb640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f819c06b4d0 0x7f819c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.218+0000 7f81adffb640 1 -- 172.21.15.94:0/351528317 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f81b40ac630 con 0x7f81b80c1aa0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.218+0000 7f81bca96640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f819c06b4d0 0x7f819c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.219+0000 7f81bca96640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f819c06b4d0 0x7f819c06d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f81b8151ff0 tx=0x7f81a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.223+0000 7f81adffb640 1 -- 172.21.15.94:0/351528317 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81b407af60 con 0x7f81b80c1aa0 2024-03-06T07:21:11.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.431+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b80c1f40 con 0x7f81b80c1aa0 2024-03-06T07:21:11.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.432+0000 7f81adffb640 1 -- 172.21.15.94:0/351528317 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5461 (secure 0 0 0) 0x7f81b407a860 con 0x7f81b80c1aa0 2024-03-06T07:21:11.434 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:11.434 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:21:11.096897+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:11.435 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 61 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f819c06b4d0 msgr2=0x7f819c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f819c06b4d0 0x7f819c06d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f81b8151ff0 tx=0x7f81a8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b80c1aa0 msgr2=0x7f81b80c4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b80c1aa0 0x7f81b80c4880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81b4033000 tx=0x7f81b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 shutdown_connections 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f819c06b4d0 0x7f819c06d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81b8155a40 0x7f81b814c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81b814f790 0x7f81b814bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 --2- 172.21.15.94:0/351528317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81b80c1aa0 0x7f81b80c4880 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 >> 172.21.15.94:0/351528317 conn(0x7f81b80c6710 msgr2=0x7f81b80a7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 shutdown_connections 2024-03-06T07:21:11.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.435+0000 7f81bf4fb640 1 -- 172.21.15.94:0/351528317 wait complete. 2024-03-06T07:21:11.449 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-03-06T07:21:11.450 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-06T07:21:11.555 INFO:teuthology.orchestra.run.smithi094.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-06T07:21:11.556 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-06T07:21:11.556 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:11.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.654+0000 7f110d484640 1 Processor -- start 2024-03-06T07:21:11.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.654+0000 7f110d484640 1 -- start start 2024-03-06T07:21:11.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f110d484640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f11081527b0 0x7f1108159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f110d484640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11080c88b0 0x7f11080c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f110d484640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11080c7c60 con 0x7f11080c7180 2024-03-06T07:21:11.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f110d484640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1108159640 con 0x7f11080c88b0 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f110d484640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11080c8060 con 0x7f11081527b0 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f11077fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11080c88b0 0x7f11080c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f11067fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f11080c7180 0x7f1108155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38774/0 (socket says 172.21.15.94:38774) 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f11077fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11080c88b0 0x7f11080c5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40758/0 (socket says 172.21.15.94:40758) 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f11067fc640 1 -- 172.21.15.94:0/1298376493 learned_addr learned my addr 172.21.15.94:0/1298376493 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.655+0000 7f1106ffd640 1 --2- 172.21.15.94:0/1298376493 >> v2:172.21.15.145:3300/0 conn(0x7f11081527b0 0x7f1108159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f11077fe640 1 -- 172.21.15.94:0/1298376493 >> v2:172.21.15.145:3300/0 conn(0x7f11081527b0 msgr2=0x7f1108159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f11077fe640 1 --2- 172.21.15.94:0/1298376493 >> v2:172.21.15.145:3300/0 conn(0x7f11081527b0 0x7f1108159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f11077fe640 1 -- 172.21.15.94:0/1298376493 >> v1:172.21.15.94:6789/0 conn(0x7f11080c7180 legacy=0x7f1108155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f11077fe640 1 -- 172.21.15.94:0/1298376493 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080c76a0 con 0x7f11080c88b0 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f11077fe640 1 --2- 172.21.15.94:0/1298376493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11080c88b0 0x7f11080c5330 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f11080cba80 tx=0x7f110002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44c0d96e8f2a6b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f1105ffb640 1 -- 172.21.15.94:0/1298376493 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11000315a0 con 0x7f11080c88b0 2024-03-06T07:21:11.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f1105ffb640 1 -- 172.21.15.94:0/1298376493 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1100037070 con 0x7f11080c88b0 2024-03-06T07:21:11.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.656+0000 7f1105ffb640 1 -- 172.21.15.94:0/1298376493 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100031960 con 0x7f11080c88b0 2024-03-06T07:21:11.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 -- 172.21.15.94:0/1298376493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11080c88b0 msgr2=0x7f11080c5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 --2- 172.21.15.94:0/1298376493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11080c88b0 0x7f11080c5330 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f11080cba80 tx=0x7f110002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 -- 172.21.15.94:0/1298376493 shutdown_connections 2024-03-06T07:21:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 --2- 172.21.15.94:0/1298376493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11080c88b0 0x7f11080c5330 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 --2- 172.21.15.94:0/1298376493 >> v2:172.21.15.145:3300/0 conn(0x7f11081527b0 0x7f1108159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 -- 172.21.15.94:0/1298376493 >> 172.21.15.94:0/1298376493 conn(0x7f11080ac560 msgr2=0x7f11080ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 -- 172.21.15.94:0/1298376493 shutdown_connections 2024-03-06T07:21:11.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.657+0000 7f110d484640 1 -- 172.21.15.94:0/1298376493 wait complete. 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.658+0000 7f110d484640 1 Processor -- start 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.658+0000 7f110d484640 1 -- start start 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f110d484640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11080c7180 0x7f1108157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f110d484640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11080c88b0 0x7f11080c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f110d484640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11081527b0 0x7f11080c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f110d484640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1108155600 con 0x7f11081527b0 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f110d484640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1108155320 con 0x7f11080c7180 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f110d484640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1108152b90 con 0x7f11080c88b0 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11081527b0 0x7f11080c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11081527b0 0x7f11080c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40760/0 (socket says 172.21.15.94:40760) 2024-03-06T07:21:11.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 -- 172.21.15.94:0/3646875966 learned_addr learned my addr 172.21.15.94:0/3646875966 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f1106ffd640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11080c7180 0x7f1108157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11067fc640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11080c88b0 0x7f11080c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 -- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11080c88b0 msgr2=0x7f11080c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11080c88b0 0x7f11080c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 -- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11080c7180 msgr2=0x7f1108157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11080c7180 0x7f1108157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 -- 172.21.15.94:0/3646875966 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11081564f0 con 0x7f11081527b0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11067fc640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11080c88b0 0x7f11080c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.659+0000 7f11077fe640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11081527b0 0x7f11080c3fe0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f110002be50 tx=0x7f11000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.660+0000 7f10ef7fe640 1 -- 172.21.15.94:0/3646875966 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100032630 con 0x7f11081527b0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.660+0000 7f10ef7fe640 1 -- 172.21.15.94:0/3646875966 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11000327d0 con 0x7f11081527b0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.660+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11081ebd20 con 0x7f11081527b0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.660+0000 7f10ef7fe640 1 -- 172.21.15.94:0/3646875966 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100032ba0 con 0x7f11081527b0 2024-03-06T07:21:11.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.660+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1108156a90 con 0x7f11081527b0 2024-03-06T07:21:11.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.661+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110804f330 con 0x7f11081527b0 2024-03-06T07:21:11.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.661+0000 7f10ef7fe640 1 -- 172.21.15.94:0/3646875966 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f110004e430 con 0x7f11081527b0 2024-03-06T07:21:11.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.661+0000 7f10ef7fe640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10f406b4d0 0x7f10f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:11.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.662+0000 7f10ef7fe640 1 -- 172.21.15.94:0/3646875966 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f110002f030 con 0x7f11081527b0 2024-03-06T07:21:11.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.662+0000 7f1106ffd640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10f406b4d0 0x7f10f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:11.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.662+0000 7f1106ffd640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10f406b4d0 0x7f10f406d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f10f8011340 tx=0x7f10f800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:11.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.666+0000 7f10ef7fe640 1 -- 172.21.15.94:0/3646875966 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1100040070 con 0x7f11081527b0 2024-03-06T07:21:11.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.873+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11080aa490 con 0x7f11081527b0 2024-03-06T07:21:11.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.873+0000 7f10ef7fe640 1 -- 172.21.15.94:0/3646875966 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5461 (secure 0 0 0) 0x7f110007aea0 con 0x7f11081527b0 2024-03-06T07:21:11.875 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:11.875 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":61,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":61,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:21:11.096897+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:11.875 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 61 2024-03-06T07:21:11.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10f406b4d0 msgr2=0x7f10f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10f406b4d0 0x7f10f406d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f10f8011340 tx=0x7f10f800b040 comp rx=0 tx=0).stop 2024-03-06T07:21:11.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11081527b0 msgr2=0x7f11080c3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:11.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11081527b0 0x7f11080c3fe0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f110002be50 tx=0x7f11000046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 shutdown_connections 2024-03-06T07:21:11.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10f406b4d0 0x7f10f406d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11081527b0 0x7f11080c3fe0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11080c88b0 0x7f11080c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 --2- 172.21.15.94:0/3646875966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11080c7180 0x7f1108157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:11.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 >> 172.21.15.94:0/3646875966 conn(0x7f11080ac560 msgr2=0x7f110814a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:11.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 shutdown_connections 2024-03-06T07:21:11.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:11.875+0000 7f110d484640 1 -- 172.21.15.94:0/3646875966 wait complete. 2024-03-06T07:21:11.888 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-06T07:21:14.029 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:14.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:16.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:17.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f516220f640 1 Processor -- start 2024-03-06T07:21:17.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f516220f640 1 -- start start 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f516220f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f515c14f790 0x7f515c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f516220f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515c0c28d0 con 0x7f515c0c1aa0 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f516220f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515c0c2bb0 con 0x7f515c155a40 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f516220f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515c0c2680 con 0x7f515c14f790 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f515b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f515c14f790 0x7f515c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f515b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f515c14f790 0x7f515c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40764/0 (socket says 172.21.15.94:40764) 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.000+0000 7f515affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f515c0c1aa0 0x7f515c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38786/0 (socket says 172.21.15.94:38786) 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.001+0000 7f515b7fe640 1 -- 172.21.15.94:0/3056068074 learned_addr learned my addr 172.21.15.94:0/3056068074 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.001+0000 7f515b7fe640 1 -- 172.21.15.94:0/3056068074 >> v1:172.21.15.145:6789/0 conn(0x7f515c155a40 legacy=0x7f515c155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.001+0000 7f515b7fe640 1 -- 172.21.15.94:0/3056068074 >> v1:172.21.15.94:6789/0 conn(0x7f515c0c1aa0 legacy=0x7f515c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.001+0000 7f515b7fe640 1 -- 172.21.15.94:0/3056068074 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c0c2e90 con 0x7f515c14f790 2024-03-06T07:21:17.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f515b7fe640 1 --2- 172.21.15.94:0/3056068074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c1529d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f515c0c2800 tx=0x7f514c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4544ed8ecb7de457 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f515a7fc640 1 -- 172.21.15.94:0/3056068074 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c02f840 con 0x7f515c14f790 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f515a7fc640 1 -- 172.21.15.94:0/3056068074 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c02f9e0 con 0x7f515c14f790 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f515a7fc640 1 -- 172.21.15.94:0/3056068074 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c02fdb0 con 0x7f515c14f790 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f516220f640 1 -- 172.21.15.94:0/3056068074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 msgr2=0x7f515c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f516220f640 1 --2- 172.21.15.94:0/3056068074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c1529d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f515c0c2800 tx=0x7f514c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f516220f640 1 -- 172.21.15.94:0/3056068074 shutdown_connections 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f516220f640 1 --2- 172.21.15.94:0/3056068074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c1529d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f516220f640 1 -- 172.21.15.94:0/3056068074 >> 172.21.15.94:0/3056068074 conn(0x7f515c0c6710 msgr2=0x7f515c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f516220f640 1 -- 172.21.15.94:0/3056068074 shutdown_connections 2024-03-06T07:21:17.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.002+0000 7f516220f640 1 -- 172.21.15.94:0/3056068074 wait complete. 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.003+0000 7f516220f640 1 Processor -- start 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f516220f640 1 -- start start 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f516220f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515c0c1aa0 0x7f515c0a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f516220f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f516220f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f515c155a40 0x7f515c14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f516220f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0c5e30 con 0x7f515c14f790 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f516220f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0c0d60 con 0x7f515c0c1aa0 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f516220f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c0c5fb0 con 0x7f515c155a40 2024-03-06T07:21:17.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f515affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f515affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c14c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40770/0 (socket says 172.21.15.94:40770) 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f515affd640 1 -- 172.21.15.94:0/892252393 learned_addr learned my addr 172.21.15.94:0/892252393 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.004+0000 7f515b7fe640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515c0c1aa0 0x7f515c0a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f515bfff640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f515c155a40 0x7f515c14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f515affd640 1 -- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f515c155a40 msgr2=0x7f515c14c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f515affd640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f515c155a40 0x7f515c14c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f515affd640 1 -- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515c0c1aa0 msgr2=0x7f515c0a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f515affd640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515c0c1aa0 0x7f515c0a6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f515affd640 1 -- 172.21.15.94:0/892252393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c14cc30 con 0x7f515c14f790 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f515affd640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c14c1b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f515000c450 tx=0x7f515000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f5143fff640 1 -- 172.21.15.94:0/892252393 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150018070 con 0x7f515c14f790 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f5143fff640 1 -- 172.21.15.94:0/892252393 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150007480 con 0x7f515c14f790 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c1ead60 con 0x7f515c14f790 2024-03-06T07:21:17.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f5143fff640 1 -- 172.21.15.94:0/892252393 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500047e0 con 0x7f515c14f790 2024-03-06T07:21:17.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.005+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c1eb320 con 0x7f515c14f790 2024-03-06T07:21:17.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.006+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c04f330 con 0x7f515c14f790 2024-03-06T07:21:17.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.007+0000 7f5143fff640 1 -- 172.21.15.94:0/892252393 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5150004980 con 0x7f515c14f790 2024-03-06T07:21:17.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.007+0000 7f5143fff640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f514806b500 0x7f514806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.007+0000 7f5143fff640 1 -- 172.21.15.94:0/892252393 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f51500897b0 con 0x7f515c14f790 2024-03-06T07:21:17.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.007+0000 7f515b7fe640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f514806b500 0x7f514806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.008+0000 7f515b7fe640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f514806b500 0x7f514806d9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f514c036000 tx=0x7f514c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.011+0000 7f5143fff640 1 -- 172.21.15.94:0/892252393 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5150004c70 con 0x7f515c14f790 2024-03-06T07:21:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.222+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f515c0c1f40 con 0x7f515c14f790 2024-03-06T07:21:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.222+0000 7f5143fff640 1 -- 172.21.15.94:0/892252393 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6271 (secure 0 0 0) 0x7f5150057ab0 con 0x7f515c14f790 2024-03-06T07:21:17.224 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:17.224 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:21:15.113184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6424":{"gid":6424,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/2023918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":2023918602},{"type":"v1","addr":"172.21.15.145:6837","nonce":2023918602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6424,"qdb_cluster":[6424]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:17.225 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 66 2024-03-06T07:21:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f514806b500 msgr2=0x7f514806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f514806b500 0x7f514806d9c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f514c036000 tx=0x7f514c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 msgr2=0x7f515c14c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c14c1b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f515000c450 tx=0x7f515000c910 comp rx=0 tx=0).stop 2024-03-06T07:21:17.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 shutdown_connections 2024-03-06T07:21:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f514806b500 0x7f514806d9c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f515c155a40 0x7f515c14c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f515c14f790 0x7f515c14c1b0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 --2- 172.21.15.94:0/892252393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515c0c1aa0 0x7f515c0a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 >> 172.21.15.94:0/892252393 conn(0x7f515c0c6710 msgr2=0x7f515c0c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 shutdown_connections 2024-03-06T07:21:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.224+0000 7f516220f640 1 -- 172.21.15.94:0/892252393 wait complete. 2024-03-06T07:21:17.237 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:21:17.237 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:21:17.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:17.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:17.312 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:21:17.312 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:21:17.313 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:17.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:21:17.333 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:21:17.333 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:17.368 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:21:17.369 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:17.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.473+0000 7fde63fae640 1 Processor -- start 2024-03-06T07:21:17.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.473+0000 7fde63fae640 1 -- start start 2024-03-06T07:21:17.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.473+0000 7fde63fae640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde5c14f790 0x7fde5c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.473+0000 7fde63fae640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fde5c0c1aa0 0x7fde5c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.473+0000 7fde63fae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde5c0c28d0 con 0x7fde5c0c5c60 2024-03-06T07:21:17.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.473+0000 7fde63fae640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5c0c2680 con 0x7fde5c14f790 2024-03-06T07:21:17.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.473+0000 7fde63fae640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5c0c2bb0 con 0x7fde5c0c1aa0 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61d4a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde5c14f790 0x7fde5c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61d4a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde5c14f790 0x7fde5c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40784/0 (socket says 172.21.15.94:40784) 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde6254b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fde5c0c5c60 0x7fde5c155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38788/0 (socket says 172.21.15.94:38788) 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61d4a640 1 -- 172.21.15.94:0/169473656 learned_addr learned my addr 172.21.15.94:0/169473656 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61549640 1 --2- 172.21.15.94:0/169473656 >> v2:172.21.15.145:3300/0 conn(0x7fde5c0c1aa0 0x7fde5c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61d4a640 1 -- 172.21.15.94:0/169473656 >> v2:172.21.15.145:3300/0 conn(0x7fde5c0c1aa0 msgr2=0x7fde5c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61d4a640 1 --2- 172.21.15.94:0/169473656 >> v2:172.21.15.145:3300/0 conn(0x7fde5c0c1aa0 0x7fde5c0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61d4a640 1 -- 172.21.15.94:0/169473656 >> v1:172.21.15.94:6789/0 conn(0x7fde5c0c5c60 legacy=0x7fde5c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde61d4a640 1 -- 172.21.15.94:0/169473656 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c0c2e90 con 0x7fde5c14f790 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.474+0000 7fde60d48640 1 -- 172.21.15.94:0/169473656 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027182239 0 0) 0x7fde5c0c28d0 con 0x7fde5c0c5c60 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde61d4a640 1 --2- 172.21.15.94:0/169473656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c1529d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fde5c070280 tx=0x7fde5402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d79a43f8473404 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde60d48640 1 -- 172.21.15.94:0/169473656 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5402cb20 con 0x7fde5c14f790 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde60d48640 1 -- 172.21.15.94:0/169473656 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5402ccc0 con 0x7fde5c14f790 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde60d48640 1 -- 172.21.15.94:0/169473656 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54034610 con 0x7fde5c14f790 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 -- 172.21.15.94:0/169473656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 msgr2=0x7fde5c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 --2- 172.21.15.94:0/169473656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c1529d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fde5c070280 tx=0x7fde5402bf80 comp rx=0 tx=0).stop 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 -- 172.21.15.94:0/169473656 shutdown_connections 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 --2- 172.21.15.94:0/169473656 >> v2:172.21.15.145:3300/0 conn(0x7fde5c0c1aa0 0x7fde5c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 --2- 172.21.15.94:0/169473656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c1529d0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 -- 172.21.15.94:0/169473656 >> 172.21.15.94:0/169473656 conn(0x7fde5c0c6710 msgr2=0x7fde5c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 -- 172.21.15.94:0/169473656 shutdown_connections 2024-03-06T07:21:17.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.475+0000 7fde63fae640 1 -- 172.21.15.94:0/169473656 wait complete. 2024-03-06T07:21:17.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.476+0000 7fde63fae640 1 Processor -- start 2024-03-06T07:21:17.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.476+0000 7fde63fae640 1 -- start start 2024-03-06T07:21:17.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde63fae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde5c0c1aa0 0x7fde5c1eb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde63fae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde5c0c5c60 0x7fde5c070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde63fae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c14de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde63fae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c06cc00 con 0x7fde5c14f790 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde63fae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c0c0d60 con 0x7fde5c0c1aa0 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde63fae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c06cd80 con 0x7fde5c0c5c60 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde6254b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c14de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde6254b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c14de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40794/0 (socket says 172.21.15.94:40794) 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde6254b640 1 -- 172.21.15.94:0/3628914304 learned_addr learned my addr 172.21.15.94:0/3628914304 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde61549640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde5c0c5c60 0x7fde5c070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.477+0000 7fde61d4a640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde5c0c1aa0 0x7fde5c1eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde6254b640 1 -- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde5c0c5c60 msgr2=0x7fde5c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde6254b640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde5c0c5c60 0x7fde5c070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde6254b640 1 -- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde5c0c1aa0 msgr2=0x7fde5c1eb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde6254b640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde5c0c1aa0 0x7fde5c1eb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde6254b640 1 -- 172.21.15.94:0/3628914304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c1eae70 con 0x7fde5c14f790 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde6254b640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c14de40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fde58001ea0 tx=0x7fde5800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde527fc640 1 -- 172.21.15.94:0/3628914304 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde580074e0 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde527fc640 1 -- 172.21.15.94:0/3628914304 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58016070 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c072d00 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde527fc640 1 -- 172.21.15.94:0/3628914304 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58004a20 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.478+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c1eb110 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.479+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5c04f330 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.479+0000 7fde527fc640 1 -- 172.21.15.94:0/3628914304 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde58004bc0 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.480+0000 7fde527fc640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde4006b4d0 0x7fde4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.480+0000 7fde527fc640 1 -- 172.21.15.94:0/3628914304 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fde5808c1e0 con 0x7fde5c14f790 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.480+0000 7fde61d4a640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde4006b4d0 0x7fde4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.480+0000 7fde61d4a640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde4006b4d0 0x7fde4006d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fde54035000 tx=0x7fde54030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.484+0000 7fde527fc640 1 -- 172.21.15.94:0/3628914304 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde5805a4d0 con 0x7fde5c14f790 2024-03-06T07:21:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.690+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde5c068450 con 0x7fde5c14f790 2024-03-06T07:21:17.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.691+0000 7fde527fc640 1 -- 172.21.15.94:0/3628914304 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6271 (secure 0 0 0) 0x7fde5805e570 con 0x7fde5c14f790 2024-03-06T07:21:17.694 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:17.694 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:21:15.113184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6424":{"gid":6424,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/2023918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":2023918602},{"type":"v1","addr":"172.21.15.145:6837","nonce":2023918602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6424,"qdb_cluster":[6424]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 66 2024-03-06T07:21:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde4006b4d0 msgr2=0x7fde4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde4006b4d0 0x7fde4006d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fde54035000 tx=0x7fde54030000 comp rx=0 tx=0).stop 2024-03-06T07:21:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 msgr2=0x7fde5c14de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c14de40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fde58001ea0 tx=0x7fde5800c8e0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 shutdown_connections 2024-03-06T07:21:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fde4006b4d0 0x7fde4006d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde5c14f790 0x7fde5c14de40 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde5c0c5c60 0x7fde5c070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 --2- 172.21.15.94:0/3628914304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde5c0c1aa0 0x7fde5c1eb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 >> 172.21.15.94:0/3628914304 conn(0x7fde5c0c6710 msgr2=0x7fde5c06d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 shutdown_connections 2024-03-06T07:21:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.693+0000 7fde63fae640 1 -- 172.21.15.94:0/3628914304 wait complete. 2024-03-06T07:21:17.706 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:17.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.805+0000 7f3ba1884640 1 Processor -- start 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.805+0000 7f3ba1884640 1 -- start start 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3ba1884640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b9c0c88c0 0x7f3b9c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3ba1884640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3ba1884640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c0c73d0 con 0x7f3b9c14f360 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3ba1884640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c0c8dd0 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3ba1884640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0be2e0 con 0x7f3b9c158b60 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3b9a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b9c0c88c0 0x7f3b9c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3b9a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b9c0c88c0 0x7f3b9c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40808/0 (socket says 172.21.15.94:40808) 2024-03-06T07:21:17.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3b9a7fc640 1 -- 172.21.15.94:0/1189057111 learned_addr learned my addr 172.21.15.94:0/1189057111 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:17.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3b9affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3b9c14f360 0x7f3b9c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38792/0 (socket says 172.21.15.94:38792) 2024-03-06T07:21:17.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.806+0000 7f3b9b7fe640 1 --2- 172.21.15.94:0/1189057111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b9a7fc640 1 -- 172.21.15.94:0/1189057111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 msgr2=0x7f3b9c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:17.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b9a7fc640 1 --2- 172.21.15.94:0/1189057111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b9a7fc640 1 -- 172.21.15.94:0/1189057111 >> v1:172.21.15.94:6789/0 conn(0x7f3b9c14f360 legacy=0x7f3b9c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b9a7fc640 1 -- 172.21.15.94:0/1189057111 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c0c7670 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b99ffb640 1 -- 172.21.15.94:0/1189057111 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829232532 0 0) 0x7f3b9c0c73d0 con 0x7f3b9c14f360 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b9a7fc640 1 --2- 172.21.15.94:0/1189057111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c155c10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009870 tx=0x7f3b9002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9dd7596cd33b29d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b99ffb640 1 -- 172.21.15.94:0/1189057111 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90035020 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b99ffb640 1 -- 172.21.15.94:0/1189057111 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b90031640 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.807+0000 7f3b99ffb640 1 -- 172.21.15.94:0/1189057111 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9002d040 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 -- 172.21.15.94:0/1189057111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 msgr2=0x7f3b9c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 --2- 172.21.15.94:0/1189057111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c155c10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009870 tx=0x7f3b9002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 -- 172.21.15.94:0/1189057111 shutdown_connections 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 --2- 172.21.15.94:0/1189057111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 --2- 172.21.15.94:0/1189057111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c155c10 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 -- 172.21.15.94:0/1189057111 >> 172.21.15.94:0/1189057111 conn(0x7f3b9c0ac570 msgr2=0x7f3b9c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 -- 172.21.15.94:0/1189057111 shutdown_connections 2024-03-06T07:21:17.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.808+0000 7f3ba1884640 1 -- 172.21.15.94:0/1189057111 wait complete. 2024-03-06T07:21:17.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.809+0000 7f3ba1884640 1 Processor -- start 2024-03-06T07:21:17.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.809+0000 7f3ba1884640 1 -- start start 2024-03-06T07:21:17.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3ba1884640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3ba1884640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b9c14f360 0x7f3b9c151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3ba1884640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c14b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3ba1884640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0c5a50 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3ba1884640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0c7c30 con 0x7f3b9c14f360 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3ba1884640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0c5bd0 con 0x7f3b9c158b60 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40810/0 (socket says 172.21.15.94:40810) 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 -- 172.21.15.94:0/378414834 learned_addr learned my addr 172.21.15.94:0/378414834 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9b7fe640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c14b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9a7fc640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b9c14f360 0x7f3b9c151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 -- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 msgr2=0x7f3b9c14b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c14b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 -- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b9c14f360 msgr2=0x7f3b9c151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b9c14f360 0x7f3b9c151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.810+0000 7f3b9affd640 1 -- 172.21.15.94:0/378414834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c1eef40 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.811+0000 7f3b9affd640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c157c80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c7c0 tx=0x7f3b8c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.811+0000 7f3b83fff640 1 -- 172.21.15.94:0/378414834 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c018070 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.811+0000 7f3b83fff640 1 -- 172.21.15.94:0/378414834 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c00a640 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.811+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c1ef1e0 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.811+0000 7f3b83fff640 1 -- 172.21.15.94:0/378414834 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00aa30 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.811+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c150bf0 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.812+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c069460 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.812+0000 7f3b83fff640 1 -- 172.21.15.94:0/378414834 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b8c01b070 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.813+0000 7f3b83fff640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b8806b500 0x7f3b8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:17.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.813+0000 7f3b83fff640 1 -- 172.21.15.94:0/378414834 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f3b8c089c20 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.813+0000 7f3b9a7fc640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b8806b500 0x7f3b8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:17.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.813+0000 7f3b9a7fc640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b8806b500 0x7f3b8806d9c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3b90036000 tx=0x7f3b9003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:17.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.817+0000 7f3b83fff640 1 -- 172.21.15.94:0/378414834 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b9c069460 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.988+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b9c04f330 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.990+0000 7f3b83fff640 1 -- 172.21.15.94:0/378414834 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20800 (secure 0 0 0) 0x7f3b8c057fa0 con 0x7f3b9c0c88c0 2024-03-06T07:21:17.992 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:17.992 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":79,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:11.096698+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/3537750458":"2024-03-07T07:21:11.096670+0000","172.21.15.145:6836/3537750458":"2024-03-07T07:21:11.096670+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b8806b500 msgr2=0x7f3b8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b8806b500 0x7f3b8806d9c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3b90036000 tx=0x7f3b9003c040 comp rx=0 tx=0).stop 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 msgr2=0x7f3b9c157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c157c80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00c7c0 tx=0x7f3b8c00cc80 comp rx=0 tx=0).stop 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 shutdown_connections 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b8806b500 0x7f3b8806d9c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b9c158b60 0x7f3b9c14b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b9c14f360 0x7f3b9c151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 --2- 172.21.15.94:0/378414834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b9c0c88c0 0x7f3b9c157c80 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 >> 172.21.15.94:0/378414834 conn(0x7f3b9c0ac570 msgr2=0x7f3b9c0c9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 shutdown_connections 2024-03-06T07:21:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:17.992+0000 7f3ba1884640 1 -- 172.21.15.94:0/378414834 wait complete. 2024-03-06T07:21:18.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:20:49.440763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:20:50.185600+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:20:59.828663+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:21:00.744064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:18.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:18.105 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:21:18.104+0000 7f4fe12d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:18.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.106+0000 7f7134439640 1 Processor -- start 2024-03-06T07:21:18.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.106+0000 7f7134439640 1 -- start start 2024-03-06T07:21:18.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f7134439640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f712c0b0070 0x7f712c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f7134439640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c06ca70 0x7f712c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f7134439640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f712c0b0450 con 0x7f712c0c5380 2024-03-06T07:21:18.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f7134439640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712c06c020 con 0x7f712c0b0070 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f7134439640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712c06cf20 con 0x7f712c06ca70 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f71321d5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f712c0b0070 0x7f712c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f71321d5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f712c0b0070 0x7f712c155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40812/0 (socket says 172.21.15.94:40812) 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f71321d5640 1 -- 172.21.15.94:0/3383823821 learned_addr learned my addr 172.21.15.94:0/3383823821 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f71329d6640 1 --1- 172.21.15.94:0/3383823821 >> v1:172.21.15.145:6789/0 conn(0x7f712c0c5380 0x7f712c14e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:60992/0 (socket says 172.21.15.94:60992) 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.107+0000 7f71319d4640 1 --2- 172.21.15.94:0/3383823821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c06ca70 0x7f712c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71321d5640 1 -- 172.21.15.94:0/3383823821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c06ca70 msgr2=0x7f712c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71321d5640 1 --2- 172.21.15.94:0/3383823821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c06ca70 0x7f712c0c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71321d5640 1 -- 172.21.15.94:0/3383823821 >> v1:172.21.15.145:6789/0 conn(0x7f712c0c5380 legacy=0x7f712c14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71321d5640 1 -- 172.21.15.94:0/3383823821 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c0c77f0 con 0x7f712c0b0070 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71321d5640 1 --2- 172.21.15.94:0/3383823821 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0b0070 0x7f712c155c30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7128009e50 tx=0x7f712802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31a9cf504061670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71311d3640 1 -- 172.21.15.94:0/3383823821 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712802ca70 con 0x7f712c0b0070 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71311d3640 1 -- 172.21.15.94:0/3383823821 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712802cc10 con 0x7f712c0b0070 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.108+0000 7f71311d3640 1 -- 172.21.15.94:0/3383823821 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71280343f0 con 0x7f712c0b0070 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 -- 172.21.15.94:0/3383823821 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0b0070 msgr2=0x7f712c155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 --2- 172.21.15.94:0/3383823821 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0b0070 0x7f712c155c30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7128009e50 tx=0x7f712802bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 -- 172.21.15.94:0/3383823821 shutdown_connections 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 --2- 172.21.15.94:0/3383823821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c06ca70 0x7f712c0c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 --2- 172.21.15.94:0/3383823821 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0b0070 0x7f712c155c30 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 -- 172.21.15.94:0/3383823821 >> 172.21.15.94:0/3383823821 conn(0x7f712c0ac590 msgr2=0x7f712c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 -- 172.21.15.94:0/3383823821 shutdown_connections 2024-03-06T07:21:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.109+0000 7f7134439640 1 -- 172.21.15.94:0/3383823821 wait complete. 2024-03-06T07:21:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.110+0000 7f7134439640 1 Processor -- start 2024-03-06T07:21:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.110+0000 7f7134439640 1 -- start start 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f7134439640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f712c06ca70 0x7f712c1eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f7134439640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c0b0070 0x7f712c154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f7134439640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0c5380 0x7f712c1eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f7134439640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c06c550 con 0x7f712c0c5380 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f7134439640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c06c270 con 0x7f712c06ca70 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f7134439640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712c06da80 con 0x7f712c0b0070 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0c5380 0x7f712c1eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0c5380 0x7f712c1eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40820/0 (socket says 172.21.15.94:40820) 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71321d5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f712c06ca70 0x7f712c1eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 -- 172.21.15.94:0/4164956347 learned_addr learned my addr 172.21.15.94:0/4164956347 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71319d4640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c0b0070 0x7f712c154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 -- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c0b0070 msgr2=0x7f712c154a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c0b0070 0x7f712c154a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 -- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f712c06ca70 msgr2=0x7f712c1eed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f712c06ca70 0x7f712c1eed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 -- 172.21.15.94:0/4164956347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c0c4870 con 0x7f712c0c5380 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71321d5640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f712c06ca70 0x7f712c1eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.111+0000 7f71329d6640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0c5380 0x7f712c1eada0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f712400c400 tx=0x7f712400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.112+0000 7f7122ffd640 1 -- 172.21.15.94:0/4164956347 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71240074e0 con 0x7f712c0c5380 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.112+0000 7f7122ffd640 1 -- 172.21.15.94:0/4164956347 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7124007680 con 0x7f712c0c5380 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.112+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c0c4b50 con 0x7f712c0c5380 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.112+0000 7f7122ffd640 1 -- 172.21.15.94:0/4164956347 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71240049e0 con 0x7f712c0c5380 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.112+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1ee840 con 0x7f712c0c5380 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.113+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c04f330 con 0x7f712c0c5380 2024-03-06T07:21:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.113+0000 7f7122ffd640 1 -- 172.21.15.94:0/4164956347 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f712401a020 con 0x7f712c0c5380 2024-03-06T07:21:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.113+0000 7f7122ffd640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711006b4d0 0x7f711006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.114+0000 7f7122ffd640 1 -- 172.21.15.94:0/4164956347 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f7124089d40 con 0x7f712c0c5380 2024-03-06T07:21:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.114+0000 7f71321d5640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711006b4d0 0x7f711006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.114+0000 7f71321d5640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711006b4d0 0x7f711006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7128035000 tx=0x7f7128030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.118+0000 7f7122ffd640 1 -- 172.21.15.94:0/4164956347 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f712c0b0450 con 0x7f712c0c5380 2024-03-06T07:21:18.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.288+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f712c06d860 con 0x7f712c0c5380 2024-03-06T07:21:18.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.289+0000 7f7122ffd640 1 -- 172.21.15.94:0/4164956347 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71240580c0 con 0x7f712c0c5380 2024-03-06T07:21:18.290 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711006b4d0 msgr2=0x7f711006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711006b4d0 0x7f711006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7128035000 tx=0x7f7128030000 comp rx=0 tx=0).stop 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0c5380 msgr2=0x7f712c1eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0c5380 0x7f712c1eada0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f712400c400 tx=0x7f712400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 shutdown_connections 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711006b4d0 0x7f711006d990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f712c0c5380 0x7f712c1eada0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f712c0b0070 0x7f712c154a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 --2- 172.21.15.94:0/4164956347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f712c06ca70 0x7f712c1eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.290+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 >> 172.21.15.94:0/4164956347 conn(0x7f712c0ac590 msgr2=0x7f712c0adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.291+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 shutdown_connections 2024-03-06T07:21:18.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.291+0000 7f7134439640 1 -- 172.21.15.94:0/4164956347 wait complete. 2024-03-06T07:21:18.303 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.403+0000 7fd3e482a640 1 Processor -- start 2024-03-06T07:21:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.403+0000 7fd3e482a640 1 -- start start 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.404+0000 7fd3e482a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd3e006c900 0x7fd3e00c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.404+0000 7fd3e482a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e00c3ef0 con 0x7fd3e00ab170 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.404+0000 7fd3e482a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e01556d0 con 0x7fd3e00c5990 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.404+0000 7fd3e482a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e01559b0 con 0x7fd3e006c900 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.404+0000 7fd3ded9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd3e00ab170 0x7fd3e00a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38804/0 (socket says 172.21.15.94:38804) 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.404+0000 7fd3ded9d640 1 -- 172.21.15.94:0/2236834927 learned_addr learned my addr 172.21.15.94:0/2236834927 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.404+0000 7fd3ddd9b640 1 --2- 172.21.15.94:0/2236834927 >> v2:172.21.15.145:3300/0 conn(0x7fd3e006c900 0x7fd3e00c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.405+0000 7fd3dd59a640 1 -- 172.21.15.94:0/2236834927 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617400973 0 0) 0x7fd3e00c3ef0 con 0x7fd3e00ab170 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.405+0000 7fd3dd59a640 1 -- 172.21.15.94:0/2236834927 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c8003520 con 0x7fd3e00ab170 2024-03-06T07:21:18.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.405+0000 7fd3ddd9b640 1 -- 172.21.15.94:0/2236834927 >> v1:172.21.15.145:6789/0 conn(0x7fd3e00c5990 legacy=0x7fd3e006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.405+0000 7fd3dd59a640 1 -- 172.21.15.94:0/2236834927 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702154601 0 0) 0x7fd3e01556d0 con 0x7fd3e00c5990 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.405+0000 7fd3ddd9b640 1 -- 172.21.15.94:0/2236834927 >> v1:172.21.15.94:6789/0 conn(0x7fd3e00ab170 legacy=0x7fd3e00a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.405+0000 7fd3ddd9b640 1 -- 172.21.15.94:0/2236834927 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0155c90 con 0x7fd3e006c900 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3ddd9b640 1 --2- 172.21.15.94:0/2236834927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e006c900 0x7fd3e00c0ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009dc0 tx=0x7fd3d402bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3cc0a90426572d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3dd59a640 1 -- 172.21.15.94:0/2236834927 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4039070 con 0x7fd3e006c900 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3dd59a640 1 -- 172.21.15.94:0/2236834927 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d402ccd0 con 0x7fd3e006c900 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3dd59a640 1 -- 172.21.15.94:0/2236834927 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4034640 con 0x7fd3e006c900 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3e482a640 1 -- 172.21.15.94:0/2236834927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e006c900 msgr2=0x7fd3e00c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2236834927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e006c900 0x7fd3e00c0ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009dc0 tx=0x7fd3d402bf20 comp rx=0 tx=0).stop 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3e482a640 1 -- 172.21.15.94:0/2236834927 shutdown_connections 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2236834927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e006c900 0x7fd3e00c0ab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3e482a640 1 -- 172.21.15.94:0/2236834927 >> 172.21.15.94:0/2236834927 conn(0x7fd3e00c6710 msgr2=0x7fd3e00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3e482a640 1 -- 172.21.15.94:0/2236834927 shutdown_connections 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.406+0000 7fd3e482a640 1 -- 172.21.15.94:0/2236834927 wait complete. 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 Processor -- start 2024-03-06T07:21:18.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 -- start start 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3e006c900 0x7fd3e01eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3e00ab170 0x7fd3e00bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e00c5990 0x7fd3e00c3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e0156610 con 0x7fd3e00ab170 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e0156490 con 0x7fd3e00c5990 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3e482a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e00c3af0 con 0x7fd3e006c900 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3ddd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3e00ab170 0x7fd3e00bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3ded9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e00c5990 0x7fd3e00c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3ddd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3e00ab170 0x7fd3e00bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40830/0 (socket says 172.21.15.94:40830) 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3de59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3e006c900 0x7fd3e01eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.408+0000 7fd3ddd9b640 1 -- 172.21.15.94:0/2580691201 learned_addr learned my addr 172.21.15.94:0/2580691201 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ded9d640 1 -- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3e006c900 msgr2=0x7fd3e01eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ded9d640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3e006c900 0x7fd3e01eb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ded9d640 1 -- 172.21.15.94:0/2580691201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3e00ab170 msgr2=0x7fd3e00bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ded9d640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3e00ab170 0x7fd3e00bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ded9d640 1 -- 172.21.15.94:0/2580691201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0073070 con 0x7fd3e00c5990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ddd9b640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3e00ab170 0x7fd3e00bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ded9d640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e00c5990 0x7fd3e00c3290 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000c420 tx=0x7fd3d000c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ceffd640 1 -- 172.21.15.94:0/2580691201 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0017070 con 0x7fd3e00c5990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ceffd640 1 -- 172.21.15.94:0/2580691201 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d0005ce0 con 0x7fd3e00c5990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e014eb20 con 0x7fd3e00c5990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3ceffd640 1 -- 172.21.15.94:0/2580691201 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00072d0 con 0x7fd3e00c5990 2024-03-06T07:21:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.409+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e00c02f0 con 0x7fd3e00c5990 2024-03-06T07:21:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.410+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e004f330 con 0x7fd3e00c5990 2024-03-06T07:21:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.410+0000 7fd3ceffd640 1 -- 172.21.15.94:0/2580691201 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3d001a070 con 0x7fd3e00c5990 2024-03-06T07:21:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.411+0000 7fd3ceffd640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd3c006b4d0 0x7fd3c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.411+0000 7fd3ceffd640 1 -- 172.21.15.94:0/2580691201 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fd3d0089a40 con 0x7fd3e00c5990 2024-03-06T07:21:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.411+0000 7fd3de59c640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd3c006b4d0 0x7fd3c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.411+0000 7fd3de59c640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd3c006b4d0 0x7fd3c006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8000f60 tx=0x7fd3d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.415+0000 7fd3ceffd640 1 -- 172.21.15.94:0/2580691201 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3d0057c90 con 0x7fd3e00c5990 2024-03-06T07:21:18.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.592+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3e00c0df0 con 0x7fd3e00c5990 2024-03-06T07:21:18.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.592+0000 7fd3ceffd640 1 -- 172.21.15.94:0/2580691201 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd3d005bd30 con 0x7fd3e00c5990 2024-03-06T07:21:18.594 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:21:18.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd3c006b4d0 msgr2=0x7fd3c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd3c006b4d0 0x7fd3c006d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8000f60 tx=0x7fd3d8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e00c5990 msgr2=0x7fd3e00c3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e00c5990 0x7fd3e00c3290 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000c420 tx=0x7fd3d000c8e0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 shutdown_connections 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd3c006b4d0 0x7fd3c006d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3e00c5990 0x7fd3e00c3290 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3e00ab170 0x7fd3e00bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 --2- 172.21.15.94:0/2580691201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3e006c900 0x7fd3e01eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 >> 172.21.15.94:0/2580691201 conn(0x7fd3e00c6710 msgr2=0x7fd3e00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 shutdown_connections 2024-03-06T07:21:18.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.594+0000 7fd3e482a640 1 -- 172.21.15.94:0/2580691201 wait complete. 2024-03-06T07:21:18.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.708+0000 7f497d4a3640 1 Processor -- start 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.708+0000 7f497d4a3640 1 -- start start 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f497d4a3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f49780b00d0 0x7f49780c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f497d4a3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f497806b8e0 0x7f497806bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f497d4a3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49780b04b0 con 0x7f49780c7210 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f497d4a3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4978156160 con 0x7f49780b00d0 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f497d4a3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49780c5710 con 0x7f497806b8e0 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f49767fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f497806b8e0 0x7f497806bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f4976ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f49780b00d0 0x7f49780c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f49767fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f497806b8e0 0x7f497806bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:42284/0 (socket says 172.21.15.94:42284) 2024-03-06T07:21:18.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f49767fc640 1 -- 172.21.15.94:0/973930086 learned_addr learned my addr 172.21.15.94:0/973930086 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.709+0000 7f49777fe640 1 --1- 172.21.15.94:0/973930086 >> v1:172.21.15.145:6789/0 conn(0x7f49780c7210 0x7f49780c76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:40444/0 (socket says 172.21.15.94:40444) 2024-03-06T07:21:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.710+0000 7f49767fc640 1 -- 172.21.15.94:0/973930086 >> v2:172.21.15.145:3300/0 conn(0x7f49780b00d0 msgr2=0x7f49780c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.710+0000 7f49767fc640 1 --2- 172.21.15.94:0/973930086 >> v2:172.21.15.145:3300/0 conn(0x7f49780b00d0 0x7f49780c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.710+0000 7f49767fc640 1 -- 172.21.15.94:0/973930086 >> v1:172.21.15.145:6789/0 conn(0x7f49780c7210 legacy=0x7f49780c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.710+0000 7f49767fc640 1 -- 172.21.15.94:0/973930086 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497806d210 con 0x7f497806b8e0 2024-03-06T07:21:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.710+0000 7f4976ffd640 1 --2- 172.21.15.94:0/973930086 >> v2:172.21.15.145:3300/0 conn(0x7f49780b00d0 0x7f49780c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:18.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.710+0000 7f49767fc640 1 --2- 172.21.15.94:0/973930086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f497806b8e0 0x7f497806bd20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f496c009870 tx=0x7f496c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c1b392b0e1fe5e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f4975ffb640 1 -- 172.21.15.94:0/973930086 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c034070 con 0x7f497806b8e0 2024-03-06T07:21:18.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f4975ffb640 1 -- 172.21.15.94:0/973930086 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496c030840 con 0x7f497806b8e0 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f4975ffb640 1 -- 172.21.15.94:0/973930086 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c030c10 con 0x7f497806b8e0 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 -- 172.21.15.94:0/973930086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f497806b8e0 msgr2=0x7f497806bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 --2- 172.21.15.94:0/973930086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f497806b8e0 0x7f497806bd20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f496c009870 tx=0x7f496c02b980 comp rx=0 tx=0).stop 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 -- 172.21.15.94:0/973930086 shutdown_connections 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 --2- 172.21.15.94:0/973930086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f497806b8e0 0x7f497806bd20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 --2- 172.21.15.94:0/973930086 >> v2:172.21.15.145:3300/0 conn(0x7f49780b00d0 0x7f49780c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 -- 172.21.15.94:0/973930086 >> 172.21.15.94:0/973930086 conn(0x7f49780ac810 msgr2=0x7f49780acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 -- 172.21.15.94:0/973930086 shutdown_connections 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.711+0000 7f497d4a3640 1 -- 172.21.15.94:0/973930086 wait complete. 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.712+0000 7f497d4a3640 1 Processor -- start 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.712+0000 7f497d4a3640 1 -- start start 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f497d4a3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f497806b8e0 0x7f49781eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f497d4a3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49780b00d0 0x7f49781e05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f497d4a3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49780c7210 0x7f49781542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f497d4a3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497806c260 con 0x7f49780c7210 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f497d4a3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978152c50 con 0x7f497806b8e0 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f497d4a3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49780c5ba0 con 0x7f49780b00d0 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f49777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49780c7210 0x7f49781542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f4976ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f497806b8e0 0x7f49781eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f49777fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49780c7210 0x7f49781542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51740/0 (socket says 172.21.15.94:51740) 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f49767fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49780b00d0 0x7f49781e05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.713+0000 7f49777fe640 1 -- 172.21.15.94:0/1616012971 learned_addr learned my addr 172.21.15.94:0/1616012971 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f4976ffd640 1 -- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49780b00d0 msgr2=0x7f49781e05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f4976ffd640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49780b00d0 0x7f49781e05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f4976ffd640 1 -- 172.21.15.94:0/1616012971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49780c7210 msgr2=0x7f49781542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f4976ffd640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49780c7210 0x7f49781542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f4976ffd640 1 -- 172.21.15.94:0/1616012971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49781e0af0 con 0x7f497806b8e0 2024-03-06T07:21:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f49777fe640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49780c7210 0x7f49781542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f49767fc640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49780b00d0 0x7f49781e05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f4976ffd640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f497806b8e0 0x7f49781eedb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f496800efc0 tx=0x7f496800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f495f7fe640 1 -- 172.21.15.94:0/1616012971 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968018020 con 0x7f497806b8e0 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f495f7fe640 1 -- 172.21.15.94:0/1616012971 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4968014680 con 0x7f497806b8e0 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f495f7fe640 1 -- 172.21.15.94:0/1616012971 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4968014a70 con 0x7f497806b8e0 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49781dfbd0 con 0x7f497806b8e0 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.714+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4978154800 con 0x7f497806b8e0 2024-03-06T07:21:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.715+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497804f330 con 0x7f497806b8e0 2024-03-06T07:21:18.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.715+0000 7f495f7fe640 1 -- 172.21.15.94:0/1616012971 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4968005d10 con 0x7f497806b8e0 2024-03-06T07:21:18.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.716+0000 7f495f7fe640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f496406b4d0 0x7f496406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:18.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.716+0000 7f495f7fe640 1 -- 172.21.15.94:0/1616012971 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f496808b2c0 con 0x7f497806b8e0 2024-03-06T07:21:18.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.716+0000 7f49767fc640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f496406b4d0 0x7f496406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:18.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.716+0000 7f49767fc640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f496406b4d0 0x7f496406d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f496c036000 tx=0x7f496c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.721+0000 7f495f7fe640 1 -- 172.21.15.94:0/1616012971 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4968022030 con 0x7f497806b8e0 2024-03-06T07:21:18.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.891+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f497806ce50 con 0x7f497806b8e0 2024-03-06T07:21:18.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.892+0000 7f495f7fe640 1 -- 172.21.15.94:0/1616012971 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4968010040 con 0x7f497806b8e0 2024-03-06T07:21:18.894 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f496406b4d0 msgr2=0x7f496406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f496406b4d0 0x7f496406d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f496c036000 tx=0x7f496c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f497806b8e0 msgr2=0x7f49781eedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f497806b8e0 0x7f49781eedb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f496800efc0 tx=0x7f496800c490 comp rx=0 tx=0).stop 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 shutdown_connections 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f496406b4d0 0x7f496406d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49780c7210 0x7f49781542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49780b00d0 0x7f49781e05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 --2- 172.21.15.94:0/1616012971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f497806b8e0 0x7f49781eedb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 >> 172.21.15.94:0/1616012971 conn(0x7f49780ac810 msgr2=0x7f49780cac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 shutdown_connections 2024-03-06T07:21:18.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:18.894+0000 7f497d4a3640 1 -- 172.21.15.94:0/1616012971 wait complete. 2024-03-06T07:21:18.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:21:19.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e36f7a640 1 Processor -- start 2024-03-06T07:21:19.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e36f7a640 1 -- start start 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e36f7a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7e300ab310 0x7f7e300ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e36f7a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e30154330 con 0x7f7e301538c0 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e36f7a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e30153dc0 con 0x7f7e300a97c0 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e36f7a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300c46a0 con 0x7f7e300ab310 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e2ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7e301538c0 0x7f7e301494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37650/0 (socket says 172.21.15.94:37650) 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.008+0000 7f7e2ffff640 1 -- 172.21.15.94:0/2098334768 learned_addr learned my addr 172.21.15.94:0/2098334768 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.009+0000 7f7e34d16640 1 --2- 172.21.15.94:0/2098334768 >> v2:172.21.15.145:3300/0 conn(0x7f7e300ab310 0x7f7e300ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.009+0000 7f7e2f7fe640 1 -- 172.21.15.94:0/2098334768 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948711555 0 0) 0x7f7e30154330 con 0x7f7e301538c0 2024-03-06T07:21:19.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.009+0000 7f7e2f7fe640 1 -- 172.21.15.94:0/2098334768 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e20003520 con 0x7f7e301538c0 2024-03-06T07:21:19.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.009+0000 7f7e34d16640 1 -- 172.21.15.94:0/2098334768 >> v1:172.21.15.145:6789/0 conn(0x7f7e300a97c0 legacy=0x7f7e300c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.009+0000 7f7e2f7fe640 1 -- 172.21.15.94:0/2098334768 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517079873 0 0) 0x7f7e30153dc0 con 0x7f7e300a97c0 2024-03-06T07:21:19.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.009+0000 7f7e34d16640 1 -- 172.21.15.94:0/2098334768 >> v1:172.21.15.94:6789/0 conn(0x7f7e301538c0 legacy=0x7f7e301494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.009+0000 7f7e34d16640 1 -- 172.21.15.94:0/2098334768 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e300c5d30 con 0x7f7e300ab310 2024-03-06T07:21:19.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e34d16640 1 --2- 172.21.15.94:0/2098334768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 0x7f7e300ab6f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ccc7ca276e59ada server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e2f7fe640 1 -- 172.21.15.94:0/2098334768 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24035020 con 0x7f7e300ab310 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e2f7fe640 1 -- 172.21.15.94:0/2098334768 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24031830 con 0x7f7e300ab310 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e2f7fe640 1 -- 172.21.15.94:0/2098334768 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24031c00 con 0x7f7e300ab310 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e36f7a640 1 -- 172.21.15.94:0/2098334768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 msgr2=0x7f7e300ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/2098334768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 0x7f7e300ab6f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e36f7a640 1 -- 172.21.15.94:0/2098334768 shutdown_connections 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/2098334768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 0x7f7e300ab6f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e36f7a640 1 -- 172.21.15.94:0/2098334768 >> 172.21.15.94:0/2098334768 conn(0x7f7e300c6710 msgr2=0x7f7e300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e36f7a640 1 -- 172.21.15.94:0/2098334768 shutdown_connections 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.010+0000 7f7e36f7a640 1 -- 172.21.15.94:0/2098334768 wait complete. 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.011+0000 7f7e36f7a640 1 Processor -- start 2024-03-06T07:21:19.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e36f7a640 1 -- start start 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e36f7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e300a97c0 0x7f7e3014b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e36f7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 0x7f7e300bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e36f7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e301538c0 0x7f7e3006c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e36f7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300abc30 con 0x7f7e301538c0 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e36f7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300c62f0 con 0x7f7e300ab310 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e36f7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30149bb0 con 0x7f7e300a97c0 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e35517640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e301538c0 0x7f7e3006c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e35517640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e301538c0 0x7f7e3006c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51746/0 (socket says 172.21.15.94:51746) 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e35517640 1 -- 172.21.15.94:0/3489255292 learned_addr learned my addr 172.21.15.94:0/3489255292 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.012+0000 7f7e34d16640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e300a97c0 0x7f7e3014b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e2ffff640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 0x7f7e300bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e35517640 1 -- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e300a97c0 msgr2=0x7f7e3014b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e35517640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e300a97c0 0x7f7e3014b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e35517640 1 -- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 msgr2=0x7f7e300bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e35517640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 0x7f7e300bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e35517640 1 -- 172.21.15.94:0/3489255292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30149f60 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e35517640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e301538c0 0x7f7e3006c530 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c760 tx=0x7f7e2800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e2d7fa640 1 -- 172.21.15.94:0/3489255292 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28003da0 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e2d7fa640 1 -- 172.21.15.94:0/3489255292 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e28005070 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3006e260 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e2d7fa640 1 -- 172.21.15.94:0/3489255292 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28005460 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.013+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3014a7e0 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.014+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e300acab0 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.015+0000 7f7e2d7fa640 1 -- 172.21.15.94:0/3489255292 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e28005600 con 0x7f7e301538c0 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.015+0000 7f7e2d7fa640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.015+0000 7f7e2d7fa640 1 -- 172.21.15.94:0/3489255292 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f7e2808a530 con 0x7f7e301538c0 2024-03-06T07:21:19.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.015+0000 7f7e34d16640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.016+0000 7f7e34d16640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7e24036000 tx=0x7f7e24008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.019+0000 7f7e2d7fa640 1 -- 172.21.15.94:0/3489255292 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e300acab0 con 0x7f7e301538c0 2024-03-06T07:21:19.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.190+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e3004f330 con 0x7f7e301538c0 2024-03-06T07:21:19.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.191+0000 7f7e2d7fa640 1 -- 172.21.15.94:0/3489255292 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v79) v1 ==== 105+0+11 (secure 0 0 0) 0x7f7e3006e260 con 0x7f7e301538c0 2024-03-06T07:21:19.193 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e1406b4d0 msgr2=0x7f7e1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7e24036000 tx=0x7f7e24008030 comp rx=0 tx=0).stop 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e301538c0 msgr2=0x7f7e3006c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e301538c0 0x7f7e3006c530 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c760 tx=0x7f7e2800cc20 comp rx=0 tx=0).stop 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 shutdown_connections 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e301538c0 0x7f7e3006c530 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e300ab310 0x7f7e300bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 --2- 172.21.15.94:0/3489255292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e300a97c0 0x7f7e3014b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 >> 172.21.15.94:0/3489255292 conn(0x7f7e300c6710 msgr2=0x7f7e30151390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 shutdown_connections 2024-03-06T07:21:19.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.193+0000 7f7e36f7a640 1 -- 172.21.15.94:0/3489255292 wait complete. 2024-03-06T07:21:19.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:21:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.307+0000 7fbac51e9640 1 Processor -- start 2024-03-06T07:21:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.307+0000 7fbac51e9640 1 -- start start 2024-03-06T07:21:19.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.307+0000 7fbac51e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac00ab310 0x7fbac00ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.307+0000 7fbac51e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbac01538c0 0x7fbac01494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.307+0000 7fbac51e9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac0154330 con 0x7fbac00a97c0 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.307+0000 7fbac51e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac00c46a0 con 0x7fbac01538c0 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.307+0000 7fbac51e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0153dc0 con 0x7fbac00ab310 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabe59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbac01538c0 0x7fbac01494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabe59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbac01538c0 0x7fbac01494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51762/0 (socket says 172.21.15.94:51762) 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabe59c640 1 -- 172.21.15.94:0/870121916 learned_addr learned my addr 172.21.15.94:0/870121916 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabed9d640 1 --2- 172.21.15.94:0/870121916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac00ab310 0x7fbac00ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabe59c640 1 -- 172.21.15.94:0/870121916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac00ab310 msgr2=0x7fbac00ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabe59c640 1 --2- 172.21.15.94:0/870121916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac00ab310 0x7fbac00ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabe59c640 1 -- 172.21.15.94:0/870121916 >> v1:172.21.15.145:6789/0 conn(0x7fbac00a97c0 legacy=0x7fbac00c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.308+0000 7fbabe59c640 1 -- 172.21.15.94:0/870121916 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac00c5d30 con 0x7fbac01538c0 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbabe59c640 1 --2- 172.21.15.94:0/870121916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac01538c0 0x7fbac01494a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbac00c4820 tx=0x7fbab402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae92ff0aaf02d8cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbabdd9b640 1 -- 172.21.15.94:0/870121916 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4035020 con 0x7fbac01538c0 2024-03-06T07:21:19.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbabdd9b640 1 -- 172.21.15.94:0/870121916 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4031640 con 0x7fbac01538c0 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbabdd9b640 1 -- 172.21.15.94:0/870121916 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab402d040 con 0x7fbac01538c0 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 -- 172.21.15.94:0/870121916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac01538c0 msgr2=0x7fbac01494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 --2- 172.21.15.94:0/870121916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac01538c0 0x7fbac01494a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbac00c4820 tx=0x7fbab402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 -- 172.21.15.94:0/870121916 shutdown_connections 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 --2- 172.21.15.94:0/870121916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac01538c0 0x7fbac01494a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 --2- 172.21.15.94:0/870121916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac00ab310 0x7fbac00ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 -- 172.21.15.94:0/870121916 >> 172.21.15.94:0/870121916 conn(0x7fbac00c6710 msgr2=0x7fbac00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 -- 172.21.15.94:0/870121916 shutdown_connections 2024-03-06T07:21:19.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.309+0000 7fbac51e9640 1 -- 172.21.15.94:0/870121916 wait complete. 2024-03-06T07:21:19.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 Processor -- start 2024-03-06T07:21:19.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 -- start start 2024-03-06T07:21:19.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac00a97c0 0x7fbac014daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbac00ab310 0x7fbac01dc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac01538c0 0x7fbac01e9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00abc30 con 0x7fbac00a97c0 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00c62f0 con 0x7fbac00ab310 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbac51e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00c5240 con 0x7fbac01538c0 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbabed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac00a97c0 0x7fbac014daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbabed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac00a97c0 0x7fbac014daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51770/0 (socket says 172.21.15.94:51770) 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.311+0000 7fbabed9d640 1 -- 172.21.15.94:0/2527435768 learned_addr learned my addr 172.21.15.94:0/2527435768 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabe59c640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbac00ab310 0x7fbac01dc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabf59e640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac01538c0 0x7fbac01e9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabed9d640 1 -- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac01538c0 msgr2=0x7fbac01e9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabed9d640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac01538c0 0x7fbac01e9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabed9d640 1 -- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbac00ab310 msgr2=0x7fbac01dc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabed9d640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbac00ab310 0x7fbac01dc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabed9d640 1 -- 172.21.15.94:0/2527435768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac014d810 con 0x7fbac00a97c0 2024-03-06T07:21:19.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbabed9d640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac00a97c0 0x7fbac014daf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbab000e9e0 tx=0x7fbab000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2527435768 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0017070 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2527435768 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000c9d0 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac01561a0 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2527435768 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000cdc0 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.312+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac01dcf40 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.313+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac0153690 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.314+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2527435768 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbab001a070 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.314+0000 7fbaa77fe640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbaac06b500 0x7fbaac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.314+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2527435768 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fbab008a9b0 con 0x7fbac00a97c0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.314+0000 7fbabe59c640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbaac06b500 0x7fbaac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.315+0000 7fbabe59c640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbaac06b500 0x7fbaac06d9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbab4036000 tx=0x7fbab403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.318+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2527435768 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbab0058d30 con 0x7fbac00a97c0 2024-03-06T07:21:19.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.491+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7fbac0155f90 con 0x7fbac00a97c0 2024-03-06T07:21:19.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.491+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2527435768 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v79) v1 ==== 101+0+11 (secure 0 0 0) 0x7fbab005cdd0 con 0x7fbac00a97c0 2024-03-06T07:21:19.493 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:21:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbaac06b500 msgr2=0x7fbaac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbaac06b500 0x7fbaac06d9c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbab4036000 tx=0x7fbab403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac00a97c0 msgr2=0x7fbac014daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac00a97c0 0x7fbac014daf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbab000e9e0 tx=0x7fbab000eea0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 shutdown_connections 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbaac06b500 0x7fbaac06d9c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbac01538c0 0x7fbac01e9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbac00ab310 0x7fbac01dc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 --2- 172.21.15.94:0/2527435768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbac00a97c0 0x7fbac014daf0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 >> 172.21.15.94:0/2527435768 conn(0x7fbac00c6710 msgr2=0x7fbac00c4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 shutdown_connections 2024-03-06T07:21:19.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.493+0000 7fbac51e9640 1 -- 172.21.15.94:0/2527435768 wait complete. 2024-03-06T07:21:19.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:19.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.604+0000 7f01e8a47640 1 Processor -- start 2024-03-06T07:21:19.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.604+0000 7f01e8a47640 1 -- start start 2024-03-06T07:21:19.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e8a47640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e41550c0 0x7f01e41554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e8a47640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e40aa660 con 0x7f01e40aa180 2024-03-06T07:21:19.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e8a47640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e40aa940 con 0x7f01e41559e0 2024-03-06T07:21:19.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e8a47640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e40c4610 con 0x7f01e41550c0 2024-03-06T07:21:19.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e2d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f01e40aa180 0x7f01e40c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37662/0 (socket says 172.21.15.94:37662) 2024-03-06T07:21:19.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e2d9d640 1 -- 172.21.15.94:0/160028563 learned_addr learned my addr 172.21.15.94:0/160028563 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e259c640 1 --2- 172.21.15.94:0/160028563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e41550c0 0x7f01e41554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e159a640 1 -- 172.21.15.94:0/160028563 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803879235 0 0) 0x7f01e40aa660 con 0x7f01e40aa180 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e159a640 1 -- 172.21.15.94:0/160028563 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01cc003520 con 0x7f01e40aa180 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e159a640 1 -- 172.21.15.94:0/160028563 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638976496 0 0) 0x7f01e40aa940 con 0x7f01e41559e0 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.605+0000 7f01e159a640 1 -- 172.21.15.94:0/160028563 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e40aa660 con 0x7f01e41559e0 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.606+0000 7f01e259c640 1 -- 172.21.15.94:0/160028563 >> v1:172.21.15.145:6789/0 conn(0x7f01e41559e0 legacy=0x7f01e414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.606+0000 7f01e259c640 1 -- 172.21.15.94:0/160028563 >> v1:172.21.15.94:6789/0 conn(0x7f01e40aa180 legacy=0x7f01e40c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.606+0000 7f01e259c640 1 -- 172.21.15.94:0/160028563 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e40c4860 con 0x7f01e41550c0 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.606+0000 7f01e259c640 1 --2- 172.21.15.94:0/160028563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e41550c0 0x7f01e41554a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f01d400aed0 tx=0x7f01d402cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e7bf8b64043271 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.606+0000 7f01e159a640 1 -- 172.21.15.94:0/160028563 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d402dab0 con 0x7f01e41550c0 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.606+0000 7f01e159a640 1 -- 172.21.15.94:0/160028563 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d402dc50 con 0x7f01e41550c0 2024-03-06T07:21:19.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.606+0000 7f01e159a640 1 -- 172.21.15.94:0/160028563 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4008440 con 0x7f01e41550c0 2024-03-06T07:21:19.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.607+0000 7f01e8a47640 1 -- 172.21.15.94:0/160028563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e41550c0 msgr2=0x7f01e41554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.607+0000 7f01e8a47640 1 --2- 172.21.15.94:0/160028563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e41550c0 0x7f01e41554a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f01d400aed0 tx=0x7f01d402cf10 comp rx=0 tx=0).stop 2024-03-06T07:21:19.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.607+0000 7f01e8a47640 1 -- 172.21.15.94:0/160028563 shutdown_connections 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.607+0000 7f01e8a47640 1 --2- 172.21.15.94:0/160028563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e41550c0 0x7f01e41554a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.607+0000 7f01e8a47640 1 -- 172.21.15.94:0/160028563 >> 172.21.15.94:0/160028563 conn(0x7f01e40c6710 msgr2=0x7f01e40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.607+0000 7f01e8a47640 1 -- 172.21.15.94:0/160028563 shutdown_connections 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.607+0000 7f01e8a47640 1 -- 172.21.15.94:0/160028563 wait complete. 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.608+0000 7f01e8a47640 1 Processor -- start 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.608+0000 7f01e8a47640 1 -- start start 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e8a47640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e40aa180 0x7f01e41ea760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e8a47640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01e41550c0 0x7f01e414a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e8a47640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e41559e0 0x7f01e41ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e8a47640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e414f310 con 0x7f01e41559e0 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e8a47640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e40c1520 con 0x7f01e41550c0 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e8a47640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e414f490 con 0x7f01e40aa180 2024-03-06T07:21:19.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e41559e0 0x7f01e41ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e41559e0 0x7f01e41ee570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51776/0 (socket says 172.21.15.94:51776) 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 -- 172.21.15.94:0/1053502217 learned_addr learned my addr 172.21.15.94:0/1053502217 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e259c640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e40aa180 0x7f01e41ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e1d9b640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01e41550c0 0x7f01e414a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 -- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e40aa180 msgr2=0x7f01e41ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e40aa180 0x7f01e41ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 -- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01e41550c0 msgr2=0x7f01e414a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01e41550c0 0x7f01e414a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.609+0000 7f01e2d9d640 1 -- 172.21.15.94:0/1053502217 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e40a8c40 con 0x7f01e41559e0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.610+0000 7f01e2d9d640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e41559e0 0x7f01e41ee570 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f01dc00c3c0 tx=0x7f01dc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.610+0000 7f01d2ffd640 1 -- 172.21.15.94:0/1053502217 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc017070 con 0x7f01e41559e0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.610+0000 7f01d2ffd640 1 -- 172.21.15.94:0/1053502217 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01dc005ce0 con 0x7f01e41559e0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.610+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e40a8f20 con 0x7f01e41559e0 2024-03-06T07:21:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.610+0000 7f01d2ffd640 1 -- 172.21.15.94:0/1053502217 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc0072d0 con 0x7f01e41559e0 2024-03-06T07:21:19.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.610+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e41ee050 con 0x7f01e41559e0 2024-03-06T07:21:19.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.611+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e41ee250 con 0x7f01e41559e0 2024-03-06T07:21:19.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.611+0000 7f01d2ffd640 1 -- 172.21.15.94:0/1053502217 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01dc01a070 con 0x7f01e41559e0 2024-03-06T07:21:19.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.611+0000 7f01d2ffd640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01c406b4d0 0x7f01c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.612+0000 7f01d2ffd640 1 -- 172.21.15.94:0/1053502217 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f01dc089900 con 0x7f01e41559e0 2024-03-06T07:21:19.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.612+0000 7f01e259c640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01c406b4d0 0x7f01c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.612+0000 7f01e259c640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01c406b4d0 0x7f01c406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f01d4036000 tx=0x7f01d4031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.616+0000 7f01d2ffd640 1 -- 172.21.15.94:0/1053502217 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01e41ee250 con 0x7f01e41559e0 2024-03-06T07:21:19.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.821+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01e4068450 con 0x7f01e41559e0 2024-03-06T07:21:19.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.822+0000 7f01d2ffd640 1 -- 172.21.15.94:0/1053502217 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6271 (secure 0 0 0) 0x7f01dc007e20 con 0x7f01e41559e0 2024-03-06T07:21:19.824 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:19.824 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:21:15.113184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6424":{"gid":6424,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/2023918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":2023918602},{"type":"v1","addr":"172.21.15.145:6837","nonce":2023918602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6424,"qdb_cluster":[6424]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:19.824 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 66 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.823+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01c406b4d0 msgr2=0x7f01c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.823+0000 7f01e8a47640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01c406b4d0 0x7f01c406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f01d4036000 tx=0x7f01d4031000 comp rx=0 tx=0).stop 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.823+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e41559e0 msgr2=0x7f01e41ee570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.823+0000 7f01e8a47640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e41559e0 0x7f01e41ee570 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f01dc00c3c0 tx=0x7f01dc001e70 comp rx=0 tx=0).stop 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 shutdown_connections 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01c406b4d0 0x7f01c406d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01e41559e0 0x7f01e41ee570 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01e41550c0 0x7f01e414a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 --2- 172.21.15.94:0/1053502217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01e40aa180 0x7f01e41ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 >> 172.21.15.94:0/1053502217 conn(0x7f01e40c6710 msgr2=0x7f01e40c16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 shutdown_connections 2024-03-06T07:21:19.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.824+0000 7f01e8a47640 1 -- 172.21.15.94:0/1053502217 wait complete. 2024-03-06T07:21:19.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:19.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.935+0000 7f05de7bc640 1 Processor -- start 2024-03-06T07:21:19.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.935+0000 7f05de7bc640 1 -- start start 2024-03-06T07:21:19.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.935+0000 7f05de7bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d8150920 0x7f05d80c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.935+0000 7f05de7bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05d80c2e70 0x7f05d815dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.935+0000 7f05de7bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d8067340 con 0x7f05d806c720 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.935+0000 7f05de7bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d8150d00 con 0x7f05d80c2e70 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.935+0000 7f05de7bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80c2010 con 0x7f05d8150920 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d8150920 0x7f05d80c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05dcd59640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05d80c2e70 0x7f05d815dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d8150920 0x7f05d80c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:42332/0 (socket says 172.21.15.94:42332) 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d77fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f05d806c720 0x7f05d8071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:40478/0 (socket says 172.21.15.94:40478) 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05dcd59640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05d80c2e70 0x7f05d815dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:43332/0 (socket says 172.21.15.94:43332) 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d7fff640 1 -- 172.21.15.94:0/1644705581 learned_addr learned my addr 172.21.15.94:0/1644705581 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d7fff640 1 -- 172.21.15.94:0/1644705581 >> v2:172.21.15.145:3300/0 conn(0x7f05d80c2e70 msgr2=0x7f05d815dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d7fff640 1 --2- 172.21.15.94:0/1644705581 >> v2:172.21.15.145:3300/0 conn(0x7f05d80c2e70 0x7f05d815dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d7fff640 1 -- 172.21.15.94:0/1644705581 >> v1:172.21.15.145:6789/0 conn(0x7f05d806c720 legacy=0x7f05d8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.936+0000 7f05d7fff640 1 -- 172.21.15.94:0/1644705581 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d8067060 con 0x7f05d8150920 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05d7fff640 1 --2- 172.21.15.94:0/1644705581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d8150920 0x7f05d80c2930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009870 tx=0x7f05c802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc069580f892a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05d6ffd640 1 -- 172.21.15.94:0/1644705581 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8035020 con 0x7f05d8150920 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05d6ffd640 1 -- 172.21.15.94:0/1644705581 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c8031640 con 0x7f05d8150920 2024-03-06T07:21:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05d6ffd640 1 -- 172.21.15.94:0/1644705581 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c802d040 con 0x7f05d8150920 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05de7bc640 1 -- 172.21.15.94:0/1644705581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d8150920 msgr2=0x7f05d80c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05de7bc640 1 --2- 172.21.15.94:0/1644705581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d8150920 0x7f05d80c2930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009870 tx=0x7f05c802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05de7bc640 1 -- 172.21.15.94:0/1644705581 shutdown_connections 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05de7bc640 1 --2- 172.21.15.94:0/1644705581 >> v2:172.21.15.145:3300/0 conn(0x7f05d80c2e70 0x7f05d815dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05de7bc640 1 --2- 172.21.15.94:0/1644705581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d8150920 0x7f05d80c2930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05de7bc640 1 -- 172.21.15.94:0/1644705581 >> 172.21.15.94:0/1644705581 conn(0x7f05d80c70b0 msgr2=0x7f05d80c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.937+0000 7f05de7bc640 1 -- 172.21.15.94:0/1644705581 shutdown_connections 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.938+0000 7f05de7bc640 1 -- 172.21.15.94:0/1644705581 wait complete. 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 Processor -- start 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 -- start start 2024-03-06T07:21:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d806c720 0x7f05d814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d80c2e70 0x7f05d8155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d8150920 0x7f05d81e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d8073210 con 0x7f05d806c720 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d8073090 con 0x7f05d8150920 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05de7bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80c1b40 con 0x7f05d80c2e70 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d806c720 0x7f05d814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05d77fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d80c2e70 0x7f05d8155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.939+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d806c720 0x7f05d814d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51782/0 (socket says 172.21.15.94:51782) 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d7fff640 1 -- 172.21.15.94:0/2487295161 learned_addr learned my addr 172.21.15.94:0/2487295161 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05dcd59640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d8150920 0x7f05d81e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d7fff640 1 -- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d80c2e70 msgr2=0x7f05d8155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d7fff640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d80c2e70 0x7f05d8155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d7fff640 1 -- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d8150920 msgr2=0x7f05d81e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d7fff640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d8150920 0x7f05d81e5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d7fff640 1 -- 172.21.15.94:0/2487295161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d81556c0 con 0x7f05d806c720 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d77fe640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d80c2e70 0x7f05d8155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d7fff640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d806c720 0x7f05d814d0c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f05c8000c00 tx=0x7f05c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d4ff9640 1 -- 172.21.15.94:0/2487295161 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8040070 con 0x7f05d806c720 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d4ff9640 1 -- 172.21.15.94:0/2487295161 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c803e070 con 0x7f05d806c720 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d81517d0 con 0x7f05d806c720 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05d4ff9640 1 -- 172.21.15.94:0/2487295161 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8035040 con 0x7f05d806c720 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.940+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d814c8c0 con 0x7f05d806c720 2024-03-06T07:21:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.941+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d8075300 con 0x7f05d806c720 2024-03-06T07:21:19.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.942+0000 7f05d4ff9640 1 -- 172.21.15.94:0/2487295161 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05c8038020 con 0x7f05d806c720 2024-03-06T07:21:19.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.942+0000 7f05d4ff9640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05c406b4d0 0x7f05c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:19.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.942+0000 7f05d4ff9640 1 -- 172.21.15.94:0/2487295161 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f05c80abfe0 con 0x7f05d806c720 2024-03-06T07:21:19.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.942+0000 7f05d77fe640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05c406b4d0 0x7f05c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:19.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.943+0000 7f05d77fe640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05c406b4d0 0x7f05c406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f05d8148da0 tx=0x7f05cc007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:19.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:19.946+0000 7f05d4ff9640 1 -- 172.21.15.94:0/2487295161 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05d8075300 con 0x7f05d806c720 2024-03-06T07:21:20.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.119+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05d804f330 con 0x7f05d806c720 2024-03-06T07:21:20.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.120+0000 7f05d4ff9640 1 -- 172.21.15.94:0/2487295161 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20800 (secure 0 0 0) 0x7f05c807a250 con 0x7f05d806c720 2024-03-06T07:21:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:20.122 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":79,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:11.096698+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/3537750458":"2024-03-07T07:21:11.096670+0000","172.21.15.145:6836/3537750458":"2024-03-07T07:21:11.096670+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05c406b4d0 msgr2=0x7f05c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05c406b4d0 0x7f05c406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f05d8148da0 tx=0x7f05cc007e80 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d806c720 msgr2=0x7f05d814d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d806c720 0x7f05d814d0c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f05c8000c00 tx=0x7f05c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 shutdown_connections 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05c406b4d0 0x7f05c406d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d8150920 0x7f05d81e5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d80c2e70 0x7f05d8155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 --2- 172.21.15.94:0/2487295161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d806c720 0x7f05d814d0c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 >> 172.21.15.94:0/2487295161 conn(0x7f05d80c70b0 msgr2=0x7f05d806dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 shutdown_connections 2024-03-06T07:21:20.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.122+0000 7f05de7bc640 1 -- 172.21.15.94:0/2487295161 wait complete. 2024-03-06T07:21:20.135 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:21:20.135 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:20.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.233+0000 7f48363bc640 1 Processor -- start 2024-03-06T07:21:20.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.233+0000 7f48363bc640 1 -- start start 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f48363bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f48300c3750 0x7f48300c1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f48363bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300a5e60 0x7f48300c1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f48363bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48300c5490 con 0x7f48300c3370 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f48363bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48300c5770 con 0x7f48300c3750 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f48363bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48300c5a50 con 0x7f48300a5e60 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f4834959640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300a5e60 0x7f48300c1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f4834959640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300a5e60 0x7f48300c1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:42358/0 (socket says 172.21.15.94:42358) 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f4834959640 1 -- 172.21.15.94:0/3785192833 learned_addr learned my addr 172.21.15.94:0/3785192833 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f482ffff640 1 --1- 172.21.15.94:0/3785192833 >> v1:172.21.15.145:6789/0 conn(0x7f48300c3370 0x7f4830155ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:40488/0 (socket says 172.21.15.94:40488) 2024-03-06T07:21:20.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.234+0000 7f482f7fe640 1 --2- 172.21.15.94:0/3785192833 >> v2:172.21.15.94:3300/0 conn(0x7f48300c3750 0x7f48300c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482f7fe640 1 -- 172.21.15.94:0/3785192833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300a5e60 msgr2=0x7f48300c1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482f7fe640 1 --2- 172.21.15.94:0/3785192833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300a5e60 0x7f48300c1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482f7fe640 1 -- 172.21.15.94:0/3785192833 >> v1:172.21.15.145:6789/0 conn(0x7f48300c3370 legacy=0x7f4830155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482effd640 1 -- 172.21.15.94:0/3785192833 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566807995 0 0) 0x7f48300c5490 con 0x7f48300c3370 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482f7fe640 1 -- 172.21.15.94:0/3785192833 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48300c5d30 con 0x7f48300c3750 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f4834959640 1 --2- 172.21.15.94:0/3785192833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300a5e60 0x7f48300c1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482f7fe640 1 --2- 172.21.15.94:0/3785192833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3750 0x7f48300c1110 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f48240097a0 tx=0x7f482402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b1aaedcdd4f6d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482effd640 1 -- 172.21.15.94:0/3785192833 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824038070 con 0x7f48300c3750 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482effd640 1 -- 172.21.15.94:0/3785192833 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824031700 con 0x7f48300c3750 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.235+0000 7f482effd640 1 -- 172.21.15.94:0/3785192833 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824031ad0 con 0x7f48300c3750 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 -- 172.21.15.94:0/3785192833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3750 msgr2=0x7f48300c1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 --2- 172.21.15.94:0/3785192833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3750 0x7f48300c1110 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f48240097a0 tx=0x7f482402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 -- 172.21.15.94:0/3785192833 shutdown_connections 2024-03-06T07:21:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 --2- 172.21.15.94:0/3785192833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300a5e60 0x7f48300c1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 --2- 172.21.15.94:0/3785192833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3750 0x7f48300c1110 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 -- 172.21.15.94:0/3785192833 >> 172.21.15.94:0/3785192833 conn(0x7f48300c6710 msgr2=0x7f48300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 -- 172.21.15.94:0/3785192833 shutdown_connections 2024-03-06T07:21:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.236+0000 7f48363bc640 1 -- 172.21.15.94:0/3785192833 wait complete. 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.237+0000 7f48363bc640 1 Processor -- start 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.237+0000 7f48363bc640 1 -- start start 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f48363bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48300a5e60 0x7f48301ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f48363bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3370 0x7f48301530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f48363bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300c3750 0x7f483006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f48363bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4830159910 con 0x7f48300c3370 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f48363bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4830159790 con 0x7f48300a5e60 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f48363bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48300c1b90 con 0x7f48300c3750 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3370 0x7f48301530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3370 0x7f48301530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51810/0 (socket says 172.21.15.94:51810) 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 -- 172.21.15.94:0/1750662542 learned_addr learned my addr 172.21.15.94:0/1750662542 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f4834959640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300c3750 0x7f483006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482ffff640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48300a5e60 0x7f48301ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 -- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300c3750 msgr2=0x7f483006d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300c3750 0x7f483006d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 -- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48300a5e60 msgr2=0x7f48301ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48300a5e60 0x7f48301ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 -- 172.21.15.94:0/1750662542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48301eec00 con 0x7f48300c3370 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482ffff640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48300a5e60 0x7f48301ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.238+0000 7f482f7fe640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3370 0x7f48301530b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4824036040 tx=0x7f48240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.239+0000 7f482cff9640 1 -- 172.21.15.94:0/1750662542 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824042070 con 0x7f48300c3370 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.239+0000 7f482cff9640 1 -- 172.21.15.94:0/1750662542 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482403d890 con 0x7f48300c3370 2024-03-06T07:21:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.239+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48301535f0 con 0x7f48300c3370 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.239+0000 7f482cff9640 1 -- 172.21.15.94:0/1750662542 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824038040 con 0x7f48300c3370 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.239+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4830070ad0 con 0x7f48300c3370 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.240+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48300a64a0 con 0x7f48300c3370 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.240+0000 7f482cff9640 1 -- 172.21.15.94:0/1750662542 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f482403b070 con 0x7f48300c3370 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.240+0000 7f482cff9640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f481c06b500 0x7f481c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.241+0000 7f482cff9640 1 -- 172.21.15.94:0/1750662542 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f48240abd30 con 0x7f48300c3370 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.241+0000 7f482ffff640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f481c06b500 0x7f481c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.241+0000 7f482ffff640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f481c06b500 0x7f481c06d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4820000f60 tx=0x7f4820007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.245+0000 7f482cff9640 1 -- 172.21.15.94:0/1750662542 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48300a64a0 con 0x7f48300c3370 2024-03-06T07:21:20.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.452+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4830068450 con 0x7f48300c3370 2024-03-06T07:21:20.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.453+0000 7f482cff9640 1 -- 172.21.15.94:0/1750662542 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6271 (secure 0 0 0) 0x7f4824030070 con 0x7f48300c3370 2024-03-06T07:21:20.455 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:20.455 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:21:15.113184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6424":{"gid":6424,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/2023918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":2023918602},{"type":"v1","addr":"172.21.15.145:6837","nonce":2023918602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6424,"qdb_cluster":[6424]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:20.455 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 66 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.454+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f481c06b500 msgr2=0x7f481c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.454+0000 7f48363bc640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f481c06b500 0x7f481c06d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4820000f60 tx=0x7f4820007c10 comp rx=0 tx=0).stop 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.454+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3370 msgr2=0x7f48301530b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.454+0000 7f48363bc640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3370 0x7f48301530b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4824036040 tx=0x7f48240046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 shutdown_connections 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f481c06b500 0x7f481c06d9c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48300c3750 0x7f483006d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48300c3370 0x7f48301530b0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 --2- 172.21.15.94:0/1750662542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48300a5e60 0x7f48301ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 >> 172.21.15.94:0/1750662542 conn(0x7f48300c6710 msgr2=0x7f48300bfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 shutdown_connections 2024-03-06T07:21:20.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.455+0000 7f48363bc640 1 -- 172.21.15.94:0/1750662542 wait complete. 2024-03-06T07:21:20.468 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.566+0000 7f11e334b640 1 Processor -- start 2024-03-06T07:21:20.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e334b640 1 -- start start 2024-03-06T07:21:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e334b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11dc156ea0 0x7f11dc0c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e334b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0ad310 con 0x7f11dc153990 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e334b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc1572a0 con 0x7f11dc1506b0 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e334b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11dc156450 con 0x7f11dc156ea0 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e18e8640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11dc156ea0 0x7f11dc0c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e08e6640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f11dc153990 0x7f11dc0acc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37674/0 (socket says 172.21.15.94:37674) 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.567+0000 7f11e08e6640 1 -- 172.21.15.94:0/3354246500 learned_addr learned my addr 172.21.15.94:0/3354246500 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.568+0000 7f11e18e8640 1 -- 172.21.15.94:0/3354246500 >> v1:172.21.15.145:6789/0 conn(0x7f11dc1506b0 legacy=0x7f11dc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.568+0000 7f11e18e8640 1 -- 172.21.15.94:0/3354246500 >> v1:172.21.15.94:6789/0 conn(0x7f11dc153990 legacy=0x7f11dc0acc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.568+0000 7f11e18e8640 1 -- 172.21.15.94:0/3354246500 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc070aa0 con 0x7f11dc156ea0 2024-03-06T07:21:20.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.568+0000 7f11d3fff640 1 -- 172.21.15.94:0/3354246500 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967249712 0 0) 0x7f11dc0ad310 con 0x7f11dc153990 2024-03-06T07:21:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.568+0000 7f11e18e8640 1 --2- 172.21.15.94:0/3354246500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc156ea0 0x7f11dc0c54a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097a0 tx=0x7f11d802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f425d83af09a7ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11d3fff640 1 -- 172.21.15.94:0/3354246500 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8034070 con 0x7f11dc156ea0 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11d3fff640 1 -- 172.21.15.94:0/3354246500 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8030610 con 0x7f11dc156ea0 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11d3fff640 1 -- 172.21.15.94:0/3354246500 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d80309e0 con 0x7f11dc156ea0 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11e334b640 1 -- 172.21.15.94:0/3354246500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc156ea0 msgr2=0x7f11dc0c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11e334b640 1 --2- 172.21.15.94:0/3354246500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc156ea0 0x7f11dc0c54a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097a0 tx=0x7f11d802b960 comp rx=0 tx=0).stop 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11e334b640 1 -- 172.21.15.94:0/3354246500 shutdown_connections 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11e334b640 1 --2- 172.21.15.94:0/3354246500 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc156ea0 0x7f11dc0c54a0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11e334b640 1 -- 172.21.15.94:0/3354246500 >> 172.21.15.94:0/3354246500 conn(0x7f11dc06b1a0 msgr2=0x7f11dc06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11e334b640 1 -- 172.21.15.94:0/3354246500 shutdown_connections 2024-03-06T07:21:20.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.569+0000 7f11e334b640 1 -- 172.21.15.94:0/3354246500 wait complete. 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.570+0000 7f11e334b640 1 Processor -- start 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.570+0000 7f11e334b640 1 -- start start 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e334b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11dc1506b0 0x7f11dc155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e334b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc153990 0x7f11dc148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e334b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11dc156ea0 0x7f11dc1e87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e334b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc1566a0 con 0x7f11dc153990 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e334b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc156a10 con 0x7f11dc1506b0 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e334b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc0c59e0 con 0x7f11dc156ea0 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc153990 0x7f11dc148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc153990 0x7f11dc148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51826/0 (socket says 172.21.15.94:51826) 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 -- 172.21.15.94:0/139211205 learned_addr learned my addr 172.21.15.94:0/139211205 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e18e8640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11dc156ea0 0x7f11dc1e87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e10e7640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11dc1506b0 0x7f11dc155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 -- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11dc156ea0 msgr2=0x7f11dc1e87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11dc156ea0 0x7f11dc1e87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 -- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11dc1506b0 msgr2=0x7f11dc155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11dc1506b0 0x7f11dc155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.571+0000 7f11e08e6640 1 -- 172.21.15.94:0/139211205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc150cf0 con 0x7f11dc153990 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.572+0000 7f11e08e6640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc153990 0x7f11dc148fb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00c4d0 tx=0x7f11cc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.572+0000 7f11d1ffb640 1 -- 172.21.15.94:0/139211205 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc018070 con 0x7f11dc153990 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.572+0000 7f11d1ffb640 1 -- 172.21.15.94:0/139211205 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11cc0074e0 con 0x7f11dc153990 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.572+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc150f90 con 0x7f11dc153990 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.572+0000 7f11d1ffb640 1 -- 172.21.15.94:0/139211205 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc004a20 con 0x7f11dc153990 2024-03-06T07:21:20.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.572+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc1ecb60 con 0x7f11dc153990 2024-03-06T07:21:20.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.573+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc04f330 con 0x7f11dc153990 2024-03-06T07:21:20.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.573+0000 7f11d1ffb640 1 -- 172.21.15.94:0/139211205 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11cc01b070 con 0x7f11dc153990 2024-03-06T07:21:20.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.573+0000 7f11d1ffb640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11c006b4d0 0x7f11c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.574+0000 7f11d1ffb640 1 -- 172.21.15.94:0/139211205 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f11cc089960 con 0x7f11dc153990 2024-03-06T07:21:20.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.574+0000 7f11e10e7640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11c006b4d0 0x7f11c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.574+0000 7f11e10e7640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11c006b4d0 0x7f11c006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f11d40012e0 tx=0x7f11d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.578+0000 7f11d1ffb640 1 -- 172.21.15.94:0/139211205 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11cc057ce0 con 0x7f11dc153990 2024-03-06T07:21:20.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.785+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11dc0700a0 con 0x7f11dc153990 2024-03-06T07:21:20.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.786+0000 7f11d1ffb640 1 -- 172.21.15.94:0/139211205 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+6271 (secure 0 0 0) 0x7f11cc05bd80 con 0x7f11dc153990 2024-03-06T07:21:20.787 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:20.788 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":66,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:20:51.223307+0000","modified":"2024-03-06T07:21:15.113184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6424":{"gid":6424,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6837/2023918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":2023918602},{"type":"v1","addr":"172.21.15.145:6837","nonce":2023918602}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6424,"qdb_cluster":[6424]},"id":5},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:03.792493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35486},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35486":{"gid":35486,"name":"d","rank":0,"incarnation":56,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6835/3330183565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3330183565},{"type":"v1","addr":"172.21.15.145:6835","nonce":3330183565}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35486,"qdb_cluster":[35486]},"id":6}]} 2024-03-06T07:21:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 66 2024-03-06T07:21:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.787+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11c006b4d0 msgr2=0x7f11c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.787+0000 7f11e334b640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11c006b4d0 0x7f11c006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f11d40012e0 tx=0x7f11d4009040 comp rx=0 tx=0).stop 2024-03-06T07:21:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.787+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc153990 msgr2=0x7f11dc148fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.787+0000 7f11e334b640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc153990 0x7f11dc148fb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00c4d0 tx=0x7f11cc00c990 comp rx=0 tx=0).stop 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.787+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 shutdown_connections 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.787+0000 7f11e334b640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11c006b4d0 0x7f11c006d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.788+0000 7f11e334b640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11dc156ea0 0x7f11dc1e87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.788+0000 7f11e334b640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11dc153990 0x7f11dc148fb0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.788+0000 7f11e334b640 1 --2- 172.21.15.94:0/139211205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11dc1506b0 0x7f11dc155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.788+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 >> 172.21.15.94:0/139211205 conn(0x7f11dc06b1a0 msgr2=0x7f11dc152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.788+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 shutdown_connections 2024-03-06T07:21:20.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.788+0000 7f11e334b640 1 -- 172.21.15.94:0/139211205 wait complete. 2024-03-06T07:21:20.800 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:20.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.899+0000 7fe27462a640 1 Processor -- start 2024-03-06T07:21:20.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.899+0000 7fe27462a640 1 -- start start 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe27462a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe26c14f790 0x7fe26c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe27462a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe26c0c1aa0 0x7fe26c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe27462a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0c28d0 con 0x7fe26c0c5c60 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe27462a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c0c2680 con 0x7fe26c14f790 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe27462a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c0c2bb0 con 0x7fe26c0c1aa0 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe2723c6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe26c14f790 0x7fe26c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe2723c6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe26c14f790 0x7fe26c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51842/0 (socket says 172.21.15.94:51842) 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe2723c6640 1 -- 172.21.15.94:0/736570481 learned_addr learned my addr 172.21.15.94:0/736570481 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe271bc5640 1 --2- 172.21.15.94:0/736570481 >> v2:172.21.15.145:3300/0 conn(0x7fe26c0c1aa0 0x7fe26c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe2723c6640 1 -- 172.21.15.94:0/736570481 >> v2:172.21.15.145:3300/0 conn(0x7fe26c0c1aa0 msgr2=0x7fe26c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe2723c6640 1 --2- 172.21.15.94:0/736570481 >> v2:172.21.15.145:3300/0 conn(0x7fe26c0c1aa0 0x7fe26c0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe2723c6640 1 -- 172.21.15.94:0/736570481 >> v1:172.21.15.145:6789/0 conn(0x7fe26c0c5c60 legacy=0x7fe26c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.900+0000 7fe2723c6640 1 -- 172.21.15.94:0/736570481 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0c2e90 con 0x7fe26c14f790 2024-03-06T07:21:20.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.901+0000 7fe2723c6640 1 --2- 172.21.15.94:0/736570481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c14f790 0x7fe26c1529d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe26c070280 tx=0x7fe26802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed50c5996e7a706f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.901+0000 7fe2713c4640 1 -- 172.21.15.94:0/736570481 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268034070 con 0x7fe26c14f790 2024-03-06T07:21:20.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.901+0000 7fe2713c4640 1 -- 172.21.15.94:0/736570481 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268030860 con 0x7fe26c14f790 2024-03-06T07:21:20.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.901+0000 7fe2713c4640 1 -- 172.21.15.94:0/736570481 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268030c30 con 0x7fe26c14f790 2024-03-06T07:21:20.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 -- 172.21.15.94:0/736570481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c14f790 msgr2=0x7fe26c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 --2- 172.21.15.94:0/736570481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c14f790 0x7fe26c1529d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe26c070280 tx=0x7fe26802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 -- 172.21.15.94:0/736570481 shutdown_connections 2024-03-06T07:21:20.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 --2- 172.21.15.94:0/736570481 >> v2:172.21.15.145:3300/0 conn(0x7fe26c0c1aa0 0x7fe26c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 --2- 172.21.15.94:0/736570481 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c14f790 0x7fe26c1529d0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 -- 172.21.15.94:0/736570481 >> 172.21.15.94:0/736570481 conn(0x7fe26c0c6710 msgr2=0x7fe26c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:20.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 -- 172.21.15.94:0/736570481 shutdown_connections 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.902+0000 7fe27462a640 1 -- 172.21.15.94:0/736570481 wait complete. 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 Processor -- start 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 -- start start 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe26c0c1aa0 0x7fe26c1eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c0c5c60 0x7fe26c1eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe26c14f790 0x7fe26c14de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c06cc00 con 0x7fe26c0c5c60 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0c0d60 con 0x7fe26c14f790 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe27462a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c06cd80 con 0x7fe26c0c1aa0 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe271bc5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c0c5c60 0x7fe26c1eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe271bc5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c0c5c60 0x7fe26c1eea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51846/0 (socket says 172.21.15.94:51846) 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.903+0000 7fe271bc5640 1 -- 172.21.15.94:0/2226685197 learned_addr learned my addr 172.21.15.94:0/2226685197 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:20.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe2723c6640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe26c0c1aa0 0x7fe26c1eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe272bc7640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe26c14f790 0x7fe26c14de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe271bc5640 1 -- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe26c0c1aa0 msgr2=0x7fe26c1eb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe271bc5640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe26c0c1aa0 0x7fe26c1eb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe271bc5640 1 -- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe26c14f790 msgr2=0x7fe26c14de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe271bc5640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe26c14f790 0x7fe26c14de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe271bc5640 1 -- 172.21.15.94:0/2226685197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c1eefa0 con 0x7fe26c0c5c60 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe271bc5640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c0c5c60 0x7fe26c1eea60 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00e9b0 tx=0x7fe25c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe262ffd640 1 -- 172.21.15.94:0/2226685197 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c017070 con 0x7fe26c0c5c60 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe262ffd640 1 -- 172.21.15.94:0/2226685197 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c00c9a0 con 0x7fe26c0c5c60 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c1eaa60 con 0x7fe26c0c5c60 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe262ffd640 1 -- 172.21.15.94:0/2226685197 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00cd70 con 0x7fe26c0c5c60 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.904+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c0c0030 con 0x7fe26c0c5c60 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.905+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c04f330 con 0x7fe26c0c5c60 2024-03-06T07:21:20.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.905+0000 7fe262ffd640 1 -- 172.21.15.94:0/2226685197 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe25c01a070 con 0x7fe26c0c5c60 2024-03-06T07:21:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.906+0000 7fe262ffd640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe25006b4d0 0x7fe25006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.906+0000 7fe262ffd640 1 -- 172.21.15.94:0/2226685197 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fe25c08a5b0 con 0x7fe26c0c5c60 2024-03-06T07:21:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.906+0000 7fe2723c6640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe25006b4d0 0x7fe25006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:20.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.906+0000 7fe2723c6640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe25006b4d0 0x7fe25006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe268036000 tx=0x7fe26803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:20.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:20.910+0000 7fe262ffd640 1 -- 172.21.15.94:0/2226685197 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe25c058930 con 0x7fe26c0c5c60 2024-03-06T07:21:21.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.083+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe26c0c1f40 con 0x7fe26c0c5c60 2024-03-06T07:21:21.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.084+0000 7fe262ffd640 1 -- 172.21.15.94:0/2226685197 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20800 (secure 0 0 0) 0x7fe25c05c9d0 con 0x7fe26c0c5c60 2024-03-06T07:21:21.086 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:21.087 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":79,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:11.096698+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:20:49.440763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-06T07:20:50.185600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/3537750458":"2024-03-07T07:21:11.096670+0000","172.21.15.145:6836/3537750458":"2024-03-07T07:21:11.096670+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe25006b4d0 msgr2=0x7fe25006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe25006b4d0 0x7fe25006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe268036000 tx=0x7fe26803b040 comp rx=0 tx=0).stop 2024-03-06T07:21:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c0c5c60 msgr2=0x7fe26c1eea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c0c5c60 0x7fe26c1eea60 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00e9b0 tx=0x7fe25c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 shutdown_connections 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe25006b4d0 0x7fe25006d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe26c14f790 0x7fe26c14de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe26c0c5c60 0x7fe26c1eea60 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 --2- 172.21.15.94:0/2226685197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe26c0c1aa0 0x7fe26c1eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 >> 172.21.15.94:0/2226685197 conn(0x7fe26c0c6710 msgr2=0x7fe26c0a9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 shutdown_connections 2024-03-06T07:21:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.086+0000 7fe27462a640 1 -- 172.21.15.94:0/2226685197 wait complete. 2024-03-06T07:21:21.099 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:21:21.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.199+0000 7f322d4b5640 1 Processor -- start 2024-03-06T07:21:21.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.199+0000 7f322d4b5640 1 -- start start 2024-03-06T07:21:21.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.199+0000 7f322d4b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280bfe90 0x7f3228155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.199+0000 7f322d4b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32280c71d0 con 0x7f32280c6d70 2024-03-06T07:21:21.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.199+0000 7f322d4b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3228159590 con 0x7f32281522d0 2024-03-06T07:21:21.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.199+0000 7f322d4b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3228158710 con 0x7f32280bfe90 2024-03-06T07:21:21.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f32267fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f32280c6d70 0x7f3228158e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37676/0 (socket says 172.21.15.94:37676) 2024-03-06T07:21:21.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f32267fc640 1 -- 172.21.15.94:0/3873101838 learned_addr learned my addr 172.21.15.94:0/3873101838 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:21.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3226ffd640 1 --2- 172.21.15.94:0/3873101838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280bfe90 0x7f3228155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3225ffb640 1 -- 172.21.15.94:0/3873101838 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228711548 0 0) 0x7f32280c71d0 con 0x7f32280c6d70 2024-03-06T07:21:21.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3225ffb640 1 -- 172.21.15.94:0/3873101838 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3210003520 con 0x7f32280c6d70 2024-03-06T07:21:21.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3225ffb640 1 -- 172.21.15.94:0/3873101838 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770971083 0 0) 0x7f3228159590 con 0x7f32281522d0 2024-03-06T07:21:21.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3225ffb640 1 -- 172.21.15.94:0/3873101838 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32280c71d0 con 0x7f32281522d0 2024-03-06T07:21:21.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3226ffd640 1 -- 172.21.15.94:0/3873101838 >> v1:172.21.15.145:6789/0 conn(0x7f32281522d0 legacy=0x7f322815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3226ffd640 1 -- 172.21.15.94:0/3873101838 >> v1:172.21.15.94:6789/0 conn(0x7f32280c6d70 legacy=0x7f3228158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.200+0000 7f3226ffd640 1 -- 172.21.15.94:0/3873101838 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32280c59d0 con 0x7f32280bfe90 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f3226ffd640 1 --2- 172.21.15.94:0/3873101838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280bfe90 0x7f3228155120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f32180097d0 tx=0x7f321802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=492f94e300f28449 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f3225ffb640 1 -- 172.21.15.94:0/3873101838 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3218035020 con 0x7f32280bfe90 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f3225ffb640 1 -- 172.21.15.94:0/3873101838 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3218031800 con 0x7f32280bfe90 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f3225ffb640 1 -- 172.21.15.94:0/3873101838 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3218031bd0 con 0x7f32280bfe90 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f322d4b5640 1 -- 172.21.15.94:0/3873101838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280bfe90 msgr2=0x7f3228155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f322d4b5640 1 --2- 172.21.15.94:0/3873101838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280bfe90 0x7f3228155120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f32180097d0 tx=0x7f321802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f322d4b5640 1 -- 172.21.15.94:0/3873101838 shutdown_connections 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f322d4b5640 1 --2- 172.21.15.94:0/3873101838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280bfe90 0x7f3228155120 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.201+0000 7f322d4b5640 1 -- 172.21.15.94:0/3873101838 >> 172.21.15.94:0/3873101838 conn(0x7f32280acaa0 msgr2=0x7f32280aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.202+0000 7f322d4b5640 1 -- 172.21.15.94:0/3873101838 shutdown_connections 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.202+0000 7f322d4b5640 1 -- 172.21.15.94:0/3873101838 wait complete. 2024-03-06T07:21:21.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 Processor -- start 2024-03-06T07:21:21.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 -- start start 2024-03-06T07:21:21.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32280bfe90 0x7f32280ad660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280c6d70 0x7f32280afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32281522d0 0x7f3228140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3228158960 con 0x7f32281522d0 2024-03-06T07:21:21.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32280c5f90 con 0x7f32280bfe90 2024-03-06T07:21:21.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.203+0000 7f322d4b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32280ca700 con 0x7f32280c6d70 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32281522d0 0x7f3228140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f3226ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32280bfe90 0x7f32280ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32281522d0 0x7f3228140340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51852/0 (socket says 172.21.15.94:51852) 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 -- 172.21.15.94:0/4143246341 learned_addr learned my addr 172.21.15.94:0/4143246341 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32267fc640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280c6d70 0x7f32280afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 -- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280c6d70 msgr2=0x7f32280afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280c6d70 0x7f32280afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 -- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32280bfe90 msgr2=0x7f32280ad660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32280bfe90 0x7f32280ad660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 -- 172.21.15.94:0/4143246341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322806cbc0 con 0x7f32281522d0 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f32277fe640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32281522d0 0x7f3228140340 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f322000c6e0 tx=0x7f322000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f320f7fe640 1 -- 172.21.15.94:0/4143246341 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220003b70 con 0x7f32281522d0 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f320f7fe640 1 -- 172.21.15.94:0/4143246341 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3220003d10 con 0x7f32281522d0 2024-03-06T07:21:21.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3228153fe0 con 0x7f32281522d0 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f320f7fe640 1 -- 172.21.15.94:0/4143246341 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32200052d0 con 0x7f32281522d0 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.204+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32281545a0 con 0x7f32281522d0 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.205+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32280c8620 con 0x7f32281522d0 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.206+0000 7f320f7fe640 1 -- 172.21.15.94:0/4143246341 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3220005470 con 0x7f32281522d0 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.206+0000 7f320f7fe640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f321406b500 0x7f321406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.206+0000 7f320f7fe640 1 -- 172.21.15.94:0/4143246341 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f322008a460 con 0x7f32281522d0 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.206+0000 7f3226ffd640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f321406b500 0x7f321406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.207+0000 7f3226ffd640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f321406b500 0x7f321406d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3218036000 tx=0x7f3218008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.210+0000 7f320f7fe640 1 -- 172.21.15.94:0/4143246341 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32280c8620 con 0x7f32281522d0 2024-03-06T07:21:21.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.419+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f32280c53d0 con 0x7f32281522d0 2024-03-06T07:21:21.421 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:21:21.420+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:21:21.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.421+0000 7f320f7fe640 1 -- 172.21.15.94:0/4143246341 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7f32200587e0 con 0x7f32281522d0 2024-03-06T07:21:21.422 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:21:21.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f321406b500 msgr2=0x7f321406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f321406b500 0x7f321406d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3218036000 tx=0x7f3218008030 comp rx=0 tx=0).stop 2024-03-06T07:21:21.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32281522d0 msgr2=0x7f3228140340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32281522d0 0x7f3228140340 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f322000c6e0 tx=0x7f322000cba0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 shutdown_connections 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f321406b500 0x7f321406d9c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f32281522d0 0x7f3228140340 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32280c6d70 0x7f32280afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 --2- 172.21.15.94:0/4143246341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32280bfe90 0x7f32280ad660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.423+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 >> 172.21.15.94:0/4143246341 conn(0x7f32280acaa0 msgr2=0x7f32280c62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.424+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 shutdown_connections 2024-03-06T07:21:21.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.424+0000 7f322d4b5640 1 -- 172.21.15.94:0/4143246341 wait complete. 2024-03-06T07:21:21.427 INFO:tasks.ceph.mds.b.smithi145.stderr: -133> 2024-03-06T07:21:18.104+0000 7f4fe12d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:21.436 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:21:21.450 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:21:21.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.539+0000 7f658cf02640 1 Processor -- start 2024-03-06T07:21:21.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.539+0000 7f658cf02640 1 -- start start 2024-03-06T07:21:21.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f658cf02640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f658806c1f0 0x7f65880c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f658cf02640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65880c2790 con 0x7f658806cad0 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f658cf02640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658806cf20 con 0x7f658814ef70 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f658cf02640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65880c3090 con 0x7f658806c1f0 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f6586d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f658806cad0 0x7f6588154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37678/0 (socket says 172.21.15.94:37678) 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f6586d9d640 1 -- 172.21.15.94:0/2614967988 learned_addr learned my addr 172.21.15.94:0/2614967988 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f6585d9b640 1 --2- 172.21.15.94:0/2614967988 >> v2:172.21.15.145:3300/0 conn(0x7f658806c1f0 0x7f65880c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f658559a640 1 -- 172.21.15.94:0/2614967988 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998006490 0 0) 0x7f65880c2790 con 0x7f658806cad0 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.540+0000 7f658559a640 1 -- 172.21.15.94:0/2614967988 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6570003520 con 0x7f658806cad0 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.541+0000 7f6585d9b640 1 -- 172.21.15.94:0/2614967988 >> v1:172.21.15.145:6789/0 conn(0x7f658814ef70 legacy=0x7f65880c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.541+0000 7f658559a640 1 -- 172.21.15.94:0/2614967988 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172191751 0 0) 0x7f658806cf20 con 0x7f658814ef70 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.541+0000 7f6585d9b640 1 -- 172.21.15.94:0/2614967988 >> v1:172.21.15.94:6789/0 conn(0x7f658806cad0 legacy=0x7f6588154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.541+0000 7f6585d9b640 1 -- 172.21.15.94:0/2614967988 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65880c1540 con 0x7f658806c1f0 2024-03-06T07:21:21.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.541+0000 7f6585d9b640 1 --2- 172.21.15.94:0/2614967988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 0x7f65880c1820 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f657c0097a0 tx=0x7f657c02dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f76848b2615c2b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.541+0000 7f658559a640 1 -- 172.21.15.94:0/2614967988 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c02f8a0 con 0x7f658806c1f0 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658559a640 1 -- 172.21.15.94:0/2614967988 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c02fa40 con 0x7f658806c1f0 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658559a640 1 -- 172.21.15.94:0/2614967988 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c02fe10 con 0x7f658806c1f0 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658cf02640 1 -- 172.21.15.94:0/2614967988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 msgr2=0x7f65880c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658cf02640 1 --2- 172.21.15.94:0/2614967988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 0x7f65880c1820 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f657c0097a0 tx=0x7f657c02dae0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658cf02640 1 -- 172.21.15.94:0/2614967988 shutdown_connections 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658cf02640 1 --2- 172.21.15.94:0/2614967988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 0x7f65880c1820 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658cf02640 1 -- 172.21.15.94:0/2614967988 >> 172.21.15.94:0/2614967988 conn(0x7f65880c6710 msgr2=0x7f65880c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658cf02640 1 -- 172.21.15.94:0/2614967988 shutdown_connections 2024-03-06T07:21:21.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.542+0000 7f658cf02640 1 -- 172.21.15.94:0/2614967988 wait complete. 2024-03-06T07:21:21.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.543+0000 7f658cf02640 1 Processor -- start 2024-03-06T07:21:21.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.543+0000 7f658cf02640 1 -- start start 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f658cf02640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 0x7f65880a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f658cf02640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658806cad0 0x7f65880a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f658cf02640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658814ef70 0x7f65880a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f658cf02640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6588155750 con 0x7f658806cad0 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f658cf02640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658814f350 con 0x7f658806c1f0 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f658cf02640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6588155340 con 0x7f658814ef70 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658806cad0 0x7f65880a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658806cad0 0x7f65880a8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51862/0 (socket says 172.21.15.94:51862) 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 -- 172.21.15.94:0/1766270940 learned_addr learned my addr 172.21.15.94:0/1766270940 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f658659c640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 0x7f65880a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6586d9d640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658814ef70 0x7f65880a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 -- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658814ef70 msgr2=0x7f65880a9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658814ef70 0x7f65880a9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 -- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 msgr2=0x7f65880a6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 0x7f65880a6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.544+0000 7f6585d9b640 1 -- 172.21.15.94:0/1766270940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65881db6f0 con 0x7f658806cad0 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.545+0000 7f6585d9b640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658806cad0 0x7f65880a8af0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f657c02e0f0 tx=0x7f657c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.545+0000 7f6576ffd640 1 -- 172.21.15.94:0/1766270940 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c049070 con 0x7f658806cad0 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.545+0000 7f6576ffd640 1 -- 172.21.15.94:0/1766270940 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c030cf0 con 0x7f658806cad0 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.545+0000 7f6576ffd640 1 -- 172.21.15.94:0/1766270940 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c039620 con 0x7f658806cad0 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.545+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65881db9d0 con 0x7f658806cad0 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.545+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65880a9030 con 0x7f658806cad0 2024-03-06T07:21:21.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.546+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658804f330 con 0x7f658806cad0 2024-03-06T07:21:21.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.546+0000 7f6576ffd640 1 -- 172.21.15.94:0/1766270940 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f657c0397c0 con 0x7f658806cad0 2024-03-06T07:21:21.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.547+0000 7f6576ffd640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f656806b500 0x7f656806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:21.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.547+0000 7f6576ffd640 1 -- 172.21.15.94:0/1766270940 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f657c0ae650 con 0x7f658806cad0 2024-03-06T07:21:21.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.547+0000 7f658659c640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f656806b500 0x7f656806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:21.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.547+0000 7f658659c640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f656806b500 0x7f656806d9c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f65780012e0 tx=0x7f6578009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:21.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.551+0000 7f6576ffd640 1 -- 172.21.15.94:0/1766270940 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f657c035030 con 0x7f658806cad0 2024-03-06T07:21:21.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:21.758+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f65880c1110 con 0x7f658806cad0 2024-03-06T07:21:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.191+0000 7f6576ffd640 1 -- 172.21.15.94:0/1766270940 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f657c07c980 con 0x7f658806cad0 2024-03-06T07:21:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f656806b500 msgr2=0x7f656806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f656806b500 0x7f656806d9c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f65780012e0 tx=0x7f6578009040 comp rx=0 tx=0).stop 2024-03-06T07:21:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658806cad0 msgr2=0x7f65880a8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658806cad0 0x7f65880a8af0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f657c02e0f0 tx=0x7f657c004730 comp rx=0 tx=0).stop 2024-03-06T07:21:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 shutdown_connections 2024-03-06T07:21:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f656806b500 0x7f656806d9c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658814ef70 0x7f65880a9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658806cad0 0x7f65880a8af0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 --2- 172.21.15.94:0/1766270940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f658806c1f0 0x7f65880a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 >> 172.21.15.94:0/1766270940 conn(0x7f65880c6710 msgr2=0x7f65880c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 shutdown_connections 2024-03-06T07:21:22.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.194+0000 7f658cf02640 1 -- 172.21.15.94:0/1766270940 wait complete. 2024-03-06T07:21:22.199 INFO:tasks.ceph.mds.d.smithi145.stderr: -497> 2024-03-06T07:21:06.139+0000 7ff1a967d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:22.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:21:22.225 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:21:22.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.312+0000 7f3743e88640 1 Processor -- start 2024-03-06T07:21:22.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.312+0000 7f3743e88640 1 -- start start 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3743e88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f373c14ebb0 0x7f373c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3743e88640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f373c0c28d0 0x7f373c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3743e88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f373c14ef90 con 0x7f373c0a9980 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3743e88640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f373c155530 con 0x7f373c0c28d0 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3743e88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f373c155a60 con 0x7f373c14ebb0 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3742425640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f373c0c28d0 0x7f373c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3742425640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f373c0c28d0 0x7f373c06dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51872/0 (socket says 172.21.15.94:51872) 2024-03-06T07:21:22.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3742425640 1 -- 172.21.15.94:0/1949058538 learned_addr learned my addr 172.21.15.94:0/1949058538 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:22.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3741c24640 1 --2- 172.21.15.94:0/1949058538 >> v2:172.21.15.145:3300/0 conn(0x7f373c14ebb0 0x7f373c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3742425640 1 -- 172.21.15.94:0/1949058538 >> v2:172.21.15.145:3300/0 conn(0x7f373c14ebb0 msgr2=0x7f373c1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:22.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3742425640 1 --2- 172.21.15.94:0/1949058538 >> v2:172.21.15.145:3300/0 conn(0x7f373c14ebb0 0x7f373c1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3742425640 1 -- 172.21.15.94:0/1949058538 >> v1:172.21.15.145:6789/0 conn(0x7f373c0a9980 legacy=0x7f373c0a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:22.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.313+0000 7f3742425640 1 -- 172.21.15.94:0/1949058538 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c155780 con 0x7f373c0c28d0 2024-03-06T07:21:22.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3742425640 1 --2- 172.21.15.94:0/1949058538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0c28d0 0x7f373c06dba0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f373c06e790 tx=0x7f373402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfd5ddd3e47213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:22.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3740c22640 1 -- 172.21.15.94:0/1949058538 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373402c9f0 con 0x7f373c0c28d0 2024-03-06T07:21:22.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3740c22640 1 -- 172.21.15.94:0/1949058538 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373402cb90 con 0x7f373c0c28d0 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3740c22640 1 -- 172.21.15.94:0/1949058538 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37340343f0 con 0x7f373c0c28d0 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3743e88640 1 -- 172.21.15.94:0/1949058538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0c28d0 msgr2=0x7f373c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3743e88640 1 --2- 172.21.15.94:0/1949058538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0c28d0 0x7f373c06dba0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f373c06e790 tx=0x7f373402be50 comp rx=0 tx=0).stop 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3743e88640 1 -- 172.21.15.94:0/1949058538 shutdown_connections 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3743e88640 1 --2- 172.21.15.94:0/1949058538 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0c28d0 0x7f373c06dba0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3743e88640 1 --2- 172.21.15.94:0/1949058538 >> v2:172.21.15.145:3300/0 conn(0x7f373c14ebb0 0x7f373c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.314+0000 7f3743e88640 1 -- 172.21.15.94:0/1949058538 >> 172.21.15.94:0/1949058538 conn(0x7f373c0c65b0 msgr2=0x7f373c0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.315+0000 7f3743e88640 1 -- 172.21.15.94:0/1949058538 shutdown_connections 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.315+0000 7f3743e88640 1 -- 172.21.15.94:0/1949058538 wait complete. 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 Processor -- start 2024-03-06T07:21:22.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 -- start start 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0a9980 0x7f373c1ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c0c28d0 0x7f373c0c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c14ebb0 0x7f373c1e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373c151c90 con 0x7f373c0a9980 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373c1519b0 con 0x7f373c0c28d0 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3743e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373c0c2d80 con 0x7f373c14ebb0 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3741c24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0a9980 0x7f373c1ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3741c24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0a9980 0x7f373c1ec5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51878/0 (socket says 172.21.15.94:51878) 2024-03-06T07:21:22.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3741c24640 1 -- 172.21.15.94:0/677862774 learned_addr learned my addr 172.21.15.94:0/677862774 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3742425640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c14ebb0 0x7f373c1e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.316+0000 7f3741423640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c0c28d0 0x7f373c0c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3741c24640 1 -- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c14ebb0 msgr2=0x7f373c1e9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3741c24640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c14ebb0 0x7f373c1e9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3741c24640 1 -- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c0c28d0 msgr2=0x7f373c0c11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3741c24640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c0c28d0 0x7f373c0c11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3741c24640 1 -- 172.21.15.94:0/677862774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c0be4c0 con 0x7f373c0a9980 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3741c24640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0a9980 0x7f373c1ec5d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f373800d620 tx=0x7f373800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f37327fc640 1 -- 172.21.15.94:0/677862774 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373800b620 con 0x7f373c0a9980 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f37327fc640 1 -- 172.21.15.94:0/677862774 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3738019070 con 0x7f373c0a9980 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373c1ecc90 con 0x7f373c0a9980 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f37327fc640 1 -- 172.21.15.94:0/677862774 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373800ba00 con 0x7f373c0a9980 2024-03-06T07:21:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.317+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373c1ed250 con 0x7f373c0a9980 2024-03-06T07:21:22.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.318+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f373c067ad0 con 0x7f373c0a9980 2024-03-06T07:21:22.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.318+0000 7f37327fc640 1 -- 172.21.15.94:0/677862774 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f373800bba0 con 0x7f373c0a9980 2024-03-06T07:21:22.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.319+0000 7f37327fc640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f372006b4d0 0x7f372006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:22.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.319+0000 7f37327fc640 1 -- 172.21.15.94:0/677862774 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f373808aa20 con 0x7f373c0a9980 2024-03-06T07:21:22.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.319+0000 7f3741423640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f372006b4d0 0x7f372006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:22.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.319+0000 7f3741423640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f372006b4d0 0x7f372006d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f373c1444d0 tx=0x7f372c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:22.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.323+0000 7f37327fc640 1 -- 172.21.15.94:0/677862774 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f373c067ad0 con 0x7f373c0a9980 2024-03-06T07:21:22.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:22.494+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f373c04f330 con 0x7f373c0a9980 2024-03-06T07:21:23.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.196+0000 7f37327fc640 1 -- 172.21.15.94:0/677862774 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3738058d00 con 0x7f373c0a9980 2024-03-06T07:21:23.198 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:21:23.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.199+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f372006b4d0 msgr2=0x7f372006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.199+0000 7f3743e88640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f372006b4d0 0x7f372006d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f373c1444d0 tx=0x7f372c007e30 comp rx=0 tx=0).stop 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.199+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0a9980 msgr2=0x7f373c1ec5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.199+0000 7f3743e88640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0a9980 0x7f373c1ec5d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f373800d620 tx=0x7f373800dae0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 shutdown_connections 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f372006b4d0 0x7f372006d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c14ebb0 0x7f373c1e9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c0c28d0 0x7f373c0c11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 --2- 172.21.15.94:0/677862774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f373c0a9980 0x7f373c1ec5d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 >> 172.21.15.94:0/677862774 conn(0x7f373c0c65b0 msgr2=0x7f373c14dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 shutdown_connections 2024-03-06T07:21:23.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.200+0000 7f3743e88640 1 -- 172.21.15.94:0/677862774 wait complete. 2024-03-06T07:21:23.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:21:23.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.321+0000 7f68c8d20640 1 Processor -- start 2024-03-06T07:21:23.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.321+0000 7f68c8d20640 1 -- start start 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.322+0000 7f68c8d20640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40c53f0 0x7f68c406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.322+0000 7f68c8d20640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c413e070 con 0x7f68c406b700 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.322+0000 7f68c8d20640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c413e420 con 0x7f68c413dc90 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.322+0000 7f68c8d20640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c406c190 con 0x7f68c40c53f0 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.323+0000 7f68c259c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f68c413dc90 0x7f68c4159230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:40530/0 (socket says 172.21.15.94:40530) 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.323+0000 7f68c259c640 1 -- 172.21.15.94:0/327931487 learned_addr learned my addr 172.21.15.94:0/327931487 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.323+0000 7f68c1d9b640 1 --2- 172.21.15.94:0/327931487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40c53f0 0x7f68c406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.323+0000 7f68c159a640 1 -- 172.21.15.94:0/327931487 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324711453 0 0) 0x7f68c413e420 con 0x7f68c413dc90 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.323+0000 7f68c159a640 1 -- 172.21.15.94:0/327931487 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b0003520 con 0x7f68c413dc90 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.323+0000 7f68c1d9b640 1 -- 172.21.15.94:0/327931487 >> v1:172.21.15.145:6789/0 conn(0x7f68c413dc90 legacy=0x7f68c4159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.323+0000 7f68c1d9b640 1 -- 172.21.15.94:0/327931487 >> v1:172.21.15.94:6789/0 conn(0x7f68c406b700 legacy=0x7f68c40c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.324+0000 7f68c1d9b640 1 -- 172.21.15.94:0/327931487 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c413e700 con 0x7f68c40c53f0 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.325+0000 7f68c1d9b640 1 --2- 172.21.15.94:0/327931487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40c53f0 0x7f68c406b1c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0097d0 tx=0x7f68ac02b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f77cffac00fabbb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.326+0000 7f68c159a640 1 -- 172.21.15.94:0/327931487 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac037070 con 0x7f68c40c53f0 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.326+0000 7f68c159a640 1 -- 172.21.15.94:0/327931487 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac030710 con 0x7f68c40c53f0 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.326+0000 7f68c159a640 1 -- 172.21.15.94:0/327931487 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac030ae0 con 0x7f68c40c53f0 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.326+0000 7f68c8d20640 1 -- 172.21.15.94:0/327931487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40c53f0 msgr2=0x7f68c406b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.326+0000 7f68c8d20640 1 --2- 172.21.15.94:0/327931487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40c53f0 0x7f68c406b1c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0097d0 tx=0x7f68ac02b6e0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.327+0000 7f68c8d20640 1 -- 172.21.15.94:0/327931487 shutdown_connections 2024-03-06T07:21:23.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.327+0000 7f68c8d20640 1 --2- 172.21.15.94:0/327931487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40c53f0 0x7f68c406b1c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.327+0000 7f68c8d20640 1 -- 172.21.15.94:0/327931487 >> 172.21.15.94:0/327931487 conn(0x7f68c40caa70 msgr2=0x7f68c40cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:23.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.327+0000 7f68c8d20640 1 -- 172.21.15.94:0/327931487 shutdown_connections 2024-03-06T07:21:23.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.327+0000 7f68c8d20640 1 -- 172.21.15.94:0/327931487 wait complete. 2024-03-06T07:21:23.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.328+0000 7f68c8d20640 1 Processor -- start 2024-03-06T07:21:23.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.328+0000 7f68c8d20640 1 -- start start 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c406b700 0x7f68c41eb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40c53f0 0x7f68c40abf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68c413dc90 0x7f68c41eefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40c61f0 con 0x7f68c413dc90 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c406bde0 con 0x7f68c40c53f0 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40c8970 con 0x7f68c406b700 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68c413dc90 0x7f68c41eefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68c413dc90 0x7f68c41eefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51890/0 (socket says 172.21.15.94:51890) 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 -- 172.21.15.94:0/197747718 learned_addr learned my addr 172.21.15.94:0/197747718 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c1d9b640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40c53f0 0x7f68c40abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c259c640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c406b700 0x7f68c41eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 -- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c406b700 msgr2=0x7f68c41eb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c406b700 0x7f68c41eb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 -- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40c53f0 msgr2=0x7f68c40abf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40c53f0 0x7f68c40abf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 -- 172.21.15.94:0/197747718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c41584e0 con 0x7f68c413dc90 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c259c640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c406b700 0x7f68c41eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c2d9d640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68c413dc90 0x7f68c41eefc0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c780 tx=0x7f68b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68baffd640 1 -- 172.21.15.94:0/197747718 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4017070 con 0x7f68c413dc90 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68baffd640 1 -- 172.21.15.94:0/197747718 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b4005270 con 0x7f68c413dc90 2024-03-06T07:21:23.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c41587c0 con 0x7f68c413dc90 2024-03-06T07:21:23.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68baffd640 1 -- 172.21.15.94:0/197747718 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b400a640 con 0x7f68c413dc90 2024-03-06T07:21:23.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.329+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c4152520 con 0x7f68c413dc90 2024-03-06T07:21:23.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.330+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c41ee890 con 0x7f68c413dc90 2024-03-06T07:21:23.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.331+0000 7f68baffd640 1 -- 172.21.15.94:0/197747718 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68b401a070 con 0x7f68c413dc90 2024-03-06T07:21:23.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.331+0000 7f68baffd640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68a406b4d0 0x7f68a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:23.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.331+0000 7f68baffd640 1 -- 172.21.15.94:0/197747718 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6777+0+0 (secure 0 0 0) 0x7f68b40899e0 con 0x7f68c413dc90 2024-03-06T07:21:23.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.331+0000 7f68c259c640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68a406b4d0 0x7f68a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:23.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.332+0000 7f68c259c640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68a406b4d0 0x7f68a406d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f68bc005c00 tx=0x7f68bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:23.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.335+0000 7f68baffd640 1 -- 172.21.15.94:0/197747718 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68b4057e70 con 0x7f68c413dc90 2024-03-06T07:21:23.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:23.508+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f68c404f330 con 0x7f68c413dc90 2024-03-06T07:21:24.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.204+0000 7f68baffd640 1 -- 172.21.15.94:0/197747718 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) v1 ==== 169+0+0 (secure 0 0 0) 0x7f68b405bf10 con 0x7f68c413dc90 2024-03-06T07:21:24.206 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:21:24.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68a406b4d0 msgr2=0x7f68a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68a406b4d0 0x7f68a406d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f68bc005c00 tx=0x7f68bc009040 comp rx=0 tx=0).stop 2024-03-06T07:21:24.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68c413dc90 msgr2=0x7f68c41eefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68c413dc90 0x7f68c41eefc0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c780 tx=0x7f68b400cc40 comp rx=0 tx=0).stop 2024-03-06T07:21:24.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 shutdown_connections 2024-03-06T07:21:24.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68a406b4d0 0x7f68a406d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68c413dc90 0x7f68c41eefc0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40c53f0 0x7f68c40abf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 --2- 172.21.15.94:0/197747718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c406b700 0x7f68c41eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 >> 172.21.15.94:0/197747718 conn(0x7f68c40caa70 msgr2=0x7f68c40cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 shutdown_connections 2024-03-06T07:21:24.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.208+0000 7f68c8d20640 1 -- 172.21.15.94:0/197747718 wait complete. 2024-03-06T07:21:24.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:24.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.324+0000 7f993f777640 1 Processor -- start 2024-03-06T07:21:24.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.324+0000 7f993f777640 1 -- start start 2024-03-06T07:21:24.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.324+0000 7f993f777640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99381550c0 0x7f99381554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.324+0000 7f993f777640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99380aa660 con 0x7f99380aa180 2024-03-06T07:21:24.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.324+0000 7f993f777640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99380aa940 con 0x7f99381559e0 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.324+0000 7f993f777640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99380c4610 con 0x7f99381550c0 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.325+0000 7f993d513640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99381550c0 0x7f99381554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.325+0000 7f993dd14640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f99380aa180 0x7f99380c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37702/0 (socket says 172.21.15.94:37702) 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.325+0000 7f993dd14640 1 -- 172.21.15.94:0/1176418903 learned_addr learned my addr 172.21.15.94:0/1176418903 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.325+0000 7f993d513640 1 -- 172.21.15.94:0/1176418903 >> v1:172.21.15.145:6789/0 conn(0x7f99381559e0 legacy=0x7f993814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.325+0000 7f993d513640 1 -- 172.21.15.94:0/1176418903 >> v1:172.21.15.94:6789/0 conn(0x7f99380aa180 legacy=0x7f99380c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.325+0000 7f992ffff640 1 -- 172.21.15.94:0/1176418903 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589826873 0 0) 0x7f99380aa660 con 0x7f99380aa180 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.325+0000 7f993d513640 1 -- 172.21.15.94:0/1176418903 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99380c4860 con 0x7f99381550c0 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993d513640 1 --2- 172.21.15.94:0/1176418903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381550c0 0x7f99381554a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f99380c4790 tx=0x7f993002cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8489b986c9fd465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f992ffff640 1 -- 172.21.15.94:0/1176418903 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993003a070 con 0x7f99381550c0 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f992ffff640 1 -- 172.21.15.94:0/1176418903 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993002da10 con 0x7f99381550c0 2024-03-06T07:21:24.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f992ffff640 1 -- 172.21.15.94:0/1176418903 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993000b040 con 0x7f99381550c0 2024-03-06T07:21:24.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993f777640 1 -- 172.21.15.94:0/1176418903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381550c0 msgr2=0x7f99381554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993f777640 1 --2- 172.21.15.94:0/1176418903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381550c0 0x7f99381554a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f99380c4790 tx=0x7f993002cc60 comp rx=0 tx=0).stop 2024-03-06T07:21:24.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993f777640 1 -- 172.21.15.94:0/1176418903 shutdown_connections 2024-03-06T07:21:24.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993f777640 1 --2- 172.21.15.94:0/1176418903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381550c0 0x7f99381554a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993f777640 1 -- 172.21.15.94:0/1176418903 >> 172.21.15.94:0/1176418903 conn(0x7f99380c6710 msgr2=0x7f99380c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993f777640 1 -- 172.21.15.94:0/1176418903 shutdown_connections 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.326+0000 7f993f777640 1 -- 172.21.15.94:0/1176418903 wait complete. 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.327+0000 7f993f777640 1 Processor -- start 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.327+0000 7f993f777640 1 -- start start 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993f777640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99380aa180 0x7f993814ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993f777640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99381550c0 0x7f9938141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993f777640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381559e0 0x7f9938141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993f777640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993814f490 con 0x7f99381559e0 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993f777640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993814f310 con 0x7f99381550c0 2024-03-06T07:21:24.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993f777640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99380a6590 con 0x7f99380aa180 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381559e0 0x7f9938141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381559e0 0x7f9938141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51914/0 (socket says 172.21.15.94:51914) 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 -- 172.21.15.94:0/3086351996 learned_addr learned my addr 172.21.15.94:0/3086351996 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993d513640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99380aa180 0x7f993814ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993cd12640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99381550c0 0x7f9938141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 -- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99380aa180 msgr2=0x7f993814ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99380aa180 0x7f993814ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 -- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99381550c0 msgr2=0x7f9938141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99381550c0 0x7f9938141820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.328+0000 7f993dd14640 1 -- 172.21.15.94:0/3086351996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99381422a0 con 0x7f99381559e0 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.329+0000 7f993dd14640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381559e0 0x7f9938141d60 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f993400c6d0 tx=0x7f993400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.329+0000 7f992dffb640 1 -- 172.21.15.94:0/3086351996 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9934018070 con 0x7f99381559e0 2024-03-06T07:21:24.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.329+0000 7f992dffb640 1 -- 172.21.15.94:0/3086351996 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9934004d60 con 0x7f99381559e0 2024-03-06T07:21:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.329+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99380a72c0 con 0x7f99381559e0 2024-03-06T07:21:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.329+0000 7f992dffb640 1 -- 172.21.15.94:0/3086351996 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993400a880 con 0x7f99381559e0 2024-03-06T07:21:24.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.329+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99380a7880 con 0x7f99381559e0 2024-03-06T07:21:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.330+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993804f330 con 0x7f99381559e0 2024-03-06T07:21:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.330+0000 7f992dffb640 1 -- 172.21.15.94:0/3086351996 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993401b070 con 0x7f99381559e0 2024-03-06T07:21:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.330+0000 7f992dffb640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f991c06b500 0x7f991c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.331+0000 7f992dffb640 1 -- 172.21.15.94:0/3086351996 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f9934089b30 con 0x7f99381559e0 2024-03-06T07:21:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.331+0000 7f993d513640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f991c06b500 0x7f991c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.331+0000 7f993d513640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f991c06b500 0x7f991c06d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9930037000 tx=0x7f993000c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.335+0000 7f992dffb640 1 -- 172.21.15.94:0/3086351996 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9934058720 con 0x7f99381559e0 2024-03-06T07:21:24.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.507+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9938068450 con 0x7f99381559e0 2024-03-06T07:21:24.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.508+0000 7f992dffb640 1 -- 172.21.15.94:0/3086351996 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17284 (secure 0 0 0) 0x7f9934058020 con 0x7f99381559e0 2024-03-06T07:21:24.510 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:24.510 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":83,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:24.197861+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6834/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6837/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6836/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6837/3537750458":"2024-03-07T07:21:11.096670+0000","172.21.15.145:6836/3537750458":"2024-03-07T07:21:11.096670+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f991c06b500 msgr2=0x7f991c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f991c06b500 0x7f991c06d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9930037000 tx=0x7f993000c040 comp rx=0 tx=0).stop 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381559e0 msgr2=0x7f9938141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381559e0 0x7f9938141d60 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f993400c6d0 tx=0x7f993400cb90 comp rx=0 tx=0).stop 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 shutdown_connections 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f991c06b500 0x7f991c06d9c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99381559e0 0x7f9938141d60 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99381550c0 0x7f9938141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 --2- 172.21.15.94:0/3086351996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99380aa180 0x7f993814ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 >> 172.21.15.94:0/3086351996 conn(0x7f99380c6710 msgr2=0x7f99380c2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 shutdown_connections 2024-03-06T07:21:24.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.510+0000 7f993f777640 1 -- 172.21.15.94:0/3086351996 wait complete. 2024-03-06T07:21:24.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:20:59.828663+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:21:00.744064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:24.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:24.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 Processor -- start 2024-03-06T07:21:24.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 -- start start 2024-03-06T07:21:24.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f17bc14d350 0x7f17bc06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0acbc0 0x7f17bc0ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f17bc0abe90 0x7f17bc156540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17bc0ac750 con 0x7f17bc14d350 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17bc0c1a40 con 0x7f17bc0abe90 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.626+0000 7f17c2447640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc153c00 con 0x7f17bc0acbc0 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f17bc14d350 0x7f17bc06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f17bc14d350 0x7f17bc06afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51918/0 (socket says 172.21.15.94:51918) 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 -- 172.21.15.94:0/1911654819 learned_addr learned my addr 172.21.15.94:0/1911654819 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bb7fe640 1 --2- 172.21.15.94:0/1911654819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0acbc0 0x7f17bc0ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17c09e4640 1 --2- 172.21.15.94:0/1911654819 >> v2:172.21.15.145:3300/0 conn(0x7f17bc0abe90 0x7f17bc156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 -- 172.21.15.94:0/1911654819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0acbc0 msgr2=0x7f17bc0ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 --2- 172.21.15.94:0/1911654819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0acbc0 0x7f17bc0ab950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 -- 172.21.15.94:0/1911654819 >> v2:172.21.15.145:3300/0 conn(0x7f17bc0abe90 msgr2=0x7f17bc156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:24.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 --2- 172.21.15.94:0/1911654819 >> v2:172.21.15.145:3300/0 conn(0x7f17bc0abe90 0x7f17bc156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bbfff640 1 -- 172.21.15.94:0/1911654819 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc0c1c90 con 0x7f17bc14d350 2024-03-06T07:21:24.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.627+0000 7f17bb7fe640 1 --2- 172.21.15.94:0/1911654819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0acbc0 0x7f17bc0ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:24.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17bbfff640 1 --2- 172.21.15.94:0/1911654819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc14d350 0x7f17bc06afc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0c8060 tx=0x7f17ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe51cd76e0060e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17baffd640 1 -- 172.21.15.94:0/1911654819 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac02e070 con 0x7f17bc14d350 2024-03-06T07:21:24.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17baffd640 1 -- 172.21.15.94:0/1911654819 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ac037070 con 0x7f17bc14d350 2024-03-06T07:21:24.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17baffd640 1 -- 172.21.15.94:0/1911654819 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ac031a00 con 0x7f17bc14d350 2024-03-06T07:21:24.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 -- 172.21.15.94:0/1911654819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc14d350 msgr2=0x7f17bc06afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 --2- 172.21.15.94:0/1911654819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc14d350 0x7f17bc06afc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0c8060 tx=0x7f17ac02b880 comp rx=0 tx=0).stop 2024-03-06T07:21:24.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 -- 172.21.15.94:0/1911654819 shutdown_connections 2024-03-06T07:21:24.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 --2- 172.21.15.94:0/1911654819 >> v2:172.21.15.145:3300/0 conn(0x7f17bc0abe90 0x7f17bc156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 --2- 172.21.15.94:0/1911654819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0acbc0 0x7f17bc0ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 --2- 172.21.15.94:0/1911654819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc14d350 0x7f17bc06afc0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 -- 172.21.15.94:0/1911654819 >> 172.21.15.94:0/1911654819 conn(0x7f17bc0c3b40 msgr2=0x7f17bc0c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 -- 172.21.15.94:0/1911654819 shutdown_connections 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.628+0000 7f17c2447640 1 -- 172.21.15.94:0/1911654819 wait complete. 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.629+0000 7f17c2447640 1 Processor -- start 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c2447640 1 -- start start 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c2447640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0abe90 0x7f17bc156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c2447640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc0acbc0 0x7f17bc1e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c2447640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17bc14d350 0x7f17bc1489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c2447640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc06ea70 con 0x7f17bc0acbc0 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c2447640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc0aa160 con 0x7f17bc14d350 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c2447640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc1ec3e0 con 0x7f17bc0abe90 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17bb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc0acbc0 0x7f17bc1e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17bb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc0acbc0 0x7f17bc1e9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51920/0 (socket says 172.21.15.94:51920) 2024-03-06T07:21:24.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17bb7fe640 1 -- 172.21.15.94:0/442357308 learned_addr learned my addr 172.21.15.94:0/442357308 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17bbfff640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0abe90 0x7f17bc156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.630+0000 7f17c09e4640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17bc14d350 0x7f17bc1489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17bb7fe640 1 -- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0abe90 msgr2=0x7f17bc156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17bb7fe640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0abe90 0x7f17bc156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17bb7fe640 1 -- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17bc14d350 msgr2=0x7f17bc1489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17bb7fe640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17bc14d350 0x7f17bc1489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17bb7fe640 1 -- 172.21.15.94:0/442357308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc1ec0e0 con 0x7f17bc0acbc0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17bb7fe640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc0acbc0 0x7f17bc1e9f40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f17b000e7e0 tx=0x7f17b000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17b8ff9640 1 -- 172.21.15.94:0/442357308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b000f040 con 0x7f17bc0acbc0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17b8ff9640 1 -- 172.21.15.94:0/442357308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b0004030 con 0x7f17bc0acbc0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17b8ff9640 1 -- 172.21.15.94:0/442357308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0004400 con 0x7f17bc0acbc0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc146540 con 0x7f17bc0acbc0 2024-03-06T07:21:24.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.631+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc146820 con 0x7f17bc0acbc0 2024-03-06T07:21:24.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.632+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17bc04f330 con 0x7f17bc0acbc0 2024-03-06T07:21:24.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.632+0000 7f17b8ff9640 1 -- 172.21.15.94:0/442357308 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17b000c670 con 0x7f17bc0acbc0 2024-03-06T07:21:24.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.633+0000 7f17b8ff9640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17a806b500 0x7f17a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.633+0000 7f17b8ff9640 1 -- 172.21.15.94:0/442357308 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f17b008a080 con 0x7f17bc0acbc0 2024-03-06T07:21:24.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.633+0000 7f17bbfff640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17a806b500 0x7f17a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.633+0000 7f17bbfff640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17a806b500 0x7f17a806d9c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17ac036000 tx=0x7f17ac03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.637+0000 7f17b8ff9640 1 -- 172.21.15.94:0/442357308 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17b0017020 con 0x7f17bc0acbc0 2024-03-06T07:21:24.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.807+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17bc0c49e0 con 0x7f17bc0acbc0 2024-03-06T07:21:24.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.807+0000 7f17b8ff9640 1 -- 172.21.15.94:0/442357308 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17b00586b0 con 0x7f17bc0acbc0 2024-03-06T07:21:24.809 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17a806b500 msgr2=0x7f17a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17a806b500 0x7f17a806d9c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17ac036000 tx=0x7f17ac03d040 comp rx=0 tx=0).stop 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc0acbc0 msgr2=0x7f17bc1e9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc0acbc0 0x7f17bc1e9f40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f17b000e7e0 tx=0x7f17b000eca0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 shutdown_connections 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17a806b500 0x7f17a806d9c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17bc14d350 0x7f17bc1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17bc0acbc0 0x7f17bc1e9f40 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 --2- 172.21.15.94:0/442357308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17bc0abe90 0x7f17bc156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 >> 172.21.15.94:0/442357308 conn(0x7f17bc0c3b40 msgr2=0x7f17bc0c1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 shutdown_connections 2024-03-06T07:21:24.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.809+0000 7f17c2447640 1 -- 172.21.15.94:0/442357308 wait complete. 2024-03-06T07:21:24.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:21:24.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.923+0000 7f9f84be9640 1 Processor -- start 2024-03-06T07:21:24.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.923+0000 7f9f84be9640 1 -- start start 2024-03-06T07:21:24.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f84be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800a8290 0x7f9f801539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f84be9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9f800c69a0 0x7f9f80072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f84be9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f800c62a0 con 0x7f9f800c4740 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f84be9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f80153f00 con 0x7f9f800c69a0 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f84be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800c4c00 con 0x7f9f800a8290 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f7e59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800a8290 0x7f9f801539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f7ed9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9f800c69a0 0x7f9f80072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f7e59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800a8290 0x7f9f801539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:42502/0 (socket says 172.21.15.94:42502) 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.924+0000 7f9f7e59c640 1 -- 172.21.15.94:0/2464874321 learned_addr learned my addr 172.21.15.94:0/2464874321 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.925+0000 7f9f7e59c640 1 -- 172.21.15.94:0/2464874321 >> v2:172.21.15.145:3300/0 conn(0x7f9f800c69a0 msgr2=0x7f9f80072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.925+0000 7f9f7e59c640 1 --2- 172.21.15.94:0/2464874321 >> v2:172.21.15.145:3300/0 conn(0x7f9f800c69a0 0x7f9f80072680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.925+0000 7f9f7e59c640 1 -- 172.21.15.94:0/2464874321 >> v1:172.21.15.145:6789/0 conn(0x7f9f800c4740 legacy=0x7f9f800701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.925+0000 7f9f7e59c640 1 -- 172.21.15.94:0/2464874321 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800c4ee0 con 0x7f9f800a8290 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.925+0000 7f9f7e59c640 1 --2- 172.21.15.94:0/2464874321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800a8290 0x7f9f801539c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009e50 tx=0x7f9f7002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fb29bd2ef5f5c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f7d59a640 1 -- 172.21.15.94:0/2464874321 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7002c9e0 con 0x7f9f800a8290 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f7d59a640 1 -- 172.21.15.94:0/2464874321 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7002cb80 con 0x7f9f800a8290 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f7d59a640 1 -- 172.21.15.94:0/2464874321 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700343f0 con 0x7f9f800a8290 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 -- 172.21.15.94:0/2464874321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800a8290 msgr2=0x7f9f801539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 --2- 172.21.15.94:0/2464874321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800a8290 0x7f9f801539c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009e50 tx=0x7f9f7002be40 comp rx=0 tx=0).stop 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 -- 172.21.15.94:0/2464874321 shutdown_connections 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 --2- 172.21.15.94:0/2464874321 >> v2:172.21.15.145:3300/0 conn(0x7f9f800c69a0 0x7f9f80072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 --2- 172.21.15.94:0/2464874321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800a8290 0x7f9f801539c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 -- 172.21.15.94:0/2464874321 >> 172.21.15.94:0/2464874321 conn(0x7f9f800c5560 msgr2=0x7f9f800c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 -- 172.21.15.94:0/2464874321 shutdown_connections 2024-03-06T07:21:24.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.926+0000 7f9f84be9640 1 -- 172.21.15.94:0/2464874321 wait complete. 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 Processor -- start 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 -- start start 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f800a8290 0x7f9f80154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800c4740 0x7f9f800adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f800c69a0 0x7f9f801ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800c6e50 con 0x7f9f800a8290 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8006fc80 con 0x7f9f800c69a0 2024-03-06T07:21:24.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f84be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f80072bc0 con 0x7f9f800c4740 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f800a8290 0x7f9f80154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f7e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f800a8290 0x7f9f80154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51936/0 (socket says 172.21.15.94:51936) 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f7e59c640 1 -- 172.21.15.94:0/906678098 learned_addr learned my addr 172.21.15.94:0/906678098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f7dd9b640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800c4740 0x7f9f800adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.928+0000 7f9f7ed9d640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f800c69a0 0x7f9f801ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f7e59c640 1 -- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800c4740 msgr2=0x7f9f800adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f7e59c640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800c4740 0x7f9f800adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f7e59c640 1 -- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f800c69a0 msgr2=0x7f9f801ec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f7e59c640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f800c69a0 0x7f9f801ec180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f7e59c640 1 -- 172.21.15.94:0/906678098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800aba40 con 0x7f9f800a8290 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f7e59c640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f800a8290 0x7f9f80154b60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009f70 tx=0x7f9f70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f6effd640 1 -- 172.21.15.94:0/906678098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70043020 con 0x7f9f800a8290 2024-03-06T07:21:24.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f6effd640 1 -- 172.21.15.94:0/906678098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7003e450 con 0x7f9f800a8290 2024-03-06T07:21:24.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f800abd20 con 0x7f9f800a8290 2024-03-06T07:21:24.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f6effd640 1 -- 172.21.15.94:0/906678098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7003e840 con 0x7f9f800a8290 2024-03-06T07:21:24.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.929+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f801550a0 con 0x7f9f800a8290 2024-03-06T07:21:24.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.930+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8004f330 con 0x7f9f800a8290 2024-03-06T07:21:24.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.930+0000 7f9f6effd640 1 -- 172.21.15.94:0/906678098 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f70046020 con 0x7f9f800a8290 2024-03-06T07:21:24.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.931+0000 7f9f6effd640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f6006b4d0 0x7f9f6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:24.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.931+0000 7f9f6effd640 1 -- 172.21.15.94:0/906678098 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f9f700acac0 con 0x7f9f800a8290 2024-03-06T07:21:24.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.931+0000 7f9f7dd9b640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f6006b4d0 0x7f9f6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:24.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.931+0000 7f9f7dd9b640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f6006b4d0 0x7f9f6006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f800aa710 tx=0x7f9f74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:24.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:24.935+0000 7f9f6effd640 1 -- 172.21.15.94:0/906678098 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f7004d030 con 0x7f9f800a8290 2024-03-06T07:21:25.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.113+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f800735e0 con 0x7f9f800a8290 2024-03-06T07:21:25.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.114+0000 7f9f6effd640 1 -- 172.21.15.94:0/906678098 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v83) v1 ==== 105+0+11 (secure 0 0 0) 0x7f9f7007b0f0 con 0x7f9f800a8290 2024-03-06T07:21:25.168 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:21:25.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.115+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f6006b4d0 msgr2=0x7f9f6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.115+0000 7f9f84be9640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f6006b4d0 0x7f9f6006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f800aa710 tx=0x7f9f74009040 comp rx=0 tx=0).stop 2024-03-06T07:21:25.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.115+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f800a8290 msgr2=0x7f9f80154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.115+0000 7f9f84be9640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f800a8290 0x7f9f80154b60 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009f70 tx=0x7f9f70004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:25.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 shutdown_connections 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f6006b4d0 0x7f9f6006d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f800c69a0 0x7f9f801ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f800c4740 0x7f9f800adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 --2- 172.21.15.94:0/906678098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f800a8290 0x7f9f80154b60 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 >> 172.21.15.94:0/906678098 conn(0x7f9f800c5560 msgr2=0x7f9f801dc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 shutdown_connections 2024-03-06T07:21:25.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.116+0000 7f9f84be9640 1 -- 172.21.15.94:0/906678098 wait complete. 2024-03-06T07:21:25.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:21:25.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.277+0000 7f2f67ef4640 1 Processor -- start 2024-03-06T07:21:25.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.277+0000 7f2f67ef4640 1 -- start start 2024-03-06T07:21:25.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.277+0000 7f2f67ef4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2f6014ef70 0x7f2f6006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f67ef4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f600c1820 0x7f2f6006cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f67ef4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f600c2790 con 0x7f2f600c2980 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f67ef4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6014f350 con 0x7f2f600c1820 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f67ef4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f600c1540 con 0x7f2f6014ef70 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f6548f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f600c1820 0x7f2f6006cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f66491640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2f600c2980 0x7f2f60154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37704/0 (socket says 172.21.15.94:37704) 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f6548f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f600c1820 0x7f2f6006cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51938/0 (socket says 172.21.15.94:51938) 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f66491640 1 -- 172.21.15.94:0/2934367615 learned_addr learned my addr 172.21.15.94:0/2934367615 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f65c90640 1 --2- 172.21.15.94:0/2934367615 >> v2:172.21.15.145:3300/0 conn(0x7f2f6014ef70 0x7f2f6006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f6548f640 1 -- 172.21.15.94:0/2934367615 >> v2:172.21.15.145:3300/0 conn(0x7f2f6014ef70 msgr2=0x7f2f6006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f6548f640 1 --2- 172.21.15.94:0/2934367615 >> v2:172.21.15.145:3300/0 conn(0x7f2f6014ef70 0x7f2f6006c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f6548f640 1 -- 172.21.15.94:0/2934367615 >> v1:172.21.15.94:6789/0 conn(0x7f2f600c2980 legacy=0x7f2f60154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f6548f640 1 -- 172.21.15.94:0/2934367615 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f600c24b0 con 0x7f2f600c1820 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f64c8e640 1 -- 172.21.15.94:0/2934367615 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819290306 0 0) 0x7f2f600c2790 con 0x7f2f600c2980 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.278+0000 7f2f65c90640 1 --2- 172.21.15.94:0/2934367615 >> v2:172.21.15.145:3300/0 conn(0x7f2f6014ef70 0x7f2f6006c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f6548f640 1 --2- 172.21.15.94:0/2934367615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c1820 0x7f2f6006cad0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f2f6006e5d0 tx=0x7f2f5002cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252323f8b79138b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f64c8e640 1 -- 172.21.15.94:0/2934367615 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5003a070 con 0x7f2f600c1820 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f64c8e640 1 -- 172.21.15.94:0/2934367615 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5002dad0 con 0x7f2f600c1820 2024-03-06T07:21:25.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f64c8e640 1 -- 172.21.15.94:0/2934367615 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5000b040 con 0x7f2f600c1820 2024-03-06T07:21:25.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f67ef4640 1 -- 172.21.15.94:0/2934367615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c1820 msgr2=0x7f2f6006cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/2934367615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c1820 0x7f2f6006cad0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f2f6006e5d0 tx=0x7f2f5002cd20 comp rx=0 tx=0).stop 2024-03-06T07:21:25.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f67ef4640 1 -- 172.21.15.94:0/2934367615 shutdown_connections 2024-03-06T07:21:25.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/2934367615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c1820 0x7f2f6006cad0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/2934367615 >> v2:172.21.15.145:3300/0 conn(0x7f2f6014ef70 0x7f2f6006c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.279+0000 7f2f67ef4640 1 -- 172.21.15.94:0/2934367615 >> 172.21.15.94:0/2934367615 conn(0x7f2f600c6710 msgr2=0x7f2f600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.280+0000 7f2f67ef4640 1 -- 172.21.15.94:0/2934367615 shutdown_connections 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.280+0000 7f2f67ef4640 1 -- 172.21.15.94:0/2934367615 wait complete. 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 Processor -- start 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 -- start start 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f600c1820 0x7f2f601daeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c2980 0x7f2f600a87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f6014ef70 0x7f2f600a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f601554c0 con 0x7f2f600c2980 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f60155340 con 0x7f2f6014ef70 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f67ef4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f600c2df0 con 0x7f2f600c1820 2024-03-06T07:21:25.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f6548f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c2980 0x7f2f600a87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f6548f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c2980 0x7f2f600a87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51954/0 (socket says 172.21.15.94:51954) 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f6548f640 1 -- 172.21.15.94:0/4040509738 learned_addr learned my addr 172.21.15.94:0/4040509738 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f65c90640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f600c1820 0x7f2f601daeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.281+0000 7f2f66491640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f6014ef70 0x7f2f600a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f6548f640 1 -- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f600c1820 msgr2=0x7f2f601daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f6548f640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f600c1820 0x7f2f601daeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f6548f640 1 -- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f6014ef70 msgr2=0x7f2f600a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f6548f640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f6014ef70 0x7f2f600a8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f6548f640 1 -- 172.21.15.94:0/4040509738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f601db750 con 0x7f2f600c2980 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f65c90640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f600c1820 0x7f2f601daeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f66491640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f6014ef70 0x7f2f600a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f6548f640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c2980 0x7f2f600a87a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f2f50000c00 tx=0x7f2f500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f567fc640 1 -- 172.21.15.94:0/4040509738 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50045070 con 0x7f2f600c2980 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f567fc640 1 -- 172.21.15.94:0/4040509738 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5003e550 con 0x7f2f600c2980 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f601467e0 con 0x7f2f600c2980 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f567fc640 1 -- 172.21.15.94:0/4040509738 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5003a070 con 0x7f2f600c2980 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.282+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6014a070 con 0x7f2f600c2980 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.283+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6004f330 con 0x7f2f600c2980 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.283+0000 7f2f567fc640 1 -- 172.21.15.94:0/4040509738 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f50048070 con 0x7f2f600c2980 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.284+0000 7f2f567fc640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f4406b4d0 0x7f2f4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.284+0000 7f2f567fc640 1 -- 172.21.15.94:0/4040509738 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f2f500abfe0 con 0x7f2f600c2980 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.284+0000 7f2f65c90640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f4406b4d0 0x7f2f4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.285+0000 7f2f65c90640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f4406b4d0 0x7f2f4406d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c00a990 tx=0x7f2f5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.288+0000 7f2f567fc640 1 -- 172.21.15.94:0/4040509738 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f5007ac60 con 0x7f2f600c2980 2024-03-06T07:21:25.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.468+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f2f600c1110 con 0x7f2f600c2980 2024-03-06T07:21:25.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.469+0000 7f2f567fc640 1 -- 172.21.15.94:0/4040509738 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v83) v1 ==== 101+0+11 (secure 0 0 0) 0x7f2f5007a560 con 0x7f2f600c2980 2024-03-06T07:21:25.471 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f4406b4d0 msgr2=0x7f2f4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f4406b4d0 0x7f2f4406d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c00a990 tx=0x7f2f5c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c2980 msgr2=0x7f2f600a87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c2980 0x7f2f600a87a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f2f50000c00 tx=0x7f2f500046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 shutdown_connections 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f4406b4d0 0x7f2f4406d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f6014ef70 0x7f2f600a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f600c2980 0x7f2f600a87a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 --2- 172.21.15.94:0/4040509738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f600c1820 0x7f2f601daeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 >> 172.21.15.94:0/4040509738 conn(0x7f2f600c6710 msgr2=0x7f2f60153990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 shutdown_connections 2024-03-06T07:21:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.471+0000 7f2f67ef4640 1 -- 172.21.15.94:0/4040509738 wait complete. 2024-03-06T07:21:25.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.583+0000 7f0b8c37a640 1 Processor -- start 2024-03-06T07:21:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.583+0000 7f0b8c37a640 1 -- start start 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8c37a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0b840c88c0 0x7f0b84155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8c37a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0b84158b60 0x7f0b84158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8c37a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b840c73d0 con 0x7f0b8414f360 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8c37a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b840c8dd0 con 0x7f0b84158b60 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8c37a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b840be2e0 con 0x7f0b840c88c0 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8a917640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0b84158b60 0x7f0b84158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8a917640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0b84158b60 0x7f0b84158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51958/0 (socket says 172.21.15.94:51958) 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8a116640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0b8414f360 0x7f0b840c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37720/0 (socket says 172.21.15.94:37720) 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.584+0000 7f0b8a917640 1 -- 172.21.15.94:0/3920811931 learned_addr learned my addr 172.21.15.94:0/3920811931 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b89915640 1 --2- 172.21.15.94:0/3920811931 >> v2:172.21.15.145:3300/0 conn(0x7f0b840c88c0 0x7f0b84155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b8a917640 1 -- 172.21.15.94:0/3920811931 >> v2:172.21.15.145:3300/0 conn(0x7f0b840c88c0 msgr2=0x7f0b84155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b8a917640 1 --2- 172.21.15.94:0/3920811931 >> v2:172.21.15.145:3300/0 conn(0x7f0b840c88c0 0x7f0b84155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b8a917640 1 -- 172.21.15.94:0/3920811931 >> v1:172.21.15.94:6789/0 conn(0x7f0b8414f360 legacy=0x7f0b840c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b8a917640 1 -- 172.21.15.94:0/3920811931 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b840c7670 con 0x7f0b84158b60 2024-03-06T07:21:25.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b89915640 1 --2- 172.21.15.94:0/3920811931 >> v2:172.21.15.145:3300/0 conn(0x7f0b840c88c0 0x7f0b84155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:25.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b8a917640 1 --2- 172.21.15.94:0/3920811931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b84158ff0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0b840c84c0 tx=0x7f0b7c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2371cc367a6592e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.585+0000 7f0b89114640 1 -- 172.21.15.94:0/3920811931 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c02ca80 con 0x7f0b84158b60 2024-03-06T07:21:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b89114640 1 -- 172.21.15.94:0/3920811931 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c02cc20 con 0x7f0b84158b60 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b89114640 1 -- 172.21.15.94:0/3920811931 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c0343f0 con 0x7f0b84158b60 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 -- 172.21.15.94:0/3920811931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 msgr2=0x7f0b84158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/3920811931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b84158ff0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0b840c84c0 tx=0x7f0b7c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 -- 172.21.15.94:0/3920811931 shutdown_connections 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/3920811931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b84158ff0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/3920811931 >> v2:172.21.15.145:3300/0 conn(0x7f0b840c88c0 0x7f0b84155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 -- 172.21.15.94:0/3920811931 >> 172.21.15.94:0/3920811931 conn(0x7f0b840ac570 msgr2=0x7f0b840ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 -- 172.21.15.94:0/3920811931 shutdown_connections 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.586+0000 7f0b8c37a640 1 -- 172.21.15.94:0/3920811931 wait complete. 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.587+0000 7f0b8c37a640 1 Processor -- start 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.587+0000 7f0b8c37a640 1 -- start start 2024-03-06T07:21:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8c37a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b840c88c0 0x7f0b84154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8c37a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b8414f360 0x7f0b8414fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8c37a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b841500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8c37a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840c5a50 con 0x7f0b84158b60 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8c37a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840c7c30 con 0x7f0b840c88c0 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8c37a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840c5bd0 con 0x7f0b8414f360 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b841500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b841500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51972/0 (socket says 172.21.15.94:51972) 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 -- 172.21.15.94:0/1837716556 learned_addr learned my addr 172.21.15.94:0/1837716556 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b89915640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b8414f360 0x7f0b8414fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a116640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b840c88c0 0x7f0b84154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 -- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b8414f360 msgr2=0x7f0b8414fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b8414f360 0x7f0b8414fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 -- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b840c88c0 msgr2=0x7f0b84154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b840c88c0 0x7f0b84154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 -- 172.21.15.94:0/1837716556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84152700 con 0x7f0b84158b60 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8a917640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b841500b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c02c4f0 tx=0x7f0b7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b7affd640 1 -- 172.21.15.94:0/1837716556 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c043020 con 0x7f0b84158b60 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b7affd640 1 -- 172.21.15.94:0/1837716556 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c03e450 con 0x7f0b84158b60 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.588+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b840bd880 con 0x7f0b84158b60 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.589+0000 7f0b7affd640 1 -- 172.21.15.94:0/1837716556 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c03e840 con 0x7f0b84158b60 2024-03-06T07:21:25.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.589+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b840c0480 con 0x7f0b84158b60 2024-03-06T07:21:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.589+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8404f330 con 0x7f0b84158b60 2024-03-06T07:21:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.590+0000 7f0b7affd640 1 -- 172.21.15.94:0/1837716556 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b7c046020 con 0x7f0b84158b60 2024-03-06T07:21:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.590+0000 7f0b7affd640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b6806b4d0 0x7f0b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.591+0000 7f0b7affd640 1 -- 172.21.15.94:0/1837716556 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f0b7c0ad120 con 0x7f0b84158b60 2024-03-06T07:21:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.591+0000 7f0b8a116640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b6806b4d0 0x7f0b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.591+0000 7f0b8a116640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b6806b4d0 0x7f0b6806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0b840c7f10 tx=0x7f0b80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.594+0000 7f0b7affd640 1 -- 172.21.15.94:0/1837716556 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b7c04d030 con 0x7f0b84158b60 2024-03-06T07:21:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.809+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b841554a0 con 0x7f0b84158b60 2024-03-06T07:21:25.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.809+0000 7f0b7affd640 1 -- 172.21.15.94:0/1837716556 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f0b7c07bfa0 con 0x7f0b84158b60 2024-03-06T07:21:25.812 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:25.812 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767091656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767091656},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767091656}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":35795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3407932030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3407932030},{"type":"v1","addr":"172.21.15.145:6837","nonce":3407932030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:25.198745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":14,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":6}]} 2024-03-06T07:21:25.813 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 71 2024-03-06T07:21:25.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b6806b4d0 msgr2=0x7f0b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b6806b4d0 0x7f0b6806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0b840c7f10 tx=0x7f0b80009040 comp rx=0 tx=0).stop 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 msgr2=0x7f0b841500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b841500b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c02c4f0 tx=0x7f0b7c004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 shutdown_connections 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b6806b4d0 0x7f0b6806d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b84158b60 0x7f0b841500b0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b8414f360 0x7f0b8414fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 --2- 172.21.15.94:0/1837716556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b840c88c0 0x7f0b84154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 >> 172.21.15.94:0/1837716556 conn(0x7f0b840ac570 msgr2=0x7f0b840ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.811+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 shutdown_connections 2024-03-06T07:21:25.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.812+0000 7f0b8c37a640 1 -- 172.21.15.94:0/1837716556 wait complete. 2024-03-06T07:21:25.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:25.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89faaf4640 1 Processor -- start 2024-03-06T07:21:25.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89faaf4640 1 -- start start 2024-03-06T07:21:25.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89faaf4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f89f4159270 0x7f89f4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89faaf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f40c73d0 con 0x7f89f40c88c0 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89faaf4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f40be2e0 con 0x7f89f414f360 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89faaf4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f4159700 con 0x7f89f4159270 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89f3fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f89f40c88c0 0x7f89f4158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37722/0 (socket says 172.21.15.94:37722) 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.925+0000 7f89f3fff640 1 -- 172.21.15.94:0/43125820 learned_addr learned my addr 172.21.15.94:0/43125820 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233604756 0 0) 0x7f89f40c73d0 con 0x7f89f40c88c0 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f9091640 1 --2- 172.21.15.94:0/43125820 >> v2:172.21.15.145:3300/0 conn(0x7f89f4159270 0x7f89f4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e4003520 con 0x7f89f40c88c0 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394439181 0 0) 0x7f89e4003520 con 0x7f89f40c88c0 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 >> v2:172.21.15.145:3300/0 conn(0x7f89f4159270 msgr2=0x7f89f4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 --2- 172.21.15.94:0/43125820 >> v2:172.21.15.145:3300/0 conn(0x7f89f4159270 0x7f89f4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 >> v1:172.21.15.145:6789/0 conn(0x7f89f414f360 legacy=0x7f89f40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40c7670 con 0x7f89f40c88c0 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f89e0001670 con 0x7f89f40c88c0 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 >> v1:172.21.15.94:6789/0 conn(0x7f89f40c88c0 legacy=0x7f89f4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89e40051c0 0x7f89e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4159700 con 0x7f89e40051c0 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e400c160 con 0x7f89e4004800 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e400c360 con 0x7f89e40039b0 2024-03-06T07:21:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f9091640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89e40051c0 0x7f89e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f3fff640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f8890640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f9091640 1 -- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89e40039b0 msgr2=0x7f89e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f9091640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f9091640 1 -- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e4004800 msgr2=0x7f89e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.926+0000 7f89f9091640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.927+0000 7f89f9091640 1 -- 172.21.15.94:0/43125820 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e4003520 con 0x7f89e40051c0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.927+0000 7f89f9091640 1 -- 172.21.15.94:0/43125820 reap_dead start 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.927+0000 7f89f9091640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89e40051c0 0x7f89e400b9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009fc0 tx=0x7f89e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a096c27291177e83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.927+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec009c10 con 0x7f89e40051c0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.927+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec02cc60 con 0x7f89e40051c0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.927+0000 7f89f37fe640 1 -- 172.21.15.94:0/43125820 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec003bd0 con 0x7f89e40051c0 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.928+0000 7f89faaf4640 1 -- 172.21.15.94:0/43125820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89e40051c0 msgr2=0x7f89e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.928+0000 7f89faaf4640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89e40051c0 0x7f89e400b9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009fc0 tx=0x7f89e4005830 comp rx=0 tx=0).stop 2024-03-06T07:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.928+0000 7f89faaf4640 1 -- 172.21.15.94:0/43125820 shutdown_connections 2024-03-06T07:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.928+0000 7f89faaf4640 1 --2- 172.21.15.94:0/43125820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89e40051c0 0x7f89e400b9c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.928+0000 7f89faaf4640 1 -- 172.21.15.94:0/43125820 >> 172.21.15.94:0/43125820 conn(0x7f89f40ac570 msgr2=0x7f89f40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.928+0000 7f89faaf4640 1 -- 172.21.15.94:0/43125820 shutdown_connections 2024-03-06T07:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.928+0000 7f89faaf4640 1 -- 172.21.15.94:0/43125820 wait complete. 2024-03-06T07:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.929+0000 7f89faaf4640 1 Processor -- start 2024-03-06T07:21:25.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.929+0000 7f89faaf4640 1 -- start start 2024-03-06T07:21:25.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89faaf4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e40051c0 0x7f89f40c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89faaf4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40c46b0 0x7f89f40b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89faaf4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f4157840 0x7f89f40ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89faaf4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f40c7c30 con 0x7f89f4157840 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89faaf4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4155c10 con 0x7f89e40051c0 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89faaf4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4155d90 con 0x7f89f40c46b0 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f4157840 0x7f89f40ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f4157840 0x7f89f40ad320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51990/0 (socket says 172.21.15.94:51990) 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 -- 172.21.15.94:0/3553789308 learned_addr learned my addr 172.21.15.94:0/3553789308 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f3fff640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40c46b0 0x7f89f40b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f8890640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e40051c0 0x7f89f40c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 -- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40c46b0 msgr2=0x7f89f40b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40c46b0 0x7f89f40b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 -- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e40051c0 msgr2=0x7f89f40c0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e40051c0 0x7f89f40c0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 -- 172.21.15.94:0/3553789308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40c0950 con 0x7f89f4157840 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.930+0000 7f89f9091640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f4157840 0x7f89f40ad320 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f89ec02c6d0 tx=0x7f89ec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.931+0000 7f89f17fa640 1 -- 172.21.15.94:0/3553789308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec003990 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.931+0000 7f89f17fa640 1 -- 172.21.15.94:0/3553789308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec044070 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.931+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f40c0f30 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.931+0000 7f89f17fa640 1 -- 172.21.15.94:0/3553789308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec003d90 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.931+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f40c11f0 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.932+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f414d470 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.932+0000 7f89f17fa640 1 -- 172.21.15.94:0/3553789308 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89ec005d10 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.932+0000 7f89f17fa640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.933+0000 7f89f17fa640 1 -- 172.21.15.94:0/3553789308 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f89ec0acbb0 con 0x7f89f4157840 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.933+0000 7f89f8890640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.933+0000 7f89f8890640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f89e8005490 tx=0x7f89e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:25.937+0000 7f89f17fa640 1 -- 172.21.15.94:0/3553789308 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89f414d470 con 0x7f89f4157840 2024-03-06T07:21:26.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.131+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89f404f330 con 0x7f89f4157840 2024-03-06T07:21:26.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.132+0000 7f89f17fa640 1 -- 172.21.15.94:0/3553789308 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17284 (secure 0 0 0) 0x7f89ec0b0020 con 0x7f89f4157840 2024-03-06T07:21:26.134 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:26.135 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":83,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:24.197861+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6834/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6837/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6836/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6837/3537750458":"2024-03-07T07:21:11.096670+0000","172.21.15.145:6836/3537750458":"2024-03-07T07:21:11.096670+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:26.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.134+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 msgr2=0x7f89d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.134+0000 7f89faaf4640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f89e8005490 tx=0x7f89e8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.134+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f4157840 msgr2=0x7f89f40ad320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.134+0000 7f89faaf4640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f4157840 0x7f89f40ad320 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f89ec02c6d0 tx=0x7f89ec009750 comp rx=0 tx=0).stop 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.134+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 shutdown_connections 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.134+0000 7f89faaf4640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.135+0000 7f89faaf4640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f4157840 0x7f89f40ad320 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.135+0000 7f89faaf4640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40c46b0 0x7f89f40b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.135+0000 7f89faaf4640 1 --2- 172.21.15.94:0/3553789308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89e40051c0 0x7f89f40c0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.135+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 >> 172.21.15.94:0/3553789308 conn(0x7f89f40ac570 msgr2=0x7f89f414e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.135+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 shutdown_connections 2024-03-06T07:21:26.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.135+0000 7f89faaf4640 1 -- 172.21.15.94:0/3553789308 wait complete. 2024-03-06T07:21:26.140 INFO:tasks.ceph.mds.a.smithi094.stderr:2024-03-06T07:21:26.139+0000 7f57b22e2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:26.148 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-06T07:21:26.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:26.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.247+0000 7f43b5781640 1 Processor -- start 2024-03-06T07:21:26.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.247+0000 7f43b5781640 1 -- start start 2024-03-06T07:21:26.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43b5781640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b0152960 0x7f43b006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43b5781640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43b00ac260 0x7f43b01559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43b5781640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b0152d40 con 0x7f43b01592d0 2024-03-06T07:21:26.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43b5781640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b0066e30 con 0x7f43b00ac260 2024-03-06T07:21:26.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43b5781640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00c1dc0 con 0x7f43b0152960 2024-03-06T07:21:26.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43af7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43b00ac260 0x7f43b01559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43af7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43b00ac260 0x7f43b01559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52000/0 (socket says 172.21.15.94:52000) 2024-03-06T07:21:26.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.248+0000 7f43af7fe640 1 -- 172.21.15.94:0/2885193012 learned_addr learned my addr 172.21.15.94:0/2885193012 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:26.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.249+0000 7f43aeffd640 1 --2- 172.21.15.94:0/2885193012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b0152960 0x7f43b006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.249+0000 7f43af7fe640 1 -- 172.21.15.94:0/2885193012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b0152960 msgr2=0x7f43b006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.249+0000 7f43af7fe640 1 --2- 172.21.15.94:0/2885193012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b0152960 0x7f43b006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.249+0000 7f43af7fe640 1 -- 172.21.15.94:0/2885193012 >> v1:172.21.15.145:6789/0 conn(0x7f43b01592d0 legacy=0x7f43b01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.249+0000 7f43af7fe640 1 -- 172.21.15.94:0/2885193012 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b00c1b50 con 0x7f43b00ac260 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.249+0000 7f43af7fe640 1 --2- 172.21.15.94:0/2885193012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b00ac260 0x7f43b01559e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009870 tx=0x7f43a002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c5c5b48d57dd20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43adffb640 1 -- 172.21.15.94:0/2885193012 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0034070 con 0x7f43b00ac260 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43adffb640 1 -- 172.21.15.94:0/2885193012 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a0030840 con 0x7f43b00ac260 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43adffb640 1 -- 172.21.15.94:0/2885193012 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0030c10 con 0x7f43b00ac260 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 -- 172.21.15.94:0/2885193012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b00ac260 msgr2=0x7f43b01559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 --2- 172.21.15.94:0/2885193012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b00ac260 0x7f43b01559e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009870 tx=0x7f43a002b980 comp rx=0 tx=0).stop 2024-03-06T07:21:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 -- 172.21.15.94:0/2885193012 shutdown_connections 2024-03-06T07:21:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 --2- 172.21.15.94:0/2885193012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b00ac260 0x7f43b01559e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 --2- 172.21.15.94:0/2885193012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b0152960 0x7f43b006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 -- 172.21.15.94:0/2885193012 >> 172.21.15.94:0/2885193012 conn(0x7f43b00c70b0 msgr2=0x7f43b00c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 -- 172.21.15.94:0/2885193012 shutdown_connections 2024-03-06T07:21:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.250+0000 7f43b5781640 1 -- 172.21.15.94:0/2885193012 wait complete. 2024-03-06T07:21:26.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.251+0000 7f43b5781640 1 Processor -- start 2024-03-06T07:21:26.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.251+0000 7f43b5781640 1 -- start start 2024-03-06T07:21:26.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43b5781640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b00ac260 0x7f43b01ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43b5781640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b0152960 0x7f43b0151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43b5781640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b006bd00 0x7f43b01435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43b5781640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00c20a0 con 0x7f43b006bd00 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43b5781640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0159760 con 0x7f43b0152960 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43b5781640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00c2220 con 0x7f43b00ac260 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b006bd00 0x7f43b01435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b006bd00 0x7f43b01435b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52014/0 (socket says 172.21.15.94:52014) 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43af7fe640 1 -- 172.21.15.94:0/3596530326 learned_addr learned my addr 172.21.15.94:0/3596530326 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43aeffd640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b00ac260 0x7f43b01ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43ae7fc640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b0152960 0x7f43b0151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.252+0000 7f43af7fe640 1 -- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b00ac260 msgr2=0x7f43b01ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43af7fe640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b00ac260 0x7f43b01ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43af7fe640 1 -- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b0152960 msgr2=0x7f43b0151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43af7fe640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b0152960 0x7f43b0151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43af7fe640 1 -- 172.21.15.94:0/3596530326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b01eeab0 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43af7fe640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b006bd00 0x7f43b01435b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009870 tx=0x7f43a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43a77fe640 1 -- 172.21.15.94:0/3596530326 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0034040 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43a77fe640 1 -- 172.21.15.94:0/3596530326 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a0039d90 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43a77fe640 1 -- 172.21.15.94:0/3596530326 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0043610 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b01eed90 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.253+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b01dfae0 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.254+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b004f330 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.254+0000 7f43a77fe640 1 -- 172.21.15.94:0/3596530326 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43a0037070 con 0x7f43b006bd00 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.255+0000 7f43a77fe640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439006b4d0 0x7f439006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.255+0000 7f43a77fe640 1 -- 172.21.15.94:0/3596530326 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f43a00ac3a0 con 0x7f43b006bd00 2024-03-06T07:21:26.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.255+0000 7f43aeffd640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439006b4d0 0x7f439006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.256+0000 7f43aeffd640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439006b4d0 0x7f439006d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f43a8003b80 tx=0x7f43a80098d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.259+0000 7f43a77fe640 1 -- 172.21.15.94:0/3596530326 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43a003e070 con 0x7f43b006bd00 2024-03-06T07:21:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.478+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43b0156270 con 0x7f43b006bd00 2024-03-06T07:21:26.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.478+0000 7f43a77fe640 1 -- 172.21.15.94:0/3596530326 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f43a007a9d0 con 0x7f43b006bd00 2024-03-06T07:21:26.481 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:26.481 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767091656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767091656},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767091656}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":35795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3407932030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3407932030},{"type":"v1","addr":"172.21.15.145:6837","nonce":3407932030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:25.198745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":14,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":6}]} 2024-03-06T07:21:26.483 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 71 2024-03-06T07:21:26.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439006b4d0 msgr2=0x7f439006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439006b4d0 0x7f439006d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f43a8003b80 tx=0x7f43a80098d0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b006bd00 msgr2=0x7f43b01435b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b006bd00 0x7f43b01435b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009870 tx=0x7f43a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 shutdown_connections 2024-03-06T07:21:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439006b4d0 0x7f439006d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43b006bd00 0x7f43b01435b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b0152960 0x7f43b0151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 --2- 172.21.15.94:0/3596530326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b00ac260 0x7f43b01ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 >> 172.21.15.94:0/3596530326 conn(0x7f43b00c70b0 msgr2=0x7f43b00c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 shutdown_connections 2024-03-06T07:21:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.480+0000 7f43b5781640 1 -- 172.21.15.94:0/3596530326 wait complete. 2024-03-06T07:21:26.493 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.593+0000 7f7108db5640 1 Processor -- start 2024-03-06T07:21:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.593+0000 7f7108db5640 1 -- start start 2024-03-06T07:21:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f7108db5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f710414f360 0x7f7104155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f7108db5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7104158b60 0x7f7104158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f7108db5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71040c73d0 con 0x7f71040c88c0 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f7108db5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71040c8dd0 con 0x7f7104158b60 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f7108db5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71040be2e0 con 0x7f710414f360 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f7102d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7104158b60 0x7f7104158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f70fbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f71040c88c0 0x7f71040c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37724/0 (socket says 172.21.15.94:37724) 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f7102d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7104158b60 0x7f7104158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52024/0 (socket says 172.21.15.94:52024) 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.594+0000 7f70fbfff640 1 -- 172.21.15.94:0/1171046344 learned_addr learned my addr 172.21.15.94:0/1171046344 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7102d9d640 1 -- 172.21.15.94:0/1171046344 >> v2:172.21.15.145:3300/0 conn(0x7f710414f360 msgr2=0x7f7104155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7102d9d640 1 --2- 172.21.15.94:0/1171046344 >> v2:172.21.15.145:3300/0 conn(0x7f710414f360 0x7f7104155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7102d9d640 1 -- 172.21.15.94:0/1171046344 >> v1:172.21.15.94:6789/0 conn(0x7f71040c88c0 legacy=0x7f71040c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7102d9d640 1 -- 172.21.15.94:0/1171046344 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040c7670 con 0x7f7104158b60 2024-03-06T07:21:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7101d9b640 1 -- 172.21.15.94:0/1171046344 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037974384 0 0) 0x7f71040c73d0 con 0x7f71040c88c0 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7102d9d640 1 --2- 172.21.15.94:0/1171046344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7104158b60 0x7f7104158ff0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f710408c840 tx=0x7f70fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c003f069853dfef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7101d9b640 1 -- 172.21.15.94:0/1171046344 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc039070 con 0x7f7104158b60 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7101d9b640 1 -- 172.21.15.94:0/1171046344 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc02cbf0 con 0x7f7104158b60 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.595+0000 7f7101d9b640 1 -- 172.21.15.94:0/1171046344 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc034440 con 0x7f7104158b60 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 -- 172.21.15.94:0/1171046344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7104158b60 msgr2=0x7f7104158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 --2- 172.21.15.94:0/1171046344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7104158b60 0x7f7104158ff0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f710408c840 tx=0x7f70fc02be40 comp rx=0 tx=0).stop 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 -- 172.21.15.94:0/1171046344 shutdown_connections 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 --2- 172.21.15.94:0/1171046344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7104158b60 0x7f7104158ff0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 --2- 172.21.15.94:0/1171046344 >> v2:172.21.15.145:3300/0 conn(0x7f710414f360 0x7f7104155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 -- 172.21.15.94:0/1171046344 >> 172.21.15.94:0/1171046344 conn(0x7f71040ac570 msgr2=0x7f71040ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 -- 172.21.15.94:0/1171046344 shutdown_connections 2024-03-06T07:21:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.596+0000 7f7108db5640 1 -- 172.21.15.94:0/1171046344 wait complete. 2024-03-06T07:21:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.598+0000 7f7108db5640 1 Processor -- start 2024-03-06T07:21:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.598+0000 7f7108db5640 1 -- start start 2024-03-06T07:21:26.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f7108db5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71040c88c0 0x7f71041ee310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f7108db5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f710414f360 0x7f71040add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f7108db5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7104158b60 0x7f71040ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f7108db5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71040c5a50 con 0x7f71040c88c0 2024-03-06T07:21:26.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f7108db5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71040c7c30 con 0x7f710414f360 2024-03-06T07:21:26.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f7108db5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71040c5bd0 con 0x7f7104158b60 2024-03-06T07:21:26.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71040c88c0 0x7f71041ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71040c88c0 0x7f71041ee310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52040/0 (socket says 172.21.15.94:52040) 2024-03-06T07:21:26.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 -- 172.21.15.94:0/2760395047 learned_addr learned my addr 172.21.15.94:0/2760395047 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:26.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f70fbfff640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f710414f360 0x7f71040add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f7102d9d640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7104158b60 0x7f71040ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 -- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7104158b60 msgr2=0x7f71040ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7104158b60 0x7f71040ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 -- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f710414f360 msgr2=0x7f71040add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f710414f360 0x7f71040add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f710259c640 1 -- 172.21.15.94:0/2760395047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040ae7b0 con 0x7f71040c88c0 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.599+0000 7f70fbfff640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f710414f360 0x7f71040add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.600+0000 7f710259c640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71040c88c0 0x7f71041ee310 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f70f400b4a0 tx=0x7f70f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.600+0000 7f70faffd640 1 -- 172.21.15.94:0/2760395047 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4014070 con 0x7f71040c88c0 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.600+0000 7f70faffd640 1 -- 172.21.15.94:0/2760395047 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f4010610 con 0x7f71040c88c0 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.600+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71040af580 con 0x7f71040c88c0 2024-03-06T07:21:26.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.600+0000 7f70faffd640 1 -- 172.21.15.94:0/2760395047 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4010a00 con 0x7f71040c88c0 2024-03-06T07:21:26.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.600+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71040afe50 con 0x7f71040c88c0 2024-03-06T07:21:26.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.601+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710414eec0 con 0x7f71040c88c0 2024-03-06T07:21:26.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.601+0000 7f70faffd640 1 -- 172.21.15.94:0/2760395047 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70f4017070 con 0x7f71040c88c0 2024-03-06T07:21:26.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.602+0000 7f70faffd640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e406b4d0 0x7f70e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.602+0000 7f70faffd640 1 -- 172.21.15.94:0/2760395047 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f70f4089e20 con 0x7f71040c88c0 2024-03-06T07:21:26.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.602+0000 7f70fbfff640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e406b4d0 0x7f70e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.603+0000 7f70fbfff640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e406b4d0 0x7f70e406d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f71040be1a0 tx=0x7f70ec007cc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.606+0000 7f70faffd640 1 -- 172.21.15.94:0/2760395047 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f710414eec0 con 0x7f71040c88c0 2024-03-06T07:21:26.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.820+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f710404f330 con 0x7f71040c88c0 2024-03-06T07:21:26.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.821+0000 7f70faffd640 1 -- 172.21.15.94:0/2760395047 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4987 (secure 0 0 0) 0x7f70f4058aa0 con 0x7f71040c88c0 2024-03-06T07:21:26.823 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:26.823 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767091656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767091656},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767091656}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":35795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3407932030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3407932030},{"type":"v1","addr":"172.21.15.145:6837","nonce":3407932030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:01.766278+0000","modified":"2024-03-06T07:21:25.198745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6318":{"gid":6318,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":14,"addr":"172.21.15.94:6834/3391508984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3391508984},{"type":"v1","addr":"172.21.15.94:6834","nonce":3391508984}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6318,"qdb_cluster":[6318]},"id":6}]} 2024-03-06T07:21:26.823 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 71 2024-03-06T07:21:26.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e406b4d0 msgr2=0x7f70e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e406b4d0 0x7f70e406d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f71040be1a0 tx=0x7f70ec007cc0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71040c88c0 msgr2=0x7f71041ee310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71040c88c0 0x7f71041ee310 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f70f400b4a0 tx=0x7f70f400b960 comp rx=0 tx=0).stop 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 shutdown_connections 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70e406b4d0 0x7f70e406d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7104158b60 0x7f71040ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f710414f360 0x7f71040add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 --2- 172.21.15.94:0/2760395047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71040c88c0 0x7f71041ee310 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 >> 172.21.15.94:0/2760395047 conn(0x7f71040ac570 msgr2=0x7f710414da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 shutdown_connections 2024-03-06T07:21:26.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.823+0000 7f7108db5640 1 -- 172.21.15.94:0/2760395047 wait complete. 2024-03-06T07:21:26.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:26.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.936+0000 7fc0e2afb640 1 Processor -- start 2024-03-06T07:21:26.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.936+0000 7fc0e2afb640 1 -- start start 2024-03-06T07:21:26.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.936+0000 7fc0e2afb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc1527b0 0x7fc0dc159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.936+0000 7fc0e2afb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc0dc0c88b0 0x7fc0dc0c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.936+0000 7fc0e2afb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0dc0c7c60 con 0x7fc0dc0c7180 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.936+0000 7fc0e2afb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0dc159640 con 0x7fc0dc0c88b0 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.936+0000 7fc0e2afb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0c8060 con 0x7fc0dc1527b0 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e0897640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc1527b0 0x7fc0dc159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e1098640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc0dc0c88b0 0x7fc0dc0c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e0897640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc1527b0 0x7fc0dc159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:42586/0 (socket says 172.21.15.94:42586) 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e0897640 1 -- 172.21.15.94:0/3334933486 learned_addr learned my addr 172.21.15.94:0/3334933486 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e0897640 1 -- 172.21.15.94:0/3334933486 >> v2:172.21.15.145:3300/0 conn(0x7fc0dc0c88b0 msgr2=0x7fc0dc0c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e0897640 1 --2- 172.21.15.94:0/3334933486 >> v2:172.21.15.145:3300/0 conn(0x7fc0dc0c88b0 0x7fc0dc0c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e0897640 1 -- 172.21.15.94:0/3334933486 >> v1:172.21.15.145:6789/0 conn(0x7fc0dc0c7180 legacy=0x7fc0dc155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.937+0000 7fc0e0897640 1 -- 172.21.15.94:0/3334933486 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc0c76a0 con 0x7fc0dc1527b0 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e0897640 1 --2- 172.21.15.94:0/3334933486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc1527b0 0x7fc0dc159100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009e50 tx=0x7fc0d402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d999a92687fd0852 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0db7fe640 1 -- 172.21.15.94:0/3334933486 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d402ca70 con 0x7fc0dc1527b0 2024-03-06T07:21:26.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0db7fe640 1 -- 172.21.15.94:0/3334933486 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d402cc10 con 0x7fc0dc1527b0 2024-03-06T07:21:26.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0db7fe640 1 -- 172.21.15.94:0/3334933486 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40343f0 con 0x7fc0dc1527b0 2024-03-06T07:21:26.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 -- 172.21.15.94:0/3334933486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc1527b0 msgr2=0x7fc0dc159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/3334933486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc1527b0 0x7fc0dc159100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009e50 tx=0x7fc0d402bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 -- 172.21.15.94:0/3334933486 shutdown_connections 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/3334933486 >> v2:172.21.15.145:3300/0 conn(0x7fc0dc0c88b0 0x7fc0dc0c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/3334933486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc1527b0 0x7fc0dc159100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 -- 172.21.15.94:0/3334933486 >> 172.21.15.94:0/3334933486 conn(0x7fc0dc0ac560 msgr2=0x7fc0dc0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 -- 172.21.15.94:0/3334933486 shutdown_connections 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.938+0000 7fc0e2afb640 1 -- 172.21.15.94:0/3334933486 wait complete. 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 Processor -- start 2024-03-06T07:21:26.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 -- start start 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0c7180 0x7fc0dc158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0c88b0 0x7fc0dc0cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0dc1527b0 0x7fc0dc1ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc155600 con 0x7fc0dc1527b0 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc155320 con 0x7fc0dc0c88b0 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e2afb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0dc0c8d60 con 0x7fc0dc0c7180 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.940+0000 7fc0e1098640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0dc1527b0 0x7fc0dc1ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0dc1527b0 0x7fc0dc1ee080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52048/0 (socket says 172.21.15.94:52048) 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 -- 172.21.15.94:0/1833855473 learned_addr learned my addr 172.21.15.94:0/1833855473 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e0897640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0c7180 0x7fc0dc158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0dbfff640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0c88b0 0x7fc0dc0cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 -- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0c7180 msgr2=0x7fc0dc158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0c7180 0x7fc0dc158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 -- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0c88b0 msgr2=0x7fc0dc0cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0c88b0 0x7fc0dc0cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 -- 172.21.15.94:0/1833855473 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc0cb270 con 0x7fc0dc1527b0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e1098640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0dc1527b0 0x7fc0dc1ee080 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00e930 tx=0x7fc0cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0d97fa640 1 -- 172.21.15.94:0/1833855473 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc017070 con 0x7fc0dc1527b0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0d97fa640 1 -- 172.21.15.94:0/1833855473 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0cc00c980 con 0x7fc0dc1527b0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0dc149060 con 0x7fc0dc1527b0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0d97fa640 1 -- 172.21.15.94:0/1833855473 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc00cd70 con 0x7fc0dc1527b0 2024-03-06T07:21:26.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.941+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0dc149410 con 0x7fc0dc1527b0 2024-03-06T07:21:26.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.942+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0dc04f330 con 0x7fc0dc1527b0 2024-03-06T07:21:26.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.943+0000 7fc0d97fa640 1 -- 172.21.15.94:0/1833855473 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0cc01a070 con 0x7fc0dc1527b0 2024-03-06T07:21:26.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.944+0000 7fc0d97fa640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c806b4d0 0x7fc0c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.944+0000 7fc0d97fa640 1 -- 172.21.15.94:0/1833855473 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7fc0cc08a780 con 0x7fc0dc1527b0 2024-03-06T07:21:26.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.944+0000 7fc0e0897640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c806b4d0 0x7fc0c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.944+0000 7fc0e0897640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c806b4d0 0x7fc0c806d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4035000 tx=0x7fc0d4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:26.952+0000 7fc0d97fa640 1 -- 172.21.15.94:0/1833855473 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0dc0cb270 con 0x7fc0dc1527b0 2024-03-06T07:21:27.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.131+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0dc06ce00 con 0x7fc0dc1527b0 2024-03-06T07:21:27.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.133+0000 7fc0d97fa640 1 -- 172.21.15.94:0/1833855473 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17284 (secure 0 0 0) 0x7fc0cc0594c0 con 0x7fc0dc1527b0 2024-03-06T07:21:27.135 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:27.135 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":83,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:24.197861+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:20:59.828663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:21:00.744064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6834/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6837/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6836/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6837/3537750458":"2024-03-07T07:21:11.096670+0000","172.21.15.145:6836/3537750458":"2024-03-07T07:21:11.096670+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.134+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c806b4d0 msgr2=0x7fc0c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.134+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c806b4d0 0x7fc0c806d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4035000 tx=0x7fc0d4030000 comp rx=0 tx=0).stop 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.134+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0dc1527b0 msgr2=0x7fc0dc1ee080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.134+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0dc1527b0 0x7fc0dc1ee080 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00e930 tx=0x7fc0cc00edf0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 shutdown_connections 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0c806b4d0 0x7fc0c806d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc0dc1527b0 0x7fc0dc1ee080 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0c88b0 0x7fc0dc0cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 --2- 172.21.15.94:0/1833855473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0c7180 0x7fc0dc158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 >> 172.21.15.94:0/1833855473 conn(0x7fc0dc0ac560 msgr2=0x7fc0dc14ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 shutdown_connections 2024-03-06T07:21:27.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.135+0000 7fc0e2afb640 1 -- 172.21.15.94:0/1833855473 wait complete. 2024-03-06T07:21:27.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-06T07:21:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.263+0000 7f1d042f4640 1 Processor -- start 2024-03-06T07:21:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.263+0000 7f1d042f4640 1 -- start start 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.264+0000 7f1d042f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc0c6ab0 0x7f1cfc1529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.264+0000 7f1d042f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1cfc1561a0 0x7f1cfc0a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.264+0000 7f1d042f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1cfc0aa430 0x7f1cfc14f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.264+0000 7f1d042f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cfc155d10 con 0x7f1cfc0aa430 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d042f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cfc0a9990 con 0x7f1cfc1561a0 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d042f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc1556c0 con 0x7f1cfc0c6ab0 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1cfc0aa430 0x7f1cfc14f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1cfc0aa430 0x7f1cfc14f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52064/0 (socket says 172.21.15.94:52064) 2024-03-06T07:21:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 -- 172.21.15.94:0/3409818258 learned_addr learned my addr 172.21.15.94:0/3409818258 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02090640 1 --2- 172.21.15.94:0/3409818258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc0c6ab0 0x7f1cfc1529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d0188f640 1 --2- 172.21.15.94:0/3409818258 >> v2:172.21.15.145:3300/0 conn(0x7f1cfc1561a0 0x7f1cfc0a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 -- 172.21.15.94:0/3409818258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc0c6ab0 msgr2=0x7f1cfc1529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 --2- 172.21.15.94:0/3409818258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc0c6ab0 0x7f1cfc1529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 -- 172.21.15.94:0/3409818258 >> v2:172.21.15.145:3300/0 conn(0x7f1cfc1561a0 msgr2=0x7f1cfc0a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 --2- 172.21.15.94:0/3409818258 >> v2:172.21.15.145:3300/0 conn(0x7f1cfc1561a0 0x7f1cfc0a9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.265+0000 7f1d02891640 1 -- 172.21.15.94:0/3409818258 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc1559a0 con 0x7f1cfc0aa430 2024-03-06T07:21:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.266+0000 7f1d02891640 1 --2- 172.21.15.94:0/3409818258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc14f110 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc06bf60 tx=0x7f1cf802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa33a9f4c304b5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.266+0000 7f1d0108e640 1 -- 172.21.15.94:0/3409818258 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf802c950 con 0x7f1cfc0aa430 2024-03-06T07:21:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.266+0000 7f1d0108e640 1 -- 172.21.15.94:0/3409818258 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf802caf0 con 0x7f1cfc0aa430 2024-03-06T07:21:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.266+0000 7f1d0108e640 1 -- 172.21.15.94:0/3409818258 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf80343f0 con 0x7f1cfc0aa430 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.266+0000 7f1d042f4640 1 -- 172.21.15.94:0/3409818258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 msgr2=0x7f1cfc14f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.266+0000 7f1d042f4640 1 --2- 172.21.15.94:0/3409818258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc14f110 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc06bf60 tx=0x7f1cf802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.267+0000 7f1d042f4640 1 -- 172.21.15.94:0/3409818258 shutdown_connections 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.267+0000 7f1d042f4640 1 --2- 172.21.15.94:0/3409818258 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc14f110 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.267+0000 7f1d042f4640 1 --2- 172.21.15.94:0/3409818258 >> v2:172.21.15.145:3300/0 conn(0x7f1cfc1561a0 0x7f1cfc0a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.267+0000 7f1d042f4640 1 --2- 172.21.15.94:0/3409818258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc0c6ab0 0x7f1cfc1529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.267+0000 7f1d042f4640 1 -- 172.21.15.94:0/3409818258 >> 172.21.15.94:0/3409818258 conn(0x7f1cfc06b1a0 msgr2=0x7f1cfc06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.267+0000 7f1d042f4640 1 -- 172.21.15.94:0/3409818258 shutdown_connections 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.267+0000 7f1d042f4640 1 -- 172.21.15.94:0/3409818258 wait complete. 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 Processor -- start 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 -- start start 2024-03-06T07:21:27.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc071cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cfc0c6ab0 0x7f1cfc14bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc1561a0 0x7f1cfc14c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc0c2c10 con 0x7f1cfc0aa430 2024-03-06T07:21:27.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc0c2a90 con 0x7f1cfc0c6ab0 2024-03-06T07:21:27.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d042f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc0c6e90 con 0x7f1cfc1561a0 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d02090640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc071cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d02090640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc071cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52066/0 (socket says 172.21.15.94:52066) 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d02090640 1 -- 172.21.15.94:0/897120666 learned_addr learned my addr 172.21.15.94:0/897120666 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d0188f640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cfc0c6ab0 0x7f1cfc14bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.268+0000 7f1d02891640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc1561a0 0x7f1cfc14c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d02090640 1 -- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc1561a0 msgr2=0x7f1cfc14c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d02090640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc1561a0 0x7f1cfc14c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d02090640 1 -- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cfc0c6ab0 msgr2=0x7f1cfc14bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d02090640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cfc0c6ab0 0x7f1cfc14bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d02090640 1 -- 172.21.15.94:0/897120666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc0c6320 con 0x7f1cfc0aa430 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d02090640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc071cb0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400e870 tx=0x7f1cf400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1cf2ffd640 1 -- 172.21.15.94:0/897120666 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4017070 con 0x7f1cfc0aa430 2024-03-06T07:21:27.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1cf2ffd640 1 -- 172.21.15.94:0/897120666 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf400ee60 con 0x7f1cfc0aa430 2024-03-06T07:21:27.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cfc0c4b00 con 0x7f1cfc0aa430 2024-03-06T07:21:27.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1cf2ffd640 1 -- 172.21.15.94:0/897120666 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf400cb80 con 0x7f1cfc0aa430 2024-03-06T07:21:27.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.269+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cfc0c50c0 con 0x7f1cfc0aa430 2024-03-06T07:21:27.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.270+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cfc04f330 con 0x7f1cfc0aa430 2024-03-06T07:21:27.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.271+0000 7f1cf2ffd640 1 -- 172.21.15.94:0/897120666 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cf401a070 con 0x7f1cfc0aa430 2024-03-06T07:21:27.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.271+0000 7f1cf2ffd640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ce006b4d0 0x7f1ce006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.271+0000 7f1cf2ffd640 1 -- 172.21.15.94:0/897120666 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f1cf408a240 con 0x7f1cfc0aa430 2024-03-06T07:21:27.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.271+0000 7f1d0188f640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ce006b4d0 0x7f1ce006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.272+0000 7f1d0188f640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ce006b4d0 0x7f1ce006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc06e950 tx=0x7f1cec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.275+0000 7f1cf2ffd640 1 -- 172.21.15.94:0/897120666 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1cf40587e0 con 0x7f1cfc0aa430 2024-03-06T07:21:27.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.485+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7f1cfc0c26b0 con 0x7f1cfc0aa430 2024-03-06T07:21:27.487 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:21:27.485+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:21:27.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.486+0000 7f1cf2ffd640 1 -- 172.21.15.94:0/897120666 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v72) v1 ==== 163+0+0 (secure 0 0 0) 0x7f1cf405c880 con 0x7f1cfc0aa430 2024-03-06T07:21:27.488 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:21:27.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ce006b4d0 msgr2=0x7f1ce006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ce006b4d0 0x7f1ce006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc06e950 tx=0x7f1cec009190 comp rx=0 tx=0).stop 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 msgr2=0x7f1cfc071cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc071cb0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400e870 tx=0x7f1cf400ed30 comp rx=0 tx=0).stop 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 shutdown_connections 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ce006b4d0 0x7f1ce006d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cfc1561a0 0x7f1cfc14c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cfc0c6ab0 0x7f1cfc14bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 --2- 172.21.15.94:0/897120666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cfc0aa430 0x7f1cfc071cb0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.489+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 >> 172.21.15.94:0/897120666 conn(0x7f1cfc06b1a0 msgr2=0x7f1cfc0a9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.490+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 shutdown_connections 2024-03-06T07:21:27.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.490+0000 7f1d042f4640 1 -- 172.21.15.94:0/897120666 wait complete. 2024-03-06T07:21:27.493 INFO:tasks.ceph.mds.a.smithi094.stderr: -64> 2024-03-06T07:21:26.139+0000 7f57b22e2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:27.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-06T07:21:27.516 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:21:27.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.603+0000 7f42ee2af640 1 Processor -- start 2024-03-06T07:21:27.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.603+0000 7f42ee2af640 1 -- start start 2024-03-06T07:21:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42ee2af640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 0x7f42e80c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42ee2af640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e80c3ef0 con 0x7f42e80ab170 2024-03-06T07:21:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42ee2af640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e81556d0 con 0x7f42e80c5990 2024-03-06T07:21:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42ee2af640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e81559b0 con 0x7f42e806c900 2024-03-06T07:21:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42ec84c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f42e80ab170 0x7f42e80a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37748/0 (socket says 172.21.15.94:37748) 2024-03-06T07:21:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42ec84c640 1 -- 172.21.15.94:0/3463087484 learned_addr learned my addr 172.21.15.94:0/3463087484 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146242358 0 0) 0x7f42e80c3ef0 con 0x7f42e80ab170 2024-03-06T07:21:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d0003520 con 0x7f42e80ab170 2024-03-06T07:21:27.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e77fe640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 0x7f42e80c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3240644206 0 0) 0x7f42d0003520 con 0x7f42e80ab170 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 msgr2=0x7f42e80c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 0x7f42e80c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 >> v1:172.21.15.145:6789/0 conn(0x7f42e80c5990 legacy=0x7f42e806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e8155c90 con 0x7f42e80ab170 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.604+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f42e00019f0 con 0x7f42e80ab170 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 >> v1:172.21.15.94:6789/0 conn(0x7f42e80ab170 legacy=0x7f42e80a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e6ffd640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 0x7f42d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e6ffd640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e6ffd640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d0004d40 0x7f42d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e81559b0 con 0x7f42d0004010 2024-03-06T07:21:27.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d000bb80 con 0x7f42d0004d40 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d000bd80 con 0x7f42e806c900 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e77fe640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e7fff640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 0x7f42d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42ec84c640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d0004d40 0x7f42d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e77fe640 1 -- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 msgr2=0x7f42d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e77fe640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e806c900 0x7f42d00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e77fe640 1 -- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d0004d40 msgr2=0x7f42d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e77fe640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d0004d40 0x7f42d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42e77fe640 1 -- 172.21.15.94:0/3463087484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0003520 con 0x7f42d0004010 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.605+0000 7f42ec84c640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d0004d40 0x7f42d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42e77fe640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42d0004800 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00ba00 tx=0x7f42d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69bec56f30694d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc036020 con 0x7f42d0004010 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc032620 con 0x7f42d0004010 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42e6ffd640 1 -- 172.21.15.94:0/3463087484 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc00b600 con 0x7f42d0004010 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42ee2af640 1 -- 172.21.15.94:0/3463087484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 msgr2=0x7f42d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42ee2af640 1 --2- 172.21.15.94:0/3463087484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42d0004800 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00ba00 tx=0x7f42d0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42ec84c640 1 -- 172.21.15.94:0/3463087484 reap_dead start 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42ee2af640 1 -- 172.21.15.94:0/3463087484 shutdown_connections 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42ee2af640 1 -- 172.21.15.94:0/3463087484 >> 172.21.15.94:0/3463087484 conn(0x7f42e80c6710 msgr2=0x7f42e80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42ee2af640 1 -- 172.21.15.94:0/3463087484 shutdown_connections 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.606+0000 7f42ee2af640 1 -- 172.21.15.94:0/3463087484 wait complete. 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.607+0000 7f42ee2af640 1 Processor -- start 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42ee2af640 1 -- start start 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42ee2af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42e81ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42ee2af640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42e8153540 0x7f42e80a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42ee2af640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e80c0030 0x7f42e813f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42ee2af640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8156610 con 0x7f42d0004010 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42ee2af640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8156490 con 0x7f42e8153540 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42ee2af640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80c3af0 con 0x7f42e80c0030 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42e81ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42e7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42e81ee0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52106/0 (socket says 172.21.15.94:52106) 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.608+0000 7f42e7fff640 1 -- 172.21.15.94:0/81538060 learned_addr learned my addr 172.21.15.94:0/81538060 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e77fe640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42e8153540 0x7f42e80a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42ec84c640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e80c0030 0x7f42e813f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e7fff640 1 -- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e80c0030 msgr2=0x7f42e813f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e7fff640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e80c0030 0x7f42e813f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e7fff640 1 -- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42e8153540 msgr2=0x7f42e80a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e7fff640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42e8153540 0x7f42e80a7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e7fff640 1 -- 172.21.15.94:0/81538060 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e8151260 con 0x7f42d0004010 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e77fe640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42e8153540 0x7f42e80a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e7fff640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42e81ee0c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f42d800f880 tx=0x7f42d800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e4ff9640 1 -- 172.21.15.94:0/81538060 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8001d20 con 0x7f42d0004010 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e4ff9640 1 -- 172.21.15.94:0/81538060 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d8018070 con 0x7f42d0004010 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e81ee8f0 con 0x7f42d0004010 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42e4ff9640 1 -- 172.21.15.94:0/81538060 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d800d730 con 0x7f42d0004010 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.609+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e81eeeb0 con 0x7f42d0004010 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.610+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e804f330 con 0x7f42d0004010 2024-03-06T07:21:27.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.611+0000 7f42e4ff9640 1 -- 172.21.15.94:0/81538060 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42d800d8d0 con 0x7f42d0004010 2024-03-06T07:21:27.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.611+0000 7f42e4ff9640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42d406b500 0x7f42d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.611+0000 7f42e4ff9640 1 -- 172.21.15.94:0/81538060 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f42d808a360 con 0x7f42d0004010 2024-03-06T07:21:27.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.611+0000 7f42e77fe640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42d406b500 0x7f42d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.612+0000 7f42e77fe640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42d406b500 0x7f42d406d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f42e806cd40 tx=0x7f42dc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.615+0000 7f42e4ff9640 1 -- 172.21.15.94:0/81538060 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42d8058ef0 con 0x7f42d0004010 2024-03-06T07:21:27.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:27.826+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f42e80c6b80 con 0x7f42d0004010 2024-03-06T07:21:28.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.488+0000 7f42e4ff9640 1 -- 172.21.15.94:0/81538060 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v73) v1 ==== 105+0+0 (secure 0 0 0) 0x7f42d80587f0 con 0x7f42d0004010 2024-03-06T07:21:28.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42d406b500 msgr2=0x7f42d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42d406b500 0x7f42d406d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f42e806cd40 tx=0x7f42dc030000 comp rx=0 tx=0).stop 2024-03-06T07:21:28.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 msgr2=0x7f42e81ee0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42e81ee0c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f42d800f880 tx=0x7f42d800fd40 comp rx=0 tx=0).stop 2024-03-06T07:21:28.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 shutdown_connections 2024-03-06T07:21:28.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42d406b500 0x7f42d406d9c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42e80c0030 0x7f42e813f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42e8153540 0x7f42e80a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 --2- 172.21.15.94:0/81538060 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d0004010 0x7f42e81ee0c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 >> 172.21.15.94:0/81538060 conn(0x7f42e80c6710 msgr2=0x7f42e806cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:28.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 shutdown_connections 2024-03-06T07:21:28.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.491+0000 7f42ee2af640 1 -- 172.21.15.94:0/81538060 wait complete. 2024-03-06T07:21:28.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-06T07:21:28.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.656+0000 7f1e1ab3e640 1 Processor -- start 2024-03-06T07:21:28.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.656+0000 7f1e1ab3e640 1 -- start start 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e1ab3e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1e140c22b0 0x7f1e14151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e1ab3e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1e140a9830 0x7f1e1414ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e1ab3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1414f490 0x7f1e14153390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e1ab3e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1406e850 con 0x7f1e140c22b0 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e1ab3e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1406d2b0 con 0x7f1e140a9830 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e1ab3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1406d590 con 0x7f1e1414f490 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1e140c22b0 0x7f1e14151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1e140c22b0 0x7f1e14151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52208/0 (socket says 172.21.15.94:52208) 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 -- 172.21.15.94:0/10471129 learned_addr learned my addr 172.21.15.94:0/10471129 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e190db640 1 --2- 172.21.15.94:0/10471129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1414f490 0x7f1e14153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e13fff640 1 --2- 172.21.15.94:0/10471129 >> v2:172.21.15.145:3300/0 conn(0x7f1e140a9830 0x7f1e1414ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 -- 172.21.15.94:0/10471129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1414f490 msgr2=0x7f1e14153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 --2- 172.21.15.94:0/10471129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1414f490 0x7f1e14153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 -- 172.21.15.94:0/10471129 >> v2:172.21.15.145:3300/0 conn(0x7f1e140a9830 msgr2=0x7f1e1414ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 --2- 172.21.15.94:0/10471129 >> v2:172.21.15.145:3300/0 conn(0x7f1e140a9830 0x7f1e1414ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.657+0000 7f1e188da640 1 -- 172.21.15.94:0/10471129 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1406d870 con 0x7f1e140c22b0 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.658+0000 7f1e188da640 1 --2- 172.21.15.94:0/10471129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140c22b0 0x7f1e14151e10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1e04009870 tx=0x7f1e0402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a279530a0c92d2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.658+0000 7f1e137fe640 1 -- 172.21.15.94:0/10471129 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04035020 con 0x7f1e140c22b0 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.658+0000 7f1e137fe640 1 -- 172.21.15.94:0/10471129 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e04031640 con 0x7f1e140c22b0 2024-03-06T07:21:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.658+0000 7f1e137fe640 1 -- 172.21.15.94:0/10471129 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0402d040 con 0x7f1e140c22b0 2024-03-06T07:21:28.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/10471129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140c22b0 msgr2=0x7f1e14151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/10471129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140c22b0 0x7f1e14151e10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1e04009870 tx=0x7f1e0402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/10471129 shutdown_connections 2024-03-06T07:21:28.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/10471129 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1414f490 0x7f1e14153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/10471129 >> v2:172.21.15.145:3300/0 conn(0x7f1e140a9830 0x7f1e1414ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/10471129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140c22b0 0x7f1e14151e10 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/10471129 >> 172.21.15.94:0/10471129 conn(0x7f1e140c65b0 msgr2=0x7f1e140c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/10471129 shutdown_connections 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.659+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/10471129 wait complete. 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.660+0000 7f1e1ab3e640 1 Processor -- start 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.660+0000 7f1e1ab3e640 1 -- start start 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e1ab3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140a9830 0x7f1e141df8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e1ab3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e140c22b0 0x7f1e141ea690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e1ab3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e1414f490 0x7f1e141eabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e1ab3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140c2810 con 0x7f1e140a9830 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e1ab3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140c2690 con 0x7f1e1414f490 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e1ab3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e141538d0 con 0x7f1e140c22b0 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140a9830 0x7f1e141df8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140a9830 0x7f1e141df8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52218/0 (socket says 172.21.15.94:52218) 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 -- 172.21.15.94:0/527717337 learned_addr learned my addr 172.21.15.94:0/527717337 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e13fff640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e140c22b0 0x7f1e141ea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e190db640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e1414f490 0x7f1e141eabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 -- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e140c22b0 msgr2=0x7f1e141ea690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e140c22b0 0x7f1e141ea690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 -- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e1414f490 msgr2=0x7f1e141eabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e1414f490 0x7f1e141eabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 -- 172.21.15.94:0/527717337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1414e710 con 0x7f1e140a9830 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e188da640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140a9830 0x7f1e141df8a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1e04000c00 tx=0x7f1e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.661+0000 7f1e117fa640 1 -- 172.21.15.94:0/527717337 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04040070 con 0x7f1e140a9830 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.662+0000 7f1e117fa640 1 -- 172.21.15.94:0/527717337 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e040399f0 con 0x7f1e140a9830 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.662+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e141edc00 con 0x7f1e140a9830 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.662+0000 7f1e117fa640 1 -- 172.21.15.94:0/527717337 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04035040 con 0x7f1e140a9830 2024-03-06T07:21:28.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.662+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e141ee1c0 con 0x7f1e140a9830 2024-03-06T07:21:28.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.663+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1404f330 con 0x7f1e140a9830 2024-03-06T07:21:28.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.663+0000 7f1e117fa640 1 -- 172.21.15.94:0/527717337 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e0404b060 con 0x7f1e140a9830 2024-03-06T07:21:28.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.663+0000 7f1e117fa640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1e0006b4d0 0x7f1e0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.663+0000 7f1e117fa640 1 -- 172.21.15.94:0/527717337 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f1e040abd40 con 0x7f1e140a9830 2024-03-06T07:21:28.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.664+0000 7f1e13fff640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1e0006b4d0 0x7f1e0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.664+0000 7f1e13fff640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1e0006b4d0 0x7f1e0006d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1e141eb7c0 tx=0x7f1e08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:28.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.668+0000 7f1e117fa640 1 -- 172.21.15.94:0/527717337 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e1414e710 con 0x7f1e140a9830 2024-03-06T07:21:28.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:28.838+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1e14074700 con 0x7f1e140a9830 2024-03-06T07:21:29.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.497+0000 7f1e117fa640 1 -- 172.21.15.94:0/527717337 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v85) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1e0407a960 con 0x7f1e140a9830 2024-03-06T07:21:29.499 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_metadata' removed 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1e0006b4d0 msgr2=0x7f1e0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1e0006b4d0 0x7f1e0006d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1e141eb7c0 tx=0x7f1e08008040 comp rx=0 tx=0).stop 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140a9830 msgr2=0x7f1e141df8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140a9830 0x7f1e141df8a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f1e04000c00 tx=0x7f1e040046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 shutdown_connections 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1e0006b4d0 0x7f1e0006d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e1414f490 0x7f1e141eabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e140c22b0 0x7f1e141ea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 --2- 172.21.15.94:0/527717337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1e140a9830 0x7f1e141df8a0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 >> 172.21.15.94:0/527717337 conn(0x7f1e140c65b0 msgr2=0x7f1e140c2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 shutdown_connections 2024-03-06T07:21:29.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.500+0000 7f1e1ab3e640 1 -- 172.21.15.94:0/527717337 wait complete. 2024-03-06T07:21:29.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-06T07:21:29.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.621+0000 7f23c33a0640 1 Processor -- start 2024-03-06T07:21:29.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.622+0000 7f23c33a0640 1 -- start start 2024-03-06T07:21:29.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c33a0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23bc156b20 0x7f23bc0a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c33a0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f23bc15c940 0x7f23bc15cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c33a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23bc0ab8f0 con 0x7f23bc0c0e00 2024-03-06T07:21:29.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c33a0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc0c26c0 con 0x7f23bc156b20 2024-03-06T07:21:29.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c33a0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc0c5e70 con 0x7f23bc15c940 2024-03-06T07:21:29.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c239e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23bc156b20 0x7f23bc0a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c239e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23bc156b20 0x7f23bc0a94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52234/0 (socket says 172.21.15.94:52234) 2024-03-06T07:21:29.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c239e640 1 -- 172.21.15.94:0/517055275 learned_addr learned my addr 172.21.15.94:0/517055275 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:29.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.623+0000 7f23c2b9f640 1 --2- 172.21.15.94:0/517055275 >> v2:172.21.15.145:3300/0 conn(0x7f23bc15c940 0x7f23bc15cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.624+0000 7f23c239e640 1 -- 172.21.15.94:0/517055275 >> v2:172.21.15.145:3300/0 conn(0x7f23bc15c940 msgr2=0x7f23bc15cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:29.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.624+0000 7f23c239e640 1 --2- 172.21.15.94:0/517055275 >> v2:172.21.15.145:3300/0 conn(0x7f23bc15c940 0x7f23bc15cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.624+0000 7f23c239e640 1 -- 172.21.15.94:0/517055275 >> v1:172.21.15.94:6789/0 conn(0x7f23bc0c0e00 legacy=0x7f23bc06d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:29.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.624+0000 7f23c239e640 1 -- 172.21.15.94:0/517055275 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0c5540 con 0x7f23bc156b20 2024-03-06T07:21:29.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.625+0000 7f23c239e640 1 --2- 172.21.15.94:0/517055275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc156b20 0x7f23bc0a94b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0aaa60 tx=0x7f23b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656d9da7695db78e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.625+0000 7f23c139c640 1 -- 172.21.15.94:0/517055275 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b802e070 con 0x7f23bc156b20 2024-03-06T07:21:29.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.625+0000 7f23c139c640 1 -- 172.21.15.94:0/517055275 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b8037070 con 0x7f23bc156b20 2024-03-06T07:21:29.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.625+0000 7f23c139c640 1 -- 172.21.15.94:0/517055275 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8031a00 con 0x7f23bc156b20 2024-03-06T07:21:29.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 -- 172.21.15.94:0/517055275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc156b20 msgr2=0x7f23bc0a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 --2- 172.21.15.94:0/517055275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc156b20 0x7f23bc0a94b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0aaa60 tx=0x7f23b802b880 comp rx=0 tx=0).stop 2024-03-06T07:21:29.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 -- 172.21.15.94:0/517055275 shutdown_connections 2024-03-06T07:21:29.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 --2- 172.21.15.94:0/517055275 >> v2:172.21.15.145:3300/0 conn(0x7f23bc15c940 0x7f23bc15cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 --2- 172.21.15.94:0/517055275 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc156b20 0x7f23bc0a94b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 -- 172.21.15.94:0/517055275 >> 172.21.15.94:0/517055275 conn(0x7f23bc0c2f00 msgr2=0x7f23bc0c32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 -- 172.21.15.94:0/517055275 shutdown_connections 2024-03-06T07:21:29.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.626+0000 7f23c33a0640 1 -- 172.21.15.94:0/517055275 wait complete. 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.627+0000 7f23c33a0640 1 Processor -- start 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.627+0000 7f23c33a0640 1 -- start start 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c33a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc0c0e00 0x7f23bc071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c33a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23bc156b20 0x7f23bc1f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c33a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23bc15c940 0x7f23bc1f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c33a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0c2910 con 0x7f23bc0c0e00 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c33a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0c5b00 con 0x7f23bc15c940 2024-03-06T07:21:29.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c33a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc06d710 con 0x7f23bc156b20 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc0c0e00 0x7f23bc071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc0c0e00 0x7f23bc071dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52244/0 (socket says 172.21.15.94:52244) 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 -- 172.21.15.94:0/172962633 learned_addr learned my addr 172.21.15.94:0/172962633 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c1b9d640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23bc156b20 0x7f23bc1f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c2b9f640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23bc15c940 0x7f23bc1f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 -- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23bc156b20 msgr2=0x7f23bc1f0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23bc156b20 0x7f23bc1f0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 -- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23bc15c940 msgr2=0x7f23bc1f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23bc15c940 0x7f23bc1f0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 -- 172.21.15.94:0/172962633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc1f2b90 con 0x7f23bc0c0e00 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c1b9d640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23bc156b20 0x7f23bc1f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c239e640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc0c0e00 0x7f23bc071dc0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f23b802b8c0 tx=0x7f23b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23b2ffd640 1 -- 172.21.15.94:0/172962633 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8032670 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23b2ffd640 1 -- 172.21.15.94:0/172962633 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b8032850 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23b2ffd640 1 -- 172.21.15.94:0/172962633 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8032c40 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.628+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc1f2e70 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.629+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc1f04e0 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.630+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc0693c0 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.630+0000 7f23b2ffd640 1 -- 172.21.15.94:0/172962633 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23b804c060 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.630+0000 7f23b2ffd640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23a006b4d0 0x7f23a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.630+0000 7f23b2ffd640 1 -- 172.21.15.94:0/172962633 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6009+0+0 (secure 0 0 0) 0x7f23b80abc20 con 0x7f23bc0c0e00 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.630+0000 7f23c1b9d640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23a006b4d0 0x7f23a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.631+0000 7f23c1b9d640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23a006b4d0 0x7f23a006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0c1240 tx=0x7f23ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.634+0000 7f23b2ffd640 1 -- 172.21.15.94:0/172962633 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23b8040070 con 0x7f23bc0c0e00 2024-03-06T07:21:29.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:29.805+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f23bc1f3040 con 0x7f23bc0c0e00 2024-03-06T07:21:30.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.499+0000 7f23b2ffd640 1 -- 172.21.15.94:0/172962633 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v86) v1 ==== 172+0+0 (secure 0 0 0) 0x7f23b807aa00 con 0x7f23bc0c0e00 2024-03-06T07:21:30.501 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_data' removed 2024-03-06T07:21:30.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.502+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23a006b4d0 msgr2=0x7f23a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.502+0000 7f23c33a0640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23a006b4d0 0x7f23a006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0c1240 tx=0x7f23ac008040 comp rx=0 tx=0).stop 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.502+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc0c0e00 msgr2=0x7f23bc071dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.502+0000 7f23c33a0640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc0c0e00 0x7f23bc071dc0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f23b802b8c0 tx=0x7f23b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 shutdown_connections 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23a006b4d0 0x7f23a006d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23bc15c940 0x7f23bc1f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23bc156b20 0x7f23bc1f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 --2- 172.21.15.94:0/172962633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23bc0c0e00 0x7f23bc071dc0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 >> 172.21.15.94:0/172962633 conn(0x7f23bc0c2f00 msgr2=0x7f23bc0aad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 shutdown_connections 2024-03-06T07:21:30.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.503+0000 7f23c33a0640 1 -- 172.21.15.94:0/172962633 wait complete. 2024-03-06T07:21:30.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-03-06T07:21:30.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.624+0000 7fee0820e640 1 Processor -- start 2024-03-06T07:21:30.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.624+0000 7fee0820e640 1 -- start start 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.624+0000 7fee0820e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee0006c900 0x7fee000c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.624+0000 7fee0820e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee000ab170 0x7fee000a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.624+0000 7fee0820e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee000c3ef0 con 0x7fee000c5990 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.624+0000 7fee0820e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee001556d0 con 0x7fee0006c900 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.624+0000 7fee0820e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee001559b0 con 0x7fee000ab170 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee057a9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee0006c900 0x7fee000c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee057a9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fee0006c900 0x7fee000c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52252/0 (socket says 172.21.15.94:52252) 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee057a9640 1 -- 172.21.15.94:0/2272466726 learned_addr learned my addr 172.21.15.94:0/2272466726 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee067ab640 1 --2- 172.21.15.94:0/2272466726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee000ab170 0x7fee000a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee057a9640 1 -- 172.21.15.94:0/2272466726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee000ab170 msgr2=0x7fee000a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee057a9640 1 --2- 172.21.15.94:0/2272466726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee000ab170 0x7fee000a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee057a9640 1 -- 172.21.15.94:0/2272466726 >> v1:172.21.15.145:6789/0 conn(0x7fee000c5990 legacy=0x7fee0006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee057a9640 1 -- 172.21.15.94:0/2272466726 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee00155c90 con 0x7fee0006c900 2024-03-06T07:21:30.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.625+0000 7fee067ab640 1 --2- 172.21.15.94:0/2272466726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee000ab170 0x7fee000a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:30.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee057a9640 1 --2- 172.21.15.94:0/2272466726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee0006c900 0x7fee000c0ab0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bb10 tx=0x7fedfc02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43e885d19dece8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee04fa8640 1 -- 172.21.15.94:0/2272466726 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc03b070 con 0x7fee0006c900 2024-03-06T07:21:30.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee04fa8640 1 -- 172.21.15.94:0/2272466726 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc036440 con 0x7fee0006c900 2024-03-06T07:21:30.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee04fa8640 1 -- 172.21.15.94:0/2272466726 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc036810 con 0x7fee0006c900 2024-03-06T07:21:30.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 -- 172.21.15.94:0/2272466726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee0006c900 msgr2=0x7fee000c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 --2- 172.21.15.94:0/2272466726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee0006c900 0x7fee000c0ab0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bb10 tx=0x7fedfc02e0c0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 -- 172.21.15.94:0/2272466726 shutdown_connections 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 --2- 172.21.15.94:0/2272466726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee000ab170 0x7fee000a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 --2- 172.21.15.94:0/2272466726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee0006c900 0x7fee000c0ab0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 -- 172.21.15.94:0/2272466726 >> 172.21.15.94:0/2272466726 conn(0x7fee000c6710 msgr2=0x7fee000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 -- 172.21.15.94:0/2272466726 shutdown_connections 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.626+0000 7fee0820e640 1 -- 172.21.15.94:0/2272466726 wait complete. 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.627+0000 7fee0820e640 1 Processor -- start 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.627+0000 7fee0820e640 1 -- start start 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee0820e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee0006c900 0x7fee00144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee0820e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee000ab170 0x7fee001e6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee0820e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee000c5990 0x7fee001e71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee0820e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee00156610 con 0x7fee000c5990 2024-03-06T07:21:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee0820e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee00156490 con 0x7fee000ab170 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee0820e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee000c3af0 con 0x7fee0006c900 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee000c5990 0x7fee001e71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee000c5990 0x7fee001e71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52260/0 (socket says 172.21.15.94:52260) 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 -- 172.21.15.94:0/1104694877 learned_addr learned my addr 172.21.15.94:0/1104694877 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee05faa640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee0006c900 0x7fee00144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 -- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee0006c900 msgr2=0x7fee00144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee0006c900 0x7fee00144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee057a9640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee000ab170 0x7fee001e6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 -- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee000ab170 msgr2=0x7fee001e6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee000ab170 0x7fee001e6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.628+0000 7fee067ab640 1 -- 172.21.15.94:0/1104694877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee001e76e0 con 0x7fee000c5990 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.629+0000 7fee067ab640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee000c5990 0x7fee001e71a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fedf800cef0 tx=0x7fedf800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.629+0000 7fedf67fc640 1 -- 172.21.15.94:0/1104694877 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf801a070 con 0x7fee000c5990 2024-03-06T07:21:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.629+0000 7fedf67fc640 1 -- 172.21.15.94:0/1104694877 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf800d860 con 0x7fee000c5990 2024-03-06T07:21:30.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.629+0000 7fedf67fc640 1 -- 172.21.15.94:0/1104694877 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf800dc50 con 0x7fee000c5990 2024-03-06T07:21:30.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.629+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee001eb030 con 0x7fee000c5990 2024-03-06T07:21:30.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.629+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee001ea550 con 0x7fee000c5990 2024-03-06T07:21:30.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.630+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0004f330 con 0x7fee000c5990 2024-03-06T07:21:30.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.630+0000 7fedf67fc640 1 -- 172.21.15.94:0/1104694877 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedf801d070 con 0x7fee000c5990 2024-03-06T07:21:30.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.631+0000 7fedf67fc640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fede406b4d0 0x7fede406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.631+0000 7fedf67fc640 1 -- 172.21.15.94:0/1104694877 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fedf808a9d0 con 0x7fee000c5990 2024-03-06T07:21:30.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.631+0000 7fee05faa640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fede406b4d0 0x7fede406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.631+0000 7fee05faa640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fede406b4d0 0x7fede406d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fedf000a750 tx=0x7fedf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.635+0000 7fedf67fc640 1 -- 172.21.15.94:0/1104694877 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedf8059a10 con 0x7fee000c5990 2024-03-06T07:21:30.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.805+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fee000c1740 con 0x7fee000c5990 2024-03-06T07:21:30.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.808+0000 7fedf67fc640 1 -- 172.21.15.94:0/1104694877 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v27) v1 ==== 96+0+0 (secure 0 0 0) 0x7fedf8059310 con 0x7fee000c5990 2024-03-06T07:21:30.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fede406b4d0 msgr2=0x7fede406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fede406b4d0 0x7fede406d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fedf000a750 tx=0x7fedf0008040 comp rx=0 tx=0).stop 2024-03-06T07:21:30.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee000c5990 msgr2=0x7fee001e71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee000c5990 0x7fee001e71a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fedf800cef0 tx=0x7fedf800a710 comp rx=0 tx=0).stop 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 shutdown_connections 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fede406b4d0 0x7fede406d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee000c5990 0x7fee001e71a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee000ab170 0x7fee001e6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 --2- 172.21.15.94:0/1104694877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee0006c900 0x7fee00144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 >> 172.21.15.94:0/1104694877 conn(0x7fee000c6710 msgr2=0x7fee0014aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 shutdown_connections 2024-03-06T07:21:30.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.810+0000 7fee0820e640 1 -- 172.21.15.94:0/1104694877 wait complete. 2024-03-06T07:21:30.823 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:21:30.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.923+0000 7f09bbe4e640 1 Processor -- start 2024-03-06T07:21:30.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.924+0000 7f09bbe4e640 1 -- start start 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.924+0000 7f09bbe4e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09b40c5990 0x7f09b40c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.924+0000 7f09bbe4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b40c3ef0 con 0x7f09b40ab170 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.924+0000 7f09bbe4e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b41556d0 con 0x7f09b406c900 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.924+0000 7f09bbe4e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b41559b0 con 0x7f09b40c5990 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.925+0000 7f09b9bea640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09b40c5990 0x7f09b40c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.925+0000 7f09b9bea640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09b40c5990 0x7f09b40c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52274/0 (socket says 172.21.15.94:52274) 2024-03-06T07:21:30.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.925+0000 7f09b9bea640 1 -- 172.21.15.94:0/2630592207 learned_addr learned my addr 172.21.15.94:0/2630592207 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:30.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.925+0000 7f09b9bea640 1 -- 172.21.15.94:0/2630592207 >> v1:172.21.15.145:6789/0 conn(0x7f09b406c900 legacy=0x7f09b40a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.925+0000 7f09b9bea640 1 -- 172.21.15.94:0/2630592207 >> v1:172.21.15.94:6789/0 conn(0x7f09b40ab170 legacy=0x7f09b406c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.925+0000 7f09b9bea640 1 -- 172.21.15.94:0/2630592207 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4155c90 con 0x7f09b40c5990 2024-03-06T07:21:30.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.925+0000 7f09b8be8640 1 -- 172.21.15.94:0/2630592207 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735715347 0 0) 0x7f09b40c3ef0 con 0x7f09b40ab170 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09b9bea640 1 --2- 172.21.15.94:0/2630592207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40c5990 0x7f09b40c0ab0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098a0 tx=0x7f09a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cd1cb542fe98c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09b8be8640 1 -- 172.21.15.94:0/2630592207 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8034070 con 0x7f09b40c5990 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09b8be8640 1 -- 172.21.15.94:0/2630592207 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8030840 con 0x7f09b40c5990 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09b8be8640 1 -- 172.21.15.94:0/2630592207 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8030c10 con 0x7f09b40c5990 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09bbe4e640 1 -- 172.21.15.94:0/2630592207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40c5990 msgr2=0x7f09b40c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/2630592207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40c5990 0x7f09b40c0ab0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f09a80098a0 tx=0x7f09a802b980 comp rx=0 tx=0).stop 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09bbe4e640 1 -- 172.21.15.94:0/2630592207 shutdown_connections 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/2630592207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40c5990 0x7f09b40c0ab0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09bbe4e640 1 -- 172.21.15.94:0/2630592207 >> 172.21.15.94:0/2630592207 conn(0x7f09b40c6710 msgr2=0x7f09b40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09bbe4e640 1 -- 172.21.15.94:0/2630592207 shutdown_connections 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.926+0000 7f09bbe4e640 1 -- 172.21.15.94:0/2630592207 wait complete. 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.927+0000 7f09bbe4e640 1 Processor -- start 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.927+0000 7f09bbe4e640 1 -- start start 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09bbe4e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09b406c900 0x7f09b4146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09bbe4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40ab170 0x7f09b41448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09bbe4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09b40c5990 0x7f09b4144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09bbe4e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4156610 con 0x7f09b40ab170 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09bbe4e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4156490 con 0x7f09b40c5990 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09bbe4e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b40c3af0 con 0x7f09b406c900 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40ab170 0x7f09b41448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40ab170 0x7f09b41448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52276/0 (socket says 172.21.15.94:52276) 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 -- 172.21.15.94:0/1814068108 learned_addr learned my addr 172.21.15.94:0/1814068108 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:30.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 -- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09b406c900 msgr2=0x7f09b4146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09b406c900 0x7f09b4146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 -- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09b40c5990 msgr2=0x7f09b4144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09b40c5990 0x7f09b4144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.928+0000 7f09b93e9640 1 -- 172.21.15.94:0/1814068108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b40a7ca0 con 0x7f09b40ab170 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.929+0000 7f09b93e9640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40ab170 0x7f09b41448a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c4e0 tx=0x7f09b000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.929+0000 7f09a67fc640 1 -- 172.21.15.94:0/1814068108 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0017070 con 0x7f09b40ab170 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.929+0000 7f09a67fc640 1 -- 172.21.15.94:0/1814068108 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b00035f0 con 0x7f09b40ab170 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.929+0000 7f09a67fc640 1 -- 172.21.15.94:0/1814068108 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b00039e0 con 0x7f09b40ab170 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.929+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b40a7f80 con 0x7f09b40ab170 2024-03-06T07:21:30.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.929+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b4152fa0 con 0x7f09b40ab170 2024-03-06T07:21:30.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.930+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b404f330 con 0x7f09b40ab170 2024-03-06T07:21:30.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.930+0000 7f09a67fc640 1 -- 172.21.15.94:0/1814068108 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09b001a070 con 0x7f09b40ab170 2024-03-06T07:21:30.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.931+0000 7f09a67fc640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f099806b500 0x7f099806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.931+0000 7f09a67fc640 1 -- 172.21.15.94:0/1814068108 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f09b0089360 con 0x7f09b40ab170 2024-03-06T07:21:30.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.931+0000 7f09b9bea640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f099806b500 0x7f099806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.931+0000 7f09b9bea640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f099806b500 0x7f099806d9c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f09a8036000 tx=0x7f09a8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:30.935+0000 7f09a67fc640 1 -- 172.21.15.94:0/1814068108 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09b0057c90 con 0x7f09b40ab170 2024-03-06T07:21:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.104+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f09b40c1740 con 0x7f09b40ab170 2024-03-06T07:21:31.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.108+0000 7f09a67fc640 1 -- 172.21.15.94:0/1814068108 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f09b005bd30 con 0x7f09b40ab170 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f099806b500 msgr2=0x7f099806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f099806b500 0x7f099806d9c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f09a8036000 tx=0x7f09a8030000 comp rx=0 tx=0).stop 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40ab170 msgr2=0x7f09b41448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40ab170 0x7f09b41448a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c4e0 tx=0x7f09b000c9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 shutdown_connections 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f099806b500 0x7f099806d9c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09b40c5990 0x7f09b4144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09b40ab170 0x7f09b41448a0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 --2- 172.21.15.94:0/1814068108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09b406c900 0x7f09b4146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 >> 172.21.15.94:0/1814068108 conn(0x7f09b40c6710 msgr2=0x7f09b414c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 shutdown_connections 2024-03-06T07:21:31.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.110+0000 7f09bbe4e640 1 -- 172.21.15.94:0/1814068108 wait complete. 2024-03-06T07:21:31.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:21:31.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.221+0000 7f9453b56640 1 Processor -- start 2024-03-06T07:21:31.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.221+0000 7f9453b56640 1 -- start start 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f9453b56640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f944c155c30 0x7f944c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f9453b56640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f944c0c5380 0x7f944c0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f9453b56640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944c0b0450 con 0x7f944c0b0070 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f9453b56640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c06cfb0 con 0x7f944c0c5380 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f9453b56640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c06c020 con 0x7f944c155c30 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f94520f3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f944c0c5380 0x7f944c0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f94520f3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f944c0c5380 0x7f944c0c5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52282/0 (socket says 172.21.15.94:52282) 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.222+0000 7f94520f3640 1 -- 172.21.15.94:0/1517891062 learned_addr learned my addr 172.21.15.94:0/1517891062 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94510f1640 1 --2- 172.21.15.94:0/1517891062 >> v2:172.21.15.145:3300/0 conn(0x7f944c155c30 0x7f944c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94520f3640 1 -- 172.21.15.94:0/1517891062 >> v2:172.21.15.145:3300/0 conn(0x7f944c155c30 msgr2=0x7f944c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94520f3640 1 --2- 172.21.15.94:0/1517891062 >> v2:172.21.15.145:3300/0 conn(0x7f944c155c30 0x7f944c06ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94520f3640 1 -- 172.21.15.94:0/1517891062 >> v1:172.21.15.94:6789/0 conn(0x7f944c0b0070 legacy=0x7f944c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94520f3640 1 -- 172.21.15.94:0/1517891062 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c1560e0 con 0x7f944c0c5380 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94508f0640 1 -- 172.21.15.94:0/1517891062 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145291375 0 0) 0x7f9448001490 con 0x7f944c0b0070 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94510f1640 1 --2- 172.21.15.94:0/1517891062 >> v2:172.21.15.145:3300/0 conn(0x7f944c155c30 0x7f944c06ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94520f3640 1 --2- 172.21.15.94:0/1517891062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0c5380 0x7f944c0c5790 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f944c06d4c0 tx=0x7f944402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66296f54bbf4a05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94508f0640 1 -- 172.21.15.94:0/1517891062 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444035020 con 0x7f944c0c5380 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94508f0640 1 -- 172.21.15.94:0/1517891062 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9444031870 con 0x7f944c0c5380 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.223+0000 7f94508f0640 1 -- 172.21.15.94:0/1517891062 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444031c40 con 0x7f944c0c5380 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 -- 172.21.15.94:0/1517891062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0c5380 msgr2=0x7f944c0c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 --2- 172.21.15.94:0/1517891062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0c5380 0x7f944c0c5790 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f944c06d4c0 tx=0x7f944402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 -- 172.21.15.94:0/1517891062 shutdown_connections 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 --2- 172.21.15.94:0/1517891062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0c5380 0x7f944c0c5790 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 --2- 172.21.15.94:0/1517891062 >> v2:172.21.15.145:3300/0 conn(0x7f944c155c30 0x7f944c06ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 -- 172.21.15.94:0/1517891062 >> 172.21.15.94:0/1517891062 conn(0x7f944c0ac590 msgr2=0x7f944c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 -- 172.21.15.94:0/1517891062 shutdown_connections 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.224+0000 7f9453b56640 1 -- 172.21.15.94:0/1517891062 wait complete. 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.225+0000 7f9453b56640 1 Processor -- start 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.225+0000 7f9453b56640 1 -- start start 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0b0070 0x7f944c1eb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f944c0c5380 0x7f944c0c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f944c155c30 0x7f944c0c2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c06c270 con 0x7f944c0b0070 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0c7ad0 con 0x7f944c155c30 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944c06c550 con 0x7f944c0c5380 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0b0070 0x7f944c1eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0b0070 0x7f944c1eb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52284/0 (socket says 172.21.15.94:52284) 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 -- 172.21.15.94:0/914482770 learned_addr learned my addr 172.21.15.94:0/914482770 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 -- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f944c0c5380 msgr2=0x7f944c0c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f944c0c5380 0x7f944c0c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 -- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f944c155c30 msgr2=0x7f944c0c2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f944c155c30 0x7f944c0c2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 -- 172.21.15.94:0/914482770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c1ebd80 con 0x7f944c0b0070 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94518f2640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0b0070 0x7f944c1eb840 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f944800c7f0 tx=0x7f944800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94427fc640 1 -- 172.21.15.94:0/914482770 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448017070 con 0x7f944c0b0070 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94427fc640 1 -- 172.21.15.94:0/914482770 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9448003da0 con 0x7f944c0b0070 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c151f20 con 0x7f944c0b0070 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f94427fc640 1 -- 172.21.15.94:0/914482770 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448005290 con 0x7f944c0b0070 2024-03-06T07:21:31.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.226+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c1524e0 con 0x7f944c0b0070 2024-03-06T07:21:31.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.227+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944c04f330 con 0x7f944c0b0070 2024-03-06T07:21:31.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.228+0000 7f94427fc640 1 -- 172.21.15.94:0/914482770 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f944801a070 con 0x7f944c0b0070 2024-03-06T07:21:31.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.228+0000 7f94427fc640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f943006b4d0 0x7f943006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.228+0000 7f94427fc640 1 -- 172.21.15.94:0/914482770 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f94480897d0 con 0x7f944c0b0070 2024-03-06T07:21:31.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.228+0000 7f94510f1640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f943006b4d0 0x7f943006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.229+0000 7f94510f1640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f943006b4d0 0x7f943006d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f943c006fd0 tx=0x7f943c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.232+0000 7f94427fc640 1 -- 172.21.15.94:0/914482770 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f944c0b0450 con 0x7f944c0b0070 2024-03-06T07:21:31.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.406+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f944c06d550 con 0x7f944c0b0070 2024-03-06T07:21:31.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.409+0000 7f94427fc640 1 -- 172.21.15.94:0/914482770 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v29) v1 ==== 96+0+0 (secure 0 0 0) 0x7f9448058750 con 0x7f944c0b0070 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f943006b4d0 msgr2=0x7f943006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f943006b4d0 0x7f943006d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f943c006fd0 tx=0x7f943c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0b0070 msgr2=0x7f944c1eb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0b0070 0x7f944c1eb840 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f944800c7f0 tx=0x7f944800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 shutdown_connections 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f943006b4d0 0x7f943006d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f944c155c30 0x7f944c0c2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f944c0c5380 0x7f944c0c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 --2- 172.21.15.94:0/914482770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f944c0b0070 0x7f944c1eb840 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 >> 172.21.15.94:0/914482770 conn(0x7f944c0ac590 msgr2=0x7f944c0ca1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 shutdown_connections 2024-03-06T07:21:31.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.411+0000 7f9453b56640 1 -- 172.21.15.94:0/914482770 wait complete. 2024-03-06T07:21:31.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.525+0000 7f1ae87c7640 1 Processor -- start 2024-03-06T07:21:31.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.525+0000 7f1ae87c7640 1 -- start start 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae87c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00b0070 0x7f1ae0155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae87c7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1ae006ca70 0x7f1ae00c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae87c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae00b0450 con 0x7f1ae00c5380 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae87c7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae006c020 con 0x7f1ae006ca70 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae87c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00c77f0 con 0x7f1ae00b0070 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae5d62640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1ae006ca70 0x7f1ae00c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae6d64640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1ae00c5380 0x7f1ae014e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35096/0 (socket says 172.21.15.94:35096) 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae6d64640 1 -- 172.21.15.94:0/109232535 learned_addr learned my addr 172.21.15.94:0/109232535 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae5d62640 1 -- 172.21.15.94:0/109232535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00b0070 msgr2=0x7f1ae0155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.526+0000 7f1ae6563640 1 --2- 172.21.15.94:0/109232535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00b0070 0x7f1ae0155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5d62640 1 --2- 172.21.15.94:0/109232535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00b0070 0x7f1ae0155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5d62640 1 -- 172.21.15.94:0/109232535 >> v1:172.21.15.94:6789/0 conn(0x7f1ae00c5380 legacy=0x7f1ae014e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5561640 1 -- 172.21.15.94:0/109232535 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370363432 0 0) 0x7f1ad8001110 con 0x7f1ae00c5380 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5d62640 1 -- 172.21.15.94:0/109232535 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00c7ad0 con 0x7f1ae006ca70 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae6563640 1 --2- 172.21.15.94:0/109232535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00b0070 0x7f1ae0155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5d62640 1 --2- 172.21.15.94:0/109232535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae006ca70 0x7f1ae00c8900 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f1ad00097a0 tx=0x7f1ad002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eeaedde87a3fbf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5561640 1 -- 172.21.15.94:0/109232535 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00315a0 con 0x7f1ae006ca70 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5561640 1 -- 172.21.15.94:0/109232535 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0037070 con 0x7f1ae006ca70 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.527+0000 7f1ae5561640 1 -- 172.21.15.94:0/109232535 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0031960 con 0x7f1ae006ca70 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 -- 172.21.15.94:0/109232535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae006ca70 msgr2=0x7f1ae00c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/109232535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae006ca70 0x7f1ae00c8900 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f1ad00097a0 tx=0x7f1ad002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 -- 172.21.15.94:0/109232535 shutdown_connections 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/109232535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae006ca70 0x7f1ae00c8900 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/109232535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00b0070 0x7f1ae0155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 -- 172.21.15.94:0/109232535 >> 172.21.15.94:0/109232535 conn(0x7f1ae00ac590 msgr2=0x7f1ae00ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 -- 172.21.15.94:0/109232535 shutdown_connections 2024-03-06T07:21:31.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.528+0000 7f1ae87c7640 1 -- 172.21.15.94:0/109232535 wait complete. 2024-03-06T07:21:31.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.529+0000 7f1ae87c7640 1 Processor -- start 2024-03-06T07:21:31.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.529+0000 7f1ae87c7640 1 -- start start 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae87c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ae006ca70 0x7f1ae014dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae87c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae00b0070 0x7f1ae0153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae87c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00c5380 0x7f1ae01eaae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae87c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae006cef0 con 0x7f1ae00b0070 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae87c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae006c550 con 0x7f1ae006ca70 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae87c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae006da80 con 0x7f1ae00c5380 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae00b0070 0x7f1ae0153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae00b0070 0x7f1ae0153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52308/0 (socket says 172.21.15.94:52308) 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 -- 172.21.15.94:0/101468440 learned_addr learned my addr 172.21.15.94:0/101468440 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae6d64640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00c5380 0x7f1ae01eaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae6563640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ae006ca70 0x7f1ae014dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 -- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00c5380 msgr2=0x7f1ae01eaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00c5380 0x7f1ae01eaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 -- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ae006ca70 msgr2=0x7f1ae014dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ae006ca70 0x7f1ae014dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 -- 172.21.15.94:0/101468440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae0155020 con 0x7f1ae00b0070 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae6563640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ae006ca70 0x7f1ae014dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae5d62640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae00b0070 0x7f1ae0153cf0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0034040 tx=0x7f1ad00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ad6ffd640 1 -- 172.21.15.94:0/101468440 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0032840 con 0x7f1ae00b0070 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ad6ffd640 1 -- 172.21.15.94:0/101468440 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0032a00 con 0x7f1ae00b0070 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ad6ffd640 1 -- 172.21.15.94:0/101468440 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0032df0 con 0x7f1ae00b0070 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.530+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae014da20 con 0x7f1ae00b0070 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.531+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae00bffe0 con 0x7f1ae00b0070 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.531+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae004f330 con 0x7f1ae00b0070 2024-03-06T07:21:31.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.532+0000 7f1ad6ffd640 1 -- 172.21.15.94:0/101468440 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ad004e430 con 0x7f1ae00b0070 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.532+0000 7f1ad6ffd640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.532+0000 7f1ad6ffd640 1 -- 172.21.15.94:0/101468440 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f1ad002f030 con 0x7f1ae00b0070 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.532+0000 7f1ae6563640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.533+0000 7f1ae6563640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00d430 tx=0x7f1adc007530 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.537+0000 7f1ad6ffd640 1 -- 172.21.15.94:0/101468440 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ad0040070 con 0x7f1ae00b0070 2024-03-06T07:21:31.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.707+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7f1ae006d860 con 0x7f1ae00b0070 2024-03-06T07:21:31.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.711+0000 7f1ad6ffd640 1 -- 172.21.15.94:0/101468440 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v30) v1 ==== 96+0+0 (secure 0 0 0) 0x7f1ad007b110 con 0x7f1ae00b0070 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ac406b4d0 msgr2=0x7f1ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00d430 tx=0x7f1adc007530 comp rx=0 tx=0).stop 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae00b0070 msgr2=0x7f1ae0153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae00b0070 0x7f1ae0153cf0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0034040 tx=0x7f1ad00046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 shutdown_connections 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ae00c5380 0x7f1ae01eaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae00b0070 0x7f1ae0153cf0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 --2- 172.21.15.94:0/101468440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ae006ca70 0x7f1ae014dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 >> 172.21.15.94:0/101468440 conn(0x7f1ae00ac590 msgr2=0x7f1ae01511a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 shutdown_connections 2024-03-06T07:21:31.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.713+0000 7f1ae87c7640 1 -- 172.21.15.94:0/101468440 wait complete. 2024-03-06T07:21:31.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-03-06T07:21:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.828+0000 7f24027b6640 1 Processor -- start 2024-03-06T07:21:31.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.828+0000 7f24027b6640 1 -- start start 2024-03-06T07:21:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f24027b6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23fc0c3710 0x7f23fc1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f24027b6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc158800 con 0x7f23fc06d9f0 2024-03-06T07:21:31.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f24027b6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc158ae0 con 0x7f23fc0c3280 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f24027b6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23fc158dc0 con 0x7f23fc0c3710 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f2400d53640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23fc0c3710 0x7f23fc1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f2400d53640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23fc0c3710 0x7f23fc1558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52322/0 (socket says 172.21.15.94:52322) 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f2400d53640 1 -- 172.21.15.94:0/1975795462 learned_addr learned my addr 172.21.15.94:0/1975795462 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f23fbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f23fc06d9f0 0x7f23fc0a7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35112/0 (socket says 172.21.15.94:35112) 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f2400d53640 1 -- 172.21.15.94:0/1975795462 >> v1:172.21.15.145:6789/0 conn(0x7f23fc0c3280 legacy=0x7f23fc1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f2400d53640 1 -- 172.21.15.94:0/1975795462 >> v1:172.21.15.94:6789/0 conn(0x7f23fc06d9f0 legacy=0x7f23fc0a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.829+0000 7f2400d53640 1 -- 172.21.15.94:0/1975795462 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc1590a0 con 0x7f23fc0c3710 2024-03-06T07:21:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.830+0000 7f2400d53640 1 --2- 172.21.15.94:0/1975795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3710 0x7f23fc1558f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009e50 tx=0x7f23f402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a83348fcdd8a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.830+0000 7f23faffd640 1 -- 172.21.15.94:0/1975795462 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f402ca80 con 0x7f23fc0c3710 2024-03-06T07:21:31.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.830+0000 7f23faffd640 1 -- 172.21.15.94:0/1975795462 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f402cc20 con 0x7f23fc0c3710 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.830+0000 7f23faffd640 1 -- 172.21.15.94:0/1975795462 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f40343f0 con 0x7f23fc0c3710 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.831+0000 7f24027b6640 1 -- 172.21.15.94:0/1975795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3710 msgr2=0x7f23fc1558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.831+0000 7f24027b6640 1 --2- 172.21.15.94:0/1975795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3710 0x7f23fc1558f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009e50 tx=0x7f23f402bee0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.831+0000 7f24027b6640 1 -- 172.21.15.94:0/1975795462 shutdown_connections 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.831+0000 7f24027b6640 1 --2- 172.21.15.94:0/1975795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3710 0x7f23fc1558f0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.831+0000 7f24027b6640 1 -- 172.21.15.94:0/1975795462 >> 172.21.15.94:0/1975795462 conn(0x7f23fc0aa630 msgr2=0x7f23fc0aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.831+0000 7f24027b6640 1 -- 172.21.15.94:0/1975795462 shutdown_connections 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.831+0000 7f24027b6640 1 -- 172.21.15.94:0/1975795462 wait complete. 2024-03-06T07:21:31.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.832+0000 7f24027b6640 1 Processor -- start 2024-03-06T07:21:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.832+0000 7f24027b6640 1 -- start start 2024-03-06T07:21:31.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f24027b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc06d9f0 0x7f23fc1dee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f24027b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3280 0x7f23fc14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f24027b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc0c3710 0x7f23fc06f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f24027b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc06ddd0 con 0x7f23fc0c3280 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f24027b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc155e30 con 0x7f23fc0c3710 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f24027b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc06df50 con 0x7f23fc06d9f0 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3280 0x7f23fc14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3280 0x7f23fc14e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52336/0 (socket says 172.21.15.94:52336) 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 -- 172.21.15.94:0/2458404683 learned_addr learned my addr 172.21.15.94:0/2458404683 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:31.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fbfff640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc06d9f0 0x7f23fc1dee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f2400d53640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc0c3710 0x7f23fc06f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 -- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc06d9f0 msgr2=0x7f23fc1dee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc06d9f0 0x7f23fc1dee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 -- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc0c3710 msgr2=0x7f23fc06f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc0c3710 0x7f23fc06f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 -- 172.21.15.94:0/2458404683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc14f740 con 0x7f23fc0c3280 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23fb7fe640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3280 0x7f23fc14e640 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f23f000c440 tx=0x7f23f000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.833+0000 7f23f8ff9640 1 -- 172.21.15.94:0/2458404683 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0017070 con 0x7f23fc0c3280 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.834+0000 7f23f8ff9640 1 -- 172.21.15.94:0/2458404683 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f00035f0 con 0x7f23fc0c3280 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.834+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc14f4a0 con 0x7f23fc0c3280 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.834+0000 7f23f8ff9640 1 -- 172.21.15.94:0/2458404683 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f00039e0 con 0x7f23fc0c3280 2024-03-06T07:21:31.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.834+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc14efc0 con 0x7f23fc0c3280 2024-03-06T07:21:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.834+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23fc04f330 con 0x7f23fc0c3280 2024-03-06T07:21:31.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.835+0000 7f23f8ff9640 1 -- 172.21.15.94:0/2458404683 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23f0003b80 con 0x7f23fc0c3280 2024-03-06T07:21:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.835+0000 7f23f8ff9640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e806b500 0x7f23e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.835+0000 7f23f8ff9640 1 -- 172.21.15.94:0/2458404683 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f23f0089400 con 0x7f23fc0c3280 2024-03-06T07:21:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.836+0000 7f23fbfff640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e806b500 0x7f23e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.836+0000 7f23fbfff640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e806b500 0x7f23e806d9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f23ec000f60 tx=0x7f23ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:31.839+0000 7f23f8ff9640 1 -- 172.21.15.94:0/2458404683 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23f0058270 con 0x7f23fc0c3280 2024-03-06T07:21:32.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.078+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7f23fc06ce70 con 0x7f23fc0c3280 2024-03-06T07:21:32.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.504+0000 7f23f8ff9640 1 -- 172.21.15.94:0/2458404683 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v200) v1 ==== 158+0+0 (secure 0 0 0) 0x7f23f00010c0 con 0x7f23fc0c3280 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.506+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e806b500 msgr2=0x7f23e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.506+0000 7f24027b6640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e806b500 0x7f23e806d9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f23ec000f60 tx=0x7f23ec009040 comp rx=0 tx=0).stop 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.506+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3280 msgr2=0x7f23fc14e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.506+0000 7f24027b6640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3280 0x7f23fc14e640 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f23f000c440 tx=0x7f23f000c900 comp rx=0 tx=0).stop 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 shutdown_connections 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23e806b500 0x7f23e806d9c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc0c3710 0x7f23fc06f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c3280 0x7f23fc14e640 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 --2- 172.21.15.94:0/2458404683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc06d9f0 0x7f23fc1dee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 >> 172.21.15.94:0/2458404683 conn(0x7f23fc0aa630 msgr2=0x7f23fc06c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 shutdown_connections 2024-03-06T07:21:32.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.507+0000 7f24027b6640 1 -- 172.21.15.94:0/2458404683 wait complete. 2024-03-06T07:21:32.520 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. ... ok 2024-03-06T07:21:32.520 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) That setting mds_join_fs at runtime affects the cluster layout. 2024-03-06T07:21:32.521 INFO:tasks.cephfs_test_runner:test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:21:32.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.620+0000 7f2cff179640 1 Processor -- start 2024-03-06T07:21:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.620+0000 7f2cff179640 1 -- start start 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cff179640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2cf81506b0 0x7f2cf8153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cff179640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2cf8156ea0 0x7f2cf80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cff179640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf8070800 con 0x7f2cf80acc00 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cff179640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf8156450 con 0x7f2cf8156ea0 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cff179640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf8156a10 con 0x7f2cf81506b0 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cf7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2cf8156ea0 0x7f2cf80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cf7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2cf8156ea0 0x7f2cf80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52346/0 (socket says 172.21.15.94:52346) 2024-03-06T07:21:32.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.621+0000 7f2cf7fff640 1 -- 172.21.15.94:0/149814010 learned_addr learned my addr 172.21.15.94:0/149814010 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf7fff640 1 -- 172.21.15.94:0/149814010 >> v2:172.21.15.145:3300/0 conn(0x7f2cf81506b0 msgr2=0x7f2cf8153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/149814010 >> v2:172.21.15.145:3300/0 conn(0x7f2cf81506b0 0x7f2cf8153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf7fff640 1 -- 172.21.15.94:0/149814010 >> v1:172.21.15.145:6789/0 conn(0x7f2cf80acc00 legacy=0x7f2cf80c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf7fff640 1 -- 172.21.15.94:0/149814010 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8070aa0 con 0x7f2cf8156ea0 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/149814010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf8156ea0 0x7f2cf80702c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2cf81565d0 tx=0x7f2ce002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e550f6c2b8d805 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf77fe640 1 -- 172.21.15.94:0/149814010 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0035020 con 0x7f2cf8156ea0 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf77fe640 1 -- 172.21.15.94:0/149814010 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0031870 con 0x7f2cf8156ea0 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.622+0000 7f2cf77fe640 1 -- 172.21.15.94:0/149814010 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0031c40 con 0x7f2cf8156ea0 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 -- 172.21.15.94:0/149814010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf8156ea0 msgr2=0x7f2cf80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 --2- 172.21.15.94:0/149814010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf8156ea0 0x7f2cf80702c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2cf81565d0 tx=0x7f2ce002b910 comp rx=0 tx=0).stop 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 -- 172.21.15.94:0/149814010 shutdown_connections 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 --2- 172.21.15.94:0/149814010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf8156ea0 0x7f2cf80702c0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 --2- 172.21.15.94:0/149814010 >> v2:172.21.15.145:3300/0 conn(0x7f2cf81506b0 0x7f2cf8153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 -- 172.21.15.94:0/149814010 >> 172.21.15.94:0/149814010 conn(0x7f2cf806b1a0 msgr2=0x7f2cf806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 -- 172.21.15.94:0/149814010 shutdown_connections 2024-03-06T07:21:32.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.623+0000 7f2cff179640 1 -- 172.21.15.94:0/149814010 wait complete. 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.624+0000 7f2cff179640 1 Processor -- start 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.624+0000 7f2cff179640 1 -- start start 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cff179640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf80acc00 0x7f2cf8151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cff179640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cf81506b0 0x7f2cf81ec310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cff179640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cf81537e0 0x7f2cf81ec850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cff179640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf80c5bb0 con 0x7f2cf80acc00 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cff179640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf81566a0 con 0x7f2cf81537e0 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cff179640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf80c5d30 con 0x7f2cf81506b0 2024-03-06T07:21:32.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf80acc00 0x7f2cf8151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf80acc00 0x7f2cf8151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52348/0 (socket says 172.21.15.94:52348) 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 -- 172.21.15.94:0/3735175677 learned_addr learned my addr 172.21.15.94:0/3735175677 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 -- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cf81506b0 msgr2=0x7f2cf81ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cf81506b0 0x7f2cf81ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 -- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cf81537e0 msgr2=0x7f2cf81ec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cf81537e0 0x7f2cf81ec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 -- 172.21.15.94:0/3735175677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8153d00 con 0x7f2cf80acc00 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.625+0000 7f2cfcf15640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf80acc00 0x7f2cf8151c30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000ba80 tx=0x7f2cf000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.626+0000 7f2cf57fa640 1 -- 172.21.15.94:0/3735175677 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0019070 con 0x7f2cf80acc00 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.626+0000 7f2cf57fa640 1 -- 172.21.15.94:0/3735175677 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf000cc20 con 0x7f2cf80acc00 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.626+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf8150e00 con 0x7f2cf80acc00 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.626+0000 7f2cf57fa640 1 -- 172.21.15.94:0/3735175677 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0014440 con 0x7f2cf80acc00 2024-03-06T07:21:32.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.626+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf81513c0 con 0x7f2cf80acc00 2024-03-06T07:21:32.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.627+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf804f330 con 0x7f2cf80acc00 2024-03-06T07:21:32.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.627+0000 7f2cf57fa640 1 -- 172.21.15.94:0/3735175677 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cf001c070 con 0x7f2cf80acc00 2024-03-06T07:21:32.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.627+0000 7f2cf57fa640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cd806b500 0x7f2cd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.628+0000 7f2cf57fa640 1 -- 172.21.15.94:0/3735175677 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f2cf0089ef0 con 0x7f2cf80acc00 2024-03-06T07:21:32.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.628+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cd806b500 0x7f2cd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.628+0000 7f2cf7fff640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cd806b500 0x7f2cd806d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0036000 tx=0x7f2ce0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.632+0000 7f2cf57fa640 1 -- 172.21.15.94:0/3735175677 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2cf0058de0 con 0x7f2cf80acc00 2024-03-06T07:21:32.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.806+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cf80700a0 con 0x7f2cf80acc00 2024-03-06T07:21:32.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.807+0000 7f2cf57fa640 1 -- 172.21.15.94:0/3735175677 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+13690 (secure 0 0 0) 0x7f2cf00586e0 con 0x7f2cf80acc00 2024-03-06T07:21:32.809 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:32.809 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":86,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:30.498191+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/3391508984":"2024-03-07T07:21:27.486286+0000","172.21.15.94:6832/3391508984":"2024-03-07T07:21:27.486286+0000","172.21.15.145:6835/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6834/3330183565":"2024-03-07T07:21:22.191279+0000","172.21.15.145:6837/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6836/2023918602":"2024-03-07T07:21:21.420819+0000","172.21.15.145:6837/3537750458":"2024-03-07T07:21:11.096670+0000","172.21.15.145:6836/3537750458":"2024-03-07T07:21:11.096670+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:32.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cd806b500 msgr2=0x7f2cd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cd806b500 0x7f2cd806d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0036000 tx=0x7f2ce0030000 comp rx=0 tx=0).stop 2024-03-06T07:21:32.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf80acc00 msgr2=0x7f2cf8151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf80acc00 0x7f2cf8151c30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000ba80 tx=0x7f2cf000bf40 comp rx=0 tx=0).stop 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 shutdown_connections 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cd806b500 0x7f2cd806d9c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cf81537e0 0x7f2cf81ec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cf81506b0 0x7f2cf81ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 --2- 172.21.15.94:0/3735175677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cf80acc00 0x7f2cf8151c30 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 >> 172.21.15.94:0/3735175677 conn(0x7f2cf806b1a0 msgr2=0x7f2cf80c4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 shutdown_connections 2024-03-06T07:21:32.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.809+0000 7f2cff179640 1 -- 172.21.15.94:0/3735175677 wait complete. 2024-03-06T07:21:32.822 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:32.822 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:32.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.923+0000 7f910175c640 1 Processor -- start 2024-03-06T07:21:32.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.923+0000 7f910175c640 1 -- start start 2024-03-06T07:21:32.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f910175c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f90fc155c30 0x7f90fc06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f910175c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f90fc0c5380 0x7f90fc0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f910175c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc0b0450 con 0x7f90fc0b0070 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f910175c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc06cfb0 con 0x7f90fc155c30 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f910175c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc06c020 con 0x7f90fc0c5380 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fa7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f90fc155c30 0x7f90fc06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fa7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f90fc155c30 0x7f90fc06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52362/0 (socket says 172.21.15.94:52362) 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fa7fc640 1 -- 172.21.15.94:0/2953502301 learned_addr learned my addr 172.21.15.94:0/2953502301 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fb7fe640 1 --2- 172.21.15.94:0/2953502301 >> v2:172.21.15.145:3300/0 conn(0x7f90fc0c5380 0x7f90fc0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fa7fc640 1 -- 172.21.15.94:0/2953502301 >> v2:172.21.15.145:3300/0 conn(0x7f90fc0c5380 msgr2=0x7f90fc0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fa7fc640 1 --2- 172.21.15.94:0/2953502301 >> v2:172.21.15.145:3300/0 conn(0x7f90fc0c5380 0x7f90fc0c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fa7fc640 1 -- 172.21.15.94:0/2953502301 >> v1:172.21.15.145:6789/0 conn(0x7f90fc0b0070 legacy=0x7f90fc0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.924+0000 7f90fa7fc640 1 -- 172.21.15.94:0/2953502301 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc1560e0 con 0x7f90fc155c30 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.925+0000 7f90fa7fc640 1 --2- 172.21.15.94:0/2953502301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc06ca70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0be030 tx=0x7f90f002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852c95454f47c917 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.925+0000 7f90f9ffb640 1 -- 172.21.15.94:0/2953502301 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f002e070 con 0x7f90fc155c30 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.925+0000 7f90f9ffb640 1 -- 172.21.15.94:0/2953502301 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f0037070 con 0x7f90fc155c30 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.925+0000 7f90f9ffb640 1 -- 172.21.15.94:0/2953502301 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f0031a00 con 0x7f90fc155c30 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 -- 172.21.15.94:0/2953502301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 msgr2=0x7f90fc06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 --2- 172.21.15.94:0/2953502301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc06ca70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0be030 tx=0x7f90f002b880 comp rx=0 tx=0).stop 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 -- 172.21.15.94:0/2953502301 shutdown_connections 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 --2- 172.21.15.94:0/2953502301 >> v2:172.21.15.145:3300/0 conn(0x7f90fc0c5380 0x7f90fc0c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 --2- 172.21.15.94:0/2953502301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc06ca70 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 -- 172.21.15.94:0/2953502301 >> 172.21.15.94:0/2953502301 conn(0x7f90fc0ac590 msgr2=0x7f90fc0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 -- 172.21.15.94:0/2953502301 shutdown_connections 2024-03-06T07:21:32.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.926+0000 7f910175c640 1 -- 172.21.15.94:0/2953502301 wait complete. 2024-03-06T07:21:32.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.927+0000 7f910175c640 1 Processor -- start 2024-03-06T07:21:32.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.927+0000 7f910175c640 1 -- start start 2024-03-06T07:21:32.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.927+0000 7f910175c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90fc0b0070 0x7f90fc1ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f910175c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90fc0c5380 0x7f90fc0c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f910175c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc0c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f910175c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc06c270 con 0x7f90fc155c30 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f910175c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0c7ad0 con 0x7f90fc0b0070 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f910175c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc06c550 con 0x7f90fc0c5380 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc0c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc0c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52370/0 (socket says 172.21.15.94:52370) 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 -- 172.21.15.94:0/2662163253 learned_addr learned my addr 172.21.15.94:0/2662163253 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90faffd640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90fc0b0070 0x7f90fc1ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fa7fc640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90fc0c5380 0x7f90fc0c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 -- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90fc0c5380 msgr2=0x7f90fc0c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90fc0c5380 0x7f90fc0c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 -- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90fc0b0070 msgr2=0x7f90fc1ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90fc0b0070 0x7f90fc1ee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 -- 172.21.15.94:0/2662163253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc1eee00 con 0x7f90fc155c30 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fa7fc640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90fc0c5380 0x7f90fc0c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:32.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90fb7fe640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc0c2c20 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f90f4007e40 tx=0x7f90f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90e37fe640 1 -- 172.21.15.94:0/2662163253 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4014620 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90e37fe640 1 -- 172.21.15.94:0/2662163253 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f4019070 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc0c4500 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f90e37fe640 1 -- 172.21.15.94:0/2662163253 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4014a20 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.928+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc0beb40 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.929+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc04f330 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.930+0000 7f90e37fe640 1 -- 172.21.15.94:0/2662163253 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90f4014bc0 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.930+0000 7f90e37fe640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f90e806b4d0 0x7f90e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.930+0000 7f90e37fe640 1 -- 172.21.15.94:0/2662163253 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f90f408a990 con 0x7f90fc155c30 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.930+0000 7f90faffd640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f90e806b4d0 0x7f90e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.931+0000 7f90faffd640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f90e806b4d0 0x7f90e806d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f90ec006fd0 tx=0x7f90ec009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:32.934+0000 7f90e37fe640 1 -- 172.21.15.94:0/2662163253 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90f4059910 con 0x7f90fc155c30 2024-03-06T07:21:33.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.106+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f90fc0c4b00 con 0x7f90fc155c30 2024-03-06T07:21:33.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.106+0000 7f90e37fe640 1 -- 172.21.15.94:0/2662163253 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90f4059210 con 0x7f90fc155c30 2024-03-06T07:21:33.108 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:21:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f90e806b4d0 msgr2=0x7f90e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f90e806b4d0 0x7f90e806d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f90ec006fd0 tx=0x7f90ec009060 comp rx=0 tx=0).stop 2024-03-06T07:21:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 msgr2=0x7f90fc0c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc0c2c20 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f90f4007e40 tx=0x7f90f400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 shutdown_connections 2024-03-06T07:21:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f90e806b4d0 0x7f90e806d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90fc155c30 0x7f90fc0c2c20 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90fc0c5380 0x7f90fc0c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 --2- 172.21.15.94:0/2662163253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90fc0b0070 0x7f90fc1ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 >> 172.21.15.94:0/2662163253 conn(0x7f90fc0ac590 msgr2=0x7f90fc0ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 shutdown_connections 2024-03-06T07:21:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.108+0000 7f910175c640 1 -- 172.21.15.94:0/2662163253 wait complete. 2024-03-06T07:21:33.121 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-03-06T07:21:33.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.218+0000 7ff05cfd7640 1 Processor -- start 2024-03-06T07:21:33.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.218+0000 7ff05cfd7640 1 -- start start 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff05cfd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0580c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff05cfd7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0581586f0 0x7ff058158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff05cfd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0581591c0 con 0x7ff0580c5480 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff05cfd7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05806df40 con 0x7ff0581586f0 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff05cfd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05806fdb0 con 0x7ff058155ba0 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff056d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0581586f0 0x7ff058158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff056d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0581586f0 0x7ff058158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52378/0 (socket says 172.21.15.94:52378) 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.219+0000 7ff056d9d640 1 -- 172.21.15.94:0/886864075 learned_addr learned my addr 172.21.15.94:0/886864075 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff055d9b640 1 --2- 172.21.15.94:0/886864075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0580c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff056d9d640 1 -- 172.21.15.94:0/886864075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 msgr2=0x7ff0580c5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff056d9d640 1 --2- 172.21.15.94:0/886864075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0580c5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff056d9d640 1 -- 172.21.15.94:0/886864075 >> v1:172.21.15.145:6789/0 conn(0x7ff0580c5480 legacy=0x7ff05806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff056d9d640 1 -- 172.21.15.94:0/886864075 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0581594a0 con 0x7ff0581586f0 2024-03-06T07:21:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff056d9d640 1 --2- 172.21.15.94:0/886864075 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff058158b80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff050009870 tx=0x7ff05002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b840f9de82cb853 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff05559a640 1 -- 172.21.15.94:0/886864075 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0500317c0 con 0x7ff0581586f0 2024-03-06T07:21:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.220+0000 7ff05559a640 1 -- 172.21.15.94:0/886864075 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff050037070 con 0x7ff0581586f0 2024-03-06T07:21:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05559a640 1 -- 172.21.15.94:0/886864075 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050031b80 con 0x7ff0581586f0 2024-03-06T07:21:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 -- 172.21.15.94:0/886864075 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 msgr2=0x7ff058158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/886864075 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff058158b80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff050009870 tx=0x7ff05002b860 comp rx=0 tx=0).stop 2024-03-06T07:21:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 -- 172.21.15.94:0/886864075 shutdown_connections 2024-03-06T07:21:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/886864075 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff058158b80 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/886864075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0580c5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 -- 172.21.15.94:0/886864075 >> 172.21.15.94:0/886864075 conn(0x7ff05806b1a0 msgr2=0x7ff05806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 -- 172.21.15.94:0/886864075 shutdown_connections 2024-03-06T07:21:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.221+0000 7ff05cfd7640 1 -- 172.21.15.94:0/886864075 wait complete. 2024-03-06T07:21:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.222+0000 7ff05cfd7640 1 Processor -- start 2024-03-06T07:21:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.222+0000 7ff05cfd7640 1 -- start start 2024-03-06T07:21:33.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05cfd7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0580c5480 0x7ff0580c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05cfd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0581ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05cfd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff0581eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05cfd7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058155680 con 0x7ff0581586f0 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05cfd7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058156000 con 0x7ff0580c5480 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05cfd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff058152a20 con 0x7ff058155ba0 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff0581eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff0581eeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52388/0 (socket says 172.21.15.94:52388) 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 -- 172.21.15.94:0/2674535 learned_addr learned my addr 172.21.15.94:0/2674535 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff055d9b640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0581ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 -- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 msgr2=0x7ff0581ee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05659c640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0580c5480 0x7ff0580c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0581ee570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 -- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0580c5480 msgr2=0x7ff0580c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0580c5480 0x7ff0580c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff056d9d640 1 -- 172.21.15.94:0/2674535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0581eba70 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff055d9b640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0581ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.223+0000 7ff05659c640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0580c5480 0x7ff0580c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.224+0000 7ff056d9d640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff0581eeab0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff05002b8a0 tx=0x7ff0500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.224+0000 7ff04effd640 1 -- 172.21.15.94:0/2674535 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050032650 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.224+0000 7ff04effd640 1 -- 172.21.15.94:0/2674535 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0500327f0 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.224+0000 7ff04effd640 1 -- 172.21.15.94:0/2674535 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050032be0 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.224+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0581ebd50 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.224+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0580c8ed0 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.225+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05804f330 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.225+0000 7ff04effd640 1 -- 172.21.15.94:0/2674535 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff05004c060 con 0x7ff0581586f0 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.226+0000 7ff04effd640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff03806b4d0 0x7ff03806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.226+0000 7ff04effd640 1 -- 172.21.15.94:0/2674535 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7ff0500ab760 con 0x7ff0581586f0 2024-03-06T07:21:33.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.226+0000 7ff05659c640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff03806b4d0 0x7ff03806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.226+0000 7ff05659c640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff03806b4d0 0x7ff03806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff0580c2900 tx=0x7ff048009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.230+0000 7ff04effd640 1 -- 172.21.15.94:0/2674535 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff050040070 con 0x7ff0581586f0 2024-03-06T07:21:33.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.467+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7ff05806ba70 con 0x7ff0581586f0 2024-03-06T07:21:33.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.509+0000 7ff04effd640 1 -- 172.21.15.94:0/2674535 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v201) v1 ==== 149+0+0 (secure 0 0 0) 0x7ff05007a800 con 0x7ff0581586f0 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff03806b4d0 msgr2=0x7ff03806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff03806b4d0 0x7ff03806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff0580c2900 tx=0x7ff048009040 comp rx=0 tx=0).stop 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 msgr2=0x7ff0581eeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff0581eeab0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff05002b8a0 tx=0x7ff0500046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 shutdown_connections 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff03806b4d0 0x7ff03806d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0581586f0 0x7ff0581eeab0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff058155ba0 0x7ff0581ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 --2- 172.21.15.94:0/2674535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0580c5480 0x7ff0580c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 >> 172.21.15.94:0/2674535 conn(0x7ff05806b1a0 msgr2=0x7ff058154b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 shutdown_connections 2024-03-06T07:21:33.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.511+0000 7ff05cfd7640 1 -- 172.21.15.94:0/2674535 wait complete. 2024-03-06T07:21:33.523 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:21:33.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.625+0000 7f7c086c9640 1 Processor -- start 2024-03-06T07:21:33.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.625+0000 7f7c086c9640 1 -- start start 2024-03-06T07:21:33.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c086c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c2aa0 0x7f7c00155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c086c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7c000c0470 0x7f7c00152350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c086c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000670c0 con 0x7f7c000c26c0 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c086c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0006b740 con 0x7f7c000c0470 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c086c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0006b0f0 con 0x7f7c000c2aa0 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06c66640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7c000c0470 0x7f7c00152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06c66640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7c000c0470 0x7f7c00152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52394/0 (socket says 172.21.15.94:52394) 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06465640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7c000c26c0 0x7f7c0006bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35128/0 (socket says 172.21.15.94:35128) 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06c66640 1 -- 172.21.15.94:0/124028337 learned_addr learned my addr 172.21.15.94:0/124028337 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c05c64640 1 --2- 172.21.15.94:0/124028337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c2aa0 0x7f7c00155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06c66640 1 -- 172.21.15.94:0/124028337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c2aa0 msgr2=0x7f7c00155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06c66640 1 --2- 172.21.15.94:0/124028337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c2aa0 0x7f7c00155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06c66640 1 -- 172.21.15.94:0/124028337 >> v1:172.21.15.94:6789/0 conn(0x7f7c000c26c0 legacy=0x7f7c0006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c05463640 1 -- 172.21.15.94:0/124028337 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411262747 0 0) 0x7f7c000670c0 con 0x7f7c000c26c0 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.626+0000 7f7c06c66640 1 -- 172.21.15.94:0/124028337 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0006b3d0 con 0x7f7c000c0470 2024-03-06T07:21:33.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.627+0000 7f7c06c66640 1 --2- 172.21.15.94:0/124028337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c0470 0x7f7c00152350 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8009e50 tx=0x7f7bf802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5054ebc5ad339096 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.627+0000 7f7c05463640 1 -- 172.21.15.94:0/124028337 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf802c9e0 con 0x7f7c000c0470 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.627+0000 7f7c05463640 1 -- 172.21.15.94:0/124028337 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf802cb80 con 0x7f7c000c0470 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.627+0000 7f7c05463640 1 -- 172.21.15.94:0/124028337 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf80343f0 con 0x7f7c000c0470 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.627+0000 7f7c086c9640 1 -- 172.21.15.94:0/124028337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c0470 msgr2=0x7f7c00152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.627+0000 7f7c086c9640 1 --2- 172.21.15.94:0/124028337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c0470 0x7f7c00152350 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8009e50 tx=0x7f7bf802be40 comp rx=0 tx=0).stop 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.628+0000 7f7c086c9640 1 -- 172.21.15.94:0/124028337 shutdown_connections 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.628+0000 7f7c086c9640 1 --2- 172.21.15.94:0/124028337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c0470 0x7f7c00152350 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.628+0000 7f7c086c9640 1 --2- 172.21.15.94:0/124028337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c2aa0 0x7f7c00155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.628+0000 7f7c086c9640 1 -- 172.21.15.94:0/124028337 >> 172.21.15.94:0/124028337 conn(0x7f7c000c3b40 msgr2=0x7f7c000c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.628+0000 7f7c086c9640 1 -- 172.21.15.94:0/124028337 shutdown_connections 2024-03-06T07:21:33.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.628+0000 7f7c086c9640 1 -- 172.21.15.94:0/124028337 wait complete. 2024-03-06T07:21:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 Processor -- start 2024-03-06T07:21:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 -- start start 2024-03-06T07:21:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c0470 0x7f7c00154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c26c0 0x7f7c00149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c000c2aa0 0x7f7c00145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000c20b0 con 0x7f7c000c26c0 2024-03-06T07:21:33.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000c1f30 con 0x7f7c000c2aa0 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c086c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00066d00 con 0x7f7c000c0470 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c05c64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c26c0 0x7f7c00149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c05c64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c26c0 0x7f7c00149c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52406/0 (socket says 172.21.15.94:52406) 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.629+0000 7f7c05c64640 1 -- 172.21.15.94:0/3784236738 learned_addr learned my addr 172.21.15.94:0/3784236738 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c06c66640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c000c2aa0 0x7f7c00145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c06465640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c0470 0x7f7c00154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c05c64640 1 -- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c0470 msgr2=0x7f7c00154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c05c64640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c0470 0x7f7c00154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c05c64640 1 -- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c000c2aa0 msgr2=0x7f7c00145740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c05c64640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c000c2aa0 0x7f7c00145740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c05c64640 1 -- 172.21.15.94:0/3784236738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000c6e20 con 0x7f7c000c26c0 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c05c64640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c26c0 0x7f7c00149c60 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000d800 tx=0x7f7bf000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7bf6ffd640 1 -- 172.21.15.94:0/3784236738 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf000b6c0 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7bf6ffd640 1 -- 172.21.15.94:0/3784236738 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf0019070 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7bf6ffd640 1 -- 172.21.15.94:0/3784236738 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf000ba80 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00154e30 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.630+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c00148b90 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.631+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0004f330 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.631+0000 7f7bf6ffd640 1 -- 172.21.15.94:0/3784236738 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bf000bc20 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.632+0000 7f7bf6ffd640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7be406b4d0 0x7f7be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.632+0000 7f7bf6ffd640 1 -- 172.21.15.94:0/3784236738 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f7bf008a5e0 con 0x7f7c000c26c0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.632+0000 7f7c06465640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7be406b4d0 0x7f7be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.633+0000 7f7c06465640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7be406b4d0 0x7f7be406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc000f60 tx=0x7f7bfc007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.636+0000 7f7bf6ffd640 1 -- 172.21.15.94:0/3784236738 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bf0059560 con 0x7f7c000c26c0 2024-03-06T07:21:33.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.808+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7c0006be80 con 0x7f7c000c26c0 2024-03-06T07:21:33.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.811+0000 7f7bf6ffd640 1 -- 172.21.15.94:0/3784236738 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v31)=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7bf0058e60 con 0x7f7c000c26c0 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7be406b4d0 msgr2=0x7f7be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7be406b4d0 0x7f7be406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc000f60 tx=0x7f7bfc007b50 comp rx=0 tx=0).stop 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c26c0 msgr2=0x7f7c00149c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c26c0 0x7f7c00149c60 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000d800 tx=0x7f7bf000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 shutdown_connections 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7be406b4d0 0x7f7be406d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c000c2aa0 0x7f7c00145740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c000c26c0 0x7f7c00149c60 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 --2- 172.21.15.94:0/3784236738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c000c0470 0x7f7c00154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 >> 172.21.15.94:0/3784236738 conn(0x7f7c000c3b40 msgr2=0x7f7c00150d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 shutdown_connections 2024-03-06T07:21:33.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:33.813+0000 7f7c086c9640 1 -- 172.21.15.94:0/3784236738 wait complete. 2024-03-06T07:21:33.826 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:21:33.826 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:33.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:33.917 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:21:33.917 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:21:33.917 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:33.933 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:21:33.933 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:21:33.933 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:33.974 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:21:33.975 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:21:33.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:34.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.073+0000 7faa77ee0640 1 Processor -- start 2024-03-06T07:21:34.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.073+0000 7faa77ee0640 1 -- start start 2024-03-06T07:21:34.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa77ee0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7014f790 0x7faa701529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa77ee0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa700c1aa0 0x7faa700c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa77ee0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa700c28d0 con 0x7faa700c5c60 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa77ee0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa700c2680 con 0x7faa700c1aa0 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa77ee0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa700c2bb0 con 0x7faa7014f790 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa7547b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa700c1aa0 0x7faa700c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa7547b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa700c1aa0 0x7faa700c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52416/0 (socket says 172.21.15.94:52416) 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa7547b640 1 -- 172.21.15.94:0/772494186 learned_addr learned my addr 172.21.15.94:0/772494186 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa75c7c640 1 --2- 172.21.15.94:0/772494186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7014f790 0x7faa701529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa7547b640 1 -- 172.21.15.94:0/772494186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7014f790 msgr2=0x7faa701529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa7547b640 1 --2- 172.21.15.94:0/772494186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7014f790 0x7faa701529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa7547b640 1 -- 172.21.15.94:0/772494186 >> v1:172.21.15.145:6789/0 conn(0x7faa700c5c60 legacy=0x7faa70155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.074+0000 7faa7547b640 1 -- 172.21.15.94:0/772494186 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa700c2e90 con 0x7faa700c1aa0 2024-03-06T07:21:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.075+0000 7faa7547b640 1 --2- 172.21.15.94:0/772494186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa700c1aa0 0x7faa700c5720 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7faa60009870 tx=0x7faa6002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17edcddd1b15f314 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.075+0000 7faa74c7a640 1 -- 172.21.15.94:0/772494186 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60035020 con 0x7faa700c1aa0 2024-03-06T07:21:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.075+0000 7faa74c7a640 1 -- 172.21.15.94:0/772494186 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60031640 con 0x7faa700c1aa0 2024-03-06T07:21:34.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.075+0000 7faa74c7a640 1 -- 172.21.15.94:0/772494186 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6002d040 con 0x7faa700c1aa0 2024-03-06T07:21:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.075+0000 7faa77ee0640 1 -- 172.21.15.94:0/772494186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa700c1aa0 msgr2=0x7faa700c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.075+0000 7faa77ee0640 1 --2- 172.21.15.94:0/772494186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa700c1aa0 0x7faa700c5720 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7faa60009870 tx=0x7faa6002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.076+0000 7faa77ee0640 1 -- 172.21.15.94:0/772494186 shutdown_connections 2024-03-06T07:21:34.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.076+0000 7faa77ee0640 1 --2- 172.21.15.94:0/772494186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa700c1aa0 0x7faa700c5720 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.076+0000 7faa77ee0640 1 --2- 172.21.15.94:0/772494186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7014f790 0x7faa701529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.076+0000 7faa77ee0640 1 -- 172.21.15.94:0/772494186 >> 172.21.15.94:0/772494186 conn(0x7faa700c6710 msgr2=0x7faa700c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.076+0000 7faa77ee0640 1 -- 172.21.15.94:0/772494186 shutdown_connections 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.076+0000 7faa77ee0640 1 -- 172.21.15.94:0/772494186 wait complete. 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 Processor -- start 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 -- start start 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa700c1aa0 0x7faa7015cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa700c5c60 0x7faa701ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa7014f790 0x7faa701ecb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7006cc00 con 0x7faa7014f790 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa700c0d60 con 0x7faa700c1aa0 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa77ee0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7006cd80 con 0x7faa700c5c60 2024-03-06T07:21:34.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa7647d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa7014f790 0x7faa701ecb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa7647d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa7014f790 0x7faa701ecb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52420/0 (socket says 172.21.15.94:52420) 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.077+0000 7faa7647d640 1 -- 172.21.15.94:0/3006870029 learned_addr learned my addr 172.21.15.94:0/3006870029 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa75c7c640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa700c1aa0 0x7faa7015cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7547b640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa700c5c60 0x7faa701ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7647d640 1 -- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa700c5c60 msgr2=0x7faa701ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7647d640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa700c5c60 0x7faa701ec620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7647d640 1 -- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa700c1aa0 msgr2=0x7faa7015cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7647d640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa700c1aa0 0x7faa7015cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7647d640 1 -- 172.21.15.94:0/3006870029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa701e9ae0 con 0x7faa7014f790 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7547b640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa700c5c60 0x7faa701ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa7647d640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa7014f790 0x7faa701ecb60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faa6800c7b0 tx=0x7faa6800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa667fc640 1 -- 172.21.15.94:0/3006870029 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68004b60 con 0x7faa7014f790 2024-03-06T07:21:34.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa667fc640 1 -- 172.21.15.94:0/3006870029 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa68004d00 con 0x7faa7014f790 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa667fc640 1 -- 172.21.15.94:0/3006870029 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6800aa90 con 0x7faa7014f790 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa700bc360 con 0x7faa7014f790 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.078+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7015d2a0 con 0x7faa7014f790 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.079+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7004f330 con 0x7faa7014f790 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.080+0000 7faa667fc640 1 -- 172.21.15.94:0/3006870029 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa6800ac30 con 0x7faa7014f790 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.080+0000 7faa667fc640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.080+0000 7faa667fc640 1 -- 172.21.15.94:0/3006870029 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7faa6808a3d0 con 0x7faa7014f790 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.080+0000 7faa75c7c640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.081+0000 7faa75c7c640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006fd0 tx=0x7faa6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.084+0000 7faa667fc640 1 -- 172.21.15.94:0/3006870029 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa68020070 con 0x7faa7014f790 2024-03-06T07:21:34.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.293+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa700c1f40 con 0x7faa7014f790 2024-03-06T07:21:34.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.293+0000 7faa667fc640 1 -- 172.21.15.94:0/3006870029 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+3706 (secure 0 0 0) 0x7faa68059350 con 0x7faa7014f790 2024-03-06T07:21:34.296 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:34.296 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.94:6835/3556888574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3556888574},{"type":"v1","addr":"172.21.15.94:6835","nonce":3556888574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767091656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767091656},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767091656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":6843,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1235147662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1235147662},{"type":"v1","addr":"172.21.15.94:6834","nonce":1235147662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":35795,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3407932030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3407932030},{"type":"v1","addr":"172.21.15.145:6837","nonce":3407932030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[]} 2024-03-06T07:21:34.296 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 73 2024-03-06T07:21:34.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.295+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 msgr2=0x7faa5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.295+0000 7faa77ee0640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006fd0 tx=0x7faa6c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.295+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa7014f790 msgr2=0x7faa701ecb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.295+0000 7faa77ee0640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa7014f790 0x7faa701ecb60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faa6800c7b0 tx=0x7faa6800cc70 comp rx=0 tx=0).stop 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 shutdown_connections 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5406b4d0 0x7faa5406d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa7014f790 0x7faa701ecb60 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa700c5c60 0x7faa701ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 --2- 172.21.15.94:0/3006870029 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa700c1aa0 0x7faa7015cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 >> 172.21.15.94:0/3006870029 conn(0x7faa700c6710 msgr2=0x7faa7006d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 shutdown_connections 2024-03-06T07:21:34.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.296+0000 7faa77ee0640 1 -- 172.21.15.94:0/3006870029 wait complete. 2024-03-06T07:21:34.308 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:21:34.309 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:21:34.309 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:21:34.309 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:34.309 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:21:34.310 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:21:34.310 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:21:34.310 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:34.310 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:21:34.310 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:21:34.310 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:21:34.310 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:34.311 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:21:34.311 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:21:34.311 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:21:34.311 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:21:34.318 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:21:34.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:21:34.319 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:21:34.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:21:34.322 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:21:34.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:21:34.323 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:21:34.323 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:21:34.361 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:21:34.361 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:34.362 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:21:34.362 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:34.365 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:21:34.365 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:34.365 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:21:34.365 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:34.366 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:21:34.438 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:21:34.438 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:21:34.447 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:21:34.448 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:21:34.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.469+0000 7fe72bdb5640 1 Processor -- start 2024-03-06T07:21:34.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.469+0000 7fe72bdb5640 1 -- start start 2024-03-06T07:21:34.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe72bdb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe7240c69f0 0x7fe7240c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe72bdb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe7240c8950 0x7fe7240acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe72bdb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe724155130 con 0x7fe7240ad2b0 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe72bdb5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe724155440 con 0x7fe7240c69f0 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe72bdb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7240c5d50 con 0x7fe7240c8950 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729b51640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe7240c69f0 0x7fe7240c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729b51640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe7240c69f0 0x7fe7240c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52460/0 (socket says 172.21.15.94:52460) 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729b51640 1 -- 172.21.15.94:0/2949939559 learned_addr learned my addr 172.21.15.94:0/2949939559 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729350640 1 --2- 172.21.15.94:0/2949939559 >> v2:172.21.15.145:3300/0 conn(0x7fe7240c8950 0x7fe7240acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729b51640 1 -- 172.21.15.94:0/2949939559 >> v2:172.21.15.145:3300/0 conn(0x7fe7240c8950 msgr2=0x7fe7240acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729b51640 1 --2- 172.21.15.94:0/2949939559 >> v2:172.21.15.145:3300/0 conn(0x7fe7240c8950 0x7fe7240acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729b51640 1 -- 172.21.15.94:0/2949939559 >> v1:172.21.15.145:6789/0 conn(0x7fe7240ad2b0 legacy=0x7fe7241558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.470+0000 7fe729b51640 1 -- 172.21.15.94:0/2949939559 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7240c6030 con 0x7fe7240c69f0 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.471+0000 7fe729b51640 1 --2- 172.21.15.94:0/2949939559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240c69f0 0x7fe7240c6dd0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe7240c7430 tx=0x7fe72002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652ef797484786ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.471+0000 7fe728b4f640 1 -- 172.21.15.94:0/2949939559 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720035020 con 0x7fe7240c69f0 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.471+0000 7fe728b4f640 1 -- 172.21.15.94:0/2949939559 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe720031800 con 0x7fe7240c69f0 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.471+0000 7fe728b4f640 1 -- 172.21.15.94:0/2949939559 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720031c10 con 0x7fe7240c69f0 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.471+0000 7fe72bdb5640 1 -- 172.21.15.94:0/2949939559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240c69f0 msgr2=0x7fe7240c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.471+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/2949939559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240c69f0 0x7fe7240c6dd0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe7240c7430 tx=0x7fe72002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.472+0000 7fe72bdb5640 1 -- 172.21.15.94:0/2949939559 shutdown_connections 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.472+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/2949939559 >> v2:172.21.15.145:3300/0 conn(0x7fe7240c8950 0x7fe7240acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.472+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/2949939559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240c69f0 0x7fe7240c6dd0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.472+0000 7fe72bdb5640 1 -- 172.21.15.94:0/2949939559 >> 172.21.15.94:0/2949939559 conn(0x7fe7240cae70 msgr2=0x7fe7240cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.472+0000 7fe72bdb5640 1 -- 172.21.15.94:0/2949939559 shutdown_connections 2024-03-06T07:21:34.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.472+0000 7fe72bdb5640 1 -- 172.21.15.94:0/2949939559 wait complete. 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 Processor -- start 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 -- start start 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240ad2b0 0x7fe72406c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7240c69f0 0x7fe7241ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7240c8950 0x7fe72414b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe724152a20 con 0x7fe7240ad2b0 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe724155fe0 con 0x7fe7240c8950 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.473+0000 7fe72bdb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe724152ba0 con 0x7fe7240c69f0 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240ad2b0 0x7fe72406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240ad2b0 0x7fe72406c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52462/0 (socket says 172.21.15.94:52462) 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 -- 172.21.15.94:0/1384766131 learned_addr learned my addr 172.21.15.94:0/1384766131 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe72a352640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7240c8950 0x7fe72414b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729350640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7240c69f0 0x7fe7241ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 -- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7240c69f0 msgr2=0x7fe7241ee0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7240c69f0 0x7fe7241ee0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 -- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7240c8950 msgr2=0x7fe72414b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7240c8950 0x7fe72414b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 -- 172.21.15.94:0/1384766131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7240c2480 con 0x7fe7240ad2b0 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe72a352640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7240c8950 0x7fe72414b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729350640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7240c69f0 0x7fe7241ee0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe729b51640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240ad2b0 0x7fe72406c280 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe720009960 tx=0x7fe7200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe71a7fc640 1 -- 172.21.15.94:0/1384766131 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720040070 con 0x7fe7240ad2b0 2024-03-06T07:21:34.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe71a7fc640 1 -- 172.21.15.94:0/1384766131 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe720039cf0 con 0x7fe7240ad2b0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72414cd10 con 0x7fe7240ad2b0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe71a7fc640 1 -- 172.21.15.94:0/1384766131 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720035040 con 0x7fe7240ad2b0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.474+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72406c7c0 con 0x7fe7240ad2b0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.475+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7240c2650 con 0x7fe7240ad2b0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.476+0000 7fe71a7fc640 1 -- 172.21.15.94:0/1384766131 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe720038020 con 0x7fe7240ad2b0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.476+0000 7fe71a7fc640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.476+0000 7fe71a7fc640 1 -- 172.21.15.94:0/1384766131 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fe7200abd10 con 0x7fe7240ad2b0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.476+0000 7fe729350640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:34.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.477+0000 7fe729350640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe70806b4d0 0x7fe70806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe724154cc0 tx=0x7fe714008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:34.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.480+0000 7fe71a7fc640 1 -- 172.21.15.94:0/1384766131 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe724155130 con 0x7fe7240ad2b0 2024-03-06T07:21:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:34.667+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe72414ca70 con 0x7fe7240ad2b0 2024-03-06T07:21:35.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.531+0000 7fe71a7fc640 1 -- 172.21.15.94:0/1384766131 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v91) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe72007ac90 con 0x7fe7240ad2b0 2024-03-06T07:21:35.533 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:21:35.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.533+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe70806b4d0 msgr2=0x7fe70806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.533+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe70806b4d0 0x7fe70806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe724154cc0 tx=0x7fe714008040 comp rx=0 tx=0).stop 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.533+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240ad2b0 msgr2=0x7fe72406c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.533+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240ad2b0 0x7fe72406c280 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe720009960 tx=0x7fe7200046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 shutdown_connections 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe70806b4d0 0x7fe70806d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7240c8950 0x7fe72414b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7240c69f0 0x7fe7241ee0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 --2- 172.21.15.94:0/1384766131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7240ad2b0 0x7fe72406c280 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 >> 172.21.15.94:0/1384766131 conn(0x7fe7240cae70 msgr2=0x7fe724151760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 shutdown_connections 2024-03-06T07:21:35.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.534+0000 7fe72bdb5640 1 -- 172.21.15.94:0/1384766131 wait complete. 2024-03-06T07:21:35.547 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:21:35.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.647+0000 7f6758c1c640 1 Processor -- start 2024-03-06T07:21:35.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.647+0000 7f6758c1c640 1 -- start start 2024-03-06T07:21:35.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f6758c1c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67540c1aa0 0x7f67541529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f6758c1c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6754155a40 0x7f6754155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f6758c1c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67540c28d0 con 0x7f675414f790 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f6758c1c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67540c2680 con 0x7f6754155a40 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f6758c1c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67540c2bb0 con 0x7f67540c1aa0 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f6752d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6754155a40 0x7f6754155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f675259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f675414f790 0x7f67540c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35154/0 (socket says 172.21.15.94:35154) 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f6752d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6754155a40 0x7f6754155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52470/0 (socket says 172.21.15.94:52470) 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.648+0000 7f675259c640 1 -- 172.21.15.94:0/342935358 learned_addr learned my addr 172.21.15.94:0/342935358 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.649+0000 7f6751d9b640 1 --2- 172.21.15.94:0/342935358 >> v2:172.21.15.145:3300/0 conn(0x7f67540c1aa0 0x7f67541529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.649+0000 7f6752d9d640 1 -- 172.21.15.94:0/342935358 >> v2:172.21.15.145:3300/0 conn(0x7f67540c1aa0 msgr2=0x7f67541529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.649+0000 7f6752d9d640 1 --2- 172.21.15.94:0/342935358 >> v2:172.21.15.145:3300/0 conn(0x7f67540c1aa0 0x7f67541529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.649+0000 7f6752d9d640 1 -- 172.21.15.94:0/342935358 >> v1:172.21.15.94:6789/0 conn(0x7f675414f790 legacy=0x7f67540c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.649+0000 7f675159a640 1 -- 172.21.15.94:0/342935358 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389334089 0 0) 0x7f67540c28d0 con 0x7f675414f790 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.649+0000 7f6752d9d640 1 -- 172.21.15.94:0/342935358 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67540c2e90 con 0x7f6754155a40 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.649+0000 7f6752d9d640 1 --2- 172.21.15.94:0/342935358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754155e50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6754070280 tx=0x7f674402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ad32ed3d2094f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f675159a640 1 -- 172.21.15.94:0/342935358 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744039070 con 0x7f6754155a40 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f675159a640 1 -- 172.21.15.94:0/342935358 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674402cc80 con 0x7f6754155a40 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f675159a640 1 -- 172.21.15.94:0/342935358 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744034660 con 0x7f6754155a40 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 -- 172.21.15.94:0/342935358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 msgr2=0x7f6754155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 --2- 172.21.15.94:0/342935358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754155e50 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6754070280 tx=0x7f674402bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 -- 172.21.15.94:0/342935358 shutdown_connections 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 --2- 172.21.15.94:0/342935358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754155e50 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 --2- 172.21.15.94:0/342935358 >> v2:172.21.15.145:3300/0 conn(0x7f67540c1aa0 0x7f67541529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 -- 172.21.15.94:0/342935358 >> 172.21.15.94:0/342935358 conn(0x7f67540c6710 msgr2=0x7f67540c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 -- 172.21.15.94:0/342935358 shutdown_connections 2024-03-06T07:21:35.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.650+0000 7f6758c1c640 1 -- 172.21.15.94:0/342935358 wait complete. 2024-03-06T07:21:35.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.651+0000 7f6758c1c640 1 Processor -- start 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.651+0000 7f6758c1c640 1 -- start start 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6758c1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67540c1aa0 0x7f6754145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6758c1c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f675414f790 0x7f6754146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6758c1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6758c1c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67540c0d60 con 0x7f6754155a40 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6758c1c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6754156390 con 0x7f675414f790 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6758c1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67540c5e30 con 0x7f67540c1aa0 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754153c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52476/0 (socket says 172.21.15.94:52476) 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 -- 172.21.15.94:0/4118989051 learned_addr learned my addr 172.21.15.94:0/4118989051 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f675259c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67540c1aa0 0x7f6754145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 -- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67540c1aa0 msgr2=0x7f6754145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6751d9b640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f675414f790 0x7f6754146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67540c1aa0 0x7f6754145ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 -- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f675414f790 msgr2=0x7f6754146b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f675414f790 0x7f6754146b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.652+0000 7f6752d9d640 1 -- 172.21.15.94:0/4118989051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67541513b0 con 0x7f6754155a40 2024-03-06T07:21:35.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.653+0000 7f6752d9d640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754153c10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6744037040 tx=0x7f6744004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.653+0000 7f6742ffd640 1 -- 172.21.15.94:0/4118989051 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744039050 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.653+0000 7f6742ffd640 1 -- 172.21.15.94:0/4118989051 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6744045070 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.653+0000 7f6742ffd640 1 -- 172.21.15.94:0/4118989051 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674403e810 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.653+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67541479b0 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.653+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6754151c10 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.654+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675404f330 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.654+0000 7f6742ffd640 1 -- 172.21.15.94:0/4118989051 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f674403e9b0 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.655+0000 7f6742ffd640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f673406b4d0 0x7f673406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.655+0000 7f6742ffd640 1 -- 172.21.15.94:0/4118989051 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f67440ace40 con 0x7f6754155a40 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.655+0000 7f675259c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f673406b4d0 0x7f673406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:35.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.655+0000 7f675259c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f673406b4d0 0x7f673406d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f674c000f60 tx=0x7f674c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:35.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.659+0000 7f6742ffd640 1 -- 172.21.15.94:0/4118989051 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f674407bf10 con 0x7f6754155a40 2024-03-06T07:21:35.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.894+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f67541551b0 con 0x7f6754155a40 2024-03-06T07:21:35.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.894+0000 7f6742ffd640 1 -- 172.21.15.94:0/4118989051 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f674407b810 con 0x7f6754155a40 2024-03-06T07:21:35.897 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:35.897 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:21:35.897 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:21:35.897 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.897 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.898 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:21:35.899 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.900 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:21:35.901 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:21:35.902 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.903 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.904 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.905 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.906 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:21:35.907 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:21:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:21:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:21:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:21:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:21:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:21:35.911 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f673406b4d0 msgr2=0x7f673406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f673406b4d0 0x7f673406d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f674c000f60 tx=0x7f674c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 msgr2=0x7f6754153c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754153c10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6744037040 tx=0x7f6744004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 shutdown_connections 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f673406b4d0 0x7f673406d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6754155a40 0x7f6754153c10 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f675414f790 0x7f6754146b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 --2- 172.21.15.94:0/4118989051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67540c1aa0 0x7f6754145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 >> 172.21.15.94:0/4118989051 conn(0x7f67540c6710 msgr2=0x7f67540c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 shutdown_connections 2024-03-06T07:21:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:35.896+0000 7f6758c1c640 1 -- 172.21.15.94:0/4118989051 wait complete. 2024-03-06T07:21:35.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:36.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.012+0000 7f8d0f8c3640 1 Processor -- start 2024-03-06T07:21:36.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.013+0000 7f8d0f8c3640 1 -- start start 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.013+0000 7f8d0f8c3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8d0814f790 0x7f8d081529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.013+0000 7f8d0f8c3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d080c28d0 con 0x7f8d08155a40 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.013+0000 7f8d0f8c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d080c2bb0 con 0x7f8d080c1aa0 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.013+0000 7f8d0f8c3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d080c2680 con 0x7f8d0814f790 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.014+0000 7f8d0d65f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8d0814f790 0x7f8d081529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.014+0000 7f8d0de60640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8d08155a40 0x7f8d08155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35166/0 (socket says 172.21.15.94:35166) 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.014+0000 7f8d0de60640 1 -- 172.21.15.94:0/2695540795 learned_addr learned my addr 172.21.15.94:0/2695540795 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.014+0000 7f8d0d65f640 1 -- 172.21.15.94:0/2695540795 >> v1:172.21.15.145:6789/0 conn(0x7f8d080c1aa0 legacy=0x7f8d080c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.014+0000 7f8d0d65f640 1 -- 172.21.15.94:0/2695540795 >> v1:172.21.15.94:6789/0 conn(0x7f8d08155a40 legacy=0x7f8d08155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.014+0000 7f8d0d65f640 1 -- 172.21.15.94:0/2695540795 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d080c2e90 con 0x7f8d0814f790 2024-03-06T07:21:36.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.014+0000 7f8cfffff640 1 -- 172.21.15.94:0/2695540795 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36646837 0 0) 0x7f8d080c28d0 con 0x7f8d08155a40 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0d65f640 1 --2- 172.21.15.94:0/2695540795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081529d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8d080c2800 tx=0x7f8d0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6c79b8b9c9e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8cfffff640 1 -- 172.21.15.94:0/2695540795 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04035020 con 0x7f8d0814f790 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8cfffff640 1 -- 172.21.15.94:0/2695540795 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04031830 con 0x7f8d0814f790 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8cfffff640 1 -- 172.21.15.94:0/2695540795 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04031c00 con 0x7f8d0814f790 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/2695540795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 msgr2=0x7f8d081529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/2695540795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081529d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8d080c2800 tx=0x7f8d0402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/2695540795 shutdown_connections 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/2695540795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081529d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/2695540795 >> 172.21.15.94:0/2695540795 conn(0x7f8d080c6710 msgr2=0x7f8d080c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/2695540795 shutdown_connections 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.015+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/2695540795 wait complete. 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.016+0000 7f8d0f8c3640 1 Processor -- start 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0f8c3640 1 -- start start 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0f8c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d080c1aa0 0x7f8d08150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0f8c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0f8c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d08155a40 0x7f8d080bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0f8c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d080c5e30 con 0x7f8d0814f790 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0f8c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d080c0d60 con 0x7f8d080c1aa0 2024-03-06T07:21:36.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0f8c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d080c5fb0 con 0x7f8d08155a40 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0ce5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0ce5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081ee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52508/0 (socket says 172.21.15.94:52508) 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0ce5e640 1 -- 172.21.15.94:0/1302650176 learned_addr learned my addr 172.21.15.94:0/1302650176 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0d65f640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d080c1aa0 0x7f8d08150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.017+0000 7f8d0de60640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d08155a40 0x7f8d080bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0ce5e640 1 -- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d08155a40 msgr2=0x7f8d080bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0ce5e640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d08155a40 0x7f8d080bfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0ce5e640 1 -- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d080c1aa0 msgr2=0x7f8d08150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0ce5e640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d080c1aa0 0x7f8d08150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0ce5e640 1 -- 172.21.15.94:0/1302650176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08154f60 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0ce5e640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081ee340 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8cf800c400 tx=0x7f8cf800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8cfdffb640 1 -- 172.21.15.94:0/1302650176 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8011790 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8cfdffb640 1 -- 172.21.15.94:0/1302650176 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf8017070 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d08155240 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8cfdffb640 1 -- 172.21.15.94:0/1302650176 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8011bb0 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.018+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d08153b20 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.019+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0804f330 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.019+0000 7f8cfdffb640 1 -- 172.21.15.94:0/1302650176 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cf801c460 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.020+0000 7f8cfdffb640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8cec06b4d0 0x7f8cec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.020+0000 7f8cfdffb640 1 -- 172.21.15.94:0/1302650176 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8cf8089d00 con 0x7f8d0814f790 2024-03-06T07:21:36.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.020+0000 7f8d0d65f640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8cec06b4d0 0x7f8cec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.021+0000 7f8d0d65f640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8cec06b4d0 0x7f8cec06d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8d04036000 tx=0x7f8d0403e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.024+0000 7f8cfdffb640 1 -- 172.21.15.94:0/1302650176 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cf8058780 con 0x7f8d0814f790 2024-03-06T07:21:36.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.194+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d080c1f40 con 0x7f8d0814f790 2024-03-06T07:21:36.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.195+0000 7f8cfdffb640 1 -- 172.21.15.94:0/1302650176 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13165 (secure 0 0 0) 0x7f8cf805c820 con 0x7f8d0814f790 2024-03-06T07:21:36.200 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:36.201 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":91,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:35.527493+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8cec06b4d0 msgr2=0x7f8cec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8cec06b4d0 0x7f8cec06d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8d04036000 tx=0x7f8d0403e040 comp rx=0 tx=0).stop 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 msgr2=0x7f8d081ee340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081ee340 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8cf800c400 tx=0x7f8cf800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 shutdown_connections 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8cec06b4d0 0x7f8cec06d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d08155a40 0x7f8d080bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d0814f790 0x7f8d081ee340 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 --2- 172.21.15.94:0/1302650176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d080c1aa0 0x7f8d08150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 >> 172.21.15.94:0/1302650176 conn(0x7f8d080c6710 msgr2=0x7f8d0806d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 shutdown_connections 2024-03-06T07:21:36.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.197+0000 7f8d0f8c3640 1 -- 172.21.15.94:0/1302650176 wait complete. 2024-03-06T07:21:36.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:36.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.312+0000 7fb0a493a640 1 Processor -- start 2024-03-06T07:21:36.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.312+0000 7fb0a493a640 1 -- start start 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.312+0000 7fb0a493a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb0a00c5990 0x7fb0a00c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb0a493a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a006c900 0x7fb0a00a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb0a493a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a00c3ef0 con 0x7fb0a00ab170 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb0a493a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a01556d0 con 0x7fb0a00c5990 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb0a493a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a01559b0 con 0x7fb0a006c900 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb0a00c5990 0x7fb0a00c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09dd9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb0a00ab170 0x7fb0a006c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35180/0 (socket says 172.21.15.94:35180) 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09dd9b640 1 -- 172.21.15.94:0/2172228159 learned_addr learned my addr 172.21.15.94:0/2172228159 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09ed9d640 1 --2- 172.21.15.94:0/2172228159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a006c900 0x7fb0a00a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09e59c640 1 -- 172.21.15.94:0/2172228159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a006c900 msgr2=0x7fb0a00a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09e59c640 1 --2- 172.21.15.94:0/2172228159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a006c900 0x7fb0a00a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09e59c640 1 -- 172.21.15.94:0/2172228159 >> v1:172.21.15.94:6789/0 conn(0x7fb0a00ab170 legacy=0x7fb0a006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09e59c640 1 -- 172.21.15.94:0/2172228159 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0155c90 con 0x7fb0a00c5990 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.313+0000 7fb09d59a640 1 -- 172.21.15.94:0/2172228159 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552856525 0 0) 0x7fb0a00c3ef0 con 0x7fb0a00ab170 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb09e59c640 1 --2- 172.21.15.94:0/2172228159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a00c5990 0x7fb0a00c0ab0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb0900097a0 tx=0x7fb09002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5d91e7d4ba7f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb09d59a640 1 -- 172.21.15.94:0/2172228159 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090035020 con 0x7fb0a00c5990 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb09d59a640 1 -- 172.21.15.94:0/2172228159 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090031830 con 0x7fb0a00c5990 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb09d59a640 1 -- 172.21.15.94:0/2172228159 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090031c00 con 0x7fb0a00c5990 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb0a493a640 1 -- 172.21.15.94:0/2172228159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a00c5990 msgr2=0x7fb0a00c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb0a493a640 1 --2- 172.21.15.94:0/2172228159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a00c5990 0x7fb0a00c0ab0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb0900097a0 tx=0x7fb09002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb0a493a640 1 -- 172.21.15.94:0/2172228159 shutdown_connections 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb0a493a640 1 --2- 172.21.15.94:0/2172228159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a006c900 0x7fb0a00a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb0a493a640 1 --2- 172.21.15.94:0/2172228159 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a00c5990 0x7fb0a00c0ab0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb0a493a640 1 -- 172.21.15.94:0/2172228159 >> 172.21.15.94:0/2172228159 conn(0x7fb0a00c6710 msgr2=0x7fb0a00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.314+0000 7fb0a493a640 1 -- 172.21.15.94:0/2172228159 shutdown_connections 2024-03-06T07:21:36.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.315+0000 7fb0a493a640 1 -- 172.21.15.94:0/2172228159 wait complete. 2024-03-06T07:21:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 Processor -- start 2024-03-06T07:21:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 -- start start 2024-03-06T07:21:36.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a006c900 0x7fb0a0071b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0a00ab170 0x7fb0a00a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a00c5990 0x7fb0a01e9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0156610 con 0x7fb0a006c900 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0156490 con 0x7fb0a00ab170 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb0a493a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00c3af0 con 0x7fb0a00c5990 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.316+0000 7fb09e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a006c900 0x7fb0a0071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a006c900 0x7fb0a0071b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52528/0 (socket says 172.21.15.94:52528) 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 -- 172.21.15.94:0/1349133064 learned_addr learned my addr 172.21.15.94:0/1349133064 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09dd9b640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0a00ab170 0x7fb0a00a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09ed9d640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a00c5990 0x7fb0a01e9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 -- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a00c5990 msgr2=0x7fb0a01e9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a00c5990 0x7fb0a01e9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 -- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0a00ab170 msgr2=0x7fb0a00a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0a00ab170 0x7fb0a00a8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 -- 172.21.15.94:0/1349133064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a01e30f0 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb09e59c640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a006c900 0x7fb0a0071b30 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb0900097a0 tx=0x7fb0900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb08effd640 1 -- 172.21.15.94:0/1349133064 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090035070 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb08effd640 1 -- 172.21.15.94:0/1349133064 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090004230 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a00c4610 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb08effd640 1 -- 172.21.15.94:0/1349133064 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09003a6f0 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.317+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a01e9b90 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.318+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a004f330 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.319+0000 7fb08effd640 1 -- 172.21.15.94:0/1349133064 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb090038020 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.319+0000 7fb08effd640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb08006b4d0 0x7fb08006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.319+0000 7fb08effd640 1 -- 172.21.15.94:0/1349133064 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb0900abcc0 con 0x7fb0a006c900 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.319+0000 7fb09dd9b640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb08006b4d0 0x7fb08006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.320+0000 7fb09dd9b640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb08006b4d0 0x7fb08006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb094007fd0 tx=0x7fb094009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.323+0000 7fb08effd640 1 -- 172.21.15.94:0/1349133064 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb090040070 con 0x7fb0a006c900 2024-03-06T07:21:36.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.495+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0a014c9a0 con 0x7fb0a006c900 2024-03-06T07:21:36.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.495+0000 7fb08effd640 1 -- 172.21.15.94:0/1349133064 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb09007ad90 con 0x7fb0a006c900 2024-03-06T07:21:36.497 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb08006b4d0 msgr2=0x7fb08006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb08006b4d0 0x7fb08006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb094007fd0 tx=0x7fb094009040 comp rx=0 tx=0).stop 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a006c900 msgr2=0x7fb0a0071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a006c900 0x7fb0a0071b30 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb0900097a0 tx=0x7fb0900046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 shutdown_connections 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb08006b4d0 0x7fb08006d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0a00c5990 0x7fb0a01e9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0a00ab170 0x7fb0a00a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 --2- 172.21.15.94:0/1349133064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0a006c900 0x7fb0a0071b30 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 >> 172.21.15.94:0/1349133064 conn(0x7fb0a00c6710 msgr2=0x7fb0a0154db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 shutdown_connections 2024-03-06T07:21:36.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.497+0000 7fb0a493a640 1 -- 172.21.15.94:0/1349133064 wait complete. 2024-03-06T07:21:36.510 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:21:36.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.608+0000 7fd5530cf640 1 Processor -- start 2024-03-06T07:21:36.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.608+0000 7fd5530cf640 1 -- start start 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd5530cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c14f790 0x7fd54c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd5530cf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd54c0c1aa0 0x7fd54c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd5530cf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0c28d0 con 0x7fd54c0c5c60 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd5530cf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c0c2680 con 0x7fd54c0c1aa0 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd5530cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0c2bb0 con 0x7fd54c14f790 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd54bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd54c0c1aa0 0x7fd54c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd55166c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd54c0c5c60 0x7fd54c155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35186/0 (socket says 172.21.15.94:35186) 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd55166c640 1 -- 172.21.15.94:0/761377021 learned_addr learned my addr 172.21.15.94:0/761377021 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd54bfff640 1 -- 172.21.15.94:0/761377021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c14f790 msgr2=0x7fd54c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd54bfff640 1 --2- 172.21.15.94:0/761377021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c14f790 0x7fd54c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd54bfff640 1 -- 172.21.15.94:0/761377021 >> v1:172.21.15.94:6789/0 conn(0x7fd54c0c5c60 legacy=0x7fd54c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd54bfff640 1 -- 172.21.15.94:0/761377021 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0c2e90 con 0x7fd54c0c1aa0 2024-03-06T07:21:36.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.609+0000 7fd54b7fe640 1 -- 172.21.15.94:0/761377021 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962116529 0 0) 0x7fd54c0c28d0 con 0x7fd54c0c5c60 2024-03-06T07:21:36.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.610+0000 7fd54bfff640 1 --2- 172.21.15.94:0/761377021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c0c1aa0 0x7fd54c0c5720 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd5400097a0 tx=0x7fd54002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4599c3b2cd983716 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.610+0000 7fd54b7fe640 1 -- 172.21.15.94:0/761377021 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540035020 con 0x7fd54c0c1aa0 2024-03-06T07:21:36.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.610+0000 7fd54b7fe640 1 -- 172.21.15.94:0/761377021 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd540031830 con 0x7fd54c0c1aa0 2024-03-06T07:21:36.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.610+0000 7fd54b7fe640 1 -- 172.21.15.94:0/761377021 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540031c00 con 0x7fd54c0c1aa0 2024-03-06T07:21:36.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 -- 172.21.15.94:0/761377021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c0c1aa0 msgr2=0x7fd54c0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 --2- 172.21.15.94:0/761377021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c0c1aa0 0x7fd54c0c5720 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd5400097a0 tx=0x7fd54002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 -- 172.21.15.94:0/761377021 shutdown_connections 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 --2- 172.21.15.94:0/761377021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c0c1aa0 0x7fd54c0c5720 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 --2- 172.21.15.94:0/761377021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c14f790 0x7fd54c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 -- 172.21.15.94:0/761377021 >> 172.21.15.94:0/761377021 conn(0x7fd54c0c6710 msgr2=0x7fd54c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 -- 172.21.15.94:0/761377021 shutdown_connections 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.611+0000 7fd5530cf640 1 -- 172.21.15.94:0/761377021 wait complete. 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.612+0000 7fd5530cf640 1 Processor -- start 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.612+0000 7fd5530cf640 1 -- start start 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd54c0c1aa0 0x7fd54c1e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c0c5c60 0x7fd54c14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c14f790 0x7fd54c150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c06cc00 con 0x7fd54c14f790 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0c0d60 con 0x7fd54c0c1aa0 2024-03-06T07:21:36.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c06cd80 con 0x7fd54c0c5c60 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c14f790 0x7fd54c150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c14f790 0x7fd54c150300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52542/0 (socket says 172.21.15.94:52542) 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 -- 172.21.15.94:0/4145759589 learned_addr learned my addr 172.21.15.94:0/4145759589 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 -- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c0c5c60 msgr2=0x7fd54c14fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c0c5c60 0x7fd54c14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 -- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd54c0c1aa0 msgr2=0x7fd54c1e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd550e6b640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd54c0c1aa0 0x7fd54c1e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd54c0c1aa0 0x7fd54c1e0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 -- 172.21.15.94:0/4145759589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c1543d0 con 0x7fd54c14f790 2024-03-06T07:21:36.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd55166c640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c14f790 0x7fd54c150300 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c810 tx=0x7fd54400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5497fa640 1 -- 172.21.15.94:0/4145759589 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544018070 con 0x7fd54c14f790 2024-03-06T07:21:36.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5497fa640 1 -- 172.21.15.94:0/4145759589 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54400a640 con 0x7fd54c14f790 2024-03-06T07:21:36.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c150840 con 0x7fd54c14f790 2024-03-06T07:21:36.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5497fa640 1 -- 172.21.15.94:0/4145759589 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54400aa30 con 0x7fd54c14f790 2024-03-06T07:21:36.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.613+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c14dd80 con 0x7fd54c14f790 2024-03-06T07:21:36.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.614+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c04f330 con 0x7fd54c14f790 2024-03-06T07:21:36.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.615+0000 7fd5497fa640 1 -- 172.21.15.94:0/4145759589 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd54401b070 con 0x7fd54c14f790 2024-03-06T07:21:36.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.615+0000 7fd5497fa640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd53806b500 0x7fd53806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.615+0000 7fd5497fa640 1 -- 172.21.15.94:0/4145759589 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd5440895d0 con 0x7fd54c14f790 2024-03-06T07:21:36.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.615+0000 7fd550e6b640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd53806b500 0x7fd53806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.616+0000 7fd550e6b640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd53806b500 0x7fd53806d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00a770 tx=0x7fd53c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.620+0000 7fd5497fa640 1 -- 172.21.15.94:0/4145759589 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5440586a0 con 0x7fd54c14f790 2024-03-06T07:21:36.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.794+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd54c0c1f40 con 0x7fd54c14f790 2024-03-06T07:21:36.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.794+0000 7fd5497fa640 1 -- 172.21.15.94:0/4145759589 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v91) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd544057fa0 con 0x7fd54c14f790 2024-03-06T07:21:36.796 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:36.796 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:21:36.796 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:21:36.796 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:21:36.796 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:21:36.797 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:21:36.797 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:21:36.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd53806b500 msgr2=0x7fd53806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd53806b500 0x7fd53806d9c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00a770 tx=0x7fd53c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:36.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c14f790 msgr2=0x7fd54c150300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c14f790 0x7fd54c150300 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c810 tx=0x7fd54400ccd0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 shutdown_connections 2024-03-06T07:21:36.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd53806b500 0x7fd53806d9c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd54c14f790 0x7fd54c150300 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd54c0c5c60 0x7fd54c14fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 --2- 172.21.15.94:0/4145759589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd54c0c1aa0 0x7fd54c1e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 >> 172.21.15.94:0/4145759589 conn(0x7fd54c0c6710 msgr2=0x7fd54c0c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 shutdown_connections 2024-03-06T07:21:36.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.796+0000 7fd5530cf640 1 -- 172.21.15.94:0/4145759589 wait complete. 2024-03-06T07:21:36.812 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:21:36.812 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:21:36.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.914+0000 7f44c6fac640 1 Processor -- start 2024-03-06T07:21:36.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.914+0000 7f44c6fac640 1 -- start start 2024-03-06T07:21:36.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.914+0000 7f44c6fac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 0x7f44c0151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.914+0000 7f44c6fac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f44c00a9830 0x7f44c014ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.914+0000 7f44c6fac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c006e850 con 0x7f44c014f490 2024-03-06T07:21:36.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.914+0000 7f44c6fac640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c006d2b0 con 0x7f44c00a9830 2024-03-06T07:21:36.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.914+0000 7f44c6fac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c006d590 con 0x7f44c00c22b0 2024-03-06T07:21:36.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c5549640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f44c014f490 0x7f44c0153390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35192/0 (socket says 172.21.15.94:35192) 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c5549640 1 -- 172.21.15.94:0/3230629579 learned_addr learned my addr 172.21.15.94:0/3230629579 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bffff640 1 --2- 172.21.15.94:0/3230629579 >> v2:172.21.15.145:3300/0 conn(0x7f44c00a9830 0x7f44c014ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c4d48640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 0x7f44c0151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260431060 0 0) 0x7f44b8001490 con 0x7f44c014f490 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b0003520 con 0x7f44c014f490 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3914057041 0 0) 0x7f44b0003520 con 0x7f44c014f490 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 msgr2=0x7f44c0151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 0x7f44c0151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 >> v2:172.21.15.145:3300/0 conn(0x7f44c00a9830 msgr2=0x7f44c014ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 --2- 172.21.15.94:0/3230629579 >> v2:172.21.15.145:3300/0 conn(0x7f44c00a9830 0x7f44c014ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c006d870 con 0x7f44c014f490 2024-03-06T07:21:36.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f44b8001bc0 con 0x7f44c014f490 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 >> v1:172.21.15.94:6789/0 conn(0x7f44c014f490 legacy=0x7f44c0153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b00039b0 0x7f44b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 0x7f44b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44b0004d40 0x7f44b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c006d2b0 con 0x7f44b0004d40 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c006d590 con 0x7f44b00039b0 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b000b9c0 con 0x7f44c00c22b0 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c5549640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44b0004d40 0x7f44b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c4d48640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b00039b0 0x7f44b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44bffff640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 0x7f44b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c5549640 1 -- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 msgr2=0x7f44b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c5549640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 0x7f44b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.915+0000 7f44c5549640 1 -- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b00039b0 msgr2=0x7f44b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.916+0000 7f44c5549640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b00039b0 0x7f44b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.916+0000 7f44c5549640 1 -- 172.21.15.94:0/3230629579 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0003520 con 0x7f44b0004d40 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.916+0000 7f44bffff640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00c22b0 0x7f44b0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.916+0000 7f44c5549640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44b0004d40 0x7f44b00051c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f44b800aa50 tx=0x7f44b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fc5462181f4814 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.916+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8038070 con 0x7f44b0004d40 2024-03-06T07:21:36.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.916+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b8033650 con 0x7f44b0004d40 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.916+0000 7f44bf7fe640 1 -- 172.21.15.94:0/3230629579 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b800a670 con 0x7f44b0004d40 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.917+0000 7f44c6fac640 1 -- 172.21.15.94:0/3230629579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44b0004d40 msgr2=0x7f44b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.917+0000 7f44c6fac640 1 --2- 172.21.15.94:0/3230629579 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44b0004d40 0x7f44b00051c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f44b800aa50 tx=0x7f44b0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.917+0000 7f44c5549640 1 -- 172.21.15.94:0/3230629579 reap_dead start 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.917+0000 7f44c6fac640 1 -- 172.21.15.94:0/3230629579 shutdown_connections 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.917+0000 7f44c6fac640 1 -- 172.21.15.94:0/3230629579 >> 172.21.15.94:0/3230629579 conn(0x7f44c00c65b0 msgr2=0x7f44c00c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.917+0000 7f44c6fac640 1 -- 172.21.15.94:0/3230629579 shutdown_connections 2024-03-06T07:21:36.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.917+0000 7f44c6fac640 1 -- 172.21.15.94:0/3230629579 wait complete. 2024-03-06T07:21:36.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.918+0000 7f44c6fac640 1 Processor -- start 2024-03-06T07:21:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.918+0000 7f44c6fac640 1 -- start start 2024-03-06T07:21:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c6fac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b0004d40 0x7f44c00a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c6fac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00aa300 0x7f44c0141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c6fac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0142020 0x7f44c0144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c6fac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00c2810 con 0x7f44c0142020 2024-03-06T07:21:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c6fac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c00c2690 con 0x7f44b0004d40 2024-03-06T07:21:36.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c6fac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c0151e10 con 0x7f44c00aa300 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0142020 0x7f44c0144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0142020 0x7f44c0144f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52554/0 (socket says 172.21.15.94:52554) 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 -- 172.21.15.94:0/1430833116 learned_addr learned my addr 172.21.15.94:0/1430833116 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c4d48640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b0004d40 0x7f44c00a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44bffff640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00aa300 0x7f44c0141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 -- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00aa300 msgr2=0x7f44c0141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00aa300 0x7f44c0141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 -- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b0004d40 msgr2=0x7f44c00a9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b0004d40 0x7f44c00a9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 -- 172.21.15.94:0/1430833116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c01ec760 con 0x7f44c0142020 2024-03-06T07:21:36.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c4d48640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b0004d40 0x7f44c00a9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:36.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c5549640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0142020 0x7f44c0144f30 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f44b8036040 tx=0x7f44c0070db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44bd7fa640 1 -- 172.21.15.94:0/1430833116 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8044070 con 0x7f44c0142020 2024-03-06T07:21:36.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44bd7fa640 1 -- 172.21.15.94:0/1430833116 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b8004d60 con 0x7f44c0142020 2024-03-06T07:21:36.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.919+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c01eccb0 con 0x7f44c0142020 2024-03-06T07:21:36.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.920+0000 7f44bd7fa640 1 -- 172.21.15.94:0/1430833116 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8038070 con 0x7f44c0142020 2024-03-06T07:21:36.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.920+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c00aba90 con 0x7f44c0142020 2024-03-06T07:21:36.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.921+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c004f330 con 0x7f44c0142020 2024-03-06T07:21:36.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.921+0000 7f44bd7fa640 1 -- 172.21.15.94:0/1430833116 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44b8047070 con 0x7f44c0142020 2024-03-06T07:21:36.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.921+0000 7f44bd7fa640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44a406b500 0x7f44a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.921+0000 7f44bd7fa640 1 -- 172.21.15.94:0/1430833116 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f44b80ab720 con 0x7f44c0142020 2024-03-06T07:21:36.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.922+0000 7f44c4d48640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44a406b500 0x7f44a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.922+0000 7f44c4d48640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44a406b500 0x7f44a406d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f44c00c64d0 tx=0x7f44b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:36.926+0000 7f44bd7fa640 1 -- 172.21.15.94:0/1430833116 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44c006e850 con 0x7f44c0142020 2024-03-06T07:21:37.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.097+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f44c0146d50 con 0x7f44c0142020 2024-03-06T07:21:37.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.536+0000 7f44bd7fa640 1 -- 172.21.15.94:0/1430833116 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v92) v1 ==== 138+0+0 (secure 0 0 0) 0x7f44b807a1a0 con 0x7f44c0142020 2024-03-06T07:21:37.540 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:21:37.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44a406b500 msgr2=0x7f44a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44a406b500 0x7f44a406d9c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f44c00c64d0 tx=0x7f44b4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:37.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0142020 msgr2=0x7f44c0144f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0142020 0x7f44c0144f30 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f44b8036040 tx=0x7f44c0070db0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 shutdown_connections 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44a406b500 0x7f44a406d9c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0142020 0x7f44c0144f30 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44c00aa300 0x7f44c0141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 --2- 172.21.15.94:0/1430833116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44b0004d40 0x7f44c00a9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 >> 172.21.15.94:0/1430833116 conn(0x7f44c00c65b0 msgr2=0x7f44c0150e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 shutdown_connections 2024-03-06T07:21:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.539+0000 7f44c6fac640 1 -- 172.21.15.94:0/1430833116 wait complete. 2024-03-06T07:21:37.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:37.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c3da37640 1 Processor -- start 2024-03-06T07:21:37.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c3da37640 1 -- start start 2024-03-06T07:21:37.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c3da37640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5c380c2410 0x7f5c380c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c3da37640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3806e790 con 0x7f5c380c3d10 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c3da37640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3806ea30 con 0x7f5c380c33f0 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c3da37640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3806ed10 con 0x7f5c380c2410 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c377fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5c380c3d10 0x7f5c3815c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35208/0 (socket says 172.21.15.94:35208) 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.666+0000 7f5c377fe640 1 -- 172.21.15.94:0/3411942705 learned_addr learned my addr 172.21.15.94:0/3411942705 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c367fc640 1 --2- 172.21.15.94:0/3411942705 >> v2:172.21.15.145:3300/0 conn(0x7f5c380c2410 0x7f5c380c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963616864 0 0) 0x7f5c3806e790 con 0x7f5c380c3d10 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c20003520 con 0x7f5c380c3d10 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4227371041 0 0) 0x7f5c20003520 con 0x7f5c380c3d10 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 >> v2:172.21.15.145:3300/0 conn(0x7f5c380c2410 msgr2=0x7f5c380c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 --2- 172.21.15.94:0/3411942705 >> v2:172.21.15.145:3300/0 conn(0x7f5c380c2410 0x7f5c380c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 >> v1:172.21.15.145:6789/0 conn(0x7f5c380c33f0 legacy=0x7f5c380c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c38152bb0 con 0x7f5c380c3d10 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f5c300019f0 con 0x7f5c380c3d10 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 >> v1:172.21.15.94:6789/0 conn(0x7f5c380c3d10 legacy=0x7f5c3815c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200039b0 0x7f5c20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200052e0 0x7f5c2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3806ed10 con 0x7f5c20004800 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2000c1b0 con 0x7f5c200039b0 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2000c3b0 con 0x7f5c200052e0 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c367fc640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c36ffd640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200039b0 0x7f5c20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c377fe640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200052e0 0x7f5c2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c367fc640 1 -- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200052e0 msgr2=0x7f5c2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c367fc640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c200052e0 0x7f5c2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c367fc640 1 -- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200039b0 msgr2=0x7f5c20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c367fc640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200039b0 0x7f5c20003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c367fc640 1 -- 172.21.15.94:0/3411942705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c20003520 con 0x7f5c20004800 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c377fe640 1 -- 172.21.15.94:0/3411942705 reap_dead start 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.667+0000 7f5c36ffd640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c200039b0 0x7f5c20003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.668+0000 7f5c367fc640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c20004c80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009a90 tx=0x7f5c20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a73b8f9fd40c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.668+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c039070 con 0x7f5c20004800 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.668+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c02c680 con 0x7f5c20004800 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.668+0000 7f5c35ffb640 1 -- 172.21.15.94:0/3411942705 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c009690 con 0x7f5c20004800 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.669+0000 7f5c3da37640 1 -- 172.21.15.94:0/3411942705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 msgr2=0x7f5c20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.669+0000 7f5c3da37640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c20004c80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009a90 tx=0x7f5c20005830 comp rx=0 tx=0).stop 2024-03-06T07:21:37.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.669+0000 7f5c3da37640 1 -- 172.21.15.94:0/3411942705 shutdown_connections 2024-03-06T07:21:37.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.669+0000 7f5c3da37640 1 --2- 172.21.15.94:0/3411942705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c20004c80 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.669+0000 7f5c3da37640 1 -- 172.21.15.94:0/3411942705 >> 172.21.15.94:0/3411942705 conn(0x7f5c380a9b00 msgr2=0x7f5c380c4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:37.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.669+0000 7f5c3da37640 1 -- 172.21.15.94:0/3411942705 shutdown_connections 2024-03-06T07:21:37.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.669+0000 7f5c3da37640 1 -- 172.21.15.94:0/3411942705 wait complete. 2024-03-06T07:21:37.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.670+0000 7f5c3da37640 1 Processor -- start 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.670+0000 7f5c3da37640 1 -- start start 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c3da37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c38145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c3da37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c38145bf0 0x7f5c380a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c3da37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c380a7b20 0x7f5c38153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c3da37640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c38152960 con 0x7f5c20004800 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c3da37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380c2e60 con 0x7f5c38145bf0 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c3da37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380c2fe0 con 0x7f5c380a7b20 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c38145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c38145440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52582/0 (socket says 172.21.15.94:52582) 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 -- 172.21.15.94:0/1311811150 learned_addr learned my addr 172.21.15.94:0/1311811150 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:37.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c367fc640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c38145bf0 0x7f5c380a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c377fe640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c380a7b20 0x7f5c38153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 -- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c380a7b20 msgr2=0x7f5c38153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c380a7b20 0x7f5c38153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 -- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c38145bf0 msgr2=0x7f5c380a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c38145bf0 0x7f5c380a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 -- 172.21.15.94:0/1311811150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3814f990 con 0x7f5c20004800 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c377fe640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c380a7b20 0x7f5c38153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c36ffd640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c38145440 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800e4e0 tx=0x7f5c2800e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.671+0000 7f5c1ffff640 1 -- 172.21.15.94:0/1311811150 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280052e0 con 0x7f5c20004800 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.672+0000 7f5c1ffff640 1 -- 172.21.15.94:0/1311811150 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c28007020 con 0x7f5c20004800 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.672+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c38156160 con 0x7f5c20004800 2024-03-06T07:21:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.672+0000 7f5c1ffff640 1 -- 172.21.15.94:0/1311811150 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28007210 con 0x7f5c20004800 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.672+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c38145980 con 0x7f5c20004800 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.673+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3804f330 con 0x7f5c20004800 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.673+0000 7f5c1ffff640 1 -- 172.21.15.94:0/1311811150 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c2801a020 con 0x7f5c20004800 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.673+0000 7f5c1ffff640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c2406b500 0x7f5c2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.674+0000 7f5c1ffff640 1 -- 172.21.15.94:0/1311811150 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f5c280895a0 con 0x7f5c20004800 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.674+0000 7f5c367fc640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c2406b500 0x7f5c2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.674+0000 7f5c367fc640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c2406b500 0x7f5c2406d9c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c036000 tx=0x7f5c2c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.678+0000 7f5c1ffff640 1 -- 172.21.15.94:0/1311811150 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c28058500 con 0x7f5c20004800 2024-03-06T07:21:37.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:37.850+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5c380c21f0 con 0x7f5c20004800 2024-03-06T07:21:38.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.543+0000 7f5c1ffff640 1 -- 172.21.15.94:0/1311811150 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v93) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5c28057e00 con 0x7f5c20004800 2024-03-06T07:21:38.546 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:21:38.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.547+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c2406b500 msgr2=0x7f5c2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:38.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.547+0000 7f5c3da37640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c2406b500 0x7f5c2406d9c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c036000 tx=0x7f5c2c030000 comp rx=0 tx=0).stop 2024-03-06T07:21:38.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.547+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 msgr2=0x7f5c38145440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:38.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.547+0000 7f5c3da37640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c38145440 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800e4e0 tx=0x7f5c2800e9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.553+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 shutdown_connections 2024-03-06T07:21:38.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.553+0000 7f5c3da37640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c2406b500 0x7f5c2406d9c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.553+0000 7f5c3da37640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c380a7b20 0x7f5c38153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.553+0000 7f5c3da37640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c38145bf0 0x7f5c380a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.553+0000 7f5c3da37640 1 --2- 172.21.15.94:0/1311811150 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c20004800 0x7f5c38145440 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.553+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 >> 172.21.15.94:0/1311811150 conn(0x7f5c380a9b00 msgr2=0x7f5c38152060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:38.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.554+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 shutdown_connections 2024-03-06T07:21:38.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.556+0000 7f5c3da37640 1 -- 172.21.15.94:0/1311811150 wait complete. 2024-03-06T07:21:38.576 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:21:38.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2afb878640 1 Processor -- start 2024-03-06T07:21:38.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2afb878640 1 -- start start 2024-03-06T07:21:38.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2afb878640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2af4150680 0x7f2af40c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2afb878640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af4153b70 con 0x7f2af40c2b10 2024-03-06T07:21:38.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2afb878640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af40c1140 con 0x7f2af40c26b0 2024-03-06T07:21:38.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2afb878640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af40c1420 con 0x7f2af4150680 2024-03-06T07:21:38.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2af9e15640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2af40c2b10 0x7f2af40a87f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54248/0 (socket says 172.21.15.94:54248) 2024-03-06T07:21:38.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2af9e15640 1 -- 172.21.15.94:0/4046147868 learned_addr learned my addr 172.21.15.94:0/4046147868 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812263119 0 0) 0x7f2af4153b70 con 0x7f2af40c2b10 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae0003520 con 0x7f2af40c2b10 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2af9614640 1 --2- 172.21.15.94:0/4046147868 >> v2:172.21.15.145:3300/0 conn(0x7f2af4150680 0x7f2af40c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2405280040 0 0) 0x7f2ae0003520 con 0x7f2af40c2b10 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 >> v2:172.21.15.145:3300/0 conn(0x7f2af4150680 msgr2=0x7f2af40c1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 --2- 172.21.15.94:0/4046147868 >> v2:172.21.15.145:3300/0 conn(0x7f2af4150680 0x7f2af40c1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 >> v1:172.21.15.145:6789/0 conn(0x7f2af40c26b0 legacy=0x7f2af41530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af40c1700 con 0x7f2af40c2b10 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f2aec001a10 con 0x7f2af40c2b10 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2af9614640 1 --2- 172.21.15.94:0/4046147868 >> v2:172.21.15.145:3300/0 conn(0x7f2af4150680 0x7f2af40c1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 >> v1:172.21.15.94:6789/0 conn(0x7f2af40c2b10 legacy=0x7f2af40a87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae0003a00 0x7f2ae0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ae0004800 0x7f2ae0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0005210 0x7f2ae000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af40c1420 con 0x7f2ae0003a00 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae000c160 con 0x7f2ae0004800 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae000c360 con 0x7f2ae0005210 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2af9614640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae0003a00 0x7f2ae0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2af8e13640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ae0004800 0x7f2ae0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.681+0000 7f2af9e15640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0005210 0x7f2ae000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9614640 1 -- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0005210 msgr2=0x7f2ae000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9614640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0005210 0x7f2ae000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9614640 1 -- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ae0004800 msgr2=0x7f2ae0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9614640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ae0004800 0x7f2ae0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9614640 1 -- 172.21.15.94:0/4046147868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae0003520 con 0x7f2ae0003a00 2024-03-06T07:21:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9e15640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0005210 0x7f2ae000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9e15640 1 -- 172.21.15.94:0/4046147868 reap_dead start 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2af9614640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae0003a00 0x7f2ae0003e20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2af00098a0 tx=0x7f2ae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c928e43012c78b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af000f040 con 0x7f2ae0003a00 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af0004230 con 0x7f2ae0003a00 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.682+0000 7f2aebfff640 1 -- 172.21.15.94:0/4046147868 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0003610 con 0x7f2ae0003a00 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.683+0000 7f2afb878640 1 -- 172.21.15.94:0/4046147868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae0003a00 msgr2=0x7f2ae0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.683+0000 7f2afb878640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae0003a00 0x7f2ae0003e20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2af00098a0 tx=0x7f2ae0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.683+0000 7f2afb878640 1 -- 172.21.15.94:0/4046147868 shutdown_connections 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.683+0000 7f2afb878640 1 --2- 172.21.15.94:0/4046147868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ae0003a00 0x7f2ae0003e20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.683+0000 7f2afb878640 1 -- 172.21.15.94:0/4046147868 >> 172.21.15.94:0/4046147868 conn(0x7f2af406afc0 msgr2=0x7f2af406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.683+0000 7f2afb878640 1 -- 172.21.15.94:0/4046147868 shutdown_connections 2024-03-06T07:21:38.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.683+0000 7f2afb878640 1 -- 172.21.15.94:0/4046147868 wait complete. 2024-03-06T07:21:38.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.684+0000 7f2afb878640 1 Processor -- start 2024-03-06T07:21:38.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.684+0000 7f2afb878640 1 -- start start 2024-03-06T07:21:38.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2afb878640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0003a00 0x7f2af4147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2afb878640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2af4155f20 0x7f2af41457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2afb878640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2af40c61e0 0x7f2af4145d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2afb878640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af4150a60 con 0x7f2af4155f20 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2afb878640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af4156680 con 0x7f2af40c61e0 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2afb878640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af4150be0 con 0x7f2ae0003a00 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2af4155f20 0x7f2af41457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2af4155f20 0x7f2af41457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48742/0 (socket says 172.21.15.94:48742) 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 -- 172.21.15.94:0/1726554884 learned_addr learned my addr 172.21.15.94:0/1726554884 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af9614640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0003a00 0x7f2af4147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af9e15640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2af40c61e0 0x7f2af4145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 -- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0003a00 msgr2=0x7f2af4147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0003a00 0x7f2af4147d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 -- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2af40c61e0 msgr2=0x7f2af4145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2af40c61e0 0x7f2af4145d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 -- 172.21.15.94:0/1726554884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af41e9690 con 0x7f2af4155f20 2024-03-06T07:21:38.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2af8e13640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2af4155f20 0x7f2af41457d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2ae400f650 tx=0x7f2ae400fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2ae9ffb640 1 -- 172.21.15.94:0/1726554884 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae40050d0 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2ae9ffb640 1 -- 172.21.15.94:0/1726554884 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae4018070 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2ae9ffb640 1 -- 172.21.15.94:0/1726554884 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae400d6b0 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.685+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af40c4330 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.686+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af4148280 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.687+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af404f330 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.687+0000 7f2ae9ffb640 1 -- 172.21.15.94:0/1726554884 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ae400d850 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.687+0000 7f2ae9ffb640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ad806b500 0x7f2ad806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.687+0000 7f2ae9ffb640 1 -- 172.21.15.94:0/1726554884 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f2ae4089ab0 con 0x7f2af4155f20 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.687+0000 7f2af9614640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ad806b500 0x7f2ad806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:38.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.688+0000 7f2af9614640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ad806b500 0x7f2ad806d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2af0035000 tx=0x7f2af0031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:38.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.692+0000 7f2ae9ffb640 1 -- 172.21.15.94:0/1726554884 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ae4058240 con 0x7f2af4155f20 2024-03-06T07:21:38.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:38.901+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2af4074700 con 0x7f2af4155f20 2024-03-06T07:21:39.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.558+0000 7f2ae9ffb640 1 -- 172.21.15.94:0/1726554884 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v75) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2ae405c2e0 con 0x7f2af4155f20 2024-03-06T07:21:39.564 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 15 and data pool 16 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ad806b500 msgr2=0x7f2ad806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ad806b500 0x7f2ad806d9c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2af0035000 tx=0x7f2af0031000 comp rx=0 tx=0).stop 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2af4155f20 msgr2=0x7f2af41457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2af4155f20 0x7f2af41457d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2ae400f650 tx=0x7f2ae400fb10 comp rx=0 tx=0).stop 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 shutdown_connections 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ad806b500 0x7f2ad806d9c0 secure :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2af0035000 tx=0x7f2af0031000 comp rx=0 tx=0).stop 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2af40c61e0 0x7f2af4145d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2af4155f20 0x7f2af41457d0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 --2- 172.21.15.94:0/1726554884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ae0003a00 0x7f2af4147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.568+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 >> 172.21.15.94:0/1726554884 conn(0x7f2af406afc0 msgr2=0x7f2af406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.569+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 shutdown_connections 2024-03-06T07:21:39.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.569+0000 7f2afb878640 1 -- 172.21.15.94:0/1726554884 wait complete. 2024-03-06T07:21:39.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:39.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 Processor -- start 2024-03-06T07:21:39.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 -- start start 2024-03-06T07:21:39.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0d3014f360 0x7f0d30155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c88c0 0x7f0d300c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d300c5880 0x7f0d30158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d300c8dd0 con 0x7f0d300c5880 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d300c73d0 con 0x7f0d3014f360 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d3726b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d300be2e0 con 0x7f0d300c88c0 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d35808640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d300c5880 0x7f0d30158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d35808640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d300c5880 0x7f0d30158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48756/0 (socket says 172.21.15.94:48756) 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.687+0000 7f0d35808640 1 -- 172.21.15.94:0/2502951029 learned_addr learned my addr 172.21.15.94:0/2502951029 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d35007640 1 --2- 172.21.15.94:0/2502951029 >> v2:172.21.15.145:3300/0 conn(0x7f0d3014f360 0x7f0d30155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d34806640 1 --2- 172.21.15.94:0/2502951029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c88c0 0x7f0d300c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d35808640 1 -- 172.21.15.94:0/2502951029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c88c0 msgr2=0x7f0d300c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d35808640 1 --2- 172.21.15.94:0/2502951029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c88c0 0x7f0d300c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d35808640 1 -- 172.21.15.94:0/2502951029 >> v2:172.21.15.145:3300/0 conn(0x7f0d3014f360 msgr2=0x7f0d30155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d35808640 1 --2- 172.21.15.94:0/2502951029 >> v2:172.21.15.145:3300/0 conn(0x7f0d3014f360 0x7f0d30155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d35808640 1 -- 172.21.15.94:0/2502951029 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d300c7670 con 0x7f0d300c5880 2024-03-06T07:21:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.688+0000 7f0d35808640 1 --2- 172.21.15.94:0/2502951029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c5880 0x7f0d30158b60 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0d3008c840 tx=0x7f0d2c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e8eb5819c476cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d27fff640 1 -- 172.21.15.94:0/2502951029 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c02c9e0 con 0x7f0d300c5880 2024-03-06T07:21:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d27fff640 1 -- 172.21.15.94:0/2502951029 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2c02cb80 con 0x7f0d300c5880 2024-03-06T07:21:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d27fff640 1 -- 172.21.15.94:0/2502951029 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c0343f0 con 0x7f0d300c5880 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 -- 172.21.15.94:0/2502951029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c5880 msgr2=0x7f0d30158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2502951029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c5880 0x7f0d30158b60 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0d3008c840 tx=0x7f0d2c02be40 comp rx=0 tx=0).stop 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 -- 172.21.15.94:0/2502951029 shutdown_connections 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2502951029 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c5880 0x7f0d30158b60 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2502951029 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c88c0 0x7f0d300c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2502951029 >> v2:172.21.15.145:3300/0 conn(0x7f0d3014f360 0x7f0d30155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 -- 172.21.15.94:0/2502951029 >> 172.21.15.94:0/2502951029 conn(0x7f0d300ac570 msgr2=0x7f0d300ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 -- 172.21.15.94:0/2502951029 shutdown_connections 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.689+0000 7f0d3726b640 1 -- 172.21.15.94:0/2502951029 wait complete. 2024-03-06T07:21:39.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.690+0000 7f0d3726b640 1 Processor -- start 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.690+0000 7f0d3726b640 1 -- start start 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d3726b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c5880 0x7f0d300c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d3726b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c88c0 0x7f0d3014e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d3726b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d3014f360 0x7f0d30147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d3726b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d301590a0 con 0x7f0d300c88c0 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d3726b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d300c7c30 con 0x7f0d3014f360 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d3726b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d30159220 con 0x7f0d300c5880 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c88c0 0x7f0d3014e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c88c0 0x7f0d3014e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48770/0 (socket says 172.21.15.94:48770) 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 -- 172.21.15.94:0/2658233941 learned_addr learned my addr 172.21.15.94:0/2658233941 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d35808640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d3014f360 0x7f0d30147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d35007640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c5880 0x7f0d300c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 -- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c5880 msgr2=0x7f0d300c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c5880 0x7f0d300c3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 -- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d3014f360 msgr2=0x7f0d30147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d3014f360 0x7f0d30147ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d34806640 1 -- 172.21.15.94:0/2658233941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d300adab0 con 0x7f0d300c88c0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.691+0000 7f0d35808640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d3014f360 0x7f0d30147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.692+0000 7f0d34806640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c88c0 0x7f0d3014e510 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0d2000e860 tx=0x7f0d2000ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.692+0000 7f0d25ffb640 1 -- 172.21.15.94:0/2658233941 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20017070 con 0x7f0d300c88c0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.692+0000 7f0d25ffb640 1 -- 172.21.15.94:0/2658233941 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2000c940 con 0x7f0d300c88c0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.692+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d300add90 con 0x7f0d300c88c0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.692+0000 7f0d25ffb640 1 -- 172.21.15.94:0/2658233941 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2000cd10 con 0x7f0d300c88c0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.692+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3014ea50 con 0x7f0d300c88c0 2024-03-06T07:21:39.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.693+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3004f330 con 0x7f0d300c88c0 2024-03-06T07:21:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.693+0000 7f0d25ffb640 1 -- 172.21.15.94:0/2658233941 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d2001a070 con 0x7f0d300c88c0 2024-03-06T07:21:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.694+0000 7f0d25ffb640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d1406b4d0 0x7f0d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.694+0000 7f0d25ffb640 1 -- 172.21.15.94:0/2658233941 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d2008a6d0 con 0x7f0d300c88c0 2024-03-06T07:21:39.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.694+0000 7f0d35007640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d1406b4d0 0x7f0d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.695+0000 7f0d35007640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d1406b4d0 0x7f0d1406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009a30 tx=0x7f0d28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.698+0000 7f0d25ffb640 1 -- 172.21.15.94:0/2658233941 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d20059450 con 0x7f0d300c88c0 2024-03-06T07:21:39.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.871+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d300c52e0 con 0x7f0d300c88c0 2024-03-06T07:21:39.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.873+0000 7f0d25ffb640 1 -- 172.21.15.94:0/2658233941 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16886 (secure 0 0 0) 0x7f0d20058d50 con 0x7f0d300c88c0 2024-03-06T07:21:39.876 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:39.876 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:38.903784+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:37.116515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:37.869999+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.874+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d1406b4d0 msgr2=0x7f0d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.874+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d1406b4d0 0x7f0d1406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009a30 tx=0x7f0d28008040 comp rx=0 tx=0).stop 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.874+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c88c0 msgr2=0x7f0d3014e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c88c0 0x7f0d3014e510 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0d2000e860 tx=0x7f0d2000ed20 comp rx=0 tx=0).stop 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 shutdown_connections 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d1406b4d0 0x7f0d1406d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d3014f360 0x7f0d30147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d300c88c0 0x7f0d3014e510 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 --2- 172.21.15.94:0/2658233941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d300c5880 0x7f0d300c3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 >> 172.21.15.94:0/2658233941 conn(0x7f0d300ac570 msgr2=0x7f0d3006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 shutdown_connections 2024-03-06T07:21:39.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.875+0000 7f0d3726b640 1 -- 172.21.15.94:0/2658233941 wait complete. 2024-03-06T07:21:39.889 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:39.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.988+0000 7f11ac3b6640 1 Processor -- start 2024-03-06T07:21:39.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.988+0000 7f11ac3b6640 1 -- start start 2024-03-06T07:21:39.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.988+0000 7f11ac3b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f11a41550c0 0x7f11a41554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11ac3b6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11a41559e0 0x7f11a414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11ac3b6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a40aa660 con 0x7f11a40aa180 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11ac3b6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a40c4610 con 0x7f11a41559e0 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11ac3b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a40aa940 con 0x7f11a41550c0 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11a9951640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11a41559e0 0x7f11a414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11a9951640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11a41559e0 0x7f11a414ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48786/0 (socket says 172.21.15.94:48786) 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11a9951640 1 -- 172.21.15.94:0/819553634 learned_addr learned my addr 172.21.15.94:0/819553634 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11aa152640 1 --2- 172.21.15.94:0/819553634 >> v2:172.21.15.145:3300/0 conn(0x7f11a41550c0 0x7f11a41554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11a9951640 1 -- 172.21.15.94:0/819553634 >> v2:172.21.15.145:3300/0 conn(0x7f11a41550c0 msgr2=0x7f11a41554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11a9951640 1 --2- 172.21.15.94:0/819553634 >> v2:172.21.15.145:3300/0 conn(0x7f11a41550c0 0x7f11a41554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11a9951640 1 -- 172.21.15.94:0/819553634 >> v1:172.21.15.145:6789/0 conn(0x7f11a40aa180 legacy=0x7f11a40c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.989+0000 7f11a9951640 1 -- 172.21.15.94:0/819553634 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a40c4860 con 0x7f11a41559e0 2024-03-06T07:21:39.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11a9951640 1 --2- 172.21.15.94:0/819553634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41559e0 0x7f11a414ec00 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f11a40c3fc0 tx=0x7f119402ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1bc6212a6e40f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11a9150640 1 -- 172.21.15.94:0/819553634 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119403a070 con 0x7f11a41559e0 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11a9150640 1 -- 172.21.15.94:0/819553634 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119402da70 con 0x7f11a41559e0 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11a9150640 1 -- 172.21.15.94:0/819553634 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119400b040 con 0x7f11a41559e0 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11ac3b6640 1 -- 172.21.15.94:0/819553634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41559e0 msgr2=0x7f11a414ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/819553634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41559e0 0x7f11a414ec00 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f11a40c3fc0 tx=0x7f119402ccc0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11ac3b6640 1 -- 172.21.15.94:0/819553634 shutdown_connections 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/819553634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41559e0 0x7f11a414ec00 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/819553634 >> v2:172.21.15.145:3300/0 conn(0x7f11a41550c0 0x7f11a41554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.990+0000 7f11ac3b6640 1 -- 172.21.15.94:0/819553634 >> 172.21.15.94:0/819553634 conn(0x7f11a40c6710 msgr2=0x7f11a40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:39.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.991+0000 7f11ac3b6640 1 -- 172.21.15.94:0/819553634 shutdown_connections 2024-03-06T07:21:39.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.991+0000 7f11ac3b6640 1 -- 172.21.15.94:0/819553634 wait complete. 2024-03-06T07:21:39.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 Processor -- start 2024-03-06T07:21:39.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 -- start start 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11a40aa180 0x7f11a40c3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41550c0 0x7f11a4148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11a41559e0 0x7f11a4148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a414f140 con 0x7f11a41550c0 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40c1560 con 0x7f11a41559e0 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11ac3b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a414f2c0 con 0x7f11a40aa180 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11a9951640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41550c0 0x7f11a4148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11a9951640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41550c0 0x7f11a4148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48798/0 (socket says 172.21.15.94:48798) 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11a9951640 1 -- 172.21.15.94:0/4121182230 learned_addr learned my addr 172.21.15.94:0/4121182230 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11aa953640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11a41559e0 0x7f11a4148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.992+0000 7f11aa152640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11a40aa180 0x7f11a40c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11a9951640 1 -- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11a40aa180 msgr2=0x7f11a40c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11a9951640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11a40aa180 0x7f11a40c3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11a9951640 1 -- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11a41559e0 msgr2=0x7f11a4148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11a9951640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11a41559e0 0x7f11a4148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11a9951640 1 -- 172.21.15.94:0/4121182230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a41ddd90 con 0x7f11a41550c0 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11aa953640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11a41559e0 0x7f11a4148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11a9951640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41550c0 0x7f11a4148970 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1194000c00 tx=0x7f1194004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f119affd640 1 -- 172.21.15.94:0/4121182230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119403a030 con 0x7f11a41550c0 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f119affd640 1 -- 172.21.15.94:0/4121182230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119403c070 con 0x7f11a41550c0 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a41db070 con 0x7f11a41550c0 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f119affd640 1 -- 172.21.15.94:0/4121182230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194035d80 con 0x7f11a41550c0 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.993+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a41db630 con 0x7f11a41550c0 2024-03-06T07:21:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.994+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a404f330 con 0x7f11a41550c0 2024-03-06T07:21:40.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.994+0000 7f119affd640 1 -- 172.21.15.94:0/4121182230 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1194047060 con 0x7f11a41550c0 2024-03-06T07:21:40.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.995+0000 7f119affd640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118806b4d0 0x7f118806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.995+0000 7f119affd640 1 -- 172.21.15.94:0/4121182230 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11940ab3b0 con 0x7f11a41550c0 2024-03-06T07:21:40.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.995+0000 7f11aa152640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118806b4d0 0x7f118806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.995+0000 7f11aa152640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118806b4d0 0x7f118806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f11a0006fd0 tx=0x7f11a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:39.999+0000 7f119affd640 1 -- 172.21.15.94:0/4121182230 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f119407a130 con 0x7f11a41550c0 2024-03-06T07:21:40.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.171+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11a4068450 con 0x7f11a41550c0 2024-03-06T07:21:40.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.173+0000 7f119affd640 1 -- 172.21.15.94:0/4121182230 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16886 (secure 0 0 0) 0x7f1194079a30 con 0x7f11a41550c0 2024-03-06T07:21:40.175 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:40.175 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:38.903784+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:37.116515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:37.869999+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.174+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118806b4d0 msgr2=0x7f118806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.174+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118806b4d0 0x7f118806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f11a0006fd0 tx=0x7f11a0008040 comp rx=0 tx=0).stop 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.174+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41550c0 msgr2=0x7f11a4148970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.174+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41550c0 0x7f11a4148970 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1194000c00 tx=0x7f1194004730 comp rx=0 tx=0).stop 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 shutdown_connections 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118806b4d0 0x7f118806d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11a41559e0 0x7f11a4148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11a41550c0 0x7f11a4148970 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 --2- 172.21.15.94:0/4121182230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11a40aa180 0x7f11a40c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 >> 172.21.15.94:0/4121182230 conn(0x7f11a40c6710 msgr2=0x7f11a41de520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:40.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 shutdown_connections 2024-03-06T07:21:40.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.175+0000 7f11ac3b6640 1 -- 172.21.15.94:0/4121182230 wait complete. 2024-03-06T07:21:40.188 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:21:40.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.290+0000 7ff0c795e640 1 Processor -- start 2024-03-06T07:21:40.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.291+0000 7ff0c795e640 1 -- start start 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.291+0000 7ff0c795e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0c00c2aa0 0x7ff0c0155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.291+0000 7ff0c795e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00670c0 con 0x7ff0c00c0470 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.291+0000 7ff0c795e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c006b0f0 con 0x7ff0c00c26c0 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.291+0000 7ff0c795e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c006b740 con 0x7ff0c00c2aa0 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.292+0000 7ff0c4ef9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0c00c2aa0 0x7ff0c0155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.292+0000 7ff0c5efb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff0c00c0470 0x7ff0c0152350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54252/0 (socket says 172.21.15.94:54252) 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.292+0000 7ff0c5efb640 1 -- 172.21.15.94:0/1048081848 learned_addr learned my addr 172.21.15.94:0/1048081848 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.292+0000 7ff0c4ef9640 1 -- 172.21.15.94:0/1048081848 >> v1:172.21.15.145:6789/0 conn(0x7ff0c00c26c0 legacy=0x7ff0c006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.292+0000 7ff0b7fff640 1 -- 172.21.15.94:0/1048081848 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093611645 0 0) 0x7ff0c00670c0 con 0x7ff0c00c0470 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.292+0000 7ff0c4ef9640 1 -- 172.21.15.94:0/1048081848 >> v1:172.21.15.94:6789/0 conn(0x7ff0c00c0470 legacy=0x7ff0c0152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.292+0000 7ff0c4ef9640 1 -- 172.21.15.94:0/1048081848 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c006b3d0 con 0x7ff0c00c2aa0 2024-03-06T07:21:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c4ef9640 1 --2- 172.21.15.94:0/1048081848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c2aa0 0x7ff0c0155bd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00097a0 tx=0x7ff0b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4654b4ffadcb02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0b7fff640 1 -- 172.21.15.94:0/1048081848 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00315a0 con 0x7ff0c00c2aa0 2024-03-06T07:21:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0b7fff640 1 -- 172.21.15.94:0/1048081848 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b0037070 con 0x7ff0c00c2aa0 2024-03-06T07:21:40.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0b7fff640 1 -- 172.21.15.94:0/1048081848 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0031960 con 0x7ff0c00c2aa0 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c795e640 1 -- 172.21.15.94:0/1048081848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c2aa0 msgr2=0x7ff0c0155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1048081848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c2aa0 0x7ff0c0155bd0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00097a0 tx=0x7ff0b002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c795e640 1 -- 172.21.15.94:0/1048081848 shutdown_connections 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1048081848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c2aa0 0x7ff0c0155bd0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c795e640 1 -- 172.21.15.94:0/1048081848 >> 172.21.15.94:0/1048081848 conn(0x7ff0c00c3b40 msgr2=0x7ff0c00c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c795e640 1 -- 172.21.15.94:0/1048081848 shutdown_connections 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.293+0000 7ff0c795e640 1 -- 172.21.15.94:0/1048081848 wait complete. 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.294+0000 7ff0c795e640 1 Processor -- start 2024-03-06T07:21:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.294+0000 7ff0c795e640 1 -- start start 2024-03-06T07:21:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c795e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c0470 0x7ff0c01edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c795e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0c00c26c0 0x7ff0c0146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c795e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0c00c2aa0 0x7ff0c0147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c795e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00c20b0 con 0x7ff0c00c0470 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c795e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00c1f30 con 0x7ff0c00c2aa0 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c795e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0066d00 con 0x7ff0c00c26c0 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c0470 0x7ff0c01edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c0470 0x7ff0c01edf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48814/0 (socket says 172.21.15.94:48814) 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 -- 172.21.15.94:0/1335598805 learned_addr learned my addr 172.21.15.94:0/1335598805 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c4ef9640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0c00c26c0 0x7ff0c0146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c5efb640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0c00c2aa0 0x7ff0c0147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 -- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0c00c26c0 msgr2=0x7ff0c0146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0c00c26c0 0x7ff0c0146dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 -- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0c00c2aa0 msgr2=0x7ff0c0147300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0c00c2aa0 0x7ff0c0147300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.295+0000 7ff0c56fa640 1 -- 172.21.15.94:0/1335598805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c0154dd0 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.296+0000 7ff0c56fa640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c0470 0x7ff0c01edf90 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c450 tx=0x7ff0bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.296+0000 7ff0b5ffb640 1 -- 172.21.15.94:0/1335598805 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0052d0 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.296+0000 7ff0b5ffb640 1 -- 172.21.15.94:0/1335598805 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc005470 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.296+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c00aa300 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.296+0000 7ff0b5ffb640 1 -- 172.21.15.94:0/1335598805 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc012870 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.296+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c014d0c0 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.297+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c004f330 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.297+0000 7ff0b5ffb640 1 -- 172.21.15.94:0/1335598805 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0bc012a10 con 0x7ff0c00c0470 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.298+0000 7ff0b5ffb640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0a406b4d0 0x7ff0a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.298+0000 7ff0b5ffb640 1 -- 172.21.15.94:0/1335598805 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff0bc089f40 con 0x7ff0c00c0470 2024-03-06T07:21:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.298+0000 7ff0c4ef9640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0a406b4d0 0x7ff0a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.298+0000 7ff0c4ef9640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0a406b4d0 0x7ff0a406d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0036000 tx=0x7ff0b0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.302+0000 7ff0b5ffb640 1 -- 172.21.15.94:0/1335598805 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0bc058cc0 con 0x7ff0c00c0470 2024-03-06T07:21:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.509+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff0c006bc70 con 0x7ff0c00c0470 2024-03-06T07:21:40.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.560+0000 7ff0b5ffb640 1 -- 172.21.15.94:0/1335598805 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff0bc0585c0 con 0x7ff0c00c0470 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0a406b4d0 msgr2=0x7ff0a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0a406b4d0 0x7ff0a406d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0036000 tx=0x7ff0b0008030 comp rx=0 tx=0).stop 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c0470 msgr2=0x7ff0c01edf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c0470 0x7ff0c01edf90 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c450 tx=0x7ff0bc00c910 comp rx=0 tx=0).stop 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 shutdown_connections 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0a406b4d0 0x7ff0a406d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0c00c2aa0 0x7ff0c0147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0c00c26c0 0x7ff0c0146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 --2- 172.21.15.94:0/1335598805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0c00c0470 0x7ff0c01edf90 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 >> 172.21.15.94:0/1335598805 conn(0x7ff0c00c3b40 msgr2=0x7ff0c0152120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:40.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 shutdown_connections 2024-03-06T07:21:40.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.562+0000 7ff0c795e640 1 -- 172.21.15.94:0/1335598805 wait complete. 2024-03-06T07:21:40.576 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:40.576 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:21:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.678+0000 7fb38308f640 1 Processor -- start 2024-03-06T07:21:40.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.678+0000 7fb38308f640 1 -- start start 2024-03-06T07:21:40.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb38308f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb37c0c5990 0x7fb37c0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb38308f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb37c0ab170 0x7fb37c06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb38308f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37c06cba0 con 0x7fb37c06c730 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb38308f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37c0c3ef0 con 0x7fb37c0ab170 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb38308f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37c1556d0 con 0x7fb37c0c5990 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb37bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb37c0ab170 0x7fb37c06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb38162c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb37c06c730 0x7fb37c1523f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54256/0 (socket says 172.21.15.94:54256) 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb38162c640 1 -- 172.21.15.94:0/3912346711 learned_addr learned my addr 172.21.15.94:0/3912346711 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb380e2b640 1 --2- 172.21.15.94:0/3912346711 >> v2:172.21.15.145:3300/0 conn(0x7fb37c0c5990 0x7fb37c0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb37bfff640 1 -- 172.21.15.94:0/3912346711 >> v2:172.21.15.145:3300/0 conn(0x7fb37c0c5990 msgr2=0x7fb37c0c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb37bfff640 1 --2- 172.21.15.94:0/3912346711 >> v2:172.21.15.145:3300/0 conn(0x7fb37c0c5990 0x7fb37c0c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb37bfff640 1 -- 172.21.15.94:0/3912346711 >> v1:172.21.15.94:6789/0 conn(0x7fb37c06c730 legacy=0x7fb37c1523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb37bfff640 1 -- 172.21.15.94:0/3912346711 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c1559b0 con 0x7fb37c0ab170 2024-03-06T07:21:40.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.679+0000 7fb37b7fe640 1 -- 172.21.15.94:0/3912346711 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222700358 0 0) 0x7fb37c06cba0 con 0x7fb37c06c730 2024-03-06T07:21:40.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb37bfff640 1 --2- 172.21.15.94:0/3912346711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c0ab170 0x7fb37c06c1f0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0c12a0 tx=0x7fb36802e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e1c45450bbbe10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb37b7fe640 1 -- 172.21.15.94:0/3912346711 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36803b070 con 0x7fb37c0ab170 2024-03-06T07:21:40.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb37b7fe640 1 -- 172.21.15.94:0/3912346711 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36802ee50 con 0x7fb37c0ab170 2024-03-06T07:21:40.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb37b7fe640 1 -- 172.21.15.94:0/3912346711 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368036660 con 0x7fb37c0ab170 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 -- 172.21.15.94:0/3912346711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c0ab170 msgr2=0x7fb37c06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 --2- 172.21.15.94:0/3912346711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c0ab170 0x7fb37c06c1f0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0c12a0 tx=0x7fb36802e0a0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 -- 172.21.15.94:0/3912346711 shutdown_connections 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 --2- 172.21.15.94:0/3912346711 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c0ab170 0x7fb37c06c1f0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 --2- 172.21.15.94:0/3912346711 >> v2:172.21.15.145:3300/0 conn(0x7fb37c0c5990 0x7fb37c0c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 -- 172.21.15.94:0/3912346711 >> 172.21.15.94:0/3912346711 conn(0x7fb37c0c6710 msgr2=0x7fb37c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 -- 172.21.15.94:0/3912346711 shutdown_connections 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.680+0000 7fb38308f640 1 -- 172.21.15.94:0/3912346711 wait complete. 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.681+0000 7fb38308f640 1 Processor -- start 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb38308f640 1 -- start start 2024-03-06T07:21:40.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb38308f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c06c730 0x7fb37c14d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb38308f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb37c0ab170 0x7fb37c147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb38308f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb37c0c5990 0x7fb37c14b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb38308f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c156490 con 0x7fb37c06c730 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb38308f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c155f70 con 0x7fb37c0ab170 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb38308f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c156610 con 0x7fb37c0c5990 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb380e2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c06c730 0x7fb37c14d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb380e2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c06c730 0x7fb37c14d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48820/0 (socket says 172.21.15.94:48820) 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.682+0000 7fb380e2b640 1 -- 172.21.15.94:0/2177867089 learned_addr learned my addr 172.21.15.94:0/2177867089 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb37bfff640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb37c0ab170 0x7fb37c147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb38162c640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb37c0c5990 0x7fb37c14b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb380e2b640 1 -- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb37c0c5990 msgr2=0x7fb37c14b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb380e2b640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb37c0c5990 0x7fb37c14b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb380e2b640 1 -- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb37c0ab170 msgr2=0x7fb37c147d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb380e2b640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb37c0ab170 0x7fb37c147d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb380e2b640 1 -- 172.21.15.94:0/2177867089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c14de80 con 0x7fb37c06c730 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb380e2b640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c06c730 0x7fb37c14d5a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb37000e870 tx=0x7fb37000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb3797fa640 1 -- 172.21.15.94:0/2177867089 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb370017070 con 0x7fb37c06c730 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb3797fa640 1 -- 172.21.15.94:0/2177867089 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37000c940 con 0x7fb37c06c730 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c155410 con 0x7fb37c06c730 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb3797fa640 1 -- 172.21.15.94:0/2177867089 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37000cd30 con 0x7fb37c06c730 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.683+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c1e9860 con 0x7fb37c06c730 2024-03-06T07:21:40.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.684+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c04f330 con 0x7fb37c06c730 2024-03-06T07:21:40.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.685+0000 7fb3797fa640 1 -- 172.21.15.94:0/2177867089 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb37001a070 con 0x7fb37c06c730 2024-03-06T07:21:40.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.685+0000 7fb3797fa640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb36006b4d0 0x7fb36006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.685+0000 7fb3797fa640 1 -- 172.21.15.94:0/2177867089 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb37008a130 con 0x7fb37c06c730 2024-03-06T07:21:40.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.685+0000 7fb37bfff640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb36006b4d0 0x7fb36006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.686+0000 7fb37bfff640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb36006b4d0 0x7fb36006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb368038000 tx=0x7fb368032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.689+0000 7fb3797fa640 1 -- 172.21.15.94:0/2177867089 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb370058860 con 0x7fb37c06c730 2024-03-06T07:21:40.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:40.897+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb37c0c63b0 con 0x7fb37c06c730 2024-03-06T07:21:41.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.564+0000 7fb3797fa640 1 -- 172.21.15.94:0/2177867089 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb37005c900 con 0x7fb37c06c730 2024-03-06T07:21:41.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb36006b4d0 msgr2=0x7fb36006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb36006b4d0 0x7fb36006d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb368038000 tx=0x7fb368032000 comp rx=0 tx=0).stop 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c06c730 msgr2=0x7fb37c14d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c06c730 0x7fb37c14d5a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb37000e870 tx=0x7fb37000ed30 comp rx=0 tx=0).stop 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 shutdown_connections 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb36006b4d0 0x7fb36006d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb37c0c5990 0x7fb37c14b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb37c0ab170 0x7fb37c147d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 --2- 172.21.15.94:0/2177867089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb37c06c730 0x7fb37c14d5a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 >> 172.21.15.94:0/2177867089 conn(0x7fb37c0c6710 msgr2=0x7fb37c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 shutdown_connections 2024-03-06T07:21:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.566+0000 7fb38308f640 1 -- 172.21.15.94:0/2177867089 wait complete. 2024-03-06T07:21:41.580 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:21:41.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:41.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.680+0000 7f789b664640 1 Processor -- start 2024-03-06T07:21:41.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.680+0000 7f789b664640 1 -- start start 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f789b664640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78941550c0 0x7f78941554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f789b664640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78941559e0 0x7f789414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f789b664640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78940aa660 con 0x7f78940aa180 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f789b664640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78940c4610 con 0x7f78941550c0 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f789b664640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78940aa940 con 0x7f78941559e0 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7899400640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78941550c0 0x7f78941554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7899400640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78941550c0 0x7f78941554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48826/0 (socket says 172.21.15.94:48826) 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7899400640 1 -- 172.21.15.94:0/3137923427 learned_addr learned my addr 172.21.15.94:0/3137923427 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:41.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7898bff640 1 --2- 172.21.15.94:0/3137923427 >> v2:172.21.15.145:3300/0 conn(0x7f78941559e0 0x7f789414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7899400640 1 -- 172.21.15.94:0/3137923427 >> v2:172.21.15.145:3300/0 conn(0x7f78941559e0 msgr2=0x7f789414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7899400640 1 --2- 172.21.15.94:0/3137923427 >> v2:172.21.15.145:3300/0 conn(0x7f78941559e0 0x7f789414ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7899400640 1 -- 172.21.15.94:0/3137923427 >> v1:172.21.15.94:6789/0 conn(0x7f78940aa180 legacy=0x7f78940c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f788bfff640 1 -- 172.21.15.94:0/3137923427 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218834934 0 0) 0x7f78940aa660 con 0x7f78940aa180 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7899400640 1 -- 172.21.15.94:0/3137923427 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78940c4860 con 0x7f78941550c0 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.681+0000 7f7898bff640 1 --2- 172.21.15.94:0/3137923427 >> v2:172.21.15.145:3300/0 conn(0x7f78941559e0 0x7f789414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.682+0000 7f7899400640 1 --2- 172.21.15.94:0/3137923427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f78941554a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f78940abe90 tx=0x7f789002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917294892d236b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.682+0000 7f788bfff640 1 -- 172.21.15.94:0/3137923427 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789002e070 con 0x7f78941550c0 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.682+0000 7f788bfff640 1 -- 172.21.15.94:0/3137923427 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7890037070 con 0x7f78941550c0 2024-03-06T07:21:41.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.682+0000 7f788bfff640 1 -- 172.21.15.94:0/3137923427 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890031a00 con 0x7f78941550c0 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 -- 172.21.15.94:0/3137923427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 msgr2=0x7f78941554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 --2- 172.21.15.94:0/3137923427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f78941554a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f78940abe90 tx=0x7f789002b880 comp rx=0 tx=0).stop 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 -- 172.21.15.94:0/3137923427 shutdown_connections 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 --2- 172.21.15.94:0/3137923427 >> v2:172.21.15.145:3300/0 conn(0x7f78941559e0 0x7f789414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 --2- 172.21.15.94:0/3137923427 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f78941554a0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 -- 172.21.15.94:0/3137923427 >> 172.21.15.94:0/3137923427 conn(0x7f78940c6710 msgr2=0x7f78940c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 -- 172.21.15.94:0/3137923427 shutdown_connections 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.683+0000 7f789b664640 1 -- 172.21.15.94:0/3137923427 wait complete. 2024-03-06T07:21:41.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.684+0000 7f789b664640 1 Processor -- start 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.684+0000 7f789b664640 1 -- start start 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f789b664640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78940aa180 0x7f78941517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f789b664640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f7894070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f789b664640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78941559e0 0x7f7894142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f789b664640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789414f140 con 0x7f78941550c0 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f789b664640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78940c1560 con 0x7f78941559e0 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f789b664640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789414f2c0 con 0x7f78940aa180 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f7894070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f7894070bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48842/0 (socket says 172.21.15.94:48842) 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 -- 172.21.15.94:0/1176919601 learned_addr learned my addr 172.21.15.94:0/1176919601 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7899c01640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78941559e0 0x7f7894142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7899400640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78940aa180 0x7f78941517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 -- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78940aa180 msgr2=0x7f78941517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78940aa180 0x7f78941517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 -- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78941559e0 msgr2=0x7f7894142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78941559e0 0x7f7894142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 -- 172.21.15.94:0/1176919601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894150a10 con 0x7f78941550c0 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7898bff640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f7894070bc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f788400e960 tx=0x7f788400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7889ffb640 1 -- 172.21.15.94:0/1176919601 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884017070 con 0x7f78941550c0 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7889ffb640 1 -- 172.21.15.94:0/1176919601 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788400ca00 con 0x7f78941550c0 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78940a8f70 con 0x7f78941550c0 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.685+0000 7f7889ffb640 1 -- 172.21.15.94:0/1176919601 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788400cdd0 con 0x7f78941550c0 2024-03-06T07:21:41.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.686+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7894150fb0 con 0x7f78941550c0 2024-03-06T07:21:41.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.687+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7894069460 con 0x7f78941550c0 2024-03-06T07:21:41.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.687+0000 7f7889ffb640 1 -- 172.21.15.94:0/1176919601 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f788401a070 con 0x7f78941550c0 2024-03-06T07:21:41.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.687+0000 7f7889ffb640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f787806b4d0 0x7f787806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:41.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.687+0000 7f7889ffb640 1 -- 172.21.15.94:0/1176919601 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f788408a130 con 0x7f78941550c0 2024-03-06T07:21:41.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.688+0000 7f7899400640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f787806b4d0 0x7f787806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:41.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.688+0000 7f7899400640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f787806b4d0 0x7f787806d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7890036000 tx=0x7f789003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:41.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.692+0000 7f7889ffb640 1 -- 172.21.15.94:0/1176919601 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7894069460 con 0x7f78941550c0 2024-03-06T07:21:41.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.899+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f789404f330 con 0x7f78941550c0 2024-03-06T07:21:41.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.899+0000 7f7889ffb640 1 -- 172.21.15.94:0/1176919601 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f7884058860 con 0x7f78941550c0 2024-03-06T07:21:41.901 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:41.901 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:40.899292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35900},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35900":{"gid":35900,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/3774173957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3774173957},{"type":"v1","addr":"172.21.15.145:6836","nonce":3774173957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35900,"qdb_cluster":[35900]},"id":7}]} 2024-03-06T07:21:41.902 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 77 2024-03-06T07:21:41.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f787806b4d0 msgr2=0x7f787806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f787806b4d0 0x7f787806d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7890036000 tx=0x7f789003d040 comp rx=0 tx=0).stop 2024-03-06T07:21:41.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 msgr2=0x7f7894070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:41.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f7894070bc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f788400e960 tx=0x7f788400ee20 comp rx=0 tx=0).stop 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 shutdown_connections 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f787806b4d0 0x7f787806d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78941559e0 0x7f7894142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78941550c0 0x7f7894070bc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 --2- 172.21.15.94:0/1176919601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78940aa180 0x7f78941517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 >> 172.21.15.94:0/1176919601 conn(0x7f78940c6710 msgr2=0x7f7894154370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 shutdown_connections 2024-03-06T07:21:41.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:41.901+0000 7f789b664640 1 -- 172.21.15.94:0/1176919601 wait complete. 2024-03-06T07:21:41.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:42.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.012+0000 7fc1c4055640 1 Processor -- start 2024-03-06T07:21:42.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.012+0000 7fc1c4055640 1 -- start start 2024-03-06T07:21:42.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.013+0000 7fc1c4055640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1bc14f310 0x7fc1bc0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.013+0000 7fc1c4055640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1bc155bb0 con 0x7fc1bc1550c0 2024-03-06T07:21:42.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.013+0000 7fc1c4055640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1bc0aa660 con 0x7fc1bc0aa180 2024-03-06T07:21:42.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.013+0000 7fc1c4055640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc0c4610 con 0x7fc1bc14f310 2024-03-06T07:21:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.013+0000 7fc1c25f2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1bc14f310 0x7fc1bc0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.013+0000 7fc1c1df1640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc1bc1550c0 0x7fc1bc1554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54274/0 (socket says 172.21.15.94:54274) 2024-03-06T07:21:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.013+0000 7fc1c1df1640 1 -- 172.21.15.94:0/1989188186 learned_addr learned my addr 172.21.15.94:0/1989188186 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c25f2640 1 -- 172.21.15.94:0/1989188186 >> v1:172.21.15.145:6789/0 conn(0x7fc1bc0aa180 legacy=0x7fc1bc14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c25f2640 1 -- 172.21.15.94:0/1989188186 >> v1:172.21.15.94:6789/0 conn(0x7fc1bc1550c0 legacy=0x7fc1bc1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c25f2640 1 -- 172.21.15.94:0/1989188186 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc0aa940 con 0x7fc1bc14f310 2024-03-06T07:21:42.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c0def640 1 -- 172.21.15.94:0/1989188186 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426680828 0 0) 0x7fc1bc155bb0 con 0x7fc1bc1550c0 2024-03-06T07:21:42.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c25f2640 1 --2- 172.21.15.94:0/1989188186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc14f310 0x7fc1bc0c5060 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009d90 tx=0x7fc1b402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36a7bcb3a235e7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c0def640 1 -- 172.21.15.94:0/1989188186 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4039070 con 0x7fc1bc14f310 2024-03-06T07:21:42.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c0def640 1 -- 172.21.15.94:0/1989188186 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b402cbd0 con 0x7fc1bc14f310 2024-03-06T07:21:42.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.014+0000 7fc1c0def640 1 -- 172.21.15.94:0/1989188186 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4034440 con 0x7fc1bc14f310 2024-03-06T07:21:42.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.015+0000 7fc1c4055640 1 -- 172.21.15.94:0/1989188186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc14f310 msgr2=0x7fc1bc0c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.015+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1989188186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc14f310 0x7fc1bc0c5060 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009d90 tx=0x7fc1b402be20 comp rx=0 tx=0).stop 2024-03-06T07:21:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.015+0000 7fc1c4055640 1 -- 172.21.15.94:0/1989188186 shutdown_connections 2024-03-06T07:21:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.015+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1989188186 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc14f310 0x7fc1bc0c5060 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.015+0000 7fc1c4055640 1 -- 172.21.15.94:0/1989188186 >> 172.21.15.94:0/1989188186 conn(0x7fc1bc0c6710 msgr2=0x7fc1bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.015+0000 7fc1c4055640 1 -- 172.21.15.94:0/1989188186 shutdown_connections 2024-03-06T07:21:42.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.015+0000 7fc1c4055640 1 -- 172.21.15.94:0/1989188186 wait complete. 2024-03-06T07:21:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.016+0000 7fc1c4055640 1 Processor -- start 2024-03-06T07:21:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.016+0000 7fc1c4055640 1 -- start start 2024-03-06T07:21:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c4055640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc0aa180 0x7fc1bc144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c4055640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1bc14f310 0x7fc1bc1520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c4055640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1bc1550c0 0x7fc1bc150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c4055640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0c1490 con 0x7fc1bc0aa180 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c4055640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0c1310 con 0x7fc1bc1550c0 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c4055640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0c31b0 con 0x7fc1bc14f310 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc0aa180 0x7fc1bc144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc0aa180 0x7fc1bc144620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48854/0 (socket says 172.21.15.94:48854) 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 -- 172.21.15.94:0/1455721272 learned_addr learned my addr 172.21.15.94:0/1455721272 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c25f2640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1bc1550c0 0x7fc1bc150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c15f0640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1bc14f310 0x7fc1bc1520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 -- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1bc14f310 msgr2=0x7fc1bc1520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1bc14f310 0x7fc1bc1520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 -- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1bc1550c0 msgr2=0x7fc1bc150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1bc1550c0 0x7fc1bc150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 -- 172.21.15.94:0/1455721272 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc072380 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.017+0000 7fc1c1df1640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc0aa180 0x7fc1bc144620 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c420 tx=0x7fc1b800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.018+0000 7fc1b27fc640 1 -- 172.21.15.94:0/1455721272 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80074e0 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.018+0000 7fc1b27fc640 1 -- 172.21.15.94:0/1455721272 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b80047e0 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.018+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1bc072660 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.018+0000 7fc1b27fc640 1 -- 172.21.15.94:0/1455721272 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8004c10 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.018+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1bc072c20 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.018+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1bc04f330 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.019+0000 7fc1b27fc640 1 -- 172.21.15.94:0/1455721272 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1b8011070 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.019+0000 7fc1b27fc640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.019+0000 7fc1b27fc640 1 -- 172.21.15.94:0/1455721272 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc1b8089d90 con 0x7fc1bc0aa180 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.020+0000 7fc1c15f0640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.020+0000 7fc1c15f0640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc1ed230 tx=0x7fc1ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.024+0000 7fc1b27fc640 1 -- 172.21.15.94:0/1455721272 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1b80584c0 con 0x7fc1bc0aa180 2024-03-06T07:21:42.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.194+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1bc0c2be0 con 0x7fc1bc0aa180 2024-03-06T07:21:42.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.196+0000 7fc1b27fc640 1 -- 172.21.15.94:0/1455721272 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16873 (secure 0 0 0) 0x7fc1b805c560 con 0x7fc1bc0aa180 2024-03-06T07:21:42.198 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:42.198 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":95,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:40.546567+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:37.116515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:37.869999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 msgr2=0x7fc1a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc1ed230 tx=0x7fc1ac009040 comp rx=0 tx=0).stop 2024-03-06T07:21:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc0aa180 msgr2=0x7fc1bc144620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc0aa180 0x7fc1bc144620 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c420 tx=0x7fc1b800c8e0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 shutdown_connections 2024-03-06T07:21:42.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1bc1550c0 0x7fc1bc150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.197+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1bc14f310 0x7fc1bc1520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.198+0000 7fc1c4055640 1 --2- 172.21.15.94:0/1455721272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1bc0aa180 0x7fc1bc144620 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.198+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 >> 172.21.15.94:0/1455721272 conn(0x7fc1bc0c6710 msgr2=0x7fc1bc14d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.198+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 shutdown_connections 2024-03-06T07:21:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.198+0000 7fc1c4055640 1 -- 172.21.15.94:0/1455721272 wait complete. 2024-03-06T07:21:42.211 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:21:42.211 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:42.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.309+0000 7fd5ffa37640 1 Processor -- start 2024-03-06T07:21:42.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.309+0000 7fd5ffa37640 1 -- start start 2024-03-06T07:21:42.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5ffa37640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd5f814f790 0x7fd5f81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5ffa37640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5f80c1aa0 0x7fd5f80c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5ffa37640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f80c28d0 con 0x7fd5f80c5c60 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5ffa37640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f80c2680 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5ffa37640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f80c2bb0 con 0x7fd5f814f790 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5fcfd2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5f80c1aa0 0x7fd5f80c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5fdfd4640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd5f80c5c60 0x7fd5f8155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54282/0 (socket says 172.21.15.94:54282) 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5fdfd4640 1 -- 172.21.15.94:0/3471205243 learned_addr learned my addr 172.21.15.94:0/3471205243 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.310+0000 7fd5fd7d3640 1 --2- 172.21.15.94:0/3471205243 >> v2:172.21.15.145:3300/0 conn(0x7fd5f814f790 0x7fd5f81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.311+0000 7fd5fcfd2640 1 -- 172.21.15.94:0/3471205243 >> v2:172.21.15.145:3300/0 conn(0x7fd5f814f790 msgr2=0x7fd5f81529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.311+0000 7fd5fcfd2640 1 --2- 172.21.15.94:0/3471205243 >> v2:172.21.15.145:3300/0 conn(0x7fd5f814f790 0x7fd5f81529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.311+0000 7fd5fcfd2640 1 -- 172.21.15.94:0/3471205243 >> v1:172.21.15.94:6789/0 conn(0x7fd5f80c5c60 legacy=0x7fd5f8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.311+0000 7fd5fcfd2640 1 -- 172.21.15.94:0/3471205243 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f80c2e90 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.311+0000 7fd5fcfd2640 1 --2- 172.21.15.94:0/3471205243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f80c5720 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd5f80c3170 tx=0x7fd5f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f83d69172ead950 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.311+0000 7fd5effff640 1 -- 172.21.15.94:0/3471205243 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4035020 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5effff640 1 -- 172.21.15.94:0/3471205243 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f4031830 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5effff640 1 -- 172.21.15.94:0/3471205243 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4031c00 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 -- 172.21.15.94:0/3471205243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 msgr2=0x7fd5f80c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/3471205243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f80c5720 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd5f80c3170 tx=0x7fd5f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 -- 172.21.15.94:0/3471205243 shutdown_connections 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/3471205243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f80c5720 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/3471205243 >> v2:172.21.15.145:3300/0 conn(0x7fd5f814f790 0x7fd5f81529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 -- 172.21.15.94:0/3471205243 >> 172.21.15.94:0/3471205243 conn(0x7fd5f80c6710 msgr2=0x7fd5f80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 -- 172.21.15.94:0/3471205243 shutdown_connections 2024-03-06T07:21:42.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.312+0000 7fd5ffa37640 1 -- 172.21.15.94:0/3471205243 wait complete. 2024-03-06T07:21:42.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.313+0000 7fd5ffa37640 1 Processor -- start 2024-03-06T07:21:42.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.313+0000 7fd5ffa37640 1 -- start start 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5ffa37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f8144a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5ffa37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5f80c5c60 0x7fd5f814fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5ffa37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5f814f790 0x7fd5f8154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5ffa37640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f806cc00 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5ffa37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80c0d60 con 0x7fd5f814f790 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5ffa37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f806cd80 con 0x7fd5f80c5c60 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f8144a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f8144a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48858/0 (socket says 172.21.15.94:48858) 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 -- 172.21.15.94:0/2045303061 learned_addr learned my addr 172.21.15.94:0/2045303061 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:42.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fdfd4640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5f814f790 0x7fd5f8154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fcfd2640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5f80c5c60 0x7fd5f814fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 -- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5f80c5c60 msgr2=0x7fd5f814fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5f80c5c60 0x7fd5f814fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 -- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5f814f790 msgr2=0x7fd5f8154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5f814f790 0x7fd5f8154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 -- 172.21.15.94:0/2045303061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f8144f70 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.314+0000 7fd5fd7d3640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f8144a30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800e870 tx=0x7fd5e800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.315+0000 7fd5edffb640 1 -- 172.21.15.94:0/2045303061 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8017070 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.315+0000 7fd5edffb640 1 -- 172.21.15.94:0/2045303061 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e800c8e0 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.315+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f8147150 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.315+0000 7fd5edffb640 1 -- 172.21.15.94:0/2045303061 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e800ccd0 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.315+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f81ec630 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.316+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f814de90 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.316+0000 7fd5edffb640 1 -- 172.21.15.94:0/2045303061 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5e801a070 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.316+0000 7fd5edffb640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5dc06b4d0 0x7fd5dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.317+0000 7fd5edffb640 1 -- 172.21.15.94:0/2045303061 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5e808a080 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.317+0000 7fd5fcfd2640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5dc06b4d0 0x7fd5dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.317+0000 7fd5fcfd2640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5dc06b4d0 0x7fd5dc06d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4036000 tx=0x7fd5f4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.321+0000 7fd5edffb640 1 -- 172.21.15.94:0/2045303061 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5e80587b0 con 0x7fd5f80c1aa0 2024-03-06T07:21:42.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.491+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5f8145860 con 0x7fd5dc06b4d0 2024-03-06T07:21:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.497+0000 7fd5edffb640 1 -- 172.21.15.94:0/2045303061 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237317 (secure 0 0 0) 0x7fd5f8145860 con 0x7fd5dc06b4d0 2024-03-06T07:21:42.501 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:42.502 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:21:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5dc06b4d0 msgr2=0x7fd5dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5dc06b4d0 0x7fd5dc06d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4036000 tx=0x7fd5f4008030 comp rx=0 tx=0).stop 2024-03-06T07:21:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 msgr2=0x7fd5f8144a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f8144a30 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800e870 tx=0x7fd5e800ed30 comp rx=0 tx=0).stop 2024-03-06T07:21:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 shutdown_connections 2024-03-06T07:21:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5dc06b4d0 0x7fd5dc06d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5f814f790 0x7fd5f8154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5f80c5c60 0x7fd5f814fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 --2- 172.21.15.94:0/2045303061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5f80c1aa0 0x7fd5f8144a30 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 >> 172.21.15.94:0/2045303061 conn(0x7fd5f80c6710 msgr2=0x7fd5f8072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 shutdown_connections 2024-03-06T07:21:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.500+0000 7fd5ffa37640 1 -- 172.21.15.94:0/2045303061 wait complete. 2024-03-06T07:21:42.512 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-06T07:21:40.679023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":258464,"kb_used_data":8960,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716256,"statfs":{"total":773094113280,"available":772829446144,"internally_reserved":0,"allocated":9175040,"data_stored":5515880,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000725"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:38.539243+0000","last_change":"2024-03-06T07:21:38.539243+0000","last_active":"2024-03-06T07:21:38.539243+0000","last_peered":"2024-03-06T07:21:38.539243+0000","last_clean":"2024-03-06T07:21:38.539243+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:38.539243+0000","last_undegraded":"2024-03-06T07:21:38.539243+0000","last_fullsized":"2024-03-06T07:21:38.539243+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:21:37.533584+0000","last_change":"2024-03-06T07:21:37.533584+0000","last_active":"2024-03-06T07:21:37.533584+0000","last_peered":"2024-03-06T07:21:37.533584+0000","last_clean":"2024-03-06T07:21:37.533584+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:21:37.533584+0000","last_undegraded":"2024-03-06T07:21:37.533584+0000","last_fullsized":"2024-03-06T07:21:37.533584+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":304,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:21:35.840281+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:21:35.840281+0000","last_peered":"2024-03-06T07:21:35.840281+0000","last_clean":"2024-03-06T07:21:35.840281+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:21:35.840281+0000","last_undegraded":"2024-03-06T07:21:35.840281+0000","last_fullsized":"2024-03-06T07:21:35.840281+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574893,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":1196032,"data_stored":1132261,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32296,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1130496,"data_stored":541893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1130496,"data_stored":541893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32300,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1130496,"data_stored":541893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32292,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1130496,"data_stored":541893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32292,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1130496,"data_stored":541893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1130496,"data_stored":541893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32352,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339488,"statfs":{"total":96636764160,"available":96603635712,"internally_reserved":0,"allocated":1196032,"data_stored":1132261,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:42.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:42.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.651+0000 7fc494af4640 1 Processor -- start 2024-03-06T07:21:42.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.651+0000 7fc494af4640 1 -- start start 2024-03-06T07:21:42.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc494af4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc4901550c0 0x7fc4901554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc494af4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4900c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc494af4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4900aa660 con 0x7fc4901559e0 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc494af4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4900c4610 con 0x7fc4901550c0 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc494af4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4900aa940 con 0x7fc4900aa180 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc48ed9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4900c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc48e59c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc4901550c0 0x7fc4901554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc487fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc4901559e0 0x7fc49014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43616/0 (socket says 172.21.15.94:43616) 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.652+0000 7fc487fff640 1 -- 172.21.15.94:0/713384668 learned_addr learned my addr 172.21.15.94:0/713384668 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48ed9d640 1 -- 172.21.15.94:0/713384668 >> v2:172.21.15.145:3300/0 conn(0x7fc4901550c0 msgr2=0x7fc4901554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48dd9b640 1 -- 172.21.15.94:0/713384668 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325571317 0 0) 0x7fc4900aa660 con 0x7fc4901559e0 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48ed9d640 1 --2- 172.21.15.94:0/713384668 >> v2:172.21.15.145:3300/0 conn(0x7fc4901550c0 0x7fc4901554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48ed9d640 1 -- 172.21.15.94:0/713384668 >> v1:172.21.15.145:6789/0 conn(0x7fc4901559e0 legacy=0x7fc49014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48ed9d640 1 -- 172.21.15.94:0/713384668 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4900c4860 con 0x7fc4900aa180 2024-03-06T07:21:42.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48e59c640 1 --2- 172.21.15.94:0/713384668 >> v2:172.21.15.145:3300/0 conn(0x7fc4901550c0 0x7fc4901554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:42.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48ed9d640 1 --2- 172.21.15.94:0/713384668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4900c5060 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc488009d90 tx=0x7fc48802ce90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=decf87ec9a921a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48dd9b640 1 -- 172.21.15.94:0/713384668 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48803a070 con 0x7fc4900aa180 2024-03-06T07:21:42.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48dd9b640 1 -- 172.21.15.94:0/713384668 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48802dc40 con 0x7fc4900aa180 2024-03-06T07:21:42.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.653+0000 7fc48dd9b640 1 -- 172.21.15.94:0/713384668 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488035440 con 0x7fc4900aa180 2024-03-06T07:21:42.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 -- 172.21.15.94:0/713384668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 msgr2=0x7fc4900c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 --2- 172.21.15.94:0/713384668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4900c5060 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc488009d90 tx=0x7fc48802ce90 comp rx=0 tx=0).stop 2024-03-06T07:21:42.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 -- 172.21.15.94:0/713384668 shutdown_connections 2024-03-06T07:21:42.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 --2- 172.21.15.94:0/713384668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4900c5060 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 --2- 172.21.15.94:0/713384668 >> v2:172.21.15.145:3300/0 conn(0x7fc4901550c0 0x7fc4901554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 -- 172.21.15.94:0/713384668 >> 172.21.15.94:0/713384668 conn(0x7fc4900c6710 msgr2=0x7fc4900c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 -- 172.21.15.94:0/713384668 shutdown_connections 2024-03-06T07:21:42.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.654+0000 7fc494af4640 1 -- 172.21.15.94:0/713384668 wait complete. 2024-03-06T07:21:42.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.655+0000 7fc494af4640 1 Processor -- start 2024-03-06T07:21:42.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.655+0000 7fc494af4640 1 -- start start 2024-03-06T07:21:42.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc494af4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4901ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc494af4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4901550c0 0x7fc49014b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc494af4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4901559e0 0x7fc49014b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc494af4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49014f310 con 0x7fc4901550c0 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc494af4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4900c1560 con 0x7fc4901559e0 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc494af4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49014f490 con 0x7fc4900aa180 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc487fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4901550c0 0x7fc49014b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc487fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4901550c0 0x7fc49014b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48872/0 (socket says 172.21.15.94:48872) 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc48e59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4901ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc487fff640 1 -- 172.21.15.94:0/3616653496 learned_addr learned my addr 172.21.15.94:0/3616653496 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc48ed9d640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4901559e0 0x7fc49014b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc48e59c640 1 -- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4901559e0 msgr2=0x7fc49014b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc48e59c640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4901559e0 0x7fc49014b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc48e59c640 1 -- 172.21.15.94:0/3616653496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4901550c0 msgr2=0x7fc49014b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc48e59c640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4901550c0 0x7fc49014b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc48e59c640 1 -- 172.21.15.94:0/3616653496 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4901ec6a0 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.656+0000 7fc487fff640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4901550c0 0x7fc49014b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.657+0000 7fc48e59c640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4901ebdd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc480010bb0 tx=0x7fc48000b510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.657+0000 7fc486ffd640 1 -- 172.21.15.94:0/3616653496 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48001e070 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.657+0000 7fc486ffd640 1 -- 172.21.15.94:0/3616653496 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48000be30 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.657+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4901485e0 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.657+0000 7fc486ffd640 1 -- 172.21.15.94:0/3616653496 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480019610 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.657+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49014bae0 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.658+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc490152a90 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.658+0000 7fc486ffd640 1 -- 172.21.15.94:0/3616653496 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc480021070 con 0x7fc4900aa180 2024-03-06T07:21:42.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.659+0000 7fc486ffd640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47006b4d0 0x7fc47006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:42.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.659+0000 7fc486ffd640 1 -- 172.21.15.94:0/3616653496 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc48008dc70 con 0x7fc4900aa180 2024-03-06T07:21:42.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.659+0000 7fc487fff640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47006b4d0 0x7fc47006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:42.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.659+0000 7fc487fff640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47006b4d0 0x7fc47006d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc4900a9180 tx=0x7fc478009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:42.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.664+0000 7fc486ffd640 1 -- 172.21.15.94:0/3616653496 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc490152a90 con 0x7fc4900aa180 2024-03-06T07:21:42.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.832+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc49004f330 con 0x7fc47006b4d0 2024-03-06T07:21:42.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.838+0000 7fc486ffd640 1 -- 172.21.15.94:0/3616653496 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252159 (secure 0 0 0) 0x7fc49004f330 con 0x7fc47006b4d0 2024-03-06T07:21:42.843 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:42.844 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:21:42.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.841+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47006b4d0 msgr2=0x7fc47006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.841+0000 7fc494af4640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47006b4d0 0x7fc47006d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc4900a9180 tx=0x7fc478009040 comp rx=0 tx=0).stop 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.841+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 msgr2=0x7fc4901ebdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.841+0000 7fc494af4640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4901ebdd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc480010bb0 tx=0x7fc48000b510 comp rx=0 tx=0).stop 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 shutdown_connections 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47006b4d0 0x7fc47006d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4901559e0 0x7fc49014b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4901550c0 0x7fc49014b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 --2- 172.21.15.94:0/3616653496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4900aa180 0x7fc4901ebdd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 >> 172.21.15.94:0/3616653496 conn(0x7fc4900c6710 msgr2=0x7fc490152ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 shutdown_connections 2024-03-06T07:21:42.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:42.842+0000 7fc494af4640 1 -- 172.21.15.94:0/3616653496 wait complete. 2024-03-06T07:21:42.854 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-06T07:21:42.679325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258856,"kb_used_data":9408,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715864,"statfs":{"total":773094113280,"available":772829044736,"internally_reserved":0,"allocated":9633792,"data_stored":5734580,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.139048"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571868+0000","last_change":"2024-03-06T07:21:39.558523+0000","last_active":"2024-03-06T07:21:40.571868+0000","last_peered":"2024-03-06T07:21:40.571868+0000","last_clean":"2024-03-06T07:21:40.571868+0000","last_became_active":"2024-03-06T07:21:39.558273+0000","last_became_peered":"2024-03-06T07:21:39.558273+0000","last_unstale":"2024-03-06T07:21:40.571868+0000","last_undegraded":"2024-03-06T07:21:40.571868+0000","last_fullsized":"2024-03-06T07:21:40.571868+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:43.593051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552087+0000","last_change":"2024-03-06T07:21:39.560386+0000","last_active":"2024-03-06T07:21:40.552087+0000","last_peered":"2024-03-06T07:21:40.552087+0000","last_clean":"2024-03-06T07:21:40.552087+0000","last_became_active":"2024-03-06T07:21:39.560055+0000","last_became_peered":"2024-03-06T07:21:39.560055+0000","last_unstale":"2024-03-06T07:21:40.552087+0000","last_undegraded":"2024-03-06T07:21:40.552087+0000","last_fullsized":"2024-03-06T07:21:40.552087+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:08.322089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571797+0000","last_change":"2024-03-06T07:21:39.568543+0000","last_active":"2024-03-06T07:21:40.571797+0000","last_peered":"2024-03-06T07:21:40.571797+0000","last_clean":"2024-03-06T07:21:40.571797+0000","last_became_active":"2024-03-06T07:21:39.568242+0000","last_became_peered":"2024-03-06T07:21:39.568242+0000","last_unstale":"2024-03-06T07:21:40.571797+0000","last_undegraded":"2024-03-06T07:21:40.571797+0000","last_fullsized":"2024-03-06T07:21:40.571797+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:28:44.600013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822735+0000","last_change":"2024-03-06T07:21:39.557569+0000","last_active":"2024-03-06T07:21:40.822735+0000","last_peered":"2024-03-06T07:21:40.822735+0000","last_clean":"2024-03-06T07:21:40.822735+0000","last_became_active":"2024-03-06T07:21:39.557327+0000","last_became_peered":"2024-03-06T07:21:39.557327+0000","last_unstale":"2024-03-06T07:21:40.822735+0000","last_undegraded":"2024-03-06T07:21:40.822735+0000","last_fullsized":"2024-03-06T07:21:40.822735+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:20.497324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822699+0000","last_change":"2024-03-06T07:21:39.561339+0000","last_active":"2024-03-06T07:21:40.822699+0000","last_peered":"2024-03-06T07:21:40.822699+0000","last_clean":"2024-03-06T07:21:40.822699+0000","last_became_active":"2024-03-06T07:21:39.561088+0000","last_became_peered":"2024-03-06T07:21:39.561088+0000","last_unstale":"2024-03-06T07:21:40.822699+0000","last_undegraded":"2024-03-06T07:21:40.822699+0000","last_fullsized":"2024-03-06T07:21:40.822699+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:14.657657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.551689+0000","last_change":"2024-03-06T07:21:39.566835+0000","last_active":"2024-03-06T07:21:40.551689+0000","last_peered":"2024-03-06T07:21:40.551689+0000","last_clean":"2024-03-06T07:21:40.551689+0000","last_became_active":"2024-03-06T07:21:39.566425+0000","last_became_peered":"2024-03-06T07:21:39.566425+0000","last_unstale":"2024-03-06T07:21:40.551689+0000","last_undegraded":"2024-03-06T07:21:40.551689+0000","last_fullsized":"2024-03-06T07:21:40.551689+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:53.384103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.551790+0000","last_change":"2024-03-06T07:21:39.565615+0000","last_active":"2024-03-06T07:21:40.551790+0000","last_peered":"2024-03-06T07:21:40.551790+0000","last_clean":"2024-03-06T07:21:40.551790+0000","last_became_active":"2024-03-06T07:21:39.565348+0000","last_became_peered":"2024-03-06T07:21:39.565348+0000","last_unstale":"2024-03-06T07:21:40.551790+0000","last_undegraded":"2024-03-06T07:21:40.551790+0000","last_fullsized":"2024-03-06T07:21:40.551790+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:39.074338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.549978+0000","last_change":"2024-03-06T07:21:39.566875+0000","last_active":"2024-03-06T07:21:40.549978+0000","last_peered":"2024-03-06T07:21:40.549978+0000","last_clean":"2024-03-06T07:21:40.549978+0000","last_became_active":"2024-03-06T07:21:39.566558+0000","last_became_peered":"2024-03-06T07:21:39.566558+0000","last_unstale":"2024-03-06T07:21:40.549978+0000","last_undegraded":"2024-03-06T07:21:40.549978+0000","last_fullsized":"2024-03-06T07:21:40.549978+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:34.253993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552140+0000","last_change":"2024-03-06T07:21:39.558160+0000","last_active":"2024-03-06T07:21:40.552140+0000","last_peered":"2024-03-06T07:21:40.552140+0000","last_clean":"2024-03-06T07:21:40.552140+0000","last_became_active":"2024-03-06T07:21:39.557372+0000","last_became_peered":"2024-03-06T07:21:39.557372+0000","last_unstale":"2024-03-06T07:21:40.552140+0000","last_undegraded":"2024-03-06T07:21:40.552140+0000","last_fullsized":"2024-03-06T07:21:40.552140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:26.511208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822335+0000","last_change":"2024-03-06T07:21:39.575352+0000","last_active":"2024-03-06T07:21:40.822335+0000","last_peered":"2024-03-06T07:21:40.822335+0000","last_clean":"2024-03-06T07:21:40.822335+0000","last_became_active":"2024-03-06T07:21:39.575068+0000","last_became_peered":"2024-03-06T07:21:39.575068+0000","last_unstale":"2024-03-06T07:21:40.822335+0000","last_undegraded":"2024-03-06T07:21:40.822335+0000","last_fullsized":"2024-03-06T07:21:40.822335+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:51.522741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573962+0000","last_change":"2024-03-06T07:21:39.572459+0000","last_active":"2024-03-06T07:21:40.573962+0000","last_peered":"2024-03-06T07:21:40.573962+0000","last_clean":"2024-03-06T07:21:40.573962+0000","last_became_active":"2024-03-06T07:21:39.572070+0000","last_became_peered":"2024-03-06T07:21:39.572070+0000","last_unstale":"2024-03-06T07:21:40.573962+0000","last_undegraded":"2024-03-06T07:21:40.573962+0000","last_fullsized":"2024-03-06T07:21:40.573962+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:00.819370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552553+0000","last_change":"2024-03-06T07:21:39.555172+0000","last_active":"2024-03-06T07:21:40.552553+0000","last_peered":"2024-03-06T07:21:40.552553+0000","last_clean":"2024-03-06T07:21:40.552553+0000","last_became_active":"2024-03-06T07:21:39.554888+0000","last_became_peered":"2024-03-06T07:21:39.554888+0000","last_unstale":"2024-03-06T07:21:40.552553+0000","last_undegraded":"2024-03-06T07:21:40.552553+0000","last_fullsized":"2024-03-06T07:21:40.552553+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:08.649060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822115+0000","last_change":"2024-03-06T07:21:39.570047+0000","last_active":"2024-03-06T07:21:40.822115+0000","last_peered":"2024-03-06T07:21:40.822115+0000","last_clean":"2024-03-06T07:21:40.822115+0000","last_became_active":"2024-03-06T07:21:39.569737+0000","last_became_peered":"2024-03-06T07:21:39.569737+0000","last_unstale":"2024-03-06T07:21:40.822115+0000","last_undegraded":"2024-03-06T07:21:40.822115+0000","last_fullsized":"2024-03-06T07:21:40.822115+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:48.822900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821578+0000","last_change":"2024-03-06T07:21:39.564660+0000","last_active":"2024-03-06T07:21:40.821578+0000","last_peered":"2024-03-06T07:21:40.821578+0000","last_clean":"2024-03-06T07:21:40.821578+0000","last_became_active":"2024-03-06T07:21:39.564380+0000","last_became_peered":"2024-03-06T07:21:39.564380+0000","last_unstale":"2024-03-06T07:21:40.821578+0000","last_undegraded":"2024-03-06T07:21:40.821578+0000","last_fullsized":"2024-03-06T07:21:40.821578+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:55.748090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821662+0000","last_change":"2024-03-06T07:21:39.569725+0000","last_active":"2024-03-06T07:21:40.821662+0000","last_peered":"2024-03-06T07:21:40.821662+0000","last_clean":"2024-03-06T07:21:40.821662+0000","last_became_active":"2024-03-06T07:21:39.569406+0000","last_became_peered":"2024-03-06T07:21:39.569406+0000","last_unstale":"2024-03-06T07:21:40.821662+0000","last_undegraded":"2024-03-06T07:21:40.821662+0000","last_fullsized":"2024-03-06T07:21:40.821662+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:03.854208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821576+0000","last_change":"2024-03-06T07:21:39.569423+0000","last_active":"2024-03-06T07:21:40.821576+0000","last_peered":"2024-03-06T07:21:40.821576+0000","last_clean":"2024-03-06T07:21:40.821576+0000","last_became_active":"2024-03-06T07:21:39.569133+0000","last_became_peered":"2024-03-06T07:21:39.569133+0000","last_unstale":"2024-03-06T07:21:40.821576+0000","last_undegraded":"2024-03-06T07:21:40.821576+0000","last_fullsized":"2024-03-06T07:21:40.821576+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:04.207248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571934+0000","last_change":"2024-03-06T07:21:39.565547+0000","last_active":"2024-03-06T07:21:40.571934+0000","last_peered":"2024-03-06T07:21:40.571934+0000","last_clean":"2024-03-06T07:21:40.571934+0000","last_became_active":"2024-03-06T07:21:39.565258+0000","last_became_peered":"2024-03-06T07:21:39.565258+0000","last_unstale":"2024-03-06T07:21:40.571934+0000","last_undegraded":"2024-03-06T07:21:40.571934+0000","last_fullsized":"2024-03-06T07:21:40.571934+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:19.711445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552240+0000","last_change":"2024-03-06T07:21:39.568695+0000","last_active":"2024-03-06T07:21:40.552240+0000","last_peered":"2024-03-06T07:21:40.552240+0000","last_clean":"2024-03-06T07:21:40.552240+0000","last_became_active":"2024-03-06T07:21:39.568218+0000","last_became_peered":"2024-03-06T07:21:39.568218+0000","last_unstale":"2024-03-06T07:21:40.552240+0000","last_undegraded":"2024-03-06T07:21:40.552240+0000","last_fullsized":"2024-03-06T07:21:40.552240+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:46.103816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571767+0000","last_change":"2024-03-06T07:21:39.562627+0000","last_active":"2024-03-06T07:21:40.571767+0000","last_peered":"2024-03-06T07:21:40.571767+0000","last_clean":"2024-03-06T07:21:40.571767+0000","last_became_active":"2024-03-06T07:21:39.562002+0000","last_became_peered":"2024-03-06T07:21:39.562002+0000","last_unstale":"2024-03-06T07:21:40.571767+0000","last_undegraded":"2024-03-06T07:21:40.571767+0000","last_fullsized":"2024-03-06T07:21:40.571767+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:13.724918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552579+0000","last_change":"2024-03-06T07:21:39.561775+0000","last_active":"2024-03-06T07:21:40.552579+0000","last_peered":"2024-03-06T07:21:40.552579+0000","last_clean":"2024-03-06T07:21:40.552579+0000","last_became_active":"2024-03-06T07:21:39.561454+0000","last_became_peered":"2024-03-06T07:21:39.561454+0000","last_unstale":"2024-03-06T07:21:40.552579+0000","last_undegraded":"2024-03-06T07:21:40.552579+0000","last_fullsized":"2024-03-06T07:21:40.552579+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:30.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550168+0000","last_change":"2024-03-06T07:21:39.570212+0000","last_active":"2024-03-06T07:21:40.550168+0000","last_peered":"2024-03-06T07:21:40.550168+0000","last_clean":"2024-03-06T07:21:40.550168+0000","last_became_active":"2024-03-06T07:21:39.569945+0000","last_became_peered":"2024-03-06T07:21:39.569945+0000","last_unstale":"2024-03-06T07:21:40.550168+0000","last_undegraded":"2024-03-06T07:21:40.550168+0000","last_fullsized":"2024-03-06T07:21:40.550168+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:01.674158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822649+0000","last_change":"2024-03-06T07:21:39.571703+0000","last_active":"2024-03-06T07:21:40.822649+0000","last_peered":"2024-03-06T07:21:40.822649+0000","last_clean":"2024-03-06T07:21:40.822649+0000","last_became_active":"2024-03-06T07:21:39.570682+0000","last_became_peered":"2024-03-06T07:21:39.570682+0000","last_unstale":"2024-03-06T07:21:40.822649+0000","last_undegraded":"2024-03-06T07:21:40.822649+0000","last_fullsized":"2024-03-06T07:21:40.822649+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:30.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571626+0000","last_change":"2024-03-06T07:21:39.566219+0000","last_active":"2024-03-06T07:21:40.571626+0000","last_peered":"2024-03-06T07:21:40.571626+0000","last_clean":"2024-03-06T07:21:40.571626+0000","last_became_active":"2024-03-06T07:21:39.565951+0000","last_became_peered":"2024-03-06T07:21:39.565951+0000","last_unstale":"2024-03-06T07:21:40.571626+0000","last_undegraded":"2024-03-06T07:21:40.571626+0000","last_fullsized":"2024-03-06T07:21:40.571626+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:34.359727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822055+0000","last_change":"2024-03-06T07:21:39.559127+0000","last_active":"2024-03-06T07:21:40.822055+0000","last_peered":"2024-03-06T07:21:40.822055+0000","last_clean":"2024-03-06T07:21:40.822055+0000","last_became_active":"2024-03-06T07:21:39.558872+0000","last_became_peered":"2024-03-06T07:21:39.558872+0000","last_unstale":"2024-03-06T07:21:40.822055+0000","last_undegraded":"2024-03-06T07:21:40.822055+0000","last_fullsized":"2024-03-06T07:21:40.822055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:03.246332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552221+0000","last_change":"2024-03-06T07:21:39.566800+0000","last_active":"2024-03-06T07:21:40.552221+0000","last_peered":"2024-03-06T07:21:40.552221+0000","last_clean":"2024-03-06T07:21:40.552221+0000","last_became_active":"2024-03-06T07:21:39.566534+0000","last_became_peered":"2024-03-06T07:21:39.566534+0000","last_unstale":"2024-03-06T07:21:40.552221+0000","last_undegraded":"2024-03-06T07:21:40.552221+0000","last_fullsized":"2024-03-06T07:21:40.552221+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:03.246332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552732+0000","last_change":"2024-03-06T07:21:39.558298+0000","last_active":"2024-03-06T07:21:40.552732+0000","last_peered":"2024-03-06T07:21:40.552732+0000","last_clean":"2024-03-06T07:21:40.552732+0000","last_became_active":"2024-03-06T07:21:39.558011+0000","last_became_peered":"2024-03-06T07:21:39.558011+0000","last_unstale":"2024-03-06T07:21:40.552732+0000","last_undegraded":"2024-03-06T07:21:40.552732+0000","last_fullsized":"2024-03-06T07:21:40.552732+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:14.657657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573951+0000","last_change":"2024-03-06T07:21:39.568785+0000","last_active":"2024-03-06T07:21:40.573951+0000","last_peered":"2024-03-06T07:21:40.573951+0000","last_clean":"2024-03-06T07:21:40.573951+0000","last_became_active":"2024-03-06T07:21:39.568499+0000","last_became_peered":"2024-03-06T07:21:39.568499+0000","last_unstale":"2024-03-06T07:21:40.573951+0000","last_undegraded":"2024-03-06T07:21:40.573951+0000","last_fullsized":"2024-03-06T07:21:40.573951+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:14.657657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552901+0000","last_change":"2024-03-06T07:21:39.570579+0000","last_active":"2024-03-06T07:21:40.552901+0000","last_peered":"2024-03-06T07:21:40.552901+0000","last_clean":"2024-03-06T07:21:40.552901+0000","last_became_active":"2024-03-06T07:21:39.570302+0000","last_became_peered":"2024-03-06T07:21:39.570302+0000","last_unstale":"2024-03-06T07:21:40.552901+0000","last_undegraded":"2024-03-06T07:21:40.552901+0000","last_fullsized":"2024-03-06T07:21:40.552901+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:30.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550294+0000","last_change":"2024-03-06T07:21:39.563779+0000","last_active":"2024-03-06T07:21:40.550294+0000","last_peered":"2024-03-06T07:21:40.550294+0000","last_clean":"2024-03-06T07:21:40.550294+0000","last_became_active":"2024-03-06T07:21:39.563541+0000","last_became_peered":"2024-03-06T07:21:39.563541+0000","last_unstale":"2024-03-06T07:21:40.550294+0000","last_undegraded":"2024-03-06T07:21:40.550294+0000","last_fullsized":"2024-03-06T07:21:40.550294+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:48.893838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552186+0000","last_change":"2024-03-06T07:21:39.575133+0000","last_active":"2024-03-06T07:21:40.552186+0000","last_peered":"2024-03-06T07:21:40.552186+0000","last_clean":"2024-03-06T07:21:40.552186+0000","last_became_active":"2024-03-06T07:21:39.574869+0000","last_became_peered":"2024-03-06T07:21:39.574869+0000","last_unstale":"2024-03-06T07:21:40.552186+0000","last_undegraded":"2024-03-06T07:21:40.552186+0000","last_fullsized":"2024-03-06T07:21:40.552186+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:46.103816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552177+0000","last_change":"2024-03-06T07:21:39.558222+0000","last_active":"2024-03-06T07:21:40.552177+0000","last_peered":"2024-03-06T07:21:40.552177+0000","last_clean":"2024-03-06T07:21:40.552177+0000","last_became_active":"2024-03-06T07:21:39.557928+0000","last_became_peered":"2024-03-06T07:21:39.557928+0000","last_unstale":"2024-03-06T07:21:40.552177+0000","last_undegraded":"2024-03-06T07:21:40.552177+0000","last_fullsized":"2024-03-06T07:21:40.552177+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:34.789108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.549939+0000","last_change":"2024-03-06T07:21:39.564487+0000","last_active":"2024-03-06T07:21:40.549939+0000","last_peered":"2024-03-06T07:21:40.549939+0000","last_clean":"2024-03-06T07:21:40.549939+0000","last_became_active":"2024-03-06T07:21:39.564238+0000","last_became_peered":"2024-03-06T07:21:39.564238+0000","last_unstale":"2024-03-06T07:21:40.549939+0000","last_undegraded":"2024-03-06T07:21:40.549939+0000","last_fullsized":"2024-03-06T07:21:40.549939+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:11.666327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571710+0000","last_change":"2024-03-06T07:21:39.556047+0000","last_active":"2024-03-06T07:21:40.571710+0000","last_peered":"2024-03-06T07:21:40.571710+0000","last_clean":"2024-03-06T07:21:40.571710+0000","last_became_active":"2024-03-06T07:21:39.555681+0000","last_became_peered":"2024-03-06T07:21:39.555681+0000","last_unstale":"2024-03-06T07:21:40.571710+0000","last_undegraded":"2024-03-06T07:21:40.571710+0000","last_fullsized":"2024-03-06T07:21:40.571710+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:30.609572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"94'4","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822756+0000","last_change":"2024-03-06T07:21:38.561342+0000","last_active":"2024-03-06T07:21:40.822756+0000","last_peered":"2024-03-06T07:21:40.822756+0000","last_clean":"2024-03-06T07:21:40.822756+0000","last_became_active":"2024-03-06T07:21:38.561062+0000","last_became_peered":"2024-03-06T07:21:38.561062+0000","last_unstale":"2024-03-06T07:21:40.822756+0000","last_undegraded":"2024-03-06T07:21:40.822756+0000","last_fullsized":"2024-03-06T07:21:40.822756+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:59.813711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550090+0000","last_change":"2024-03-06T07:21:39.566642+0000","last_active":"2024-03-06T07:21:40.550090+0000","last_peered":"2024-03-06T07:21:40.550090+0000","last_clean":"2024-03-06T07:21:40.550090+0000","last_became_active":"2024-03-06T07:21:39.566319+0000","last_became_peered":"2024-03-06T07:21:39.566319+0000","last_unstale":"2024-03-06T07:21:40.550090+0000","last_undegraded":"2024-03-06T07:21:40.550090+0000","last_fullsized":"2024-03-06T07:21:40.550090+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:01.084486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550190+0000","last_change":"2024-03-06T07:21:38.559655+0000","last_active":"2024-03-06T07:21:40.550190+0000","last_peered":"2024-03-06T07:21:40.550190+0000","last_clean":"2024-03-06T07:21:40.550190+0000","last_became_active":"2024-03-06T07:21:38.559285+0000","last_became_peered":"2024-03-06T07:21:38.559285+0000","last_unstale":"2024-03-06T07:21:40.550190+0000","last_undegraded":"2024-03-06T07:21:40.550190+0000","last_fullsized":"2024-03-06T07:21:40.550190+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:16.366085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552343+0000","last_change":"2024-03-06T07:21:39.562234+0000","last_active":"2024-03-06T07:21:40.552343+0000","last_peered":"2024-03-06T07:21:40.552343+0000","last_clean":"2024-03-06T07:21:40.552343+0000","last_became_active":"2024-03-06T07:21:39.561944+0000","last_became_peered":"2024-03-06T07:21:39.561944+0000","last_unstale":"2024-03-06T07:21:40.552343+0000","last_undegraded":"2024-03-06T07:21:40.552343+0000","last_fullsized":"2024-03-06T07:21:40.552343+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:51.522741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.551980+0000","last_change":"2024-03-06T07:21:38.567926+0000","last_active":"2024-03-06T07:21:40.551980+0000","last_peered":"2024-03-06T07:21:40.551980+0000","last_clean":"2024-03-06T07:21:40.551980+0000","last_became_active":"2024-03-06T07:21:38.567657+0000","last_became_peered":"2024-03-06T07:21:38.567657+0000","last_unstale":"2024-03-06T07:21:40.551980+0000","last_undegraded":"2024-03-06T07:21:40.551980+0000","last_fullsized":"2024-03-06T07:21:40.551980+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:29.378583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.574035+0000","last_change":"2024-03-06T07:21:39.565064+0000","last_active":"2024-03-06T07:21:40.574035+0000","last_peered":"2024-03-06T07:21:40.574035+0000","last_clean":"2024-03-06T07:21:40.574035+0000","last_became_active":"2024-03-06T07:21:39.564796+0000","last_became_peered":"2024-03-06T07:21:39.564796+0000","last_unstale":"2024-03-06T07:21:40.574035+0000","last_undegraded":"2024-03-06T07:21:40.574035+0000","last_fullsized":"2024-03-06T07:21:40.574035+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:08.867469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822289+0000","last_change":"2024-03-06T07:21:38.563969+0000","last_active":"2024-03-06T07:21:40.822289+0000","last_peered":"2024-03-06T07:21:40.822289+0000","last_clean":"2024-03-06T07:21:40.822289+0000","last_became_active":"2024-03-06T07:21:38.563668+0000","last_became_peered":"2024-03-06T07:21:38.563668+0000","last_unstale":"2024-03-06T07:21:40.822289+0000","last_undegraded":"2024-03-06T07:21:40.822289+0000","last_fullsized":"2024-03-06T07:21:40.822289+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:42.680526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822835+0000","last_change":"2024-03-06T07:21:39.561952+0000","last_active":"2024-03-06T07:21:40.822835+0000","last_peered":"2024-03-06T07:21:40.822835+0000","last_clean":"2024-03-06T07:21:40.822835+0000","last_became_active":"2024-03-06T07:21:39.561696+0000","last_became_peered":"2024-03-06T07:21:39.561696+0000","last_unstale":"2024-03-06T07:21:40.822835+0000","last_undegraded":"2024-03-06T07:21:40.822835+0000","last_fullsized":"2024-03-06T07:21:40.822835+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:49.478703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821631+0000","last_change":"2024-03-06T07:21:38.565509+0000","last_active":"2024-03-06T07:21:40.821631+0000","last_peered":"2024-03-06T07:21:40.821631+0000","last_clean":"2024-03-06T07:21:40.821631+0000","last_became_active":"2024-03-06T07:21:38.565200+0000","last_became_peered":"2024-03-06T07:21:38.565200+0000","last_unstale":"2024-03-06T07:21:40.821631+0000","last_undegraded":"2024-03-06T07:21:40.821631+0000","last_fullsized":"2024-03-06T07:21:40.821631+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:07.643401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571922+0000","last_change":"2024-03-06T07:21:39.559156+0000","last_active":"2024-03-06T07:21:40.571922+0000","last_peered":"2024-03-06T07:21:40.571922+0000","last_clean":"2024-03-06T07:21:40.571922+0000","last_became_active":"2024-03-06T07:21:39.558738+0000","last_became_peered":"2024-03-06T07:21:39.558738+0000","last_unstale":"2024-03-06T07:21:40.571922+0000","last_undegraded":"2024-03-06T07:21:40.571922+0000","last_fullsized":"2024-03-06T07:21:40.571922+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:36.102485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573711+0000","last_change":"2024-03-06T07:21:38.563488+0000","last_active":"2024-03-06T07:21:40.573711+0000","last_peered":"2024-03-06T07:21:40.573711+0000","last_clean":"2024-03-06T07:21:40.573711+0000","last_became_active":"2024-03-06T07:21:38.563238+0000","last_became_peered":"2024-03-06T07:21:38.563238+0000","last_unstale":"2024-03-06T07:21:40.573711+0000","last_undegraded":"2024-03-06T07:21:40.573711+0000","last_fullsized":"2024-03-06T07:21:40.573711+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:48.266835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552379+0000","last_change":"2024-03-06T07:21:39.571528+0000","last_active":"2024-03-06T07:21:40.552379+0000","last_peered":"2024-03-06T07:21:40.552379+0000","last_clean":"2024-03-06T07:21:40.552379+0000","last_became_active":"2024-03-06T07:21:39.569232+0000","last_became_peered":"2024-03-06T07:21:39.569232+0000","last_unstale":"2024-03-06T07:21:40.552379+0000","last_undegraded":"2024-03-06T07:21:40.552379+0000","last_fullsized":"2024-03-06T07:21:40.552379+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:03.246332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.551594+0000","last_change":"2024-03-06T07:21:38.562638+0000","last_active":"2024-03-06T07:21:40.551594+0000","last_peered":"2024-03-06T07:21:40.551594+0000","last_clean":"2024-03-06T07:21:40.551594+0000","last_became_active":"2024-03-06T07:21:38.562357+0000","last_became_peered":"2024-03-06T07:21:38.562357+0000","last_unstale":"2024-03-06T07:21:40.551594+0000","last_undegraded":"2024-03-06T07:21:40.551594+0000","last_fullsized":"2024-03-06T07:21:40.551594+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.631937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822645+0000","last_change":"2024-03-06T07:21:39.558741+0000","last_active":"2024-03-06T07:21:40.822645+0000","last_peered":"2024-03-06T07:21:40.822645+0000","last_clean":"2024-03-06T07:21:40.822645+0000","last_became_active":"2024-03-06T07:21:39.558469+0000","last_became_peered":"2024-03-06T07:21:39.558469+0000","last_unstale":"2024-03-06T07:21:40.822645+0000","last_undegraded":"2024-03-06T07:21:40.822645+0000","last_fullsized":"2024-03-06T07:21:40.822645+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:56.808564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822004+0000","last_change":"2024-03-06T07:21:38.556711+0000","last_active":"2024-03-06T07:21:40.822004+0000","last_peered":"2024-03-06T07:21:40.822004+0000","last_clean":"2024-03-06T07:21:40.822004+0000","last_became_active":"2024-03-06T07:21:38.556454+0000","last_became_peered":"2024-03-06T07:21:38.556454+0000","last_unstale":"2024-03-06T07:21:40.822004+0000","last_undegraded":"2024-03-06T07:21:40.822004+0000","last_fullsized":"2024-03-06T07:21:40.822004+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:13.651998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571559+0000","last_change":"2024-03-06T07:21:39.556208+0000","last_active":"2024-03-06T07:21:40.571559+0000","last_peered":"2024-03-06T07:21:40.571559+0000","last_clean":"2024-03-06T07:21:40.571559+0000","last_became_active":"2024-03-06T07:21:39.555754+0000","last_became_peered":"2024-03-06T07:21:39.555754+0000","last_unstale":"2024-03-06T07:21:40.571559+0000","last_undegraded":"2024-03-06T07:21:40.571559+0000","last_fullsized":"2024-03-06T07:21:40.571559+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:31.920977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552321+0000","last_change":"2024-03-06T07:21:38.563281+0000","last_active":"2024-03-06T07:21:40.552321+0000","last_peered":"2024-03-06T07:21:40.552321+0000","last_clean":"2024-03-06T07:21:40.552321+0000","last_became_active":"2024-03-06T07:21:38.563016+0000","last_became_peered":"2024-03-06T07:21:38.563016+0000","last_unstale":"2024-03-06T07:21:40.552321+0000","last_undegraded":"2024-03-06T07:21:40.552321+0000","last_fullsized":"2024-03-06T07:21:40.552321+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:42.680526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822820+0000","last_change":"2024-03-06T07:21:39.557951+0000","last_active":"2024-03-06T07:21:40.822820+0000","last_peered":"2024-03-06T07:21:40.822820+0000","last_clean":"2024-03-06T07:21:40.822820+0000","last_became_active":"2024-03-06T07:21:39.557666+0000","last_became_peered":"2024-03-06T07:21:39.557666+0000","last_unstale":"2024-03-06T07:21:40.822820+0000","last_undegraded":"2024-03-06T07:21:40.822820+0000","last_fullsized":"2024-03-06T07:21:40.822820+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:40.396311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571719+0000","last_change":"2024-03-06T07:21:38.557167+0000","last_active":"2024-03-06T07:21:40.571719+0000","last_peered":"2024-03-06T07:21:40.571719+0000","last_clean":"2024-03-06T07:21:40.571719+0000","last_became_active":"2024-03-06T07:21:38.556899+0000","last_became_peered":"2024-03-06T07:21:38.556899+0000","last_unstale":"2024-03-06T07:21:40.571719+0000","last_undegraded":"2024-03-06T07:21:40.571719+0000","last_fullsized":"2024-03-06T07:21:40.571719+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:54.742431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550140+0000","last_change":"2024-03-06T07:21:39.565889+0000","last_active":"2024-03-06T07:21:40.550140+0000","last_peered":"2024-03-06T07:21:40.550140+0000","last_clean":"2024-03-06T07:21:40.550140+0000","last_became_active":"2024-03-06T07:21:39.565628+0000","last_became_peered":"2024-03-06T07:21:39.565628+0000","last_unstale":"2024-03-06T07:21:40.550140+0000","last_undegraded":"2024-03-06T07:21:40.550140+0000","last_fullsized":"2024-03-06T07:21:40.550140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:05.334666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"94'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821705+0000","last_change":"2024-03-06T07:21:38.555958+0000","last_active":"2024-03-06T07:21:40.821705+0000","last_peered":"2024-03-06T07:21:40.821705+0000","last_clean":"2024-03-06T07:21:40.821705+0000","last_became_active":"2024-03-06T07:21:38.555657+0000","last_became_peered":"2024-03-06T07:21:38.555657+0000","last_unstale":"2024-03-06T07:21:40.821705+0000","last_undegraded":"2024-03-06T07:21:40.821705+0000","last_fullsized":"2024-03-06T07:21:40.821705+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:29.036077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822772+0000","last_change":"2024-03-06T07:21:39.571490+0000","last_active":"2024-03-06T07:21:40.822772+0000","last_peered":"2024-03-06T07:21:40.822772+0000","last_clean":"2024-03-06T07:21:40.822772+0000","last_became_active":"2024-03-06T07:21:39.571202+0000","last_became_peered":"2024-03-06T07:21:39.571202+0000","last_unstale":"2024-03-06T07:21:40.822772+0000","last_undegraded":"2024-03-06T07:21:40.822772+0000","last_fullsized":"2024-03-06T07:21:40.822772+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:00.819370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552801+0000","last_change":"2024-03-06T07:21:38.555822+0000","last_active":"2024-03-06T07:21:40.552801+0000","last_peered":"2024-03-06T07:21:40.552801+0000","last_clean":"2024-03-06T07:21:40.552801+0000","last_became_active":"2024-03-06T07:21:38.555546+0000","last_became_peered":"2024-03-06T07:21:38.555546+0000","last_unstale":"2024-03-06T07:21:40.552801+0000","last_undegraded":"2024-03-06T07:21:40.552801+0000","last_fullsized":"2024-03-06T07:21:40.552801+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.622299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573970+0000","last_change":"2024-03-06T07:21:39.563900+0000","last_active":"2024-03-06T07:21:40.573970+0000","last_peered":"2024-03-06T07:21:40.573970+0000","last_clean":"2024-03-06T07:21:40.573970+0000","last_became_active":"2024-03-06T07:21:39.562560+0000","last_became_peered":"2024-03-06T07:21:39.562560+0000","last_unstale":"2024-03-06T07:21:40.573970+0000","last_undegraded":"2024-03-06T07:21:40.573970+0000","last_fullsized":"2024-03-06T07:21:40.573970+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:22.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822795+0000","last_change":"2024-03-06T07:21:38.563044+0000","last_active":"2024-03-06T07:21:40.822795+0000","last_peered":"2024-03-06T07:21:40.822795+0000","last_clean":"2024-03-06T07:21:40.822795+0000","last_became_active":"2024-03-06T07:21:38.562787+0000","last_became_peered":"2024-03-06T07:21:38.562787+0000","last_unstale":"2024-03-06T07:21:40.822795+0000","last_undegraded":"2024-03-06T07:21:40.822795+0000","last_fullsized":"2024-03-06T07:21:40.822795+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:45.451298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552537+0000","last_change":"2024-03-06T07:21:39.557239+0000","last_active":"2024-03-06T07:21:40.552537+0000","last_peered":"2024-03-06T07:21:40.552537+0000","last_clean":"2024-03-06T07:21:40.552537+0000","last_became_active":"2024-03-06T07:21:39.556817+0000","last_became_peered":"2024-03-06T07:21:39.556817+0000","last_unstale":"2024-03-06T07:21:40.552537+0000","last_undegraded":"2024-03-06T07:21:40.552537+0000","last_fullsized":"2024-03-06T07:21:40.552537+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:51.522741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822317+0000","last_change":"2024-03-06T07:21:38.559810+0000","last_active":"2024-03-06T07:21:40.822317+0000","last_peered":"2024-03-06T07:21:40.822317+0000","last_clean":"2024-03-06T07:21:40.822317+0000","last_became_active":"2024-03-06T07:21:38.558988+0000","last_became_peered":"2024-03-06T07:21:38.558988+0000","last_unstale":"2024-03-06T07:21:40.822317+0000","last_undegraded":"2024-03-06T07:21:40.822317+0000","last_fullsized":"2024-03-06T07:21:40.822317+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:39.390652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550014+0000","last_change":"2024-03-06T07:21:39.565269+0000","last_active":"2024-03-06T07:21:40.550014+0000","last_peered":"2024-03-06T07:21:40.550014+0000","last_clean":"2024-03-06T07:21:40.550014+0000","last_became_active":"2024-03-06T07:21:39.565018+0000","last_became_peered":"2024-03-06T07:21:39.565018+0000","last_unstale":"2024-03-06T07:21:40.550014+0000","last_undegraded":"2024-03-06T07:21:40.550014+0000","last_fullsized":"2024-03-06T07:21:40.550014+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:16.320990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.549921+0000","last_change":"2024-03-06T07:21:38.566362+0000","last_active":"2024-03-06T07:21:40.549921+0000","last_peered":"2024-03-06T07:21:40.549921+0000","last_clean":"2024-03-06T07:21:40.549921+0000","last_became_active":"2024-03-06T07:21:38.566024+0000","last_became_peered":"2024-03-06T07:21:38.566024+0000","last_unstale":"2024-03-06T07:21:40.549921+0000","last_undegraded":"2024-03-06T07:21:40.549921+0000","last_fullsized":"2024-03-06T07:21:40.549921+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:23.548190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571970+0000","last_change":"2024-03-06T07:21:39.567036+0000","last_active":"2024-03-06T07:21:40.571970+0000","last_peered":"2024-03-06T07:21:40.571970+0000","last_clean":"2024-03-06T07:21:40.571970+0000","last_became_active":"2024-03-06T07:21:39.566754+0000","last_became_peered":"2024-03-06T07:21:39.566754+0000","last_unstale":"2024-03-06T07:21:40.571970+0000","last_undegraded":"2024-03-06T07:21:40.571970+0000","last_fullsized":"2024-03-06T07:21:40.571970+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:06.114883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"94'8","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.551558+0000","last_change":"2024-03-06T07:21:38.563529+0000","last_active":"2024-03-06T07:21:40.551558+0000","last_peered":"2024-03-06T07:21:40.551558+0000","last_clean":"2024-03-06T07:21:40.551558+0000","last_became_active":"2024-03-06T07:21:38.563228+0000","last_became_peered":"2024-03-06T07:21:38.563228+0000","last_unstale":"2024-03-06T07:21:40.551558+0000","last_undegraded":"2024-03-06T07:21:40.551558+0000","last_fullsized":"2024-03-06T07:21:40.551558+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:02.187205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550072+0000","last_change":"2024-03-06T07:21:39.564325+0000","last_active":"2024-03-06T07:21:40.550072+0000","last_peered":"2024-03-06T07:21:40.550072+0000","last_clean":"2024-03-06T07:21:40.550072+0000","last_became_active":"2024-03-06T07:21:39.561176+0000","last_became_peered":"2024-03-06T07:21:39.561176+0000","last_unstale":"2024-03-06T07:21:40.550072+0000","last_undegraded":"2024-03-06T07:21:40.550072+0000","last_fullsized":"2024-03-06T07:21:40.550072+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:55.272896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552238+0000","last_change":"2024-03-06T07:21:38.558253+0000","last_active":"2024-03-06T07:21:40.552238+0000","last_peered":"2024-03-06T07:21:40.552238+0000","last_clean":"2024-03-06T07:21:40.552238+0000","last_became_active":"2024-03-06T07:21:38.557733+0000","last_became_peered":"2024-03-06T07:21:38.557733+0000","last_unstale":"2024-03-06T07:21:40.552238+0000","last_undegraded":"2024-03-06T07:21:40.552238+0000","last_fullsized":"2024-03-06T07:21:40.552238+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:18:30.608394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552601+0000","last_change":"2024-03-06T07:21:39.557529+0000","last_active":"2024-03-06T07:21:40.552601+0000","last_peered":"2024-03-06T07:21:40.552601+0000","last_clean":"2024-03-06T07:21:40.552601+0000","last_became_active":"2024-03-06T07:21:39.557254+0000","last_became_peered":"2024-03-06T07:21:39.557254+0000","last_unstale":"2024-03-06T07:21:40.552601+0000","last_undegraded":"2024-03-06T07:21:40.552601+0000","last_fullsized":"2024-03-06T07:21:40.552601+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:08.649060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821758+0000","last_change":"2024-03-06T07:21:38.559383+0000","last_active":"2024-03-06T07:21:40.821758+0000","last_peered":"2024-03-06T07:21:40.821758+0000","last_clean":"2024-03-06T07:21:40.821758+0000","last_became_active":"2024-03-06T07:21:38.559112+0000","last_became_peered":"2024-03-06T07:21:38.559112+0000","last_unstale":"2024-03-06T07:21:40.821758+0000","last_undegraded":"2024-03-06T07:21:40.821758+0000","last_fullsized":"2024-03-06T07:21:40.821758+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:13.651998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822620+0000","last_change":"2024-03-06T07:21:39.562687+0000","last_active":"2024-03-06T07:21:40.822620+0000","last_peered":"2024-03-06T07:21:40.822620+0000","last_clean":"2024-03-06T07:21:40.822620+0000","last_became_active":"2024-03-06T07:21:39.562431+0000","last_became_peered":"2024-03-06T07:21:39.562431+0000","last_unstale":"2024-03-06T07:21:40.822620+0000","last_undegraded":"2024-03-06T07:21:40.822620+0000","last_fullsized":"2024-03-06T07:21:40.822620+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:08.649060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.574031+0000","last_change":"2024-03-06T07:21:38.565498+0000","last_active":"2024-03-06T07:21:40.574031+0000","last_peered":"2024-03-06T07:21:40.574031+0000","last_clean":"2024-03-06T07:21:40.574031+0000","last_became_active":"2024-03-06T07:21:38.565232+0000","last_became_peered":"2024-03-06T07:21:38.565232+0000","last_unstale":"2024-03-06T07:21:40.574031+0000","last_undegraded":"2024-03-06T07:21:40.574031+0000","last_fullsized":"2024-03-06T07:21:40.574031+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:52.378445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571702+0000","last_change":"2024-03-06T07:21:39.557441+0000","last_active":"2024-03-06T07:21:40.571702+0000","last_peered":"2024-03-06T07:21:40.571702+0000","last_clean":"2024-03-06T07:21:40.571702+0000","last_became_active":"2024-03-06T07:21:39.557186+0000","last_became_peered":"2024-03-06T07:21:39.557186+0000","last_unstale":"2024-03-06T07:21:40.571702+0000","last_undegraded":"2024-03-06T07:21:40.571702+0000","last_fullsized":"2024-03-06T07:21:40.571702+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:19.892943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550281+0000","last_change":"2024-03-06T07:21:38.560200+0000","last_active":"2024-03-06T07:21:40.550281+0000","last_peered":"2024-03-06T07:21:40.550281+0000","last_clean":"2024-03-06T07:21:40.550281+0000","last_became_active":"2024-03-06T07:21:38.559944+0000","last_became_peered":"2024-03-06T07:21:38.559944+0000","last_unstale":"2024-03-06T07:21:40.550281+0000","last_undegraded":"2024-03-06T07:21:40.550281+0000","last_fullsized":"2024-03-06T07:21:40.550281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:12.347227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571915+0000","last_change":"2024-03-06T07:21:39.562781+0000","last_active":"2024-03-06T07:21:40.571915+0000","last_peered":"2024-03-06T07:21:40.571915+0000","last_clean":"2024-03-06T07:21:40.571915+0000","last_became_active":"2024-03-06T07:21:39.562383+0000","last_became_peered":"2024-03-06T07:21:39.562383+0000","last_unstale":"2024-03-06T07:21:40.571915+0000","last_undegraded":"2024-03-06T07:21:40.571915+0000","last_fullsized":"2024-03-06T07:21:40.571915+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:43.834756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.549919+0000","last_change":"2024-03-06T07:21:38.562192+0000","last_active":"2024-03-06T07:21:40.549919+0000","last_peered":"2024-03-06T07:21:40.549919+0000","last_clean":"2024-03-06T07:21:40.549919+0000","last_became_active":"2024-03-06T07:21:38.561921+0000","last_became_peered":"2024-03-06T07:21:38.561921+0000","last_unstale":"2024-03-06T07:21:40.549919+0000","last_undegraded":"2024-03-06T07:21:40.549919+0000","last_fullsized":"2024-03-06T07:21:40.549919+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:19.887188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822561+0000","last_change":"2024-03-06T07:21:39.560217+0000","last_active":"2024-03-06T07:21:40.822561+0000","last_peered":"2024-03-06T07:21:40.822561+0000","last_clean":"2024-03-06T07:21:40.822561+0000","last_became_active":"2024-03-06T07:21:39.559960+0000","last_became_peered":"2024-03-06T07:21:39.559960+0000","last_unstale":"2024-03-06T07:21:40.822561+0000","last_undegraded":"2024-03-06T07:21:40.822561+0000","last_fullsized":"2024-03-06T07:21:40.822561+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:57.633704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.551909+0000","last_change":"2024-03-06T07:21:38.557631+0000","last_active":"2024-03-06T07:21:40.551909+0000","last_peered":"2024-03-06T07:21:40.551909+0000","last_clean":"2024-03-06T07:21:40.551909+0000","last_became_active":"2024-03-06T07:21:38.557341+0000","last_became_peered":"2024-03-06T07:21:38.557341+0000","last_unstale":"2024-03-06T07:21:40.551909+0000","last_undegraded":"2024-03-06T07:21:40.551909+0000","last_fullsized":"2024-03-06T07:21:40.551909+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:19.715201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550341+0000","last_change":"2024-03-06T07:21:39.566108+0000","last_active":"2024-03-06T07:21:40.550341+0000","last_peered":"2024-03-06T07:21:40.550341+0000","last_clean":"2024-03-06T07:21:40.550341+0000","last_became_active":"2024-03-06T07:21:39.565808+0000","last_became_peered":"2024-03-06T07:21:39.565808+0000","last_unstale":"2024-03-06T07:21:40.550341+0000","last_undegraded":"2024-03-06T07:21:40.550341+0000","last_fullsized":"2024-03-06T07:21:40.550341+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:45.907216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550067+0000","last_change":"2024-03-06T07:21:38.562814+0000","last_active":"2024-03-06T07:21:40.550067+0000","last_peered":"2024-03-06T07:21:40.550067+0000","last_clean":"2024-03-06T07:21:40.550067+0000","last_became_active":"2024-03-06T07:21:38.562548+0000","last_became_peered":"2024-03-06T07:21:38.562548+0000","last_unstale":"2024-03-06T07:21:40.550067+0000","last_undegraded":"2024-03-06T07:21:40.550067+0000","last_fullsized":"2024-03-06T07:21:40.550067+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:43.727793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.571634+0000","last_change":"2024-03-06T07:21:39.555965+0000","last_active":"2024-03-06T07:21:40.571634+0000","last_peered":"2024-03-06T07:21:40.571634+0000","last_clean":"2024-03-06T07:21:40.571634+0000","last_became_active":"2024-03-06T07:21:39.555134+0000","last_became_peered":"2024-03-06T07:21:39.555134+0000","last_unstale":"2024-03-06T07:21:40.571634+0000","last_undegraded":"2024-03-06T07:21:40.571634+0000","last_fullsized":"2024-03-06T07:21:40.571634+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:06.160344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822433+0000","last_change":"2024-03-06T07:21:38.562429+0000","last_active":"2024-03-06T07:21:40.822433+0000","last_peered":"2024-03-06T07:21:40.822433+0000","last_clean":"2024-03-06T07:21:40.822433+0000","last_became_active":"2024-03-06T07:21:38.562148+0000","last_became_peered":"2024-03-06T07:21:38.562148+0000","last_unstale":"2024-03-06T07:21:40.822433+0000","last_undegraded":"2024-03-06T07:21:40.822433+0000","last_fullsized":"2024-03-06T07:21:40.822433+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.631937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.551612+0000","last_change":"2024-03-06T07:21:39.555218+0000","last_active":"2024-03-06T07:21:40.551612+0000","last_peered":"2024-03-06T07:21:40.551612+0000","last_clean":"2024-03-06T07:21:40.551612+0000","last_became_active":"2024-03-06T07:21:39.554928+0000","last_became_peered":"2024-03-06T07:21:39.554928+0000","last_unstale":"2024-03-06T07:21:40.551612+0000","last_undegraded":"2024-03-06T07:21:40.551612+0000","last_fullsized":"2024-03-06T07:21:40.551612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:34.789108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"95'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573196+0000","last_change":"2024-03-06T07:21:38.563141+0000","last_active":"2024-03-06T07:21:40.573196+0000","last_peered":"2024-03-06T07:21:40.573196+0000","last_clean":"2024-03-06T07:21:40.573196+0000","last_became_active":"2024-03-06T07:21:38.562889+0000","last_became_peered":"2024-03-06T07:21:38.562889+0000","last_unstale":"2024-03-06T07:21:40.573196+0000","last_undegraded":"2024-03-06T07:21:40.573196+0000","last_fullsized":"2024-03-06T07:21:40.573196+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:02.848549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822221+0000","last_change":"2024-03-06T07:21:39.562546+0000","last_active":"2024-03-06T07:21:40.822221+0000","last_peered":"2024-03-06T07:21:40.822221+0000","last_clean":"2024-03-06T07:21:40.822221+0000","last_became_active":"2024-03-06T07:21:39.562279+0000","last_became_peered":"2024-03-06T07:21:39.562279+0000","last_unstale":"2024-03-06T07:21:40.822221+0000","last_undegraded":"2024-03-06T07:21:40.822221+0000","last_fullsized":"2024-03-06T07:21:40.822221+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:57:55.843723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552644+0000","last_change":"2024-03-06T07:21:38.559995+0000","last_active":"2024-03-06T07:21:40.552644+0000","last_peered":"2024-03-06T07:21:40.552644+0000","last_clean":"2024-03-06T07:21:40.552644+0000","last_became_active":"2024-03-06T07:21:38.559713+0000","last_became_peered":"2024-03-06T07:21:38.559713+0000","last_unstale":"2024-03-06T07:21:40.552644+0000","last_undegraded":"2024-03-06T07:21:40.552644+0000","last_fullsized":"2024-03-06T07:21:40.552644+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:02.068806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573877+0000","last_change":"2024-03-06T07:21:39.569139+0000","last_active":"2024-03-06T07:21:40.573877+0000","last_peered":"2024-03-06T07:21:40.573877+0000","last_clean":"2024-03-06T07:21:40.573877+0000","last_became_active":"2024-03-06T07:21:39.568825+0000","last_became_peered":"2024-03-06T07:21:39.568825+0000","last_unstale":"2024-03-06T07:21:40.573877+0000","last_undegraded":"2024-03-06T07:21:40.573877+0000","last_fullsized":"2024-03-06T07:21:40.573877+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:30.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.822374+0000","last_change":"2024-03-06T07:21:38.565594+0000","last_active":"2024-03-06T07:21:40.822374+0000","last_peered":"2024-03-06T07:21:40.822374+0000","last_clean":"2024-03-06T07:21:40.822374+0000","last_became_active":"2024-03-06T07:21:38.565309+0000","last_became_peered":"2024-03-06T07:21:38.565309+0000","last_unstale":"2024-03-06T07:21:40.822374+0000","last_undegraded":"2024-03-06T07:21:40.822374+0000","last_fullsized":"2024-03-06T07:21:40.822374+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:05.130997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550257+0000","last_change":"2024-03-06T07:21:39.566116+0000","last_active":"2024-03-06T07:21:40.550257+0000","last_peered":"2024-03-06T07:21:40.550257+0000","last_clean":"2024-03-06T07:21:40.550257+0000","last_became_active":"2024-03-06T07:21:39.565781+0000","last_became_peered":"2024-03-06T07:21:39.565781+0000","last_unstale":"2024-03-06T07:21:40.550257+0000","last_undegraded":"2024-03-06T07:21:40.550257+0000","last_fullsized":"2024-03-06T07:21:40.550257+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:30.709573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552017+0000","last_change":"2024-03-06T07:21:38.558434+0000","last_active":"2024-03-06T07:21:40.552017+0000","last_peered":"2024-03-06T07:21:40.552017+0000","last_clean":"2024-03-06T07:21:40.552017+0000","last_became_active":"2024-03-06T07:21:38.558170+0000","last_became_peered":"2024-03-06T07:21:38.558170+0000","last_unstale":"2024-03-06T07:21:40.552017+0000","last_undegraded":"2024-03-06T07:21:40.552017+0000","last_fullsized":"2024-03-06T07:21:40.552017+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:45.519268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550075+0000","last_change":"2024-03-06T07:21:39.568656+0000","last_active":"2024-03-06T07:21:40.550075+0000","last_peered":"2024-03-06T07:21:40.550075+0000","last_clean":"2024-03-06T07:21:40.550075+0000","last_became_active":"2024-03-06T07:21:39.568146+0000","last_became_peered":"2024-03-06T07:21:39.568146+0000","last_unstale":"2024-03-06T07:21:40.550075+0000","last_undegraded":"2024-03-06T07:21:40.550075+0000","last_fullsized":"2024-03-06T07:21:40.550075+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:15.274387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550157+0000","last_change":"2024-03-06T07:21:38.565679+0000","last_active":"2024-03-06T07:21:40.550157+0000","last_peered":"2024-03-06T07:21:40.550157+0000","last_clean":"2024-03-06T07:21:40.550157+0000","last_became_active":"2024-03-06T07:21:38.565398+0000","last_became_peered":"2024-03-06T07:21:38.565398+0000","last_unstale":"2024-03-06T07:21:40.550157+0000","last_undegraded":"2024-03-06T07:21:40.550157+0000","last_fullsized":"2024-03-06T07:21:40.550157+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:41.889616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821818+0000","last_change":"2024-03-06T07:21:39.577625+0000","last_active":"2024-03-06T07:21:40.821818+0000","last_peered":"2024-03-06T07:21:40.821818+0000","last_clean":"2024-03-06T07:21:40.821818+0000","last_became_active":"2024-03-06T07:21:39.577351+0000","last_became_peered":"2024-03-06T07:21:39.577351+0000","last_unstale":"2024-03-06T07:21:40.821818+0000","last_undegraded":"2024-03-06T07:21:40.821818+0000","last_fullsized":"2024-03-06T07:21:40.821818+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:28.742450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.552016+0000","last_change":"2024-03-06T07:21:38.559216+0000","last_active":"2024-03-06T07:21:40.552016+0000","last_peered":"2024-03-06T07:21:40.552016+0000","last_clean":"2024-03-06T07:21:40.552016+0000","last_became_active":"2024-03-06T07:21:38.558963+0000","last_became_peered":"2024-03-06T07:21:38.558963+0000","last_unstale":"2024-03-06T07:21:40.552016+0000","last_undegraded":"2024-03-06T07:21:40.552016+0000","last_fullsized":"2024-03-06T07:21:40.552016+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:24.414184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573976+0000","last_change":"2024-03-06T07:21:39.574070+0000","last_active":"2024-03-06T07:21:40.573976+0000","last_peered":"2024-03-06T07:21:40.573976+0000","last_clean":"2024-03-06T07:21:40.573976+0000","last_became_active":"2024-03-06T07:21:39.573810+0000","last_became_peered":"2024-03-06T07:21:39.573810+0000","last_unstale":"2024-03-06T07:21:40.573976+0000","last_undegraded":"2024-03-06T07:21:40.573976+0000","last_fullsized":"2024-03-06T07:21:40.573976+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:40.396311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550162+0000","last_change":"2024-03-06T07:21:38.565658+0000","last_active":"2024-03-06T07:21:40.550162+0000","last_peered":"2024-03-06T07:21:40.550162+0000","last_clean":"2024-03-06T07:21:40.550162+0000","last_became_active":"2024-03-06T07:21:38.565359+0000","last_became_peered":"2024-03-06T07:21:38.565359+0000","last_unstale":"2024-03-06T07:21:40.550162+0000","last_undegraded":"2024-03-06T07:21:40.550162+0000","last_fullsized":"2024-03-06T07:21:40.550162+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:02.434383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.573721+0000","last_change":"2024-03-06T07:21:39.564333+0000","last_active":"2024-03-06T07:21:40.573721+0000","last_peered":"2024-03-06T07:21:40.573721+0000","last_clean":"2024-03-06T07:21:40.573721+0000","last_became_active":"2024-03-06T07:21:39.564054+0000","last_became_peered":"2024-03-06T07:21:39.564054+0000","last_unstale":"2024-03-06T07:21:40.573721+0000","last_undegraded":"2024-03-06T07:21:40.573721+0000","last_fullsized":"2024-03-06T07:21:40.573721+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.539243+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:08.649060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.550250+0000","last_change":"2024-03-06T07:21:38.564490+0000","last_active":"2024-03-06T07:21:40.550250+0000","last_peered":"2024-03-06T07:21:40.550250+0000","last_clean":"2024-03-06T07:21:40.550250+0000","last_became_active":"2024-03-06T07:21:38.564221+0000","last_became_peered":"2024-03-06T07:21:38.564221+0000","last_unstale":"2024-03-06T07:21:40.550250+0000","last_undegraded":"2024-03-06T07:21:40.550250+0000","last_fullsized":"2024-03-06T07:21:40.550250+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:37.533584+0000","last_clean_scrub_stamp":"2024-03-06T07:21:37.533584+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:08.253097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":312,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-06T07:21:40.821467+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:21:40.821467+0000","last_peered":"2024-03-06T07:21:40.821467+0000","last_clean":"2024-03-06T07:21:40.821467+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:21:40.821467+0000","last_undegraded":"2024-03-06T07:21:40.821467+0000","last_fullsized":"2024-03-06T07:21:40.821467+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32408,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339432,"statfs":{"total":96636764160,"available":96603578368,"internally_reserved":0,"allocated":1253376,"data_stored":1159156,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574894,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32352,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339488,"statfs":{"total":96636764160,"available":96603635712,"internally_reserved":0,"allocated":1200128,"data_stored":570844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1179648,"data_stored":568628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1187840,"data_stored":568720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":569210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":568662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":568662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1261568,"data_stored":1160698,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:42.858 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:21:42.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:21:43.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.000+0000 7f57a961c640 1 Processor -- start 2024-03-06T07:21:43.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.000+0000 7f57a961c640 1 -- start start 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a961c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a414e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a961c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a40b0450 con 0x7f57a4155c30 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a961c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a41560e0 con 0x7f57a40b0070 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a961c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a406c020 con 0x7f57a406ca70 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a27fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f57a4155c30 0x7f57a40c5380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54296/0 (socket says 172.21.15.94:54296) 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a27fc640 1 -- 172.21.15.94:0/1626587110 learned_addr learned my addr 172.21.15.94:0/1626587110 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a37fe640 1 --2- 172.21.15.94:0/1626587110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a1ffb640 1 -- 172.21.15.94:0/1626587110 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641325980 0 0) 0x7f5798001110 con 0x7f57a4155c30 2024-03-06T07:21:43.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a1ffb640 1 -- 172.21.15.94:0/1626587110 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f578c003520 con 0x7f57a4155c30 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a1ffb640 1 -- 172.21.15.94:0/1626587110 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713767935 0 0) 0x7f57a41560e0 con 0x7f57a40b0070 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a37fe640 1 -- 172.21.15.94:0/1626587110 >> v1:172.21.15.145:6789/0 conn(0x7f57a40b0070 legacy=0x7f57a40c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a37fe640 1 -- 172.21.15.94:0/1626587110 >> v1:172.21.15.94:6789/0 conn(0x7f57a4155c30 legacy=0x7f57a40c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.001+0000 7f57a37fe640 1 -- 172.21.15.94:0/1626587110 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a40c77f0 con 0x7f57a406ca70 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.002+0000 7f57a37fe640 1 --2- 172.21.15.94:0/1626587110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a414e800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f57a41562b0 tx=0x7f579c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0079e5b683facc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.002+0000 7f57a1ffb640 1 -- 172.21.15.94:0/1626587110 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c039070 con 0x7f57a406ca70 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.002+0000 7f57a1ffb640 1 -- 172.21.15.94:0/1626587110 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579c02cc40 con 0x7f57a406ca70 2024-03-06T07:21:43.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.002+0000 7f57a1ffb640 1 -- 172.21.15.94:0/1626587110 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579c034440 con 0x7f57a406ca70 2024-03-06T07:21:43.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.003+0000 7f57a961c640 1 -- 172.21.15.94:0/1626587110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 msgr2=0x7f57a414e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.003+0000 7f57a961c640 1 --2- 172.21.15.94:0/1626587110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a414e800 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f57a41562b0 tx=0x7f579c02be90 comp rx=0 tx=0).stop 2024-03-06T07:21:43.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.003+0000 7f57a961c640 1 -- 172.21.15.94:0/1626587110 shutdown_connections 2024-03-06T07:21:43.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.003+0000 7f57a961c640 1 --2- 172.21.15.94:0/1626587110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a414e800 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.003+0000 7f57a961c640 1 -- 172.21.15.94:0/1626587110 >> 172.21.15.94:0/1626587110 conn(0x7f57a40ac590 msgr2=0x7f57a40ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.003+0000 7f57a961c640 1 -- 172.21.15.94:0/1626587110 shutdown_connections 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.003+0000 7f57a961c640 1 -- 172.21.15.94:0/1626587110 wait complete. 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.004+0000 7f57a961c640 1 Processor -- start 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.004+0000 7f57a961c640 1 -- start start 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a961c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a41eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a961c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57a40b0070 0x7f57a40c0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a961c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57a4155c30 0x7f57a4150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a961c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a406c550 con 0x7f57a40b0070 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a961c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a406c270 con 0x7f57a4155c30 2024-03-06T07:21:43.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a961c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a406da80 con 0x7f57a406ca70 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57a40b0070 0x7f57a40c0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57a40b0070 0x7f57a40c0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48876/0 (socket says 172.21.15.94:48876) 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 -- 172.21.15.94:0/135338700 learned_addr learned my addr 172.21.15.94:0/135338700 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a2ffd640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a41eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a37fe640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57a4155c30 0x7f57a4150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 -- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 msgr2=0x7f57a41eaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a41eaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 -- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57a4155c30 msgr2=0x7f57a4150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57a4155c30 0x7f57a4150eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 -- 172.21.15.94:0/135338700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a40c4460 con 0x7f57a40b0070 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.005+0000 7f57a27fc640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57a40b0070 0x7f57a40c0320 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f579800c3a0 tx=0x7f579800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.006+0000 7f578b7fe640 1 -- 172.21.15.94:0/135338700 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798017070 con 0x7f57a40b0070 2024-03-06T07:21:43.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.006+0000 7f578b7fe640 1 -- 172.21.15.94:0/135338700 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5798005ce0 con 0x7f57a40b0070 2024-03-06T07:21:43.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.006+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a40c4740 con 0x7f57a40b0070 2024-03-06T07:21:43.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.006+0000 7f578b7fe640 1 -- 172.21.15.94:0/135338700 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798007310 con 0x7f57a40b0070 2024-03-06T07:21:43.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.006+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a40c4d00 con 0x7f57a40b0070 2024-03-06T07:21:43.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.007+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a404f330 con 0x7f57a40b0070 2024-03-06T07:21:43.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.007+0000 7f578b7fe640 1 -- 172.21.15.94:0/135338700 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f579801a070 con 0x7f57a40b0070 2024-03-06T07:21:43.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.007+0000 7f578b7fe640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f579006b500 0x7f579006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.008+0000 7f578b7fe640 1 -- 172.21.15.94:0/135338700 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f57980896f0 con 0x7f57a40b0070 2024-03-06T07:21:43.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.008+0000 7f57a2ffd640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f579006b500 0x7f579006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.008+0000 7f57a2ffd640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f579006b500 0x7f579006d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5794000f60 tx=0x7f5794009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.012+0000 7f578b7fe640 1 -- 172.21.15.94:0/135338700 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57a40b0450 con 0x7f57a40b0070 2024-03-06T07:21:43.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.249+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f57a406d860 con 0x7f57a40b0070 2024-03-06T07:21:43.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.250+0000 7f578b7fe640 1 -- 172.21.15.94:0/135338700 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5798057d90 con 0x7f57a40b0070 2024-03-06T07:21:43.252 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:21:43.252 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:21:43.252 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:21:43.253 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:21:43.253 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:21:43.253 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:21:43.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f579006b500 msgr2=0x7f579006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f579006b500 0x7f579006d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5794000f60 tx=0x7f5794009040 comp rx=0 tx=0).stop 2024-03-06T07:21:43.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57a40b0070 msgr2=0x7f57a40c0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57a40b0070 0x7f57a40c0320 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f579800c3a0 tx=0x7f579800c860 comp rx=0 tx=0).stop 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 shutdown_connections 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f579006b500 0x7f579006d9c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f57a4155c30 0x7f57a4150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57a40b0070 0x7f57a40c0320 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 --2- 172.21.15.94:0/135338700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57a406ca70 0x7f57a41eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 >> 172.21.15.94:0/135338700 conn(0x7f57a40ac590 msgr2=0x7f57a40c97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 shutdown_connections 2024-03-06T07:21:43.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.252+0000 7f57a961c640 1 -- 172.21.15.94:0/135338700 wait complete. 2024-03-06T07:21:43.265 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:43.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.363+0000 7f56020fb640 1 Processor -- start 2024-03-06T07:21:43.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.363+0000 7f56020fb640 1 -- start start 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f56020fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc155c80 0x7f55fc06ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f56020fb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55fc0720a0 0x7f55fc06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f56020fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0c1ae0 con 0x7f55fc152520 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f56020fb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55fc0c1dc0 con 0x7f55fc0720a0 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f56020fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc06e450 con 0x7f55fc155c80 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55fc0720a0 0x7f55fc06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f55fc152520 0x7f55fc0c12b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54300/0 (socket says 172.21.15.94:54300) 2024-03-06T07:21:43.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fb7fe640 1 -- 172.21.15.94:0/1507709304 learned_addr learned my addr 172.21.15.94:0/1507709304 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55faffd640 1 --2- 172.21.15.94:0/1507709304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc155c80 0x7f55fc06ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fbfff640 1 -- 172.21.15.94:0/1507709304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc155c80 msgr2=0x7f55fc06ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fbfff640 1 --2- 172.21.15.94:0/1507709304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc155c80 0x7f55fc06ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fbfff640 1 -- 172.21.15.94:0/1507709304 >> v1:172.21.15.94:6789/0 conn(0x7f55fc152520 legacy=0x7f55fc0c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fa7fc640 1 -- 172.21.15.94:0/1507709304 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466293206 0 0) 0x7f55fc0c1ae0 con 0x7f55fc152520 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.364+0000 7f55fbfff640 1 -- 172.21.15.94:0/1507709304 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc06e730 con 0x7f55fc0720a0 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.365+0000 7f55fbfff640 1 --2- 172.21.15.94:0/1507709304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc0720a0 0x7f55fc06afc0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009d90 tx=0x7f55f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b3563f74fc250a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.365+0000 7f55fa7fc640 1 -- 172.21.15.94:0/1507709304 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4039070 con 0x7f55fc0720a0 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.365+0000 7f55fa7fc640 1 -- 172.21.15.94:0/1507709304 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f402cc50 con 0x7f55fc0720a0 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.365+0000 7f55fa7fc640 1 -- 172.21.15.94:0/1507709304 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4034440 con 0x7f55fc0720a0 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 -- 172.21.15.94:0/1507709304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc0720a0 msgr2=0x7f55fc06afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 --2- 172.21.15.94:0/1507709304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc0720a0 0x7f55fc06afc0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009d90 tx=0x7f55f402bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 -- 172.21.15.94:0/1507709304 shutdown_connections 2024-03-06T07:21:43.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 --2- 172.21.15.94:0/1507709304 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc0720a0 0x7f55fc06afc0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 --2- 172.21.15.94:0/1507709304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc155c80 0x7f55fc06ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 -- 172.21.15.94:0/1507709304 >> 172.21.15.94:0/1507709304 conn(0x7f55fc0c76b0 msgr2=0x7f55fc0c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 -- 172.21.15.94:0/1507709304 shutdown_connections 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.366+0000 7f56020fb640 1 -- 172.21.15.94:0/1507709304 wait complete. 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.367+0000 7f56020fb640 1 Processor -- start 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.367+0000 7f56020fb640 1 -- start start 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f56020fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc0720a0 0x7f55fc147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f56020fb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55fc152520 0x7f55fc14d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f56020fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc155c80 0x7f55fc1575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f56020fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc152a80 con 0x7f55fc155c80 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f56020fb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc152900 con 0x7f55fc152520 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f56020fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc06b500 con 0x7f55fc0720a0 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc155c80 0x7f55fc1575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc155c80 0x7f55fc1575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48904/0 (socket says 172.21.15.94:48904) 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 -- 172.21.15.94:0/196042645 learned_addr learned my addr 172.21.15.94:0/196042645 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fb7fe640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc0720a0 0x7f55fc147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55faffd640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55fc152520 0x7f55fc14d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 -- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc0720a0 msgr2=0x7f55fc147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc0720a0 0x7f55fc147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 -- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55fc152520 msgr2=0x7f55fc14d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55fc152520 0x7f55fc14d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 -- 172.21.15.94:0/196042645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc148650 con 0x7f55fc155c80 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.368+0000 7f55fbfff640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc155c80 0x7f55fc1575d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009d90 tx=0x7f55f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.369+0000 7f55e3fff640 1 -- 172.21.15.94:0/196042645 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4044020 con 0x7f55fc155c80 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.369+0000 7f55e3fff640 1 -- 172.21.15.94:0/196042645 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f403a040 con 0x7f55fc155c80 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.369+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc1479b0 con 0x7f55fc155c80 2024-03-06T07:21:43.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.369+0000 7f55e3fff640 1 -- 172.21.15.94:0/196042645 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4039050 con 0x7f55fc155c80 2024-03-06T07:21:43.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.369+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc156af0 con 0x7f55fc155c80 2024-03-06T07:21:43.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.370+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc069460 con 0x7f55fc155c80 2024-03-06T07:21:43.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.370+0000 7f55e3fff640 1 -- 172.21.15.94:0/196042645 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55f4003990 con 0x7f55fc155c80 2024-03-06T07:21:43.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.370+0000 7f55e3fff640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55e806b4d0 0x7f55e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.370+0000 7f55e3fff640 1 -- 172.21.15.94:0/196042645 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55f40ac5b0 con 0x7f55fc155c80 2024-03-06T07:21:43.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.371+0000 7f55fb7fe640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55e806b4d0 0x7f55e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.371+0000 7f55fb7fe640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55e806b4d0 0x7f55e806d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f55ec000f60 tx=0x7f55ec007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.375+0000 7f55e3fff640 1 -- 172.21.15.94:0/196042645 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55fc069460 con 0x7f55fc155c80 2024-03-06T07:21:43.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.583+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55fc0c0b70 con 0x7f55fc155c80 2024-03-06T07:21:43.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.584+0000 7f55e3fff640 1 -- 172.21.15.94:0/196042645 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4988 (secure 0 0 0) 0x7f55f404e060 con 0x7f55fc155c80 2024-03-06T07:21:43.586 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:43.586 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:40.899292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35900},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35900":{"gid":35900,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/3774173957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3774173957},{"type":"v1","addr":"172.21.15.145:6836","nonce":3774173957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35900,"qdb_cluster":[35900]},"id":7}]} 2024-03-06T07:21:43.586 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 77 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55e806b4d0 msgr2=0x7f55e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55e806b4d0 0x7f55e806d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f55ec000f60 tx=0x7f55ec007d50 comp rx=0 tx=0).stop 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc155c80 msgr2=0x7f55fc1575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc155c80 0x7f55fc1575d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009d90 tx=0x7f55f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 shutdown_connections 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55e806b4d0 0x7f55e806d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55fc155c80 0x7f55fc1575d0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55fc152520 0x7f55fc14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 --2- 172.21.15.94:0/196042645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55fc0720a0 0x7f55fc147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 >> 172.21.15.94:0/196042645 conn(0x7f55fc0c76b0 msgr2=0x7f55fc151d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 shutdown_connections 2024-03-06T07:21:43.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:43.586+0000 7f56020fb640 1 -- 172.21.15.94:0/196042645 wait complete. 2024-03-06T07:21:44.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:38.904104+0000', 'modified': '2024-03-06T07:21:40.899292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 35900}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_35900': {'gid': 35900, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/3774173957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3774173957}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3774173957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 35900, 'qdb_cluster': [35900]} 2024-03-06T07:21:44.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:44.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:21:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.102+0000 7f631c839640 1 Processor -- start 2024-03-06T07:21:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.102+0000 7f631c839640 1 -- start start 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631c839640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f63181561a0 0x7f63181529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631c839640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f631814f110 0x7f631814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631c839640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63180a9990 con 0x7f63180c6ab0 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631c839640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6318155d10 con 0x7f631814f110 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631c839640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63181556c0 con 0x7f63181561a0 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f6316d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f631814f110 0x7f631814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f63180c6ab0 0x7f63180a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54308/0 (socket says 172.21.15.94:54308) 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631659c640 1 -- 172.21.15.94:0/3423426394 learned_addr learned my addr 172.21.15.94:0/3423426394 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f6316d9d640 1 --2- 172.21.15.94:0/3423426394 >> v2:172.21.15.94:3300/0 conn(0x7f631814f110 0x7f631814f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48914/0 (socket says 172.21.15.94:48914) 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f6315d9b640 1 --2- 172.21.15.94:0/3423426394 >> v2:172.21.15.145:3300/0 conn(0x7f63181561a0 0x7f63181529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f6316d9d640 1 -- 172.21.15.94:0/3423426394 >> v2:172.21.15.145:3300/0 conn(0x7f63181561a0 msgr2=0x7f63181529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f6316d9d640 1 --2- 172.21.15.94:0/3423426394 >> v2:172.21.15.145:3300/0 conn(0x7f63181561a0 0x7f63181529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f6316d9d640 1 -- 172.21.15.94:0/3423426394 >> v1:172.21.15.94:6789/0 conn(0x7f63180c6ab0 legacy=0x7f63180a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f6316d9d640 1 -- 172.21.15.94:0/3423426394 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63181559a0 con 0x7f631814f110 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.103+0000 7f631559a640 1 -- 172.21.15.94:0/3423426394 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931418388 0 0) 0x7f63180a9990 con 0x7f63180c6ab0 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.104+0000 7f6316d9d640 1 --2- 172.21.15.94:0/3423426394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f631814f110 0x7f631814f5a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f63180c0b80 tx=0x7f631002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b27ee08885aee87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.104+0000 7f631559a640 1 -- 172.21.15.94:0/3423426394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631002e9d0 con 0x7f631814f110 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.104+0000 7f631559a640 1 -- 172.21.15.94:0/3423426394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631002eb70 con 0x7f631814f110 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.104+0000 7f631559a640 1 -- 172.21.15.94:0/3423426394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310009540 con 0x7f631814f110 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 -- 172.21.15.94:0/3423426394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f631814f110 msgr2=0x7f631814f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 --2- 172.21.15.94:0/3423426394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f631814f110 0x7f631814f5a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f63180c0b80 tx=0x7f631002e030 comp rx=0 tx=0).stop 2024-03-06T07:21:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 -- 172.21.15.94:0/3423426394 shutdown_connections 2024-03-06T07:21:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 --2- 172.21.15.94:0/3423426394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f631814f110 0x7f631814f5a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 --2- 172.21.15.94:0/3423426394 >> v2:172.21.15.145:3300/0 conn(0x7f63181561a0 0x7f63181529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 -- 172.21.15.94:0/3423426394 >> 172.21.15.94:0/3423426394 conn(0x7f631806b1a0 msgr2=0x7f631806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 -- 172.21.15.94:0/3423426394 shutdown_connections 2024-03-06T07:21:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.105+0000 7f631c839640 1 -- 172.21.15.94:0/3423426394 wait complete. 2024-03-06T07:21:44.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.106+0000 7f631c839640 1 Processor -- start 2024-03-06T07:21:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.106+0000 7f631c839640 1 -- start start 2024-03-06T07:21:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631c839640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63180c6ab0 0x7f63181513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631c839640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f631814f110 0x7f6318142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631c839640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63181561a0 0x7f6318141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631c839640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63180aa600 con 0x7f63180c6ab0 2024-03-06T07:21:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631c839640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63180c2a90 con 0x7f63181561a0 2024-03-06T07:21:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631c839640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6318154880 con 0x7f631814f110 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63180c6ab0 0x7f63181513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63180c6ab0 0x7f63181513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48916/0 (socket says 172.21.15.94:48916) 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 -- 172.21.15.94:0/3620472123 learned_addr learned my addr 172.21.15.94:0/3620472123 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f6316d9d640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63181561a0 0x7f6318141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f6315d9b640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f631814f110 0x7f6318142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 -- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f631814f110 msgr2=0x7f6318142960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f631814f110 0x7f6318142960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 -- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63181561a0 msgr2=0x7f6318141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63181561a0 0x7f6318141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 -- 172.21.15.94:0/3620472123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6318154b60 con 0x7f63180c6ab0 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f6315d9b640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f631814f110 0x7f6318142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f6316d9d640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63181561a0 0x7f6318141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631659c640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63180c6ab0 0x7f63181513b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f630800c430 tx=0x7f630800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f6306ffd640 1 -- 172.21.15.94:0/3620472123 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63080074e0 con 0x7f63180c6ab0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f6306ffd640 1 -- 172.21.15.94:0/3620472123 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63080047e0 con 0x7f63180c6ab0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6318154e40 con 0x7f63180c6ab0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.107+0000 7f6306ffd640 1 -- 172.21.15.94:0/3620472123 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308004bd0 con 0x7f63180c6ab0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.108+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6318155400 con 0x7f63180c6ab0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.109+0000 7f6306ffd640 1 -- 172.21.15.94:0/3620472123 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6308011070 con 0x7f63180c6ab0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.109+0000 7f6306ffd640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f62f806b4d0 0x7f62f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.110+0000 7f6306ffd640 1 -- 172.21.15.94:0/3620472123 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6308089f70 con 0x7f63180c6ab0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.110+0000 7f6315d9b640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f62f806b4d0 0x7f62f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.110+0000 7f6315d9b640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f62f806b4d0 0x7f62f806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f631806b990 tx=0x7f630c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.110+0000 7f631f426740 1 Processor -- start 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631f426740 1 -- start start 2024-03-06T07:21:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f5523a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631f426740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5523f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f55244b0 0x5626f5520680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631f426740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f548a6b0 con 0x5626f5526330 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f5433a60 con 0x5626f55244b0 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f5486750 con 0x5626f5525f50 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f6315d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5523f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f6315d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5523f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48920/0 (socket says 172.21.15.94:48920) 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f6315d9b640 1 -- 172.21.15.94:0/1034038557 learned_addr learned my addr 172.21.15.94:0/1034038557 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f6316d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f55244b0 0x5626f5520680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.111+0000 7f631659c640 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f5523a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f6315d9b640 1 -- 172.21.15.94:0/1034038557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 msgr2=0x5626f5523a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f6315d9b640 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f5523a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f6315d9b640 1 -- 172.21.15.94:0/1034038557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f55244b0 msgr2=0x5626f5520680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f6315d9b640 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f55244b0 0x5626f5520680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f6315d9b640 1 -- 172.21.15.94:0/1034038557 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f5451f30 con 0x5626f5526330 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f6315d9b640 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5523f70 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f630c011e30 tx=0x7f630c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b1f960a20fd6eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f62f7fff640 1 -- 172.21.15.94:0/1034038557 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c011e60 con 0x5626f5526330 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f62f7fff640 1 -- 172.21.15.94:0/1034038557 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c026070 con 0x5626f5526330 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.112+0000 7f62f7fff640 1 -- 172.21.15.94:0/1034038557 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c021640 con 0x5626f5526330 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 -- 172.21.15.94:0/1034038557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 msgr2=0x5626f5523f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5523f70 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f630c011e30 tx=0x7f630c014000 comp rx=0 tx=0).stop 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 -- 172.21.15.94:0/1034038557 shutdown_connections 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f55244b0 0x5626f5520680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5523f70 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 --2- 172.21.15.94:0/1034038557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f5523a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 -- 172.21.15.94:0/1034038557 >> 172.21.15.94:0/1034038557 conn(0x5626f55227d0 msgr2=0x5626f55473c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 -- 172.21.15.94:0/1034038557 shutdown_connections 2024-03-06T07:21:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 -- 172.21.15.94:0/1034038557 wait complete. 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 -- start start 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.113+0000 7f631f426740 1 Processor -- start 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f55467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631f426740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5549010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f554a490 0x5626f554a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631f426740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f5548230 con 0x5626f5526330 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f5446ea0 con 0x5626f554a490 2024-03-06T07:21:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f5547c70 con 0x5626f5525f50 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631f426740 10 client.0 _ll_register_callbacks cb 0x5626f5446870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f6315d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5549010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f6315d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5549010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48924/0 (socket says 172.21.15.94:48924) 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f6315d9b640 1 -- 172.21.15.94:0/1782031232 learned_addr learned my addr 172.21.15.94:0/1782031232 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f631659c640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f55467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.114+0000 7f6316d9d640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f554a490 0x5626f554a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631659c640 1 -- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f554a490 msgr2=0x5626f554a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631659c640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f554a490 0x5626f554a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631659c640 1 -- 172.21.15.94:0/1782031232 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 msgr2=0x5626f5549010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631659c640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5549010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631659c640 1 -- 172.21.15.94:0/1782031232 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626f5447270 con 0x5626f5525f50 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f6315d9b640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5549010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631659c640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f55467b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6308088370 tx=0x7f6308058cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f12e4f7e6a2e247 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f62f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.145:3301/0 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f62f57fa640 1 -- 172.21.15.94:0/1782031232 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630805c740 con 0x5626f5525f50 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f62f57fa640 1 -- 172.21.15.94:0/1782031232 <== mon.2 v2:172.21.15.145:3301/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6308057a40 con 0x5626f5525f50 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f630c026070 con 0x5626f5525f50 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f62f57fa640 1 -- 172.21.15.94:0/1782031232 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6308060d00 con 0x5626f5525f50 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 10 client.35972 fetch_fsmap learned FSMap version 77 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x5626f519f2d0 con 0x5626f5525f50 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f62f57fa640 1 -- 172.21.15.94:0/1782031232 <== mon.2 v2:172.21.15.145:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c026070 con 0x5626f5525f50 2024-03-06T07:21:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f62f57fa640 1 -- 172.21.15.94:0/1782031232 <== mon.2 v2:172.21.15.145:3301/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6308054340 con 0x5626f5525f50 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 10 client.35972 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 10 client.35972 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 20 client.35972 populate_metadata read hostname 'smithi094' 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55766a0 0x5626f5581260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 4 client.35972 mds_command: new command op to 35900 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.115+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 --> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626f5497bf0 con 0x5626f55766a0 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.116+0000 7f6315d9b640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55766a0 0x5626f5581260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.116+0000 7f62f57fa640 10 client.35972 ms_handle_connect on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.116+0000 7f6315d9b640 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55766a0 0x5626f5581260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ab3490436322ee8 server_cookie=73ec098d4b1fea8e in_seq=0 out_seq=0 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.116+0000 7f62f57fa640 1 -- 172.21.15.94:0/1782031232 <== mds.0 v2:172.21.15.145:6834/3774173957 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5626f5497bf0 con 0x5626f55766a0 2024-03-06T07:21:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.116+0000 7f62f57fa640 10 client.35972 handle_command_reply: tid=0 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.116+0000 7f631f426740 1 client.35972 shutdown 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 msgr2=0x5626f55467b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f55467b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6308088370 tx=0x7f6308058cf0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 shutdown_connections 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55766a0 0x5626f5581260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f554a490 0x5626f554a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5526330 0x5626f5549010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 --2- 172.21.15.94:0/1782031232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5525f50 0x5626f55467b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f62f57fa640 0 client.35972 ms_handle_reset on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 >> 172.21.15.94:0/1782031232 conn(0x5626f55244b0 msgr2=0x5626f55248e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 shutdown_connections 2024-03-06T07:21:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 1 -- 172.21.15.94:0/1782031232 wait complete. 2024-03-06T07:21:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.117+0000 7f631f426740 20 client.35972 trim_cache size 0 max 16384 2024-03-06T07:21:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.133+0000 7f631f426740 1 Processor -- start 2024-03-06T07:21:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.133+0000 7f631f426740 1 -- start start 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631f426740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f551f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f5520f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55214c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631f426740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63181f27d0 con 0x5626f5525f50 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63181ea6a0 con 0x5626f5526330 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63181f2250 con 0x5626f554a490 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f551f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f551f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48932/0 (socket says 172.21.15.94:48932) 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 -- 172.21.15.94:0/3614945294 learned_addr learned my addr 172.21.15.94:0/3614945294 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f6315d9b640 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f5520f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f6316d9d640 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55214c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 -- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 msgr2=0x5626f55214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55214c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 -- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 msgr2=0x5626f5520f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f5520f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 -- 172.21.15.94:0/3614945294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f52696b0 con 0x5626f5525f50 2024-03-06T07:21:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.134+0000 7f631659c640 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f551f510 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6308054a50 tx=0x7f630806e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd7224d8abdd87d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f62f77fe640 1 -- 172.21.15.94:0/3614945294 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630805c740 con 0x5626f5525f50 2024-03-06T07:21:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f62f77fe640 1 -- 172.21.15.94:0/3614945294 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6308070030 con 0x5626f5525f50 2024-03-06T07:21:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f62f77fe640 1 -- 172.21.15.94:0/3614945294 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308057a40 con 0x5626f5525f50 2024-03-06T07:21:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 -- 172.21.15.94:0/3614945294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 msgr2=0x5626f551f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f551f510 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6308054a50 tx=0x7f630806e000 comp rx=0 tx=0).stop 2024-03-06T07:21:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 -- 172.21.15.94:0/3614945294 shutdown_connections 2024-03-06T07:21:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55214c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f5520f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 --2- 172.21.15.94:0/3614945294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f551f510 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 -- 172.21.15.94:0/3614945294 >> 172.21.15.94:0/3614945294 conn(0x5626f55244b0 msgr2=0x5626f55432c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 -- 172.21.15.94:0/3614945294 shutdown_connections 2024-03-06T07:21:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.135+0000 7f631f426740 1 -- 172.21.15.94:0/3614945294 wait complete. 2024-03-06T07:21:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 -- start start 2024-03-06T07:21:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 Processor -- start 2024-03-06T07:21:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f5547180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f55493a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55d1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f554c930 con 0x5626f5525f50 2024-03-06T07:21:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f554c190 con 0x5626f5526330 2024-03-06T07:21:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.136+0000 7f631f426740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f5549fc0 con 0x5626f554a490 2024-03-06T07:21:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631f426740 10 client.0 _ll_register_callbacks cb 0x7f6308057fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f5547180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f5547180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48942/0 (socket says 172.21.15.94:48942) 2024-03-06T07:21:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 -- 172.21.15.94:0/550205244 learned_addr learned my addr 172.21.15.94:0/550205244 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f6315d9b640 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f55493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f6316d9d640 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 -- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 msgr2=0x5626f55d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55d1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 -- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 msgr2=0x5626f55493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f55493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 -- 172.21.15.94:0/550205244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626f5549030 con 0x5626f5525f50 2024-03-06T07:21:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631659c640 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f5547180 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f630800cdf0 tx=0x7f6308088270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a31a03e5780179 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f62f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:21:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f62f57fa640 1 -- 172.21.15.94:0/550205244 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630805c740 con 0x5626f5525f50 2024-03-06T07:21:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f62f57fa640 1 -- 172.21.15.94:0/550205244 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6308088070 con 0x5626f5525f50 2024-03-06T07:21:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6308070030 con 0x5626f5525f50 2024-03-06T07:21:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f62f57fa640 1 -- 172.21.15.94:0/550205244 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f630807f050 con 0x5626f5525f50 2024-03-06T07:21:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631f426740 10 client.7119 fetch_fsmap learned FSMap version 77 2024-03-06T07:21:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x5626f5542a50 con 0x5626f5525f50 2024-03-06T07:21:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.137+0000 7f62f57fa640 1 -- 172.21.15.94:0/550205244 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6308070030 con 0x5626f5525f50 2024-03-06T07:21:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f62f57fa640 1 -- 172.21.15.94:0/550205244 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f63080574b0 con 0x5626f5525f50 2024-03-06T07:21:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 10 client.7119 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:21:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 10 client.7119 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:21:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 20 client.7119 populate_metadata read hostname 'smithi094' 2024-03-06T07:21:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55d1d40 0x5626f5543250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 4 client.7119 mds_command: new command op to 35900 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 --> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5626f5491f40 con 0x5626f55d1d40 2024-03-06T07:21:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f6315d9b640 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55d1d40 0x5626f5543250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f62f57fa640 10 client.7119 ms_handle_connect on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f6315d9b640 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55d1d40 0x5626f5543250 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=652f890c92e1368 server_cookie=c7083a35be2d909c in_seq=0 out_seq=0 2024-03-06T07:21:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f62f57fa640 1 -- 172.21.15.94:0/550205244 <== mds.0 v2:172.21.15.145:6834/3774173957 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5626f5491f40 con 0x5626f55d1d40 2024-03-06T07:21:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f62f57fa640 10 client.7119 handle_command_reply: tid=0 2024-03-06T07:21:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 1 client.7119 shutdown 2024-03-06T07:21:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 msgr2=0x5626f5547180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.138+0000 7f631f426740 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f5547180 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f630800cdf0 tx=0x7f6308088270 comp rx=0 tx=0).stop 2024-03-06T07:21:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 shutdown_connections 2024-03-06T07:21:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5626f55d1d40 0x5626f5543250 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f554a490 0x5626f55d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5526330 0x5626f55493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f62f57fa640 0 client.7119 ms_handle_reset on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 --2- 172.21.15.94:0/550205244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5525f50 0x5626f5547180 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 >> 172.21.15.94:0/550205244 conn(0x5626f55244b0 msgr2=0x5626f551f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 shutdown_connections 2024-03-06T07:21:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 1 -- 172.21.15.94:0/550205244 wait complete. 2024-03-06T07:21:44.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631f426740 20 client.7119 trim_cache size 0 max 16384 2024-03-06T07:21:44.155 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:21:44.155 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:21:44.156 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:21:44.156 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 35900, 2024-03-06T07:21:44.156 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:21:44.156 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:21:44.156 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:21:44.156 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.5770114409999998, 2024-03-06T07:21:44.157 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 77, 2024-03-06T07:21:44.157 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 95, 2024-03-06T07:21:44.157 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 95, 2024-03-06T07:21:44.157 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 9.6912506430000001 2024-03-06T07:21:44.157 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:21:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f62f806b4d0 msgr2=0x7f62f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f62f806b4d0 0x7f62f806d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f631806b990 tx=0x7f630c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63180c6ab0 msgr2=0x7f63181513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63180c6ab0 0x7f63181513b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f630800c430 tx=0x7f630800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 shutdown_connections 2024-03-06T07:21:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f62f806b4d0 0x7f62f806d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63181561a0 0x7f6318141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f631814f110 0x7f6318142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 --2- 172.21.15.94:0/3620472123 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63180c6ab0 0x7f63181513b0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 >> 172.21.15.94:0/3620472123 conn(0x7f631806b1a0 msgr2=0x7f63180c15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 shutdown_connections 2024-03-06T07:21:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.139+0000 7f631c839640 1 -- 172.21.15.94:0/3620472123 wait complete. 2024-03-06T07:21:44.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:44.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.260+0000 7fe2a7c0e640 1 Processor -- start 2024-03-06T07:21:44.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.261+0000 7fe2a7c0e640 1 -- start start 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.261+0000 7fe2a7c0e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe2a0152960 0x7fe2a006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.261+0000 7fe2a7c0e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2a00ac260 0x7fe2a01559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.261+0000 7fe2a7c0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a0152d40 con 0x7fe2a01592d0 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.261+0000 7fe2a7c0e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a0066e30 con 0x7fe2a00ac260 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.261+0000 7fe2a7c0e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a00c1dc0 con 0x7fe2a0152960 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a61ab640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2a00ac260 0x7fe2a01559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a51a9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe2a01592d0 0x7fe2a01552d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54318/0 (socket says 172.21.15.94:54318) 2024-03-06T07:21:44.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a51a9640 1 -- 172.21.15.94:0/4030517045 learned_addr learned my addr 172.21.15.94:0/4030517045 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:44.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a59aa640 1 --2- 172.21.15.94:0/4030517045 >> v2:172.21.15.145:3300/0 conn(0x7fe2a0152960 0x7fe2a006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a61ab640 1 -- 172.21.15.94:0/4030517045 >> v2:172.21.15.145:3300/0 conn(0x7fe2a0152960 msgr2=0x7fe2a006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a61ab640 1 --2- 172.21.15.94:0/4030517045 >> v2:172.21.15.145:3300/0 conn(0x7fe2a0152960 0x7fe2a006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a61ab640 1 -- 172.21.15.94:0/4030517045 >> v1:172.21.15.94:6789/0 conn(0x7fe2a01592d0 legacy=0x7fe2a01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a61ab640 1 -- 172.21.15.94:0/4030517045 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00c1b50 con 0x7fe2a00ac260 2024-03-06T07:21:44.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a49a8640 1 -- 172.21.15.94:0/4030517045 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882070312 0 0) 0x7fe2a0152d40 con 0x7fe2a01592d0 2024-03-06T07:21:44.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.262+0000 7fe2a59aa640 1 --2- 172.21.15.94:0/4030517045 >> v2:172.21.15.145:3300/0 conn(0x7fe2a0152960 0x7fe2a006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:44.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a61ab640 1 --2- 172.21.15.94:0/4030517045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a00ac260 0x7fe2a01559e0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe2a006b1c0 tx=0x7fe29802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89da67309c1c0cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a49a8640 1 -- 172.21.15.94:0/4030517045 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298039070 con 0x7fe2a00ac260 2024-03-06T07:21:44.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a49a8640 1 -- 172.21.15.94:0/4030517045 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29802cc80 con 0x7fe2a00ac260 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a49a8640 1 -- 172.21.15.94:0/4030517045 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298034660 con 0x7fe2a00ac260 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/4030517045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a00ac260 msgr2=0x7fe2a01559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/4030517045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a00ac260 0x7fe2a01559e0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe2a006b1c0 tx=0x7fe29802bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/4030517045 shutdown_connections 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/4030517045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a00ac260 0x7fe2a01559e0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/4030517045 >> v2:172.21.15.145:3300/0 conn(0x7fe2a0152960 0x7fe2a006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/4030517045 >> 172.21.15.94:0/4030517045 conn(0x7fe2a00c70b0 msgr2=0x7fe2a00c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/4030517045 shutdown_connections 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.263+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/4030517045 wait complete. 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.264+0000 7fe2a7c0e640 1 Processor -- start 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a7c0e640 1 -- start start 2024-03-06T07:21:44.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a7c0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2a00ac260 0x7fe2a0156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a7c0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2a0152960 0x7fe2a01ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a7c0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0153b50 0x7fe2a014e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a7c0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00c20a0 con 0x7fe2a0153b50 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a7c0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a0159760 con 0x7fe2a0152960 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a7c0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00c2220 con 0x7fe2a00ac260 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a61ab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0153b50 0x7fe2a014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a61ab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0153b50 0x7fe2a014e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48958/0 (socket says 172.21.15.94:48958) 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a61ab640 1 -- 172.21.15.94:0/810410834 learned_addr learned my addr 172.21.15.94:0/810410834 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a51a9640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2a0152960 0x7fe2a01ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.265+0000 7fe2a59aa640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2a00ac260 0x7fe2a0156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a61ab640 1 -- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2a00ac260 msgr2=0x7fe2a0156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a61ab640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2a00ac260 0x7fe2a0156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a61ab640 1 -- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2a0152960 msgr2=0x7fe2a01ee120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a61ab640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2a0152960 0x7fe2a01ee120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a61ab640 1 -- 172.21.15.94:0/810410834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a01ee960 con 0x7fe2a0153b50 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a61ab640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0153b50 0x7fe2a014e460 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe298037040 tx=0x7fe298004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2967fc640 1 -- 172.21.15.94:0/810410834 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298044020 con 0x7fe2a0153b50 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2967fc640 1 -- 172.21.15.94:0/810410834 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29803a040 con 0x7fe2a0153b50 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a01eec40 con 0x7fe2a0153b50 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2967fc640 1 -- 172.21.15.94:0/810410834 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298039050 con 0x7fe2a0153b50 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.266+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a01ee660 con 0x7fe2a0153b50 2024-03-06T07:21:44.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.267+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a0075300 con 0x7fe2a0153b50 2024-03-06T07:21:44.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:44.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.267+0000 7fe2967fc640 1 -- 172.21.15.94:0/810410834 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe298003b90 con 0x7fe2a0153b50 2024-03-06T07:21:44.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.268+0000 7fe2967fc640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe28406b500 0x7fe28406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.268+0000 7fe2967fc640 1 -- 172.21.15.94:0/810410834 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe2980ac970 con 0x7fe2a0153b50 2024-03-06T07:21:44.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.268+0000 7fe2a59aa640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe28406b500 0x7fe28406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.268+0000 7fe2a59aa640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe28406b500 0x7fe28406d9c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00ab4e0 tx=0x7fe290009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.272+0000 7fe2967fc640 1 -- 172.21.15.94:0/810410834 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2a0075300 con 0x7fe2a0153b50 2024-03-06T07:21:44.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:44.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.483+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2a0156270 con 0x7fe2a0153b50 2024-03-06T07:21:44.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.483+0000 7fe2967fc640 1 -- 172.21.15.94:0/810410834 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fe29804e020 con 0x7fe2a0153b50 2024-03-06T07:21:44.485 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:44.486 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:40.899292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35900},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35900":{"gid":35900,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/3774173957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3774173957},{"type":"v1","addr":"172.21.15.145:6836","nonce":3774173957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35900,"qdb_cluster":[35900]},"id":7}]} 2024-03-06T07:21:44.486 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 77 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe28406b500 msgr2=0x7fe28406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe28406b500 0x7fe28406d9c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00ab4e0 tx=0x7fe290009190 comp rx=0 tx=0).stop 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0153b50 msgr2=0x7fe2a014e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0153b50 0x7fe2a014e460 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe298037040 tx=0x7fe298004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 shutdown_connections 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe28406b500 0x7fe28406d9c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2a0153b50 0x7fe2a014e460 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2a0152960 0x7fe2a01ee120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 --2- 172.21.15.94:0/810410834 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2a00ac260 0x7fe2a0156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 >> 172.21.15.94:0/810410834 conn(0x7fe2a00c70b0 msgr2=0x7fe2a006afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 shutdown_connections 2024-03-06T07:21:44.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.485+0000 7fe2a7c0e640 1 -- 172.21.15.94:0/810410834 wait complete. 2024-03-06T07:21:44.498 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:38.904104+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_35900": { "addr": "172.21.15.145:6836/3774173957", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 3774173957, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 3774173957, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35900, "incarnation": 75, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-06T07:21:40.899292+0000", "qdb_cluster": [ 35900 ], "qdb_leader": 35900, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 35900 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/744582506", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 744582506, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 744582506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/3090166857", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3090166857, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 3090166857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6945, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3798967274", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3798967274, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3798967274, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35888, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:44.499 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:44.499 INFO:tasks.cephfs.test_failover:current = [{'gid': 35888, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3798967274', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3798967274}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3798967274}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/744582506', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 744582506}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 744582506}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6945, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/3090166857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3090166857}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 3090166857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 35900, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/3774173957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3774173957}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3774173957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:21:44.499 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-06T07:21:44.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.601+0000 7f9182483640 1 Processor -- start 2024-03-06T07:21:44.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.601+0000 7f9182483640 1 -- start start 2024-03-06T07:21:44.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.602+0000 7f9182483640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c14f650 0x7f917c071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.602+0000 7f9182483640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917c072440 con 0x7f917c0c3150 2024-03-06T07:21:44.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.602+0000 7f9182483640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917c0c2b10 con 0x7f917c152c20 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.602+0000 7f9182483640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917c152140 con 0x7f917c14f650 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.602+0000 7f917b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f917c0c3150 0x7f917c0c5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54320/0 (socket says 172.21.15.94:54320) 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.602+0000 7f917b7fe640 1 -- 172.21.15.94:0/1219944948 learned_addr learned my addr 172.21.15.94:0/1219944948 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.602+0000 7f917bfff640 1 --2- 172.21.15.94:0/1219944948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c14f650 0x7f917c071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.603+0000 7f917affd640 1 -- 172.21.15.94:0/1219944948 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859357163 0 0) 0x7f9170001110 con 0x7f917c0c3150 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.603+0000 7f917affd640 1 -- 172.21.15.94:0/1219944948 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9164003520 con 0x7f917c0c3150 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.603+0000 7f917affd640 1 -- 172.21.15.94:0/1219944948 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170823729 0 0) 0x7f917c0c2b10 con 0x7f917c152c20 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.603+0000 7f917affd640 1 -- 172.21.15.94:0/1219944948 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9170001110 con 0x7f917c152c20 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.603+0000 7f917bfff640 1 -- 172.21.15.94:0/1219944948 >> v1:172.21.15.145:6789/0 conn(0x7f917c152c20 legacy=0x7f917c15c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.603+0000 7f917bfff640 1 -- 172.21.15.94:0/1219944948 >> v1:172.21.15.94:6789/0 conn(0x7f917c0c3150 legacy=0x7f917c0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.603+0000 7f917bfff640 1 -- 172.21.15.94:0/1219944948 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c152420 con 0x7f917c14f650 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f917bfff640 1 --2- 172.21.15.94:0/1219944948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c14f650 0x7f917c071880 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f916c0097d0 tx=0x7f916c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e93e9a32752b27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f917affd640 1 -- 172.21.15.94:0/1219944948 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c035020 con 0x7f917c14f650 2024-03-06T07:21:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f917affd640 1 -- 172.21.15.94:0/1219944948 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916c031890 con 0x7f917c14f650 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f917affd640 1 -- 172.21.15.94:0/1219944948 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c031c40 con 0x7f917c14f650 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f9182483640 1 -- 172.21.15.94:0/1219944948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c14f650 msgr2=0x7f917c071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f9182483640 1 --2- 172.21.15.94:0/1219944948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c14f650 0x7f917c071880 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f916c0097d0 tx=0x7f916c02b930 comp rx=0 tx=0).stop 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f9182483640 1 -- 172.21.15.94:0/1219944948 shutdown_connections 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f9182483640 1 --2- 172.21.15.94:0/1219944948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c14f650 0x7f917c071880 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f9182483640 1 -- 172.21.15.94:0/1219944948 >> 172.21.15.94:0/1219944948 conn(0x7f917c0c4290 msgr2=0x7f917c0c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f9182483640 1 -- 172.21.15.94:0/1219944948 shutdown_connections 2024-03-06T07:21:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.604+0000 7f9182483640 1 -- 172.21.15.94:0/1219944948 wait complete. 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.605+0000 7f9182483640 1 Processor -- start 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9182483640 1 -- start start 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9182483640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f917c0c3150 0x7f917c150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9182483640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f917c14f650 0x7f917c0a7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9182483640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c152c20 0x7f917c14ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9182483640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c0c40c0 con 0x7f917c14f650 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9182483640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c0c3f40 con 0x7f917c0c3150 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9182483640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917c0c5da0 con 0x7f917c152c20 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f917b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f917c14f650 0x7f917c0a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f917b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f917c14f650 0x7f917c0a7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48974/0 (socket says 172.21.15.94:48974) 2024-03-06T07:21:44.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f917b7fe640 1 -- 172.21.15.94:0/1440690663 learned_addr learned my addr 172.21.15.94:0/1440690663 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f9180a20640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c152c20 0x7f917c14ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.606+0000 7f917bfff640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f917c0c3150 0x7f917c150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f917b7fe640 1 -- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c152c20 msgr2=0x7f917c14ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f917b7fe640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c152c20 0x7f917c14ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f917b7fe640 1 -- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f917c0c3150 msgr2=0x7f917c150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f917b7fe640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f917c0c3150 0x7f917c150bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f917b7fe640 1 -- 172.21.15.94:0/1440690663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c14a0a0 con 0x7f917c14f650 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f917b7fe640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f917c14f650 0x7f917c0a7490 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f917000c680 tx=0x7f917000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f9178ff9640 1 -- 172.21.15.94:0/1440690663 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170017070 con 0x7f917c14f650 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f9178ff9640 1 -- 172.21.15.94:0/1440690663 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9170005ab0 con 0x7f917c14f650 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917c14a380 con 0x7f917c14f650 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f9178ff9640 1 -- 172.21.15.94:0/1440690663 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170007070 con 0x7f917c14f650 2024-03-06T07:21:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.607+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917c1506b0 con 0x7f917c14f650 2024-03-06T07:21:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.608+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917c1504a0 con 0x7f917c14f650 2024-03-06T07:21:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.609+0000 7f9178ff9640 1 -- 172.21.15.94:0/1440690663 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f917001a070 con 0x7f917c14f650 2024-03-06T07:21:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.609+0000 7f9178ff9640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916806b4d0 0x7f916806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.609+0000 7f9178ff9640 1 -- 172.21.15.94:0/1440690663 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9170089820 con 0x7f917c14f650 2024-03-06T07:21:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.609+0000 7f917bfff640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916806b4d0 0x7f916806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.610+0000 7f917bfff640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916806b4d0 0x7f916806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f916c036000 tx=0x7f916c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.613+0000 7f9178ff9640 1 -- 172.21.15.94:0/1440690663 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f917c072440 con 0x7f917c14f650 2024-03-06T07:21:44.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.785+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f917c04f330 con 0x7f917c14f650 2024-03-06T07:21:44.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.787+0000 7f9178ff9640 1 -- 172.21.15.94:0/1440690663 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v32)=0 v32) v1 ==== 116+0+0 (secure 0 0 0) 0x7f9170058010 con 0x7f917c14f650 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916806b4d0 msgr2=0x7f916806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916806b4d0 0x7f916806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f916c036000 tx=0x7f916c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f917c14f650 msgr2=0x7f917c0a7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f917c14f650 0x7f917c0a7490 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f917000c680 tx=0x7f917000cb40 comp rx=0 tx=0).stop 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 shutdown_connections 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916806b4d0 0x7f916806d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f917c152c20 0x7f917c14ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f917c14f650 0x7f917c0a7490 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 --2- 172.21.15.94:0/1440690663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f917c0c3150 0x7f917c150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 >> 172.21.15.94:0/1440690663 conn(0x7f917c0c4290 msgr2=0x7f917c071b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.789+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 shutdown_connections 2024-03-06T07:21:44.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.790+0000 7f9182483640 1 -- 172.21.15.94:0/1440690663 wait complete. 2024-03-06T07:21:44.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:44.819 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:21:44.817+0000 7fca29a23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:44.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.941+0000 7fe2497d7640 1 Processor -- start 2024-03-06T07:21:44.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.941+0000 7fe2497d7640 1 -- start start 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2497d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2497d7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2440c7350 0x7fe2440c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2497d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24406c150 con 0x7fe24414f250 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2497d7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24406c480 con 0x7fe2440c7350 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2497d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2440c8cc0 con 0x7fe2440c8840 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2437fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2440c7350 0x7fe2440c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe242ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe24414f250 0x7fe244155c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54332/0 (socket says 172.21.15.94:54332) 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe242ffd640 1 -- 172.21.15.94:0/2932911989 learned_addr learned my addr 172.21.15.94:0/2932911989 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe23bfff640 1 --2- 172.21.15.94:0/2932911989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2437fe640 1 -- 172.21.15.94:0/2932911989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 msgr2=0x7fe24406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2437fe640 1 --2- 172.21.15.94:0/2932911989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2437fe640 1 -- 172.21.15.94:0/2932911989 >> v1:172.21.15.94:6789/0 conn(0x7fe24414f250 legacy=0x7fe244155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2427fc640 1 -- 172.21.15.94:0/2932911989 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975807961 0 0) 0x7fe24406c150 con 0x7fe24414f250 2024-03-06T07:21:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.942+0000 7fe2437fe640 1 -- 172.21.15.94:0/2932911989 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24414f630 con 0x7fe2440c7350 2024-03-06T07:21:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.943+0000 7fe2437fe640 1 --2- 172.21.15.94:0/2932911989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2440c7350 0x7fe2440c77e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009d90 tx=0x7fe23c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a0c64491f00bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.943+0000 7fe2427fc640 1 -- 172.21.15.94:0/2932911989 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c039070 con 0x7fe2440c7350 2024-03-06T07:21:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.943+0000 7fe2427fc640 1 -- 172.21.15.94:0/2932911989 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c02cb30 con 0x7fe2440c7350 2024-03-06T07:21:44.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.943+0000 7fe2427fc640 1 -- 172.21.15.94:0/2932911989 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c034440 con 0x7fe2440c7350 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.943+0000 7fe2497d7640 1 -- 172.21.15.94:0/2932911989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2440c7350 msgr2=0x7fe2440c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.943+0000 7fe2497d7640 1 --2- 172.21.15.94:0/2932911989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2440c7350 0x7fe2440c77e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009d90 tx=0x7fe23c02bd80 comp rx=0 tx=0).stop 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.944+0000 7fe2497d7640 1 -- 172.21.15.94:0/2932911989 shutdown_connections 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.944+0000 7fe2497d7640 1 --2- 172.21.15.94:0/2932911989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2440c7350 0x7fe2440c77e0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.944+0000 7fe2497d7640 1 --2- 172.21.15.94:0/2932911989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.944+0000 7fe2497d7640 1 -- 172.21.15.94:0/2932911989 >> 172.21.15.94:0/2932911989 conn(0x7fe2440ac4d0 msgr2=0x7fe2440ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.944+0000 7fe2497d7640 1 -- 172.21.15.94:0/2932911989 shutdown_connections 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.944+0000 7fe2497d7640 1 -- 172.21.15.94:0/2932911989 wait complete. 2024-03-06T07:21:44.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 Processor -- start 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 -- start start 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2440c7350 0x7fe2440af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24414bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe24414f250 0x7fe24414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24406daf0 con 0x7fe24414f250 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2440c5940 con 0x7fe2440c7350 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2497d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2440c5c20 con 0x7fe2440c8840 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2437fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe24414f250 0x7fe24414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2437fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe24414f250 0x7fe24414c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48992/0 (socket says 172.21.15.94:48992) 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe2437fe640 1 -- 172.21.15.94:0/228411096 learned_addr learned my addr 172.21.15.94:0/228411096 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe242ffd640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2440c7350 0x7fe2440af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.945+0000 7fe23bfff640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24414bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2437fe640 1 -- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 msgr2=0x7fe24414bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2437fe640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24414bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2437fe640 1 -- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2440c7350 msgr2=0x7fe2440af230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2437fe640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2440c7350 0x7fe2440af230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2437fe640 1 -- 172.21.15.94:0/228411096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2440be450 con 0x7fe24414f250 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2437fe640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe24414f250 0x7fe24414c1a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009d90 tx=0x7fe23c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe23b7fe640 1 -- 172.21.15.94:0/228411096 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c044020 con 0x7fe24414f250 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe23b7fe640 1 -- 172.21.15.94:0/228411096 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c03a040 con 0x7fe24414f250 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe23b7fe640 1 -- 172.21.15.94:0/228411096 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c039050 con 0x7fe24414f250 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2441ee450 con 0x7fe24414f250 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.946+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2441eea10 con 0x7fe24414f250 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.947+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24404f330 con 0x7fe24414f250 2024-03-06T07:21:44.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.947+0000 7fe23b7fe640 1 -- 172.21.15.94:0/228411096 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe23c003bb0 con 0x7fe24414f250 2024-03-06T07:21:44.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.948+0000 7fe23b7fe640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe22406b4d0 0x7fe22406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.948+0000 7fe23b7fe640 1 -- 172.21.15.94:0/228411096 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe23c0ac840 con 0x7fe24414f250 2024-03-06T07:21:44.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.948+0000 7fe242ffd640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe22406b4d0 0x7fe22406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.948+0000 7fe242ffd640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe22406b4d0 0x7fe22406d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe234007fd0 tx=0x7fe234007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:44.952+0000 7fe23b7fe640 1 -- 172.21.15.94:0/228411096 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe23c04e060 con 0x7fe24414f250 2024-03-06T07:21:45.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.157+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe24406d860 con 0x7fe24414f250 2024-03-06T07:21:45.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.157+0000 7fe23b7fe640 1 -- 172.21.15.94:0/228411096 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4988 (secure 0 0 0) 0x7fe23c07af70 con 0x7fe24414f250 2024-03-06T07:21:45.159 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:45.159 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":77,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6658,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:40.899292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":35900},"failed":[],"damaged":[],"stopped":[],"info":{"gid_35900":{"gid":35900,"name":"d","rank":0,"incarnation":75,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/3774173957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3774173957},{"type":"v1","addr":"172.21.15.145:6836","nonce":3774173957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":35900,"qdb_cluster":[35900]},"id":7}]} 2024-03-06T07:21:45.160 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 77 2024-03-06T07:21:45.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe22406b4d0 msgr2=0x7fe22406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe22406b4d0 0x7fe22406d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe234007fd0 tx=0x7fe234007de0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe24414f250 msgr2=0x7fe24414c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe24414f250 0x7fe24414c1a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009d90 tx=0x7fe23c004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:45.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 shutdown_connections 2024-03-06T07:21:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe22406b4d0 0x7fe22406d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe24414f250 0x7fe24414c1a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2440c8840 0x7fe24414bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 --2- 172.21.15.94:0/228411096 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2440c7350 0x7fe2440af230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 >> 172.21.15.94:0/228411096 conn(0x7fe2440ac4d0 msgr2=0x7fe24414e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 shutdown_connections 2024-03-06T07:21:45.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.159+0000 7fe2497d7640 1 -- 172.21.15.94:0/228411096 wait complete. 2024-03-06T07:21:45.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:38.904104+0000', 'modified': '2024-03-06T07:21:40.899292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 35900}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_35900': {'gid': 35900, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/3774173957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3774173957}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3774173957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 35900, 'qdb_cluster': [35900]} 2024-03-06T07:21:45.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:45.576 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:21:45.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.676+0000 7f4f83466640 1 Processor -- start 2024-03-06T07:21:45.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.677+0000 7f4f83466640 1 -- start start 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.677+0000 7f4f83466640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f7c0c27b0 0x7f4f7c0c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.677+0000 7f4f83466640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c158720 0x7f4f7c158b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.677+0000 7f4f83466640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c155150 con 0x7f4f7c152740 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.677+0000 7f4f83466640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7c155d00 con 0x7f4f7c0c27b0 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.677+0000 7f4f83466640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1553d0 con 0x7f4f7c158720 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f81202640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f4f7c152740 0x7f4f7c06afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:43638/0 (socket says 172.21.15.94:43638) 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f80a01640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f7c0c27b0 0x7f4f7c0c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f81202640 1 -- 172.21.15.94:0/896598369 learned_addr learned my addr 172.21.15.94:0/896598369 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f81a03640 1 --2- 172.21.15.94:0/896598369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c158720 0x7f4f7c158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f80a01640 1 -- 172.21.15.94:0/896598369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c158720 msgr2=0x7f4f7c158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f80a01640 1 --2- 172.21.15.94:0/896598369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c158720 0x7f4f7c158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f73fff640 1 -- 172.21.15.94:0/896598369 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371426166 0 0) 0x7f4f6c001110 con 0x7f4f7c152740 2024-03-06T07:21:45.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f80a01640 1 -- 172.21.15.94:0/896598369 >> v1:172.21.15.145:6789/0 conn(0x7f4f7c152740 legacy=0x7f4f7c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.678+0000 7f4f80a01640 1 -- 172.21.15.94:0/896598369 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c1556b0 con 0x7f4f7c0c27b0 2024-03-06T07:21:45.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.679+0000 7f4f80a01640 1 --2- 172.21.15.94:0/896598369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c0c27b0 0x7f4f7c0c20b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c155e80 tx=0x7f4f7802b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8cacaf2907c8285 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:45.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.679+0000 7f4f73fff640 1 -- 172.21.15.94:0/896598369 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78031770 con 0x7f4f7c0c27b0 2024-03-06T07:21:45.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.679+0000 7f4f73fff640 1 -- 172.21.15.94:0/896598369 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f78037070 con 0x7f4f7c0c27b0 2024-03-06T07:21:45.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.679+0000 7f4f73fff640 1 -- 172.21.15.94:0/896598369 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78031b30 con 0x7f4f7c0c27b0 2024-03-06T07:21:45.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 -- 172.21.15.94:0/896598369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c0c27b0 msgr2=0x7f4f7c0c20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 --2- 172.21.15.94:0/896598369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c0c27b0 0x7f4f7c0c20b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c155e80 tx=0x7f4f7802b810 comp rx=0 tx=0).stop 2024-03-06T07:21:45.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 -- 172.21.15.94:0/896598369 shutdown_connections 2024-03-06T07:21:45.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 --2- 172.21.15.94:0/896598369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c158720 0x7f4f7c158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 --2- 172.21.15.94:0/896598369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c0c27b0 0x7f4f7c0c20b0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 -- 172.21.15.94:0/896598369 >> 172.21.15.94:0/896598369 conn(0x7f4f7c0c41b0 msgr2=0x7f4f7c0c4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:45.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 -- 172.21.15.94:0/896598369 shutdown_connections 2024-03-06T07:21:45.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.680+0000 7f4f83466640 1 -- 172.21.15.94:0/896598369 wait complete. 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.681+0000 7f4f83466640 1 Processor -- start 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.681+0000 7f4f83466640 1 -- start start 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.681+0000 7f4f83466640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f7c0c27b0 0x7f4f7c147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.681+0000 7f4f83466640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c152740 0x7f4f7c0c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.681+0000 7f4f83466640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c158720 0x7f4f7c1478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f83466640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c152b20 con 0x7f4f7c158720 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f83466640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c06be80 con 0x7f4f7c0c27b0 2024-03-06T07:21:45.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f83466640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c06b6d0 con 0x7f4f7c152740 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c158720 0x7f4f7c1478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c158720 0x7f4f7c1478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48998/0 (socket says 172.21.15.94:48998) 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 -- 172.21.15.94:0/1952732182 learned_addr learned my addr 172.21.15.94:0/1952732182 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81202640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f7c0c27b0 0x7f4f7c147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f80a01640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c152740 0x7f4f7c0c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 -- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c152740 msgr2=0x7f4f7c0c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c152740 0x7f4f7c0c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 -- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f7c0c27b0 msgr2=0x7f4f7c147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f7c0c27b0 0x7f4f7c147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 -- 172.21.15.94:0/1952732182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c1476a0 con 0x7f4f7c158720 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f81a03640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c158720 0x7f4f7c1478b0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400efc0 tx=0x7f4f7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f71ffb640 1 -- 172.21.15.94:0/1952732182 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74018020 con 0x7f4f7c158720 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f71ffb640 1 -- 172.21.15.94:0/1952732182 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f74014480 con 0x7f4f7c158720 2024-03-06T07:21:45.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f71ffb640 1 -- 172.21.15.94:0/1952732182 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74014870 con 0x7f4f7c158720 2024-03-06T07:21:45.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c1ebbd0 con 0x7f4f7c158720 2024-03-06T07:21:45.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.682+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c1eb330 con 0x7f4f7c158720 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.684+0000 7f4f71ffb640 1 -- 172.21.15.94:0/1952732182 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f74005d10 con 0x7f4f7c158720 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.684+0000 7f4f71ffb640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f6006b500 0x7f4f6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.684+0000 7f4f71ffb640 1 -- 172.21.15.94:0/1952732182 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f7408ac20 con 0x7f4f7c158720 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.684+0000 7f4f81202640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f6006b500 0x7f4f6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.685+0000 7f4f81202640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f6006b500 0x7f4f6006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000f60 tx=0x7f4f6c007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.685+0000 7f4f86049740 1 Processor -- start 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.685+0000 7f4f86049740 1 -- start start 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f86049740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe214f50 0x5596fe212a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe212f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe2134b0 0x5596fe20f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f86049740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596fe1796b0 con 0x5596fe214f50 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596fe122a60 con 0x5596fe215330 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596fe175750 con 0x5596fe2134b0 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe214f50 0x5596fe212a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe214f50 0x5596fe212a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49014/0 (socket says 172.21.15.94:49014) 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 -- 172.21.15.94:0/3168379014 learned_addr learned my addr 172.21.15.94:0/3168379014 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f80a01640 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe212f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81a03640 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe2134b0 0x5596fe20f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 -- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe2134b0 msgr2=0x5596fe20f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe2134b0 0x5596fe20f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 -- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 msgr2=0x5596fe212f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe212f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.686+0000 7f4f81202640 1 -- 172.21.15.94:0/3168379014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596fe140f30 con 0x5596fe214f50 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f81202640 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe214f50 0x5596fe212a30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0018f0 tx=0x7f4f6c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd010ae94ae5c51f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f5effd640 1 -- 172.21.15.94:0/3168379014 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c026070 con 0x5596fe214f50 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f5effd640 1 -- 172.21.15.94:0/3168379014 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c013b80 con 0x5596fe214f50 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f5effd640 1 -- 172.21.15.94:0/3168379014 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c012350 con 0x5596fe214f50 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 -- 172.21.15.94:0/3168379014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe214f50 msgr2=0x5596fe212a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe214f50 0x5596fe212a30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0018f0 tx=0x7f4f6c013000 comp rx=0 tx=0).stop 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 -- 172.21.15.94:0/3168379014 shutdown_connections 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe2134b0 0x5596fe20f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe212f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 --2- 172.21.15.94:0/3168379014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe214f50 0x5596fe212a30 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 -- 172.21.15.94:0/3168379014 >> 172.21.15.94:0/3168379014 conn(0x5596fe2117d0 msgr2=0x5596fe2363c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 -- 172.21.15.94:0/3168379014 shutdown_connections 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.687+0000 7f4f86049740 1 -- 172.21.15.94:0/3168379014 wait complete. 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.688+0000 7f4f86049740 1 -- start start 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.688+0000 7f4f86049740 1 Processor -- start 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe237600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe23a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f86049740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe2625a0 0x5596fe24dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f86049740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596fe237230 con 0x5596fe2625a0 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596fe135ea0 con 0x5596fe215330 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596fe236c70 con 0x5596fe214f50 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f86049740 10 client.0 _ll_register_callbacks cb 0x5596fe135870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:45.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe2625a0 0x5596fe24dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe2625a0 0x5596fe24dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49016/0 (socket says 172.21.15.94:49016) 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81202640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe237600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 -- 172.21.15.94:0/3227738827 learned_addr learned my addr 172.21.15.94:0/3227738827 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f80a01640 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe23a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 -- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 msgr2=0x5596fe237600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe237600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 -- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 msgr2=0x5596fe23a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe23a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 -- 172.21.15.94:0/3227738827 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596fe136270 con 0x5596fe2625a0 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f81a03640 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe2625a0 0x5596fe24dae0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4f74089050 tx=0x7f4f740594d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c5c47ef2b919ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.689+0000 7f4f57fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f57fff640 1 -- 172.21.15.94:0/3227738827 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74087060 con 0x5596fe2625a0 2024-03-06T07:21:45.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f57fff640 1 -- 172.21.15.94:0/3227738827 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f740587a0 con 0x5596fe2625a0 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f6c013b80 con 0x5596fe2625a0 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f57fff640 1 -- 172.21.15.94:0/3227738827 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f740589a0 con 0x5596fe2625a0 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 10 client.7155 fetch_fsmap learned FSMap version 77 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x5596fde8e2d0 con 0x5596fe2625a0 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f57fff640 1 -- 172.21.15.94:0/3227738827 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c013b80 con 0x5596fe2625a0 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f57fff640 1 -- 172.21.15.94:0/3227738827 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4f7405f380 con 0x5596fe2625a0 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 10 client.7155 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 10 client.7155 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 20 client.7155 populate_metadata read hostname 'smithi094' 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe265780 0x5596fe270540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 4 client.7155 mds_command: new command op to 35900 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 --> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596fe186bf0 con 0x5596fe265780 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.690+0000 7f4f81202640 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe265780 0x5596fe270540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f81202640 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe265780 0x5596fe270540 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42966ac4b81eb533 server_cookie=e34defc4163c55f3 in_seq=0 out_seq=0 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f57fff640 10 client.7155 ms_handle_connect on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f57fff640 1 -- 172.21.15.94:0/3227738827 <== mds.0 v2:172.21.15.145:6834/3774173957 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5596fe186bf0 con 0x5596fe265780 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f57fff640 10 client.7155 handle_command_reply: tid=0 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 client.7155 shutdown 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe2625a0 msgr2=0x5596fe24dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe2625a0 0x5596fe24dae0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4f74089050 tx=0x7f4f740594d0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 shutdown_connections 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe265780 0x5596fe270540 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe2625a0 0x5596fe24dae0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe215330 0x5596fe23a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f57fff640 0 client.7155 ms_handle_reset on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 --2- 172.21.15.94:0/3227738827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe237600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 >> 172.21.15.94:0/3227738827 conn(0x5596fe2134b0 msgr2=0x5596fe251b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:45.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.691+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 shutdown_connections 2024-03-06T07:21:45.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.692+0000 7f4f86049740 1 -- 172.21.15.94:0/3227738827 wait complete. 2024-03-06T07:21:45.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.692+0000 7f4f86049740 20 client.7155 trim_cache size 0 max 16384 2024-03-06T07:21:45.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.708+0000 7f4f86049740 1 Processor -- start 2024-03-06T07:21:45.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.708+0000 7f4f86049740 1 -- start start 2024-03-06T07:21:45.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe2107b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f86049740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe238980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe238ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f86049740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1f6f10 con 0x5596fe215330 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1eede0 con 0x5596fe2625a0 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1f6910 con 0x5596fe214f50 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe238980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe238980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49018/0 (socket says 172.21.15.94:49018) 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 -- 172.21.15.94:0/1724690678 learned_addr learned my addr 172.21.15.94:0/1724690678 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f81a03640 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe238ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f81202640 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe2107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 -- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 msgr2=0x5596fe2107b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe2107b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 -- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 msgr2=0x5596fe238ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe238ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 -- 172.21.15.94:0/1724690678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596fdf586b0 con 0x5596fe215330 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f81202640 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe2107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f80a01640 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe238980 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009890 tx=0x7f4f78030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b54617914932a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.709+0000 7f4f5e7fc640 1 -- 172.21.15.94:0/1724690678 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78031b40 con 0x5596fe215330 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f5e7fc640 1 -- 172.21.15.94:0/1724690678 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f78037040 con 0x5596fe215330 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f5e7fc640 1 -- 172.21.15.94:0/1724690678 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78032400 con 0x5596fe215330 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 -- 172.21.15.94:0/1724690678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 msgr2=0x5596fe238980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe238980 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009890 tx=0x7f4f78030000 comp rx=0 tx=0).stop 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 -- 172.21.15.94:0/1724690678 shutdown_connections 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe238ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe238980 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 --2- 172.21.15.94:0/1724690678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe2107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 -- 172.21.15.94:0/1724690678 >> 172.21.15.94:0/1724690678 conn(0x5596fe2134b0 msgr2=0x5596fe2381f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 -- 172.21.15.94:0/1724690678 shutdown_connections 2024-03-06T07:21:45.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.710+0000 7f4f86049740 1 -- 172.21.15.94:0/1724690678 wait complete. 2024-03-06T07:21:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 -- start start 2024-03-06T07:21:45.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 Processor -- start 2024-03-06T07:21:45.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe267f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe25dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe2bde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596fe236560 con 0x5596fe215330 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596fe261b20 con 0x5596fe2625a0 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596fe239750 con 0x5596fe214f50 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.711+0000 7f4f86049740 10 client.0 _ll_register_callbacks cb 0x7f4f74058d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe25dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe25dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49030/0 (socket says 172.21.15.94:49030) 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 -- 172.21.15.94:0/1810428844 learned_addr learned my addr 172.21.15.94:0/1810428844 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f81a03640 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe2bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f81202640 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe267f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 -- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 msgr2=0x5596fe267f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe267f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 -- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 msgr2=0x5596fe2bde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe2bde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 -- 172.21.15.94:0/1810428844 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596fe235e60 con 0x5596fe215330 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f80a01640 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe25dd00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4f7802be20 tx=0x7f4f78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad96ee7969abdadf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f57fff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f57fff640 1 -- 172.21.15.94:0/1810428844 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78004220 con 0x5596fe215330 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f57fff640 1 -- 172.21.15.94:0/1810428844 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f780043c0 con 0x5596fe215330 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f78037040 con 0x5596fe215330 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f57fff640 1 -- 172.21.15.94:0/1810428844 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f7805e050 con 0x5596fe215330 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f86049740 10 client.7161 fetch_fsmap learned FSMap version 77 2024-03-06T07:21:45.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x5596fe251dd0 con 0x5596fe215330 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.712+0000 7f4f57fff640 1 -- 172.21.15.94:0/1810428844 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f78037040 con 0x5596fe215330 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f57fff640 1 -- 172.21.15.94:0/1810428844 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4f78003610 con 0x5596fe215330 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 10 client.7161 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 10 client.7161 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 20 client.7161 populate_metadata read hostname 'smithi094' 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe2c40a0 0x5596fe2c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 4 client.7161 mds_command: new command op to 35900 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 --> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5596fe180f40 con 0x5596fe2c40a0 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f81202640 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe2c40a0 0x5596fe2c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f57fff640 10 client.7161 ms_handle_connect on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f81202640 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe2c40a0 0x5596fe2c4480 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bac3a178983064be server_cookie=6ef6c40159a32700 in_seq=0 out_seq=0 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f57fff640 1 -- 172.21.15.94:0/1810428844 <== mds.0 v2:172.21.15.145:6834/3774173957 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5596fe180f40 con 0x5596fe2c40a0 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f57fff640 10 client.7161 handle_command_reply: tid=0 2024-03-06T07:21:45.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 client.7161 shutdown 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 msgr2=0x5596fe25dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe25dd00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4f7802be20 tx=0x7f4f78004730 comp rx=0 tx=0).stop 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 shutdown_connections 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:6834/3774173957,v1:172.21.15.145:6836/3774173957] conn(0x5596fe2c40a0 0x5596fe2c4480 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596fe2625a0 0x5596fe2bde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f57fff640 0 client.7161 ms_handle_reset on v2:172.21.15.145:6834/3774173957 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596fe215330 0x5596fe25dd00 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 --2- 172.21.15.94:0/1810428844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596fe214f50 0x5596fe267f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.713+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 >> 172.21.15.94:0/1810428844 conn(0x5596fe2134b0 msgr2=0x5596fe254600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 shutdown_connections 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f86049740 1 -- 172.21.15.94:0/1810428844 wait complete. 2024-03-06T07:21:45.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f86049740 20 client.7161 trim_cache size 0 max 16384 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 35900, 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 6.1519946250000004, 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 77, 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 95, 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 95, 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 11.266234273 2024-03-06T07:21:45.723 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:21:45.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f6006b500 msgr2=0x7f4f6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f6006b500 0x7f4f6006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000f60 tx=0x7f4f6c007b50 comp rx=0 tx=0).stop 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c158720 msgr2=0x7f4f7c1478b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c158720 0x7f4f7c1478b0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400efc0 tx=0x7f4f7400c490 comp rx=0 tx=0).stop 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 shutdown_connections 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f6006b500 0x7f4f6006d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f7c158720 0x7f4f7c1478b0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f7c152740 0x7f4f7c0c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 --2- 172.21.15.94:0/1952732182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f7c0c27b0 0x7f4f7c147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 >> 172.21.15.94:0/1952732182 conn(0x7f4f7c0c41b0 msgr2=0x7f4f7c072790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 shutdown_connections 2024-03-06T07:21:45.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:45.714+0000 7f4f83466640 1 -- 172.21.15.94:0/1952732182 wait complete. 2024-03-06T07:21:45.727 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:38.904104+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_35900": { "addr": "172.21.15.145:6836/3774173957", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 3774173957, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 3774173957, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35900, "incarnation": 75, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-06T07:21:40.899292+0000", "qdb_cluster": [ 35900 ], "qdb_leader": 35900, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 35900 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/744582506", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 744582506, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 744582506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6658, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/3090166857", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3090166857, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 3090166857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6945, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3798967274", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3798967274, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3798967274, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35888, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:45.728 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 7, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:45.728 INFO:tasks.cephfs.test_failover:current = [{'gid': 35888, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3798967274', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3798967274}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3798967274}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6658, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/744582506', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 744582506}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 744582506}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6945, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/3090166857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3090166857}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 3090166857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 35900, 'name': 'd', 'rank': 0, 'incarnation': 75, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/3774173957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3774173957}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3774173957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:21:45.728 DEBUG:tasks.cephfs.test_failover:7 != -1 2024-03-06T07:21:45.728 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:21:46.807 INFO:tasks.ceph.mds.d.smithi145.stderr: -104> 2024-03-06T07:21:44.817+0000 7fca29a23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:46.832 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:21:49.819 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:21:49.816+0000 7f628b2ca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:50.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:50.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.842+0000 7fda61b93640 1 Processor -- start 2024-03-06T07:21:50.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.842+0000 7fda61b93640 1 -- start start 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.842+0000 7fda61b93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c14f360 0x7fda5c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.842+0000 7fda61b93640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fda5c158b60 0x7fda5c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.842+0000 7fda61b93640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5c0c73d0 con 0x7fda5c0c88c0 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.842+0000 7fda61b93640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5c0c8dd0 con 0x7fda5c158b60 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.842+0000 7fda61b93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0be2e0 con 0x7fda5c14f360 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5b7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c14f360 0x7fda5c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5bfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fda5c158b60 0x7fda5c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5affd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fda5c0c88c0 0x7fda5c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37564/0 (socket says 172.21.15.94:37564) 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5bfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fda5c158b60 0x7fda5c158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:50654/0 (socket says 172.21.15.94:50654) 2024-03-06T07:21:50.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5b7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c14f360 0x7fda5c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:37474/0 (socket says 172.21.15.94:37474) 2024-03-06T07:21:50.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5bfff640 1 -- 172.21.15.94:0/3331994755 learned_addr learned my addr 172.21.15.94:0/3331994755 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:50.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5bfff640 1 -- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c14f360 msgr2=0x7fda5c155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5bfff640 1 --2- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c14f360 0x7fda5c155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5bfff640 1 -- 172.21.15.94:0/3331994755 >> v1:172.21.15.145:6789/0 conn(0x7fda5c0c88c0 legacy=0x7fda5c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5bfff640 1 -- 172.21.15.94:0/3331994755 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c0c7670 con 0x7fda5c158b60 2024-03-06T07:21:50.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5a7fc640 1 -- 172.21.15.94:0/3331994755 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691825271 0 0) 0x7fda5c0c73d0 con 0x7fda5c0c88c0 2024-03-06T07:21:50.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.843+0000 7fda5b7fe640 1 --2- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c14f360 0x7fda5c155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:50.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda5bfff640 1 --2- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c158b60 0x7fda5c158ff0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fda54009e50 tx=0x7fda5402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daad5f126dc763a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda5a7fc640 1 -- 172.21.15.94:0/3331994755 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5402ca70 con 0x7fda5c158b60 2024-03-06T07:21:50.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda5a7fc640 1 -- 172.21.15.94:0/3331994755 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5402cc10 con 0x7fda5c158b60 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda5a7fc640 1 -- 172.21.15.94:0/3331994755 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda540343f0 con 0x7fda5c158b60 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda61b93640 1 -- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c158b60 msgr2=0x7fda5c158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda61b93640 1 --2- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c158b60 0x7fda5c158ff0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fda54009e50 tx=0x7fda5402bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda61b93640 1 -- 172.21.15.94:0/3331994755 shutdown_connections 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda61b93640 1 --2- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c158b60 0x7fda5c158ff0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda61b93640 1 --2- 172.21.15.94:0/3331994755 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c14f360 0x7fda5c155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.844+0000 7fda61b93640 1 -- 172.21.15.94:0/3331994755 >> 172.21.15.94:0/3331994755 conn(0x7fda5c0ac570 msgr2=0x7fda5c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.845+0000 7fda61b93640 1 -- 172.21.15.94:0/3331994755 shutdown_connections 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.845+0000 7fda61b93640 1 -- 172.21.15.94:0/3331994755 wait complete. 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 Processor -- start 2024-03-06T07:21:50.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 -- start start 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c88c0 0x7fda5c0c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c14f360 0x7fda5c0c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c158b60 0x7fda5c1e7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0c5a50 con 0x7fda5c0c88c0 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0c7c30 con 0x7fda5c14f360 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.846+0000 7fda61b93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0c5bd0 con 0x7fda5c158b60 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c88c0 0x7fda5c0c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c88c0 0x7fda5c0c4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35294/0 (socket says 172.21.15.94:35294) 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 -- 172.21.15.94:0/1788312729 learned_addr learned my addr 172.21.15.94:0/1788312729 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5affd640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c14f360 0x7fda5c0c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5bfff640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c158b60 0x7fda5c1e7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 -- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c158b60 msgr2=0x7fda5c1e7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c158b60 0x7fda5c1e7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 -- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c14f360 msgr2=0x7fda5c0c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c14f360 0x7fda5c0c4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 -- 172.21.15.94:0/1788312729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c1576c0 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda5b7fe640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c88c0 0x7fda5c0c4100 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00ec10 tx=0x7fda4c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda43fff640 1 -- 172.21.15.94:0/1788312729 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c017070 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda43fff640 1 -- 172.21.15.94:0/1788312729 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c00cc30 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c1579a0 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda43fff640 1 -- 172.21.15.94:0/1788312729 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c013470 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.847+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c157f60 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.848+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c153710 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.849+0000 7fda43fff640 1 -- 172.21.15.94:0/1788312729 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda4c01a070 con 0x7fda5c0c88c0 2024-03-06T07:21:50.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.849+0000 7fda43fff640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4806b500 0x7fda4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.849+0000 7fda43fff640 1 -- 172.21.15.94:0/1788312729 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fda4c08a590 con 0x7fda5c0c88c0 2024-03-06T07:21:50.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.849+0000 7fda5affd640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4806b500 0x7fda4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.850+0000 7fda5affd640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4806b500 0x7fda4806d9c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0bdd60 tx=0x7fda50007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:50.853+0000 7fda43fff640 1 -- 172.21.15.94:0/1788312729 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda5c153710 con 0x7fda5c0c88c0 2024-03-06T07:21:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.058+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda5c06cda0 con 0x7fda5c0c88c0 2024-03-06T07:21:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.059+0000 7fda43fff640 1 -- 172.21.15.94:0/1788312729 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fda4c058be0 con 0x7fda5c0c88c0 2024-03-06T07:21:51.061 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:51.061 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":36014,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1145122167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1145122167},{"type":"v1","addr":"172.21.15.145:6836","nonce":1145122167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:49.806589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6658":{"gid":6658,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6658,"qdb_cluster":[6658]},"id":7}]} 2024-03-06T07:21:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 82 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4806b500 msgr2=0x7fda4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4806b500 0x7fda4806d9c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0bdd60 tx=0x7fda50007e40 comp rx=0 tx=0).stop 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c88c0 msgr2=0x7fda5c0c4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c88c0 0x7fda5c0c4100 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00ec10 tx=0x7fda4c00c520 comp rx=0 tx=0).stop 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 shutdown_connections 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda4806b500 0x7fda4806d9c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda5c158b60 0x7fda5c1e7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda5c14f360 0x7fda5c0c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 --2- 172.21.15.94:0/1788312729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda5c0c88c0 0x7fda5c0c4100 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 >> 172.21.15.94:0/1788312729 conn(0x7fda5c0ac570 msgr2=0x7fda5c06c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 shutdown_connections 2024-03-06T07:21:51.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.061+0000 7fda61b93640 1 -- 172.21.15.94:0/1788312729 wait complete. 2024-03-06T07:21:51.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:38.904104+0000', 'modified': '2024-03-06T07:21:49.806589+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6658}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6658': {'gid': 6658, 'name': 'b', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.145:6837/744582506', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 744582506}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 744582506}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6658, 'qdb_cluster': [6658]} 2024-03-06T07:21:51.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:21:51.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:21:51.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.579+0000 7f8902feb640 1 Processor -- start 2024-03-06T07:21:51.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.579+0000 7f8902feb640 1 -- start start 2024-03-06T07:21:51.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8902feb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc06d680 0x7f88fc152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8902feb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f88fc072f40 0x7f88fc0c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8902feb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc0c1140 con 0x7f88fc0c2720 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8902feb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88fc0c1d50 con 0x7f88fc072f40 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8902feb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc1525a0 con 0x7f88fc06d680 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f88fbfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f88fc072f40 0x7f88fc0c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8900d87640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc06d680 0x7f88fc152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8901588640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f88fc0c2720 0x7f88fc154bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37576/0 (socket says 172.21.15.94:37576) 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f8901588640 1 -- 172.21.15.94:0/2502311426 learned_addr learned my addr 172.21.15.94:0/2502311426 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f88fbfff640 1 -- 172.21.15.94:0/2502311426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc06d680 msgr2=0x7f88fc152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f88fbfff640 1 --2- 172.21.15.94:0/2502311426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc06d680 0x7f88fc152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f88fbfff640 1 -- 172.21.15.94:0/2502311426 >> v1:172.21.15.145:6789/0 conn(0x7f88fc0c2720 legacy=0x7f88fc154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.580+0000 7f88fb7fe640 1 -- 172.21.15.94:0/2502311426 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304033884 0 0) 0x7f88fc0c1140 con 0x7f88fc0c2720 2024-03-06T07:21:51.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.581+0000 7f88fbfff640 1 -- 172.21.15.94:0/2502311426 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0c1420 con 0x7f88fc072f40 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.581+0000 7f88fbfff640 1 --2- 172.21.15.94:0/2502311426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 0x7f88fc0c21e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f88f00097a0 tx=0x7f88f002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0d6334f1a2ffa99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.581+0000 7f88fb7fe640 1 -- 172.21.15.94:0/2502311426 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f00315a0 con 0x7f88fc072f40 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.581+0000 7f88fb7fe640 1 -- 172.21.15.94:0/2502311426 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0037070 con 0x7f88fc072f40 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.581+0000 7f88fb7fe640 1 -- 172.21.15.94:0/2502311426 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0031960 con 0x7f88fc072f40 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 -- 172.21.15.94:0/2502311426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 msgr2=0x7f88fc0c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 --2- 172.21.15.94:0/2502311426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 0x7f88fc0c21e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f88f00097a0 tx=0x7f88f002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 -- 172.21.15.94:0/2502311426 shutdown_connections 2024-03-06T07:21:51.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 --2- 172.21.15.94:0/2502311426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 0x7f88fc0c21e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 --2- 172.21.15.94:0/2502311426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc06d680 0x7f88fc152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 -- 172.21.15.94:0/2502311426 >> 172.21.15.94:0/2502311426 conn(0x7f88fc06b1a0 msgr2=0x7f88fc06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 -- 172.21.15.94:0/2502311426 shutdown_connections 2024-03-06T07:21:51.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.582+0000 7f8902feb640 1 -- 172.21.15.94:0/2502311426 wait complete. 2024-03-06T07:21:51.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.583+0000 7f8902feb640 1 Processor -- start 2024-03-06T07:21:51.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.583+0000 7f8902feb640 1 -- start start 2024-03-06T07:21:51.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8902feb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88fc06d680 0x7f88fc0c04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8902feb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 0x7f88fc14a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8902feb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc0c2720 0x7f88fc14e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8902feb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc06da60 con 0x7f88fc06d680 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8902feb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc0c19e0 con 0x7f88fc072f40 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8902feb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc06dbe0 con 0x7f88fc0c2720 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88fc06d680 0x7f88fc0c04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88fc06d680 0x7f88fc0c04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35300/0 (socket says 172.21.15.94:35300) 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 -- 172.21.15.94:0/360353610 learned_addr learned my addr 172.21.15.94:0/360353610 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f88fbfff640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 0x7f88fc14a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8901588640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc0c2720 0x7f88fc14e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 -- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc0c2720 msgr2=0x7f88fc14e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc0c2720 0x7f88fc14e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 -- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 msgr2=0x7f88fc14a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 0x7f88fc14a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 -- 172.21.15.94:0/360353610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc14cce0 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.584+0000 7f8900d87640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88fc06d680 0x7f88fc0c04b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00e9e0 tx=0x7f88ec00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.585+0000 7f88f97fa640 1 -- 172.21.15.94:0/360353610 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec017070 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.585+0000 7f88f97fa640 1 -- 172.21.15.94:0/360353610 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec00c9d0 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.585+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc14cfc0 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.585+0000 7f88f97fa640 1 -- 172.21.15.94:0/360353610 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00cdc0 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.585+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc150580 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.586+0000 7f88f97fa640 1 -- 172.21.15.94:0/360353610 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88ec01a070 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.587+0000 7f88f97fa640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88e806b500 0x7f88e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.587+0000 7f88f97fa640 1 -- 172.21.15.94:0/360353610 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f88ec08a6e0 con 0x7f88fc06d680 2024-03-06T07:21:51.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.587+0000 7f88fbfff640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88e806b500 0x7f88e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f88fbfff640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88e806b500 0x7f88e806d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88f0036000 tx=0x7f88f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 Processor -- start 2024-03-06T07:21:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 -- start start 2024-03-06T07:21:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f553a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f553a550 0x5626f5583be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5574420 0x5626f5534680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f549e6b0 con 0x5626f5583800 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f5447a60 con 0x5626f553a550 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f549a750 con 0x5626f5574420 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8900d87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f553a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8900d87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f553a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35312/0 (socket says 172.21.15.94:35312) 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8900d87640 1 -- 172.21.15.94:0/1845879418 learned_addr learned my addr 172.21.15.94:0/1845879418 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f88fbfff640 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f553a550 0x5626f5583be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.588+0000 7f8901588640 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5574420 0x5626f5534680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8900d87640 1 -- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5574420 msgr2=0x5626f5534680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8900d87640 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5574420 0x5626f5534680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8900d87640 1 -- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f553a550 msgr2=0x5626f5583be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8900d87640 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f553a550 0x5626f5583be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8900d87640 1 -- 172.21.15.94:0/1845879418 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f5465f30 con 0x5626f5583800 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8900d87640 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f553a010 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f88ec088a30 tx=0x7f88ec058d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a9b59566fb0812 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f88e27fc640 1 -- 172.21.15.94:0/1845879418 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec021030 con 0x5626f5583800 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f88e27fc640 1 -- 172.21.15.94:0/1845879418 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec088e50 con 0x5626f5583800 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f88e27fc640 1 -- 172.21.15.94:0/1845879418 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec05f0c0 con 0x5626f5583800 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8905bcb740 1 -- 172.21.15.94:0/1845879418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 msgr2=0x5626f553a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f553a010 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f88ec088a30 tx=0x7f88ec058d40 comp rx=0 tx=0).stop 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8905bcb740 1 -- 172.21.15.94:0/1845879418 shutdown_connections 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5574420 0x5626f5534680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f553a550 0x5626f5583be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1845879418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f553a010 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.589+0000 7f8905bcb740 1 -- 172.21.15.94:0/1845879418 >> 172.21.15.94:0/1845879418 conn(0x5626f55367d0 msgr2=0x5626f555b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.590+0000 7f8905bcb740 1 -- 172.21.15.94:0/1845879418 shutdown_connections 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.590+0000 7f8905bcb740 1 -- 172.21.15.94:0/1845879418 wait complete. 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.590+0000 7f8905bcb740 1 -- start start 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.590+0000 7f8905bcb740 1 Processor -- start 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f558b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f5575100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5575640 0x5626f5557fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8905bcb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f5578590 con 0x5626f5575640 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f545aea0 con 0x5626f5574420 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f555bdb0 con 0x5626f5583800 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8905bcb740 10 client.0 _ll_register_callbacks cb 0x5626f545a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5575640 0x5626f5557fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5575640 0x5626f5557fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35326/0 (socket says 172.21.15.94:35326) 2024-03-06T07:21:51.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8900d87640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f558b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 -- 172.21.15.94:0/2323219023 learned_addr learned my addr 172.21.15.94:0/2323219023 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f88fbfff640 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f5575100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 -- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 msgr2=0x5626f5575100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f5575100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 -- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 msgr2=0x5626f558b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f558b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8901588640 1 -- 172.21.15.94:0/2323219023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626f545b270 con 0x5626f5575640 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.591+0000 7f8900d87640 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f558b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8901588640 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5575640 0x5626f5557fd0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f88f4007320 tx=0x7f88f40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabfb5569539c37e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f88db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f88db7fe640 1 -- 172.21.15.94:0/2323219023 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f40127c0 con 0x5626f5575640 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f88db7fe640 1 -- 172.21.15.94:0/2323219023 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f88f4012960 con 0x5626f5575640 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88ec088e50 con 0x5626f5575640 2024-03-06T07:21:51.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f88db7fe640 1 -- 172.21.15.94:0/2323219023 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88f4022050 con 0x5626f5575640 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 10 client.7176 fetch_fsmap learned FSMap version 82 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x5626f51b32d0 con 0x5626f5575640 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f88db7fe640 1 -- 172.21.15.94:0/2323219023 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec088e50 con 0x5626f5575640 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f88db7fe640 1 -- 172.21.15.94:0/2323219023 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f88f4025020 con 0x5626f5575640 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 10 client.7176 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 10 client.7176 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 20 client.7176 populate_metadata read hostname 'smithi094' 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f558e9b0 0x5626f55998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 4 client.7176 mds_command: new command op to 6658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 --> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626f54abbf0 con 0x5626f558e9b0 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.592+0000 7f8900d87640 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f558e9b0 0x5626f55998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.593+0000 7f88db7fe640 10 client.7176 ms_handle_connect on v2:172.21.15.145:6835/744582506 2024-03-06T07:21:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.593+0000 7f8900d87640 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f558e9b0 0x5626f55998b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5933344cfcceefb server_cookie=5b8ee76497b8f6b6 in_seq=0 out_seq=0 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f88db7fe640 1 -- 172.21.15.94:0/2323219023 <== mds.0 v2:172.21.15.145:6835/744582506 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5626f54abbf0 con 0x5626f558e9b0 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f88db7fe640 10 client.7176 handle_command_reply: tid=0 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 client.7176 shutdown 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5575640 msgr2=0x5626f5557fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5575640 0x5626f5557fd0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f88f4007320 tx=0x7f88f40077e0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 shutdown_connections 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f558e9b0 0x5626f55998b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5575640 0x5626f5557fd0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f5575100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 --2- 172.21.15.94:0/2323219023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f558b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f88db7fe640 0 client.7176 ms_handle_reset on v2:172.21.15.145:6835/744582506 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 >> 172.21.15.94:0/2323219023 conn(0x5626f553a550 msgr2=0x5626f555b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 shutdown_connections 2024-03-06T07:21:51.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 1 -- 172.21.15.94:0/2323219023 wait complete. 2024-03-06T07:21:51.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.594+0000 7f8905bcb740 20 client.7176 trim_cache size 0 max 16384 2024-03-06T07:21:51.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.610+0000 7f8905bcb740 1 Processor -- start 2024-03-06T07:21:51.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.610+0000 7f8905bcb740 1 -- start start 2024-03-06T07:21:51.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f557f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5575640 0x5626f5582d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f55385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8905bcb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc1f2ba0 con 0x5626f5583800 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc0c5970 con 0x5626f5574420 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc0c5f70 con 0x5626f5575640 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f55385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f55385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35332/0 (socket says 172.21.15.94:35332) 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 -- 172.21.15.94:0/1916274103 learned_addr learned my addr 172.21.15.94:0/1916274103 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8900d87640 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f557f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f88fbfff640 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5575640 0x5626f5582d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 -- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5575640 msgr2=0x5626f5582d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5575640 0x5626f5582d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 -- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 msgr2=0x5626f557f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f557f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8901588640 1 -- 172.21.15.94:0/1916274103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626f527d6b0 con 0x5626f5583800 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.611+0000 7f8900d87640 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f557f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8901588640 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f55385b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f88f4004d60 tx=0x7f88f400c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb48d27be6c61fbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f88e1ffb640 1 -- 172.21.15.94:0/1916274103 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4012890 con 0x5626f5583800 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f88e1ffb640 1 -- 172.21.15.94:0/1916274103 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f400b040 con 0x5626f5583800 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f88e1ffb640 1 -- 172.21.15.94:0/1916274103 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4015440 con 0x5626f5583800 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 -- 172.21.15.94:0/1916274103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 msgr2=0x5626f55385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f55385b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f88f4004d60 tx=0x7f88f400c000 comp rx=0 tx=0).stop 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 -- 172.21.15.94:0/1916274103 shutdown_connections 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5583800 0x5626f55385b0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5575640 0x5626f5582d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1916274103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5574420 0x5626f557f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 -- 172.21.15.94:0/1916274103 >> 172.21.15.94:0/1916274103 conn(0x5626f553a550 msgr2=0x5626f558af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 -- 172.21.15.94:0/1916274103 shutdown_connections 2024-03-06T07:21:51.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.612+0000 7f8905bcb740 1 -- 172.21.15.94:0/1916274103 wait complete. 2024-03-06T07:21:51.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.613+0000 7f8905bcb740 1 -- start start 2024-03-06T07:21:51.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.613+0000 7f8905bcb740 1 Processor -- start 2024-03-06T07:21:51.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5574420 0x5626f555bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5575640 0x5626f5576f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f55774d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f5575b60 con 0x5626f5574420 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626f558e830 con 0x5626f5575640 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626f5576830 con 0x5626f5583800 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 10 client.0 _ll_register_callbacks cb 0x7f88f4001f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5574420 0x5626f555bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5574420 0x5626f555bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35336/0 (socket says 172.21.15.94:35336) 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 -- 172.21.15.94:0/1073962256 learned_addr learned my addr 172.21.15.94:0/1073962256 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f88fbfff640 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5575640 0x5626f5576f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8901588640 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f55774d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 -- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 msgr2=0x5626f55774d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f55774d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 -- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5575640 msgr2=0x5626f5576f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5575640 0x5626f5576f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 -- 172.21.15.94:0/1073962256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626f559eeb0 con 0x5626f5574420 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8900d87640 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5574420 0x5626f555bb70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f88ec088b10 tx=0x7f88ec08a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431dd113a89dfde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f88db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f88db7fe640 1 -- 172.21.15.94:0/1073962256 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec021040 con 0x5626f5574420 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f88db7fe640 1 -- 172.21.15.94:0/1073962256 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f88ec083cb0 con 0x5626f5574420 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.614+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88f400b040 con 0x5626f5574420 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88db7fe640 1 -- 172.21.15.94:0/1073962256 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec054fe0 con 0x5626f5574420 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 10 client.7182 fetch_fsmap learned FSMap version 82 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x5626f555c0b0 con 0x5626f5574420 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88db7fe640 1 -- 172.21.15.94:0/1073962256 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f400b040 con 0x5626f5574420 2024-03-06T07:21:51.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88db7fe640 1 -- 172.21.15.94:0/1073962256 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f88ec077be0 con 0x5626f5574420 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 10 client.7182 fetch_fsmap finished waiting for FSMap version 82 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 10 client.7182 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 20 client.7182 populate_metadata read hostname 'smithi094' 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f5535330 0x5626f5535710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 4 client.7182 mds_command: new command op to 6658 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 --> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5626f553ad40 con 0x5626f5535330 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88fbfff640 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f5535330 0x5626f5535710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88db7fe640 10 client.7182 ms_handle_connect on v2:172.21.15.145:6835/744582506 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88fbfff640 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f5535330 0x5626f5535710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad25a131ce5fed40 server_cookie=7d7a9e318466d7c3 in_seq=0 out_seq=0 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88db7fe640 1 -- 172.21.15.94:0/1073962256 <== mds.0 v2:172.21.15.145:6835/744582506 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5626f553ad40 con 0x5626f5535330 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f88db7fe640 10 client.7182 handle_command_reply: tid=0 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.615+0000 7f8905bcb740 1 client.7182 shutdown 2024-03-06T07:21:51.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5574420 msgr2=0x5626f555bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5574420 0x5626f555bb70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f88ec088b10 tx=0x7f88ec08a070 comp rx=0 tx=0).stop 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 shutdown_connections 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:6835/744582506,v1:172.21.15.145:6837/744582506] conn(0x5626f5535330 0x5626f5535710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5626f5583800 0x5626f55774d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5626f5575640 0x5626f5576f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f88db7fe640 0 client.7182 ms_handle_reset on v2:172.21.15.145:6835/744582506 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 --2- 172.21.15.94:0/1073962256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5626f5574420 0x5626f555bb70 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 >> 172.21.15.94:0/1073962256 conn(0x5626f553a550 msgr2=0x5626f557d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 shutdown_connections 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 1 -- 172.21.15.94:0/1073962256 wait complete. 2024-03-06T07:21:51.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8905bcb740 20 client.7182 trim_cache size 0 max 16384 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 6658, 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.815746903, 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 82, 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 96, 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 96, 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 17.168414688999999 2024-03-06T07:21:51.625 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:21:51.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88e806b500 msgr2=0x7f88e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88e806b500 0x7f88e806d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f88f0036000 tx=0x7f88f0008030 comp rx=0 tx=0).stop 2024-03-06T07:21:51.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88fc06d680 msgr2=0x7f88fc0c04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88fc06d680 0x7f88fc0c04b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00e9e0 tx=0x7f88ec00eea0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 shutdown_connections 2024-03-06T07:21:51.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f88e806b500 0x7f88e806d9c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88fc0c2720 0x7f88fc14e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88fc072f40 0x7f88fc14a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 --2- 172.21.15.94:0/360353610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88fc06d680 0x7f88fc0c04b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 >> 172.21.15.94:0/360353610 conn(0x7f88fc06b1a0 msgr2=0x7f88fc06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.616+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 shutdown_connections 2024-03-06T07:21:51.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.617+0000 7f8902feb640 1 -- 172.21.15.94:0/360353610 wait complete. 2024-03-06T07:21:51.629 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:21:38.904104+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6658": { "addr": "172.21.15.145:6837/744582506", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 744582506, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 744582506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6658, "incarnation": 79, "join_fscid": 7, "name": "b", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 96, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-03-06T07:21:49.806589+0000", "qdb_cluster": [ 6658 ], "qdb_leader": 6658, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6658 } } } ], "standbys": [ { "addr": "172.21.15.94:6834/3090166857", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3090166857, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 3090166857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6945, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3798967274", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3798967274, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3798967274, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 74, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 35888, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6836/1145122167", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1145122167, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1145122167, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 80, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 36014, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:21:51.630 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 7, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:21:51.630 INFO:tasks.cephfs.test_failover:current = [{'gid': 35888, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3798967274', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3798967274}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3798967274}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 6658, 'name': 'b', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.145:6837/744582506', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 744582506}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 744582506}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6945, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/3090166857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3090166857}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 3090166857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 74}, {'gid': 36014, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/1145122167', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1145122167}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1145122167}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 80}] 2024-03-06T07:21:51.630 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:21:51.630 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:21:51.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:51.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:21:51.716 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:21:51.716 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:21:51.716 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:51.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:21:51.731 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:21:51.732 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:21:51.772 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:21:51.772 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:51.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.871+0000 7fa0a1681640 1 Processor -- start 2024-03-06T07:21:51.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.871+0000 7fa0a1681640 1 -- start start 2024-03-06T07:21:51.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.871+0000 7fa0a1681640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa09c158db0 0x7fa09c0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa0a1681640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c155110 con 0x7fa09c0c88b0 2024-03-06T07:21:51.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa0a1681640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c0c8f40 con 0x7fa09c0c5330 2024-03-06T07:21:51.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa0a1681640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09c1559e0 con 0x7fa09c158db0 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa09b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa09c0c88b0 0x7fa09c15fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36484/0 (socket says 172.21.15.94:36484) 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa09b7fe640 1 -- 172.21.15.94:0/2079685648 learned_addr learned my addr 172.21.15.94:0/2079685648 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa09a7fc640 1 --2- 172.21.15.94:0/2079685648 >> v2:172.21.15.145:3300/0 conn(0x7fa09c158db0 0x7fa09c0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa099ffb640 1 -- 172.21.15.94:0/2079685648 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730755102 0 0) 0x7fa094001490 con 0x7fa09c0c88b0 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa099ffb640 1 -- 172.21.15.94:0/2079685648 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa084003520 con 0x7fa09c0c88b0 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa09a7fc640 1 -- 172.21.15.94:0/2079685648 >> v1:172.21.15.145:6789/0 conn(0x7fa09c0c5330 legacy=0x7fa09c1586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa09a7fc640 1 -- 172.21.15.94:0/2079685648 >> v1:172.21.15.94:6789/0 conn(0x7fa09c0c88b0 legacy=0x7fa09c15fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.872+0000 7fa09a7fc640 1 -- 172.21.15.94:0/2079685648 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c155390 con 0x7fa09c158db0 2024-03-06T07:21:51.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.873+0000 7fa09a7fc640 1 --2- 172.21.15.94:0/2079685648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c158db0 0x7fa09c0c5710 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa0900097a0 tx=0x7fa09002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bf90294a24e736c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.873+0000 7fa099ffb640 1 -- 172.21.15.94:0/2079685648 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090035020 con 0x7fa09c158db0 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.873+0000 7fa099ffb640 1 -- 172.21.15.94:0/2079685648 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa090031830 con 0x7fa09c158db0 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.873+0000 7fa099ffb640 1 -- 172.21.15.94:0/2079685648 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090031be0 con 0x7fa09c158db0 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.874+0000 7fa0a1681640 1 -- 172.21.15.94:0/2079685648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c158db0 msgr2=0x7fa09c0c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.874+0000 7fa0a1681640 1 --2- 172.21.15.94:0/2079685648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c158db0 0x7fa09c0c5710 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa0900097a0 tx=0x7fa09002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.874+0000 7fa0a1681640 1 -- 172.21.15.94:0/2079685648 shutdown_connections 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.874+0000 7fa0a1681640 1 --2- 172.21.15.94:0/2079685648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c158db0 0x7fa09c0c5710 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.874+0000 7fa0a1681640 1 -- 172.21.15.94:0/2079685648 >> 172.21.15.94:0/2079685648 conn(0x7fa09c0ac560 msgr2=0x7fa09c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.874+0000 7fa0a1681640 1 -- 172.21.15.94:0/2079685648 shutdown_connections 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.874+0000 7fa0a1681640 1 -- 172.21.15.94:0/2079685648 wait complete. 2024-03-06T07:21:51.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.875+0000 7fa0a1681640 1 Processor -- start 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.875+0000 7fa0a1681640 1 -- start start 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.875+0000 7fa0a1681640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa09c0c5330 0x7fa09c14d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.875+0000 7fa0a1681640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c0c88b0 0x7fa09c14b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0a1681640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa09c149a60 0x7fa09c0c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0a1681640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c1529d0 con 0x7fa09c0c5330 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0a1681640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c06d090 con 0x7fa09c0c88b0 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0a1681640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c152b50 con 0x7fa09c149a60 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa09c0c5330 0x7fa09c14d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa09c0c5330 0x7fa09c14d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35348/0 (socket says 172.21.15.94:35348) 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 -- 172.21.15.94:0/3982745709 learned_addr learned my addr 172.21.15.94:0/3982745709 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09a7fc640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c0c88b0 0x7fa09c14b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09b7fe640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa09c149a60 0x7fa09c0c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 -- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa09c149a60 msgr2=0x7fa09c0c38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa09c149a60 0x7fa09c0c38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 -- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c0c88b0 msgr2=0x7fa09c14b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c0c88b0 0x7fa09c14b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 -- 172.21.15.94:0/3982745709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c14dfe0 con 0x7fa09c0c5330 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa09affd640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa09c0c5330 0x7fa09c14d900 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00c400 tx=0x7fa08c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0837fe640 1 -- 172.21.15.94:0/3982745709 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c0052d0 con 0x7fa09c0c5330 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0837fe640 1 -- 172.21.15.94:0/3982745709 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c005490 con 0x7fa09c0c5330 2024-03-06T07:21:51.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0837fe640 1 -- 172.21.15.94:0/3982745709 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c007260 con 0x7fa09c0c5330 2024-03-06T07:21:51.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c1ee3b0 con 0x7fa09c0c5330 2024-03-06T07:21:51.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.876+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c1ee970 con 0x7fa09c0c5330 2024-03-06T07:21:51.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.877+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c04f330 con 0x7fa09c0c5330 2024-03-06T07:21:51.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.878+0000 7fa0837fe640 1 -- 172.21.15.94:0/3982745709 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa08c007400 con 0x7fa09c0c5330 2024-03-06T07:21:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.878+0000 7fa0837fe640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa08806b4d0 0x7fa08806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.878+0000 7fa0837fe640 1 -- 172.21.15.94:0/3982745709 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa08c0897e0 con 0x7fa09c0c5330 2024-03-06T07:21:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.879+0000 7fa09a7fc640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa08806b4d0 0x7fa08806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.879+0000 7fa09a7fc640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa08806b4d0 0x7fa08806d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa090036000 tx=0x7fa090008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:51.883+0000 7fa0837fe640 1 -- 172.21.15.94:0/3982745709 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa08c057ec0 con 0x7fa09c0c5330 2024-03-06T07:21:52.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.090+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa09c06cdc0 con 0x7fa09c0c5330 2024-03-06T07:21:52.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.091+0000 7fa0837fe640 1 -- 172.21.15.94:0/3982745709 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fa08c05bf60 con 0x7fa09c0c5330 2024-03-06T07:21:52.093 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:52.093 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":36014,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1145122167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1145122167},{"type":"v1","addr":"172.21.15.145:6836","nonce":1145122167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:49.806589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6658":{"gid":6658,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6658,"qdb_cluster":[6658]},"id":7}]} 2024-03-06T07:21:52.093 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 82 2024-03-06T07:21:52.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa08806b4d0 msgr2=0x7fa08806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa08806b4d0 0x7fa08806d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa090036000 tx=0x7fa090008030 comp rx=0 tx=0).stop 2024-03-06T07:21:52.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa09c0c5330 msgr2=0x7fa09c14d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa09c0c5330 0x7fa09c14d900 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00c400 tx=0x7fa08c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 shutdown_connections 2024-03-06T07:21:52.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa08806b4d0 0x7fa08806d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa09c149a60 0x7fa09c0c38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa09c0c88b0 0x7fa09c14b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 --2- 172.21.15.94:0/3982745709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa09c0c5330 0x7fa09c14d900 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 >> 172.21.15.94:0/3982745709 conn(0x7fa09c0ac560 msgr2=0x7fa09c154920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 shutdown_connections 2024-03-06T07:21:52.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.093+0000 7fa0a1681640 1 -- 172.21.15.94:0/3982745709 wait complete. 2024-03-06T07:21:52.106 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.204+0000 7fe19e2c9640 1 Processor -- start 2024-03-06T07:21:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.204+0000 7fe19e2c9640 1 -- start start 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe19e2c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe19814f790 0x7fe1981529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe19e2c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1980c1aa0 0x7fe1980c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe19e2c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1980c28d0 con 0x7fe1980c5c60 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe19e2c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1980c2680 con 0x7fe19814f790 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe19e2c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980c2bb0 con 0x7fe1980c1aa0 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe1977fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1980c1aa0 0x7fe1980c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe197fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe19814f790 0x7fe1981529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe19c866640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fe1980c5c60 0x7fe198155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37580/0 (socket says 172.21.15.94:37580) 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe19c866640 1 -- 172.21.15.94:0/2518292944 learned_addr learned my addr 172.21.15.94:0/2518292944 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe197fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe19814f790 0x7fe1981529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:50738/0 (socket says 172.21.15.94:50738) 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe1977fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1980c1aa0 0x7fe1980c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:37568/0 (socket says 172.21.15.94:37568) 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe197fff640 1 -- 172.21.15.94:0/2518292944 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1980c1aa0 msgr2=0x7fe1980c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe197fff640 1 --2- 172.21.15.94:0/2518292944 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1980c1aa0 0x7fe1980c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe197fff640 1 -- 172.21.15.94:0/2518292944 >> v1:172.21.15.145:6789/0 conn(0x7fe1980c5c60 legacy=0x7fe198155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.205+0000 7fe197fff640 1 -- 172.21.15.94:0/2518292944 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1980c2e90 con 0x7fe19814f790 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.206+0000 7fe197fff640 1 --2- 172.21.15.94:0/2518292944 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19814f790 0x7fe1981529d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097a0 tx=0x7fe18802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5012b16ab6322104 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.206+0000 7fe196ffd640 1 -- 172.21.15.94:0/2518292944 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188035020 con 0x7fe19814f790 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.206+0000 7fe196ffd640 1 -- 172.21.15.94:0/2518292944 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe188031830 con 0x7fe19814f790 2024-03-06T07:21:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.206+0000 7fe196ffd640 1 -- 172.21.15.94:0/2518292944 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188031c00 con 0x7fe19814f790 2024-03-06T07:21:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 -- 172.21.15.94:0/2518292944 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19814f790 msgr2=0x7fe1981529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/2518292944 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19814f790 0x7fe1981529d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097a0 tx=0x7fe18802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 -- 172.21.15.94:0/2518292944 shutdown_connections 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/2518292944 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1980c1aa0 0x7fe1980c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/2518292944 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19814f790 0x7fe1981529d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 -- 172.21.15.94:0/2518292944 >> 172.21.15.94:0/2518292944 conn(0x7fe1980c6710 msgr2=0x7fe1980c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 -- 172.21.15.94:0/2518292944 shutdown_connections 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.207+0000 7fe19e2c9640 1 -- 172.21.15.94:0/2518292944 wait complete. 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.208+0000 7fe19e2c9640 1 Processor -- start 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.208+0000 7fe19e2c9640 1 -- start start 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe19e2c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1980c1aa0 0x7fe198072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe19e2c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1980c5c60 0x7fe1981debb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe19e2c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19814f790 0x7fe198153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe19e2c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19806cc00 con 0x7fe1980c1aa0 2024-03-06T07:21:52.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe19e2c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980c0d60 con 0x7fe1980c5c60 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe19e2c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19806cd80 con 0x7fe19814f790 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1980c1aa0 0x7fe198072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1980c1aa0 0x7fe198072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35362/0 (socket says 172.21.15.94:35362) 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 -- 172.21.15.94:0/3818075623 learned_addr learned my addr 172.21.15.94:0/3818075623 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe1977fe640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1980c5c60 0x7fe1981debb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe19c866640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19814f790 0x7fe198153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 -- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19814f790 msgr2=0x7fe198153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19814f790 0x7fe198153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 -- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1980c5c60 msgr2=0x7fe1981debb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1980c5c60 0x7fe1981debb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 -- 172.21.15.94:0/3818075623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1981e97f0 con 0x7fe1980c1aa0 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe1977fe640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1980c5c60 0x7fe1981debb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe197fff640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1980c1aa0 0x7fe198072ee0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097a0 tx=0x7fe1880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.209+0000 7fe194ff9640 1 -- 172.21.15.94:0/3818075623 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18803a970 con 0x7fe1980c1aa0 2024-03-06T07:21:52.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.210+0000 7fe194ff9640 1 -- 172.21.15.94:0/3818075623 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18803ab10 con 0x7fe1980c1aa0 2024-03-06T07:21:52.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.210+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe198154430 con 0x7fe1980c1aa0 2024-03-06T07:21:52.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.210+0000 7fe194ff9640 1 -- 172.21.15.94:0/3818075623 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1880443f0 con 0x7fe1980c1aa0 2024-03-06T07:21:52.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.210+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1981df0f0 con 0x7fe1980c1aa0 2024-03-06T07:21:52.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.210+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19804f330 con 0x7fe1980c1aa0 2024-03-06T07:21:52.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.211+0000 7fe194ff9640 1 -- 172.21.15.94:0/3818075623 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe18804b060 con 0x7fe1980c1aa0 2024-03-06T07:21:52.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.211+0000 7fe194ff9640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe18406b500 0x7fe18406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.211+0000 7fe194ff9640 1 -- 172.21.15.94:0/3818075623 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe1880abde0 con 0x7fe1980c1aa0 2024-03-06T07:21:52.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.211+0000 7fe1977fe640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe18406b500 0x7fe18406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.212+0000 7fe1977fe640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe18406b500 0x7fe18406d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe1980a6a90 tx=0x7fe18c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.216+0000 7fe194ff9640 1 -- 172.21.15.94:0/3818075623 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe18803f070 con 0x7fe1980c1aa0 2024-03-06T07:21:52.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.387+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1980c0b40 con 0x7fe1980c1aa0 2024-03-06T07:21:52.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.388+0000 7fe194ff9640 1 -- 172.21.15.94:0/3818075623 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+17004 (secure 0 0 0) 0x7fe18807a440 con 0x7fe1980c1aa0 2024-03-06T07:21:52.390 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:52.390 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:46.796076+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:37.116515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:37.869999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/3774173957":"2024-03-07T07:21:46.796043+0000","172.21.15.145:6834/3774173957":"2024-03-07T07:21:46.796043+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:52.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe18406b500 msgr2=0x7fe18406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe18406b500 0x7fe18406d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe1980a6a90 tx=0x7fe18c009190 comp rx=0 tx=0).stop 2024-03-06T07:21:52.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1980c1aa0 msgr2=0x7fe198072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1980c1aa0 0x7fe198072ee0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097a0 tx=0x7fe1880046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 shutdown_connections 2024-03-06T07:21:52.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe18406b500 0x7fe18406d9c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19814f790 0x7fe198153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1980c5c60 0x7fe1981debb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 --2- 172.21.15.94:0/3818075623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1980c1aa0 0x7fe198072ee0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 >> 172.21.15.94:0/3818075623 conn(0x7fe1980c6710 msgr2=0x7fe1980c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 shutdown_connections 2024-03-06T07:21:52.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.390+0000 7fe19e2c9640 1 -- 172.21.15.94:0/3818075623 wait complete. 2024-03-06T07:21:52.403 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:37.116515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:37.869999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:52.403 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:52.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.504+0000 7f920038a640 1 Processor -- start 2024-03-06T07:21:52.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.504+0000 7f920038a640 1 -- start start 2024-03-06T07:21:52.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.505+0000 7f920038a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91f8155bd0 0x7f91f8152350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.505+0000 7f920038a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f80670c0 con 0x7f91f80c26c0 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.505+0000 7f920038a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f806b0f0 con 0x7f91f80c2aa0 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.505+0000 7f920038a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f8156060 con 0x7f91f8155bd0 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.505+0000 7f91fe927640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91f8155bd0 0x7f91f8152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.505+0000 7f91fe126640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f91f80c26c0 0x7f91f806bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36486/0 (socket says 172.21.15.94:36486) 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.505+0000 7f91fe126640 1 -- 172.21.15.94:0/1300523658 learned_addr learned my addr 172.21.15.94:0/1300523658 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.506+0000 7f91fe927640 1 -- 172.21.15.94:0/1300523658 >> v1:172.21.15.145:6789/0 conn(0x7f91f80c2aa0 legacy=0x7f91f80c0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.506+0000 7f91fe927640 1 -- 172.21.15.94:0/1300523658 >> v1:172.21.15.94:6789/0 conn(0x7f91f80c26c0 legacy=0x7f91f806bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.506+0000 7f91fe927640 1 -- 172.21.15.94:0/1300523658 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f806b3d0 con 0x7f91f8155bd0 2024-03-06T07:21:52.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.506+0000 7f91fd124640 1 -- 172.21.15.94:0/1300523658 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858690572 0 0) 0x7f91f80670c0 con 0x7f91f80c26c0 2024-03-06T07:21:52.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.506+0000 7f91fe927640 1 --2- 172.21.15.94:0/1300523658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8152350 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009d90 tx=0x7f91f002bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4163ea9e87f236fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f91fd124640 1 -- 172.21.15.94:0/1300523658 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0039070 con 0x7f91f8155bd0 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f91fd124640 1 -- 172.21.15.94:0/1300523658 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f002ca40 con 0x7f91f8155bd0 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f91fd124640 1 -- 172.21.15.94:0/1300523658 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f002ce10 con 0x7f91f8155bd0 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f920038a640 1 -- 172.21.15.94:0/1300523658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 msgr2=0x7f91f8152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f920038a640 1 --2- 172.21.15.94:0/1300523658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8152350 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009d90 tx=0x7f91f002bc90 comp rx=0 tx=0).stop 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f920038a640 1 -- 172.21.15.94:0/1300523658 shutdown_connections 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f920038a640 1 --2- 172.21.15.94:0/1300523658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8152350 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f920038a640 1 -- 172.21.15.94:0/1300523658 >> 172.21.15.94:0/1300523658 conn(0x7f91f80c3b40 msgr2=0x7f91f80c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f920038a640 1 -- 172.21.15.94:0/1300523658 shutdown_connections 2024-03-06T07:21:52.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.507+0000 7f920038a640 1 -- 172.21.15.94:0/1300523658 wait complete. 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.508+0000 7f920038a640 1 Processor -- start 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.508+0000 7f920038a640 1 -- start start 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f920038a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91f80c26c0 0x7f91f81edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f920038a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f80c2aa0 0x7f91f8146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f920038a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f920038a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8066d00 con 0x7f91f8155bd0 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f920038a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f806b740 con 0x7f91f80c26c0 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f920038a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8066e80 con 0x7f91f80c2aa0 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8146e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35374/0 (socket says 172.21.15.94:35374) 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 -- 172.21.15.94:0/196969927 learned_addr learned my addr 172.21.15.94:0/196969927 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:52.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe126640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91f80c26c0 0x7f91f81edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fd925640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f80c2aa0 0x7f91f8146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 -- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f80c2aa0 msgr2=0x7f91f8146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f80c2aa0 0x7f91f8146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 -- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91f80c26c0 msgr2=0x7f91f81edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91f80c26c0 0x7f91f81edca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.509+0000 7f91fe927640 1 -- 172.21.15.94:0/196969927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f81ee570 con 0x7f91f8155bd0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.510+0000 7f91fe927640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8146e90 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f91f0037040 tx=0x7f91f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.510+0000 7f91eeffd640 1 -- 172.21.15.94:0/196969927 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0039050 con 0x7f91f8155bd0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.510+0000 7f91eeffd640 1 -- 172.21.15.94:0/196969927 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f003a040 con 0x7f91f8155bd0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.510+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f80ac840 con 0x7f91f8155bd0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.510+0000 7f91eeffd640 1 -- 172.21.15.94:0/196969927 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f003e610 con 0x7f91f8155bd0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.510+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f80abb80 con 0x7f91f8155bd0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.511+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f804f330 con 0x7f91f8155bd0 2024-03-06T07:21:52.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.511+0000 7f91eeffd640 1 -- 172.21.15.94:0/196969927 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91f0047030 con 0x7f91f8155bd0 2024-03-06T07:21:52.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.512+0000 7f91eeffd640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91dc06b500 0x7f91dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.512+0000 7f91eeffd640 1 -- 172.21.15.94:0/196969927 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f91f00ac270 con 0x7f91f8155bd0 2024-03-06T07:21:52.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.512+0000 7f91fe126640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91dc06b500 0x7f91dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.512+0000 7f91fe126640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91dc06b500 0x7f91dc06d9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f91f4007fd0 tx=0x7f91f4007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.516+0000 7f91eeffd640 1 -- 172.21.15.94:0/196969927 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91f007aa10 con 0x7f91f8155bd0 2024-03-06T07:21:52.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.686+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91f8067590 con 0x7f91f8155bd0 2024-03-06T07:21:52.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.687+0000 7f91eeffd640 1 -- 172.21.15.94:0/196969927 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91f007eab0 con 0x7f91f8155bd0 2024-03-06T07:21:52.689 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:21:52.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91dc06b500 msgr2=0x7f91dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91dc06b500 0x7f91dc06d9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f91f4007fd0 tx=0x7f91f4007240 comp rx=0 tx=0).stop 2024-03-06T07:21:52.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 msgr2=0x7f91f8146e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8146e90 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f91f0037040 tx=0x7f91f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:52.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 shutdown_connections 2024-03-06T07:21:52.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91dc06b500 0x7f91dc06d9c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91f8155bd0 0x7f91f8146e90 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f80c2aa0 0x7f91f8146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 --2- 172.21.15.94:0/196969927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91f80c26c0 0x7f91f81edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 >> 172.21.15.94:0/196969927 conn(0x7f91f80c3b40 msgr2=0x7f91f80c4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 shutdown_connections 2024-03-06T07:21:52.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.689+0000 7f920038a640 1 -- 172.21.15.94:0/196969927 wait complete. 2024-03-06T07:21:52.702 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:52.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.803+0000 7f67fdf32640 1 Processor -- start 2024-03-06T07:21:52.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.803+0000 7f67fdf32640 1 -- start start 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67fdf32640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67f80c4740 0x7f67f81539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67fdf32640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f80c62a0 con 0x7f67f80c69a0 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67fdf32640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f80c4c00 con 0x7f67f80a8290 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67fdf32640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f8153f00 con 0x7f67f80c4740 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67f7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f67f80c69a0 0x7f67f8072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36502/0 (socket says 172.21.15.94:36502) 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67f7fff640 1 -- 172.21.15.94:0/413038345 learned_addr learned my addr 172.21.15.94:0/413038345 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67eeffd640 1 --2- 172.21.15.94:0/413038345 >> v2:172.21.15.145:3300/0 conn(0x7f67f80c4740 0x7f67f81539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67f6ffd640 1 -- 172.21.15.94:0/413038345 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669368702 0 0) 0x7f67f80c62a0 con 0x7f67f80c69a0 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.804+0000 7f67f6ffd640 1 -- 172.21.15.94:0/413038345 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e4003520 con 0x7f67f80c69a0 2024-03-06T07:21:52.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.805+0000 7f67eeffd640 1 -- 172.21.15.94:0/413038345 >> v1:172.21.15.145:6789/0 conn(0x7f67f80a8290 legacy=0x7f67f80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.805+0000 7f67f6ffd640 1 -- 172.21.15.94:0/413038345 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551234864 0 0) 0x7f67f80c4c00 con 0x7f67f80a8290 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.805+0000 7f67eeffd640 1 -- 172.21.15.94:0/413038345 >> v1:172.21.15.94:6789/0 conn(0x7f67f80c69a0 legacy=0x7f67f8072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.805+0000 7f67eeffd640 1 -- 172.21.15.94:0/413038345 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f80c4ee0 con 0x7f67f80c4740 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.805+0000 7f67eeffd640 1 --2- 172.21.15.94:0/413038345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80c4740 0x7f67f81539c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f67e00097a0 tx=0x7f67e002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d82c8861c6ad008f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.805+0000 7f67f6ffd640 1 -- 172.21.15.94:0/413038345 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e00315a0 con 0x7f67f80c4740 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67f6ffd640 1 -- 172.21.15.94:0/413038345 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e0037070 con 0x7f67f80c4740 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67f6ffd640 1 -- 172.21.15.94:0/413038345 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0031960 con 0x7f67f80c4740 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67fdf32640 1 -- 172.21.15.94:0/413038345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80c4740 msgr2=0x7f67f81539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67fdf32640 1 --2- 172.21.15.94:0/413038345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80c4740 0x7f67f81539c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f67e00097a0 tx=0x7f67e002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67fdf32640 1 -- 172.21.15.94:0/413038345 shutdown_connections 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67fdf32640 1 --2- 172.21.15.94:0/413038345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80c4740 0x7f67f81539c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67fdf32640 1 -- 172.21.15.94:0/413038345 >> 172.21.15.94:0/413038345 conn(0x7f67f80c5560 msgr2=0x7f67f80c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67fdf32640 1 -- 172.21.15.94:0/413038345 shutdown_connections 2024-03-06T07:21:52.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.806+0000 7f67fdf32640 1 -- 172.21.15.94:0/413038345 wait complete. 2024-03-06T07:21:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.807+0000 7f67fdf32640 1 Processor -- start 2024-03-06T07:21:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.807+0000 7f67fdf32640 1 -- start start 2024-03-06T07:21:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67fdf32640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80a8290 0x7f67f8156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67fdf32640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67f80c4740 0x7f67f81e9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67fdf32640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f80c69a0 0x7f67f814c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67fdf32640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f8072d90 con 0x7f67f80c69a0 2024-03-06T07:21:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67fdf32640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f806fc80 con 0x7f67f80a8290 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67fdf32640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f8072f10 con 0x7f67f80c4740 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f80c69a0 0x7f67f814c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f80c69a0 0x7f67f814c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35376/0 (socket says 172.21.15.94:35376) 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 -- 172.21.15.94:0/3033627197 learned_addr learned my addr 172.21.15.94:0/3033627197 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67eeffd640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67f80c4740 0x7f67f81e9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f77fe640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80a8290 0x7f67f8156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 -- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67f80c4740 msgr2=0x7f67f81e9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67f80c4740 0x7f67f81e9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 -- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80a8290 msgr2=0x7f67f8156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80a8290 0x7f67f8156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.808+0000 7f67f7fff640 1 -- 172.21.15.94:0/3033627197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f80abcf0 con 0x7f67f80c69a0 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.809+0000 7f67f7fff640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f80c69a0 0x7f67f814c230 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f67f000cad0 tx=0x7f67f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.809+0000 7f67f4ff9640 1 -- 172.21.15.94:0/3033627197 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f0017070 con 0x7f67f80c69a0 2024-03-06T07:21:52.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.809+0000 7f67f4ff9640 1 -- 172.21.15.94:0/3033627197 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f000ac30 con 0x7f67f80c69a0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.809+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f80abfd0 con 0x7f67f80c69a0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.809+0000 7f67f4ff9640 1 -- 172.21.15.94:0/3033627197 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f0012400 con 0x7f67f80c69a0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.809+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f80be300 con 0x7f67f80c69a0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.810+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f804f330 con 0x7f67f80c69a0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.810+0000 7f67f4ff9640 1 -- 172.21.15.94:0/3033627197 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67f001a070 con 0x7f67f80c69a0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.811+0000 7f67f4ff9640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67d806b4d0 0x7f67d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.811+0000 7f67f4ff9640 1 -- 172.21.15.94:0/3033627197 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f67f008a050 con 0x7f67f80c69a0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.811+0000 7f67f77fe640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67d806b4d0 0x7f67d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.811+0000 7f67f77fe640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67d806b4d0 0x7f67d806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f67e8000f60 tx=0x7f67e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.815+0000 7f67f4ff9640 1 -- 172.21.15.94:0/3033627197 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67f0058730 con 0x7f67f80c69a0 2024-03-06T07:21:52.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.985+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f67f80c8480 con 0x7f67f80c69a0 2024-03-06T07:21:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.985+0000 7f67f4ff9640 1 -- 172.21.15.94:0/3033627197 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7f67f005c7d0 con 0x7f67f80c69a0 2024-03-06T07:21:52.987 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:21:52.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67d806b4d0 msgr2=0x7f67d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67d806b4d0 0x7f67d806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f67e8000f60 tx=0x7f67e8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f80c69a0 msgr2=0x7f67f814c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f80c69a0 0x7f67f814c230 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f67f000cad0 tx=0x7f67f000cf90 comp rx=0 tx=0).stop 2024-03-06T07:21:52.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 shutdown_connections 2024-03-06T07:21:52.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67d806b4d0 0x7f67d806d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f80c69a0 0x7f67f814c230 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67f80c4740 0x7f67f81e9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 --2- 172.21.15.94:0/3033627197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67f80a8290 0x7f67f8156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 >> 172.21.15.94:0/3033627197 conn(0x7f67f80c5560 msgr2=0x7f67f8150150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 shutdown_connections 2024-03-06T07:21:52.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:52.987+0000 7f67fdf32640 1 -- 172.21.15.94:0/3033627197 wait complete. 2024-03-06T07:21:53.000 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:53.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.101+0000 7f79084c3640 1 Processor -- start 2024-03-06T07:21:53.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.101+0000 7f79084c3640 1 -- start start 2024-03-06T07:21:53.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.102+0000 7f79084c3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f790006ca70 0x7f790014e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.102+0000 7f79084c3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79000b0450 con 0x7f79000b0070 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.102+0000 7f79084c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79001560e0 con 0x7f7900155c30 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.102+0000 7f79084c3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790006c020 con 0x7f790006ca70 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.102+0000 7f790625f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f79000b0070 0x7f79000c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36510/0 (socket says 172.21.15.94:36510) 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.102+0000 7f790625f640 1 -- 172.21.15.94:0/32580958 learned_addr learned my addr 172.21.15.94:0/32580958 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.102+0000 7f7906a60640 1 --2- 172.21.15.94:0/32580958 >> v2:172.21.15.145:3300/0 conn(0x7f790006ca70 0x7f790014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.103+0000 7f790525d640 1 -- 172.21.15.94:0/32580958 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857041923 0 0) 0x7f78fc001490 con 0x7f79000b0070 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.103+0000 7f790525d640 1 -- 172.21.15.94:0/32580958 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ec003520 con 0x7f79000b0070 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.103+0000 7f7906a60640 1 -- 172.21.15.94:0/32580958 >> v1:172.21.15.145:6789/0 conn(0x7f7900155c30 legacy=0x7f79000c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.103+0000 7f7906a60640 1 -- 172.21.15.94:0/32580958 >> v1:172.21.15.94:6789/0 conn(0x7f79000b0070 legacy=0x7f79000c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.103+0000 7f7906a60640 1 -- 172.21.15.94:0/32580958 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79000c77f0 con 0x7f790006ca70 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.103+0000 7f7906a60640 1 --2- 172.21.15.94:0/32580958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790006ca70 0x7f790014e800 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f78f80097a0 tx=0x7f78f802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b41cb4d3c0d78739 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f790525d640 1 -- 172.21.15.94:0/32580958 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8035020 con 0x7f790006ca70 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f790525d640 1 -- 172.21.15.94:0/32580958 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f8031830 con 0x7f790006ca70 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f790525d640 1 -- 172.21.15.94:0/32580958 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8031be0 con 0x7f790006ca70 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f79084c3640 1 -- 172.21.15.94:0/32580958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790006ca70 msgr2=0x7f790014e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f79084c3640 1 --2- 172.21.15.94:0/32580958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790006ca70 0x7f790014e800 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f78f80097a0 tx=0x7f78f802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f79084c3640 1 -- 172.21.15.94:0/32580958 shutdown_connections 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f79084c3640 1 --2- 172.21.15.94:0/32580958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790006ca70 0x7f790014e800 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f79084c3640 1 -- 172.21.15.94:0/32580958 >> 172.21.15.94:0/32580958 conn(0x7f79000ac590 msgr2=0x7f79000ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f79084c3640 1 -- 172.21.15.94:0/32580958 shutdown_connections 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.104+0000 7f79084c3640 1 -- 172.21.15.94:0/32580958 wait complete. 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.105+0000 7f79084c3640 1 Processor -- start 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.105+0000 7f79084c3640 1 -- start start 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f79084c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f790006ca70 0x7f79001eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f79084c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79000b0070 0x7f7900158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f79084c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7900155c30 0x7f79001548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f79084c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790006c550 con 0x7f790006ca70 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f79084c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790006c270 con 0x7f7900155c30 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f79084c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790006da80 con 0x7f79000b0070 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f790006ca70 0x7f79001eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f790006ca70 0x7f79001eaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35386/0 (socket says 172.21.15.94:35386) 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 -- 172.21.15.94:0/1478624824 learned_addr learned my addr 172.21.15.94:0/1478624824 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:53.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f7906a60640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7900155c30 0x7f79001548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f7905a5e640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79000b0070 0x7f7900158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 -- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79000b0070 msgr2=0x7f7900158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79000b0070 0x7f7900158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 -- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7900155c30 msgr2=0x7f79001548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7900155c30 0x7f79001548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.106+0000 7f790625f640 1 -- 172.21.15.94:0/1478624824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79000c3e40 con 0x7f790006ca70 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.107+0000 7f790625f640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f790006ca70 0x7f79001eaea0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00cbd0 tx=0x7f78fc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.107+0000 7f78f6ffd640 1 -- 172.21.15.94:0/1478624824 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc017070 con 0x7f790006ca70 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.107+0000 7f78f6ffd640 1 -- 172.21.15.94:0/1478624824 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc00ace0 con 0x7f790006ca70 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.107+0000 7f78f6ffd640 1 -- 172.21.15.94:0/1478624824 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc012620 con 0x7f790006ca70 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.107+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79001eb3e0 con 0x7f790006ca70 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.107+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79000c3b60 con 0x7f790006ca70 2024-03-06T07:21:53.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.108+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790004f330 con 0x7f790006ca70 2024-03-06T07:21:53.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.108+0000 7f78f6ffd640 1 -- 172.21.15.94:0/1478624824 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78fc01a070 con 0x7f790006ca70 2024-03-06T07:21:53.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.109+0000 7f78f6ffd640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b4d0 0x7f78e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.109+0000 7f78f6ffd640 1 -- 172.21.15.94:0/1478624824 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f78fc08a380 con 0x7f790006ca70 2024-03-06T07:21:53.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.109+0000 7f7905a5e640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b4d0 0x7f78e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.109+0000 7f7905a5e640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b4d0 0x7f78e406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f79001e8ce0 tx=0x7f78f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.113+0000 7f78f6ffd640 1 -- 172.21.15.94:0/1478624824 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79000b0450 con 0x7f790006ca70 2024-03-06T07:21:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.284+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f79000ada00 con 0x7f790006ca70 2024-03-06T07:21:53.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.285+0000 7f78f6ffd640 1 -- 172.21.15.94:0/1478624824 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f78fc058a60 con 0x7f790006ca70 2024-03-06T07:21:53.287 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b4d0 msgr2=0x7f78e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b4d0 0x7f78e406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f79001e8ce0 tx=0x7f78f0009040 comp rx=0 tx=0).stop 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f790006ca70 msgr2=0x7f79001eaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f790006ca70 0x7f79001eaea0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00cbd0 tx=0x7f78fc00a430 comp rx=0 tx=0).stop 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 shutdown_connections 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b4d0 0x7f78e406d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7900155c30 0x7f79001548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79000b0070 0x7f7900158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 --2- 172.21.15.94:0/1478624824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f790006ca70 0x7f79001eaea0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 >> 172.21.15.94:0/1478624824 conn(0x7f79000ac590 msgr2=0x7f79000ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 shutdown_connections 2024-03-06T07:21:53.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.287+0000 7f79084c3640 1 -- 172.21.15.94:0/1478624824 wait complete. 2024-03-06T07:21:53.299 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:53.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.397+0000 7fc881193640 1 Processor -- start 2024-03-06T07:21:53.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.397+0000 7fc881193640 1 -- start start 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc881193640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc87c0c88c0 0x7fc87c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc881193640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c0c73d0 con 0x7fc87c158b60 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc881193640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c0be2e0 con 0x7fc87c14f360 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc881193640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c159700 con 0x7fc87c0c88c0 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc87a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc87c0c88c0 0x7fc87c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc87b59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc87c158b60 0x7fc87c158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36526/0 (socket says 172.21.15.94:36526) 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc87b59e640 1 -- 172.21.15.94:0/704497428 learned_addr learned my addr 172.21.15.94:0/704497428 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc87a59c640 1 -- 172.21.15.94:0/704497428 >> v1:172.21.15.145:6789/0 conn(0x7fc87c14f360 legacy=0x7fc87c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc87a59c640 1 -- 172.21.15.94:0/704497428 >> v1:172.21.15.94:6789/0 conn(0x7fc87c158b60 legacy=0x7fc87c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc879d9b640 1 -- 172.21.15.94:0/704497428 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871688999 0 0) 0x7fc87c0be2e0 con 0x7fc87c14f360 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.398+0000 7fc87a59c640 1 -- 172.21.15.94:0/704497428 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0c7670 con 0x7fc87c0c88c0 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.399+0000 7fc87a59c640 1 --2- 172.21.15.94:0/704497428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c0c88c0 0x7fc87c155c10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc8700097a0 tx=0x7fc87002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f643f4c5f9d65b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.399+0000 7fc879d9b640 1 -- 172.21.15.94:0/704497428 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870037070 con 0x7fc87c0c88c0 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.399+0000 7fc879d9b640 1 -- 172.21.15.94:0/704497428 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc870030710 con 0x7fc87c0c88c0 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.399+0000 7fc879d9b640 1 -- 172.21.15.94:0/704497428 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870030ae0 con 0x7fc87c0c88c0 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.400+0000 7fc881193640 1 -- 172.21.15.94:0/704497428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c0c88c0 msgr2=0x7fc87c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.400+0000 7fc881193640 1 --2- 172.21.15.94:0/704497428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c0c88c0 0x7fc87c155c10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc8700097a0 tx=0x7fc87002b710 comp rx=0 tx=0).stop 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.400+0000 7fc881193640 1 -- 172.21.15.94:0/704497428 shutdown_connections 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.400+0000 7fc881193640 1 --2- 172.21.15.94:0/704497428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c0c88c0 0x7fc87c155c10 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.400+0000 7fc881193640 1 -- 172.21.15.94:0/704497428 >> 172.21.15.94:0/704497428 conn(0x7fc87c0ac570 msgr2=0x7fc87c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.400+0000 7fc881193640 1 -- 172.21.15.94:0/704497428 shutdown_connections 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.400+0000 7fc881193640 1 -- 172.21.15.94:0/704497428 wait complete. 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.401+0000 7fc881193640 1 Processor -- start 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.401+0000 7fc881193640 1 -- start start 2024-03-06T07:21:53.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc881193640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc87c0c88c0 0x7fc87c157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc881193640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c14f360 0x7fc87c0c0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc881193640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc87c158b60 0x7fc87c0c0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc881193640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0c7c30 con 0x7fc87c14f360 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc881193640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0c5a50 con 0x7fc87c0c88c0 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc881193640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0c5bd0 con 0x7fc87c158b60 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c14f360 0x7fc87c0c0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c14f360 0x7fc87c0c0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35402/0 (socket says 172.21.15.94:35402) 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 -- 172.21.15.94:0/1054495463 learned_addr learned my addr 172.21.15.94:0/1054495463 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87b59e640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc87c158b60 0x7fc87c0c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87ad9d640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc87c0c88c0 0x7fc87c157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 -- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc87c158b60 msgr2=0x7fc87c0c0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc87c158b60 0x7fc87c0c0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 -- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc87c0c88c0 msgr2=0x7fc87c157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc87c0c88c0 0x7fc87c157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.402+0000 7fc87a59c640 1 -- 172.21.15.94:0/1054495463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c1eb250 con 0x7fc87c14f360 2024-03-06T07:21:53.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.403+0000 7fc87a59c640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c14f360 0x7fc87c0c0100 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc8700098c0 tx=0x7fc8700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.403+0000 7fc8637fe640 1 -- 172.21.15.94:0/1054495463 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870042070 con 0x7fc87c14f360 2024-03-06T07:21:53.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.403+0000 7fc8637fe640 1 -- 172.21.15.94:0/1054495463 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc870032a40 con 0x7fc87c14f360 2024-03-06T07:21:53.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.403+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c14b570 con 0x7fc87c14f360 2024-03-06T07:21:53.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.403+0000 7fc8637fe640 1 -- 172.21.15.94:0/1054495463 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870037040 con 0x7fc87c14f360 2024-03-06T07:21:53.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.403+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c0be870 con 0x7fc87c14f360 2024-03-06T07:21:53.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.404+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c04f330 con 0x7fc87c14f360 2024-03-06T07:21:53.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.404+0000 7fc8637fe640 1 -- 172.21.15.94:0/1054495463 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc870045070 con 0x7fc87c14f360 2024-03-06T07:21:53.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.405+0000 7fc8637fe640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc86806b4d0 0x7fc86806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.405+0000 7fc8637fe640 1 -- 172.21.15.94:0/1054495463 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc8700ac0f0 con 0x7fc87c14f360 2024-03-06T07:21:53.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.405+0000 7fc87ad9d640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc86806b4d0 0x7fc86806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.405+0000 7fc87ad9d640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc86806b4d0 0x7fc86806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0014a0 tx=0x7fc86c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.409+0000 7fc8637fe640 1 -- 172.21.15.94:0/1054495463 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc87007a8f0 con 0x7fc87c14f360 2024-03-06T07:21:53.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.614+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc87c0c8d50 con 0x7fc87c14f360 2024-03-06T07:21:53.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.615+0000 7fc8637fe640 1 -- 172.21.15.94:0/1054495463 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fc87007e990 con 0x7fc87c14f360 2024-03-06T07:21:53.617 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:53.617 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":36014,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1145122167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1145122167},{"type":"v1","addr":"172.21.15.145:6836","nonce":1145122167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:49.806589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6658":{"gid":6658,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6658,"qdb_cluster":[6658]},"id":7}]} 2024-03-06T07:21:53.617 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 82 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.616+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc86806b4d0 msgr2=0x7fc86806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.616+0000 7fc881193640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc86806b4d0 0x7fc86806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0014a0 tx=0x7fc86c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c14f360 msgr2=0x7fc87c0c0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c14f360 0x7fc87c0c0100 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc8700098c0 tx=0x7fc8700046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 shutdown_connections 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc86806b4d0 0x7fc86806d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc87c158b60 0x7fc87c0c0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc87c14f360 0x7fc87c0c0100 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 --2- 172.21.15.94:0/1054495463 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc87c0c88c0 0x7fc87c157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 >> 172.21.15.94:0/1054495463 conn(0x7fc87c0ac570 msgr2=0x7fc87c14bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 shutdown_connections 2024-03-06T07:21:53.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.617+0000 7fc881193640 1 -- 172.21.15.94:0/1054495463 wait complete. 2024-03-06T07:21:53.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.728+0000 7f9303240640 1 Processor -- start 2024-03-06T07:21:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.728+0000 7f9303240640 1 -- start start 2024-03-06T07:21:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f9303240640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f92fc1550c0 0x7f92fc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f9303240640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc1559e0 0x7f92fc14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f9303240640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f92fc0aa180 0x7f92fc14f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f9303240640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92fc0c4610 con 0x7f92fc0aa180 2024-03-06T07:21:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f9303240640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92fc0aa660 con 0x7f92fc1550c0 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f9303240640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0aa940 con 0x7f92fc1559e0 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f93017dd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f92fc0aa180 0x7f92fc14f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f93017dd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f92fc0aa180 0x7f92fc14f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35408/0 (socket says 172.21.15.94:35408) 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f93017dd640 1 -- 172.21.15.94:0/1153128835 learned_addr learned my addr 172.21.15.94:0/1153128835 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f9300fdc640 1 --2- 172.21.15.94:0/1153128835 >> v2:172.21.15.145:3300/0 conn(0x7f92fc1550c0 0x7f92fc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f92fbfff640 1 --2- 172.21.15.94:0/1153128835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc1559e0 0x7f92fc14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f93017dd640 1 -- 172.21.15.94:0/1153128835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc1559e0 msgr2=0x7f92fc14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f93017dd640 1 --2- 172.21.15.94:0/1153128835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc1559e0 0x7f92fc14ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f93017dd640 1 -- 172.21.15.94:0/1153128835 >> v2:172.21.15.145:3300/0 conn(0x7f92fc1550c0 msgr2=0x7f92fc1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.729+0000 7f93017dd640 1 --2- 172.21.15.94:0/1153128835 >> v2:172.21.15.145:3300/0 conn(0x7f92fc1550c0 0x7f92fc1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.730+0000 7f93017dd640 1 -- 172.21.15.94:0/1153128835 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc0c4860 con 0x7f92fc0aa180 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.730+0000 7f93017dd640 1 --2- 172.21.15.94:0/1153128835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc0aa180 0x7f92fc14f140 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f92fc0c3fc0 tx=0x7f92f402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1612ce738115202e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.730+0000 7f92fb7fe640 1 -- 172.21.15.94:0/1153128835 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f402c950 con 0x7f92fc0aa180 2024-03-06T07:21:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.730+0000 7f92fb7fe640 1 -- 172.21.15.94:0/1153128835 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f402caf0 con 0x7f92fc0aa180 2024-03-06T07:21:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.730+0000 7f92fb7fe640 1 -- 172.21.15.94:0/1153128835 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f40343f0 con 0x7f92fc0aa180 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 -- 172.21.15.94:0/1153128835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc0aa180 msgr2=0x7f92fc14f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 --2- 172.21.15.94:0/1153128835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc0aa180 0x7f92fc14f140 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f92fc0c3fc0 tx=0x7f92f402bdb0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 -- 172.21.15.94:0/1153128835 shutdown_connections 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 --2- 172.21.15.94:0/1153128835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc0aa180 0x7f92fc14f140 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 --2- 172.21.15.94:0/1153128835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc1559e0 0x7f92fc14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 --2- 172.21.15.94:0/1153128835 >> v2:172.21.15.145:3300/0 conn(0x7f92fc1550c0 0x7f92fc1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 -- 172.21.15.94:0/1153128835 >> 172.21.15.94:0/1153128835 conn(0x7f92fc0c6710 msgr2=0x7f92fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 -- 172.21.15.94:0/1153128835 shutdown_connections 2024-03-06T07:21:53.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.731+0000 7f9303240640 1 -- 172.21.15.94:0/1153128835 wait complete. 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.732+0000 7f9303240640 1 Processor -- start 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.732+0000 7f9303240640 1 -- start start 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f9303240640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc0aa180 0x7f92fc14b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f9303240640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc1550c0 0x7f92fc06c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f9303240640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92fc1559e0 0x7f92fc072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f9303240640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0c3210 con 0x7f92fc1550c0 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f9303240640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0c1520 con 0x7f92fc1559e0 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f9303240640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0c3390 con 0x7f92fc0aa180 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc1550c0 0x7f92fc06c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc1550c0 0x7f92fc06c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35424/0 (socket says 172.21.15.94:35424) 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 -- 172.21.15.94:0/2318614739 learned_addr learned my addr 172.21.15.94:0/2318614739 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f93017dd640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92fc1559e0 0x7f92fc072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f9300fdc640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc0aa180 0x7f92fc14b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 -- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc0aa180 msgr2=0x7f92fc14b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc0aa180 0x7f92fc14b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 -- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92fc1559e0 msgr2=0x7f92fc072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92fc1559e0 0x7f92fc072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 -- 172.21.15.94:0/2318614739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc072f20 con 0x7f92fc1550c0 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.733+0000 7f92fbfff640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc1550c0 0x7f92fc06c480 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f92f000e8a0 tx=0x7f92f000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.734+0000 7f92f97fa640 1 -- 172.21.15.94:0/2318614739 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0017070 con 0x7f92fc1550c0 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.734+0000 7f92f97fa640 1 -- 172.21.15.94:0/2318614739 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f000c940 con 0x7f92fc1550c0 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.734+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92fc073200 con 0x7f92fc1550c0 2024-03-06T07:21:53.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.734+0000 7f92f97fa640 1 -- 172.21.15.94:0/2318614739 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f000cd30 con 0x7f92fc1550c0 2024-03-06T07:21:53.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.734+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92fc1e8b10 con 0x7f92fc1550c0 2024-03-06T07:21:53.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.735+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92fc069460 con 0x7f92fc1550c0 2024-03-06T07:21:53.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.735+0000 7f92f97fa640 1 -- 172.21.15.94:0/2318614739 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92f001a070 con 0x7f92fc1550c0 2024-03-06T07:21:53.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.735+0000 7f92f97fa640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92e806b500 0x7f92e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.736+0000 7f92f97fa640 1 -- 172.21.15.94:0/2318614739 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f92f008a4d0 con 0x7f92fc1550c0 2024-03-06T07:21:53.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.736+0000 7f9300fdc640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92e806b500 0x7f92e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.736+0000 7f9300fdc640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92e806b500 0x7f92e806d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f92fc06e970 tx=0x7f92ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.740+0000 7f92f97fa640 1 -- 172.21.15.94:0/2318614739 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92fc069460 con 0x7f92fc1550c0 2024-03-06T07:21:53.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.913+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92fc1e8640 con 0x7f92fc1550c0 2024-03-06T07:21:53.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.914+0000 7f92f97fa640 1 -- 172.21.15.94:0/2318614739 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+17004 (secure 0 0 0) 0x7f92f0058bb0 con 0x7f92fc1550c0 2024-03-06T07:21:53.916 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:53.916 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:46.796076+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:37.116515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:37.869999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/3774173957":"2024-03-07T07:21:46.796043+0000","172.21.15.145:6834/3774173957":"2024-03-07T07:21:46.796043+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92e806b500 msgr2=0x7f92e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92e806b500 0x7f92e806d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f92fc06e970 tx=0x7f92ec009190 comp rx=0 tx=0).stop 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc1550c0 msgr2=0x7f92fc06c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc1550c0 0x7f92fc06c480 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f92f000e8a0 tx=0x7f92f000ed60 comp rx=0 tx=0).stop 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 shutdown_connections 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92e806b500 0x7f92e806d9c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92fc1559e0 0x7f92fc072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc1550c0 0x7f92fc06c480 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 --2- 172.21.15.94:0/2318614739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92fc0aa180 0x7f92fc14b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 >> 172.21.15.94:0/2318614739 conn(0x7f92fc0c6710 msgr2=0x7f92fc153780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 shutdown_connections 2024-03-06T07:21:53.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:53.916+0000 7f9303240640 1 -- 172.21.15.94:0/2318614739 wait complete. 2024-03-06T07:21:53.929 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:21:53.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:54.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.027+0000 7fd530383640 1 Processor -- start 2024-03-06T07:21:54.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.027+0000 7fd530383640 1 -- start start 2024-03-06T07:21:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd530383640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd5281554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd530383640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd5280aa180 0x7fd5280c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd530383640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5280aa660 con 0x7fd5281559e0 2024-03-06T07:21:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd530383640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5280c4610 con 0x7fd5280aa180 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd530383640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5280aa940 con 0x7fd5281550c0 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd52e920640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd5280aa180 0x7fd5280c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd52d91e640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd5281559e0 0x7fd52814ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37632/0 (socket says 172.21.15.94:37632) 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd52d91e640 1 -- 172.21.15.94:0/4015651390 learned_addr learned my addr 172.21.15.94:0/4015651390 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.028+0000 7fd52e11f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd5281554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52d11d640 1 -- 172.21.15.94:0/4015651390 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042653740 0 0) 0x7fd5280aa660 con 0x7fd5281559e0 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52d11d640 1 -- 172.21.15.94:0/4015651390 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd514003520 con 0x7fd5281559e0 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52e920640 1 -- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 msgr2=0x7fd5281554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52e920640 1 --2- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd5281554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52e920640 1 -- 172.21.15.94:0/4015651390 >> v1:172.21.15.145:6789/0 conn(0x7fd5281559e0 legacy=0x7fd52814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52e920640 1 -- 172.21.15.94:0/4015651390 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5280c4860 con 0x7fd5280aa180 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52e11f640 1 --2- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd5281554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.029+0000 7fd52e920640 1 --2- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 0x7fd5280c5060 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd5200097a0 tx=0x7fd52002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0bb6e743fb62ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd52d11d640 1 -- 172.21.15.94:0/4015651390 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520038070 con 0x7fd5280aa180 2024-03-06T07:21:54.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd52d11d640 1 -- 172.21.15.94:0/4015651390 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd520031700 con 0x7fd5280aa180 2024-03-06T07:21:54.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd52d11d640 1 -- 172.21.15.94:0/4015651390 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520031ad0 con 0x7fd5280aa180 2024-03-06T07:21:54.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 -- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 msgr2=0x7fd5280c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 --2- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 0x7fd5280c5060 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd5200097a0 tx=0x7fd52002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 -- 172.21.15.94:0/4015651390 shutdown_connections 2024-03-06T07:21:54.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 --2- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 0x7fd5280c5060 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 --2- 172.21.15.94:0/4015651390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd5281554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 -- 172.21.15.94:0/4015651390 >> 172.21.15.94:0/4015651390 conn(0x7fd5280c6710 msgr2=0x7fd5280c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 -- 172.21.15.94:0/4015651390 shutdown_connections 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.030+0000 7fd530383640 1 -- 172.21.15.94:0/4015651390 wait complete. 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.031+0000 7fd530383640 1 Processor -- start 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.031+0000 7fd530383640 1 -- start start 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd530383640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 0x7fd5280a7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd530383640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd528149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd530383640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5281559e0 0x7fd5281e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd530383640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52814f310 con 0x7fd5281559e0 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd530383640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5280c1520 con 0x7fd5280aa180 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd530383640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52814f490 con 0x7fd5281550c0 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5281559e0 0x7fd5281e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5281559e0 0x7fd5281e9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35440/0 (socket says 172.21.15.94:35440) 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 -- 172.21.15.94:0/2784468452 learned_addr learned my addr 172.21.15.94:0/2784468452 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52d91e640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd528149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e11f640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 0x7fd5280a7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 -- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 msgr2=0x7fd528149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd528149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 -- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 msgr2=0x7fd5280a7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 0x7fd5280a7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.032+0000 7fd52e920640 1 -- 172.21.15.94:0/2784468452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5280a6ee0 con 0x7fd5281559e0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.033+0000 7fd52e920640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5281559e0 0x7fd5281e9c10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd520036040 tx=0x7fd5200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.033+0000 7fd51effd640 1 -- 172.21.15.94:0/2784468452 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520042070 con 0x7fd5281559e0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.033+0000 7fd51effd640 1 -- 172.21.15.94:0/2784468452 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52003d890 con 0x7fd5281559e0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.033+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52806e500 con 0x7fd5281559e0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.033+0000 7fd51effd640 1 -- 172.21.15.94:0/2784468452 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520038040 con 0x7fd5281559e0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.033+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52806e7e0 con 0x7fd5281559e0 2024-03-06T07:21:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.034+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd52804f330 con 0x7fd5281559e0 2024-03-06T07:21:54.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.034+0000 7fd51effd640 1 -- 172.21.15.94:0/2784468452 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd520045070 con 0x7fd5281559e0 2024-03-06T07:21:54.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.035+0000 7fd51effd640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50c06b500 0x7fd50c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.035+0000 7fd51effd640 1 -- 172.21.15.94:0/2784468452 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd5200abc70 con 0x7fd5281559e0 2024-03-06T07:21:54.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.035+0000 7fd52e11f640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50c06b500 0x7fd50c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.035+0000 7fd52e11f640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50c06b500 0x7fd50c06d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd524009bd0 tx=0x7fd52400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.039+0000 7fd51effd640 1 -- 172.21.15.94:0/2784468452 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd520030070 con 0x7fd5281559e0 2024-03-06T07:21:54.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.245+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd528068450 con 0x7fd5281559e0 2024-03-06T07:21:54.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.246+0000 7fd51effd640 1 -- 172.21.15.94:0/2784468452 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4984 (secure 0 0 0) 0x7fd52007a2c0 con 0x7fd5281559e0 2024-03-06T07:21:54.248 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:54.248 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":36014,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1145122167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1145122167},{"type":"v1","addr":"172.21.15.145:6836","nonce":1145122167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:49.806589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6658":{"gid":6658,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6658,"qdb_cluster":[6658]},"id":7}]} 2024-03-06T07:21:54.248 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 82 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50c06b500 msgr2=0x7fd50c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50c06b500 0x7fd50c06d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd524009bd0 tx=0x7fd52400b040 comp rx=0 tx=0).stop 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5281559e0 msgr2=0x7fd5281e9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5281559e0 0x7fd5281e9c10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd520036040 tx=0x7fd5200046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 shutdown_connections 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50c06b500 0x7fd50c06d9c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5281559e0 0x7fd5281e9c10 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5281550c0 0x7fd528149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 --2- 172.21.15.94:0/2784468452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5280aa180 0x7fd5280a7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 >> 172.21.15.94:0/2784468452 conn(0x7fd5280c6710 msgr2=0x7fd5280c16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 shutdown_connections 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.248+0000 7fd530383640 1 -- 172.21.15.94:0/2784468452 wait complete. 2024-03-06T07:21:54.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:54.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.358+0000 7f444069a640 1 Processor -- start 2024-03-06T07:21:54.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.358+0000 7f444069a640 1 -- start start 2024-03-06T07:21:54.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f444069a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f443814f360 0x7f4438155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f444069a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f44380c88c0 0x7f44380c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f444069a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44380c73d0 con 0x7f44380c5880 2024-03-06T07:21:54.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f444069a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44380c8dd0 con 0x7f44380c88c0 2024-03-06T07:21:54.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f444069a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44380be2e0 con 0x7f443814f360 2024-03-06T07:21:54.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f443dc35640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f44380c88c0 0x7f44380c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f443ec37640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f44380c5880 0x7f4438158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36538/0 (socket says 172.21.15.94:36538) 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f443ec37640 1 -- 172.21.15.94:0/2309481922 learned_addr learned my addr 172.21.15.94:0/2309481922 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.359+0000 7f443e436640 1 --2- 172.21.15.94:0/2309481922 >> v2:172.21.15.145:3300/0 conn(0x7f443814f360 0x7f4438155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443dc35640 1 -- 172.21.15.94:0/2309481922 >> v2:172.21.15.145:3300/0 conn(0x7f443814f360 msgr2=0x7f4438155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443dc35640 1 --2- 172.21.15.94:0/2309481922 >> v2:172.21.15.145:3300/0 conn(0x7f443814f360 0x7f4438155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443dc35640 1 -- 172.21.15.94:0/2309481922 >> v1:172.21.15.94:6789/0 conn(0x7f44380c5880 legacy=0x7f4438158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443dc35640 1 -- 172.21.15.94:0/2309481922 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44380c7670 con 0x7f44380c88c0 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443d434640 1 -- 172.21.15.94:0/2309481922 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682094257 0 0) 0x7f44380c73d0 con 0x7f44380c5880 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443dc35640 1 --2- 172.21.15.94:0/2309481922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44380c88c0 0x7f44380c5340 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f443808c840 tx=0x7f442802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf21065045ca69b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443d434640 1 -- 172.21.15.94:0/2309481922 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44280315a0 con 0x7f44380c88c0 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443d434640 1 -- 172.21.15.94:0/2309481922 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4428037070 con 0x7f44380c88c0 2024-03-06T07:21:54.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.360+0000 7f443d434640 1 -- 172.21.15.94:0/2309481922 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428031960 con 0x7f44380c88c0 2024-03-06T07:21:54.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 -- 172.21.15.94:0/2309481922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44380c88c0 msgr2=0x7f44380c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 --2- 172.21.15.94:0/2309481922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44380c88c0 0x7f44380c5340 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f443808c840 tx=0x7f442802b840 comp rx=0 tx=0).stop 2024-03-06T07:21:54.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 -- 172.21.15.94:0/2309481922 shutdown_connections 2024-03-06T07:21:54.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 --2- 172.21.15.94:0/2309481922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44380c88c0 0x7f44380c5340 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 --2- 172.21.15.94:0/2309481922 >> v2:172.21.15.145:3300/0 conn(0x7f443814f360 0x7f4438155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 -- 172.21.15.94:0/2309481922 >> 172.21.15.94:0/2309481922 conn(0x7f44380ac570 msgr2=0x7f44380ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 -- 172.21.15.94:0/2309481922 shutdown_connections 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.361+0000 7f444069a640 1 -- 172.21.15.94:0/2309481922 wait complete. 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.362+0000 7f444069a640 1 Processor -- start 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.362+0000 7f444069a640 1 -- start start 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f444069a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44380c5880 0x7f44380c0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f444069a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44380c88c0 0x7f44380af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f444069a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f443814f360 0x7f44380af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f444069a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4438159270 con 0x7f443814f360 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f444069a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44380c7c30 con 0x7f44380c5880 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f444069a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44381593f0 con 0x7f44380c88c0 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f443814f360 0x7f44380af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f443814f360 0x7f44380af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35460/0 (socket says 172.21.15.94:35460) 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 -- 172.21.15.94:0/3104785308 learned_addr learned my addr 172.21.15.94:0/3104785308 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443e436640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44380c5880 0x7f44380c0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443dc35640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44380c88c0 0x7f44380af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 -- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44380c88c0 msgr2=0x7f44380af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44380c88c0 0x7f44380af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 -- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44380c5880 msgr2=0x7f44380c0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44380c5880 0x7f44380c0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 -- 172.21.15.94:0/3104785308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44380adcf0 con 0x7f443814f360 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443dc35640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44380c88c0 0x7f44380af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f443ec37640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f443814f360 0x7f44380af660 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f443000c480 tx=0x7f443000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.363+0000 7f442effd640 1 -- 172.21.15.94:0/3104785308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430018070 con 0x7f443814f360 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.364+0000 7f442effd640 1 -- 172.21.15.94:0/3104785308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4430007500 con 0x7f443814f360 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.364+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44380ada10 con 0x7f443814f360 2024-03-06T07:21:54.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.364+0000 7f442effd640 1 -- 172.21.15.94:0/3104785308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4430004a20 con 0x7f443814f360 2024-03-06T07:21:54.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.364+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44381eb580 con 0x7f443814f360 2024-03-06T07:21:54.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.365+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4438069460 con 0x7f443814f360 2024-03-06T07:21:54.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.365+0000 7f442effd640 1 -- 172.21.15.94:0/3104785308 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f443001b070 con 0x7f443814f360 2024-03-06T07:21:54.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.365+0000 7f442effd640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f441c06b500 0x7f441c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.365+0000 7f442effd640 1 -- 172.21.15.94:0/3104785308 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f44300895b0 con 0x7f443814f360 2024-03-06T07:21:54.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.365+0000 7f443e436640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f441c06b500 0x7f441c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.366+0000 7f443e436640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f441c06b500 0x7f441c06d9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4434011340 tx=0x7f443400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.370+0000 7f442effd640 1 -- 172.21.15.94:0/3104785308 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4438069460 con 0x7f443814f360 2024-03-06T07:21:54.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.580+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f443806cda0 con 0x7f443814f360 2024-03-06T07:21:54.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.581+0000 7f442effd640 1 -- 172.21.15.94:0/3104785308 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4984 (secure 0 0 0) 0x7f4430057c90 con 0x7f443814f360 2024-03-06T07:21:54.583 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:54.583 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":36014,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1145122167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1145122167},{"type":"v1","addr":"172.21.15.145:6836","nonce":1145122167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:38.904104+0000","modified":"2024-03-06T07:21:49.806589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6658},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6658":{"gid":6658,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6837/744582506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":744582506},{"type":"v1","addr":"172.21.15.145:6837","nonce":744582506}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6658,"qdb_cluster":[6658]},"id":7}]} 2024-03-06T07:21:54.584 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 82 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f441c06b500 msgr2=0x7f441c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f441c06b500 0x7f441c06d9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4434011340 tx=0x7f443400b040 comp rx=0 tx=0).stop 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f443814f360 msgr2=0x7f44380af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f443814f360 0x7f44380af660 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f443000c480 tx=0x7f443000c940 comp rx=0 tx=0).stop 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 shutdown_connections 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f441c06b500 0x7f441c06d9c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f443814f360 0x7f44380af660 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44380c88c0 0x7f44380af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 --2- 172.21.15.94:0/3104785308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44380c5880 0x7f44380c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 >> 172.21.15.94:0/3104785308 conn(0x7f44380ac570 msgr2=0x7f44380c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 shutdown_connections 2024-03-06T07:21:54.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.583+0000 7f444069a640 1 -- 172.21.15.94:0/3104785308 wait complete. 2024-03-06T07:21:54.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:54.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.693+0000 7f6262081640 1 Processor -- start 2024-03-06T07:21:54.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.693+0000 7f6262081640 1 -- start start 2024-03-06T07:21:54.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f6262081640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f625c14f360 0x7f625c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f6262081640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f6262081640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c0c73d0 con 0x7f625c0c88c0 2024-03-06T07:21:54.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f6262081640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625c0c8dd0 con 0x7f625c14f360 2024-03-06T07:21:54.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f6262081640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0be2e0 con 0x7f625c158b60 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f625affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f625c0c88c0 0x7f625c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36540/0 (socket says 172.21.15.94:36540) 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f625b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f625c14f360 0x7f625c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f625affd640 1 -- 172.21.15.94:0/3316885575 learned_addr learned my addr 172.21.15.94:0/3316885575 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.694+0000 7f625bfff640 1 --2- 172.21.15.94:0/3316885575 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.695+0000 7f625b7fe640 1 -- 172.21.15.94:0/3316885575 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 msgr2=0x7f625c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.695+0000 7f625b7fe640 1 --2- 172.21.15.94:0/3316885575 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.695+0000 7f625b7fe640 1 -- 172.21.15.94:0/3316885575 >> v1:172.21.15.94:6789/0 conn(0x7f625c0c88c0 legacy=0x7f625c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.695+0000 7f625b7fe640 1 -- 172.21.15.94:0/3316885575 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c0c7670 con 0x7f625c14f360 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.695+0000 7f625a7fc640 1 -- 172.21.15.94:0/3316885575 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917534160 0 0) 0x7f625c0c73d0 con 0x7f625c0c88c0 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.695+0000 7f625b7fe640 1 --2- 172.21.15.94:0/3316885575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c14f360 0x7f625c155c10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f624c009d90 tx=0x7f624c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5f0605c79033e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.695+0000 7f625a7fc640 1 -- 172.21.15.94:0/3316885575 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c039070 con 0x7f625c14f360 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f625a7fc640 1 -- 172.21.15.94:0/3316885575 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c02cbc0 con 0x7f625c14f360 2024-03-06T07:21:54.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f625a7fc640 1 -- 172.21.15.94:0/3316885575 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c034440 con 0x7f625c14f360 2024-03-06T07:21:54.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 -- 172.21.15.94:0/3316885575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c14f360 msgr2=0x7f625c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 --2- 172.21.15.94:0/3316885575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c14f360 0x7f625c155c10 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f624c009d90 tx=0x7f624c02be10 comp rx=0 tx=0).stop 2024-03-06T07:21:54.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 -- 172.21.15.94:0/3316885575 shutdown_connections 2024-03-06T07:21:54.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 --2- 172.21.15.94:0/3316885575 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 --2- 172.21.15.94:0/3316885575 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c14f360 0x7f625c155c10 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 -- 172.21.15.94:0/3316885575 >> 172.21.15.94:0/3316885575 conn(0x7f625c0ac570 msgr2=0x7f625c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 -- 172.21.15.94:0/3316885575 shutdown_connections 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.696+0000 7f6262081640 1 -- 172.21.15.94:0/3316885575 wait complete. 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.697+0000 7f6262081640 1 Processor -- start 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.697+0000 7f6262081640 1 -- start start 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f6262081640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c0c88c0 0x7f625c1eb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f6262081640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625c14f360 0x7f625c1ee320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f6262081640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f6262081640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0c5a50 con 0x7f625c0c88c0 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f6262081640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0c7c30 con 0x7f625c14f360 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f6262081640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0c5bd0 con 0x7f625c158b60 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c0c88c0 0x7f625c1eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c0c88c0 0x7f625c1eb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35490/0 (socket says 172.21.15.94:35490) 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 -- 172.21.15.94:0/532260735 learned_addr learned my addr 172.21.15.94:0/532260735 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625affd640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625c14f360 0x7f625c1ee320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625bfff640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 -- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 msgr2=0x7f625c154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 -- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625c14f360 msgr2=0x7f625c1ee320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625c14f360 0x7f625c1ee320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.698+0000 7f625b7fe640 1 -- 172.21.15.94:0/532260735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c1ee860 con 0x7f625c0c88c0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.699+0000 7f625b7fe640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c0c88c0 0x7f625c1eb810 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f624c009d90 tx=0x7f624c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.699+0000 7f6243fff640 1 -- 172.21.15.94:0/532260735 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c039050 con 0x7f625c0c88c0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.699+0000 7f6243fff640 1 -- 172.21.15.94:0/532260735 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c03a040 con 0x7f625c0c88c0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.699+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c158760 con 0x7f625c0c88c0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.699+0000 7f6243fff640 1 -- 172.21.15.94:0/532260735 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c03e610 con 0x7f625c0c88c0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.699+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625c158480 con 0x7f625c0c88c0 2024-03-06T07:21:54.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.700+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c04f330 con 0x7f625c0c88c0 2024-03-06T07:21:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.700+0000 7f6243fff640 1 -- 172.21.15.94:0/532260735 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f624c047020 con 0x7f625c0c88c0 2024-03-06T07:21:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.701+0000 7f6243fff640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f624806b4d0 0x7f624806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.701+0000 7f6243fff640 1 -- 172.21.15.94:0/532260735 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f624c0ac550 con 0x7f625c0c88c0 2024-03-06T07:21:54.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.701+0000 7f625affd640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f624806b4d0 0x7f624806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.701+0000 7f625affd640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f624806b4d0 0x7f624806d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f625c1552d0 tx=0x7f6250007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.705+0000 7f6243fff640 1 -- 172.21.15.94:0/532260735 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f624c04e020 con 0x7f625c0c88c0 2024-03-06T07:21:54.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.879+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625c06cda0 con 0x7f625c0c88c0 2024-03-06T07:21:54.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.880+0000 7f6243fff640 1 -- 172.21.15.94:0/532260735 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+17004 (secure 0 0 0) 0x7f624c07ac30 con 0x7f625c0c88c0 2024-03-06T07:21:54.882 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:54.883 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:46.796076+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:37.116515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:37.869999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/3774173957":"2024-03-07T07:21:46.796043+0000","172.21.15.145:6834/3774173957":"2024-03-07T07:21:46.796043+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f624806b4d0 msgr2=0x7f624806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f624806b4d0 0x7f624806d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f625c1552d0 tx=0x7f6250007de0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c0c88c0 msgr2=0x7f625c1eb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c0c88c0 0x7f625c1eb810 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f624c009d90 tx=0x7f624c004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 shutdown_connections 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f624806b4d0 0x7f624806d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f625c158b60 0x7f625c154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625c14f360 0x7f625c1ee320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 --2- 172.21.15.94:0/532260735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f625c0c88c0 0x7f625c1eb810 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.882+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 >> 172.21.15.94:0/532260735 conn(0x7f625c0ac570 msgr2=0x7f625c0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.883+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 shutdown_connections 2024-03-06T07:21:54.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.883+0000 7f6262081640 1 -- 172.21.15.94:0/532260735 wait complete. 2024-03-06T07:21:54.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:21:54.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.995+0000 7f3c2df5c640 1 Processor -- start 2024-03-06T07:21:54.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.995+0000 7f3c2df5c640 1 -- start start 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c2df5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c28152720 0x7f3c281593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c2df5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3c2806f550 0x7f3c280c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c2df5c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c280733f0 con 0x7f3c280c7d20 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c2df5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c28155ca0 con 0x7f3c2806f550 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c2df5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c281561d0 con 0x7f3c28152720 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c27fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3c280c7d20 0x7f3c2806fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36554/0 (socket says 172.21.15.94:36554) 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c27fff640 1 -- 172.21.15.94:0/4260236632 learned_addr learned my addr 172.21.15.94:0/4260236632 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c26ffd640 1 --2- 172.21.15.94:0/4260236632 >> v2:172.21.15.145:3300/0 conn(0x7f3c2806f550 0x7f3c280c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c277fe640 1 --2- 172.21.15.94:0/4260236632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c28152720 0x7f3c281593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c267fc640 1 -- 172.21.15.94:0/4260236632 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130782361 0 0) 0x7f3c280733f0 con 0x7f3c280c7d20 2024-03-06T07:21:55.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c267fc640 1 -- 172.21.15.94:0/4260236632 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c14003520 con 0x7f3c280c7d20 2024-03-06T07:21:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c277fe640 1 -- 172.21.15.94:0/4260236632 >> v2:172.21.15.145:3300/0 conn(0x7f3c2806f550 msgr2=0x7f3c280c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c277fe640 1 --2- 172.21.15.94:0/4260236632 >> v2:172.21.15.145:3300/0 conn(0x7f3c2806f550 0x7f3c280c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c277fe640 1 -- 172.21.15.94:0/4260236632 >> v1:172.21.15.94:6789/0 conn(0x7f3c280c7d20 legacy=0x7f3c2806fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c277fe640 1 -- 172.21.15.94:0/4260236632 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28155ef0 con 0x7f3c28152720 2024-03-06T07:21:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.996+0000 7f3c26ffd640 1 --2- 172.21.15.94:0/4260236632 >> v2:172.21.15.145:3300/0 conn(0x7f3c2806f550 0x7f3c280c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:55.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.997+0000 7f3c277fe640 1 --2- 172.21.15.94:0/4260236632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c28152720 0x7f3c281593c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c180097a0 tx=0x7f3c1802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed464c9a8f88f9ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.997+0000 7f3c267fc640 1 -- 172.21.15.94:0/4260236632 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c180315a0 con 0x7f3c28152720 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.997+0000 7f3c267fc640 1 -- 172.21.15.94:0/4260236632 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c18037070 con 0x7f3c28152720 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.997+0000 7f3c267fc640 1 -- 172.21.15.94:0/4260236632 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18031960 con 0x7f3c28152720 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4260236632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c28152720 msgr2=0x7f3c281593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4260236632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c28152720 0x7f3c281593c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c180097a0 tx=0x7f3c1802b840 comp rx=0 tx=0).stop 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4260236632 shutdown_connections 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4260236632 >> v2:172.21.15.145:3300/0 conn(0x7f3c2806f550 0x7f3c280c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4260236632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c28152720 0x7f3c281593c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4260236632 >> 172.21.15.94:0/4260236632 conn(0x7f3c2806ea40 msgr2=0x7f3c2806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4260236632 shutdown_connections 2024-03-06T07:21:55.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.998+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4260236632 wait complete. 2024-03-06T07:21:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.999+0000 7f3c2df5c640 1 Processor -- start 2024-03-06T07:21:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.999+0000 7f3c2df5c640 1 -- start start 2024-03-06T07:21:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.999+0000 7f3c2df5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c2806f550 0x7f3c28154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:54.999+0000 7f3c2df5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c280c7d20 0x7f3c28148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c2df5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c28152720 0x7f3c281eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c2df5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28158ea0 con 0x7f3c28152720 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c2df5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28158bc0 con 0x7f3c2806f550 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c2df5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280704a0 con 0x7f3c280c7d20 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c28152720 0x7f3c281eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c28152720 0x7f3c281eb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35496/0 (socket says 172.21.15.94:35496) 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 -- 172.21.15.94:0/4138248051 learned_addr learned my addr 172.21.15.94:0/4138248051 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c26ffd640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c280c7d20 0x7f3c28148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c277fe640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c2806f550 0x7f3c28154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 -- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c280c7d20 msgr2=0x7f3c28148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c280c7d20 0x7f3c28148f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 -- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c2806f550 msgr2=0x7f3c28154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c2806f550 0x7f3c28154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 -- 172.21.15.94:0/4138248051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28149480 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c27fff640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c28152720 0x7f3c281eb8b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000caa0 tx=0x7f3c2000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c1ffff640 1 -- 172.21.15.94:0/4138248051 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20017070 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c1ffff640 1 -- 172.21.15.94:0/4138248051 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2000ac00 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c280c3390 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c1ffff640 1 -- 172.21.15.94:0/4138248051 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20012400 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.000+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c28153c20 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.001+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c28153a50 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.002+0000 7f3c1ffff640 1 -- 172.21.15.94:0/4138248051 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c2001a070 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.002+0000 7f3c1ffff640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c0806b4d0 0x7f3c0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.002+0000 7f3c1ffff640 1 -- 172.21.15.94:0/4138248051 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3c20089f10 con 0x7f3c28152720 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.002+0000 7f3c277fe640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c0806b4d0 0x7f3c0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.003+0000 7f3c277fe640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c0806b4d0 0x7f3c0806d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3c18036000 tx=0x7f3c1803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.007+0000 7f3c1ffff640 1 -- 172.21.15.94:0/4138248051 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c200585f0 con 0x7f3c28152720 2024-03-06T07:21:55.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.212+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c2804f330 con 0x7f3c28152720 2024-03-06T07:21:55.214 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:21:55.213+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:21:55.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.214+0000 7f3c1ffff640 1 -- 172.21.15.94:0/4138248051 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3c2005c690 con 0x7f3c28152720 2024-03-06T07:21:55.216 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:21:55.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c0806b4d0 msgr2=0x7f3c0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c0806b4d0 0x7f3c0806d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3c18036000 tx=0x7f3c1803d040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c28152720 msgr2=0x7f3c281eb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c28152720 0x7f3c281eb8b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000caa0 tx=0x7f3c2000cf60 comp rx=0 tx=0).stop 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 shutdown_connections 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c0806b4d0 0x7f3c0806d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c28152720 0x7f3c281eb8b0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c280c7d20 0x7f3c28148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 --2- 172.21.15.94:0/4138248051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c2806f550 0x7f3c28154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 >> 172.21.15.94:0/4138248051 conn(0x7f3c2806ea40 msgr2=0x7f3c280a9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 shutdown_connections 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.216+0000 7f3c2df5c640 1 -- 172.21.15.94:0/4138248051 wait complete. 2024-03-06T07:21:55.224 INFO:tasks.ceph.mds.b.smithi145.stderr: -311> 2024-03-06T07:21:49.816+0000 7f628b2ca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:55.229 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:21:55.247 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:21:55.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe827f7640 1 Processor -- start 2024-03-06T07:21:55.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe827f7640 1 -- start start 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe827f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efe7c14ef70 0x7efe7c06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe827f7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe7c06cad0 0x7efe7c154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe827f7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c0c2790 con 0x7efe7c0c1820 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe827f7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c0c3090 con 0x7efe7c14ef70 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe827f7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c14f350 con 0x7efe7c06cad0 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe7bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efe7c14ef70 0x7efe7c06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.330+0000 7efe7b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efe7c0c1820 0x7efe7c0c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36566/0 (socket says 172.21.15.94:36566) 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe7b7fe640 1 -- 172.21.15.94:0/1209323952 learned_addr learned my addr 172.21.15.94:0/1209323952 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe80d94640 1 --2- 172.21.15.94:0/1209323952 >> v2:172.21.15.145:3300/0 conn(0x7efe7c06cad0 0x7efe7c154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe7bfff640 1 -- 172.21.15.94:0/1209323952 >> v2:172.21.15.145:3300/0 conn(0x7efe7c06cad0 msgr2=0x7efe7c154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe7bfff640 1 --2- 172.21.15.94:0/1209323952 >> v2:172.21.15.145:3300/0 conn(0x7efe7c06cad0 0x7efe7c154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe7bfff640 1 -- 172.21.15.94:0/1209323952 >> v1:172.21.15.94:6789/0 conn(0x7efe7c0c1820 legacy=0x7efe7c0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe7bfff640 1 -- 172.21.15.94:0/1209323952 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c06cf40 con 0x7efe7c14ef70 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe7affd640 1 -- 172.21.15.94:0/1209323952 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987611206 0 0) 0x7efe7c0c2790 con 0x7efe7c0c1820 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.331+0000 7efe80d94640 1 --2- 172.21.15.94:0/1209323952 >> v2:172.21.15.145:3300/0 conn(0x7efe7c06cad0 0x7efe7c154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe7bfff640 1 --2- 172.21.15.94:0/1209323952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c14ef70 0x7efe7c06c1f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7efe7c0c76b0 tx=0x7efe6c02cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c565a36e1219e36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe7affd640 1 -- 172.21.15.94:0/1209323952 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c03a070 con 0x7efe7c14ef70 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe7affd640 1 -- 172.21.15.94:0/1209323952 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c02dad0 con 0x7efe7c14ef70 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe7affd640 1 -- 172.21.15.94:0/1209323952 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c00b040 con 0x7efe7c14ef70 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 -- 172.21.15.94:0/1209323952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c14ef70 msgr2=0x7efe7c06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 --2- 172.21.15.94:0/1209323952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c14ef70 0x7efe7c06c1f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7efe7c0c76b0 tx=0x7efe6c02cd20 comp rx=0 tx=0).stop 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 -- 172.21.15.94:0/1209323952 shutdown_connections 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 --2- 172.21.15.94:0/1209323952 >> v2:172.21.15.145:3300/0 conn(0x7efe7c06cad0 0x7efe7c154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 --2- 172.21.15.94:0/1209323952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c14ef70 0x7efe7c06c1f0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 -- 172.21.15.94:0/1209323952 >> 172.21.15.94:0/1209323952 conn(0x7efe7c0c6710 msgr2=0x7efe7c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 -- 172.21.15.94:0/1209323952 shutdown_connections 2024-03-06T07:21:55.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.332+0000 7efe827f7640 1 -- 172.21.15.94:0/1209323952 wait complete. 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.333+0000 7efe827f7640 1 Processor -- start 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe827f7640 1 -- start start 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe827f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c06cad0 0x7efe7c0a8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe827f7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe7c0c1820 0x7efe7c14c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe827f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe7c14ef70 0x7efe7c0aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe827f7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c155750 con 0x7efe7c06cad0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe827f7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c0c24b0 con 0x7efe7c0c1820 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe827f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c155170 con 0x7efe7c14ef70 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c06cad0 0x7efe7c0a8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe7bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c06cad0 0x7efe7c0a8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35522/0 (socket says 172.21.15.94:35522) 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe7bfff640 1 -- 172.21.15.94:0/3984653685 learned_addr learned my addr 172.21.15.94:0/3984653685 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe7b7fe640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe7c0c1820 0x7efe7c14c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.334+0000 7efe80d94640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe7c14ef70 0x7efe7c0aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe7bfff640 1 -- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe7c14ef70 msgr2=0x7efe7c0aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe7bfff640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe7c14ef70 0x7efe7c0aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe7bfff640 1 -- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe7c0c1820 msgr2=0x7efe7c14c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe7bfff640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe7c0c1820 0x7efe7c14c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe7bfff640 1 -- 172.21.15.94:0/3984653685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c0a6b30 con 0x7efe7c06cad0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe7b7fe640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe7c0c1820 0x7efe7c14c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe7bfff640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c06cad0 0x7efe7c0a8bb0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7efe6c038040 tx=0x7efe6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe78ff9640 1 -- 172.21.15.94:0/3984653685 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c03a040 con 0x7efe7c06cad0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe78ff9640 1 -- 172.21.15.94:0/3984653685 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c004030 con 0x7efe7c06cad0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe78ff9640 1 -- 172.21.15.94:0/3984653685 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c004400 con 0x7efe7c06cad0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c1ea5f0 con 0x7efe7c06cad0 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.335+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c0a64e0 con 0x7efe7c06cad0 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.336+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c04f330 con 0x7efe7c06cad0 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.336+0000 7efe78ff9640 1 -- 172.21.15.94:0/3984653685 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe6c003610 con 0x7efe7c06cad0 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.337+0000 7efe78ff9640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe6806b4d0 0x7efe6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.337+0000 7efe78ff9640 1 -- 172.21.15.94:0/3984653685 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6273+0+0 (secure 0 0 0) 0x7efe6c0adf50 con 0x7efe7c06cad0 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.337+0000 7efe7b7fe640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe6806b4d0 0x7efe6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.337+0000 7efe7b7fe640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe6806b4d0 0x7efe6806d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efe7c1e42f0 tx=0x7efe70007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.341+0000 7efe78ff9640 1 -- 172.21.15.94:0/3984653685 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe7c06cf40 con 0x7efe7c06cad0 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:55.550+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efe7c0c1110 con 0x7efe7c06cad0 2024-03-06T07:21:56.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.215+0000 7efe78ff9640 1 -- 172.21.15.94:0/3984653685 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) v1 ==== 104+0+0 (secure 0 0 0) 0x7efe6c07c530 con 0x7efe7c06cad0 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.217+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe6806b4d0 msgr2=0x7efe6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.217+0000 7efe827f7640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe6806b4d0 0x7efe6806d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efe7c1e42f0 tx=0x7efe70007c10 comp rx=0 tx=0).stop 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c06cad0 msgr2=0x7efe7c0a8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c06cad0 0x7efe7c0a8bb0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7efe6c038040 tx=0x7efe6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 shutdown_connections 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe6806b4d0 0x7efe6806d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe7c14ef70 0x7efe7c0aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe7c0c1820 0x7efe7c14c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 --2- 172.21.15.94:0/3984653685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe7c06cad0 0x7efe7c0a8bb0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 >> 172.21.15.94:0/3984653685 conn(0x7efe7c0c6710 msgr2=0x7efe7c152a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 shutdown_connections 2024-03-06T07:21:56.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.218+0000 7efe827f7640 1 -- 172.21.15.94:0/3984653685 wait complete. 2024-03-06T07:21:56.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:21:56.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.336+0000 7f3960ff7640 1 Processor -- start 2024-03-06T07:21:56.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.336+0000 7f3960ff7640 1 -- start start 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.336+0000 7f3960ff7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 0x7f395c151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.336+0000 7f3960ff7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c06e850 con 0x7f395c153390 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.336+0000 7f3960ff7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c06d2b0 con 0x7f395c0c22b0 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.336+0000 7f3960ff7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395c06d590 con 0x7f395c0a9830 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f395ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f395c153390 0x7f395c153820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36570/0 (socket says 172.21.15.94:36570) 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f395ad9d640 1 -- 172.21.15.94:0/2263445445 learned_addr learned my addr 172.21.15.94:0/2263445445 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357685039 0 0) 0x7f3954001490 con 0x7f395c153390 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3953fff640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 0x7f395c151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3948003520 con 0x7f395c153390 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2937534314 0 0) 0x7f3948003520 con 0x7f395c153390 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 msgr2=0x7f395c151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 0x7f395c151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 >> v1:172.21.15.145:6789/0 conn(0x7f395c0c22b0 legacy=0x7f395c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c06d870 con 0x7f395c153390 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f3954001bc0 con 0x7f395c153390 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 >> v1:172.21.15.94:6789/0 conn(0x7f395c153390 legacy=0x7f395c153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39480039e0 0x7f3948003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 0x7f3948004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3948004d40 0x7f39480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c06d590 con 0x7f3948004d40 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394800bb50 con 0x7f39480039e0 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394800bd50 con 0x7f395c0a9830 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f395ad9d640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3948004d40 0x7f39480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f3953fff640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 0x7f3948004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.337+0000 7f395a59c640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39480039e0 0x7f3948003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f395ad9d640 1 -- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 msgr2=0x7f3948004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f395ad9d640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c0a9830 0x7f3948004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f395ad9d640 1 -- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39480039e0 msgr2=0x7f3948003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f395ad9d640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39480039e0 0x7f3948003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f395ad9d640 1 -- 172.21.15.94:0/2263445445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3948003520 con 0x7f3948004d40 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f395ad9d640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3948004d40 0x7f39480051c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f395400aa20 tx=0x7f3948005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a51b31e6a7e231 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954038070 con 0x7f3948004d40 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3954033620 con 0x7f3948004d40 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.338+0000 7f3959d9b640 1 -- 172.21.15.94:0/2263445445 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395400a640 con 0x7f3948004d40 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.339+0000 7f3960ff7640 1 -- 172.21.15.94:0/2263445445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3948004d40 msgr2=0x7f39480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.339+0000 7f3960ff7640 1 --2- 172.21.15.94:0/2263445445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3948004d40 0x7f39480051c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f395400aa20 tx=0x7f3948005830 comp rx=0 tx=0).stop 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.339+0000 7f395ad9d640 1 -- 172.21.15.94:0/2263445445 reap_dead start 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.339+0000 7f3960ff7640 1 -- 172.21.15.94:0/2263445445 shutdown_connections 2024-03-06T07:21:56.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.339+0000 7f3960ff7640 1 -- 172.21.15.94:0/2263445445 >> 172.21.15.94:0/2263445445 conn(0x7f395c0c65b0 msgr2=0x7f395c0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.339+0000 7f3960ff7640 1 -- 172.21.15.94:0/2263445445 shutdown_connections 2024-03-06T07:21:56.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.339+0000 7f3960ff7640 1 -- 172.21.15.94:0/2263445445 wait complete. 2024-03-06T07:21:56.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.340+0000 7f3960ff7640 1 Processor -- start 2024-03-06T07:21:56.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.340+0000 7f3960ff7640 1 -- start start 2024-03-06T07:21:56.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3960ff7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f395c151400 0x7f395c1ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3960ff7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f395c1db7f0 0x7f395c1e7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3960ff7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c14bf80 0x7f395c1e7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3960ff7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c153f30 con 0x7f395c1db7f0 2024-03-06T07:21:56.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3960ff7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c14f660 con 0x7f395c151400 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3960ff7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395c0c6270 con 0x7f395c14bf80 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f395c1db7f0 0x7f395c1e7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f395c1db7f0 0x7f395c1e7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35542/0 (socket says 172.21.15.94:35542) 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 -- 172.21.15.94:0/3871456000 learned_addr learned my addr 172.21.15.94:0/3871456000 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 -- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c14bf80 msgr2=0x7f395c1e7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c14bf80 0x7f395c1e7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 -- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f395c151400 msgr2=0x7f395c1ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f395c151400 0x7f395c1ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 -- 172.21.15.94:0/3871456000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c0c59d0 con 0x7f395c1db7f0 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.341+0000 7f3953fff640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f395c1db7f0 0x7f395c1e7000 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f39440090b0 tx=0x7f3944003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.342+0000 7f3952ffd640 1 -- 172.21.15.94:0/3871456000 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944009b20 con 0x7f395c1db7f0 2024-03-06T07:21:56.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.342+0000 7f3952ffd640 1 -- 172.21.15.94:0/3871456000 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394401c070 con 0x7f395c1db7f0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.342+0000 7f3952ffd640 1 -- 172.21.15.94:0/3871456000 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944017420 con 0x7f395c1db7f0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.342+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c0c5cb0 con 0x7f395c1db7f0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.342+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c1e6ae0 con 0x7f395c1db7f0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.343+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c155170 con 0x7f395c1db7f0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.343+0000 7f3952ffd640 1 -- 172.21.15.94:0/3871456000 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39440175c0 con 0x7f395c1db7f0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.343+0000 7f3952ffd640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393c06b4d0 0x7f393c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.344+0000 7f3952ffd640 1 -- 172.21.15.94:0/3871456000 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f394408caf0 con 0x7f395c1db7f0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.344+0000 7f395a59c640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393c06b4d0 0x7f393c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.344+0000 7f395a59c640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393c06b4d0 0x7f393c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f394c001750 tx=0x7f394c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.348+0000 7f3952ffd640 1 -- 172.21.15.94:0/3871456000 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f395c155170 con 0x7f395c1db7f0 2024-03-06T07:21:56.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:56.518+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f395c0c2690 con 0x7f395c1db7f0 2024-03-06T07:21:57.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.220+0000 7f3952ffd640 1 -- 172.21.15.94:0/3871456000 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) v1 ==== 181+0+0 (secure 0 0 0) 0x7f395c0c2690 con 0x7f395c1db7f0 2024-03-06T07:21:57.222 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:21:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393c06b4d0 msgr2=0x7f393c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393c06b4d0 0x7f393c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f394c001750 tx=0x7f394c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f395c1db7f0 msgr2=0x7f395c1e7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f395c1db7f0 0x7f395c1e7000 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f39440090b0 tx=0x7f3944003d50 comp rx=0 tx=0).stop 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 shutdown_connections 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393c06b4d0 0x7f393c06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f395c14bf80 0x7f395c1e7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f395c1db7f0 0x7f395c1e7000 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 --2- 172.21.15.94:0/3871456000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f395c151400 0x7f395c1ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.223+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 >> 172.21.15.94:0/3871456000 conn(0x7f395c0c65b0 msgr2=0x7f395c0ac0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.224+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 shutdown_connections 2024-03-06T07:21:57.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.224+0000 7f3960ff7640 1 -- 172.21.15.94:0/3871456000 wait complete. 2024-03-06T07:21:57.240 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:21:57.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.347+0000 7fd5e24c2640 1 Processor -- start 2024-03-06T07:21:57.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.347+0000 7fd5e24c2640 1 -- start start 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.347+0000 7fd5e24c2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5dc0c22b0 0x7fd5dc151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.347+0000 7fd5e24c2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5dc06e850 con 0x7fd5dc0a9830 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.347+0000 7fd5e24c2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5dc06d2b0 con 0x7fd5dc153390 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.347+0000 7fd5e24c2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5dc06d590 con 0x7fd5dc0c22b0 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.348+0000 7fd5dbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5dc0c22b0 0x7fd5dc151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.348+0000 7fd5db7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd5dc0a9830 0x7fd5dc14ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36574/0 (socket says 172.21.15.94:36574) 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.348+0000 7fd5db7fe640 1 -- 172.21.15.94:0/2879191783 learned_addr learned my addr 172.21.15.94:0/2879191783 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.348+0000 7fd5dbfff640 1 -- 172.21.15.94:0/2879191783 >> v1:172.21.15.145:6789/0 conn(0x7fd5dc153390 legacy=0x7fd5dc153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.348+0000 7fd5dbfff640 1 -- 172.21.15.94:0/2879191783 >> v1:172.21.15.94:6789/0 conn(0x7fd5dc0a9830 legacy=0x7fd5dc14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.348+0000 7fd5daffd640 1 -- 172.21.15.94:0/2879191783 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785600521 0 0) 0x7fd5d0001110 con 0x7fd5dc0a9830 2024-03-06T07:21:57.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.348+0000 7fd5dbfff640 1 -- 172.21.15.94:0/2879191783 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc06d870 con 0x7fd5dc0c22b0 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5dbfff640 1 --2- 172.21.15.94:0/2879191783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc0c22b0 0x7fd5dc151e10 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0097d0 tx=0x7fd5cc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a08c8cc6583674 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5daffd640 1 -- 172.21.15.94:0/2879191783 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc035020 con 0x7fd5dc0c22b0 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5daffd640 1 -- 172.21.15.94:0/2879191783 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5cc031810 con 0x7fd5dc0c22b0 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5daffd640 1 -- 172.21.15.94:0/2879191783 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc031be0 con 0x7fd5dc0c22b0 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2879191783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc0c22b0 msgr2=0x7fd5dc151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2879191783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc0c22b0 0x7fd5dc151e10 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0097d0 tx=0x7fd5cc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2879191783 shutdown_connections 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2879191783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc0c22b0 0x7fd5dc151e10 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2879191783 >> 172.21.15.94:0/2879191783 conn(0x7fd5dc0c65b0 msgr2=0x7fd5dc0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2879191783 shutdown_connections 2024-03-06T07:21:57.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.349+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2879191783 wait complete. 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.350+0000 7fd5e24c2640 1 Processor -- start 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e24c2640 1 -- start start 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e24c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5dc0a9830 0x7fd5dc0c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e24c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5dc0c22b0 0x7fd5dc0c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e24c2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc153390 0x7fd5dc0c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e24c2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc154090 con 0x7fd5dc153390 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e24c2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc153f10 con 0x7fd5dc0c22b0 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e24c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc14f660 con 0x7fd5dc0a9830 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e0a5f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc153390 0x7fd5dc0c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e0a5f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc153390 0x7fd5dc0c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35552/0 (socket says 172.21.15.94:35552) 2024-03-06T07:21:57.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.351+0000 7fd5e0a5f640 1 -- 172.21.15.94:0/2789482418 learned_addr learned my addr 172.21.15.94:0/2789482418 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5dbfff640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5dc0a9830 0x7fd5dc0c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5db7fe640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5dc0c22b0 0x7fd5dc0c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e0a5f640 1 -- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5dc0a9830 msgr2=0x7fd5dc0c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e0a5f640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5dc0a9830 0x7fd5dc0c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e0a5f640 1 -- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5dc0c22b0 msgr2=0x7fd5dc0c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e0a5f640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5dc0c22b0 0x7fd5dc0c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e0a5f640 1 -- 172.21.15.94:0/2789482418 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc0c5e20 con 0x7fd5dc153390 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e0a5f640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc153390 0x7fd5dc0c42f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400c7b0 tx=0x7fd5d400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5d8ff9640 1 -- 172.21.15.94:0/2789482418 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4004b60 con 0x7fd5dc153390 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5d8ff9640 1 -- 172.21.15.94:0/2789482418 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d4004d00 con 0x7fd5dc153390 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5dc151b70 con 0x7fd5dc153390 2024-03-06T07:21:57.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5d8ff9640 1 -- 172.21.15.94:0/2789482418 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d400aa90 con 0x7fd5dc153390 2024-03-06T07:21:57.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.352+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc145500 con 0x7fd5dc153390 2024-03-06T07:21:57.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.353+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5dc075300 con 0x7fd5dc153390 2024-03-06T07:21:57.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.354+0000 7fd5d8ff9640 1 -- 172.21.15.94:0/2789482418 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5d400ac30 con 0x7fd5dc153390 2024-03-06T07:21:57.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.354+0000 7fd5d8ff9640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5c806b4d0 0x7fd5c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.354+0000 7fd5d8ff9640 1 -- 172.21.15.94:0/2789482418 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5864+0+0 (secure 0 0 0) 0x7fd5d408a790 con 0x7fd5dc153390 2024-03-06T07:21:57.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.354+0000 7fd5dbfff640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5c806b4d0 0x7fd5c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.355+0000 7fd5dbfff640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5c806b4d0 0x7fd5c806d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc036000 tx=0x7fd5cc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.359+0000 7fd5d8ff9640 1 -- 172.21.15.94:0/2789482418 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5dc075300 con 0x7fd5dc153390 2024-03-06T07:21:57.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:57.537+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd5dc04f330 con 0x7fd5dc153390 2024-03-06T07:21:58.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.221+0000 7fd5d8ff9640 1 -- 172.21.15.94:0/2789482418 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd5d4059600 con 0x7fd5dc153390 2024-03-06T07:21:58.223 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.224+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5c806b4d0 msgr2=0x7fd5c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.224+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5c806b4d0 0x7fd5c806d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc036000 tx=0x7fd5cc03c040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.224+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc153390 msgr2=0x7fd5dc0c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.224+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc153390 0x7fd5dc0c42f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400c7b0 tx=0x7fd5d400cc70 comp rx=0 tx=0).stop 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.224+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 shutdown_connections 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.225+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5c806b4d0 0x7fd5c806d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.225+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5dc153390 0x7fd5dc0c42f0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.225+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5dc0c22b0 0x7fd5dc0c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.225+0000 7fd5e24c2640 1 --2- 172.21.15.94:0/2789482418 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5dc0a9830 0x7fd5dc0c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.225+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 >> 172.21.15.94:0/2789482418 conn(0x7fd5dc0c65b0 msgr2=0x7fd5dc071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.225+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 shutdown_connections 2024-03-06T07:21:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.225+0000 7fd5e24c2640 1 -- 172.21.15.94:0/2789482418 wait complete. 2024-03-06T07:21:58.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:21:58.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.351+0000 7ff0b8d5e640 1 Processor -- start 2024-03-06T07:21:58.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.351+0000 7ff0b8d5e640 1 -- start start 2024-03-06T07:21:58.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b8d5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 0x7ff0b40c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b8d5e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b40c3ef0 con 0x7ff0b406c900 2024-03-06T07:21:58.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b8d5e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b41556d0 con 0x7ff0b40ab170 2024-03-06T07:21:58.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b8d5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b41559b0 con 0x7ff0b40c5990 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b2d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff0b406c900 0x7ff0b40a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36576/0 (socket says 172.21.15.94:36576) 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b2d9d640 1 -- 172.21.15.94:0/2319628334 learned_addr learned my addr 172.21.15.94:0/2319628334 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060904360 0 0) 0x7ff0b40c3ef0 con 0x7ff0b406c900 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a0003520 con 0x7ff0b406c900 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b259c640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 0x7ff0b40c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23893464 0 0) 0x7ff0a0003520 con 0x7ff0b406c900 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 msgr2=0x7ff0b40c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 0x7ff0b40c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 >> v1:172.21.15.145:6789/0 conn(0x7ff0b40ab170 legacy=0x7ff0b406c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4155c90 con 0x7ff0b406c900 2024-03-06T07:21:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7ff0ac0019f0 con 0x7ff0b406c900 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 >> v1:172.21.15.94:6789/0 conn(0x7ff0b406c900 legacy=0x7ff0b40a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 0x7ff0a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a0004010 0x7ff0a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0a0004d40 0x7ff0a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b41559b0 con 0x7ff0a0004d40 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a000bb80 con 0x7ff0a0004010 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a000bd80 con 0x7ff0b40c5990 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b259c640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 0x7ff0a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0b2d9d640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0a0004d40 0x7ff0a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.352+0000 7ff0abfff640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a0004010 0x7ff0a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.353+0000 7ff0b2d9d640 1 -- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 msgr2=0x7ff0a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.353+0000 7ff0b2d9d640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b40c5990 0x7ff0a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.353+0000 7ff0b2d9d640 1 -- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a0004010 msgr2=0x7ff0a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.353+0000 7ff0b2d9d640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a0004010 0x7ff0a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.353+0000 7ff0b2d9d640 1 -- 172.21.15.94:0/2319628334 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a0003520 con 0x7ff0a0004d40 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b2d9d640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0a0004d40 0x7ff0a00051c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00a810 tx=0x7ff0a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702170b406586b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00a460 con 0x7ff0a0004d40 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac034650 con 0x7ff0a0004d40 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b1d9b640 1 -- 172.21.15.94:0/2319628334 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac034a80 con 0x7ff0a0004d40 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2319628334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0a0004d40 msgr2=0x7ff0a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b8d5e640 1 --2- 172.21.15.94:0/2319628334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0a0004d40 0x7ff0a00051c0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00a810 tx=0x7ff0a0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:58.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b2d9d640 1 -- 172.21.15.94:0/2319628334 reap_dead start 2024-03-06T07:21:58.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2319628334 shutdown_connections 2024-03-06T07:21:58.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2319628334 >> 172.21.15.94:0/2319628334 conn(0x7ff0b40c6710 msgr2=0x7ff0b40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2319628334 shutdown_connections 2024-03-06T07:21:58.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.354+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2319628334 wait complete. 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.355+0000 7ff0b8d5e640 1 Processor -- start 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b8d5e640 1 -- start start 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b8d5e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b41508f0 0x7ff0b4148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b8d5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0b4153010 0x7ff0b4148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b8d5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b41e8b30 0x7ff0b41e8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b8d5e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4156610 con 0x7ff0b4153010 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b8d5e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4156490 con 0x7ff0b41508f0 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b8d5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40c3af0 con 0x7ff0b41e8b30 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0b4153010 0x7ff0b4148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0b4153010 0x7ff0b4148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35580/0 (socket says 172.21.15.94:35580) 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0abfff640 1 -- 172.21.15.94:0/2948794859 learned_addr learned my addr 172.21.15.94:0/2948794859 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b259c640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b41508f0 0x7ff0b4148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.356+0000 7ff0b2d9d640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b41e8b30 0x7ff0b41e8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0abfff640 1 -- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b41e8b30 msgr2=0x7ff0b41e8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0abfff640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b41e8b30 0x7ff0b41e8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0abfff640 1 -- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b41508f0 msgr2=0x7ff0b4148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0abfff640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b41508f0 0x7ff0b4148870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0abfff640 1 -- 172.21.15.94:0/2948794859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b41e0250 con 0x7ff0b4153010 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0b259c640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b41508f0 0x7ff0b4148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0abfff640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0b4153010 0x7ff0b4148db0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00f7f0 tx=0x7ff09c00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0aaffd640 1 -- 172.21.15.94:0/2948794859 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c0050d0 con 0x7ff0b4153010 2024-03-06T07:21:58.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0aaffd640 1 -- 172.21.15.94:0/2948794859 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c018070 con 0x7ff0b4153010 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b41dff70 con 0x7ff0b4153010 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0aaffd640 1 -- 172.21.15.94:0/2948794859 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00d6d0 con 0x7ff0b4153010 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.357+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b4153800 con 0x7ff0b4153010 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.358+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b404f330 con 0x7ff0b4153010 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.358+0000 7ff0aaffd640 1 -- 172.21.15.94:0/2948794859 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff09c00d870 con 0x7ff0b4153010 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.359+0000 7ff0aaffd640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff09406b500 0x7ff09406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.359+0000 7ff0aaffd640 1 -- 172.21.15.94:0/2948794859 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7ff09c089e70 con 0x7ff0b4153010 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.359+0000 7ff0b259c640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff09406b500 0x7ff09406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.359+0000 7ff0b259c640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff09406b500 0x7ff09406d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4006fd0 tx=0x7ff0a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.363+0000 7ff0aaffd640 1 -- 172.21.15.94:0/2948794859 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff09c058ea0 con 0x7ff0b4153010 2024-03-06T07:21:58.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.534+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff0b40c1740 con 0x7ff0b4153010 2024-03-06T07:21:58.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.537+0000 7ff0aaffd640 1 -- 172.21.15.94:0/2948794859 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff09c0587a0 con 0x7ff0b4153010 2024-03-06T07:21:58.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff09406b500 msgr2=0x7ff09406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff09406b500 0x7ff09406d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4006fd0 tx=0x7ff0a4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0b4153010 msgr2=0x7ff0b4148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0b4153010 0x7ff0b4148db0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00f7f0 tx=0x7ff09c00fcb0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 shutdown_connections 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff09406b500 0x7ff09406d9c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b41e8b30 0x7ff0b41e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0b4153010 0x7ff0b4148db0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 --2- 172.21.15.94:0/2948794859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b41508f0 0x7ff0b4148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 >> 172.21.15.94:0/2948794859 conn(0x7ff0b40c6710 msgr2=0x7ff0b40aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 shutdown_connections 2024-03-06T07:21:58.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.539+0000 7ff0b8d5e640 1 -- 172.21.15.94:0/2948794859 wait complete. 2024-03-06T07:21:58.552 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:21:58.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.652+0000 7f93309a0640 1 Processor -- start 2024-03-06T07:21:58.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.652+0000 7f93309a0640 1 -- start start 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f93309a0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f932c1538c0 0x7f932c1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f93309a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c154330 con 0x7f932c0a97c0 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f93309a0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c153dc0 con 0x7f932c0ab310 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f93309a0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c0c46a0 con 0x7f932c1538c0 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f9329d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f932c1538c0 0x7f932c1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f932ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f932c0a97c0 0x7f932c0c4b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52432/0 (socket says 172.21.15.94:52432) 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f932ad9d640 1 -- 172.21.15.94:0/4268399174 learned_addr learned my addr 172.21.15.94:0/4268399174 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f9329d9b640 1 -- 172.21.15.94:0/4268399174 >> v1:172.21.15.145:6789/0 conn(0x7f932c0ab310 legacy=0x7f932c0ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f9329d9b640 1 -- 172.21.15.94:0/4268399174 >> v1:172.21.15.94:6789/0 conn(0x7f932c0a97c0 legacy=0x7f932c0c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f9329d9b640 1 -- 172.21.15.94:0/4268399174 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c0c5d30 con 0x7f932c1538c0 2024-03-06T07:21:58.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.653+0000 7f932959a640 1 -- 172.21.15.94:0/4268399174 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458552658 0 0) 0x7f932c154330 con 0x7f932c0a97c0 2024-03-06T07:21:58.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.654+0000 7f9329d9b640 1 --2- 172.21.15.94:0/4268399174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c1494a0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f93200097a0 tx=0x7f932002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7547765d7b09d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.654+0000 7f932959a640 1 -- 172.21.15.94:0/4268399174 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93200315a0 con 0x7f932c1538c0 2024-03-06T07:21:58.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.654+0000 7f932959a640 1 -- 172.21.15.94:0/4268399174 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9320037070 con 0x7f932c1538c0 2024-03-06T07:21:58.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.654+0000 7f932959a640 1 -- 172.21.15.94:0/4268399174 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320031960 con 0x7f932c1538c0 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.655+0000 7f93309a0640 1 -- 172.21.15.94:0/4268399174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 msgr2=0x7f932c1494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.655+0000 7f93309a0640 1 --2- 172.21.15.94:0/4268399174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c1494a0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f93200097a0 tx=0x7f932002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.655+0000 7f93309a0640 1 -- 172.21.15.94:0/4268399174 shutdown_connections 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.655+0000 7f93309a0640 1 --2- 172.21.15.94:0/4268399174 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c1494a0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.655+0000 7f93309a0640 1 -- 172.21.15.94:0/4268399174 >> 172.21.15.94:0/4268399174 conn(0x7f932c0c6710 msgr2=0x7f932c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.655+0000 7f93309a0640 1 -- 172.21.15.94:0/4268399174 shutdown_connections 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.655+0000 7f93309a0640 1 -- 172.21.15.94:0/4268399174 wait complete. 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.656+0000 7f93309a0640 1 Processor -- start 2024-03-06T07:21:58.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.656+0000 7f93309a0640 1 -- start start 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f93309a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c0a97c0 0x7f932c14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f93309a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f932c0ab310 0x7f932c1e9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f93309a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c14b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f93309a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c5240 con 0x7f932c1538c0 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f93309a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c62f0 con 0x7f932c0a97c0 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f93309a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c53c0 con 0x7f932c0ab310 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c14b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c14b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37910/0 (socket says 172.21.15.94:37910) 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 -- 172.21.15.94:0/1086762643 learned_addr learned my addr 172.21.15.94:0/1086762643 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:58.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932a59c640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c0a97c0 0x7f932c14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 -- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f932c0ab310 msgr2=0x7f932c1e9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f9329d9b640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f932c0ab310 0x7f932c1e9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f932c0ab310 0x7f932c1e9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 -- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c0a97c0 msgr2=0x7f932c14fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c0a97c0 0x7f932c14fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 -- 172.21.15.94:0/1086762643 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c070910 con 0x7f932c1538c0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932a59c640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c0a97c0 0x7f932c14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f9329d9b640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f932c0ab310 0x7f932c1e9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.657+0000 7f932ad9d640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c14b9a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f932400c7d0 tx=0x7f932400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.658+0000 7f931affd640 1 -- 172.21.15.94:0/1086762643 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324018070 con 0x7f932c1538c0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.658+0000 7f931affd640 1 -- 172.21.15.94:0/1086762643 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932400a640 con 0x7f932c1538c0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.658+0000 7f931affd640 1 -- 172.21.15.94:0/1086762643 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932400aa70 con 0x7f932c1538c0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.658+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c14f090 con 0x7f932c1538c0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.658+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c070630 con 0x7f932c1538c0 2024-03-06T07:21:58.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.659+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c04f330 con 0x7f932c1538c0 2024-03-06T07:21:58.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.659+0000 7f931affd640 1 -- 172.21.15.94:0/1086762643 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f932401b070 con 0x7f932c1538c0 2024-03-06T07:21:58.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.660+0000 7f931affd640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f930c06b4d0 0x7f930c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.660+0000 7f931affd640 1 -- 172.21.15.94:0/1086762643 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f9324089670 con 0x7f932c1538c0 2024-03-06T07:21:58.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.660+0000 7f932a59c640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f930c06b4d0 0x7f930c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.660+0000 7f932a59c640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f930c06b4d0 0x7f930c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f932c1e63a0 tx=0x7f931c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.664+0000 7f931affd640 1 -- 172.21.15.94:0/1086762643 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93240586a0 con 0x7f932c1538c0 2024-03-06T07:21:58.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.835+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f932c156560 con 0x7f932c1538c0 2024-03-06T07:21:58.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.838+0000 7f931affd640 1 -- 172.21.15.94:0/1086762643 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v34) v1 ==== 96+0+0 (secure 0 0 0) 0x7f9324057fa0 con 0x7f932c1538c0 2024-03-06T07:21:58.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f930c06b4d0 msgr2=0x7f930c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f930c06b4d0 0x7f930c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f932c1e63a0 tx=0x7f931c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 msgr2=0x7f932c14b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c14b9a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f932400c7d0 tx=0x7f932400cc90 comp rx=0 tx=0).stop 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 shutdown_connections 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f930c06b4d0 0x7f930c06d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f932c1538c0 0x7f932c14b9a0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f932c0ab310 0x7f932c1e9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 --2- 172.21.15.94:0/1086762643 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f932c0a97c0 0x7f932c14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.840+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 >> 172.21.15.94:0/1086762643 conn(0x7f932c0c6710 msgr2=0x7f932c0c4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.841+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 shutdown_connections 2024-03-06T07:21:58.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.841+0000 7f93309a0640 1 -- 172.21.15.94:0/1086762643 wait complete. 2024-03-06T07:21:58.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-03-06T07:21:58.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.951+0000 7f2bd4c66640 1 Processor -- start 2024-03-06T07:21:58.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.951+0000 7f2bd4c66640 1 -- start start 2024-03-06T07:21:58.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.952+0000 7f2bd4c66640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.952+0000 7f2bd4c66640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2bd0156ea0 0x7f2bd00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.952+0000 7f2bd4c66640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd0070800 con 0x7f2bd00acc00 2024-03-06T07:21:58.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.952+0000 7f2bd4c66640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd0157320 con 0x7f2bd0156ea0 2024-03-06T07:21:58.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.952+0000 7f2bd4c66640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd0156450 con 0x7f2bd01506b0 2024-03-06T07:21:58.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.952+0000 7f2bce59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.952+0000 7f2bcdd9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2bd0156ea0 0x7f2bd00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bce59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:60222/0 (socket says 172.21.15.94:60222) 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bcdd9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2bd0156ea0 0x7f2bd00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:46310/0 (socket says 172.21.15.94:46310) 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bced9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2bd00acc00 0x7f2bd00c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52762/0 (socket says 172.21.15.94:52762) 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bce59c640 1 -- 172.21.15.94:0/3211919231 learned_addr learned my addr 172.21.15.94:0/3211919231 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bce59c640 1 -- 172.21.15.94:0/3211919231 >> v2:172.21.15.145:3300/0 conn(0x7f2bd0156ea0 msgr2=0x7f2bd00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bce59c640 1 --2- 172.21.15.94:0/3211919231 >> v2:172.21.15.145:3300/0 conn(0x7f2bd0156ea0 0x7f2bd00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bce59c640 1 -- 172.21.15.94:0/3211919231 >> v1:172.21.15.145:6789/0 conn(0x7f2bd00acc00 legacy=0x7f2bd00c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bce59c640 1 -- 172.21.15.94:0/3211919231 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd0070aa0 con 0x7f2bd01506b0 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bcd59a640 1 -- 172.21.15.94:0/3211919231 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639115671 0 0) 0x7f2bd0070800 con 0x7f2bd00acc00 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.953+0000 7f2bcdd9b640 1 --2- 172.21.15.94:0/3211919231 >> v2:172.21.15.145:3300/0 conn(0x7f2bd0156ea0 0x7f2bd00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:58.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.954+0000 7f2bce59c640 1 --2- 172.21.15.94:0/3211919231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0153990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009e50 tx=0x7f2bc002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d38a78db0509b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.954+0000 7f2bcd59a640 1 -- 172.21.15.94:0/3211919231 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc002cb00 con 0x7f2bd01506b0 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.954+0000 7f2bcd59a640 1 -- 172.21.15.94:0/3211919231 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc002cca0 con 0x7f2bd01506b0 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.954+0000 7f2bcd59a640 1 -- 172.21.15.94:0/3211919231 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0034610 con 0x7f2bd01506b0 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.954+0000 7f2bd4c66640 1 -- 172.21.15.94:0/3211919231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 msgr2=0x7f2bd0153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.954+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/3211919231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0153990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009e50 tx=0x7f2bc002bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.955+0000 7f2bd4c66640 1 -- 172.21.15.94:0/3211919231 shutdown_connections 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.955+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/3211919231 >> v2:172.21.15.145:3300/0 conn(0x7f2bd0156ea0 0x7f2bd00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.955+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/3211919231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0153990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.955+0000 7f2bd4c66640 1 -- 172.21.15.94:0/3211919231 >> 172.21.15.94:0/3211919231 conn(0x7f2bd006b1a0 msgr2=0x7f2bd006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.955+0000 7f2bd4c66640 1 -- 172.21.15.94:0/3211919231 shutdown_connections 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.955+0000 7f2bd4c66640 1 -- 172.21.15.94:0/3211919231 wait complete. 2024-03-06T07:21:58.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 Processor -- start 2024-03-06T07:21:58.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 -- start start 2024-03-06T07:21:58.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bd00acc00 0x7f2bd01eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bd0156ea0 0x7f2bd0155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd01566a0 con 0x7f2bd00acc00 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd0156a10 con 0x7f2bd0156ea0 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.956+0000 7f2bd4c66640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c5bb0 con 0x7f2bd01506b0 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bd00acc00 0x7f2bd01eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bd00acc00 0x7f2bd01eaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37926/0 (socket says 172.21.15.94:37926) 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 -- 172.21.15.94:0/150535444 learned_addr learned my addr 172.21.15.94:0/150535444 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bced9d640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bd0156ea0 0x7f2bd0155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bcdd9b640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 -- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 msgr2=0x7f2bd0148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0148d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 -- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bd0156ea0 msgr2=0x7f2bd0155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bd0156ea0 0x7f2bd0155260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 -- 172.21.15.94:0/150535444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd00c26c0 con 0x7f2bd00acc00 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bced9d640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bd0156ea0 0x7f2bd0155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:58.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bce59c640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bd00acc00 0x7f2bd01eaac0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0036040 tx=0x7f2bc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bbeffd640 1 -- 172.21.15.94:0/150535444 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0043020 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bbeffd640 1 -- 172.21.15.94:0/150535444 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc002ce50 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd01ee160 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bbeffd640 1 -- 172.21.15.94:0/150535444 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0009510 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.957+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd01ee400 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.958+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd004f330 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.959+0000 7f2bbeffd640 1 -- 172.21.15.94:0/150535444 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bc0046020 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.959+0000 7f2bbeffd640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2bb006b500 0x7f2bb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.959+0000 7f2bbeffd640 1 -- 172.21.15.94:0/150535444 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2bc00ac630 con 0x7f2bd00acc00 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.959+0000 7f2bcdd9b640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2bb006b500 0x7f2bb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.960+0000 7f2bcdd9b640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2bb006b500 0x7f2bb006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0149cb0 tx=0x7f2bc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:58.963+0000 7f2bbeffd640 1 -- 172.21.15.94:0/150535444 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bc004d030 con 0x7f2bd00acc00 2024-03-06T07:21:59.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.198+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7f2bd00700a0 con 0x7f2bd00acc00 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.222+0000 7f2bbeffd640 1 -- 172.21.15.94:0/150535444 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v225) v1 ==== 146+0+0 (secure 0 0 0) 0x7f2bc003c350 con 0x7f2bd00acc00 2024-03-06T07:21:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2bb006b500 msgr2=0x7f2bb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2bb006b500 0x7f2bb006d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0149cb0 tx=0x7f2bc4008040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bd00acc00 msgr2=0x7f2bd01eaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bd00acc00 0x7f2bd01eaac0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0036040 tx=0x7f2bc0004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 shutdown_connections 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2bb006b500 0x7f2bb006d9c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bd0156ea0 0x7f2bd0155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bd01506b0 0x7f2bd0148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 --2- 172.21.15.94:0/150535444 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bd00acc00 0x7f2bd01eaac0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 >> 172.21.15.94:0/150535444 conn(0x7f2bd006b1a0 msgr2=0x7f2bd00ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 shutdown_connections 2024-03-06T07:21:59.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.224+0000 7f2bd4c66640 1 -- 172.21.15.94:0/150535444 wait complete. 2024-03-06T07:21:59.237 INFO:tasks.cephfs_test_runner:That setting mds_join_fs at runtime affects the cluster layout. ... ok 2024-03-06T07:21:59.238 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) That a standby-replay daemon with weak affinity is replaced by a stronger one. 2024-03-06T07:21:59.238 INFO:tasks.cephfs_test_runner:test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:21:59.238 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.339+0000 7fc232888640 1 Processor -- start 2024-03-06T07:21:59.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.339+0000 7fc232888640 1 -- start start 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.339+0000 7fc232888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c14f360 0x7fc22c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.339+0000 7fc232888640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc22c0c88c0 0x7fc22c0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.339+0000 7fc232888640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c0c73d0 con 0x7fc22c0c5880 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.339+0000 7fc232888640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc22c0c8dd0 con 0x7fc22c0c88c0 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.339+0000 7fc232888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c0be2e0 con 0x7fc22c14f360 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c14f360 0x7fc22c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22b7fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc22c0c88c0 0x7fc22c0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c14f360 0x7fc22c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:60238/0 (socket says 172.21.15.94:60238) 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22bfff640 1 -- 172.21.15.94:0/72116996 learned_addr learned my addr 172.21.15.94:0/72116996 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:59.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22bfff640 1 -- 172.21.15.94:0/72116996 >> v2:172.21.15.145:3300/0 conn(0x7fc22c0c88c0 msgr2=0x7fc22c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22bfff640 1 --2- 172.21.15.94:0/72116996 >> v2:172.21.15.145:3300/0 conn(0x7fc22c0c88c0 0x7fc22c0c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22bfff640 1 -- 172.21.15.94:0/72116996 >> v1:172.21.15.145:6789/0 conn(0x7fc22c0c5880 legacy=0x7fc22c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22bfff640 1 -- 172.21.15.94:0/72116996 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0c7670 con 0x7fc22c14f360 2024-03-06T07:21:59.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.340+0000 7fc22b7fe640 1 --2- 172.21.15.94:0/72116996 >> v2:172.21.15.145:3300/0 conn(0x7fc22c0c88c0 0x7fc22c0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:59.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.341+0000 7fc22bfff640 1 --2- 172.21.15.94:0/72116996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c14f360 0x7fc22c155c10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc220009870 tx=0x7fc22002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de82fd7f599bcc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.341+0000 7fc22affd640 1 -- 172.21.15.94:0/72116996 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220035020 con 0x7fc22c14f360 2024-03-06T07:21:59.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.341+0000 7fc22affd640 1 -- 172.21.15.94:0/72116996 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc220031640 con 0x7fc22c14f360 2024-03-06T07:21:59.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.341+0000 7fc22affd640 1 -- 172.21.15.94:0/72116996 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22002d040 con 0x7fc22c14f360 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 -- 172.21.15.94:0/72116996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c14f360 msgr2=0x7fc22c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 --2- 172.21.15.94:0/72116996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c14f360 0x7fc22c155c10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc220009870 tx=0x7fc22002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 -- 172.21.15.94:0/72116996 shutdown_connections 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 --2- 172.21.15.94:0/72116996 >> v2:172.21.15.145:3300/0 conn(0x7fc22c0c88c0 0x7fc22c0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 --2- 172.21.15.94:0/72116996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c14f360 0x7fc22c155c10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 -- 172.21.15.94:0/72116996 >> 172.21.15.94:0/72116996 conn(0x7fc22c0ac570 msgr2=0x7fc22c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 -- 172.21.15.94:0/72116996 shutdown_connections 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.342+0000 7fc232888640 1 -- 172.21.15.94:0/72116996 wait complete. 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.343+0000 7fc232888640 1 Processor -- start 2024-03-06T07:21:59.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.343+0000 7fc232888640 1 -- start start 2024-03-06T07:21:59.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.343+0000 7fc232888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c0c5880 0x7fc22c156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc232888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc22c0c88c0 0x7fc22c0c09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc232888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc22c14f360 0x7fc22c0c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc232888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c159270 con 0x7fc22c0c88c0 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc232888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c0c7c30 con 0x7fc22c14f360 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc232888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c1593f0 con 0x7fc22c0c5880 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc22c0c88c0 0x7fc22c0c09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc22c0c88c0 0x7fc22c0c09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37936/0 (socket says 172.21.15.94:37936) 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 -- 172.21.15.94:0/2524256937 learned_addr learned my addr 172.21.15.94:0/2524256937 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22bfff640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c0c5880 0x7fc22c156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc230e25640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc22c14f360 0x7fc22c0c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 -- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c0c5880 msgr2=0x7fc22c156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c0c5880 0x7fc22c156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 -- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc22c14f360 msgr2=0x7fc22c0c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc22c14f360 0x7fc22c0c0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 -- 172.21.15.94:0/2524256937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0c1430 con 0x7fc22c0c88c0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc22b7fe640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc22c0c88c0 0x7fc22c0c09b0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00e9e0 tx=0x7fc21c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc228ff9640 1 -- 172.21.15.94:0/2524256937 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c017070 con 0x7fc22c0c88c0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc228ff9640 1 -- 172.21.15.94:0/2524256937 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c00ca30 con 0x7fc22c0c88c0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc228ff9640 1 -- 172.21.15.94:0/2524256937 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c00ce20 con 0x7fc22c0c88c0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.344+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22c0aff90 con 0x7fc22c0c88c0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.345+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22c158080 con 0x7fc22c0c88c0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.346+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22c04f330 con 0x7fc22c0c88c0 2024-03-06T07:21:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.346+0000 7fc228ff9640 1 -- 172.21.15.94:0/2524256937 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc21c01a070 con 0x7fc22c0c88c0 2024-03-06T07:21:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.346+0000 7fc228ff9640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc21806b4d0 0x7fc21806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.346+0000 7fc228ff9640 1 -- 172.21.15.94:0/2524256937 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc21c089b70 con 0x7fc22c0c88c0 2024-03-06T07:21:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.347+0000 7fc22bfff640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc21806b4d0 0x7fc21806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.347+0000 7fc22bfff640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc21806b4d0 0x7fc21806d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc220036000 tx=0x7fc22003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.351+0000 7fc228ff9640 1 -- 172.21.15.94:0/2524256937 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc22c0c1430 con 0x7fc22c0c88c0 2024-03-06T07:21:59.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.524+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc22c0b0680 con 0x7fc22c0c88c0 2024-03-06T07:21:59.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.524+0000 7fc228ff9640 1 -- 172.21.15.94:0/2524256937 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+13426 (secure 0 0 0) 0x7fc21c058550 con 0x7fc22c0c88c0 2024-03-06T07:21:59.526 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:21:59.526 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":99,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:21:58.220943+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/744582506":"2024-03-07T07:21:55.213711+0000","172.21.15.145:6835/744582506":"2024-03-07T07:21:55.213711+0000","172.21.15.145:6836/3774173957":"2024-03-07T07:21:46.796043+0000","172.21.15.145:6834/3774173957":"2024-03-07T07:21:46.796043+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc21806b4d0 msgr2=0x7fc21806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc21806b4d0 0x7fc21806d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc220036000 tx=0x7fc22003c040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc22c0c88c0 msgr2=0x7fc22c0c09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc22c0c88c0 0x7fc22c0c09b0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00e9e0 tx=0x7fc21c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 shutdown_connections 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc21806b4d0 0x7fc21806d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc22c14f360 0x7fc22c0c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc22c0c88c0 0x7fc22c0c09b0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 --2- 172.21.15.94:0/2524256937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc22c0c5880 0x7fc22c156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.526+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 >> 172.21.15.94:0/2524256937 conn(0x7fc22c0ac570 msgr2=0x7fc22c153310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.527+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 shutdown_connections 2024-03-06T07:21:59.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.527+0000 7fc232888640 1 -- 172.21.15.94:0/2524256937 wait complete. 2024-03-06T07:21:59.539 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:59.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:59.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.641+0000 7f0a474fa640 1 Processor -- start 2024-03-06T07:21:59.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.641+0000 7f0a474fa640 1 -- start start 2024-03-06T07:21:59.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a474fa640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0a400c5990 0x7f0a400c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a474fa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0a400ab170 0x7f0a4006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a474fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 0x7f0a401523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a474fa640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4006cba0 con 0x7f0a400ab170 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a474fa640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a400c3ef0 con 0x7f0a400c5990 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a474fa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a401556d0 con 0x7f0a4006c730 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0a400ab170 0x7f0a4006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0a400ab170 0x7f0a4006c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37952/0 (socket says 172.21.15.94:37952) 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 -- 172.21.15.94:0/3595043243 learned_addr learned my addr 172.21.15.94:0/3595043243 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 -- 172.21.15.94:0/3595043243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 msgr2=0x7f0a401523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 --2- 172.21.15.94:0/3595043243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 0x7f0a401523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 -- 172.21.15.94:0/3595043243 >> v2:172.21.15.145:3300/0 conn(0x7f0a400c5990 msgr2=0x7f0a400c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 --2- 172.21.15.94:0/3595043243 >> v2:172.21.15.145:3300/0 conn(0x7f0a400c5990 0x7f0a400c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.642+0000 7f0a44a95640 1 -- 172.21.15.94:0/3595043243 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a401559b0 con 0x7f0a400ab170 2024-03-06T07:21:59.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a44a95640 1 --2- 172.21.15.94:0/3595043243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400ab170 0x7f0a4006c1f0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f0a400c12a0 tx=0x7f0a3c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172ba67988cea50a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a37fff640 1 -- 172.21.15.94:0/3595043243 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c035020 con 0x7f0a400ab170 2024-03-06T07:21:59.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a37fff640 1 -- 172.21.15.94:0/3595043243 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c02d040 con 0x7f0a400ab170 2024-03-06T07:21:59.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a37fff640 1 -- 172.21.15.94:0/3595043243 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c031a80 con 0x7f0a400ab170 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a474fa640 1 -- 172.21.15.94:0/3595043243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400ab170 msgr2=0x7f0a4006c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a474fa640 1 --2- 172.21.15.94:0/3595043243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400ab170 0x7f0a4006c1f0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f0a400c12a0 tx=0x7f0a3c02b900 comp rx=0 tx=0).stop 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a474fa640 1 -- 172.21.15.94:0/3595043243 shutdown_connections 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a474fa640 1 --2- 172.21.15.94:0/3595043243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 0x7f0a401523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a474fa640 1 --2- 172.21.15.94:0/3595043243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400ab170 0x7f0a4006c1f0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a474fa640 1 --2- 172.21.15.94:0/3595043243 >> v2:172.21.15.145:3300/0 conn(0x7f0a400c5990 0x7f0a400c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.643+0000 7f0a474fa640 1 -- 172.21.15.94:0/3595043243 >> 172.21.15.94:0/3595043243 conn(0x7f0a400c6710 msgr2=0x7f0a400c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.644+0000 7f0a474fa640 1 -- 172.21.15.94:0/3595043243 shutdown_connections 2024-03-06T07:21:59.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.644+0000 7f0a474fa640 1 -- 172.21.15.94:0/3595043243 wait complete. 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 Processor -- start 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 -- start start 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 0x7f0a4014ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a400ab170 0x7f0a400a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400c5990 0x7f0a40072210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40156490 con 0x7f0a400c5990 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40155f70 con 0x7f0a400ab170 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a474fa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40156610 con 0x7f0a4006c730 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a45a97640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400c5990 0x7f0a40072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a45a97640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400c5990 0x7f0a40072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37958/0 (socket says 172.21.15.94:37958) 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.645+0000 7f0a45a97640 1 -- 172.21.15.94:0/1332370997 learned_addr learned my addr 172.21.15.94:0/1332370997 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a44a95640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a400ab170 0x7f0a400a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a45296640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 0x7f0a4014ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a45a97640 1 -- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 msgr2=0x7f0a4014ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a45a97640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 0x7f0a4014ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a45a97640 1 -- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a400ab170 msgr2=0x7f0a400a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a45a97640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a400ab170 0x7f0a400a7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a45a97640 1 -- 172.21.15.94:0/1332370997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4014fb90 con 0x7f0a400c5990 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a45a97640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400c5990 0x7f0a40072210 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800b9a0 tx=0x7f0a3800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a35ffb640 1 -- 172.21.15.94:0/1332370997 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38019070 con 0x7f0a400c5990 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a35ffb640 1 -- 172.21.15.94:0/1332370997 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3800cbc0 con 0x7f0a400c5990 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a401e7880 con 0x7f0a400c5990 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a35ffb640 1 -- 172.21.15.94:0/1332370997 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a38014440 con 0x7f0a400c5990 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.646+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a401e7e20 con 0x7f0a400c5990 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.647+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4004f330 con 0x7f0a400c5990 2024-03-06T07:21:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.647+0000 7f0a35ffb640 1 -- 172.21.15.94:0/1332370997 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a3801c070 con 0x7f0a400c5990 2024-03-06T07:21:59.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.648+0000 7f0a35ffb640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a2406b500 0x7f0a2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.648+0000 7f0a35ffb640 1 -- 172.21.15.94:0/1332370997 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f0a38089d60 con 0x7f0a400c5990 2024-03-06T07:21:59.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.648+0000 7f0a45296640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a2406b500 0x7f0a2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.648+0000 7f0a45296640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a2406b500 0x7f0a2406d9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000a810 tx=0x7f0a30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.652+0000 7f0a35ffb640 1 -- 172.21.15.94:0/1332370997 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a38058d90 con 0x7f0a400c5990 2024-03-06T07:21:59.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.823+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a400aa180 con 0x7f0a400c5990 2024-03-06T07:21:59.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.823+0000 7f0a35ffb640 1 -- 172.21.15.94:0/1332370997 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a38058690 con 0x7f0a400c5990 2024-03-06T07:21:59.826 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:21:59.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a2406b500 msgr2=0x7f0a2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a2406b500 0x7f0a2406d9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000a810 tx=0x7f0a30008040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400c5990 msgr2=0x7f0a40072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400c5990 0x7f0a40072210 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800b9a0 tx=0x7f0a3800be60 comp rx=0 tx=0).stop 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 shutdown_connections 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a2406b500 0x7f0a2406d9c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a400c5990 0x7f0a40072210 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a400ab170 0x7f0a400a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 --2- 172.21.15.94:0/1332370997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a4006c730 0x7f0a4014ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 >> 172.21.15.94:0/1332370997 conn(0x7f0a400c6710 msgr2=0x7f0a400c27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 shutdown_connections 2024-03-06T07:21:59.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.825+0000 7f0a474fa640 1 -- 172.21.15.94:0/1332370997 wait complete. 2024-03-06T07:21:59.838 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-03-06T07:21:59.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.937+0000 7fe8ecafb640 1 Processor -- start 2024-03-06T07:21:59.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.937+0000 7fe8ecafb640 1 -- start start 2024-03-06T07:21:59.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.937+0000 7fe8ecafb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe8e814f790 0x7fe8e81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.937+0000 7fe8ecafb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 0x7fe8e8155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.937+0000 7fe8ecafb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e81563d0 con 0x7fe8e80c1aa0 2024-03-06T07:21:59.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.937+0000 7fe8ecafb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e80c2680 con 0x7fe8e814f790 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.937+0000 7fe8ecafb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c28d0 con 0x7fe8e8155a40 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e5d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe8e80c1aa0 0x7fe8e80c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52436/0 (socket says 172.21.15.94:52436) 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/3723643438 learned_addr learned my addr 172.21.15.94:0/3723643438 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e6d9d640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 0x7fe8e8155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e659c640 1 --2- 172.21.15.94:0/3723643438 >> v2:172.21.15.145:3300/0 conn(0x7fe8e814f790 0x7fe8e81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451680476 0 0) 0x7fe8e81563d0 con 0x7fe8e80c1aa0 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0003520 con 0x7fe8e80c1aa0 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552654147 0 0) 0x7fe8d0003520 con 0x7fe8e80c1aa0 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 msgr2=0x7fe8e8155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 0x7fe8e8155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 >> v2:172.21.15.145:3300/0 conn(0x7fe8e814f790 msgr2=0x7fe8e81529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 --2- 172.21.15.94:0/3723643438 >> v2:172.21.15.145:3300/0 conn(0x7fe8e814f790 0x7fe8e81529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e80c2bb0 con 0x7fe8e80c1aa0 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fe8dc001670 con 0x7fe8e80c1aa0 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 >> v1:172.21.15.94:6789/0 conn(0x7fe8e80c1aa0 legacy=0x7fe8e80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 0x7fe8d00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8d0003f30 0x7fe8d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0004d40 0x7fe8d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c2680 con 0x7fe8d0003f30 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c28d0 con 0x7fe8d0004d40 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000b9c0 con 0x7fe8e8155a40 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e5d9b640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8d0003f30 0x7fe8d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e6d9d640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0004d40 0x7fe8d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.938+0000 7fe8e659c640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 0x7fe8d00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 msgr2=0x7fe8d00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e5d9b640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e8155a40 0x7fe8d00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0004d40 msgr2=0x7fe8d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e5d9b640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0004d40 0x7fe8d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/3723643438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0003520 con 0x7fe8d0003f30 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e5d9b640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8d0003f30 0x7fe8d0004800 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc001050 tx=0x7fe8d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc192738fe5af3bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc007d70 con 0x7fe8d0003f30 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc030070 con 0x7fe8d0003f30 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.939+0000 7fe8e559a640 1 -- 172.21.15.94:0/3723643438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc005d30 con 0x7fe8d0003f30 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.940+0000 7fe8ecafb640 1 -- 172.21.15.94:0/3723643438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8d0003f30 msgr2=0x7fe8d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.940+0000 7fe8ecafb640 1 --2- 172.21.15.94:0/3723643438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8d0003f30 0x7fe8d0004800 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc001050 tx=0x7fe8d0005830 comp rx=0 tx=0).stop 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.940+0000 7fe8e6d9d640 1 -- 172.21.15.94:0/3723643438 reap_dead start 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.940+0000 7fe8ecafb640 1 -- 172.21.15.94:0/3723643438 shutdown_connections 2024-03-06T07:21:59.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.940+0000 7fe8ecafb640 1 -- 172.21.15.94:0/3723643438 >> 172.21.15.94:0/3723643438 conn(0x7fe8e80c6710 msgr2=0x7fe8e80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.940+0000 7fe8ecafb640 1 -- 172.21.15.94:0/3723643438 shutdown_connections 2024-03-06T07:21:59.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.940+0000 7fe8ecafb640 1 -- 172.21.15.94:0/3723643438 wait complete. 2024-03-06T07:21:59.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.941+0000 7fe8ecafb640 1 Processor -- start 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.941+0000 7fe8ecafb640 1 -- start start 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8ecafb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0003f30 0x7fe8e81544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8ecafb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8e814a750 0x7fe8e814abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8ecafb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e814b110 0x7fe8e81e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8ecafb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c0d60 con 0x7fe8e814a750 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8ecafb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8155520 con 0x7fe8d0003f30 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8ecafb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c5e30 con 0x7fe8e814b110 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8e814a750 0x7fe8e814abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8e814a750 0x7fe8e814abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37974/0 (socket says 172.21.15.94:37974) 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/1475299680 learned_addr learned my addr 172.21.15.94:0/1475299680 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:21:59.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e659c640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0003f30 0x7fe8e81544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e6d9d640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e814b110 0x7fe8e81e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e814b110 msgr2=0x7fe8e81e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e814b110 0x7fe8e81e9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0003f30 msgr2=0x7fe8e81544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0003f30 0x7fe8e81544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e5d9b640 1 -- 172.21.15.94:0/1475299680 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e8142a80 con 0x7fe8e814a750 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.942+0000 7fe8e659c640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0003f30 0x7fe8e81544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.943+0000 7fe8e5d9b640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8e814a750 0x7fe8e814abd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80c1f10 tx=0x7fe8dc000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.943+0000 7fe8d6ffd640 1 -- 172.21.15.94:0/1475299680 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc004bf0 con 0x7fe8e814a750 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.943+0000 7fe8d6ffd640 1 -- 172.21.15.94:0/1475299680 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc004d90 con 0x7fe8e814a750 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.943+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e814c2d0 con 0x7fe8e814a750 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.943+0000 7fe8d6ffd640 1 -- 172.21.15.94:0/1475299680 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc007270 con 0x7fe8e814a750 2024-03-06T07:21:59.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.943+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e814c570 con 0x7fe8e814a750 2024-03-06T07:21:59.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.944+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e804f330 con 0x7fe8e814a750 2024-03-06T07:21:59.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.944+0000 7fe8d6ffd640 1 -- 172.21.15.94:0/1475299680 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8dc007410 con 0x7fe8e814a750 2024-03-06T07:21:59.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.944+0000 7fe8d6ffd640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe8c806b4a0 0x7fe8c806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.945+0000 7fe8d6ffd640 1 -- 172.21.15.94:0/1475299680 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fe8dc0ab780 con 0x7fe8e814a750 2024-03-06T07:21:59.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.945+0000 7fe8e659c640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe8c806b4a0 0x7fe8c806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.945+0000 7fe8e659c640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe8c806b4a0 0x7fe8c806d960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe8e81db720 tx=0x7fe8e000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:21:59.949+0000 7fe8d6ffd640 1 -- 172.21.15.94:0/1475299680 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8dc07a7b0 con 0x7fe8e814a750 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.187+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7fe8e80c1f40 con 0x7fe8e814a750 2024-03-06T07:22:00.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.224+0000 7fe8d6ffd640 1 -- 172.21.15.94:0/1475299680 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v226) v1 ==== 156+0+0 (secure 0 0 0) 0x7fe8dc07a0b0 con 0x7fe8e814a750 2024-03-06T07:22:00.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe8c806b4a0 msgr2=0x7fe8c806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe8c806b4a0 0x7fe8c806d960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe8e81db720 tx=0x7fe8e000b040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8e814a750 msgr2=0x7fe8e814abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8e814a750 0x7fe8e814abd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fe8e80c1f10 tx=0x7fe8dc000fe0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 shutdown_connections 2024-03-06T07:22:00.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe8c806b4a0 0x7fe8c806d960 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8e814b110 0x7fe8e81e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8e814a750 0x7fe8e814abd0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 --2- 172.21.15.94:0/1475299680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8d0003f30 0x7fe8e81544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 >> 172.21.15.94:0/1475299680 conn(0x7fe8e80c6710 msgr2=0x7fe8e80a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 shutdown_connections 2024-03-06T07:22:00.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.226+0000 7fe8ecafb640 1 -- 172.21.15.94:0/1475299680 wait complete. 2024-03-06T07:22:00.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:00.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5690ba0640 1 Processor -- start 2024-03-06T07:22:00.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5690ba0640 1 -- start start 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5690ba0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f568c14ef70 0x7f568c06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5690ba0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f568c0c1820 0x7f568c06cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5690ba0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568c0c2790 con 0x7f568c0c2980 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5690ba0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c14f350 con 0x7f568c0c1820 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5690ba0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c0c1540 con 0x7f568c14ef70 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.340+0000 7f5683fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f568c0c1820 0x7f568c06cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f568ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f568c0c2980 0x7f568c154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52450/0 (socket says 172.21.15.94:52450) 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f568ad9d640 1 -- 172.21.15.94:0/4216144263 learned_addr learned my addr 172.21.15.94:0/4216144263 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f568a59c640 1 --2- 172.21.15.94:0/4216144263 >> v2:172.21.15.145:3300/0 conn(0x7f568c14ef70 0x7f568c06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f5683fff640 1 -- 172.21.15.94:0/4216144263 >> v2:172.21.15.145:3300/0 conn(0x7f568c14ef70 msgr2=0x7f568c06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f5683fff640 1 --2- 172.21.15.94:0/4216144263 >> v2:172.21.15.145:3300/0 conn(0x7f568c14ef70 0x7f568c06c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f5683fff640 1 -- 172.21.15.94:0/4216144263 >> v1:172.21.15.94:6789/0 conn(0x7f568c0c2980 legacy=0x7f568c154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f5683fff640 1 -- 172.21.15.94:0/4216144263 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0c24b0 con 0x7f568c0c1820 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.341+0000 7f5689d9b640 1 -- 172.21.15.94:0/4216144263 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503293213 0 0) 0x7f568c0c2790 con 0x7f568c0c2980 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5683fff640 1 --2- 172.21.15.94:0/4216144263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c06cad0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f568c06e5d0 tx=0x7f567402cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501d16638adebc09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5689d9b640 1 -- 172.21.15.94:0/4216144263 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567403a070 con 0x7f568c0c1820 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5689d9b640 1 -- 172.21.15.94:0/4216144263 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567402dad0 con 0x7f568c0c1820 2024-03-06T07:22:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5689d9b640 1 -- 172.21.15.94:0/4216144263 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567400b040 con 0x7f568c0c1820 2024-03-06T07:22:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 -- 172.21.15.94:0/4216144263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 msgr2=0x7f568c06cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4216144263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c06cad0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f568c06e5d0 tx=0x7f567402cd20 comp rx=0 tx=0).stop 2024-03-06T07:22:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 -- 172.21.15.94:0/4216144263 shutdown_connections 2024-03-06T07:22:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4216144263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c06cad0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4216144263 >> v2:172.21.15.145:3300/0 conn(0x7f568c14ef70 0x7f568c06c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 -- 172.21.15.94:0/4216144263 >> 172.21.15.94:0/4216144263 conn(0x7f568c0c6710 msgr2=0x7f568c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 -- 172.21.15.94:0/4216144263 shutdown_connections 2024-03-06T07:22:00.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.342+0000 7f5690ba0640 1 -- 172.21.15.94:0/4216144263 wait complete. 2024-03-06T07:22:00.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.343+0000 7f5690ba0640 1 Processor -- start 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.343+0000 7f5690ba0640 1 -- start start 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f5690ba0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c0a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f5690ba0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f568c0c2980 0x7f568c0bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f5690ba0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f568c14ef70 0x7f568c1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f5690ba0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c1554c0 con 0x7f568c0c1820 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f5690ba0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c155340 con 0x7f568c0c2980 2024-03-06T07:22:00.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f5690ba0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c0c2df0 con 0x7f568c14ef70 2024-03-06T07:22:00.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c0a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c0a6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37990/0 (socket says 172.21.15.94:37990) 2024-03-06T07:22:00.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 -- 172.21.15.94:0/4242608764 learned_addr learned my addr 172.21.15.94:0/4242608764 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:00.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 -- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f568c14ef70 msgr2=0x7f568c1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f568c14ef70 0x7f568c1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 -- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f568c0c2980 msgr2=0x7f568c0bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f568c0c2980 0x7f568c0bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.344+0000 7f568a59c640 1 -- 172.21.15.94:0/4242608764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c073120 con 0x7f568c0c1820 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.345+0000 7f568a59c640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c0a6e00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f567c00d5f0 tx=0x7f567c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.345+0000 7f5682ffd640 1 -- 172.21.15.94:0/4242608764 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c0125f0 con 0x7f568c0c1820 2024-03-06T07:22:00.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.345+0000 7f5682ffd640 1 -- 172.21.15.94:0/4242608764 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567c019070 con 0x7f568c0c1820 2024-03-06T07:22:00.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.345+0000 7f5682ffd640 1 -- 172.21.15.94:0/4242608764 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c0129b0 con 0x7f568c0c1820 2024-03-06T07:22:00.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.345+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c072360 con 0x7f568c0c1820 2024-03-06T07:22:00.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.345+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c072c00 con 0x7f568c0c1820 2024-03-06T07:22:00.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.346+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c04f330 con 0x7f568c0c1820 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.346+0000 7f5682ffd640 1 -- 172.21.15.94:0/4242608764 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f567c012b50 con 0x7f568c0c1820 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.346+0000 7f5682ffd640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f566c06b4d0 0x7f566c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.347+0000 7f5682ffd640 1 -- 172.21.15.94:0/4242608764 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f567c08a490 con 0x7f568c0c1820 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.347+0000 7f5683fff640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f566c06b4d0 0x7f566c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.347+0000 7f5683fff640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f566c06b4d0 0x7f566c06d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5674037000 tx=0x7f567402f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.351+0000 7f5682ffd640 1 -- 172.21.15.94:0/4242608764 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f567c0594c0 con 0x7f568c0c1820 2024-03-06T07:22:00.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.521+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f568c0c1110 con 0x7f568c0c1820 2024-03-06T07:22:00.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.524+0000 7f5682ffd640 1 -- 172.21.15.94:0/4242608764 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f567c058dc0 con 0x7f568c0c1820 2024-03-06T07:22:00.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f566c06b4d0 msgr2=0x7f566c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f566c06b4d0 0x7f566c06d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5674037000 tx=0x7f567402f000 comp rx=0 tx=0).stop 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 msgr2=0x7f568c0a6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c0a6e00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f567c00d5f0 tx=0x7f567c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 shutdown_connections 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f566c06b4d0 0x7f566c06d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f568c14ef70 0x7f568c1494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f568c0c2980 0x7f568c0bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 --2- 172.21.15.94:0/4242608764 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f568c0c1820 0x7f568c0a6e00 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 >> 172.21.15.94:0/4242608764 conn(0x7f568c0c6710 msgr2=0x7f568c152c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 shutdown_connections 2024-03-06T07:22:00.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.526+0000 7f5690ba0640 1 -- 172.21.15.94:0/4242608764 wait complete. 2024-03-06T07:22:00.539 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:00.539 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:00.582 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:00.634 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:00.635 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:22:00.635 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:00.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:22:00.650 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:22:00.650 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:00.667 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:22:00.667 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:22:00.667 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:00.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.789+0000 7fb7c73f4640 1 Processor -- start 2024-03-06T07:22:00.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.789+0000 7fb7c73f4640 1 -- start start 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c73f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb7c00c1aa0 0x7fb7c01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c73f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb7c0155a40 0x7fb7c0155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c73f4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00c28d0 con 0x7fb7c014f790 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c73f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c00c2680 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c73f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c00c2bb0 con 0x7fb7c0155a40 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c498f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb7c00c1aa0 0x7fb7c01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c5190640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb7c014f790 0x7fb7c00c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52462/0 (socket says 172.21.15.94:52462) 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c5190640 1 -- 172.21.15.94:0/2312630266 learned_addr learned my addr 172.21.15.94:0/2312630266 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c5991640 1 --2- 172.21.15.94:0/2312630266 >> v2:172.21.15.145:3300/0 conn(0x7fb7c0155a40 0x7fb7c0155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c498f640 1 -- 172.21.15.94:0/2312630266 >> v2:172.21.15.145:3300/0 conn(0x7fb7c0155a40 msgr2=0x7fb7c0155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c498f640 1 --2- 172.21.15.94:0/2312630266 >> v2:172.21.15.145:3300/0 conn(0x7fb7c0155a40 0x7fb7c0155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c498f640 1 -- 172.21.15.94:0/2312630266 >> v1:172.21.15.94:6789/0 conn(0x7fb7c014f790 legacy=0x7fb7c00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7c498f640 1 -- 172.21.15.94:0/2312630266 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00c2e90 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.790+0000 7fb7b7fff640 1 -- 172.21.15.94:0/2312630266 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626077488 0 0) 0x7fb7c00c28d0 con 0x7fb7c014f790 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.791+0000 7fb7c498f640 1 --2- 172.21.15.94:0/2312630266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c01529d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb7c006c960 tx=0x7fb7b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b7c964cb2f4a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.791+0000 7fb7b7fff640 1 -- 172.21.15.94:0/2312630266 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0035020 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.791+0000 7fb7b7fff640 1 -- 172.21.15.94:0/2312630266 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b0031830 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.791+0000 7fb7b7fff640 1 -- 172.21.15.94:0/2312630266 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0031c00 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2312630266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 msgr2=0x7fb7c01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2312630266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c01529d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb7c006c960 tx=0x7fb7b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2312630266 shutdown_connections 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2312630266 >> v2:172.21.15.145:3300/0 conn(0x7fb7c0155a40 0x7fb7c0155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2312630266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c01529d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2312630266 >> 172.21.15.94:0/2312630266 conn(0x7fb7c00c6710 msgr2=0x7fb7c00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2312630266 shutdown_connections 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.792+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2312630266 wait complete. 2024-03-06T07:22:00.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.793+0000 7fb7c73f4640 1 Processor -- start 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.793+0000 7fb7c73f4640 1 -- start start 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c73f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c00bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c73f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7c014f790 0x7fb7c014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c73f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7c0155a40 0x7fb7c01e8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c73f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00c0d60 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c73f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c0156390 con 0x7fb7c014f790 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c73f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00c5e30 con 0x7fb7c0155a40 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c00bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c00bf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38000/0 (socket says 172.21.15.94:38000) 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 -- 172.21.15.94:0/2082367952 learned_addr learned my addr 172.21.15.94:0/2082367952 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 -- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7c0155a40 msgr2=0x7fb7c01e8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7c0155a40 0x7fb7c01e8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 -- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7c014f790 msgr2=0x7fb7c014be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7c014f790 0x7fb7c014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 -- 172.21.15.94:0/2082367952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c014c9d0 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c5190640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c00bf770 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c4d0 tx=0x7fb7bc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7b5ffb640 1 -- 172.21.15.94:0/2082367952 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc018070 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7b5ffb640 1 -- 172.21.15.94:0/2082367952 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc007500 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c01ebc40 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.794+0000 7fb7b5ffb640 1 -- 172.21.15.94:0/2082367952 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc004a20 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.795+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c01ec200 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.796+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c004f330 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.796+0000 7fb7b5ffb640 1 -- 172.21.15.94:0/2082367952 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7bc01b070 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.796+0000 7fb7b5ffb640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.796+0000 7fb7b5ffb640 1 -- 172.21.15.94:0/2082367952 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb7bc089360 con 0x7fb7c00c1aa0 2024-03-06T07:22:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.797+0000 7fb7c498f640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.797+0000 7fb7c498f640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7a406b4d0 0x7fb7a406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0036000 tx=0x7fb7b0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:00.800+0000 7fb7b5ffb640 1 -- 172.21.15.94:0/2082367952 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7bc058390 con 0x7fb7c00c1aa0 2024-03-06T07:22:01.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.007+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7c00c1f40 con 0x7fb7c00c1aa0 2024-03-06T07:22:01.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.007+0000 7fb7b5ffb640 1 -- 172.21.15.94:0/2082367952 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3708 (secure 0 0 0) 0x7fb7bc057c90 con 0x7fb7c00c1aa0 2024-03-06T07:22:01.009 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:01.009 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6945,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3090166857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3090166857},{"type":"v1","addr":"172.21.15.94:6834","nonce":3090166857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":35888,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3798967274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3798967274},{"type":"v1","addr":"172.21.15.94:6835","nonce":3798967274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":36014,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1145122167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1145122167},{"type":"v1","addr":"172.21.15.145:6836","nonce":1145122167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":36089,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/4015623207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":4015623207},{"type":"v1","addr":"172.21.15.145:6837","nonce":4015623207}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[]} 2024-03-06T07:22:01.010 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 84 2024-03-06T07:22:01.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7a406b4d0 msgr2=0x7fb7a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7a406b4d0 0x7fb7a406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0036000 tx=0x7fb7b0030000 comp rx=0 tx=0).stop 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 msgr2=0x7fb7c00bf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c00bf770 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c4d0 tx=0x7fb7bc00c990 comp rx=0 tx=0).stop 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 shutdown_connections 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7c0155a40 0x7fb7c01e8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7c014f790 0x7fb7c014be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 --2- 172.21.15.94:0/2082367952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00c1aa0 0x7fb7c00bf770 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 >> 172.21.15.94:0/2082367952 conn(0x7fb7c00c6710 msgr2=0x7fb7c014d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 shutdown_connections 2024-03-06T07:22:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.009+0000 7fb7c73f4640 1 -- 172.21.15.94:0/2082367952 wait complete. 2024-03-06T07:22:01.022 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:01.022 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:01.022 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:01.022 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.023 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:01.023 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:01.023 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:01.023 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.023 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:01.023 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:01.023 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:01.024 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.024 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:01.024 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:01.024 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:01.024 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:01.032 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:01.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:01.033 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:01.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:01.036 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:01.036 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:01.037 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:01.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:01.074 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:01.074 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.075 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:01.075 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.080 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:01.080 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.080 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:01.080 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:01.081 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:01.149 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:22:01.150 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:22:01.163 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:22:01.163 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:22:01.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.184+0000 7f77620c3640 1 Processor -- start 2024-03-06T07:22:01.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.184+0000 7f77620c3640 1 -- start start 2024-03-06T07:22:01.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f77620c3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f775c0c1150 0x7f775c06b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f77620c3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f775c0c53d0 0x7f775c0c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f77620c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c159520 0x7f775c1552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f77620c3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775c159090 con 0x7f775c0c53d0 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f77620c3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775c13e420 con 0x7f775c0c1150 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f77620c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c158a40 con 0x7f775c159520 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f775c0c53d0 0x7f775c0c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f775c0c53d0 0x7f775c0c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38044/0 (socket says 172.21.15.94:38044) 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 -- 172.21.15.94:0/1452077141 learned_addr learned my addr 172.21.15.94:0/1452077141 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 -- 172.21.15.94:0/1452077141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c159520 msgr2=0x7f775c1552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775b7fe640 1 --2- 172.21.15.94:0/1452077141 >> v2:172.21.15.145:3300/0 conn(0x7f775c0c1150 0x7f775c06b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 --2- 172.21.15.94:0/1452077141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c159520 0x7f775c1552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 -- 172.21.15.94:0/1452077141 >> v2:172.21.15.145:3300/0 conn(0x7f775c0c1150 msgr2=0x7f775c06b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 --2- 172.21.15.94:0/1452077141 >> v2:172.21.15.145:3300/0 conn(0x7f775c0c1150 0x7f775c06b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775affd640 1 -- 172.21.15.94:0/1452077141 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c158d20 con 0x7f775c0c53d0 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.185+0000 7f775b7fe640 1 --2- 172.21.15.94:0/1452077141 >> v2:172.21.15.145:3300/0 conn(0x7f775c0c1150 0x7f775c06b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:01.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f775affd640 1 --2- 172.21.15.94:0/1452077141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c0c53d0 0x7f775c0c8950 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f775c0c7f00 tx=0x7f775002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a586b2204618943a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f775a7fc640 1 -- 172.21.15.94:0/1452077141 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750038070 con 0x7f775c0c53d0 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f775a7fc640 1 -- 172.21.15.94:0/1452077141 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775002f050 con 0x7f775c0c53d0 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f775a7fc640 1 -- 172.21.15.94:0/1452077141 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750031970 con 0x7f775c0c53d0 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 -- 172.21.15.94:0/1452077141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c0c53d0 msgr2=0x7f775c0c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 --2- 172.21.15.94:0/1452077141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c0c53d0 0x7f775c0c8950 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f775c0c7f00 tx=0x7f775002b7f0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 -- 172.21.15.94:0/1452077141 shutdown_connections 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 --2- 172.21.15.94:0/1452077141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c159520 0x7f775c1552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 --2- 172.21.15.94:0/1452077141 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c0c53d0 0x7f775c0c8950 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 --2- 172.21.15.94:0/1452077141 >> v2:172.21.15.145:3300/0 conn(0x7f775c0c1150 0x7f775c06b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 -- 172.21.15.94:0/1452077141 >> 172.21.15.94:0/1452077141 conn(0x7f775c0ac680 msgr2=0x7f775c0aca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 -- 172.21.15.94:0/1452077141 shutdown_connections 2024-03-06T07:22:01.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.186+0000 7f77620c3640 1 -- 172.21.15.94:0/1452077141 wait complete. 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.188+0000 7f77620c3640 1 Processor -- start 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.188+0000 7f77620c3640 1 -- start start 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f775c0c1150 0x7f775c136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c0c53d0 0x7f775c1ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c159520 0x7f775c0c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c13e7d0 con 0x7f775c159520 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c0c1530 con 0x7f775c0c1150 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c13e950 con 0x7f775c0c53d0 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c159520 0x7f775c0c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c159520 0x7f775c0c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38054/0 (socket says 172.21.15.94:38054) 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 -- 172.21.15.94:0/2978911337 learned_addr learned my addr 172.21.15.94:0/2978911337 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:01.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775b7fe640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f775c0c1150 0x7f775c136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775affd640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c0c53d0 0x7f775c1ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 -- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c0c53d0 msgr2=0x7f775c1ee2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c0c53d0 0x7f775c1ee2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 -- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f775c0c1150 msgr2=0x7f775c136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f775c0c1150 0x7f775c136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 -- 172.21.15.94:0/2978911337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c0c2350 con 0x7f775c159520 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f775bfff640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c159520 0x7f775c0c41c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f775400ba10 tx=0x7f775400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f7743fff640 1 -- 172.21.15.94:0/2978911337 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754019070 con 0x7f775c159520 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f7743fff640 1 -- 172.21.15.94:0/2978911337 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775400cb60 con 0x7f775c159520 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f7743fff640 1 -- 172.21.15.94:0/2978911337 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754014440 con 0x7f775c159520 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c1eec70 con 0x7f775c159520 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.189+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c137230 con 0x7f775c159520 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.191+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c04f330 con 0x7f775c159520 2024-03-06T07:22:01.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.191+0000 7f7743fff640 1 -- 172.21.15.94:0/2978911337 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f775401c070 con 0x7f775c159520 2024-03-06T07:22:01.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.191+0000 7f7743fff640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774806b4d0 0x7f774806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.192+0000 7f7743fff640 1 -- 172.21.15.94:0/2978911337 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f7754089c70 con 0x7f775c159520 2024-03-06T07:22:01.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.192+0000 7f775b7fe640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774806b4d0 0x7f774806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.192+0000 7f775b7fe640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774806b4d0 0x7f774806d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f775c158780 tx=0x7f774c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.196+0000 7f7743fff640 1 -- 172.21.15.94:0/2978911337 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7754058650 con 0x7f775c159520 2024-03-06T07:22:01.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:01.388+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f775c0ca3e0 con 0x7f775c159520 2024-03-06T07:22:02.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.237+0000 7f7743fff640 1 -- 172.21.15.94:0/2978911337 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f775405c6f0 con 0x7f775c159520 2024-03-06T07:22:02.239 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774806b4d0 msgr2=0x7f774806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774806b4d0 0x7f774806d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f775c158780 tx=0x7f774c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c159520 msgr2=0x7f775c0c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c159520 0x7f775c0c41c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f775400ba10 tx=0x7f775400bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 shutdown_connections 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774806b4d0 0x7f774806d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f775c159520 0x7f775c0c41c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f775c0c53d0 0x7f775c1ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 --2- 172.21.15.94:0/2978911337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f775c0c1150 0x7f775c136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 >> 172.21.15.94:0/2978911337 conn(0x7f775c0ac680 msgr2=0x7f775c0c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 shutdown_connections 2024-03-06T07:22:02.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.241+0000 7f77620c3640 1 -- 172.21.15.94:0/2978911337 wait complete. 2024-03-06T07:22:02.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.360+0000 7f0cdd8cf640 1 Processor -- start 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.360+0000 7f0cdd8cf640 1 -- start start 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cdd8cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd80c5480 0x7f0cd8155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cdd8cf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0cd81586f0 0x7f0cd8158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cdd8cf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd80c5cc0 con 0x7f0cd80c5860 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cdd8cf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd806df40 con 0x7f0cd81586f0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cdd8cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd806fdb0 con 0x7f0cd80c5480 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cd77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0cd81586f0 0x7f0cd8158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cd67fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0cd80c5860 0x7f0cd806f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52484/0 (socket says 172.21.15.94:52484) 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cd67fc640 1 -- 172.21.15.94:0/2524025005 learned_addr learned my addr 172.21.15.94:0/2524025005 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cd77fe640 1 -- 172.21.15.94:0/2524025005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd80c5480 msgr2=0x7f0cd8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cd77fe640 1 --2- 172.21.15.94:0/2524025005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd80c5480 0x7f0cd8155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cd77fe640 1 -- 172.21.15.94:0/2524025005 >> v1:172.21.15.94:6789/0 conn(0x7f0cd80c5860 legacy=0x7f0cd806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.362+0000 7f0cd77fe640 1 -- 172.21.15.94:0/2524025005 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd81591c0 con 0x7f0cd81586f0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.361+0000 7f0cd5ffb640 1 -- 172.21.15.94:0/2524025005 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033083568 0 0) 0x7f0cd80c5cc0 con 0x7f0cd80c5860 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.362+0000 7f0cd77fe640 1 --2- 172.21.15.94:0/2524025005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd81586f0 0x7f0cd8158b80 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00097a0 tx=0x7f0cd002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf9e88e623fdcb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.362+0000 7f0cd5ffb640 1 -- 172.21.15.94:0/2524025005 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00315a0 con 0x7f0cd81586f0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.362+0000 7f0cd5ffb640 1 -- 172.21.15.94:0/2524025005 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd0037070 con 0x7f0cd81586f0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.362+0000 7f0cd5ffb640 1 -- 172.21.15.94:0/2524025005 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0031960 con 0x7f0cd81586f0 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/2524025005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd81586f0 msgr2=0x7f0cd8158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/2524025005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd81586f0 0x7f0cd8158b80 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00097a0 tx=0x7f0cd002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/2524025005 shutdown_connections 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/2524025005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd81586f0 0x7f0cd8158b80 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/2524025005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd80c5480 0x7f0cd8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/2524025005 >> 172.21.15.94:0/2524025005 conn(0x7f0cd806b1a0 msgr2=0x7f0cd806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/2524025005 shutdown_connections 2024-03-06T07:22:02.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.363+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/2524025005 wait complete. 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.364+0000 7f0cdd8cf640 1 Processor -- start 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.364+0000 7f0cdd8cf640 1 -- start start 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cdd8cf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd80c5480 0x7f0cd8153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cdd8cf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cd80c5860 0x7f0cd81eadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cdd8cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd81586f0 0x7f0cd81eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cdd8cf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8155680 con 0x7f0cd80c5480 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cdd8cf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd81553a0 con 0x7f0cd80c5860 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cdd8cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8152a20 con 0x7f0cd81586f0 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd80c5480 0x7f0cd8153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd80c5480 0x7f0cd8153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38068/0 (socket says 172.21.15.94:38068) 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 -- 172.21.15.94:0/3859598320 learned_addr learned my addr 172.21.15.94:0/3859598320 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 -- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd81586f0 msgr2=0x7f0cd81eb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd81586f0 0x7f0cd81eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd67fc640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cd80c5860 0x7f0cd81eadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 -- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cd80c5860 msgr2=0x7f0cd81eadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cd80c5860 0x7f0cd81eadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 -- 172.21.15.94:0/3859598320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd81ee350 con 0x7f0cd80c5480 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd67fc640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cd80c5860 0x7f0cd81eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.365+0000 7f0cd6ffd640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd80c5480 0x7f0cd8153680 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800ba50 tx=0x7f0cc800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.366+0000 7f0cbffff640 1 -- 172.21.15.94:0/3859598320 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8019070 con 0x7f0cd80c5480 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.366+0000 7f0cbffff640 1 -- 172.21.15.94:0/3859598320 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc800cc00 con 0x7f0cd80c5480 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.366+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd80ad8c0 con 0x7f0cd80c5480 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.366+0000 7f0cbffff640 1 -- 172.21.15.94:0/3859598320 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8014440 con 0x7f0cd80c5480 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.366+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd81ee8f0 con 0x7f0cd80c5480 2024-03-06T07:22:02.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.367+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8069460 con 0x7f0cd80c5480 2024-03-06T07:22:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.367+0000 7f0cbffff640 1 -- 172.21.15.94:0/3859598320 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cc801c070 con 0x7f0cd80c5480 2024-03-06T07:22:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.367+0000 7f0cbffff640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cc406b4d0 0x7f0cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.368+0000 7f0cbffff640 1 -- 172.21.15.94:0/3859598320 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0cc8089d90 con 0x7f0cd80c5480 2024-03-06T07:22:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.368+0000 7f0cd67fc640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cc406b4d0 0x7f0cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.368+0000 7f0cd67fc640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cc406b4d0 0x7f0cc406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80c6590 tx=0x7f0ccc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.372+0000 7f0cbffff640 1 -- 172.21.15.94:0/3859598320 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cd8069460 con 0x7f0cd80c5480 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.608+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0cd804f330 con 0x7f0cd80c5480 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.609+0000 7f0cbffff640 1 -- 172.21.15.94:0/3859598320 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f0cc8058e60 con 0x7f0cd80c5480 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.611 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:02.612 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.613 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.614 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.615 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.616 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:22:02.617 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.618 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.619 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:22:02.620 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:22:02.621 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:02.622 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:02.623 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:22:02.624 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cc406b4d0 msgr2=0x7f0cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cc406b4d0 0x7f0cc406d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80c6590 tx=0x7f0ccc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd80c5480 msgr2=0x7f0cd8153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd80c5480 0x7f0cd8153680 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800ba50 tx=0x7f0cc800bf10 comp rx=0 tx=0).stop 2024-03-06T07:22:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 shutdown_connections 2024-03-06T07:22:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cc406b4d0 0x7f0cc406d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cd81586f0 0x7f0cd81eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cd80c5860 0x7f0cd81eadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 --2- 172.21.15.94:0/3859598320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cd80c5480 0x7f0cd8153680 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 >> 172.21.15.94:0/3859598320 conn(0x7f0cd806b1a0 msgr2=0x7f0cd806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 shutdown_connections 2024-03-06T07:22:02.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.611+0000 7f0cdd8cf640 1 -- 172.21.15.94:0/3859598320 wait complete. 2024-03-06T07:22:02.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:02.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.728+0000 7fb5a3015640 1 Processor -- start 2024-03-06T07:22:02.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.728+0000 7fb5a3015640 1 -- start start 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a3015640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c1320 0x7fb59c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a3015640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb59c0c6000 0x7fb59c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a3015640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c153a60 con 0x7fb59c150580 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a3015640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59c153d40 con 0x7fb59c0c6000 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a3015640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c06c2c0 con 0x7fb59c0c1320 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a15b2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb59c0c6000 0x7fb59c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a0db1640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb59c150580 0x7fb59c06dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52490/0 (socket says 172.21.15.94:52490) 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a0db1640 1 -- 172.21.15.94:0/3300303145 learned_addr learned my addr 172.21.15.94:0/3300303145 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a15b2640 1 -- 172.21.15.94:0/3300303145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c1320 msgr2=0x7fb59c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a15b2640 1 --2- 172.21.15.94:0/3300303145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c1320 0x7fb59c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a15b2640 1 -- 172.21.15.94:0/3300303145 >> v1:172.21.15.94:6789/0 conn(0x7fb59c150580 legacy=0x7fb59c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb5a15b2640 1 -- 172.21.15.94:0/3300303145 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c06c5a0 con 0x7fb59c0c6000 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.729+0000 7fb59b7fe640 1 -- 172.21.15.94:0/3300303145 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157390570 0 0) 0x7fb59c153a60 con 0x7fb59c150580 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.730+0000 7fb5a15b2640 1 --2- 172.21.15.94:0/3300303145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c0c6000 0x7fb59c156bf0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb594009d90 tx=0x7fb59402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5b1200e08b1eac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.730+0000 7fb59b7fe640 1 -- 172.21.15.94:0/3300303145 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594039070 con 0x7fb59c0c6000 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.730+0000 7fb59b7fe640 1 -- 172.21.15.94:0/3300303145 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59402cbc0 con 0x7fb59c0c6000 2024-03-06T07:22:02.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.730+0000 7fb59b7fe640 1 -- 172.21.15.94:0/3300303145 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594034440 con 0x7fb59c0c6000 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 -- 172.21.15.94:0/3300303145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c0c6000 msgr2=0x7fb59c156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 --2- 172.21.15.94:0/3300303145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c0c6000 0x7fb59c156bf0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb594009d90 tx=0x7fb59402be10 comp rx=0 tx=0).stop 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 -- 172.21.15.94:0/3300303145 shutdown_connections 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 --2- 172.21.15.94:0/3300303145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c0c6000 0x7fb59c156bf0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 --2- 172.21.15.94:0/3300303145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c1320 0x7fb59c1566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 -- 172.21.15.94:0/3300303145 >> 172.21.15.94:0/3300303145 conn(0x7fb59c0c6710 msgr2=0x7fb59c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 -- 172.21.15.94:0/3300303145 shutdown_connections 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.731+0000 7fb5a3015640 1 -- 172.21.15.94:0/3300303145 wait complete. 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.732+0000 7fb5a3015640 1 Processor -- start 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.732+0000 7fb5a3015640 1 -- start start 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a3015640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb59c0c1320 0x7fb59c1e98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a3015640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c6000 0x7fb59c1474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a3015640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c150580 0x7fb59c1ea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a3015640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c06e5a0 con 0x7fb59c150580 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a3015640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c06e420 con 0x7fb59c0c1320 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a3015640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0c0c40 con 0x7fb59c0c6000 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c150580 0x7fb59c1ea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c150580 0x7fb59c1ea2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38086/0 (socket says 172.21.15.94:38086) 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 -- 172.21.15.94:0/4095998479 learned_addr learned my addr 172.21.15.94:0/4095998479 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a0db1640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb59c0c1320 0x7fb59c1e98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 -- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c6000 msgr2=0x7fb59c1474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c6000 0x7fb59c1474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 -- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb59c0c1320 msgr2=0x7fb59c1e98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb59c0c1320 0x7fb59c1e98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 -- 172.21.15.94:0/4095998479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c1e9de0 con 0x7fb59c150580 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a15b2640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c150580 0x7fb59c1ea2c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb594009eb0 tx=0x7fb594004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5997fa640 1 -- 172.21.15.94:0/4095998479 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594044020 con 0x7fb59c150580 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5997fa640 1 -- 172.21.15.94:0/4095998479 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59403f470 con 0x7fb59c150580 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c154e80 con 0x7fb59c150580 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.733+0000 7fb5997fa640 1 -- 172.21.15.94:0/4095998479 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594039040 con 0x7fb59c150580 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.734+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c1ec840 con 0x7fb59c150580 2024-03-06T07:22:02.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.735+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59c04f330 con 0x7fb59c150580 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.735+0000 7fb5997fa640 1 -- 172.21.15.94:0/4095998479 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb594047020 con 0x7fb59c150580 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.735+0000 7fb5997fa640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb58806b4d0 0x7fb58806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.735+0000 7fb5997fa640 1 -- 172.21.15.94:0/4095998479 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb5940acbf0 con 0x7fb59c150580 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.735+0000 7fb5a0db1640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb58806b4d0 0x7fb58806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.736+0000 7fb5a0db1640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb58806b4d0 0x7fb58806d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb58c000f60 tx=0x7fb58c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.740+0000 7fb5997fa640 1 -- 172.21.15.94:0/4095998479 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb59404e030 con 0x7fb59c150580 2024-03-06T07:22:02.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.913+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb59c0c0780 con 0x7fb59c150580 2024-03-06T07:22:02.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.913+0000 7fb5997fa640 1 -- 172.21.15.94:0/4095998479 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+13166 (secure 0 0 0) 0x7fb59407bde0 con 0x7fb59c150580 2024-03-06T07:22:02.915 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:02.915 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":104,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:02.237335+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb58806b4d0 msgr2=0x7fb58806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb58806b4d0 0x7fb58806d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb58c000f60 tx=0x7fb58c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c150580 msgr2=0x7fb59c1ea2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c150580 0x7fb59c1ea2c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb594009eb0 tx=0x7fb594004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 shutdown_connections 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb58806b4d0 0x7fb58806d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb59c150580 0x7fb59c1ea2c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb59c0c6000 0x7fb59c1474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 --2- 172.21.15.94:0/4095998479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb59c0c1320 0x7fb59c1e98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 >> 172.21.15.94:0/4095998479 conn(0x7fb59c0c6710 msgr2=0x7fb59c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 shutdown_connections 2024-03-06T07:22:02.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:02.916+0000 7fb5a3015640 1 -- 172.21.15.94:0/4095998479 wait complete. 2024-03-06T07:22:02.929 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:02.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.030+0000 7ff411f65640 1 Processor -- start 2024-03-06T07:22:03.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.030+0000 7ff411f65640 1 -- start start 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff411f65640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff40c0ab310 0x7ff40c0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff411f65640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40c154330 con 0x7ff40c1538c0 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff411f65640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40c153dc0 con 0x7ff40c0a97c0 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff411f65640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c0c46a0 con 0x7ff40c0ab310 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff40b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff40c0ab310 0x7ff40c0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff40affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff40c1538c0 0x7ff40c1494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52502/0 (socket says 172.21.15.94:52502) 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff40affd640 1 -- 172.21.15.94:0/2907245234 learned_addr learned my addr 172.21.15.94:0/2907245234 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:03.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff40b7fe640 1 -- 172.21.15.94:0/2907245234 >> v1:172.21.15.145:6789/0 conn(0x7ff40c0a97c0 legacy=0x7ff40c0c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff40b7fe640 1 -- 172.21.15.94:0/2907245234 >> v1:172.21.15.94:6789/0 conn(0x7ff40c1538c0 legacy=0x7ff40c1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.031+0000 7ff40b7fe640 1 -- 172.21.15.94:0/2907245234 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c0c5d30 con 0x7ff40c0ab310 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.032+0000 7ff40b7fe640 1 --2- 172.21.15.94:0/2907245234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c0ab6f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0097a0 tx=0x7ff3fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dbe9e4f245956b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.032+0000 7ff40a7fc640 1 -- 172.21.15.94:0/2907245234 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc035020 con 0x7ff40c0ab310 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.032+0000 7ff40a7fc640 1 -- 172.21.15.94:0/2907245234 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc031830 con 0x7ff40c0ab310 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.032+0000 7ff40a7fc640 1 -- 172.21.15.94:0/2907245234 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc031c00 con 0x7ff40c0ab310 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.033+0000 7ff411f65640 1 -- 172.21.15.94:0/2907245234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 msgr2=0x7ff40c0ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.033+0000 7ff411f65640 1 --2- 172.21.15.94:0/2907245234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c0ab6f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0097a0 tx=0x7ff3fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.033+0000 7ff411f65640 1 -- 172.21.15.94:0/2907245234 shutdown_connections 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.033+0000 7ff411f65640 1 --2- 172.21.15.94:0/2907245234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c0ab6f0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.033+0000 7ff411f65640 1 -- 172.21.15.94:0/2907245234 >> 172.21.15.94:0/2907245234 conn(0x7ff40c0c6710 msgr2=0x7ff40c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.033+0000 7ff411f65640 1 -- 172.21.15.94:0/2907245234 shutdown_connections 2024-03-06T07:22:03.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.033+0000 7ff411f65640 1 -- 172.21.15.94:0/2907245234 wait complete. 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.034+0000 7ff411f65640 1 Processor -- start 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.034+0000 7ff411f65640 1 -- start start 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0a97c0 0x7ff40c1eed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c06d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c1538c0 0x7ff40c1ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c0abc30 con 0x7ff40c0ab310 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c0c62f0 con 0x7ff40c0a97c0 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c149bb0 con 0x7ff40c1538c0 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c06d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c06d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38110/0 (socket says 172.21.15.94:38110) 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 -- 172.21.15.94:0/864286868 learned_addr learned my addr 172.21.15.94:0/864286868 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40b7fe640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0a97c0 0x7ff40c1eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40bfff640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c1538c0 0x7ff40c1ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 -- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c1538c0 msgr2=0x7ff40c1ec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c1538c0 0x7ff40c1ec180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 -- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0a97c0 msgr2=0x7ff40c1eed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0a97c0 0x7ff40c1eed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 -- 172.21.15.94:0/864286868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c1507c0 con 0x7ff40c0ab310 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40b7fe640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0a97c0 0x7ff40c1eed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40bfff640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c1538c0 0x7ff40c1ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff40affd640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c06d6e0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c3c0 tx=0x7ff40000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff3f3fff640 1 -- 172.21.15.94:0/864286868 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400017070 con 0x7ff40c0ab310 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff3f3fff640 1 -- 172.21.15.94:0/864286868 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff400005ce0 con 0x7ff40c0ab310 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40c14da30 con 0x7ff40c0ab310 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff3f3fff640 1 -- 172.21.15.94:0/864286868 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4000072d0 con 0x7ff40c0ab310 2024-03-06T07:22:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.035+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40c1ee280 con 0x7ff40c0ab310 2024-03-06T07:22:03.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.036+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40c04f330 con 0x7ff40c0ab310 2024-03-06T07:22:03.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.037+0000 7ff3f3fff640 1 -- 172.21.15.94:0/864286868 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff40001a070 con 0x7ff40c0ab310 2024-03-06T07:22:03.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.037+0000 7ff3f3fff640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b500 0x7ff3f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.037+0000 7ff3f3fff640 1 -- 172.21.15.94:0/864286868 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff400089280 con 0x7ff40c0ab310 2024-03-06T07:22:03.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.037+0000 7ff40b7fe640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b500 0x7ff3f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.038+0000 7ff40b7fe640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b500 0x7ff3f806d9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc036000 tx=0x7ff3fc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.041+0000 7ff3f3fff640 1 -- 172.21.15.94:0/864286868 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff400057be0 con 0x7ff40c0ab310 2024-03-06T07:22:03.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.211+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff40c0c6c90 con 0x7ff40c0ab310 2024-03-06T07:22:03.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.212+0000 7ff3f3fff640 1 -- 172.21.15.94:0/864286868 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff40005bc80 con 0x7ff40c0ab310 2024-03-06T07:22:03.216 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:22:03.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b500 msgr2=0x7ff3f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b500 0x7ff3f806d9c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc036000 tx=0x7ff3fc008030 comp rx=0 tx=0).stop 2024-03-06T07:22:03.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 msgr2=0x7ff40c06d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c06d6e0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c3c0 tx=0x7ff40000c880 comp rx=0 tx=0).stop 2024-03-06T07:22:03.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 shutdown_connections 2024-03-06T07:22:03.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff3f806b500 0x7ff3f806d9c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff40c1538c0 0x7ff40c1ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff40c0ab310 0x7ff40c06d6e0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 --2- 172.21.15.94:0/864286868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff40c0a97c0 0x7ff40c1eed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.213+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 >> 172.21.15.94:0/864286868 conn(0x7ff40c0c6710 msgr2=0x7ff40c0a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.214+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 shutdown_connections 2024-03-06T07:22:03.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.214+0000 7ff411f65640 1 -- 172.21.15.94:0/864286868 wait complete. 2024-03-06T07:22:03.226 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:03.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.324+0000 7f7ebb681640 1 Processor -- start 2024-03-06T07:22:03.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.324+0000 7f7ebb681640 1 -- start start 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7ebb681640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7eb40aa180 0x7f7eb40c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7ebb681640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7eb41550c0 0x7f7eb4155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7ebb681640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb40aa660 con 0x7f7eb414ec00 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7ebb681640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb40c4610 con 0x7f7eb41550c0 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7ebb681640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb40aa940 con 0x7f7eb40aa180 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb9c1e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7eb41550c0 0x7f7eb4155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb9c1e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7eb41550c0 0x7f7eb4155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38114/0 (socket says 172.21.15.94:38114) 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/1014305775 learned_addr learned my addr 172.21.15.94:0/1014305775 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb941d640 1 --2- 172.21.15.94:0/1014305775 >> v2:172.21.15.145:3300/0 conn(0x7f7eb40aa180 0x7f7eb40c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/1014305775 >> v2:172.21.15.145:3300/0 conn(0x7f7eb40aa180 msgr2=0x7f7eb40c31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb9c1e640 1 --2- 172.21.15.94:0/1014305775 >> v2:172.21.15.145:3300/0 conn(0x7f7eb40aa180 0x7f7eb40c31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/1014305775 >> v1:172.21.15.145:6789/0 conn(0x7f7eb414ec00 legacy=0x7f7eb414f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.325+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/1014305775 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb40c4860 con 0x7f7eb41550c0 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.326+0000 7f7eb9c1e640 1 --2- 172.21.15.94:0/1014305775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb4155550 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7eb40c3fc0 tx=0x7f7eac02dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244d4d588ced4d9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.326+0000 7f7eabfff640 1 -- 172.21.15.94:0/1014305775 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac02e970 con 0x7f7eb41550c0 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.326+0000 7f7eabfff640 1 -- 172.21.15.94:0/1014305775 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac02eb10 con 0x7f7eb41550c0 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.326+0000 7f7eabfff640 1 -- 172.21.15.94:0/1014305775 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac009600 con 0x7f7eb41550c0 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 -- 172.21.15.94:0/1014305775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 msgr2=0x7f7eb4155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 --2- 172.21.15.94:0/1014305775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb4155550 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7eb40c3fc0 tx=0x7f7eac02dfd0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 -- 172.21.15.94:0/1014305775 shutdown_connections 2024-03-06T07:22:03.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 --2- 172.21.15.94:0/1014305775 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb4155550 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 --2- 172.21.15.94:0/1014305775 >> v2:172.21.15.145:3300/0 conn(0x7f7eb40aa180 0x7f7eb40c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 -- 172.21.15.94:0/1014305775 >> 172.21.15.94:0/1014305775 conn(0x7f7eb40c6710 msgr2=0x7f7eb40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 -- 172.21.15.94:0/1014305775 shutdown_connections 2024-03-06T07:22:03.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.327+0000 7f7ebb681640 1 -- 172.21.15.94:0/1014305775 wait complete. 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.328+0000 7f7ebb681640 1 Processor -- start 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.328+0000 7f7ebb681640 1 -- start start 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ebb681640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40aa180 0x7f7eb4146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ebb681640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb414ec00 0x7f7eb41469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ebb681640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb41ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ebb681640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40c3870 con 0x7f7eb41550c0 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ebb681640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40c36f0 con 0x7f7eb414ec00 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ebb681640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40c1310 con 0x7f7eb40aa180 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb41ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb41ea3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38124/0 (socket says 172.21.15.94:38124) 2024-03-06T07:22:03.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/3556967185 learned_addr learned my addr 172.21.15.94:0/3556967185 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb941d640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40aa180 0x7f7eb4146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb8c1c640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb414ec00 0x7f7eb41469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40aa180 msgr2=0x7f7eb4146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40aa180 0x7f7eb4146490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb414ec00 msgr2=0x7f7eb41469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb414ec00 0x7f7eb41469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 -- 172.21.15.94:0/3556967185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb414e330 con 0x7f7eb41550c0 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb8c1c640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb414ec00 0x7f7eb41469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7eb9c1e640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb41ea3f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00d240 tx=0x7f7eac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ea9ffb640 1 -- 172.21.15.94:0/3556967185 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac02e970 con 0x7f7eb41550c0 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ea9ffb640 1 -- 172.21.15.94:0/3556967185 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac045070 con 0x7f7eb41550c0 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb414e610 con 0x7f7eb41550c0 2024-03-06T07:22:03.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.329+0000 7f7ea9ffb640 1 -- 172.21.15.94:0/3556967185 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac036a60 con 0x7f7eb41550c0 2024-03-06T07:22:03.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.330+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb41475c0 con 0x7f7eb41550c0 2024-03-06T07:22:03.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.331+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb404f330 con 0x7f7eb41550c0 2024-03-06T07:22:03.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.331+0000 7f7ea9ffb640 1 -- 172.21.15.94:0/3556967185 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7eac036c00 con 0x7f7eb41550c0 2024-03-06T07:22:03.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.331+0000 7f7ea9ffb640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e9806b500 0x7f7e9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.331+0000 7f7ea9ffb640 1 -- 172.21.15.94:0/3556967185 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7eac0ac920 con 0x7f7eb41550c0 2024-03-06T07:22:03.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.331+0000 7f7eb941d640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e9806b500 0x7f7e9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.332+0000 7f7eb941d640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e9806b500 0x7f7e9806d9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0006fd0 tx=0x7f7eb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.336+0000 7f7ea9ffb640 1 -- 172.21.15.94:0/3556967185 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7eac04e020 con 0x7f7eb41550c0 2024-03-06T07:22:03.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.509+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7eb406e4a0 con 0x7f7eb41550c0 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.509+0000 7f7ea9ffb640 1 -- 172.21.15.94:0/3556967185 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7eac07b970 con 0x7f7eb41550c0 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:03.511 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e9806b500 msgr2=0x7f7e9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e9806b500 0x7f7e9806d9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0006fd0 tx=0x7f7eb0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:03.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 msgr2=0x7f7eb41ea3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb41ea3f0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00d240 tx=0x7f7eac004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 shutdown_connections 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7e9806b500 0x7f7e9806d9c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb41550c0 0x7f7eb41ea3f0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb414ec00 0x7f7eb41469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 --2- 172.21.15.94:0/3556967185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40aa180 0x7f7eb4146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 >> 172.21.15.94:0/3556967185 conn(0x7f7eb40c6710 msgr2=0x7f7eb40c39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 shutdown_connections 2024-03-06T07:22:03.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.511+0000 7f7ebb681640 1 -- 172.21.15.94:0/3556967185 wait complete. 2024-03-06T07:22:03.524 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:03.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:03.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.627+0000 7f7345aeb640 1 Processor -- start 2024-03-06T07:22:03.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.627+0000 7f7345aeb640 1 -- start start 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f7345aeb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f73400a9830 0x7f7340151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f7345aeb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7340153390 0x7f7340153800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f7345aeb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734006e850 con 0x7f73400c22b0 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f7345aeb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734006d2b0 con 0x7f7340153390 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f7345aeb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734006d590 con 0x7f73400a9830 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7340153390 0x7f7340153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f73400c22b0 0x7f734014ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52516/0 (socket says 172.21.15.94:52516) 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733f7fe640 1 -- 172.21.15.94:0/1906198226 learned_addr learned my addr 172.21.15.94:0/1906198226 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733effd640 1 --2- 172.21.15.94:0/1906198226 >> v2:172.21.15.145:3300/0 conn(0x7f73400a9830 0x7f7340151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733ffff640 1 -- 172.21.15.94:0/1906198226 >> v2:172.21.15.145:3300/0 conn(0x7f73400a9830 msgr2=0x7f7340151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733ffff640 1 --2- 172.21.15.94:0/1906198226 >> v2:172.21.15.145:3300/0 conn(0x7f73400a9830 0x7f7340151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733ffff640 1 -- 172.21.15.94:0/1906198226 >> v1:172.21.15.94:6789/0 conn(0x7f73400c22b0 legacy=0x7f734014ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733ffff640 1 -- 172.21.15.94:0/1906198226 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734006d870 con 0x7f7340153390 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.628+0000 7f733effd640 1 --2- 172.21.15.94:0/1906198226 >> v2:172.21.15.145:3300/0 conn(0x7f73400a9830 0x7f7340151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.629+0000 7f733ffff640 1 --2- 172.21.15.94:0/1906198226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f7340153800 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f7338009d90 tx=0x7f733802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ad273fd3263429 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.629+0000 7f733e7fc640 1 -- 172.21.15.94:0/1906198226 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7338039070 con 0x7f7340153390 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.629+0000 7f733e7fc640 1 -- 172.21.15.94:0/1906198226 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733802cc60 con 0x7f7340153390 2024-03-06T07:22:03.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.629+0000 7f733e7fc640 1 -- 172.21.15.94:0/1906198226 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7338034660 con 0x7f7340153390 2024-03-06T07:22:03.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 -- 172.21.15.94:0/1906198226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 msgr2=0x7f7340153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 --2- 172.21.15.94:0/1906198226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f7340153800 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f7338009d90 tx=0x7f733802beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 -- 172.21.15.94:0/1906198226 shutdown_connections 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 --2- 172.21.15.94:0/1906198226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f7340153800 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 --2- 172.21.15.94:0/1906198226 >> v2:172.21.15.145:3300/0 conn(0x7f73400a9830 0x7f7340151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 -- 172.21.15.94:0/1906198226 >> 172.21.15.94:0/1906198226 conn(0x7f73400c65b0 msgr2=0x7f73400c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 -- 172.21.15.94:0/1906198226 shutdown_connections 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.630+0000 7f7345aeb640 1 -- 172.21.15.94:0/1906198226 wait complete. 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.631+0000 7f7345aeb640 1 Processor -- start 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.631+0000 7f7345aeb640 1 -- start start 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73400a9830 0x7f73401493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73400c22b0 0x7f734014db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f73400c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734014f7e0 con 0x7f7340153390 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734014f660 con 0x7f73400a9830 2024-03-06T07:22:03.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73400c2690 con 0x7f73400c22b0 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f73400c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f73400c3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38142/0 (socket says 172.21.15.94:38142) 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 -- 172.21.15.94:0/2605155630 learned_addr learned my addr 172.21.15.94:0/2605155630 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733f7fe640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73400a9830 0x7f73401493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 -- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73400c22b0 msgr2=0x7f734014db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73400c22b0 0x7f734014db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 -- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73400a9830 msgr2=0x7f73401493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73400a9830 0x7f73401493a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 -- 172.21.15.94:0/2605155630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73401e9770 con 0x7f7340153390 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f733ffff640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f73400c3b50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f7338009eb0 tx=0x7f7338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7327fff640 1 -- 172.21.15.94:0/2605155630 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7338039050 con 0x7f7340153390 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7327fff640 1 -- 172.21.15.94:0/2605155630 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7338045070 con 0x7f7340153390 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7327fff640 1 -- 172.21.15.94:0/2605155630 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733803e830 con 0x7f7340153390 2024-03-06T07:22:03.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73401e89b0 con 0x7f7340153390 2024-03-06T07:22:03.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.632+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73401e8f70 con 0x7f7340153390 2024-03-06T07:22:03.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.633+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f734004f330 con 0x7f7340153390 2024-03-06T07:22:03.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.634+0000 7f7327fff640 1 -- 172.21.15.94:0/2605155630 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f733803e9d0 con 0x7f7340153390 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.634+0000 7f7327fff640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f732c06b4d0 0x7f732c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.634+0000 7f7327fff640 1 -- 172.21.15.94:0/2605155630 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f73380ace90 con 0x7f7340153390 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.634+0000 7f733f7fe640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f732c06b4d0 0x7f732c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.635+0000 7f733f7fe640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f732c06b4d0 0x7f732c06d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7330000f60 tx=0x7f7330009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.639+0000 7f7327fff640 1 -- 172.21.15.94:0/2605155630 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f733807c080 con 0x7f7340153390 2024-03-06T07:22:03.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:03.808+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f73400c6d10 con 0x7f7340153390 2024-03-06T07:22:04.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.246+0000 7f7327fff640 1 -- 172.21.15.94:0/2605155630 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7f733807b980 con 0x7f7340153390 2024-03-06T07:22:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f732c06b4d0 msgr2=0x7f732c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f732c06b4d0 0x7f732c06d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7330000f60 tx=0x7f7330009040 comp rx=0 tx=0).stop 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 msgr2=0x7f73400c3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f73400c3b50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f7338009eb0 tx=0x7f7338004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 shutdown_connections 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f732c06b4d0 0x7f732c06d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7340153390 0x7f73400c3b50 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73400c22b0 0x7f734014db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 --2- 172.21.15.94:0/2605155630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73400a9830 0x7f73401493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 >> 172.21.15.94:0/2605155630 conn(0x7f73400c65b0 msgr2=0x7f7340153cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 shutdown_connections 2024-03-06T07:22:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.249+0000 7f7345aeb640 1 -- 172.21.15.94:0/2605155630 wait complete. 2024-03-06T07:22:04.266 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:04.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.374+0000 7f7014a64640 1 Processor -- start 2024-03-06T07:22:04.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.374+0000 7f7014a64640 1 -- start start 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.374+0000 7f7014a64640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f70100ab170 0x7f70100c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.374+0000 7f7014a64640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70100c3ef0 con 0x7f701006c900 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.374+0000 7f7014a64640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70101556d0 con 0x7f70100c5990 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.374+0000 7f7014a64640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70101559b0 con 0x7f70100ab170 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f7007fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f701006c900 0x7f70100a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52530/0 (socket says 172.21.15.94:52530) 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f7007fff640 1 -- 172.21.15.94:0/3955672386 learned_addr learned my addr 172.21.15.94:0/3955672386 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700ed9d640 1 --2- 172.21.15.94:0/3955672386 >> v2:172.21.15.145:3300/0 conn(0x7f70100ab170 0x7f70100c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624113505 0 0) 0x7f70100c3ef0 con 0x7f701006c900 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc003520 con 0x7f701006c900 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953867874 0 0) 0x7f6ffc003520 con 0x7f701006c900 2024-03-06T07:22:04.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 >> v2:172.21.15.145:3300/0 conn(0x7f70100ab170 msgr2=0x7f70100c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 --2- 172.21.15.94:0/3955672386 >> v2:172.21.15.145:3300/0 conn(0x7f70100ab170 0x7f70100c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 >> v1:172.21.15.145:6789/0 conn(0x7f70100c5990 legacy=0x7f701006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010155c90 con 0x7f701006c900 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f6ff8001670 con 0x7f701006c900 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 >> v1:172.21.15.94:6789/0 conn(0x7f701006c900 legacy=0x7f70100a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f6ffc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ffc004800 0x7f6ffc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ffc0051c0 0x7f6ffc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101559b0 con 0x7f6ffc0039e0 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc00c090 con 0x7f6ffc004800 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc00c290 con 0x7f6ffc0051c0 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700e59c640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f6ffc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f7007fff640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ffc004800 0x7f6ffc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.375+0000 7f700ed9d640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ffc0051c0 0x7f6ffc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700e59c640 1 -- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ffc0051c0 msgr2=0x7f6ffc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700e59c640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ffc0051c0 0x7f6ffc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700e59c640 1 -- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ffc004800 msgr2=0x7f6ffc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700e59c640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ffc004800 0x7f6ffc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700ed9d640 1 -- 172.21.15.94:0/3955672386 reap_dead start 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700e59c640 1 -- 172.21.15.94:0/3955672386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc003520 con 0x7f6ffc0039e0 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700e59c640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f6ffc003e00 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f7000001750 tx=0x7f6ffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560829280b6cbc68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000001370 con 0x7f6ffc0039e0 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70000052d0 con 0x7f6ffc0039e0 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.376+0000 7f700dd9b640 1 -- 172.21.15.94:0/3955672386 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000003830 con 0x7f6ffc0039e0 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.377+0000 7f7014a64640 1 -- 172.21.15.94:0/3955672386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 msgr2=0x7f6ffc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.377+0000 7f7014a64640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f6ffc003e00 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f7000001750 tx=0x7f6ffc005830 comp rx=0 tx=0).stop 2024-03-06T07:22:04.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.377+0000 7f7014a64640 1 -- 172.21.15.94:0/3955672386 shutdown_connections 2024-03-06T07:22:04.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.377+0000 7f7014a64640 1 --2- 172.21.15.94:0/3955672386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f6ffc003e00 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.377+0000 7f7014a64640 1 -- 172.21.15.94:0/3955672386 >> 172.21.15.94:0/3955672386 conn(0x7f70100c6710 msgr2=0x7f70100c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.377+0000 7f7014a64640 1 -- 172.21.15.94:0/3955672386 shutdown_connections 2024-03-06T07:22:04.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.377+0000 7f7014a64640 1 -- 172.21.15.94:0/3955672386 wait complete. 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.378+0000 7f7014a64640 1 Processor -- start 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.378+0000 7f7014a64640 1 -- start start 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f7014a64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f7010071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f7014a64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7010152ed0 0x7f70100705f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f7014a64640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70100c0780 0x7f7010154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f7014a64640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7010156610 con 0x7f6ffc0039e0 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f7014a64640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7010156490 con 0x7f70100c0780 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f7014a64640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100c3af0 con 0x7f7010152ed0 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f700e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f7010071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f700e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f7010071fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38164/0 (socket says 172.21.15.94:38164) 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f700e59c640 1 -- 172.21.15.94:0/2502833402 learned_addr learned my addr 172.21.15.94:0/2502833402 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:04.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f7007fff640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7010152ed0 0x7f70100705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.379+0000 7f700ed9d640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70100c0780 0x7f7010154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f700e59c640 1 -- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7010152ed0 msgr2=0x7f70100705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f700e59c640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7010152ed0 0x7f70100705f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f700e59c640 1 -- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70100c0780 msgr2=0x7f7010154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f700e59c640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70100c0780 0x7f7010154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f700e59c640 1 -- 172.21.15.94:0/2502833402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010070b30 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f700e59c640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f7010071fc0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7000000c00 tx=0x7f7000001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f7006ffd640 1 -- 172.21.15.94:0/2502833402 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000041070 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f7006ffd640 1 -- 172.21.15.94:0/2502833402 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000005480 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7010152b40 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f7006ffd640 1 -- 172.21.15.94:0/2502833402 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000001370 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.380+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701014e8d0 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.381+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701014e3e0 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.381+0000 7f7006ffd640 1 -- 172.21.15.94:0/2502833402 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7000044070 con 0x7f6ffc0039e0 2024-03-06T07:22:04.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.382+0000 7f7006ffd640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ff006b500 0x7f6ff006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.382+0000 7f7006ffd640 1 -- 172.21.15.94:0/2502833402 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f70000ab9b0 con 0x7f6ffc0039e0 2024-03-06T07:22:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.382+0000 7f7007fff640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ff006b500 0x7f6ff006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.383+0000 7f7007fff640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ff006b500 0x7f6ff006d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7010152950 tx=0x7f6ff80078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.386+0000 7f7006ffd640 1 -- 172.21.15.94:0/2502833402 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f700007a2c0 con 0x7f6ffc0039e0 2024-03-06T07:22:04.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:04.560+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f701004f330 con 0x7f6ffc0039e0 2024-03-06T07:22:05.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.255+0000 7f7006ffd640 1 -- 172.21.15.94:0/2502833402 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7f700007e360 con 0x7f6ffc0039e0 2024-03-06T07:22:05.257 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ff006b500 msgr2=0x7f6ff006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ff006b500 0x7f6ff006d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7010152950 tx=0x7f6ff80078f0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 msgr2=0x7f7010071fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f7010071fc0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7000000c00 tx=0x7f7000001120 comp rx=0 tx=0).stop 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 shutdown_connections 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ff006b500 0x7f6ff006d9c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70100c0780 0x7f7010154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7010152ed0 0x7f70100705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 --2- 172.21.15.94:0/2502833402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ffc0039e0 0x7f7010071fc0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.258+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 >> 172.21.15.94:0/2502833402 conn(0x7f70100c6710 msgr2=0x7f701014b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.259+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 shutdown_connections 2024-03-06T07:22:05.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.259+0000 7f7014a64640 1 -- 172.21.15.94:0/2502833402 wait complete. 2024-03-06T07:22:05.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.388+0000 7f797d833640 1 Processor -- start 2024-03-06T07:22:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.388+0000 7f797d833640 1 -- start start 2024-03-06T07:22:05.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f797d833640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f79780a6290 0x7f79780a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f797d833640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780c1f90 con 0x7f7978155cf0 2024-03-06T07:22:05.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f797d833640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780c11d0 con 0x7f79780a8ff0 2024-03-06T07:22:05.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f797d833640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797814f850 con 0x7f79780a6290 2024-03-06T07:22:05.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79777fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7978155cf0 0x7f79780aa360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52536/0 (socket says 172.21.15.94:52536) 2024-03-06T07:22:05.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79777fe640 1 -- 172.21.15.94:0/4039862922 learned_addr learned my addr 172.21.15.94:0/4039862922 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f796ffff640 1 --2- 172.21.15.94:0/4039862922 >> v2:172.21.15.145:3300/0 conn(0x7f79780a6290 0x7f79780a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258440255 0 0) 0x7f79780c1f90 con 0x7f7978155cf0 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7964003520 con 0x7f7978155cf0 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1304062625 0 0) 0x7f7964003520 con 0x7f7978155cf0 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 >> v2:172.21.15.145:3300/0 conn(0x7f79780a6290 msgr2=0x7f79780a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 --2- 172.21.15.94:0/4039862922 >> v2:172.21.15.145:3300/0 conn(0x7f79780a6290 0x7f79780a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 >> v1:172.21.15.145:6789/0 conn(0x7f79780a8ff0 legacy=0x7f79780ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79780c14b0 con 0x7f7978155cf0 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f79700019f0 con 0x7f7978155cf0 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f796ffff640 1 --2- 172.21.15.94:0/4039862922 >> v2:172.21.15.145:3300/0 conn(0x7f79780a6290 0x7f79780a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 >> v1:172.21.15.94:6789/0 conn(0x7f7978155cf0 legacy=0x7f79780aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f7964003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7964004800 0x7f7964004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.389+0000 7f79767fc640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79640052e0 0x7f796400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797814f850 con 0x7f79640039b0 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796400c1b0 con 0x7f7964004800 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796400c3b0 con 0x7f79640052e0 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f7976ffd640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f7964003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f796ffff640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7964004800 0x7f7964004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f79777fe640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79640052e0 0x7f796400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f7976ffd640 1 -- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79640052e0 msgr2=0x7f796400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f7976ffd640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79640052e0 0x7f796400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f7976ffd640 1 -- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7964004800 msgr2=0x7f7964004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f7976ffd640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7964004800 0x7f7964004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f7976ffd640 1 -- 172.21.15.94:0/4039862922 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964003520 con 0x7f79640039b0 2024-03-06T07:22:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.390+0000 7f79777fe640 1 -- 172.21.15.94:0/4039862922 reap_dead start 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f7976ffd640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f7964003dd0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7968001720 tx=0x7f7964005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6eded99e15fe978 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968001370 con 0x7f79640039b0 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79680052d0 con 0x7f79640039b0 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f79767fc640 1 -- 172.21.15.94:0/4039862922 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968003830 con 0x7f79640039b0 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f797d833640 1 -- 172.21.15.94:0/4039862922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 msgr2=0x7f7964003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f797d833640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f7964003dd0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7968001720 tx=0x7f7964005830 comp rx=0 tx=0).stop 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f797d833640 1 -- 172.21.15.94:0/4039862922 shutdown_connections 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f797d833640 1 --2- 172.21.15.94:0/4039862922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f7964003dd0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f797d833640 1 -- 172.21.15.94:0/4039862922 >> 172.21.15.94:0/4039862922 conn(0x7f797806afc0 msgr2=0x7f797806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f797d833640 1 -- 172.21.15.94:0/4039862922 shutdown_connections 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.391+0000 7f797d833640 1 -- 172.21.15.94:0/4039862922 wait complete. 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.392+0000 7f797d833640 1 Processor -- start 2024-03-06T07:22:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f797d833640 1 -- start start 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f797d833640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f79781e8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f797d833640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7978070280 0x7f79781543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f797d833640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7978143e70 0x7f7978071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f797d833640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79780c1a70 con 0x7f79640039b0 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f797d833640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7978156260 con 0x7f7978070280 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f797d833640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79780c0a30 con 0x7f7978143e70 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f7976ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f79781e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f7976ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f79781e8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38188/0 (socket says 172.21.15.94:38188) 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f7976ffd640 1 -- 172.21.15.94:0/1576928438 learned_addr learned my addr 172.21.15.94:0/1576928438 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f79777fe640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7978143e70 0x7f7978071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.393+0000 7f796ffff640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7978070280 0x7f79781543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7976ffd640 1 -- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7978143e70 msgr2=0x7f7978071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7976ffd640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7978143e70 0x7f7978071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7976ffd640 1 -- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7978070280 msgr2=0x7f79781543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7976ffd640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7978070280 0x7f79781543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7976ffd640 1 -- 172.21.15.94:0/1576928438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978154d90 con 0x7f79640039b0 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f79777fe640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7978143e70 0x7f7978071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7976ffd640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f79781e8630 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7968005040 tx=0x7f7968001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7974ff9640 1 -- 172.21.15.94:0/1576928438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968041020 con 0x7f79640039b0 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7974ff9640 1 -- 172.21.15.94:0/1576928438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7968005480 con 0x7f79640039b0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7978155070 con 0x7f79640039b0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f7974ff9640 1 -- 172.21.15.94:0/1576928438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968007a30 con 0x7f79640039b0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.394+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79781548e0 con 0x7f79640039b0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.395+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7978155730 con 0x7f79640039b0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.395+0000 7f7974ff9640 1 -- 172.21.15.94:0/1576928438 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7968044020 con 0x7f79640039b0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.396+0000 7f7974ff9640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f795806b4d0 0x7f795806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.396+0000 7f7974ff9640 1 -- 172.21.15.94:0/1576928438 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f79680abcf0 con 0x7f79640039b0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.396+0000 7f796ffff640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f795806b4d0 0x7f795806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.396+0000 7f796ffff640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f795806b4d0 0x7f795806d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f79780aa180 tx=0x7f7960009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.400+0000 7f7974ff9640 1 -- 172.21.15.94:0/1576928438 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79680b0070 con 0x7f79640039b0 2024-03-06T07:22:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:05.609+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f797804f330 con 0x7f79640039b0 2024-03-06T07:22:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.256+0000 7f7974ff9640 1 -- 172.21.15.94:0/1576928438 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v86) v1 ==== 174+0+0 (secure 0 0 0) 0x7f796804b030 con 0x7f79640039b0 2024-03-06T07:22:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 17 and data pool 18 2024-03-06T07:22:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.259+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f795806b4d0 msgr2=0x7f795806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.259+0000 7f797d833640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f795806b4d0 0x7f795806d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f79780aa180 tx=0x7f7960009190 comp rx=0 tx=0).stop 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.259+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 msgr2=0x7f79781e8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.259+0000 7f797d833640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f79781e8630 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7968005040 tx=0x7f7968001120 comp rx=0 tx=0).stop 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.260+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 shutdown_connections 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.260+0000 7f797d833640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f795806b4d0 0x7f795806d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.260+0000 7f797d833640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7978143e70 0x7f7978071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.260+0000 7f797d833640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7978070280 0x7f79781543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.260+0000 7f797d833640 1 --2- 172.21.15.94:0/1576928438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79640039b0 0x7f79781e8630 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.260+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 >> 172.21.15.94:0/1576928438 conn(0x7f797806afc0 msgr2=0x7f79780c66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.261+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 shutdown_connections 2024-03-06T07:22:06.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.261+0000 7f797d833640 1 -- 172.21.15.94:0/1576928438 wait complete. 2024-03-06T07:22:06.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:06.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.392+0000 7fa638881640 1 Processor -- start 2024-03-06T07:22:06.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.392+0000 7fa638881640 1 -- start start 2024-03-06T07:22:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa638881640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6340c1aa0 0x7fa6341529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa638881640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa634155a40 0x7fa634155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa638881640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6341563d0 con 0x7fa63414f790 2024-03-06T07:22:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa638881640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6340c2680 con 0x7fa6340c1aa0 2024-03-06T07:22:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa638881640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6340c28d0 con 0x7fa634155a40 2024-03-06T07:22:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa631d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6340c1aa0 0x7fa6341529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa63259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa63414f790 0x7fa6340c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52546/0 (socket says 172.21.15.94:52546) 2024-03-06T07:22:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa63259c640 1 -- 172.21.15.94:0/572361066 learned_addr learned my addr 172.21.15.94:0/572361066 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa631d9b640 1 -- 172.21.15.94:0/572361066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa634155a40 msgr2=0x7fa634155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa631d9b640 1 --2- 172.21.15.94:0/572361066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa634155a40 0x7fa634155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa631d9b640 1 -- 172.21.15.94:0/572361066 >> v1:172.21.15.94:6789/0 conn(0x7fa63414f790 legacy=0x7fa6340c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.393+0000 7fa631d9b640 1 -- 172.21.15.94:0/572361066 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6340c2bb0 con 0x7fa6340c1aa0 2024-03-06T07:22:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.394+0000 7fa631d9b640 1 --2- 172.21.15.94:0/572361066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6340c1aa0 0x7fa6341529d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fa6280097a0 tx=0x7fa62802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d173e220d839a7e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.394+0000 7fa63159a640 1 -- 172.21.15.94:0/572361066 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628035020 con 0x7fa6340c1aa0 2024-03-06T07:22:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.394+0000 7fa63159a640 1 -- 172.21.15.94:0/572361066 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628031830 con 0x7fa6340c1aa0 2024-03-06T07:22:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.394+0000 7fa63159a640 1 -- 172.21.15.94:0/572361066 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628031c00 con 0x7fa6340c1aa0 2024-03-06T07:22:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 -- 172.21.15.94:0/572361066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6340c1aa0 msgr2=0x7fa6341529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 --2- 172.21.15.94:0/572361066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6340c1aa0 0x7fa6341529d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fa6280097a0 tx=0x7fa62802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 -- 172.21.15.94:0/572361066 shutdown_connections 2024-03-06T07:22:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 --2- 172.21.15.94:0/572361066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa634155a40 0x7fa634155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 --2- 172.21.15.94:0/572361066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6340c1aa0 0x7fa6341529d0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 -- 172.21.15.94:0/572361066 >> 172.21.15.94:0/572361066 conn(0x7fa6340c6710 msgr2=0x7fa6340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 -- 172.21.15.94:0/572361066 shutdown_connections 2024-03-06T07:22:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.395+0000 7fa638881640 1 -- 172.21.15.94:0/572361066 wait complete. 2024-03-06T07:22:06.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.396+0000 7fa638881640 1 Processor -- start 2024-03-06T07:22:06.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.396+0000 7fa638881640 1 -- start start 2024-03-06T07:22:06.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa638881640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6340c1aa0 0x7fa6341def50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa638881640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa63414f790 0x7fa634150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa638881640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa634155a40 0x7fa6341e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa638881640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6340c0d60 con 0x7fa63414f790 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa638881640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa634155520 con 0x7fa634155a40 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa638881640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6340c5e30 con 0x7fa6340c1aa0 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa63414f790 0x7fa634150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa63414f790 0x7fa634150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38218/0 (socket says 172.21.15.94:38218) 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 -- 172.21.15.94:0/4050958996 learned_addr learned my addr 172.21.15.94:0/4050958996 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa63259c640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6340c1aa0 0x7fa6341def50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa632d9d640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa634155a40 0x7fa6341e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 -- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6340c1aa0 msgr2=0x7fa6341def50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6340c1aa0 0x7fa6341def50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 -- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa634155a40 msgr2=0x7fa6341e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa634155a40 0x7fa6341e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa631d9b640 1 -- 172.21.15.94:0/4050958996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa634073330 con 0x7fa63414f790 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.397+0000 7fa632d9d640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa634155a40 0x7fa6341e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.398+0000 7fa631d9b640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa63414f790 0x7fa634150bc0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fa62802b910 tx=0x7fa6280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.398+0000 7fa622ffd640 1 -- 172.21.15.94:0/4050958996 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628041070 con 0x7fa63414f790 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.398+0000 7fa622ffd640 1 -- 172.21.15.94:0/4050958996 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62803a970 con 0x7fa63414f790 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.398+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6340717b0 con 0x7fa63414f790 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.398+0000 7fa622ffd640 1 -- 172.21.15.94:0/4050958996 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628035040 con 0x7fa63414f790 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.398+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa634071d70 con 0x7fa63414f790 2024-03-06T07:22:06.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.399+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63404f330 con 0x7fa63414f790 2024-03-06T07:22:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.399+0000 7fa622ffd640 1 -- 172.21.15.94:0/4050958996 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa628038020 con 0x7fa63414f790 2024-03-06T07:22:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.399+0000 7fa622ffd640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa61406b4d0 0x7fa61406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.400+0000 7fa622ffd640 1 -- 172.21.15.94:0/4050958996 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6280abe70 con 0x7fa63414f790 2024-03-06T07:22:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.400+0000 7fa63259c640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa61406b4d0 0x7fa61406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.400+0000 7fa63259c640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa61406b4d0 0x7fa61406d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa6240012e0 tx=0x7fa624009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.404+0000 7fa622ffd640 1 -- 172.21.15.94:0/4050958996 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa62807ab60 con 0x7fa63414f790 2024-03-06T07:22:06.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.576+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6340c1f40 con 0x7fa63414f790 2024-03-06T07:22:06.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.577+0000 7fa622ffd640 1 -- 172.21.15.94:0/4050958996 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16965 (secure 0 0 0) 0x7fa62807a460 con 0x7fa63414f790 2024-03-06T07:22:06.579 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:06.579 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":107,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:05.610382+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.827640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.579202+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa61406b4d0 msgr2=0x7fa61406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa61406b4d0 0x7fa61406d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa6240012e0 tx=0x7fa624009040 comp rx=0 tx=0).stop 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa63414f790 msgr2=0x7fa634150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa63414f790 0x7fa634150bc0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fa62802b910 tx=0x7fa6280046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 shutdown_connections 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa61406b4d0 0x7fa61406d990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa634155a40 0x7fa6341e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa63414f790 0x7fa634150bc0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 --2- 172.21.15.94:0/4050958996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6340c1aa0 0x7fa6341def50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 >> 172.21.15.94:0/4050958996 conn(0x7fa6340c6710 msgr2=0x7fa6340c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 shutdown_connections 2024-03-06T07:22:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.579+0000 7fa638881640 1 -- 172.21.15.94:0/4050958996 wait complete. 2024-03-06T07:22:06.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:06.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.690+0000 7ff232cd4640 1 Processor -- start 2024-03-06T07:22:06.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.690+0000 7ff232cd4640 1 -- start start 2024-03-06T07:22:06.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff232cd4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 0x7ff22c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff232cd4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22c0c28d0 con 0x7ff22c155a40 2024-03-06T07:22:06.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff232cd4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22c0c2bb0 con 0x7ff22c14f790 2024-03-06T07:22:06.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff232cd4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0c2680 con 0x7ff22c0c1aa0 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff231271640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff22c155a40 0x7ff22c155ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52556/0 (socket says 172.21.15.94:52556) 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff231271640 1 -- 172.21.15.94:0/3254583241 learned_addr learned my addr 172.21.15.94:0/3254583241 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 0x7ff22c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343279021 0 0) 0x7ff22c0c28d0 con 0x7ff22c155a40 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff21c003520 con 0x7ff22c155a40 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165969180 0 0) 0x7ff22c0c2bb0 con 0x7ff22c14f790 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22c0c28d0 con 0x7ff22c14f790 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502518859 0 0) 0x7ff21c003520 con 0x7ff22c155a40 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 msgr2=0x7ff22c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 0x7ff22c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 >> v1:172.21.15.145:6789/0 conn(0x7ff22c14f790 legacy=0x7ff22c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c0c2e90 con 0x7ff22c155a40 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7ff2240019f0 con 0x7ff22c155a40 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 >> v1:172.21.15.94:6789/0 conn(0x7ff22c155a40 legacy=0x7ff22c155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 0x7ff21c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff21c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.691+0000 7ff22b7fe640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff21c004d40 0x7ff21c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0c2680 con 0x7ff21c0040d0 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c00baa0 con 0x7ff21c004d40 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c00bca0 con 0x7ff22c0c1aa0 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff21c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff231271640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff21c004d40 0x7ff21c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff230a70640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 0x7ff21c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22bfff640 1 -- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 msgr2=0x7ff21c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0c1aa0 0x7ff21c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22bfff640 1 -- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff21c004d40 msgr2=0x7ff21c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff21c004d40 0x7ff21c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.692+0000 7ff22bfff640 1 -- 172.21.15.94:0/3254583241 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c003520 con 0x7ff21c0040d0 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff22bfff640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff21c004800 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff218009ac0 tx=0x7ff21c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c23f453086b7782 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218039070 con 0x7ff21c0040d0 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21802c6b0 con 0x7ff21c0040d0 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff22b7fe640 1 -- 172.21.15.94:0/3254583241 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218009690 con 0x7ff21c0040d0 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff232cd4640 1 -- 172.21.15.94:0/3254583241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 msgr2=0x7ff21c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff232cd4640 1 --2- 172.21.15.94:0/3254583241 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff21c004800 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff218009ac0 tx=0x7ff21c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff231271640 1 -- 172.21.15.94:0/3254583241 reap_dead start 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff232cd4640 1 -- 172.21.15.94:0/3254583241 shutdown_connections 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff232cd4640 1 -- 172.21.15.94:0/3254583241 >> 172.21.15.94:0/3254583241 conn(0x7ff22c0c6710 msgr2=0x7ff22c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff232cd4640 1 -- 172.21.15.94:0/3254583241 shutdown_connections 2024-03-06T07:22:06.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.693+0000 7ff232cd4640 1 -- 172.21.15.94:0/3254583241 wait complete. 2024-03-06T07:22:06.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.694+0000 7ff232cd4640 1 Processor -- start 2024-03-06T07:22:06.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.694+0000 7ff232cd4640 1 -- start start 2024-03-06T07:22:06.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff232cd4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff22c1df9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff232cd4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22c072040 0x7ff22c1eab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff232cd4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0708b0 0x7ff22c1eb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff232cd4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0c5e30 con 0x7ff21c0040d0 2024-03-06T07:22:06.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff232cd4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0c0d60 con 0x7ff22c072040 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff232cd4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0c5fb0 con 0x7ff22c0708b0 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff230a70640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff22c1df9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff230a70640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff22c1df9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38242/0 (socket says 172.21.15.94:38242) 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff230a70640 1 -- 172.21.15.94:0/2441953524 learned_addr learned my addr 172.21.15.94:0/2441953524 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff231271640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0708b0 0x7ff22c1eb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22c072040 0x7ff22c1eab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff230a70640 1 -- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0708b0 msgr2=0x7ff22c1eb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff230a70640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0708b0 0x7ff22c1eb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff230a70640 1 -- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22c072040 msgr2=0x7ff22c1eab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.695+0000 7ff230a70640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22c072040 0x7ff22c1eab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.696+0000 7ff230a70640 1 -- 172.21.15.94:0/2441953524 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c14e330 con 0x7ff21c0040d0 2024-03-06T07:22:06.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.696+0000 7ff230a70640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff22c1df9a0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff22000f7d0 tx=0x7ff22000fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.696+0000 7ff2297fa640 1 -- 172.21.15.94:0/2441953524 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22000fe80 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.696+0000 7ff2297fa640 1 -- 172.21.15.94:0/2441953524 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff220018070 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.696+0000 7ff2297fa640 1 -- 172.21.15.94:0/2441953524 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22000d6d0 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.696+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22c1eed70 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.696+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22c1eea40 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.697+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22c04f330 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.697+0000 7ff2297fa640 1 -- 172.21.15.94:0/2441953524 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff22000d870 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.698+0000 7ff2297fa640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b500 0x7ff21006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.698+0000 7ff2297fa640 1 -- 172.21.15.94:0/2441953524 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff22008a230 con 0x7ff21c0040d0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.698+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b500 0x7ff21006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.698+0000 7ff22bfff640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b500 0x7ff21006d9c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff218036000 tx=0x7ff218030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.702+0000 7ff2297fa640 1 -- 172.21.15.94:0/2441953524 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff220058f20 con 0x7ff21c0040d0 2024-03-06T07:22:06.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.873+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff22c068450 con 0x7ff21c0040d0 2024-03-06T07:22:06.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.874+0000 7ff2297fa640 1 -- 172.21.15.94:0/2441953524 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16965 (secure 0 0 0) 0x7ff220058820 con 0x7ff21c0040d0 2024-03-06T07:22:06.876 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:06.877 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":107,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:05.610382+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.827640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.579202+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.876+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b500 msgr2=0x7ff21006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.876+0000 7ff232cd4640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b500 0x7ff21006d9c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff218036000 tx=0x7ff218030000 comp rx=0 tx=0).stop 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.876+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 msgr2=0x7ff22c1df9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.876+0000 7ff232cd4640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff22c1df9a0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff22000f7d0 tx=0x7ff22000fc90 comp rx=0 tx=0).stop 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 shutdown_connections 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b500 0x7ff21006d9c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff22c0708b0 0x7ff22c1eb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22c072040 0x7ff22c1eab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 --2- 172.21.15.94:0/2441953524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c0040d0 0x7ff22c1df9a0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 >> 172.21.15.94:0/2441953524 conn(0x7ff22c0c6710 msgr2=0x7ff22c0a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 shutdown_connections 2024-03-06T07:22:06.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.877+0000 7ff232cd4640 1 -- 172.21.15.94:0/2441953524 wait complete. 2024-03-06T07:22:06.890 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:06.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.991+0000 7f23a80df640 1 Processor -- start 2024-03-06T07:22:06.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.991+0000 7f23a80df640 1 -- start start 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.991+0000 7f23a80df640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23a00ab310 0x7f23a00ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.991+0000 7f23a80df640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f23a00a97c0 0x7f23a00c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.991+0000 7f23a80df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a0154330 con 0x7f23a01538c0 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.991+0000 7f23a80df640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a00c46a0 con 0x7f23a00ab310 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.991+0000 7f23a80df640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a0153dc0 con 0x7f23a00a97c0 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a5e7b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23a00ab310 0x7f23a00ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a5e7b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23a00ab310 0x7f23a00ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38246/0 (socket says 172.21.15.94:38246) 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a5e7b640 1 -- 172.21.15.94:0/1663042734 learned_addr learned my addr 172.21.15.94:0/1663042734 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a667c640 1 --2- 172.21.15.94:0/1663042734 >> v2:172.21.15.145:3300/0 conn(0x7f23a00a97c0 0x7f23a00c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a5e7b640 1 -- 172.21.15.94:0/1663042734 >> v2:172.21.15.145:3300/0 conn(0x7f23a00a97c0 msgr2=0x7f23a00c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a5e7b640 1 --2- 172.21.15.94:0/1663042734 >> v2:172.21.15.145:3300/0 conn(0x7f23a00a97c0 0x7f23a00c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a5e7b640 1 -- 172.21.15.94:0/1663042734 >> v1:172.21.15.145:6789/0 conn(0x7f23a01538c0 legacy=0x7f23a01494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.992+0000 7f23a5e7b640 1 -- 172.21.15.94:0/1663042734 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00c5d30 con 0x7f23a00ab310 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a5e7b640 1 --2- 172.21.15.94:0/1663042734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a00ab6f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f23a014d490 tx=0x7f239402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8240169a725774f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a4e79640 1 -- 172.21.15.94:0/1663042734 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239402e070 con 0x7f23a00ab310 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a4e79640 1 -- 172.21.15.94:0/1663042734 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2394037070 con 0x7f23a00ab310 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a4e79640 1 -- 172.21.15.94:0/1663042734 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394031a00 con 0x7f23a00ab310 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 -- 172.21.15.94:0/1663042734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 msgr2=0x7f23a00ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 --2- 172.21.15.94:0/1663042734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a00ab6f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f23a014d490 tx=0x7f239402b880 comp rx=0 tx=0).stop 2024-03-06T07:22:06.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 -- 172.21.15.94:0/1663042734 shutdown_connections 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 --2- 172.21.15.94:0/1663042734 >> v2:172.21.15.145:3300/0 conn(0x7f23a00a97c0 0x7f23a00c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 --2- 172.21.15.94:0/1663042734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a00ab6f0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 -- 172.21.15.94:0/1663042734 >> 172.21.15.94:0/1663042734 conn(0x7f23a00c6710 msgr2=0x7f23a00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 -- 172.21.15.94:0/1663042734 shutdown_connections 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.993+0000 7f23a80df640 1 -- 172.21.15.94:0/1663042734 wait complete. 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.994+0000 7f23a80df640 1 Processor -- start 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a80df640 1 -- start start 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a80df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a00a97c0 0x7f23a00bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a80df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a01ec300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a80df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a01538c0 0x7f23a01486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a80df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00abc30 con 0x7f23a00ab310 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a80df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00c62f0 con 0x7f23a01538c0 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a80df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0149bb0 con 0x7f23a00a97c0 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a567a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a01ec300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a567a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a01ec300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38248/0 (socket says 172.21.15.94:38248) 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a567a640 1 -- 172.21.15.94:0/579185864 learned_addr learned my addr 172.21.15.94:0/579185864 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a667c640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a01538c0 0x7f23a01486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.995+0000 7f23a5e7b640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a00a97c0 0x7f23a00bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a567a640 1 -- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a00a97c0 msgr2=0x7f23a00bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a567a640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a00a97c0 0x7f23a00bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a567a640 1 -- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a01538c0 msgr2=0x7f23a01486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a567a640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a01538c0 0x7f23a01486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a567a640 1 -- 172.21.15.94:0/579185864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a01ef380 con 0x7f23a00ab310 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a567a640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a01ec300 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f239c00b3c0 tx=0x7f239c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23927fc640 1 -- 172.21.15.94:0/579185864 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c011830 con 0x7f23a00ab310 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23927fc640 1 -- 172.21.15.94:0/579185864 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c016070 con 0x7f23a00ab310 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a006c540 con 0x7f23a00ab310 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23927fc640 1 -- 172.21.15.94:0/579185864 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c011c10 con 0x7f23a00ab310 2024-03-06T07:22:07.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.996+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a006cb00 con 0x7f23a00ab310 2024-03-06T07:22:07.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.997+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a004f330 con 0x7f23a00ab310 2024-03-06T07:22:07.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.997+0000 7f23927fc640 1 -- 172.21.15.94:0/579185864 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f239c01b460 con 0x7f23a00ab310 2024-03-06T07:22:07.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.998+0000 7f23927fc640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238406b4d0 0x7f238406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.998+0000 7f23927fc640 1 -- 172.21.15.94:0/579185864 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f239c089c20 con 0x7f23a00ab310 2024-03-06T07:22:07.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.998+0000 7f23a5e7b640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238406b4d0 0x7f238406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:06.999+0000 7f23a5e7b640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238406b4d0 0x7f238406d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2394036000 tx=0x7f239403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:07.002+0000 7f23927fc640 1 -- 172.21.15.94:0/579185864 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f239c020020 con 0x7f23a00ab310 2024-03-06T07:22:07.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:07.211+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f23a0153120 con 0x7f23a00ab310 2024-03-06T07:22:08.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.199+0000 7f23927fc640 1 -- 172.21.15.94:0/579185864 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v88) v1 ==== 118+0+0 (secure 0 0 0) 0x7f239c058350 con 0x7f23a00ab310 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238406b4d0 msgr2=0x7f238406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238406b4d0 0x7f238406d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2394036000 tx=0x7f239403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 msgr2=0x7f23a01ec300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a01ec300 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f239c00b3c0 tx=0x7f239c00b880 comp rx=0 tx=0).stop 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 shutdown_connections 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238406b4d0 0x7f238406d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a01538c0 0x7f23a01486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00ab310 0x7f23a01ec300 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 --2- 172.21.15.94:0/579185864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a00a97c0 0x7f23a00bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 >> 172.21.15.94:0/579185864 conn(0x7f23a00c6710 msgr2=0x7f23a00c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 shutdown_connections 2024-03-06T07:22:08.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.202+0000 7f23a80df640 1 -- 172.21.15.94:0/579185864 wait complete. 2024-03-06T07:22:08.217 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:08.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:08.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.324+0000 7fe6add25640 1 Processor -- start 2024-03-06T07:22:08.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.324+0000 7fe6add25640 1 -- start start 2024-03-06T07:22:08.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6add25640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6a80a97c0 0x7fe6a80c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6add25640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a8154330 con 0x7fe6a80ab310 2024-03-06T07:22:08.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6add25640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a8153dc0 con 0x7fe6a81538c0 2024-03-06T07:22:08.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6add25640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a80c46a0 con 0x7fe6a80a97c0 2024-03-06T07:22:08.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6a7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6a80a97c0 0x7fe6a80c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6a77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe6a80ab310 0x7fe6a80ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52572/0 (socket says 172.21.15.94:52572) 2024-03-06T07:22:08.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6a7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6a80a97c0 0x7fe6a80c4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38254/0 (socket says 172.21.15.94:38254) 2024-03-06T07:22:08.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6a77fe640 1 -- 172.21.15.94:0/1392904853 learned_addr learned my addr 172.21.15.94:0/1392904853 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:08.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6a7fff640 1 -- 172.21.15.94:0/1392904853 >> v1:172.21.15.145:6789/0 conn(0x7fe6a81538c0 legacy=0x7fe6a81494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6a7fff640 1 -- 172.21.15.94:0/1392904853 >> v1:172.21.15.94:6789/0 conn(0x7fe6a80ab310 legacy=0x7fe6a80ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.325+0000 7fe6a7fff640 1 -- 172.21.15.94:0/1392904853 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a80c5d30 con 0x7fe6a80a97c0 2024-03-06T07:22:08.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/1392904853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a80c4b30 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009d90 tx=0x7fe6a002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93582590b2b67199 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6a67fc640 1 -- 172.21.15.94:0/1392904853 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0039070 con 0x7fe6a80a97c0 2024-03-06T07:22:08.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6a67fc640 1 -- 172.21.15.94:0/1392904853 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a002cc60 con 0x7fe6a80a97c0 2024-03-06T07:22:08.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6a67fc640 1 -- 172.21.15.94:0/1392904853 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0034660 con 0x7fe6a80a97c0 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6add25640 1 -- 172.21.15.94:0/1392904853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 msgr2=0x7fe6a80c4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6add25640 1 --2- 172.21.15.94:0/1392904853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a80c4b30 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009d90 tx=0x7fe6a002beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6add25640 1 -- 172.21.15.94:0/1392904853 shutdown_connections 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6add25640 1 --2- 172.21.15.94:0/1392904853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a80c4b30 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.326+0000 7fe6add25640 1 -- 172.21.15.94:0/1392904853 >> 172.21.15.94:0/1392904853 conn(0x7fe6a80c6710 msgr2=0x7fe6a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.327+0000 7fe6add25640 1 -- 172.21.15.94:0/1392904853 shutdown_connections 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.327+0000 7fe6add25640 1 -- 172.21.15.94:0/1392904853 wait complete. 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 Processor -- start 2024-03-06T07:22:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 -- start start 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a81eaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a80ab310 0x7fe6a81ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a81538c0 0x7fe6a80bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8149bb0 con 0x7fe6a80a97c0 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a80c62f0 con 0x7fe6a81538c0 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6add25640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a8149d30 con 0x7fe6a80ab310 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.328+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a81eaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a81eaa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38260/0 (socket says 172.21.15.94:38260) 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 -- 172.21.15.94:0/891680763 learned_addr learned my addr 172.21.15.94:0/891680763 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a81538c0 0x7fe6a80bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a6ffd640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a80ab310 0x7fe6a81ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 -- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a80ab310 msgr2=0x7fe6a81ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a80ab310 0x7fe6a81ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 -- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a81538c0 msgr2=0x7fe6a80bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a81538c0 0x7fe6a80bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 -- 172.21.15.94:0/891680763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a8156f90 con 0x7fe6a80a97c0 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a7fff640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a81538c0 0x7fe6a80bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6a77fe640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a81eaa50 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe69800c4a0 tx=0x7fe69800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe68ffff640 1 -- 172.21.15.94:0/891680763 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698017070 con 0x7fe6a80a97c0 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe68ffff640 1 -- 172.21.15.94:0/891680763 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe698005ce0 con 0x7fe6a80a97c0 2024-03-06T07:22:08.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a8148960 con 0x7fe6a80a97c0 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe68ffff640 1 -- 172.21.15.94:0/891680763 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698007270 con 0x7fe6a80a97c0 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.329+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a81edfc0 con 0x7fe6a80a97c0 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.330+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a804f330 con 0x7fe6a80a97c0 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.330+0000 7fe68ffff640 1 -- 172.21.15.94:0/891680763 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe69801a070 con 0x7fe6a80a97c0 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.331+0000 7fe68ffff640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe69406b4d0 0x7fe69406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.331+0000 7fe68ffff640 1 -- 172.21.15.94:0/891680763 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe6980894d0 con 0x7fe6a80a97c0 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.331+0000 7fe6a6ffd640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe69406b4d0 0x7fe69406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.331+0000 7fe6a6ffd640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe69406b4d0 0x7fe69406d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe69c007fd0 tx=0x7fe69c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.335+0000 7fe68ffff640 1 -- 172.21.15.94:0/891680763 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe698057c00 con 0x7fe6a80a97c0 2024-03-06T07:22:08.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:08.542+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe6a80c5a70 con 0x7fe6a80a97c0 2024-03-06T07:22:09.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.277+0000 7fe68ffff640 1 -- 172.21.15.94:0/891680763 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v89) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe69805bca0 con 0x7fe6a80a97c0 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe69406b4d0 msgr2=0x7fe69406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe69406b4d0 0x7fe69406d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe69c007fd0 tx=0x7fe69c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 msgr2=0x7fe6a81eaa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a81eaa50 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe69800c4a0 tx=0x7fe69800c960 comp rx=0 tx=0).stop 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 shutdown_connections 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe69406b4d0 0x7fe69406d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a81538c0 0x7fe6a80bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a80ab310 0x7fe6a81ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 --2- 172.21.15.94:0/891680763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6a80a97c0 0x7fe6a81eaa50 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 >> 172.21.15.94:0/891680763 conn(0x7fe6a80c6710 msgr2=0x7fe6a8152510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 shutdown_connections 2024-03-06T07:22:09.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.280+0000 7fe6add25640 1 -- 172.21.15.94:0/891680763 wait complete. 2024-03-06T07:22:09.294 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:09.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.396+0000 7fdb14042640 1 Processor -- start 2024-03-06T07:22:09.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.396+0000 7fdb14042640 1 -- start start 2024-03-06T07:22:09.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb14042640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb0c14f790 0x7fdb0c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb14042640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c155a40 0x7fdb0c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb14042640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c1563d0 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb14042640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0c0c2680 con 0x7fdb0c14f790 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb14042640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0c28d0 con 0x7fdb0c155a40 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb115dd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdb0c0c1aa0 0x7fdb0c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54670/0 (socket says 172.21.15.94:54670) 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb115dd640 1 -- 172.21.15.94:0/2674205647 learned_addr learned my addr 172.21.15.94:0/2674205647 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb11dde640 1 --2- 172.21.15.94:0/2674205647 >> v2:172.21.15.145:3300/0 conn(0x7fdb0c14f790 0x7fdb0c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb125df640 1 --2- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c155a40 0x7fdb0c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb10ddc640 1 -- 172.21.15.94:0/2674205647 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211330321 0 0) 0x7fdb0c1563d0 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.397+0000 7fdb10ddc640 1 -- 172.21.15.94:0/2674205647 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf8003520 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.398+0000 7fdb11dde640 1 -- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c155a40 msgr2=0x7fdb0c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.398+0000 7fdb11dde640 1 --2- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c155a40 0x7fdb0c155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.398+0000 7fdb11dde640 1 -- 172.21.15.94:0/2674205647 >> v1:172.21.15.94:6789/0 conn(0x7fdb0c0c1aa0 legacy=0x7fdb0c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.398+0000 7fdb11dde640 1 -- 172.21.15.94:0/2674205647 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c0c2bb0 con 0x7fdb0c14f790 2024-03-06T07:22:09.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.398+0000 7fdb125df640 1 --2- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c155a40 0x7fdb0c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.398+0000 7fdb11dde640 1 --2- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c14f790 0x7fdb0c1529d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdb080097a0 tx=0x7fdb0802fce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b02e336191a88097 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.398+0000 7fdb10ddc640 1 -- 172.21.15.94:0/2674205647 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0803d070 con 0x7fdb0c14f790 2024-03-06T07:22:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb10ddc640 1 -- 172.21.15.94:0/2674205647 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb08030a90 con 0x7fdb0c14f790 2024-03-06T07:22:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb10ddc640 1 -- 172.21.15.94:0/2674205647 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08030e60 con 0x7fdb0c14f790 2024-03-06T07:22:09.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 -- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c14f790 msgr2=0x7fdb0c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 --2- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c14f790 0x7fdb0c1529d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdb080097a0 tx=0x7fdb0802fce0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 -- 172.21.15.94:0/2674205647 shutdown_connections 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 --2- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c155a40 0x7fdb0c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 --2- 172.21.15.94:0/2674205647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c14f790 0x7fdb0c1529d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 -- 172.21.15.94:0/2674205647 >> 172.21.15.94:0/2674205647 conn(0x7fdb0c0c6710 msgr2=0x7fdb0c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 -- 172.21.15.94:0/2674205647 shutdown_connections 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.399+0000 7fdb14042640 1 -- 172.21.15.94:0/2674205647 wait complete. 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.400+0000 7fdb14042640 1 Processor -- start 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.400+0000 7fdb14042640 1 -- start start 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb14042640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb0c0c1aa0 0x7fdb0c154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb14042640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c14f790 0x7fdb0c0bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb14042640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c155a40 0x7fdb0c154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb14042640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0c0d60 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb14042640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c155520 con 0x7fdb0c155a40 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb14042640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0c5e30 con 0x7fdb0c14f790 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb0c0c1aa0 0x7fdb0c154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb0c0c1aa0 0x7fdb0c154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33474/0 (socket says 172.21.15.94:33474) 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 -- 172.21.15.94:0/1014406242 learned_addr learned my addr 172.21.15.94:0/1014406242 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb125df640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c155a40 0x7fdb0c154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb115dd640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c14f790 0x7fdb0c0bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 -- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c14f790 msgr2=0x7fdb0c0bffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c14f790 0x7fdb0c0bffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 -- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c155a40 msgr2=0x7fdb0c154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c155a40 0x7fdb0c154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 -- 172.21.15.94:0/1014406242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c1ead70 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.401+0000 7fdb11dde640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb0c0c1aa0 0x7fdb0c154f10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fdb080098c0 tx=0x7fdb080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.402+0000 7fdb027fc640 1 -- 172.21.15.94:0/1014406242 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0803d040 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.402+0000 7fdb027fc640 1 -- 172.21.15.94:0/1014406242 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb08030890 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.402+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c072d50 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.402+0000 7fdb027fc640 1 -- 172.21.15.94:0/1014406242 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08006e10 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.402+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c1eaa90 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.403+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c04f330 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.403+0000 7fdb027fc640 1 -- 172.21.15.94:0/1014406242 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb0804a020 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.403+0000 7fdb027fc640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaf006b4d0 0x7fdaf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.403+0000 7fdb027fc640 1 -- 172.21.15.94:0/1014406242 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdb080b0000 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.403+0000 7fdb115dd640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaf006b4d0 0x7fdaf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.404+0000 7fdb115dd640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaf006b4d0 0x7fdaf006d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0ab650 tx=0x7fdafc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.408+0000 7fdb027fc640 1 -- 172.21.15.94:0/1014406242 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb0807edc0 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.612+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0c151610 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.613+0000 7fdb027fc640 1 -- 172.21.15.94:0/1014406242 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4990 (secure 0 0 0) 0x7fdb0807e6c0 con 0x7fdb0c0c1aa0 2024-03-06T07:22:09.615 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:09.615 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2722633283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2722633283},{"type":"v1","addr":"172.21.15.145:6837","nonce":2722633283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:08.543964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":36134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_36134":{"gid":36134,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/4001418202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4001418202},{"type":"v1","addr":"172.21.15.145:6836","nonce":4001418202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":36134,"qdb_cluster":[36134]},"id":8}]} 2024-03-06T07:22:09.615 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 89 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaf006b4d0 msgr2=0x7fdaf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaf006b4d0 0x7fdaf006d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c0ab650 tx=0x7fdafc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb0c0c1aa0 msgr2=0x7fdb0c154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb0c0c1aa0 0x7fdb0c154f10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fdb080098c0 tx=0x7fdb080046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 shutdown_connections 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdaf006b4d0 0x7fdaf006d990 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb0c155a40 0x7fdb0c154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb0c14f790 0x7fdb0c0bffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 --2- 172.21.15.94:0/1014406242 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb0c0c1aa0 0x7fdb0c154f10 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 >> 172.21.15.94:0/1014406242 conn(0x7fdb0c0c6710 msgr2=0x7fdb0c0a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 shutdown_connections 2024-03-06T07:22:09.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.615+0000 7fdb14042640 1 -- 172.21.15.94:0/1014406242 wait complete. 2024-03-06T07:22:09.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:09.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.725+0000 7f49f1088640 1 Processor -- start 2024-03-06T07:22:09.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.725+0000 7f49f1088640 1 -- start start 2024-03-06T07:22:09.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49f1088640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f49ec0c1aa0 0x7f49ec1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49f1088640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f49ec155a40 0x7f49ec155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49f1088640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec0c28d0 con 0x7f49ec14f790 2024-03-06T07:22:09.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49f1088640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ec0c2680 con 0x7f49ec0c1aa0 2024-03-06T07:22:09.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49f1088640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ec0c2bb0 con 0x7f49ec155a40 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49ea59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f49ec0c1aa0 0x7f49ec1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49ea59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f49ec0c1aa0 0x7f49ec1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33482/0 (socket says 172.21.15.94:33482) 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49ea59c640 1 -- 172.21.15.94:0/3190541520 learned_addr learned my addr 172.21.15.94:0/3190541520 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.726+0000 7f49eb59e640 1 --2- 172.21.15.94:0/3190541520 >> v2:172.21.15.145:3300/0 conn(0x7f49ec155a40 0x7f49ec155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49eb59e640 1 -- 172.21.15.94:0/3190541520 >> v2:172.21.15.94:3300/0 conn(0x7f49ec0c1aa0 msgr2=0x7f49ec1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49eb59e640 1 --2- 172.21.15.94:0/3190541520 >> v2:172.21.15.94:3300/0 conn(0x7f49ec0c1aa0 0x7f49ec1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49eb59e640 1 -- 172.21.15.94:0/3190541520 >> v1:172.21.15.145:6789/0 conn(0x7f49ec14f790 legacy=0x7f49ec0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49e9d9b640 1 -- 172.21.15.94:0/3190541520 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831286656 0 0) 0x7f49ec0c28d0 con 0x7f49ec14f790 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49eb59e640 1 -- 172.21.15.94:0/3190541520 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec0c2e90 con 0x7f49ec155a40 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49ea59c640 1 --2- 172.21.15.94:0/3190541520 >> v2:172.21.15.94:3300/0 conn(0x7f49ec0c1aa0 0x7f49ec1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49eb59e640 1 --2- 172.21.15.94:0/3190541520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec155a40 0x7f49ec155e50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f49ec06c960 tx=0x7f49e402bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baa239fe7694cbb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.727+0000 7f49e9d9b640 1 -- 172.21.15.94:0/3190541520 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4039070 con 0x7f49ec155a40 2024-03-06T07:22:09.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49e9d9b640 1 -- 172.21.15.94:0/3190541520 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e402cc70 con 0x7f49ec155a40 2024-03-06T07:22:09.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49e9d9b640 1 -- 172.21.15.94:0/3190541520 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4034660 con 0x7f49ec155a40 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 -- 172.21.15.94:0/3190541520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec155a40 msgr2=0x7f49ec155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 --2- 172.21.15.94:0/3190541520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec155a40 0x7f49ec155e50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f49ec06c960 tx=0x7f49e402bec0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 -- 172.21.15.94:0/3190541520 shutdown_connections 2024-03-06T07:22:09.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 --2- 172.21.15.94:0/3190541520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec155a40 0x7f49ec155e50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 --2- 172.21.15.94:0/3190541520 >> v2:172.21.15.94:3300/0 conn(0x7f49ec0c1aa0 0x7f49ec1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 -- 172.21.15.94:0/3190541520 >> 172.21.15.94:0/3190541520 conn(0x7f49ec0c6710 msgr2=0x7f49ec0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 -- 172.21.15.94:0/3190541520 shutdown_connections 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.728+0000 7f49f1088640 1 -- 172.21.15.94:0/3190541520 wait complete. 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.729+0000 7f49f1088640 1 Processor -- start 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.729+0000 7f49f1088640 1 -- start start 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49f1088640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec0c1aa0 0x7f49ec072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49f1088640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ec14f790 0x7f49ec1543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49f1088640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ec155a40 0x7f49ec14fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49f1088640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0c0d60 con 0x7f49ec14f790 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49f1088640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec156390 con 0x7f49ec0c1aa0 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49f1088640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ec0c5e30 con 0x7f49ec155a40 2024-03-06T07:22:09.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ec14f790 0x7f49ec1543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ec14f790 0x7f49ec1543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33498/0 (socket says 172.21.15.94:33498) 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49ea59c640 1 -- 172.21.15.94:0/3570404425 learned_addr learned my addr 172.21.15.94:0/3570404425 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49ead9d640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec0c1aa0 0x7f49ec072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.730+0000 7f49eb59e640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ec155a40 0x7f49ec14fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49ea59c640 1 -- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ec155a40 msgr2=0x7f49ec14fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49ea59c640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ec155a40 0x7f49ec14fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49ea59c640 1 -- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec0c1aa0 msgr2=0x7f49ec072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49ea59c640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec0c1aa0 0x7f49ec072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49ea59c640 1 -- 172.21.15.94:0/3570404425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec1539d0 con 0x7f49ec14f790 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49eb59e640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ec155a40 0x7f49ec14fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49ea59c640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ec14f790 0x7f49ec1543e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f49e000e9a0 tx=0x7f49e000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49d37fe640 1 -- 172.21.15.94:0/3570404425 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0017070 con 0x7f49ec14f790 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49d37fe640 1 -- 172.21.15.94:0/3570404425 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e000ca00 con 0x7f49ec14f790 2024-03-06T07:22:09.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49d37fe640 1 -- 172.21.15.94:0/3570404425 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e000cdb0 con 0x7f49ec14f790 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ec153cb0 con 0x7f49ec14f790 2024-03-06T07:22:09.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.731+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ec070da0 con 0x7f49ec14f790 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.732+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ec04f330 con 0x7f49ec14f790 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.732+0000 7f49d37fe640 1 -- 172.21.15.94:0/3570404425 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49e001a070 con 0x7f49ec14f790 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.733+0000 7f49d37fe640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49d806b500 0x7f49d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.733+0000 7f49d37fe640 1 -- 172.21.15.94:0/3570404425 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f49e008a4e0 con 0x7f49ec14f790 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.733+0000 7f49ead9d640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49d806b500 0x7f49d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:09.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.733+0000 7f49ead9d640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49d806b500 0x7f49d806d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f49dc000f60 tx=0x7f49dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:09.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.737+0000 7f49d37fe640 1 -- 172.21.15.94:0/3570404425 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49e0058c10 con 0x7f49ec14f790 2024-03-06T07:22:09.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.909+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49ec1521d0 con 0x7f49ec14f790 2024-03-06T07:22:09.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.910+0000 7f49d37fe640 1 -- 172.21.15.94:0/3570404425 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16952 (secure 0 0 0) 0x7f49e005ccb0 con 0x7f49ec14f790 2024-03-06T07:22:09.912 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:09.912 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":108,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:07.268021+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.827640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.579202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:09.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49d806b500 msgr2=0x7f49d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49d806b500 0x7f49d806d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f49dc000f60 tx=0x7f49dc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ec14f790 msgr2=0x7f49ec1543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ec14f790 0x7f49ec1543e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f49e000e9a0 tx=0x7f49e000ee60 comp rx=0 tx=0).stop 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 shutdown_connections 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f49d806b500 0x7f49d806d9c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ec155a40 0x7f49ec14fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ec14f790 0x7f49ec1543e0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 --2- 172.21.15.94:0/3570404425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ec0c1aa0 0x7f49ec072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 >> 172.21.15.94:0/3570404425 conn(0x7f49ec0c6710 msgr2=0x7f49ec0c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 shutdown_connections 2024-03-06T07:22:09.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:09.912+0000 7f49f1088640 1 -- 172.21.15.94:0/3570404425 wait complete. 2024-03-06T07:22:09.925 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:09.925 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:10.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.023+0000 7f3306439640 1 Processor -- start 2024-03-06T07:22:10.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f3306439640 1 -- start start 2024-03-06T07:22:10.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f3306439640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f330014f790 0x7f33001529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f3306439640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c1aa0 0x7f33000c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f3306439640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f33000c5c60 0x7f3300155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f3306439640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33000c2680 con 0x7f330014f790 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f3306439640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33000c28d0 con 0x7f33000c5c60 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f3306439640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33000c2bb0 con 0x7f33000c1aa0 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f32fffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f330014f790 0x7f33001529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f32fffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f330014f790 0x7f33001529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33502/0 (socket says 172.21.15.94:33502) 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f32fffff640 1 -- 172.21.15.94:0/3237073722 learned_addr learned my addr 172.21.15.94:0/3237073722 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f32ff7fe640 1 --2- 172.21.15.94:0/3237073722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c1aa0 0x7f33000c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.024+0000 7f33049d6640 1 --2- 172.21.15.94:0/3237073722 >> v2:172.21.15.145:3300/0 conn(0x7f33000c5c60 0x7f3300155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.025+0000 7f32fffff640 1 -- 172.21.15.94:0/3237073722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c1aa0 msgr2=0x7f33000c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.025+0000 7f32fffff640 1 --2- 172.21.15.94:0/3237073722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c1aa0 0x7f33000c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.025+0000 7f32fffff640 1 -- 172.21.15.94:0/3237073722 >> v2:172.21.15.145:3300/0 conn(0x7f33000c5c60 msgr2=0x7f3300155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.025+0000 7f32fffff640 1 --2- 172.21.15.94:0/3237073722 >> v2:172.21.15.145:3300/0 conn(0x7f33000c5c60 0x7f3300155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.025+0000 7f32fffff640 1 -- 172.21.15.94:0/3237073722 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33000c2e90 con 0x7f330014f790 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.025+0000 7f32ff7fe640 1 --2- 172.21.15.94:0/3237073722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c1aa0 0x7f33000c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.025+0000 7f32fffff640 1 --2- 172.21.15.94:0/3237073722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f330014f790 0x7f33001529d0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f330006c960 tx=0x7f32f002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b7060138fad9e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f32feffd640 1 -- 172.21.15.94:0/3237073722 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f003b070 con 0x7f330014f790 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f32feffd640 1 -- 172.21.15.94:0/3237073722 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f0036440 con 0x7f330014f790 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f32feffd640 1 -- 172.21.15.94:0/3237073722 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0036810 con 0x7f330014f790 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 -- 172.21.15.94:0/3237073722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f330014f790 msgr2=0x7f33001529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 --2- 172.21.15.94:0/3237073722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f330014f790 0x7f33001529d0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f330006c960 tx=0x7f32f002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 -- 172.21.15.94:0/3237073722 shutdown_connections 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 --2- 172.21.15.94:0/3237073722 >> v2:172.21.15.145:3300/0 conn(0x7f33000c5c60 0x7f3300155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 --2- 172.21.15.94:0/3237073722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c1aa0 0x7f33000c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 --2- 172.21.15.94:0/3237073722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f330014f790 0x7f33001529d0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 -- 172.21.15.94:0/3237073722 >> 172.21.15.94:0/3237073722 conn(0x7f33000c6710 msgr2=0x7f33000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 -- 172.21.15.94:0/3237073722 shutdown_connections 2024-03-06T07:22:10.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.026+0000 7f3306439640 1 -- 172.21.15.94:0/3237073722 wait complete. 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.027+0000 7f3306439640 1 Processor -- start 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f3306439640 1 -- start start 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f3306439640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33000c1aa0 0x7f3300072f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f3306439640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c5c60 0x7f33001505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f3306439640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f330014f790 0x7f33001dc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f3306439640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330006cc00 con 0x7f33000c1aa0 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f3306439640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33000c0d60 con 0x7f330014f790 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f3306439640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330006cd80 con 0x7f33000c5c60 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f32fffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33000c1aa0 0x7f3300072f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f32fffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33000c1aa0 0x7f3300072f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33512/0 (socket says 172.21.15.94:33512) 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f32fffff640 1 -- 172.21.15.94:0/316224457 learned_addr learned my addr 172.21.15.94:0/316224457 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f33049d6640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f330014f790 0x7f33001dc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.028+0000 7f32ff7fe640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c5c60 0x7f33001505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fffff640 1 -- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c5c60 msgr2=0x7f33001505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fffff640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c5c60 0x7f33001505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fffff640 1 -- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f330014f790 msgr2=0x7f33001dc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fffff640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f330014f790 0x7f33001dc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fffff640 1 -- 172.21.15.94:0/316224457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33001dd840 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fffff640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33000c1aa0 0x7f3300072f90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f32f002e120 tx=0x7f32f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fcff9640 1 -- 172.21.15.94:0/316224457 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0045070 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fcff9640 1 -- 172.21.15.94:0/316224457 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f0040870 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f32fcff9640 1 -- 172.21.15.94:0/316224457 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f003b040 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33000734d0 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.029+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33001e7470 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.030+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330004f330 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.030+0000 7f32fcff9640 1 -- 172.21.15.94:0/316224457 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32f0048070 con 0x7f33000c1aa0 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.031+0000 7f32fcff9640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f32ec06b4d0 0x7f32ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.031+0000 7f32fcff9640 1 -- 172.21.15.94:0/316224457 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32f00aeda0 con 0x7f33000c1aa0 2024-03-06T07:22:10.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.031+0000 7f32ff7fe640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f32ec06b4d0 0x7f32ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.031+0000 7f32ff7fe640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f32ec06b4d0 0x7f32ec06d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009560 tx=0x7f32f4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.035+0000 7f32fcff9640 1 -- 172.21.15.94:0/316224457 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32f007d5f0 con 0x7f33000c1aa0 2024-03-06T07:22:10.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.201+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3300072de0 con 0x7f32ec06b4d0 2024-03-06T07:22:10.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.207+0000 7f32fcff9640 1 -- 172.21.15.94:0/316224457 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252562 (secure 0 0 0) 0x7f3300072de0 con 0x7f32ec06b4d0 2024-03-06T07:22:10.212 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:10.213 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f32ec06b4d0 msgr2=0x7f32ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f32ec06b4d0 0x7f32ec06d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009560 tx=0x7f32f4009190 comp rx=0 tx=0).stop 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33000c1aa0 msgr2=0x7f3300072f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33000c1aa0 0x7f3300072f90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f32f002e120 tx=0x7f32f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 shutdown_connections 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f32ec06b4d0 0x7f32ec06d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f330014f790 0x7f33001dc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33000c5c60 0x7f33001505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 --2- 172.21.15.94:0/316224457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33000c1aa0 0x7f3300072f90 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.210+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 >> 172.21.15.94:0/316224457 conn(0x7f33000c6710 msgr2=0x7f33000c4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.211+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 shutdown_connections 2024-03-06T07:22:10.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.211+0000 7f3306439640 1 -- 172.21.15.94:0/316224457 wait complete. 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-03-06T07:22:08.684374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260140,"kb_used_data":10648,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714580,"statfs":{"total":773094113280,"available":772827729920,"internally_reserved":0,"allocated":10903552,"data_stored":6338004,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.435909"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274390+0000","last_change":"2024-03-06T07:22:06.274189+0000","last_active":"2024-03-06T07:22:07.274390+0000","last_peered":"2024-03-06T07:22:07.274390+0000","last_clean":"2024-03-06T07:22:07.274390+0000","last_became_active":"2024-03-06T07:22:06.273926+0000","last_became_peered":"2024-03-06T07:22:06.273926+0000","last_unstale":"2024-03-06T07:22:07.274390+0000","last_undegraded":"2024-03-06T07:22:07.274390+0000","last_fullsized":"2024-03-06T07:22:07.274390+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:46.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324168+0000","last_change":"2024-03-06T07:22:06.285476+0000","last_active":"2024-03-06T07:22:07.324168+0000","last_peered":"2024-03-06T07:22:07.324168+0000","last_clean":"2024-03-06T07:22:07.324168+0000","last_became_active":"2024-03-06T07:22:06.284773+0000","last_became_peered":"2024-03-06T07:22:06.284773+0000","last_unstale":"2024-03-06T07:22:07.324168+0000","last_undegraded":"2024-03-06T07:22:07.324168+0000","last_fullsized":"2024-03-06T07:22:07.324168+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:36.281548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324075+0000","last_change":"2024-03-06T07:22:06.283289+0000","last_active":"2024-03-06T07:22:07.324075+0000","last_peered":"2024-03-06T07:22:07.324075+0000","last_clean":"2024-03-06T07:22:07.324075+0000","last_became_active":"2024-03-06T07:22:06.283018+0000","last_became_peered":"2024-03-06T07:22:06.283018+0000","last_unstale":"2024-03-06T07:22:07.324075+0000","last_undegraded":"2024-03-06T07:22:07.324075+0000","last_fullsized":"2024-03-06T07:22:07.324075+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.811925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323941+0000","last_change":"2024-03-06T07:22:06.283187+0000","last_active":"2024-03-06T07:22:07.323941+0000","last_peered":"2024-03-06T07:22:07.323941+0000","last_clean":"2024-03-06T07:22:07.323941+0000","last_became_active":"2024-03-06T07:22:06.282893+0000","last_became_peered":"2024-03-06T07:22:06.282893+0000","last_unstale":"2024-03-06T07:22:07.323941+0000","last_undegraded":"2024-03-06T07:22:07.323941+0000","last_fullsized":"2024-03-06T07:22:07.323941+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:02.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273919+0000","last_change":"2024-03-06T07:22:06.284103+0000","last_active":"2024-03-06T07:22:07.273919+0000","last_peered":"2024-03-06T07:22:07.273919+0000","last_clean":"2024-03-06T07:22:07.273919+0000","last_became_active":"2024-03-06T07:22:06.283840+0000","last_became_peered":"2024-03-06T07:22:06.283840+0000","last_unstale":"2024-03-06T07:22:07.273919+0000","last_undegraded":"2024-03-06T07:22:07.273919+0000","last_fullsized":"2024-03-06T07:22:07.273919+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:58.629934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323872+0000","last_change":"2024-03-06T07:22:06.267539+0000","last_active":"2024-03-06T07:22:07.323872+0000","last_peered":"2024-03-06T07:22:07.323872+0000","last_clean":"2024-03-06T07:22:07.323872+0000","last_became_active":"2024-03-06T07:22:06.267202+0000","last_became_peered":"2024-03-06T07:22:06.267202+0000","last_unstale":"2024-03-06T07:22:07.323872+0000","last_undegraded":"2024-03-06T07:22:07.323872+0000","last_fullsized":"2024-03-06T07:22:07.323872+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:32.128497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323698+0000","last_change":"2024-03-06T07:22:06.286083+0000","last_active":"2024-03-06T07:22:07.323698+0000","last_peered":"2024-03-06T07:22:07.323698+0000","last_clean":"2024-03-06T07:22:07.323698+0000","last_became_active":"2024-03-06T07:22:06.285818+0000","last_became_peered":"2024-03-06T07:22:06.285818+0000","last_unstale":"2024-03-06T07:22:07.323698+0000","last_undegraded":"2024-03-06T07:22:07.323698+0000","last_fullsized":"2024-03-06T07:22:07.323698+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:05.683743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323651+0000","last_change":"2024-03-06T07:22:06.286474+0000","last_active":"2024-03-06T07:22:07.323651+0000","last_peered":"2024-03-06T07:22:07.323651+0000","last_clean":"2024-03-06T07:22:07.323651+0000","last_became_active":"2024-03-06T07:22:06.285226+0000","last_became_peered":"2024-03-06T07:22:06.285226+0000","last_unstale":"2024-03-06T07:22:07.323651+0000","last_undegraded":"2024-03-06T07:22:07.323651+0000","last_fullsized":"2024-03-06T07:22:07.323651+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:27.472310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324389+0000","last_change":"2024-03-06T07:22:06.288562+0000","last_active":"2024-03-06T07:22:07.324389+0000","last_peered":"2024-03-06T07:22:07.324389+0000","last_clean":"2024-03-06T07:22:07.324389+0000","last_became_active":"2024-03-06T07:22:06.288282+0000","last_became_peered":"2024-03-06T07:22:06.288282+0000","last_unstale":"2024-03-06T07:22:07.324389+0000","last_undegraded":"2024-03-06T07:22:07.324389+0000","last_fullsized":"2024-03-06T07:22:07.324389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:30.644551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324421+0000","last_change":"2024-03-06T07:22:06.271898+0000","last_active":"2024-03-06T07:22:07.324421+0000","last_peered":"2024-03-06T07:22:07.324421+0000","last_clean":"2024-03-06T07:22:07.324421+0000","last_became_active":"2024-03-06T07:22:06.271643+0000","last_became_peered":"2024-03-06T07:22:06.271643+0000","last_unstale":"2024-03-06T07:22:07.324421+0000","last_undegraded":"2024-03-06T07:22:07.324421+0000","last_fullsized":"2024-03-06T07:22:07.324421+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:40.433875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274007+0000","last_change":"2024-03-06T07:22:06.279646+0000","last_active":"2024-03-06T07:22:07.274007+0000","last_peered":"2024-03-06T07:22:07.274007+0000","last_clean":"2024-03-06T07:22:07.274007+0000","last_became_active":"2024-03-06T07:22:06.279342+0000","last_became_peered":"2024-03-06T07:22:06.279342+0000","last_unstale":"2024-03-06T07:22:07.274007+0000","last_undegraded":"2024-03-06T07:22:07.274007+0000","last_fullsized":"2024-03-06T07:22:07.274007+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:01.068684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325059+0000","last_change":"2024-03-06T07:22:06.275948+0000","last_active":"2024-03-06T07:22:07.325059+0000","last_peered":"2024-03-06T07:22:07.325059+0000","last_clean":"2024-03-06T07:22:07.325059+0000","last_became_active":"2024-03-06T07:22:06.275683+0000","last_became_peered":"2024-03-06T07:22:06.275683+0000","last_unstale":"2024-03-06T07:22:07.325059+0000","last_undegraded":"2024-03-06T07:22:07.325059+0000","last_fullsized":"2024-03-06T07:22:07.325059+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:01.110061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324301+0000","last_change":"2024-03-06T07:22:06.283085+0000","last_active":"2024-03-06T07:22:07.324301+0000","last_peered":"2024-03-06T07:22:07.324301+0000","last_clean":"2024-03-06T07:22:07.324301+0000","last_became_active":"2024-03-06T07:22:06.282786+0000","last_became_peered":"2024-03-06T07:22:06.282786+0000","last_unstale":"2024-03-06T07:22:07.324301+0000","last_undegraded":"2024-03-06T07:22:07.324301+0000","last_fullsized":"2024-03-06T07:22:07.324301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:57.318529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324783+0000","last_change":"2024-03-06T07:22:06.283315+0000","last_active":"2024-03-06T07:22:07.324783+0000","last_peered":"2024-03-06T07:22:07.324783+0000","last_clean":"2024-03-06T07:22:07.324783+0000","last_became_active":"2024-03-06T07:22:06.282764+0000","last_became_peered":"2024-03-06T07:22:06.282764+0000","last_unstale":"2024-03-06T07:22:07.324783+0000","last_undegraded":"2024-03-06T07:22:07.324783+0000","last_fullsized":"2024-03-06T07:22:07.324783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:47.730533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273403+0000","last_change":"2024-03-06T07:22:06.280055+0000","last_active":"2024-03-06T07:22:07.273403+0000","last_peered":"2024-03-06T07:22:07.273403+0000","last_clean":"2024-03-06T07:22:07.273403+0000","last_became_active":"2024-03-06T07:22:06.279760+0000","last_became_peered":"2024-03-06T07:22:06.279760+0000","last_unstale":"2024-03-06T07:22:07.273403+0000","last_undegraded":"2024-03-06T07:22:07.273403+0000","last_fullsized":"2024-03-06T07:22:07.273403+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:00.533569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323660+0000","last_change":"2024-03-06T07:22:06.270471+0000","last_active":"2024-03-06T07:22:07.323660+0000","last_peered":"2024-03-06T07:22:07.323660+0000","last_clean":"2024-03-06T07:22:07.323660+0000","last_became_active":"2024-03-06T07:22:06.270190+0000","last_became_peered":"2024-03-06T07:22:06.270190+0000","last_unstale":"2024-03-06T07:22:07.323660+0000","last_undegraded":"2024-03-06T07:22:07.323660+0000","last_fullsized":"2024-03-06T07:22:07.323660+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:32.708206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324491+0000","last_change":"2024-03-06T07:22:06.277248+0000","last_active":"2024-03-06T07:22:07.324491+0000","last_peered":"2024-03-06T07:22:07.324491+0000","last_clean":"2024-03-06T07:22:07.324491+0000","last_became_active":"2024-03-06T07:22:06.276981+0000","last_became_peered":"2024-03-06T07:22:06.276981+0000","last_unstale":"2024-03-06T07:22:07.324491+0000","last_undegraded":"2024-03-06T07:22:07.324491+0000","last_fullsized":"2024-03-06T07:22:07.324491+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:10.302008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324299+0000","last_change":"2024-03-06T07:22:06.285784+0000","last_active":"2024-03-06T07:22:07.324299+0000","last_peered":"2024-03-06T07:22:07.324299+0000","last_clean":"2024-03-06T07:22:07.324299+0000","last_became_active":"2024-03-06T07:22:06.284444+0000","last_became_peered":"2024-03-06T07:22:06.284444+0000","last_unstale":"2024-03-06T07:22:07.324299+0000","last_undegraded":"2024-03-06T07:22:07.324299+0000","last_fullsized":"2024-03-06T07:22:07.324299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:54.348798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273406+0000","last_change":"2024-03-06T07:22:06.291869+0000","last_active":"2024-03-06T07:22:07.273406+0000","last_peered":"2024-03-06T07:22:07.273406+0000","last_clean":"2024-03-06T07:22:07.273406+0000","last_became_active":"2024-03-06T07:22:06.291524+0000","last_became_peered":"2024-03-06T07:22:06.291524+0000","last_unstale":"2024-03-06T07:22:07.273406+0000","last_undegraded":"2024-03-06T07:22:07.273406+0000","last_fullsized":"2024-03-06T07:22:07.273406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:43.785073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272443+0000","last_change":"2024-03-06T07:22:06.271230+0000","last_active":"2024-03-06T07:22:07.272443+0000","last_peered":"2024-03-06T07:22:07.272443+0000","last_clean":"2024-03-06T07:22:07.272443+0000","last_became_active":"2024-03-06T07:22:06.270940+0000","last_became_peered":"2024-03-06T07:22:06.270940+0000","last_unstale":"2024-03-06T07:22:07.272443+0000","last_undegraded":"2024-03-06T07:22:07.272443+0000","last_fullsized":"2024-03-06T07:22:07.272443+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:35.031046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324040+0000","last_change":"2024-03-06T07:22:06.287793+0000","last_active":"2024-03-06T07:22:07.324040+0000","last_peered":"2024-03-06T07:22:07.324040+0000","last_clean":"2024-03-06T07:22:07.324040+0000","last_became_active":"2024-03-06T07:22:06.287433+0000","last_became_peered":"2024-03-06T07:22:06.287433+0000","last_unstale":"2024-03-06T07:22:07.324040+0000","last_undegraded":"2024-03-06T07:22:07.324040+0000","last_fullsized":"2024-03-06T07:22:07.324040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:20.360914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274597+0000","last_change":"2024-03-06T07:22:06.273316+0000","last_active":"2024-03-06T07:22:07.274597+0000","last_peered":"2024-03-06T07:22:07.274597+0000","last_clean":"2024-03-06T07:22:07.274597+0000","last_became_active":"2024-03-06T07:22:06.273014+0000","last_became_peered":"2024-03-06T07:22:06.273014+0000","last_unstale":"2024-03-06T07:22:07.274597+0000","last_undegraded":"2024-03-06T07:22:07.274597+0000","last_fullsized":"2024-03-06T07:22:07.274597+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:57.318529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324273+0000","last_change":"2024-03-06T07:22:06.287966+0000","last_active":"2024-03-06T07:22:07.324273+0000","last_peered":"2024-03-06T07:22:07.324273+0000","last_clean":"2024-03-06T07:22:07.324273+0000","last_became_active":"2024-03-06T07:22:06.287405+0000","last_became_peered":"2024-03-06T07:22:06.287405+0000","last_unstale":"2024-03-06T07:22:07.324273+0000","last_undegraded":"2024-03-06T07:22:07.324273+0000","last_fullsized":"2024-03-06T07:22:07.324273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:45.778003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324210+0000","last_change":"2024-03-06T07:22:06.286120+0000","last_active":"2024-03-06T07:22:07.324210+0000","last_peered":"2024-03-06T07:22:07.324210+0000","last_clean":"2024-03-06T07:22:07.324210+0000","last_became_active":"2024-03-06T07:22:06.285851+0000","last_became_peered":"2024-03-06T07:22:06.285851+0000","last_unstale":"2024-03-06T07:22:07.324210+0000","last_undegraded":"2024-03-06T07:22:07.324210+0000","last_fullsized":"2024-03-06T07:22:07.324210+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:02.811442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324493+0000","last_change":"2024-03-06T07:22:06.280900+0000","last_active":"2024-03-06T07:22:07.324493+0000","last_peered":"2024-03-06T07:22:07.324493+0000","last_clean":"2024-03-06T07:22:07.324493+0000","last_became_active":"2024-03-06T07:22:06.280415+0000","last_became_peered":"2024-03-06T07:22:06.280415+0000","last_unstale":"2024-03-06T07:22:07.324493+0000","last_undegraded":"2024-03-06T07:22:07.324493+0000","last_fullsized":"2024-03-06T07:22:07.324493+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:03.152066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272615+0000","last_change":"2024-03-06T07:22:06.271002+0000","last_active":"2024-03-06T07:22:07.272615+0000","last_peered":"2024-03-06T07:22:07.272615+0000","last_clean":"2024-03-06T07:22:07.272615+0000","last_became_active":"2024-03-06T07:22:06.270705+0000","last_became_peered":"2024-03-06T07:22:06.270705+0000","last_unstale":"2024-03-06T07:22:07.272615+0000","last_undegraded":"2024-03-06T07:22:07.272615+0000","last_fullsized":"2024-03-06T07:22:07.272615+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:12.812773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273555+0000","last_change":"2024-03-06T07:22:06.288907+0000","last_active":"2024-03-06T07:22:07.273555+0000","last_peered":"2024-03-06T07:22:07.273555+0000","last_clean":"2024-03-06T07:22:07.273555+0000","last_became_active":"2024-03-06T07:22:06.288635+0000","last_became_peered":"2024-03-06T07:22:06.288635+0000","last_unstale":"2024-03-06T07:22:07.273555+0000","last_undegraded":"2024-03-06T07:22:07.273555+0000","last_fullsized":"2024-03-06T07:22:07.273555+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:03.086475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324917+0000","last_change":"2024-03-06T07:22:06.275729+0000","last_active":"2024-03-06T07:22:07.324917+0000","last_peered":"2024-03-06T07:22:07.324917+0000","last_clean":"2024-03-06T07:22:07.324917+0000","last_became_active":"2024-03-06T07:22:06.275468+0000","last_became_peered":"2024-03-06T07:22:06.275468+0000","last_unstale":"2024-03-06T07:22:07.324917+0000","last_undegraded":"2024-03-06T07:22:07.324917+0000","last_fullsized":"2024-03-06T07:22:07.324917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:29.775060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272558+0000","last_change":"2024-03-06T07:22:06.269348+0000","last_active":"2024-03-06T07:22:07.272558+0000","last_peered":"2024-03-06T07:22:07.272558+0000","last_clean":"2024-03-06T07:22:07.272558+0000","last_became_active":"2024-03-06T07:22:06.269049+0000","last_became_peered":"2024-03-06T07:22:06.269049+0000","last_unstale":"2024-03-06T07:22:07.272558+0000","last_undegraded":"2024-03-06T07:22:07.272558+0000","last_fullsized":"2024-03-06T07:22:07.272558+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:01.498065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324905+0000","last_change":"2024-03-06T07:22:06.276527+0000","last_active":"2024-03-06T07:22:07.324905+0000","last_peered":"2024-03-06T07:22:07.324905+0000","last_clean":"2024-03-06T07:22:07.324905+0000","last_became_active":"2024-03-06T07:22:06.276167+0000","last_became_peered":"2024-03-06T07:22:06.276167+0000","last_unstale":"2024-03-06T07:22:07.324905+0000","last_undegraded":"2024-03-06T07:22:07.324905+0000","last_fullsized":"2024-03-06T07:22:07.324905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:27.387382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324295+0000","last_change":"2024-03-06T07:22:06.274422+0000","last_active":"2024-03-06T07:22:07.324295+0000","last_peered":"2024-03-06T07:22:07.324295+0000","last_clean":"2024-03-06T07:22:07.324295+0000","last_became_active":"2024-03-06T07:22:06.274116+0000","last_became_peered":"2024-03-06T07:22:06.274116+0000","last_unstale":"2024-03-06T07:22:07.324295+0000","last_undegraded":"2024-03-06T07:22:07.324295+0000","last_fullsized":"2024-03-06T07:22:07.324295+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:32.823840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324316+0000","last_change":"2024-03-06T07:22:06.283397+0000","last_active":"2024-03-06T07:22:07.324316+0000","last_peered":"2024-03-06T07:22:07.324316+0000","last_clean":"2024-03-06T07:22:07.324316+0000","last_became_active":"2024-03-06T07:22:06.283134+0000","last_became_peered":"2024-03-06T07:22:06.283134+0000","last_unstale":"2024-03-06T07:22:07.324316+0000","last_undegraded":"2024-03-06T07:22:07.324316+0000","last_fullsized":"2024-03-06T07:22:07.324316+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:04.668491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324692+0000","last_change":"2024-03-06T07:22:06.272969+0000","last_active":"2024-03-06T07:22:07.324692+0000","last_peered":"2024-03-06T07:22:07.324692+0000","last_clean":"2024-03-06T07:22:07.324692+0000","last_became_active":"2024-03-06T07:22:06.272688+0000","last_became_peered":"2024-03-06T07:22:06.272688+0000","last_unstale":"2024-03-06T07:22:07.324692+0000","last_undegraded":"2024-03-06T07:22:07.324692+0000","last_fullsized":"2024-03-06T07:22:07.324692+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:02.811442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273206+0000","last_change":"2024-03-06T07:22:05.274770+0000","last_active":"2024-03-06T07:22:07.273206+0000","last_peered":"2024-03-06T07:22:07.273206+0000","last_clean":"2024-03-06T07:22:07.273206+0000","last_became_active":"2024-03-06T07:22:05.274459+0000","last_became_peered":"2024-03-06T07:22:05.274459+0000","last_unstale":"2024-03-06T07:22:07.273206+0000","last_undegraded":"2024-03-06T07:22:07.273206+0000","last_fullsized":"2024-03-06T07:22:07.273206+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:23.337554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324068+0000","last_change":"2024-03-06T07:22:06.267869+0000","last_active":"2024-03-06T07:22:07.324068+0000","last_peered":"2024-03-06T07:22:07.324068+0000","last_clean":"2024-03-06T07:22:07.324068+0000","last_became_active":"2024-03-06T07:22:06.267608+0000","last_became_peered":"2024-03-06T07:22:06.267608+0000","last_unstale":"2024-03-06T07:22:07.324068+0000","last_undegraded":"2024-03-06T07:22:07.324068+0000","last_fullsized":"2024-03-06T07:22:07.324068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:30.342334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324538+0000","last_change":"2024-03-06T07:22:05.269390+0000","last_active":"2024-03-06T07:22:07.324538+0000","last_peered":"2024-03-06T07:22:07.324538+0000","last_clean":"2024-03-06T07:22:07.324538+0000","last_became_active":"2024-03-06T07:22:05.269045+0000","last_became_peered":"2024-03-06T07:22:05.269045+0000","last_unstale":"2024-03-06T07:22:07.324538+0000","last_undegraded":"2024-03-06T07:22:07.324538+0000","last_fullsized":"2024-03-06T07:22:07.324538+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:22.512414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324156+0000","last_change":"2024-03-06T07:22:06.281995+0000","last_active":"2024-03-06T07:22:07.324156+0000","last_peered":"2024-03-06T07:22:07.324156+0000","last_clean":"2024-03-06T07:22:07.324156+0000","last_became_active":"2024-03-06T07:22:06.281688+0000","last_became_peered":"2024-03-06T07:22:06.281688+0000","last_unstale":"2024-03-06T07:22:07.324156+0000","last_undegraded":"2024-03-06T07:22:07.324156+0000","last_fullsized":"2024-03-06T07:22:07.324156+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:10.543713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324124+0000","last_change":"2024-03-06T07:22:05.276257+0000","last_active":"2024-03-06T07:22:07.324124+0000","last_peered":"2024-03-06T07:22:07.324124+0000","last_clean":"2024-03-06T07:22:07.324124+0000","last_became_active":"2024-03-06T07:22:05.275951+0000","last_became_peered":"2024-03-06T07:22:05.275951+0000","last_unstale":"2024-03-06T07:22:07.324124+0000","last_undegraded":"2024-03-06T07:22:07.324124+0000","last_fullsized":"2024-03-06T07:22:07.324124+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:56.313423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324464+0000","last_change":"2024-03-06T07:22:06.287709+0000","last_active":"2024-03-06T07:22:07.324464+0000","last_peered":"2024-03-06T07:22:07.324464+0000","last_clean":"2024-03-06T07:22:07.324464+0000","last_became_active":"2024-03-06T07:22:06.287390+0000","last_became_peered":"2024-03-06T07:22:06.287390+0000","last_unstale":"2024-03-06T07:22:07.324464+0000","last_undegraded":"2024-03-06T07:22:07.324464+0000","last_fullsized":"2024-03-06T07:22:07.324464+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:04.872464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324858+0000","last_change":"2024-03-06T07:22:05.265795+0000","last_active":"2024-03-06T07:22:07.324858+0000","last_peered":"2024-03-06T07:22:07.324858+0000","last_clean":"2024-03-06T07:22:07.324858+0000","last_became_active":"2024-03-06T07:22:05.265116+0000","last_became_peered":"2024-03-06T07:22:05.265116+0000","last_unstale":"2024-03-06T07:22:07.324858+0000","last_undegraded":"2024-03-06T07:22:07.324858+0000","last_fullsized":"2024-03-06T07:22:07.324858+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:46.201174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324934+0000","last_change":"2024-03-06T07:22:06.283260+0000","last_active":"2024-03-06T07:22:07.324934+0000","last_peered":"2024-03-06T07:22:07.324934+0000","last_clean":"2024-03-06T07:22:07.324934+0000","last_became_active":"2024-03-06T07:22:06.282667+0000","last_became_peered":"2024-03-06T07:22:06.282667+0000","last_unstale":"2024-03-06T07:22:07.324934+0000","last_undegraded":"2024-03-06T07:22:07.324934+0000","last_fullsized":"2024-03-06T07:22:07.324934+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:32.823840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274013+0000","last_change":"2024-03-06T07:22:05.269608+0000","last_active":"2024-03-06T07:22:07.274013+0000","last_peered":"2024-03-06T07:22:07.274013+0000","last_clean":"2024-03-06T07:22:07.274013+0000","last_became_active":"2024-03-06T07:22:05.269226+0000","last_became_peered":"2024-03-06T07:22:05.269226+0000","last_unstale":"2024-03-06T07:22:07.274013+0000","last_undegraded":"2024-03-06T07:22:07.274013+0000","last_fullsized":"2024-03-06T07:22:07.274013+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.526750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273553+0000","last_change":"2024-03-06T07:22:06.284758+0000","last_active":"2024-03-06T07:22:07.273553+0000","last_peered":"2024-03-06T07:22:07.273553+0000","last_clean":"2024-03-06T07:22:07.273553+0000","last_became_active":"2024-03-06T07:22:06.284472+0000","last_became_peered":"2024-03-06T07:22:06.284472+0000","last_unstale":"2024-03-06T07:22:07.273553+0000","last_undegraded":"2024-03-06T07:22:07.273553+0000","last_fullsized":"2024-03-06T07:22:07.273553+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:03.086475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323969+0000","last_change":"2024-03-06T07:22:05.270816+0000","last_active":"2024-03-06T07:22:07.323969+0000","last_peered":"2024-03-06T07:22:07.323969+0000","last_clean":"2024-03-06T07:22:07.323969+0000","last_became_active":"2024-03-06T07:22:05.270550+0000","last_became_peered":"2024-03-06T07:22:05.270550+0000","last_unstale":"2024-03-06T07:22:07.323969+0000","last_undegraded":"2024-03-06T07:22:07.323969+0000","last_fullsized":"2024-03-06T07:22:07.323969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:53.343691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325032+0000","last_change":"2024-03-06T07:22:06.269423+0000","last_active":"2024-03-06T07:22:07.325032+0000","last_peered":"2024-03-06T07:22:07.325032+0000","last_clean":"2024-03-06T07:22:07.325032+0000","last_became_active":"2024-03-06T07:22:06.269180+0000","last_became_peered":"2024-03-06T07:22:06.269180+0000","last_unstale":"2024-03-06T07:22:07.325032+0000","last_undegraded":"2024-03-06T07:22:07.325032+0000","last_fullsized":"2024-03-06T07:22:07.325032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:40.036671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324354+0000","last_change":"2024-03-06T07:22:05.271075+0000","last_active":"2024-03-06T07:22:07.324354+0000","last_peered":"2024-03-06T07:22:07.324354+0000","last_clean":"2024-03-06T07:22:07.324354+0000","last_became_active":"2024-03-06T07:22:05.270790+0000","last_became_peered":"2024-03-06T07:22:05.270790+0000","last_unstale":"2024-03-06T07:22:07.324354+0000","last_undegraded":"2024-03-06T07:22:07.324354+0000","last_fullsized":"2024-03-06T07:22:07.324354+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:15.182553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324421+0000","last_change":"2024-03-06T07:22:06.273126+0000","last_active":"2024-03-06T07:22:07.324421+0000","last_peered":"2024-03-06T07:22:07.324421+0000","last_clean":"2024-03-06T07:22:07.324421+0000","last_became_active":"2024-03-06T07:22:06.272851+0000","last_became_peered":"2024-03-06T07:22:06.272851+0000","last_unstale":"2024-03-06T07:22:07.324421+0000","last_undegraded":"2024-03-06T07:22:07.324421+0000","last_fullsized":"2024-03-06T07:22:07.324421+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:11.308970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323988+0000","last_change":"2024-03-06T07:22:05.266985+0000","last_active":"2024-03-06T07:22:07.323988+0000","last_peered":"2024-03-06T07:22:07.323988+0000","last_clean":"2024-03-06T07:22:07.323988+0000","last_became_active":"2024-03-06T07:22:05.266735+0000","last_became_peered":"2024-03-06T07:22:05.266735+0000","last_unstale":"2024-03-06T07:22:07.323988+0000","last_undegraded":"2024-03-06T07:22:07.323988+0000","last_fullsized":"2024-03-06T07:22:07.323988+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:35.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273865+0000","last_change":"2024-03-06T07:22:06.290954+0000","last_active":"2024-03-06T07:22:07.273865+0000","last_peered":"2024-03-06T07:22:07.273865+0000","last_clean":"2024-03-06T07:22:07.273865+0000","last_became_active":"2024-03-06T07:22:06.290677+0000","last_became_peered":"2024-03-06T07:22:06.290677+0000","last_unstale":"2024-03-06T07:22:07.273865+0000","last_undegraded":"2024-03-06T07:22:07.273865+0000","last_fullsized":"2024-03-06T07:22:07.273865+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:58.629934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273304+0000","last_change":"2024-03-06T07:22:05.277283+0000","last_active":"2024-03-06T07:22:07.273304+0000","last_peered":"2024-03-06T07:22:07.273304+0000","last_clean":"2024-03-06T07:22:07.273304+0000","last_became_active":"2024-03-06T07:22:05.276995+0000","last_became_peered":"2024-03-06T07:22:05.276995+0000","last_unstale":"2024-03-06T07:22:07.273304+0000","last_undegraded":"2024-03-06T07:22:07.273304+0000","last_fullsized":"2024-03-06T07:22:07.273304+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:22.512414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324565+0000","last_change":"2024-03-06T07:22:06.278005+0000","last_active":"2024-03-06T07:22:07.324565+0000","last_peered":"2024-03-06T07:22:07.324565+0000","last_clean":"2024-03-06T07:22:07.324565+0000","last_became_active":"2024-03-06T07:22:06.277741+0000","last_became_peered":"2024-03-06T07:22:06.277741+0000","last_unstale":"2024-03-06T07:22:07.324565+0000","last_undegraded":"2024-03-06T07:22:07.324565+0000","last_fullsized":"2024-03-06T07:22:07.324565+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:02.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324161+0000","last_change":"2024-03-06T07:22:05.272997+0000","last_active":"2024-03-06T07:22:07.324161+0000","last_peered":"2024-03-06T07:22:07.324161+0000","last_clean":"2024-03-06T07:22:07.324161+0000","last_became_active":"2024-03-06T07:22:05.272736+0000","last_became_peered":"2024-03-06T07:22:05.272736+0000","last_unstale":"2024-03-06T07:22:07.324161+0000","last_undegraded":"2024-03-06T07:22:07.324161+0000","last_fullsized":"2024-03-06T07:22:07.324161+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:59.528462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324425+0000","last_change":"2024-03-06T07:22:06.285930+0000","last_active":"2024-03-06T07:22:07.324425+0000","last_peered":"2024-03-06T07:22:07.324425+0000","last_clean":"2024-03-06T07:22:07.324425+0000","last_became_active":"2024-03-06T07:22:06.285683+0000","last_became_peered":"2024-03-06T07:22:06.285683+0000","last_unstale":"2024-03-06T07:22:07.324425+0000","last_undegraded":"2024-03-06T07:22:07.324425+0000","last_fullsized":"2024-03-06T07:22:07.324425+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:12.415589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324536+0000","last_change":"2024-03-06T07:22:05.276323+0000","last_active":"2024-03-06T07:22:07.324536+0000","last_peered":"2024-03-06T07:22:07.324536+0000","last_clean":"2024-03-06T07:22:07.324536+0000","last_became_active":"2024-03-06T07:22:05.276045+0000","last_became_peered":"2024-03-06T07:22:05.276045+0000","last_unstale":"2024-03-06T07:22:07.324536+0000","last_undegraded":"2024-03-06T07:22:07.324536+0000","last_fullsized":"2024-03-06T07:22:07.324536+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:51.610768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324488+0000","last_change":"2024-03-06T07:22:06.280686+0000","last_active":"2024-03-06T07:22:07.324488+0000","last_peered":"2024-03-06T07:22:07.324488+0000","last_clean":"2024-03-06T07:22:07.324488+0000","last_became_active":"2024-03-06T07:22:06.280079+0000","last_became_peered":"2024-03-06T07:22:06.280079+0000","last_unstale":"2024-03-06T07:22:07.324488+0000","last_undegraded":"2024-03-06T07:22:07.324488+0000","last_fullsized":"2024-03-06T07:22:07.324488+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.906790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323933+0000","last_change":"2024-03-06T07:22:05.272839+0000","last_active":"2024-03-06T07:22:07.323933+0000","last_peered":"2024-03-06T07:22:07.323933+0000","last_clean":"2024-03-06T07:22:07.323933+0000","last_became_active":"2024-03-06T07:22:05.272588+0000","last_became_peered":"2024-03-06T07:22:05.272588+0000","last_unstale":"2024-03-06T07:22:07.323933+0000","last_undegraded":"2024-03-06T07:22:07.323933+0000","last_fullsized":"2024-03-06T07:22:07.323933+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:31.818733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273488+0000","last_change":"2024-03-06T07:22:06.281798+0000","last_active":"2024-03-06T07:22:07.273488+0000","last_peered":"2024-03-06T07:22:07.273488+0000","last_clean":"2024-03-06T07:22:07.273488+0000","last_became_active":"2024-03-06T07:22:06.281285+0000","last_became_peered":"2024-03-06T07:22:06.281285+0000","last_unstale":"2024-03-06T07:22:07.273488+0000","last_undegraded":"2024-03-06T07:22:07.273488+0000","last_fullsized":"2024-03-06T07:22:07.273488+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:00.533569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324658+0000","last_change":"2024-03-06T07:22:05.275597+0000","last_active":"2024-03-06T07:22:07.324658+0000","last_peered":"2024-03-06T07:22:07.324658+0000","last_clean":"2024-03-06T07:22:07.324658+0000","last_became_active":"2024-03-06T07:22:05.274043+0000","last_became_peered":"2024-03-06T07:22:05.274043+0000","last_unstale":"2024-03-06T07:22:07.324658+0000","last_undegraded":"2024-03-06T07:22:07.324658+0000","last_fullsized":"2024-03-06T07:22:07.324658+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:09.538606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324807+0000","last_change":"2024-03-06T07:22:06.283922+0000","last_active":"2024-03-06T07:22:07.324807+0000","last_peered":"2024-03-06T07:22:07.324807+0000","last_clean":"2024-03-06T07:22:07.324807+0000","last_became_active":"2024-03-06T07:22:06.283634+0000","last_became_peered":"2024-03-06T07:22:06.283634+0000","last_unstale":"2024-03-06T07:22:07.324807+0000","last_undegraded":"2024-03-06T07:22:07.324807+0000","last_fullsized":"2024-03-06T07:22:07.324807+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:46.420402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325246+0000","last_change":"2024-03-06T07:22:05.277243+0000","last_active":"2024-03-06T07:22:07.325246+0000","last_peered":"2024-03-06T07:22:07.325246+0000","last_clean":"2024-03-06T07:22:07.325246+0000","last_became_active":"2024-03-06T07:22:05.276996+0000","last_became_peered":"2024-03-06T07:22:05.276996+0000","last_unstale":"2024-03-06T07:22:07.325246+0000","last_undegraded":"2024-03-06T07:22:07.325246+0000","last_fullsized":"2024-03-06T07:22:07.325246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:03.663384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272379+0000","last_change":"2024-03-06T07:22:06.275074+0000","last_active":"2024-03-06T07:22:07.272379+0000","last_peered":"2024-03-06T07:22:07.272379+0000","last_clean":"2024-03-06T07:22:07.272379+0000","last_became_active":"2024-03-06T07:22:06.274819+0000","last_became_peered":"2024-03-06T07:22:06.274819+0000","last_unstale":"2024-03-06T07:22:07.272379+0000","last_undegraded":"2024-03-06T07:22:07.272379+0000","last_fullsized":"2024-03-06T07:22:07.272379+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:05.783295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324480+0000","last_change":"2024-03-06T07:22:05.272234+0000","last_active":"2024-03-06T07:22:07.324480+0000","last_peered":"2024-03-06T07:22:07.324480+0000","last_clean":"2024-03-06T07:22:07.324480+0000","last_became_active":"2024-03-06T07:22:05.271944+0000","last_became_peered":"2024-03-06T07:22:05.271944+0000","last_unstale":"2024-03-06T07:22:07.324480+0000","last_undegraded":"2024-03-06T07:22:07.324480+0000","last_fullsized":"2024-03-06T07:22:07.324480+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:01.806336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324005+0000","last_change":"2024-03-06T07:22:06.275296+0000","last_active":"2024-03-06T07:22:07.324005+0000","last_peered":"2024-03-06T07:22:07.324005+0000","last_clean":"2024-03-06T07:22:07.324005+0000","last_became_active":"2024-03-06T07:22:06.275041+0000","last_became_peered":"2024-03-06T07:22:06.275041+0000","last_unstale":"2024-03-06T07:22:07.324005+0000","last_undegraded":"2024-03-06T07:22:07.324005+0000","last_fullsized":"2024-03-06T07:22:07.324005+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:10.302008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324600+0000","last_change":"2024-03-06T07:22:05.276314+0000","last_active":"2024-03-06T07:22:07.324600+0000","last_peered":"2024-03-06T07:22:07.324600+0000","last_clean":"2024-03-06T07:22:07.324600+0000","last_became_active":"2024-03-06T07:22:05.276059+0000","last_became_peered":"2024-03-06T07:22:05.276059+0000","last_unstale":"2024-03-06T07:22:07.324600+0000","last_undegraded":"2024-03-06T07:22:07.324600+0000","last_fullsized":"2024-03-06T07:22:07.324600+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:23.337554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274146+0000","last_change":"2024-03-06T07:22:06.273627+0000","last_active":"2024-03-06T07:22:07.274146+0000","last_peered":"2024-03-06T07:22:07.274146+0000","last_clean":"2024-03-06T07:22:07.274146+0000","last_became_active":"2024-03-06T07:22:06.273328+0000","last_became_peered":"2024-03-06T07:22:06.273328+0000","last_unstale":"2024-03-06T07:22:07.274146+0000","last_undegraded":"2024-03-06T07:22:07.274146+0000","last_fullsized":"2024-03-06T07:22:07.274146+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:32.869301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272620+0000","last_change":"2024-03-06T07:22:05.275558+0000","last_active":"2024-03-06T07:22:07.272620+0000","last_peered":"2024-03-06T07:22:07.272620+0000","last_clean":"2024-03-06T07:22:07.272620+0000","last_became_active":"2024-03-06T07:22:05.275307+0000","last_became_peered":"2024-03-06T07:22:05.275307+0000","last_unstale":"2024-03-06T07:22:07.272620+0000","last_undegraded":"2024-03-06T07:22:07.272620+0000","last_fullsized":"2024-03-06T07:22:07.272620+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:56.088092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324463+0000","last_change":"2024-03-06T07:22:06.282996+0000","last_active":"2024-03-06T07:22:07.324463+0000","last_peered":"2024-03-06T07:22:07.324463+0000","last_clean":"2024-03-06T07:22:07.324463+0000","last_became_active":"2024-03-06T07:22:06.282325+0000","last_became_peered":"2024-03-06T07:22:06.282325+0000","last_unstale":"2024-03-06T07:22:07.324463+0000","last_undegraded":"2024-03-06T07:22:07.324463+0000","last_fullsized":"2024-03-06T07:22:07.324463+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:45.997367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324194+0000","last_change":"2024-03-06T07:22:05.278909+0000","last_active":"2024-03-06T07:22:07.324194+0000","last_peered":"2024-03-06T07:22:07.324194+0000","last_clean":"2024-03-06T07:22:07.324194+0000","last_became_active":"2024-03-06T07:22:05.278643+0000","last_became_peered":"2024-03-06T07:22:05.278643+0000","last_unstale":"2024-03-06T07:22:07.324194+0000","last_undegraded":"2024-03-06T07:22:07.324194+0000","last_fullsized":"2024-03-06T07:22:07.324194+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:39.428768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324610+0000","last_change":"2024-03-06T07:22:06.290910+0000","last_active":"2024-03-06T07:22:07.324610+0000","last_peered":"2024-03-06T07:22:07.324610+0000","last_clean":"2024-03-06T07:22:07.324610+0000","last_became_active":"2024-03-06T07:22:06.290638+0000","last_became_peered":"2024-03-06T07:22:06.290638+0000","last_unstale":"2024-03-06T07:22:07.324610+0000","last_undegraded":"2024-03-06T07:22:07.324610+0000","last_fullsized":"2024-03-06T07:22:07.324610+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:55.574795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324030+0000","last_change":"2024-03-06T07:22:05.276401+0000","last_active":"2024-03-06T07:22:07.324030+0000","last_peered":"2024-03-06T07:22:07.324030+0000","last_clean":"2024-03-06T07:22:07.324030+0000","last_became_active":"2024-03-06T07:22:05.276140+0000","last_became_peered":"2024-03-06T07:22:05.276140+0000","last_unstale":"2024-03-06T07:22:07.324030+0000","last_undegraded":"2024-03-06T07:22:07.324030+0000","last_fullsized":"2024-03-06T07:22:07.324030+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:00.063577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323649+0000","last_change":"2024-03-06T07:22:06.282249+0000","last_active":"2024-03-06T07:22:07.323649+0000","last_peered":"2024-03-06T07:22:07.323649+0000","last_clean":"2024-03-06T07:22:07.323649+0000","last_became_active":"2024-03-06T07:22:06.281968+0000","last_became_peered":"2024-03-06T07:22:06.281968+0000","last_unstale":"2024-03-06T07:22:07.323649+0000","last_undegraded":"2024-03-06T07:22:07.323649+0000","last_fullsized":"2024-03-06T07:22:07.323649+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:20.517880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323864+0000","last_change":"2024-03-06T07:22:05.275874+0000","last_active":"2024-03-06T07:22:07.323864+0000","last_peered":"2024-03-06T07:22:07.323864+0000","last_clean":"2024-03-06T07:22:07.323864+0000","last_became_active":"2024-03-06T07:22:05.273346+0000","last_became_peered":"2024-03-06T07:22:05.273346+0000","last_unstale":"2024-03-06T07:22:07.323864+0000","last_undegraded":"2024-03-06T07:22:07.323864+0000","last_fullsized":"2024-03-06T07:22:07.323864+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:26.467203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324252+0000","last_change":"2024-03-06T07:22:06.279486+0000","last_active":"2024-03-06T07:22:07.324252+0000","last_peered":"2024-03-06T07:22:07.324252+0000","last_clean":"2024-03-06T07:22:07.324252+0000","last_became_active":"2024-03-06T07:22:06.279191+0000","last_became_peered":"2024-03-06T07:22:06.279191+0000","last_unstale":"2024-03-06T07:22:07.324252+0000","last_undegraded":"2024-03-06T07:22:07.324252+0000","last_fullsized":"2024-03-06T07:22:07.324252+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:14.577314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323908+0000","last_change":"2024-03-06T07:22:05.278429+0000","last_active":"2024-03-06T07:22:07.323908+0000","last_peered":"2024-03-06T07:22:07.323908+0000","last_clean":"2024-03-06T07:22:07.323908+0000","last_became_active":"2024-03-06T07:22:05.278167+0000","last_became_peered":"2024-03-06T07:22:05.278167+0000","last_unstale":"2024-03-06T07:22:07.323908+0000","last_undegraded":"2024-03-06T07:22:07.323908+0000","last_fullsized":"2024-03-06T07:22:07.323908+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:31.864194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325204+0000","last_change":"2024-03-06T07:22:06.281426+0000","last_active":"2024-03-06T07:22:07.325204+0000","last_peered":"2024-03-06T07:22:07.325204+0000","last_clean":"2024-03-06T07:22:07.325204+0000","last_became_active":"2024-03-06T07:22:06.281025+0000","last_became_peered":"2024-03-06T07:22:06.281025+0000","last_unstale":"2024-03-06T07:22:07.325204+0000","last_undegraded":"2024-03-06T07:22:07.325204+0000","last_fullsized":"2024-03-06T07:22:07.325204+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:29.084213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324566+0000","last_change":"2024-03-06T07:22:05.264875+0000","last_active":"2024-03-06T07:22:07.324566+0000","last_peered":"2024-03-06T07:22:07.324566+0000","last_clean":"2024-03-06T07:22:07.324566+0000","last_became_active":"2024-03-06T07:22:05.264500+0000","last_became_peered":"2024-03-06T07:22:05.264500+0000","last_unstale":"2024-03-06T07:22:07.324566+0000","last_undegraded":"2024-03-06T07:22:07.324566+0000","last_fullsized":"2024-03-06T07:22:07.324566+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:45.415295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324265+0000","last_change":"2024-03-06T07:22:06.281551+0000","last_active":"2024-03-06T07:22:07.324265+0000","last_peered":"2024-03-06T07:22:07.324265+0000","last_clean":"2024-03-06T07:22:07.324265+0000","last_became_active":"2024-03-06T07:22:06.281257+0000","last_became_peered":"2024-03-06T07:22:06.281257+0000","last_unstale":"2024-03-06T07:22:07.324265+0000","last_undegraded":"2024-03-06T07:22:07.324265+0000","last_fullsized":"2024-03-06T07:22:07.324265+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:46.420402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323662+0000","last_change":"2024-03-06T07:22:05.275590+0000","last_active":"2024-03-06T07:22:07.323662+0000","last_peered":"2024-03-06T07:22:07.323662+0000","last_clean":"2024-03-06T07:22:07.323662+0000","last_became_active":"2024-03-06T07:22:05.275317+0000","last_became_peered":"2024-03-06T07:22:05.275317+0000","last_unstale":"2024-03-06T07:22:07.323662+0000","last_undegraded":"2024-03-06T07:22:07.323662+0000","last_fullsized":"2024-03-06T07:22:07.323662+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:10.303863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325056+0000","last_change":"2024-03-06T07:22:06.273592+0000","last_active":"2024-03-06T07:22:07.325056+0000","last_peered":"2024-03-06T07:22:07.325056+0000","last_clean":"2024-03-06T07:22:07.325056+0000","last_became_active":"2024-03-06T07:22:06.273190+0000","last_became_peered":"2024-03-06T07:22:06.273190+0000","last_unstale":"2024-03-06T07:22:07.325056+0000","last_undegraded":"2024-03-06T07:22:07.325056+0000","last_fullsized":"2024-03-06T07:22:07.325056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:20.668909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324678+0000","last_change":"2024-03-06T07:22:05.275628+0000","last_active":"2024-03-06T07:22:07.324678+0000","last_peered":"2024-03-06T07:22:07.324678+0000","last_clean":"2024-03-06T07:22:07.324678+0000","last_became_active":"2024-03-06T07:22:05.275226+0000","last_became_peered":"2024-03-06T07:22:05.275226+0000","last_unstale":"2024-03-06T07:22:07.324678+0000","last_undegraded":"2024-03-06T07:22:07.324678+0000","last_fullsized":"2024-03-06T07:22:07.324678+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:47.908766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324255+0000","last_change":"2024-03-06T07:22:06.285329+0000","last_active":"2024-03-06T07:22:07.324255+0000","last_peered":"2024-03-06T07:22:07.324255+0000","last_clean":"2024-03-06T07:22:07.324255+0000","last_became_active":"2024-03-06T07:22:06.285067+0000","last_became_peered":"2024-03-06T07:22:06.285067+0000","last_unstale":"2024-03-06T07:22:07.324255+0000","last_undegraded":"2024-03-06T07:22:07.324255+0000","last_fullsized":"2024-03-06T07:22:07.324255+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:46.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273806+0000","last_change":"2024-03-06T07:22:05.277565+0000","last_active":"2024-03-06T07:22:07.273806+0000","last_peered":"2024-03-06T07:22:07.273806+0000","last_clean":"2024-03-06T07:22:07.273806+0000","last_became_active":"2024-03-06T07:22:05.276867+0000","last_became_peered":"2024-03-06T07:22:05.276867+0000","last_unstale":"2024-03-06T07:22:07.273806+0000","last_undegraded":"2024-03-06T07:22:07.273806+0000","last_fullsized":"2024-03-06T07:22:07.273806+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:21.547573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324410+0000","last_change":"2024-03-06T07:22:06.289399+0000","last_active":"2024-03-06T07:22:07.324410+0000","last_peered":"2024-03-06T07:22:07.324410+0000","last_clean":"2024-03-06T07:22:07.324410+0000","last_became_active":"2024-03-06T07:22:06.289121+0000","last_became_peered":"2024-03-06T07:22:06.289121+0000","last_unstale":"2024-03-06T07:22:07.324410+0000","last_undegraded":"2024-03-06T07:22:07.324410+0000","last_fullsized":"2024-03-06T07:22:07.324410+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:45.514208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273657+0000","last_change":"2024-03-06T07:22:05.275590+0000","last_active":"2024-03-06T07:22:07.273657+0000","last_peered":"2024-03-06T07:22:07.273657+0000","last_clean":"2024-03-06T07:22:07.273657+0000","last_became_active":"2024-03-06T07:22:05.275287+0000","last_became_peered":"2024-03-06T07:22:05.275287+0000","last_unstale":"2024-03-06T07:22:07.273657+0000","last_undegraded":"2024-03-06T07:22:07.273657+0000","last_fullsized":"2024-03-06T07:22:07.273657+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.526750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272557+0000","last_change":"2024-03-06T07:22:06.276631+0000","last_active":"2024-03-06T07:22:07.272557+0000","last_peered":"2024-03-06T07:22:07.272557+0000","last_clean":"2024-03-06T07:22:07.272557+0000","last_became_active":"2024-03-06T07:22:06.276329+0000","last_became_peered":"2024-03-06T07:22:06.276329+0000","last_unstale":"2024-03-06T07:22:07.272557+0000","last_undegraded":"2024-03-06T07:22:07.272557+0000","last_fullsized":"2024-03-06T07:22:07.272557+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:53.220165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325138+0000","last_change":"2024-03-06T07:22:05.270328+0000","last_active":"2024-03-06T07:22:07.325138+0000","last_peered":"2024-03-06T07:22:07.325138+0000","last_clean":"2024-03-06T07:22:07.325138+0000","last_became_active":"2024-03-06T07:22:05.270021+0000","last_became_peered":"2024-03-06T07:22:05.270021+0000","last_unstale":"2024-03-06T07:22:07.325138+0000","last_undegraded":"2024-03-06T07:22:07.325138+0000","last_fullsized":"2024-03-06T07:22:07.325138+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:13.572207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323890+0000","last_change":"2024-03-06T07:22:06.274953+0000","last_active":"2024-03-06T07:22:07.323890+0000","last_peered":"2024-03-06T07:22:07.323890+0000","last_clean":"2024-03-06T07:22:07.323890+0000","last_became_active":"2024-03-06T07:22:06.274702+0000","last_became_peered":"2024-03-06T07:22:06.274702+0000","last_unstale":"2024-03-06T07:22:07.323890+0000","last_undegraded":"2024-03-06T07:22:07.323890+0000","last_fullsized":"2024-03-06T07:22:07.323890+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:48.093500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274103+0000","last_change":"2024-03-06T07:22:05.271821+0000","last_active":"2024-03-06T07:22:07.274103+0000","last_peered":"2024-03-06T07:22:07.274103+0000","last_clean":"2024-03-06T07:22:07.274103+0000","last_became_active":"2024-03-06T07:22:05.271340+0000","last_became_peered":"2024-03-06T07:22:05.271340+0000","last_unstale":"2024-03-06T07:22:07.274103+0000","last_undegraded":"2024-03-06T07:22:07.274103+0000","last_fullsized":"2024-03-06T07:22:07.274103+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:21.547573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324391+0000","last_change":"2024-03-06T07:22:06.282198+0000","last_active":"2024-03-06T07:22:07.324391+0000","last_peered":"2024-03-06T07:22:07.324391+0000","last_clean":"2024-03-06T07:22:07.324391+0000","last_became_active":"2024-03-06T07:22:06.281940+0000","last_became_peered":"2024-03-06T07:22:06.281940+0000","last_unstale":"2024-03-06T07:22:07.324391+0000","last_undegraded":"2024-03-06T07:22:07.324391+0000","last_fullsized":"2024-03-06T07:22:07.324391+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:31.566918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273851+0000","last_change":"2024-03-06T07:22:05.274943+0000","last_active":"2024-03-06T07:22:07.273851+0000","last_peered":"2024-03-06T07:22:07.273851+0000","last_clean":"2024-03-06T07:22:07.273851+0000","last_became_active":"2024-03-06T07:22:05.272239+0000","last_became_peered":"2024-03-06T07:22:05.272239+0000","last_unstale":"2024-03-06T07:22:07.273851+0000","last_undegraded":"2024-03-06T07:22:07.273851+0000","last_fullsized":"2024-03-06T07:22:07.273851+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:54.446300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324017+0000","last_change":"2024-03-06T07:22:06.283287+0000","last_active":"2024-03-06T07:22:07.324017+0000","last_peered":"2024-03-06T07:22:07.324017+0000","last_clean":"2024-03-06T07:22:07.324017+0000","last_became_active":"2024-03-06T07:22:06.283024+0000","last_became_peered":"2024-03-06T07:22:06.283024+0000","last_unstale":"2024-03-06T07:22:07.324017+0000","last_undegraded":"2024-03-06T07:22:07.324017+0000","last_fullsized":"2024-03-06T07:22:07.324017+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:55.574795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323732+0000","last_change":"2024-03-06T07:22:05.270394+0000","last_active":"2024-03-06T07:22:07.323732+0000","last_peered":"2024-03-06T07:22:07.323732+0000","last_clean":"2024-03-06T07:22:07.323732+0000","last_became_active":"2024-03-06T07:22:05.270110+0000","last_became_peered":"2024-03-06T07:22:05.270110+0000","last_unstale":"2024-03-06T07:22:07.323732+0000","last_undegraded":"2024-03-06T07:22:07.323732+0000","last_fullsized":"2024-03-06T07:22:07.323732+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:14.806818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324544+0000","last_change":"2024-03-06T07:22:06.288799+0000","last_active":"2024-03-06T07:22:07.324544+0000","last_peered":"2024-03-06T07:22:07.324544+0000","last_clean":"2024-03-06T07:22:07.324544+0000","last_became_active":"2024-03-06T07:22:06.288501+0000","last_became_peered":"2024-03-06T07:22:06.288501+0000","last_unstale":"2024-03-06T07:22:07.324544+0000","last_undegraded":"2024-03-06T07:22:07.324544+0000","last_fullsized":"2024-03-06T07:22:07.324544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:10.543713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323687+0000","last_change":"2024-03-06T07:22:05.276925+0000","last_active":"2024-03-06T07:22:07.323687+0000","last_peered":"2024-03-06T07:22:07.323687+0000","last_clean":"2024-03-06T07:22:07.323687+0000","last_became_active":"2024-03-06T07:22:05.276594+0000","last_became_peered":"2024-03-06T07:22:05.276594+0000","last_unstale":"2024-03-06T07:22:07.323687+0000","last_undegraded":"2024-03-06T07:22:07.323687+0000","last_fullsized":"2024-03-06T07:22:07.323687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:45.596793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324099+0000","last_change":"2024-03-06T07:22:06.287902+0000","last_active":"2024-03-06T07:22:07.324099+0000","last_peered":"2024-03-06T07:22:07.324099+0000","last_clean":"2024-03-06T07:22:07.324099+0000","last_became_active":"2024-03-06T07:22:06.287635+0000","last_became_peered":"2024-03-06T07:22:06.287635+0000","last_unstale":"2024-03-06T07:22:07.324099+0000","last_undegraded":"2024-03-06T07:22:07.324099+0000","last_fullsized":"2024-03-06T07:22:07.324099+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:28.209748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323853+0000","last_change":"2024-03-06T07:22:05.275435+0000","last_active":"2024-03-06T07:22:07.323853+0000","last_peered":"2024-03-06T07:22:07.323853+0000","last_clean":"2024-03-06T07:22:07.323853+0000","last_became_active":"2024-03-06T07:22:05.275199+0000","last_became_peered":"2024-03-06T07:22:05.275199+0000","last_unstale":"2024-03-06T07:22:07.323853+0000","last_undegraded":"2024-03-06T07:22:07.323853+0000","last_fullsized":"2024-03-06T07:22:07.323853+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:09.296901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":337,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323801+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:22:07.323801+0000","last_peered":"2024-03-06T07:22:07.323801+0000","last_clean":"2024-03-06T07:22:07.323801+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:22:07.323801+0000","last_undegraded":"2024-03-06T07:22:07.323801+0000","last_fullsized":"2024-03-06T07:22:07.323801+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1417216,"data_stored":1233527,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1351680,"data_stored":649657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1343488,"data_stored":642543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":643995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1363968,"data_stored":650721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32488,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339352,"statfs":{"total":96636764160,"available":96603496448,"internally_reserved":0,"allocated":1335296,"data_stored":642383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1339392,"data_stored":642405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":1232773,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:10.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:10.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.362+0000 7f7d53bbc640 1 Processor -- start 2024-03-06T07:22:10.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.363+0000 7f7d53bbc640 1 -- start start 2024-03-06T07:22:10.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.363+0000 7f7d53bbc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7d4c06f100 0x7f7d4c06b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.363+0000 7f7d53bbc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c0a70f0 con 0x7f7d4c1536c0 2024-03-06T07:22:10.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.363+0000 7f7d53bbc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c1567f0 con 0x7f7d4c06afc0 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.363+0000 7f7d53bbc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c150a20 con 0x7f7d4c06f100 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.363+0000 7f7d51157640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7d4c1536c0 0x7f7d4c15b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54684/0 (socket says 172.21.15.94:54684) 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.363+0000 7f7d51157640 1 -- 172.21.15.94:0/969091202 learned_addr learned my addr 172.21.15.94:0/969091202 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d52159640 1 --2- 172.21.15.94:0/969091202 >> v2:172.21.15.145:3300/0 conn(0x7f7d4c06f100 0x7f7d4c06b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d50956640 1 -- 172.21.15.94:0/969091202 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348190168 0 0) 0x7f7d4c0a70f0 con 0x7f7d4c1536c0 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d50956640 1 -- 172.21.15.94:0/969091202 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d38003520 con 0x7f7d4c1536c0 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d50956640 1 -- 172.21.15.94:0/969091202 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910073007 0 0) 0x7f7d4c1567f0 con 0x7f7d4c06afc0 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d50956640 1 -- 172.21.15.94:0/969091202 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c0a70f0 con 0x7f7d4c06afc0 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d52159640 1 -- 172.21.15.94:0/969091202 >> v1:172.21.15.145:6789/0 conn(0x7f7d4c06afc0 legacy=0x7f7d4c152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d52159640 1 -- 172.21.15.94:0/969091202 >> v1:172.21.15.94:6789/0 conn(0x7f7d4c1536c0 legacy=0x7f7d4c15b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.364+0000 7f7d52159640 1 -- 172.21.15.94:0/969091202 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c156ad0 con 0x7f7d4c06f100 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d52159640 1 --2- 172.21.15.94:0/969091202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c06f100 0x7f7d4c06b3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009d90 tx=0x7f7d4402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c1b08a2554f71e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d50956640 1 -- 172.21.15.94:0/969091202 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44039070 con 0x7f7d4c06f100 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d50956640 1 -- 172.21.15.94:0/969091202 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4402cbd0 con 0x7f7d4c06f100 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d50956640 1 -- 172.21.15.94:0/969091202 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44034440 con 0x7f7d4c06f100 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d53bbc640 1 -- 172.21.15.94:0/969091202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c06f100 msgr2=0x7f7d4c06b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/969091202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c06f100 0x7f7d4c06b3a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009d90 tx=0x7f7d4402be20 comp rx=0 tx=0).stop 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d53bbc640 1 -- 172.21.15.94:0/969091202 shutdown_connections 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/969091202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c06f100 0x7f7d4c06b3a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d53bbc640 1 -- 172.21.15.94:0/969091202 >> 172.21.15.94:0/969091202 conn(0x7f7d4c0c4290 msgr2=0x7f7d4c0c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d53bbc640 1 -- 172.21.15.94:0/969091202 shutdown_connections 2024-03-06T07:22:10.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.365+0000 7f7d53bbc640 1 -- 172.21.15.94:0/969091202 wait complete. 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.366+0000 7f7d53bbc640 1 Processor -- start 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.366+0000 7f7d53bbc640 1 -- start start 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d53bbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4c06afc0 0x7f7d4c06c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d53bbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d4c06f100 0x7f7d4c1e9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d53bbc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c1536c0 0x7f7d4c1ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d53bbc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c06f530 con 0x7f7d4c06f100 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d53bbc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c1565a0 con 0x7f7d4c1536c0 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d53bbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0c7360 con 0x7f7d4c06afc0 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d51157640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d4c06f100 0x7f7d4c1e9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d51157640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d4c06f100 0x7f7d4c1e9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33516/0 (socket says 172.21.15.94:33516) 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d51157640 1 -- 172.21.15.94:0/1362245034 learned_addr learned my addr 172.21.15.94:0/1362245034 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d51958640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4c06afc0 0x7f7d4c06c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.367+0000 7f7d52159640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c1536c0 0x7f7d4c1ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d51157640 1 -- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4c06afc0 msgr2=0x7f7d4c06c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d51157640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4c06afc0 0x7f7d4c06c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d51157640 1 -- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c1536c0 msgr2=0x7f7d4c1ea4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d51157640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c1536c0 0x7f7d4c1ea4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d51157640 1 -- 172.21.15.94:0/1362245034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c1ec000 con 0x7f7d4c06f100 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d51157640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d4c06f100 0x7f7d4c1e9f60 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c410 tx=0x7f7d3c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d427fc640 1 -- 172.21.15.94:0/1362245034 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c017070 con 0x7f7d4c06f100 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d427fc640 1 -- 172.21.15.94:0/1362245034 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c005ce0 con 0x7f7d4c06f100 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c1ec2b0 con 0x7f7d4c06f100 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d427fc640 1 -- 172.21.15.94:0/1362245034 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0072f0 con 0x7f7d4c06f100 2024-03-06T07:22:10.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.368+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c1ec870 con 0x7f7d4c06f100 2024-03-06T07:22:10.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.369+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c04f330 con 0x7f7d4c06f100 2024-03-06T07:22:10.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.369+0000 7f7d427fc640 1 -- 172.21.15.94:0/1362245034 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d3c01a070 con 0x7f7d4c06f100 2024-03-06T07:22:10.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.370+0000 7f7d427fc640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7d3006b4d0 0x7f7d3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.370+0000 7f7d427fc640 1 -- 172.21.15.94:0/1362245034 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d3c089670 con 0x7f7d4c06f100 2024-03-06T07:22:10.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.370+0000 7f7d51958640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7d3006b4d0 0x7f7d3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.370+0000 7f7d51958640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7d3006b4d0 0x7f7d3006d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7d48007fd0 tx=0x7f7d48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.374+0000 7f7d427fc640 1 -- 172.21.15.94:0/1362245034 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d3c057e60 con 0x7f7d4c06f100 2024-03-06T07:22:10.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.543+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d4c06ece0 con 0x7f7d3006b4d0 2024-03-06T07:22:10.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.549+0000 7f7d427fc640 1 -- 172.21.15.94:0/1362245034 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252562 (secure 0 0 0) 0x7f7d4c06ece0 con 0x7f7d3006b4d0 2024-03-06T07:22:10.554 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:10.555 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:22:10.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7d3006b4d0 msgr2=0x7f7d3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7d3006b4d0 0x7f7d3006d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7d48007fd0 tx=0x7f7d48009040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d4c06f100 msgr2=0x7f7d4c1e9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d4c06f100 0x7f7d4c1e9f60 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c410 tx=0x7f7d3c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 shutdown_connections 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7d3006b4d0 0x7f7d3006d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4c1536c0 0x7f7d4c1ea4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d4c06f100 0x7f7d4c1e9f60 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 --2- 172.21.15.94:0/1362245034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4c06afc0 0x7f7d4c06c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 >> 172.21.15.94:0/1362245034 conn(0x7f7d4c0c4290 msgr2=0x7f7d4c154c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 shutdown_connections 2024-03-06T07:22:10.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.552+0000 7f7d53bbc640 1 -- 172.21.15.94:0/1362245034 wait complete. 2024-03-06T07:22:10.565 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-03-06T07:22:08.684374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260140,"kb_used_data":10648,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714580,"statfs":{"total":773094113280,"available":772827729920,"internally_reserved":0,"allocated":10903552,"data_stored":6338004,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.435909"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274390+0000","last_change":"2024-03-06T07:22:06.274189+0000","last_active":"2024-03-06T07:22:07.274390+0000","last_peered":"2024-03-06T07:22:07.274390+0000","last_clean":"2024-03-06T07:22:07.274390+0000","last_became_active":"2024-03-06T07:22:06.273926+0000","last_became_peered":"2024-03-06T07:22:06.273926+0000","last_unstale":"2024-03-06T07:22:07.274390+0000","last_undegraded":"2024-03-06T07:22:07.274390+0000","last_fullsized":"2024-03-06T07:22:07.274390+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:46.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324168+0000","last_change":"2024-03-06T07:22:06.285476+0000","last_active":"2024-03-06T07:22:07.324168+0000","last_peered":"2024-03-06T07:22:07.324168+0000","last_clean":"2024-03-06T07:22:07.324168+0000","last_became_active":"2024-03-06T07:22:06.284773+0000","last_became_peered":"2024-03-06T07:22:06.284773+0000","last_unstale":"2024-03-06T07:22:07.324168+0000","last_undegraded":"2024-03-06T07:22:07.324168+0000","last_fullsized":"2024-03-06T07:22:07.324168+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:36.281548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324075+0000","last_change":"2024-03-06T07:22:06.283289+0000","last_active":"2024-03-06T07:22:07.324075+0000","last_peered":"2024-03-06T07:22:07.324075+0000","last_clean":"2024-03-06T07:22:07.324075+0000","last_became_active":"2024-03-06T07:22:06.283018+0000","last_became_peered":"2024-03-06T07:22:06.283018+0000","last_unstale":"2024-03-06T07:22:07.324075+0000","last_undegraded":"2024-03-06T07:22:07.324075+0000","last_fullsized":"2024-03-06T07:22:07.324075+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:15.811925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323941+0000","last_change":"2024-03-06T07:22:06.283187+0000","last_active":"2024-03-06T07:22:07.323941+0000","last_peered":"2024-03-06T07:22:07.323941+0000","last_clean":"2024-03-06T07:22:07.323941+0000","last_became_active":"2024-03-06T07:22:06.282893+0000","last_became_peered":"2024-03-06T07:22:06.282893+0000","last_unstale":"2024-03-06T07:22:07.323941+0000","last_undegraded":"2024-03-06T07:22:07.323941+0000","last_fullsized":"2024-03-06T07:22:07.323941+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:02.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273919+0000","last_change":"2024-03-06T07:22:06.284103+0000","last_active":"2024-03-06T07:22:07.273919+0000","last_peered":"2024-03-06T07:22:07.273919+0000","last_clean":"2024-03-06T07:22:07.273919+0000","last_became_active":"2024-03-06T07:22:06.283840+0000","last_became_peered":"2024-03-06T07:22:06.283840+0000","last_unstale":"2024-03-06T07:22:07.273919+0000","last_undegraded":"2024-03-06T07:22:07.273919+0000","last_fullsized":"2024-03-06T07:22:07.273919+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:58.629934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323872+0000","last_change":"2024-03-06T07:22:06.267539+0000","last_active":"2024-03-06T07:22:07.323872+0000","last_peered":"2024-03-06T07:22:07.323872+0000","last_clean":"2024-03-06T07:22:07.323872+0000","last_became_active":"2024-03-06T07:22:06.267202+0000","last_became_peered":"2024-03-06T07:22:06.267202+0000","last_unstale":"2024-03-06T07:22:07.323872+0000","last_undegraded":"2024-03-06T07:22:07.323872+0000","last_fullsized":"2024-03-06T07:22:07.323872+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:32.128497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323698+0000","last_change":"2024-03-06T07:22:06.286083+0000","last_active":"2024-03-06T07:22:07.323698+0000","last_peered":"2024-03-06T07:22:07.323698+0000","last_clean":"2024-03-06T07:22:07.323698+0000","last_became_active":"2024-03-06T07:22:06.285818+0000","last_became_peered":"2024-03-06T07:22:06.285818+0000","last_unstale":"2024-03-06T07:22:07.323698+0000","last_undegraded":"2024-03-06T07:22:07.323698+0000","last_fullsized":"2024-03-06T07:22:07.323698+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:05.683743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323651+0000","last_change":"2024-03-06T07:22:06.286474+0000","last_active":"2024-03-06T07:22:07.323651+0000","last_peered":"2024-03-06T07:22:07.323651+0000","last_clean":"2024-03-06T07:22:07.323651+0000","last_became_active":"2024-03-06T07:22:06.285226+0000","last_became_peered":"2024-03-06T07:22:06.285226+0000","last_unstale":"2024-03-06T07:22:07.323651+0000","last_undegraded":"2024-03-06T07:22:07.323651+0000","last_fullsized":"2024-03-06T07:22:07.323651+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:27.472310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324389+0000","last_change":"2024-03-06T07:22:06.288562+0000","last_active":"2024-03-06T07:22:07.324389+0000","last_peered":"2024-03-06T07:22:07.324389+0000","last_clean":"2024-03-06T07:22:07.324389+0000","last_became_active":"2024-03-06T07:22:06.288282+0000","last_became_peered":"2024-03-06T07:22:06.288282+0000","last_unstale":"2024-03-06T07:22:07.324389+0000","last_undegraded":"2024-03-06T07:22:07.324389+0000","last_fullsized":"2024-03-06T07:22:07.324389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:30.644551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324421+0000","last_change":"2024-03-06T07:22:06.271898+0000","last_active":"2024-03-06T07:22:07.324421+0000","last_peered":"2024-03-06T07:22:07.324421+0000","last_clean":"2024-03-06T07:22:07.324421+0000","last_became_active":"2024-03-06T07:22:06.271643+0000","last_became_peered":"2024-03-06T07:22:06.271643+0000","last_unstale":"2024-03-06T07:22:07.324421+0000","last_undegraded":"2024-03-06T07:22:07.324421+0000","last_fullsized":"2024-03-06T07:22:07.324421+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:40.433875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274007+0000","last_change":"2024-03-06T07:22:06.279646+0000","last_active":"2024-03-06T07:22:07.274007+0000","last_peered":"2024-03-06T07:22:07.274007+0000","last_clean":"2024-03-06T07:22:07.274007+0000","last_became_active":"2024-03-06T07:22:06.279342+0000","last_became_peered":"2024-03-06T07:22:06.279342+0000","last_unstale":"2024-03-06T07:22:07.274007+0000","last_undegraded":"2024-03-06T07:22:07.274007+0000","last_fullsized":"2024-03-06T07:22:07.274007+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:01.068684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325059+0000","last_change":"2024-03-06T07:22:06.275948+0000","last_active":"2024-03-06T07:22:07.325059+0000","last_peered":"2024-03-06T07:22:07.325059+0000","last_clean":"2024-03-06T07:22:07.325059+0000","last_became_active":"2024-03-06T07:22:06.275683+0000","last_became_peered":"2024-03-06T07:22:06.275683+0000","last_unstale":"2024-03-06T07:22:07.325059+0000","last_undegraded":"2024-03-06T07:22:07.325059+0000","last_fullsized":"2024-03-06T07:22:07.325059+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:01.110061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324301+0000","last_change":"2024-03-06T07:22:06.283085+0000","last_active":"2024-03-06T07:22:07.324301+0000","last_peered":"2024-03-06T07:22:07.324301+0000","last_clean":"2024-03-06T07:22:07.324301+0000","last_became_active":"2024-03-06T07:22:06.282786+0000","last_became_peered":"2024-03-06T07:22:06.282786+0000","last_unstale":"2024-03-06T07:22:07.324301+0000","last_undegraded":"2024-03-06T07:22:07.324301+0000","last_fullsized":"2024-03-06T07:22:07.324301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:57.318529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324783+0000","last_change":"2024-03-06T07:22:06.283315+0000","last_active":"2024-03-06T07:22:07.324783+0000","last_peered":"2024-03-06T07:22:07.324783+0000","last_clean":"2024-03-06T07:22:07.324783+0000","last_became_active":"2024-03-06T07:22:06.282764+0000","last_became_peered":"2024-03-06T07:22:06.282764+0000","last_unstale":"2024-03-06T07:22:07.324783+0000","last_undegraded":"2024-03-06T07:22:07.324783+0000","last_fullsized":"2024-03-06T07:22:07.324783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:47.730533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273403+0000","last_change":"2024-03-06T07:22:06.280055+0000","last_active":"2024-03-06T07:22:07.273403+0000","last_peered":"2024-03-06T07:22:07.273403+0000","last_clean":"2024-03-06T07:22:07.273403+0000","last_became_active":"2024-03-06T07:22:06.279760+0000","last_became_peered":"2024-03-06T07:22:06.279760+0000","last_unstale":"2024-03-06T07:22:07.273403+0000","last_undegraded":"2024-03-06T07:22:07.273403+0000","last_fullsized":"2024-03-06T07:22:07.273403+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:00.533569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323660+0000","last_change":"2024-03-06T07:22:06.270471+0000","last_active":"2024-03-06T07:22:07.323660+0000","last_peered":"2024-03-06T07:22:07.323660+0000","last_clean":"2024-03-06T07:22:07.323660+0000","last_became_active":"2024-03-06T07:22:06.270190+0000","last_became_peered":"2024-03-06T07:22:06.270190+0000","last_unstale":"2024-03-06T07:22:07.323660+0000","last_undegraded":"2024-03-06T07:22:07.323660+0000","last_fullsized":"2024-03-06T07:22:07.323660+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:32.708206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324491+0000","last_change":"2024-03-06T07:22:06.277248+0000","last_active":"2024-03-06T07:22:07.324491+0000","last_peered":"2024-03-06T07:22:07.324491+0000","last_clean":"2024-03-06T07:22:07.324491+0000","last_became_active":"2024-03-06T07:22:06.276981+0000","last_became_peered":"2024-03-06T07:22:06.276981+0000","last_unstale":"2024-03-06T07:22:07.324491+0000","last_undegraded":"2024-03-06T07:22:07.324491+0000","last_fullsized":"2024-03-06T07:22:07.324491+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:10.302008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324299+0000","last_change":"2024-03-06T07:22:06.285784+0000","last_active":"2024-03-06T07:22:07.324299+0000","last_peered":"2024-03-06T07:22:07.324299+0000","last_clean":"2024-03-06T07:22:07.324299+0000","last_became_active":"2024-03-06T07:22:06.284444+0000","last_became_peered":"2024-03-06T07:22:06.284444+0000","last_unstale":"2024-03-06T07:22:07.324299+0000","last_undegraded":"2024-03-06T07:22:07.324299+0000","last_fullsized":"2024-03-06T07:22:07.324299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:54.348798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273406+0000","last_change":"2024-03-06T07:22:06.291869+0000","last_active":"2024-03-06T07:22:07.273406+0000","last_peered":"2024-03-06T07:22:07.273406+0000","last_clean":"2024-03-06T07:22:07.273406+0000","last_became_active":"2024-03-06T07:22:06.291524+0000","last_became_peered":"2024-03-06T07:22:06.291524+0000","last_unstale":"2024-03-06T07:22:07.273406+0000","last_undegraded":"2024-03-06T07:22:07.273406+0000","last_fullsized":"2024-03-06T07:22:07.273406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:43.785073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272443+0000","last_change":"2024-03-06T07:22:06.271230+0000","last_active":"2024-03-06T07:22:07.272443+0000","last_peered":"2024-03-06T07:22:07.272443+0000","last_clean":"2024-03-06T07:22:07.272443+0000","last_became_active":"2024-03-06T07:22:06.270940+0000","last_became_peered":"2024-03-06T07:22:06.270940+0000","last_unstale":"2024-03-06T07:22:07.272443+0000","last_undegraded":"2024-03-06T07:22:07.272443+0000","last_fullsized":"2024-03-06T07:22:07.272443+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:35.031046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324040+0000","last_change":"2024-03-06T07:22:06.287793+0000","last_active":"2024-03-06T07:22:07.324040+0000","last_peered":"2024-03-06T07:22:07.324040+0000","last_clean":"2024-03-06T07:22:07.324040+0000","last_became_active":"2024-03-06T07:22:06.287433+0000","last_became_peered":"2024-03-06T07:22:06.287433+0000","last_unstale":"2024-03-06T07:22:07.324040+0000","last_undegraded":"2024-03-06T07:22:07.324040+0000","last_fullsized":"2024-03-06T07:22:07.324040+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:20.360914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274597+0000","last_change":"2024-03-06T07:22:06.273316+0000","last_active":"2024-03-06T07:22:07.274597+0000","last_peered":"2024-03-06T07:22:07.274597+0000","last_clean":"2024-03-06T07:22:07.274597+0000","last_became_active":"2024-03-06T07:22:06.273014+0000","last_became_peered":"2024-03-06T07:22:06.273014+0000","last_unstale":"2024-03-06T07:22:07.274597+0000","last_undegraded":"2024-03-06T07:22:07.274597+0000","last_fullsized":"2024-03-06T07:22:07.274597+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:57.318529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324273+0000","last_change":"2024-03-06T07:22:06.287966+0000","last_active":"2024-03-06T07:22:07.324273+0000","last_peered":"2024-03-06T07:22:07.324273+0000","last_clean":"2024-03-06T07:22:07.324273+0000","last_became_active":"2024-03-06T07:22:06.287405+0000","last_became_peered":"2024-03-06T07:22:06.287405+0000","last_unstale":"2024-03-06T07:22:07.324273+0000","last_undegraded":"2024-03-06T07:22:07.324273+0000","last_fullsized":"2024-03-06T07:22:07.324273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:45.778003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324210+0000","last_change":"2024-03-06T07:22:06.286120+0000","last_active":"2024-03-06T07:22:07.324210+0000","last_peered":"2024-03-06T07:22:07.324210+0000","last_clean":"2024-03-06T07:22:07.324210+0000","last_became_active":"2024-03-06T07:22:06.285851+0000","last_became_peered":"2024-03-06T07:22:06.285851+0000","last_unstale":"2024-03-06T07:22:07.324210+0000","last_undegraded":"2024-03-06T07:22:07.324210+0000","last_fullsized":"2024-03-06T07:22:07.324210+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:02.811442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324493+0000","last_change":"2024-03-06T07:22:06.280900+0000","last_active":"2024-03-06T07:22:07.324493+0000","last_peered":"2024-03-06T07:22:07.324493+0000","last_clean":"2024-03-06T07:22:07.324493+0000","last_became_active":"2024-03-06T07:22:06.280415+0000","last_became_peered":"2024-03-06T07:22:06.280415+0000","last_unstale":"2024-03-06T07:22:07.324493+0000","last_undegraded":"2024-03-06T07:22:07.324493+0000","last_fullsized":"2024-03-06T07:22:07.324493+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:03.152066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272615+0000","last_change":"2024-03-06T07:22:06.271002+0000","last_active":"2024-03-06T07:22:07.272615+0000","last_peered":"2024-03-06T07:22:07.272615+0000","last_clean":"2024-03-06T07:22:07.272615+0000","last_became_active":"2024-03-06T07:22:06.270705+0000","last_became_peered":"2024-03-06T07:22:06.270705+0000","last_unstale":"2024-03-06T07:22:07.272615+0000","last_undegraded":"2024-03-06T07:22:07.272615+0000","last_fullsized":"2024-03-06T07:22:07.272615+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:12.812773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273555+0000","last_change":"2024-03-06T07:22:06.288907+0000","last_active":"2024-03-06T07:22:07.273555+0000","last_peered":"2024-03-06T07:22:07.273555+0000","last_clean":"2024-03-06T07:22:07.273555+0000","last_became_active":"2024-03-06T07:22:06.288635+0000","last_became_peered":"2024-03-06T07:22:06.288635+0000","last_unstale":"2024-03-06T07:22:07.273555+0000","last_undegraded":"2024-03-06T07:22:07.273555+0000","last_fullsized":"2024-03-06T07:22:07.273555+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:03.086475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324917+0000","last_change":"2024-03-06T07:22:06.275729+0000","last_active":"2024-03-06T07:22:07.324917+0000","last_peered":"2024-03-06T07:22:07.324917+0000","last_clean":"2024-03-06T07:22:07.324917+0000","last_became_active":"2024-03-06T07:22:06.275468+0000","last_became_peered":"2024-03-06T07:22:06.275468+0000","last_unstale":"2024-03-06T07:22:07.324917+0000","last_undegraded":"2024-03-06T07:22:07.324917+0000","last_fullsized":"2024-03-06T07:22:07.324917+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:23:29.775060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272558+0000","last_change":"2024-03-06T07:22:06.269348+0000","last_active":"2024-03-06T07:22:07.272558+0000","last_peered":"2024-03-06T07:22:07.272558+0000","last_clean":"2024-03-06T07:22:07.272558+0000","last_became_active":"2024-03-06T07:22:06.269049+0000","last_became_peered":"2024-03-06T07:22:06.269049+0000","last_unstale":"2024-03-06T07:22:07.272558+0000","last_undegraded":"2024-03-06T07:22:07.272558+0000","last_fullsized":"2024-03-06T07:22:07.272558+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:01.498065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324905+0000","last_change":"2024-03-06T07:22:06.276527+0000","last_active":"2024-03-06T07:22:07.324905+0000","last_peered":"2024-03-06T07:22:07.324905+0000","last_clean":"2024-03-06T07:22:07.324905+0000","last_became_active":"2024-03-06T07:22:06.276167+0000","last_became_peered":"2024-03-06T07:22:06.276167+0000","last_unstale":"2024-03-06T07:22:07.324905+0000","last_undegraded":"2024-03-06T07:22:07.324905+0000","last_fullsized":"2024-03-06T07:22:07.324905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:27.387382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324295+0000","last_change":"2024-03-06T07:22:06.274422+0000","last_active":"2024-03-06T07:22:07.324295+0000","last_peered":"2024-03-06T07:22:07.324295+0000","last_clean":"2024-03-06T07:22:07.324295+0000","last_became_active":"2024-03-06T07:22:06.274116+0000","last_became_peered":"2024-03-06T07:22:06.274116+0000","last_unstale":"2024-03-06T07:22:07.324295+0000","last_undegraded":"2024-03-06T07:22:07.324295+0000","last_fullsized":"2024-03-06T07:22:07.324295+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:32.823840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324316+0000","last_change":"2024-03-06T07:22:06.283397+0000","last_active":"2024-03-06T07:22:07.324316+0000","last_peered":"2024-03-06T07:22:07.324316+0000","last_clean":"2024-03-06T07:22:07.324316+0000","last_became_active":"2024-03-06T07:22:06.283134+0000","last_became_peered":"2024-03-06T07:22:06.283134+0000","last_unstale":"2024-03-06T07:22:07.324316+0000","last_undegraded":"2024-03-06T07:22:07.324316+0000","last_fullsized":"2024-03-06T07:22:07.324316+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:04.668491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324692+0000","last_change":"2024-03-06T07:22:06.272969+0000","last_active":"2024-03-06T07:22:07.324692+0000","last_peered":"2024-03-06T07:22:07.324692+0000","last_clean":"2024-03-06T07:22:07.324692+0000","last_became_active":"2024-03-06T07:22:06.272688+0000","last_became_peered":"2024-03-06T07:22:06.272688+0000","last_unstale":"2024-03-06T07:22:07.324692+0000","last_undegraded":"2024-03-06T07:22:07.324692+0000","last_fullsized":"2024-03-06T07:22:07.324692+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:02.811442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273206+0000","last_change":"2024-03-06T07:22:05.274770+0000","last_active":"2024-03-06T07:22:07.273206+0000","last_peered":"2024-03-06T07:22:07.273206+0000","last_clean":"2024-03-06T07:22:07.273206+0000","last_became_active":"2024-03-06T07:22:05.274459+0000","last_became_peered":"2024-03-06T07:22:05.274459+0000","last_unstale":"2024-03-06T07:22:07.273206+0000","last_undegraded":"2024-03-06T07:22:07.273206+0000","last_fullsized":"2024-03-06T07:22:07.273206+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:23.337554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324068+0000","last_change":"2024-03-06T07:22:06.267869+0000","last_active":"2024-03-06T07:22:07.324068+0000","last_peered":"2024-03-06T07:22:07.324068+0000","last_clean":"2024-03-06T07:22:07.324068+0000","last_became_active":"2024-03-06T07:22:06.267608+0000","last_became_peered":"2024-03-06T07:22:06.267608+0000","last_unstale":"2024-03-06T07:22:07.324068+0000","last_undegraded":"2024-03-06T07:22:07.324068+0000","last_fullsized":"2024-03-06T07:22:07.324068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:30.342334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324538+0000","last_change":"2024-03-06T07:22:05.269390+0000","last_active":"2024-03-06T07:22:07.324538+0000","last_peered":"2024-03-06T07:22:07.324538+0000","last_clean":"2024-03-06T07:22:07.324538+0000","last_became_active":"2024-03-06T07:22:05.269045+0000","last_became_peered":"2024-03-06T07:22:05.269045+0000","last_unstale":"2024-03-06T07:22:07.324538+0000","last_undegraded":"2024-03-06T07:22:07.324538+0000","last_fullsized":"2024-03-06T07:22:07.324538+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:22.512414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324156+0000","last_change":"2024-03-06T07:22:06.281995+0000","last_active":"2024-03-06T07:22:07.324156+0000","last_peered":"2024-03-06T07:22:07.324156+0000","last_clean":"2024-03-06T07:22:07.324156+0000","last_became_active":"2024-03-06T07:22:06.281688+0000","last_became_peered":"2024-03-06T07:22:06.281688+0000","last_unstale":"2024-03-06T07:22:07.324156+0000","last_undegraded":"2024-03-06T07:22:07.324156+0000","last_fullsized":"2024-03-06T07:22:07.324156+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:10.543713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324124+0000","last_change":"2024-03-06T07:22:05.276257+0000","last_active":"2024-03-06T07:22:07.324124+0000","last_peered":"2024-03-06T07:22:07.324124+0000","last_clean":"2024-03-06T07:22:07.324124+0000","last_became_active":"2024-03-06T07:22:05.275951+0000","last_became_peered":"2024-03-06T07:22:05.275951+0000","last_unstale":"2024-03-06T07:22:07.324124+0000","last_undegraded":"2024-03-06T07:22:07.324124+0000","last_fullsized":"2024-03-06T07:22:07.324124+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:56.313423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324464+0000","last_change":"2024-03-06T07:22:06.287709+0000","last_active":"2024-03-06T07:22:07.324464+0000","last_peered":"2024-03-06T07:22:07.324464+0000","last_clean":"2024-03-06T07:22:07.324464+0000","last_became_active":"2024-03-06T07:22:06.287390+0000","last_became_peered":"2024-03-06T07:22:06.287390+0000","last_unstale":"2024-03-06T07:22:07.324464+0000","last_undegraded":"2024-03-06T07:22:07.324464+0000","last_fullsized":"2024-03-06T07:22:07.324464+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:04.872464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324858+0000","last_change":"2024-03-06T07:22:05.265795+0000","last_active":"2024-03-06T07:22:07.324858+0000","last_peered":"2024-03-06T07:22:07.324858+0000","last_clean":"2024-03-06T07:22:07.324858+0000","last_became_active":"2024-03-06T07:22:05.265116+0000","last_became_peered":"2024-03-06T07:22:05.265116+0000","last_unstale":"2024-03-06T07:22:07.324858+0000","last_undegraded":"2024-03-06T07:22:07.324858+0000","last_fullsized":"2024-03-06T07:22:07.324858+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:46.201174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324934+0000","last_change":"2024-03-06T07:22:06.283260+0000","last_active":"2024-03-06T07:22:07.324934+0000","last_peered":"2024-03-06T07:22:07.324934+0000","last_clean":"2024-03-06T07:22:07.324934+0000","last_became_active":"2024-03-06T07:22:06.282667+0000","last_became_peered":"2024-03-06T07:22:06.282667+0000","last_unstale":"2024-03-06T07:22:07.324934+0000","last_undegraded":"2024-03-06T07:22:07.324934+0000","last_fullsized":"2024-03-06T07:22:07.324934+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:32.823840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274013+0000","last_change":"2024-03-06T07:22:05.269608+0000","last_active":"2024-03-06T07:22:07.274013+0000","last_peered":"2024-03-06T07:22:07.274013+0000","last_clean":"2024-03-06T07:22:07.274013+0000","last_became_active":"2024-03-06T07:22:05.269226+0000","last_became_peered":"2024-03-06T07:22:05.269226+0000","last_unstale":"2024-03-06T07:22:07.274013+0000","last_undegraded":"2024-03-06T07:22:07.274013+0000","last_fullsized":"2024-03-06T07:22:07.274013+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.526750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273553+0000","last_change":"2024-03-06T07:22:06.284758+0000","last_active":"2024-03-06T07:22:07.273553+0000","last_peered":"2024-03-06T07:22:07.273553+0000","last_clean":"2024-03-06T07:22:07.273553+0000","last_became_active":"2024-03-06T07:22:06.284472+0000","last_became_peered":"2024-03-06T07:22:06.284472+0000","last_unstale":"2024-03-06T07:22:07.273553+0000","last_undegraded":"2024-03-06T07:22:07.273553+0000","last_fullsized":"2024-03-06T07:22:07.273553+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:03.086475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323969+0000","last_change":"2024-03-06T07:22:05.270816+0000","last_active":"2024-03-06T07:22:07.323969+0000","last_peered":"2024-03-06T07:22:07.323969+0000","last_clean":"2024-03-06T07:22:07.323969+0000","last_became_active":"2024-03-06T07:22:05.270550+0000","last_became_peered":"2024-03-06T07:22:05.270550+0000","last_unstale":"2024-03-06T07:22:07.323969+0000","last_undegraded":"2024-03-06T07:22:07.323969+0000","last_fullsized":"2024-03-06T07:22:07.323969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:53.343691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325032+0000","last_change":"2024-03-06T07:22:06.269423+0000","last_active":"2024-03-06T07:22:07.325032+0000","last_peered":"2024-03-06T07:22:07.325032+0000","last_clean":"2024-03-06T07:22:07.325032+0000","last_became_active":"2024-03-06T07:22:06.269180+0000","last_became_peered":"2024-03-06T07:22:06.269180+0000","last_unstale":"2024-03-06T07:22:07.325032+0000","last_undegraded":"2024-03-06T07:22:07.325032+0000","last_fullsized":"2024-03-06T07:22:07.325032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:40.036671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324354+0000","last_change":"2024-03-06T07:22:05.271075+0000","last_active":"2024-03-06T07:22:07.324354+0000","last_peered":"2024-03-06T07:22:07.324354+0000","last_clean":"2024-03-06T07:22:07.324354+0000","last_became_active":"2024-03-06T07:22:05.270790+0000","last_became_peered":"2024-03-06T07:22:05.270790+0000","last_unstale":"2024-03-06T07:22:07.324354+0000","last_undegraded":"2024-03-06T07:22:07.324354+0000","last_fullsized":"2024-03-06T07:22:07.324354+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:15.182553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324421+0000","last_change":"2024-03-06T07:22:06.273126+0000","last_active":"2024-03-06T07:22:07.324421+0000","last_peered":"2024-03-06T07:22:07.324421+0000","last_clean":"2024-03-06T07:22:07.324421+0000","last_became_active":"2024-03-06T07:22:06.272851+0000","last_became_peered":"2024-03-06T07:22:06.272851+0000","last_unstale":"2024-03-06T07:22:07.324421+0000","last_undegraded":"2024-03-06T07:22:07.324421+0000","last_fullsized":"2024-03-06T07:22:07.324421+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:11.308970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323988+0000","last_change":"2024-03-06T07:22:05.266985+0000","last_active":"2024-03-06T07:22:07.323988+0000","last_peered":"2024-03-06T07:22:07.323988+0000","last_clean":"2024-03-06T07:22:07.323988+0000","last_became_active":"2024-03-06T07:22:05.266735+0000","last_became_peered":"2024-03-06T07:22:05.266735+0000","last_unstale":"2024-03-06T07:22:07.323988+0000","last_undegraded":"2024-03-06T07:22:07.323988+0000","last_fullsized":"2024-03-06T07:22:07.323988+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:35.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273865+0000","last_change":"2024-03-06T07:22:06.290954+0000","last_active":"2024-03-06T07:22:07.273865+0000","last_peered":"2024-03-06T07:22:07.273865+0000","last_clean":"2024-03-06T07:22:07.273865+0000","last_became_active":"2024-03-06T07:22:06.290677+0000","last_became_peered":"2024-03-06T07:22:06.290677+0000","last_unstale":"2024-03-06T07:22:07.273865+0000","last_undegraded":"2024-03-06T07:22:07.273865+0000","last_fullsized":"2024-03-06T07:22:07.273865+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:58.629934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273304+0000","last_change":"2024-03-06T07:22:05.277283+0000","last_active":"2024-03-06T07:22:07.273304+0000","last_peered":"2024-03-06T07:22:07.273304+0000","last_clean":"2024-03-06T07:22:07.273304+0000","last_became_active":"2024-03-06T07:22:05.276995+0000","last_became_peered":"2024-03-06T07:22:05.276995+0000","last_unstale":"2024-03-06T07:22:07.273304+0000","last_undegraded":"2024-03-06T07:22:07.273304+0000","last_fullsized":"2024-03-06T07:22:07.273304+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:22.512414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324565+0000","last_change":"2024-03-06T07:22:06.278005+0000","last_active":"2024-03-06T07:22:07.324565+0000","last_peered":"2024-03-06T07:22:07.324565+0000","last_clean":"2024-03-06T07:22:07.324565+0000","last_became_active":"2024-03-06T07:22:06.277741+0000","last_became_peered":"2024-03-06T07:22:06.277741+0000","last_unstale":"2024-03-06T07:22:07.324565+0000","last_undegraded":"2024-03-06T07:22:07.324565+0000","last_fullsized":"2024-03-06T07:22:07.324565+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:02.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324161+0000","last_change":"2024-03-06T07:22:05.272997+0000","last_active":"2024-03-06T07:22:07.324161+0000","last_peered":"2024-03-06T07:22:07.324161+0000","last_clean":"2024-03-06T07:22:07.324161+0000","last_became_active":"2024-03-06T07:22:05.272736+0000","last_became_peered":"2024-03-06T07:22:05.272736+0000","last_unstale":"2024-03-06T07:22:07.324161+0000","last_undegraded":"2024-03-06T07:22:07.324161+0000","last_fullsized":"2024-03-06T07:22:07.324161+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:19:59.528462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324425+0000","last_change":"2024-03-06T07:22:06.285930+0000","last_active":"2024-03-06T07:22:07.324425+0000","last_peered":"2024-03-06T07:22:07.324425+0000","last_clean":"2024-03-06T07:22:07.324425+0000","last_became_active":"2024-03-06T07:22:06.285683+0000","last_became_peered":"2024-03-06T07:22:06.285683+0000","last_unstale":"2024-03-06T07:22:07.324425+0000","last_undegraded":"2024-03-06T07:22:07.324425+0000","last_fullsized":"2024-03-06T07:22:07.324425+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:12.415589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324536+0000","last_change":"2024-03-06T07:22:05.276323+0000","last_active":"2024-03-06T07:22:07.324536+0000","last_peered":"2024-03-06T07:22:07.324536+0000","last_clean":"2024-03-06T07:22:07.324536+0000","last_became_active":"2024-03-06T07:22:05.276045+0000","last_became_peered":"2024-03-06T07:22:05.276045+0000","last_unstale":"2024-03-06T07:22:07.324536+0000","last_undegraded":"2024-03-06T07:22:07.324536+0000","last_fullsized":"2024-03-06T07:22:07.324536+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:51.610768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324488+0000","last_change":"2024-03-06T07:22:06.280686+0000","last_active":"2024-03-06T07:22:07.324488+0000","last_peered":"2024-03-06T07:22:07.324488+0000","last_clean":"2024-03-06T07:22:07.324488+0000","last_became_active":"2024-03-06T07:22:06.280079+0000","last_became_peered":"2024-03-06T07:22:06.280079+0000","last_unstale":"2024-03-06T07:22:07.324488+0000","last_undegraded":"2024-03-06T07:22:07.324488+0000","last_fullsized":"2024-03-06T07:22:07.324488+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:29.906790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323933+0000","last_change":"2024-03-06T07:22:05.272839+0000","last_active":"2024-03-06T07:22:07.323933+0000","last_peered":"2024-03-06T07:22:07.323933+0000","last_clean":"2024-03-06T07:22:07.323933+0000","last_became_active":"2024-03-06T07:22:05.272588+0000","last_became_peered":"2024-03-06T07:22:05.272588+0000","last_unstale":"2024-03-06T07:22:07.323933+0000","last_undegraded":"2024-03-06T07:22:07.323933+0000","last_fullsized":"2024-03-06T07:22:07.323933+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:31.818733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273488+0000","last_change":"2024-03-06T07:22:06.281798+0000","last_active":"2024-03-06T07:22:07.273488+0000","last_peered":"2024-03-06T07:22:07.273488+0000","last_clean":"2024-03-06T07:22:07.273488+0000","last_became_active":"2024-03-06T07:22:06.281285+0000","last_became_peered":"2024-03-06T07:22:06.281285+0000","last_unstale":"2024-03-06T07:22:07.273488+0000","last_undegraded":"2024-03-06T07:22:07.273488+0000","last_fullsized":"2024-03-06T07:22:07.273488+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:00.533569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324658+0000","last_change":"2024-03-06T07:22:05.275597+0000","last_active":"2024-03-06T07:22:07.324658+0000","last_peered":"2024-03-06T07:22:07.324658+0000","last_clean":"2024-03-06T07:22:07.324658+0000","last_became_active":"2024-03-06T07:22:05.274043+0000","last_became_peered":"2024-03-06T07:22:05.274043+0000","last_unstale":"2024-03-06T07:22:07.324658+0000","last_undegraded":"2024-03-06T07:22:07.324658+0000","last_fullsized":"2024-03-06T07:22:07.324658+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:09.538606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324807+0000","last_change":"2024-03-06T07:22:06.283922+0000","last_active":"2024-03-06T07:22:07.324807+0000","last_peered":"2024-03-06T07:22:07.324807+0000","last_clean":"2024-03-06T07:22:07.324807+0000","last_became_active":"2024-03-06T07:22:06.283634+0000","last_became_peered":"2024-03-06T07:22:06.283634+0000","last_unstale":"2024-03-06T07:22:07.324807+0000","last_undegraded":"2024-03-06T07:22:07.324807+0000","last_fullsized":"2024-03-06T07:22:07.324807+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:46.420402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325246+0000","last_change":"2024-03-06T07:22:05.277243+0000","last_active":"2024-03-06T07:22:07.325246+0000","last_peered":"2024-03-06T07:22:07.325246+0000","last_clean":"2024-03-06T07:22:07.325246+0000","last_became_active":"2024-03-06T07:22:05.276996+0000","last_became_peered":"2024-03-06T07:22:05.276996+0000","last_unstale":"2024-03-06T07:22:07.325246+0000","last_undegraded":"2024-03-06T07:22:07.325246+0000","last_fullsized":"2024-03-06T07:22:07.325246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:03.663384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272379+0000","last_change":"2024-03-06T07:22:06.275074+0000","last_active":"2024-03-06T07:22:07.272379+0000","last_peered":"2024-03-06T07:22:07.272379+0000","last_clean":"2024-03-06T07:22:07.272379+0000","last_became_active":"2024-03-06T07:22:06.274819+0000","last_became_peered":"2024-03-06T07:22:06.274819+0000","last_unstale":"2024-03-06T07:22:07.272379+0000","last_undegraded":"2024-03-06T07:22:07.272379+0000","last_fullsized":"2024-03-06T07:22:07.272379+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:05.783295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324480+0000","last_change":"2024-03-06T07:22:05.272234+0000","last_active":"2024-03-06T07:22:07.324480+0000","last_peered":"2024-03-06T07:22:07.324480+0000","last_clean":"2024-03-06T07:22:07.324480+0000","last_became_active":"2024-03-06T07:22:05.271944+0000","last_became_peered":"2024-03-06T07:22:05.271944+0000","last_unstale":"2024-03-06T07:22:07.324480+0000","last_undegraded":"2024-03-06T07:22:07.324480+0000","last_fullsized":"2024-03-06T07:22:07.324480+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:01.806336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324005+0000","last_change":"2024-03-06T07:22:06.275296+0000","last_active":"2024-03-06T07:22:07.324005+0000","last_peered":"2024-03-06T07:22:07.324005+0000","last_clean":"2024-03-06T07:22:07.324005+0000","last_became_active":"2024-03-06T07:22:06.275041+0000","last_became_peered":"2024-03-06T07:22:06.275041+0000","last_unstale":"2024-03-06T07:22:07.324005+0000","last_undegraded":"2024-03-06T07:22:07.324005+0000","last_fullsized":"2024-03-06T07:22:07.324005+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:10.302008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324600+0000","last_change":"2024-03-06T07:22:05.276314+0000","last_active":"2024-03-06T07:22:07.324600+0000","last_peered":"2024-03-06T07:22:07.324600+0000","last_clean":"2024-03-06T07:22:07.324600+0000","last_became_active":"2024-03-06T07:22:05.276059+0000","last_became_peered":"2024-03-06T07:22:05.276059+0000","last_unstale":"2024-03-06T07:22:07.324600+0000","last_undegraded":"2024-03-06T07:22:07.324600+0000","last_fullsized":"2024-03-06T07:22:07.324600+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:23.337554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274146+0000","last_change":"2024-03-06T07:22:06.273627+0000","last_active":"2024-03-06T07:22:07.274146+0000","last_peered":"2024-03-06T07:22:07.274146+0000","last_clean":"2024-03-06T07:22:07.274146+0000","last_became_active":"2024-03-06T07:22:06.273328+0000","last_became_peered":"2024-03-06T07:22:06.273328+0000","last_unstale":"2024-03-06T07:22:07.274146+0000","last_undegraded":"2024-03-06T07:22:07.274146+0000","last_fullsized":"2024-03-06T07:22:07.274146+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:32.869301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272620+0000","last_change":"2024-03-06T07:22:05.275558+0000","last_active":"2024-03-06T07:22:07.272620+0000","last_peered":"2024-03-06T07:22:07.272620+0000","last_clean":"2024-03-06T07:22:07.272620+0000","last_became_active":"2024-03-06T07:22:05.275307+0000","last_became_peered":"2024-03-06T07:22:05.275307+0000","last_unstale":"2024-03-06T07:22:07.272620+0000","last_undegraded":"2024-03-06T07:22:07.272620+0000","last_fullsized":"2024-03-06T07:22:07.272620+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:42:56.088092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324463+0000","last_change":"2024-03-06T07:22:06.282996+0000","last_active":"2024-03-06T07:22:07.324463+0000","last_peered":"2024-03-06T07:22:07.324463+0000","last_clean":"2024-03-06T07:22:07.324463+0000","last_became_active":"2024-03-06T07:22:06.282325+0000","last_became_peered":"2024-03-06T07:22:06.282325+0000","last_unstale":"2024-03-06T07:22:07.324463+0000","last_undegraded":"2024-03-06T07:22:07.324463+0000","last_fullsized":"2024-03-06T07:22:07.324463+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:37:45.997367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324194+0000","last_change":"2024-03-06T07:22:05.278909+0000","last_active":"2024-03-06T07:22:07.324194+0000","last_peered":"2024-03-06T07:22:07.324194+0000","last_clean":"2024-03-06T07:22:07.324194+0000","last_became_active":"2024-03-06T07:22:05.278643+0000","last_became_peered":"2024-03-06T07:22:05.278643+0000","last_unstale":"2024-03-06T07:22:07.324194+0000","last_undegraded":"2024-03-06T07:22:07.324194+0000","last_fullsized":"2024-03-06T07:22:07.324194+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:39.428768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324610+0000","last_change":"2024-03-06T07:22:06.290910+0000","last_active":"2024-03-06T07:22:07.324610+0000","last_peered":"2024-03-06T07:22:07.324610+0000","last_clean":"2024-03-06T07:22:07.324610+0000","last_became_active":"2024-03-06T07:22:06.290638+0000","last_became_peered":"2024-03-06T07:22:06.290638+0000","last_unstale":"2024-03-06T07:22:07.324610+0000","last_undegraded":"2024-03-06T07:22:07.324610+0000","last_fullsized":"2024-03-06T07:22:07.324610+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:55.574795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324030+0000","last_change":"2024-03-06T07:22:05.276401+0000","last_active":"2024-03-06T07:22:07.324030+0000","last_peered":"2024-03-06T07:22:07.324030+0000","last_clean":"2024-03-06T07:22:07.324030+0000","last_became_active":"2024-03-06T07:22:05.276140+0000","last_became_peered":"2024-03-06T07:22:05.276140+0000","last_unstale":"2024-03-06T07:22:07.324030+0000","last_undegraded":"2024-03-06T07:22:07.324030+0000","last_fullsized":"2024-03-06T07:22:07.324030+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:00.063577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323649+0000","last_change":"2024-03-06T07:22:06.282249+0000","last_active":"2024-03-06T07:22:07.323649+0000","last_peered":"2024-03-06T07:22:07.323649+0000","last_clean":"2024-03-06T07:22:07.323649+0000","last_became_active":"2024-03-06T07:22:06.281968+0000","last_became_peered":"2024-03-06T07:22:06.281968+0000","last_unstale":"2024-03-06T07:22:07.323649+0000","last_undegraded":"2024-03-06T07:22:07.323649+0000","last_fullsized":"2024-03-06T07:22:07.323649+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:20.517880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323864+0000","last_change":"2024-03-06T07:22:05.275874+0000","last_active":"2024-03-06T07:22:07.323864+0000","last_peered":"2024-03-06T07:22:07.323864+0000","last_clean":"2024-03-06T07:22:07.323864+0000","last_became_active":"2024-03-06T07:22:05.273346+0000","last_became_peered":"2024-03-06T07:22:05.273346+0000","last_unstale":"2024-03-06T07:22:07.323864+0000","last_undegraded":"2024-03-06T07:22:07.323864+0000","last_fullsized":"2024-03-06T07:22:07.323864+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:26.467203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324252+0000","last_change":"2024-03-06T07:22:06.279486+0000","last_active":"2024-03-06T07:22:07.324252+0000","last_peered":"2024-03-06T07:22:07.324252+0000","last_clean":"2024-03-06T07:22:07.324252+0000","last_became_active":"2024-03-06T07:22:06.279191+0000","last_became_peered":"2024-03-06T07:22:06.279191+0000","last_unstale":"2024-03-06T07:22:07.324252+0000","last_undegraded":"2024-03-06T07:22:07.324252+0000","last_fullsized":"2024-03-06T07:22:07.324252+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:14.577314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323908+0000","last_change":"2024-03-06T07:22:05.278429+0000","last_active":"2024-03-06T07:22:07.323908+0000","last_peered":"2024-03-06T07:22:07.323908+0000","last_clean":"2024-03-06T07:22:07.323908+0000","last_became_active":"2024-03-06T07:22:05.278167+0000","last_became_peered":"2024-03-06T07:22:05.278167+0000","last_unstale":"2024-03-06T07:22:07.323908+0000","last_undegraded":"2024-03-06T07:22:07.323908+0000","last_fullsized":"2024-03-06T07:22:07.323908+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:31.864194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325204+0000","last_change":"2024-03-06T07:22:06.281426+0000","last_active":"2024-03-06T07:22:07.325204+0000","last_peered":"2024-03-06T07:22:07.325204+0000","last_clean":"2024-03-06T07:22:07.325204+0000","last_became_active":"2024-03-06T07:22:06.281025+0000","last_became_peered":"2024-03-06T07:22:06.281025+0000","last_unstale":"2024-03-06T07:22:07.325204+0000","last_undegraded":"2024-03-06T07:22:07.325204+0000","last_fullsized":"2024-03-06T07:22:07.325204+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:29.084213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324566+0000","last_change":"2024-03-06T07:22:05.264875+0000","last_active":"2024-03-06T07:22:07.324566+0000","last_peered":"2024-03-06T07:22:07.324566+0000","last_clean":"2024-03-06T07:22:07.324566+0000","last_became_active":"2024-03-06T07:22:05.264500+0000","last_became_peered":"2024-03-06T07:22:05.264500+0000","last_unstale":"2024-03-06T07:22:07.324566+0000","last_undegraded":"2024-03-06T07:22:07.324566+0000","last_fullsized":"2024-03-06T07:22:07.324566+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:45.415295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324265+0000","last_change":"2024-03-06T07:22:06.281551+0000","last_active":"2024-03-06T07:22:07.324265+0000","last_peered":"2024-03-06T07:22:07.324265+0000","last_clean":"2024-03-06T07:22:07.324265+0000","last_became_active":"2024-03-06T07:22:06.281257+0000","last_became_peered":"2024-03-06T07:22:06.281257+0000","last_unstale":"2024-03-06T07:22:07.324265+0000","last_undegraded":"2024-03-06T07:22:07.324265+0000","last_fullsized":"2024-03-06T07:22:07.324265+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:43:46.420402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323662+0000","last_change":"2024-03-06T07:22:05.275590+0000","last_active":"2024-03-06T07:22:07.323662+0000","last_peered":"2024-03-06T07:22:07.323662+0000","last_clean":"2024-03-06T07:22:07.323662+0000","last_became_active":"2024-03-06T07:22:05.275317+0000","last_became_peered":"2024-03-06T07:22:05.275317+0000","last_unstale":"2024-03-06T07:22:07.323662+0000","last_undegraded":"2024-03-06T07:22:07.323662+0000","last_fullsized":"2024-03-06T07:22:07.323662+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:10.303863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325056+0000","last_change":"2024-03-06T07:22:06.273592+0000","last_active":"2024-03-06T07:22:07.325056+0000","last_peered":"2024-03-06T07:22:07.325056+0000","last_clean":"2024-03-06T07:22:07.325056+0000","last_became_active":"2024-03-06T07:22:06.273190+0000","last_became_peered":"2024-03-06T07:22:06.273190+0000","last_unstale":"2024-03-06T07:22:07.325056+0000","last_undegraded":"2024-03-06T07:22:07.325056+0000","last_fullsized":"2024-03-06T07:22:07.325056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:20.668909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324678+0000","last_change":"2024-03-06T07:22:05.275628+0000","last_active":"2024-03-06T07:22:07.324678+0000","last_peered":"2024-03-06T07:22:07.324678+0000","last_clean":"2024-03-06T07:22:07.324678+0000","last_became_active":"2024-03-06T07:22:05.275226+0000","last_became_peered":"2024-03-06T07:22:05.275226+0000","last_unstale":"2024-03-06T07:22:07.324678+0000","last_undegraded":"2024-03-06T07:22:07.324678+0000","last_fullsized":"2024-03-06T07:22:07.324678+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:47.908766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324255+0000","last_change":"2024-03-06T07:22:06.285329+0000","last_active":"2024-03-06T07:22:07.324255+0000","last_peered":"2024-03-06T07:22:07.324255+0000","last_clean":"2024-03-06T07:22:07.324255+0000","last_became_active":"2024-03-06T07:22:06.285067+0000","last_became_peered":"2024-03-06T07:22:06.285067+0000","last_unstale":"2024-03-06T07:22:07.324255+0000","last_undegraded":"2024-03-06T07:22:07.324255+0000","last_fullsized":"2024-03-06T07:22:07.324255+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:46.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273806+0000","last_change":"2024-03-06T07:22:05.277565+0000","last_active":"2024-03-06T07:22:07.273806+0000","last_peered":"2024-03-06T07:22:07.273806+0000","last_clean":"2024-03-06T07:22:07.273806+0000","last_became_active":"2024-03-06T07:22:05.276867+0000","last_became_peered":"2024-03-06T07:22:05.276867+0000","last_unstale":"2024-03-06T07:22:07.273806+0000","last_undegraded":"2024-03-06T07:22:07.273806+0000","last_fullsized":"2024-03-06T07:22:07.273806+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:21.547573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324410+0000","last_change":"2024-03-06T07:22:06.289399+0000","last_active":"2024-03-06T07:22:07.324410+0000","last_peered":"2024-03-06T07:22:07.324410+0000","last_clean":"2024-03-06T07:22:07.324410+0000","last_became_active":"2024-03-06T07:22:06.289121+0000","last_became_peered":"2024-03-06T07:22:06.289121+0000","last_unstale":"2024-03-06T07:22:07.324410+0000","last_undegraded":"2024-03-06T07:22:07.324410+0000","last_fullsized":"2024-03-06T07:22:07.324410+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:45.514208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273657+0000","last_change":"2024-03-06T07:22:05.275590+0000","last_active":"2024-03-06T07:22:07.273657+0000","last_peered":"2024-03-06T07:22:07.273657+0000","last_clean":"2024-03-06T07:22:07.273657+0000","last_became_active":"2024-03-06T07:22:05.275287+0000","last_became_peered":"2024-03-06T07:22:05.275287+0000","last_unstale":"2024-03-06T07:22:07.273657+0000","last_undegraded":"2024-03-06T07:22:07.273657+0000","last_fullsized":"2024-03-06T07:22:07.273657+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.526750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.272557+0000","last_change":"2024-03-06T07:22:06.276631+0000","last_active":"2024-03-06T07:22:07.272557+0000","last_peered":"2024-03-06T07:22:07.272557+0000","last_clean":"2024-03-06T07:22:07.272557+0000","last_became_active":"2024-03-06T07:22:06.276329+0000","last_became_peered":"2024-03-06T07:22:06.276329+0000","last_unstale":"2024-03-06T07:22:07.272557+0000","last_undegraded":"2024-03-06T07:22:07.272557+0000","last_fullsized":"2024-03-06T07:22:07.272557+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:53.220165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.325138+0000","last_change":"2024-03-06T07:22:05.270328+0000","last_active":"2024-03-06T07:22:07.325138+0000","last_peered":"2024-03-06T07:22:07.325138+0000","last_clean":"2024-03-06T07:22:07.325138+0000","last_became_active":"2024-03-06T07:22:05.270021+0000","last_became_peered":"2024-03-06T07:22:05.270021+0000","last_unstale":"2024-03-06T07:22:07.325138+0000","last_undegraded":"2024-03-06T07:22:07.325138+0000","last_fullsized":"2024-03-06T07:22:07.325138+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:13.572207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323890+0000","last_change":"2024-03-06T07:22:06.274953+0000","last_active":"2024-03-06T07:22:07.323890+0000","last_peered":"2024-03-06T07:22:07.323890+0000","last_clean":"2024-03-06T07:22:07.323890+0000","last_became_active":"2024-03-06T07:22:06.274702+0000","last_became_peered":"2024-03-06T07:22:06.274702+0000","last_unstale":"2024-03-06T07:22:07.323890+0000","last_undegraded":"2024-03-06T07:22:07.323890+0000","last_fullsized":"2024-03-06T07:22:07.323890+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:48.093500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.274103+0000","last_change":"2024-03-06T07:22:05.271821+0000","last_active":"2024-03-06T07:22:07.274103+0000","last_peered":"2024-03-06T07:22:07.274103+0000","last_clean":"2024-03-06T07:22:07.274103+0000","last_became_active":"2024-03-06T07:22:05.271340+0000","last_became_peered":"2024-03-06T07:22:05.271340+0000","last_unstale":"2024-03-06T07:22:07.274103+0000","last_undegraded":"2024-03-06T07:22:07.274103+0000","last_fullsized":"2024-03-06T07:22:07.274103+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:21.547573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324391+0000","last_change":"2024-03-06T07:22:06.282198+0000","last_active":"2024-03-06T07:22:07.324391+0000","last_peered":"2024-03-06T07:22:07.324391+0000","last_clean":"2024-03-06T07:22:07.324391+0000","last_became_active":"2024-03-06T07:22:06.281940+0000","last_became_peered":"2024-03-06T07:22:06.281940+0000","last_unstale":"2024-03-06T07:22:07.324391+0000","last_undegraded":"2024-03-06T07:22:07.324391+0000","last_fullsized":"2024-03-06T07:22:07.324391+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:31.566918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.273851+0000","last_change":"2024-03-06T07:22:05.274943+0000","last_active":"2024-03-06T07:22:07.273851+0000","last_peered":"2024-03-06T07:22:07.273851+0000","last_clean":"2024-03-06T07:22:07.273851+0000","last_became_active":"2024-03-06T07:22:05.272239+0000","last_became_peered":"2024-03-06T07:22:05.272239+0000","last_unstale":"2024-03-06T07:22:07.273851+0000","last_undegraded":"2024-03-06T07:22:07.273851+0000","last_fullsized":"2024-03-06T07:22:07.273851+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:54.446300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324017+0000","last_change":"2024-03-06T07:22:06.283287+0000","last_active":"2024-03-06T07:22:07.324017+0000","last_peered":"2024-03-06T07:22:07.324017+0000","last_clean":"2024-03-06T07:22:07.324017+0000","last_became_active":"2024-03-06T07:22:06.283024+0000","last_became_peered":"2024-03-06T07:22:06.283024+0000","last_unstale":"2024-03-06T07:22:07.324017+0000","last_undegraded":"2024-03-06T07:22:07.324017+0000","last_fullsized":"2024-03-06T07:22:07.324017+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:55.574795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323732+0000","last_change":"2024-03-06T07:22:05.270394+0000","last_active":"2024-03-06T07:22:07.323732+0000","last_peered":"2024-03-06T07:22:07.323732+0000","last_clean":"2024-03-06T07:22:07.323732+0000","last_became_active":"2024-03-06T07:22:05.270110+0000","last_became_peered":"2024-03-06T07:22:05.270110+0000","last_unstale":"2024-03-06T07:22:07.323732+0000","last_undegraded":"2024-03-06T07:22:07.323732+0000","last_fullsized":"2024-03-06T07:22:07.323732+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:14.806818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324544+0000","last_change":"2024-03-06T07:22:06.288799+0000","last_active":"2024-03-06T07:22:07.324544+0000","last_peered":"2024-03-06T07:22:07.324544+0000","last_clean":"2024-03-06T07:22:07.324544+0000","last_became_active":"2024-03-06T07:22:06.288501+0000","last_became_peered":"2024-03-06T07:22:06.288501+0000","last_unstale":"2024-03-06T07:22:07.324544+0000","last_undegraded":"2024-03-06T07:22:07.324544+0000","last_fullsized":"2024-03-06T07:22:07.324544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:10.543713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323687+0000","last_change":"2024-03-06T07:22:05.276925+0000","last_active":"2024-03-06T07:22:07.323687+0000","last_peered":"2024-03-06T07:22:07.323687+0000","last_clean":"2024-03-06T07:22:07.323687+0000","last_became_active":"2024-03-06T07:22:05.276594+0000","last_became_peered":"2024-03-06T07:22:05.276594+0000","last_unstale":"2024-03-06T07:22:07.323687+0000","last_undegraded":"2024-03-06T07:22:07.323687+0000","last_fullsized":"2024-03-06T07:22:07.323687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:13:45.596793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.324099+0000","last_change":"2024-03-06T07:22:06.287902+0000","last_active":"2024-03-06T07:22:07.324099+0000","last_peered":"2024-03-06T07:22:07.324099+0000","last_clean":"2024-03-06T07:22:07.324099+0000","last_became_active":"2024-03-06T07:22:06.287635+0000","last_became_peered":"2024-03-06T07:22:06.287635+0000","last_unstale":"2024-03-06T07:22:07.324099+0000","last_undegraded":"2024-03-06T07:22:07.324099+0000","last_fullsized":"2024-03-06T07:22:07.324099+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:05.248200+0000","last_clean_scrub_stamp":"2024-03-06T07:22:05.248200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:28.209748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323853+0000","last_change":"2024-03-06T07:22:05.275435+0000","last_active":"2024-03-06T07:22:07.323853+0000","last_peered":"2024-03-06T07:22:07.323853+0000","last_clean":"2024-03-06T07:22:07.323853+0000","last_became_active":"2024-03-06T07:22:05.275199+0000","last_became_peered":"2024-03-06T07:22:05.275199+0000","last_unstale":"2024-03-06T07:22:07.323853+0000","last_undegraded":"2024-03-06T07:22:07.323853+0000","last_fullsized":"2024-03-06T07:22:07.323853+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:04.243093+0000","last_clean_scrub_stamp":"2024-03-06T07:22:04.243093+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:09.296901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":337,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:22:07.323801+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:22:07.323801+0000","last_peered":"2024-03-06T07:22:07.323801+0000","last_clean":"2024-03-06T07:22:07.323801+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:22:07.323801+0000","last_undegraded":"2024-03-06T07:22:07.323801+0000","last_fullsized":"2024-03-06T07:22:07.323801+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1417216,"data_stored":1233527,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1351680,"data_stored":649657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1343488,"data_stored":642543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":643995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1363968,"data_stored":650721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32488,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339352,"statfs":{"total":96636764160,"available":96603496448,"internally_reserved":0,"allocated":1335296,"data_stored":642383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32492,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339348,"statfs":{"total":96636764160,"available":96603492352,"internally_reserved":0,"allocated":1339392,"data_stored":642405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":1232773,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:10.569 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:10.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:22:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.712+0000 7f9704488640 1 Processor -- start 2024-03-06T07:22:10.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.712+0000 7f9704488640 1 -- start start 2024-03-06T07:22:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9704488640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f96fc0c5990 0x7f96fc0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9704488640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9704488640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f96fc06c730 0x7f96fc1523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9704488640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc06cbe0 con 0x7f96fc0c5990 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9704488640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0c3ef0 con 0x7f96fc06c730 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9704488640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc1556d0 con 0x7f96fc0ab170 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f96fc0c5990 0x7f96fc0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f96fc0c5990 0x7f96fc0c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33522/0 (socket says 172.21.15.94:33522) 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 -- 172.21.15.94:0/1289838959 learned_addr learned my addr 172.21.15.94:0/1289838959 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702a25640 1 --2- 172.21.15.94:0/1289838959 >> v2:172.21.15.145:3300/0 conn(0x7f96fc06c730 0x7f96fc1523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9701a23640 1 --2- 172.21.15.94:0/1289838959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 -- 172.21.15.94:0/1289838959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 msgr2=0x7f96fc06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 --2- 172.21.15.94:0/1289838959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc06c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 -- 172.21.15.94:0/1289838959 >> v2:172.21.15.145:3300/0 conn(0x7f96fc06c730 msgr2=0x7f96fc1523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 --2- 172.21.15.94:0/1289838959 >> v2:172.21.15.145:3300/0 conn(0x7f96fc06c730 0x7f96fc1523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.713+0000 7f9702224640 1 -- 172.21.15.94:0/1289838959 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc1559b0 con 0x7f96fc0c5990 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.714+0000 7f9702224640 1 --2- 172.21.15.94:0/1289838959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc0c5990 0x7f96fc0c0ab0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0c4100 tx=0x7f96f802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8882329215b8309 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.714+0000 7f9701222640 1 -- 172.21.15.94:0/1289838959 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f803b070 con 0x7f96fc0c5990 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.714+0000 7f9701222640 1 -- 172.21.15.94:0/1289838959 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f8036440 con 0x7f96fc0c5990 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.714+0000 7f9701222640 1 -- 172.21.15.94:0/1289838959 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8036810 con 0x7f96fc0c5990 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 -- 172.21.15.94:0/1289838959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc0c5990 msgr2=0x7f96fc0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 --2- 172.21.15.94:0/1289838959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc0c5990 0x7f96fc0c0ab0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0c4100 tx=0x7f96f802e0c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 -- 172.21.15.94:0/1289838959 shutdown_connections 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 --2- 172.21.15.94:0/1289838959 >> v2:172.21.15.145:3300/0 conn(0x7f96fc06c730 0x7f96fc1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 --2- 172.21.15.94:0/1289838959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc06c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 --2- 172.21.15.94:0/1289838959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc0c5990 0x7f96fc0c0ab0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 -- 172.21.15.94:0/1289838959 >> 172.21.15.94:0/1289838959 conn(0x7f96fc0c6710 msgr2=0x7f96fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 -- 172.21.15.94:0/1289838959 shutdown_connections 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.715+0000 7f9704488640 1 -- 172.21.15.94:0/1289838959 wait complete. 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.716+0000 7f9704488640 1 Processor -- start 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.716+0000 7f9704488640 1 -- start start 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc06c730 0x7f96fc1482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc144a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96fc0c5990 0x7f96fc144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc156490 con 0x7f96fc06c730 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc155f70 con 0x7f96fc0c5990 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc156610 con 0x7f96fc0ab170 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc06c730 0x7f96fc1482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc06c730 0x7f96fc1482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33528/0 (socket says 172.21.15.94:33528) 2024-03-06T07:22:10.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 -- 172.21.15.94:0/1371915709 learned_addr learned my addr 172.21.15.94:0/1371915709 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702a25640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96fc0c5990 0x7f96fc144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9701a23640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 -- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 msgr2=0x7f96fc144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc144a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 -- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96fc0c5990 msgr2=0x7f96fc144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96fc0c5990 0x7f96fc144fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 -- 172.21.15.94:0/1371915709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc1537f0 con 0x7f96fc06c730 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9701a23640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9702224640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc06c730 0x7f96fc1482e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f96f802e100 tx=0x7f96f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f96f2ffd640 1 -- 172.21.15.94:0/1371915709 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8046070 con 0x7f96fc06c730 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f96f2ffd640 1 -- 172.21.15.94:0/1371915709 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f803d050 con 0x7f96fc06c730 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc0a79f0 con 0x7f96fc06c730 2024-03-06T07:22:10.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f96f2ffd640 1 -- 172.21.15.94:0/1371915709 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f803b030 con 0x7f96fc06c730 2024-03-06T07:22:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.717+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc148820 con 0x7f96fc06c730 2024-03-06T07:22:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.718+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc04f330 con 0x7f96fc06c730 2024-03-06T07:22:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.719+0000 7f96f2ffd640 1 -- 172.21.15.94:0/1371915709 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96f8049070 con 0x7f96fc06c730 2024-03-06T07:22:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.719+0000 7f96f2ffd640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96e006b4d0 0x7f96e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.719+0000 7f96f2ffd640 1 -- 172.21.15.94:0/1371915709 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f96f80abd30 con 0x7f96fc06c730 2024-03-06T07:22:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.719+0000 7f9701a23640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96e006b4d0 0x7f96e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.720+0000 7f9701a23640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96e006b4d0 0x7f96e006d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f96fc149af0 tx=0x7f96ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.723+0000 7f96f2ffd640 1 -- 172.21.15.94:0/1371915709 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96f807a580 con 0x7f96fc06c730 2024-03-06T07:22:10.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.958+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f96fc153d30 con 0x7f96fc06c730 2024-03-06T07:22:10.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.960+0000 7f96f2ffd640 1 -- 172.21.15.94:0/1371915709 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f96f807e620 con 0x7f96fc06c730 2024-03-06T07:22:10.962 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:22:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:22:10.962 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:22:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:22:10.963 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:10.963 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:22:10.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96e006b4d0 msgr2=0x7f96e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96e006b4d0 0x7f96e006d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f96fc149af0 tx=0x7f96ec008040 comp rx=0 tx=0).stop 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc06c730 msgr2=0x7f96fc1482e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc06c730 0x7f96fc1482e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f96f802e100 tx=0x7f96f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 shutdown_connections 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96e006b4d0 0x7f96e006d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96fc0c5990 0x7f96fc144fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96fc0ab170 0x7f96fc144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 --2- 172.21.15.94:0/1371915709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96fc06c730 0x7f96fc1482e0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 >> 172.21.15.94:0/1371915709 conn(0x7f96fc0c6710 msgr2=0x7f96fc06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 shutdown_connections 2024-03-06T07:22:10.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:10.963+0000 7f9704488640 1 -- 172.21.15.94:0/1371915709 wait complete. 2024-03-06T07:22:10.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.073+0000 7f07aee84640 1 Processor -- start 2024-03-06T07:22:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.073+0000 7f07aee84640 1 -- start start 2024-03-06T07:22:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07aee84640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07a8150580 0x7f07a806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07aee84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a80c1320 0x7f07a806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07aee84640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f07a80c6000 0x7f07a81566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07aee84640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a8153a60 con 0x7f07a8150580 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07aee84640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a8153d40 con 0x7f07a80c6000 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07aee84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a806c2c0 con 0x7f07a80c1320 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07a8150580 0x7f07a806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07a8150580 0x7f07a806dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33542/0 (socket says 172.21.15.94:33542) 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 -- 172.21.15.94:0/113607345 learned_addr learned my addr 172.21.15.94:0/113607345 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07ad421640 1 --2- 172.21.15.94:0/113607345 >> v2:172.21.15.145:3300/0 conn(0x7f07a80c6000 0x7f07a81566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07a7fff640 1 --2- 172.21.15.94:0/113607345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a80c1320 0x7f07a806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 -- 172.21.15.94:0/113607345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a80c1320 msgr2=0x7f07a806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 --2- 172.21.15.94:0/113607345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a80c1320 0x7f07a806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 -- 172.21.15.94:0/113607345 >> v2:172.21.15.145:3300/0 conn(0x7f07a80c6000 msgr2=0x7f07a81566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 --2- 172.21.15.94:0/113607345 >> v2:172.21.15.145:3300/0 conn(0x7f07a80c6000 0x7f07a81566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.074+0000 7f07acc20640 1 -- 172.21.15.94:0/113607345 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a806c5a0 con 0x7f07a8150580 2024-03-06T07:22:11.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.075+0000 7f07acc20640 1 --2- 172.21.15.94:0/113607345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a8150580 0x7f07a806dd10 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f07a8152fe0 tx=0x7f079c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca74d12a4cdc202 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.075+0000 7f07a77fe640 1 -- 172.21.15.94:0/113607345 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c02e070 con 0x7f07a8150580 2024-03-06T07:22:11.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.075+0000 7f07a77fe640 1 -- 172.21.15.94:0/113607345 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c037070 con 0x7f07a8150580 2024-03-06T07:22:11.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.075+0000 7f07a77fe640 1 -- 172.21.15.94:0/113607345 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c031a00 con 0x7f07a8150580 2024-03-06T07:22:11.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 -- 172.21.15.94:0/113607345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a8150580 msgr2=0x7f07a806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 --2- 172.21.15.94:0/113607345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a8150580 0x7f07a806dd10 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f07a8152fe0 tx=0x7f079c02b880 comp rx=0 tx=0).stop 2024-03-06T07:22:11.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 -- 172.21.15.94:0/113607345 shutdown_connections 2024-03-06T07:22:11.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 --2- 172.21.15.94:0/113607345 >> v2:172.21.15.145:3300/0 conn(0x7f07a80c6000 0x7f07a81566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 --2- 172.21.15.94:0/113607345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a80c1320 0x7f07a806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 --2- 172.21.15.94:0/113607345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a8150580 0x7f07a806dd10 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 -- 172.21.15.94:0/113607345 >> 172.21.15.94:0/113607345 conn(0x7f07a80c6710 msgr2=0x7f07a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 -- 172.21.15.94:0/113607345 shutdown_connections 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.076+0000 7f07aee84640 1 -- 172.21.15.94:0/113607345 wait complete. 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.077+0000 7f07aee84640 1 Processor -- start 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.077+0000 7f07aee84640 1 -- start start 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07aee84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07a80c1320 0x7f07a81e2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07aee84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a80c6000 0x7f07a81ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07aee84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a8150580 0x7f07a81477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07aee84640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80c0dc0 con 0x7f07a80c6000 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07aee84640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80c0c40 con 0x7f07a80c1320 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07aee84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a8156bf0 con 0x7f07a8150580 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a80c6000 0x7f07a81ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a80c6000 0x7f07a81ede30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33556/0 (socket says 172.21.15.94:33556) 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 -- 172.21.15.94:0/38029449 learned_addr learned my addr 172.21.15.94:0/38029449 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07acc20640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07a80c1320 0x7f07a81e2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07ad421640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a8150580 0x7f07a81477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 -- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a8150580 msgr2=0x7f07a81477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a8150580 0x7f07a81477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 -- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07a80c1320 msgr2=0x7f07a81e2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07a80c1320 0x7f07a81e2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 -- 172.21.15.94:0/38029449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a80c2f90 con 0x7f07a80c6000 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a7fff640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a80c6000 0x7f07a81ede30 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f079400e890 tx=0x7f079400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.078+0000 7f07a57fa640 1 -- 172.21.15.94:0/38029449 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794017070 con 0x7f07a80c6000 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.079+0000 7f07a57fa640 1 -- 172.21.15.94:0/38029449 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079400c940 con 0x7f07a80c6000 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.079+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a81560b0 con 0x7f07a80c6000 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.079+0000 7f07a57fa640 1 -- 172.21.15.94:0/38029449 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079400cd10 con 0x7f07a80c6000 2024-03-06T07:22:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.079+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a81e2550 con 0x7f07a80c6000 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.079+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a804f330 con 0x7f07a80c6000 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.080+0000 7f07a57fa640 1 -- 172.21.15.94:0/38029449 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f079401a070 con 0x7f07a80c6000 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.080+0000 7f07a57fa640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078c06b4d0 0x7f078c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.080+0000 7f07a57fa640 1 -- 172.21.15.94:0/38029449 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f079408a210 con 0x7f07a80c6000 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.080+0000 7f07acc20640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078c06b4d0 0x7f078c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.081+0000 7f07acc20640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078c06b4d0 0x7f078c06d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f079c036000 tx=0x7f079c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.085+0000 7f07a57fa640 1 -- 172.21.15.94:0/38029449 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0794058940 con 0x7f07a80c6000 2024-03-06T07:22:11.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.292+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07a80c5de0 con 0x7f07a80c6000 2024-03-06T07:22:11.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.292+0000 7f07a57fa640 1 -- 172.21.15.94:0/38029449 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4990 (secure 0 0 0) 0x7f079405c9e0 con 0x7f07a80c6000 2024-03-06T07:22:11.295 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:11.295 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2722633283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2722633283},{"type":"v1","addr":"172.21.15.145:6837","nonce":2722633283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:08.543964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":36134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_36134":{"gid":36134,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/4001418202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4001418202},{"type":"v1","addr":"172.21.15.145:6836","nonce":4001418202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":36134,"qdb_cluster":[36134]},"id":8}]} 2024-03-06T07:22:11.295 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 89 2024-03-06T07:22:11.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.294+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078c06b4d0 msgr2=0x7f078c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.294+0000 7f07aee84640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078c06b4d0 0x7f078c06d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f079c036000 tx=0x7f079c03d040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.294+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a80c6000 msgr2=0x7f07a81ede30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.294+0000 7f07aee84640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a80c6000 0x7f07a81ede30 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f079400e890 tx=0x7f079400ed50 comp rx=0 tx=0).stop 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 shutdown_connections 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f078c06b4d0 0x7f078c06d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07a8150580 0x7f07a81477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07a80c6000 0x7f07a81ede30 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 --2- 172.21.15.94:0/38029449 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07a80c1320 0x7f07a81e2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 >> 172.21.15.94:0/38029449 conn(0x7f07a80c6710 msgr2=0x7f07a80c4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 shutdown_connections 2024-03-06T07:22:11.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.295+0000 7f07aee84640 1 -- 172.21.15.94:0/38029449 wait complete. 2024-03-06T07:22:11.534 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:22:11.533+0000 7f550a843640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:11.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:05.610686+0000', 'modified': '2024-03-06T07:22:08.543964+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 36134}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_36134': {'gid': 36134, 'name': 'd', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/4001418202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 4001418202}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 4001418202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 36134, 'qdb_cluster': [36134]} 2024-03-06T07:22:11.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:11.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:22:11.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.812+0000 7f4f5d97a640 1 Processor -- start 2024-03-06T07:22:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.812+0000 7f4f5d97a640 1 -- start start 2024-03-06T07:22:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.812+0000 7f4f5d97a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f581561a0 0x7f4f581529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.812+0000 7f4f5d97a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f580a9990 con 0x7f4f580c6ab0 2024-03-06T07:22:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.812+0000 7f4f5d97a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f581556c0 con 0x7f4f5814f110 2024-03-06T07:22:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.812+0000 7f4f5d97a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f58155d10 con 0x7f4f581561a0 2024-03-06T07:22:11.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.813+0000 7f4f567fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f581561a0 0x7f4f581529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.813+0000 7f4f56ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4f580c6ab0 0x7f4f580a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54688/0 (socket says 172.21.15.94:54688) 2024-03-06T07:22:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.813+0000 7f4f56ffd640 1 -- 172.21.15.94:0/3130613272 learned_addr learned my addr 172.21.15.94:0/3130613272 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.813+0000 7f4f567fc640 1 -- 172.21.15.94:0/3130613272 >> v1:172.21.15.145:6789/0 conn(0x7f4f5814f110 legacy=0x7f4f5815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.813+0000 7f4f567fc640 1 -- 172.21.15.94:0/3130613272 >> v1:172.21.15.94:6789/0 conn(0x7f4f580c6ab0 legacy=0x7f4f580a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.813+0000 7f4f55ffb640 1 -- 172.21.15.94:0/3130613272 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908202275 0 0) 0x7f4f580a9990 con 0x7f4f580c6ab0 2024-03-06T07:22:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.813+0000 7f4f567fc640 1 -- 172.21.15.94:0/3130613272 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f581559a0 con 0x7f4f581561a0 2024-03-06T07:22:11.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f567fc640 1 --2- 172.21.15.94:0/3130613272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f581561a0 0x7f4f581529e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0097a0 tx=0x7f4f4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39634ce6670f2e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f55ffb640 1 -- 172.21.15.94:0/3130613272 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c035020 con 0x7f4f581561a0 2024-03-06T07:22:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f55ffb640 1 -- 172.21.15.94:0/3130613272 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c031830 con 0x7f4f581561a0 2024-03-06T07:22:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f55ffb640 1 -- 172.21.15.94:0/3130613272 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c031c00 con 0x7f4f581561a0 2024-03-06T07:22:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f5d97a640 1 -- 172.21.15.94:0/3130613272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f581561a0 msgr2=0x7f4f581529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/3130613272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f581561a0 0x7f4f581529e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0097a0 tx=0x7f4f4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f5d97a640 1 -- 172.21.15.94:0/3130613272 shutdown_connections 2024-03-06T07:22:11.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/3130613272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f581561a0 0x7f4f581529e0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f5d97a640 1 -- 172.21.15.94:0/3130613272 >> 172.21.15.94:0/3130613272 conn(0x7f4f5806b1a0 msgr2=0x7f4f5806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f5d97a640 1 -- 172.21.15.94:0/3130613272 shutdown_connections 2024-03-06T07:22:11.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.814+0000 7f4f5d97a640 1 -- 172.21.15.94:0/3130613272 wait complete. 2024-03-06T07:22:11.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.815+0000 7f4f5d97a640 1 Processor -- start 2024-03-06T07:22:11.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.816+0000 7f4f5d97a640 1 -- start start 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.816+0000 7f4f5d97a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f580c6ab0 0x7f4f581ee6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.816+0000 7f4f5d97a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5814f110 0x7f4f581527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.816+0000 7f4f5d97a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f581561a0 0x7f4f58145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.816+0000 7f4f5d97a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f580aa600 con 0x7f4f5814f110 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.816+0000 7f4f5d97a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f580c2a90 con 0x7f4f581561a0 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.816+0000 7f4f5d97a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f58153f80 con 0x7f4f580c6ab0 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5814f110 0x7f4f581527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5814f110 0x7f4f581527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33568/0 (socket says 172.21.15.94:33568) 2024-03-06T07:22:11.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 -- 172.21.15.94:0/1601912540 learned_addr learned my addr 172.21.15.94:0/1601912540 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f577fe640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f581561a0 0x7f4f58145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f580c6ab0 0x7f4f581ee6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 -- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f580c6ab0 msgr2=0x7f4f581ee6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f580c6ab0 0x7f4f581ee6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 -- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f581561a0 msgr2=0x7f4f58145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f581561a0 0x7f4f58145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 -- 172.21.15.94:0/1601912540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f581dfe50 con 0x7f4f5814f110 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f567fc640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5814f110 0x7f4f581527c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c02bee0 tx=0x7f4f4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f3ffff640 1 -- 172.21.15.94:0/1601912540 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c041070 con 0x7f4f5814f110 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f3ffff640 1 -- 172.21.15.94:0/1601912540 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c03aa30 con 0x7f4f5814f110 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f3ffff640 1 -- 172.21.15.94:0/1601912540 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c035040 con 0x7f4f5814f110 2024-03-06T07:22:11.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5814dd60 con 0x7f4f5814f110 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.817+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5814d7a0 con 0x7f4f5814f110 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.819+0000 7f4f3ffff640 1 -- 172.21.15.94:0/1601912540 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f4c038020 con 0x7f4f5814f110 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.819+0000 7f4f3ffff640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4406b4d0 0x7f4f4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.819+0000 7f4f3ffff640 1 -- 172.21.15.94:0/1601912540 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f4c0abd00 con 0x7f4f5814f110 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.819+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4406b4d0 0x7f4f4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.820+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4406b4d0 0x7f4f4406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4f48000f60 tx=0x7f4f48007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.820+0000 7f4f60564740 1 Processor -- start 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.820+0000 7f4f60564740 1 -- start start 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f60564740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472b6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b74b0 0x55f1472b3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f60564740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14721d6b0 con 0x55f1472b9330 2024-03-06T07:22:11.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1471c6a60 con 0x55f1472b74b0 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f147219750 con 0x55f1472b8f50 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472b6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472b6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33570/0 (socket says 172.21.15.94:33570) 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 -- 172.21.15.94:0/4255041843 learned_addr learned my addr 172.21.15.94:0/4255041843 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f577fe640 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b74b0 0x55f1472b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 -- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 msgr2=0x55f1472b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 -- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b74b0 msgr2=0x55f1472b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b74b0 0x55f1472b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.821+0000 7f4f567fc640 1 -- 172.21.15.94:0/4255041843 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1471e4f30 con 0x55f1472b9330 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f567fc640 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472b6f70 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c07a3c0 tx=0x7f4f4c07a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6608518dc05b09d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f3d7fa640 1 -- 172.21.15.94:0/4255041843 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c082a90 con 0x55f1472b9330 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f3d7fa640 1 -- 172.21.15.94:0/4255041843 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c0784d0 con 0x55f1472b9330 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f3d7fa640 1 -- 172.21.15.94:0/4255041843 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c07e170 con 0x55f1472b9330 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 -- 172.21.15.94:0/4255041843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 msgr2=0x55f1472b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472b6f70 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c07a3c0 tx=0x7f4f4c07a430 comp rx=0 tx=0).stop 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 -- 172.21.15.94:0/4255041843 shutdown_connections 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b74b0 0x55f1472b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472b6f70 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 --2- 172.21.15.94:0/4255041843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 -- 172.21.15.94:0/4255041843 >> 172.21.15.94:0/4255041843 conn(0x55f1472b57d0 msgr2=0x55f1472da3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 -- 172.21.15.94:0/4255041843 shutdown_connections 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.822+0000 7f4f60564740 1 -- 172.21.15.94:0/4255041843 wait complete. 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.823+0000 7f4f60564740 1 -- start start 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.823+0000 7f4f60564740 1 Processor -- start 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b8f50 0x55f1472d97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f1472d5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14721d6b0 con 0x55f1472b8f50 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1471d9ea0 con 0x55f1472f29f0 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1472db230 con 0x55f1472b9330 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 10 client.0 _ll_register_callbacks cb 0x55f1471d9870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:11.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b8f50 0x55f1472d97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b8f50 0x55f1472d97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33582/0 (socket says 172.21.15.94:33582) 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 -- 172.21.15.94:0/805495830 learned_addr learned my addr 172.21.15.94:0/805495830 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f577fe640 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f1472d5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f567fc640 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 -- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 msgr2=0x55f1472dc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 -- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 msgr2=0x55f1472d5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f1472d5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 -- 172.21.15.94:0/805495830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1471da270 con 0x55f1472b8f50 2024-03-06T07:22:11.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b8f50 0x55f1472d97b0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4f480136a0 tx=0x7f4f48000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b84792a3e4e30da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f367fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f367fc640 1 -- 172.21.15.94:0/805495830 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48020580 con 0x55f1472b8f50 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f367fc640 1 -- 172.21.15.94:0/805495830 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f48020720 con 0x55f1472b8f50 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f4c0784d0 con 0x55f1472b8f50 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f367fc640 1 -- 172.21.15.94:0/805495830 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f4802e050 con 0x55f1472b8f50 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.824+0000 7f4f60564740 10 client.7512 fetch_fsmap learned FSMap version 89 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55f146f322d0 con 0x55f1472b8f50 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f367fc640 1 -- 172.21.15.94:0/805495830 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c0784d0 con 0x55f1472b8f50 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f367fc640 1 -- 172.21.15.94:0/805495830 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 89) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4f48031020 con 0x55f1472b8f50 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f60564740 10 client.7512 fetch_fsmap finished waiting for FSMap version 89 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f60564740 10 client.7512 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f60564740 20 client.7512 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:11.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f60564740 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1473095d0 0x55f147314110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f60564740 4 client.7512 mds_command: new command op to 36134 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 --> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f14722abf0 con 0x55f1473095d0 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f567fc640 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1473095d0 0x55f147314110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f367fc640 10 client.7512 ms_handle_connect on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.825+0000 7f4f567fc640 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1473095d0 0x55f147314110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=268237ffc45071f server_cookie=46ea5f62ee908af5 in_seq=0 out_seq=0 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f367fc640 1 -- 172.21.15.94:0/805495830 <== mds.0 v2:172.21.15.145:6834/4001418202 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f14722abf0 con 0x55f1473095d0 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f367fc640 10 client.7512 handle_command_reply: tid=0 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 client.7512 shutdown 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b8f50 msgr2=0x55f1472d97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b8f50 0x55f1472d97b0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4f480136a0 tx=0x7f4f48000ed0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 shutdown_connections 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f1472d5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1473095d0 0x55f147314110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f367fc640 0 client.7512 ms_handle_reset on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:11.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 --2- 172.21.15.94:0/805495830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b8f50 0x55f1472d97b0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 >> 172.21.15.94:0/805495830 conn(0x55f1472b74b0 msgr2=0x55f1472b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 shutdown_connections 2024-03-06T07:22:11.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 1 -- 172.21.15.94:0/805495830 wait complete. 2024-03-06T07:22:11.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.826+0000 7f4f60564740 20 client.7512 trim_cache size 0 max 16384 2024-03-06T07:22:11.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.843+0000 7f4f60564740 1 Processor -- start 2024-03-06T07:22:11.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.843+0000 7f4f60564740 1 -- start start 2024-03-06T07:22:11.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.843+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b8f50 0x55f1472b2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.843+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.843+0000 7f4f60564740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472f29f0 0x55f1472ddae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.843+0000 7f4f60564740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581f7b10 con 0x55f1472f29f0 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581f6c40 con 0x55f1472b8f50 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f58073250 con 0x55f1472b9330 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472f29f0 0x55f1472ddae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472f29f0 0x55f1472ddae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33594/0 (socket says 172.21.15.94:33594) 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 -- 172.21.15.94:0/4004345028 learned_addr learned my addr 172.21.15.94:0/4004345028 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b8f50 0x55f1472b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f567fc640 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 -- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 msgr2=0x55f1472dd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 -- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b8f50 msgr2=0x55f1472b2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b8f50 0x55f1472b2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 -- 172.21.15.94:0/4004345028 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f146ffc6b0 con 0x55f1472f29f0 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f577fe640 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472f29f0 0x55f1472ddae0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4f50002030 tx=0x7f4f50001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdc1d193a863e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f3cff9640 1 -- 172.21.15.94:0/4004345028 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50006b50 con 0x55f1472f29f0 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f3cff9640 1 -- 172.21.15.94:0/4004345028 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f50006cf0 con 0x55f1472f29f0 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.844+0000 7f4f3cff9640 1 -- 172.21.15.94:0/4004345028 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50003b70 con 0x55f1472f29f0 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 -- 172.21.15.94:0/4004345028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472f29f0 msgr2=0x55f1472ddae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472f29f0 0x55f1472ddae0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f4f50002030 tx=0x7f4f50001670 comp rx=0 tx=0).stop 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 -- 172.21.15.94:0/4004345028 shutdown_connections 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472f29f0 0x55f1472ddae0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b9330 0x55f1472dd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 --2- 172.21.15.94:0/4004345028 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472b8f50 0x55f1472b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 -- 172.21.15.94:0/4004345028 >> 172.21.15.94:0/4004345028 conn(0x55f1472b74b0 msgr2=0x55f147302c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 -- 172.21.15.94:0/4004345028 shutdown_connections 2024-03-06T07:22:11.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 -- 172.21.15.94:0/4004345028 wait complete. 2024-03-06T07:22:11.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 -- start start 2024-03-06T07:22:11.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.845+0000 7f4f60564740 1 Processor -- start 2024-03-06T07:22:11.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472d8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f60564740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472dda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f60564740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f147362a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f60564740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1472de3e0 con 0x55f1472b9330 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581f6640 con 0x55f1472f29f0 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f60564740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f1472dfa50 con 0x55f1472b8f50 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f60564740 10 client.0 _ll_register_callbacks cb 0x7f4f48004e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472dda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472dda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33610/0 (socket says 172.21.15.94:33610) 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 -- 172.21.15.94:0/4283704787 learned_addr learned my addr 172.21.15.94:0/4283704787 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472d8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f577fe640 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f147362a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 -- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 msgr2=0x55f1472d8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472d8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 -- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 msgr2=0x55f147362a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f147362a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.846+0000 7f4f567fc640 1 -- 172.21.15.94:0/4283704787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1472b71a0 con 0x55f1472b9330 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f567fc640 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472dda30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c07e690 tx=0x7f4f4c03d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6960aaa46d453563 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f367fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f367fc640 1 -- 172.21.15.94:0/4283704787 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c03aea0 con 0x55f1472b9330 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f367fc640 1 -- 172.21.15.94:0/4283704787 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f4c07e170 con 0x55f1472b9330 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f50006cf0 con 0x55f1472b9330 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f367fc640 1 -- 172.21.15.94:0/4283704787 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f4c076aa0 con 0x55f1472b9330 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 10 client.7518 fetch_fsmap learned FSMap version 89 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55f1472d8cc0 con 0x55f1472b9330 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f367fc640 1 -- 172.21.15.94:0/4283704787 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f50006cf0 con 0x55f1472b9330 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f367fc640 1 -- 172.21.15.94:0/4283704787 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 89) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4f4c079880 con 0x55f1472b9330 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 10 client.7518 fetch_fsmap finished waiting for FSMap version 89 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 10 client.7518 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 20 client.7518 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1472db690 0x55f1472dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 4 client.7518 mds_command: new command op to 36134 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 --> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f147224f40 con 0x55f1472db690 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.847+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1472db690 0x55f1472dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f56ffd640 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1472db690 0x55f1472dba70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2336dd75416d15e server_cookie=4952b259032c6a89 in_seq=0 out_seq=0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f367fc640 10 client.7518 ms_handle_connect on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f367fc640 1 -- 172.21.15.94:0/4283704787 <== mds.0 v2:172.21.15.145:6834/4001418202 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55f147224f40 con 0x55f1472db690 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f367fc640 10 client.7518 handle_command_reply: tid=0 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 client.7518 shutdown 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 msgr2=0x55f1472dda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472dda30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c07e690 tx=0x7f4f4c03d070 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 shutdown_connections 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f1472f29f0 0x55f147362a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f1472b9330 0x55f1472dda30 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55f1472db690 0x55f1472dba70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 --2- 172.21.15.94:0/4283704787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f1472b8f50 0x55f1472d8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f367fc640 0 client.7518 ms_handle_reset on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 >> 172.21.15.94:0/4283704787 conn(0x55f1472b74b0 msgr2=0x55f1472d5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 shutdown_connections 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 1 -- 172.21.15.94:0/4283704787 wait complete. 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.848+0000 7f4f60564740 20 client.7518 trim_cache size 0 max 16384 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 36134, 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 5.5907528060000002, 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 89, 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 108, 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 10.685432464 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4406b4d0 msgr2=0x7f4f4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4406b4d0 0x7f4f4406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4f48000f60 tx=0x7f4f48007e40 comp rx=0 tx=0).stop 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5814f110 msgr2=0x7f4f581527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5814f110 0x7f4f581527c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c02bee0 tx=0x7f4f4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 shutdown_connections 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4406b4d0 0x7f4f4406d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f581561a0 0x7f4f58145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5814f110 0x7f4f581527c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 --2- 172.21.15.94:0/1601912540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f580c6ab0 0x7f4f581ee6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 >> 172.21.15.94:0/1601912540 conn(0x7f4f5806b1a0 msgr2=0x7f4f580ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 shutdown_connections 2024-03-06T07:22:11.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.849+0000 7f4f5d97a640 1 -- 172.21.15.94:0/1601912540 wait complete. 2024-03-06T07:22:11.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:11.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.964+0000 7f05a38b5640 1 Processor -- start 2024-03-06T07:22:11.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.964+0000 7f05a38b5640 1 -- start start 2024-03-06T07:22:11.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a38b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f059c0aa180 0x7f059c14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a38b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c14f140 0x7f059c0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a38b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c155bb0 con 0x7f059c1550c0 2024-03-06T07:22:11.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a38b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c0c4610 con 0x7f059c0aa180 2024-03-06T07:22:11.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a38b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c14f5f0 con 0x7f059c14f140 2024-03-06T07:22:11.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a1651640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f059c1550c0 0x7f059c1554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54694/0 (socket says 172.21.15.94:54694) 2024-03-06T07:22:11.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a1651640 1 -- 172.21.15.94:0/1106930412 learned_addr learned my addr 172.21.15.94:0/1106930412 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:11.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a1e52640 1 --2- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c14f140 0x7f059c0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f05a0e50640 1 --2- 172.21.15.94:0/1106930412 >> v2:172.21.15.145:3300/0 conn(0x7f059c0aa180 0x7f059c14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f0593fff640 1 -- 172.21.15.94:0/1106930412 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503485640 0 0) 0x7f059c155bb0 con 0x7f059c1550c0 2024-03-06T07:22:11.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.965+0000 7f0593fff640 1 -- 172.21.15.94:0/1106930412 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0588003520 con 0x7f059c1550c0 2024-03-06T07:22:11.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f05a0e50640 1 -- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c14f140 msgr2=0x7f059c0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f05a0e50640 1 --2- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c14f140 0x7f059c0c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f05a0e50640 1 -- 172.21.15.94:0/1106930412 >> v1:172.21.15.94:6789/0 conn(0x7f059c1550c0 legacy=0x7f059c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f05a0e50640 1 -- 172.21.15.94:0/1106930412 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0aa660 con 0x7f059c0aa180 2024-03-06T07:22:11.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f05a1e52640 1 --2- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c14f140 0x7f059c0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f05a0e50640 1 --2- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c0aa180 0x7f059c14ec00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f058c00a940 tx=0x7f058c02cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3496769db231dcea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f0593fff640 1 -- 172.21.15.94:0/1106930412 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c03a070 con 0x7f059c0aa180 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f0593fff640 1 -- 172.21.15.94:0/1106930412 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c02da20 con 0x7f059c0aa180 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.966+0000 7f0593fff640 1 -- 172.21.15.94:0/1106930412 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c00b040 con 0x7f059c0aa180 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 -- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c0aa180 msgr2=0x7f059c14ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c0aa180 0x7f059c14ec00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f058c00a940 tx=0x7f058c02cc70 comp rx=0 tx=0).stop 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 -- 172.21.15.94:0/1106930412 shutdown_connections 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c14f140 0x7f059c0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1106930412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c0aa180 0x7f059c14ec00 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 -- 172.21.15.94:0/1106930412 >> 172.21.15.94:0/1106930412 conn(0x7f059c0c6710 msgr2=0x7f059c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 -- 172.21.15.94:0/1106930412 shutdown_connections 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.967+0000 7f05a38b5640 1 -- 172.21.15.94:0/1106930412 wait complete. 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.968+0000 7f05a38b5640 1 Processor -- start 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.968+0000 7f05a38b5640 1 -- start start 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a38b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c0aa180 0x7f059c14da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a38b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f059c14f140 0x7f059c1ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a38b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c1550c0 0x7f059c1ee370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a38b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0c1310 con 0x7f059c14f140 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a38b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0c4860 con 0x7f059c1550c0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a38b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c0c1490 con 0x7f059c0aa180 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f059c14f140 0x7f059c1ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f059c14f140 0x7f059c1ede30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33616/0 (socket says 172.21.15.94:33616) 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 -- 172.21.15.94:0/1509710577 learned_addr learned my addr 172.21.15.94:0/1509710577 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a1e52640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c1550c0 0x7f059c1ee370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a1651640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c0aa180 0x7f059c14da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 -- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c0aa180 msgr2=0x7f059c14da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c0aa180 0x7f059c14da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 -- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c1550c0 msgr2=0x7f059c1ee370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c1550c0 0x7f059c1ee370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 -- 172.21.15.94:0/1509710577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0a5ee0 con 0x7f059c14f140 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.969+0000 7f05a0e50640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f059c14f140 0x7f059c1ede30 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f058c038040 tx=0x7f058c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.970+0000 7f0591ffb640 1 -- 172.21.15.94:0/1509710577 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c045070 con 0x7f059c14f140 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.970+0000 7f0591ffb640 1 -- 172.21.15.94:0/1509710577 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c036ce0 con 0x7f059c14f140 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.970+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c0a6c40 con 0x7f059c14f140 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.970+0000 7f0591ffb640 1 -- 172.21.15.94:0/1509710577 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c03a070 con 0x7f059c14f140 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.970+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c154c00 con 0x7f059c14f140 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.971+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c04f330 con 0x7f059c14f140 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.971+0000 7f0591ffb640 1 -- 172.21.15.94:0/1509710577 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f058c03e070 con 0x7f059c14f140 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.971+0000 7f0591ffb640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f058006b4d0 0x7f058006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.971+0000 7f0591ffb640 1 -- 172.21.15.94:0/1509710577 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f058c0ab500 con 0x7f059c14f140 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.972+0000 7f05a1651640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f058006b4d0 0x7f058006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.972+0000 7f05a1651640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f058006b4d0 0x7f058006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0598005880 tx=0x7f0598009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:11.976+0000 7f0591ffb640 1 -- 172.21.15.94:0/1509710577 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f058c079bb0 con 0x7f059c14f140 2024-03-06T07:22:12.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.184+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f059c068450 con 0x7f059c14f140 2024-03-06T07:22:12.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.184+0000 7f0591ffb640 1 -- 172.21.15.94:0/1509710577 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4990 (secure 0 0 0) 0x7f058c07dc50 con 0x7f059c14f140 2024-03-06T07:22:12.186 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:12.187 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2722633283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2722633283},{"type":"v1","addr":"172.21.15.145:6837","nonce":2722633283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:08.543964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":36134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_36134":{"gid":36134,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/4001418202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4001418202},{"type":"v1","addr":"172.21.15.145:6836","nonce":4001418202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":36134,"qdb_cluster":[36134]},"id":8}]} 2024-03-06T07:22:12.187 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 89 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f058006b4d0 msgr2=0x7f058006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f058006b4d0 0x7f058006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0598005880 tx=0x7f0598009040 comp rx=0 tx=0).stop 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f059c14f140 msgr2=0x7f059c1ede30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f059c14f140 0x7f059c1ede30 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f058c038040 tx=0x7f058c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 shutdown_connections 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f058006b4d0 0x7f058006d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f059c1550c0 0x7f059c1ee370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f059c14f140 0x7f059c1ede30 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 --2- 172.21.15.94:0/1509710577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f059c0aa180 0x7f059c14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.186+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 >> 172.21.15.94:0/1509710577 conn(0x7f059c0c6710 msgr2=0x7f059c145e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.187+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 shutdown_connections 2024-03-06T07:22:12.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.187+0000 7f05a38b5640 1 -- 172.21.15.94:0/1509710577 wait complete. 2024-03-06T07:22:12.200 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:05.610686+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_36134": { "addr": "172.21.15.145:6836/4001418202", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 4001418202, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 4001418202, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 36134, "incarnation": 86, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:22:08.543964+0000", "qdb_cluster": [ 36134 ], "qdb_leader": 36134, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 36134 } } } ], "standbys": [ { "addr": "172.21.15.94:6834/2226553926", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 2226553926, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 2226553926, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7350, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3093758100", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3093758100, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3093758100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7353, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/2722633283", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 2722633283, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 2722633283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7371, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:12.200 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:22:12.200 INFO:tasks.cephfs.test_failover:current = [{'gid': 7350, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7371, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/2722633283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2722633283}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2722633283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7353, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3093758100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3093758100}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3093758100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 36134, 'name': 'd', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/4001418202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 4001418202}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 4001418202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:22:12.200 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs 2024-03-06T07:22:12.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.302+0000 7f24ec646640 1 Processor -- start 2024-03-06T07:22:12.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.302+0000 7f24ec646640 1 -- start start 2024-03-06T07:22:12.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24ec646640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e40c1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24ec646640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24e40a7cd0 0x7f24e40c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24ec646640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e4155f40 con 0x7f24e4152880 2024-03-06T07:22:12.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24ec646640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e4072510 con 0x7f24e40a7cd0 2024-03-06T07:22:12.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24ec646640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e4072db0 con 0x7f24e40c6710 2024-03-06T07:22:12.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24eabe3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24e40a7cd0 0x7f24e40c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24eabe3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24e40a7cd0 0x7f24e40c23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33624/0 (socket says 172.21.15.94:33624) 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24eabe3640 1 -- 172.21.15.94:0/3187873370 learned_addr learned my addr 172.21.15.94:0/3187873370 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24ea3e2640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f24e4152880 0x7f24e40c3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:59574/0 (socket says 172.21.15.94:59574) 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.303+0000 7f24e9be1640 1 --2- 172.21.15.94:0/3187873370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e40c1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.304+0000 7f24e93e0640 1 -- 172.21.15.94:0/3187873370 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268388405 0 0) 0x7f24e4155f40 con 0x7f24e4152880 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.304+0000 7f24eabe3640 1 -- 172.21.15.94:0/3187873370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 msgr2=0x7f24e40c1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.304+0000 7f24eabe3640 1 --2- 172.21.15.94:0/3187873370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e40c1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.304+0000 7f24eabe3640 1 -- 172.21.15.94:0/3187873370 >> v1:172.21.15.145:6789/0 conn(0x7f24e4152880 legacy=0x7f24e40c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.304+0000 7f24eabe3640 1 -- 172.21.15.94:0/3187873370 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e4072760 con 0x7f24e40a7cd0 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.304+0000 7f24eabe3640 1 --2- 172.21.15.94:0/3187873370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e40c23c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009870 tx=0x7f24dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a6a4f27296bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.304+0000 7f24e93e0640 1 -- 172.21.15.94:0/3187873370 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc035020 con 0x7f24e40a7cd0 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24e93e0640 1 -- 172.21.15.94:0/3187873370 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc031640 con 0x7f24e40a7cd0 2024-03-06T07:22:12.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24e93e0640 1 -- 172.21.15.94:0/3187873370 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc02d040 con 0x7f24e40a7cd0 2024-03-06T07:22:12.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 -- 172.21.15.94:0/3187873370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 msgr2=0x7f24e40c23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 --2- 172.21.15.94:0/3187873370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e40c23c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009870 tx=0x7f24dc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 -- 172.21.15.94:0/3187873370 shutdown_connections 2024-03-06T07:22:12.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 --2- 172.21.15.94:0/3187873370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e40c23c0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 --2- 172.21.15.94:0/3187873370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e40c1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 -- 172.21.15.94:0/3187873370 >> 172.21.15.94:0/3187873370 conn(0x7f24e40aa5c0 msgr2=0x7f24e40aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 -- 172.21.15.94:0/3187873370 shutdown_connections 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.305+0000 7f24ec646640 1 -- 172.21.15.94:0/3187873370 wait complete. 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.306+0000 7f24ec646640 1 Processor -- start 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.306+0000 7f24ec646640 1 -- start start 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ec646640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e41ea900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ec646640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ec646640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24e4152880 0x7f24e4153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ec646640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e40c3aa0 con 0x7f24e40a7cd0 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ec646640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e40c3920 con 0x7f24e4152880 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ec646640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e4155b80 con 0x7f24e40c6710 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e41ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e41ea900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33628/0 (socket says 172.21.15.94:33628) 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 -- 172.21.15.94:0/1837545069 learned_addr learned my addr 172.21.15.94:0/1837545069 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:12.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24e9be1640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24eabe3640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24e4152880 0x7f24e4153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 -- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 msgr2=0x7f24e406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e406fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 -- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24e4152880 msgr2=0x7f24e4153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24e4152880 0x7f24e4153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 -- 172.21.15.94:0/1837545069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e41eae40 con 0x7f24e40a7cd0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.307+0000 7f24ea3e2640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e41ea900 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f24e000c7c0 tx=0x7f24e000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.308+0000 7f24daffd640 1 -- 172.21.15.94:0/1837545069 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0018070 con 0x7f24e40a7cd0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.308+0000 7f24daffd640 1 -- 172.21.15.94:0/1837545069 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e000a640 con 0x7f24e40a7cd0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.308+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e41eb0e0 con 0x7f24e40a7cd0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.308+0000 7f24daffd640 1 -- 172.21.15.94:0/1837545069 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e000aa30 con 0x7f24e40a7cd0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.308+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e41eb6a0 con 0x7f24e40a7cd0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.309+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e404f330 con 0x7f24e40a7cd0 2024-03-06T07:22:12.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.309+0000 7f24daffd640 1 -- 172.21.15.94:0/1837545069 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24e001b070 con 0x7f24e40a7cd0 2024-03-06T07:22:12.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.309+0000 7f24daffd640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24c806b500 0x7f24c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.310+0000 7f24daffd640 1 -- 172.21.15.94:0/1837545069 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24e0089840 con 0x7f24e40a7cd0 2024-03-06T07:22:12.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.310+0000 7f24e9be1640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24c806b500 0x7f24c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.310+0000 7f24e9be1640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24c806b500 0x7f24c806d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f24e406d690 tx=0x7f24d4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.314+0000 7f24daffd640 1 -- 172.21.15.94:0/1837545069 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24e0057f70 con 0x7f24e40a7cd0 2024-03-06T07:22:12.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.487+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f24e41eb8a0 con 0x7f24e40a7cd0 2024-03-06T07:22:12.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.490+0000 7f24daffd640 1 -- 172.21.15.94:0/1837545069 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v36)=0 v36) v1 ==== 116+0+0 (secure 0 0 0) 0x7f24e005c010 con 0x7f24e40a7cd0 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24c806b500 msgr2=0x7f24c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24c806b500 0x7f24c806d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f24e406d690 tx=0x7f24d4009190 comp rx=0 tx=0).stop 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 msgr2=0x7f24e41ea900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e41ea900 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f24e000c7c0 tx=0x7f24e000cc80 comp rx=0 tx=0).stop 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 shutdown_connections 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24c806b500 0x7f24c806d9c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24e4152880 0x7f24e4153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24e40c6710 0x7f24e406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 --2- 172.21.15.94:0/1837545069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24e40a7cd0 0x7f24e41ea900 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 >> 172.21.15.94:0/1837545069 conn(0x7f24e40aa5c0 msgr2=0x7f24e40c1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 shutdown_connections 2024-03-06T07:22:12.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.492+0000 7f24ec646640 1 -- 172.21.15.94:0/1837545069 wait complete. 2024-03-06T07:22:12.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:12.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f2b125640 1 Processor -- start 2024-03-06T07:22:12.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f2b125640 1 -- start start 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f2b125640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4f24159100 0x7f4f240c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f2b125640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f240c7c60 con 0x7f4f240c7180 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f2b125640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f240c8060 con 0x7f4f241527b0 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f2b125640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f240c8df0 con 0x7f4f24159100 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f23fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4f240c7180 0x7f4f240c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54696/0 (socket says 172.21.15.94:54696) 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.603+0000 7f4f23fff640 1 -- 172.21.15.94:0/8496699 learned_addr learned my addr 172.21.15.94:0/8496699 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.604+0000 7f4f296c2640 1 --2- 172.21.15.94:0/8496699 >> v2:172.21.15.145:3300/0 conn(0x7f4f24159100 0x7f4f240c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.604+0000 7f4f237fe640 1 -- 172.21.15.94:0/8496699 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061171052 0 0) 0x7f4f10001110 con 0x7f4f240c7180 2024-03-06T07:22:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.604+0000 7f4f237fe640 1 -- 172.21.15.94:0/8496699 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14003520 con 0x7f4f240c7180 2024-03-06T07:22:12.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.604+0000 7f4f296c2640 1 -- 172.21.15.94:0/8496699 >> v1:172.21.15.145:6789/0 conn(0x7f4f241527b0 legacy=0x7f4f24155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.604+0000 7f4f296c2640 1 -- 172.21.15.94:0/8496699 >> v1:172.21.15.94:6789/0 conn(0x7f4f240c7180 legacy=0x7f4f240c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.604+0000 7f4f296c2640 1 -- 172.21.15.94:0/8496699 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f240c76a0 con 0x7f4f24159100 2024-03-06T07:22:12.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f296c2640 1 --2- 172.21.15.94:0/8496699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 0x7f4f240c88b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009d90 tx=0x7f4f1c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91c61369ff333bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f237fe640 1 -- 172.21.15.94:0/8496699 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c039070 con 0x7f4f24159100 2024-03-06T07:22:12.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f237fe640 1 -- 172.21.15.94:0/8496699 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c02cc60 con 0x7f4f24159100 2024-03-06T07:22:12.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f237fe640 1 -- 172.21.15.94:0/8496699 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c034640 con 0x7f4f24159100 2024-03-06T07:22:12.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f2b125640 1 -- 172.21.15.94:0/8496699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 msgr2=0x7f4f240c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f2b125640 1 --2- 172.21.15.94:0/8496699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 0x7f4f240c88b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009d90 tx=0x7f4f1c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f2b125640 1 -- 172.21.15.94:0/8496699 shutdown_connections 2024-03-06T07:22:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f2b125640 1 --2- 172.21.15.94:0/8496699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 0x7f4f240c88b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f2b125640 1 -- 172.21.15.94:0/8496699 >> 172.21.15.94:0/8496699 conn(0x7f4f240ac560 msgr2=0x7f4f240ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f2b125640 1 -- 172.21.15.94:0/8496699 shutdown_connections 2024-03-06T07:22:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.605+0000 7f4f2b125640 1 -- 172.21.15.94:0/8496699 wait complete. 2024-03-06T07:22:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.606+0000 7f4f2b125640 1 Processor -- start 2024-03-06T07:22:12.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.606+0000 7f4f2b125640 1 -- start start 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f2b125640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f240c7180 0x7f4f241eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f2b125640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f241527b0 0x7f4f24154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f2b125640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 0x7f4f2414b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f2b125640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f24155600 con 0x7f4f240c7180 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f2b125640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f24155320 con 0x7f4f24159100 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f2b125640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f24152b90 con 0x7f4f241527b0 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f28ec1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f240c7180 0x7f4f241eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f28ec1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f240c7180 0x7f4f241eea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33632/0 (socket says 172.21.15.94:33632) 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f28ec1640 1 -- 172.21.15.94:0/445522559 learned_addr learned my addr 172.21.15.94:0/445522559 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:12.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f23fff640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f241527b0 0x7f4f24154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.607+0000 7f4f296c2640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 0x7f4f2414b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f28ec1640 1 -- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f241527b0 msgr2=0x7f4f24154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f28ec1640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f241527b0 0x7f4f24154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f28ec1640 1 -- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 msgr2=0x7f4f2414b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f28ec1640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 0x7f4f2414b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f28ec1640 1 -- 172.21.15.94:0/445522559 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f241eefa0 con 0x7f4f240c7180 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f28ec1640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f240c7180 0x7f4f241eea60 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c400 tx=0x7f4f1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f217fa640 1 -- 172.21.15.94:0/445522559 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f180052d0 con 0x7f4f240c7180 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f217fa640 1 -- 172.21.15.94:0/445522559 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f18005490 con 0x7f4f240c7180 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f240ae7e0 con 0x7f4f240c7180 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f217fa640 1 -- 172.21.15.94:0/445522559 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18007260 con 0x7f4f240c7180 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.608+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f24156590 con 0x7f4f240c7180 2024-03-06T07:22:12.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.609+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f241568c0 con 0x7f4f240c7180 2024-03-06T07:22:12.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.609+0000 7f4f217fa640 1 -- 172.21.15.94:0/445522559 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f18007400 con 0x7f4f240c7180 2024-03-06T07:22:12.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.610+0000 7f4f217fa640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f0806b4d0 0x7f4f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.610+0000 7f4f217fa640 1 -- 172.21.15.94:0/445522559 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f180896b0 con 0x7f4f240c7180 2024-03-06T07:22:12.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.610+0000 7f4f23fff640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f0806b4d0 0x7f4f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.610+0000 7f4f23fff640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f0806b4d0 0x7f4f0806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4f240c57e0 tx=0x7f4f10007b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.614+0000 7f4f217fa640 1 -- 172.21.15.94:0/445522559 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f18057de0 con 0x7f4f240c7180 2024-03-06T07:22:12.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.821+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f2404f330 con 0x7f4f240c7180 2024-03-06T07:22:12.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.822+0000 7f4f217fa640 1 -- 172.21.15.94:0/445522559 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+4990 (secure 0 0 0) 0x7f4f1805be80 con 0x7f4f240c7180 2024-03-06T07:22:12.824 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:12.824 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":89,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2722633283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2722633283},{"type":"v1","addr":"172.21.15.145:6837","nonce":2722633283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:08.543964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":36134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_36134":{"gid":36134,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/4001418202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4001418202},{"type":"v1","addr":"172.21.15.145:6836","nonce":4001418202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":36134,"qdb_cluster":[36134]},"id":8}]} 2024-03-06T07:22:12.824 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 89 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f0806b4d0 msgr2=0x7f4f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f0806b4d0 0x7f4f0806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4f240c57e0 tx=0x7f4f10007b70 comp rx=0 tx=0).stop 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f240c7180 msgr2=0x7f4f241eea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f240c7180 0x7f4f241eea60 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c400 tx=0x7f4f1800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 shutdown_connections 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f0806b4d0 0x7f4f0806d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f24159100 0x7f4f2414b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f241527b0 0x7f4f24154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 --2- 172.21.15.94:0/445522559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f240c7180 0x7f4f241eea60 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 >> 172.21.15.94:0/445522559 conn(0x7f4f240ac560 msgr2=0x7f4f241519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 shutdown_connections 2024-03-06T07:22:12.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:12.824+0000 7f4f2b125640 1 -- 172.21.15.94:0/445522559 wait complete. 2024-03-06T07:22:13.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:05.610686+0000', 'modified': '2024-03-06T07:22:08.543964+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 36134}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_36134': {'gid': 36134, 'name': 'd', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/4001418202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 4001418202}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 4001418202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 36134, 'qdb_cluster': [36134]} 2024-03-06T07:22:13.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:13.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:22:13.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.343+0000 7fbed459a640 1 Processor -- start 2024-03-06T07:22:13.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.343+0000 7fbed459a640 1 -- start start 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed459a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbecc06c240 0x7fbecc0c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed459a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbecc0ab860 0x7fbecc1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed459a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbecc1553c0 con 0x7fbecc0ac2b0 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed459a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbecc1556a0 con 0x7fbecc0ab860 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed459a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbecc155980 con 0x7fbecc06c240 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2b37640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbecc0ab860 0x7fbecc1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2b37640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbecc0ab860 0x7fbecc1588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33648/0 (socket says 172.21.15.94:33648) 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2b37640 1 -- 172.21.15.94:0/3851741898 learned_addr learned my addr 172.21.15.94:0/3851741898 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:13.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2336640 1 --2- 172.21.15.94:0/3851741898 >> v2:172.21.15.145:3300/0 conn(0x7fbecc06c240 0x7fbecc0c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2b37640 1 -- 172.21.15.94:0/3851741898 >> v2:172.21.15.145:3300/0 conn(0x7fbecc06c240 msgr2=0x7fbecc0c4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2b37640 1 --2- 172.21.15.94:0/3851741898 >> v2:172.21.15.145:3300/0 conn(0x7fbecc06c240 0x7fbecc0c4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2b37640 1 -- 172.21.15.94:0/3851741898 >> v1:172.21.15.145:6789/0 conn(0x7fbecc0ac2b0 legacy=0x7fbecc0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.344+0000 7fbed2b37640 1 -- 172.21.15.94:0/3851741898 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc155c60 con 0x7fbecc0ab860 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.345+0000 7fbed2b37640 1 --2- 172.21.15.94:0/3851741898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc0ab860 0x7fbecc1588a0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0c8000 tx=0x7fbec402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79edef28802aa085 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.345+0000 7fbed1334640 1 -- 172.21.15.94:0/3851741898 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec402e070 con 0x7fbecc0ab860 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.345+0000 7fbed1334640 1 -- 172.21.15.94:0/3851741898 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec4037070 con 0x7fbecc0ab860 2024-03-06T07:22:13.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.345+0000 7fbed1334640 1 -- 172.21.15.94:0/3851741898 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4031a00 con 0x7fbecc0ab860 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 -- 172.21.15.94:0/3851741898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc0ab860 msgr2=0x7fbecc1588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 --2- 172.21.15.94:0/3851741898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc0ab860 0x7fbecc1588a0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0c8000 tx=0x7fbec402b880 comp rx=0 tx=0).stop 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 -- 172.21.15.94:0/3851741898 shutdown_connections 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 --2- 172.21.15.94:0/3851741898 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc0ab860 0x7fbecc1588a0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 --2- 172.21.15.94:0/3851741898 >> v2:172.21.15.145:3300/0 conn(0x7fbecc06c240 0x7fbecc0c4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 -- 172.21.15.94:0/3851741898 >> 172.21.15.94:0/3851741898 conn(0x7fbecc0c7670 msgr2=0x7fbecc0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 -- 172.21.15.94:0/3851741898 shutdown_connections 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.346+0000 7fbed459a640 1 -- 172.21.15.94:0/3851741898 wait complete. 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.347+0000 7fbed459a640 1 Processor -- start 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.347+0000 7fbed459a640 1 -- start start 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc06c240 0x7fbecc1490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbecc0ab860 0x7fbecc147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbecc0ac2b0 0x7fbecc147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc0c4830 con 0x7fbecc06c240 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc156180 con 0x7fbecc0ab860 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc06c620 con 0x7fbecc0ac2b0 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc06c240 0x7fbecc1490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc06c240 0x7fbecc1490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33658/0 (socket says 172.21.15.94:33658) 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 -- 172.21.15.94:0/1729918847 learned_addr learned my addr 172.21.15.94:0/1729918847 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed1b35640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbecc0ab860 0x7fbecc147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2b37640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbecc0ac2b0 0x7fbecc147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 -- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbecc0ac2b0 msgr2=0x7fbecc147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbecc0ac2b0 0x7fbecc147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 -- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbecc0ab860 msgr2=0x7fbecc147810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbecc0ab860 0x7fbecc147810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 -- 172.21.15.94:0/1729918847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc14a850 con 0x7fbecc06c240 2024-03-06T07:22:13.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2b37640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbecc0ac2b0 0x7fbecc147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed2336640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc06c240 0x7fbecc1490e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fbec800efc0 tx=0x7fbec800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbec2ffd640 1 -- 172.21.15.94:0/1729918847 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8018020 con 0x7fbecc06c240 2024-03-06T07:22:13.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbec2ffd640 1 -- 172.21.15.94:0/1729918847 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec8014480 con 0x7fbecc06c240 2024-03-06T07:22:13.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbecc147360 con 0x7fbecc06c240 2024-03-06T07:22:13.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbec2ffd640 1 -- 172.21.15.94:0/1729918847 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8014850 con 0x7fbecc06c240 2024-03-06T07:22:13.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.348+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc1eb3a0 con 0x7fbecc06c240 2024-03-06T07:22:13.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.350+0000 7fbec2ffd640 1 -- 172.21.15.94:0/1729918847 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbec8005d10 con 0x7fbecc06c240 2024-03-06T07:22:13.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.350+0000 7fbec2ffd640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbeb006b500 0x7fbeb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.351+0000 7fbec2ffd640 1 -- 172.21.15.94:0/1729918847 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbec808ad20 con 0x7fbecc06c240 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.351+0000 7fbed1b35640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbeb006b500 0x7fbeb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.351+0000 7fbed1b35640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbeb006b500 0x7fbeb006d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0c3fb0 tx=0x7fbebc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.351+0000 7fbed7184740 1 Processor -- start 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed7184740 1 -- start start 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed7184740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f908f50 0x55a45f906a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f909330 0x55a45f906f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f9074b0 0x55a45f903680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed7184740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a45f86d6b0 con 0x55a45f908f50 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a45f816a60 con 0x55a45f9074b0 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a45f869750 con 0x55a45f909330 2024-03-06T07:22:13.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed2336640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f908f50 0x55a45f906a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed2336640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f908f50 0x55a45f906a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33660/0 (socket says 172.21.15.94:33660) 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed2336640 1 -- 172.21.15.94:0/831151035 learned_addr learned my addr 172.21.15.94:0/831151035 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed1b35640 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f909330 0x55a45f906f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.352+0000 7fbed2b37640 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f9074b0 0x55a45f903680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbed2336640 1 -- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f909330 msgr2=0x55a45f906f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbed2336640 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f909330 0x55a45f906f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbed2336640 1 -- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f9074b0 msgr2=0x55a45f903680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbed2336640 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f9074b0 0x55a45f903680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbed2336640 1 -- 172.21.15.94:0/831151035 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a45f834f30 con 0x55a45f908f50 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbed2336640 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f908f50 0x55a45f906a30 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fbec8055800 tx=0x7fbec8059460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45317d32538a226 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbeaffff640 1 -- 172.21.15.94:0/831151035 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec80576a0 con 0x55a45f908f50 2024-03-06T07:22:13.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbeaffff640 1 -- 172.21.15.94:0/831151035 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec805d160 con 0x55a45f908f50 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.353+0000 7fbeaffff640 1 -- 172.21.15.94:0/831151035 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec80587f0 con 0x55a45f908f50 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 -- 172.21.15.94:0/831151035 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f908f50 msgr2=0x55a45f906a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f908f50 0x55a45f906a30 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fbec8055800 tx=0x7fbec8059460 comp rx=0 tx=0).stop 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 -- 172.21.15.94:0/831151035 shutdown_connections 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f9074b0 0x55a45f903680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f909330 0x55a45f906f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 --2- 172.21.15.94:0/831151035 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f908f50 0x55a45f906a30 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 -- 172.21.15.94:0/831151035 >> 172.21.15.94:0/831151035 conn(0x55a45f9057d0 msgr2=0x55a45f92a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 -- 172.21.15.94:0/831151035 shutdown_connections 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 -- 172.21.15.94:0/831151035 wait complete. 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 -- start start 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.354+0000 7fbed7184740 1 Processor -- start 2024-03-06T07:22:13.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f908f50 0x55a45f907fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f909330 0x55a45f959460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed7184740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f92bdd0 0x55a45f924f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed7184740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a45f92b230 con 0x55a45f92bdd0 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a45f829ea0 con 0x55a45f909330 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a45f92ac70 con 0x55a45f908f50 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed7184740 10 client.0 _ll_register_callbacks cb 0x55a45f829870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed2b37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f92bdd0 0x55a45f924f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed1b35640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f909330 0x55a45f959460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed2336640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f908f50 0x55a45f907fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed2b37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f92bdd0 0x55a45f924f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33670/0 (socket says 172.21.15.94:33670) 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.355+0000 7fbed2b37640 1 -- 172.21.15.94:0/2180631837 learned_addr learned my addr 172.21.15.94:0/2180631837 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed2b37640 1 -- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f908f50 msgr2=0x55a45f907fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed2b37640 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f908f50 0x55a45f907fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed2b37640 1 -- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f909330 msgr2=0x55a45f959460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed2b37640 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f909330 0x55a45f959460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed2b37640 1 -- 172.21.15.94:0/2180631837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a45f82a270 con 0x55a45f92bdd0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed1b35640 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f909330 0x55a45f959460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed2b37640 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f92bdd0 0x55a45f924f20 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fbec402b8c0 tx=0x7fbec4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bc2556517e73c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbead7fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbead7fa640 1 -- 172.21.15.94:0/2180631837 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec404a020 con 0x55a45f92bdd0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbead7fa640 1 -- 172.21.15.94:0/2180631837 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbec4032670 con 0x55a45f92bdd0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbec805d160 con 0x55a45f92bdd0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbead7fa640 1 -- 172.21.15.94:0/2180631837 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbec4054050 con 0x55a45f92bdd0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 10 client.7548 fetch_fsmap learned FSMap version 89 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55a45f5822d0 con 0x55a45f92bdd0 2024-03-06T07:22:13.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbead7fa640 1 -- 172.21.15.94:0/2180631837 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec805d160 con 0x55a45f92bdd0 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbead7fa640 1 -- 172.21.15.94:0/2180631837 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 89) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbec4057020 con 0x55a45f92bdd0 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 10 client.7548 fetch_fsmap finished waiting for FSMap version 89 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 10 client.7548 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 20 client.7548 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f929f60 0x55a45f964760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 4 client.7548 mds_command: new command op to 36134 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.356+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 --> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a45f87abf0 con 0x55a45f929f60 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.357+0000 7fbed2336640 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f929f60 0x55a45f964760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.357+0000 7fbead7fa640 10 client.7548 ms_handle_connect on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.357+0000 7fbed2336640 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f929f60 0x55a45f964760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dbd8b4a971204d7 server_cookie=4d6290234e412abd in_seq=0 out_seq=0 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbead7fa640 1 -- 172.21.15.94:0/2180631837 <== mds.0 v2:172.21.15.145:6834/4001418202 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a45f87abf0 con 0x55a45f929f60 2024-03-06T07:22:13.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbead7fa640 10 client.7548 handle_command_reply: tid=0 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 client.7548 shutdown 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f92bdd0 msgr2=0x55a45f924f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f92bdd0 0x55a45f924f20 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fbec402b8c0 tx=0x7fbec4004730 comp rx=0 tx=0).stop 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 shutdown_connections 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f92bdd0 0x55a45f924f20 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f909330 0x55a45f959460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f929f60 0x55a45f964760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 --2- 172.21.15.94:0/2180631837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f908f50 0x55a45f907fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 >> 172.21.15.94:0/2180631837 conn(0x55a45f9074b0 msgr2=0x55a45f926e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbead7fa640 0 client.7548 ms_handle_reset on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 shutdown_connections 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 1 -- 172.21.15.94:0/2180631837 wait complete. 2024-03-06T07:22:13.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.358+0000 7fbed7184740 20 client.7548 trim_cache size 0 max 16384 2024-03-06T07:22:13.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.374+0000 7fbed7184740 1 Processor -- start 2024-03-06T07:22:13.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed7184740 1 -- start start 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f955b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed7184740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f9045f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f904b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed7184740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc1f6df0 con 0x55a45f909330 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc146370 con 0x55a45f908f50 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc1f67f0 con 0x55a45f92bdd0 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f9045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f9045f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33686/0 (socket says 172.21.15.94:33686) 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 -- 172.21.15.94:0/3319965098 learned_addr learned my addr 172.21.15.94:0/3319965098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed2b37640 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f904b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed2336640 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f955b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 -- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 msgr2=0x55a45f904b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f904b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 -- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 msgr2=0x55a45f955b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f955b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.375+0000 7fbed1b35640 1 -- 172.21.15.94:0/3319965098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a45f64c6b0 con 0x55a45f909330 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed2b37640 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f904b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed2336640 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f955b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed1b35640 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f9045f0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00ff60 tx=0x7fbebc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3014f9b121d674c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbeaf7fe640 1 -- 172.21.15.94:0/3319965098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc007400 con 0x55a45f909330 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbeaf7fe640 1 -- 172.21.15.94:0/3319965098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc0075a0 con 0x55a45f909330 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbeaf7fe640 1 -- 172.21.15.94:0/3319965098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0225f0 con 0x55a45f909330 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 -- 172.21.15.94:0/3319965098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 msgr2=0x55a45f9045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f9045f0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00ff60 tx=0x7fbebc012000 comp rx=0 tx=0).stop 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 -- 172.21.15.94:0/3319965098 shutdown_connections 2024-03-06T07:22:13.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f904b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f9045f0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 --2- 172.21.15.94:0/3319965098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f955b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 -- 172.21.15.94:0/3319965098 >> 172.21.15.94:0/3319965098 conn(0x55a45f9074b0 msgr2=0x55a45f924cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 -- 172.21.15.94:0/3319965098 shutdown_connections 2024-03-06T07:22:13.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.376+0000 7fbed7184740 1 -- 172.21.15.94:0/3319965098 wait complete. 2024-03-06T07:22:13.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.377+0000 7fbed7184740 1 -- start start 2024-03-06T07:22:13.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.377+0000 7fbed7184740 1 Processor -- start 2024-03-06T07:22:13.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f9255e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f927010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f951a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a45f925460 con 0x55a45f909330 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a45f92ee70 con 0x55a45f908f50 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a45f9293b0 con 0x55a45f92bdd0 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 10 client.0 _ll_register_callbacks cb 0x7fbec40319e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f927010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f927010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33702/0 (socket says 172.21.15.94:33702) 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 -- 172.21.15.94:0/561175732 learned_addr learned my addr 172.21.15.94:0/561175732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed2b37640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f951a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed2336640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f9255e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 -- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 msgr2=0x55a45f951a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f951a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 -- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 msgr2=0x55a45f9255e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f9255e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 -- 172.21.15.94:0/561175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a45f9282c0 con 0x55a45f909330 2024-03-06T07:22:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed2336640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f9255e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed1b35640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f927010 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbebc006f70 tx=0x7fbebc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b7d0d6df01b55d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbead7fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbead7fa640 1 -- 172.21.15.94:0/561175732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc03b060 con 0x55a45f909330 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbead7fa640 1 -- 172.21.15.94:0/561175732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbebc036400 con 0x55a45f909330 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.378+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbebc0075a0 con 0x55a45f909330 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbead7fa640 1 -- 172.21.15.94:0/561175732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbebc041050 con 0x55a45f909330 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 10 client.7554 fetch_fsmap learned FSMap version 89 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) v3 -- 0x55a45f947070 con 0x55a45f909330 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbead7fa640 1 -- 172.21.15.94:0/561175732 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc0075a0 con 0x55a45f909330 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbead7fa640 1 -- 172.21.15.94:0/561175732 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 89) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbebc044020 con 0x55a45f909330 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 10 client.7554 fetch_fsmap finished waiting for FSMap version 89 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 10 client.7554 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:13.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 20 client.7554 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f952350 0x55a45f9b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 4 client.7554 mds_command: new command op to 36134 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 --> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a45f874f40 con 0x55a45f952350 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed2336640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f952350 0x55a45f9b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbead7fa640 10 client.7554 ms_handle_connect on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.379+0000 7fbed2336640 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f952350 0x55a45f9b1f00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88b3229a2e7f2647 server_cookie=66acd94ca12aa72d in_seq=0 out_seq=0 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbead7fa640 1 -- 172.21.15.94:0/561175732 <== mds.0 v2:172.21.15.145:6834/4001418202 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55a45f874f40 con 0x55a45f952350 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbead7fa640 10 client.7554 handle_command_reply: tid=0 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 client.7554 shutdown 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 msgr2=0x55a45f927010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f927010 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbebc006f70 tx=0x7fbebc000ed0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 shutdown_connections 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:6834/4001418202,v1:172.21.15.145:6836/4001418202] conn(0x55a45f952350 0x55a45f9b1f00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a45f92bdd0 0x55a45f951a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a45f909330 0x55a45f927010 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbead7fa640 0 client.7554 ms_handle_reset on v2:172.21.15.145:6834/4001418202 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 --2- 172.21.15.94:0/561175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a45f908f50 0x55a45f9255e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 >> 172.21.15.94:0/561175732 conn(0x55a45f9074b0 msgr2=0x55a45f924b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 shutdown_connections 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 1 -- 172.21.15.94:0/561175732 wait complete. 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed7184740 20 client.7554 trim_cache size 0 max 16384 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:13.389 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 36134, 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 7.1224079339999999, 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 89, 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 108, 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 12.217087086999999 2024-03-06T07:22:13.390 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:13.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbeb006b500 msgr2=0x7fbeb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbeb006b500 0x7fbeb006d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0c3fb0 tx=0x7fbebc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:13.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc06c240 msgr2=0x7fbecc1490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc06c240 0x7fbecc1490e0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fbec800efc0 tx=0x7fbec800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:13.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 shutdown_connections 2024-03-06T07:22:13.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbeb006b500 0x7fbeb006d9c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbecc0ac2b0 0x7fbecc147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbecc0ab860 0x7fbecc147810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 --2- 172.21.15.94:0/1729918847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbecc06c240 0x7fbecc1490e0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 >> 172.21.15.94:0/1729918847 conn(0x7fbecc0c7670 msgr2=0x7fbecc0c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.380+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 shutdown_connections 2024-03-06T07:22:13.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:13.381+0000 7fbed459a640 1 -- 172.21.15.94:0/1729918847 wait complete. 2024-03-06T07:22:13.393 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:05.610686+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_36134": { "addr": "172.21.15.145:6836/4001418202", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 4001418202, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 4001418202, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 36134, "incarnation": 86, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:22:08.543964+0000", "qdb_cluster": [ 36134 ], "qdb_leader": 36134, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 36134 } } } ], "standbys": [ { "addr": "172.21.15.94:6834/2226553926", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 2226553926, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 2226553926, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7350, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3093758100", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3093758100, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3093758100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7353, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/2722633283", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 2722633283, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 2722633283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7371, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:13.394 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 8, 'name': 'a', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:22:13.394 INFO:tasks.cephfs.test_failover:current = [{'gid': 7350, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7371, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/2722633283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2722633283}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2722633283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7353, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3093758100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3093758100}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3093758100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 36134, 'name': 'd', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/4001418202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 4001418202}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 4001418202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:22:13.394 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-03-06T07:22:13.394 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:22:13.501 INFO:tasks.ceph.mds.d.smithi145.stderr: -141> 2024-03-06T07:22:11.533+0000 7f550a843640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:13.525 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:22:14.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:14.608 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:16.533 INFO:tasks.ceph.mds.a.smithi094.stderr:2024-03-06T07:22:16.532+0000 7f1621a01640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:18.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.495+0000 7fed6f59e640 1 Processor -- start 2024-03-06T07:22:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.495+0000 7fed6f59e640 1 -- start start 2024-03-06T07:22:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6f59e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fed700c2ef0 0x7fed701536a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6f59e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed700aa990 0x7fed7006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6f59e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed700ab7d0 con 0x7fed70149710 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6f59e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed700ab210 con 0x7fed700aa990 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6f59e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed700ab4f0 con 0x7fed700c2ef0 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed700aa990 0x7fed7006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed700aa990 0x7fed7006bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44802/0 (socket says 172.21.15.94:44802) 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6ed9d640 1 -- 172.21.15.94:0/776439452 learned_addr learned my addr 172.21.15.94:0/776439452 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6ed9d640 1 -- 172.21.15.94:0/776439452 >> v2:172.21.15.145:3300/0 conn(0x7fed700c2ef0 msgr2=0x7fed701536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6ed9d640 1 --2- 172.21.15.94:0/776439452 >> v2:172.21.15.145:3300/0 conn(0x7fed700c2ef0 0x7fed701536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6ed9d640 1 -- 172.21.15.94:0/776439452 >> v1:172.21.15.145:6789/0 conn(0x7fed70149710 legacy=0x7fed700c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.496+0000 7fed6ed9d640 1 -- 172.21.15.94:0/776439452 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70152ea0 con 0x7fed700aa990 2024-03-06T07:22:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.497+0000 7fed6ed9d640 1 --2- 172.21.15.94:0/776439452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed700aa990 0x7fed7006bfd0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fed700a9df0 tx=0x7fed6802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c7a8ba7910871a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.497+0000 7fed6dd9b640 1 -- 172.21.15.94:0/776439452 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6802c9f0 con 0x7fed700aa990 2024-03-06T07:22:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.497+0000 7fed6dd9b640 1 -- 172.21.15.94:0/776439452 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6802cb90 con 0x7fed700aa990 2024-03-06T07:22:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.497+0000 7fed6dd9b640 1 -- 172.21.15.94:0/776439452 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680343f0 con 0x7fed700aa990 2024-03-06T07:22:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 -- 172.21.15.94:0/776439452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed700aa990 msgr2=0x7fed7006bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 --2- 172.21.15.94:0/776439452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed700aa990 0x7fed7006bfd0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fed700a9df0 tx=0x7fed6802be50 comp rx=0 tx=0).stop 2024-03-06T07:22:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 -- 172.21.15.94:0/776439452 shutdown_connections 2024-03-06T07:22:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 --2- 172.21.15.94:0/776439452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed700aa990 0x7fed7006bfd0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 --2- 172.21.15.94:0/776439452 >> v2:172.21.15.145:3300/0 conn(0x7fed700c2ef0 0x7fed701536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 -- 172.21.15.94:0/776439452 >> 172.21.15.94:0/776439452 conn(0x7fed700c3960 msgr2=0x7fed700c3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 -- 172.21.15.94:0/776439452 shutdown_connections 2024-03-06T07:22:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.498+0000 7fed6f59e640 1 -- 172.21.15.94:0/776439452 wait complete. 2024-03-06T07:22:18.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 Processor -- start 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 -- start start 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700aa990 0x7fed700c05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700c2ef0 0x7fed700c0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed70149710 0x7fed701dc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed70149af0 con 0x7fed70149710 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed70153be0 con 0x7fed700aa990 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.500+0000 7fed6f59e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed70149c70 con 0x7fed700c2ef0 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed70149710 0x7fed701dc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed70149710 0x7fed701dc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44814/0 (socket says 172.21.15.94:44814) 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 -- 172.21.15.94:0/2156412415 learned_addr learned my addr 172.21.15.94:0/2156412415 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed67fff640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700c2ef0 0x7fed700c0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6e59c640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700aa990 0x7fed700c05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 -- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700c2ef0 msgr2=0x7fed700c0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700c2ef0 0x7fed700c0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 -- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700aa990 msgr2=0x7fed700c05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700aa990 0x7fed700c05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 -- 172.21.15.94:0/2156412415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed701466d0 con 0x7fed70149710 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed67fff640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700c2ef0 0x7fed700c0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6e59c640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700aa990 0x7fed700c05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6ed9d640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed70149710 0x7fed701dc3f0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fed68036040 tx=0x7fed68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed66ffd640 1 -- 172.21.15.94:0/2156412415 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68043020 con 0x7fed70149710 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed66ffd640 1 -- 172.21.15.94:0/2156412415 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6803e450 con 0x7fed70149710 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed70154fb0 con 0x7fed70149710 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed66ffd640 1 -- 172.21.15.94:0/2156412415 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6803e820 con 0x7fed70149710 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.501+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701e93e0 con 0x7fed70149710 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.502+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed70069430 con 0x7fed70149710 2024-03-06T07:22:18.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.503+0000 7fed66ffd640 1 -- 172.21.15.94:0/2156412415 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed68046020 con 0x7fed70149710 2024-03-06T07:22:18.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.503+0000 7fed66ffd640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed5006b500 0x7fed5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.503+0000 7fed66ffd640 1 -- 172.21.15.94:0/2156412415 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fed680aca30 con 0x7fed70149710 2024-03-06T07:22:18.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.503+0000 7fed6e59c640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed5006b500 0x7fed5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.504+0000 7fed6e59c640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed5006b500 0x7fed5006d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fed60006fd0 tx=0x7fed60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.512+0000 7fed66ffd640 1 -- 172.21.15.94:0/2156412415 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed6804d070 con 0x7fed70149710 2024-03-06T07:22:18.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.727+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed700c2ad0 con 0x7fed70149710 2024-03-06T07:22:18.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.728+0000 7fed66ffd640 1 -- 172.21.15.94:0/2156412415 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4986 (secure 0 0 0) 0x7fed6807b760 con 0x7fed70149710 2024-03-06T07:22:18.730 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:18.730 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":94,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/2722633283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2722633283},{"type":"v1","addr":"172.21.15.145:6837","nonce":2722633283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:16.505437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:18.730 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 94 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed5006b500 msgr2=0x7fed5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed5006b500 0x7fed5006d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fed60006fd0 tx=0x7fed60008040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed70149710 msgr2=0x7fed701dc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed70149710 0x7fed701dc3f0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fed68036040 tx=0x7fed68004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 shutdown_connections 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed5006b500 0x7fed5006d9c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed70149710 0x7fed701dc3f0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700c2ef0 0x7fed700c0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 --2- 172.21.15.94:0/2156412415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700aa990 0x7fed700c05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 >> 172.21.15.94:0/2156412415 conn(0x7fed700c3960 msgr2=0x7fed70150fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 shutdown_connections 2024-03-06T07:22:18.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:18.730+0000 7fed6f59e640 1 -- 172.21.15.94:0/2156412415 wait complete. 2024-03-06T07:22:19.146 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:05.610686+0000', 'modified': '2024-03-06T07:22:16.505437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 109, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7350}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7350': {'gid': 7350, 'name': 'a', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7350, 'qdb_cluster': [7350]} 2024-03-06T07:22:19.146 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:19.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:22:19.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.251+0000 7fc3aef1e640 1 Processor -- start 2024-03-06T07:22:19.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.251+0000 7fc3aef1e640 1 -- start start 2024-03-06T07:22:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3aef1e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc3a80c8950 0x7fc3a80c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3aef1e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3a81558d0 0x7fc3a8155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3aef1e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a8155130 con 0x7fc3a80c69f0 2024-03-06T07:22:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3aef1e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a8155440 con 0x7fc3a81558d0 2024-03-06T07:22:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3aef1e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80ad480 con 0x7fc3a80c8950 2024-03-06T07:22:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3ad4bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3a81558d0 0x7fc3a8155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3ad4bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3a81558d0 0x7fc3a8155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44828/0 (socket says 172.21.15.94:44828) 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1671158294 learned_addr learned my addr 172.21.15.94:0/1671158294 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/1671158294 >> v2:172.21.15.145:3300/0 conn(0x7fc3a80c8950 0x7fc3a80c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1671158294 >> v2:172.21.15.145:3300/0 conn(0x7fc3a80c8950 msgr2=0x7fc3a80c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/1671158294 >> v2:172.21.15.145:3300/0 conn(0x7fc3a80c8950 0x7fc3a80c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1671158294 >> v1:172.21.15.145:6789/0 conn(0x7fc3a80c69f0 legacy=0x7fc3a80acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.252+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1671158294 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80c5d50 con 0x7fc3a81558d0 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.253+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/1671158294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a8155d00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80c7430 tx=0x7fc3a002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0f2ade69561587 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.253+0000 7fc3a77fe640 1 -- 172.21.15.94:0/1671158294 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a002ca80 con 0x7fc3a81558d0 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.253+0000 7fc3a77fe640 1 -- 172.21.15.94:0/1671158294 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a002cc20 con 0x7fc3a81558d0 2024-03-06T07:22:19.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.253+0000 7fc3a77fe640 1 -- 172.21.15.94:0/1671158294 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00343f0 con 0x7fc3a81558d0 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1671158294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 msgr2=0x7fc3a8155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1671158294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a8155d00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80c7430 tx=0x7fc3a002bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1671158294 shutdown_connections 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1671158294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a8155d00 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1671158294 >> v2:172.21.15.145:3300/0 conn(0x7fc3a80c8950 0x7fc3a80c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1671158294 >> 172.21.15.94:0/1671158294 conn(0x7fc3a80cae70 msgr2=0x7fc3a80cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1671158294 shutdown_connections 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.254+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1671158294 wait complete. 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.255+0000 7fc3aef1e640 1 Processor -- start 2024-03-06T07:22:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.255+0000 7fc3aef1e640 1 -- start start 2024-03-06T07:22:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3aef1e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a80c69f0 0x7fc3a80c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3aef1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a80c8950 0x7fc3a814ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3aef1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a81ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3aef1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8152a20 con 0x7fc3a81558d0 2024-03-06T07:22:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3aef1e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80c6310 con 0x7fc3a80c69f0 2024-03-06T07:22:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3aef1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8152ba0 con 0x7fc3a80c8950 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a81ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a81ee350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44838/0 (socket says 172.21.15.94:44838) 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1429282266 learned_addr learned my addr 172.21.15.94:0/1429282266 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a80c8950 0x7fc3a814ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3accba640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a80c69f0 0x7fc3a80c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a80c8950 msgr2=0x7fc3a814ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a80c8950 0x7fc3a814ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a80c69f0 msgr2=0x7fc3a80c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a80c69f0 0x7fc3a80c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/1429282266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a81587c0 con 0x7fc3a81558d0 2024-03-06T07:22:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.257+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a81ee350 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036040 tx=0x7fc3a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.258+0000 7fc3a57fa640 1 -- 172.21.15.94:0/1429282266 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0043020 con 0x7fc3a81558d0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.258+0000 7fc3a57fa640 1 -- 172.21.15.94:0/1429282266 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a003e450 con 0x7fc3a81558d0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.258+0000 7fc3a57fa640 1 -- 172.21.15.94:0/1429282266 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0009510 con 0x7fc3a81558d0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.258+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a8158aa0 con 0x7fc3a81558d0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.258+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a814f190 con 0x7fc3a81558d0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.259+0000 7fc3a57fa640 1 -- 172.21.15.94:0/1429282266 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3a0046020 con 0x7fc3a81558d0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.259+0000 7fc3a57fa640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc38c06b4d0 0x7fc38c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.259+0000 7fc3a57fa640 1 -- 172.21.15.94:0/1429282266 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc3a00ac990 con 0x7fc3a81558d0 2024-03-06T07:22:19.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.260+0000 7fc3accba640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc38c06b4d0 0x7fc38c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.260+0000 7fc3accba640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc38c06b4d0 0x7fc38c06d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80ac3b0 tx=0x7fc39c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.260+0000 7fc3b1be6740 1 Processor -- start 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.260+0000 7fc3b1be6740 1 -- start start 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfb5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb64b0 0x55f4bcfb2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4bcf1c6b0 con 0x55f4bcfb7f50 2024-03-06T07:22:19.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4bcec5a60 con 0x55f4bcfb8330 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4bcf18750 con 0x55f4bcfb64b0 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfb5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfb5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44840/0 (socket says 172.21.15.94:44840) 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 -- 172.21.15.94:0/1514857997 learned_addr learned my addr 172.21.15.94:0/1514857997 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb64b0 0x55f4bcfb2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 -- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb64b0 msgr2=0x55f4bcfb2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb64b0 0x55f4bcfb2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 -- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 msgr2=0x55f4bcfb5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 -- 172.21.15.94:0/1514857997 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4bcee3f30 con 0x55f4bcfb7f50 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.261+0000 7fc3accba640 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfb5a30 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00f950 tx=0x7fc39c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f62db0508a93d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3927fc640 1 -- 172.21.15.94:0/1514857997 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c025070 con 0x55f4bcfb7f50 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3927fc640 1 -- 172.21.15.94:0/1514857997 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c01b050 con 0x55f4bcfb7f50 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3927fc640 1 -- 172.21.15.94:0/1514857997 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0013a0 con 0x55f4bcfb7f50 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 -- 172.21.15.94:0/1514857997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 msgr2=0x55f4bcfb5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfb5a30 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00f950 tx=0x7fc39c012000 comp rx=0 tx=0).stop 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 -- 172.21.15.94:0/1514857997 shutdown_connections 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb64b0 0x55f4bcfb2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/1514857997 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfb5a30 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 -- 172.21.15.94:0/1514857997 >> 172.21.15.94:0/1514857997 conn(0x55f4bcfb47d0 msgr2=0x55f4bcfd93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 -- 172.21.15.94:0/1514857997 shutdown_connections 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.262+0000 7fc3b1be6740 1 -- 172.21.15.94:0/1514857997 wait complete. 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.263+0000 7fc3b1be6740 1 -- start start 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.263+0000 7fc3b1be6740 1 Processor -- start 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb7f50 0x55f4bcff4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd007f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfde0a0 0x55f4bcff09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4bcfda230 con 0x55f4bcfde0a0 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4bced8ea0 con 0x55f4bcfb8330 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4bcfd9c70 con 0x55f4bcfb7f50 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3b1be6740 10 client.0 _ll_register_callbacks cb 0x55f4bced8870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfde0a0 0x55f4bcff09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfde0a0 0x55f4bcff09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44852/0 (socket says 172.21.15.94:44852) 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/793013628 learned_addr learned my addr 172.21.15.94:0/793013628 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3accba640 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb7f50 0x55f4bcff4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd007f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb7f50 msgr2=0x55f4bcff4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb7f50 0x55f4bcff4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 msgr2=0x55f4bd007f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd007f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 -- 172.21.15.94:0/793013628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4bced9270 con 0x55f4bcfde0a0 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfde0a0 0x55f4bcff09e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fc3a007af80 tx=0x7fc3a007a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846c0de21258fcb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3877fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3877fe640 1 -- 172.21.15.94:0/793013628 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0083650 con 0x55f4bcfde0a0 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.264+0000 7fc3877fe640 1 -- 172.21.15.94:0/793013628 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc3a00831c0 con 0x55f4bcfde0a0 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc39c01b050 con 0x55f4bcfde0a0 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3877fe640 1 -- 172.21.15.94:0/793013628 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3a00833c0 con 0x55f4bcfde0a0 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 10 client.7575 fetch_fsmap learned FSMap version 94 2024-03-06T07:22:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55f4bcc312d0 con 0x55f4bcfde0a0 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3877fe640 1 -- 172.21.15.94:0/793013628 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c01b050 con 0x55f4bcfde0a0 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3877fe640 1 -- 172.21.15.94:0/793013628 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 94) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc3a00810b0 con 0x55f4bcfde0a0 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 10 client.7575 fetch_fsmap finished waiting for FSMap version 94 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 10 client.7575 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 20 client.7575 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bcfda9f0 0x55f4bcfdadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 4 client.7575 mds_command: new command op to 7350 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 --> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4bcf29bf0 con 0x55f4bcfda9f0 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3accba640 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bcfda9f0 0x55f4bcfdadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3accba640 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bcfda9f0 0x55f4bcfdadd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68ba1c238234006e server_cookie=eb0cfcb59715b034 in_seq=0 out_seq=0 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.265+0000 7fc3877fe640 10 client.7575 ms_handle_connect on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3877fe640 1 -- 172.21.15.94:0/793013628 <== mds.0 v2:172.21.15.94:6832/2226553926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f4bcf29bf0 con 0x55f4bcfda9f0 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3877fe640 10 client.7575 handle_command_reply: tid=0 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 client.7575 shutdown 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfde0a0 msgr2=0x55f4bcff09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfde0a0 0x55f4bcff09e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fc3a007af80 tx=0x7fc3a007a950 comp rx=0 tx=0).stop 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 shutdown_connections 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bcfda9f0 0x55f4bcfdadd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfde0a0 0x55f4bcff09e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd007f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/793013628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfb7f50 0x55f4bcff4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3877fe640 0 client.7575 ms_handle_reset on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 >> 172.21.15.94:0/793013628 conn(0x55f4bcfb64b0 msgr2=0x55f4bcfd90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 shutdown_connections 2024-03-06T07:22:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 1 -- 172.21.15.94:0/793013628 wait complete. 2024-03-06T07:22:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.266+0000 7fc3b1be6740 20 client.7575 trim_cache size 0 max 16384 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 Processor -- start 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 -- start start 2024-03-06T07:22:19.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfdadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bcfb5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a81f70c0 con 0x55f4bcfb8330 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8149340 con 0x55f4bcfb7f50 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a81f6ac0 con 0x55f4bcfde0a0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3a7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3a7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44860/0 (socket says 172.21.15.94:44860) 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3a7fff640 1 -- 172.21.15.94:0/2387927979 learned_addr learned my addr 172.21.15.94:0/2387927979 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.289+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bcfb5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3accba640 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfdadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3a7fff640 1 -- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 msgr2=0x55f4bcfb5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bcfb5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3a7fff640 1 -- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb7f50 msgr2=0x55f4bcfdadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfdadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3a7fff640 1 -- 172.21.15.94:0/2387927979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4bcff6720 con 0x55f4bcfb8330 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3accba640 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfdadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb55a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fc394009be0 tx=0x7fc394009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7740dc28a99decd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc391ffb640 1 -- 172.21.15.94:0/2387927979 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394004030 con 0x55f4bcfb8330 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc391ffb640 1 -- 172.21.15.94:0/2387927979 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3940041d0 con 0x55f4bcfb8330 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc391ffb640 1 -- 172.21.15.94:0/2387927979 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394003610 con 0x55f4bcfb8330 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 -- 172.21.15.94:0/2387927979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb8330 msgr2=0x55f4bcfb55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb55a0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fc394009be0 tx=0x7fc394009190 comp rx=0 tx=0).stop 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 -- 172.21.15.94:0/2387927979 shutdown_connections 2024-03-06T07:22:19.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bcfb5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb8330 0x55f4bcfb55a0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/2387927979 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb7f50 0x55f4bcfdadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 -- 172.21.15.94:0/2387927979 >> 172.21.15.94:0/2387927979 conn(0x55f4bcfb64b0 msgr2=0x55f4bd001c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 -- 172.21.15.94:0/2387927979 shutdown_connections 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.290+0000 7fc3b1be6740 1 -- 172.21.15.94:0/2387927979 wait complete. 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.291+0000 7fc3b1be6740 1 -- start start 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.291+0000 7fc3b1be6740 1 Processor -- start 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bd0664e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd008da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3b1be6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bd0092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4bcfda720 con 0x55f4bcfb7f50 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4bcfda230 con 0x55f4bcfb8330 2024-03-06T07:22:19.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3b1be6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4bd0049a0 con 0x55f4bcfde0a0 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3b1be6740 10 client.0 _ll_register_callbacks cb 0x7fc3a007adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bd0664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bd0664e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44870/0 (socket says 172.21.15.94:44870) 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 -- 172.21.15.94:0/3520092911 learned_addr learned my addr 172.21.15.94:0/3520092911 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3ad4bb640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bd0092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd008da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 -- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 msgr2=0x55f4bd0092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bd0092e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 -- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 msgr2=0x55f4bd008da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd008da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 -- 172.21.15.94:0/3520092911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4bccfb6b0 con 0x55f4bcfb7f50 2024-03-06T07:22:19.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd008da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3accba640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bd0664e0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00f5a0 tx=0x7fc39c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbbb3bdfb3de6b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.292+0000 7fc3877fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3877fe640 1 -- 172.21.15.94:0/3520092911 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c034020 con 0x55f4bcfb7f50 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3877fe640 1 -- 172.21.15.94:0/3520092911 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc39c030100 con 0x55f4bcfb7f50 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3940041d0 con 0x55f4bcfb7f50 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3877fe640 1 -- 172.21.15.94:0/3520092911 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c03e050 con 0x55f4bcfb7f50 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 10 client.7581 fetch_fsmap learned FSMap version 94 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x55f4bcfb9260 con 0x55f4bcfb7f50 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3877fe640 1 -- 172.21.15.94:0/3520092911 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3940041d0 con 0x55f4bcfb7f50 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3877fe640 1 -- 172.21.15.94:0/3520092911 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 94) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc39c041020 con 0x55f4bcfb7f50 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 10 client.7581 fetch_fsmap finished waiting for FSMap version 94 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 10 client.7581 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:19.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 20 client.7581 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bd061e30 0x55f4bd064690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 4 client.7581 mds_command: new command op to 7350 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 --> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f4bcf23f40 con 0x55f4bd061e30 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bd061e30 0x55f4bd064690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3877fe640 10 client.7581 ms_handle_connect on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.293+0000 7fc3a7fff640 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bd061e30 0x55f4bd064690 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8af377afc0a1c14d server_cookie=3317e6087f9f8faa in_seq=0 out_seq=0 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3877fe640 1 -- 172.21.15.94:0/3520092911 <== mds.0 v2:172.21.15.94:6832/2226553926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55f4bcf23f40 con 0x55f4bd061e30 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3877fe640 10 client.7581 handle_command_reply: tid=0 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 client.7581 shutdown 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 msgr2=0x55f4bd0664e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bd0664e0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00f5a0 tx=0x7fc39c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 shutdown_connections 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55f4bd061e30 0x55f4bd064690 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f4bcfde0a0 0x55f4bd0092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3877fe640 0 client.7581 ms_handle_reset on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f4bcfb8330 0x55f4bd008da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 --2- 172.21.15.94:0/3520092911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f4bcfb7f50 0x55f4bd0664e0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 >> 172.21.15.94:0/3520092911 conn(0x55f4bcfb64b0 msgr2=0x55f4bcfb55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 shutdown_connections 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 1 -- 172.21.15.94:0/3520092911 wait complete. 2024-03-06T07:22:19.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3b1be6740 20 client.7581 trim_cache size 0 max 16384 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7350, 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 5.7989350760000002, 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 94, 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 109, 2024-03-06T07:22:19.306 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 109, 2024-03-06T07:22:19.307 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 18.144820511999999 2024-03-06T07:22:19.307 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:19.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc38c06b4d0 msgr2=0x7fc38c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc38c06b4d0 0x7fc38c06d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80ac3b0 tx=0x7fc39c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:19.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 msgr2=0x7fc3a81ee350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.294+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a81ee350 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036040 tx=0x7fc3a0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 shutdown_connections 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc38c06b4d0 0x7fc38c06d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3a81558d0 0x7fc3a81ee350 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a80c8950 0x7fc3a814ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 --2- 172.21.15.94:0/1429282266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a80c69f0 0x7fc3a80c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 >> 172.21.15.94:0/1429282266 conn(0x7fc3a80cae70 msgr2=0x7fc3a81517b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 shutdown_connections 2024-03-06T07:22:19.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.295+0000 7fc3aef1e640 1 -- 172.21.15.94:0/1429282266 wait complete. 2024-03-06T07:22:19.310 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:05.610686+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7350": { "addr": "172.21.15.94:6834/2226553926", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 2226553926, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 2226553926, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7350, "incarnation": 91, "join_fscid": 8, "name": "a", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 109, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:22:16.505437+0000", "qdb_cluster": [ 7350 ], "qdb_leader": 7350, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7350 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/3485950735", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 3485950735, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 3485950735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 92, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7144, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3093758100", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3093758100, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3093758100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7353, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/2722633283", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 2722633283, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 2722633283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7371, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:19.311 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 8, 'name': 'a', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:22:19.311 INFO:tasks.cephfs.test_failover:current = [{'gid': 7350, 'name': 'a', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7371, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/2722633283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2722633283}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2722633283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7353, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3093758100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3093758100}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3093758100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7144, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/3485950735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3485950735}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3485950735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 92}] 2024-03-06T07:22:19.311 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:22:19.311 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-06T07:22:19.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.414+0000 7fda9d565640 1 Processor -- start 2024-03-06T07:22:19.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.414+0000 7fda9d565640 1 -- start start 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda9d565640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fda980a8290 0x7fda981539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda9d565640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fda980c4740 0x7fda980701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda9d565640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 0x7fda98072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda9d565640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda98153f00 con 0x7fda980a8290 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda9d565640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda980c62a0 con 0x7fda980c4740 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda9d565640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980706e0 con 0x7fda980c69a0 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fda980a8290 0x7fda981539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fda980a8290 0x7fda981539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44880/0 (socket says 172.21.15.94:44880) 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 -- 172.21.15.94:0/2873026452 learned_addr learned my addr 172.21.15.94:0/2873026452 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda977fe640 1 --2- 172.21.15.94:0/2873026452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 0x7fda98072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda8ffff640 1 --2- 172.21.15.94:0/2873026452 >> v2:172.21.15.145:3300/0 conn(0x7fda980c4740 0x7fda980701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 -- 172.21.15.94:0/2873026452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 msgr2=0x7fda98072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 --2- 172.21.15.94:0/2873026452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 0x7fda98072680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 -- 172.21.15.94:0/2873026452 >> v2:172.21.15.145:3300/0 conn(0x7fda980c4740 msgr2=0x7fda980701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 --2- 172.21.15.94:0/2873026452 >> v2:172.21.15.145:3300/0 conn(0x7fda980c4740 0x7fda980701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.415+0000 7fda96ffd640 1 -- 172.21.15.94:0/2873026452 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980c4c00 con 0x7fda980a8290 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.416+0000 7fda96ffd640 1 --2- 172.21.15.94:0/2873026452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda981539c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fda980ab750 tx=0x7fda8802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3318721a477542df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.416+0000 7fda967fc640 1 -- 172.21.15.94:0/2873026452 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88038070 con 0x7fda980a8290 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.416+0000 7fda967fc640 1 -- 172.21.15.94:0/2873026452 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda88031740 con 0x7fda980a8290 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.416+0000 7fda967fc640 1 -- 172.21.15.94:0/2873026452 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8802f050 con 0x7fda980a8290 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 -- 172.21.15.94:0/2873026452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 msgr2=0x7fda981539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 --2- 172.21.15.94:0/2873026452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda981539c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fda980ab750 tx=0x7fda8802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 -- 172.21.15.94:0/2873026452 shutdown_connections 2024-03-06T07:22:19.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 --2- 172.21.15.94:0/2873026452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 0x7fda98072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 --2- 172.21.15.94:0/2873026452 >> v2:172.21.15.145:3300/0 conn(0x7fda980c4740 0x7fda980701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 --2- 172.21.15.94:0/2873026452 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda981539c0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 -- 172.21.15.94:0/2873026452 >> 172.21.15.94:0/2873026452 conn(0x7fda980c5560 msgr2=0x7fda980c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 -- 172.21.15.94:0/2873026452 shutdown_connections 2024-03-06T07:22:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.417+0000 7fda9d565640 1 -- 172.21.15.94:0/2873026452 wait complete. 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.419+0000 7fda9d565640 1 Processor -- start 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.419+0000 7fda9d565640 1 -- start start 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda9d565640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda980c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda9d565640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda980c4740 0x7fda9814f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda9d565640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 0x7fda9814fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda9d565640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9806fc80 con 0x7fda980a8290 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda9d565640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9806f9a0 con 0x7fda980c4740 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda9d565640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980c6df0 con 0x7fda980c69a0 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda980c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda980c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44882/0 (socket says 172.21.15.94:44882) 2024-03-06T07:22:19.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 -- 172.21.15.94:0/917195683 learned_addr learned my addr 172.21.15.94:0/917195683 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 -- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 msgr2=0x7fda9814fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 0x7fda9814fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 -- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda980c4740 msgr2=0x7fda9814f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda980c4740 0x7fda9814f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 -- 172.21.15.94:0/917195683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980abaf0 con 0x7fda980a8290 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda96ffd640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda980c2100 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fda8802b820 tx=0x7fda880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda8f7fe640 1 -- 172.21.15.94:0/917195683 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88042020 con 0x7fda980a8290 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda8f7fe640 1 -- 172.21.15.94:0/917195683 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda880338c0 con 0x7fda980a8290 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda980ae5a0 con 0x7fda980a8290 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.420+0000 7fda8f7fe640 1 -- 172.21.15.94:0/917195683 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88038040 con 0x7fda980a8290 2024-03-06T07:22:19.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.421+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda980c2640 con 0x7fda980a8290 2024-03-06T07:22:19.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.422+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda98150d30 con 0x7fda980a8290 2024-03-06T07:22:19.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.422+0000 7fda8f7fe640 1 -- 172.21.15.94:0/917195683 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda88045020 con 0x7fda980a8290 2024-03-06T07:22:19.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.422+0000 7fda8f7fe640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda7806b500 0x7fda7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.423+0000 7fda8f7fe640 1 -- 172.21.15.94:0/917195683 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fda880abf30 con 0x7fda980a8290 2024-03-06T07:22:19.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.423+0000 7fda8ffff640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda7806b500 0x7fda7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.423+0000 7fda8ffff640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda7806b500 0x7fda7806d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fda9806e2b0 tx=0x7fda80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.431+0000 7fda8f7fe640 1 -- 172.21.15.94:0/917195683 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda8807ac60 con 0x7fda980a8290 2024-03-06T07:22:19.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.646+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fda9804f330 con 0x7fda980a8290 2024-03-06T07:22:19.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.780+0000 7fda8f7fe640 1 -- 172.21.15.94:0/917195683 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v95) v1 ==== 121+0+0 (secure 0 0 0) 0x7fda88046e40 con 0x7fda980a8290 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda7806b500 msgr2=0x7fda7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda7806b500 0x7fda7806d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fda9806e2b0 tx=0x7fda80008040 comp rx=0 tx=0).stop 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 msgr2=0x7fda980c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda980c2100 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fda8802b820 tx=0x7fda880046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 shutdown_connections 2024-03-06T07:22:19.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fda7806b500 0x7fda7806d9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda980c69a0 0x7fda9814fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda980c4740 0x7fda9814f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 --2- 172.21.15.94:0/917195683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fda980a8290 0x7fda980c2100 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 >> 172.21.15.94:0/917195683 conn(0x7fda980c5560 msgr2=0x7fda980c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 shutdown_connections 2024-03-06T07:22:19.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.782+0000 7fda9d565640 1 -- 172.21.15.94:0/917195683 wait complete. 2024-03-06T07:22:19.796 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:19.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.899+0000 7fb488057640 1 Processor -- start 2024-03-06T07:22:19.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.899+0000 7fb488057640 1 -- start start 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.899+0000 7fb488057640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb4801529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.899+0000 7fb488057640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb480155a40 0x7fb480155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.899+0000 7fb488057640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb480156410 con 0x7fb48014f790 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.899+0000 7fb488057640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4800c2680 con 0x7fb480155a40 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb488057640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800c28d0 con 0x7fb4800c1aa0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb485df3640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb48014f790 0x7fb4800c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37048/0 (socket says 172.21.15.94:37048) 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb485df3640 1 -- 172.21.15.94:0/3180108049 learned_addr learned my addr 172.21.15.94:0/3180108049 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb4865f4640 1 --2- 172.21.15.94:0/3180108049 >> v2:172.21.15.145:3300/0 conn(0x7fb480155a40 0x7fb480155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb4855f2640 1 --2- 172.21.15.94:0/3180108049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb4801529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb484df1640 1 -- 172.21.15.94:0/3180108049 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814753166 0 0) 0x7fb480156410 con 0x7fb48014f790 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb484df1640 1 -- 172.21.15.94:0/3180108049 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46c003520 con 0x7fb48014f790 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb484df1640 1 -- 172.21.15.94:0/3180108049 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3806102646 0 0) 0x7fb46c003520 con 0x7fb48014f790 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb4855f2640 1 -- 172.21.15.94:0/3180108049 >> v2:172.21.15.145:3300/0 conn(0x7fb480155a40 msgr2=0x7fb480155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb4855f2640 1 --2- 172.21.15.94:0/3180108049 >> v2:172.21.15.145:3300/0 conn(0x7fb480155a40 0x7fb480155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb4855f2640 1 -- 172.21.15.94:0/3180108049 >> v1:172.21.15.94:6789/0 conn(0x7fb48014f790 legacy=0x7fb4800c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.900+0000 7fb4855f2640 1 -- 172.21.15.94:0/3180108049 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4800c2bb0 con 0x7fb4800c1aa0 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.901+0000 7fb4855f2640 1 --2- 172.21.15.94:0/3180108049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb4801529d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb4700097a0 tx=0x7fb47002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f502fd4f5a92a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.901+0000 7fb484df1640 1 -- 172.21.15.94:0/3180108049 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470037070 con 0x7fb4800c1aa0 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.901+0000 7fb484df1640 1 -- 172.21.15.94:0/3180108049 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb470030510 con 0x7fb4800c1aa0 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.901+0000 7fb484df1640 1 -- 172.21.15.94:0/3180108049 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c001670 con 0x7fb4800c1aa0 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 -- 172.21.15.94:0/3180108049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 msgr2=0x7fb4801529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 --2- 172.21.15.94:0/3180108049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb4801529d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb4700097a0 tx=0x7fb47002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:19.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 -- 172.21.15.94:0/3180108049 shutdown_connections 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 --2- 172.21.15.94:0/3180108049 >> v2:172.21.15.145:3300/0 conn(0x7fb480155a40 0x7fb480155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 --2- 172.21.15.94:0/3180108049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb4801529d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 -- 172.21.15.94:0/3180108049 >> 172.21.15.94:0/3180108049 conn(0x7fb4800c6710 msgr2=0x7fb4800c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 -- 172.21.15.94:0/3180108049 shutdown_connections 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.902+0000 7fb488057640 1 -- 172.21.15.94:0/3180108049 wait complete. 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.903+0000 7fb488057640 1 Processor -- start 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.903+0000 7fb488057640 1 -- start start 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb488057640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb480148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb488057640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb48014f790 0x7fb4801ddb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb488057640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb480155a40 0x7fb48006d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb488057640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800c0d60 con 0x7fb48014f790 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb488057640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb480155520 con 0x7fb480155a40 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb488057640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800c5e30 con 0x7fb4800c1aa0 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb48014f790 0x7fb4801ddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb48014f790 0x7fb4801ddb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44898/0 (socket says 172.21.15.94:44898) 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 -- 172.21.15.94:0/1911213974 learned_addr learned my addr 172.21.15.94:0/1911213974 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 -- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 msgr2=0x7fb480148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4865f4640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb480155a40 0x7fb48006d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb480148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 -- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb480155a40 msgr2=0x7fb48006d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb480155a40 0x7fb48006d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 -- 172.21.15.94:0/1911213974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4801dbc60 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4855f2640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb48014f790 0x7fb4801ddb00 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fb470000c00 tx=0x7fb4700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4767fc640 1 -- 172.21.15.94:0/1911213974 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470041020 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4767fc640 1 -- 172.21.15.94:0/1911213974 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47003c870 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4801dbf40 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.904+0000 7fb4767fc640 1 -- 172.21.15.94:0/1911213974 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470037040 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.905+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4801dc500 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.905+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48004f330 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.906+0000 7fb4767fc640 1 -- 172.21.15.94:0/1911213974 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb470044020 con 0x7fb48014f790 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.906+0000 7fb4767fc640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b4d0 0x7fb46406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.906+0000 7fb4767fc640 1 -- 172.21.15.94:0/1911213974 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb4700abd30 con 0x7fb48014f790 2024-03-06T07:22:19.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.907+0000 7fb485df3640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b4d0 0x7fb46406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.907+0000 7fb485df3640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b4d0 0x7fb46406d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb47c001eb0 tx=0x7fb47c001a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:19.911+0000 7fb4767fc640 1 -- 172.21.15.94:0/1911213974 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb47004b030 con 0x7fb48014f790 2024-03-06T07:22:20.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.119+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4800a7680 con 0x7fb48014f790 2024-03-06T07:22:20.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.120+0000 7fb4767fc640 1 -- 172.21.15.94:0/1911213974 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4993 (secure 0 0 0) 0x7fb47003ae20 con 0x7fb48014f790 2024-03-06T07:22:20.122 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:20.122 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":96,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:19.783264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7371":{"gid":7371,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6837/2722633283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2722633283},{"type":"v1","addr":"172.21.15.145:6837","nonce":2722633283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:20.122 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 96 2024-03-06T07:22:20.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b4d0 msgr2=0x7fb46406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b4d0 0x7fb46406d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb47c001eb0 tx=0x7fb47c001a40 comp rx=0 tx=0).stop 2024-03-06T07:22:20.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb48014f790 msgr2=0x7fb4801ddb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb48014f790 0x7fb4801ddb00 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fb470000c00 tx=0x7fb4700046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 shutdown_connections 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb46406b4d0 0x7fb46406d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb480155a40 0x7fb48006d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb48014f790 0x7fb4801ddb00 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 --2- 172.21.15.94:0/1911213974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4800c1aa0 0x7fb480148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 >> 172.21.15.94:0/1911213974 conn(0x7fb4800c6710 msgr2=0x7fb480153940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 shutdown_connections 2024-03-06T07:22:20.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.121+0000 7fb488057640 1 -- 172.21.15.94:0/1911213974 wait complete. 2024-03-06T07:22:20.134 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-03-06T07:22:20.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.235+0000 7f8eda46d640 1 Processor -- start 2024-03-06T07:22:20.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.235+0000 7f8eda46d640 1 -- start start 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.236+0000 7f8eda46d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8ed41505f0 0x7f8ed4156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.236+0000 7f8eda46d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed40c6710 0x7f8ed40c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.236+0000 7f8eda46d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed4073580 con 0x7f8ed40c7f60 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.236+0000 7f8eda46d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed4070b20 con 0x7f8ed41505f0 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.236+0000 7f8eda46d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40701f0 con 0x7f8ed40c6710 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed8a0a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed40c6710 0x7f8ed40c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed3fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8ed41505f0 0x7f8ed4156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed8a0a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed40c6710 0x7f8ed40c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53374/0 (socket says 172.21.15.94:53374) 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed3fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8ed41505f0 0x7f8ed4156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:43382/0 (socket says 172.21.15.94:43382) 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed8a0a640 1 -- 172.21.15.94:0/2819956791 learned_addr learned my addr 172.21.15.94:0/2819956791 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed8a0a640 1 -- 172.21.15.94:0/2819956791 >> v2:172.21.15.145:3300/0 conn(0x7f8ed41505f0 msgr2=0x7f8ed4156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed8a0a640 1 --2- 172.21.15.94:0/2819956791 >> v2:172.21.15.145:3300/0 conn(0x7f8ed41505f0 0x7f8ed4156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed8a0a640 1 -- 172.21.15.94:0/2819956791 >> v1:172.21.15.145:6789/0 conn(0x7f8ed40c7f60 legacy=0x7f8ed4072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed8a0a640 1 -- 172.21.15.94:0/2819956791 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed40704d0 con 0x7f8ed40c6710 2024-03-06T07:22:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.237+0000 7f8ed3fff640 1 --2- 172.21.15.94:0/2819956791 >> v2:172.21.15.145:3300/0 conn(0x7f8ed41505f0 0x7f8ed4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8ed8a0a640 1 --2- 172.21.15.94:0/2819956791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed40c6710 0x7f8ed40c6ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009e50 tx=0x7f8ecc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9691275a4d1b15a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8ed2ffd640 1 -- 172.21.15.94:0/2819956791 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc02cb00 con 0x7f8ed40c6710 2024-03-06T07:22:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8ed2ffd640 1 -- 172.21.15.94:0/2819956791 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc02cca0 con 0x7f8ed40c6710 2024-03-06T07:22:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8ed2ffd640 1 -- 172.21.15.94:0/2819956791 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc034610 con 0x7f8ed40c6710 2024-03-06T07:22:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8eda46d640 1 -- 172.21.15.94:0/2819956791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed40c6710 msgr2=0x7f8ed40c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8eda46d640 1 --2- 172.21.15.94:0/2819956791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed40c6710 0x7f8ed40c6ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009e50 tx=0x7f8ecc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8eda46d640 1 -- 172.21.15.94:0/2819956791 shutdown_connections 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8eda46d640 1 --2- 172.21.15.94:0/2819956791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed40c6710 0x7f8ed40c6ba0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8eda46d640 1 --2- 172.21.15.94:0/2819956791 >> v2:172.21.15.145:3300/0 conn(0x7f8ed41505f0 0x7f8ed4156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.238+0000 7f8eda46d640 1 -- 172.21.15.94:0/2819956791 >> 172.21.15.94:0/2819956791 conn(0x7f8ed40aa5c0 msgr2=0x7f8ed40aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.239+0000 7f8eda46d640 1 -- 172.21.15.94:0/2819956791 shutdown_connections 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.239+0000 7f8eda46d640 1 -- 172.21.15.94:0/2819956791 wait complete. 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 Processor -- start 2024-03-06T07:22:20.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 -- start start 2024-03-06T07:22:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed40c6710 0x7f8ed4152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed40c7f60 0x7f8ed414b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed41505f0 0x7f8ed414bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4072c20 con 0x7f8ed40c6710 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed406fec0 con 0x7f8ed40c7f60 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8eda46d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4072da0 con 0x7f8ed41505f0 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.240+0000 7f8ed3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed40c6710 0x7f8ed4152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed40c6710 0x7f8ed4152f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44900/0 (socket says 172.21.15.94:44900) 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 -- 172.21.15.94:0/179473271 learned_addr learned my addr 172.21.15.94:0/179473271 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed8a0a640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed41505f0 0x7f8ed414bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed37fe640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed40c7f60 0x7f8ed414b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 -- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed41505f0 msgr2=0x7f8ed414bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed41505f0 0x7f8ed414bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 -- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed40c7f60 msgr2=0x7f8ed414b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed40c7f60 0x7f8ed414b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 -- 172.21.15.94:0/179473271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed41527b0 con 0x7f8ed40c6710 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed3fff640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed40c6710 0x7f8ed4152f50 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00e990 tx=0x7f8ebc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed0ff9640 1 -- 172.21.15.94:0/179473271 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc017070 con 0x7f8ed40c6710 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed0ff9640 1 -- 172.21.15.94:0/179473271 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc00c9e0 con 0x7f8ed40c6710 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8ed0ff9640 1 -- 172.21.15.94:0/179473271 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00cdb0 con 0x7f8ed40c6710 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed4154270 con 0x7f8ed40c6710 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.241+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed4152a30 con 0x7f8ed40c6710 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.242+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed404f330 con 0x7f8ed40c6710 2024-03-06T07:22:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.243+0000 7f8ed0ff9640 1 -- 172.21.15.94:0/179473271 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ebc01a070 con 0x7f8ed40c6710 2024-03-06T07:22:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.243+0000 7f8ed0ff9640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8eb406b4d0 0x7f8eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.243+0000 7f8ed0ff9640 1 -- 172.21.15.94:0/179473271 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8ebc08a1d0 con 0x7f8ed40c6710 2024-03-06T07:22:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.244+0000 7f8ed37fe640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8eb406b4d0 0x7f8eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.244+0000 7f8ed37fe640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8eb406b4d0 0x7f8eb406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4007fd0 tx=0x7f8ec4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.248+0000 7f8ed0ff9640 1 -- 172.21.15.94:0/179473271 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ed41527b0 con 0x7f8ed40c6710 2024-03-06T07:22:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.425+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f8ed40a87d0 con 0x7f8ed40c6710 2024-03-06T07:22:20.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.428+0000 7f8ed0ff9640 1 -- 172.21.15.94:0/179473271 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v37)=0 v37) v1 ==== 116+0+0 (secure 0 0 0) 0x7f8ebc058f00 con 0x7f8ed40c6710 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8eb406b4d0 msgr2=0x7f8eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8eb406b4d0 0x7f8eb406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4007fd0 tx=0x7f8ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed40c6710 msgr2=0x7f8ed4152f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed40c6710 0x7f8ed4152f50 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00e990 tx=0x7f8ebc00ee50 comp rx=0 tx=0).stop 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 shutdown_connections 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8eb406b4d0 0x7f8eb406d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed41505f0 0x7f8ed414bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed40c7f60 0x7f8ed414b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 --2- 172.21.15.94:0/179473271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed40c6710 0x7f8ed4152f50 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 >> 172.21.15.94:0/179473271 conn(0x7f8ed40aa5c0 msgr2=0x7f8ed40ae140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 shutdown_connections 2024-03-06T07:22:20.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.431+0000 7f8eda46d640 1 -- 172.21.15.94:0/179473271 wait complete. 2024-03-06T07:22:20.444 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.542+0000 7f18f955e640 1 Processor -- start 2024-03-06T07:22:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.543+0000 7f18f955e640 1 -- start start 2024-03-06T07:22:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f955e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18f4152920 0x7f18f4155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f955e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18f40c2190 0x7f18f40c2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f955e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f40c1280 0x7f18f40c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f955e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f4156160 con 0x7f18f40c2190 2024-03-06T07:22:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f955e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f406afc0 con 0x7f18f4152920 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f955e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f406b2a0 con 0x7f18f40c1280 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18f40c2190 0x7f18f40c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f18f40c2190 0x7f18f40c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44902/0 (socket says 172.21.15.94:44902) 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 -- 172.21.15.94:0/746090044 learned_addr learned my addr 172.21.15.94:0/746090044 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f37fe640 1 --2- 172.21.15.94:0/746090044 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f40c1280 0x7f18f40c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18f2ffd640 1 --2- 172.21.15.94:0/746090044 >> v2:172.21.15.145:3300/0 conn(0x7f18f4152920 0x7f18f4155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 -- 172.21.15.94:0/746090044 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f40c1280 msgr2=0x7f18f40c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 --2- 172.21.15.94:0/746090044 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f40c1280 0x7f18f40c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 -- 172.21.15.94:0/746090044 >> v2:172.21.15.145:3300/0 conn(0x7f18f4152920 msgr2=0x7f18f4155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 --2- 172.21.15.94:0/746090044 >> v2:172.21.15.145:3300/0 conn(0x7f18f4152920 0x7f18f4155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.544+0000 7f18ebfff640 1 -- 172.21.15.94:0/746090044 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f406b580 con 0x7f18f40c2190 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.545+0000 7f18ebfff640 1 --2- 172.21.15.94:0/746090044 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c2190 0x7f18f40c2620 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009870 tx=0x7f18dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7506947db1e526b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.545+0000 7f18f27fc640 1 -- 172.21.15.94:0/746090044 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc035020 con 0x7f18f40c2190 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.545+0000 7f18f27fc640 1 -- 172.21.15.94:0/746090044 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc031640 con 0x7f18f40c2190 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.545+0000 7f18f27fc640 1 -- 172.21.15.94:0/746090044 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc02d040 con 0x7f18f40c2190 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 -- 172.21.15.94:0/746090044 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c2190 msgr2=0x7f18f40c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 --2- 172.21.15.94:0/746090044 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c2190 0x7f18f40c2620 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009870 tx=0x7f18dc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 -- 172.21.15.94:0/746090044 shutdown_connections 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 --2- 172.21.15.94:0/746090044 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f40c1280 0x7f18f40c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 --2- 172.21.15.94:0/746090044 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c2190 0x7f18f40c2620 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 --2- 172.21.15.94:0/746090044 >> v2:172.21.15.145:3300/0 conn(0x7f18f4152920 0x7f18f4155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 -- 172.21.15.94:0/746090044 >> 172.21.15.94:0/746090044 conn(0x7f18f40c4290 msgr2=0x7f18f40c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 -- 172.21.15.94:0/746090044 shutdown_connections 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.546+0000 7f18f955e640 1 -- 172.21.15.94:0/746090044 wait complete. 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.547+0000 7f18f955e640 1 Processor -- start 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.547+0000 7f18f955e640 1 -- start start 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.547+0000 7f18f955e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c1280 0x7f18f4152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.547+0000 7f18f955e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f40c2190 0x7f18f4156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.547+0000 7f18f955e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f4152920 0x7f18f4158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.547+0000 7f18f955e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40c1c50 con 0x7f18f40c1280 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.547+0000 7f18f955e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40c0d60 con 0x7f18f40c2190 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f955e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40c1dd0 con 0x7f18f4152920 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c1280 0x7f18f4152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c1280 0x7f18f4152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44910/0 (socket says 172.21.15.94:44910) 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 -- 172.21.15.94:0/3231215593 learned_addr learned my addr 172.21.15.94:0/3231215593 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f37fe640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f4152920 0x7f18f4158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18ebfff640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f40c2190 0x7f18f4156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 -- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f4152920 msgr2=0x7f18f4158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f4152920 0x7f18f4158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 -- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f40c2190 msgr2=0x7f18f4156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f40c2190 0x7f18f4156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 -- 172.21.15.94:0/3231215593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f41461c0 con 0x7f18f40c1280 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18ebfff640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f40c2190 0x7f18f4156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f2ffd640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c1280 0x7f18f4152d00 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f18e400d830 tx=0x7f18e400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18eb7fe640 1 -- 172.21.15.94:0/3231215593 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400b6c0 con 0x7f18f40c1280 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18eb7fe640 1 -- 172.21.15.94:0/3231215593 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4019070 con 0x7f18f40c1280 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f4145d00 con 0x7f18f40c1280 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18eb7fe640 1 -- 172.21.15.94:0/3231215593 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400ba80 con 0x7f18f40c1280 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.548+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f406f430 con 0x7f18f40c1280 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.549+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f404f330 con 0x7f18f40c1280 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.550+0000 7f18eb7fe640 1 -- 172.21.15.94:0/3231215593 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18e401e460 con 0x7f18f40c1280 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.550+0000 7f18eb7fe640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18d406b500 0x7f18d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.550+0000 7f18eb7fe640 1 -- 172.21.15.94:0/3231215593 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f18e408aa20 con 0x7f18f40c1280 2024-03-06T07:22:20.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.550+0000 7f18ebfff640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18d406b500 0x7f18d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.551+0000 7f18ebfff640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18d406b500 0x7f18d406d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f18dc036000 tx=0x7f18dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.556+0000 7f18eb7fe640 1 -- 172.21.15.94:0/3231215593 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18e40596d0 con 0x7f18f40c1280 2024-03-06T07:22:20.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.771+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18f4068460 con 0x7f18f40c1280 2024-03-06T07:22:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.772+0000 7f18eb7fe640 1 -- 172.21.15.94:0/3231215593 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+4993 (secure 0 0 0) 0x7f18e4058fd0 con 0x7f18f40c1280 2024-03-06T07:22:20.774 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:20.774 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":96,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":96,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:19.783264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7371":{"gid":7371,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.145:6837/2722633283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2722633283},{"type":"v1","addr":"172.21.15.145:6837","nonce":2722633283}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:20.774 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 96 2024-03-06T07:22:20.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18d406b500 msgr2=0x7f18d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18d406b500 0x7f18d406d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f18dc036000 tx=0x7f18dc008030 comp rx=0 tx=0).stop 2024-03-06T07:22:20.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c1280 msgr2=0x7f18f4152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c1280 0x7f18f4152d00 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f18e400d830 tx=0x7f18e400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 shutdown_connections 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f18d406b500 0x7f18d406d9c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18f4152920 0x7f18f4158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18f40c2190 0x7f18f4156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 --2- 172.21.15.94:0/3231215593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18f40c1280 0x7f18f4152d00 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 >> 172.21.15.94:0/3231215593 conn(0x7f18f40c4290 msgr2=0x7f18f4150010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 shutdown_connections 2024-03-06T07:22:20.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:20.774+0000 7f18f955e640 1 -- 172.21.15.94:0/3231215593 wait complete. 2024-03-06T07:22:21.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-06T07:22:05.610686+0000', 'modified': '2024-03-06T07:22:19.783264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 109, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7350}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7350': {'gid': 7350, 'name': 'a', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7371': {'gid': 7371, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.145:6837/2722633283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2722633283}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2722633283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7350, 'qdb_cluster': [7350]} 2024-03-06T07:22:21.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:21.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:22:21.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.321+0000 7f90b953e640 1 Processor -- start 2024-03-06T07:22:21.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.321+0000 7f90b953e640 1 -- start start 2024-03-06T07:22:21.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b953e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f90b406f550 0x7f90b41593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b953e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40733f0 con 0x7f90b4152720 2024-03-06T07:22:21.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b953e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b4155ef0 con 0x7f90b4072cd0 2024-03-06T07:22:21.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b953e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b4155ca0 con 0x7f90b406f550 2024-03-06T07:22:21.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f90b4152720 0x7f90b40c77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37058/0 (socket says 172.21.15.94:37058) 2024-03-06T07:22:21.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b2ffd640 1 -- 172.21.15.94:0/3775955445 learned_addr learned my addr 172.21.15.94:0/3775955445 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:21.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660230408 0 0) 0x7f90b40733f0 con 0x7f90b4152720 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0003520 con 0x7f90b4152720 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90abfff640 1 --2- 172.21.15.94:0/3775955445 >> v2:172.21.15.145:3300/0 conn(0x7f90b406f550 0x7f90b41593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1311715683 0 0) 0x7f90a0003520 con 0x7f90b4152720 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 >> v2:172.21.15.145:3300/0 conn(0x7f90b406f550 msgr2=0x7f90b41593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 --2- 172.21.15.94:0/3775955445 >> v2:172.21.15.145:3300/0 conn(0x7f90b406f550 0x7f90b41593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 >> v1:172.21.15.145:6789/0 conn(0x7f90b4072cd0 legacy=0x7f90b406fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b41588e0 con 0x7f90b4152720 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f90a4001670 con 0x7f90b4152720 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 >> v1:172.21.15.94:6789/0 conn(0x7f90b4152720 legacy=0x7f90b40c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00039b0 0x7f90a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a0004800 0x7f90a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90a00052e0 0x7f90a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4155ca0 con 0x7f90a00052e0 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a00149c0 con 0x7f90a0004800 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.322+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0014bc0 con 0x7f90a00039b0 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00039b0 0x7f90a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90b37fe640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90a00052e0 0x7f90a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90abfff640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a0004800 0x7f90a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90b37fe640 1 -- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00039b0 msgr2=0x7f90a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90b37fe640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00039b0 0x7f90a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90b37fe640 1 -- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a0004800 msgr2=0x7f90a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90b37fe640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a0004800 0x7f90a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.323+0000 7f90b37fe640 1 -- 172.21.15.94:0/3775955445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a0003520 con 0x7f90a00052e0 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.324+0000 7f90b37fe640 1 -- 172.21.15.94:0/3775955445 reap_dead start 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.324+0000 7f90b37fe640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90a00052e0 0x7f90a000b9c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f90ac001e20 tx=0x7f90a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d446f347ae849c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.324+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac001a70 con 0x7f90a00052e0 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.324+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac003b70 con 0x7f90a00052e0 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.324+0000 7f90b27fc640 1 -- 172.21.15.94:0/3775955445 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac0050d0 con 0x7f90a00052e0 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.325+0000 7f90b953e640 1 -- 172.21.15.94:0/3775955445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90a00052e0 msgr2=0x7f90a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.325+0000 7f90b953e640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90a00052e0 0x7f90a000b9c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f90ac001e20 tx=0x7f90a0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.325+0000 7f90b953e640 1 -- 172.21.15.94:0/3775955445 shutdown_connections 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.325+0000 7f90b953e640 1 --2- 172.21.15.94:0/3775955445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90a00052e0 0x7f90a000b9c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.325+0000 7f90b953e640 1 -- 172.21.15.94:0/3775955445 >> 172.21.15.94:0/3775955445 conn(0x7f90b406ea40 msgr2=0x7f90b406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.325+0000 7f90b953e640 1 -- 172.21.15.94:0/3775955445 shutdown_connections 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.325+0000 7f90b953e640 1 -- 172.21.15.94:0/3775955445 wait complete. 2024-03-06T07:22:21.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.326+0000 7f90b953e640 1 Processor -- start 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.326+0000 7f90b953e640 1 -- start start 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.326+0000 7f90b953e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00052e0 0x7f90b4145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.326+0000 7f90b953e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90b4144680 0x7f90b4144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b953e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90b4147070 0x7f90b41474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b953e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4158ea0 con 0x7f90b4147070 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b953e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b41561d0 con 0x7f90b4144680 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b953e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4152b00 con 0x7f90a00052e0 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90b4147070 0x7f90b41474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90b4147070 0x7f90b41474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44932/0 (socket says 172.21.15.94:44932) 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 -- 172.21.15.94:0/69963992 learned_addr learned my addr 172.21.15.94:0/69963992 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00052e0 0x7f90b4145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90abfff640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90b4144680 0x7f90b4144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 -- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00052e0 msgr2=0x7f90b4145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00052e0 0x7f90b4145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 -- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90b4144680 msgr2=0x7f90b4144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90b4144680 0x7f90b4144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 -- 172.21.15.94:0/69963992 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b41468b0 con 0x7f90b4147070 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00052e0 0x7f90b4145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b37fe640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90b4147070 0x7f90b41474f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f90ac02c040 tx=0x7f90ac0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90ab7fe640 1 -- 172.21.15.94:0/69963992 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac037460 con 0x7f90b4147070 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90ab7fe640 1 -- 172.21.15.94:0/69963992 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac037600 con 0x7f90b4147070 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b41e76d0 con 0x7f90b4147070 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90ab7fe640 1 -- 172.21.15.94:0/69963992 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac0379b0 con 0x7f90b4147070 2024-03-06T07:22:21.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.327+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b41e79b0 con 0x7f90b4147070 2024-03-06T07:22:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.329+0000 7f90ab7fe640 1 -- 172.21.15.94:0/69963992 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90ac037b50 con 0x7f90b4147070 2024-03-06T07:22:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.330+0000 7f90ab7fe640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f909406b4d0 0x7f909406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.330+0000 7f90ab7fe640 1 -- 172.21.15.94:0/69963992 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f90ac0abec0 con 0x7f90b4147070 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.330+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f909406b4d0 0x7f909406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.330+0000 7f90bc207740 1 Processor -- start 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90bc207740 1 -- start start 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f909406b4d0 0x7f909406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f90b40c2bd0 tx=0x7f90a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c00e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c00e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90bc207740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00e94b0 0x55c4c00e5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90bc207740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c004f6b0 con 0x55c4c00e94b0 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bfff8a60 con 0x55c4c00eb330 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4c004b750 con 0x55c4c00eaf50 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00e94b0 0x55c4c00e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00e94b0 0x55c4c00e5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44942/0 (socket says 172.21.15.94:44942) 2024-03-06T07:22:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90b37fe640 1 -- 172.21.15.94:0/2414715145 learned_addr learned my addr 172.21.15.94:0/2414715145 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90abfff640 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c00e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.331+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c00e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90b37fe640 1 -- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 msgr2=0x55c4c00e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c00e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90b37fe640 1 -- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 msgr2=0x55c4c00e8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c00e8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90b37fe640 1 -- 172.21.15.94:0/2414715145 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4c0016f30 con 0x55c4c00e94b0 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00e94b0 0x55c4c00e5680 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f90ac001650 tx=0x7f90ac07a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb00f8abad8b04cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90a8ff9640 1 -- 172.21.15.94:0/2414715145 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac047040 con 0x55c4c00e94b0 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90a8ff9640 1 -- 172.21.15.94:0/2414715145 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac082b70 con 0x55c4c00e94b0 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.332+0000 7f90a8ff9640 1 -- 172.21.15.94:0/2414715145 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac076b80 con 0x55c4c00e94b0 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 -- 172.21.15.94:0/2414715145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00e94b0 msgr2=0x55c4c00e5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00e94b0 0x55c4c00e5680 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f90ac001650 tx=0x7f90ac07a510 comp rx=0 tx=0).stop 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 -- 172.21.15.94:0/2414715145 shutdown_connections 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00e94b0 0x55c4c00e5680 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c00e8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 --2- 172.21.15.94:0/2414715145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c00e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 -- 172.21.15.94:0/2414715145 >> 172.21.15.94:0/2414715145 conn(0x55c4c00e77d0 msgr2=0x55c4c010c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 -- 172.21.15.94:0/2414715145 shutdown_connections 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 -- 172.21.15.94:0/2414715145 wait complete. 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 -- start start 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.333+0000 7f90bc207740 1 Processor -- start 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90bc207740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00eaf50 0x55c4c0111d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0124820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c010d690 0x55c4c0112320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90bc207740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c004b750 con 0x55c4c00eaf50 2024-03-06T07:22:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c000bea0 con 0x55c4c00eb330 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4c004f6b0 con 0x55c4c010d690 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00eaf50 0x55c4c0111d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90bc207740 10 client.0 _ll_register_callbacks cb 0x55c4c000b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00eaf50 0x55c4c0111d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44946/0 (socket says 172.21.15.94:44946) 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 -- 172.21.15.94:0/2318611289 learned_addr learned my addr 172.21.15.94:0/2318611289 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c010d690 0x55c4c0112320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90abfff640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0124820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 -- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c010d690 msgr2=0x55c4c0112320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c010d690 0x55c4c0112320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 -- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 msgr2=0x55c4c0124820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0124820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b2ffd640 1 -- 172.21.15.94:0/2318611289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4c000c270 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.334+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c010d690 0x55c4c0112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00eaf50 0x55c4c0111d70 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f90a40159c0 tx=0x7f90b4158ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567f88f743f59d5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f9091ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f9091ffb640 1 -- 172.21.15.94:0/2318611289 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4026070 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f9091ffb640 1 -- 172.21.15.94:0/2318611289 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f90a4014ae0 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90ac082b70 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f9091ffb640 1 -- 172.21.15.94:0/2318611289 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90a4032050 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 10 client.7614 fetch_fsmap learned FSMap version 96 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) v3 -- 0x55c4bfd642d0 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f9091ffb640 1 -- 172.21.15.94:0/2318611289 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac082b70 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f9091ffb640 1 -- 172.21.15.94:0/2318611289 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 96) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f90a4035020 con 0x55c4c00eaf50 2024-03-06T07:22:21.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 10 client.7614 fetch_fsmap finished waiting for FSMap version 96 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 10 client.7614 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 20 client.7614 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c013b150 0x55c4c0145f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 4 client.7614 mds_command: new command op to 7350 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 --> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4c005cbf0 con 0x55c4c013b150 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.335+0000 7f90abfff640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c013b150 0x55c4c0145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f9091ffb640 10 client.7614 ms_handle_connect on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90abfff640 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c013b150 0x55c4c0145f50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f5bd6b1af6c69e server_cookie=c5361db832dd19f7 in_seq=0 out_seq=0 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f9091ffb640 1 -- 172.21.15.94:0/2318611289 <== mds.0 v2:172.21.15.94:6832/2226553926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c4c005cbf0 con 0x55c4c013b150 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f9091ffb640 10 client.7614 handle_command_reply: tid=0 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 client.7614 shutdown 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00eaf50 msgr2=0x55c4c0111d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00eaf50 0x55c4c0111d70 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f90a40159c0 tx=0x7f90b4158ab0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 shutdown_connections 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c013b150 0x55c4c0145f50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c010d690 0x55c4c0112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0124820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 --2- 172.21.15.94:0/2318611289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c00eaf50 0x55c4c0111d70 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 >> 172.21.15.94:0/2318611289 conn(0x55c4c00e94b0 msgr2=0x55c4c00e98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f9091ffb640 0 client.7614 ms_handle_reset on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 shutdown_connections 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 1 -- 172.21.15.94:0/2318611289 wait complete. 2024-03-06T07:22:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.336+0000 7f90bc207740 20 client.7614 trim_cache size 0 max 16384 2024-03-06T07:22:21.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.353+0000 7f90bc207740 1 Processor -- start 2024-03-06T07:22:21.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.353+0000 7f90bc207740 1 -- start start 2024-03-06T07:22:21.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eaf50 0x55c4c014b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eb330 0x55c4c00e88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90bc207740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c00e4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90bc207740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b41f40d0 con 0x55c4c010d690 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b41f2680 con 0x55c4c00eaf50 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b41f2c80 con 0x55c4c00eb330 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c00e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c00e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44962/0 (socket says 172.21.15.94:44962) 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 -- 172.21.15.94:0/1580132361 learned_addr learned my addr 172.21.15.94:0/1580132361 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 -- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eb330 msgr2=0x55c4c00e88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90abfff640 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eb330 0x55c4c00e88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eb330 0x55c4c00e88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eaf50 0x55c4c014b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 -- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eaf50 msgr2=0x55c4c014b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eaf50 0x55c4c014b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 -- 172.21.15.94:0/1580132361 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4bfe2e6b0 con 0x55c4c010d690 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f90b37fe640 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c00e4510 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f90ac07a4a0 tx=0x7f90ac07d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1eac49c90b7f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f9093fff640 1 -- 172.21.15.94:0/1580132361 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac047070 con 0x55c4c010d690 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f9093fff640 1 -- 172.21.15.94:0/1580132361 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac0a7070 con 0x55c4c010d690 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.354+0000 7f9093fff640 1 -- 172.21.15.94:0/1580132361 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac082b70 con 0x55c4c010d690 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 -- 172.21.15.94:0/1580132361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 msgr2=0x55c4c00e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c00e4510 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f90ac07a4a0 tx=0x7f90ac07d000 comp rx=0 tx=0).stop 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 -- 172.21.15.94:0/1580132361 shutdown_connections 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c00e4510 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eb330 0x55c4c00e88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 --2- 172.21.15.94:0/1580132361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eaf50 0x55c4c014b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 -- 172.21.15.94:0/1580132361 >> 172.21.15.94:0/1580132361 conn(0x55c4c00e94b0 msgr2=0x55c4c00e9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 -- 172.21.15.94:0/1580132361 shutdown_connections 2024-03-06T07:22:21.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 -- 172.21.15.94:0/1580132361 wait complete. 2024-03-06T07:22:21.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 -- start start 2024-03-06T07:22:21.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.355+0000 7f90bc207740 1 Processor -- start 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c0111dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90bc207740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0199a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90bc207740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c0190220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90bc207740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c0111850 con 0x55c4c010d690 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4bfff8a60 con 0x55c4c00eb330 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90bc207740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4c00e61a0 con 0x55c4c00eaf50 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90bc207740 10 client.0 _ll_register_callbacks cb 0x7f90a4015640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c0190220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90b37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c0190220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44976/0 (socket says 172.21.15.94:44976) 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90b37fe640 1 -- 172.21.15.94:0/2074849200 learned_addr learned my addr 172.21.15.94:0/2074849200 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:21.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c0111dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.356+0000 7f90abfff640 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90b37fe640 1 -- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 msgr2=0x55c4c0111dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c0111dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90b37fe640 1 -- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 msgr2=0x55c4c0199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90b37fe640 1 -- 172.21.15.94:0/2074849200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4c00e4cc0 con 0x55c4c010d690 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90b37fe640 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c0190220 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f90ac07a4a0 tx=0x7f90ac043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff15369a9716f198 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f9091ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f9091ffb640 1 -- 172.21.15.94:0/2074849200 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac047040 con 0x55c4c010d690 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f9091ffb640 1 -- 172.21.15.94:0/2074849200 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f90ac0a3b70 con 0x55c4c010d690 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90ac0a7070 con 0x55c4c010d690 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f9091ffb640 1 -- 172.21.15.94:0/2074849200 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90ac07e370 con 0x55c4c010d690 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 10 client.7620 fetch_fsmap learned FSMap version 96 2024-03-06T07:22:21.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) v3 -- 0x55c4c00e4cc0 con 0x55c4c010d690 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f9091ffb640 1 -- 172.21.15.94:0/2074849200 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac0a7070 con 0x55c4c010d690 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f9091ffb640 1 -- 172.21.15.94:0/2074849200 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 96) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f90ac079960 con 0x55c4c010d690 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 10 client.7620 fetch_fsmap finished waiting for FSMap version 96 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 10 client.7620 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 20 client.7620 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c0190760 0x55c4c012a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 4 client.7620 mds_command: new command op to 7350 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 --> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c4c0056f40 con 0x55c4c0190760 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.357+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c0190760 0x55c4c012a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90b2ffd640 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c0190760 0x55c4c012a650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2142a3ffb72fcbc5 server_cookie=fdf8c529b6d152ca in_seq=0 out_seq=0 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f9091ffb640 10 client.7620 ms_handle_connect on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f9091ffb640 1 -- 172.21.15.94:0/2074849200 <== mds.0 v2:172.21.15.94:6832/2226553926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55c4c0056f40 con 0x55c4c0190760 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f9091ffb640 10 client.7620 handle_command_reply: tid=0 2024-03-06T07:22:21.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 client.7620 shutdown 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 msgr2=0x55c4c0190220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c0190220 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f90ac07a4a0 tx=0x7f90ac043000 comp rx=0 tx=0).stop 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 shutdown_connections 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c4c0190760 0x55c4c012a650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4c010d690 0x55c4c0190220 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4c00eb330 0x55c4c0199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f9091ffb640 0 client.7620 ms_handle_reset on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 --2- 172.21.15.94:0/2074849200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4c00eaf50 0x55c4c0111dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 >> 172.21.15.94:0/2074849200 conn(0x55c4c00e94b0 msgr2=0x55c4c00e8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 shutdown_connections 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 1 -- 172.21.15.94:0/2074849200 wait complete. 2024-03-06T07:22:21.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.358+0000 7f90bc207740 20 client.7620 trim_cache size 0 max 16384 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7350, 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 7.8630959809999998, 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 96, 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 109, 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 109, 2024-03-06T07:22:21.368 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 20.208981223999999 2024-03-06T07:22:21.369 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:21.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f909406b4d0 msgr2=0x7f909406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f909406b4d0 0x7f909406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f90b40c2bd0 tx=0x7f90a4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90b4147070 msgr2=0x7f90b41474f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90b4147070 0x7f90b41474f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f90ac02c040 tx=0x7f90ac0014a0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 shutdown_connections 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f909406b4d0 0x7f909406d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90b4147070 0x7f90b41474f0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90b4144680 0x7f90b4144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 --2- 172.21.15.94:0/69963992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a00052e0 0x7f90b4145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 >> 172.21.15.94:0/69963992 conn(0x7f90b406ea40 msgr2=0x7f90b406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 shutdown_connections 2024-03-06T07:22:21.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:21.359+0000 7f90b953e640 1 -- 172.21.15.94:0/69963992 wait complete. 2024-03-06T07:22:21.371 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 96, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:05.610686+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 96, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7350": { "addr": "172.21.15.94:6834/2226553926", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 2226553926, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 2226553926, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7350, "incarnation": 91, "join_fscid": 8, "name": "a", "rank": 0, "state": "up:active", "state_seq": 7 }, "gid_7371": { "addr": "172.21.15.145:6837/2722633283", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 2722633283, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 2722633283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7371, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:standby-replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 109, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:22:19.783264+0000", "qdb_cluster": [ 7350 ], "qdb_leader": 7350, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7350 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/3485950735", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 3485950735, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 3485950735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 92, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7144, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3093758100", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3093758100, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3093758100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7353, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:21.372 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 8, 'name': 'a', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:standby-replay'}] 2024-03-06T07:22:21.373 INFO:tasks.cephfs.test_failover:current = [{'gid': 7350, 'name': 'a', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7371, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.145:6837/2722633283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2722633283}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2722633283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7353, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3093758100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3093758100}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3093758100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7144, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/3485950735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3485950735}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3485950735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 92}] 2024-03-06T07:22:21.373 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-03-06T07:22:21.373 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:22:22.229 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:22:26.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.475+0000 7fd2b71b5640 1 Processor -- start 2024-03-06T07:22:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.475+0000 7fd2b71b5640 1 -- start start 2024-03-06T07:22:26.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b71b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b00c1aa0 0x7fd2b01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b71b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd2b0155a40 0x7fd2b0155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b71b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b0156410 con 0x7fd2b014f790 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b71b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b00c2680 con 0x7fd2b0155a40 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b71b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00c28d0 con 0x7fd2b00c1aa0 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b5752640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd2b0155a40 0x7fd2b0155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b5752640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd2b0155a40 0x7fd2b0155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44980/0 (socket says 172.21.15.94:44980) 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2b5752640 1 -- 172.21.15.94:0/3004049792 learned_addr learned my addr 172.21.15.94:0/3004049792 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.476+0000 7fd2affff640 1 --2- 172.21.15.94:0/3004049792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b00c1aa0 0x7fd2b01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2b5752640 1 -- 172.21.15.94:0/3004049792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b00c1aa0 msgr2=0x7fd2b01529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2b5752640 1 --2- 172.21.15.94:0/3004049792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b00c1aa0 0x7fd2b01529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2b5752640 1 -- 172.21.15.94:0/3004049792 >> v1:172.21.15.145:6789/0 conn(0x7fd2b014f790 legacy=0x7fd2b00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2b5752640 1 -- 172.21.15.94:0/3004049792 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b00c2bb0 con 0x7fd2b0155a40 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2b5752640 1 --2- 172.21.15.94:0/3004049792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b0155ed0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009e50 tx=0x7fd2a802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665f587b61d835ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2af7fe640 1 -- 172.21.15.94:0/3004049792 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a802cb00 con 0x7fd2b0155a40 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2af7fe640 1 -- 172.21.15.94:0/3004049792 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a802cca0 con 0x7fd2b0155a40 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.477+0000 7fd2af7fe640 1 -- 172.21.15.94:0/3004049792 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8034610 con 0x7fd2b0155a40 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3004049792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 msgr2=0x7fd2b0155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3004049792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b0155ed0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009e50 tx=0x7fd2a802bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3004049792 shutdown_connections 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3004049792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b0155ed0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3004049792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b00c1aa0 0x7fd2b01529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3004049792 >> 172.21.15.94:0/3004049792 conn(0x7fd2b00c6710 msgr2=0x7fd2b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3004049792 shutdown_connections 2024-03-06T07:22:26.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.478+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3004049792 wait complete. 2024-03-06T07:22:26.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.479+0000 7fd2b71b5640 1 Processor -- start 2024-03-06T07:22:26.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.479+0000 7fd2b71b5640 1 -- start start 2024-03-06T07:22:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b71b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2b00c1aa0 0x7fd2b01ecbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b71b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b014f790 0x7fd2b01e6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b71b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b01e9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b71b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00c0d60 con 0x7fd2b0155a40 2024-03-06T07:22:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b71b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b0155520 con 0x7fd2b00c1aa0 2024-03-06T07:22:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b71b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00c5e30 con 0x7fd2b014f790 2024-03-06T07:22:26.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b01e9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b01e9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44982/0 (socket says 172.21.15.94:44982) 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 -- 172.21.15.94:0/3570028380 learned_addr learned my addr 172.21.15.94:0/3570028380 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 -- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b014f790 msgr2=0x7fd2b01e6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b014f790 0x7fd2b01e6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 -- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2b00c1aa0 msgr2=0x7fd2b01ecbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2b00c1aa0 0x7fd2b01ecbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 -- 172.21.15.94:0/3570028380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b01db810 con 0x7fd2b0155a40 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.480+0000 7fd2b5752640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b01e9650 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009e50 tx=0x7fd2a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.481+0000 7fd2ad7fa640 1 -- 172.21.15.94:0/3570028380 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8043020 con 0x7fd2b0155a40 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.481+0000 7fd2ad7fa640 1 -- 172.21.15.94:0/3570028380 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a802ce50 con 0x7fd2b0155a40 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.481+0000 7fd2ad7fa640 1 -- 172.21.15.94:0/3570028380 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8009510 con 0x7fd2b0155a40 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.481+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b01e7180 con 0x7fd2b0155a40 2024-03-06T07:22:26.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.481+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b0072de0 con 0x7fd2b0155a40 2024-03-06T07:22:26.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.482+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b004f330 con 0x7fd2b0155a40 2024-03-06T07:22:26.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.482+0000 7fd2ad7fa640 1 -- 172.21.15.94:0/3570028380 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2a8046020 con 0x7fd2b0155a40 2024-03-06T07:22:26.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.482+0000 7fd2ad7fa640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd29c06b500 0x7fd29c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.483+0000 7fd2ad7fa640 1 -- 172.21.15.94:0/3570028380 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd2a80ac980 con 0x7fd2b0155a40 2024-03-06T07:22:26.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.483+0000 7fd2b4f51640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd29c06b500 0x7fd29c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.483+0000 7fd2b4f51640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd29c06b500 0x7fd29c06d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0000f60 tx=0x7fd2a00079d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.487+0000 7fd2ad7fa640 1 -- 172.21.15.94:0/3570028380 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2a804d030 con 0x7fd2b0155a40 2024-03-06T07:22:26.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.702+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2b00a5f60 con 0x7fd2b0155a40 2024-03-06T07:22:26.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.703+0000 7fd2ad7fa640 1 -- 172.21.15.94:0/3570028380 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4990 (secure 0 0 0) 0x7fd2a807b6b0 con 0x7fd2b0155a40 2024-03-06T07:22:26.705 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:26.705 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1342040049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1342040049},{"type":"v1","addr":"172.21.15.145:6837","nonce":1342040049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:22.199458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:26.705 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 98 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd29c06b500 msgr2=0x7fd29c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd29c06b500 0x7fd29c06d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0000f60 tx=0x7fd2a00079d0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 msgr2=0x7fd2b01e9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b01e9650 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009e50 tx=0x7fd2a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 shutdown_connections 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd29c06b500 0x7fd29c06d9c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2b0155a40 0x7fd2b01e9650 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2b014f790 0x7fd2b01e6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 --2- 172.21.15.94:0/3570028380 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2b00c1aa0 0x7fd2b01ecbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 >> 172.21.15.94:0/3570028380 conn(0x7fd2b00c6710 msgr2=0x7fd2b00a8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 shutdown_connections 2024-03-06T07:22:26.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:26.706+0000 7fd2b71b5640 1 -- 172.21.15.94:0/3570028380 wait complete. 2024-03-06T07:22:27.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-06T07:22:05.610686+0000', 'modified': '2024-03-06T07:22:22.199458+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7350}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7144': {'gid': 7144, 'name': 'd', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 3, 'addr': '172.21.15.145:6836/3485950735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3485950735}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3485950735}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7350': {'gid': 7350, 'name': 'a', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7350, 'qdb_cluster': [7350]} 2024-03-06T07:22:27.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:27.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:22:27.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.222+0000 7f1868cb5640 1 Processor -- start 2024-03-06T07:22:27.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.222+0000 7f1868cb5640 1 -- start start 2024-03-06T07:22:27.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.223+0000 7f1868cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f18640c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.223+0000 7f1868cb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1864155130 con 0x7f18641558d0 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.223+0000 7f1868cb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18640ad480 con 0x7f18640c8950 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.223+0000 7f1868cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1864155440 con 0x7f18640c69f0 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.223+0000 7f1862d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f18641558d0 0x7f186415fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37060/0 (socket says 172.21.15.94:37060) 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.223+0000 7f1862d9d640 1 -- 172.21.15.94:0/3861961527 learned_addr learned my addr 172.21.15.94:0/3861961527 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.223+0000 7f186259c640 1 --2- 172.21.15.94:0/3861961527 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f18640c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.224+0000 7f186159a640 1 -- 172.21.15.94:0/3861961527 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425983179 0 0) 0x7f185c001490 con 0x7f18641558d0 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.224+0000 7f186159a640 1 -- 172.21.15.94:0/3861961527 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f184c003520 con 0x7f18641558d0 2024-03-06T07:22:27.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.224+0000 7f186159a640 1 -- 172.21.15.94:0/3861961527 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007988888 0 0) 0x7f18640ad480 con 0x7f18640c8950 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.224+0000 7f186159a640 1 -- 172.21.15.94:0/3861961527 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c001490 con 0x7f18640c8950 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.224+0000 7f186259c640 1 -- 172.21.15.94:0/3861961527 >> v1:172.21.15.145:6789/0 conn(0x7f18640c8950 legacy=0x7f18640acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.224+0000 7f186259c640 1 -- 172.21.15.94:0/3861961527 >> v1:172.21.15.94:6789/0 conn(0x7f18641558d0 legacy=0x7f186415fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.224+0000 7f186259c640 1 -- 172.21.15.94:0/3861961527 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640c5d50 con 0x7f18640c69f0 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f186259c640 1 --2- 172.21.15.94:0/3861961527 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f18640c6dd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f18540097d0 tx=0x7f185402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92710e44aefd8f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f186159a640 1 -- 172.21.15.94:0/3861961527 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854035020 con 0x7f18640c69f0 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f186159a640 1 -- 172.21.15.94:0/3861961527 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1854031800 con 0x7f18640c69f0 2024-03-06T07:22:27.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f186159a640 1 -- 172.21.15.94:0/3861961527 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854031bd0 con 0x7f18640c69f0 2024-03-06T07:22:27.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f1868cb5640 1 -- 172.21.15.94:0/3861961527 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 msgr2=0x7f18640c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f1868cb5640 1 --2- 172.21.15.94:0/3861961527 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f18640c6dd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f18540097d0 tx=0x7f185402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f1868cb5640 1 -- 172.21.15.94:0/3861961527 shutdown_connections 2024-03-06T07:22:27.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f1868cb5640 1 --2- 172.21.15.94:0/3861961527 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f18640c6dd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f1868cb5640 1 -- 172.21.15.94:0/3861961527 >> 172.21.15.94:0/3861961527 conn(0x7f18640cae70 msgr2=0x7f18640cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f1868cb5640 1 -- 172.21.15.94:0/3861961527 shutdown_connections 2024-03-06T07:22:27.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.225+0000 7f1868cb5640 1 -- 172.21.15.94:0/3861961527 wait complete. 2024-03-06T07:22:27.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 Processor -- start 2024-03-06T07:22:27.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 -- start start 2024-03-06T07:22:27.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f1864158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18640c8950 0x7f1864154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18641558d0 0x7f18641edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1864152a20 con 0x7f18641558d0 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18640c6310 con 0x7f18640c8950 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1868cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1864152ba0 con 0x7f18640c69f0 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1862d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18641558d0 0x7f18641edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1862d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18641558d0 0x7f18641edd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44986/0 (socket says 172.21.15.94:44986) 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.227+0000 7f1862d9d640 1 -- 172.21.15.94:0/1340932309 learned_addr learned my addr 172.21.15.94:0/1340932309 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f186259c640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f1864158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1862d9d640 1 -- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 msgr2=0x7f1864158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1862d9d640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f1864158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1862d9d640 1 -- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18640c8950 msgr2=0x7f1864154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1861d9b640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18640c8950 0x7f1864154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1862d9d640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18640c8950 0x7f1864154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1862d9d640 1 -- 172.21.15.94:0/1340932309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640b0550 con 0x7f18641558d0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1862d9d640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18641558d0 0x7f18641edd80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f185c001f90 tx=0x7f185c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1852ffd640 1 -- 172.21.15.94:0/1340932309 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c017070 con 0x7f18641558d0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1852ffd640 1 -- 172.21.15.94:0/1340932309 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c012400 con 0x7f18641558d0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186414caa0 con 0x7f18641558d0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1852ffd640 1 -- 172.21.15.94:0/1340932309 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0127f0 con 0x7f18641558d0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.228+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18640b0270 con 0x7f18641558d0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.229+0000 7f1852ffd640 1 -- 172.21.15.94:0/1340932309 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f185c01a070 con 0x7f18641558d0 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.230+0000 7f1852ffd640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184406b4d0 0x7f184406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.230+0000 7f1852ffd640 1 -- 172.21.15.94:0/1340932309 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f185c08a310 con 0x7f18641558d0 2024-03-06T07:22:27.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.230+0000 7f186259c640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184406b4d0 0x7f184406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.231+0000 7f186259c640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184406b4d0 0x7f184406d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1854036000 tx=0x7f185403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.231+0000 7f186b979740 1 Processor -- start 2024-03-06T07:22:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.231+0000 7f186b979740 1 -- start start 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39ac52a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f186b979740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39ac52f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f186b979740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c39abb96b0 con 0x55c39ac55330 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c39ab62a60 con 0x55c39ac54f50 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c39abb5750 con 0x55c39ac534b0 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39ac52f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39ac52f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44996/0 (socket says 172.21.15.94:44996) 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 -- 172.21.15.94:0/1680108534 learned_addr learned my addr 172.21.15.94:0/1680108534 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f186259c640 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39ac52a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1862d9d640 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 -- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 msgr2=0x55c39ac4f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 -- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 msgr2=0x55c39ac52a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39ac52a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 -- 172.21.15.94:0/1680108534 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c39ab80f30 con 0x55c39ac55330 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.232+0000 7f1861d9b640 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39ac52f70 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f18580016d0 tx=0x7f1858001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e355cafc9dca494 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f1843fff640 1 -- 172.21.15.94:0/1680108534 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858016020 con 0x55c39ac55330 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f1843fff640 1 -- 172.21.15.94:0/1680108534 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185800db40 con 0x55c39ac55330 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f1843fff640 1 -- 172.21.15.94:0/1680108534 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18580123f0 con 0x55c39ac55330 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 -- 172.21.15.94:0/1680108534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 msgr2=0x55c39ac52f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39ac52f70 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f18580016d0 tx=0x7f1858001b90 comp rx=0 tx=0).stop 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 -- 172.21.15.94:0/1680108534 shutdown_connections 2024-03-06T07:22:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39ac52f70 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 --2- 172.21.15.94:0/1680108534 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39ac52a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 -- 172.21.15.94:0/1680108534 >> 172.21.15.94:0/1680108534 conn(0x55c39ac517d0 msgr2=0x55c39ac763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 -- 172.21.15.94:0/1680108534 shutdown_connections 2024-03-06T07:22:27.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.233+0000 7f186b979740 1 -- 172.21.15.94:0/1680108534 wait complete. 2024-03-06T07:22:27.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.234+0000 7f186b979740 1 -- start start 2024-03-06T07:22:27.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.234+0000 7f186b979740 1 Processor -- start 2024-03-06T07:22:27.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac94a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 0x55c39ac78a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac78f90 0x55c39ac71dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c39ac77230 con 0x55c39ac54f50 2024-03-06T07:22:27.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c39ab75ea0 con 0x55c39ac55330 2024-03-06T07:22:27.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c39ac76c70 con 0x55c39ac78f90 2024-03-06T07:22:27.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 10 client.0 _ll_register_callbacks cb 0x55c39ab75870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:27.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac94a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac94a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45008/0 (socket says 172.21.15.94:45008) 2024-03-06T07:22:27.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 -- 172.21.15.94:0/3006669556 learned_addr learned my addr 172.21.15.94:0/3006669556 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 -- 172.21.15.94:0/3006669556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac78f90 msgr2=0x55c39ac71dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac78f90 0x55c39ac71dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 -- 172.21.15.94:0/3006669556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 msgr2=0x55c39ac78a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 0x55c39ac78a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 -- 172.21.15.94:0/3006669556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c39ab76270 con 0x55c39ac54f50 2024-03-06T07:22:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186259c640 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac94a70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f185402beb0 tx=0x7f1854004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d30bce6e0dc5e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f18417fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f18417fa640 1 -- 172.21.15.94:0/3006669556 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854035040 con 0x55c39ac54f50 2024-03-06T07:22:27.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f18417fa640 1 -- 172.21.15.94:0/3006669556 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f185403ae00 con 0x55c39ac54f50 2024-03-06T07:22:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.235+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f185800db40 con 0x55c39ac54f50 2024-03-06T07:22:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f18417fa640 1 -- 172.21.15.94:0/3006669556 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1854045610 con 0x55c39ac54f50 2024-03-06T07:22:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 10 client.7644 fetch_fsmap learned FSMap version 98 2024-03-06T07:22:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x55c39a8ce2d0 con 0x55c39ac54f50 2024-03-06T07:22:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f18417fa640 1 -- 172.21.15.94:0/3006669556 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185800db40 con 0x55c39ac54f50 2024-03-06T07:22:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f18417fa640 1 -- 172.21.15.94:0/3006669556 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 98) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f1854057050 con 0x55c39ac54f50 2024-03-06T07:22:27.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 10 client.7644 fetch_fsmap finished waiting for FSMap version 98 2024-03-06T07:22:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 10 client.7644 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 20 client.7644 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39aca5680 0x55c39acb02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 4 client.7644 mds_command: new command op to 7350 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 --> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c39abc6bf0 con 0x55c39aca5680 2024-03-06T07:22:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f1861d9b640 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39aca5680 0x55c39acb02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f18417fa640 10 client.7644 ms_handle_connect on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.236+0000 7f1861d9b640 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39aca5680 0x55c39acb02a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a87a218832790351 server_cookie=b13693c22765bf0b in_seq=0 out_seq=0 2024-03-06T07:22:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f18417fa640 1 -- 172.21.15.94:0/3006669556 <== mds.0 v2:172.21.15.94:6832/2226553926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c39abc6bf0 con 0x55c39aca5680 2024-03-06T07:22:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f18417fa640 10 client.7644 handle_command_reply: tid=0 2024-03-06T07:22:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 client.7644 shutdown 2024-03-06T07:22:27.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 msgr2=0x55c39ac94a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac94a70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f185402beb0 tx=0x7f1854004730 comp rx=0 tx=0).stop 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 shutdown_connections 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39aca5680 0x55c39acb02a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac78f90 0x55c39ac71dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 0x55c39ac78a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 --2- 172.21.15.94:0/3006669556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac94a70 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f18417fa640 0 client.7644 ms_handle_reset on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:27.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 >> 172.21.15.94:0/3006669556 conn(0x55c39ac534b0 msgr2=0x55c39ac538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 shutdown_connections 2024-03-06T07:22:27.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 1 -- 172.21.15.94:0/3006669556 wait complete. 2024-03-06T07:22:27.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.237+0000 7f186b979740 20 client.7644 trim_cache size 0 max 16384 2024-03-06T07:22:27.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 Processor -- start 2024-03-06T07:22:27.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 -- start start 2024-03-06T07:22:27.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39aca1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39aca1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18641f7c30 con 0x55c39ac55330 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186414c6e0 con 0x55c39ac54f50 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.254+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18641f6820 con 0x55c39ac534b0 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39aca1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39aca1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45016/0 (socket says 172.21.15.94:45016) 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 -- 172.21.15.94:0/2853064170 learned_addr learned my addr 172.21.15.94:0/2853064170 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f186259c640 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 -- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 msgr2=0x55c39ac4e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 -- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 msgr2=0x55c39aca1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1861d9b640 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39aca1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39aca1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 -- 172.21.15.94:0/2853064170 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c39a9986b0 con 0x55c39ac55330 2024-03-06T07:22:27.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f186259c640 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f1862d9d640 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39aca1b70 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f185c058920 tx=0x7f185c058ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48ffe7440d79f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f18437fe640 1 -- 172.21.15.94:0/2853064170 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c056c90 con 0x55c39ac55330 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f18437fe640 1 -- 172.21.15.94:0/2853064170 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c055000 con 0x55c39ac55330 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.255+0000 7f18437fe640 1 -- 172.21.15.94:0/2853064170 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c060da0 con 0x55c39ac55330 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 -- 172.21.15.94:0/2853064170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 msgr2=0x55c39aca1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39aca1b70 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f185c058920 tx=0x7f185c058ad0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 -- 172.21.15.94:0/2853064170 shutdown_connections 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac55330 0x55c39aca1b70 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac54f50 0x55c39aca1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 --2- 172.21.15.94:0/2853064170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ac4e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 -- 172.21.15.94:0/2853064170 >> 172.21.15.94:0/2853064170 conn(0x55c39ac50e70 msgr2=0x55c39ac94880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 -- 172.21.15.94:0/2853064170 shutdown_connections 2024-03-06T07:22:27.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 -- 172.21.15.94:0/2853064170 wait complete. 2024-03-06T07:22:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 -- start start 2024-03-06T07:22:27.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.256+0000 7f186b979740 1 Processor -- start 2024-03-06T07:22:27.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ad015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186b979740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac8d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186b979740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 0x55c39aca9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186b979740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c39ac77a60 con 0x55c39ac54f50 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c39ac8e600 con 0x55c39ac55330 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186b979740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c39aca50f0 con 0x55c39ac534b0 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186b979740 10 client.0 _ll_register_callbacks cb 0x7f1854009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac8d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac8d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45018/0 (socket says 172.21.15.94:45018) 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f1861d9b640 1 -- 172.21.15.94:0/1354382943 learned_addr learned my addr 172.21.15.94:0/1354382943 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f186259c640 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ad015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.257+0000 7f1862d9d640 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 0x55c39aca9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f1861d9b640 1 -- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 msgr2=0x55c39ad015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f1861d9b640 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ad015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f1861d9b640 1 -- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 msgr2=0x55c39aca9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f1861d9b640 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 0x55c39aca9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f1861d9b640 1 -- 172.21.15.94:0/1354382943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c39ac8da10 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f1861d9b640 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac8d4d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x55c39ac945d0 tx=0x7f1858000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e272a64c205575f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f18417fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f18417fa640 1 -- 172.21.15.94:0/1354382943 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185802b060 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f18417fa640 1 -- 172.21.15.94:0/1354382943 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1858005270 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f185c055000 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f18417fa640 1 -- 172.21.15.94:0/1354382943 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1858031050 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 10 client.7650 fetch_fsmap learned FSMap version 98 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x55c39ac7b1a0 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f18417fa640 1 -- 172.21.15.94:0/1354382943 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c055000 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f18417fa640 1 -- 172.21.15.94:0/1354382943 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 98) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f1858034020 con 0x55c39ac54f50 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 10 client.7650 fetch_fsmap finished waiting for FSMap version 98 2024-03-06T07:22:27.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 10 client.7650 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 20 client.7650 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39ac78f90 0x55c39ad04bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 4 client.7650 mds_command: new command op to 7350 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.258+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 --> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c39abc0f40 con 0x55c39ac78f90 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186259c640 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39ac78f90 0x55c39ad04bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f18417fa640 10 client.7650 ms_handle_connect on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186259c640 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39ac78f90 0x55c39ad04bb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e22d3c948731e60 server_cookie=cafd84f9e857bef9 in_seq=0 out_seq=0 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f18417fa640 1 -- 172.21.15.94:0/1354382943 <== mds.0 v2:172.21.15.94:6832/2226553926 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c39abc0f40 con 0x55c39ac78f90 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f18417fa640 10 client.7650 handle_command_reply: tid=0 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 client.7650 shutdown 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 msgr2=0x55c39ac8d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac8d4d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x55c39ac945d0 tx=0x7f1858000bc0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 shutdown_connections 2024-03-06T07:22:27.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:6832/2226553926,v1:172.21.15.94:6834/2226553926] conn(0x55c39ac78f90 0x55c39ad04bb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c39ac55330 0x55c39aca9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c39ac54f50 0x55c39ac8d4d0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 --2- 172.21.15.94:0/1354382943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c39ac534b0 0x55c39ad015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f18417fa640 0 client.7650 ms_handle_reset on v2:172.21.15.94:6832/2226553926 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 >> 172.21.15.94:0/1354382943 conn(0x55c39ac50e70 msgr2=0x55c39ac4e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 shutdown_connections 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 1 -- 172.21.15.94:0/1354382943 wait complete. 2024-03-06T07:22:27.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.259+0000 7f186b979740 20 client.7650 trim_cache size 0 max 16384 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7350, 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 13.764257034, 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 97, 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 109, 2024-03-06T07:22:27.269 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 109, 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 26.110146952000001 2024-03-06T07:22:27.270 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184406b4d0 msgr2=0x7f184406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184406b4d0 0x7f184406d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1854036000 tx=0x7f185403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18641558d0 msgr2=0x7f18641edd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18641558d0 0x7f18641edd80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f185c001f90 tx=0x7f185c00a430 comp rx=0 tx=0).stop 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 shutdown_connections 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184406b4d0 0x7f184406d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18641558d0 0x7f18641edd80 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18640c8950 0x7f1864154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 --2- 172.21.15.94:0/1340932309 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18640c69f0 0x7f1864158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 >> 172.21.15.94:0/1340932309 conn(0x7f18640cae70 msgr2=0x7f18640cb250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 shutdown_connections 2024-03-06T07:22:27.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.260+0000 7f1868cb5640 1 -- 172.21.15.94:0/1340932309 wait complete. 2024-03-06T07:22:27.276 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:05.610686+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 97, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7144": { "addr": "172.21.15.145:6836/3485950735", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 3485950735, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 3485950735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7144, "incarnation": 0, "join_fscid": 8, "name": "d", "rank": 0, "state": "up:standby-replay", "state_seq": 3 }, "gid_7350": { "addr": "172.21.15.94:6834/2226553926", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 2226553926, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 2226553926, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7350, "incarnation": 91, "join_fscid": 8, "name": "a", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-03-06T07:22:22.199458+0000", "qdb_cluster": [ 7350 ], "qdb_leader": 7350, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7350 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/1342040049", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 1342040049, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 1342040049, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 98, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7201, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3093758100", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3093758100, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3093758100, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 85, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7353, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:27.277 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 8, 'name': 'a', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:standby-replay'}] 2024-03-06T07:22:27.277 INFO:tasks.cephfs.test_failover:current = [{'gid': 7350, 'name': 'a', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.94:6834/2226553926', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 2226553926}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 2226553926}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7201, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/1342040049', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1342040049}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1342040049}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 98}, {'gid': 7353, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3093758100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3093758100}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3093758100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 85}, {'gid': 7144, 'name': 'd', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 3, 'addr': '172.21.15.145:6836/3485950735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3485950735}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 3485950735}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:22:27.277 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:22:27.278 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:27.278 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:27.323 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:27.369 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:27.369 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:22:27.369 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:27.385 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:22:27.385 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:22:27.385 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:27.403 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:22:27.403 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:27.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.525+0000 7f2f8f5ac640 1 Processor -- start 2024-03-06T07:22:27.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.525+0000 7f2f8f5ac640 1 -- start start 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8f5ac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2f881506b0 0x7f2f88153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8f5ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f880acc00 0x7f2f880c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8f5ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f88070aa0 con 0x7f2f88156ea0 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8f5ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f88156450 con 0x7f2f880acc00 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8f5ac640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f88156a10 con 0x7f2f881506b0 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8db49640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f880acc00 0x7f2f880c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8db49640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2f880acc00 0x7f2f880c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45032/0 (socket says 172.21.15.94:45032) 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8db49640 1 -- 172.21.15.94:0/3563448841 learned_addr learned my addr 172.21.15.94:0/3563448841 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8db49640 1 -- 172.21.15.94:0/3563448841 >> v2:172.21.15.145:3300/0 conn(0x7f2f881506b0 msgr2=0x7f2f88153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8db49640 1 --2- 172.21.15.94:0/3563448841 >> v2:172.21.15.145:3300/0 conn(0x7f2f881506b0 0x7f2f88153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.526+0000 7f2f8db49640 1 -- 172.21.15.94:0/3563448841 >> v1:172.21.15.145:6789/0 conn(0x7f2f88156ea0 legacy=0x7f2f880702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.527+0000 7f2f8db49640 1 -- 172.21.15.94:0/3563448841 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f88070d80 con 0x7f2f880acc00 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.527+0000 7f2f8db49640 1 --2- 172.21.15.94:0/3563448841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f880acc00 0x7f2f880c54a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f2f880c97e0 tx=0x7f2f8002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc95df5dd4c777a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.527+0000 7f2f7ffff640 1 -- 172.21.15.94:0/3563448841 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8002ca80 con 0x7f2f880acc00 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.527+0000 7f2f7ffff640 1 -- 172.21.15.94:0/3563448841 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8002cc20 con 0x7f2f880acc00 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.527+0000 7f2f7ffff640 1 -- 172.21.15.94:0/3563448841 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f800343f0 con 0x7f2f880acc00 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/3563448841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f880acc00 msgr2=0x7f2f880c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/3563448841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f880acc00 0x7f2f880c54a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f2f880c97e0 tx=0x7f2f8002bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/3563448841 shutdown_connections 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/3563448841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f880acc00 0x7f2f880c54a0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/3563448841 >> v2:172.21.15.145:3300/0 conn(0x7f2f881506b0 0x7f2f88153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/3563448841 >> 172.21.15.94:0/3563448841 conn(0x7f2f8806b1a0 msgr2=0x7f2f8806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/3563448841 shutdown_connections 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.528+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/3563448841 wait complete. 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.529+0000 7f2f8f5ac640 1 Processor -- start 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.529+0000 7f2f8f5ac640 1 -- start start 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8f5ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f880acc00 0x7f2f880aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8f5ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f881506b0 0x7f2f880aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8f5ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f88156ea0 0x7f2f8814ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8f5ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f880ad1d0 con 0x7f2f88156ea0 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8f5ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f880ad050 con 0x7f2f880acc00 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8f5ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f880ad350 con 0x7f2f881506b0 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f88156ea0 0x7f2f8814ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f88156ea0 0x7f2f8814ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45044/0 (socket says 172.21.15.94:45044) 2024-03-06T07:22:27.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 -- 172.21.15.94:0/2352978074 learned_addr learned my addr 172.21.15.94:0/2352978074 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8cb47640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f881506b0 0x7f2f880aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8d348640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f880acc00 0x7f2f880aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 -- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f881506b0 msgr2=0x7f2f880aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f881506b0 0x7f2f880aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 -- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f880acc00 msgr2=0x7f2f880aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f880acc00 0x7f2f880aaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 -- 172.21.15.94:0/2352978074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f880a8ee0 con 0x7f2f88156ea0 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8d348640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f880acc00 0x7f2f880aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8cb47640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f881506b0 0x7f2f880aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.530+0000 7f2f8db49640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f88156ea0 0x7f2f8814ac30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f2f80036040 tx=0x7f2f80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.531+0000 7f2f7dffb640 1 -- 172.21.15.94:0/2352978074 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80043020 con 0x7f2f88156ea0 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.531+0000 7f2f7dffb640 1 -- 172.21.15.94:0/2352978074 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8003e450 con 0x7f2f88156ea0 2024-03-06T07:22:27.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.531+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f881e9340 con 0x7f2f88156ea0 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.531+0000 7f2f7dffb640 1 -- 172.21.15.94:0/2352978074 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80009510 con 0x7f2f88156ea0 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.531+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8814c500 con 0x7f2f88156ea0 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.532+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8804f330 con 0x7f2f88156ea0 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.532+0000 7f2f7dffb640 1 -- 172.21.15.94:0/2352978074 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f80046020 con 0x7f2f88156ea0 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.533+0000 7f2f7dffb640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f6c06b4d0 0x7f2f6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.533+0000 7f2f7dffb640 1 -- 172.21.15.94:0/2352978074 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2f800acc30 con 0x7f2f88156ea0 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.533+0000 7f2f8d348640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f6c06b4d0 0x7f2f6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.534+0000 7f2f8d348640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f6c06b4d0 0x7f2f6c06d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2f880c4070 tx=0x7f2f84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.537+0000 7f2f7dffb640 1 -- 172.21.15.94:0/2352978074 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f8004d030 con 0x7f2f88156ea0 2024-03-06T07:22:27.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.761+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f880700a0 con 0x7f2f88156ea0 2024-03-06T07:22:27.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.761+0000 7f2f7dffb640 1 -- 172.21.15.94:0/2352978074 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4990 (secure 0 0 0) 0x7f2f8007ba20 con 0x7f2f88156ea0 2024-03-06T07:22:27.764 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:27.764 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1342040049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1342040049},{"type":"v1","addr":"172.21.15.145:6837","nonce":1342040049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:22.199458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:27.764 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 98 2024-03-06T07:22:27.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f6c06b4d0 msgr2=0x7f2f6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f6c06b4d0 0x7f2f6c06d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2f880c4070 tx=0x7f2f84008040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f88156ea0 msgr2=0x7f2f8814ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f88156ea0 0x7f2f8814ac30 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f2f80036040 tx=0x7f2f80004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 shutdown_connections 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f6c06b4d0 0x7f2f6c06d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2f88156ea0 0x7f2f8814ac30 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f881506b0 0x7f2f880aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 --2- 172.21.15.94:0/2352978074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f880acc00 0x7f2f880aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 >> 172.21.15.94:0/2352978074 conn(0x7f2f8806b1a0 msgr2=0x7f2f880c48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 shutdown_connections 2024-03-06T07:22:27.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.764+0000 7f2f8f5ac640 1 -- 172.21.15.94:0/2352978074 wait complete. 2024-03-06T07:22:27.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.880+0000 7fc1097d6640 1 Processor -- start 2024-03-06T07:22:27.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.880+0000 7fc1097d6640 1 -- start start 2024-03-06T07:22:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc1097d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1040c5480 0x7fc104155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc1097d6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc1040c5860 0x7fc10406f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc1097d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1041586f0 0x7fc104158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc1097d6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10406df40 con 0x7fc1040c5480 2024-03-06T07:22:27.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc1097d6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1040c5cc0 con 0x7fc1040c5860 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc1097d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1041591c0 con 0x7fc1041586f0 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc102ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1040c5480 0x7fc104155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc102ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1040c5480 0x7fc104155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45060/0 (socket says 172.21.15.94:45060) 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.881+0000 7fc102ffd640 1 -- 172.21.15.94:0/3565482169 learned_addr learned my addr 172.21.15.94:0/3565482169 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc1037fe640 1 --2- 172.21.15.94:0/3565482169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1041586f0 0x7fc104158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc1027fc640 1 --2- 172.21.15.94:0/3565482169 >> v2:172.21.15.145:3300/0 conn(0x7fc1040c5860 0x7fc10406f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc102ffd640 1 -- 172.21.15.94:0/3565482169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1041586f0 msgr2=0x7fc104158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc102ffd640 1 --2- 172.21.15.94:0/3565482169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1041586f0 0x7fc104158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc102ffd640 1 -- 172.21.15.94:0/3565482169 >> v2:172.21.15.145:3300/0 conn(0x7fc1040c5860 msgr2=0x7fc10406f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc102ffd640 1 --2- 172.21.15.94:0/3565482169 >> v2:172.21.15.145:3300/0 conn(0x7fc1040c5860 0x7fc10406f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc102ffd640 1 -- 172.21.15.94:0/3565482169 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1041594a0 con 0x7fc1040c5480 2024-03-06T07:22:27.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc102ffd640 1 --2- 172.21.15.94:0/3565482169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1040c5480 0x7fc104155ba0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc1040c9770 tx=0x7fc0f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393bfc593d797c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.882+0000 7fc101ffb640 1 -- 172.21.15.94:0/3565482169 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f802e070 con 0x7fc1040c5480 2024-03-06T07:22:27.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc101ffb640 1 -- 172.21.15.94:0/3565482169 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f8037070 con 0x7fc1040c5480 2024-03-06T07:22:27.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc101ffb640 1 -- 172.21.15.94:0/3565482169 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8031a00 con 0x7fc1040c5480 2024-03-06T07:22:27.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 -- 172.21.15.94:0/3565482169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1040c5480 msgr2=0x7fc104155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 --2- 172.21.15.94:0/3565482169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1040c5480 0x7fc104155ba0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc1040c9770 tx=0x7fc0f802b880 comp rx=0 tx=0).stop 2024-03-06T07:22:27.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 -- 172.21.15.94:0/3565482169 shutdown_connections 2024-03-06T07:22:27.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 --2- 172.21.15.94:0/3565482169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1041586f0 0x7fc104158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 --2- 172.21.15.94:0/3565482169 >> v2:172.21.15.145:3300/0 conn(0x7fc1040c5860 0x7fc10406f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 --2- 172.21.15.94:0/3565482169 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1040c5480 0x7fc104155ba0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 -- 172.21.15.94:0/3565482169 >> 172.21.15.94:0/3565482169 conn(0x7fc10406b1a0 msgr2=0x7fc10406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 -- 172.21.15.94:0/3565482169 shutdown_connections 2024-03-06T07:22:27.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.883+0000 7fc1097d6640 1 -- 172.21.15.94:0/3565482169 wait complete. 2024-03-06T07:22:27.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.885+0000 7fc1097d6640 1 Processor -- start 2024-03-06T07:22:27.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.885+0000 7fc1097d6640 1 -- start start 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.886+0000 7fc1097d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1040c5480 0x7fc104157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.886+0000 7fc1097d6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1040c5860 0x7fc1041ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.886+0000 7fc1097d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1041586f0 0x7fc1040ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.886+0000 7fc1097d6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc104152a20 con 0x7fc1041586f0 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.886+0000 7fc1097d6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc104155680 con 0x7fc1040c5860 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.886+0000 7fc1097d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc104155050 con 0x7fc1040c5480 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.886+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1041586f0 0x7fc1040ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1041586f0 0x7fc1040ab990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45070/0 (socket says 172.21.15.94:45070) 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 -- 172.21.15.94:0/1715095674 learned_addr learned my addr 172.21.15.94:0/1715095674 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc102ffd640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1040c5480 0x7fc104157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1027fc640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1040c5860 0x7fc1041ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 -- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1040c5480 msgr2=0x7fc104157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1040c5480 0x7fc104157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 -- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1040c5860 msgr2=0x7fc1041ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1040c5860 0x7fc1041ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 -- 172.21.15.94:0/1715095674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc104156ef0 con 0x7fc1041586f0 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1037fe640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1041586f0 0x7fc1040ab990 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00cd50 tx=0x7fc0fc00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc0eb7fe640 1 -- 172.21.15.94:0/1715095674 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc003b90 con 0x7fc1041586f0 2024-03-06T07:22:27.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc0eb7fe640 1 -- 172.21.15.94:0/1715095674 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc003d30 con 0x7fc1041586f0 2024-03-06T07:22:27.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1041eef70 con 0x7fc1041586f0 2024-03-06T07:22:27.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc0eb7fe640 1 -- 172.21.15.94:0/1715095674 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc004240 con 0x7fc1041586f0 2024-03-06T07:22:27.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.887+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1041ef250 con 0x7fc1041586f0 2024-03-06T07:22:27.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.888+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10404f330 con 0x7fc1041586f0 2024-03-06T07:22:27.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.889+0000 7fc0eb7fe640 1 -- 172.21.15.94:0/1715095674 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0fc0043e0 con 0x7fc1041586f0 2024-03-06T07:22:27.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.889+0000 7fc0eb7fe640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f006b4d0 0x7fc0f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.889+0000 7fc0eb7fe640 1 -- 172.21.15.94:0/1715095674 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc0fc08ab50 con 0x7fc1041586f0 2024-03-06T07:22:27.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.889+0000 7fc102ffd640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f006b4d0 0x7fc0f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.890+0000 7fc102ffd640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f006b4d0 0x7fc0f006d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8036000 tx=0x7fc0f803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:27.893+0000 7fc0eb7fe640 1 -- 172.21.15.94:0/1715095674 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0fc059230 con 0x7fc1041586f0 2024-03-06T07:22:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.072+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1040c2670 con 0x7fc1041586f0 2024-03-06T07:22:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.073+0000 7fc0eb7fe640 1 -- 172.21.15.94:0/1715095674 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+17083 (secure 0 0 0) 0x7fc0fc05d2d0 con 0x7fc1041586f0 2024-03-06T07:22:28.075 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:28.075 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":109,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:13.494390+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.827640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.579202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/4001418202":"2024-03-07T07:22:13.494369+0000","172.21.15.145:6834/4001418202":"2024-03-07T07:22:13.494369+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:28.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f006b4d0 msgr2=0x7fc0f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f006b4d0 0x7fc0f006d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8036000 tx=0x7fc0f803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:28.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1041586f0 msgr2=0x7fc1040ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1041586f0 0x7fc1040ab990 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00cd50 tx=0x7fc0fc00a660 comp rx=0 tx=0).stop 2024-03-06T07:22:28.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 shutdown_connections 2024-03-06T07:22:28.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f006b4d0 0x7fc0f006d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1041586f0 0x7fc1040ab990 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1040c5860 0x7fc1041ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 --2- 172.21.15.94:0/1715095674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1040c5480 0x7fc104157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 >> 172.21.15.94:0/1715095674 conn(0x7fc10406b1a0 msgr2=0x7fc104153820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 shutdown_connections 2024-03-06T07:22:28.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.075+0000 7fc1097d6640 1 -- 172.21.15.94:0/1715095674 wait complete. 2024-03-06T07:22:28.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:03.827640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:04.579202+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:28.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:28.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.194+0000 7f4f6d088640 1 Processor -- start 2024-03-06T07:22:28.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.194+0000 7f4f6d088640 1 -- start start 2024-03-06T07:22:28.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6d088640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4f6814ef70 0x7f4f6806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6d088640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f6806cad0 0x7f4f68154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6d088640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680c2790 con 0x7f4f680c1820 2024-03-06T07:22:28.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6d088640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f680c3090 con 0x7f4f6806cad0 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6d088640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6814f350 con 0x7f4f6814ef70 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6759e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f6806cad0 0x7f4f68154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6759e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f6806cad0 0x7f4f68154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45072/0 (socket says 172.21.15.94:45072) 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6759e640 1 -- 172.21.15.94:0/3179377767 learned_addr learned my addr 172.21.15.94:0/3179377767 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f66d9d640 1 --2- 172.21.15.94:0/3179377767 >> v2:172.21.15.145:3300/0 conn(0x7f4f6814ef70 0x7f4f6806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6759e640 1 -- 172.21.15.94:0/3179377767 >> v2:172.21.15.145:3300/0 conn(0x7f4f6814ef70 msgr2=0x7f4f6806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6759e640 1 --2- 172.21.15.94:0/3179377767 >> v2:172.21.15.145:3300/0 conn(0x7f4f6814ef70 0x7f4f6806c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6759e640 1 -- 172.21.15.94:0/3179377767 >> v1:172.21.15.145:6789/0 conn(0x7f4f680c1820 legacy=0x7f4f680c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.195+0000 7f4f6759e640 1 -- 172.21.15.94:0/3179377767 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6806cf40 con 0x7f4f6806cad0 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.196+0000 7f4f6759e640 1 --2- 172.21.15.94:0/3179377767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f6806cad0 0x7f4f68154c30 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4f6806e5d0 tx=0x7f4f5802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a8663990ff5b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.196+0000 7f4f65d9b640 1 -- 172.21.15.94:0/3179377767 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5802c9f0 con 0x7f4f6806cad0 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.196+0000 7f4f65d9b640 1 -- 172.21.15.94:0/3179377767 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5802cb90 con 0x7f4f6806cad0 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.196+0000 7f4f65d9b640 1 -- 172.21.15.94:0/3179377767 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f580343f0 con 0x7f4f6806cad0 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.196+0000 7f4f6d088640 1 -- 172.21.15.94:0/3179377767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f6806cad0 msgr2=0x7f4f68154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.196+0000 7f4f6d088640 1 --2- 172.21.15.94:0/3179377767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f6806cad0 0x7f4f68154c30 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4f6806e5d0 tx=0x7f4f5802be50 comp rx=0 tx=0).stop 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.197+0000 7f4f6d088640 1 -- 172.21.15.94:0/3179377767 shutdown_connections 2024-03-06T07:22:28.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.197+0000 7f4f6d088640 1 --2- 172.21.15.94:0/3179377767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f6806cad0 0x7f4f68154c30 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.197+0000 7f4f6d088640 1 --2- 172.21.15.94:0/3179377767 >> v2:172.21.15.145:3300/0 conn(0x7f4f6814ef70 0x7f4f6806c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.197+0000 7f4f6d088640 1 -- 172.21.15.94:0/3179377767 >> 172.21.15.94:0/3179377767 conn(0x7f4f680c6710 msgr2=0x7f4f680c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.197+0000 7f4f6d088640 1 -- 172.21.15.94:0/3179377767 shutdown_connections 2024-03-06T07:22:28.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.197+0000 7f4f6d088640 1 -- 172.21.15.94:0/3179377767 wait complete. 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.199+0000 7f4f6d088640 1 Processor -- start 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.199+0000 7f4f6d088640 1 -- start start 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6d088640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f6806cad0 0x7f4f6814d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6d088640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f680c1820 0x7f4f680bf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6d088640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f6814ef70 0x7f4f680bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6d088640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68155750 con 0x7f4f680c1820 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6d088640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f680c24b0 con 0x7f4f6814ef70 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6d088640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68155170 con 0x7f4f6806cad0 2024-03-06T07:22:28.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f680c1820 0x7f4f680bf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f680c1820 0x7f4f680bf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45076/0 (socket says 172.21.15.94:45076) 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 -- 172.21.15.94:0/934769074 learned_addr learned my addr 172.21.15.94:0/934769074 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f66d9d640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f6806cad0 0x7f4f6814d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6759e640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f6814ef70 0x7f4f680bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 -- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f6806cad0 msgr2=0x7f4f6814d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f6806cad0 0x7f4f6814d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 -- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f6814ef70 msgr2=0x7f4f680bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f6814ef70 0x7f4f680bfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 -- 172.21.15.94:0/934769074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f680bbf00 con 0x7f4f680c1820 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6759e640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f6814ef70 0x7f4f680bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f66d9d640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f6806cad0 0x7f4f6814d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:28.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.200+0000 7f4f6659c640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f680c1820 0x7f4f680bf5f0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c440 tx=0x7f4f5c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.201+0000 7f4f4f7fe640 1 -- 172.21.15.94:0/934769074 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c017070 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.201+0000 7f4f4f7fe640 1 -- 172.21.15.94:0/934769074 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c0035f0 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.201+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f680c0070 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.201+0000 7f4f4f7fe640 1 -- 172.21.15.94:0/934769074 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0039e0 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.201+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f681516e0 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.202+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6804f330 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.202+0000 7f4f4f7fe640 1 -- 172.21.15.94:0/934769074 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f5c01a070 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.202+0000 7f4f4f7fe640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f5406b4d0 0x7f4f5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.203+0000 7f4f4f7fe640 1 -- 172.21.15.94:0/934769074 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4f5c0896a0 con 0x7f4f680c1820 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.203+0000 7f4f66d9d640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f5406b4d0 0x7f4f5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.203+0000 7f4f66d9d640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f5406b4d0 0x7f4f5406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000b820 tx=0x7f4f60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.207+0000 7f4f4f7fe640 1 -- 172.21.15.94:0/934769074 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f5c0583d0 con 0x7f4f680c1820 2024-03-06T07:22:28.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.385+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f6806dc80 con 0x7f4f680c1820 2024-03-06T07:22:28.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.385+0000 7f4f4f7fe640 1 -- 172.21.15.94:0/934769074 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f5c0010c0 con 0x7f4f680c1820 2024-03-06T07:22:28.387 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:22:28.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f5406b4d0 msgr2=0x7f4f5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f5406b4d0 0x7f4f5406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000b820 tx=0x7f4f60008040 comp rx=0 tx=0).stop 2024-03-06T07:22:28.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f680c1820 msgr2=0x7f4f680bf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f680c1820 0x7f4f680bf5f0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c440 tx=0x7f4f5c00c900 comp rx=0 tx=0).stop 2024-03-06T07:22:28.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 shutdown_connections 2024-03-06T07:22:28.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f5406b4d0 0x7f4f5406d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f6814ef70 0x7f4f680bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f680c1820 0x7f4f680bf5f0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 --2- 172.21.15.94:0/934769074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f6806cad0 0x7f4f6814d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 >> 172.21.15.94:0/934769074 conn(0x7f4f680c6710 msgr2=0x7f4f680ab520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 shutdown_connections 2024-03-06T07:22:28.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.387+0000 7f4f6d088640 1 -- 172.21.15.94:0/934769074 wait complete. 2024-03-06T07:22:28.403 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:28.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66c5c37640 1 Processor -- start 2024-03-06T07:22:28.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66c5c37640 1 -- start start 2024-03-06T07:22:28.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66c5c37640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f66c00b00d0 0x7f66c00c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66c5c37640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c00b04b0 con 0x7f66c00c7210 2024-03-06T07:22:28.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66c5c37640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c006d210 con 0x7f66c006b8e0 2024-03-06T07:22:28.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66c5c37640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c0156160 con 0x7f66c00b00d0 2024-03-06T07:22:28.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66bf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f66c00b00d0 0x7f66c00c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66bffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f66c00c7210 0x7f66c00c7680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50446/0 (socket says 172.21.15.94:50446) 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.511+0000 7f66bffff640 1 -- 172.21.15.94:0/688826897 learned_addr learned my addr 172.21.15.94:0/688826897 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.512+0000 7f66bf7fe640 1 -- 172.21.15.94:0/688826897 >> v1:172.21.15.145:6789/0 conn(0x7f66c006b8e0 legacy=0x7f66c006bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.512+0000 7f66bf7fe640 1 -- 172.21.15.94:0/688826897 >> v1:172.21.15.94:6789/0 conn(0x7f66c00c7210 legacy=0x7f66c00c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.512+0000 7f66bf7fe640 1 -- 172.21.15.94:0/688826897 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0155e80 con 0x7f66c00b00d0 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.512+0000 7f66beffd640 1 -- 172.21.15.94:0/688826897 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420971915 0 0) 0x7f66c00b04b0 con 0x7f66c00c7210 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.512+0000 7f66bf7fe640 1 --2- 172.21.15.94:0/688826897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c8ff0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f66c01562e0 tx=0x7f66b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e515f6a1f591edf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66beffd640 1 -- 172.21.15.94:0/688826897 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0035020 con 0x7f66c00b00d0 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66beffd640 1 -- 172.21.15.94:0/688826897 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b0031830 con 0x7f66c00b00d0 2024-03-06T07:22:28.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66beffd640 1 -- 172.21.15.94:0/688826897 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0031c00 con 0x7f66c00b00d0 2024-03-06T07:22:28.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66c5c37640 1 -- 172.21.15.94:0/688826897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 msgr2=0x7f66c00c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66c5c37640 1 --2- 172.21.15.94:0/688826897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c8ff0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f66c01562e0 tx=0x7f66b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66c5c37640 1 -- 172.21.15.94:0/688826897 shutdown_connections 2024-03-06T07:22:28.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66c5c37640 1 --2- 172.21.15.94:0/688826897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c8ff0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66c5c37640 1 -- 172.21.15.94:0/688826897 >> 172.21.15.94:0/688826897 conn(0x7f66c00ac810 msgr2=0x7f66c00acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66c5c37640 1 -- 172.21.15.94:0/688826897 shutdown_connections 2024-03-06T07:22:28.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.513+0000 7f66c5c37640 1 -- 172.21.15.94:0/688826897 wait complete. 2024-03-06T07:22:28.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.515+0000 7f66c5c37640 1 Processor -- start 2024-03-06T07:22:28.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.515+0000 7f66c5c37640 1 -- start start 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66c5c37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66c006b8e0 0x7f66c00c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66c5c37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66c5c37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66c00c7210 0x7f66c0158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66c5c37640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c0152c50 con 0x7f66c00b00d0 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66c5c37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c00c5710 con 0x7f66c00c7210 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66c5c37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c00c7d90 con 0x7f66c006b8e0 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40286/0 (socket says 172.21.15.94:40286) 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 -- 172.21.15.94:0/3481974610 learned_addr learned my addr 172.21.15.94:0/3481974610 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 -- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66c006b8e0 msgr2=0x7f66c00c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66c006b8e0 0x7f66c00c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 -- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66c00c7210 msgr2=0x7f66c0158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66c00c7210 0x7f66c0158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 -- 172.21.15.94:0/3481974610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c01576f0 con 0x7f66c00b00d0 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.516+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c21f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c410 tx=0x7f66a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.517+0000 7f66bcff9640 1 -- 172.21.15.94:0/3481974610 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8017070 con 0x7f66c00b00d0 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.517+0000 7f66bcff9640 1 -- 172.21.15.94:0/3481974610 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a80035f0 con 0x7f66c00b00d0 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.517+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c01579d0 con 0x7f66c00b00d0 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.517+0000 7f66bcff9640 1 -- 172.21.15.94:0/3481974610 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8003a20 con 0x7f66c00b00d0 2024-03-06T07:22:28.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.517+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c0157f90 con 0x7f66c00b00d0 2024-03-06T07:22:28.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.518+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c0075300 con 0x7f66c00b00d0 2024-03-06T07:22:28.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.518+0000 7f66bcff9640 1 -- 172.21.15.94:0/3481974610 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66a801a070 con 0x7f66c00b00d0 2024-03-06T07:22:28.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.518+0000 7f66bcff9640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a006b4d0 0x7f66a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.519+0000 7f66bcff9640 1 -- 172.21.15.94:0/3481974610 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f66a8089750 con 0x7f66c00b00d0 2024-03-06T07:22:28.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.519+0000 7f66bf7fe640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a006b4d0 0x7f66a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.519+0000 7f66bf7fe640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a006b4d0 0x7f66a006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f66b0036000 tx=0x7f66b0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.527+0000 7f66bcff9640 1 -- 172.21.15.94:0/3481974610 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66c0075300 con 0x7f66c00b00d0 2024-03-06T07:22:28.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.703+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f66c006ce50 con 0x7f66c00b00d0 2024-03-06T07:22:28.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.703+0000 7f66bcff9640 1 -- 172.21.15.94:0/3481974610 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) v1 ==== 104+0+11 (secure 0 0 0) 0x7f66a8058480 con 0x7f66c00b00d0 2024-03-06T07:22:28.705 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:22:28.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a006b4d0 msgr2=0x7f66a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a006b4d0 0x7f66a006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f66b0036000 tx=0x7f66b0030000 comp rx=0 tx=0).stop 2024-03-06T07:22:28.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 msgr2=0x7f66c00c21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c21f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f66a800c410 tx=0x7f66a800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 shutdown_connections 2024-03-06T07:22:28.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a006b4d0 0x7f66a006d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66c00c7210 0x7f66c0158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66c00b00d0 0x7f66c00c21f0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 --2- 172.21.15.94:0/3481974610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66c006b8e0 0x7f66c00c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 >> 172.21.15.94:0/3481974610 conn(0x7f66c00ac810 msgr2=0x7f66c0153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 shutdown_connections 2024-03-06T07:22:28.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.705+0000 7f66c5c37640 1 -- 172.21.15.94:0/3481974610 wait complete. 2024-03-06T07:22:28.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:28.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.819+0000 7f96ca62a640 1 Processor -- start 2024-03-06T07:22:28.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.819+0000 7f96ca62a640 1 -- start start 2024-03-06T07:22:28.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96ca62a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f96c40c5880 0x7f96c40c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96ca62a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c5c60 0x7f96c40c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96ca62a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f96c40c7720 0x7f96c406f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96ca62a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c40723b0 con 0x7f96c40c5880 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96ca62a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c40728e0 con 0x7f96c40c7720 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96ca62a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c4072600 con 0x7f96c40c5c60 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96c3fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f96c40c5880 0x7f96c40c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96c3fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f96c40c5880 0x7f96c40c6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40302/0 (socket says 172.21.15.94:40302) 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96c3fff640 1 -- 172.21.15.94:0/432400095 learned_addr learned my addr 172.21.15.94:0/432400095 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96c37fe640 1 --2- 172.21.15.94:0/432400095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c5c60 0x7f96c40c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.820+0000 7f96c8bc7640 1 --2- 172.21.15.94:0/432400095 >> v2:172.21.15.145:3300/0 conn(0x7f96c40c7720 0x7f96c406f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.821+0000 7f96c3fff640 1 -- 172.21.15.94:0/432400095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c5c60 msgr2=0x7f96c40c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.821+0000 7f96c3fff640 1 --2- 172.21.15.94:0/432400095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c5c60 0x7f96c40c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.821+0000 7f96c3fff640 1 -- 172.21.15.94:0/432400095 >> v2:172.21.15.145:3300/0 conn(0x7f96c40c7720 msgr2=0x7f96c406f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.821+0000 7f96c3fff640 1 --2- 172.21.15.94:0/432400095 >> v2:172.21.15.145:3300/0 conn(0x7f96c40c7720 0x7f96c406f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.821+0000 7f96c3fff640 1 -- 172.21.15.94:0/432400095 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4153af0 con 0x7f96c40c5880 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.821+0000 7f96c3fff640 1 --2- 172.21.15.94:0/432400095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c40c6ca0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009870 tx=0x7f96b802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a406b608d9665b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96c2ffd640 1 -- 172.21.15.94:0/432400095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8038070 con 0x7f96c40c5880 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96c2ffd640 1 -- 172.21.15.94:0/432400095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b8031740 con 0x7f96c40c5880 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96c2ffd640 1 -- 172.21.15.94:0/432400095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b802f050 con 0x7f96c40c5880 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 -- 172.21.15.94:0/432400095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 msgr2=0x7f96c40c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 --2- 172.21.15.94:0/432400095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c40c6ca0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009870 tx=0x7f96b802b7e0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 -- 172.21.15.94:0/432400095 shutdown_connections 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 --2- 172.21.15.94:0/432400095 >> v2:172.21.15.145:3300/0 conn(0x7f96c40c7720 0x7f96c406f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 --2- 172.21.15.94:0/432400095 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c5c60 0x7f96c40c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 --2- 172.21.15.94:0/432400095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c40c6ca0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 -- 172.21.15.94:0/432400095 >> 172.21.15.94:0/432400095 conn(0x7f96c40aa630 msgr2=0x7f96c40aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 -- 172.21.15.94:0/432400095 shutdown_connections 2024-03-06T07:22:28.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.822+0000 7f96ca62a640 1 -- 172.21.15.94:0/432400095 wait complete. 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.823+0000 7f96ca62a640 1 Processor -- start 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.823+0000 7f96ca62a640 1 -- start start 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96ca62a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c41dca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96ca62a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c40c5c60 0x7f96c414c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96ca62a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c7720 0x7f96c414cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96ca62a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c4073190 con 0x7f96c40c5880 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96ca62a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c40c6780 con 0x7f96c40c5c60 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96ca62a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c4073310 con 0x7f96c40c7720 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c41dca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c41dca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40304/0 (socket says 172.21.15.94:40304) 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 -- 172.21.15.94:0/1119190033 learned_addr learned my addr 172.21.15.94:0/1119190033 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 -- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c7720 msgr2=0x7f96c414cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c7720 0x7f96c414cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 -- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c40c5c60 msgr2=0x7f96c414c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c40c5c60 0x7f96c414c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.824+0000 7f96c3fff640 1 -- 172.21.15.94:0/1119190033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c414d3e0 con 0x7f96c40c5880 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.825+0000 7f96c3fff640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c41dca70 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f96b8000c00 tx=0x7f96b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.825+0000 7f96c0ff9640 1 -- 172.21.15.94:0/1119190033 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8042070 con 0x7f96c40c5880 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.825+0000 7f96c0ff9640 1 -- 172.21.15.94:0/1119190033 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b803d670 con 0x7f96c40c5880 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.825+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c41e97b0 con 0x7f96c40c5880 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.825+0000 7f96c0ff9640 1 -- 172.21.15.94:0/1119190033 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8038040 con 0x7f96c40c5880 2024-03-06T07:22:28.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.825+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c41e89f0 con 0x7f96c40c5880 2024-03-06T07:22:28.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.826+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c40ae780 con 0x7f96c40c5880 2024-03-06T07:22:28.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.826+0000 7f96c0ff9640 1 -- 172.21.15.94:0/1119190033 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96b8045070 con 0x7f96c40c5880 2024-03-06T07:22:28.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.827+0000 7f96c0ff9640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96b006b500 0x7f96b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.827+0000 7f96c0ff9640 1 -- 172.21.15.94:0/1119190033 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f96b80abd50 con 0x7f96c40c5880 2024-03-06T07:22:28.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.827+0000 7f96c37fe640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96b006b500 0x7f96b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.827+0000 7f96c37fe640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96b006b500 0x7f96b006d9c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f96b4006a90 tx=0x7f96b4006a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:28.835+0000 7f96c0ff9640 1 -- 172.21.15.94:0/1119190033 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96c40ae780 con 0x7f96c40c5880 2024-03-06T07:22:29.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.007+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f96c404f330 con 0x7f96c40c5880 2024-03-06T07:22:29.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.008+0000 7f96c0ff9640 1 -- 172.21.15.94:0/1119190033 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) v1 ==== 100+0+11 (secure 0 0 0) 0x7f96b80af030 con 0x7f96c40c5880 2024-03-06T07:22:29.010 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:22:29.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.009+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96b006b500 msgr2=0x7f96b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.009+0000 7f96ca62a640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96b006b500 0x7f96b006d9c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f96b4006a90 tx=0x7f96b4006a20 comp rx=0 tx=0).stop 2024-03-06T07:22:29.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 msgr2=0x7f96c41dca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c41dca70 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f96b8000c00 tx=0x7f96b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 shutdown_connections 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f96b006b500 0x7f96b006d9c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96c40c7720 0x7f96c414cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96c40c5c60 0x7f96c414c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 --2- 172.21.15.94:0/1119190033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96c40c5880 0x7f96c41dca70 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 >> 172.21.15.94:0/1119190033 conn(0x7f96c40aa630 msgr2=0x7f96c4151490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 shutdown_connections 2024-03-06T07:22:29.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.010+0000 7f96ca62a640 1 -- 172.21.15.94:0/1119190033 wait complete. 2024-03-06T07:22:29.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:29.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.123+0000 7fed294b5640 1 Processor -- start 2024-03-06T07:22:29.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.123+0000 7fed294b5640 1 -- start start 2024-03-06T07:22:29.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed294b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed241506b0 0x7fed24153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed294b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed24156ea0 0x7fed240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed294b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed24070800 con 0x7fed240acc00 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed294b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed24157320 con 0x7fed24156ea0 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed294b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed24156450 con 0x7fed241506b0 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed1bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed24156ea0 0x7fed240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed237fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fed240acc00 0x7fed240c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50458/0 (socket says 172.21.15.94:50458) 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed237fe640 1 -- 172.21.15.94:0/869486311 learned_addr learned my addr 172.21.15.94:0/869486311 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed22ffd640 1 --2- 172.21.15.94:0/869486311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed241506b0 0x7fed24153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed1bfff640 1 -- 172.21.15.94:0/869486311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed241506b0 msgr2=0x7fed24153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed1bfff640 1 --2- 172.21.15.94:0/869486311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed241506b0 0x7fed24153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed1bfff640 1 -- 172.21.15.94:0/869486311 >> v1:172.21.15.94:6789/0 conn(0x7fed240acc00 legacy=0x7fed240c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed1bfff640 1 -- 172.21.15.94:0/869486311 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed24070aa0 con 0x7fed24156ea0 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed227fc640 1 -- 172.21.15.94:0/869486311 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754435131 0 0) 0x7fed24070800 con 0x7fed240acc00 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.124+0000 7fed22ffd640 1 --2- 172.21.15.94:0/869486311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed241506b0 0x7fed24153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.125+0000 7fed1bfff640 1 --2- 172.21.15.94:0/869486311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed240702c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0097a0 tx=0x7fed0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4323cbc4399f7e1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.125+0000 7fed227fc640 1 -- 172.21.15.94:0/869486311 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c035020 con 0x7fed24156ea0 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.125+0000 7fed227fc640 1 -- 172.21.15.94:0/869486311 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c031830 con 0x7fed24156ea0 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.125+0000 7fed227fc640 1 -- 172.21.15.94:0/869486311 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c031c00 con 0x7fed24156ea0 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 -- 172.21.15.94:0/869486311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 msgr2=0x7fed240702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 --2- 172.21.15.94:0/869486311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed240702c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0097a0 tx=0x7fed0c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 -- 172.21.15.94:0/869486311 shutdown_connections 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 --2- 172.21.15.94:0/869486311 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed240702c0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 --2- 172.21.15.94:0/869486311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed241506b0 0x7fed24153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 -- 172.21.15.94:0/869486311 >> 172.21.15.94:0/869486311 conn(0x7fed2406b1a0 msgr2=0x7fed2406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 -- 172.21.15.94:0/869486311 shutdown_connections 2024-03-06T07:22:29.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.126+0000 7fed294b5640 1 -- 172.21.15.94:0/869486311 wait complete. 2024-03-06T07:22:29.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.127+0000 7fed294b5640 1 Processor -- start 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.127+0000 7fed294b5640 1 -- start start 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed294b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed240acc00 0x7fed240c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed294b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed241506b0 0x7fed240aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed294b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed24146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed294b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed241566a0 con 0x7fed24156ea0 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed294b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed24156a10 con 0x7fed241506b0 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed294b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed240c5bb0 con 0x7fed240acc00 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed24146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed24146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40316/0 (socket says 172.21.15.94:40316) 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 -- 172.21.15.94:0/1465796703 learned_addr learned my addr 172.21.15.94:0/1465796703 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed22ffd640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed240acc00 0x7fed240c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed1bfff640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed241506b0 0x7fed240aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 -- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed240acc00 msgr2=0x7fed240c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed240acc00 0x7fed240c1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 -- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed241506b0 msgr2=0x7fed240aa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed241506b0 0x7fed240aa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 -- 172.21.15.94:0/1465796703 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240c4560 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed1bfff640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed241506b0 0x7fed240aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed237fe640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed24146a10 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00ca70 tx=0x7fed1c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed1b7fe640 1 -- 172.21.15.94:0/1465796703 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c00abd0 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed1b7fe640 1 -- 172.21.15.94:0/1465796703 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c00ad70 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed240c4840 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.128+0000 7fed1b7fe640 1 -- 172.21.15.94:0/1465796703 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c012640 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.129+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed240c22a0 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.129+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2404f330 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.130+0000 7fed1b7fe640 1 -- 172.21.15.94:0/1465796703 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed1c0127e0 con 0x7fed24156ea0 2024-03-06T07:22:29.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.130+0000 7fed1b7fe640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed0406b4d0 0x7fed0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.130+0000 7fed1b7fe640 1 -- 172.21.15.94:0/1465796703 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fed1c08a8d0 con 0x7fed24156ea0 2024-03-06T07:22:29.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.130+0000 7fed22ffd640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed0406b4d0 0x7fed0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.131+0000 7fed22ffd640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed0406b4d0 0x7fed0406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fed1400d8b0 tx=0x7fed14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.135+0000 7fed1b7fe640 1 -- 172.21.15.94:0/1465796703 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed1c059600 con 0x7fed24156ea0 2024-03-06T07:22:29.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.349+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed240700a0 con 0x7fed24156ea0 2024-03-06T07:22:29.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.350+0000 7fed1b7fe640 1 -- 172.21.15.94:0/1465796703 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4990 (secure 0 0 0) 0x7fed1c058f00 con 0x7fed24156ea0 2024-03-06T07:22:29.352 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:29.352 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1342040049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1342040049},{"type":"v1","addr":"172.21.15.145:6837","nonce":1342040049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:22.199458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 98 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed0406b4d0 msgr2=0x7fed0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed0406b4d0 0x7fed0406d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fed1400d8b0 tx=0x7fed14008040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 msgr2=0x7fed24146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed24146a10 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00ca70 tx=0x7fed1c00cf30 comp rx=0 tx=0).stop 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 shutdown_connections 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fed0406b4d0 0x7fed0406d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed24156ea0 0x7fed24146a10 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed241506b0 0x7fed240aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 --2- 172.21.15.94:0/1465796703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed240acc00 0x7fed240c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 >> 172.21.15.94:0/1465796703 conn(0x7fed2406b1a0 msgr2=0x7fed2414e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 shutdown_connections 2024-03-06T07:22:29.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.352+0000 7fed294b5640 1 -- 172.21.15.94:0/1465796703 wait complete. 2024-03-06T07:22:29.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:29.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc159ff7640 1 Processor -- start 2024-03-06T07:22:29.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc159ff7640 1 -- start start 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc159ff7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540c88c0 0x7fc154155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc159ff7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1540c73d0 con 0x7fc154158b60 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc159ff7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1540be2e0 con 0x7fc15414f360 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc159ff7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc154159700 con 0x7fc1540c88c0 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc153fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc154158b60 0x7fc154158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50466/0 (socket says 172.21.15.94:50466) 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.474+0000 7fc153fff640 1 -- 172.21.15.94:0/2591979833 learned_addr learned my addr 172.21.15.94:0/2591979833 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc152ffd640 1 --2- 172.21.15.94:0/2591979833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540c88c0 0x7fc154155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722100866 0 0) 0x7fc1540c73d0 con 0x7fc154158b60 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc140003520 con 0x7fc154158b60 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797006745 0 0) 0x7fc1540be2e0 con 0x7fc15414f360 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1540c73d0 con 0x7fc15414f360 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026273745 0 0) 0x7fc140003520 con 0x7fc154158b60 2024-03-06T07:22:29.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc152ffd640 1 -- 172.21.15.94:0/2591979833 >> v1:172.21.15.145:6789/0 conn(0x7fc15414f360 legacy=0x7fc1540c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc152ffd640 1 -- 172.21.15.94:0/2591979833 >> v1:172.21.15.94:6789/0 conn(0x7fc154158b60 legacy=0x7fc154158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.475+0000 7fc152ffd640 1 -- 172.21.15.94:0/2591979833 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540c7670 con 0x7fc1540c88c0 2024-03-06T07:22:29.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.476+0000 7fc152ffd640 1 --2- 172.21.15.94:0/2591979833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540c88c0 0x7fc154155c10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0097d0 tx=0x7fc13c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f65cae0aa3dd7d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.476+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c037070 con 0x7fc1540c88c0 2024-03-06T07:22:29.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.476+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c030510 con 0x7fc1540c88c0 2024-03-06T07:22:29.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.476+0000 7fc1527fc640 1 -- 172.21.15.94:0/2591979833 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c001670 con 0x7fc1540c88c0 2024-03-06T07:22:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.476+0000 7fc159ff7640 1 -- 172.21.15.94:0/2591979833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540c88c0 msgr2=0x7fc154155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.476+0000 7fc159ff7640 1 --2- 172.21.15.94:0/2591979833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540c88c0 0x7fc154155c10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0097d0 tx=0x7fc13c02b770 comp rx=0 tx=0).stop 2024-03-06T07:22:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.477+0000 7fc159ff7640 1 -- 172.21.15.94:0/2591979833 shutdown_connections 2024-03-06T07:22:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.477+0000 7fc159ff7640 1 --2- 172.21.15.94:0/2591979833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540c88c0 0x7fc154155c10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.477+0000 7fc159ff7640 1 -- 172.21.15.94:0/2591979833 >> 172.21.15.94:0/2591979833 conn(0x7fc1540ac570 msgr2=0x7fc1540ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.477+0000 7fc159ff7640 1 -- 172.21.15.94:0/2591979833 shutdown_connections 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.477+0000 7fc159ff7640 1 -- 172.21.15.94:0/2591979833 wait complete. 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.478+0000 7fc159ff7640 1 Processor -- start 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.478+0000 7fc159ff7640 1 -- start start 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.478+0000 7fc159ff7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1540c88c0 0x7fc154154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc159ff7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc15414f360 0x7fc1541ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc159ff7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc154158b60 0x7fc1541ee960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc159ff7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540c7c30 con 0x7fc1540c88c0 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc159ff7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540c5a50 con 0x7fc154158b60 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc159ff7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540c5bd0 con 0x7fc15414f360 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1540c88c0 0x7fc154154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1540c88c0 0x7fc154154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40332/0 (socket says 172.21.15.94:40332) 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 -- 172.21.15.94:0/3275631704 learned_addr learned my addr 172.21.15.94:0/3275631704 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 -- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc15414f360 msgr2=0x7fc1541ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc153fff640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc154158b60 0x7fc1541ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc152ffd640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc15414f360 0x7fc1541ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc15414f360 0x7fc1541ee420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 -- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc154158b60 msgr2=0x7fc1541ee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc154158b60 0x7fc1541ee960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 -- 172.21.15.94:0/3275631704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540af2e0 con 0x7fc1540c88c0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc1537fe640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1540c88c0 0x7fc154154380 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc14400c830 tx=0x7fc14400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc14bfff640 1 -- 172.21.15.94:0/3275631704 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144003b70 con 0x7fc1540c88c0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc14bfff640 1 -- 172.21.15.94:0/3275631704 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc144003d10 con 0x7fc1540c88c0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1540c23b0 con 0x7fc1540c88c0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc14bfff640 1 -- 172.21.15.94:0/3275631704 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14400aa90 con 0x7fc1540c88c0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.479+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1540c2970 con 0x7fc1540c88c0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.480+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1541eeea0 con 0x7fc1540c88c0 2024-03-06T07:22:29.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.481+0000 7fc14bfff640 1 -- 172.21.15.94:0/3275631704 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc14400ac30 con 0x7fc1540c88c0 2024-03-06T07:22:29.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.481+0000 7fc14bfff640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc13406b4d0 0x7fc13406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.481+0000 7fc14bfff640 1 -- 172.21.15.94:0/3275631704 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc14408a370 con 0x7fc1540c88c0 2024-03-06T07:22:29.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.481+0000 7fc152ffd640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc13406b4d0 0x7fc13406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.482+0000 7fc152ffd640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc13406b4d0 0x7fc13406d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc13c034000 tx=0x7fc13c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.485+0000 7fc14bfff640 1 -- 172.21.15.94:0/3275631704 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc144059160 con 0x7fc1540c88c0 2024-03-06T07:22:29.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.658+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc15404f330 con 0x7fc1540c88c0 2024-03-06T07:22:29.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.659+0000 7fc14bfff640 1 -- 172.21.15.94:0/3275631704 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+17083 (secure 0 0 0) 0x7fc144058a60 con 0x7fc1540c88c0 2024-03-06T07:22:29.661 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:29.661 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":109,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:13.494390+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.827640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.579202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/4001418202":"2024-03-07T07:22:13.494369+0000","172.21.15.145:6834/4001418202":"2024-03-07T07:22:13.494369+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:29.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc13406b4d0 msgr2=0x7fc13406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc13406b4d0 0x7fc13406d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc13c034000 tx=0x7fc13c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:29.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1540c88c0 msgr2=0x7fc154154380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1540c88c0 0x7fc154154380 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc14400c830 tx=0x7fc14400ccf0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 shutdown_connections 2024-03-06T07:22:29.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc13406b4d0 0x7fc13406d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc154158b60 0x7fc1541ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc15414f360 0x7fc1541ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 --2- 172.21.15.94:0/3275631704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1540c88c0 0x7fc154154380 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 >> 172.21.15.94:0/3275631704 conn(0x7fc1540ac570 msgr2=0x7fc1540ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 shutdown_connections 2024-03-06T07:22:29.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.661+0000 7fc159ff7640 1 -- 172.21.15.94:0/3275631704 wait complete. 2024-03-06T07:22:29.674 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:22:29.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:29.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.773+0000 7f8634de9640 1 Processor -- start 2024-03-06T07:22:29.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.774+0000 7f8634de9640 1 -- start start 2024-03-06T07:22:29.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.774+0000 7f8634de9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f863014f360 0x7f8630155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.774+0000 7f8634de9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f86300c88c0 0x7f86300c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.774+0000 7f8634de9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c5880 0x7f8630158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.774+0000 7f8634de9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86300c8dd0 con 0x7f863014f360 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.774+0000 7f8634de9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86300c73d0 con 0x7f86300c88c0 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.774+0000 7f8634de9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86300be2e0 con 0x7f86300c5880 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f863014f360 0x7f8630155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f863014f360 0x7f8630155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40334/0 (socket says 172.21.15.94:40334) 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 -- 172.21.15.94:0/3938964483 learned_addr learned my addr 172.21.15.94:0/3938964483 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862ed9d640 1 --2- 172.21.15.94:0/3938964483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c5880 0x7f8630158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862dd9b640 1 --2- 172.21.15.94:0/3938964483 >> v2:172.21.15.145:3300/0 conn(0x7f86300c88c0 0x7f86300c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 -- 172.21.15.94:0/3938964483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c5880 msgr2=0x7f8630158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 --2- 172.21.15.94:0/3938964483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c5880 0x7f8630158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 -- 172.21.15.94:0/3938964483 >> v2:172.21.15.145:3300/0 conn(0x7f86300c88c0 msgr2=0x7f86300c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 --2- 172.21.15.94:0/3938964483 >> v2:172.21.15.145:3300/0 conn(0x7f86300c88c0 0x7f86300c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862e59c640 1 -- 172.21.15.94:0/3938964483 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300c7670 con 0x7f863014f360 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.775+0000 7f862dd9b640 1 --2- 172.21.15.94:0/3938964483 >> v2:172.21.15.145:3300/0 conn(0x7f86300c88c0 0x7f86300c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f862e59c640 1 --2- 172.21.15.94:0/3938964483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f360 0x7f8630155c10 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f86300c84c0 tx=0x7f862802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0f20bcc626b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f862d59a640 1 -- 172.21.15.94:0/3938964483 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862802c9e0 con 0x7f863014f360 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f862d59a640 1 -- 172.21.15.94:0/3938964483 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862802cb80 con 0x7f863014f360 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f862d59a640 1 -- 172.21.15.94:0/3938964483 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86280343f0 con 0x7f863014f360 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 -- 172.21.15.94:0/3938964483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f360 msgr2=0x7f8630155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 --2- 172.21.15.94:0/3938964483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f360 0x7f8630155c10 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f86300c84c0 tx=0x7f862802be40 comp rx=0 tx=0).stop 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 -- 172.21.15.94:0/3938964483 shutdown_connections 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 --2- 172.21.15.94:0/3938964483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c5880 0x7f8630158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 --2- 172.21.15.94:0/3938964483 >> v2:172.21.15.145:3300/0 conn(0x7f86300c88c0 0x7f86300c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 --2- 172.21.15.94:0/3938964483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f360 0x7f8630155c10 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 -- 172.21.15.94:0/3938964483 >> 172.21.15.94:0/3938964483 conn(0x7f86300ac570 msgr2=0x7f86300ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 -- 172.21.15.94:0/3938964483 shutdown_connections 2024-03-06T07:22:29.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.776+0000 7f8634de9640 1 -- 172.21.15.94:0/3938964483 wait complete. 2024-03-06T07:22:29.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 Processor -- start 2024-03-06T07:22:29.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 -- start start 2024-03-06T07:22:29.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86300c5880 0x7f86301580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c88c0 0x7f86301eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f863014f360 0x7f8630154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86301590a0 con 0x7f86300c5880 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86300c7c30 con 0x7f863014f360 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f8634de9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8630159220 con 0x7f86300c88c0 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f862e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86300c5880 0x7f86301580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f862e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86300c5880 0x7f86301580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40338/0 (socket says 172.21.15.94:40338) 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.778+0000 7f862e59c640 1 -- 172.21.15.94:0/482633251 learned_addr learned my addr 172.21.15.94:0/482633251 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862dd9b640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c88c0 0x7f86301eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862ed9d640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f863014f360 0x7f8630154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862e59c640 1 -- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c88c0 msgr2=0x7f86301eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862e59c640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c88c0 0x7f86301eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862e59c640 1 -- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f863014f360 msgr2=0x7f8630154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862e59c640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f863014f360 0x7f8630154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862e59c640 1 -- 172.21.15.94:0/482633251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86301eae90 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862dd9b640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c88c0 0x7f86301eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862ed9d640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f863014f360 0x7f8630154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f862e59c640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86300c5880 0x7f86301580c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8628009f40 tx=0x7f8628004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f8626ffd640 1 -- 172.21.15.94:0/482633251 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8628043020 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f8626ffd640 1 -- 172.21.15.94:0/482633251 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862803e450 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86301eb130 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f8626ffd640 1 -- 172.21.15.94:0/482633251 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8628009510 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.779+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8630154c40 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.780+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86300b05f0 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.781+0000 7f8626ffd640 1 -- 172.21.15.94:0/482633251 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8628046020 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.781+0000 7f8626ffd640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861006b4d0 0x7f861006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.781+0000 7f8626ffd640 1 -- 172.21.15.94:0/482633251 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f86280acbe0 con 0x7f86300c5880 2024-03-06T07:22:29.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.781+0000 7f862dd9b640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861006b4d0 0x7f861006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.782+0000 7f862dd9b640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861006b4d0 0x7f861006d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f86301ee250 tx=0x7f8618008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:29.790+0000 7f8626ffd640 1 -- 172.21.15.94:0/482633251 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86300b05f0 con 0x7f86300c5880 2024-03-06T07:22:30.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.005+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f863004f330 con 0x7f86300c5880 2024-03-06T07:22:30.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.006+0000 7f8626ffd640 1 -- 172.21.15.94:0/482633251 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4990 (secure 0 0 0) 0x7f862804d030 con 0x7f86300c5880 2024-03-06T07:22:30.008 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:30.008 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1342040049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1342040049},{"type":"v1","addr":"172.21.15.145:6837","nonce":1342040049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:22.199458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:30.008 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 98 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861006b4d0 msgr2=0x7f861006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861006b4d0 0x7f861006d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f86301ee250 tx=0x7f8618008040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86300c5880 msgr2=0x7f86301580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86300c5880 0x7f86301580c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8628009f40 tx=0x7f8628004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 shutdown_connections 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861006b4d0 0x7f861006d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f863014f360 0x7f8630154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86300c88c0 0x7f86301eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 --2- 172.21.15.94:0/482633251 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86300c5880 0x7f86301580c0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 >> 172.21.15.94:0/482633251 conn(0x7f86300ac570 msgr2=0x7f863014d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 shutdown_connections 2024-03-06T07:22:30.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.008+0000 7f8634de9640 1 -- 172.21.15.94:0/482633251 wait complete. 2024-03-06T07:22:30.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:30.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf2d316640 1 Processor -- start 2024-03-06T07:22:30.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf2d316640 1 -- start start 2024-03-06T07:22:30.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf2d316640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcf2806c720 0x7fcf28071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf2d316640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf2d316640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf28067340 con 0x7fcf28150920 2024-03-06T07:22:30.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf2d316640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf28072400 con 0x7fcf2806c720 2024-03-06T07:22:30.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf2d316640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf28150d00 con 0x7fcf280c2410 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.120+0000 7fcf267fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcf2806c720 0x7fcf28071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf267fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcf2806c720 0x7fcf28071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40340/0 (socket says 172.21.15.94:40340) 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf267fc640 1 -- 172.21.15.94:0/348223164 learned_addr learned my addr 172.21.15.94:0/348223164 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf277fe640 1 --2- 172.21.15.94:0/348223164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf267fc640 1 -- 172.21.15.94:0/348223164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 msgr2=0x7fcf28071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf267fc640 1 --2- 172.21.15.94:0/348223164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28071ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf267fc640 1 -- 172.21.15.94:0/348223164 >> v1:172.21.15.145:6789/0 conn(0x7fcf28150920 legacy=0x7fcf280c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf267fc640 1 -- 172.21.15.94:0/348223164 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf280c2010 con 0x7fcf2806c720 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.121+0000 7fcf277fe640 1 --2- 172.21.15.94:0/348223164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28071ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf267fc640 1 --2- 172.21.15.94:0/348223164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28071980 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009870 tx=0x7fcf1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1a46205b7e5a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf25ffb640 1 -- 172.21.15.94:0/348223164 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c035020 con 0x7fcf2806c720 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf25ffb640 1 -- 172.21.15.94:0/348223164 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c031640 con 0x7fcf2806c720 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf25ffb640 1 -- 172.21.15.94:0/348223164 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c02d040 con 0x7fcf2806c720 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 -- 172.21.15.94:0/348223164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 msgr2=0x7fcf28071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 --2- 172.21.15.94:0/348223164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28071980 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009870 tx=0x7fcf1c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 -- 172.21.15.94:0/348223164 shutdown_connections 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 --2- 172.21.15.94:0/348223164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28071ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 --2- 172.21.15.94:0/348223164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28071980 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 -- 172.21.15.94:0/348223164 >> 172.21.15.94:0/348223164 conn(0x7fcf280c70b0 msgr2=0x7fcf280c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 -- 172.21.15.94:0/348223164 shutdown_connections 2024-03-06T07:22:30.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.122+0000 7fcf2d316640 1 -- 172.21.15.94:0/348223164 wait complete. 2024-03-06T07:22:30.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.123+0000 7fcf2d316640 1 Processor -- start 2024-03-06T07:22:30.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf2d316640 1 -- start start 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf2d316640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28142450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf2d316640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf2d316640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf28150920 0x7fcf281ec6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf2d316640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf280c3040 con 0x7fcf2806c720 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf2d316640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf28072b70 con 0x7fcf28150920 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf2d316640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf280c31c0 con 0x7fcf280c2410 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf26ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf26ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28142450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40348/0 (socket says 172.21.15.94:40348) 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf26ffd640 1 -- 172.21.15.94:0/2592280583 learned_addr learned my addr 172.21.15.94:0/2592280583 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf277fe640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf28150920 0x7fcf281ec6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.124+0000 7fcf267fc640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf26ffd640 1 -- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 msgr2=0x7fcf28144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf26ffd640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf26ffd640 1 -- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf28150920 msgr2=0x7fcf281ec6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf26ffd640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf28150920 0x7fcf281ec6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf26ffd640 1 -- 172.21.15.94:0/2592280583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28157770 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf277fe640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf28150920 0x7fcf281ec6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf26ffd640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28142450 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c3c0 tx=0x7fcf1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf0f7fe640 1 -- 172.21.15.94:0/2592280583 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180052d0 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf0f7fe640 1 -- 172.21.15.94:0/2592280583 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18005490 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf28142e40 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf0f7fe640 1 -- 172.21.15.94:0/2592280583 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18007260 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.125+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf281e8d00 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.126+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf28154650 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.126+0000 7fcf0f7fe640 1 -- 172.21.15.94:0/2592280583 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf18007400 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.127+0000 7fcf0f7fe640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.127+0000 7fcf0f7fe640 1 -- 172.21.15.94:0/2592280583 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcf180897d0 con 0x7fcf2806c720 2024-03-06T07:22:30.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.127+0000 7fcf267fc640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.128+0000 7fcf267fc640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b4d0 0x7fcf1406d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c036000 tx=0x7fcf1c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.131+0000 7fcf0f7fe640 1 -- 172.21.15.94:0/2592280583 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf28157770 con 0x7fcf2806c720 2024-03-06T07:22:30.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.347+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf2804f330 con 0x7fcf2806c720 2024-03-06T07:22:30.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.347+0000 7fcf0f7fe640 1 -- 172.21.15.94:0/2592280583 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4990 (secure 0 0 0) 0x7fcf18057e20 con 0x7fcf2806c720 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1342040049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1342040049},{"type":"v1","addr":"172.21.15.145:6837","nonce":1342040049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":97,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-06T07:22:05.610686+0000","modified":"2024-03-06T07:22:22.199458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"d","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.145:6836/3485950735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3485950735},{"type":"v1","addr":"172.21.15.145:6836","nonce":3485950735}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7350":{"gid":7350,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":7,"addr":"172.21.15.94:6834/2226553926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2226553926},{"type":"v1","addr":"172.21.15.94:6834","nonce":2226553926}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7350,"qdb_cluster":[7350]},"id":8}]} 2024-03-06T07:22:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 98 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.349+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b4d0 msgr2=0x7fcf1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.349+0000 7fcf2d316640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b4d0 0x7fcf1406d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c036000 tx=0x7fcf1c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.349+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 msgr2=0x7fcf28142450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.349+0000 7fcf2d316640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28142450 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c3c0 tx=0x7fcf1800c880 comp rx=0 tx=0).stop 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 shutdown_connections 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf1406b4d0 0x7fcf1406d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf28150920 0x7fcf281ec6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf280c2410 0x7fcf28144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 --2- 172.21.15.94:0/2592280583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf2806c720 0x7fcf28142450 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 >> 172.21.15.94:0/2592280583 conn(0x7fcf280c70b0 msgr2=0x7fcf2806b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 shutdown_connections 2024-03-06T07:22:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.350+0000 7fcf2d316640 1 -- 172.21.15.94:0/2592280583 wait complete. 2024-03-06T07:22:30.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:30.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.461+0000 7f47f09ae640 1 Processor -- start 2024-03-06T07:22:30.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.461+0000 7f47f09ae640 1 -- start start 2024-03-06T07:22:30.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47f09ae640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47ec0724b0 0x7f47ec159090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47f09ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec072890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47f09ae640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f47ec0c67f0 0x7f47ec155090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47f09ae640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec0a8410 con 0x7f47ec0724b0 2024-03-06T07:22:30.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47f09ae640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec0a7e50 con 0x7f47ec0c67f0 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47f09ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec0a8130 con 0x7f47ec0c3940 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47ea59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47ec0724b0 0x7f47ec159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47ea59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47ec0724b0 0x7f47ec159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40354/0 (socket says 172.21.15.94:40354) 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47ea59c640 1 -- 172.21.15.94:0/842740676 learned_addr learned my addr 172.21.15.94:0/842740676 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.462+0000 7f47e9d9b640 1 --2- 172.21.15.94:0/842740676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47ead9d640 1 --2- 172.21.15.94:0/842740676 >> v2:172.21.15.145:3300/0 conn(0x7f47ec0c67f0 0x7f47ec155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47ea59c640 1 -- 172.21.15.94:0/842740676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 msgr2=0x7f47ec072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47ea59c640 1 --2- 172.21.15.94:0/842740676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec072890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47ea59c640 1 -- 172.21.15.94:0/842740676 >> v2:172.21.15.145:3300/0 conn(0x7f47ec0c67f0 msgr2=0x7f47ec155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47ea59c640 1 --2- 172.21.15.94:0/842740676 >> v2:172.21.15.145:3300/0 conn(0x7f47ec0c67f0 0x7f47ec155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47ea59c640 1 -- 172.21.15.94:0/842740676 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec0c3420 con 0x7f47ec0724b0 2024-03-06T07:22:30.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47ea59c640 1 --2- 172.21.15.94:0/842740676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec159090 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06e280 tx=0x7f47dc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8db0272c1ca614 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47e959a640 1 -- 172.21.15.94:0/842740676 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc038070 con 0x7f47ec0724b0 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47e959a640 1 -- 172.21.15.94:0/842740676 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc02f050 con 0x7f47ec0724b0 2024-03-06T07:22:30.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.463+0000 7f47e959a640 1 -- 172.21.15.94:0/842740676 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc031970 con 0x7f47ec0724b0 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 -- 172.21.15.94:0/842740676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 msgr2=0x7f47ec159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 --2- 172.21.15.94:0/842740676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec159090 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06e280 tx=0x7f47dc02b7f0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 -- 172.21.15.94:0/842740676 shutdown_connections 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 --2- 172.21.15.94:0/842740676 >> v2:172.21.15.145:3300/0 conn(0x7f47ec0c67f0 0x7f47ec155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 --2- 172.21.15.94:0/842740676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 --2- 172.21.15.94:0/842740676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec159090 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 -- 172.21.15.94:0/842740676 >> 172.21.15.94:0/842740676 conn(0x7f47ec0aa6a0 msgr2=0x7f47ec0aaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 -- 172.21.15.94:0/842740676 shutdown_connections 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.464+0000 7f47f09ae640 1 -- 172.21.15.94:0/842740676 wait complete. 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.465+0000 7f47f09ae640 1 Processor -- start 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.465+0000 7f47f09ae640 1 -- start start 2024-03-06T07:22:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47f09ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec1ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47f09ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47f09ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47ec0c67f0 0x7f47ec1572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47f09ae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec152ae0 con 0x7f47ec0724b0 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47f09ae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec152960 con 0x7f47ec0c67f0 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47f09ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec1555d0 con 0x7f47ec0c3940 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec1ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec1ee2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40362/0 (socket says 172.21.15.94:40362) 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 -- 172.21.15.94:0/2596484411 learned_addr learned my addr 172.21.15.94:0/2596484411 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ead9d640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47ec0c67f0 0x7f47ec1572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47e9d9b640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 -- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 msgr2=0x7f47ec156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 -- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47ec0c67f0 msgr2=0x7f47ec1572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47ec0c67f0 0x7f47ec1572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.466+0000 7f47ea59c640 1 -- 172.21.15.94:0/2596484411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec154c00 con 0x7f47ec0724b0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.467+0000 7f47ea59c640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec1ee2a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f47dc02b830 tx=0x7f47dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.467+0000 7f47daffd640 1 -- 172.21.15.94:0/2596484411 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc042070 con 0x7f47ec0724b0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.467+0000 7f47daffd640 1 -- 172.21.15.94:0/2596484411 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc03d890 con 0x7f47ec0724b0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.467+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec158880 con 0x7f47ec0724b0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.467+0000 7f47daffd640 1 -- 172.21.15.94:0/2596484411 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc038040 con 0x7f47ec0724b0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.467+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec0abb80 con 0x7f47ec0724b0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.468+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec075300 con 0x7f47ec0724b0 2024-03-06T07:22:30.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.468+0000 7f47daffd640 1 -- 172.21.15.94:0/2596484411 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47dc045070 con 0x7f47ec0724b0 2024-03-06T07:22:30.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.469+0000 7f47daffd640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47cc06b4d0 0x7f47cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.469+0000 7f47daffd640 1 -- 172.21.15.94:0/2596484411 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f47dc0abfa0 con 0x7f47ec0724b0 2024-03-06T07:22:30.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.469+0000 7f47e9d9b640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47cc06b4d0 0x7f47cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.469+0000 7f47e9d9b640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47cc06b4d0 0x7f47cc06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0c7e50 tx=0x7f47e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.473+0000 7f47daffd640 1 -- 172.21.15.94:0/2596484411 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47ec075300 con 0x7f47ec0724b0 2024-03-06T07:22:30.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.764+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47ec04f330 con 0x7f47ec0724b0 2024-03-06T07:22:30.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.765+0000 7f47daffd640 1 -- 172.21.15.94:0/2596484411 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+17083 (secure 0 0 0) 0x7f47dc07a680 con 0x7f47ec0724b0 2024-03-06T07:22:30.768 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:30.768 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":109,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:13.494390+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:03.827640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:04.579202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/4001418202":"2024-03-07T07:22:13.494369+0000","172.21.15.145:6834/4001418202":"2024-03-07T07:22:13.494369+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47cc06b4d0 msgr2=0x7f47cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47cc06b4d0 0x7f47cc06d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f47ec0c7e50 tx=0x7f47e0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 msgr2=0x7f47ec1ee2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec1ee2a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f47dc02b830 tx=0x7f47dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 shutdown_connections 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47cc06b4d0 0x7f47cc06d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47ec0c67f0 0x7f47ec1572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47ec0c3940 0x7f47ec156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 --2- 172.21.15.94:0/2596484411 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47ec0724b0 0x7f47ec1ee2a0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 >> 172.21.15.94:0/2596484411 conn(0x7f47ec0aa6a0 msgr2=0x7f47ec0c75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 shutdown_connections 2024-03-06T07:22:30.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.768+0000 7f47f09ae640 1 -- 172.21.15.94:0/2596484411 wait complete. 2024-03-06T07:22:30.781 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:22:30.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.880+0000 7f14a593e640 1 Processor -- start 2024-03-06T07:22:30.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.880+0000 7f14a593e640 1 -- start start 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f14a593e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14a006ff10 0x7f14a0153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f14a593e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c7170 0x7f14a0154160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f14a593e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a00a8390 con 0x7f14a00c6c40 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f14a593e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a00c7620 con 0x7f14a006ff10 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f14a593e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00730f0 con 0x7f14a00c7170 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14a006ff10 0x7f14a0153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f14a00c6c40 0x7f14a00c8810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50478/0 (socket says 172.21.15.94:50478) 2024-03-06T07:22:30.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149f7fe640 1 -- 172.21.15.94:0/943132388 learned_addr learned my addr 172.21.15.94:0/943132388 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149e7fc640 1 --2- 172.21.15.94:0/943132388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c7170 0x7f14a0154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149dffb640 1 -- 172.21.15.94:0/943132388 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222026831 0 0) 0x7f14a00a8390 con 0x7f14a00c6c40 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149dffb640 1 -- 172.21.15.94:0/943132388 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148c003520 con 0x7f14a00c6c40 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149effd640 1 -- 172.21.15.94:0/943132388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c7170 msgr2=0x7f14a0154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149effd640 1 --2- 172.21.15.94:0/943132388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c7170 0x7f14a0154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149effd640 1 -- 172.21.15.94:0/943132388 >> v1:172.21.15.94:6789/0 conn(0x7f14a00c6c40 legacy=0x7f14a00c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.881+0000 7f149effd640 1 -- 172.21.15.94:0/943132388 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a00733d0 con 0x7f14a006ff10 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.882+0000 7f149effd640 1 --2- 172.21.15.94:0/943132388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a006ff10 0x7f14a0153c20 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f14980097a0 tx=0x7f149802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58362b4fb7d3d635 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.882+0000 7f149dffb640 1 -- 172.21.15.94:0/943132388 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498038070 con 0x7f14a006ff10 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.882+0000 7f149dffb640 1 -- 172.21.15.94:0/943132388 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1498031700 con 0x7f14a006ff10 2024-03-06T07:22:30.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.882+0000 7f149dffb640 1 -- 172.21.15.94:0/943132388 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498031ab0 con 0x7f14a006ff10 2024-03-06T07:22:30.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 -- 172.21.15.94:0/943132388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a006ff10 msgr2=0x7f14a0153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 --2- 172.21.15.94:0/943132388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a006ff10 0x7f14a0153c20 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f14980097a0 tx=0x7f149802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 -- 172.21.15.94:0/943132388 shutdown_connections 2024-03-06T07:22:30.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 --2- 172.21.15.94:0/943132388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c7170 0x7f14a0154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 --2- 172.21.15.94:0/943132388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a006ff10 0x7f14a0153c20 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 -- 172.21.15.94:0/943132388 >> 172.21.15.94:0/943132388 conn(0x7f14a006b970 msgr2=0x7f14a00aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 -- 172.21.15.94:0/943132388 shutdown_connections 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.883+0000 7f14a593e640 1 -- 172.21.15.94:0/943132388 wait complete. 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.884+0000 7f14a593e640 1 Processor -- start 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.884+0000 7f14a593e640 1 -- start start 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f14a593e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a006ff10 0x7f14a00c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f14a593e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c6c40 0x7f14a01eab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f14a593e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a00c7170 0x7f14a01eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f14a593e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0070470 con 0x7f14a00c7170 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f14a593e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00702f0 con 0x7f14a006ff10 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f14a593e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00c6860 con 0x7f14a00c6c40 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a00c7170 0x7f14a01eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a00c7170 0x7f14a01eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40372/0 (socket says 172.21.15.94:40372) 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f149f7fe640 1 -- 172.21.15.94:0/3474787109 learned_addr learned my addr 172.21.15.94:0/3474787109 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:30.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f149e7fc640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c6c40 0x7f14a01eab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.885+0000 7f149effd640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a006ff10 0x7f14a00c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f149f7fe640 1 -- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c6c40 msgr2=0x7f14a01eab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f149f7fe640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c6c40 0x7f14a01eab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f149f7fe640 1 -- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a006ff10 msgr2=0x7f14a00c1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f149f7fe640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a006ff10 0x7f14a00c1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f149f7fe640 1 -- 172.21.15.94:0/3474787109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a01eec60 con 0x7f14a00c7170 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f149e7fc640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c6c40 0x7f14a01eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f149f7fe640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a00c7170 0x7f14a01eb080 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f149000cb70 tx=0x7f149000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f1497fff640 1 -- 172.21.15.94:0/3474787109 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490017070 con 0x7f14a00c7170 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f1497fff640 1 -- 172.21.15.94:0/3474787109 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149000ad40 con 0x7f14a00c7170 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a009af20 con 0x7f14a00c7170 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f1497fff640 1 -- 172.21.15.94:0/3474787109 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490012620 con 0x7f14a00c7170 2024-03-06T07:22:30.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.886+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a01eef00 con 0x7f14a00c7170 2024-03-06T07:22:30.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.887+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a004f330 con 0x7f14a00c7170 2024-03-06T07:22:30.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.888+0000 7f1497fff640 1 -- 172.21.15.94:0/3474787109 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f149001a070 con 0x7f14a00c7170 2024-03-06T07:22:30.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.888+0000 7f1497fff640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.888+0000 7f1497fff640 1 -- 172.21.15.94:0/3474787109 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f149008a120 con 0x7f14a00c7170 2024-03-06T07:22:30.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.888+0000 7f149effd640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.889+0000 7f149effd640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f148006b4d0 0x7f148006d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1498035000 tx=0x7f1498008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:30.897+0000 7f1497fff640 1 -- 172.21.15.94:0/3474787109 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1490058800 con 0x7f14a00c7170 2024-03-06T07:22:31.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.110+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f14a006fa80 con 0x7f14a00c7170 2024-03-06T07:22:31.113 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:22:31.111+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:31.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.112+0000 7f1497fff640 1 -- 172.21.15.94:0/3474787109 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v99) v1 ==== 161+0+0 (secure 0 0 0) 0x7f149005c8a0 con 0x7f14a00c7170 2024-03-06T07:22:31.114 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f148006b4d0 msgr2=0x7f148006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f148006b4d0 0x7f148006d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1498035000 tx=0x7f1498008030 comp rx=0 tx=0).stop 2024-03-06T07:22:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a00c7170 msgr2=0x7f14a01eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a00c7170 0x7f14a01eb080 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f149000cb70 tx=0x7f149000a430 comp rx=0 tx=0).stop 2024-03-06T07:22:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 shutdown_connections 2024-03-06T07:22:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a00c7170 0x7f14a01eb080 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14a00c6c40 0x7f14a01eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 --2- 172.21.15.94:0/3474787109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14a006ff10 0x7f14a00c1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 >> 172.21.15.94:0/3474787109 conn(0x7f14a006b970 msgr2=0x7f14a00c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 shutdown_connections 2024-03-06T07:22:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.115+0000 7f14a593e640 1 -- 172.21.15.94:0/3474787109 wait complete. 2024-03-06T07:22:31.119 INFO:tasks.ceph.mds.a.smithi094.stderr: -596> 2024-03-06T07:22:16.532+0000 7f1621a01640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:31.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:22:31.143 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:22:31.144 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:22:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.233+0000 7f66aaad4640 1 Processor -- start 2024-03-06T07:22:31.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.233+0000 7f66aaad4640 1 -- start start 2024-03-06T07:22:31.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66aaad4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 0x7f66a4073090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66aaad4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f66a40c69a0 0x7f66a4158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66aaad4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a4159580 con 0x7f66a4158bd0 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66aaad4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a40c3460 con 0x7f66a40c69a0 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66aaad4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40c2050 con 0x7f66a406d840 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66a9071640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f66a4158bd0 0x7f66a40c24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50490/0 (socket says 172.21.15.94:50490) 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66a9071640 1 -- 172.21.15.94:0/4237143329 learned_addr learned my addr 172.21.15.94:0/4237143329 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66a3fff640 1 --2- 172.21.15.94:0/4237143329 >> v2:172.21.15.145:3300/0 conn(0x7f66a40c69a0 0x7f66a4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131000868 0 0) 0x7f66a4159580 con 0x7f66a4158bd0 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.234+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6690003520 con 0x7f66a4158bd0 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a8870640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 0x7f66a4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89670649 0 0) 0x7f6690003520 con 0x7f66a4158bd0 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 msgr2=0x7f66a4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 0x7f66a4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 >> v2:172.21.15.145:3300/0 conn(0x7f66a40c69a0 msgr2=0x7f66a4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 --2- 172.21.15.94:0/4237143329 >> v2:172.21.15.145:3300/0 conn(0x7f66a40c69a0 0x7f66a4158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40c36b0 con 0x7f66a4158bd0 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f669c001670 con 0x7f66a4158bd0 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 >> v1:172.21.15.94:6789/0 conn(0x7f66a4158bd0 legacy=0x7f66a40c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 0x7f66900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6690003f80 0x7f6690004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40c3460 con 0x7f6690004d40 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40c2050 con 0x7f6690003f80 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669000baf0 con 0x7f66a406d840 2024-03-06T07:22:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a9071640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a8870640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 0x7f66900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a3fff640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6690003f80 0x7f6690004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a9071640 1 -- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 msgr2=0x7f66900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a9071640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a406d840 0x7f66900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a9071640 1 -- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6690003f80 msgr2=0x7f6690004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a9071640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6690003f80 0x7f6690004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.235+0000 7f66a9071640 1 -- 172.21.15.94:0/4237143329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6690003520 con 0x7f6690004d40 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.236+0000 7f66a9071640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66900051c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f669c001050 tx=0x7f6690005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886e76fbf6d3b055 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.236+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c007ca0 con 0x7f6690004d40 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.236+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c030070 con 0x7f6690004d40 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.236+0000 7f66a37fe640 1 -- 172.21.15.94:0/4237143329 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c005d10 con 0x7f6690004d40 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.237+0000 7f66aaad4640 1 -- 172.21.15.94:0/4237143329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 msgr2=0x7f66900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.237+0000 7f66aaad4640 1 --2- 172.21.15.94:0/4237143329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66900051c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f669c001050 tx=0x7f6690005830 comp rx=0 tx=0).stop 2024-03-06T07:22:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.237+0000 7f66a9071640 1 -- 172.21.15.94:0/4237143329 reap_dead start 2024-03-06T07:22:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.237+0000 7f66aaad4640 1 -- 172.21.15.94:0/4237143329 shutdown_connections 2024-03-06T07:22:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.237+0000 7f66aaad4640 1 -- 172.21.15.94:0/4237143329 >> 172.21.15.94:0/4237143329 conn(0x7f66a40c5560 msgr2=0x7f66a40c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.237+0000 7f66aaad4640 1 -- 172.21.15.94:0/4237143329 shutdown_connections 2024-03-06T07:22:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.237+0000 7f66aaad4640 1 -- 172.21.15.94:0/4237143329 wait complete. 2024-03-06T07:22:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.238+0000 7f66aaad4640 1 Processor -- start 2024-03-06T07:22:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.238+0000 7f66aaad4640 1 -- start start 2024-03-06T07:22:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66a40ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66a4155d20 0x7f66a406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a4153b70 0x7f66a406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a406dc20 con 0x7f6690004d40 2024-03-06T07:22:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40c3c70 con 0x7f66a4155d20 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a406dda0 con 0x7f66a4153b70 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66a40ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66a40ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40390/0 (socket says 172.21.15.94:40390) 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 -- 172.21.15.94:0/1636272470 learned_addr learned my addr 172.21.15.94:0/1636272470 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a3fff640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66a4155d20 0x7f66a406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a9071640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a4153b70 0x7f66a406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 -- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a4153b70 msgr2=0x7f66a406f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a4153b70 0x7f66a406f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 -- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66a4155d20 msgr2=0x7f66a406ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66a4155d20 0x7f66a406ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 -- 172.21.15.94:0/1636272470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40adaf0 con 0x7f6690004d40 2024-03-06T07:22:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a3fff640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66a4155d20 0x7f66a406ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a9071640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a4153b70 0x7f66a406f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a8870640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66a40ae400 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f668c011a00 tx=0x7f668c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a17fa640 1 -- 172.21.15.94:0/1636272470 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c00fa90 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a17fa640 1 -- 172.21.15.94:0/1636272470 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c00fc30 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a414c540 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66a17fa640 1 -- 172.21.15.94:0/1636272470 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c014440 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.239+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a40ae940 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.240+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a404f330 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.241+0000 7f66a17fa640 1 -- 172.21.15.94:0/1636272470 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f668c012040 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.241+0000 7f66a17fa640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f668406b4d0 0x7f668406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.241+0000 7f66a17fa640 1 -- 172.21.15.94:0/1636272470 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f668c08aa10 con 0x7f6690004d40 2024-03-06T07:22:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.241+0000 7f66a3fff640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f668406b4d0 0x7f668406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.242+0000 7f66a3fff640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f668406b4d0 0x7f668406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f66a40ac8d0 tx=0x7f669400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.245+0000 7f66a17fa640 1 -- 172.21.15.94:0/1636272470 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f668c019030 con 0x7f6690004d40 2024-03-06T07:22:31.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:31.452+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f66a4074700 con 0x7f6690004d40 2024-03-06T07:22:32.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.114+0000 7f66a17fa640 1 -- 172.21.15.94:0/1636272470 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v100) v1 ==== 104+0+0 (secure 0 0 0) 0x7f668c0596e0 con 0x7f6690004d40 2024-03-06T07:22:32.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f668406b4d0 msgr2=0x7f668406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f668406b4d0 0x7f668406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f66a40ac8d0 tx=0x7f669400b040 comp rx=0 tx=0).stop 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 msgr2=0x7f66a40ae400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66a40ae400 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f668c011a00 tx=0x7f668c011ec0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 shutdown_connections 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f668406b4d0 0x7f668406d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66a4153b70 0x7f66a406f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66a4155d20 0x7f66a406ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 --2- 172.21.15.94:0/1636272470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6690004d40 0x7f66a40ae400 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 >> 172.21.15.94:0/1636272470 conn(0x7f66a40c5560 msgr2=0x7f66a4072580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 shutdown_connections 2024-03-06T07:22:32.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.117+0000 7f66aaad4640 1 -- 172.21.15.94:0/1636272470 wait complete. 2024-03-06T07:22:32.133 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:32.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.285+0000 7f900831e640 1 Processor -- start 2024-03-06T07:22:32.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.285+0000 7f900831e640 1 -- start start 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.285+0000 7f900831e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 0x7f90001521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f900831e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900014ef90 con 0x7f90000c28d0 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f900831e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9000155780 con 0x7f90000a9980 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f900831e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9000155530 con 0x7f900014ebb0 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90068bb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f90000c28d0 0x7f900006dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50494/0 (socket says 172.21.15.94:50494) 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90068bb640 1 -- 172.21.15.94:0/75297541 learned_addr learned my addr 172.21.15.94:0/75297541 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90060ba640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 0x7f90001521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69296845 0 0) 0x7f8ff8001490 con 0x7f90000c28d0 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fec003520 con 0x7f90000c28d0 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385324740 0 0) 0x7f9000155780 con 0x7f90000a9980 2024-03-06T07:22:32.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8001490 con 0x7f90000a9980 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825908837 0 0) 0x7f8fec003520 con 0x7f90000c28d0 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 msgr2=0x7f90001521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 0x7f90001521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 >> v1:172.21.15.145:6789/0 conn(0x7f90000a9980 legacy=0x7f90000a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90001516d0 con 0x7f90000c28d0 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f8ff8001bc0 con 0x7f90000c28d0 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 >> v1:172.21.15.94:6789/0 conn(0x7f90000c28d0 legacy=0x7f900006dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fec003b70 0x7f8fec003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec004800 0x7f8fec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 0x7f8fec005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9000155530 con 0x7f8fec003b70 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec00dd60 con 0x7f8fec004800 2024-03-06T07:22:32.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec00df30 con 0x7f900014ebb0 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.286+0000 7f90060ba640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fec003b70 0x7f8fec003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90058b9640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec004800 0x7f8fec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90068bb640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 0x7f8fec005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90060ba640 1 -- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 msgr2=0x7f8fec005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90060ba640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f900014ebb0 0x7f8fec005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90060ba640 1 -- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec004800 msgr2=0x7f8fec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90060ba640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec004800 0x7f8fec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90060ba640 1 -- 172.21.15.94:0/75297541 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec003520 con 0x7f8fec003b70 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90058b9640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec004800 0x7f8fec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90060ba640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fec003b70 0x7f8fec003f70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009af0 tx=0x7f8fec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec3fa38adb2e748 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.287+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc039070 con 0x7f8fec003b70 2024-03-06T07:22:32.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc02c6b0 con 0x7f8fec003b70 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f90050b8640 1 -- 172.21.15.94:0/75297541 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc009690 con 0x7f8fec003b70 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f900831e640 1 -- 172.21.15.94:0/75297541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fec003b70 msgr2=0x7f8fec003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f900831e640 1 --2- 172.21.15.94:0/75297541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fec003b70 0x7f8fec003f70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009af0 tx=0x7f8fec005830 comp rx=0 tx=0).stop 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f90068bb640 1 -- 172.21.15.94:0/75297541 reap_dead start 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f900831e640 1 -- 172.21.15.94:0/75297541 shutdown_connections 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f900831e640 1 -- 172.21.15.94:0/75297541 >> 172.21.15.94:0/75297541 conn(0x7f90000c65b0 msgr2=0x7f90000c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f900831e640 1 -- 172.21.15.94:0/75297541 shutdown_connections 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.288+0000 7f900831e640 1 -- 172.21.15.94:0/75297541 wait complete. 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.289+0000 7f900831e640 1 Processor -- start 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.289+0000 7f900831e640 1 -- start start 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f900831e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec003b70 0x7f90001e81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f900831e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90001eacb0 0x7f90000c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f900831e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90001e7520 0x7f90001495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f900831e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9000151c90 con 0x7f90001e7520 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f900831e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9000155a60 con 0x7f8fec003b70 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f900831e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90000c2d80 con 0x7f90001eacb0 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90001e7520 0x7f90001495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90001e7520 0x7f90001495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40396/0 (socket says 172.21.15.94:40396) 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 -- 172.21.15.94:0/2612603021 learned_addr learned my addr 172.21.15.94:0/2612603021 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90060ba640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec003b70 0x7f90001e81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90058b9640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90001eacb0 0x7f90000c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 -- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90001eacb0 msgr2=0x7f90000c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90001eacb0 0x7f90000c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 -- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec003b70 msgr2=0x7f90001e81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec003b70 0x7f90001e81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.290+0000 7f90068bb640 1 -- 172.21.15.94:0/2612603021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9000072460 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.291+0000 7f90068bb640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90001e7520 0x7f90001495d0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f900006d2e0 tx=0x7f8ff800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.291+0000 7f8ff6ffd640 1 -- 172.21.15.94:0/2612603021 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8007a80 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.291+0000 7f8ff6ffd640 1 -- 172.21.15.94:0/2612603021 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff800ed40 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.291+0000 7f8ff6ffd640 1 -- 172.21.15.94:0/2612603021 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8014670 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.291+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9000072740 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.291+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90001e7000 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.292+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9000155190 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.292+0000 7f8ff6ffd640 1 -- 172.21.15.94:0/2612603021 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ff8014810 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.292+0000 7f8ff6ffd640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.293+0000 7f8ff6ffd640 1 -- 172.21.15.94:0/2612603021 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8ff808a1b0 con 0x7f90001e7520 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.293+0000 7f90060ba640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.293+0000 7f90060ba640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe406b4d0 0x7f8fe406d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc036000 tx=0x7f8ffc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.297+0000 7f8ff6ffd640 1 -- 172.21.15.94:0/2612603021 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9000155190 con 0x7f90001e7520 2024-03-06T07:22:32.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:32.469+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9000070260 con 0x7f90001e7520 2024-03-06T07:22:33.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.229+0000 7f8ff6ffd640 1 -- 172.21.15.94:0/2612603021 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v111) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9000070260 con 0x7f90001e7520 2024-03-06T07:22:33.231 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.232+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe406b4d0 msgr2=0x7f8fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.232+0000 7f900831e640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe406b4d0 0x7f8fe406d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc036000 tx=0x7f8ffc030000 comp rx=0 tx=0).stop 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.232+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90001e7520 msgr2=0x7f90001495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.232+0000 7f900831e640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90001e7520 0x7f90001495d0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f900006d2e0 tx=0x7f8ff800e530 comp rx=0 tx=0).stop 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 shutdown_connections 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe406b4d0 0x7f8fe406d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f90001e7520 0x7f90001495d0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90001eacb0 0x7f90000c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 --2- 172.21.15.94:0/2612603021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fec003b70 0x7f90001e81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 >> 172.21.15.94:0/2612603021 conn(0x7f90000c65b0 msgr2=0x7f90000ab580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 shutdown_connections 2024-03-06T07:22:33.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.233+0000 7f900831e640 1 -- 172.21.15.94:0/2612603021 wait complete. 2024-03-06T07:22:33.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:22:33.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.353+0000 7fb872d93640 1 Processor -- start 2024-03-06T07:22:33.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.353+0000 7fb872d93640 1 -- start start 2024-03-06T07:22:33.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb872d93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 0x7fb86c151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb872d93640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c06e850 con 0x7fb86c153390 2024-03-06T07:22:33.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb872d93640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c06d2b0 con 0x7fb86c0a9830 2024-03-06T07:22:33.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb872d93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c06d590 con 0x7fb86c0c22b0 2024-03-06T07:22:33.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb871330640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb86c153390 0x7fb86c153820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50508/0 (socket says 172.21.15.94:50508) 2024-03-06T07:22:33.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb871330640 1 -- 172.21.15.94:0/1720285932 learned_addr learned my addr 172.21.15.94:0/1720285932 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:33.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb870b2f640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 0x7fb86c151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995709828 0 0) 0x7fb864001490 con 0x7fb86c153390 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb854003520 con 0x7fb86c153390 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727613346 0 0) 0x7fb854003520 con 0x7fb86c153390 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 msgr2=0x7fb86c151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb86b7fe640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 0x7fb86c151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.354+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 >> v1:172.21.15.145:6789/0 conn(0x7fb86c0a9830 legacy=0x7fb86c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c06d870 con 0x7fb86c153390 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fb864001bc0 con 0x7fb86c153390 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 >> v1:172.21.15.94:6789/0 conn(0x7fb86c153390 legacy=0x7fb86c153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 0x7fb8540039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb854003f20 0x7fb854004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb854004d40 0x7fb8540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c06d590 con 0x7fb854004d40 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85400ba20 con 0x7fb854003f20 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85400bc20 con 0x7fb86c0c22b0 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb871330640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb854004d40 0x7fb8540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb870b2f640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 0x7fb8540039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb86bfff640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb854003f20 0x7fb854004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb871330640 1 -- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 msgr2=0x7fb8540039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb871330640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c0c22b0 0x7fb8540039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb871330640 1 -- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb854003f20 msgr2=0x7fb854004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb871330640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb854003f20 0x7fb854004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.355+0000 7fb871330640 1 -- 172.21.15.94:0/1720285932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854003520 con 0x7fb854004d40 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb871330640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb854004d40 0x7fb8540051c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fb86400aa30 tx=0x7fb854005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c3c988f0ba1606 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86400a650 con 0x7fb854004d40 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb864034630 con 0x7fb854004d40 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb86b7fe640 1 -- 172.21.15.94:0/1720285932 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864034a20 con 0x7fb854004d40 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb872d93640 1 -- 172.21.15.94:0/1720285932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb854004d40 msgr2=0x7fb8540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb872d93640 1 --2- 172.21.15.94:0/1720285932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb854004d40 0x7fb8540051c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fb86400aa30 tx=0x7fb854005830 comp rx=0 tx=0).stop 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb871330640 1 -- 172.21.15.94:0/1720285932 reap_dead start 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb872d93640 1 -- 172.21.15.94:0/1720285932 shutdown_connections 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb872d93640 1 -- 172.21.15.94:0/1720285932 >> 172.21.15.94:0/1720285932 conn(0x7fb86c0c65b0 msgr2=0x7fb86c0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb872d93640 1 -- 172.21.15.94:0/1720285932 shutdown_connections 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.356+0000 7fb872d93640 1 -- 172.21.15.94:0/1720285932 wait complete. 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.357+0000 7fb872d93640 1 Processor -- start 2024-03-06T07:22:33.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb872d93640 1 -- start start 2024-03-06T07:22:33.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb872d93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c1de730 0x7fb86c1ec450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb872d93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb86c0aa980 0x7fb86c151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb872d93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb86c151990 0x7fb86c0c0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb872d93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c153f30 con 0x7fb86c0aa980 2024-03-06T07:22:33.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb872d93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c14f660 con 0x7fb86c151990 2024-03-06T07:22:33.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb872d93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c1ec990 con 0x7fb86c1de730 2024-03-06T07:22:33.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb86bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb86c0aa980 0x7fb86c151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb86bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb86c0aa980 0x7fb86c151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40422/0 (socket says 172.21.15.94:40422) 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.358+0000 7fb86bfff640 1 -- 172.21.15.94:0/822080564 learned_addr learned my addr 172.21.15.94:0/822080564 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb871330640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb86c151990 0x7fb86c0c0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb870b2f640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c1de730 0x7fb86c1ec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb86bfff640 1 -- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c1de730 msgr2=0x7fb86c1ec450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb86bfff640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c1de730 0x7fb86c1ec450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb86bfff640 1 -- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb86c151990 msgr2=0x7fb86c0c0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb86bfff640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb86c151990 0x7fb86c0c0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb86bfff640 1 -- 172.21.15.94:0/822080564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c150990 con 0x7fb86c0aa980 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb871330640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb86c151990 0x7fb86c0c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb86bfff640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb86c0aa980 0x7fb86c151450 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb86000f7c0 tx=0x7fb86000fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb8697fa640 1 -- 172.21.15.94:0/822080564 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8600052e0 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb8697fa640 1 -- 172.21.15.94:0/822080564 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb860018070 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb8697fa640 1 -- 172.21.15.94:0/822080564 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86000d970 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c146c10 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.359+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c150c50 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.360+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c04f330 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.361+0000 7fb8697fa640 1 -- 172.21.15.94:0/822080564 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb86000db10 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.361+0000 7fb8697fa640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb85806b500 0x7fb85806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.361+0000 7fb8697fa640 1 -- 172.21.15.94:0/822080564 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5835+0+0 (secure 0 0 0) 0x7fb86008a180 con 0x7fb86c0aa980 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.361+0000 7fb870b2f640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb85806b500 0x7fb85806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.362+0000 7fb870b2f640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb85806b500 0x7fb85806d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009b00 tx=0x7fb85c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.365+0000 7fb8697fa640 1 -- 172.21.15.94:0/822080564 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb86001a310 con 0x7fb86c0aa980 2024-03-06T07:22:33.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:33.543+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb86c1562a0 con 0x7fb86c0aa980 2024-03-06T07:22:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.237+0000 7fb8697fa640 1 -- 172.21.15.94:0/822080564 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v112) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb860059010 con 0x7fb86c0aa980 2024-03-06T07:22:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:22:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.240+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb85806b500 msgr2=0x7fb85806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.240+0000 7fb872d93640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb85806b500 0x7fb85806d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009b00 tx=0x7fb85c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.240+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb86c0aa980 msgr2=0x7fb86c151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.240+0000 7fb872d93640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb86c0aa980 0x7fb86c151450 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb86000f7c0 tx=0x7fb86000fc80 comp rx=0 tx=0).stop 2024-03-06T07:22:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 shutdown_connections 2024-03-06T07:22:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb85806b500 0x7fb85806d9c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb86c151990 0x7fb86c0c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb86c0aa980 0x7fb86c151450 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 --2- 172.21.15.94:0/822080564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb86c1de730 0x7fb86c1ec450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 >> 172.21.15.94:0/822080564 conn(0x7fb86c0c65b0 msgr2=0x7fb86c155390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 shutdown_connections 2024-03-06T07:22:34.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.241+0000 7fb872d93640 1 -- 172.21.15.94:0/822080564 wait complete. 2024-03-06T07:22:34.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.359+0000 7fd14035e640 1 Processor -- start 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.359+0000 7fd14035e640 1 -- start start 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd14035e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 0x7fd1380c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd14035e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380c3ef0 con 0x7fd1380ab170 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd14035e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1381556d0 con 0x7fd13806c900 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd14035e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1381559b0 con 0x7fd1380c5990 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd13d8f9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd1380ab170 0x7fd13806c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50514/0 (socket says 172.21.15.94:50514) 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd13d8f9640 1 -- 172.21.15.94:0/1109252197 learned_addr learned my addr 172.21.15.94:0/1109252197 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488799899 0 0) 0x7fd1380c3ef0 con 0x7fd1380ab170 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd124003520 con 0x7fd1380ab170 2024-03-06T07:22:34.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.360+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 0x7fd1380c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 675818079 0 0) 0x7fd124003520 con 0x7fd1380ab170 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 msgr2=0x7fd1380c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 0x7fd1380c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 >> v1:172.21.15.145:6789/0 conn(0x7fd13806c900 legacy=0x7fd1380a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd138155c90 con 0x7fd1380ab170 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fd128001670 con 0x7fd1380ab170 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 >> v1:172.21.15.94:6789/0 conn(0x7fd1380ab170 legacy=0x7fd13806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1240039e0 0x7fd124003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd124004800 0x7fd124004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 0x7fd1240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1381559b0 con 0x7fd1240039e0 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12400ba70 con 0x7fd124004800 2024-03-06T07:22:34.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12400de80 con 0x7fd1380c5990 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1240039e0 0x7fd124003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13d8f9640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd124004800 0x7fd124004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13e8fb640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 0x7fd1240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13e0fa640 1 -- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 msgr2=0x7fd1240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1380c5990 0x7fd1240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13e0fa640 1 -- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd124004800 msgr2=0x7fd124004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd124004800 0x7fd124004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.361+0000 7fd13e0fa640 1 -- 172.21.15.94:0/1109252197 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124003520 con 0x7fd1240039e0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.362+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1240039e0 0x7fd124003e00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd134009a90 tx=0x7fd124005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6164a56aa7befc6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.362+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134039070 con 0x7fd1240039e0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.362+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13402c680 con 0x7fd1240039e0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.362+0000 7fd13d0f8640 1 -- 172.21.15.94:0/1109252197 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134009690 con 0x7fd1240039e0 2024-03-06T07:22:34.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.362+0000 7fd14035e640 1 -- 172.21.15.94:0/1109252197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1240039e0 msgr2=0x7fd124003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.362+0000 7fd14035e640 1 --2- 172.21.15.94:0/1109252197 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1240039e0 0x7fd124003e00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd134009a90 tx=0x7fd124005830 comp rx=0 tx=0).stop 2024-03-06T07:22:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.362+0000 7fd13e8fb640 1 -- 172.21.15.94:0/1109252197 reap_dead start 2024-03-06T07:22:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.363+0000 7fd14035e640 1 -- 172.21.15.94:0/1109252197 shutdown_connections 2024-03-06T07:22:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.363+0000 7fd14035e640 1 -- 172.21.15.94:0/1109252197 >> 172.21.15.94:0/1109252197 conn(0x7fd1380c6710 msgr2=0x7fd1380c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.363+0000 7fd14035e640 1 -- 172.21.15.94:0/1109252197 shutdown_connections 2024-03-06T07:22:34.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.363+0000 7fd14035e640 1 -- 172.21.15.94:0/1109252197 wait complete. 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 Processor -- start 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 -- start start 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1240039e0 0x7fd1380a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1381449a0 0x7fd138144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1381eb1c0 0x7fd1381eb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd138156610 con 0x7fd1381eb1c0 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd138156490 con 0x7fd1381449a0 2024-03-06T07:22:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd14035e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380c3af0 con 0x7fd1240039e0 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd13e8fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1381eb1c0 0x7fd1381eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd13e8fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1381eb1c0 0x7fd1381eb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40438/0 (socket says 172.21.15.94:40438) 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.364+0000 7fd13e8fb640 1 -- 172.21.15.94:0/330025828 learned_addr learned my addr 172.21.15.94:0/330025828 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13d8f9640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1381449a0 0x7fd138144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1240039e0 0x7fd1380a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e8fb640 1 -- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1240039e0 msgr2=0x7fd1380a7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e8fb640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1240039e0 0x7fd1380a7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e8fb640 1 -- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1381449a0 msgr2=0x7fd138144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e8fb640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1381449a0 0x7fd138144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e8fb640 1 -- 172.21.15.94:0/330025828 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13814a9e0 con 0x7fd1381eb1c0 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1240039e0 0x7fd1380a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd13e8fb640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1381eb1c0 0x7fd1381eb640 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd1380c12a0 tx=0x7fd13000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd12effd640 1 -- 172.21.15.94:0/330025828 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130003970 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd12effd640 1 -- 172.21.15.94:0/330025828 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd130018070 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd12effd640 1 -- 172.21.15.94:0/330025828 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130003d50 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1381eacc0 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.365+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1380c0070 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.366+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1381eae90 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.367+0000 7fd12effd640 1 -- 172.21.15.94:0/330025828 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1300050d0 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.367+0000 7fd12effd640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd11c06b4d0 0x7fd11c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.367+0000 7fd12effd640 1 -- 172.21.15.94:0/330025828 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd130089d10 con 0x7fd1381eb1c0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.367+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd11c06b4d0 0x7fd11c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.368+0000 7fd13e0fa640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd11c06b4d0 0x7fd11c06d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd134036000 tx=0x7fd134030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.371+0000 7fd12effd640 1 -- 172.21.15.94:0/330025828 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd130058d40 con 0x7fd1381eb1c0 2024-03-06T07:22:34.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.543+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd13804f330 con 0x7fd1381eb1c0 2024-03-06T07:22:34.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.546+0000 7fd12effd640 1 -- 172.21.15.94:0/330025828 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd13001be40 con 0x7fd1381eb1c0 2024-03-06T07:22:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd11c06b4d0 msgr2=0x7fd11c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd11c06b4d0 0x7fd11c06d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd134036000 tx=0x7fd134030000 comp rx=0 tx=0).stop 2024-03-06T07:22:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1381eb1c0 msgr2=0x7fd1381eb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1381eb1c0 0x7fd1381eb640 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd1380c12a0 tx=0x7fd13000ec10 comp rx=0 tx=0).stop 2024-03-06T07:22:34.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 shutdown_connections 2024-03-06T07:22:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd11c06b4d0 0x7fd11c06d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd1381eb1c0 0x7fd1381eb640 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd1381449a0 0x7fd138144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 --2- 172.21.15.94:0/330025828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd1240039e0 0x7fd1380a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 >> 172.21.15.94:0/330025828 conn(0x7fd1380c6710 msgr2=0x7fd138149680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 shutdown_connections 2024-03-06T07:22:34.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.548+0000 7fd14035e640 1 -- 172.21.15.94:0/330025828 wait complete. 2024-03-06T07:22:34.560 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:22:34.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.662+0000 7f3b2ab79640 1 Processor -- start 2024-03-06T07:22:34.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b2ab79640 1 -- start start 2024-03-06T07:22:34.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b2ab79640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b24155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b2ab79640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3b2406ca70 0x7f3b240c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b2ab79640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b240c5380 0x7f3b240c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b2ab79640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2406c020 con 0x7f3b240c5380 2024-03-06T07:22:34.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b2ab79640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240b0450 con 0x7f3b2406ca70 2024-03-06T07:22:34.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b2ab79640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240c77f0 con 0x7f3b240b0070 2024-03-06T07:22:34.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b29116640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b240c5380 0x7f3b240c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b29116640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3b240c5380 0x7f3b240c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40450/0 (socket says 172.21.15.94:40450) 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.663+0000 7f3b29116640 1 -- 172.21.15.94:0/502992183 learned_addr learned my addr 172.21.15.94:0/502992183 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b29116640 1 -- 172.21.15.94:0/502992183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 msgr2=0x7f3b24155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b28915640 1 --2- 172.21.15.94:0/502992183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b24155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b23fff640 1 --2- 172.21.15.94:0/502992183 >> v2:172.21.15.145:3300/0 conn(0x7f3b2406ca70 0x7f3b240c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b29116640 1 --2- 172.21.15.94:0/502992183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b24155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b29116640 1 -- 172.21.15.94:0/502992183 >> v2:172.21.15.145:3300/0 conn(0x7f3b2406ca70 msgr2=0x7f3b240c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b29116640 1 --2- 172.21.15.94:0/502992183 >> v2:172.21.15.145:3300/0 conn(0x7f3b2406ca70 0x7f3b240c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b29116640 1 -- 172.21.15.94:0/502992183 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240c7ad0 con 0x7f3b240c5380 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.664+0000 7f3b29116640 1 --2- 172.21.15.94:0/502992183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b240c57d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3b240cba90 tx=0x7f3b1c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405ab5bb8538c34f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b237fe640 1 -- 172.21.15.94:0/502992183 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c02c9e0 con 0x7f3b240c5380 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b237fe640 1 -- 172.21.15.94:0/502992183 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c02cb80 con 0x7f3b240c5380 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b237fe640 1 -- 172.21.15.94:0/502992183 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0343f0 con 0x7f3b240c5380 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 -- 172.21.15.94:0/502992183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 msgr2=0x7f3b240c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/502992183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b240c57d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3b240cba90 tx=0x7f3b1c02be40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 -- 172.21.15.94:0/502992183 shutdown_connections 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/502992183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b240c57d0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/502992183 >> v2:172.21.15.145:3300/0 conn(0x7f3b2406ca70 0x7f3b240c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/502992183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b24155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 -- 172.21.15.94:0/502992183 >> 172.21.15.94:0/502992183 conn(0x7f3b240ac590 msgr2=0x7f3b240ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 -- 172.21.15.94:0/502992183 shutdown_connections 2024-03-06T07:22:34.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.665+0000 7f3b2ab79640 1 -- 172.21.15.94:0/502992183 wait complete. 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.666+0000 7f3b2ab79640 1 Processor -- start 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.666+0000 7f3b2ab79640 1 -- start start 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b2ab79640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2406ca70 0x7f3b2414bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b2ab79640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b2414a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b2ab79640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b2414d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b2ab79640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2406cef0 con 0x7f3b240c5380 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b2ab79640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2406c550 con 0x7f3b2406ca70 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b2ab79640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2406da80 con 0x7f3b240b0070 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b2414d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b2414d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40460/0 (socket says 172.21.15.94:40460) 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 -- 172.21.15.94:0/3595808743 learned_addr learned my addr 172.21.15.94:0/3595808743 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b28915640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2406ca70 0x7f3b2414bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b23fff640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b2414a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 -- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 msgr2=0x7f3b2414a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b2414a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 -- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2406ca70 msgr2=0x7f3b2414bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2406ca70 0x7f3b2414bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.667+0000 7f3b29116640 1 -- 172.21.15.94:0/3595808743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2414e8e0 con 0x7f3b240c5380 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.668+0000 7f3b29116640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b2414d810 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009f40 tx=0x7f3b1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.668+0000 7f3b217fa640 1 -- 172.21.15.94:0/3595808743 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c043020 con 0x7f3b240c5380 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.668+0000 7f3b217fa640 1 -- 172.21.15.94:0/3595808743 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c03e450 con 0x7f3b240c5380 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.668+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2414ad00 con 0x7f3b240c5380 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.668+0000 7f3b217fa640 1 -- 172.21.15.94:0/3595808743 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c009510 con 0x7f3b240c5380 2024-03-06T07:22:34.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.668+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b241eb950 con 0x7f3b240c5380 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.669+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2404f330 con 0x7f3b240c5380 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.669+0000 7f3b217fa640 1 -- 172.21.15.94:0/3595808743 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b1c046020 con 0x7f3b240c5380 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.670+0000 7f3b217fa640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b0806b4d0 0x7f3b0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.670+0000 7f3b217fa640 1 -- 172.21.15.94:0/3595808743 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3b1c0ac940 con 0x7f3b240c5380 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.670+0000 7f3b28915640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b0806b4d0 0x7f3b0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.670+0000 7f3b28915640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b0806b4d0 0x7f3b0806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3b18006fd0 tx=0x7f3b18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.674+0000 7f3b217fa640 1 -- 172.21.15.94:0/3595808743 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b1c04d030 con 0x7f3b240c5380 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.844+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f3b2406d860 con 0x7f3b240c5380 2024-03-06T07:22:34.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.847+0000 7f3b217fa640 1 -- 172.21.15.94:0/3595808743 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v39) v1 ==== 96+0+0 (secure 0 0 0) 0x7f3b1c07b970 con 0x7f3b240c5380 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b0806b4d0 msgr2=0x7f3b0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b0806b4d0 0x7f3b0806d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3b18006fd0 tx=0x7f3b18008040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 msgr2=0x7f3b2414d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b2414d810 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009f40 tx=0x7f3b1c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 shutdown_connections 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3b0806b4d0 0x7f3b0806d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3b240c5380 0x7f3b2414d810 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b240b0070 0x7f3b2414a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 --2- 172.21.15.94:0/3595808743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2406ca70 0x7f3b2414bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 >> 172.21.15.94:0/3595808743 conn(0x7f3b240ac590 msgr2=0x7f3b240abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 shutdown_connections 2024-03-06T07:22:34.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.849+0000 7f3b2ab79640 1 -- 172.21.15.94:0/3595808743 wait complete. 2024-03-06T07:22:34.862 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-06T07:22:34.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.962+0000 7fc37926f640 1 Processor -- start 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.962+0000 7fc37926f640 1 -- start start 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc37926f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc37406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc37926f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740b0450 con 0x7fc3740c5380 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc37926f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3741560e0 con 0x7fc3740b0070 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc37926f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406cfb0 con 0x7fc374155c30 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc3737fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc3740c5380 0x7fc37414e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50516/0 (socket says 172.21.15.94:50516) 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc3737fe640 1 -- 172.21.15.94:0/2606735799 learned_addr learned my addr 172.21.15.94:0/2606735799 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc3727fc640 1 --2- 172.21.15.94:0/2606735799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc37406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234269111 0 0) 0x7fc36c001490 con 0x7fc3740c5380 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c003520 con 0x7fc3740c5380 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881931804 0 0) 0x7fc3741560e0 con 0x7fc3740b0070 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc36c001490 con 0x7fc3740b0070 2024-03-06T07:22:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 71096992 0 0) 0x7fc35c003520 con 0x7fc3740c5380 2024-03-06T07:22:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc3727fc640 1 -- 172.21.15.94:0/2606735799 >> v1:172.21.15.145:6789/0 conn(0x7fc3740b0070 legacy=0x7fc3740c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc3727fc640 1 -- 172.21.15.94:0/2606735799 >> v1:172.21.15.94:6789/0 conn(0x7fc3740c5380 legacy=0x7fc37414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.963+0000 7fc3727fc640 1 -- 172.21.15.94:0/2606735799 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740c77f0 con 0x7fc374155c30 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.964+0000 7fc3727fc640 1 --2- 172.21.15.94:0/2606735799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc37406ca70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc3680097d0 tx=0x7fc36802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90b2633d9c5b883e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.964+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368037070 con 0x7fc374155c30 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.964+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc368030510 con 0x7fc374155c30 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.964+0000 7fc371ffb640 1 -- 172.21.15.94:0/2606735799 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c001b40 con 0x7fc374155c30 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.965+0000 7fc37926f640 1 -- 172.21.15.94:0/2606735799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 msgr2=0x7fc37406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.965+0000 7fc37926f640 1 --2- 172.21.15.94:0/2606735799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc37406ca70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc3680097d0 tx=0x7fc36802b770 comp rx=0 tx=0).stop 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.965+0000 7fc37926f640 1 -- 172.21.15.94:0/2606735799 shutdown_connections 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.965+0000 7fc37926f640 1 --2- 172.21.15.94:0/2606735799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc37406ca70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.965+0000 7fc37926f640 1 -- 172.21.15.94:0/2606735799 >> 172.21.15.94:0/2606735799 conn(0x7fc3740ac590 msgr2=0x7fc3740ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.965+0000 7fc37926f640 1 -- 172.21.15.94:0/2606735799 shutdown_connections 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.965+0000 7fc37926f640 1 -- 172.21.15.94:0/2606735799 wait complete. 2024-03-06T07:22:34.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.966+0000 7fc37926f640 1 Processor -- start 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.966+0000 7fc37926f640 1 -- start start 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc37926f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3740b0070 0x7fc374154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc37926f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3740c5380 0x7fc3741e7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc37926f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc3741e8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc37926f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406c270 con 0x7fc3740c5380 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc37926f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406c020 con 0x7fc3740b0070 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc37926f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37406c550 con 0x7fc374155c30 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3740c5380 0x7fc3741e7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3740c5380 0x7fc3741e7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40464/0 (socket says 172.21.15.94:40464) 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 -- 172.21.15.94:0/2129198088 learned_addr learned my addr 172.21.15.94:0/2129198088 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc372ffd640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3740b0070 0x7fc374154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3737fe640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc3741e8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 -- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 msgr2=0x7fc3741e8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc3741e8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 -- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3740b0070 msgr2=0x7fc374154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3740b0070 0x7fc374154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 -- 172.21.15.94:0/2129198088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3741eb690 con 0x7fc3740c5380 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3737fe640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc3741e8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.967+0000 7fc3727fc640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3740c5380 0x7fc3741e7bd0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc3680097d0 tx=0x7fc3680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.968+0000 7fc35b7fe640 1 -- 172.21.15.94:0/2129198088 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368037040 con 0x7fc3740c5380 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.968+0000 7fc35b7fe640 1 -- 172.21.15.94:0/2129198088 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc368004250 con 0x7fc3740c5380 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.968+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3741eb970 con 0x7fc3740c5380 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.968+0000 7fc35b7fe640 1 -- 172.21.15.94:0/2129198088 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368003610 con 0x7fc3740c5380 2024-03-06T07:22:34.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.968+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3740c1db0 con 0x7fc3740c5380 2024-03-06T07:22:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.969+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3740678a0 con 0x7fc3740c5380 2024-03-06T07:22:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.969+0000 7fc35b7fe640 1 -- 172.21.15.94:0/2129198088 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3680037b0 con 0x7fc3740c5380 2024-03-06T07:22:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.969+0000 7fc35b7fe640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc36006b4d0 0x7fc36006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.969+0000 7fc35b7fe640 1 -- 172.21.15.94:0/2129198088 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc3680ac140 con 0x7fc3740c5380 2024-03-06T07:22:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.970+0000 7fc372ffd640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc36006b4d0 0x7fc36006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.970+0000 7fc372ffd640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc36006b4d0 0x7fc36006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc364000f60 tx=0x7fc364009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:34.974+0000 7fc35b7fe640 1 -- 172.21.15.94:0/2129198088 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3740678a0 con 0x7fc3740c5380 2024-03-06T07:22:35.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.145+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7fc37404f330 con 0x7fc3740c5380 2024-03-06T07:22:35.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.148+0000 7fc35b7fe640 1 -- 172.21.15.94:0/2129198088 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v40) v1 ==== 96+0+0 (secure 0 0 0) 0x7fc36807b170 con 0x7fc3740c5380 2024-03-06T07:22:35.152 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc36006b4d0 msgr2=0x7fc36006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc36006b4d0 0x7fc36006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc364000f60 tx=0x7fc364009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3740c5380 msgr2=0x7fc3741e7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3740c5380 0x7fc3741e7bd0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc3680097d0 tx=0x7fc3680046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 shutdown_connections 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc36006b4d0 0x7fc36006d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc374155c30 0x7fc3741e8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3740c5380 0x7fc3741e7bd0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 --2- 172.21.15.94:0/2129198088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3740b0070 0x7fc374154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 >> 172.21.15.94:0/2129198088 conn(0x7fc3740ac590 msgr2=0x7fc3740ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.150+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 shutdown_connections 2024-03-06T07:22:35.153 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.151+0000 7fc37926f640 1 -- 172.21.15.94:0/2129198088 wait complete. 2024-03-06T07:22:35.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-03-06T07:22:35.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.262+0000 7f357d7b6640 1 Processor -- start 2024-03-06T07:22:35.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.262+0000 7f357d7b6640 1 -- start start 2024-03-06T07:22:35.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f357d7b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35780c5480 0x7f3578155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f357d7b6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f35781586f0 0x7f3578158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f357d7b6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35780c5cc0 con 0x7f35780c5860 2024-03-06T07:22:35.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f357d7b6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f357806df40 con 0x7f35781586f0 2024-03-06T07:22:35.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f357d7b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357806fdb0 con 0x7f35780c5480 2024-03-06T07:22:35.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f35777fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f35781586f0 0x7f3578158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f35777fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f35781586f0 0x7f3578158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40478/0 (socket says 172.21.15.94:40478) 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f35777fe640 1 -- 172.21.15.94:0/393512334 learned_addr learned my addr 172.21.15.94:0/393512334 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f35777fe640 1 -- 172.21.15.94:0/393512334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35780c5480 msgr2=0x7f3578155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f35777fe640 1 --2- 172.21.15.94:0/393512334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35780c5480 0x7f3578155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f35777fe640 1 -- 172.21.15.94:0/393512334 >> v1:172.21.15.145:6789/0 conn(0x7f35780c5860 legacy=0x7f357806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.263+0000 7f35777fe640 1 -- 172.21.15.94:0/393512334 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35781591c0 con 0x7f35781586f0 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.264+0000 7f35777fe640 1 --2- 172.21.15.94:0/393512334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35781586f0 0x7f3578158b80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3570009870 tx=0x7f357002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3941c5c936185c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.264+0000 7f3575ffb640 1 -- 172.21.15.94:0/393512334 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35700317c0 con 0x7f35781586f0 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.264+0000 7f3575ffb640 1 -- 172.21.15.94:0/393512334 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3570037070 con 0x7f35781586f0 2024-03-06T07:22:35.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.264+0000 7f3575ffb640 1 -- 172.21.15.94:0/393512334 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3570031b80 con 0x7f35781586f0 2024-03-06T07:22:35.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 -- 172.21.15.94:0/393512334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35781586f0 msgr2=0x7f3578158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 --2- 172.21.15.94:0/393512334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35781586f0 0x7f3578158b80 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3570009870 tx=0x7f357002b860 comp rx=0 tx=0).stop 2024-03-06T07:22:35.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 -- 172.21.15.94:0/393512334 shutdown_connections 2024-03-06T07:22:35.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 --2- 172.21.15.94:0/393512334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35781586f0 0x7f3578158b80 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 --2- 172.21.15.94:0/393512334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35780c5480 0x7f3578155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 -- 172.21.15.94:0/393512334 >> 172.21.15.94:0/393512334 conn(0x7f357806b1a0 msgr2=0x7f357806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 -- 172.21.15.94:0/393512334 shutdown_connections 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.265+0000 7f357d7b6640 1 -- 172.21.15.94:0/393512334 wait complete. 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.266+0000 7f357d7b6640 1 Processor -- start 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.266+0000 7f357d7b6640 1 -- start start 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f357d7b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35780c5480 0x7f35780abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f357d7b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35780c5860 0x7f35780c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f357d7b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35781586f0 0x7f35780c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f357d7b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3578155680 con 0x7f35780c5860 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f357d7b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35781553a0 con 0x7f35780c5480 2024-03-06T07:22:35.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f357d7b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3578152a20 con 0x7f35781586f0 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35780c5860 0x7f35780c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35780c5860 0x7f35780c3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40484/0 (socket says 172.21.15.94:40484) 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 -- 172.21.15.94:0/732968428 learned_addr learned my addr 172.21.15.94:0/732968428 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35777fe640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35781586f0 0x7f35780c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 -- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35781586f0 msgr2=0x7f35780c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f3576ffd640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35780c5480 0x7f35780abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35781586f0 0x7f35780c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 -- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35780c5480 msgr2=0x7f35780abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35780c5480 0x7f35780abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 -- 172.21.15.94:0/732968428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35781eb760 con 0x7f35780c5860 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35777fe640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35781586f0 0x7f35780c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f3576ffd640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35780c5480 0x7f35780abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f35767fc640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35780c5860 0x7f35780c3d50 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f356c00c410 tx=0x7f356c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f355f7fe640 1 -- 172.21.15.94:0/732968428 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c017070 con 0x7f35780c5860 2024-03-06T07:22:35.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f355f7fe640 1 -- 172.21.15.94:0/732968428 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c0035f0 con 0x7f35780c5860 2024-03-06T07:22:35.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.267+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357814e360 con 0x7f35780c5860 2024-03-06T07:22:35.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.268+0000 7f355f7fe640 1 -- 172.21.15.94:0/732968428 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c003a20 con 0x7f35780c5860 2024-03-06T07:22:35.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.268+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35780c3330 con 0x7f35780c5860 2024-03-06T07:22:35.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.269+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357804f330 con 0x7f35780c5860 2024-03-06T07:22:35.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.269+0000 7f355f7fe640 1 -- 172.21.15.94:0/732968428 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f356c01a070 con 0x7f35780c5860 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.269+0000 7f355f7fe640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f356406b500 0x7f356406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.269+0000 7f355f7fe640 1 -- 172.21.15.94:0/732968428 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f356c0894f0 con 0x7f35780c5860 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.269+0000 7f3576ffd640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f356406b500 0x7f356406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.270+0000 7f3576ffd640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f356406b500 0x7f356406d9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3578150660 tx=0x7f35680073e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.273+0000 7f355f7fe640 1 -- 172.21.15.94:0/732968428 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f356c058520 con 0x7f35780c5860 2024-03-06T07:22:35.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:35.511+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7f35781584d0 con 0x7f35780c5860 2024-03-06T07:22:36.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.240+0000 7f355f7fe640 1 -- 172.21.15.94:0/732968428 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v259) v1 ==== 153+0+0 (secure 0 0 0) 0x7f356c0010c0 con 0x7f35780c5860 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f356406b500 msgr2=0x7f356406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f356406b500 0x7f356406d9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3578150660 tx=0x7f35680073e0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35780c5860 msgr2=0x7f35780c3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35780c5860 0x7f35780c3d50 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f356c00c410 tx=0x7f356c00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 shutdown_connections 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f356406b500 0x7f356406d9c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35781586f0 0x7f35780c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35780c5860 0x7f35780c3d50 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 --2- 172.21.15.94:0/732968428 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35780c5480 0x7f35780abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 >> 172.21.15.94:0/732968428 conn(0x7f357806b1a0 msgr2=0x7f357806ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 shutdown_connections 2024-03-06T07:22:36.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.243+0000 7f357d7b6640 1 -- 172.21.15.94:0/732968428 wait complete. 2024-03-06T07:22:36.258 INFO:tasks.cephfs_test_runner:That a standby-replay daemon with weak affinity is replaced by a stronger one. ... ok 2024-03-06T07:22:36.258 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) That a sole MDS with mds_join_fs set will come back as active eventually even after failover. 2024-03-06T07:22:36.258 INFO:tasks.cephfs_test_runner:test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:22:36.259 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.363+0000 7f437b878640 1 Processor -- start 2024-03-06T07:22:36.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.363+0000 7f437b878640 1 -- start start 2024-03-06T07:22:36.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f437b878640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4374152920 0x7f4374155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f437b878640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f43740c8a00 0x7f4374158f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f437b878640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374159460 0x7f43740c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f437b878640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43741587b0 con 0x7f4374152920 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f437b878640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4374156180 con 0x7f43740c8a00 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f437b878640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43740c7760 con 0x7f4374159460 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4374152920 0x7f4374155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4374152920 0x7f4374155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40494/0 (socket says 172.21.15.94:40494) 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 -- 172.21.15.94:0/2556996565 learned_addr learned my addr 172.21.15.94:0/2556996565 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4378e13640 1 --2- 172.21.15.94:0/2556996565 >> v2:172.21.15.145:3300/0 conn(0x7f43740c8a00 0x7f4374158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379e15640 1 --2- 172.21.15.94:0/2556996565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374159460 0x7f43740c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 -- 172.21.15.94:0/2556996565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374159460 msgr2=0x7f43740c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 --2- 172.21.15.94:0/2556996565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374159460 0x7f43740c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 -- 172.21.15.94:0/2556996565 >> v2:172.21.15.145:3300/0 conn(0x7f43740c8a00 msgr2=0x7f4374158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 --2- 172.21.15.94:0/2556996565 >> v2:172.21.15.145:3300/0 conn(0x7f43740c8a00 0x7f4374158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4379614640 1 -- 172.21.15.94:0/2556996565 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43740c7a40 con 0x7f4374152920 2024-03-06T07:22:36.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.364+0000 7f4378e13640 1 --2- 172.21.15.94:0/2556996565 >> v2:172.21.15.145:3300/0 conn(0x7f43740c8a00 0x7f4374158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.365+0000 7f4379614640 1 --2- 172.21.15.94:0/2556996565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4374152920 0x7f4374155c40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f4374067c30 tx=0x7f436402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c6b508fc1747e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.365+0000 7f436bfff640 1 -- 172.21.15.94:0/2556996565 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436402e070 con 0x7f4374152920 2024-03-06T07:22:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.365+0000 7f436bfff640 1 -- 172.21.15.94:0/2556996565 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364037070 con 0x7f4374152920 2024-03-06T07:22:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.365+0000 7f436bfff640 1 -- 172.21.15.94:0/2556996565 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364031a00 con 0x7f4374152920 2024-03-06T07:22:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 -- 172.21.15.94:0/2556996565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4374152920 msgr2=0x7f4374155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 --2- 172.21.15.94:0/2556996565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4374152920 0x7f4374155c40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f4374067c30 tx=0x7f436402b880 comp rx=0 tx=0).stop 2024-03-06T07:22:36.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 -- 172.21.15.94:0/2556996565 shutdown_connections 2024-03-06T07:22:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 --2- 172.21.15.94:0/2556996565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374159460 0x7f43740c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 --2- 172.21.15.94:0/2556996565 >> v2:172.21.15.145:3300/0 conn(0x7f43740c8a00 0x7f4374158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 --2- 172.21.15.94:0/2556996565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4374152920 0x7f4374155c40 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 -- 172.21.15.94:0/2556996565 >> 172.21.15.94:0/2556996565 conn(0x7f43740ac560 msgr2=0x7f43740ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 -- 172.21.15.94:0/2556996565 shutdown_connections 2024-03-06T07:22:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.366+0000 7f437b878640 1 -- 172.21.15.94:0/2556996565 wait complete. 2024-03-06T07:22:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 Processor -- start 2024-03-06T07:22:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 -- start start 2024-03-06T07:22:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43740c8a00 0x7f43740c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374152920 0x7f43741eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43740c4650 0x7f43741eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437406d030 con 0x7f43740c8a00 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374158a00 con 0x7f43740c4650 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.367+0000 7f437b878640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43740c7390 con 0x7f4374152920 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43740c8a00 0x7f43740c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43740c8a00 0x7f43740c9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40508/0 (socket says 172.21.15.94:40508) 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 -- 172.21.15.94:0/315715701 learned_addr learned my addr 172.21.15.94:0/315715701 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379e15640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43740c4650 0x7f43741eb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4378e13640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374152920 0x7f43741eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 -- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374152920 msgr2=0x7f43741eb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374152920 0x7f43741eb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 -- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43740c4650 msgr2=0x7f43741eb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43740c4650 0x7f43741eb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 -- 172.21.15.94:0/315715701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43741ef120 con 0x7f43740c8a00 2024-03-06T07:22:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4379614640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43740c8a00 0x7f43740c9130 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f4364000c00 tx=0x7f43640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4369ffb640 1 -- 172.21.15.94:0/315715701 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364032670 con 0x7f43740c8a00 2024-03-06T07:22:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4369ffb640 1 -- 172.21.15.94:0/315715701 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364032850 con 0x7f43740c8a00 2024-03-06T07:22:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4374155bb0 con 0x7f43740c8a00 2024-03-06T07:22:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.368+0000 7f4369ffb640 1 -- 172.21.15.94:0/315715701 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364032c40 con 0x7f43740c8a00 2024-03-06T07:22:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.369+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437414cbc0 con 0x7f43740c8a00 2024-03-06T07:22:36.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.370+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437404f330 con 0x7f43740c8a00 2024-03-06T07:22:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.370+0000 7f4369ffb640 1 -- 172.21.15.94:0/315715701 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f436404c060 con 0x7f43740c8a00 2024-03-06T07:22:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.370+0000 7f4369ffb640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f435806b500 0x7f435806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.370+0000 7f4369ffb640 1 -- 172.21.15.94:0/315715701 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f43640abab0 con 0x7f43740c8a00 2024-03-06T07:22:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.371+0000 7f4378e13640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f435806b500 0x7f435806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.371+0000 7f4378e13640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f435806b500 0x7f435806d9c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4370006fd0 tx=0x7f4370008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.375+0000 7f4369ffb640 1 -- 172.21.15.94:0/315715701 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4364040070 con 0x7f43740c8a00 2024-03-06T07:22:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.547+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43740be1f0 con 0x7f43740c8a00 2024-03-06T07:22:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.548+0000 7f4369ffb640 1 -- 172.21.15.94:0/315715701 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+13435 (secure 0 0 0) 0x7f436407ab70 con 0x7f43740c8a00 2024-03-06T07:22:36.550 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:36.550 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":112,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:34.231027+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/2226553926":"2024-03-07T07:22:31.112093+0000","172.21.15.94:6832/2226553926":"2024-03-07T07:22:31.112093+0000","172.21.15.145:6836/4001418202":"2024-03-07T07:22:13.494369+0000","172.21.15.145:6834/4001418202":"2024-03-07T07:22:13.494369+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f435806b500 msgr2=0x7f435806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f435806b500 0x7f435806d9c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4370006fd0 tx=0x7f4370008040 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43740c8a00 msgr2=0x7f43740c9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43740c8a00 0x7f43740c9130 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f4364000c00 tx=0x7f43640046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 shutdown_connections 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f435806b500 0x7f435806d9c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43740c4650 0x7f43741eb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4374152920 0x7f43741eb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 --2- 172.21.15.94:0/315715701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43740c8a00 0x7f43740c9130 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 >> 172.21.15.94:0/315715701 conn(0x7f43740ac560 msgr2=0x7f4374151b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 shutdown_connections 2024-03-06T07:22:36.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.550+0000 7f437b878640 1 -- 172.21.15.94:0/315715701 wait complete. 2024-03-06T07:22:36.563 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:36.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:36.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.664+0000 7f4eeec42640 1 Processor -- start 2024-03-06T07:22:36.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.664+0000 7f4eeec42640 1 -- start start 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.664+0000 7f4eeec42640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 0x7f4ee80c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.664+0000 7f4eeec42640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4ee80ab170 0x7f4ee806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.664+0000 7f4eeec42640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee80c3ef0 con 0x7f4ee806c730 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.664+0000 7f4eeec42640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee806cbe0 con 0x7f4ee80ab170 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.664+0000 7f4eeec42640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee81556d0 con 0x7f4ee80c5990 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4eed1df640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4ee806c730 0x7f4ee81523f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50526/0 (socket says 172.21.15.94:50526) 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4eed1df640 1 -- 172.21.15.94:0/3019516420 learned_addr learned my addr 172.21.15.94:0/3019516420 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee7fff640 1 --2- 172.21.15.94:0/3019516420 >> v2:172.21.15.145:3300/0 conn(0x7f4ee80ab170 0x7f4ee806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4eec9de640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 0x7f4ee80c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051191524 0 0) 0x7f4ee80c3ef0 con 0x7f4ee806c730 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed8003520 con 0x7f4ee806c730 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385205471 0 0) 0x7f4ed8003520 con 0x7f4ee806c730 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 msgr2=0x7f4ee80c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 0x7f4ee80c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 >> v2:172.21.15.145:3300/0 conn(0x7f4ee80ab170 msgr2=0x7f4ee806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 --2- 172.21.15.94:0/3019516420 >> v2:172.21.15.145:3300/0 conn(0x7f4ee80ab170 0x7f4ee806c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee81559b0 con 0x7f4ee806c730 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f4ee00019f0 con 0x7f4ee806c730 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 >> v1:172.21.15.94:6789/0 conn(0x7f4ee806c730 legacy=0x7f4ee81523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed80039b0 0x7f4ed8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 0x7f4ed8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ed8004d40 0x7f4ed800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee81556d0 con 0x7f4ed8004d40 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80051c0 con 0x7f4ed80039b0 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8005390 con 0x7f4ee80c5990 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4eed1df640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ed8004d40 0x7f4ed800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4eec9de640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed80039b0 0x7f4ed8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.665+0000 7f4ee7fff640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 0x7f4ed8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4eed1df640 1 -- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 msgr2=0x7f4ed8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4eed1df640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80c5990 0x7f4ed8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4eed1df640 1 -- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed80039b0 msgr2=0x7f4ed8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4eed1df640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed80039b0 0x7f4ed8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4eed1df640 1 -- 172.21.15.94:0/3019516420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8003520 con 0x7f4ed8004d40 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4eed1df640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ed8004d40 0x7f4ed800b9c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000a870 tx=0x7f4ed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51110b992818fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee000a4c0 con 0x7f4ed8004d40 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee0034630 con 0x7f4ed8004d40 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.666+0000 7f4ee77fe640 1 -- 172.21.15.94:0/3019516420 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0034a20 con 0x7f4ed8004d40 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.667+0000 7f4eeec42640 1 -- 172.21.15.94:0/3019516420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ed8004d40 msgr2=0x7f4ed800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.667+0000 7f4eeec42640 1 --2- 172.21.15.94:0/3019516420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ed8004d40 0x7f4ed800b9c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000a870 tx=0x7f4ed8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.667+0000 7f4eed1df640 1 -- 172.21.15.94:0/3019516420 reap_dead start 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.667+0000 7f4eeec42640 1 -- 172.21.15.94:0/3019516420 shutdown_connections 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.667+0000 7f4eeec42640 1 -- 172.21.15.94:0/3019516420 >> 172.21.15.94:0/3019516420 conn(0x7f4ee80c6710 msgr2=0x7f4ee80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.667+0000 7f4eeec42640 1 -- 172.21.15.94:0/3019516420 shutdown_connections 2024-03-06T07:22:36.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.667+0000 7f4eeec42640 1 -- 172.21.15.94:0/3019516420 wait complete. 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.668+0000 7f4eeec42640 1 Processor -- start 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.668+0000 7f4eeec42640 1 -- start start 2024-03-06T07:22:36.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eeec42640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed8004d40 0x7f4ee81ea160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eeec42640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee814c8c0 0x7f4ee81db950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eeec42640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ee8148970 0x7f4ee8148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eeec42640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8156490 con 0x7f4ee8148970 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eeec42640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8155f70 con 0x7f4ed8004d40 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eeec42640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8156610 con 0x7f4ee814c8c0 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ee8148970 0x7f4ee8148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ee8148970 0x7f4ee8148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40530/0 (socket says 172.21.15.94:40530) 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 -- 172.21.15.94:0/253523379 learned_addr learned my addr 172.21.15.94:0/253523379 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 -- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee814c8c0 msgr2=0x7f4ee81db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee814c8c0 0x7f4ee81db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 -- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed8004d40 msgr2=0x7f4ee81ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed8004d40 0x7f4ee81ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 -- 172.21.15.94:0/253523379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8155340 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4eed1df640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ee8148970 0x7f4ee8148e00 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000a990 tx=0x7f4ee8155b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4ee57fa640 1 -- 172.21.15.94:0/253523379 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0039040 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.669+0000 7f4ee57fa640 1 -- 172.21.15.94:0/253523379 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee0043070 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.670+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee814a910 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.670+0000 7f4ee57fa640 1 -- 172.21.15.94:0/253523379 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0002030 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.670+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee8154560 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.671+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee804f330 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.671+0000 7f4ee57fa640 1 -- 172.21.15.94:0/253523379 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ee0034630 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.671+0000 7f4ee57fa640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ecc06b500 0x7f4ecc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.672+0000 7f4ee57fa640 1 -- 172.21.15.94:0/253523379 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4ee00abef0 con 0x7f4ee8148970 2024-03-06T07:22:36.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.672+0000 7f4eec9de640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ecc06b500 0x7f4ecc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.672+0000 7f4eec9de640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ecc06b500 0x7f4ecc06d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009a40 tx=0x7f4edc009320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.676+0000 7f4ee57fa640 1 -- 172.21.15.94:0/253523379 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ee8155340 con 0x7f4ee8148970 2024-03-06T07:22:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.845+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ee80c28c0 con 0x7f4ee8148970 2024-03-06T07:22:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.845+0000 7f4ee57fa640 1 -- 172.21.15.94:0/253523379 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ee007ae90 con 0x7f4ee8148970 2024-03-06T07:22:36.847 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:22:36.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ecc06b500 msgr2=0x7f4ecc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ecc06b500 0x7f4ecc06d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009a40 tx=0x7f4edc009320 comp rx=0 tx=0).stop 2024-03-06T07:22:36.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ee8148970 msgr2=0x7f4ee8148e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ee8148970 0x7f4ee8148e00 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000a990 tx=0x7f4ee8155b80 comp rx=0 tx=0).stop 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 shutdown_connections 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ecc06b500 0x7f4ecc06d9c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ee8148970 0x7f4ee8148e00 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee814c8c0 0x7f4ee81db950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 --2- 172.21.15.94:0/253523379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ed8004d40 0x7f4ee81ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 >> 172.21.15.94:0/253523379 conn(0x7f4ee80c6710 msgr2=0x7f4ee80c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 shutdown_connections 2024-03-06T07:22:36.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.847+0000 7f4eeec42640 1 -- 172.21.15.94:0/253523379 wait complete. 2024-03-06T07:22:36.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-03-06T07:22:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.959+0000 7f8fb151e640 1 Processor -- start 2024-03-06T07:22:36.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.959+0000 7f8fb151e640 1 -- start start 2024-03-06T07:22:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fb151e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fb151e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac152d40 con 0x7f8fac1592d0 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fb151e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac0c1b50 con 0x7f8fac0ac260 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fb151e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac066e30 con 0x7f8fac152960 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8faa7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8fac1592d0 0x7f8fac1552d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50532/0 (socket says 172.21.15.94:50532) 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8faa7fc640 1 -- 172.21.15.94:0/3616736090 learned_addr learned my addr 172.21.15.94:0/3616736090 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8faaffd640 1 --2- 172.21.15.94:0/3616736090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506197229 0 0) 0x7f8fac152d40 con 0x7f8fac1592d0 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94003520 con 0x7f8fac1592d0 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184454780 0 0) 0x7f8fac0c1b50 con 0x7f8fac0ac260 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fac152d40 con 0x7f8fac0ac260 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8faaffd640 1 -- 172.21.15.94:0/3616736090 >> v1:172.21.15.145:6789/0 conn(0x7f8fac0ac260 legacy=0x7f8fac15fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8faaffd640 1 -- 172.21.15.94:0/3616736090 >> v1:172.21.15.94:6789/0 conn(0x7f8fac1592d0 legacy=0x7f8fac1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.960+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073531039 0 0) 0x7f8f94003520 con 0x7f8fac1592d0 2024-03-06T07:22:36.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.961+0000 7f8faaffd640 1 -- 172.21.15.94:0/3616736090 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac0c1dc0 con 0x7f8fac152960 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.961+0000 7f8faaffd640 1 --2- 172.21.15.94:0/3616736090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac06dc90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009dc0 tx=0x7f8f9c02bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=334a6429e4c32a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c039070 con 0x7f8fac152960 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c02c910 con 0x7f8fac152960 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fa9ffb640 1 -- 172.21.15.94:0/3616736090 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0001670 con 0x7f8fac152960 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fb151e640 1 -- 172.21.15.94:0/3616736090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 msgr2=0x7f8fac06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3616736090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac06dc90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009dc0 tx=0x7f8f9c02bde0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fb151e640 1 -- 172.21.15.94:0/3616736090 shutdown_connections 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3616736090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac06dc90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fb151e640 1 -- 172.21.15.94:0/3616736090 >> 172.21.15.94:0/3616736090 conn(0x7f8fac0c70b0 msgr2=0x7f8fac0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fb151e640 1 -- 172.21.15.94:0/3616736090 shutdown_connections 2024-03-06T07:22:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.962+0000 7f8fb151e640 1 -- 172.21.15.94:0/3616736090 wait complete. 2024-03-06T07:22:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.963+0000 7f8fb151e640 1 Processor -- start 2024-03-06T07:22:36.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.963+0000 7f8fb151e640 1 -- start start 2024-03-06T07:22:36.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fb151e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fac0ac260 0x7f8fac146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fb151e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fb151e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fac0728b0 0x7f8fac1eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fb151e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0c20a0 con 0x7f8fac0728b0 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fb151e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac159760 con 0x7f8fac0ac260 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fb151e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0c2220 con 0x7f8fac152960 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fac0728b0 0x7f8fac1eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fac0728b0 0x7f8fac1eb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40544/0 (socket says 172.21.15.94:40544) 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 -- 172.21.15.94:0/3197967230 learned_addr learned my addr 172.21.15.94:0/3197967230 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8faaffd640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fac0ac260 0x7f8fac146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8faa7fc640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 -- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 msgr2=0x7f8fac072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac072370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 -- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fac0ac260 msgr2=0x7f8fac146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fac0ac260 0x7f8fac146490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.964+0000 7f8fab7fe640 1 -- 172.21.15.94:0/3197967230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac1ee7b0 con 0x7f8fac0728b0 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.965+0000 7f8fab7fe640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fac0728b0 0x7f8fac1eb590 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400c430 tx=0x7f8fa400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.965+0000 7f8f937fe640 1 -- 172.21.15.94:0/3197967230 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4017070 con 0x7f8fac0728b0 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.965+0000 7f8f937fe640 1 -- 172.21.15.94:0/3197967230 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa40052d0 con 0x7f8fac0728b0 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.965+0000 7f8f937fe640 1 -- 172.21.15.94:0/3197967230 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4003830 con 0x7f8fac0728b0 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.965+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac1eea90 con 0x7f8fac0728b0 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.965+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac1ef050 con 0x7f8fac0728b0 2024-03-06T07:22:36.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.966+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac069460 con 0x7f8fac0728b0 2024-03-06T07:22:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.967+0000 7f8f937fe640 1 -- 172.21.15.94:0/3197967230 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fa401a070 con 0x7f8fac0728b0 2024-03-06T07:22:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.967+0000 7f8f937fe640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f9806b4d0 0x7f8f9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.967+0000 7f8f937fe640 1 -- 172.21.15.94:0/3197967230 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f8fa4089440 con 0x7f8fac0728b0 2024-03-06T07:22:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.967+0000 7f8faaffd640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f9806b4d0 0x7f8f9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.968+0000 7f8faaffd640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f9806b4d0 0x7f8f9806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c036000 tx=0x7f8f9c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:36.971+0000 7f8f937fe640 1 -- 172.21.15.94:0/3197967230 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fac069460 con 0x7f8fac0728b0 2024-03-06T07:22:37.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.209+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7f8fac04f330 con 0x7f8fac0728b0 2024-03-06T07:22:37.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.247+0000 7f8f937fe640 1 -- 172.21.15.94:0/3197967230 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v260) v1 ==== 148+0+0 (secure 0 0 0) 0x7f8fa4058470 con 0x7f8fac0728b0 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f9806b4d0 msgr2=0x7f8f9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f9806b4d0 0x7f8f9806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c036000 tx=0x7f8f9c030000 comp rx=0 tx=0).stop 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fac0728b0 msgr2=0x7f8fac1eb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fac0728b0 0x7f8fac1eb590 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400c430 tx=0x7f8fa400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 shutdown_connections 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f9806b4d0 0x7f8f9806d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8fac0728b0 0x7f8fac1eb590 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fac152960 0x7f8fac072370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 --2- 172.21.15.94:0/3197967230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fac0ac260 0x7f8fac146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 >> 172.21.15.94:0/3197967230 conn(0x7f8fac0c70b0 msgr2=0x7f8fac0c4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 shutdown_connections 2024-03-06T07:22:37.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.250+0000 7f8fb151e640 1 -- 172.21.15.94:0/3197967230 wait complete. 2024-03-06T07:22:37.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:37.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.363+0000 7f90446cf640 1 Processor -- start 2024-03-06T07:22:37.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.363+0000 7f90446cf640 1 -- start start 2024-03-06T07:22:37.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.363+0000 7f90446cf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f903c0c4c60 0x7f903c0c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.363+0000 7f90446cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c152e40 0x7f903c0724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.363+0000 7f90446cf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903c0c80b0 con 0x7f903c070a50 2024-03-06T07:22:37.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.363+0000 7f90446cf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903c070ee0 con 0x7f903c0c4c60 2024-03-06T07:22:37.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f90446cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c156940 con 0x7f903c152e40 2024-03-06T07:22:37.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9042c6c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c152e40 0x7f903c0724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f904246b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f903c0c4c60 0x7f903c0c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9042c6c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c152e40 0x7f903c0724d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:58716/0 (socket says 172.21.15.94:58716) 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9042c6c640 1 -- 172.21.15.94:0/2361148638 learned_addr learned my addr 172.21.15.94:0/2361148638 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9041c6a640 1 --1- 172.21.15.94:0/2361148638 >> v1:172.21.15.145:6789/0 conn(0x7f903c070a50 0x7f903c156dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:45156/0 (socket says 172.21.15.94:45156) 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9041469640 1 -- 172.21.15.94:0/2361148638 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444848406 0 0) 0x7f903c0c80b0 con 0x7f903c070a50 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9041469640 1 -- 172.21.15.94:0/2361148638 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9028003520 con 0x7f903c070a50 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9042c6c640 1 -- 172.21.15.94:0/2361148638 >> v2:172.21.15.145:3300/0 conn(0x7f903c0c4c60 msgr2=0x7f903c0c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9042c6c640 1 --2- 172.21.15.94:0/2361148638 >> v2:172.21.15.145:3300/0 conn(0x7f903c0c4c60 0x7f903c0c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9042c6c640 1 -- 172.21.15.94:0/2361148638 >> v1:172.21.15.145:6789/0 conn(0x7f903c070a50 legacy=0x7f903c156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f9042c6c640 1 -- 172.21.15.94:0/2361148638 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0c8390 con 0x7f903c152e40 2024-03-06T07:22:37.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.364+0000 7f904246b640 1 --2- 172.21.15.94:0/2361148638 >> v2:172.21.15.145:3300/0 conn(0x7f903c0c4c60 0x7f903c0c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.365+0000 7f9042c6c640 1 --2- 172.21.15.94:0/2361148638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c152e40 0x7f903c0724d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9034009870 tx=0x7f903402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce408df6a75190ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.365+0000 7f9041469640 1 -- 172.21.15.94:0/2361148638 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034035020 con 0x7f903c152e40 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.365+0000 7f9041469640 1 -- 172.21.15.94:0/2361148638 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9034031640 con 0x7f903c152e40 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.365+0000 7f9041469640 1 -- 172.21.15.94:0/2361148638 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903402d040 con 0x7f903c152e40 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 -- 172.21.15.94:0/2361148638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c152e40 msgr2=0x7f903c0724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 --2- 172.21.15.94:0/2361148638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c152e40 0x7f903c0724d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9034009870 tx=0x7f903402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 -- 172.21.15.94:0/2361148638 shutdown_connections 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 --2- 172.21.15.94:0/2361148638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c152e40 0x7f903c0724d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 --2- 172.21.15.94:0/2361148638 >> v2:172.21.15.145:3300/0 conn(0x7f903c0c4c60 0x7f903c0c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 -- 172.21.15.94:0/2361148638 >> 172.21.15.94:0/2361148638 conn(0x7f903c0aa6a0 msgr2=0x7f903c0aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 -- 172.21.15.94:0/2361148638 shutdown_connections 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.366+0000 7f90446cf640 1 -- 172.21.15.94:0/2361148638 wait complete. 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.367+0000 7f90446cf640 1 Processor -- start 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.367+0000 7f90446cf640 1 -- start start 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c070a50 0x7f903c1469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c4c60 0x7f903c1eb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c152e40 0x7f903c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c072a10 con 0x7f903c152e40 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c1532f0 con 0x7f903c0c4c60 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c072b90 con 0x7f903c070a50 2024-03-06T07:22:37.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c152e40 0x7f903c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c152e40 0x7f903c1521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40560/0 (socket says 172.21.15.94:40560) 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 -- 172.21.15.94:0/2615203305 learned_addr learned my addr 172.21.15.94:0/2615203305 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f904246b640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c070a50 0x7f903c1469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9041c6a640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c4c60 0x7f903c1eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 -- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c070a50 msgr2=0x7f903c1469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c070a50 0x7f903c1469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 -- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c4c60 msgr2=0x7f903c1eb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c4c60 0x7f903c1eb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 -- 172.21.15.94:0/2615203305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c14af30 con 0x7f903c152e40 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f904246b640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c070a50 0x7f903c1469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9042c6c640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c152e40 0x7f903c1521a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f90340090b0 tx=0x7f90340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9032ffd640 1 -- 172.21.15.94:0/2615203305 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034035040 con 0x7f903c152e40 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9032ffd640 1 -- 172.21.15.94:0/2615203305 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9034003820 con 0x7f903c152e40 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c14b210 con 0x7f903c152e40 2024-03-06T07:22:37.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f9032ffd640 1 -- 172.21.15.94:0/2615203305 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034003bd0 con 0x7f903c152e40 2024-03-06T07:22:37.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.368+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c14b7d0 con 0x7f903c152e40 2024-03-06T07:22:37.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.369+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c04f330 con 0x7f903c152e40 2024-03-06T07:22:37.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.370+0000 7f9032ffd640 1 -- 172.21.15.94:0/2615203305 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9034042020 con 0x7f903c152e40 2024-03-06T07:22:37.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.370+0000 7f9032ffd640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902006b4d0 0x7f902006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.370+0000 7f9032ffd640 1 -- 172.21.15.94:0/2615203305 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f90340abd00 con 0x7f903c152e40 2024-03-06T07:22:37.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.370+0000 7f904246b640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902006b4d0 0x7f902006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.371+0000 7f904246b640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902006b4d0 0x7f902006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f903c0c3ef0 tx=0x7f903800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.375+0000 7f9032ffd640 1 -- 172.21.15.94:0/2615203305 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f903407ad30 con 0x7f903c152e40 2024-03-06T07:22:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.546+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f903c0c2fd0 con 0x7f903c152e40 2024-03-06T07:22:37.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.550+0000 7f9032ffd640 1 -- 172.21.15.94:0/2615203305 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v41)=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7f903407a630 con 0x7f903c152e40 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902006b4d0 msgr2=0x7f902006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902006b4d0 0x7f902006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f903c0c3ef0 tx=0x7f903800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c152e40 msgr2=0x7f903c1521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c152e40 0x7f903c1521a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f90340090b0 tx=0x7f90340046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 shutdown_connections 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f902006b4d0 0x7f902006d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f903c152e40 0x7f903c1521a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f903c0c4c60 0x7f903c1eb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 --2- 172.21.15.94:0/2615203305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f903c070a50 0x7f903c1469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 >> 172.21.15.94:0/2615203305 conn(0x7f903c0aa6a0 msgr2=0x7f903c1dc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 shutdown_connections 2024-03-06T07:22:37.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.552+0000 7f90446cf640 1 -- 172.21.15.94:0/2615203305 wait complete. 2024-03-06T07:22:37.564 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:37.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:37.614 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:37.662 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:37.662 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:22:37.662 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:37.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:22:37.678 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:22:37.678 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:37.695 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:22:37.695 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:22:37.695 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:37.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.816+0000 7f319a5ac640 1 Processor -- start 2024-03-06T07:22:37.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.816+0000 7f319a5ac640 1 -- start start 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f319a5ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3194072720 0x7f31941564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f319a5ac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3194156a00 0x7f3194150290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f319a5ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3194156f90 con 0x7f31941507d0 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f319a5ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31940c38d0 con 0x7f3194072720 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f319a5ac640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3194157270 con 0x7f3194156a00 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f3193fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3194072720 0x7f31941564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f3193fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3194072720 0x7f31941564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40566/0 (socket says 172.21.15.94:40566) 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f3193fff640 1 -- 172.21.15.94:0/580718390 learned_addr learned my addr 172.21.15.94:0/580718390 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f3193fff640 1 -- 172.21.15.94:0/580718390 >> v2:172.21.15.145:3300/0 conn(0x7f3194156a00 msgr2=0x7f3194150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f3193fff640 1 --2- 172.21.15.94:0/580718390 >> v2:172.21.15.145:3300/0 conn(0x7f3194156a00 0x7f3194150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f3193fff640 1 -- 172.21.15.94:0/580718390 >> v1:172.21.15.145:6789/0 conn(0x7f31941507d0 legacy=0x7f31940c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.817+0000 7f3193fff640 1 -- 172.21.15.94:0/580718390 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31940c3280 con 0x7f3194072720 2024-03-06T07:22:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.818+0000 7f3193fff640 1 --2- 172.21.15.94:0/580718390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31941564c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f319406eaa0 tx=0x7f318402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735eee99ed8b851a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.818+0000 7f3192ffd640 1 -- 172.21.15.94:0/580718390 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318402e070 con 0x7f3194072720 2024-03-06T07:22:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.818+0000 7f3192ffd640 1 -- 172.21.15.94:0/580718390 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184037070 con 0x7f3194072720 2024-03-06T07:22:37.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.818+0000 7f3192ffd640 1 -- 172.21.15.94:0/580718390 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184031a00 con 0x7f3194072720 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 -- 172.21.15.94:0/580718390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 msgr2=0x7f31941564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 --2- 172.21.15.94:0/580718390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31941564c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f319406eaa0 tx=0x7f318402b880 comp rx=0 tx=0).stop 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 -- 172.21.15.94:0/580718390 shutdown_connections 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 --2- 172.21.15.94:0/580718390 >> v2:172.21.15.145:3300/0 conn(0x7f3194156a00 0x7f3194150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 --2- 172.21.15.94:0/580718390 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31941564c0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 -- 172.21.15.94:0/580718390 >> 172.21.15.94:0/580718390 conn(0x7f31940aa630 msgr2=0x7f31940aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 -- 172.21.15.94:0/580718390 shutdown_connections 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.819+0000 7f319a5ac640 1 -- 172.21.15.94:0/580718390 wait complete. 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.820+0000 7f319a5ac640 1 Processor -- start 2024-03-06T07:22:37.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.820+0000 7f319a5ac640 1 -- start start 2024-03-06T07:22:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31940ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31941507d0 0x7f31940c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3194156a00 0x7f31940c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194072b00 con 0x7f3194072720 2024-03-06T07:22:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319406f8f0 con 0x7f31941507d0 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3194072c80 con 0x7f3194156a00 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31940ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31940ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40574/0 (socket says 172.21.15.94:40574) 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 -- 172.21.15.94:0/816222166 learned_addr learned my addr 172.21.15.94:0/816222166 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f31937fe640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31941507d0 0x7f31940c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3198b49640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3194156a00 0x7f31940c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 -- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3194156a00 msgr2=0x7f31940c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3194156a00 0x7f31940c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 -- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31941507d0 msgr2=0x7f31940c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31941507d0 0x7f31940c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 -- 172.21.15.94:0/816222166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31940c3e00 con 0x7f3194072720 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3198b49640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3194156a00 0x7f31940c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3193fff640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31940ad860 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f3184000c00 tx=0x7f31840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3190ff9640 1 -- 172.21.15.94:0/816222166 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184032880 con 0x7f3194072720 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3190ff9640 1 -- 172.21.15.94:0/816222166 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184032a20 con 0x7f3194072720 2024-03-06T07:22:37.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3194151e60 con 0x7f3194072720 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f3190ff9640 1 -- 172.21.15.94:0/816222166 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184032e10 con 0x7f3194072720 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.821+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31940c4350 con 0x7f3194072720 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.822+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319404f330 con 0x7f3194072720 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.823+0000 7f3190ff9640 1 -- 172.21.15.94:0/816222166 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f318404c060 con 0x7f3194072720 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.823+0000 7f3190ff9640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f318006b4d0 0x7f318006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.823+0000 7f3190ff9640 1 -- 172.21.15.94:0/816222166 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f31840ab900 con 0x7f3194072720 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.823+0000 7f31937fe640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f318006b4d0 0x7f318006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.824+0000 7f31937fe640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f318006b4d0 0x7f318006d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f31940ae2f0 tx=0x7f3188008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:37.827+0000 7f3190ff9640 1 -- 172.21.15.94:0/816222166 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3184040070 con 0x7f3194072720 2024-03-06T07:22:38.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.039+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f319414b830 con 0x7f3194072720 2024-03-06T07:22:38.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.040+0000 7f3190ff9640 1 -- 172.21.15.94:0/816222166 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3706 (secure 0 0 0) 0x7f318407a8a0 con 0x7f3194072720 2024-03-06T07:22:38.042 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:38.042 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":100,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7201,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1342040049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1342040049},{"type":"v1","addr":"172.21.15.145:6837","nonce":1342040049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":7353,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3093758100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3093758100},{"type":"v1","addr":"172.21.15.94:6835","nonce":3093758100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":7743,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/714133878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":714133878},{"type":"v1","addr":"172.21.15.94:6834","nonce":714133878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100},{"gid":36359,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3298010342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3298010342},{"type":"v1","addr":"172.21.15.145:6836","nonce":3298010342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[]} 2024-03-06T07:22:38.042 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 100 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f318006b4d0 msgr2=0x7f318006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f318006b4d0 0x7f318006d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f31940ae2f0 tx=0x7f3188008040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 msgr2=0x7f31940ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31940ad860 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f3184000c00 tx=0x7f31840046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 shutdown_connections 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f318006b4d0 0x7f318006d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3194156a00 0x7f31940c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31941507d0 0x7f31940c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 --2- 172.21.15.94:0/816222166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3194072720 0x7f31940ad860 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.041+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 >> 172.21.15.94:0/816222166 conn(0x7f31940aa630 msgr2=0x7f3194148bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.042+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 shutdown_connections 2024-03-06T07:22:38.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.042+0000 7f319a5ac640 1 -- 172.21.15.94:0/816222166 wait complete. 2024-03-06T07:22:38.054 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:38.054 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:38.055 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:38.055 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:38.055 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:38.055 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:38.055 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:38.055 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:38.056 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:38.056 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:38.056 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:38.056 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:38.056 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:38.056 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:38.057 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:38.057 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:38.064 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:38.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:38.065 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:38.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:38.068 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:38.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:38.069 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:38.069 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:38.107 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:38.107 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:38.108 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:38.108 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:38.110 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:38.110 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:38.111 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:38.111 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:38.111 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:38.184 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:22:38.184 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:22:38.192 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:22:38.193 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:22:38.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.215+0000 7fce5b95e640 1 Processor -- start 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.216+0000 7fce5b95e640 1 -- start start 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.216+0000 7fce5b95e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fce540c69f0 0x7fce540c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.216+0000 7fce5b95e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce54155130 con 0x7fce541558d0 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.216+0000 7fce5b95e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce540ad480 con 0x7fce540c8950 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.216+0000 7fce5b95e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce54155440 con 0x7fce540c69f0 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce596fa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fce540c69f0 0x7fce540c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce59efb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fce541558d0 0x7fce5415fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50542/0 (socket says 172.21.15.94:50542) 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce59efb640 1 -- 172.21.15.94:0/1736897619 learned_addr learned my addr 172.21.15.94:0/1736897619 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce596fa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fce540c69f0 0x7fce540c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40614/0 (socket says 172.21.15.94:40614) 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce596fa640 1 -- 172.21.15.94:0/1736897619 >> v1:172.21.15.145:6789/0 conn(0x7fce540c8950 legacy=0x7fce540acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce596fa640 1 -- 172.21.15.94:0/1736897619 >> v1:172.21.15.94:6789/0 conn(0x7fce541558d0 legacy=0x7fce5415fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce596fa640 1 -- 172.21.15.94:0/1736897619 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce540c5d50 con 0x7fce540c69f0 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.217+0000 7fce4bfff640 1 -- 172.21.15.94:0/1736897619 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964510335 0 0) 0x7fce4c001490 con 0x7fce541558d0 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce596fa640 1 --2- 172.21.15.94:0/1736897619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c69f0 0x7fce540c6dd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fce440097a0 tx=0x7fce4402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d92783e601c572 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce4bfff640 1 -- 172.21.15.94:0/1736897619 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440315a0 con 0x7fce540c69f0 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce4bfff640 1 -- 172.21.15.94:0/1736897619 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44037070 con 0x7fce540c69f0 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce4bfff640 1 -- 172.21.15.94:0/1736897619 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44031960 con 0x7fce540c69f0 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce5b95e640 1 -- 172.21.15.94:0/1736897619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c69f0 msgr2=0x7fce540c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1736897619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c69f0 0x7fce540c6dd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fce440097a0 tx=0x7fce4402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce5b95e640 1 -- 172.21.15.94:0/1736897619 shutdown_connections 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1736897619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c69f0 0x7fce540c6dd0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce5b95e640 1 -- 172.21.15.94:0/1736897619 >> 172.21.15.94:0/1736897619 conn(0x7fce540cae70 msgr2=0x7fce540cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce5b95e640 1 -- 172.21.15.94:0/1736897619 shutdown_connections 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.218+0000 7fce5b95e640 1 -- 172.21.15.94:0/1736897619 wait complete. 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.219+0000 7fce5b95e640 1 Processor -- start 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.219+0000 7fce5b95e640 1 -- start start 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce5b95e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce540c69f0 0x7fce5406c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce5b95e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c8950 0x7fce5414c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce5b95e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce541558d0 0x7fce5414ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce5b95e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce54152a20 con 0x7fce540c8950 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce5b95e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce540c6310 con 0x7fce541558d0 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce5b95e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce54152ba0 con 0x7fce540c69f0 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c8950 0x7fce5414c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c8950 0x7fce5414c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40630/0 (socket says 172.21.15.94:40630) 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 -- 172.21.15.94:0/1053660996 learned_addr learned my addr 172.21.15.94:0/1053660996 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce596fa640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce540c69f0 0x7fce5406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce59efb640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce541558d0 0x7fce5414ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 -- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce540c69f0 msgr2=0x7fce5406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce540c69f0 0x7fce5406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 -- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce541558d0 msgr2=0x7fce5414ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce541558d0 0x7fce5414ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.220+0000 7fce58ef9640 1 -- 172.21.15.94:0/1053660996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce540b0c60 con 0x7fce540c8950 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.221+0000 7fce58ef9640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c8950 0x7fce5414c8e0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fce5000c460 tx=0x7fce5000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.221+0000 7fce49ffb640 1 -- 172.21.15.94:0/1053660996 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50016020 con 0x7fce540c8950 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.221+0000 7fce49ffb640 1 -- 172.21.15.94:0/1053660996 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce50012870 con 0x7fce540c8950 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.221+0000 7fce49ffb640 1 -- 172.21.15.94:0/1053660996 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50012c60 con 0x7fce540c8950 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.221+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce540b0980 con 0x7fce540c8950 2024-03-06T07:22:38.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.221+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce540c1ea0 con 0x7fce540c8950 2024-03-06T07:22:38.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.222+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5404f330 con 0x7fce540c8950 2024-03-06T07:22:38.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.223+0000 7fce49ffb640 1 -- 172.21.15.94:0/1053660996 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce50019020 con 0x7fce540c8950 2024-03-06T07:22:38.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.223+0000 7fce49ffb640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce3806b500 0x7fce3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.223+0000 7fce49ffb640 1 -- 172.21.15.94:0/1053660996 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fce500898b0 con 0x7fce540c8950 2024-03-06T07:22:38.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.223+0000 7fce596fa640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce3806b500 0x7fce3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.224+0000 7fce596fa640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce3806b500 0x7fce3806d9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fce44036000 tx=0x7fce4403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.227+0000 7fce49ffb640 1 -- 172.21.15.94:0/1053660996 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce50020070 con 0x7fce540c8950 2024-03-06T07:22:38.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:38.416+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fce540c4d80 con 0x7fce540c8950 2024-03-06T07:22:39.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.258+0000 7fce49ffb640 1 -- 172.21.15.94:0/1053660996 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v117) v1 ==== 97+0+0 (secure 0 0 0) 0x7fce50058290 con 0x7fce540c8950 2024-03-06T07:22:39.261 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:22:39.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce3806b500 msgr2=0x7fce3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce3806b500 0x7fce3806d9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fce44036000 tx=0x7fce4403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c8950 msgr2=0x7fce5414c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c8950 0x7fce5414c8e0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fce5000c460 tx=0x7fce5000c920 comp rx=0 tx=0).stop 2024-03-06T07:22:39.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 shutdown_connections 2024-03-06T07:22:39.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce3806b500 0x7fce3806d9c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce541558d0 0x7fce5414ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fce540c8950 0x7fce5414c8e0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 --2- 172.21.15.94:0/1053660996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce540c69f0 0x7fce5406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 >> 172.21.15.94:0/1053660996 conn(0x7fce540cae70 msgr2=0x7fce54151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 shutdown_connections 2024-03-06T07:22:39.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.261+0000 7fce5b95e640 1 -- 172.21.15.94:0/1053660996 wait complete. 2024-03-06T07:22:39.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:39.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.379+0000 7f5a2d81c640 1 Processor -- start 2024-03-06T07:22:39.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.379+0000 7f5a2d81c640 1 -- start start 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a2d81c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5a280c88c0 0x7f5a28155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a2d81c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5a28158b60 0x7f5a28158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a2d81c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a280c73d0 con 0x7f5a2814f360 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a2d81c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a280c8dd0 con 0x7f5a280c88c0 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a2d81c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a280be2e0 con 0x7f5a28158b60 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a267fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5a280c88c0 0x7f5a28155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a267fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5a280c88c0 0x7f5a28155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36084/0 (socket says 172.21.15.94:36084) 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a267fc640 1 -- 172.21.15.94:0/2648881085 learned_addr learned my addr 172.21.15.94:0/2648881085 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:39.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a267fc640 1 -- 172.21.15.94:0/2648881085 >> v2:172.21.15.145:3300/0 conn(0x7f5a28158b60 msgr2=0x7f5a28158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a267fc640 1 --2- 172.21.15.94:0/2648881085 >> v2:172.21.15.145:3300/0 conn(0x7f5a28158b60 0x7f5a28158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a267fc640 1 -- 172.21.15.94:0/2648881085 >> v1:172.21.15.145:6789/0 conn(0x7f5a2814f360 legacy=0x7f5a280c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.380+0000 7f5a267fc640 1 -- 172.21.15.94:0/2648881085 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a280c7670 con 0x7f5a280c88c0 2024-03-06T07:22:39.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.381+0000 7f5a267fc640 1 --2- 172.21.15.94:0/2648881085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a280c88c0 0x7f5a28155c10 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5a28152d60 tx=0x7f5a1c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af666f7a1eaa685f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.381+0000 7f5a25ffb640 1 -- 172.21.15.94:0/2648881085 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c02e070 con 0x7f5a280c88c0 2024-03-06T07:22:39.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.381+0000 7f5a25ffb640 1 -- 172.21.15.94:0/2648881085 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c037070 con 0x7f5a280c88c0 2024-03-06T07:22:39.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.381+0000 7f5a25ffb640 1 -- 172.21.15.94:0/2648881085 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c031a00 con 0x7f5a280c88c0 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2648881085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a280c88c0 msgr2=0x7f5a28155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2648881085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a280c88c0 0x7f5a28155c10 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5a28152d60 tx=0x7f5a1c02b880 comp rx=0 tx=0).stop 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2648881085 shutdown_connections 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2648881085 >> v2:172.21.15.145:3300/0 conn(0x7f5a28158b60 0x7f5a28158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2648881085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a280c88c0 0x7f5a28155c10 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2648881085 >> 172.21.15.94:0/2648881085 conn(0x7f5a280ac570 msgr2=0x7f5a280ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2648881085 shutdown_connections 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.382+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2648881085 wait complete. 2024-03-06T07:22:39.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.383+0000 7f5a2d81c640 1 Processor -- start 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.383+0000 7f5a2d81c640 1 -- start start 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a280c88c0 0x7f5a281eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a2814f360 0x7f5a281575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a28158b60 0x7f5a280bef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a280c5a50 con 0x7f5a2814f360 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a280c7c30 con 0x7f5a28158b60 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a280c5bd0 con 0x7f5a280c88c0 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a2814f360 0x7f5a281575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a2814f360 0x7f5a281575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36100/0 (socket says 172.21.15.94:36100) 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 -- 172.21.15.94:0/2906993562 learned_addr learned my addr 172.21.15.94:0/2906993562 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a26ffd640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a280c88c0 0x7f5a281eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a277fe640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a28158b60 0x7f5a280bef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 -- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a280c88c0 msgr2=0x7f5a281eb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a280c88c0 0x7f5a281eb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 -- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a28158b60 msgr2=0x7f5a280bef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a28158b60 0x7f5a280bef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 -- 172.21.15.94:0/2906993562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a281eef30 con 0x7f5a2814f360 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a277fe640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a28158b60 0x7f5a280bef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a267fc640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a2814f360 0x7f5a281575b0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c02b8c0 tx=0x7f5a1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a0ffff640 1 -- 172.21.15.94:0/2906993562 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c032880 con 0x7f5a2814f360 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a0ffff640 1 -- 172.21.15.94:0/2906993562 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c032a20 con 0x7f5a2814f360 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a281ef1d0 con 0x7f5a2814f360 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a0ffff640 1 -- 172.21.15.94:0/2906993562 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c032e10 con 0x7f5a2814f360 2024-03-06T07:22:39.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.384+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a280c10d0 con 0x7f5a2814f360 2024-03-06T07:22:39.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.385+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2804f330 con 0x7f5a2814f360 2024-03-06T07:22:39.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.386+0000 7f5a0ffff640 1 -- 172.21.15.94:0/2906993562 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a1c04c060 con 0x7f5a2814f360 2024-03-06T07:22:39.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.386+0000 7f5a0ffff640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.386+0000 7f5a0ffff640 1 -- 172.21.15.94:0/2906993562 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5a1c0abaf0 con 0x7f5a2814f360 2024-03-06T07:22:39.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.386+0000 7f5a26ffd640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.387+0000 7f5a26ffd640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a1406b4d0 0x7f5a1406d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5a180015d0 tx=0x7f5a18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.390+0000 7f5a0ffff640 1 -- 172.21.15.94:0/2906993562 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a1c040070 con 0x7f5a2814f360 2024-03-06T07:22:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.626+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5a280bece0 con 0x7f5a2814f360 2024-03-06T07:22:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.627+0000 7f5a0ffff640 1 -- 172.21.15.94:0/2906993562 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f5a1c07a690 con 0x7f5a2814f360 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:39.630 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.631 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:22:39.632 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.633 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.634 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.635 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.636 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.637 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:22:39.638 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.639 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:22:39.640 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.628+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a1406b4d0 msgr2=0x7f5a1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a1406b4d0 0x7f5a1406d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5a180015d0 tx=0x7f5a18009040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a2814f360 msgr2=0x7f5a281575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a2814f360 0x7f5a281575b0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c02b8c0 tx=0x7f5a1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 shutdown_connections 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a28158b60 0x7f5a280bef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5a2814f360 0x7f5a281575b0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 --2- 172.21.15.94:0/2906993562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a280c88c0 0x7f5a281eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 >> 172.21.15.94:0/2906993562 conn(0x7f5a280ac570 msgr2=0x7f5a280cb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 shutdown_connections 2024-03-06T07:22:39.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.629+0000 7f5a2d81c640 1 -- 172.21.15.94:0/2906993562 wait complete. 2024-03-06T07:22:39.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:39.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.741+0000 7f22b4f5e640 1 Processor -- start 2024-03-06T07:22:39.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.741+0000 7f22b4f5e640 1 -- start start 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22b4f5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22b4f5e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f22b01559e0 0x7f22b014ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22b4f5e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f22b00aa180 0x7f22b014f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22b4f5e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b00c4610 con 0x7f22b00aa180 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22b4f5e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b00aa660 con 0x7f22b01559e0 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22b4f5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00aa940 con 0x7f22b01550c0 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22aed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f22b00aa180 0x7f22b014f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22aed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f22b00aa180 0x7f22b014f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36104/0 (socket says 172.21.15.94:36104) 2024-03-06T07:22:39.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22aed9d640 1 -- 172.21.15.94:0/1124862645 learned_addr learned my addr 172.21.15.94:0/1124862645 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22add9b640 1 --2- 172.21.15.94:0/1124862645 >> v2:172.21.15.145:3300/0 conn(0x7f22b01559e0 0x7f22b014ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22ae59c640 1 --2- 172.21.15.94:0/1124862645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22aed9d640 1 -- 172.21.15.94:0/1124862645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 msgr2=0x7f22b01554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22aed9d640 1 --2- 172.21.15.94:0/1124862645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b01554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22aed9d640 1 -- 172.21.15.94:0/1124862645 >> v2:172.21.15.145:3300/0 conn(0x7f22b01559e0 msgr2=0x7f22b014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.742+0000 7f22aed9d640 1 --2- 172.21.15.94:0/1124862645 >> v2:172.21.15.145:3300/0 conn(0x7f22b01559e0 0x7f22b014ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.743+0000 7f22aed9d640 1 -- 172.21.15.94:0/1124862645 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b00c4860 con 0x7f22b00aa180 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.743+0000 7f22ae59c640 1 --2- 172.21.15.94:0/1124862645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b01554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.743+0000 7f22aed9d640 1 --2- 172.21.15.94:0/1124862645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b00aa180 0x7f22b014f140 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f22b00c77b0 tx=0x7f22a802dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76965788b6422a7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.743+0000 7f22ad59a640 1 -- 172.21.15.94:0/1124862645 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a802e970 con 0x7f22b00aa180 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.743+0000 7f22ad59a640 1 -- 172.21.15.94:0/1124862645 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a802eb10 con 0x7f22b00aa180 2024-03-06T07:22:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.743+0000 7f22ad59a640 1 -- 172.21.15.94:0/1124862645 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a8009600 con 0x7f22b00aa180 2024-03-06T07:22:39.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 -- 172.21.15.94:0/1124862645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b00aa180 msgr2=0x7f22b014f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/1124862645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b00aa180 0x7f22b014f140 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f22b00c77b0 tx=0x7f22a802dfd0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 -- 172.21.15.94:0/1124862645 shutdown_connections 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/1124862645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b00aa180 0x7f22b014f140 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/1124862645 >> v2:172.21.15.145:3300/0 conn(0x7f22b01559e0 0x7f22b014ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/1124862645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b01554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 -- 172.21.15.94:0/1124862645 >> 172.21.15.94:0/1124862645 conn(0x7f22b00c6710 msgr2=0x7f22b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 -- 172.21.15.94:0/1124862645 shutdown_connections 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.744+0000 7f22b4f5e640 1 -- 172.21.15.94:0/1124862645 wait complete. 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.745+0000 7f22b4f5e640 1 Processor -- start 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.745+0000 7f22b4f5e640 1 -- start start 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22b4f5e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22b00aa180 0x7f22b0142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22b4f5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b0141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22b4f5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b01559e0 0x7f22b0141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22b4f5e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00c3210 con 0x7f22b01559e0 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22b4f5e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00c1520 con 0x7f22b00aa180 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22b4f5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b00c3390 con 0x7f22b01550c0 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b01559e0 0x7f22b0141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b01559e0 0x7f22b0141dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36120/0 (socket says 172.21.15.94:36120) 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 -- 172.21.15.94:0/72083126 learned_addr learned my addr 172.21.15.94:0/72083126 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22add9b640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b0141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22ae59c640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22b00aa180 0x7f22b0142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 -- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 msgr2=0x7f22b0141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b0141890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 -- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22b00aa180 msgr2=0x7f22b0142cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22b00aa180 0x7f22b0142cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22aed9d640 1 -- 172.21.15.94:0/72083126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b0149190 con 0x7f22b01559e0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.746+0000 7f22add9b640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b0141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.747+0000 7f22aed9d640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b01559e0 0x7f22b0141dd0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f22a800d240 tx=0x7f22a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.747+0000 7f229effd640 1 -- 172.21.15.94:0/72083126 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a8045020 con 0x7f22b01559e0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.747+0000 7f229effd640 1 -- 172.21.15.94:0/72083126 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a8040660 con 0x7f22b01559e0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.747+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b0149470 con 0x7f22b01559e0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.747+0000 7f229effd640 1 -- 172.21.15.94:0/72083126 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a802e970 con 0x7f22b01559e0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.747+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b0143210 con 0x7f22b01559e0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.748+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b0075300 con 0x7f22b01559e0 2024-03-06T07:22:39.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.748+0000 7f229effd640 1 -- 172.21.15.94:0/72083126 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22a8048020 con 0x7f22b01559e0 2024-03-06T07:22:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.748+0000 7f229effd640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f229006b4d0 0x7f229006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.749+0000 7f229effd640 1 -- 172.21.15.94:0/72083126 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f22a80aea40 con 0x7f22b01559e0 2024-03-06T07:22:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.749+0000 7f22ae59c640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f229006b4d0 0x7f229006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.749+0000 7f22ae59c640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f229006b4d0 0x7f229006d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f22a0006b10 tx=0x7f22a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.754+0000 7f229effd640 1 -- 172.21.15.94:0/72083126 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22b0075300 con 0x7f22b01559e0 2024-03-06T07:22:39.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.924+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22b004f330 con 0x7f22b01559e0 2024-03-06T07:22:39.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.925+0000 7f229effd640 1 -- 172.21.15.94:0/72083126 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13174 (secure 0 0 0) 0x7f22a804f030 con 0x7f22b01559e0 2024-03-06T07:22:39.928 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:39.929 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":117,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:39.259193+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:39.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f229006b4d0 msgr2=0x7f229006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f229006b4d0 0x7f229006d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f22a0006b10 tx=0x7f22a0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:39.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b01559e0 msgr2=0x7f22b0141dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b01559e0 0x7f22b0141dd0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f22a800d240 tx=0x7f22a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 shutdown_connections 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f229006b4d0 0x7f229006d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22b01559e0 0x7f22b0141dd0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22b01550c0 0x7f22b0141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 --2- 172.21.15.94:0/72083126 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22b00aa180 0x7f22b0142cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 >> 172.21.15.94:0/72083126 conn(0x7f22b00c6710 msgr2=0x7f22b0152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 shutdown_connections 2024-03-06T07:22:39.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:39.927+0000 7f22b4f5e640 1 -- 172.21.15.94:0/72083126 wait complete. 2024-03-06T07:22:39.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:39.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:40.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.042+0000 7f08b86f4640 1 Processor -- start 2024-03-06T07:22:40.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.042+0000 7f08b86f4640 1 -- start start 2024-03-06T07:22:40.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b86f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f08b01538c0 0x7f08b01494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b86f4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b0154330 con 0x7f08b00ab310 2024-03-06T07:22:40.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b86f4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b0153dc0 con 0x7f08b00a97c0 2024-03-06T07:22:40.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b86f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b00c46a0 con 0x7f08b01538c0 2024-03-06T07:22:40.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b6490640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f08b00ab310 0x7f08b00ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33894/0 (socket says 172.21.15.94:33894) 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b6490640 1 -- 172.21.15.94:0/2352559698 learned_addr learned my addr 172.21.15.94:0/2352559698 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245416924 0 0) 0x7f08b0154330 con 0x7f08b00ab310 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f089c003520 con 0x7f08b00ab310 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.043+0000 7f08b5c8f640 1 --2- 172.21.15.94:0/2352559698 >> v2:172.21.15.145:3300/0 conn(0x7f08b01538c0 0x7f08b01494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232836997 0 0) 0x7f089c003520 con 0x7f08b00ab310 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 >> v2:172.21.15.145:3300/0 conn(0x7f08b01538c0 msgr2=0x7f08b01494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 --2- 172.21.15.94:0/2352559698 >> v2:172.21.15.145:3300/0 conn(0x7f08b01538c0 0x7f08b01494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 >> v1:172.21.15.145:6789/0 conn(0x7f08b00a97c0 legacy=0x7f08b00c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00c5d30 con 0x7f08b00ab310 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f08ac001670 con 0x7f08b00ab310 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 >> v1:172.21.15.94:6789/0 conn(0x7f08b00ab310 legacy=0x7f08b00ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f089c0039b0 0x7f089c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c004800 0x7f089c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f089c0051c0 0x7f089c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c46a0 con 0x7f089c0039b0 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c00c160 con 0x7f089c0051c0 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c00c360 con 0x7f089c004800 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6490640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f089c0039b0 0x7f089c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b5c8f640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c004800 0x7f089c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6c91640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f089c0051c0 0x7f089c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6490640 1 -- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c004800 msgr2=0x7f089c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6490640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c004800 0x7f089c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6490640 1 -- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f089c0051c0 msgr2=0x7f089c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6490640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f089c0051c0 0x7f089c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6c91640 1 -- 172.21.15.94:0/2352559698 reap_dead start 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.044+0000 7f08b6490640 1 -- 172.21.15.94:0/2352559698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c003520 con 0x7f089c0039b0 2024-03-06T07:22:40.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b6490640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f089c0039b0 0x7f089c003dd0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00c1c0 tx=0x7f089c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769873924b38abb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac007d00 con 0x7f089c0039b0 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac030070 con 0x7f089c0039b0 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b548e640 1 -- 172.21.15.94:0/2352559698 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac005d70 con 0x7f089c0039b0 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b86f4640 1 -- 172.21.15.94:0/2352559698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f089c0039b0 msgr2=0x7f089c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b86f4640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f089c0039b0 0x7f089c003dd0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f08ac00c1c0 tx=0x7f089c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b86f4640 1 -- 172.21.15.94:0/2352559698 shutdown_connections 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b86f4640 1 --2- 172.21.15.94:0/2352559698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f089c0039b0 0x7f089c003dd0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b86f4640 1 -- 172.21.15.94:0/2352559698 >> 172.21.15.94:0/2352559698 conn(0x7f08b00c6710 msgr2=0x7f08b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b86f4640 1 -- 172.21.15.94:0/2352559698 shutdown_connections 2024-03-06T07:22:40.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.045+0000 7f08b86f4640 1 -- 172.21.15.94:0/2352559698 wait complete. 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.046+0000 7f08b86f4640 1 Processor -- start 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b86f4640 1 -- start start 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b86f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c0039b0 0x7f08b01ec0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b86f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08b01ec600 0x7f08b00727e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b86f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08b0072d20 0x7f08b01e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b86f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00ab6f0 con 0x7f08b0072d20 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b86f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00c62f0 con 0x7f08b01ec600 2024-03-06T07:22:40.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b86f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00ab870 con 0x7f089c0039b0 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b6c91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08b0072d20 0x7f08b01e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b6c91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08b0072d20 0x7f08b01e88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36132/0 (socket says 172.21.15.94:36132) 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b6c91640 1 -- 172.21.15.94:0/121779794 learned_addr learned my addr 172.21.15.94:0/121779794 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b6490640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c0039b0 0x7f08b01ec0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b5c8f640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08b01ec600 0x7f08b00727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b6c91640 1 -- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c0039b0 msgr2=0x7f08b01ec0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.047+0000 7f08b6c91640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c0039b0 0x7f08b01ec0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08b6c91640 1 -- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08b01ec600 msgr2=0x7f08b00727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08b6c91640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08b01ec600 0x7f08b00727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08b6c91640 1 -- 172.21.15.94:0/121779794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b01eca80 con 0x7f08b0072d20 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08b6c91640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08b0072d20 0x7f08b01e88e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f08a800f750 tx=0x7f08a800fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08a6ffd640 1 -- 172.21.15.94:0/121779794 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a801a070 con 0x7f08b0072d20 2024-03-06T07:22:40.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08a6ffd640 1 -- 172.21.15.94:0/121779794 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a800fe60 con 0x7f08b0072d20 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b0072300 con 0x7f08b0072d20 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08a6ffd640 1 -- 172.21.15.94:0/121779794 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8003970 con 0x7f08b0072d20 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.048+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b00731a0 con 0x7f08b0072d20 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.049+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b004f330 con 0x7f08b0072d20 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.049+0000 7f08a6ffd640 1 -- 172.21.15.94:0/121779794 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08a801d070 con 0x7f08b0072d20 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.050+0000 7f08a6ffd640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f089406b4d0 0x7f089406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.050+0000 7f08a6ffd640 1 -- 172.21.15.94:0/121779794 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f08a8089310 con 0x7f08b0072d20 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.050+0000 7f08b6490640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f089406b4d0 0x7f089406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.051+0000 7f08b6490640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f089406b4d0 0x7f089406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f08ac034000 tx=0x7f08ac007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.054+0000 7f08a6ffd640 1 -- 172.21.15.94:0/121779794 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08a8057d90 con 0x7f08b0072d20 2024-03-06T07:22:40.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.226+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08b0154330 con 0x7f08b0072d20 2024-03-06T07:22:40.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.227+0000 7f08a6ffd640 1 -- 172.21.15.94:0/121779794 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08a8087e20 con 0x7f08b0072d20 2024-03-06T07:22:40.230 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f089406b4d0 msgr2=0x7f089406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f089406b4d0 0x7f089406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f08ac034000 tx=0x7f08ac007000 comp rx=0 tx=0).stop 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08b0072d20 msgr2=0x7f08b01e88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08b0072d20 0x7f08b01e88e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f08a800f750 tx=0x7f08a800fb50 comp rx=0 tx=0).stop 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 shutdown_connections 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f089406b4d0 0x7f089406d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08b0072d20 0x7f08b01e88e0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08b01ec600 0x7f08b00727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 --2- 172.21.15.94:0/121779794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f089c0039b0 0x7f08b01ec0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 >> 172.21.15.94:0/121779794 conn(0x7f08b00c6710 msgr2=0x7f08b00c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 shutdown_connections 2024-03-06T07:22:40.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.229+0000 7f08b86f4640 1 -- 172.21.15.94:0/121779794 wait complete. 2024-03-06T07:22:40.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:40.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.340+0000 7f2697bb6640 1 Processor -- start 2024-03-06T07:22:40.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.340+0000 7f2697bb6640 1 -- start start 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2697bb6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2697bb6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f26900c1aa0 0x7f26900c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2697bb6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26900c28d0 con 0x7f26900c5c60 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2697bb6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26900c2680 con 0x7f26900c1aa0 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2697bb6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26900c2bb0 con 0x7f269014f790 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2695151640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f26900c1aa0 0x7f26900c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2696153640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f26900c5c60 0x7f2690155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33904/0 (socket says 172.21.15.94:33904) 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2696153640 1 -- 172.21.15.94:0/354153247 learned_addr learned my addr 172.21.15.94:0/354153247 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2695952640 1 --2- 172.21.15.94:0/354153247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2695151640 1 -- 172.21.15.94:0/354153247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 msgr2=0x7f26901529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2695151640 1 --2- 172.21.15.94:0/354153247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2695151640 1 -- 172.21.15.94:0/354153247 >> v1:172.21.15.94:6789/0 conn(0x7f26900c5c60 legacy=0x7f2690155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2694950640 1 -- 172.21.15.94:0/354153247 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557574259 0 0) 0x7f26900c28d0 con 0x7f26900c5c60 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.341+0000 7f2695151640 1 -- 172.21.15.94:0/354153247 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26900c2e90 con 0x7f26900c1aa0 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.342+0000 7f2695151640 1 --2- 172.21.15.94:0/354153247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c1aa0 0x7f26900c5720 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f26800097a0 tx=0x7f268002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a173a2d0d7c07491 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.342+0000 7f2694950640 1 -- 172.21.15.94:0/354153247 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680035020 con 0x7f26900c1aa0 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.342+0000 7f2694950640 1 -- 172.21.15.94:0/354153247 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2680031830 con 0x7f26900c1aa0 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.342+0000 7f2694950640 1 -- 172.21.15.94:0/354153247 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680031c00 con 0x7f26900c1aa0 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 -- 172.21.15.94:0/354153247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c1aa0 msgr2=0x7f26900c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 --2- 172.21.15.94:0/354153247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c1aa0 0x7f26900c5720 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f26800097a0 tx=0x7f268002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 -- 172.21.15.94:0/354153247 shutdown_connections 2024-03-06T07:22:40.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 --2- 172.21.15.94:0/354153247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c1aa0 0x7f26900c5720 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 --2- 172.21.15.94:0/354153247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 -- 172.21.15.94:0/354153247 >> 172.21.15.94:0/354153247 conn(0x7f26900c6710 msgr2=0x7f26900c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 -- 172.21.15.94:0/354153247 shutdown_connections 2024-03-06T07:22:40.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.343+0000 7f2697bb6640 1 -- 172.21.15.94:0/354153247 wait complete. 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 Processor -- start 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 -- start start 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26900c1aa0 0x7f2690152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c5c60 0x7f26901e8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901e8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269006cc00 con 0x7f26900c5c60 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26900c0d60 con 0x7f26900c1aa0 2024-03-06T07:22:40.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2697bb6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269006cd80 con 0x7f269014f790 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2695151640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c5c60 0x7f26901e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2695151640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c5c60 0x7f26901e8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36148/0 (socket says 172.21.15.94:36148) 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.344+0000 7f2695151640 1 -- 172.21.15.94:0/1706479716 learned_addr learned my addr 172.21.15.94:0/1706479716 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695952640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26900c1aa0 0x7f2690152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2696153640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901e8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695151640 1 -- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 msgr2=0x7f26901e8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695151640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901e8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695151640 1 -- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26900c1aa0 msgr2=0x7f2690152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695151640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26900c1aa0 0x7f2690152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695151640 1 -- 172.21.15.94:0/1706479716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26901550a0 con 0x7f26900c5c60 2024-03-06T07:22:40.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695952640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26900c1aa0 0x7f2690152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2696153640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901e8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2695151640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c5c60 0x7f26901e8450 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f26800098c0 tx=0x7f26800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f26867fc640 1 -- 172.21.15.94:0/1706479716 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268003ba00 con 0x7f26900c5c60 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f26867fc640 1 -- 172.21.15.94:0/1706479716 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268003bba0 con 0x7f26900c5c60 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2690072850 con 0x7f26900c5c60 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f26867fc640 1 -- 172.21.15.94:0/1706479716 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26800453f0 con 0x7f26900c5c60 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.345+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2690071c50 con 0x7f26900c5c60 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.346+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269004f330 con 0x7f26900c5c60 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.347+0000 7f26867fc640 1 -- 172.21.15.94:0/1706479716 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2680038020 con 0x7f26900c5c60 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.347+0000 7f26867fc640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f267406b4d0 0x7f267406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.347+0000 7f26867fc640 1 -- 172.21.15.94:0/1706479716 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f26800abd80 con 0x7f26900c5c60 2024-03-06T07:22:40.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.347+0000 7f2695952640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f267406b4d0 0x7f267406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.348+0000 7f2695952640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f267406b4d0 0x7f267406d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f268c006fd0 tx=0x7f268c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.351+0000 7f26867fc640 1 -- 172.21.15.94:0/1706479716 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26901550a0 con 0x7f26900c5c60 2024-03-06T07:22:40.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.526+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f26900c1f40 con 0x7f26900c5c60 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.527+0000 7f26867fc640 1 -- 172.21.15.94:0/1706479716 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v117) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2680040070 con 0x7f26900c5c60 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:22:40.530 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f267406b4d0 msgr2=0x7f267406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f267406b4d0 0x7f267406d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f268c006fd0 tx=0x7f268c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c5c60 msgr2=0x7f26901e8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c5c60 0x7f26901e8450 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f26800098c0 tx=0x7f26800046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 shutdown_connections 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f267406b4d0 0x7f267406d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f269014f790 0x7f26901e8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26900c5c60 0x7f26901e8450 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 --2- 172.21.15.94:0/1706479716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26900c1aa0 0x7f2690152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 >> 172.21.15.94:0/1706479716 conn(0x7f26900c6710 msgr2=0x7f26900a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 shutdown_connections 2024-03-06T07:22:40.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.529+0000 7f2697bb6640 1 -- 172.21.15.94:0/1706479716 wait complete. 2024-03-06T07:22:40.542 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:40.543 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.643+0000 7f95537b6640 1 Processor -- start 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.643+0000 7f95537b6640 1 -- start start 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f95537b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f954c0c5a20 0x7f954c1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f95537b6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c155d10 con 0x7f954c14bb20 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f95537b6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c159260 con 0x7f954c0c2330 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f95537b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c0c3070 con 0x7f954c0c5a20 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f9551552640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f954c14bb20 0x7f954c0c5310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33912/0 (socket says 172.21.15.94:33912) 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f9551552640 1 -- 172.21.15.94:0/2518116187 learned_addr learned my addr 172.21.15.94:0/2518116187 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171086580 0 0) 0x7f954c155d10 con 0x7f954c14bb20 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.644+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9538003520 con 0x7f954c14bb20 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551d53640 1 --2- 172.21.15.94:0/2518116187 >> v2:172.21.15.145:3300/0 conn(0x7f954c0c5a20 0x7f954c1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379440380 0 0) 0x7f9538003520 con 0x7f954c14bb20 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 >> v2:172.21.15.145:3300/0 conn(0x7f954c0c5a20 msgr2=0x7f954c1557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 --2- 172.21.15.94:0/2518116187 >> v2:172.21.15.145:3300/0 conn(0x7f954c0c5a20 0x7f954c1557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 >> v1:172.21.15.145:6789/0 conn(0x7f954c0c2330 legacy=0x7f954c1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c0a7d70 con 0x7f954c14bb20 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f95480019f0 con 0x7f954c14bb20 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 >> v1:172.21.15.94:6789/0 conn(0x7f954c14bb20 legacy=0x7f954c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95380039b0 0x7f9538003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9538004800 0x7f9538004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380051c0 0x7f953800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0c3070 con 0x7f95380039b0 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953800c160 con 0x7f9538004800 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953800c360 con 0x7f95380051c0 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551552640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95380039b0 0x7f9538003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9550d51640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9538004800 0x7f9538004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551d53640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380051c0 0x7f953800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551552640 1 -- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380051c0 msgr2=0x7f953800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551552640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380051c0 0x7f953800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551552640 1 -- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9538004800 msgr2=0x7f9538004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551552640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9538004800 0x7f9538004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551552640 1 -- 172.21.15.94:0/2518116187 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9538003520 con 0x7f95380039b0 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551d53640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380051c0 0x7f953800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:40.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.645+0000 7f9551d53640 1 -- 172.21.15.94:0/2518116187 reap_dead start 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.646+0000 7f9551552640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95380039b0 0x7f9538003dd0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f954800a860 tx=0x7f9538005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf61f8cdd4d0250 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.646+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954800a480 con 0x7f95380039b0 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.646+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9548034630 con 0x7f95380039b0 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.646+0000 7f9543fff640 1 -- 172.21.15.94:0/2518116187 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9548034aa0 con 0x7f95380039b0 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.647+0000 7f95537b6640 1 -- 172.21.15.94:0/2518116187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95380039b0 msgr2=0x7f9538003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.647+0000 7f95537b6640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95380039b0 0x7f9538003dd0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f954800a860 tx=0x7f9538005830 comp rx=0 tx=0).stop 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.647+0000 7f95537b6640 1 -- 172.21.15.94:0/2518116187 shutdown_connections 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.647+0000 7f95537b6640 1 --2- 172.21.15.94:0/2518116187 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95380039b0 0x7f9538003dd0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.647+0000 7f95537b6640 1 -- 172.21.15.94:0/2518116187 >> 172.21.15.94:0/2518116187 conn(0x7f954c0c88f0 msgr2=0x7f954c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.647+0000 7f95537b6640 1 -- 172.21.15.94:0/2518116187 shutdown_connections 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.647+0000 7f95537b6640 1 -- 172.21.15.94:0/2518116187 wait complete. 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 Processor -- start 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 -- start start 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380039b0 0x7f954c157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f954c0ab760 0x7f954c1eabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f954c14b230 0x7f954c1eb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c159540 con 0x7f954c14b230 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0a80e0 con 0x7f954c0ab760 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.648+0000 7f95537b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c14bf00 con 0x7f95380039b0 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f954c14b230 0x7f954c1eb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f954c14b230 0x7f954c1eb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36180/0 (socket says 172.21.15.94:36180) 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 -- 172.21.15.94:0/577740987 learned_addr learned my addr 172.21.15.94:0/577740987 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9550d51640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f954c0ab760 0x7f954c1eabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551552640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380039b0 0x7f954c157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 -- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380039b0 msgr2=0x7f954c157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380039b0 0x7f954c157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 -- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f954c0ab760 msgr2=0x7f954c1eabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f954c0ab760 0x7f954c1eabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 -- 172.21.15.94:0/577740987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c14a8c0 con 0x7f954c14b230 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9551d53640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f954c14b230 0x7f954c1eb110 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f954c073570 tx=0x7f9544009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9541ffb640 1 -- 172.21.15.94:0/577740987 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9544019020 con 0x7f954c14b230 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9541ffb640 1 -- 172.21.15.94:0/577740987 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9544003610 con 0x7f954c14b230 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954c1ee2c0 con 0x7f954c14b230 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f9541ffb640 1 -- 172.21.15.94:0/577740987 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9544003a00 con 0x7f954c14b230 2024-03-06T07:22:40.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.649+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c1582f0 con 0x7f954c14b230 2024-03-06T07:22:40.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.650+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954c04f330 con 0x7f954c14b230 2024-03-06T07:22:40.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.651+0000 7f9541ffb640 1 -- 172.21.15.94:0/577740987 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f954401c020 con 0x7f954c14b230 2024-03-06T07:22:40.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.651+0000 7f9541ffb640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f953006b4d0 0x7f953006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.651+0000 7f9541ffb640 1 -- 172.21.15.94:0/577740987 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f954408a4f0 con 0x7f954c14b230 2024-03-06T07:22:40.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.651+0000 7f9551552640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f953006b4d0 0x7f953006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.652+0000 7f9551552640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f953006b4d0 0x7f953006d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9548034000 tx=0x7f954802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.656+0000 7f9541ffb640 1 -- 172.21.15.94:0/577740987 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9544023020 con 0x7f954c14b230 2024-03-06T07:22:40.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:40.828+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f954c1527b0 con 0x7f954c14b230 2024-03-06T07:22:41.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.263+0000 7f9541ffb640 1 -- 172.21.15.94:0/577740987 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v118) v1 ==== 138+0+0 (secure 0 0 0) 0x7f95440595c0 con 0x7f954c14b230 2024-03-06T07:22:41.266 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:41.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f953006b4d0 msgr2=0x7f953006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f953006b4d0 0x7f953006d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9548034000 tx=0x7f954802f000 comp rx=0 tx=0).stop 2024-03-06T07:22:41.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f954c14b230 msgr2=0x7f954c1eb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f954c14b230 0x7f954c1eb110 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f954c073570 tx=0x7f9544009f90 comp rx=0 tx=0).stop 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 shutdown_connections 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f953006b4d0 0x7f953006d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f954c14b230 0x7f954c1eb110 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f954c0ab760 0x7f954c1eabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 --2- 172.21.15.94:0/577740987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95380039b0 0x7f954c157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 >> 172.21.15.94:0/577740987 conn(0x7f954c0c88f0 msgr2=0x7f954c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 shutdown_connections 2024-03-06T07:22:41.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.267+0000 7f95537b6640 1 -- 172.21.15.94:0/577740987 wait complete. 2024-03-06T07:22:41.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.390+0000 7f012b6b8640 1 Processor -- start 2024-03-06T07:22:41.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.390+0000 7f012b6b8640 1 -- start start 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.390+0000 7f012b6b8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 0x7f012406cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.390+0000 7f012b6b8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01240c2790 con 0x7f012414ef70 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.390+0000 7f012b6b8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01240c1540 con 0x7f012406c1f0 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.390+0000 7f012b6b8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01240c3090 con 0x7f01240c1820 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f0129454640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f012414ef70 0x7f01240c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33928/0 (socket says 172.21.15.94:33928) 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f0129454640 1 -- 172.21.15.94:0/2908102294 learned_addr learned my addr 172.21.15.94:0/2908102294 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f0129c55640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 0x7f012406cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261171897 0 0) 0x7f01240c2790 con 0x7f012414ef70 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110004630 con 0x7f012414ef70 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096552712 0 0) 0x7f01240c1540 con 0x7f012406c1f0 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01240c2790 con 0x7f012406c1f0 2024-03-06T07:22:41.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533025585 0 0) 0x7f0110004630 con 0x7f012414ef70 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 msgr2=0x7f012406cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 0x7f012406cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 >> v1:172.21.15.145:6789/0 conn(0x7f012406c1f0 legacy=0x7f0124154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01240c24b0 con 0x7f012414ef70 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f0120001670 con 0x7f012414ef70 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f0129c55640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 0x7f012406cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 >> v1:172.21.15.94:6789/0 conn(0x7f012414ef70 legacy=0x7f01240c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110004c80 0x7f0110005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0110005910 0x7f0110005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 0x7f01100062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01240c3090 con 0x7f0110005910 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011000cbb0 con 0x7f0110004c80 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011000cdb0 con 0x7f01240c1820 2024-03-06T07:22:41.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f0128c53640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0110005910 0x7f0110005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.391+0000 7f0129454640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110004c80 0x7f0110005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0129c55640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 0x7f01100062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0128c53640 1 -- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 msgr2=0x7f01100062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0128c53640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01240c1820 0x7f01100062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0128c53640 1 -- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110004c80 msgr2=0x7f0110005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0128c53640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110004c80 0x7f0110005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0128c53640 1 -- 172.21.15.94:0/2908102294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0110004630 con 0x7f0110005910 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0129454640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110004c80 0x7f0110005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f0128c53640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0110005910 0x7f0110005d90 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f01140018a0 tx=0x7f0110006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438ee004a3f05ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.392+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114037070 con 0x7f0110005910 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01140052d0 con 0x7f0110005910 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f011bfff640 1 -- 172.21.15.94:0/2908102294 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01140014f0 con 0x7f0110005910 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f012b6b8640 1 -- 172.21.15.94:0/2908102294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0110005910 msgr2=0x7f0110005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f012b6b8640 1 --2- 172.21.15.94:0/2908102294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0110005910 0x7f0110005d90 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f01140018a0 tx=0x7f0110006940 comp rx=0 tx=0).stop 2024-03-06T07:22:41.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f0129c55640 1 -- 172.21.15.94:0/2908102294 reap_dead start 2024-03-06T07:22:41.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f012b6b8640 1 -- 172.21.15.94:0/2908102294 shutdown_connections 2024-03-06T07:22:41.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f012b6b8640 1 -- 172.21.15.94:0/2908102294 >> 172.21.15.94:0/2908102294 conn(0x7f01240c6710 msgr2=0x7f01240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f012b6b8640 1 -- 172.21.15.94:0/2908102294 shutdown_connections 2024-03-06T07:22:41.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.393+0000 7f012b6b8640 1 -- 172.21.15.94:0/2908102294 wait complete. 2024-03-06T07:22:41.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.394+0000 7f012b6b8640 1 Processor -- start 2024-03-06T07:22:41.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.394+0000 7f012b6b8640 1 -- start start 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f012b6b8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110005910 0x7f01241e86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f012b6b8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0124145550 0x7f01240bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f012b6b8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01240c0080 0x7f01241ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f012b6b8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0124155340 con 0x7f01240c0080 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f012b6b8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012414f350 con 0x7f0110005910 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f012b6b8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01241554c0 con 0x7f0124145550 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01240c0080 0x7f01241ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01240c0080 0x7f01241ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36188/0 (socket says 172.21.15.94:36188) 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 -- 172.21.15.94:0/596854477 learned_addr learned my addr 172.21.15.94:0/596854477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0128c53640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0124145550 0x7f01240bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129454640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110005910 0x7f01241e86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 -- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0124145550 msgr2=0x7f01240bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0124145550 0x7f01240bfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 -- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110005910 msgr2=0x7f01241e86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110005910 0x7f01241e86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 -- 172.21.15.94:0/596854477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01241e8c00 con 0x7f01240c0080 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0128c53640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0124145550 0x7f01240bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129454640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110005910 0x7f01241e86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.395+0000 7f0129c55640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01240c0080 0x7f01241ee560 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f012406e5a0 tx=0x7f011c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.396+0000 7f0119ffb640 1 -- 172.21.15.94:0/596854477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c011e00 con 0x7f01240c0080 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.396+0000 7f0119ffb640 1 -- 172.21.15.94:0/596854477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c01b070 con 0x7f01240c0080 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.396+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01241eb1a0 con 0x7f01240c0080 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.396+0000 7f0119ffb640 1 -- 172.21.15.94:0/596854477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c003990 con 0x7f01240c0080 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.396+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01241eb760 con 0x7f01240c0080 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.397+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012404f330 con 0x7f01240c0080 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.397+0000 7f0119ffb640 1 -- 172.21.15.94:0/596854477 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f011c005d10 con 0x7f01240c0080 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.398+0000 7f0119ffb640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f010806b500 0x7f010806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.398+0000 7f0119ffb640 1 -- 172.21.15.94:0/596854477 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f011c08a580 con 0x7f01240c0080 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.398+0000 7f0129454640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f010806b500 0x7f010806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.399+0000 7f0129454640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f010806b500 0x7f010806d9c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0120001770 tx=0x7f0120000f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.402+0000 7f0119ffb640 1 -- 172.21.15.94:0/596854477 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f011c014350 con 0x7f01240c0080 2024-03-06T07:22:41.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:41.574+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0124152530 con 0x7f01240c0080 2024-03-06T07:22:42.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.273+0000 7f0119ffb640 1 -- 172.21.15.94:0/596854477 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v119) v1 ==== 170+0+0 (secure 0 0 0) 0x7f011c059450 con 0x7f01240c0080 2024-03-06T07:22:42.275 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:22:42.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.275+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f010806b500 msgr2=0x7f010806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.275+0000 7f012b6b8640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f010806b500 0x7f010806d9c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0120001770 tx=0x7f0120000f90 comp rx=0 tx=0).stop 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.275+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01240c0080 msgr2=0x7f01241ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.275+0000 7f012b6b8640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01240c0080 0x7f01241ee560 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f012406e5a0 tx=0x7f011c009f90 comp rx=0 tx=0).stop 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.276+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 shutdown_connections 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.276+0000 7f012b6b8640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f010806b500 0x7f010806d9c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.276+0000 7f012b6b8640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01240c0080 0x7f01241ee560 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.276+0000 7f012b6b8640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0124145550 0x7f01240bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.276+0000 7f012b6b8640 1 --2- 172.21.15.94:0/596854477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0110005910 0x7f01241e86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.276+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 >> 172.21.15.94:0/596854477 conn(0x7f01240c6710 msgr2=0x7f0124147340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.277+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 shutdown_connections 2024-03-06T07:22:42.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.280+0000 7f012b6b8640 1 -- 172.21.15.94:0/596854477 wait complete. 2024-03-06T07:22:42.303 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:42.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.404+0000 7f1f07a81640 1 Processor -- start 2024-03-06T07:22:42.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.404+0000 7f1f07a81640 1 -- start start 2024-03-06T07:22:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f07a81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000a8ff0 0x7f1f000ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f07a81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1f000ab730 0x7f1f000aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f07a81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000c1f90 con 0x7f1f000a6290 2024-03-06T07:22:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f07a81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0014f850 con 0x7f1f000ab730 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f07a81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00156260 con 0x7f1f000a8ff0 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f0581d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000a8ff0 0x7f1f000ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f0601e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1f000ab730 0x7f1f000aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f0501c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f1f000a6290 0x7f1f000a9e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:59690/0 (socket says 172.21.15.94:59690) 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.405+0000 7f1f0501c640 1 -- 172.21.15.94:0/148065166 learned_addr learned my addr 172.21.15.94:0/148065166 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.406+0000 7f1f0581d640 1 -- 172.21.15.94:0/148065166 >> v2:172.21.15.145:3300/0 conn(0x7f1f000ab730 msgr2=0x7f1f000aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.406+0000 7f1f0581d640 1 --2- 172.21.15.94:0/148065166 >> v2:172.21.15.145:3300/0 conn(0x7f1f000ab730 0x7f1f000aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.406+0000 7f1f0581d640 1 -- 172.21.15.94:0/148065166 >> v1:172.21.15.145:6789/0 conn(0x7f1f000a6290 legacy=0x7f1f000a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.406+0000 7f1f0581d640 1 -- 172.21.15.94:0/148065166 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000c11d0 con 0x7f1f000a8ff0 2024-03-06T07:22:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.406+0000 7f1f0601e640 1 --2- 172.21.15.94:0/148065166 >> v2:172.21.15.145:3300/0 conn(0x7f1f000ab730 0x7f1f000aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:42.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.406+0000 7f1f0581d640 1 --2- 172.21.15.94:0/148065166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000a8ff0 0x7f1f000ab1f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0097a0 tx=0x7f1efc02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecb2434b23fa999b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f0481b640 1 -- 172.21.15.94:0/148065166 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc038070 con 0x7f1f000a8ff0 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f0481b640 1 -- 172.21.15.94:0/148065166 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc031700 con 0x7f1f000a8ff0 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f0481b640 1 -- 172.21.15.94:0/148065166 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc031ad0 con 0x7f1f000a8ff0 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 -- 172.21.15.94:0/148065166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000a8ff0 msgr2=0x7f1f000ab1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 --2- 172.21.15.94:0/148065166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000a8ff0 0x7f1f000ab1f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0097a0 tx=0x7f1efc02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 -- 172.21.15.94:0/148065166 shutdown_connections 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 --2- 172.21.15.94:0/148065166 >> v2:172.21.15.145:3300/0 conn(0x7f1f000ab730 0x7f1f000aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 --2- 172.21.15.94:0/148065166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000a8ff0 0x7f1f000ab1f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 -- 172.21.15.94:0/148065166 >> 172.21.15.94:0/148065166 conn(0x7f1f0006afc0 msgr2=0x7f1f0006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 -- 172.21.15.94:0/148065166 shutdown_connections 2024-03-06T07:22:42.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.407+0000 7f1f07a81640 1 -- 172.21.15.94:0/148065166 wait complete. 2024-03-06T07:22:42.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.408+0000 7f1f07a81640 1 Processor -- start 2024-03-06T07:22:42.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.408+0000 7f1f07a81640 1 -- start start 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f07a81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f000a8ff0 0x7f1f001e8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f07a81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f000ab730 0x7f1f00152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f07a81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000c0260 0x7f1f001ec820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f07a81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000c1a70 con 0x7f1f000a8ff0 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f07a81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000c1790 con 0x7f1f000ab730 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f07a81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00155cf0 con 0x7f1f000c0260 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f000a8ff0 0x7f1f001e8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f000a8ff0 0x7f1f001e8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36202/0 (socket says 172.21.15.94:36202) 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 -- 172.21.15.94:0/3662932273 learned_addr learned my addr 172.21.15.94:0/3662932273 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0601e640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000c0260 0x7f1f001ec820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0501c640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f000ab730 0x7f1f00152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 -- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000c0260 msgr2=0x7f1f001ec820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000c0260 0x7f1f001ec820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 -- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f000ab730 msgr2=0x7f1f00152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f000ab730 0x7f1f00152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0581d640 1 -- 172.21.15.94:0/3662932273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000717d0 con 0x7f1f000a8ff0 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.409+0000 7f1f0501c640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f000ab730 0x7f1f00152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.410+0000 7f1f0581d640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f000a8ff0 0x7f1f001e8b40 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1efc036040 tx=0x7f1efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.410+0000 7f1ef67fc640 1 -- 172.21.15.94:0/3662932273 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc042070 con 0x7f1f000a8ff0 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.410+0000 7f1ef67fc640 1 -- 172.21.15.94:0/3662932273 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc03d950 con 0x7f1f000a8ff0 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.410+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f00071a70 con 0x7f1f000a8ff0 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.410+0000 7f1ef67fc640 1 -- 172.21.15.94:0/3662932273 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc038040 con 0x7f1f000a8ff0 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.410+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f00152560 con 0x7f1f000a8ff0 2024-03-06T07:22:42.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.411+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f001ec0f0 con 0x7f1f000a8ff0 2024-03-06T07:22:42.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.411+0000 7f1ef67fc640 1 -- 172.21.15.94:0/3662932273 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1efc03b070 con 0x7f1f000a8ff0 2024-03-06T07:22:42.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.412+0000 7f1ef67fc640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ee406b500 0x7f1ee406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.412+0000 7f1ef67fc640 1 -- 172.21.15.94:0/3662932273 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f1efc0aba30 con 0x7f1f000a8ff0 2024-03-06T07:22:42.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.412+0000 7f1f0501c640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ee406b500 0x7f1ee406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.413+0000 7f1f0501c640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ee406b500 0x7f1ee406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1f00073650 tx=0x7f1ef0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.416+0000 7f1ef67fc640 1 -- 172.21.15.94:0/3662932273 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1efc030070 con 0x7f1f000a8ff0 2024-03-06T07:22:42.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:42.623+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1f0006b860 con 0x7f1f000a8ff0 2024-03-06T07:22:43.285 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 19 and data pool 20 2024-03-06T07:22:43.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.283+0000 7f1ef67fc640 1 -- 172.21.15.94:0/3662932273 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v102) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1efc07a130 con 0x7f1f000a8ff0 2024-03-06T07:22:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.297+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ee406b500 msgr2=0x7f1ee406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.297+0000 7f1f07a81640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ee406b500 0x7f1ee406d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1f00073650 tx=0x7f1ef0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.297+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f000a8ff0 msgr2=0x7f1f001e8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.297+0000 7f1f07a81640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f000a8ff0 0x7f1f001e8b40 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f1efc036040 tx=0x7f1efc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.298+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 shutdown_connections 2024-03-06T07:22:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.298+0000 7f1f07a81640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ee406b500 0x7f1ee406d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.298+0000 7f1f07a81640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f000c0260 0x7f1f001ec820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.298+0000 7f1f07a81640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f000ab730 0x7f1f00152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.298+0000 7f1f07a81640 1 --2- 172.21.15.94:0/3662932273 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1f000a8ff0 0x7f1f001e8b40 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.298+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 >> 172.21.15.94:0/3662932273 conn(0x7f1f0006afc0 msgr2=0x7f1f000c4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.298+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 shutdown_connections 2024-03-06T07:22:43.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.300+0000 7f1f07a81640 1 -- 172.21.15.94:0/3662932273 wait complete. 2024-03-06T07:22:43.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:43.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.421+0000 7f154e0c2640 1 Processor -- start 2024-03-06T07:22:43.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.421+0000 7f154e0c2640 1 -- start start 2024-03-06T07:22:43.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f154e0c2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15480c1320 0x7f15481566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f154e0c2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f15480c6000 0x7f1548156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f154e0c2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1548153a60 con 0x7f1548150580 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f154e0c2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1548153d40 con 0x7f15480c1320 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f154e0c2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154806c2c0 con 0x7f15480c6000 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f1546ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15480c1320 0x7f15481566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f15477fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1548150580 0x7f154806dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33938/0 (socket says 172.21.15.94:33938) 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f1546ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15480c1320 0x7f15481566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36206/0 (socket says 172.21.15.94:36206) 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f15477fe640 1 -- 172.21.15.94:0/96138784 learned_addr learned my addr 172.21.15.94:0/96138784 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f1546ffd640 1 -- 172.21.15.94:0/96138784 >> v2:172.21.15.145:3300/0 conn(0x7f15480c6000 msgr2=0x7f1548156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f1547fff640 1 --2- 172.21.15.94:0/96138784 >> v2:172.21.15.145:3300/0 conn(0x7f15480c6000 0x7f1548156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f15467fc640 1 -- 172.21.15.94:0/96138784 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648401483 0 0) 0x7f1548153a60 con 0x7f1548150580 2024-03-06T07:22:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f1546ffd640 1 --2- 172.21.15.94:0/96138784 >> v2:172.21.15.145:3300/0 conn(0x7f15480c6000 0x7f1548156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f1546ffd640 1 -- 172.21.15.94:0/96138784 >> v1:172.21.15.94:6789/0 conn(0x7f1548150580 legacy=0x7f154806dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.422+0000 7f1546ffd640 1 -- 172.21.15.94:0/96138784 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154806c5a0 con 0x7f15480c1320 2024-03-06T07:22:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.423+0000 7f1546ffd640 1 --2- 172.21.15.94:0/96138784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c1320 0x7f15481566b0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f15480c5960 tx=0x7f153c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bdbd78accdb3d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.423+0000 7f15467fc640 1 -- 172.21.15.94:0/96138784 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c0315a0 con 0x7f15480c1320 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.423+0000 7f15467fc640 1 -- 172.21.15.94:0/96138784 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c037070 con 0x7f15480c1320 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.423+0000 7f15467fc640 1 -- 172.21.15.94:0/96138784 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c031960 con 0x7f15480c1320 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 -- 172.21.15.94:0/96138784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c1320 msgr2=0x7f15481566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 --2- 172.21.15.94:0/96138784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c1320 0x7f15481566b0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f15480c5960 tx=0x7f153c02b840 comp rx=0 tx=0).stop 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 -- 172.21.15.94:0/96138784 shutdown_connections 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 --2- 172.21.15.94:0/96138784 >> v2:172.21.15.145:3300/0 conn(0x7f15480c6000 0x7f1548156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 --2- 172.21.15.94:0/96138784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c1320 0x7f15481566b0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 -- 172.21.15.94:0/96138784 >> 172.21.15.94:0/96138784 conn(0x7f15480c6710 msgr2=0x7f15480c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 -- 172.21.15.94:0/96138784 shutdown_connections 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.424+0000 7f154e0c2640 1 -- 172.21.15.94:0/96138784 wait complete. 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.425+0000 7f154e0c2640 1 Processor -- start 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.425+0000 7f154e0c2640 1 -- start start 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f154e0c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15480c1320 0x7f1548148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f154e0c2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c6000 0x7f1548146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f154e0c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1548150580 0x7f15481470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f154e0c2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154806e5a0 con 0x7f15480c6000 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f154e0c2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154806e420 con 0x7f15480c1320 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f154e0c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15480c0c40 con 0x7f1548150580 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c6000 0x7f1548146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c6000 0x7f1548146ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36208/0 (socket says 172.21.15.94:36208) 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 -- 172.21.15.94:0/79219636 learned_addr learned my addr 172.21.15.94:0/79219636 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f15477fe640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15480c1320 0x7f1548148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1547fff640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1548150580 0x7f15481470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 -- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1548150580 msgr2=0x7f15481470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1548150580 0x7f15481470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 -- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15480c1320 msgr2=0x7f1548148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15480c1320 0x7f1548148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 -- 172.21.15.94:0/79219636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480a6330 con 0x7f15480c6000 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1547fff640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1548150580 0x7f15481470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.426+0000 7f1546ffd640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c6000 0x7f1548146ba0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f153c000c00 tx=0x7f153c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.427+0000 7f152ffff640 1 -- 172.21.15.94:0/79219636 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c004230 con 0x7f15480c6000 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.427+0000 7f152ffff640 1 -- 172.21.15.94:0/79219636 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c0043d0 con 0x7f15480c6000 2024-03-06T07:22:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.427+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15481542d0 con 0x7f15480c6000 2024-03-06T07:22:43.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.427+0000 7f152ffff640 1 -- 172.21.15.94:0/79219636 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c032860 con 0x7f15480c6000 2024-03-06T07:22:43.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.427+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15480a7d00 con 0x7f15480c6000 2024-03-06T07:22:43.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.428+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154804f330 con 0x7f15480c6000 2024-03-06T07:22:43.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.428+0000 7f152ffff640 1 -- 172.21.15.94:0/79219636 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f153c032a00 con 0x7f15480c6000 2024-03-06T07:22:43.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.428+0000 7f152ffff640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f153406b4d0 0x7f153406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.428+0000 7f152ffff640 1 -- 172.21.15.94:0/79219636 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f153c02f030 con 0x7f15480c6000 2024-03-06T07:22:43.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.429+0000 7f15477fe640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f153406b4d0 0x7f153406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.429+0000 7f15477fe640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f153406b4d0 0x7f153406d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1538007fd0 tx=0x7f1538007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.433+0000 7f152ffff640 1 -- 172.21.15.94:0/79219636 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15480a6330 con 0x7f15480c6000 2024-03-06T07:22:43.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.605+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15480c3a40 con 0x7f15480c6000 2024-03-06T07:22:43.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.607+0000 7f152ffff640 1 -- 172.21.15.94:0/79219636 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16913 (secure 0 0 0) 0x7f153c041070 con 0x7f15480c6000 2024-03-06T07:22:43.609 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:43.609 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":120,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:42.624555+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:40.848613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:41.593837+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f153406b4d0 msgr2=0x7f153406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f153406b4d0 0x7f153406d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1538007fd0 tx=0x7f1538007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c6000 msgr2=0x7f1548146ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c6000 0x7f1548146ba0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f153c000c00 tx=0x7f153c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 shutdown_connections 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f153406b4d0 0x7f153406d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1548150580 0x7f15481470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15480c6000 0x7f1548146ba0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 --2- 172.21.15.94:0/79219636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15480c1320 0x7f1548148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 >> 172.21.15.94:0/79219636 conn(0x7f15480c6710 msgr2=0x7f1548153630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 shutdown_connections 2024-03-06T07:22:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.609+0000 7f154e0c2640 1 -- 172.21.15.94:0/79219636 wait complete. 2024-03-06T07:22:43.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:43.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.720+0000 7ff4cd50f640 1 Processor -- start 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.720+0000 7ff4cd50f640 1 -- start start 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4cd50f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4c81506b0 0x7ff4c8153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4cd50f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff4c80acc00 0x7ff4c80c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4cd50f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c8070aa0 con 0x7ff4c8156ea0 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4cd50f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c8156450 con 0x7ff4c81506b0 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4cd50f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c8156a10 con 0x7ff4c80acc00 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4c6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4c81506b0 0x7ff4c8153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4c67fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff4c8156ea0 0x7ff4c80702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33940/0 (socket says 172.21.15.94:33940) 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4c67fc640 1 -- 172.21.15.94:0/1916231875 learned_addr learned my addr 172.21.15.94:0/1916231875 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4c77fe640 1 --2- 172.21.15.94:0/1916231875 >> v2:172.21.15.145:3300/0 conn(0x7ff4c80acc00 0x7ff4c80c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4c6ffd640 1 -- 172.21.15.94:0/1916231875 >> v2:172.21.15.145:3300/0 conn(0x7ff4c80acc00 msgr2=0x7ff4c80c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.721+0000 7ff4c6ffd640 1 --2- 172.21.15.94:0/1916231875 >> v2:172.21.15.145:3300/0 conn(0x7ff4c80acc00 0x7ff4c80c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.722+0000 7ff4c6ffd640 1 -- 172.21.15.94:0/1916231875 >> v1:172.21.15.94:6789/0 conn(0x7ff4c8156ea0 legacy=0x7ff4c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.722+0000 7ff4c6ffd640 1 -- 172.21.15.94:0/1916231875 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c8070d80 con 0x7ff4c81506b0 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.722+0000 7ff4c6ffd640 1 --2- 172.21.15.94:0/1916231875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c81506b0 0x7ff4c8153990 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8073610 tx=0x7ff4b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8be93318ca9893 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.722+0000 7ff4c5ffb640 1 -- 172.21.15.94:0/1916231875 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8039070 con 0x7ff4c81506b0 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.722+0000 7ff4c5ffb640 1 -- 172.21.15.94:0/1916231875 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b802cc80 con 0x7ff4c81506b0 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.722+0000 7ff4c5ffb640 1 -- 172.21.15.94:0/1916231875 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8034660 con 0x7ff4c81506b0 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1916231875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c81506b0 msgr2=0x7ff4c8153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1916231875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c81506b0 0x7ff4c8153990 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8073610 tx=0x7ff4b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1916231875 shutdown_connections 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1916231875 >> v2:172.21.15.145:3300/0 conn(0x7ff4c80acc00 0x7ff4c80c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1916231875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c81506b0 0x7ff4c8153990 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1916231875 >> 172.21.15.94:0/1916231875 conn(0x7ff4c806b1a0 msgr2=0x7ff4c806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1916231875 shutdown_connections 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.723+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1916231875 wait complete. 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.724+0000 7ff4cd50f640 1 Processor -- start 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.724+0000 7ff4cd50f640 1 -- start start 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4cd50f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c80acc00 0x7ff4c80a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4cd50f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c81506b0 0x7ff4c81ec010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4cd50f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4c8156ea0 0x7ff4c814baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4cd50f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80ad1d0 con 0x7ff4c80acc00 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4cd50f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80ad050 con 0x7ff4c81506b0 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4cd50f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80ad350 con 0x7ff4c8156ea0 2024-03-06T07:22:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c80acc00 0x7ff4c80a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c80acc00 0x7ff4c80a85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36218/0 (socket says 172.21.15.94:36218) 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 -- 172.21.15.94:0/1714888093 learned_addr learned my addr 172.21.15.94:0/1714888093 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 -- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4c8156ea0 msgr2=0x7ff4c814baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4c8156ea0 0x7ff4c814baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 -- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c81506b0 msgr2=0x7ff4c81ec010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c81506b0 0x7ff4c81ec010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 -- 172.21.15.94:0/1714888093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c81544c0 con 0x7ff4c80acc00 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.725+0000 7ff4c6ffd640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c80acc00 0x7ff4c80a85c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8037040 tx=0x7ff4b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.726+0000 7ff4af7fe640 1 -- 172.21.15.94:0/1714888093 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8039050 con 0x7ff4c80acc00 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.726+0000 7ff4af7fe640 1 -- 172.21.15.94:0/1714888093 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8045070 con 0x7ff4c80acc00 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.726+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c80aa3f0 con 0x7ff4c80acc00 2024-03-06T07:22:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.726+0000 7ff4af7fe640 1 -- 172.21.15.94:0/1714888093 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b803e850 con 0x7ff4c80acc00 2024-03-06T07:22:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.726+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c80aa690 con 0x7ff4c80acc00 2024-03-06T07:22:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.727+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c804f330 con 0x7ff4c80acc00 2024-03-06T07:22:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.727+0000 7ff4af7fe640 1 -- 172.21.15.94:0/1714888093 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4b803e9f0 con 0x7ff4c80acc00 2024-03-06T07:22:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.727+0000 7ff4af7fe640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4b406b4d0 0x7ff4b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.728+0000 7ff4af7fe640 1 -- 172.21.15.94:0/1714888093 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4b80ad1e0 con 0x7ff4c80acc00 2024-03-06T07:22:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.728+0000 7ff4c67fc640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4b406b4d0 0x7ff4b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.728+0000 7ff4c67fc640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4b406b4d0 0x7ff4b406d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000f60 tx=0x7ff4bc007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.732+0000 7ff4af7fe640 1 -- 172.21.15.94:0/1714888093 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4b807c080 con 0x7ff4c80acc00 2024-03-06T07:22:43.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.905+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4c8153f60 con 0x7ff4c80acc00 2024-03-06T07:22:43.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.906+0000 7ff4af7fe640 1 -- 172.21.15.94:0/1714888093 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+16913 (secure 0 0 0) 0x7ff4c8153f60 con 0x7ff4c80acc00 2024-03-06T07:22:43.908 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:43.908 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":120,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:42.624555+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:40.848613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:41.593837+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4b406b4d0 msgr2=0x7ff4b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4b406b4d0 0x7ff4b406d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000f60 tx=0x7ff4bc007a80 comp rx=0 tx=0).stop 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c80acc00 msgr2=0x7ff4c80a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c80acc00 0x7ff4c80a85c0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8037040 tx=0x7ff4b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 shutdown_connections 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4b406b4d0 0x7ff4b406d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4c8156ea0 0x7ff4c814baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c81506b0 0x7ff4c81ec010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 --2- 172.21.15.94:0/1714888093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4c80acc00 0x7ff4c80a85c0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 >> 172.21.15.94:0/1714888093 conn(0x7ff4c806b1a0 msgr2=0x7ff4c8152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 shutdown_connections 2024-03-06T07:22:43.911 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:43.908+0000 7ff4cd50f640 1 -- 172.21.15.94:0/1714888093 wait complete. 2024-03-06T07:22:43.921 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.022+0000 7f43bc9f6640 1 Processor -- start 2024-03-06T07:22:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.022+0000 7f43bc9f6640 1 -- start start 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43bc9f6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f43b814ef70 0x7f43b80c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43bc9f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b80c2790 con 0x7f43b806c1f0 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43bc9f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b806cf20 con 0x7f43b806cad0 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43bc9f6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b806c5d0 con 0x7f43b814ef70 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43b659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f43b806c1f0 0x7f43b8154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33942/0 (socket says 172.21.15.94:33942) 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43b659c640 1 -- 172.21.15.94:0/263673046 learned_addr learned my addr 172.21.15.94:0/263673046 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43b5d9b640 1 --2- 172.21.15.94:0/263673046 >> v2:172.21.15.145:3300/0 conn(0x7f43b814ef70 0x7f43b80c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087059284 0 0) 0x7f43b80c2790 con 0x7f43b806c1f0 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.023+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a0003520 con 0x7f43b806c1f0 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598610981 0 0) 0x7f43a0003520 con 0x7f43b806c1f0 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 >> v2:172.21.15.145:3300/0 conn(0x7f43b814ef70 msgr2=0x7f43b80c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 --2- 172.21.15.94:0/263673046 >> v2:172.21.15.145:3300/0 conn(0x7f43b814ef70 0x7f43b80c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 >> v1:172.21.15.145:6789/0 conn(0x7f43b806cad0 legacy=0x7f43b80c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b80c1540 con 0x7f43b806c1f0 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f43a8001670 con 0x7f43b806c1f0 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 >> v1:172.21.15.94:6789/0 conn(0x7f43b806c1f0 legacy=0x7f43b8154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43a00051c0 0x7f43a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b806c5d0 con 0x7f43a00039b0 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a000c160 con 0x7f43a0004800 2024-03-06T07:22:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a000c360 con 0x7f43a00051c0 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b659c640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b5d9b640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b6d9d640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43a00051c0 0x7f43a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b659c640 1 -- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43a00051c0 msgr2=0x7f43a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b659c640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43a00051c0 0x7f43a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b659c640 1 -- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43a0004800 msgr2=0x7f43a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b659c640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b659c640 1 -- 172.21.15.94:0/263673046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0003520 con 0x7f43a00039b0 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b6d9d640 1 -- 172.21.15.94:0/263673046 reap_dead start 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.024+0000 7f43b5d9b640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43a0004800 0x7f43a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43b659c640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f43a800c210 tx=0x7f43a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60f6c938507a513 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8007d30 con 0x7f43a00039b0 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8030070 con 0x7f43a00039b0 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43b559a640 1 -- 172.21.15.94:0/263673046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a802cda0 con 0x7f43a00039b0 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43bc9f6640 1 -- 172.21.15.94:0/263673046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 msgr2=0x7f43a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f43a800c210 tx=0x7f43a0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43bc9f6640 1 -- 172.21.15.94:0/263673046 shutdown_connections 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/263673046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43a0003dd0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43bc9f6640 1 -- 172.21.15.94:0/263673046 >> 172.21.15.94:0/263673046 conn(0x7f43b80c6710 msgr2=0x7f43b80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43bc9f6640 1 -- 172.21.15.94:0/263673046 shutdown_connections 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.025+0000 7f43bc9f6640 1 -- 172.21.15.94:0/263673046 wait complete. 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 Processor -- start 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 -- start start 2024-03-06T07:22:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43b8070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b8150430 0x7f43b814ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b81e7c00 0x7f43b81e8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b80c1820 con 0x7f43a00039b0 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b8155750 con 0x7f43b8150430 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43bc9f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b80c19a0 con 0x7f43b81e7c00 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43b659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43b8070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.027+0000 7f43b659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43b8070d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36234/0 (socket says 172.21.15.94:36234) 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b659c640 1 -- 172.21.15.94:0/164342465 learned_addr learned my addr 172.21.15.94:0/164342465 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b5d9b640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b8150430 0x7f43b814ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b6d9d640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b81e7c00 0x7f43b81e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b659c640 1 -- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b81e7c00 msgr2=0x7f43b81e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b659c640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b81e7c00 0x7f43b81e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b659c640 1 -- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b8150430 msgr2=0x7f43b814ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b659c640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b8150430 0x7f43b814ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b659c640 1 -- 172.21.15.94:0/164342465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b8152190 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b5d9b640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b8150430 0x7f43b814ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43b659c640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43b8070d30 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f43a800c210 tx=0x7f43a8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43a6ffd640 1 -- 172.21.15.94:0/164342465 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8043070 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43a6ffd640 1 -- 172.21.15.94:0/164342465 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8004bf0 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b8150a00 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43a6ffd640 1 -- 172.21.15.94:0/164342465 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8007d10 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.028+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b8150fc0 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.029+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b804f330 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.029+0000 7f43a6ffd640 1 -- 172.21.15.94:0/164342465 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43a8046070 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.030+0000 7f43a6ffd640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.030+0000 7f43a6ffd640 1 -- 172.21.15.94:0/164342465 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43a80ae3d0 con 0x7f43a00039b0 2024-03-06T07:22:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.030+0000 7f43b5d9b640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.031+0000 7f43b5d9b640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f43b8150830 tx=0x7f43ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.034+0000 7f43a6ffd640 1 -- 172.21.15.94:0/164342465 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43a804d030 con 0x7f43a00039b0 2024-03-06T07:22:44.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.239+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f43b80c1110 con 0x7f43a00039b0 2024-03-06T07:22:44.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.287+0000 7f43a6ffd640 1 -- 172.21.15.94:0/164342465 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v103) v1 ==== 118+0+0 (secure 0 0 0) 0x7f43a803ce20 con 0x7f43a00039b0 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439806b4d0 msgr2=0x7f439806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f43b8150830 tx=0x7f43ac008040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 msgr2=0x7f43b8070d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43b8070d30 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f43a800c210 tx=0x7f43a8001a40 comp rx=0 tx=0).stop 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 shutdown_connections 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43b81e7c00 0x7f43b81e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43b8150430 0x7f43b814ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 --2- 172.21.15.94:0/164342465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43a00039b0 0x7f43b8070d30 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 >> 172.21.15.94:0/164342465 conn(0x7f43b80c6710 msgr2=0x7f43b80c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 shutdown_connections 2024-03-06T07:22:44.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.290+0000 7f43bc9f6640 1 -- 172.21.15.94:0/164342465 wait complete. 2024-03-06T07:22:44.303 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:44.303 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:44.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.405+0000 7f39afbc7640 1 Processor -- start 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.406+0000 7f39afbc7640 1 -- start start 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.406+0000 7f39afbc7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39a806c900 0x7f39a80c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.406+0000 7f39afbc7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f39a80ab170 0x7f39a80a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.406+0000 7f39afbc7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a80c3ef0 con 0x7f39a80c5990 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.406+0000 7f39afbc7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a81556d0 con 0x7f39a806c900 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.406+0000 7f39afbc7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a81559b0 con 0x7f39a80ab170 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.406+0000 7f39ad162640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39a806c900 0x7f39a80c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ad162640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39a806c900 0x7f39a80c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36250/0 (socket says 172.21.15.94:36250) 2024-03-06T07:22:44.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ad162640 1 -- 172.21.15.94:0/2543812987 learned_addr learned my addr 172.21.15.94:0/2543812987 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:44.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ae164640 1 --2- 172.21.15.94:0/2543812987 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ab170 0x7f39a80a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ad162640 1 -- 172.21.15.94:0/2543812987 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ab170 msgr2=0x7f39a80a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ad162640 1 --2- 172.21.15.94:0/2543812987 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ab170 0x7f39a80a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ad162640 1 -- 172.21.15.94:0/2543812987 >> v1:172.21.15.145:6789/0 conn(0x7f39a80c5990 legacy=0x7f39a806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ad162640 1 -- 172.21.15.94:0/2543812987 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8155c90 con 0x7f39a806c900 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.407+0000 7f39ad162640 1 --2- 172.21.15.94:0/2543812987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80c0ab0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f3998009870 tx=0x7f399802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2378bca90cbddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39ac961640 1 -- 172.21.15.94:0/2543812987 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998034070 con 0x7f39a806c900 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39ac961640 1 -- 172.21.15.94:0/2543812987 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3998030860 con 0x7f39a806c900 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39ac961640 1 -- 172.21.15.94:0/2543812987 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998030c30 con 0x7f39a806c900 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 -- 172.21.15.94:0/2543812987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 msgr2=0x7f39a80c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 --2- 172.21.15.94:0/2543812987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80c0ab0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f3998009870 tx=0x7f399802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 -- 172.21.15.94:0/2543812987 shutdown_connections 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 --2- 172.21.15.94:0/2543812987 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ab170 0x7f39a80a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 --2- 172.21.15.94:0/2543812987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80c0ab0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 -- 172.21.15.94:0/2543812987 >> 172.21.15.94:0/2543812987 conn(0x7f39a80c6710 msgr2=0x7f39a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 -- 172.21.15.94:0/2543812987 shutdown_connections 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.408+0000 7f39afbc7640 1 -- 172.21.15.94:0/2543812987 wait complete. 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.409+0000 7f39afbc7640 1 Processor -- start 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.409+0000 7f39afbc7640 1 -- start start 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39afbc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39afbc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80ab170 0x7f39a8070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39afbc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80c5990 0x7f39a8141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39afbc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a8156610 con 0x7f39a806c900 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39afbc7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a8156490 con 0x7f39a80c5990 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39afbc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80c3af0 con 0x7f39a80ab170 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80a75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36256/0 (socket says 172.21.15.94:36256) 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 -- 172.21.15.94:0/241404950 learned_addr learned my addr 172.21.15.94:0/241404950 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ae164640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80c5990 0x7f39a8141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad162640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80ab170 0x7f39a8070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 -- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80ab170 msgr2=0x7f39a8070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80ab170 0x7f39a8070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 -- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80c5990 msgr2=0x7f39a8141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80c5990 0x7f39a8141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.410+0000 7f39ad963640 1 -- 172.21.15.94:0/241404950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a81532a0 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.411+0000 7f39ad963640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80a75f0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c3f0 tx=0x7f39a400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.411+0000 7f399e7fc640 1 -- 172.21.15.94:0/241404950 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a40052d0 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.411+0000 7f399e7fc640 1 -- 172.21.15.94:0/241404950 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a4005490 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.411+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a8152fc0 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.411+0000 7f399e7fc640 1 -- 172.21.15.94:0/241404950 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a4007260 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.411+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a81e90a0 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.412+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a80c5d70 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.412+0000 7f399e7fc640 1 -- 172.21.15.94:0/241404950 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39a4007400 con 0x7f39a806c900 2024-03-06T07:22:44.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.412+0000 7f399e7fc640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f398c06b4d0 0x7f398c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.413+0000 7f399e7fc640 1 -- 172.21.15.94:0/241404950 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f39a40897c0 con 0x7f39a806c900 2024-03-06T07:22:44.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.413+0000 7f39ad162640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f398c06b4d0 0x7f398c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.413+0000 7f39ad162640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f398c06b4d0 0x7f398c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3998036000 tx=0x7f3998008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.417+0000 7f399e7fc640 1 -- 172.21.15.94:0/241404950 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39a80c5d70 con 0x7f39a806c900 2024-03-06T07:22:44.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:44.624+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f39a804f330 con 0x7f39a806c900 2024-03-06T07:22:44.804 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:44.806 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:45.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.291+0000 7f399e7fc640 1 -- 172.21.15.94:0/241404950 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v104) v1 ==== 122+0+0 (secure 0 0 0) 0x7f39a405ce20 con 0x7f39a806c900 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f398c06b4d0 msgr2=0x7f398c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f398c06b4d0 0x7f398c06d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3998036000 tx=0x7f3998008030 comp rx=0 tx=0).stop 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 msgr2=0x7f39a80a75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80a75f0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c3f0 tx=0x7f39a400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 shutdown_connections 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f398c06b4d0 0x7f398c06d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80c5990 0x7f39a8141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80ab170 0x7f39a8070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 --2- 172.21.15.94:0/241404950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39a806c900 0x7f39a80a75f0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 >> 172.21.15.94:0/241404950 conn(0x7f39a80c6710 msgr2=0x7f39a80a9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 shutdown_connections 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.293+0000 7f39afbc7640 1 -- 172.21.15.94:0/241404950 wait complete. 2024-03-06T07:22:45.306 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:45.306 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:45.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.404+0000 7f91e6dd6640 1 Processor -- start 2024-03-06T07:22:45.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.404+0000 7f91e6dd6640 1 -- start start 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91e6dd6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f91e00c5480 0x7f91e0155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91e6dd6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91e01586f0 0x7f91e0158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91e6dd6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00c5cc0 con 0x7f91e00c5860 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91e6dd6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e006df40 con 0x7f91e01586f0 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91e6dd6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e006fdb0 con 0x7f91e00c5480 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91e5373640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91e01586f0 0x7f91e0158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91dffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f91e00c5860 0x7f91e006f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33944/0 (socket says 172.21.15.94:33944) 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91dffff640 1 -- 172.21.15.94:0/992786509 learned_addr learned my addr 172.21.15.94:0/992786509 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:45.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.405+0000 7f91e4b72640 1 --2- 172.21.15.94:0/992786509 >> v2:172.21.15.145:3300/0 conn(0x7f91e00c5480 0x7f91e0155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.406+0000 7f91e5373640 1 -- 172.21.15.94:0/992786509 >> v2:172.21.15.145:3300/0 conn(0x7f91e00c5480 msgr2=0x7f91e0155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.406+0000 7f91e5373640 1 --2- 172.21.15.94:0/992786509 >> v2:172.21.15.145:3300/0 conn(0x7f91e00c5480 0x7f91e0155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.406+0000 7f91e5373640 1 -- 172.21.15.94:0/992786509 >> v1:172.21.15.94:6789/0 conn(0x7f91e00c5860 legacy=0x7f91e006f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.406+0000 7f91e5373640 1 -- 172.21.15.94:0/992786509 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e01591c0 con 0x7f91e01586f0 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.406+0000 7f91df7fe640 1 -- 172.21.15.94:0/992786509 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584149485 0 0) 0x7f91e00c5cc0 con 0x7f91e00c5860 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.406+0000 7f91e5373640 1 --2- 172.21.15.94:0/992786509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e0158b80 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f91e00c1cf0 tx=0x7f91d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84d3c382c28d74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.406+0000 7f91df7fe640 1 -- 172.21.15.94:0/992786509 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8039070 con 0x7f91e01586f0 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91df7fe640 1 -- 172.21.15.94:0/992786509 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d802cbf0 con 0x7f91e01586f0 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91df7fe640 1 -- 172.21.15.94:0/992786509 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8034440 con 0x7f91e01586f0 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 -- 172.21.15.94:0/992786509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 msgr2=0x7f91e0158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/992786509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e0158b80 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f91e00c1cf0 tx=0x7f91d802be40 comp rx=0 tx=0).stop 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 -- 172.21.15.94:0/992786509 shutdown_connections 2024-03-06T07:22:45.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/992786509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e0158b80 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/992786509 >> v2:172.21.15.145:3300/0 conn(0x7f91e00c5480 0x7f91e0155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 -- 172.21.15.94:0/992786509 >> 172.21.15.94:0/992786509 conn(0x7f91e006b1a0 msgr2=0x7f91e006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 -- 172.21.15.94:0/992786509 shutdown_connections 2024-03-06T07:22:45.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.407+0000 7f91e6dd6640 1 -- 172.21.15.94:0/992786509 wait complete. 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.408+0000 7f91e6dd6640 1 Processor -- start 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.408+0000 7f91e6dd6640 1 -- start start 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e6dd6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e00c5480 0x7f91e00abb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e6dd6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e00c5860 0x7f91e00ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e6dd6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e014eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e6dd6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e0155680 con 0x7f91e01586f0 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e6dd6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e01553a0 con 0x7f91e00c5860 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e6dd6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e0152a20 con 0x7f91e00c5480 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e014eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e014eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36266/0 (socket says 172.21.15.94:36266) 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 -- 172.21.15.94:0/2029579367 learned_addr learned my addr 172.21.15.94:0/2029579367 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91dffff640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e00c5860 0x7f91e00ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e4b72640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e00c5480 0x7f91e00abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 -- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e00c5480 msgr2=0x7f91e00abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e00c5480 0x7f91e00abb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 -- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e00c5860 msgr2=0x7f91e00ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e00c5860 0x7f91e00ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.409+0000 7f91e5373640 1 -- 172.21.15.94:0/2029579367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00adcc0 con 0x7f91e01586f0 2024-03-06T07:22:45.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.410+0000 7f91e5373640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e014eb80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f91d8037040 tx=0x7f91d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.410+0000 7f91dd7fa640 1 -- 172.21.15.94:0/2029579367 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8044020 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.410+0000 7f91dd7fa640 1 -- 172.21.15.94:0/2029579367 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d803a040 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.410+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e014cc40 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.410+0000 7f91dd7fa640 1 -- 172.21.15.94:0/2029579367 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8039050 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.410+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e01eba10 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.411+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e0152c90 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.411+0000 7f91dd7fa640 1 -- 172.21.15.94:0/2029579367 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91d8003bb0 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.412+0000 7f91dd7fa640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91cc06b4d0 0x7f91cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.412+0000 7f91dd7fa640 1 -- 172.21.15.94:0/2029579367 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f91d80ac720 con 0x7f91e01586f0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.412+0000 7f91e4b72640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91cc06b4d0 0x7f91cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.412+0000 7f91e4b72640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91cc06b4d0 0x7f91cc06d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f91e006d340 tx=0x7f91d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.416+0000 7f91dd7fa640 1 -- 172.21.15.94:0/2029579367 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91e0152c90 con 0x7f91e01586f0 2024-03-06T07:22:45.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.622+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91e00ad7d0 con 0x7f91e01586f0 2024-03-06T07:22:45.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.623+0000 7f91dd7fa640 1 -- 172.21.15.94:0/2029579367 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4985 (secure 0 0 0) 0x7f91d804e060 con 0x7f91e01586f0 2024-03-06T07:22:45.625 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:45.625 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7827,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/282207355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":282207355},{"type":"v1","addr":"172.21.15.94:6834","nonce":282207355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:22:44.626233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7842},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7842":{"gid":7842,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/198029515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":198029515},{"type":"v1","addr":"172.21.15.145:6836","nonce":198029515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7842,"qdb_cluster":[7842]},"id":9}]} 2024-03-06T07:22:45.625 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 104 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91cc06b4d0 msgr2=0x7f91cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91cc06b4d0 0x7f91cc06d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f91e006d340 tx=0x7f91d0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 msgr2=0x7f91e014eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e014eb80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f91d8037040 tx=0x7f91d8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 shutdown_connections 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91cc06b4d0 0x7f91cc06d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91e01586f0 0x7f91e014eb80 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e00c5860 0x7f91e00ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 --2- 172.21.15.94:0/2029579367 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e00c5480 0x7f91e00abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 >> 172.21.15.94:0/2029579367 conn(0x7f91e006b1a0 msgr2=0x7f91e006ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 shutdown_connections 2024-03-06T07:22:45.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.625+0000 7f91e6dd6640 1 -- 172.21.15.94:0/2029579367 wait complete. 2024-03-06T07:22:45.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:45.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.735+0000 7f78ad1f6640 1 Processor -- start 2024-03-06T07:22:45.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.736+0000 7f78ad1f6640 1 -- start start 2024-03-06T07:22:45.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.736+0000 7f78ad1f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.736+0000 7f78ad1f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a80c28d0 con 0x7f78a814f790 2024-03-06T07:22:45.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.736+0000 7f78ad1f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a80c2bb0 con 0x7f78a8155a40 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.736+0000 7f78ad1f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80c2680 con 0x7f78a80c1aa0 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.736+0000 7f78a6d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f78a814f790 0x7f78a80c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33956/0 (socket says 172.21.15.94:33956) 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.736+0000 7f78a6d9d640 1 -- 172.21.15.94:0/1985998103 learned_addr learned my addr 172.21.15.94:0/1985998103 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a659c640 1 --2- 172.21.15.94:0/1985998103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a5d9b640 1 -- 172.21.15.94:0/1985998103 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496205577 0 0) 0x7f78a80c28d0 con 0x7f78a814f790 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a5d9b640 1 -- 172.21.15.94:0/1985998103 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7894003520 con 0x7f78a814f790 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a5d9b640 1 -- 172.21.15.94:0/1985998103 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844722893 0 0) 0x7f78a80c2bb0 con 0x7f78a8155a40 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a5d9b640 1 -- 172.21.15.94:0/1985998103 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a80c28d0 con 0x7f78a8155a40 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a659c640 1 -- 172.21.15.94:0/1985998103 >> v1:172.21.15.145:6789/0 conn(0x7f78a8155a40 legacy=0x7f78a8155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a659c640 1 -- 172.21.15.94:0/1985998103 >> v1:172.21.15.94:6789/0 conn(0x7f78a814f790 legacy=0x7f78a80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.737+0000 7f78a659c640 1 -- 172.21.15.94:0/1985998103 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80c2e90 con 0x7f78a80c1aa0 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78a659c640 1 --2- 172.21.15.94:0/1985998103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81529d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78900097d0 tx=0x7f789002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd1e455cc841c520 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78a5d9b640 1 -- 172.21.15.94:0/1985998103 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890035020 con 0x7f78a80c1aa0 2024-03-06T07:22:45.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78a5d9b640 1 -- 172.21.15.94:0/1985998103 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7890031890 con 0x7f78a80c1aa0 2024-03-06T07:22:45.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78a5d9b640 1 -- 172.21.15.94:0/1985998103 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890031c60 con 0x7f78a80c1aa0 2024-03-06T07:22:45.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78ad1f6640 1 -- 172.21.15.94:0/1985998103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 msgr2=0x7f78a81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/1985998103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81529d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78900097d0 tx=0x7f789002b930 comp rx=0 tx=0).stop 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78ad1f6640 1 -- 172.21.15.94:0/1985998103 shutdown_connections 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/1985998103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81529d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78ad1f6640 1 -- 172.21.15.94:0/1985998103 >> 172.21.15.94:0/1985998103 conn(0x7f78a80c6710 msgr2=0x7f78a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78ad1f6640 1 -- 172.21.15.94:0/1985998103 shutdown_connections 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.738+0000 7f78ad1f6640 1 -- 172.21.15.94:0/1985998103 wait complete. 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 Processor -- start 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 -- start start 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81dfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a814f790 0x7f78a80702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78a8155a40 0x7f78a8072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80c5e30 con 0x7f78a814f790 2024-03-06T07:22:45.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80c0d60 con 0x7f78a8155a40 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78ad1f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80c5fb0 con 0x7f78a80c1aa0 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.740+0000 7f78a659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a814f790 0x7f78a80702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a814f790 0x7f78a80702f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36268/0 (socket says 172.21.15.94:36268) 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 -- 172.21.15.94:0/2639738056 learned_addr learned my addr 172.21.15.94:0/2639738056 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a6d9d640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81dfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a759e640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78a8155a40 0x7f78a8072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 -- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 msgr2=0x7f78a81dfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81dfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 -- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78a8155a40 msgr2=0x7f78a8072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78a8155a40 0x7f78a8072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 -- 172.21.15.94:0/2639738056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a81ecfe0 con 0x7f78a814f790 2024-03-06T07:22:45.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78a659c640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a814f790 0x7f78a80702f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f78900098f0 tx=0x7f78900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f789f7fe640 1 -- 172.21.15.94:0/2639738056 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890035040 con 0x7f78a814f790 2024-03-06T07:22:45.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f789f7fe640 1 -- 172.21.15.94:0/2639738056 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7890030d20 con 0x7f78a814f790 2024-03-06T07:22:45.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a81e0330 con 0x7f78a814f790 2024-03-06T07:22:45.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f789f7fe640 1 -- 172.21.15.94:0/2639738056 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890006e10 con 0x7f78a814f790 2024-03-06T07:22:45.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.741+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a81df310 con 0x7f78a814f790 2024-03-06T07:22:45.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.742+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a8067ad0 con 0x7f78a814f790 2024-03-06T07:22:45.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.743+0000 7f789f7fe640 1 -- 172.21.15.94:0/2639738056 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7890038020 con 0x7f78a814f790 2024-03-06T07:22:45.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.743+0000 7f789f7fe640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f788806b4d0 0x7f788806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.743+0000 7f789f7fe640 1 -- 172.21.15.94:0/2639738056 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78900ac910 con 0x7f78a814f790 2024-03-06T07:22:45.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.743+0000 7f78a6d9d640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f788806b4d0 0x7f788806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.744+0000 7f78a6d9d640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f788806b4d0 0x7f788806d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f78a00012e0 tx=0x7f78a0001ad0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.747+0000 7f789f7fe640 1 -- 172.21.15.94:0/2639738056 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78a8067ad0 con 0x7f78a814f790 2024-03-06T07:22:45.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.918+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78a804f330 con 0x7f78a814f790 2024-03-06T07:22:45.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.919+0000 7f789f7fe640 1 -- 172.21.15.94:0/2639738056 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16900 (secure 0 0 0) 0x7f789002d040 con 0x7f78a814f790 2024-03-06T07:22:45.921 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:45.921 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":121,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:22:44.283246+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:40.848613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:41.593837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f788806b4d0 msgr2=0x7f788806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f788806b4d0 0x7f788806d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f78a00012e0 tx=0x7f78a0001ad0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a814f790 msgr2=0x7f78a80702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a814f790 0x7f78a80702f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f78900098f0 tx=0x7f78900046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 shutdown_connections 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f788806b4d0 0x7f788806d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78a8155a40 0x7f78a8072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78a814f790 0x7f78a80702f0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 --2- 172.21.15.94:0/2639738056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78a80c1aa0 0x7f78a81dfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 >> 172.21.15.94:0/2639738056 conn(0x7f78a80c6710 msgr2=0x7f78a8153c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 shutdown_connections 2024-03-06T07:22:45.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:45.921+0000 7f78ad1f6640 1 -- 172.21.15.94:0/2639738056 wait complete. 2024-03-06T07:22:45.934 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:45.934 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:46.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.031+0000 7f05dc0fb640 1 Processor -- start 2024-03-06T07:22:46.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.031+0000 7f05dc0fb640 1 -- start start 2024-03-06T07:22:46.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05dc0fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d4150580 0x7f05d406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05dc0fb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05d40c6000 0x7f05d406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05dc0fb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d4153a60 con 0x7f05d40c1320 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05dc0fb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d4153d40 con 0x7f05d40c6000 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05dc0fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d406c2c0 con 0x7f05d4150580 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05d9e97640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d4150580 0x7f05d406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05da698640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05d40c6000 0x7f05d406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05d9e97640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d4150580 0x7f05d406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:57922/0 (socket says 172.21.15.94:57922) 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05da698640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05d40c6000 0x7f05d406e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:39648/0 (socket says 172.21.15.94:39648) 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05d9e97640 1 -- 172.21.15.94:0/2406511767 learned_addr learned my addr 172.21.15.94:0/2406511767 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.032+0000 7f05d9696640 1 --1- 172.21.15.94:0/2406511767 >> v1:172.21.15.145:6789/0 conn(0x7f05d40c1320 0x7f05d41566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:59748/0 (socket says 172.21.15.94:59748) 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.033+0000 7f05d9e97640 1 -- 172.21.15.94:0/2406511767 >> v2:172.21.15.145:3300/0 conn(0x7f05d40c6000 msgr2=0x7f05d406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.033+0000 7f05d9e97640 1 --2- 172.21.15.94:0/2406511767 >> v2:172.21.15.145:3300/0 conn(0x7f05d40c6000 0x7f05d406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.033+0000 7f05d9e97640 1 -- 172.21.15.94:0/2406511767 >> v1:172.21.15.145:6789/0 conn(0x7f05d40c1320 legacy=0x7f05d41566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.033+0000 7f05d9e97640 1 -- 172.21.15.94:0/2406511767 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d406c5a0 con 0x7f05d4150580 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.033+0000 7f05d9e97640 1 --2- 172.21.15.94:0/2406511767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d4150580 0x7f05d406dd10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009e50 tx=0x7f05d002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ed0cb32c059093 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05d8e95640 1 -- 172.21.15.94:0/2406511767 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d002ca70 con 0x7f05d4150580 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05d8e95640 1 -- 172.21.15.94:0/2406511767 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d002cc10 con 0x7f05d4150580 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05d8e95640 1 -- 172.21.15.94:0/2406511767 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d00343f0 con 0x7f05d4150580 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 -- 172.21.15.94:0/2406511767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d4150580 msgr2=0x7f05d406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/2406511767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d4150580 0x7f05d406dd10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009e50 tx=0x7f05d002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 -- 172.21.15.94:0/2406511767 shutdown_connections 2024-03-06T07:22:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/2406511767 >> v2:172.21.15.145:3300/0 conn(0x7f05d40c6000 0x7f05d406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/2406511767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d4150580 0x7f05d406dd10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 -- 172.21.15.94:0/2406511767 >> 172.21.15.94:0/2406511767 conn(0x7f05d40c6710 msgr2=0x7f05d40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 -- 172.21.15.94:0/2406511767 shutdown_connections 2024-03-06T07:22:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.034+0000 7f05dc0fb640 1 -- 172.21.15.94:0/2406511767 wait complete. 2024-03-06T07:22:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.035+0000 7f05dc0fb640 1 Processor -- start 2024-03-06T07:22:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.035+0000 7f05dc0fb640 1 -- start start 2024-03-06T07:22:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05dc0fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d40c1320 0x7f05d4154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05dc0fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d40c6000 0x7f05d41e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05dc0fb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d4150580 0x7f05d40a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05dc0fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40c0dc0 con 0x7f05d40c1320 2024-03-06T07:22:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05dc0fb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40c0c40 con 0x7f05d4150580 2024-03-06T07:22:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05dc0fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4156dc0 con 0x7f05d40c6000 2024-03-06T07:22:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d40c1320 0x7f05d4154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9696640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d40c6000 0x7f05d41e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d40c1320 0x7f05d4154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36282/0 (socket says 172.21.15.94:36282) 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9696640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d40c6000 0x7f05d41e8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:57932/0 (socket says 172.21.15.94:57932) 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05da698640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d4150580 0x7f05d40a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 -- 172.21.15.94:0/4108048812 learned_addr learned my addr 172.21.15.94:0/4108048812 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 -- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d40c6000 msgr2=0x7f05d41e8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d40c6000 0x7f05d41e8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 -- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d4150580 msgr2=0x7f05d40a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d4150580 0x7f05d40a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.036+0000 7f05d9e97640 1 -- 172.21.15.94:0/4108048812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d40c3050 con 0x7f05d40c1320 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.037+0000 7f05d9e97640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d40c1320 0x7f05d4154df0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009f70 tx=0x7f05d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.037+0000 7f05ca7fc640 1 -- 172.21.15.94:0/4108048812 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0044020 con 0x7f05d40c1320 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.037+0000 7f05ca7fc640 1 -- 172.21.15.94:0/4108048812 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d00406b0 con 0x7f05d40c1320 2024-03-06T07:22:46.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.037+0000 7f05ca7fc640 1 -- 172.21.15.94:0/4108048812 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0009510 con 0x7f05d40c1320 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.037+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d40c3330 con 0x7f05d40c1320 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.037+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d41ec740 con 0x7f05d40c1320 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.038+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d404f330 con 0x7f05d40c1320 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.038+0000 7f05ca7fc640 1 -- 172.21.15.94:0/4108048812 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05d0047020 con 0x7f05d40c1320 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.038+0000 7f05ca7fc640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05b806b4d0 0x7f05b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.039+0000 7f05ca7fc640 1 -- 172.21.15.94:0/4108048812 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f05d00acc10 con 0x7f05d40c1320 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.039+0000 7f05d9696640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05b806b4d0 0x7f05b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.039+0000 7f05d9696640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05b806b4d0 0x7f05b806d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f05d40a6280 tx=0x7f05c4000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.043+0000 7f05ca7fc640 1 -- 172.21.15.94:0/4108048812 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05d004e030 con 0x7f05d40c1320 2024-03-06T07:22:46.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.210+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05d4156450 con 0x7f05b806b4d0 2024-03-06T07:22:46.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.216+0000 7f05ca7fc640 1 -- 172.21.15.94:0/4108048812 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247154 (secure 0 0 0) 0x7f05c4004c70 con 0x7f05b806b4d0 2024-03-06T07:22:46.223 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:46.225 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:22:46.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05b806b4d0 msgr2=0x7f05b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05b806b4d0 0x7f05b806d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f05d40a6280 tx=0x7f05c4000ed0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d40c1320 msgr2=0x7f05d4154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d40c1320 0x7f05d4154df0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009f70 tx=0x7f05d0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 shutdown_connections 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05b806b4d0 0x7f05b806d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05d4150580 0x7f05d40a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05d40c6000 0x7f05d41e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 --2- 172.21.15.94:0/4108048812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05d40c1320 0x7f05d4154df0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 >> 172.21.15.94:0/4108048812 conn(0x7f05d40c6710 msgr2=0x7f05d414e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.219+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 shutdown_connections 2024-03-06T07:22:46.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.220+0000 7f05dc0fb640 1 -- 172.21.15.94:0/4108048812 wait complete. 2024-03-06T07:22:46.232 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-03-06T07:22:44.693154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261148,"kb_used_data":11692,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713572,"statfs":{"total":773094113280,"available":772826697728,"internally_reserved":0,"allocated":11972608,"data_stored":6858699,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001252"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279032+0000","last_change":"2024-03-06T07:22:42.279032+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279032+0000","last_undegraded":"2024-03-06T07:22:42.279032+0000","last_fullsized":"2024-03-06T07:22:42.279032+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284203+0000","last_change":"2024-03-06T07:22:42.284203+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284203+0000","last_undegraded":"2024-03-06T07:22:42.284203+0000","last_fullsized":"2024-03-06T07:22:42.284203+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284568+0000","last_change":"2024-03-06T07:22:42.284568+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284568+0000","last_undegraded":"2024-03-06T07:22:42.284568+0000","last_fullsized":"2024-03-06T07:22:42.284568+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279457+0000","last_change":"2024-03-06T07:22:42.279457+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279457+0000","last_undegraded":"2024-03-06T07:22:42.279457+0000","last_fullsized":"2024-03-06T07:22:42.279457+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283015+0000","last_change":"2024-03-06T07:22:42.283015+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283015+0000","last_undegraded":"2024-03-06T07:22:42.283015+0000","last_fullsized":"2024-03-06T07:22:42.283015+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292601+0000","last_change":"2024-03-06T07:22:42.292601+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292601+0000","last_undegraded":"2024-03-06T07:22:42.292601+0000","last_fullsized":"2024-03-06T07:22:42.292601+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.285418+0000","last_change":"2024-03-06T07:22:42.285418+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.285418+0000","last_undegraded":"2024-03-06T07:22:42.285418+0000","last_fullsized":"2024-03-06T07:22:42.285418+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282898+0000","last_change":"2024-03-06T07:22:42.282898+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282898+0000","last_undegraded":"2024-03-06T07:22:42.282898+0000","last_fullsized":"2024-03-06T07:22:42.282898+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282437+0000","last_change":"2024-03-06T07:22:42.282437+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282437+0000","last_undegraded":"2024-03-06T07:22:42.282437+0000","last_fullsized":"2024-03-06T07:22:42.282437+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283810+0000","last_change":"2024-03-06T07:22:42.283810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283810+0000","last_undegraded":"2024-03-06T07:22:42.283810+0000","last_fullsized":"2024-03-06T07:22:42.283810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278978+0000","last_change":"2024-03-06T07:22:42.278978+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278978+0000","last_undegraded":"2024-03-06T07:22:42.278978+0000","last_fullsized":"2024-03-06T07:22:42.278978+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282804+0000","last_change":"2024-03-06T07:22:42.282804+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282804+0000","last_undegraded":"2024-03-06T07:22:42.282804+0000","last_fullsized":"2024-03-06T07:22:42.282804+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278897+0000","last_change":"2024-03-06T07:22:42.278897+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278897+0000","last_undegraded":"2024-03-06T07:22:42.278897+0000","last_fullsized":"2024-03-06T07:22:42.278897+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278443+0000","last_change":"2024-03-06T07:22:42.278443+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278443+0000","last_undegraded":"2024-03-06T07:22:42.278443+0000","last_fullsized":"2024-03-06T07:22:42.278443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283228+0000","last_change":"2024-03-06T07:22:42.283228+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283228+0000","last_undegraded":"2024-03-06T07:22:42.283228+0000","last_fullsized":"2024-03-06T07:22:42.283228+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282177+0000","last_change":"2024-03-06T07:22:42.282177+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282177+0000","last_undegraded":"2024-03-06T07:22:42.282177+0000","last_fullsized":"2024-03-06T07:22:42.282177+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279562+0000","last_change":"2024-03-06T07:22:42.279562+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279562+0000","last_undegraded":"2024-03-06T07:22:42.279562+0000","last_fullsized":"2024-03-06T07:22:42.279562+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.293286+0000","last_change":"2024-03-06T07:22:42.293286+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.293286+0000","last_undegraded":"2024-03-06T07:22:42.293286+0000","last_fullsized":"2024-03-06T07:22:42.293286+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280297+0000","last_change":"2024-03-06T07:22:42.280297+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280297+0000","last_undegraded":"2024-03-06T07:22:42.280297+0000","last_fullsized":"2024-03-06T07:22:42.280297+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283843+0000","last_change":"2024-03-06T07:22:42.283843+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283843+0000","last_undegraded":"2024-03-06T07:22:42.283843+0000","last_fullsized":"2024-03-06T07:22:42.283843+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278394+0000","last_change":"2024-03-06T07:22:42.278394+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278394+0000","last_undegraded":"2024-03-06T07:22:42.278394+0000","last_fullsized":"2024-03-06T07:22:42.278394+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284222+0000","last_change":"2024-03-06T07:22:42.284222+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284222+0000","last_undegraded":"2024-03-06T07:22:42.284222+0000","last_fullsized":"2024-03-06T07:22:42.284222+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282160+0000","last_change":"2024-03-06T07:22:42.282160+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282160+0000","last_undegraded":"2024-03-06T07:22:42.282160+0000","last_fullsized":"2024-03-06T07:22:42.282160+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286365+0000","last_change":"2024-03-06T07:22:42.286365+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286365+0000","last_undegraded":"2024-03-06T07:22:42.286365+0000","last_fullsized":"2024-03-06T07:22:42.286365+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286778+0000","last_change":"2024-03-06T07:22:42.286778+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286778+0000","last_undegraded":"2024-03-06T07:22:42.286778+0000","last_fullsized":"2024-03-06T07:22:42.286778+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292889+0000","last_change":"2024-03-06T07:22:42.292889+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292889+0000","last_undegraded":"2024-03-06T07:22:42.292889+0000","last_fullsized":"2024-03-06T07:22:42.292889+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282008+0000","last_change":"2024-03-06T07:22:42.282008+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282008+0000","last_undegraded":"2024-03-06T07:22:42.282008+0000","last_fullsized":"2024-03-06T07:22:42.282008+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286216+0000","last_change":"2024-03-06T07:22:42.286216+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286216+0000","last_undegraded":"2024-03-06T07:22:42.286216+0000","last_fullsized":"2024-03-06T07:22:42.286216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281552+0000","last_change":"2024-03-06T07:22:42.281552+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281552+0000","last_undegraded":"2024-03-06T07:22:42.281552+0000","last_fullsized":"2024-03-06T07:22:42.281552+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281197+0000","last_change":"2024-03-06T07:22:42.281197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281197+0000","last_undegraded":"2024-03-06T07:22:42.281197+0000","last_fullsized":"2024-03-06T07:22:42.281197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281292+0000","last_change":"2024-03-06T07:22:42.281292+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281292+0000","last_undegraded":"2024-03-06T07:22:42.281292+0000","last_fullsized":"2024-03-06T07:22:42.281292+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289168+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288869+0000","last_became_peered":"2024-03-06T07:22:42.288869+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:22.433887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277947+0000","last_change":"2024-03-06T07:22:42.277947+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277947+0000","last_undegraded":"2024-03-06T07:22:42.277947+0000","last_fullsized":"2024-03-06T07:22:42.277947+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.284819+0000","last_change":"2024-03-06T07:22:42.284230+0000","last_active":"2024-03-06T07:22:42.284819+0000","last_peered":"2024-03-06T07:22:42.284819+0000","last_clean":"2024-03-06T07:22:42.284819+0000","last_became_active":"2024-03-06T07:22:42.283796+0000","last_became_peered":"2024-03-06T07:22:42.283796+0000","last_unstale":"2024-03-06T07:22:42.284819+0000","last_undegraded":"2024-03-06T07:22:42.284819+0000","last_fullsized":"2024-03-06T07:22:42.284819+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:28.750113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283761+0000","last_change":"2024-03-06T07:22:42.283761+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283761+0000","last_undegraded":"2024-03-06T07:22:42.283761+0000","last_fullsized":"2024-03-06T07:22:42.283761+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299956+0000","last_change":"2024-03-06T07:22:42.298123+0000","last_active":"2024-03-06T07:22:42.299956+0000","last_peered":"2024-03-06T07:22:42.299956+0000","last_clean":"2024-03-06T07:22:42.299956+0000","last_became_active":"2024-03-06T07:22:42.297656+0000","last_became_peered":"2024-03-06T07:22:42.297656+0000","last_unstale":"2024-03-06T07:22:42.299956+0000","last_undegraded":"2024-03-06T07:22:42.299956+0000","last_fullsized":"2024-03-06T07:22:42.299956+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:56.531364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277964+0000","last_change":"2024-03-06T07:22:42.277964+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277964+0000","last_undegraded":"2024-03-06T07:22:42.277964+0000","last_fullsized":"2024-03-06T07:22:42.277964+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298336+0000","last_change":"2024-03-06T07:22:42.295819+0000","last_active":"2024-03-06T07:22:42.298336+0000","last_peered":"2024-03-06T07:22:42.298336+0000","last_clean":"2024-03-06T07:22:42.298336+0000","last_became_active":"2024-03-06T07:22:42.295541+0000","last_became_peered":"2024-03-06T07:22:42.295541+0000","last_unstale":"2024-03-06T07:22:42.298336+0000","last_undegraded":"2024-03-06T07:22:42.298336+0000","last_fullsized":"2024-03-06T07:22:42.298336+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:51.235851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282847+0000","last_change":"2024-03-06T07:22:42.282847+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282847+0000","last_undegraded":"2024-03-06T07:22:42.282847+0000","last_fullsized":"2024-03-06T07:22:42.282847+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289767+0000","last_change":"2024-03-06T07:22:42.287120+0000","last_active":"2024-03-06T07:22:42.289767+0000","last_peered":"2024-03-06T07:22:42.289767+0000","last_clean":"2024-03-06T07:22:42.289767+0000","last_became_active":"2024-03-06T07:22:42.286844+0000","last_became_peered":"2024-03-06T07:22:42.286844+0000","last_unstale":"2024-03-06T07:22:42.289767+0000","last_undegraded":"2024-03-06T07:22:42.289767+0000","last_fullsized":"2024-03-06T07:22:42.289767+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.165551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283519+0000","last_change":"2024-03-06T07:22:42.283519+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283519+0000","last_undegraded":"2024-03-06T07:22:42.283519+0000","last_fullsized":"2024-03-06T07:22:42.283519+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.290193+0000","last_change":"2024-03-06T07:22:42.289388+0000","last_active":"2024-03-06T07:22:42.290193+0000","last_peered":"2024-03-06T07:22:42.290193+0000","last_clean":"2024-03-06T07:22:42.290193+0000","last_became_active":"2024-03-06T07:22:42.288148+0000","last_became_peered":"2024-03-06T07:22:42.288148+0000","last_unstale":"2024-03-06T07:22:42.290193+0000","last_undegraded":"2024-03-06T07:22:42.290193+0000","last_fullsized":"2024-03-06T07:22:42.290193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:37.212217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279197+0000","last_change":"2024-03-06T07:22:42.279197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279197+0000","last_undegraded":"2024-03-06T07:22:42.279197+0000","last_fullsized":"2024-03-06T07:22:42.279197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.287719+0000","last_change":"2024-03-06T07:22:42.285183+0000","last_active":"2024-03-06T07:22:42.287719+0000","last_peered":"2024-03-06T07:22:42.287719+0000","last_clean":"2024-03-06T07:22:42.287719+0000","last_became_active":"2024-03-06T07:22:42.284924+0000","last_became_peered":"2024-03-06T07:22:42.284924+0000","last_unstale":"2024-03-06T07:22:42.287719+0000","last_undegraded":"2024-03-06T07:22:42.287719+0000","last_fullsized":"2024-03-06T07:22:42.287719+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:21.527693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292509+0000","last_change":"2024-03-06T07:22:42.292509+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292509+0000","last_undegraded":"2024-03-06T07:22:42.292509+0000","last_fullsized":"2024-03-06T07:22:42.292509+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296297+0000","last_change":"2024-03-06T07:22:42.289621+0000","last_active":"2024-03-06T07:22:42.296297+0000","last_peered":"2024-03-06T07:22:42.296297+0000","last_clean":"2024-03-06T07:22:42.296297+0000","last_became_active":"2024-03-06T07:22:42.289239+0000","last_became_peered":"2024-03-06T07:22:42.289239+0000","last_unstale":"2024-03-06T07:22:42.296297+0000","last_undegraded":"2024-03-06T07:22:42.296297+0000","last_fullsized":"2024-03-06T07:22:42.296297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:40.681976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277967+0000","last_change":"2024-03-06T07:22:42.277967+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277967+0000","last_undegraded":"2024-03-06T07:22:42.277967+0000","last_fullsized":"2024-03-06T07:22:42.277967+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291813+0000","last_change":"2024-03-06T07:22:42.287554+0000","last_active":"2024-03-06T07:22:42.291813+0000","last_peered":"2024-03-06T07:22:42.291813+0000","last_clean":"2024-03-06T07:22:42.291813+0000","last_became_active":"2024-03-06T07:22:42.285213+0000","last_became_peered":"2024-03-06T07:22:42.285213+0000","last_unstale":"2024-03-06T07:22:42.291813+0000","last_undegraded":"2024-03-06T07:22:42.291813+0000","last_fullsized":"2024-03-06T07:22:42.291813+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:07.580402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281750+0000","last_change":"2024-03-06T07:22:42.281750+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281750+0000","last_undegraded":"2024-03-06T07:22:42.281750+0000","last_fullsized":"2024-03-06T07:22:42.281750+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288109+0000","last_change":"2024-03-06T07:22:42.286796+0000","last_active":"2024-03-06T07:22:42.288109+0000","last_peered":"2024-03-06T07:22:42.288109+0000","last_clean":"2024-03-06T07:22:42.288109+0000","last_became_active":"2024-03-06T07:22:42.286558+0000","last_became_peered":"2024-03-06T07:22:42.286558+0000","last_unstale":"2024-03-06T07:22:42.288109+0000","last_undegraded":"2024-03-06T07:22:42.288109+0000","last_fullsized":"2024-03-06T07:22:42.288109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:11.589911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281997+0000","last_change":"2024-03-06T07:22:42.281997+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281997+0000","last_undegraded":"2024-03-06T07:22:42.281997+0000","last_fullsized":"2024-03-06T07:22:42.281997+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299969+0000","last_change":"2024-03-06T07:22:42.298748+0000","last_active":"2024-03-06T07:22:42.299969+0000","last_peered":"2024-03-06T07:22:42.299969+0000","last_clean":"2024-03-06T07:22:42.299969+0000","last_became_active":"2024-03-06T07:22:42.298459+0000","last_became_peered":"2024-03-06T07:22:42.298459+0000","last_unstale":"2024-03-06T07:22:42.299969+0000","last_undegraded":"2024-03-06T07:22:42.299969+0000","last_fullsized":"2024-03-06T07:22:42.299969+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:08.721690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281717+0000","last_change":"2024-03-06T07:22:42.281717+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281717+0000","last_undegraded":"2024-03-06T07:22:42.281717+0000","last_fullsized":"2024-03-06T07:22:42.281717+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299706+0000","last_change":"2024-03-06T07:22:42.297582+0000","last_active":"2024-03-06T07:22:42.299706+0000","last_peered":"2024-03-06T07:22:42.299706+0000","last_clean":"2024-03-06T07:22:42.299706+0000","last_became_active":"2024-03-06T07:22:42.297309+0000","last_became_peered":"2024-03-06T07:22:42.297309+0000","last_unstale":"2024-03-06T07:22:42.299706+0000","last_undegraded":"2024-03-06T07:22:42.299706+0000","last_fullsized":"2024-03-06T07:22:42.299706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:00.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281667+0000","last_change":"2024-03-06T07:22:42.281667+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281667+0000","last_undegraded":"2024-03-06T07:22:42.281667+0000","last_fullsized":"2024-03-06T07:22:42.281667+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288179+0000","last_change":"2024-03-06T07:22:42.287402+0000","last_active":"2024-03-06T07:22:42.288179+0000","last_peered":"2024-03-06T07:22:42.288179+0000","last_clean":"2024-03-06T07:22:42.288179+0000","last_became_active":"2024-03-06T07:22:42.287169+0000","last_became_peered":"2024-03-06T07:22:42.287169+0000","last_unstale":"2024-03-06T07:22:42.288179+0000","last_undegraded":"2024-03-06T07:22:42.288179+0000","last_fullsized":"2024-03-06T07:22:42.288179+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:39.487955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290484+0000","last_change":"2024-03-06T07:22:42.290484+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290484+0000","last_undegraded":"2024-03-06T07:22:42.290484+0000","last_fullsized":"2024-03-06T07:22:42.290484+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296242+0000","last_change":"2024-03-06T07:22:42.289384+0000","last_active":"2024-03-06T07:22:42.296242+0000","last_peered":"2024-03-06T07:22:42.296242+0000","last_clean":"2024-03-06T07:22:42.296242+0000","last_became_active":"2024-03-06T07:22:42.289045+0000","last_became_peered":"2024-03-06T07:22:42.289045+0000","last_unstale":"2024-03-06T07:22:42.296242+0000","last_undegraded":"2024-03-06T07:22:42.296242+0000","last_fullsized":"2024-03-06T07:22:42.296242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:08.837325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280053+0000","last_change":"2024-03-06T07:22:42.280053+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280053+0000","last_undegraded":"2024-03-06T07:22:42.280053+0000","last_fullsized":"2024-03-06T07:22:42.280053+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.295035+0000","last_change":"2024-03-06T07:22:42.294517+0000","last_active":"2024-03-06T07:22:42.295035+0000","last_peered":"2024-03-06T07:22:42.295035+0000","last_clean":"2024-03-06T07:22:42.295035+0000","last_became_active":"2024-03-06T07:22:42.294249+0000","last_became_peered":"2024-03-06T07:22:42.294249+0000","last_unstale":"2024-03-06T07:22:42.295035+0000","last_undegraded":"2024-03-06T07:22:42.295035+0000","last_fullsized":"2024-03-06T07:22:42.295035+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:12.295033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284708+0000","last_change":"2024-03-06T07:22:42.284708+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284708+0000","last_undegraded":"2024-03-06T07:22:42.284708+0000","last_fullsized":"2024-03-06T07:22:42.284708+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291835+0000","last_change":"2024-03-06T07:22:42.288271+0000","last_active":"2024-03-06T07:22:42.291835+0000","last_peered":"2024-03-06T07:22:42.291835+0000","last_clean":"2024-03-06T07:22:42.291835+0000","last_became_active":"2024-03-06T07:22:42.287981+0000","last_became_peered":"2024-03-06T07:22:42.287981+0000","last_unstale":"2024-03-06T07:22:42.291835+0000","last_undegraded":"2024-03-06T07:22:42.291835+0000","last_fullsized":"2024-03-06T07:22:42.291835+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.744017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286139+0000","last_change":"2024-03-06T07:22:42.286139+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286139+0000","last_undegraded":"2024-03-06T07:22:42.286139+0000","last_fullsized":"2024-03-06T07:22:42.286139+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289825+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288884+0000","last_became_peered":"2024-03-06T07:22:42.288884+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:05.920275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290881+0000","last_change":"2024-03-06T07:22:42.290881+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290881+0000","last_undegraded":"2024-03-06T07:22:42.290881+0000","last_fullsized":"2024-03-06T07:22:42.290881+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289606+0000","last_change":"2024-03-06T07:22:42.286639+0000","last_active":"2024-03-06T07:22:42.289606+0000","last_peered":"2024-03-06T07:22:42.289606+0000","last_clean":"2024-03-06T07:22:42.289606+0000","last_became_active":"2024-03-06T07:22:42.286392+0000","last_became_peered":"2024-03-06T07:22:42.286392+0000","last_unstale":"2024-03-06T07:22:42.289606+0000","last_undegraded":"2024-03-06T07:22:42.289606+0000","last_fullsized":"2024-03-06T07:22:42.289606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:40.885949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281607+0000","last_change":"2024-03-06T07:22:42.281607+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281607+0000","last_undegraded":"2024-03-06T07:22:42.281607+0000","last_fullsized":"2024-03-06T07:22:42.281607+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.285286+0000","last_change":"2024-03-06T07:22:42.284042+0000","last_active":"2024-03-06T07:22:42.285286+0000","last_peered":"2024-03-06T07:22:42.285286+0000","last_clean":"2024-03-06T07:22:42.285286+0000","last_became_active":"2024-03-06T07:22:42.283781+0000","last_became_peered":"2024-03-06T07:22:42.283781+0000","last_unstale":"2024-03-06T07:22:42.285286+0000","last_undegraded":"2024-03-06T07:22:42.285286+0000","last_fullsized":"2024-03-06T07:22:42.285286+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.123546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279871+0000","last_change":"2024-03-06T07:22:42.279871+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279871+0000","last_undegraded":"2024-03-06T07:22:42.279871+0000","last_fullsized":"2024-03-06T07:22:42.279871+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298230+0000","last_change":"2024-03-06T07:22:42.297131+0000","last_active":"2024-03-06T07:22:42.298230+0000","last_peered":"2024-03-06T07:22:42.298230+0000","last_clean":"2024-03-06T07:22:42.298230+0000","last_became_active":"2024-03-06T07:22:42.296815+0000","last_became_peered":"2024-03-06T07:22:42.296815+0000","last_unstale":"2024-03-06T07:22:42.298230+0000","last_undegraded":"2024-03-06T07:22:42.298230+0000","last_fullsized":"2024-03-06T07:22:42.298230+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:56.374398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277869+0000","last_change":"2024-03-06T07:22:42.277869+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277869+0000","last_undegraded":"2024-03-06T07:22:42.277869+0000","last_fullsized":"2024-03-06T07:22:42.277869+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289232+0000","last_change":"2024-03-06T07:22:42.288618+0000","last_active":"2024-03-06T07:22:42.289232+0000","last_peered":"2024-03-06T07:22:42.289232+0000","last_clean":"2024-03-06T07:22:42.289232+0000","last_became_active":"2024-03-06T07:22:42.287863+0000","last_became_peered":"2024-03-06T07:22:42.287863+0000","last_unstale":"2024-03-06T07:22:42.289232+0000","last_undegraded":"2024-03-06T07:22:42.289232+0000","last_fullsized":"2024-03-06T07:22:42.289232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:06.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278812+0000","last_change":"2024-03-06T07:22:42.278812+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278812+0000","last_undegraded":"2024-03-06T07:22:42.278812+0000","last_fullsized":"2024-03-06T07:22:42.278812+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298112+0000","last_change":"2024-03-06T07:22:42.293801+0000","last_active":"2024-03-06T07:22:42.298112+0000","last_peered":"2024-03-06T07:22:42.298112+0000","last_clean":"2024-03-06T07:22:42.298112+0000","last_became_active":"2024-03-06T07:22:42.293537+0000","last_became_peered":"2024-03-06T07:22:42.293537+0000","last_unstale":"2024-03-06T07:22:42.298112+0000","last_undegraded":"2024-03-06T07:22:42.298112+0000","last_fullsized":"2024-03-06T07:22:42.298112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:05.097698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283044+0000","last_change":"2024-03-06T07:22:42.283044+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283044+0000","last_undegraded":"2024-03-06T07:22:42.283044+0000","last_fullsized":"2024-03-06T07:22:42.283044+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289637+0000","last_change":"2024-03-06T07:22:42.287323+0000","last_active":"2024-03-06T07:22:42.289637+0000","last_peered":"2024-03-06T07:22:42.289637+0000","last_clean":"2024-03-06T07:22:42.289637+0000","last_became_active":"2024-03-06T07:22:42.287077+0000","last_became_peered":"2024-03-06T07:22:42.287077+0000","last_unstale":"2024-03-06T07:22:42.289637+0000","last_undegraded":"2024-03-06T07:22:42.289637+0000","last_fullsized":"2024-03-06T07:22:42.289637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:24.927357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278746+0000","last_change":"2024-03-06T07:22:42.278746+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278746+0000","last_undegraded":"2024-03-06T07:22:42.278746+0000","last_fullsized":"2024-03-06T07:22:42.278746+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291008+0000","last_change":"2024-03-06T07:22:42.290240+0000","last_active":"2024-03-06T07:22:42.291008+0000","last_peered":"2024-03-06T07:22:42.291008+0000","last_clean":"2024-03-06T07:22:42.291008+0000","last_became_active":"2024-03-06T07:22:42.289934+0000","last_became_peered":"2024-03-06T07:22:42.289934+0000","last_unstale":"2024-03-06T07:22:42.291008+0000","last_undegraded":"2024-03-06T07:22:42.291008+0000","last_fullsized":"2024-03-06T07:22:42.291008+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:58.470531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282416+0000","last_change":"2024-03-06T07:22:42.282416+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282416+0000","last_undegraded":"2024-03-06T07:22:42.282416+0000","last_fullsized":"2024-03-06T07:22:42.282416+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.293325+0000","last_change":"2024-03-06T07:22:42.292354+0000","last_active":"2024-03-06T07:22:42.293325+0000","last_peered":"2024-03-06T07:22:42.293325+0000","last_clean":"2024-03-06T07:22:42.293325+0000","last_became_active":"2024-03-06T07:22:42.292033+0000","last_became_peered":"2024-03-06T07:22:42.292033+0000","last_unstale":"2024-03-06T07:22:42.293325+0000","last_undegraded":"2024-03-06T07:22:42.293325+0000","last_fullsized":"2024-03-06T07:22:42.293325+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:31.588280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.289810+0000","last_change":"2024-03-06T07:22:42.289810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.289810+0000","last_undegraded":"2024-03-06T07:22:42.289810+0000","last_fullsized":"2024-03-06T07:22:42.289810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291470+0000","last_change":"2024-03-06T07:22:42.288768+0000","last_active":"2024-03-06T07:22:42.291470+0000","last_peered":"2024-03-06T07:22:42.291470+0000","last_clean":"2024-03-06T07:22:42.291470+0000","last_became_active":"2024-03-06T07:22:42.288476+0000","last_became_peered":"2024-03-06T07:22:42.288476+0000","last_unstale":"2024-03-06T07:22:42.291470+0000","last_undegraded":"2024-03-06T07:22:42.291470+0000","last_fullsized":"2024-03-06T07:22:42.291470+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.908267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281207+0000","last_change":"2024-03-06T07:22:42.281207+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281207+0000","last_undegraded":"2024-03-06T07:22:42.281207+0000","last_fullsized":"2024-03-06T07:22:42.281207+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294770+0000","last_change":"2024-03-06T07:22:42.293943+0000","last_active":"2024-03-06T07:22:42.294770+0000","last_peered":"2024-03-06T07:22:42.294770+0000","last_clean":"2024-03-06T07:22:42.294770+0000","last_became_active":"2024-03-06T07:22:42.293661+0000","last_became_peered":"2024-03-06T07:22:42.293661+0000","last_unstale":"2024-03-06T07:22:42.294770+0000","last_undegraded":"2024-03-06T07:22:42.294770+0000","last_fullsized":"2024-03-06T07:22:42.294770+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:51.825409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280504+0000","last_change":"2024-03-06T07:22:42.280504+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280504+0000","last_undegraded":"2024-03-06T07:22:42.280504+0000","last_fullsized":"2024-03-06T07:22:42.280504+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291595+0000","last_change":"2024-03-06T07:22:42.287760+0000","last_active":"2024-03-06T07:22:42.291595+0000","last_peered":"2024-03-06T07:22:42.291595+0000","last_clean":"2024-03-06T07:22:42.291595+0000","last_became_active":"2024-03-06T07:22:42.287466+0000","last_became_peered":"2024-03-06T07:22:42.287466+0000","last_unstale":"2024-03-06T07:22:42.291595+0000","last_undegraded":"2024-03-06T07:22:42.291595+0000","last_fullsized":"2024-03-06T07:22:42.291595+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:48.429074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280985+0000","last_change":"2024-03-06T07:22:42.280985+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280985+0000","last_undegraded":"2024-03-06T07:22:42.280985+0000","last_fullsized":"2024-03-06T07:22:42.280985+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298342+0000","last_change":"2024-03-06T07:22:42.296157+0000","last_active":"2024-03-06T07:22:42.298342+0000","last_peered":"2024-03-06T07:22:42.298342+0000","last_clean":"2024-03-06T07:22:42.298342+0000","last_became_active":"2024-03-06T07:22:42.295713+0000","last_became_peered":"2024-03-06T07:22:42.295713+0000","last_unstale":"2024-03-06T07:22:42.298342+0000","last_undegraded":"2024-03-06T07:22:42.298342+0000","last_fullsized":"2024-03-06T07:22:42.298342+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:11.158468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281379+0000","last_change":"2024-03-06T07:22:42.281379+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281379+0000","last_undegraded":"2024-03-06T07:22:42.281379+0000","last_fullsized":"2024-03-06T07:22:42.281379+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294125+0000","last_change":"2024-03-06T07:22:42.293654+0000","last_active":"2024-03-06T07:22:42.294125+0000","last_peered":"2024-03-06T07:22:42.294125+0000","last_clean":"2024-03-06T07:22:42.294125+0000","last_became_active":"2024-03-06T07:22:42.293361+0000","last_became_peered":"2024-03-06T07:22:42.293361+0000","last_unstale":"2024-03-06T07:22:42.294125+0000","last_undegraded":"2024-03-06T07:22:42.294125+0000","last_fullsized":"2024-03-06T07:22:42.294125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:03.485795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290349+0000","last_change":"2024-03-06T07:22:42.290349+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290349+0000","last_undegraded":"2024-03-06T07:22:42.290349+0000","last_fullsized":"2024-03-06T07:22:42.290349+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291039+0000","last_change":"2024-03-06T07:22:42.290440+0000","last_active":"2024-03-06T07:22:42.291039+0000","last_peered":"2024-03-06T07:22:42.291039+0000","last_clean":"2024-03-06T07:22:42.291039+0000","last_became_active":"2024-03-06T07:22:42.290162+0000","last_became_peered":"2024-03-06T07:22:42.290162+0000","last_unstale":"2024-03-06T07:22:42.291039+0000","last_undegraded":"2024-03-06T07:22:42.291039+0000","last_fullsized":"2024-03-06T07:22:42.291039+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:02.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283399+0000","last_change":"2024-03-06T07:22:42.283399+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283399+0000","last_undegraded":"2024-03-06T07:22:42.283399+0000","last_fullsized":"2024-03-06T07:22:42.283399+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298125+0000","last_change":"2024-03-06T07:22:42.293720+0000","last_active":"2024-03-06T07:22:42.298125+0000","last_peered":"2024-03-06T07:22:42.298125+0000","last_clean":"2024-03-06T07:22:42.298125+0000","last_became_active":"2024-03-06T07:22:42.293429+0000","last_became_peered":"2024-03-06T07:22:42.293429+0000","last_unstale":"2024-03-06T07:22:42.298125+0000","last_undegraded":"2024-03-06T07:22:42.298125+0000","last_fullsized":"2024-03-06T07:22:42.298125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:55.471196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":357,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291293+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:22:42.291293+0000","last_peered":"2024-03-06T07:22:42.291293+0000","last_clean":"2024-03-06T07:22:42.291293+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:22:42.291293+0000","last_undegraded":"2024-03-06T07:22:42.291293+0000","last_fullsized":"2024-03-06T07:22:42.291293+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1490944,"data_stored":715572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:46.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:46.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.372+0000 7f33c31c9640 1 Processor -- start 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.372+0000 7f33c31c9640 1 -- start start 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.372+0000 7f33c31c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc073250 0x7f33bc0c3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.372+0000 7f33c31c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f33bc0c3950 0x7f33bc15fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.372+0000 7f33c31c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33bc0bdc20 con 0x7f33bc072e70 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.372+0000 7f33c31c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33bc06c8e0 con 0x7f33bc0c3950 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.372+0000 7f33c31c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc06ce80 con 0x7f33bc073250 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33c0f65640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f33bc072e70 0x7f33bc0bd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33970/0 (socket says 172.21.15.94:33970) 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33c0f65640 1 -- 172.21.15.94:0/3552462650 learned_addr learned my addr 172.21.15.94:0/3552462650 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33c1766640 1 --2- 172.21.15.94:0/3552462650 >> v2:172.21.15.145:3300/0 conn(0x7f33bc0c3950 0x7f33bc15fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33bbfff640 1 --2- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc073250 0x7f33bc0c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33bb7fe640 1 -- 172.21.15.94:0/3552462650 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082075217 0 0) 0x7f33bc0bdc20 con 0x7f33bc072e70 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33bb7fe640 1 -- 172.21.15.94:0/3552462650 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a8003520 con 0x7f33bc072e70 2024-03-06T07:22:46.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33c1766640 1 -- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc073250 msgr2=0x7f33bc0c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33c1766640 1 --2- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc073250 0x7f33bc0c3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33c1766640 1 -- 172.21.15.94:0/3552462650 >> v1:172.21.15.94:6789/0 conn(0x7f33bc072e70 legacy=0x7f33bc0bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.373+0000 7f33c1766640 1 -- 172.21.15.94:0/3552462650 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc0c1b00 con 0x7f33bc0c3950 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c1766640 1 --2- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc0c3950 0x7f33bc15fce0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f33a40097a0 tx=0x7f33a402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fbfa936e0fec32d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33bb7fe640 1 -- 172.21.15.94:0/3552462650 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a40315a0 con 0x7f33bc0c3950 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33bb7fe640 1 -- 172.21.15.94:0/3552462650 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a4037070 con 0x7f33bc0c3950 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33bb7fe640 1 -- 172.21.15.94:0/3552462650 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4031940 con 0x7f33bc0c3950 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 -- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc0c3950 msgr2=0x7f33bc15fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc0c3950 0x7f33bc15fce0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f33a40097a0 tx=0x7f33a402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 -- 172.21.15.94:0/3552462650 shutdown_connections 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc0c3950 0x7f33bc15fce0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3552462650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc073250 0x7f33bc0c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 -- 172.21.15.94:0/3552462650 >> 172.21.15.94:0/3552462650 conn(0x7f33bc06b1a0 msgr2=0x7f33bc06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 -- 172.21.15.94:0/3552462650 shutdown_connections 2024-03-06T07:22:46.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.374+0000 7f33c31c9640 1 -- 172.21.15.94:0/3552462650 wait complete. 2024-03-06T07:22:46.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.375+0000 7f33c31c9640 1 Processor -- start 2024-03-06T07:22:46.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c31c9640 1 -- start start 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c31c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33bc072e70 0x7f33bc0aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c31c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc073250 0x7f33bc157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c31c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc0c3950 0x7f33bc1e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c31c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0c6760 con 0x7f33bc072e70 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c31c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc0c4a00 con 0x7f33bc073250 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c31c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc1573f0 con 0x7f33bc0c3950 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c0f65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33bc072e70 0x7f33bc0aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c0f65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33bc072e70 0x7f33bc0aa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36290/0 (socket says 172.21.15.94:36290) 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.376+0000 7f33c0f65640 1 -- 172.21.15.94:0/3699559073 learned_addr learned my addr 172.21.15.94:0/3699559073 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c1766640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc0c3950 0x7f33bc1e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33bbfff640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc073250 0x7f33bc157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c0f65640 1 -- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc0c3950 msgr2=0x7f33bc1e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c0f65640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc0c3950 0x7f33bc1e7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c0f65640 1 -- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc073250 msgr2=0x7f33bc157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c0f65640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc073250 0x7f33bc157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c0f65640 1 -- 172.21.15.94:0/3699559073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc1590f0 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c0f65640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33bc072e70 0x7f33bc0aa1e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f33b400caa0 tx=0x7f33b400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33b97fa640 1 -- 172.21.15.94:0/3699559073 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4017070 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33b97fa640 1 -- 172.21.15.94:0/3699559073 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b400ac00 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc1593d0 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33b97fa640 1 -- 172.21.15.94:0/3699559073 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4012400 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.377+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc158300 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.378+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc04f330 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.379+0000 7f33b97fa640 1 -- 172.21.15.94:0/3699559073 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33b401a070 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.379+0000 7f33b97fa640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f339c06b500 0x7f339c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.379+0000 7f33b97fa640 1 -- 172.21.15.94:0/3699559073 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f33b408a080 con 0x7f33bc072e70 2024-03-06T07:22:46.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.379+0000 7f33bbfff640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f339c06b500 0x7f339c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.380+0000 7f33bbfff640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f339c06b500 0x7f339c06d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f33bc072840 tx=0x7f33ac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.384+0000 7f33b97fa640 1 -- 172.21.15.94:0/3699559073 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33b40587b0 con 0x7f33bc072e70 2024-03-06T07:22:46.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.552+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33bc1598f0 con 0x7f339c06b500 2024-03-06T07:22:46.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.558+0000 7f33b97fa640 1 -- 172.21.15.94:0/3699559073 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247154 (secure 0 0 0) 0x7f33bc1598f0 con 0x7f339c06b500 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:46.564 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f339c06b500 msgr2=0x7f339c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f339c06b500 0x7f339c06d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f33bc072840 tx=0x7f33ac009190 comp rx=0 tx=0).stop 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33bc072e70 msgr2=0x7f33bc0aa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33bc072e70 0x7f33bc0aa1e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f33b400caa0 tx=0x7f33b400cf60 comp rx=0 tx=0).stop 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 shutdown_connections 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f339c06b500 0x7f339c06d9c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33bc0c3950 0x7f33bc1e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33bc073250 0x7f33bc157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 --2- 172.21.15.94:0/3699559073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f33bc072e70 0x7f33bc0aa1e0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 >> 172.21.15.94:0/3699559073 conn(0x7f33bc06b1a0 msgr2=0x7f33bc0c28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 shutdown_connections 2024-03-06T07:22:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.561+0000 7f33c31c9640 1 -- 172.21.15.94:0/3699559073 wait complete. 2024-03-06T07:22:46.574 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-03-06T07:22:44.693154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261148,"kb_used_data":11692,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713572,"statfs":{"total":773094113280,"available":772826697728,"internally_reserved":0,"allocated":11972608,"data_stored":6858699,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001252"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279032+0000","last_change":"2024-03-06T07:22:42.279032+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279032+0000","last_undegraded":"2024-03-06T07:22:42.279032+0000","last_fullsized":"2024-03-06T07:22:42.279032+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284203+0000","last_change":"2024-03-06T07:22:42.284203+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284203+0000","last_undegraded":"2024-03-06T07:22:42.284203+0000","last_fullsized":"2024-03-06T07:22:42.284203+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284568+0000","last_change":"2024-03-06T07:22:42.284568+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284568+0000","last_undegraded":"2024-03-06T07:22:42.284568+0000","last_fullsized":"2024-03-06T07:22:42.284568+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279457+0000","last_change":"2024-03-06T07:22:42.279457+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279457+0000","last_undegraded":"2024-03-06T07:22:42.279457+0000","last_fullsized":"2024-03-06T07:22:42.279457+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283015+0000","last_change":"2024-03-06T07:22:42.283015+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283015+0000","last_undegraded":"2024-03-06T07:22:42.283015+0000","last_fullsized":"2024-03-06T07:22:42.283015+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292601+0000","last_change":"2024-03-06T07:22:42.292601+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292601+0000","last_undegraded":"2024-03-06T07:22:42.292601+0000","last_fullsized":"2024-03-06T07:22:42.292601+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.285418+0000","last_change":"2024-03-06T07:22:42.285418+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.285418+0000","last_undegraded":"2024-03-06T07:22:42.285418+0000","last_fullsized":"2024-03-06T07:22:42.285418+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282898+0000","last_change":"2024-03-06T07:22:42.282898+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282898+0000","last_undegraded":"2024-03-06T07:22:42.282898+0000","last_fullsized":"2024-03-06T07:22:42.282898+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282437+0000","last_change":"2024-03-06T07:22:42.282437+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282437+0000","last_undegraded":"2024-03-06T07:22:42.282437+0000","last_fullsized":"2024-03-06T07:22:42.282437+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283810+0000","last_change":"2024-03-06T07:22:42.283810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283810+0000","last_undegraded":"2024-03-06T07:22:42.283810+0000","last_fullsized":"2024-03-06T07:22:42.283810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278978+0000","last_change":"2024-03-06T07:22:42.278978+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278978+0000","last_undegraded":"2024-03-06T07:22:42.278978+0000","last_fullsized":"2024-03-06T07:22:42.278978+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282804+0000","last_change":"2024-03-06T07:22:42.282804+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282804+0000","last_undegraded":"2024-03-06T07:22:42.282804+0000","last_fullsized":"2024-03-06T07:22:42.282804+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278897+0000","last_change":"2024-03-06T07:22:42.278897+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278897+0000","last_undegraded":"2024-03-06T07:22:42.278897+0000","last_fullsized":"2024-03-06T07:22:42.278897+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278443+0000","last_change":"2024-03-06T07:22:42.278443+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278443+0000","last_undegraded":"2024-03-06T07:22:42.278443+0000","last_fullsized":"2024-03-06T07:22:42.278443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283228+0000","last_change":"2024-03-06T07:22:42.283228+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283228+0000","last_undegraded":"2024-03-06T07:22:42.283228+0000","last_fullsized":"2024-03-06T07:22:42.283228+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282177+0000","last_change":"2024-03-06T07:22:42.282177+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282177+0000","last_undegraded":"2024-03-06T07:22:42.282177+0000","last_fullsized":"2024-03-06T07:22:42.282177+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279562+0000","last_change":"2024-03-06T07:22:42.279562+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279562+0000","last_undegraded":"2024-03-06T07:22:42.279562+0000","last_fullsized":"2024-03-06T07:22:42.279562+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.293286+0000","last_change":"2024-03-06T07:22:42.293286+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.293286+0000","last_undegraded":"2024-03-06T07:22:42.293286+0000","last_fullsized":"2024-03-06T07:22:42.293286+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280297+0000","last_change":"2024-03-06T07:22:42.280297+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280297+0000","last_undegraded":"2024-03-06T07:22:42.280297+0000","last_fullsized":"2024-03-06T07:22:42.280297+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283843+0000","last_change":"2024-03-06T07:22:42.283843+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283843+0000","last_undegraded":"2024-03-06T07:22:42.283843+0000","last_fullsized":"2024-03-06T07:22:42.283843+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278394+0000","last_change":"2024-03-06T07:22:42.278394+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278394+0000","last_undegraded":"2024-03-06T07:22:42.278394+0000","last_fullsized":"2024-03-06T07:22:42.278394+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284222+0000","last_change":"2024-03-06T07:22:42.284222+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284222+0000","last_undegraded":"2024-03-06T07:22:42.284222+0000","last_fullsized":"2024-03-06T07:22:42.284222+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282160+0000","last_change":"2024-03-06T07:22:42.282160+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282160+0000","last_undegraded":"2024-03-06T07:22:42.282160+0000","last_fullsized":"2024-03-06T07:22:42.282160+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286365+0000","last_change":"2024-03-06T07:22:42.286365+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286365+0000","last_undegraded":"2024-03-06T07:22:42.286365+0000","last_fullsized":"2024-03-06T07:22:42.286365+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286778+0000","last_change":"2024-03-06T07:22:42.286778+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286778+0000","last_undegraded":"2024-03-06T07:22:42.286778+0000","last_fullsized":"2024-03-06T07:22:42.286778+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292889+0000","last_change":"2024-03-06T07:22:42.292889+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292889+0000","last_undegraded":"2024-03-06T07:22:42.292889+0000","last_fullsized":"2024-03-06T07:22:42.292889+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282008+0000","last_change":"2024-03-06T07:22:42.282008+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282008+0000","last_undegraded":"2024-03-06T07:22:42.282008+0000","last_fullsized":"2024-03-06T07:22:42.282008+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286216+0000","last_change":"2024-03-06T07:22:42.286216+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286216+0000","last_undegraded":"2024-03-06T07:22:42.286216+0000","last_fullsized":"2024-03-06T07:22:42.286216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281552+0000","last_change":"2024-03-06T07:22:42.281552+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281552+0000","last_undegraded":"2024-03-06T07:22:42.281552+0000","last_fullsized":"2024-03-06T07:22:42.281552+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281197+0000","last_change":"2024-03-06T07:22:42.281197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281197+0000","last_undegraded":"2024-03-06T07:22:42.281197+0000","last_fullsized":"2024-03-06T07:22:42.281197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281292+0000","last_change":"2024-03-06T07:22:42.281292+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281292+0000","last_undegraded":"2024-03-06T07:22:42.281292+0000","last_fullsized":"2024-03-06T07:22:42.281292+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289168+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288869+0000","last_became_peered":"2024-03-06T07:22:42.288869+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:22.433887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277947+0000","last_change":"2024-03-06T07:22:42.277947+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277947+0000","last_undegraded":"2024-03-06T07:22:42.277947+0000","last_fullsized":"2024-03-06T07:22:42.277947+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.284819+0000","last_change":"2024-03-06T07:22:42.284230+0000","last_active":"2024-03-06T07:22:42.284819+0000","last_peered":"2024-03-06T07:22:42.284819+0000","last_clean":"2024-03-06T07:22:42.284819+0000","last_became_active":"2024-03-06T07:22:42.283796+0000","last_became_peered":"2024-03-06T07:22:42.283796+0000","last_unstale":"2024-03-06T07:22:42.284819+0000","last_undegraded":"2024-03-06T07:22:42.284819+0000","last_fullsized":"2024-03-06T07:22:42.284819+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:28.750113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283761+0000","last_change":"2024-03-06T07:22:42.283761+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283761+0000","last_undegraded":"2024-03-06T07:22:42.283761+0000","last_fullsized":"2024-03-06T07:22:42.283761+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299956+0000","last_change":"2024-03-06T07:22:42.298123+0000","last_active":"2024-03-06T07:22:42.299956+0000","last_peered":"2024-03-06T07:22:42.299956+0000","last_clean":"2024-03-06T07:22:42.299956+0000","last_became_active":"2024-03-06T07:22:42.297656+0000","last_became_peered":"2024-03-06T07:22:42.297656+0000","last_unstale":"2024-03-06T07:22:42.299956+0000","last_undegraded":"2024-03-06T07:22:42.299956+0000","last_fullsized":"2024-03-06T07:22:42.299956+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:56.531364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277964+0000","last_change":"2024-03-06T07:22:42.277964+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277964+0000","last_undegraded":"2024-03-06T07:22:42.277964+0000","last_fullsized":"2024-03-06T07:22:42.277964+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298336+0000","last_change":"2024-03-06T07:22:42.295819+0000","last_active":"2024-03-06T07:22:42.298336+0000","last_peered":"2024-03-06T07:22:42.298336+0000","last_clean":"2024-03-06T07:22:42.298336+0000","last_became_active":"2024-03-06T07:22:42.295541+0000","last_became_peered":"2024-03-06T07:22:42.295541+0000","last_unstale":"2024-03-06T07:22:42.298336+0000","last_undegraded":"2024-03-06T07:22:42.298336+0000","last_fullsized":"2024-03-06T07:22:42.298336+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:51.235851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282847+0000","last_change":"2024-03-06T07:22:42.282847+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282847+0000","last_undegraded":"2024-03-06T07:22:42.282847+0000","last_fullsized":"2024-03-06T07:22:42.282847+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289767+0000","last_change":"2024-03-06T07:22:42.287120+0000","last_active":"2024-03-06T07:22:42.289767+0000","last_peered":"2024-03-06T07:22:42.289767+0000","last_clean":"2024-03-06T07:22:42.289767+0000","last_became_active":"2024-03-06T07:22:42.286844+0000","last_became_peered":"2024-03-06T07:22:42.286844+0000","last_unstale":"2024-03-06T07:22:42.289767+0000","last_undegraded":"2024-03-06T07:22:42.289767+0000","last_fullsized":"2024-03-06T07:22:42.289767+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.165551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283519+0000","last_change":"2024-03-06T07:22:42.283519+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283519+0000","last_undegraded":"2024-03-06T07:22:42.283519+0000","last_fullsized":"2024-03-06T07:22:42.283519+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.290193+0000","last_change":"2024-03-06T07:22:42.289388+0000","last_active":"2024-03-06T07:22:42.290193+0000","last_peered":"2024-03-06T07:22:42.290193+0000","last_clean":"2024-03-06T07:22:42.290193+0000","last_became_active":"2024-03-06T07:22:42.288148+0000","last_became_peered":"2024-03-06T07:22:42.288148+0000","last_unstale":"2024-03-06T07:22:42.290193+0000","last_undegraded":"2024-03-06T07:22:42.290193+0000","last_fullsized":"2024-03-06T07:22:42.290193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:37.212217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279197+0000","last_change":"2024-03-06T07:22:42.279197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279197+0000","last_undegraded":"2024-03-06T07:22:42.279197+0000","last_fullsized":"2024-03-06T07:22:42.279197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.287719+0000","last_change":"2024-03-06T07:22:42.285183+0000","last_active":"2024-03-06T07:22:42.287719+0000","last_peered":"2024-03-06T07:22:42.287719+0000","last_clean":"2024-03-06T07:22:42.287719+0000","last_became_active":"2024-03-06T07:22:42.284924+0000","last_became_peered":"2024-03-06T07:22:42.284924+0000","last_unstale":"2024-03-06T07:22:42.287719+0000","last_undegraded":"2024-03-06T07:22:42.287719+0000","last_fullsized":"2024-03-06T07:22:42.287719+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:21.527693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292509+0000","last_change":"2024-03-06T07:22:42.292509+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292509+0000","last_undegraded":"2024-03-06T07:22:42.292509+0000","last_fullsized":"2024-03-06T07:22:42.292509+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296297+0000","last_change":"2024-03-06T07:22:42.289621+0000","last_active":"2024-03-06T07:22:42.296297+0000","last_peered":"2024-03-06T07:22:42.296297+0000","last_clean":"2024-03-06T07:22:42.296297+0000","last_became_active":"2024-03-06T07:22:42.289239+0000","last_became_peered":"2024-03-06T07:22:42.289239+0000","last_unstale":"2024-03-06T07:22:42.296297+0000","last_undegraded":"2024-03-06T07:22:42.296297+0000","last_fullsized":"2024-03-06T07:22:42.296297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:40.681976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277967+0000","last_change":"2024-03-06T07:22:42.277967+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277967+0000","last_undegraded":"2024-03-06T07:22:42.277967+0000","last_fullsized":"2024-03-06T07:22:42.277967+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291813+0000","last_change":"2024-03-06T07:22:42.287554+0000","last_active":"2024-03-06T07:22:42.291813+0000","last_peered":"2024-03-06T07:22:42.291813+0000","last_clean":"2024-03-06T07:22:42.291813+0000","last_became_active":"2024-03-06T07:22:42.285213+0000","last_became_peered":"2024-03-06T07:22:42.285213+0000","last_unstale":"2024-03-06T07:22:42.291813+0000","last_undegraded":"2024-03-06T07:22:42.291813+0000","last_fullsized":"2024-03-06T07:22:42.291813+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:07.580402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281750+0000","last_change":"2024-03-06T07:22:42.281750+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281750+0000","last_undegraded":"2024-03-06T07:22:42.281750+0000","last_fullsized":"2024-03-06T07:22:42.281750+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288109+0000","last_change":"2024-03-06T07:22:42.286796+0000","last_active":"2024-03-06T07:22:42.288109+0000","last_peered":"2024-03-06T07:22:42.288109+0000","last_clean":"2024-03-06T07:22:42.288109+0000","last_became_active":"2024-03-06T07:22:42.286558+0000","last_became_peered":"2024-03-06T07:22:42.286558+0000","last_unstale":"2024-03-06T07:22:42.288109+0000","last_undegraded":"2024-03-06T07:22:42.288109+0000","last_fullsized":"2024-03-06T07:22:42.288109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:11.589911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281997+0000","last_change":"2024-03-06T07:22:42.281997+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281997+0000","last_undegraded":"2024-03-06T07:22:42.281997+0000","last_fullsized":"2024-03-06T07:22:42.281997+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299969+0000","last_change":"2024-03-06T07:22:42.298748+0000","last_active":"2024-03-06T07:22:42.299969+0000","last_peered":"2024-03-06T07:22:42.299969+0000","last_clean":"2024-03-06T07:22:42.299969+0000","last_became_active":"2024-03-06T07:22:42.298459+0000","last_became_peered":"2024-03-06T07:22:42.298459+0000","last_unstale":"2024-03-06T07:22:42.299969+0000","last_undegraded":"2024-03-06T07:22:42.299969+0000","last_fullsized":"2024-03-06T07:22:42.299969+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:08.721690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281717+0000","last_change":"2024-03-06T07:22:42.281717+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281717+0000","last_undegraded":"2024-03-06T07:22:42.281717+0000","last_fullsized":"2024-03-06T07:22:42.281717+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299706+0000","last_change":"2024-03-06T07:22:42.297582+0000","last_active":"2024-03-06T07:22:42.299706+0000","last_peered":"2024-03-06T07:22:42.299706+0000","last_clean":"2024-03-06T07:22:42.299706+0000","last_became_active":"2024-03-06T07:22:42.297309+0000","last_became_peered":"2024-03-06T07:22:42.297309+0000","last_unstale":"2024-03-06T07:22:42.299706+0000","last_undegraded":"2024-03-06T07:22:42.299706+0000","last_fullsized":"2024-03-06T07:22:42.299706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:00.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281667+0000","last_change":"2024-03-06T07:22:42.281667+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281667+0000","last_undegraded":"2024-03-06T07:22:42.281667+0000","last_fullsized":"2024-03-06T07:22:42.281667+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288179+0000","last_change":"2024-03-06T07:22:42.287402+0000","last_active":"2024-03-06T07:22:42.288179+0000","last_peered":"2024-03-06T07:22:42.288179+0000","last_clean":"2024-03-06T07:22:42.288179+0000","last_became_active":"2024-03-06T07:22:42.287169+0000","last_became_peered":"2024-03-06T07:22:42.287169+0000","last_unstale":"2024-03-06T07:22:42.288179+0000","last_undegraded":"2024-03-06T07:22:42.288179+0000","last_fullsized":"2024-03-06T07:22:42.288179+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:39.487955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290484+0000","last_change":"2024-03-06T07:22:42.290484+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290484+0000","last_undegraded":"2024-03-06T07:22:42.290484+0000","last_fullsized":"2024-03-06T07:22:42.290484+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296242+0000","last_change":"2024-03-06T07:22:42.289384+0000","last_active":"2024-03-06T07:22:42.296242+0000","last_peered":"2024-03-06T07:22:42.296242+0000","last_clean":"2024-03-06T07:22:42.296242+0000","last_became_active":"2024-03-06T07:22:42.289045+0000","last_became_peered":"2024-03-06T07:22:42.289045+0000","last_unstale":"2024-03-06T07:22:42.296242+0000","last_undegraded":"2024-03-06T07:22:42.296242+0000","last_fullsized":"2024-03-06T07:22:42.296242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:08.837325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280053+0000","last_change":"2024-03-06T07:22:42.280053+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280053+0000","last_undegraded":"2024-03-06T07:22:42.280053+0000","last_fullsized":"2024-03-06T07:22:42.280053+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.295035+0000","last_change":"2024-03-06T07:22:42.294517+0000","last_active":"2024-03-06T07:22:42.295035+0000","last_peered":"2024-03-06T07:22:42.295035+0000","last_clean":"2024-03-06T07:22:42.295035+0000","last_became_active":"2024-03-06T07:22:42.294249+0000","last_became_peered":"2024-03-06T07:22:42.294249+0000","last_unstale":"2024-03-06T07:22:42.295035+0000","last_undegraded":"2024-03-06T07:22:42.295035+0000","last_fullsized":"2024-03-06T07:22:42.295035+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:12.295033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284708+0000","last_change":"2024-03-06T07:22:42.284708+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284708+0000","last_undegraded":"2024-03-06T07:22:42.284708+0000","last_fullsized":"2024-03-06T07:22:42.284708+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291835+0000","last_change":"2024-03-06T07:22:42.288271+0000","last_active":"2024-03-06T07:22:42.291835+0000","last_peered":"2024-03-06T07:22:42.291835+0000","last_clean":"2024-03-06T07:22:42.291835+0000","last_became_active":"2024-03-06T07:22:42.287981+0000","last_became_peered":"2024-03-06T07:22:42.287981+0000","last_unstale":"2024-03-06T07:22:42.291835+0000","last_undegraded":"2024-03-06T07:22:42.291835+0000","last_fullsized":"2024-03-06T07:22:42.291835+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.744017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286139+0000","last_change":"2024-03-06T07:22:42.286139+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286139+0000","last_undegraded":"2024-03-06T07:22:42.286139+0000","last_fullsized":"2024-03-06T07:22:42.286139+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289825+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288884+0000","last_became_peered":"2024-03-06T07:22:42.288884+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:05.920275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290881+0000","last_change":"2024-03-06T07:22:42.290881+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290881+0000","last_undegraded":"2024-03-06T07:22:42.290881+0000","last_fullsized":"2024-03-06T07:22:42.290881+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289606+0000","last_change":"2024-03-06T07:22:42.286639+0000","last_active":"2024-03-06T07:22:42.289606+0000","last_peered":"2024-03-06T07:22:42.289606+0000","last_clean":"2024-03-06T07:22:42.289606+0000","last_became_active":"2024-03-06T07:22:42.286392+0000","last_became_peered":"2024-03-06T07:22:42.286392+0000","last_unstale":"2024-03-06T07:22:42.289606+0000","last_undegraded":"2024-03-06T07:22:42.289606+0000","last_fullsized":"2024-03-06T07:22:42.289606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:40.885949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281607+0000","last_change":"2024-03-06T07:22:42.281607+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281607+0000","last_undegraded":"2024-03-06T07:22:42.281607+0000","last_fullsized":"2024-03-06T07:22:42.281607+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.285286+0000","last_change":"2024-03-06T07:22:42.284042+0000","last_active":"2024-03-06T07:22:42.285286+0000","last_peered":"2024-03-06T07:22:42.285286+0000","last_clean":"2024-03-06T07:22:42.285286+0000","last_became_active":"2024-03-06T07:22:42.283781+0000","last_became_peered":"2024-03-06T07:22:42.283781+0000","last_unstale":"2024-03-06T07:22:42.285286+0000","last_undegraded":"2024-03-06T07:22:42.285286+0000","last_fullsized":"2024-03-06T07:22:42.285286+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.123546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279871+0000","last_change":"2024-03-06T07:22:42.279871+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279871+0000","last_undegraded":"2024-03-06T07:22:42.279871+0000","last_fullsized":"2024-03-06T07:22:42.279871+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298230+0000","last_change":"2024-03-06T07:22:42.297131+0000","last_active":"2024-03-06T07:22:42.298230+0000","last_peered":"2024-03-06T07:22:42.298230+0000","last_clean":"2024-03-06T07:22:42.298230+0000","last_became_active":"2024-03-06T07:22:42.296815+0000","last_became_peered":"2024-03-06T07:22:42.296815+0000","last_unstale":"2024-03-06T07:22:42.298230+0000","last_undegraded":"2024-03-06T07:22:42.298230+0000","last_fullsized":"2024-03-06T07:22:42.298230+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:56.374398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277869+0000","last_change":"2024-03-06T07:22:42.277869+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277869+0000","last_undegraded":"2024-03-06T07:22:42.277869+0000","last_fullsized":"2024-03-06T07:22:42.277869+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289232+0000","last_change":"2024-03-06T07:22:42.288618+0000","last_active":"2024-03-06T07:22:42.289232+0000","last_peered":"2024-03-06T07:22:42.289232+0000","last_clean":"2024-03-06T07:22:42.289232+0000","last_became_active":"2024-03-06T07:22:42.287863+0000","last_became_peered":"2024-03-06T07:22:42.287863+0000","last_unstale":"2024-03-06T07:22:42.289232+0000","last_undegraded":"2024-03-06T07:22:42.289232+0000","last_fullsized":"2024-03-06T07:22:42.289232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:06.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278812+0000","last_change":"2024-03-06T07:22:42.278812+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278812+0000","last_undegraded":"2024-03-06T07:22:42.278812+0000","last_fullsized":"2024-03-06T07:22:42.278812+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298112+0000","last_change":"2024-03-06T07:22:42.293801+0000","last_active":"2024-03-06T07:22:42.298112+0000","last_peered":"2024-03-06T07:22:42.298112+0000","last_clean":"2024-03-06T07:22:42.298112+0000","last_became_active":"2024-03-06T07:22:42.293537+0000","last_became_peered":"2024-03-06T07:22:42.293537+0000","last_unstale":"2024-03-06T07:22:42.298112+0000","last_undegraded":"2024-03-06T07:22:42.298112+0000","last_fullsized":"2024-03-06T07:22:42.298112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:05.097698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283044+0000","last_change":"2024-03-06T07:22:42.283044+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283044+0000","last_undegraded":"2024-03-06T07:22:42.283044+0000","last_fullsized":"2024-03-06T07:22:42.283044+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289637+0000","last_change":"2024-03-06T07:22:42.287323+0000","last_active":"2024-03-06T07:22:42.289637+0000","last_peered":"2024-03-06T07:22:42.289637+0000","last_clean":"2024-03-06T07:22:42.289637+0000","last_became_active":"2024-03-06T07:22:42.287077+0000","last_became_peered":"2024-03-06T07:22:42.287077+0000","last_unstale":"2024-03-06T07:22:42.289637+0000","last_undegraded":"2024-03-06T07:22:42.289637+0000","last_fullsized":"2024-03-06T07:22:42.289637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:24.927357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278746+0000","last_change":"2024-03-06T07:22:42.278746+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278746+0000","last_undegraded":"2024-03-06T07:22:42.278746+0000","last_fullsized":"2024-03-06T07:22:42.278746+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291008+0000","last_change":"2024-03-06T07:22:42.290240+0000","last_active":"2024-03-06T07:22:42.291008+0000","last_peered":"2024-03-06T07:22:42.291008+0000","last_clean":"2024-03-06T07:22:42.291008+0000","last_became_active":"2024-03-06T07:22:42.289934+0000","last_became_peered":"2024-03-06T07:22:42.289934+0000","last_unstale":"2024-03-06T07:22:42.291008+0000","last_undegraded":"2024-03-06T07:22:42.291008+0000","last_fullsized":"2024-03-06T07:22:42.291008+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:58.470531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282416+0000","last_change":"2024-03-06T07:22:42.282416+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282416+0000","last_undegraded":"2024-03-06T07:22:42.282416+0000","last_fullsized":"2024-03-06T07:22:42.282416+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.293325+0000","last_change":"2024-03-06T07:22:42.292354+0000","last_active":"2024-03-06T07:22:42.293325+0000","last_peered":"2024-03-06T07:22:42.293325+0000","last_clean":"2024-03-06T07:22:42.293325+0000","last_became_active":"2024-03-06T07:22:42.292033+0000","last_became_peered":"2024-03-06T07:22:42.292033+0000","last_unstale":"2024-03-06T07:22:42.293325+0000","last_undegraded":"2024-03-06T07:22:42.293325+0000","last_fullsized":"2024-03-06T07:22:42.293325+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:31.588280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.289810+0000","last_change":"2024-03-06T07:22:42.289810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.289810+0000","last_undegraded":"2024-03-06T07:22:42.289810+0000","last_fullsized":"2024-03-06T07:22:42.289810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291470+0000","last_change":"2024-03-06T07:22:42.288768+0000","last_active":"2024-03-06T07:22:42.291470+0000","last_peered":"2024-03-06T07:22:42.291470+0000","last_clean":"2024-03-06T07:22:42.291470+0000","last_became_active":"2024-03-06T07:22:42.288476+0000","last_became_peered":"2024-03-06T07:22:42.288476+0000","last_unstale":"2024-03-06T07:22:42.291470+0000","last_undegraded":"2024-03-06T07:22:42.291470+0000","last_fullsized":"2024-03-06T07:22:42.291470+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.908267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281207+0000","last_change":"2024-03-06T07:22:42.281207+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281207+0000","last_undegraded":"2024-03-06T07:22:42.281207+0000","last_fullsized":"2024-03-06T07:22:42.281207+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294770+0000","last_change":"2024-03-06T07:22:42.293943+0000","last_active":"2024-03-06T07:22:42.294770+0000","last_peered":"2024-03-06T07:22:42.294770+0000","last_clean":"2024-03-06T07:22:42.294770+0000","last_became_active":"2024-03-06T07:22:42.293661+0000","last_became_peered":"2024-03-06T07:22:42.293661+0000","last_unstale":"2024-03-06T07:22:42.294770+0000","last_undegraded":"2024-03-06T07:22:42.294770+0000","last_fullsized":"2024-03-06T07:22:42.294770+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:51.825409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280504+0000","last_change":"2024-03-06T07:22:42.280504+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280504+0000","last_undegraded":"2024-03-06T07:22:42.280504+0000","last_fullsized":"2024-03-06T07:22:42.280504+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291595+0000","last_change":"2024-03-06T07:22:42.287760+0000","last_active":"2024-03-06T07:22:42.291595+0000","last_peered":"2024-03-06T07:22:42.291595+0000","last_clean":"2024-03-06T07:22:42.291595+0000","last_became_active":"2024-03-06T07:22:42.287466+0000","last_became_peered":"2024-03-06T07:22:42.287466+0000","last_unstale":"2024-03-06T07:22:42.291595+0000","last_undegraded":"2024-03-06T07:22:42.291595+0000","last_fullsized":"2024-03-06T07:22:42.291595+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:48.429074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280985+0000","last_change":"2024-03-06T07:22:42.280985+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280985+0000","last_undegraded":"2024-03-06T07:22:42.280985+0000","last_fullsized":"2024-03-06T07:22:42.280985+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298342+0000","last_change":"2024-03-06T07:22:42.296157+0000","last_active":"2024-03-06T07:22:42.298342+0000","last_peered":"2024-03-06T07:22:42.298342+0000","last_clean":"2024-03-06T07:22:42.298342+0000","last_became_active":"2024-03-06T07:22:42.295713+0000","last_became_peered":"2024-03-06T07:22:42.295713+0000","last_unstale":"2024-03-06T07:22:42.298342+0000","last_undegraded":"2024-03-06T07:22:42.298342+0000","last_fullsized":"2024-03-06T07:22:42.298342+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:11.158468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281379+0000","last_change":"2024-03-06T07:22:42.281379+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281379+0000","last_undegraded":"2024-03-06T07:22:42.281379+0000","last_fullsized":"2024-03-06T07:22:42.281379+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294125+0000","last_change":"2024-03-06T07:22:42.293654+0000","last_active":"2024-03-06T07:22:42.294125+0000","last_peered":"2024-03-06T07:22:42.294125+0000","last_clean":"2024-03-06T07:22:42.294125+0000","last_became_active":"2024-03-06T07:22:42.293361+0000","last_became_peered":"2024-03-06T07:22:42.293361+0000","last_unstale":"2024-03-06T07:22:42.294125+0000","last_undegraded":"2024-03-06T07:22:42.294125+0000","last_fullsized":"2024-03-06T07:22:42.294125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:03.485795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290349+0000","last_change":"2024-03-06T07:22:42.290349+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290349+0000","last_undegraded":"2024-03-06T07:22:42.290349+0000","last_fullsized":"2024-03-06T07:22:42.290349+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291039+0000","last_change":"2024-03-06T07:22:42.290440+0000","last_active":"2024-03-06T07:22:42.291039+0000","last_peered":"2024-03-06T07:22:42.291039+0000","last_clean":"2024-03-06T07:22:42.291039+0000","last_became_active":"2024-03-06T07:22:42.290162+0000","last_became_peered":"2024-03-06T07:22:42.290162+0000","last_unstale":"2024-03-06T07:22:42.291039+0000","last_undegraded":"2024-03-06T07:22:42.291039+0000","last_fullsized":"2024-03-06T07:22:42.291039+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:02.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283399+0000","last_change":"2024-03-06T07:22:42.283399+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283399+0000","last_undegraded":"2024-03-06T07:22:42.283399+0000","last_fullsized":"2024-03-06T07:22:42.283399+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298125+0000","last_change":"2024-03-06T07:22:42.293720+0000","last_active":"2024-03-06T07:22:42.298125+0000","last_peered":"2024-03-06T07:22:42.298125+0000","last_clean":"2024-03-06T07:22:42.298125+0000","last_became_active":"2024-03-06T07:22:42.293429+0000","last_became_peered":"2024-03-06T07:22:42.293429+0000","last_unstale":"2024-03-06T07:22:42.298125+0000","last_undegraded":"2024-03-06T07:22:42.298125+0000","last_fullsized":"2024-03-06T07:22:42.298125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:55.471196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":357,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291293+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:22:42.291293+0000","last_peered":"2024-03-06T07:22:42.291293+0000","last_clean":"2024-03-06T07:22:42.291293+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:22:42.291293+0000","last_undegraded":"2024-03-06T07:22:42.291293+0000","last_fullsized":"2024-03-06T07:22:42.291293+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1490944,"data_stored":715572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:46.578 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:46.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.714+0000 7fd262629640 1 Processor -- start 2024-03-06T07:22:46.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.714+0000 7fd262629640 1 -- start start 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.714+0000 7fd262629640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.714+0000 7fd262629640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd25c0c1aa0 0x7fd25c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.714+0000 7fd262629640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25c0c28d0 con 0x7fd25c0c5c60 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.714+0000 7fd262629640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25c0c2680 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.714+0000 7fd262629640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c0c2bb0 con 0x7fd25c14f790 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd25c0c1aa0 0x7fd25c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd25c0c1aa0 0x7fd25c0c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36296/0 (socket says 172.21.15.94:36296) 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25b7fe640 1 -- 172.21.15.94:0/3498086785 learned_addr learned my addr 172.21.15.94:0/3498086785 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25bfff640 1 --2- 172.21.15.94:0/3498086785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd260bc6640 1 --1- 172.21.15.94:0/3498086785 >> v1:172.21.15.145:6789/0 conn(0x7fd25c0c5c60 0x7fd25c155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:59758/0 (socket says 172.21.15.94:59758) 2024-03-06T07:22:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25bfff640 1 -- 172.21.15.94:0/3498086785 >> v2:172.21.15.94:3300/0 conn(0x7fd25c0c1aa0 msgr2=0x7fd25c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25bfff640 1 --2- 172.21.15.94:0/3498086785 >> v2:172.21.15.94:3300/0 conn(0x7fd25c0c1aa0 0x7fd25c0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25bfff640 1 -- 172.21.15.94:0/3498086785 >> v1:172.21.15.145:6789/0 conn(0x7fd25c0c5c60 legacy=0x7fd25c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25affd640 1 -- 172.21.15.94:0/3498086785 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844182269 0 0) 0x7fd25c0c28d0 con 0x7fd25c0c5c60 2024-03-06T07:22:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25bfff640 1 -- 172.21.15.94:0/3498086785 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c0c2e90 con 0x7fd25c14f790 2024-03-06T07:22:46.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.715+0000 7fd25b7fe640 1 --2- 172.21.15.94:0/3498086785 >> v2:172.21.15.94:3300/0 conn(0x7fd25c0c1aa0 0x7fd25c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd25bfff640 1 --2- 172.21.15.94:0/3498086785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c1529d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0097a0 tx=0x7fd24c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c780a2807e7c331 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd25affd640 1 -- 172.21.15.94:0/3498086785 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c035020 con 0x7fd25c14f790 2024-03-06T07:22:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd25affd640 1 -- 172.21.15.94:0/3498086785 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24c031830 con 0x7fd25c14f790 2024-03-06T07:22:46.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd25affd640 1 -- 172.21.15.94:0/3498086785 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c031c00 con 0x7fd25c14f790 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 -- 172.21.15.94:0/3498086785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 msgr2=0x7fd25c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 --2- 172.21.15.94:0/3498086785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c1529d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0097a0 tx=0x7fd24c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 -- 172.21.15.94:0/3498086785 shutdown_connections 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 --2- 172.21.15.94:0/3498086785 >> v2:172.21.15.94:3300/0 conn(0x7fd25c0c1aa0 0x7fd25c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 --2- 172.21.15.94:0/3498086785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c1529d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 -- 172.21.15.94:0/3498086785 >> 172.21.15.94:0/3498086785 conn(0x7fd25c0c6710 msgr2=0x7fd25c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 -- 172.21.15.94:0/3498086785 shutdown_connections 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.716+0000 7fd262629640 1 -- 172.21.15.94:0/3498086785 wait complete. 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.717+0000 7fd262629640 1 Processor -- start 2024-03-06T07:22:46.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd262629640 1 -- start start 2024-03-06T07:22:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd262629640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd25c0c1aa0 0x7fd25c154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd262629640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25c0c5c60 0x7fd25c1e99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd262629640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd262629640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c06cc00 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd262629640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c0c0d60 con 0x7fd25c0c5c60 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd262629640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c06cd80 con 0x7fd25c14f790 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd25bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd25c0c1aa0 0x7fd25c154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd25bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd25c0c1aa0 0x7fd25c154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36310/0 (socket says 172.21.15.94:36310) 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd25bfff640 1 -- 172.21.15.94:0/2103024209 learned_addr learned my addr 172.21.15.94:0/2103024209 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd25b7fe640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25c0c5c60 0x7fd25c1e99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.718+0000 7fd260bc6640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd25bfff640 1 -- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 msgr2=0x7fd25c153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd25bfff640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd25bfff640 1 -- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25c0c5c60 msgr2=0x7fd25c1e99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd25bfff640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25c0c5c60 0x7fd25c1e99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd25bfff640 1 -- 172.21.15.94:0/2103024209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c153ef0 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd25bfff640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd25c0c1aa0 0x7fd25c154fb0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd24c034040 tx=0x7fd24c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd258ff9640 1 -- 172.21.15.94:0/2103024209 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c03b980 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd258ff9640 1 -- 172.21.15.94:0/2103024209 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24c03bb40 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd258ff9640 1 -- 172.21.15.94:0/2103024209 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c0453f0 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25c070470 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.719+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25c1e6ef0 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.720+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25c04f330 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.720+0000 7fd258ff9640 1 -- 172.21.15.94:0/2103024209 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd24c04c060 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.721+0000 7fd258ff9640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.721+0000 7fd258ff9640 1 -- 172.21.15.94:0/2103024209 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd24c0abcf0 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.721+0000 7fd25b7fe640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.721+0000 7fd25b7fe640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd24806b4d0 0x7fd24806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd25c1554f0 tx=0x7fd250009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.725+0000 7fd258ff9640 1 -- 172.21.15.94:0/2103024209 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd24c040070 con 0x7fd25c0c1aa0 2024-03-06T07:22:46.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.894+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd25c0c1f40 con 0x7fd24806b4d0 2024-03-06T07:22:46.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.900+0000 7fd258ff9640 1 -- 172.21.15.94:0/2103024209 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247154 (secure 0 0 0) 0x7fd25c0c1f40 con 0x7fd24806b4d0 2024-03-06T07:22:46.904 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:46.905 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:22:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd24806b4d0 msgr2=0x7fd24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd24806b4d0 0x7fd24806d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd25c1554f0 tx=0x7fd250009240 comp rx=0 tx=0).stop 2024-03-06T07:22:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd25c0c1aa0 msgr2=0x7fd25c154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd25c0c1aa0 0x7fd25c154fb0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd24c034040 tx=0x7fd24c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 shutdown_connections 2024-03-06T07:22:46.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd24806b4d0 0x7fd24806d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd25c14f790 0x7fd25c153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25c0c5c60 0x7fd25c1e99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 --2- 172.21.15.94:0/2103024209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd25c0c1aa0 0x7fd25c154fb0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 >> 172.21.15.94:0/2103024209 conn(0x7fd25c0c6710 msgr2=0x7fd25c0c42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 shutdown_connections 2024-03-06T07:22:46.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:46.903+0000 7fd262629640 1 -- 172.21.15.94:0/2103024209 wait complete. 2024-03-06T07:22:46.916 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-03-06T07:22:46.693446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261148,"kb_used_data":11692,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713572,"statfs":{"total":773094113280,"available":772826697728,"internally_reserved":0,"allocated":11972608,"data_stored":6858699,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.427010"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279032+0000","last_change":"2024-03-06T07:22:42.279032+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279032+0000","last_undegraded":"2024-03-06T07:22:42.279032+0000","last_fullsized":"2024-03-06T07:22:42.279032+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284203+0000","last_change":"2024-03-06T07:22:42.284203+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284203+0000","last_undegraded":"2024-03-06T07:22:42.284203+0000","last_fullsized":"2024-03-06T07:22:42.284203+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284568+0000","last_change":"2024-03-06T07:22:42.284568+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284568+0000","last_undegraded":"2024-03-06T07:22:42.284568+0000","last_fullsized":"2024-03-06T07:22:42.284568+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279457+0000","last_change":"2024-03-06T07:22:42.279457+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279457+0000","last_undegraded":"2024-03-06T07:22:42.279457+0000","last_fullsized":"2024-03-06T07:22:42.279457+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283015+0000","last_change":"2024-03-06T07:22:42.283015+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283015+0000","last_undegraded":"2024-03-06T07:22:42.283015+0000","last_fullsized":"2024-03-06T07:22:42.283015+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292601+0000","last_change":"2024-03-06T07:22:42.292601+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292601+0000","last_undegraded":"2024-03-06T07:22:42.292601+0000","last_fullsized":"2024-03-06T07:22:42.292601+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.285418+0000","last_change":"2024-03-06T07:22:42.285418+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.285418+0000","last_undegraded":"2024-03-06T07:22:42.285418+0000","last_fullsized":"2024-03-06T07:22:42.285418+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282898+0000","last_change":"2024-03-06T07:22:42.282898+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282898+0000","last_undegraded":"2024-03-06T07:22:42.282898+0000","last_fullsized":"2024-03-06T07:22:42.282898+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282437+0000","last_change":"2024-03-06T07:22:42.282437+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282437+0000","last_undegraded":"2024-03-06T07:22:42.282437+0000","last_fullsized":"2024-03-06T07:22:42.282437+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283810+0000","last_change":"2024-03-06T07:22:42.283810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283810+0000","last_undegraded":"2024-03-06T07:22:42.283810+0000","last_fullsized":"2024-03-06T07:22:42.283810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278978+0000","last_change":"2024-03-06T07:22:42.278978+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278978+0000","last_undegraded":"2024-03-06T07:22:42.278978+0000","last_fullsized":"2024-03-06T07:22:42.278978+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282804+0000","last_change":"2024-03-06T07:22:42.282804+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282804+0000","last_undegraded":"2024-03-06T07:22:42.282804+0000","last_fullsized":"2024-03-06T07:22:42.282804+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278897+0000","last_change":"2024-03-06T07:22:42.278897+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278897+0000","last_undegraded":"2024-03-06T07:22:42.278897+0000","last_fullsized":"2024-03-06T07:22:42.278897+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278443+0000","last_change":"2024-03-06T07:22:42.278443+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278443+0000","last_undegraded":"2024-03-06T07:22:42.278443+0000","last_fullsized":"2024-03-06T07:22:42.278443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283228+0000","last_change":"2024-03-06T07:22:42.283228+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283228+0000","last_undegraded":"2024-03-06T07:22:42.283228+0000","last_fullsized":"2024-03-06T07:22:42.283228+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282177+0000","last_change":"2024-03-06T07:22:42.282177+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282177+0000","last_undegraded":"2024-03-06T07:22:42.282177+0000","last_fullsized":"2024-03-06T07:22:42.282177+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279562+0000","last_change":"2024-03-06T07:22:42.279562+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279562+0000","last_undegraded":"2024-03-06T07:22:42.279562+0000","last_fullsized":"2024-03-06T07:22:42.279562+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.293286+0000","last_change":"2024-03-06T07:22:42.293286+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.293286+0000","last_undegraded":"2024-03-06T07:22:42.293286+0000","last_fullsized":"2024-03-06T07:22:42.293286+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280297+0000","last_change":"2024-03-06T07:22:42.280297+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280297+0000","last_undegraded":"2024-03-06T07:22:42.280297+0000","last_fullsized":"2024-03-06T07:22:42.280297+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283843+0000","last_change":"2024-03-06T07:22:42.283843+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283843+0000","last_undegraded":"2024-03-06T07:22:42.283843+0000","last_fullsized":"2024-03-06T07:22:42.283843+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278394+0000","last_change":"2024-03-06T07:22:42.278394+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278394+0000","last_undegraded":"2024-03-06T07:22:42.278394+0000","last_fullsized":"2024-03-06T07:22:42.278394+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284222+0000","last_change":"2024-03-06T07:22:42.284222+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284222+0000","last_undegraded":"2024-03-06T07:22:42.284222+0000","last_fullsized":"2024-03-06T07:22:42.284222+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282160+0000","last_change":"2024-03-06T07:22:42.282160+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282160+0000","last_undegraded":"2024-03-06T07:22:42.282160+0000","last_fullsized":"2024-03-06T07:22:42.282160+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286365+0000","last_change":"2024-03-06T07:22:42.286365+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286365+0000","last_undegraded":"2024-03-06T07:22:42.286365+0000","last_fullsized":"2024-03-06T07:22:42.286365+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286778+0000","last_change":"2024-03-06T07:22:42.286778+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286778+0000","last_undegraded":"2024-03-06T07:22:42.286778+0000","last_fullsized":"2024-03-06T07:22:42.286778+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292889+0000","last_change":"2024-03-06T07:22:42.292889+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292889+0000","last_undegraded":"2024-03-06T07:22:42.292889+0000","last_fullsized":"2024-03-06T07:22:42.292889+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282008+0000","last_change":"2024-03-06T07:22:42.282008+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282008+0000","last_undegraded":"2024-03-06T07:22:42.282008+0000","last_fullsized":"2024-03-06T07:22:42.282008+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286216+0000","last_change":"2024-03-06T07:22:42.286216+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286216+0000","last_undegraded":"2024-03-06T07:22:42.286216+0000","last_fullsized":"2024-03-06T07:22:42.286216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281552+0000","last_change":"2024-03-06T07:22:42.281552+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281552+0000","last_undegraded":"2024-03-06T07:22:42.281552+0000","last_fullsized":"2024-03-06T07:22:42.281552+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281197+0000","last_change":"2024-03-06T07:22:42.281197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281197+0000","last_undegraded":"2024-03-06T07:22:42.281197+0000","last_fullsized":"2024-03-06T07:22:42.281197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281292+0000","last_change":"2024-03-06T07:22:42.281292+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281292+0000","last_undegraded":"2024-03-06T07:22:42.281292+0000","last_fullsized":"2024-03-06T07:22:42.281292+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289168+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288869+0000","last_became_peered":"2024-03-06T07:22:42.288869+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:22.433887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277947+0000","last_change":"2024-03-06T07:22:42.277947+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277947+0000","last_undegraded":"2024-03-06T07:22:42.277947+0000","last_fullsized":"2024-03-06T07:22:42.277947+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.284819+0000","last_change":"2024-03-06T07:22:42.284230+0000","last_active":"2024-03-06T07:22:42.284819+0000","last_peered":"2024-03-06T07:22:42.284819+0000","last_clean":"2024-03-06T07:22:42.284819+0000","last_became_active":"2024-03-06T07:22:42.283796+0000","last_became_peered":"2024-03-06T07:22:42.283796+0000","last_unstale":"2024-03-06T07:22:42.284819+0000","last_undegraded":"2024-03-06T07:22:42.284819+0000","last_fullsized":"2024-03-06T07:22:42.284819+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:28.750113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283761+0000","last_change":"2024-03-06T07:22:42.283761+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283761+0000","last_undegraded":"2024-03-06T07:22:42.283761+0000","last_fullsized":"2024-03-06T07:22:42.283761+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299956+0000","last_change":"2024-03-06T07:22:42.298123+0000","last_active":"2024-03-06T07:22:42.299956+0000","last_peered":"2024-03-06T07:22:42.299956+0000","last_clean":"2024-03-06T07:22:42.299956+0000","last_became_active":"2024-03-06T07:22:42.297656+0000","last_became_peered":"2024-03-06T07:22:42.297656+0000","last_unstale":"2024-03-06T07:22:42.299956+0000","last_undegraded":"2024-03-06T07:22:42.299956+0000","last_fullsized":"2024-03-06T07:22:42.299956+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:56.531364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277964+0000","last_change":"2024-03-06T07:22:42.277964+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277964+0000","last_undegraded":"2024-03-06T07:22:42.277964+0000","last_fullsized":"2024-03-06T07:22:42.277964+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298336+0000","last_change":"2024-03-06T07:22:42.295819+0000","last_active":"2024-03-06T07:22:42.298336+0000","last_peered":"2024-03-06T07:22:42.298336+0000","last_clean":"2024-03-06T07:22:42.298336+0000","last_became_active":"2024-03-06T07:22:42.295541+0000","last_became_peered":"2024-03-06T07:22:42.295541+0000","last_unstale":"2024-03-06T07:22:42.298336+0000","last_undegraded":"2024-03-06T07:22:42.298336+0000","last_fullsized":"2024-03-06T07:22:42.298336+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:51.235851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282847+0000","last_change":"2024-03-06T07:22:42.282847+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282847+0000","last_undegraded":"2024-03-06T07:22:42.282847+0000","last_fullsized":"2024-03-06T07:22:42.282847+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289767+0000","last_change":"2024-03-06T07:22:42.287120+0000","last_active":"2024-03-06T07:22:42.289767+0000","last_peered":"2024-03-06T07:22:42.289767+0000","last_clean":"2024-03-06T07:22:42.289767+0000","last_became_active":"2024-03-06T07:22:42.286844+0000","last_became_peered":"2024-03-06T07:22:42.286844+0000","last_unstale":"2024-03-06T07:22:42.289767+0000","last_undegraded":"2024-03-06T07:22:42.289767+0000","last_fullsized":"2024-03-06T07:22:42.289767+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.165551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283519+0000","last_change":"2024-03-06T07:22:42.283519+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283519+0000","last_undegraded":"2024-03-06T07:22:42.283519+0000","last_fullsized":"2024-03-06T07:22:42.283519+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.290193+0000","last_change":"2024-03-06T07:22:42.289388+0000","last_active":"2024-03-06T07:22:42.290193+0000","last_peered":"2024-03-06T07:22:42.290193+0000","last_clean":"2024-03-06T07:22:42.290193+0000","last_became_active":"2024-03-06T07:22:42.288148+0000","last_became_peered":"2024-03-06T07:22:42.288148+0000","last_unstale":"2024-03-06T07:22:42.290193+0000","last_undegraded":"2024-03-06T07:22:42.290193+0000","last_fullsized":"2024-03-06T07:22:42.290193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:37.212217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279197+0000","last_change":"2024-03-06T07:22:42.279197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279197+0000","last_undegraded":"2024-03-06T07:22:42.279197+0000","last_fullsized":"2024-03-06T07:22:42.279197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.287719+0000","last_change":"2024-03-06T07:22:42.285183+0000","last_active":"2024-03-06T07:22:42.287719+0000","last_peered":"2024-03-06T07:22:42.287719+0000","last_clean":"2024-03-06T07:22:42.287719+0000","last_became_active":"2024-03-06T07:22:42.284924+0000","last_became_peered":"2024-03-06T07:22:42.284924+0000","last_unstale":"2024-03-06T07:22:42.287719+0000","last_undegraded":"2024-03-06T07:22:42.287719+0000","last_fullsized":"2024-03-06T07:22:42.287719+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:21.527693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292509+0000","last_change":"2024-03-06T07:22:42.292509+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292509+0000","last_undegraded":"2024-03-06T07:22:42.292509+0000","last_fullsized":"2024-03-06T07:22:42.292509+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296297+0000","last_change":"2024-03-06T07:22:42.289621+0000","last_active":"2024-03-06T07:22:42.296297+0000","last_peered":"2024-03-06T07:22:42.296297+0000","last_clean":"2024-03-06T07:22:42.296297+0000","last_became_active":"2024-03-06T07:22:42.289239+0000","last_became_peered":"2024-03-06T07:22:42.289239+0000","last_unstale":"2024-03-06T07:22:42.296297+0000","last_undegraded":"2024-03-06T07:22:42.296297+0000","last_fullsized":"2024-03-06T07:22:42.296297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:40.681976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277967+0000","last_change":"2024-03-06T07:22:42.277967+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277967+0000","last_undegraded":"2024-03-06T07:22:42.277967+0000","last_fullsized":"2024-03-06T07:22:42.277967+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291813+0000","last_change":"2024-03-06T07:22:42.287554+0000","last_active":"2024-03-06T07:22:42.291813+0000","last_peered":"2024-03-06T07:22:42.291813+0000","last_clean":"2024-03-06T07:22:42.291813+0000","last_became_active":"2024-03-06T07:22:42.285213+0000","last_became_peered":"2024-03-06T07:22:42.285213+0000","last_unstale":"2024-03-06T07:22:42.291813+0000","last_undegraded":"2024-03-06T07:22:42.291813+0000","last_fullsized":"2024-03-06T07:22:42.291813+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:07.580402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281750+0000","last_change":"2024-03-06T07:22:42.281750+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281750+0000","last_undegraded":"2024-03-06T07:22:42.281750+0000","last_fullsized":"2024-03-06T07:22:42.281750+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288109+0000","last_change":"2024-03-06T07:22:42.286796+0000","last_active":"2024-03-06T07:22:42.288109+0000","last_peered":"2024-03-06T07:22:42.288109+0000","last_clean":"2024-03-06T07:22:42.288109+0000","last_became_active":"2024-03-06T07:22:42.286558+0000","last_became_peered":"2024-03-06T07:22:42.286558+0000","last_unstale":"2024-03-06T07:22:42.288109+0000","last_undegraded":"2024-03-06T07:22:42.288109+0000","last_fullsized":"2024-03-06T07:22:42.288109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:11.589911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281997+0000","last_change":"2024-03-06T07:22:42.281997+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281997+0000","last_undegraded":"2024-03-06T07:22:42.281997+0000","last_fullsized":"2024-03-06T07:22:42.281997+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299969+0000","last_change":"2024-03-06T07:22:42.298748+0000","last_active":"2024-03-06T07:22:42.299969+0000","last_peered":"2024-03-06T07:22:42.299969+0000","last_clean":"2024-03-06T07:22:42.299969+0000","last_became_active":"2024-03-06T07:22:42.298459+0000","last_became_peered":"2024-03-06T07:22:42.298459+0000","last_unstale":"2024-03-06T07:22:42.299969+0000","last_undegraded":"2024-03-06T07:22:42.299969+0000","last_fullsized":"2024-03-06T07:22:42.299969+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:08.721690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281717+0000","last_change":"2024-03-06T07:22:42.281717+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281717+0000","last_undegraded":"2024-03-06T07:22:42.281717+0000","last_fullsized":"2024-03-06T07:22:42.281717+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299706+0000","last_change":"2024-03-06T07:22:42.297582+0000","last_active":"2024-03-06T07:22:42.299706+0000","last_peered":"2024-03-06T07:22:42.299706+0000","last_clean":"2024-03-06T07:22:42.299706+0000","last_became_active":"2024-03-06T07:22:42.297309+0000","last_became_peered":"2024-03-06T07:22:42.297309+0000","last_unstale":"2024-03-06T07:22:42.299706+0000","last_undegraded":"2024-03-06T07:22:42.299706+0000","last_fullsized":"2024-03-06T07:22:42.299706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:00.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281667+0000","last_change":"2024-03-06T07:22:42.281667+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281667+0000","last_undegraded":"2024-03-06T07:22:42.281667+0000","last_fullsized":"2024-03-06T07:22:42.281667+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288179+0000","last_change":"2024-03-06T07:22:42.287402+0000","last_active":"2024-03-06T07:22:42.288179+0000","last_peered":"2024-03-06T07:22:42.288179+0000","last_clean":"2024-03-06T07:22:42.288179+0000","last_became_active":"2024-03-06T07:22:42.287169+0000","last_became_peered":"2024-03-06T07:22:42.287169+0000","last_unstale":"2024-03-06T07:22:42.288179+0000","last_undegraded":"2024-03-06T07:22:42.288179+0000","last_fullsized":"2024-03-06T07:22:42.288179+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:39.487955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290484+0000","last_change":"2024-03-06T07:22:42.290484+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290484+0000","last_undegraded":"2024-03-06T07:22:42.290484+0000","last_fullsized":"2024-03-06T07:22:42.290484+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296242+0000","last_change":"2024-03-06T07:22:42.289384+0000","last_active":"2024-03-06T07:22:42.296242+0000","last_peered":"2024-03-06T07:22:42.296242+0000","last_clean":"2024-03-06T07:22:42.296242+0000","last_became_active":"2024-03-06T07:22:42.289045+0000","last_became_peered":"2024-03-06T07:22:42.289045+0000","last_unstale":"2024-03-06T07:22:42.296242+0000","last_undegraded":"2024-03-06T07:22:42.296242+0000","last_fullsized":"2024-03-06T07:22:42.296242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:08.837325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280053+0000","last_change":"2024-03-06T07:22:42.280053+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280053+0000","last_undegraded":"2024-03-06T07:22:42.280053+0000","last_fullsized":"2024-03-06T07:22:42.280053+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.295035+0000","last_change":"2024-03-06T07:22:42.294517+0000","last_active":"2024-03-06T07:22:42.295035+0000","last_peered":"2024-03-06T07:22:42.295035+0000","last_clean":"2024-03-06T07:22:42.295035+0000","last_became_active":"2024-03-06T07:22:42.294249+0000","last_became_peered":"2024-03-06T07:22:42.294249+0000","last_unstale":"2024-03-06T07:22:42.295035+0000","last_undegraded":"2024-03-06T07:22:42.295035+0000","last_fullsized":"2024-03-06T07:22:42.295035+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:12.295033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284708+0000","last_change":"2024-03-06T07:22:42.284708+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284708+0000","last_undegraded":"2024-03-06T07:22:42.284708+0000","last_fullsized":"2024-03-06T07:22:42.284708+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291835+0000","last_change":"2024-03-06T07:22:42.288271+0000","last_active":"2024-03-06T07:22:42.291835+0000","last_peered":"2024-03-06T07:22:42.291835+0000","last_clean":"2024-03-06T07:22:42.291835+0000","last_became_active":"2024-03-06T07:22:42.287981+0000","last_became_peered":"2024-03-06T07:22:42.287981+0000","last_unstale":"2024-03-06T07:22:42.291835+0000","last_undegraded":"2024-03-06T07:22:42.291835+0000","last_fullsized":"2024-03-06T07:22:42.291835+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.744017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286139+0000","last_change":"2024-03-06T07:22:42.286139+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286139+0000","last_undegraded":"2024-03-06T07:22:42.286139+0000","last_fullsized":"2024-03-06T07:22:42.286139+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289825+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288884+0000","last_became_peered":"2024-03-06T07:22:42.288884+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:05.920275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290881+0000","last_change":"2024-03-06T07:22:42.290881+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290881+0000","last_undegraded":"2024-03-06T07:22:42.290881+0000","last_fullsized":"2024-03-06T07:22:42.290881+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289606+0000","last_change":"2024-03-06T07:22:42.286639+0000","last_active":"2024-03-06T07:22:42.289606+0000","last_peered":"2024-03-06T07:22:42.289606+0000","last_clean":"2024-03-06T07:22:42.289606+0000","last_became_active":"2024-03-06T07:22:42.286392+0000","last_became_peered":"2024-03-06T07:22:42.286392+0000","last_unstale":"2024-03-06T07:22:42.289606+0000","last_undegraded":"2024-03-06T07:22:42.289606+0000","last_fullsized":"2024-03-06T07:22:42.289606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:40.885949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281607+0000","last_change":"2024-03-06T07:22:42.281607+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281607+0000","last_undegraded":"2024-03-06T07:22:42.281607+0000","last_fullsized":"2024-03-06T07:22:42.281607+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.285286+0000","last_change":"2024-03-06T07:22:42.284042+0000","last_active":"2024-03-06T07:22:42.285286+0000","last_peered":"2024-03-06T07:22:42.285286+0000","last_clean":"2024-03-06T07:22:42.285286+0000","last_became_active":"2024-03-06T07:22:42.283781+0000","last_became_peered":"2024-03-06T07:22:42.283781+0000","last_unstale":"2024-03-06T07:22:42.285286+0000","last_undegraded":"2024-03-06T07:22:42.285286+0000","last_fullsized":"2024-03-06T07:22:42.285286+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.123546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279871+0000","last_change":"2024-03-06T07:22:42.279871+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279871+0000","last_undegraded":"2024-03-06T07:22:42.279871+0000","last_fullsized":"2024-03-06T07:22:42.279871+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298230+0000","last_change":"2024-03-06T07:22:42.297131+0000","last_active":"2024-03-06T07:22:42.298230+0000","last_peered":"2024-03-06T07:22:42.298230+0000","last_clean":"2024-03-06T07:22:42.298230+0000","last_became_active":"2024-03-06T07:22:42.296815+0000","last_became_peered":"2024-03-06T07:22:42.296815+0000","last_unstale":"2024-03-06T07:22:42.298230+0000","last_undegraded":"2024-03-06T07:22:42.298230+0000","last_fullsized":"2024-03-06T07:22:42.298230+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:56.374398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277869+0000","last_change":"2024-03-06T07:22:42.277869+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277869+0000","last_undegraded":"2024-03-06T07:22:42.277869+0000","last_fullsized":"2024-03-06T07:22:42.277869+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289232+0000","last_change":"2024-03-06T07:22:42.288618+0000","last_active":"2024-03-06T07:22:42.289232+0000","last_peered":"2024-03-06T07:22:42.289232+0000","last_clean":"2024-03-06T07:22:42.289232+0000","last_became_active":"2024-03-06T07:22:42.287863+0000","last_became_peered":"2024-03-06T07:22:42.287863+0000","last_unstale":"2024-03-06T07:22:42.289232+0000","last_undegraded":"2024-03-06T07:22:42.289232+0000","last_fullsized":"2024-03-06T07:22:42.289232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:06.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278812+0000","last_change":"2024-03-06T07:22:42.278812+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278812+0000","last_undegraded":"2024-03-06T07:22:42.278812+0000","last_fullsized":"2024-03-06T07:22:42.278812+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298112+0000","last_change":"2024-03-06T07:22:42.293801+0000","last_active":"2024-03-06T07:22:42.298112+0000","last_peered":"2024-03-06T07:22:42.298112+0000","last_clean":"2024-03-06T07:22:42.298112+0000","last_became_active":"2024-03-06T07:22:42.293537+0000","last_became_peered":"2024-03-06T07:22:42.293537+0000","last_unstale":"2024-03-06T07:22:42.298112+0000","last_undegraded":"2024-03-06T07:22:42.298112+0000","last_fullsized":"2024-03-06T07:22:42.298112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:05.097698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283044+0000","last_change":"2024-03-06T07:22:42.283044+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283044+0000","last_undegraded":"2024-03-06T07:22:42.283044+0000","last_fullsized":"2024-03-06T07:22:42.283044+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289637+0000","last_change":"2024-03-06T07:22:42.287323+0000","last_active":"2024-03-06T07:22:42.289637+0000","last_peered":"2024-03-06T07:22:42.289637+0000","last_clean":"2024-03-06T07:22:42.289637+0000","last_became_active":"2024-03-06T07:22:42.287077+0000","last_became_peered":"2024-03-06T07:22:42.287077+0000","last_unstale":"2024-03-06T07:22:42.289637+0000","last_undegraded":"2024-03-06T07:22:42.289637+0000","last_fullsized":"2024-03-06T07:22:42.289637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:24.927357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278746+0000","last_change":"2024-03-06T07:22:42.278746+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278746+0000","last_undegraded":"2024-03-06T07:22:42.278746+0000","last_fullsized":"2024-03-06T07:22:42.278746+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291008+0000","last_change":"2024-03-06T07:22:42.290240+0000","last_active":"2024-03-06T07:22:42.291008+0000","last_peered":"2024-03-06T07:22:42.291008+0000","last_clean":"2024-03-06T07:22:42.291008+0000","last_became_active":"2024-03-06T07:22:42.289934+0000","last_became_peered":"2024-03-06T07:22:42.289934+0000","last_unstale":"2024-03-06T07:22:42.291008+0000","last_undegraded":"2024-03-06T07:22:42.291008+0000","last_fullsized":"2024-03-06T07:22:42.291008+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:58.470531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282416+0000","last_change":"2024-03-06T07:22:42.282416+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282416+0000","last_undegraded":"2024-03-06T07:22:42.282416+0000","last_fullsized":"2024-03-06T07:22:42.282416+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.293325+0000","last_change":"2024-03-06T07:22:42.292354+0000","last_active":"2024-03-06T07:22:42.293325+0000","last_peered":"2024-03-06T07:22:42.293325+0000","last_clean":"2024-03-06T07:22:42.293325+0000","last_became_active":"2024-03-06T07:22:42.292033+0000","last_became_peered":"2024-03-06T07:22:42.292033+0000","last_unstale":"2024-03-06T07:22:42.293325+0000","last_undegraded":"2024-03-06T07:22:42.293325+0000","last_fullsized":"2024-03-06T07:22:42.293325+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:31.588280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.289810+0000","last_change":"2024-03-06T07:22:42.289810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.289810+0000","last_undegraded":"2024-03-06T07:22:42.289810+0000","last_fullsized":"2024-03-06T07:22:42.289810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291470+0000","last_change":"2024-03-06T07:22:42.288768+0000","last_active":"2024-03-06T07:22:42.291470+0000","last_peered":"2024-03-06T07:22:42.291470+0000","last_clean":"2024-03-06T07:22:42.291470+0000","last_became_active":"2024-03-06T07:22:42.288476+0000","last_became_peered":"2024-03-06T07:22:42.288476+0000","last_unstale":"2024-03-06T07:22:42.291470+0000","last_undegraded":"2024-03-06T07:22:42.291470+0000","last_fullsized":"2024-03-06T07:22:42.291470+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.908267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281207+0000","last_change":"2024-03-06T07:22:42.281207+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281207+0000","last_undegraded":"2024-03-06T07:22:42.281207+0000","last_fullsized":"2024-03-06T07:22:42.281207+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294770+0000","last_change":"2024-03-06T07:22:42.293943+0000","last_active":"2024-03-06T07:22:42.294770+0000","last_peered":"2024-03-06T07:22:42.294770+0000","last_clean":"2024-03-06T07:22:42.294770+0000","last_became_active":"2024-03-06T07:22:42.293661+0000","last_became_peered":"2024-03-06T07:22:42.293661+0000","last_unstale":"2024-03-06T07:22:42.294770+0000","last_undegraded":"2024-03-06T07:22:42.294770+0000","last_fullsized":"2024-03-06T07:22:42.294770+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:51.825409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280504+0000","last_change":"2024-03-06T07:22:42.280504+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280504+0000","last_undegraded":"2024-03-06T07:22:42.280504+0000","last_fullsized":"2024-03-06T07:22:42.280504+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291595+0000","last_change":"2024-03-06T07:22:42.287760+0000","last_active":"2024-03-06T07:22:42.291595+0000","last_peered":"2024-03-06T07:22:42.291595+0000","last_clean":"2024-03-06T07:22:42.291595+0000","last_became_active":"2024-03-06T07:22:42.287466+0000","last_became_peered":"2024-03-06T07:22:42.287466+0000","last_unstale":"2024-03-06T07:22:42.291595+0000","last_undegraded":"2024-03-06T07:22:42.291595+0000","last_fullsized":"2024-03-06T07:22:42.291595+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:48.429074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280985+0000","last_change":"2024-03-06T07:22:42.280985+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280985+0000","last_undegraded":"2024-03-06T07:22:42.280985+0000","last_fullsized":"2024-03-06T07:22:42.280985+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298342+0000","last_change":"2024-03-06T07:22:42.296157+0000","last_active":"2024-03-06T07:22:42.298342+0000","last_peered":"2024-03-06T07:22:42.298342+0000","last_clean":"2024-03-06T07:22:42.298342+0000","last_became_active":"2024-03-06T07:22:42.295713+0000","last_became_peered":"2024-03-06T07:22:42.295713+0000","last_unstale":"2024-03-06T07:22:42.298342+0000","last_undegraded":"2024-03-06T07:22:42.298342+0000","last_fullsized":"2024-03-06T07:22:42.298342+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:11.158468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281379+0000","last_change":"2024-03-06T07:22:42.281379+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281379+0000","last_undegraded":"2024-03-06T07:22:42.281379+0000","last_fullsized":"2024-03-06T07:22:42.281379+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294125+0000","last_change":"2024-03-06T07:22:42.293654+0000","last_active":"2024-03-06T07:22:42.294125+0000","last_peered":"2024-03-06T07:22:42.294125+0000","last_clean":"2024-03-06T07:22:42.294125+0000","last_became_active":"2024-03-06T07:22:42.293361+0000","last_became_peered":"2024-03-06T07:22:42.293361+0000","last_unstale":"2024-03-06T07:22:42.294125+0000","last_undegraded":"2024-03-06T07:22:42.294125+0000","last_fullsized":"2024-03-06T07:22:42.294125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:03.485795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290349+0000","last_change":"2024-03-06T07:22:42.290349+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290349+0000","last_undegraded":"2024-03-06T07:22:42.290349+0000","last_fullsized":"2024-03-06T07:22:42.290349+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291039+0000","last_change":"2024-03-06T07:22:42.290440+0000","last_active":"2024-03-06T07:22:42.291039+0000","last_peered":"2024-03-06T07:22:42.291039+0000","last_clean":"2024-03-06T07:22:42.291039+0000","last_became_active":"2024-03-06T07:22:42.290162+0000","last_became_peered":"2024-03-06T07:22:42.290162+0000","last_unstale":"2024-03-06T07:22:42.291039+0000","last_undegraded":"2024-03-06T07:22:42.291039+0000","last_fullsized":"2024-03-06T07:22:42.291039+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:02.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283399+0000","last_change":"2024-03-06T07:22:42.283399+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283399+0000","last_undegraded":"2024-03-06T07:22:42.283399+0000","last_fullsized":"2024-03-06T07:22:42.283399+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298125+0000","last_change":"2024-03-06T07:22:42.293720+0000","last_active":"2024-03-06T07:22:42.298125+0000","last_peered":"2024-03-06T07:22:42.298125+0000","last_clean":"2024-03-06T07:22:42.298125+0000","last_became_active":"2024-03-06T07:22:42.293429+0000","last_became_peered":"2024-03-06T07:22:42.293429+0000","last_unstale":"2024-03-06T07:22:42.298125+0000","last_undegraded":"2024-03-06T07:22:42.298125+0000","last_fullsized":"2024-03-06T07:22:42.298125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:55.471196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":357,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291293+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:22:42.291293+0000","last_peered":"2024-03-06T07:22:42.291293+0000","last_clean":"2024-03-06T07:22:42.291293+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:22:42.291293+0000","last_undegraded":"2024-03-06T07:22:42.291293+0000","last_fullsized":"2024-03-06T07:22:42.291293+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1490944,"data_stored":715572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3f is not active+clean 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.279032+0000', 'last_change': '2024-03-06T07:22:42.279032+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.279032+0000', 'last_undegraded': '2024-03-06T07:22:42.279032+0000', 'last_fullsized': '2024-03-06T07:22:42.279032+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3e is not active+clean 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.284203+0000', 'last_change': '2024-03-06T07:22:42.284203+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.284203+0000', 'last_undegraded': '2024-03-06T07:22:42.284203+0000', 'last_fullsized': '2024-03-06T07:22:42.284203+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3d is not active+clean 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.286663+0000', 'last_change': '2024-03-06T07:22:42.286663+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.286663+0000', 'last_undegraded': '2024-03-06T07:22:42.286663+0000', 'last_fullsized': '2024-03-06T07:22:42.286663+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3c is not active+clean 2024-03-06T07:22:46.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.284568+0000', 'last_change': '2024-03-06T07:22:42.284568+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.284568+0000', 'last_undegraded': '2024-03-06T07:22:42.284568+0000', 'last_fullsized': '2024-03-06T07:22:42.284568+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3b is not active+clean 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.279457+0000', 'last_change': '2024-03-06T07:22:42.279457+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.279457+0000', 'last_undegraded': '2024-03-06T07:22:42.279457+0000', 'last_fullsized': '2024-03-06T07:22:42.279457+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3a is not active+clean 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283015+0000', 'last_change': '2024-03-06T07:22:42.283015+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283015+0000', 'last_undegraded': '2024-03-06T07:22:42.283015+0000', 'last_fullsized': '2024-03-06T07:22:42.283015+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.39 is not active+clean 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.292601+0000', 'last_change': '2024-03-06T07:22:42.292601+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.292601+0000', 'last_undegraded': '2024-03-06T07:22:42.292601+0000', 'last_fullsized': '2024-03-06T07:22:42.292601+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.38 is not active+clean 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.285418+0000', 'last_change': '2024-03-06T07:22:42.285418+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.285418+0000', 'last_undegraded': '2024-03-06T07:22:42.285418+0000', 'last_fullsized': '2024-03-06T07:22:42.285418+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.37 is not active+clean 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282898+0000', 'last_change': '2024-03-06T07:22:42.282898+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282898+0000', 'last_undegraded': '2024-03-06T07:22:42.282898+0000', 'last_fullsized': '2024-03-06T07:22:42.282898+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.36 is not active+clean 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282437+0000', 'last_change': '2024-03-06T07:22:42.282437+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282437+0000', 'last_undegraded': '2024-03-06T07:22:42.282437+0000', 'last_fullsized': '2024-03-06T07:22:42.282437+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.35 is not active+clean 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283810+0000', 'last_change': '2024-03-06T07:22:42.283810+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283810+0000', 'last_undegraded': '2024-03-06T07:22:42.283810+0000', 'last_fullsized': '2024-03-06T07:22:42.283810+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.34 is not active+clean 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.278978+0000', 'last_change': '2024-03-06T07:22:42.278978+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.278978+0000', 'last_undegraded': '2024-03-06T07:22:42.278978+0000', 'last_fullsized': '2024-03-06T07:22:42.278978+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.33 is not active+clean 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282804+0000', 'last_change': '2024-03-06T07:22:42.282804+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282804+0000', 'last_undegraded': '2024-03-06T07:22:42.282804+0000', 'last_fullsized': '2024-03-06T07:22:42.282804+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.32 is not active+clean 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.278897+0000', 'last_change': '2024-03-06T07:22:42.278897+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.278897+0000', 'last_undegraded': '2024-03-06T07:22:42.278897+0000', 'last_fullsized': '2024-03-06T07:22:42.278897+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.31 is not active+clean 2024-03-06T07:22:46.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.278443+0000', 'last_change': '2024-03-06T07:22:42.278443+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.278443+0000', 'last_undegraded': '2024-03-06T07:22:42.278443+0000', 'last_fullsized': '2024-03-06T07:22:42.278443+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.30 is not active+clean 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283228+0000', 'last_change': '2024-03-06T07:22:42.283228+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283228+0000', 'last_undegraded': '2024-03-06T07:22:42.283228+0000', 'last_fullsized': '2024-03-06T07:22:42.283228+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2f is not active+clean 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282177+0000', 'last_change': '2024-03-06T07:22:42.282177+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282177+0000', 'last_undegraded': '2024-03-06T07:22:42.282177+0000', 'last_fullsized': '2024-03-06T07:22:42.282177+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2e is not active+clean 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.279562+0000', 'last_change': '2024-03-06T07:22:42.279562+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.279562+0000', 'last_undegraded': '2024-03-06T07:22:42.279562+0000', 'last_fullsized': '2024-03-06T07:22:42.279562+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2d is not active+clean 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.293286+0000', 'last_change': '2024-03-06T07:22:42.293286+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.293286+0000', 'last_undegraded': '2024-03-06T07:22:42.293286+0000', 'last_fullsized': '2024-03-06T07:22:42.293286+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2c is not active+clean 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.280297+0000', 'last_change': '2024-03-06T07:22:42.280297+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.280297+0000', 'last_undegraded': '2024-03-06T07:22:42.280297+0000', 'last_fullsized': '2024-03-06T07:22:42.280297+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2b is not active+clean 2024-03-06T07:22:46.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283843+0000', 'last_change': '2024-03-06T07:22:42.283843+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283843+0000', 'last_undegraded': '2024-03-06T07:22:42.283843+0000', 'last_fullsized': '2024-03-06T07:22:42.283843+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2a is not active+clean 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.278394+0000', 'last_change': '2024-03-06T07:22:42.278394+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.278394+0000', 'last_undegraded': '2024-03-06T07:22:42.278394+0000', 'last_fullsized': '2024-03-06T07:22:42.278394+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.29 is not active+clean 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.284222+0000', 'last_change': '2024-03-06T07:22:42.284222+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.284222+0000', 'last_undegraded': '2024-03-06T07:22:42.284222+0000', 'last_fullsized': '2024-03-06T07:22:42.284222+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.28 is not active+clean 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282160+0000', 'last_change': '2024-03-06T07:22:42.282160+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282160+0000', 'last_undegraded': '2024-03-06T07:22:42.282160+0000', 'last_fullsized': '2024-03-06T07:22:42.282160+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.27 is not active+clean 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.286365+0000', 'last_change': '2024-03-06T07:22:42.286365+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.286365+0000', 'last_undegraded': '2024-03-06T07:22:42.286365+0000', 'last_fullsized': '2024-03-06T07:22:42.286365+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.26 is not active+clean 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.286778+0000', 'last_change': '2024-03-06T07:22:42.286778+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.286778+0000', 'last_undegraded': '2024-03-06T07:22:42.286778+0000', 'last_fullsized': '2024-03-06T07:22:42.286778+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.25 is not active+clean 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.292889+0000', 'last_change': '2024-03-06T07:22:42.292889+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.292889+0000', 'last_undegraded': '2024-03-06T07:22:42.292889+0000', 'last_fullsized': '2024-03-06T07:22:42.292889+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.24 is not active+clean 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282008+0000', 'last_change': '2024-03-06T07:22:42.282008+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282008+0000', 'last_undegraded': '2024-03-06T07:22:42.282008+0000', 'last_fullsized': '2024-03-06T07:22:42.282008+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.23 is not active+clean 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.286663+0000', 'last_change': '2024-03-06T07:22:42.286663+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.286663+0000', 'last_undegraded': '2024-03-06T07:22:42.286663+0000', 'last_fullsized': '2024-03-06T07:22:42.286663+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.22 is not active+clean 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.286216+0000', 'last_change': '2024-03-06T07:22:42.286216+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.286216+0000', 'last_undegraded': '2024-03-06T07:22:42.286216+0000', 'last_fullsized': '2024-03-06T07:22:42.286216+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.21 is not active+clean 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281552+0000', 'last_change': '2024-03-06T07:22:42.281552+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281552+0000', 'last_undegraded': '2024-03-06T07:22:42.281552+0000', 'last_fullsized': '2024-03-06T07:22:42.281552+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.20 is not active+clean 2024-03-06T07:22:46.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281197+0000', 'last_change': '2024-03-06T07:22:42.281197+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281197+0000', 'last_undegraded': '2024-03-06T07:22:42.281197+0000', 'last_fullsized': '2024-03-06T07:22:42.281197+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.18 is not active+clean 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281292+0000', 'last_change': '2024-03-06T07:22:42.281292+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281292+0000', 'last_undegraded': '2024-03-06T07:22:42.281292+0000', 'last_fullsized': '2024-03-06T07:22:42.281292+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.19 is not active+clean 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.277947+0000', 'last_change': '2024-03-06T07:22:42.277947+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.277947+0000', 'last_undegraded': '2024-03-06T07:22:42.277947+0000', 'last_fullsized': '2024-03-06T07:22:42.277947+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1a is not active+clean 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283761+0000', 'last_change': '2024-03-06T07:22:42.283761+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283761+0000', 'last_undegraded': '2024-03-06T07:22:42.283761+0000', 'last_fullsized': '2024-03-06T07:22:42.283761+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1b is not active+clean 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.277964+0000', 'last_change': '2024-03-06T07:22:42.277964+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.277964+0000', 'last_undegraded': '2024-03-06T07:22:42.277964+0000', 'last_fullsized': '2024-03-06T07:22:42.277964+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1c is not active+clean 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282847+0000', 'last_change': '2024-03-06T07:22:42.282847+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282847+0000', 'last_undegraded': '2024-03-06T07:22:42.282847+0000', 'last_fullsized': '2024-03-06T07:22:42.282847+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1d is not active+clean 2024-03-06T07:22:46.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283519+0000', 'last_change': '2024-03-06T07:22:42.283519+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283519+0000', 'last_undegraded': '2024-03-06T07:22:42.283519+0000', 'last_fullsized': '2024-03-06T07:22:42.283519+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1e is not active+clean 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.279197+0000', 'last_change': '2024-03-06T07:22:42.279197+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.279197+0000', 'last_undegraded': '2024-03-06T07:22:42.279197+0000', 'last_fullsized': '2024-03-06T07:22:42.279197+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1f is not active+clean 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.292509+0000', 'last_change': '2024-03-06T07:22:42.292509+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.292509+0000', 'last_undegraded': '2024-03-06T07:22:42.292509+0000', 'last_fullsized': '2024-03-06T07:22:42.292509+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.10 is not active+clean 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.277967+0000', 'last_change': '2024-03-06T07:22:42.277967+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.277967+0000', 'last_undegraded': '2024-03-06T07:22:42.277967+0000', 'last_fullsized': '2024-03-06T07:22:42.277967+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.11 is not active+clean 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281750+0000', 'last_change': '2024-03-06T07:22:42.281750+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281750+0000', 'last_undegraded': '2024-03-06T07:22:42.281750+0000', 'last_fullsized': '2024-03-06T07:22:42.281750+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.12 is not active+clean 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281997+0000', 'last_change': '2024-03-06T07:22:42.281997+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281997+0000', 'last_undegraded': '2024-03-06T07:22:42.281997+0000', 'last_fullsized': '2024-03-06T07:22:42.281997+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.13 is not active+clean 2024-03-06T07:22:46.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281717+0000', 'last_change': '2024-03-06T07:22:42.281717+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281717+0000', 'last_undegraded': '2024-03-06T07:22:42.281717+0000', 'last_fullsized': '2024-03-06T07:22:42.281717+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.14 is not active+clean 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281667+0000', 'last_change': '2024-03-06T07:22:42.281667+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281667+0000', 'last_undegraded': '2024-03-06T07:22:42.281667+0000', 'last_fullsized': '2024-03-06T07:22:42.281667+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.16 is not active+clean 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.290484+0000', 'last_change': '2024-03-06T07:22:42.290484+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.290484+0000', 'last_undegraded': '2024-03-06T07:22:42.290484+0000', 'last_fullsized': '2024-03-06T07:22:42.290484+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.17 is not active+clean 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.280053+0000', 'last_change': '2024-03-06T07:22:42.280053+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.280053+0000', 'last_undegraded': '2024-03-06T07:22:42.280053+0000', 'last_fullsized': '2024-03-06T07:22:42.280053+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.8 is not active+clean 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.284708+0000', 'last_change': '2024-03-06T07:22:42.284708+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.284708+0000', 'last_undegraded': '2024-03-06T07:22:42.284708+0000', 'last_fullsized': '2024-03-06T07:22:42.284708+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.9 is not active+clean 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.286139+0000', 'last_change': '2024-03-06T07:22:42.286139+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.286139+0000', 'last_undegraded': '2024-03-06T07:22:42.286139+0000', 'last_fullsized': '2024-03-06T07:22:42.286139+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.a is not active+clean 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.290881+0000', 'last_change': '2024-03-06T07:22:42.290881+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.290881+0000', 'last_undegraded': '2024-03-06T07:22:42.290881+0000', 'last_fullsized': '2024-03-06T07:22:42.290881+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.b is not active+clean 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281607+0000', 'last_change': '2024-03-06T07:22:42.281607+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281607+0000', 'last_undegraded': '2024-03-06T07:22:42.281607+0000', 'last_fullsized': '2024-03-06T07:22:42.281607+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.c is not active+clean 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.279871+0000', 'last_change': '2024-03-06T07:22:42.279871+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.279871+0000', 'last_undegraded': '2024-03-06T07:22:42.279871+0000', 'last_fullsized': '2024-03-06T07:22:42.279871+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.d is not active+clean 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.277869+0000', 'last_change': '2024-03-06T07:22:42.277869+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.277869+0000', 'last_undegraded': '2024-03-06T07:22:42.277869+0000', 'last_fullsized': '2024-03-06T07:22:42.277869+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.e is not active+clean 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.278812+0000', 'last_change': '2024-03-06T07:22:42.278812+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.278812+0000', 'last_undegraded': '2024-03-06T07:22:42.278812+0000', 'last_fullsized': '2024-03-06T07:22:42.278812+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.f is not active+clean 2024-03-06T07:22:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283044+0000', 'last_change': '2024-03-06T07:22:42.283044+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283044+0000', 'last_undegraded': '2024-03-06T07:22:42.283044+0000', 'last_fullsized': '2024-03-06T07:22:42.283044+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.0 is not active+clean 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.278746+0000', 'last_change': '2024-03-06T07:22:42.278746+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.278746+0000', 'last_undegraded': '2024-03-06T07:22:42.278746+0000', 'last_fullsized': '2024-03-06T07:22:42.278746+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.1 is not active+clean 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.282416+0000', 'last_change': '2024-03-06T07:22:42.282416+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.282416+0000', 'last_undegraded': '2024-03-06T07:22:42.282416+0000', 'last_fullsized': '2024-03-06T07:22:42.282416+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.2 is not active+clean 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.289810+0000', 'last_change': '2024-03-06T07:22:42.289810+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.289810+0000', 'last_undegraded': '2024-03-06T07:22:42.289810+0000', 'last_fullsized': '2024-03-06T07:22:42.289810+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.3 is not active+clean 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281207+0000', 'last_change': '2024-03-06T07:22:42.281207+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281207+0000', 'last_undegraded': '2024-03-06T07:22:42.281207+0000', 'last_fullsized': '2024-03-06T07:22:42.281207+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.4 is not active+clean 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.280504+0000', 'last_change': '2024-03-06T07:22:42.280504+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.280504+0000', 'last_undegraded': '2024-03-06T07:22:42.280504+0000', 'last_fullsized': '2024-03-06T07:22:42.280504+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.5 is not active+clean 2024-03-06T07:22:46.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.280985+0000', 'last_change': '2024-03-06T07:22:42.280985+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.280985+0000', 'last_undegraded': '2024-03-06T07:22:42.280985+0000', 'last_fullsized': '2024-03-06T07:22:42.280985+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.6 is not active+clean 2024-03-06T07:22:46.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.281379+0000', 'last_change': '2024-03-06T07:22:42.281379+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.281379+0000', 'last_undegraded': '2024-03-06T07:22:42.281379+0000', 'last_fullsized': '2024-03-06T07:22:42.281379+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:22:46.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.7 is not active+clean 2024-03-06T07:22:46.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.290349+0000', 'last_change': '2024-03-06T07:22:42.290349+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.290349+0000', 'last_undegraded': '2024-03-06T07:22:42.290349+0000', 'last_fullsized': '2024-03-06T07:22:42.290349+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:22:46.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 20.15 is not active+clean 2024-03-06T07:22:46.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '20.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 119, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:22:42.283399+0000', 'last_change': '2024-03-06T07:22:42.283399+0000', 'last_active': '2024-03-06T07:22:42.266236+0000', 'last_peered': '2024-03-06T07:22:42.266236+0000', 'last_clean': '2024-03-06T07:22:42.266236+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:22:42.283399+0000', 'last_undegraded': '2024-03-06T07:22:42.283399+0000', 'last_fullsized': '2024-03-06T07:22:42.283399+0000', 'mapping_epoch': 119, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 119, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'last_clean_scrub_stamp': '2024-03-06T07:22:42.266236+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:22:46.931 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:22:47.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.053+0000 7ff1fd0a4640 1 Processor -- start 2024-03-06T07:22:47.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.053+0000 7ff1fd0a4640 1 -- start start 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1fd0a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1f814f360 0x7ff1f8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1fd0a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff1f80c88c0 0x7ff1f80c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1fd0a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f8158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1fd0a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f80c8dd0 con 0x7ff1f814f360 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1fd0a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f80c73d0 con 0x7ff1f80c88c0 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1fd0a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80be2e0 con 0x7ff1f80c5880 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1f814f360 0x7ff1f8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff1f814f360 0x7ff1f8155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36326/0 (socket says 172.21.15.94:36326) 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 -- 172.21.15.94:0/2959673007 learned_addr learned my addr 172.21.15.94:0/2959673007 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f759e640 1 --2- 172.21.15.94:0/2959673007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f8158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f659c640 1 --2- 172.21.15.94:0/2959673007 >> v2:172.21.15.145:3300/0 conn(0x7ff1f80c88c0 0x7ff1f80c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 -- 172.21.15.94:0/2959673007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 msgr2=0x7ff1f8158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 --2- 172.21.15.94:0/2959673007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f8158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 -- 172.21.15.94:0/2959673007 >> v2:172.21.15.145:3300/0 conn(0x7ff1f80c88c0 msgr2=0x7ff1f80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 --2- 172.21.15.94:0/2959673007 >> v2:172.21.15.145:3300/0 conn(0x7ff1f80c88c0 0x7ff1f80c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.054+0000 7ff1f6d9d640 1 -- 172.21.15.94:0/2959673007 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80c7670 con 0x7ff1f814f360 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.055+0000 7ff1f6d9d640 1 --2- 172.21.15.94:0/2959673007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f814f360 0x7ff1f8155c10 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80c84c0 tx=0x7ff1e802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee82757733d5a02f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.055+0000 7ff1f5d9b640 1 -- 172.21.15.94:0/2959673007 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e802c9e0 con 0x7ff1f814f360 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.055+0000 7ff1f5d9b640 1 -- 172.21.15.94:0/2959673007 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e802cb80 con 0x7ff1f814f360 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.055+0000 7ff1f5d9b640 1 -- 172.21.15.94:0/2959673007 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80343f0 con 0x7ff1f814f360 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/2959673007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f814f360 msgr2=0x7ff1f8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/2959673007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f814f360 0x7ff1f8155c10 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80c84c0 tx=0x7ff1e802be40 comp rx=0 tx=0).stop 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/2959673007 shutdown_connections 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/2959673007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f8158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/2959673007 >> v2:172.21.15.145:3300/0 conn(0x7ff1f80c88c0 0x7ff1f80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/2959673007 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f814f360 0x7ff1f8155c10 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/2959673007 >> 172.21.15.94:0/2959673007 conn(0x7ff1f80ac570 msgr2=0x7ff1f80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/2959673007 shutdown_connections 2024-03-06T07:22:47.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.056+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/2959673007 wait complete. 2024-03-06T07:22:47.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.057+0000 7ff1fd0a4640 1 Processor -- start 2024-03-06T07:22:47.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.057+0000 7ff1fd0a4640 1 -- start start 2024-03-06T07:22:47.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1fd0a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f81580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1fd0a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f80c88c0 0x7ff1f81eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1fd0a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1f814f360 0x7ff1f8154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1fd0a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f81590a0 con 0x7ff1f80c88c0 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1fd0a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80c7c30 con 0x7ff1f814f360 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1fd0a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f8159220 con 0x7ff1f80c5880 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f80c88c0 0x7ff1f81eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f80c88c0 0x7ff1f81eed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36332/0 (socket says 172.21.15.94:36332) 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 -- 172.21.15.94:0/1673666866 learned_addr learned my addr 172.21.15.94:0/1673666866 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f759e640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1f814f360 0x7ff1f8154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f6d9d640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f81580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 -- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 msgr2=0x7ff1f81580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f81580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 -- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1f814f360 msgr2=0x7ff1f8154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1f814f360 0x7ff1f8154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f659c640 1 -- 172.21.15.94:0/1673666866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f81eae90 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.058+0000 7ff1f759e640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1f814f360 0x7ff1f8154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.059+0000 7ff1f659c640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f80c88c0 0x7ff1f81eed00 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00e9b0 tx=0x7ff1ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.059+0000 7ff1df7fe640 1 -- 172.21.15.94:0/1673666866 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec017070 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.059+0000 7ff1df7fe640 1 -- 172.21.15.94:0/1673666866 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ec00c9a0 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.059+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f81eb130 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.059+0000 7ff1df7fe640 1 -- 172.21.15.94:0/1673666866 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec00cd70 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.059+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f8154c40 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.060+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f80b05f0 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.060+0000 7ff1df7fe640 1 -- 172.21.15.94:0/1673666866 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1ec01a070 con 0x7ff1f80c88c0 2024-03-06T07:22:47.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.061+0000 7ff1df7fe640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff1e406b4d0 0x7ff1e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.061+0000 7ff1df7fe640 1 -- 172.21.15.94:0/1673666866 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff1ec08a690 con 0x7ff1f80c88c0 2024-03-06T07:22:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.061+0000 7ff1f6d9d640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff1e406b4d0 0x7ff1e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.061+0000 7ff1f6d9d640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff1e406b4d0 0x7ff1e406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8035000 tx=0x7ff1e8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.065+0000 7ff1df7fe640 1 -- 172.21.15.94:0/1673666866 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1f80b05f0 con 0x7ff1f80c88c0 2024-03-06T07:22:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.305+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff1f804f330 con 0x7ff1f80c88c0 2024-03-06T07:22:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.306+0000 7ff1df7fe640 1 -- 172.21.15.94:0/1673666866 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1456 (secure 0 0 0) 0x7ff1ec058d30 con 0x7ff1f80c88c0 2024-03-06T07:22:47.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:47.308 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":293,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":121,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709482,"num_in_osds":8,"osd_in_since":1709709474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":267415552,"bytes_avail":772826697728,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":104,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"b","status":"up:active","gid":7842}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-03-06T07:22:22.687878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff1e406b4d0 msgr2=0x7ff1e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff1e406b4d0 0x7ff1e406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8035000 tx=0x7ff1e8030000 comp rx=0 tx=0).stop 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f80c88c0 msgr2=0x7ff1f81eed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f80c88c0 0x7ff1f81eed00 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00e9b0 tx=0x7ff1ec00ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 shutdown_connections 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff1e406b4d0 0x7ff1e406d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1f814f360 0x7ff1f8154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff1f80c88c0 0x7ff1f81eed00 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 --2- 172.21.15.94:0/1673666866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1f80c5880 0x7ff1f81580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 >> 172.21.15.94:0/1673666866 conn(0x7ff1f80ac570 msgr2=0x7ff1f814d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 shutdown_connections 2024-03-06T07:22:47.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.308+0000 7ff1fd0a4640 1 -- 172.21.15.94:0/1673666866 wait complete. 2024-03-06T07:22:47.320 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:22:47.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.420+0000 7f9d6a1ac640 1 Processor -- start 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.420+0000 7f9d6a1ac640 1 -- start start 2024-03-06T07:22:47.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d6a1ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9d6414f790 0x7f9d641529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d6a1ac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9d640c1aa0 0x7f9d640c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d6a1ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d640c28d0 con 0x7f9d640c5c60 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d6a1ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d640c2680 con 0x7f9d6414f790 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d6a1ac640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d640c2bb0 con 0x7f9d640c1aa0 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d637fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9d6414f790 0x7f9d641529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d637fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9d6414f790 0x7f9d641529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36340/0 (socket says 172.21.15.94:36340) 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d637fe640 1 -- 172.21.15.94:0/154684754 learned_addr learned my addr 172.21.15.94:0/154684754 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d62ffd640 1 --2- 172.21.15.94:0/154684754 >> v2:172.21.15.145:3300/0 conn(0x7f9d640c1aa0 0x7f9d640c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d637fe640 1 -- 172.21.15.94:0/154684754 >> v2:172.21.15.145:3300/0 conn(0x7f9d640c1aa0 msgr2=0x7f9d640c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d637fe640 1 --2- 172.21.15.94:0/154684754 >> v2:172.21.15.145:3300/0 conn(0x7f9d640c1aa0 0x7f9d640c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d637fe640 1 -- 172.21.15.94:0/154684754 >> v1:172.21.15.145:6789/0 conn(0x7f9d640c5c60 legacy=0x7f9d64155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d637fe640 1 -- 172.21.15.94:0/154684754 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640c2e90 con 0x7f9d6414f790 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.421+0000 7f9d62ffd640 1 --2- 172.21.15.94:0/154684754 >> v2:172.21.15.145:3300/0 conn(0x7f9d640c1aa0 0x7f9d640c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d637fe640 1 --2- 172.21.15.94:0/154684754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6414f790 0x7f9d641529d0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f9d64070280 tx=0x7f9d5802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615bd71ea3a2766d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d627fc640 1 -- 172.21.15.94:0/154684754 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5803b070 con 0x7f9d6414f790 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d627fc640 1 -- 172.21.15.94:0/154684754 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d58036440 con 0x7f9d6414f790 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d627fc640 1 -- 172.21.15.94:0/154684754 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58036810 con 0x7f9d6414f790 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/154684754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6414f790 msgr2=0x7f9d641529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/154684754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6414f790 0x7f9d641529d0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f9d64070280 tx=0x7f9d5802e170 comp rx=0 tx=0).stop 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/154684754 shutdown_connections 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/154684754 >> v2:172.21.15.145:3300/0 conn(0x7f9d640c1aa0 0x7f9d640c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/154684754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d6414f790 0x7f9d641529d0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/154684754 >> 172.21.15.94:0/154684754 conn(0x7f9d640c6710 msgr2=0x7f9d640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/154684754 shutdown_connections 2024-03-06T07:22:47.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.422+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/154684754 wait complete. 2024-03-06T07:22:47.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 Processor -- start 2024-03-06T07:22:47.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 -- start start 2024-03-06T07:22:47.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d640c1aa0 0x7f9d64154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d640c5c60 0x7f9d6414c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d6414f790 0x7f9d641e6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6406cc00 con 0x7f9d640c1aa0 2024-03-06T07:22:47.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640c0d60 con 0x7f9d640c5c60 2024-03-06T07:22:47.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d6a1ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6406cd80 con 0x7f9d6414f790 2024-03-06T07:22:47.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d640c1aa0 0x7f9d64154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d640c1aa0 0x7f9d64154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36350/0 (socket says 172.21.15.94:36350) 2024-03-06T07:22:47.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.424+0000 7f9d637fe640 1 -- 172.21.15.94:0/3040227047 learned_addr learned my addr 172.21.15.94:0/3040227047 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:47.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d62ffd640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d640c5c60 0x7f9d6414c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d63fff640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d6414f790 0x7f9d641e6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d637fe640 1 -- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d6414f790 msgr2=0x7f9d641e6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d637fe640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d6414f790 0x7f9d641e6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d637fe640 1 -- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d640c5c60 msgr2=0x7f9d6414c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d637fe640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d640c5c60 0x7f9d6414c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d637fe640 1 -- 172.21.15.94:0/3040227047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6414cfc0 con 0x7f9d640c1aa0 2024-03-06T07:22:47.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d637fe640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d640c1aa0 0x7f9d64154370 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f9d5802e1b0 tx=0x7f9d580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d4bfff640 1 -- 172.21.15.94:0/3040227047 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58045020 con 0x7f9d640c1aa0 2024-03-06T07:22:47.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d4bfff640 1 -- 172.21.15.94:0/3040227047 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d580346e0 con 0x7f9d640c1aa0 2024-03-06T07:22:47.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d4bfff640 1 -- 172.21.15.94:0/3040227047 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5803b040 con 0x7f9d640c1aa0 2024-03-06T07:22:47.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d641eac10 con 0x7f9d640c1aa0 2024-03-06T07:22:47.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.425+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d641eaef0 con 0x7f9d640c1aa0 2024-03-06T07:22:47.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.426+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6404f330 con 0x7f9d640c1aa0 2024-03-06T07:22:47.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.427+0000 7f9d4bfff640 1 -- 172.21.15.94:0/3040227047 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d58048020 con 0x7f9d640c1aa0 2024-03-06T07:22:47.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.427+0000 7f9d4bfff640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d5006b4d0 0x7f9d5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.427+0000 7f9d4bfff640 1 -- 172.21.15.94:0/3040227047 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9d580aebf0 con 0x7f9d640c1aa0 2024-03-06T07:22:47.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.427+0000 7f9d62ffd640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d5006b4d0 0x7f9d5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.428+0000 7f9d62ffd640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d5006b4d0 0x7f9d5006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9d640c14b0 tx=0x7f9d54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.431+0000 7f9d4bfff640 1 -- 172.21.15.94:0/3040227047 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d5804f030 con 0x7f9d640c1aa0 2024-03-06T07:22:47.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.601+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d6414d2a0 con 0x7f9d5006b4d0 2024-03-06T07:22:47.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.607+0000 7f9d4bfff640 1 -- 172.21.15.94:0/3040227047 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247154 (secure 0 0 0) 0x7f9d6414d2a0 con 0x7f9d5006b4d0 2024-03-06T07:22:47.612 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:47.613 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:22:47.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d5006b4d0 msgr2=0x7f9d5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d5006b4d0 0x7f9d5006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9d640c14b0 tx=0x7f9d54008040 comp rx=0 tx=0).stop 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d640c1aa0 msgr2=0x7f9d64154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d640c1aa0 0x7f9d64154370 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f9d5802e1b0 tx=0x7f9d580046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 shutdown_connections 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d5006b4d0 0x7f9d5006d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d6414f790 0x7f9d641e6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d640c5c60 0x7f9d6414c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 --2- 172.21.15.94:0/3040227047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d640c1aa0 0x7f9d64154370 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 >> 172.21.15.94:0/3040227047 conn(0x7f9d640c6710 msgr2=0x7f9d640a9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 shutdown_connections 2024-03-06T07:22:47.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:47.610+0000 7f9d6a1ac640 1 -- 172.21.15.94:0/3040227047 wait complete. 2024-03-06T07:22:47.623 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-03-06T07:22:46.693446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261148,"kb_used_data":11692,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713572,"statfs":{"total":773094113280,"available":772826697728,"internally_reserved":0,"allocated":11972608,"data_stored":6858699,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.427010"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279032+0000","last_change":"2024-03-06T07:22:42.279032+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279032+0000","last_undegraded":"2024-03-06T07:22:42.279032+0000","last_fullsized":"2024-03-06T07:22:42.279032+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284203+0000","last_change":"2024-03-06T07:22:42.284203+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284203+0000","last_undegraded":"2024-03-06T07:22:42.284203+0000","last_fullsized":"2024-03-06T07:22:42.284203+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284568+0000","last_change":"2024-03-06T07:22:42.284568+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284568+0000","last_undegraded":"2024-03-06T07:22:42.284568+0000","last_fullsized":"2024-03-06T07:22:42.284568+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279457+0000","last_change":"2024-03-06T07:22:42.279457+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279457+0000","last_undegraded":"2024-03-06T07:22:42.279457+0000","last_fullsized":"2024-03-06T07:22:42.279457+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283015+0000","last_change":"2024-03-06T07:22:42.283015+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283015+0000","last_undegraded":"2024-03-06T07:22:42.283015+0000","last_fullsized":"2024-03-06T07:22:42.283015+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292601+0000","last_change":"2024-03-06T07:22:42.292601+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292601+0000","last_undegraded":"2024-03-06T07:22:42.292601+0000","last_fullsized":"2024-03-06T07:22:42.292601+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.285418+0000","last_change":"2024-03-06T07:22:42.285418+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.285418+0000","last_undegraded":"2024-03-06T07:22:42.285418+0000","last_fullsized":"2024-03-06T07:22:42.285418+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282898+0000","last_change":"2024-03-06T07:22:42.282898+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282898+0000","last_undegraded":"2024-03-06T07:22:42.282898+0000","last_fullsized":"2024-03-06T07:22:42.282898+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282437+0000","last_change":"2024-03-06T07:22:42.282437+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282437+0000","last_undegraded":"2024-03-06T07:22:42.282437+0000","last_fullsized":"2024-03-06T07:22:42.282437+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283810+0000","last_change":"2024-03-06T07:22:42.283810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283810+0000","last_undegraded":"2024-03-06T07:22:42.283810+0000","last_fullsized":"2024-03-06T07:22:42.283810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278978+0000","last_change":"2024-03-06T07:22:42.278978+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278978+0000","last_undegraded":"2024-03-06T07:22:42.278978+0000","last_fullsized":"2024-03-06T07:22:42.278978+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282804+0000","last_change":"2024-03-06T07:22:42.282804+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282804+0000","last_undegraded":"2024-03-06T07:22:42.282804+0000","last_fullsized":"2024-03-06T07:22:42.282804+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278897+0000","last_change":"2024-03-06T07:22:42.278897+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278897+0000","last_undegraded":"2024-03-06T07:22:42.278897+0000","last_fullsized":"2024-03-06T07:22:42.278897+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278443+0000","last_change":"2024-03-06T07:22:42.278443+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278443+0000","last_undegraded":"2024-03-06T07:22:42.278443+0000","last_fullsized":"2024-03-06T07:22:42.278443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283228+0000","last_change":"2024-03-06T07:22:42.283228+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283228+0000","last_undegraded":"2024-03-06T07:22:42.283228+0000","last_fullsized":"2024-03-06T07:22:42.283228+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282177+0000","last_change":"2024-03-06T07:22:42.282177+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282177+0000","last_undegraded":"2024-03-06T07:22:42.282177+0000","last_fullsized":"2024-03-06T07:22:42.282177+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279562+0000","last_change":"2024-03-06T07:22:42.279562+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279562+0000","last_undegraded":"2024-03-06T07:22:42.279562+0000","last_fullsized":"2024-03-06T07:22:42.279562+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.293286+0000","last_change":"2024-03-06T07:22:42.293286+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.293286+0000","last_undegraded":"2024-03-06T07:22:42.293286+0000","last_fullsized":"2024-03-06T07:22:42.293286+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280297+0000","last_change":"2024-03-06T07:22:42.280297+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280297+0000","last_undegraded":"2024-03-06T07:22:42.280297+0000","last_fullsized":"2024-03-06T07:22:42.280297+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283843+0000","last_change":"2024-03-06T07:22:42.283843+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283843+0000","last_undegraded":"2024-03-06T07:22:42.283843+0000","last_fullsized":"2024-03-06T07:22:42.283843+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278394+0000","last_change":"2024-03-06T07:22:42.278394+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278394+0000","last_undegraded":"2024-03-06T07:22:42.278394+0000","last_fullsized":"2024-03-06T07:22:42.278394+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284222+0000","last_change":"2024-03-06T07:22:42.284222+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284222+0000","last_undegraded":"2024-03-06T07:22:42.284222+0000","last_fullsized":"2024-03-06T07:22:42.284222+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282160+0000","last_change":"2024-03-06T07:22:42.282160+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282160+0000","last_undegraded":"2024-03-06T07:22:42.282160+0000","last_fullsized":"2024-03-06T07:22:42.282160+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286365+0000","last_change":"2024-03-06T07:22:42.286365+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286365+0000","last_undegraded":"2024-03-06T07:22:42.286365+0000","last_fullsized":"2024-03-06T07:22:42.286365+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286778+0000","last_change":"2024-03-06T07:22:42.286778+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286778+0000","last_undegraded":"2024-03-06T07:22:42.286778+0000","last_fullsized":"2024-03-06T07:22:42.286778+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292889+0000","last_change":"2024-03-06T07:22:42.292889+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292889+0000","last_undegraded":"2024-03-06T07:22:42.292889+0000","last_fullsized":"2024-03-06T07:22:42.292889+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282008+0000","last_change":"2024-03-06T07:22:42.282008+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282008+0000","last_undegraded":"2024-03-06T07:22:42.282008+0000","last_fullsized":"2024-03-06T07:22:42.282008+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286663+0000","last_change":"2024-03-06T07:22:42.286663+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286663+0000","last_undegraded":"2024-03-06T07:22:42.286663+0000","last_fullsized":"2024-03-06T07:22:42.286663+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286216+0000","last_change":"2024-03-06T07:22:42.286216+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286216+0000","last_undegraded":"2024-03-06T07:22:42.286216+0000","last_fullsized":"2024-03-06T07:22:42.286216+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281552+0000","last_change":"2024-03-06T07:22:42.281552+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281552+0000","last_undegraded":"2024-03-06T07:22:42.281552+0000","last_fullsized":"2024-03-06T07:22:42.281552+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281197+0000","last_change":"2024-03-06T07:22:42.281197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281197+0000","last_undegraded":"2024-03-06T07:22:42.281197+0000","last_fullsized":"2024-03-06T07:22:42.281197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281292+0000","last_change":"2024-03-06T07:22:42.281292+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281292+0000","last_undegraded":"2024-03-06T07:22:42.281292+0000","last_fullsized":"2024-03-06T07:22:42.281292+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289168+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288869+0000","last_became_peered":"2024-03-06T07:22:42.288869+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:22.433887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277947+0000","last_change":"2024-03-06T07:22:42.277947+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277947+0000","last_undegraded":"2024-03-06T07:22:42.277947+0000","last_fullsized":"2024-03-06T07:22:42.277947+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.284819+0000","last_change":"2024-03-06T07:22:42.284230+0000","last_active":"2024-03-06T07:22:42.284819+0000","last_peered":"2024-03-06T07:22:42.284819+0000","last_clean":"2024-03-06T07:22:42.284819+0000","last_became_active":"2024-03-06T07:22:42.283796+0000","last_became_peered":"2024-03-06T07:22:42.283796+0000","last_unstale":"2024-03-06T07:22:42.284819+0000","last_undegraded":"2024-03-06T07:22:42.284819+0000","last_fullsized":"2024-03-06T07:22:42.284819+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:28.750113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283761+0000","last_change":"2024-03-06T07:22:42.283761+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283761+0000","last_undegraded":"2024-03-06T07:22:42.283761+0000","last_fullsized":"2024-03-06T07:22:42.283761+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299956+0000","last_change":"2024-03-06T07:22:42.298123+0000","last_active":"2024-03-06T07:22:42.299956+0000","last_peered":"2024-03-06T07:22:42.299956+0000","last_clean":"2024-03-06T07:22:42.299956+0000","last_became_active":"2024-03-06T07:22:42.297656+0000","last_became_peered":"2024-03-06T07:22:42.297656+0000","last_unstale":"2024-03-06T07:22:42.299956+0000","last_undegraded":"2024-03-06T07:22:42.299956+0000","last_fullsized":"2024-03-06T07:22:42.299956+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:56.531364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277964+0000","last_change":"2024-03-06T07:22:42.277964+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277964+0000","last_undegraded":"2024-03-06T07:22:42.277964+0000","last_fullsized":"2024-03-06T07:22:42.277964+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298336+0000","last_change":"2024-03-06T07:22:42.295819+0000","last_active":"2024-03-06T07:22:42.298336+0000","last_peered":"2024-03-06T07:22:42.298336+0000","last_clean":"2024-03-06T07:22:42.298336+0000","last_became_active":"2024-03-06T07:22:42.295541+0000","last_became_peered":"2024-03-06T07:22:42.295541+0000","last_unstale":"2024-03-06T07:22:42.298336+0000","last_undegraded":"2024-03-06T07:22:42.298336+0000","last_fullsized":"2024-03-06T07:22:42.298336+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:51.235851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282847+0000","last_change":"2024-03-06T07:22:42.282847+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282847+0000","last_undegraded":"2024-03-06T07:22:42.282847+0000","last_fullsized":"2024-03-06T07:22:42.282847+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289767+0000","last_change":"2024-03-06T07:22:42.287120+0000","last_active":"2024-03-06T07:22:42.289767+0000","last_peered":"2024-03-06T07:22:42.289767+0000","last_clean":"2024-03-06T07:22:42.289767+0000","last_became_active":"2024-03-06T07:22:42.286844+0000","last_became_peered":"2024-03-06T07:22:42.286844+0000","last_unstale":"2024-03-06T07:22:42.289767+0000","last_undegraded":"2024-03-06T07:22:42.289767+0000","last_fullsized":"2024-03-06T07:22:42.289767+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.165551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283519+0000","last_change":"2024-03-06T07:22:42.283519+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283519+0000","last_undegraded":"2024-03-06T07:22:42.283519+0000","last_fullsized":"2024-03-06T07:22:42.283519+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.290193+0000","last_change":"2024-03-06T07:22:42.289388+0000","last_active":"2024-03-06T07:22:42.290193+0000","last_peered":"2024-03-06T07:22:42.290193+0000","last_clean":"2024-03-06T07:22:42.290193+0000","last_became_active":"2024-03-06T07:22:42.288148+0000","last_became_peered":"2024-03-06T07:22:42.288148+0000","last_unstale":"2024-03-06T07:22:42.290193+0000","last_undegraded":"2024-03-06T07:22:42.290193+0000","last_fullsized":"2024-03-06T07:22:42.290193+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:37.212217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279197+0000","last_change":"2024-03-06T07:22:42.279197+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279197+0000","last_undegraded":"2024-03-06T07:22:42.279197+0000","last_fullsized":"2024-03-06T07:22:42.279197+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.287719+0000","last_change":"2024-03-06T07:22:42.285183+0000","last_active":"2024-03-06T07:22:42.287719+0000","last_peered":"2024-03-06T07:22:42.287719+0000","last_clean":"2024-03-06T07:22:42.287719+0000","last_became_active":"2024-03-06T07:22:42.284924+0000","last_became_peered":"2024-03-06T07:22:42.284924+0000","last_unstale":"2024-03-06T07:22:42.287719+0000","last_undegraded":"2024-03-06T07:22:42.287719+0000","last_fullsized":"2024-03-06T07:22:42.287719+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:21.527693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.292509+0000","last_change":"2024-03-06T07:22:42.292509+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.292509+0000","last_undegraded":"2024-03-06T07:22:42.292509+0000","last_fullsized":"2024-03-06T07:22:42.292509+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296297+0000","last_change":"2024-03-06T07:22:42.289621+0000","last_active":"2024-03-06T07:22:42.296297+0000","last_peered":"2024-03-06T07:22:42.296297+0000","last_clean":"2024-03-06T07:22:42.296297+0000","last_became_active":"2024-03-06T07:22:42.289239+0000","last_became_peered":"2024-03-06T07:22:42.289239+0000","last_unstale":"2024-03-06T07:22:42.296297+0000","last_undegraded":"2024-03-06T07:22:42.296297+0000","last_fullsized":"2024-03-06T07:22:42.296297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:40.681976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277967+0000","last_change":"2024-03-06T07:22:42.277967+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277967+0000","last_undegraded":"2024-03-06T07:22:42.277967+0000","last_fullsized":"2024-03-06T07:22:42.277967+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291813+0000","last_change":"2024-03-06T07:22:42.287554+0000","last_active":"2024-03-06T07:22:42.291813+0000","last_peered":"2024-03-06T07:22:42.291813+0000","last_clean":"2024-03-06T07:22:42.291813+0000","last_became_active":"2024-03-06T07:22:42.285213+0000","last_became_peered":"2024-03-06T07:22:42.285213+0000","last_unstale":"2024-03-06T07:22:42.291813+0000","last_undegraded":"2024-03-06T07:22:42.291813+0000","last_fullsized":"2024-03-06T07:22:42.291813+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:07.580402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281750+0000","last_change":"2024-03-06T07:22:42.281750+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281750+0000","last_undegraded":"2024-03-06T07:22:42.281750+0000","last_fullsized":"2024-03-06T07:22:42.281750+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288109+0000","last_change":"2024-03-06T07:22:42.286796+0000","last_active":"2024-03-06T07:22:42.288109+0000","last_peered":"2024-03-06T07:22:42.288109+0000","last_clean":"2024-03-06T07:22:42.288109+0000","last_became_active":"2024-03-06T07:22:42.286558+0000","last_became_peered":"2024-03-06T07:22:42.286558+0000","last_unstale":"2024-03-06T07:22:42.288109+0000","last_undegraded":"2024-03-06T07:22:42.288109+0000","last_fullsized":"2024-03-06T07:22:42.288109+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:11.589911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281997+0000","last_change":"2024-03-06T07:22:42.281997+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281997+0000","last_undegraded":"2024-03-06T07:22:42.281997+0000","last_fullsized":"2024-03-06T07:22:42.281997+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299969+0000","last_change":"2024-03-06T07:22:42.298748+0000","last_active":"2024-03-06T07:22:42.299969+0000","last_peered":"2024-03-06T07:22:42.299969+0000","last_clean":"2024-03-06T07:22:42.299969+0000","last_became_active":"2024-03-06T07:22:42.298459+0000","last_became_peered":"2024-03-06T07:22:42.298459+0000","last_unstale":"2024-03-06T07:22:42.299969+0000","last_undegraded":"2024-03-06T07:22:42.299969+0000","last_fullsized":"2024-03-06T07:22:42.299969+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:08.721690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281717+0000","last_change":"2024-03-06T07:22:42.281717+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281717+0000","last_undegraded":"2024-03-06T07:22:42.281717+0000","last_fullsized":"2024-03-06T07:22:42.281717+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.299706+0000","last_change":"2024-03-06T07:22:42.297582+0000","last_active":"2024-03-06T07:22:42.299706+0000","last_peered":"2024-03-06T07:22:42.299706+0000","last_clean":"2024-03-06T07:22:42.299706+0000","last_became_active":"2024-03-06T07:22:42.297309+0000","last_became_peered":"2024-03-06T07:22:42.297309+0000","last_unstale":"2024-03-06T07:22:42.299706+0000","last_undegraded":"2024-03-06T07:22:42.299706+0000","last_fullsized":"2024-03-06T07:22:42.299706+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:00.525005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281667+0000","last_change":"2024-03-06T07:22:42.281667+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281667+0000","last_undegraded":"2024-03-06T07:22:42.281667+0000","last_fullsized":"2024-03-06T07:22:42.281667+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.288179+0000","last_change":"2024-03-06T07:22:42.287402+0000","last_active":"2024-03-06T07:22:42.288179+0000","last_peered":"2024-03-06T07:22:42.288179+0000","last_clean":"2024-03-06T07:22:42.288179+0000","last_became_active":"2024-03-06T07:22:42.287169+0000","last_became_peered":"2024-03-06T07:22:42.287169+0000","last_unstale":"2024-03-06T07:22:42.288179+0000","last_undegraded":"2024-03-06T07:22:42.288179+0000","last_fullsized":"2024-03-06T07:22:42.288179+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:39.487955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290484+0000","last_change":"2024-03-06T07:22:42.290484+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290484+0000","last_undegraded":"2024-03-06T07:22:42.290484+0000","last_fullsized":"2024-03-06T07:22:42.290484+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296242+0000","last_change":"2024-03-06T07:22:42.289384+0000","last_active":"2024-03-06T07:22:42.296242+0000","last_peered":"2024-03-06T07:22:42.296242+0000","last_clean":"2024-03-06T07:22:42.296242+0000","last_became_active":"2024-03-06T07:22:42.289045+0000","last_became_peered":"2024-03-06T07:22:42.289045+0000","last_unstale":"2024-03-06T07:22:42.296242+0000","last_undegraded":"2024-03-06T07:22:42.296242+0000","last_fullsized":"2024-03-06T07:22:42.296242+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:08.837325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280053+0000","last_change":"2024-03-06T07:22:42.280053+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280053+0000","last_undegraded":"2024-03-06T07:22:42.280053+0000","last_fullsized":"2024-03-06T07:22:42.280053+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.295035+0000","last_change":"2024-03-06T07:22:42.294517+0000","last_active":"2024-03-06T07:22:42.295035+0000","last_peered":"2024-03-06T07:22:42.295035+0000","last_clean":"2024-03-06T07:22:42.295035+0000","last_became_active":"2024-03-06T07:22:42.294249+0000","last_became_peered":"2024-03-06T07:22:42.294249+0000","last_unstale":"2024-03-06T07:22:42.295035+0000","last_undegraded":"2024-03-06T07:22:42.295035+0000","last_fullsized":"2024-03-06T07:22:42.295035+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:12.295033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.284708+0000","last_change":"2024-03-06T07:22:42.284708+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.284708+0000","last_undegraded":"2024-03-06T07:22:42.284708+0000","last_fullsized":"2024-03-06T07:22:42.284708+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291835+0000","last_change":"2024-03-06T07:22:42.288271+0000","last_active":"2024-03-06T07:22:42.291835+0000","last_peered":"2024-03-06T07:22:42.291835+0000","last_clean":"2024-03-06T07:22:42.291835+0000","last_became_active":"2024-03-06T07:22:42.287981+0000","last_became_peered":"2024-03-06T07:22:42.287981+0000","last_unstale":"2024-03-06T07:22:42.291835+0000","last_undegraded":"2024-03-06T07:22:42.291835+0000","last_fullsized":"2024-03-06T07:22:42.291835+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.744017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.286139+0000","last_change":"2024-03-06T07:22:42.286139+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.286139+0000","last_undegraded":"2024-03-06T07:22:42.286139+0000","last_fullsized":"2024-03-06T07:22:42.286139+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.296273+0000","last_change":"2024-03-06T07:22:42.289825+0000","last_active":"2024-03-06T07:22:42.296273+0000","last_peered":"2024-03-06T07:22:42.296273+0000","last_clean":"2024-03-06T07:22:42.296273+0000","last_became_active":"2024-03-06T07:22:42.288884+0000","last_became_peered":"2024-03-06T07:22:42.288884+0000","last_unstale":"2024-03-06T07:22:42.296273+0000","last_undegraded":"2024-03-06T07:22:42.296273+0000","last_fullsized":"2024-03-06T07:22:42.296273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:05.920275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290881+0000","last_change":"2024-03-06T07:22:42.290881+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290881+0000","last_undegraded":"2024-03-06T07:22:42.290881+0000","last_fullsized":"2024-03-06T07:22:42.290881+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289606+0000","last_change":"2024-03-06T07:22:42.286639+0000","last_active":"2024-03-06T07:22:42.289606+0000","last_peered":"2024-03-06T07:22:42.289606+0000","last_clean":"2024-03-06T07:22:42.289606+0000","last_became_active":"2024-03-06T07:22:42.286392+0000","last_became_peered":"2024-03-06T07:22:42.286392+0000","last_unstale":"2024-03-06T07:22:42.289606+0000","last_undegraded":"2024-03-06T07:22:42.289606+0000","last_fullsized":"2024-03-06T07:22:42.289606+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:40.885949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281607+0000","last_change":"2024-03-06T07:22:42.281607+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281607+0000","last_undegraded":"2024-03-06T07:22:42.281607+0000","last_fullsized":"2024-03-06T07:22:42.281607+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.285286+0000","last_change":"2024-03-06T07:22:42.284042+0000","last_active":"2024-03-06T07:22:42.285286+0000","last_peered":"2024-03-06T07:22:42.285286+0000","last_clean":"2024-03-06T07:22:42.285286+0000","last_became_active":"2024-03-06T07:22:42.283781+0000","last_became_peered":"2024-03-06T07:22:42.283781+0000","last_unstale":"2024-03-06T07:22:42.285286+0000","last_undegraded":"2024-03-06T07:22:42.285286+0000","last_fullsized":"2024-03-06T07:22:42.285286+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.123546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.279871+0000","last_change":"2024-03-06T07:22:42.279871+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.279871+0000","last_undegraded":"2024-03-06T07:22:42.279871+0000","last_fullsized":"2024-03-06T07:22:42.279871+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298230+0000","last_change":"2024-03-06T07:22:42.297131+0000","last_active":"2024-03-06T07:22:42.298230+0000","last_peered":"2024-03-06T07:22:42.298230+0000","last_clean":"2024-03-06T07:22:42.298230+0000","last_became_active":"2024-03-06T07:22:42.296815+0000","last_became_peered":"2024-03-06T07:22:42.296815+0000","last_unstale":"2024-03-06T07:22:42.298230+0000","last_undegraded":"2024-03-06T07:22:42.298230+0000","last_fullsized":"2024-03-06T07:22:42.298230+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:56.374398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.277869+0000","last_change":"2024-03-06T07:22:42.277869+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.277869+0000","last_undegraded":"2024-03-06T07:22:42.277869+0000","last_fullsized":"2024-03-06T07:22:42.277869+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289232+0000","last_change":"2024-03-06T07:22:42.288618+0000","last_active":"2024-03-06T07:22:42.289232+0000","last_peered":"2024-03-06T07:22:42.289232+0000","last_clean":"2024-03-06T07:22:42.289232+0000","last_became_active":"2024-03-06T07:22:42.287863+0000","last_became_peered":"2024-03-06T07:22:42.287863+0000","last_unstale":"2024-03-06T07:22:42.289232+0000","last_undegraded":"2024-03-06T07:22:42.289232+0000","last_fullsized":"2024-03-06T07:22:42.289232+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:06.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278812+0000","last_change":"2024-03-06T07:22:42.278812+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278812+0000","last_undegraded":"2024-03-06T07:22:42.278812+0000","last_fullsized":"2024-03-06T07:22:42.278812+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298112+0000","last_change":"2024-03-06T07:22:42.293801+0000","last_active":"2024-03-06T07:22:42.298112+0000","last_peered":"2024-03-06T07:22:42.298112+0000","last_clean":"2024-03-06T07:22:42.298112+0000","last_became_active":"2024-03-06T07:22:42.293537+0000","last_became_peered":"2024-03-06T07:22:42.293537+0000","last_unstale":"2024-03-06T07:22:42.298112+0000","last_undegraded":"2024-03-06T07:22:42.298112+0000","last_fullsized":"2024-03-06T07:22:42.298112+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:05.097698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283044+0000","last_change":"2024-03-06T07:22:42.283044+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283044+0000","last_undegraded":"2024-03-06T07:22:42.283044+0000","last_fullsized":"2024-03-06T07:22:42.283044+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.289637+0000","last_change":"2024-03-06T07:22:42.287323+0000","last_active":"2024-03-06T07:22:42.289637+0000","last_peered":"2024-03-06T07:22:42.289637+0000","last_clean":"2024-03-06T07:22:42.289637+0000","last_became_active":"2024-03-06T07:22:42.287077+0000","last_became_peered":"2024-03-06T07:22:42.287077+0000","last_unstale":"2024-03-06T07:22:42.289637+0000","last_undegraded":"2024-03-06T07:22:42.289637+0000","last_fullsized":"2024-03-06T07:22:42.289637+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:24.927357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.278746+0000","last_change":"2024-03-06T07:22:42.278746+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.278746+0000","last_undegraded":"2024-03-06T07:22:42.278746+0000","last_fullsized":"2024-03-06T07:22:42.278746+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291008+0000","last_change":"2024-03-06T07:22:42.290240+0000","last_active":"2024-03-06T07:22:42.291008+0000","last_peered":"2024-03-06T07:22:42.291008+0000","last_clean":"2024-03-06T07:22:42.291008+0000","last_became_active":"2024-03-06T07:22:42.289934+0000","last_became_peered":"2024-03-06T07:22:42.289934+0000","last_unstale":"2024-03-06T07:22:42.291008+0000","last_undegraded":"2024-03-06T07:22:42.291008+0000","last_fullsized":"2024-03-06T07:22:42.291008+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:58.470531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.282416+0000","last_change":"2024-03-06T07:22:42.282416+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.282416+0000","last_undegraded":"2024-03-06T07:22:42.282416+0000","last_fullsized":"2024-03-06T07:22:42.282416+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.293325+0000","last_change":"2024-03-06T07:22:42.292354+0000","last_active":"2024-03-06T07:22:42.293325+0000","last_peered":"2024-03-06T07:22:42.293325+0000","last_clean":"2024-03-06T07:22:42.293325+0000","last_became_active":"2024-03-06T07:22:42.292033+0000","last_became_peered":"2024-03-06T07:22:42.292033+0000","last_unstale":"2024-03-06T07:22:42.293325+0000","last_undegraded":"2024-03-06T07:22:42.293325+0000","last_fullsized":"2024-03-06T07:22:42.293325+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:31.588280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.289810+0000","last_change":"2024-03-06T07:22:42.289810+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.289810+0000","last_undegraded":"2024-03-06T07:22:42.289810+0000","last_fullsized":"2024-03-06T07:22:42.289810+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291470+0000","last_change":"2024-03-06T07:22:42.288768+0000","last_active":"2024-03-06T07:22:42.291470+0000","last_peered":"2024-03-06T07:22:42.291470+0000","last_clean":"2024-03-06T07:22:42.291470+0000","last_became_active":"2024-03-06T07:22:42.288476+0000","last_became_peered":"2024-03-06T07:22:42.288476+0000","last_unstale":"2024-03-06T07:22:42.291470+0000","last_undegraded":"2024-03-06T07:22:42.291470+0000","last_fullsized":"2024-03-06T07:22:42.291470+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.908267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281207+0000","last_change":"2024-03-06T07:22:42.281207+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281207+0000","last_undegraded":"2024-03-06T07:22:42.281207+0000","last_fullsized":"2024-03-06T07:22:42.281207+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294770+0000","last_change":"2024-03-06T07:22:42.293943+0000","last_active":"2024-03-06T07:22:42.294770+0000","last_peered":"2024-03-06T07:22:42.294770+0000","last_clean":"2024-03-06T07:22:42.294770+0000","last_became_active":"2024-03-06T07:22:42.293661+0000","last_became_peered":"2024-03-06T07:22:42.293661+0000","last_unstale":"2024-03-06T07:22:42.294770+0000","last_undegraded":"2024-03-06T07:22:42.294770+0000","last_fullsized":"2024-03-06T07:22:42.294770+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:51.825409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280504+0000","last_change":"2024-03-06T07:22:42.280504+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280504+0000","last_undegraded":"2024-03-06T07:22:42.280504+0000","last_fullsized":"2024-03-06T07:22:42.280504+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291595+0000","last_change":"2024-03-06T07:22:42.287760+0000","last_active":"2024-03-06T07:22:42.291595+0000","last_peered":"2024-03-06T07:22:42.291595+0000","last_clean":"2024-03-06T07:22:42.291595+0000","last_became_active":"2024-03-06T07:22:42.287466+0000","last_became_peered":"2024-03-06T07:22:42.287466+0000","last_unstale":"2024-03-06T07:22:42.291595+0000","last_undegraded":"2024-03-06T07:22:42.291595+0000","last_fullsized":"2024-03-06T07:22:42.291595+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:48.429074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.280985+0000","last_change":"2024-03-06T07:22:42.280985+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.280985+0000","last_undegraded":"2024-03-06T07:22:42.280985+0000","last_fullsized":"2024-03-06T07:22:42.280985+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298342+0000","last_change":"2024-03-06T07:22:42.296157+0000","last_active":"2024-03-06T07:22:42.298342+0000","last_peered":"2024-03-06T07:22:42.298342+0000","last_clean":"2024-03-06T07:22:42.298342+0000","last_became_active":"2024-03-06T07:22:42.295713+0000","last_became_peered":"2024-03-06T07:22:42.295713+0000","last_unstale":"2024-03-06T07:22:42.298342+0000","last_undegraded":"2024-03-06T07:22:42.298342+0000","last_fullsized":"2024-03-06T07:22:42.298342+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:11.158468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.281379+0000","last_change":"2024-03-06T07:22:42.281379+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.281379+0000","last_undegraded":"2024-03-06T07:22:42.281379+0000","last_fullsized":"2024-03-06T07:22:42.281379+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.294125+0000","last_change":"2024-03-06T07:22:42.293654+0000","last_active":"2024-03-06T07:22:42.294125+0000","last_peered":"2024-03-06T07:22:42.294125+0000","last_clean":"2024-03-06T07:22:42.294125+0000","last_became_active":"2024-03-06T07:22:42.293361+0000","last_became_peered":"2024-03-06T07:22:42.293361+0000","last_unstale":"2024-03-06T07:22:42.294125+0000","last_undegraded":"2024-03-06T07:22:42.294125+0000","last_fullsized":"2024-03-06T07:22:42.294125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:03.485795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.290349+0000","last_change":"2024-03-06T07:22:42.290349+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.290349+0000","last_undegraded":"2024-03-06T07:22:42.290349+0000","last_fullsized":"2024-03-06T07:22:42.290349+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291039+0000","last_change":"2024-03-06T07:22:42.290440+0000","last_active":"2024-03-06T07:22:42.291039+0000","last_peered":"2024-03-06T07:22:42.291039+0000","last_clean":"2024-03-06T07:22:42.291039+0000","last_became_active":"2024-03-06T07:22:42.290162+0000","last_became_peered":"2024-03-06T07:22:42.290162+0000","last_unstale":"2024-03-06T07:22:42.291039+0000","last_undegraded":"2024-03-06T07:22:42.291039+0000","last_fullsized":"2024-03-06T07:22:42.291039+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:02.519897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":1,"reported_epoch":119,"state":"creating+peering","last_fresh":"2024-03-06T07:22:42.283399+0000","last_change":"2024-03-06T07:22:42.283399+0000","last_active":"2024-03-06T07:22:42.266236+0000","last_peered":"2024-03-06T07:22:42.266236+0000","last_clean":"2024-03-06T07:22:42.266236+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:22:42.283399+0000","last_undegraded":"2024-03-06T07:22:42.283399+0000","last_fullsized":"2024-03-06T07:22:42.283399+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.298125+0000","last_change":"2024-03-06T07:22:42.293720+0000","last_active":"2024-03-06T07:22:42.298125+0000","last_peered":"2024-03-06T07:22:42.298125+0000","last_clean":"2024-03-06T07:22:42.298125+0000","last_became_active":"2024-03-06T07:22:42.293429+0000","last_became_peered":"2024-03-06T07:22:42.293429+0000","last_unstale":"2024-03-06T07:22:42.298125+0000","last_undegraded":"2024-03-06T07:22:42.298125+0000","last_fullsized":"2024-03-06T07:22:42.298125+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:55.471196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":357,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-06T07:22:42.291293+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:22:42.291293+0000","last_peered":"2024-03-06T07:22:42.291293+0000","last_clean":"2024-03-06T07:22:42.291293+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:22:42.291293+0000","last_undegraded":"2024-03-06T07:22:42.291293+0000","last_fullsized":"2024-03-06T07:22:42.291293+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1490944,"data_stored":715572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1478656,"data_stored":708913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32692,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339148,"statfs":{"total":96636764160,"available":96603287552,"internally_reserved":0,"allocated":1544192,"data_stored":1299281,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:48.558 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:22:48.557+0000 7f3ee55a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:50.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.738+0000 7fd2f238a640 1 Processor -- start 2024-03-06T07:22:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.738+0000 7fd2f238a640 1 -- start start 2024-03-06T07:22:50.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.738+0000 7fd2f238a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd2ec14f790 0x7fd2ec1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.738+0000 7fd2f238a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd2ec155a40 0x7fd2ec155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f238a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0c28d0 con 0x7fd2ec0c1aa0 2024-03-06T07:22:50.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f238a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0c2680 con 0x7fd2ec155a40 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f238a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0c2bb0 con 0x7fd2ec14f790 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f0927640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd2ec155a40 0x7fd2ec155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2eb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd2ec0c1aa0 0x7fd2ec0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37070/0 (socket says 172.21.15.94:37070) 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2eb7fe640 1 -- 172.21.15.94:0/3244502498 learned_addr learned my addr 172.21.15.94:0/3244502498 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2ebfff640 1 --2- 172.21.15.94:0/3244502498 >> v2:172.21.15.145:3300/0 conn(0x7fd2ec14f790 0x7fd2ec1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2eaffd640 1 -- 172.21.15.94:0/3244502498 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299562818 0 0) 0x7fd2ec0c28d0 con 0x7fd2ec0c1aa0 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2eaffd640 1 -- 172.21.15.94:0/3244502498 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d4003520 con 0x7fd2ec0c1aa0 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f0927640 1 -- 172.21.15.94:0/3244502498 >> v2:172.21.15.145:3300/0 conn(0x7fd2ec14f790 msgr2=0x7fd2ec1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f0927640 1 --2- 172.21.15.94:0/3244502498 >> v2:172.21.15.145:3300/0 conn(0x7fd2ec14f790 0x7fd2ec1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f0927640 1 -- 172.21.15.94:0/3244502498 >> v1:172.21.15.94:6789/0 conn(0x7fd2ec0c1aa0 legacy=0x7fd2ec0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.739+0000 7fd2f0927640 1 -- 172.21.15.94:0/3244502498 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0c2e90 con 0x7fd2ec155a40 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.740+0000 7fd2f0927640 1 --2- 172.21.15.94:0/3244502498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec155e50 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0c3170 tx=0x7fd2e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4791251af5f2dad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.740+0000 7fd2eaffd640 1 -- 172.21.15.94:0/3244502498 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4039070 con 0x7fd2ec155a40 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.740+0000 7fd2eaffd640 1 -- 172.21.15.94:0/3244502498 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e402cc80 con 0x7fd2ec155a40 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.740+0000 7fd2eaffd640 1 -- 172.21.15.94:0/3244502498 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4034660 con 0x7fd2ec155a40 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.740+0000 7fd2f238a640 1 -- 172.21.15.94:0/3244502498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 msgr2=0x7fd2ec155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.740+0000 7fd2f238a640 1 --2- 172.21.15.94:0/3244502498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec155e50 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0c3170 tx=0x7fd2e402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.741+0000 7fd2f238a640 1 -- 172.21.15.94:0/3244502498 shutdown_connections 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.741+0000 7fd2f238a640 1 --2- 172.21.15.94:0/3244502498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec155e50 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.741+0000 7fd2f238a640 1 --2- 172.21.15.94:0/3244502498 >> v2:172.21.15.145:3300/0 conn(0x7fd2ec14f790 0x7fd2ec1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.741+0000 7fd2f238a640 1 -- 172.21.15.94:0/3244502498 >> 172.21.15.94:0/3244502498 conn(0x7fd2ec0c6710 msgr2=0x7fd2ec0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.741+0000 7fd2f238a640 1 -- 172.21.15.94:0/3244502498 shutdown_connections 2024-03-06T07:22:50.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.741+0000 7fd2f238a640 1 -- 172.21.15.94:0/3244502498 wait complete. 2024-03-06T07:22:50.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.742+0000 7fd2f238a640 1 Processor -- start 2024-03-06T07:22:50.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.742+0000 7fd2f238a640 1 -- start start 2024-03-06T07:22:50.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2ec0c1aa0 0x7fd2ec1dba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2ec14f790 0x7fd2ec14d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0c0d60 con 0x7fd2ec155a40 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec156390 con 0x7fd2ec14f790 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0c5e30 con 0x7fd2ec0c1aa0 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec150580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50930/0 (socket says 172.21.15.94:50930) 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 -- 172.21.15.94:0/112145055 learned_addr learned my addr 172.21.15.94:0/112145055 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2ebfff640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2ec0c1aa0 0x7fd2ec1dba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2eb7fe640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2ec14f790 0x7fd2ec14d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 -- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2ec0c1aa0 msgr2=0x7fd2ec1dba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2ec0c1aa0 0x7fd2ec1dba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 -- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2ec14f790 msgr2=0x7fd2ec14d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2ec14f790 0x7fd2ec14d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 -- 172.21.15.94:0/112145055 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec14d6d0 con 0x7fd2ec155a40 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f0927640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec150580 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009e80 tx=0x7fd2e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2e8ff9640 1 -- 172.21.15.94:0/112145055 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4039050 con 0x7fd2ec155a40 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2e8ff9640 1 -- 172.21.15.94:0/112145055 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e4045070 con 0x7fd2ec155a40 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2e8ff9640 1 -- 172.21.15.94:0/112145055 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4009450 con 0x7fd2ec155a40 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec1dfe40 con 0x7fd2ec155a40 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.743+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec1e0100 con 0x7fd2ec155a40 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.745+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec1dfc30 con 0x7fd2ec155a40 2024-03-06T07:22:50.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.745+0000 7fd2e8ff9640 1 -- 172.21.15.94:0/112145055 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2e403e850 con 0x7fd2ec155a40 2024-03-06T07:22:50.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.745+0000 7fd2e8ff9640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd2d806b4d0 0x7fd2d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.745+0000 7fd2e8ff9640 1 -- 172.21.15.94:0/112145055 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2e40ad0e0 con 0x7fd2ec155a40 2024-03-06T07:22:50.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.745+0000 7fd2ebfff640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd2d806b4d0 0x7fd2d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.746+0000 7fd2ebfff640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd2d806b4d0 0x7fd2d806d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec067cb0 tx=0x7fd2dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.750+0000 7fd2e8ff9640 1 -- 172.21.15.94:0/112145055 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2e407b930 con 0x7fd2ec155a40 2024-03-06T07:22:50.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.918+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2ec1db740 con 0x7fd2d806b4d0 2024-03-06T07:22:50.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.924+0000 7fd2e8ff9640 1 -- 172.21.15.94:0/112145055 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252563 (secure 0 0 0) 0x7fd2dc011a30 con 0x7fd2d806b4d0 2024-03-06T07:22:50.933 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:50.934 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:22:50.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd2d806b4d0 msgr2=0x7fd2d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd2d806b4d0 0x7fd2d806d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec067cb0 tx=0x7fd2dc009190 comp rx=0 tx=0).stop 2024-03-06T07:22:50.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 msgr2=0x7fd2ec150580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec150580 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009e80 tx=0x7fd2e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:50.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 shutdown_connections 2024-03-06T07:22:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd2d806b4d0 0x7fd2d806d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2ec155a40 0x7fd2ec150580 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2ec14f790 0x7fd2ec14d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 --2- 172.21.15.94:0/112145055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2ec0c1aa0 0x7fd2ec1dba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 >> 172.21.15.94:0/112145055 conn(0x7fd2ec0c6710 msgr2=0x7fd2ec0c4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 shutdown_connections 2024-03-06T07:22:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:50.927+0000 7fd2f238a640 1 -- 172.21.15.94:0/112145055 wait complete. 2024-03-06T07:22:50.940 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-03-06T07:22:50.694168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261480,"kb_used_data":12032,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713240,"statfs":{"total":773094113280,"available":772826357760,"internally_reserved":0,"allocated":12320768,"data_stored":7021004,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001740"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347652+0000","last_change":"2024-03-06T07:22:43.293116+0000","last_active":"2024-03-06T07:22:44.347652+0000","last_peered":"2024-03-06T07:22:44.347652+0000","last_clean":"2024-03-06T07:22:44.347652+0000","last_became_active":"2024-03-06T07:22:43.292511+0000","last_became_peered":"2024-03-06T07:22:43.292511+0000","last_unstale":"2024-03-06T07:22:44.347652+0000","last_undegraded":"2024-03-06T07:22:44.347652+0000","last_fullsized":"2024-03-06T07:22:44.347652+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:39.829479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293721+0000","last_change":"2024-03-06T07:22:43.304899+0000","last_active":"2024-03-06T07:22:44.293721+0000","last_peered":"2024-03-06T07:22:44.293721+0000","last_clean":"2024-03-06T07:22:44.293721+0000","last_became_active":"2024-03-06T07:22:43.304588+0000","last_became_peered":"2024-03-06T07:22:43.304588+0000","last_unstale":"2024-03-06T07:22:44.293721+0000","last_undegraded":"2024-03-06T07:22:44.293721+0000","last_fullsized":"2024-03-06T07:22:44.293721+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:38.128097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.288064+0000","last_change":"2024-03-06T07:22:43.296475+0000","last_active":"2024-03-06T07:22:44.288064+0000","last_peered":"2024-03-06T07:22:44.288064+0000","last_clean":"2024-03-06T07:22:44.288064+0000","last_became_active":"2024-03-06T07:22:43.296212+0000","last_became_peered":"2024-03-06T07:22:43.296212+0000","last_unstale":"2024-03-06T07:22:44.288064+0000","last_undegraded":"2024-03-06T07:22:44.288064+0000","last_fullsized":"2024-03-06T07:22:44.288064+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:17.054707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293466+0000","last_change":"2024-03-06T07:22:43.302442+0000","last_active":"2024-03-06T07:22:44.293466+0000","last_peered":"2024-03-06T07:22:44.293466+0000","last_clean":"2024-03-06T07:22:44.293466+0000","last_became_active":"2024-03-06T07:22:43.302152+0000","last_became_peered":"2024-03-06T07:22:43.302152+0000","last_unstale":"2024-03-06T07:22:44.293466+0000","last_undegraded":"2024-03-06T07:22:44.293466+0000","last_fullsized":"2024-03-06T07:22:44.293466+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:09.146533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347053+0000","last_change":"2024-03-06T07:22:43.291444+0000","last_active":"2024-03-06T07:22:44.347053+0000","last_peered":"2024-03-06T07:22:44.347053+0000","last_clean":"2024-03-06T07:22:44.347053+0000","last_became_active":"2024-03-06T07:22:43.291140+0000","last_became_peered":"2024-03-06T07:22:43.291140+0000","last_unstale":"2024-03-06T07:22:44.347053+0000","last_undegraded":"2024-03-06T07:22:44.347053+0000","last_fullsized":"2024-03-06T07:22:44.347053+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.890883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347527+0000","last_change":"2024-03-06T07:22:43.289234+0000","last_active":"2024-03-06T07:22:44.347527+0000","last_peered":"2024-03-06T07:22:44.347527+0000","last_clean":"2024-03-06T07:22:44.347527+0000","last_became_active":"2024-03-06T07:22:43.288768+0000","last_became_peered":"2024-03-06T07:22:43.288768+0000","last_unstale":"2024-03-06T07:22:44.347527+0000","last_undegraded":"2024-03-06T07:22:44.347527+0000","last_fullsized":"2024-03-06T07:22:44.347527+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:23.015403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346375+0000","last_change":"2024-03-06T07:22:43.298550+0000","last_active":"2024-03-06T07:22:44.346375+0000","last_peered":"2024-03-06T07:22:44.346375+0000","last_clean":"2024-03-06T07:22:44.346375+0000","last_became_active":"2024-03-06T07:22:43.298289+0000","last_became_peered":"2024-03-06T07:22:43.298289+0000","last_unstale":"2024-03-06T07:22:44.346375+0000","last_undegraded":"2024-03-06T07:22:44.346375+0000","last_fullsized":"2024-03-06T07:22:44.346375+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:18.223471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.286983+0000","last_change":"2024-03-06T07:22:43.295190+0000","last_active":"2024-03-06T07:22:44.286983+0000","last_peered":"2024-03-06T07:22:44.286983+0000","last_clean":"2024-03-06T07:22:44.286983+0000","last_became_active":"2024-03-06T07:22:43.294878+0000","last_became_peered":"2024-03-06T07:22:43.294878+0000","last_unstale":"2024-03-06T07:22:44.286983+0000","last_undegraded":"2024-03-06T07:22:44.286983+0000","last_fullsized":"2024-03-06T07:22:44.286983+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:08.584954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.348308+0000","last_change":"2024-03-06T07:22:43.290943+0000","last_active":"2024-03-06T07:22:44.348308+0000","last_peered":"2024-03-06T07:22:44.348308+0000","last_clean":"2024-03-06T07:22:44.348308+0000","last_became_active":"2024-03-06T07:22:43.290630+0000","last_became_peered":"2024-03-06T07:22:43.290630+0000","last_unstale":"2024-03-06T07:22:44.348308+0000","last_undegraded":"2024-03-06T07:22:44.348308+0000","last_fullsized":"2024-03-06T07:22:44.348308+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:03.524449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.285960+0000","last_change":"2024-03-06T07:22:43.291737+0000","last_active":"2024-03-06T07:22:44.285960+0000","last_peered":"2024-03-06T07:22:44.285960+0000","last_clean":"2024-03-06T07:22:44.285960+0000","last_became_active":"2024-03-06T07:22:43.291468+0000","last_became_peered":"2024-03-06T07:22:43.291468+0000","last_unstale":"2024-03-06T07:22:44.285960+0000","last_undegraded":"2024-03-06T07:22:44.285960+0000","last_fullsized":"2024-03-06T07:22:44.285960+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.529739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293521+0000","last_change":"2024-03-06T07:22:43.303944+0000","last_active":"2024-03-06T07:22:44.293521+0000","last_peered":"2024-03-06T07:22:44.293521+0000","last_clean":"2024-03-06T07:22:44.293521+0000","last_became_active":"2024-03-06T07:22:43.303402+0000","last_became_peered":"2024-03-06T07:22:43.303402+0000","last_unstale":"2024-03-06T07:22:44.293521+0000","last_undegraded":"2024-03-06T07:22:44.293521+0000","last_fullsized":"2024-03-06T07:22:44.293521+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:23.015403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347441+0000","last_change":"2024-03-06T07:22:43.294176+0000","last_active":"2024-03-06T07:22:44.347441+0000","last_peered":"2024-03-06T07:22:44.347441+0000","last_clean":"2024-03-06T07:22:44.347441+0000","last_became_active":"2024-03-06T07:22:43.293892+0000","last_became_peered":"2024-03-06T07:22:43.293892+0000","last_unstale":"2024-03-06T07:22:44.347441+0000","last_undegraded":"2024-03-06T07:22:44.347441+0000","last_fullsized":"2024-03-06T07:22:44.347441+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:47.320044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347691+0000","last_change":"2024-03-06T07:22:43.289400+0000","last_active":"2024-03-06T07:22:44.347691+0000","last_peered":"2024-03-06T07:22:44.347691+0000","last_clean":"2024-03-06T07:22:44.347691+0000","last_became_active":"2024-03-06T07:22:43.289140+0000","last_became_peered":"2024-03-06T07:22:43.289140+0000","last_unstale":"2024-03-06T07:22:44.347691+0000","last_undegraded":"2024-03-06T07:22:44.347691+0000","last_fullsized":"2024-03-06T07:22:44.347691+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:06.793096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347501+0000","last_change":"2024-03-06T07:22:43.298870+0000","last_active":"2024-03-06T07:22:44.347501+0000","last_peered":"2024-03-06T07:22:44.347501+0000","last_clean":"2024-03-06T07:22:44.347501+0000","last_became_active":"2024-03-06T07:22:43.298611+0000","last_became_peered":"2024-03-06T07:22:43.298611+0000","last_unstale":"2024-03-06T07:22:44.347501+0000","last_undegraded":"2024-03-06T07:22:44.347501+0000","last_fullsized":"2024-03-06T07:22:44.347501+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:47.561749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346631+0000","last_change":"2024-03-06T07:22:43.298069+0000","last_active":"2024-03-06T07:22:44.346631+0000","last_peered":"2024-03-06T07:22:44.346631+0000","last_clean":"2024-03-06T07:22:44.346631+0000","last_became_active":"2024-03-06T07:22:43.297541+0000","last_became_peered":"2024-03-06T07:22:43.297541+0000","last_unstale":"2024-03-06T07:22:44.346631+0000","last_undegraded":"2024-03-06T07:22:44.346631+0000","last_fullsized":"2024-03-06T07:22:44.346631+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:09.887337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347803+0000","last_change":"2024-03-06T07:22:43.287777+0000","last_active":"2024-03-06T07:22:44.347803+0000","last_peered":"2024-03-06T07:22:44.347803+0000","last_clean":"2024-03-06T07:22:44.347803+0000","last_became_active":"2024-03-06T07:22:43.287480+0000","last_became_peered":"2024-03-06T07:22:43.287480+0000","last_unstale":"2024-03-06T07:22:44.347803+0000","last_undegraded":"2024-03-06T07:22:44.347803+0000","last_fullsized":"2024-03-06T07:22:44.347803+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:57.686946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.348361+0000","last_change":"2024-03-06T07:22:43.287189+0000","last_active":"2024-03-06T07:22:44.348361+0000","last_peered":"2024-03-06T07:22:44.348361+0000","last_clean":"2024-03-06T07:22:44.348361+0000","last_became_active":"2024-03-06T07:22:43.286899+0000","last_became_peered":"2024-03-06T07:22:43.286899+0000","last_unstale":"2024-03-06T07:22:44.348361+0000","last_undegraded":"2024-03-06T07:22:44.348361+0000","last_fullsized":"2024-03-06T07:22:44.348361+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:37.841013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346995+0000","last_change":"2024-03-06T07:22:43.294218+0000","last_active":"2024-03-06T07:22:44.346995+0000","last_peered":"2024-03-06T07:22:44.346995+0000","last_clean":"2024-03-06T07:22:44.346995+0000","last_became_active":"2024-03-06T07:22:43.293944+0000","last_became_peered":"2024-03-06T07:22:43.293944+0000","last_unstale":"2024-03-06T07:22:44.346995+0000","last_undegraded":"2024-03-06T07:22:44.346995+0000","last_fullsized":"2024-03-06T07:22:44.346995+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.529739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346802+0000","last_change":"2024-03-06T07:22:43.297564+0000","last_active":"2024-03-06T07:22:44.346802+0000","last_peered":"2024-03-06T07:22:44.346802+0000","last_clean":"2024-03-06T07:22:44.346802+0000","last_became_active":"2024-03-06T07:22:43.297309+0000","last_became_peered":"2024-03-06T07:22:43.297309+0000","last_unstale":"2024-03-06T07:22:44.346802+0000","last_undegraded":"2024-03-06T07:22:44.346802+0000","last_fullsized":"2024-03-06T07:22:44.346802+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:19.987690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347340+0000","last_change":"2024-03-06T07:22:43.304843+0000","last_active":"2024-03-06T07:22:44.347340+0000","last_peered":"2024-03-06T07:22:44.347340+0000","last_clean":"2024-03-06T07:22:44.347340+0000","last_became_active":"2024-03-06T07:22:43.304588+0000","last_became_peered":"2024-03-06T07:22:43.304588+0000","last_unstale":"2024-03-06T07:22:44.347340+0000","last_undegraded":"2024-03-06T07:22:44.347340+0000","last_fullsized":"2024-03-06T07:22:44.347340+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:48.773566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293572+0000","last_change":"2024-03-06T07:22:43.299902+0000","last_active":"2024-03-06T07:22:44.293572+0000","last_peered":"2024-03-06T07:22:44.293572+0000","last_clean":"2024-03-06T07:22:44.293572+0000","last_became_active":"2024-03-06T07:22:43.299454+0000","last_became_peered":"2024-03-06T07:22:43.299454+0000","last_unstale":"2024-03-06T07:22:44.293572+0000","last_undegraded":"2024-03-06T07:22:44.293572+0000","last_fullsized":"2024-03-06T07:22:44.293572+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:06.793096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347359+0000","last_change":"2024-03-06T07:22:43.302055+0000","last_active":"2024-03-06T07:22:44.347359+0000","last_peered":"2024-03-06T07:22:44.347359+0000","last_clean":"2024-03-06T07:22:44.347359+0000","last_became_active":"2024-03-06T07:22:43.301744+0000","last_became_peered":"2024-03-06T07:22:43.301744+0000","last_unstale":"2024-03-06T07:22:44.347359+0000","last_undegraded":"2024-03-06T07:22:44.347359+0000","last_fullsized":"2024-03-06T07:22:44.347359+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:17.451911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293349+0000","last_change":"2024-03-06T07:22:43.302772+0000","last_active":"2024-03-06T07:22:44.293349+0000","last_peered":"2024-03-06T07:22:44.293349+0000","last_clean":"2024-03-06T07:22:44.293349+0000","last_became_active":"2024-03-06T07:22:43.302496+0000","last_became_peered":"2024-03-06T07:22:43.302496+0000","last_unstale":"2024-03-06T07:22:44.293349+0000","last_undegraded":"2024-03-06T07:22:44.293349+0000","last_fullsized":"2024-03-06T07:22:44.293349+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:25.111536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347937+0000","last_change":"2024-03-06T07:22:43.288476+0000","last_active":"2024-03-06T07:22:44.347937+0000","last_peered":"2024-03-06T07:22:44.347937+0000","last_clean":"2024-03-06T07:22:44.347937+0000","last_became_active":"2024-03-06T07:22:43.288233+0000","last_became_peered":"2024-03-06T07:22:43.288233+0000","last_unstale":"2024-03-06T07:22:44.347937+0000","last_undegraded":"2024-03-06T07:22:44.347937+0000","last_fullsized":"2024-03-06T07:22:44.347937+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:29.754664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287383+0000","last_change":"2024-03-06T07:22:43.284220+0000","last_active":"2024-03-06T07:22:44.287383+0000","last_peered":"2024-03-06T07:22:44.287383+0000","last_clean":"2024-03-06T07:22:44.287383+0000","last_became_active":"2024-03-06T07:22:43.283941+0000","last_became_peered":"2024-03-06T07:22:43.283941+0000","last_unstale":"2024-03-06T07:22:44.287383+0000","last_undegraded":"2024-03-06T07:22:44.287383+0000","last_fullsized":"2024-03-06T07:22:44.287383+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:49.433625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287780+0000","last_change":"2024-03-06T07:22:43.298793+0000","last_active":"2024-03-06T07:22:44.287780+0000","last_peered":"2024-03-06T07:22:44.287780+0000","last_clean":"2024-03-06T07:22:44.287780+0000","last_became_active":"2024-03-06T07:22:43.298369+0000","last_became_peered":"2024-03-06T07:22:43.298369+0000","last_unstale":"2024-03-06T07:22:44.287780+0000","last_undegraded":"2024-03-06T07:22:44.287780+0000","last_fullsized":"2024-03-06T07:22:44.287780+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:04.405418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346878+0000","last_change":"2024-03-06T07:22:43.302827+0000","last_active":"2024-03-06T07:22:44.346878+0000","last_peered":"2024-03-06T07:22:44.346878+0000","last_clean":"2024-03-06T07:22:44.346878+0000","last_became_active":"2024-03-06T07:22:43.302548+0000","last_became_peered":"2024-03-06T07:22:43.302548+0000","last_unstale":"2024-03-06T07:22:44.346878+0000","last_undegraded":"2024-03-06T07:22:44.346878+0000","last_fullsized":"2024-03-06T07:22:44.346878+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.823704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.286055+0000","last_change":"2024-03-06T07:22:43.296272+0000","last_active":"2024-03-06T07:22:44.286055+0000","last_peered":"2024-03-06T07:22:44.286055+0000","last_clean":"2024-03-06T07:22:44.286055+0000","last_became_active":"2024-03-06T07:22:43.295994+0000","last_became_peered":"2024-03-06T07:22:43.295994+0000","last_unstale":"2024-03-06T07:22:44.286055+0000","last_undegraded":"2024-03-06T07:22:44.286055+0000","last_fullsized":"2024-03-06T07:22:44.286055+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.823704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.288450+0000","last_change":"2024-03-06T07:22:43.289012+0000","last_active":"2024-03-06T07:22:44.288450+0000","last_peered":"2024-03-06T07:22:44.288450+0000","last_clean":"2024-03-06T07:22:44.288450+0000","last_became_active":"2024-03-06T07:22:43.288099+0000","last_became_peered":"2024-03-06T07:22:43.288099+0000","last_unstale":"2024-03-06T07:22:44.288450+0000","last_undegraded":"2024-03-06T07:22:44.288450+0000","last_fullsized":"2024-03-06T07:22:44.288450+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:52.240403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287174+0000","last_change":"2024-03-06T07:22:43.290590+0000","last_active":"2024-03-06T07:22:44.287174+0000","last_peered":"2024-03-06T07:22:44.287174+0000","last_clean":"2024-03-06T07:22:44.287174+0000","last_became_active":"2024-03-06T07:22:43.290289+0000","last_became_peered":"2024-03-06T07:22:43.290289+0000","last_unstale":"2024-03-06T07:22:44.287174+0000","last_undegraded":"2024-03-06T07:22:44.287174+0000","last_fullsized":"2024-03-06T07:22:44.287174+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:31.366834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.286917+0000","last_change":"2024-03-06T07:22:43.286496+0000","last_active":"2024-03-06T07:22:44.286917+0000","last_peered":"2024-03-06T07:22:44.286917+0000","last_clean":"2024-03-06T07:22:44.286917+0000","last_became_active":"2024-03-06T07:22:43.285036+0000","last_became_peered":"2024-03-06T07:22:43.285036+0000","last_unstale":"2024-03-06T07:22:44.286917+0000","last_undegraded":"2024-03-06T07:22:44.286917+0000","last_fullsized":"2024-03-06T07:22:44.286917+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:31.366834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.285858+0000","last_change":"2024-03-06T07:22:43.290505+0000","last_active":"2024-03-06T07:22:44.285858+0000","last_peered":"2024-03-06T07:22:44.285858+0000","last_clean":"2024-03-06T07:22:44.285858+0000","last_became_active":"2024-03-06T07:22:43.290179+0000","last_became_peered":"2024-03-06T07:22:43.290179+0000","last_unstale":"2024-03-06T07:22:44.285858+0000","last_undegraded":"2024-03-06T07:22:44.285858+0000","last_fullsized":"2024-03-06T07:22:44.285858+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:06.681371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287412+0000","last_change":"2024-03-06T07:22:43.287912+0000","last_active":"2024-03-06T07:22:44.287412+0000","last_peered":"2024-03-06T07:22:44.287412+0000","last_clean":"2024-03-06T07:22:44.287412+0000","last_became_active":"2024-03-06T07:22:43.287648+0000","last_became_peered":"2024-03-06T07:22:43.287648+0000","last_unstale":"2024-03-06T07:22:44.287412+0000","last_undegraded":"2024-03-06T07:22:44.287412+0000","last_fullsized":"2024-03-06T07:22:44.287412+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.748569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"120'4","reported_seq":18,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287492+0000","last_change":"2024-03-06T07:22:42.289168+0000","last_active":"2024-03-06T07:22:44.287492+0000","last_peered":"2024-03-06T07:22:44.287492+0000","last_clean":"2024-03-06T07:22:44.287492+0000","last_became_active":"2024-03-06T07:22:42.288869+0000","last_became_peered":"2024-03-06T07:22:42.288869+0000","last_unstale":"2024-03-06T07:22:44.287492+0000","last_undegraded":"2024-03-06T07:22:44.287492+0000","last_fullsized":"2024-03-06T07:22:44.287492+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:31.588280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346709+0000","last_change":"2024-03-06T07:22:43.294585+0000","last_active":"2024-03-06T07:22:44.346709+0000","last_peered":"2024-03-06T07:22:44.346709+0000","last_clean":"2024-03-06T07:22:44.346709+0000","last_became_active":"2024-03-06T07:22:43.294307+0000","last_became_peered":"2024-03-06T07:22:43.294307+0000","last_unstale":"2024-03-06T07:22:44.346709+0000","last_undegraded":"2024-03-06T07:22:44.346709+0000","last_fullsized":"2024-03-06T07:22:44.346709+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:23.619936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347403+0000","last_change":"2024-03-06T07:22:42.284230+0000","last_active":"2024-03-06T07:22:44.347403+0000","last_peered":"2024-03-06T07:22:44.347403+0000","last_clean":"2024-03-06T07:22:44.347403+0000","last_became_active":"2024-03-06T07:22:42.283796+0000","last_became_peered":"2024-03-06T07:22:42.283796+0000","last_unstale":"2024-03-06T07:22:44.347403+0000","last_undegraded":"2024-03-06T07:22:44.347403+0000","last_fullsized":"2024-03-06T07:22:44.347403+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:31.240192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293499+0000","last_change":"2024-03-06T07:22:43.302820+0000","last_active":"2024-03-06T07:22:44.293499+0000","last_peered":"2024-03-06T07:22:44.293499+0000","last_clean":"2024-03-06T07:22:44.293499+0000","last_became_active":"2024-03-06T07:22:43.302549+0000","last_became_peered":"2024-03-06T07:22:43.302549+0000","last_unstale":"2024-03-06T07:22:44.293499+0000","last_undegraded":"2024-03-06T07:22:44.293499+0000","last_fullsized":"2024-03-06T07:22:44.293499+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:07.360370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346901+0000","last_change":"2024-03-06T07:22:42.298123+0000","last_active":"2024-03-06T07:22:44.346901+0000","last_peered":"2024-03-06T07:22:44.346901+0000","last_clean":"2024-03-06T07:22:44.346901+0000","last_became_active":"2024-03-06T07:22:42.297656+0000","last_became_peered":"2024-03-06T07:22:42.297656+0000","last_unstale":"2024-03-06T07:22:44.346901+0000","last_undegraded":"2024-03-06T07:22:44.346901+0000","last_fullsized":"2024-03-06T07:22:44.346901+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:05.788544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346915+0000","last_change":"2024-03-06T07:22:43.297740+0000","last_active":"2024-03-06T07:22:44.346915+0000","last_peered":"2024-03-06T07:22:44.346915+0000","last_clean":"2024-03-06T07:22:44.346915+0000","last_became_active":"2024-03-06T07:22:43.297437+0000","last_became_peered":"2024-03-06T07:22:43.297437+0000","last_unstale":"2024-03-06T07:22:44.346915+0000","last_undegraded":"2024-03-06T07:22:44.346915+0000","last_fullsized":"2024-03-06T07:22:44.346915+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:34.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346506+0000","last_change":"2024-03-06T07:22:42.295819+0000","last_active":"2024-03-06T07:22:44.346506+0000","last_peered":"2024-03-06T07:22:44.346506+0000","last_clean":"2024-03-06T07:22:44.346506+0000","last_became_active":"2024-03-06T07:22:42.295541+0000","last_became_peered":"2024-03-06T07:22:42.295541+0000","last_unstale":"2024-03-06T07:22:44.346506+0000","last_undegraded":"2024-03-06T07:22:44.346506+0000","last_fullsized":"2024-03-06T07:22:44.346506+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:41.697228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287513+0000","last_change":"2024-03-06T07:22:43.289135+0000","last_active":"2024-03-06T07:22:44.287513+0000","last_peered":"2024-03-06T07:22:44.287513+0000","last_clean":"2024-03-06T07:22:44.287513+0000","last_became_active":"2024-03-06T07:22:43.288885+0000","last_became_peered":"2024-03-06T07:22:43.288885+0000","last_unstale":"2024-03-06T07:22:44.287513+0000","last_undegraded":"2024-03-06T07:22:44.287513+0000","last_fullsized":"2024-03-06T07:22:44.287513+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:04.405418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287705+0000","last_change":"2024-03-06T07:22:42.287120+0000","last_active":"2024-03-06T07:22:44.287705+0000","last_peered":"2024-03-06T07:22:44.287705+0000","last_clean":"2024-03-06T07:22:44.287705+0000","last_became_active":"2024-03-06T07:22:42.286844+0000","last_became_peered":"2024-03-06T07:22:42.286844+0000","last_unstale":"2024-03-06T07:22:44.287705+0000","last_undegraded":"2024-03-06T07:22:44.287705+0000","last_fullsized":"2024-03-06T07:22:44.287705+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:21.791487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287170+0000","last_change":"2024-03-06T07:22:43.295812+0000","last_active":"2024-03-06T07:22:44.287170+0000","last_peered":"2024-03-06T07:22:44.287170+0000","last_clean":"2024-03-06T07:22:44.287170+0000","last_became_active":"2024-03-06T07:22:43.295545+0000","last_became_peered":"2024-03-06T07:22:43.295545+0000","last_unstale":"2024-03-06T07:22:44.287170+0000","last_undegraded":"2024-03-06T07:22:44.287170+0000","last_fullsized":"2024-03-06T07:22:44.287170+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:49.433625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346932+0000","last_change":"2024-03-06T07:22:42.289388+0000","last_active":"2024-03-06T07:22:44.346932+0000","last_peered":"2024-03-06T07:22:44.346932+0000","last_clean":"2024-03-06T07:22:44.346932+0000","last_became_active":"2024-03-06T07:22:42.288148+0000","last_became_peered":"2024-03-06T07:22:42.288148+0000","last_unstale":"2024-03-06T07:22:44.346932+0000","last_undegraded":"2024-03-06T07:22:44.346932+0000","last_fullsized":"2024-03-06T07:22:44.346932+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:46.185571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347162+0000","last_change":"2024-03-06T07:22:43.303528+0000","last_active":"2024-03-06T07:22:44.347162+0000","last_peered":"2024-03-06T07:22:44.347162+0000","last_clean":"2024-03-06T07:22:44.347162+0000","last_became_active":"2024-03-06T07:22:43.303231+0000","last_became_peered":"2024-03-06T07:22:43.303231+0000","last_unstale":"2024-03-06T07:22:44.347162+0000","last_undegraded":"2024-03-06T07:22:44.347162+0000","last_fullsized":"2024-03-06T07:22:44.347162+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:33.503886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346809+0000","last_change":"2024-03-06T07:22:42.285183+0000","last_active":"2024-03-06T07:22:44.346809+0000","last_peered":"2024-03-06T07:22:44.346809+0000","last_clean":"2024-03-06T07:22:44.346809+0000","last_became_active":"2024-03-06T07:22:42.284924+0000","last_became_peered":"2024-03-06T07:22:42.284924+0000","last_unstale":"2024-03-06T07:22:44.346809+0000","last_undegraded":"2024-03-06T07:22:44.346809+0000","last_fullsized":"2024-03-06T07:22:44.346809+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:40:30.362283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346703+0000","last_change":"2024-03-06T07:22:43.301596+0000","last_active":"2024-03-06T07:22:44.346703+0000","last_peered":"2024-03-06T07:22:44.346703+0000","last_clean":"2024-03-06T07:22:44.346703+0000","last_became_active":"2024-03-06T07:22:43.300378+0000","last_became_peered":"2024-03-06T07:22:43.300378+0000","last_unstale":"2024-03-06T07:22:44.346703+0000","last_undegraded":"2024-03-06T07:22:44.346703+0000","last_fullsized":"2024-03-06T07:22:44.346703+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:44.542169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.288277+0000","last_change":"2024-03-06T07:22:42.289621+0000","last_active":"2024-03-06T07:22:44.288277+0000","last_peered":"2024-03-06T07:22:44.288277+0000","last_clean":"2024-03-06T07:22:44.288277+0000","last_became_active":"2024-03-06T07:22:42.289239+0000","last_became_peered":"2024-03-06T07:22:42.289239+0000","last_unstale":"2024-03-06T07:22:44.288277+0000","last_undegraded":"2024-03-06T07:22:44.288277+0000","last_fullsized":"2024-03-06T07:22:44.288277+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:38.908267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346533+0000","last_change":"2024-03-06T07:22:43.288991+0000","last_active":"2024-03-06T07:22:44.346533+0000","last_peered":"2024-03-06T07:22:44.346533+0000","last_clean":"2024-03-06T07:22:44.346533+0000","last_became_active":"2024-03-06T07:22:43.288710+0000","last_became_peered":"2024-03-06T07:22:43.288710+0000","last_unstale":"2024-03-06T07:22:44.346533+0000","last_undegraded":"2024-03-06T07:22:44.346533+0000","last_fullsized":"2024-03-06T07:22:44.346533+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:38.086720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293685+0000","last_change":"2024-03-06T07:22:42.287554+0000","last_active":"2024-03-06T07:22:44.293685+0000","last_peered":"2024-03-06T07:22:44.293685+0000","last_clean":"2024-03-06T07:22:44.293685+0000","last_became_active":"2024-03-06T07:22:42.285213+0000","last_became_peered":"2024-03-06T07:22:42.285213+0000","last_unstale":"2024-03-06T07:22:44.293685+0000","last_undegraded":"2024-03-06T07:22:44.293685+0000","last_fullsized":"2024-03-06T07:22:44.293685+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:03.400867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346968+0000","last_change":"2024-03-06T07:22:43.291903+0000","last_active":"2024-03-06T07:22:44.346968+0000","last_peered":"2024-03-06T07:22:44.346968+0000","last_clean":"2024-03-06T07:22:44.346968+0000","last_became_active":"2024-03-06T07:22:43.291620+0000","last_became_peered":"2024-03-06T07:22:43.291620+0000","last_unstale":"2024-03-06T07:22:44.346968+0000","last_undegraded":"2024-03-06T07:22:44.346968+0000","last_fullsized":"2024-03-06T07:22:44.346968+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:07.360370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347555+0000","last_change":"2024-03-06T07:22:42.286796+0000","last_active":"2024-03-06T07:22:44.347555+0000","last_peered":"2024-03-06T07:22:44.347555+0000","last_clean":"2024-03-06T07:22:44.347555+0000","last_became_active":"2024-03-06T07:22:42.286558+0000","last_became_peered":"2024-03-06T07:22:42.286558+0000","last_unstale":"2024-03-06T07:22:44.347555+0000","last_undegraded":"2024-03-06T07:22:44.347555+0000","last_fullsized":"2024-03-06T07:22:44.347555+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:31.464891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287337+0000","last_change":"2024-03-06T07:22:43.301325+0000","last_active":"2024-03-06T07:22:44.287337+0000","last_peered":"2024-03-06T07:22:44.287337+0000","last_clean":"2024-03-06T07:22:44.287337+0000","last_became_active":"2024-03-06T07:22:43.301063+0000","last_became_peered":"2024-03-06T07:22:43.301063+0000","last_unstale":"2024-03-06T07:22:44.287337+0000","last_undegraded":"2024-03-06T07:22:44.287337+0000","last_fullsized":"2024-03-06T07:22:44.287337+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:08.584954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"120'3","reported_seq":17,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346984+0000","last_change":"2024-03-06T07:22:42.298748+0000","last_active":"2024-03-06T07:22:44.346984+0000","last_peered":"2024-03-06T07:22:44.346984+0000","last_clean":"2024-03-06T07:22:44.346984+0000","last_became_active":"2024-03-06T07:22:42.298459+0000","last_became_peered":"2024-03-06T07:22:42.298459+0000","last_unstale":"2024-03-06T07:22:44.346984+0000","last_undegraded":"2024-03-06T07:22:44.346984+0000","last_fullsized":"2024-03-06T07:22:44.346984+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:56.682394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.288520+0000","last_change":"2024-03-06T07:22:43.294398+0000","last_active":"2024-03-06T07:22:44.288520+0000","last_peered":"2024-03-06T07:22:44.288520+0000","last_clean":"2024-03-06T07:22:44.288520+0000","last_became_active":"2024-03-06T07:22:43.293436+0000","last_became_peered":"2024-03-06T07:22:43.293436+0000","last_unstale":"2024-03-06T07:22:44.288520+0000","last_undegraded":"2024-03-06T07:22:44.288520+0000","last_fullsized":"2024-03-06T07:22:44.288520+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:12.163019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"120'6","reported_seq":20,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346666+0000","last_change":"2024-03-06T07:22:42.297582+0000","last_active":"2024-03-06T07:22:44.346666+0000","last_peered":"2024-03-06T07:22:44.346666+0000","last_clean":"2024-03-06T07:22:44.346666+0000","last_became_active":"2024-03-06T07:22:42.297309+0000","last_became_peered":"2024-03-06T07:22:42.297309+0000","last_unstale":"2024-03-06T07:22:44.346666+0000","last_undegraded":"2024-03-06T07:22:44.346666+0000","last_fullsized":"2024-03-06T07:22:44.346666+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:08.141982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.286115+0000","last_change":"2024-03-06T07:22:43.291171+0000","last_active":"2024-03-06T07:22:44.286115+0000","last_peered":"2024-03-06T07:22:44.286115+0000","last_clean":"2024-03-06T07:22:44.286115+0000","last_became_active":"2024-03-06T07:22:43.290867+0000","last_became_peered":"2024-03-06T07:22:43.290867+0000","last_unstale":"2024-03-06T07:22:44.286115+0000","last_undegraded":"2024-03-06T07:22:44.286115+0000","last_fullsized":"2024-03-06T07:22:44.286115+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:33.503886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347013+0000","last_change":"2024-03-06T07:22:42.287402+0000","last_active":"2024-03-06T07:22:44.347013+0000","last_peered":"2024-03-06T07:22:44.347013+0000","last_clean":"2024-03-06T07:22:44.347013+0000","last_became_active":"2024-03-06T07:22:42.287169+0000","last_became_peered":"2024-03-06T07:22:42.287169+0000","last_unstale":"2024-03-06T07:22:44.347013+0000","last_undegraded":"2024-03-06T07:22:44.347013+0000","last_fullsized":"2024-03-06T07:22:44.347013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:06.576650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346864+0000","last_change":"2024-03-06T07:22:43.303534+0000","last_active":"2024-03-06T07:22:44.346864+0000","last_peered":"2024-03-06T07:22:44.346864+0000","last_clean":"2024-03-06T07:22:44.346864+0000","last_became_active":"2024-03-06T07:22:43.303259+0000","last_became_peered":"2024-03-06T07:22:43.303259+0000","last_unstale":"2024-03-06T07:22:44.346864+0000","last_undegraded":"2024-03-06T07:22:44.346864+0000","last_fullsized":"2024-03-06T07:22:44.346864+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.529739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287682+0000","last_change":"2024-03-06T07:22:42.289384+0000","last_active":"2024-03-06T07:22:44.287682+0000","last_peered":"2024-03-06T07:22:44.287682+0000","last_clean":"2024-03-06T07:22:44.287682+0000","last_became_active":"2024-03-06T07:22:42.289045+0000","last_became_peered":"2024-03-06T07:22:42.289045+0000","last_unstale":"2024-03-06T07:22:44.287682+0000","last_undegraded":"2024-03-06T07:22:44.287682+0000","last_fullsized":"2024-03-06T07:22:44.287682+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.165551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347147+0000","last_change":"2024-03-06T07:22:43.290021+0000","last_active":"2024-03-06T07:22:44.347147+0000","last_peered":"2024-03-06T07:22:44.347147+0000","last_clean":"2024-03-06T07:22:44.347147+0000","last_became_active":"2024-03-06T07:22:43.289692+0000","last_became_peered":"2024-03-06T07:22:43.289692+0000","last_unstale":"2024-03-06T07:22:44.347147+0000","last_undegraded":"2024-03-06T07:22:44.347147+0000","last_fullsized":"2024-03-06T07:22:44.347147+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:40.154204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"120'8","reported_seq":22,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.348031+0000","last_change":"2024-03-06T07:22:42.294517+0000","last_active":"2024-03-06T07:22:44.348031+0000","last_peered":"2024-03-06T07:22:44.348031+0000","last_clean":"2024-03-06T07:22:44.348031+0000","last_became_active":"2024-03-06T07:22:42.294249+0000","last_became_peered":"2024-03-06T07:22:42.294249+0000","last_unstale":"2024-03-06T07:22:44.348031+0000","last_undegraded":"2024-03-06T07:22:44.348031+0000","last_fullsized":"2024-03-06T07:22:44.348031+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:03.400867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.288103+0000","last_change":"2024-03-06T07:22:43.291254+0000","last_active":"2024-03-06T07:22:44.288103+0000","last_peered":"2024-03-06T07:22:44.288103+0000","last_clean":"2024-03-06T07:22:44.288103+0000","last_became_active":"2024-03-06T07:22:43.290860+0000","last_became_peered":"2024-03-06T07:22:43.290860+0000","last_unstale":"2024-03-06T07:22:44.288103+0000","last_undegraded":"2024-03-06T07:22:44.288103+0000","last_fullsized":"2024-03-06T07:22:44.288103+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:06.102249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293752+0000","last_change":"2024-03-06T07:22:42.288271+0000","last_active":"2024-03-06T07:22:44.293752+0000","last_peered":"2024-03-06T07:22:44.293752+0000","last_clean":"2024-03-06T07:22:44.293752+0000","last_became_active":"2024-03-06T07:22:42.287981+0000","last_became_peered":"2024-03-06T07:22:42.287981+0000","last_unstale":"2024-03-06T07:22:44.293752+0000","last_undegraded":"2024-03-06T07:22:44.293752+0000","last_fullsized":"2024-03-06T07:22:44.293752+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:16.050156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346437+0000","last_change":"2024-03-06T07:22:43.302832+0000","last_active":"2024-03-06T07:22:44.346437+0000","last_peered":"2024-03-06T07:22:44.346437+0000","last_clean":"2024-03-06T07:22:44.346437+0000","last_became_active":"2024-03-06T07:22:43.302556+0000","last_became_peered":"2024-03-06T07:22:43.302556+0000","last_unstale":"2024-03-06T07:22:44.346437+0000","last_undegraded":"2024-03-06T07:22:44.346437+0000","last_fullsized":"2024-03-06T07:22:44.346437+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:37.505357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287886+0000","last_change":"2024-03-06T07:22:42.289825+0000","last_active":"2024-03-06T07:22:44.287886+0000","last_peered":"2024-03-06T07:22:44.287886+0000","last_clean":"2024-03-06T07:22:44.287886+0000","last_became_active":"2024-03-06T07:22:42.288884+0000","last_became_peered":"2024-03-06T07:22:42.288884+0000","last_unstale":"2024-03-06T07:22:44.287886+0000","last_undegraded":"2024-03-06T07:22:44.287886+0000","last_fullsized":"2024-03-06T07:22:44.287886+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:56.374398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346761+0000","last_change":"2024-03-06T07:22:43.303059+0000","last_active":"2024-03-06T07:22:44.346761+0000","last_peered":"2024-03-06T07:22:44.346761+0000","last_clean":"2024-03-06T07:22:44.346761+0000","last_became_active":"2024-03-06T07:22:43.302244+0000","last_became_peered":"2024-03-06T07:22:43.302244+0000","last_unstale":"2024-03-06T07:22:44.346761+0000","last_undegraded":"2024-03-06T07:22:44.346761+0000","last_fullsized":"2024-03-06T07:22:44.346761+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:48.741247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"120'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.348267+0000","last_change":"2024-03-06T07:22:42.286639+0000","last_active":"2024-03-06T07:22:44.348267+0000","last_peered":"2024-03-06T07:22:44.348267+0000","last_clean":"2024-03-06T07:22:44.348267+0000","last_became_active":"2024-03-06T07:22:42.286392+0000","last_became_peered":"2024-03-06T07:22:42.286392+0000","last_unstale":"2024-03-06T07:22:44.348267+0000","last_undegraded":"2024-03-06T07:22:44.348267+0000","last_fullsized":"2024-03-06T07:22:44.348267+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.744017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.286055+0000","last_change":"2024-03-06T07:22:43.291297+0000","last_active":"2024-03-06T07:22:44.286055+0000","last_peered":"2024-03-06T07:22:44.286055+0000","last_clean":"2024-03-06T07:22:44.286055+0000","last_became_active":"2024-03-06T07:22:43.291039+0000","last_became_peered":"2024-03-06T07:22:43.291039+0000","last_unstale":"2024-03-06T07:22:44.286055+0000","last_undegraded":"2024-03-06T07:22:44.286055+0000","last_fullsized":"2024-03-06T07:22:44.286055+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:20.890883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347671+0000","last_change":"2024-03-06T07:22:42.284042+0000","last_active":"2024-03-06T07:22:44.347671+0000","last_peered":"2024-03-06T07:22:44.347671+0000","last_clean":"2024-03-06T07:22:44.347671+0000","last_became_active":"2024-03-06T07:22:42.283781+0000","last_became_peered":"2024-03-06T07:22:42.283781+0000","last_unstale":"2024-03-06T07:22:44.347671+0000","last_undegraded":"2024-03-06T07:22:44.347671+0000","last_fullsized":"2024-03-06T07:22:44.347671+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:36.836462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347611+0000","last_change":"2024-03-06T07:22:43.288066+0000","last_active":"2024-03-06T07:22:44.347611+0000","last_peered":"2024-03-06T07:22:44.347611+0000","last_clean":"2024-03-06T07:22:44.347611+0000","last_became_active":"2024-03-06T07:22:43.287773+0000","last_became_peered":"2024-03-06T07:22:43.287773+0000","last_unstale":"2024-03-06T07:22:44.347611+0000","last_undegraded":"2024-03-06T07:22:44.347611+0000","last_fullsized":"2024-03-06T07:22:44.347611+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:41.180473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287821+0000","last_change":"2024-03-06T07:22:42.297131+0000","last_active":"2024-03-06T07:22:44.287821+0000","last_peered":"2024-03-06T07:22:44.287821+0000","last_clean":"2024-03-06T07:22:44.287821+0000","last_became_active":"2024-03-06T07:22:42.296815+0000","last_became_peered":"2024-03-06T07:22:42.296815+0000","last_unstale":"2024-03-06T07:22:44.287821+0000","last_undegraded":"2024-03-06T07:22:44.287821+0000","last_fullsized":"2024-03-06T07:22:44.287821+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:21.527693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347412+0000","last_change":"2024-03-06T07:22:43.298219+0000","last_active":"2024-03-06T07:22:44.347412+0000","last_peered":"2024-03-06T07:22:44.347412+0000","last_clean":"2024-03-06T07:22:44.347412+0000","last_became_active":"2024-03-06T07:22:43.297945+0000","last_became_peered":"2024-03-06T07:22:43.297945+0000","last_unstale":"2024-03-06T07:22:44.347412+0000","last_undegraded":"2024-03-06T07:22:44.347412+0000","last_fullsized":"2024-03-06T07:22:44.347412+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:48.327006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347444+0000","last_change":"2024-03-06T07:22:42.288618+0000","last_active":"2024-03-06T07:22:44.347444+0000","last_peered":"2024-03-06T07:22:44.347444+0000","last_clean":"2024-03-06T07:22:44.347444+0000","last_became_active":"2024-03-06T07:22:42.287863+0000","last_became_peered":"2024-03-06T07:22:42.287863+0000","last_unstale":"2024-03-06T07:22:44.347444+0000","last_undegraded":"2024-03-06T07:22:44.347444+0000","last_fullsized":"2024-03-06T07:22:44.347444+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:58.566165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347303+0000","last_change":"2024-03-06T07:22:43.301308+0000","last_active":"2024-03-06T07:22:44.347303+0000","last_peered":"2024-03-06T07:22:44.347303+0000","last_clean":"2024-03-06T07:22:44.347303+0000","last_became_active":"2024-03-06T07:22:43.301043+0000","last_became_peered":"2024-03-06T07:22:43.301043+0000","last_unstale":"2024-03-06T07:22:44.347303+0000","last_undegraded":"2024-03-06T07:22:44.347303+0000","last_fullsized":"2024-03-06T07:22:44.347303+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:13.720467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346560+0000","last_change":"2024-03-06T07:22:42.293801+0000","last_active":"2024-03-06T07:22:44.346560+0000","last_peered":"2024-03-06T07:22:44.346560+0000","last_clean":"2024-03-06T07:22:44.346560+0000","last_became_active":"2024-03-06T07:22:42.293537+0000","last_became_peered":"2024-03-06T07:22:42.293537+0000","last_unstale":"2024-03-06T07:22:44.346560+0000","last_undegraded":"2024-03-06T07:22:44.346560+0000","last_fullsized":"2024-03-06T07:22:44.346560+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:22.010852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293715+0000","last_change":"2024-03-06T07:22:43.289373+0000","last_active":"2024-03-06T07:22:44.293715+0000","last_peered":"2024-03-06T07:22:44.293715+0000","last_clean":"2024-03-06T07:22:44.293715+0000","last_became_active":"2024-03-06T07:22:43.289009+0000","last_became_peered":"2024-03-06T07:22:43.289009+0000","last_unstale":"2024-03-06T07:22:44.293715+0000","last_undegraded":"2024-03-06T07:22:44.293715+0000","last_fullsized":"2024-03-06T07:22:44.293715+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:57.686946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.348128+0000","last_change":"2024-03-06T07:22:42.287323+0000","last_active":"2024-03-06T07:22:44.348128+0000","last_peered":"2024-03-06T07:22:44.348128+0000","last_clean":"2024-03-06T07:22:44.348128+0000","last_became_active":"2024-03-06T07:22:42.287077+0000","last_became_peered":"2024-03-06T07:22:42.287077+0000","last_unstale":"2024-03-06T07:22:44.348128+0000","last_undegraded":"2024-03-06T07:22:44.348128+0000","last_fullsized":"2024-03-06T07:22:44.348128+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:16.050156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347013+0000","last_change":"2024-03-06T07:22:43.302778+0000","last_active":"2024-03-06T07:22:44.347013+0000","last_peered":"2024-03-06T07:22:44.347013+0000","last_clean":"2024-03-06T07:22:44.347013+0000","last_became_active":"2024-03-06T07:22:43.302524+0000","last_became_peered":"2024-03-06T07:22:43.302524+0000","last_unstale":"2024-03-06T07:22:44.347013+0000","last_undegraded":"2024-03-06T07:22:44.347013+0000","last_fullsized":"2024-03-06T07:22:44.347013+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.823704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"120'2","reported_seq":16,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347629+0000","last_change":"2024-03-06T07:22:42.290240+0000","last_active":"2024-03-06T07:22:44.347629+0000","last_peered":"2024-03-06T07:22:44.347629+0000","last_clean":"2024-03-06T07:22:44.347629+0000","last_became_active":"2024-03-06T07:22:42.289934+0000","last_became_peered":"2024-03-06T07:22:42.289934+0000","last_unstale":"2024-03-06T07:22:44.347629+0000","last_undegraded":"2024-03-06T07:22:44.347629+0000","last_fullsized":"2024-03-06T07:22:44.347629+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:51.545342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287588+0000","last_change":"2024-03-06T07:22:43.293939+0000","last_active":"2024-03-06T07:22:44.287588+0000","last_peered":"2024-03-06T07:22:44.287588+0000","last_clean":"2024-03-06T07:22:44.287588+0000","last_became_active":"2024-03-06T07:22:43.293669+0000","last_became_peered":"2024-03-06T07:22:43.293669+0000","last_unstale":"2024-03-06T07:22:44.287588+0000","last_undegraded":"2024-03-06T07:22:44.287588+0000","last_fullsized":"2024-03-06T07:22:44.287588+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.748569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"121'1","reported_seq":15,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.288095+0000","last_change":"2024-03-06T07:22:42.292354+0000","last_active":"2024-03-06T07:22:44.288095+0000","last_peered":"2024-03-06T07:22:44.288095+0000","last_clean":"2024-03-06T07:22:44.288095+0000","last_became_active":"2024-03-06T07:22:42.292033+0000","last_became_peered":"2024-03-06T07:22:42.292033+0000","last_unstale":"2024-03-06T07:22:44.288095+0000","last_undegraded":"2024-03-06T07:22:44.288095+0000","last_fullsized":"2024-03-06T07:22:44.288095+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:06.658036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346398+0000","last_change":"2024-03-06T07:22:43.296534+0000","last_active":"2024-03-06T07:22:44.346398+0000","last_peered":"2024-03-06T07:22:44.346398+0000","last_clean":"2024-03-06T07:22:44.346398+0000","last_became_active":"2024-03-06T07:22:43.296224+0000","last_became_peered":"2024-03-06T07:22:43.296224+0000","last_unstale":"2024-03-06T07:22:44.346398+0000","last_undegraded":"2024-03-06T07:22:44.346398+0000","last_fullsized":"2024-03-06T07:22:44.346398+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:41.573878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.287263+0000","last_change":"2024-03-06T07:22:42.288768+0000","last_active":"2024-03-06T07:22:44.287263+0000","last_peered":"2024-03-06T07:22:44.287263+0000","last_clean":"2024-03-06T07:22:44.287263+0000","last_became_active":"2024-03-06T07:22:42.288476+0000","last_became_peered":"2024-03-06T07:22:42.288476+0000","last_unstale":"2024-03-06T07:22:44.287263+0000","last_undegraded":"2024-03-06T07:22:44.287263+0000","last_fullsized":"2024-03-06T07:22:44.287263+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:04.223232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.286192+0000","last_change":"2024-03-06T07:22:43.289679+0000","last_active":"2024-03-06T07:22:44.286192+0000","last_peered":"2024-03-06T07:22:44.286192+0000","last_clean":"2024-03-06T07:22:44.286192+0000","last_became_active":"2024-03-06T07:22:43.289382+0000","last_became_peered":"2024-03-06T07:22:43.289382+0000","last_unstale":"2024-03-06T07:22:44.286192+0000","last_undegraded":"2024-03-06T07:22:44.286192+0000","last_fullsized":"2024-03-06T07:22:44.286192+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:13.720467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347921+0000","last_change":"2024-03-06T07:22:42.293943+0000","last_active":"2024-03-06T07:22:44.347921+0000","last_peered":"2024-03-06T07:22:44.347921+0000","last_clean":"2024-03-06T07:22:44.347921+0000","last_became_active":"2024-03-06T07:22:42.293661+0000","last_became_peered":"2024-03-06T07:22:42.293661+0000","last_unstale":"2024-03-06T07:22:44.347921+0000","last_undegraded":"2024-03-06T07:22:44.347921+0000","last_fullsized":"2024-03-06T07:22:44.347921+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:07.580402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347839+0000","last_change":"2024-03-06T07:22:43.295259+0000","last_active":"2024-03-06T07:22:44.347839+0000","last_peered":"2024-03-06T07:22:44.347839+0000","last_clean":"2024-03-06T07:22:44.347839+0000","last_became_active":"2024-03-06T07:22:43.294970+0000","last_became_peered":"2024-03-06T07:22:43.294970+0000","last_unstale":"2024-03-06T07:22:44.347839+0000","last_undegraded":"2024-03-06T07:22:44.347839+0000","last_fullsized":"2024-03-06T07:22:44.347839+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:38.128097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.293424+0000","last_change":"2024-03-06T07:22:42.287760+0000","last_active":"2024-03-06T07:22:44.293424+0000","last_peered":"2024-03-06T07:22:44.293424+0000","last_clean":"2024-03-06T07:22:44.293424+0000","last_became_active":"2024-03-06T07:22:42.287466+0000","last_became_peered":"2024-03-06T07:22:42.287466+0000","last_unstale":"2024-03-06T07:22:44.293424+0000","last_undegraded":"2024-03-06T07:22:44.293424+0000","last_fullsized":"2024-03-06T07:22:44.293424+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:55.471196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.348313+0000","last_change":"2024-03-06T07:22:43.290283+0000","last_active":"2024-03-06T07:22:44.348313+0000","last_peered":"2024-03-06T07:22:44.348313+0000","last_clean":"2024-03-06T07:22:44.348313+0000","last_became_active":"2024-03-06T07:22:43.290022+0000","last_became_peered":"2024-03-06T07:22:43.290022+0000","last_unstale":"2024-03-06T07:22:44.348313+0000","last_undegraded":"2024-03-06T07:22:44.348313+0000","last_fullsized":"2024-03-06T07:22:44.348313+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:10.020653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346685+0000","last_change":"2024-03-06T07:22:42.296157+0000","last_active":"2024-03-06T07:22:44.346685+0000","last_peered":"2024-03-06T07:22:44.346685+0000","last_clean":"2024-03-06T07:22:44.346685+0000","last_became_active":"2024-03-06T07:22:42.295713+0000","last_became_peered":"2024-03-06T07:22:42.295713+0000","last_unstale":"2024-03-06T07:22:44.346685+0000","last_undegraded":"2024-03-06T07:22:44.346685+0000","last_fullsized":"2024-03-06T07:22:44.346685+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:06.355818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.348069+0000","last_change":"2024-03-06T07:22:43.295468+0000","last_active":"2024-03-06T07:22:44.348069+0000","last_peered":"2024-03-06T07:22:44.348069+0000","last_clean":"2024-03-06T07:22:44.348069+0000","last_became_active":"2024-03-06T07:22:43.295205+0000","last_became_peered":"2024-03-06T07:22:43.295205+0000","last_unstale":"2024-03-06T07:22:44.348069+0000","last_undegraded":"2024-03-06T07:22:44.348069+0000","last_fullsized":"2024-03-06T07:22:44.348069+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:38.216769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347058+0000","last_change":"2024-03-06T07:22:42.293654+0000","last_active":"2024-03-06T07:22:44.347058+0000","last_peered":"2024-03-06T07:22:44.347058+0000","last_clean":"2024-03-06T07:22:44.347058+0000","last_became_active":"2024-03-06T07:22:42.293361+0000","last_became_peered":"2024-03-06T07:22:42.293361+0000","last_unstale":"2024-03-06T07:22:44.347058+0000","last_undegraded":"2024-03-06T07:22:44.347058+0000","last_fullsized":"2024-03-06T07:22:44.347058+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:48.429074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346871+0000","last_change":"2024-03-06T07:22:43.297785+0000","last_active":"2024-03-06T07:22:44.346871+0000","last_peered":"2024-03-06T07:22:44.346871+0000","last_clean":"2024-03-06T07:22:44.346871+0000","last_became_active":"2024-03-06T07:22:43.296984+0000","last_became_peered":"2024-03-06T07:22:43.296984+0000","last_unstale":"2024-03-06T07:22:44.346871+0000","last_undegraded":"2024-03-06T07:22:44.346871+0000","last_fullsized":"2024-03-06T07:22:44.346871+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:06.681371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.347096+0000","last_change":"2024-03-06T07:22:42.290440+0000","last_active":"2024-03-06T07:22:44.347096+0000","last_peered":"2024-03-06T07:22:44.347096+0000","last_clean":"2024-03-06T07:22:44.347096+0000","last_became_active":"2024-03-06T07:22:42.290162+0000","last_became_peered":"2024-03-06T07:22:42.290162+0000","last_unstale":"2024-03-06T07:22:44.347096+0000","last_undegraded":"2024-03-06T07:22:44.347096+0000","last_fullsized":"2024-03-06T07:22:44.347096+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:09.016101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.288267+0000","last_change":"2024-03-06T07:22:43.295665+0000","last_active":"2024-03-06T07:22:44.288267+0000","last_peered":"2024-03-06T07:22:44.288267+0000","last_clean":"2024-03-06T07:22:44.288267+0000","last_became_active":"2024-03-06T07:22:43.295376+0000","last_became_peered":"2024-03-06T07:22:43.295376+0000","last_unstale":"2024-03-06T07:22:44.288267+0000","last_undegraded":"2024-03-06T07:22:44.288267+0000","last_fullsized":"2024-03-06T07:22:44.288267+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:42.266236+0000","last_clean_scrub_stamp":"2024-03-06T07:22:42.266236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:56.475748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346536+0000","last_change":"2024-03-06T07:22:42.293720+0000","last_active":"2024-03-06T07:22:44.346536+0000","last_peered":"2024-03-06T07:22:44.346536+0000","last_clean":"2024-03-06T07:22:44.346536+0000","last_became_active":"2024-03-06T07:22:42.293429+0000","last_became_peered":"2024-03-06T07:22:42.293429+0000","last_unstale":"2024-03-06T07:22:44.346536+0000","last_undegraded":"2024-03-06T07:22:44.346536+0000","last_fullsized":"2024-03-06T07:22:44.346536+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":118,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:41.261684+0000","last_clean_scrub_stamp":"2024-03-06T07:22:41.261684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:24.106985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":361,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-03-06T07:22:44.346318+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:22:44.346318+0000","last_peered":"2024-03-06T07:22:44.346318+0000","last_clean":"2024-03-06T07:22:44.346318+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:22:44.346318+0000","last_undegraded":"2024-03-06T07:22:44.346318+0000","last_fullsized":"2024-03-06T07:22:44.346318+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1597440,"data_stored":1321501,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1523712,"data_stored":730103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1519616,"data_stored":729453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":729431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":731113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1523712,"data_stored":730047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1519616,"data_stored":729501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1589248,"data_stored":1319855,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:50.943 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:50.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:22:51.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.085+0000 7f29c32a4640 1 Processor -- start 2024-03-06T07:22:51.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.085+0000 7f29c32a4640 1 -- start start 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c32a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29bc0c5990 0x7f29bc0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c32a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29bc0ab170 0x7f29bc06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c32a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc06cba0 con 0x7f29bc06c730 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c32a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29bc0c3ef0 con 0x7f29bc0ab170 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c32a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29bc1556d0 con 0x7f29bc0c5990 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c083f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29bc0ab170 0x7f29bc06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c1841640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f29bc06c730 0x7f29bc1523f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37076/0 (socket says 172.21.15.94:37076) 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c1841640 1 -- 172.21.15.94:0/1667423215 learned_addr learned my addr 172.21.15.94:0/1667423215 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c1040640 1 --2- 172.21.15.94:0/1667423215 >> v2:172.21.15.145:3300/0 conn(0x7f29bc0c5990 0x7f29bc0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c083f640 1 -- 172.21.15.94:0/1667423215 >> v2:172.21.15.145:3300/0 conn(0x7f29bc0c5990 msgr2=0x7f29bc0c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c083f640 1 --2- 172.21.15.94:0/1667423215 >> v2:172.21.15.145:3300/0 conn(0x7f29bc0c5990 0x7f29bc0c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c083f640 1 -- 172.21.15.94:0/1667423215 >> v1:172.21.15.94:6789/0 conn(0x7f29bc06c730 legacy=0x7f29bc1523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29b3fff640 1 -- 172.21.15.94:0/1667423215 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415748837 0 0) 0x7f29bc06cba0 con 0x7f29bc06c730 2024-03-06T07:22:51.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.086+0000 7f29c083f640 1 -- 172.21.15.94:0/1667423215 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc1559b0 con 0x7f29bc0ab170 2024-03-06T07:22:51.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.087+0000 7f29c083f640 1 --2- 172.21.15.94:0/1667423215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0ab170 0x7f29bc06c1f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0c12a0 tx=0x7f29ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17916c9fba9fa014 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.087+0000 7f29b3fff640 1 -- 172.21.15.94:0/1667423215 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac035020 con 0x7f29bc0ab170 2024-03-06T07:22:51.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.087+0000 7f29b3fff640 1 -- 172.21.15.94:0/1667423215 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ac031830 con 0x7f29bc0ab170 2024-03-06T07:22:51.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.087+0000 7f29b3fff640 1 -- 172.21.15.94:0/1667423215 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac031c00 con 0x7f29bc0ab170 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 -- 172.21.15.94:0/1667423215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0ab170 msgr2=0x7f29bc06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 --2- 172.21.15.94:0/1667423215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0ab170 0x7f29bc06c1f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0c12a0 tx=0x7f29ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 -- 172.21.15.94:0/1667423215 shutdown_connections 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 --2- 172.21.15.94:0/1667423215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0ab170 0x7f29bc06c1f0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 --2- 172.21.15.94:0/1667423215 >> v2:172.21.15.145:3300/0 conn(0x7f29bc0c5990 0x7f29bc0c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 -- 172.21.15.94:0/1667423215 >> 172.21.15.94:0/1667423215 conn(0x7f29bc0c6710 msgr2=0x7f29bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 -- 172.21.15.94:0/1667423215 shutdown_connections 2024-03-06T07:22:51.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.088+0000 7f29c32a4640 1 -- 172.21.15.94:0/1667423215 wait complete. 2024-03-06T07:22:51.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 Processor -- start 2024-03-06T07:22:51.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 -- start start 2024-03-06T07:22:51.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29bc06c730 0x7f29bc154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29bc0ab170 0x7f29bc14d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0c5990 0x7f29bc14d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc156490 con 0x7f29bc0c5990 2024-03-06T07:22:51.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc155f70 con 0x7f29bc0ab170 2024-03-06T07:22:51.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c32a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc156610 con 0x7f29bc06c730 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c1841640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0c5990 0x7f29bc14d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.089+0000 7f29c1841640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0c5990 0x7f29bc14d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50948/0 (socket says 172.21.15.94:50948) 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1841640 1 -- 172.21.15.94:0/2205634616 learned_addr learned my addr 172.21.15.94:0/2205634616 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1040640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29bc06c730 0x7f29bc154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c083f640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29bc0ab170 0x7f29bc14d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1841640 1 -- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29bc06c730 msgr2=0x7f29bc154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1841640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29bc06c730 0x7f29bc154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1841640 1 -- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29bc0ab170 msgr2=0x7f29bc14d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1841640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29bc0ab170 0x7f29bc14d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1841640 1 -- 172.21.15.94:0/2205634616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc14cc60 con 0x7f29bc0c5990 2024-03-06T07:22:51.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c1841640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0c5990 0x7f29bc14d660 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f29b400c7a0 tx=0x7f29b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29b1ffb640 1 -- 172.21.15.94:0/2205634616 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4018070 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29b1ffb640 1 -- 172.21.15.94:0/2205634616 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b400a640 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29bc1552a0 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29b1ffb640 1 -- 172.21.15.94:0/2205634616 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b400aab0 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.090+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29bc1e9350 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.091+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29bc04f330 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.092+0000 7f29b1ffb640 1 -- 172.21.15.94:0/2205634616 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29b401b070 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.092+0000 7f29b1ffb640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29a006b500 0x7f29a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.092+0000 7f29b1ffb640 1 -- 172.21.15.94:0/2205634616 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29b4089890 con 0x7f29bc0c5990 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.092+0000 7f29c1040640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29a006b500 0x7f29a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.093+0000 7f29c1040640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29a006b500 0x7f29a006d9c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009aa0 tx=0x7f29b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.096+0000 7f29b1ffb640 1 -- 172.21.15.94:0/2205634616 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29b4057f30 con 0x7f29bc0c5990 2024-03-06T07:22:51.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.337+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f29bc0c1740 con 0x7f29bc0c5990 2024-03-06T07:22:51.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.338+0000 7f29b1ffb640 1 -- 172.21.15.94:0/2205634616 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f29b405bfd0 con 0x7f29bc0c5990 2024-03-06T07:22:51.340 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:22:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:22:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:22:51.340 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:22:51.341 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:51.341 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:22:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29a006b500 msgr2=0x7f29a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29a006b500 0x7f29a006d9c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009aa0 tx=0x7f29b8009190 comp rx=0 tx=0).stop 2024-03-06T07:22:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0c5990 msgr2=0x7f29bc14d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0c5990 0x7f29bc14d660 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f29b400c7a0 tx=0x7f29b400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 shutdown_connections 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29a006b500 0x7f29a006d9c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29bc0c5990 0x7f29bc14d660 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29bc0ab170 0x7f29bc14d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 --2- 172.21.15.94:0/2205634616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29bc06c730 0x7f29bc154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 >> 172.21.15.94:0/2205634616 conn(0x7f29bc0c6710 msgr2=0x7f29bc14aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 shutdown_connections 2024-03-06T07:22:51.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.340+0000 7f29c32a4640 1 -- 172.21.15.94:0/2205634616 wait complete. 2024-03-06T07:22:51.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:51.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.451+0000 7f1db6eeb640 1 Processor -- start 2024-03-06T07:22:51.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.451+0000 7f1db6eeb640 1 -- start start 2024-03-06T07:22:51.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1db6eeb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db006c720 0x7f1db0071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1db6eeb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db0067340 con 0x7f1db00c2410 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1db6eeb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db0150d00 con 0x7f1db0150920 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1db6eeb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00c2010 con 0x7f1db006c720 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1db5488640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1db00c2410 0x7f1db015dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37086/0 (socket says 172.21.15.94:37086) 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1db5488640 1 -- 172.21.15.94:0/271652257 learned_addr learned my addr 172.21.15.94:0/271652257 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1daffff640 1 --2- 172.21.15.94:0/271652257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db006c720 0x7f1db0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1daf7fe640 1 -- 172.21.15.94:0/271652257 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797184877 0 0) 0x7f1db0067340 con 0x7f1db00c2410 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.452+0000 7f1daf7fe640 1 -- 172.21.15.94:0/271652257 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d98003520 con 0x7f1db00c2410 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.453+0000 7f1daf7fe640 1 -- 172.21.15.94:0/271652257 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774931801 0 0) 0x7f1db0150d00 con 0x7f1db0150920 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.453+0000 7f1daf7fe640 1 -- 172.21.15.94:0/271652257 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0067340 con 0x7f1db0150920 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.453+0000 7f1daffff640 1 -- 172.21.15.94:0/271652257 >> v1:172.21.15.145:6789/0 conn(0x7f1db0150920 legacy=0x7f1db00c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.453+0000 7f1daffff640 1 -- 172.21.15.94:0/271652257 >> v1:172.21.15.94:6789/0 conn(0x7f1db00c2410 legacy=0x7f1db015dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.453+0000 7f1daffff640 1 -- 172.21.15.94:0/271652257 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0067060 con 0x7f1db006c720 2024-03-06T07:22:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.453+0000 7f1daffff640 1 --2- 172.21.15.94:0/271652257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db006c720 0x7f1db0071980 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1da40097d0 tx=0x7f1da402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=979e8de5f6741d8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1daf7fe640 1 -- 172.21.15.94:0/271652257 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4035020 con 0x7f1db006c720 2024-03-06T07:22:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1daf7fe640 1 -- 172.21.15.94:0/271652257 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4031800 con 0x7f1db006c720 2024-03-06T07:22:51.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1daf7fe640 1 -- 172.21.15.94:0/271652257 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4031bd0 con 0x7f1db006c720 2024-03-06T07:22:51.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1db6eeb640 1 -- 172.21.15.94:0/271652257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db006c720 msgr2=0x7f1db0071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/271652257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db006c720 0x7f1db0071980 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1da40097d0 tx=0x7f1da402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1db6eeb640 1 -- 172.21.15.94:0/271652257 shutdown_connections 2024-03-06T07:22:51.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/271652257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db006c720 0x7f1db0071980 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1db6eeb640 1 -- 172.21.15.94:0/271652257 >> 172.21.15.94:0/271652257 conn(0x7f1db00c70b0 msgr2=0x7f1db00c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1db6eeb640 1 -- 172.21.15.94:0/271652257 shutdown_connections 2024-03-06T07:22:51.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.454+0000 7f1db6eeb640 1 -- 172.21.15.94:0/271652257 wait complete. 2024-03-06T07:22:51.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.455+0000 7f1db6eeb640 1 Processor -- start 2024-03-06T07:22:51.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.455+0000 7f1db6eeb640 1 -- start start 2024-03-06T07:22:51.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db6eeb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db006c720 0x7f1db0153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db6eeb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00c2410 0x7f1db0144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db6eeb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0150920 0x7f1db0142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db6eeb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00c31c0 con 0x7f1db0150920 2024-03-06T07:22:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db6eeb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00c3040 con 0x7f1db006c720 2024-03-06T07:22:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db6eeb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0073090 con 0x7f1db00c2410 2024-03-06T07:22:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0150920 0x7f1db0142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0150920 0x7f1db0142bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50952/0 (socket says 172.21.15.94:50952) 2024-03-06T07:22:51.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 -- 172.21.15.94:0/3954404603 learned_addr learned my addr 172.21.15.94:0/3954404603 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:51.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1daffff640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00c2410 0x7f1db0144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db4c87640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db006c720 0x7f1db0153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 -- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00c2410 msgr2=0x7f1db0144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00c2410 0x7f1db0144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 -- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db006c720 msgr2=0x7f1db0153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db006c720 0x7f1db0153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.456+0000 7f1db5488640 1 -- 172.21.15.94:0/3954404603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0152810 con 0x7f1db0150920 2024-03-06T07:22:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.457+0000 7f1db5488640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0150920 0x7f1db0142bd0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f1da800caa0 tx=0x7f1da800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.457+0000 7f1dad7fa640 1 -- 172.21.15.94:0/3954404603 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8017070 con 0x7f1db0150920 2024-03-06T07:22:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.457+0000 7f1dad7fa640 1 -- 172.21.15.94:0/3954404603 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da800ac20 con 0x7f1db0150920 2024-03-06T07:22:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.457+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db0152af0 con 0x7f1db0150920 2024-03-06T07:22:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.457+0000 7f1dad7fa640 1 -- 172.21.15.94:0/3954404603 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8012640 con 0x7f1db0150920 2024-03-06T07:22:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.457+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db01530b0 con 0x7f1db0150920 2024-03-06T07:22:51.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.458+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db01532f0 con 0x7f1db0150920 2024-03-06T07:22:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.458+0000 7f1dad7fa640 1 -- 172.21.15.94:0/3954404603 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1da801a070 con 0x7f1db0150920 2024-03-06T07:22:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.459+0000 7f1dad7fa640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.459+0000 7f1dad7fa640 1 -- 172.21.15.94:0/3954404603 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1da808a0c0 con 0x7f1db0150920 2024-03-06T07:22:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.459+0000 7f1db4c87640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.459+0000 7f1db4c87640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1da0007fd0 tx=0x7f1da0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.463+0000 7f1dad7fa640 1 -- 172.21.15.94:0/3954404603 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1db01532f0 con 0x7f1db0150920 2024-03-06T07:22:51.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.669+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1db004f330 con 0x7f1db0150920 2024-03-06T07:22:51.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.670+0000 7f1dad7fa640 1 -- 172.21.15.94:0/3954404603 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4985 (secure 0 0 0) 0x7f1da8058760 con 0x7f1db0150920 2024-03-06T07:22:51.672 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:51.672 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7827,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/282207355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":282207355},{"type":"v1","addr":"172.21.15.94:6834","nonce":282207355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:22:44.626233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7842},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7842":{"gid":7842,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/198029515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":198029515},{"type":"v1","addr":"172.21.15.145:6836","nonce":198029515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7842,"qdb_cluster":[7842]},"id":9}]} 2024-03-06T07:22:51.672 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 104 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 msgr2=0x7f1d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1da0007fd0 tx=0x7f1da0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0150920 msgr2=0x7f1db0142bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0150920 0x7f1db0142bd0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f1da800caa0 tx=0x7f1da800cf60 comp rx=0 tx=0).stop 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 shutdown_connections 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0150920 0x7f1db0142bd0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00c2410 0x7f1db0144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 --2- 172.21.15.94:0/3954404603 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db006c720 0x7f1db0153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 >> 172.21.15.94:0/3954404603 conn(0x7f1db00c70b0 msgr2=0x7f1db00c7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 shutdown_connections 2024-03-06T07:22:51.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:51.672+0000 7f1db6eeb640 1 -- 172.21.15.94:0/3954404603 wait complete. 2024-03-06T07:22:52.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:42.624925+0000', 'modified': '2024-03-06T07:22:44.626233+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7842}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7842': {'gid': 7842, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/198029515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 198029515}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 198029515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7842, 'qdb_cluster': [7842]} 2024-03-06T07:22:52.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:52.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:22:52.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.188+0000 7f6a69eb8640 1 Processor -- start 2024-03-06T07:22:52.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.188+0000 7f6a69eb8640 1 -- start start 2024-03-06T07:22:52.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a69eb8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6a640c6ab0 0x7f6a641529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a69eb8640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6a6414f110 0x7f6a6414f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a69eb8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a640a9990 con 0x7f6a641561a0 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a69eb8640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a64155d10 con 0x7f6a6414f110 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a69eb8640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a641556c0 con 0x7f6a640c6ab0 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a63fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6a6414f110 0x7f6a6414f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a62ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6a641561a0 0x7f6a640a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37100/0 (socket says 172.21.15.94:37100) 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a62ffd640 1 -- 172.21.15.94:0/2355018295 learned_addr learned my addr 172.21.15.94:0/2355018295 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a637fe640 1 --2- 172.21.15.94:0/2355018295 >> v2:172.21.15.145:3300/0 conn(0x7f6a640c6ab0 0x7f6a641529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a63fff640 1 -- 172.21.15.94:0/2355018295 >> v2:172.21.15.145:3300/0 conn(0x7f6a640c6ab0 msgr2=0x7f6a641529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a63fff640 1 --2- 172.21.15.94:0/2355018295 >> v2:172.21.15.145:3300/0 conn(0x7f6a640c6ab0 0x7f6a641529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a63fff640 1 -- 172.21.15.94:0/2355018295 >> v1:172.21.15.94:6789/0 conn(0x7f6a641561a0 legacy=0x7f6a640a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a63fff640 1 -- 172.21.15.94:0/2355018295 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a641559a0 con 0x7f6a6414f110 2024-03-06T07:22:52.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.189+0000 7f6a627fc640 1 -- 172.21.15.94:0/2355018295 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081141802 0 0) 0x7f6a640a9990 con 0x7f6a641561a0 2024-03-06T07:22:52.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.190+0000 7f6a63fff640 1 --2- 172.21.15.94:0/2355018295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414f5a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6a640c74f0 tx=0x7f6a5c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec98807aedd7d961 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.190+0000 7f6a627fc640 1 -- 172.21.15.94:0/2355018295 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c039070 con 0x7f6a6414f110 2024-03-06T07:22:52.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.190+0000 7f6a627fc640 1 -- 172.21.15.94:0/2355018295 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c02cbf0 con 0x7f6a6414f110 2024-03-06T07:22:52.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.190+0000 7f6a627fc640 1 -- 172.21.15.94:0/2355018295 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c034440 con 0x7f6a6414f110 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 -- 172.21.15.94:0/2355018295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 msgr2=0x7f6a6414f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/2355018295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414f5a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6a640c74f0 tx=0x7f6a5c02be40 comp rx=0 tx=0).stop 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 -- 172.21.15.94:0/2355018295 shutdown_connections 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/2355018295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414f5a0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/2355018295 >> v2:172.21.15.145:3300/0 conn(0x7f6a640c6ab0 0x7f6a641529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 -- 172.21.15.94:0/2355018295 >> 172.21.15.94:0/2355018295 conn(0x7f6a6406b1a0 msgr2=0x7f6a6406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 -- 172.21.15.94:0/2355018295 shutdown_connections 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.191+0000 7f6a69eb8640 1 -- 172.21.15.94:0/2355018295 wait complete. 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 Processor -- start 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 -- start start 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a640c6ab0 0x7f6a640bfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a641561a0 0x7f6a6414b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a640aa600 con 0x7f6a6414f110 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a640c2a90 con 0x7f6a640c6ab0 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.192+0000 7f6a69eb8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6414a770 con 0x7f6a641561a0 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50966/0 (socket says 172.21.15.94:50966) 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 -- 172.21.15.94:0/4273918301 learned_addr learned my addr 172.21.15.94:0/4273918301 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a637fe640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a640c6ab0 0x7f6a640bfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a63fff640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a641561a0 0x7f6a6414b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 -- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a641561a0 msgr2=0x7f6a6414b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a641561a0 0x7f6a6414b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 -- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a640c6ab0 msgr2=0x7f6a640bfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a640c6ab0 0x7f6a640bfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 -- 172.21.15.94:0/4273918301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a640c5ad0 con 0x7f6a6414f110 2024-03-06T07:22:52.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414ac00 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800c4d0 tx=0x7f6a5800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a4bfff640 1 -- 172.21.15.94:0/4273918301 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58018070 con 0x7f6a6414f110 2024-03-06T07:22:52.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a4bfff640 1 -- 172.21.15.94:0/4273918301 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a58007500 con 0x7f6a6414f110 2024-03-06T07:22:52.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a640c57f0 con 0x7f6a6414f110 2024-03-06T07:22:52.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a4bfff640 1 -- 172.21.15.94:0/4273918301 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58004a20 con 0x7f6a6414f110 2024-03-06T07:22:52.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.193+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a641547a0 con 0x7f6a6414f110 2024-03-06T07:22:52.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.195+0000 7f6a4bfff640 1 -- 172.21.15.94:0/4273918301 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a5801b070 con 0x7f6a6414f110 2024-03-06T07:22:52.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.195+0000 7f6a4bfff640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a5006b4d0 0x7f6a5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.195+0000 7f6a4bfff640 1 -- 172.21.15.94:0/4273918301 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6a58089590 con 0x7f6a6414f110 2024-03-06T07:22:52.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.195+0000 7f6a637fe640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a5006b4d0 0x7f6a5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.196+0000 7f6a637fe640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a5006b4d0 0x7f6a5006d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009aa0 tx=0x7f6a54009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.196+0000 7f6a6cb77740 1 Processor -- start 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.196+0000 7f6a6cb77740 1 -- start start 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b20f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f4330 0x55b5b20f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f24b0 0x55b5b20ee680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b20586b0 con 0x55b5b20f3f50 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b2001a60 con 0x55b5b20f4330 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5b2054750 con 0x55b5b20f24b0 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b20f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b20f1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50978/0 (socket says 172.21.15.94:50978) 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 -- 172.21.15.94:0/146411838 learned_addr learned my addr 172.21.15.94:0/146411838 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f4330 0x55b5b20f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a63fff640 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f24b0 0x55b5b20ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 -- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f24b0 msgr2=0x55b5b20ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f24b0 0x55b5b20ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 -- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f4330 msgr2=0x55b5b20f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f4330 0x55b5b20f1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 -- 172.21.15.94:0/146411838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5b201ff30 con 0x55b5b20f3f50 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f4330 0x55b5b20f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.197+0000 7f6a637fe640 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b20f1a30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6a54004440 tx=0x7f6a54014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209161434ecc0b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a497fa640 1 -- 172.21.15.94:0/146411838 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5401b070 con 0x55b5b20f3f50 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a497fa640 1 -- 172.21.15.94:0/146411838 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a54027070 con 0x55b5b20f3f50 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a497fa640 1 -- 172.21.15.94:0/146411838 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54003a20 con 0x55b5b20f3f50 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 -- 172.21.15.94:0/146411838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 msgr2=0x55b5b20f1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b20f1a30 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6a54004440 tx=0x7f6a54014000 comp rx=0 tx=0).stop 2024-03-06T07:22:52.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 -- 172.21.15.94:0/146411838 shutdown_connections 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f24b0 0x55b5b20ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f4330 0x55b5b20f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/146411838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b20f1a30 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 -- 172.21.15.94:0/146411838 >> 172.21.15.94:0/146411838 conn(0x55b5b20f07d0 msgr2=0x55b5b21153c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 -- 172.21.15.94:0/146411838 shutdown_connections 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.198+0000 7f6a6cb77740 1 -- 172.21.15.94:0/146411838 wait complete. 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.199+0000 7f6a6cb77740 1 -- start start 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.199+0000 7f6a6cb77740 1 Processor -- start 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f3f50 0x55b5b2119e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b2116600 0x55b5b212cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b2116230 con 0x55b5b2116600 2024-03-06T07:22:52.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b2014ea0 con 0x55b5b20f3f50 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5b2115c70 con 0x55b5b20f4330 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 10 client.0 _ll_register_callbacks cb 0x55b5b2014870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b2116600 0x55b5b212cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b2116600 0x55b5b212cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50986/0 (socket says 172.21.15.94:50986) 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:49372/0 (socket says 172.21.15.94:49372) 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f3f50 0x55b5b2119e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 -- 172.21.15.94:0/1055542065 learned_addr learned my addr 172.21.15.94:0/1055542065 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 -- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 msgr2=0x55b5b212fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 -- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f3f50 msgr2=0x55b5b2119e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f3f50 0x55b5b2119e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 -- 172.21.15.94:0/1055542065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5b2015270 con 0x55b5b2116600 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a637fe640 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f3f50 0x55b5b2119e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:52.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a63fff640 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b2116600 0x55b5b212cb00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c02c450 tx=0x7f6a5c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e30e2d6de40cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a427fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a427fc640 1 -- 172.21.15.94:0/1055542065 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c045070 con 0x55b5b2116600 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a427fc640 1 -- 172.21.15.94:0/1055542065 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6a5c04ac50 con 0x55b5b2116600 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.200+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a54027070 con 0x55b5b2116600 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a427fc640 1 -- 172.21.15.94:0/1055542065 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a5c039070 con 0x55b5b2116600 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 10 client.8007 fetch_fsmap learned FSMap version 104 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x55b5b1d6d2d0 con 0x55b5b2116600 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a427fc640 1 -- 172.21.15.94:0/1055542065 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a54027070 con 0x55b5b2116600 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a427fc640 1 -- 172.21.15.94:0/1055542065 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 104) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6a5c004040 con 0x55b5b2116600 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 10 client.8007 fetch_fsmap finished waiting for FSMap version 104 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 10 client.8007 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 20 client.8007 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b2141400 0x55b5b214f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 4 client.8007 mds_command: new command op to 7842 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 --> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5b2065bf0 con 0x55b5b2141400 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a637fe640 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b2141400 0x55b5b214f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a637fe640 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b2141400 0x55b5b214f4a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c99d96ffebf968d server_cookie=49f07ee7707677dc in_seq=0 out_seq=0 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.201+0000 7f6a427fc640 10 client.8007 ms_handle_connect on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a427fc640 1 -- 172.21.15.94:0/1055542065 <== mds.0 v2:172.21.15.145:6834/198029515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b5b2065bf0 con 0x55b5b2141400 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a427fc640 10 client.8007 handle_command_reply: tid=0 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 client.8007 shutdown 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b2116600 msgr2=0x55b5b212cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b2116600 0x55b5b212cb00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c02c450 tx=0x7f6a5c004ab0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 shutdown_connections 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b2141400 0x55b5b214f4a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b2116600 0x55b5b212cb00 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1055542065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b20f3f50 0x55b5b2119e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 >> 172.21.15.94:0/1055542065 conn(0x55b5b20f24b0 msgr2=0x55b5b20ed510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a427fc640 0 client.8007 ms_handle_reset on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 shutdown_connections 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1055542065 wait complete. 2024-03-06T07:22:52.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.202+0000 7f6a6cb77740 20 client.8007 trim_cache size 0 max 16384 2024-03-06T07:22:52.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.219+0000 7f6a6cb77740 1 Processor -- start 2024-03-06T07:22:52.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.219+0000 7f6a6cb77740 1 -- start start 2024-03-06T07:22:52.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b21150a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b212c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a641f6c40 con 0x55b5b20f3f50 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64154210 con 0x55b5b2116600 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a641f6640 con 0x55b5b20f4330 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b21150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b21150a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51000/0 (socket says 172.21.15.94:51000) 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 -- 172.21.15.94:0/1296060206 learned_addr learned my addr 172.21.15.94:0/1296060206 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a63fff640 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b212c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 -- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 msgr2=0x55b5b212c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 -- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 msgr2=0x55b5b212c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b212c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 -- 172.21.15.94:0/1296060206 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5b1e376b0 con 0x55b5b20f3f50 2024-03-06T07:22:52.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a637fe640 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b21150a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f6a54024040 tx=0x7f6a54016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85d47a906a4fc27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a48ff9640 1 -- 172.21.15.94:0/1296060206 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5401b040 con 0x55b5b20f3f50 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a48ff9640 1 -- 172.21.15.94:0/1296060206 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a54027040 con 0x55b5b20f3f50 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.220+0000 7f6a48ff9640 1 -- 172.21.15.94:0/1296060206 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54003fd0 con 0x55b5b20f3f50 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1296060206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 msgr2=0x55b5b21150a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b21150a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f6a54024040 tx=0x7f6a54016000 comp rx=0 tx=0).stop 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1296060206 shutdown_connections 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b212c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b212c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/1296060206 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b21150a0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1296060206 >> 172.21.15.94:0/1296060206 conn(0x55b5b20f24b0 msgr2=0x55b5b2119c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1296060206 shutdown_connections 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.221+0000 7f6a6cb77740 1 -- 172.21.15.94:0/1296060206 wait complete. 2024-03-06T07:22:52.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 -- start start 2024-03-06T07:22:52.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 Processor -- start 2024-03-06T07:22:52.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b212c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b2133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b219c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b211afa0 con 0x55b5b20f3f50 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b2144840 con 0x55b5b2116600 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5b2140850 con 0x55b5b20f4330 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b212c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a6cb77740 10 client.0 _ll_register_callbacks cb 0x7f6a5c02c9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b212c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51010/0 (socket says 172.21.15.94:51010) 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.222+0000 7f6a637fe640 1 -- 172.21.15.94:0/2683396533 learned_addr learned my addr 172.21.15.94:0/2683396533 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a63fff640 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b219c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b2133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a637fe640 1 -- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 msgr2=0x55b5b2133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a637fe640 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b2133bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a637fe640 1 -- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 msgr2=0x55b5b219c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a637fe640 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b219c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a637fe640 1 -- 172.21.15.94:0/2683396533 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5b20ed5b0 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a637fe640 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b212c840 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6a54011690 tx=0x7f6a54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba55984ea9b14029 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a427fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a427fc640 1 -- 172.21.15.94:0/2683396533 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54046060 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a427fc640 1 -- 172.21.15.94:0/2683396533 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6a54003f40 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a54027040 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a427fc640 1 -- 172.21.15.94:0/2683396533 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a5401b030 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a6cb77740 10 client.8013 fetch_fsmap learned FSMap version 104 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x55b5b2115750 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a427fc640 1 -- 172.21.15.94:0/2683396533 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a54027040 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a427fc640 1 -- 172.21.15.94:0/2683396533 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 104) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6a54035070 con 0x55b5b20f3f50 2024-03-06T07:22:52.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a6cb77740 10 client.8013 fetch_fsmap finished waiting for FSMap version 104 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a6cb77740 10 client.8013 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.223+0000 7f6a6cb77740 20 client.8013 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b219c6c0 0x55b5b21534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 4 client.8013 mds_command: new command op to 7842 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 --> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5b205ff40 con 0x55b5b219c6c0 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b219c6c0 0x55b5b21534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a62ffd640 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b219c6c0 0x55b5b21534c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4655761d90268bc5 server_cookie=42e96ec8cb2beab0 in_seq=0 out_seq=0 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a427fc640 10 client.8013 ms_handle_connect on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a427fc640 1 -- 172.21.15.94:0/2683396533 <== mds.0 v2:172.21.15.145:6834/198029515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55b5b205ff40 con 0x55b5b219c6c0 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a427fc640 10 client.8013 handle_command_reply: tid=0 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 client.8013 shutdown 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 msgr2=0x55b5b212c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b212c840 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f6a54011690 tx=0x7f6a54004730 comp rx=0 tx=0).stop 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 shutdown_connections 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55b5b219c6c0 0x55b5b21534c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b5b2116600 0x55b5b219c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a427fc640 0 client.8013 ms_handle_reset on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b5b20f4330 0x55b5b2133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 --2- 172.21.15.94:0/2683396533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b5b20f3f50 0x55b5b212c840 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 >> 172.21.15.94:0/2683396533 conn(0x55b5b20f24b0 msgr2=0x55b5b20e9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 shutdown_connections 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 1 -- 172.21.15.94:0/2683396533 wait complete. 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.224+0000 7f6a6cb77740 20 client.8013 trim_cache size 0 max 16384 2024-03-06T07:22:52.233 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7842, 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 8.939682522, 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 104, 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 121, 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 121, 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 14.031896456 2024-03-06T07:22:52.234 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a5006b4d0 msgr2=0x7f6a5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a5006b4d0 0x7f6a5006d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009aa0 tx=0x7f6a54009190 comp rx=0 tx=0).stop 2024-03-06T07:22:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 msgr2=0x7f6a6414ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414ac00 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800c4d0 tx=0x7f6a5800c990 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 shutdown_connections 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a5006b4d0 0x7f6a5006d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a641561a0 0x7f6a6414b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6a6414f110 0x7f6a6414ac00 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 --2- 172.21.15.94:0/4273918301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a640c6ab0 0x7f6a640bfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 >> 172.21.15.94:0/4273918301 conn(0x7f6a6406b1a0 msgr2=0x7f6a64144860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 shutdown_connections 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.225+0000 7f6a69eb8640 1 -- 172.21.15.94:0/4273918301 wait complete. 2024-03-06T07:22:52.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:52.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.336+0000 7f956a555640 1 Processor -- start 2024-03-06T07:22:52.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.336+0000 7f956a555640 1 -- start start 2024-03-06T07:22:52.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f956a555640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f956414f360 0x7f9564155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f956a555640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f9564158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f956a555640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95640c73d0 con 0x7f95640c88c0 2024-03-06T07:22:52.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f956a555640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95640c8dd0 con 0x7f956414f360 2024-03-06T07:22:52.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f956a555640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95640be2e0 con 0x7f9564158b60 2024-03-06T07:22:52.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f9568af2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f9564158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f9563fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f956414f360 0x7f9564155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f9568af2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f9564158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:49398/0 (socket says 172.21.15.94:49398) 2024-03-06T07:22:52.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f95637fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f95640c88c0 0x7f95640c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:51626/0 (socket says 172.21.15.94:51626) 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.337+0000 7f9568af2640 1 -- 172.21.15.94:0/2875873943 learned_addr learned my addr 172.21.15.94:0/2875873943 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.338+0000 7f9563fff640 1 -- 172.21.15.94:0/2875873943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 msgr2=0x7f9564158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.338+0000 7f9563fff640 1 --2- 172.21.15.94:0/2875873943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f9564158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.338+0000 7f9563fff640 1 -- 172.21.15.94:0/2875873943 >> v1:172.21.15.145:6789/0 conn(0x7f95640c88c0 legacy=0x7f95640c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.338+0000 7f9563fff640 1 -- 172.21.15.94:0/2875873943 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95640c7670 con 0x7f956414f360 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.338+0000 7f9563fff640 1 --2- 172.21.15.94:0/2875873943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956414f360 0x7f9564155c10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f95540097a0 tx=0x7f955402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb2dbffe84bbe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f9562ffd640 1 -- 172.21.15.94:0/2875873943 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95540315a0 con 0x7f956414f360 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f9562ffd640 1 -- 172.21.15.94:0/2875873943 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9554037070 con 0x7f956414f360 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f9562ffd640 1 -- 172.21.15.94:0/2875873943 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554031960 con 0x7f956414f360 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 -- 172.21.15.94:0/2875873943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956414f360 msgr2=0x7f9564155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 --2- 172.21.15.94:0/2875873943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956414f360 0x7f9564155c10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f95540097a0 tx=0x7f955402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 -- 172.21.15.94:0/2875873943 shutdown_connections 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 --2- 172.21.15.94:0/2875873943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f9564158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 --2- 172.21.15.94:0/2875873943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956414f360 0x7f9564155c10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 -- 172.21.15.94:0/2875873943 >> 172.21.15.94:0/2875873943 conn(0x7f95640ac570 msgr2=0x7f95640ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 -- 172.21.15.94:0/2875873943 shutdown_connections 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.339+0000 7f956a555640 1 -- 172.21.15.94:0/2875873943 wait complete. 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.340+0000 7f956a555640 1 Processor -- start 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.340+0000 7f956a555640 1 -- start start 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f956a555640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640c88c0 0x7f956414ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f956a555640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414f360 0x7f956414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f956a555640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f95641ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f956a555640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640c5a50 con 0x7f956414f360 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f956a555640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640c7c30 con 0x7f95640c88c0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f956a555640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95640c5bd0 con 0x7f9564158b60 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f95637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414f360 0x7f956414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f9563fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640c88c0 0x7f956414ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f95637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414f360 0x7f956414bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51016/0 (socket says 172.21.15.94:51016) 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f95637fe640 1 -- 172.21.15.94:0/55833836 learned_addr learned my addr 172.21.15.94:0/55833836 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f9568af2640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f95641ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f9563fff640 1 -- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 msgr2=0x7f95641ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f9563fff640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f95641ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f9563fff640 1 -- 172.21.15.94:0/55833836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414f360 msgr2=0x7f956414bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.341+0000 7f9563fff640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414f360 0x7f956414bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f9563fff640 1 -- 172.21.15.94:0/55833836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95641ec060 con 0x7f95640c88c0 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f95637fe640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414f360 0x7f956414bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f9563fff640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640c88c0 0x7f956414ba50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9554033000 tx=0x7f95540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f9560ff9640 1 -- 172.21.15.94:0/55833836 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554032630 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f9560ff9640 1 -- 172.21.15.94:0/55833836 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95540327f0 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95640ae910 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f9560ff9640 1 -- 172.21.15.94:0/55833836 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554032be0 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.342+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95640c1390 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.343+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956404f330 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.343+0000 7f9560ff9640 1 -- 172.21.15.94:0/55833836 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f955404e430 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.344+0000 7f9560ff9640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.344+0000 7f9560ff9640 1 -- 172.21.15.94:0/55833836 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f955402f030 con 0x7f95640c88c0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.344+0000 7f95637fe640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.344+0000 7f95637fe640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f95640ad030 tx=0x7f9558007ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.348+0000 7f9560ff9640 1 -- 172.21.15.94:0/55833836 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9554040070 con 0x7f95640c88c0 2024-03-06T07:22:52.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.557+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95640aee30 con 0x7f95640c88c0 2024-03-06T07:22:52.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.558+0000 7f9560ff9640 1 -- 172.21.15.94:0/55833836 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4985 (secure 0 0 0) 0x7f955407a6f0 con 0x7f95640c88c0 2024-03-06T07:22:52.560 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:52.560 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7827,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/282207355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":282207355},{"type":"v1","addr":"172.21.15.94:6834","nonce":282207355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:22:44.626233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7842},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7842":{"gid":7842,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/198029515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":198029515},{"type":"v1","addr":"172.21.15.145:6836","nonce":198029515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7842,"qdb_cluster":[7842]},"id":9}]} 2024-03-06T07:22:52.560 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 104 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 msgr2=0x7f955006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f95640ad030 tx=0x7f9558007ee0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640c88c0 msgr2=0x7f956414ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640c88c0 0x7f956414ba50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9554033000 tx=0x7f95540046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 shutdown_connections 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564158b60 0x7f95641ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414f360 0x7f956414bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 --2- 172.21.15.94:0/55833836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640c88c0 0x7f956414ba50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 >> 172.21.15.94:0/55833836 conn(0x7f95640ac570 msgr2=0x7f95640ca040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 shutdown_connections 2024-03-06T07:22:52.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.560+0000 7f956a555640 1 -- 172.21.15.94:0/55833836 wait complete. 2024-03-06T07:22:52.573 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 104, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:42.624925+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 104, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7842": { "addr": "172.21.15.145:6836/198029515", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 198029515, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 198029515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7842, "incarnation": 102, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:22:44.626233+0000", "qdb_cluster": [ 7842 ], "qdb_leader": 7842, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7842 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/1356561672", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 1356561672, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 1356561672, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/1458830160", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 1458830160, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 1458830160, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7824, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/282207355", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 282207355, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 282207355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7827, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:52.574 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:22:52.574 INFO:tasks.cephfs.test_failover:current = [{'gid': 7824, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/1458830160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 1458830160}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 1458830160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7842, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/198029515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 198029515}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 198029515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7827, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/282207355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 282207355}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 282207355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/1356561672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1356561672}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1356561672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:22:52.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-06T07:22:52.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.676+0000 7f94b2bac640 1 Processor -- start 2024-03-06T07:22:52.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.676+0000 7f94b2bac640 1 -- start start 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b2bac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f94ac0a8ff0 0x7f94ac0ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b2bac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0ab730 0x7f94ac0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b2bac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac0c1f90 con 0x7f94ac0a6290 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b2bac640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac14f850 con 0x7f94ac0a8ff0 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b2bac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac156260 con 0x7f94ac0ab730 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b1149640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0ab730 0x7f94ac0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b0948640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f94ac0a8ff0 0x7f94ac0ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94abfff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f94ac0a6290 0x7f94ac0a9e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:51632/0 (socket says 172.21.15.94:51632) 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94abfff640 1 -- 172.21.15.94:0/558650767 learned_addr learned my addr 172.21.15.94:0/558650767 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b1149640 1 -- 172.21.15.94:0/558650767 >> v2:172.21.15.145:3300/0 conn(0x7f94ac0a8ff0 msgr2=0x7f94ac0ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b1149640 1 --2- 172.21.15.94:0/558650767 >> v2:172.21.15.145:3300/0 conn(0x7f94ac0a8ff0 0x7f94ac0ab1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b1149640 1 -- 172.21.15.94:0/558650767 >> v1:172.21.15.145:6789/0 conn(0x7f94ac0a6290 legacy=0x7f94ac0a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b1149640 1 -- 172.21.15.94:0/558650767 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac0c11d0 con 0x7f94ac0ab730 2024-03-06T07:22:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.677+0000 7f94b0948640 1 --2- 172.21.15.94:0/558650767 >> v2:172.21.15.145:3300/0 conn(0x7f94ac0a8ff0 0x7f94ac0ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.678+0000 7f94b1149640 1 --2- 172.21.15.94:0/558650767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0ab730 0x7f94ac0aa530 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009d90 tx=0x7f94a402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ae952042c92614 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.678+0000 7f94ab7fe640 1 -- 172.21.15.94:0/558650767 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4039070 con 0x7f94ac0ab730 2024-03-06T07:22:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.678+0000 7f94ab7fe640 1 -- 172.21.15.94:0/558650767 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a402cb30 con 0x7f94ac0ab730 2024-03-06T07:22:52.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.678+0000 7f94ab7fe640 1 -- 172.21.15.94:0/558650767 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4034440 con 0x7f94ac0ab730 2024-03-06T07:22:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 -- 172.21.15.94:0/558650767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0ab730 msgr2=0x7f94ac0aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558650767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0ab730 0x7f94ac0aa530 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009d90 tx=0x7f94a402bd80 comp rx=0 tx=0).stop 2024-03-06T07:22:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 -- 172.21.15.94:0/558650767 shutdown_connections 2024-03-06T07:22:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558650767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0ab730 0x7f94ac0aa530 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558650767 >> v2:172.21.15.145:3300/0 conn(0x7f94ac0a8ff0 0x7f94ac0ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 -- 172.21.15.94:0/558650767 >> 172.21.15.94:0/558650767 conn(0x7f94ac06afc0 msgr2=0x7f94ac06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 -- 172.21.15.94:0/558650767 shutdown_connections 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.679+0000 7f94b2bac640 1 -- 172.21.15.94:0/558650767 wait complete. 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.680+0000 7f94b2bac640 1 Processor -- start 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.680+0000 7f94b2bac640 1 -- start start 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b2bac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0a8ff0 0x7f94ac0bd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b2bac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94ac0ab730 0x7f94ac072650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b2bac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94ac14c4f0 0x7f94ac0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b2bac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0c1a70 con 0x7f94ac14c4f0 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b2bac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0c1790 con 0x7f94ac0ab730 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b2bac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac155cf0 con 0x7f94ac0a8ff0 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94ac14c4f0 0x7f94ac0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94ac14c4f0 0x7f94ac0c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51030/0 (socket says 172.21.15.94:51030) 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 -- 172.21.15.94:0/558142819 learned_addr learned my addr 172.21.15.94:0/558142819 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94abfff640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94ac0ab730 0x7f94ac072650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b0948640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0a8ff0 0x7f94ac0bd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 -- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0a8ff0 msgr2=0x7f94ac0bd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0a8ff0 0x7f94ac0bd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 -- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94ac0ab730 msgr2=0x7f94ac072650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94ac0ab730 0x7f94ac072650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 -- 172.21.15.94:0/558142819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac0c7100 con 0x7f94ac14c4f0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b1149640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94ac14c4f0 0x7f94ac0c5720 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009eb0 tx=0x7f94a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94a97fa640 1 -- 172.21.15.94:0/558142819 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a4039050 con 0x7f94ac14c4f0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94a97fa640 1 -- 172.21.15.94:0/558142819 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a4045070 con 0x7f94ac14c4f0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94a97fa640 1 -- 172.21.15.94:0/558142819 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a403e830 con 0x7f94ac14c4f0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.681+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac0c73e0 con 0x7f94ac14c4f0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.682+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac072b90 con 0x7f94ac14c4f0 2024-03-06T07:22:52.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.682+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac04f330 con 0x7f94ac14c4f0 2024-03-06T07:22:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.683+0000 7f94a97fa640 1 -- 172.21.15.94:0/558142819 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94a403e9d0 con 0x7f94ac14c4f0 2024-03-06T07:22:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.683+0000 7f94a97fa640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f949006b4d0 0x7f949006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.683+0000 7f94a97fa640 1 -- 172.21.15.94:0/558142819 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f94a40ad200 con 0x7f94ac14c4f0 2024-03-06T07:22:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.683+0000 7f94b0948640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f949006b4d0 0x7f949006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.684+0000 7f94b0948640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f949006b4d0 0x7f949006d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f94a0009560 tx=0x7f94a0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.688+0000 7f94a97fa640 1 -- 172.21.15.94:0/558142819 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94a407ba50 con 0x7f94ac14c4f0 2024-03-06T07:22:52.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.859+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f94ac0c24e0 con 0x7f94ac14c4f0 2024-03-06T07:22:52.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.861+0000 7f94a97fa640 1 -- 172.21.15.94:0/558142819 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v42)=0 v42) v1 ==== 116+0+0 (secure 0 0 0) 0x7f94a407faf0 con 0x7f94ac14c4f0 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.863+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f949006b4d0 msgr2=0x7f949006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.863+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f949006b4d0 0x7f949006d990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f94a0009560 tx=0x7f94a0009190 comp rx=0 tx=0).stop 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94ac14c4f0 msgr2=0x7f94ac0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94ac14c4f0 0x7f94ac0c5720 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009eb0 tx=0x7f94a4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 shutdown_connections 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f949006b4d0 0x7f949006d990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94ac14c4f0 0x7f94ac0c5720 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94ac0ab730 0x7f94ac072650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 --2- 172.21.15.94:0/558142819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94ac0a8ff0 0x7f94ac0bd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 >> 172.21.15.94:0/558142819 conn(0x7f94ac06afc0 msgr2=0x7f94ac06dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 shutdown_connections 2024-03-06T07:22:52.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.864+0000 7f94b2bac640 1 -- 172.21.15.94:0/558142819 wait complete. 2024-03-06T07:22:52.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:52.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.974+0000 7fbae8adf640 1 Processor -- start 2024-03-06T07:22:52.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.974+0000 7fbae8adf640 1 -- start start 2024-03-06T07:22:52.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae8adf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbae40c1320 0x7fbae41566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae8adf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbae40c6000 0x7fbae4156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae8adf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae4153a60 con 0x7fbae4150580 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae8adf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae4153d40 con 0x7fbae40c6000 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae8adf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae406c2c0 con 0x7fbae40c1320 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae2d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbae40c6000 0x7fbae4156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae2d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbae40c6000 0x7fbae4156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51032/0 (socket says 172.21.15.94:51032) 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbae4150580 0x7fbae406dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37110/0 (socket says 172.21.15.94:37110) 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae2d9d640 1 -- 172.21.15.94:0/2495755326 learned_addr learned my addr 172.21.15.94:0/2495755326 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae1d9b640 1 --2- 172.21.15.94:0/2495755326 >> v2:172.21.15.145:3300/0 conn(0x7fbae40c1320 0x7fbae41566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae159a640 1 -- 172.21.15.94:0/2495755326 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204275973 0 0) 0x7fbae4153a60 con 0x7fbae4150580 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae159a640 1 -- 172.21.15.94:0/2495755326 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbacc003520 con 0x7fbae4150580 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae2d9d640 1 -- 172.21.15.94:0/2495755326 >> v2:172.21.15.145:3300/0 conn(0x7fbae40c1320 msgr2=0x7fbae41566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae2d9d640 1 --2- 172.21.15.94:0/2495755326 >> v2:172.21.15.145:3300/0 conn(0x7fbae40c1320 0x7fbae41566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae2d9d640 1 -- 172.21.15.94:0/2495755326 >> v1:172.21.15.94:6789/0 conn(0x7fbae4150580 legacy=0x7fbae406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.975+0000 7fbae2d9d640 1 -- 172.21.15.94:0/2495755326 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae406c5a0 con 0x7fbae40c6000 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.976+0000 7fbae2d9d640 1 --2- 172.21.15.94:0/2495755326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c6000 0x7fbae4156bf0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fbae40c7730 tx=0x7fbadc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2438c79f86cc37e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.976+0000 7fbae159a640 1 -- 172.21.15.94:0/2495755326 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc02ca80 con 0x7fbae40c6000 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.976+0000 7fbae159a640 1 -- 172.21.15.94:0/2495755326 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc02cc20 con 0x7fbae40c6000 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.976+0000 7fbae159a640 1 -- 172.21.15.94:0/2495755326 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0343f0 con 0x7fbae40c6000 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 -- 172.21.15.94:0/2495755326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c6000 msgr2=0x7fbae4156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 --2- 172.21.15.94:0/2495755326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c6000 0x7fbae4156bf0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fbae40c7730 tx=0x7fbadc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 -- 172.21.15.94:0/2495755326 shutdown_connections 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 --2- 172.21.15.94:0/2495755326 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c6000 0x7fbae4156bf0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 --2- 172.21.15.94:0/2495755326 >> v2:172.21.15.145:3300/0 conn(0x7fbae40c1320 0x7fbae41566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 -- 172.21.15.94:0/2495755326 >> 172.21.15.94:0/2495755326 conn(0x7fbae40c6710 msgr2=0x7fbae40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 -- 172.21.15.94:0/2495755326 shutdown_connections 2024-03-06T07:22:52.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.977+0000 7fbae8adf640 1 -- 172.21.15.94:0/2495755326 wait complete. 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.978+0000 7fbae8adf640 1 Processor -- start 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.978+0000 7fbae8adf640 1 -- start start 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c1320 0x7fbae41e85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbae40c6000 0x7fbae41e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbae4150580 0x7fbae4152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae406e5a0 con 0x7fbae40c1320 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae406e420 con 0x7fbae40c6000 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40c0c40 con 0x7fbae4150580 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c1320 0x7fbae41e85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c1320 0x7fbae41e85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51046/0 (socket says 172.21.15.94:51046) 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 -- 172.21.15.94:0/4102856943 learned_addr learned my addr 172.21.15.94:0/4102856943 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae1d9b640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbae40c6000 0x7fbae41e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae2d9d640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbae4150580 0x7fbae4152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 -- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbae4150580 msgr2=0x7fbae4152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbae4150580 0x7fbae4152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 -- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbae40c6000 msgr2=0x7fbae41e5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbae40c6000 0x7fbae41e5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 -- 172.21.15.94:0/4102856943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae41e6260 con 0x7fbae40c1320 2024-03-06T07:22:52.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae2d9d640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbae4150580 0x7fbae4152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae1d9b640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbae40c6000 0x7fbae41e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae259c640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c1320 0x7fbae41e85f0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbad400c3c0 tx=0x7fbad400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbad2ffd640 1 -- 172.21.15.94:0/4102856943 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad40074e0 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbad2ffd640 1 -- 172.21.15.94:0/4102856943 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad40047e0 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbad2ffd640 1 -- 172.21.15.94:0/4102856943 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad4004bd0 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae4072900 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.979+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae4072ec0 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.980+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae41ebbe0 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.981+0000 7fbad2ffd640 1 -- 172.21.15.94:0/4102856943 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbad401a020 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.981+0000 7fbad2ffd640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbac406b4d0 0x7fbac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.981+0000 7fbad2ffd640 1 -- 172.21.15.94:0/4102856943 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbad4089b30 con 0x7fbae40c1320 2024-03-06T07:22:52.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.981+0000 7fbae1d9b640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbac406b4d0 0x7fbac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.982+0000 7fbae1d9b640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbac406b4d0 0x7fbac406d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbae40c3c00 tx=0x7fbad8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:52.986+0000 7fbad2ffd640 1 -- 172.21.15.94:0/4102856943 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbad4010300 con 0x7fbae40c1320 2024-03-06T07:22:53.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.191+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbae414fd00 con 0x7fbae40c1320 2024-03-06T07:22:53.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.191+0000 7fbad2ffd640 1 -- 172.21.15.94:0/4102856943 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4985 (secure 0 0 0) 0x7fbad4058970 con 0x7fbae40c1320 2024-03-06T07:22:53.194 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:53.194 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":104,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7827,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/282207355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":282207355},{"type":"v1","addr":"172.21.15.94:6834","nonce":282207355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:22:44.626233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7842},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7842":{"gid":7842,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/198029515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":198029515},{"type":"v1","addr":"172.21.15.145:6836","nonce":198029515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7842,"qdb_cluster":[7842]},"id":9}]} 2024-03-06T07:22:53.194 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 104 2024-03-06T07:22:53.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbac406b4d0 msgr2=0x7fbac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbac406b4d0 0x7fbac406d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbae40c3c00 tx=0x7fbad8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c1320 msgr2=0x7fbae41e85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c1320 0x7fbae41e85f0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbad400c3c0 tx=0x7fbad400c880 comp rx=0 tx=0).stop 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 shutdown_connections 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbac406b4d0 0x7fbac406d990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbae4150580 0x7fbae4152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbae40c6000 0x7fbae41e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 --2- 172.21.15.94:0/4102856943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbae40c1320 0x7fbae41e85f0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 >> 172.21.15.94:0/4102856943 conn(0x7fbae40c6710 msgr2=0x7fbae40a8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 shutdown_connections 2024-03-06T07:22:53.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.193+0000 7fbae8adf640 1 -- 172.21.15.94:0/4102856943 wait complete. 2024-03-06T07:22:53.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:42.624925+0000', 'modified': '2024-03-06T07:22:44.626233+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7842}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7842': {'gid': 7842, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/198029515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 198029515}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 198029515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7842, 'qdb_cluster': [7842]} 2024-03-06T07:22:53.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:53.609 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:22:53.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.710+0000 7f0afaba1640 1 Processor -- start 2024-03-06T07:22:53.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.710+0000 7f0afaba1640 1 -- start start 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0afaba1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af41551f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0afaba1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0af40a7ff0 0x7f0af4155730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0afaba1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af40733f0 con 0x7f0af4152b40 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0afaba1640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0af4072d70 con 0x7f0af40a7ff0 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0afaba1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af40c82c0 con 0x7f0af40724d0 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0af913e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0af40a7ff0 0x7f0af4155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0af913e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0af40a7ff0 0x7f0af4155730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51050/0 (socket says 172.21.15.94:51050) 2024-03-06T07:22:53.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0af913e640 1 -- 172.21.15.94:0/1776193160 learned_addr learned my addr 172.21.15.94:0/1776193160 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.711+0000 7f0af3fff640 1 --2- 172.21.15.94:0/1776193160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af41551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af913e640 1 -- 172.21.15.94:0/1776193160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 msgr2=0x7f0af41551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af913e640 1 --2- 172.21.15.94:0/1776193160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af41551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af913e640 1 -- 172.21.15.94:0/1776193160 >> v1:172.21.15.145:6789/0 conn(0x7f0af4152b40 legacy=0x7f0af40c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af37fe640 1 -- 172.21.15.94:0/1776193160 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997562631 0 0) 0x7f0af40733f0 con 0x7f0af4152b40 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af913e640 1 -- 172.21.15.94:0/1776193160 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af406fd30 con 0x7f0af40a7ff0 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af913e640 1 --2- 172.21.15.94:0/1776193160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af40a7ff0 0x7f0af4155730 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009e50 tx=0x7f0aec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e3515c24594c6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af37fe640 1 -- 172.21.15.94:0/1776193160 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec02ca70 con 0x7f0af40a7ff0 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af37fe640 1 -- 172.21.15.94:0/1776193160 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aec02cc10 con 0x7f0af40a7ff0 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.712+0000 7f0af37fe640 1 -- 172.21.15.94:0/1776193160 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0343f0 con 0x7f0af40a7ff0 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 -- 172.21.15.94:0/1776193160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af40a7ff0 msgr2=0x7f0af4155730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 --2- 172.21.15.94:0/1776193160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af40a7ff0 0x7f0af4155730 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009e50 tx=0x7f0aec02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 -- 172.21.15.94:0/1776193160 shutdown_connections 2024-03-06T07:22:53.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 --2- 172.21.15.94:0/1776193160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af40a7ff0 0x7f0af4155730 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 --2- 172.21.15.94:0/1776193160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af41551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 -- 172.21.15.94:0/1776193160 >> 172.21.15.94:0/1776193160 conn(0x7f0af406b970 msgr2=0x7f0af40aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 -- 172.21.15.94:0/1776193160 shutdown_connections 2024-03-06T07:22:53.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.713+0000 7f0afaba1640 1 -- 172.21.15.94:0/1776193160 wait complete. 2024-03-06T07:22:53.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.714+0000 7f0afaba1640 1 Processor -- start 2024-03-06T07:22:53.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.714+0000 7f0afaba1640 1 -- start start 2024-03-06T07:22:53.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0afaba1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af4156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0afaba1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0af40a7ff0 0x7f0af41eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0afaba1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af4152b40 0x7f0af41eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0afaba1640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af4155df0 con 0x7f0af4152b40 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0afaba1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af4155c70 con 0x7f0af40a7ff0 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0afaba1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af4070280 con 0x7f0af40724d0 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af4152b40 0x7f0af41eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af4152b40 0x7f0af41eb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51052/0 (socket says 172.21.15.94:51052) 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 -- 172.21.15.94:0/3073486243 learned_addr learned my addr 172.21.15.94:0/3073486243 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af3fff640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0af40a7ff0 0x7f0af41eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af893d640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af4156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 -- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 msgr2=0x7f0af4156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af4156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 -- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0af40a7ff0 msgr2=0x7f0af41eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0af40a7ff0 0x7f0af41eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 -- 172.21.15.94:0/3073486243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af41590c0 con 0x7f0af4152b40 2024-03-06T07:22:53.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.715+0000 7f0af913e640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af4152b40 0x7f0af41eb670 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009f70 tx=0x7f0aec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.716+0000 7f0af17fa640 1 -- 172.21.15.94:0/3073486243 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec043020 con 0x7f0af4152b40 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.716+0000 7f0af17fa640 1 -- 172.21.15.94:0/3073486243 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aec03e450 con 0x7f0af4152b40 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.716+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af41593a0 con 0x7f0af4152b40 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.716+0000 7f0af17fa640 1 -- 172.21.15.94:0/3073486243 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec009510 con 0x7f0af4152b40 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.716+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af40c5f90 con 0x7f0af4152b40 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.717+0000 7f0af17fa640 1 -- 172.21.15.94:0/3073486243 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0aec046020 con 0x7f0af4152b40 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.717+0000 7f0af17fa640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ae006b4d0 0x7f0ae006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.718+0000 7f0af17fa640 1 -- 172.21.15.94:0/3073486243 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0aec0ac9f0 con 0x7f0af4152b40 2024-03-06T07:22:53.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.718+0000 7f0af893d640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ae006b4d0 0x7f0ae006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.718+0000 7f0af893d640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ae006b4d0 0x7f0ae006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4000f60 tx=0x7f0ae4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 Processor -- start 2024-03-06T07:22:53.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 -- start start 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208a91f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a924b0 0x560208a8e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602089f86b0 con 0x560208a94330 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602089a1a60 con 0x560208a93f50 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602089f4750 con 0x560208a924b0 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0af3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208a91f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0af3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208a91f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51062/0 (socket says 172.21.15.94:51062) 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0af3fff640 1 -- 172.21.15.94:0/4010890867 learned_addr learned my addr 172.21.15.94:0/4010890867 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0af913e640 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a924b0 0x560208a8e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.719+0000 7f0af893d640 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0af3fff640 1 -- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a924b0 msgr2=0x560208a8e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0af3fff640 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a924b0 0x560208a8e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0af3fff640 1 -- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 msgr2=0x560208a91a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0af3fff640 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0af3fff640 1 -- 172.21.15.94:0/4010890867 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602089bff30 con 0x560208a94330 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0af3fff640 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208a91f70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009c10 tx=0x7f0ae8011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61cf853d77640cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0ada7fc640 1 -- 172.21.15.94:0/4010890867 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8019070 con 0x560208a94330 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0ada7fc640 1 -- 172.21.15.94:0/4010890867 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae8004030 con 0x560208a94330 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0ada7fc640 1 -- 172.21.15.94:0/4010890867 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8004420 con 0x560208a94330 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0afd788740 1 -- 172.21.15.94:0/4010890867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 msgr2=0x560208a91f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.720+0000 7f0afd788740 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208a91f70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8009c10 tx=0x7f0ae8011690 comp rx=0 tx=0).stop 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 -- 172.21.15.94:0/4010890867 shutdown_connections 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a924b0 0x560208a8e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208a91f70 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 --2- 172.21.15.94:0/4010890867 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 -- 172.21.15.94:0/4010890867 >> 172.21.15.94:0/4010890867 conn(0x560208a907d0 msgr2=0x560208ab53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 -- 172.21.15.94:0/4010890867 shutdown_connections 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 -- 172.21.15.94:0/4010890867 wait complete. 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 -- start start 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.721+0000 7f0afd788740 1 Processor -- start 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208ab47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0afd788740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208ae07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208ab0dc0 0x560208ab1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0afd788740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x560208ab6230 con 0x560208a94330 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602089b4ea0 con 0x560208a93f50 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x560208ab5c70 con 0x560208ab0dc0 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0afd788740 10 client.0 _ll_register_callbacks cb 0x5602089b4870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:53.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208ae07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208ae07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51066/0 (socket says 172.21.15.94:51066) 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 -- 172.21.15.94:0/1678863389 learned_addr learned my addr 172.21.15.94:0/1678863389 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af893d640 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208ab47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af913e640 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208ab0dc0 0x560208ab1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 -- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208ab0dc0 msgr2=0x560208ab1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208ab0dc0 0x560208ab1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 -- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 msgr2=0x560208ab47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208ab47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.722+0000 7f0af3fff640 1 -- 172.21.15.94:0/1678863389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602089b5270 con 0x560208a94330 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0af3fff640 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208ae07b0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8018000 tx=0x7f0ae8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b1e28134547c6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0ad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0ad37fe640 1 -- 172.21.15.94:0/1678863389 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae8019040 con 0x560208a94330 2024-03-06T07:22:53.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0ad37fe640 1 -- 172.21.15.94:0/1678863389 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0ae8003a20 con 0x560208a94330 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ae8004030 con 0x560208a94330 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0ad37fe640 1 -- 172.21.15.94:0/1678863389 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ae8003c20 con 0x560208a94330 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 10 client.8040 fetch_fsmap learned FSMap version 104 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x56020870d2d0 con 0x560208a94330 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0ad37fe640 1 -- 172.21.15.94:0/1678863389 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae8004030 con 0x560208a94330 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0ad37fe640 1 -- 172.21.15.94:0/1678863389 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 104) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0ae8005990 con 0x560208a94330 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 10 client.8040 fetch_fsmap finished waiting for FSMap version 104 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 10 client.8040 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 20 client.8040 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208ae4640 0x560208aef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 4 client.8040 mds_command: new command op to 7842 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 --> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560208a05bf0 con 0x560208ae4640 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.723+0000 7f0af893d640 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208ae4640 0x560208aef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.724+0000 7f0ad37fe640 10 client.8040 ms_handle_connect on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.724+0000 7f0af893d640 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208ae4640 0x560208aef3a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2514b1444eb33cf server_cookie=e39f8ecef236f075 in_seq=0 out_seq=0 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.724+0000 7f0ad37fe640 1 -- 172.21.15.94:0/1678863389 <== mds.0 v2:172.21.15.145:6834/198029515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560208a05bf0 con 0x560208ae4640 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.724+0000 7f0ad37fe640 10 client.8040 handle_command_reply: tid=0 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.724+0000 7f0afd788740 1 client.8040 shutdown 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 msgr2=0x560208ae07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208ae07b0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8018000 tx=0x7f0ae8004730 comp rx=0 tx=0).stop 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 shutdown_connections 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208ae4640 0x560208aef3a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208ab0dc0 0x560208ab1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208a94330 0x560208ae07b0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0ad37fe640 0 client.8040 ms_handle_reset on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 --2- 172.21.15.94:0/1678863389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208ab47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 >> 172.21.15.94:0/1678863389 conn(0x560208a924b0 msgr2=0x560208a8f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 shutdown_connections 2024-03-06T07:22:53.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 1 -- 172.21.15.94:0/1678863389 wait complete. 2024-03-06T07:22:53.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.725+0000 7f0afd788740 20 client.8040 trim_cache size 0 max 16384 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.741+0000 7f0afd788740 1 Processor -- start 2024-03-06T07:22:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.741+0000 7f0afd788740 1 -- start start 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a93f50 0x560208a8d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a94330 0x560208acd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0afd788740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208acd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0afd788740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af41f6d80 con 0x560208ab0dc0 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af41eec50 con 0x560208a94330 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af41f6780 con 0x560208a93f50 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208acd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208acd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51080/0 (socket says 172.21.15.94:51080) 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 -- 172.21.15.94:0/228464961 learned_addr learned my addr 172.21.15.94:0/228464961 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af893d640 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a93f50 0x560208a8d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af3fff640 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a94330 0x560208acd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 -- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a93f50 msgr2=0x560208a8d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a93f50 0x560208a8d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 -- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a94330 msgr2=0x560208acd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a94330 0x560208acd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 -- 172.21.15.94:0/228464961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602087d76b0 con 0x560208ab0dc0 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af893d640 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a93f50 0x560208a8d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0af913e640 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208acd800 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0aad90 tx=0x7f0aec009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ae02c5d9b8da9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0ad9ffb640 1 -- 172.21.15.94:0/228464961 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec0a8070 con 0x560208ab0dc0 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0ad9ffb640 1 -- 172.21.15.94:0/228464961 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aec0836f0 con 0x560208ab0dc0 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.742+0000 7f0ad9ffb640 1 -- 172.21.15.94:0/228464961 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec077700 con 0x560208ab0dc0 2024-03-06T07:22:53.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 -- 172.21.15.94:0/228464961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 msgr2=0x560208acd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208acd800 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0aad90 tx=0x7f0aec009040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 -- 172.21.15.94:0/228464961 shutdown_connections 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208acd800 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a94330 0x560208acd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 --2- 172.21.15.94:0/228464961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a93f50 0x560208a8d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 -- 172.21.15.94:0/228464961 >> 172.21.15.94:0/228464961 conn(0x560208a924b0 msgr2=0x560208ab84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 -- 172.21.15.94:0/228464961 shutdown_connections 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 -- 172.21.15.94:0/228464961 wait complete. 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.743+0000 7f0afd788740 1 -- start start 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 1 Processor -- start 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a94330 0x560208a94bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208b3d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x560208ae1050 con 0x560208ab0dc0 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x560208ab8e90 con 0x560208a93f50 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x560208ab1780 con 0x560208a94330 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0afd788740 10 client.0 _ll_register_callbacks cb 0x7f0ae8009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0af913e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208b3d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0af913e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208b3d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51092/0 (socket says 172.21.15.94:51092) 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.744+0000 7f0af913e640 1 -- 172.21.15.94:0/2003706976 learned_addr learned my addr 172.21.15.94:0/2003706976 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af3fff640 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a94330 0x560208a94bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af893d640 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af913e640 1 -- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a94330 msgr2=0x560208a94bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af913e640 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a94330 0x560208a94bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af913e640 1 -- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 msgr2=0x560208a91770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af913e640 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af913e640 1 -- 172.21.15.94:0/2003706976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560208b43630 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0af913e640 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208b3d2b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f0aec07b250 tx=0x7f0aec090040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f29fab29da04fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0ad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0ad37fe640 1 -- 172.21.15.94:0/2003706976 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec09e060 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0ad37fe640 1 -- 172.21.15.94:0/2003706976 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0aec083260 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0aec0836f0 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0ad37fe640 1 -- 172.21.15.94:0/2003706976 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0aec098070 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 10 client.8046 fetch_fsmap learned FSMap version 104 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=2+}) v3 -- 0x560208af4680 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0ad37fe640 1 -- 172.21.15.94:0/2003706976 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aec0836f0 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0ad37fe640 1 -- 172.21.15.94:0/2003706976 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 104) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0aec0a4200 con 0x560208ab0dc0 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 10 client.8046 fetch_fsmap finished waiting for FSMap version 104 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 10 client.8046 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 20 client.8046 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:53.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208b3d7f0 0x560208aaf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 4 client.8046 mds_command: new command op to 7842 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.745+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 --> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5602089fff40 con 0x560208b3d7f0 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0af893d640 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208b3d7f0 0x560208aaf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0ad37fe640 10 client.8046 ms_handle_connect on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0af893d640 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208b3d7f0 0x560208aaf990 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85ff47a1dd74ae2c server_cookie=a8121cafd9ecd2d1 in_seq=0 out_seq=0 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0ad37fe640 1 -- 172.21.15.94:0/2003706976 <== mds.0 v2:172.21.15.145:6834/198029515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5602089fff40 con 0x560208b3d7f0 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0ad37fe640 10 client.8046 handle_command_reply: tid=0 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 client.8046 shutdown 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 msgr2=0x560208b3d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208b3d2b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f0aec07b250 tx=0x7f0aec090040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 shutdown_connections 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x560208b3d7f0 0x560208aaf990 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x560208ab0dc0 0x560208b3d2b0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0ad37fe640 0 client.8046 ms_handle_reset on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560208a94330 0x560208a94bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 --2- 172.21.15.94:0/2003706976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560208a93f50 0x560208a91770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 >> 172.21.15.94:0/2003706976 conn(0x560208a924b0 msgr2=0x560208a8d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 shutdown_connections 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.746+0000 7f0afd788740 1 -- 172.21.15.94:0/2003706976 wait complete. 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afd788740 20 client.8046 trim_cache size 0 max 16384 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:53.756 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7842, 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 10.461676873, 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 104, 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 121, 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 121, 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 15.553886127 2024-03-06T07:22:53.757 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ae006b4d0 msgr2=0x7f0ae006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ae006b4d0 0x7f0ae006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4000f60 tx=0x7f0ae4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af4152b40 msgr2=0x7f0af41eb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af4152b40 0x7f0af41eb670 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009f70 tx=0x7f0aec004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 shutdown_connections 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ae006b4d0 0x7f0ae006d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0af4152b40 0x7f0af41eb670 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0af40a7ff0 0x7f0af41eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 --2- 172.21.15.94:0/3073486243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0af40724d0 0x7f0af4156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 >> 172.21.15.94:0/3073486243 conn(0x7f0af406b970 msgr2=0x7f0af414e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 shutdown_connections 2024-03-06T07:22:53.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:53.747+0000 7f0afaba1640 1 -- 172.21.15.94:0/3073486243 wait complete. 2024-03-06T07:22:53.760 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 104, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:42.624925+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 104, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7842": { "addr": "172.21.15.145:6836/198029515", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 198029515, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 198029515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7842, "incarnation": 102, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:22:44.626233+0000", "qdb_cluster": [ 7842 ], "qdb_leader": 7842, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7842 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/1356561672", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 1356561672, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 1356561672, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/1458830160", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 1458830160, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 1458830160, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7824, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/282207355", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 282207355, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 282207355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7827, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:53.761 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 9, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:22:53.761 INFO:tasks.cephfs.test_failover:current = [{'gid': 7824, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/1458830160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 1458830160}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 1458830160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7842, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/198029515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 198029515}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 198029515}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7827, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/282207355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 282207355}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 282207355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/1356561672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1356561672}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1356561672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:22:53.761 DEBUG:tasks.cephfs.test_failover:9 != -1 2024-03-06T07:22:53.761 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:22:58.762 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.872+0000 7f166bf72640 1 Processor -- start 2024-03-06T07:22:58.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.872+0000 7f166bf72640 1 -- start start 2024-03-06T07:22:58.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.873+0000 7f166bf72640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f166414f360 0x7f1664155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.873+0000 7f166bf72640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640c73d0 con 0x7f1664158b60 2024-03-06T07:22:58.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.873+0000 7f166bf72640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640be2e0 con 0x7f16640c88c0 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.873+0000 7f166bf72640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1664159700 con 0x7f166414f360 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.873+0000 7f1669d0e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f166414f360 0x7f1664155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.873+0000 7f166a50f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1664158b60 0x7f1664158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49232/0 (socket says 172.21.15.94:49232) 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.873+0000 7f166a50f640 1 -- 172.21.15.94:0/649963108 learned_addr learned my addr 172.21.15.94:0/649963108 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.874+0000 7f1669d0e640 1 -- 172.21.15.94:0/649963108 >> v1:172.21.15.145:6789/0 conn(0x7f16640c88c0 legacy=0x7f16640c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.874+0000 7f1669d0e640 1 -- 172.21.15.94:0/649963108 >> v1:172.21.15.94:6789/0 conn(0x7f1664158b60 legacy=0x7f1664158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.874+0000 7f1669d0e640 1 -- 172.21.15.94:0/649963108 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640c7670 con 0x7f166414f360 2024-03-06T07:22:58.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.874+0000 7f1668d0c640 1 -- 172.21.15.94:0/649963108 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613977455 0 0) 0x7f16640c73d0 con 0x7f1664158b60 2024-03-06T07:22:58.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.874+0000 7f1669d0e640 1 --2- 172.21.15.94:0/649963108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166414f360 0x7f1664155c10 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f1660009d90 tx=0x7f166002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a5d1434806cd7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f1668d0c640 1 -- 172.21.15.94:0/649963108 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660039070 con 0x7f166414f360 2024-03-06T07:22:58.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f1668d0c640 1 -- 172.21.15.94:0/649963108 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166002cc60 con 0x7f166414f360 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f1668d0c640 1 -- 172.21.15.94:0/649963108 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660034660 con 0x7f166414f360 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f166bf72640 1 -- 172.21.15.94:0/649963108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166414f360 msgr2=0x7f1664155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f166bf72640 1 --2- 172.21.15.94:0/649963108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166414f360 0x7f1664155c10 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f1660009d90 tx=0x7f166002beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f166bf72640 1 -- 172.21.15.94:0/649963108 shutdown_connections 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f166bf72640 1 --2- 172.21.15.94:0/649963108 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166414f360 0x7f1664155c10 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f166bf72640 1 -- 172.21.15.94:0/649963108 >> 172.21.15.94:0/649963108 conn(0x7f16640ac570 msgr2=0x7f16640ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f166bf72640 1 -- 172.21.15.94:0/649963108 shutdown_connections 2024-03-06T07:22:58.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.875+0000 7f166bf72640 1 -- 172.21.15.94:0/649963108 wait complete. 2024-03-06T07:22:58.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.876+0000 7f166bf72640 1 Processor -- start 2024-03-06T07:22:58.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.876+0000 7f166bf72640 1 -- start start 2024-03-06T07:22:58.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166bf72640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16640c88c0 0x7f16641ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166bf72640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f166414f360 0x7f16641eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166bf72640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1664158b60 0x7f166414bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166bf72640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c7c30 con 0x7f16640c88c0 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166bf72640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c5a50 con 0x7f166414f360 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166bf72640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c5bd0 con 0x7f1664158b60 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16640c88c0 0x7f16641ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16640c88c0 0x7f16641ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47400/0 (socket says 172.21.15.94:47400) 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 -- 172.21.15.94:0/2317504140 learned_addr learned my addr 172.21.15.94:0/2317504140 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166a50f640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1664158b60 0x7f166414bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f166950d640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f166414f360 0x7f16641eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 -- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1664158b60 msgr2=0x7f166414bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1664158b60 0x7f166414bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 -- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f166414f360 msgr2=0x7f16641eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f166414f360 0x7f16641eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.877+0000 7f1669d0e640 1 -- 172.21.15.94:0/2317504140 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16641ef030 con 0x7f16640c88c0 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.878+0000 7f1669d0e640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16640c88c0 0x7f16641ee5b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f1660037040 tx=0x7f1660004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.878+0000 7f165a7fc640 1 -- 172.21.15.94:0/2317504140 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660039050 con 0x7f16640c88c0 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.878+0000 7f165a7fc640 1 -- 172.21.15.94:0/2317504140 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166003a040 con 0x7f16640c88c0 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.878+0000 7f165a7fc640 1 -- 172.21.15.94:0/2317504140 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660003bd0 con 0x7f16640c88c0 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.878+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16640c11b0 con 0x7f16640c88c0 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.878+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16641ec6c0 con 0x7f16640c88c0 2024-03-06T07:22:58.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.879+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166404f330 con 0x7f16640c88c0 2024-03-06T07:22:58.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.879+0000 7f165a7fc640 1 -- 172.21.15.94:0/2317504140 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1660047030 con 0x7f16640c88c0 2024-03-06T07:22:58.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.880+0000 7f165a7fc640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164806b4d0 0x7f164806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.880+0000 7f165a7fc640 1 -- 172.21.15.94:0/2317504140 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f16600ac7b0 con 0x7f16640c88c0 2024-03-06T07:22:58.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.880+0000 7f166950d640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164806b4d0 0x7f164806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.880+0000 7f166950d640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164806b4d0 0x7f164806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f16641549d0 tx=0x7f1654009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:58.884+0000 7f165a7fc640 1 -- 172.21.15.94:0/2317504140 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f166004e020 con 0x7f16640c88c0 2024-03-06T07:22:59.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.094+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16640c8d50 con 0x7f16640c88c0 2024-03-06T07:22:59.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.094+0000 7f165a7fc640 1 -- 172.21.15.94:0/2317504140 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4984 (secure 0 0 0) 0x7f166007aee0 con 0x7f16640c88c0 2024-03-06T07:22:59.096 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:22:59.096 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":105,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7827,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/282207355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":282207355},{"type":"v1","addr":"172.21.15.94:6834","nonce":282207355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:22:55.313430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7842},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7842":{"gid":7842,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":6,"addr":"172.21.15.145:6836/198029515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":198029515},{"type":"v1","addr":"172.21.15.145:6836","nonce":198029515}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7842,"qdb_cluster":[7842]},"id":9}]} 2024-03-06T07:22:59.097 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 105 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164806b4d0 msgr2=0x7f164806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164806b4d0 0x7f164806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f16641549d0 tx=0x7f1654009040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16640c88c0 msgr2=0x7f16641ee5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16640c88c0 0x7f16641ee5b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f1660037040 tx=0x7f1660004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 shutdown_connections 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164806b4d0 0x7f164806d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1664158b60 0x7f166414bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f166414f360 0x7f16641eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 --2- 172.21.15.94:0/2317504140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16640c88c0 0x7f16641ee5b0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 >> 172.21.15.94:0/2317504140 conn(0x7f16640ac570 msgr2=0x7f16640ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 shutdown_connections 2024-03-06T07:22:59.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.096+0000 7f166bf72640 1 -- 172.21.15.94:0/2317504140 wait complete. 2024-03-06T07:22:59.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:42.624925+0000', 'modified': '2024-03-06T07:22:55.313430+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7842}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7842': {'gid': 7842, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.145:6836/198029515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 198029515}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 198029515}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7842, 'qdb_cluster': [7842]} 2024-03-06T07:22:59.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:59.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:22:59.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.618+0000 7fec83925640 1 Processor -- start 2024-03-06T07:22:59.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.618+0000 7fec83925640 1 -- start start 2024-03-06T07:22:59.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec83925640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fec7c0c6ab0 0x7fec7c1529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec83925640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec7c14f110 0x7fec7c14f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec83925640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7c0a9990 con 0x7fec7c1561a0 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec83925640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7c155d10 con 0x7fec7c0c6ab0 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec83925640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7c1556c0 con 0x7fec7c14f110 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec816c1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fec7c0c6ab0 0x7fec7c1529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec80ec0640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fec7c1561a0 0x7fec7c0a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49240/0 (socket says 172.21.15.94:49240) 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec80ec0640 1 -- 172.21.15.94:0/3381567533 learned_addr learned my addr 172.21.15.94:0/3381567533 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec81ec2640 1 --2- 172.21.15.94:0/3381567533 >> v2:172.21.15.145:3300/0 conn(0x7fec7c14f110 0x7fec7c14f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec73fff640 1 -- 172.21.15.94:0/3381567533 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501283842 0 0) 0x7fec7c0a9990 con 0x7fec7c1561a0 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec73fff640 1 -- 172.21.15.94:0/3381567533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec68003520 con 0x7fec7c1561a0 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec816c1640 1 -- 172.21.15.94:0/3381567533 >> v2:172.21.15.145:3300/0 conn(0x7fec7c14f110 msgr2=0x7fec7c14f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec816c1640 1 --2- 172.21.15.94:0/3381567533 >> v2:172.21.15.145:3300/0 conn(0x7fec7c14f110 0x7fec7c14f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec816c1640 1 -- 172.21.15.94:0/3381567533 >> v1:172.21.15.94:6789/0 conn(0x7fec7c1561a0 legacy=0x7fec7c0a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.619+0000 7fec816c1640 1 -- 172.21.15.94:0/3381567533 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c1559a0 con 0x7fec7c0c6ab0 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.620+0000 7fec816c1640 1 --2- 172.21.15.94:0/3381567533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c0c6ab0 0x7fec7c1529e0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fec7c155c80 tx=0x7fec7802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70043ef28d77aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.620+0000 7fec73fff640 1 -- 172.21.15.94:0/3381567533 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec780315a0 con 0x7fec7c0c6ab0 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.620+0000 7fec73fff640 1 -- 172.21.15.94:0/3381567533 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec78037070 con 0x7fec7c0c6ab0 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.620+0000 7fec73fff640 1 -- 172.21.15.94:0/3381567533 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78031960 con 0x7fec7c0c6ab0 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.620+0000 7fec83925640 1 -- 172.21.15.94:0/3381567533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c0c6ab0 msgr2=0x7fec7c1529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.620+0000 7fec83925640 1 --2- 172.21.15.94:0/3381567533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c0c6ab0 0x7fec7c1529e0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fec7c155c80 tx=0x7fec7802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.621+0000 7fec83925640 1 -- 172.21.15.94:0/3381567533 shutdown_connections 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.621+0000 7fec83925640 1 --2- 172.21.15.94:0/3381567533 >> v2:172.21.15.145:3300/0 conn(0x7fec7c14f110 0x7fec7c14f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.621+0000 7fec83925640 1 --2- 172.21.15.94:0/3381567533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c0c6ab0 0x7fec7c1529e0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.621+0000 7fec83925640 1 -- 172.21.15.94:0/3381567533 >> 172.21.15.94:0/3381567533 conn(0x7fec7c06b1a0 msgr2=0x7fec7c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.621+0000 7fec83925640 1 -- 172.21.15.94:0/3381567533 shutdown_connections 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.621+0000 7fec83925640 1 -- 172.21.15.94:0/3381567533 wait complete. 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 Processor -- start 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 -- start start 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7c0c6ab0 0x7fec7c0c0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c14f110 0x7fec7c151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c1561a0 0x7fec7c1e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0aa600 con 0x7fec7c1561a0 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0c2a90 con 0x7fec7c14f110 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec83925640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1e7df0 con 0x7fec7c0c6ab0 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec81ec2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c1561a0 0x7fec7c1e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec81ec2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c1561a0 0x7fec7c1e8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47424/0 (socket says 172.21.15.94:47424) 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec81ec2640 1 -- 172.21.15.94:0/2115421956 learned_addr learned my addr 172.21.15.94:0/2115421956 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.622+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c14f110 0x7fec7c151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec816c1640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7c0c6ab0 0x7fec7c0c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec81ec2640 1 -- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7c0c6ab0 msgr2=0x7fec7c0c0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec81ec2640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7c0c6ab0 0x7fec7c0c0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec81ec2640 1 -- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c14f110 msgr2=0x7fec7c151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec81ec2640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c14f110 0x7fec7c151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec81ec2640 1 -- 172.21.15.94:0/2115421956 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c153f10 con 0x7fec7c1561a0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec81ec2640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c1561a0 0x7fec7c1e8280 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fec7400efc0 tx=0x7fec7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec71ffb640 1 -- 172.21.15.94:0/2115421956 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74018020 con 0x7fec7c1561a0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec71ffb640 1 -- 172.21.15.94:0/2115421956 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74014480 con 0x7fec7c1561a0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7c1541f0 con 0x7fec7c1561a0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec71ffb640 1 -- 172.21.15.94:0/2115421956 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74014870 con 0x7fec7c1561a0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.623+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7c0c4260 con 0x7fec7c1561a0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.625+0000 7fec71ffb640 1 -- 172.21.15.94:0/2115421956 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec74005d10 con 0x7fec7c1561a0 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.625+0000 7fec71ffb640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec6006b4d0 0x7fec6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.625+0000 7fec71ffb640 1 -- 172.21.15.94:0/2115421956 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec7408b030 con 0x7fec7c1561a0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.625+0000 7fec816c1640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec6006b4d0 0x7fec6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.626+0000 7fec816c1640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec6006b4d0 0x7fec6006d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fec78036000 tx=0x7fec7803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.626+0000 7fec865ec740 1 Processor -- start 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.626+0000 7fec865ec740 1 -- start start 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415f81010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec865ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415f81550 0x55a415fcabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fbb420 0x55a415f7b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec865ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a415ee56b0 con 0x55a415f81550 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a415e8ea60 con 0x55a415fca800 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a415ee1750 con 0x55a415fbb420 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415f81550 0x55a415fcabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415f81550 0x55a415fcabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47426/0 (socket says 172.21.15.94:47426) 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 -- 172.21.15.94:0/2320414968 learned_addr learned my addr 172.21.15.94:0/2320414968 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec816c1640 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415f81010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec81ec2640 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fbb420 0x55a415f7b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 -- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fbb420 msgr2=0x55a415f7b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fbb420 0x55a415f7b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 -- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 msgr2=0x55a415f81010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415f81010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 -- 172.21.15.94:0/2320414968 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a415eacf30 con 0x55a415f81550 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415f81550 0x55a415fcabe0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fec6c000f60 tx=0x7fec6c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf88f8a394d6ef21 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec66ffd640 1 -- 172.21.15.94:0/2320414968 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00fc10 con 0x55a415f81550 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec66ffd640 1 -- 172.21.15.94:0/2320414968 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c016020 con 0x55a415f81550 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.627+0000 7fec66ffd640 1 -- 172.21.15.94:0/2320414968 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0058b0 con 0x55a415f81550 2024-03-06T07:22:59.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 -- 172.21.15.94:0/2320414968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415f81550 msgr2=0x55a415fcabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415f81550 0x55a415fcabe0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fec6c000f60 tx=0x7fec6c000bc0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 -- 172.21.15.94:0/2320414968 shutdown_connections 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fbb420 0x55a415f7b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415f81550 0x55a415fcabe0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 --2- 172.21.15.94:0/2320414968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415f81010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 -- 172.21.15.94:0/2320414968 >> 172.21.15.94:0/2320414968 conn(0x55a415f7d7d0 msgr2=0x55a415fa23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 -- 172.21.15.94:0/2320414968 shutdown_connections 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 -- 172.21.15.94:0/2320414968 wait complete. 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.628+0000 7fec865ec740 1 -- start start 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 1 Processor -- start 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fa13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fc8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a415fbf590 con 0x55a415fbb420 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a415ea1ea0 con 0x55a415fca800 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a415fa2db0 con 0x55a415fc8c50 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec865ec740 10 client.0 _ll_register_callbacks cb 0x55a415ea1870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.629+0000 7fec816c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fa13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec80ec0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fc8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec81ec2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fa13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47436/0 (socket says 172.21.15.94:47436) 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 -- 172.21.15.94:0/2695095740 learned_addr learned my addr 172.21.15.94:0/2695095740 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fc8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:38090/0 (socket says 172.21.15.94:38090) 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 -- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 msgr2=0x55a415f9eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 -- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 msgr2=0x55a415fc8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fc8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 -- 172.21.15.94:0/2695095740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a415ea2270 con 0x55a415fbb420 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec81ec2640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fc8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec816c1640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fa13d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fec78009860 tx=0x7fec7802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ee5bcd31e43b94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec5bfff640 1 -- 172.21.15.94:0/2695095740 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78055060 con 0x55a415fbb420 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec5bfff640 1 -- 172.21.15.94:0/2695095740 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec78045630 con 0x55a415fbb420 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec6c016020 con 0x55a415fbb420 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec5bfff640 1 -- 172.21.15.94:0/2695095740 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec78032e40 con 0x55a415fbb420 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec865ec740 10 client.8070 fetch_fsmap learned FSMap version 105 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) v3 -- 0x55a415bfa2d0 con 0x55a415fbb420 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec5bfff640 1 -- 172.21.15.94:0/2695095740 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c016020 con 0x55a415fbb420 2024-03-06T07:22:59.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.630+0000 7fec5bfff640 1 -- 172.21.15.94:0/2695095740 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 105) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fec78031070 con 0x55a415fbb420 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec865ec740 10 client.8070 fetch_fsmap finished waiting for FSMap version 105 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec865ec740 10 client.8070 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec865ec740 20 client.8070 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec865ec740 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fd5bd0 0x55a415fe0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec865ec740 4 client.8070 mds_command: new command op to 7842 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 --> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a415ef2bf0 con 0x55a415fd5bd0 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fd5bd0 0x55a415fe0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec5bfff640 10 client.8070 ms_handle_connect on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.631+0000 7fec80ec0640 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fd5bd0 0x55a415fe0790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7625d42b028316dd server_cookie=b5f2308406d859fa in_seq=0 out_seq=0 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec5bfff640 1 -- 172.21.15.94:0/2695095740 <== mds.0 v2:172.21.15.145:6834/198029515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a415ef2bf0 con 0x55a415fd5bd0 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec5bfff640 10 client.8070 handle_command_reply: tid=0 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 client.8070 shutdown 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 msgr2=0x55a415fa13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fa13d0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fec78009860 tx=0x7fec7802e000 comp rx=0 tx=0).stop 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 shutdown_connections 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fd5bd0 0x55a415fe0790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec5bfff640 0 client.8070 ms_handle_reset on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fc8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 --2- 172.21.15.94:0/2695095740 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fa13d0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 >> 172.21.15.94:0/2695095740 conn(0x55a415f81550 msgr2=0x55a415f7ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 shutdown_connections 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 1 -- 172.21.15.94:0/2695095740 wait complete. 2024-03-06T07:22:59.642 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.632+0000 7fec865ec740 20 client.8070 trim_cache size 0 max 16384 2024-03-06T07:22:59.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 Processor -- start 2024-03-06T07:22:59.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 -- start start 2024-03-06T07:22:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fbb420 0x55a415fc7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fca800 0x55a415fa0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c06d500 con 0x55a415fca800 2024-03-06T07:22:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c14e860 con 0x55a415fbb420 2024-03-06T07:22:59.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0bf770 con 0x55a415fc8c50 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.649+0000 7fec81ec2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fca800 0x55a415fa0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec816c1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fbb420 0x55a415fc7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fca800 0x55a415fa0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47440/0 (socket says 172.21.15.94:47440) 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 -- 172.21.15.94:0/1460126207 learned_addr learned my addr 172.21.15.94:0/1460126207 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec80ec0640 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 -- 172.21.15.94:0/1460126207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 msgr2=0x55a415f9fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 -- 172.21.15.94:0/1460126207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fbb420 msgr2=0x55a415fc7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fbb420 0x55a415fc7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 -- 172.21.15.94:0/1460126207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a415cc46b0 con 0x55a415fca800 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec81ec2640 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fca800 0x55a415fa0060 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fec740596f0 tx=0x7fec7405c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee8ee648154b4af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec667fc640 1 -- 172.21.15.94:0/1460126207 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74057a60 con 0x55a415fca800 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec667fc640 1 -- 172.21.15.94:0/1460126207 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74071150 con 0x55a415fca800 2024-03-06T07:22:59.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.650+0000 7fec667fc640 1 -- 172.21.15.94:0/1460126207 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74058df0 con 0x55a415fca800 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 -- 172.21.15.94:0/1460126207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fca800 msgr2=0x55a415fa0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fca800 0x55a415fa0060 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fec740596f0 tx=0x7fec7405c040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 -- 172.21.15.94:0/1460126207 shutdown_connections 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fca800 0x55a415fa0060 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415f9fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 --2- 172.21.15.94:0/1460126207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fbb420 0x55a415fc7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 -- 172.21.15.94:0/1460126207 >> 172.21.15.94:0/1460126207 conn(0x55a415f81550 msgr2=0x55a415fa11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 -- 172.21.15.94:0/1460126207 shutdown_connections 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 -- 172.21.15.94:0/1460126207 wait complete. 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 -- start start 2024-03-06T07:22:59.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.651+0000 7fec865ec740 1 Processor -- start 2024-03-06T07:22:59.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec865ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fbe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415fa0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec865ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fb63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec865ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a415fbc5d0 con 0x55a415fbb420 2024-03-06T07:22:59.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a415fbd610 con 0x55a415fca800 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec865ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a415fbab00 con 0x55a415fc8c50 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec865ec740 10 client.0 _ll_register_callbacks cb 0x7fec78031bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fbe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fbe440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47450/0 (socket says 172.21.15.94:47450) 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 -- 172.21.15.94:0/3809610738 learned_addr learned my addr 172.21.15.94:0/3809610738 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec80ec0640 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415fa0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec81ec2640 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fb63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 -- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 msgr2=0x55a415fa0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415fa0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 -- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 msgr2=0x55a415fb63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fb63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.652+0000 7fec816c1640 1 -- 172.21.15.94:0/3809610738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a415fc0740 con 0x55a415fbb420 2024-03-06T07:22:59.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec816c1640 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fbe440 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fec7802be50 tx=0x7fec78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b258c8aafc62f17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec5bfff640 1 -- 172.21.15.94:0/3809610738 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78063020 con 0x55a415fbb420 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec5bfff640 1 -- 172.21.15.94:0/3809610738 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec78031070 con 0x55a415fbb420 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec74071150 con 0x55a415fbb420 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec5bfff640 1 -- 172.21.15.94:0/3809610738 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec7806a050 con 0x55a415fbb420 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 10 client.8076 fetch_fsmap learned FSMap version 105 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) v3 -- 0x55a415fb6e30 con 0x55a415fbb420 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec5bfff640 1 -- 172.21.15.94:0/3809610738 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74071150 con 0x55a415fbb420 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec5bfff640 1 -- 172.21.15.94:0/3809610738 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 105) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fec7806d020 con 0x55a415fbb420 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 10 client.8076 fetch_fsmap finished waiting for FSMap version 105 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 10 client.8076 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 20 client.8076 populate_metadata read hostname 'smithi094' 2024-03-06T07:22:59.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fb98d0 0x55a415fb9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 4 client.8076 mds_command: new command op to 7842 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 --> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a415eecf40 con 0x55a415fb98d0 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.653+0000 7fec80ec0640 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fb98d0 0x55a415fb9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec80ec0640 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fb98d0 0x55a415fb9cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb295591c0bf3f60 server_cookie=6d993d9f18d5e5a0 in_seq=0 out_seq=0 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec5bfff640 10 client.8076 ms_handle_connect on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec5bfff640 1 -- 172.21.15.94:0/3809610738 <== mds.0 v2:172.21.15.145:6834/198029515 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55a415eecf40 con 0x55a415fb98d0 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec5bfff640 10 client.8076 handle_command_reply: tid=0 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 client.8076 shutdown 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 msgr2=0x55a415fbe440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fbe440 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fec7802be50 tx=0x7fec78004730 comp rx=0 tx=0).stop 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 shutdown_connections 2024-03-06T07:22:59.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:6834/198029515,v1:172.21.15.145:6836/198029515] conn(0x55a415fb98d0 0x55a415fb9cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a415fca800 0x55a415fb63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a415fc8c50 0x55a415fa0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec5bfff640 0 client.8076 ms_handle_reset on v2:172.21.15.145:6834/198029515 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 --2- 172.21.15.94:0/3809610738 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55a415fbb420 0x55a415fbe440 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 >> 172.21.15.94:0/3809610738 conn(0x55a415f81550 msgr2=0x55a415f9e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 shutdown_connections 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 1 -- 172.21.15.94:0/3809610738 wait complete. 2024-03-06T07:22:59.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.654+0000 7fec865ec740 20 client.8076 trim_cache size 0 max 16384 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7842, 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 16.369391127, 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 105, 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 121, 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 121, 2024-03-06T07:22:59.664 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 21.461600670999999 2024-03-06T07:22:59.665 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:22:59.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec6006b4d0 msgr2=0x7fec6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec6006b4d0 0x7fec6006d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fec78036000 tx=0x7fec7803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c1561a0 msgr2=0x7fec7c1e8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c1561a0 0x7fec7c1e8280 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fec7400efc0 tx=0x7fec7400c490 comp rx=0 tx=0).stop 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 shutdown_connections 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec6006b4d0 0x7fec6006d990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec7c1561a0 0x7fec7c1e8280 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c14f110 0x7fec7c151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 --2- 172.21.15.94:0/2115421956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7c0c6ab0 0x7fec7c0c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 >> 172.21.15.94:0/2115421956 conn(0x7fec7c06b1a0 msgr2=0x7fec7c0a9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 shutdown_connections 2024-03-06T07:22:59.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.655+0000 7fec83925640 1 -- 172.21.15.94:0/2115421956 wait complete. 2024-03-06T07:22:59.667 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 105, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:42.624925+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7842": { "addr": "172.21.15.145:6836/198029515", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 198029515, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 198029515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7842, "incarnation": 102, "join_fscid": 9, "name": "b", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:22:55.313430+0000", "qdb_cluster": [ 7842 ], "qdb_leader": 7842, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7842 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/1356561672", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 1356561672, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 1356561672, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/1458830160", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 1458830160, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 1458830160, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7824, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/282207355", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 282207355, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 282207355, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7827, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:22:59.668 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 9, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:22:59.668 INFO:tasks.cephfs.test_failover:current = [{'gid': 7824, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/1458830160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 1458830160}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 1458830160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7842, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.145:6836/198029515', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 198029515}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 198029515}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7827, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/282207355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 282207355}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 282207355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/1356561672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1356561672}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1356561672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:22:59.669 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:22:59.669 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 9:0 2024-03-06T07:22:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.768+0000 7f5389302640 1 Processor -- start 2024-03-06T07:22:59.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.768+0000 7f5389302640 1 -- start start 2024-03-06T07:22:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f5389302640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5384152b40 0x7f53840c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f5389302640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53840724d0 0x7f53841551f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f5389302640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53840733f0 con 0x7f53840a7ff0 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f5389302640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5384072d70 con 0x7f5384152b40 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f5389302640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53840c82c0 con 0x7f53840724d0 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f5382ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5384152b40 0x7f53840c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f53837fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f53840a7ff0 0x7f5384155730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49248/0 (socket says 172.21.15.94:49248) 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f53837fe640 1 -- 172.21.15.94:0/2851100954 learned_addr learned my addr 172.21.15.94:0/2851100954 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.769+0000 7f53827fc640 1 --2- 172.21.15.94:0/2851100954 >> v2:172.21.15.145:3300/0 conn(0x7f53840724d0 0x7f53841551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5382ffd640 1 -- 172.21.15.94:0/2851100954 >> v2:172.21.15.145:3300/0 conn(0x7f53840724d0 msgr2=0x7f53841551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5382ffd640 1 --2- 172.21.15.94:0/2851100954 >> v2:172.21.15.145:3300/0 conn(0x7f53840724d0 0x7f53841551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5382ffd640 1 -- 172.21.15.94:0/2851100954 >> v1:172.21.15.94:6789/0 conn(0x7f53840a7ff0 legacy=0x7f5384155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5382ffd640 1 -- 172.21.15.94:0/2851100954 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538406fd30 con 0x7f5384152b40 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5381ffb640 1 -- 172.21.15.94:0/2851100954 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563572333 0 0) 0x7f53840733f0 con 0x7f53840a7ff0 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5382ffd640 1 --2- 172.21.15.94:0/2851100954 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5384152b40 0x7f53840c8810 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f53840c5330 tx=0x7f537402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6621b938c86d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5381ffb640 1 -- 172.21.15.94:0/2851100954 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374039070 con 0x7f5384152b40 2024-03-06T07:22:59.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.770+0000 7f5381ffb640 1 -- 172.21.15.94:0/2851100954 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537402cbf0 con 0x7f5384152b40 2024-03-06T07:22:59.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5381ffb640 1 -- 172.21.15.94:0/2851100954 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374034440 con 0x7f5384152b40 2024-03-06T07:22:59.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 -- 172.21.15.94:0/2851100954 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5384152b40 msgr2=0x7f53840c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 --2- 172.21.15.94:0/2851100954 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5384152b40 0x7f53840c8810 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f53840c5330 tx=0x7f537402be40 comp rx=0 tx=0).stop 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 -- 172.21.15.94:0/2851100954 shutdown_connections 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 --2- 172.21.15.94:0/2851100954 >> v2:172.21.15.145:3300/0 conn(0x7f53840724d0 0x7f53841551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 --2- 172.21.15.94:0/2851100954 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5384152b40 0x7f53840c8810 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 -- 172.21.15.94:0/2851100954 >> 172.21.15.94:0/2851100954 conn(0x7f538406b970 msgr2=0x7f53840aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 -- 172.21.15.94:0/2851100954 shutdown_connections 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.771+0000 7f5389302640 1 -- 172.21.15.94:0/2851100954 wait complete. 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.772+0000 7f5389302640 1 Processor -- start 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.772+0000 7f5389302640 1 -- start start 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f5389302640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840724d0 0x7f53840c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f5389302640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840a7ff0 0x7f53840a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f5389302640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5384152b40 0x7f53841e78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f5389302640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53840c8d50 con 0x7f53840a7ff0 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f5389302640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5384070490 con 0x7f5384152b40 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f5389302640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53840c8ed0 con 0x7f53840724d0 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840a7ff0 0x7f53840a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840a7ff0 0x7f53840a96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47466/0 (socket says 172.21.15.94:47466) 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 -- 172.21.15.94:0/1339292735 learned_addr learned my addr 172.21.15.94:0/1339292735 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f5382ffd640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840724d0 0x7f53840c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53837fe640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5384152b40 0x7f53841e78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 -- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840724d0 msgr2=0x7f53840c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840724d0 0x7f53840c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 -- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5384152b40 msgr2=0x7f53841e78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5384152b40 0x7f53841e78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.773+0000 7f53827fc640 1 -- 172.21.15.94:0/1339292735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840c4010 con 0x7f53840a7ff0 2024-03-06T07:22:59.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.774+0000 7f53827fc640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840a7ff0 0x7f53840a96b0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f537800e870 tx=0x7f537800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.774+0000 7f536b7fe640 1 -- 172.21.15.94:0/1339292735 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378017070 con 0x7f53840a7ff0 2024-03-06T07:22:59.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.774+0000 7f536b7fe640 1 -- 172.21.15.94:0/1339292735 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537800c940 con 0x7f53840a7ff0 2024-03-06T07:22:59.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.774+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53841490a0 con 0x7f53840a7ff0 2024-03-06T07:22:59.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.774+0000 7f536b7fe640 1 -- 172.21.15.94:0/1339292735 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537800cd30 con 0x7f53840a7ff0 2024-03-06T07:22:59.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.774+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5384149660 con 0x7f53840a7ff0 2024-03-06T07:22:59.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.775+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538404f330 con 0x7f53840a7ff0 2024-03-06T07:22:59.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.775+0000 7f536b7fe640 1 -- 172.21.15.94:0/1339292735 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f537801a070 con 0x7f53840a7ff0 2024-03-06T07:22:59.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.776+0000 7f536b7fe640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f537006b4d0 0x7f537006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.776+0000 7f536b7fe640 1 -- 172.21.15.94:0/1339292735 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f537808a0c0 con 0x7f53840a7ff0 2024-03-06T07:22:59.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.776+0000 7f5382ffd640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f537006b4d0 0x7f537006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.776+0000 7f5382ffd640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f537006b4d0 0x7f537006d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5374036000 tx=0x7f5374030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:22:59.780+0000 7f536b7fe640 1 -- 172.21.15.94:0/1339292735 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53780587f0 con 0x7f53840a7ff0 2024-03-06T07:23:00.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.019+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "9:0"} v 0) v1 -- 0x7f538406fb10 con 0x7f53840a7ff0 2024-03-06T07:23:00.022 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:23:00.020+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:00.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.021+0000 7f536b7fe640 1 -- 172.21.15.94:0/1339292735 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "9:0"}]=0 failed mds gid 7842 v106) v1 ==== 97+0+0 (secure 0 0 0) 0x7f537805c890 con 0x7f53840a7ff0 2024-03-06T07:23:00.023 INFO:teuthology.orchestra.run.smithi094.stderr:failed mds gid 7842 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f537006b4d0 msgr2=0x7f537006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f537006b4d0 0x7f537006d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5374036000 tx=0x7f5374030000 comp rx=0 tx=0).stop 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840a7ff0 msgr2=0x7f53840a96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840a7ff0 0x7f53840a96b0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f537800e870 tx=0x7f537800ed30 comp rx=0 tx=0).stop 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 shutdown_connections 2024-03-06T07:23:00.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f537006b4d0 0x7f537006d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5384152b40 0x7f53841e78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840a7ff0 0x7f53840a96b0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 --2- 172.21.15.94:0/1339292735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840724d0 0x7f53840c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 >> 172.21.15.94:0/1339292735 conn(0x7f538406b970 msgr2=0x7f538414d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 shutdown_connections 2024-03-06T07:23:00.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.024+0000 7f5389302640 1 -- 172.21.15.94:0/1339292735 wait complete. 2024-03-06T07:23:00.027 INFO:tasks.ceph.mds.b.smithi145.stderr: -476> 2024-03-06T07:22:48.557+0000 7f3ee55a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:00.039 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:00.052 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.137+0000 7f598a82a640 1 Processor -- start 2024-03-06T07:23:00.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.137+0000 7f598a82a640 1 -- start start 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f598a82a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5984152960 0x7f598406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f598a82a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f59840ac260 0x7f59841559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f598a82a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5984152d40 con 0x7f59841592d0 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f598a82a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5984066e30 con 0x7f5984152960 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f598a82a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59840c1dc0 con 0x7f59840ac260 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5983fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5984152960 0x7f598406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f59837fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f59841592d0 0x7f59841552d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38746/0 (socket says 172.21.15.94:38746) 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5983fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5984152960 0x7f598406dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47476/0 (socket says 172.21.15.94:47476) 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f59837fe640 1 -- 172.21.15.94:0/2771186858 learned_addr learned my addr 172.21.15.94:0/2771186858 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:00.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5988dc7640 1 --2- 172.21.15.94:0/2771186858 >> v2:172.21.15.145:3300/0 conn(0x7f59840ac260 0x7f59841559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5983fff640 1 -- 172.21.15.94:0/2771186858 >> v2:172.21.15.145:3300/0 conn(0x7f59840ac260 msgr2=0x7f59841559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5983fff640 1 --2- 172.21.15.94:0/2771186858 >> v2:172.21.15.145:3300/0 conn(0x7f59840ac260 0x7f59841559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5982ffd640 1 -- 172.21.15.94:0/2771186858 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37951233 0 0) 0x7f5984152d40 con 0x7f59841592d0 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5983fff640 1 -- 172.21.15.94:0/2771186858 >> v1:172.21.15.145:6789/0 conn(0x7f59841592d0 legacy=0x7f59841552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.138+0000 7f5983fff640 1 -- 172.21.15.94:0/2771186858 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840c1b50 con 0x7f5984152960 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.139+0000 7f5983fff640 1 --2- 172.21.15.94:0/2771186858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f598406dc90 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f598406b380 tx=0x7f597402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38a0a6ca93fe1ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.139+0000 7f5982ffd640 1 -- 172.21.15.94:0/2771186858 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974035020 con 0x7f5984152960 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.139+0000 7f5982ffd640 1 -- 172.21.15.94:0/2771186858 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974031830 con 0x7f5984152960 2024-03-06T07:23:00.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.139+0000 7f5982ffd640 1 -- 172.21.15.94:0/2771186858 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974031c00 con 0x7f5984152960 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 -- 172.21.15.94:0/2771186858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 msgr2=0x7f598406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 --2- 172.21.15.94:0/2771186858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f598406dc90 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f598406b380 tx=0x7f597402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 -- 172.21.15.94:0/2771186858 shutdown_connections 2024-03-06T07:23:00.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 --2- 172.21.15.94:0/2771186858 >> v2:172.21.15.145:3300/0 conn(0x7f59840ac260 0x7f59841559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 --2- 172.21.15.94:0/2771186858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f598406dc90 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 -- 172.21.15.94:0/2771186858 >> 172.21.15.94:0/2771186858 conn(0x7f59840c70b0 msgr2=0x7f59840c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 -- 172.21.15.94:0/2771186858 shutdown_connections 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.140+0000 7f598a82a640 1 -- 172.21.15.94:0/2771186858 wait complete. 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.141+0000 7f598a82a640 1 Processor -- start 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.141+0000 7f598a82a640 1 -- start start 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ac260 0x7f59841eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f59841204a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59841219a0 0x7f59841446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840c20a0 con 0x7f5984152960 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984159760 con 0x7f59840ac260 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840c2220 con 0x7f59841219a0 2024-03-06T07:23:00.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f59841204a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f59841204a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47488/0 (socket says 172.21.15.94:47488) 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 -- 172.21.15.94:0/3033308931 learned_addr learned my addr 172.21.15.94:0/3033308931 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f5988dc7640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59841219a0 0x7f59841446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 -- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59841219a0 msgr2=0x7f59841446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f5983fff640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ac260 0x7f59841eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59841219a0 0x7f59841446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 -- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ac260 msgr2=0x7f59841eb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ac260 0x7f59841eb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 -- 172.21.15.94:0/3033308931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5984156670 con 0x7f5984152960 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f5988dc7640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59841219a0 0x7f59841446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f59837fe640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f59841204a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f597800c4d0 tx=0x7f597800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f5980ff9640 1 -- 172.21.15.94:0/3033308931 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978018070 con 0x7f5984152960 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f5980ff9640 1 -- 172.21.15.94:0/3033308931 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5978007500 con 0x7f5984152960 2024-03-06T07:23:00.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f5980ff9640 1 -- 172.21.15.94:0/3033308931 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978004a20 con 0x7f5984152960 2024-03-06T07:23:00.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5984070c50 con 0x7f5984152960 2024-03-06T07:23:00.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.142+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5984071210 con 0x7f5984152960 2024-03-06T07:23:00.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.143+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598404f330 con 0x7f5984152960 2024-03-06T07:23:00.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.144+0000 7f5980ff9640 1 -- 172.21.15.94:0/3033308931 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f597801b070 con 0x7f5984152960 2024-03-06T07:23:00.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.144+0000 7f5980ff9640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f597006b4d0 0x7f597006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.144+0000 7f5980ff9640 1 -- 172.21.15.94:0/3033308931 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5978089660 con 0x7f5984152960 2024-03-06T07:23:00.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.144+0000 7f5983fff640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f597006b4d0 0x7f597006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.145+0000 7f5983fff640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f597006b4d0 0x7f597006d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5974036000 tx=0x7f5974008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.148+0000 7f5980ff9640 1 -- 172.21.15.94:0/3033308931 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5978058390 con 0x7f5984152960 2024-03-06T07:23:00.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.356+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59841523a0 con 0x7f5984152960 2024-03-06T07:23:00.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.356+0000 7f5980ff9640 1 -- 172.21.15.94:0/3033308931 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4188 (secure 0 0 0) 0x7f5978057c90 con 0x7f5984152960 2024-03-06T07:23:00.359 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:00.359 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":107,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:00.025312+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7827":{"gid":7827,"name":"c","rank":0,"incarnation":107,"state":"up:replay","state_seq":1,"addr":"172.21.15.94:6834/282207355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":282207355},{"type":"v1","addr":"172.21.15.94:6834","nonce":282207355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-06T07:23:00.359 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 107 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.358+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f597006b4d0 msgr2=0x7f597006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.358+0000 7f598a82a640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f597006b4d0 0x7f597006d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5974036000 tx=0x7f5974008030 comp rx=0 tx=0).stop 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 msgr2=0x7f59841204a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f59841204a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f597800c4d0 tx=0x7f597800c990 comp rx=0 tx=0).stop 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 shutdown_connections 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f597006b4d0 0x7f597006d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59841219a0 0x7f59841446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5984152960 0x7f59841204a0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 --2- 172.21.15.94:0/3033308931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ac260 0x7f59841eb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 >> 172.21.15.94:0/3033308931 conn(0x7f59840c70b0 msgr2=0x7f59840c36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 shutdown_connections 2024-03-06T07:23:00.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:00.359+0000 7f598a82a640 1 -- 172.21.15.94:0/3033308931 wait complete. 2024-03-06T07:23:00.774 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:42.624925+0000', 'modified': '2024-03-06T07:23:00.025312+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7827}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7827': {'gid': 7827, 'name': 'c', 'rank': 0, 'incarnation': 107, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.94:6834/282207355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 282207355}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 282207355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:23:00.775 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7827:up:replay 2024-03-06T07:23:01.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:01.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.885+0000 7f11b0dbc640 1 Processor -- start 2024-03-06T07:23:01.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.885+0000 7f11b0dbc640 1 -- start start 2024-03-06T07:23:01.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11b0dbc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11ac150580 0x7f11ac06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11b0dbc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f11ac0c6000 0x7f11ac06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11b0dbc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ac153a60 con 0x7f11ac0c1320 2024-03-06T07:23:01.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11b0dbc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac153d40 con 0x7f11ac150580 2024-03-06T07:23:01.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11b0dbc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac06c2c0 con 0x7f11ac0c6000 2024-03-06T07:23:01.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11aa59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11ac150580 0x7f11ac06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11aa59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f11ac150580 0x7f11ac06dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47502/0 (socket says 172.21.15.94:47502) 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.886+0000 7f11aa59c640 1 -- 172.21.15.94:0/3413050338 learned_addr learned my addr 172.21.15.94:0/3413050338 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.887+0000 7f11aad9d640 1 --2- 172.21.15.94:0/3413050338 >> v2:172.21.15.145:3300/0 conn(0x7f11ac0c6000 0x7f11ac06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.887+0000 7f11aa59c640 1 -- 172.21.15.94:0/3413050338 >> v2:172.21.15.145:3300/0 conn(0x7f11ac0c6000 msgr2=0x7f11ac06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.887+0000 7f11aa59c640 1 --2- 172.21.15.94:0/3413050338 >> v2:172.21.15.145:3300/0 conn(0x7f11ac0c6000 0x7f11ac06e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.887+0000 7f11aa59c640 1 -- 172.21.15.94:0/3413050338 >> v1:172.21.15.145:6789/0 conn(0x7f11ac0c1320 legacy=0x7f11ac1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.887+0000 7f11aa59c640 1 -- 172.21.15.94:0/3413050338 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac06c5a0 con 0x7f11ac150580 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.887+0000 7f11aa59c640 1 --2- 172.21.15.94:0/3413050338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac150580 0x7f11ac06dd10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0c5960 tx=0x7f119c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88cc2c9285d631b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.887+0000 7f11a959a640 1 -- 172.21.15.94:0/3413050338 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c02c9f0 con 0x7f11ac150580 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11a959a640 1 -- 172.21.15.94:0/3413050338 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119c02cb90 con 0x7f11ac150580 2024-03-06T07:23:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11a959a640 1 -- 172.21.15.94:0/3413050338 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c0343f0 con 0x7f11ac150580 2024-03-06T07:23:01.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 -- 172.21.15.94:0/3413050338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac150580 msgr2=0x7f11ac06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/3413050338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac150580 0x7f11ac06dd10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0c5960 tx=0x7f119c02be50 comp rx=0 tx=0).stop 2024-03-06T07:23:01.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 -- 172.21.15.94:0/3413050338 shutdown_connections 2024-03-06T07:23:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/3413050338 >> v2:172.21.15.145:3300/0 conn(0x7f11ac0c6000 0x7f11ac06e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/3413050338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac150580 0x7f11ac06dd10 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 -- 172.21.15.94:0/3413050338 >> 172.21.15.94:0/3413050338 conn(0x7f11ac0c6710 msgr2=0x7f11ac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 -- 172.21.15.94:0/3413050338 shutdown_connections 2024-03-06T07:23:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.888+0000 7f11b0dbc640 1 -- 172.21.15.94:0/3413050338 wait complete. 2024-03-06T07:23:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.889+0000 7f11b0dbc640 1 Processor -- start 2024-03-06T07:23:01.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11b0dbc640 1 -- start start 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11b0dbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11ac0c1320 0x7f11ac1ec900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11b0dbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac0c6000 0x7f11ac0c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11b0dbc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11ac150580 0x7f11ac1e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11b0dbc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac0c0dc0 con 0x7f11ac0c6000 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11b0dbc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac0c0c40 con 0x7f11ac150580 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11b0dbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac156dc0 con 0x7f11ac0c1320 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11a9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac0c6000 0x7f11ac0c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11a9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac0c6000 0x7f11ac0c2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47506/0 (socket says 172.21.15.94:47506) 2024-03-06T07:23:01.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11a9d9b640 1 -- 172.21.15.94:0/2100834350 learned_addr learned my addr 172.21.15.94:0/2100834350 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11aa59c640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11ac0c1320 0x7f11ac1ec900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.890+0000 7f11aad9d640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11ac150580 0x7f11ac1e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11a9d9b640 1 -- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11ac0c1320 msgr2=0x7f11ac1ec900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11a9d9b640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11ac0c1320 0x7f11ac1ec900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11a9d9b640 1 -- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11ac150580 msgr2=0x7f11ac1e89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11a9d9b640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11ac150580 0x7f11ac1e89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11a9d9b640 1 -- 172.21.15.94:0/2100834350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac1ece40 con 0x7f11ac0c6000 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11aad9d640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11ac150580 0x7f11ac1e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11a9d9b640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac0c6000 0x7f11ac0c2e10 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f11a000c430 tx=0x7f11a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f119affd640 1 -- 172.21.15.94:0/2100834350 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0011790 con 0x7f11ac0c6000 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f119affd640 1 -- 172.21.15.94:0/2100834350 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a0017070 con 0x7f11ac0c6000 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac0731c0 con 0x7f11ac0c6000 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f119affd640 1 -- 172.21.15.94:0/2100834350 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0011bb0 con 0x7f11ac0c6000 2024-03-06T07:23:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.891+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac070f40 con 0x7f11ac0c6000 2024-03-06T07:23:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.892+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac071180 con 0x7f11ac0c6000 2024-03-06T07:23:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.892+0000 7f119affd640 1 -- 172.21.15.94:0/2100834350 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11a001c460 con 0x7f11ac0c6000 2024-03-06T07:23:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.893+0000 7f119affd640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118c06b4d0 0x7f118c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.893+0000 7f119affd640 1 -- 172.21.15.94:0/2100834350 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f11a008a110 con 0x7f11ac0c6000 2024-03-06T07:23:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.893+0000 7f11aa59c640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118c06b4d0 0x7f118c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:01.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.894+0000 7f11aa59c640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118c06b4d0 0x7f118c06d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f119c035000 tx=0x7f119c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:01.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:01.897+0000 7f119affd640 1 -- 172.21.15.94:0/2100834350 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11ac071180 con 0x7f11ac0c6000 2024-03-06T07:23:02.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.105+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11ac04f330 con 0x7f11ac0c6000 2024-03-06T07:23:02.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.105+0000 7f119affd640 1 -- 172.21.15.94:0/2100834350 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4982 (secure 0 0 0) 0x7f11a0058e40 con 0x7f11ac0c6000 2024-03-06T07:23:02.107 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:02.107 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":109,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7540,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1862149336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1862149336},{"type":"v1","addr":"172.21.15.145:6836","nonce":1862149336}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:01.029633+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7827},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7827":{"gid":7827,"name":"c","rank":0,"incarnation":107,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.94:6834/282207355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":282207355},{"type":"v1","addr":"172.21.15.94:6834","nonce":282207355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-06T07:23:02.108 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 109 2024-03-06T07:23:02.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118c06b4d0 msgr2=0x7f118c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118c06b4d0 0x7f118c06d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f119c035000 tx=0x7f119c030000 comp rx=0 tx=0).stop 2024-03-06T07:23:02.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac0c6000 msgr2=0x7f11ac0c2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac0c6000 0x7f11ac0c2e10 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f11a000c430 tx=0x7f11a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 shutdown_connections 2024-03-06T07:23:02.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f118c06b4d0 0x7f118c06d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11ac150580 0x7f11ac1e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11ac0c6000 0x7f11ac0c2e10 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 --2- 172.21.15.94:0/2100834350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11ac0c1320 0x7f11ac1ec900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 >> 172.21.15.94:0/2100834350 conn(0x7f11ac0c6710 msgr2=0x7f11ac0c4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 shutdown_connections 2024-03-06T07:23:02.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:02.107+0000 7f11b0dbc640 1 -- 172.21.15.94:0/2100834350 wait complete. 2024-03-06T07:23:02.233 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:23:02.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:42.624925+0000', 'modified': '2024-03-06T07:23:01.029633+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 122, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7827}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7827': {'gid': 7827, 'name': 'c', 'rank': 0, 'incarnation': 107, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.94:6834/282207355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 282207355}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 282207355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:23:02.524 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7827:up:rejoin 2024-03-06T07:23:03.525 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.635+0000 7f675e888640 1 Processor -- start 2024-03-06T07:23:03.637 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.635+0000 7f675e888640 1 -- start start 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f675e888640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6758150580 0x7f675806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f675e888640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67580c1320 0x7f675806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f675e888640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6758153a60 con 0x7f67580c6000 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f675e888640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6758153d40 con 0x7f6758150580 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f675e888640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675806c2c0 con 0x7f67580c1320 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6757fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6758150580 0x7f675806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6757fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6758150580 0x7f675806dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47538/0 (socket says 172.21.15.94:47538) 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f675ce25640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f67580c6000 0x7f67581566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49274/0 (socket says 172.21.15.94:49274) 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6757fff640 1 -- 172.21.15.94:0/1648926480 learned_addr learned my addr 172.21.15.94:0/1648926480 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:03.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f67577fe640 1 --2- 172.21.15.94:0/1648926480 >> v2:172.21.15.145:3300/0 conn(0x7f67580c1320 0x7f675806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6757fff640 1 -- 172.21.15.94:0/1648926480 >> v2:172.21.15.145:3300/0 conn(0x7f67580c1320 msgr2=0x7f675806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6757fff640 1 --2- 172.21.15.94:0/1648926480 >> v2:172.21.15.145:3300/0 conn(0x7f67580c1320 0x7f675806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6757fff640 1 -- 172.21.15.94:0/1648926480 >> v1:172.21.15.94:6789/0 conn(0x7f67580c6000 legacy=0x7f67581566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6757fff640 1 -- 172.21.15.94:0/1648926480 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675806c5a0 con 0x7f6758150580 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f6756ffd640 1 -- 172.21.15.94:0/1648926480 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859445541 0 0) 0x7f6758153a60 con 0x7f67580c6000 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.636+0000 7f67577fe640 1 --2- 172.21.15.94:0/1648926480 >> v2:172.21.15.145:3300/0 conn(0x7f67580c1320 0x7f675806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.637+0000 7f6757fff640 1 --2- 172.21.15.94:0/1648926480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f675806dd10 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f67580c7730 tx=0x7f674802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352dc937cd348c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.637+0000 7f6756ffd640 1 -- 172.21.15.94:0/1648926480 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748035020 con 0x7f6758150580 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.637+0000 7f6756ffd640 1 -- 172.21.15.94:0/1648926480 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6748031870 con 0x7f6758150580 2024-03-06T07:23:03.640 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.637+0000 7f6756ffd640 1 -- 172.21.15.94:0/1648926480 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748031c40 con 0x7f6758150580 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 -- 172.21.15.94:0/1648926480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 msgr2=0x7f675806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 --2- 172.21.15.94:0/1648926480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f675806dd10 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f67580c7730 tx=0x7f674802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 -- 172.21.15.94:0/1648926480 shutdown_connections 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 --2- 172.21.15.94:0/1648926480 >> v2:172.21.15.145:3300/0 conn(0x7f67580c1320 0x7f675806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 --2- 172.21.15.94:0/1648926480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f675806dd10 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 -- 172.21.15.94:0/1648926480 >> 172.21.15.94:0/1648926480 conn(0x7f67580c6710 msgr2=0x7f67580c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 -- 172.21.15.94:0/1648926480 shutdown_connections 2024-03-06T07:23:03.643 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.638+0000 7f675e888640 1 -- 172.21.15.94:0/1648926480 wait complete. 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.639+0000 7f675e888640 1 Processor -- start 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.639+0000 7f675e888640 1 -- start start 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675e888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67580c1320 0x7f67580bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675e888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67580c6000 0x7f6758154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675e888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f6758154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675e888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580c0dc0 con 0x7f6758150580 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675e888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580c0c40 con 0x7f67580c6000 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675e888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6758156dc0 con 0x7f67580c1320 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f6758154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f6758154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47554/0 (socket says 172.21.15.94:47554) 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 -- 172.21.15.94:0/1256133822 learned_addr learned my addr 172.21.15.94:0/1256133822 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:03.644 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f6757fff640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67580c1320 0x7f67580bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 -- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67580c1320 msgr2=0x7f67580bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67580c1320 0x7f67580bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 -- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67580c6000 msgr2=0x7f6758154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67580c6000 0x7f6758154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 -- 172.21.15.94:0/1256133822 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67580a6fc0 con 0x7f6758150580 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675ce25640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f6758154d40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f675000c810 tx=0x7f675000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f6754ff9640 1 -- 172.21.15.94:0/1256133822 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6750018070 con 0x7f6758150580 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f6754ff9640 1 -- 172.21.15.94:0/1256133822 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675000a640 con 0x7f6758150580 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.640+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67580ab8b0 con 0x7f6758150580 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.641+0000 7f6754ff9640 1 -- 172.21.15.94:0/1256133822 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675000aa30 con 0x7f6758150580 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.641+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6758156250 con 0x7f6758150580 2024-03-06T07:23:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.641+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675804f330 con 0x7f6758150580 2024-03-06T07:23:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.642+0000 7f6754ff9640 1 -- 172.21.15.94:0/1256133822 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f675001b070 con 0x7f6758150580 2024-03-06T07:23:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.642+0000 7f6754ff9640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674406b4d0 0x7f674406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.642+0000 7f6754ff9640 1 -- 172.21.15.94:0/1256133822 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f6750089a70 con 0x7f6758150580 2024-03-06T07:23:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.643+0000 7f6757fff640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674406b4d0 0x7f674406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.643+0000 7f6757fff640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674406b4d0 0x7f674406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6748036000 tx=0x7f6748008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.647+0000 7f6754ff9640 1 -- 172.21.15.94:0/1256133822 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67500586a0 con 0x7f6758150580 2024-03-06T07:23:03.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.856+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67580c3a40 con 0x7f6758150580 2024-03-06T07:23:03.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.857+0000 7f6754ff9640 1 -- 172.21.15.94:0/1256133822 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+4988 (secure 0 0 0) 0x7f6750057fa0 con 0x7f6758150580 2024-03-06T07:23:03.859 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:03.859 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":111,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":8112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1268765110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1268765110},{"type":"v1","addr":"172.21.15.94:6834","nonce":1268765110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:02.218006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7540},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7540":{"gid":7540,"name":"b","rank":0,"incarnation":110,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.145:6836/1862149336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1862149336},{"type":"v1","addr":"172.21.15.145:6836","nonce":1862149336}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":9}]} 2024-03-06T07:23:03.859 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 111 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674406b4d0 msgr2=0x7f674406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674406b4d0 0x7f674406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6748036000 tx=0x7f6748008030 comp rx=0 tx=0).stop 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 msgr2=0x7f6758154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f6758154d40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f675000c810 tx=0x7f675000ccd0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 shutdown_connections 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674406b4d0 0x7f674406d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6758150580 0x7f6758154d40 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67580c6000 0x7f6758154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 --2- 172.21.15.94:0/1256133822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67580c1320 0x7f67580bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 >> 172.21.15.94:0/1256133822 conn(0x7f67580c6710 msgr2=0x7f675814dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 shutdown_connections 2024-03-06T07:23:03.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:03.859+0000 7f675e888640 1 -- 172.21.15.94:0/1256133822 wait complete. 2024-03-06T07:23:04.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:42.624925+0000', 'modified': '2024-03-06T07:23:02.218006+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7540}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7540': {'gid': 7540, 'name': 'b', 'rank': 0, 'incarnation': 110, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.145:6836/1862149336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1862149336}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1862149336}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:23:04.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7540:up:reconnect 2024-03-06T07:23:05.276 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:05.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.386+0000 7fb8845c9640 1 Processor -- start 2024-03-06T07:23:05.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.386+0000 7fb8845c9640 1 -- start start 2024-03-06T07:23:05.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb8845c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c152520 0x7fb87c06ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb8845c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb87c0720a0 0x7fb87c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb8845c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0c1ae0 con 0x7fb87c155c80 2024-03-06T07:23:05.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb8845c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c0c1dc0 con 0x7fb87c0720a0 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb8845c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c06e450 con 0x7fb87c152520 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb881b64640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb87c155c80 0x7fb87c0c12b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49278/0 (socket says 172.21.15.94:49278) 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb881b64640 1 -- 172.21.15.94:0/924434483 learned_addr learned my addr 172.21.15.94:0/924434483 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb882365640 1 --2- 172.21.15.94:0/924434483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c152520 0x7fb87c06ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb882b66640 1 --2- 172.21.15.94:0/924434483 >> v2:172.21.15.145:3300/0 conn(0x7fb87c0720a0 0x7fb87c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb881363640 1 -- 172.21.15.94:0/924434483 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389446887 0 0) 0x7fb87c0c1ae0 con 0x7fb87c155c80 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb881363640 1 -- 172.21.15.94:0/924434483 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb868003520 con 0x7fb87c155c80 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb882365640 1 -- 172.21.15.94:0/924434483 >> v2:172.21.15.145:3300/0 conn(0x7fb87c0720a0 msgr2=0x7fb87c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb882365640 1 --2- 172.21.15.94:0/924434483 >> v2:172.21.15.145:3300/0 conn(0x7fb87c0720a0 0x7fb87c06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb882365640 1 -- 172.21.15.94:0/924434483 >> v1:172.21.15.94:6789/0 conn(0x7fb87c155c80 legacy=0x7fb87c0c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb882365640 1 -- 172.21.15.94:0/924434483 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c06e730 con 0x7fb87c152520 2024-03-06T07:23:05.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.387+0000 7fb882b66640 1 --2- 172.21.15.94:0/924434483 >> v2:172.21.15.145:3300/0 conn(0x7fb87c0720a0 0x7fb87c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.388+0000 7fb882365640 1 --2- 172.21.15.94:0/924434483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c152520 0x7fb87c06ef30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb874009d90 tx=0x7fb87402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355f8863d06952f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.388+0000 7fb881363640 1 -- 172.21.15.94:0/924434483 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874039070 con 0x7fb87c152520 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.388+0000 7fb881363640 1 -- 172.21.15.94:0/924434483 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87402cc50 con 0x7fb87c152520 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.388+0000 7fb881363640 1 -- 172.21.15.94:0/924434483 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874034440 con 0x7fb87c152520 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 -- 172.21.15.94:0/924434483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c152520 msgr2=0x7fb87c06ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 --2- 172.21.15.94:0/924434483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c152520 0x7fb87c06ef30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb874009d90 tx=0x7fb87402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 -- 172.21.15.94:0/924434483 shutdown_connections 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 --2- 172.21.15.94:0/924434483 >> v2:172.21.15.145:3300/0 conn(0x7fb87c0720a0 0x7fb87c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 --2- 172.21.15.94:0/924434483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c152520 0x7fb87c06ef30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 -- 172.21.15.94:0/924434483 >> 172.21.15.94:0/924434483 conn(0x7fb87c0c76b0 msgr2=0x7fb87c0c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 -- 172.21.15.94:0/924434483 shutdown_connections 2024-03-06T07:23:05.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.389+0000 7fb8845c9640 1 -- 172.21.15.94:0/924434483 wait complete. 2024-03-06T07:23:05.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.390+0000 7fb8845c9640 1 Processor -- start 2024-03-06T07:23:05.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.390+0000 7fb8845c9640 1 -- start start 2024-03-06T07:23:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb8845c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c0720a0 0x7fb87c1e1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb8845c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb87c152520 0x7fb87c158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb8845c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c155c80 0x7fb87c157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb8845c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c152a80 con 0x7fb87c0720a0 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb8845c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c152900 con 0x7fb87c152520 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb8845c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c06b500 con 0x7fb87c155c80 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c0720a0 0x7fb87c1e1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c0720a0 0x7fb87c1e1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47556/0 (socket says 172.21.15.94:47556) 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 -- 172.21.15.94:0/3907047454 learned_addr learned my addr 172.21.15.94:0/3907047454 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882b66640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c155c80 0x7fb87c157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb881b64640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb87c152520 0x7fb87c158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 -- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c155c80 msgr2=0x7fb87c157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c155c80 0x7fb87c157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 -- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb87c152520 msgr2=0x7fb87c158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb87c152520 0x7fb87c158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 -- 172.21.15.94:0/3907047454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c1e1640 con 0x7fb87c0720a0 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.391+0000 7fb882365640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c0720a0 0x7fb87c1e1100 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb874037040 tx=0x7fb874004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.392+0000 7fb872ffd640 1 -- 172.21.15.94:0/3907047454 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874039050 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.392+0000 7fb872ffd640 1 -- 172.21.15.94:0/3907047454 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874045070 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.392+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c1ead30 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.392+0000 7fb872ffd640 1 -- 172.21.15.94:0/3907047454 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb874009450 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.392+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c1eb2f0 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.393+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c04f330 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.393+0000 7fb872ffd640 1 -- 172.21.15.94:0/3907047454 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb87403e610 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.393+0000 7fb872ffd640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb86006b4d0 0x7fb86006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.394+0000 7fb872ffd640 1 -- 172.21.15.94:0/3907047454 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fb8740acdb0 con 0x7fb87c0720a0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.394+0000 7fb881b64640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb86006b4d0 0x7fb86006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.394+0000 7fb881b64640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb86006b4d0 0x7fb86006d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb87c1572d0 tx=0x7fb86c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.398+0000 7fb872ffd640 1 -- 172.21.15.94:0/3907047454 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb87407bb20 con 0x7fb87c0720a0 2024-03-06T07:23:05.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.605+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb87c0c0b70 con 0x7fb87c0720a0 2024-03-06T07:23:05.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.605+0000 7fb872ffd640 1 -- 172.21.15.94:0/3907047454 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4992 (secure 0 0 0) 0x7fb87407b420 con 0x7fb87c0720a0 2024-03-06T07:23:05.607 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:05.607 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":113,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":8112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1268765110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1268765110},{"type":"v1","addr":"172.21.15.94:6834","nonce":1268765110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:05.210092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7540},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7540":{"gid":7540,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6836/1862149336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1862149336},{"type":"v1","addr":"172.21.15.145:6836","nonce":1862149336}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7540,"qdb_cluster":[7540]},"id":9}]} 2024-03-06T07:23:05.608 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 113 2024-03-06T07:23:05.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb86006b4d0 msgr2=0x7fb86006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb86006b4d0 0x7fb86006d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb87c1572d0 tx=0x7fb86c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c0720a0 msgr2=0x7fb87c1e1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c0720a0 0x7fb87c1e1100 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb874037040 tx=0x7fb874004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 shutdown_connections 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb86006b4d0 0x7fb86006d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb87c155c80 0x7fb87c157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb87c152520 0x7fb87c158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 --2- 172.21.15.94:0/3907047454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb87c0720a0 0x7fb87c1e1100 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 >> 172.21.15.94:0/3907047454 conn(0x7fb87c0c76b0 msgr2=0x7fb87c0c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 shutdown_connections 2024-03-06T07:23:05.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:05.607+0000 7fb8845c9640 1 -- 172.21.15.94:0/3907047454 wait complete. 2024-03-06T07:23:06.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 113, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:42.624925+0000', 'modified': '2024-03-06T07:23:05.210092+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7540}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7540': {'gid': 7540, 'name': 'b', 'rank': 0, 'incarnation': 110, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.145:6836/1862149336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1862149336}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1862149336}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7540, 'qdb_cluster': [7540]} 2024-03-06T07:23:06.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:06.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:23:06.034 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:23:06.032+0000 7f798e912640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:06.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.124+0000 7f23a60c9640 1 Processor -- start 2024-03-06T07:23:06.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.124+0000 7f23a60c9640 1 -- start start 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f23a60c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f23a60c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23a006d990 0x7f23a006de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f23a60c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a0070550 con 0x7f23a00c0fa0 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f23a60c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a006e360 con 0x7f23a006d990 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f23a60c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0070830 con 0x7f23a0072f00 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23a006d990 0x7f23a006de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23a006d990 0x7f23a006de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47566/0 (socket says 172.21.15.94:47566) 2024-03-06T07:23:06.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239ffff640 1 -- 172.21.15.94:0/77637438 learned_addr learned my addr 172.21.15.94:0/77637438 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239f7fe640 1 --2- 172.21.15.94:0/77637438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239ffff640 1 -- 172.21.15.94:0/77637438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 msgr2=0x7f23a0153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239ffff640 1 --2- 172.21.15.94:0/77637438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239ffff640 1 -- 172.21.15.94:0/77637438 >> v1:172.21.15.145:6789/0 conn(0x7f23a00c0fa0 legacy=0x7f23a00c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.125+0000 7f239ffff640 1 -- 172.21.15.94:0/77637438 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0070b10 con 0x7f23a006d990 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.126+0000 7f239ffff640 1 --2- 172.21.15.94:0/77637438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a006d990 0x7f23a006de20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f2398009e50 tx=0x7f239802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8b5437825f7eeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.126+0000 7f239e7fc640 1 -- 172.21.15.94:0/77637438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239802cb00 con 0x7f23a006d990 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.126+0000 7f239e7fc640 1 -- 172.21.15.94:0/77637438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239802cca0 con 0x7f23a006d990 2024-03-06T07:23:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.126+0000 7f239e7fc640 1 -- 172.21.15.94:0/77637438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398034610 con 0x7f23a006d990 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 -- 172.21.15.94:0/77637438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a006d990 msgr2=0x7f23a006de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 --2- 172.21.15.94:0/77637438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a006d990 0x7f23a006de20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f2398009e50 tx=0x7f239802bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 -- 172.21.15.94:0/77637438 shutdown_connections 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 --2- 172.21.15.94:0/77637438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a006d990 0x7f23a006de20 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 --2- 172.21.15.94:0/77637438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 -- 172.21.15.94:0/77637438 >> 172.21.15.94:0/77637438 conn(0x7f23a006b1a0 msgr2=0x7f23a006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 -- 172.21.15.94:0/77637438 shutdown_connections 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.127+0000 7f23a60c9640 1 -- 172.21.15.94:0/77637438 wait complete. 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.128+0000 7f23a60c9640 1 Processor -- start 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.128+0000 7f23a60c9640 1 -- start start 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f23a60c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a006d990 0x7f23a01eca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f23a60c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f23a60c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00c0fa0 0x7f23a01e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f23a60c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00c70d0 con 0x7f23a00c0fa0 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f23a60c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00c6f50 con 0x7f23a006d990 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f23a60c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0070df0 con 0x7f23a0072f00 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00c0fa0 0x7f23a01e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00c0fa0 0x7f23a01e8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47576/0 (socket says 172.21.15.94:47576) 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 -- 172.21.15.94:0/2786325888 learned_addr learned my addr 172.21.15.94:0/2786325888 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239effd640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239f7fe640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a006d990 0x7f23a01eca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 -- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 msgr2=0x7f23a0156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 -- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a006d990 msgr2=0x7f23a01eca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a006d990 0x7f23a01eca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 -- 172.21.15.94:0/2786325888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a01501c0 con 0x7f23a00c0fa0 2024-03-06T07:23:06.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.129+0000 7f239ffff640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00c0fa0 0x7f23a01e8ce0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f2398036040 tx=0x7f2398004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.130+0000 7f2387fff640 1 -- 172.21.15.94:0/2786325888 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398043020 con 0x7f23a00c0fa0 2024-03-06T07:23:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.130+0000 7f2387fff640 1 -- 172.21.15.94:0/2786325888 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239802ce50 con 0x7f23a00c0fa0 2024-03-06T07:23:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.130+0000 7f2387fff640 1 -- 172.21.15.94:0/2786325888 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398009510 con 0x7f23a00c0fa0 2024-03-06T07:23:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.130+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a01504a0 con 0x7f23a00c0fa0 2024-03-06T07:23:06.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.130+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a01dd0d0 con 0x7f23a00c0fa0 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.131+0000 7f2387fff640 1 -- 172.21.15.94:0/2786325888 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2398046020 con 0x7f23a00c0fa0 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.131+0000 7f2387fff640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238c06b500 0x7f238c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.131+0000 7f2387fff640 1 -- 172.21.15.94:0/2786325888 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f23980ac890 con 0x7f23a00c0fa0 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.132+0000 7f239f7fe640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238c06b500 0x7f238c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.132+0000 7f239f7fe640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238c06b500 0x7f238c06d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2390009b40 tx=0x7f2390009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.132+0000 7f23a8d8f740 1 Processor -- start 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.132+0000 7f23a8d8f740 1 -- start start 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b57fc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b57fc550 0x5651b5845be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b57f6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b57606b0 con 0x5651b5845800 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b5709a60 con 0x5651b57fc550 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651b575c750 con 0x5651b5836420 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f239f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b57fc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f239f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b57fc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47586/0 (socket says 172.21.15.94:47586) 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f239f7fe640 1 -- 172.21.15.94:0/4035698317 learned_addr learned my addr 172.21.15.94:0/4035698317 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f239effd640 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b57fc550 0x5651b5845be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.133+0000 7f239ffff640 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b57f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f239f7fe640 1 -- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 msgr2=0x5651b57f6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f239f7fe640 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b57f6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f239f7fe640 1 -- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b57fc550 msgr2=0x5651b5845be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f239f7fe640 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b57fc550 0x5651b5845be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f239f7fe640 1 -- 172.21.15.94:0/4035698317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651b5727f30 con 0x5651b5845800 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f239f7fe640 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b57fc010 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f2390011ee0 tx=0x7f2390014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b06c669a781be6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f23857fa640 1 -- 172.21.15.94:0/4035698317 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239001b070 con 0x5651b5845800 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f23857fa640 1 -- 172.21.15.94:0/4035698317 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2390027070 con 0x5651b5845800 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.134+0000 7f23857fa640 1 -- 172.21.15.94:0/4035698317 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390003a40 con 0x5651b5845800 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4035698317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 msgr2=0x5651b57fc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b57fc010 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f2390011ee0 tx=0x7f2390014000 comp rx=0 tx=0).stop 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4035698317 shutdown_connections 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b57f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b57fc550 0x5651b5845be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4035698317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b57fc010 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4035698317 >> 172.21.15.94:0/4035698317 conn(0x5651b57f87d0 msgr2=0x5651b581d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4035698317 shutdown_connections 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4035698317 wait complete. 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 -- start start 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.135+0000 7f23a8d8f740 1 Processor -- start 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5836420 0x5651b5818c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5845800 0x5651b5841270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b5820310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b583a590 con 0x5651b5836420 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b571cea0 con 0x5651b58192e0 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651b581ddb0 con 0x5651b5845800 2024-03-06T07:23:06.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f23a8d8f740 10 client.0 _ll_register_callbacks cb 0x5651b571c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5836420 0x5651b5818c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5836420 0x5651b5818c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47592/0 (socket says 172.21.15.94:47592) 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 -- 172.21.15.94:0/2166925222 learned_addr learned my addr 172.21.15.94:0/2166925222 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239effd640 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5845800 0x5651b5841270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239ffff640 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b5820310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 -- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5845800 msgr2=0x5651b5841270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5845800 0x5651b5841270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 -- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 msgr2=0x5651b5820310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b5820310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 -- 172.21.15.94:0/2166925222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651b571d270 con 0x5651b5836420 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f239f7fe640 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5836420 0x5651b5818c60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f23900096a0 tx=0x7f2390004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69579e5a495d8850 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.136+0000 7f237e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f237e7fc640 1 -- 172.21.15.94:0/2166925222 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239001b040 con 0x5651b5836420 2024-03-06T07:23:06.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f237e7fc640 1 -- 172.21.15.94:0/2166925222 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f2390022880 con 0x5651b5836420 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2390027070 con 0x5651b5836420 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f237e7fc640 1 -- 172.21.15.94:0/2166925222 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f239003d050 con 0x5651b5836420 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 10 client.8139 fetch_fsmap learned FSMap version 113 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x5651b54752d0 con 0x5651b5836420 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f237e7fc640 1 -- 172.21.15.94:0/2166925222 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2390027070 con 0x5651b5836420 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f237e7fc640 1 -- 172.21.15.94:0/2166925222 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 113) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2390040020 con 0x5651b5836420 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 10 client.8139 fetch_fsmap finished waiting for FSMap version 113 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 10 client.8139 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 20 client.8139 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b58209d0 0x5651b57faa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 4 client.8139 mds_command: new command op to 7540 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 --> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651b576dbf0 con 0x5651b58209d0 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.137+0000 7f239effd640 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b58209d0 0x5651b57faa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f237e7fc640 10 client.8139 ms_handle_connect on v2:172.21.15.145:6834/1862149336 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f239effd640 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b58209d0 0x5651b57faa50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3efa61365456605 server_cookie=2fc75fe755b5e3ba in_seq=0 out_seq=0 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f237e7fc640 1 -- 172.21.15.94:0/2166925222 <== mds.0 v2:172.21.15.145:6834/1862149336 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5651b576dbf0 con 0x5651b58209d0 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f237e7fc640 10 client.8139 handle_command_reply: tid=0 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 client.8139 shutdown 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5836420 msgr2=0x5651b5818c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5836420 0x5651b5818c60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f23900096a0 tx=0x7f2390004730 comp rx=0 tx=0).stop 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 shutdown_connections 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b58209d0 0x5651b57faa50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b5820310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5845800 0x5651b5841270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f237e7fc640 0 client.8139 ms_handle_reset on v2:172.21.15.145:6834/1862149336 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/2166925222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5836420 0x5651b5818c60 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.138+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 >> 172.21.15.94:0/2166925222 conn(0x5651b57fc550 msgr2=0x5651b5842bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.139+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 shutdown_connections 2024-03-06T07:23:06.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.139+0000 7f23a8d8f740 1 -- 172.21.15.94:0/2166925222 wait complete. 2024-03-06T07:23:06.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.139+0000 7f23a8d8f740 20 client.8139 trim_cache size 0 max 16384 2024-03-06T07:23:06.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.155+0000 7f23a8d8f740 1 Processor -- start 2024-03-06T07:23:06.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.155+0000 7f23a8d8f740 1 -- start start 2024-03-06T07:23:06.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b581d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b581db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b5857ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00ab100 con 0x5651b5845800 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a0149f60 con 0x5651b58192e0 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a00aab00 con 0x5651b5836420 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b5857ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b5857ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47600/0 (socket says 172.21.15.94:47600) 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 -- 172.21.15.94:0/542490121 learned_addr learned my addr 172.21.15.94:0/542490121 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239effd640 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b581db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239f7fe640 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b581d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 -- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 msgr2=0x5651b581db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b581db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 -- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 msgr2=0x5651b581d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b581d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 -- 172.21.15.94:0/542490121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651b553f6b0 con 0x5651b5845800 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f239ffff640 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b5857ae0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f239807af30 tx=0x7f23980ac040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cefb29605adc749 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f2384ff9640 1 -- 172.21.15.94:0/542490121 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23980a9070 con 0x5651b5845800 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f2384ff9640 1 -- 172.21.15.94:0/542490121 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239807f040 con 0x5651b5845800 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.156+0000 7f2384ff9640 1 -- 172.21.15.94:0/542490121 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398083600 con 0x5651b5845800 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 -- 172.21.15.94:0/542490121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 msgr2=0x5651b5857ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b5857ae0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f239807af30 tx=0x7f23980ac040 comp rx=0 tx=0).stop 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 -- 172.21.15.94:0/542490121 shutdown_connections 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b5857ae0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b5836420 0x5651b581db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/542490121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b58192e0 0x5651b581d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 -- 172.21.15.94:0/542490121 >> 172.21.15.94:0/542490121 conn(0x5651b57fc550 msgr2=0x5651b581a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 -- 172.21.15.94:0/542490121 shutdown_connections 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.157+0000 7f23a8d8f740 1 -- 172.21.15.94:0/542490121 wait complete. 2024-03-06T07:23:06.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 -- start start 2024-03-06T07:23:06.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 Processor -- start 2024-03-06T07:23:06.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b58192e0 0x5651b5840af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b5836420 0x5651b5831c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b583b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b5820850 con 0x5651b5845800 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651b58444f0 con 0x5651b5836420 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651b5844870 con 0x5651b58192e0 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f23a8d8f740 10 client.0 _ll_register_callbacks cb 0x7f2390011de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f239ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b583b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f239ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b583b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47604/0 (socket says 172.21.15.94:47604) 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f239ffff640 1 -- 172.21.15.94:0/4102171027 learned_addr learned my addr 172.21.15.94:0/4102171027 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f239f7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b58192e0 0x5651b5840af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.158+0000 7f239effd640 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b5836420 0x5651b5831c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f239ffff640 1 -- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b58192e0 msgr2=0x5651b5840af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f239ffff640 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b58192e0 0x5651b5840af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f239ffff640 1 -- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b5836420 msgr2=0x5651b5831c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f239ffff640 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b5836420 0x5651b5831c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f239ffff640 1 -- 172.21.15.94:0/4102171027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651b58356c0 con 0x5651b5845800 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f239ffff640 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b583b960 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f239802c570 tx=0x7f2398081040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d425fe66c430bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f237e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f237e7fc640 1 -- 172.21.15.94:0/4102171027 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398089060 con 0x5651b5845800 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f237e7fc640 1 -- 172.21.15.94:0/4102171027 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f239802cb10 con 0x5651b5845800 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f239807f040 con 0x5651b5845800 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f237e7fc640 1 -- 172.21.15.94:0/4102171027 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23980a9040 con 0x5651b5845800 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 10 client.8145 fetch_fsmap learned FSMap version 113 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x5651b583c450 con 0x5651b5845800 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f237e7fc640 1 -- 172.21.15.94:0/4102171027 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239807f040 con 0x5651b5845800 2024-03-06T07:23:06.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f237e7fc640 1 -- 172.21.15.94:0/4102171027 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 113) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f239807b200 con 0x5651b5845800 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 10 client.8145 fetch_fsmap finished waiting for FSMap version 113 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 10 client.8145 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 20 client.8145 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b583f2d0 0x5651b58b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 4 client.8145 mds_command: new command op to 7540 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.159+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 --> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5651b5767f40 con 0x5651b583f2d0 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f239f7fe640 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b583f2d0 0x5651b58b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f237e7fc640 10 client.8145 ms_handle_connect on v2:172.21.15.145:6834/1862149336 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f239f7fe640 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b583f2d0 0x5651b58b0a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ae6162b63dd0518 server_cookie=97be198e2409b5e3 in_seq=0 out_seq=0 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f237e7fc640 1 -- 172.21.15.94:0/4102171027 <== mds.0 v2:172.21.15.145:6834/1862149336 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5651b5767f40 con 0x5651b583f2d0 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f237e7fc640 10 client.8145 handle_command_reply: tid=0 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 client.8145 shutdown 2024-03-06T07:23:06.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 msgr2=0x5651b583b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b583b960 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f239802c570 tx=0x7f2398081040 comp rx=0 tx=0).stop 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 shutdown_connections 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:6834/1862149336,v1:172.21.15.145:6836/1862149336] conn(0x5651b583f2d0 0x5651b58b0a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5651b5845800 0x5651b583b960 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5651b5836420 0x5651b5831c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f237e7fc640 0 client.8145 ms_handle_reset on v2:172.21.15.145:6834/1862149336 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 --2- 172.21.15.94:0/4102171027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5651b58192e0 0x5651b5840af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 >> 172.21.15.94:0/4102171027 conn(0x5651b57fc550 msgr2=0x5651b581d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 shutdown_connections 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.160+0000 7f23a8d8f740 1 -- 172.21.15.94:0/4102171027 wait complete. 2024-03-06T07:23:06.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a8d8f740 20 client.8145 trim_cache size 0 max 16384 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7540, 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 3.9539932489999998, 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 113, 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 123, 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 123, 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 6.10884698 2024-03-06T07:23:06.170 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:06.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238c06b500 msgr2=0x7f238c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238c06b500 0x7f238c06d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2390009b40 tx=0x7f2390009190 comp rx=0 tx=0).stop 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00c0fa0 msgr2=0x7f23a01e8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00c0fa0 0x7f23a01e8ce0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f2398036040 tx=0x7f2398004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 shutdown_connections 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f238c06b500 0x7f238c06d9c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23a00c0fa0 0x7f23a01e8ce0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23a0072f00 0x7f23a0156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 --2- 172.21.15.94:0/2786325888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23a006d990 0x7f23a01eca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 >> 172.21.15.94:0/2786325888 conn(0x7f23a006b1a0 msgr2=0x7f23a006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 shutdown_connections 2024-03-06T07:23:06.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.161+0000 7f23a60c9640 1 -- 172.21.15.94:0/2786325888 wait complete. 2024-03-06T07:23:06.173 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 113, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:22:42.624925+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 113, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7540": { "addr": "172.21.15.145:6836/1862149336", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1862149336, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1862149336, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7540, "incarnation": 110, "join_fscid": 9, "name": "b", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 123, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-03-06T07:23:05.210092+0000", "qdb_cluster": [ 7540 ], "qdb_leader": 7540, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7540 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/1356561672", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 1356561672, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 1356561672, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7321, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/1458830160", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 1458830160, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 1458830160, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 101, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7824, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6834/1268765110", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 1268765110, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 1268765110, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8112, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:06.174 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 9, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:06.174 INFO:tasks.cephfs.test_failover:current = [{'gid': 7824, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/1458830160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 1458830160}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 1458830160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}, {'gid': 7540, 'name': 'b', 'rank': 0, 'incarnation': 110, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.145:6836/1862149336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1862149336}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1862149336}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8112, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6834/1268765110', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 1268765110}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 1268765110}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 111}, {'gid': 7321, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/1356561672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1356561672}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 1356561672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 101}] 2024-03-06T07:23:06.175 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:23:06.175 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:06.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:06.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:06.260 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:06.261 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:23:06.261 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:06.277 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:23:06.278 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:23:06.278 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:06.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:23:06.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:06.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.416+0000 7f0079f5c640 1 Processor -- start 2024-03-06T07:23:06.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.416+0000 7f0079f5c640 1 -- start start 2024-03-06T07:23:06.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.416+0000 7f0079f5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00740c1aa0 0x7f00741529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.416+0000 7f0079f5c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00740c28d0 con 0x7f007414f790 2024-03-06T07:23:06.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.416+0000 7f0079f5c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00740c2bb0 con 0x7f0074155a40 2024-03-06T07:23:06.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.416+0000 7f0079f5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00740c2680 con 0x7f00740c1aa0 2024-03-06T07:23:06.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.416+0000 7f00737fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f007414f790 0x7f00740c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49286/0 (socket says 172.21.15.94:49286) 2024-03-06T07:23:06.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f00737fe640 1 -- 172.21.15.94:0/2803732338 learned_addr learned my addr 172.21.15.94:0/2803732338 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f006affd640 1 --2- 172.21.15.94:0/2803732338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00740c1aa0 0x7f00741529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f0072ffd640 1 -- 172.21.15.94:0/2803732338 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797506469 0 0) 0x7f00740c28d0 con 0x7f007414f790 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f0072ffd640 1 -- 172.21.15.94:0/2803732338 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0060003520 con 0x7f007414f790 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f0072ffd640 1 -- 172.21.15.94:0/2803732338 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188840098 0 0) 0x7f00740c2bb0 con 0x7f0074155a40 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f0072ffd640 1 -- 172.21.15.94:0/2803732338 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00740c28d0 con 0x7f0074155a40 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f006affd640 1 -- 172.21.15.94:0/2803732338 >> v1:172.21.15.145:6789/0 conn(0x7f0074155a40 legacy=0x7f0074155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f006affd640 1 -- 172.21.15.94:0/2803732338 >> v1:172.21.15.94:6789/0 conn(0x7f007414f790 legacy=0x7f00740c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.417+0000 7f006affd640 1 -- 172.21.15.94:0/2803732338 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00740c2e90 con 0x7f00740c1aa0 2024-03-06T07:23:06.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f006affd640 1 --2- 172.21.15.94:0/2803732338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00740c1aa0 0x7f00741529d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f005c0097d0 tx=0x7f005c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=652b2beb55f5a2f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0072ffd640 1 -- 172.21.15.94:0/2803732338 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c035020 con 0x7f00740c1aa0 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0072ffd640 1 -- 172.21.15.94:0/2803732338 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c031890 con 0x7f00740c1aa0 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0072ffd640 1 -- 172.21.15.94:0/2803732338 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c031c60 con 0x7f00740c1aa0 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0079f5c640 1 -- 172.21.15.94:0/2803732338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00740c1aa0 msgr2=0x7f00741529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0079f5c640 1 --2- 172.21.15.94:0/2803732338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00740c1aa0 0x7f00741529d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f005c0097d0 tx=0x7f005c02b930 comp rx=0 tx=0).stop 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0079f5c640 1 -- 172.21.15.94:0/2803732338 shutdown_connections 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0079f5c640 1 --2- 172.21.15.94:0/2803732338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00740c1aa0 0x7f00741529d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.418+0000 7f0079f5c640 1 -- 172.21.15.94:0/2803732338 >> 172.21.15.94:0/2803732338 conn(0x7f00740c6710 msgr2=0x7f00740c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.419+0000 7f0079f5c640 1 -- 172.21.15.94:0/2803732338 shutdown_connections 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.419+0000 7f0079f5c640 1 -- 172.21.15.94:0/2803732338 wait complete. 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 Processor -- start 2024-03-06T07:23:06.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 -- start start 2024-03-06T07:23:06.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00740c1aa0 0x7f0074070950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f007414f790 0x7f00740c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0074155a40 0x7f0074148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00740c5e30 con 0x7f00740c1aa0 2024-03-06T07:23:06.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00740c0d60 con 0x7f007414f790 2024-03-06T07:23:06.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.420+0000 7f0079f5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00740c5fb0 con 0x7f0074155a40 2024-03-06T07:23:06.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00740c1aa0 0x7f0074070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00740c1aa0 0x7f0074070950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47620/0 (socket says 172.21.15.94:47620) 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 -- 172.21.15.94:0/1102601107 learned_addr learned my addr 172.21.15.94:0/1102601107 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f006affd640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f007414f790 0x7f00740c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f0073fff640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0074155a40 0x7f0074148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 -- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0074155a40 msgr2=0x7f0074148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0074155a40 0x7f0074148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 -- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f007414f790 msgr2=0x7f00740c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f007414f790 0x7f00740c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 -- 172.21.15.94:0/1102601107 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0074070e90 con 0x7f00740c1aa0 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f0073fff640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0074155a40 0x7f0074148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f00737fe640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00740c1aa0 0x7f0074070950 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f006400c500 tx=0x7f006400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f0070ff9640 1 -- 172.21.15.94:0/1102601107 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064017070 con 0x7f00740c1aa0 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f0070ff9640 1 -- 172.21.15.94:0/1102601107 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064005ce0 con 0x7f00740c1aa0 2024-03-06T07:23:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f0070ff9640 1 -- 172.21.15.94:0/1102601107 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00640072f0 con 0x7f00740c1aa0 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00740a7740 con 0x7f00740c1aa0 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.421+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00740a7d00 con 0x7f00740c1aa0 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.422+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0074069460 con 0x7f00740c1aa0 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.423+0000 7f0070ff9640 1 -- 172.21.15.94:0/1102601107 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f006401a070 con 0x7f00740c1aa0 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.423+0000 7f0070ff9640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f005406b4d0 0x7f005406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.423+0000 7f0070ff9640 1 -- 172.21.15.94:0/1102601107 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f00640896d0 con 0x7f00740c1aa0 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.423+0000 7f006affd640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f005406b4d0 0x7f005406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.424+0000 7f006affd640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f005406b4d0 0x7f005406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f005c036000 tx=0x7f005c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.427+0000 7f0070ff9640 1 -- 172.21.15.94:0/1102601107 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0074069460 con 0x7f00740c1aa0 2024-03-06T07:23:06.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.633+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f007404f330 con 0x7f00740c1aa0 2024-03-06T07:23:06.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.634+0000 7f0070ff9640 1 -- 172.21.15.94:0/1102601107 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4992 (secure 0 0 0) 0x7f0074070e90 con 0x7f00740c1aa0 2024-03-06T07:23:06.636 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:06.636 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":113,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":8112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1268765110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1268765110},{"type":"v1","addr":"172.21.15.94:6834","nonce":1268765110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:05.210092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7540},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7540":{"gid":7540,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6836/1862149336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1862149336},{"type":"v1","addr":"172.21.15.145:6836","nonce":1862149336}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7540,"qdb_cluster":[7540]},"id":9}]} 2024-03-06T07:23:06.637 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 113 2024-03-06T07:23:06.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f005406b4d0 msgr2=0x7f005406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f005406b4d0 0x7f005406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f005c036000 tx=0x7f005c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:06.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00740c1aa0 msgr2=0x7f0074070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00740c1aa0 0x7f0074070950 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f006400c500 tx=0x7f006400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 shutdown_connections 2024-03-06T07:23:06.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f005406b4d0 0x7f005406d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0074155a40 0x7f0074148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f007414f790 0x7f00740c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 --2- 172.21.15.94:0/1102601107 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00740c1aa0 0x7f0074070950 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 >> 172.21.15.94:0/1102601107 conn(0x7f00740c6710 msgr2=0x7f007406cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 shutdown_connections 2024-03-06T07:23:06.639 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.636+0000 7f0079f5c640 1 -- 172.21.15.94:0/1102601107 wait complete. 2024-03-06T07:23:06.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.747+0000 7ffb9bf02640 1 Processor -- start 2024-03-06T07:23:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9bf02640 1 -- start start 2024-03-06T07:23:06.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9bf02640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb9414f360 0x7ffb94155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9bf02640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb94158b60 0x7ffb94158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9bf02640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940c73d0 con 0x7ffb940c88c0 2024-03-06T07:23:06.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9bf02640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb940c8dd0 con 0x7ffb94158b60 2024-03-06T07:23:06.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9bf02640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940be2e0 con 0x7ffb9414f360 2024-03-06T07:23:06.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9a49f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb94158b60 0x7ffb94158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9a49f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb94158b60 0x7ffb94158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47630/0 (socket says 172.21.15.94:47630) 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb9a49f640 1 -- 172.21.15.94:0/458417935 learned_addr learned my addr 172.21.15.94:0/458417935 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.748+0000 7ffb99c9e640 1 --2- 172.21.15.94:0/458417935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb9414f360 0x7ffb94155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.749+0000 7ffb9a49f640 1 -- 172.21.15.94:0/458417935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb9414f360 msgr2=0x7ffb94155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.749+0000 7ffb9a49f640 1 --2- 172.21.15.94:0/458417935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb9414f360 0x7ffb94155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.749+0000 7ffb9a49f640 1 -- 172.21.15.94:0/458417935 >> v1:172.21.15.145:6789/0 conn(0x7ffb940c88c0 legacy=0x7ffb940c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.749+0000 7ffb9a49f640 1 -- 172.21.15.94:0/458417935 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940c7670 con 0x7ffb94158b60 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.749+0000 7ffb98c9c640 1 -- 172.21.15.94:0/458417935 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065299489 0 0) 0x7ffb940c73d0 con 0x7ffb940c88c0 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.749+0000 7ffb9a49f640 1 --2- 172.21.15.94:0/458417935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb94158b60 0x7ffb94158ff0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009e50 tx=0x7ffb8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788851ef9957019c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb98c9c640 1 -- 172.21.15.94:0/458417935 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c02ca70 con 0x7ffb94158b60 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb98c9c640 1 -- 172.21.15.94:0/458417935 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8c02cc10 con 0x7ffb94158b60 2024-03-06T07:23:06.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb98c9c640 1 -- 172.21.15.94:0/458417935 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c0343f0 con 0x7ffb94158b60 2024-03-06T07:23:06.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 -- 172.21.15.94:0/458417935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb94158b60 msgr2=0x7ffb94158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/458417935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb94158b60 0x7ffb94158ff0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009e50 tx=0x7ffb8c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 -- 172.21.15.94:0/458417935 shutdown_connections 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/458417935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb94158b60 0x7ffb94158ff0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/458417935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb9414f360 0x7ffb94155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 -- 172.21.15.94:0/458417935 >> 172.21.15.94:0/458417935 conn(0x7ffb940ac570 msgr2=0x7ffb940ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 -- 172.21.15.94:0/458417935 shutdown_connections 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.750+0000 7ffb9bf02640 1 -- 172.21.15.94:0/458417935 wait complete. 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.751+0000 7ffb9bf02640 1 Processor -- start 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.751+0000 7ffb9bf02640 1 -- start start 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9bf02640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c88c0 0x7ffb941ee460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9bf02640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9414f360 0x7ffb9414bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9bf02640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb94158b60 0x7ffb9414c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9bf02640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940c5a50 con 0x7ffb9414f360 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9bf02640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940c7c30 con 0x7ffb940c88c0 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9bf02640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb940c5bd0 con 0x7ffb94158b60 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9414f360 0x7ffb9414bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9414f360 0x7ffb9414bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47638/0 (socket says 172.21.15.94:47638) 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 -- 172.21.15.94:0/4218616005 learned_addr learned my addr 172.21.15.94:0/4218616005 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb99c9e640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c88c0 0x7ffb941ee460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9a49f640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb94158b60 0x7ffb9414c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 -- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb94158b60 msgr2=0x7ffb9414c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb94158b60 0x7ffb9414c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 -- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c88c0 msgr2=0x7ffb941ee460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c88c0 0x7ffb941ee460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.752+0000 7ffb9949d640 1 -- 172.21.15.94:0/4218616005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940c00c0 con 0x7ffb9414f360 2024-03-06T07:23:06.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.753+0000 7ffb9949d640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9414f360 0x7ffb9414bc50 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c410 tx=0x7ffb8800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.753+0000 7ffb867fc640 1 -- 172.21.15.94:0/4218616005 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88017070 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.753+0000 7ffb867fc640 1 -- 172.21.15.94:0/4218616005 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb880035f0 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.753+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb941eee50 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.753+0000 7ffb867fc640 1 -- 172.21.15.94:0/4218616005 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb88003a00 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.753+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb940c1060 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.754+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb94075300 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.754+0000 7ffb867fc640 1 -- 172.21.15.94:0/4218616005 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb8801a070 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.754+0000 7ffb867fc640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb7806b4d0 0x7ffb7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.755+0000 7ffb867fc640 1 -- 172.21.15.94:0/4218616005 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7ffb880896f0 con 0x7ffb9414f360 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.755+0000 7ffb99c9e640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb7806b4d0 0x7ffb7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.755+0000 7ffb99c9e640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb7806b4d0 0x7ffb7806d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009aa0 tx=0x7ffb90009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.759+0000 7ffb867fc640 1 -- 172.21.15.94:0/4218616005 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb94075300 con 0x7ffb9414f360 2024-03-06T07:23:06.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.930+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb9404f330 con 0x7ffb9414f360 2024-03-06T07:23:06.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.931+0000 7ffb867fc640 1 -- 172.21.15.94:0/4218616005 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+17157 (secure 0 0 0) 0x7ffb8805e0c0 con 0x7ffb9414f360 2024-03-06T07:23:06.933 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:06.933 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":123,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:02.205043+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:40.848613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:41.593837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.94:6832/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.145:6836/198029515":"2024-03-07T07:23:00.021121+0000","172.21.15.145:6834/198029515":"2024-03-07T07:23:00.021121+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:06.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb7806b4d0 msgr2=0x7ffb7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb7806b4d0 0x7ffb7806d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009aa0 tx=0x7ffb90009190 comp rx=0 tx=0).stop 2024-03-06T07:23:06.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9414f360 msgr2=0x7ffb9414bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9414f360 0x7ffb9414bc50 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c410 tx=0x7ffb8800c8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 shutdown_connections 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb7806b4d0 0x7ffb7806d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb94158b60 0x7ffb9414c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb9414f360 0x7ffb9414bc50 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 --2- 172.21.15.94:0/4218616005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb940c88c0 0x7ffb941ee460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 >> 172.21.15.94:0/4218616005 conn(0x7ffb940ac570 msgr2=0x7ffb9414d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 shutdown_connections 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:06.933+0000 7ffb9bf02640 1 -- 172.21.15.94:0/4218616005 wait complete. 2024-03-06T07:23:06.946 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:40.848613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:41.593837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:06.946 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:07.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.048+0000 7f895308d640 1 Processor -- start 2024-03-06T07:23:07.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.048+0000 7f895308d640 1 -- start start 2024-03-06T07:23:07.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f895308d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f894c0b0070 0x7f894c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f895308d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c06ca70 0x7f894c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f895308d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c0b0450 con 0x7f894c0c5380 2024-03-06T07:23:07.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f895308d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894c06c020 con 0x7f894c0b0070 2024-03-06T07:23:07.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f895308d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c06cf20 con 0x7f894c06ca70 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f894bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c06ca70 0x7f894c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f8950e29640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f894c0b0070 0x7f894c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f895162a640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f894c0c5380 0x7f894c14e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:38784/0 (socket says 172.21.15.94:38784) 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f894bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c06ca70 0x7f894c0c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:58186/0 (socket says 172.21.15.94:58186) 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.049+0000 7f895162a640 1 -- 172.21.15.94:0/1219780695 learned_addr learned my addr 172.21.15.94:0/1219780695 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f894b7fe640 1 -- 172.21.15.94:0/1219780695 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499973569 0 0) 0x7f8944001490 con 0x7f894c0c5380 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f894b7fe640 1 -- 172.21.15.94:0/1219780695 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934003520 con 0x7f894c0c5380 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f894bfff640 1 -- 172.21.15.94:0/1219780695 >> v2:172.21.15.145:3300/0 conn(0x7f894c0b0070 msgr2=0x7f894c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f894bfff640 1 --2- 172.21.15.94:0/1219780695 >> v2:172.21.15.145:3300/0 conn(0x7f894c0b0070 0x7f894c155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f894bfff640 1 -- 172.21.15.94:0/1219780695 >> v1:172.21.15.145:6789/0 conn(0x7f894c0c5380 legacy=0x7f894c14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f894bfff640 1 -- 172.21.15.94:0/1219780695 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c0c77f0 con 0x7f894c06ca70 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f8950e29640 1 --2- 172.21.15.94:0/1219780695 >> v2:172.21.15.145:3300/0 conn(0x7f894c0b0070 0x7f894c155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:07.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.050+0000 7f894bfff640 1 --2- 172.21.15.94:0/1219780695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c06ca70 0x7f894c0c8900 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f89400097a0 tx=0x7f894002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257d3f2a6d87ab8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f894b7fe640 1 -- 172.21.15.94:0/1219780695 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89400315a0 con 0x7f894c06ca70 2024-03-06T07:23:07.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f894b7fe640 1 -- 172.21.15.94:0/1219780695 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8940037070 con 0x7f894c06ca70 2024-03-06T07:23:07.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f894b7fe640 1 -- 172.21.15.94:0/1219780695 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940031940 con 0x7f894c06ca70 2024-03-06T07:23:07.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 -- 172.21.15.94:0/1219780695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c06ca70 msgr2=0x7f894c0c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 --2- 172.21.15.94:0/1219780695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c06ca70 0x7f894c0c8900 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f89400097a0 tx=0x7f894002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:07.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 -- 172.21.15.94:0/1219780695 shutdown_connections 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 --2- 172.21.15.94:0/1219780695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c06ca70 0x7f894c0c8900 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 --2- 172.21.15.94:0/1219780695 >> v2:172.21.15.145:3300/0 conn(0x7f894c0b0070 0x7f894c155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 -- 172.21.15.94:0/1219780695 >> 172.21.15.94:0/1219780695 conn(0x7f894c0ac590 msgr2=0x7f894c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 -- 172.21.15.94:0/1219780695 shutdown_connections 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.051+0000 7f895308d640 1 -- 172.21.15.94:0/1219780695 wait complete. 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.052+0000 7f895308d640 1 Processor -- start 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.052+0000 7f895308d640 1 -- start start 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895308d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f894c06ca70 0x7f894c0c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895308d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c0b0070 0x7f894c14e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895308d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f894c0c5380 0x7f894c157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895308d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c06c550 con 0x7f894c0c5380 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895308d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c06c270 con 0x7f894c06ca70 2024-03-06T07:23:07.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895308d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c06da80 con 0x7f894c0b0070 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f894c0c5380 0x7f894c157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f894bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c0b0070 0x7f894c14e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f894c0c5380 0x7f894c157f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47640/0 (socket says 172.21.15.94:47640) 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 -- 172.21.15.94:0/3690830163 learned_addr learned my addr 172.21.15.94:0/3690830163 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f8950e29640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f894c06ca70 0x7f894c0c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 -- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c0b0070 msgr2=0x7f894c14e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c0b0070 0x7f894c14e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 -- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f894c06ca70 msgr2=0x7f894c0c14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f894c06ca70 0x7f894c0c14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.053+0000 7f895162a640 1 -- 172.21.15.94:0/3690830163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c1eed70 con 0x7f894c0c5380 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.054+0000 7f895162a640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f894c0c5380 0x7f894c157f80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f894400cb50 tx=0x7f894400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.054+0000 7f89497fa640 1 -- 172.21.15.94:0/3690830163 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8944017070 con 0x7f894c0c5380 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.054+0000 7f89497fa640 1 -- 172.21.15.94:0/3690830163 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894400ad40 con 0x7f894c0c5380 2024-03-06T07:23:07.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.054+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894c1ef050 con 0x7f894c0c5380 2024-03-06T07:23:07.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.054+0000 7f89497fa640 1 -- 172.21.15.94:0/3690830163 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8944012620 con 0x7f894c0c5380 2024-03-06T07:23:07.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.054+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c0c4bd0 con 0x7f894c0c5380 2024-03-06T07:23:07.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.055+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c0c85d0 con 0x7f894c0c5380 2024-03-06T07:23:07.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.055+0000 7f89497fa640 1 -- 172.21.15.94:0/3690830163 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f894401a070 con 0x7f894c0c5380 2024-03-06T07:23:07.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.056+0000 7f89497fa640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f893806b4d0 0x7f893806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.056+0000 7f89497fa640 1 -- 172.21.15.94:0/3690830163 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f894408a180 con 0x7f894c0c5380 2024-03-06T07:23:07.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.056+0000 7f8950e29640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f893806b4d0 0x7f893806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.056+0000 7f8950e29640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f893806b4d0 0x7f893806d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f894c157e60 tx=0x7f893c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.060+0000 7f89497fa640 1 -- 172.21.15.94:0/3690830163 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f894c0c85d0 con 0x7f894c0c5380 2024-03-06T07:23:07.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.233+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f894c0c46e0 con 0x7f894c0c5380 2024-03-06T07:23:07.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.233+0000 7f89497fa640 1 -- 172.21.15.94:0/3690830163 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7f894c0b0450 con 0x7f894c0c5380 2024-03-06T07:23:07.235 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:23:07.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f893806b4d0 msgr2=0x7f893806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f893806b4d0 0x7f893806d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f894c157e60 tx=0x7f893c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:07.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f894c0c5380 msgr2=0x7f894c157f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f894c0c5380 0x7f894c157f80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f894400cb50 tx=0x7f894400a430 comp rx=0 tx=0).stop 2024-03-06T07:23:07.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 shutdown_connections 2024-03-06T07:23:07.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f893806b4d0 0x7f893806d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f894c0c5380 0x7f894c157f80 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f894c0b0070 0x7f894c14e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 --2- 172.21.15.94:0/3690830163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f894c06ca70 0x7f894c0c14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 >> 172.21.15.94:0/3690830163 conn(0x7f894c0ac590 msgr2=0x7f894c155890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 shutdown_connections 2024-03-06T07:23:07.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.235+0000 7f895308d640 1 -- 172.21.15.94:0/3690830163 wait complete. 2024-03-06T07:23:07.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:07.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.351+0000 7fdc20450640 1 Processor -- start 2024-03-06T07:23:07.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.351+0000 7fdc20450640 1 -- start start 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc20450640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c26c0 0x7fdc18155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc20450640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdc180c2aa0 0x7fdc1806bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc20450640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdc180c0470 0x7fdc18152350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc20450640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1806b740 con 0x7fdc180c2aa0 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc20450640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1806c110 con 0x7fdc180c0470 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc20450640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180670c0 con 0x7fdc180c26c0 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdc180c2aa0 0x7fdc1806bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdc180c2aa0 0x7fdc1806bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47646/0 (socket says 172.21.15.94:47646) 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 -- 172.21.15.94:0/4126573641 learned_addr learned my addr 172.21.15.94:0/4126573641 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1e1ec640 1 --2- 172.21.15.94:0/4126573641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c26c0 0x7fdc18155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1e9ed640 1 --2- 172.21.15.94:0/4126573641 >> v2:172.21.15.145:3300/0 conn(0x7fdc180c0470 0x7fdc18152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 -- 172.21.15.94:0/4126573641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c26c0 msgr2=0x7fdc18155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 --2- 172.21.15.94:0/4126573641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c26c0 0x7fdc18155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 -- 172.21.15.94:0/4126573641 >> v2:172.21.15.145:3300/0 conn(0x7fdc180c0470 msgr2=0x7fdc18152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 --2- 172.21.15.94:0/4126573641 >> v2:172.21.15.145:3300/0 conn(0x7fdc180c0470 0x7fdc18152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.352+0000 7fdc1d9eb640 1 -- 172.21.15.94:0/4126573641 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1806b0f0 con 0x7fdc180c2aa0 2024-03-06T07:23:07.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.353+0000 7fdc1d9eb640 1 --2- 172.21.15.94:0/4126573641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc1806bbd0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fdc180c49f0 tx=0x7fdc0802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c123534ee7ee55c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.353+0000 7fdc1d1ea640 1 -- 172.21.15.94:0/4126573641 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08038070 con 0x7fdc180c2aa0 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.353+0000 7fdc1d1ea640 1 -- 172.21.15.94:0/4126573641 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0802f050 con 0x7fdc180c2aa0 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.353+0000 7fdc1d1ea640 1 -- 172.21.15.94:0/4126573641 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08031970 con 0x7fdc180c2aa0 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 -- 172.21.15.94:0/4126573641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 msgr2=0x7fdc1806bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 --2- 172.21.15.94:0/4126573641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc1806bbd0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fdc180c49f0 tx=0x7fdc0802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 -- 172.21.15.94:0/4126573641 shutdown_connections 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 --2- 172.21.15.94:0/4126573641 >> v2:172.21.15.145:3300/0 conn(0x7fdc180c0470 0x7fdc18152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 --2- 172.21.15.94:0/4126573641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc1806bbd0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 --2- 172.21.15.94:0/4126573641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c26c0 0x7fdc18155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 -- 172.21.15.94:0/4126573641 >> 172.21.15.94:0/4126573641 conn(0x7fdc180c3b40 msgr2=0x7fdc180c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 -- 172.21.15.94:0/4126573641 shutdown_connections 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.354+0000 7fdc20450640 1 -- 172.21.15.94:0/4126573641 wait complete. 2024-03-06T07:23:07.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 Processor -- start 2024-03-06T07:23:07.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 -- start start 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c0470 0x7fdc180c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdc180c26c0 0x7fdc1813ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc181ee540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180c1f30 con 0x7fdc180c2aa0 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180c0900 con 0x7fdc180c26c0 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.355+0000 7fdc20450640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180c20b0 con 0x7fdc180c0470 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc181ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc181ee540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47656/0 (socket says 172.21.15.94:47656) 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 -- 172.21.15.94:0/3273825506 learned_addr learned my addr 172.21.15.94:0/3273825506 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e1ec640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c0470 0x7fdc180c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1d9eb640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdc180c26c0 0x7fdc1813ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 -- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c0470 msgr2=0x7fdc180c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c0470 0x7fdc180c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 -- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdc180c26c0 msgr2=0x7fdc1813ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdc180c26c0 0x7fdc1813ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 -- 172.21.15.94:0/3273825506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1813b190 con 0x7fdc180c2aa0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc1e9ed640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc181ee540 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000efc0 tx=0x7fdc1000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc0effd640 1 -- 172.21.15.94:0/3273825506 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10018020 con 0x7fdc180c2aa0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc0effd640 1 -- 172.21.15.94:0/3273825506 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc10014480 con 0x7fdc180c2aa0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc181eea80 con 0x7fdc180c2aa0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc0effd640 1 -- 172.21.15.94:0/3273825506 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10014870 con 0x7fdc180c2aa0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.356+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc180c4e80 con 0x7fdc180c2aa0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.357+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc1804f330 con 0x7fdc180c2aa0 2024-03-06T07:23:07.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.358+0000 7fdc0effd640 1 -- 172.21.15.94:0/3273825506 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc10005d10 con 0x7fdc180c2aa0 2024-03-06T07:23:07.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.358+0000 7fdc0effd640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.358+0000 7fdc0effd640 1 -- 172.21.15.94:0/3273825506 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fdc1008ad30 con 0x7fdc180c2aa0 2024-03-06T07:23:07.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.358+0000 7fdc1e1ec640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.359+0000 7fdc1e1ec640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdc180c5fb0 tx=0x7fdc14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.362+0000 7fdc0effd640 1 -- 172.21.15.94:0/3273825506 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc10022070 con 0x7fdc180c2aa0 2024-03-06T07:23:07.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.534+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdc18145690 con 0x7fdc180c2aa0 2024-03-06T07:23:07.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.534+0000 7fdc0effd640 1 -- 172.21.15.94:0/3273825506 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v123) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdc1813b190 con 0x7fdc180c2aa0 2024-03-06T07:23:07.536 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:23:07.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdbfc06b4d0 msgr2=0x7fdbfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdc180c5fb0 tx=0x7fdc14009190 comp rx=0 tx=0).stop 2024-03-06T07:23:07.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 msgr2=0x7fdc181ee540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc181ee540 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000efc0 tx=0x7fdc1000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:07.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 shutdown_connections 2024-03-06T07:23:07.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdbfc06b4d0 0x7fdbfc06d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdc180c2aa0 0x7fdc181ee540 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdc180c26c0 0x7fdc1813ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 --2- 172.21.15.94:0/3273825506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdc180c0470 0x7fdc180c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 >> 172.21.15.94:0/3273825506 conn(0x7fdc180c3b40 msgr2=0x7fdc1814c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 shutdown_connections 2024-03-06T07:23:07.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.536+0000 7fdc20450640 1 -- 172.21.15.94:0/3273825506 wait complete. 2024-03-06T07:23:07.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:07.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.650+0000 7f9d8d33e640 1 Processor -- start 2024-03-06T07:23:07.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.650+0000 7f9d8d33e640 1 -- start start 2024-03-06T07:23:07.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.650+0000 7f9d8d33e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9d8806c1f0 0x7f9d880c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.650+0000 7f9d8d33e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806cad0 0x7f9d88154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.650+0000 7f9d8d33e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d880c2790 con 0x7f9d8814ef70 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.650+0000 7f9d8d33e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d880c3090 con 0x7f9d8806c1f0 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.650+0000 7f9d8d33e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8814f350 con 0x7f9d8806cad0 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d86ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9d8814ef70 0x7f9d880c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49298/0 (socket says 172.21.15.94:49298) 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d86ffd640 1 -- 172.21.15.94:0/3216540382 learned_addr learned my addr 172.21.15.94:0/3216540382 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d7ffff640 1 --2- 172.21.15.94:0/3216540382 >> v2:172.21.15.145:3300/0 conn(0x7f9d8806c1f0 0x7f9d880c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d877fe640 1 --2- 172.21.15.94:0/3216540382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806cad0 0x7f9d88154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d867fc640 1 -- 172.21.15.94:0/3216540382 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235218590 0 0) 0x7f9d880c2790 con 0x7f9d8814ef70 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d867fc640 1 -- 172.21.15.94:0/3216540382 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d74003520 con 0x7f9d8814ef70 2024-03-06T07:23:07.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d877fe640 1 -- 172.21.15.94:0/3216540382 >> v2:172.21.15.145:3300/0 conn(0x7f9d8806c1f0 msgr2=0x7f9d880c1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d877fe640 1 --2- 172.21.15.94:0/3216540382 >> v2:172.21.15.145:3300/0 conn(0x7f9d8806c1f0 0x7f9d880c1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d877fe640 1 -- 172.21.15.94:0/3216540382 >> v1:172.21.15.94:6789/0 conn(0x7f9d8814ef70 legacy=0x7f9d880c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d877fe640 1 -- 172.21.15.94:0/3216540382 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d880c1540 con 0x7f9d8806cad0 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.651+0000 7f9d7ffff640 1 --2- 172.21.15.94:0/3216540382 >> v2:172.21.15.145:3300/0 conn(0x7f9d8806c1f0 0x7f9d880c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d877fe640 1 --2- 172.21.15.94:0/3216540382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806cad0 0x7f9d88154c30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9d80009d90 tx=0x7f9d8002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c98768e58bd5220c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d867fc640 1 -- 172.21.15.94:0/3216540382 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80039070 con 0x7f9d8806cad0 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d867fc640 1 -- 172.21.15.94:0/3216540382 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8002cbc0 con 0x7f9d8806cad0 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d867fc640 1 -- 172.21.15.94:0/3216540382 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80034440 con 0x7f9d8806cad0 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3216540382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806cad0 msgr2=0x7f9d88154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3216540382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806cad0 0x7f9d88154c30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9d80009d90 tx=0x7f9d8002be10 comp rx=0 tx=0).stop 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3216540382 shutdown_connections 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3216540382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806cad0 0x7f9d88154c30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3216540382 >> v2:172.21.15.145:3300/0 conn(0x7f9d8806c1f0 0x7f9d880c1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.652+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3216540382 >> 172.21.15.94:0/3216540382 conn(0x7f9d880c6710 msgr2=0x7f9d880c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.653+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3216540382 shutdown_connections 2024-03-06T07:23:07.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.653+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3216540382 wait complete. 2024-03-06T07:23:07.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 Processor -- start 2024-03-06T07:23:07.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 -- start start 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806c1f0 0x7f9d88146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d8806cad0 0x7f9d88144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d8814ef70 0x7f9d88142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88155750 con 0x7f9d8814ef70 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8806cf60 con 0x7f9d8806cad0 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d8d33e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88155170 con 0x7f9d8806c1f0 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d8814ef70 0x7f9d88142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d877fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d8814ef70 0x7f9d88142430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47658/0 (socket says 172.21.15.94:47658) 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d877fe640 1 -- 172.21.15.94:0/3065810553 learned_addr learned my addr 172.21.15.94:0/3065810553 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d86ffd640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806c1f0 0x7f9d88146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.654+0000 7f9d7ffff640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d8806cad0 0x7f9d88144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d877fe640 1 -- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806c1f0 msgr2=0x7f9d88146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d877fe640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806c1f0 0x7f9d88146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d877fe640 1 -- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d8806cad0 msgr2=0x7f9d88144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d877fe640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d8806cad0 0x7f9d88144710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d877fe640 1 -- 172.21.15.94:0/3065810553 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8815c480 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d877fe640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d8814ef70 0x7f9d88142430 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9d80009d90 tx=0x7f9d80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d7f7fe640 1 -- 172.21.15.94:0/3065810553 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80044020 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d7f7fe640 1 -- 172.21.15.94:0/3065810553 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8003f470 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8815ca00 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d7f7fe640 1 -- 172.21.15.94:0/3065810553 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80039040 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.655+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8815c720 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.656+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d880c1e90 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.656+0000 7f9d7f7fe640 1 -- 172.21.15.94:0/3065810553 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d80047020 con 0x7f9d8814ef70 2024-03-06T07:23:07.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.657+0000 7f9d7f7fe640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d6806b4d0 0x7f9d6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.657+0000 7f9d7f7fe640 1 -- 172.21.15.94:0/3065810553 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f9d800acc40 con 0x7f9d8814ef70 2024-03-06T07:23:07.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.657+0000 7f9d86ffd640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d6806b4d0 0x7f9d6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.657+0000 7f9d86ffd640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d6806b4d0 0x7f9d6806d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005880 tx=0x7f9d78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.661+0000 7f9d7f7fe640 1 -- 172.21.15.94:0/3065810553 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d880c1e90 con 0x7f9d8814ef70 2024-03-06T07:23:07.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.829+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9d8804f330 con 0x7f9d8814ef70 2024-03-06T07:23:07.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.829+0000 7f9d7f7fe640 1 -- 172.21.15.94:0/3065810553 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v123) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9d8004e030 con 0x7f9d8814ef70 2024-03-06T07:23:07.831 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:23:07.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d6806b4d0 msgr2=0x7f9d6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d6806b4d0 0x7f9d6806d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005880 tx=0x7f9d78009040 comp rx=0 tx=0).stop 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d8814ef70 msgr2=0x7f9d88142430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d8814ef70 0x7f9d88142430 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9d80009d90 tx=0x7f9d80004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 shutdown_connections 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d6806b4d0 0x7f9d6806d990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d8814ef70 0x7f9d88142430 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d8806cad0 0x7f9d88144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 --2- 172.21.15.94:0/3065810553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d8806c1f0 0x7f9d88146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 >> 172.21.15.94:0/3065810553 conn(0x7f9d880c6710 msgr2=0x7f9d8815b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 shutdown_connections 2024-03-06T07:23:07.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.831+0000 7f9d8d33e640 1 -- 172.21.15.94:0/3065810553 wait complete. 2024-03-06T07:23:07.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:07.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.943+0000 7fc631b16640 1 Processor -- start 2024-03-06T07:23:07.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.943+0000 7fc631b16640 1 -- start start 2024-03-06T07:23:07.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc631b16640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c1550c0 0x7fc62c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc631b16640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc62c1559e0 0x7fc62c14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc631b16640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62c0aa660 con 0x7fc62c0aa180 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc631b16640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc62c0c4610 con 0x7fc62c1559e0 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc631b16640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62c0aa940 con 0x7fc62c1550c0 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc623fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc62c1559e0 0x7fc62c14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc623fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc62c1559e0 0x7fc62c14ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47664/0 (socket says 172.21.15.94:47664) 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc623fff640 1 -- 172.21.15.94:0/2991686737 learned_addr learned my addr 172.21.15.94:0/2991686737 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc62bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc62c0aa180 0x7fc62c0c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49306/0 (socket says 172.21.15.94:49306) 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc62b7fe640 1 --2- 172.21.15.94:0/2991686737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c1550c0 0x7fc62c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc623fff640 1 -- 172.21.15.94:0/2991686737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c1550c0 msgr2=0x7fc62c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc623fff640 1 --2- 172.21.15.94:0/2991686737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c1550c0 0x7fc62c1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc623fff640 1 -- 172.21.15.94:0/2991686737 >> v1:172.21.15.94:6789/0 conn(0x7fc62c0aa180 legacy=0x7fc62c0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc623fff640 1 -- 172.21.15.94:0/2991686737 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c0c4860 con 0x7fc62c1559e0 2024-03-06T07:23:07.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.944+0000 7fc62affd640 1 -- 172.21.15.94:0/2991686737 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564629404 0 0) 0x7fc62c0aa660 con 0x7fc62c0aa180 2024-03-06T07:23:07.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.945+0000 7fc623fff640 1 --2- 172.21.15.94:0/2991686737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1559e0 0x7fc62c14ec00 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc61402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83916e06735973e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.945+0000 7fc62affd640 1 -- 172.21.15.94:0/2991686737 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61402e070 con 0x7fc62c1559e0 2024-03-06T07:23:07.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.945+0000 7fc62affd640 1 -- 172.21.15.94:0/2991686737 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc614037070 con 0x7fc62c1559e0 2024-03-06T07:23:07.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.945+0000 7fc62affd640 1 -- 172.21.15.94:0/2991686737 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614031a00 con 0x7fc62c1559e0 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.945+0000 7fc631b16640 1 -- 172.21.15.94:0/2991686737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1559e0 msgr2=0x7fc62c14ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.945+0000 7fc631b16640 1 --2- 172.21.15.94:0/2991686737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1559e0 0x7fc62c14ec00 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc61402b880 comp rx=0 tx=0).stop 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.946+0000 7fc631b16640 1 -- 172.21.15.94:0/2991686737 shutdown_connections 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.946+0000 7fc631b16640 1 --2- 172.21.15.94:0/2991686737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1559e0 0x7fc62c14ec00 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.946+0000 7fc631b16640 1 --2- 172.21.15.94:0/2991686737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c1550c0 0x7fc62c1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.946+0000 7fc631b16640 1 -- 172.21.15.94:0/2991686737 >> 172.21.15.94:0/2991686737 conn(0x7fc62c0c6710 msgr2=0x7fc62c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.946+0000 7fc631b16640 1 -- 172.21.15.94:0/2991686737 shutdown_connections 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.946+0000 7fc631b16640 1 -- 172.21.15.94:0/2991686737 wait complete. 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 Processor -- start 2024-03-06T07:23:07.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 -- start start 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c0aa180 0x7fc62c149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1550c0 0x7fc62c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62c1559e0 0x7fc62c06ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62c14f140 con 0x7fc62c1550c0 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62c0c1520 con 0x7fc62c1559e0 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc631b16640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62c14f2c0 con 0x7fc62c0aa180 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc623fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1550c0 0x7fc62c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc623fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1550c0 0x7fc62c06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47670/0 (socket says 172.21.15.94:47670) 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc623fff640 1 -- 172.21.15.94:0/657735563 learned_addr learned my addr 172.21.15.94:0/657735563 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc62b7fe640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c0aa180 0x7fc62c149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.947+0000 7fc62bfff640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62c1559e0 0x7fc62c06ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc623fff640 1 -- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c0aa180 msgr2=0x7fc62c149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc623fff640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c0aa180 0x7fc62c149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc623fff640 1 -- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62c1559e0 msgr2=0x7fc62c06ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc623fff640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62c1559e0 0x7fc62c06ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc623fff640 1 -- 172.21.15.94:0/657735563 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c1e6930 con 0x7fc62c1550c0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc62bfff640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62c1559e0 0x7fc62c06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc623fff640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1550c0 0x7fc62c06c910 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc6140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc628ff9640 1 -- 172.21.15.94:0/657735563 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614032670 con 0x7fc62c1550c0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc628ff9640 1 -- 172.21.15.94:0/657735563 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc614032830 con 0x7fc62c1550c0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62c1e6c10 con 0x7fc62c1550c0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc628ff9640 1 -- 172.21.15.94:0/657735563 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614032c20 con 0x7fc62c1550c0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.948+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62c1e71d0 con 0x7fc62c1550c0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.949+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62c04f330 con 0x7fc62c1550c0 2024-03-06T07:23:07.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.949+0000 7fc628ff9640 1 -- 172.21.15.94:0/657735563 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc61404c060 con 0x7fc62c1550c0 2024-03-06T07:23:07.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.950+0000 7fc628ff9640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60c06b4d0 0x7fc60c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.950+0000 7fc628ff9640 1 -- 172.21.15.94:0/657735563 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fc6140abce0 con 0x7fc62c1550c0 2024-03-06T07:23:07.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.950+0000 7fc62b7fe640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60c06b4d0 0x7fc60c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.950+0000 7fc62b7fe640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60c06b4d0 0x7fc60c06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009b40 tx=0x7fc61c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:07.954+0000 7fc628ff9640 1 -- 172.21.15.94:0/657735563 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc614040070 con 0x7fc62c1550c0 2024-03-06T07:23:08.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.161+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc62c0aa660 con 0x7fc62c1550c0 2024-03-06T07:23:08.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.162+0000 7fc628ff9640 1 -- 172.21.15.94:0/657735563 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4992 (secure 0 0 0) 0x7fc61407a9b0 con 0x7fc62c1550c0 2024-03-06T07:23:08.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:08.164 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":113,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":8112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1268765110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1268765110},{"type":"v1","addr":"172.21.15.94:6834","nonce":1268765110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:05.210092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7540},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7540":{"gid":7540,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6836/1862149336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1862149336},{"type":"v1","addr":"172.21.15.145:6836","nonce":1862149336}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7540,"qdb_cluster":[7540]},"id":9}]} 2024-03-06T07:23:08.164 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 113 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.163+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60c06b4d0 msgr2=0x7fc60c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.163+0000 7fc631b16640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60c06b4d0 0x7fc60c06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009b40 tx=0x7fc61c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.163+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1550c0 msgr2=0x7fc62c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.163+0000 7fc631b16640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1550c0 0x7fc62c06c910 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fc614009870 tx=0x7fc6140046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 shutdown_connections 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60c06b4d0 0x7fc60c06d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62c1559e0 0x7fc62c06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62c1550c0 0x7fc62c06c910 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 --2- 172.21.15.94:0/657735563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc62c0aa180 0x7fc62c149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 >> 172.21.15.94:0/657735563 conn(0x7fc62c0c6710 msgr2=0x7fc62c153c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 shutdown_connections 2024-03-06T07:23:08.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.164+0000 7fc631b16640 1 -- 172.21.15.94:0/657735563 wait complete. 2024-03-06T07:23:08.177 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:08.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.275+0000 7fa846b5c640 1 Processor -- start 2024-03-06T07:23:08.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.275+0000 7fa846b5c640 1 -- start start 2024-03-06T07:23:08.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa846b5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa84014f360 0x7fa840155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa846b5c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8400c88c0 0x7fa8400c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa846b5c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8400c73d0 con 0x7fa8400c5880 2024-03-06T07:23:08.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa846b5c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8400c8dd0 con 0x7fa8400c88c0 2024-03-06T07:23:08.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa846b5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8400be2e0 con 0x7fa84014f360 2024-03-06T07:23:08.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa83ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8400c88c0 0x7fa8400c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa83ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8400c88c0 0x7fa8400c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47674/0 (socket says 172.21.15.94:47674) 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa83ffff640 1 -- 172.21.15.94:0/2710451428 learned_addr learned my addr 172.21.15.94:0/2710451428 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.276+0000 7fa8448f8640 1 --2- 172.21.15.94:0/2710451428 >> v2:172.21.15.145:3300/0 conn(0x7fa84014f360 0x7fa840155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.277+0000 7fa83ffff640 1 -- 172.21.15.94:0/2710451428 >> v2:172.21.15.145:3300/0 conn(0x7fa84014f360 msgr2=0x7fa840155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.277+0000 7fa83ffff640 1 --2- 172.21.15.94:0/2710451428 >> v2:172.21.15.145:3300/0 conn(0x7fa84014f360 0x7fa840155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.277+0000 7fa83ffff640 1 -- 172.21.15.94:0/2710451428 >> v1:172.21.15.145:6789/0 conn(0x7fa8400c5880 legacy=0x7fa840158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.277+0000 7fa83ffff640 1 -- 172.21.15.94:0/2710451428 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8400c7670 con 0x7fa8400c88c0 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.277+0000 7fa83ffff640 1 --2- 172.21.15.94:0/2710451428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8400c88c0 0x7fa8400c5340 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fa84008c840 tx=0x7fa83402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3694e1c776931574 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.277+0000 7fa83f7fe640 1 -- 172.21.15.94:0/2710451428 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834035020 con 0x7fa8400c88c0 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa83f7fe640 1 -- 172.21.15.94:0/2710451428 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834031870 con 0x7fa8400c88c0 2024-03-06T07:23:08.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa83f7fe640 1 -- 172.21.15.94:0/2710451428 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834031c40 con 0x7fa8400c88c0 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 -- 172.21.15.94:0/2710451428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8400c88c0 msgr2=0x7fa8400c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2710451428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8400c88c0 0x7fa8400c5340 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fa84008c840 tx=0x7fa83402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 -- 172.21.15.94:0/2710451428 shutdown_connections 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2710451428 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8400c88c0 0x7fa8400c5340 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2710451428 >> v2:172.21.15.145:3300/0 conn(0x7fa84014f360 0x7fa840155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 -- 172.21.15.94:0/2710451428 >> 172.21.15.94:0/2710451428 conn(0x7fa8400ac570 msgr2=0x7fa8400ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 -- 172.21.15.94:0/2710451428 shutdown_connections 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.278+0000 7fa846b5c640 1 -- 172.21.15.94:0/2710451428 wait complete. 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.279+0000 7fa846b5c640 1 Processor -- start 2024-03-06T07:23:08.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.279+0000 7fa846b5c640 1 -- start start 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa846b5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8400c5880 0x7fa84014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa846b5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8400c88c0 0x7fa84014bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa846b5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa84014f360 0x7fa8401ee070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa846b5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa840159270 con 0x7fa84014f360 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa846b5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8400c7c30 con 0x7fa8400c5880 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa846b5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8401593f0 con 0x7fa8400c88c0 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa84014f360 0x7fa8401ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa84014f360 0x7fa8401ee070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47678/0 (socket says 172.21.15.94:47678) 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 -- 172.21.15.94:0/2130034677 learned_addr learned my addr 172.21.15.94:0/2130034677 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa83ffff640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8400c88c0 0x7fa84014bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8448f8640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8400c5880 0x7fa84014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 -- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8400c88c0 msgr2=0x7fa84014bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8400c88c0 0x7fa84014bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 -- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8400c5880 msgr2=0x7fa84014b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8400c5880 0x7fa84014b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.280+0000 7fa8450f9640 1 -- 172.21.15.94:0/2130034677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8400c1470 con 0x7fa84014f360 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.281+0000 7fa8450f9640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa84014f360 0x7fa8401ee070 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fa83800c7a0 tx=0x7fa83800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.281+0000 7fa83d7fa640 1 -- 172.21.15.94:0/2130034677 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838003b70 con 0x7fa84014f360 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.281+0000 7fa83d7fa640 1 -- 172.21.15.94:0/2130034677 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa838003d30 con 0x7fa84014f360 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.281+0000 7fa83d7fa640 1 -- 172.21.15.94:0/2130034677 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83800aa90 con 0x7fa84014f360 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.281+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8400af750 con 0x7fa84014f360 2024-03-06T07:23:08.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.281+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8401eecd0 con 0x7fa84014f360 2024-03-06T07:23:08.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.282+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8400afa40 con 0x7fa84014f360 2024-03-06T07:23:08.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.282+0000 7fa83d7fa640 1 -- 172.21.15.94:0/2130034677 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa83800ac30 con 0x7fa84014f360 2024-03-06T07:23:08.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.282+0000 7fa83d7fa640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa82c06b4d0 0x7fa82c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.283+0000 7fa83d7fa640 1 -- 172.21.15.94:0/2130034677 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa83808a1b0 con 0x7fa84014f360 2024-03-06T07:23:08.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.283+0000 7fa8448f8640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa82c06b4d0 0x7fa82c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.283+0000 7fa8448f8640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa82c06b4d0 0x7fa82c06d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa830009aa0 tx=0x7fa830009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.287+0000 7fa83d7fa640 1 -- 172.21.15.94:0/2130034677 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa838058830 con 0x7fa84014f360 2024-03-06T07:23:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.457+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa84004f330 con 0x7fa84014f360 2024-03-06T07:23:08.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.458+0000 7fa83d7fa640 1 -- 172.21.15.94:0/2130034677 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+17157 (secure 0 0 0) 0x7fa83805c8d0 con 0x7fa84014f360 2024-03-06T07:23:08.460 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:08.460 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":123,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:02.205043+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:40.848613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:41.593837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.94:6832/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.145:6836/198029515":"2024-03-07T07:23:00.021121+0000","172.21.15.145:6834/198029515":"2024-03-07T07:23:00.021121+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:08.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa82c06b4d0 msgr2=0x7fa82c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa82c06b4d0 0x7fa82c06d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa830009aa0 tx=0x7fa830009190 comp rx=0 tx=0).stop 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa84014f360 msgr2=0x7fa8401ee070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa84014f360 0x7fa8401ee070 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fa83800c7a0 tx=0x7fa83800cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 shutdown_connections 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa82c06b4d0 0x7fa82c06d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa84014f360 0x7fa8401ee070 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8400c88c0 0x7fa84014bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 --2- 172.21.15.94:0/2130034677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8400c5880 0x7fa84014b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.460+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 >> 172.21.15.94:0/2130034677 conn(0x7fa8400ac570 msgr2=0x7fa8400c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.461+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 shutdown_connections 2024-03-06T07:23:08.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.461+0000 7fa846b5c640 1 -- 172.21.15.94:0/2130034677 wait complete. 2024-03-06T07:23:08.473 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:08.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:08.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.572+0000 7f87f64c9640 1 Processor -- start 2024-03-06T07:23:08.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.572+0000 7f87f64c9640 1 -- start start 2024-03-06T07:23:08.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.572+0000 7f87f64c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87f00c1aa0 0x7f87f01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.572+0000 7f87f64c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f0155a40 0x7f87f0155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.572+0000 7f87f64c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f01563d0 con 0x7f87f014f790 2024-03-06T07:23:08.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.572+0000 7f87f64c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f00c2680 con 0x7f87f00c1aa0 2024-03-06T07:23:08.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87f64c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00c28d0 con 0x7f87f0155a40 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87ef7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87f00c1aa0 0x7f87f01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87ef7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87f00c1aa0 0x7f87f01529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39706/0 (socket says 172.21.15.94:39706) 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87ef7fe640 1 -- 172.21.15.94:0/3053562698 learned_addr learned my addr 172.21.15.94:0/3053562698 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87f4a66640 1 --2- 172.21.15.94:0/3053562698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f0155a40 0x7f87f0155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87ef7fe640 1 -- 172.21.15.94:0/3053562698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f0155a40 msgr2=0x7f87f0155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87ef7fe640 1 --2- 172.21.15.94:0/3053562698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f0155a40 0x7f87f0155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87ef7fe640 1 -- 172.21.15.94:0/3053562698 >> v1:172.21.15.145:6789/0 conn(0x7f87f014f790 legacy=0x7f87f00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.573+0000 7f87ef7fe640 1 -- 172.21.15.94:0/3053562698 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00c2bb0 con 0x7f87f00c1aa0 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87ef7fe640 1 --2- 172.21.15.94:0/3053562698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f00c1aa0 0x7f87f01529d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f87e400bb10 tx=0x7f87e402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b007460a9af46049 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87eeffd640 1 -- 172.21.15.94:0/3053562698 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e403b070 con 0x7f87f00c1aa0 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87eeffd640 1 -- 172.21.15.94:0/3053562698 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e4036440 con 0x7f87f00c1aa0 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87eeffd640 1 -- 172.21.15.94:0/3053562698 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4036810 con 0x7f87f00c1aa0 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 -- 172.21.15.94:0/3053562698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f00c1aa0 msgr2=0x7f87f01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 --2- 172.21.15.94:0/3053562698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f00c1aa0 0x7f87f01529d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f87e400bb10 tx=0x7f87e402e150 comp rx=0 tx=0).stop 2024-03-06T07:23:08.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 -- 172.21.15.94:0/3053562698 shutdown_connections 2024-03-06T07:23:08.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 --2- 172.21.15.94:0/3053562698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f0155a40 0x7f87f0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 --2- 172.21.15.94:0/3053562698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f00c1aa0 0x7f87f01529d0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 -- 172.21.15.94:0/3053562698 >> 172.21.15.94:0/3053562698 conn(0x7f87f00c6710 msgr2=0x7f87f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 -- 172.21.15.94:0/3053562698 shutdown_connections 2024-03-06T07:23:08.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.574+0000 7f87f64c9640 1 -- 172.21.15.94:0/3053562698 wait complete. 2024-03-06T07:23:08.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.575+0000 7f87f64c9640 1 Processor -- start 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87f64c9640 1 -- start start 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87f64c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f00c1aa0 0x7f87f00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87f64c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f014f790 0x7f87f014a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87f64c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87f0155a40 0x7f87f014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87f64c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00c0d60 con 0x7f87f014f790 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87f64c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0155520 con 0x7f87f0155a40 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87f64c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00c5e30 con 0x7f87f00c1aa0 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87ef7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f014f790 0x7f87f014a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87ef7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f014f790 0x7f87f014a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39708/0 (socket says 172.21.15.94:39708) 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.576+0000 7f87ef7fe640 1 -- 172.21.15.94:0/2095488106 learned_addr learned my addr 172.21.15.94:0/2095488106 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87effff640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f00c1aa0 0x7f87f00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87f4a66640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87f0155a40 0x7f87f014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ef7fe640 1 -- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f00c1aa0 msgr2=0x7f87f00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ef7fe640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f00c1aa0 0x7f87f00712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ef7fe640 1 -- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87f0155a40 msgr2=0x7f87f014ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ef7fe640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87f0155a40 0x7f87f014ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ef7fe640 1 -- 172.21.15.94:0/2095488106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f014b710 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ef7fe640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f014f790 0x7f87f014a920 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f87e402e190 tx=0x7f87e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ecff9640 1 -- 172.21.15.94:0/2095488106 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4045030 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ecff9640 1 -- 172.21.15.94:0/2095488106 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e4034440 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f0070330 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87ecff9640 1 -- 172.21.15.94:0/2095488106 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e403b040 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.577+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f00a9290 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.578+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f004f330 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.579+0000 7f87ecff9640 1 -- 172.21.15.94:0/2095488106 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87e4048070 con 0x7f87f014f790 2024-03-06T07:23:08.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.579+0000 7f87ecff9640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.579+0000 7f87ecff9640 1 -- 172.21.15.94:0/2095488106 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f87e40ae8c0 con 0x7f87f014f790 2024-03-06T07:23:08.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.579+0000 7f87effff640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.580+0000 7f87effff640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87dc06b4d0 0x7f87dc06d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f87e0001780 tx=0x7f87e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.583+0000 7f87ecff9640 1 -- 172.21.15.94:0/2095488106 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87e404f070 con 0x7f87f014f790 2024-03-06T07:23:08.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.792+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f00c1f40 con 0x7f87f014f790 2024-03-06T07:23:08.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.793+0000 7f87ecff9640 1 -- 172.21.15.94:0/2095488106 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4992 (secure 0 0 0) 0x7f87e407cf40 con 0x7f87f014f790 2024-03-06T07:23:08.795 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:08.795 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":113,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":8112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1268765110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1268765110},{"type":"v1","addr":"172.21.15.94:6834","nonce":1268765110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:05.210092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7540},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7540":{"gid":7540,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6836/1862149336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1862149336},{"type":"v1","addr":"172.21.15.145:6836","nonce":1862149336}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7540,"qdb_cluster":[7540]},"id":9}]} 2024-03-06T07:23:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 113 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87dc06b4d0 msgr2=0x7f87dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87dc06b4d0 0x7f87dc06d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f87e0001780 tx=0x7f87e0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f014f790 msgr2=0x7f87f014a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f014f790 0x7f87f014a920 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f87e402e190 tx=0x7f87e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 shutdown_connections 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87f0155a40 0x7f87f014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87f014f790 0x7f87f014a920 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 --2- 172.21.15.94:0/2095488106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87f00c1aa0 0x7f87f00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 >> 172.21.15.94:0/2095488106 conn(0x7f87f00c6710 msgr2=0x7f87f00a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 shutdown_connections 2024-03-06T07:23:08.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.795+0000 7f87f64c9640 1 -- 172.21.15.94:0/2095488106 wait complete. 2024-03-06T07:23:08.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:08.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.906+0000 7f7777c88640 1 Processor -- start 2024-03-06T07:23:08.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.906+0000 7f7777c88640 1 -- start start 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7777c88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77701550c0 0x7f77701554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7777c88640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77700aa660 con 0x7f77701559e0 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7777c88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77700aa940 con 0x7f77700aa180 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7777c88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77700c4610 con 0x7f77701550c0 2024-03-06T07:23:08.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7775223640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f77701559e0 0x7f777014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47226/0 (socket says 172.21.15.94:47226) 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7775223640 1 -- 172.21.15.94:0/3008614897 learned_addr learned my addr 172.21.15.94:0/3008614897 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7775a24640 1 --2- 172.21.15.94:0/3008614897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77701550c0 0x7f77701554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7774a22640 1 -- 172.21.15.94:0/3008614897 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775097429 0 0) 0x7f77700aa660 con 0x7f77701559e0 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7774a22640 1 -- 172.21.15.94:0/3008614897 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775c003440 con 0x7f77701559e0 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7774a22640 1 -- 172.21.15.94:0/3008614897 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091243607 0 0) 0x7f77700aa940 con 0x7f77700aa180 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7775a24640 1 -- 172.21.15.94:0/3008614897 >> v1:172.21.15.145:6789/0 conn(0x7f77700aa180 legacy=0x7f77700c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7775a24640 1 -- 172.21.15.94:0/3008614897 >> v1:172.21.15.94:6789/0 conn(0x7f77701559e0 legacy=0x7f777014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.907+0000 7f7775a24640 1 -- 172.21.15.94:0/3008614897 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77700c4860 con 0x7f77701550c0 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.908+0000 7f7775a24640 1 --2- 172.21.15.94:0/3008614897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77701550c0 0x7f77701554a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f776c0097d0 tx=0x7f776c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221764dbd361ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.908+0000 7f7774a22640 1 -- 172.21.15.94:0/3008614897 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0315a0 con 0x7f77701550c0 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.908+0000 7f7774a22640 1 -- 172.21.15.94:0/3008614897 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c037070 con 0x7f77701550c0 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.908+0000 7f7774a22640 1 -- 172.21.15.94:0/3008614897 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c031960 con 0x7f77701550c0 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.909+0000 7f7777c88640 1 -- 172.21.15.94:0/3008614897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77701550c0 msgr2=0x7f77701554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.909+0000 7f7777c88640 1 --2- 172.21.15.94:0/3008614897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77701550c0 0x7f77701554a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f776c0097d0 tx=0x7f776c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.909+0000 7f7777c88640 1 -- 172.21.15.94:0/3008614897 shutdown_connections 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.909+0000 7f7777c88640 1 --2- 172.21.15.94:0/3008614897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77701550c0 0x7f77701554a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.909+0000 7f7777c88640 1 -- 172.21.15.94:0/3008614897 >> 172.21.15.94:0/3008614897 conn(0x7f77700c6710 msgr2=0x7f77700c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.909+0000 7f7777c88640 1 -- 172.21.15.94:0/3008614897 shutdown_connections 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.909+0000 7f7777c88640 1 -- 172.21.15.94:0/3008614897 wait complete. 2024-03-06T07:23:08.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.910+0000 7f7777c88640 1 Processor -- start 2024-03-06T07:23:08.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.910+0000 7f7777c88640 1 -- start start 2024-03-06T07:23:08.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77700aa180 0x7f777014a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77701550c0 0x7f77701e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77701559e0 0x7f77701449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777014f310 con 0x7f77701550c0 2024-03-06T07:23:08.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77700c1520 con 0x7f77701559e0 2024-03-06T07:23:08.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777014f490 con 0x7f77700aa180 2024-03-06T07:23:08.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77701550c0 0x7f77701e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77701550c0 0x7f77701e9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39718/0 (socket says 172.21.15.94:39718) 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 -- 172.21.15.94:0/637379952 learned_addr learned my addr 172.21.15.94:0/637379952 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775a24640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77700aa180 0x7f777014a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7776225640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77701559e0 0x7f77701449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 -- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77700aa180 msgr2=0x7f777014a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77700aa180 0x7f777014a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 -- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77701559e0 msgr2=0x7f77701449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77701559e0 0x7f77701449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 -- 172.21.15.94:0/637379952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77700a6bc0 con 0x7f77701550c0 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775a24640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77700aa180 0x7f777014a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7775223640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77701550c0 0x7f77701e9e50 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f776000c390 tx=0x7f776000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f77667fc640 1 -- 172.21.15.94:0/637379952 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7760017070 con 0x7f77701550c0 2024-03-06T07:23:08.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f77667fc640 1 -- 172.21.15.94:0/637379952 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760005ce0 con 0x7f77701550c0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77700a6920 con 0x7f77701550c0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f77667fc640 1 -- 172.21.15.94:0/637379952 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77600072f0 con 0x7f77701550c0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.911+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77701e9930 con 0x7f77701550c0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.913+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777004f330 con 0x7f77701550c0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.913+0000 7f77667fc640 1 -- 172.21.15.94:0/637379952 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f776001a070 con 0x7f77701550c0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.913+0000 7f77667fc640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f775406b4d0 0x7f775406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.913+0000 7f77667fc640 1 -- 172.21.15.94:0/637379952 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f77600897b0 con 0x7f77701550c0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.913+0000 7f7775a24640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f775406b4d0 0x7f775406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.914+0000 7f7775a24640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f775406b4d0 0x7f775406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f776c036000 tx=0x7f776c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:08.917+0000 7f77667fc640 1 -- 172.21.15.94:0/637379952 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7760057da0 con 0x7f77701550c0 2024-03-06T07:23:09.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.122+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7770068450 con 0x7f77701550c0 2024-03-06T07:23:09.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.123+0000 7f77667fc640 1 -- 172.21.15.94:0/637379952 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4992 (secure 0 0 0) 0x7f776005be40 con 0x7f77701550c0 2024-03-06T07:23:09.125 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:09.126 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":113,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":8112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1268765110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1268765110},{"type":"v1","addr":"172.21.15.94:6834","nonce":1268765110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:42.624925+0000","modified":"2024-03-06T07:23:05.210092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7540},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7540":{"gid":7540,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6836/1862149336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1862149336},{"type":"v1","addr":"172.21.15.145:6836","nonce":1862149336}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7540,"qdb_cluster":[7540]},"id":9}]} 2024-03-06T07:23:09.128 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 113 2024-03-06T07:23:09.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f775406b4d0 msgr2=0x7f775406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f775406b4d0 0x7f775406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f776c036000 tx=0x7f776c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:09.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77701550c0 msgr2=0x7f77701e9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77701550c0 0x7f77701e9e50 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f776000c390 tx=0x7f776000c850 comp rx=0 tx=0).stop 2024-03-06T07:23:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 shutdown_connections 2024-03-06T07:23:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f775406b4d0 0x7f775406d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77701559e0 0x7f77701449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77701550c0 0x7f77701e9e50 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 --2- 172.21.15.94:0/637379952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77700aa180 0x7f777014a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 >> 172.21.15.94:0/637379952 conn(0x7f77700c6710 msgr2=0x7f77700c16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 shutdown_connections 2024-03-06T07:23:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.125+0000 7f7777c88640 1 -- 172.21.15.94:0/637379952 wait complete. 2024-03-06T07:23:09.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:09.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.238+0000 7f6ab675e640 1 Processor -- start 2024-03-06T07:23:09.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.238+0000 7f6ab675e640 1 -- start start 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6ab675e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6ab00c3750 0x7f6ab00c1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6ab675e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00a5e60 0x7f6ab00c1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6ab675e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab00c5490 con 0x7f6ab00c3370 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6ab675e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab00c5770 con 0x7f6ab00c3750 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6ab675e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00c5a50 con 0x7f6ab00a5e60 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6aaffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6ab00c3370 0x7f6ab0155ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47230/0 (socket says 172.21.15.94:47230) 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6aaffff640 1 -- 172.21.15.94:0/2554583134 learned_addr learned my addr 172.21.15.94:0/2554583134 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6aaf7fe640 1 --2- 172.21.15.94:0/2554583134 >> v2:172.21.15.145:3300/0 conn(0x7f6ab00c3750 0x7f6ab00c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6ab4cfb640 1 --2- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00a5e60 0x7f6ab00c1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6aaeffd640 1 -- 172.21.15.94:0/2554583134 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420198262 0 0) 0x7f6ab00c5490 con 0x7f6ab00c3370 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.239+0000 7f6aaeffd640 1 -- 172.21.15.94:0/2554583134 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a98003520 con 0x7f6ab00c3370 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.240+0000 7f6aaf7fe640 1 -- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00a5e60 msgr2=0x7f6ab00c1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.240+0000 7f6aaf7fe640 1 --2- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00a5e60 0x7f6ab00c1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.240+0000 7f6aaf7fe640 1 -- 172.21.15.94:0/2554583134 >> v1:172.21.15.94:6789/0 conn(0x7f6ab00c3370 legacy=0x7f6ab0155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.240+0000 7f6aaf7fe640 1 -- 172.21.15.94:0/2554583134 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab00c5d30 con 0x7f6ab00c3750 2024-03-06T07:23:09.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.240+0000 7f6ab4cfb640 1 --2- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00a5e60 0x7f6ab00c1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:09.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.240+0000 7f6aaf7fe640 1 --2- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00c3750 0x7f6ab00c1110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40097a0 tx=0x7f6aa402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0743f367df8bda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.240+0000 7f6aaeffd640 1 -- 172.21.15.94:0/2554583134 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4038070 con 0x7f6ab00c3750 2024-03-06T07:23:09.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6aaeffd640 1 -- 172.21.15.94:0/2554583134 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa4031700 con 0x7f6ab00c3750 2024-03-06T07:23:09.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6aaeffd640 1 -- 172.21.15.94:0/2554583134 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4031ab0 con 0x7f6ab00c3750 2024-03-06T07:23:09.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 -- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00c3750 msgr2=0x7f6ab00c1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 --2- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00c3750 0x7f6ab00c1110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40097a0 tx=0x7f6aa402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 -- 172.21.15.94:0/2554583134 shutdown_connections 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 --2- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00a5e60 0x7f6ab00c1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 --2- 172.21.15.94:0/2554583134 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00c3750 0x7f6ab00c1110 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 -- 172.21.15.94:0/2554583134 >> 172.21.15.94:0/2554583134 conn(0x7f6ab00c6710 msgr2=0x7f6ab00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 -- 172.21.15.94:0/2554583134 shutdown_connections 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.241+0000 7f6ab675e640 1 -- 172.21.15.94:0/2554583134 wait complete. 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.242+0000 7f6ab675e640 1 Processor -- start 2024-03-06T07:23:09.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.242+0000 7f6ab675e640 1 -- start start 2024-03-06T07:23:09.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab675e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00a5e60 0x7f6ab01eb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab675e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00c3370 0x7f6ab0072c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab675e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ab00c3750 0x7f6ab0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab675e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0159910 con 0x7f6ab00c3750 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab675e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0159790 con 0x7f6ab00a5e60 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab675e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00c1b90 con 0x7f6ab00c3370 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ab00c3750 0x7f6ab0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ab00c3750 0x7f6ab0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39726/0 (socket says 172.21.15.94:39726) 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 -- 172.21.15.94:0/3663581142 learned_addr learned my addr 172.21.15.94:0/3663581142 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6aaffff640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00a5e60 0x7f6ab01eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6aaf7fe640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00c3370 0x7f6ab0072c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 -- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00c3370 msgr2=0x7f6ab0072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00c3370 0x7f6ab0072c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 -- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00a5e60 msgr2=0x7f6ab01eb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00a5e60 0x7f6ab01eb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 -- 172.21.15.94:0/3663581142 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0070cb0 con 0x7f6ab00c3750 2024-03-06T07:23:09.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.243+0000 7f6ab4cfb640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ab00c3750 0x7f6ab0073140 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800efc0 tx=0x7f6aa800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.244+0000 7f6aacff9640 1 -- 172.21.15.94:0/3663581142 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8018030 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.244+0000 7f6aacff9640 1 -- 172.21.15.94:0/3663581142 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa800cdf0 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.244+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab0072270 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.244+0000 7f6aacff9640 1 -- 172.21.15.94:0/3663581142 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8003bb0 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.244+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0070820 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.245+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab00c2be0 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.245+0000 7f6aacff9640 1 -- 172.21.15.94:0/3663581142 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6aa8005d10 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.246+0000 7f6aacff9640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.246+0000 7f6aacff9640 1 -- 172.21.15.94:0/3663581142 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f6aa808abd0 con 0x7f6ab00c3750 2024-03-06T07:23:09.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.246+0000 7f6aaffff640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.246+0000 7f6aaffff640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0005880 tx=0x7f6aa0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.250+0000 7f6aacff9640 1 -- 172.21.15.94:0/3663581142 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ab00c2be0 con 0x7f6ab00c3750 2024-03-06T07:23:09.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.421+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ab004f330 con 0x7f6ab00c3750 2024-03-06T07:23:09.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.422+0000 7f6aacff9640 1 -- 172.21.15.94:0/3663581142 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+17157 (secure 0 0 0) 0x7f6aa8022070 con 0x7f6ab00c3750 2024-03-06T07:23:09.424 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:09.424 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":123,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:02.205043+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:40.848613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:41.593837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.94:6832/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.145:6836/198029515":"2024-03-07T07:23:00.021121+0000","172.21.15.145:6834/198029515":"2024-03-07T07:23:00.021121+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a9c06b500 msgr2=0x7f6a9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0005880 tx=0x7f6aa0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ab00c3750 msgr2=0x7f6ab0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ab00c3750 0x7f6ab0073140 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f6aa800efc0 tx=0x7f6aa800c490 comp rx=0 tx=0).stop 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 shutdown_connections 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6a9c06b500 0x7f6a9c06d9c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ab00c3750 0x7f6ab0073140 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ab00c3370 0x7f6ab0072c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 --2- 172.21.15.94:0/3663581142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ab00a5e60 0x7f6ab01eb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 >> 172.21.15.94:0/3663581142 conn(0x7f6ab00c6710 msgr2=0x7f6ab00c23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 shutdown_connections 2024-03-06T07:23:09.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.424+0000 7f6ab675e640 1 -- 172.21.15.94:0/3663581142 wait complete. 2024-03-06T07:23:09.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:09.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.538+0000 7f5e03716640 1 Processor -- start 2024-03-06T07:23:09.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.538+0000 7f5e03716640 1 -- start start 2024-03-06T07:23:09.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e03716640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5dfc072f00 0x7f5dfc153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e03716640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e03716640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5dfc0c6d80 0x7f5dfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e03716640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dfc070550 con 0x7f5dfc072f00 2024-03-06T07:23:09.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e03716640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dfc070830 con 0x7f5dfc0c6d80 2024-03-06T07:23:09.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e03716640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc070b10 con 0x7f5dfc0c0fa0 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5dfc072f00 0x7f5dfc153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5dfc072f00 0x7f5dfc153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39740/0 (socket says 172.21.15.94:39740) 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 -- 172.21.15.94:0/303354829 learned_addr learned my addr 172.21.15.94:0/303354829 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e00cb1640 1 --2- 172.21.15.94:0/303354829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e01cb3640 1 --2- 172.21.15.94:0/303354829 >> v2:172.21.15.145:3300/0 conn(0x7f5dfc0c6d80 0x7f5dfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 -- 172.21.15.94:0/303354829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c0fa0 msgr2=0x7f5dfc0c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 --2- 172.21.15.94:0/303354829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 -- 172.21.15.94:0/303354829 >> v2:172.21.15.145:3300/0 conn(0x7f5dfc0c6d80 msgr2=0x7f5dfc06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 --2- 172.21.15.94:0/303354829 >> v2:172.21.15.145:3300/0 conn(0x7f5dfc0c6d80 0x7f5dfc06d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.539+0000 7f5e014b2640 1 -- 172.21.15.94:0/303354829 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc0732e0 con 0x7f5dfc072f00 2024-03-06T07:23:09.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.540+0000 7f5e014b2640 1 --2- 172.21.15.94:0/303354829 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc153990 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0c0e50 tx=0x7f5df802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5029073e2ac8884 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.540+0000 7f5df3fff640 1 -- 172.21.15.94:0/303354829 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8035020 con 0x7f5dfc072f00 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.540+0000 7f5df3fff640 1 -- 172.21.15.94:0/303354829 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8031870 con 0x7f5dfc072f00 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.540+0000 7f5df3fff640 1 -- 172.21.15.94:0/303354829 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8031c40 con 0x7f5dfc072f00 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.540+0000 7f5e03716640 1 -- 172.21.15.94:0/303354829 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 msgr2=0x7f5dfc153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.540+0000 7f5e03716640 1 --2- 172.21.15.94:0/303354829 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc153990 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0c0e50 tx=0x7f5df802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.541+0000 7f5e03716640 1 -- 172.21.15.94:0/303354829 shutdown_connections 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.541+0000 7f5e03716640 1 --2- 172.21.15.94:0/303354829 >> v2:172.21.15.145:3300/0 conn(0x7f5dfc0c6d80 0x7f5dfc06d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.541+0000 7f5e03716640 1 --2- 172.21.15.94:0/303354829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.541+0000 7f5e03716640 1 --2- 172.21.15.94:0/303354829 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc153990 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.541+0000 7f5e03716640 1 -- 172.21.15.94:0/303354829 >> 172.21.15.94:0/303354829 conn(0x7f5dfc06b1a0 msgr2=0x7f5dfc06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.541+0000 7f5e03716640 1 -- 172.21.15.94:0/303354829 shutdown_connections 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.541+0000 7f5e03716640 1 -- 172.21.15.94:0/303354829 wait complete. 2024-03-06T07:23:09.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 Processor -- start 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 -- start start 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c6d80 0x7f5dfc151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc072580 con 0x7f5dfc072f00 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc072400 con 0x7f5dfc0c0fa0 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e03716640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc06ded0 con 0x7f5dfc0c6d80 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e014b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.542+0000 7f5e014b2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc14fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39746/0 (socket says 172.21.15.94:39746) 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e014b2640 1 -- 172.21.15.94:0/3806054596 learned_addr learned my addr 172.21.15.94:0/3806054596 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e00cb1640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e01cb3640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c6d80 0x7f5dfc151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e014b2640 1 -- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c6d80 msgr2=0x7f5dfc151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e014b2640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c6d80 0x7f5dfc151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e014b2640 1 -- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dfc0c0fa0 msgr2=0x7f5dfc0c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e014b2640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e014b2640 1 -- 172.21.15.94:0/3806054596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc1ebb40 con 0x7f5dfc072f00 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e00cb1640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e01cb3640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c6d80 0x7f5dfc151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e014b2640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc14fed0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f5df8000c00 tx=0x7f5df80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5df1ffb640 1 -- 172.21.15.94:0/3806054596 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8035040 con 0x7f5dfc072f00 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5df1ffb640 1 -- 172.21.15.94:0/3806054596 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8030d00 con 0x7f5dfc072f00 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dfc1ebe20 con 0x7f5dfc072f00 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5df1ffb640 1 -- 172.21.15.94:0/3806054596 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df80456a0 con 0x7f5dfc072f00 2024-03-06T07:23:09.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.543+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dfc1ec3e0 con 0x7f5dfc072f00 2024-03-06T07:23:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.544+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dfc04f330 con 0x7f5dfc072f00 2024-03-06T07:23:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.545+0000 7f5df1ffb640 1 -- 172.21.15.94:0/3806054596 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5df8038020 con 0x7f5dfc072f00 2024-03-06T07:23:09.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.545+0000 7f5df1ffb640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5de006b500 0x7f5de006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.545+0000 7f5df1ffb640 1 -- 172.21.15.94:0/3806054596 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f5df80ac770 con 0x7f5dfc072f00 2024-03-06T07:23:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.545+0000 7f5e00cb1640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5de006b500 0x7f5de006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.546+0000 7f5e00cb1640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5de006b500 0x7f5de006d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc153460 tx=0x7f5dec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.549+0000 7f5df1ffb640 1 -- 172.21.15.94:0/3806054596 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5df807b440 con 0x7f5dfc072f00 2024-03-06T07:23:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.759+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5dfc0abee0 con 0x7f5dfc072f00 2024-03-06T07:23:09.761 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:23:09.760+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:09.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.761+0000 7f5df1ffb640 1 -- 172.21.15.94:0/3806054596 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v114) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5df807ad40 con 0x7f5dfc072f00 2024-03-06T07:23:09.762 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:09.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5de006b500 msgr2=0x7f5de006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5de006b500 0x7f5de006d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc153460 tx=0x7f5dec008040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 msgr2=0x7f5dfc14fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc14fed0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f5df8000c00 tx=0x7f5df80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 shutdown_connections 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5de006b500 0x7f5de006d9c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dfc0c6d80 0x7f5dfc151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dfc0c0fa0 0x7f5dfc0c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 --2- 172.21.15.94:0/3806054596 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5dfc072f00 0x7f5dfc14fed0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 >> 172.21.15.94:0/3806054596 conn(0x7f5dfc06b1a0 msgr2=0x7f5dfc0c5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 shutdown_connections 2024-03-06T07:23:09.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.763+0000 7f5e03716640 1 -- 172.21.15.94:0/3806054596 wait complete. 2024-03-06T07:23:09.767 INFO:tasks.ceph.mds.b.smithi145.stderr: -159> 2024-03-06T07:23:06.032+0000 7f798e912640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:09.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:09.790 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:23:09.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.880+0000 7efe455e2640 1 Processor -- start 2024-03-06T07:23:09.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.880+0000 7efe455e2640 1 -- start start 2024-03-06T07:23:09.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe455e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe4014e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe455e2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe400b0450 con 0x7efe400b0070 2024-03-06T07:23:09.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe455e2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe401560e0 con 0x7efe40155c30 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe455e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4006c020 con 0x7efe4006ca70 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efe400b0070 0x7efe400c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47244/0 (socket says 172.21.15.94:47244) 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3effd640 1 -- 172.21.15.94:0/534944396 learned_addr learned my addr 172.21.15.94:0/534944396 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3f7fe640 1 --2- 172.21.15.94:0/534944396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe4014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3dffb640 1 -- 172.21.15.94:0/534944396 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945018910 0 0) 0x7efe30001110 con 0x7efe400b0070 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3dffb640 1 -- 172.21.15.94:0/534944396 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe28003520 con 0x7efe400b0070 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3dffb640 1 -- 172.21.15.94:0/534944396 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386334521 0 0) 0x7efe401560e0 con 0x7efe40155c30 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3f7fe640 1 -- 172.21.15.94:0/534944396 >> v1:172.21.15.145:6789/0 conn(0x7efe40155c30 legacy=0x7efe400c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3f7fe640 1 -- 172.21.15.94:0/534944396 >> v1:172.21.15.94:6789/0 conn(0x7efe400b0070 legacy=0x7efe400c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.881+0000 7efe3f7fe640 1 -- 172.21.15.94:0/534944396 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe400c77f0 con 0x7efe4006ca70 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.882+0000 7efe3f7fe640 1 --2- 172.21.15.94:0/534944396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe4014e800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe401562b0 tx=0x7efe3802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22619a8fe24ec28b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.882+0000 7efe3dffb640 1 -- 172.21.15.94:0/534944396 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38039070 con 0x7efe4006ca70 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.882+0000 7efe3dffb640 1 -- 172.21.15.94:0/534944396 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3802cc40 con 0x7efe4006ca70 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.882+0000 7efe3dffb640 1 -- 172.21.15.94:0/534944396 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38034440 con 0x7efe4006ca70 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.883+0000 7efe455e2640 1 -- 172.21.15.94:0/534944396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 msgr2=0x7efe4014e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.883+0000 7efe455e2640 1 --2- 172.21.15.94:0/534944396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe4014e800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efe401562b0 tx=0x7efe3802be90 comp rx=0 tx=0).stop 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.883+0000 7efe455e2640 1 -- 172.21.15.94:0/534944396 shutdown_connections 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.883+0000 7efe455e2640 1 --2- 172.21.15.94:0/534944396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe4014e800 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.883+0000 7efe455e2640 1 -- 172.21.15.94:0/534944396 >> 172.21.15.94:0/534944396 conn(0x7efe400ac590 msgr2=0x7efe400ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.883+0000 7efe455e2640 1 -- 172.21.15.94:0/534944396 shutdown_connections 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.883+0000 7efe455e2640 1 -- 172.21.15.94:0/534944396 wait complete. 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.884+0000 7efe455e2640 1 Processor -- start 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.884+0000 7efe455e2640 1 -- start start 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe455e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe40154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe455e2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe400b0070 0x7efe40157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe455e2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe40155c30 0x7efe4014ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe455e2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4006c550 con 0x7efe400b0070 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe455e2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4006c270 con 0x7efe40155c30 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe455e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4006da80 con 0x7efe4006ca70 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe400b0070 0x7efe40157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe400b0070 0x7efe40157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39758/0 (socket says 172.21.15.94:39758) 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 -- 172.21.15.94:0/2544331767 learned_addr learned my addr 172.21.15.94:0/2544331767 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3effd640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe40154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3f7fe640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe40155c30 0x7efe4014ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 -- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 msgr2=0x7efe40154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe40154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 -- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe40155c30 msgr2=0x7efe4014ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe40155c30 0x7efe4014ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 -- 172.21.15.94:0/2544331767 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe401eecd0 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.885+0000 7efe3e7fc640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe400b0070 0x7efe40157600 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7efe3400c410 tx=0x7efe3400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.886+0000 7efe277fe640 1 -- 172.21.15.94:0/2544331767 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34017070 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.886+0000 7efe277fe640 1 -- 172.21.15.94:0/2544331767 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe340035f0 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.886+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe400c46f0 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.886+0000 7efe277fe640 1 -- 172.21.15.94:0/2544331767 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe340039e0 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.886+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe401550b0 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.886+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe400c44e0 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.887+0000 7efe277fe640 1 -- 172.21.15.94:0/2544331767 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe3401a070 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.887+0000 7efe277fe640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe2c06b4d0 0x7efe2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.887+0000 7efe277fe640 1 -- 172.21.15.94:0/2544331767 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6359+0+0 (secure 0 0 0) 0x7efe340897a0 con 0x7efe400b0070 2024-03-06T07:23:09.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.888+0000 7efe3effd640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe2c06b4d0 0x7efe2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.888+0000 7efe3effd640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe2c06b4d0 0x7efe2c06d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efe30005880 tx=0x7efe30009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:09.892+0000 7efe277fe640 1 -- 172.21.15.94:0/2544331767 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe34057dd0 con 0x7efe400b0070 2024-03-06T07:23:10.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.098+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efe4004f330 con 0x7efe400b0070 2024-03-06T07:23:10.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.762+0000 7efe277fe640 1 -- 172.21.15.94:0/2544331767 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v115) v1 ==== 104+0+0 (secure 0 0 0) 0x7efe3405be70 con 0x7efe400b0070 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe2c06b4d0 msgr2=0x7efe2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe2c06b4d0 0x7efe2c06d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efe30005880 tx=0x7efe30009040 comp rx=0 tx=0).stop 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe400b0070 msgr2=0x7efe40157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe400b0070 0x7efe40157600 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7efe3400c410 tx=0x7efe3400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 shutdown_connections 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe2c06b4d0 0x7efe2c06d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe40155c30 0x7efe4014ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efe400b0070 0x7efe40157600 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 --2- 172.21.15.94:0/2544331767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4006ca70 0x7efe40154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 >> 172.21.15.94:0/2544331767 conn(0x7efe400ac590 msgr2=0x7efe40151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 shutdown_connections 2024-03-06T07:23:10.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.765+0000 7efe455e2640 1 -- 172.21.15.94:0/2544331767 wait complete. 2024-03-06T07:23:10.780 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.887+0000 7f2951815640 1 Processor -- start 2024-03-06T07:23:10.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.887+0000 7f2951815640 1 -- start start 2024-03-06T07:23:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f2951815640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f294c0c6930 0x7f294c06f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f2951815640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c152e90 con 0x7f294c0c3c20 2024-03-06T07:23:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f2951815640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c0c6f10 con 0x7f294c153910 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f2951815640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294c153480 con 0x7f294c0c6930 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f294a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f294c0c6930 0x7f294c06f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f294affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f294c0c3c20 0x7f294c156480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47246/0 (socket says 172.21.15.94:47246) 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f294a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f294c0c6930 0x7f294c06f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39772/0 (socket says 172.21.15.94:39772) 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f294affd640 1 -- 172.21.15.94:0/38836942 learned_addr learned my addr 172.21.15.94:0/38836942 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f294a7fc640 1 -- 172.21.15.94:0/38836942 >> v1:172.21.15.145:6789/0 conn(0x7f294c153910 legacy=0x7f294c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f294a7fc640 1 -- 172.21.15.94:0/38836942 >> v1:172.21.15.94:6789/0 conn(0x7f294c0c3c20 legacy=0x7f294c156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f2949ffb640 1 -- 172.21.15.94:0/38836942 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151663878 0 0) 0x7f293c001490 con 0x7f294c0c3c20 2024-03-06T07:23:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.888+0000 7f294a7fc640 1 -- 172.21.15.94:0/38836942 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c0c71f0 con 0x7f294c0c6930 2024-03-06T07:23:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.889+0000 7f294a7fc640 1 --2- 172.21.15.94:0/38836942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c06f8d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f294c153600 tx=0x7f294002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e4c45b9a9df938 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.889+0000 7f2949ffb640 1 -- 172.21.15.94:0/38836942 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940034070 con 0x7f294c0c6930 2024-03-06T07:23:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.889+0000 7f2949ffb640 1 -- 172.21.15.94:0/38836942 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940030610 con 0x7f294c0c6930 2024-03-06T07:23:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.889+0000 7f2949ffb640 1 -- 172.21.15.94:0/38836942 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29400309e0 con 0x7f294c0c6930 2024-03-06T07:23:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.890+0000 7f2951815640 1 -- 172.21.15.94:0/38836942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 msgr2=0x7f294c06f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.890+0000 7f2951815640 1 --2- 172.21.15.94:0/38836942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c06f8d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f294c153600 tx=0x7f294002b960 comp rx=0 tx=0).stop 2024-03-06T07:23:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.890+0000 7f2951815640 1 -- 172.21.15.94:0/38836942 shutdown_connections 2024-03-06T07:23:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.890+0000 7f2951815640 1 --2- 172.21.15.94:0/38836942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c06f8d0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.890+0000 7f2951815640 1 -- 172.21.15.94:0/38836942 >> 172.21.15.94:0/38836942 conn(0x7f294c0c4300 msgr2=0x7f294c0c46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.890+0000 7f2951815640 1 -- 172.21.15.94:0/38836942 shutdown_connections 2024-03-06T07:23:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.890+0000 7f2951815640 1 -- 172.21.15.94:0/38836942 wait complete. 2024-03-06T07:23:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.891+0000 7f2951815640 1 Processor -- start 2024-03-06T07:23:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.891+0000 7f2951815640 1 -- start start 2024-03-06T07:23:10.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2951815640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f294c0c3c20 0x7f294c146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2951815640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c1ec550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2951815640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f294c153910 0x7f294c14efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2951815640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c06b6d0 con 0x7f294c0c6930 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2951815640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c153d60 con 0x7f294c153910 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2951815640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c06b850 con 0x7f294c0c3c20 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c1ec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c1ec550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39778/0 (socket says 172.21.15.94:39778) 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 -- 172.21.15.94:0/1158240778 learned_addr learned my addr 172.21.15.94:0/1158240778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294affd640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f294c0c3c20 0x7f294c146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294b7fe640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f294c153910 0x7f294c14efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 -- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f294c0c3c20 msgr2=0x7f294c146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f294c0c3c20 0x7f294c146c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 -- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f294c153910 msgr2=0x7f294c14efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f294c153910 0x7f294c14efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 -- 172.21.15.94:0/1158240778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c0aa820 con 0x7f294c0c6930 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f294a7fc640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c1ec550 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2940000c00 tx=0x7f29400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2933fff640 1 -- 172.21.15.94:0/1158240778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940034040 con 0x7f294c0c6930 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2933fff640 1 -- 172.21.15.94:0/1158240778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940004030 con 0x7f294c0c6930 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.892+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c0aad20 con 0x7f294c0c6930 2024-03-06T07:23:10.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.893+0000 7f2933fff640 1 -- 172.21.15.94:0/1158240778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940004420 con 0x7f294c0c6930 2024-03-06T07:23:10.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.893+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c1ecfd0 con 0x7f294c0c6930 2024-03-06T07:23:10.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.894+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c075300 con 0x7f294c0c6930 2024-03-06T07:23:10.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.894+0000 7f2933fff640 1 -- 172.21.15.94:0/1158240778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2940037070 con 0x7f294c0c6930 2024-03-06T07:23:10.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.894+0000 7f2933fff640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f293806b4d0 0x7f293806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.894+0000 7f2933fff640 1 -- 172.21.15.94:0/1158240778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f29400ac0c0 con 0x7f294c0c6930 2024-03-06T07:23:10.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.895+0000 7f294affd640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f293806b4d0 0x7f293806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.895+0000 7f294affd640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f293806b4d0 0x7f293806d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f293c0012e0 tx=0x7f293c007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:10.899+0000 7f2933fff640 1 -- 172.21.15.94:0/1158240778 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f294c075300 con 0x7f294c0c6930 2024-03-06T07:23:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.069+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f294c0aa9f0 con 0x7f294c0c6930 2024-03-06T07:23:11.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.763+0000 7f2933fff640 1 -- 172.21.15.94:0/1158240778 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v125) v1 ==== 181+0+0 (secure 0 0 0) 0x7f294007acb0 con 0x7f294c0c6930 2024-03-06T07:23:11.765 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f293806b4d0 msgr2=0x7f293806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f293806b4d0 0x7f293806d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f293c0012e0 tx=0x7f293c007cf0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 msgr2=0x7f294c1ec550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c1ec550 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2940000c00 tx=0x7f29400046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 shutdown_connections 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f293806b4d0 0x7f293806d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f294c153910 0x7f294c14efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f294c0c6930 0x7f294c1ec550 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 --2- 172.21.15.94:0/1158240778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f294c0c3c20 0x7f294c146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 >> 172.21.15.94:0/1158240778 conn(0x7f294c0c4300 msgr2=0x7f294c150a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 shutdown_connections 2024-03-06T07:23:11.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.767+0000 7f2951815640 1 -- 172.21.15.94:0/1158240778 wait complete. 2024-03-06T07:23:11.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:11.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.889+0000 7faeefe29640 1 Processor -- start 2024-03-06T07:23:11.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.889+0000 7faeefe29640 1 -- start start 2024-03-06T07:23:11.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeefe29640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faee80a9980 0x7faee81521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeefe29640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 0x7faee806dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeefe29640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee814ef90 con 0x7faee814ebb0 2024-03-06T07:23:11.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeefe29640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee8155530 con 0x7faee80a9980 2024-03-06T07:23:11.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeefe29640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee8155a60 con 0x7faee80c28d0 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeedbc5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7faee814ebb0 0x7faee80a9fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47258/0 (socket says 172.21.15.94:47258) 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeedbc5640 1 -- 172.21.15.94:0/877862721 learned_addr learned my addr 172.21.15.94:0/877862721 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454597895 0 0) 0x7faee4001110 con 0x7faee814ebb0 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeee3c6640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 0x7faee806dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed4003520 con 0x7faee814ebb0 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeed3c4640 1 --2- 172.21.15.94:0/877862721 >> v2:172.21.15.145:3300/0 conn(0x7faee80a9980 0x7faee81521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2746758266 0 0) 0x7faed4003520 con 0x7faee814ebb0 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 msgr2=0x7faee806dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 0x7faee806dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 >> v2:172.21.15.145:3300/0 conn(0x7faee80a9980 msgr2=0x7faee81521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 --2- 172.21.15.94:0/877862721 >> v2:172.21.15.145:3300/0 conn(0x7faee80a9980 0x7faee81521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8155780 con 0x7faee814ebb0 2024-03-06T07:23:11.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7faee4001840 con 0x7faee814ebb0 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 >> v1:172.21.15.94:6789/0 conn(0x7faee814ebb0 legacy=0x7faee80a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faed40039b0 0x7faed4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 0x7faed4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faed40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee8155530 con 0x7faed4004d40 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee8155a60 con 0x7faed40039b0 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.890+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed400b9f0 con 0x7faee80c28d0 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeee3c6640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faed40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeedbc5640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faed40039b0 0x7faed4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeed3c4640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 0x7faed4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeee3c6640 1 -- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 msgr2=0x7faed4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeee3c6640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 0x7faed4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeee3c6640 1 -- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faed40039b0 msgr2=0x7faed4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeee3c6640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faed40039b0 0x7faed4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeee3c6640 1 -- 172.21.15.94:0/877862721 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4003520 con 0x7faed4004d40 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeed3c4640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee80c28d0 0x7faed4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.891+0000 7faeee3c6640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faed40051c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7faee0009fc0 tx=0x7faed4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da43bf1dc26ec7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0039070 con 0x7faed4004d40 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee002cc20 con 0x7faed4004d40 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeecbc3640 1 -- 172.21.15.94:0/877862721 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee0009c10 con 0x7faed4004d40 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeefe29640 1 -- 172.21.15.94:0/877862721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 msgr2=0x7faed40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeefe29640 1 --2- 172.21.15.94:0/877862721 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faed40051c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7faee0009fc0 tx=0x7faed4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeee3c6640 1 -- 172.21.15.94:0/877862721 reap_dead start 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeefe29640 1 -- 172.21.15.94:0/877862721 shutdown_connections 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeefe29640 1 -- 172.21.15.94:0/877862721 >> 172.21.15.94:0/877862721 conn(0x7faee80c65b0 msgr2=0x7faee80c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeefe29640 1 -- 172.21.15.94:0/877862721 shutdown_connections 2024-03-06T07:23:11.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.892+0000 7faeefe29640 1 -- 172.21.15.94:0/877862721 wait complete. 2024-03-06T07:23:11.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.893+0000 7faeefe29640 1 Processor -- start 2024-03-06T07:23:11.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.893+0000 7faeefe29640 1 -- start start 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeefe29640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faee8148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeefe29640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faee81ebb30 0x7faee8071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeefe29640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee8152a20 0x7faee80c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeefe29640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee8151c90 con 0x7faed4004d40 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeefe29640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81519b0 con 0x7faee81ebb30 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeefe29640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80c2d80 con 0x7faee8152a20 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faee8148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faee8148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39804/0 (socket says 172.21.15.94:39804) 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 -- 172.21.15.94:0/2837565447 learned_addr learned my addr 172.21.15.94:0/2837565447 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeed3c4640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faee81ebb30 0x7faee8071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeee3c6640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee8152a20 0x7faee80c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 -- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee8152a20 msgr2=0x7faee80c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee8152a20 0x7faee80c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 -- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faee81ebb30 msgr2=0x7faee8071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faee81ebb30 0x7faee8071fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 -- 172.21.15.94:0/2837565447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee81ef090 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeed3c4640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faee81ebb30 0x7faee8071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.894+0000 7faeedbc5640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faee8148f60 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7faee4007310 tx=0x7faee40077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.895+0000 7faede7fc640 1 -- 172.21.15.94:0/2837565447 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee401a070 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.895+0000 7faede7fc640 1 -- 172.21.15.94:0/2837565447 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee400f930 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.895+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee81ef330 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.895+0000 7faede7fc640 1 -- 172.21.15.94:0/2837565447 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee400fd00 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.895+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee80724e0 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.896+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee8067ad0 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.896+0000 7faede7fc640 1 -- 172.21.15.94:0/2837565447 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faee401d360 con 0x7faed4004d40 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.896+0000 7faede7fc640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faecc06b500 0x7faecc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.897+0000 7faede7fc640 1 -- 172.21.15.94:0/2837565447 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5921+0+0 (secure 0 0 0) 0x7faee4089b50 con 0x7faed4004d40 2024-03-06T07:23:11.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.897+0000 7faeed3c4640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faecc06b500 0x7faecc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.897+0000 7faeed3c4640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faecc06b500 0x7faecc06d9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faee8153b00 tx=0x7faed8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:11.901+0000 7faede7fc640 1 -- 172.21.15.94:0/2837565447 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faee8067ad0 con 0x7faed4004d40 2024-03-06T07:23:12.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.072+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faee804f330 con 0x7faed4004d40 2024-03-06T07:23:12.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.765+0000 7faede7fc640 1 -- 172.21.15.94:0/2837565447 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v126) v1 ==== 169+0+0 (secure 0 0 0) 0x7faee814ef90 con 0x7faed4004d40 2024-03-06T07:23:12.767 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faecc06b500 msgr2=0x7faecc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faecc06b500 0x7faecc06d9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faee8153b00 tx=0x7faed8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:12.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 msgr2=0x7faee8148f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faee8148f60 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7faee4007310 tx=0x7faee40077d0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 shutdown_connections 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faecc06b500 0x7faecc06d9c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faee8152a20 0x7faee80c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faee81ebb30 0x7faee8071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 --2- 172.21.15.94:0/2837565447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faed4004d40 0x7faee8148f60 secure :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7faee4007310 tx=0x7faee40077d0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 >> 172.21.15.94:0/2837565447 conn(0x7faee80c65b0 msgr2=0x7faee813b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.770+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 shutdown_connections 2024-03-06T07:23:12.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.771+0000 7faeefe29640 1 -- 172.21.15.94:0/2837565447 wait complete. 2024-03-06T07:23:12.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:12.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.892+0000 7f652e25c640 1 Processor -- start 2024-03-06T07:23:12.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.892+0000 7f652e25c640 1 -- start start 2024-03-06T07:23:12.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f652e25c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6528071c20 0x7f652814f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f652e25c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6528067350 con 0x7f652814f740 2024-03-06T07:23:12.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f652e25c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652806c320 con 0x7f6528066d00 2024-03-06T07:23:12.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f652e25c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f652814fbb0 con 0x7f6528071c20 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f6527fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f652814f740 0x7f6528152760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47268/0 (socket says 172.21.15.94:47268) 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f6527fff640 1 -- 172.21.15.94:0/540305215 learned_addr learned my addr 172.21.15.94:0/540305215 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f65277fe640 1 --2- 172.21.15.94:0/540305215 >> v2:172.21.15.145:3300/0 conn(0x7f6528071c20 0x7f652814f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837406662 0 0) 0x7f6520001490 con 0x7f652814f740 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.893+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6510003520 con 0x7f652814f740 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3939439041 0 0) 0x7f6510003520 con 0x7f652814f740 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 >> v2:172.21.15.145:3300/0 conn(0x7f6528071c20 msgr2=0x7f652814f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 --2- 172.21.15.94:0/540305215 >> v2:172.21.15.145:3300/0 conn(0x7f6528071c20 0x7f652814f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 >> v1:172.21.15.145:6789/0 conn(0x7f6528066d00 legacy=0x7f6528152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65280aca90 con 0x7f652814f740 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f6520001bc0 con 0x7f652814f740 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 >> v1:172.21.15.94:6789/0 conn(0x7f652814f740 legacy=0x7f6528152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65100039b0 0x7f6510003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6510004800 0x7f6510004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100052e0 0x7f651000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652814fbb0 con 0x7f65100039b0 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651000c1b0 con 0x7f6510004800 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651000c3b0 con 0x7f65100052e0 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65277fe640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65100039b0 0x7f6510003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f6527fff640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100052e0 0x7f651000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f6526ffd640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6510004800 0x7f6510004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65277fe640 1 -- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100052e0 msgr2=0x7f651000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65277fe640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100052e0 0x7f651000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65277fe640 1 -- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6510004800 msgr2=0x7f6510004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65277fe640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6510004800 0x7f6510004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f6527fff640 1 -- 172.21.15.94:0/540305215 reap_dead start 2024-03-06T07:23:12.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.894+0000 7f65277fe640 1 -- 172.21.15.94:0/540305215 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6510003520 con 0x7f65100039b0 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f65277fe640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65100039b0 0x7f6510003dd0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f6518009a90 tx=0x7f6510005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293a9e0569cf2bae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518039070 con 0x7f65100039b0 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651802c680 con 0x7f65100039b0 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f65267fc640 1 -- 172.21.15.94:0/540305215 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518009690 con 0x7f65100039b0 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f652e25c640 1 -- 172.21.15.94:0/540305215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65100039b0 msgr2=0x7f6510003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f652e25c640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65100039b0 0x7f6510003dd0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f6518009a90 tx=0x7f6510005830 comp rx=0 tx=0).stop 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f652e25c640 1 -- 172.21.15.94:0/540305215 shutdown_connections 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f652e25c640 1 --2- 172.21.15.94:0/540305215 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65100039b0 0x7f6510003dd0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f652e25c640 1 -- 172.21.15.94:0/540305215 >> 172.21.15.94:0/540305215 conn(0x7f65280c3b40 msgr2=0x7f65280c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f652e25c640 1 -- 172.21.15.94:0/540305215 shutdown_connections 2024-03-06T07:23:12.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.895+0000 7f652e25c640 1 -- 172.21.15.94:0/540305215 wait complete. 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.896+0000 7f652e25c640 1 Processor -- start 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f652e25c640 1 -- start start 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f652e25c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100039b0 0x7f652813cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f652e25c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65280c68b0 0x7f652814cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f652e25c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f652814d7a0 0x7f652814dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f652e25c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65280716c0 con 0x7f65280c68b0 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f652e25c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6528071540 con 0x7f652814d7a0 2024-03-06T07:23:12.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f652e25c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6528072000 con 0x7f65100039b0 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f6526ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65280c68b0 0x7f652814cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f6526ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65280c68b0 0x7f652814cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39820/0 (socket says 172.21.15.94:39820) 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.897+0000 7f6526ffd640 1 -- 172.21.15.94:0/3334013590 learned_addr learned my addr 172.21.15.94:0/3334013590 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f65277fe640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100039b0 0x7f652813cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f6527fff640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f652814d7a0 0x7f652814dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f6526ffd640 1 -- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100039b0 msgr2=0x7f652813cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f6526ffd640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100039b0 0x7f652813cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f6526ffd640 1 -- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f652814d7a0 msgr2=0x7f652814dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f6526ffd640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f652814d7a0 0x7f652814dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f6526ffd640 1 -- 172.21.15.94:0/3334013590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652814ec50 con 0x7f65280c68b0 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f65277fe640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100039b0 0x7f652813cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:12.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f6526ffd640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65280c68b0 0x7f652814cc90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f651c007fd0 tx=0x7f651c00e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f650ffff640 1 -- 172.21.15.94:0/3334013590 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c019070 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f650ffff640 1 -- 172.21.15.94:0/3334013590 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c0052e0 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65281e7030 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f650ffff640 1 -- 172.21.15.94:0/3334013590 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651c003810 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.898+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65280c5320 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.899+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f652804f330 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.899+0000 7f650ffff640 1 -- 172.21.15.94:0/3334013590 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f651c01c070 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.900+0000 7f650ffff640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f651406b500 0x7f651406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.900+0000 7f650ffff640 1 -- 172.21.15.94:0/3334013590 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f651c0896b0 con 0x7f65280c68b0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.900+0000 7f65277fe640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f651406b500 0x7f651406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.901+0000 7f65277fe640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f651406b500 0x7f651406d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6518036000 tx=0x7f6518030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:12.904+0000 7f650ffff640 1 -- 172.21.15.94:0/3334013590 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f651c058030 con 0x7f65280c68b0 2024-03-06T07:23:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.075+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f652813d1b0 con 0x7f65280c68b0 2024-03-06T07:23:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.078+0000 7f650ffff640 1 -- 172.21.15.94:0/3334013590 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f651c05c0d0 con 0x7f65280c68b0 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f651406b500 msgr2=0x7f651406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f651406b500 0x7f651406d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6518036000 tx=0x7f6518030000 comp rx=0 tx=0).stop 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65280c68b0 msgr2=0x7f652814cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65280c68b0 0x7f652814cc90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f651c007fd0 tx=0x7f651c00e8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 shutdown_connections 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f651406b500 0x7f651406d9c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f652814d7a0 0x7f652814dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65280c68b0 0x7f652814cc90 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 --2- 172.21.15.94:0/3334013590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65100039b0 0x7f652813cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 >> 172.21.15.94:0/3334013590 conn(0x7f65280c3b40 msgr2=0x7f65280ab6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 shutdown_connections 2024-03-06T07:23:13.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.080+0000 7f652e25c640 1 -- 172.21.15.94:0/3334013590 wait complete. 2024-03-06T07:23:13.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:23:13.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.195+0000 7fe4574f4640 1 Processor -- start 2024-03-06T07:23:13.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.195+0000 7fe4574f4640 1 -- start start 2024-03-06T07:23:13.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe4574f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450150330 0x7fe450156470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe4574f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe450152ea0 0x7fe450072430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe4574f4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe450156be0 con 0x7fe450072970 2024-03-06T07:23:13.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe4574f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4501569b0 con 0x7fe450152ea0 2024-03-06T07:23:13.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe4574f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe450156ec0 con 0x7fe450150330 2024-03-06T07:23:13.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455290640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450150330 0x7fe450156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe454a8f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe450152ea0 0x7fe450072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455290640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450150330 0x7fe450156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:49950/0 (socket says 172.21.15.94:49950) 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe454a8f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe450152ea0 0x7fe450072430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:40376/0 (socket says 172.21.15.94:40376) 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455a91640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fe450072970 0x7fe4500c38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49250/0 (socket says 172.21.15.94:49250) 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455290640 1 -- 172.21.15.94:0/1645363164 learned_addr learned my addr 172.21.15.94:0/1645363164 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455290640 1 -- 172.21.15.94:0/1645363164 >> v2:172.21.15.145:3300/0 conn(0x7fe450152ea0 msgr2=0x7fe450072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455290640 1 --2- 172.21.15.94:0/1645363164 >> v2:172.21.15.145:3300/0 conn(0x7fe450152ea0 0x7fe450072430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455290640 1 -- 172.21.15.94:0/1645363164 >> v1:172.21.15.145:6789/0 conn(0x7fe450072970 legacy=0x7fe4500c38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe455290640 1 -- 172.21.15.94:0/1645363164 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4501571a0 con 0x7fe450150330 2024-03-06T07:23:13.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.196+0000 7fe454a8f640 1 --2- 172.21.15.94:0/1645363164 >> v2:172.21.15.145:3300/0 conn(0x7fe450152ea0 0x7fe450072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.197+0000 7fe455290640 1 --2- 172.21.15.94:0/1645363164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450150330 0x7fe450156470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009870 tx=0x7fe44c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f455dbc29223c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.197+0000 7fe447fff640 1 -- 172.21.15.94:0/1645363164 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c034070 con 0x7fe450150330 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.197+0000 7fe447fff640 1 -- 172.21.15.94:0/1645363164 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c030840 con 0x7fe450150330 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.197+0000 7fe447fff640 1 -- 172.21.15.94:0/1645363164 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c030c10 con 0x7fe450150330 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 -- 172.21.15.94:0/1645363164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450150330 msgr2=0x7fe450156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 --2- 172.21.15.94:0/1645363164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450150330 0x7fe450156470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009870 tx=0x7fe44c02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 -- 172.21.15.94:0/1645363164 shutdown_connections 2024-03-06T07:23:13.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 --2- 172.21.15.94:0/1645363164 >> v2:172.21.15.145:3300/0 conn(0x7fe450152ea0 0x7fe450072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 --2- 172.21.15.94:0/1645363164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450150330 0x7fe450156470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 -- 172.21.15.94:0/1645363164 >> 172.21.15.94:0/1645363164 conn(0x7fe4500aa5c0 msgr2=0x7fe4500aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 -- 172.21.15.94:0/1645363164 shutdown_connections 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.198+0000 7fe4574f4640 1 -- 172.21.15.94:0/1645363164 wait complete. 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.199+0000 7fe4574f4640 1 Processor -- start 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.199+0000 7fe4574f4640 1 -- start start 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe4574f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450072970 0x7fe45014ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe4574f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450150330 0x7fe4500abad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe4574f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe450152ea0 0x7fe4501e8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe4574f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe450150890 con 0x7fe450150330 2024-03-06T07:23:13.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe4574f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe450150710 con 0x7fe450152ea0 2024-03-06T07:23:13.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe4574f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4500c3fe0 con 0x7fe450072970 2024-03-06T07:23:13.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450150330 0x7fe4500abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450150330 0x7fe4500abad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39826/0 (socket says 172.21.15.94:39826) 2024-03-06T07:23:13.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 -- 172.21.15.94:0/2289951620 learned_addr learned my addr 172.21.15.94:0/2289951620 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe455290640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450072970 0x7fe45014ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe455a91640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe450152ea0 0x7fe4501e8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 -- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450072970 msgr2=0x7fe45014ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450072970 0x7fe45014ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 -- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe450152ea0 msgr2=0x7fe4501e8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe450152ea0 0x7fe4501e8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 -- 172.21.15.94:0/2289951620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4501df9c0 con 0x7fe450150330 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe454a8f640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450150330 0x7fe4500abad0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e990 tx=0x7fe44000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe445ffb640 1 -- 172.21.15.94:0/2289951620 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440017070 con 0x7fe450150330 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe445ffb640 1 -- 172.21.15.94:0/2289951620 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44000c9e0 con 0x7fe450150330 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe445ffb640 1 -- 172.21.15.94:0/2289951620 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44000cdb0 con 0x7fe450150330 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.200+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4501561e0 con 0x7fe450150330 2024-03-06T07:23:13.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.201+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4501e8910 con 0x7fe450150330 2024-03-06T07:23:13.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.201+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45004f330 con 0x7fe450150330 2024-03-06T07:23:13.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.202+0000 7fe445ffb640 1 -- 172.21.15.94:0/2289951620 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe44001a070 con 0x7fe450150330 2024-03-06T07:23:13.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.202+0000 7fe445ffb640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe43406b4d0 0x7fe43406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.202+0000 7fe445ffb640 1 -- 172.21.15.94:0/2289951620 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe440089f30 con 0x7fe450150330 2024-03-06T07:23:13.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.202+0000 7fe455290640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe43406b4d0 0x7fe43406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.203+0000 7fe455290640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe43406b4d0 0x7fe43406d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe44c036000 tx=0x7fe44c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.207+0000 7fe445ffb640 1 -- 172.21.15.94:0/2289951620 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4501df9c0 con 0x7fe450150330 2024-03-06T07:23:13.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.382+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7fe45014e950 con 0x7fe450150330 2024-03-06T07:23:13.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.385+0000 7fe445ffb640 1 -- 172.21.15.94:0/2289951620 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v44) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe440058f00 con 0x7fe450150330 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe43406b4d0 msgr2=0x7fe43406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe43406b4d0 0x7fe43406d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe44c036000 tx=0x7fe44c03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450150330 msgr2=0x7fe4500abad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450150330 0x7fe4500abad0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e990 tx=0x7fe44000ee50 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 shutdown_connections 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe43406b4d0 0x7fe43406d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe450152ea0 0x7fe4501e8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe450150330 0x7fe4500abad0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 --2- 172.21.15.94:0/2289951620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe450072970 0x7fe45014ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 >> 172.21.15.94:0/2289951620 conn(0x7fe4500aa5c0 msgr2=0x7fe45014d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 shutdown_connections 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.387+0000 7fe4574f4640 1 -- 172.21.15.94:0/2289951620 wait complete. 2024-03-06T07:23:13.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-03-06T07:23:13.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.498+0000 7f42653ac640 1 Processor -- start 2024-03-06T07:23:13.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.498+0000 7f42653ac640 1 -- start start 2024-03-06T07:23:13.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f42653ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 0x7f4260155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f42653ac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600c73d0 con 0x7f42600c88c0 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f42653ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42600be2e0 con 0x7f426014f360 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f42653ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4260159700 con 0x7f4260159270 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f425e7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f42600c88c0 0x7f4260158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47278/0 (socket says 172.21.15.94:47278) 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f425e7fc640 1 -- 172.21.15.94:0/1192316669 learned_addr learned my addr 172.21.15.94:0/1192316669 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f425f7fe640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 0x7f4260155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939110674 0 0) 0x7f42600c73d0 con 0x7f42600c88c0 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.499+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c003520 con 0x7f42600c88c0 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504929259 0 0) 0x7f42600be2e0 con 0x7f426014f360 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42600c73d0 con 0x7f426014f360 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605519719 0 0) 0x7f424c003520 con 0x7f42600c88c0 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 msgr2=0x7f4260155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 0x7f4260155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 >> v1:172.21.15.145:6789/0 conn(0x7f426014f360 legacy=0x7f42600c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42600c7670 con 0x7f42600c88c0 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f4248001670 con 0x7f42600c88c0 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 >> v1:172.21.15.94:6789/0 conn(0x7f42600c88c0 legacy=0x7f4260158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f424c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 0x7f424c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c004d40 0x7f424c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260159700 con 0x7f424c003b70 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c00baf0 con 0x7f424c004d40 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424c00bcf0 con 0x7f4260159270 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425effd640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f424c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425e7fc640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 0x7f424c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425f7fe640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c004d40 0x7f424c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425effd640 1 -- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 msgr2=0x7f424c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425effd640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4260159270 0x7f424c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425effd640 1 -- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c004d40 msgr2=0x7f424c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425effd640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f424c004d40 0x7f424c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.500+0000 7f425effd640 1 -- 172.21.15.94:0/1192316669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c003520 con 0x7f424c003b70 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.501+0000 7f425effd640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f424c003f70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4250001fd0 tx=0x7f424c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9456a26936841b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.501+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250001bf0 con 0x7f424c003b70 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.501+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250003b70 con 0x7f424c003b70 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.501+0000 7f425dffb640 1 -- 172.21.15.94:0/1192316669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42500050d0 con 0x7f424c003b70 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.502+0000 7f42653ac640 1 -- 172.21.15.94:0/1192316669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 msgr2=0x7f424c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.502+0000 7f42653ac640 1 --2- 172.21.15.94:0/1192316669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f424c003f70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4250001fd0 tx=0x7f424c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.502+0000 7f425f7fe640 1 -- 172.21.15.94:0/1192316669 reap_dead start 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.502+0000 7f42653ac640 1 -- 172.21.15.94:0/1192316669 shutdown_connections 2024-03-06T07:23:13.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.502+0000 7f42653ac640 1 -- 172.21.15.94:0/1192316669 >> 172.21.15.94:0/1192316669 conn(0x7f42600ac570 msgr2=0x7f42600ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.502+0000 7f42653ac640 1 -- 172.21.15.94:0/1192316669 shutdown_connections 2024-03-06T07:23:13.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.502+0000 7f42653ac640 1 -- 172.21.15.94:0/1192316669 wait complete. 2024-03-06T07:23:13.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.503+0000 7f42653ac640 1 Processor -- start 2024-03-06T07:23:13.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.503+0000 7f42653ac640 1 -- start start 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f4260157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4260152980 0x7f426014bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42601e1460 0x7f426014c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600c7c30 con 0x7f424c003b70 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42600c5a50 con 0x7f4260152980 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42600c5bd0 con 0x7f42601e1460 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f4260157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f4260157840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39852/0 (socket says 172.21.15.94:39852) 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 -- 172.21.15.94:0/33038181 learned_addr learned my addr 172.21.15.94:0/33038181 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425f7fe640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42601e1460 0x7f426014c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425e7fc640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4260152980 0x7f426014bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 -- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42601e1460 msgr2=0x7f426014c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42601e1460 0x7f426014c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 -- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4260152980 msgr2=0x7f426014bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4260152980 0x7f426014bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 -- 172.21.15.94:0/33038181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426014c5b0 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f425effd640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f4260157840 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f4250005040 tx=0x7f4250001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42577fe640 1 -- 172.21.15.94:0/33038181 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250041020 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42577fe640 1 -- 172.21.15.94:0/33038181 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250003970 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42600be740 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42577fe640 1 -- 172.21.15.94:0/33038181 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42500054e0 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.504+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4260158300 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.505+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42600c8590 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.506+0000 7f42577fe640 1 -- 172.21.15.94:0/33038181 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4250044020 con 0x7f424c003b70 2024-03-06T07:23:13.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.506+0000 7f42577fe640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424006b4d0 0x7f424006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.506+0000 7f42577fe640 1 -- 172.21.15.94:0/33038181 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f42500ab880 con 0x7f424c003b70 2024-03-06T07:23:13.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.506+0000 7f425e7fc640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424006b4d0 0x7f424006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.507+0000 7f425e7fc640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424006b4d0 0x7f424006d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f42600bdd80 tx=0x7f4248009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.511+0000 7f42577fe640 1 -- 172.21.15.94:0/33038181 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42600c8590 con 0x7f424c003b70 2024-03-06T07:23:13.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.749+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7f426004f330 con 0x7f424c003b70 2024-03-06T07:23:13.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.767+0000 7f42577fe640 1 -- 172.21.15.94:0/33038181 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v293) v1 ==== 145+0+0 (secure 0 0 0) 0x7f425004b030 con 0x7f424c003b70 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424006b4d0 msgr2=0x7f424006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424006b4d0 0x7f424006d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f42600bdd80 tx=0x7f4248009040 comp rx=0 tx=0).stop 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 msgr2=0x7f4260157840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f4260157840 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f4250005040 tx=0x7f4250001650 comp rx=0 tx=0).stop 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 shutdown_connections 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424006b4d0 0x7f424006d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42601e1460 0x7f426014c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4260152980 0x7f426014bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 --2- 172.21.15.94:0/33038181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f424c003b70 0x7f4260157840 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 >> 172.21.15.94:0/33038181 conn(0x7f42600ac570 msgr2=0x7f426006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 shutdown_connections 2024-03-06T07:23:13.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.770+0000 7f42653ac640 1 -- 172.21.15.94:0/33038181 wait complete. 2024-03-06T07:23:13.782 INFO:tasks.cephfs_test_runner:That a sole MDS with mds_join_fs set will come back as active eventually even after failover. ... ok 2024-03-06T07:23:13.783 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will cause failover if another high-affinity standby exists. 2024-03-06T07:23:13.783 INFO:tasks.cephfs_test_runner:test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:23:13.783 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:13.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.881+0000 7f3ef2de9640 1 Processor -- start 2024-03-06T07:23:13.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.881+0000 7f3ef2de9640 1 -- start start 2024-03-06T07:23:13.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.881+0000 7f3ef2de9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3eec0c3370 0x7f3eec155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.881+0000 7f3ef2de9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0c5490 con 0x7f3eec0c3750 2024-03-06T07:23:13.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3ef2de9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0c5770 con 0x7f3eec0a5e60 2024-03-06T07:23:13.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3ef2de9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eec0c5a50 con 0x7f3eec0c3370 2024-03-06T07:23:13.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3ef0b85640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3eec0c3370 0x7f3eec155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3eebfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3eec0c3750 0x7f3eec0c1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47292/0 (socket says 172.21.15.94:47292) 2024-03-06T07:23:13.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3ef0b85640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3eec0c3370 0x7f3eec155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39862/0 (socket says 172.21.15.94:39862) 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3eebfff640 1 -- 172.21.15.94:0/1044229522 learned_addr learned my addr 172.21.15.94:0/1044229522 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3ef0b85640 1 -- 172.21.15.94:0/1044229522 >> v1:172.21.15.145:6789/0 conn(0x7f3eec0a5e60 legacy=0x7f3eec15fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3ef0b85640 1 -- 172.21.15.94:0/1044229522 >> v1:172.21.15.94:6789/0 conn(0x7f3eec0c3750 legacy=0x7f3eec0c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3ef0b85640 1 -- 172.21.15.94:0/1044229522 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec0c5d30 con 0x7f3eec0c3370 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.882+0000 7f3eeb7fe640 1 -- 172.21.15.94:0/1044229522 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384546112 0 0) 0x7f3eec0c5490 con 0x7f3eec0c3750 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef0b85640 1 --2- 172.21.15.94:0/1044229522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec155ad0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00097a0 tx=0x7f3ee002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6761abeadd74728 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3eeb7fe640 1 -- 172.21.15.94:0/1044229522 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0038070 con 0x7f3eec0c3370 2024-03-06T07:23:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3eeb7fe640 1 -- 172.21.15.94:0/1044229522 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee0031700 con 0x7f3eec0c3370 2024-03-06T07:23:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3eeb7fe640 1 -- 172.21.15.94:0/1044229522 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0031ad0 con 0x7f3eec0c3370 2024-03-06T07:23:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef2de9640 1 -- 172.21.15.94:0/1044229522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 msgr2=0x7f3eec155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/1044229522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec155ad0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f3ee00097a0 tx=0x7f3ee002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef2de9640 1 -- 172.21.15.94:0/1044229522 shutdown_connections 2024-03-06T07:23:13.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/1044229522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec155ad0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef2de9640 1 -- 172.21.15.94:0/1044229522 >> 172.21.15.94:0/1044229522 conn(0x7f3eec0c6710 msgr2=0x7f3eec0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef2de9640 1 -- 172.21.15.94:0/1044229522 shutdown_connections 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.883+0000 7f3ef2de9640 1 -- 172.21.15.94:0/1044229522 wait complete. 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.884+0000 7f3ef2de9640 1 Processor -- start 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.884+0000 7f3ef2de9640 1 -- start start 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3ef2de9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3eec0a5e60 0x7f3eec070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3ef2de9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3ef2de9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3eec0c3750 0x7f3eec14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3ef2de9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec159910 con 0x7f3eec0c3370 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3ef2de9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec159790 con 0x7f3eec0c3750 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3ef2de9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec0c4ed0 con 0x7f3eec0a5e60 2024-03-06T07:23:13.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3eebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3eebfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec06d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39864/0 (socket says 172.21.15.94:39864) 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.885+0000 7f3eebfff640 1 -- 172.21.15.94:0/3946862412 learned_addr learned my addr 172.21.15.94:0/3946862412 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ef0b85640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3eec0a5e60 0x7f3eec070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ef1386640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3eec0c3750 0x7f3eec14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3eebfff640 1 -- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3eec0a5e60 msgr2=0x7f3eec070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3eebfff640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3eec0a5e60 0x7f3eec070ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3eebfff640 1 -- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3eec0c3750 msgr2=0x7f3eec14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3eebfff640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3eec0c3750 0x7f3eec14ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3eebfff640 1 -- 172.21.15.94:0/3946862412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec157af0 con 0x7f3eec0c3370 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ef1386640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3eec0c3750 0x7f3eec14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3eebfff640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec06d390 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c430 tx=0x7f3ed800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ee97fa640 1 -- 172.21.15.94:0/3946862412 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80074e0 con 0x7f3eec0c3370 2024-03-06T07:23:13.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ee97fa640 1 -- 172.21.15.94:0/3946862412 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed80047e0 con 0x7f3eec0c3370 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec157580 con 0x7f3eec0c3370 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ee97fa640 1 -- 172.21.15.94:0/3946862412 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8004bd0 con 0x7f3eec0c3370 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.886+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec06d8d0 con 0x7f3eec0c3370 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.887+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eec158430 con 0x7f3eec0c3370 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.887+0000 7f3ee97fa640 1 -- 172.21.15.94:0/3946862412 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ed8011070 con 0x7f3eec0c3370 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.888+0000 7f3ee97fa640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed006b4d0 0x7f3ed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.888+0000 7f3ee97fa640 1 -- 172.21.15.94:0/3946862412 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f3ed8089d60 con 0x7f3eec0c3370 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.888+0000 7f3ef0b85640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed006b4d0 0x7f3ed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.889+0000 7f3ef0b85640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed006b4d0 0x7f3ed006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0035000 tx=0x7f3ee0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:13.892+0000 7f3ee97fa640 1 -- 172.21.15.94:0/3946862412 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ed8058ca0 con 0x7f3eec0c3370 2024-03-06T07:23:14.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.062+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eec157cc0 con 0x7f3eec0c3370 2024-03-06T07:23:14.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.063+0000 7f3ee97fa640 1 -- 172.21.15.94:0/3946862412 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+13563 (secure 0 0 0) 0x7f3ed80585a0 con 0x7f3eec0c3370 2024-03-06T07:23:14.065 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:14.065 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":126,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:12.764835+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/1862149336":"2024-03-07T07:23:09.760690+0000","172.21.15.145:6834/1862149336":"2024-03-07T07:23:09.760690+0000","172.21.15.94:6834/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.94:6832/282207355":"2024-03-07T07:23:02.205014+0000","172.21.15.145:6836/198029515":"2024-03-07T07:23:00.021121+0000","172.21.15.145:6834/198029515":"2024-03-07T07:23:00.021121+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:14.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed006b4d0 msgr2=0x7f3ed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed006b4d0 0x7f3ed006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0035000 tx=0x7f3ee0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:14.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 msgr2=0x7f3eec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec06d390 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c430 tx=0x7f3ed800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 shutdown_connections 2024-03-06T07:23:14.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed006b4d0 0x7f3ed006d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3eec0c3750 0x7f3eec14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3eec0c3370 0x7f3eec06d390 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 --2- 172.21.15.94:0/3946862412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3eec0a5e60 0x7f3eec070ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 >> 172.21.15.94:0/3946862412 conn(0x7f3eec0c6710 msgr2=0x7f3eec0a70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 shutdown_connections 2024-03-06T07:23:14.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.065+0000 7f3ef2de9640 1 -- 172.21.15.94:0/3946862412 wait complete. 2024-03-06T07:23:14.078 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:14.079 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.180+0000 7fe51dbb5640 1 Processor -- start 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.180+0000 7fe51dbb5640 1 -- start start 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.180+0000 7fe51dbb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5181528d0 0x7fe5180724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.180+0000 7fe51dbb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe518072a10 0x7fe518155220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.180+0000 7fe51dbb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe518152cb0 con 0x7fe5180c67f0 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.180+0000 7fe51dbb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5180c38b0 con 0x7fe518072a10 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.180+0000 7fe51dbb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5181594f0 con 0x7fe5181528d0 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe5177fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5181528d0 0x7fe5180724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe517fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fe5180c67f0 0x7fe518155760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49272/0 (socket says 172.21.15.94:49272) 2024-03-06T07:23:14.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe517fff640 1 -- 172.21.15.94:0/2681116615 learned_addr learned my addr 172.21.15.94:0/2681116615 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:14.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe516ffd640 1 --2- 172.21.15.94:0/2681116615 >> v2:172.21.15.145:3300/0 conn(0x7fe518072a10 0x7fe518155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe5177fe640 1 -- 172.21.15.94:0/2681116615 >> v2:172.21.15.145:3300/0 conn(0x7fe518072a10 msgr2=0x7fe518155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe5177fe640 1 --2- 172.21.15.94:0/2681116615 >> v2:172.21.15.145:3300/0 conn(0x7fe518072a10 0x7fe518155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe5177fe640 1 -- 172.21.15.94:0/2681116615 >> v1:172.21.15.145:6789/0 conn(0x7fe5180c67f0 legacy=0x7fe518155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.181+0000 7fe5177fe640 1 -- 172.21.15.94:0/2681116615 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5180a7f70 con 0x7fe5181528d0 2024-03-06T07:23:14.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.182+0000 7fe5177fe640 1 --2- 172.21.15.94:0/2681116615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5181528d0 0x7fe5180724d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5080097a0 tx=0x7fe50802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b66e138f100e85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.182+0000 7fe5167fc640 1 -- 172.21.15.94:0/2681116615 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508034070 con 0x7fe5181528d0 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.182+0000 7fe5167fc640 1 -- 172.21.15.94:0/2681116615 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe508030610 con 0x7fe5181528d0 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.182+0000 7fe5167fc640 1 -- 172.21.15.94:0/2681116615 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080309e0 con 0x7fe5181528d0 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 -- 172.21.15.94:0/2681116615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5181528d0 msgr2=0x7fe5180724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/2681116615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5181528d0 0x7fe5180724d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5080097a0 tx=0x7fe50802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 -- 172.21.15.94:0/2681116615 shutdown_connections 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/2681116615 >> v2:172.21.15.145:3300/0 conn(0x7fe518072a10 0x7fe518155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/2681116615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5181528d0 0x7fe5180724d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 -- 172.21.15.94:0/2681116615 >> 172.21.15.94:0/2681116615 conn(0x7fe5180aa6a0 msgr2=0x7fe5180aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 -- 172.21.15.94:0/2681116615 shutdown_connections 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.183+0000 7fe51dbb5640 1 -- 172.21.15.94:0/2681116615 wait complete. 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.184+0000 7fe51dbb5640 1 Processor -- start 2024-03-06T07:23:14.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.184+0000 7fe51dbb5640 1 -- start start 2024-03-06T07:23:14.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe518072a10 0x7fe518156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5180c67f0 0x7fe5181ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5181528d0 0x7fe5180706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5181591a0 con 0x7fe5181528d0 2024-03-06T07:23:14.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5180a8250 con 0x7fe518072a10 2024-03-06T07:23:14.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5180c4ef0 con 0x7fe5180c67f0 2024-03-06T07:23:14.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5181528d0 0x7fe5180706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5181528d0 0x7fe5180706d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39876/0 (socket says 172.21.15.94:39876) 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 -- 172.21.15.94:0/1400734852 learned_addr learned my addr 172.21.15.94:0/1400734852 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe516ffd640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5180c67f0 0x7fe5181ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe5177fe640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe518072a10 0x7fe518156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 -- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5180c67f0 msgr2=0x7fe5181ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5180c67f0 0x7fe5181ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 -- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe518072a10 msgr2=0x7fe518156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe518072a10 0x7fe518156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 -- 172.21.15.94:0/1400734852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5181491b0 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe5177fe640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe518072a10 0x7fe518156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe517fff640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5181528d0 0x7fe5180706d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe51000cb20 tx=0x7fe51000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe4fffff640 1 -- 172.21.15.94:0/1400734852 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510017070 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe4fffff640 1 -- 172.21.15.94:0/1400734852 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51000ac40 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe518148d10 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe4fffff640 1 -- 172.21.15.94:0/1400734852 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510012400 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.185+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe518070c10 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.186+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51804f330 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.187+0000 7fe4fffff640 1 -- 172.21.15.94:0/1400734852 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe51001a070 con 0x7fe5181528d0 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.187+0000 7fe4fffff640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.187+0000 7fe4fffff640 1 -- 172.21.15.94:0/1400734852 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fe510089d50 con 0x7fe5181528d0 2024-03-06T07:23:14.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.187+0000 7fe5177fe640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.188+0000 7fe5177fe640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe50406b4d0 0x7fe50406d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe508036000 tx=0x7fe508008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.191+0000 7fe4fffff640 1 -- 172.21.15.94:0/1400734852 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe51008d030 con 0x7fe5181528d0 2024-03-06T07:23:14.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.365+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe518148ee0 con 0x7fe5181528d0 2024-03-06T07:23:14.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.366+0000 7fe4fffff640 1 -- 172.21.15.94:0/1400734852 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe510058650 con 0x7fe5181528d0 2024-03-06T07:23:14.368 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe50406b4d0 msgr2=0x7fe50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe50406b4d0 0x7fe50406d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe508036000 tx=0x7fe508008030 comp rx=0 tx=0).stop 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5181528d0 msgr2=0x7fe5180706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5181528d0 0x7fe5180706d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe51000cb20 tx=0x7fe51000a060 comp rx=0 tx=0).stop 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 shutdown_connections 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5181528d0 0x7fe5180706d0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5180c67f0 0x7fe5181ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 --2- 172.21.15.94:0/1400734852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe518072a10 0x7fe518156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 >> 172.21.15.94:0/1400734852 conn(0x7fe5180aa6a0 msgr2=0x7fe51806d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 shutdown_connections 2024-03-06T07:23:14.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.368+0000 7fe51dbb5640 1 -- 172.21.15.94:0/1400734852 wait complete. 2024-03-06T07:23:14.381 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-03-06T07:23:14.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.480+0000 7f209ea78640 1 Processor -- start 2024-03-06T07:23:14.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.480+0000 7f209ea78640 1 -- start start 2024-03-06T07:23:14.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.480+0000 7f209ea78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209814f790 0x7f20981529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f209ea78640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20980c1aa0 0x7f20980c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f209ea78640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20980c28d0 con 0x7f20980c5c60 2024-03-06T07:23:14.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f209ea78640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20980c2680 con 0x7f20980c1aa0 2024-03-06T07:23:14.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f209ea78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20980c2bb0 con 0x7f209814f790 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f2097fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20980c1aa0 0x7f20980c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f2097fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20980c1aa0 0x7f20980c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39892/0 (socket says 172.21.15.94:39892) 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f209d015640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f20980c5c60 0x7f2098155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47296/0 (socket says 172.21.15.94:47296) 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f2097fff640 1 -- 172.21.15.94:0/3955290525 learned_addr learned my addr 172.21.15.94:0/3955290525 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f209c814640 1 --2- 172.21.15.94:0/3955290525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209814f790 0x7f20981529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f2097fff640 1 -- 172.21.15.94:0/3955290525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209814f790 msgr2=0x7f20981529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f2097fff640 1 --2- 172.21.15.94:0/3955290525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209814f790 0x7f20981529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f2097fff640 1 -- 172.21.15.94:0/3955290525 >> v1:172.21.15.94:6789/0 conn(0x7f20980c5c60 legacy=0x7f2098155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f2097fff640 1 -- 172.21.15.94:0/3955290525 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980c2e90 con 0x7f20980c1aa0 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.481+0000 7f20977fe640 1 -- 172.21.15.94:0/3955290525 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451195531 0 0) 0x7f20980c28d0 con 0x7f20980c5c60 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f2097fff640 1 --2- 172.21.15.94:0/3955290525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20980c1aa0 0x7f20980c5720 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f2084009870 tx=0x7f208402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f384aabf47bd25c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f20977fe640 1 -- 172.21.15.94:0/3955290525 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084034070 con 0x7f20980c1aa0 2024-03-06T07:23:14.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f20977fe640 1 -- 172.21.15.94:0/3955290525 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084030840 con 0x7f20980c1aa0 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f20977fe640 1 -- 172.21.15.94:0/3955290525 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084030c10 con 0x7f20980c1aa0 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f209ea78640 1 -- 172.21.15.94:0/3955290525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20980c1aa0 msgr2=0x7f20980c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f209ea78640 1 --2- 172.21.15.94:0/3955290525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20980c1aa0 0x7f20980c5720 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f2084009870 tx=0x7f208402b980 comp rx=0 tx=0).stop 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f209ea78640 1 -- 172.21.15.94:0/3955290525 shutdown_connections 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f209ea78640 1 --2- 172.21.15.94:0/3955290525 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20980c1aa0 0x7f20980c5720 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f209ea78640 1 --2- 172.21.15.94:0/3955290525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209814f790 0x7f20981529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.482+0000 7f209ea78640 1 -- 172.21.15.94:0/3955290525 >> 172.21.15.94:0/3955290525 conn(0x7f20980c6710 msgr2=0x7f20980c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.483+0000 7f209ea78640 1 -- 172.21.15.94:0/3955290525 shutdown_connections 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.483+0000 7f209ea78640 1 -- 172.21.15.94:0/3955290525 wait complete. 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 Processor -- start 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 -- start start 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20980c1aa0 0x7f209815cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20980c5c60 0x7f2098071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f209814f790 0x7f20980722e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209806cc00 con 0x7f209814f790 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20980c0d60 con 0x7f20980c1aa0 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.484+0000 7f209ea78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209806cd80 con 0x7f20980c5c60 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f209814f790 0x7f20980722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f209814f790 0x7f20980722e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39906/0 (socket says 172.21.15.94:39906) 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 -- 172.21.15.94:0/1015211200 learned_addr learned my addr 172.21.15.94:0/1015211200 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f2097fff640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20980c5c60 0x7f2098071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209c814640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20980c1aa0 0x7f209815cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 -- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20980c5c60 msgr2=0x7f2098071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20980c5c60 0x7f2098071da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 -- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20980c1aa0 msgr2=0x7f209815cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20980c1aa0 0x7f209815cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 -- 172.21.15.94:0/1015211200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980bb8d0 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209d015640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f209814f790 0x7f20980722e0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f209000c7d0 tx=0x7f209000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f20957fa640 1 -- 172.21.15.94:0/1015211200 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090018070 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f20957fa640 1 -- 172.21.15.94:0/1015211200 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209000a640 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20980c0320 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f20957fa640 1 -- 172.21.15.94:0/1015211200 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209000aa70 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.485+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2098152880 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.486+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20980c59a0 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.487+0000 7f20957fa640 1 -- 172.21.15.94:0/1015211200 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f209000ac10 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.487+0000 7f20957fa640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f207c06b500 0x7f207c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.487+0000 7f20957fa640 1 -- 172.21.15.94:0/1015211200 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f2090089790 con 0x7f209814f790 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.487+0000 7f209c814640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f207c06b500 0x7f207c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.488+0000 7f209c814640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f207c06b500 0x7f207c06d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f208c009970 tx=0x7f208c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.491+0000 7f20957fa640 1 -- 172.21.15.94:0/1015211200 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20980c59a0 con 0x7f209814f790 2024-03-06T07:23:14.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.730+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7f209804f330 con 0x7f209814f790 2024-03-06T07:23:14.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.771+0000 7f20957fa640 1 -- 172.21.15.94:0/1015211200 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v294) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2090058650 con 0x7f209814f790 2024-03-06T07:23:14.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f207c06b500 msgr2=0x7f207c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f207c06b500 0x7f207c06d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f208c009970 tx=0x7f208c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:14.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f209814f790 msgr2=0x7f20980722e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f209814f790 0x7f20980722e0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f209000c7d0 tx=0x7f209000cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 shutdown_connections 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f207c06b500 0x7f207c06d9c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f209814f790 0x7f20980722e0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20980c5c60 0x7f2098071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 --2- 172.21.15.94:0/1015211200 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20980c1aa0 0x7f209815cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 >> 172.21.15.94:0/1015211200 conn(0x7f20980c6710 msgr2=0x7f20980c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 shutdown_connections 2024-03-06T07:23:14.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.773+0000 7f209ea78640 1 -- 172.21.15.94:0/1015211200 wait complete. 2024-03-06T07:23:14.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.887+0000 7fbcc3c9a640 1 Processor -- start 2024-03-06T07:23:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.887+0000 7fbcc3c9a640 1 -- start start 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc3c9a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbcbc0b0070 0x7fbcbc155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc3c9a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbcbc06ca70 0x7fbcbc0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc3c9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0c5380 0x7fbcbc0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc3c9a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcbc06c020 con 0x7fbcbc0b0070 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc3c9a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcbc0b0450 con 0x7fbcbc06ca70 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc3c9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc06cf20 con 0x7fbcbc0c5380 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc1a36640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbcbc0b0070 0x7fbcbc155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc1a36640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbcbc0b0070 0x7fbcbc155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39910/0 (socket says 172.21.15.94:39910) 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.888+0000 7fbcc1a36640 1 -- 172.21.15.94:0/2167270909 learned_addr learned my addr 172.21.15.94:0/2167270909 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.889+0000 7fbcc1a36640 1 -- 172.21.15.94:0/2167270909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0c5380 msgr2=0x7fbcbc0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.889+0000 7fbcc1a36640 1 --2- 172.21.15.94:0/2167270909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0c5380 0x7fbcbc0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.889+0000 7fbcc1a36640 1 -- 172.21.15.94:0/2167270909 >> v2:172.21.15.145:3300/0 conn(0x7fbcbc06ca70 msgr2=0x7fbcbc0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.889+0000 7fbcc1a36640 1 --2- 172.21.15.94:0/2167270909 >> v2:172.21.15.145:3300/0 conn(0x7fbcbc06ca70 0x7fbcbc0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.889+0000 7fbcc1a36640 1 -- 172.21.15.94:0/2167270909 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc0c77f0 con 0x7fbcbc0b0070 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.889+0000 7fbcc1a36640 1 --2- 172.21.15.94:0/2167270909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc0b0070 0x7fbcbc155c30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc06d4c0 tx=0x7fbcb802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88012f25eb7a6a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.889+0000 7fbcc0a34640 1 -- 172.21.15.94:0/2167270909 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb802c9e0 con 0x7fbcbc0b0070 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc0a34640 1 -- 172.21.15.94:0/2167270909 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb802cb80 con 0x7fbcbc0b0070 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc0a34640 1 -- 172.21.15.94:0/2167270909 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb80343f0 con 0x7fbcbc0b0070 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/2167270909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc0b0070 msgr2=0x7fbcbc155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/2167270909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc0b0070 0x7fbcbc155c30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc06d4c0 tx=0x7fbcb802be40 comp rx=0 tx=0).stop 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/2167270909 shutdown_connections 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/2167270909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0c5380 0x7fbcbc0c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/2167270909 >> v2:172.21.15.145:3300/0 conn(0x7fbcbc06ca70 0x7fbcbc0c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/2167270909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc0b0070 0x7fbcbc155c30 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/2167270909 >> 172.21.15.94:0/2167270909 conn(0x7fbcbc0ac590 msgr2=0x7fbcbc0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/2167270909 shutdown_connections 2024-03-06T07:23:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.890+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/2167270909 wait complete. 2024-03-06T07:23:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.891+0000 7fbcc3c9a640 1 Processor -- start 2024-03-06T07:23:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.891+0000 7fbcc3c9a640 1 -- start start 2024-03-06T07:23:14.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc3c9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc06ca70 0x7fbcbc0c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc3c9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0b0070 0x7fbcbc1ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc3c9a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcbc0c5380 0x7fbcbc0ab500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc3c9a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc06c550 con 0x7fbcbc06ca70 2024-03-06T07:23:14.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc3c9a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc06c270 con 0x7fbcbc0c5380 2024-03-06T07:23:14.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc3c9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc06da80 con 0x7fbcbc0b0070 2024-03-06T07:23:14.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc06ca70 0x7fbcbc0c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc06ca70 0x7fbcbc0c0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39916/0 (socket says 172.21.15.94:39916) 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 -- 172.21.15.94:0/1817106866 learned_addr learned my addr 172.21.15.94:0/1817106866 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 -- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0b0070 msgr2=0x7fbcbc1ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0b0070 0x7fbcbc1ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 -- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcbc0c5380 msgr2=0x7fbcbc0ab500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcbc0c5380 0x7fbcbc0ab500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 -- 172.21.15.94:0/1817106866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc0bec50 con 0x7fbcbc06ca70 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcc1a36640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc06ca70 0x7fbcbc0c0d10 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8036040 tx=0x7fbcb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.892+0000 7fbcb27fc640 1 -- 172.21.15.94:0/1817106866 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8043020 con 0x7fbcbc06ca70 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.893+0000 7fbcb27fc640 1 -- 172.21.15.94:0/1817106866 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb803e450 con 0x7fbcbc06ca70 2024-03-06T07:23:14.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.893+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcbc0ca640 con 0x7fbcbc06ca70 2024-03-06T07:23:14.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.893+0000 7fbcb27fc640 1 -- 172.21.15.94:0/1817106866 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8009510 con 0x7fbcbc06ca70 2024-03-06T07:23:14.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.893+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcbc1582e0 con 0x7fbcbc06ca70 2024-03-06T07:23:14.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.893+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcbc04f330 con 0x7fbcbc06ca70 2024-03-06T07:23:14.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.894+0000 7fbcb27fc640 1 -- 172.21.15.94:0/1817106866 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcb8005d10 con 0x7fbcbc06ca70 2024-03-06T07:23:14.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.894+0000 7fbcb27fc640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbca006b4d0 0x7fbca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.894+0000 7fbcb27fc640 1 -- 172.21.15.94:0/1817106866 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fbcb80acd60 con 0x7fbcbc06ca70 2024-03-06T07:23:14.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.894+0000 7fbcc1235640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbca006b4d0 0x7fbca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.895+0000 7fbcc1235640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbca006b4d0 0x7fbca006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc1e8c50 tx=0x7fbcac006bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:14.898+0000 7fbcb27fc640 1 -- 172.21.15.94:0/1817106866 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcb802c9f0 con 0x7fbcbc06ca70 2024-03-06T07:23:14.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:14.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:15.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.076+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbcbc06d860 con 0x7fbcbc06ca70 2024-03-06T07:23:15.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.079+0000 7fbcb27fc640 1 -- 172.21.15.94:0/1817106866 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v45)=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbcb807bd30 con 0x7fbcbc06ca70 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbca006b4d0 msgr2=0x7fbca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbca006b4d0 0x7fbca006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc1e8c50 tx=0x7fbcac006bf0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc06ca70 msgr2=0x7fbcbc0c0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc06ca70 0x7fbcbc0c0d10 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8036040 tx=0x7fbcb8004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 shutdown_connections 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbca006b4d0 0x7fbca006d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcbc0c5380 0x7fbcbc0ab500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcbc0b0070 0x7fbcbc1ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 --2- 172.21.15.94:0/1817106866 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcbc06ca70 0x7fbcbc0c0d10 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 >> 172.21.15.94:0/1817106866 conn(0x7fbcbc0ac590 msgr2=0x7fbcbc0adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 shutdown_connections 2024-03-06T07:23:15.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.081+0000 7fbcc3c9a640 1 -- 172.21.15.94:0/1817106866 wait complete. 2024-03-06T07:23:15.093 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:15.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:15.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:15.190 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:15.190 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:23:15.190 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:15.206 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:23:15.206 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:23:15.206 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:15.223 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:23:15.223 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:23:15.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:15.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.344+0000 7f7ecfbbc640 1 Processor -- start 2024-03-06T07:23:15.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.345+0000 7f7ecfbbc640 1 -- start start 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.345+0000 7f7ecfbbc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7ec81550c0 0x7f7ec81554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.345+0000 7f7ecfbbc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80aa660 con 0x7f7ec80aa180 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.345+0000 7f7ecfbbc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80aa940 con 0x7f7ec81559e0 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.345+0000 7f7ecfbbc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec80c4610 con 0x7f7ec81550c0 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.345+0000 7f7ece159640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7ec80aa180 0x7f7ec80c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47302/0 (socket says 172.21.15.94:47302) 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.345+0000 7f7ece159640 1 -- 172.21.15.94:0/3109467384 learned_addr learned my addr 172.21.15.94:0/3109467384 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101547831 0 0) 0x7f7ec80aa660 con 0x7f7ec80aa180 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4003520 con 0x7f7ec80aa180 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd958640 1 --2- 172.21.15.94:0/3109467384 >> v2:172.21.15.145:3300/0 conn(0x7f7ec81550c0 0x7f7ec81554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848311349 0 0) 0x7f7eb4003520 con 0x7f7ec80aa180 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 >> v2:172.21.15.145:3300/0 conn(0x7f7ec81550c0 msgr2=0x7f7ec81554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 --2- 172.21.15.94:0/3109467384 >> v2:172.21.15.145:3300/0 conn(0x7f7ec81550c0 0x7f7ec81554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 >> v1:172.21.15.145:6789/0 conn(0x7f7ec81559e0 legacy=0x7f7ec814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80c4860 con 0x7f7ec80aa180 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f7ec00019f0 con 0x7f7ec80aa180 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 >> v1:172.21.15.94:6789/0 conn(0x7f7ec80aa180 legacy=0x7f7ec80c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7eb4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb4004800 0x7f7eb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40052e0 0x7f7eb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80c4610 con 0x7f7eb40039b0 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb400c1b0 con 0x7f7eb4004800 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb400c3b0 con 0x7f7eb40052e0 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd958640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7eb4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd157640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb4004800 0x7f7eb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ece159640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40052e0 0x7f7eb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd958640 1 -- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40052e0 msgr2=0x7f7eb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd958640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7eb40052e0 0x7f7eb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd958640 1 -- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb4004800 msgr2=0x7f7eb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd958640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb4004800 0x7f7eb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ece159640 1 -- 172.21.15.94:0/3109467384 reap_dead start 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd958640 1 -- 172.21.15.94:0/3109467384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4003520 con 0x7f7eb40039b0 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.346+0000 7f7ecd157640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7eb4004800 0x7f7eb4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.347+0000 7f7ecd958640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7eb4003df0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009ac0 tx=0x7f7eb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f97a51cfb5455e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.347+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc039070 con 0x7f7eb40039b0 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.347+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc02c6b0 con 0x7f7eb40039b0 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.347+0000 7f7ecc956640 1 -- 172.21.15.94:0/3109467384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc009690 con 0x7f7eb40039b0 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.348+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/3109467384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 msgr2=0x7f7eb4003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.348+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7eb4003df0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009ac0 tx=0x7f7eb4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.348+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/3109467384 shutdown_connections 2024-03-06T07:23:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.348+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/3109467384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7eb4003df0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.348+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/3109467384 >> 172.21.15.94:0/3109467384 conn(0x7f7ec80c6710 msgr2=0x7f7ec80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.348+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/3109467384 shutdown_connections 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.348+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/3109467384 wait complete. 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.349+0000 7f7ecfbbc640 1 Processor -- start 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.349+0000 7f7ecfbbc640 1 -- start start 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7ec80703a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8151e00 0x7f7ec8144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ec8144c40 0x7f7ec813e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81554a0 con 0x7f7eb40039b0 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80c1520 con 0x7f7ec8151e00 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8155620 con 0x7f7ec8144c40 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7ec80703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7ec80703a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39938/0 (socket says 172.21.15.94:39938) 2024-03-06T07:23:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 -- 172.21.15.94:0/2261552677 learned_addr learned my addr 172.21.15.94:0/2261552677 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 -- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ec8144c40 msgr2=0x7f7ec813e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ec8144c40 0x7f7ec813e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 -- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8151e00 msgr2=0x7f7ec8144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8151e00 0x7f7ec8144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 -- 172.21.15.94:0/2261552677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec806c520 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecd958640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7ec80703a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc037040 tx=0x7f7ebc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7eba7fc640 1 -- 172.21.15.94:0/2261552677 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc039050 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7eba7fc640 1 -- 172.21.15.94:0/2261552677 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc03a040 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec80c03d0 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7eba7fc640 1 -- 172.21.15.94:0/2261552677 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0044e0 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.350+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec80a5cb0 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.351+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec804f330 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.352+0000 7f7eba7fc640 1 -- 172.21.15.94:0/2261552677 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ebc047020 con 0x7f7eb40039b0 2024-03-06T07:23:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.352+0000 7f7eba7fc640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7eac06b500 0x7f7eac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.352+0000 7f7eba7fc640 1 -- 172.21.15.94:0/2261552677 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f7ebc0ab950 con 0x7f7eb40039b0 2024-03-06T07:23:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.353+0000 7f7ecd157640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7eac06b500 0x7f7eac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.353+0000 7f7ecd157640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7eac06b500 0x7f7eac06d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7ec81536b0 tx=0x7f7ec4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.356+0000 7f7eba7fc640 1 -- 172.21.15.94:0/2261552677 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ebc07a920 con 0x7f7eb40039b0 2024-03-06T07:23:15.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.562+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ec8068450 con 0x7f7eb40039b0 2024-03-06T07:23:15.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.563+0000 7f7eba7fc640 1 -- 172.21.15.94:0/2261552677 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+3707 (secure 0 0 0) 0x7f7ebc07a220 con 0x7f7eb40039b0 2024-03-06T07:23:15.565 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:15.565 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":115,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7321,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1356561672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1356561672},{"type":"v1","addr":"172.21.15.145:6837","nonce":1356561672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":7654,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/411838683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":411838683},{"type":"v1","addr":"172.21.15.145:6836","nonce":411838683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1458830160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1458830160},{"type":"v1","addr":"172.21.15.94:6835","nonce":1458830160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101},{"gid":8112,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1268765110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1268765110},{"type":"v1","addr":"172.21.15.94:6834","nonce":1268765110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[]} 2024-03-06T07:23:15.565 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 115 2024-03-06T07:23:15.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.564+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7eac06b500 msgr2=0x7f7eac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.564+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7eac06b500 0x7f7eac06d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7ec81536b0 tx=0x7f7ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.564+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 msgr2=0x7f7ec80703a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.564+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7ec80703a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc037040 tx=0x7f7ebc004730 comp rx=0 tx=0).stop 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 shutdown_connections 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7eac06b500 0x7f7eac06d9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ec8144c40 0x7f7ec813e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ec8151e00 0x7f7ec8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 --2- 172.21.15.94:0/2261552677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7eb40039b0 0x7f7ec80703a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 >> 172.21.15.94:0/2261552677 conn(0x7f7ec80c6710 msgr2=0x7f7ec80c16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 shutdown_connections 2024-03-06T07:23:15.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.565+0000 7f7ecfbbc640 1 -- 172.21.15.94:0/2261552677 wait complete. 2024-03-06T07:23:15.578 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:15.578 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:15.578 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:15.578 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:15.578 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:15.578 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:15.579 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:15.579 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:15.579 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:15.579 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:15.579 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:15.579 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:15.579 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:15.579 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:15.580 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:15.580 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:15.587 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:15.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:15.588 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:15.588 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:15.591 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:15.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:15.592 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:15.592 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:15.630 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:15.630 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:15.631 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:15.631 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:15.634 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:15.634 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:15.634 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:15.635 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:15.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:15.706 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:23:15.707 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:23:15.719 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:23:15.720 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:23:15.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.739+0000 7f538c33a640 1 Processor -- start 2024-03-06T07:23:15.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.739+0000 7f538c33a640 1 -- start start 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538c33a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53840c69f0 0x7f53840c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538c33a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53840c8950 0x7f53840acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538c33a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384155130 con 0x7f53840ad2b0 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538c33a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5384155440 con 0x7f53840c69f0 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538c33a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53840c5d50 con 0x7f53840c8950 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538a0d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53840c69f0 0x7f53840c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538a0d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53840c69f0 0x7f53840c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39978/0 (socket says 172.21.15.94:39978) 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538a0d6640 1 -- 172.21.15.94:0/739999334 learned_addr learned my addr 172.21.15.94:0/739999334 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:15.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f53898d5640 1 --2- 172.21.15.94:0/739999334 >> v2:172.21.15.145:3300/0 conn(0x7f53840c8950 0x7f53840acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538a0d6640 1 -- 172.21.15.94:0/739999334 >> v2:172.21.15.145:3300/0 conn(0x7f53840c8950 msgr2=0x7f53840acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538a0d6640 1 --2- 172.21.15.94:0/739999334 >> v2:172.21.15.145:3300/0 conn(0x7f53840c8950 0x7f53840acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538a0d6640 1 -- 172.21.15.94:0/739999334 >> v1:172.21.15.145:6789/0 conn(0x7f53840ad2b0 legacy=0x7f53841558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.740+0000 7f538a0d6640 1 -- 172.21.15.94:0/739999334 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840c6030 con 0x7f53840c69f0 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.741+0000 7f538a0d6640 1 --2- 172.21.15.94:0/739999334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53840c6dd0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f53840c7430 tx=0x7f538002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81808eddb846f4e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.741+0000 7f53890d4640 1 -- 172.21.15.94:0/739999334 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538002e070 con 0x7f53840c69f0 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.741+0000 7f53890d4640 1 -- 172.21.15.94:0/739999334 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380037070 con 0x7f53840c69f0 2024-03-06T07:23:15.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.741+0000 7f53890d4640 1 -- 172.21.15.94:0/739999334 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380031a00 con 0x7f53840c69f0 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 -- 172.21.15.94:0/739999334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 msgr2=0x7f53840c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 --2- 172.21.15.94:0/739999334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53840c6dd0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f53840c7430 tx=0x7f538002b880 comp rx=0 tx=0).stop 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 -- 172.21.15.94:0/739999334 shutdown_connections 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 --2- 172.21.15.94:0/739999334 >> v2:172.21.15.145:3300/0 conn(0x7f53840c8950 0x7f53840acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 --2- 172.21.15.94:0/739999334 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53840c6dd0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 -- 172.21.15.94:0/739999334 >> 172.21.15.94:0/739999334 conn(0x7f53840cae70 msgr2=0x7f53840cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 -- 172.21.15.94:0/739999334 shutdown_connections 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.742+0000 7f538c33a640 1 -- 172.21.15.94:0/739999334 wait complete. 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.743+0000 7f538c33a640 1 Processor -- start 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.743+0000 7f538c33a640 1 -- start start 2024-03-06T07:23:15.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53840ad2b0 0x7f538406c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53841ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840c8950 0x7f538414b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5384152a20 con 0x7f53840c69f0 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5384155fe0 con 0x7f53840ad2b0 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5384152ba0 con 0x7f53840c8950 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53841ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53841ee0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39980/0 (socket says 172.21.15.94:39980) 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 -- 172.21.15.94:0/4132540317 learned_addr learned my addr 172.21.15.94:0/4132540317 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538a0d6640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53840ad2b0 0x7f538406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538a8d7640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840c8950 0x7f538414b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 -- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840c8950 msgr2=0x7f538414b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840c8950 0x7f538414b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 -- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53840ad2b0 msgr2=0x7f538406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53840ad2b0 0x7f538406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 -- 172.21.15.94:0/4132540317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840c2480 con 0x7f53840c69f0 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538a0d6640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53840ad2b0 0x7f538406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f53898d5640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53841ee0d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f537800c820 tx=0x7f537800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f5376ffd640 1 -- 172.21.15.94:0/4132540317 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378015070 con 0x7f53840c69f0 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f5376ffd640 1 -- 172.21.15.94:0/4132540317 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378004220 con 0x7f53840c69f0 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538414cd10 con 0x7f53840c69f0 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f5376ffd640 1 -- 172.21.15.94:0/4132540317 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537800ce10 con 0x7f53840c69f0 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.744+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538406c7c0 con 0x7f53840c69f0 2024-03-06T07:23:15.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.745+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538414ca70 con 0x7f53840c69f0 2024-03-06T07:23:15.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.746+0000 7f5376ffd640 1 -- 172.21.15.94:0/4132540317 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5378007830 con 0x7f53840c69f0 2024-03-06T07:23:15.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.746+0000 7f5376ffd640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536806b4d0 0x7f536806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.746+0000 7f5376ffd640 1 -- 172.21.15.94:0/4132540317 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f5378089f20 con 0x7f53840c69f0 2024-03-06T07:23:15.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.746+0000 7f538a0d6640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536806b4d0 0x7f536806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.747+0000 7f538a0d6640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536806b4d0 0x7f536806d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5380036000 tx=0x7f538003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.751+0000 7f5376ffd640 1 -- 172.21.15.94:0/4132540317 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5378058ef0 con 0x7f53840c69f0 2024-03-06T07:23:15.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:15.936+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f53840c2650 con 0x7f53840c69f0 2024-03-06T07:23:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.791+0000 7f5376ffd640 1 -- 172.21.15.94:0/4132540317 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v131) v1 ==== 97+0+0 (secure 0 0 0) 0x7f53780587f0 con 0x7f53840c69f0 2024-03-06T07:23:16.793 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:23:16.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536806b4d0 msgr2=0x7f536806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536806b4d0 0x7f536806d990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5380036000 tx=0x7f538003d040 comp rx=0 tx=0).stop 2024-03-06T07:23:16.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 msgr2=0x7f53841ee0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53841ee0d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f537800c820 tx=0x7f537800cce0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 shutdown_connections 2024-03-06T07:23:16.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f536806b4d0 0x7f536806d990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53840c8950 0x7f538414b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53840c69f0 0x7f53841ee0d0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 --2- 172.21.15.94:0/4132540317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53840ad2b0 0x7f538406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 >> 172.21.15.94:0/4132540317 conn(0x7f53840cae70 msgr2=0x7f5384151760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 shutdown_connections 2024-03-06T07:23:16.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.794+0000 7f538c33a640 1 -- 172.21.15.94:0/4132540317 wait complete. 2024-03-06T07:23:16.809 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:16.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.913+0000 7f759209a640 1 Processor -- start 2024-03-06T07:23:16.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.913+0000 7f759209a640 1 -- start start 2024-03-06T07:23:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.913+0000 7f759209a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 0x7f758c0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f759209a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758c155bb0 con 0x7f758c0aa180 2024-03-06T07:23:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f759209a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758c0aa660 con 0x7f758c1550c0 2024-03-06T07:23:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f759209a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0c4610 con 0x7f758c14f310 2024-03-06T07:23:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f758c0aa180 0x7f758c14ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47318/0 (socket says 172.21.15.94:47318) 2024-03-06T07:23:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758affd640 1 -- 172.21.15.94:0/3924816937 learned_addr learned my addr 172.21.15.94:0/3924816937 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102636967 0 0) 0x7f758c155bb0 con 0x7f758c0aa180 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7574003440 con 0x7f758c0aa180 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758bfff640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 0x7f758c0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1692504647 0 0) 0x7f7574003440 con 0x7f758c0aa180 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 msgr2=0x7f758c0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 0x7f758c0c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 >> v1:172.21.15.145:6789/0 conn(0x7f758c1550c0 legacy=0x7f758c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c0aa940 con 0x7f758c0aa180 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f7580001670 con 0x7f758c0aa180 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 >> v1:172.21.15.94:6789/0 conn(0x7f758c0aa180 legacy=0x7f758c14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7574003ad0 0x7f7574003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f757400ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 0x7f757400d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0c4610 con 0x7f7574008970 2024-03-06T07:23:16.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757400de50 con 0x7f7574003ad0 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757400e050 con 0x7f758c14f310 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.914+0000 7f758affd640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f757400ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758bfff640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 0x7f757400d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758b7fe640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7574003ad0 0x7f7574003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758affd640 1 -- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 msgr2=0x7f757400d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758affd640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f310 0x7f757400d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758affd640 1 -- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7574003ad0 msgr2=0x7f7574003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758affd640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7574003ad0 0x7f7574003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758affd640 1 -- 172.21.15.94:0/3924816937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7574003440 con 0x7f7574008970 2024-03-06T07:23:16.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758affd640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f757400ae00 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f758c0aa590 tx=0x7f7574003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a218c45c07a68a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580007d30 con 0x7f7574008970 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7580030070 con 0x7f7574008970 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.915+0000 7f758a7fc640 1 -- 172.21.15.94:0/3924816937 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580005d30 con 0x7f7574008970 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.916+0000 7f759209a640 1 -- 172.21.15.94:0/3924816937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 msgr2=0x7f757400ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.916+0000 7f759209a640 1 --2- 172.21.15.94:0/3924816937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f757400ae00 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f758c0aa590 tx=0x7f7574003730 comp rx=0 tx=0).stop 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.916+0000 7f758bfff640 1 -- 172.21.15.94:0/3924816937 reap_dead start 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.916+0000 7f759209a640 1 -- 172.21.15.94:0/3924816937 shutdown_connections 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.916+0000 7f759209a640 1 -- 172.21.15.94:0/3924816937 >> 172.21.15.94:0/3924816937 conn(0x7f758c0c6710 msgr2=0x7f758c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.916+0000 7f759209a640 1 -- 172.21.15.94:0/3924816937 shutdown_connections 2024-03-06T07:23:16.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.916+0000 7f759209a640 1 -- 172.21.15.94:0/3924816937 wait complete. 2024-03-06T07:23:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.917+0000 7f759209a640 1 Processor -- start 2024-03-06T07:23:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.917+0000 7f759209a640 1 -- start start 2024-03-06T07:23:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f759209a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f758c0a8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f759209a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f7a0 0x7f758c151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f759209a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f758c151590 0x7f758c0a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f759209a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0c1490 con 0x7f7574008970 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f759209a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0c1310 con 0x7f758c151590 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f759209a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0c3210 con 0x7f758c14f7a0 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f758c0a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f758c0a8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39986/0 (socket says 172.21.15.94:39986) 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 -- 172.21.15.94:0/2466864445 learned_addr learned my addr 172.21.15.94:0/2466864445 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 -- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f7a0 msgr2=0x7f758c151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f7a0 0x7f758c151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 -- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f758c151590 msgr2=0x7f758c0a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f758c151590 0x7f758c0a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 -- 172.21.15.94:0/2466864445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c0a91a0 con 0x7f7574008970 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.918+0000 7f758b7fe640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f758c0a8990 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f757c00e490 tx=0x7f757c00e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.919+0000 7f7573fff640 1 -- 172.21.15.94:0/2466864445 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0052e0 con 0x7f7574008970 2024-03-06T07:23:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.919+0000 7f7573fff640 1 -- 172.21.15.94:0/2466864445 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c018070 con 0x7f7574008970 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.919+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758c1ea5c0 con 0x7f7574008970 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.919+0000 7f7573fff640 1 -- 172.21.15.94:0/2466864445 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c003830 con 0x7f7574008970 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.919+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758c0a6960 con 0x7f7574008970 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.920+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758c04f330 con 0x7f7574008970 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.920+0000 7f7573fff640 1 -- 172.21.15.94:0/2466864445 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f757c0039d0 con 0x7f7574008970 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.920+0000 7f7573fff640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f757806b500 0x7f757806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.920+0000 7f7573fff640 1 -- 172.21.15.94:0/2466864445 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f757c089bc0 con 0x7f7574008970 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.921+0000 7f758affd640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f757806b500 0x7f757806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.921+0000 7f758affd640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f757806b500 0x7f757806d9c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7580034000 tx=0x7f7580039040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:16.925+0000 7f7573fff640 1 -- 172.21.15.94:0/2466864445 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f757c058c90 con 0x7f7574008970 2024-03-06T07:23:17.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.162+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f758c068450 con 0x7f7574008970 2024-03-06T07:23:17.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.162+0000 7f7573fff640 1 -- 172.21.15.94:0/2466864445 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f757c058590 con 0x7f7574008970 2024-03-06T07:23:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:17.164 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:23:17.164 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:23:17.165 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.166 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:23:17.167 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.168 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.171 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:23:17.172 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.173 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.174 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.175 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.176 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:23:17.177 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:23:17.178 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f757806b500 msgr2=0x7f757806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f757806b500 0x7f757806d9c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7580034000 tx=0x7f7580039040 comp rx=0 tx=0).stop 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 msgr2=0x7f758c0a8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f758c0a8990 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f757c00e490 tx=0x7f757c00e950 comp rx=0 tx=0).stop 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 shutdown_connections 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f757806b500 0x7f757806d9c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f758c151590 0x7f758c0a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f758c14f7a0 0x7f758c151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 --2- 172.21.15.94:0/2466864445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7574008970 0x7f758c0a8990 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 >> 172.21.15.94:0/2466864445 conn(0x7f758c0c6710 msgr2=0x7f758c14eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 shutdown_connections 2024-03-06T07:23:17.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.164+0000 7f759209a640 1 -- 172.21.15.94:0/2466864445 wait complete. 2024-03-06T07:23:17.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:17.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.281+0000 7f82b3ecf640 1 Processor -- start 2024-03-06T07:23:17.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.282+0000 7f82b3ecf640 1 -- start start 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.282+0000 7f82b3ecf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f82ac150580 0x7f82ac06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.282+0000 7f82b3ecf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 0x7f82ac06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.282+0000 7f82b3ecf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f82ac0c6000 0x7f82ac1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.282+0000 7f82b3ecf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac153a60 con 0x7f82ac0c6000 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.282+0000 7f82b3ecf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac153d40 con 0x7f82ac150580 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.282+0000 7f82b3ecf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac06c2c0 con 0x7f82ac0c1320 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f82ac0c6000 0x7f82ac1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f82ac0c6000 0x7f82ac1566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40002/0 (socket says 172.21.15.94:40002) 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 -- 172.21.15.94:0/2776788281 learned_addr learned my addr 172.21.15.94:0/2776788281 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 -- 172.21.15.94:0/2776788281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 msgr2=0x7f82ac06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 --2- 172.21.15.94:0/2776788281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 0x7f82ac06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 -- 172.21.15.94:0/2776788281 >> v2:172.21.15.145:3300/0 conn(0x7f82ac150580 msgr2=0x7f82ac06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 --2- 172.21.15.94:0/2776788281 >> v2:172.21.15.145:3300/0 conn(0x7f82ac150580 0x7f82ac06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.283+0000 7f82b246c640 1 -- 172.21.15.94:0/2776788281 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac06c5a0 con 0x7f82ac0c6000 2024-03-06T07:23:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b246c640 1 --2- 172.21.15.94:0/2776788281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac1566b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0a9550 tx=0x7f82a402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58a9dfd88edbee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b0c69640 1 -- 172.21.15.94:0/2776788281 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a402c9e0 con 0x7f82ac0c6000 2024-03-06T07:23:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b0c69640 1 -- 172.21.15.94:0/2776788281 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a402cb80 con 0x7f82ac0c6000 2024-03-06T07:23:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b0c69640 1 -- 172.21.15.94:0/2776788281 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a40343f0 con 0x7f82ac0c6000 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 -- 172.21.15.94:0/2776788281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 msgr2=0x7f82ac1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/2776788281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac1566b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0a9550 tx=0x7f82a402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 -- 172.21.15.94:0/2776788281 shutdown_connections 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/2776788281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac1566b0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/2776788281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 0x7f82ac06e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/2776788281 >> v2:172.21.15.145:3300/0 conn(0x7f82ac150580 0x7f82ac06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 -- 172.21.15.94:0/2776788281 >> 172.21.15.94:0/2776788281 conn(0x7f82ac0c6710 msgr2=0x7f82ac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 -- 172.21.15.94:0/2776788281 shutdown_connections 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.284+0000 7f82b3ecf640 1 -- 172.21.15.94:0/2776788281 wait complete. 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.285+0000 7f82b3ecf640 1 Processor -- start 2024-03-06T07:23:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.285+0000 7f82b3ecf640 1 -- start start 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b3ecf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 0x7f82ac149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b3ecf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b3ecf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82ac150580 0x7f82ac14bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b3ecf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0c0dc0 con 0x7f82ac0c6000 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b3ecf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0c0c40 con 0x7f82ac150580 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b3ecf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac156bf0 con 0x7f82ac0c1320 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b146a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b146a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40018/0 (socket says 172.21.15.94:40018) 2024-03-06T07:23:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b146a640 1 -- 172.21.15.94:0/1044571804 learned_addr learned my addr 172.21.15.94:0/1044571804 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b1c6b640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 0x7f82ac149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.286+0000 7f82b246c640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82ac150580 0x7f82ac14bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b146a640 1 -- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 msgr2=0x7f82ac149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b146a640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 0x7f82ac149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b146a640 1 -- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82ac150580 msgr2=0x7f82ac14bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b146a640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82ac150580 0x7f82ac14bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b146a640 1 -- 172.21.15.94:0/1044571804 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac1dc9c0 con 0x7f82ac0c6000 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b146a640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac06ded0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f82a800b4a0 tx=0x7f82a800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82a27fc640 1 -- 172.21.15.94:0/1044571804 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8014070 con 0x7f82ac0c6000 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82a27fc640 1 -- 172.21.15.94:0/1044571804 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a8010610 con 0x7f82ac0c6000 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82a27fc640 1 -- 172.21.15.94:0/1044571804 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8010a00 con 0x7f82ac0c6000 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac1dc6e0 con 0x7f82ac0c6000 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.287+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac1ecdd0 con 0x7f82ac0c6000 2024-03-06T07:23:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.288+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac14bc60 con 0x7f82ac0c6000 2024-03-06T07:23:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.288+0000 7f82a27fc640 1 -- 172.21.15.94:0/1044571804 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82a8017070 con 0x7f82ac0c6000 2024-03-06T07:23:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.289+0000 7f82a27fc640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f829006b500 0x7f829006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.289+0000 7f82a27fc640 1 -- 172.21.15.94:0/1044571804 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f82a80898e0 con 0x7f82ac0c6000 2024-03-06T07:23:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.289+0000 7f82b1c6b640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f829006b500 0x7f829006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.289+0000 7f82b1c6b640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f829006b500 0x7f829006d9c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f829c0098e0 tx=0x7f829c009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.293+0000 7f82a27fc640 1 -- 172.21.15.94:0/1044571804 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82ac14bc60 con 0x7f82ac0c6000 2024-03-06T07:23:17.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.465+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82ac0c5de0 con 0x7f82ac0c6000 2024-03-06T07:23:17.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.465+0000 7f82a27fc640 1 -- 172.21.15.94:0/1044571804 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+13174 (secure 0 0 0) 0x7f82a8058360 con 0x7f82ac0c6000 2024-03-06T07:23:17.467 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:17.467 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":131,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:16.789989+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f829006b500 msgr2=0x7f829006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f829006b500 0x7f829006d9c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f829c0098e0 tx=0x7f829c009060 comp rx=0 tx=0).stop 2024-03-06T07:23:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 msgr2=0x7f82ac06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac06ded0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f82a800b4a0 tx=0x7f82a800b960 comp rx=0 tx=0).stop 2024-03-06T07:23:17.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 shutdown_connections 2024-03-06T07:23:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f829006b500 0x7f829006d9c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82ac150580 0x7f82ac14bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82ac0c6000 0x7f82ac06ded0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 --2- 172.21.15.94:0/1044571804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82ac0c1320 0x7f82ac149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 >> 172.21.15.94:0/1044571804 conn(0x7f82ac0c6710 msgr2=0x7f82ac1515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 shutdown_connections 2024-03-06T07:23:17.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.467+0000 7f82b3ecf640 1 -- 172.21.15.94:0/1044571804 wait complete. 2024-03-06T07:23:17.480 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:17.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:17.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.581+0000 7f59cdec9640 1 Processor -- start 2024-03-06T07:23:17.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59cdec9640 1 -- start start 2024-03-06T07:23:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59cdec9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f59c806d750 0x7f59c80732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59cdec9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806db30 0x7f59c80c1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59cdec9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c8156060 con 0x7f59c80c3a70 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59cdec9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c8155b30 con 0x7f59c806d750 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59cdec9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80a8430 con 0x7f59c806db30 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59c77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f59c806d750 0x7f59c80732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59c6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f59c80c3a70 0x7f59c8072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47330/0 (socket says 172.21.15.94:47330) 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.582+0000 7f59c6ffd640 1 -- 172.21.15.94:0/1712648751 learned_addr learned my addr 172.21.15.94:0/1712648751 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c7fff640 1 --2- 172.21.15.94:0/1712648751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806db30 0x7f59c80c1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c67fc640 1 -- 172.21.15.94:0/1712648751 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355172715 0 0) 0x7f59c8156060 con 0x7f59c80c3a70 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c67fc640 1 -- 172.21.15.94:0/1712648751 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b0003520 con 0x7f59c80c3a70 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c77fe640 1 -- 172.21.15.94:0/1712648751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806db30 msgr2=0x7f59c80c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c77fe640 1 --2- 172.21.15.94:0/1712648751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806db30 0x7f59c80c1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c77fe640 1 -- 172.21.15.94:0/1712648751 >> v1:172.21.15.94:6789/0 conn(0x7f59c80c3a70 legacy=0x7f59c8072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c77fe640 1 -- 172.21.15.94:0/1712648751 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8155d80 con 0x7f59c806d750 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.583+0000 7f59c77fe640 1 --2- 172.21.15.94:0/1712648751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c806d750 0x7f59c80732a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f59b80097a0 tx=0x7f59b802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7527c5b7b7ae0f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59c67fc640 1 -- 172.21.15.94:0/1712648751 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8038070 con 0x7f59c806d750 2024-03-06T07:23:17.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59c67fc640 1 -- 172.21.15.94:0/1712648751 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8031700 con 0x7f59c806d750 2024-03-06T07:23:17.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59c67fc640 1 -- 172.21.15.94:0/1712648751 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8031ad0 con 0x7f59c806d750 2024-03-06T07:23:17.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 -- 172.21.15.94:0/1712648751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c806d750 msgr2=0x7f59c80732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 --2- 172.21.15.94:0/1712648751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c806d750 0x7f59c80732a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f59b80097a0 tx=0x7f59b802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 -- 172.21.15.94:0/1712648751 shutdown_connections 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 --2- 172.21.15.94:0/1712648751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806db30 0x7f59c80c1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 --2- 172.21.15.94:0/1712648751 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c806d750 0x7f59c80732a0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 -- 172.21.15.94:0/1712648751 >> 172.21.15.94:0/1712648751 conn(0x7f59c80c5560 msgr2=0x7f59c80c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 -- 172.21.15.94:0/1712648751 shutdown_connections 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.584+0000 7f59cdec9640 1 -- 172.21.15.94:0/1712648751 wait complete. 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.585+0000 7f59cdec9640 1 Processor -- start 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.585+0000 7f59cdec9640 1 -- start start 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59cdec9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806d750 0x7f59c806ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59cdec9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59c806db30 0x7f59c806f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59cdec9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c80c3a70 0x7f59c80ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59cdec9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80a7e50 con 0x7f59c80c3a70 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59cdec9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80a7cd0 con 0x7f59c806db30 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59cdec9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80a7fd0 con 0x7f59c806d750 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c80c3a70 0x7f59c80ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c80c3a70 0x7f59c80ad6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40034/0 (socket says 172.21.15.94:40034) 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c7fff640 1 -- 172.21.15.94:0/2135925237 learned_addr learned my addr 172.21.15.94:0/2135925237 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c77fe640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806d750 0x7f59c806ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c6ffd640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59c806db30 0x7f59c806f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c7fff640 1 -- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806d750 msgr2=0x7f59c806ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c7fff640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806d750 0x7f59c806ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c7fff640 1 -- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59c806db30 msgr2=0x7f59c806f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.586+0000 7f59c7fff640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59c806db30 0x7f59c806f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59c7fff640 1 -- 172.21.15.94:0/2135925237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c81eba70 con 0x7f59c80c3a70 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59c77fe640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806d750 0x7f59c806ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59c6ffd640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59c806db30 0x7f59c806f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:17.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59c7fff640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c80c3a70 0x7f59c80ad6c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007d80 tx=0x7f59c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59affff640 1 -- 172.21.15.94:0/2135925237 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0014420 con 0x7f59c80c3a70 2024-03-06T07:23:17.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59affff640 1 -- 172.21.15.94:0/2135925237 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0019070 con 0x7f59c80c3a70 2024-03-06T07:23:17.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c806e720 con 0x7f59c80c3a70 2024-03-06T07:23:17.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59affff640 1 -- 172.21.15.94:0/2135925237 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0014800 con 0x7f59c80c3a70 2024-03-06T07:23:17.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.587+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c8072a70 con 0x7f59c80c3a70 2024-03-06T07:23:17.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.588+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c804f330 con 0x7f59c80c3a70 2024-03-06T07:23:17.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.588+0000 7f59affff640 1 -- 172.21.15.94:0/2135925237 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59c00149a0 con 0x7f59c80c3a70 2024-03-06T07:23:17.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.589+0000 7f59affff640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f59b406b4d0 0x7f59b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.589+0000 7f59affff640 1 -- 172.21.15.94:0/2135925237 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f59c008a7e0 con 0x7f59c80c3a70 2024-03-06T07:23:17.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.589+0000 7f59c77fe640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f59b406b4d0 0x7f59b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.589+0000 7f59c77fe640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f59b406b4d0 0x7f59b406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f59b8035000 tx=0x7f59b8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.593+0000 7f59affff640 1 -- 172.21.15.94:0/2135925237 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59c00598b0 con 0x7f59c80c3a70 2024-03-06T07:23:17.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.763+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59c80c87b0 con 0x7f59c80c3a70 2024-03-06T07:23:17.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.764+0000 7f59affff640 1 -- 172.21.15.94:0/2135925237 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59c00591b0 con 0x7f59c80c3a70 2024-03-06T07:23:17.766 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f59b406b4d0 msgr2=0x7f59b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f59b406b4d0 0x7f59b406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f59b8035000 tx=0x7f59b8008030 comp rx=0 tx=0).stop 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c80c3a70 msgr2=0x7f59c80ad6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c80c3a70 0x7f59c80ad6c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007d80 tx=0x7f59c000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 shutdown_connections 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f59b406b4d0 0x7f59b406d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f59c80c3a70 0x7f59c80ad6c0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59c806db30 0x7f59c806f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 --2- 172.21.15.94:0/2135925237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59c806d750 0x7f59c806ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 >> 172.21.15.94:0/2135925237 conn(0x7f59c80c5560 msgr2=0x7f59c8151130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 shutdown_connections 2024-03-06T07:23:17.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.766+0000 7f59cdec9640 1 -- 172.21.15.94:0/2135925237 wait complete. 2024-03-06T07:23:17.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:17.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.881+0000 7f92f7ca4640 1 Processor -- start 2024-03-06T07:23:17.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.881+0000 7f92f7ca4640 1 -- start start 2024-03-06T07:23:17.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f7ca4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f92f00c1320 0x7f92f01566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f7ca4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0153a60 con 0x7f92f00c6000 2024-03-06T07:23:17.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f7ca4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0153d40 con 0x7f92f0150580 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f7ca4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f006c2c0 con 0x7f92f00c1320 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f523f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f92f00c1320 0x7f92f01566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f6241640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f92f00c6000 0x7f92f0156bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47334/0 (socket says 172.21.15.94:47334) 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f523f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f92f00c1320 0x7f92f01566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40036/0 (socket says 172.21.15.94:40036) 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f6241640 1 -- 172.21.15.94:0/3378305919 learned_addr learned my addr 172.21.15.94:0/3378305919 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f523f640 1 -- 172.21.15.94:0/3378305919 >> v1:172.21.15.145:6789/0 conn(0x7f92f0150580 legacy=0x7f92f006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f523f640 1 -- 172.21.15.94:0/3378305919 >> v1:172.21.15.94:6789/0 conn(0x7f92f00c6000 legacy=0x7f92f0156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f4a3e640 1 -- 172.21.15.94:0/3378305919 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723831451 0 0) 0x7f92f0153a60 con 0x7f92f00c6000 2024-03-06T07:23:17.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.882+0000 7f92f523f640 1 -- 172.21.15.94:0/3378305919 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f006c5a0 con 0x7f92f00c1320 2024-03-06T07:23:17.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.883+0000 7f92f523f640 1 --2- 172.21.15.94:0/3378305919 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f00c1320 0x7f92f01566b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f92e00097a0 tx=0x7f92e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d244f941937ff99b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.883+0000 7f92f4a3e640 1 -- 172.21.15.94:0/3378305919 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e00315a0 con 0x7f92f00c1320 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.883+0000 7f92f4a3e640 1 -- 172.21.15.94:0/3378305919 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e0037070 con 0x7f92f00c1320 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.883+0000 7f92f4a3e640 1 -- 172.21.15.94:0/3378305919 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e0031960 con 0x7f92f00c1320 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.884+0000 7f92f7ca4640 1 -- 172.21.15.94:0/3378305919 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f00c1320 msgr2=0x7f92f01566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.884+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/3378305919 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f00c1320 0x7f92f01566b0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f92e00097a0 tx=0x7f92e002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.884+0000 7f92f7ca4640 1 -- 172.21.15.94:0/3378305919 shutdown_connections 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.884+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/3378305919 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f00c1320 0x7f92f01566b0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.884+0000 7f92f7ca4640 1 -- 172.21.15.94:0/3378305919 >> 172.21.15.94:0/3378305919 conn(0x7f92f00c6710 msgr2=0x7f92f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.884+0000 7f92f7ca4640 1 -- 172.21.15.94:0/3378305919 shutdown_connections 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.884+0000 7f92f7ca4640 1 -- 172.21.15.94:0/3378305919 wait complete. 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.885+0000 7f92f7ca4640 1 Processor -- start 2024-03-06T07:23:17.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.885+0000 7f92f7ca4640 1 -- start start 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f7ca4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92f00c1320 0x7f92f01ec440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f7ca4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92f00c6000 0x7f92f00c2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f7ca4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f0150580 0x7f92f00c2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f7ca4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f006e420 con 0x7f92f0150580 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f7ca4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0150960 con 0x7f92f00c6000 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f7ca4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f006e5a0 con 0x7f92f00c1320 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f0150580 0x7f92f00c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f0150580 0x7f92f00c2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40038/0 (socket says 172.21.15.94:40038) 2024-03-06T07:23:17.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 -- 172.21.15.94:0/2335870114 learned_addr learned my addr 172.21.15.94:0/2335870114 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f5a40640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92f00c1320 0x7f92f01ec440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f523f640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92f00c6000 0x7f92f00c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 -- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92f00c1320 msgr2=0x7f92f01ec440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92f00c1320 0x7f92f01ec440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 -- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92f00c6000 msgr2=0x7f92f00c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92f00c6000 0x7f92f00c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 -- 172.21.15.94:0/2335870114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f00a8160 con 0x7f92f0150580 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f6241640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f0150580 0x7f92f00c2970 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f92e800c810 tx=0x7f92e800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92e67fc640 1 -- 172.21.15.94:0/2335870114 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8018070 con 0x7f92f0150580 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92e67fc640 1 -- 172.21.15.94:0/2335870114 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e800a640 con 0x7f92f0150580 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92e67fc640 1 -- 172.21.15.94:0/2335870114 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e800aa30 con 0x7f92f0150580 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.886+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f0155280 con 0x7f92f0150580 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.887+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f00a7650 con 0x7f92f0150580 2024-03-06T07:23:17.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.887+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f004f330 con 0x7f92f0150580 2024-03-06T07:23:17.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.888+0000 7f92e67fc640 1 -- 172.21.15.94:0/2335870114 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92e801b070 con 0x7f92f0150580 2024-03-06T07:23:17.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.888+0000 7f92e67fc640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92d406b4d0 0x7f92d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.888+0000 7f92e67fc640 1 -- 172.21.15.94:0/2335870114 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f92e80895c0 con 0x7f92f0150580 2024-03-06T07:23:17.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.888+0000 7f92f5a40640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92d406b4d0 0x7f92d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.889+0000 7f92f5a40640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92d406b4d0 0x7f92d406d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f92ec006fd0 tx=0x7f92ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:17.892+0000 7f92e67fc640 1 -- 172.21.15.94:0/2335870114 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92e8058690 con 0x7f92f0150580 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.065+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f92f00c3a40 con 0x7f92f0150580 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.065+0000 7f92e67fc640 1 -- 172.21.15.94:0/2335870114 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v131) v1 ==== 84+0+65 (secure 0 0 0) 0x7f92e8057f90 con 0x7f92f0150580 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:23:18.067 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92d406b4d0 msgr2=0x7f92d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92d406b4d0 0x7f92d406d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f92ec006fd0 tx=0x7f92ec008040 comp rx=0 tx=0).stop 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f0150580 msgr2=0x7f92f00c2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f0150580 0x7f92f00c2970 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f92e800c810 tx=0x7f92e800ccd0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 shutdown_connections 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f92d406b4d0 0x7f92d406d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92f0150580 0x7f92f00c2970 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f92f00c6000 0x7f92f00c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 --2- 172.21.15.94:0/2335870114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92f00c1320 0x7f92f01ec440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 >> 172.21.15.94:0/2335870114 conn(0x7f92f00c6710 msgr2=0x7f92f014e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 shutdown_connections 2024-03-06T07:23:18.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.067+0000 7f92f7ca4640 1 -- 172.21.15.94:0/2335870114 wait complete. 2024-03-06T07:23:18.079 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:18.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:18.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.181+0000 7f9399d3a640 1 Processor -- start 2024-03-06T07:23:18.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.181+0000 7f9399d3a640 1 -- start start 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9399d3a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93940c1860 0x7f939406bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9399d3a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f939406c510 0x7f9394158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9399d3a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9394155820 con 0x7f93941527a0 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9399d3a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9394158820 con 0x7f939406c510 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9399d3a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93940c2b10 con 0x7f93940c1860 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9393fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f939406c510 0x7f9394158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9393fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f939406c510 0x7f9394158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40048/0 (socket says 172.21.15.94:40048) 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9393fff640 1 -- 172.21.15.94:0/2581885720 learned_addr learned my addr 172.21.15.94:0/2581885720 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:18.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9392ffd640 1 --2- 172.21.15.94:0/2581885720 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93940c1860 0x7f939406bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9393fff640 1 -- 172.21.15.94:0/2581885720 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93940c1860 msgr2=0x7f939406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9393fff640 1 --2- 172.21.15.94:0/2581885720 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93940c1860 0x7f939406bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9393fff640 1 -- 172.21.15.94:0/2581885720 >> v1:172.21.15.145:6789/0 conn(0x7f93941527a0 legacy=0x7f9394155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.182+0000 7f9393fff640 1 -- 172.21.15.94:0/2581885720 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394155b00 con 0x7f939406c510 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.183+0000 7f9393fff640 1 --2- 172.21.15.94:0/2581885720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939406c510 0x7f9394158cb0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f938c009e50 tx=0x7f938c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d773fe9922d3925d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.183+0000 7f93927fc640 1 -- 172.21.15.94:0/2581885720 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c02ca70 con 0x7f939406c510 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.183+0000 7f93927fc640 1 -- 172.21.15.94:0/2581885720 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c02cc10 con 0x7f939406c510 2024-03-06T07:23:18.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.183+0000 7f93927fc640 1 -- 172.21.15.94:0/2581885720 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c0343f0 con 0x7f939406c510 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 -- 172.21.15.94:0/2581885720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939406c510 msgr2=0x7f9394158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 --2- 172.21.15.94:0/2581885720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939406c510 0x7f9394158cb0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f938c009e50 tx=0x7f938c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 -- 172.21.15.94:0/2581885720 shutdown_connections 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 --2- 172.21.15.94:0/2581885720 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939406c510 0x7f9394158cb0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 --2- 172.21.15.94:0/2581885720 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93940c1860 0x7f939406bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 -- 172.21.15.94:0/2581885720 >> 172.21.15.94:0/2581885720 conn(0x7f93940c3960 msgr2=0x7f93940c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 -- 172.21.15.94:0/2581885720 shutdown_connections 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.184+0000 7f9399d3a640 1 -- 172.21.15.94:0/2581885720 wait complete. 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.185+0000 7f9399d3a640 1 Processor -- start 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.185+0000 7f9399d3a640 1 -- start start 2024-03-06T07:23:18.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f939406c510 0x7f93941e1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93940c1860 0x7f9394154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93941527a0 0x7f9394153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9394152d00 con 0x7f93940c1860 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9394152b80 con 0x7f93941527a0 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939406db70 con 0x7f939406c510 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93940c1860 0x7f9394154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93940c1860 0x7f9394154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40050/0 (socket says 172.21.15.94:40050) 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 -- 172.21.15.94:0/518982780 learned_addr learned my addr 172.21.15.94:0/518982780 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f93937fe640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f939406c510 0x7f93941e1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9393fff640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93941527a0 0x7f9394153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 -- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f939406c510 msgr2=0x7f93941e1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f939406c510 0x7f93941e1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 -- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93941527a0 msgr2=0x7f9394153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93941527a0 0x7f9394153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 -- 172.21.15.94:0/518982780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939414dc30 con 0x7f93940c1860 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9393fff640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93941527a0 0x7f9394153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9392ffd640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93940c1860 0x7f9394154bc0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f938800e980 tx=0x7f938800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f937bfff640 1 -- 172.21.15.94:0/518982780 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9388017070 con 0x7f93940c1860 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f937bfff640 1 -- 172.21.15.94:0/518982780 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938800ca00 con 0x7f93940c1860 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9394152f50 con 0x7f93940c1860 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f937bfff640 1 -- 172.21.15.94:0/518982780 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938800cdd0 con 0x7f93940c1860 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.186+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93941eb7b0 con 0x7f93940c1860 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.187+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93941eb2e0 con 0x7f93940c1860 2024-03-06T07:23:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.188+0000 7f937bfff640 1 -- 172.21.15.94:0/518982780 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f938801a070 con 0x7f93940c1860 2024-03-06T07:23:18.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.188+0000 7f937bfff640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938006b4d0 0x7f938006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.188+0000 7f937bfff640 1 -- 172.21.15.94:0/518982780 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9388089f70 con 0x7f93940c1860 2024-03-06T07:23:18.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.188+0000 7f93937fe640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938006b4d0 0x7f938006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.189+0000 7f93937fe640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938006b4d0 0x7f938006d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9394156670 tx=0x7f9384009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.192+0000 7f937bfff640 1 -- 172.21.15.94:0/518982780 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9388058970 con 0x7f93940c1860 2024-03-06T07:23:18.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.363+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f939404f330 con 0x7f93940c1860 2024-03-06T07:23:18.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.797+0000 7f937bfff640 1 -- 172.21.15.94:0/518982780 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v132) v1 ==== 138+0+0 (secure 0 0 0) 0x7f938805ca10 con 0x7f93940c1860 2024-03-06T07:23:18.799 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:18.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.799+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938006b4d0 msgr2=0x7f938006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.799+0000 7f9399d3a640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938006b4d0 0x7f938006d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9394156670 tx=0x7f9384009040 comp rx=0 tx=0).stop 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.799+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93940c1860 msgr2=0x7f9394154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.799+0000 7f9399d3a640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93940c1860 0x7f9394154bc0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f938800e980 tx=0x7f938800ee40 comp rx=0 tx=0).stop 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 shutdown_connections 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938006b4d0 0x7f938006d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93941527a0 0x7f9394153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93940c1860 0x7f9394154bc0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 --2- 172.21.15.94:0/518982780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f939406c510 0x7f93941e1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 >> 172.21.15.94:0/518982780 conn(0x7f93940c3960 msgr2=0x7f9394150e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 shutdown_connections 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.800+0000 7f9399d3a640 1 -- 172.21.15.94:0/518982780 wait complete. 2024-03-06T07:23:18.816 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:18.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.924+0000 7f9dfcd65640 1 Processor -- start 2024-03-06T07:23:18.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.924+0000 7f9dfcd65640 1 -- start start 2024-03-06T07:23:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.924+0000 7f9dfcd65640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9df8155c30 0x7f9df806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.924+0000 7f9dfcd65640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80c5380 0x7f9df80c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.924+0000 7f9dfcd65640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df80b0450 con 0x7f9df80b0070 2024-03-06T07:23:18.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.924+0000 7f9dfcd65640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df806cfb0 con 0x7f9df8155c30 2024-03-06T07:23:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.924+0000 7f9dfcd65640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df806c020 con 0x7f9df80c5380 2024-03-06T07:23:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9deffff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9df8155c30 0x7f9df806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9df6d9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80c5380 0x7f9df80c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9df659c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f9df80b0070 0x7f9df80c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:34462/0 (socket says 172.21.15.94:34462) 2024-03-06T07:23:18.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9df659c640 1 -- 172.21.15.94:0/1235770976 learned_addr learned my addr 172.21.15.94:0/1235770976 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9deffff640 1 -- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80c5380 msgr2=0x7f9df80c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9deffff640 1 --2- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80c5380 0x7f9df80c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9deffff640 1 -- 172.21.15.94:0/1235770976 >> v1:172.21.15.145:6789/0 conn(0x7f9df80b0070 legacy=0x7f9df80c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9deffff640 1 -- 172.21.15.94:0/1235770976 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df81560e0 con 0x7f9df8155c30 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.925+0000 7f9df6d9d640 1 --2- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80c5380 0x7f9df80c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.926+0000 7f9deffff640 1 --2- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df8155c30 0x7f9df806ca70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9de00097a0 tx=0x7f9de002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30e9aedcb80816b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.926+0000 7f9df5d9b640 1 -- 172.21.15.94:0/1235770976 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0031780 con 0x7f9df8155c30 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.926+0000 7f9df5d9b640 1 -- 172.21.15.94:0/1235770976 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de0037070 con 0x7f9df8155c30 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.926+0000 7f9df5d9b640 1 -- 172.21.15.94:0/1235770976 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0031b40 con 0x7f9df8155c30 2024-03-06T07:23:18.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.926+0000 7f9dfcd65640 1 -- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df8155c30 msgr2=0x7f9df806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.926+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df8155c30 0x7f9df806ca70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9de00097a0 tx=0x7f9de002b820 comp rx=0 tx=0).stop 2024-03-06T07:23:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.927+0000 7f9dfcd65640 1 -- 172.21.15.94:0/1235770976 shutdown_connections 2024-03-06T07:23:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.927+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80c5380 0x7f9df80c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.927+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/1235770976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df8155c30 0x7f9df806ca70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.927+0000 7f9dfcd65640 1 -- 172.21.15.94:0/1235770976 >> 172.21.15.94:0/1235770976 conn(0x7f9df80ac590 msgr2=0x7f9df80ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.927+0000 7f9dfcd65640 1 -- 172.21.15.94:0/1235770976 shutdown_connections 2024-03-06T07:23:18.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.927+0000 7f9dfcd65640 1 -- 172.21.15.94:0/1235770976 wait complete. 2024-03-06T07:23:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 Processor -- start 2024-03-06T07:23:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 -- start start 2024-03-06T07:23:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80b0070 0x7f9df8158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df80c5380 0x7f9df80c3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9df8155c30 0x7f9df81ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df806c270 con 0x7f9df8155c30 2024-03-06T07:23:18.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80c7ad0 con 0x7f9df80c5380 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.928+0000 7f9dfcd65640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df806c550 con 0x7f9df80b0070 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9df8155c30 0x7f9df81ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9df8155c30 0x7f9df81ee1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48892/0 (socket says 172.21.15.94:48892) 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 -- 172.21.15.94:0/4131108831 learned_addr learned my addr 172.21.15.94:0/4131108831 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df659c640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80b0070 0x7f9df8158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9deffff640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df80c5380 0x7f9df80c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 -- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80b0070 msgr2=0x7f9df8158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80b0070 0x7f9df8158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 -- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df80c5380 msgr2=0x7f9df80c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df80c5380 0x7f9df80c3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 -- 172.21.15.94:0/4131108831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df81ee9e0 con 0x7f9df8155c30 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9df6d9d640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9df8155c30 0x7f9df81ee1a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f9df000efc0 tx=0x7f9df0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9deeffd640 1 -- 172.21.15.94:0/4131108831 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0018030 con 0x7f9df8155c30 2024-03-06T07:23:18.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9deeffd640 1 -- 172.21.15.94:0/4131108831 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df000cde0 con 0x7f9df8155c30 2024-03-06T07:23:18.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df81eec80 con 0x7f9df8155c30 2024-03-06T07:23:18.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9deeffd640 1 -- 172.21.15.94:0/4131108831 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0003bb0 con 0x7f9df8155c30 2024-03-06T07:23:18.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.929+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df81ee6e0 con 0x7f9df8155c30 2024-03-06T07:23:18.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.930+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df804f330 con 0x7f9df8155c30 2024-03-06T07:23:18.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.931+0000 7f9deeffd640 1 -- 172.21.15.94:0/4131108831 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9df0005d10 con 0x7f9df8155c30 2024-03-06T07:23:18.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.931+0000 7f9deeffd640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dd806b4d0 0x7f9dd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.931+0000 7f9deeffd640 1 -- 172.21.15.94:0/4131108831 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f9df008aa80 con 0x7f9df8155c30 2024-03-06T07:23:18.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.932+0000 7f9df659c640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dd806b4d0 0x7f9dd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.932+0000 7f9df659c640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dd806b4d0 0x7f9dd806d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9df8155340 tx=0x7f9de8007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:18.936+0000 7f9deeffd640 1 -- 172.21.15.94:0/4131108831 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9df0022070 con 0x7f9df8155c30 2024-03-06T07:23:19.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.108+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9df806d860 con 0x7f9df8155c30 2024-03-06T07:23:19.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.799+0000 7f9deeffd640 1 -- 172.21.15.94:0/4131108831 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v133) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9df0010040 con 0x7f9df8155c30 2024-03-06T07:23:19.801 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:23:19.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dd806b4d0 msgr2=0x7f9dd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dd806b4d0 0x7f9dd806d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9df8155340 tx=0x7f9de8007e80 comp rx=0 tx=0).stop 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9df8155c30 msgr2=0x7f9df81ee1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9df8155c30 0x7f9df81ee1a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f9df000efc0 tx=0x7f9df0009f90 comp rx=0 tx=0).stop 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 shutdown_connections 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dd806b4d0 0x7f9dd806d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9df8155c30 0x7f9df81ee1a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9df80c5380 0x7f9df80c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 --2- 172.21.15.94:0/4131108831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9df80b0070 0x7f9df8158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 >> 172.21.15.94:0/4131108831 conn(0x7f9df80ac590 msgr2=0x7f9df80c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 shutdown_connections 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.802+0000 7f9dfcd65640 1 -- 172.21.15.94:0/4131108831 wait complete. 2024-03-06T07:23:19.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.932+0000 7f9521888640 1 Processor -- start 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.932+0000 7f9521888640 1 -- start start 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f9521888640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f951c0c7370 0x7f951c0c7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f9521888640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951c155260 con 0x7f951c1527f0 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f9521888640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f951c155540 con 0x7f951c06c0b0 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f9521888640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951c0c8f70 con 0x7f951c0c7370 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f951a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f951c0c7370 0x7f951c0c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f951a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f951c0c7370 0x7f951c0c7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48896/0 (socket says 172.21.15.94:48896) 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f951affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f951c1527f0 0x7f951c0c8860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54304/0 (socket says 172.21.15.94:54304) 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f951a7fc640 1 -- 172.21.15.94:0/1627022011 learned_addr learned my addr 172.21.15.94:0/1627022011 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f951a7fc640 1 -- 172.21.15.94:0/1627022011 >> v1:172.21.15.145:6789/0 conn(0x7f951c06c0b0 legacy=0x7f951c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f951a7fc640 1 -- 172.21.15.94:0/1627022011 >> v1:172.21.15.94:6789/0 conn(0x7f951c1527f0 legacy=0x7f951c0c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f951a7fc640 1 -- 172.21.15.94:0/1627022011 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c155820 con 0x7f951c0c7370 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.933+0000 7f9519ffb640 1 -- 172.21.15.94:0/1627022011 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932935365 0 0) 0x7f951c155260 con 0x7f951c1527f0 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f951a7fc640 1 --2- 172.21.15.94:0/1627022011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c0c7800 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9510009870 tx=0x7f951002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3c8704569a1b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9519ffb640 1 -- 172.21.15.94:0/1627022011 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510035020 con 0x7f951c0c7370 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9519ffb640 1 -- 172.21.15.94:0/1627022011 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9510031870 con 0x7f951c0c7370 2024-03-06T07:23:19.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9519ffb640 1 -- 172.21.15.94:0/1627022011 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510031c40 con 0x7f951c0c7370 2024-03-06T07:23:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9521888640 1 -- 172.21.15.94:0/1627022011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 msgr2=0x7f951c0c7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9521888640 1 --2- 172.21.15.94:0/1627022011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c0c7800 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f9510009870 tx=0x7f951002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9521888640 1 -- 172.21.15.94:0/1627022011 shutdown_connections 2024-03-06T07:23:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9521888640 1 --2- 172.21.15.94:0/1627022011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c0c7800 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9521888640 1 -- 172.21.15.94:0/1627022011 >> 172.21.15.94:0/1627022011 conn(0x7f951c0ac4f0 msgr2=0x7f951c0ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.934+0000 7f9521888640 1 -- 172.21.15.94:0/1627022011 shutdown_connections 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.935+0000 7f9521888640 1 -- 172.21.15.94:0/1627022011 wait complete. 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 Processor -- start 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 -- start start 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f951c06c0b0 0x7f951c14ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c1eb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f951c1527f0 0x7f951c14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c06cbd0 con 0x7f951c0c7370 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951c0be1a0 con 0x7f951c1527f0 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f9521888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f951c06cd50 con 0x7f951c06c0b0 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f951a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c1eb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f951a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c1eb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48898/0 (socket says 172.21.15.94:48898) 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.936+0000 7f951a7fc640 1 -- 172.21.15.94:0/1231044854 learned_addr learned my addr 172.21.15.94:0/1231044854 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:19.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951affd640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f951c06c0b0 0x7f951c14ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951b7fe640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f951c1527f0 0x7f951c14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951a7fc640 1 -- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f951c06c0b0 msgr2=0x7f951c14ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951a7fc640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f951c06c0b0 0x7f951c14ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951a7fc640 1 -- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f951c1527f0 msgr2=0x7f951c14d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951a7fc640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f951c1527f0 0x7f951c14d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951a7fc640 1 -- 172.21.15.94:0/1231044854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c0afee0 con 0x7f951c0c7370 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f951a7fc640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c1eb740 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9510009870 tx=0x7f95100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f9503fff640 1 -- 172.21.15.94:0/1231044854 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510040070 con 0x7f951c0c7370 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f9503fff640 1 -- 172.21.15.94:0/1231044854 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951003a420 con 0x7f951c0c7370 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951c146090 con 0x7f951c0c7370 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f9503fff640 1 -- 172.21.15.94:0/1231044854 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510035040 con 0x7f951c0c7370 2024-03-06T07:23:19.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.937+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951c14e6a0 con 0x7f951c0c7370 2024-03-06T07:23:19.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.938+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951c04f330 con 0x7f951c0c7370 2024-03-06T07:23:19.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.939+0000 7f9503fff640 1 -- 172.21.15.94:0/1231044854 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f951004b060 con 0x7f951c0c7370 2024-03-06T07:23:19.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.939+0000 7f9503fff640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f950806b4d0 0x7f950806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.939+0000 7f9503fff640 1 -- 172.21.15.94:0/1231044854 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f95100ac1c0 con 0x7f951c0c7370 2024-03-06T07:23:19.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.939+0000 7f951affd640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f950806b4d0 0x7f950806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.940+0000 7f951affd640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f950806b4d0 0x7f950806d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f950c000f60 tx=0x7f950c007d20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:19.944+0000 7f9503fff640 1 -- 172.21.15.94:0/1231044854 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f951002f320 con 0x7f951c0c7370 2024-03-06T07:23:20.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.149+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f951c06c490 con 0x7f951c0c7370 2024-03-06T07:23:20.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.823+0000 7f9503fff640 1 -- 172.21.15.94:0/1231044854 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v117) v1 ==== 174+0+0 (secure 0 0 0) 0x7f951007afa0 con 0x7f951c0c7370 2024-03-06T07:23:20.826 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 21 and data pool 22 2024-03-06T07:23:20.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.834+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f950806b4d0 msgr2=0x7f950806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.834+0000 7f9521888640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f950806b4d0 0x7f950806d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f950c000f60 tx=0x7f950c007d20 comp rx=0 tx=0).stop 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.834+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 msgr2=0x7f951c1eb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.834+0000 7f9521888640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c1eb740 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9510009870 tx=0x7f95100046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.835+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 shutdown_connections 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.835+0000 7f9521888640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f950806b4d0 0x7f950806d990 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.835+0000 7f9521888640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f951c1527f0 0x7f951c14d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.835+0000 7f9521888640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f951c0c7370 0x7f951c1eb740 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.835+0000 7f9521888640 1 --2- 172.21.15.94:0/1231044854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f951c06c0b0 0x7f951c14ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.835+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 >> 172.21.15.94:0/1231044854 conn(0x7f951c0ac4f0 msgr2=0x7f951c0c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.836+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 shutdown_connections 2024-03-06T07:23:20.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.837+0000 7f9521888640 1 -- 172.21.15.94:0/1231044854 wait complete. 2024-03-06T07:23:20.854 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:20.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.955+0000 7f98eda39640 1 Processor -- start 2024-03-06T07:23:20.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.955+0000 7f98eda39640 1 -- start start 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98eda39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e806afc0 0x7f98e806f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98eda39640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f98e81536c0 0x7f98e815d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98eda39640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e806b800 con 0x7f98e806b3a0 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98eda39640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e8150a20 con 0x7f98e81536c0 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98eda39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e81565a0 con 0x7f98e806afc0 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98e77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f98e81536c0 0x7f98e815d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98e67fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f98e806b3a0 0x7f98e8152fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54320/0 (socket says 172.21.15.94:54320) 2024-03-06T07:23:20.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98e67fc640 1 -- 172.21.15.94:0/3948059747 learned_addr learned my addr 172.21.15.94:0/3948059747 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98e6ffd640 1 --2- 172.21.15.94:0/3948059747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e806afc0 0x7f98e806f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98e77fe640 1 -- 172.21.15.94:0/3948059747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e806afc0 msgr2=0x7f98e806f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98e77fe640 1 --2- 172.21.15.94:0/3948059747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e806afc0 0x7f98e806f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.957+0000 7f98e77fe640 1 -- 172.21.15.94:0/3948059747 >> v1:172.21.15.94:6789/0 conn(0x7f98e806b3a0 legacy=0x7f98e8152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.956+0000 7f98e5ffb640 1 -- 172.21.15.94:0/3948059747 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205549711 0 0) 0x7f98e806b800 con 0x7f98e806b3a0 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.957+0000 7f98e77fe640 1 -- 172.21.15.94:0/3948059747 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80a70f0 con 0x7f98e81536c0 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.957+0000 7f98e6ffd640 1 --2- 172.21.15.94:0/3948059747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e806afc0 0x7f98e806f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.957+0000 7f98e77fe640 1 --2- 172.21.15.94:0/3948059747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e81536c0 0x7f98e815d930 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f98e00097a0 tx=0x7f98e002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a4d5c9630c96c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.957+0000 7f98e5ffb640 1 -- 172.21.15.94:0/3948059747 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0038070 con 0x7f98e81536c0 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.957+0000 7f98e5ffb640 1 -- 172.21.15.94:0/3948059747 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e0031700 con 0x7f98e81536c0 2024-03-06T07:23:20.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.957+0000 7f98e5ffb640 1 -- 172.21.15.94:0/3948059747 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0031ad0 con 0x7f98e81536c0 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 -- 172.21.15.94:0/3948059747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e81536c0 msgr2=0x7f98e815d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 --2- 172.21.15.94:0/3948059747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e81536c0 0x7f98e815d930 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f98e00097a0 tx=0x7f98e002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 -- 172.21.15.94:0/3948059747 shutdown_connections 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 --2- 172.21.15.94:0/3948059747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e81536c0 0x7f98e815d930 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 --2- 172.21.15.94:0/3948059747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e806afc0 0x7f98e806f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 -- 172.21.15.94:0/3948059747 >> 172.21.15.94:0/3948059747 conn(0x7f98e80c4290 msgr2=0x7f98e80c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 -- 172.21.15.94:0/3948059747 shutdown_connections 2024-03-06T07:23:20.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.958+0000 7f98eda39640 1 -- 172.21.15.94:0/3948059747 wait complete. 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.959+0000 7f98eda39640 1 Processor -- start 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.959+0000 7f98eda39640 1 -- start start 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98eda39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e806afc0 0x7f98e80ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98eda39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e806b3a0 0x7f98e81e9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98eda39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e81536c0 0x7f98e80c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98eda39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e8156db0 con 0x7f98e806afc0 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98eda39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e8156ad0 con 0x7f98e806b3a0 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98eda39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80c7360 con 0x7f98e81536c0 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e806afc0 0x7f98e80ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e806afc0 0x7f98e80ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48916/0 (socket says 172.21.15.94:48916) 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 -- 172.21.15.94:0/2479817658 learned_addr learned my addr 172.21.15.94:0/2479817658 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 -- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e81536c0 msgr2=0x7f98e80c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e81536c0 0x7f98e80c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 -- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e806b3a0 msgr2=0x7f98e81e9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e806b3a0 0x7f98e81e9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 -- 172.21.15.94:0/2479817658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e81e9540 con 0x7f98e806afc0 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98e6ffd640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e806afc0 0x7f98e80ab7e0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f98d800ce00 tx=0x7f98d800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.960+0000 7f98cffff640 1 -- 172.21.15.94:0/2479817658 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d800f890 con 0x7f98e806afc0 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.961+0000 7f98cffff640 1 -- 172.21.15.94:0/2479817658 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d800fa30 con 0x7f98e806afc0 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.961+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e80aa9a0 con 0x7f98e806afc0 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.961+0000 7f98cffff640 1 -- 172.21.15.94:0/2479817658 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d800fe40 con 0x7f98e806afc0 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.961+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e8151d70 con 0x7f98e806afc0 2024-03-06T07:23:20.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.961+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e804f330 con 0x7f98e806afc0 2024-03-06T07:23:20.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.962+0000 7f98cffff640 1 -- 172.21.15.94:0/2479817658 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98d8018400 con 0x7f98e806afc0 2024-03-06T07:23:20.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.962+0000 7f98cffff640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f98d406b4d0 0x7f98d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.963+0000 7f98cffff640 1 -- 172.21.15.94:0/2479817658 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f98d808b0e0 con 0x7f98e806afc0 2024-03-06T07:23:20.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.963+0000 7f98e67fc640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f98d406b4d0 0x7f98d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.963+0000 7f98e67fc640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f98d406b4d0 0x7f98d406d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f98dc007fd0 tx=0x7f98dc007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:20.967+0000 7f98cffff640 1 -- 172.21.15.94:0/2479817658 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98d8014030 con 0x7f98e806afc0 2024-03-06T07:23:21.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.139+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98e8068460 con 0x7f98e806afc0 2024-03-06T07:23:21.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.140+0000 7f98cffff640 1 -- 172.21.15.94:0/2479817658 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f98d800ae20 con 0x7f98e806afc0 2024-03-06T07:23:21.142 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:21.142 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":134,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:20.150517+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:18.382044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:19.127404+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f98d406b4d0 msgr2=0x7f98d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f98d406b4d0 0x7f98d406d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f98dc007fd0 tx=0x7f98dc007960 comp rx=0 tx=0).stop 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e806afc0 msgr2=0x7f98e80ab7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e806afc0 0x7f98e80ab7e0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f98d800ce00 tx=0x7f98d800a650 comp rx=0 tx=0).stop 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 shutdown_connections 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f98d406b4d0 0x7f98d406d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e81536c0 0x7f98e80c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e806b3a0 0x7f98e81e9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 --2- 172.21.15.94:0/2479817658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f98e806afc0 0x7f98e80ab7e0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 >> 172.21.15.94:0/2479817658 conn(0x7f98e80c4290 msgr2=0x7f98e80c4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 shutdown_connections 2024-03-06T07:23:21.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.142+0000 7f98eda39640 1 -- 172.21.15.94:0/2479817658 wait complete. 2024-03-06T07:23:21.155 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:21.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.255+0000 7f2a10756640 1 Processor -- start 2024-03-06T07:23:21.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.255+0000 7f2a10756640 1 -- start start 2024-03-06T07:23:21.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a10756640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2a08150920 0x7f2a080c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a10756640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a08071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a10756640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a08067340 con 0x7f2a080c2e70 2024-03-06T07:23:21.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a10756640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a08150d00 con 0x7f2a08150920 2024-03-06T07:23:21.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a10756640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080c2010 con 0x7f2a0806c720 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0e4f2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2a08150920 0x7f2a080c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0ecf3640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2a080c2e70 0x7f2a081532e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:34484/0 (socket says 172.21.15.94:34484) 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0ecf3640 1 -- 172.21.15.94:0/4266739121 learned_addr learned my addr 172.21.15.94:0/4266739121 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0dcf1640 1 --2- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a08071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0e4f2640 1 -- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 msgr2=0x7f2a08071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0e4f2640 1 --2- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a08071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0e4f2640 1 -- 172.21.15.94:0/4266739121 >> v1:172.21.15.145:6789/0 conn(0x7f2a080c2e70 legacy=0x7f2a081532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0e4f2640 1 -- 172.21.15.94:0/4266739121 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08067060 con 0x7f2a08150920 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.256+0000 7f2a0d4f0640 1 -- 172.21.15.94:0/4266739121 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174724239 0 0) 0x7f2a08067340 con 0x7f2a080c2e70 2024-03-06T07:23:21.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.257+0000 7f2a0e4f2640 1 --2- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 0x7f2a080c2930 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=189d0bddfa786312 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.257+0000 7f2a0d4f0640 1 -- 172.21.15.94:0/4266739121 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a000315a0 con 0x7f2a08150920 2024-03-06T07:23:21.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.257+0000 7f2a0d4f0640 1 -- 172.21.15.94:0/4266739121 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a00037070 con 0x7f2a08150920 2024-03-06T07:23:21.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.257+0000 7f2a0d4f0640 1 -- 172.21.15.94:0/4266739121 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00031960 con 0x7f2a08150920 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 -- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 msgr2=0x7f2a080c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 --2- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 0x7f2a080c2930 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 -- 172.21.15.94:0/4266739121 shutdown_connections 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 --2- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a08071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 --2- 172.21.15.94:0/4266739121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 0x7f2a080c2930 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 -- 172.21.15.94:0/4266739121 >> 172.21.15.94:0/4266739121 conn(0x7f2a080c70b0 msgr2=0x7f2a080c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 -- 172.21.15.94:0/4266739121 shutdown_connections 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.258+0000 7f2a10756640 1 -- 172.21.15.94:0/4266739121 wait complete. 2024-03-06T07:23:21.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.259+0000 7f2a10756640 1 Processor -- start 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.259+0000 7f2a10756640 1 -- start start 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a10756640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a081525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a10756640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a080c2e70 0x7f2a0806b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a10756640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 0x7f2a081ec930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a10756640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08073210 con 0x7f2a080c2e70 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a10756640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08073090 con 0x7f2a08150920 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a10756640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a080c1b40 con 0x7f2a0806c720 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a080c2e70 0x7f2a0806b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a080c2e70 0x7f2a0806b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48926/0 (socket says 172.21.15.94:48926) 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 -- 172.21.15.94:0/2053253371 learned_addr learned my addr 172.21.15.94:0/2053253371 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0e4f2640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a081525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0ecf3640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 0x7f2a081ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 -- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 msgr2=0x7f2a081525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a081525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 -- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 msgr2=0x7f2a081ec930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 0x7f2a081ec930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 -- 172.21.15.94:0/2053253371 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a081ec170 con 0x7f2a080c2e70 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.260+0000 7f2a0dcf1640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a080c2e70 0x7f2a0806b230 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00e9e0 tx=0x7f29fc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.261+0000 7f2a06ffd640 1 -- 172.21.15.94:0/2053253371 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc017070 con 0x7f2a080c2e70 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.261+0000 7f2a06ffd640 1 -- 172.21.15.94:0/2053253371 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc00c9d0 con 0x7f2a080c2e70 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.261+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a081e9760 con 0x7f2a080c2e70 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.261+0000 7f2a06ffd640 1 -- 172.21.15.94:0/2053253371 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00cdc0 con 0x7f2a080c2e70 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.261+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a081e9a40 con 0x7f2a080c2e70 2024-03-06T07:23:21.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.262+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a08074ac0 con 0x7f2a080c2e70 2024-03-06T07:23:21.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.262+0000 7f2a06ffd640 1 -- 172.21.15.94:0/2053253371 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29fc01a070 con 0x7f2a080c2e70 2024-03-06T07:23:21.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.262+0000 7f2a06ffd640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29f406b4d0 0x7f29f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.262+0000 7f2a06ffd640 1 -- 172.21.15.94:0/2053253371 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29fc08a460 con 0x7f2a080c2e70 2024-03-06T07:23:21.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.263+0000 7f2a0e4f2640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29f406b4d0 0x7f29f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.263+0000 7f2a0e4f2640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29f406b4d0 0x7f29f406d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2a00036000 tx=0x7f2a00008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.267+0000 7f2a06ffd640 1 -- 172.21.15.94:0/2053253371 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a08074ac0 con 0x7f2a080c2e70 2024-03-06T07:23:21.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.438+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a0804f330 con 0x7f2a080c2e70 2024-03-06T07:23:21.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.439+0000 7f2a06ffd640 1 -- 172.21.15.94:0/2053253371 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f29fc0590c0 con 0x7f2a080c2e70 2024-03-06T07:23:21.441 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:21.441 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":134,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:20.150517+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:18.382044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:19.127404+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:21.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29f406b4d0 msgr2=0x7f29f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29f406b4d0 0x7f29f406d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2a00036000 tx=0x7f2a00008030 comp rx=0 tx=0).stop 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a080c2e70 msgr2=0x7f2a0806b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a080c2e70 0x7f2a0806b230 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00e9e0 tx=0x7f29fc00eea0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 shutdown_connections 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f29f406b4d0 0x7f29f406d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a08150920 0x7f2a081ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2a080c2e70 0x7f2a0806b230 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 --2- 172.21.15.94:0/2053253371 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0806c720 0x7f2a081525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 >> 172.21.15.94:0/2053253371 conn(0x7f2a080c70b0 msgr2=0x7f2a0814fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 shutdown_connections 2024-03-06T07:23:21.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.441+0000 7f2a10756640 1 -- 172.21.15.94:0/2053253371 wait complete. 2024-03-06T07:23:21.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:21.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.555+0000 7fd9bf156640 1 Processor -- start 2024-03-06T07:23:21.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.555+0000 7fd9bf156640 1 -- start start 2024-03-06T07:23:21.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bf156640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9b80c26c0 0x7fd9b8155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bf156640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd9b80c2aa0 0x7fd9b806bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bf156640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b806c110 con 0x7fd9b80c0470 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bf156640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b806b740 con 0x7fd9b80c26c0 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bf156640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b80670c0 con 0x7fd9b80c2aa0 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bcef2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9b80c26c0 0x7fd9b8155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bcef2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9b80c26c0 0x7fd9b8155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48934/0 (socket says 172.21.15.94:48934) 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bcef2640 1 -- 172.21.15.94:0/3143267599 learned_addr learned my addr 172.21.15.94:0/3143267599 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9b7fff640 1 --2- 172.21.15.94:0/3143267599 >> v2:172.21.15.145:3300/0 conn(0x7fd9b80c2aa0 0x7fd9b806bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bcef2640 1 -- 172.21.15.94:0/3143267599 >> v2:172.21.15.145:3300/0 conn(0x7fd9b80c2aa0 msgr2=0x7fd9b806bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bcef2640 1 --2- 172.21.15.94:0/3143267599 >> v2:172.21.15.145:3300/0 conn(0x7fd9b80c2aa0 0x7fd9b806bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bcef2640 1 -- 172.21.15.94:0/3143267599 >> v1:172.21.15.145:6789/0 conn(0x7fd9b80c0470 legacy=0x7fd9b8152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9bcef2640 1 -- 172.21.15.94:0/3143267599 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b806b0f0 con 0x7fd9b80c26c0 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.556+0000 7fd9b7fff640 1 --2- 172.21.15.94:0/3143267599 >> v2:172.21.15.145:3300/0 conn(0x7fd9b80c2aa0 0x7fd9b806bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:21.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.557+0000 7fd9bcef2640 1 --2- 172.21.15.94:0/3143267599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b8155bd0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fd9b806f590 tx=0x7fd9ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de5a2122653c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.557+0000 7fd9b77fe640 1 -- 172.21.15.94:0/3143267599 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac02e070 con 0x7fd9b80c26c0 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.557+0000 7fd9b77fe640 1 -- 172.21.15.94:0/3143267599 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ac037070 con 0x7fd9b80c26c0 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.557+0000 7fd9b77fe640 1 -- 172.21.15.94:0/3143267599 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac031a00 con 0x7fd9b80c26c0 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 -- 172.21.15.94:0/3143267599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 msgr2=0x7fd9b8155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 --2- 172.21.15.94:0/3143267599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b8155bd0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fd9b806f590 tx=0x7fd9ac02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 -- 172.21.15.94:0/3143267599 shutdown_connections 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 --2- 172.21.15.94:0/3143267599 >> v2:172.21.15.145:3300/0 conn(0x7fd9b80c2aa0 0x7fd9b806bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 --2- 172.21.15.94:0/3143267599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b8155bd0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 -- 172.21.15.94:0/3143267599 >> 172.21.15.94:0/3143267599 conn(0x7fd9b80c3b40 msgr2=0x7fd9b80c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 -- 172.21.15.94:0/3143267599 shutdown_connections 2024-03-06T07:23:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.558+0000 7fd9bf156640 1 -- 172.21.15.94:0/3143267599 wait complete. 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.559+0000 7fd9bf156640 1 Processor -- start 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.559+0000 7fd9bf156640 1 -- start start 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b80c0470 0x7fd9b814ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b814dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b80c2aa0 0x7fd9b81e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c1f30 con 0x7fd9b80c26c0 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c0900 con 0x7fd9b80c0470 2024-03-06T07:23:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b80c20b0 con 0x7fd9b80c2aa0 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b814dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b814dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48948/0 (socket says 172.21.15.94:48948) 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 -- 172.21.15.94:0/1236832384 learned_addr learned my addr 172.21.15.94:0/1236832384 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bd6f3640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b80c2aa0 0x7fd9b81e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bcef2640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b80c0470 0x7fd9b814ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 -- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b80c2aa0 msgr2=0x7fd9b81e86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b80c2aa0 0x7fd9b81e86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 -- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b80c0470 msgr2=0x7fd9b814ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b80c0470 0x7fd9b814ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 -- 172.21.15.94:0/1236832384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b814d240 con 0x7fd9b80c26c0 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bd6f3640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b80c2aa0 0x7fd9b81e86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bcef2640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b80c0470 0x7fd9b814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b7fff640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b814dff0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d7d0 tx=0x7fd9a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b57fa640 1 -- 172.21.15.94:0/1236832384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4004030 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b57fa640 1 -- 172.21.15.94:0/1236832384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a40041d0 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b81eea70 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9b57fa640 1 -- 172.21.15.94:0/1236832384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400b6c0 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.560+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b814d7f0 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.561+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b804f330 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.562+0000 7fd9b57fa640 1 -- 172.21.15.94:0/1236832384 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9a400b860 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.562+0000 7fd9b57fa640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd99c06b500 0x7fd99c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.562+0000 7fd9b57fa640 1 -- 172.21.15.94:0/1236832384 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd9a408a550 con 0x7fd9b80c26c0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.562+0000 7fd9bcef2640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd99c06b500 0x7fd99c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.563+0000 7fd9bcef2640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd99c06b500 0x7fd99c06d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac036000 tx=0x7fd9ac008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.566+0000 7fd9b57fa640 1 -- 172.21.15.94:0/1236832384 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9a40592d0 con 0x7fd9b80c26c0 2024-03-06T07:23:21.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.774+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd9b80713d0 con 0x7fd9b80c26c0 2024-03-06T07:23:21.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.827+0000 7fd9b57fa640 1 -- 172.21.15.94:0/1236832384 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v118) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd9a4058bd0 con 0x7fd9b80c26c0 2024-03-06T07:23:21.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd99c06b500 msgr2=0x7fd99c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd99c06b500 0x7fd99c06d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac036000 tx=0x7fd9ac008030 comp rx=0 tx=0).stop 2024-03-06T07:23:21.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 msgr2=0x7fd9b814dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b814dff0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d7d0 tx=0x7fd9a400dc90 comp rx=0 tx=0).stop 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 shutdown_connections 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd99c06b500 0x7fd99c06d9c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b80c2aa0 0x7fd9b81e86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9b80c26c0 0x7fd9b814dff0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 --2- 172.21.15.94:0/1236832384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b80c0470 0x7fd9b814ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 >> 172.21.15.94:0/1236832384 conn(0x7fd9b80c3b40 msgr2=0x7fd9b8151860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 shutdown_connections 2024-03-06T07:23:21.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.829+0000 7fd9bf156640 1 -- 172.21.15.94:0/1236832384 wait complete. 2024-03-06T07:23:21.842 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:21.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:21.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.946+0000 7fcee065c640 1 Processor -- start 2024-03-06T07:23:21.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.946+0000 7fcee065c640 1 -- start start 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fcee065c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fced8155c30 0x7fced806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fcee065c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced80c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fcee065c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80b0450 con 0x7fced80b0070 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fcee065c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced806cfb0 con 0x7fced8155c30 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fcee065c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced806c020 con 0x7fced80c5380 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fceddbf7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fced8155c30 0x7fced806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fceddbf7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fced8155c30 0x7fced806ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48964/0 (socket says 172.21.15.94:48964) 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fceddbf7640 1 -- 172.21.15.94:0/2872594365 learned_addr learned my addr 172.21.15.94:0/2872594365 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:21.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fcedebf9640 1 --2- 172.21.15.94:0/2872594365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced80c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fceddbf7640 1 -- 172.21.15.94:0/2872594365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 msgr2=0x7fced80c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fceddbf7640 1 --2- 172.21.15.94:0/2872594365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced80c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fceddbf7640 1 -- 172.21.15.94:0/2872594365 >> v1:172.21.15.145:6789/0 conn(0x7fced80b0070 legacy=0x7fced80c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.947+0000 7fceddbf7640 1 -- 172.21.15.94:0/2872594365 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced81560e0 con 0x7fced8155c30 2024-03-06T07:23:21.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.948+0000 7fceddbf7640 1 --2- 172.21.15.94:0/2872594365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced806ca70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fcec8009870 tx=0x7fcec802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c20fc5cc2d58db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.948+0000 7fcedd3f6640 1 -- 172.21.15.94:0/2872594365 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8035020 con 0x7fced8155c30 2024-03-06T07:23:21.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.948+0000 7fcedd3f6640 1 -- 172.21.15.94:0/2872594365 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec8031640 con 0x7fced8155c30 2024-03-06T07:23:21.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.948+0000 7fcedd3f6640 1 -- 172.21.15.94:0/2872594365 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec802d040 con 0x7fced8155c30 2024-03-06T07:23:21.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.948+0000 7fcee065c640 1 -- 172.21.15.94:0/2872594365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 msgr2=0x7fced806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.948+0000 7fcee065c640 1 --2- 172.21.15.94:0/2872594365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced806ca70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fcec8009870 tx=0x7fcec802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.949+0000 7fcee065c640 1 -- 172.21.15.94:0/2872594365 shutdown_connections 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.949+0000 7fcee065c640 1 --2- 172.21.15.94:0/2872594365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced80c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.949+0000 7fcee065c640 1 --2- 172.21.15.94:0/2872594365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced806ca70 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.949+0000 7fcee065c640 1 -- 172.21.15.94:0/2872594365 >> 172.21.15.94:0/2872594365 conn(0x7fced80ac590 msgr2=0x7fced80ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.949+0000 7fcee065c640 1 -- 172.21.15.94:0/2872594365 shutdown_connections 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.949+0000 7fcee065c640 1 -- 172.21.15.94:0/2872594365 wait complete. 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 Processor -- start 2024-03-06T07:23:21.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 -- start start 2024-03-06T07:23:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fced80b0070 0x7fced81549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced814ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced814bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced806c270 con 0x7fced8155c30 2024-03-06T07:23:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced80c7ad0 con 0x7fced80b0070 2024-03-06T07:23:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcee065c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced806c550 con 0x7fced80c5380 2024-03-06T07:23:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcedebf9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced814bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcedebf9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced814bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48974/0 (socket says 172.21.15.94:48974) 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.950+0000 7fcedebf9640 1 -- 172.21.15.94:0/837096301 learned_addr learned my addr 172.21.15.94:0/837096301 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fceddbf7640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced814ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcede3f8640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fced80b0070 0x7fced81549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcedebf9640 1 -- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 msgr2=0x7fced814ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcedebf9640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced814ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcedebf9640 1 -- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fced80b0070 msgr2=0x7fced81549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcedebf9640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fced80b0070 0x7fced81549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcedebf9640 1 -- 172.21.15.94:0/837096301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8155620 con 0x7fced8155c30 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcede3f8640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fced80b0070 0x7fced81549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcedebf9640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced814bf40 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fced000efc0 tx=0x7fced000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fceceffd640 1 -- 172.21.15.94:0/837096301 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0018020 con 0x7fced8155c30 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fceceffd640 1 -- 172.21.15.94:0/837096301 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0014480 con 0x7fced8155c30 2024-03-06T07:23:21.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced8158110 con 0x7fced8155c30 2024-03-06T07:23:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fceceffd640 1 -- 172.21.15.94:0/837096301 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0014870 con 0x7fced8155c30 2024-03-06T07:23:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.951+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced81ec030 con 0x7fced8155c30 2024-03-06T07:23:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.952+0000 7fceceffd640 1 -- 172.21.15.94:0/837096301 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fced0005d10 con 0x7fced8155c30 2024-03-06T07:23:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.952+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced80c85d0 con 0x7fced8155c30 2024-03-06T07:23:21.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.953+0000 7fceceffd640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcebc06b4d0 0x7fcebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.953+0000 7fceceffd640 1 -- 172.21.15.94:0/837096301 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fced008b160 con 0x7fced8155c30 2024-03-06T07:23:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.953+0000 7fcede3f8640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcebc06b4d0 0x7fcebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.954+0000 7fcede3f8640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcebc06b4d0 0x7fcebc06d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fced40015e0 tx=0x7fced4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:21.957+0000 7fceceffd640 1 -- 172.21.15.94:0/837096301 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fced80c85d0 con 0x7fced8155c30 2024-03-06T07:23:22.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.173+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fced806d860 con 0x7fced8155c30 2024-03-06T07:23:22.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.831+0000 7fceceffd640 1 -- 172.21.15.94:0/837096301 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v119) v1 ==== 122+0+0 (secure 0 0 0) 0x7fced0022030 con 0x7fced8155c30 2024-03-06T07:23:22.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcebc06b4d0 msgr2=0x7fcebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcebc06b4d0 0x7fcebc06d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fced40015e0 tx=0x7fced4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 msgr2=0x7fced814bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced814bf40 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fced000efc0 tx=0x7fced000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 shutdown_connections 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcebc06b4d0 0x7fcebc06d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fced8155c30 0x7fced814bf40 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fced80c5380 0x7fced814ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 --2- 172.21.15.94:0/837096301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fced80b0070 0x7fced81549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 >> 172.21.15.94:0/837096301 conn(0x7fced80ac590 msgr2=0x7fced80ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 shutdown_connections 2024-03-06T07:23:22.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.833+0000 7fcee065c640 1 -- 172.21.15.94:0/837096301 wait complete. 2024-03-06T07:23:22.846 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:22.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:22.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.944+0000 7f19abe39640 1 Processor -- start 2024-03-06T07:23:22.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.944+0000 7f19abe39640 1 -- start start 2024-03-06T07:23:22.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19abe39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f19a414f360 0x7f19a4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19abe39640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a40c73d0 con 0x7f19a40c88c0 2024-03-06T07:23:22.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19abe39640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a40be2e0 con 0x7f19a4158b60 2024-03-06T07:23:22.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19abe39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a4159700 con 0x7f19a414f360 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a93d4640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f19a40c88c0 0x7f19a40c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54334/0 (socket says 172.21.15.94:54334) 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a93d4640 1 -- 172.21.15.94:0/1994577841 learned_addr learned my addr 172.21.15.94:0/1994577841 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a9bd5640 1 --2- 172.21.15.94:0/1994577841 >> v2:172.21.15.145:3300/0 conn(0x7f19a414f360 0x7f19a4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988689630 0 0) 0x7f19a40c73d0 con 0x7f19a40c88c0 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1990003520 con 0x7f19a40c88c0 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307253769 0 0) 0x7f19a40be2e0 con 0x7f19a4158b60 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a40c73d0 con 0x7f19a4158b60 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3258375383 0 0) 0x7f1990003520 con 0x7f19a40c88c0 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 >> v2:172.21.15.145:3300/0 conn(0x7f19a414f360 msgr2=0x7f19a4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 --2- 172.21.15.94:0/1994577841 >> v2:172.21.15.145:3300/0 conn(0x7f19a414f360 0x7f19a4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 >> v1:172.21.15.145:6789/0 conn(0x7f19a4158b60 legacy=0x7f19a4158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a40c7670 con 0x7f19a40c88c0 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f1994001670 con 0x7f19a40c88c0 2024-03-06T07:23:22.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 >> v1:172.21.15.94:6789/0 conn(0x7f19a40c88c0 legacy=0x7f19a40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1990003b90 0x7f1990003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f1990004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1990005210 0x7f199000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4159700 con 0x7f1990004800 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199000c140 con 0x7f1990003b90 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.945+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199000c340 con 0x7f1990005210 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19a93d4640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f1990004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19a9bd5640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1990003b90 0x7f1990003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19aa3d6640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1990005210 0x7f199000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19a93d4640 1 -- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1990005210 msgr2=0x7f199000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19a93d4640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1990005210 0x7f199000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19a93d4640 1 -- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1990003b90 msgr2=0x7f1990003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19a93d4640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1990003b90 0x7f1990003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19a93d4640 1 -- 172.21.15.94:0/1994577841 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990003520 con 0x7f1990004800 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19aa3d6640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1990005210 0x7f199000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.946+0000 7f19aa3d6640 1 -- 172.21.15.94:0/1994577841 reap_dead start 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19a93d4640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f1990004c80 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f19a40c8d20 tx=0x7f1990005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4034f88daa81cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1994007c40 con 0x7f1990004800 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1994030070 con 0x7f1990004800 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19a8bd3640 1 -- 172.21.15.94:0/1994577841 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1994005d90 con 0x7f1990004800 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19abe39640 1 -- 172.21.15.94:0/1994577841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 msgr2=0x7f1990004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19abe39640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f1990004c80 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f19a40c8d20 tx=0x7f1990005830 comp rx=0 tx=0).stop 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19abe39640 1 -- 172.21.15.94:0/1994577841 shutdown_connections 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19abe39640 1 --2- 172.21.15.94:0/1994577841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f1990004c80 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19abe39640 1 -- 172.21.15.94:0/1994577841 >> 172.21.15.94:0/1994577841 conn(0x7f19a40ac570 msgr2=0x7f19a40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19abe39640 1 -- 172.21.15.94:0/1994577841 shutdown_connections 2024-03-06T07:23:22.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.947+0000 7f19abe39640 1 -- 172.21.15.94:0/1994577841 wait complete. 2024-03-06T07:23:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.948+0000 7f19abe39640 1 Processor -- start 2024-03-06T07:23:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19abe39640 1 -- start start 2024-03-06T07:23:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19abe39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f19a414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19abe39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a4154860 0x7f19a414ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19abe39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a414b0d0 0x7f19a40c0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19abe39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40c7c30 con 0x7f1990004800 2024-03-06T07:23:22.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19abe39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4155c10 con 0x7f19a414b0d0 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19abe39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4155d90 con 0x7f19a4154860 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19a9bd5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f19a414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19a9bd5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f19a414f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48990/0 (socket says 172.21.15.94:48990) 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19a9bd5640 1 -- 172.21.15.94:0/1461723639 learned_addr learned my addr 172.21.15.94:0/1461723639 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19a93d4640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a4154860 0x7f19a414ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.949+0000 7f19aa3d6640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a414b0d0 0x7f19a40c0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19a9bd5640 1 -- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a4154860 msgr2=0x7f19a414ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19a9bd5640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a4154860 0x7f19a414ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19a9bd5640 1 -- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a414b0d0 msgr2=0x7f19a40c0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19a9bd5640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a414b0d0 0x7f19a40c0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19a9bd5640 1 -- 172.21.15.94:0/1461723639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4152010 con 0x7f1990004800 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19a9bd5640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f19a414f740 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f19a0011a70 tx=0x7f19a0011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f199a7fc640 1 -- 172.21.15.94:0/1461723639 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a000fb50 con 0x7f1990004800 2024-03-06T07:23:22.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f199a7fc640 1 -- 172.21.15.94:0/1461723639 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a000fcf0 con 0x7f1990004800 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a4152560 con 0x7f1990004800 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f199a7fc640 1 -- 172.21.15.94:0/1461723639 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a000fea0 con 0x7f1990004800 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.950+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a40bfe20 con 0x7f1990004800 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.951+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a404f330 con 0x7f1990004800 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.952+0000 7f199a7fc640 1 -- 172.21.15.94:0/1461723639 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19a0012040 con 0x7f1990004800 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.952+0000 7f199a7fc640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.952+0000 7f199a7fc640 1 -- 172.21.15.94:0/1461723639 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f19a008a8b0 con 0x7f1990004800 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.953+0000 7f19a93d4640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.953+0000 7f19a93d4640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1994034000 tx=0x7f1994009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:22.957+0000 7f199a7fc640 1 -- 172.21.15.94:0/1461723639 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19a0019030 con 0x7f1990004800 2024-03-06T07:23:23.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.166+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a40c8d50 con 0x7f1990004800 2024-03-06T07:23:23.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.166+0000 7f199a7fc640 1 -- 172.21.15.94:0/1461723639 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4990 (secure 0 0 0) 0x7f19a0058f60 con 0x7f1990004800 2024-03-06T07:23:23.168 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:23.168 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":119,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1362727065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1362727065},{"type":"v1","addr":"172.21.15.145:6836","nonce":1362727065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:22.175096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/392019985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":392019985},{"type":"v1","addr":"172.21.15.94:6835","nonce":392019985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:23:23.169 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 119 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.168+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 msgr2=0x7f198806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.168+0000 7f19abe39640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1994034000 tx=0x7f1994009030 comp rx=0 tx=0).stop 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.168+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 msgr2=0x7f19a414f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.168+0000 7f19abe39640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f19a414f740 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f19a0011a70 tx=0x7f19a0011f30 comp rx=0 tx=0).stop 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 shutdown_connections 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a414b0d0 0x7f19a40c0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a4154860 0x7f19a414ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 --2- 172.21.15.94:0/1461723639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1990004800 0x7f19a414f740 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 >> 172.21.15.94:0/1461723639 conn(0x7f19a40ac570 msgr2=0x7f19a414d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 shutdown_connections 2024-03-06T07:23:23.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.169+0000 7f19abe39640 1 -- 172.21.15.94:0/1461723639 wait complete. 2024-03-06T07:23:23.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.281+0000 7fe2dcdae640 1 Processor -- start 2024-03-06T07:23:23.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.281+0000 7fe2dcdae640 1 -- start start 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2dcdae640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe2d814f360 0x7fe2d8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2dcdae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 0x7fe2d8158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2dcdae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d80c73d0 con 0x7fe2d80c88c0 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2dcdae640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d80c8dd0 con 0x7fe2d814f360 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2dcdae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80be2e0 con 0x7fe2d8158b60 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2d5d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe2d80c88c0 0x7fe2d80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54348/0 (socket says 172.21.15.94:54348) 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2d5d9b640 1 -- 172.21.15.94:0/1944300173 learned_addr learned my addr 172.21.15.94:0/1944300173 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 0x7fe2d8158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026052616 0 0) 0x7fe2d80c73d0 con 0x7fe2d80c88c0 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.282+0000 7fe2d659c640 1 --2- 172.21.15.94:0/1944300173 >> v2:172.21.15.145:3300/0 conn(0x7fe2d814f360 0x7fe2d8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c0003520 con 0x7fe2d80c88c0 2024-03-06T07:23:23.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 314601574 0 0) 0x7fe2c0003520 con 0x7fe2d80c88c0 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 msgr2=0x7fe2d8158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 0x7fe2d8158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 >> v2:172.21.15.145:3300/0 conn(0x7fe2d814f360 msgr2=0x7fe2d8155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 --2- 172.21.15.94:0/1944300173 >> v2:172.21.15.145:3300/0 conn(0x7fe2d814f360 0x7fe2d8155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d80c7670 con 0x7fe2d80c88c0 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fe2cc001670 con 0x7fe2d80c88c0 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 >> v1:172.21.15.94:6789/0 conn(0x7fe2d80c88c0 legacy=0x7fe2d80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 0x7fe2c00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0003f30 0x7fe2c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2c0004d40 0x7fe2c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80be2e0 con 0x7fe2c0004d40 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c000b9c0 con 0x7fe2c0003f30 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c000bb90 con 0x7fe2d8158b60 2024-03-06T07:23:23.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2c0004d40 0x7fe2c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d659c640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 0x7fe2c00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d5d9b640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0003f30 0x7fe2c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 msgr2=0x7fe2c00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d8158b60 0x7fe2c00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0003f30 msgr2=0x7fe2c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0003f30 0x7fe2c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.283+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/1944300173 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0003520 con 0x7fe2c0004d40 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.284+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2c0004d40 0x7fe2c00051c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800e170 tx=0x7fe2c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec4ffeed3e9fb5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.284+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8039070 con 0x7fe2c0004d40 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.284+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c802cbf0 con 0x7fe2c0004d40 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.284+0000 7fe2d559a640 1 -- 172.21.15.94:0/1944300173 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8009c10 con 0x7fe2c0004d40 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.285+0000 7fe2dcdae640 1 -- 172.21.15.94:0/1944300173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2c0004d40 msgr2=0x7fe2c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.285+0000 7fe2dcdae640 1 --2- 172.21.15.94:0/1944300173 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2c0004d40 0x7fe2c00051c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800e170 tx=0x7fe2c0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:23.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.285+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/1944300173 reap_dead start 2024-03-06T07:23:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.285+0000 7fe2dcdae640 1 -- 172.21.15.94:0/1944300173 shutdown_connections 2024-03-06T07:23:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.285+0000 7fe2dcdae640 1 -- 172.21.15.94:0/1944300173 >> 172.21.15.94:0/1944300173 conn(0x7fe2d80ac570 msgr2=0x7fe2d80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.285+0000 7fe2dcdae640 1 -- 172.21.15.94:0/1944300173 shutdown_connections 2024-03-06T07:23:23.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.285+0000 7fe2dcdae640 1 -- 172.21.15.94:0/1944300173 wait complete. 2024-03-06T07:23:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.286+0000 7fe2dcdae640 1 Processor -- start 2024-03-06T07:23:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.286+0000 7fe2dcdae640 1 -- start start 2024-03-06T07:23:23.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2dcdae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0004d40 0x7fe2d81ee060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2dcdae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d80b0480 0x7fe2d8150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2dcdae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2d8150e50 0x7fe2d814bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2dcdae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80c5a50 con 0x7fe2d8150e50 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2dcdae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80c7c30 con 0x7fe2c0004d40 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2dcdae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80c5bd0 con 0x7fe2d80b0480 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2d8150e50 0x7fe2d814bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2d8150e50 0x7fe2d814bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49006/0 (socket says 172.21.15.94:49006) 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/3310432630 learned_addr learned my addr 172.21.15.94:0/3310432630 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d5d9b640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d80b0480 0x7fe2d8150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d80b0480 msgr2=0x7fe2d8150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d80b0480 0x7fe2d8150910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0004d40 msgr2=0x7fe2d81ee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0004d40 0x7fe2d81ee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 -- 172.21.15.94:0/3310432630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d81512e0 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2d6d9d640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2d8150e50 0x7fe2d814bae0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800e170 tx=0x7fe2c8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.287+0000 7fe2c6ffd640 1 -- 172.21.15.94:0/3310432630 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8039030 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.288+0000 7fe2c6ffd640 1 -- 172.21.15.94:0/3310432630 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c803a040 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.288+0000 7fe2c6ffd640 1 -- 172.21.15.94:0/3310432630 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8003990 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.288+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d81ee8a0 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.288+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d81526f0 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.288+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d804f330 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.289+0000 7fe2c6ffd640 1 -- 172.21.15.94:0/3310432630 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2c8045030 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.290+0000 7fe2c6ffd640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b4d0 0x7fe2b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.290+0000 7fe2c6ffd640 1 -- 172.21.15.94:0/3310432630 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe2c80ac370 con 0x7fe2d8150e50 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.290+0000 7fe2d659c640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b4d0 0x7fe2b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.291+0000 7fe2d659c640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b4d0 0x7fe2b806d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006e80 tx=0x7fe2d0006df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.294+0000 7fe2c6ffd640 1 -- 172.21.15.94:0/3310432630 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2c804c020 con 0x7fe2d8150e50 2024-03-06T07:23:23.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.471+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2d806cda0 con 0x7fe2d8150e50 2024-03-06T07:23:23.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.472+0000 7fe2c6ffd640 1 -- 172.21.15.94:0/3310432630 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+16892 (secure 0 0 0) 0x7fe2c807b0f0 con 0x7fe2d8150e50 2024-03-06T07:23:23.474 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:23.475 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":135,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:21.811253+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:18.382044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:19.127404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:23.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b4d0 msgr2=0x7fe2b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b4d0 0x7fe2b806d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006e80 tx=0x7fe2d0006df0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2d8150e50 msgr2=0x7fe2d814bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2d8150e50 0x7fe2d814bae0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800e170 tx=0x7fe2c8009750 comp rx=0 tx=0).stop 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 shutdown_connections 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe2b806b4d0 0x7fe2b806d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2d8150e50 0x7fe2d814bae0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d80b0480 0x7fe2d8150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 --2- 172.21.15.94:0/3310432630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2c0004d40 0x7fe2d81ee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 >> 172.21.15.94:0/3310432630 conn(0x7fe2d80ac570 msgr2=0x7fe2d80c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 shutdown_connections 2024-03-06T07:23:23.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.474+0000 7fe2dcdae640 1 -- 172.21.15.94:0/3310432630 wait complete. 2024-03-06T07:23:23.488 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:23.488 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:23.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.586+0000 7ff602b3a640 1 Processor -- start 2024-03-06T07:23:23.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.586+0000 7ff602b3a640 1 -- start start 2024-03-06T07:23:23.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff602b3a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc14f360 0x7ff5fc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff602b3a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0c73d0 con 0x7ff5fc158b60 2024-03-06T07:23:23.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff602b3a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0be2e0 con 0x7ff5fc0c88c0 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff602b3a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc159700 con 0x7ff5fc14f360 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff6010d7640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff5fc158b60 0x7ff5fc158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54358/0 (socket says 172.21.15.94:54358) 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff6010d7640 1 -- 172.21.15.94:0/2924290805 learned_addr learned my addr 172.21.15.94:0/2924290805 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff6008d6640 1 --2- 172.21.15.94:0/2924290805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc14f360 0x7ff5fc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff5fb7fe640 1 -- 172.21.15.94:0/2924290805 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235393220 0 0) 0x7ff5fc0c73d0 con 0x7ff5fc158b60 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff5fb7fe640 1 -- 172.21.15.94:0/2924290805 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ec003520 con 0x7ff5fc158b60 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff5fb7fe640 1 -- 172.21.15.94:0/2924290805 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284631658 0 0) 0x7ff5ec003520 con 0x7ff5fc158b60 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff6008d6640 1 -- 172.21.15.94:0/2924290805 >> v1:172.21.15.145:6789/0 conn(0x7ff5fc0c88c0 legacy=0x7ff5fc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff6008d6640 1 -- 172.21.15.94:0/2924290805 >> v1:172.21.15.94:6789/0 conn(0x7ff5fc158b60 legacy=0x7ff5fc158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.587+0000 7ff6008d6640 1 -- 172.21.15.94:0/2924290805 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0c7670 con 0x7ff5fc14f360 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.588+0000 7ff6008d6640 1 --2- 172.21.15.94:0/2924290805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc14f360 0x7ff5fc155c10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097d0 tx=0x7ff5f002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d3da2fb3e389151 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.588+0000 7ff5fb7fe640 1 -- 172.21.15.94:0/2924290805 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0037070 con 0x7ff5fc14f360 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.588+0000 7ff5fb7fe640 1 -- 172.21.15.94:0/2924290805 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f0030510 con 0x7ff5fc14f360 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.588+0000 7ff5fb7fe640 1 -- 172.21.15.94:0/2924290805 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40019f0 con 0x7ff5fc14f360 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.589+0000 7ff602b3a640 1 -- 172.21.15.94:0/2924290805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc14f360 msgr2=0x7ff5fc155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.589+0000 7ff602b3a640 1 --2- 172.21.15.94:0/2924290805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc14f360 0x7ff5fc155c10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097d0 tx=0x7ff5f002b770 comp rx=0 tx=0).stop 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.589+0000 7ff602b3a640 1 -- 172.21.15.94:0/2924290805 shutdown_connections 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.589+0000 7ff602b3a640 1 --2- 172.21.15.94:0/2924290805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc14f360 0x7ff5fc155c10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.589+0000 7ff602b3a640 1 -- 172.21.15.94:0/2924290805 >> 172.21.15.94:0/2924290805 conn(0x7ff5fc0ac570 msgr2=0x7ff5fc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.589+0000 7ff602b3a640 1 -- 172.21.15.94:0/2924290805 shutdown_connections 2024-03-06T07:23:23.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.589+0000 7ff602b3a640 1 -- 172.21.15.94:0/2924290805 wait complete. 2024-03-06T07:23:23.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.590+0000 7ff602b3a640 1 Processor -- start 2024-03-06T07:23:23.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.590+0000 7ff602b3a640 1 -- start start 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff602b3a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc0c88c0 0x7ff5fc0c4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff602b3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5fc14f360 0x7ff5fc0ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff602b3a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5fc158b60 0x7ff5fc0ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff602b3a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0c7c30 con 0x7ff5fc14f360 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff602b3a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0c5a50 con 0x7ff5fc158b60 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff602b3a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0c5bd0 con 0x7ff5fc0c88c0 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5fc14f360 0x7ff5fc0ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5fc14f360 0x7ff5fc0ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49008/0 (socket says 172.21.15.94:49008) 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 -- 172.21.15.94:0/1796873050 learned_addr learned my addr 172.21.15.94:0/1796873050 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff6010d7640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5fc158b60 0x7ff5fc0ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff6008d6640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc0c88c0 0x7ff5fc0c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 -- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc0c88c0 msgr2=0x7ff5fc0c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc0c88c0 0x7ff5fc0c4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 -- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5fc158b60 msgr2=0x7ff5fc0ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5fc158b60 0x7ff5fc0ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.591+0000 7ff5fbfff640 1 -- 172.21.15.94:0/1796873050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0b04f0 con 0x7ff5fc14f360 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.592+0000 7ff5fbfff640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5fc14f360 0x7ff5fc0ae090 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800c3c0 tx=0x7ff5e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.592+0000 7ff5f97fa640 1 -- 172.21.15.94:0/1796873050 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8017070 con 0x7ff5fc14f360 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.592+0000 7ff5f97fa640 1 -- 172.21.15.94:0/1796873050 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e80052f0 con 0x7ff5fc14f360 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.592+0000 7ff5f97fa640 1 -- 172.21.15.94:0/1796873050 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8003830 con 0x7ff5fc14f360 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.592+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc0aed20 con 0x7ff5fc14f360 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.592+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc0c2c90 con 0x7ff5fc14f360 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.593+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc0add60 con 0x7ff5fc14f360 2024-03-06T07:23:23.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.593+0000 7ff5f97fa640 1 -- 172.21.15.94:0/1796873050 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5e801a070 con 0x7ff5fc14f360 2024-03-06T07:23:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.593+0000 7ff5f97fa640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5e006b4d0 0x7ff5e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.594+0000 7ff5f97fa640 1 -- 172.21.15.94:0/1796873050 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff5e8089600 con 0x7ff5fc14f360 2024-03-06T07:23:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.594+0000 7ff6008d6640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5e006b4d0 0x7ff5e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.594+0000 7ff6008d6640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5e006b4d0 0x7ff5e006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0034000 tx=0x7ff5f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.598+0000 7ff5f97fa640 1 -- 172.21.15.94:0/1796873050 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5e8058380 con 0x7ff5fc14f360 2024-03-06T07:23:23.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.772+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5fc04f330 con 0x7ff5e006b4d0 2024-03-06T07:23:23.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.779+0000 7ff5f97fa640 1 -- 172.21.15.94:0/1796873050 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252562 (secure 0 0 0) 0x7ff5fc04f330 con 0x7ff5e006b4d0 2024-03-06T07:23:23.784 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:23.785 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5e006b4d0 msgr2=0x7ff5e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5e006b4d0 0x7ff5e006d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0034000 tx=0x7ff5f0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5fc14f360 msgr2=0x7ff5fc0ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5fc14f360 0x7ff5fc0ae090 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800c3c0 tx=0x7ff5e800c880 comp rx=0 tx=0).stop 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 shutdown_connections 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5e006b4d0 0x7ff5e006d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5fc158b60 0x7ff5fc0ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5fc14f360 0x7ff5fc0ae090 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 --2- 172.21.15.94:0/1796873050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5fc0c88c0 0x7ff5fc0c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 >> 172.21.15.94:0/1796873050 conn(0x7ff5fc0ac570 msgr2=0x7ff5fc06b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 shutdown_connections 2024-03-06T07:23:23.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.782+0000 7ff602b3a640 1 -- 172.21.15.94:0/1796873050 wait complete. 2024-03-06T07:23:23.795 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-03-06T07:23:22.700733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262880,"kb_used_data":13404,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711840,"statfs":{"total":773094113280,"available":772824924160,"internally_reserved":0,"allocated":13725696,"data_stored":7701688,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000837"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831874+0000","last_change":"2024-03-06T07:23:20.833717+0000","last_active":"2024-03-06T07:23:21.831874+0000","last_peered":"2024-03-06T07:23:21.831874+0000","last_clean":"2024-03-06T07:23:21.831874+0000","last_became_active":"2024-03-06T07:23:20.833391+0000","last_became_peered":"2024-03-06T07:23:20.833391+0000","last_unstale":"2024-03-06T07:23:21.831874+0000","last_undegraded":"2024-03-06T07:23:21.831874+0000","last_fullsized":"2024-03-06T07:23:21.831874+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:07.020050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814127+0000","last_change":"2024-03-06T07:23:20.831542+0000","last_active":"2024-03-06T07:23:21.814127+0000","last_peered":"2024-03-06T07:23:21.814127+0000","last_clean":"2024-03-06T07:23:21.814127+0000","last_became_active":"2024-03-06T07:23:20.830456+0000","last_became_peered":"2024-03-06T07:23:20.830456+0000","last_unstale":"2024-03-06T07:23:21.814127+0000","last_undegraded":"2024-03-06T07:23:21.814127+0000","last_fullsized":"2024-03-06T07:23:21.814127+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:47.901171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814272+0000","last_change":"2024-03-06T07:23:20.830517+0000","last_active":"2024-03-06T07:23:21.814272+0000","last_peered":"2024-03-06T07:23:21.814272+0000","last_clean":"2024-03-06T07:23:21.814272+0000","last_became_active":"2024-03-06T07:23:20.830209+0000","last_became_peered":"2024-03-06T07:23:20.830209+0000","last_unstale":"2024-03-06T07:23:21.814272+0000","last_undegraded":"2024-03-06T07:23:21.814272+0000","last_fullsized":"2024-03-06T07:23:21.814272+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:21.123558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815246+0000","last_change":"2024-03-06T07:23:20.823795+0000","last_active":"2024-03-06T07:23:21.815246+0000","last_peered":"2024-03-06T07:23:21.815246+0000","last_clean":"2024-03-06T07:23:21.815246+0000","last_became_active":"2024-03-06T07:23:20.823520+0000","last_became_peered":"2024-03-06T07:23:20.823520+0000","last_unstale":"2024-03-06T07:23:21.815246+0000","last_undegraded":"2024-03-06T07:23:21.815246+0000","last_fullsized":"2024-03-06T07:23:21.815246+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:21.355665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927614+0000","last_change":"2024-03-06T07:23:20.833801+0000","last_active":"2024-03-06T07:23:21.927614+0000","last_peered":"2024-03-06T07:23:21.927614+0000","last_clean":"2024-03-06T07:23:21.927614+0000","last_became_active":"2024-03-06T07:23:20.833264+0000","last_became_peered":"2024-03-06T07:23:20.833264+0000","last_unstale":"2024-03-06T07:23:21.927614+0000","last_undegraded":"2024-03-06T07:23:21.927614+0000","last_fullsized":"2024-03-06T07:23:21.927614+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:51.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927668+0000","last_change":"2024-03-06T07:23:20.832761+0000","last_active":"2024-03-06T07:23:21.927668+0000","last_peered":"2024-03-06T07:23:21.927668+0000","last_clean":"2024-03-06T07:23:21.927668+0000","last_became_active":"2024-03-06T07:23:20.832484+0000","last_became_peered":"2024-03-06T07:23:20.832484+0000","last_unstale":"2024-03-06T07:23:21.927668+0000","last_undegraded":"2024-03-06T07:23:21.927668+0000","last_fullsized":"2024-03-06T07:23:21.927668+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:45.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831564+0000","last_change":"2024-03-06T07:23:20.833437+0000","last_active":"2024-03-06T07:23:21.831564+0000","last_peered":"2024-03-06T07:23:21.831564+0000","last_clean":"2024-03-06T07:23:21.831564+0000","last_became_active":"2024-03-06T07:23:20.833161+0000","last_became_peered":"2024-03-06T07:23:20.833161+0000","last_unstale":"2024-03-06T07:23:21.831564+0000","last_undegraded":"2024-03-06T07:23:21.831564+0000","last_fullsized":"2024-03-06T07:23:21.831564+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:58.636532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815124+0000","last_change":"2024-03-06T07:23:20.823571+0000","last_active":"2024-03-06T07:23:21.815124+0000","last_peered":"2024-03-06T07:23:21.815124+0000","last_clean":"2024-03-06T07:23:21.815124+0000","last_became_active":"2024-03-06T07:23:20.822951+0000","last_became_peered":"2024-03-06T07:23:20.822951+0000","last_unstale":"2024-03-06T07:23:21.815124+0000","last_undegraded":"2024-03-06T07:23:21.815124+0000","last_fullsized":"2024-03-06T07:23:21.815124+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:22.074129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927958+0000","last_change":"2024-03-06T07:23:20.831489+0000","last_active":"2024-03-06T07:23:21.927958+0000","last_peered":"2024-03-06T07:23:21.927958+0000","last_clean":"2024-03-06T07:23:21.927958+0000","last_became_active":"2024-03-06T07:23:20.831199+0000","last_became_peered":"2024-03-06T07:23:20.831199+0000","last_unstale":"2024-03-06T07:23:21.927958+0000","last_undegraded":"2024-03-06T07:23:21.927958+0000","last_fullsized":"2024-03-06T07:23:21.927958+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:32.771906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928065+0000","last_change":"2024-03-06T07:23:20.823116+0000","last_active":"2024-03-06T07:23:21.928065+0000","last_peered":"2024-03-06T07:23:21.928065+0000","last_clean":"2024-03-06T07:23:21.928065+0000","last_became_active":"2024-03-06T07:23:20.822781+0000","last_became_peered":"2024-03-06T07:23:20.822781+0000","last_unstale":"2024-03-06T07:23:21.928065+0000","last_undegraded":"2024-03-06T07:23:21.928065+0000","last_fullsized":"2024-03-06T07:23:21.928065+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:24.917625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927629+0000","last_change":"2024-03-06T07:23:20.830890+0000","last_active":"2024-03-06T07:23:21.927629+0000","last_peered":"2024-03-06T07:23:21.927629+0000","last_clean":"2024-03-06T07:23:21.927629+0000","last_became_active":"2024-03-06T07:23:20.830419+0000","last_became_peered":"2024-03-06T07:23:20.830419+0000","last_unstale":"2024-03-06T07:23:21.927629+0000","last_undegraded":"2024-03-06T07:23:21.927629+0000","last_fullsized":"2024-03-06T07:23:21.927629+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:49.694980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815405+0000","last_change":"2024-03-06T07:23:20.835609+0000","last_active":"2024-03-06T07:23:21.815405+0000","last_peered":"2024-03-06T07:23:21.815405+0000","last_clean":"2024-03-06T07:23:21.815405+0000","last_became_active":"2024-03-06T07:23:20.835034+0000","last_became_peered":"2024-03-06T07:23:20.835034+0000","last_unstale":"2024-03-06T07:23:21.815405+0000","last_undegraded":"2024-03-06T07:23:21.815405+0000","last_fullsized":"2024-03-06T07:23:21.815405+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:58.422844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927353+0000","last_change":"2024-03-06T07:23:20.821260+0000","last_active":"2024-03-06T07:23:21.927353+0000","last_peered":"2024-03-06T07:23:21.927353+0000","last_clean":"2024-03-06T07:23:21.927353+0000","last_became_active":"2024-03-06T07:23:20.820990+0000","last_became_peered":"2024-03-06T07:23:20.820990+0000","last_unstale":"2024-03-06T07:23:21.927353+0000","last_undegraded":"2024-03-06T07:23:21.927353+0000","last_fullsized":"2024-03-06T07:23:21.927353+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:19.586412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815635+0000","last_change":"2024-03-06T07:23:20.837633+0000","last_active":"2024-03-06T07:23:21.815635+0000","last_peered":"2024-03-06T07:23:21.815635+0000","last_clean":"2024-03-06T07:23:21.815635+0000","last_became_active":"2024-03-06T07:23:20.837271+0000","last_became_peered":"2024-03-06T07:23:20.837271+0000","last_unstale":"2024-03-06T07:23:21.815635+0000","last_undegraded":"2024-03-06T07:23:21.815635+0000","last_fullsized":"2024-03-06T07:23:21.815635+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:11.035847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814220+0000","last_change":"2024-03-06T07:23:20.832684+0000","last_active":"2024-03-06T07:23:21.814220+0000","last_peered":"2024-03-06T07:23:21.814220+0000","last_clean":"2024-03-06T07:23:21.814220+0000","last_became_active":"2024-03-06T07:23:20.832100+0000","last_became_peered":"2024-03-06T07:23:20.832100+0000","last_unstale":"2024-03-06T07:23:21.814220+0000","last_undegraded":"2024-03-06T07:23:21.814220+0000","last_fullsized":"2024-03-06T07:23:21.814220+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.177104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928028+0000","last_change":"2024-03-06T07:23:20.826760+0000","last_active":"2024-03-06T07:23:21.928028+0000","last_peered":"2024-03-06T07:23:21.928028+0000","last_clean":"2024-03-06T07:23:21.928028+0000","last_became_active":"2024-03-06T07:23:20.823900+0000","last_became_peered":"2024-03-06T07:23:20.823900+0000","last_unstale":"2024-03-06T07:23:21.928028+0000","last_undegraded":"2024-03-06T07:23:21.928028+0000","last_fullsized":"2024-03-06T07:23:21.928028+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:00.676714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816475+0000","last_change":"2024-03-06T07:23:20.829449+0000","last_active":"2024-03-06T07:23:21.816475+0000","last_peered":"2024-03-06T07:23:21.816475+0000","last_clean":"2024-03-06T07:23:21.816475+0000","last_became_active":"2024-03-06T07:23:20.829171+0000","last_became_peered":"2024-03-06T07:23:20.829171+0000","last_unstale":"2024-03-06T07:23:21.816475+0000","last_undegraded":"2024-03-06T07:23:21.816475+0000","last_fullsized":"2024-03-06T07:23:21.816475+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:18.712434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927233+0000","last_change":"2024-03-06T07:23:20.838681+0000","last_active":"2024-03-06T07:23:21.927233+0000","last_peered":"2024-03-06T07:23:21.927233+0000","last_clean":"2024-03-06T07:23:21.927233+0000","last_became_active":"2024-03-06T07:23:20.838403+0000","last_became_peered":"2024-03-06T07:23:20.838403+0000","last_unstale":"2024-03-06T07:23:21.927233+0000","last_undegraded":"2024-03-06T07:23:21.927233+0000","last_fullsized":"2024-03-06T07:23:21.927233+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:43.634210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836218+0000","last_change":"2024-03-06T07:23:20.830707+0000","last_active":"2024-03-06T07:23:21.836218+0000","last_peered":"2024-03-06T07:23:21.836218+0000","last_clean":"2024-03-06T07:23:21.836218+0000","last_became_active":"2024-03-06T07:23:20.828572+0000","last_became_peered":"2024-03-06T07:23:20.828572+0000","last_unstale":"2024-03-06T07:23:21.836218+0000","last_undegraded":"2024-03-06T07:23:21.836218+0000","last_fullsized":"2024-03-06T07:23:21.836218+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:53.216867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927737+0000","last_change":"2024-03-06T07:23:20.823018+0000","last_active":"2024-03-06T07:23:21.927737+0000","last_peered":"2024-03-06T07:23:21.927737+0000","last_clean":"2024-03-06T07:23:21.927737+0000","last_became_active":"2024-03-06T07:23:20.822746+0000","last_became_peered":"2024-03-06T07:23:20.822746+0000","last_unstale":"2024-03-06T07:23:21.927737+0000","last_undegraded":"2024-03-06T07:23:21.927737+0000","last_fullsized":"2024-03-06T07:23:21.927737+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.177104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927487+0000","last_change":"2024-03-06T07:23:20.829029+0000","last_active":"2024-03-06T07:23:21.927487+0000","last_peered":"2024-03-06T07:23:21.927487+0000","last_clean":"2024-03-06T07:23:21.927487+0000","last_became_active":"2024-03-06T07:23:20.828741+0000","last_became_peered":"2024-03-06T07:23:20.828741+0000","last_unstale":"2024-03-06T07:23:21.927487+0000","last_undegraded":"2024-03-06T07:23:21.927487+0000","last_fullsized":"2024-03-06T07:23:21.927487+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:17.302710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831576+0000","last_change":"2024-03-06T07:23:20.834834+0000","last_active":"2024-03-06T07:23:21.831576+0000","last_peered":"2024-03-06T07:23:21.831576+0000","last_clean":"2024-03-06T07:23:21.831576+0000","last_became_active":"2024-03-06T07:23:20.834581+0000","last_became_peered":"2024-03-06T07:23:20.834581+0000","last_unstale":"2024-03-06T07:23:21.831576+0000","last_undegraded":"2024-03-06T07:23:21.831576+0000","last_fullsized":"2024-03-06T07:23:21.831576+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:45.761837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815799+0000","last_change":"2024-03-06T07:23:20.824357+0000","last_active":"2024-03-06T07:23:21.815799+0000","last_peered":"2024-03-06T07:23:21.815799+0000","last_clean":"2024-03-06T07:23:21.815799+0000","last_became_active":"2024-03-06T07:23:20.823953+0000","last_became_peered":"2024-03-06T07:23:20.823953+0000","last_unstale":"2024-03-06T07:23:21.815799+0000","last_undegraded":"2024-03-06T07:23:21.815799+0000","last_fullsized":"2024-03-06T07:23:21.815799+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.061700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927023+0000","last_change":"2024-03-06T07:23:20.827550+0000","last_active":"2024-03-06T07:23:21.927023+0000","last_peered":"2024-03-06T07:23:21.927023+0000","last_clean":"2024-03-06T07:23:21.927023+0000","last_became_active":"2024-03-06T07:23:20.827222+0000","last_became_peered":"2024-03-06T07:23:20.827222+0000","last_unstale":"2024-03-06T07:23:21.927023+0000","last_undegraded":"2024-03-06T07:23:21.927023+0000","last_fullsized":"2024-03-06T07:23:21.927023+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:54.586668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831952+0000","last_change":"2024-03-06T07:23:20.832808+0000","last_active":"2024-03-06T07:23:21.831952+0000","last_peered":"2024-03-06T07:23:21.831952+0000","last_clean":"2024-03-06T07:23:21.831952+0000","last_became_active":"2024-03-06T07:23:20.832484+0000","last_became_peered":"2024-03-06T07:23:20.832484+0000","last_unstale":"2024-03-06T07:23:21.831952+0000","last_undegraded":"2024-03-06T07:23:21.831952+0000","last_fullsized":"2024-03-06T07:23:21.831952+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:25.362417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831873+0000","last_change":"2024-03-06T07:23:20.833900+0000","last_active":"2024-03-06T07:23:21.831873+0000","last_peered":"2024-03-06T07:23:21.831873+0000","last_clean":"2024-03-06T07:23:21.831873+0000","last_became_active":"2024-03-06T07:23:20.833375+0000","last_became_peered":"2024-03-06T07:23:20.833375+0000","last_unstale":"2024-03-06T07:23:21.831873+0000","last_undegraded":"2024-03-06T07:23:21.831873+0000","last_fullsized":"2024-03-06T07:23:21.831873+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:54.587065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927006+0000","last_change":"2024-03-06T07:23:20.829523+0000","last_active":"2024-03-06T07:23:21.927006+0000","last_peered":"2024-03-06T07:23:21.927006+0000","last_clean":"2024-03-06T07:23:21.927006+0000","last_became_active":"2024-03-06T07:23:20.829233+0000","last_became_peered":"2024-03-06T07:23:20.829233+0000","last_unstale":"2024-03-06T07:23:21.927006+0000","last_undegraded":"2024-03-06T07:23:21.927006+0000","last_fullsized":"2024-03-06T07:23:21.927006+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:12.845913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.926942+0000","last_change":"2024-03-06T07:23:20.824645+0000","last_active":"2024-03-06T07:23:21.926942+0000","last_peered":"2024-03-06T07:23:21.926942+0000","last_clean":"2024-03-06T07:23:21.926942+0000","last_became_active":"2024-03-06T07:23:20.824385+0000","last_became_peered":"2024-03-06T07:23:20.824385+0000","last_unstale":"2024-03-06T07:23:21.926942+0000","last_undegraded":"2024-03-06T07:23:21.926942+0000","last_fullsized":"2024-03-06T07:23:21.926942+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:41.937379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816592+0000","last_change":"2024-03-06T07:23:20.823326+0000","last_active":"2024-03-06T07:23:21.816592+0000","last_peered":"2024-03-06T07:23:21.816592+0000","last_clean":"2024-03-06T07:23:21.816592+0000","last_became_active":"2024-03-06T07:23:20.822842+0000","last_became_peered":"2024-03-06T07:23:20.822842+0000","last_unstale":"2024-03-06T07:23:21.816592+0000","last_undegraded":"2024-03-06T07:23:21.816592+0000","last_fullsized":"2024-03-06T07:23:21.816592+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:58.422844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927336+0000","last_change":"2024-03-06T07:23:20.828196+0000","last_active":"2024-03-06T07:23:21.927336+0000","last_peered":"2024-03-06T07:23:21.927336+0000","last_clean":"2024-03-06T07:23:21.927336+0000","last_became_active":"2024-03-06T07:23:20.827920+0000","last_became_peered":"2024-03-06T07:23:20.827920+0000","last_unstale":"2024-03-06T07:23:21.927336+0000","last_undegraded":"2024-03-06T07:23:21.927336+0000","last_fullsized":"2024-03-06T07:23:21.927336+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:34.007709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927853+0000","last_change":"2024-03-06T07:23:20.823387+0000","last_active":"2024-03-06T07:23:21.927853+0000","last_peered":"2024-03-06T07:23:21.927853+0000","last_clean":"2024-03-06T07:23:21.927853+0000","last_became_active":"2024-03-06T07:23:20.823117+0000","last_became_peered":"2024-03-06T07:23:20.823117+0000","last_unstale":"2024-03-06T07:23:21.927853+0000","last_undegraded":"2024-03-06T07:23:21.927853+0000","last_fullsized":"2024-03-06T07:23:21.927853+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:26.473077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815624+0000","last_change":"2024-03-06T07:23:20.823473+0000","last_active":"2024-03-06T07:23:21.815624+0000","last_peered":"2024-03-06T07:23:21.815624+0000","last_clean":"2024-03-06T07:23:21.815624+0000","last_became_active":"2024-03-06T07:23:20.822556+0000","last_became_peered":"2024-03-06T07:23:20.822556+0000","last_unstale":"2024-03-06T07:23:21.815624+0000","last_undegraded":"2024-03-06T07:23:21.815624+0000","last_fullsized":"2024-03-06T07:23:21.815624+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:44.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814153+0000","last_change":"2024-03-06T07:23:20.829495+0000","last_active":"2024-03-06T07:23:21.814153+0000","last_peered":"2024-03-06T07:23:21.814153+0000","last_clean":"2024-03-06T07:23:21.814153+0000","last_became_active":"2024-03-06T07:23:20.829216+0000","last_became_peered":"2024-03-06T07:23:20.829216+0000","last_unstale":"2024-03-06T07:23:21.814153+0000","last_undegraded":"2024-03-06T07:23:21.814153+0000","last_fullsized":"2024-03-06T07:23:21.814153+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:51.539447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"134'4","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928064+0000","last_change":"2024-03-06T07:23:19.819221+0000","last_active":"2024-03-06T07:23:21.928064+0000","last_peered":"2024-03-06T07:23:21.928064+0000","last_clean":"2024-03-06T07:23:21.928064+0000","last_became_active":"2024-03-06T07:23:19.818971+0000","last_became_peered":"2024-03-06T07:23:19.818971+0000","last_unstale":"2024-03-06T07:23:21.928064+0000","last_undegraded":"2024-03-06T07:23:21.928064+0000","last_fullsized":"2024-03-06T07:23:21.928064+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:20.118726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814279+0000","last_change":"2024-03-06T07:23:20.831095+0000","last_active":"2024-03-06T07:23:21.814279+0000","last_peered":"2024-03-06T07:23:21.814279+0000","last_clean":"2024-03-06T07:23:21.814279+0000","last_became_active":"2024-03-06T07:23:20.830835+0000","last_became_peered":"2024-03-06T07:23:20.830835+0000","last_unstale":"2024-03-06T07:23:21.814279+0000","last_undegraded":"2024-03-06T07:23:21.814279+0000","last_fullsized":"2024-03-06T07:23:21.814279+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:29.260882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835995+0000","last_change":"2024-03-06T07:23:19.822328+0000","last_active":"2024-03-06T07:23:21.835995+0000","last_peered":"2024-03-06T07:23:21.835995+0000","last_clean":"2024-03-06T07:23:21.835995+0000","last_became_active":"2024-03-06T07:23:19.821927+0000","last_became_peered":"2024-03-06T07:23:19.821927+0000","last_unstale":"2024-03-06T07:23:21.835995+0000","last_undegraded":"2024-03-06T07:23:21.835995+0000","last_fullsized":"2024-03-06T07:23:21.835995+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:53.582233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816739+0000","last_change":"2024-03-06T07:23:20.826012+0000","last_active":"2024-03-06T07:23:21.816739+0000","last_peered":"2024-03-06T07:23:21.816739+0000","last_clean":"2024-03-06T07:23:21.816739+0000","last_became_active":"2024-03-06T07:23:20.825726+0000","last_became_peered":"2024-03-06T07:23:20.825726+0000","last_unstale":"2024-03-06T07:23:21.816739+0000","last_undegraded":"2024-03-06T07:23:21.816739+0000","last_fullsized":"2024-03-06T07:23:21.816739+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:51.252428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836259+0000","last_change":"2024-03-06T07:23:19.822963+0000","last_active":"2024-03-06T07:23:21.836259+0000","last_peered":"2024-03-06T07:23:21.836259+0000","last_clean":"2024-03-06T07:23:21.836259+0000","last_became_active":"2024-03-06T07:23:19.822663+0000","last_became_peered":"2024-03-06T07:23:19.822663+0000","last_unstale":"2024-03-06T07:23:21.836259+0000","last_undegraded":"2024-03-06T07:23:21.836259+0000","last_fullsized":"2024-03-06T07:23:21.836259+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:49.157011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836191+0000","last_change":"2024-03-06T07:23:20.835612+0000","last_active":"2024-03-06T07:23:21.836191+0000","last_peered":"2024-03-06T07:23:21.836191+0000","last_clean":"2024-03-06T07:23:21.836191+0000","last_became_active":"2024-03-06T07:23:20.835350+0000","last_became_peered":"2024-03-06T07:23:20.835350+0000","last_unstale":"2024-03-06T07:23:21.836191+0000","last_undegraded":"2024-03-06T07:23:21.836191+0000","last_fullsized":"2024-03-06T07:23:21.836191+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:02.387042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814219+0000","last_change":"2024-03-06T07:23:19.814213+0000","last_active":"2024-03-06T07:23:21.814219+0000","last_peered":"2024-03-06T07:23:21.814219+0000","last_clean":"2024-03-06T07:23:21.814219+0000","last_became_active":"2024-03-06T07:23:19.813940+0000","last_became_peered":"2024-03-06T07:23:19.813940+0000","last_unstale":"2024-03-06T07:23:21.814219+0000","last_undegraded":"2024-03-06T07:23:21.814219+0000","last_fullsized":"2024-03-06T07:23:21.814219+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:16.681332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927210+0000","last_change":"2024-03-06T07:23:20.822303+0000","last_active":"2024-03-06T07:23:21.927210+0000","last_peered":"2024-03-06T07:23:21.927210+0000","last_clean":"2024-03-06T07:23:21.927210+0000","last_became_active":"2024-03-06T07:23:20.821635+0000","last_became_peered":"2024-03-06T07:23:20.821635+0000","last_unstale":"2024-03-06T07:23:21.927210+0000","last_undegraded":"2024-03-06T07:23:21.927210+0000","last_fullsized":"2024-03-06T07:23:21.927210+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:51.539447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816639+0000","last_change":"2024-03-06T07:23:19.814612+0000","last_active":"2024-03-06T07:23:21.816639+0000","last_peered":"2024-03-06T07:23:21.816639+0000","last_clean":"2024-03-06T07:23:21.816639+0000","last_became_active":"2024-03-06T07:23:19.814108+0000","last_became_peered":"2024-03-06T07:23:19.814108+0000","last_unstale":"2024-03-06T07:23:21.816639+0000","last_undegraded":"2024-03-06T07:23:21.816639+0000","last_fullsized":"2024-03-06T07:23:21.816639+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:14.655226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815888+0000","last_change":"2024-03-06T07:23:20.824411+0000","last_active":"2024-03-06T07:23:21.815888+0000","last_peered":"2024-03-06T07:23:21.815888+0000","last_clean":"2024-03-06T07:23:21.815888+0000","last_became_active":"2024-03-06T07:23:20.824153+0000","last_became_peered":"2024-03-06T07:23:20.824153+0000","last_unstale":"2024-03-06T07:23:21.815888+0000","last_undegraded":"2024-03-06T07:23:21.815888+0000","last_fullsized":"2024-03-06T07:23:21.815888+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:51.252428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927940+0000","last_change":"2024-03-06T07:23:19.821284+0000","last_active":"2024-03-06T07:23:21.927940+0000","last_peered":"2024-03-06T07:23:21.927940+0000","last_clean":"2024-03-06T07:23:21.927940+0000","last_became_active":"2024-03-06T07:23:19.820988+0000","last_became_peered":"2024-03-06T07:23:19.820988+0000","last_unstale":"2024-03-06T07:23:21.927940+0000","last_undegraded":"2024-03-06T07:23:21.927940+0000","last_fullsized":"2024-03-06T07:23:21.927940+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:49.172272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.813939+0000","last_change":"2024-03-06T07:23:20.826258+0000","last_active":"2024-03-06T07:23:21.813939+0000","last_peered":"2024-03-06T07:23:21.813939+0000","last_clean":"2024-03-06T07:23:21.813939+0000","last_became_active":"2024-03-06T07:23:20.825979+0000","last_became_peered":"2024-03-06T07:23:20.825979+0000","last_unstale":"2024-03-06T07:23:21.813939+0000","last_undegraded":"2024-03-06T07:23:21.813939+0000","last_fullsized":"2024-03-06T07:23:21.813939+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:25.362417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831518+0000","last_change":"2024-03-06T07:23:19.820573+0000","last_active":"2024-03-06T07:23:21.831518+0000","last_peered":"2024-03-06T07:23:21.831518+0000","last_clean":"2024-03-06T07:23:21.831518+0000","last_became_active":"2024-03-06T07:23:19.820283+0000","last_became_peered":"2024-03-06T07:23:19.820283+0000","last_unstale":"2024-03-06T07:23:21.831518+0000","last_undegraded":"2024-03-06T07:23:21.831518+0000","last_fullsized":"2024-03-06T07:23:21.831518+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:43.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831689+0000","last_change":"2024-03-06T07:23:20.828026+0000","last_active":"2024-03-06T07:23:21.831689+0000","last_peered":"2024-03-06T07:23:21.831689+0000","last_clean":"2024-03-06T07:23:21.831689+0000","last_became_active":"2024-03-06T07:23:20.827770+0000","last_became_peered":"2024-03-06T07:23:20.827770+0000","last_unstale":"2024-03-06T07:23:21.831689+0000","last_undegraded":"2024-03-06T07:23:21.831689+0000","last_fullsized":"2024-03-06T07:23:21.831689+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:57.608000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927970+0000","last_change":"2024-03-06T07:23:19.822141+0000","last_active":"2024-03-06T07:23:21.927970+0000","last_peered":"2024-03-06T07:23:21.927970+0000","last_clean":"2024-03-06T07:23:21.927970+0000","last_became_active":"2024-03-06T07:23:19.821888+0000","last_became_peered":"2024-03-06T07:23:19.821888+0000","last_unstale":"2024-03-06T07:23:21.927970+0000","last_undegraded":"2024-03-06T07:23:21.927970+0000","last_fullsized":"2024-03-06T07:23:21.927970+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:13.643734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816600+0000","last_change":"2024-03-06T07:23:20.830015+0000","last_active":"2024-03-06T07:23:21.816600+0000","last_peered":"2024-03-06T07:23:21.816600+0000","last_clean":"2024-03-06T07:23:21.816600+0000","last_became_active":"2024-03-06T07:23:20.829751+0000","last_became_peered":"2024-03-06T07:23:20.829751+0000","last_unstale":"2024-03-06T07:23:21.816600+0000","last_undegraded":"2024-03-06T07:23:21.816600+0000","last_fullsized":"2024-03-06T07:23:21.816600+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:11.035847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836199+0000","last_change":"2024-03-06T07:23:19.818446+0000","last_active":"2024-03-06T07:23:21.836199+0000","last_peered":"2024-03-06T07:23:21.836199+0000","last_clean":"2024-03-06T07:23:21.836199+0000","last_became_active":"2024-03-06T07:23:19.818187+0000","last_became_peered":"2024-03-06T07:23:19.818187+0000","last_unstale":"2024-03-06T07:23:21.836199+0000","last_undegraded":"2024-03-06T07:23:21.836199+0000","last_fullsized":"2024-03-06T07:23:21.836199+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:24.357585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927758+0000","last_change":"2024-03-06T07:23:20.830670+0000","last_active":"2024-03-06T07:23:21.927758+0000","last_peered":"2024-03-06T07:23:21.927758+0000","last_clean":"2024-03-06T07:23:21.927758+0000","last_became_active":"2024-03-06T07:23:20.830068+0000","last_became_peered":"2024-03-06T07:23:20.830068+0000","last_unstale":"2024-03-06T07:23:21.927758+0000","last_undegraded":"2024-03-06T07:23:21.927758+0000","last_fullsized":"2024-03-06T07:23:21.927758+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:39.061517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.832024+0000","last_change":"2024-03-06T07:23:19.821359+0000","last_active":"2024-03-06T07:23:21.832024+0000","last_peered":"2024-03-06T07:23:21.832024+0000","last_clean":"2024-03-06T07:23:21.832024+0000","last_became_active":"2024-03-06T07:23:19.821086+0000","last_became_peered":"2024-03-06T07:23:19.821086+0000","last_unstale":"2024-03-06T07:23:21.832024+0000","last_undegraded":"2024-03-06T07:23:21.832024+0000","last_fullsized":"2024-03-06T07:23:21.832024+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:10.031014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927371+0000","last_change":"2024-03-06T07:23:20.837942+0000","last_active":"2024-03-06T07:23:21.927371+0000","last_peered":"2024-03-06T07:23:21.927371+0000","last_clean":"2024-03-06T07:23:21.927371+0000","last_became_active":"2024-03-06T07:23:20.837637+0000","last_became_peered":"2024-03-06T07:23:20.837637+0000","last_unstale":"2024-03-06T07:23:21.927371+0000","last_undegraded":"2024-03-06T07:23:21.927371+0000","last_fullsized":"2024-03-06T07:23:21.927371+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:29.772363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"134'3","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927849+0000","last_change":"2024-03-06T07:23:19.812634+0000","last_active":"2024-03-06T07:23:21.927849+0000","last_peered":"2024-03-06T07:23:21.927849+0000","last_clean":"2024-03-06T07:23:21.927849+0000","last_became_active":"2024-03-06T07:23:19.812378+0000","last_became_peered":"2024-03-06T07:23:19.812378+0000","last_unstale":"2024-03-06T07:23:21.927849+0000","last_undegraded":"2024-03-06T07:23:21.927849+0000","last_fullsized":"2024-03-06T07:23:21.927849+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:18.417628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815268+0000","last_change":"2024-03-06T07:23:20.829353+0000","last_active":"2024-03-06T07:23:21.815268+0000","last_peered":"2024-03-06T07:23:21.815268+0000","last_clean":"2024-03-06T07:23:21.815268+0000","last_became_active":"2024-03-06T07:23:20.829066+0000","last_became_peered":"2024-03-06T07:23:20.829066+0000","last_unstale":"2024-03-06T07:23:21.815268+0000","last_undegraded":"2024-03-06T07:23:21.815268+0000","last_fullsized":"2024-03-06T07:23:21.815268+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:26.273208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927710+0000","last_change":"2024-03-06T07:23:19.817081+0000","last_active":"2024-03-06T07:23:21.927710+0000","last_peered":"2024-03-06T07:23:21.927710+0000","last_clean":"2024-03-06T07:23:21.927710+0000","last_became_active":"2024-03-06T07:23:19.816830+0000","last_became_peered":"2024-03-06T07:23:19.816830+0000","last_unstale":"2024-03-06T07:23:21.927710+0000","last_undegraded":"2024-03-06T07:23:21.927710+0000","last_fullsized":"2024-03-06T07:23:21.927710+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:28.256050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.813980+0000","last_change":"2024-03-06T07:23:20.832427+0000","last_active":"2024-03-06T07:23:21.813980+0000","last_peered":"2024-03-06T07:23:21.813980+0000","last_clean":"2024-03-06T07:23:21.813980+0000","last_became_active":"2024-03-06T07:23:20.830865+0000","last_became_peered":"2024-03-06T07:23:20.830865+0000","last_unstale":"2024-03-06T07:23:21.813980+0000","last_undegraded":"2024-03-06T07:23:21.813980+0000","last_fullsized":"2024-03-06T07:23:21.813980+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:50.161843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815481+0000","last_change":"2024-03-06T07:23:19.815723+0000","last_active":"2024-03-06T07:23:21.815481+0000","last_peered":"2024-03-06T07:23:21.815481+0000","last_clean":"2024-03-06T07:23:21.815481+0000","last_became_active":"2024-03-06T07:23:19.815374+0000","last_became_peered":"2024-03-06T07:23:19.815374+0000","last_unstale":"2024-03-06T07:23:21.815481+0000","last_undegraded":"2024-03-06T07:23:21.815481+0000","last_fullsized":"2024-03-06T07:23:21.815481+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:08.771871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927807+0000","last_change":"2024-03-06T07:23:20.821930+0000","last_active":"2024-03-06T07:23:21.927807+0000","last_peered":"2024-03-06T07:23:21.927807+0000","last_clean":"2024-03-06T07:23:21.927807+0000","last_became_active":"2024-03-06T07:23:20.820943+0000","last_became_peered":"2024-03-06T07:23:20.820943+0000","last_unstale":"2024-03-06T07:23:21.927807+0000","last_undegraded":"2024-03-06T07:23:21.927807+0000","last_fullsized":"2024-03-06T07:23:21.927807+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:55.112513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815987+0000","last_change":"2024-03-06T07:23:19.817408+0000","last_active":"2024-03-06T07:23:21.815987+0000","last_peered":"2024-03-06T07:23:21.815987+0000","last_clean":"2024-03-06T07:23:21.815987+0000","last_became_active":"2024-03-06T07:23:19.817072+0000","last_became_peered":"2024-03-06T07:23:19.817072+0000","last_unstale":"2024-03-06T07:23:21.815987+0000","last_undegraded":"2024-03-06T07:23:21.815987+0000","last_fullsized":"2024-03-06T07:23:21.815987+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:24.931887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.813988+0000","last_change":"2024-03-06T07:23:20.824640+0000","last_active":"2024-03-06T07:23:21.813988+0000","last_peered":"2024-03-06T07:23:21.813988+0000","last_clean":"2024-03-06T07:23:21.813988+0000","last_became_active":"2024-03-06T07:23:20.824357+0000","last_became_peered":"2024-03-06T07:23:20.824357+0000","last_unstale":"2024-03-06T07:23:21.813988+0000","last_undegraded":"2024-03-06T07:23:21.813988+0000","last_fullsized":"2024-03-06T07:23:21.813988+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:11.943936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"134'8","reported_seq":22,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831458+0000","last_change":"2024-03-06T07:23:19.822083+0000","last_active":"2024-03-06T07:23:21.831458+0000","last_peered":"2024-03-06T07:23:21.831458+0000","last_clean":"2024-03-06T07:23:21.831458+0000","last_became_active":"2024-03-06T07:23:19.821768+0000","last_became_peered":"2024-03-06T07:23:19.821768+0000","last_unstale":"2024-03-06T07:23:21.831458+0000","last_undegraded":"2024-03-06T07:23:21.831458+0000","last_fullsized":"2024-03-06T07:23:21.831458+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:57.418012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835959+0000","last_change":"2024-03-06T07:23:20.828815+0000","last_active":"2024-03-06T07:23:21.835959+0000","last_peered":"2024-03-06T07:23:21.835959+0000","last_clean":"2024-03-06T07:23:21.835959+0000","last_became_active":"2024-03-06T07:23:20.828422+0000","last_became_peered":"2024-03-06T07:23:20.828422+0000","last_unstale":"2024-03-06T07:23:21.835959+0000","last_undegraded":"2024-03-06T07:23:21.835959+0000","last_fullsized":"2024-03-06T07:23:21.835959+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:14.592219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927475+0000","last_change":"2024-03-06T07:23:19.821216+0000","last_active":"2024-03-06T07:23:21.927475+0000","last_peered":"2024-03-06T07:23:21.927475+0000","last_clean":"2024-03-06T07:23:21.927475+0000","last_became_active":"2024-03-06T07:23:19.820903+0000","last_became_peered":"2024-03-06T07:23:19.820903+0000","last_unstale":"2024-03-06T07:23:21.927475+0000","last_undegraded":"2024-03-06T07:23:21.927475+0000","last_fullsized":"2024-03-06T07:23:21.927475+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:02.459037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836190+0000","last_change":"2024-03-06T07:23:20.830697+0000","last_active":"2024-03-06T07:23:21.836190+0000","last_peered":"2024-03-06T07:23:21.836190+0000","last_clean":"2024-03-06T07:23:21.836190+0000","last_became_active":"2024-03-06T07:23:20.829015+0000","last_became_peered":"2024-03-06T07:23:20.829015+0000","last_unstale":"2024-03-06T07:23:21.836190+0000","last_undegraded":"2024-03-06T07:23:21.836190+0000","last_fullsized":"2024-03-06T07:23:21.836190+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:30.818350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836033+0000","last_change":"2024-03-06T07:23:19.820024+0000","last_active":"2024-03-06T07:23:21.836033+0000","last_peered":"2024-03-06T07:23:21.836033+0000","last_clean":"2024-03-06T07:23:21.836033+0000","last_became_active":"2024-03-06T07:23:19.819703+0000","last_became_peered":"2024-03-06T07:23:19.819703+0000","last_unstale":"2024-03-06T07:23:21.836033+0000","last_undegraded":"2024-03-06T07:23:21.836033+0000","last_fullsized":"2024-03-06T07:23:21.836033+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:06.015217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831952+0000","last_change":"2024-03-06T07:23:20.839134+0000","last_active":"2024-03-06T07:23:21.831952+0000","last_peered":"2024-03-06T07:23:21.831952+0000","last_clean":"2024-03-06T07:23:21.831952+0000","last_became_active":"2024-03-06T07:23:20.838836+0000","last_became_peered":"2024-03-06T07:23:20.838836+0000","last_unstale":"2024-03-06T07:23:21.831952+0000","last_undegraded":"2024-03-06T07:23:21.831952+0000","last_fullsized":"2024-03-06T07:23:21.831952+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:50.161843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814367+0000","last_change":"2024-03-06T07:23:19.815390+0000","last_active":"2024-03-06T07:23:21.814367+0000","last_peered":"2024-03-06T07:23:21.814367+0000","last_clean":"2024-03-06T07:23:21.814367+0000","last_became_active":"2024-03-06T07:23:19.815132+0000","last_became_peered":"2024-03-06T07:23:19.815132+0000","last_unstale":"2024-03-06T07:23:21.814367+0000","last_undegraded":"2024-03-06T07:23:21.814367+0000","last_fullsized":"2024-03-06T07:23:21.814367+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:25.300694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.832033+0000","last_change":"2024-03-06T07:23:20.832539+0000","last_active":"2024-03-06T07:23:21.832033+0000","last_peered":"2024-03-06T07:23:21.832033+0000","last_clean":"2024-03-06T07:23:21.832033+0000","last_became_active":"2024-03-06T07:23:20.832135+0000","last_became_peered":"2024-03-06T07:23:20.832135+0000","last_unstale":"2024-03-06T07:23:21.832033+0000","last_undegraded":"2024-03-06T07:23:21.832033+0000","last_fullsized":"2024-03-06T07:23:21.832033+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:11.943936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815783+0000","last_change":"2024-03-06T07:23:19.815651+0000","last_active":"2024-03-06T07:23:21.815783+0000","last_peered":"2024-03-06T07:23:21.815783+0000","last_clean":"2024-03-06T07:23:21.815783+0000","last_became_active":"2024-03-06T07:23:19.815351+0000","last_became_peered":"2024-03-06T07:23:19.815351+0000","last_unstale":"2024-03-06T07:23:21.815783+0000","last_undegraded":"2024-03-06T07:23:21.815783+0000","last_fullsized":"2024-03-06T07:23:21.815783+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:23.717250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835951+0000","last_change":"2024-03-06T07:23:20.823541+0000","last_active":"2024-03-06T07:23:21.835951+0000","last_peered":"2024-03-06T07:23:21.835951+0000","last_clean":"2024-03-06T07:23:21.835951+0000","last_became_active":"2024-03-06T07:23:20.823235+0000","last_became_peered":"2024-03-06T07:23:20.823235+0000","last_unstale":"2024-03-06T07:23:21.835951+0000","last_undegraded":"2024-03-06T07:23:21.835951+0000","last_fullsized":"2024-03-06T07:23:21.835951+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:21.123558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927506+0000","last_change":"2024-03-06T07:23:19.815230+0000","last_active":"2024-03-06T07:23:21.927506+0000","last_peered":"2024-03-06T07:23:21.927506+0000","last_clean":"2024-03-06T07:23:21.927506+0000","last_became_active":"2024-03-06T07:23:19.814945+0000","last_became_peered":"2024-03-06T07:23:19.814945+0000","last_unstale":"2024-03-06T07:23:21.927506+0000","last_undegraded":"2024-03-06T07:23:21.927506+0000","last_fullsized":"2024-03-06T07:23:21.927506+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:10.939104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814084+0000","last_change":"2024-03-06T07:23:20.827053+0000","last_active":"2024-03-06T07:23:21.814084+0000","last_peered":"2024-03-06T07:23:21.814084+0000","last_clean":"2024-03-06T07:23:21.814084+0000","last_became_active":"2024-03-06T07:23:20.826752+0000","last_became_peered":"2024-03-06T07:23:20.826752+0000","last_unstale":"2024-03-06T07:23:21.814084+0000","last_undegraded":"2024-03-06T07:23:21.814084+0000","last_fullsized":"2024-03-06T07:23:21.814084+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:15.340945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815978+0000","last_change":"2024-03-06T07:23:19.812535+0000","last_active":"2024-03-06T07:23:21.815978+0000","last_peered":"2024-03-06T07:23:21.815978+0000","last_clean":"2024-03-06T07:23:21.815978+0000","last_became_active":"2024-03-06T07:23:19.812215+0000","last_became_peered":"2024-03-06T07:23:19.812215+0000","last_unstale":"2024-03-06T07:23:21.815978+0000","last_undegraded":"2024-03-06T07:23:21.815978+0000","last_fullsized":"2024-03-06T07:23:21.815978+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:34.214074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835933+0000","last_change":"2024-03-06T07:23:20.827750+0000","last_active":"2024-03-06T07:23:21.835933+0000","last_peered":"2024-03-06T07:23:21.835933+0000","last_clean":"2024-03-06T07:23:21.835933+0000","last_became_active":"2024-03-06T07:23:20.827438+0000","last_became_peered":"2024-03-06T07:23:20.827438+0000","last_unstale":"2024-03-06T07:23:21.835933+0000","last_undegraded":"2024-03-06T07:23:21.835933+0000","last_fullsized":"2024-03-06T07:23:21.835933+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:29.260882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928200+0000","last_change":"2024-03-06T07:23:19.814485+0000","last_active":"2024-03-06T07:23:21.928200+0000","last_peered":"2024-03-06T07:23:21.928200+0000","last_clean":"2024-03-06T07:23:21.928200+0000","last_became_active":"2024-03-06T07:23:19.814240+0000","last_became_peered":"2024-03-06T07:23:19.814240+0000","last_unstale":"2024-03-06T07:23:21.928200+0000","last_undegraded":"2024-03-06T07:23:21.928200+0000","last_fullsized":"2024-03-06T07:23:21.928200+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:55.295513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814146+0000","last_change":"2024-03-06T07:23:20.831880+0000","last_active":"2024-03-06T07:23:21.814146+0000","last_peered":"2024-03-06T07:23:21.814146+0000","last_clean":"2024-03-06T07:23:21.814146+0000","last_became_active":"2024-03-06T07:23:20.830420+0000","last_became_peered":"2024-03-06T07:23:20.830420+0000","last_unstale":"2024-03-06T07:23:21.814146+0000","last_undegraded":"2024-03-06T07:23:21.814146+0000","last_fullsized":"2024-03-06T07:23:21.814146+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:14.648567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815536+0000","last_change":"2024-03-06T07:23:19.813521+0000","last_active":"2024-03-06T07:23:21.815536+0000","last_peered":"2024-03-06T07:23:21.815536+0000","last_clean":"2024-03-06T07:23:21.815536+0000","last_became_active":"2024-03-06T07:23:19.813250+0000","last_became_peered":"2024-03-06T07:23:19.813250+0000","last_unstale":"2024-03-06T07:23:21.815536+0000","last_undegraded":"2024-03-06T07:23:21.815536+0000","last_fullsized":"2024-03-06T07:23:21.815536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:06.281792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928292+0000","last_change":"2024-03-06T07:23:20.834654+0000","last_active":"2024-03-06T07:23:21.928292+0000","last_peered":"2024-03-06T07:23:21.928292+0000","last_clean":"2024-03-06T07:23:21.928292+0000","last_became_active":"2024-03-06T07:23:20.834179+0000","last_became_peered":"2024-03-06T07:23:20.834179+0000","last_unstale":"2024-03-06T07:23:21.928292+0000","last_undegraded":"2024-03-06T07:23:21.928292+0000","last_fullsized":"2024-03-06T07:23:21.928292+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:56.219442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816449+0000","last_change":"2024-03-06T07:23:19.816231+0000","last_active":"2024-03-06T07:23:21.816449+0000","last_peered":"2024-03-06T07:23:21.816449+0000","last_clean":"2024-03-06T07:23:21.816449+0000","last_became_active":"2024-03-06T07:23:19.815988+0000","last_became_peered":"2024-03-06T07:23:19.815988+0000","last_unstale":"2024-03-06T07:23:21.816449+0000","last_undegraded":"2024-03-06T07:23:21.816449+0000","last_fullsized":"2024-03-06T07:23:21.816449+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:14.032486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816054+0000","last_change":"2024-03-06T07:23:20.833907+0000","last_active":"2024-03-06T07:23:21.816054+0000","last_peered":"2024-03-06T07:23:21.816054+0000","last_clean":"2024-03-06T07:23:21.816054+0000","last_became_active":"2024-03-06T07:23:20.833642+0000","last_became_peered":"2024-03-06T07:23:20.833642+0000","last_unstale":"2024-03-06T07:23:21.816054+0000","last_undegraded":"2024-03-06T07:23:21.816054+0000","last_fullsized":"2024-03-06T07:23:21.816054+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:26.305527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"135'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.833226+0000","last_change":"2024-03-06T07:23:19.821237+0000","last_active":"2024-03-06T07:23:21.833226+0000","last_peered":"2024-03-06T07:23:21.833226+0000","last_clean":"2024-03-06T07:23:21.833226+0000","last_became_active":"2024-03-06T07:23:19.820953+0000","last_became_peered":"2024-03-06T07:23:19.820953+0000","last_unstale":"2024-03-06T07:23:21.833226+0000","last_undegraded":"2024-03-06T07:23:21.833226+0000","last_fullsized":"2024-03-06T07:23:21.833226+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:50.247595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.832031+0000","last_change":"2024-03-06T07:23:20.834169+0000","last_active":"2024-03-06T07:23:21.832031+0000","last_peered":"2024-03-06T07:23:21.832031+0000","last_clean":"2024-03-06T07:23:21.832031+0000","last_became_active":"2024-03-06T07:23:20.833884+0000","last_became_peered":"2024-03-06T07:23:20.833884+0000","last_unstale":"2024-03-06T07:23:21.832031+0000","last_undegraded":"2024-03-06T07:23:21.832031+0000","last_fullsized":"2024-03-06T07:23:21.832031+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:15.340945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814364+0000","last_change":"2024-03-06T07:23:19.813583+0000","last_active":"2024-03-06T07:23:21.814364+0000","last_peered":"2024-03-06T07:23:21.814364+0000","last_clean":"2024-03-06T07:23:21.814364+0000","last_became_active":"2024-03-06T07:23:19.813329+0000","last_became_peered":"2024-03-06T07:23:19.813329+0000","last_unstale":"2024-03-06T07:23:21.814364+0000","last_undegraded":"2024-03-06T07:23:21.814364+0000","last_fullsized":"2024-03-06T07:23:21.814364+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:17.707602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928054+0000","last_change":"2024-03-06T07:23:20.835301+0000","last_active":"2024-03-06T07:23:21.928054+0000","last_peered":"2024-03-06T07:23:21.928054+0000","last_clean":"2024-03-06T07:23:21.928054+0000","last_became_active":"2024-03-06T07:23:20.835035+0000","last_became_peered":"2024-03-06T07:23:20.835035+0000","last_unstale":"2024-03-06T07:23:21.928054+0000","last_undegraded":"2024-03-06T07:23:21.928054+0000","last_fullsized":"2024-03-06T07:23:21.928054+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:56.836724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928321+0000","last_change":"2024-03-06T07:23:19.820735+0000","last_active":"2024-03-06T07:23:21.928321+0000","last_peered":"2024-03-06T07:23:21.928321+0000","last_clean":"2024-03-06T07:23:21.928321+0000","last_became_active":"2024-03-06T07:23:19.820274+0000","last_became_peered":"2024-03-06T07:23:19.820274+0000","last_unstale":"2024-03-06T07:23:21.928321+0000","last_undegraded":"2024-03-06T07:23:21.928321+0000","last_fullsized":"2024-03-06T07:23:21.928321+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:50.534615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815400+0000","last_change":"2024-03-06T07:23:20.825424+0000","last_active":"2024-03-06T07:23:21.815400+0000","last_peered":"2024-03-06T07:23:21.815400+0000","last_clean":"2024-03-06T07:23:21.815400+0000","last_became_active":"2024-03-06T07:23:20.825147+0000","last_became_peered":"2024-03-06T07:23:20.825147+0000","last_unstale":"2024-03-06T07:23:21.815400+0000","last_undegraded":"2024-03-06T07:23:21.815400+0000","last_fullsized":"2024-03-06T07:23:21.815400+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:57.519651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831684+0000","last_change":"2024-03-06T07:23:19.818560+0000","last_active":"2024-03-06T07:23:21.831684+0000","last_peered":"2024-03-06T07:23:21.831684+0000","last_clean":"2024-03-06T07:23:21.831684+0000","last_became_active":"2024-03-06T07:23:19.818303+0000","last_became_peered":"2024-03-06T07:23:19.818303+0000","last_unstale":"2024-03-06T07:23:21.831684+0000","last_undegraded":"2024-03-06T07:23:21.831684+0000","last_fullsized":"2024-03-06T07:23:21.831684+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:20.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815924+0000","last_change":"2024-03-06T07:23:20.837173+0000","last_active":"2024-03-06T07:23:21.815924+0000","last_peered":"2024-03-06T07:23:21.815924+0000","last_clean":"2024-03-06T07:23:21.815924+0000","last_became_active":"2024-03-06T07:23:20.836902+0000","last_became_peered":"2024-03-06T07:23:20.836902+0000","last_unstale":"2024-03-06T07:23:21.815924+0000","last_undegraded":"2024-03-06T07:23:21.815924+0000","last_fullsized":"2024-03-06T07:23:21.815924+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:17.686165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927428+0000","last_change":"2024-03-06T07:23:19.815802+0000","last_active":"2024-03-06T07:23:21.927428+0000","last_peered":"2024-03-06T07:23:21.927428+0000","last_clean":"2024-03-06T07:23:21.927428+0000","last_became_active":"2024-03-06T07:23:19.815539+0000","last_became_peered":"2024-03-06T07:23:19.815539+0000","last_unstale":"2024-03-06T07:23:21.927428+0000","last_undegraded":"2024-03-06T07:23:21.927428+0000","last_fullsized":"2024-03-06T07:23:21.927428+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:41.017474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814116+0000","last_change":"2024-03-06T07:23:20.826890+0000","last_active":"2024-03-06T07:23:21.814116+0000","last_peered":"2024-03-06T07:23:21.814116+0000","last_clean":"2024-03-06T07:23:21.814116+0000","last_became_active":"2024-03-06T07:23:20.826612+0000","last_became_peered":"2024-03-06T07:23:20.826612+0000","last_unstale":"2024-03-06T07:23:21.814116+0000","last_undegraded":"2024-03-06T07:23:21.814116+0000","last_fullsized":"2024-03-06T07:23:21.814116+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:56.300346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831647+0000","last_change":"2024-03-06T07:23:19.818290+0000","last_active":"2024-03-06T07:23:21.831647+0000","last_peered":"2024-03-06T07:23:21.831647+0000","last_clean":"2024-03-06T07:23:21.831647+0000","last_became_active":"2024-03-06T07:23:19.818009+0000","last_became_peered":"2024-03-06T07:23:19.818009+0000","last_unstale":"2024-03-06T07:23:21.831647+0000","last_undegraded":"2024-03-06T07:23:21.831647+0000","last_fullsized":"2024-03-06T07:23:21.831647+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:00.056867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816386+0000","last_change":"2024-03-06T07:23:20.823101+0000","last_active":"2024-03-06T07:23:21.816386+0000","last_peered":"2024-03-06T07:23:21.816386+0000","last_clean":"2024-03-06T07:23:21.816386+0000","last_became_active":"2024-03-06T07:23:20.822481+0000","last_became_peered":"2024-03-06T07:23:20.822481+0000","last_unstale":"2024-03-06T07:23:21.816386+0000","last_undegraded":"2024-03-06T07:23:21.816386+0000","last_fullsized":"2024-03-06T07:23:21.816386+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:21.355665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835868+0000","last_change":"2024-03-06T07:23:19.817553+0000","last_active":"2024-03-06T07:23:21.835868+0000","last_peered":"2024-03-06T07:23:21.835868+0000","last_clean":"2024-03-06T07:23:21.835868+0000","last_became_active":"2024-03-06T07:23:19.817263+0000","last_became_peered":"2024-03-06T07:23:19.817263+0000","last_unstale":"2024-03-06T07:23:21.835868+0000","last_undegraded":"2024-03-06T07:23:21.835868+0000","last_fullsized":"2024-03-06T07:23:21.835868+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:56.603168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816828+0000","last_change":"2024-03-06T07:23:20.819636+0000","last_active":"2024-03-06T07:23:21.816828+0000","last_peered":"2024-03-06T07:23:21.816828+0000","last_clean":"2024-03-06T07:23:21.816828+0000","last_became_active":"2024-03-06T07:23:20.819333+0000","last_became_peered":"2024-03-06T07:23:20.819333+0000","last_unstale":"2024-03-06T07:23:21.816828+0000","last_undegraded":"2024-03-06T07:23:21.816828+0000","last_fullsized":"2024-03-06T07:23:21.816828+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:17.686165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"134'6","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927829+0000","last_change":"2024-03-06T07:23:19.813814+0000","last_active":"2024-03-06T07:23:21.927829+0000","last_peered":"2024-03-06T07:23:21.927829+0000","last_clean":"2024-03-06T07:23:21.927829+0000","last_became_active":"2024-03-06T07:23:19.813559+0000","last_became_peered":"2024-03-06T07:23:19.813559+0000","last_unstale":"2024-03-06T07:23:21.927829+0000","last_undegraded":"2024-03-06T07:23:21.927829+0000","last_fullsized":"2024-03-06T07:23:21.927829+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:14.336112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":386,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835931+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:23:21.835931+0000","last_peered":"2024-03-06T07:23:21.835931+0000","last_clean":"2024-03-06T07:23:21.835931+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:23:21.835931+0000","last_undegraded":"2024-03-06T07:23:21.835931+0000","last_fullsized":"2024-03-06T07:23:21.835931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1765376,"data_stored":1407416,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1699840,"data_stored":817096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1683456,"data_stored":810334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":811946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":817642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":817138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1687552,"data_stored":810404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1785856,"data_stored":1409712,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:23.800 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:23.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.935+0000 7f20e3255640 1 Processor -- start 2024-03-06T07:23:23.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.935+0000 7f20e3255640 1 -- start start 2024-03-06T07:23:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e3255640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc14f360 0x7f20dc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e3255640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f20dc0c88c0 0x7f20dc0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e3255640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20dc0c5880 0x7f20dc158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e3255640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20dc0c8dd0 con 0x7f20dc0c5880 2024-03-06T07:23:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e3255640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20dc0c73d0 con 0x7f20dc0c88c0 2024-03-06T07:23:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e3255640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0be2e0 con 0x7f20dc14f360 2024-03-06T07:23:23.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e17f2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20dc0c5880 0x7f20dc158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e17f2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20dc0c5880 0x7f20dc158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49024/0 (socket says 172.21.15.94:49024) 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e17f2640 1 -- 172.21.15.94:0/2342214802 learned_addr learned my addr 172.21.15.94:0/2342214802 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20dbfff640 1 --2- 172.21.15.94:0/2342214802 >> v2:172.21.15.145:3300/0 conn(0x7f20dc0c88c0 0x7f20dc0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e0ff1640 1 --2- 172.21.15.94:0/2342214802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc14f360 0x7f20dc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e17f2640 1 -- 172.21.15.94:0/2342214802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc14f360 msgr2=0x7f20dc155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.936+0000 7f20e17f2640 1 --2- 172.21.15.94:0/2342214802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc14f360 0x7f20dc155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.937+0000 7f20e17f2640 1 -- 172.21.15.94:0/2342214802 >> v2:172.21.15.145:3300/0 conn(0x7f20dc0c88c0 msgr2=0x7f20dc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.937+0000 7f20e17f2640 1 --2- 172.21.15.94:0/2342214802 >> v2:172.21.15.145:3300/0 conn(0x7f20dc0c88c0 0x7f20dc0c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.937+0000 7f20e17f2640 1 -- 172.21.15.94:0/2342214802 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc0c7670 con 0x7f20dc0c5880 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.937+0000 7f20e17f2640 1 --2- 172.21.15.94:0/2342214802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc0c5880 0x7f20dc158b60 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0cbac0 tx=0x7f20d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9175e06561bb3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.937+0000 7f20db7fe640 1 -- 172.21.15.94:0/2342214802 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d402c9e0 con 0x7f20dc0c5880 2024-03-06T07:23:23.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.937+0000 7f20db7fe640 1 -- 172.21.15.94:0/2342214802 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d402cb80 con 0x7f20dc0c5880 2024-03-06T07:23:23.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.937+0000 7f20db7fe640 1 -- 172.21.15.94:0/2342214802 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d40343f0 con 0x7f20dc0c5880 2024-03-06T07:23:23.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 -- 172.21.15.94:0/2342214802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc0c5880 msgr2=0x7f20dc158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 --2- 172.21.15.94:0/2342214802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc0c5880 0x7f20dc158b60 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0cbac0 tx=0x7f20d402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:23.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 -- 172.21.15.94:0/2342214802 shutdown_connections 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 --2- 172.21.15.94:0/2342214802 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc0c5880 0x7f20dc158b60 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 --2- 172.21.15.94:0/2342214802 >> v2:172.21.15.145:3300/0 conn(0x7f20dc0c88c0 0x7f20dc0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 --2- 172.21.15.94:0/2342214802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc14f360 0x7f20dc155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 -- 172.21.15.94:0/2342214802 >> 172.21.15.94:0/2342214802 conn(0x7f20dc0ac570 msgr2=0x7f20dc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 -- 172.21.15.94:0/2342214802 shutdown_connections 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.938+0000 7f20e3255640 1 -- 172.21.15.94:0/2342214802 wait complete. 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.939+0000 7f20e3255640 1 Processor -- start 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.939+0000 7f20e3255640 1 -- start start 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e3255640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20dc0c5880 0x7f20dc1ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e3255640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc0c88c0 0x7f20dc0ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e3255640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc14f360 0x7f20dc156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e3255640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc1590a0 con 0x7f20dc14f360 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e3255640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0c7c30 con 0x7f20dc0c5880 2024-03-06T07:23:23.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e3255640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc159220 con 0x7f20dc0c88c0 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc14f360 0x7f20dc156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc14f360 0x7f20dc156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49036/0 (socket says 172.21.15.94:49036) 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 -- 172.21.15.94:0/1248877759 learned_addr learned my addr 172.21.15.94:0/1248877759 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20dbfff640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc0c88c0 0x7f20dc0ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e0ff1640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20dc0c5880 0x7f20dc1ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 -- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc0c88c0 msgr2=0x7f20dc0ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc0c88c0 0x7f20dc0ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 -- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20dc0c5880 msgr2=0x7f20dc1ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20dc0c5880 0x7f20dc1ee780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.940+0000 7f20e17f2640 1 -- 172.21.15.94:0/1248877759 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc0b0560 con 0x7f20dc14f360 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.941+0000 7f20e17f2640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc14f360 0x7f20dc156ea0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f20d4036040 tx=0x7f20d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.941+0000 7f20d97fa640 1 -- 172.21.15.94:0/1248877759 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4043020 con 0x7f20dc14f360 2024-03-06T07:23:23.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.941+0000 7f20d97fa640 1 -- 172.21.15.94:0/1248877759 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d403e450 con 0x7f20dc14f360 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.941+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc0ac2d0 con 0x7f20dc14f360 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.941+0000 7f20d97fa640 1 -- 172.21.15.94:0/1248877759 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d4009510 con 0x7f20dc14f360 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.941+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc157d60 con 0x7f20dc14f360 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.942+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc04f330 con 0x7f20dc14f360 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.942+0000 7f20d97fa640 1 -- 172.21.15.94:0/1248877759 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20d4046020 con 0x7f20dc14f360 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.943+0000 7f20d97fa640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f20c006b4d0 0x7f20c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.943+0000 7f20d97fa640 1 -- 172.21.15.94:0/1248877759 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f20d40aca00 con 0x7f20dc14f360 2024-03-06T07:23:23.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.943+0000 7f20e0ff1640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f20c006b4d0 0x7f20c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.944+0000 7f20e0ff1640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f20c006b4d0 0x7f20c006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0ac540 tx=0x7f20d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:23.947+0000 7f20d97fa640 1 -- 172.21.15.94:0/1248877759 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20d404d030 con 0x7f20dc14f360 2024-03-06T07:23:24.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.118+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20dc06cda0 con 0x7f20c006b4d0 2024-03-06T07:23:24.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.124+0000 7f20d97fa640 1 -- 172.21.15.94:0/1248877759 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252562 (secure 0 0 0) 0x7f20dc06cda0 con 0x7f20c006b4d0 2024-03-06T07:23:24.130 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:24.132 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:23:24.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f20c006b4d0 msgr2=0x7f20c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f20c006b4d0 0x7f20c006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0ac540 tx=0x7f20d0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:24.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc14f360 msgr2=0x7f20dc156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc14f360 0x7f20dc156ea0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f20d4036040 tx=0x7f20d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 shutdown_connections 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f20c006b4d0 0x7f20c006d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20dc14f360 0x7f20dc156ea0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20dc0c88c0 0x7f20dc0ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 --2- 172.21.15.94:0/1248877759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20dc0c5880 0x7f20dc1ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 >> 172.21.15.94:0/1248877759 conn(0x7f20dc0ac570 msgr2=0x7f20dc0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 shutdown_connections 2024-03-06T07:23:24.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.127+0000 7f20e3255640 1 -- 172.21.15.94:0/1248877759 wait complete. 2024-03-06T07:23:24.140 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-03-06T07:23:22.700733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262880,"kb_used_data":13404,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711840,"statfs":{"total":773094113280,"available":772824924160,"internally_reserved":0,"allocated":13725696,"data_stored":7701688,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000837"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831874+0000","last_change":"2024-03-06T07:23:20.833717+0000","last_active":"2024-03-06T07:23:21.831874+0000","last_peered":"2024-03-06T07:23:21.831874+0000","last_clean":"2024-03-06T07:23:21.831874+0000","last_became_active":"2024-03-06T07:23:20.833391+0000","last_became_peered":"2024-03-06T07:23:20.833391+0000","last_unstale":"2024-03-06T07:23:21.831874+0000","last_undegraded":"2024-03-06T07:23:21.831874+0000","last_fullsized":"2024-03-06T07:23:21.831874+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:07.020050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814127+0000","last_change":"2024-03-06T07:23:20.831542+0000","last_active":"2024-03-06T07:23:21.814127+0000","last_peered":"2024-03-06T07:23:21.814127+0000","last_clean":"2024-03-06T07:23:21.814127+0000","last_became_active":"2024-03-06T07:23:20.830456+0000","last_became_peered":"2024-03-06T07:23:20.830456+0000","last_unstale":"2024-03-06T07:23:21.814127+0000","last_undegraded":"2024-03-06T07:23:21.814127+0000","last_fullsized":"2024-03-06T07:23:21.814127+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:47.901171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814272+0000","last_change":"2024-03-06T07:23:20.830517+0000","last_active":"2024-03-06T07:23:21.814272+0000","last_peered":"2024-03-06T07:23:21.814272+0000","last_clean":"2024-03-06T07:23:21.814272+0000","last_became_active":"2024-03-06T07:23:20.830209+0000","last_became_peered":"2024-03-06T07:23:20.830209+0000","last_unstale":"2024-03-06T07:23:21.814272+0000","last_undegraded":"2024-03-06T07:23:21.814272+0000","last_fullsized":"2024-03-06T07:23:21.814272+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:21.123558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815246+0000","last_change":"2024-03-06T07:23:20.823795+0000","last_active":"2024-03-06T07:23:21.815246+0000","last_peered":"2024-03-06T07:23:21.815246+0000","last_clean":"2024-03-06T07:23:21.815246+0000","last_became_active":"2024-03-06T07:23:20.823520+0000","last_became_peered":"2024-03-06T07:23:20.823520+0000","last_unstale":"2024-03-06T07:23:21.815246+0000","last_undegraded":"2024-03-06T07:23:21.815246+0000","last_fullsized":"2024-03-06T07:23:21.815246+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:21.355665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927614+0000","last_change":"2024-03-06T07:23:20.833801+0000","last_active":"2024-03-06T07:23:21.927614+0000","last_peered":"2024-03-06T07:23:21.927614+0000","last_clean":"2024-03-06T07:23:21.927614+0000","last_became_active":"2024-03-06T07:23:20.833264+0000","last_became_peered":"2024-03-06T07:23:20.833264+0000","last_unstale":"2024-03-06T07:23:21.927614+0000","last_undegraded":"2024-03-06T07:23:21.927614+0000","last_fullsized":"2024-03-06T07:23:21.927614+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:51.684228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927668+0000","last_change":"2024-03-06T07:23:20.832761+0000","last_active":"2024-03-06T07:23:21.927668+0000","last_peered":"2024-03-06T07:23:21.927668+0000","last_clean":"2024-03-06T07:23:21.927668+0000","last_became_active":"2024-03-06T07:23:20.832484+0000","last_became_peered":"2024-03-06T07:23:20.832484+0000","last_unstale":"2024-03-06T07:23:21.927668+0000","last_undegraded":"2024-03-06T07:23:21.927668+0000","last_fullsized":"2024-03-06T07:23:21.927668+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:45.550052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831564+0000","last_change":"2024-03-06T07:23:20.833437+0000","last_active":"2024-03-06T07:23:21.831564+0000","last_peered":"2024-03-06T07:23:21.831564+0000","last_clean":"2024-03-06T07:23:21.831564+0000","last_became_active":"2024-03-06T07:23:20.833161+0000","last_became_peered":"2024-03-06T07:23:20.833161+0000","last_unstale":"2024-03-06T07:23:21.831564+0000","last_undegraded":"2024-03-06T07:23:21.831564+0000","last_fullsized":"2024-03-06T07:23:21.831564+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:58.636532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815124+0000","last_change":"2024-03-06T07:23:20.823571+0000","last_active":"2024-03-06T07:23:21.815124+0000","last_peered":"2024-03-06T07:23:21.815124+0000","last_clean":"2024-03-06T07:23:21.815124+0000","last_became_active":"2024-03-06T07:23:20.822951+0000","last_became_peered":"2024-03-06T07:23:20.822951+0000","last_unstale":"2024-03-06T07:23:21.815124+0000","last_undegraded":"2024-03-06T07:23:21.815124+0000","last_fullsized":"2024-03-06T07:23:21.815124+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:22.074129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927958+0000","last_change":"2024-03-06T07:23:20.831489+0000","last_active":"2024-03-06T07:23:21.927958+0000","last_peered":"2024-03-06T07:23:21.927958+0000","last_clean":"2024-03-06T07:23:21.927958+0000","last_became_active":"2024-03-06T07:23:20.831199+0000","last_became_peered":"2024-03-06T07:23:20.831199+0000","last_unstale":"2024-03-06T07:23:21.927958+0000","last_undegraded":"2024-03-06T07:23:21.927958+0000","last_fullsized":"2024-03-06T07:23:21.927958+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:32.771906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928065+0000","last_change":"2024-03-06T07:23:20.823116+0000","last_active":"2024-03-06T07:23:21.928065+0000","last_peered":"2024-03-06T07:23:21.928065+0000","last_clean":"2024-03-06T07:23:21.928065+0000","last_became_active":"2024-03-06T07:23:20.822781+0000","last_became_peered":"2024-03-06T07:23:20.822781+0000","last_unstale":"2024-03-06T07:23:21.928065+0000","last_undegraded":"2024-03-06T07:23:21.928065+0000","last_fullsized":"2024-03-06T07:23:21.928065+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:24.917625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927629+0000","last_change":"2024-03-06T07:23:20.830890+0000","last_active":"2024-03-06T07:23:21.927629+0000","last_peered":"2024-03-06T07:23:21.927629+0000","last_clean":"2024-03-06T07:23:21.927629+0000","last_became_active":"2024-03-06T07:23:20.830419+0000","last_became_peered":"2024-03-06T07:23:20.830419+0000","last_unstale":"2024-03-06T07:23:21.927629+0000","last_undegraded":"2024-03-06T07:23:21.927629+0000","last_fullsized":"2024-03-06T07:23:21.927629+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:49.694980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815405+0000","last_change":"2024-03-06T07:23:20.835609+0000","last_active":"2024-03-06T07:23:21.815405+0000","last_peered":"2024-03-06T07:23:21.815405+0000","last_clean":"2024-03-06T07:23:21.815405+0000","last_became_active":"2024-03-06T07:23:20.835034+0000","last_became_peered":"2024-03-06T07:23:20.835034+0000","last_unstale":"2024-03-06T07:23:21.815405+0000","last_undegraded":"2024-03-06T07:23:21.815405+0000","last_fullsized":"2024-03-06T07:23:21.815405+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:58.422844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927353+0000","last_change":"2024-03-06T07:23:20.821260+0000","last_active":"2024-03-06T07:23:21.927353+0000","last_peered":"2024-03-06T07:23:21.927353+0000","last_clean":"2024-03-06T07:23:21.927353+0000","last_became_active":"2024-03-06T07:23:20.820990+0000","last_became_peered":"2024-03-06T07:23:20.820990+0000","last_unstale":"2024-03-06T07:23:21.927353+0000","last_undegraded":"2024-03-06T07:23:21.927353+0000","last_fullsized":"2024-03-06T07:23:21.927353+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:19.586412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815635+0000","last_change":"2024-03-06T07:23:20.837633+0000","last_active":"2024-03-06T07:23:21.815635+0000","last_peered":"2024-03-06T07:23:21.815635+0000","last_clean":"2024-03-06T07:23:21.815635+0000","last_became_active":"2024-03-06T07:23:20.837271+0000","last_became_peered":"2024-03-06T07:23:20.837271+0000","last_unstale":"2024-03-06T07:23:21.815635+0000","last_undegraded":"2024-03-06T07:23:21.815635+0000","last_fullsized":"2024-03-06T07:23:21.815635+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:11.035847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814220+0000","last_change":"2024-03-06T07:23:20.832684+0000","last_active":"2024-03-06T07:23:21.814220+0000","last_peered":"2024-03-06T07:23:21.814220+0000","last_clean":"2024-03-06T07:23:21.814220+0000","last_became_active":"2024-03-06T07:23:20.832100+0000","last_became_peered":"2024-03-06T07:23:20.832100+0000","last_unstale":"2024-03-06T07:23:21.814220+0000","last_undegraded":"2024-03-06T07:23:21.814220+0000","last_fullsized":"2024-03-06T07:23:21.814220+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.177104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928028+0000","last_change":"2024-03-06T07:23:20.826760+0000","last_active":"2024-03-06T07:23:21.928028+0000","last_peered":"2024-03-06T07:23:21.928028+0000","last_clean":"2024-03-06T07:23:21.928028+0000","last_became_active":"2024-03-06T07:23:20.823900+0000","last_became_peered":"2024-03-06T07:23:20.823900+0000","last_unstale":"2024-03-06T07:23:21.928028+0000","last_undegraded":"2024-03-06T07:23:21.928028+0000","last_fullsized":"2024-03-06T07:23:21.928028+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:00.676714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816475+0000","last_change":"2024-03-06T07:23:20.829449+0000","last_active":"2024-03-06T07:23:21.816475+0000","last_peered":"2024-03-06T07:23:21.816475+0000","last_clean":"2024-03-06T07:23:21.816475+0000","last_became_active":"2024-03-06T07:23:20.829171+0000","last_became_peered":"2024-03-06T07:23:20.829171+0000","last_unstale":"2024-03-06T07:23:21.816475+0000","last_undegraded":"2024-03-06T07:23:21.816475+0000","last_fullsized":"2024-03-06T07:23:21.816475+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:18.712434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927233+0000","last_change":"2024-03-06T07:23:20.838681+0000","last_active":"2024-03-06T07:23:21.927233+0000","last_peered":"2024-03-06T07:23:21.927233+0000","last_clean":"2024-03-06T07:23:21.927233+0000","last_became_active":"2024-03-06T07:23:20.838403+0000","last_became_peered":"2024-03-06T07:23:20.838403+0000","last_unstale":"2024-03-06T07:23:21.927233+0000","last_undegraded":"2024-03-06T07:23:21.927233+0000","last_fullsized":"2024-03-06T07:23:21.927233+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:43.634210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836218+0000","last_change":"2024-03-06T07:23:20.830707+0000","last_active":"2024-03-06T07:23:21.836218+0000","last_peered":"2024-03-06T07:23:21.836218+0000","last_clean":"2024-03-06T07:23:21.836218+0000","last_became_active":"2024-03-06T07:23:20.828572+0000","last_became_peered":"2024-03-06T07:23:20.828572+0000","last_unstale":"2024-03-06T07:23:21.836218+0000","last_undegraded":"2024-03-06T07:23:21.836218+0000","last_fullsized":"2024-03-06T07:23:21.836218+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:53.216867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927737+0000","last_change":"2024-03-06T07:23:20.823018+0000","last_active":"2024-03-06T07:23:21.927737+0000","last_peered":"2024-03-06T07:23:21.927737+0000","last_clean":"2024-03-06T07:23:21.927737+0000","last_became_active":"2024-03-06T07:23:20.822746+0000","last_became_peered":"2024-03-06T07:23:20.822746+0000","last_unstale":"2024-03-06T07:23:21.927737+0000","last_undegraded":"2024-03-06T07:23:21.927737+0000","last_fullsized":"2024-03-06T07:23:21.927737+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:50.177104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927487+0000","last_change":"2024-03-06T07:23:20.829029+0000","last_active":"2024-03-06T07:23:21.927487+0000","last_peered":"2024-03-06T07:23:21.927487+0000","last_clean":"2024-03-06T07:23:21.927487+0000","last_became_active":"2024-03-06T07:23:20.828741+0000","last_became_peered":"2024-03-06T07:23:20.828741+0000","last_unstale":"2024-03-06T07:23:21.927487+0000","last_undegraded":"2024-03-06T07:23:21.927487+0000","last_fullsized":"2024-03-06T07:23:21.927487+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:17.302710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831576+0000","last_change":"2024-03-06T07:23:20.834834+0000","last_active":"2024-03-06T07:23:21.831576+0000","last_peered":"2024-03-06T07:23:21.831576+0000","last_clean":"2024-03-06T07:23:21.831576+0000","last_became_active":"2024-03-06T07:23:20.834581+0000","last_became_peered":"2024-03-06T07:23:20.834581+0000","last_unstale":"2024-03-06T07:23:21.831576+0000","last_undegraded":"2024-03-06T07:23:21.831576+0000","last_fullsized":"2024-03-06T07:23:21.831576+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:45.761837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815799+0000","last_change":"2024-03-06T07:23:20.824357+0000","last_active":"2024-03-06T07:23:21.815799+0000","last_peered":"2024-03-06T07:23:21.815799+0000","last_clean":"2024-03-06T07:23:21.815799+0000","last_became_active":"2024-03-06T07:23:20.823953+0000","last_became_peered":"2024-03-06T07:23:20.823953+0000","last_unstale":"2024-03-06T07:23:21.815799+0000","last_undegraded":"2024-03-06T07:23:21.815799+0000","last_fullsized":"2024-03-06T07:23:21.815799+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:01.061700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927023+0000","last_change":"2024-03-06T07:23:20.827550+0000","last_active":"2024-03-06T07:23:21.927023+0000","last_peered":"2024-03-06T07:23:21.927023+0000","last_clean":"2024-03-06T07:23:21.927023+0000","last_became_active":"2024-03-06T07:23:20.827222+0000","last_became_peered":"2024-03-06T07:23:20.827222+0000","last_unstale":"2024-03-06T07:23:21.927023+0000","last_undegraded":"2024-03-06T07:23:21.927023+0000","last_fullsized":"2024-03-06T07:23:21.927023+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:54.586668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831952+0000","last_change":"2024-03-06T07:23:20.832808+0000","last_active":"2024-03-06T07:23:21.831952+0000","last_peered":"2024-03-06T07:23:21.831952+0000","last_clean":"2024-03-06T07:23:21.831952+0000","last_became_active":"2024-03-06T07:23:20.832484+0000","last_became_peered":"2024-03-06T07:23:20.832484+0000","last_unstale":"2024-03-06T07:23:21.831952+0000","last_undegraded":"2024-03-06T07:23:21.831952+0000","last_fullsized":"2024-03-06T07:23:21.831952+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:25.362417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831873+0000","last_change":"2024-03-06T07:23:20.833900+0000","last_active":"2024-03-06T07:23:21.831873+0000","last_peered":"2024-03-06T07:23:21.831873+0000","last_clean":"2024-03-06T07:23:21.831873+0000","last_became_active":"2024-03-06T07:23:20.833375+0000","last_became_peered":"2024-03-06T07:23:20.833375+0000","last_unstale":"2024-03-06T07:23:21.831873+0000","last_undegraded":"2024-03-06T07:23:21.831873+0000","last_fullsized":"2024-03-06T07:23:21.831873+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:54.587065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927006+0000","last_change":"2024-03-06T07:23:20.829523+0000","last_active":"2024-03-06T07:23:21.927006+0000","last_peered":"2024-03-06T07:23:21.927006+0000","last_clean":"2024-03-06T07:23:21.927006+0000","last_became_active":"2024-03-06T07:23:20.829233+0000","last_became_peered":"2024-03-06T07:23:20.829233+0000","last_unstale":"2024-03-06T07:23:21.927006+0000","last_undegraded":"2024-03-06T07:23:21.927006+0000","last_fullsized":"2024-03-06T07:23:21.927006+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:12.845913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.926942+0000","last_change":"2024-03-06T07:23:20.824645+0000","last_active":"2024-03-06T07:23:21.926942+0000","last_peered":"2024-03-06T07:23:21.926942+0000","last_clean":"2024-03-06T07:23:21.926942+0000","last_became_active":"2024-03-06T07:23:20.824385+0000","last_became_peered":"2024-03-06T07:23:20.824385+0000","last_unstale":"2024-03-06T07:23:21.926942+0000","last_undegraded":"2024-03-06T07:23:21.926942+0000","last_fullsized":"2024-03-06T07:23:21.926942+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:41.937379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816592+0000","last_change":"2024-03-06T07:23:20.823326+0000","last_active":"2024-03-06T07:23:21.816592+0000","last_peered":"2024-03-06T07:23:21.816592+0000","last_clean":"2024-03-06T07:23:21.816592+0000","last_became_active":"2024-03-06T07:23:20.822842+0000","last_became_peered":"2024-03-06T07:23:20.822842+0000","last_unstale":"2024-03-06T07:23:21.816592+0000","last_undegraded":"2024-03-06T07:23:21.816592+0000","last_fullsized":"2024-03-06T07:23:21.816592+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:58.422844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927336+0000","last_change":"2024-03-06T07:23:20.828196+0000","last_active":"2024-03-06T07:23:21.927336+0000","last_peered":"2024-03-06T07:23:21.927336+0000","last_clean":"2024-03-06T07:23:21.927336+0000","last_became_active":"2024-03-06T07:23:20.827920+0000","last_became_peered":"2024-03-06T07:23:20.827920+0000","last_unstale":"2024-03-06T07:23:21.927336+0000","last_undegraded":"2024-03-06T07:23:21.927336+0000","last_fullsized":"2024-03-06T07:23:21.927336+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:47:34.007709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927853+0000","last_change":"2024-03-06T07:23:20.823387+0000","last_active":"2024-03-06T07:23:21.927853+0000","last_peered":"2024-03-06T07:23:21.927853+0000","last_clean":"2024-03-06T07:23:21.927853+0000","last_became_active":"2024-03-06T07:23:20.823117+0000","last_became_peered":"2024-03-06T07:23:20.823117+0000","last_unstale":"2024-03-06T07:23:21.927853+0000","last_undegraded":"2024-03-06T07:23:21.927853+0000","last_fullsized":"2024-03-06T07:23:21.927853+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:26.473077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815624+0000","last_change":"2024-03-06T07:23:20.823473+0000","last_active":"2024-03-06T07:23:21.815624+0000","last_peered":"2024-03-06T07:23:21.815624+0000","last_clean":"2024-03-06T07:23:21.815624+0000","last_became_active":"2024-03-06T07:23:20.822556+0000","last_became_peered":"2024-03-06T07:23:20.822556+0000","last_unstale":"2024-03-06T07:23:21.815624+0000","last_undegraded":"2024-03-06T07:23:21.815624+0000","last_fullsized":"2024-03-06T07:23:21.815624+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:44.213332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814153+0000","last_change":"2024-03-06T07:23:20.829495+0000","last_active":"2024-03-06T07:23:21.814153+0000","last_peered":"2024-03-06T07:23:21.814153+0000","last_clean":"2024-03-06T07:23:21.814153+0000","last_became_active":"2024-03-06T07:23:20.829216+0000","last_became_peered":"2024-03-06T07:23:20.829216+0000","last_unstale":"2024-03-06T07:23:21.814153+0000","last_undegraded":"2024-03-06T07:23:21.814153+0000","last_fullsized":"2024-03-06T07:23:21.814153+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:51.539447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"134'4","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928064+0000","last_change":"2024-03-06T07:23:19.819221+0000","last_active":"2024-03-06T07:23:21.928064+0000","last_peered":"2024-03-06T07:23:21.928064+0000","last_clean":"2024-03-06T07:23:21.928064+0000","last_became_active":"2024-03-06T07:23:19.818971+0000","last_became_peered":"2024-03-06T07:23:19.818971+0000","last_unstale":"2024-03-06T07:23:21.928064+0000","last_undegraded":"2024-03-06T07:23:21.928064+0000","last_fullsized":"2024-03-06T07:23:21.928064+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:20.118726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814279+0000","last_change":"2024-03-06T07:23:20.831095+0000","last_active":"2024-03-06T07:23:21.814279+0000","last_peered":"2024-03-06T07:23:21.814279+0000","last_clean":"2024-03-06T07:23:21.814279+0000","last_became_active":"2024-03-06T07:23:20.830835+0000","last_became_peered":"2024-03-06T07:23:20.830835+0000","last_unstale":"2024-03-06T07:23:21.814279+0000","last_undegraded":"2024-03-06T07:23:21.814279+0000","last_fullsized":"2024-03-06T07:23:21.814279+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:29.260882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835995+0000","last_change":"2024-03-06T07:23:19.822328+0000","last_active":"2024-03-06T07:23:21.835995+0000","last_peered":"2024-03-06T07:23:21.835995+0000","last_clean":"2024-03-06T07:23:21.835995+0000","last_became_active":"2024-03-06T07:23:19.821927+0000","last_became_peered":"2024-03-06T07:23:19.821927+0000","last_unstale":"2024-03-06T07:23:21.835995+0000","last_undegraded":"2024-03-06T07:23:21.835995+0000","last_fullsized":"2024-03-06T07:23:21.835995+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:53.582233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816739+0000","last_change":"2024-03-06T07:23:20.826012+0000","last_active":"2024-03-06T07:23:21.816739+0000","last_peered":"2024-03-06T07:23:21.816739+0000","last_clean":"2024-03-06T07:23:21.816739+0000","last_became_active":"2024-03-06T07:23:20.825726+0000","last_became_peered":"2024-03-06T07:23:20.825726+0000","last_unstale":"2024-03-06T07:23:21.816739+0000","last_undegraded":"2024-03-06T07:23:21.816739+0000","last_fullsized":"2024-03-06T07:23:21.816739+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:51.252428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836259+0000","last_change":"2024-03-06T07:23:19.822963+0000","last_active":"2024-03-06T07:23:21.836259+0000","last_peered":"2024-03-06T07:23:21.836259+0000","last_clean":"2024-03-06T07:23:21.836259+0000","last_became_active":"2024-03-06T07:23:19.822663+0000","last_became_peered":"2024-03-06T07:23:19.822663+0000","last_unstale":"2024-03-06T07:23:21.836259+0000","last_undegraded":"2024-03-06T07:23:21.836259+0000","last_fullsized":"2024-03-06T07:23:21.836259+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:49.157011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836191+0000","last_change":"2024-03-06T07:23:20.835612+0000","last_active":"2024-03-06T07:23:21.836191+0000","last_peered":"2024-03-06T07:23:21.836191+0000","last_clean":"2024-03-06T07:23:21.836191+0000","last_became_active":"2024-03-06T07:23:20.835350+0000","last_became_peered":"2024-03-06T07:23:20.835350+0000","last_unstale":"2024-03-06T07:23:21.836191+0000","last_undegraded":"2024-03-06T07:23:21.836191+0000","last_fullsized":"2024-03-06T07:23:21.836191+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:02.387042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814219+0000","last_change":"2024-03-06T07:23:19.814213+0000","last_active":"2024-03-06T07:23:21.814219+0000","last_peered":"2024-03-06T07:23:21.814219+0000","last_clean":"2024-03-06T07:23:21.814219+0000","last_became_active":"2024-03-06T07:23:19.813940+0000","last_became_peered":"2024-03-06T07:23:19.813940+0000","last_unstale":"2024-03-06T07:23:21.814219+0000","last_undegraded":"2024-03-06T07:23:21.814219+0000","last_fullsized":"2024-03-06T07:23:21.814219+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:16.681332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927210+0000","last_change":"2024-03-06T07:23:20.822303+0000","last_active":"2024-03-06T07:23:21.927210+0000","last_peered":"2024-03-06T07:23:21.927210+0000","last_clean":"2024-03-06T07:23:21.927210+0000","last_became_active":"2024-03-06T07:23:20.821635+0000","last_became_peered":"2024-03-06T07:23:20.821635+0000","last_unstale":"2024-03-06T07:23:21.927210+0000","last_undegraded":"2024-03-06T07:23:21.927210+0000","last_fullsized":"2024-03-06T07:23:21.927210+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:51.539447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816639+0000","last_change":"2024-03-06T07:23:19.814612+0000","last_active":"2024-03-06T07:23:21.816639+0000","last_peered":"2024-03-06T07:23:21.816639+0000","last_clean":"2024-03-06T07:23:21.816639+0000","last_became_active":"2024-03-06T07:23:19.814108+0000","last_became_peered":"2024-03-06T07:23:19.814108+0000","last_unstale":"2024-03-06T07:23:21.816639+0000","last_undegraded":"2024-03-06T07:23:21.816639+0000","last_fullsized":"2024-03-06T07:23:21.816639+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:29:14.655226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815888+0000","last_change":"2024-03-06T07:23:20.824411+0000","last_active":"2024-03-06T07:23:21.815888+0000","last_peered":"2024-03-06T07:23:21.815888+0000","last_clean":"2024-03-06T07:23:21.815888+0000","last_became_active":"2024-03-06T07:23:20.824153+0000","last_became_peered":"2024-03-06T07:23:20.824153+0000","last_unstale":"2024-03-06T07:23:21.815888+0000","last_undegraded":"2024-03-06T07:23:21.815888+0000","last_fullsized":"2024-03-06T07:23:21.815888+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:51.252428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927940+0000","last_change":"2024-03-06T07:23:19.821284+0000","last_active":"2024-03-06T07:23:21.927940+0000","last_peered":"2024-03-06T07:23:21.927940+0000","last_clean":"2024-03-06T07:23:21.927940+0000","last_became_active":"2024-03-06T07:23:19.820988+0000","last_became_peered":"2024-03-06T07:23:19.820988+0000","last_unstale":"2024-03-06T07:23:21.927940+0000","last_undegraded":"2024-03-06T07:23:21.927940+0000","last_fullsized":"2024-03-06T07:23:21.927940+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:49.172272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.813939+0000","last_change":"2024-03-06T07:23:20.826258+0000","last_active":"2024-03-06T07:23:21.813939+0000","last_peered":"2024-03-06T07:23:21.813939+0000","last_clean":"2024-03-06T07:23:21.813939+0000","last_became_active":"2024-03-06T07:23:20.825979+0000","last_became_peered":"2024-03-06T07:23:20.825979+0000","last_unstale":"2024-03-06T07:23:21.813939+0000","last_undegraded":"2024-03-06T07:23:21.813939+0000","last_fullsized":"2024-03-06T07:23:21.813939+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:25.362417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831518+0000","last_change":"2024-03-06T07:23:19.820573+0000","last_active":"2024-03-06T07:23:21.831518+0000","last_peered":"2024-03-06T07:23:21.831518+0000","last_clean":"2024-03-06T07:23:21.831518+0000","last_became_active":"2024-03-06T07:23:19.820283+0000","last_became_peered":"2024-03-06T07:23:19.820283+0000","last_unstale":"2024-03-06T07:23:21.831518+0000","last_undegraded":"2024-03-06T07:23:21.831518+0000","last_fullsized":"2024-03-06T07:23:21.831518+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:43.208499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831689+0000","last_change":"2024-03-06T07:23:20.828026+0000","last_active":"2024-03-06T07:23:21.831689+0000","last_peered":"2024-03-06T07:23:21.831689+0000","last_clean":"2024-03-06T07:23:21.831689+0000","last_became_active":"2024-03-06T07:23:20.827770+0000","last_became_peered":"2024-03-06T07:23:20.827770+0000","last_unstale":"2024-03-06T07:23:21.831689+0000","last_undegraded":"2024-03-06T07:23:21.831689+0000","last_fullsized":"2024-03-06T07:23:21.831689+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:57.608000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927970+0000","last_change":"2024-03-06T07:23:19.822141+0000","last_active":"2024-03-06T07:23:21.927970+0000","last_peered":"2024-03-06T07:23:21.927970+0000","last_clean":"2024-03-06T07:23:21.927970+0000","last_became_active":"2024-03-06T07:23:19.821888+0000","last_became_peered":"2024-03-06T07:23:19.821888+0000","last_unstale":"2024-03-06T07:23:21.927970+0000","last_undegraded":"2024-03-06T07:23:21.927970+0000","last_fullsized":"2024-03-06T07:23:21.927970+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:13.643734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816600+0000","last_change":"2024-03-06T07:23:20.830015+0000","last_active":"2024-03-06T07:23:21.816600+0000","last_peered":"2024-03-06T07:23:21.816600+0000","last_clean":"2024-03-06T07:23:21.816600+0000","last_became_active":"2024-03-06T07:23:20.829751+0000","last_became_peered":"2024-03-06T07:23:20.829751+0000","last_unstale":"2024-03-06T07:23:21.816600+0000","last_undegraded":"2024-03-06T07:23:21.816600+0000","last_fullsized":"2024-03-06T07:23:21.816600+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:11.035847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836199+0000","last_change":"2024-03-06T07:23:19.818446+0000","last_active":"2024-03-06T07:23:21.836199+0000","last_peered":"2024-03-06T07:23:21.836199+0000","last_clean":"2024-03-06T07:23:21.836199+0000","last_became_active":"2024-03-06T07:23:19.818187+0000","last_became_peered":"2024-03-06T07:23:19.818187+0000","last_unstale":"2024-03-06T07:23:21.836199+0000","last_undegraded":"2024-03-06T07:23:21.836199+0000","last_fullsized":"2024-03-06T07:23:21.836199+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:24.357585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927758+0000","last_change":"2024-03-06T07:23:20.830670+0000","last_active":"2024-03-06T07:23:21.927758+0000","last_peered":"2024-03-06T07:23:21.927758+0000","last_clean":"2024-03-06T07:23:21.927758+0000","last_became_active":"2024-03-06T07:23:20.830068+0000","last_became_peered":"2024-03-06T07:23:20.830068+0000","last_unstale":"2024-03-06T07:23:21.927758+0000","last_undegraded":"2024-03-06T07:23:21.927758+0000","last_fullsized":"2024-03-06T07:23:21.927758+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:39.061517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.832024+0000","last_change":"2024-03-06T07:23:19.821359+0000","last_active":"2024-03-06T07:23:21.832024+0000","last_peered":"2024-03-06T07:23:21.832024+0000","last_clean":"2024-03-06T07:23:21.832024+0000","last_became_active":"2024-03-06T07:23:19.821086+0000","last_became_peered":"2024-03-06T07:23:19.821086+0000","last_unstale":"2024-03-06T07:23:21.832024+0000","last_undegraded":"2024-03-06T07:23:21.832024+0000","last_fullsized":"2024-03-06T07:23:21.832024+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:10.031014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927371+0000","last_change":"2024-03-06T07:23:20.837942+0000","last_active":"2024-03-06T07:23:21.927371+0000","last_peered":"2024-03-06T07:23:21.927371+0000","last_clean":"2024-03-06T07:23:21.927371+0000","last_became_active":"2024-03-06T07:23:20.837637+0000","last_became_peered":"2024-03-06T07:23:20.837637+0000","last_unstale":"2024-03-06T07:23:21.927371+0000","last_undegraded":"2024-03-06T07:23:21.927371+0000","last_fullsized":"2024-03-06T07:23:21.927371+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:29.772363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"134'3","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927849+0000","last_change":"2024-03-06T07:23:19.812634+0000","last_active":"2024-03-06T07:23:21.927849+0000","last_peered":"2024-03-06T07:23:21.927849+0000","last_clean":"2024-03-06T07:23:21.927849+0000","last_became_active":"2024-03-06T07:23:19.812378+0000","last_became_peered":"2024-03-06T07:23:19.812378+0000","last_unstale":"2024-03-06T07:23:21.927849+0000","last_undegraded":"2024-03-06T07:23:21.927849+0000","last_fullsized":"2024-03-06T07:23:21.927849+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:31:18.417628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815268+0000","last_change":"2024-03-06T07:23:20.829353+0000","last_active":"2024-03-06T07:23:21.815268+0000","last_peered":"2024-03-06T07:23:21.815268+0000","last_clean":"2024-03-06T07:23:21.815268+0000","last_became_active":"2024-03-06T07:23:20.829066+0000","last_became_peered":"2024-03-06T07:23:20.829066+0000","last_unstale":"2024-03-06T07:23:21.815268+0000","last_undegraded":"2024-03-06T07:23:21.815268+0000","last_fullsized":"2024-03-06T07:23:21.815268+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:26.273208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927710+0000","last_change":"2024-03-06T07:23:19.817081+0000","last_active":"2024-03-06T07:23:21.927710+0000","last_peered":"2024-03-06T07:23:21.927710+0000","last_clean":"2024-03-06T07:23:21.927710+0000","last_became_active":"2024-03-06T07:23:19.816830+0000","last_became_peered":"2024-03-06T07:23:19.816830+0000","last_unstale":"2024-03-06T07:23:21.927710+0000","last_undegraded":"2024-03-06T07:23:21.927710+0000","last_fullsized":"2024-03-06T07:23:21.927710+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:28.256050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.813980+0000","last_change":"2024-03-06T07:23:20.832427+0000","last_active":"2024-03-06T07:23:21.813980+0000","last_peered":"2024-03-06T07:23:21.813980+0000","last_clean":"2024-03-06T07:23:21.813980+0000","last_became_active":"2024-03-06T07:23:20.830865+0000","last_became_peered":"2024-03-06T07:23:20.830865+0000","last_unstale":"2024-03-06T07:23:21.813980+0000","last_undegraded":"2024-03-06T07:23:21.813980+0000","last_fullsized":"2024-03-06T07:23:21.813980+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:50.161843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815481+0000","last_change":"2024-03-06T07:23:19.815723+0000","last_active":"2024-03-06T07:23:21.815481+0000","last_peered":"2024-03-06T07:23:21.815481+0000","last_clean":"2024-03-06T07:23:21.815481+0000","last_became_active":"2024-03-06T07:23:19.815374+0000","last_became_peered":"2024-03-06T07:23:19.815374+0000","last_unstale":"2024-03-06T07:23:21.815481+0000","last_undegraded":"2024-03-06T07:23:21.815481+0000","last_fullsized":"2024-03-06T07:23:21.815481+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:08.771871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927807+0000","last_change":"2024-03-06T07:23:20.821930+0000","last_active":"2024-03-06T07:23:21.927807+0000","last_peered":"2024-03-06T07:23:21.927807+0000","last_clean":"2024-03-06T07:23:21.927807+0000","last_became_active":"2024-03-06T07:23:20.820943+0000","last_became_peered":"2024-03-06T07:23:20.820943+0000","last_unstale":"2024-03-06T07:23:21.927807+0000","last_undegraded":"2024-03-06T07:23:21.927807+0000","last_fullsized":"2024-03-06T07:23:21.927807+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:55.112513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815987+0000","last_change":"2024-03-06T07:23:19.817408+0000","last_active":"2024-03-06T07:23:21.815987+0000","last_peered":"2024-03-06T07:23:21.815987+0000","last_clean":"2024-03-06T07:23:21.815987+0000","last_became_active":"2024-03-06T07:23:19.817072+0000","last_became_peered":"2024-03-06T07:23:19.817072+0000","last_unstale":"2024-03-06T07:23:21.815987+0000","last_undegraded":"2024-03-06T07:23:21.815987+0000","last_fullsized":"2024-03-06T07:23:21.815987+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:24.931887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.813988+0000","last_change":"2024-03-06T07:23:20.824640+0000","last_active":"2024-03-06T07:23:21.813988+0000","last_peered":"2024-03-06T07:23:21.813988+0000","last_clean":"2024-03-06T07:23:21.813988+0000","last_became_active":"2024-03-06T07:23:20.824357+0000","last_became_peered":"2024-03-06T07:23:20.824357+0000","last_unstale":"2024-03-06T07:23:21.813988+0000","last_undegraded":"2024-03-06T07:23:21.813988+0000","last_fullsized":"2024-03-06T07:23:21.813988+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:11.943936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"134'8","reported_seq":22,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831458+0000","last_change":"2024-03-06T07:23:19.822083+0000","last_active":"2024-03-06T07:23:21.831458+0000","last_peered":"2024-03-06T07:23:21.831458+0000","last_clean":"2024-03-06T07:23:21.831458+0000","last_became_active":"2024-03-06T07:23:19.821768+0000","last_became_peered":"2024-03-06T07:23:19.821768+0000","last_unstale":"2024-03-06T07:23:21.831458+0000","last_undegraded":"2024-03-06T07:23:21.831458+0000","last_fullsized":"2024-03-06T07:23:21.831458+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:57.418012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835959+0000","last_change":"2024-03-06T07:23:20.828815+0000","last_active":"2024-03-06T07:23:21.835959+0000","last_peered":"2024-03-06T07:23:21.835959+0000","last_clean":"2024-03-06T07:23:21.835959+0000","last_became_active":"2024-03-06T07:23:20.828422+0000","last_became_peered":"2024-03-06T07:23:20.828422+0000","last_unstale":"2024-03-06T07:23:21.835959+0000","last_undegraded":"2024-03-06T07:23:21.835959+0000","last_fullsized":"2024-03-06T07:23:21.835959+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:14.592219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927475+0000","last_change":"2024-03-06T07:23:19.821216+0000","last_active":"2024-03-06T07:23:21.927475+0000","last_peered":"2024-03-06T07:23:21.927475+0000","last_clean":"2024-03-06T07:23:21.927475+0000","last_became_active":"2024-03-06T07:23:19.820903+0000","last_became_peered":"2024-03-06T07:23:19.820903+0000","last_unstale":"2024-03-06T07:23:21.927475+0000","last_undegraded":"2024-03-06T07:23:21.927475+0000","last_fullsized":"2024-03-06T07:23:21.927475+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:02.459037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836190+0000","last_change":"2024-03-06T07:23:20.830697+0000","last_active":"2024-03-06T07:23:21.836190+0000","last_peered":"2024-03-06T07:23:21.836190+0000","last_clean":"2024-03-06T07:23:21.836190+0000","last_became_active":"2024-03-06T07:23:20.829015+0000","last_became_peered":"2024-03-06T07:23:20.829015+0000","last_unstale":"2024-03-06T07:23:21.836190+0000","last_undegraded":"2024-03-06T07:23:21.836190+0000","last_fullsized":"2024-03-06T07:23:21.836190+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:30.818350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.836033+0000","last_change":"2024-03-06T07:23:19.820024+0000","last_active":"2024-03-06T07:23:21.836033+0000","last_peered":"2024-03-06T07:23:21.836033+0000","last_clean":"2024-03-06T07:23:21.836033+0000","last_became_active":"2024-03-06T07:23:19.819703+0000","last_became_peered":"2024-03-06T07:23:19.819703+0000","last_unstale":"2024-03-06T07:23:21.836033+0000","last_undegraded":"2024-03-06T07:23:21.836033+0000","last_fullsized":"2024-03-06T07:23:21.836033+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:06.015217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831952+0000","last_change":"2024-03-06T07:23:20.839134+0000","last_active":"2024-03-06T07:23:21.831952+0000","last_peered":"2024-03-06T07:23:21.831952+0000","last_clean":"2024-03-06T07:23:21.831952+0000","last_became_active":"2024-03-06T07:23:20.838836+0000","last_became_peered":"2024-03-06T07:23:20.838836+0000","last_unstale":"2024-03-06T07:23:21.831952+0000","last_undegraded":"2024-03-06T07:23:21.831952+0000","last_fullsized":"2024-03-06T07:23:21.831952+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:50.161843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814367+0000","last_change":"2024-03-06T07:23:19.815390+0000","last_active":"2024-03-06T07:23:21.814367+0000","last_peered":"2024-03-06T07:23:21.814367+0000","last_clean":"2024-03-06T07:23:21.814367+0000","last_became_active":"2024-03-06T07:23:19.815132+0000","last_became_peered":"2024-03-06T07:23:19.815132+0000","last_unstale":"2024-03-06T07:23:21.814367+0000","last_undegraded":"2024-03-06T07:23:21.814367+0000","last_fullsized":"2024-03-06T07:23:21.814367+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:25.300694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.832033+0000","last_change":"2024-03-06T07:23:20.832539+0000","last_active":"2024-03-06T07:23:21.832033+0000","last_peered":"2024-03-06T07:23:21.832033+0000","last_clean":"2024-03-06T07:23:21.832033+0000","last_became_active":"2024-03-06T07:23:20.832135+0000","last_became_peered":"2024-03-06T07:23:20.832135+0000","last_unstale":"2024-03-06T07:23:21.832033+0000","last_undegraded":"2024-03-06T07:23:21.832033+0000","last_fullsized":"2024-03-06T07:23:21.832033+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:11.943936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815783+0000","last_change":"2024-03-06T07:23:19.815651+0000","last_active":"2024-03-06T07:23:21.815783+0000","last_peered":"2024-03-06T07:23:21.815783+0000","last_clean":"2024-03-06T07:23:21.815783+0000","last_became_active":"2024-03-06T07:23:19.815351+0000","last_became_peered":"2024-03-06T07:23:19.815351+0000","last_unstale":"2024-03-06T07:23:21.815783+0000","last_undegraded":"2024-03-06T07:23:21.815783+0000","last_fullsized":"2024-03-06T07:23:21.815783+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:23.717250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835951+0000","last_change":"2024-03-06T07:23:20.823541+0000","last_active":"2024-03-06T07:23:21.835951+0000","last_peered":"2024-03-06T07:23:21.835951+0000","last_clean":"2024-03-06T07:23:21.835951+0000","last_became_active":"2024-03-06T07:23:20.823235+0000","last_became_peered":"2024-03-06T07:23:20.823235+0000","last_unstale":"2024-03-06T07:23:21.835951+0000","last_undegraded":"2024-03-06T07:23:21.835951+0000","last_fullsized":"2024-03-06T07:23:21.835951+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:21.123558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927506+0000","last_change":"2024-03-06T07:23:19.815230+0000","last_active":"2024-03-06T07:23:21.927506+0000","last_peered":"2024-03-06T07:23:21.927506+0000","last_clean":"2024-03-06T07:23:21.927506+0000","last_became_active":"2024-03-06T07:23:19.814945+0000","last_became_peered":"2024-03-06T07:23:19.814945+0000","last_unstale":"2024-03-06T07:23:21.927506+0000","last_undegraded":"2024-03-06T07:23:21.927506+0000","last_fullsized":"2024-03-06T07:23:21.927506+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:10.939104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814084+0000","last_change":"2024-03-06T07:23:20.827053+0000","last_active":"2024-03-06T07:23:21.814084+0000","last_peered":"2024-03-06T07:23:21.814084+0000","last_clean":"2024-03-06T07:23:21.814084+0000","last_became_active":"2024-03-06T07:23:20.826752+0000","last_became_peered":"2024-03-06T07:23:20.826752+0000","last_unstale":"2024-03-06T07:23:21.814084+0000","last_undegraded":"2024-03-06T07:23:21.814084+0000","last_fullsized":"2024-03-06T07:23:21.814084+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:15.340945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815978+0000","last_change":"2024-03-06T07:23:19.812535+0000","last_active":"2024-03-06T07:23:21.815978+0000","last_peered":"2024-03-06T07:23:21.815978+0000","last_clean":"2024-03-06T07:23:21.815978+0000","last_became_active":"2024-03-06T07:23:19.812215+0000","last_became_peered":"2024-03-06T07:23:19.812215+0000","last_unstale":"2024-03-06T07:23:21.815978+0000","last_undegraded":"2024-03-06T07:23:21.815978+0000","last_fullsized":"2024-03-06T07:23:21.815978+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:34.214074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835933+0000","last_change":"2024-03-06T07:23:20.827750+0000","last_active":"2024-03-06T07:23:21.835933+0000","last_peered":"2024-03-06T07:23:21.835933+0000","last_clean":"2024-03-06T07:23:21.835933+0000","last_became_active":"2024-03-06T07:23:20.827438+0000","last_became_peered":"2024-03-06T07:23:20.827438+0000","last_unstale":"2024-03-06T07:23:21.835933+0000","last_undegraded":"2024-03-06T07:23:21.835933+0000","last_fullsized":"2024-03-06T07:23:21.835933+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:29.260882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928200+0000","last_change":"2024-03-06T07:23:19.814485+0000","last_active":"2024-03-06T07:23:21.928200+0000","last_peered":"2024-03-06T07:23:21.928200+0000","last_clean":"2024-03-06T07:23:21.928200+0000","last_became_active":"2024-03-06T07:23:19.814240+0000","last_became_peered":"2024-03-06T07:23:19.814240+0000","last_unstale":"2024-03-06T07:23:21.928200+0000","last_undegraded":"2024-03-06T07:23:21.928200+0000","last_fullsized":"2024-03-06T07:23:21.928200+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:55.295513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814146+0000","last_change":"2024-03-06T07:23:20.831880+0000","last_active":"2024-03-06T07:23:21.814146+0000","last_peered":"2024-03-06T07:23:21.814146+0000","last_clean":"2024-03-06T07:23:21.814146+0000","last_became_active":"2024-03-06T07:23:20.830420+0000","last_became_peered":"2024-03-06T07:23:20.830420+0000","last_unstale":"2024-03-06T07:23:21.814146+0000","last_undegraded":"2024-03-06T07:23:21.814146+0000","last_fullsized":"2024-03-06T07:23:21.814146+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:14.648567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815536+0000","last_change":"2024-03-06T07:23:19.813521+0000","last_active":"2024-03-06T07:23:21.815536+0000","last_peered":"2024-03-06T07:23:21.815536+0000","last_clean":"2024-03-06T07:23:21.815536+0000","last_became_active":"2024-03-06T07:23:19.813250+0000","last_became_peered":"2024-03-06T07:23:19.813250+0000","last_unstale":"2024-03-06T07:23:21.815536+0000","last_undegraded":"2024-03-06T07:23:21.815536+0000","last_fullsized":"2024-03-06T07:23:21.815536+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:06.281792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928292+0000","last_change":"2024-03-06T07:23:20.834654+0000","last_active":"2024-03-06T07:23:21.928292+0000","last_peered":"2024-03-06T07:23:21.928292+0000","last_clean":"2024-03-06T07:23:21.928292+0000","last_became_active":"2024-03-06T07:23:20.834179+0000","last_became_peered":"2024-03-06T07:23:20.834179+0000","last_unstale":"2024-03-06T07:23:21.928292+0000","last_undegraded":"2024-03-06T07:23:21.928292+0000","last_fullsized":"2024-03-06T07:23:21.928292+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:56.219442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816449+0000","last_change":"2024-03-06T07:23:19.816231+0000","last_active":"2024-03-06T07:23:21.816449+0000","last_peered":"2024-03-06T07:23:21.816449+0000","last_clean":"2024-03-06T07:23:21.816449+0000","last_became_active":"2024-03-06T07:23:19.815988+0000","last_became_peered":"2024-03-06T07:23:19.815988+0000","last_unstale":"2024-03-06T07:23:21.816449+0000","last_undegraded":"2024-03-06T07:23:21.816449+0000","last_fullsized":"2024-03-06T07:23:21.816449+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:14.032486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816054+0000","last_change":"2024-03-06T07:23:20.833907+0000","last_active":"2024-03-06T07:23:21.816054+0000","last_peered":"2024-03-06T07:23:21.816054+0000","last_clean":"2024-03-06T07:23:21.816054+0000","last_became_active":"2024-03-06T07:23:20.833642+0000","last_became_peered":"2024-03-06T07:23:20.833642+0000","last_unstale":"2024-03-06T07:23:21.816054+0000","last_undegraded":"2024-03-06T07:23:21.816054+0000","last_fullsized":"2024-03-06T07:23:21.816054+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:26.305527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"135'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.833226+0000","last_change":"2024-03-06T07:23:19.821237+0000","last_active":"2024-03-06T07:23:21.833226+0000","last_peered":"2024-03-06T07:23:21.833226+0000","last_clean":"2024-03-06T07:23:21.833226+0000","last_became_active":"2024-03-06T07:23:19.820953+0000","last_became_peered":"2024-03-06T07:23:19.820953+0000","last_unstale":"2024-03-06T07:23:21.833226+0000","last_undegraded":"2024-03-06T07:23:21.833226+0000","last_fullsized":"2024-03-06T07:23:21.833226+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:58:50.247595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.832031+0000","last_change":"2024-03-06T07:23:20.834169+0000","last_active":"2024-03-06T07:23:21.832031+0000","last_peered":"2024-03-06T07:23:21.832031+0000","last_clean":"2024-03-06T07:23:21.832031+0000","last_became_active":"2024-03-06T07:23:20.833884+0000","last_became_peered":"2024-03-06T07:23:20.833884+0000","last_unstale":"2024-03-06T07:23:21.832031+0000","last_undegraded":"2024-03-06T07:23:21.832031+0000","last_fullsized":"2024-03-06T07:23:21.832031+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:15.340945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814364+0000","last_change":"2024-03-06T07:23:19.813583+0000","last_active":"2024-03-06T07:23:21.814364+0000","last_peered":"2024-03-06T07:23:21.814364+0000","last_clean":"2024-03-06T07:23:21.814364+0000","last_became_active":"2024-03-06T07:23:19.813329+0000","last_became_peered":"2024-03-06T07:23:19.813329+0000","last_unstale":"2024-03-06T07:23:21.814364+0000","last_undegraded":"2024-03-06T07:23:21.814364+0000","last_fullsized":"2024-03-06T07:23:21.814364+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:17.707602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928054+0000","last_change":"2024-03-06T07:23:20.835301+0000","last_active":"2024-03-06T07:23:21.928054+0000","last_peered":"2024-03-06T07:23:21.928054+0000","last_clean":"2024-03-06T07:23:21.928054+0000","last_became_active":"2024-03-06T07:23:20.835035+0000","last_became_peered":"2024-03-06T07:23:20.835035+0000","last_unstale":"2024-03-06T07:23:21.928054+0000","last_undegraded":"2024-03-06T07:23:21.928054+0000","last_fullsized":"2024-03-06T07:23:21.928054+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:56.836724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.928321+0000","last_change":"2024-03-06T07:23:19.820735+0000","last_active":"2024-03-06T07:23:21.928321+0000","last_peered":"2024-03-06T07:23:21.928321+0000","last_clean":"2024-03-06T07:23:21.928321+0000","last_became_active":"2024-03-06T07:23:19.820274+0000","last_became_peered":"2024-03-06T07:23:19.820274+0000","last_unstale":"2024-03-06T07:23:21.928321+0000","last_undegraded":"2024-03-06T07:23:21.928321+0000","last_fullsized":"2024-03-06T07:23:21.928321+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:50.534615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815400+0000","last_change":"2024-03-06T07:23:20.825424+0000","last_active":"2024-03-06T07:23:21.815400+0000","last_peered":"2024-03-06T07:23:21.815400+0000","last_clean":"2024-03-06T07:23:21.815400+0000","last_became_active":"2024-03-06T07:23:20.825147+0000","last_became_peered":"2024-03-06T07:23:20.825147+0000","last_unstale":"2024-03-06T07:23:21.815400+0000","last_undegraded":"2024-03-06T07:23:21.815400+0000","last_fullsized":"2024-03-06T07:23:21.815400+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:57.519651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831684+0000","last_change":"2024-03-06T07:23:19.818560+0000","last_active":"2024-03-06T07:23:21.831684+0000","last_peered":"2024-03-06T07:23:21.831684+0000","last_clean":"2024-03-06T07:23:21.831684+0000","last_became_active":"2024-03-06T07:23:19.818303+0000","last_became_peered":"2024-03-06T07:23:19.818303+0000","last_unstale":"2024-03-06T07:23:21.831684+0000","last_undegraded":"2024-03-06T07:23:21.831684+0000","last_fullsized":"2024-03-06T07:23:21.831684+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:20.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.815924+0000","last_change":"2024-03-06T07:23:20.837173+0000","last_active":"2024-03-06T07:23:21.815924+0000","last_peered":"2024-03-06T07:23:21.815924+0000","last_clean":"2024-03-06T07:23:21.815924+0000","last_became_active":"2024-03-06T07:23:20.836902+0000","last_became_peered":"2024-03-06T07:23:20.836902+0000","last_unstale":"2024-03-06T07:23:21.815924+0000","last_undegraded":"2024-03-06T07:23:21.815924+0000","last_fullsized":"2024-03-06T07:23:21.815924+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:17.686165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927428+0000","last_change":"2024-03-06T07:23:19.815802+0000","last_active":"2024-03-06T07:23:21.927428+0000","last_peered":"2024-03-06T07:23:21.927428+0000","last_clean":"2024-03-06T07:23:21.927428+0000","last_became_active":"2024-03-06T07:23:19.815539+0000","last_became_peered":"2024-03-06T07:23:19.815539+0000","last_unstale":"2024-03-06T07:23:21.927428+0000","last_undegraded":"2024-03-06T07:23:21.927428+0000","last_fullsized":"2024-03-06T07:23:21.927428+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:41.017474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.814116+0000","last_change":"2024-03-06T07:23:20.826890+0000","last_active":"2024-03-06T07:23:21.814116+0000","last_peered":"2024-03-06T07:23:21.814116+0000","last_clean":"2024-03-06T07:23:21.814116+0000","last_became_active":"2024-03-06T07:23:20.826612+0000","last_became_peered":"2024-03-06T07:23:20.826612+0000","last_unstale":"2024-03-06T07:23:21.814116+0000","last_undegraded":"2024-03-06T07:23:21.814116+0000","last_fullsized":"2024-03-06T07:23:21.814116+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:56.300346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.831647+0000","last_change":"2024-03-06T07:23:19.818290+0000","last_active":"2024-03-06T07:23:21.831647+0000","last_peered":"2024-03-06T07:23:21.831647+0000","last_clean":"2024-03-06T07:23:21.831647+0000","last_became_active":"2024-03-06T07:23:19.818009+0000","last_became_peered":"2024-03-06T07:23:19.818009+0000","last_unstale":"2024-03-06T07:23:21.831647+0000","last_undegraded":"2024-03-06T07:23:21.831647+0000","last_fullsized":"2024-03-06T07:23:21.831647+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:00.056867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816386+0000","last_change":"2024-03-06T07:23:20.823101+0000","last_active":"2024-03-06T07:23:21.816386+0000","last_peered":"2024-03-06T07:23:21.816386+0000","last_clean":"2024-03-06T07:23:21.816386+0000","last_became_active":"2024-03-06T07:23:20.822481+0000","last_became_peered":"2024-03-06T07:23:20.822481+0000","last_unstale":"2024-03-06T07:23:21.816386+0000","last_undegraded":"2024-03-06T07:23:21.816386+0000","last_fullsized":"2024-03-06T07:23:21.816386+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:21.355665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835868+0000","last_change":"2024-03-06T07:23:19.817553+0000","last_active":"2024-03-06T07:23:21.835868+0000","last_peered":"2024-03-06T07:23:21.835868+0000","last_clean":"2024-03-06T07:23:21.835868+0000","last_became_active":"2024-03-06T07:23:19.817263+0000","last_became_peered":"2024-03-06T07:23:19.817263+0000","last_unstale":"2024-03-06T07:23:21.835868+0000","last_undegraded":"2024-03-06T07:23:21.835868+0000","last_fullsized":"2024-03-06T07:23:21.835868+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:56.603168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.816828+0000","last_change":"2024-03-06T07:23:20.819636+0000","last_active":"2024-03-06T07:23:21.816828+0000","last_peered":"2024-03-06T07:23:21.816828+0000","last_clean":"2024-03-06T07:23:21.816828+0000","last_became_active":"2024-03-06T07:23:20.819333+0000","last_became_peered":"2024-03-06T07:23:20.819333+0000","last_unstale":"2024-03-06T07:23:21.816828+0000","last_undegraded":"2024-03-06T07:23:21.816828+0000","last_fullsized":"2024-03-06T07:23:21.816828+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:19.798196+0000","last_clean_scrub_stamp":"2024-03-06T07:23:19.798196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:17.686165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"134'6","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.927829+0000","last_change":"2024-03-06T07:23:19.813814+0000","last_active":"2024-03-06T07:23:21.927829+0000","last_peered":"2024-03-06T07:23:21.927829+0000","last_clean":"2024-03-06T07:23:21.927829+0000","last_became_active":"2024-03-06T07:23:19.813559+0000","last_became_peered":"2024-03-06T07:23:19.813559+0000","last_unstale":"2024-03-06T07:23:21.927829+0000","last_undegraded":"2024-03-06T07:23:21.927829+0000","last_fullsized":"2024-03-06T07:23:21.927829+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:18.793364+0000","last_clean_scrub_stamp":"2024-03-06T07:23:18.793364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:14.336112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":386,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.835931+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:23:21.835931+0000","last_peered":"2024-03-06T07:23:21.835931+0000","last_clean":"2024-03-06T07:23:21.835931+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:23:21.835931+0000","last_undegraded":"2024-03-06T07:23:21.835931+0000","last_fullsized":"2024-03-06T07:23:21.835931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1765376,"data_stored":1407416,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1699840,"data_stored":817096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1683456,"data_stored":810334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":811946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":817642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":817138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1687552,"data_stored":810404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1785856,"data_stored":1409712,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:24.144 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:24.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.287+0000 7f02bdb13640 1 Processor -- start 2024-03-06T07:23:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.287+0000 7f02bdb13640 1 -- start start 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02bdb13640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f02b80a8290 0x7f02b81539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02bdb13640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f02b80c4740 0x7f02b80701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02bdb13640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80c62a0 con 0x7f02b80c69a0 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02bdb13640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b8153f00 con 0x7f02b80c4740 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02bdb13640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b80706e0 con 0x7f02b80a8290 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02aeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f02b80c4740 0x7f02b80701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02aeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f02b80c4740 0x7f02b80701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49048/0 (socket says 172.21.15.94:49048) 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.288+0000 7f02aeffd640 1 -- 172.21.15.94:0/2523189246 learned_addr learned my addr 172.21.15.94:0/2523189246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.289+0000 7f02b77fe640 1 --2- 172.21.15.94:0/2523189246 >> v2:172.21.15.145:3300/0 conn(0x7f02b80a8290 0x7f02b81539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.289+0000 7f02aeffd640 1 -- 172.21.15.94:0/2523189246 >> v2:172.21.15.145:3300/0 conn(0x7f02b80a8290 msgr2=0x7f02b81539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.289+0000 7f02aeffd640 1 --2- 172.21.15.94:0/2523189246 >> v2:172.21.15.145:3300/0 conn(0x7f02b80a8290 0x7f02b81539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.289+0000 7f02aeffd640 1 -- 172.21.15.94:0/2523189246 >> v1:172.21.15.145:6789/0 conn(0x7f02b80c69a0 legacy=0x7f02b8072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.289+0000 7f02aeffd640 1 -- 172.21.15.94:0/2523189246 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80c4c00 con 0x7f02b80c4740 2024-03-06T07:23:24.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.289+0000 7f02aeffd640 1 --2- 172.21.15.94:0/2523189246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80c4740 0x7f02b80701a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f02b80735b0 tx=0x7f02a002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3157bd1e29243df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02b6ffd640 1 -- 172.21.15.94:0/2523189246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0038070 con 0x7f02b80c4740 2024-03-06T07:23:24.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02b6ffd640 1 -- 172.21.15.94:0/2523189246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a002f050 con 0x7f02b80c4740 2024-03-06T07:23:24.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02b6ffd640 1 -- 172.21.15.94:0/2523189246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0031970 con 0x7f02b80c4740 2024-03-06T07:23:24.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 -- 172.21.15.94:0/2523189246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80c4740 msgr2=0x7f02b80701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 --2- 172.21.15.94:0/2523189246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80c4740 0x7f02b80701a0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f02b80735b0 tx=0x7f02a002b7f0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 -- 172.21.15.94:0/2523189246 shutdown_connections 2024-03-06T07:23:24.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 --2- 172.21.15.94:0/2523189246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80c4740 0x7f02b80701a0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 --2- 172.21.15.94:0/2523189246 >> v2:172.21.15.145:3300/0 conn(0x7f02b80a8290 0x7f02b81539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 -- 172.21.15.94:0/2523189246 >> 172.21.15.94:0/2523189246 conn(0x7f02b80c5560 msgr2=0x7f02b80c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 -- 172.21.15.94:0/2523189246 shutdown_connections 2024-03-06T07:23:24.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.290+0000 7f02bdb13640 1 -- 172.21.15.94:0/2523189246 wait complete. 2024-03-06T07:23:24.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.291+0000 7f02bdb13640 1 Processor -- start 2024-03-06T07:23:24.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.291+0000 7f02bdb13640 1 -- start start 2024-03-06T07:23:24.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02bdb13640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80a8290 0x7f02b81e0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02bdb13640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02b80c4740 0x7f02b80adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02bdb13640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02b80c69a0 0x7f02b80ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02bdb13640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b806fc80 con 0x7f02b80a8290 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02bdb13640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b806f9a0 con 0x7f02b80c4740 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02bdb13640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b8072d90 con 0x7f02b80c69a0 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80a8290 0x7f02b81e0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80a8290 0x7f02b81e0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49052/0 (socket says 172.21.15.94:49052) 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 -- 172.21.15.94:0/344567335 learned_addr learned my addr 172.21.15.94:0/344567335 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b7fff640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02b80c69a0 0x7f02b80ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02aeffd640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02b80c4740 0x7f02b80adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 -- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02b80c69a0 msgr2=0x7f02b80ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02b80c69a0 0x7f02b80ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 -- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02b80c4740 msgr2=0x7f02b80adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02b80c4740 0x7f02b80adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.292+0000 7f02b77fe640 1 -- 172.21.15.94:0/344567335 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b81eefd0 con 0x7f02b80a8290 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.293+0000 7f02b77fe640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80a8290 0x7f02b81e0180 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f02a800d830 tx=0x7f02a800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.293+0000 7f02b4ff9640 1 -- 172.21.15.94:0/344567335 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a800b6c0 con 0x7f02b80a8290 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.293+0000 7f02b4ff9640 1 -- 172.21.15.94:0/344567335 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a8019070 con 0x7f02b80a8290 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.293+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b8155a30 con 0x7f02b80a8290 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.293+0000 7f02b4ff9640 1 -- 172.21.15.94:0/344567335 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a800ba80 con 0x7f02b80a8290 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.293+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b8155ff0 con 0x7f02b80a8290 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.294+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b804f330 con 0x7f02b80a8290 2024-03-06T07:23:24.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.294+0000 7f02b4ff9640 1 -- 172.21.15.94:0/344567335 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02a801e460 con 0x7f02b80a8290 2024-03-06T07:23:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.295+0000 7f02b4ff9640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f029806b4d0 0x7f029806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.295+0000 7f02b4ff9640 1 -- 172.21.15.94:0/344567335 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f02a808a9b0 con 0x7f02b80a8290 2024-03-06T07:23:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.295+0000 7f02aeffd640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f029806b4d0 0x7f029806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.295+0000 7f02aeffd640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f029806b4d0 0x7f029806d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f02a0035000 tx=0x7f02a0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.299+0000 7f02b4ff9640 1 -- 172.21.15.94:0/344567335 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02a8059730 con 0x7f02b80a8290 2024-03-06T07:23:24.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.590+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f02b80c30b0 con 0x7f02b80a8290 2024-03-06T07:23:24.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.592+0000 7f02b4ff9640 1 -- 172.21.15.94:0/344567335 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f02a8059030 con 0x7f02b80a8290 2024-03-06T07:23:24.594 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:23:24.594 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:23:24.595 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:23:24.595 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:23:24.595 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:24.595 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f029806b4d0 msgr2=0x7f029806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f029806b4d0 0x7f029806d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f02a0035000 tx=0x7f02a0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80a8290 msgr2=0x7f02b81e0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80a8290 0x7f02b81e0180 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f02a800d830 tx=0x7f02a800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 shutdown_connections 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f029806b4d0 0x7f029806d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02b80c69a0 0x7f02b80ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02b80c4740 0x7f02b80adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 --2- 172.21.15.94:0/344567335 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b80a8290 0x7f02b81e0180 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 >> 172.21.15.94:0/344567335 conn(0x7f02b80c5560 msgr2=0x7f02b81517e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 shutdown_connections 2024-03-06T07:23:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.595+0000 7f02bdb13640 1 -- 172.21.15.94:0/344567335 wait complete. 2024-03-06T07:23:24.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:24.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.708+0000 7f62c11f7640 1 Processor -- start 2024-03-06T07:23:24.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.708+0000 7f62c11f7640 1 -- start start 2024-03-06T07:23:24.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62c11f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 0x7f62bc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62c11f7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0c5cc0 con 0x7f62bc1586f0 2024-03-06T07:23:24.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62c11f7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc1591c0 con 0x7f62bc0c5860 2024-03-06T07:23:24.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62c11f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc06df40 con 0x7f62bc0c5480 2024-03-06T07:23:24.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62bb59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f62bc1586f0 0x7f62bc15f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54364/0 (socket says 172.21.15.94:54364) 2024-03-06T07:23:24.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62bb59e640 1 -- 172.21.15.94:0/1511630257 learned_addr learned my addr 172.21.15.94:0/1511630257 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422262331 0 0) 0x7f62bc0c5cc0 con 0x7f62bc1586f0 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a8003520 con 0x7f62bc1586f0 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62bad9d640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 0x7f62bc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3001321678 0 0) 0x7f62a8003520 con 0x7f62bc1586f0 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 msgr2=0x7f62bc155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 0x7f62bc155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 >> v1:172.21.15.145:6789/0 conn(0x7f62bc0c5860 legacy=0x7f62bc06f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc1594a0 con 0x7f62bc1586f0 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f62b40019f0 con 0x7f62bc1586f0 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 >> v1:172.21.15.94:6789/0 conn(0x7f62bc1586f0 legacy=0x7f62bc15f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.709+0000 7f62ba59c640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a80039e0 0x7f62a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62ba59c640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 0x7f62a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62ba59c640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc06df40 con 0x7f62a8004d40 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a800baf0 con 0x7f62a80039e0 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a800bcf0 con 0x7f62bc0c5480 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bb59e640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62b3fff640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 0x7f62a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bad9d640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a80039e0 0x7f62a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bb59e640 1 -- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 msgr2=0x7f62a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bb59e640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 0x7f62a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bb59e640 1 -- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a80039e0 msgr2=0x7f62a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bb59e640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a80039e0 0x7f62a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bb59e640 1 -- 172.21.15.94:0/1511630257 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a8003520 con 0x7f62a8004d40 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62b3fff640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc0c5480 0x7f62a8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.710+0000 7f62bad9d640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a80039e0 0x7f62a8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62bb59e640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62a80051c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f62b400a850 tx=0x7f62a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e0635893b44a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b400a4a0 con 0x7f62a8004d40 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b4034650 con 0x7f62a8004d40 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62ba59c640 1 -- 172.21.15.94:0/1511630257 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4034a40 con 0x7f62a8004d40 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62c11f7640 1 -- 172.21.15.94:0/1511630257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 msgr2=0x7f62a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62c11f7640 1 --2- 172.21.15.94:0/1511630257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62a80051c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f62b400a850 tx=0x7f62a8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62bb59e640 1 -- 172.21.15.94:0/1511630257 reap_dead start 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62c11f7640 1 -- 172.21.15.94:0/1511630257 shutdown_connections 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62c11f7640 1 -- 172.21.15.94:0/1511630257 >> 172.21.15.94:0/1511630257 conn(0x7f62bc06b1a0 msgr2=0x7f62bc06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62c11f7640 1 -- 172.21.15.94:0/1511630257 shutdown_connections 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.711+0000 7f62c11f7640 1 -- 172.21.15.94:0/1511630257 wait complete. 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 Processor -- start 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 -- start start 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62bc0c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62bc0c3730 0x7f62bc0c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc1ee2e0 0x7f62bc1ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc155680 con 0x7f62a8004d40 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc06fdb0 con 0x7f62bc0c3730 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62c11f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc152a20 con 0x7f62bc1ee2e0 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62bad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62bc0c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62bad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62bc0c4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49070/0 (socket says 172.21.15.94:49070) 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.713+0000 7f62bad9d640 1 -- 172.21.15.94:0/3965573263 learned_addr learned my addr 172.21.15.94:0/3965573263 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62b3fff640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62bc0c3730 0x7f62bc0c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62bb59e640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc1ee2e0 0x7f62bc1ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62bad9d640 1 -- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc1ee2e0 msgr2=0x7f62bc1ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62bad9d640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc1ee2e0 0x7f62bc1ee760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62bad9d640 1 -- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62bc0c3730 msgr2=0x7f62bc0c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62bad9d640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62bc0c3730 0x7f62bc0c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62bad9d640 1 -- 172.21.15.94:0/3965573263 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0c40d0 con 0x7f62a8004d40 2024-03-06T07:23:24.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62b3fff640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62bc0c3730 0x7f62bc0c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62bad9d640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62bc0c4800 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f62ac011600 tx=0x7f62ac011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62b37fe640 1 -- 172.21.15.94:0/3965573263 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00f920 con 0x7f62a8004d40 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62b37fe640 1 -- 172.21.15.94:0/3965573263 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ac00fac0 con 0x7f62a8004d40 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc0ab2d0 con 0x7f62a8004d40 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62b37fe640 1 -- 172.21.15.94:0/3965573263 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00fe90 con 0x7f62a8004d40 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.714+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc14db90 con 0x7f62a8004d40 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.715+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc04f330 con 0x7f62a8004d40 2024-03-06T07:23:24.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.716+0000 7f62b37fe640 1 -- 172.21.15.94:0/3965573263 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62ac015460 con 0x7f62a8004d40 2024-03-06T07:23:24.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.716+0000 7f62b37fe640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f629c06b670 0x7f629c06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.716+0000 7f62b37fe640 1 -- 172.21.15.94:0/3965573263 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f62ac08acb0 con 0x7f62a8004d40 2024-03-06T07:23:24.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.716+0000 7f62b3fff640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f629c06b670 0x7f629c06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.717+0000 7f62b3fff640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f629c06b670 0x7f629c06db30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f62bc1551a0 tx=0x7f62a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.720+0000 7f62b37fe640 1 -- 172.21.15.94:0/3965573263 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62bc0c40d0 con 0x7f62a8004d40 2024-03-06T07:23:24.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.935+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62bc0ab0b0 con 0x7f62a8004d40 2024-03-06T07:23:24.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.936+0000 7f62b37fe640 1 -- 172.21.15.94:0/3965573263 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4990 (secure 0 0 0) 0x7f62ac019030 con 0x7f62a8004d40 2024-03-06T07:23:24.938 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:24.938 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":119,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1362727065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1362727065},{"type":"v1","addr":"172.21.15.145:6836","nonce":1362727065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:22.175096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/392019985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":392019985},{"type":"v1","addr":"172.21.15.94:6835","nonce":392019985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:23:24.938 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 119 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f629c06b670 msgr2=0x7f629c06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f629c06b670 0x7f629c06db30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f62bc1551a0 tx=0x7f62a4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 msgr2=0x7f62bc0c4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62bc0c4800 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f62ac011600 tx=0x7f62ac011ac0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 shutdown_connections 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f629c06b670 0x7f629c06db30 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62bc1ee2e0 0x7f62bc1ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62bc0c3730 0x7f62bc0c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 --2- 172.21.15.94:0/3965573263 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f62a8004d40 0x7f62bc0c4800 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 >> 172.21.15.94:0/3965573263 conn(0x7f62bc06b1a0 msgr2=0x7f62bc1513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 shutdown_connections 2024-03-06T07:23:24.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:24.938+0000 7f62c11f7640 1 -- 172.21.15.94:0/3965573263 wait complete. 2024-03-06T07:23:25.353 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:20.150839+0000', 'modified': '2024-03-06T07:23:22.175096+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8301}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8301': {'gid': 8301, 'name': 'c', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.94:6835/392019985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 392019985}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 392019985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8301, 'qdb_cluster': [8301]} 2024-03-06T07:23:25.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:25.354 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:23:25.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.454+0000 7fccd5acf640 1 Processor -- start 2024-03-06T07:23:25.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.454+0000 7fccd5acf640 1 -- start start 2024-03-06T07:23:25.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fccd5acf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fccd00c69f0 0x7fccd00c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fccd5acf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fccd00c8950 0x7fccd00acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fccd5acf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd0155130 con 0x7fccd00ad2b0 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fccd5acf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd0155440 con 0x7fccd00c69f0 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fccd5acf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd00c5d50 con 0x7fccd00c8950 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fcccf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fccd00c69f0 0x7fccd00c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fcccffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fccd00ad2b0 0x7fccd01558d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54376/0 (socket says 172.21.15.94:54376) 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fcccffff640 1 -- 172.21.15.94:0/3037160704 learned_addr learned my addr 172.21.15.94:0/3037160704 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fcccf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fccd00c69f0 0x7fccd00c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49084/0 (socket says 172.21.15.94:49084) 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.455+0000 7fccceffd640 1 --2- 172.21.15.94:0/3037160704 >> v2:172.21.15.145:3300/0 conn(0x7fccd00c8950 0x7fccd00acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.456+0000 7fcccf7fe640 1 -- 172.21.15.94:0/3037160704 >> v2:172.21.15.145:3300/0 conn(0x7fccd00c8950 msgr2=0x7fccd00acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.456+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/3037160704 >> v2:172.21.15.145:3300/0 conn(0x7fccd00c8950 0x7fccd00acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.456+0000 7fcccf7fe640 1 -- 172.21.15.94:0/3037160704 >> v1:172.21.15.94:6789/0 conn(0x7fccd00ad2b0 legacy=0x7fccd01558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.456+0000 7fcccf7fe640 1 -- 172.21.15.94:0/3037160704 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd00c6030 con 0x7fccd00c69f0 2024-03-06T07:23:25.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.456+0000 7fccce7fc640 1 -- 172.21.15.94:0/3037160704 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980542045 0 0) 0x7fccc8001490 con 0x7fccd00ad2b0 2024-03-06T07:23:25.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.456+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/3037160704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd00c6dd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fccd00c7430 tx=0x7fccc002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83b21fe837fd93b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.456+0000 7fccce7fc640 1 -- 172.21.15.94:0/3037160704 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc00315a0 con 0x7fccd00c69f0 2024-03-06T07:23:25.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccce7fc640 1 -- 172.21.15.94:0/3037160704 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0037070 con 0x7fccd00c69f0 2024-03-06T07:23:25.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccce7fc640 1 -- 172.21.15.94:0/3037160704 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0031960 con 0x7fccd00c69f0 2024-03-06T07:23:25.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 -- 172.21.15.94:0/3037160704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 msgr2=0x7fccd00c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 --2- 172.21.15.94:0/3037160704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd00c6dd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fccd00c7430 tx=0x7fccc002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 -- 172.21.15.94:0/3037160704 shutdown_connections 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 --2- 172.21.15.94:0/3037160704 >> v2:172.21.15.145:3300/0 conn(0x7fccd00c8950 0x7fccd00acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 --2- 172.21.15.94:0/3037160704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd00c6dd0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 -- 172.21.15.94:0/3037160704 >> 172.21.15.94:0/3037160704 conn(0x7fccd00cae70 msgr2=0x7fccd00cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 -- 172.21.15.94:0/3037160704 shutdown_connections 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.457+0000 7fccd5acf640 1 -- 172.21.15.94:0/3037160704 wait complete. 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.458+0000 7fccd5acf640 1 Processor -- start 2024-03-06T07:23:25.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccd5acf640 1 -- start start 2024-03-06T07:23:25.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccd5acf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd00ad2b0 0x7fccd006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccd5acf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd01ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccd5acf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd00c8950 0x7fccd01eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccd5acf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd0152a20 con 0x7fccd00c69f0 2024-03-06T07:23:25.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccd5acf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd0155fe0 con 0x7fccd00ad2b0 2024-03-06T07:23:25.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccd5acf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd0152ba0 con 0x7fccd00c8950 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd01ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd01ee610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49086/0 (socket says 172.21.15.94:49086) 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.459+0000 7fccceffd640 1 -- 172.21.15.94:0/2655900565 learned_addr learned my addr 172.21.15.94:0/2655900565 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd00ad2b0 0x7fccd006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fcccffff640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd00c8950 0x7fccd01eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccceffd640 1 -- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd00c8950 msgr2=0x7fccd01eeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccceffd640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd00c8950 0x7fccd01eeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccceffd640 1 -- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd00ad2b0 msgr2=0x7fccd006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccceffd640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd00ad2b0 0x7fccd006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccceffd640 1 -- 172.21.15.94:0/2655900565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd014d360 con 0x7fccd00c69f0 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd00ad2b0 0x7fccd006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccceffd640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd01ee610 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fccc400d800 tx=0x7fccc400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccb7fff640 1 -- 172.21.15.94:0/2655900565 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400b6c0 con 0x7fccd00c69f0 2024-03-06T07:23:25.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccb7fff640 1 -- 172.21.15.94:0/2655900565 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc4019070 con 0x7fccd00c69f0 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccb7fff640 1 -- 172.21.15.94:0/2655900565 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400ba80 con 0x7fccd00c69f0 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd01ebf70 con 0x7fccd00c69f0 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.460+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd01eb490 con 0x7fccd00c69f0 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.462+0000 7fccb7fff640 1 -- 172.21.15.94:0/2655900565 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccc400bc20 con 0x7fccd00c69f0 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.462+0000 7fccb7fff640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fccbc06b4d0 0x7fccbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.462+0000 7fccb7fff640 1 -- 172.21.15.94:0/2655900565 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fccc408a870 con 0x7fccd00c69f0 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.462+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fccbc06b4d0 0x7fccbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 Processor -- start 2024-03-06T07:23:25.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 -- start start 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fccbc06b4d0 0x7fccbc06d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036000 tx=0x7fccc0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c35a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c35f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c364b0 0x561537c32680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x561537b9c6b0 con 0x561537c38330 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561537b45a60 con 0x561537c37f50 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.463+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561537b98750 con 0x561537c364b0 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c35f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c35f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49102/0 (socket says 172.21.15.94:49102) 2024-03-06T07:23:25.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 -- 172.21.15.94:0/3328183558 learned_addr learned my addr 172.21.15.94:0/3328183558 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fcccffff640 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c364b0 0x561537c32680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c35a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 -- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c364b0 msgr2=0x561537c32680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c364b0 0x561537c32680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 -- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 msgr2=0x561537c35a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c35a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 -- 172.21.15.94:0/3328183558 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561537b63f30 con 0x561537c38330 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccceffd640 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c35f70 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fccc4061040 tx=0x7fccc4011340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cfb9e74e2d9de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccb57fa640 1 -- 172.21.15.94:0/3328183558 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4087070 con 0x561537c38330 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccb57fa640 1 -- 172.21.15.94:0/3328183558 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc405cfb0 con 0x561537c38330 2024-03-06T07:23:25.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.464+0000 7fccb57fa640 1 -- 172.21.15.94:0/3328183558 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4070070 con 0x561537c38330 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 -- 172.21.15.94:0/3328183558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 msgr2=0x561537c35f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c35f70 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fccc4061040 tx=0x7fccc4011340 comp rx=0 tx=0).stop 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 -- 172.21.15.94:0/3328183558 shutdown_connections 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c364b0 0x561537c32680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c35f70 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 --2- 172.21.15.94:0/3328183558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c35a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 -- 172.21.15.94:0/3328183558 >> 172.21.15.94:0/3328183558 conn(0x561537c347d0 msgr2=0x561537c593c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 -- 172.21.15.94:0/3328183558 shutdown_connections 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 -- 172.21.15.94:0/3328183558 wait complete. 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.465+0000 7fccd878a740 1 -- start start 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 1 Processor -- start 2024-03-06T07:23:25.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c37f50 0x561537c5ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c5e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c5d1f0 0x561537c54fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x561537c5a230 con 0x561537c38330 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561537b58ea0 con 0x561537c5d1f0 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561537c59c70 con 0x561537c37f50 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccd878a740 10 client.0 _ll_register_callbacks cb 0x561537b58870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c5e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c5e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49118/0 (socket says 172.21.15.94:49118) 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fccceffd640 1 -- 172.21.15.94:0/1335647824 learned_addr learned my addr 172.21.15.94:0/1335647824 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.466+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c37f50 0x561537c5ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fcccffff640 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c5d1f0 0x561537c54fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccceffd640 1 -- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c37f50 msgr2=0x561537c5ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccceffd640 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c37f50 0x561537c5ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccceffd640 1 -- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c5d1f0 msgr2=0x561537c54fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccceffd640 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c5d1f0 0x561537c54fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccceffd640 1 -- 172.21.15.94:0/1335647824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561537b59270 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccceffd640 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c5e0b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fccc4058b60 tx=0x7fccc4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53269d10cbb400e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccae7fc640 1 -- 172.21.15.94:0/1335647824 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4075070 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccae7fc640 1 -- 172.21.15.94:0/1335647824 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fccc40554d0 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccc405cfb0 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccae7fc640 1 -- 172.21.15.94:0/1335647824 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccc407f050 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 10 client.8445 fetch_fsmap learned FSMap version 119 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x5615378b12d0 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccae7fc640 1 -- 172.21.15.94:0/1335647824 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc405cfb0 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccae7fc640 1 -- 172.21.15.94:0/1335647824 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 119) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fccc4082020 con 0x561537c38330 2024-03-06T07:23:25.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 10 client.8445 fetch_fsmap finished waiting for FSMap version 119 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 10 client.8445 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 20 client.8445 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c5ad00 0x561537c93760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 4 client.8445 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.467+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 --> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561537ba9bf0 con 0x561537c5ad00 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.468+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c5ad00 0x561537c93760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.468+0000 7fccae7fc640 10 client.8445 ms_handle_connect on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.468+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c5ad00 0x561537c93760 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=838e800a72b8b506 server_cookie=1e87e992dfd01e76 in_seq=0 out_seq=0 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.468+0000 7fccae7fc640 1 -- 172.21.15.94:0/1335647824 <== mds.0 v2:172.21.15.94:6834/392019985 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561537ba9bf0 con 0x561537c5ad00 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.468+0000 7fccae7fc640 10 client.8445 handle_command_reply: tid=0 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.468+0000 7fccd878a740 1 client.8445 shutdown 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 msgr2=0x561537c5e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c5e0b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fccc4058b60 tx=0x7fccc4010000 comp rx=0 tx=0).stop 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 shutdown_connections 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c5ad00 0x561537c93760 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c5d1f0 0x561537c54fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c38330 0x561537c5e0b0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccae7fc640 0 client.8445 ms_handle_reset on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 --2- 172.21.15.94:0/1335647824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c37f50 0x561537c5ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 >> 172.21.15.94:0/1335647824 conn(0x561537c364b0 msgr2=0x561537c368e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 shutdown_connections 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 1 -- 172.21.15.94:0/1335647824 wait complete. 2024-03-06T07:23:25.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.469+0000 7fccd878a740 20 client.8445 trim_cache size 0 max 16384 2024-03-06T07:23:25.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.485+0000 7fccd878a740 1 Processor -- start 2024-03-06T07:23:25.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.485+0000 7fccd878a740 1 -- start start 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccd878a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c37f50 0x561537c31700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c38330 0x561537c81c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c5d1f0 0x561537c82180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccd878a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd01f6fc0 con 0x561537c37f50 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd0156d50 con 0x561537c38330 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd01f6b70 con 0x561537c5d1f0 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c37f50 0x561537c31700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c37f50 0x561537c31700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49130/0 (socket says 172.21.15.94:49130) 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 -- 172.21.15.94:0/2273187039 learned_addr learned my addr 172.21.15.94:0/2273187039 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccffff640 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c5d1f0 0x561537c82180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccceffd640 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c38330 0x561537c81c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 -- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c5d1f0 msgr2=0x561537c82180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c5d1f0 0x561537c82180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 -- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c38330 msgr2=0x561537c81c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c38330 0x561537c81c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fcccf7fe640 1 -- 172.21.15.94:0/2273187039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56153797b6b0 con 0x561537c37f50 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.486+0000 7fccceffd640 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c38330 0x561537c81c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c37f50 0x561537c31700 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fccc0034040 tx=0x7fccc0031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5822e77356bae7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccb4ff9640 1 -- 172.21.15.94:0/2273187039 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc005a070 con 0x561537c37f50 2024-03-06T07:23:25.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccb4ff9640 1 -- 172.21.15.94:0/2273187039 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0037040 con 0x561537c37f50 2024-03-06T07:23:25.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccb4ff9640 1 -- 172.21.15.94:0/2273187039 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0054bc0 con 0x561537c37f50 2024-03-06T07:23:25.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 -- 172.21.15.94:0/2273187039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c37f50 msgr2=0x561537c31700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c37f50 0x561537c31700 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fccc0034040 tx=0x7fccc0031000 comp rx=0 tx=0).stop 2024-03-06T07:23:25.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 -- 172.21.15.94:0/2273187039 shutdown_connections 2024-03-06T07:23:25.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c5d1f0 0x561537c82180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c38330 0x561537c81c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 --2- 172.21.15.94:0/2273187039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c37f50 0x561537c31700 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 -- 172.21.15.94:0/2273187039 >> 172.21.15.94:0/2273187039 conn(0x561537c364b0 msgr2=0x561537c2dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 -- 172.21.15.94:0/2273187039 shutdown_connections 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.487+0000 7fccd878a740 1 -- 172.21.15.94:0/2273187039 wait complete. 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.488+0000 7fccd878a740 1 -- start start 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.488+0000 7fccd878a740 1 Processor -- start 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c856b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c38330 0x561537c82040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c5d1f0 0x561537c51a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x561537c880c0 con 0x561537c5d1f0 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561537c5a230 con 0x561537c37f50 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561537c883c0 con 0x561537c38330 2024-03-06T07:23:25.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 10 client.0 _ll_register_callbacks cb 0x7fccc4006d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c5d1f0 0x561537c51a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c5d1f0 0x561537c51a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49138/0 (socket says 172.21.15.94:49138) 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 -- 172.21.15.94:0/1029797076 learned_addr learned my addr 172.21.15.94:0/1029797076 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccceffd640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c38330 0x561537c82040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c856b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 -- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c38330 msgr2=0x561537c82040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c38330 0x561537c82040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 -- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 msgr2=0x561537c856b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c856b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 -- 172.21.15.94:0/1029797076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561537c85bf0 con 0x561537c5d1f0 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c856b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fcccffff640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c5d1f0 0x561537c51a10 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fccc80012e0 tx=0x7fccc8014a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c8b448ec45ce0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccae7fc640 1 -- 172.21.15.94:0/1029797076 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc80167b0 con 0x561537c5d1f0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccae7fc640 1 -- 172.21.15.94:0/1029797076 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fccc8016950 con 0x561537c5d1f0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.489+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccc0037040 con 0x561537c5d1f0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccae7fc640 1 -- 172.21.15.94:0/1029797076 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccc8026050 con 0x561537c5d1f0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 10 client.8451 fetch_fsmap learned FSMap version 119 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x561537c52470 con 0x561537c5d1f0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccae7fc640 1 -- 172.21.15.94:0/1029797076 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0037040 con 0x561537c5d1f0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccae7fc640 1 -- 172.21.15.94:0/1029797076 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 119) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fccc8029020 con 0x561537c5d1f0 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 10 client.8451 fetch_fsmap finished waiting for FSMap version 119 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 10 client.8451 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 20 client.8451 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c80c30 0x561537c81010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 4 client.8451 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:25.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 --> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561537ba3f40 con 0x561537c80c30 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c80c30 0x561537c81010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccae7fc640 10 client.8451 ms_handle_connect on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fcccf7fe640 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c80c30 0x561537c81010 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b275c4c8d9c08159 server_cookie=11a656c03879a741 in_seq=0 out_seq=0 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccae7fc640 1 -- 172.21.15.94:0/1029797076 <== mds.0 v2:172.21.15.94:6834/392019985 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x561537ba3f40 con 0x561537c80c30 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccae7fc640 10 client.8451 handle_command_reply: tid=0 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.490+0000 7fccd878a740 1 client.8451 shutdown 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c5d1f0 msgr2=0x561537c51a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c5d1f0 0x561537c51a10 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fccc80012e0 tx=0x7fccc8014a50 comp rx=0 tx=0).stop 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 shutdown_connections 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x561537c80c30 0x561537c81010 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x561537c5d1f0 0x561537c51a10 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccae7fc640 0 client.8451 ms_handle_reset on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:25.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561537c38330 0x561537c82040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 --2- 172.21.15.94:0/1029797076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561537c37f50 0x561537c856b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 >> 172.21.15.94:0/1029797076 conn(0x561537c364b0 msgr2=0x561537c31700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 shutdown_connections 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 1 -- 172.21.15.94:0/1029797076 wait complete. 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd878a740 20 client.8451 trim_cache size 0 max 16384 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:23:25.501 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 8301, 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.6672660669999999, 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 119, 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 135, 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 135, 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 9.7830935799999992 2024-03-06T07:23:25.502 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:25.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fccbc06b4d0 msgr2=0x7fccbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fccbc06b4d0 0x7fccbc06d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036000 tx=0x7fccc0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:25.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 msgr2=0x7fccd01ee610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd01ee610 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fccc400d800 tx=0x7fccc400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 shutdown_connections 2024-03-06T07:23:25.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fccbc06b4d0 0x7fccbc06d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd00c8950 0x7fccd01eeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccd00c69f0 0x7fccd01ee610 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 --2- 172.21.15.94:0/2655900565 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd00ad2b0 0x7fccd006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.491+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 >> 172.21.15.94:0/2655900565 conn(0x7fccd00cae70 msgr2=0x7fccd00bfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.492+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 shutdown_connections 2024-03-06T07:23:25.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.492+0000 7fccd5acf640 1 -- 172.21.15.94:0/2655900565 wait complete. 2024-03-06T07:23:25.508 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:25.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.609+0000 7f1ac5d25640 1 Processor -- start 2024-03-06T07:23:25.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.609+0000 7f1ac5d25640 1 -- start start 2024-03-06T07:23:25.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1ac5d25640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac014f360 0x7f1ac0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1ac5d25640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ac0158b60 0x7f1ac0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1ac5d25640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00c73d0 con 0x7f1ac00c88c0 2024-03-06T07:23:25.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1ac5d25640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac00c8dd0 con 0x7f1ac0158b60 2024-03-06T07:23:25.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1ac5d25640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00be2e0 con 0x7f1ac014f360 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1abf7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac014f360 0x7f1ac0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1abffff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ac0158b60 0x7f1ac0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1ab6ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f1ac00c88c0 0x7f1ac00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:34534/0 (socket says 172.21.15.94:34534) 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/131439216 learned_addr learned my addr 172.21.15.94:0/131439216 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1abf7fe640 1 -- 172.21.15.94:0/131439216 >> v2:172.21.15.145:3300/0 conn(0x7f1ac0158b60 msgr2=0x7f1ac0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1abf7fe640 1 --2- 172.21.15.94:0/131439216 >> v2:172.21.15.145:3300/0 conn(0x7f1ac0158b60 0x7f1ac0158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1abf7fe640 1 -- 172.21.15.94:0/131439216 >> v1:172.21.15.145:6789/0 conn(0x7f1ac00c88c0 legacy=0x7f1ac00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.610+0000 7f1abf7fe640 1 -- 172.21.15.94:0/131439216 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00c7670 con 0x7f1ac014f360 2024-03-06T07:23:25.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.611+0000 7f1abf7fe640 1 --2- 172.21.15.94:0/131439216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac014f360 0x7f1ac0155c10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00097a0 tx=0x7f1ab002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d710cf2fc4ec927 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.611+0000 7f1abeffd640 1 -- 172.21.15.94:0/131439216 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00315a0 con 0x7f1ac014f360 2024-03-06T07:23:25.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.611+0000 7f1abeffd640 1 -- 172.21.15.94:0/131439216 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab0037070 con 0x7f1ac014f360 2024-03-06T07:23:25.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.611+0000 7f1abeffd640 1 -- 172.21.15.94:0/131439216 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0031960 con 0x7f1ac014f360 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 -- 172.21.15.94:0/131439216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac014f360 msgr2=0x7f1ac0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/131439216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac014f360 0x7f1ac0155c10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00097a0 tx=0x7f1ab002b840 comp rx=0 tx=0).stop 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 -- 172.21.15.94:0/131439216 shutdown_connections 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/131439216 >> v2:172.21.15.145:3300/0 conn(0x7f1ac0158b60 0x7f1ac0158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/131439216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac014f360 0x7f1ac0155c10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 -- 172.21.15.94:0/131439216 >> 172.21.15.94:0/131439216 conn(0x7f1ac00ac570 msgr2=0x7f1ac00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 -- 172.21.15.94:0/131439216 shutdown_connections 2024-03-06T07:23:25.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.612+0000 7f1ac5d25640 1 -- 172.21.15.94:0/131439216 wait complete. 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.613+0000 7f1ac5d25640 1 Processor -- start 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.613+0000 7f1ac5d25640 1 -- start start 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ac5d25640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ac00c88c0 0x7f1ac01df370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ac5d25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac014f360 0x7f1ac01eb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ac5d25640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac0158b60 0x7f1ac0157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ac5d25640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00c5a50 con 0x7f1ac014f360 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ac5d25640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00c7c30 con 0x7f1ac00c88c0 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ac5d25640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00c5bd0 con 0x7f1ac0158b60 2024-03-06T07:23:25.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac014f360 0x7f1ac01eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac014f360 0x7f1ac01eb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49142/0 (socket says 172.21.15.94:49142) 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/760842013 learned_addr learned my addr 172.21.15.94:0/760842013 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1abffff640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac0158b60 0x7f1ac0157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1abf7fe640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ac00c88c0 0x7f1ac01df370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac0158b60 msgr2=0x7f1ac0157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac0158b60 0x7f1ac0157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ac00c88c0 msgr2=0x7f1ac01df370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ac00c88c0 0x7f1ac01df370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 -- 172.21.15.94:0/760842013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00b0410 con 0x7f1ac014f360 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.614+0000 7f1ab6ffd640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac014f360 0x7f1ac01eb690 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c470 tx=0x7f1aa800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.615+0000 7f1abcff9640 1 -- 172.21.15.94:0/760842013 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8017070 con 0x7f1ac014f360 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.615+0000 7f1abcff9640 1 -- 172.21.15.94:0/760842013 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa8005ce0 con 0x7f1ac014f360 2024-03-06T07:23:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.615+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac00b0130 con 0x7f1ac014f360 2024-03-06T07:23:25.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.615+0000 7f1abcff9640 1 -- 172.21.15.94:0/760842013 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80072d0 con 0x7f1ac014f360 2024-03-06T07:23:25.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.615+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac01deb70 con 0x7f1ac014f360 2024-03-06T07:23:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.616+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac004f330 con 0x7f1ac014f360 2024-03-06T07:23:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.616+0000 7f1abcff9640 1 -- 172.21.15.94:0/760842013 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1aa801a070 con 0x7f1ac014f360 2024-03-06T07:23:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.616+0000 7f1abcff9640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1aa006b500 0x7f1aa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.616+0000 7f1abcff9640 1 -- 172.21.15.94:0/760842013 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1aa8089560 con 0x7f1ac014f360 2024-03-06T07:23:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.617+0000 7f1abf7fe640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1aa006b500 0x7f1aa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.617+0000 7f1abf7fe640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1aa006b500 0x7f1aa006d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036000 tx=0x7f1ab003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.621+0000 7f1abcff9640 1 -- 172.21.15.94:0/760842013 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1aa8057c00 con 0x7f1ac014f360 2024-03-06T07:23:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.871+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ac00b0680 con 0x7f1ac014f360 2024-03-06T07:23:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.872+0000 7f1abcff9640 1 -- 172.21.15.94:0/760842013 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4990 (secure 0 0 0) 0x7f1aa805bca0 con 0x7f1ac014f360 2024-03-06T07:23:25.873 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:25.873 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":119,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1362727065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1362727065},{"type":"v1","addr":"172.21.15.145:6836","nonce":1362727065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:22.175096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/392019985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":392019985},{"type":"v1","addr":"172.21.15.94:6835","nonce":392019985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:23:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 119 2024-03-06T07:23:25.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.873+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1aa006b500 msgr2=0x7f1aa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1aa006b500 0x7f1aa006d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0036000 tx=0x7f1ab003d040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac014f360 msgr2=0x7f1ac01eb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac014f360 0x7f1ac01eb690 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800c470 tx=0x7f1aa800c930 comp rx=0 tx=0).stop 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 shutdown_connections 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1aa006b500 0x7f1aa006d9c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ac0158b60 0x7f1ac0157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ac014f360 0x7f1ac01eb690 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 --2- 172.21.15.94:0/760842013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ac00c88c0 0x7f1ac01df370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 >> 172.21.15.94:0/760842013 conn(0x7f1ac00ac570 msgr2=0x7f1ac01e1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 shutdown_connections 2024-03-06T07:23:25.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.874+0000 7f1ac5d25640 1 -- 172.21.15.94:0/760842013 wait complete. 2024-03-06T07:23:25.887 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 119, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:20.150839+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 119, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8301": { "addr": "172.21.15.94:6835/392019985", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6834", "nonce": 392019985, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 392019985, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8301, "incarnation": 117, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:23:22.175096+0000", "qdb_cluster": [ 8301 ], "qdb_leader": 8301, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8301 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/1362727065", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1362727065, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1362727065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7723, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/3826543722", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 3826543722, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 3826543722, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7726, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6833/3405703874", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3405703874, "type": "v2" }, { "addr": "172.21.15.94:6833", "nonce": 3405703874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8298, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:25.887 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:25.887 INFO:tasks.cephfs.test_failover:current = [{'gid': 8298, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6833/3405703874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3405703874}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 3405703874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7726, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/3826543722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3826543722}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3826543722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 8301, 'name': 'c', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.94:6835/392019985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 392019985}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 392019985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7723, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/1362727065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1362727065}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1362727065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}] 2024-03-06T07:23:25.887 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-03-06T07:23:25.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.996+0000 7fc259a47640 1 Processor -- start 2024-03-06T07:23:25.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.996+0000 7fc259a47640 1 -- start start 2024-03-06T07:23:26.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc259a47640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc2541505f0 0x7fc254156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc259a47640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2540c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc259a47640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc254073580 con 0x7fc2540c7f60 2024-03-06T07:23:26.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc259a47640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc254070b20 con 0x7fc2541505f0 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc259a47640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2540701f0 con 0x7fc2540c6710 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc24a7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc2540c7f60 0x7fc254072510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54386/0 (socket says 172.21.15.94:54386) 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc24a7fc640 1 -- 172.21.15.94:0/2855574306 learned_addr learned my addr 172.21.15.94:0/2855574306 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc252ffd640 1 --2- 172.21.15.94:0/2855574306 >> v2:172.21.15.145:3300/0 conn(0x7fc2541505f0 0x7fc254156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.997+0000 7fc2537fe640 1 --2- 172.21.15.94:0/2855574306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2540c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.998+0000 7fc2527fc640 1 -- 172.21.15.94:0/2855574306 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949778553 0 0) 0x7fc254073580 con 0x7fc2540c7f60 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.998+0000 7fc2527fc640 1 -- 172.21.15.94:0/2855574306 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc240003520 con 0x7fc2540c7f60 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.998+0000 7fc2537fe640 1 -- 172.21.15.94:0/2855574306 >> v2:172.21.15.145:3300/0 conn(0x7fc2541505f0 msgr2=0x7fc254156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.998+0000 7fc2537fe640 1 --2- 172.21.15.94:0/2855574306 >> v2:172.21.15.145:3300/0 conn(0x7fc2541505f0 0x7fc254156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.998+0000 7fc2537fe640 1 -- 172.21.15.94:0/2855574306 >> v1:172.21.15.94:6789/0 conn(0x7fc2540c7f60 legacy=0x7fc254072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.998+0000 7fc2537fe640 1 -- 172.21.15.94:0/2855574306 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2540704d0 con 0x7fc2540c6710 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.998+0000 7fc252ffd640 1 --2- 172.21.15.94:0/2855574306 >> v2:172.21.15.145:3300/0 conn(0x7fc2541505f0 0x7fc254156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:26.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc2537fe640 1 --2- 172.21.15.94:0/2855574306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2540c6ba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009d90 tx=0x7fc24c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=204ebe7422390df8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc2527fc640 1 -- 172.21.15.94:0/2855574306 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c039070 con 0x7fc2540c6710 2024-03-06T07:23:26.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc2527fc640 1 -- 172.21.15.94:0/2855574306 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c02cc50 con 0x7fc2540c6710 2024-03-06T07:23:26.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc2527fc640 1 -- 172.21.15.94:0/2855574306 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c034440 con 0x7fc2540c6710 2024-03-06T07:23:26.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 -- 172.21.15.94:0/2855574306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 msgr2=0x7fc2540c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 --2- 172.21.15.94:0/2855574306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2540c6ba0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009d90 tx=0x7fc24c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 -- 172.21.15.94:0/2855574306 shutdown_connections 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 --2- 172.21.15.94:0/2855574306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2540c6ba0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 --2- 172.21.15.94:0/2855574306 >> v2:172.21.15.145:3300/0 conn(0x7fc2541505f0 0x7fc254156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 -- 172.21.15.94:0/2855574306 >> 172.21.15.94:0/2855574306 conn(0x7fc2540aa5c0 msgr2=0x7fc2540aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 -- 172.21.15.94:0/2855574306 shutdown_connections 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:25.999+0000 7fc259a47640 1 -- 172.21.15.94:0/2855574306 wait complete. 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.000+0000 7fc259a47640 1 Processor -- start 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc259a47640 1 -- start start 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc259a47640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2541ec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc259a47640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2540c7f60 0x7fc2540c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc259a47640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2541505f0 0x7fc254155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc259a47640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc254072c20 con 0x7fc2541505f0 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc259a47640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25406fec0 con 0x7fc2540c7f60 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc259a47640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc254072da0 con 0x7fc2540c6710 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc2537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2541505f0 0x7fc254155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc2537fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2541505f0 0x7fc254155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49156/0 (socket says 172.21.15.94:49156) 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc2537fe640 1 -- 172.21.15.94:0/2679525190 learned_addr learned my addr 172.21.15.94:0/2679525190 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc252ffd640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2541ec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.001+0000 7fc24a7fc640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2540c7f60 0x7fc2540c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc2537fe640 1 -- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 msgr2=0x7fc2541ec5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc2537fe640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2541ec5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc2537fe640 1 -- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2540c7f60 msgr2=0x7fc2540c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc2537fe640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2540c7f60 0x7fc2540c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc2537fe640 1 -- 172.21.15.94:0/2679525190 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2541e8fe0 con 0x7fc2541505f0 2024-03-06T07:23:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc2537fe640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2541505f0 0x7fc254155270 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009eb0 tx=0x7fc24c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc250ff9640 1 -- 172.21.15.94:0/2679525190 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c039050 con 0x7fc2541505f0 2024-03-06T07:23:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc250ff9640 1 -- 172.21.15.94:0/2679525190 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c045070 con 0x7fc2541505f0 2024-03-06T07:23:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2541e92c0 con 0x7fc2541505f0 2024-03-06T07:23:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc250ff9640 1 -- 172.21.15.94:0/2679525190 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c009450 con 0x7fc2541505f0 2024-03-06T07:23:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.002+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2541ebdc0 con 0x7fc2541505f0 2024-03-06T07:23:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.003+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25404f330 con 0x7fc2541505f0 2024-03-06T07:23:26.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.003+0000 7fc250ff9640 1 -- 172.21.15.94:0/2679525190 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc24c03e610 con 0x7fc2541505f0 2024-03-06T07:23:26.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.004+0000 7fc250ff9640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc23406b500 0x7fc23406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.004+0000 7fc250ff9640 1 -- 172.21.15.94:0/2679525190 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc24c0ac5e0 con 0x7fc2541505f0 2024-03-06T07:23:26.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.004+0000 7fc252ffd640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc23406b500 0x7fc23406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.005+0000 7fc252ffd640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc23406b500 0x7fc23406d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc244009560 tx=0x7fc244009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.008+0000 7fc250ff9640 1 -- 172.21.15.94:0/2679525190 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc24c04e020 con 0x7fc2541505f0 2024-03-06T07:23:26.086 INFO:tasks.ceph.mds.c.smithi094.stderr:2024-03-06T07:23:26.084+0000 7f6bcf43b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:26.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.187+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fc2540c40c0 con 0x7fc2541505f0 2024-03-06T07:23:26.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.190+0000 7fc250ff9640 1 -- 172.21.15.94:0/2679525190 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v46)=0 v46) v1 ==== 116+0+0 (secure 0 0 0) 0x7fc24c07ac80 con 0x7fc2541505f0 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc23406b500 msgr2=0x7fc23406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc23406b500 0x7fc23406d9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc244009560 tx=0x7fc244009190 comp rx=0 tx=0).stop 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2541505f0 msgr2=0x7fc254155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2541505f0 0x7fc254155270 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009eb0 tx=0x7fc24c004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 shutdown_connections 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc23406b500 0x7fc23406d9c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2541505f0 0x7fc254155270 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2540c7f60 0x7fc2540c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 --2- 172.21.15.94:0/2679525190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2540c6710 0x7fc2541ec5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 >> 172.21.15.94:0/2679525190 conn(0x7fc2540aa5c0 msgr2=0x7fc2540a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 shutdown_connections 2024-03-06T07:23:26.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.192+0000 7fc259a47640 1 -- 172.21.15.94:0/2679525190 wait complete. 2024-03-06T07:23:26.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-03-06T07:23:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.309+0000 7f84900ed640 1 Processor -- start 2024-03-06T07:23:26.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.309+0000 7f84900ed640 1 -- start start 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f84900ed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84880ca1a0 0x7f84880c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f84900ed640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f84880c5310 0x7f84880c9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f84900ed640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8488155d20 con 0x7f84880af720 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f84900ed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84880ad900 con 0x7f84880ca1a0 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f84900ed640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8488155fc0 con 0x7f84880c5310 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f848d688640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84880ca1a0 0x7f84880c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f848d688640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84880ca1a0 0x7f84880c8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49168/0 (socket says 172.21.15.94:49168) 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f848d688640 1 -- 172.21.15.94:0/4143117734 learned_addr learned my addr 172.21.15.94:0/4143117734 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.310+0000 7f848e68a640 1 --2- 172.21.15.94:0/4143117734 >> v2:172.21.15.145:3300/0 conn(0x7f84880c5310 0x7f84880c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848d688640 1 -- 172.21.15.94:0/4143117734 >> v2:172.21.15.145:3300/0 conn(0x7f84880c5310 msgr2=0x7f84880c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848d688640 1 --2- 172.21.15.94:0/4143117734 >> v2:172.21.15.145:3300/0 conn(0x7f84880c5310 0x7f84880c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848d688640 1 -- 172.21.15.94:0/4143117734 >> v1:172.21.15.145:6789/0 conn(0x7f84880af720 legacy=0x7f848814ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848d688640 1 -- 172.21.15.94:0/4143117734 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84880adb50 con 0x7f84880ca1a0 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848e68a640 1 --2- 172.21.15.94:0/4143117734 >> v2:172.21.15.145:3300/0 conn(0x7f84880c5310 0x7f84880c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848d688640 1 --2- 172.21.15.94:0/4143117734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880ca1a0 0x7f84880c8b30 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f848808c630 tx=0x7f848402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb65e9dbd260c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848ce87640 1 -- 172.21.15.94:0/4143117734 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8484038070 con 0x7f84880ca1a0 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848ce87640 1 -- 172.21.15.94:0/4143117734 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8484031520 con 0x7f84880ca1a0 2024-03-06T07:23:26.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.311+0000 7f848ce87640 1 -- 172.21.15.94:0/4143117734 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84840318f0 con 0x7f84880ca1a0 2024-03-06T07:23:26.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 -- 172.21.15.94:0/4143117734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880ca1a0 msgr2=0x7f84880c8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 --2- 172.21.15.94:0/4143117734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880ca1a0 0x7f84880c8b30 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f848808c630 tx=0x7f848402b7c0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 -- 172.21.15.94:0/4143117734 shutdown_connections 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 --2- 172.21.15.94:0/4143117734 >> v2:172.21.15.145:3300/0 conn(0x7f84880c5310 0x7f84880c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 --2- 172.21.15.94:0/4143117734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880ca1a0 0x7f84880c8b30 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 -- 172.21.15.94:0/4143117734 >> 172.21.15.94:0/4143117734 conn(0x7f84880ac7e0 msgr2=0x7f84880acbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 -- 172.21.15.94:0/4143117734 shutdown_connections 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.312+0000 7f84900ed640 1 -- 172.21.15.94:0/4143117734 wait complete. 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.313+0000 7f84900ed640 1 Processor -- start 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.313+0000 7f84900ed640 1 -- start start 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f84900ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880af720 0x7f848814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f84900ed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84880c5310 0x7f84880c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f84900ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84880ca1a0 0x7f84881ead20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f84900ed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84880adfb0 con 0x7f84880af720 2024-03-06T07:23:26.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f84900ed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84880ade30 con 0x7f84880c5310 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f84900ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848814f4a0 con 0x7f84880ca1a0 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880af720 0x7f848814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880af720 0x7f848814bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49172/0 (socket says 172.21.15.94:49172) 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 -- 172.21.15.94:0/1916136886 learned_addr learned my addr 172.21.15.94:0/1916136886 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 -- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84880ca1a0 msgr2=0x7f84881ead20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84880ca1a0 0x7f84881ead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 -- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84880c5310 msgr2=0x7f84880c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84880c5310 0x7f84880c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 -- 172.21.15.94:0/1916136886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84881434c0 con 0x7f84880af720 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f848de89640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880af720 0x7f848814bdb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f847800c430 tx=0x7f847800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f847e7fc640 1 -- 172.21.15.94:0/1916136886 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478011790 con 0x7f84880af720 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f847e7fc640 1 -- 172.21.15.94:0/1916136886 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478017070 con 0x7f84880af720 2024-03-06T07:23:26.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.314+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84881437a0 con 0x7f84880af720 2024-03-06T07:23:26.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.315+0000 7f847e7fc640 1 -- 172.21.15.94:0/1916136886 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478011bb0 con 0x7f84880af720 2024-03-06T07:23:26.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.315+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8488143d60 con 0x7f84880af720 2024-03-06T07:23:26.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.316+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848804f330 con 0x7f84880af720 2024-03-06T07:23:26.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.316+0000 7f847e7fc640 1 -- 172.21.15.94:0/1916136886 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f847801c460 con 0x7f84880af720 2024-03-06T07:23:26.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.316+0000 7f847e7fc640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f846c06b4d0 0x7f846c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.317+0000 7f847e7fc640 1 -- 172.21.15.94:0/1916136886 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f847808a1e0 con 0x7f84880af720 2024-03-06T07:23:26.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.317+0000 7f848d688640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f846c06b4d0 0x7f846c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.317+0000 7f848d688640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f846c06b4d0 0x7f846c06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8484035000 tx=0x7f848402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.321+0000 7f847e7fc640 1 -- 172.21.15.94:0/1916136886 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8478058f60 con 0x7f84880af720 2024-03-06T07:23:26.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.498+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f84880c9f80 con 0x7f84880af720 2024-03-06T07:23:26.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.501+0000 7f847e7fc640 1 -- 172.21.15.94:0/1916136886 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v47)=0 v47) v1 ==== 116+0+0 (secure 0 0 0) 0x7f8478058860 con 0x7f84880af720 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f846c06b4d0 msgr2=0x7f846c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f846c06b4d0 0x7f846c06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8484035000 tx=0x7f848402e000 comp rx=0 tx=0).stop 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880af720 msgr2=0x7f848814bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880af720 0x7f848814bdb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f847800c430 tx=0x7f847800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 shutdown_connections 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f846c06b4d0 0x7f846c06d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84880ca1a0 0x7f84881ead20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84880c5310 0x7f84880c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 --2- 172.21.15.94:0/1916136886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84880af720 0x7f848814bdb0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 >> 172.21.15.94:0/1916136886 conn(0x7f84880ac7e0 msgr2=0x7f84880c93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 shutdown_connections 2024-03-06T07:23:26.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.503+0000 7f84900ed640 1 -- 172.21.15.94:0/1916136886 wait complete. 2024-03-06T07:23:26.519 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.622+0000 7faa73cc2640 1 Processor -- start 2024-03-06T07:23:26.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.622+0000 7faa73cc2640 1 -- start start 2024-03-06T07:23:26.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.622+0000 7faa73cc2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa6c14f360 0x7faa6c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.622+0000 7faa73cc2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faa6c158b60 0x7faa6c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa73cc2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c0c73d0 con 0x7faa6c0c88c0 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa73cc2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6c0c8dd0 con 0x7faa6c14f360 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa73cc2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6c0be2e0 con 0x7faa6c158b60 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa71a5e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa6c14f360 0x7faa6c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa71a5e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faa6c14f360 0x7faa6c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49174/0 (socket says 172.21.15.94:49174) 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa71a5e640 1 -- 172.21.15.94:0/2211001484 learned_addr learned my addr 172.21.15.94:0/2211001484 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa7125d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7faa6c0c88c0 0x7faa6c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54402/0 (socket says 172.21.15.94:54402) 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa7225f640 1 --2- 172.21.15.94:0/2211001484 >> v2:172.21.15.145:3300/0 conn(0x7faa6c158b60 0x7faa6c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa71a5e640 1 -- 172.21.15.94:0/2211001484 >> v2:172.21.15.145:3300/0 conn(0x7faa6c158b60 msgr2=0x7faa6c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa71a5e640 1 --2- 172.21.15.94:0/2211001484 >> v2:172.21.15.145:3300/0 conn(0x7faa6c158b60 0x7faa6c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa71a5e640 1 -- 172.21.15.94:0/2211001484 >> v1:172.21.15.94:6789/0 conn(0x7faa6c0c88c0 legacy=0x7faa6c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.623+0000 7faa71a5e640 1 -- 172.21.15.94:0/2211001484 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c0c7670 con 0x7faa6c14f360 2024-03-06T07:23:26.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa71a5e640 1 --2- 172.21.15.94:0/2211001484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c14f360 0x7faa6c155c10 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7faa6c152d60 tx=0x7faa6802b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab32ec480ba3e19c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa70a5c640 1 -- 172.21.15.94:0/2211001484 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68037070 con 0x7faa6c14f360 2024-03-06T07:23:26.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa70a5c640 1 -- 172.21.15.94:0/2211001484 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa68030710 con 0x7faa6c14f360 2024-03-06T07:23:26.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa70a5c640 1 -- 172.21.15.94:0/2211001484 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68030ae0 con 0x7faa6c14f360 2024-03-06T07:23:26.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa73cc2640 1 -- 172.21.15.94:0/2211001484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c14f360 msgr2=0x7faa6c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2211001484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c14f360 0x7faa6c155c10 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7faa6c152d60 tx=0x7faa6802b760 comp rx=0 tx=0).stop 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa73cc2640 1 -- 172.21.15.94:0/2211001484 shutdown_connections 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2211001484 >> v2:172.21.15.145:3300/0 conn(0x7faa6c158b60 0x7faa6c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2211001484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c14f360 0x7faa6c155c10 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.624+0000 7faa73cc2640 1 -- 172.21.15.94:0/2211001484 >> 172.21.15.94:0/2211001484 conn(0x7faa6c0ac570 msgr2=0x7faa6c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.625+0000 7faa73cc2640 1 -- 172.21.15.94:0/2211001484 shutdown_connections 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.625+0000 7faa73cc2640 1 -- 172.21.15.94:0/2211001484 wait complete. 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 Processor -- start 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 -- start start 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa6c0c88c0 0x7faa6c0bf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa6c14f360 0x7faa6c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c158b60 0x7faa6c1ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c0c5a50 con 0x7faa6c158b60 2024-03-06T07:23:26.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c0c7c30 con 0x7faa6c14f360 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa73cc2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c0c5bd0 con 0x7faa6c0c88c0 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa7225f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c158b60 0x7faa6c1ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa7225f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c158b60 0x7faa6c1ebdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49186/0 (socket says 172.21.15.94:49186) 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.626+0000 7faa7225f640 1 -- 172.21.15.94:0/2438389045 learned_addr learned my addr 172.21.15.94:0/2438389045 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa71a5e640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa6c0c88c0 0x7faa6c0bf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7125d640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa6c14f360 0x7faa6c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7225f640 1 -- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa6c0c88c0 msgr2=0x7faa6c0bf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7225f640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa6c0c88c0 0x7faa6c0bf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7225f640 1 -- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa6c14f360 msgr2=0x7faa6c0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7225f640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa6c14f360 0x7faa6c0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7225f640 1 -- 172.21.15.94:0/2438389045 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c0bf0a0 con 0x7faa6c158b60 2024-03-06T07:23:26.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa71a5e640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa6c0c88c0 0x7faa6c0bf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7125d640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa6c14f360 0x7faa6c0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa7225f640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c158b60 0x7faa6c1ebdb0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7faa64007e40 tx=0x7faa6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa627fc640 1 -- 172.21.15.94:0/2438389045 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64014620 con 0x7faa6c158b60 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa627fc640 1 -- 172.21.15.94:0/2438389045 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64019070 con 0x7faa6c158b60 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c1eeba0 con 0x7faa6c158b60 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa627fc640 1 -- 172.21.15.94:0/2438389045 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64014a20 con 0x7faa6c158b60 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.627+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c152980 con 0x7faa6c158b60 2024-03-06T07:23:26.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.628+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c04f330 con 0x7faa6c158b60 2024-03-06T07:23:26.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.629+0000 7faa627fc640 1 -- 172.21.15.94:0/2438389045 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa64014bc0 con 0x7faa6c158b60 2024-03-06T07:23:26.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.629+0000 7faa627fc640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5006b4d0 0x7faa5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.629+0000 7faa627fc640 1 -- 172.21.15.94:0/2438389045 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa6408ab80 con 0x7faa6c158b60 2024-03-06T07:23:26.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.630+0000 7faa71a5e640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5006b4d0 0x7faa5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.630+0000 7faa71a5e640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5006b4d0 0x7faa5006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faa68034000 tx=0x7faa68008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.634+0000 7faa627fc640 1 -- 172.21.15.94:0/2438389045 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa64059900 con 0x7faa6c158b60 2024-03-06T07:23:26.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.848+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa6c06cda0 con 0x7faa6c158b60 2024-03-06T07:23:26.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.849+0000 7faa627fc640 1 -- 172.21.15.94:0/2438389045 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+4990 (secure 0 0 0) 0x7faa64059200 con 0x7faa6c158b60 2024-03-06T07:23:26.851 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:26.851 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":119,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1362727065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1362727065},{"type":"v1","addr":"172.21.15.145:6836","nonce":1362727065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:22.175096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8301},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/392019985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":392019985},{"type":"v1","addr":"172.21.15.94:6835","nonce":392019985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8301,"qdb_cluster":[8301]},"id":10}]} 2024-03-06T07:23:26.851 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 119 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5006b4d0 msgr2=0x7faa5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5006b4d0 0x7faa5006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7faa68034000 tx=0x7faa68008030 comp rx=0 tx=0).stop 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c158b60 msgr2=0x7faa6c1ebdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c158b60 0x7faa6c1ebdb0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7faa64007e40 tx=0x7faa6400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 shutdown_connections 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faa5006b4d0 0x7faa5006d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faa6c158b60 0x7faa6c1ebdb0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa6c14f360 0x7faa6c0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 --2- 172.21.15.94:0/2438389045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa6c0c88c0 0x7faa6c0bf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 >> 172.21.15.94:0/2438389045 conn(0x7faa6c0ac570 msgr2=0x7faa6c14dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 shutdown_connections 2024-03-06T07:23:26.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:26.851+0000 7faa73cc2640 1 -- 172.21.15.94:0/2438389045 wait complete. 2024-03-06T07:23:27.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 119, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:20.150839+0000', 'modified': '2024-03-06T07:23:22.175096+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8301}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8301': {'gid': 8301, 'name': 'c', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.94:6835/392019985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 392019985}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 392019985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8301, 'qdb_cluster': [8301]} 2024-03-06T07:23:27.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:27.266 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:23:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.367+0000 7fe4805e9640 1 Processor -- start 2024-03-06T07:23:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.367+0000 7fe4805e9640 1 -- start start 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe4805e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4780c8950 0x7fe4780c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe4805e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe4781558d0 0x7fe478155d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe4805e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe478155130 con 0x7fe4780c69f0 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe4805e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe478155440 con 0x7fe4781558d0 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe4805e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780ad480 con 0x7fe4780c8950 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47eb86640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe4781558d0 0x7fe478155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47e385640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe4780c69f0 0x7fe4780acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54406/0 (socket says 172.21.15.94:54406) 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47e385640 1 -- 172.21.15.94:0/4137480813 learned_addr learned my addr 172.21.15.94:0/4137480813 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47db84640 1 --2- 172.21.15.94:0/4137480813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4780c8950 0x7fe4780c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47eb86640 1 -- 172.21.15.94:0/4137480813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4780c8950 msgr2=0x7fe4780c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47eb86640 1 --2- 172.21.15.94:0/4137480813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4780c8950 0x7fe4780c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47eb86640 1 -- 172.21.15.94:0/4137480813 >> v1:172.21.15.94:6789/0 conn(0x7fe4780c69f0 legacy=0x7fe4780acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47eb86640 1 -- 172.21.15.94:0/4137480813 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780c5d50 con 0x7fe4781558d0 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.368+0000 7fe47d383640 1 -- 172.21.15.94:0/4137480813 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386491316 0 0) 0x7fe474001110 con 0x7fe4780c69f0 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.369+0000 7fe47eb86640 1 --2- 172.21.15.94:0/4137480813 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4781558d0 0x7fe478155d60 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe470009d90 tx=0x7fe47002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6739bef464d18a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.369+0000 7fe47d383640 1 -- 172.21.15.94:0/4137480813 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470039070 con 0x7fe4781558d0 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.369+0000 7fe47d383640 1 -- 172.21.15.94:0/4137480813 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47002cbc0 con 0x7fe4781558d0 2024-03-06T07:23:27.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.369+0000 7fe47d383640 1 -- 172.21.15.94:0/4137480813 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470034440 con 0x7fe4781558d0 2024-03-06T07:23:27.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 -- 172.21.15.94:0/4137480813 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4781558d0 msgr2=0x7fe478155d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 --2- 172.21.15.94:0/4137480813 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4781558d0 0x7fe478155d60 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe470009d90 tx=0x7fe47002be10 comp rx=0 tx=0).stop 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 -- 172.21.15.94:0/4137480813 shutdown_connections 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 --2- 172.21.15.94:0/4137480813 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4781558d0 0x7fe478155d60 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 --2- 172.21.15.94:0/4137480813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4780c8950 0x7fe4780c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 -- 172.21.15.94:0/4137480813 >> 172.21.15.94:0/4137480813 conn(0x7fe4780cae70 msgr2=0x7fe4780cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 -- 172.21.15.94:0/4137480813 shutdown_connections 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.370+0000 7fe4805e9640 1 -- 172.21.15.94:0/4137480813 wait complete. 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.371+0000 7fe4805e9640 1 Processor -- start 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.371+0000 7fe4805e9640 1 -- start start 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe4805e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4780c69f0 0x7fe478156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe4805e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4780c8950 0x7fe4781eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe4805e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4781558d0 0x7fe47806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe4805e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe478152a20 con 0x7fe4780c69f0 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe4805e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780c6310 con 0x7fe4780c8950 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe4805e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe478152ba0 con 0x7fe4781558d0 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4780c69f0 0x7fe478156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4780c69f0 0x7fe478156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49192/0 (socket says 172.21.15.94:49192) 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 -- 172.21.15.94:0/438427069 learned_addr learned my addr 172.21.15.94:0/438427069 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47eb86640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4781558d0 0x7fe47806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47db84640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4780c8950 0x7fe4781eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 -- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4781558d0 msgr2=0x7fe47806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4781558d0 0x7fe47806c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 -- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4780c8950 msgr2=0x7fe4781eb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4780c8950 0x7fe4781eb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 -- 172.21.15.94:0/438427069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe478156a60 con 0x7fe4780c69f0 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe47e385640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4780c69f0 0x7fe478156520 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c430 tx=0x7fe47400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe46effd640 1 -- 172.21.15.94:0/438427069 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740074e0 con 0x7fe4780c69f0 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe46effd640 1 -- 172.21.15.94:0/438427069 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4740047e0 con 0x7fe4780c69f0 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.372+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4781ebb70 con 0x7fe4780c69f0 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.373+0000 7fe46effd640 1 -- 172.21.15.94:0/438427069 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474004bd0 con 0x7fe4780c69f0 2024-03-06T07:23:27.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.373+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4780c46d0 con 0x7fe4780c69f0 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.374+0000 7fe46effd640 1 -- 172.21.15.94:0/438427069 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe474011070 con 0x7fe4780c69f0 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.374+0000 7fe46effd640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe45c06b4d0 0x7fe45c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.374+0000 7fe46effd640 1 -- 172.21.15.94:0/438427069 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe47408a000 con 0x7fe4780c69f0 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.374+0000 7fe47db84640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe45c06b4d0 0x7fe45c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.375+0000 7fe47db84640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe45c06b4d0 0x7fe45c06d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe468006fd0 tx=0x7fe468008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.375+0000 7fe4831d0740 1 Processor -- start 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe4831d0740 1 -- start start 2024-03-06T07:23:27.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881df3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881df3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881df44b0 0x562881df0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe4831d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x562881d5a6b0 con 0x562881df6330 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x562881d03a60 con 0x562881df5f50 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x562881d56750 con 0x562881df44b0 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe47db84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881df3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe47db84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881df3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49200/0 (socket says 172.21.15.94:49200) 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe47db84640 1 -- 172.21.15.94:0/4004967199 learned_addr learned my addr 172.21.15.94:0/4004967199 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe47eb86640 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881df44b0 0x562881df0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.376+0000 7fe47e385640 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881df3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe47db84640 1 -- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881df44b0 msgr2=0x562881df0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe47db84640 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881df44b0 0x562881df0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe47db84640 1 -- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 msgr2=0x562881df3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe47db84640 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881df3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe47db84640 1 -- 172.21.15.94:0/4004967199 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562881d21f30 con 0x562881df6330 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe47db84640 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881df3f70 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe46800b590 tx=0x7fe468012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2c361dae7c8271 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe45bfff640 1 -- 172.21.15.94:0/4004967199 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46801b050 con 0x562881df6330 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe45bfff640 1 -- 172.21.15.94:0/4004967199 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468025070 con 0x562881df6330 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.377+0000 7fe45bfff640 1 -- 172.21.15.94:0/4004967199 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468014d80 con 0x562881df6330 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 -- 172.21.15.94:0/4004967199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 msgr2=0x562881df3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881df3f70 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fe46800b590 tx=0x7fe468012000 comp rx=0 tx=0).stop 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 -- 172.21.15.94:0/4004967199 shutdown_connections 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881df44b0 0x562881df0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881df3f70 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 --2- 172.21.15.94:0/4004967199 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881df3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 -- 172.21.15.94:0/4004967199 >> 172.21.15.94:0/4004967199 conn(0x562881df27d0 msgr2=0x562881e173c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 -- 172.21.15.94:0/4004967199 shutdown_connections 2024-03-06T07:23:27.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 -- 172.21.15.94:0/4004967199 wait complete. 2024-03-06T07:23:27.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 -- start start 2024-03-06T07:23:27.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.378+0000 7fe4831d0740 1 Processor -- start 2024-03-06T07:23:27.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881e167b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e45b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e12da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe4831d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x562881e18230 con 0x562881df6330 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x562881d16ea0 con 0x562881df5f50 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x562881e17c70 con 0x562881e1b360 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe4831d0740 10 client.0 _ll_register_callbacks cb 0x562881d16870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe47db84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e45b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe47db84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e45b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49202/0 (socket says 172.21.15.94:49202) 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.379+0000 7fe47db84640 1 -- 172.21.15.94:0/3251545350 learned_addr learned my addr 172.21.15.94:0/3251545350 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47e385640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881e167b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47eb86640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e12da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47db84640 1 -- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 msgr2=0x562881e12da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47db84640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e12da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47db84640 1 -- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 msgr2=0x562881e167b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47db84640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881e167b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47db84640 1 -- 172.21.15.94:0/3251545350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562881d17270 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47e385640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881e167b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe47db84640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e45b50 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe468021000 tx=0x7fe468004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd64789ca8c70a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4597fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4597fa640 1 -- 172.21.15.94:0/3251545350 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468020910 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4597fa640 1 -- 172.21.15.94:0/3251545350 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe468020ab0 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe468025070 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4597fa640 1 -- 172.21.15.94:0/3251545350 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe46801b020 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4831d0740 10 client.8490 fetch_fsmap learned FSMap version 119 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x562881a6f2d0 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.380+0000 7fe4597fa640 1 -- 172.21.15.94:0/3251545350 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468025070 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4597fa640 1 -- 172.21.15.94:0/3251545350 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 119) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe468020cb0 con 0x562881df6330 2024-03-06T07:23:27.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4831d0740 10 client.8490 fetch_fsmap finished waiting for FSMap version 119 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4831d0740 10 client.8490 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4831d0740 20 client.8490 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e18f30 0x562881e51040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4831d0740 4 client.8490 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 --> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562881d67bf0 con 0x562881e18f30 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe47e385640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e18f30 0x562881e51040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe4597fa640 10 client.8490 ms_handle_connect on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.381+0000 7fe47e385640 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e18f30 0x562881e51040 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e86ca2d0c6bb8250 server_cookie=b93a73dcf4326012 in_seq=0 out_seq=0 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4597fa640 1 -- 172.21.15.94:0/3251545350 <== mds.0 v2:172.21.15.94:6834/392019985 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562881d67bf0 con 0x562881e18f30 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4597fa640 10 client.8490 handle_command_reply: tid=0 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 client.8490 shutdown 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 msgr2=0x562881e45b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e45b50 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fe468021000 tx=0x7fe468004730 comp rx=0 tx=0).stop 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 shutdown_connections 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e18f30 0x562881e51040 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e12da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e45b50 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4597fa640 0 client.8490 ms_handle_reset on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3251545350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881e167b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 >> 172.21.15.94:0/3251545350 conn(0x562881df44b0 msgr2=0x562881df48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 shutdown_connections 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 1 -- 172.21.15.94:0/3251545350 wait complete. 2024-03-06T07:23:27.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.382+0000 7fe4831d0740 20 client.8490 trim_cache size 0 max 16384 2024-03-06T07:23:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.398+0000 7fe4831d0740 1 Processor -- start 2024-03-06T07:23:27.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.398+0000 7fe4831d0740 1 -- start start 2024-03-06T07:23:27.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df5f50 0x562881df3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df6330 0x562881def510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e35580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe4831d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781f6db0 con 0x562881df5f50 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781eecb0 con 0x562881df6330 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781f67e0 con 0x562881e1b360 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df5f50 0x562881df3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df5f50 0x562881df3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49218/0 (socket says 172.21.15.94:49218) 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 -- 172.21.15.94:0/3235943343 learned_addr learned my addr 172.21.15.94:0/3235943343 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47eb86640 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e35580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47db84640 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df6330 0x562881def510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 -- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 msgr2=0x562881e35580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e35580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 -- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df6330 msgr2=0x562881def510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df6330 0x562881def510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.399+0000 7fe47e385640 1 -- 172.21.15.94:0/3235943343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562881b396b0 con 0x562881df5f50 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe47e385640 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df5f50 0x562881df3d80 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fe474058580 tx=0x7fe47405b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c371a924a86beb04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe45b7fe640 1 -- 172.21.15.94:0/3235943343 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474056690 con 0x562881df5f50 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe45b7fe640 1 -- 172.21.15.94:0/3235943343 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4740568f0 con 0x562881df5f50 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe45b7fe640 1 -- 172.21.15.94:0/3235943343 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47401ad50 con 0x562881df5f50 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 -- 172.21.15.94:0/3235943343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df5f50 msgr2=0x562881df3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df5f50 0x562881df3d80 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fe474058580 tx=0x7fe47405b040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 -- 172.21.15.94:0/3235943343 shutdown_connections 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e35580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df6330 0x562881def510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3235943343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df5f50 0x562881df3d80 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 -- 172.21.15.94:0/3235943343 >> 172.21.15.94:0/3235943343 conn(0x562881df44b0 msgr2=0x562881e42460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 -- 172.21.15.94:0/3235943343 shutdown_connections 2024-03-06T07:23:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.400+0000 7fe4831d0740 1 -- 172.21.15.94:0/3235943343 wait complete. 2024-03-06T07:23:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.401+0000 7fe4831d0740 1 -- start start 2024-03-06T07:23:27.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.401+0000 7fe4831d0740 1 Processor -- start 2024-03-06T07:23:27.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881def910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e489c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4831d0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e3e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4831d0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x562881e16fd0 con 0x562881df6330 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x562881e1c5c0 con 0x562881df5f50 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4831d0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x562881e13800 con 0x562881e1b360 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4831d0740 10 client.0 _ll_register_callbacks cb 0x7fe46800f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49226/0 (socket says 172.21.15.94:49226) 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 -- 172.21.15.94:0/3728750384 learned_addr learned my addr 172.21.15.94:0/3728750384 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47eb86640 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e3e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47e385640 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881def910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 -- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 msgr2=0x562881e3e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e3e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 -- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 msgr2=0x562881def910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881def910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 -- 172.21.15.94:0/3728750384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562881e4c690 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe47db84640 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e489c0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fe468008010 tx=0x7fe468015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16832c282183944e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.402+0000 7fe4597fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4597fa640 1 -- 172.21.15.94:0/3728750384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468020b20 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4597fa640 1 -- 172.21.15.94:0/3728750384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe468020cc0 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4740568f0 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4597fa640 1 -- 172.21.15.94:0/3728750384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=119) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe46804e050 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 10 client.8496 fetch_fsmap learned FSMap version 119 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=119,monmap=2+}) v3 -- 0x562881e49420 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4597fa640 1 -- 172.21.15.94:0/3728750384 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4740568f0 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4597fa640 1 -- 172.21.15.94:0/3728750384 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 119) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe468051020 con 0x562881df6330 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 10 client.8496 fetch_fsmap finished waiting for FSMap version 119 2024-03-06T07:23:27.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 10 client.8496 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 20 client.8496 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e9e6c0 0x562881e9eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 4 client.8496 mds_command: new command op to 8301 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 --> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562881d61f40 con 0x562881e9e6c0 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.403+0000 7fe47e385640 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e9e6c0 0x562881e9eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4597fa640 10 client.8496 ms_handle_connect on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe47e385640 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e9e6c0 0x562881e9eaa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5958797719bc63a4 server_cookie=8a340850934df213 in_seq=0 out_seq=0 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4597fa640 1 -- 172.21.15.94:0/3728750384 <== mds.0 v2:172.21.15.94:6834/392019985 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x562881d61f40 con 0x562881e9e6c0 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4597fa640 10 client.8496 handle_command_reply: tid=0 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 client.8496 shutdown 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 msgr2=0x562881e489c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e489c0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fe468008010 tx=0x7fe468015000 comp rx=0 tx=0).stop 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 shutdown_connections 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:6834/392019985,v1:172.21.15.94:6835/392019985] conn(0x562881e9e6c0 0x562881e9eaa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x562881e1b360 0x562881e3e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x562881df6330 0x562881e489c0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4597fa640 0 client.8496 ms_handle_reset on v2:172.21.15.94:6834/392019985 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 --2- 172.21.15.94:0/3728750384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x562881df5f50 0x562881def910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 >> 172.21.15.94:0/3728750384 conn(0x562881df44b0 msgr2=0x562881e42370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 shutdown_connections 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 1 -- 172.21.15.94:0/3728750384 wait complete. 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.404+0000 7fe4831d0740 20 client.8496 trim_cache size 0 max 16384 2024-03-06T07:23:27.412 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 8301, 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 6.5805544500000002, 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 119, 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 135, 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 135, 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 11.696381935 2024-03-06T07:23:27.413 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:27.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe45c06b4d0 msgr2=0x7fe45c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe45c06b4d0 0x7fe45c06d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe468006fd0 tx=0x7fe468008040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4780c69f0 msgr2=0x7fe478156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4780c69f0 0x7fe478156520 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c430 tx=0x7fe47400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 shutdown_connections 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe45c06b4d0 0x7fe45c06d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4781558d0 0x7fe47806c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4780c8950 0x7fe4781eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 --2- 172.21.15.94:0/438427069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4780c69f0 0x7fe478156520 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 >> 172.21.15.94:0/438427069 conn(0x7fe4780cae70 msgr2=0x7fe478150a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 shutdown_connections 2024-03-06T07:23:27.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:27.405+0000 7fe4805e9640 1 -- 172.21.15.94:0/438427069 wait complete. 2024-03-06T07:23:27.417 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 119, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:20.150839+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 119, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8301": { "addr": "172.21.15.94:6835/392019985", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6834", "nonce": 392019985, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 392019985, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8301, "incarnation": 117, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:23:22.175096+0000", "qdb_cluster": [ 8301 ], "qdb_leader": 8301, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8301 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/1362727065", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1362727065, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1362727065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7723, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/3826543722", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 3826543722, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 3826543722, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7726, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6833/3405703874", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3405703874, "type": "v2" }, { "addr": "172.21.15.94:6833", "nonce": 3405703874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8298, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:27.418 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-03-06T07:23:27.418 INFO:tasks.cephfs.test_failover:current = [{'gid': 8298, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6833/3405703874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3405703874}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 3405703874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7726, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/3826543722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3826543722}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3826543722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 8301, 'name': 'c', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.94:6835/392019985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 392019985}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 392019985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7723, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/1362727065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1362727065}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1362727065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}] 2024-03-06T07:23:27.419 DEBUG:tasks.cephfs.test_failover:10 != -1 2024-03-06T07:23:27.419 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:23:28.509 INFO:tasks.ceph.mds.c.smithi094.stderr: -104> 2024-03-06T07:23:26.084+0000 7f6bcf43b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:28.534 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:23:32.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.529+0000 7f7aba5f6640 1 Processor -- start 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7aba5f6640 1 -- start start 2024-03-06T07:23:32.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7aba5f6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ab41506b0 0x7f7ab4153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7aba5f6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7ab40acc00 0x7f7ab40c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7aba5f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab4070aa0 con 0x7f7ab4156ea0 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7aba5f6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab4156450 con 0x7f7ab41506b0 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7aba5f6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab4156a10 con 0x7f7ab40acc00 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7ab3fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ab41506b0 0x7f7ab4153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7ab37fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7ab4156ea0 0x7f7ab40702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36398/0 (socket says 172.21.15.94:36398) 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.530+0000 7f7ab37fe640 1 -- 172.21.15.94:0/2601524213 learned_addr learned my addr 172.21.15.94:0/2601524213 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.531+0000 7f7ab8b93640 1 --2- 172.21.15.94:0/2601524213 >> v2:172.21.15.145:3300/0 conn(0x7f7ab40acc00 0x7f7ab40c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.531+0000 7f7ab3fff640 1 -- 172.21.15.94:0/2601524213 >> v2:172.21.15.145:3300/0 conn(0x7f7ab40acc00 msgr2=0x7f7ab40c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.531+0000 7f7ab3fff640 1 --2- 172.21.15.94:0/2601524213 >> v2:172.21.15.145:3300/0 conn(0x7f7ab40acc00 0x7f7ab40c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.531+0000 7f7ab3fff640 1 -- 172.21.15.94:0/2601524213 >> v1:172.21.15.94:6789/0 conn(0x7f7ab4156ea0 legacy=0x7f7ab40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.531+0000 7f7ab3fff640 1 -- 172.21.15.94:0/2601524213 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4070d80 con 0x7f7ab41506b0 2024-03-06T07:23:32.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.531+0000 7f7ab2ffd640 1 -- 172.21.15.94:0/2601524213 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304143041 0 0) 0x7f7ab4070aa0 con 0x7f7ab4156ea0 2024-03-06T07:23:32.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7ab3fff640 1 --2- 172.21.15.94:0/2601524213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab41506b0 0x7f7ab4153990 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4073610 tx=0x7f7aa402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b73889ace4e94ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7ab2ffd640 1 -- 172.21.15.94:0/2601524213 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4035020 con 0x7f7ab41506b0 2024-03-06T07:23:32.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7ab2ffd640 1 -- 172.21.15.94:0/2601524213 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa4031830 con 0x7f7ab41506b0 2024-03-06T07:23:32.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7ab2ffd640 1 -- 172.21.15.94:0/2601524213 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4031c00 con 0x7f7ab41506b0 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 -- 172.21.15.94:0/2601524213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab41506b0 msgr2=0x7f7ab4153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/2601524213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab41506b0 0x7f7ab4153990 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4073610 tx=0x7f7aa402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 -- 172.21.15.94:0/2601524213 shutdown_connections 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/2601524213 >> v2:172.21.15.145:3300/0 conn(0x7f7ab40acc00 0x7f7ab40c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/2601524213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab41506b0 0x7f7ab4153990 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 -- 172.21.15.94:0/2601524213 >> 172.21.15.94:0/2601524213 conn(0x7f7ab406b1a0 msgr2=0x7f7ab406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 -- 172.21.15.94:0/2601524213 shutdown_connections 2024-03-06T07:23:32.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.532+0000 7f7aba5f6640 1 -- 172.21.15.94:0/2601524213 wait complete. 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.533+0000 7f7aba5f6640 1 Processor -- start 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7aba5f6640 1 -- start start 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7aba5f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab40acc00 0x7f7ab4152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7aba5f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ab41506b0 0x7f7ab40c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7aba5f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ab4156ea0 0x7f7ab414c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7aba5f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab40ad1d0 con 0x7f7ab40acc00 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7aba5f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab40ad050 con 0x7f7ab4156ea0 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7aba5f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab40ad350 con 0x7f7ab41506b0 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7ab3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab40acc00 0x7f7ab4152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7ab3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab40acc00 0x7f7ab4152970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37776/0 (socket says 172.21.15.94:37776) 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7ab3fff640 1 -- 172.21.15.94:0/3496990976 learned_addr learned my addr 172.21.15.94:0/3496990976 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7ab8b93640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ab4156ea0 0x7f7ab414c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.534+0000 7f7ab37fe640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ab41506b0 0x7f7ab40c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab3fff640 1 -- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ab41506b0 msgr2=0x7f7ab40c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab3fff640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ab41506b0 0x7f7ab40c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab3fff640 1 -- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ab4156ea0 msgr2=0x7f7ab414c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab3fff640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ab4156ea0 0x7f7ab414c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab3fff640 1 -- 172.21.15.94:0/3496990976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4147080 con 0x7f7ab40acc00 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab3fff640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab40acc00 0x7f7ab4152970 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009890 tx=0x7f7aa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab0ff9640 1 -- 172.21.15.94:0/3496990976 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4041070 con 0x7f7ab40acc00 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab0ff9640 1 -- 172.21.15.94:0/3496990976 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa403b880 con 0x7f7ab40acc00 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab414afa0 con 0x7f7ab40acc00 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7ab0ff9640 1 -- 172.21.15.94:0/3496990976 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4035040 con 0x7f7ab40acc00 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.535+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab4147340 con 0x7f7ab40acc00 2024-03-06T07:23:32.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.536+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab404f330 con 0x7f7ab40acc00 2024-03-06T07:23:32.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.537+0000 7f7ab0ff9640 1 -- 172.21.15.94:0/3496990976 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7aa4038020 con 0x7f7ab40acc00 2024-03-06T07:23:32.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.537+0000 7f7ab0ff9640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7aa006b4d0 0x7f7aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.537+0000 7f7ab0ff9640 1 -- 172.21.15.94:0/3496990976 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7aa40abc50 con 0x7f7ab40acc00 2024-03-06T07:23:32.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.537+0000 7f7ab37fe640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7aa006b4d0 0x7f7aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.538+0000 7f7ab37fe640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7aa006b4d0 0x7f7aa006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7ab41503e0 tx=0x7f7aa8007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.542+0000 7f7ab0ff9640 1 -- 172.21.15.94:0/3496990976 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7aa40b1020 con 0x7f7ab40acc00 2024-03-06T07:23:32.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.752+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ab40700a0 con 0x7f7ab40acc00 2024-03-06T07:23:32.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.752+0000 7f7ab0ff9640 1 -- 172.21.15.94:0/3496990976 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4992 (secure 0 0 0) 0x7f7aa4031800 con 0x7f7ab40acc00 2024-03-06T07:23:32.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:32.754 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":124,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:31.523761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7723},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7723":{"gid":7723,"name":"d","rank":0,"incarnation":121,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6836/1362727065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1362727065},{"type":"v1","addr":"172.21.15.145:6836","nonce":1362727065}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7723,"qdb_cluster":[7723]},"id":10}]} 2024-03-06T07:23:32.755 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 124 2024-03-06T07:23:32.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7aa006b4d0 msgr2=0x7f7aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7aa006b4d0 0x7f7aa006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7ab41503e0 tx=0x7f7aa8007de0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab40acc00 msgr2=0x7f7ab4152970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab40acc00 0x7f7ab4152970 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009890 tx=0x7f7aa40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 shutdown_connections 2024-03-06T07:23:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7aa006b4d0 0x7f7aa006d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ab4156ea0 0x7f7ab414c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ab41506b0 0x7f7ab40c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 --2- 172.21.15.94:0/3496990976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ab40acc00 0x7f7ab4152970 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 >> 172.21.15.94:0/3496990976 conn(0x7f7ab406b1a0 msgr2=0x7f7ab40c35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 shutdown_connections 2024-03-06T07:23:32.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:32.754+0000 7f7aba5f6640 1 -- 172.21.15.94:0/3496990976 wait complete. 2024-03-06T07:23:33.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:20.150839+0000', 'modified': '2024-03-06T07:23:31.523761+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7723}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7723': {'gid': 7723, 'name': 'd', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.145:6836/1362727065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1362727065}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1362727065}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7723, 'qdb_cluster': [7723]} 2024-03-06T07:23:33.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:33.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:23:33.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.272+0000 7f4c62a0e640 1 Processor -- start 2024-03-06T07:23:33.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.272+0000 7f4c62a0e640 1 -- start start 2024-03-06T07:23:33.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c62a0e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4c5c152b40 0x7f4c5c0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c62a0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5c0733f0 con 0x7f4c5c0724d0 2024-03-06T07:23:33.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c62a0e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5c0c82c0 con 0x7f4c5c0a7ff0 2024-03-06T07:23:33.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c62a0e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5c072d70 con 0x7f4c5c152b40 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4c5c0724d0 0x7f4c5c1551f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36414/0 (socket says 172.21.15.94:36414) 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/2620935048 learned_addr learned my addr 172.21.15.94:0/2620935048 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/2620935048 >> v2:172.21.15.145:3300/0 conn(0x7f4c5c152b40 0x7f4c5c0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5affd640 1 -- 172.21.15.94:0/2620935048 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894469915 0 0) 0x7f4c5c0733f0 con 0x7f4c5c0724d0 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5affd640 1 -- 172.21.15.94:0/2620935048 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c44003520 con 0x7f4c5c0724d0 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5bfff640 1 -- 172.21.15.94:0/2620935048 >> v1:172.21.15.145:6789/0 conn(0x7f4c5c0a7ff0 legacy=0x7f4c5c15fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5bfff640 1 -- 172.21.15.94:0/2620935048 >> v1:172.21.15.94:6789/0 conn(0x7f4c5c0724d0 legacy=0x7f4c5c1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.273+0000 7f4c5bfff640 1 -- 172.21.15.94:0/2620935048 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c06fd30 con 0x7f4c5c152b40 2024-03-06T07:23:33.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.274+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/2620935048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c152b40 0x7f4c5c0c8810 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0097a0 tx=0x7f4c4c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6613e167ae18cf5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.274+0000 7f4c5affd640 1 -- 172.21.15.94:0/2620935048 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c035020 con 0x7f4c5c152b40 2024-03-06T07:23:33.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.274+0000 7f4c5affd640 1 -- 172.21.15.94:0/2620935048 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c031830 con 0x7f4c5c152b40 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.274+0000 7f4c5affd640 1 -- 172.21.15.94:0/2620935048 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c031be0 con 0x7f4c5c152b40 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.275+0000 7f4c62a0e640 1 -- 172.21.15.94:0/2620935048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c152b40 msgr2=0x7f4c5c0c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.275+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/2620935048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c152b40 0x7f4c5c0c8810 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0097a0 tx=0x7f4c4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.275+0000 7f4c62a0e640 1 -- 172.21.15.94:0/2620935048 shutdown_connections 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.275+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/2620935048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c152b40 0x7f4c5c0c8810 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.275+0000 7f4c62a0e640 1 -- 172.21.15.94:0/2620935048 >> 172.21.15.94:0/2620935048 conn(0x7f4c5c06b970 msgr2=0x7f4c5c0aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.275+0000 7f4c62a0e640 1 -- 172.21.15.94:0/2620935048 shutdown_connections 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.275+0000 7f4c62a0e640 1 -- 172.21.15.94:0/2620935048 wait complete. 2024-03-06T07:23:33.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.276+0000 7f4c62a0e640 1 Processor -- start 2024-03-06T07:23:33.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.276+0000 7f4c62a0e640 1 -- start start 2024-03-06T07:23:33.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c62a0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c5c0724d0 0x7f4c5c14c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c62a0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c0a7ff0 0x7f4c5c0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c62a0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c5c0ad9e0 0x7f4c5c1eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c62a0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0c8d50 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c62a0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0704d0 con 0x7f4c5c0a7ff0 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c62a0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0c8ed0 con 0x7f4c5c0724d0 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c5c0ad9e0 0x7f4c5c1eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c5c0ad9e0 0x7f4c5c1eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37778/0 (socket says 172.21.15.94:37778) 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 -- 172.21.15.94:0/278053855 learned_addr learned my addr 172.21.15.94:0/278053855 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c0a7ff0 0x7f4c5c0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c5c0724d0 0x7f4c5c14c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 -- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c5c0724d0 msgr2=0x7f4c5c14c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c5c0724d0 0x7f4c5c14c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 -- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c0a7ff0 msgr2=0x7f4c5c0a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c0a7ff0 0x7f4c5c0a9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 -- 172.21.15.94:0/278053855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0aa800 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.277+0000 7f4c60fab640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c5c0ad9e0 0x7f4c5c1eb080 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400c6b0 tx=0x7f4c5400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.278+0000 7f4c58ff9640 1 -- 172.21.15.94:0/278053855 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54003b70 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.278+0000 7f4c58ff9640 1 -- 172.21.15.94:0/278053855 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54003d30 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.278+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c158a70 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.278+0000 7f4c58ff9640 1 -- 172.21.15.94:0/278053855 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54005310 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.278+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5c159030 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.279+0000 7f4c58ff9640 1 -- 172.21.15.94:0/278053855 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c540054b0 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.279+0000 7f4c58ff9640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c4806b4d0 0x7f4c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.280+0000 7f4c58ff9640 1 -- 172.21.15.94:0/278053855 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4c5408a070 con 0x7f4c5c0ad9e0 2024-03-06T07:23:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.280+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c4806b4d0 0x7f4c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.280+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c4806b4d0 0x7f4c4806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c036000 tx=0x7f4c4c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.280+0000 7f4c655f7740 1 Processor -- start 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.280+0000 7f4c655f7740 1 -- start start 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69ef50 0x55c4da69ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69f330 0x55c4da69cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69d4b0 0x55c4da699680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c655f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4da6036b0 con 0x55c4da69f330 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4da5aca60 con 0x55c4da69d4b0 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4da5ff750 con 0x55c4da69ef50 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69f330 0x55c4da69cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69f330 0x55c4da69cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37784/0 (socket says 172.21.15.94:37784) 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/2810323745 learned_addr learned my addr 172.21.15.94:0/2810323745 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69ef50 0x55c4da69ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c60fab640 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69d4b0 0x55c4da699680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69ef50 msgr2=0x55c4da69ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69ef50 0x55c4da69ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69d4b0 msgr2=0x55c4da699680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69d4b0 0x55c4da699680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c5b7fe640 1 -- 172.21.15.94:0/2810323745 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4da5caf30 con 0x55c4da69f330 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.281+0000 7f4c60fab640 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69d4b0 0x55c4da699680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69f330 0x55c4da69cf70 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f4c50000f60 tx=0x7f4c50007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8712dd7532c938ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c41ffb640 1 -- 172.21.15.94:0/2810323745 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c500016c0 con 0x55c4da69f330 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c41ffb640 1 -- 172.21.15.94:0/2810323745 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c50018070 con 0x55c4da69f330 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c41ffb640 1 -- 172.21.15.94:0/2810323745 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50001a60 con 0x55c4da69f330 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 -- 172.21.15.94:0/2810323745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69f330 msgr2=0x55c4da69cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69f330 0x55c4da69cf70 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f4c50000f60 tx=0x7f4c50007410 comp rx=0 tx=0).stop 2024-03-06T07:23:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 -- 172.21.15.94:0/2810323745 shutdown_connections 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69d4b0 0x55c4da699680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69f330 0x55c4da69cf70 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2810323745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69ef50 0x55c4da69ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 -- 172.21.15.94:0/2810323745 >> 172.21.15.94:0/2810323745 conn(0x55c4da69b7d0 msgr2=0x55c4da6c03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 -- 172.21.15.94:0/2810323745 shutdown_connections 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.282+0000 7f4c655f7740 1 -- 172.21.15.94:0/2810323745 wait complete. 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.283+0000 7f4c655f7740 1 -- start start 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.283+0000 7f4c655f7740 1 Processor -- start 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6c1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69f330 0x55c4da6bd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da6c4a30 0x55c4da6bbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4da6c1230 con 0x55c4da69ef50 2024-03-06T07:23:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4da5bfea0 con 0x55c4da6c4a30 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4da6c0c70 con 0x55c4da69f330 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 10 client.0 _ll_register_callbacks cb 0x55c4da5bf870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6c1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37796/0 (socket says 172.21.15.94:37796) 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 -- 172.21.15.94:0/2699438875 learned_addr learned my addr 172.21.15.94:0/2699438875 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69f330 0x55c4da6bd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c60fab640 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da6c4a30 0x55c4da6bbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 -- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69f330 msgr2=0x55c4da6bd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69f330 0x55c4da6bd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 -- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da6c4a30 msgr2=0x55c4da6bbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da6c4a30 0x55c4da6bbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 -- 172.21.15.94:0/2699438875 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4da5c0270 con 0x55c4da69ef50 2024-03-06T07:23:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6c1600 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009890 tx=0x7f4c4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a21cbcc192726e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c3affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c3affd640 1 -- 172.21.15.94:0/2699438875 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c035050 con 0x55c4da69ef50 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c3affd640 1 -- 172.21.15.94:0/2699438875 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4c4c003d10 con 0x55c4da69ef50 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.284+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c50018070 con 0x55c4da69ef50 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c3affd640 1 -- 172.21.15.94:0/2699438875 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c4c03b430 con 0x55c4da69ef50 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 10 client.8520 fetch_fsmap learned FSMap version 124 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55c4da3182d0 con 0x55c4da69ef50 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c3affd640 1 -- 172.21.15.94:0/2699438875 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c50018070 con 0x55c4da69ef50 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c3affd640 1 -- 172.21.15.94:0/2699438875 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 124) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4c4c045070 con 0x55c4da69ef50 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 10 client.8520 fetch_fsmap finished waiting for FSMap version 124 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 10 client.8520 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 20 client.8520 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6fa720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 4 client.8520 mds_command: new command op to 7723 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 --> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4da610bf0 con 0x55c4da6ef800 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6fa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c3affd640 10 client.8520 ms_handle_connect on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.285+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6fa720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bbff576801ebb0 server_cookie=b20a6e5ecd83d8d in_seq=0 out_seq=0 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c3affd640 1 -- 172.21.15.94:0/2699438875 <== mds.0 v2:172.21.15.145:6834/1362727065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c4da610bf0 con 0x55c4da6ef800 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c3affd640 10 client.8520 handle_command_reply: tid=0 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 client.8520 shutdown 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 msgr2=0x55c4da6c1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6c1600 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009890 tx=0x7f4c4c004730 comp rx=0 tx=0).stop 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 shutdown_connections 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da6c4a30 0x55c4da6bbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da69f330 0x55c4da6bd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6fa720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c3affd640 0 client.8520 ms_handle_reset on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:33.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 --2- 172.21.15.94:0/2699438875 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6c1600 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 >> 172.21.15.94:0/2699438875 conn(0x55c4da69d4b0 msgr2=0x55c4da698510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 shutdown_connections 2024-03-06T07:23:33.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.286+0000 7f4c655f7740 1 -- 172.21.15.94:0/2699438875 wait complete. 2024-03-06T07:23:33.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.287+0000 7f4c655f7740 20 client.8520 trim_cache size 0 max 16384 2024-03-06T07:23:33.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.303+0000 7f4c655f7740 1 Processor -- start 2024-03-06T07:23:33.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.303+0000 7f4c655f7740 1 -- start start 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.303+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da698510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da69c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da69cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c655f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c1f80c0 con 0x55c4da69ef50 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c1f7200 con 0x55c4da69f330 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c14bce0 con 0x55c4da6c4a30 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da698510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da698510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37812/0 (socket says 172.21.15.94:37812) 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 -- 172.21.15.94:0/717516819 learned_addr learned my addr 172.21.15.94:0/717516819 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c60fab640 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da69cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da69c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 -- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 msgr2=0x55c4da69cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da69cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 -- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 msgr2=0x55c4da69c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da69c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 -- 172.21.15.94:0/717516819 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4da3e26b0 con 0x55c4da69ef50 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da69c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c60fab640 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da69cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da698510 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c006c70 tx=0x7f4c4c04f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96be760709aa27e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c417fa640 1 -- 172.21.15.94:0/717516819 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c035040 con 0x55c4da69ef50 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c417fa640 1 -- 172.21.15.94:0/717516819 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c061020 con 0x55c4da69ef50 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.304+0000 7f4c417fa640 1 -- 172.21.15.94:0/717516819 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c003d60 con 0x55c4da69ef50 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 -- 172.21.15.94:0/717516819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 msgr2=0x55c4da698510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da698510 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c006c70 tx=0x7f4c4c04f000 comp rx=0 tx=0).stop 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 -- 172.21.15.94:0/717516819 shutdown_connections 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da69cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da69c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 --2- 172.21.15.94:0/717516819 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da698510 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 -- 172.21.15.94:0/717516819 >> 172.21.15.94:0/717516819 conn(0x55c4da69d4b0 msgr2=0x55c4da6bcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 -- 172.21.15.94:0/717516819 shutdown_connections 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 -- 172.21.15.94:0/717516819 wait complete. 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.305+0000 7f4c655f7740 1 -- start start 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 1 Processor -- start 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da74b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da6f5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4da6ef680 con 0x55c4da69ef50 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c1f6c30 con 0x55c4da69f330 2024-03-06T07:23:33.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4da6bc410 con 0x55c4da6c4a30 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c655f7740 10 client.0 _ll_register_callbacks cb 0x7f4c4c004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6e7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37818/0 (socket says 172.21.15.94:37818) 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3847348533 learned_addr learned my addr 172.21.15.94:0/3847348533 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da74b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.306+0000 7f4c60fab640 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da6f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 msgr2=0x55c4da6f5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da6f5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 msgr2=0x55c4da74b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da74b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c5bfff640 1 -- 172.21.15.94:0/3847348533 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4da74bc50 con 0x55c4da69ef50 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c5bfff640 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6e7e20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009560 tx=0x7f4c4c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d49a3cb26347fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c3affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c3affd640 1 -- 172.21.15.94:0/3847348533 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c035040 con 0x55c4da69ef50 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c3affd640 1 -- 172.21.15.94:0/3847348533 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4c4c03b840 con 0x55c4da69ef50 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c4c061020 con 0x55c4da69ef50 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c3affd640 1 -- 172.21.15.94:0/3847348533 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c4c071050 con 0x55c4da69ef50 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 10 client.8526 fetch_fsmap learned FSMap version 124 2024-03-06T07:23:33.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55c4da69cb60 con 0x55c4da69ef50 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c3affd640 1 -- 172.21.15.94:0/3847348533 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c061020 con 0x55c4da69ef50 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c3affd640 1 -- 172.21.15.94:0/3847348533 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 124) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4c4c074020 con 0x55c4da69ef50 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 10 client.8526 fetch_fsmap finished waiting for FSMap version 124 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 10 client.8526 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 20 client.8526 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6f5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 4 client.8526 mds_command: new command op to 7723 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:33.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 --> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c4da60af40 con 0x55c4da6ef800 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.307+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c3affd640 10 client.8526 ms_handle_connect on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c5b7fe640 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6f5600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee5b3e5d974da628 server_cookie=65fe4c419c89404c in_seq=0 out_seq=0 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c3affd640 1 -- 172.21.15.94:0/3847348533 <== mds.0 v2:172.21.15.145:6834/1362727065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x7f4c50018160 con 0x55c4da6ef800 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c3affd640 10 client.8526 handle_command_reply: tid=0 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 client.8526 shutdown 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 msgr2=0x55c4da6e7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6e7e20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009560 tx=0x7f4c4c031000 comp rx=0 tx=0).stop 2024-03-06T07:23:33.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 shutdown_connections 2024-03-06T07:23:33.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c4da6c4a30 0x55c4da6f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c4da69f330 0x55c4da74b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55c4da6ef800 0x55c4da6f5600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 --2- 172.21.15.94:0/3847348533 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c4da69ef50 0x55c4da6e7e20 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c3affd640 0 client.8526 ms_handle_reset on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 >> 172.21.15.94:0/3847348533 conn(0x55c4da69d4b0 msgr2=0x55c4da698510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 shutdown_connections 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 1 -- 172.21.15.94:0/3847348533 wait complete. 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.308+0000 7f4c655f7740 20 client.8526 trim_cache size 0 max 16384 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7723, 2024-03-06T07:23:33.324 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.8026837750000002, 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 124, 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 136, 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 136, 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 17.588676320000001 2024-03-06T07:23:33.325 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:33.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c4806b4d0 msgr2=0x7f4c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c4806b4d0 0x7f4c4806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c036000 tx=0x7f4c4c03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c5c0ad9e0 msgr2=0x7f4c5c1eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c5c0ad9e0 0x7f4c5c1eb080 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400c6b0 tx=0x7f4c5400cb70 comp rx=0 tx=0).stop 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 shutdown_connections 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4c4806b4d0 0x7f4c4806d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c5c0ad9e0 0x7f4c5c1eb080 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c5c0a7ff0 0x7f4c5c0a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 --2- 172.21.15.94:0/278053855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c5c0724d0 0x7f4c5c14c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 >> 172.21.15.94:0/278053855 conn(0x7f4c5c06b970 msgr2=0x7f4c5c0c3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 shutdown_connections 2024-03-06T07:23:33.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:33.309+0000 7f4c62a0e640 1 -- 172.21.15.94:0/278053855 wait complete. 2024-03-06T07:23:33.327 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 124, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:20.150839+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7723": { "addr": "172.21.15.145:6836/1362727065", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1362727065, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1362727065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7723, "incarnation": 121, "join_fscid": 10, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 136, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:23:31.523761+0000", "qdb_cluster": [ 7723 ], "qdb_leader": 7723, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7723 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/3826543722", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 3826543722, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 3826543722, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7726, "incarnation": 0, "join_fscid": 10, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 4 }, { "addr": "172.21.15.94:6833/3405703874", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3405703874, "type": "v2" }, { "addr": "172.21.15.94:6833", "nonce": 3405703874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8298, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/801675853", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6834", "nonce": 801675853, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 801675853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 122, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8499, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:33.329 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-03-06T07:23:33.329 INFO:tasks.cephfs.test_failover:current = [{'gid': 8298, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6833/3405703874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3405703874}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 3405703874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7726, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 4, 'addr': '172.21.15.145:6837/3826543722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3826543722}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3826543722}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 120}, {'gid': 8499, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/801675853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 801675853}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 801675853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 122}, {'gid': 7723, 'name': 'd', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.145:6836/1362727065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1362727065}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1362727065}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:23:33.329 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:23:36.092 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:23:36.091+0000 7fb25a748640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:38.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:38.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.440+0000 7ff22a5ae640 1 Processor -- start 2024-03-06T07:23:38.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.440+0000 7ff22a5ae640 1 -- start start 2024-03-06T07:23:38.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.440+0000 7ff22a5ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff224155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.440+0000 7ff22a5ae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240c73d0 con 0x7ff22414f360 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.440+0000 7ff22a5ae640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240be2e0 con 0x7ff2240c88c0 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.440+0000 7ff22a5ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff224159700 con 0x7ff224159270 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.440+0000 7ff223fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff22414f360 0x7ff2240c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36646/0 (socket says 172.21.15.94:36646) 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff223fff640 1 -- 172.21.15.94:0/3864806583 learned_addr learned my addr 172.21.15.94:0/3864806583 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff228b4b640 1 --2- 172.21.15.94:0/3864806583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff224155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff222ffd640 1 -- 172.21.15.94:0/3864806583 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886888884 0 0) 0x7ff2240c73d0 con 0x7ff22414f360 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff222ffd640 1 -- 172.21.15.94:0/3864806583 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff20c003520 con 0x7ff22414f360 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff222ffd640 1 -- 172.21.15.94:0/3864806583 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97709847 0 0) 0x7ff2240be2e0 con 0x7ff2240c88c0 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff222ffd640 1 -- 172.21.15.94:0/3864806583 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2240c73d0 con 0x7ff2240c88c0 2024-03-06T07:23:38.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff228b4b640 1 -- 172.21.15.94:0/3864806583 >> v1:172.21.15.145:6789/0 conn(0x7ff2240c88c0 legacy=0x7ff224158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff228b4b640 1 -- 172.21.15.94:0/3864806583 >> v1:172.21.15.94:6789/0 conn(0x7ff22414f360 legacy=0x7ff2240c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.441+0000 7ff228b4b640 1 -- 172.21.15.94:0/3864806583 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240c7670 con 0x7ff224159270 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.442+0000 7ff228b4b640 1 --2- 172.21.15.94:0/3864806583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff224155c10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009dc0 tx=0x7ff21c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e41203a7a51780ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.442+0000 7ff222ffd640 1 -- 172.21.15.94:0/3864806583 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c039070 con 0x7ff224159270 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.442+0000 7ff222ffd640 1 -- 172.21.15.94:0/3864806583 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c02cc40 con 0x7ff224159270 2024-03-06T07:23:38.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.442+0000 7ff222ffd640 1 -- 172.21.15.94:0/3864806583 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c034440 con 0x7ff224159270 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.443+0000 7ff22a5ae640 1 -- 172.21.15.94:0/3864806583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 msgr2=0x7ff224155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.443+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/3864806583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff224155c10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009dc0 tx=0x7ff21c02be90 comp rx=0 tx=0).stop 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.443+0000 7ff22a5ae640 1 -- 172.21.15.94:0/3864806583 shutdown_connections 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.443+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/3864806583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff224155c10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.443+0000 7ff22a5ae640 1 -- 172.21.15.94:0/3864806583 >> 172.21.15.94:0/3864806583 conn(0x7ff2240ac570 msgr2=0x7ff2240ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.443+0000 7ff22a5ae640 1 -- 172.21.15.94:0/3864806583 shutdown_connections 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.443+0000 7ff22a5ae640 1 -- 172.21.15.94:0/3864806583 wait complete. 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.444+0000 7ff22a5ae640 1 Processor -- start 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.444+0000 7ff22a5ae640 1 -- start start 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2240c88c0 0x7ff22414e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22414f360 0x7ff2240c2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff2240c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240c7c30 con 0x7ff2240c88c0 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240c5a50 con 0x7ff22414f360 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240c5bd0 con 0x7ff224159270 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2240c88c0 0x7ff22414e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2240c88c0 0x7ff22414e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39966/0 (socket says 172.21.15.94:39966) 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 -- 172.21.15.94:0/171719604 learned_addr learned my addr 172.21.15.94:0/171719604 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff2237fe640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22414f360 0x7ff2240c2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff228b4b640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff2240c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 -- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 msgr2=0x7ff2240c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff2240c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 -- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22414f360 msgr2=0x7ff2240c2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22414f360 0x7ff2240c2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 -- 172.21.15.94:0/171719604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff224149a10 con 0x7ff2240c88c0 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff223fff640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2240c88c0 0x7ff22414e3e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff21400c500 tx=0x7ff21400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff220ff9640 1 -- 172.21.15.94:0/171719604 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214017070 con 0x7ff2240c88c0 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff220ff9640 1 -- 172.21.15.94:0/171719604 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff214005ce0 con 0x7ff2240c88c0 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2241eed80 con 0x7ff2240c88c0 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff220ff9640 1 -- 172.21.15.94:0/171719604 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2140072f0 con 0x7ff2240c88c0 2024-03-06T07:23:38.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.445+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22414e920 con 0x7ff2240c88c0 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.447+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22404f330 con 0x7ff2240c88c0 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.447+0000 7ff220ff9640 1 -- 172.21.15.94:0/171719604 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff21401a070 con 0x7ff2240c88c0 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.447+0000 7ff220ff9640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b4d0 0x7ff21006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.448+0000 7ff220ff9640 1 -- 172.21.15.94:0/171719604 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff2140896f0 con 0x7ff2240c88c0 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.448+0000 7ff2237fe640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b4d0 0x7ff21006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:38.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.448+0000 7ff2237fe640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b4d0 0x7ff21006d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff218007fd0 tx=0x7ff218009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:38.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.452+0000 7ff220ff9640 1 -- 172.21.15.94:0/171719604 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff214057dd0 con 0x7ff2240c88c0 2024-03-06T07:23:38.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.660+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2240c8d50 con 0x7ff2240c88c0 2024-03-06T07:23:38.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.660+0000 7ff220ff9640 1 -- 172.21.15.94:0/171719604 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4992 (secure 0 0 0) 0x7ff21405be70 con 0x7ff2240c88c0 2024-03-06T07:23:38.662 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:38.662 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":124,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:31.523761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7723},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7723":{"gid":7723,"name":"d","rank":0,"incarnation":121,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6836/1362727065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1362727065},{"type":"v1","addr":"172.21.15.145:6836","nonce":1362727065}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7723,"qdb_cluster":[7723]},"id":10}]} 2024-03-06T07:23:38.663 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 124 2024-03-06T07:23:38.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b4d0 msgr2=0x7ff21006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b4d0 0x7ff21006d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff218007fd0 tx=0x7ff218009040 comp rx=0 tx=0).stop 2024-03-06T07:23:38.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2240c88c0 msgr2=0x7ff22414e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:38.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2240c88c0 0x7ff22414e3e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ff21400c500 tx=0x7ff21400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 shutdown_connections 2024-03-06T07:23:38.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff21006b4d0 0x7ff21006d990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff224159270 0x7ff2240c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff22414f360 0x7ff2240c2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 --2- 172.21.15.94:0/171719604 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff2240c88c0 0x7ff22414e3e0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:38.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 >> 172.21.15.94:0/171719604 conn(0x7ff2240ac570 msgr2=0x7ff2241547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:38.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 shutdown_connections 2024-03-06T07:23:38.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:38.662+0000 7ff22a5ae640 1 -- 172.21.15.94:0/171719604 wait complete. 2024-03-06T07:23:39.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:20.150839+0000', 'modified': '2024-03-06T07:23:31.523761+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7723}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7723': {'gid': 7723, 'name': 'd', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.145:6836/1362727065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1362727065}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1362727065}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7723, 'qdb_cluster': [7723]} 2024-03-06T07:23:39.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:39.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:23:39.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.178+0000 7f9c4edbc640 1 Processor -- start 2024-03-06T07:23:39.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.178+0000 7f9c4edbc640 1 -- start start 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4edbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c481571e0 0x7f9c4806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4edbc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9c480a9ea0 0x7f9c4815d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4edbc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4806fa80 con 0x7f9c480c20b0 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4edbc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c48153100 con 0x7f9c480a9ea0 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4edbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480a75d0 con 0x7f9c481571e0 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4d359640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9c480a9ea0 0x7f9c4815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c47fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9c480c20b0 0x7f9c48153590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36654/0 (socket says 172.21.15.94:36654) 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4d359640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9c480a9ea0 0x7f9c4815d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39968/0 (socket says 172.21.15.94:39968) 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c47fff640 1 -- 172.21.15.94:0/4269836058 learned_addr learned my addr 172.21.15.94:0/4269836058 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/4269836058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c481571e0 0x7f9c4806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4d359640 1 -- 172.21.15.94:0/4269836058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c481571e0 msgr2=0x7f9c4806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4d359640 1 --2- 172.21.15.94:0/4269836058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c481571e0 0x7f9c4806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4d359640 1 -- 172.21.15.94:0/4269836058 >> v1:172.21.15.94:6789/0 conn(0x7f9c480c20b0 legacy=0x7f9c48153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c4d359640 1 -- 172.21.15.94:0/4269836058 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480a78b0 con 0x7f9c480a9ea0 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.179+0000 7f9c477fe640 1 -- 172.21.15.94:0/4269836058 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545673043 0 0) 0x7f9c4806fa80 con 0x7f9c480c20b0 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.180+0000 7f9c4d359640 1 --2- 172.21.15.94:0/4269836058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480a9ea0 0x7f9c4815d850 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f9c40009d90 tx=0x7f9c4002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7004518316f75403 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.180+0000 7f9c477fe640 1 -- 172.21.15.94:0/4269836058 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40039070 con 0x7f9c480a9ea0 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.180+0000 7f9c477fe640 1 -- 172.21.15.94:0/4269836058 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4002cbc0 con 0x7f9c480a9ea0 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.180+0000 7f9c477fe640 1 -- 172.21.15.94:0/4269836058 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40034440 con 0x7f9c480a9ea0 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 -- 172.21.15.94:0/4269836058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480a9ea0 msgr2=0x7f9c4815d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/4269836058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480a9ea0 0x7f9c4815d850 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f9c40009d90 tx=0x7f9c4002be10 comp rx=0 tx=0).stop 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 -- 172.21.15.94:0/4269836058 shutdown_connections 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/4269836058 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480a9ea0 0x7f9c4815d850 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/4269836058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c481571e0 0x7f9c4806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 -- 172.21.15.94:0/4269836058 >> 172.21.15.94:0/4269836058 conn(0x7f9c480c41b0 msgr2=0x7f9c480c4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 -- 172.21.15.94:0/4269836058 shutdown_connections 2024-03-06T07:23:39.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.181+0000 7f9c4edbc640 1 -- 172.21.15.94:0/4269836058 wait complete. 2024-03-06T07:23:39.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.182+0000 7f9c4edbc640 1 Processor -- start 2024-03-06T07:23:39.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.182+0000 7f9c4edbc640 1 -- start start 2024-03-06T07:23:39.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4edbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c480a9ea0 0x7f9c481eab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4edbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480c20b0 0x7f9c481de980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4edbc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c481571e0 0x7f9c481ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4edbc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480a7e70 con 0x7f9c480c20b0 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4edbc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480a72c0 con 0x7f9c481571e0 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4edbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48153ca0 con 0x7f9c480a9ea0 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480c20b0 0x7f9c481de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480c20b0 0x7f9c481de980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39982/0 (socket says 172.21.15.94:39982) 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 -- 172.21.15.94:0/76930010 learned_addr learned my addr 172.21.15.94:0/76930010 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c480a9ea0 0x7f9c481eab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c4d359640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c481571e0 0x7f9c481ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 -- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c480a9ea0 msgr2=0x7f9c481eab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c480a9ea0 0x7f9c481eab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 -- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c481571e0 msgr2=0x7f9c481ee8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c481571e0 0x7f9c481ee8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 -- 172.21.15.94:0/76930010 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c481ee160 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.183+0000 7f9c47fff640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480c20b0 0x7f9c481de980 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00c4d0 tx=0x7f9c3c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.184+0000 7f9c457fa640 1 -- 172.21.15.94:0/76930010 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c018070 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.184+0000 7f9c457fa640 1 -- 172.21.15.94:0/76930010 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3c0074e0 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.184+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4814cee0 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.184+0000 7f9c457fa640 1 -- 172.21.15.94:0/76930010 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c004a20 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.184+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c481deec0 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.185+0000 7f9c457fa640 1 -- 172.21.15.94:0/76930010 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c3c01b070 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.185+0000 7f9c457fa640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c3406b4d0 0x7f9c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.186+0000 7f9c457fa640 1 -- 172.21.15.94:0/76930010 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9c3c089600 con 0x7f9c480c20b0 2024-03-06T07:23:39.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.186+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c3406b4d0 0x7f9c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.186+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c3406b4d0 0x7f9c3406d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9c480c15c0 tx=0x7f9c38009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.186+0000 7f9c519a1740 1 Processor -- start 2024-03-06T07:23:39.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.186+0000 7f9c519a1740 1 -- start start 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2df50 0x55bb3cc2ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3cc2bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2c4b0 0x55bb3cc28680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c519a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cb926b0 con 0x55bb3cc2e330 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cb3ba60 con 0x55bb3cc2df50 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cb8e750 con 0x55bb3cc2c4b0 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3cc2bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3cc2bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39996/0 (socket says 172.21.15.94:39996) 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 -- 172.21.15.94:0/2988928125 learned_addr learned my addr 172.21.15.94:0/2988928125 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2df50 0x55bb3cc2ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c4d359640 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2c4b0 0x55bb3cc28680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 -- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2c4b0 msgr2=0x55bb3cc28680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2c4b0 0x55bb3cc28680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 -- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2df50 msgr2=0x55bb3cc2ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2df50 0x55bb3cc2ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.187+0000 7f9c47fff640 1 -- 172.21.15.94:0/2988928125 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb3cb59f30 con 0x55bb3cc2e330 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c47fff640 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3cc2bf70 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c007fd0 tx=0x7f9c3c057da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68dfd2b559873b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c2e7fc640 1 -- 172.21.15.94:0/2988928125 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c05e070 con 0x55bb3cc2e330 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c2e7fc640 1 -- 172.21.15.94:0/2988928125 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3c055fe0 con 0x55bb3cc2e330 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c2e7fc640 1 -- 172.21.15.94:0/2988928125 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c004bc0 con 0x55bb3cc2e330 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 -- 172.21.15.94:0/2988928125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 msgr2=0x55bb3cc2bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3cc2bf70 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c007fd0 tx=0x7f9c3c057da0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 -- 172.21.15.94:0/2988928125 shutdown_connections 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2c4b0 0x55bb3cc28680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3cc2bf70 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 --2- 172.21.15.94:0/2988928125 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2df50 0x55bb3cc2ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 -- 172.21.15.94:0/2988928125 >> 172.21.15.94:0/2988928125 conn(0x55bb3cc2a7d0 msgr2=0x55bb3cc4f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 -- 172.21.15.94:0/2988928125 shutdown_connections 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.188+0000 7f9c519a1740 1 -- 172.21.15.94:0/2988928125 wait complete. 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.189+0000 7f9c519a1740 1 -- start start 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.189+0000 7f9c519a1740 1 Processor -- start 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc49c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc53a30 0x55bb3cc7e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c519a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cc50230 con 0x55bb3cc53a30 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cb4eea0 con 0x55bb3cc2e330 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cc4fc70 con 0x55bb3cc2df50 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c519a1740 10 client.0 _ll_register_callbacks cb 0x55bb3cb4e870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc53a30 0x55bb3cc7e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4cb58640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc49c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc53a30 0x55bb3cc7e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40004/0 (socket says 172.21.15.94:40004) 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 -- 172.21.15.94:0/853003927 learned_addr learned my addr 172.21.15.94:0/853003927 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c47fff640 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 -- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 msgr2=0x55bb3cc49c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc49c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 -- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 msgr2=0x55bb3cc4be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 -- 172.21.15.94:0/853003927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb3cb4f270 con 0x55bb3cc53a30 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c4d359640 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc53a30 0x55bb3cc7e300 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9c4002be50 tx=0x7f9c40004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3273a47bc299b6e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c277fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c277fe640 1 -- 172.21.15.94:0/853003927 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40039040 con 0x55bb3cc53a30 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.190+0000 7f9c277fe640 1 -- 172.21.15.94:0/853003927 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9c4004bcc0 con 0x55bb3cc53a30 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c3c055fe0 con 0x55bb3cc53a30 2024-03-06T07:23:39.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c277fe640 1 -- 172.21.15.94:0/853003927 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c40041640 con 0x55bb3cc53a30 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 10 client.8547 fetch_fsmap learned FSMap version 124 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55bb3c8a72d0 con 0x55bb3cc53a30 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c277fe640 1 -- 172.21.15.94:0/853003927 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3c055fe0 con 0x55bb3cc53a30 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c277fe640 1 -- 172.21.15.94:0/853003927 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 124) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9c40052050 con 0x55bb3cc53a30 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 10 client.8547 fetch_fsmap finished waiting for FSMap version 124 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 10 client.8547 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 20 client.8547 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc66940 0x55bb3cc66d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 4 client.8547 mds_command: new command op to 7723 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 --> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb3cb9fbf0 con 0x55bb3cc66940 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc66940 0x55bb3cc66d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc66940 0x55bb3cc66d20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ad707768afb9200 server_cookie=e1796a1c4e68b982 in_seq=0 out_seq=0 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.191+0000 7f9c277fe640 10 client.8547 ms_handle_connect on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:39.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c277fe640 1 -- 172.21.15.94:0/853003927 <== mds.0 v2:172.21.15.145:6834/1362727065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bb3cb9fbf0 con 0x55bb3cc66940 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c277fe640 10 client.8547 handle_command_reply: tid=0 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 client.8547 shutdown 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc53a30 msgr2=0x55bb3cc7e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc53a30 0x55bb3cc7e300 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9c4002be50 tx=0x7f9c40004ab0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 shutdown_connections 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc66940 0x55bb3cc66d20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc53a30 0x55bb3cc7e300 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 --2- 172.21.15.94:0/853003927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc49c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c277fe640 0 client.8547 ms_handle_reset on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 >> 172.21.15.94:0/853003927 conn(0x55bb3cc2c4b0 msgr2=0x55bb3cc28590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 shutdown_connections 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 1 -- 172.21.15.94:0/853003927 wait complete. 2024-03-06T07:23:39.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.192+0000 7f9c519a1740 20 client.8547 trim_cache size 0 max 16384 2024-03-06T07:23:39.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 Processor -- start 2024-03-06T07:23:39.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 -- start start 2024-03-06T07:23:39.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2df50 0x55bb3cc50a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc53a30 0x55bb3cc4f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c481f6e10 con 0x55bb3cc2df50 2024-03-06T07:23:39.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c481e23c0 con 0x55bb3cc2e330 2024-03-06T07:23:39.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.209+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c481f6810 con 0x55bb3cc53a30 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2df50 0x55bb3cc50a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2df50 0x55bb3cc50a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40014/0 (socket says 172.21.15.94:40014) 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 -- 172.21.15.94:0/3876694070 learned_addr learned my addr 172.21.15.94:0/3876694070 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4d359640 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc53a30 0x55bb3cc4f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c47fff640 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 -- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc53a30 msgr2=0x55bb3cc4f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc53a30 0x55bb3cc4f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 -- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 msgr2=0x55bb3cc4eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 -- 172.21.15.94:0/3876694070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb3c9716b0 con 0x55bb3cc2df50 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4d359640 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc53a30 0x55bb3cc4f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2df50 0x55bb3cc50a00 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9c38016040 tx=0x7f9c38018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93df2577eee431e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c2dffb640 1 -- 172.21.15.94:0/3876694070 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38003750 con 0x55bb3cc2df50 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c2dffb640 1 -- 172.21.15.94:0/3876694070 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c38030070 con 0x55bb3cc2df50 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.210+0000 7f9c2dffb640 1 -- 172.21.15.94:0/3876694070 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38003b30 con 0x55bb3cc2df50 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 -- 172.21.15.94:0/3876694070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2df50 msgr2=0x55bb3cc50a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2df50 0x55bb3cc50a00 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f9c38016040 tx=0x7f9c38018000 comp rx=0 tx=0).stop 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 -- 172.21.15.94:0/3876694070 shutdown_connections 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc53a30 0x55bb3cc4f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc2e330 0x55bb3cc4eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 --2- 172.21.15.94:0/3876694070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2df50 0x55bb3cc50a00 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 -- 172.21.15.94:0/3876694070 >> 172.21.15.94:0/3876694070 conn(0x55bb3cc2c4b0 msgr2=0x55bb3cc4bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 -- 172.21.15.94:0/3876694070 shutdown_connections 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 -- 172.21.15.94:0/3876694070 wait complete. 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 -- start start 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.211+0000 7f9c519a1740 1 Processor -- start 2024-03-06T07:23:39.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc4f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3ccdd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c519a1740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc53a30 0x55bb3ccd6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c519a1740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cc7dde0 con 0x55bb3cc2e330 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cc667c0 con 0x55bb3cc53a30 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c519a1740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb3cc7a790 con 0x55bb3cc2df50 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c519a1740 10 client.0 _ll_register_callbacks cb 0x7f9c4002cad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3ccdd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3ccdd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40020/0 (socket says 172.21.15.94:40020) 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 -- 172.21.15.94:0/553774778 learned_addr learned my addr 172.21.15.94:0/553774778 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c4d359640 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc53a30 0x55bb3ccd6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc4f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 -- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 msgr2=0x55bb3cc4f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc4f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 -- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc53a30 msgr2=0x55bb3ccd6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc53a30 0x55bb3ccd6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.212+0000 7f9c47fff640 1 -- 172.21.15.94:0/553774778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb3cc2d7e0 con 0x55bb3cc2e330 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c47fff640 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3ccdd260 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c087a30 tx=0x7f9c3c05b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aaff7ba02a38fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c277fe640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c277fe640 1 -- 172.21.15.94:0/553774778 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c05e070 con 0x55bb3cc2e330 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c277fe640 1 -- 172.21.15.94:0/553774778 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9c3c004dc0 con 0x55bb3cc2e330 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c38030070 con 0x55bb3cc2e330 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c277fe640 1 -- 172.21.15.94:0/553774778 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c3c057ab0 con 0x55bb3cc2e330 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 10 client.8553 fetch_fsmap learned FSMap version 124 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55bb3cc2be30 con 0x55bb3cc2e330 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c277fe640 1 -- 172.21.15.94:0/553774778 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c38030070 con 0x55bb3cc2e330 2024-03-06T07:23:39.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c277fe640 1 -- 172.21.15.94:0/553774778 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 124) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9c3c057130 con 0x55bb3cc2e330 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 10 client.8553 fetch_fsmap finished waiting for FSMap version 124 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 10 client.8553 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 20 client.8553 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc776a0 0x55bb3cc77a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 4 client.8553 mds_command: new command op to 7723 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 --> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bb3cb99f40 con 0x55bb3cc776a0 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.213+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc776a0 0x55bb3cc77a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c4cb58640 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc776a0 0x55bb3cc77a80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34844c7f61641486 server_cookie=cb8a9408bfd2853a in_seq=0 out_seq=0 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c277fe640 10 client.8553 ms_handle_connect on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c277fe640 1 -- 172.21.15.94:0/553774778 <== mds.0 v2:172.21.15.145:6834/1362727065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55bb3cb99f40 con 0x55bb3cc776a0 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c277fe640 10 client.8553 handle_command_reply: tid=0 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 client.8553 shutdown 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 msgr2=0x55bb3ccdd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3ccdd260 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c087a30 tx=0x7f9c3c05b000 comp rx=0 tx=0).stop 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 shutdown_connections 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bb3cc53a30 0x55bb3ccd6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55bb3cc776a0 0x55bb3cc77a80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55bb3cc2e330 0x55bb3ccdd260 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c277fe640 0 client.8553 ms_handle_reset on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 --2- 172.21.15.94:0/553774778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bb3cc2df50 0x55bb3cc4f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 >> 172.21.15.94:0/553774778 conn(0x55bb3cc2c4b0 msgr2=0x55bb3cc2f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 shutdown_connections 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 1 -- 172.21.15.94:0/553774778 wait complete. 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.214+0000 7f9c519a1740 20 client.8553 trim_cache size 0 max 16384 2024-03-06T07:23:39.223 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7723, 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 10.708561620999999, 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 124, 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 136, 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 136, 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 23.494554181000002 2024-03-06T07:23:39.224 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:39.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c3406b4d0 msgr2=0x7f9c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c3406b4d0 0x7f9c3406d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9c480c15c0 tx=0x7f9c38009190 comp rx=0 tx=0).stop 2024-03-06T07:23:39.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480c20b0 msgr2=0x7f9c481de980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480c20b0 0x7f9c481de980 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00c4d0 tx=0x7f9c3c00c990 comp rx=0 tx=0).stop 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 shutdown_connections 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c3406b4d0 0x7f9c3406d990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c481571e0 0x7f9c481ee8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c480c20b0 0x7f9c481de980 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 --2- 172.21.15.94:0/76930010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c480a9ea0 0x7f9c481eab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 >> 172.21.15.94:0/76930010 conn(0x7f9c480c41b0 msgr2=0x7f9c48155fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 shutdown_connections 2024-03-06T07:23:39.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.215+0000 7f9c4edbc640 1 -- 172.21.15.94:0/76930010 wait complete. 2024-03-06T07:23:39.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:23:39.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.327+0000 7f00b913e640 1 Processor -- start 2024-03-06T07:23:39.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00b913e640 1 -- start start 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00b913e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f00b41538c0 0x7f00b41494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00b913e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b4154330 con 0x7f00b40ab310 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00b913e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b4153dc0 con 0x7f00b40a97c0 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00b913e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b40c46a0 con 0x7f00b41538c0 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00b2d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f00b40ab310 0x7f00b40ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36660/0 (socket says 172.21.15.94:36660) 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00b2d9d640 1 -- 172.21.15.94:0/3909495708 learned_addr learned my addr 172.21.15.94:0/3909495708 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.328+0000 7f00abfff640 1 --2- 172.21.15.94:0/3909495708 >> v2:172.21.15.145:3300/0 conn(0x7f00b41538c0 0x7f00b41494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.329+0000 7f00b259c640 1 -- 172.21.15.94:0/3909495708 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351096307 0 0) 0x7f00b4154330 con 0x7f00b40ab310 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.329+0000 7f00b259c640 1 -- 172.21.15.94:0/3909495708 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a0003520 con 0x7f00b40ab310 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.329+0000 7f00abfff640 1 -- 172.21.15.94:0/3909495708 >> v1:172.21.15.145:6789/0 conn(0x7f00b40a97c0 legacy=0x7f00b40c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.329+0000 7f00abfff640 1 -- 172.21.15.94:0/3909495708 >> v1:172.21.15.94:6789/0 conn(0x7f00b40ab310 legacy=0x7f00b40ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.329+0000 7f00abfff640 1 -- 172.21.15.94:0/3909495708 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b40c5d30 con 0x7f00b41538c0 2024-03-06T07:23:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00abfff640 1 --2- 172.21.15.94:0/3909495708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b41538c0 0x7f00b41494a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f009c0097a0 tx=0x7f009c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fc9711e851e54c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b259c640 1 -- 172.21.15.94:0/3909495708 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c035020 con 0x7f00b41538c0 2024-03-06T07:23:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b259c640 1 -- 172.21.15.94:0/3909495708 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c031830 con 0x7f00b41538c0 2024-03-06T07:23:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b259c640 1 -- 172.21.15.94:0/3909495708 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c031be0 con 0x7f00b41538c0 2024-03-06T07:23:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b913e640 1 -- 172.21.15.94:0/3909495708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b41538c0 msgr2=0x7f00b41494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b913e640 1 --2- 172.21.15.94:0/3909495708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b41538c0 0x7f00b41494a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f009c0097a0 tx=0x7f009c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b913e640 1 -- 172.21.15.94:0/3909495708 shutdown_connections 2024-03-06T07:23:39.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b913e640 1 --2- 172.21.15.94:0/3909495708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b41538c0 0x7f00b41494a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b913e640 1 -- 172.21.15.94:0/3909495708 >> 172.21.15.94:0/3909495708 conn(0x7f00b40c6710 msgr2=0x7f00b40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b913e640 1 -- 172.21.15.94:0/3909495708 shutdown_connections 2024-03-06T07:23:39.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.330+0000 7f00b913e640 1 -- 172.21.15.94:0/3909495708 wait complete. 2024-03-06T07:23:39.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.331+0000 7f00b913e640 1 Processor -- start 2024-03-06T07:23:39.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b913e640 1 -- start start 2024-03-06T07:23:39.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b913e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00b40a97c0 0x7f00b4072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b913e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b40ab310 0x7f00b4071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b913e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00b41538c0 0x7f00b414ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b913e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40c5240 con 0x7f00b40a97c0 2024-03-06T07:23:39.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b913e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40c62f0 con 0x7f00b40ab310 2024-03-06T07:23:39.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b913e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b40c53c0 con 0x7f00b41538c0 2024-03-06T07:23:39.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00b40a97c0 0x7f00b4072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00b40a97c0 0x7f00b4072fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40028/0 (socket says 172.21.15.94:40028) 2024-03-06T07:23:39.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b2d9d640 1 -- 172.21.15.94:0/4238605281 learned_addr learned my addr 172.21.15.94:0/4238605281 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:39.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00abfff640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b40ab310 0x7f00b4071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.332+0000 7f00b359e640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00b41538c0 0x7f00b414ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b2d9d640 1 -- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00b41538c0 msgr2=0x7f00b414ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b2d9d640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00b41538c0 0x7f00b414ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b2d9d640 1 -- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b40ab310 msgr2=0x7f00b4071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b2d9d640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b40ab310 0x7f00b4071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b2d9d640 1 -- 172.21.15.94:0/4238605281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4070940 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b2d9d640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00b40a97c0 0x7f00b4072fd0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f00a400c500 tx=0x7f00a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00ab7fe640 1 -- 172.21.15.94:0/4238605281 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4017070 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00ab7fe640 1 -- 172.21.15.94:0/4238605281 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a4005ce0 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b41ebe10 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00ab7fe640 1 -- 172.21.15.94:0/4238605281 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a4007330 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.333+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b41ec3d0 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.334+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b41ec5d0 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.334+0000 7f00ab7fe640 1 -- 172.21.15.94:0/4238605281 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00a401a070 con 0x7f00b40a97c0 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.335+0000 7f00ab7fe640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f009406b4d0 0x7f009406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.335+0000 7f00ab7fe640 1 -- 172.21.15.94:0/4238605281 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f00a4089710 con 0x7f00b40a97c0 2024-03-06T07:23:39.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.335+0000 7f00abfff640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f009406b4d0 0x7f009406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.335+0000 7f00abfff640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f009406b4d0 0x7f009406d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f009c036000 tx=0x7f009c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.339+0000 7f00ab7fe640 1 -- 172.21.15.94:0/4238605281 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f00b41ec5d0 con 0x7f00b40a97c0 2024-03-06T07:23:39.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.509+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f00b404f330 con 0x7f00b40a97c0 2024-03-06T07:23:39.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.511+0000 7f00ab7fe640 1 -- 172.21.15.94:0/4238605281 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v48) v1 ==== 96+0+0 (secure 0 0 0) 0x7f00a4057eb0 con 0x7f00b40a97c0 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f009406b4d0 msgr2=0x7f009406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f009406b4d0 0x7f009406d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f009c036000 tx=0x7f009c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00b40a97c0 msgr2=0x7f00b4072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00b40a97c0 0x7f00b4072fd0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f00a400c500 tx=0x7f00a400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 shutdown_connections 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f009406b4d0 0x7f009406d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00b41538c0 0x7f00b414ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00b40ab310 0x7f00b4071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 --2- 172.21.15.94:0/4238605281 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f00b40a97c0 0x7f00b4072fd0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 >> 172.21.15.94:0/4238605281 conn(0x7f00b40c6710 msgr2=0x7f00b40c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 shutdown_connections 2024-03-06T07:23:39.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.514+0000 7f00b913e640 1 -- 172.21.15.94:0/4238605281 wait complete. 2024-03-06T07:23:39.526 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.625+0000 7fbff6cc2640 1 Processor -- start 2024-03-06T07:23:39.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.625+0000 7fbff6cc2640 1 -- start start 2024-03-06T07:23:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.626+0000 7fbff6cc2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbff0153990 0x7fbff0156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.626+0000 7fbff6cc2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff0070aa0 con 0x7fbff01506b0 2024-03-06T07:23:39.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.626+0000 7fbff6cc2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff0070d80 con 0x7fbff00acc00 2024-03-06T07:23:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.626+0000 7fbff6cc2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff0156450 con 0x7fbff0153990 2024-03-06T07:23:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.626+0000 7fbff4a5e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbff01506b0 0x7fbff00702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36666/0 (socket says 172.21.15.94:36666) 2024-03-06T07:23:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.626+0000 7fbff4a5e640 1 -- 172.21.15.94:0/873862432 learned_addr learned my addr 172.21.15.94:0/873862432 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.626+0000 7fbfeffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbff0153990 0x7fbff0156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.627+0000 7fbfeffff640 1 -- 172.21.15.94:0/873862432 >> v1:172.21.15.145:6789/0 conn(0x7fbff00acc00 legacy=0x7fbff00c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.627+0000 7fbfeffff640 1 -- 172.21.15.94:0/873862432 >> v1:172.21.15.94:6789/0 conn(0x7fbff01506b0 legacy=0x7fbff00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.627+0000 7fbfeffff640 1 -- 172.21.15.94:0/873862432 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff01566a0 con 0x7fbff0153990 2024-03-06T07:23:39.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.627+0000 7fbfeffff640 1 --2- 172.21.15.94:0/873862432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff0153990 0x7fbff0156ea0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40097a0 tx=0x7fbfe402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9e949bc9e431c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.627+0000 7fbfef7fe640 1 -- 172.21.15.94:0/873862432 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4034070 con 0x7fbff0153990 2024-03-06T07:23:39.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbfef7fe640 1 -- 172.21.15.94:0/873862432 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4030610 con 0x7fbff0153990 2024-03-06T07:23:39.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbfef7fe640 1 -- 172.21.15.94:0/873862432 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40309e0 con 0x7fbff0153990 2024-03-06T07:23:39.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbff6cc2640 1 -- 172.21.15.94:0/873862432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff0153990 msgr2=0x7fbff0156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/873862432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff0153990 0x7fbff0156ea0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40097a0 tx=0x7fbfe402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:39.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbff6cc2640 1 -- 172.21.15.94:0/873862432 shutdown_connections 2024-03-06T07:23:39.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/873862432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff0153990 0x7fbff0156ea0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbff6cc2640 1 -- 172.21.15.94:0/873862432 >> 172.21.15.94:0/873862432 conn(0x7fbff006b1a0 msgr2=0x7fbff006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbff6cc2640 1 -- 172.21.15.94:0/873862432 shutdown_connections 2024-03-06T07:23:39.631 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.628+0000 7fbff6cc2640 1 -- 172.21.15.94:0/873862432 wait complete. 2024-03-06T07:23:39.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.629+0000 7fbff6cc2640 1 Processor -- start 2024-03-06T07:23:39.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.629+0000 7fbff6cc2640 1 -- start start 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff6cc2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbff00acc00 0x7fbff01ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff6cc2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff01506b0 0x7fbff014a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff6cc2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbff0153990 0x7fbff00c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff6cc2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff00ad000 con 0x7fbff01506b0 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff6cc2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0156a10 con 0x7fbff00acc00 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff6cc2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff00ad180 con 0x7fbff0153990 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff01506b0 0x7fbff014a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff01506b0 0x7fbff014a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40040/0 (socket says 172.21.15.94:40040) 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 -- 172.21.15.94:0/4095296006 learned_addr learned my addr 172.21.15.94:0/4095296006 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff4a5e640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbff00acc00 0x7fbff01ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff525f640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbff0153990 0x7fbff00c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 -- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbff0153990 msgr2=0x7fbff00c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.634 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbff0153990 0x7fbff00c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 -- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbff00acc00 msgr2=0x7fbff01ecc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbff00acc00 0x7fbff01ecc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 -- 172.21.15.94:0/4095296006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff00c4080 con 0x7fbff01506b0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbff525f640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbff0153990 0x7fbff00c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.630+0000 7fbfeffff640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff01506b0 0x7fbff014a950 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fbfe402bf70 tx=0x7fbfe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.631+0000 7fbfed7fa640 1 -- 172.21.15.94:0/4095296006 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4034040 con 0x7fbff01506b0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.631+0000 7fbfed7fa640 1 -- 172.21.15.94:0/4095296006 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe403ace0 con 0x7fbff01506b0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.631+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff01ed140 con 0x7fbff01506b0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.631+0000 7fbfed7fa640 1 -- 172.21.15.94:0/4095296006 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4044630 con 0x7fbff01506b0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.631+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff01e9750 con 0x7fbff01506b0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.632+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff004f330 con 0x7fbff01506b0 2024-03-06T07:23:39.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.632+0000 7fbfed7fa640 1 -- 172.21.15.94:0/4095296006 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfe4037070 con 0x7fbff01506b0 2024-03-06T07:23:39.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.632+0000 7fbfed7fa640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbfdc06b4d0 0x7fbfdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:39.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.632+0000 7fbfed7fa640 1 -- 172.21.15.94:0/4095296006 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fbfe40ac210 con 0x7fbff01506b0 2024-03-06T07:23:39.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.633+0000 7fbff4a5e640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbfdc06b4d0 0x7fbfdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:39.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.633+0000 7fbff4a5e640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbfdc06b4d0 0x7fbfdc06d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005880 tx=0x7fbfe0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:39.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.637+0000 7fbfed7fa640 1 -- 172.21.15.94:0/4095296006 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbff00c4080 con 0x7fbff01506b0 2024-03-06T07:23:39.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.843+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbff00ad480 con 0x7fbff01506b0 2024-03-06T07:23:39.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.843+0000 7fbfed7fa640 1 -- 172.21.15.94:0/4095296006 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4992 (secure 0 0 0) 0x7fbfe403f070 con 0x7fbff01506b0 2024-03-06T07:23:39.846 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:39.846 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":124,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":120},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:31.523761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7723},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7723":{"gid":7723,"name":"d","rank":0,"incarnation":121,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6836/1362727065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1362727065},{"type":"v1","addr":"172.21.15.145:6836","nonce":1362727065}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7723,"qdb_cluster":[7723]},"id":10}]} 2024-03-06T07:23:39.846 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 124 2024-03-06T07:23:39.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbfdc06b4d0 msgr2=0x7fbfdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbfdc06b4d0 0x7fbfdc06d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005880 tx=0x7fbfe0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:39.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff01506b0 msgr2=0x7fbff014a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff01506b0 0x7fbff014a950 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fbfe402bf70 tx=0x7fbfe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 shutdown_connections 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbfdc06b4d0 0x7fbfdc06d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbff0153990 0x7fbff00c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbff01506b0 0x7fbff014a950 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 --2- 172.21.15.94:0/4095296006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbff00acc00 0x7fbff01ecc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 >> 172.21.15.94:0/4095296006 conn(0x7fbff006b1a0 msgr2=0x7fbff00702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 shutdown_connections 2024-03-06T07:23:39.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:39.845+0000 7fbff6cc2640 1 -- 172.21.15.94:0/4095296006 wait complete. 2024-03-06T07:23:40.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:20.150839+0000', 'modified': '2024-03-06T07:23:31.523761+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7723}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7723': {'gid': 7723, 'name': 'd', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.145:6836/1362727065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1362727065}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1362727065}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7723, 'qdb_cluster': [7723]} 2024-03-06T07:23:40.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:40.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:23:40.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.363+0000 7fb68270c640 1 Processor -- start 2024-03-06T07:23:40.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.363+0000 7fb68270c640 1 -- start start 2024-03-06T07:23:40.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb68270c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb67c06c240 0x7fb67c0c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb68270c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb67c0ab860 0x7fb67c1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb68270c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c1553c0 con 0x7fb67c0ac2b0 2024-03-06T07:23:40.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb68270c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c1556a0 con 0x7fb67c06c240 2024-03-06T07:23:40.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb68270c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c155980 con 0x7fb67c0ab860 2024-03-06T07:23:40.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb67bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb67c06c240 0x7fb67c0c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb67bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb67c06c240 0x7fb67c0c4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40056/0 (socket says 172.21.15.94:40056) 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb67bfff640 1 -- 172.21.15.94:0/3576509358 learned_addr learned my addr 172.21.15.94:0/3576509358 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.364+0000 7fb680ca9640 1 --2- 172.21.15.94:0/3576509358 >> v2:172.21.15.145:3300/0 conn(0x7fb67c0ab860 0x7fb67c1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67bfff640 1 -- 172.21.15.94:0/3576509358 >> v2:172.21.15.145:3300/0 conn(0x7fb67c0ab860 msgr2=0x7fb67c1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3576509358 >> v2:172.21.15.145:3300/0 conn(0x7fb67c0ab860 0x7fb67c1588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67bfff640 1 -- 172.21.15.94:0/3576509358 >> v1:172.21.15.94:6789/0 conn(0x7fb67c0ac2b0 legacy=0x7fb67c0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67affd640 1 -- 172.21.15.94:0/3576509358 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917802736 0 0) 0x7fb67c1553c0 con 0x7fb67c0ac2b0 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67bfff640 1 -- 172.21.15.94:0/3576509358 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c155c60 con 0x7fb67c06c240 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3576509358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c06c240 0x7fb67c0c4250 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb67c06caf0 tx=0x7fb66c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf97892c8d1dbee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67affd640 1 -- 172.21.15.94:0/3576509358 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c02e070 con 0x7fb67c06c240 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.365+0000 7fb67affd640 1 -- 172.21.15.94:0/3576509358 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c037070 con 0x7fb67c06c240 2024-03-06T07:23:40.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb67affd640 1 -- 172.21.15.94:0/3576509358 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c031a00 con 0x7fb67c06c240 2024-03-06T07:23:40.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 -- 172.21.15.94:0/3576509358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c06c240 msgr2=0x7fb67c0c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 --2- 172.21.15.94:0/3576509358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c06c240 0x7fb67c0c4250 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb67c06caf0 tx=0x7fb66c02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:40.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 -- 172.21.15.94:0/3576509358 shutdown_connections 2024-03-06T07:23:40.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 --2- 172.21.15.94:0/3576509358 >> v2:172.21.15.145:3300/0 conn(0x7fb67c0ab860 0x7fb67c1588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 --2- 172.21.15.94:0/3576509358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c06c240 0x7fb67c0c4250 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 -- 172.21.15.94:0/3576509358 >> 172.21.15.94:0/3576509358 conn(0x7fb67c0c7670 msgr2=0x7fb67c0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 -- 172.21.15.94:0/3576509358 shutdown_connections 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.366+0000 7fb68270c640 1 -- 172.21.15.94:0/3576509358 wait complete. 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.367+0000 7fb68270c640 1 Processor -- start 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.367+0000 7fb68270c640 1 -- start start 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb68270c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb67c06c240 0x7fb67c1edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb68270c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c0ab860 0x7fb67c139710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb68270c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb67c0ac2b0 0x7fb67c139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb68270c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0c4830 con 0x7fb67c0ab860 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb68270c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c156180 con 0x7fb67c06c240 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb68270c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c06c620 con 0x7fb67c0ac2b0 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c0ab860 0x7fb67c139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c0ab860 0x7fb67c139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40058/0 (socket says 172.21.15.94:40058) 2024-03-06T07:23:40.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3212429785 learned_addr learned my addr 172.21.15.94:0/3212429785 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb67c06c240 0x7fb67c1edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb680ca9640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb67c0ac2b0 0x7fb67c139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb67c0ac2b0 msgr2=0x7fb67c139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb67c0ac2b0 0x7fb67c139c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb67c06c240 msgr2=0x7fb67c1edbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb67c06c240 0x7fb67c1edbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.368+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3212429785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c1dfec0 con 0x7fb67c0ab860 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.369+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c0ab860 0x7fb67c139710 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb67000c410 tx=0x7fb67000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.369+0000 7fb678ff9640 1 -- 172.21.15.94:0/3212429785 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670017070 con 0x7fb67c0ab860 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.369+0000 7fb678ff9640 1 -- 172.21.15.94:0/3212429785 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6700035f0 con 0x7fb67c0ab860 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.369+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c1dfbe0 con 0x7fb67c0ab860 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.369+0000 7fb678ff9640 1 -- 172.21.15.94:0/3212429785 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6700039e0 con 0x7fb67c0ab860 2024-03-06T07:23:40.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.369+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c1ee770 con 0x7fb67c0ab860 2024-03-06T07:23:40.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.370+0000 7fb678ff9640 1 -- 172.21.15.94:0/3212429785 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb67001a070 con 0x7fb67c0ab860 2024-03-06T07:23:40.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.371+0000 7fb678ff9640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb66806b4d0 0x7fb66806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.371+0000 7fb678ff9640 1 -- 172.21.15.94:0/3212429785 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb6700896f0 con 0x7fb67c0ab860 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.371+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb66806b4d0 0x7fb66806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.371+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb66806b4d0 0x7fb66806d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb66c036000 tx=0x7fb66c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.371+0000 7fb6853d5740 1 Processor -- start 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb6853d5740 1 -- start start 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ad8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47ad8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47ad94b0 0x55ac47ad5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb6853d5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac47a3f6b0 con 0x55ac47ad94b0 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac479e8a60 con 0x55ac47adb330 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac47a3b750 con 0x55ac47adaf50 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb680ca9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47ad94b0 0x55ac47ad5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ad8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb680ca9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47ad94b0 0x55ac47ad5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40062/0 (socket says 172.21.15.94:40062) 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb680ca9640 1 -- 172.21.15.94:0/1432403975 learned_addr learned my addr 172.21.15.94:0/1432403975 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.372+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47ad8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb67bfff640 1 -- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 msgr2=0x55ac47ad8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb67bfff640 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47ad8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb67bfff640 1 -- 172.21.15.94:0/1432403975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47ad94b0 msgr2=0x55ac47ad5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb67bfff640 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47ad94b0 0x55ac47ad5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb67bfff640 1 -- 172.21.15.94:0/1432403975 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac47a06f30 con 0x55ac47adaf50 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb680ca9640 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47ad94b0 0x55ac47ad5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb67bfff640 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ad8a30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb66c033000 tx=0x7fb66c030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63b8e5f47152049 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb661ffb640 1 -- 172.21.15.94:0/1432403975 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c032b40 con 0x55ac47adaf50 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb661ffb640 1 -- 172.21.15.94:0/1432403975 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c037040 con 0x55ac47adaf50 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.373+0000 7fb661ffb640 1 -- 172.21.15.94:0/1432403975 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0453f0 con 0x55ac47adaf50 2024-03-06T07:23:40.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 -- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 msgr2=0x55ac47ad8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ad8a30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb66c033000 tx=0x7fb66c030000 comp rx=0 tx=0).stop 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 -- 172.21.15.94:0/1432403975 shutdown_connections 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47ad94b0 0x55ac47ad5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47ad8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1432403975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ad8a30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 -- 172.21.15.94:0/1432403975 >> 172.21.15.94:0/1432403975 conn(0x55ac47ad77d0 msgr2=0x55ac47afc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 -- 172.21.15.94:0/1432403975 shutdown_connections 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 -- 172.21.15.94:0/1432403975 wait complete. 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 -- start start 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.374+0000 7fb6853d5740 1 Processor -- start 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47b17210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47adb330 0x55ac47affaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47afee60 0x55ac47b13ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb6853d5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac47afd230 con 0x55ac47adb330 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac479fbea0 con 0x55ac47adaf50 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac47afcc70 con 0x55ac47afee60 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb6853d5740 10 client.0 _ll_register_callbacks cb 0x55ac479fb870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47adb330 0x55ac47affaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47b17210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47adb330 0x55ac47affaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40076/0 (socket says 172.21.15.94:40076) 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3242033277 learned_addr learned my addr 172.21.15.94:0/3242033277 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb680ca9640 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47afee60 0x55ac47b13ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47afee60 msgr2=0x55ac47b13ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47afee60 0x55ac47b13ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 msgr2=0x55ac47b17210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47b17210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.375+0000 7fb67b7fe640 1 -- 172.21.15.94:0/3242033277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac479fc270 con 0x55ac47adb330 2024-03-06T07:23:40.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47adb330 0x55ac47affaf0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb6700050a0 tx=0x7fb670057f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121848cba8458189 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb65affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb65affd640 1 -- 172.21.15.94:0/3242033277 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670087e60 con 0x55ac47adb330 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb65affd640 1 -- 172.21.15.94:0/3242033277 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb670003b80 con 0x55ac47adb330 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66c037040 con 0x55ac47adb330 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb65affd640 1 -- 172.21.15.94:0/3242033277 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb670003da0 con 0x55ac47adb330 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 10 client.8583 fetch_fsmap learned FSMap version 124 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55ac477542d0 con 0x55ac47adb330 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb65affd640 1 -- 172.21.15.94:0/3242033277 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c037040 con 0x55ac47adb330 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb65affd640 1 -- 172.21.15.94:0/3242033277 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 124) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb67006bce0 con 0x55ac47adb330 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 10 client.8583 fetch_fsmap finished waiting for FSMap version 124 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 10 client.8583 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 20 client.8583 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b2b530 0x55ac47b2b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 4 client.8583 mds_command: new command op to 7723 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 --> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac47a4cbf0 con 0x55ac47b2b530 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.376+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b2b530 0x55ac47b2b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.377+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b2b530 0x55ac47b2b910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b7cc56a8b185732 server_cookie=99e33ddec45b68db in_seq=0 out_seq=0 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.377+0000 7fb65affd640 10 client.8583 ms_handle_connect on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.377+0000 7fb65affd640 1 -- 172.21.15.94:0/3242033277 <== mds.0 v2:172.21.15.145:6834/1362727065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ac47a4cbf0 con 0x55ac47b2b530 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.377+0000 7fb65affd640 10 client.8583 handle_command_reply: tid=0 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.377+0000 7fb6853d5740 1 client.8583 shutdown 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.377+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47adb330 msgr2=0x55ac47affaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.377+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47adb330 0x55ac47affaf0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb6700050a0 tx=0x7fb670057f10 comp rx=0 tx=0).stop 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 shutdown_connections 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47afee60 0x55ac47b13ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b2b530 0x55ac47b2b910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47adb330 0x55ac47affaf0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb65affd640 0 client.8583 ms_handle_reset on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3242033277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47b17210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 >> 172.21.15.94:0/3242033277 conn(0x55ac47ad94b0 msgr2=0x55ac47ad4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 shutdown_connections 2024-03-06T07:23:40.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 1 -- 172.21.15.94:0/3242033277 wait complete. 2024-03-06T07:23:40.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.378+0000 7fb6853d5740 20 client.8583 trim_cache size 0 max 16384 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.394+0000 7fb6853d5740 1 Processor -- start 2024-03-06T07:23:40.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.394+0000 7fb6853d5740 1 -- start start 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47ad8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adb330 0x55ac47ad8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b24a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6853d5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1f6c50 con 0x55ac47afee60 2024-03-06T07:23:40.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1f6050 con 0x55ac47adaf50 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1f6650 con 0x55ac47adb330 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b24a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b24a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40078/0 (socket says 172.21.15.94:40078) 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 -- 172.21.15.94:0/3318650689 learned_addr learned my addr 172.21.15.94:0/3318650689 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb67bfff640 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47ad8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adb330 0x55ac47ad8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 -- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adb330 msgr2=0x55ac47ad8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adb330 0x55ac47ad8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 -- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 msgr2=0x55ac47ad8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47ad8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 -- 172.21.15.94:0/3318650689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac4781e6b0 con 0x55ac47afee60 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb680ca9640 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b24a30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fb674017780 tx=0x7fb674009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e645d2b04b02b536 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6617fa640 1 -- 172.21.15.94:0/3318650689 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674024070 con 0x55ac47afee60 2024-03-06T07:23:40.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6617fa640 1 -- 172.21.15.94:0/3318650689 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb674015dc0 con 0x55ac47afee60 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.395+0000 7fb6617fa640 1 -- 172.21.15.94:0/3318650689 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67401f650 con 0x55ac47afee60 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 -- 172.21.15.94:0/3318650689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 msgr2=0x55ac47b24a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b24a30 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fb674017780 tx=0x7fb674009750 comp rx=0 tx=0).stop 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 -- 172.21.15.94:0/3318650689 shutdown_connections 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b24a30 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adb330 0x55ac47ad8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 --2- 172.21.15.94:0/3318650689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adaf50 0x55ac47ad8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 -- 172.21.15.94:0/3318650689 >> 172.21.15.94:0/3318650689 conn(0x55ac47ad94b0 msgr2=0x55ac47b01470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 -- 172.21.15.94:0/3318650689 shutdown_connections 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 -- 172.21.15.94:0/3318650689 wait complete. 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.396+0000 7fb6853d5740 1 -- start start 2024-03-06T07:23:40.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 1 Processor -- start 2024-03-06T07:23:40.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ada0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47afd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b84d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac47b01190 con 0x55ac47afee60 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac47b2b3b0 con 0x55ac47adb330 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac47b279e0 con 0x55ac47adaf50 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb6853d5740 10 client.0 _ll_register_callbacks cb 0x7fb670057730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb680ca9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b84d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ada0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb680ca9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b84d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40084/0 (socket says 172.21.15.94:40084) 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb67bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ada0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:48362/0 (socket says 172.21.15.94:48362) 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb680ca9640 1 -- 172.21.15.94:0/1967568012 learned_addr learned my addr 172.21.15.94:0/1967568012 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.397+0000 7fb67b7fe640 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47afd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb680ca9640 1 -- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 msgr2=0x55ac47ada0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb680ca9640 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ada0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb680ca9640 1 -- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 msgr2=0x55ac47afd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb680ca9640 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47afd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb680ca9640 1 -- 172.21.15.94:0/1967568012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac47b17920 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb680ca9640 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b84d30 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb674009cf0 tx=0x7fb674004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5de6055ddd262b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb65affd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb65affd640 1 -- 172.21.15.94:0/1967568012 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674024040 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb65affd640 1 -- 172.21.15.94:0/1967568012 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb674015bc0 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb674015dc0 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb65affd640 1 -- 172.21.15.94:0/1967568012 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=124) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb674036050 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 10 client.8589 fetch_fsmap learned FSMap version 124 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=124,monmap=2+}) v3 -- 0x55ac47b1af10 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb65affd640 1 -- 172.21.15.94:0/1967568012 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb674015dc0 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb65affd640 1 -- 172.21.15.94:0/1967568012 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 124) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fb674039020 con 0x55ac47afee60 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 10 client.8589 fetch_fsmap finished waiting for FSMap version 124 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 10 client.8589 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 20 client.8589 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:40.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b19dc0 0x55ac47b1a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 4 client.8589 mds_command: new command op to 7723 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.398+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 --> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ac47a46f40 con 0x55ac47b19dc0 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb67bfff640 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b19dc0 0x55ac47b1a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb65affd640 10 client.8589 ms_handle_connect on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb67bfff640 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b19dc0 0x55ac47b1a1a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb0780edc728905e server_cookie=40646417fc3e3df0 in_seq=0 out_seq=0 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb65affd640 1 -- 172.21.15.94:0/1967568012 <== mds.0 v2:172.21.15.145:6834/1362727065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55ac47a46f40 con 0x55ac47b19dc0 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb65affd640 10 client.8589 handle_command_reply: tid=0 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 client.8589 shutdown 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 msgr2=0x55ac47b84d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b84d30 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb674009cf0 tx=0x7fb674004ab0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 shutdown_connections 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:6834/1362727065,v1:172.21.15.145:6836/1362727065] conn(0x55ac47b19dc0 0x55ac47b1a1a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ac47afee60 0x55ac47b84d30 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ac47adb330 0x55ac47afd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb65affd640 0 client.8589 ms_handle_reset on v2:172.21.15.145:6834/1362727065 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 --2- 172.21.15.94:0/1967568012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ac47adaf50 0x55ac47ada0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 >> 172.21.15.94:0/1967568012 conn(0x55ac47ad94b0 msgr2=0x55ac47ad8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 shutdown_connections 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 1 -- 172.21.15.94:0/1967568012 wait complete. 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.399+0000 7fb6853d5740 20 client.8589 trim_cache size 0 max 16384 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:23:40.409 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7723, 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 11.894763391, 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 124, 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 136, 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 136, 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 24.680755731000001 2024-03-06T07:23:40.410 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:40.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb66806b4d0 msgr2=0x7fb66806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb66806b4d0 0x7fb66806d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb66c036000 tx=0x7fb66c03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:40.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c0ab860 msgr2=0x7fb67c139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:40.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c0ab860 0x7fb67c139710 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb67000c410 tx=0x7fb67000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 shutdown_connections 2024-03-06T07:23:40.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb66806b4d0 0x7fb66806d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb67c0ac2b0 0x7fb67c139c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb67c0ab860 0x7fb67c139710 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 --2- 172.21.15.94:0/3212429785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb67c06c240 0x7fb67c1edbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:40.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 >> 172.21.15.94:0/3212429785 conn(0x7fb67c0c7670 msgr2=0x7fb67c0c6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:40.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 shutdown_connections 2024-03-06T07:23:40.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:40.400+0000 7fb68270c640 1 -- 172.21.15.94:0/3212429785 wait complete. 2024-03-06T07:23:40.413 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 124, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:20.150839+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7723": { "addr": "172.21.15.145:6836/1362727065", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1362727065, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1362727065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7723, "incarnation": 121, "join_fscid": 10, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 136, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:23:31.523761+0000", "qdb_cluster": [ 7723 ], "qdb_leader": 7723, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7723 } } } ], "standbys": [ { "addr": "172.21.15.145:6837/3826543722", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 3826543722, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 3826543722, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7726, "incarnation": 0, "join_fscid": 10, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 4 }, { "addr": "172.21.15.94:6833/3405703874", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3405703874, "type": "v2" }, { "addr": "172.21.15.94:6833", "nonce": 3405703874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8298, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/801675853", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6834", "nonce": 801675853, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 801675853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 122, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8499, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:40.414 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:40.414 INFO:tasks.cephfs.test_failover:current = [{'gid': 8298, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6833/3405703874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3405703874}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 3405703874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7726, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 4, 'addr': '172.21.15.145:6837/3826543722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3826543722}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3826543722}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 120}, {'gid': 8499, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/801675853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 801675853}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 801675853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 122}, {'gid': 7723, 'name': 'd', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.145:6836/1362727065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1362727065}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1362727065}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-03-06T07:23:40.415 DEBUG:tasks.cephfs.test_failover:'up:active' != 'up:standby' - up:active + up:standby 2024-03-06T07:23:40.415 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:23:42.790 INFO:tasks.ceph.mds.d.smithi145.stderr: -321> 2024-03-06T07:23:36.091+0000 7fb25a748640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:42.814 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:23:45.028 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:45.031 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:45.416 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.525+0000 7f659b281640 1 Processor -- start 2024-03-06T07:23:45.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.526+0000 7f659b281640 1 -- start start 2024-03-06T07:23:45.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.526+0000 7f659b281640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f659414f360 0x7f6594155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.526+0000 7f659b281640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f65940c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.526+0000 7f659b281640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65940c73d0 con 0x7f65940c5880 2024-03-06T07:23:45.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.526+0000 7f659b281640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65940c8dd0 con 0x7f659414f360 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.526+0000 7f659b281640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65940be2e0 con 0x7f65940c88c0 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.526+0000 7f659901d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f659414f360 0x7f6594155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659901d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f659414f360 0x7f6594155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40098/0 (socket says 172.21.15.94:40098) 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659901d640 1 -- 172.21.15.94:0/106586220 learned_addr learned my addr 172.21.15.94:0/106586220 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659881c640 1 --2- 172.21.15.94:0/106586220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f65940c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659901d640 1 -- 172.21.15.94:0/106586220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 msgr2=0x7f65940c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659901d640 1 --2- 172.21.15.94:0/106586220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f65940c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659901d640 1 -- 172.21.15.94:0/106586220 >> v1:172.21.15.145:6789/0 conn(0x7f65940c5880 legacy=0x7f6594158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659901d640 1 -- 172.21.15.94:0/106586220 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940c7670 con 0x7f659414f360 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.527+0000 7f659881c640 1 --2- 172.21.15.94:0/106586220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f65940c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659901d640 1 --2- 172.21.15.94:0/106586220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f6594155c10 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6584009870 tx=0x7f658402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ea0a1278cfaf29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f658bfff640 1 -- 172.21.15.94:0/106586220 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584035020 con 0x7f659414f360 2024-03-06T07:23:45.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f658bfff640 1 -- 172.21.15.94:0/106586220 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6584031640 con 0x7f659414f360 2024-03-06T07:23:45.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f658bfff640 1 -- 172.21.15.94:0/106586220 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658402d040 con 0x7f659414f360 2024-03-06T07:23:45.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 -- 172.21.15.94:0/106586220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 msgr2=0x7f6594155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 --2- 172.21.15.94:0/106586220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f6594155c10 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6584009870 tx=0x7f658402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 -- 172.21.15.94:0/106586220 shutdown_connections 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 --2- 172.21.15.94:0/106586220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f65940c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 --2- 172.21.15.94:0/106586220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f6594155c10 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 -- 172.21.15.94:0/106586220 >> 172.21.15.94:0/106586220 conn(0x7f65940ac570 msgr2=0x7f65940ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 -- 172.21.15.94:0/106586220 shutdown_connections 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.528+0000 7f659b281640 1 -- 172.21.15.94:0/106586220 wait complete. 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 Processor -- start 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 -- start start 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65940c5880 0x7f65941ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f6594157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f65940afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6594159270 con 0x7f659414f360 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940c7c30 con 0x7f65940c5880 2024-03-06T07:23:45.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659b281640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65941593f0 con 0x7f65940c88c0 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659981e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f65940afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659981e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f65940afcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40102/0 (socket says 172.21.15.94:40102) 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659981e640 1 -- 172.21.15.94:0/3325587763 learned_addr learned my addr 172.21.15.94:0/3325587763 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659881c640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f6594157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.530+0000 7f659901d640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65940c5880 0x7f65941ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659981e640 1 -- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 msgr2=0x7f6594157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659981e640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f6594157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659981e640 1 -- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65940c5880 msgr2=0x7f65941ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659981e640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65940c5880 0x7f65941ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659981e640 1 -- 172.21.15.94:0/3325587763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65941eec30 con 0x7f659414f360 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659901d640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65940c5880 0x7f65941ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659881c640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f6594157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659981e640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f65940afcc0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f658c00c7d0 tx=0x7f658c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f6589ffb640 1 -- 172.21.15.94:0/3325587763 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c003b70 con 0x7f659414f360 2024-03-06T07:23:45.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f6589ffb640 1 -- 172.21.15.94:0/3325587763 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c003d30 con 0x7f659414f360 2024-03-06T07:23:45.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f6589ffb640 1 -- 172.21.15.94:0/3325587763 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c00aa90 con 0x7f659414f360 2024-03-06T07:23:45.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65940c0e20 con 0x7f659414f360 2024-03-06T07:23:45.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.531+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65941ebd40 con 0x7f659414f360 2024-03-06T07:23:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.532+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659404f330 con 0x7f659414f360 2024-03-06T07:23:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.532+0000 7f6589ffb640 1 -- 172.21.15.94:0/3325587763 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f658c00ac30 con 0x7f659414f360 2024-03-06T07:23:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.533+0000 7f6589ffb640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f657806b4d0 0x7f657806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.533+0000 7f6589ffb640 1 -- 172.21.15.94:0/3325587763 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f658c08a390 con 0x7f659414f360 2024-03-06T07:23:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.533+0000 7f659901d640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f657806b4d0 0x7f657806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.534+0000 7f659901d640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f657806b4d0 0x7f657806d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6584036000 tx=0x7f658403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.537+0000 7f6589ffb640 1 -- 172.21.15.94:0/3325587763 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f658c058fc0 con 0x7f659414f360 2024-03-06T07:23:45.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.744+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f659406cda0 con 0x7f659414f360 2024-03-06T07:23:45.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.745+0000 7f6589ffb640 1 -- 172.21.15.94:0/3325587763 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4986 (secure 0 0 0) 0x7f658c0588c0 con 0x7f659414f360 2024-03-06T07:23:45.747 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:45.747 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":128,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":36917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1486437600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1486437600},{"type":"v1","addr":"172.21.15.145:6836","nonce":1486437600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:43.788617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7726},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7726":{"gid":7726,"name":"b","rank":0,"incarnation":126,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":10}]} 2024-03-06T07:23:45.747 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 128 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f657806b4d0 msgr2=0x7f657806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f657806b4d0 0x7f657806d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f6584036000 tx=0x7f658403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 msgr2=0x7f65940afcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f65940afcc0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f658c00c7d0 tx=0x7f658c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 shutdown_connections 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f657806b4d0 0x7f657806d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659414f360 0x7f65940afcc0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940c88c0 0x7f6594157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 --2- 172.21.15.94:0/3325587763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65940c5880 0x7f65941ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 >> 172.21.15.94:0/3325587763 conn(0x7f65940ac570 msgr2=0x7f65940ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 shutdown_connections 2024-03-06T07:23:45.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:45.747+0000 7f659b281640 1 -- 172.21.15.94:0/3325587763 wait complete. 2024-03-06T07:23:46.092 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:23:46.090+0000 7efeba6a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:46.162 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:20.150839+0000', 'modified': '2024-03-06T07:23:43.788617+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7726}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7726': {'gid': 7726, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.145:6837/3826543722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3826543722}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3826543722}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:23:46.162 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7726:up:rejoin 2024-03-06T07:23:47.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.273+0000 7fb6a360e640 1 Processor -- start 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.274+0000 7fb6a360e640 1 -- start start 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.274+0000 7fb6a360e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.274+0000 7fb6a360e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb69c158b60 0x7fb69c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.274+0000 7fb6a360e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0c73d0 con 0x7fb69c14f360 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.274+0000 7fb6a360e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69c0c8dd0 con 0x7fb69c158b60 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.274+0000 7fb6a360e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c0be2e0 con 0x7fb69c0c88c0 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.274+0000 7fb6a0ba9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a1bab640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb69c158b60 0x7fb69c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a13aa640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fb69c14f360 0x7fb69c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52586/0 (socket says 172.21.15.94:52586) 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a13aa640 1 -- 172.21.15.94:0/352649261 learned_addr learned my addr 172.21.15.94:0/352649261 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb693fff640 1 -- 172.21.15.94:0/352649261 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213980706 0 0) 0x7fb69c0c73d0 con 0x7fb69c14f360 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb693fff640 1 -- 172.21.15.94:0/352649261 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb688003520 con 0x7fb69c14f360 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a0ba9640 1 -- 172.21.15.94:0/352649261 >> v2:172.21.15.145:3300/0 conn(0x7fb69c158b60 msgr2=0x7fb69c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a0ba9640 1 --2- 172.21.15.94:0/352649261 >> v2:172.21.15.145:3300/0 conn(0x7fb69c158b60 0x7fb69c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a0ba9640 1 -- 172.21.15.94:0/352649261 >> v1:172.21.15.145:6789/0 conn(0x7fb69c14f360 legacy=0x7fb69c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a0ba9640 1 -- 172.21.15.94:0/352649261 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c0c7670 con 0x7fb69c0c88c0 2024-03-06T07:23:47.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.275+0000 7fb6a1bab640 1 --2- 172.21.15.94:0/352649261 >> v2:172.21.15.145:3300/0 conn(0x7fb69c158b60 0x7fb69c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:47.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb6a0ba9640 1 --2- 172.21.15.94:0/352649261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c155c10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0097a0 tx=0x7fb68c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2348b16865cc278d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb693fff640 1 -- 172.21.15.94:0/352649261 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0315a0 con 0x7fb69c0c88c0 2024-03-06T07:23:47.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb693fff640 1 -- 172.21.15.94:0/352649261 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c037070 con 0x7fb69c0c88c0 2024-03-06T07:23:47.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb693fff640 1 -- 172.21.15.94:0/352649261 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c031960 con 0x7fb69c0c88c0 2024-03-06T07:23:47.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb6a360e640 1 -- 172.21.15.94:0/352649261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 msgr2=0x7fb69c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb6a360e640 1 --2- 172.21.15.94:0/352649261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c155c10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0097a0 tx=0x7fb68c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb6a360e640 1 -- 172.21.15.94:0/352649261 shutdown_connections 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb6a360e640 1 --2- 172.21.15.94:0/352649261 >> v2:172.21.15.145:3300/0 conn(0x7fb69c158b60 0x7fb69c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb6a360e640 1 --2- 172.21.15.94:0/352649261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c155c10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.276+0000 7fb6a360e640 1 -- 172.21.15.94:0/352649261 >> 172.21.15.94:0/352649261 conn(0x7fb69c0ac570 msgr2=0x7fb69c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.277+0000 7fb6a360e640 1 -- 172.21.15.94:0/352649261 shutdown_connections 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.277+0000 7fb6a360e640 1 -- 172.21.15.94:0/352649261 wait complete. 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 Processor -- start 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 -- start start 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c1eb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb69c14f360 0x7fb69c0aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb69c158b60 0x7fb69c14b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c0c5a50 con 0x7fb69c14f360 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c0c7c30 con 0x7fb69c158b60 2024-03-06T07:23:47.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a360e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c0c5bd0 con 0x7fb69c0c88c0 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a0ba9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb69c14f360 0x7fb69c0aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a1bab640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb69c158b60 0x7fb69c14b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a0ba9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb69c14f360 0x7fb69c0aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40104/0 (socket says 172.21.15.94:40104) 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a0ba9640 1 -- 172.21.15.94:0/2604746052 learned_addr learned my addr 172.21.15.94:0/2604746052 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.278+0000 7fb6a13aa640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c1eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a1bab640 1 -- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 msgr2=0x7fb69c1eb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a1bab640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c1eb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a1bab640 1 -- 172.21.15.94:0/2604746052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb69c14f360 msgr2=0x7fb69c0aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a1bab640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb69c14f360 0x7fb69c0aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a1bab640 1 -- 172.21.15.94:0/2604746052 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c0c1490 con 0x7fb69c158b60 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a1bab640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb69c158b60 0x7fb69c14b9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb69400e880 tx=0x7fb69400ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb691ffb640 1 -- 172.21.15.94:0/2604746052 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694017070 con 0x7fb69c158b60 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb691ffb640 1 -- 172.21.15.94:0/2604746052 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb694004020 con 0x7fb69c158b60 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c0c0ed0 con 0x7fb69c158b60 2024-03-06T07:23:47.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb691ffb640 1 -- 172.21.15.94:0/2604746052 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6940043f0 con 0x7fb69c158b60 2024-03-06T07:23:47.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.279+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c0ad660 con 0x7fb69c158b60 2024-03-06T07:23:47.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.280+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c04f330 con 0x7fb69c158b60 2024-03-06T07:23:47.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.281+0000 7fb691ffb640 1 -- 172.21.15.94:0/2604746052 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb69400c940 con 0x7fb69c158b60 2024-03-06T07:23:47.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.281+0000 7fb691ffb640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb68006b500 0x7fb68006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.281+0000 7fb691ffb640 1 -- 172.21.15.94:0/2604746052 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fb69408a1f0 con 0x7fb69c158b60 2024-03-06T07:23:47.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.281+0000 7fb6a13aa640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb68006b500 0x7fb68006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.282+0000 7fb6a13aa640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb68006b500 0x7fb68006d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb698005c00 tx=0x7fb698009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.286+0000 7fb691ffb640 1 -- 172.21.15.94:0/2604746052 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb694058750 con 0x7fb69c158b60 2024-03-06T07:23:47.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.494+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb69c06cda0 con 0x7fb69c158b60 2024-03-06T07:23:47.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.494+0000 7fb691ffb640 1 -- 172.21.15.94:0/2604746052 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4994 (secure 0 0 0) 0x7fb69405c7f0 con 0x7fb69c158b60 2024-03-06T07:23:47.496 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:47.497 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":36917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1486437600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1486437600},{"type":"v1","addr":"172.21.15.145:6836","nonce":1486437600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:45.793611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7726},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7726":{"gid":7726,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7726,"qdb_cluster":[7726]},"id":10}]} 2024-03-06T07:23:47.497 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 129 2024-03-06T07:23:47.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb68006b500 msgr2=0x7fb68006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb68006b500 0x7fb68006d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb698005c00 tx=0x7fb698009040 comp rx=0 tx=0).stop 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb69c158b60 msgr2=0x7fb69c14b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb69c158b60 0x7fb69c14b9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb69400e880 tx=0x7fb69400ed40 comp rx=0 tx=0).stop 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 shutdown_connections 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb68006b500 0x7fb68006d9c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb69c158b60 0x7fb69c14b9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb69c14f360 0x7fb69c0aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.496+0000 7fb6a360e640 1 --2- 172.21.15.94:0/2604746052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb69c0c88c0 0x7fb69c1eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.497+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 >> 172.21.15.94:0/2604746052 conn(0x7fb69c0ac570 msgr2=0x7fb69c0c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.497+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 shutdown_connections 2024-03-06T07:23:47.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:47.497+0000 7fb6a360e640 1 -- 172.21.15.94:0/2604746052 wait complete. 2024-03-06T07:23:47.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:20.150839+0000', 'modified': '2024-03-06T07:23:45.793611+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7726}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7726': {'gid': 7726, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.145:6837/3826543722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3826543722}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3826543722}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7726, 'qdb_cluster': [7726]} 2024-03-06T07:23:47.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:47.912 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:23:48.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.013+0000 7fa49db25640 1 Processor -- start 2024-03-06T07:23:48.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.013+0000 7fa49db25640 1 -- start start 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa49db25640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980c69f0 0x7fa4980c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa49db25640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa4980c8950 0x7fa4980acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa49db25640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa498155130 con 0x7fa4980ad2b0 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa49db25640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa498155440 con 0x7fa4980c8950 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa49db25640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c5d50 con 0x7fa4980c69f0 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa496ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa4980c8950 0x7fa4980acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa496ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa4980c8950 0x7fa4980acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40110/0 (socket says 172.21.15.94:40110) 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa496ffd640 1 -- 172.21.15.94:0/1205351016 learned_addr learned my addr 172.21.15.94:0/1205351016 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.014+0000 7fa4977fe640 1 --2- 172.21.15.94:0/1205351016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980c69f0 0x7fa4980c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa496ffd640 1 -- 172.21.15.94:0/1205351016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980c69f0 msgr2=0x7fa4980c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa496ffd640 1 --2- 172.21.15.94:0/1205351016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980c69f0 0x7fa4980c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa496ffd640 1 -- 172.21.15.94:0/1205351016 >> v1:172.21.15.145:6789/0 conn(0x7fa4980ad2b0 legacy=0x7fa4981558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa496ffd640 1 -- 172.21.15.94:0/1205351016 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4980c6030 con 0x7fa4980c8950 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa4967fc640 1 -- 172.21.15.94:0/1205351016 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154870673 0 0) 0x7fa490001110 con 0x7fa4980ad2b0 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa496ffd640 1 --2- 172.21.15.94:0/1205351016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c8950 0x7fa4980acd70 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fa480009870 tx=0x7fa48002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774bc234c04904ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa4967fc640 1 -- 172.21.15.94:0/1205351016 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480035020 con 0x7fa4980c8950 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa4967fc640 1 -- 172.21.15.94:0/1205351016 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480031640 con 0x7fa4980c8950 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.015+0000 7fa4967fc640 1 -- 172.21.15.94:0/1205351016 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48002d040 con 0x7fa4980c8950 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 -- 172.21.15.94:0/1205351016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c8950 msgr2=0x7fa4980acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 --2- 172.21.15.94:0/1205351016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c8950 0x7fa4980acd70 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fa480009870 tx=0x7fa48002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 -- 172.21.15.94:0/1205351016 shutdown_connections 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 --2- 172.21.15.94:0/1205351016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c8950 0x7fa4980acd70 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 --2- 172.21.15.94:0/1205351016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980c69f0 0x7fa4980c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 -- 172.21.15.94:0/1205351016 >> 172.21.15.94:0/1205351016 conn(0x7fa4980cae70 msgr2=0x7fa4980cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 -- 172.21.15.94:0/1205351016 shutdown_connections 2024-03-06T07:23:48.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.016+0000 7fa49db25640 1 -- 172.21.15.94:0/1205351016 wait complete. 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.017+0000 7fa49db25640 1 Processor -- start 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.017+0000 7fa49db25640 1 -- start start 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa49db25640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980ad2b0 0x7fa49806c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa49db25640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c69f0 0x7fa4980ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa49db25640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4980c8950 0x7fa4980adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa49db25640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa498152a20 con 0x7fa4980c69f0 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa49db25640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa498155fe0 con 0x7fa4980c8950 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa49db25640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa498152ba0 con 0x7fa4980ad2b0 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c69f0 0x7fa4980ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c69f0 0x7fa4980ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40124/0 (socket says 172.21.15.94:40124) 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 -- 172.21.15.94:0/3390015554 learned_addr learned my addr 172.21.15.94:0/3390015554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa4977fe640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980ad2b0 0x7fa49806c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa497fff640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4980c8950 0x7fa4980adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 -- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980ad2b0 msgr2=0x7fa49806c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980ad2b0 0x7fa49806c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 -- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4980c8950 msgr2=0x7fa4980adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4980c8950 0x7fa4980adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.018+0000 7fa496ffd640 1 -- 172.21.15.94:0/3390015554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4981585a0 con 0x7fa4980c69f0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.019+0000 7fa496ffd640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c69f0 0x7fa4980ad770 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fa48002b930 tx=0x7fa4800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.019+0000 7fa48ffff640 1 -- 172.21.15.94:0/3390015554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480040070 con 0x7fa4980c69f0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.019+0000 7fa48ffff640 1 -- 172.21.15.94:0/3390015554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48003e070 con 0x7fa4980c69f0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.019+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4980afb30 con 0x7fa4980c69f0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.019+0000 7fa48ffff640 1 -- 172.21.15.94:0/3390015554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480035040 con 0x7fa4980c69f0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.019+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4980b00f0 con 0x7fa4980c69f0 2024-03-06T07:23:48.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.020+0000 7fa48ffff640 1 -- 172.21.15.94:0/3390015554 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa480038020 con 0x7fa4980c69f0 2024-03-06T07:23:48.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.020+0000 7fa48ffff640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47806b500 0x7fa47806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.021+0000 7fa48ffff640 1 -- 172.21.15.94:0/3390015554 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa4800abcb0 con 0x7fa4980c69f0 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.021+0000 7fa4977fe640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47806b500 0x7fa47806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.021+0000 7fa4977fe640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47806b500 0x7fa47806d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa4880113e0 tx=0x7fa48800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.021+0000 7fa4a07ec740 1 Processor -- start 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.021+0000 7fa4a07ec740 1 -- start start 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599a3330 0x5652599a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a14b0 0x56525999d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652599076b0 con 0x5652599a3330 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652598b0a60 con 0x5652599a2f50 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x565259903750 con 0x5652599a14b0 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa496ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599a3330 0x5652599a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa496ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599a3330 0x5652599a0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40138/0 (socket says 172.21.15.94:40138) 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa496ffd640 1 -- 172.21.15.94:0/2417364103 learned_addr learned my addr 172.21.15.94:0/2417364103 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa4977fe640 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.022+0000 7fa497fff640 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a14b0 0x56525999d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa496ffd640 1 -- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a14b0 msgr2=0x56525999d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa496ffd640 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a14b0 0x56525999d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa496ffd640 1 -- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 msgr2=0x5652599a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa496ffd640 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa496ffd640 1 -- 172.21.15.94:0/2417364103 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652598cef30 con 0x5652599a3330 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa4977fe640 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa496ffd640 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599a3330 0x5652599a0f70 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fa480076670 tx=0x7fa48007a350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cbf75ef68e8b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa48d7fa640 1 -- 172.21.15.94:0/2417364103 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800a8070 con 0x5652599a3330 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa48d7fa640 1 -- 172.21.15.94:0/2417364103 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480076880 con 0x5652599a3330 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa48d7fa640 1 -- 172.21.15.94:0/2417364103 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480039e10 con 0x5652599a3330 2024-03-06T07:23:48.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2417364103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599a3330 msgr2=0x5652599a0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.023+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599a3330 0x5652599a0f70 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fa480076670 tx=0x7fa48007a350 comp rx=0 tx=0).stop 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2417364103 shutdown_connections 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a14b0 0x56525999d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599a3330 0x5652599a0f70 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2417364103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2417364103 >> 172.21.15.94:0/2417364103 conn(0x56525999f7d0 msgr2=0x5652599c43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2417364103 shutdown_connections 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2417364103 wait complete. 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 -- start start 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.024+0000 7fa4a07ec740 1 Processor -- start 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599f3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a3330 0x5652599c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652599076b0 con 0x5652599c38f0 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652598c3ea0 con 0x5652599a2f50 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652599c5230 con 0x5652599a3330 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4a07ec740 10 client.0 _ll_register_callbacks cb 0x5652598c3870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4977fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40140/0 (socket says 172.21.15.94:40140) 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 -- 172.21.15.94:0/2955776212 learned_addr learned my addr 172.21.15.94:0/2955776212 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa496ffd640 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a3330 0x5652599c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 -- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a3330 msgr2=0x5652599c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a3330 0x5652599c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 -- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 msgr2=0x5652599f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599f3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 -- 172.21.15.94:0/2955776212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652598c4270 con 0x5652599c38f0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4977fe640 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa497fff640 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599c5690 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fa4900122f0 tx=0x7fa4900127b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2393fd1f034eafd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4767fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.025+0000 7fa4767fc640 1 -- 172.21.15.94:0/2955776212 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490005320 con 0x5652599c38f0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4767fc640 1 -- 172.21.15.94:0/2955776212 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa4900189c0 con 0x5652599c38f0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa480076880 con 0x5652599c38f0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4767fc640 1 -- 172.21.15.94:0/2955776212 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa490018bc0 con 0x5652599c38f0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 10 client.8613 fetch_fsmap learned FSMap version 129 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x56525961c2d0 con 0x5652599c38f0 2024-03-06T07:23:48.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4767fc640 1 -- 172.21.15.94:0/2955776212 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480076880 con 0x5652599c38f0 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4767fc640 1 -- 172.21.15.94:0/2955776212 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 129) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa490007050 con 0x5652599c38f0 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 10 client.8613 fetch_fsmap finished waiting for FSMap version 129 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 10 client.8613 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 20 client.8613 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c6030 0x5652599fe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 4 client.8613 mds_command: new command op to 7726 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 --> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565259914bf0 con 0x5652599c6030 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.026+0000 7fa4977fe640 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c6030 0x5652599fe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4767fc640 10 client.8613 ms_handle_connect on v2:172.21.15.145:6835/3826543722 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4977fe640 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c6030 0x5652599fe1d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b488ccc8307c370 server_cookie=c006ef7c8e9144f3 in_seq=0 out_seq=0 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4767fc640 1 -- 172.21.15.94:0/2955776212 <== mds.0 v2:172.21.15.145:6835/3826543722 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565259914bf0 con 0x5652599c6030 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4767fc640 10 client.8613 handle_command_reply: tid=0 2024-03-06T07:23:48.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 client.8613 shutdown 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 msgr2=0x5652599c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599c5690 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fa4900122f0 tx=0x7fa4900127b0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 shutdown_connections 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599c5690 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a3330 0x5652599c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c6030 0x5652599fe1d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2955776212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a2f50 0x5652599f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4767fc640 0 client.8613 ms_handle_reset on v2:172.21.15.145:6835/3826543722 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 >> 172.21.15.94:0/2955776212 conn(0x5652599a14b0 msgr2=0x5652599a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 shutdown_connections 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.027+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2955776212 wait complete. 2024-03-06T07:23:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.028+0000 7fa4a07ec740 20 client.8613 trim_cache size 0 max 16384 2024-03-06T07:23:48.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.044+0000 7fa4a07ec740 1 Processor -- start 2024-03-06T07:23:48.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.044+0000 7fa4a07ec740 1 -- start start 2024-03-06T07:23:48.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x5652599ca400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4981f7f10 con 0x5652599c38f0 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4981f70b0 con 0x5652599a3330 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4981f6500 con 0x5652599a2f50 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40148/0 (socket says 172.21.15.94:40148) 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 -- 172.21.15.94:0/2938431370 learned_addr learned my addr 172.21.15.94:0/2938431370 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa4977fe640 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa496ffd640 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x5652599ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 -- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 msgr2=0x5652599dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599dc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 -- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 msgr2=0x5652599ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x5652599ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.045+0000 7fa497fff640 1 -- 172.21.15.94:0/2938431370 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652596e66b0 con 0x5652599c38f0 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa497fff640 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599ca940 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fa4900050e0 tx=0x7fa49000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747b547e47e4e7cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa48cff9640 1 -- 172.21.15.94:0/2938431370 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49002d070 con 0x5652599c38f0 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa48cff9640 1 -- 172.21.15.94:0/2938431370 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4900075a0 con 0x5652599c38f0 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa48cff9640 1 -- 172.21.15.94:0/2938431370 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa490003810 con 0x5652599c38f0 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2938431370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 msgr2=0x5652599ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599ca940 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fa4900050e0 tx=0x7fa49000f000 comp rx=0 tx=0).stop 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2938431370 shutdown_connections 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599ca940 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x5652599ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/2938431370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2938431370 >> 172.21.15.94:0/2938431370 conn(0x5652599a14b0 msgr2=0x565259998c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2938431370 shutdown_connections 2024-03-06T07:23:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.046+0000 7fa4a07ec740 1 -- 172.21.15.94:0/2938431370 wait complete. 2024-03-06T07:23:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.047+0000 7fa4a07ec740 1 -- start start 2024-03-06T07:23:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.047+0000 7fa4a07ec740 1 Processor -- start 2024-03-06T07:23:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599db240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x565259a02680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652599c5eb0 con 0x5652599c38f0 2024-03-06T07:23:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4981f6ab0 con 0x5652599a3330 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652599c41f0 con 0x5652599a2f50 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 10 client.0 _ll_register_callbacks cb 0x7fa490001ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa496ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x565259a02680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40154/0 (socket says 172.21.15.94:40154) 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 -- 172.21.15.94:0/3314020315 learned_addr learned my addr 172.21.15.94:0/3314020315 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4977fe640 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599db240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 -- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 msgr2=0x5652599db240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599db240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 -- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 msgr2=0x565259a02680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x565259a02680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 -- 172.21.15.94:0/3314020315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56525999ccd0 con 0x5652599c38f0 2024-03-06T07:23:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa497fff640 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599a0cf0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fa490007570 tx=0x7fa490000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ceab7d65813f5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4767fc640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4767fc640 1 -- 172.21.15.94:0/3314020315 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49002d040 con 0x5652599c38f0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4767fc640 1 -- 172.21.15.94:0/3314020315 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa490005320 con 0x5652599c38f0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4900075a0 con 0x5652599c38f0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4767fc640 1 -- 172.21.15.94:0/3314020315 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa49003e050 con 0x5652599c38f0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 10 client.8619 fetch_fsmap learned FSMap version 129 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.048+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x5652599db810 con 0x5652599c38f0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4767fc640 1 -- 172.21.15.94:0/3314020315 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4900075a0 con 0x5652599c38f0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4767fc640 1 -- 172.21.15.94:0/3314020315 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 129) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa490041020 con 0x5652599c38f0 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4a07ec740 10 client.8619 fetch_fsmap finished waiting for FSMap version 129 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4a07ec740 10 client.8619 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4a07ec740 20 client.8619 populate_metadata read hostname 'smithi094' 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c1f00 0x5652599c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4a07ec740 4 client.8619 mds_command: new command op to 7726 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 --> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56525990ef40 con 0x5652599c1f00 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4977fe640 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c1f00 0x5652599c22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4767fc640 10 client.8619 ms_handle_connect on v2:172.21.15.145:6835/3826543722 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4977fe640 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c1f00 0x5652599c22e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1c4afbba8e1656 server_cookie=57747a09e221d27 in_seq=0 out_seq=0 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4767fc640 1 -- 172.21.15.94:0/3314020315 <== mds.0 v2:172.21.15.145:6835/3826543722 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x56525990ef40 con 0x5652599c1f00 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4767fc640 10 client.8619 handle_command_reply: tid=0 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.049+0000 7fa4a07ec740 1 client.8619 shutdown 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 msgr2=0x5652599a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599a0cf0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fa490007570 tx=0x7fa490000ed0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 shutdown_connections 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5652599c38f0 0x5652599a0cf0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:6835/3826543722,v1:172.21.15.145:6837/3826543722] conn(0x5652599c1f00 0x5652599c22e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5652599a3330 0x565259a02680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 --2- 172.21.15.94:0/3314020315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5652599a2f50 0x5652599db240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4767fc640 0 client.8619 ms_handle_reset on v2:172.21.15.145:6835/3826543722 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 >> 172.21.15.94:0/3314020315 conn(0x5652599a14b0 msgr2=0x5652599a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 shutdown_connections 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 1 -- 172.21.15.94:0/3314020315 wait complete. 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa4a07ec740 20 client.8619 trim_cache size 0 max 16384 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:23:48.059 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 7726, 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 5.2667191960000004, 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 129, 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 137, 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 137, 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 32.330056911 2024-03-06T07:23:48.060 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:23:48.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47806b500 msgr2=0x7fa47806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47806b500 0x7fa47806d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa4880113e0 tx=0x7fa48800b040 comp rx=0 tx=0).stop 2024-03-06T07:23:48.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c69f0 msgr2=0x7fa4980ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c69f0 0x7fa4980ad770 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fa48002b930 tx=0x7fa4800046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 shutdown_connections 2024-03-06T07:23:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47806b500 0x7fa47806d9c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4980c8950 0x7fa4980adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4980c69f0 0x7fa4980ad770 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 --2- 172.21.15.94:0/3390015554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4980ad2b0 0x7fa49806c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 >> 172.21.15.94:0/3390015554 conn(0x7fa4980cae70 msgr2=0x7fa4981558d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 shutdown_connections 2024-03-06T07:23:48.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.050+0000 7fa49db25640 1 -- 172.21.15.94:0/3390015554 wait complete. 2024-03-06T07:23:48.063 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 129, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:23:20.150839+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7726": { "addr": "172.21.15.145:6837/3826543722", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 3826543722, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 3826543722, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7726, "incarnation": 126, "join_fscid": 10, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 137, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-03-06T07:23:45.793611+0000", "qdb_cluster": [ 7726 ], "qdb_leader": 7726, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7726 } } } ], "standbys": [ { "addr": "172.21.15.94:6833/3405703874", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 3405703874, "type": "v2" }, { "addr": "172.21.15.94:6833", "nonce": 3405703874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8298, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/801675853", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6834", "nonce": 801675853, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 801675853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 122, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8499, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6836/1486437600", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1486437600, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1486437600, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 127, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 36917, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:23:48.064 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:23:48.064 INFO:tasks.cephfs.test_failover:current = [{'gid': 8298, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6833/3405703874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 3405703874}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 3405703874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7726, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.145:6837/3826543722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 3826543722}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3826543722}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8499, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/801675853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 801675853}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 801675853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 122}, {'gid': 36917, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/1486437600', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1486437600}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1486437600}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 127}] 2024-03-06T07:23:48.064 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:23:48.064 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:48.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:48.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:48.162 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:48.162 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:23:48.162 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:48.177 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:23:48.178 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:23:48.178 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:48.218 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:23:48.218 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:48.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.315+0000 7fcc97439640 1 Processor -- start 2024-03-06T07:23:48.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.315+0000 7fcc97439640 1 -- start start 2024-03-06T07:23:48.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc97439640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc90155ba0 0x7fcc900c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc97439640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcc901586f0 0x7fcc90158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc97439640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc901591c0 con 0x7fcc900c5480 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc97439640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9006df40 con 0x7fcc901586f0 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc97439640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9006fdb0 con 0x7fcc90155ba0 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc959d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcc901586f0 0x7fcc90158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc959d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcc901586f0 0x7fcc90158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40156/0 (socket says 172.21.15.94:40156) 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc959d6640 1 -- 172.21.15.94:0/370566832 learned_addr learned my addr 172.21.15.94:0/370566832 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.316+0000 7fcc949d4640 1 --2- 172.21.15.94:0/370566832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc90155ba0 0x7fcc900c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc959d6640 1 -- 172.21.15.94:0/370566832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc90155ba0 msgr2=0x7fcc900c5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc959d6640 1 --2- 172.21.15.94:0/370566832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc90155ba0 0x7fcc900c5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc959d6640 1 -- 172.21.15.94:0/370566832 >> v1:172.21.15.145:6789/0 conn(0x7fcc900c5480 legacy=0x7fcc9006f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc959d6640 1 -- 172.21.15.94:0/370566832 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc901594a0 con 0x7fcc901586f0 2024-03-06T07:23:48.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc87fff640 1 -- 172.21.15.94:0/370566832 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661551778 0 0) 0x7fcc901591c0 con 0x7fcc900c5480 2024-03-06T07:23:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc959d6640 1 --2- 172.21.15.94:0/370566832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc901586f0 0x7fcc90158b80 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009870 tx=0x7fcc8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cb7859b9685250 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc87fff640 1 -- 172.21.15.94:0/370566832 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88035020 con 0x7fcc901586f0 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.317+0000 7fcc87fff640 1 -- 172.21.15.94:0/370566832 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc88031640 con 0x7fcc901586f0 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc87fff640 1 -- 172.21.15.94:0/370566832 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8802d040 con 0x7fcc901586f0 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 -- 172.21.15.94:0/370566832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc901586f0 msgr2=0x7fcc90158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 --2- 172.21.15.94:0/370566832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc901586f0 0x7fcc90158b80 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009870 tx=0x7fcc8802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 -- 172.21.15.94:0/370566832 shutdown_connections 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 --2- 172.21.15.94:0/370566832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc901586f0 0x7fcc90158b80 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 --2- 172.21.15.94:0/370566832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc90155ba0 0x7fcc900c5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 -- 172.21.15.94:0/370566832 >> 172.21.15.94:0/370566832 conn(0x7fcc9006b1a0 msgr2=0x7fcc9006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 -- 172.21.15.94:0/370566832 shutdown_connections 2024-03-06T07:23:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.318+0000 7fcc97439640 1 -- 172.21.15.94:0/370566832 wait complete. 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.319+0000 7fcc97439640 1 Processor -- start 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.319+0000 7fcc97439640 1 -- start start 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc97439640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c5480 0x7fcc901e13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc97439640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc90155ba0 0x7fcc901e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc97439640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901586f0 0x7fcc901eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc97439640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90155680 con 0x7fcc900c5480 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc97439640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90156000 con 0x7fcc90155ba0 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc97439640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc90152a20 con 0x7fcc901586f0 2024-03-06T07:23:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c5480 0x7fcc901e13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c5480 0x7fcc901e13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40160/0 (socket says 172.21.15.94:40160) 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 -- 172.21.15.94:0/427167780 learned_addr learned my addr 172.21.15.94:0/427167780 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc959d6640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901586f0 0x7fcc901eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc949d4640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc90155ba0 0x7fcc901e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 -- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901586f0 msgr2=0x7fcc901eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901586f0 0x7fcc901eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 -- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc90155ba0 msgr2=0x7fcc901e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc90155ba0 0x7fcc901e18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 -- 172.21.15.94:0/427167780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900c4010 con 0x7fcc900c5480 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.320+0000 7fcc951d5640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c5480 0x7fcc901e13b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00c890 tx=0x7fcc8c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.321+0000 7fcc85ffb640 1 -- 172.21.15.94:0/427167780 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c017070 con 0x7fcc900c5480 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.321+0000 7fcc85ffb640 1 -- 172.21.15.94:0/427167780 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8c003dc0 con 0x7fcc900c5480 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.321+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc901e22e0 con 0x7fcc900c5480 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.321+0000 7fcc85ffb640 1 -- 172.21.15.94:0/427167780 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c0052b0 con 0x7fcc900c5480 2024-03-06T07:23:48.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.321+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc900c3810 con 0x7fcc900c5480 2024-03-06T07:23:48.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.322+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9004f330 con 0x7fcc900c5480 2024-03-06T07:23:48.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.322+0000 7fcc85ffb640 1 -- 172.21.15.94:0/427167780 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc8c01a070 con 0x7fcc900c5480 2024-03-06T07:23:48.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.322+0000 7fcc85ffb640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7406b4d0 0x7fcc7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.323+0000 7fcc85ffb640 1 -- 172.21.15.94:0/427167780 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fcc8c089a00 con 0x7fcc900c5480 2024-03-06T07:23:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.323+0000 7fcc949d4640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7406b4d0 0x7fcc7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.323+0000 7fcc949d4640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7406b4d0 0x7fcc7406d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcc900c9700 tx=0x7fcc80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.327+0000 7fcc85ffb640 1 -- 172.21.15.94:0/427167780 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc8c058080 con 0x7fcc900c5480 2024-03-06T07:23:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.534+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc9006cfb0 con 0x7fcc900c5480 2024-03-06T07:23:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.535+0000 7fcc85ffb640 1 -- 172.21.15.94:0/427167780 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4994 (secure 0 0 0) 0x7fcc8c05c120 con 0x7fcc900c5480 2024-03-06T07:23:48.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:48.537 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":36917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1486437600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1486437600},{"type":"v1","addr":"172.21.15.145:6836","nonce":1486437600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:45.793611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7726},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7726":{"gid":7726,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7726,"qdb_cluster":[7726]},"id":10}]} 2024-03-06T07:23:48.537 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 129 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7406b4d0 msgr2=0x7fcc7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7406b4d0 0x7fcc7406d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcc900c9700 tx=0x7fcc80009190 comp rx=0 tx=0).stop 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c5480 msgr2=0x7fcc901e13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c5480 0x7fcc901e13b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00c890 tx=0x7fcc8c00cd50 comp rx=0 tx=0).stop 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 shutdown_connections 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc7406b4d0 0x7fcc7406d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc901586f0 0x7fcc901eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc90155ba0 0x7fcc901e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 --2- 172.21.15.94:0/427167780 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc900c5480 0x7fcc901e13b0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 >> 172.21.15.94:0/427167780 conn(0x7fcc9006b1a0 msgr2=0x7fcc9006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 shutdown_connections 2024-03-06T07:23:48.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.537+0000 7fcc97439640 1 -- 172.21.15.94:0/427167780 wait complete. 2024-03-06T07:23:48.550 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9189847640 1 Processor -- start 2024-03-06T07:23:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9189847640 1 -- start start 2024-03-06T07:23:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9189847640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f91840c20b0 0x7f918406b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9189847640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91840c25c0 0x7f9184153890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9189847640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9184153dd0 con 0x7f918406afc0 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9189847640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91840a9530 con 0x7f91840c25c0 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9189847640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91840a9810 con 0x7f91840c20b0 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f91837fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91840c25c0 0x7f9184153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f91837fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91840c25c0 0x7f9184153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38920/0 (socket says 172.21.15.94:38920) 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f9182ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f918406afc0 0x7f9184153180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47732/0 (socket says 172.21.15.94:47732) 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.648+0000 7f91837fe640 1 -- 172.21.15.94:0/885481077 learned_addr learned my addr 172.21.15.94:0/885481077 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.649+0000 7f91827fc640 1 --2- 172.21.15.94:0/885481077 >> v2:172.21.15.145:3300/0 conn(0x7f91840c20b0 0x7f918406b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.649+0000 7f91837fe640 1 -- 172.21.15.94:0/885481077 >> v2:172.21.15.145:3300/0 conn(0x7f91840c20b0 msgr2=0x7f918406b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.649+0000 7f91837fe640 1 --2- 172.21.15.94:0/885481077 >> v2:172.21.15.145:3300/0 conn(0x7f91840c20b0 0x7f918406b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.649+0000 7f91837fe640 1 -- 172.21.15.94:0/885481077 >> v1:172.21.15.94:6789/0 conn(0x7f918406afc0 legacy=0x7f9184153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.649+0000 7f91837fe640 1 -- 172.21.15.94:0/885481077 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91840a9af0 con 0x7f91840c25c0 2024-03-06T07:23:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.649+0000 7f9181ffb640 1 -- 172.21.15.94:0/885481077 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673743916 0 0) 0x7f9184153dd0 con 0x7f918406afc0 2024-03-06T07:23:48.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.649+0000 7f91827fc640 1 --2- 172.21.15.94:0/885481077 >> v2:172.21.15.145:3300/0 conn(0x7f91840c20b0 0x7f918406b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:48.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f91837fe640 1 --2- 172.21.15.94:0/885481077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c25c0 0x7f9184153890 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9184067830 tx=0x7f917402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5263375315fed219 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9181ffb640 1 -- 172.21.15.94:0/885481077 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917402e070 con 0x7f91840c25c0 2024-03-06T07:23:48.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9181ffb640 1 -- 172.21.15.94:0/885481077 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9174037070 con 0x7f91840c25c0 2024-03-06T07:23:48.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9181ffb640 1 -- 172.21.15.94:0/885481077 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9174031a00 con 0x7f91840c25c0 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 -- 172.21.15.94:0/885481077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c25c0 msgr2=0x7f9184153890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 --2- 172.21.15.94:0/885481077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c25c0 0x7f9184153890 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9184067830 tx=0x7f917402b880 comp rx=0 tx=0).stop 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 -- 172.21.15.94:0/885481077 shutdown_connections 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 --2- 172.21.15.94:0/885481077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c25c0 0x7f9184153890 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 --2- 172.21.15.94:0/885481077 >> v2:172.21.15.145:3300/0 conn(0x7f91840c20b0 0x7f918406b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 -- 172.21.15.94:0/885481077 >> 172.21.15.94:0/885481077 conn(0x7f91840c76b0 msgr2=0x7f91840c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 -- 172.21.15.94:0/885481077 shutdown_connections 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.650+0000 7f9189847640 1 -- 172.21.15.94:0/885481077 wait complete. 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.651+0000 7f9189847640 1 Processor -- start 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9189847640 1 -- start start 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9189847640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f918406afc0 0x7f9184156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9189847640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c20b0 0x7f9184149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9189847640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91840c25c0 0x7f91840c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9189847640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9184150980 con 0x7f91840c20b0 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9189847640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91840aa0b0 con 0x7f91840c25c0 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9189847640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9184150b00 con 0x7f918406afc0 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f91827fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c20b0 0x7f9184149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f91827fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c20b0 0x7f9184149f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38936/0 (socket says 172.21.15.94:38936) 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f91827fc640 1 -- 172.21.15.94:0/418657897 learned_addr learned my addr 172.21.15.94:0/418657897 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f91837fe640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91840c25c0 0x7f91840c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.652+0000 7f9182ffd640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f918406afc0 0x7f9184156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f91827fc640 1 -- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f918406afc0 msgr2=0x7f9184156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f91827fc640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f918406afc0 0x7f9184156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f91827fc640 1 -- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91840c25c0 msgr2=0x7f91840c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f91827fc640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91840c25c0 0x7f91840c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f91827fc640 1 -- 172.21.15.94:0/418657897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184154270 con 0x7f91840c20b0 2024-03-06T07:23:48.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f91827fc640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c20b0 0x7f9184149f10 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f916c012fc0 tx=0x7f916c010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f917bfff640 1 -- 172.21.15.94:0/418657897 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c010da0 con 0x7f91840c20b0 2024-03-06T07:23:48.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f917bfff640 1 -- 172.21.15.94:0/418657897 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916c01d070 con 0x7f91840c20b0 2024-03-06T07:23:48.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9184154550 con 0x7f91840c20b0 2024-03-06T07:23:48.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f917bfff640 1 -- 172.21.15.94:0/418657897 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c0186c0 con 0x7f91840c20b0 2024-03-06T07:23:48.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.653+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91841e6eb0 con 0x7f91840c20b0 2024-03-06T07:23:48.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.654+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918404f330 con 0x7f91840c20b0 2024-03-06T07:23:48.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.654+0000 7f917bfff640 1 -- 172.21.15.94:0/418657897 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f916c016050 con 0x7f91840c20b0 2024-03-06T07:23:48.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.655+0000 7f917bfff640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916406b4d0 0x7f916406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.655+0000 7f917bfff640 1 -- 172.21.15.94:0/418657897 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f916c08de10 con 0x7f91840c20b0 2024-03-06T07:23:48.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.655+0000 7f9182ffd640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916406b4d0 0x7f916406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.655+0000 7f9182ffd640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916406b4d0 0x7f916406d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f917c0012e0 tx=0x7f917c001680 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.659+0000 7f917bfff640 1 -- 172.21.15.94:0/418657897 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f916c014040 con 0x7f91840c20b0 2024-03-06T07:23:48.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.832+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91840c1a50 con 0x7f91840c20b0 2024-03-06T07:23:48.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.834+0000 7f917bfff640 1 -- 172.21.15.94:0/418657897 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+17151 (secure 0 0 0) 0x7f916c05c490 con 0x7f91840c20b0 2024-03-06T07:23:48.836 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:48.836 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":137,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:42.782345+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:18.382044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:19.127404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.145:6834/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.94:6835/392019985":"2024-03-07T07:23:28.504159+0000","172.21.15.94:6834/392019985":"2024-03-07T07:23:28.504159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916406b4d0 msgr2=0x7f916406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916406b4d0 0x7f916406d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f917c0012e0 tx=0x7f917c001680 comp rx=0 tx=0).stop 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c20b0 msgr2=0x7f9184149f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c20b0 0x7f9184149f10 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f916c012fc0 tx=0x7f916c010490 comp rx=0 tx=0).stop 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 shutdown_connections 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f916406b4d0 0x7f916406d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91840c25c0 0x7f91840c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91840c20b0 0x7f9184149f10 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 --2- 172.21.15.94:0/418657897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f918406afc0 0x7f9184156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 >> 172.21.15.94:0/418657897 conn(0x7f91840c76b0 msgr2=0x7f91840c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 shutdown_connections 2024-03-06T07:23:48.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.836+0000 7f9189847640 1 -- 172.21.15.94:0/418657897 wait complete. 2024-03-06T07:23:48.849 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:18.382044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:19.127404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:48.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.950+0000 7f105fd55640 1 Processor -- start 2024-03-06T07:23:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105fd55640 1 -- start start 2024-03-06T07:23:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105fd55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f10580c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105fd55640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f10580ab170 0x7f105806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105fd55640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580c3ef0 con 0x7f105806c730 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105fd55640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105806cbe0 con 0x7f10580ab170 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105fd55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10581556d0 con 0x7f10580c5990 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105e2f2640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f105806c730 0x7f10581523f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47740/0 (socket says 172.21.15.94:47740) 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.951+0000 7f105e2f2640 1 -- 172.21.15.94:0/3440740182 learned_addr learned my addr 172.21.15.94:0/3440740182 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105d2f0640 1 --2- 172.21.15.94:0/3440740182 >> v2:172.21.15.145:3300/0 conn(0x7f10580ab170 0x7f105806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105daf1640 1 --2- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f10580c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105caef640 1 -- 172.21.15.94:0/3440740182 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921495945 0 0) 0x7f10580c3ef0 con 0x7f105806c730 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105caef640 1 -- 172.21.15.94:0/3440740182 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040003520 con 0x7f105806c730 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105d2f0640 1 -- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 msgr2=0x7f10580c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105d2f0640 1 --2- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f10580c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105d2f0640 1 -- 172.21.15.94:0/3440740182 >> v1:172.21.15.94:6789/0 conn(0x7f105806c730 legacy=0x7f10581523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105d2f0640 1 -- 172.21.15.94:0/3440740182 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10581559b0 con 0x7f10580ab170 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.952+0000 7f105daf1640 1 --2- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f10580c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105d2f0640 1 --2- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 0x7f105806c1f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f104c0097a0 tx=0x7f104c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d83a1635ba2b7be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105caef640 1 -- 172.21.15.94:0/3440740182 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c0315a0 con 0x7f10580ab170 2024-03-06T07:23:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105caef640 1 -- 172.21.15.94:0/3440740182 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c037070 con 0x7f10580ab170 2024-03-06T07:23:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105caef640 1 -- 172.21.15.94:0/3440740182 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c031940 con 0x7f10580ab170 2024-03-06T07:23:48.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 -- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 msgr2=0x7f105806c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 --2- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 0x7f105806c1f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f104c0097a0 tx=0x7f104c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 -- 172.21.15.94:0/3440740182 shutdown_connections 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 --2- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 0x7f105806c1f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 --2- 172.21.15.94:0/3440740182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f10580c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 -- 172.21.15.94:0/3440740182 >> 172.21.15.94:0/3440740182 conn(0x7f10580c6710 msgr2=0x7f10580c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 -- 172.21.15.94:0/3440740182 shutdown_connections 2024-03-06T07:23:48.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.953+0000 7f105fd55640 1 -- 172.21.15.94:0/3440740182 wait complete. 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.954+0000 7f105fd55640 1 Processor -- start 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105fd55640 1 -- start start 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105fd55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f105806c730 0x7f10580bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105fd55640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 0x7f1058070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105fd55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f1058070890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105fd55640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058156490 con 0x7f105806c730 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105fd55640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058155f70 con 0x7f10580ab170 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105fd55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1058156610 con 0x7f10580c5990 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105daf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f105806c730 0x7f10580bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105daf1640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f105806c730 0x7f10580bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38938/0 (socket says 172.21.15.94:38938) 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105daf1640 1 -- 172.21.15.94:0/3264754996 learned_addr learned my addr 172.21.15.94:0/3264754996 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:48.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105d2f0640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 0x7f1058070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.955+0000 7f105e2f2640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f1058070890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105daf1640 1 -- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 msgr2=0x7f1058070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105daf1640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f1058070890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105daf1640 1 -- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 msgr2=0x7f1058070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105daf1640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 0x7f1058070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105daf1640 1 -- 172.21.15.94:0/3264754996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580a7520 con 0x7f105806c730 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105daf1640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f105806c730 0x7f10580bfdb0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f105000efc0 tx=0x7f105000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f10567fc640 1 -- 172.21.15.94:0/3264754996 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050018020 con 0x7f105806c730 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f10567fc640 1 -- 172.21.15.94:0/3264754996 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1050014680 con 0x7f105806c730 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10580a7800 con 0x7f105806c730 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f10567fc640 1 -- 172.21.15.94:0/3264754996 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050014a50 con 0x7f105806c730 2024-03-06T07:23:48.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.956+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10580c02f0 con 0x7f105806c730 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.957+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10580c7020 con 0x7f105806c730 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.957+0000 7f10567fc640 1 -- 172.21.15.94:0/3264754996 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f105001b020 con 0x7f105806c730 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.958+0000 7f10567fc640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f104406b4d0 0x7f104406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.958+0000 7f10567fc640 1 -- 172.21.15.94:0/3264754996 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f105008b0f0 con 0x7f105806c730 2024-03-06T07:23:48.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.958+0000 7f105d2f0640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f104406b4d0 0x7f104406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.958+0000 7f105d2f0640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f104406b4d0 0x7f104406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f104c036000 tx=0x7f104c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:48.962+0000 7f10567fc640 1 -- 172.21.15.94:0/3264754996 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10580c7020 con 0x7f105806c730 2024-03-06T07:23:49.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.131+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f105804f330 con 0x7f105806c730 2024-03-06T07:23:49.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.131+0000 7f10567fc640 1 -- 172.21.15.94:0/3264754996 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v137) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1050022020 con 0x7f105806c730 2024-03-06T07:23:49.133 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:23:49.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f104406b4d0 msgr2=0x7f104406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f104406b4d0 0x7f104406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f104c036000 tx=0x7f104c03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f105806c730 msgr2=0x7f10580bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f105806c730 0x7f10580bfdb0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f105000efc0 tx=0x7f105000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 shutdown_connections 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f104406b4d0 0x7f104406d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580c5990 0x7f1058070890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10580ab170 0x7f1058070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 --2- 172.21.15.94:0/3264754996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f105806c730 0x7f10580bfdb0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 >> 172.21.15.94:0/3264754996 conn(0x7f10580c6710 msgr2=0x7f105814ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 shutdown_connections 2024-03-06T07:23:49.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.133+0000 7f105fd55640 1 -- 172.21.15.94:0/3264754996 wait complete. 2024-03-06T07:23:49.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.247+0000 7f94a94fb640 1 Processor -- start 2024-03-06T07:23:49.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.247+0000 7f94a94fb640 1 -- start start 2024-03-06T07:23:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a94fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a40b0070 0x7f94a4155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a94fb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f94a406ca70 0x7f94a40c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a94fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a40b0450 con 0x7f94a40c5380 2024-03-06T07:23:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a94fb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a406c020 con 0x7f94a406ca70 2024-03-06T07:23:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a94fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40c77f0 con 0x7f94a40b0070 2024-03-06T07:23:49.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a37fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f94a40c5380 0x7f94a414e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47756/0 (socket says 172.21.15.94:47756) 2024-03-06T07:23:49.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a37fe640 1 -- 172.21.15.94:0/380642559 learned_addr learned my addr 172.21.15.94:0/380642559 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:49.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f949a7fc640 1 --2- 172.21.15.94:0/380642559 >> v2:172.21.15.145:3300/0 conn(0x7f94a406ca70 0x7f94a40c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a2ffd640 1 --2- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a40b0070 0x7f94a4155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a27fc640 1 -- 172.21.15.94:0/380642559 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985816702 0 0) 0x7f949c001490 con 0x7f94a40c5380 2024-03-06T07:23:49.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a27fc640 1 -- 172.21.15.94:0/380642559 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490003520 con 0x7f94a40c5380 2024-03-06T07:23:49.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f949a7fc640 1 -- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a40b0070 msgr2=0x7f94a4155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f949a7fc640 1 --2- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a40b0070 0x7f94a4155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f949a7fc640 1 -- 172.21.15.94:0/380642559 >> v1:172.21.15.94:6789/0 conn(0x7f94a40c5380 legacy=0x7f94a414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f949a7fc640 1 -- 172.21.15.94:0/380642559 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40c7ad0 con 0x7f94a406ca70 2024-03-06T07:23:49.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.248+0000 7f94a2ffd640 1 --2- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a40b0070 0x7f94a4155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:49.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.249+0000 7f949a7fc640 1 --2- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a406ca70 0x7f94a40c8900 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f948c0097a0 tx=0x7f948c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bacc1b30f0ecbbf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.249+0000 7f94a27fc640 1 -- 172.21.15.94:0/380642559 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c0315a0 con 0x7f94a406ca70 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.249+0000 7f94a27fc640 1 -- 172.21.15.94:0/380642559 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948c037070 con 0x7f94a406ca70 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.249+0000 7f94a27fc640 1 -- 172.21.15.94:0/380642559 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c031960 con 0x7f94a406ca70 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 -- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a406ca70 msgr2=0x7f94a40c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 --2- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a406ca70 0x7f94a40c8900 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f948c0097a0 tx=0x7f948c02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 -- 172.21.15.94:0/380642559 shutdown_connections 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 --2- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a406ca70 0x7f94a40c8900 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 --2- 172.21.15.94:0/380642559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a40b0070 0x7f94a4155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 -- 172.21.15.94:0/380642559 >> 172.21.15.94:0/380642559 conn(0x7f94a40ac590 msgr2=0x7f94a40ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 -- 172.21.15.94:0/380642559 shutdown_connections 2024-03-06T07:23:49.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.250+0000 7f94a94fb640 1 -- 172.21.15.94:0/380642559 wait complete. 2024-03-06T07:23:49.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.251+0000 7f94a94fb640 1 Processor -- start 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.251+0000 7f94a94fb640 1 -- start start 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a94fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a406ca70 0x7f94a4154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a94fb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a40b0070 0x7f94a41ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a94fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94a40c5380 0x7f94a41ee690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a94fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a406cef0 con 0x7f94a40c5380 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a94fb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a406c550 con 0x7f94a40b0070 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a94fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a406da80 con 0x7f94a406ca70 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94a40c5380 0x7f94a41ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94a40c5380 0x7f94a41ee690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38952/0 (socket says 172.21.15.94:38952) 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 -- 172.21.15.94:0/3910110727 learned_addr learned my addr 172.21.15.94:0/3910110727 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a2ffd640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a406ca70 0x7f94a4154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f949a7fc640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a40b0070 0x7f94a41ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 -- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a406ca70 msgr2=0x7f94a4154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a406ca70 0x7f94a4154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 -- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a40b0070 msgr2=0x7f94a41ee150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a40b0070 0x7f94a41ee150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 -- 172.21.15.94:0/3910110727 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40c3500 con 0x7f94a40c5380 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a37fe640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94a40c5380 0x7f94a41ee690 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f949c00cb40 tx=0x7f949c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f949bfff640 1 -- 172.21.15.94:0/3910110727 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c017070 con 0x7f94a40c5380 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f949bfff640 1 -- 172.21.15.94:0/3910110727 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c00ac40 con 0x7f94a40c5380 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a40c37e0 con 0x7f94a40c5380 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.252+0000 7f949bfff640 1 -- 172.21.15.94:0/3910110727 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c012400 con 0x7f94a40c5380 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.253+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a4154960 con 0x7f94a40c5380 2024-03-06T07:23:49.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.253+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a404f330 con 0x7f94a40c5380 2024-03-06T07:23:49.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.254+0000 7f949bfff640 1 -- 172.21.15.94:0/3910110727 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f949c01a070 con 0x7f94a40c5380 2024-03-06T07:23:49.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.254+0000 7f949bfff640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f948406b4d0 0x7f948406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.254+0000 7f949bfff640 1 -- 172.21.15.94:0/3910110727 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f949c08a0d0 con 0x7f94a40c5380 2024-03-06T07:23:49.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.254+0000 7f94a2ffd640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f948406b4d0 0x7f948406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.255+0000 7f94a2ffd640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f948406b4d0 0x7f948406d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f94a40be000 tx=0x7f9494009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.259+0000 7f949bfff640 1 -- 172.21.15.94:0/3910110727 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94a40b0450 con 0x7f94a40c5380 2024-03-06T07:23:49.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.430+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94a406d860 con 0x7f94a40c5380 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.430+0000 7f949bfff640 1 -- 172.21.15.94:0/3910110727 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v137) v1 ==== 104+0+11 (secure 0 0 0) 0x7f949c058750 con 0x7f94a40c5380 2024-03-06T07:23:49.432 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f948406b4d0 msgr2=0x7f948406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f948406b4d0 0x7f948406d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f94a40be000 tx=0x7f9494009190 comp rx=0 tx=0).stop 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94a40c5380 msgr2=0x7f94a41ee690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94a40c5380 0x7f94a41ee690 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f949c00cb40 tx=0x7f949c00a060 comp rx=0 tx=0).stop 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 shutdown_connections 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f948406b4d0 0x7f948406d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94a40c5380 0x7f94a41ee690 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f94a40b0070 0x7f94a41ee150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 --2- 172.21.15.94:0/3910110727 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94a406ca70 0x7f94a4154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 >> 172.21.15.94:0/3910110727 conn(0x7f94a40ac590 msgr2=0x7f94a40ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 shutdown_connections 2024-03-06T07:23:49.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.432+0000 7f94a94fb640 1 -- 172.21.15.94:0/3910110727 wait complete. 2024-03-06T07:23:49.445 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:49.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.545+0000 7f99f83dd640 1 Processor -- start 2024-03-06T07:23:49.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.545+0000 7f99f83dd640 1 -- start start 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.546+0000 7f99f83dd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f006c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.546+0000 7f99f83dd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f99f00c1820 0x7f99f006cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.546+0000 7f99f83dd640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f00c2790 con 0x7f99f00c2980 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.546+0000 7f99f83dd640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f00c1ca0 con 0x7f99f00c1820 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.546+0000 7f99f83dd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f014f350 con 0x7f99f014ef70 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f697a640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f99f00c2980 0x7f99f0154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49264/0 (socket says 172.21.15.94:49264) 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f697a640 1 -- 172.21.15.94:0/249386253 learned_addr learned my addr 172.21.15.94:0/249386253 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f6179640 1 --2- 172.21.15.94:0/249386253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f006c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f5978640 1 --2- 172.21.15.94:0/249386253 >> v2:172.21.15.145:3300/0 conn(0x7f99f00c1820 0x7f99f006cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f5177640 1 -- 172.21.15.94:0/249386253 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984708213 0 0) 0x7f99f00c2790 con 0x7f99f00c2980 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f5177640 1 -- 172.21.15.94:0/249386253 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99dc003520 con 0x7f99f00c2980 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f6179640 1 -- 172.21.15.94:0/249386253 >> v2:172.21.15.145:3300/0 conn(0x7f99f00c1820 msgr2=0x7f99f006cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f6179640 1 --2- 172.21.15.94:0/249386253 >> v2:172.21.15.145:3300/0 conn(0x7f99f00c1820 0x7f99f006cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f6179640 1 -- 172.21.15.94:0/249386253 >> v1:172.21.15.145:6789/0 conn(0x7f99f00c2980 legacy=0x7f99f0154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f6179640 1 -- 172.21.15.94:0/249386253 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00c1540 con 0x7f99f014ef70 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.547+0000 7f99f5978640 1 --2- 172.21.15.94:0/249386253 >> v2:172.21.15.145:3300/0 conn(0x7f99f00c1820 0x7f99f006cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f6179640 1 --2- 172.21.15.94:0/249386253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f006c1f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d5b7d46cdba8a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f5177640 1 -- 172.21.15.94:0/249386253 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec0315a0 con 0x7f99f014ef70 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f5177640 1 -- 172.21.15.94:0/249386253 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec037070 con 0x7f99f014ef70 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f5177640 1 -- 172.21.15.94:0/249386253 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec031960 con 0x7f99f014ef70 2024-03-06T07:23:49.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f83dd640 1 -- 172.21.15.94:0/249386253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 msgr2=0x7f99f006c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f83dd640 1 --2- 172.21.15.94:0/249386253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f006c1f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec02b840 comp rx=0 tx=0).stop 2024-03-06T07:23:49.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f83dd640 1 -- 172.21.15.94:0/249386253 shutdown_connections 2024-03-06T07:23:49.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f83dd640 1 --2- 172.21.15.94:0/249386253 >> v2:172.21.15.145:3300/0 conn(0x7f99f00c1820 0x7f99f006cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f83dd640 1 --2- 172.21.15.94:0/249386253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f006c1f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.548+0000 7f99f83dd640 1 -- 172.21.15.94:0/249386253 >> 172.21.15.94:0/249386253 conn(0x7f99f00c6710 msgr2=0x7f99f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.549+0000 7f99f83dd640 1 -- 172.21.15.94:0/249386253 shutdown_connections 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.549+0000 7f99f83dd640 1 -- 172.21.15.94:0/249386253 wait complete. 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 Processor -- start 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 -- start start 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99f00c1820 0x7f99f00bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f00c2980 0x7f99f00721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f01516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0155340 con 0x7f99f00c1820 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f0155750 con 0x7f99f00c2980 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f83dd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f01554c0 con 0x7f99f014ef70 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f6179640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99f00c1820 0x7f99f00bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f6179640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99f00c1820 0x7f99f00bfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38954/0 (socket says 172.21.15.94:38954) 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f6179640 1 -- 172.21.15.94:0/2777709445 learned_addr learned my addr 172.21.15.94:0/2777709445 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f5978640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f00c2980 0x7f99f00721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.550+0000 7f99f697a640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f01516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f6179640 1 -- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 msgr2=0x7f99f01516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f6179640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f01516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f6179640 1 -- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f00c2980 msgr2=0x7f99f00721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f6179640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f00c2980 0x7f99f00721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f6179640 1 -- 172.21.15.94:0/2777709445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f01550c0 con 0x7f99f00c1820 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f6179640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99f00c1820 0x7f99f00bfad0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0098c0 tx=0x7f99ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99e6ffd640 1 -- 172.21.15.94:0/2777709445 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec049070 con 0x7f99f00c1820 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99e6ffd640 1 -- 172.21.15.94:0/2777709445 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec032430 con 0x7f99f00c1820 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f0154540 con 0x7f99f00c1820 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99e6ffd640 1 -- 172.21.15.94:0/2777709445 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec032800 con 0x7f99f00c1820 2024-03-06T07:23:49.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.551+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f0154b00 con 0x7f99f00c1820 2024-03-06T07:23:49.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.552+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f004f330 con 0x7f99f00c1820 2024-03-06T07:23:49.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.552+0000 7f99e6ffd640 1 -- 172.21.15.94:0/2777709445 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99ec0329a0 con 0x7f99f00c1820 2024-03-06T07:23:49.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.553+0000 7f99e6ffd640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99d406b4d0 0x7f99d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.553+0000 7f99e6ffd640 1 -- 172.21.15.94:0/2777709445 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f99ec02f020 con 0x7f99f00c1820 2024-03-06T07:23:49.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.553+0000 7f99f5978640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99d406b4d0 0x7f99d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.554+0000 7f99f5978640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99d406b4d0 0x7f99d406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f99f014e0b0 tx=0x7f99e000a2a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.557+0000 7f99e6ffd640 1 -- 172.21.15.94:0/2777709445 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99ec079e40 con 0x7f99f00c1820 2024-03-06T07:23:49.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.728+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99f00c1110 con 0x7f99f00c1820 2024-03-06T07:23:49.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.729+0000 7f99e6ffd640 1 -- 172.21.15.94:0/2777709445 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v137) v1 ==== 100+0+11 (secure 0 0 0) 0x7f99ec07dee0 con 0x7f99f00c1820 2024-03-06T07:23:49.731 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99d406b4d0 msgr2=0x7f99d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99d406b4d0 0x7f99d406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f99f014e0b0 tx=0x7f99e000a2a0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99f00c1820 msgr2=0x7f99f00bfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99f00c1820 0x7f99f00bfad0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0098c0 tx=0x7f99ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 shutdown_connections 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99d406b4d0 0x7f99d406d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99f014ef70 0x7f99f01516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f00c2980 0x7f99f00721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 --2- 172.21.15.94:0/2777709445 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99f00c1820 0x7f99f00bfad0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 >> 172.21.15.94:0/2777709445 conn(0x7f99f00c6710 msgr2=0x7f99f00ab520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 shutdown_connections 2024-03-06T07:23:49.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.731+0000 7f99f83dd640 1 -- 172.21.15.94:0/2777709445 wait complete. 2024-03-06T07:23:49.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:49.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.842+0000 7f8a43e39640 1 Processor -- start 2024-03-06T07:23:49.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.842+0000 7f8a43e39640 1 -- start start 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a43e39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8a3c14f360 0x7f8a3c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a43e39640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8a3c0c88c0 0x7f8a3c0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a43e39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c5880 0x7f8a3c158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a43e39640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a3c0c8dd0 con 0x7f8a3c0c88c0 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a43e39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a3c0c73d0 con 0x7f8a3c14f360 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a43e39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c0be2e0 con 0x7f8a3c0c5880 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a413d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8a3c0c88c0 0x7f8a3c0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a413d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8a3c0c88c0 0x7f8a3c0c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38964/0 (socket says 172.21.15.94:38964) 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a413d4640 1 -- 172.21.15.94:0/772133797 learned_addr learned my addr 172.21.15.94:0/772133797 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a41bd5640 1 --2- 172.21.15.94:0/772133797 >> v2:172.21.15.145:3300/0 conn(0x7f8a3c14f360 0x7f8a3c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.843+0000 7f8a423d6640 1 --2- 172.21.15.94:0/772133797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c5880 0x7f8a3c158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a413d4640 1 -- 172.21.15.94:0/772133797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c5880 msgr2=0x7f8a3c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a413d4640 1 --2- 172.21.15.94:0/772133797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c5880 0x7f8a3c158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a413d4640 1 -- 172.21.15.94:0/772133797 >> v2:172.21.15.145:3300/0 conn(0x7f8a3c14f360 msgr2=0x7f8a3c155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a413d4640 1 --2- 172.21.15.94:0/772133797 >> v2:172.21.15.145:3300/0 conn(0x7f8a3c14f360 0x7f8a3c155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a413d4640 1 -- 172.21.15.94:0/772133797 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0c7670 con 0x7f8a3c0c88c0 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a413d4640 1 --2- 172.21.15.94:0/772133797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c0c88c0 0x7f8a3c0c5340 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c08c840 tx=0x7f8a2c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd744f35485c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a40bd3640 1 -- 172.21.15.94:0/772133797 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c02e070 con 0x7f8a3c0c88c0 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a40bd3640 1 -- 172.21.15.94:0/772133797 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c037070 con 0x7f8a3c0c88c0 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.844+0000 7f8a40bd3640 1 -- 172.21.15.94:0/772133797 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c031a00 con 0x7f8a3c0c88c0 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 -- 172.21.15.94:0/772133797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c0c88c0 msgr2=0x7f8a3c0c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 --2- 172.21.15.94:0/772133797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c0c88c0 0x7f8a3c0c5340 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c08c840 tx=0x7f8a2c02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 -- 172.21.15.94:0/772133797 shutdown_connections 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 --2- 172.21.15.94:0/772133797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c5880 0x7f8a3c158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 --2- 172.21.15.94:0/772133797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c0c88c0 0x7f8a3c0c5340 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 --2- 172.21.15.94:0/772133797 >> v2:172.21.15.145:3300/0 conn(0x7f8a3c14f360 0x7f8a3c155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 -- 172.21.15.94:0/772133797 >> 172.21.15.94:0/772133797 conn(0x7f8a3c0ac570 msgr2=0x7f8a3c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 -- 172.21.15.94:0/772133797 shutdown_connections 2024-03-06T07:23:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.845+0000 7f8a43e39640 1 -- 172.21.15.94:0/772133797 wait complete. 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.846+0000 7f8a43e39640 1 Processor -- start 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.846+0000 7f8a43e39640 1 -- start start 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a43e39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a3c0c5880 0x7f8a3c0c10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a43e39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c88c0 0x7f8a3c0c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a43e39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c14f360 0x7f8a3c0c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a43e39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c1590a0 con 0x7f8a3c14f360 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a43e39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c0c7c30 con 0x7f8a3c0c5880 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a43e39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c159220 con 0x7f8a3c0c88c0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c14f360 0x7f8a3c0c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c14f360 0x7f8a3c0c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38970/0 (socket says 172.21.15.94:38970) 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 -- 172.21.15.94:0/1367926022 learned_addr learned my addr 172.21.15.94:0/1367926022 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a41bd5640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a3c0c5880 0x7f8a3c0c10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a413d4640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c88c0 0x7f8a3c0c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 -- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c88c0 msgr2=0x7f8a3c0c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c88c0 0x7f8a3c0c06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 -- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a3c0c5880 msgr2=0x7f8a3c0c10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a3c0c5880 0x7f8a3c0c10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 -- 172.21.15.94:0/1367926022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0be740 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.847+0000 7f8a423d6640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c14f360 0x7f8a3c0c9ea0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400e830 tx=0x7f8a3400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.848+0000 7f8a327fc640 1 -- 172.21.15.94:0/1367926022 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34017070 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.848+0000 7f8a327fc640 1 -- 172.21.15.94:0/1367926022 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3400c940 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.848+0000 7f8a327fc640 1 -- 172.21.15.94:0/1367926022 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3400cd30 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.848+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c1ee880 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.848+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c1eeb60 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.849+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3c1ee6b0 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.849+0000 7f8a327fc640 1 -- 172.21.15.94:0/1367926022 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a3401a070 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.849+0000 7f8a327fc640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a2006b500 0x7f8a2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.850+0000 7f8a327fc640 1 -- 172.21.15.94:0/1367926022 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8a3408a5f0 con 0x7f8a3c14f360 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.850+0000 7f8a41bd5640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a2006b500 0x7f8a2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.850+0000 7f8a41bd5640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a2006b500 0x7f8a2006d9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8a38011340 tx=0x7f8a3800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:49.854+0000 7f8a327fc640 1 -- 172.21.15.94:0/1367926022 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a34058c70 con 0x7f8a3c14f360 2024-03-06T07:23:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.059+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a3c04f330 con 0x7f8a3c14f360 2024-03-06T07:23:50.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.060+0000 7f8a327fc640 1 -- 172.21.15.94:0/1367926022 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4994 (secure 0 0 0) 0x7f8a3405cd10 con 0x7f8a3c14f360 2024-03-06T07:23:50.062 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:50.062 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":36917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1486437600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1486437600},{"type":"v1","addr":"172.21.15.145:6836","nonce":1486437600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:45.793611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7726},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7726":{"gid":7726,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7726,"qdb_cluster":[7726]},"id":10}]} 2024-03-06T07:23:50.062 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 129 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a2006b500 msgr2=0x7f8a2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a2006b500 0x7f8a2006d9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8a38011340 tx=0x7f8a3800b040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c14f360 msgr2=0x7f8a3c0c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c14f360 0x7f8a3c0c9ea0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400e830 tx=0x7f8a3400ecf0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 shutdown_connections 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a2006b500 0x7f8a2006d9c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a3c14f360 0x7f8a3c0c9ea0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a3c0c88c0 0x7f8a3c0c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 --2- 172.21.15.94:0/1367926022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a3c0c5880 0x7f8a3c0c10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 >> 172.21.15.94:0/1367926022 conn(0x7f8a3c0ac570 msgr2=0x7f8a3c0cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 shutdown_connections 2024-03-06T07:23:50.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.062+0000 7f8a43e39640 1 -- 172.21.15.94:0/1367926022 wait complete. 2024-03-06T07:23:50.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:50.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.173+0000 7fca1b4d6640 1 Processor -- start 2024-03-06T07:23:50.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.173+0000 7fca1b4d6640 1 -- start start 2024-03-06T07:23:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca1b4d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca1406afc0 0x7fca1406b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca1b4d6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fca140c25c0 0x7fca14153890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca1b4d6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca14153dd0 con 0x7fca140c20b0 2024-03-06T07:23:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca1b4d6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca140a9530 con 0x7fca140c25c0 2024-03-06T07:23:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca1b4d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca140a9810 con 0x7fca1406afc0 2024-03-06T07:23:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19272640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca1406afc0 0x7fca1406b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19a73640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fca140c25c0 0x7fca14153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19272640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca1406afc0 0x7fca1406b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:37848/0 (socket says 172.21.15.94:37848) 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19272640 1 -- 172.21.15.94:0/3342167886 learned_addr learned my addr 172.21.15.94:0/3342167886 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca18a71640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fca140c20b0 0x7fca14153180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49268/0 (socket says 172.21.15.94:49268) 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19a73640 1 --2- 172.21.15.94:0/3342167886 >> v2:172.21.15.145:3300/0 conn(0x7fca140c25c0 0x7fca14153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:57784/0 (socket says 172.21.15.94:57784) 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19272640 1 -- 172.21.15.94:0/3342167886 >> v2:172.21.15.145:3300/0 conn(0x7fca140c25c0 msgr2=0x7fca14153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19272640 1 --2- 172.21.15.94:0/3342167886 >> v2:172.21.15.145:3300/0 conn(0x7fca140c25c0 0x7fca14153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.174+0000 7fca19272640 1 -- 172.21.15.94:0/3342167886 >> v1:172.21.15.145:6789/0 conn(0x7fca140c20b0 legacy=0x7fca14153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.175+0000 7fca19272640 1 -- 172.21.15.94:0/3342167886 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140a9af0 con 0x7fca1406afc0 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.175+0000 7fca19a73640 1 --2- 172.21.15.94:0/3342167886 >> v2:172.21.15.145:3300/0 conn(0x7fca140c25c0 0x7fca14153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.175+0000 7fca19272640 1 --2- 172.21.15.94:0/3342167886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca1406afc0 0x7fca1406b3a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fca10009870 tx=0x7fca1002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=206ee82a20dd3eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.175+0000 7fca0bfff640 1 -- 172.21.15.94:0/3342167886 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca100317c0 con 0x7fca1406afc0 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.175+0000 7fca0bfff640 1 -- 172.21.15.94:0/3342167886 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca10037070 con 0x7fca1406afc0 2024-03-06T07:23:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.175+0000 7fca0bfff640 1 -- 172.21.15.94:0/3342167886 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10031b80 con 0x7fca1406afc0 2024-03-06T07:23:50.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 -- 172.21.15.94:0/3342167886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca1406afc0 msgr2=0x7fca1406b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/3342167886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca1406afc0 0x7fca1406b3a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fca10009870 tx=0x7fca1002b860 comp rx=0 tx=0).stop 2024-03-06T07:23:50.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 -- 172.21.15.94:0/3342167886 shutdown_connections 2024-03-06T07:23:50.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/3342167886 >> v2:172.21.15.145:3300/0 conn(0x7fca140c25c0 0x7fca14153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/3342167886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca1406afc0 0x7fca1406b3a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 -- 172.21.15.94:0/3342167886 >> 172.21.15.94:0/3342167886 conn(0x7fca140c76b0 msgr2=0x7fca140c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 -- 172.21.15.94:0/3342167886 shutdown_connections 2024-03-06T07:23:50.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.176+0000 7fca1b4d6640 1 -- 172.21.15.94:0/3342167886 wait complete. 2024-03-06T07:23:50.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.177+0000 7fca1b4d6640 1 Processor -- start 2024-03-06T07:23:50.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.177+0000 7fca1b4d6640 1 -- start start 2024-03-06T07:23:50.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca1b4d6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca1406afc0 0x7fca140c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca1b4d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca140c20b0 0x7fca141df7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca1b4d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca140c25c0 0x7fca140acf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca1b4d6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca14150980 con 0x7fca140c25c0 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca1b4d6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca140aa0b0 con 0x7fca1406afc0 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca1b4d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca14150b00 con 0x7fca140c20b0 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca140c25c0 0x7fca140acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca18a71640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca140c20b0 0x7fca141df7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca140c25c0 0x7fca140acf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38986/0 (socket says 172.21.15.94:38986) 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 -- 172.21.15.94:0/2140704584 learned_addr learned my addr 172.21.15.94:0/2140704584 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19272640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca1406afc0 0x7fca140c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 -- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca140c20b0 msgr2=0x7fca141df7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca140c20b0 0x7fca141df7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 -- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca1406afc0 msgr2=0x7fca140c5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca1406afc0 0x7fca140c5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 -- 172.21.15.94:0/2140704584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140c7140 con 0x7fca140c25c0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca18a71640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca140c20b0 0x7fca141df7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.178+0000 7fca19a73640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca140c25c0 0x7fca140acf60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00efc0 tx=0x7fca0c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.179+0000 7fca09ffb640 1 -- 172.21.15.94:0/2140704584 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c018030 con 0x7fca140c25c0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.179+0000 7fca09ffb640 1 -- 172.21.15.94:0/2140704584 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c00cd80 con 0x7fca140c25c0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.179+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1414b5a0 con 0x7fca140c25c0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.179+0000 7fca09ffb640 1 -- 172.21.15.94:0/2140704584 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c003bb0 con 0x7fca140c25c0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.179+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1414b840 con 0x7fca140c25c0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.180+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1404f330 con 0x7fca140c25c0 2024-03-06T07:23:50.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.180+0000 7fca09ffb640 1 -- 172.21.15.94:0/2140704584 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca0c005d10 con 0x7fca140c25c0 2024-03-06T07:23:50.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.180+0000 7fca09ffb640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc9f806b500 0x7fc9f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.181+0000 7fca09ffb640 1 -- 172.21.15.94:0/2140704584 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fca0c08ac00 con 0x7fca140c25c0 2024-03-06T07:23:50.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.181+0000 7fca19272640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc9f806b500 0x7fc9f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.181+0000 7fca19272640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc9f806b500 0x7fc9f806d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fca10036000 tx=0x7fca1003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.185+0000 7fca09ffb640 1 -- 172.21.15.94:0/2140704584 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca0c091080 con 0x7fca140c25c0 2024-03-06T07:23:50.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.356+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca140c1a50 con 0x7fca140c25c0 2024-03-06T07:23:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.357+0000 7fca09ffb640 1 -- 172.21.15.94:0/2140704584 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+17151 (secure 0 0 0) 0x7fca0c022070 con 0x7fca140c25c0 2024-03-06T07:23:50.359 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:50.359 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":137,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:42.782345+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:18.382044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:19.127404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.145:6834/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.94:6835/392019985":"2024-03-07T07:23:28.504159+0000","172.21.15.94:6834/392019985":"2024-03-07T07:23:28.504159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc9f806b500 msgr2=0x7fc9f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc9f806b500 0x7fc9f806d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fca10036000 tx=0x7fca1003d040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca140c25c0 msgr2=0x7fca140acf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca140c25c0 0x7fca140acf60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00efc0 tx=0x7fca0c009f90 comp rx=0 tx=0).stop 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 shutdown_connections 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc9f806b500 0x7fc9f806d9c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca140c25c0 0x7fca140acf60 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca140c20b0 0x7fca141df7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 --2- 172.21.15.94:0/2140704584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca1406afc0 0x7fca140c5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 >> 172.21.15.94:0/2140704584 conn(0x7fca140c76b0 msgr2=0x7fca140c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 shutdown_connections 2024-03-06T07:23:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.359+0000 7fca1b4d6640 1 -- 172.21.15.94:0/2140704584 wait complete. 2024-03-06T07:23:50.372 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:50.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:50.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.471+0000 7fca331c7640 1 Processor -- start 2024-03-06T07:23:50.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.471+0000 7fca331c7640 1 -- start start 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.471+0000 7fca331c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c0c1320 0x7fca2c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.471+0000 7fca331c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c153a60 con 0x7fca2c0c6000 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.471+0000 7fca331c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c153d40 con 0x7fca2c150580 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.471+0000 7fca331c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c06c2c0 con 0x7fca2c0c1320 2024-03-06T07:23:50.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.471+0000 7fca31764640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fca2c0c6000 0x7fca2c156bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47768/0 (socket says 172.21.15.94:47768) 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca31764640 1 -- 172.21.15.94:0/4241795449 learned_addr learned my addr 172.21.15.94:0/4241795449 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2bfff640 1 --2- 172.21.15.94:0/4241795449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c0c1320 0x7fca2c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2b7fe640 1 -- 172.21.15.94:0/4241795449 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085833859 0 0) 0x7fca2c153a60 con 0x7fca2c0c6000 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2b7fe640 1 -- 172.21.15.94:0/4241795449 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca1c003520 con 0x7fca2c0c6000 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2b7fe640 1 -- 172.21.15.94:0/4241795449 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842198137 0 0) 0x7fca2c153d40 con 0x7fca2c150580 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2b7fe640 1 -- 172.21.15.94:0/4241795449 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca2c153a60 con 0x7fca2c150580 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2bfff640 1 -- 172.21.15.94:0/4241795449 >> v1:172.21.15.145:6789/0 conn(0x7fca2c150580 legacy=0x7fca2c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2bfff640 1 -- 172.21.15.94:0/4241795449 >> v1:172.21.15.94:6789/0 conn(0x7fca2c0c6000 legacy=0x7fca2c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.472+0000 7fca2bfff640 1 -- 172.21.15.94:0/4241795449 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c06c5a0 con 0x7fca2c0c1320 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca2bfff640 1 --2- 172.21.15.94:0/4241795449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c0c1320 0x7fca2c1566b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fca180097d0 tx=0x7fca1802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aa0a629e779f04b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca2b7fe640 1 -- 172.21.15.94:0/4241795449 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18035020 con 0x7fca2c0c1320 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca2b7fe640 1 -- 172.21.15.94:0/4241795449 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca18031800 con 0x7fca2c0c1320 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca2b7fe640 1 -- 172.21.15.94:0/4241795449 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18031bd0 con 0x7fca2c0c1320 2024-03-06T07:23:50.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca331c7640 1 -- 172.21.15.94:0/4241795449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c0c1320 msgr2=0x7fca2c1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca331c7640 1 --2- 172.21.15.94:0/4241795449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c0c1320 0x7fca2c1566b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fca180097d0 tx=0x7fca1802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca331c7640 1 -- 172.21.15.94:0/4241795449 shutdown_connections 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca331c7640 1 --2- 172.21.15.94:0/4241795449 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c0c1320 0x7fca2c1566b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca331c7640 1 -- 172.21.15.94:0/4241795449 >> 172.21.15.94:0/4241795449 conn(0x7fca2c0c6710 msgr2=0x7fca2c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca331c7640 1 -- 172.21.15.94:0/4241795449 shutdown_connections 2024-03-06T07:23:50.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.473+0000 7fca331c7640 1 -- 172.21.15.94:0/4241795449 wait complete. 2024-03-06T07:23:50.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.474+0000 7fca331c7640 1 Processor -- start 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.474+0000 7fca331c7640 1 -- start start 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca331c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca2c0c1320 0x7fca2c1e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca331c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca2c0c6000 0x7fca2c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca331c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c150580 0x7fca2c0bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca331c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c06e420 con 0x7fca2c0c6000 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca331c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c150960 con 0x7fca2c0c1320 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca331c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c06e5a0 con 0x7fca2c150580 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca2bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca2c0c6000 0x7fca2c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca2bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca2c0c6000 0x7fca2c155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38998/0 (socket says 172.21.15.94:38998) 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca2bfff640 1 -- 172.21.15.94:0/1894570337 learned_addr learned my addr 172.21.15.94:0/1894570337 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:50.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca30f63640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca2c0c1320 0x7fca2c1e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.475+0000 7fca31764640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c150580 0x7fca2c0bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca2bfff640 1 -- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c150580 msgr2=0x7fca2c0bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca2bfff640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c150580 0x7fca2c0bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca2bfff640 1 -- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca2c0c1320 msgr2=0x7fca2c1e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca2bfff640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca2c0c1320 0x7fca2c1e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca2bfff640 1 -- 172.21.15.94:0/1894570337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c072df0 con 0x7fca2c0c6000 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca2bfff640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca2c0c6000 0x7fca2c155ba0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fca18034040 tx=0x7fca180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca297fa640 1 -- 172.21.15.94:0/1894570337 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18040070 con 0x7fca2c0c6000 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca297fa640 1 -- 172.21.15.94:0/1894570337 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca18031e30 con 0x7fca2c0c6000 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c14a8c0 con 0x7fca2c0c6000 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca297fa640 1 -- 172.21.15.94:0/1894570337 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18035040 con 0x7fca2c0c6000 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.476+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c14ab60 con 0x7fca2c0c6000 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.477+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c04f330 con 0x7fca2c0c6000 2024-03-06T07:23:50.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.477+0000 7fca297fa640 1 -- 172.21.15.94:0/1894570337 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca1804b060 con 0x7fca2c0c6000 2024-03-06T07:23:50.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.478+0000 7fca297fa640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.478+0000 7fca297fa640 1 -- 172.21.15.94:0/1894570337 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fca180abc60 con 0x7fca2c0c6000 2024-03-06T07:23:50.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.478+0000 7fca30f63640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.478+0000 7fca30f63640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fca200011e0 tx=0x7fca20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.482+0000 7fca297fa640 1 -- 172.21.15.94:0/1894570337 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca1807a260 con 0x7fca2c0c6000 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.688+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca2c0c5de0 con 0x7fca2c0c6000 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.689+0000 7fca297fa640 1 -- 172.21.15.94:0/1894570337 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4994 (secure 0 0 0) 0x7fca1807e300 con 0x7fca2c0c6000 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":36917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1486437600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1486437600},{"type":"v1","addr":"172.21.15.145:6836","nonce":1486437600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:45.793611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7726},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7726":{"gid":7726,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7726,"qdb_cluster":[7726]},"id":10}]} 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 129 2024-03-06T07:23:50.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.690+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca1006b4d0 msgr2=0x7fca1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.690+0000 7fca331c7640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fca200011e0 tx=0x7fca20009040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.690+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca2c0c6000 msgr2=0x7fca2c155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.690+0000 7fca331c7640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca2c0c6000 0x7fca2c155ba0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fca18034040 tx=0x7fca180046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 shutdown_connections 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca2c150580 0x7fca2c0bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca2c0c6000 0x7fca2c155ba0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 --2- 172.21.15.94:0/1894570337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca2c0c1320 0x7fca2c1e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 >> 172.21.15.94:0/1894570337 conn(0x7fca2c0c6710 msgr2=0x7fca2c151260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 shutdown_connections 2024-03-06T07:23:50.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.691+0000 7fca331c7640 1 -- 172.21.15.94:0/1894570337 wait complete. 2024-03-06T07:23:50.704 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.802+0000 7fd99c113640 1 Processor -- start 2024-03-06T07:23:50.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.802+0000 7fd99c113640 1 -- start start 2024-03-06T07:23:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd99c113640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd99414f360 0x7fd994155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd99c113640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9940c88c0 0x7fd9940c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd99c113640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940c73d0 con 0x7fd9940c5880 2024-03-06T07:23:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd99c113640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9940c8dd0 con 0x7fd9940c88c0 2024-03-06T07:23:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd99c113640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9940be2e0 con 0x7fd99414f360 2024-03-06T07:23:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd9996ae640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9940c88c0 0x7fd9940c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd9996ae640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9940c88c0 0x7fd9940c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39012/0 (socket says 172.21.15.94:39012) 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd9996ae640 1 -- 172.21.15.94:0/3081174114 learned_addr learned my addr 172.21.15.94:0/3081174114 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.803+0000 7fd999eaf640 1 --2- 172.21.15.94:0/3081174114 >> v2:172.21.15.145:3300/0 conn(0x7fd99414f360 0x7fd994155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd9996ae640 1 -- 172.21.15.94:0/3081174114 >> v2:172.21.15.145:3300/0 conn(0x7fd99414f360 msgr2=0x7fd994155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd9996ae640 1 --2- 172.21.15.94:0/3081174114 >> v2:172.21.15.145:3300/0 conn(0x7fd99414f360 0x7fd994155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd9996ae640 1 -- 172.21.15.94:0/3081174114 >> v1:172.21.15.145:6789/0 conn(0x7fd9940c5880 legacy=0x7fd994158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd9996ae640 1 -- 172.21.15.94:0/3081174114 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940c7670 con 0x7fd9940c88c0 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd999eaf640 1 --2- 172.21.15.94:0/3081174114 >> v2:172.21.15.145:3300/0 conn(0x7fd99414f360 0x7fd994155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd9996ae640 1 --2- 172.21.15.94:0/3081174114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c88c0 0x7fd9940c5340 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fd99408c840 tx=0x7fd98402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc7051ae92719a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd998ead640 1 -- 172.21.15.94:0/3081174114 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984035020 con 0x7fd9940c88c0 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd998ead640 1 -- 172.21.15.94:0/3081174114 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd984031870 con 0x7fd9940c88c0 2024-03-06T07:23:50.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.804+0000 7fd998ead640 1 -- 172.21.15.94:0/3081174114 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984031c40 con 0x7fd9940c88c0 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 -- 172.21.15.94:0/3081174114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c88c0 msgr2=0x7fd9940c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 --2- 172.21.15.94:0/3081174114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c88c0 0x7fd9940c5340 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fd99408c840 tx=0x7fd98402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 -- 172.21.15.94:0/3081174114 shutdown_connections 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 --2- 172.21.15.94:0/3081174114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c88c0 0x7fd9940c5340 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 --2- 172.21.15.94:0/3081174114 >> v2:172.21.15.145:3300/0 conn(0x7fd99414f360 0x7fd994155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 -- 172.21.15.94:0/3081174114 >> 172.21.15.94:0/3081174114 conn(0x7fd9940ac570 msgr2=0x7fd9940ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 -- 172.21.15.94:0/3081174114 shutdown_connections 2024-03-06T07:23:50.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.805+0000 7fd99c113640 1 -- 172.21.15.94:0/3081174114 wait complete. 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.806+0000 7fd99c113640 1 Processor -- start 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.806+0000 7fd99c113640 1 -- start start 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c5880 0x7fd9940ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940c88c0 0x7fd9940c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99414f360 0x7fd99414bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd994159270 con 0x7fd9940c5880 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940c7c30 con 0x7fd9940c88c0 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9941593f0 con 0x7fd99414f360 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c5880 0x7fd9940ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c5880 0x7fd9940ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39022/0 (socket says 172.21.15.94:39022) 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 -- 172.21.15.94:0/1747807332 learned_addr learned my addr 172.21.15.94:0/1747807332 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:50.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd9996ae640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940c88c0 0x7fd9940c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99a6b0640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99414f360 0x7fd99414bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 -- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99414f360 msgr2=0x7fd99414bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99414f360 0x7fd99414bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 -- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940c88c0 msgr2=0x7fd9940c0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940c88c0 0x7fd9940c0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 -- 172.21.15.94:0/1747807332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940c1d30 con 0x7fd9940c5880 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99a6b0640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99414f360 0x7fd99414bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd999eaf640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c5880 0x7fd9940ab7d0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fd99000efc0 tx=0x7fd99000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd98a7fc640 1 -- 172.21.15.94:0/1747807332 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd990018020 con 0x7fd9940c5880 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd98a7fc640 1 -- 172.21.15.94:0/1747807332 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd990014680 con 0x7fd9940c5880 2024-03-06T07:23:50.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9941edf40 con 0x7fd9940c5880 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd98a7fc640 1 -- 172.21.15.94:0/1747807332 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd990014a50 con 0x7fd9940c5880 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.807+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9941ee4c0 con 0x7fd9940c5880 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.808+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99404f330 con 0x7fd9940c5880 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.809+0000 7fd98a7fc640 1 -- 172.21.15.94:0/1747807332 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd990005d10 con 0x7fd9940c5880 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.809+0000 7fd98a7fc640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd97806b4d0 0x7fd97806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.809+0000 7fd98a7fc640 1 -- 172.21.15.94:0/1747807332 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fd99008b0b0 con 0x7fd9940c5880 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.809+0000 7fd9996ae640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd97806b4d0 0x7fd97806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.810+0000 7fd9996ae640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd97806b4d0 0x7fd97806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd984036000 tx=0x7fd98403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:50.814+0000 7fd98a7fc640 1 -- 172.21.15.94:0/1747807332 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd990022030 con 0x7fd9940c5880 2024-03-06T07:23:51.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.022+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd99406cda0 con 0x7fd9940c5880 2024-03-06T07:23:51.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.023+0000 7fd98a7fc640 1 -- 172.21.15.94:0/1747807332 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+4994 (secure 0 0 0) 0x7fd9940c1d30 con 0x7fd9940c5880 2024-03-06T07:23:51.025 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:51.025 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":129,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":36917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1486437600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1486437600},{"type":"v1","addr":"172.21.15.145:6836","nonce":1486437600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:20.150839+0000","modified":"2024-03-06T07:23:45.793611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7726},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7726":{"gid":7726,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/3826543722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3826543722},{"type":"v1","addr":"172.21.15.145:6837","nonce":3826543722}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7726,"qdb_cluster":[7726]},"id":10}]} 2024-03-06T07:23:51.025 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 129 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd97806b4d0 msgr2=0x7fd97806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd97806b4d0 0x7fd97806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd984036000 tx=0x7fd98403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c5880 msgr2=0x7fd9940ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c5880 0x7fd9940ab7d0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fd99000efc0 tx=0x7fd99000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 shutdown_connections 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd97806b4d0 0x7fd97806d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99414f360 0x7fd99414bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940c88c0 0x7fd9940c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 --2- 172.21.15.94:0/1747807332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd9940c5880 0x7fd9940ab7d0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 >> 172.21.15.94:0/1747807332 conn(0x7fd9940ac570 msgr2=0x7fd9941542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 shutdown_connections 2024-03-06T07:23:51.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.025+0000 7fd99c113640 1 -- 172.21.15.94:0/1747807332 wait complete. 2024-03-06T07:23:51.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:51.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.136+0000 7f35af902640 1 Processor -- start 2024-03-06T07:23:51.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.136+0000 7f35af902640 1 -- start start 2024-03-06T07:23:51.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35af902640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35a80c3280 0x7f35a80c3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35af902640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a81551e0 0x7f35a8155630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35af902640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a8158800 con 0x7f35a806d9f0 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35af902640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a8158ae0 con 0x7f35a80c3280 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35af902640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a8158dc0 con 0x7f35a81551e0 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ade9f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a81551e0 0x7f35a8155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ace9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35a80c3280 0x7f35a80c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ade9f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a81551e0 0x7f35a8155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:37896/0 (socket says 172.21.15.94:37896) 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ad69e640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f35a806d9f0 0x7f35a80a7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49306/0 (socket says 172.21.15.94:49306) 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ade9f640 1 -- 172.21.15.94:0/4182559252 learned_addr learned my addr 172.21.15.94:0/4182559252 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ade9f640 1 -- 172.21.15.94:0/4182559252 >> v2:172.21.15.145:3300/0 conn(0x7f35a80c3280 msgr2=0x7f35a80c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ade9f640 1 --2- 172.21.15.94:0/4182559252 >> v2:172.21.15.145:3300/0 conn(0x7f35a80c3280 0x7f35a80c3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ade9f640 1 -- 172.21.15.94:0/4182559252 >> v1:172.21.15.145:6789/0 conn(0x7f35a806d9f0 legacy=0x7f35a80a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ade9f640 1 -- 172.21.15.94:0/4182559252 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a81590a0 con 0x7f35a81551e0 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.137+0000 7f35ace9d640 1 --2- 172.21.15.94:0/4182559252 >> v2:172.21.15.145:3300/0 conn(0x7f35a80c3280 0x7f35a80c3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.138+0000 7f35ade9f640 1 --2- 172.21.15.94:0/4182559252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a81551e0 0x7f35a8155630 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009e50 tx=0x7f35a002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2a968a89589e2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.138+0000 7f359ffff640 1 -- 172.21.15.94:0/4182559252 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a002ca70 con 0x7f35a81551e0 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.138+0000 7f359ffff640 1 -- 172.21.15.94:0/4182559252 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a002cc10 con 0x7f35a81551e0 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.138+0000 7f359ffff640 1 -- 172.21.15.94:0/4182559252 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00343f0 con 0x7f35a81551e0 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 -- 172.21.15.94:0/4182559252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a81551e0 msgr2=0x7f35a8155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 --2- 172.21.15.94:0/4182559252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a81551e0 0x7f35a8155630 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009e50 tx=0x7f35a002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 -- 172.21.15.94:0/4182559252 shutdown_connections 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 --2- 172.21.15.94:0/4182559252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a81551e0 0x7f35a8155630 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 --2- 172.21.15.94:0/4182559252 >> v2:172.21.15.145:3300/0 conn(0x7f35a80c3280 0x7f35a80c3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 -- 172.21.15.94:0/4182559252 >> 172.21.15.94:0/4182559252 conn(0x7f35a80aa630 msgr2=0x7f35a80aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 -- 172.21.15.94:0/4182559252 shutdown_connections 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.139+0000 7f35af902640 1 -- 172.21.15.94:0/4182559252 wait complete. 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.140+0000 7f35af902640 1 Processor -- start 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.140+0000 7f35af902640 1 -- start start 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35af902640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a806d9f0 0x7f35a81ee650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35af902640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35a80c3280 0x7f35a81574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35af902640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35a81551e0 0x7f35a8157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35af902640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a806df50 con 0x7f35a80c3280 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35af902640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a806ddd0 con 0x7f35a81551e0 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35af902640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a80c6910 con 0x7f35a806d9f0 2024-03-06T07:23:51.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35a80c3280 0x7f35a81574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35a80c3280 0x7f35a81574e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39030/0 (socket says 172.21.15.94:39030) 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ad69e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a806d9f0 0x7f35a81ee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 -- 172.21.15.94:0/2496309290 learned_addr learned my addr 172.21.15.94:0/2496309290 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ade9f640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35a81551e0 0x7f35a8157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 -- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a806d9f0 msgr2=0x7f35a81ee650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a806d9f0 0x7f35a81ee650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 -- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35a81551e0 msgr2=0x7f35a8157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35a81551e0 0x7f35a8157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 -- 172.21.15.94:0/2496309290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a81ef0d0 con 0x7f35a80c3280 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ad69e640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a806d9f0 0x7f35a81ee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f35ace9d640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35a80c3280 0x7f35a81574e0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f359800ebe0 tx=0x7f359800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.141+0000 7f359dffb640 1 -- 172.21.15.94:0/2496309290 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598017070 con 0x7f35a80c3280 2024-03-06T07:23:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.142+0000 7f359dffb640 1 -- 172.21.15.94:0/2496309290 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359800cc30 con 0x7f35a80c3280 2024-03-06T07:23:51.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.142+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a80ab6c0 con 0x7f35a80c3280 2024-03-06T07:23:51.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.142+0000 7f359dffb640 1 -- 172.21.15.94:0/2496309290 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598013470 con 0x7f35a80c3280 2024-03-06T07:23:51.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.142+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a80abc80 con 0x7f35a80c3280 2024-03-06T07:23:51.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.143+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a8067910 con 0x7f35a80c3280 2024-03-06T07:23:51.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.143+0000 7f359dffb640 1 -- 172.21.15.94:0/2496309290 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f359801a070 con 0x7f35a80c3280 2024-03-06T07:23:51.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.144+0000 7f359dffb640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f358c06b4d0 0x7f358c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.144+0000 7f359dffb640 1 -- 172.21.15.94:0/2496309290 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f359808a290 con 0x7f35a80c3280 2024-03-06T07:23:51.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.144+0000 7f35ad69e640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f358c06b4d0 0x7f358c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.144+0000 7f35ad69e640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f358c06b4d0 0x7f358c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f35a4007fd0 tx=0x7f35a4007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.148+0000 7f359dffb640 1 -- 172.21.15.94:0/2496309290 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35a8067910 con 0x7f35a80c3280 2024-03-06T07:23:51.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.320+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35a804f330 con 0x7f35a80c3280 2024-03-06T07:23:51.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.321+0000 7f359dffb640 1 -- 172.21.15.94:0/2496309290 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+17151 (secure 0 0 0) 0x7f3598058910 con 0x7f35a80c3280 2024-03-06T07:23:51.323 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:51.323 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":137,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:42.782345+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:18.382044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:19.127404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.145:6834/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.94:6835/392019985":"2024-03-07T07:23:28.504159+0000","172.21.15.94:6834/392019985":"2024-03-07T07:23:28.504159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f358c06b4d0 msgr2=0x7f358c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f358c06b4d0 0x7f358c06d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f35a4007fd0 tx=0x7f35a4007e80 comp rx=0 tx=0).stop 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35a80c3280 msgr2=0x7f35a81574e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35a80c3280 0x7f35a81574e0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f359800ebe0 tx=0x7f359800c520 comp rx=0 tx=0).stop 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 shutdown_connections 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f358c06b4d0 0x7f358c06d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35a81551e0 0x7f35a8157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35a80c3280 0x7f35a81574e0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 --2- 172.21.15.94:0/2496309290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a806d9f0 0x7f35a81ee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 >> 172.21.15.94:0/2496309290 conn(0x7f35a80aa630 msgr2=0x7f35a8153450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 shutdown_connections 2024-03-06T07:23:51.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.323+0000 7f35af902640 1 -- 172.21.15.94:0/2496309290 wait complete. 2024-03-06T07:23:51.336 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:51.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.435+0000 7f8f6aafb640 1 Processor -- start 2024-03-06T07:23:51.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.435+0000 7f8f6aafb640 1 -- start start 2024-03-06T07:23:51.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f6aafb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f6aafb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8f64072cd0 0x7f8f6406fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f6aafb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640733f0 con 0x7f8f6406f550 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f6aafb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f64155ca0 con 0x7f8f64072cd0 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f6aafb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f641561d0 con 0x7f8f64152720 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f68897640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f69098640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8f64072cd0 0x7f8f6406fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f68897640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:37908/0 (socket says 172.21.15.94:37908) 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f68897640 1 -- 172.21.15.94:0/4128172850 learned_addr learned my addr 172.21.15.94:0/4128172850 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.436+0000 7f8f63fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f8f6406f550 0x7f8f640c77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49316/0 (socket says 172.21.15.94:49316) 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f68897640 1 -- 172.21.15.94:0/4128172850 >> v2:172.21.15.145:3300/0 conn(0x7f8f64072cd0 msgr2=0x7f8f6406fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f68897640 1 --2- 172.21.15.94:0/4128172850 >> v2:172.21.15.145:3300/0 conn(0x7f8f64072cd0 0x7f8f6406fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f68897640 1 -- 172.21.15.94:0/4128172850 >> v1:172.21.15.145:6789/0 conn(0x7f8f6406f550 legacy=0x7f8f640c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f68897640 1 -- 172.21.15.94:0/4128172850 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64155ef0 con 0x7f8f64152720 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f68897640 1 --2- 172.21.15.94:0/4128172850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641593c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009870 tx=0x7f8f5402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c4a2bd4a1e1166e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f637fe640 1 -- 172.21.15.94:0/4128172850 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54035020 con 0x7f8f64152720 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f637fe640 1 -- 172.21.15.94:0/4128172850 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54031640 con 0x7f8f64152720 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.437+0000 7f8f637fe640 1 -- 172.21.15.94:0/4128172850 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5402d040 con 0x7f8f64152720 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 -- 172.21.15.94:0/4128172850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 msgr2=0x7f8f641593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/4128172850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641593c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009870 tx=0x7f8f5402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 -- 172.21.15.94:0/4128172850 shutdown_connections 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/4128172850 >> v2:172.21.15.145:3300/0 conn(0x7f8f64072cd0 0x7f8f6406fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/4128172850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641593c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 -- 172.21.15.94:0/4128172850 >> 172.21.15.94:0/4128172850 conn(0x7f8f6406ea40 msgr2=0x7f8f6406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 -- 172.21.15.94:0/4128172850 shutdown_connections 2024-03-06T07:23:51.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.438+0000 7f8f6aafb640 1 -- 172.21.15.94:0/4128172850 wait complete. 2024-03-06T07:23:51.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.439+0000 7f8f6aafb640 1 Processor -- start 2024-03-06T07:23:51.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.439+0000 7f8f6aafb640 1 -- start start 2024-03-06T07:23:51.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f6aafb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f6406f550 0x7f8f64148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f6aafb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f64072cd0 0x7f8f64157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f6aafb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641ee180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f6aafb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64158ea0 con 0x7f8f6406f550 2024-03-06T07:23:51.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f6aafb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64158bc0 con 0x7f8f64072cd0 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f6aafb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640c7ef0 con 0x7f8f64152720 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f6406f550 0x7f8f64148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f6406f550 0x7f8f64148ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39032/0 (socket says 172.21.15.94:39032) 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 -- 172.21.15.94:0/668626384 learned_addr learned my addr 172.21.15.94:0/668626384 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f63fff640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f64072cd0 0x7f8f64157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f69098640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 -- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 msgr2=0x7f8f641ee180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641ee180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 -- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f64072cd0 msgr2=0x7f8f64157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f64072cd0 0x7f8f64157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.440+0000 7f8f68897640 1 -- 172.21.15.94:0/668626384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64147c80 con 0x7f8f6406f550 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.441+0000 7f8f68897640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f6406f550 0x7f8f64148ba0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f8f54000c00 tx=0x7f8f540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.441+0000 7f8f617fa640 1 -- 172.21.15.94:0/668626384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54040070 con 0x7f8f6406f550 2024-03-06T07:23:51.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.441+0000 7f8f617fa640 1 -- 172.21.15.94:0/668626384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f540399f0 con 0x7f8f6406f550 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.441+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f64155390 con 0x7f8f6406f550 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.441+0000 7f8f617fa640 1 -- 172.21.15.94:0/668626384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54035040 con 0x7f8f6406f550 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.441+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f64153a70 con 0x7f8f6406f550 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.442+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f641eeb10 con 0x7f8f6406f550 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.442+0000 7f8f617fa640 1 -- 172.21.15.94:0/668626384 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f5404b060 con 0x7f8f6406f550 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.443+0000 7f8f617fa640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f5006b500 0x7f8f5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.443+0000 7f8f617fa640 1 -- 172.21.15.94:0/668626384 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8f540abaf0 con 0x7f8f6406f550 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.443+0000 7f8f63fff640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f5006b500 0x7f8f5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.443+0000 7f8f63fff640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f5006b500 0x7f8f5006d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8f640c7170 tx=0x7f8f58007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.447+0000 7f8f617fa640 1 -- 172.21.15.94:0/668626384 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f5407a0e0 con 0x7f8f6406f550 2024-03-06T07:23:51.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.655+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8f6404f330 con 0x7f8f6406f550 2024-03-06T07:23:51.657 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:23:51.655+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:51.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.656+0000 7f8f617fa640 1 -- 172.21.15.94:0/668626384 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v130) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8f5407e180 con 0x7f8f6406f550 2024-03-06T07:23:51.658 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f5006b500 msgr2=0x7f8f5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f5006b500 0x7f8f5006d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8f640c7170 tx=0x7f8f58007e80 comp rx=0 tx=0).stop 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f6406f550 msgr2=0x7f8f64148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f6406f550 0x7f8f64148ba0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f8f54000c00 tx=0x7f8f540046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 shutdown_connections 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f5006b500 0x7f8f5006d9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f64152720 0x7f8f641ee180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f64072cd0 0x7f8f64157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 --2- 172.21.15.94:0/668626384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f6406f550 0x7f8f64148ba0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 >> 172.21.15.94:0/668626384 conn(0x7f8f6406ea40 msgr2=0x7f8f6406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 shutdown_connections 2024-03-06T07:23:51.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.659+0000 7f8f6aafb640 1 -- 172.21.15.94:0/668626384 wait complete. 2024-03-06T07:23:51.662 INFO:tasks.ceph.mds.b.smithi145.stderr: -202> 2024-03-06T07:23:46.090+0000 7efeba6a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:51.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:51.685 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:23:51.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.771+0000 7fdf373c9640 1 Processor -- start 2024-03-06T07:23:51.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf373c9640 1 -- start start 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf373c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdf300b00d0 0x7fdf300c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf373c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdf3006b8e0 0x7fdf3006bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf373c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf300b04b0 con 0x7fdf300c7210 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf373c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf30156160 con 0x7fdf300b00d0 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf373c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf300c5710 con 0x7fdf3006b8e0 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf35165640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdf300b00d0 0x7fdf300c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf35165640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdf300b00d0 0x7fdf300c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39048/0 (socket says 172.21.15.94:39048) 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.772+0000 7fdf35165640 1 -- 172.21.15.94:0/1342526590 learned_addr learned my addr 172.21.15.94:0/1342526590 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.773+0000 7fdf35165640 1 -- 172.21.15.94:0/1342526590 >> v2:172.21.15.145:3300/0 conn(0x7fdf3006b8e0 msgr2=0x7fdf3006bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.773+0000 7fdf34964640 1 --2- 172.21.15.94:0/1342526590 >> v2:172.21.15.145:3300/0 conn(0x7fdf3006b8e0 0x7fdf3006bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.773+0000 7fdf35165640 1 --2- 172.21.15.94:0/1342526590 >> v2:172.21.15.145:3300/0 conn(0x7fdf3006b8e0 0x7fdf3006bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.773+0000 7fdf35165640 1 -- 172.21.15.94:0/1342526590 >> v1:172.21.15.145:6789/0 conn(0x7fdf300c7210 legacy=0x7fdf300c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.773+0000 7fdf35165640 1 -- 172.21.15.94:0/1342526590 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3006d210 con 0x7fdf300b00d0 2024-03-06T07:23:51.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.773+0000 7fdf35165640 1 --2- 172.21.15.94:0/1342526590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf300b00d0 0x7fdf300c8ff0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fdf3006ce20 tx=0x7fdf2c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc569a80c21e0b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf27fff640 1 -- 172.21.15.94:0/1342526590 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c035020 con 0x7fdf300b00d0 2024-03-06T07:23:51.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf27fff640 1 -- 172.21.15.94:0/1342526590 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c031870 con 0x7fdf300b00d0 2024-03-06T07:23:51.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf27fff640 1 -- 172.21.15.94:0/1342526590 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c031c40 con 0x7fdf300b00d0 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 -- 172.21.15.94:0/1342526590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf300b00d0 msgr2=0x7fdf300c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 --2- 172.21.15.94:0/1342526590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf300b00d0 0x7fdf300c8ff0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fdf3006ce20 tx=0x7fdf2c02b910 comp rx=0 tx=0).stop 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 -- 172.21.15.94:0/1342526590 shutdown_connections 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 --2- 172.21.15.94:0/1342526590 >> v2:172.21.15.145:3300/0 conn(0x7fdf3006b8e0 0x7fdf3006bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 --2- 172.21.15.94:0/1342526590 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf300b00d0 0x7fdf300c8ff0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 -- 172.21.15.94:0/1342526590 >> 172.21.15.94:0/1342526590 conn(0x7fdf300ac810 msgr2=0x7fdf300acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 -- 172.21.15.94:0/1342526590 shutdown_connections 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.774+0000 7fdf373c9640 1 -- 172.21.15.94:0/1342526590 wait complete. 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.775+0000 7fdf373c9640 1 Processor -- start 2024-03-06T07:23:51.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.775+0000 7fdf373c9640 1 -- start start 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf373c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf3006b8e0 0x7fdf301eeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf373c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf300b00d0 0x7fdf30153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf373c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf300c7210 0x7fdf301eab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf373c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3006c260 con 0x7fdf3006b8e0 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf373c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf30152c50 con 0x7fdf300c7210 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf373c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300c5ba0 con 0x7fdf300b00d0 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf3006b8e0 0x7fdf301eeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf3006b8e0 0x7fdf301eeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39058/0 (socket says 172.21.15.94:39058) 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 -- 172.21.15.94:0/2177279488 learned_addr learned my addr 172.21.15.94:0/2177279488 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf34964640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf300b00d0 0x7fdf30153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35966640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf300c7210 0x7fdf301eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 -- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf300b00d0 msgr2=0x7fdf30153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf300b00d0 0x7fdf30153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 -- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf300c7210 msgr2=0x7fdf301eab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf300c7210 0x7fdf301eab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35165640 1 -- 172.21.15.94:0/2177279488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf30159210 con 0x7fdf3006b8e0 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.776+0000 7fdf35966640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf300c7210 0x7fdf301eab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.777+0000 7fdf35165640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf3006b8e0 0x7fdf301eeac0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c000c00 tx=0x7fdf2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.777+0000 7fdf25ffb640 1 -- 172.21.15.94:0/2177279488 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c040070 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.777+0000 7fdf25ffb640 1 -- 172.21.15.94:0/2177279488 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c03a420 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.777+0000 7fdf25ffb640 1 -- 172.21.15.94:0/2177279488 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c035040 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.777+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf301594b0 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.777+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf30153d90 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.778+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3004f330 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.778+0000 7fdf25ffb640 1 -- 172.21.15.94:0/2177279488 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf2c04b060 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.779+0000 7fdf25ffb640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdf1406b4d0 0x7fdf1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.779+0000 7fdf25ffb640 1 -- 172.21.15.94:0/2177279488 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6359+0+0 (secure 0 0 0) 0x7fdf2c0ac3a0 con 0x7fdf3006b8e0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.779+0000 7fdf34964640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdf1406b4d0 0x7fdf1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.779+0000 7fdf34964640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdf1406b4d0 0x7fdf1406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdf20006fd0 tx=0x7fdf20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.783+0000 7fdf25ffb640 1 -- 172.21.15.94:0/2177279488 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf2c07b020 con 0x7fdf3006b8e0 2024-03-06T07:23:51.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:51.992+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdf3006ce50 con 0x7fdf3006b8e0 2024-03-06T07:23:52.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.658+0000 7fdf25ffb640 1 -- 172.21.15.94:0/2177279488 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v131) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdf2c07a920 con 0x7fdf3006b8e0 2024-03-06T07:23:52.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdf1406b4d0 msgr2=0x7fdf1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdf1406b4d0 0x7fdf1406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdf20006fd0 tx=0x7fdf20008040 comp rx=0 tx=0).stop 2024-03-06T07:23:52.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf3006b8e0 msgr2=0x7fdf301eeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf3006b8e0 0x7fdf301eeac0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c000c00 tx=0x7fdf2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 shutdown_connections 2024-03-06T07:23:52.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdf1406b4d0 0x7fdf1406d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf300c7210 0x7fdf301eab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf300b00d0 0x7fdf30153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 --2- 172.21.15.94:0/2177279488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf3006b8e0 0x7fdf301eeac0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 >> 172.21.15.94:0/2177279488 conn(0x7fdf300ac810 msgr2=0x7fdf301542d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 shutdown_connections 2024-03-06T07:23:52.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.660+0000 7fdf373c9640 1 -- 172.21.15.94:0/2177279488 wait complete. 2024-03-06T07:23:52.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:52.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.780+0000 7f4d8f9dd640 1 Processor -- start 2024-03-06T07:23:52.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.781+0000 7f4d8f9dd640 1 -- start start 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.781+0000 7f4d8f9dd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d880c22b0 0x7f4d88151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.781+0000 7f4d8f9dd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880a9830 0x7f4d8814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.781+0000 7f4d8f9dd640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8806e850 con 0x7f4d8814f490 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.781+0000 7f4d8f9dd640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8806d2b0 con 0x7f4d880c22b0 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.781+0000 7f4d8f9dd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8806d590 con 0x7f4d880a9830 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8d779640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d880c22b0 0x7f4d88151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8cf78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880a9830 0x7f4d8814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8cf78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880a9830 0x7f4d8814ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:37936/0 (socket says 172.21.15.94:37936) 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8df7a640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f4d8814f490 0x7f4d88153390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49342/0 (socket says 172.21.15.94:49342) 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8d779640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d880c22b0 0x7f4d88151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:57884/0 (socket says 172.21.15.94:57884) 2024-03-06T07:23:52.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8cf78640 1 -- 172.21.15.94:0/3477799801 learned_addr learned my addr 172.21.15.94:0/3477799801 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8d779640 1 -- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880a9830 msgr2=0x7f4d8814ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8d779640 1 --2- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880a9830 0x7f4d8814ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8d779640 1 -- 172.21.15.94:0/3477799801 >> v1:172.21.15.145:6789/0 conn(0x7f4d8814f490 legacy=0x7f4d88153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8d779640 1 -- 172.21.15.94:0/3477799801 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8806d870 con 0x7f4d880c22b0 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d7ffff640 1 -- 172.21.15.94:0/3477799801 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561366550 0 0) 0x7f4d80001490 con 0x7f4d8814f490 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.782+0000 7f4d8cf78640 1 --2- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880a9830 0x7f4d8814ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.783+0000 7f4d8d779640 1 --2- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880c22b0 0x7f4d88151e10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4d840097a0 tx=0x7f4d8402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f135802c1a10956 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.783+0000 7f4d7ffff640 1 -- 172.21.15.94:0/3477799801 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84035020 con 0x7f4d880c22b0 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.783+0000 7f4d7ffff640 1 -- 172.21.15.94:0/3477799801 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d84031830 con 0x7f4d880c22b0 2024-03-06T07:23:52.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.783+0000 7f4d7ffff640 1 -- 172.21.15.94:0/3477799801 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84031c00 con 0x7f4d880c22b0 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880c22b0 msgr2=0x7f4d88151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880c22b0 0x7f4d88151e10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4d840097a0 tx=0x7f4d8402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/3477799801 shutdown_connections 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880a9830 0x7f4d8814ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/3477799801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880c22b0 0x7f4d88151e10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/3477799801 >> 172.21.15.94:0/3477799801 conn(0x7f4d880c65b0 msgr2=0x7f4d880c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/3477799801 shutdown_connections 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.784+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/3477799801 wait complete. 2024-03-06T07:23:52.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 Processor -- start 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 -- start start 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880a9830 0x7f4d88150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880c22b0 0x7f4d881dc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d8814f490 0x7f4d881e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880c2810 con 0x7f4d8814f490 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880c2690 con 0x7f4d880a9830 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8f9dd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d88153aa0 con 0x7f4d880c22b0 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.785+0000 7f4d8df7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d8814f490 0x7f4d881e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d8814f490 0x7f4d881e8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39062/0 (socket says 172.21.15.94:39062) 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 -- 172.21.15.94:0/1354623040 learned_addr learned my addr 172.21.15.94:0/1354623040 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8cf78640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880c22b0 0x7f4d881dc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8d779640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880a9830 0x7f4d88150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 -- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880c22b0 msgr2=0x7f4d881dc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880c22b0 0x7f4d881dc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 -- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880a9830 msgr2=0x7f4d88150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880a9830 0x7f4d88150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 -- 172.21.15.94:0/1354623040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d881ec5d0 con 0x7f4d8814f490 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8df7a640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d8814f490 0x7f4d881e8be0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c730 tx=0x7f4d8000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d7dffb640 1 -- 172.21.15.94:0/1354623040 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80018070 con 0x7f4d8814f490 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d7dffb640 1 -- 172.21.15.94:0/1354623040 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8000ce40 con 0x7f4d8814f490 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d881ec8b0 con 0x7f4d8814f490 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d7dffb640 1 -- 172.21.15.94:0/1354623040 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8000a860 con 0x7f4d8814f490 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.786+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d880ab610 con 0x7f4d8814f490 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.787+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8804f330 con 0x7f4d8814f490 2024-03-06T07:23:52.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.788+0000 7f4d7dffb640 1 -- 172.21.15.94:0/1354623040 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d8001b070 con 0x7f4d8814f490 2024-03-06T07:23:52.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.788+0000 7f4d7dffb640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d6c06b4d0 0x7f4d6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.788+0000 7f4d7dffb640 1 -- 172.21.15.94:0/1354623040 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f4d80089950 con 0x7f4d8814f490 2024-03-06T07:23:52.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.788+0000 7f4d8d779640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d6c06b4d0 0x7f4d6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.789+0000 7f4d8d779640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d6c06b4d0 0x7f4d6c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4d84036000 tx=0x7f4d8403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.792+0000 7f4d7dffb640 1 -- 172.21.15.94:0/1354623040 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d8806e850 con 0x7f4d8814f490 2024-03-06T07:23:52.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:52.969+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4d88154ed0 con 0x7f4d8814f490 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.824+0000 7f4d7dffb640 1 -- 172.21.15.94:0/1354623040 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v139) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4d80057f80 con 0x7f4d8814f490 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d6c06b4d0 msgr2=0x7f4d6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d6c06b4d0 0x7f4d6c06d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4d84036000 tx=0x7f4d8403d040 comp rx=0 tx=0).stop 2024-03-06T07:23:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d8814f490 msgr2=0x7f4d881e8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d8814f490 0x7f4d881e8be0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c730 tx=0x7f4d8000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 shutdown_connections 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d6c06b4d0 0x7f4d6c06d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d8814f490 0x7f4d881e8be0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d880c22b0 0x7f4d881dc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 --2- 172.21.15.94:0/1354623040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d880a9830 0x7f4d88150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 >> 172.21.15.94:0/1354623040 conn(0x7f4d880c65b0 msgr2=0x7f4d880c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.827+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 shutdown_connections 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.828+0000 7f4d8f9dd640 1 -- 172.21.15.94:0/1354623040 wait complete. 2024-03-06T07:23:53.842 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:53.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.950+0000 7f25d4da0640 1 Processor -- start 2024-03-06T07:23:53.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.950+0000 7f25d4da0640 1 -- start start 2024-03-06T07:23:53.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25d4da0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00abb60 0x7f25d01502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25d4da0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f25d00c1d10 0x7f25d0156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25d4da0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25d00c2920 0x7f25d0156980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25d4da0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d00711f0 con 0x7f25d00c2920 2024-03-06T07:23:53.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25d4da0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d0070d60 con 0x7f25d00c1d10 2024-03-06T07:23:53.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25d4da0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0070430 con 0x7f25d00abb60 2024-03-06T07:23:53.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25ced9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25d00c2920 0x7f25d0156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25ced9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25d00c2920 0x7f25d0156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39068/0 (socket says 172.21.15.94:39068) 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25ced9d640 1 -- 172.21.15.94:0/2039896735 learned_addr learned my addr 172.21.15.94:0/2039896735 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25cdd9b640 1 --2- 172.21.15.94:0/2039896735 >> v2:172.21.15.145:3300/0 conn(0x7f25d00c1d10 0x7f25d0156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.951+0000 7f25ce59c640 1 --2- 172.21.15.94:0/2039896735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00abb60 0x7f25d01502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25ced9d640 1 -- 172.21.15.94:0/2039896735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00abb60 msgr2=0x7f25d01502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25ced9d640 1 --2- 172.21.15.94:0/2039896735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00abb60 0x7f25d01502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25ced9d640 1 -- 172.21.15.94:0/2039896735 >> v2:172.21.15.145:3300/0 conn(0x7f25d00c1d10 msgr2=0x7f25d0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25ced9d640 1 --2- 172.21.15.94:0/2039896735 >> v2:172.21.15.145:3300/0 conn(0x7f25d00c1d10 0x7f25d0156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25ced9d640 1 -- 172.21.15.94:0/2039896735 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0070710 con 0x7f25d00c2920 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25ced9d640 1 --2- 172.21.15.94:0/2039896735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d0156980 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f25d00ac3f0 tx=0x7f25c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45966f54189d797 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25cd59a640 1 -- 172.21.15.94:0/2039896735 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c802c9e0 con 0x7f25d00c2920 2024-03-06T07:23:53.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25cd59a640 1 -- 172.21.15.94:0/2039896735 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c802cb80 con 0x7f25d00c2920 2024-03-06T07:23:53.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.952+0000 7f25cd59a640 1 -- 172.21.15.94:0/2039896735 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c80343f0 con 0x7f25d00c2920 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 -- 172.21.15.94:0/2039896735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 msgr2=0x7f25d0156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 --2- 172.21.15.94:0/2039896735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d0156980 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f25d00ac3f0 tx=0x7f25c802be40 comp rx=0 tx=0).stop 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 -- 172.21.15.94:0/2039896735 shutdown_connections 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 --2- 172.21.15.94:0/2039896735 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d0156980 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 --2- 172.21.15.94:0/2039896735 >> v2:172.21.15.145:3300/0 conn(0x7f25d00c1d10 0x7f25d0156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 --2- 172.21.15.94:0/2039896735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00abb60 0x7f25d01502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 -- 172.21.15.94:0/2039896735 >> 172.21.15.94:0/2039896735 conn(0x7f25d00c65b0 msgr2=0x7f25d00c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 -- 172.21.15.94:0/2039896735 shutdown_connections 2024-03-06T07:23:53.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.953+0000 7f25d4da0640 1 -- 172.21.15.94:0/2039896735 wait complete. 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.954+0000 7f25d4da0640 1 Processor -- start 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.954+0000 7f25d4da0640 1 -- start start 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25d4da0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d00abb60 0x7f25d0151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25d4da0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d01e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25d4da0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00c54b0 0x7f25d00bbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25d4da0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0157040 con 0x7f25d00c2920 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25d4da0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0156ec0 con 0x7f25d00abb60 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25d4da0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00abf40 con 0x7f25d00c54b0 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d01e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d01e94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39078/0 (socket says 172.21.15.94:39078) 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 -- 172.21.15.94:0/3017287679 learned_addr learned my addr 172.21.15.94:0/3017287679 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:53.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25ce59c640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d00abb60 0x7f25d0151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25ced9d640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00c54b0 0x7f25d00bbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 -- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00c54b0 msgr2=0x7f25d00bbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00c54b0 0x7f25d00bbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 -- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d00abb60 msgr2=0x7f25d0151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d00abb60 0x7f25d0151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.955+0000 7f25cdd9b640 1 -- 172.21.15.94:0/3017287679 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00bd770 con 0x7f25d00c2920 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.956+0000 7f25cdd9b640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d01e94d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f25c400d830 tx=0x7f25c400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.956+0000 7f25beffd640 1 -- 172.21.15.94:0/3017287679 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c400b6c0 con 0x7f25d00c2920 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.956+0000 7f25beffd640 1 -- 172.21.15.94:0/3017287679 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c4019070 con 0x7f25d00c2920 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.956+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d00c5970 con 0x7f25d00c2920 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.956+0000 7f25beffd640 1 -- 172.21.15.94:0/3017287679 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c400ba80 con 0x7f25d00c2920 2024-03-06T07:23:53.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.956+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d0150f40 con 0x7f25d00c2920 2024-03-06T07:23:53.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.957+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d004f330 con 0x7f25d00c2920 2024-03-06T07:23:53.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.957+0000 7f25beffd640 1 -- 172.21.15.94:0/3017287679 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25c401e460 con 0x7f25d00c2920 2024-03-06T07:23:53.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.958+0000 7f25beffd640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.958+0000 7f25beffd640 1 -- 172.21.15.94:0/3017287679 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5921+0+0 (secure 0 0 0) 0x7f25c408a730 con 0x7f25d00c2920 2024-03-06T07:23:53.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.958+0000 7f25ce59c640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.959+0000 7f25ce59c640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f25b006b4d0 0x7f25b006d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25c0006fd0 tx=0x7f25c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:53.962+0000 7f25beffd640 1 -- 172.21.15.94:0/3017287679 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25c4058f10 con 0x7f25d00c2920 2024-03-06T07:23:54.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.133+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f25d014ea00 con 0x7f25d00c2920 2024-03-06T07:23:54.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.828+0000 7f25beffd640 1 -- 172.21.15.94:0/3017287679 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v140) v1 ==== 169+0+0 (secure 0 0 0) 0x7f25c405cfb0 con 0x7f25d00c2920 2024-03-06T07:23:54.831 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:54.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f25b006b4d0 msgr2=0x7f25b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f25b006b4d0 0x7f25b006d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25c0006fd0 tx=0x7f25c0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 msgr2=0x7f25d01e94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d01e94d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f25c400d830 tx=0x7f25c400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 shutdown_connections 2024-03-06T07:23:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f25b006b4d0 0x7f25b006d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d00c54b0 0x7f25d00bbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25d00c2920 0x7f25d01e94d0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 --2- 172.21.15.94:0/3017287679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d00abb60 0x7f25d0151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.832+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 >> 172.21.15.94:0/3017287679 conn(0x7f25d00c65b0 msgr2=0x7f25d01500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.833+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 shutdown_connections 2024-03-06T07:23:54.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.833+0000 7f25d4da0640 1 -- 172.21.15.94:0/3017287679 wait complete. 2024-03-06T07:23:54.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.955+0000 7f478b33a640 1 Processor -- start 2024-03-06T07:23:54.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.956+0000 7f478b33a640 1 -- start start 2024-03-06T07:23:54.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.956+0000 7f478b33a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4784155c30 0x7f478406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.956+0000 7f478b33a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47840c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.956+0000 7f478b33a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47840b0450 con 0x7f47840b0070 2024-03-06T07:23:54.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.956+0000 7f478b33a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478406cfb0 con 0x7f4784155c30 2024-03-06T07:23:54.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.956+0000 7f478b33a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478406c020 con 0x7f47840c5380 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47888d5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4784155c30 0x7f478406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47898d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47840c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47890d6640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f47840b0070 0x7f47840c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49358/0 (socket says 172.21.15.94:49358) 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47890d6640 1 -- 172.21.15.94:0/1691017950 learned_addr learned my addr 172.21.15.94:0/1691017950 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47888d5640 1 -- 172.21.15.94:0/1691017950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 msgr2=0x7f47840c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47888d5640 1 --2- 172.21.15.94:0/1691017950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47840c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47888d5640 1 -- 172.21.15.94:0/1691017950 >> v1:172.21.15.145:6789/0 conn(0x7f47840b0070 legacy=0x7f47840c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.957+0000 7f47888d5640 1 -- 172.21.15.94:0/1691017950 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47841560e0 con 0x7f4784155c30 2024-03-06T07:23:54.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.958+0000 7f47888d5640 1 --2- 172.21.15.94:0/1691017950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 0x7f478406ca70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f47740097a0 tx=0x7f477402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5131b1826634711b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.958+0000 7f477bfff640 1 -- 172.21.15.94:0/1691017950 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740315a0 con 0x7f4784155c30 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.958+0000 7f477bfff640 1 -- 172.21.15.94:0/1691017950 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774037070 con 0x7f4784155c30 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.958+0000 7f477bfff640 1 -- 172.21.15.94:0/1691017950 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774031960 con 0x7f4784155c30 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 -- 172.21.15.94:0/1691017950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 msgr2=0x7f478406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 --2- 172.21.15.94:0/1691017950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 0x7f478406ca70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f47740097a0 tx=0x7f477402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 -- 172.21.15.94:0/1691017950 shutdown_connections 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 --2- 172.21.15.94:0/1691017950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47840c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 --2- 172.21.15.94:0/1691017950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 0x7f478406ca70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 -- 172.21.15.94:0/1691017950 >> 172.21.15.94:0/1691017950 conn(0x7f47840ac590 msgr2=0x7f47840ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 -- 172.21.15.94:0/1691017950 shutdown_connections 2024-03-06T07:23:54.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.959+0000 7f478b33a640 1 -- 172.21.15.94:0/1691017950 wait complete. 2024-03-06T07:23:54.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.960+0000 7f478b33a640 1 Processor -- start 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.960+0000 7f478b33a640 1 -- start start 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f478b33a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47840b0070 0x7f4784154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f478b33a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47841eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f478b33a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 0x7f47841eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f478b33a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478406c270 con 0x7f47840b0070 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f478b33a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47840c7ad0 con 0x7f4784155c30 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f478b33a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478406c550 con 0x7f47840c5380 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47840b0070 0x7f4784154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47840b0070 0x7f4784154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39088/0 (socket says 172.21.15.94:39088) 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 -- 172.21.15.94:0/3084052698 learned_addr learned my addr 172.21.15.94:0/3084052698 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:54.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47898d7640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 0x7f47841eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47888d5640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47841eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 -- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 msgr2=0x7f47841eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47841eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 -- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 msgr2=0x7f47841eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 0x7f47841eb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 -- 172.21.15.94:0/3084052698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47840ac310 con 0x7f47840b0070 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f47890d6640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47840b0070 0x7f4784154920 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c3c0 tx=0x7f477c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f4779ffb640 1 -- 172.21.15.94:0/3084052698 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c017070 con 0x7f47840b0070 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f4779ffb640 1 -- 172.21.15.94:0/3084052698 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477c005ce0 con 0x7f47840b0070 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f4779ffb640 1 -- 172.21.15.94:0/3084052698 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c0072f0 con 0x7f47840b0070 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.961+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47841eeb40 con 0x7f47840b0070 2024-03-06T07:23:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.962+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47840c46a0 con 0x7f47840b0070 2024-03-06T07:23:54.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.963+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478404f330 con 0x7f47840b0070 2024-03-06T07:23:54.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.963+0000 7f4779ffb640 1 -- 172.21.15.94:0/3084052698 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f477c01a070 con 0x7f47840b0070 2024-03-06T07:23:54.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.963+0000 7f4779ffb640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f476806b500 0x7f476806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.963+0000 7f4779ffb640 1 -- 172.21.15.94:0/3084052698 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f477c0893d0 con 0x7f47840b0070 2024-03-06T07:23:54.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.963+0000 7f47888d5640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f476806b500 0x7f476806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.964+0000 7f47888d5640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f476806b500 0x7f476806d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4774036000 tx=0x7f477403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:54.967+0000 7f4779ffb640 1 -- 172.21.15.94:0/3084052698 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47840b0450 con 0x7f47840b0070 2024-03-06T07:23:55.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.143+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f478406d860 con 0x7f47840b0070 2024-03-06T07:23:55.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.146+0000 7f4779ffb640 1 -- 172.21.15.94:0/3084052698 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7f477c057cd0 con 0x7f47840b0070 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f476806b500 msgr2=0x7f476806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f476806b500 0x7f476806d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4774036000 tx=0x7f477403d040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47840b0070 msgr2=0x7f4784154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47840b0070 0x7f4784154920 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f477c00c3c0 tx=0x7f477c00c880 comp rx=0 tx=0).stop 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 shutdown_connections 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f476806b500 0x7f476806d9c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4784155c30 0x7f47841eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840c5380 0x7f47841eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 --2- 172.21.15.94:0/3084052698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47840b0070 0x7f4784154920 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 >> 172.21.15.94:0/3084052698 conn(0x7f47840ac590 msgr2=0x7f47840af0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 shutdown_connections 2024-03-06T07:23:55.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.149+0000 7f478b33a640 1 -- 172.21.15.94:0/3084052698 wait complete. 2024-03-06T07:23:55.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.261+0000 7f9c12763640 1 Processor -- start 2024-03-06T07:23:55.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.261+0000 7f9c12763640 1 -- start start 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c12763640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9c0c06c900 0x7f9c0c0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c12763640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0c3ef0 con 0x7f9c0c0c5990 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c12763640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c1556d0 con 0x7f9c0c0ab170 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c12763640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c1559b0 con 0x7f9c0c06c900 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c0bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9c0c0c5990 0x7f9c0c06c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47780/0 (socket says 172.21.15.94:47780) 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c0bfff640 1 -- 172.21.15.94:0/1289731889 learned_addr learned my addr 172.21.15.94:0/1289731889 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477595224 0 0) 0x7f9c0c0c3ef0 con 0x7f9c0c0c5990 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c0b7fe640 1 --2- 172.21.15.94:0/1289731889 >> v2:172.21.15.145:3300/0 conn(0x7f9c0c06c900 0x7f9c0c0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.262+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4005730 con 0x7f9c0c0c5990 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299082221 0 0) 0x7f9bf4005730 con 0x7f9c0c0c5990 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 >> v2:172.21.15.145:3300/0 conn(0x7f9c0c06c900 msgr2=0x7f9c0c0c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 --2- 172.21.15.94:0/1289731889 >> v2:172.21.15.145:3300/0 conn(0x7f9c0c06c900 0x7f9c0c0c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 >> v1:172.21.15.145:6789/0 conn(0x7f9c0c0ab170 legacy=0x7f9c0c0a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c155c90 con 0x7f9c0c0c5990 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9bfc001670 con 0x7f9c0c0c5990 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 >> v1:172.21.15.94:6789/0 conn(0x7f9c0c0c5990 legacy=0x7f9c0c06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bf4005bc0 0x7f9bf4005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bf4006a10 0x7f9bf4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf40073d0 0x7f9bf400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c1559b0 con 0x7f9bf4005bc0 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf400e270 con 0x7f9bf4006a10 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf400e470 con 0x7f9bf40073d0 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0bfff640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bf4005bc0 0x7f9bf4005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0b7fe640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bf4006a10 0x7f9bf4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c10d00640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf40073d0 0x7f9bf400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0bfff640 1 -- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf40073d0 msgr2=0x7f9bf400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0bfff640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf40073d0 0x7f9bf400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0bfff640 1 -- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bf4006a10 msgr2=0x7f9bf4006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0bfff640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bf4006a10 0x7f9bf4006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0bfff640 1 -- 172.21.15.94:0/1289731889 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4005730 con 0x7f9bf4005bc0 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c10d00640 1 -- 172.21.15.94:0/1289731889 reap_dead start 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.263+0000 7f9c0b7fe640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bf4006a10 0x7f9bf4006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c0bfff640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bf4005bc0 0x7f9bf4005fe0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00c240 tx=0x7f9bf4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec56a19325bcb20d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc007cf0 con 0x7f9bf4005bc0 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc030070 con 0x7f9bf4005bc0 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c0affd640 1 -- 172.21.15.94:0/1289731889 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc005d50 con 0x7f9bf4005bc0 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c12763640 1 -- 172.21.15.94:0/1289731889 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bf4005bc0 msgr2=0x7f9bf4005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c12763640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bf4005bc0 0x7f9bf4005fe0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00c240 tx=0x7f9bf4007a40 comp rx=0 tx=0).stop 2024-03-06T07:23:55.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c12763640 1 -- 172.21.15.94:0/1289731889 shutdown_connections 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c12763640 1 --2- 172.21.15.94:0/1289731889 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bf4005bc0 0x7f9bf4005fe0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c12763640 1 -- 172.21.15.94:0/1289731889 >> 172.21.15.94:0/1289731889 conn(0x7f9c0c0c6710 msgr2=0x7f9c0c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c12763640 1 -- 172.21.15.94:0/1289731889 shutdown_connections 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.264+0000 7f9c12763640 1 -- 172.21.15.94:0/1289731889 wait complete. 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.265+0000 7f9c12763640 1 Processor -- start 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c12763640 1 -- start start 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c12763640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf4005bc0 0x7f9c0c14d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c12763640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c0c05e0 0x7f9c0c0bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c12763640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c0c141a10 0x7f9c0c141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c12763640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c156610 con 0x7f9c0c141a10 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c12763640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c156490 con 0x7f9c0c0c05e0 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c12763640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0c3af0 con 0x7f9bf4005bc0 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c10d00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c0c141a10 0x7f9c0c141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c10d00640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c0c141a10 0x7f9c0c141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39102/0 (socket says 172.21.15.94:39102) 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c10d00640 1 -- 172.21.15.94:0/523592122 learned_addr learned my addr 172.21.15.94:0/523592122 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c0bfff640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf4005bc0 0x7f9c0c14d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.266+0000 7f9c0b7fe640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c0c05e0 0x7f9c0c0bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c10d00640 1 -- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf4005bc0 msgr2=0x7f9c0c14d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c10d00640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf4005bc0 0x7f9c0c14d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c10d00640 1 -- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c0c05e0 msgr2=0x7f9c0c0bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c10d00640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c0c05e0 0x7f9c0c0bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c10d00640 1 -- 172.21.15.94:0/523592122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c14e2b0 con 0x7f9c0c141a10 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c10d00640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c0c141a10 0x7f9c0c141e90 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0abe90 tx=0x7f9c0400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c08ff9640 1 -- 172.21.15.94:0/523592122 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04003970 con 0x7f9c0c141a10 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c08ff9640 1 -- 172.21.15.94:0/523592122 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c04018070 con 0x7f9c0c141a10 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c14dfd0 con 0x7f9c0c141a10 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c08ff9640 1 -- 172.21.15.94:0/523592122 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04003d70 con 0x7f9c0c141a10 2024-03-06T07:23:55.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.267+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c1449d0 con 0x7f9c0c141a10 2024-03-06T07:23:55.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.268+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c04f330 con 0x7f9c0c141a10 2024-03-06T07:23:55.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.268+0000 7f9c08ff9640 1 -- 172.21.15.94:0/523592122 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c040050d0 con 0x7f9c0c141a10 2024-03-06T07:23:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.269+0000 7f9c08ff9640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf806b500 0x7f9bf806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.269+0000 7f9c08ff9640 1 -- 172.21.15.94:0/523592122 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f9c04089af0 con 0x7f9c0c141a10 2024-03-06T07:23:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.269+0000 7f9c0bfff640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf806b500 0x7f9bf806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.270+0000 7f9c0bfff640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf806b500 0x7f9bf806d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc034000 tx=0x7f9bfc007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.273+0000 7f9c08ff9640 1 -- 172.21.15.94:0/523592122 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c04058470 con 0x7f9c0c141a10 2024-03-06T07:23:55.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.444+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f9c0c0c4ba0 con 0x7f9c0c141a10 2024-03-06T07:23:55.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.448+0000 7f9c08ff9640 1 -- 172.21.15.94:0/523592122 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v50) v1 ==== 96+0+0 (secure 0 0 0) 0x7f9c0405c510 con 0x7f9c0c141a10 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf806b500 msgr2=0x7f9bf806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf806b500 0x7f9bf806d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc034000 tx=0x7f9bfc007000 comp rx=0 tx=0).stop 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c0c141a10 msgr2=0x7f9c0c141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c0c141a10 0x7f9c0c141e90 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0abe90 tx=0x7f9c0400ec10 comp rx=0 tx=0).stop 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 shutdown_connections 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf806b500 0x7f9bf806d9c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c0c141a10 0x7f9c0c141e90 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c0c05e0 0x7f9c0c0bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 --2- 172.21.15.94:0/523592122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bf4005bc0 0x7f9c0c14d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 >> 172.21.15.94:0/523592122 conn(0x7f9c0c0c6710 msgr2=0x7f9c0c0aa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 shutdown_connections 2024-03-06T07:23:55.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.450+0000 7f9c12763640 1 -- 172.21.15.94:0/523592122 wait complete. 2024-03-06T07:23:55.463 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:23:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.563+0000 7fdb1d9bc640 1 Processor -- start 2024-03-06T07:23:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.564+0000 7fdb1d9bc640 1 -- start start 2024-03-06T07:23:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.564+0000 7fdb1d9bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb180b0070 0x7fdb18155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.564+0000 7fdb1d9bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb1806ca70 0x7fdb180c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.564+0000 7fdb1d9bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb180b0450 con 0x7fdb180c5380 2024-03-06T07:23:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.564+0000 7fdb1d9bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb1806c020 con 0x7fdb180b0070 2024-03-06T07:23:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.564+0000 7fdb1d9bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1806cf20 con 0x7fdb1806ca70 2024-03-06T07:23:55.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb16ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb180b0070 0x7fdb18155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb167fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb1806ca70 0x7fdb180c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb16ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb180b0070 0x7fdb18155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:57966/0 (socket says 172.21.15.94:57966) 2024-03-06T07:23:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb167fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb1806ca70 0x7fdb180c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:38010/0 (socket says 172.21.15.94:38010) 2024-03-06T07:23:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb177fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fdb180c5380 0x7fdb1814e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49368/0 (socket says 172.21.15.94:49368) 2024-03-06T07:23:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb16ffd640 1 -- 172.21.15.94:0/174226225 learned_addr learned my addr 172.21.15.94:0/174226225 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:55.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb16ffd640 1 -- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb1806ca70 msgr2=0x7fdb180c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb16ffd640 1 --2- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb1806ca70 0x7fdb180c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb16ffd640 1 -- 172.21.15.94:0/174226225 >> v1:172.21.15.145:6789/0 conn(0x7fdb180c5380 legacy=0x7fdb1814e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb16ffd640 1 -- 172.21.15.94:0/174226225 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb180c77f0 con 0x7fdb180b0070 2024-03-06T07:23:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb15ffb640 1 -- 172.21.15.94:0/174226225 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617832537 0 0) 0x7fdb10001110 con 0x7fdb180c5380 2024-03-06T07:23:55.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.565+0000 7fdb167fc640 1 --2- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb1806ca70 0x7fdb180c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:55.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb16ffd640 1 --2- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 0x7fdb18155c30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009e50 tx=0x7fdb0802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=887e380b8b2c6d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb15ffb640 1 -- 172.21.15.94:0/174226225 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0802ca70 con 0x7fdb180b0070 2024-03-06T07:23:55.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb15ffb640 1 -- 172.21.15.94:0/174226225 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0802cc10 con 0x7fdb180b0070 2024-03-06T07:23:55.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb15ffb640 1 -- 172.21.15.94:0/174226225 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb080343f0 con 0x7fdb180b0070 2024-03-06T07:23:55.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 msgr2=0x7fdb18155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 0x7fdb18155c30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009e50 tx=0x7fdb0802bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/174226225 shutdown_connections 2024-03-06T07:23:55.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb1806ca70 0x7fdb180c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/174226225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 0x7fdb18155c30 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/174226225 >> 172.21.15.94:0/174226225 conn(0x7fdb180ac590 msgr2=0x7fdb180ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/174226225 shutdown_connections 2024-03-06T07:23:55.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.566+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/174226225 wait complete. 2024-03-06T07:23:55.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.567+0000 7fdb1d9bc640 1 Processor -- start 2024-03-06T07:23:55.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb1d9bc640 1 -- start start 2024-03-06T07:23:55.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb1d9bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb1806ca70 0x7fdb181545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb1d9bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 0x7fdb1814ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb1d9bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb180c5380 0x7fdb1814b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb1d9bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1806c550 con 0x7fdb1806ca70 2024-03-06T07:23:55.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb1d9bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1806c270 con 0x7fdb180b0070 2024-03-06T07:23:55.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb1d9bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1806da80 con 0x7fdb180c5380 2024-03-06T07:23:55.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb1806ca70 0x7fdb181545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb16ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb1806ca70 0x7fdb181545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39114/0 (socket says 172.21.15.94:39114) 2024-03-06T07:23:55.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.568+0000 7fdb16ffd640 1 -- 172.21.15.94:0/783270426 learned_addr learned my addr 172.21.15.94:0/783270426 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:55.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb167fc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 0x7fdb1814ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb177fe640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb180c5380 0x7fdb1814b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb16ffd640 1 -- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb180c5380 msgr2=0x7fdb1814b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb16ffd640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb180c5380 0x7fdb1814b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb16ffd640 1 -- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 msgr2=0x7fdb1814ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb16ffd640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 0x7fdb1814ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb16ffd640 1 -- 172.21.15.94:0/783270426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb181592f0 con 0x7fdb1806ca70 2024-03-06T07:23:55.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb16ffd640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb1806ca70 0x7fdb181545b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fdb08036040 tx=0x7fdb08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdafffff640 1 -- 172.21.15.94:0/783270426 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08043020 con 0x7fdb1806ca70 2024-03-06T07:23:55.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdafffff640 1 -- 172.21.15.94:0/783270426 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0803e450 con 0x7fdb1806ca70 2024-03-06T07:23:55.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb18157360 con 0x7fdb1806ca70 2024-03-06T07:23:55.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdafffff640 1 -- 172.21.15.94:0/783270426 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08009510 con 0x7fdb1806ca70 2024-03-06T07:23:55.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.569+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb18158ff0 con 0x7fdb1806ca70 2024-03-06T07:23:55.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.570+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1804f330 con 0x7fdb1806ca70 2024-03-06T07:23:55.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.571+0000 7fdafffff640 1 -- 172.21.15.94:0/783270426 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb08046020 con 0x7fdb1806ca70 2024-03-06T07:23:55.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.571+0000 7fdafffff640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0406b4d0 0x7fdb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.571+0000 7fdafffff640 1 -- 172.21.15.94:0/783270426 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fdb080ac720 con 0x7fdb1806ca70 2024-03-06T07:23:55.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.571+0000 7fdb167fc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0406b4d0 0x7fdb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.572+0000 7fdb167fc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0406b4d0 0x7fdb0406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009560 tx=0x7fdb0c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.575+0000 7fdafffff640 1 -- 172.21.15.94:0/783270426 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb0804d030 con 0x7fdb1806ca70 2024-03-06T07:23:55.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.747+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fdb1806d860 con 0x7fdb1806ca70 2024-03-06T07:23:55.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.747+0000 7fdafffff640 1 -- 172.21.15.94:0/783270426 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v50) v1 ==== 96+0+0 (secure 0 0 0) 0x7fdb0807b020 con 0x7fdb1806ca70 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0406b4d0 msgr2=0x7fdb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0406b4d0 0x7fdb0406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009560 tx=0x7fdb0c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb1806ca70 msgr2=0x7fdb181545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb1806ca70 0x7fdb181545b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fdb08036040 tx=0x7fdb08004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 shutdown_connections 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0406b4d0 0x7fdb0406d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb180c5380 0x7fdb1814b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb180b0070 0x7fdb1814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 --2- 172.21.15.94:0/783270426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb1806ca70 0x7fdb181545b0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 >> 172.21.15.94:0/783270426 conn(0x7fdb180ac590 msgr2=0x7fdb18151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 shutdown_connections 2024-03-06T07:23:55.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.749+0000 7fdb1d9bc640 1 -- 172.21.15.94:0/783270426 wait complete. 2024-03-06T07:23:55.761 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-03-06T07:23:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.861+0000 7f60a4516640 1 Processor -- start 2024-03-06T07:23:55.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.861+0000 7f60a4516640 1 -- start start 2024-03-06T07:23:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a4516640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f609c14f790 0x7f609c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a4516640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609c0c28d0 con 0x7f609c155a40 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a4516640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609c0c2bb0 con 0x7f609c0c1aa0 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a4516640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f609c0c2680 con 0x7f609c14f790 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a2ab3640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f609c155a40 0x7f609c155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47796/0 (socket says 172.21.15.94:47796) 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a2ab3640 1 -- 172.21.15.94:0/872750253 learned_addr learned my addr 172.21.15.94:0/872750253 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a22b2640 1 --2- 172.21.15.94:0/872750253 >> v2:172.21.15.145:3300/0 conn(0x7f609c14f790 0x7f609c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547419011 0 0) 0x7f609c0c28d0 con 0x7f609c155a40 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6088003520 con 0x7f609c155a40 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2333501643 0 0) 0x7f6088003520 con 0x7f609c155a40 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 >> v2:172.21.15.145:3300/0 conn(0x7f609c14f790 msgr2=0x7f609c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 --2- 172.21.15.94:0/872750253 >> v2:172.21.15.145:3300/0 conn(0x7f609c14f790 0x7f609c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 >> v1:172.21.15.145:6789/0 conn(0x7f609c0c1aa0 legacy=0x7f609c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c0c2e90 con 0x7f609c155a40 2024-03-06T07:23:55.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f60940019f0 con 0x7f609c155a40 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 >> v1:172.21.15.94:6789/0 conn(0x7f609c155a40 legacy=0x7f609c155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.862+0000 7f60a12b0640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880039b0 0x7f6088003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a12b0640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6088004800 0x7f6088004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a12b0640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60880051c0 0x7f608800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c0c2680 con 0x7f60880051c0 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608800c090 con 0x7f6088004800 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608800c290 con 0x7f60880039b0 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a2ab3640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60880051c0 0x7f608800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a22b2640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880039b0 0x7f6088003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a1ab1640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6088004800 0x7f6088004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a2ab3640 1 -- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880039b0 msgr2=0x7f6088003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a2ab3640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880039b0 0x7f6088003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a2ab3640 1 -- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6088004800 msgr2=0x7f6088004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a2ab3640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6088004800 0x7f6088004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.863+0000 7f60a2ab3640 1 -- 172.21.15.94:0/872750253 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088003520 con 0x7f60880051c0 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a2ab3640 1 -- 172.21.15.94:0/872750253 reap_dead start 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a2ab3640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60880051c0 0x7f608800b9c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f609400a840 tx=0x7f6088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2d63eb9f41673f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609400a460 con 0x7f60880051c0 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6094034650 con 0x7f60880051c0 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a12b0640 1 -- 172.21.15.94:0/872750253 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094034a40 con 0x7f60880051c0 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a4516640 1 -- 172.21.15.94:0/872750253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60880051c0 msgr2=0x7f608800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a4516640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60880051c0 0x7f608800b9c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f609400a840 tx=0x7f6088005830 comp rx=0 tx=0).stop 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a4516640 1 -- 172.21.15.94:0/872750253 shutdown_connections 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a4516640 1 --2- 172.21.15.94:0/872750253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60880051c0 0x7f608800b9c0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a4516640 1 -- 172.21.15.94:0/872750253 >> 172.21.15.94:0/872750253 conn(0x7f609c0c6710 msgr2=0x7f609c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a4516640 1 -- 172.21.15.94:0/872750253 shutdown_connections 2024-03-06T07:23:55.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.864+0000 7f60a4516640 1 -- 172.21.15.94:0/872750253 wait complete. 2024-03-06T07:23:55.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.865+0000 7f60a4516640 1 Processor -- start 2024-03-06T07:23:55.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a4516640 1 -- start start 2024-03-06T07:23:55.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a4516640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880051c0 0x7f609c0bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a4516640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f609c1dded0 0x7f609c146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a4516640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f609c147050 0x7f609c1ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a4516640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c06cc00 con 0x7f609c147050 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a4516640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c0c0d60 con 0x7f609c1dded0 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a4516640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c06cd80 con 0x7f60880051c0 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a2ab3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f609c147050 0x7f609c1ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a2ab3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f609c147050 0x7f609c1ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39124/0 (socket says 172.21.15.94:39124) 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a2ab3640 1 -- 172.21.15.94:0/3393517729 learned_addr learned my addr 172.21.15.94:0/3393517729 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a22b2640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880051c0 0x7f609c0bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.866+0000 7f60a1ab1640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f609c1dded0 0x7f609c146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a2ab3640 1 -- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880051c0 msgr2=0x7f609c0bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a2ab3640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880051c0 0x7f609c0bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a2ab3640 1 -- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f609c1dded0 msgr2=0x7f609c146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a2ab3640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f609c1dded0 0x7f609c146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a2ab3640 1 -- 172.21.15.94:0/3393517729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c1474d0 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a2ab3640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f609c147050 0x7f609c1ead60 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f609400a5d0 tx=0x7f609c0c2d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f6092ffd640 1 -- 172.21.15.94:0/3393517729 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094044070 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f6092ffd640 1 -- 172.21.15.94:0/3393517729 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60940019b0 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c14c8b0 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f6092ffd640 1 -- 172.21.15.94:0/3393517729 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094039040 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.867+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c0a71c0 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.868+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c04f330 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.868+0000 7f6092ffd640 1 -- 172.21.15.94:0/3393517729 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6094047070 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.869+0000 7f6092ffd640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608006b500 0x7f608006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.869+0000 7f6092ffd640 1 -- 172.21.15.94:0/3393517729 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f60940ab680 con 0x7f609c147050 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.869+0000 7f60a22b2640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608006b500 0x7f608006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.869+0000 7f60a22b2640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608006b500 0x7f608006d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6098009b40 tx=0x7f6098009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:55.873+0000 7f6092ffd640 1 -- 172.21.15.94:0/3393517729 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f609c14c8b0 con 0x7f609c147050 2024-03-06T07:23:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.110+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7f609c0ab5c0 con 0x7f609c147050 2024-03-06T07:23:56.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.834+0000 7f6092ffd640 1 -- 172.21.15.94:0/3393517729 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v332) v1 ==== 144+0+0 (secure 0 0 0) 0x7f609407a000 con 0x7f609c147050 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608006b500 msgr2=0x7f608006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608006b500 0x7f608006d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6098009b40 tx=0x7f6098009190 comp rx=0 tx=0).stop 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f609c147050 msgr2=0x7f609c1ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f609c147050 0x7f609c1ead60 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f609400a5d0 tx=0x7f609c0c2d80 comp rx=0 tx=0).stop 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 shutdown_connections 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608006b500 0x7f608006d9c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f609c147050 0x7f609c1ead60 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f609c1dded0 0x7f609c146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 --2- 172.21.15.94:0/3393517729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60880051c0 0x7f609c0bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 >> 172.21.15.94:0/3393517729 conn(0x7f609c0c6710 msgr2=0x7f609c072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 shutdown_connections 2024-03-06T07:23:56.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.837+0000 7f60a4516640 1 -- 172.21.15.94:0/3393517729 wait complete. 2024-03-06T07:23:56.852 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will cause failover if another high-affinity standby exists. ... ok 2024-03-06T07:23:56.852 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) That a vanilla standby is preferred over others with mds_join_fs set to another fs. 2024-03-06T07:23:56.852 INFO:tasks.cephfs_test_runner:test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-03-06T07:23:56.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:56.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.956+0000 7f2400b7a640 1 Processor -- start 2024-03-06T07:23:56.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.956+0000 7f2400b7a640 1 -- start start 2024-03-06T07:23:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f2400b7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc14f790 0x7f23fc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f2400b7a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f23fc0c1aa0 0x7f23fc0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f2400b7a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23fc0c5c60 0x7f23fc155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f2400b7a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23fc0c2680 con 0x7f23fc0c5c60 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f2400b7a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23fc0c28d0 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f2400b7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc0c2bb0 con 0x7f23fc14f790 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23fc0c5c60 0x7f23fc155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f23fc0c5c60 0x7f23fc155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39138/0 (socket says 172.21.15.94:39138) 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 -- 172.21.15.94:0/2175423935 learned_addr learned my addr 172.21.15.94:0/2175423935 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fa59c640 1 --2- 172.21.15.94:0/2175423935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc14f790 0x7f23fc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23f9d9b640 1 --2- 172.21.15.94:0/2175423935 >> v2:172.21.15.145:3300/0 conn(0x7f23fc0c1aa0 0x7f23fc0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 -- 172.21.15.94:0/2175423935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc14f790 msgr2=0x7f23fc1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 --2- 172.21.15.94:0/2175423935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc14f790 0x7f23fc1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 -- 172.21.15.94:0/2175423935 >> v2:172.21.15.145:3300/0 conn(0x7f23fc0c1aa0 msgr2=0x7f23fc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 --2- 172.21.15.94:0/2175423935 >> v2:172.21.15.145:3300/0 conn(0x7f23fc0c1aa0 0x7f23fc0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.957+0000 7f23fad9d640 1 -- 172.21.15.94:0/2175423935 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0c2e90 con 0x7f23fc0c5c60 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.958+0000 7f23fad9d640 1 --2- 172.21.15.94:0/2175423935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c5c60 0x7f23fc155a40 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0c76f0 tx=0x7f23f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8ff7ce0b71f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.958+0000 7f23f959a640 1 -- 172.21.15.94:0/2175423935 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4035020 con 0x7f23fc0c5c60 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.958+0000 7f23f959a640 1 -- 172.21.15.94:0/2175423935 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f4031870 con 0x7f23fc0c5c60 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.958+0000 7f23f959a640 1 -- 172.21.15.94:0/2175423935 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4031c40 con 0x7f23fc0c5c60 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 -- 172.21.15.94:0/2175423935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c5c60 msgr2=0x7f23fc155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 --2- 172.21.15.94:0/2175423935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c5c60 0x7f23fc155a40 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0c76f0 tx=0x7f23f402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 -- 172.21.15.94:0/2175423935 shutdown_connections 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 --2- 172.21.15.94:0/2175423935 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c5c60 0x7f23fc155a40 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 --2- 172.21.15.94:0/2175423935 >> v2:172.21.15.145:3300/0 conn(0x7f23fc0c1aa0 0x7f23fc0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 --2- 172.21.15.94:0/2175423935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc14f790 0x7f23fc1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 -- 172.21.15.94:0/2175423935 >> 172.21.15.94:0/2175423935 conn(0x7f23fc0c6710 msgr2=0x7f23fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 -- 172.21.15.94:0/2175423935 shutdown_connections 2024-03-06T07:23:56.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.959+0000 7f2400b7a640 1 -- 172.21.15.94:0/2175423935 wait complete. 2024-03-06T07:23:56.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.960+0000 7f2400b7a640 1 Processor -- start 2024-03-06T07:23:56.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.960+0000 7f2400b7a640 1 -- start start 2024-03-06T07:23:56.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f2400b7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c1aa0 0x7f23fc071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f2400b7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc0c5c60 0x7f23fc1e9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f2400b7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc14f790 0x7f23fc151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f2400b7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc06cc00 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f2400b7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc0c0d60 con 0x7f23fc14f790 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f2400b7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc06cd80 con 0x7f23fc0c5c60 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c1aa0 0x7f23fc071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c1aa0 0x7f23fc071660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39152/0 (socket says 172.21.15.94:39152) 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 -- 172.21.15.94:0/10315401 learned_addr learned my addr 172.21.15.94:0/10315401 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fad9d640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc14f790 0x7f23fc151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23f9d9b640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc0c5c60 0x7f23fc1e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 -- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc0c5c60 msgr2=0x7f23fc1e9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc0c5c60 0x7f23fc1e9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 -- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc14f790 msgr2=0x7f23fc151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc14f790 0x7f23fc151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fa59c640 1 -- 172.21.15.94:0/10315401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc071c40 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23f9d9b640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc0c5c60 0x7f23fc1e9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.961+0000 7f23fad9d640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc14f790 0x7f23fc151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.962+0000 7f23fa59c640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c1aa0 0x7f23fc071660 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f23ec007e70 tx=0x7f23ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.962+0000 7f23eaffd640 1 -- 172.21.15.94:0/10315401 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec014620 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.962+0000 7f23eaffd640 1 -- 172.21.15.94:0/10315401 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec019070 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.962+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc1571a0 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.962+0000 7f23eaffd640 1 -- 172.21.15.94:0/10315401 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec014a00 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.962+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc157760 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.963+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23fc04f330 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.963+0000 7f23eaffd640 1 -- 172.21.15.94:0/10315401 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23ec014ba0 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.964+0000 7f23eaffd640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23dc06b4d0 0x7f23dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.964+0000 7f23eaffd640 1 -- 172.21.15.94:0/10315401 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f23ec08a9e0 con 0x7f23fc0c1aa0 2024-03-06T07:23:56.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.964+0000 7f23f9d9b640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23dc06b4d0 0x7f23dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.965+0000 7f23f9d9b640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23dc06b4d0 0x7f23dc06d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0a83d0 tx=0x7f23f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:56.968+0000 7f23eaffd640 1 -- 172.21.15.94:0/10315401 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23ec059920 con 0x7f23fc0c1aa0 2024-03-06T07:23:57.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.139+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23fc0c1f40 con 0x7f23fc0c1aa0 2024-03-06T07:23:57.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.140+0000 7f23eaffd640 1 -- 172.21.15.94:0/10315401 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+13565 (secure 0 0 0) 0x7f23ec059220 con 0x7f23fc0c1aa0 2024-03-06T07:23:57.142 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:57.142 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":140,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:54.825293+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/3826543722":"2024-03-07T07:23:51.656305+0000","172.21.15.145:6836/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.145:6834/1362727065":"2024-03-07T07:23:42.782315+0000","172.21.15.94:6835/392019985":"2024-03-07T07:23:28.504159+0000","172.21.15.145:6835/3826543722":"2024-03-07T07:23:51.656305+0000","172.21.15.94:6834/392019985":"2024-03-07T07:23:28.504159+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23dc06b4d0 msgr2=0x7f23dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23dc06b4d0 0x7f23dc06d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0a83d0 tx=0x7f23f0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c1aa0 msgr2=0x7f23fc071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c1aa0 0x7f23fc071660 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f23ec007e70 tx=0x7f23ec00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 shutdown_connections 2024-03-06T07:23:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f23dc06b4d0 0x7f23dc06d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23fc14f790 0x7f23fc151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23fc0c5c60 0x7f23fc1e9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 --2- 172.21.15.94:0/10315401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f23fc0c1aa0 0x7f23fc071660 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 >> 172.21.15.94:0/10315401 conn(0x7f23fc0c6710 msgr2=0x7f23fc0a9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 shutdown_connections 2024-03-06T07:23:57.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.142+0000 7f2400b7a640 1 -- 172.21.15.94:0/10315401 wait complete. 2024-03-06T07:23:57.155 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:57.155 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:57.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.256+0000 7f1666502640 1 Processor -- start 2024-03-06T07:23:57.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.256+0000 7f1666502640 1 -- start start 2024-03-06T07:23:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.257+0000 7f1666502640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f166006ca70 0x7f166014e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.257+0000 7f1666502640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16600b0450 con 0x7f16600b0070 2024-03-06T07:23:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.257+0000 7f1666502640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16601560e0 con 0x7f1660155c30 2024-03-06T07:23:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.257+0000 7f1666502640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166006c020 con 0x7f166006ca70 2024-03-06T07:23:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.257+0000 7f1664a9f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f166006ca70 0x7f166014e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.257+0000 7f165ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f16600b0070 0x7f16600c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47806/0 (socket says 172.21.15.94:47806) 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.257+0000 7f165ffff640 1 -- 172.21.15.94:0/1441794086 learned_addr learned my addr 172.21.15.94:0/1441794086 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.258+0000 7f1664a9f640 1 -- 172.21.15.94:0/1441794086 >> v1:172.21.15.145:6789/0 conn(0x7f1660155c30 legacy=0x7f16600c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.258+0000 7f1664a9f640 1 -- 172.21.15.94:0/1441794086 >> v1:172.21.15.94:6789/0 conn(0x7f16600b0070 legacy=0x7f16600c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.258+0000 7f165effd640 1 -- 172.21.15.94:0/1441794086 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291057476 0 0) 0x7f1650001110 con 0x7f16600b0070 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.258+0000 7f1664a9f640 1 -- 172.21.15.94:0/1441794086 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16600c77f0 con 0x7f166006ca70 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.258+0000 7f1664a9f640 1 --2- 172.21.15.94:0/1441794086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f166014e800 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f1658009d90 tx=0x7f165802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3bce8bcffdd2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f165effd640 1 -- 172.21.15.94:0/1441794086 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658039070 con 0x7f166006ca70 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f165effd640 1 -- 172.21.15.94:0/1441794086 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165802cbd0 con 0x7f166006ca70 2024-03-06T07:23:57.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f165effd640 1 -- 172.21.15.94:0/1441794086 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658034440 con 0x7f166006ca70 2024-03-06T07:23:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f1666502640 1 -- 172.21.15.94:0/1441794086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 msgr2=0x7f166014e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f1666502640 1 --2- 172.21.15.94:0/1441794086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f166014e800 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f1658009d90 tx=0x7f165802be20 comp rx=0 tx=0).stop 2024-03-06T07:23:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f1666502640 1 -- 172.21.15.94:0/1441794086 shutdown_connections 2024-03-06T07:23:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f1666502640 1 --2- 172.21.15.94:0/1441794086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f166014e800 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f1666502640 1 -- 172.21.15.94:0/1441794086 >> 172.21.15.94:0/1441794086 conn(0x7f16600ac590 msgr2=0x7f16600ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f1666502640 1 -- 172.21.15.94:0/1441794086 shutdown_connections 2024-03-06T07:23:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.259+0000 7f1666502640 1 -- 172.21.15.94:0/1441794086 wait complete. 2024-03-06T07:23:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.260+0000 7f1666502640 1 Processor -- start 2024-03-06T07:23:57.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.260+0000 7f1666502640 1 -- start start 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f1666502640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f16601eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f1666502640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16600b0070 0x7f1660158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f1666502640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1660155c30 0x7f16601ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f1666502640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166006c550 con 0x7f166006ca70 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f1666502640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166006c270 con 0x7f16600b0070 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f1666502640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166006da80 con 0x7f1660155c30 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f16601eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f16601eaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39172/0 (socket says 172.21.15.94:39172) 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 -- 172.21.15.94:0/3391445561 learned_addr learned my addr 172.21.15.94:0/3391445561 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:57.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 -- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1660155c30 msgr2=0x7f16601ee720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1660155c30 0x7f16601ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 -- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16600b0070 msgr2=0x7f1660158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16600b0070 0x7f1660158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 -- 172.21.15.94:0/3391445561 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1660158ed0 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.261+0000 7f165ffff640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f16601eaa40 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f165000c7b0 tx=0x7f165000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.262+0000 7f165cff9640 1 -- 172.21.15.94:0/3391445561 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650018070 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.262+0000 7f165cff9640 1 -- 172.21.15.94:0/3391445561 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16500072d0 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.262+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16600c4850 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.262+0000 7f165cff9640 1 -- 172.21.15.94:0/3391445561 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16500047e0 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.262+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16601ee200 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.263+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166004f330 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.263+0000 7f165cff9640 1 -- 172.21.15.94:0/3391445561 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f165001b070 con 0x7f166006ca70 2024-03-06T07:23:57.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.263+0000 7f165cff9640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164c06b4d0 0x7f164c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.264+0000 7f165cff9640 1 -- 172.21.15.94:0/3391445561 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f1650089440 con 0x7f166006ca70 2024-03-06T07:23:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.264+0000 7f165f7fe640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164c06b4d0 0x7f164c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.264+0000 7f165f7fe640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164c06b4d0 0x7f164c06d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f166014ed50 tx=0x7f16540079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.268+0000 7f165cff9640 1 -- 172.21.15.94:0/3391445561 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16600b0450 con 0x7f166006ca70 2024-03-06T07:23:57.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.439+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f166014eda0 con 0x7f166006ca70 2024-03-06T07:23:57.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.440+0000 7f165cff9640 1 -- 172.21.15.94:0/3391445561 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1650058380 con 0x7f166006ca70 2024-03-06T07:23:57.442 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164c06b4d0 msgr2=0x7f164c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164c06b4d0 0x7f164c06d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f166014ed50 tx=0x7f16540079b0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 msgr2=0x7f16601eaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f16601eaa40 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f165000c7b0 tx=0x7f165000cc70 comp rx=0 tx=0).stop 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 shutdown_connections 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164c06b4d0 0x7f164c06d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1660155c30 0x7f16601ee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16600b0070 0x7f1660158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 --2- 172.21.15.94:0/3391445561 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f166006ca70 0x7f16601eaa40 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 >> 172.21.15.94:0/3391445561 conn(0x7f16600ac590 msgr2=0x7f16600ad7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 shutdown_connections 2024-03-06T07:23:57.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.442+0000 7f1666502640 1 -- 172.21.15.94:0/3391445561 wait complete. 2024-03-06T07:23:57.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-03-06T07:23:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.553+0000 7fc1bb0b5640 1 Processor -- start 2024-03-06T07:23:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.553+0000 7fc1bb0b5640 1 -- start start 2024-03-06T07:23:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1bb0b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 0x7fc1b406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1bb0b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1b40c1320 0x7fc1b406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1bb0b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b4153a60 con 0x7fc1b40c6000 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1bb0b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b4153d40 con 0x7fc1b40c1320 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1bb0b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b406c2c0 con 0x7fc1b4150580 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1b3fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1b40c1320 0x7fc1b406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1b3fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1b40c1320 0x7fc1b406e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39188/0 (socket says 172.21.15.94:39188) 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1b3fff640 1 -- 172.21.15.94:0/291170881 learned_addr learned my addr 172.21.15.94:0/291170881 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1b9652640 1 --1- 172.21.15.94:0/291170881 >> v1:172.21.15.94:6789/0 conn(0x7fc1b40c6000 0x7fc1b41566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47816/0 (socket says 172.21.15.94:47816) 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1b3fff640 1 -- 172.21.15.94:0/291170881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 msgr2=0x7fc1b406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1b8e51640 1 --2- 172.21.15.94:0/291170881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 0x7fc1b406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.554+0000 7fc1b37fe640 1 -- 172.21.15.94:0/291170881 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029548453 0 0) 0x7fc1b4153a60 con 0x7fc1b40c6000 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.555+0000 7fc1b3fff640 1 --2- 172.21.15.94:0/291170881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 0x7fc1b406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.555+0000 7fc1b3fff640 1 -- 172.21.15.94:0/291170881 >> v1:172.21.15.94:6789/0 conn(0x7fc1b40c6000 legacy=0x7fc1b41566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.555+0000 7fc1b3fff640 1 -- 172.21.15.94:0/291170881 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b406c5a0 con 0x7fc1b40c1320 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.555+0000 7fc1b3fff640 1 --2- 172.21.15.94:0/291170881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b406e250 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009870 tx=0x7fc1a802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a9e4a2731f0c99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.555+0000 7fc1b37fe640 1 -- 172.21.15.94:0/291170881 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8035020 con 0x7fc1b40c1320 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.555+0000 7fc1b37fe640 1 -- 172.21.15.94:0/291170881 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8031640 con 0x7fc1b40c1320 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.555+0000 7fc1b37fe640 1 -- 172.21.15.94:0/291170881 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a802d040 con 0x7fc1b40c1320 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/291170881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 msgr2=0x7fc1b406e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/291170881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b406e250 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009870 tx=0x7fc1a802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/291170881 shutdown_connections 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/291170881 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b406e250 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/291170881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 0x7fc1b406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/291170881 >> 172.21.15.94:0/291170881 conn(0x7fc1b40c6710 msgr2=0x7fc1b40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/291170881 shutdown_connections 2024-03-06T07:23:57.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.556+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/291170881 wait complete. 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.557+0000 7fc1bb0b5640 1 Processor -- start 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.557+0000 7fc1bb0b5640 1 -- start start 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1bb0b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b4148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1bb0b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1b40c6000 0x7fc1b40bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1bb0b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 0x7fc1b41ec290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1bb0b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40c0dc0 con 0x7fc1b40c1320 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1bb0b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40c0c40 con 0x7fc1b40c6000 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1bb0b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4156dc0 con 0x7fc1b4150580 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b4148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b4148a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39200/0 (socket says 172.21.15.94:39200) 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 -- 172.21.15.94:0/576134437 learned_addr learned my addr 172.21.15.94:0/576134437 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 -- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 msgr2=0x7fc1b41ec290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 0x7fc1b41ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 -- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1b40c6000 msgr2=0x7fc1b40bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1b40c6000 0x7fc1b40bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 -- 172.21.15.94:0/576134437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4149f80 con 0x7fc1b40c1320 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b8e51640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b4148a50 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400c820 tx=0x7fc1a400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.558+0000 7fc1b17fa640 1 -- 172.21.15.94:0/576134437 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4015070 con 0x7fc1b40c1320 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.559+0000 7fc1b17fa640 1 -- 172.21.15.94:0/576134437 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4004020 con 0x7fc1b40c1320 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.559+0000 7fc1b17fa640 1 -- 172.21.15.94:0/576134437 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a40043f0 con 0x7fc1b40c1320 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.559+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b41499c0 con 0x7fc1b40c1320 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.559+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b40c1b50 con 0x7fc1b40c1320 2024-03-06T07:23:57.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.559+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b404f330 con 0x7fc1b40c1320 2024-03-06T07:23:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.560+0000 7fc1b17fa640 1 -- 172.21.15.94:0/576134437 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1a4007830 con 0x7fc1b40c1320 2024-03-06T07:23:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.560+0000 7fc1b17fa640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.560+0000 7fc1b17fa640 1 -- 172.21.15.94:0/576134437 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fc1a4089f60 con 0x7fc1b40c1320 2024-03-06T07:23:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.561+0000 7fc1b3fff640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.561+0000 7fc1b3fff640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8036000 tx=0x7fc1a8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.565+0000 7fc1b17fa640 1 -- 172.21.15.94:0/576134437 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1b4149f80 con 0x7fc1b40c1320 2024-03-06T07:23:57.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.800+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7fc1b40c4060 con 0x7fc1b40c1320 2024-03-06T07:23:57.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.836+0000 7fc1b17fa640 1 -- 172.21.15.94:0/576134437 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v333) v1 ==== 149+0+0 (secure 0 0 0) 0x7fc1a4058ea0 con 0x7fc1b40c1320 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 msgr2=0x7fc1a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8036000 tx=0x7fc1a8030000 comp rx=0 tx=0).stop 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 msgr2=0x7fc1b4148a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b4148a50 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400c820 tx=0x7fc1a400cce0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 shutdown_connections 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc1a006b4d0 0x7fc1a006d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1b4150580 0x7fc1b41ec290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1b40c6000 0x7fc1b40bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 --2- 172.21.15.94:0/576134437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1b40c1320 0x7fc1b4148a50 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 >> 172.21.15.94:0/576134437 conn(0x7fc1b40c6710 msgr2=0x7fc1b40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 shutdown_connections 2024-03-06T07:23:57.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.838+0000 7fc1bb0b5640 1 -- 172.21.15.94:0/576134437 wait complete. 2024-03-06T07:23:57.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:57.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd2686b640 1 Processor -- start 2024-03-06T07:23:57.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd2686b640 1 -- start start 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd2686b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbd2006afc0 0x7fbd2006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd2686b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd200c2580 0x7fbd20159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd2686b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd200c5840 con 0x7fbd200c2120 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd2686b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd20158790 con 0x7fbd2006afc0 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd2686b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd200c5b20 con 0x7fbd200c2580 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd1ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbd2006afc0 0x7fbd2006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd1ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbd2006afc0 0x7fbd2006b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39216/0 (socket says 172.21.15.94:39216) 2024-03-06T07:23:57.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd1ffff640 1 -- 172.21.15.94:0/2281804541 learned_addr learned my addr 172.21.15.94:0/2281804541 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.952+0000 7fbd1f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbd200c2120 0x7fbd20158c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47826/0 (socket says 172.21.15.94:47826) 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd24e08640 1 --2- 172.21.15.94:0/2281804541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd200c2580 0x7fbd20159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd1ffff640 1 -- 172.21.15.94:0/2281804541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd200c2580 msgr2=0x7fbd20159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd1ffff640 1 --2- 172.21.15.94:0/2281804541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd200c2580 0x7fbd20159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd1ffff640 1 -- 172.21.15.94:0/2281804541 >> v1:172.21.15.94:6789/0 conn(0x7fbd200c2120 legacy=0x7fbd20158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd1ffff640 1 -- 172.21.15.94:0/2281804541 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd200c5e00 con 0x7fbd2006afc0 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.953+0000 7fbd1effd640 1 -- 172.21.15.94:0/2281804541 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319008090 0 0) 0x7fbd200c5840 con 0x7fbd200c2120 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd1ffff640 1 --2- 172.21.15.94:0/2281804541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd2006afc0 0x7fbd2006b3a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009870 tx=0x7fbd1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af362959dcd3c2ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd1effd640 1 -- 172.21.15.94:0/2281804541 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14035020 con 0x7fbd2006afc0 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd1effd640 1 -- 172.21.15.94:0/2281804541 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd14031640 con 0x7fbd2006afc0 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd1effd640 1 -- 172.21.15.94:0/2281804541 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1402d040 con 0x7fbd2006afc0 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 -- 172.21.15.94:0/2281804541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd2006afc0 msgr2=0x7fbd2006b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 --2- 172.21.15.94:0/2281804541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd2006afc0 0x7fbd2006b3a0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009870 tx=0x7fbd1402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 -- 172.21.15.94:0/2281804541 shutdown_connections 2024-03-06T07:23:57.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 --2- 172.21.15.94:0/2281804541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd200c2580 0x7fbd20159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 --2- 172.21.15.94:0/2281804541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd2006afc0 0x7fbd2006b3a0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 -- 172.21.15.94:0/2281804541 >> 172.21.15.94:0/2281804541 conn(0x7fbd200c4220 msgr2=0x7fbd200c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 -- 172.21.15.94:0/2281804541 shutdown_connections 2024-03-06T07:23:57.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.954+0000 7fbd2686b640 1 -- 172.21.15.94:0/2281804541 wait complete. 2024-03-06T07:23:57.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.955+0000 7fbd2686b640 1 Processor -- start 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd2686b640 1 -- start start 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd2686b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd2006afc0 0x7fbd2006c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd2686b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd200c2120 0x7fbd20149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd2686b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd200c2580 0x7fbd20149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd2686b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd200c2b30 con 0x7fbd200c2580 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd2686b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd20155220 con 0x7fbd200c2120 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd2686b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd200c2cb0 con 0x7fbd2006afc0 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd24e08640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd200c2580 0x7fbd20149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd24e08640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd200c2580 0x7fbd20149cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39230/0 (socket says 172.21.15.94:39230) 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd24e08640 1 -- 172.21.15.94:0/1919296676 learned_addr learned my addr 172.21.15.94:0/1919296676 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd1ffff640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd2006afc0 0x7fbd2006c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.956+0000 7fbd1f7fe640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd200c2120 0x7fbd20149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd24e08640 1 -- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd2006afc0 msgr2=0x7fbd2006c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd24e08640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd2006afc0 0x7fbd2006c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd24e08640 1 -- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd200c2120 msgr2=0x7fbd20149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd24e08640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd200c2120 0x7fbd20149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd24e08640 1 -- 172.21.15.94:0/1919296676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd200734c0 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd24e08640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd200c2580 0x7fbd20149cc0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbd18007d80 tx=0x7fbd1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd1cff9640 1 -- 172.21.15.94:0/1919296676 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18014420 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd1cff9640 1 -- 172.21.15.94:0/1919296676 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd18019070 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd1cff9640 1 -- 172.21.15.94:0/1919296676 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18014800 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2014d5f0 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.957+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd201569c0 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.958+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2004f330 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.958+0000 7fbd1cff9640 1 -- 172.21.15.94:0/1919296676 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd180149a0 con 0x7fbd200c2580 2024-03-06T07:23:57.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.959+0000 7fbd1cff9640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0c06b500 0x7fbd0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.959+0000 7fbd1cff9640 1 -- 172.21.15.94:0/1919296676 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fbd1808a7c0 con 0x7fbd200c2580 2024-03-06T07:23:57.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.959+0000 7fbd1ffff640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0c06b500 0x7fbd0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.959+0000 7fbd1ffff640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0c06b500 0x7fbd0c06d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbd14036000 tx=0x7fbd1403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:57.963+0000 7fbd1cff9640 1 -- 172.21.15.94:0/1919296676 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd200734c0 con 0x7fbd200c2580 2024-03-06T07:23:58.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.136+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbd2006be40 con 0x7fbd200c2580 2024-03-06T07:23:58.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.140+0000 7fbd1cff9640 1 -- 172.21.15.94:0/1919296676 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbd180590b0 con 0x7fbd200c2580 2024-03-06T07:23:58.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0c06b500 msgr2=0x7fbd0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0c06b500 0x7fbd0c06d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbd14036000 tx=0x7fbd1403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:58.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd200c2580 msgr2=0x7fbd20149cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd200c2580 0x7fbd20149cc0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbd18007d80 tx=0x7fbd1800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 shutdown_connections 2024-03-06T07:23:58.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0c06b500 0x7fbd0c06d9c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd200c2580 0x7fbd20149cc0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd200c2120 0x7fbd20149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 --2- 172.21.15.94:0/1919296676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd2006afc0 0x7fbd2006c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 >> 172.21.15.94:0/1919296676 conn(0x7fbd200c4220 msgr2=0x7fbd2006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 shutdown_connections 2024-03-06T07:23:58.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.142+0000 7fbd2686b640 1 -- 172.21.15.94:0/1919296676 wait complete. 2024-03-06T07:23:58.155 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:58.155 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:58.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:58.245 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:58.245 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:23:58.245 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:58.261 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:23:58.261 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:23:58.261 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:58.304 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:23:58.304 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:23:58.304 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:58.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.401+0000 7f96478f4640 1 Processor -- start 2024-03-06T07:23:58.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.401+0000 7f96478f4640 1 -- start start 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f96478f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9640153990 0x7f9640156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f96478f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96400acc00 0x7f96400c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f96478f4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9640070aa0 con 0x7f96401506b0 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f96478f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9640156450 con 0x7f9640153990 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f96478f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9640156a10 con 0x7f96400acc00 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f9644e8f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9640153990 0x7f9640156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f9645690640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f96401506b0 0x7f96400702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47828/0 (socket says 172.21.15.94:47828) 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.402+0000 7f9645690640 1 -- 172.21.15.94:0/3158974303 learned_addr learned my addr 172.21.15.94:0/3158974303 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9644e8f640 1 -- 172.21.15.94:0/3158974303 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96400acc00 msgr2=0x7f96400c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9644e8f640 1 --2- 172.21.15.94:0/3158974303 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96400acc00 0x7f96400c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9644e8f640 1 -- 172.21.15.94:0/3158974303 >> v1:172.21.15.94:6789/0 conn(0x7f96401506b0 legacy=0x7f96400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9644e8f640 1 -- 172.21.15.94:0/3158974303 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640070d80 con 0x7f9640153990 2024-03-06T07:23:58.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9637fff640 1 -- 172.21.15.94:0/3158974303 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737315966 0 0) 0x7f9640070aa0 con 0x7f96401506b0 2024-03-06T07:23:58.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9644e8f640 1 --2- 172.21.15.94:0/3158974303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9640153990 0x7f9640156ea0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f96300097a0 tx=0x7f963002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f45e3de6df0825 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9637fff640 1 -- 172.21.15.94:0/3158974303 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630035020 con 0x7f9640153990 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9637fff640 1 -- 172.21.15.94:0/3158974303 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9630031830 con 0x7f9640153990 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.403+0000 7f9637fff640 1 -- 172.21.15.94:0/3158974303 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630031c00 con 0x7f9640153990 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 -- 172.21.15.94:0/3158974303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9640153990 msgr2=0x7f9640156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 --2- 172.21.15.94:0/3158974303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9640153990 0x7f9640156ea0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f96300097a0 tx=0x7f963002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 -- 172.21.15.94:0/3158974303 shutdown_connections 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 --2- 172.21.15.94:0/3158974303 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96400acc00 0x7f96400c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 --2- 172.21.15.94:0/3158974303 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9640153990 0x7f9640156ea0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 -- 172.21.15.94:0/3158974303 >> 172.21.15.94:0/3158974303 conn(0x7f964006b1a0 msgr2=0x7f964006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 -- 172.21.15.94:0/3158974303 shutdown_connections 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.404+0000 7f96478f4640 1 -- 172.21.15.94:0/3158974303 wait complete. 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.405+0000 7f96478f4640 1 Processor -- start 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.405+0000 7f96478f4640 1 -- start start 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f96478f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96400acc00 0x7f96400c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f96478f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96401506b0 0x7f964006db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f96478f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9640153990 0x7f96401ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f96478f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96400ad1d0 con 0x7f96401506b0 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f96478f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96400ad050 con 0x7f96400acc00 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f96478f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96400ad350 con 0x7f9640153990 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96401506b0 0x7f964006db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96401506b0 0x7f964006db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39250/0 (socket says 172.21.15.94:39250) 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 -- 172.21.15.94:0/3249002908 learned_addr learned my addr 172.21.15.94:0/3249002908 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:58.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9645e91640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9640153990 0x7f96401ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9645690640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96400acc00 0x7f96400c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 -- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9640153990 msgr2=0x7f96401ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9640153990 0x7f96401ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 -- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96400acc00 msgr2=0x7f96400c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96400acc00 0x7f96400c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 -- 172.21.15.94:0/3249002908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96400c4be0 con 0x7f96401506b0 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9645e91640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9640153990 0x7f96401ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9645690640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96400acc00 0x7f96400c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.406+0000 7f9644e8f640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96401506b0 0x7f964006db00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f96300098c0 tx=0x7f96300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.407+0000 7f9635ffb640 1 -- 172.21.15.94:0/3249002908 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963003ba80 con 0x7f96401506b0 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.407+0000 7f9635ffb640 1 -- 172.21.15.94:0/3249002908 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963003bc40 con 0x7f96401506b0 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.407+0000 7f9635ffb640 1 -- 172.21.15.94:0/3249002908 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96300453f0 con 0x7f96401506b0 2024-03-06T07:23:58.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.407+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96401edec0 con 0x7f96401506b0 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.407+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964006e040 con 0x7f96401506b0 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.408+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964004f330 con 0x7f96401506b0 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.408+0000 7f9635ffb640 1 -- 172.21.15.94:0/3249002908 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f963004c060 con 0x7f96401506b0 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.409+0000 7f9635ffb640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f962406b500 0x7f962406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.409+0000 7f9635ffb640 1 -- 172.21.15.94:0/3249002908 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f96300abe30 con 0x7f96401506b0 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.409+0000 7f9645690640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f962406b500 0x7f962406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.409+0000 7f9645690640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f962406b500 0x7f962406d9c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f963c0012e0 tx=0x7f963c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.413+0000 7f9635ffb640 1 -- 172.21.15.94:0/3249002908 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9630040070 con 0x7f96401506b0 2024-03-06T07:23:58.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.620+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96400700a0 con 0x7f96401506b0 2024-03-06T07:23:58.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.621+0000 7f9635ffb640 1 -- 172.21.15.94:0/3249002908 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+3705 (secure 0 0 0) 0x7f963002f2e0 con 0x7f96401506b0 2024-03-06T07:23:58.623 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:23:58.623 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":131,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8062,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/823810752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":823810752},{"type":"v1","addr":"172.21.15.145:6837","nonce":823810752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131},{"gid":8298,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/3405703874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3405703874},{"type":"v1","addr":"172.21.15.94:6833","nonce":3405703874}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/801675853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":801675853},{"type":"v1","addr":"172.21.15.94:6835","nonce":801675853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122},{"gid":36917,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1486437600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1486437600},{"type":"v1","addr":"172.21.15.145:6836","nonce":1486437600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":127}],"filesystems":[]} 2024-03-06T07:23:58.623 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 131 2024-03-06T07:23:58.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f962406b500 msgr2=0x7f962406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f962406b500 0x7f962406d9c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f963c0012e0 tx=0x7f963c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96401506b0 msgr2=0x7f964006db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96401506b0 0x7f964006db00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f96300098c0 tx=0x7f96300046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 shutdown_connections 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f962406b500 0x7f962406d9c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9640153990 0x7f96401ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96401506b0 0x7f964006db00 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 --2- 172.21.15.94:0/3249002908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96400acc00 0x7f96400c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.622+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 >> 172.21.15.94:0/3249002908 conn(0x7f964006b1a0 msgr2=0x7f96400c5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.623+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 shutdown_connections 2024-03-06T07:23:58.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.623+0000 7f96478f4640 1 -- 172.21.15.94:0/3249002908 wait complete. 2024-03-06T07:23:58.635 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:58.636 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:58.636 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:58.636 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:58.636 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:58.636 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:58.636 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:58.637 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:58.637 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:58.637 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:58.637 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:58.637 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:58.637 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:58.637 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:58.638 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:58.638 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:58.645 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:58.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:58.646 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:58.646 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:58.649 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:58.649 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:58.650 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:58.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:58.689 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:58.689 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:58.689 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:58.690 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:58.692 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:58.693 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:58.694 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:58.694 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:58.694 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:58.765 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:23:58.765 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:23:58.765 INFO:tasks.ceph.mds.c.smithi094.stderr:2024-03-06T07:23:58.764+0000 7f1f60d54640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:23:58.774 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:23:58.774 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:23:58.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.800+0000 7f0d11779640 1 Processor -- start 2024-03-06T07:23:58.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.800+0000 7f0d11779640 1 -- start start 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d11779640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14eba0 0x7f0d0c0a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d11779640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d0c0733f0 0x7f0d0c0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d11779640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0d0c0c2230 0x7f0d0c0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d11779640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c14ef80 con 0x7f0d0c0733f0 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d11779640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c4a90 con 0x7f0d0c0c2230 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d11779640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c151f10 con 0x7f0d0c14eba0 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d0c0733f0 0x7f0d0c0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d0c0733f0 0x7f0d0c0c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36752/0 (socket says 172.21.15.94:36752) 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/2280857247 learned_addr learned my addr 172.21.15.94:0/2280857247 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0affd640 1 --2- 172.21.15.94:0/2280857247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14eba0 0x7f0d0c0a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/2280857247 >> v2:172.21.15.145:3300/0 conn(0x7f0d0c0c2230 0x7f0d0c0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/2280857247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14eba0 msgr2=0x7f0d0c0a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 --2- 172.21.15.94:0/2280857247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14eba0 0x7f0d0c0a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/2280857247 >> v2:172.21.15.145:3300/0 conn(0x7f0d0c0c2230 msgr2=0x7f0d0c0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 --2- 172.21.15.94:0/2280857247 >> v2:172.21.15.145:3300/0 conn(0x7f0d0c0c2230 0x7f0d0c0c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.801+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/2280857247 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0c4d30 con 0x7f0d0c0733f0 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.802+0000 7f0d0a7fc640 1 --2- 172.21.15.94:0/2280857247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c0733f0 0x7f0d0c0c5250 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0c1c50 tx=0x7f0cfc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07f150dcc85a667 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.802+0000 7f0d09ffb640 1 -- 172.21.15.94:0/2280857247 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc02e070 con 0x7f0d0c0733f0 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.802+0000 7f0d09ffb640 1 -- 172.21.15.94:0/2280857247 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc037070 con 0x7f0d0c0733f0 2024-03-06T07:23:58.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.802+0000 7f0d09ffb640 1 -- 172.21.15.94:0/2280857247 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc031a00 con 0x7f0d0c0733f0 2024-03-06T07:23:58.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 -- 172.21.15.94:0/2280857247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c0733f0 msgr2=0x7f0d0c0c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 --2- 172.21.15.94:0/2280857247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c0733f0 0x7f0d0c0c5250 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0c1c50 tx=0x7f0cfc02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:58.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 -- 172.21.15.94:0/2280857247 shutdown_connections 2024-03-06T07:23:58.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 --2- 172.21.15.94:0/2280857247 >> v2:172.21.15.145:3300/0 conn(0x7f0d0c0c2230 0x7f0d0c0c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 --2- 172.21.15.94:0/2280857247 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c0733f0 0x7f0d0c0c5250 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 --2- 172.21.15.94:0/2280857247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14eba0 0x7f0d0c0a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 -- 172.21.15.94:0/2280857247 >> 172.21.15.94:0/2280857247 conn(0x7f0d0c0c6af0 msgr2=0x7f0d0c0c6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 -- 172.21.15.94:0/2280857247 shutdown_connections 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.803+0000 7f0d11779640 1 -- 172.21.15.94:0/2280857247 wait complete. 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.804+0000 7f0d11779640 1 Processor -- start 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.804+0000 7f0d11779640 1 -- start start 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0733f0 0x7f0d0c0abf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c0c2230 0x7f0d0c13ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14eba0 0x7f0d0c13f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c073020 con 0x7f0d0c14eba0 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c072ea0 con 0x7f0d0c0c2230 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c5cd0 con 0x7f0d0c0733f0 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14eba0 0x7f0d0c13f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14eba0 0x7f0d0c13f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36756/0 (socket says 172.21.15.94:36756) 2024-03-06T07:23:58.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 -- 172.21.15.94:0/3053371659 learned_addr learned my addr 172.21.15.94:0/3053371659 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0affd640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0733f0 0x7f0d0c0abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0a7fc640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c0c2230 0x7f0d0c13ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 -- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0733f0 msgr2=0x7f0d0c0abf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0733f0 0x7f0d0c0abf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 -- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c0c2230 msgr2=0x7f0d0c13ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c0c2230 0x7f0d0c13ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 -- 172.21.15.94:0/3053371659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c1e94e0 con 0x7f0d0c14eba0 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0a7fc640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c0c2230 0x7f0d0c13ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14eba0 0x7f0d0c13f2c0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400eed0 tx=0x7f0d0400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0cf37fe640 1 -- 172.21.15.94:0/3053371659 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04018020 con 0x7f0d0c14eba0 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0cf37fe640 1 -- 172.21.15.94:0/3053371659 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0400bd30 con 0x7f0d0c14eba0 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 -- 172.21.15.94:0/3053371659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c1e97c0 con 0x7f0d0c14eba0 2024-03-06T07:23:58.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0cf37fe640 1 -- 172.21.15.94:0/3053371659 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04014640 con 0x7f0d0c14eba0 2024-03-06T07:23:58.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.805+0000 7f0d11779640 1 -- 172.21.15.94:0/3053371659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c0ac4a0 con 0x7f0d0c14eba0 2024-03-06T07:23:58.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.806+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce4005290 con 0x7f0d0c14eba0 2024-03-06T07:23:58.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.807+0000 7f0cf37fe640 1 -- 172.21.15.94:0/3053371659 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d04005d10 con 0x7f0d0c14eba0 2024-03-06T07:23:58.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.807+0000 7f0cf37fe640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.807+0000 7f0cf37fe640 1 -- 172.21.15.94:0/3053371659 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f0d0408a9a0 con 0x7f0d0c14eba0 2024-03-06T07:23:58.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.808+0000 7f0d0affd640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.808+0000 7f0d0affd640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b4d0 0x7f0cf806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c1db050 tx=0x7f0d00009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.812+0000 7f0cf37fe640 1 -- 172.21.15.94:0/3053371659 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d04022030 con 0x7f0d0c14eba0 2024-03-06T07:23:59.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:58.998+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0ce4005850 con 0x7f0d0c14eba0 2024-03-06T07:23:59.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.854+0000 7f0cf37fe640 1 -- 172.21.15.94:0/3053371659 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v145) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0d040598e0 con 0x7f0d0c14eba0 2024-03-06T07:23:59.856 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b4d0 msgr2=0x7f0cf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b4d0 0x7f0cf806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c1db050 tx=0x7f0d00009190 comp rx=0 tx=0).stop 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14eba0 msgr2=0x7f0d0c13f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14eba0 0x7f0d0c13f2c0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400eed0 tx=0x7f0d0400b480 comp rx=0 tx=0).stop 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 shutdown_connections 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14eba0 0x7f0d0c13f2c0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c0c2230 0x7f0d0c13ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 --2- 172.21.15.94:0/3053371659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0733f0 0x7f0d0c0abf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 >> 172.21.15.94:0/3053371659 conn(0x7f0d0c0c6af0 msgr2=0x7f0d0c0c3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 shutdown_connections 2024-03-06T07:23:59.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.857+0000 7f0cf17fa640 1 -- 172.21.15.94:0/3053371659 wait complete. 2024-03-06T07:23:59.872 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:59.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.975+0000 7f8f9edd7640 1 Processor -- start 2024-03-06T07:23:59.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.975+0000 7f8f9edd7640 1 -- start start 2024-03-06T07:23:59.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f9edd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 0x7f8f981529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f9edd7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f980c28d0 con 0x7f8f980c1aa0 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f9edd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f980c2bb0 con 0x7f8f98155a40 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f9edd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980c2680 con 0x7f8f9814f790 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f97fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8f980c1aa0 0x7f8f980c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37876/0 (socket says 172.21.15.94:37876) 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f97fff640 1 -- 172.21.15.94:0/4252316 learned_addr learned my addr 172.21.15.94:0/4252316 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f9cb73640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 0x7f8f981529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498691752 0 0) 0x7f8f980c28d0 con 0x7f8f980c1aa0 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.976+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f80003520 con 0x7f8f980c1aa0 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2477626138 0 0) 0x7f8f80003520 con 0x7f8f980c1aa0 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 msgr2=0x7f8f981529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 0x7f8f981529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 >> v1:172.21.15.145:6789/0 conn(0x7f8f98155a40 legacy=0x7f8f98155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f980c2e90 con 0x7f8f980c1aa0 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f8f8c001670 con 0x7f8f980c1aa0 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 >> v1:172.21.15.94:6789/0 conn(0x7f8f980c1aa0 legacy=0x7f8f980c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 0x7f8f800039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f80004d40 0x7f8f800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980c2680 con 0x7f8f80003f60 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8000baa0 con 0x7f8f80004d40 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8000deb0 con 0x7f8f9814f790 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f97fff640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f9cb73640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 0x7f8f800039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f9d374640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f80004d40 0x7f8f800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f97fff640 1 -- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 msgr2=0x7f8f800039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f97fff640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f9814f790 0x7f8f800039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f97fff640 1 -- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f80004d40 msgr2=0x7f8f800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f97fff640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f80004d40 0x7f8f800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f97fff640 1 -- 172.21.15.94:0/4252316 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80003520 con 0x7f8f80003f60 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.977+0000 7f8f9d374640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f80004d40 0x7f8f800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.978+0000 7f8f97fff640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f80004800 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00c240 tx=0x7f8f80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea73d8350d5356 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.978+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c007d10 con 0x7f8f80003f60 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.978+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c030070 con 0x7f8f80003f60 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.978+0000 7f8f977fe640 1 -- 172.21.15.94:0/4252316 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c02cdc0 con 0x7f8f80003f60 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.978+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4252316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 msgr2=0x7f8f80004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.978+0000 7f8f9edd7640 1 --2- 172.21.15.94:0/4252316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f80004800 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00c240 tx=0x7f8f80005830 comp rx=0 tx=0).stop 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.979+0000 7f8f9d374640 1 -- 172.21.15.94:0/4252316 reap_dead start 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.979+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4252316 shutdown_connections 2024-03-06T07:23:59.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.979+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4252316 >> 172.21.15.94:0/4252316 conn(0x7f8f980c6710 msgr2=0x7f8f980c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.979+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4252316 shutdown_connections 2024-03-06T07:23:59.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.979+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4252316 wait complete. 2024-03-06T07:23:59.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.980+0000 7f8f9edd7640 1 Processor -- start 2024-03-06T07:23:59.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.980+0000 7f8f9edd7640 1 -- start start 2024-03-06T07:23:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.980+0000 7f8f9edd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f9814d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.980+0000 7f8f9edd7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f98148f50 0x7f8f980bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9edd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f981ebad0 0x7f8f98148300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9edd7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980c5e30 con 0x7f8f80003f60 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9edd7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980c0d60 con 0x7f8f98148f50 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9edd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980c5fb0 con 0x7f8f981ebad0 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f9814d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f9814d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36770/0 (socket says 172.21.15.94:36770) 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 -- 172.21.15.94:0/4260795207 learned_addr learned my addr 172.21.15.94:0/4260795207 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 -- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f981ebad0 msgr2=0x7f8f98148300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f981ebad0 0x7f8f98148300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 -- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f98148f50 msgr2=0x7f8f980bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f98148f50 0x7f8f980bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 -- 172.21.15.94:0/4260795207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f981dc850 con 0x7f8f80003f60 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9cb73640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f9814d190 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8f880119a0 tx=0x7f8f88011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f957fa640 1 -- 172.21.15.94:0/4260795207 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8800faf0 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f957fa640 1 -- 172.21.15.94:0/4260795207 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8800fc90 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9814a770 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f957fa640 1 -- 172.21.15.94:0/4260795207 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88014440 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.981+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f981eb2d0 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.982+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9804f330 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.983+0000 7f8f957fa640 1 -- 172.21.15.94:0/4260795207 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f88012040 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.983+0000 7f8f957fa640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f8406b500 0x7f8f8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.983+0000 7f8f957fa640 1 -- 172.21.15.94:0/4260795207 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8f8808a6f0 con 0x7f8f80003f60 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.983+0000 7f8f97fff640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f8406b500 0x7f8f8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.984+0000 7f8f97fff640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f8406b500 0x7f8f8406d9c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c034000 tx=0x7f8f8c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:23:59.987+0000 7f8f957fa640 1 -- 172.21.15.94:0/4260795207 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f88019030 con 0x7f8f80003f60 2024-03-06T07:24:00.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.227+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8f9806cbe0 con 0x7f8f80003f60 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.228+0000 7f8f957fa640 1 -- 172.21.15.94:0/4260795207 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f8f88059730 con 0x7f8f80003f60 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:00.230 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.231 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.232 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.233 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.234 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.235 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.236 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.237 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:24:00.238 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:24:00.239 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.240 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:00.241 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:00.242 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:24:00.242 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f8406b500 msgr2=0x7f8f8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f8406b500 0x7f8f8406d9c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c034000 tx=0x7f8f8c007000 comp rx=0 tx=0).stop 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 msgr2=0x7f8f9814d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f9814d190 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8f880119a0 tx=0x7f8f88011e60 comp rx=0 tx=0).stop 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 shutdown_connections 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8f8406b500 0x7f8f8406d9c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f981ebad0 0x7f8f98148300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f98148f50 0x7f8f980bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 --2- 172.21.15.94:0/4260795207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f80003f60 0x7f8f9814d190 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 >> 172.21.15.94:0/4260795207 conn(0x7f8f980c6710 msgr2=0x7f8f980a9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 shutdown_connections 2024-03-06T07:24:00.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.230+0000 7f8f9edd7640 1 -- 172.21.15.94:0/4260795207 wait complete. 2024-03-06T07:24:00.245 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.344+0000 7f7816237640 1 Processor -- start 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.344+0000 7f7816237640 1 -- start start 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.344+0000 7f7816237640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100c3750 0x7f78100c1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f7816237640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78100a5e60 0x7f78100c1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f7816237640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100c5490 con 0x7f78100c3370 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f7816237640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78100c5770 con 0x7f78100a5e60 2024-03-06T07:24:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f7816237640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78100c5a50 con 0x7f78100c3750 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780effd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100c3750 0x7f78100c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780ffff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78100a5e60 0x7f78100c1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780f7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f78100c3370 0x7f7810155ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52928/0 (socket says 172.21.15.94:52928) 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780effd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100c3750 0x7f78100c1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:41644/0 (socket says 172.21.15.94:41644) 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780f7fe640 1 -- 172.21.15.94:0/527558372 learned_addr learned my addr 172.21.15.94:0/527558372 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780ffff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78100a5e60 0x7f78100c1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:57386/0 (socket says 172.21.15.94:57386) 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780effd640 1 -- 172.21.15.94:0/527558372 >> v2:172.21.15.145:3300/0 conn(0x7f78100a5e60 msgr2=0x7f78100c1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780effd640 1 --2- 172.21.15.94:0/527558372 >> v2:172.21.15.145:3300/0 conn(0x7f78100a5e60 0x7f78100c1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780effd640 1 -- 172.21.15.94:0/527558372 >> v1:172.21.15.145:6789/0 conn(0x7f78100c3370 legacy=0x7f7810155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780effd640 1 -- 172.21.15.94:0/527558372 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78100c5d30 con 0x7f78100c3750 2024-03-06T07:24:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.345+0000 7f780ffff640 1 --2- 172.21.15.94:0/527558372 >> v2:172.21.15.145:3300/0 conn(0x7f78100a5e60 0x7f78100c1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:00.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.346+0000 7f780effd640 1 --2- 172.21.15.94:0/527558372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100c3750 0x7f78100c1110 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f78040097a0 tx=0x7f780402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3489d457b78122a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.346+0000 7f780e7fc640 1 -- 172.21.15.94:0/527558372 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804038070 con 0x7f78100c3750 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.346+0000 7f780e7fc640 1 -- 172.21.15.94:0/527558372 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7804031700 con 0x7f78100c3750 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.346+0000 7f780e7fc640 1 -- 172.21.15.94:0/527558372 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804031ad0 con 0x7f78100c3750 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 -- 172.21.15.94:0/527558372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100c3750 msgr2=0x7f78100c1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 --2- 172.21.15.94:0/527558372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100c3750 0x7f78100c1110 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f78040097a0 tx=0x7f780402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 -- 172.21.15.94:0/527558372 shutdown_connections 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 --2- 172.21.15.94:0/527558372 >> v2:172.21.15.145:3300/0 conn(0x7f78100a5e60 0x7f78100c1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 --2- 172.21.15.94:0/527558372 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100c3750 0x7f78100c1110 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 -- 172.21.15.94:0/527558372 >> 172.21.15.94:0/527558372 conn(0x7f78100c6710 msgr2=0x7f78100c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 -- 172.21.15.94:0/527558372 shutdown_connections 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.347+0000 7f7816237640 1 -- 172.21.15.94:0/527558372 wait complete. 2024-03-06T07:24:00.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.348+0000 7f7816237640 1 Processor -- start 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.348+0000 7f7816237640 1 -- start start 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f7816237640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100a5e60 0x7f7810072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f7816237640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78100c3370 0x7f781014d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f7816237640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78100c3750 0x7f781014dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f7816237640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7810159910 con 0x7f78100c3370 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f7816237640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7810159790 con 0x7f78100c3750 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f7816237640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78100c1b90 con 0x7f78100a5e60 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78100c3370 0x7f781014d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78100c3370 0x7f781014d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36778/0 (socket says 172.21.15.94:36778) 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 -- 172.21.15.94:0/2401134256 learned_addr learned my addr 172.21.15.94:0/2401134256 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780f7fe640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100a5e60 0x7f7810072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780ffff640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78100c3750 0x7f781014dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 -- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100a5e60 msgr2=0x7f7810072d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100a5e60 0x7f7810072d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 -- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78100c3750 msgr2=0x7f781014dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78100c3750 0x7f781014dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 -- 172.21.15.94:0/2401134256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781014e6d0 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f780effd640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78100c3370 0x7f781014d7c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7804036040 tx=0x7f78040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.349+0000 7f77f7fff640 1 -- 172.21.15.94:0/2401134256 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804042070 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.350+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78101ead80 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.350+0000 7f77f7fff640 1 -- 172.21.15.94:0/2401134256 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780403d890 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.350+0000 7f77f7fff640 1 -- 172.21.15.94:0/2401134256 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804038040 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.350+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78101583f0 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.350+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781004f330 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.351+0000 7f77f7fff640 1 -- 172.21.15.94:0/2401134256 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f780403b070 con 0x7f78100c3370 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.351+0000 7f77f7fff640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f77fc06b4d0 0x7f77fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.351+0000 7f77f7fff640 1 -- 172.21.15.94:0/2401134256 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f78040ab5a0 con 0x7f78100c3370 2024-03-06T07:24:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.351+0000 7f780f7fe640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f77fc06b4d0 0x7f77fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.352+0000 7f780f7fe640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f77fc06b4d0 0x7f77fc06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7800007fd0 tx=0x7f7800007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.355+0000 7f77f7fff640 1 -- 172.21.15.94:0/2401134256 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7804030070 con 0x7f78100c3370 2024-03-06T07:24:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.527+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78100a65b0 con 0x7f78100c3370 2024-03-06T07:24:00.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.528+0000 7f77f7fff640 1 -- 172.21.15.94:0/2401134256 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+13174 (secure 0 0 0) 0x7f780407a0c0 con 0x7f78100c3370 2024-03-06T07:24:00.530 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:00.530 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":145,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:23:59.853236+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f77fc06b4d0 msgr2=0x7f77fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f77fc06b4d0 0x7f77fc06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7800007fd0 tx=0x7f7800007e80 comp rx=0 tx=0).stop 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78100c3370 msgr2=0x7f781014d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78100c3370 0x7f781014d7c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7804036040 tx=0x7f78040046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 shutdown_connections 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f77fc06b4d0 0x7f77fc06d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78100c3750 0x7f781014dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78100c3370 0x7f781014d7c0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 --2- 172.21.15.94:0/2401134256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78100a5e60 0x7f7810072d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 >> 172.21.15.94:0/2401134256 conn(0x7f78100c6710 msgr2=0x7f7810151f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 shutdown_connections 2024-03-06T07:24:00.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.530+0000 7f7816237640 1 -- 172.21.15.94:0/2401134256 wait complete. 2024-03-06T07:24:00.544 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:00.544 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:00.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.645+0000 7f6893c8a640 1 Processor -- start 2024-03-06T07:24:00.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.645+0000 7f6893c8a640 1 -- start start 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6893c8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f688c06b3a0 0x7f688c158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6893c8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c159160 0x7f688c15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6893c8a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c06b800 con 0x7f688c06afc0 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6893c8a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c158790 con 0x7f688c06b3a0 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6893c8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c0c5840 con 0x7f688c159160 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6891225640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f688c06b3a0 0x7f688c158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6891225640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f688c06b3a0 0x7f688c158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36786/0 (socket says 172.21.15.94:36786) 2024-03-06T07:24:00.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6891225640 1 -- 172.21.15.94:0/454781756 learned_addr learned my addr 172.21.15.94:0/454781756 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6892227640 1 --2- 172.21.15.94:0/454781756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c159160 0x7f688c15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6891225640 1 -- 172.21.15.94:0/454781756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c159160 msgr2=0x7f688c15fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.646+0000 7f6891225640 1 --2- 172.21.15.94:0/454781756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c159160 0x7f688c15fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.647+0000 7f6891225640 1 -- 172.21.15.94:0/454781756 >> v1:172.21.15.145:6789/0 conn(0x7f688c06afc0 legacy=0x7f688c0c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.647+0000 7f6891225640 1 -- 172.21.15.94:0/454781756 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c0c5b20 con 0x7f688c06b3a0 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.647+0000 7f6892227640 1 --2- 172.21.15.94:0/454781756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c159160 0x7f688c15fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.647+0000 7f6891225640 1 --2- 172.21.15.94:0/454781756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c06b3a0 0x7f688c158c20 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f687c009870 tx=0x7f687c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1d272895780ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.647+0000 7f6890a24640 1 -- 172.21.15.94:0/454781756 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c0317c0 con 0x7f688c06b3a0 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.647+0000 7f6890a24640 1 -- 172.21.15.94:0/454781756 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c037070 con 0x7f688c06b3a0 2024-03-06T07:24:00.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.647+0000 7f6890a24640 1 -- 172.21.15.94:0/454781756 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c031b80 con 0x7f688c06b3a0 2024-03-06T07:24:00.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 -- 172.21.15.94:0/454781756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c06b3a0 msgr2=0x7f688c158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 --2- 172.21.15.94:0/454781756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c06b3a0 0x7f688c158c20 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f687c009870 tx=0x7f687c02b860 comp rx=0 tx=0).stop 2024-03-06T07:24:00.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 -- 172.21.15.94:0/454781756 shutdown_connections 2024-03-06T07:24:00.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 --2- 172.21.15.94:0/454781756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c159160 0x7f688c15fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 --2- 172.21.15.94:0/454781756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c06b3a0 0x7f688c158c20 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 -- 172.21.15.94:0/454781756 >> 172.21.15.94:0/454781756 conn(0x7f688c0c4220 msgr2=0x7f688c0c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 -- 172.21.15.94:0/454781756 shutdown_connections 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.648+0000 7f6893c8a640 1 -- 172.21.15.94:0/454781756 wait complete. 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.649+0000 7f6893c8a640 1 Processor -- start 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.649+0000 7f6893c8a640 1 -- start start 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6893c8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f688c06afc0 0x7f688c1519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6893c8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c06b3a0 0x7f688c1eebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6893c8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c159160 0x7f688c1ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6893c8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c1595f0 con 0x7f688c159160 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6893c8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c0c60e0 con 0x7f688c06afc0 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6893c8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c155220 con 0x7f688c06b3a0 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c159160 0x7f688c1ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c159160 0x7f688c1ef120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36794/0 (socket says 172.21.15.94:36794) 2024-03-06T07:24:00.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 -- 172.21.15.94:0/1010079861 learned_addr learned my addr 172.21.15.94:0/1010079861 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6891225640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c06b3a0 0x7f688c1eebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6891a26640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f688c06afc0 0x7f688c1519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 -- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c06b3a0 msgr2=0x7f688c1eebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c06b3a0 0x7f688c1eebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 -- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f688c06afc0 msgr2=0x7f688c1519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f688c06afc0 0x7f688c1519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 -- 172.21.15.94:0/1010079861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c1e0420 con 0x7f688c159160 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6891a26640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f688c06afc0 0x7f688c1519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.650+0000 7f6892227640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c159160 0x7f688c1ef120 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f688400c7f0 tx=0x7f688400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.651+0000 7f68827fc640 1 -- 172.21.15.94:0/1010079861 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884015020 con 0x7f688c159160 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.651+0000 7f68827fc640 1 -- 172.21.15.94:0/1010079861 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6884004230 con 0x7f688c159160 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.651+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688c1e0700 con 0x7f688c159160 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.651+0000 7f68827fc640 1 -- 172.21.15.94:0/1010079861 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6884007620 con 0x7f688c159160 2024-03-06T07:24:00.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.651+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688c1ee6c0 con 0x7f688c159160 2024-03-06T07:24:00.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.651+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688c04f330 con 0x7f688c159160 2024-03-06T07:24:00.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.652+0000 7f68827fc640 1 -- 172.21.15.94:0/1010079861 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6884018020 con 0x7f688c159160 2024-03-06T07:24:00.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.652+0000 7f68827fc640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f687006b4d0 0x7f687006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.652+0000 7f68827fc640 1 -- 172.21.15.94:0/1010079861 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6884089fa0 con 0x7f688c159160 2024-03-06T07:24:00.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.653+0000 7f6891a26640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f687006b4d0 0x7f687006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.653+0000 7f6891a26640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f687006b4d0 0x7f687006d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f68880045b0 tx=0x7f6888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.657+0000 7f68827fc640 1 -- 172.21.15.94:0/1010079861 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f688401f020 con 0x7f688c159160 2024-03-06T07:24:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.828+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f688c0adab0 con 0x7f688c159160 2024-03-06T07:24:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.828+0000 7f68827fc640 1 -- 172.21.15.94:0/1010079861 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6884059130 con 0x7f688c159160 2024-03-06T07:24:00.830 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:24:00.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f687006b4d0 msgr2=0x7f687006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f687006b4d0 0x7f687006d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f68880045b0 tx=0x7f6888009040 comp rx=0 tx=0).stop 2024-03-06T07:24:00.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c159160 msgr2=0x7f688c1ef120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c159160 0x7f688c1ef120 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f688400c7f0 tx=0x7f688400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 shutdown_connections 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f687006b4d0 0x7f687006d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f688c159160 0x7f688c1ef120 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688c06b3a0 0x7f688c1eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 --2- 172.21.15.94:0/1010079861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f688c06afc0 0x7f688c1519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 >> 172.21.15.94:0/1010079861 conn(0x7f688c0c4220 msgr2=0x7f688c073e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 shutdown_connections 2024-03-06T07:24:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.830+0000 7f6893c8a640 1 -- 172.21.15.94:0/1010079861 wait complete. 2024-03-06T07:24:00.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:00.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.942+0000 7f513731e640 1 Processor -- start 2024-03-06T07:24:00.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.942+0000 7f513731e640 1 -- start start 2024-03-06T07:24:00.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.942+0000 7f513731e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301550c0 0x7f51301554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f513731e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51301559e0 0x7f513014ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f513731e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51300aa660 con 0x7f51300aa180 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f513731e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51300c4610 con 0x7f51301559e0 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f513731e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51300aa940 con 0x7f51301550c0 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51348b9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51301559e0 0x7f513014ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51348b9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51301559e0 0x7f513014ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36808/0 (socket says 172.21.15.94:36808) 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51348b9640 1 -- 172.21.15.94:0/1253402524 learned_addr learned my addr 172.21.15.94:0/1253402524 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51350ba640 1 --2- 172.21.15.94:0/1253402524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301550c0 0x7f51301554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51348b9640 1 -- 172.21.15.94:0/1253402524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301550c0 msgr2=0x7f51301554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51348b9640 1 --2- 172.21.15.94:0/1253402524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301550c0 0x7f51301554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51348b9640 1 -- 172.21.15.94:0/1253402524 >> v1:172.21.15.145:6789/0 conn(0x7f51300aa180 legacy=0x7f51300c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51348b9640 1 -- 172.21.15.94:0/1253402524 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300c4860 con 0x7f51301559e0 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.943+0000 7f51350ba640 1 --2- 172.21.15.94:0/1253402524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301550c0 0x7f51301554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:00.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f51348b9640 1 --2- 172.21.15.94:0/1253402524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51301559e0 0x7f513014ec00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f512c009870 tx=0x7f512c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96560c917d8342c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f5127fff640 1 -- 172.21.15.94:0/1253402524 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c038070 con 0x7f51301559e0 2024-03-06T07:24:00.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f5127fff640 1 -- 172.21.15.94:0/1253402524 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c031740 con 0x7f51301559e0 2024-03-06T07:24:00.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f5127fff640 1 -- 172.21.15.94:0/1253402524 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c02f050 con 0x7f51301559e0 2024-03-06T07:24:00.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f513731e640 1 -- 172.21.15.94:0/1253402524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51301559e0 msgr2=0x7f513014ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f513731e640 1 --2- 172.21.15.94:0/1253402524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51301559e0 0x7f513014ec00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f512c009870 tx=0x7f512c02b7e0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f513731e640 1 -- 172.21.15.94:0/1253402524 shutdown_connections 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f513731e640 1 --2- 172.21.15.94:0/1253402524 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51301559e0 0x7f513014ec00 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f513731e640 1 --2- 172.21.15.94:0/1253402524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301550c0 0x7f51301554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.944+0000 7f513731e640 1 -- 172.21.15.94:0/1253402524 >> 172.21.15.94:0/1253402524 conn(0x7f51300c6710 msgr2=0x7f51300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.945+0000 7f513731e640 1 -- 172.21.15.94:0/1253402524 shutdown_connections 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.945+0000 7f513731e640 1 -- 172.21.15.94:0/1253402524 wait complete. 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 Processor -- start 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 -- start start 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51300aa180 0x7f51301dc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51301550c0 0x7f51300a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301559e0 0x7f51300a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513014f140 con 0x7f51300aa180 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300c1520 con 0x7f51301550c0 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f513731e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513014f2c0 con 0x7f51301559e0 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f51350ba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51300aa180 0x7f51301dc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f51350ba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51300aa180 0x7f51301dc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36822/0 (socket says 172.21.15.94:36822) 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f51350ba640 1 -- 172.21.15.94:0/3121272857 learned_addr learned my addr 172.21.15.94:0/3121272857 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f51358bb640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301559e0 0x7f51300a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.946+0000 7f51348b9640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51301550c0 0x7f51300a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f51350ba640 1 -- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301559e0 msgr2=0x7f51300a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f51350ba640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301559e0 0x7f51300a6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f51350ba640 1 -- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51301550c0 msgr2=0x7f51300a6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f51350ba640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51301550c0 0x7f51300a6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f51350ba640 1 -- 172.21.15.94:0/3121272857 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51301eacd0 con 0x7f51300aa180 2024-03-06T07:24:00.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f51350ba640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51300aa180 0x7f51301dc090 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f512000c820 tx=0x7f512000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f5125ffb640 1 -- 172.21.15.94:0/3121272857 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5120015070 con 0x7f51300aa180 2024-03-06T07:24:00.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f5125ffb640 1 -- 172.21.15.94:0/3121272857 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5120007a30 con 0x7f51300aa180 2024-03-06T07:24:00.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513006d860 con 0x7f51300aa180 2024-03-06T07:24:00.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f5125ffb640 1 -- 172.21.15.94:0/3121272857 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5120007e00 con 0x7f51300aa180 2024-03-06T07:24:00.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.947+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513006dde0 con 0x7f51300aa180 2024-03-06T07:24:00.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.948+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5130152fd0 con 0x7f51300aa180 2024-03-06T07:24:00.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.948+0000 7f5125ffb640 1 -- 172.21.15.94:0/3121272857 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5120018070 con 0x7f51300aa180 2024-03-06T07:24:00.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.949+0000 7f5125ffb640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f511406b500 0x7f511406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.949+0000 7f5125ffb640 1 -- 172.21.15.94:0/3121272857 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5120089f40 con 0x7f51300aa180 2024-03-06T07:24:00.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.949+0000 7f51348b9640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f511406b500 0x7f511406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.950+0000 7f51348b9640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f511406b500 0x7f511406d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f512c035000 tx=0x7f512c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:00.953+0000 7f5125ffb640 1 -- 172.21.15.94:0/3121272857 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5130152fd0 con 0x7f51300aa180 2024-03-06T07:24:01.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.124+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5130068450 con 0x7f51300aa180 2024-03-06T07:24:01.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.124+0000 7f5125ffb640 1 -- 172.21.15.94:0/3121272857 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v145) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5120059010 con 0x7f51300aa180 2024-03-06T07:24:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:01.126 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:24:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:01.126 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:24:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:24:01.127 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:01.127 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:24:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f511406b500 msgr2=0x7f511406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f511406b500 0x7f511406d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f512c035000 tx=0x7f512c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51300aa180 msgr2=0x7f51301dc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51300aa180 0x7f51301dc090 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f512000c820 tx=0x7f512000cc20 comp rx=0 tx=0).stop 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 shutdown_connections 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f511406b500 0x7f511406d9c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51301559e0 0x7f51300a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51301550c0 0x7f51300a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 --2- 172.21.15.94:0/3121272857 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51300aa180 0x7f51301dc090 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 >> 172.21.15.94:0/3121272857 conn(0x7f51300c6710 msgr2=0x7f5130153f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 shutdown_connections 2024-03-06T07:24:01.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.126+0000 7f513731e640 1 -- 172.21.15.94:0/3121272857 wait complete. 2024-03-06T07:24:01.139 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:01.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:01.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.241+0000 7f05f5b4b640 1 Processor -- start 2024-03-06T07:24:01.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.241+0000 7f05f5b4b640 1 -- start start 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05f5b4b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 0x7f05f01502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05f5b4b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05f00c1d10 0x7f05f0156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05f5b4b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f05f00c2920 0x7f05f0156980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05f5b4b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f00711f0 con 0x7f05f00c2920 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05f5b4b640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f0070d60 con 0x7f05f00c1d10 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05f5b4b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0070430 con 0x7f05f00abb60 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f05f00c2920 0x7f05f0156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f05f00c2920 0x7f05f0156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36828/0 (socket says 172.21.15.94:36828) 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 -- 172.21.15.94:0/2771430739 learned_addr learned my addr 172.21.15.94:0/2771430739 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 -- 172.21.15.94:0/2771430739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 msgr2=0x7f05f01502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 --2- 172.21.15.94:0/2771430739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 0x7f05f01502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 -- 172.21.15.94:0/2771430739 >> v2:172.21.15.145:3300/0 conn(0x7f05f00c1d10 msgr2=0x7f05f0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 --2- 172.21.15.94:0/2771430739 >> v2:172.21.15.145:3300/0 conn(0x7f05f00c1d10 0x7f05f0156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.242+0000 7f05effff640 1 -- 172.21.15.94:0/2771430739 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0070710 con 0x7f05f00c2920 2024-03-06T07:24:01.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.243+0000 7f05effff640 1 --2- 172.21.15.94:0/2771430739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f0156980 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f05f00ac3f0 tx=0x7f05e802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2df983e5f8d665c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.243+0000 7f05ee7fc640 1 -- 172.21.15.94:0/2771430739 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e802c9e0 con 0x7f05f00c2920 2024-03-06T07:24:01.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.243+0000 7f05ee7fc640 1 -- 172.21.15.94:0/2771430739 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e802cb80 con 0x7f05f00c2920 2024-03-06T07:24:01.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.243+0000 7f05ee7fc640 1 -- 172.21.15.94:0/2771430739 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e80343f0 con 0x7f05f00c2920 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.243+0000 7f05f5b4b640 1 -- 172.21.15.94:0/2771430739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 msgr2=0x7f05f0156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.243+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/2771430739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f0156980 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f05f00ac3f0 tx=0x7f05e802be40 comp rx=0 tx=0).stop 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.244+0000 7f05f5b4b640 1 -- 172.21.15.94:0/2771430739 shutdown_connections 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.244+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/2771430739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f0156980 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.244+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/2771430739 >> v2:172.21.15.145:3300/0 conn(0x7f05f00c1d10 0x7f05f0156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.244+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/2771430739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 0x7f05f01502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.244+0000 7f05f5b4b640 1 -- 172.21.15.94:0/2771430739 >> 172.21.15.94:0/2771430739 conn(0x7f05f00c65b0 msgr2=0x7f05f00c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.244+0000 7f05f5b4b640 1 -- 172.21.15.94:0/2771430739 shutdown_connections 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.244+0000 7f05f5b4b640 1 -- 172.21.15.94:0/2771430739 wait complete. 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 Processor -- start 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 -- start start 2024-03-06T07:24:01.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 0x7f05f01dcb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f00c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0152d90 0x7f05f01e8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0157040 con 0x7f05f00c2920 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0156ec0 con 0x7f05f0152d90 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.245+0000 7f05f5b4b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00abf40 con 0x7f05f00abb60 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f00c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f00c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36844/0 (socket says 172.21.15.94:36844) 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 -- 172.21.15.94:0/320478024 learned_addr learned my addr 172.21.15.94:0/320478024 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 -- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 msgr2=0x7f05f01dcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 0x7f05f01dcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 -- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0152d90 msgr2=0x7f05f01e8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0152d90 0x7f05f01e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 -- 172.21.15.94:0/320478024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0153e30 con 0x7f05f00c2920 2024-03-06T07:24:01.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05eeffd640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f00c5440 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f05e400b4a0 tx=0x7f05e400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05d7fff640 1 -- 172.21.15.94:0/320478024 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e4014070 con 0x7f05f00c2920 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05d7fff640 1 -- 172.21.15.94:0/320478024 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e4010610 con 0x7f05f00c2920 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f014dc10 con 0x7f05f00c2920 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05d7fff640 1 -- 172.21.15.94:0/320478024 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e4010a00 con 0x7f05f00c2920 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.246+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f01dc350 con 0x7f05f00c2920 2024-03-06T07:24:01.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.247+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f004f330 con 0x7f05f00c2920 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.247+0000 7f05d7fff640 1 -- 172.21.15.94:0/320478024 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05e4017070 con 0x7f05f00c2920 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.248+0000 7f05d7fff640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05dc06b4d0 0x7f05dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.248+0000 7f05d7fff640 1 -- 172.21.15.94:0/320478024 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f05e40899b0 con 0x7f05f00c2920 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.248+0000 7f05ef7fe640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05dc06b4d0 0x7f05dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.249+0000 7f05ef7fe640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05dc06b4d0 0x7f05dc06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f05f0156b70 tx=0x7f05e0006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.252+0000 7f05d7fff640 1 -- 172.21.15.94:0/320478024 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05e4058a80 con 0x7f05f00c2920 2024-03-06T07:24:01.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.424+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f05f0072980 con 0x7f05f00c2920 2024-03-06T07:24:01.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.863+0000 7f05d7fff640 1 -- 172.21.15.94:0/320478024 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v146) v1 ==== 138+0+0 (secure 0 0 0) 0x7f05e4058380 con 0x7f05f00c2920 2024-03-06T07:24:01.865 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05dc06b4d0 msgr2=0x7f05dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05dc06b4d0 0x7f05dc06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f05f0156b70 tx=0x7f05e0006c90 comp rx=0 tx=0).stop 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 msgr2=0x7f05f00c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f00c5440 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f05e400b4a0 tx=0x7f05e400b960 comp rx=0 tx=0).stop 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 shutdown_connections 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f05dc06b4d0 0x7f05dc06d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f05f0152d90 0x7f05f01e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f05f00c2920 0x7f05f00c5440 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 --2- 172.21.15.94:0/320478024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05f00abb60 0x7f05f01dcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 >> 172.21.15.94:0/320478024 conn(0x7f05f00c65b0 msgr2=0x7f05f01500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.866+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 shutdown_connections 2024-03-06T07:24:01.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.867+0000 7f05f5b4b640 1 -- 172.21.15.94:0/320478024 wait complete. 2024-03-06T07:24:01.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.989+0000 7f422fe39640 1 Processor -- start 2024-03-06T07:24:01.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.990+0000 7f422fe39640 1 -- start start 2024-03-06T07:24:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.990+0000 7f422fe39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 0x7f42280ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.990+0000 7f422fe39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f42281538c0 0x7f42281494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.990+0000 7f422fe39640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4228154330 con 0x7f42280a97c0 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.990+0000 7f422fe39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42280c46a0 con 0x7f42281538c0 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.990+0000 7f422fe39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4228153dc0 con 0x7f42280ab310 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422e3d6640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f42280a97c0 0x7f42280c4b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37880/0 (socket says 172.21.15.94:37880) 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422e3d6640 1 -- 172.21.15.94:0/1880233574 learned_addr learned my addr 172.21.15.94:0/1880233574 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422dbd5640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 0x7f42280ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422d3d4640 1 --2- 172.21.15.94:0/1880233574 >> v2:172.21.15.145:3300/0 conn(0x7f42281538c0 0x7f42281494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703015351 0 0) 0x7f4228154330 con 0x7f42280a97c0 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4214003520 con 0x7f42280a97c0 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752800384 0 0) 0x7f4214003520 con 0x7f42280a97c0 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 msgr2=0x7f42280ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 0x7f42280ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 >> v2:172.21.15.145:3300/0 conn(0x7f42281538c0 msgr2=0x7f42281494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 --2- 172.21.15.94:0/1880233574 >> v2:172.21.15.145:3300/0 conn(0x7f42281538c0 0x7f42281494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42280c5d30 con 0x7f42280a97c0 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f42200019f0 con 0x7f42280a97c0 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 >> v1:172.21.15.94:6789/0 conn(0x7f42280a97c0 legacy=0x7f42280c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 0x7f42140039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4214003f50 0x7f4214004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4214004d40 0x7f42140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280c46a0 con 0x7f4214003f50 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4228153dc0 con 0x7f4214004d40 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421400ba40 con 0x7f42280ab310 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422d3d4640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4214003f50 0x7f4214004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422dbd5640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 0x7f42140039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.991+0000 7f422e3d6640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4214004d40 0x7f42140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422d3d4640 1 -- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 msgr2=0x7f42140039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422d3d4640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42280ab310 0x7f42140039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422d3d4640 1 -- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4214004d40 msgr2=0x7f42140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422d3d4640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4214004d40 0x7f42140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422d3d4640 1 -- 172.21.15.94:0/1880233574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4214003520 con 0x7f4214003f50 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422d3d4640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4214003f50 0x7f4214004800 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f421c009ac0 tx=0x7f4214005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753618ce10565605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c039070 con 0x7f4214003f50 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.992+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c02c710 con 0x7f4214003f50 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422cbd3640 1 -- 172.21.15.94:0/1880233574 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c009690 con 0x7f4214003f50 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422fe39640 1 -- 172.21.15.94:0/1880233574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4214003f50 msgr2=0x7f4214004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422fe39640 1 --2- 172.21.15.94:0/1880233574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4214003f50 0x7f4214004800 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f421c009ac0 tx=0x7f4214005830 comp rx=0 tx=0).stop 2024-03-06T07:24:02.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422e3d6640 1 -- 172.21.15.94:0/1880233574 reap_dead start 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422fe39640 1 -- 172.21.15.94:0/1880233574 shutdown_connections 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422fe39640 1 -- 172.21.15.94:0/1880233574 >> 172.21.15.94:0/1880233574 conn(0x7f42280c6710 msgr2=0x7f42280c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422fe39640 1 -- 172.21.15.94:0/1880233574 shutdown_connections 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.993+0000 7f422fe39640 1 -- 172.21.15.94:0/1880233574 wait complete. 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.994+0000 7f422fe39640 1 Processor -- start 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.994+0000 7f422fe39640 1 -- start start 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422fe39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4214003f50 0x7f4228071320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422fe39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4228071860 0x7f42281e9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422fe39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42281ea4c0 0x7f422814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422fe39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280c5240 con 0x7f42281ea4c0 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422fe39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280c62f0 con 0x7f4228071860 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422fe39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42280c53c0 con 0x7f4214003f50 2024-03-06T07:24:02.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42281ea4c0 0x7f422814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42281ea4c0 0x7f422814f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36860/0 (socket says 172.21.15.94:36860) 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 -- 172.21.15.94:0/1971563686 learned_addr learned my addr 172.21.15.94:0/1971563686 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422dbd5640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4214003f50 0x7f4228071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422d3d4640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4228071860 0x7f42281e9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 -- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4214003f50 msgr2=0x7f4228071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4214003f50 0x7f4228071320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 -- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4228071860 msgr2=0x7f42281e9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4228071860 0x7f42281e9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.995+0000 7f422e3d6640 1 -- 172.21.15.94:0/1971563686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42280be870 con 0x7f42281ea4c0 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.996+0000 7f422e3d6640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42281ea4c0 0x7f422814f500 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f4220001560 tx=0x7f4220007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.996+0000 7f421a7fc640 1 -- 172.21.15.94:0/1971563686 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220017020 con 0x7f42281ea4c0 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.996+0000 7f421a7fc640 1 -- 172.21.15.94:0/1971563686 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422000ab00 con 0x7f42281ea4c0 2024-03-06T07:24:02.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.996+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42280beea0 con 0x7f42281ea4c0 2024-03-06T07:24:02.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.996+0000 7f421a7fc640 1 -- 172.21.15.94:0/1971563686 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42200133f0 con 0x7f42281ea4c0 2024-03-06T07:24:02.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.996+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42280bb730 con 0x7f42281ea4c0 2024-03-06T07:24:02.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.997+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422804f330 con 0x7f42281ea4c0 2024-03-06T07:24:02.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.997+0000 7f421a7fc640 1 -- 172.21.15.94:0/1971563686 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f422001a020 con 0x7f42281ea4c0 2024-03-06T07:24:02.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.998+0000 7f421a7fc640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f420c06b500 0x7f420c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.998+0000 7f421a7fc640 1 -- 172.21.15.94:0/1971563686 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f422008a700 con 0x7f42281ea4c0 2024-03-06T07:24:02.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.998+0000 7f422dbd5640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f420c06b500 0x7f420c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:01.998+0000 7f422dbd5640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f420c06b500 0x7f420c06d9c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f42280a5da0 tx=0x7f4224012040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.002+0000 7f421a7fc640 1 -- 172.21.15.94:0/1971563686 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4220021020 con 0x7f42281ea4c0 2024-03-06T07:24:02.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.173+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f42280a9330 con 0x7f42281ea4c0 2024-03-06T07:24:02.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.868+0000 7f421a7fc640 1 -- 172.21.15.94:0/1971563686 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v147) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4220059660 con 0x7f42281ea4c0 2024-03-06T07:24:02.871 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f420c06b500 msgr2=0x7f420c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f420c06b500 0x7f420c06d9c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f42280a5da0 tx=0x7f4224012040 comp rx=0 tx=0).stop 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42281ea4c0 msgr2=0x7f422814f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42281ea4c0 0x7f422814f500 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f4220001560 tx=0x7f4220007a40 comp rx=0 tx=0).stop 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 shutdown_connections 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f420c06b500 0x7f420c06d9c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42281ea4c0 0x7f422814f500 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4228071860 0x7f42281e9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 --2- 172.21.15.94:0/1971563686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4214003f50 0x7f4228071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 >> 172.21.15.94:0/1971563686 conn(0x7f42280c6710 msgr2=0x7f422806dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 shutdown_connections 2024-03-06T07:24:02.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:02.872+0000 7f422fe39640 1 -- 172.21.15.94:0/1971563686 wait complete. 2024-03-06T07:24:02.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:03.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f915d765640 1 Processor -- start 2024-03-06T07:24:03.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f915d765640 1 -- start start 2024-03-06T07:24:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f915d765640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f915d765640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91580c77d0 0x7f915806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f915d765640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9158155260 con 0x7f91580c7370 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f915d765640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91580c7c00 con 0x7f91580c77d0 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f915d765640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9158155540 con 0x7f91581527f0 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f91577fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91580c77d0 0x7f915806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f91577fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f91580c77d0 0x7f915806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36874/0 (socket says 172.21.15.94:36874) 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.008+0000 7f91577fe640 1 -- 172.21.15.94:0/3522870412 learned_addr learned my addr 172.21.15.94:0/3522870412 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.009+0000 7f9156ffd640 1 --2- 172.21.15.94:0/3522870412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.009+0000 7f91577fe640 1 -- 172.21.15.94:0/3522870412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 msgr2=0x7f91580c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.009+0000 7f91577fe640 1 --2- 172.21.15.94:0/3522870412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.009+0000 7f91577fe640 1 -- 172.21.15.94:0/3522870412 >> v1:172.21.15.145:6789/0 conn(0x7f91580c7370 legacy=0x7f915806c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.009+0000 7f91577fe640 1 -- 172.21.15.94:0/3522870412 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9158155820 con 0x7f91580c77d0 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.009+0000 7f91577fe640 1 --2- 172.21.15.94:0/3522870412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c77d0 0x7f915806c7c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f9150009e50 tx=0x7f915002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a143034ee812d2cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f91567fc640 1 -- 172.21.15.94:0/3522870412 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915002c9e0 con 0x7f91580c77d0 2024-03-06T07:24:03.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f91567fc640 1 -- 172.21.15.94:0/3522870412 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915002cb80 con 0x7f91580c77d0 2024-03-06T07:24:03.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f91567fc640 1 -- 172.21.15.94:0/3522870412 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91500343f0 con 0x7f91580c77d0 2024-03-06T07:24:03.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 -- 172.21.15.94:0/3522870412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c77d0 msgr2=0x7f915806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 --2- 172.21.15.94:0/3522870412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c77d0 0x7f915806c7c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f9150009e50 tx=0x7f915002be40 comp rx=0 tx=0).stop 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 -- 172.21.15.94:0/3522870412 shutdown_connections 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 --2- 172.21.15.94:0/3522870412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c77d0 0x7f915806c7c0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 --2- 172.21.15.94:0/3522870412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 -- 172.21.15.94:0/3522870412 >> 172.21.15.94:0/3522870412 conn(0x7f91580ac4f0 msgr2=0x7f91580ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 -- 172.21.15.94:0/3522870412 shutdown_connections 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.010+0000 7f915d765640 1 -- 172.21.15.94:0/3522870412 wait complete. 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.011+0000 7f915d765640 1 Processor -- start 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f915d765640 1 -- start start 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f915d765640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c7370 0x7f915814b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f915d765640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91580c77d0 0x7f91580c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f915d765640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f915d765640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91580be1a0 con 0x7f91580c7370 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f915d765640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915806cd00 con 0x7f91580c77d0 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f915d765640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91580c8da0 con 0x7f91581527f0 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c7370 0x7f915814b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f9156ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c7370 0x7f915814b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36878/0 (socket says 172.21.15.94:36878) 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f9156ffd640 1 -- 172.21.15.94:0/1381574121 learned_addr learned my addr 172.21.15.94:0/1381574121 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.012+0000 7f91577fe640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f9156ffd640 1 -- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 msgr2=0x7f91580c21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f9156ffd640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f9156ffd640 1 -- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91580c77d0 msgr2=0x7f91580c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f914e7fc640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91580c77d0 0x7f91580c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f9156ffd640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91580c77d0 0x7f91580c4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f9156ffd640 1 -- 172.21.15.94:0/1381574121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91580c1a30 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f91577fe640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f9156ffd640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c7370 0x7f915814b730 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f914000e8a0 tx=0x7f914000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f914ffff640 1 -- 172.21.15.94:0/1381574121 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140017070 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f914ffff640 1 -- 172.21.15.94:0/1381574121 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914000c940 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915814c120 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f914ffff640 1 -- 172.21.15.94:0/1381574121 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914000cd30 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.013+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91580afa60 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.014+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915804f330 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.014+0000 7f914ffff640 1 -- 172.21.15.94:0/1381574121 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f914001a070 con 0x7f91580c7370 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.015+0000 7f914ffff640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.015+0000 7f914ffff640 1 -- 172.21.15.94:0/1381574121 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f914008a140 con 0x7f91580c7370 2024-03-06T07:24:03.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.015+0000 7f914e7fc640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.016+0000 7f914e7fc640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f915814ec80 tx=0x7f9148009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.019+0000 7f914ffff640 1 -- 172.21.15.94:0/1381574121 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9140058f20 con 0x7f91580c7370 2024-03-06T07:24:03.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.228+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f915806d860 con 0x7f91580c7370 2024-03-06T07:24:03.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.870+0000 7f914ffff640 1 -- 172.21.15.94:0/1381574121 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v133) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9140058820 con 0x7f91580c7370 2024-03-06T07:24:03.872 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 23 and data pool 24 2024-03-06T07:24:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.874+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f913806b4d0 msgr2=0x7f913806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.874+0000 7f915d765640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f915814ec80 tx=0x7f9148009040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.874+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c7370 msgr2=0x7f915814b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.874+0000 7f915d765640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c7370 0x7f915814b730 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f914000e8a0 tx=0x7f914000ed60 comp rx=0 tx=0).stop 2024-03-06T07:24:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.875+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 shutdown_connections 2024-03-06T07:24:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.875+0000 7f915d765640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.875+0000 7f915d765640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91581527f0 0x7f91580c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.875+0000 7f915d765640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91580c77d0 0x7f91580c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.875+0000 7f915d765640 1 --2- 172.21.15.94:0/1381574121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91580c7370 0x7f915814b730 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.875+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 >> 172.21.15.94:0/1381574121 conn(0x7f91580ac4f0 msgr2=0x7f91580ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.876+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 shutdown_connections 2024-03-06T07:24:03.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:03.878+0000 7f915d765640 1 -- 172.21.15.94:0/1381574121 wait complete. 2024-03-06T07:24:03.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:04.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.010+0000 7ff0f716b640 1 Processor -- start 2024-03-06T07:24:04.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.010+0000 7ff0f716b640 1 -- start start 2024-03-06T07:24:04.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f716b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0f0159270 0x7ff0f0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f716b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f00c73d0 con 0x7ff0f00c88c0 2024-03-06T07:24:04.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f716b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f00be2e0 con 0x7ff0f014f360 2024-03-06T07:24:04.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f716b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f0159700 con 0x7ff0f0159270 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f5708640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0f0159270 0x7ff0f0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f5708640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0f0159270 0x7ff0f0155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36886/0 (socket says 172.21.15.94:36886) 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0effff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff0f00c88c0 0x7ff0f0158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37884/0 (socket says 172.21.15.94:37884) 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f5708640 1 -- 172.21.15.94:0/3977811135 learned_addr learned my addr 172.21.15.94:0/3977811135 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f5708640 1 -- 172.21.15.94:0/3977811135 >> v1:172.21.15.145:6789/0 conn(0x7ff0f014f360 legacy=0x7ff0f00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0f5708640 1 -- 172.21.15.94:0/3977811135 >> v1:172.21.15.94:6789/0 conn(0x7ff0f00c88c0 legacy=0x7ff0f0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.011+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3977811135 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550798117 0 0) 0x7ff0f00c73d0 con 0x7ff0f00c88c0 2024-03-06T07:24:04.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.012+0000 7ff0f5708640 1 -- 172.21.15.94:0/3977811135 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f00c7670 con 0x7ff0f0159270 2024-03-06T07:24:04.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.012+0000 7ff0f5708640 1 --2- 172.21.15.94:0/3977811135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f0159270 0x7ff0f0155c10 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009870 tx=0x7ff0e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d441ff2120a50cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.012+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3977811135 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0035020 con 0x7ff0f0159270 2024-03-06T07:24:04.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.012+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3977811135 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e0031870 con 0x7ff0f0159270 2024-03-06T07:24:04.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.012+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3977811135 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0031c40 con 0x7ff0f0159270 2024-03-06T07:24:04.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.013+0000 7ff0f716b640 1 -- 172.21.15.94:0/3977811135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f0159270 msgr2=0x7ff0f0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.013+0000 7ff0f716b640 1 --2- 172.21.15.94:0/3977811135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f0159270 0x7ff0f0155c10 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009870 tx=0x7ff0e002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:04.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.013+0000 7ff0f716b640 1 -- 172.21.15.94:0/3977811135 shutdown_connections 2024-03-06T07:24:04.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.013+0000 7ff0f716b640 1 --2- 172.21.15.94:0/3977811135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f0159270 0x7ff0f0155c10 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.013+0000 7ff0f716b640 1 -- 172.21.15.94:0/3977811135 >> 172.21.15.94:0/3977811135 conn(0x7ff0f00ac570 msgr2=0x7ff0f00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.013+0000 7ff0f716b640 1 -- 172.21.15.94:0/3977811135 shutdown_connections 2024-03-06T07:24:04.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.013+0000 7ff0f716b640 1 -- 172.21.15.94:0/3977811135 wait complete. 2024-03-06T07:24:04.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.014+0000 7ff0f716b640 1 Processor -- start 2024-03-06T07:24:04.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.014+0000 7ff0f716b640 1 -- start start 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f716b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00c88c0 0x7ff0f00bf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f716b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f014f360 0x7ff0f01ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f716b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f0159270 0x7ff0f01ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f716b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c7c30 con 0x7ff0f00c88c0 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f716b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c5a50 con 0x7ff0f014f360 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f716b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c5bd0 con 0x7ff0f0159270 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00c88c0 0x7ff0f00bf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00c88c0 0x7ff0f00bf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36898/0 (socket says 172.21.15.94:36898) 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 -- 172.21.15.94:0/168900907 learned_addr learned my addr 172.21.15.94:0/168900907 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0effff640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f014f360 0x7ff0f01ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f5708640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f0159270 0x7ff0f01ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 -- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f0159270 msgr2=0x7ff0f01ee4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f0159270 0x7ff0f01ee4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 -- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f014f360 msgr2=0x7ff0f01ec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f014f360 0x7ff0f01ec180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 -- 172.21.15.94:0/168900907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f00c0fc0 con 0x7ff0f00c88c0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.015+0000 7ff0f4f07640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00c88c0 0x7ff0f00bf880 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800c7c0 tx=0x7ff0e800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.016+0000 7ff0ed7fa640 1 -- 172.21.15.94:0/168900907 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8016020 con 0x7ff0f00c88c0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.016+0000 7ff0ed7fa640 1 -- 172.21.15.94:0/168900907 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e800aa50 con 0x7ff0f00c88c0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.016+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f00c12a0 con 0x7ff0f00c88c0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.016+0000 7ff0ed7fa640 1 -- 172.21.15.94:0/168900907 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8012420 con 0x7ff0f00c88c0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.016+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f00bf360 con 0x7ff0f00c88c0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.016+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f006b6d0 con 0x7ff0f00c88c0 2024-03-06T07:24:04.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.017+0000 7ff0ed7fa640 1 -- 172.21.15.94:0/168900907 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0e8019020 con 0x7ff0f00c88c0 2024-03-06T07:24:04.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.017+0000 7ff0ed7fa640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.018+0000 7ff0ed7fa640 1 -- 172.21.15.94:0/168900907 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff0e808a0d0 con 0x7ff0f00c88c0 2024-03-06T07:24:04.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.018+0000 7ff0effff640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.018+0000 7ff0effff640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff0f01e8b50 tx=0x7ff0e4007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.022+0000 7ff0ed7fa640 1 -- 172.21.15.94:0/168900907 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0f006b6d0 con 0x7ff0f00c88c0 2024-03-06T07:24:04.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.193+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0f004f330 con 0x7ff0f00c88c0 2024-03-06T07:24:04.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.194+0000 7ff0ed7fa640 1 -- 172.21.15.94:0/168900907 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16897 (secure 0 0 0) 0x7ff0e8020070 con 0x7ff0f00c88c0 2024-03-06T07:24:04.196 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:04.197 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":148,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:03.229570+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:04.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 msgr2=0x7ff0dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff0f01e8b50 tx=0x7ff0e4007b50 comp rx=0 tx=0).stop 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00c88c0 msgr2=0x7ff0f00bf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00c88c0 0x7ff0f00bf880 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800c7c0 tx=0x7ff0e800cc80 comp rx=0 tx=0).stop 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 shutdown_connections 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f0159270 0x7ff0f01ee4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f014f360 0x7ff0f01ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.196+0000 7ff0f716b640 1 --2- 172.21.15.94:0/168900907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00c88c0 0x7ff0f00bf880 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.197+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 >> 172.21.15.94:0/168900907 conn(0x7ff0f00ac570 msgr2=0x7ff0f00c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.197+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 shutdown_connections 2024-03-06T07:24:04.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.197+0000 7ff0f716b640 1 -- 172.21.15.94:0/168900907 wait complete. 2024-03-06T07:24:04.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:04.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.308+0000 7f1bf0372640 1 Processor -- start 2024-03-06T07:24:04.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.308+0000 7f1bf0372640 1 -- start start 2024-03-06T07:24:04.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bf0372640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80c3370 0x7f1be8155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bf0372640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1be80a5e60 0x7f1be8156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bf0372640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be80c5490 con 0x7f1be80c3750 2024-03-06T07:24:04.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bf0372640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be80c5770 con 0x7f1be80a5e60 2024-03-06T07:24:04.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bf0372640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be80c5a50 con 0x7f1be80c3370 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee10e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80c3370 0x7f1be8155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee90f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1be80a5e60 0x7f1be8156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bed90d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f1be80c3750 0x7f1be80c1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52966/0 (socket says 172.21.15.94:52966) 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee10e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80c3370 0x7f1be8155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:41762/0 (socket says 172.21.15.94:41762) 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bed90d640 1 -- 172.21.15.94:0/316298077 learned_addr learned my addr 172.21.15.94:0/316298077 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee10e640 1 -- 172.21.15.94:0/316298077 >> v2:172.21.15.145:3300/0 conn(0x7f1be80a5e60 msgr2=0x7f1be8156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee10e640 1 --2- 172.21.15.94:0/316298077 >> v2:172.21.15.145:3300/0 conn(0x7f1be80a5e60 0x7f1be8156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee10e640 1 -- 172.21.15.94:0/316298077 >> v1:172.21.15.145:6789/0 conn(0x7f1be80c3750 legacy=0x7f1be80c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee10e640 1 -- 172.21.15.94:0/316298077 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be80c5d30 con 0x7f1be80c3370 2024-03-06T07:24:04.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.309+0000 7f1bee90f640 1 --2- 172.21.15.94:0/316298077 >> v2:172.21.15.145:3300/0 conn(0x7f1be80a5e60 0x7f1be8156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.310+0000 7f1bee10e640 1 --2- 172.21.15.94:0/316298077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80c3370 0x7f1be8155ad0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1be40097a0 tx=0x7f1be402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c83318ec04a9a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.310+0000 7f1bed10c640 1 -- 172.21.15.94:0/316298077 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4038070 con 0x7f1be80c3370 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.310+0000 7f1bed10c640 1 -- 172.21.15.94:0/316298077 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be4031700 con 0x7f1be80c3370 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.310+0000 7f1bed10c640 1 -- 172.21.15.94:0/316298077 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4031ad0 con 0x7f1be80c3370 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 -- 172.21.15.94:0/316298077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80c3370 msgr2=0x7f1be8155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 --2- 172.21.15.94:0/316298077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80c3370 0x7f1be8155ad0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1be40097a0 tx=0x7f1be402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 -- 172.21.15.94:0/316298077 shutdown_connections 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 --2- 172.21.15.94:0/316298077 >> v2:172.21.15.145:3300/0 conn(0x7f1be80a5e60 0x7f1be8156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 --2- 172.21.15.94:0/316298077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80c3370 0x7f1be8155ad0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 -- 172.21.15.94:0/316298077 >> 172.21.15.94:0/316298077 conn(0x7f1be80c6710 msgr2=0x7f1be80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 -- 172.21.15.94:0/316298077 shutdown_connections 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.311+0000 7f1bf0372640 1 -- 172.21.15.94:0/316298077 wait complete. 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.312+0000 7f1bf0372640 1 Processor -- start 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.312+0000 7f1bf0372640 1 -- start start 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80a5e60 0x7f1be806ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1be80c3370 0x7f1be8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1be80c3750 0x7f1be8154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8159910 con 0x7f1be80c3370 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8159790 con 0x7f1be80c3750 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be80c4ed0 con 0x7f1be80a5e60 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1be80c3370 0x7f1be8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1be80c3370 0x7f1be8157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36914/0 (socket says 172.21.15.94:36914) 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 -- 172.21.15.94:0/3967739111 learned_addr learned my addr 172.21.15.94:0/3967739111 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bee10e640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80a5e60 0x7f1be806ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bee90f640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1be80c3750 0x7f1be8154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 -- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80a5e60 msgr2=0x7f1be806ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80a5e60 0x7f1be806ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 -- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1be80c3750 msgr2=0x7f1be8154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1be80c3750 0x7f1be8154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 -- 172.21.15.94:0/3967739111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be81eeb80 con 0x7f1be80c3370 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bed90d640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1be80c3370 0x7f1be8157eb0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800c3f0 tx=0x7f1bd800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bdeffd640 1 -- 172.21.15.94:0/3967739111 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8017070 con 0x7f1be80c3370 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bdeffd640 1 -- 172.21.15.94:0/3967739111 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd8005ce0 con 0x7f1be80c3370 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be806d600 con 0x7f1be80c3370 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bdeffd640 1 -- 172.21.15.94:0/3967739111 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd80072f0 con 0x7f1be80c3370 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.313+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be81ee0a0 con 0x7f1be80c3370 2024-03-06T07:24:04.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.314+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be804f330 con 0x7f1be80c3370 2024-03-06T07:24:04.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.315+0000 7f1bdeffd640 1 -- 172.21.15.94:0/3967739111 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bd801a070 con 0x7f1be80c3370 2024-03-06T07:24:04.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.315+0000 7f1bdeffd640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bcc06b4d0 0x7f1bcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.315+0000 7f1bdeffd640 1 -- 172.21.15.94:0/3967739111 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1bd80895a0 con 0x7f1be80c3370 2024-03-06T07:24:04.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.315+0000 7f1bee10e640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bcc06b4d0 0x7f1bcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.316+0000 7f1bee10e640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bcc06b4d0 0x7f1bcc06d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1be4035000 tx=0x7f1be4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.319+0000 7f1bdeffd640 1 -- 172.21.15.94:0/3967739111 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bd8057c40 con 0x7f1be80c3370 2024-03-06T07:24:04.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.492+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1be8068450 con 0x7f1be80c3370 2024-03-06T07:24:04.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.493+0000 7f1bdeffd640 1 -- 172.21.15.94:0/3967739111 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16897 (secure 0 0 0) 0x7f1bd805bce0 con 0x7f1be80c3370 2024-03-06T07:24:04.495 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:04.495 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":148,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:03.229570+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:04.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bcc06b4d0 msgr2=0x7f1bcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bcc06b4d0 0x7f1bcc06d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1be4035000 tx=0x7f1be4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:04.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1be80c3370 msgr2=0x7f1be8157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1be80c3370 0x7f1be8157eb0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800c3f0 tx=0x7f1bd800c7f0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 shutdown_connections 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1bcc06b4d0 0x7f1bcc06d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1be80c3750 0x7f1be8154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1be80c3370 0x7f1be8157eb0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 --2- 172.21.15.94:0/3967739111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1be80a5e60 0x7f1be806ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 >> 172.21.15.94:0/3967739111 conn(0x7f1be80c6710 msgr2=0x7f1be80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 shutdown_connections 2024-03-06T07:24:04.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.495+0000 7f1bf0372640 1 -- 172.21.15.94:0/3967739111 wait complete. 2024-03-06T07:24:04.508 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.610+0000 7f6f7905c640 1 Processor -- start 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.610+0000 7f6f7905c640 1 -- start start 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.610+0000 7f6f7905c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f741538c0 0x7f6f741494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.610+0000 7f6f7905c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f740a97c0 0x7f6f741499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.610+0000 7f6f7905c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f74154330 con 0x7f6f740ab310 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.610+0000 7f6f7905c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f740c46a0 con 0x7f6f740a97c0 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.610+0000 7f6f7905c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f74153dc0 con 0x7f6f741538c0 2024-03-06T07:24:04.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f7359e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f740a97c0 0x7f6f741499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f6bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f741538c0 0x7f6f741494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f72d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6f740ab310 0x7f6f740ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52970/0 (socket says 172.21.15.94:52970) 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f7359e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f740a97c0 0x7f6f741499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:57502/0 (socket says 172.21.15.94:57502) 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f72d9d640 1 -- 172.21.15.94:0/3939674828 learned_addr learned my addr 172.21.15.94:0/3939674828 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f6bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f741538c0 0x7f6f741494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:41774/0 (socket says 172.21.15.94:41774) 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f7259c640 1 -- 172.21.15.94:0/3939674828 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831283399 0 0) 0x7f6f74154330 con 0x7f6f740ab310 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f7259c640 1 -- 172.21.15.94:0/3939674828 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f60003520 con 0x7f6f740ab310 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f6bfff640 1 -- 172.21.15.94:0/3939674828 >> v2:172.21.15.145:3300/0 conn(0x7f6f740a97c0 msgr2=0x7f6f741499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f6bfff640 1 --2- 172.21.15.94:0/3939674828 >> v2:172.21.15.145:3300/0 conn(0x7f6f740a97c0 0x7f6f741499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f6bfff640 1 -- 172.21.15.94:0/3939674828 >> v1:172.21.15.145:6789/0 conn(0x7f6f740ab310 legacy=0x7f6f740ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.611+0000 7f6f6bfff640 1 -- 172.21.15.94:0/3939674828 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740c5d30 con 0x7f6f741538c0 2024-03-06T07:24:04.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.612+0000 7f6f6bfff640 1 --2- 172.21.15.94:0/3939674828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f741538c0 0x7f6f741494a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097a0 tx=0x7f6f5c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe86e150984d6ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.612+0000 7f6f7259c640 1 -- 172.21.15.94:0/3939674828 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c0315a0 con 0x7f6f741538c0 2024-03-06T07:24:04.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.612+0000 7f6f7259c640 1 -- 172.21.15.94:0/3939674828 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5c037070 con 0x7f6f741538c0 2024-03-06T07:24:04.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.612+0000 7f6f7259c640 1 -- 172.21.15.94:0/3939674828 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c031940 con 0x7f6f741538c0 2024-03-06T07:24:04.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 -- 172.21.15.94:0/3939674828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f741538c0 msgr2=0x7f6f741494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3939674828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f741538c0 0x7f6f741494a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097a0 tx=0x7f6f5c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:04.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 -- 172.21.15.94:0/3939674828 shutdown_connections 2024-03-06T07:24:04.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3939674828 >> v2:172.21.15.145:3300/0 conn(0x7f6f740a97c0 0x7f6f741499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3939674828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f741538c0 0x7f6f741494a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 -- 172.21.15.94:0/3939674828 >> 172.21.15.94:0/3939674828 conn(0x7f6f740c6710 msgr2=0x7f6f740c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 -- 172.21.15.94:0/3939674828 shutdown_connections 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.613+0000 7f6f7905c640 1 -- 172.21.15.94:0/3939674828 wait complete. 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.614+0000 7f6f7905c640 1 Processor -- start 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.614+0000 7f6f7905c640 1 -- start start 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7905c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f740a97c0 0x7f6f741506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7905c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f740ab310 0x7f6f7414cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7905c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f741538c0 0x7f6f740bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7905c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f740c4b30 con 0x7f6f740a97c0 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7905c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f740c62f0 con 0x7f6f741538c0 2024-03-06T07:24:04.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7905c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f740c4cb0 con 0x7f6f740ab310 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f740a97c0 0x7f6f741506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f740a97c0 0x7f6f741506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36916/0 (socket says 172.21.15.94:36916) 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 -- 172.21.15.94:0/3098697726 learned_addr learned my addr 172.21.15.94:0/3098697726 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f6bfff640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f740ab310 0x7f6f7414cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7359e640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f741538c0 0x7f6f740bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 -- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f740ab310 msgr2=0x7f6f7414cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f740ab310 0x7f6f7414cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 -- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f741538c0 msgr2=0x7f6f740bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f741538c0 0x7f6f740bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 -- 172.21.15.94:0/3098697726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740a80a0 con 0x7f6f740a97c0 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f72d9d640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f740a97c0 0x7f6f741506a0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c430 tx=0x7f6f6400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f6b7fe640 1 -- 172.21.15.94:0/3098697726 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64017070 con 0x7f6f740a97c0 2024-03-06T07:24:04.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f6b7fe640 1 -- 172.21.15.94:0/3098697726 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f64005ce0 con 0x7f6f740a97c0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f6b7fe640 1 -- 172.21.15.94:0/3098697726 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f640072f0 con 0x7f6f740a97c0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.615+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7406c420 con 0x7f6f740a97c0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.616+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7406c9e0 con 0x7f6f740a97c0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.617+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7404f330 con 0x7f6f740a97c0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.617+0000 7f6f6b7fe640 1 -- 172.21.15.94:0/3098697726 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f6401a070 con 0x7f6f740a97c0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.617+0000 7f6f6b7fe640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f5406b4d0 0x7f6f5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.617+0000 7f6f6b7fe640 1 -- 172.21.15.94:0/3098697726 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6f640897b0 con 0x7f6f740a97c0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.618+0000 7f6f6bfff640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f5406b4d0 0x7f6f5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.618+0000 7f6f6bfff640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f5406b4d0 0x7f6f5406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c036000 tx=0x7f6f5c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.622+0000 7f6f6b7fe640 1 -- 172.21.15.94:0/3098697726 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f64058530 con 0x7f6f740a97c0 2024-03-06T07:24:04.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.832+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6f740c5a70 con 0x7f6f740a97c0 2024-03-06T07:24:04.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.888+0000 7f6f6b7fe640 1 -- 172.21.15.94:0/3098697726 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v134) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6f64057e30 con 0x7f6f740a97c0 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f5406b4d0 msgr2=0x7f6f5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f5406b4d0 0x7f6f5406d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c036000 tx=0x7f6f5c03d040 comp rx=0 tx=0).stop 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f740a97c0 msgr2=0x7f6f741506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f740a97c0 0x7f6f741506a0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c430 tx=0x7f6f6400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 shutdown_connections 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f5406b4d0 0x7f6f5406d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f741538c0 0x7f6f740bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f740ab310 0x7f6f7414cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 --2- 172.21.15.94:0/3098697726 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6f740a97c0 0x7f6f741506a0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 >> 172.21.15.94:0/3098697726 conn(0x7f6f740c6710 msgr2=0x7f6f74154e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 shutdown_connections 2024-03-06T07:24:04.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:04.891+0000 7f6f7905c640 1 -- 172.21.15.94:0/3098697726 wait complete. 2024-03-06T07:24:04.904 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:04.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:05.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.006+0000 7fb04ab8c640 1 Processor -- start 2024-03-06T07:24:05.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.006+0000 7fb04ab8c640 1 -- start start 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb04ab8c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb044072cd0 0x7fb044155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb04ab8c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb044152db0 con 0x7fb044073290 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb04ab8c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0441590f0 con 0x7fb0441529d0 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb04ab8c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440c6dd0 con 0x7fb044072cd0 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb049129640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb044073290 0x7fb044155660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37886/0 (socket says 172.21.15.94:37886) 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb049129640 1 -- 172.21.15.94:0/1745254142 learned_addr learned my addr 172.21.15.94:0/1745254142 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503675226 0 0) 0x7fb03c001490 con 0x7fb044073290 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb02c003520 con 0x7fb044073290 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.007+0000 7fb043fff640 1 --2- 172.21.15.94:0/1745254142 >> v2:172.21.15.145:3300/0 conn(0x7fb044072cd0 0x7fb044155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3445934009 0 0) 0x7fb02c003520 con 0x7fb044073290 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 >> v2:172.21.15.145:3300/0 conn(0x7fb044072cd0 msgr2=0x7fb044155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 --2- 172.21.15.94:0/1745254142 >> v2:172.21.15.145:3300/0 conn(0x7fb044072cd0 0x7fb044155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 >> v1:172.21.15.145:6789/0 conn(0x7fb0441529d0 legacy=0x7fb0440c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb044159650 con 0x7fb044073290 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fb03c001bc0 con 0x7fb044073290 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 >> v1:172.21.15.94:6789/0 conn(0x7fb044073290 legacy=0x7fb044155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb02c0039b0 0x7fb02c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02c004800 0x7fb02c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0052e0 0x7fb02c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440c6dd0 con 0x7fb02c0039b0 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c00c1b0 con 0x7fb02c004800 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c00c3b0 con 0x7fb02c0052e0 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb048928640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb02c0039b0 0x7fb02c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb049129640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0052e0 0x7fb02c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb043fff640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02c004800 0x7fb02c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb048928640 1 -- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0052e0 msgr2=0x7fb02c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb048928640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0052e0 0x7fb02c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb048928640 1 -- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02c004800 msgr2=0x7fb02c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb048928640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02c004800 0x7fb02c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb048928640 1 -- 172.21.15.94:0/1745254142 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c003520 con 0x7fb02c0039b0 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb049129640 1 -- 172.21.15.94:0/1745254142 reap_dead start 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.008+0000 7fb043fff640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb02c004800 0x7fb02c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.009+0000 7fb048928640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb02c0039b0 0x7fb02c003dd0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb034001720 tx=0x7fb02c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5452da04c584f18a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.009+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034001370 con 0x7fb02c0039b0 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.009+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0340052d0 con 0x7fb02c0039b0 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.009+0000 7fb0437fe640 1 -- 172.21.15.94:0/1745254142 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034003830 con 0x7fb02c0039b0 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.009+0000 7fb04ab8c640 1 -- 172.21.15.94:0/1745254142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb02c0039b0 msgr2=0x7fb02c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.009+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb02c0039b0 0x7fb02c003dd0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb034001720 tx=0x7fb02c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.010+0000 7fb04ab8c640 1 -- 172.21.15.94:0/1745254142 shutdown_connections 2024-03-06T07:24:05.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.010+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/1745254142 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb02c0039b0 0x7fb02c003dd0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.010+0000 7fb04ab8c640 1 -- 172.21.15.94:0/1745254142 >> 172.21.15.94:0/1745254142 conn(0x7fb0440aa630 msgr2=0x7fb0440aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.010+0000 7fb04ab8c640 1 -- 172.21.15.94:0/1745254142 shutdown_connections 2024-03-06T07:24:05.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.010+0000 7fb04ab8c640 1 -- 172.21.15.94:0/1745254142 wait complete. 2024-03-06T07:24:05.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 Processor -- start 2024-03-06T07:24:05.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 -- start start 2024-03-06T07:24:05.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0039b0 0x7fb0440a8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04414f9e0 0x7fb04414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb04414e580 0x7fb04414b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb044159370 con 0x7fb04414f9e0 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440c6940 con 0x7fb04414e580 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.011+0000 7fb04ab8c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440723b0 con 0x7fb02c0039b0 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04414f9e0 0x7fb04414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04414f9e0 0x7fb04414b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36932/0 (socket says 172.21.15.94:36932) 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 -- 172.21.15.94:0/2913586349 learned_addr learned my addr 172.21.15.94:0/2913586349 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb048928640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0039b0 0x7fb0440a8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb049129640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb04414e580 0x7fb04414b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 -- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0039b0 msgr2=0x7fb0440a8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0039b0 0x7fb0440a8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 -- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb04414e580 msgr2=0x7fb04414b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb04414e580 0x7fb04414b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 -- 172.21.15.94:0/2913586349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04414d6f0 con 0x7fb04414f9e0 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb048928640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0039b0 0x7fb0440a8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb049129640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb04414e580 0x7fb04414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb043fff640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04414f9e0 0x7fb04414b2a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb038011510 tx=0x7fb0380119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb0417fa640 1 -- 172.21.15.94:0/2913586349 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038015870 con 0x7fb04414f9e0 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb0417fa640 1 -- 172.21.15.94:0/2913586349 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb038015a10 con 0x7fb04414f9e0 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04414bf20 con 0x7fb04414f9e0 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb0417fa640 1 -- 172.21.15.94:0/2913586349 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb038015e00 con 0x7fb04414f9e0 2024-03-06T07:24:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.012+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440a94f0 con 0x7fb04414f9e0 2024-03-06T07:24:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.013+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04414cbd0 con 0x7fb04414f9e0 2024-03-06T07:24:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.014+0000 7fb0417fa640 1 -- 172.21.15.94:0/2913586349 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb03801d400 con 0x7fb04414f9e0 2024-03-06T07:24:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.014+0000 7fb0417fa640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb03006b500 0x7fb03006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.014+0000 7fb0417fa640 1 -- 172.21.15.94:0/2913586349 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb03808aab0 con 0x7fb04414f9e0 2024-03-06T07:24:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.014+0000 7fb048928640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb03006b500 0x7fb03006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.015+0000 7fb048928640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb03006b500 0x7fb03006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb034033000 tx=0x7fb034009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.018+0000 7fb0417fa640 1 -- 172.21.15.94:0/2913586349 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb038019030 con 0x7fb04414f9e0 2024-03-06T07:24:05.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.233+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb04404f330 con 0x7fb04414f9e0 2024-03-06T07:24:05.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.889+0000 7fb0417fa640 1 -- 172.21.15.94:0/2913586349 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v135) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb038059830 con 0x7fb04414f9e0 2024-03-06T07:24:05.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb03006b500 msgr2=0x7fb03006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb03006b500 0x7fb03006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb034033000 tx=0x7fb034009030 comp rx=0 tx=0).stop 2024-03-06T07:24:05.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04414f9e0 msgr2=0x7fb04414b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04414f9e0 0x7fb04414b2a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb038011510 tx=0x7fb0380119d0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 shutdown_connections 2024-03-06T07:24:05.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb03006b500 0x7fb03006d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb04414e580 0x7fb04414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb04414f9e0 0x7fb04414b2a0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 --2- 172.21.15.94:0/2913586349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb02c0039b0 0x7fb0440a8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 >> 172.21.15.94:0/2913586349 conn(0x7fb0440aa630 msgr2=0x7fb0440c48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 shutdown_connections 2024-03-06T07:24:05.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:05.892+0000 7fb04ab8c640 1 -- 172.21.15.94:0/2913586349 wait complete. 2024-03-06T07:24:05.905 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:05.905 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:06.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.004+0000 7f1898f5e640 1 Processor -- start 2024-03-06T07:24:06.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.004+0000 7f1898f5e640 1 -- start start 2024-03-06T07:24:06.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f1898f5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f189414f360 0x7f1894155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f1898f5e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18940c73d0 con 0x7f1894158b60 2024-03-06T07:24:06.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f1898f5e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18940be2e0 con 0x7f18940c88c0 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f1898f5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1894159700 con 0x7f189414f360 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f1892d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1894158b60 0x7f1894158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37888/0 (socket says 172.21.15.94:37888) 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f1892d9d640 1 -- 172.21.15.94:0/1063759530 learned_addr learned my addr 172.21.15.94:0/1063759530 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189259c640 1 --2- 172.21.15.94:0/1063759530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f189414f360 0x7f1894155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429196566 0 0) 0x7f18940c73d0 con 0x7f1894158b60 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187c003520 con 0x7f1894158b60 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332537805 0 0) 0x7f18940be2e0 con 0x7f18940c88c0 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18940c73d0 con 0x7f18940c88c0 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2446879000 0 0) 0x7f187c003520 con 0x7f1894158b60 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189259c640 1 -- 172.21.15.94:0/1063759530 >> v1:172.21.15.145:6789/0 conn(0x7f18940c88c0 legacy=0x7f18940c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189259c640 1 -- 172.21.15.94:0/1063759530 >> v1:172.21.15.94:6789/0 conn(0x7f1894158b60 legacy=0x7f1894158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.005+0000 7f189259c640 1 -- 172.21.15.94:0/1063759530 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18940c7670 con 0x7f189414f360 2024-03-06T07:24:06.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.006+0000 7f189259c640 1 --2- 172.21.15.94:0/1063759530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f189414f360 0x7f1894155c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f18840097d0 tx=0x7f188402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc4e9ba1e87c8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.006+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1884037070 con 0x7f189414f360 2024-03-06T07:24:06.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.006+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1884030510 con 0x7f189414f360 2024-03-06T07:24:06.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.006+0000 7f189159a640 1 -- 172.21.15.94:0/1063759530 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c0019f0 con 0x7f189414f360 2024-03-06T07:24:06.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.007+0000 7f1898f5e640 1 -- 172.21.15.94:0/1063759530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f189414f360 msgr2=0x7f1894155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.007+0000 7f1898f5e640 1 --2- 172.21.15.94:0/1063759530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f189414f360 0x7f1894155c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f18840097d0 tx=0x7f188402b770 comp rx=0 tx=0).stop 2024-03-06T07:24:06.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.007+0000 7f1898f5e640 1 -- 172.21.15.94:0/1063759530 shutdown_connections 2024-03-06T07:24:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.007+0000 7f1898f5e640 1 --2- 172.21.15.94:0/1063759530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f189414f360 0x7f1894155c10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.007+0000 7f1898f5e640 1 -- 172.21.15.94:0/1063759530 >> 172.21.15.94:0/1063759530 conn(0x7f18940ac570 msgr2=0x7f18940ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.007+0000 7f1898f5e640 1 -- 172.21.15.94:0/1063759530 shutdown_connections 2024-03-06T07:24:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.007+0000 7f1898f5e640 1 -- 172.21.15.94:0/1063759530 wait complete. 2024-03-06T07:24:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.008+0000 7f1898f5e640 1 Processor -- start 2024-03-06T07:24:06.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.008+0000 7f1898f5e640 1 -- start start 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1898f5e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940c88c0 0x7f18941ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1898f5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f189414f360 0x7f18941ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1898f5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1894158b60 0x7f189414bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1898f5e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940c7c30 con 0x7f189414f360 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1898f5e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18940c5a50 con 0x7f18940c88c0 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1898f5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18940c5bd0 con 0x7f1894158b60 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f189414f360 0x7f18941ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f189414f360 0x7f18941ee990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36944/0 (socket says 172.21.15.94:36944) 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 -- 172.21.15.94:0/245779210 learned_addr learned my addr 172.21.15.94:0/245779210 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:06.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1892d9d640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1894158b60 0x7f189414bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f189259c640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940c88c0 0x7f18941ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 -- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1894158b60 msgr2=0x7f189414bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1894158b60 0x7f189414bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 -- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940c88c0 msgr2=0x7f18941ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940c88c0 0x7f18941ee450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.009+0000 7f1891d9b640 1 -- 172.21.15.94:0/245779210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18941eeed0 con 0x7f189414f360 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.010+0000 7f1891d9b640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f189414f360 0x7f18941ee990 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f188800c3f0 tx=0x7f188800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.010+0000 7f1882ffd640 1 -- 172.21.15.94:0/245779210 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888017070 con 0x7f189414f360 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.010+0000 7f1882ffd640 1 -- 172.21.15.94:0/245779210 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18880052f0 con 0x7f189414f360 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.010+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18940afa80 con 0x7f189414f360 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.010+0000 7f1882ffd640 1 -- 172.21.15.94:0/245779210 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888003830 con 0x7f189414f360 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.010+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189414f900 con 0x7f189414f360 2024-03-06T07:24:06.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.011+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189404f330 con 0x7f189414f360 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.011+0000 7f1882ffd640 1 -- 172.21.15.94:0/245779210 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f188801a070 con 0x7f189414f360 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.012+0000 7f1882ffd640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.012+0000 7f1882ffd640 1 -- 172.21.15.94:0/245779210 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f18880896b0 con 0x7f189414f360 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.012+0000 7f189259c640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.013+0000 7f189259c640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f187406b500 0x7f187406d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1884034000 tx=0x7f1884008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.016+0000 7f1882ffd640 1 -- 172.21.15.94:0/245779210 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18880584f0 con 0x7f189414f360 2024-03-06T07:24:06.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.306+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18940c8d50 con 0x7f189414f360 2024-03-06T07:24:06.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.307+0000 7f1882ffd640 1 -- 172.21.15.94:0/245779210 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4990 (secure 0 0 0) 0x7f1888057df0 con 0x7f189414f360 2024-03-06T07:24:06.309 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:06.309 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":135,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8155,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:05.235078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/1613596341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1613596341},{"type":"v1","addr":"172.21.15.94:6835","nonce":1613596341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":11}]} 2024-03-06T07:24:06.309 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 135 2024-03-06T07:24:06.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f187406b500 msgr2=0x7f187406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f187406b500 0x7f187406d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1884034000 tx=0x7f1884008030 comp rx=0 tx=0).stop 2024-03-06T07:24:06.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f189414f360 msgr2=0x7f18941ee990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f189414f360 0x7f18941ee990 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f188800c3f0 tx=0x7f188800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 shutdown_connections 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f187406b500 0x7f187406d9c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1894158b60 0x7f189414bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f189414f360 0x7f18941ee990 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 --2- 172.21.15.94:0/245779210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940c88c0 0x7f18941ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 >> 172.21.15.94:0/245779210 conn(0x7f18940ac570 msgr2=0x7f18941537a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 shutdown_connections 2024-03-06T07:24:06.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.309+0000 7f1898f5e640 1 -- 172.21.15.94:0/245779210 wait complete. 2024-03-06T07:24:06.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:06.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.425+0000 7fbaaeb5c640 1 Processor -- start 2024-03-06T07:24:06.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.425+0000 7fbaaeb5c640 1 -- start start 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaaeb5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbaa814f360 0x7fbaa8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaaeb5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaaeb5c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa80c73d0 con 0x7fbaa80c5880 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaaeb5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa80c8dd0 con 0x7fbaa814f360 2024-03-06T07:24:06.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaaeb5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa80be2e0 con 0x7fbaa80c88c0 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaa7fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaac8f8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbaa814f360 0x7fbaa8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaad0f9640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fbaa80c5880 0x7fbaa8158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52978/0 (socket says 172.21.15.94:52978) 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.426+0000 7fbaad0f9640 1 -- 172.21.15.94:0/2740496421 learned_addr learned my addr 172.21.15.94:0/2740496421 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.427+0000 7fbaac8f8640 1 -- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 msgr2=0x7fbaa80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.427+0000 7fbaac8f8640 1 --2- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.427+0000 7fbaac8f8640 1 -- 172.21.15.94:0/2740496421 >> v1:172.21.15.145:6789/0 conn(0x7fbaa80c5880 legacy=0x7fbaa8158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.427+0000 7fbaac8f8640 1 -- 172.21.15.94:0/2740496421 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa80c7670 con 0x7fbaa814f360 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.427+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2740496421 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338087213 0 0) 0x7fbaa80c73d0 con 0x7fbaa80c5880 2024-03-06T07:24:06.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.427+0000 7fbaa7fff640 1 --2- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.427+0000 7fbaac8f8640 1 --2- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa814f360 0x7fbaa8155c10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fba980097a0 tx=0x7fba9802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=589873ea1f63e2c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2740496421 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba980315a0 con 0x7fbaa814f360 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2740496421 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba98037070 con 0x7fbaa814f360 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaa77fe640 1 -- 172.21.15.94:0/2740496421 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98031960 con 0x7fbaa814f360 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa814f360 msgr2=0x7fbaa8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa814f360 0x7fbaa8155c10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fba980097a0 tx=0x7fba9802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/2740496421 shutdown_connections 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/2740496421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa814f360 0x7fbaa8155c10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/2740496421 >> 172.21.15.94:0/2740496421 conn(0x7fbaa80ac570 msgr2=0x7fbaa80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/2740496421 shutdown_connections 2024-03-06T07:24:06.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.428+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/2740496421 wait complete. 2024-03-06T07:24:06.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.429+0000 7fbaaeb5c640 1 Processor -- start 2024-03-06T07:24:06.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.429+0000 7fbaaeb5c640 1 -- start start 2024-03-06T07:24:06.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaaeb5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa80c5880 0x7fbaa8157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaaeb5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaaeb5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa814f360 0x7fbaa80c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaaeb5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa8159270 con 0x7fbaa814f360 2024-03-06T07:24:06.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaaeb5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa80c7c30 con 0x7fbaa80c5880 2024-03-06T07:24:06.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaaeb5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81593f0 con 0x7fbaa80c88c0 2024-03-06T07:24:06.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaad0f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa814f360 0x7fbaa80c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaad0f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa814f360 0x7fbaa80c2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36960/0 (socket says 172.21.15.94:36960) 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaad0f9640 1 -- 172.21.15.94:0/4074258498 learned_addr learned my addr 172.21.15.94:0/4074258498 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaa7fff640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.430+0000 7fbaac8f8640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa80c5880 0x7fbaa8157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaad0f9640 1 -- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 msgr2=0x7fbaa80c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaad0f9640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaad0f9640 1 -- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa80c5880 msgr2=0x7fbaa8157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaad0f9640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa80c5880 0x7fbaa8157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaad0f9640 1 -- 172.21.15.94:0/4074258498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa81eba90 con 0x7fbaa814f360 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaad0f9640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa814f360 0x7fbaa80c2da0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000c7a0 tx=0x7fbaa000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaa57fa640 1 -- 172.21.15.94:0/4074258498 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0018070 con 0x7fbaa814f360 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaa57fa640 1 -- 172.21.15.94:0/4074258498 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa000a5e0 con 0x7fbaa814f360 2024-03-06T07:24:06.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa81eacd0 con 0x7fbaa814f360 2024-03-06T07:24:06.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaa57fa640 1 -- 172.21.15.94:0/4074258498 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa000aa50 con 0x7fbaa814f360 2024-03-06T07:24:06.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.431+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa81eb290 con 0x7fbaa814f360 2024-03-06T07:24:06.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.432+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa80c8590 con 0x7fbaa814f360 2024-03-06T07:24:06.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.432+0000 7fbaa57fa640 1 -- 172.21.15.94:0/4074258498 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaa001b070 con 0x7fbaa814f360 2024-03-06T07:24:06.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.433+0000 7fbaa57fa640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba8806b4d0 0x7fba8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.433+0000 7fbaa57fa640 1 -- 172.21.15.94:0/4074258498 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbaa0089810 con 0x7fbaa814f360 2024-03-06T07:24:06.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.433+0000 7fbaac8f8640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba8806b4d0 0x7fba8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.434+0000 7fbaac8f8640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba8806b4d0 0x7fba8806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fba98036000 tx=0x7fba9803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.441+0000 7fbaa57fa640 1 -- 172.21.15.94:0/4074258498 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaa80c8590 con 0x7fbaa814f360 2024-03-06T07:24:06.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.622+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaa80c64c0 con 0x7fbaa814f360 2024-03-06T07:24:06.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.623+0000 7fbaa57fa640 1 -- 172.21.15.94:0/4074258498 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16884 (secure 0 0 0) 0x7fbaa80c64c0 con 0x7fbaa814f360 2024-03-06T07:24:06.625 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:06.626 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":149,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:04.871334+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:06.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba8806b4d0 msgr2=0x7fba8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba8806b4d0 0x7fba8806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fba98036000 tx=0x7fba9803d040 comp rx=0 tx=0).stop 2024-03-06T07:24:06.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa814f360 msgr2=0x7fbaa80c2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa814f360 0x7fbaa80c2da0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000c7a0 tx=0x7fbaa000cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 shutdown_connections 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba8806b4d0 0x7fba8806d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa814f360 0x7fbaa80c2da0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa80c88c0 0x7fbaa80c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 --2- 172.21.15.94:0/4074258498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa80c5880 0x7fbaa8157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 >> 172.21.15.94:0/4074258498 conn(0x7fbaa80ac570 msgr2=0x7fbaa80c9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 shutdown_connections 2024-03-06T07:24:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.625+0000 7fbaaeb5c640 1 -- 172.21.15.94:0/4074258498 wait complete. 2024-03-06T07:24:06.639 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:06.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:06.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.737+0000 7f6fea671640 1 Processor -- start 2024-03-06T07:24:06.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.737+0000 7f6fea671640 1 -- start start 2024-03-06T07:24:06.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fea671640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6fe4153990 0x7f6fe4156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fea671640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6fe40acc00 0x7f6fe40c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fea671640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe4070aa0 con 0x7f6fe41506b0 2024-03-06T07:24:06.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fea671640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe4156450 con 0x7f6fe40acc00 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fea671640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe4156a10 con 0x7f6fe4153990 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe8c0e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6fe40acc00 0x7f6fe40c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe8c0e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6fe40acc00 0x7f6fe40c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36962/0 (socket says 172.21.15.94:36962) 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe8c0e640 1 -- 172.21.15.94:0/1454535170 learned_addr learned my addr 172.21.15.94:0/1454535170 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe37fe640 1 --2- 172.21.15.94:0/1454535170 >> v2:172.21.15.145:3300/0 conn(0x7f6fe4153990 0x7f6fe4156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe8c0e640 1 -- 172.21.15.94:0/1454535170 >> v2:172.21.15.145:3300/0 conn(0x7f6fe4153990 msgr2=0x7f6fe4156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe8c0e640 1 --2- 172.21.15.94:0/1454535170 >> v2:172.21.15.145:3300/0 conn(0x7f6fe4153990 0x7f6fe4156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe8c0e640 1 -- 172.21.15.94:0/1454535170 >> v1:172.21.15.145:6789/0 conn(0x7f6fe41506b0 legacy=0x7f6fe40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe8c0e640 1 -- 172.21.15.94:0/1454535170 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4070d80 con 0x7f6fe40acc00 2024-03-06T07:24:06.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.739+0000 7f6fe37fe640 1 --2- 172.21.15.94:0/1454535170 >> v2:172.21.15.145:3300/0 conn(0x7f6fe4153990 0x7f6fe4156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.740+0000 7f6fe8c0e640 1 --2- 172.21.15.94:0/1454535170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40c54a0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4150a90 tx=0x7f6fdc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec14d0ece4f16fce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.740+0000 7f6fe2ffd640 1 -- 172.21.15.94:0/1454535170 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc02cb20 con 0x7f6fe40acc00 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.740+0000 7f6fe2ffd640 1 -- 172.21.15.94:0/1454535170 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc02ccc0 con 0x7f6fe40acc00 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.740+0000 7f6fe2ffd640 1 -- 172.21.15.94:0/1454535170 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc034610 con 0x7f6fe40acc00 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 -- 172.21.15.94:0/1454535170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 msgr2=0x7f6fe40c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 --2- 172.21.15.94:0/1454535170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40c54a0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4150a90 tx=0x7f6fdc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 -- 172.21.15.94:0/1454535170 shutdown_connections 2024-03-06T07:24:06.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 --2- 172.21.15.94:0/1454535170 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40c54a0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 --2- 172.21.15.94:0/1454535170 >> v2:172.21.15.145:3300/0 conn(0x7f6fe4153990 0x7f6fe4156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 -- 172.21.15.94:0/1454535170 >> 172.21.15.94:0/1454535170 conn(0x7f6fe406b1a0 msgr2=0x7f6fe406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 -- 172.21.15.94:0/1454535170 shutdown_connections 2024-03-06T07:24:06.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.741+0000 7f6fea671640 1 -- 172.21.15.94:0/1454535170 wait complete. 2024-03-06T07:24:06.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.742+0000 7f6fea671640 1 Processor -- start 2024-03-06T07:24:06.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.742+0000 7f6fea671640 1 -- start start 2024-03-06T07:24:06.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fea671640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40a8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fea671640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fe41506b0 0x7f6fe4155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fea671640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fe4153990 0x7f6fe41ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fea671640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40c5b60 con 0x7f6fe40acc00 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fea671640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40c59e0 con 0x7f6fe41506b0 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fea671640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40c5ce0 con 0x7f6fe4153990 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40a8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40a8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36970/0 (socket says 172.21.15.94:36970) 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 -- 172.21.15.94:0/2707218357 learned_addr learned my addr 172.21.15.94:0/2707218357 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe8c0e640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fe4153990 0x7f6fe41ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe37fe640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fe41506b0 0x7f6fe4155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 -- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fe4153990 msgr2=0x7f6fe41ec6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fe4153990 0x7f6fe41ec6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 -- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fe41506b0 msgr2=0x7f6fe4155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fe41506b0 0x7f6fe4155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 -- 172.21.15.94:0/2707218357 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40aaa80 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.743+0000 7f6fe3fff640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40a8d40 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c400 tx=0x7f6fd400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.744+0000 7f6fe0ff9640 1 -- 172.21.15.94:0/2707218357 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40052d0 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.744+0000 7f6fe0ff9640 1 -- 172.21.15.94:0/2707218357 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd4005490 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.744+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40aad60 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.744+0000 7f6fe0ff9640 1 -- 172.21.15.94:0/2707218357 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4007260 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.744+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe41ec190 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.745+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe404f330 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.745+0000 7f6fe0ff9640 1 -- 172.21.15.94:0/2707218357 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fd4007400 con 0x7f6fe40acc00 2024-03-06T07:24:06.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.746+0000 7f6fe0ff9640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fc406b4d0 0x7f6fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.746+0000 7f6fe0ff9640 1 -- 172.21.15.94:0/2707218357 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6fd40898e0 con 0x7f6fe40acc00 2024-03-06T07:24:06.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.746+0000 7f6fe37fe640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fc406b4d0 0x7f6fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.746+0000 7f6fe37fe640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fc406b4d0 0x7f6fc406d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6fe414a890 tx=0x7f6fcc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:06.754+0000 7f6fe0ff9640 1 -- 172.21.15.94:0/2707218357 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fd40585d0 con 0x7f6fe40acc00 2024-03-06T07:24:07.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.008+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fe40700a0 con 0x7f6fc406b4d0 2024-03-06T07:24:07.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.014+0000 7f6fe0ff9640 1 -- 172.21.15.94:0/2707218357 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240102 (secure 0 0 0) 0x7f6fe40700a0 con 0x7f6fc406b4d0 2024-03-06T07:24:07.019 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:07.020 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fc406b4d0 msgr2=0x7f6fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fc406b4d0 0x7f6fc406d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6fe414a890 tx=0x7f6fcc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 msgr2=0x7f6fe40a8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40a8d40 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400c400 tx=0x7f6fd400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 shutdown_connections 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fc406b4d0 0x7f6fc406d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fe4153990 0x7f6fe41ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fe41506b0 0x7f6fe4155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 --2- 172.21.15.94:0/2707218357 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fe40acc00 0x7f6fe40a8d40 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 >> 172.21.15.94:0/2707218357 conn(0x7f6fe406b1a0 msgr2=0x7f6fe414aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 shutdown_connections 2024-03-06T07:24:07.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.017+0000 7f6fea671640 1 -- 172.21.15.94:0/2707218357 wait complete. 2024-03-06T07:24:07.030 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-06T07:24:06.710374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":263904,"kb_used_data":14424,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710816,"statfs":{"total":773094113280,"available":772823875584,"internally_reserved":0,"allocated":14770176,"data_stored":8217294,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.844093"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867866+0000","last_change":"2024-03-06T07:24:01.867866+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867866+0000","last_undegraded":"2024-03-06T07:24:01.867866+0000","last_fullsized":"2024-03-06T07:24:01.867866+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870688+0000","last_change":"2024-03-06T07:24:01.870688+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870688+0000","last_undegraded":"2024-03-06T07:24:01.870688+0000","last_fullsized":"2024-03-06T07:24:01.870688+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872086+0000","last_change":"2024-03-06T07:24:01.872086+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872086+0000","last_undegraded":"2024-03-06T07:24:01.872086+0000","last_fullsized":"2024-03-06T07:24:01.872086+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868997+0000","last_change":"2024-03-06T07:24:01.868997+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868997+0000","last_undegraded":"2024-03-06T07:24:01.868997+0000","last_fullsized":"2024-03-06T07:24:01.868997+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870174+0000","last_change":"2024-03-06T07:24:01.870174+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870174+0000","last_undegraded":"2024-03-06T07:24:01.870174+0000","last_fullsized":"2024-03-06T07:24:01.870174+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867878+0000","last_change":"2024-03-06T07:24:01.867878+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867878+0000","last_undegraded":"2024-03-06T07:24:01.867878+0000","last_fullsized":"2024-03-06T07:24:01.867878+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869433+0000","last_change":"2024-03-06T07:24:01.869433+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869433+0000","last_undegraded":"2024-03-06T07:24:01.869433+0000","last_fullsized":"2024-03-06T07:24:01.869433+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870509+0000","last_change":"2024-03-06T07:24:01.870509+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870509+0000","last_undegraded":"2024-03-06T07:24:01.870509+0000","last_fullsized":"2024-03-06T07:24:01.870509+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869660+0000","last_change":"2024-03-06T07:24:01.869660+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869660+0000","last_undegraded":"2024-03-06T07:24:01.869660+0000","last_fullsized":"2024-03-06T07:24:01.869660+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871686+0000","last_change":"2024-03-06T07:24:01.871686+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871686+0000","last_undegraded":"2024-03-06T07:24:01.871686+0000","last_fullsized":"2024-03-06T07:24:01.871686+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870325+0000","last_change":"2024-03-06T07:24:01.870325+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870325+0000","last_undegraded":"2024-03-06T07:24:01.870325+0000","last_fullsized":"2024-03-06T07:24:01.870325+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872643+0000","last_change":"2024-03-06T07:24:01.872643+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872643+0000","last_undegraded":"2024-03-06T07:24:01.872643+0000","last_fullsized":"2024-03-06T07:24:01.872643+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868967+0000","last_change":"2024-03-06T07:24:01.868967+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868967+0000","last_undegraded":"2024-03-06T07:24:01.868967+0000","last_fullsized":"2024-03-06T07:24:01.868967+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870125+0000","last_change":"2024-03-06T07:24:01.870125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870125+0000","last_undegraded":"2024-03-06T07:24:01.870125+0000","last_fullsized":"2024-03-06T07:24:01.870125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869423+0000","last_change":"2024-03-06T07:24:01.869423+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869423+0000","last_undegraded":"2024-03-06T07:24:01.869423+0000","last_fullsized":"2024-03-06T07:24:01.869423+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871014+0000","last_change":"2024-03-06T07:24:01.871014+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871014+0000","last_undegraded":"2024-03-06T07:24:01.871014+0000","last_fullsized":"2024-03-06T07:24:01.871014+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869024+0000","last_change":"2024-03-06T07:24:01.869024+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869024+0000","last_undegraded":"2024-03-06T07:24:01.869024+0000","last_fullsized":"2024-03-06T07:24:01.869024+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871314+0000","last_change":"2024-03-06T07:24:01.871314+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871314+0000","last_undegraded":"2024-03-06T07:24:01.871314+0000","last_fullsized":"2024-03-06T07:24:01.871314+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872125+0000","last_change":"2024-03-06T07:24:01.872125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872125+0000","last_undegraded":"2024-03-06T07:24:01.872125+0000","last_fullsized":"2024-03-06T07:24:01.872125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867188+0000","last_change":"2024-03-06T07:24:01.867188+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867188+0000","last_undegraded":"2024-03-06T07:24:01.867188+0000","last_fullsized":"2024-03-06T07:24:01.867188+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871466+0000","last_change":"2024-03-06T07:24:01.871466+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871466+0000","last_undegraded":"2024-03-06T07:24:01.871466+0000","last_fullsized":"2024-03-06T07:24:01.871466+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871506+0000","last_change":"2024-03-06T07:24:01.871506+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871506+0000","last_undegraded":"2024-03-06T07:24:01.871506+0000","last_fullsized":"2024-03-06T07:24:01.871506+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870911+0000","last_change":"2024-03-06T07:24:01.870911+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870911+0000","last_undegraded":"2024-03-06T07:24:01.870911+0000","last_fullsized":"2024-03-06T07:24:01.870911+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869750+0000","last_change":"2024-03-06T07:24:01.869750+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869750+0000","last_undegraded":"2024-03-06T07:24:01.869750+0000","last_fullsized":"2024-03-06T07:24:01.869750+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867056+0000","last_change":"2024-03-06T07:24:01.867056+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867056+0000","last_undegraded":"2024-03-06T07:24:01.867056+0000","last_fullsized":"2024-03-06T07:24:01.867056+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867868+0000","last_change":"2024-03-06T07:24:01.867868+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867868+0000","last_undegraded":"2024-03-06T07:24:01.867868+0000","last_fullsized":"2024-03-06T07:24:01.867868+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870777+0000","last_change":"2024-03-06T07:24:01.870777+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870777+0000","last_undegraded":"2024-03-06T07:24:01.870777+0000","last_fullsized":"2024-03-06T07:24:01.870777+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872023+0000","last_change":"2024-03-06T07:24:01.872023+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872023+0000","last_undegraded":"2024-03-06T07:24:01.872023+0000","last_fullsized":"2024-03-06T07:24:01.872023+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869619+0000","last_change":"2024-03-06T07:24:01.869619+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869619+0000","last_undegraded":"2024-03-06T07:24:01.869619+0000","last_fullsized":"2024-03-06T07:24:01.869619+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871251+0000","last_change":"2024-03-06T07:24:01.871251+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871251+0000","last_undegraded":"2024-03-06T07:24:01.871251+0000","last_fullsized":"2024-03-06T07:24:01.871251+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":408,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.873009+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:01.873009+0000","last_peered":"2024-03-06T07:24:01.873009+0000","last_clean":"2024-03-06T07:24:01.873009+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:01.873009+0000","last_undegraded":"2024-03-06T07:24:01.873009+0000","last_fullsized":"2024-03-06T07:24:01.873009+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574922,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1867776,"data_stored":1456888,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1474463,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:07.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:07.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.168+0000 7f68783f7640 1 Processor -- start 2024-03-06T07:24:07.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.168+0000 7f68783f7640 1 -- start start 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f68783f7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f687014f360 0x7f6870155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f68783f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68700c88c0 0x7f68700c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f68783f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68700c5880 0x7f6870158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f68783f7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700c8dd0 con 0x7f68700c5880 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f68783f7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700c73d0 con 0x7f687014f360 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f68783f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700be2e0 con 0x7f68700c88c0 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f6876994640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68700c5880 0x7f6870158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f6876994640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68700c5880 0x7f6870158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36974/0 (socket says 172.21.15.94:36974) 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.169+0000 7f6876994640 1 -- 172.21.15.94:0/1691346079 learned_addr learned my addr 172.21.15.94:0/1691346079 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.170+0000 7f6876994640 1 -- 172.21.15.94:0/1691346079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68700c88c0 msgr2=0x7f68700c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.170+0000 7f6876994640 1 --2- 172.21.15.94:0/1691346079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68700c88c0 0x7f68700c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.170+0000 7f6876994640 1 -- 172.21.15.94:0/1691346079 >> v2:172.21.15.145:3300/0 conn(0x7f687014f360 msgr2=0x7f6870155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.170+0000 7f6876994640 1 --2- 172.21.15.94:0/1691346079 >> v2:172.21.15.145:3300/0 conn(0x7f687014f360 0x7f6870155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.170+0000 7f6876994640 1 -- 172.21.15.94:0/1691346079 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700c7670 con 0x7f68700c5880 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.170+0000 7f6876994640 1 --2- 172.21.15.94:0/1691346079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f6870158b60 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f687008c840 tx=0x7f686802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5ca6cbf2234abf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f6875191640 1 -- 172.21.15.94:0/1691346079 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686802c9e0 con 0x7f68700c5880 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f6875191640 1 -- 172.21.15.94:0/1691346079 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686802cb80 con 0x7f68700c5880 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f6875191640 1 -- 172.21.15.94:0/1691346079 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68680343f0 con 0x7f68700c5880 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 -- 172.21.15.94:0/1691346079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 msgr2=0x7f6870158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 --2- 172.21.15.94:0/1691346079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f6870158b60 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f687008c840 tx=0x7f686802be40 comp rx=0 tx=0).stop 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 -- 172.21.15.94:0/1691346079 shutdown_connections 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 --2- 172.21.15.94:0/1691346079 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f6870158b60 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 --2- 172.21.15.94:0/1691346079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68700c88c0 0x7f68700c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 --2- 172.21.15.94:0/1691346079 >> v2:172.21.15.145:3300/0 conn(0x7f687014f360 0x7f6870155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 -- 172.21.15.94:0/1691346079 >> 172.21.15.94:0/1691346079 conn(0x7f68700ac570 msgr2=0x7f68700ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 -- 172.21.15.94:0/1691346079 shutdown_connections 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.171+0000 7f68783f7640 1 -- 172.21.15.94:0/1691346079 wait complete. 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.172+0000 7f68783f7640 1 Processor -- start 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.172+0000 7f68783f7640 1 -- start start 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f68783f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f68701eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f68783f7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68700c88c0 0x7f68700abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f68783f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f687014f360 0x7f6870157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f68783f7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701590a0 con 0x7f68700c5880 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f68783f7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700c7c30 con 0x7f68700c88c0 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f68783f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6870159220 con 0x7f687014f360 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f68701eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f68701eb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36976/0 (socket says 172.21.15.94:36976) 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 -- 172.21.15.94:0/3552888388 learned_addr learned my addr 172.21.15.94:0/3552888388 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 -- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f687014f360 msgr2=0x7f6870157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f687014f360 0x7f6870157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 -- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68700c88c0 msgr2=0x7f68700abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6875992640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68700c88c0 0x7f68700abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68700c88c0 0x7f68700abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6876193640 1 -- 172.21.15.94:0/3552888388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68701eade0 con 0x7f68700c5880 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.173+0000 7f6875992640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68700c88c0 0x7f68700abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.174+0000 7f6876193640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f68701eb8c0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f686c00b440 tx=0x7f686c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.174+0000 7f6866ffd640 1 -- 172.21.15.94:0/3552888388 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c015020 con 0x7f68700c5880 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.174+0000 7f6866ffd640 1 -- 172.21.15.94:0/3552888388 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c011890 con 0x7f68700c5880 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.174+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68701eb0c0 con 0x7f68700c5880 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.174+0000 7f6866ffd640 1 -- 172.21.15.94:0/3552888388 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c011c80 con 0x7f68700c5880 2024-03-06T07:24:07.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.174+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68700ac180 con 0x7f68700c5880 2024-03-06T07:24:07.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.175+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687004f330 con 0x7f68700c5880 2024-03-06T07:24:07.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.175+0000 7f6866ffd640 1 -- 172.21.15.94:0/3552888388 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f686c018020 con 0x7f68700c5880 2024-03-06T07:24:07.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.176+0000 7f6866ffd640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f685406b4d0 0x7f685406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.176+0000 7f6866ffd640 1 -- 172.21.15.94:0/3552888388 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f686c08a4a0 con 0x7f68700c5880 2024-03-06T07:24:07.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.176+0000 7f6875992640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f685406b4d0 0x7f685406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.176+0000 7f6875992640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f685406b4d0 0x7f685406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6860006fd0 tx=0x7f6860008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.180+0000 7f6866ffd640 1 -- 172.21.15.94:0/3552888388 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f686c01f020 con 0x7f68700c5880 2024-03-06T07:24:07.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.354+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68701ee240 con 0x7f685406b4d0 2024-03-06T07:24:07.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.360+0000 7f6866ffd640 1 -- 172.21.15.94:0/3552888388 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240102 (secure 0 0 0) 0x7f68701ee240 con 0x7f685406b4d0 2024-03-06T07:24:07.364 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:07.365 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:24:07.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f685406b4d0 msgr2=0x7f685406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f685406b4d0 0x7f685406d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6860006fd0 tx=0x7f6860008040 comp rx=0 tx=0).stop 2024-03-06T07:24:07.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 msgr2=0x7f68701eb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f68701eb8c0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f686c00b440 tx=0x7f686c00b900 comp rx=0 tx=0).stop 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 shutdown_connections 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f685406b4d0 0x7f685406d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f687014f360 0x7f6870157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68700c88c0 0x7f68700abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 --2- 172.21.15.94:0/3552888388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68700c5880 0x7f68701eb8c0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 >> 172.21.15.94:0/3552888388 conn(0x7f68700ac570 msgr2=0x7f687014e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 shutdown_connections 2024-03-06T07:24:07.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.362+0000 7f68783f7640 1 -- 172.21.15.94:0/3552888388 wait complete. 2024-03-06T07:24:07.375 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-06T07:24:06.710374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":263904,"kb_used_data":14424,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710816,"statfs":{"total":773094113280,"available":772823875584,"internally_reserved":0,"allocated":14770176,"data_stored":8217294,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.844093"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867866+0000","last_change":"2024-03-06T07:24:01.867866+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867866+0000","last_undegraded":"2024-03-06T07:24:01.867866+0000","last_fullsized":"2024-03-06T07:24:01.867866+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870688+0000","last_change":"2024-03-06T07:24:01.870688+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870688+0000","last_undegraded":"2024-03-06T07:24:01.870688+0000","last_fullsized":"2024-03-06T07:24:01.870688+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872086+0000","last_change":"2024-03-06T07:24:01.872086+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872086+0000","last_undegraded":"2024-03-06T07:24:01.872086+0000","last_fullsized":"2024-03-06T07:24:01.872086+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868997+0000","last_change":"2024-03-06T07:24:01.868997+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868997+0000","last_undegraded":"2024-03-06T07:24:01.868997+0000","last_fullsized":"2024-03-06T07:24:01.868997+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870174+0000","last_change":"2024-03-06T07:24:01.870174+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870174+0000","last_undegraded":"2024-03-06T07:24:01.870174+0000","last_fullsized":"2024-03-06T07:24:01.870174+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867878+0000","last_change":"2024-03-06T07:24:01.867878+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867878+0000","last_undegraded":"2024-03-06T07:24:01.867878+0000","last_fullsized":"2024-03-06T07:24:01.867878+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869433+0000","last_change":"2024-03-06T07:24:01.869433+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869433+0000","last_undegraded":"2024-03-06T07:24:01.869433+0000","last_fullsized":"2024-03-06T07:24:01.869433+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870509+0000","last_change":"2024-03-06T07:24:01.870509+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870509+0000","last_undegraded":"2024-03-06T07:24:01.870509+0000","last_fullsized":"2024-03-06T07:24:01.870509+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869660+0000","last_change":"2024-03-06T07:24:01.869660+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869660+0000","last_undegraded":"2024-03-06T07:24:01.869660+0000","last_fullsized":"2024-03-06T07:24:01.869660+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871686+0000","last_change":"2024-03-06T07:24:01.871686+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871686+0000","last_undegraded":"2024-03-06T07:24:01.871686+0000","last_fullsized":"2024-03-06T07:24:01.871686+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870325+0000","last_change":"2024-03-06T07:24:01.870325+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870325+0000","last_undegraded":"2024-03-06T07:24:01.870325+0000","last_fullsized":"2024-03-06T07:24:01.870325+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872643+0000","last_change":"2024-03-06T07:24:01.872643+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872643+0000","last_undegraded":"2024-03-06T07:24:01.872643+0000","last_fullsized":"2024-03-06T07:24:01.872643+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868967+0000","last_change":"2024-03-06T07:24:01.868967+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868967+0000","last_undegraded":"2024-03-06T07:24:01.868967+0000","last_fullsized":"2024-03-06T07:24:01.868967+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870125+0000","last_change":"2024-03-06T07:24:01.870125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870125+0000","last_undegraded":"2024-03-06T07:24:01.870125+0000","last_fullsized":"2024-03-06T07:24:01.870125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869423+0000","last_change":"2024-03-06T07:24:01.869423+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869423+0000","last_undegraded":"2024-03-06T07:24:01.869423+0000","last_fullsized":"2024-03-06T07:24:01.869423+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871014+0000","last_change":"2024-03-06T07:24:01.871014+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871014+0000","last_undegraded":"2024-03-06T07:24:01.871014+0000","last_fullsized":"2024-03-06T07:24:01.871014+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869024+0000","last_change":"2024-03-06T07:24:01.869024+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869024+0000","last_undegraded":"2024-03-06T07:24:01.869024+0000","last_fullsized":"2024-03-06T07:24:01.869024+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871314+0000","last_change":"2024-03-06T07:24:01.871314+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871314+0000","last_undegraded":"2024-03-06T07:24:01.871314+0000","last_fullsized":"2024-03-06T07:24:01.871314+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872125+0000","last_change":"2024-03-06T07:24:01.872125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872125+0000","last_undegraded":"2024-03-06T07:24:01.872125+0000","last_fullsized":"2024-03-06T07:24:01.872125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867188+0000","last_change":"2024-03-06T07:24:01.867188+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867188+0000","last_undegraded":"2024-03-06T07:24:01.867188+0000","last_fullsized":"2024-03-06T07:24:01.867188+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871466+0000","last_change":"2024-03-06T07:24:01.871466+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871466+0000","last_undegraded":"2024-03-06T07:24:01.871466+0000","last_fullsized":"2024-03-06T07:24:01.871466+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871506+0000","last_change":"2024-03-06T07:24:01.871506+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871506+0000","last_undegraded":"2024-03-06T07:24:01.871506+0000","last_fullsized":"2024-03-06T07:24:01.871506+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870911+0000","last_change":"2024-03-06T07:24:01.870911+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870911+0000","last_undegraded":"2024-03-06T07:24:01.870911+0000","last_fullsized":"2024-03-06T07:24:01.870911+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869750+0000","last_change":"2024-03-06T07:24:01.869750+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869750+0000","last_undegraded":"2024-03-06T07:24:01.869750+0000","last_fullsized":"2024-03-06T07:24:01.869750+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867056+0000","last_change":"2024-03-06T07:24:01.867056+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867056+0000","last_undegraded":"2024-03-06T07:24:01.867056+0000","last_fullsized":"2024-03-06T07:24:01.867056+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867868+0000","last_change":"2024-03-06T07:24:01.867868+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867868+0000","last_undegraded":"2024-03-06T07:24:01.867868+0000","last_fullsized":"2024-03-06T07:24:01.867868+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870777+0000","last_change":"2024-03-06T07:24:01.870777+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870777+0000","last_undegraded":"2024-03-06T07:24:01.870777+0000","last_fullsized":"2024-03-06T07:24:01.870777+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872023+0000","last_change":"2024-03-06T07:24:01.872023+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872023+0000","last_undegraded":"2024-03-06T07:24:01.872023+0000","last_fullsized":"2024-03-06T07:24:01.872023+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869619+0000","last_change":"2024-03-06T07:24:01.869619+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869619+0000","last_undegraded":"2024-03-06T07:24:01.869619+0000","last_fullsized":"2024-03-06T07:24:01.869619+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871251+0000","last_change":"2024-03-06T07:24:01.871251+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871251+0000","last_undegraded":"2024-03-06T07:24:01.871251+0000","last_fullsized":"2024-03-06T07:24:01.871251+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":408,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.873009+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:01.873009+0000","last_peered":"2024-03-06T07:24:01.873009+0000","last_clean":"2024-03-06T07:24:01.873009+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:01.873009+0000","last_undegraded":"2024-03-06T07:24:01.873009+0000","last_fullsized":"2024-03-06T07:24:01.873009+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574922,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1867776,"data_stored":1456888,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1474463,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:07.379 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:07.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.516+0000 7fee2f839640 1 Processor -- start 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.516+0000 7fee2f839640 1 -- start start 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2f839640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 0x7fee2806ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2f839640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fee280720a0 0x7fee2806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2f839640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee280c1ae0 con 0x7fee28155c80 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2f839640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee280c1dc0 con 0x7fee280720a0 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2f839640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2806e450 con 0x7fee28152520 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2cdd4640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fee28155c80 0x7fee280c12b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37892/0 (socket says 172.21.15.94:37892) 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2cdd4640 1 -- 172.21.15.94:0/4272836521 learned_addr learned my addr 172.21.15.94:0/4272836521 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658989058 0 0) 0x7fee280c1ae0 con 0x7fee28155c80 2024-03-06T07:24:07.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee14003520 con 0x7fee28155c80 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 0x7fee2806ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee2ddd6640 1 --2- 172.21.15.94:0/4272836521 >> v2:172.21.15.145:3300/0 conn(0x7fee280720a0 0x7fee2806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4261959128 0 0) 0x7fee14003520 con 0x7fee28155c80 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 msgr2=0x7fee2806ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 0x7fee2806ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 >> v2:172.21.15.145:3300/0 conn(0x7fee280720a0 msgr2=0x7fee2806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 --2- 172.21.15.94:0/4272836521 >> v2:172.21.15.145:3300/0 conn(0x7fee280720a0 0x7fee2806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2806e730 con 0x7fee28155c80 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fee18001670 con 0x7fee28155c80 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 >> v1:172.21.15.94:6789/0 conn(0x7fee28155c80 legacy=0x7fee280c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee140039b0 0x7fee14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee14004800 0x7fee14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 0x7fee140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee280c1dc0 con 0x7fee140039b0 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2806e450 con 0x7fee14004800 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.517+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1400b9c0 con 0x7fee28152520 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee140039b0 0x7fee14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2ddd6640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 0x7fee140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2cdd4640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee14004800 0x7fee14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2d5d5640 1 -- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 msgr2=0x7fee140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee28152520 0x7fee140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2d5d5640 1 -- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee14004800 msgr2=0x7fee14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee14004800 0x7fee14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2d5d5640 1 -- 172.21.15.94:0/4272836521 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee14003520 con 0x7fee140039b0 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.518+0000 7fee2cdd4640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee14004800 0x7fee14004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.519+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee140039b0 0x7fee14003dd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fee24009ac0 tx=0x7fee14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2265261b33bd7b62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.519+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24039070 con 0x7fee140039b0 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.519+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2402c6b0 con 0x7fee140039b0 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.519+0000 7fee1ffff640 1 -- 172.21.15.94:0/4272836521 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24009690 con 0x7fee140039b0 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.519+0000 7fee2f839640 1 -- 172.21.15.94:0/4272836521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee140039b0 msgr2=0x7fee14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.519+0000 7fee2f839640 1 --2- 172.21.15.94:0/4272836521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee140039b0 0x7fee14003dd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fee24009ac0 tx=0x7fee14005830 comp rx=0 tx=0).stop 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.519+0000 7fee2ddd6640 1 -- 172.21.15.94:0/4272836521 reap_dead start 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.520+0000 7fee2f839640 1 -- 172.21.15.94:0/4272836521 shutdown_connections 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.520+0000 7fee2f839640 1 -- 172.21.15.94:0/4272836521 >> 172.21.15.94:0/4272836521 conn(0x7fee280c76b0 msgr2=0x7fee280c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.520+0000 7fee2f839640 1 -- 172.21.15.94:0/4272836521 shutdown_connections 2024-03-06T07:24:07.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.520+0000 7fee2f839640 1 -- 172.21.15.94:0/4272836521 wait complete. 2024-03-06T07:24:07.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.521+0000 7fee2f839640 1 Processor -- start 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.521+0000 7fee2f839640 1 -- start start 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.522+0000 7fee2f839640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee140039b0 0x7fee2814ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.522+0000 7fee2f839640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee280ac5b0 0x7fee280aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.522+0000 7fee2f839640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee281eac10 0x7fee281eb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.522+0000 7fee2f839640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee28152a80 con 0x7fee280ac5b0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.522+0000 7fee2f839640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee28152900 con 0x7fee140039b0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.522+0000 7fee2f839640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2806b500 con 0x7fee281eac10 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee280ac5b0 0x7fee280aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee280ac5b0 0x7fee280aca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36984/0 (socket says 172.21.15.94:36984) 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 -- 172.21.15.94:0/3967846707 learned_addr learned my addr 172.21.15.94:0/3967846707 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee140039b0 0x7fee2814ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2ddd6640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee281eac10 0x7fee281eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 -- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee281eac10 msgr2=0x7fee281eb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee281eac10 0x7fee281eb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 -- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee140039b0 msgr2=0x7fee2814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee140039b0 0x7fee2814ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 -- 172.21.15.94:0/3967846707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee281ee2e0 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2cdd4640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee280ac5b0 0x7fee280aca10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e7f0 tx=0x7fee1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee1dffb640 1 -- 172.21.15.94:0/3967846707 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1800c8d0 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee1dffb640 1 -- 172.21.15.94:0/3967846707 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee18018070 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee281ee5c0 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee1dffb640 1 -- 172.21.15.94:0/3967846707 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1800cc70 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.523+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee281eeb80 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.524+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee281eed80 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.525+0000 7fee1dffb640 1 -- 172.21.15.94:0/3967846707 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee18013420 con 0x7fee280ac5b0 2024-03-06T07:24:07.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.525+0000 7fee1dffb640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee0c06b500 0x7fee0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.525+0000 7fee1dffb640 1 -- 172.21.15.94:0/3967846707 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fee18089d40 con 0x7fee280ac5b0 2024-03-06T07:24:07.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.526+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee0c06b500 0x7fee0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.526+0000 7fee2d5d5640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee0c06b500 0x7fee0c06d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fee24036000 tx=0x7fee24030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.530+0000 7fee1dffb640 1 -- 172.21.15.94:0/3967846707 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee281eed80 con 0x7fee280ac5b0 2024-03-06T07:24:07.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.704+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee2804f330 con 0x7fee0c06b500 2024-03-06T07:24:07.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.711+0000 7fee1dffb640 1 -- 172.21.15.94:0/3967846707 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240102 (secure 0 0 0) 0x7fee2804f330 con 0x7fee0c06b500 2024-03-06T07:24:07.715 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:07.716 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.713+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee0c06b500 msgr2=0x7fee0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.713+0000 7fee2f839640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee0c06b500 0x7fee0c06d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fee24036000 tx=0x7fee24030000 comp rx=0 tx=0).stop 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.713+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee280ac5b0 msgr2=0x7fee280aca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.713+0000 7fee2f839640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee280ac5b0 0x7fee280aca10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e7f0 tx=0x7fee1800ecb0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.713+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 shutdown_connections 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.714+0000 7fee2f839640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fee0c06b500 0x7fee0c06d9c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.714+0000 7fee2f839640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee281eac10 0x7fee281eb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.714+0000 7fee2f839640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fee280ac5b0 0x7fee280aca10 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.714+0000 7fee2f839640 1 --2- 172.21.15.94:0/3967846707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee140039b0 0x7fee2814ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.714+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 >> 172.21.15.94:0/3967846707 conn(0x7fee280c76b0 msgr2=0x7fee28152e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.714+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 shutdown_connections 2024-03-06T07:24:07.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.714+0000 7fee2f839640 1 -- 172.21.15.94:0/3967846707 wait complete. 2024-03-06T07:24:07.726 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-06T07:24:06.710374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":263904,"kb_used_data":14424,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710816,"statfs":{"total":773094113280,"available":772823875584,"internally_reserved":0,"allocated":14770176,"data_stored":8217294,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.844093"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867866+0000","last_change":"2024-03-06T07:24:01.867866+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867866+0000","last_undegraded":"2024-03-06T07:24:01.867866+0000","last_fullsized":"2024-03-06T07:24:01.867866+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870688+0000","last_change":"2024-03-06T07:24:01.870688+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870688+0000","last_undegraded":"2024-03-06T07:24:01.870688+0000","last_fullsized":"2024-03-06T07:24:01.870688+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872086+0000","last_change":"2024-03-06T07:24:01.872086+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872086+0000","last_undegraded":"2024-03-06T07:24:01.872086+0000","last_fullsized":"2024-03-06T07:24:01.872086+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868997+0000","last_change":"2024-03-06T07:24:01.868997+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868997+0000","last_undegraded":"2024-03-06T07:24:01.868997+0000","last_fullsized":"2024-03-06T07:24:01.868997+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870174+0000","last_change":"2024-03-06T07:24:01.870174+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870174+0000","last_undegraded":"2024-03-06T07:24:01.870174+0000","last_fullsized":"2024-03-06T07:24:01.870174+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867878+0000","last_change":"2024-03-06T07:24:01.867878+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867878+0000","last_undegraded":"2024-03-06T07:24:01.867878+0000","last_fullsized":"2024-03-06T07:24:01.867878+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869433+0000","last_change":"2024-03-06T07:24:01.869433+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869433+0000","last_undegraded":"2024-03-06T07:24:01.869433+0000","last_fullsized":"2024-03-06T07:24:01.869433+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870509+0000","last_change":"2024-03-06T07:24:01.870509+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870509+0000","last_undegraded":"2024-03-06T07:24:01.870509+0000","last_fullsized":"2024-03-06T07:24:01.870509+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869660+0000","last_change":"2024-03-06T07:24:01.869660+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869660+0000","last_undegraded":"2024-03-06T07:24:01.869660+0000","last_fullsized":"2024-03-06T07:24:01.869660+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871686+0000","last_change":"2024-03-06T07:24:01.871686+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871686+0000","last_undegraded":"2024-03-06T07:24:01.871686+0000","last_fullsized":"2024-03-06T07:24:01.871686+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870325+0000","last_change":"2024-03-06T07:24:01.870325+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870325+0000","last_undegraded":"2024-03-06T07:24:01.870325+0000","last_fullsized":"2024-03-06T07:24:01.870325+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872643+0000","last_change":"2024-03-06T07:24:01.872643+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872643+0000","last_undegraded":"2024-03-06T07:24:01.872643+0000","last_fullsized":"2024-03-06T07:24:01.872643+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868967+0000","last_change":"2024-03-06T07:24:01.868967+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868967+0000","last_undegraded":"2024-03-06T07:24:01.868967+0000","last_fullsized":"2024-03-06T07:24:01.868967+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870125+0000","last_change":"2024-03-06T07:24:01.870125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870125+0000","last_undegraded":"2024-03-06T07:24:01.870125+0000","last_fullsized":"2024-03-06T07:24:01.870125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869423+0000","last_change":"2024-03-06T07:24:01.869423+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869423+0000","last_undegraded":"2024-03-06T07:24:01.869423+0000","last_fullsized":"2024-03-06T07:24:01.869423+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871014+0000","last_change":"2024-03-06T07:24:01.871014+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871014+0000","last_undegraded":"2024-03-06T07:24:01.871014+0000","last_fullsized":"2024-03-06T07:24:01.871014+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869024+0000","last_change":"2024-03-06T07:24:01.869024+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869024+0000","last_undegraded":"2024-03-06T07:24:01.869024+0000","last_fullsized":"2024-03-06T07:24:01.869024+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871314+0000","last_change":"2024-03-06T07:24:01.871314+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871314+0000","last_undegraded":"2024-03-06T07:24:01.871314+0000","last_fullsized":"2024-03-06T07:24:01.871314+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872125+0000","last_change":"2024-03-06T07:24:01.872125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872125+0000","last_undegraded":"2024-03-06T07:24:01.872125+0000","last_fullsized":"2024-03-06T07:24:01.872125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867188+0000","last_change":"2024-03-06T07:24:01.867188+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867188+0000","last_undegraded":"2024-03-06T07:24:01.867188+0000","last_fullsized":"2024-03-06T07:24:01.867188+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871466+0000","last_change":"2024-03-06T07:24:01.871466+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871466+0000","last_undegraded":"2024-03-06T07:24:01.871466+0000","last_fullsized":"2024-03-06T07:24:01.871466+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871506+0000","last_change":"2024-03-06T07:24:01.871506+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871506+0000","last_undegraded":"2024-03-06T07:24:01.871506+0000","last_fullsized":"2024-03-06T07:24:01.871506+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870911+0000","last_change":"2024-03-06T07:24:01.870911+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870911+0000","last_undegraded":"2024-03-06T07:24:01.870911+0000","last_fullsized":"2024-03-06T07:24:01.870911+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869750+0000","last_change":"2024-03-06T07:24:01.869750+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869750+0000","last_undegraded":"2024-03-06T07:24:01.869750+0000","last_fullsized":"2024-03-06T07:24:01.869750+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867056+0000","last_change":"2024-03-06T07:24:01.867056+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867056+0000","last_undegraded":"2024-03-06T07:24:01.867056+0000","last_fullsized":"2024-03-06T07:24:01.867056+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867868+0000","last_change":"2024-03-06T07:24:01.867868+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867868+0000","last_undegraded":"2024-03-06T07:24:01.867868+0000","last_fullsized":"2024-03-06T07:24:01.867868+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870777+0000","last_change":"2024-03-06T07:24:01.870777+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870777+0000","last_undegraded":"2024-03-06T07:24:01.870777+0000","last_fullsized":"2024-03-06T07:24:01.870777+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872023+0000","last_change":"2024-03-06T07:24:01.872023+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872023+0000","last_undegraded":"2024-03-06T07:24:01.872023+0000","last_fullsized":"2024-03-06T07:24:01.872023+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869619+0000","last_change":"2024-03-06T07:24:01.869619+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869619+0000","last_undegraded":"2024-03-06T07:24:01.869619+0000","last_fullsized":"2024-03-06T07:24:01.869619+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871251+0000","last_change":"2024-03-06T07:24:01.871251+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871251+0000","last_undegraded":"2024-03-06T07:24:01.871251+0000","last_fullsized":"2024-03-06T07:24:01.871251+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":408,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.873009+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:01.873009+0000","last_peered":"2024-03-06T07:24:01.873009+0000","last_clean":"2024-03-06T07:24:01.873009+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:01.873009+0000","last_undegraded":"2024-03-06T07:24:01.873009+0000","last_fullsized":"2024-03-06T07:24:01.873009+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574922,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1867776,"data_stored":1456888,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1474463,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:07.730 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3f is not active+clean 2024-03-06T07:24:07.730 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3e is not active+clean 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3d is not active+clean 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3c is not active+clean 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3b is not active+clean 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3a is not active+clean 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.731 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.39 is not active+clean 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.38 is not active+clean 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.37 is not active+clean 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.36 is not active+clean 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.35 is not active+clean 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.34 is not active+clean 2024-03-06T07:24:07.732 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.33 is not active+clean 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.32 is not active+clean 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.31 is not active+clean 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.30 is not active+clean 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2f is not active+clean 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2e is not active+clean 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2d is not active+clean 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2c is not active+clean 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2b is not active+clean 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2a is not active+clean 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.29 is not active+clean 2024-03-06T07:24:07.734 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.28 is not active+clean 2024-03-06T07:24:07.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.27 is not active+clean 2024-03-06T07:24:07.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.26 is not active+clean 2024-03-06T07:24:07.735 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.735 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.25 is not active+clean 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.24 is not active+clean 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.23 is not active+clean 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.22 is not active+clean 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.21 is not active+clean 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.20 is not active+clean 2024-03-06T07:24:07.736 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.10 is not active+clean 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1f is not active+clean 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.867866+0000', 'last_change': '2024-03-06T07:24:01.867866+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.867866+0000', 'last_undegraded': '2024-03-06T07:24:01.867866+0000', 'last_fullsized': '2024-03-06T07:24:01.867866+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.11 is not active+clean 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1e is not active+clean 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.870688+0000', 'last_change': '2024-03-06T07:24:01.870688+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.870688+0000', 'last_undegraded': '2024-03-06T07:24:01.870688+0000', 'last_fullsized': '2024-03-06T07:24:01.870688+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.12 is not active+clean 2024-03-06T07:24:07.737 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1d is not active+clean 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.872086+0000', 'last_change': '2024-03-06T07:24:01.872086+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.872086+0000', 'last_undegraded': '2024-03-06T07:24:01.872086+0000', 'last_fullsized': '2024-03-06T07:24:01.872086+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.13 is not active+clean 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1c is not active+clean 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.867857+0000', 'last_change': '2024-03-06T07:24:01.867857+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.867857+0000', 'last_undegraded': '2024-03-06T07:24:01.867857+0000', 'last_fullsized': '2024-03-06T07:24:01.867857+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.14 is not active+clean 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1b is not active+clean 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.868997+0000', 'last_change': '2024-03-06T07:24:01.868997+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.868997+0000', 'last_undegraded': '2024-03-06T07:24:01.868997+0000', 'last_fullsized': '2024-03-06T07:24:01.868997+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:07.738 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.15 is not active+clean 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1a is not active+clean 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.870174+0000', 'last_change': '2024-03-06T07:24:01.870174+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.870174+0000', 'last_undegraded': '2024-03-06T07:24:01.870174+0000', 'last_fullsized': '2024-03-06T07:24:01.870174+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.16 is not active+clean 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.19 is not active+clean 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.867878+0000', 'last_change': '2024-03-06T07:24:01.867878+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.867878+0000', 'last_undegraded': '2024-03-06T07:24:01.867878+0000', 'last_fullsized': '2024-03-06T07:24:01.867878+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.17 is not active+clean 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.18 is not active+clean 2024-03-06T07:24:07.739 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.869433+0000', 'last_change': '2024-03-06T07:24:01.869433+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.869433+0000', 'last_undegraded': '2024-03-06T07:24:01.869433+0000', 'last_fullsized': '2024-03-06T07:24:01.869433+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.18 is not active+clean 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.17 is not active+clean 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.870509+0000', 'last_change': '2024-03-06T07:24:01.870509+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.870509+0000', 'last_undegraded': '2024-03-06T07:24:01.870509+0000', 'last_fullsized': '2024-03-06T07:24:01.870509+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1a is not active+clean 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.15 is not active+clean 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.867857+0000', 'last_change': '2024-03-06T07:24:01.867857+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.867857+0000', 'last_undegraded': '2024-03-06T07:24:01.867857+0000', 'last_fullsized': '2024-03-06T07:24:01.867857+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1b is not active+clean 2024-03-06T07:24:07.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.14 is not active+clean 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.869660+0000', 'last_change': '2024-03-06T07:24:01.869660+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.869660+0000', 'last_undegraded': '2024-03-06T07:24:01.869660+0000', 'last_fullsized': '2024-03-06T07:24:01.869660+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1c is not active+clean 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.13 is not active+clean 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.871686+0000', 'last_change': '2024-03-06T07:24:01.871686+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.871686+0000', 'last_undegraded': '2024-03-06T07:24:01.871686+0000', 'last_fullsized': '2024-03-06T07:24:01.871686+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1d is not active+clean 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.12 is not active+clean 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.870325+0000', 'last_change': '2024-03-06T07:24:01.870325+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.870325+0000', 'last_undegraded': '2024-03-06T07:24:01.870325+0000', 'last_fullsized': '2024-03-06T07:24:01.870325+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1e is not active+clean 2024-03-06T07:24:07.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.11 is not active+clean 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.872643+0000', 'last_change': '2024-03-06T07:24:01.872643+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.872643+0000', 'last_undegraded': '2024-03-06T07:24:01.872643+0000', 'last_fullsized': '2024-03-06T07:24:01.872643+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1f is not active+clean 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.10 is not active+clean 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.868967+0000', 'last_change': '2024-03-06T07:24:01.868967+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.868967+0000', 'last_undegraded': '2024-03-06T07:24:01.868967+0000', 'last_fullsized': '2024-03-06T07:24:01.868967+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.0 is not active+clean 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.f is not active+clean 2024-03-06T07:24:07.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.870125+0000', 'last_change': '2024-03-06T07:24:01.870125+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.870125+0000', 'last_undegraded': '2024-03-06T07:24:01.870125+0000', 'last_fullsized': '2024-03-06T07:24:01.870125+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.1 is not active+clean 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.e is not active+clean 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.869423+0000', 'last_change': '2024-03-06T07:24:01.869423+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.869423+0000', 'last_undegraded': '2024-03-06T07:24:01.869423+0000', 'last_fullsized': '2024-03-06T07:24:01.869423+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.2 is not active+clean 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.d is not active+clean 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.871014+0000', 'last_change': '2024-03-06T07:24:01.871014+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.871014+0000', 'last_undegraded': '2024-03-06T07:24:01.871014+0000', 'last_fullsized': '2024-03-06T07:24:01.871014+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.3 is not active+clean 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.c is not active+clean 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.869024+0000', 'last_change': '2024-03-06T07:24:01.869024+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.869024+0000', 'last_undegraded': '2024-03-06T07:24:01.869024+0000', 'last_fullsized': '2024-03-06T07:24:01.869024+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.4 is not active+clean 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.b is not active+clean 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.871314+0000', 'last_change': '2024-03-06T07:24:01.871314+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.871314+0000', 'last_undegraded': '2024-03-06T07:24:01.871314+0000', 'last_fullsized': '2024-03-06T07:24:01.871314+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.5 is not active+clean 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.a is not active+clean 2024-03-06T07:24:07.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.872125+0000', 'last_change': '2024-03-06T07:24:01.872125+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.872125+0000', 'last_undegraded': '2024-03-06T07:24:01.872125+0000', 'last_fullsized': '2024-03-06T07:24:01.872125+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.6 is not active+clean 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.9 is not active+clean 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.867188+0000', 'last_change': '2024-03-06T07:24:01.867188+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.867188+0000', 'last_undegraded': '2024-03-06T07:24:01.867188+0000', 'last_fullsized': '2024-03-06T07:24:01.867188+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.7 is not active+clean 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.8 is not active+clean 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.871466+0000', 'last_change': '2024-03-06T07:24:01.871466+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.871466+0000', 'last_undegraded': '2024-03-06T07:24:01.871466+0000', 'last_fullsized': '2024-03-06T07:24:01.871466+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.8 is not active+clean 2024-03-06T07:24:07.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.7 is not active+clean 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.871506+0000', 'last_change': '2024-03-06T07:24:01.871506+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.871506+0000', 'last_undegraded': '2024-03-06T07:24:01.871506+0000', 'last_fullsized': '2024-03-06T07:24:01.871506+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.9 is not active+clean 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.6 is not active+clean 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.870911+0000', 'last_change': '2024-03-06T07:24:01.870911+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.870911+0000', 'last_undegraded': '2024-03-06T07:24:01.870911+0000', 'last_fullsized': '2024-03-06T07:24:01.870911+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.a is not active+clean 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.5 is not active+clean 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.869750+0000', 'last_change': '2024-03-06T07:24:01.869750+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.869750+0000', 'last_undegraded': '2024-03-06T07:24:01.869750+0000', 'last_fullsized': '2024-03-06T07:24:01.869750+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.b is not active+clean 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.4 is not active+clean 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.867056+0000', 'last_change': '2024-03-06T07:24:01.867056+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.867056+0000', 'last_undegraded': '2024-03-06T07:24:01.867056+0000', 'last_fullsized': '2024-03-06T07:24:01.867056+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.c is not active+clean 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.3 is not active+clean 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.867868+0000', 'last_change': '2024-03-06T07:24:01.867868+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.867868+0000', 'last_undegraded': '2024-03-06T07:24:01.867868+0000', 'last_fullsized': '2024-03-06T07:24:01.867868+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.d is not active+clean 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.2 is not active+clean 2024-03-06T07:24:07.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.870777+0000', 'last_change': '2024-03-06T07:24:01.870777+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.870777+0000', 'last_undegraded': '2024-03-06T07:24:01.870777+0000', 'last_fullsized': '2024-03-06T07:24:01.870777+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.e is not active+clean 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.1 is not active+clean 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.872023+0000', 'last_change': '2024-03-06T07:24:01.872023+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.872023+0000', 'last_undegraded': '2024-03-06T07:24:01.872023+0000', 'last_fullsized': '2024-03-06T07:24:01.872023+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.f is not active+clean 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.0 is not active+clean 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.869619+0000', 'last_change': '2024-03-06T07:24:01.869619+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.869619+0000', 'last_undegraded': '2024-03-06T07:24:01.869619+0000', 'last_fullsized': '2024-03-06T07:24:01.869619+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 24.19 is not active+clean 2024-03-06T07:24:07.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '24.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:24:02.864943+0000', 'last_change': '2024-03-06T07:24:02.864943+0000', 'last_active': '2024-03-06T07:24:02.864943+0000', 'last_peered': '2024-03-06T07:24:02.864943+0000', 'last_clean': '2024-03-06T07:24:02.864943+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:02.864943+0000', 'last_undegraded': '2024-03-06T07:24:02.864943+0000', 'last_fullsized': '2024-03-06T07:24:02.864943+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:02.864943+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:24:07.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 23.16 is not active+clean 2024-03-06T07:24:07.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '23.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 146, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:01.871251+0000', 'last_change': '2024-03-06T07:24:01.871251+0000', 'last_active': '2024-03-06T07:24:01.859528+0000', 'last_peered': '2024-03-06T07:24:01.859528+0000', 'last_clean': '2024-03-06T07:24:01.859528+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:01.871251+0000', 'last_undegraded': '2024-03-06T07:24:01.871251+0000', 'last_fullsized': '2024-03-06T07:24:01.871251+0000', 'mapping_epoch': 146, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 146, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:01.859528+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:07.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:24:07.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.868+0000 7fb802356640 1 Processor -- start 2024-03-06T07:24:07.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.868+0000 7fb802356640 1 -- start start 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.868+0000 7fb802356640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb7fc0c88c0 0x7fb7fc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.868+0000 7fb802356640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0c73d0 con 0x7fb7fc14f360 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb802356640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0be2e0 con 0x7fb7fc158b60 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb802356640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7fc159700 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7fb7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb7fc0c88c0 0x7fb7fc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7fbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb7fc14f360 0x7fb7fc0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37908/0 (socket says 172.21.15.94:37908) 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7fb7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb7fc0c88c0 0x7fb7fc155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37000/0 (socket says 172.21.15.94:37000) 2024-03-06T07:24:07.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7fbfff640 1 -- 172.21.15.94:0/1474980105 learned_addr learned my addr 172.21.15.94:0/1474980105 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:07.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7fb7fe640 1 -- 172.21.15.94:0/1474980105 >> v1:172.21.15.145:6789/0 conn(0x7fb7fc158b60 legacy=0x7fb7fc158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7fb7fe640 1 -- 172.21.15.94:0/1474980105 >> v1:172.21.15.94:6789/0 conn(0x7fb7fc14f360 legacy=0x7fb7fc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7fb7fe640 1 -- 172.21.15.94:0/1474980105 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc0c7670 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.869+0000 7fb7faffd640 1 -- 172.21.15.94:0/1474980105 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068347178 0 0) 0x7fb7fc0c73d0 con 0x7fb7fc14f360 2024-03-06T07:24:07.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb7fb7fe640 1 --2- 172.21.15.94:0/1474980105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc155c10 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00097a0 tx=0x7fb7f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a5ad9e87c00c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb7faffd640 1 -- 172.21.15.94:0/1474980105 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0035020 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb7faffd640 1 -- 172.21.15.94:0/1474980105 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f0031830 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb7faffd640 1 -- 172.21.15.94:0/1474980105 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0031c00 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb802356640 1 -- 172.21.15.94:0/1474980105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 msgr2=0x7fb7fc155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb802356640 1 --2- 172.21.15.94:0/1474980105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc155c10 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00097a0 tx=0x7fb7f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb802356640 1 -- 172.21.15.94:0/1474980105 shutdown_connections 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb802356640 1 --2- 172.21.15.94:0/1474980105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc155c10 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb802356640 1 -- 172.21.15.94:0/1474980105 >> 172.21.15.94:0/1474980105 conn(0x7fb7fc0ac570 msgr2=0x7fb7fc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb802356640 1 -- 172.21.15.94:0/1474980105 shutdown_connections 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.870+0000 7fb802356640 1 -- 172.21.15.94:0/1474980105 wait complete. 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.871+0000 7fb802356640 1 Processor -- start 2024-03-06T07:24:07.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb802356640 1 -- start start 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb802356640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc0c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb802356640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7fc14f360 0x7fb7fc14bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb802356640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7fc158b60 0x7fb7fc0c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb802356640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0c7c30 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb802356640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0c5a50 con 0x7fb7fc14f360 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb802356640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0c5bd0 con 0x7fb7fc158b60 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb7fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc0c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb7fbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc0c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37010/0 (socket says 172.21.15.94:37010) 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.872+0000 7fb7fbfff640 1 -- 172.21.15.94:0/3105769606 learned_addr learned my addr 172.21.15.94:0/3105769606 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7fb7fe640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7fc14f360 0x7fb7fc14bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb8008f3640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7fc158b60 0x7fb7fc0c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7fbfff640 1 -- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7fc158b60 msgr2=0x7fb7fc0c0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7fbfff640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7fc158b60 0x7fb7fc0c0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7fbfff640 1 -- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7fc14f360 msgr2=0x7fb7fc14bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7fbfff640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7fc14f360 0x7fb7fc14bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7fbfff640 1 -- 172.21.15.94:0/3105769606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc0be590 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7fbfff640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc0c4a00 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c4d0 tx=0x7fb7ec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7f8ff9640 1 -- 172.21.15.94:0/3105769606 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec018070 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7f8ff9640 1 -- 172.21.15.94:0/3105769606 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0074e0 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc1eb750 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb7f8ff9640 1 -- 172.21.15.94:0/3105769606 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec004a20 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.873+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc1eba30 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.874+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc04f330 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.875+0000 7fb7f8ff9640 1 -- 172.21.15.94:0/3105769606 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7ec01b070 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.875+0000 7fb7f8ff9640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7e806b500 0x7fb7e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.875+0000 7fb7f8ff9640 1 -- 172.21.15.94:0/3105769606 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7ec0894d0 con 0x7fb7fc0c88c0 2024-03-06T07:24:07.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.875+0000 7fb7fb7fe640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7e806b500 0x7fb7e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.876+0000 7fb7fb7fe640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7e806b500 0x7fb7e806d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0036000 tx=0x7fb7f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:07.879+0000 7fb7f8ff9640 1 -- 172.21.15.94:0/3105769606 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7ec057c00 con 0x7fb7fc0c88c0 2024-03-06T07:24:08.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.126+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb7fc0c8d50 con 0x7fb7fc0c88c0 2024-03-06T07:24:08.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.127+0000 7fb7f8ff9640 1 -- 172.21.15.94:0/3105769606 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1533 (secure 0 0 0) 0x7fb7ec05bca0 con 0x7fb7fc0c88c0 2024-03-06T07:24:08.129 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:08.129 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":374,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":149,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709482,"num_in_osds":8,"osd_in_since":1709709474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":270237696,"bytes_avail":772823875584,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":135,"id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"c","status":"up:active","gid":8772}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":15,"modified":"2024-03-06T07:23:28.702199+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7e806b500 msgr2=0x7fb7e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7e806b500 0x7fb7e806d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0036000 tx=0x7fb7f0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 msgr2=0x7fb7fc0c4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc0c4a00 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c4d0 tx=0x7fb7ec00c990 comp rx=0 tx=0).stop 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 shutdown_connections 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb7e806b500 0x7fb7e806d9c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7fc158b60 0x7fb7fc0c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7fc14f360 0x7fb7fc14bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 --2- 172.21.15.94:0/3105769606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7fc0c88c0 0x7fb7fc0c4a00 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 >> 172.21.15.94:0/3105769606 conn(0x7fb7fc0ac570 msgr2=0x7fb7fc0ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 shutdown_connections 2024-03-06T07:24:08.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.129+0000 7fb802356640 1 -- 172.21.15.94:0/3105769606 wait complete. 2024-03-06T07:24:08.141 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:24:08.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:08.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.241+0000 7fe323dbc640 1 Processor -- start 2024-03-06T07:24:08.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe323dbc640 1 -- start start 2024-03-06T07:24:08.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe323dbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe323dbc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe31c0c1aa0 0x7fe31c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe323dbc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe31c0c5c60 0x7fe31c155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe323dbc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c0c2680 con 0x7fe31c0c5c60 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe323dbc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c0c28d0 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe323dbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c0c2bb0 con 0x7fe31c14f790 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe322359640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe31c0c5c60 0x7fe31c155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe322359640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe31c0c5c60 0x7fe31c155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37014/0 (socket says 172.21.15.94:37014) 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.242+0000 7fe322359640 1 -- 172.21.15.94:0/4064167684 learned_addr learned my addr 172.21.15.94:0/4064167684 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe321357640 1 --2- 172.21.15.94:0/4064167684 >> v2:172.21.15.145:3300/0 conn(0x7fe31c0c1aa0 0x7fe31c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe321b58640 1 --2- 172.21.15.94:0/4064167684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe322359640 1 -- 172.21.15.94:0/4064167684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 msgr2=0x7fe31c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe322359640 1 --2- 172.21.15.94:0/4064167684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe322359640 1 -- 172.21.15.94:0/4064167684 >> v2:172.21.15.145:3300/0 conn(0x7fe31c0c1aa0 msgr2=0x7fe31c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe322359640 1 --2- 172.21.15.94:0/4064167684 >> v2:172.21.15.145:3300/0 conn(0x7fe31c0c1aa0 0x7fe31c0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe322359640 1 -- 172.21.15.94:0/4064167684 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0c2e90 con 0x7fe31c0c5c60 2024-03-06T07:24:08.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe321357640 1 --2- 172.21.15.94:0/4064167684 >> v2:172.21.15.145:3300/0 conn(0x7fe31c0c1aa0 0x7fe31c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.243+0000 7fe322359640 1 --2- 172.21.15.94:0/4064167684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c5c60 0x7fe31c155a40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0c76f0 tx=0x7fe30c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1b809f88c7894f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe320b56640 1 -- 172.21.15.94:0/4064167684 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c02ca70 con 0x7fe31c0c5c60 2024-03-06T07:24:08.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe320b56640 1 -- 172.21.15.94:0/4064167684 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c02cc10 con 0x7fe31c0c5c60 2024-03-06T07:24:08.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe320b56640 1 -- 172.21.15.94:0/4064167684 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0343f0 con 0x7fe31c0c5c60 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 -- 172.21.15.94:0/4064167684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c5c60 msgr2=0x7fe31c155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4064167684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c5c60 0x7fe31c155a40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0c76f0 tx=0x7fe30c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 -- 172.21.15.94:0/4064167684 shutdown_connections 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4064167684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c5c60 0x7fe31c155a40 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4064167684 >> v2:172.21.15.145:3300/0 conn(0x7fe31c0c1aa0 0x7fe31c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4064167684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 -- 172.21.15.94:0/4064167684 >> 172.21.15.94:0/4064167684 conn(0x7fe31c0c6710 msgr2=0x7fe31c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 -- 172.21.15.94:0/4064167684 shutdown_connections 2024-03-06T07:24:08.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.244+0000 7fe323dbc640 1 -- 172.21.15.94:0/4064167684 wait complete. 2024-03-06T07:24:08.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.246+0000 7fe323dbc640 1 Processor -- start 2024-03-06T07:24:08.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.246+0000 7fe323dbc640 1 -- start start 2024-03-06T07:24:08.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe323dbc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c1aa0 0x7fe31c14e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe323dbc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe31c0c5c60 0x7fe31c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe323dbc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c14c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe323dbc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c06cc00 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe323dbc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c0c0d60 con 0x7fe31c0c5c60 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe323dbc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c06cd80 con 0x7fe31c14f790 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c1aa0 0x7fe31c14e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c1aa0 0x7fe31c14e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37026/0 (socket says 172.21.15.94:37026) 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 -- 172.21.15.94:0/4272165651 learned_addr learned my addr 172.21.15.94:0/4272165651 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321357640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe31c0c5c60 0x7fe31c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe322359640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c14c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 -- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 msgr2=0x7fe31c14c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c14c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 -- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe31c0c5c60 msgr2=0x7fe31c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe31c0c5c60 0x7fe31c070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 -- 172.21.15.94:0/4272165651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0c4bd0 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321357640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe31c0c5c60 0x7fe31c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe321b58640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c1aa0 0x7fe31c14e120 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe31400d620 tx=0x7fe31400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.247+0000 7fe31a7fc640 1 -- 172.21.15.94:0/4272165651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31400b620 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.248+0000 7fe31a7fc640 1 -- 172.21.15.94:0/4272165651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe314019070 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.248+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c1ec160 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.248+0000 7fe31a7fc640 1 -- 172.21.15.94:0/4272165651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31400b9e0 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.248+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c1ec720 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.248+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c04f330 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.249+0000 7fe31a7fc640 1 -- 172.21.15.94:0/4272165651 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe31400bb80 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.249+0000 7fe31a7fc640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe30806b4d0 0x7fe30806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.249+0000 7fe31a7fc640 1 -- 172.21.15.94:0/4272165651 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe31408a700 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.250+0000 7fe321357640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe30806b4d0 0x7fe30806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.250+0000 7fe321357640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe30806b4d0 0x7fe30806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe31c153220 tx=0x7fe310008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.254+0000 7fe31a7fc640 1 -- 172.21.15.94:0/4272165651 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe314059480 con 0x7fe31c0c1aa0 2024-03-06T07:24:08.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.422+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe31c0714c0 con 0x7fe30806b4d0 2024-03-06T07:24:08.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.428+0000 7fe31a7fc640 1 -- 172.21.15.94:0/4272165651 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240102 (secure 0 0 0) 0x7fe310010840 con 0x7fe30806b4d0 2024-03-06T07:24:08.432 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:08.433 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:24:08.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe30806b4d0 msgr2=0x7fe30806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe30806b4d0 0x7fe30806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe31c153220 tx=0x7fe310008040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c1aa0 msgr2=0x7fe31c14e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c1aa0 0x7fe31c14e120 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe31400d620 tx=0x7fe31400dae0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 shutdown_connections 2024-03-06T07:24:08.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe30806b4d0 0x7fe30806d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe31c14f790 0x7fe31c14c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe31c0c5c60 0x7fe31c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 --2- 172.21.15.94:0/4272165651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe31c0c1aa0 0x7fe31c14e120 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 >> 172.21.15.94:0/4272165651 conn(0x7fe31c0c6710 msgr2=0x7fe31c0a9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 shutdown_connections 2024-03-06T07:24:08.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:08.431+0000 7fe323dbc640 1 -- 172.21.15.94:0/4272165651 wait complete. 2024-03-06T07:24:08.444 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-06T07:24:06.710374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":263904,"kb_used_data":14424,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710816,"statfs":{"total":773094113280,"available":772823875584,"internally_reserved":0,"allocated":14770176,"data_stored":8217294,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.844093"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867866+0000","last_change":"2024-03-06T07:24:01.867866+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867866+0000","last_undegraded":"2024-03-06T07:24:01.867866+0000","last_fullsized":"2024-03-06T07:24:01.867866+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870688+0000","last_change":"2024-03-06T07:24:01.870688+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870688+0000","last_undegraded":"2024-03-06T07:24:01.870688+0000","last_fullsized":"2024-03-06T07:24:01.870688+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872086+0000","last_change":"2024-03-06T07:24:01.872086+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872086+0000","last_undegraded":"2024-03-06T07:24:01.872086+0000","last_fullsized":"2024-03-06T07:24:01.872086+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868997+0000","last_change":"2024-03-06T07:24:01.868997+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868997+0000","last_undegraded":"2024-03-06T07:24:01.868997+0000","last_fullsized":"2024-03-06T07:24:01.868997+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870174+0000","last_change":"2024-03-06T07:24:01.870174+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870174+0000","last_undegraded":"2024-03-06T07:24:01.870174+0000","last_fullsized":"2024-03-06T07:24:01.870174+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867878+0000","last_change":"2024-03-06T07:24:01.867878+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867878+0000","last_undegraded":"2024-03-06T07:24:01.867878+0000","last_fullsized":"2024-03-06T07:24:01.867878+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869433+0000","last_change":"2024-03-06T07:24:01.869433+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869433+0000","last_undegraded":"2024-03-06T07:24:01.869433+0000","last_fullsized":"2024-03-06T07:24:01.869433+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870509+0000","last_change":"2024-03-06T07:24:01.870509+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870509+0000","last_undegraded":"2024-03-06T07:24:01.870509+0000","last_fullsized":"2024-03-06T07:24:01.870509+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867857+0000","last_change":"2024-03-06T07:24:01.867857+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867857+0000","last_undegraded":"2024-03-06T07:24:01.867857+0000","last_fullsized":"2024-03-06T07:24:01.867857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869660+0000","last_change":"2024-03-06T07:24:01.869660+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869660+0000","last_undegraded":"2024-03-06T07:24:01.869660+0000","last_fullsized":"2024-03-06T07:24:01.869660+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871686+0000","last_change":"2024-03-06T07:24:01.871686+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871686+0000","last_undegraded":"2024-03-06T07:24:01.871686+0000","last_fullsized":"2024-03-06T07:24:01.871686+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870325+0000","last_change":"2024-03-06T07:24:01.870325+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870325+0000","last_undegraded":"2024-03-06T07:24:01.870325+0000","last_fullsized":"2024-03-06T07:24:01.870325+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872643+0000","last_change":"2024-03-06T07:24:01.872643+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872643+0000","last_undegraded":"2024-03-06T07:24:01.872643+0000","last_fullsized":"2024-03-06T07:24:01.872643+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.868967+0000","last_change":"2024-03-06T07:24:01.868967+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.868967+0000","last_undegraded":"2024-03-06T07:24:01.868967+0000","last_fullsized":"2024-03-06T07:24:01.868967+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870125+0000","last_change":"2024-03-06T07:24:01.870125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870125+0000","last_undegraded":"2024-03-06T07:24:01.870125+0000","last_fullsized":"2024-03-06T07:24:01.870125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869423+0000","last_change":"2024-03-06T07:24:01.869423+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869423+0000","last_undegraded":"2024-03-06T07:24:01.869423+0000","last_fullsized":"2024-03-06T07:24:01.869423+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871014+0000","last_change":"2024-03-06T07:24:01.871014+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871014+0000","last_undegraded":"2024-03-06T07:24:01.871014+0000","last_fullsized":"2024-03-06T07:24:01.871014+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869024+0000","last_change":"2024-03-06T07:24:01.869024+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869024+0000","last_undegraded":"2024-03-06T07:24:01.869024+0000","last_fullsized":"2024-03-06T07:24:01.869024+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871314+0000","last_change":"2024-03-06T07:24:01.871314+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871314+0000","last_undegraded":"2024-03-06T07:24:01.871314+0000","last_fullsized":"2024-03-06T07:24:01.871314+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872125+0000","last_change":"2024-03-06T07:24:01.872125+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872125+0000","last_undegraded":"2024-03-06T07:24:01.872125+0000","last_fullsized":"2024-03-06T07:24:01.872125+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867188+0000","last_change":"2024-03-06T07:24:01.867188+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867188+0000","last_undegraded":"2024-03-06T07:24:01.867188+0000","last_fullsized":"2024-03-06T07:24:01.867188+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871466+0000","last_change":"2024-03-06T07:24:01.871466+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871466+0000","last_undegraded":"2024-03-06T07:24:01.871466+0000","last_fullsized":"2024-03-06T07:24:01.871466+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871506+0000","last_change":"2024-03-06T07:24:01.871506+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871506+0000","last_undegraded":"2024-03-06T07:24:01.871506+0000","last_fullsized":"2024-03-06T07:24:01.871506+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870911+0000","last_change":"2024-03-06T07:24:01.870911+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870911+0000","last_undegraded":"2024-03-06T07:24:01.870911+0000","last_fullsized":"2024-03-06T07:24:01.870911+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869750+0000","last_change":"2024-03-06T07:24:01.869750+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869750+0000","last_undegraded":"2024-03-06T07:24:01.869750+0000","last_fullsized":"2024-03-06T07:24:01.869750+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867056+0000","last_change":"2024-03-06T07:24:01.867056+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867056+0000","last_undegraded":"2024-03-06T07:24:01.867056+0000","last_fullsized":"2024-03-06T07:24:01.867056+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.867868+0000","last_change":"2024-03-06T07:24:01.867868+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.867868+0000","last_undegraded":"2024-03-06T07:24:01.867868+0000","last_fullsized":"2024-03-06T07:24:01.867868+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.870777+0000","last_change":"2024-03-06T07:24:01.870777+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.870777+0000","last_undegraded":"2024-03-06T07:24:01.870777+0000","last_fullsized":"2024-03-06T07:24:01.870777+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.872023+0000","last_change":"2024-03-06T07:24:01.872023+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.872023+0000","last_undegraded":"2024-03-06T07:24:01.872023+0000","last_fullsized":"2024-03-06T07:24:01.872023+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.869619+0000","last_change":"2024-03-06T07:24:01.869619+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.869619+0000","last_undegraded":"2024-03-06T07:24:01.869619+0000","last_fullsized":"2024-03-06T07:24:01.869619+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:24:02.864943+0000","last_change":"2024-03-06T07:24:02.864943+0000","last_active":"2024-03-06T07:24:02.864943+0000","last_peered":"2024-03-06T07:24:02.864943+0000","last_clean":"2024-03-06T07:24:02.864943+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:02.864943+0000","last_undegraded":"2024-03-06T07:24:02.864943+0000","last_fullsized":"2024-03-06T07:24:02.864943+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":1,"reported_epoch":146,"state":"creating+peering","last_fresh":"2024-03-06T07:24:01.871251+0000","last_change":"2024-03-06T07:24:01.871251+0000","last_active":"2024-03-06T07:24:01.859528+0000","last_peered":"2024-03-06T07:24:01.859528+0000","last_clean":"2024-03-06T07:24:01.859528+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:01.871251+0000","last_undegraded":"2024-03-06T07:24:01.871251+0000","last_fullsized":"2024-03-06T07:24:01.871251+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":408,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.873009+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:01.873009+0000","last_peered":"2024-03-06T07:24:01.873009+0000","last_clean":"2024-03-06T07:24:01.873009+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:01.873009+0000","last_undegraded":"2024-03-06T07:24:01.873009+0000","last_fullsized":"2024-03-06T07:24:01.873009+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574922,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1867776,"data_stored":1456888,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":877886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1839104,"data_stored":884095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574922,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":1474463,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:09.153 INFO:tasks.ceph.mds.c.smithi094.stderr:2024-03-06T07:24:09.151+0000 7f1f5ad48640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:11.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.549+0000 7f70c9a37640 1 Processor -- start 2024-03-06T07:24:11.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.549+0000 7f70c9a37640 1 -- start start 2024-03-06T07:24:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c9a37640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f70c41506b0 0x7f70c4153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c9a37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c40c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c9a37640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c4070aa0 con 0x7f70c4156ea0 2024-03-06T07:24:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c9a37640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c4156450 con 0x7f70c41506b0 2024-03-06T07:24:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c9a37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4156a10 con 0x7f70c40acc00 2024-03-06T07:24:11.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c2ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f70c41506b0 0x7f70c4153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70bbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f70c4156ea0 0x7f70c40702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47654/0 (socket says 172.21.15.94:47654) 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70bbfff640 1 -- 172.21.15.94:0/3423828736 learned_addr learned my addr 172.21.15.94:0/3423828736 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c37fe640 1 --2- 172.21.15.94:0/3423828736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c40c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c2ffd640 1 -- 172.21.15.94:0/3423828736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 msgr2=0x7f70c40c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c2ffd640 1 --2- 172.21.15.94:0/3423828736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c40c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c2ffd640 1 -- 172.21.15.94:0/3423828736 >> v1:172.21.15.94:6789/0 conn(0x7f70c4156ea0 legacy=0x7f70c40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c2ffd640 1 -- 172.21.15.94:0/3423828736 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4070d80 con 0x7f70c41506b0 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c27fc640 1 -- 172.21.15.94:0/3423828736 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625572797 0 0) 0x7f70c4070aa0 con 0x7f70c4156ea0 2024-03-06T07:24:11.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.551+0000 7f70c37fe640 1 --2- 172.21.15.94:0/3423828736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c40c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.552+0000 7f70c2ffd640 1 --2- 172.21.15.94:0/3423828736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c41506b0 0x7f70c4153990 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f70b40097a0 tx=0x7f70b402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf03b0aa1a7b21e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.552+0000 7f70c27fc640 1 -- 172.21.15.94:0/3423828736 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4035020 con 0x7f70c41506b0 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.552+0000 7f70c27fc640 1 -- 172.21.15.94:0/3423828736 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b4031810 con 0x7f70c41506b0 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.552+0000 7f70c27fc640 1 -- 172.21.15.94:0/3423828736 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4031be0 con 0x7f70c41506b0 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 -- 172.21.15.94:0/3423828736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c41506b0 msgr2=0x7f70c4153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 --2- 172.21.15.94:0/3423828736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c41506b0 0x7f70c4153990 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f70b40097a0 tx=0x7f70b402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 -- 172.21.15.94:0/3423828736 shutdown_connections 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 --2- 172.21.15.94:0/3423828736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c40c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 --2- 172.21.15.94:0/3423828736 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c41506b0 0x7f70c4153990 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 -- 172.21.15.94:0/3423828736 >> 172.21.15.94:0/3423828736 conn(0x7f70c406b1a0 msgr2=0x7f70c406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 -- 172.21.15.94:0/3423828736 shutdown_connections 2024-03-06T07:24:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.553+0000 7f70c9a37640 1 -- 172.21.15.94:0/3423828736 wait complete. 2024-03-06T07:24:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.554+0000 7f70c9a37640 1 Processor -- start 2024-03-06T07:24:11.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.554+0000 7f70c9a37640 1 -- start start 2024-03-06T07:24:11.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c4151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70c41506b0 0x7f70c41ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4156ea0 0x7f70c41ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40ad1d0 con 0x7f70c4156ea0 2024-03-06T07:24:11.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40ad050 con 0x7f70c41506b0 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40ad350 con 0x7f70c40acc00 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4156ea0 0x7f70c41ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4156ea0 0x7f70c41ee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43074/0 (socket says 172.21.15.94:43074) 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 -- 172.21.15.94:0/394003277 learned_addr learned my addr 172.21.15.94:0/394003277 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c2ffd640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c4151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 -- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 msgr2=0x7f70c4151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70bbfff640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70c41506b0 0x7f70c41ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c4151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 -- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70c41506b0 msgr2=0x7f70c41ee0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70c41506b0 0x7f70c41ee0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 -- 172.21.15.94:0/394003277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c406e050 con 0x7f70c4156ea0 2024-03-06T07:24:11.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c37fe640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4156ea0 0x7f70c41ee620 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00ce40 tx=0x7f70bc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c0ff9640 1 -- 172.21.15.94:0/394003277 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc01a070 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c0ff9640 1 -- 172.21.15.94:0/394003277 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc00d840 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c406c0d0 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c0ff9640 1 -- 172.21.15.94:0/394003277 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc00dc30 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.555+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c406c690 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.556+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c4069460 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.557+0000 7f70c0ff9640 1 -- 172.21.15.94:0/394003277 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70bc01d070 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.557+0000 7f70c0ff9640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70a406b4d0 0x7f70a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.558+0000 7f70c0ff9640 1 -- 172.21.15.94:0/394003277 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f70bc08abc0 con 0x7f70c4156ea0 2024-03-06T07:24:11.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.558+0000 7f70c2ffd640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70a406b4d0 0x7f70a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.558+0000 7f70c2ffd640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70a406b4d0 0x7f70a406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f70b4036000 tx=0x7f70b4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.562+0000 7f70c0ff9640 1 -- 172.21.15.94:0/394003277 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70c4069460 con 0x7f70c4156ea0 2024-03-06T07:24:11.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.745+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70c404f330 con 0x7f70a406b4d0 2024-03-06T07:24:11.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.751+0000 7f70c0ff9640 1 -- 172.21.15.94:0/394003277 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252554 (secure 0 0 0) 0x7f70c404f330 con 0x7f70a406b4d0 2024-03-06T07:24:11.756 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:11.757 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70a406b4d0 msgr2=0x7f70a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70a406b4d0 0x7f70a406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f70b4036000 tx=0x7f70b4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4156ea0 msgr2=0x7f70c41ee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4156ea0 0x7f70c41ee620 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00ce40 tx=0x7f70bc00a710 comp rx=0 tx=0).stop 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 shutdown_connections 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f70a406b4d0 0x7f70a406d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f70c4156ea0 0x7f70c41ee620 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70c41506b0 0x7f70c41ee0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 --2- 172.21.15.94:0/394003277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70c40acc00 0x7f70c4151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 >> 172.21.15.94:0/394003277 conn(0x7f70c406b1a0 msgr2=0x7f70c40c5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 shutdown_connections 2024-03-06T07:24:11.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.754+0000 7f70c9a37640 1 -- 172.21.15.94:0/394003277 wait complete. 2024-03-06T07:24:11.767 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-03-06T07:24:10.711489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264280,"kb_used_data":14836,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710440,"statfs":{"total":773094113280,"available":772823490560,"internally_reserved":0,"allocated":15192064,"data_stored":8415768,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.839943"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550372+0000","last_change":"2024-03-06T07:24:03.878818+0000","last_active":"2024-03-06T07:24:06.550372+0000","last_peered":"2024-03-06T07:24:06.550372+0000","last_clean":"2024-03-06T07:24:06.550372+0000","last_became_active":"2024-03-06T07:24:03.878538+0000","last_became_peered":"2024-03-06T07:24:03.878538+0000","last_unstale":"2024-03-06T07:24:06.550372+0000","last_undegraded":"2024-03-06T07:24:06.550372+0000","last_fullsized":"2024-03-06T07:24:06.550372+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:43.198749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550319+0000","last_change":"2024-03-06T07:24:03.887970+0000","last_active":"2024-03-06T07:24:06.550319+0000","last_peered":"2024-03-06T07:24:06.550319+0000","last_clean":"2024-03-06T07:24:06.550319+0000","last_became_active":"2024-03-06T07:24:03.887681+0000","last_became_peered":"2024-03-06T07:24:03.887681+0000","last_unstale":"2024-03-06T07:24:06.550319+0000","last_undegraded":"2024-03-06T07:24:06.550319+0000","last_fullsized":"2024-03-06T07:24:06.550319+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:32.211048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550046+0000","last_change":"2024-03-06T07:24:03.885308+0000","last_active":"2024-03-06T07:24:06.550046+0000","last_peered":"2024-03-06T07:24:06.550046+0000","last_clean":"2024-03-06T07:24:06.550046+0000","last_became_active":"2024-03-06T07:24:03.885013+0000","last_became_peered":"2024-03-06T07:24:03.885013+0000","last_unstale":"2024-03-06T07:24:06.550046+0000","last_undegraded":"2024-03-06T07:24:06.550046+0000","last_fullsized":"2024-03-06T07:24:06.550046+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:45.339308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550360+0000","last_change":"2024-03-06T07:24:03.885984+0000","last_active":"2024-03-06T07:24:06.550360+0000","last_peered":"2024-03-06T07:24:06.550360+0000","last_clean":"2024-03-06T07:24:06.550360+0000","last_became_active":"2024-03-06T07:24:03.885293+0000","last_became_peered":"2024-03-06T07:24:03.885293+0000","last_unstale":"2024-03-06T07:24:06.550360+0000","last_undegraded":"2024-03-06T07:24:06.550360+0000","last_fullsized":"2024-03-06T07:24:06.550360+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:34.171582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549953+0000","last_change":"2024-03-06T07:24:03.884021+0000","last_active":"2024-03-06T07:24:06.549953+0000","last_peered":"2024-03-06T07:24:06.549953+0000","last_clean":"2024-03-06T07:24:06.549953+0000","last_became_active":"2024-03-06T07:24:03.883424+0000","last_became_peered":"2024-03-06T07:24:03.883424+0000","last_unstale":"2024-03-06T07:24:06.549953+0000","last_undegraded":"2024-03-06T07:24:06.549953+0000","last_fullsized":"2024-03-06T07:24:06.549953+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549779+0000","last_change":"2024-03-06T07:24:03.889706+0000","last_active":"2024-03-06T07:24:06.549779+0000","last_peered":"2024-03-06T07:24:06.549779+0000","last_clean":"2024-03-06T07:24:06.549779+0000","last_became_active":"2024-03-06T07:24:03.889392+0000","last_became_peered":"2024-03-06T07:24:03.889392+0000","last_unstale":"2024-03-06T07:24:06.549779+0000","last_undegraded":"2024-03-06T07:24:06.549779+0000","last_fullsized":"2024-03-06T07:24:06.549779+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877049+0000","last_change":"2024-03-06T07:24:03.889999+0000","last_active":"2024-03-06T07:24:05.877049+0000","last_peered":"2024-03-06T07:24:05.877049+0000","last_clean":"2024-03-06T07:24:05.877049+0000","last_became_active":"2024-03-06T07:24:03.889432+0000","last_became_peered":"2024-03-06T07:24:03.889432+0000","last_unstale":"2024-03-06T07:24:05.877049+0000","last_undegraded":"2024-03-06T07:24:05.877049+0000","last_fullsized":"2024-03-06T07:24:05.877049+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877048+0000","last_change":"2024-03-06T07:24:03.895907+0000","last_active":"2024-03-06T07:24:05.877048+0000","last_peered":"2024-03-06T07:24:05.877048+0000","last_clean":"2024-03-06T07:24:05.877048+0000","last_became_active":"2024-03-06T07:24:03.895663+0000","last_became_peered":"2024-03-06T07:24:03.895663+0000","last_unstale":"2024-03-06T07:24:05.877048+0000","last_undegraded":"2024-03-06T07:24:05.877048+0000","last_fullsized":"2024-03-06T07:24:05.877048+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877466+0000","last_change":"2024-03-06T07:24:03.887528+0000","last_active":"2024-03-06T07:24:05.877466+0000","last_peered":"2024-03-06T07:24:05.877466+0000","last_clean":"2024-03-06T07:24:05.877466+0000","last_became_active":"2024-03-06T07:24:03.887269+0000","last_became_peered":"2024-03-06T07:24:03.887269+0000","last_unstale":"2024-03-06T07:24:05.877466+0000","last_undegraded":"2024-03-06T07:24:05.877466+0000","last_fullsized":"2024-03-06T07:24:05.877466+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:32.211048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877506+0000","last_change":"2024-03-06T07:24:03.897717+0000","last_active":"2024-03-06T07:24:05.877506+0000","last_peered":"2024-03-06T07:24:05.877506+0000","last_clean":"2024-03-06T07:24:05.877506+0000","last_became_active":"2024-03-06T07:24:03.897445+0000","last_became_peered":"2024-03-06T07:24:03.897445+0000","last_unstale":"2024-03-06T07:24:05.877506+0000","last_undegraded":"2024-03-06T07:24:05.877506+0000","last_fullsized":"2024-03-06T07:24:05.877506+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.492678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876177+0000","last_change":"2024-03-06T07:24:03.892692+0000","last_active":"2024-03-06T07:24:04.876177+0000","last_peered":"2024-03-06T07:24:04.876177+0000","last_clean":"2024-03-06T07:24:04.876177+0000","last_became_active":"2024-03-06T07:24:03.892434+0000","last_became_peered":"2024-03-06T07:24:03.892434+0000","last_unstale":"2024-03-06T07:24:04.876177+0000","last_undegraded":"2024-03-06T07:24:04.876177+0000","last_fullsized":"2024-03-06T07:24:04.876177+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:48.146871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550399+0000","last_change":"2024-03-06T07:24:03.887559+0000","last_active":"2024-03-06T07:24:06.550399+0000","last_peered":"2024-03-06T07:24:06.550399+0000","last_clean":"2024-03-06T07:24:06.550399+0000","last_became_active":"2024-03-06T07:24:03.887275+0000","last_became_peered":"2024-03-06T07:24:03.887275+0000","last_unstale":"2024-03-06T07:24:06.550399+0000","last_undegraded":"2024-03-06T07:24:06.550399+0000","last_fullsized":"2024-03-06T07:24:06.550399+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.492678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874513+0000","last_change":"2024-03-06T07:24:03.894465+0000","last_active":"2024-03-06T07:24:04.874513+0000","last_peered":"2024-03-06T07:24:04.874513+0000","last_clean":"2024-03-06T07:24:04.874513+0000","last_became_active":"2024-03-06T07:24:03.894192+0000","last_became_peered":"2024-03-06T07:24:03.894192+0000","last_unstale":"2024-03-06T07:24:04.874513+0000","last_undegraded":"2024-03-06T07:24:04.874513+0000","last_fullsized":"2024-03-06T07:24:04.874513+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:48.848727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549517+0000","last_change":"2024-03-06T07:24:03.890072+0000","last_active":"2024-03-06T07:24:06.549517+0000","last_peered":"2024-03-06T07:24:06.549517+0000","last_clean":"2024-03-06T07:24:06.549517+0000","last_became_active":"2024-03-06T07:24:03.889801+0000","last_became_peered":"2024-03-06T07:24:03.889801+0000","last_unstale":"2024-03-06T07:24:06.549517+0000","last_undegraded":"2024-03-06T07:24:06.549517+0000","last_fullsized":"2024-03-06T07:24:06.549517+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.875848+0000","last_change":"2024-03-06T07:24:03.887600+0000","last_active":"2024-03-06T07:24:04.875848+0000","last_peered":"2024-03-06T07:24:04.875848+0000","last_clean":"2024-03-06T07:24:04.875848+0000","last_became_active":"2024-03-06T07:24:03.887214+0000","last_became_peered":"2024-03-06T07:24:03.887214+0000","last_unstale":"2024-03-06T07:24:04.875848+0000","last_undegraded":"2024-03-06T07:24:04.875848+0000","last_fullsized":"2024-03-06T07:24:04.875848+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:16.405620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549902+0000","last_change":"2024-03-06T07:24:03.884458+0000","last_active":"2024-03-06T07:24:06.549902+0000","last_peered":"2024-03-06T07:24:06.549902+0000","last_clean":"2024-03-06T07:24:06.549902+0000","last_became_active":"2024-03-06T07:24:03.883983+0000","last_became_peered":"2024-03-06T07:24:03.883983+0000","last_unstale":"2024-03-06T07:24:06.549902+0000","last_undegraded":"2024-03-06T07:24:06.549902+0000","last_fullsized":"2024-03-06T07:24:06.549902+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:36.691438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877517+0000","last_change":"2024-03-06T07:24:03.888459+0000","last_active":"2024-03-06T07:24:05.877517+0000","last_peered":"2024-03-06T07:24:05.877517+0000","last_clean":"2024-03-06T07:24:05.877517+0000","last_became_active":"2024-03-06T07:24:03.888203+0000","last_became_peered":"2024-03-06T07:24:03.888203+0000","last_unstale":"2024-03-06T07:24:05.877517+0000","last_undegraded":"2024-03-06T07:24:05.877517+0000","last_fullsized":"2024-03-06T07:24:05.877517+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:35.615657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876126+0000","last_change":"2024-03-06T07:24:03.885816+0000","last_active":"2024-03-06T07:24:04.876126+0000","last_peered":"2024-03-06T07:24:04.876126+0000","last_clean":"2024-03-06T07:24:04.876126+0000","last_became_active":"2024-03-06T07:24:03.885561+0000","last_became_peered":"2024-03-06T07:24:03.885561+0000","last_unstale":"2024-03-06T07:24:04.876126+0000","last_undegraded":"2024-03-06T07:24:04.876126+0000","last_fullsized":"2024-03-06T07:24:04.876126+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:33.243851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876360+0000","last_change":"2024-03-06T07:24:03.887681+0000","last_active":"2024-03-06T07:24:04.876360+0000","last_peered":"2024-03-06T07:24:04.876360+0000","last_clean":"2024-03-06T07:24:04.876360+0000","last_became_active":"2024-03-06T07:24:03.887375+0000","last_became_peered":"2024-03-06T07:24:03.887375+0000","last_unstale":"2024-03-06T07:24:04.876360+0000","last_undegraded":"2024-03-06T07:24:04.876360+0000","last_fullsized":"2024-03-06T07:24:04.876360+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:06.466254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284791+0000","last_change":"2024-03-06T07:24:03.884294+0000","last_active":"2024-03-06T07:24:06.284791+0000","last_peered":"2024-03-06T07:24:06.284791+0000","last_clean":"2024-03-06T07:24:06.284791+0000","last_became_active":"2024-03-06T07:24:03.884050+0000","last_became_peered":"2024-03-06T07:24:03.884050+0000","last_unstale":"2024-03-06T07:24:06.284791+0000","last_undegraded":"2024-03-06T07:24:06.284791+0000","last_fullsized":"2024-03-06T07:24:06.284791+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:01.779181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874610+0000","last_change":"2024-03-06T07:24:03.894163+0000","last_active":"2024-03-06T07:24:04.874610+0000","last_peered":"2024-03-06T07:24:04.874610+0000","last_clean":"2024-03-06T07:24:04.874610+0000","last_became_active":"2024-03-06T07:24:03.893792+0000","last_became_peered":"2024-03-06T07:24:03.893792+0000","last_unstale":"2024-03-06T07:24:04.874610+0000","last_undegraded":"2024-03-06T07:24:04.874610+0000","last_fullsized":"2024-03-06T07:24:04.874610+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:43.594819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549949+0000","last_change":"2024-03-06T07:24:03.892392+0000","last_active":"2024-03-06T07:24:06.549949+0000","last_peered":"2024-03-06T07:24:06.549949+0000","last_clean":"2024-03-06T07:24:06.549949+0000","last_became_active":"2024-03-06T07:24:03.892131+0000","last_became_peered":"2024-03-06T07:24:03.892131+0000","last_unstale":"2024-03-06T07:24:06.549949+0000","last_undegraded":"2024-03-06T07:24:06.549949+0000","last_fullsized":"2024-03-06T07:24:06.549949+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549354+0000","last_change":"2024-03-06T07:24:03.890324+0000","last_active":"2024-03-06T07:24:06.549354+0000","last_peered":"2024-03-06T07:24:06.549354+0000","last_clean":"2024-03-06T07:24:06.549354+0000","last_became_active":"2024-03-06T07:24:03.890037+0000","last_became_peered":"2024-03-06T07:24:03.890037+0000","last_unstale":"2024-03-06T07:24:06.549354+0000","last_undegraded":"2024-03-06T07:24:06.549354+0000","last_fullsized":"2024-03-06T07:24:06.549354+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:57.609556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549742+0000","last_change":"2024-03-06T07:24:03.884245+0000","last_active":"2024-03-06T07:24:06.549742+0000","last_peered":"2024-03-06T07:24:06.549742+0000","last_clean":"2024-03-06T07:24:06.549742+0000","last_became_active":"2024-03-06T07:24:03.883649+0000","last_became_peered":"2024-03-06T07:24:03.883649+0000","last_unstale":"2024-03-06T07:24:06.549742+0000","last_undegraded":"2024-03-06T07:24:06.549742+0000","last_fullsized":"2024-03-06T07:24:06.549742+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:36.283613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874786+0000","last_change":"2024-03-06T07:24:03.898098+0000","last_active":"2024-03-06T07:24:04.874786+0000","last_peered":"2024-03-06T07:24:04.874786+0000","last_clean":"2024-03-06T07:24:04.874786+0000","last_became_active":"2024-03-06T07:24:03.895994+0000","last_became_peered":"2024-03-06T07:24:03.895994+0000","last_unstale":"2024-03-06T07:24:04.874786+0000","last_undegraded":"2024-03-06T07:24:04.874786+0000","last_fullsized":"2024-03-06T07:24:04.874786+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:31.833931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284780+0000","last_change":"2024-03-06T07:24:03.879536+0000","last_active":"2024-03-06T07:24:06.284780+0000","last_peered":"2024-03-06T07:24:06.284780+0000","last_clean":"2024-03-06T07:24:06.284780+0000","last_became_active":"2024-03-06T07:24:03.879280+0000","last_became_peered":"2024-03-06T07:24:03.879280+0000","last_unstale":"2024-03-06T07:24:06.284780+0000","last_undegraded":"2024-03-06T07:24:06.284780+0000","last_fullsized":"2024-03-06T07:24:06.284780+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:09.372273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550214+0000","last_change":"2024-03-06T07:24:03.893792+0000","last_active":"2024-03-06T07:24:06.550214+0000","last_peered":"2024-03-06T07:24:06.550214+0000","last_clean":"2024-03-06T07:24:06.550214+0000","last_became_active":"2024-03-06T07:24:03.893240+0000","last_became_peered":"2024-03-06T07:24:03.893240+0000","last_unstale":"2024-03-06T07:24:06.550214+0000","last_undegraded":"2024-03-06T07:24:06.550214+0000","last_fullsized":"2024-03-06T07:24:06.550214+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:51.104075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874742+0000","last_change":"2024-03-06T07:24:03.894449+0000","last_active":"2024-03-06T07:24:04.874742+0000","last_peered":"2024-03-06T07:24:04.874742+0000","last_clean":"2024-03-06T07:24:04.874742+0000","last_became_active":"2024-03-06T07:24:03.894007+0000","last_became_peered":"2024-03-06T07:24:03.894007+0000","last_unstale":"2024-03-06T07:24:04.874742+0000","last_undegraded":"2024-03-06T07:24:04.874742+0000","last_fullsized":"2024-03-06T07:24:04.874742+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:37.861026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550685+0000","last_change":"2024-03-06T07:24:03.887453+0000","last_active":"2024-03-06T07:24:06.550685+0000","last_peered":"2024-03-06T07:24:06.550685+0000","last_clean":"2024-03-06T07:24:06.550685+0000","last_became_active":"2024-03-06T07:24:03.887167+0000","last_became_peered":"2024-03-06T07:24:03.887167+0000","last_unstale":"2024-03-06T07:24:06.550685+0000","last_undegraded":"2024-03-06T07:24:06.550685+0000","last_fullsized":"2024-03-06T07:24:06.550685+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:01.897485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876577+0000","last_change":"2024-03-06T07:24:03.885011+0000","last_active":"2024-03-06T07:24:04.876577+0000","last_peered":"2024-03-06T07:24:04.876577+0000","last_clean":"2024-03-06T07:24:04.876577+0000","last_became_active":"2024-03-06T07:24:03.884571+0000","last_became_peered":"2024-03-06T07:24:03.884571+0000","last_unstale":"2024-03-06T07:24:04.876577+0000","last_undegraded":"2024-03-06T07:24:04.876577+0000","last_fullsized":"2024-03-06T07:24:04.876577+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:12.327628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.875838+0000","last_change":"2024-03-06T07:24:03.891048+0000","last_active":"2024-03-06T07:24:04.875838+0000","last_peered":"2024-03-06T07:24:04.875838+0000","last_clean":"2024-03-06T07:24:04.875838+0000","last_became_active":"2024-03-06T07:24:03.890787+0000","last_became_peered":"2024-03-06T07:24:03.890787+0000","last_unstale":"2024-03-06T07:24:04.875838+0000","last_undegraded":"2024-03-06T07:24:04.875838+0000","last_fullsized":"2024-03-06T07:24:04.875838+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:39.286189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876281+0000","last_change":"2024-03-06T07:24:03.884181+0000","last_active":"2024-03-06T07:24:04.876281+0000","last_peered":"2024-03-06T07:24:04.876281+0000","last_clean":"2024-03-06T07:24:04.876281+0000","last_became_active":"2024-03-06T07:24:03.883923+0000","last_became_peered":"2024-03-06T07:24:03.883923+0000","last_unstale":"2024-03-06T07:24:04.876281+0000","last_undegraded":"2024-03-06T07:24:04.876281+0000","last_fullsized":"2024-03-06T07:24:04.876281+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:04.190305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549775+0000","last_change":"2024-03-06T07:24:03.893008+0000","last_active":"2024-03-06T07:24:06.549775+0000","last_peered":"2024-03-06T07:24:06.549775+0000","last_clean":"2024-03-06T07:24:06.549775+0000","last_became_active":"2024-03-06T07:24:03.892568+0000","last_became_peered":"2024-03-06T07:24:03.892568+0000","last_unstale":"2024-03-06T07:24:06.549775+0000","last_undegraded":"2024-03-06T07:24:06.549775+0000","last_fullsized":"2024-03-06T07:24:06.549775+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":18,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550516+0000","last_change":"2024-03-06T07:24:02.889021+0000","last_active":"2024-03-06T07:24:06.550516+0000","last_peered":"2024-03-06T07:24:06.550516+0000","last_clean":"2024-03-06T07:24:06.550516+0000","last_became_active":"2024-03-06T07:24:02.888760+0000","last_became_peered":"2024-03-06T07:24:02.888760+0000","last_unstale":"2024-03-06T07:24:06.550516+0000","last_undegraded":"2024-03-06T07:24:06.550516+0000","last_fullsized":"2024-03-06T07:24:06.550516+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:56.709899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874471+0000","last_change":"2024-03-06T07:24:03.895263+0000","last_active":"2024-03-06T07:24:04.874471+0000","last_peered":"2024-03-06T07:24:04.874471+0000","last_clean":"2024-03-06T07:24:04.874471+0000","last_became_active":"2024-03-06T07:24:03.894999+0000","last_became_peered":"2024-03-06T07:24:03.894999+0000","last_unstale":"2024-03-06T07:24:04.874471+0000","last_undegraded":"2024-03-06T07:24:04.874471+0000","last_fullsized":"2024-03-06T07:24:04.874471+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:37.830052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549559+0000","last_change":"2024-03-06T07:24:02.881058+0000","last_active":"2024-03-06T07:24:06.549559+0000","last_peered":"2024-03-06T07:24:06.549559+0000","last_clean":"2024-03-06T07:24:06.549559+0000","last_became_active":"2024-03-06T07:24:02.880664+0000","last_became_peered":"2024-03-06T07:24:02.880664+0000","last_unstale":"2024-03-06T07:24:06.549559+0000","last_undegraded":"2024-03-06T07:24:06.549559+0000","last_fullsized":"2024-03-06T07:24:06.549559+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:06.978957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549737+0000","last_change":"2024-03-06T07:24:03.890715+0000","last_active":"2024-03-06T07:24:06.549737+0000","last_peered":"2024-03-06T07:24:06.549737+0000","last_clean":"2024-03-06T07:24:06.549737+0000","last_became_active":"2024-03-06T07:24:03.890464+0000","last_became_peered":"2024-03-06T07:24:03.890464+0000","last_unstale":"2024-03-06T07:24:06.549737+0000","last_undegraded":"2024-03-06T07:24:06.549737+0000","last_fullsized":"2024-03-06T07:24:06.549737+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876415+0000","last_change":"2024-03-06T07:24:02.881481+0000","last_active":"2024-03-06T07:24:04.876415+0000","last_peered":"2024-03-06T07:24:04.876415+0000","last_clean":"2024-03-06T07:24:04.876415+0000","last_became_active":"2024-03-06T07:24:02.881205+0000","last_became_peered":"2024-03-06T07:24:02.881205+0000","last_unstale":"2024-03-06T07:24:04.876415+0000","last_undegraded":"2024-03-06T07:24:04.876415+0000","last_fullsized":"2024-03-06T07:24:04.876415+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:01.647744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550688+0000","last_change":"2024-03-06T07:24:03.892672+0000","last_active":"2024-03-06T07:24:06.550688+0000","last_peered":"2024-03-06T07:24:06.550688+0000","last_clean":"2024-03-06T07:24:06.550688+0000","last_became_active":"2024-03-06T07:24:03.891741+0000","last_became_peered":"2024-03-06T07:24:03.891741+0000","last_unstale":"2024-03-06T07:24:06.550688+0000","last_undegraded":"2024-03-06T07:24:06.550688+0000","last_fullsized":"2024-03-06T07:24:06.550688+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:39.675776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284857+0000","last_change":"2024-03-06T07:24:02.880928+0000","last_active":"2024-03-06T07:24:06.284857+0000","last_peered":"2024-03-06T07:24:06.284857+0000","last_clean":"2024-03-06T07:24:06.284857+0000","last_became_active":"2024-03-06T07:24:02.880661+0000","last_became_peered":"2024-03-06T07:24:02.880661+0000","last_unstale":"2024-03-06T07:24:06.284857+0000","last_undegraded":"2024-03-06T07:24:06.284857+0000","last_fullsized":"2024-03-06T07:24:06.284857+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:01.167170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877289+0000","last_change":"2024-03-06T07:24:03.898477+0000","last_active":"2024-03-06T07:24:05.877289+0000","last_peered":"2024-03-06T07:24:05.877289+0000","last_clean":"2024-03-06T07:24:05.877289+0000","last_became_active":"2024-03-06T07:24:03.897814+0000","last_became_peered":"2024-03-06T07:24:03.897814+0000","last_unstale":"2024-03-06T07:24:05.877289+0000","last_undegraded":"2024-03-06T07:24:05.877289+0000","last_fullsized":"2024-03-06T07:24:05.877289+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:51.104075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877108+0000","last_change":"2024-03-06T07:24:02.878433+0000","last_active":"2024-03-06T07:24:05.877108+0000","last_peered":"2024-03-06T07:24:05.877108+0000","last_clean":"2024-03-06T07:24:05.877108+0000","last_became_active":"2024-03-06T07:24:02.878153+0000","last_became_peered":"2024-03-06T07:24:02.878153+0000","last_unstale":"2024-03-06T07:24:05.877108+0000","last_undegraded":"2024-03-06T07:24:05.877108+0000","last_fullsized":"2024-03-06T07:24:05.877108+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:27.611384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.285052+0000","last_change":"2024-03-06T07:24:03.891217+0000","last_active":"2024-03-06T07:24:06.285052+0000","last_peered":"2024-03-06T07:24:06.285052+0000","last_clean":"2024-03-06T07:24:06.285052+0000","last_became_active":"2024-03-06T07:24:03.890968+0000","last_became_peered":"2024-03-06T07:24:03.890968+0000","last_unstale":"2024-03-06T07:24:06.285052+0000","last_undegraded":"2024-03-06T07:24:06.285052+0000","last_fullsized":"2024-03-06T07:24:06.285052+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:26.282549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549651+0000","last_change":"2024-03-06T07:24:02.890689+0000","last_active":"2024-03-06T07:24:06.549651+0000","last_peered":"2024-03-06T07:24:06.549651+0000","last_clean":"2024-03-06T07:24:06.549651+0000","last_became_active":"2024-03-06T07:24:02.890424+0000","last_became_peered":"2024-03-06T07:24:02.890424+0000","last_unstale":"2024-03-06T07:24:06.549651+0000","last_undegraded":"2024-03-06T07:24:06.549651+0000","last_fullsized":"2024-03-06T07:24:06.549651+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284823+0000","last_change":"2024-03-06T07:24:03.878910+0000","last_active":"2024-03-06T07:24:06.284823+0000","last_peered":"2024-03-06T07:24:06.284823+0000","last_clean":"2024-03-06T07:24:06.284823+0000","last_became_active":"2024-03-06T07:24:03.878660+0000","last_became_peered":"2024-03-06T07:24:03.878660+0000","last_unstale":"2024-03-06T07:24:06.284823+0000","last_undegraded":"2024-03-06T07:24:06.284823+0000","last_fullsized":"2024-03-06T07:24:06.284823+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:52.217554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284593+0000","last_change":"2024-03-06T07:24:02.891744+0000","last_active":"2024-03-06T07:24:06.284593+0000","last_peered":"2024-03-06T07:24:06.284593+0000","last_clean":"2024-03-06T07:24:06.284593+0000","last_became_active":"2024-03-06T07:24:02.891480+0000","last_became_peered":"2024-03-06T07:24:02.891480+0000","last_unstale":"2024-03-06T07:24:06.284593+0000","last_undegraded":"2024-03-06T07:24:06.284593+0000","last_fullsized":"2024-03-06T07:24:06.284593+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:07.998051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550437+0000","last_change":"2024-03-06T07:24:03.885245+0000","last_active":"2024-03-06T07:24:06.550437+0000","last_peered":"2024-03-06T07:24:06.550437+0000","last_clean":"2024-03-06T07:24:06.550437+0000","last_became_active":"2024-03-06T07:24:03.884959+0000","last_became_peered":"2024-03-06T07:24:03.884959+0000","last_unstale":"2024-03-06T07:24:06.550437+0000","last_undegraded":"2024-03-06T07:24:06.550437+0000","last_fullsized":"2024-03-06T07:24:06.550437+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:44.253526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877107+0000","last_change":"2024-03-06T07:24:02.886573+0000","last_active":"2024-03-06T07:24:05.877107+0000","last_peered":"2024-03-06T07:24:05.877107+0000","last_clean":"2024-03-06T07:24:05.877107+0000","last_became_active":"2024-03-06T07:24:02.885928+0000","last_became_peered":"2024-03-06T07:24:02.885928+0000","last_unstale":"2024-03-06T07:24:05.877107+0000","last_undegraded":"2024-03-06T07:24:05.877107+0000","last_fullsized":"2024-03-06T07:24:05.877107+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549798+0000","last_change":"2024-03-06T07:24:03.892190+0000","last_active":"2024-03-06T07:24:06.549798+0000","last_peered":"2024-03-06T07:24:06.549798+0000","last_clean":"2024-03-06T07:24:06.549798+0000","last_became_active":"2024-03-06T07:24:03.891822+0000","last_became_peered":"2024-03-06T07:24:03.891822+0000","last_unstale":"2024-03-06T07:24:06.549798+0000","last_undegraded":"2024-03-06T07:24:06.549798+0000","last_fullsized":"2024-03-06T07:24:06.549798+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874611+0000","last_change":"2024-03-06T07:24:02.892342+0000","last_active":"2024-03-06T07:24:04.874611+0000","last_peered":"2024-03-06T07:24:04.874611+0000","last_clean":"2024-03-06T07:24:04.874611+0000","last_became_active":"2024-03-06T07:24:02.892089+0000","last_became_peered":"2024-03-06T07:24:02.892089+0000","last_unstale":"2024-03-06T07:24:04.874611+0000","last_undegraded":"2024-03-06T07:24:04.874611+0000","last_fullsized":"2024-03-06T07:24:04.874611+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874503+0000","last_change":"2024-03-06T07:24:03.892347+0000","last_active":"2024-03-06T07:24:04.874503+0000","last_peered":"2024-03-06T07:24:04.874503+0000","last_clean":"2024-03-06T07:24:04.874503+0000","last_became_active":"2024-03-06T07:24:03.892074+0000","last_became_peered":"2024-03-06T07:24:03.892074+0000","last_unstale":"2024-03-06T07:24:04.874503+0000","last_undegraded":"2024-03-06T07:24:04.874503+0000","last_fullsized":"2024-03-06T07:24:04.874503+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:20.749704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":17,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550062+0000","last_change":"2024-03-06T07:24:02.886419+0000","last_active":"2024-03-06T07:24:06.550062+0000","last_peered":"2024-03-06T07:24:06.550062+0000","last_clean":"2024-03-06T07:24:06.550062+0000","last_became_active":"2024-03-06T07:24:02.886064+0000","last_became_peered":"2024-03-06T07:24:02.886064+0000","last_unstale":"2024-03-06T07:24:06.550062+0000","last_undegraded":"2024-03-06T07:24:06.550062+0000","last_fullsized":"2024-03-06T07:24:06.550062+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:29.962503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284810+0000","last_change":"2024-03-06T07:24:03.886038+0000","last_active":"2024-03-06T07:24:06.284810+0000","last_peered":"2024-03-06T07:24:06.284810+0000","last_clean":"2024-03-06T07:24:06.284810+0000","last_became_active":"2024-03-06T07:24:03.885779+0000","last_became_peered":"2024-03-06T07:24:03.885779+0000","last_unstale":"2024-03-06T07:24:06.284810+0000","last_undegraded":"2024-03-06T07:24:06.284810+0000","last_fullsized":"2024-03-06T07:24:06.284810+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:01.188435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":20,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874792+0000","last_change":"2024-03-06T07:24:02.889795+0000","last_active":"2024-03-06T07:24:04.874792+0000","last_peered":"2024-03-06T07:24:04.874792+0000","last_clean":"2024-03-06T07:24:04.874792+0000","last_became_active":"2024-03-06T07:24:02.889534+0000","last_became_peered":"2024-03-06T07:24:02.889534+0000","last_unstale":"2024-03-06T07:24:04.874792+0000","last_undegraded":"2024-03-06T07:24:04.874792+0000","last_fullsized":"2024-03-06T07:24:04.874792+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:26.432122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876040+0000","last_change":"2024-03-06T07:24:03.883274+0000","last_active":"2024-03-06T07:24:04.876040+0000","last_peered":"2024-03-06T07:24:04.876040+0000","last_clean":"2024-03-06T07:24:04.876040+0000","last_became_active":"2024-03-06T07:24:03.882970+0000","last_became_peered":"2024-03-06T07:24:03.882970+0000","last_unstale":"2024-03-06T07:24:04.876040+0000","last_undegraded":"2024-03-06T07:24:04.876040+0000","last_fullsized":"2024-03-06T07:24:04.876040+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:34.606194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549716+0000","last_change":"2024-03-06T07:24:02.886772+0000","last_active":"2024-03-06T07:24:06.549716+0000","last_peered":"2024-03-06T07:24:06.549716+0000","last_clean":"2024-03-06T07:24:06.549716+0000","last_became_active":"2024-03-06T07:24:02.886444+0000","last_became_peered":"2024-03-06T07:24:02.886444+0000","last_unstale":"2024-03-06T07:24:06.549716+0000","last_undegraded":"2024-03-06T07:24:06.549716+0000","last_fullsized":"2024-03-06T07:24:06.549716+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:14.833238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877420+0000","last_change":"2024-03-06T07:24:03.899462+0000","last_active":"2024-03-06T07:24:05.877420+0000","last_peered":"2024-03-06T07:24:05.877420+0000","last_clean":"2024-03-06T07:24:05.877420+0000","last_became_active":"2024-03-06T07:24:03.899154+0000","last_became_peered":"2024-03-06T07:24:03.899154+0000","last_unstale":"2024-03-06T07:24:05.877420+0000","last_undegraded":"2024-03-06T07:24:05.877420+0000","last_fullsized":"2024-03-06T07:24:05.877420+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:05.985567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874652+0000","last_change":"2024-03-06T07:24:02.886145+0000","last_active":"2024-03-06T07:24:04.874652+0000","last_peered":"2024-03-06T07:24:04.874652+0000","last_clean":"2024-03-06T07:24:04.874652+0000","last_became_active":"2024-03-06T07:24:02.885886+0000","last_became_peered":"2024-03-06T07:24:02.885886+0000","last_unstale":"2024-03-06T07:24:04.874652+0000","last_undegraded":"2024-03-06T07:24:04.874652+0000","last_fullsized":"2024-03-06T07:24:04.874652+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:43.257419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876129+0000","last_change":"2024-03-06T07:24:03.883971+0000","last_active":"2024-03-06T07:24:04.876129+0000","last_peered":"2024-03-06T07:24:04.876129+0000","last_clean":"2024-03-06T07:24:04.876129+0000","last_became_active":"2024-03-06T07:24:03.883705+0000","last_became_peered":"2024-03-06T07:24:03.883705+0000","last_unstale":"2024-03-06T07:24:04.876129+0000","last_undegraded":"2024-03-06T07:24:04.876129+0000","last_fullsized":"2024-03-06T07:24:04.876129+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:34.136559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876203+0000","last_change":"2024-03-06T07:24:02.883880+0000","last_active":"2024-03-06T07:24:04.876203+0000","last_peered":"2024-03-06T07:24:04.876203+0000","last_clean":"2024-03-06T07:24:04.876203+0000","last_became_active":"2024-03-06T07:24:02.883488+0000","last_became_peered":"2024-03-06T07:24:02.883488+0000","last_unstale":"2024-03-06T07:24:04.876203+0000","last_undegraded":"2024-03-06T07:24:04.876203+0000","last_fullsized":"2024-03-06T07:24:04.876203+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:31.545966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.285038+0000","last_change":"2024-03-06T07:24:03.883761+0000","last_active":"2024-03-06T07:24:06.285038+0000","last_peered":"2024-03-06T07:24:06.285038+0000","last_clean":"2024-03-06T07:24:06.285038+0000","last_became_active":"2024-03-06T07:24:03.882991+0000","last_became_peered":"2024-03-06T07:24:03.882991+0000","last_unstale":"2024-03-06T07:24:06.285038+0000","last_undegraded":"2024-03-06T07:24:06.285038+0000","last_fullsized":"2024-03-06T07:24:06.285038+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:28.659458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":22,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877115+0000","last_change":"2024-03-06T07:24:02.885178+0000","last_active":"2024-03-06T07:24:05.877115+0000","last_peered":"2024-03-06T07:24:05.877115+0000","last_clean":"2024-03-06T07:24:05.877115+0000","last_became_active":"2024-03-06T07:24:02.880268+0000","last_became_peered":"2024-03-06T07:24:02.880268+0000","last_unstale":"2024-03-06T07:24:05.877115+0000","last_undegraded":"2024-03-06T07:24:05.877115+0000","last_fullsized":"2024-03-06T07:24:05.877115+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:14.833238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877042+0000","last_change":"2024-03-06T07:24:03.888753+0000","last_active":"2024-03-06T07:24:05.877042+0000","last_peered":"2024-03-06T07:24:05.877042+0000","last_clean":"2024-03-06T07:24:05.877042+0000","last_became_active":"2024-03-06T07:24:03.888491+0000","last_became_peered":"2024-03-06T07:24:03.888491+0000","last_unstale":"2024-03-06T07:24:05.877042+0000","last_undegraded":"2024-03-06T07:24:05.877042+0000","last_fullsized":"2024-03-06T07:24:05.877042+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874818+0000","last_change":"2024-03-06T07:24:02.886460+0000","last_active":"2024-03-06T07:24:04.874818+0000","last_peered":"2024-03-06T07:24:04.874818+0000","last_clean":"2024-03-06T07:24:04.874818+0000","last_became_active":"2024-03-06T07:24:02.885733+0000","last_became_peered":"2024-03-06T07:24:02.885733+0000","last_unstale":"2024-03-06T07:24:04.874818+0000","last_undegraded":"2024-03-06T07:24:04.874818+0000","last_fullsized":"2024-03-06T07:24:04.874818+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:59.543991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876895+0000","last_change":"2024-03-06T07:24:03.889875+0000","last_active":"2024-03-06T07:24:04.876895+0000","last_peered":"2024-03-06T07:24:04.876895+0000","last_clean":"2024-03-06T07:24:04.876895+0000","last_became_active":"2024-03-06T07:24:03.889040+0000","last_became_peered":"2024-03-06T07:24:03.889040+0000","last_unstale":"2024-03-06T07:24:04.876895+0000","last_undegraded":"2024-03-06T07:24:04.876895+0000","last_fullsized":"2024-03-06T07:24:04.876895+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:57.658966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877554+0000","last_change":"2024-03-06T07:24:02.888816+0000","last_active":"2024-03-06T07:24:05.877554+0000","last_peered":"2024-03-06T07:24:05.877554+0000","last_clean":"2024-03-06T07:24:05.877554+0000","last_became_active":"2024-03-06T07:24:02.888090+0000","last_became_peered":"2024-03-06T07:24:02.888090+0000","last_unstale":"2024-03-06T07:24:05.877554+0000","last_undegraded":"2024-03-06T07:24:05.877554+0000","last_fullsized":"2024-03-06T07:24:05.877554+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:42.738046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876910+0000","last_change":"2024-03-06T07:24:03.890038+0000","last_active":"2024-03-06T07:24:04.876910+0000","last_peered":"2024-03-06T07:24:04.876910+0000","last_clean":"2024-03-06T07:24:04.876910+0000","last_became_active":"2024-03-06T07:24:03.889784+0000","last_became_peered":"2024-03-06T07:24:03.889784+0000","last_unstale":"2024-03-06T07:24:04.876910+0000","last_undegraded":"2024-03-06T07:24:04.876910+0000","last_fullsized":"2024-03-06T07:24:04.876910+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:35.461921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549600+0000","last_change":"2024-03-06T07:24:02.887549+0000","last_active":"2024-03-06T07:24:06.549600+0000","last_peered":"2024-03-06T07:24:06.549600+0000","last_clean":"2024-03-06T07:24:06.549600+0000","last_became_active":"2024-03-06T07:24:02.887264+0000","last_became_peered":"2024-03-06T07:24:02.887264+0000","last_unstale":"2024-03-06T07:24:06.549600+0000","last_undegraded":"2024-03-06T07:24:06.549600+0000","last_fullsized":"2024-03-06T07:24:06.549600+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:15.400205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876024+0000","last_change":"2024-03-06T07:24:03.897943+0000","last_active":"2024-03-06T07:24:04.876024+0000","last_peered":"2024-03-06T07:24:04.876024+0000","last_clean":"2024-03-06T07:24:04.876024+0000","last_became_active":"2024-03-06T07:24:03.897665+0000","last_became_peered":"2024-03-06T07:24:03.897665+0000","last_unstale":"2024-03-06T07:24:04.876024+0000","last_undegraded":"2024-03-06T07:24:04.876024+0000","last_fullsized":"2024-03-06T07:24:04.876024+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:36.087346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877207+0000","last_change":"2024-03-06T07:24:02.881968+0000","last_active":"2024-03-06T07:24:05.877207+0000","last_peered":"2024-03-06T07:24:05.877207+0000","last_clean":"2024-03-06T07:24:05.877207+0000","last_became_active":"2024-03-06T07:24:02.881702+0000","last_became_peered":"2024-03-06T07:24:02.881702+0000","last_unstale":"2024-03-06T07:24:05.877207+0000","last_undegraded":"2024-03-06T07:24:05.877207+0000","last_fullsized":"2024-03-06T07:24:05.877207+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:37.173846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874853+0000","last_change":"2024-03-06T07:24:03.894250+0000","last_active":"2024-03-06T07:24:04.874853+0000","last_peered":"2024-03-06T07:24:04.874853+0000","last_clean":"2024-03-06T07:24:04.874853+0000","last_became_active":"2024-03-06T07:24:03.893672+0000","last_became_peered":"2024-03-06T07:24:03.893672+0000","last_unstale":"2024-03-06T07:24:04.874853+0000","last_undegraded":"2024-03-06T07:24:04.874853+0000","last_fullsized":"2024-03-06T07:24:04.874853+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:55.684312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549810+0000","last_change":"2024-03-06T07:24:02.883228+0000","last_active":"2024-03-06T07:24:06.549810+0000","last_peered":"2024-03-06T07:24:06.549810+0000","last_clean":"2024-03-06T07:24:06.549810+0000","last_became_active":"2024-03-06T07:24:02.882971+0000","last_became_peered":"2024-03-06T07:24:02.882971+0000","last_unstale":"2024-03-06T07:24:06.549810+0000","last_undegraded":"2024-03-06T07:24:06.549810+0000","last_fullsized":"2024-03-06T07:24:06.549810+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:42.738046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549687+0000","last_change":"2024-03-06T07:24:03.889481+0000","last_active":"2024-03-06T07:24:06.549687+0000","last_peered":"2024-03-06T07:24:06.549687+0000","last_clean":"2024-03-06T07:24:06.549687+0000","last_became_active":"2024-03-06T07:24:03.889258+0000","last_became_peered":"2024-03-06T07:24:03.889258+0000","last_unstale":"2024-03-06T07:24:06.549687+0000","last_undegraded":"2024-03-06T07:24:06.549687+0000","last_fullsized":"2024-03-06T07:24:06.549687+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876961+0000","last_change":"2024-03-06T07:24:02.889856+0000","last_active":"2024-03-06T07:24:04.876961+0000","last_peered":"2024-03-06T07:24:04.876961+0000","last_clean":"2024-03-06T07:24:04.876961+0000","last_became_active":"2024-03-06T07:24:02.889589+0000","last_became_peered":"2024-03-06T07:24:02.889589+0000","last_unstale":"2024-03-06T07:24:04.876961+0000","last_undegraded":"2024-03-06T07:24:04.876961+0000","last_fullsized":"2024-03-06T07:24:04.876961+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:09.042646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549635+0000","last_change":"2024-03-06T07:24:03.890643+0000","last_active":"2024-03-06T07:24:06.549635+0000","last_peered":"2024-03-06T07:24:06.549635+0000","last_clean":"2024-03-06T07:24:06.549635+0000","last_became_active":"2024-03-06T07:24:03.890406+0000","last_became_peered":"2024-03-06T07:24:03.890406+0000","last_unstale":"2024-03-06T07:24:06.549635+0000","last_undegraded":"2024-03-06T07:24:06.549635+0000","last_fullsized":"2024-03-06T07:24:06.549635+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284672+0000","last_change":"2024-03-06T07:24:02.882070+0000","last_active":"2024-03-06T07:24:06.284672+0000","last_peered":"2024-03-06T07:24:06.284672+0000","last_clean":"2024-03-06T07:24:06.284672+0000","last_became_active":"2024-03-06T07:24:02.881582+0000","last_became_peered":"2024-03-06T07:24:02.881582+0000","last_unstale":"2024-03-06T07:24:06.284672+0000","last_undegraded":"2024-03-06T07:24:06.284672+0000","last_fullsized":"2024-03-06T07:24:06.284672+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:37.816763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550596+0000","last_change":"2024-03-06T07:24:03.890706+0000","last_active":"2024-03-06T07:24:06.550596+0000","last_peered":"2024-03-06T07:24:06.550596+0000","last_clean":"2024-03-06T07:24:06.550596+0000","last_became_active":"2024-03-06T07:24:03.890369+0000","last_became_peered":"2024-03-06T07:24:03.890369+0000","last_unstale":"2024-03-06T07:24:06.550596+0000","last_undegraded":"2024-03-06T07:24:06.550596+0000","last_fullsized":"2024-03-06T07:24:06.550596+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:35.615657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549911+0000","last_change":"2024-03-06T07:24:02.885639+0000","last_active":"2024-03-06T07:24:06.549911+0000","last_peered":"2024-03-06T07:24:06.549911+0000","last_clean":"2024-03-06T07:24:06.549911+0000","last_became_active":"2024-03-06T07:24:02.885357+0000","last_became_peered":"2024-03-06T07:24:02.885357+0000","last_unstale":"2024-03-06T07:24:06.549911+0000","last_undegraded":"2024-03-06T07:24:06.549911+0000","last_fullsized":"2024-03-06T07:24:06.549911+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.875714+0000","last_change":"2024-03-06T07:24:03.888914+0000","last_active":"2024-03-06T07:24:04.875714+0000","last_peered":"2024-03-06T07:24:04.875714+0000","last_clean":"2024-03-06T07:24:04.875714+0000","last_became_active":"2024-03-06T07:24:03.888605+0000","last_became_peered":"2024-03-06T07:24:03.888605+0000","last_unstale":"2024-03-06T07:24:04.875714+0000","last_undegraded":"2024-03-06T07:24:04.875714+0000","last_fullsized":"2024-03-06T07:24:04.875714+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:07.984372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":16,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549647+0000","last_change":"2024-03-06T07:24:02.884638+0000","last_active":"2024-03-06T07:24:06.549647+0000","last_peered":"2024-03-06T07:24:06.549647+0000","last_clean":"2024-03-06T07:24:06.549647+0000","last_became_active":"2024-03-06T07:24:02.884370+0000","last_became_peered":"2024-03-06T07:24:02.884370+0000","last_unstale":"2024-03-06T07:24:06.549647+0000","last_undegraded":"2024-03-06T07:24:06.549647+0000","last_fullsized":"2024-03-06T07:24:06.549647+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:38.280774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877154+0000","last_change":"2024-03-06T07:24:03.889791+0000","last_active":"2024-03-06T07:24:05.877154+0000","last_peered":"2024-03-06T07:24:05.877154+0000","last_clean":"2024-03-06T07:24:05.877154+0000","last_became_active":"2024-03-06T07:24:03.889121+0000","last_became_peered":"2024-03-06T07:24:03.889121+0000","last_unstale":"2024-03-06T07:24:05.877154+0000","last_undegraded":"2024-03-06T07:24:05.877154+0000","last_fullsized":"2024-03-06T07:24:05.877154+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":15,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874821+0000","last_change":"2024-03-06T07:24:02.891610+0000","last_active":"2024-03-06T07:24:04.874821+0000","last_peered":"2024-03-06T07:24:04.874821+0000","last_clean":"2024-03-06T07:24:04.874821+0000","last_became_active":"2024-03-06T07:24:02.891294+0000","last_became_peered":"2024-03-06T07:24:02.891294+0000","last_unstale":"2024-03-06T07:24:04.874821+0000","last_undegraded":"2024-03-06T07:24:04.874821+0000","last_fullsized":"2024-03-06T07:24:04.874821+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:39.405811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284783+0000","last_change":"2024-03-06T07:24:03.881428+0000","last_active":"2024-03-06T07:24:06.284783+0000","last_peered":"2024-03-06T07:24:06.284783+0000","last_clean":"2024-03-06T07:24:06.284783+0000","last_became_active":"2024-03-06T07:24:03.881168+0000","last_became_peered":"2024-03-06T07:24:03.881168+0000","last_unstale":"2024-03-06T07:24:06.284783+0000","last_undegraded":"2024-03-06T07:24:06.284783+0000","last_fullsized":"2024-03-06T07:24:06.284783+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:00.752911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874583+0000","last_change":"2024-03-06T07:24:02.887206+0000","last_active":"2024-03-06T07:24:04.874583+0000","last_peered":"2024-03-06T07:24:04.874583+0000","last_clean":"2024-03-06T07:24:04.874583+0000","last_became_active":"2024-03-06T07:24:02.886946+0000","last_became_peered":"2024-03-06T07:24:02.886946+0000","last_unstale":"2024-03-06T07:24:04.874583+0000","last_undegraded":"2024-03-06T07:24:04.874583+0000","last_fullsized":"2024-03-06T07:24:04.874583+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:11.489172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:05.877101+0000","last_change":"2024-03-06T07:24:03.887606+0000","last_active":"2024-03-06T07:24:05.877101+0000","last_peered":"2024-03-06T07:24:05.877101+0000","last_clean":"2024-03-06T07:24:05.877101+0000","last_became_active":"2024-03-06T07:24:03.887318+0000","last_became_peered":"2024-03-06T07:24:03.887318+0000","last_unstale":"2024-03-06T07:24:05.877101+0000","last_undegraded":"2024-03-06T07:24:05.877101+0000","last_fullsized":"2024-03-06T07:24:05.877101+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.284952+0000","last_change":"2024-03-06T07:24:02.891159+0000","last_active":"2024-03-06T07:24:06.284952+0000","last_peered":"2024-03-06T07:24:06.284952+0000","last_clean":"2024-03-06T07:24:06.284952+0000","last_became_active":"2024-03-06T07:24:02.890886+0000","last_became_peered":"2024-03-06T07:24:02.890886+0000","last_unstale":"2024-03-06T07:24:06.284952+0000","last_undegraded":"2024-03-06T07:24:06.284952+0000","last_fullsized":"2024-03-06T07:24:06.284952+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:57.098650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550546+0000","last_change":"2024-03-06T07:24:03.886641+0000","last_active":"2024-03-06T07:24:06.550546+0000","last_peered":"2024-03-06T07:24:06.550546+0000","last_clean":"2024-03-06T07:24:06.550546+0000","last_became_active":"2024-03-06T07:24:03.885674+0000","last_became_peered":"2024-03-06T07:24:03.885674+0000","last_unstale":"2024-03-06T07:24:06.550546+0000","last_undegraded":"2024-03-06T07:24:06.550546+0000","last_fullsized":"2024-03-06T07:24:06.550546+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:32.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549965+0000","last_change":"2024-03-06T07:24:02.888862+0000","last_active":"2024-03-06T07:24:06.549965+0000","last_peered":"2024-03-06T07:24:06.549965+0000","last_clean":"2024-03-06T07:24:06.549965+0000","last_became_active":"2024-03-06T07:24:02.888609+0000","last_became_peered":"2024-03-06T07:24:02.888609+0000","last_unstale":"2024-03-06T07:24:06.549965+0000","last_undegraded":"2024-03-06T07:24:06.549965+0000","last_fullsized":"2024-03-06T07:24:06.549965+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:38.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550266+0000","last_change":"2024-03-06T07:24:03.893828+0000","last_active":"2024-03-06T07:24:06.550266+0000","last_peered":"2024-03-06T07:24:06.550266+0000","last_clean":"2024-03-06T07:24:06.550266+0000","last_became_active":"2024-03-06T07:24:03.893578+0000","last_became_peered":"2024-03-06T07:24:03.893578+0000","last_unstale":"2024-03-06T07:24:06.550266+0000","last_undegraded":"2024-03-06T07:24:06.550266+0000","last_fullsized":"2024-03-06T07:24:06.550266+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:05.985567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874549+0000","last_change":"2024-03-06T07:24:02.884916+0000","last_active":"2024-03-06T07:24:04.874549+0000","last_peered":"2024-03-06T07:24:04.874549+0000","last_clean":"2024-03-06T07:24:04.874549+0000","last_became_active":"2024-03-06T07:24:02.883806+0000","last_became_peered":"2024-03-06T07:24:02.883806+0000","last_unstale":"2024-03-06T07:24:04.874549+0000","last_undegraded":"2024-03-06T07:24:04.874549+0000","last_fullsized":"2024-03-06T07:24:04.874549+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:52.367669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874723+0000","last_change":"2024-03-06T07:24:03.895097+0000","last_active":"2024-03-06T07:24:04.874723+0000","last_peered":"2024-03-06T07:24:04.874723+0000","last_clean":"2024-03-06T07:24:04.874723+0000","last_became_active":"2024-03-06T07:24:03.894829+0000","last_became_peered":"2024-03-06T07:24:03.894829+0000","last_unstale":"2024-03-06T07:24:04.874723+0000","last_undegraded":"2024-03-06T07:24:04.874723+0000","last_fullsized":"2024-03-06T07:24:04.874723+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:52.563689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.876224+0000","last_change":"2024-03-06T07:24:02.883059+0000","last_active":"2024-03-06T07:24:04.876224+0000","last_peered":"2024-03-06T07:24:04.876224+0000","last_clean":"2024-03-06T07:24:04.876224+0000","last_became_active":"2024-03-06T07:24:02.882626+0000","last_became_peered":"2024-03-06T07:24:02.882626+0000","last_unstale":"2024-03-06T07:24:04.876224+0000","last_undegraded":"2024-03-06T07:24:04.876224+0000","last_fullsized":"2024-03-06T07:24:04.876224+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:54.907245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.285095+0000","last_change":"2024-03-06T07:24:03.885590+0000","last_active":"2024-03-06T07:24:06.285095+0000","last_peered":"2024-03-06T07:24:06.285095+0000","last_clean":"2024-03-06T07:24:06.285095+0000","last_became_active":"2024-03-06T07:24:03.885328+0000","last_became_peered":"2024-03-06T07:24:03.885328+0000","last_unstale":"2024-03-06T07:24:06.285095+0000","last_undegraded":"2024-03-06T07:24:06.285095+0000","last_fullsized":"2024-03-06T07:24:06.285095+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:32.966742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549221+0000","last_change":"2024-03-06T07:24:02.885607+0000","last_active":"2024-03-06T07:24:06.549221+0000","last_peered":"2024-03-06T07:24:06.549221+0000","last_clean":"2024-03-06T07:24:06.549221+0000","last_became_active":"2024-03-06T07:24:02.885342+0000","last_became_peered":"2024-03-06T07:24:02.885342+0000","last_unstale":"2024-03-06T07:24:06.549221+0000","last_undegraded":"2024-03-06T07:24:06.549221+0000","last_fullsized":"2024-03-06T07:24:06.549221+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:37.173846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.549799+0000","last_change":"2024-03-06T07:24:03.891512+0000","last_active":"2024-03-06T07:24:06.549799+0000","last_peered":"2024-03-06T07:24:06.549799+0000","last_clean":"2024-03-06T07:24:06.549799+0000","last_became_active":"2024-03-06T07:24:03.891241+0000","last_became_peered":"2024-03-06T07:24:03.891241+0000","last_unstale":"2024-03-06T07:24:06.549799+0000","last_undegraded":"2024-03-06T07:24:06.549799+0000","last_fullsized":"2024-03-06T07:24:06.549799+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:04.874725+0000","last_change":"2024-03-06T07:24:02.891039+0000","last_active":"2024-03-06T07:24:04.874725+0000","last_peered":"2024-03-06T07:24:04.874725+0000","last_clean":"2024-03-06T07:24:04.874725+0000","last_became_active":"2024-03-06T07:24:02.890670+0000","last_became_peered":"2024-03-06T07:24:02.890670+0000","last_unstale":"2024-03-06T07:24:04.874725+0000","last_undegraded":"2024-03-06T07:24:04.874725+0000","last_fullsized":"2024-03-06T07:24:04.874725+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:06.218892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":415,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-06T07:24:06.550056+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:06.550056+0000","last_peered":"2024-03-06T07:24:06.550056+0000","last_clean":"2024-03-06T07:24:06.550056+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:06.550056+0000","last_undegraded":"2024-03-06T07:24:06.550056+0000","last_fullsized":"2024-03-06T07:24:06.550056+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574923,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1929216,"data_stored":1488289,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574923,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1875968,"data_stored":904613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574923,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33020,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1884160,"data_stored":904737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574923,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1892352,"data_stored":905377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1896448,"data_stored":906829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574923,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1875968,"data_stored":904613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1888256,"data_stored":906225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574923,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1949696,"data_stored":1495085,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:11.771 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:11.771 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:11.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.919+0000 7f95c98df640 1 Processor -- start 2024-03-06T07:24:11.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c98df640 1 -- start start 2024-03-06T07:24:11.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c98df640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95c40c57d0 0x7f95c4155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c98df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4152dd0 con 0x7f95c41529f0 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c98df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c4158b00 con 0x7f95c40c5370 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c98df640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c4159710 con 0x7f95c40c57d0 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95c40c57d0 0x7f95c4155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95c40c57d0 0x7f95c4155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43086/0 (socket says 172.21.15.94:43086) 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f95c41529f0 0x7f95c40c88f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47670/0 (socket says 172.21.15.94:47670) 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.920+0000 7f95c37fe640 1 -- 172.21.15.94:0/1997603651 learned_addr learned my addr 172.21.15.94:0/1997603651 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.921+0000 7f95c37fe640 1 -- 172.21.15.94:0/1997603651 >> v1:172.21.15.145:6789/0 conn(0x7f95c40c5370 legacy=0x7f95c4155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.921+0000 7f95c37fe640 1 -- 172.21.15.94:0/1997603651 >> v1:172.21.15.94:6789/0 conn(0x7f95c41529f0 legacy=0x7f95c40c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.921+0000 7f95c37fe640 1 -- 172.21.15.94:0/1997603651 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4158de0 con 0x7f95c40c57d0 2024-03-06T07:24:11.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.921+0000 7f95c1ffb640 1 -- 172.21.15.94:0/1997603651 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164491742 0 0) 0x7f95c4152dd0 con 0x7f95c41529f0 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.921+0000 7f95c37fe640 1 --2- 172.21.15.94:0/1997603651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4155c40 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009e50 tx=0x7f95bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9aa7341c45aaba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c1ffb640 1 -- 172.21.15.94:0/1997603651 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc02ca80 con 0x7f95c40c57d0 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c1ffb640 1 -- 172.21.15.94:0/1997603651 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc02cc20 con 0x7f95c40c57d0 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c1ffb640 1 -- 172.21.15.94:0/1997603651 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0343f0 con 0x7f95c40c57d0 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c98df640 1 -- 172.21.15.94:0/1997603651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 msgr2=0x7f95c4155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c98df640 1 --2- 172.21.15.94:0/1997603651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4155c40 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009e50 tx=0x7f95bc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c98df640 1 -- 172.21.15.94:0/1997603651 shutdown_connections 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c98df640 1 --2- 172.21.15.94:0/1997603651 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4155c40 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c98df640 1 -- 172.21.15.94:0/1997603651 >> 172.21.15.94:0/1997603651 conn(0x7f95c40ac580 msgr2=0x7f95c40ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c98df640 1 -- 172.21.15.94:0/1997603651 shutdown_connections 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.922+0000 7f95c98df640 1 -- 172.21.15.94:0/1997603651 wait complete. 2024-03-06T07:24:11.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.923+0000 7f95c98df640 1 Processor -- start 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.923+0000 7f95c98df640 1 -- start start 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c98df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95c40c5370 0x7f95c414df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c98df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c98df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95c41529f0 0x7f95c4157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c98df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c40c68b0 con 0x7f95c40c57d0 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c98df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c40c9000 con 0x7f95c40c5370 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c98df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c41593a0 con 0x7f95c41529f0 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43094/0 (socket says 172.21.15.94:43094) 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 -- 172.21.15.94:0/2190800904 learned_addr learned my addr 172.21.15.94:0/2190800904 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c37fe640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95c41529f0 0x7f95c4157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c2ffd640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95c40c5370 0x7f95c414df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 -- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95c41529f0 msgr2=0x7f95c4157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95c41529f0 0x7f95c4157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 -- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95c40c5370 msgr2=0x7f95c414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95c40c5370 0x7f95c414df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c27fc640 1 -- 172.21.15.94:0/2190800904 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c41eaf50 con 0x7f95c40c57d0 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c37fe640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95c41529f0 0x7f95c4157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.924+0000 7f95c2ffd640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95c40c5370 0x7f95c414df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.925+0000 7f95c27fc640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4157940 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f95b800c460 tx=0x7f95b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.925+0000 7f95abfff640 1 -- 172.21.15.94:0/2190800904 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8016020 con 0x7f95c40c57d0 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.925+0000 7f95abfff640 1 -- 172.21.15.94:0/2190800904 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b8012870 con 0x7f95c40c57d0 2024-03-06T07:24:11.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.925+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c41eb230 con 0x7f95c40c57d0 2024-03-06T07:24:11.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.925+0000 7f95abfff640 1 -- 172.21.15.94:0/2190800904 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8012c60 con 0x7f95c40c57d0 2024-03-06T07:24:11.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.925+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c41eb7f0 con 0x7f95c40c57d0 2024-03-06T07:24:11.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.926+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c404f330 con 0x7f95c40c57d0 2024-03-06T07:24:11.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.926+0000 7f95abfff640 1 -- 172.21.15.94:0/2190800904 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95b8019020 con 0x7f95c40c57d0 2024-03-06T07:24:11.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.927+0000 7f95abfff640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95b006b4d0 0x7f95b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.927+0000 7f95abfff640 1 -- 172.21.15.94:0/2190800904 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95b808a020 con 0x7f95c40c57d0 2024-03-06T07:24:11.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.927+0000 7f95c2ffd640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95b006b4d0 0x7f95b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.928+0000 7f95c2ffd640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95b006b4d0 0x7f95b006d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f95b4000f60 tx=0x7f95b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:11.931+0000 7f95abfff640 1 -- 172.21.15.94:0/2190800904 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95b8020030 con 0x7f95c40c57d0 2024-03-06T07:24:12.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.220+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f95c406bfd0 con 0x7f95c40c57d0 2024-03-06T07:24:12.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.223+0000 7f95abfff640 1 -- 172.21.15.94:0/2190800904 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f95b8058da0 con 0x7f95c40c57d0 2024-03-06T07:24:12.224 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:24:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:24:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:24:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:24:12.225 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:24:12.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95b006b4d0 msgr2=0x7f95b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95b006b4d0 0x7f95b006d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f95b4000f60 tx=0x7f95b4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:12.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 msgr2=0x7f95c4157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4157940 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f95b800c460 tx=0x7f95b800c920 comp rx=0 tx=0).stop 2024-03-06T07:24:12.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 shutdown_connections 2024-03-06T07:24:12.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95b006b4d0 0x7f95b006d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95c41529f0 0x7f95c4157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c40c57d0 0x7f95c4157940 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 --2- 172.21.15.94:0/2190800904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95c40c5370 0x7f95c414df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 >> 172.21.15.94:0/2190800904 conn(0x7f95c40ac580 msgr2=0x7f95c40ab910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 shutdown_connections 2024-03-06T07:24:12.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.225+0000 7f95c98df640 1 -- 172.21.15.94:0/2190800904 wait complete. 2024-03-06T07:24:12.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:12.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.341+0000 7fa990548640 1 Processor -- start 2024-03-06T07:24:12.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.341+0000 7fa990548640 1 -- start start 2024-03-06T07:24:12.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa990548640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9880c88c0 0x7fa988155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa990548640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa988158b60 0x7fa988158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa990548640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9880c73d0 con 0x7fa98814f360 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa990548640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9880c8dd0 con 0x7fa9880c88c0 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa990548640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9880be2e0 con 0x7fa988158b60 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98dae3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9880c88c0 0x7fa988155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98dae3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9880c88c0 0x7fa988155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43110/0 (socket says 172.21.15.94:43110) 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98e2e4640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa98814f360 0x7fa9880c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47686/0 (socket says 172.21.15.94:47686) 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98dae3640 1 -- 172.21.15.94:0/768833684 learned_addr learned my addr 172.21.15.94:0/768833684 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98dae3640 1 -- 172.21.15.94:0/768833684 >> v2:172.21.15.145:3300/0 conn(0x7fa988158b60 msgr2=0x7fa988158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98eae5640 1 --2- 172.21.15.94:0/768833684 >> v2:172.21.15.145:3300/0 conn(0x7fa988158b60 0x7fa988158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98dae3640 1 --2- 172.21.15.94:0/768833684 >> v2:172.21.15.145:3300/0 conn(0x7fa988158b60 0x7fa988158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98dae3640 1 -- 172.21.15.94:0/768833684 >> v1:172.21.15.94:6789/0 conn(0x7fa98814f360 legacy=0x7fa9880c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98d2e2640 1 -- 172.21.15.94:0/768833684 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089964976 0 0) 0x7fa9880c73d0 con 0x7fa98814f360 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98dae3640 1 -- 172.21.15.94:0/768833684 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9880c7670 con 0x7fa9880c88c0 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.342+0000 7fa98eae5640 1 --2- 172.21.15.94:0/768833684 >> v2:172.21.15.145:3300/0 conn(0x7fa988158b60 0x7fa988158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.343+0000 7fa98dae3640 1 --2- 172.21.15.94:0/768833684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9880c88c0 0x7fa988155c10 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fa988152d60 tx=0x7fa97802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f3dc98c4627182 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.343+0000 7fa98d2e2640 1 -- 172.21.15.94:0/768833684 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978035020 con 0x7fa9880c88c0 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.343+0000 7fa98d2e2640 1 -- 172.21.15.94:0/768833684 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa978031870 con 0x7fa9880c88c0 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.343+0000 7fa98d2e2640 1 -- 172.21.15.94:0/768833684 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978031c40 con 0x7fa9880c88c0 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.343+0000 7fa990548640 1 -- 172.21.15.94:0/768833684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9880c88c0 msgr2=0x7fa988155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.343+0000 7fa990548640 1 --2- 172.21.15.94:0/768833684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9880c88c0 0x7fa988155c10 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fa988152d60 tx=0x7fa97802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.344+0000 7fa990548640 1 -- 172.21.15.94:0/768833684 shutdown_connections 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.344+0000 7fa990548640 1 --2- 172.21.15.94:0/768833684 >> v2:172.21.15.145:3300/0 conn(0x7fa988158b60 0x7fa988158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.344+0000 7fa990548640 1 --2- 172.21.15.94:0/768833684 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9880c88c0 0x7fa988155c10 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.344+0000 7fa990548640 1 -- 172.21.15.94:0/768833684 >> 172.21.15.94:0/768833684 conn(0x7fa9880ac570 msgr2=0x7fa9880ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.344+0000 7fa990548640 1 -- 172.21.15.94:0/768833684 shutdown_connections 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.344+0000 7fa990548640 1 -- 172.21.15.94:0/768833684 wait complete. 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 Processor -- start 2024-03-06T07:24:12.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 -- start start 2024-03-06T07:24:12.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9880c88c0 0x7fa9881eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98814f360 0x7fa9880c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa988158b60 0x7fa98814af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880c5a50 con 0x7fa988158b60 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880c7c30 con 0x7fa9880c88c0 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa990548640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880c5bd0 con 0x7fa98814f360 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa98eae5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa988158b60 0x7fa98814af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa98eae5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa988158b60 0x7fa98814af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43122/0 (socket says 172.21.15.94:43122) 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.345+0000 7fa98eae5640 1 -- 172.21.15.94:0/480987180 learned_addr learned my addr 172.21.15.94:0/480987180 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98dae3640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98814f360 0x7fa9880c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98e2e4640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9880c88c0 0x7fa9881eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98eae5640 1 -- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98814f360 msgr2=0x7fa9880c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98eae5640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98814f360 0x7fa9880c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98eae5640 1 -- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9880c88c0 msgr2=0x7fa9881eb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98eae5640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9880c88c0 0x7fa9881eb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98eae5640 1 -- 172.21.15.94:0/480987180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9880be690 con 0x7fa988158b60 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98dae3640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98814f360 0x7fa9880c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa98eae5640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa988158b60 0x7fa98814af90 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa98000cef0 tx=0x7fa98000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa97effd640 1 -- 172.21.15.94:0/480987180 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98001a070 con 0x7fa988158b60 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa97effd640 1 -- 172.21.15.94:0/480987180 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98000d860 con 0x7fa988158b60 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9880be970 con 0x7fa988158b60 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa97effd640 1 -- 172.21.15.94:0/480987180 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98000dc50 con 0x7fa988158b60 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.346+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9880c12a0 con 0x7fa988158b60 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.347+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98804f330 con 0x7fa988158b60 2024-03-06T07:24:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.348+0000 7fa97effd640 1 -- 172.21.15.94:0/480987180 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98001d070 con 0x7fa988158b60 2024-03-06T07:24:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.348+0000 7fa97effd640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa96c06b4d0 0x7fa96c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.348+0000 7fa97effd640 1 -- 172.21.15.94:0/480987180 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa98008ab80 con 0x7fa988158b60 2024-03-06T07:24:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.348+0000 7fa98e2e4640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa96c06b4d0 0x7fa96c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.349+0000 7fa98e2e4640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa96c06b4d0 0x7fa96c06d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa984007fd0 tx=0x7fa984009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.353+0000 7fa97effd640 1 -- 172.21.15.94:0/480987180 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9800599c0 con 0x7fa988158b60 2024-03-06T07:24:12.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.565+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9880c1640 con 0x7fa988158b60 2024-03-06T07:24:12.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.565+0000 7fa97effd640 1 -- 172.21.15.94:0/480987180 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+4990 (secure 0 0 0) 0x7fa9800592c0 con 0x7fa988158b60 2024-03-06T07:24:12.567 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:12.567 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":135,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8155,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8769,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:05.235078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/1613596341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1613596341},{"type":"v1","addr":"172.21.15.94:6835","nonce":1613596341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":11}]} 2024-03-06T07:24:12.567 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 135 2024-03-06T07:24:12.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa96c06b4d0 msgr2=0x7fa96c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa96c06b4d0 0x7fa96c06d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa984007fd0 tx=0x7fa984009040 comp rx=0 tx=0).stop 2024-03-06T07:24:12.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa988158b60 msgr2=0x7fa98814af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa988158b60 0x7fa98814af90 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa98000cef0 tx=0x7fa98000a710 comp rx=0 tx=0).stop 2024-03-06T07:24:12.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 shutdown_connections 2024-03-06T07:24:12.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa96c06b4d0 0x7fa96c06d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa988158b60 0x7fa98814af90 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98814f360 0x7fa9880c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 --2- 172.21.15.94:0/480987180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9880c88c0 0x7fa9881eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 >> 172.21.15.94:0/480987180 conn(0x7fa9880ac570 msgr2=0x7fa9880c9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 shutdown_connections 2024-03-06T07:24:12.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:12.567+0000 7fa990548640 1 -- 172.21.15.94:0/480987180 wait complete. 2024-03-06T07:24:12.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 135, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:03.229941+0000', 'modified': '2024-03-06T07:24:05.235078+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8772}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8772': {'gid': 8772, 'name': 'c', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.94:6835/1613596341', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 1613596341}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 1613596341}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8772, 'qdb_cluster': [8772]} 2024-03-06T07:24:12.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:12.983 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:24:13.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.082+0000 7f975cf32640 1 Processor -- start 2024-03-06T07:24:13.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.082+0000 7f975cf32640 1 -- start start 2024-03-06T07:24:13.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.083+0000 7f975cf32640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 0x7f9758152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.083+0000 7f975cf32640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97580c1140 con 0x7f975806d680 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.083+0000 7f975cf32640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97580c1420 con 0x7f9758072f40 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.083+0000 7f975cf32640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97580c1d50 con 0x7f97580c28f0 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.083+0000 7f975659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f975806d680 0x7f97580c21e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47702/0 (socket says 172.21.15.94:47702) 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.083+0000 7f975659c640 1 -- 172.21.15.94:0/1029493763 learned_addr learned my addr 172.21.15.94:0/1029493763 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f9756d9d640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 0x7f9758152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697179917 0 0) 0x7f97580c1140 con 0x7f975806d680 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9740003520 con 0x7f975806d680 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132793234 0 0) 0x7f97580c1420 con 0x7f9758072f40 2024-03-06T07:24:13.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97580c1140 con 0x7f9758072f40 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237651049 0 0) 0x7f9740003520 con 0x7f975806d680 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 msgr2=0x7f9758152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 0x7f9758152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 >> v1:172.21.15.145:6789/0 conn(0x7f9758072f40 legacy=0x7f975814e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97580c1700 con 0x7f975806d680 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f97500015f0 con 0x7f975806d680 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 >> v1:172.21.15.94:6789/0 conn(0x7f975806d680 legacy=0x7f97580c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9740003b70 0x7f9740003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f9740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 0x7f9740005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97580c1d50 con 0x7f9740004800 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974000bb50 con 0x7f9740003b70 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974000bd20 con 0x7f97580c28f0 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f9740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f9756d9d640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 0x7f9740005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.084+0000 7f975659c640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9740003b70 0x7f9740003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f9755d9b640 1 -- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 msgr2=0x7f9740005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97580c28f0 0x7f9740005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f9755d9b640 1 -- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9740003b70 msgr2=0x7f9740003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9740003b70 0x7f9740003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f9755d9b640 1 -- 172.21.15.94:0/1029493763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9740003520 con 0x7f9740004800 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f9740004c80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f974c0018d0 tx=0x7f9740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b7bfc98e424725 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c037070 con 0x7f9740004800 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c0052d0 con 0x7f9740004800 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.085+0000 7f975559a640 1 -- 172.21.15.94:0/1029493763 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c0014f0 con 0x7f9740004800 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.086+0000 7f975cf32640 1 -- 172.21.15.94:0/1029493763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 msgr2=0x7f9740004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.086+0000 7f975cf32640 1 --2- 172.21.15.94:0/1029493763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f9740004c80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f974c0018d0 tx=0x7f9740005830 comp rx=0 tx=0).stop 2024-03-06T07:24:13.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.086+0000 7f9756d9d640 1 -- 172.21.15.94:0/1029493763 reap_dead start 2024-03-06T07:24:13.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.086+0000 7f975cf32640 1 -- 172.21.15.94:0/1029493763 shutdown_connections 2024-03-06T07:24:13.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.086+0000 7f975cf32640 1 -- 172.21.15.94:0/1029493763 >> 172.21.15.94:0/1029493763 conn(0x7f975806b1a0 msgr2=0x7f975806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.086+0000 7f975cf32640 1 -- 172.21.15.94:0/1029493763 shutdown_connections 2024-03-06T07:24:13.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.086+0000 7f975cf32640 1 -- 172.21.15.94:0/1029493763 wait complete. 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.087+0000 7f975cf32640 1 Processor -- start 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.087+0000 7f975cf32640 1 -- start start 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975cf32640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f97581404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975cf32640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9758145cc0 0x7f97581444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975cf32640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f975813e230 0x7f975814ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975cf32640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975806da60 con 0x7f9740004800 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975cf32640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97580c2d80 con 0x7f975813e230 2024-03-06T07:24:13.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975cf32640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975806dbe0 con 0x7f9758145cc0 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f97581404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f97581404e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43140/0 (socket says 172.21.15.94:43140) 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 -- 172.21.15.94:0/3675129569 learned_addr learned my addr 172.21.15.94:0/3675129569 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f9756d9d640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f975813e230 0x7f975814ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f9755d9b640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9758145cc0 0x7f97581444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 -- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9758145cc0 msgr2=0x7f97581444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9758145cc0 0x7f97581444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 -- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f975813e230 msgr2=0x7f975814ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f975813e230 0x7f975814ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.088+0000 7f975659c640 1 -- 172.21.15.94:0/3675129569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97580c0510 con 0x7f9740004800 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.089+0000 7f975659c640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f97581404e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f9750004e00 tx=0x7f9750007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.089+0000 7f9746ffd640 1 -- 172.21.15.94:0/3675129569 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975000f7b0 con 0x7f9740004800 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.089+0000 7f9746ffd640 1 -- 172.21.15.94:0/3675129569 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9750018070 con 0x7f9740004800 2024-03-06T07:24:13.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.089+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975813f9a0 con 0x7f9740004800 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.089+0000 7f9746ffd640 1 -- 172.21.15.94:0/3675129569 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975000fbb0 con 0x7f9740004800 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.089+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97580bbdd0 con 0x7f9740004800 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.091+0000 7f9746ffd640 1 -- 172.21.15.94:0/3675129569 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f975000fd50 con 0x7f9740004800 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.091+0000 7f9746ffd640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f973806b4d0 0x7f973806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.091+0000 7f9746ffd640 1 -- 172.21.15.94:0/3675129569 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f975008a150 con 0x7f9740004800 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.091+0000 7f9755d9b640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f973806b4d0 0x7f973806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 Processor -- start 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 -- start start 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f9755d9b640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f973806b4d0 0x7f973806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f974c034000 tx=0x7f974c031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a74f50 0x55b109a72a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a72f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a734b0 0x55b109a6f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1099d96b0 con 0x55b109a74f50 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b109982a60 con 0x55b109a75330 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.092+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b1099d5750 con 0x55b109a734b0 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a74f50 0x55b109a72a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a74f50 0x55b109a72a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43150/0 (socket says 172.21.15.94:43150) 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 -- 172.21.15.94:0/1315934690 learned_addr learned my addr 172.21.15.94:0/1315934690 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a72f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f9756d9d640 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a734b0 0x55b109a6f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 -- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a734b0 msgr2=0x55b109a6f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a734b0 0x55b109a6f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 -- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 msgr2=0x55b109a72f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a72f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 -- 172.21.15.94:0/1315934690 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1099a0f30 con 0x55b109a74f50 2024-03-06T07:24:13.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a72f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f975659c640 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a74f50 0x55b109a72a30 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f97500609b0 tx=0x7f9750058410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8ce6237a816f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f9737fff640 1 -- 172.21.15.94:0/1315934690 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975005c920 con 0x55b109a74f50 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f9737fff640 1 -- 172.21.15.94:0/1315934690 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9750060ee0 con 0x55b109a74f50 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.093+0000 7f9737fff640 1 -- 172.21.15.94:0/1315934690 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97500701d0 con 0x55b109a74f50 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 -- 172.21.15.94:0/1315934690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a74f50 msgr2=0x55b109a72a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a74f50 0x55b109a72a30 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f97500609b0 tx=0x7f9750058410 comp rx=0 tx=0).stop 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 -- 172.21.15.94:0/1315934690 shutdown_connections 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a734b0 0x55b109a6f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a72f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1315934690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a74f50 0x55b109a72a30 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 -- 172.21.15.94:0/1315934690 >> 172.21.15.94:0/1315934690 conn(0x55b109a717d0 msgr2=0x55b109a963c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 -- 172.21.15.94:0/1315934690 shutdown_connections 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 -- 172.21.15.94:0/1315934690 wait complete. 2024-03-06T07:24:13.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 -- start start 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.094+0000 7f975fbf0740 1 Processor -- start 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a957b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109ac21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 0x55b109a981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f975fbf0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b109a97230 con 0x55b109a75330 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b109995ea0 con 0x55b109a97d30 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b109a96c70 con 0x55b109a74f50 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f975fbf0740 10 client.0 _ll_register_callbacks cb 0x55b109995870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f9755d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109ac21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f9755d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109ac21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43158/0 (socket says 172.21.15.94:43158) 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.095+0000 7f9755d9b640 1 -- 172.21.15.94:0/1389517619 learned_addr learned my addr 172.21.15.94:0/1389517619 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f975659c640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f9756d9d640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 0x55b109a981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f9755d9b640 1 -- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 msgr2=0x55b109a957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a957b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f9755d9b640 1 -- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 msgr2=0x55b109a981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 0x55b109a981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f9755d9b640 1 -- 172.21.15.94:0/1389517619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b109996270 con 0x55b109a75330 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f975659c640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f9755d9b640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109ac21c0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f974c02be20 tx=0x7f974c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b383714027d0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f97357fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f97357fa640 1 -- 172.21.15.94:0/1389517619 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c04c070 con 0x55b109a75330 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f97357fa640 1 -- 172.21.15.94:0/1389517619 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f974c006c60 con 0x55b109a75330 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9750060ee0 con 0x55b109a75330 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f97357fa640 1 -- 172.21.15.94:0/1389517619 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f974c057050 con 0x55b109a75330 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f975fbf0740 10 client.8928 fetch_fsmap learned FSMap version 135 2024-03-06T07:24:13.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) v3 -- 0x55b1096ee2d0 con 0x55b109a75330 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f97357fa640 1 -- 172.21.15.94:0/1389517619 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9750060ee0 con 0x55b109a75330 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f97357fa640 1 -- 172.21.15.94:0/1389517619 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 135) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f974c05a020 con 0x55b109a75330 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f975fbf0740 10 client.8928 fetch_fsmap finished waiting for FSMap version 135 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.096+0000 7f975fbf0740 10 client.8928 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.097+0000 7f975fbf0740 20 client.8928 populate_metadata read hostname 'smithi094' 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.097+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109ac5730 0x55b109ad02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.097+0000 7f975fbf0740 4 client.8928 mds_command: new command op to 8772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.097+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 --> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b1099e6bf0 con 0x55b109ac5730 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.097+0000 7f975659c640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109ac5730 0x55b109ad02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.097+0000 7f975659c640 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109ac5730 0x55b109ad02f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=686af6856b134671 server_cookie=5206e2873bea2c68 in_seq=0 out_seq=0 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.097+0000 7f97357fa640 10 client.8928 ms_handle_connect on v2:172.21.15.94:6833/1613596341 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f97357fa640 1 -- 172.21.15.94:0/1389517619 <== mds.0 v2:172.21.15.94:6833/1613596341 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b1099e6bf0 con 0x55b109ac5730 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f97357fa640 10 client.8928 handle_command_reply: tid=0 2024-03-06T07:24:13.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 client.8928 shutdown 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 msgr2=0x55b109ac21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109ac21c0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f974c02be20 tx=0x7f974c02e000 comp rx=0 tx=0).stop 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 shutdown_connections 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 0x55b109a981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109ac21c0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109ac5730 0x55b109ad02f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 --2- 172.21.15.94:0/1389517619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f97357fa640 0 client.8928 ms_handle_reset on v2:172.21.15.94:6833/1613596341 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 >> 172.21.15.94:0/1389517619 conn(0x55b109a734b0 msgr2=0x55b109ab1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 shutdown_connections 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 1 -- 172.21.15.94:0/1389517619 wait complete. 2024-03-06T07:24:13.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.098+0000 7f975fbf0740 20 client.8928 trim_cache size 0 max 16384 2024-03-06T07:24:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.114+0000 7f975fbf0740 1 Processor -- start 2024-03-06T07:24:13.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.114+0000 7f975fbf0740 1 -- start start 2024-03-06T07:24:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109ab5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a6e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a97d30 0x55b109a6ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f975fbf0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97581f6060 con 0x55b109a97d30 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97581f5220 con 0x55b109a75330 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97581f4620 con 0x55b109a74f50 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a97d30 0x55b109a6ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a97d30 0x55b109a6ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43174/0 (socket says 172.21.15.94:43174) 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 -- 172.21.15.94:0/4055548385 learned_addr learned my addr 172.21.15.94:0/4055548385 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 -- 172.21.15.94:0/4055548385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 msgr2=0x55b109ab5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 --2- 172.21.15.94:0/4055548385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109ab5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 -- 172.21.15.94:0/4055548385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 msgr2=0x55b109a6e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 --2- 172.21.15.94:0/4055548385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a6e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.115+0000 7f9756d9d640 1 -- 172.21.15.94:0/4055548385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1097b86b0 con 0x55b109a97d30 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f9756d9d640 1 --2- 172.21.15.94:0/4055548385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a97d30 0x55b109a6ea50 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f97580abe90 tx=0x7f9748009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a3576f2af717b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f97377fe640 1 -- 172.21.15.94:0/4055548385 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748021630 con 0x55b109a97d30 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f97377fe640 1 -- 172.21.15.94:0/4055548385 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97480217d0 con 0x55b109a97d30 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f97377fe640 1 -- 172.21.15.94:0/4055548385 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9748021b80 con 0x55b109a97d30 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 -- 172.21.15.94:0/4055548385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a97d30 msgr2=0x55b109a6ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 --2- 172.21.15.94:0/4055548385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a97d30 0x55b109a6ea50 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f97580abe90 tx=0x7f9748009750 comp rx=0 tx=0).stop 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 -- 172.21.15.94:0/4055548385 shutdown_connections 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 --2- 172.21.15.94:0/4055548385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a97d30 0x55b109a6ea50 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 --2- 172.21.15.94:0/4055548385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a75330 0x55b109a6e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 --2- 172.21.15.94:0/4055548385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109ab5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 -- 172.21.15.94:0/4055548385 >> 172.21.15.94:0/4055548385 conn(0x55b109a734b0 msgr2=0x55b109ab3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 -- 172.21.15.94:0/4055548385 shutdown_connections 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.116+0000 7f975fbf0740 1 -- 172.21.15.94:0/4055548385 wait complete. 2024-03-06T07:24:13.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.117+0000 7f975fbf0740 1 -- start start 2024-03-06T07:24:13.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.117+0000 7f975fbf0740 1 Processor -- start 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a90d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109b20a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 0x55b109b1dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b109ac2700 con 0x55b109a75330 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97581f4c20 con 0x55b109a97d30 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b109a9bc30 con 0x55b109a74f50 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 10 client.0 _ll_register_callbacks cb 0x7f9740003880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109b20a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109b20a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43182/0 (socket says 172.21.15.94:43182) 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 -- 172.21.15.94:0/3303825859 learned_addr learned my addr 172.21.15.94:0/3303825859 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 -- 172.21.15.94:0/3303825859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 msgr2=0x55b109a90d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a90d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 -- 172.21.15.94:0/3303825859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 msgr2=0x55b109b1dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 0x55b109b1dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 -- 172.21.15.94:0/3303825859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b109a70e50 con 0x55b109a75330 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f9755d9b640 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109b20a30 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f974c005620 tx=0x7f974c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51478087157195fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f97357fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f97357fa640 1 -- 172.21.15.94:0/3303825859 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c04c070 con 0x55b109a75330 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f97357fa640 1 -- 172.21.15.94:0/3303825859 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f974c0035f0 con 0x55b109a75330 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97480217d0 con 0x55b109a75330 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.118+0000 7f97357fa640 1 -- 172.21.15.94:0/3303825859 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) v2 ==== 24+0+0 (secure 0 0 0) 0x7f974c06a050 con 0x55b109a75330 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 10 client.8934 fetch_fsmap learned FSMap version 135 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) v3 -- 0x55b109b21140 con 0x55b109a75330 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f97357fa640 1 -- 172.21.15.94:0/3303825859 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97480217d0 con 0x55b109a75330 2024-03-06T07:24:13.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f97357fa640 1 -- 172.21.15.94:0/3303825859 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 135) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f974c06d020 con 0x55b109a75330 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 10 client.8934 fetch_fsmap finished waiting for FSMap version 135 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 10 client.8934 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 20 client.8934 populate_metadata read hostname 'smithi094' 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109b24020 0x55b109b24400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 4 client.8934 mds_command: new command op to 8772 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 --> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b1099e0f40 con 0x55b109b24020 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975659c640 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109b24020 0x55b109b24400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f97357fa640 10 client.8934 ms_handle_connect on v2:172.21.15.94:6833/1613596341 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.119+0000 7f975659c640 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109b24020 0x55b109b24400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea6c6814bc9c5605 server_cookie=264fc71065ed6538 in_seq=0 out_seq=0 2024-03-06T07:24:13.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f97357fa640 1 -- 172.21.15.94:0/3303825859 <== mds.0 v2:172.21.15.94:6833/1613596341 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55b1099e0f40 con 0x55b109b24020 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f97357fa640 10 client.8934 handle_command_reply: tid=0 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 client.8934 shutdown 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 msgr2=0x55b109b20a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109b20a30 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f974c005620 tx=0x7f974c0012d0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 shutdown_connections 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b109a97d30 0x55b109b1dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55b109a75330 0x55b109b20a30 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.94:6833/1613596341,v1:172.21.15.94:6835/1613596341] conn(0x55b109b24020 0x55b109b24400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 --2- 172.21.15.94:0/3303825859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b109a74f50 0x55b109a90d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f97357fa640 0 client.8934 ms_handle_reset on v2:172.21.15.94:6833/1613596341 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 >> 172.21.15.94:0/3303825859 conn(0x55b109a734b0 msgr2=0x55b109ad5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 shutdown_connections 2024-03-06T07:24:13.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 1 -- 172.21.15.94:0/3303825859 wait complete. 2024-03-06T07:24:13.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975fbf0740 20 client.8934 trim_cache size 0 max 16384 2024-03-06T07:24:13.129 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:24:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:24:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:24:13.129 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 8772, 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 9.2490577159999994, 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 135, 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 149, 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 149, 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 14.355086961 2024-03-06T07:24:13.130 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f973806b4d0 msgr2=0x7f973806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975cf32640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f973806b4d0 0x7f973806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f974c034000 tx=0x7f974c031000 comp rx=0 tx=0).stop 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 msgr2=0x7f97581404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.120+0000 7f975cf32640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f97581404e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f9750004e00 tx=0x7f9750007430 comp rx=0 tx=0).stop 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 shutdown_connections 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f973806b4d0 0x7f973806d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f975813e230 0x7f975814ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9758145cc0 0x7f97581444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 --2- 172.21.15.94:0/3675129569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9740004800 0x7f97581404e0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 >> 172.21.15.94:0/3675129569 conn(0x7f975806b1a0 msgr2=0x7f97580c4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 shutdown_connections 2024-03-06T07:24:13.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.121+0000 7f975cf32640 1 -- 172.21.15.94:0/3675129569 wait complete. 2024-03-06T07:24:13.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:13.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.236+0000 7ff52194b640 1 Processor -- start 2024-03-06T07:24:13.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.236+0000 7ff52194b640 1 -- start start 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.236+0000 7ff52194b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c1527b0 0x7ff51c159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.236+0000 7ff52194b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c0c7c60 con 0x7ff51c0c7180 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.236+0000 7ff52194b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c0c8060 con 0x7ff51c0c88b0 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.236+0000 7ff52194b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c159640 con 0x7ff51c1527b0 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff51a7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff51c0c7180 0x7ff51c155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47718/0 (socket says 172.21.15.94:47718) 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff51a7fc640 1 -- 172.21.15.94:0/922082973 learned_addr learned my addr 172.21.15.94:0/922082973 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff51affd640 1 --2- 172.21.15.94:0/922082973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c1527b0 0x7ff51c159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff519ffb640 1 -- 172.21.15.94:0/922082973 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985277922 0 0) 0x7ff510001110 con 0x7ff51c0c7180 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff519ffb640 1 -- 172.21.15.94:0/922082973 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff504003520 con 0x7ff51c0c7180 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff519ffb640 1 -- 172.21.15.94:0/922082973 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542008949 0 0) 0x7ff51c0c8060 con 0x7ff51c0c88b0 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff519ffb640 1 -- 172.21.15.94:0/922082973 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff510001110 con 0x7ff51c0c88b0 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff51affd640 1 -- 172.21.15.94:0/922082973 >> v1:172.21.15.145:6789/0 conn(0x7ff51c0c88b0 legacy=0x7ff51c0c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff51affd640 1 -- 172.21.15.94:0/922082973 >> v1:172.21.15.94:6789/0 conn(0x7ff51c0c7180 legacy=0x7ff51c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.237+0000 7ff51affd640 1 -- 172.21.15.94:0/922082973 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c0c76a0 con 0x7ff51c1527b0 2024-03-06T07:24:13.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.238+0000 7ff51affd640 1 --2- 172.21.15.94:0/922082973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c1527b0 0x7ff51c159100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0097d0 tx=0x7ff50c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a23973804ac1d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.238+0000 7ff519ffb640 1 -- 172.21.15.94:0/922082973 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c035020 con 0x7ff51c1527b0 2024-03-06T07:24:13.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.238+0000 7ff519ffb640 1 -- 172.21.15.94:0/922082973 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c031800 con 0x7ff51c1527b0 2024-03-06T07:24:13.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.238+0000 7ff519ffb640 1 -- 172.21.15.94:0/922082973 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c031bd0 con 0x7ff51c1527b0 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.238+0000 7ff52194b640 1 -- 172.21.15.94:0/922082973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c1527b0 msgr2=0x7ff51c159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.238+0000 7ff52194b640 1 --2- 172.21.15.94:0/922082973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c1527b0 0x7ff51c159100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0097d0 tx=0x7ff50c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.239+0000 7ff52194b640 1 -- 172.21.15.94:0/922082973 shutdown_connections 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.239+0000 7ff52194b640 1 --2- 172.21.15.94:0/922082973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c1527b0 0x7ff51c159100 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.239+0000 7ff52194b640 1 -- 172.21.15.94:0/922082973 >> 172.21.15.94:0/922082973 conn(0x7ff51c0ac560 msgr2=0x7ff51c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.239+0000 7ff52194b640 1 -- 172.21.15.94:0/922082973 shutdown_connections 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.239+0000 7ff52194b640 1 -- 172.21.15.94:0/922082973 wait complete. 2024-03-06T07:24:13.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 Processor -- start 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 -- start start 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51c0c7180 0x7ff51c1eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c0c88b0 0x7ff51c14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51c1527b0 0x7ff51c0c2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c155600 con 0x7ff51c0c7180 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c155320 con 0x7ff51c1527b0 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.240+0000 7ff52194b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0c8d60 con 0x7ff51c0c88b0 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51c0c7180 0x7ff51c1eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51c0c7180 0x7ff51c1eedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43192/0 (socket says 172.21.15.94:43192) 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 -- 172.21.15.94:0/2991462933 learned_addr learned my addr 172.21.15.94:0/2991462933 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51a7fc640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c0c88b0 0x7ff51c14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51b7fe640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51c1527b0 0x7ff51c0c2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 -- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c0c88b0 msgr2=0x7ff51c14de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c0c88b0 0x7ff51c14de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 -- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51c1527b0 msgr2=0x7ff51c0c2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51c1527b0 0x7ff51c0c2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 -- 172.21.15.94:0/2991462933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c154010 con 0x7ff51c0c7180 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff51affd640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51c0c7180 0x7ff51c1eedd0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0098f0 tx=0x7ff50c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff503fff640 1 -- 172.21.15.94:0/2991462933 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c047070 con 0x7ff51c0c7180 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff503fff640 1 -- 172.21.15.94:0/2991462933 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c03ad60 con 0x7ff51c0c7180 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c1542f0 con 0x7ff51c0c7180 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff503fff640 1 -- 172.21.15.94:0/2991462933 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c042650 con 0x7ff51c0c7180 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.241+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c0c1d60 con 0x7ff51c0c7180 2024-03-06T07:24:13.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.242+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c04f330 con 0x7ff51c0c7180 2024-03-06T07:24:13.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.243+0000 7ff503fff640 1 -- 172.21.15.94:0/2991462933 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff50c04a070 con 0x7ff51c0c7180 2024-03-06T07:24:13.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.243+0000 7ff503fff640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff50806b4d0 0x7ff50806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.243+0000 7ff503fff640 1 -- 172.21.15.94:0/2991462933 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff50c0abec0 con 0x7ff51c0c7180 2024-03-06T07:24:13.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.243+0000 7ff51a7fc640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff50806b4d0 0x7ff50806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.244+0000 7ff51a7fc640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff50806b4d0 0x7ff50806d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff51c14eec0 tx=0x7ff510007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.247+0000 7ff503fff640 1 -- 172.21.15.94:0/2991462933 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff50c03f070 con 0x7ff51c0c7180 2024-03-06T07:24:13.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.538+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff51c06ce00 con 0x7ff51c0c7180 2024-03-06T07:24:13.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.539+0000 7ff503fff640 1 -- 172.21.15.94:0/2991462933 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16884 (secure 0 0 0) 0x7ff50c07a560 con 0x7ff51c0c7180 2024-03-06T07:24:13.541 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:13.541 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":149,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:04.871334+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:13.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff50806b4d0 msgr2=0x7ff50806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff50806b4d0 0x7ff50806d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff51c14eec0 tx=0x7ff510007d70 comp rx=0 tx=0).stop 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51c0c7180 msgr2=0x7ff51c1eedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51c0c7180 0x7ff51c1eedd0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0098f0 tx=0x7ff50c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 shutdown_connections 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff50806b4d0 0x7ff50806d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51c1527b0 0x7ff51c0c2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51c0c88b0 0x7ff51c14de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 --2- 172.21.15.94:0/2991462933 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff51c0c7180 0x7ff51c1eedd0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 >> 172.21.15.94:0/2991462933 conn(0x7ff51c0ac560 msgr2=0x7ff51c0c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 shutdown_connections 2024-03-06T07:24:13.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.541+0000 7ff52194b640 1 -- 172.21.15.94:0/2991462933 wait complete. 2024-03-06T07:24:13.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:01.443144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:02.192757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:13.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.658+0000 7f4a6544e640 1 Processor -- start 2024-03-06T07:24:13.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.658+0000 7f4a6544e640 1 -- start start 2024-03-06T07:24:13.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a6544e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 0x7f4a601539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a6544e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a600c62a0 con 0x7f4a600c69a0 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a6544e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a600c4c00 con 0x7f4a600a8290 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a6544e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60153f00 con 0x7f4a600c4740 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a5f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4a600c69a0 0x7f4a60072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47726/0 (socket says 172.21.15.94:47726) 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a5f7fe640 1 -- 172.21.15.94:0/4220218050 learned_addr learned my addr 172.21.15.94:0/4220218050 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a5e7fc640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 0x7f4a601539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444985702 0 0) 0x7f4a600c62a0 con 0x7f4a600c69a0 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.659+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a48003520 con 0x7f4a600c69a0 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 845436719 0 0) 0x7f4a48003520 con 0x7f4a600c69a0 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 msgr2=0x7f4a601539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 0x7f4a601539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 >> v1:172.21.15.145:6789/0 conn(0x7f4a600a8290 legacy=0x7f4a600701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a600c4ee0 con 0x7f4a600c69a0 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f4a50001670 con 0x7f4a600c69a0 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 >> v1:172.21.15.94:6789/0 conn(0x7f4a600c69a0 legacy=0x7f4a60072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a480039e0 0x7f4a48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 0x7f4a48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a48004d40 0x7f4a480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60153f00 con 0x7f4a480039e0 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4800bb80 con 0x7f4a48004d40 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4800bd80 con 0x7f4a600c4740 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5effd640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a480039e0 0x7f4a48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5e7fc640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 0x7f4a48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5f7fe640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a48004d40 0x7f4a480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5effd640 1 -- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 msgr2=0x7f4a48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5effd640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 0x7f4a48004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5effd640 1 -- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a48004d40 msgr2=0x7f4a480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5effd640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a48004d40 0x7f4a480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5effd640 1 -- 172.21.15.94:0/4220218050 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a48003520 con 0x7f4a480039e0 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.660+0000 7f4a5e7fc640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600c4740 0x7f4a48004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.661+0000 7f4a5effd640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a480039e0 0x7f4a48003e00 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f4a58001e50 tx=0x7f4a48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7363410f28a914 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.661+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58001a70 con 0x7f4a480039e0 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.661+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a58003b70 con 0x7f4a480039e0 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.661+0000 7f4a5dffb640 1 -- 172.21.15.94:0/4220218050 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a580050d0 con 0x7f4a480039e0 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.662+0000 7f4a6544e640 1 -- 172.21.15.94:0/4220218050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a480039e0 msgr2=0x7f4a48003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.662+0000 7f4a6544e640 1 --2- 172.21.15.94:0/4220218050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a480039e0 0x7f4a48003e00 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f4a58001e50 tx=0x7f4a48005830 comp rx=0 tx=0).stop 2024-03-06T07:24:13.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.662+0000 7f4a5f7fe640 1 -- 172.21.15.94:0/4220218050 reap_dead start 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.662+0000 7f4a6544e640 1 -- 172.21.15.94:0/4220218050 shutdown_connections 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.662+0000 7f4a6544e640 1 -- 172.21.15.94:0/4220218050 >> 172.21.15.94:0/4220218050 conn(0x7f4a600c5560 msgr2=0x7f4a600c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.662+0000 7f4a6544e640 1 -- 172.21.15.94:0/4220218050 shutdown_connections 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.662+0000 7f4a6544e640 1 -- 172.21.15.94:0/4220218050 wait complete. 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.663+0000 7f4a6544e640 1 Processor -- start 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.663+0000 7f4a6544e640 1 -- start start 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.663+0000 7f4a6544e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a480039e0 0x7f4a6014d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a6544e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a6014dd70 0x7f4a600a8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a6544e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600a8fd0 0x7f4a601ee780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a6544e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a600c6e50 con 0x7f4a6014dd70 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a6544e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6006fc80 con 0x7f4a480039e0 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a6544e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60072d90 con 0x7f4a600a8fd0 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a6014dd70 0x7f4a600a8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a6014dd70 0x7f4a600a8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43202/0 (socket says 172.21.15.94:43202) 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 -- 172.21.15.94:0/2527953599 learned_addr learned my addr 172.21.15.94:0/2527953599 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5f7fe640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600a8fd0 0x7f4a601ee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5effd640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a480039e0 0x7f4a6014d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 -- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600a8fd0 msgr2=0x7f4a601ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600a8fd0 0x7f4a601ee780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 -- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a480039e0 msgr2=0x7f4a6014d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a480039e0 0x7f4a6014d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 -- 172.21.15.94:0/2527953599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a60156950 con 0x7f4a6014dd70 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a5e7fc640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a6014dd70 0x7f4a600a8a90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f4a54011730 tx=0x7f4a54011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a477fe640 1 -- 172.21.15.94:0/2527953599 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5400f9c0 con 0x7f4a6014dd70 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a477fe640 1 -- 172.21.15.94:0/2527953599 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5400fb60 con 0x7f4a6014dd70 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a60154f30 con 0x7f4a6014dd70 2024-03-06T07:24:13.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a477fe640 1 -- 172.21.15.94:0/2527953599 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54015460 con 0x7f4a6014dd70 2024-03-06T07:24:13.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.664+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a60156ed0 con 0x7f4a6014dd70 2024-03-06T07:24:13.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.665+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6004f330 con 0x7f4a6014dd70 2024-03-06T07:24:13.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.666+0000 7f4a477fe640 1 -- 172.21.15.94:0/2527953599 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a54015600 con 0x7f4a6014dd70 2024-03-06T07:24:13.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.666+0000 7f4a477fe640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.666+0000 7f4a477fe640 1 -- 172.21.15.94:0/2527953599 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a5408ac30 con 0x7f4a6014dd70 2024-03-06T07:24:13.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.667+0000 7f4a5effd640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.667+0000 7f4a5effd640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4a58034000 tx=0x7f4a58009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.675+0000 7f4a477fe640 1 -- 172.21.15.94:0/2527953599 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a54019030 con 0x7f4a6014dd70 2024-03-06T07:24:13.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.850+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a600c8480 con 0x7f4a6014dd70 2024-03-06T07:24:13.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.851+0000 7f4a477fe640 1 -- 172.21.15.94:0/2527953599 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a540599b0 con 0x7f4a6014dd70 2024-03-06T07:24:13.853 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a4c06b4d0 msgr2=0x7f4a4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4a58034000 tx=0x7f4a58009030 comp rx=0 tx=0).stop 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a6014dd70 msgr2=0x7f4a600a8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a6014dd70 0x7f4a600a8a90 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f4a54011730 tx=0x7f4a54011bf0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 shutdown_connections 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a600a8fd0 0x7f4a601ee780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a6014dd70 0x7f4a600a8a90 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 --2- 172.21.15.94:0/2527953599 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a480039e0 0x7f4a6014d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 >> 172.21.15.94:0/2527953599 conn(0x7f4a600c5560 msgr2=0x7f4a60152420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 shutdown_connections 2024-03-06T07:24:13.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.853+0000 7f4a6544e640 1 -- 172.21.15.94:0/2527953599 wait complete. 2024-03-06T07:24:13.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:13.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.968+0000 7fd577471640 1 Processor -- start 2024-03-06T07:24:13.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.968+0000 7fd577471640 1 -- start start 2024-03-06T07:24:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd577471640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5700b0070 0x7fd570155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd577471640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5700c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd577471640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700b0450 con 0x7fd57006ca70 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd577471640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57006c020 con 0x7fd5700b0070 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd577471640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57006cf20 con 0x7fd5700c5380 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd57520d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5700b0070 0x7fd570155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd57520d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5700b0070 0x7fd570155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43210/0 (socket says 172.21.15.94:43210) 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd57520d640 1 -- 172.21.15.94:0/2104255246 learned_addr learned my addr 172.21.15.94:0/2104255246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.969+0000 7fd575a0e640 1 --2- 172.21.15.94:0/2104255246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5700c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd57520d640 1 -- 172.21.15.94:0/2104255246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 msgr2=0x7fd5700c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd57520d640 1 --2- 172.21.15.94:0/2104255246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5700c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd57520d640 1 -- 172.21.15.94:0/2104255246 >> v1:172.21.15.145:6789/0 conn(0x7fd57006ca70 legacy=0x7fd5700c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd57520d640 1 -- 172.21.15.94:0/2104255246 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700c77f0 con 0x7fd5700b0070 2024-03-06T07:24:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd57520d640 1 --2- 172.21.15.94:0/2104255246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd570155c30 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fd56c009e50 tx=0x7fd56c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9d19513d60744d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd567fff640 1 -- 172.21.15.94:0/2104255246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c02c9e0 con 0x7fd5700b0070 2024-03-06T07:24:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd567fff640 1 -- 172.21.15.94:0/2104255246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c02cb80 con 0x7fd5700b0070 2024-03-06T07:24:13.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.970+0000 7fd567fff640 1 -- 172.21.15.94:0/2104255246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0343f0 con 0x7fd5700b0070 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 -- 172.21.15.94:0/2104255246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 msgr2=0x7fd570155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 --2- 172.21.15.94:0/2104255246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd570155c30 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fd56c009e50 tx=0x7fd56c02be40 comp rx=0 tx=0).stop 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 -- 172.21.15.94:0/2104255246 shutdown_connections 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 --2- 172.21.15.94:0/2104255246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5700c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 --2- 172.21.15.94:0/2104255246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd570155c30 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 -- 172.21.15.94:0/2104255246 >> 172.21.15.94:0/2104255246 conn(0x7fd5700ac590 msgr2=0x7fd5700ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 -- 172.21.15.94:0/2104255246 shutdown_connections 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.971+0000 7fd577471640 1 -- 172.21.15.94:0/2104255246 wait complete. 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.973+0000 7fd577471640 1 Processor -- start 2024-03-06T07:24:13.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.973+0000 7fd577471640 1 -- start start 2024-03-06T07:24:13.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd577471640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57006ca70 0x7fd5701546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd577471640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd5701eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd577471640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5701eb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd577471640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57006c550 con 0x7fd5700b0070 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd577471640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57006c270 con 0x7fd57006ca70 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd577471640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57006da80 con 0x7fd5700c5380 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd5701eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd5701eb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43216/0 (socket says 172.21.15.94:43216) 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 -- 172.21.15.94:0/1863163122 learned_addr learned my addr 172.21.15.94:0/1863163122 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd575a0e640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5701eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd57520d640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57006ca70 0x7fd5701546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 -- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 msgr2=0x7fd5701eb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5701eb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 -- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57006ca70 msgr2=0x7fd5701546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57006ca70 0x7fd5701546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 -- 172.21.15.94:0/1863163122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5701ee980 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd574a0c640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd5701eb450 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd56000b3f0 tx=0x7fd56000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd565ffb640 1 -- 172.21.15.94:0/1863163122 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560015020 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd565ffb640 1 -- 172.21.15.94:0/1863163122 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd560011830 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5701eec60 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.974+0000 7fd565ffb640 1 -- 172.21.15.94:0/1863163122 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560011c20 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.975+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5701ef220 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.976+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5700ae020 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.976+0000 7fd565ffb640 1 -- 172.21.15.94:0/1863163122 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd560018020 con 0x7fd5700b0070 2024-03-06T07:24:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.977+0000 7fd565ffb640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd55406b4d0 0x7fd55406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.977+0000 7fd565ffb640 1 -- 172.21.15.94:0/1863163122 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd560089e20 con 0x7fd5700b0070 2024-03-06T07:24:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.977+0000 7fd57520d640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd55406b4d0 0x7fd55406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.977+0000 7fd57520d640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd55406b4d0 0x7fd55406d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd56c035000 tx=0x7fd56c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:13.981+0000 7fd565ffb640 1 -- 172.21.15.94:0/1863163122 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5700ae020 con 0x7fd5700b0070 2024-03-06T07:24:14.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.152+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd57004f330 con 0x7fd5700b0070 2024-03-06T07:24:14.154 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.153+0000 7fd565ffb640 1 -- 172.21.15.94:0/1863163122 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd56001f030 con 0x7fd5700b0070 2024-03-06T07:24:14.154 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.154+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd55406b4d0 msgr2=0x7fd55406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.154+0000 7fd577471640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd55406b4d0 0x7fd55406d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd56c035000 tx=0x7fd56c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 msgr2=0x7fd5701eb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd5701eb450 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd56000b3f0 tx=0x7fd56000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 shutdown_connections 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd55406b4d0 0x7fd55406d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5700c5380 0x7fd5701eb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5700b0070 0x7fd5701eb450 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 --2- 172.21.15.94:0/1863163122 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd57006ca70 0x7fd5701546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 >> 172.21.15.94:0/1863163122 conn(0x7fd5700ac590 msgr2=0x7fd5701515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 shutdown_connections 2024-03-06T07:24:14.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.155+0000 7fd577471640 1 -- 172.21.15.94:0/1863163122 wait complete. 2024-03-06T07:24:14.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:14.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.269+0000 7f161c543640 1 Processor -- start 2024-03-06T07:24:14.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.269+0000 7f161c543640 1 -- start start 2024-03-06T07:24:14.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161c543640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b0070 0x7f1614155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161c543640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f16140c5380 0x7f16140c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161c543640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16140b0450 con 0x7f161406ca70 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161c543640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f161406c020 con 0x7f16140c5380 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161c543640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16140c77f0 con 0x7f16140b0070 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f1619ade640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f161406ca70 0x7f16140c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47742/0 (socket says 172.21.15.94:47742) 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f1619ade640 1 -- 172.21.15.94:0/394167658 learned_addr learned my addr 172.21.15.94:0/394167658 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161aae0640 1 --2- 172.21.15.94:0/394167658 >> v2:172.21.15.94:3300/0 conn(0x7f16140c5380 0x7f16140c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161a2df640 1 --2- 172.21.15.94:0/394167658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b0070 0x7f1614155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f16192dd640 1 -- 172.21.15.94:0/394167658 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722351503 0 0) 0x7f1604001110 con 0x7f161406ca70 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f16192dd640 1 -- 172.21.15.94:0/394167658 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1600003520 con 0x7f161406ca70 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161aae0640 1 -- 172.21.15.94:0/394167658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b0070 msgr2=0x7f1614155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161aae0640 1 --2- 172.21.15.94:0/394167658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b0070 0x7f1614155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161aae0640 1 -- 172.21.15.94:0/394167658 >> v1:172.21.15.94:6789/0 conn(0x7f161406ca70 legacy=0x7f16140c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.271+0000 7f161aae0640 1 -- 172.21.15.94:0/394167658 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16140c7ad0 con 0x7f16140c5380 2024-03-06T07:24:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.272+0000 7f161aae0640 1 --2- 172.21.15.94:0/394167658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16140c5380 0x7f16140c57d0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f160c009d90 tx=0x7f160c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bdde107d854a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.272+0000 7f16192dd640 1 -- 172.21.15.94:0/394167658 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c039070 con 0x7f16140c5380 2024-03-06T07:24:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.272+0000 7f16192dd640 1 -- 172.21.15.94:0/394167658 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160c02cbc0 con 0x7f16140c5380 2024-03-06T07:24:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.272+0000 7f16192dd640 1 -- 172.21.15.94:0/394167658 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160c034440 con 0x7f16140c5380 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 -- 172.21.15.94:0/394167658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16140c5380 msgr2=0x7f16140c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 --2- 172.21.15.94:0/394167658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16140c5380 0x7f16140c57d0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f160c009d90 tx=0x7f160c02be10 comp rx=0 tx=0).stop 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 -- 172.21.15.94:0/394167658 shutdown_connections 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 --2- 172.21.15.94:0/394167658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16140c5380 0x7f16140c57d0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 --2- 172.21.15.94:0/394167658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b0070 0x7f1614155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 -- 172.21.15.94:0/394167658 >> 172.21.15.94:0/394167658 conn(0x7f16140ac590 msgr2=0x7f16140ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 -- 172.21.15.94:0/394167658 shutdown_connections 2024-03-06T07:24:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.273+0000 7f161c543640 1 -- 172.21.15.94:0/394167658 wait complete. 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 Processor -- start 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 -- start start 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f161406ca70 0x7f16141ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b0070 0x7f16141eece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140c5380 0x7f16140c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161406cef0 con 0x7f161406ca70 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161406c550 con 0x7f16140b0070 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.274+0000 7f161c543640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161406da80 con 0x7f16140c5380 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f161406ca70 0x7f16141ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f161406ca70 0x7f16141ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43242/0 (socket says 172.21.15.94:43242) 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 -- 172.21.15.94:0/4198621947 learned_addr learned my addr 172.21.15.94:0/4198621947 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:14.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161aae0640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140c5380 0x7f16140c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f1619ade640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b0070 0x7f16141eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 -- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140c5380 msgr2=0x7f16140c4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140c5380 0x7f16140c4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 -- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b0070 msgr2=0x7f16141eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b0070 0x7f16141eece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 -- 172.21.15.94:0/4198621947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16141ebcf0 con 0x7f161406ca70 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161aae0640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140c5380 0x7f16140c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161a2df640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f161406ca70 0x7f16141ead60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f161000d800 tx=0x7f161000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f160affd640 1 -- 172.21.15.94:0/4198621947 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161000b6c0 con 0x7f161406ca70 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f160affd640 1 -- 172.21.15.94:0/4198621947 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1610019070 con 0x7f161406ca70 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1614155100 con 0x7f161406ca70 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f160affd640 1 -- 172.21.15.94:0/4198621947 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161000ba80 con 0x7f161406ca70 2024-03-06T07:24:14.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.275+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16140c39e0 con 0x7f161406ca70 2024-03-06T07:24:14.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.276+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16140c3c20 con 0x7f161406ca70 2024-03-06T07:24:14.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.277+0000 7f160affd640 1 -- 172.21.15.94:0/4198621947 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f161000bc20 con 0x7f161406ca70 2024-03-06T07:24:14.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.277+0000 7f160affd640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15f806b4d0 0x7f15f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.277+0000 7f160affd640 1 -- 172.21.15.94:0/4198621947 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f161008a7b0 con 0x7f161406ca70 2024-03-06T07:24:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.278+0000 7f1619ade640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15f806b4d0 0x7f15f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.278+0000 7f1619ade640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15f806b4d0 0x7f15f806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f16040012b0 tx=0x7f1604009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.282+0000 7f160affd640 1 -- 172.21.15.94:0/4198621947 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16140c3c20 con 0x7f161406ca70 2024-03-06T07:24:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.458+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f161404f330 con 0x7f161406ca70 2024-03-06T07:24:14.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.459+0000 7f160affd640 1 -- 172.21.15.94:0/4198621947 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1610059530 con 0x7f161406ca70 2024-03-06T07:24:14.461 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.460+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15f806b4d0 msgr2=0x7f15f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15f806b4d0 0x7f15f806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f16040012b0 tx=0x7f1604009040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f161406ca70 msgr2=0x7f16141ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f161406ca70 0x7f16141ead60 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f161000d800 tx=0x7f161000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 shutdown_connections 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15f806b4d0 0x7f15f806d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140c5380 0x7f16140c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b0070 0x7f16141eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 --2- 172.21.15.94:0/4198621947 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f161406ca70 0x7f16141ead60 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 >> 172.21.15.94:0/4198621947 conn(0x7f16140ac590 msgr2=0x7f1614151230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 shutdown_connections 2024-03-06T07:24:14.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.461+0000 7f161c543640 1 -- 172.21.15.94:0/4198621947 wait complete. 2024-03-06T07:24:14.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:14.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.572+0000 7fcc77a39640 1 Processor -- start 2024-03-06T07:24:14.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.572+0000 7fcc77a39640 1 -- start start 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.573+0000 7fcc77a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc7014f650 0x7fcc70071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.573+0000 7fcc77a39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcc70152c20 0x7fcc7015c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.573+0000 7fcc77a39640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc70072440 con 0x7fcc700c3150 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.573+0000 7fcc77a39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc700c2b10 con 0x7fcc70152c20 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.573+0000 7fcc77a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc70152140 con 0x7fcc7014f650 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc757d5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc7014f650 0x7fcc70071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc75fd6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcc70152c20 0x7fcc7015c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc757d5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc7014f650 0x7fcc70071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:42610/0 (socket says 172.21.15.94:42610) 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc74fd4640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fcc700c3150 0x7fcc700c5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:57470/0 (socket says 172.21.15.94:57470) 2024-03-06T07:24:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc757d5640 1 -- 172.21.15.94:0/1523837582 learned_addr learned my addr 172.21.15.94:0/1523837582 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc757d5640 1 -- 172.21.15.94:0/1523837582 >> v2:172.21.15.145:3300/0 conn(0x7fcc70152c20 msgr2=0x7fcc7015c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc757d5640 1 --2- 172.21.15.94:0/1523837582 >> v2:172.21.15.145:3300/0 conn(0x7fcc70152c20 0x7fcc7015c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc757d5640 1 -- 172.21.15.94:0/1523837582 >> v1:172.21.15.145:6789/0 conn(0x7fcc700c3150 legacy=0x7fcc700c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc757d5640 1 -- 172.21.15.94:0/1523837582 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc70152420 con 0x7fcc7014f650 2024-03-06T07:24:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.574+0000 7fcc75fd6640 1 --2- 172.21.15.94:0/1523837582 >> v2:172.21.15.145:3300/0 conn(0x7fcc70152c20 0x7fcc7015c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc757d5640 1 --2- 172.21.15.94:0/1523837582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc7014f650 0x7fcc70071880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009870 tx=0x7fcc6002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40099b9ded69dc4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc67fff640 1 -- 172.21.15.94:0/1523837582 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60034070 con 0x7fcc7014f650 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc67fff640 1 -- 172.21.15.94:0/1523837582 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc60030840 con 0x7fcc7014f650 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc67fff640 1 -- 172.21.15.94:0/1523837582 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60030c10 con 0x7fcc7014f650 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc77a39640 1 -- 172.21.15.94:0/1523837582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc7014f650 msgr2=0x7fcc70071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1523837582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc7014f650 0x7fcc70071880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009870 tx=0x7fcc6002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc77a39640 1 -- 172.21.15.94:0/1523837582 shutdown_connections 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1523837582 >> v2:172.21.15.145:3300/0 conn(0x7fcc70152c20 0x7fcc7015c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1523837582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc7014f650 0x7fcc70071880 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc77a39640 1 -- 172.21.15.94:0/1523837582 >> 172.21.15.94:0/1523837582 conn(0x7fcc700c4290 msgr2=0x7fcc700c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.575+0000 7fcc77a39640 1 -- 172.21.15.94:0/1523837582 shutdown_connections 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.576+0000 7fcc77a39640 1 -- 172.21.15.94:0/1523837582 wait complete. 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 Processor -- start 2024-03-06T07:24:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 -- start start 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc700c3150 0x7fcc700ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc7014f650 0x7fcc701557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc70152c20 0x7fcc701ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc700c40c0 con 0x7fcc70152c20 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc700c3f40 con 0x7fcc7014f650 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc77a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc700c5da0 con 0x7fcc700c3150 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc75fd6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc70152c20 0x7fcc701ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc75fd6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc70152c20 0x7fcc701ee4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43254/0 (socket says 172.21.15.94:43254) 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc75fd6640 1 -- 172.21.15.94:0/1259238100 learned_addr learned my addr 172.21.15.94:0/1259238100 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc757d5640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc700c3150 0x7fcc700ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.577+0000 7fcc74fd4640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc7014f650 0x7fcc701557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc75fd6640 1 -- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc700c3150 msgr2=0x7fcc700ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc75fd6640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc700c3150 0x7fcc700ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc75fd6640 1 -- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc7014f650 msgr2=0x7fcc701557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc75fd6640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc7014f650 0x7fcc701557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc75fd6640 1 -- 172.21.15.94:0/1259238100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc701eed00 con 0x7fcc70152c20 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc74fd4640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc7014f650 0x7fcc701557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc75fd6640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc70152c20 0x7fcc701ee4c0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009e10 tx=0x7fcc6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc65ffb640 1 -- 172.21.15.94:0/1259238100 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68018020 con 0x7fcc70152c20 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc65ffb640 1 -- 172.21.15.94:0/1259238100 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc68014480 con 0x7fcc70152c20 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc701eb900 con 0x7fcc70152c20 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc65ffb640 1 -- 172.21.15.94:0/1259238100 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc68014850 con 0x7fcc70152c20 2024-03-06T07:24:14.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.578+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc701ebbe0 con 0x7fcc70152c20 2024-03-06T07:24:14.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.579+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7004f330 con 0x7fcc70152c20 2024-03-06T07:24:14.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.580+0000 7fcc65ffb640 1 -- 172.21.15.94:0/1259238100 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc68005d10 con 0x7fcc70152c20 2024-03-06T07:24:14.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.580+0000 7fcc65ffb640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc5406b500 0x7fcc5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.580+0000 7fcc65ffb640 1 -- 172.21.15.94:0/1259238100 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcc6808afd0 con 0x7fcc70152c20 2024-03-06T07:24:14.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.580+0000 7fcc757d5640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc5406b500 0x7fcc5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.581+0000 7fcc757d5640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc5406b500 0x7fcc5406d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcc60036000 tx=0x7fcc60008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.584+0000 7fcc65ffb640 1 -- 172.21.15.94:0/1259238100 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc68022030 con 0x7fcc70152c20 2024-03-06T07:24:14.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.765+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcc70068460 con 0x7fcc70152c20 2024-03-06T07:24:14.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.765+0000 7fcc65ffb640 1 -- 172.21.15.94:0/1259238100 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v149) v1 ==== 84+0+209 (secure 0 0 0) 0x7fcc68010040 con 0x7fcc70152c20 2024-03-06T07:24:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:14.767 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:24:14.767 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 23, 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": "cephfs_metadata" 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 24, 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": "cephfs_data" 2024-03-06T07:24:14.768 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:14.769 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc5406b500 msgr2=0x7fcc5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc5406b500 0x7fcc5406d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcc60036000 tx=0x7fcc60008030 comp rx=0 tx=0).stop 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc70152c20 msgr2=0x7fcc701ee4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc70152c20 0x7fcc701ee4c0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009e10 tx=0x7fcc6800c490 comp rx=0 tx=0).stop 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 shutdown_connections 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc5406b500 0x7fcc5406d9c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc70152c20 0x7fcc701ee4c0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc7014f650 0x7fcc701557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 --2- 172.21.15.94:0/1259238100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc700c3150 0x7fcc700ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 >> 172.21.15.94:0/1259238100 conn(0x7fcc700c4290 msgr2=0x7fcc700c6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 shutdown_connections 2024-03-06T07:24:14.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.768+0000 7fcc77a39640 1 -- 172.21.15.94:0/1259238100 wait complete. 2024-03-06T07:24:14.780 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-03-06T07:24:14.780 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-03-06T07:24:14.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.884+0000 7f617deed640 1 Processor -- start 2024-03-06T07:24:14.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.884+0000 7f617deed640 1 -- start start 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f617deed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6178072fc0 0x7f61780724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f617deed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61780c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f617deed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61780c6220 con 0x7f61780c5e40 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f617deed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6178070210 con 0x7f6178072fc0 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f617deed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61780704d0 con 0x7f6178072a10 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f616effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6178072fc0 0x7f61780724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f61777fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f61780c5e40 0x7f6178153940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47756/0 (socket says 172.21.15.94:47756) 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f61777fe640 1 -- 172.21.15.94:0/216215555 learned_addr learned my addr 172.21.15.94:0/216215555 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:14.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f616effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6178072fc0 0x7f61780724d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43268/0 (socket says 172.21.15.94:43268) 2024-03-06T07:24:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f6177fff640 1 --2- 172.21.15.94:0/216215555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61780c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f616effd640 1 -- 172.21.15.94:0/216215555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 msgr2=0x7f61780c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f616effd640 1 --2- 172.21.15.94:0/216215555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61780c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f616effd640 1 -- 172.21.15.94:0/216215555 >> v1:172.21.15.94:6789/0 conn(0x7f61780c5e40 legacy=0x7f6178153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f616effd640 1 -- 172.21.15.94:0/216215555 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617806f610 con 0x7f6178072fc0 2024-03-06T07:24:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f6176ffd640 1 -- 172.21.15.94:0/216215555 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768878345 0 0) 0x7f61780c6220 con 0x7f61780c5e40 2024-03-06T07:24:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.886+0000 7f6177fff640 1 --2- 172.21.15.94:0/216215555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61780c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:14.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.887+0000 7f616effd640 1 --2- 172.21.15.94:0/216215555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f61780724d0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f61600097a0 tx=0x7f616002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b270d876d3663ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.887+0000 7f6176ffd640 1 -- 172.21.15.94:0/216215555 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61600315a0 con 0x7f6178072fc0 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.887+0000 7f6176ffd640 1 -- 172.21.15.94:0/216215555 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6160037070 con 0x7f6178072fc0 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.887+0000 7f6176ffd640 1 -- 172.21.15.94:0/216215555 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160031960 con 0x7f6178072fc0 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 -- 172.21.15.94:0/216215555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 msgr2=0x7f61780724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 --2- 172.21.15.94:0/216215555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f61780724d0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f61600097a0 tx=0x7f616002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 -- 172.21.15.94:0/216215555 shutdown_connections 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 --2- 172.21.15.94:0/216215555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61780c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 --2- 172.21.15.94:0/216215555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f61780724d0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 -- 172.21.15.94:0/216215555 >> 172.21.15.94:0/216215555 conn(0x7f61780c88f0 msgr2=0x7f61780c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 -- 172.21.15.94:0/216215555 shutdown_connections 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.888+0000 7f617deed640 1 -- 172.21.15.94:0/216215555 wait complete. 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.889+0000 7f617deed640 1 Processor -- start 2024-03-06T07:24:14.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.889+0000 7f617deed640 1 -- start start 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f617deed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61781e92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f617deed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f617814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f617deed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61780c5e40 0x7f617814bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f617deed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61780c4c10 con 0x7f6178072fc0 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f617deed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617806fbd0 con 0x7f61780c5e40 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f617deed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61780c4d90 con 0x7f6178072a10 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f617814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f617814b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43284/0 (socket says 172.21.15.94:43284) 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 -- 172.21.15.94:0/593744317 learned_addr learned my addr 172.21.15.94:0/593744317 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f61777fe640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61781e92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f6177fff640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61780c5e40 0x7f617814bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 -- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 msgr2=0x7f61781e92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61781e92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 -- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61780c5e40 msgr2=0x7f617814bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61780c5e40 0x7f617814bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 -- 172.21.15.94:0/593744317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61780c0d60 con 0x7f6178072fc0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f61777fe640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61781e92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f616effd640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f617814b680 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f61600097a0 tx=0x7f61600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.890+0000 7f6174ff9640 1 -- 172.21.15.94:0/593744317 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160004230 con 0x7f6178072fc0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.891+0000 7f6174ff9640 1 -- 172.21.15.94:0/593744317 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61600043f0 con 0x7f6178072fc0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.891+0000 7f6174ff9640 1 -- 172.21.15.94:0/593744317 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160003850 con 0x7f6178072fc0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.891+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61780ac7e0 con 0x7f6178072fc0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.891+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61780be0e0 con 0x7f6178072fc0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.892+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617804f330 con 0x7f6178072fc0 2024-03-06T07:24:14.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.892+0000 7f6174ff9640 1 -- 172.21.15.94:0/593744317 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61600039f0 con 0x7f6178072fc0 2024-03-06T07:24:14.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.892+0000 7f6174ff9640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f615806b4d0 0x7f615806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.893+0000 7f6174ff9640 1 -- 172.21.15.94:0/593744317 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f616002f030 con 0x7f6178072fc0 2024-03-06T07:24:14.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.893+0000 7f61777fe640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f615806b4d0 0x7f615806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.893+0000 7f61777fe640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f615806b4d0 0x7f615806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6168000f60 tx=0x7f6168009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:14.897+0000 7f6174ff9640 1 -- 172.21.15.94:0/593744317 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6160041070 con 0x7f6178072fc0 2024-03-06T07:24:15.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.075+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f617806f3f0 con 0x7f6178072fc0 2024-03-06T07:24:15.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:15.184 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:15.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.230+0000 7f6174ff9640 1 -- 172.21.15.94:0/593744317 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v150) v1 ==== 140+0+0 (secure 0 0 0) 0x7f6160044340 con 0x7f6178072fc0 2024-03-06T07:24:15.232 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_metadata' created 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f615806b4d0 msgr2=0x7f615806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f615806b4d0 0x7f615806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6168000f60 tx=0x7f6168009040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 msgr2=0x7f617814b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f617814b680 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f61600097a0 tx=0x7f61600046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 shutdown_connections 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f615806b4d0 0x7f615806d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61780c5e40 0x7f617814bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6178072fc0 0x7f617814b680 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 --2- 172.21.15.94:0/593744317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6178072a10 0x7f61781e92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 >> 172.21.15.94:0/593744317 conn(0x7f61780c88f0 msgr2=0x7f61780a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 shutdown_connections 2024-03-06T07:24:15.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.233+0000 7f617deed640 1 -- 172.21.15.94:0/593744317 wait complete. 2024-03-06T07:24:15.250 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.356+0000 7f44795c7640 1 Processor -- start 2024-03-06T07:24:15.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.356+0000 7f44795c7640 1 -- start start 2024-03-06T07:24:15.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44795c7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4474072cd0 0x7f4474155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44795c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4474152db0 con 0x7f4474073290 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44795c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44741590f0 con 0x7f44741529d0 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44795c7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44740c6dd0 con 0x7f4474072cd0 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44727fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4474072cd0 0x7f4474155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44737fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4474073290 0x7f4474155660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47762/0 (socket says 172.21.15.94:47762) 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44737fe640 1 -- 172.21.15.94:0/4094608484 learned_addr learned my addr 172.21.15.94:0/4094608484 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44727fc640 1 -- 172.21.15.94:0/4094608484 >> v1:172.21.15.145:6789/0 conn(0x7f44741529d0 legacy=0x7f44740c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44727fc640 1 -- 172.21.15.94:0/4094608484 >> v1:172.21.15.94:6789/0 conn(0x7f4474073290 legacy=0x7f4474155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f44727fc640 1 -- 172.21.15.94:0/4094608484 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4474159650 con 0x7f4474072cd0 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.357+0000 7f4471ffb640 1 -- 172.21.15.94:0/4094608484 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749525420 0 0) 0x7f446c001490 con 0x7f4474073290 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f44727fc640 1 --2- 172.21.15.94:0/4094608484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4474072cd0 0x7f4474155120 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f445c0097a0 tx=0x7f445c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65fdae144b971f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f4471ffb640 1 -- 172.21.15.94:0/4094608484 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c038070 con 0x7f4474072cd0 2024-03-06T07:24:15.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f4471ffb640 1 -- 172.21.15.94:0/4094608484 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445c031700 con 0x7f4474072cd0 2024-03-06T07:24:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f4471ffb640 1 -- 172.21.15.94:0/4094608484 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c031ad0 con 0x7f4474072cd0 2024-03-06T07:24:15.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f44795c7640 1 -- 172.21.15.94:0/4094608484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4474072cd0 msgr2=0x7f4474155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f44795c7640 1 --2- 172.21.15.94:0/4094608484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4474072cd0 0x7f4474155120 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f445c0097a0 tx=0x7f445c02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f44795c7640 1 -- 172.21.15.94:0/4094608484 shutdown_connections 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f44795c7640 1 --2- 172.21.15.94:0/4094608484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4474072cd0 0x7f4474155120 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.358+0000 7f44795c7640 1 -- 172.21.15.94:0/4094608484 >> 172.21.15.94:0/4094608484 conn(0x7f44740aa630 msgr2=0x7f44740aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.359+0000 7f44795c7640 1 -- 172.21.15.94:0/4094608484 shutdown_connections 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.359+0000 7f44795c7640 1 -- 172.21.15.94:0/4094608484 wait complete. 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 Processor -- start 2024-03-06T07:24:15.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 -- start start 2024-03-06T07:24:15.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4474072cd0 0x7f447406e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4474073290 0x7f447414cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44741529d0 0x7f447414d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4474155d70 con 0x7f44741529d0 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740c6940 con 0x7f4474072cd0 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44795c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4474159370 con 0x7f4474073290 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44741529d0 0x7f447414d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.360+0000 7f44737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44741529d0 0x7f447414d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43300/0 (socket says 172.21.15.94:43300) 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44737fe640 1 -- 172.21.15.94:0/1517859880 learned_addr learned my addr 172.21.15.94:0/1517859880 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f4472ffd640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4474072cd0 0x7f447406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44727fc640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4474073290 0x7f447414cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44737fe640 1 -- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4474073290 msgr2=0x7f447414cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44737fe640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4474073290 0x7f447414cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44737fe640 1 -- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4474072cd0 msgr2=0x7f447406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44737fe640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4474072cd0 0x7f447406e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44737fe640 1 -- 172.21.15.94:0/1517859880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44741ef140 con 0x7f44741529d0 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44737fe640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44741529d0 0x7f447414d2b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f446c00c7b0 tx=0x7f446c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f446b7fe640 1 -- 172.21.15.94:0/1517859880 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c018070 con 0x7f44741529d0 2024-03-06T07:24:15.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f446b7fe640 1 -- 172.21.15.94:0/1517859880 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c00a640 con 0x7f44741529d0 2024-03-06T07:24:15.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f446b7fe640 1 -- 172.21.15.94:0/1517859880 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c00ce60 con 0x7f44741529d0 2024-03-06T07:24:15.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44741ef420 con 0x7f44741529d0 2024-03-06T07:24:15.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.361+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447406eb90 con 0x7f44741529d0 2024-03-06T07:24:15.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.362+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44740c5780 con 0x7f44741529d0 2024-03-06T07:24:15.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.363+0000 7f446b7fe640 1 -- 172.21.15.94:0/1517859880 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f446c01b070 con 0x7f44741529d0 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.363+0000 7f446b7fe640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f445806b4d0 0x7f445806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.364+0000 7f446b7fe640 1 -- 172.21.15.94:0/1517859880 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6472+0+0 (secure 0 0 0) 0x7f446c0899a0 con 0x7f44741529d0 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.364+0000 7f4472ffd640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f445806b4d0 0x7f445806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.364+0000 7f4472ffd640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f445806b4d0 0x7f445806d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4464006fd0 tx=0x7f4464009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.368+0000 7f446b7fe640 1 -- 172.21.15.94:0/1517859880 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4474152db0 con 0x7f44741529d0 2024-03-06T07:24:15.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:15.552+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f44740723b0 con 0x7f44741529d0 2024-03-06T07:24:16.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.232+0000 7f446b7fe640 1 -- 172.21.15.94:0/1517859880 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v151) v1 ==== 172+0+0 (secure 0 0 0) 0x7f44740723b0 con 0x7f44741529d0 2024-03-06T07:24:16.234 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_data' created 2024-03-06T07:24:16.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f445806b4d0 msgr2=0x7f445806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f445806b4d0 0x7f445806d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4464006fd0 tx=0x7f4464009060 comp rx=0 tx=0).stop 2024-03-06T07:24:16.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44741529d0 msgr2=0x7f447414d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44741529d0 0x7f447414d2b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f446c00c7b0 tx=0x7f446c00cc70 comp rx=0 tx=0).stop 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 shutdown_connections 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f445806b4d0 0x7f445806d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44741529d0 0x7f447414d2b0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4474073290 0x7f447414cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 --2- 172.21.15.94:0/1517859880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4474072cd0 0x7f447406e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 >> 172.21.15.94:0/1517859880 conn(0x7f44740aa630 msgr2=0x7f4474157e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 shutdown_connections 2024-03-06T07:24:16.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.235+0000 7f44795c7640 1 -- 172.21.15.94:0/1517859880 wait complete. 2024-03-06T07:24:16.262 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-03-06T07:24:16.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.405+0000 7fdd23e78640 1 Processor -- start 2024-03-06T07:24:16.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.406+0000 7fdd23e78640 1 -- start start 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd23e78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 0x7fdd1c0c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd23e78640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c155260 con 0x7fdd1c0c77d0 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd23e78640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c155540 con 0x7fdd1c0c7370 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd23e78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c7c00 con 0x7fdd1c1527f0 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd22415640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdd1c0c77d0 0x7fdd1c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47770/0 (socket says 172.21.15.94:47770) 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd22415640 1 -- 172.21.15.94:0/2252427416 learned_addr learned my addr 172.21.15.94:0/2252427416 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd21c14640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 0x7fdd1c0c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843219318 0 0) 0x7fdd1c155260 con 0x7fdd1c0c77d0 2024-03-06T07:24:16.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.407+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd08003520 con 0x7fdd1c0c77d0 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3186108040 0 0) 0x7fdd08003520 con 0x7fdd1c0c77d0 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 msgr2=0x7fdd1c0c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 0x7fdd1c0c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 >> v1:172.21.15.145:6789/0 conn(0x7fdd1c0c7370 legacy=0x7fdd1c06c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c155820 con 0x7fdd1c0c77d0 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fdd18001670 con 0x7fdd1c0c77d0 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 >> v1:172.21.15.94:6789/0 conn(0x7fdd1c0c77d0 legacy=0x7fdd1c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd080039e0 0x7fdd08003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 0x7fdd08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd08004d40 0x7fdd080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c7c00 con 0x7fdd080039e0 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0800bb80 con 0x7fdd08004d40 2024-03-06T07:24:16.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0800bd80 con 0x7fdd1c1527f0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd21c14640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd080039e0 0x7fdd08003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd21413640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 0x7fdd08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd22415640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd08004d40 0x7fdd080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd21c14640 1 -- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 msgr2=0x7fdd08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd21c14640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c1527f0 0x7fdd08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd21c14640 1 -- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd08004d40 msgr2=0x7fdd080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd21c14640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd08004d40 0x7fdd080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.408+0000 7fdd21c14640 1 -- 172.21.15.94:0/2252427416 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08003520 con 0x7fdd080039e0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.409+0000 7fdd21c14640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd080039e0 0x7fdd08003e00 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009fc0 tx=0x7fdd08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eca8ea78d8d95cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.409+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14039070 con 0x7fdd080039e0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.409+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1402cc20 con 0x7fdd080039e0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.409+0000 7fdd20c12640 1 -- 172.21.15.94:0/2252427416 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14009c10 con 0x7fdd080039e0 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.410+0000 7fdd23e78640 1 -- 172.21.15.94:0/2252427416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd080039e0 msgr2=0x7fdd08003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.410+0000 7fdd23e78640 1 --2- 172.21.15.94:0/2252427416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd080039e0 0x7fdd08003e00 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009fc0 tx=0x7fdd08005830 comp rx=0 tx=0).stop 2024-03-06T07:24:16.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.410+0000 7fdd22415640 1 -- 172.21.15.94:0/2252427416 reap_dead start 2024-03-06T07:24:16.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.410+0000 7fdd23e78640 1 -- 172.21.15.94:0/2252427416 shutdown_connections 2024-03-06T07:24:16.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.410+0000 7fdd23e78640 1 -- 172.21.15.94:0/2252427416 >> 172.21.15.94:0/2252427416 conn(0x7fdd1c0ac4f0 msgr2=0x7fdd1c0ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.410+0000 7fdd23e78640 1 -- 172.21.15.94:0/2252427416 shutdown_connections 2024-03-06T07:24:16.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.410+0000 7fdd23e78640 1 -- 172.21.15.94:0/2252427416 wait complete. 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 Processor -- start 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 -- start start 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd080039e0 0x7fdd1c0b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c0ad5e0 0x7fdd1c153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c0ada50 0x7fdd1c0adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c8860 con 0x7fdd1c0ada50 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0be1a0 con 0x7fdd080039e0 2024-03-06T07:24:16.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.411+0000 7fdd23e78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c89e0 con 0x7fdd1c0ad5e0 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c0ada50 0x7fdd1c0adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c0ada50 0x7fdd1c0adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43316/0 (socket says 172.21.15.94:43316) 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 -- 172.21.15.94:0/3885080597 learned_addr learned my addr 172.21.15.94:0/3885080597 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd21413640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c0ad5e0 0x7fdd1c153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd21c14640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd080039e0 0x7fdd1c0b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 -- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c0ad5e0 msgr2=0x7fdd1c153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c0ad5e0 0x7fdd1c153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 -- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd080039e0 msgr2=0x7fdd1c0b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd080039e0 0x7fdd1c0b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 -- 172.21.15.94:0/3885080597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c14d9d0 con 0x7fdd1c0ada50 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd22415640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c0ada50 0x7fdd1c0adf00 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c06d590 tx=0x7fdd18007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd127fc640 1 -- 172.21.15.94:0/3885080597 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800f6a0 con 0x7fdd1c0ada50 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd127fc640 1 -- 172.21.15.94:0/3885080597 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd18018070 con 0x7fdd1c0ada50 2024-03-06T07:24:16.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c14bd40 con 0x7fdd1c0ada50 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd127fc640 1 -- 172.21.15.94:0/3885080597 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800fac0 con 0x7fdd1c0ada50 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.412+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c14ee30 con 0x7fdd1c0ada50 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.413+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c04f330 con 0x7fdd1c0ada50 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.414+0000 7fdd127fc640 1 -- 172.21.15.94:0/3885080597 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd1800fc60 con 0x7fdd1c0ada50 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.415+0000 7fdd127fc640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd0006b4d0 0x7fdd0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.415+0000 7fdd127fc640 1 -- 172.21.15.94:0/3885080597 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6855+0+0 (secure 0 0 0) 0x7fdd1808a0f0 con 0x7fdd1c0ada50 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.415+0000 7fdd21c14640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd0006b4d0 0x7fdd0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.416+0000 7fdd21c14640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd0006b4d0 0x7fdd0006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036000 tx=0x7fdd1402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.420+0000 7fdd127fc640 1 -- 172.21.15.94:0/3885080597 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd18058530 con 0x7fdd1c0ada50 2024-03-06T07:24:16.636 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:16.634+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7fdd1c06d860 con 0x7fdd1c0ada50 2024-03-06T07:24:17.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.258+0000 7fdd127fc640 1 -- 172.21.15.94:0/3885080597 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 25 and data pool 26 v136) v1 ==== 177+0+0 (secure 0 0 0) 0x7fdd1805c5d0 con 0x7fdd1c0ada50 2024-03-06T07:24:17.260 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 25 and data pool 26 2024-03-06T07:24:17.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.267+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd0006b4d0 msgr2=0x7fdd0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.267+0000 7fdd23e78640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd0006b4d0 0x7fdd0006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036000 tx=0x7fdd1402e000 comp rx=0 tx=0).stop 2024-03-06T07:24:17.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.267+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c0ada50 msgr2=0x7fdd1c0adf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.267+0000 7fdd23e78640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c0ada50 0x7fdd1c0adf00 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c06d590 tx=0x7fdd18007430 comp rx=0 tx=0).stop 2024-03-06T07:24:17.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.268+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 shutdown_connections 2024-03-06T07:24:17.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.268+0000 7fdd23e78640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdd0006b4d0 0x7fdd0006d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.268+0000 7fdd23e78640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdd1c0ada50 0x7fdd1c0adf00 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.268+0000 7fdd23e78640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd1c0ad5e0 0x7fdd1c153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.268+0000 7fdd23e78640 1 --2- 172.21.15.94:0/3885080597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd080039e0 0x7fdd1c0b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.268+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 >> 172.21.15.94:0/3885080597 conn(0x7fdd1c0ac4f0 msgr2=0x7fdd1c151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.273+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 shutdown_connections 2024-03-06T07:24:17.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.274+0000 7fdd23e78640 1 -- 172.21.15.94:0/3885080597 wait complete. 2024-03-06T07:24:17.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:17.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.388+0000 7f1074d3e640 1 Processor -- start 2024-03-06T07:24:17.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.388+0000 7f1074d3e640 1 -- start start 2024-03-06T07:24:17.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.388+0000 7f1074d3e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f10700c88c0 0x7f1070155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.388+0000 7f1074d3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 0x7f1070158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.388+0000 7f1074d3e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10700c73d0 con 0x7f107014f360 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.388+0000 7f1074d3e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10700c8dd0 con 0x7f10700c88c0 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.388+0000 7f1074d3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10700be2e0 con 0x7f1070158b60 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106e59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f107014f360 0x7f10700c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47782/0 (socket says 172.21.15.94:47782) 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106e59c640 1 -- 172.21.15.94:0/2905373937 learned_addr learned my addr 172.21.15.94:0/2905373937 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106ed9d640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 0x7f1070158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106dd9b640 1 --2- 172.21.15.94:0/2905373937 >> v2:172.21.15.145:3300/0 conn(0x7f10700c88c0 0x7f1070155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878079343 0 0) 0x7f10700c73d0 con 0x7f107014f360 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1058003520 con 0x7f107014f360 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197721593 0 0) 0x7f1058003520 con 0x7f107014f360 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 msgr2=0x7f1070158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 0x7f1070158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 >> v2:172.21.15.145:3300/0 conn(0x7f10700c88c0 msgr2=0x7f1070155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 --2- 172.21.15.94:0/2905373937 >> v2:172.21.15.145:3300/0 conn(0x7f10700c88c0 0x7f1070155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700c7670 con 0x7f107014f360 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f1060001670 con 0x7f107014f360 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106ed9d640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 0x7f1070158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106dd9b640 1 --2- 172.21.15.94:0/2905373937 >> v2:172.21.15.145:3300/0 conn(0x7f10700c88c0 0x7f1070155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 >> v1:172.21.15.94:6789/0 conn(0x7f107014f360 legacy=0x7f10700c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 0x7f10580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1058003ef0 0x7f1058004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058004d40 0x7f10580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10700be2e0 con 0x7f1058003ef0 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105800b9c0 con 0x7f1058004d40 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105800bb40 con 0x7f1070158b60 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106dd9b640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1058003ef0 0x7f1058004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106e59c640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 0x7f10580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.389+0000 7f106ed9d640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058004d40 0x7f10580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.390+0000 7f106dd9b640 1 -- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 msgr2=0x7f10580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.390+0000 7f106dd9b640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1070158b60 0x7f10580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.390+0000 7f106dd9b640 1 -- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058004d40 msgr2=0x7f10580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.390+0000 7f106dd9b640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058004d40 0x7f10580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.390+0000 7f106dd9b640 1 -- 172.21.15.94:0/2905373937 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058003520 con 0x7f1058003ef0 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.390+0000 7f106dd9b640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1058003ef0 0x7f1058004800 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1064009c60 tx=0x7f1058005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317dada81e31e0a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064039070 con 0x7f1058003ef0 2024-03-06T07:24:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106402c950 con 0x7f1058003ef0 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f106d59a640 1 -- 172.21.15.94:0/2905373937 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064009860 con 0x7f1058003ef0 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f1074d3e640 1 -- 172.21.15.94:0/2905373937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1058003ef0 msgr2=0x7f1058004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f1074d3e640 1 --2- 172.21.15.94:0/2905373937 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1058003ef0 0x7f1058004800 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1064009c60 tx=0x7f1058005830 comp rx=0 tx=0).stop 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f106ed9d640 1 -- 172.21.15.94:0/2905373937 reap_dead start 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f1074d3e640 1 -- 172.21.15.94:0/2905373937 shutdown_connections 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f1074d3e640 1 -- 172.21.15.94:0/2905373937 >> 172.21.15.94:0/2905373937 conn(0x7f10700ac570 msgr2=0x7f10700ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f1074d3e640 1 -- 172.21.15.94:0/2905373937 shutdown_connections 2024-03-06T07:24:17.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.391+0000 7f1074d3e640 1 -- 172.21.15.94:0/2905373937 wait complete. 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.392+0000 7f1074d3e640 1 Processor -- start 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.392+0000 7f1074d3e640 1 -- start start 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f1074d3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058003ef0 0x7f10701eb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f1074d3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10700afbf0 0x7f10700bf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f1074d3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10700bf690 0x7f107014bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f1074d3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10700c5a50 con 0x7f10700afbf0 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f1074d3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10700c7c30 con 0x7f1058003ef0 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f1074d3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10700c5bd0 con 0x7f10700bf690 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10700afbf0 0x7f10700bf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10700afbf0 0x7f10700bf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43338/0 (socket says 172.21.15.94:43338) 2024-03-06T07:24:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 -- 172.21.15.94:0/446427391 learned_addr learned my addr 172.21.15.94:0/446427391 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106ed9d640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10700bf690 0x7f107014bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106e59c640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058003ef0 0x7f10701eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 -- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10700bf690 msgr2=0x7f107014bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10700bf690 0x7f107014bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 -- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058003ef0 msgr2=0x7f10701eb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058003ef0 0x7f10701eb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106dd9b640 1 -- 172.21.15.94:0/446427391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700bfb10 con 0x7f10700afbf0 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.393+0000 7f106e59c640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058003ef0 0x7f10701eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.394+0000 7f106dd9b640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10700afbf0 0x7f10700bf150 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1064009d80 tx=0x7f1064004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.394+0000 7f105effd640 1 -- 172.21.15.94:0/446427391 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064039050 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.394+0000 7f105effd640 1 -- 172.21.15.94:0/446427391 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064045070 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.394+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10700c0130 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.394+0000 7f105effd640 1 -- 172.21.15.94:0/446427391 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064009860 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.394+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10700be950 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.395+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107004f330 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.395+0000 7f105effd640 1 -- 172.21.15.94:0/446427391 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f106404a460 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.396+0000 7f105effd640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f105006b500 0x7f105006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.396+0000 7f105effd640 1 -- 172.21.15.94:0/446427391 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f10640ac8a0 con 0x7f10700afbf0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.396+0000 7f106e59c640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f105006b500 0x7f105006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.397+0000 7f106e59c640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f105006b500 0x7f105006d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1060001670 tx=0x7f1060009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.400+0000 7f105effd640 1 -- 172.21.15.94:0/446427391 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f106407ac80 con 0x7f10700afbf0 2024-03-06T07:24:17.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.593+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f107006cda0 con 0x7f10700afbf0 2024-03-06T07:24:17.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.595+0000 7f105effd640 1 -- 172.21.15.94:0/446427391 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+20615 (secure 0 0 0) 0x7f106407ed20 con 0x7f10700afbf0 2024-03-06T07:24:17.597 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:17.597 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":152,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:16.636050+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:17.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f105006b500 msgr2=0x7f105006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f105006b500 0x7f105006d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1060001670 tx=0x7f1060009040 comp rx=0 tx=0).stop 2024-03-06T07:24:17.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10700afbf0 msgr2=0x7f10700bf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10700afbf0 0x7f10700bf150 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1064009d80 tx=0x7f1064004730 comp rx=0 tx=0).stop 2024-03-06T07:24:17.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 shutdown_connections 2024-03-06T07:24:17.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f105006b500 0x7f105006d9c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10700bf690 0x7f107014bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10700afbf0 0x7f10700bf150 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 --2- 172.21.15.94:0/446427391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1058003ef0 0x7f10701eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 >> 172.21.15.94:0/446427391 conn(0x7f10700ac570 msgr2=0x7f10700c9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 shutdown_connections 2024-03-06T07:24:17.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.597+0000 7f1074d3e640 1 -- 172.21.15.94:0/446427391 wait complete. 2024-03-06T07:24:17.610 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:17.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.710+0000 7f87e1e66640 1 Processor -- start 2024-03-06T07:24:17.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.710+0000 7f87e1e66640 1 -- start start 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.710+0000 7f87e1e66640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f87dc159270 0x7f87dc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.710+0000 7f87e1e66640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87dc0c73d0 con 0x7f87dc14f360 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.710+0000 7f87e1e66640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87dc0be2e0 con 0x7f87dc0c88c0 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.710+0000 7f87e1e66640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87dc159700 con 0x7f87dc159270 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87db7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f87dc14f360 0x7f87dc0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47798/0 (socket says 172.21.15.94:47798) 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87db7fe640 1 -- 172.21.15.94:0/3360191747 learned_addr learned my addr 172.21.15.94:0/3360191747 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87dbfff640 1 --2- 172.21.15.94:0/3360191747 >> v2:172.21.15.145:3300/0 conn(0x7f87dc159270 0x7f87dc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912383412 0 0) 0x7f87dc0c73d0 con 0x7f87dc14f360 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c4003520 con 0x7f87dc14f360 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102106563 0 0) 0x7f87c4003520 con 0x7f87dc14f360 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 >> v2:172.21.15.145:3300/0 conn(0x7f87dc159270 msgr2=0x7f87dc155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 --2- 172.21.15.94:0/3360191747 >> v2:172.21.15.145:3300/0 conn(0x7f87dc159270 0x7f87dc155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 >> v1:172.21.15.145:6789/0 conn(0x7f87dc0c88c0 legacy=0x7f87dc158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc0c7670 con 0x7f87dc14f360 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f87cc001670 con 0x7f87dc14f360 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 >> v1:172.21.15.94:6789/0 conn(0x7f87dc14f360 legacy=0x7f87dc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40039b0 0x7f87c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87c4004800 0x7f87c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87c40051c0 0x7f87c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc159700 con 0x7f87c40051c0 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c400c160 con 0x7f87c40039b0 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c400c360 con 0x7f87c4004800 2024-03-06T07:24:17.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87dbfff640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87c40051c0 0x7f87c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87db7fe640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40039b0 0x7f87c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.711+0000 7f87daffd640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87c4004800 0x7f87c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.712+0000 7f87dbfff640 1 -- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87c4004800 msgr2=0x7f87c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.712+0000 7f87dbfff640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87c4004800 0x7f87c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.712+0000 7f87dbfff640 1 -- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40039b0 msgr2=0x7f87c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.712+0000 7f87dbfff640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40039b0 0x7f87c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.712+0000 7f87dbfff640 1 -- 172.21.15.94:0/3360191747 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c4003520 con 0x7f87c40051c0 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.712+0000 7f87dbfff640 1 -- 172.21.15.94:0/3360191747 reap_dead start 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.712+0000 7f87dbfff640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87c40051c0 0x7f87c400b9c0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009fc0 tx=0x7f87c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ea4ed66414f6ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4009c10 con 0x7f87c40051c0 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d402cc60 con 0x7f87c40051c0 2024-03-06T07:24:17.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87da7fc640 1 -- 172.21.15.94:0/3360191747 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003bd0 con 0x7f87c40051c0 2024-03-06T07:24:17.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87e1e66640 1 -- 172.21.15.94:0/3360191747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87c40051c0 msgr2=0x7f87c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87e1e66640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87c40051c0 0x7f87c400b9c0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009fc0 tx=0x7f87c4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87e1e66640 1 -- 172.21.15.94:0/3360191747 shutdown_connections 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87e1e66640 1 --2- 172.21.15.94:0/3360191747 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87c40051c0 0x7f87c400b9c0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87e1e66640 1 -- 172.21.15.94:0/3360191747 >> 172.21.15.94:0/3360191747 conn(0x7f87dc0ac570 msgr2=0x7f87dc0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87e1e66640 1 -- 172.21.15.94:0/3360191747 shutdown_connections 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.713+0000 7f87e1e66640 1 -- 172.21.15.94:0/3360191747 wait complete. 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.714+0000 7f87e1e66640 1 Processor -- start 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87e1e66640 1 -- start start 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87e1e66640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40051c0 0x7f87dc14dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87e1e66640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87dc148ab0 0x7f87dc148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87e1e66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87dc1566a0 0x7f87dc156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87e1e66640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0c7c30 con 0x7f87dc1566a0 2024-03-06T07:24:17.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87e1e66640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc155c10 con 0x7f87c40051c0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87e1e66640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc155d90 con 0x7f87dc148ab0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87dbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87dc1566a0 0x7f87dc156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87dbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87dc1566a0 0x7f87dc156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43358/0 (socket says 172.21.15.94:43358) 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87dbfff640 1 -- 172.21.15.94:0/2400491441 learned_addr learned my addr 172.21.15.94:0/2400491441 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87daffd640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87dc148ab0 0x7f87dc148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.715+0000 7f87db7fe640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40051c0 0x7f87dc14dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87dbfff640 1 -- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87dc148ab0 msgr2=0x7f87dc148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87dbfff640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87dc148ab0 0x7f87dc148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87dbfff640 1 -- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40051c0 msgr2=0x7f87dc14dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87dbfff640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40051c0 0x7f87dc14dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87dbfff640 1 -- 172.21.15.94:0/2400491441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc1ef610 con 0x7f87dc1566a0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87dbfff640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87dc1566a0 0x7f87dc156b20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f87d4035d30 tx=0x7f87d4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87c3fff640 1 -- 172.21.15.94:0/2400491441 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003990 con 0x7f87dc1566a0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87c3fff640 1 -- 172.21.15.94:0/2400491441 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d4044070 con 0x7f87dc1566a0 2024-03-06T07:24:17.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc152ee0 con 0x7f87dc1566a0 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87c3fff640 1 -- 172.21.15.94:0/2400491441 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003d50 con 0x7f87dc1566a0 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.716+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc153180 con 0x7f87dc1566a0 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.717+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc06d110 con 0x7f87dc1566a0 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.717+0000 7f87c3fff640 1 -- 172.21.15.94:0/2400491441 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87d4005d10 con 0x7f87dc1566a0 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.718+0000 7f87c3fff640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c806b500 0x7f87c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.718+0000 7f87c3fff640 1 -- 172.21.15.94:0/2400491441 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f87d40acc90 con 0x7f87dc1566a0 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.718+0000 7f87db7fe640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c806b500 0x7f87c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:17.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.718+0000 7f87db7fe640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c806b500 0x7f87c806d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f87cc001cf0 tx=0x7f87cc0078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:17.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.722+0000 7f87c3fff640 1 -- 172.21.15.94:0/2400491441 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87dc06d110 con 0x7f87dc1566a0 2024-03-06T07:24:17.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.988+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87dc04f330 con 0x7f87dc1566a0 2024-03-06T07:24:17.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.990+0000 7f87c3fff640 1 -- 172.21.15.94:0/2400491441 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+20615 (secure 0 0 0) 0x7f87d407b100 con 0x7f87dc1566a0 2024-03-06T07:24:17.992 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:17.992 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":152,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:16.636050+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c806b500 msgr2=0x7f87c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c806b500 0x7f87c806d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f87cc001cf0 tx=0x7f87cc0078f0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87dc1566a0 msgr2=0x7f87dc156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:17.994 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87dc1566a0 0x7f87dc156b20 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f87d4035d30 tx=0x7f87d4009750 comp rx=0 tx=0).stop 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 shutdown_connections 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f87c806b500 0x7f87c806d9c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87dc1566a0 0x7f87dc156b20 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87dc148ab0 0x7f87dc148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 --2- 172.21.15.94:0/2400491441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87c40051c0 0x7f87dc14dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 >> 172.21.15.94:0/2400491441 conn(0x7f87dc0ac570 msgr2=0x7f87dc14c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 shutdown_connections 2024-03-06T07:24:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:17.992+0000 7f87e1e66640 1 -- 172.21.15.94:0/2400491441 wait complete. 2024-03-06T07:24:18.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-03-06T07:24:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.108+0000 7f60a5193640 1 Processor -- start 2024-03-06T07:24:18.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.108+0000 7f60a5193640 1 -- start start 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f60a5193640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f60a006bff0 0x7f60a00c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f60a5193640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a00b04b0 con 0x7f60a00c7210 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f60a5193640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a006d210 con 0x7f60a00b00d0 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f60a5193640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a0156160 con 0x7f60a006bff0 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f609e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f60a006bff0 0x7f60a00c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f609f59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f60a00c7210 0x7f60a00c7680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47810/0 (socket says 172.21.15.94:47810) 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f609e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f60a006bff0 0x7f60a00c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43362/0 (socket says 172.21.15.94:43362) 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f609f59e640 1 -- 172.21.15.94:0/867609389 learned_addr learned my addr 172.21.15.94:0/867609389 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:18.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f609e59c640 1 -- 172.21.15.94:0/867609389 >> v1:172.21.15.145:6789/0 conn(0x7f60a00b00d0 legacy=0x7f60a006b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f609e59c640 1 -- 172.21.15.94:0/867609389 >> v1:172.21.15.94:6789/0 conn(0x7f60a00c7210 legacy=0x7f60a00c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.109+0000 7f609e59c640 1 -- 172.21.15.94:0/867609389 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a0155e80 con 0x7f60a006bff0 2024-03-06T07:24:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.110+0000 7f609e59c640 1 --2- 172.21.15.94:0/867609389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a006bff0 0x7f60a00c8ff0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f60a01562e0 tx=0x7f609402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68525a204bf0f2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f609dd9b640 1 -- 172.21.15.94:0/867609389 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094034070 con 0x7f60a006bff0 2024-03-06T07:24:18.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f609dd9b640 1 -- 172.21.15.94:0/867609389 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6094030610 con 0x7f60a006bff0 2024-03-06T07:24:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f609dd9b640 1 -- 172.21.15.94:0/867609389 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60940309e0 con 0x7f60a006bff0 2024-03-06T07:24:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f60a5193640 1 -- 172.21.15.94:0/867609389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a006bff0 msgr2=0x7f60a00c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f60a5193640 1 --2- 172.21.15.94:0/867609389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a006bff0 0x7f60a00c8ff0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f60a01562e0 tx=0x7f609402b960 comp rx=0 tx=0).stop 2024-03-06T07:24:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f60a5193640 1 -- 172.21.15.94:0/867609389 shutdown_connections 2024-03-06T07:24:18.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f60a5193640 1 --2- 172.21.15.94:0/867609389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a006bff0 0x7f60a00c8ff0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f60a5193640 1 -- 172.21.15.94:0/867609389 >> 172.21.15.94:0/867609389 conn(0x7f60a00ac810 msgr2=0x7f60a00acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:18.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f60a5193640 1 -- 172.21.15.94:0/867609389 shutdown_connections 2024-03-06T07:24:18.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.111+0000 7f60a5193640 1 -- 172.21.15.94:0/867609389 wait complete. 2024-03-06T07:24:18.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 Processor -- start 2024-03-06T07:24:18.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 -- start start 2024-03-06T07:24:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60a006bff0 0x7f60a014f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a00b00d0 0x7f60a01ee090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60a00c7210 0x7f60a014e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0152c50 con 0x7f60a00b00d0 2024-03-06T07:24:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a00c5710 con 0x7f60a006bff0 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f60a5193640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a00c7d90 con 0x7f60a00c7210 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f609e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a00b00d0 0x7f60a01ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f609e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a00b00d0 0x7f60a01ee090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43366/0 (socket says 172.21.15.94:43366) 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.113+0000 7f609e59c640 1 -- 172.21.15.94:0/722212522 learned_addr learned my addr 172.21.15.94:0/722212522 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609ed9d640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60a006bff0 0x7f60a014f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609f59e640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60a00c7210 0x7f60a014e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609e59c640 1 -- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60a00c7210 msgr2=0x7f60a014e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609e59c640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60a00c7210 0x7f60a014e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609e59c640 1 -- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60a006bff0 msgr2=0x7f60a014f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609e59c640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60a006bff0 0x7f60a014f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609e59c640 1 -- 172.21.15.94:0/722212522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a014f410 con 0x7f60a00b00d0 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f609e59c640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a00b00d0 0x7f60a01ee090 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f6094009890 tx=0x7f60940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f60877fe640 1 -- 172.21.15.94:0/722212522 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094046020 con 0x7f60a00b00d0 2024-03-06T07:24:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f60877fe640 1 -- 172.21.15.94:0/722212522 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6094039a10 con 0x7f60a00b00d0 2024-03-06T07:24:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a0159350 con 0x7f60a00b00d0 2024-03-06T07:24:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f60877fe640 1 -- 172.21.15.94:0/722212522 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094034040 con 0x7f60a00b00d0 2024-03-06T07:24:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.114+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a01ee8d0 con 0x7f60a00b00d0 2024-03-06T07:24:18.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.115+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a004f330 con 0x7f60a00b00d0 2024-03-06T07:24:18.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.116+0000 7f60877fe640 1 -- 172.21.15.94:0/722212522 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6094037070 con 0x7f60a00b00d0 2024-03-06T07:24:18.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.116+0000 7f60877fe640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608c06b500 0x7f608c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:18.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.116+0000 7f60877fe640 1 -- 172.21.15.94:0/722212522 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f60940ac430 con 0x7f60a00b00d0 2024-03-06T07:24:18.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.116+0000 7f609ed9d640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608c06b500 0x7f608c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:18.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.117+0000 7f609ed9d640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608c06b500 0x7f608c06d9c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6090001780 tx=0x7f6090009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:18.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.125+0000 7f60877fe640 1 -- 172.21.15.94:0/722212522 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6094050020 con 0x7f60a00b00d0 2024-03-06T07:24:18.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:18.404+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f60a006ce50 con 0x7f60a00b00d0 2024-03-06T07:24:19.152 INFO:tasks.ceph.mds.a.smithi094.stderr:2024-03-06T07:24:19.151+0000 7f3c0e760640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:19.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.270+0000 7f60877fe640 1 -- 172.21.15.94:0/722212522 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v139) v1 ==== 119+0+0 (secure 0 0 0) 0x7f609407a810 con 0x7f60a00b00d0 2024-03-06T07:24:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608c06b500 msgr2=0x7f608c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608c06b500 0x7f608c06d9c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6090001780 tx=0x7f6090009040 comp rx=0 tx=0).stop 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a00b00d0 msgr2=0x7f60a01ee090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a00b00d0 0x7f60a01ee090 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f6094009890 tx=0x7f60940046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 shutdown_connections 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f608c06b500 0x7f608c06d9c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60a00c7210 0x7f60a014e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60a00b00d0 0x7f60a01ee090 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 --2- 172.21.15.94:0/722212522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60a006bff0 0x7f60a014f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 >> 172.21.15.94:0/722212522 conn(0x7f60a00ac810 msgr2=0x7f60a00ae4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 shutdown_connections 2024-03-06T07:24:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.272+0000 7f60a5193640 1 -- 172.21.15.94:0/722212522 wait complete. 2024-03-06T07:24:19.285 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:19.286 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-03-06T07:24:19.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.388+0000 7f51d7e88640 1 Processor -- start 2024-03-06T07:24:19.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.388+0000 7f51d7e88640 1 -- start start 2024-03-06T07:24:19.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d7e88640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51d0155c30 0x7f51d006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d7e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00c5380 0x7f51d00c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d7e88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d00b0450 con 0x7f51d00b0070 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d7e88640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d006cfb0 con 0x7f51d0155c30 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d7e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d006c020 con 0x7f51d00c5380 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d5423640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51d0155c30 0x7f51d006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d5423640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51d0155c30 0x7f51d006ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58888/0 (socket says 172.21.15.94:58888) 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d5423640 1 -- 172.21.15.94:0/245300816 learned_addr learned my addr 172.21.15.94:0/245300816 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.389+0000 7f51d6425640 1 --2- 172.21.15.94:0/245300816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00c5380 0x7f51d00c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.390+0000 7f51d5423640 1 -- 172.21.15.94:0/245300816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00c5380 msgr2=0x7f51d00c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.390+0000 7f51d5423640 1 --2- 172.21.15.94:0/245300816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00c5380 0x7f51d00c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.390+0000 7f51d5423640 1 -- 172.21.15.94:0/245300816 >> v1:172.21.15.145:6789/0 conn(0x7f51d00b0070 legacy=0x7f51d00c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.390+0000 7f51d5423640 1 -- 172.21.15.94:0/245300816 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d01560e0 con 0x7f51d0155c30 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.390+0000 7f51d5423640 1 --2- 172.21.15.94:0/245300816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d006ca70 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009870 tx=0x7f51c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e5a640a676cc31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d4c22640 1 -- 172.21.15.94:0/245300816 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0035020 con 0x7f51d0155c30 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d4c22640 1 -- 172.21.15.94:0/245300816 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c0031640 con 0x7f51d0155c30 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d4c22640 1 -- 172.21.15.94:0/245300816 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c002d040 con 0x7f51d0155c30 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 -- 172.21.15.94:0/245300816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 msgr2=0x7f51d006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 --2- 172.21.15.94:0/245300816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d006ca70 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009870 tx=0x7f51c002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 -- 172.21.15.94:0/245300816 shutdown_connections 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 --2- 172.21.15.94:0/245300816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00c5380 0x7f51d00c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 --2- 172.21.15.94:0/245300816 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d006ca70 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 -- 172.21.15.94:0/245300816 >> 172.21.15.94:0/245300816 conn(0x7f51d00ac590 msgr2=0x7f51d00ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 -- 172.21.15.94:0/245300816 shutdown_connections 2024-03-06T07:24:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.391+0000 7f51d7e88640 1 -- 172.21.15.94:0/245300816 wait complete. 2024-03-06T07:24:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.393+0000 7f51d7e88640 1 Processor -- start 2024-03-06T07:24:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.393+0000 7f51d7e88640 1 -- start start 2024-03-06T07:24:19.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d7e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00b0070 0x7f51d01549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d7e88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51d00c5380 0x7f51d014ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d7e88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d014bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d7e88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d006c270 con 0x7f51d0155c30 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d7e88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d00c7ad0 con 0x7f51d00c5380 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d7e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d006c550 con 0x7f51d00b0070 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d6425640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d014bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d6425640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d014bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58892/0 (socket says 172.21.15.94:58892) 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d6425640 1 -- 172.21.15.94:0/3285456823 learned_addr learned my addr 172.21.15.94:0/3285456823 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d5c24640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00b0070 0x7f51d01549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d5423640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51d00c5380 0x7f51d014ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d6425640 1 -- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00b0070 msgr2=0x7f51d01549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d6425640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00b0070 0x7f51d01549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d6425640 1 -- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51d00c5380 msgr2=0x7f51d014ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.394+0000 7f51d6425640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51d00c5380 0x7f51d014ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.395+0000 7f51d6425640 1 -- 172.21.15.94:0/3285456823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d0155620 con 0x7f51d0155c30 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.395+0000 7f51d6425640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d014bf40 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f51c800efc0 tx=0x7f51c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.395+0000 7f51c67fc640 1 -- 172.21.15.94:0/3285456823 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8018020 con 0x7f51d0155c30 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.395+0000 7f51c67fc640 1 -- 172.21.15.94:0/3285456823 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8014480 con 0x7f51d0155c30 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.395+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d0158110 con 0x7f51d0155c30 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.395+0000 7f51c67fc640 1 -- 172.21.15.94:0/3285456823 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8014870 con 0x7f51d0155c30 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.395+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d01ec030 con 0x7f51d0155c30 2024-03-06T07:24:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.396+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d00c85d0 con 0x7f51d0155c30 2024-03-06T07:24:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.396+0000 7f51c67fc640 1 -- 172.21.15.94:0/3285456823 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51c8005d10 con 0x7f51d0155c30 2024-03-06T07:24:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.397+0000 7f51c67fc640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51b406b4d0 0x7f51b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.397+0000 7f51c67fc640 1 -- 172.21.15.94:0/3285456823 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f51c808b500 con 0x7f51d0155c30 2024-03-06T07:24:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.397+0000 7f51d5c24640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51b406b4d0 0x7f51b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.398+0000 7f51d5c24640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51b406b4d0 0x7f51b406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f51cc007fd0 tx=0x7f51cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.401+0000 7f51c67fc640 1 -- 172.21.15.94:0/3285456823 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51d00c85d0 con 0x7f51d0155c30 2024-03-06T07:24:19.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:19.613+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f51d006d860 con 0x7f51d0155c30 2024-03-06T07:24:20.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.271+0000 7f51c67fc640 1 -- 172.21.15.94:0/3285456823 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v140) v1 ==== 123+0+0 (secure 0 0 0) 0x7f51c8022030 con 0x7f51d0155c30 2024-03-06T07:24:20.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51b406b4d0 msgr2=0x7f51b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51b406b4d0 0x7f51b406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f51cc007fd0 tx=0x7f51cc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 msgr2=0x7f51d014bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d014bf40 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f51c800efc0 tx=0x7f51c800c490 comp rx=0 tx=0).stop 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 shutdown_connections 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51b406b4d0 0x7f51b406d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51d0155c30 0x7f51d014bf40 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51d00c5380 0x7f51d014ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 --2- 172.21.15.94:0/3285456823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51d00b0070 0x7f51d01549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 >> 172.21.15.94:0/3285456823 conn(0x7f51d00ac590 msgr2=0x7f51d00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 shutdown_connections 2024-03-06T07:24:20.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.273+0000 7f51d7e88640 1 -- 172.21.15.94:0/3285456823 wait complete. 2024-03-06T07:24:20.289 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-03-06T07:24:20.290 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.388+0000 7f3ee558c640 1 Processor -- start 2024-03-06T07:24:20.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.388+0000 7f3ee558c640 1 -- start start 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3ee558c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ee01527b0 0x7f3ee0159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3ee558c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c88b0 0x7f3ee00c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3ee558c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee00c7c60 con 0x7f3ee00c7180 2024-03-06T07:24:20.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3ee558c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee0159640 con 0x7f3ee01527b0 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3ee558c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee00c8060 con 0x7f3ee00c88b0 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ee01527b0 0x7f3ee0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ee01527b0 0x7f3ee0159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58904/0 (socket says 172.21.15.94:58904) 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edeffd640 1 -- 172.21.15.94:0/3437070514 learned_addr learned my addr 172.21.15.94:0/3437070514 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edf7fe640 1 --2- 172.21.15.94:0/3437070514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c88b0 0x7f3ee00c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edeffd640 1 -- 172.21.15.94:0/3437070514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c88b0 msgr2=0x7f3ee00c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edeffd640 1 --2- 172.21.15.94:0/3437070514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c88b0 0x7f3ee00c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edeffd640 1 -- 172.21.15.94:0/3437070514 >> v1:172.21.15.145:6789/0 conn(0x7f3ee00c7180 legacy=0x7f3ee0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.389+0000 7f3edeffd640 1 -- 172.21.15.94:0/3437070514 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee00c76a0 con 0x7f3ee01527b0 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.390+0000 7f3edf7fe640 1 --2- 172.21.15.94:0/3437070514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c88b0 0x7f3ee00c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.390+0000 7f3edeffd640 1 --2- 172.21.15.94:0/3437070514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee01527b0 0x7f3ee0159100 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009870 tx=0x7f3ed002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8631efbfd820a10c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.390+0000 7f3ede7fc640 1 -- 172.21.15.94:0/3437070514 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00317c0 con 0x7f3ee01527b0 2024-03-06T07:24:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.390+0000 7f3ede7fc640 1 -- 172.21.15.94:0/3437070514 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed0037070 con 0x7f3ee01527b0 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.390+0000 7f3ede7fc640 1 -- 172.21.15.94:0/3437070514 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0031b80 con 0x7f3ee01527b0 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 -- 172.21.15.94:0/3437070514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee01527b0 msgr2=0x7f3ee0159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 --2- 172.21.15.94:0/3437070514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee01527b0 0x7f3ee0159100 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009870 tx=0x7f3ed002b860 comp rx=0 tx=0).stop 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 -- 172.21.15.94:0/3437070514 shutdown_connections 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 --2- 172.21.15.94:0/3437070514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c88b0 0x7f3ee00c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 --2- 172.21.15.94:0/3437070514 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee01527b0 0x7f3ee0159100 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 -- 172.21.15.94:0/3437070514 >> 172.21.15.94:0/3437070514 conn(0x7f3ee00ac560 msgr2=0x7f3ee00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 -- 172.21.15.94:0/3437070514 shutdown_connections 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.391+0000 7f3ee558c640 1 -- 172.21.15.94:0/3437070514 wait complete. 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.393+0000 7f3ee558c640 1 Processor -- start 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.393+0000 7f3ee558c640 1 -- start start 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ee558c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c7180 0x7f3ee00c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ee558c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee00c88b0 0x7f3ee0154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ee558c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee01527b0 0x7f3ee014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ee558c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0155600 con 0x7f3ee00c88b0 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ee558c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0155320 con 0x7f3ee01527b0 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ee558c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee00c8d60 con 0x7f3ee00c7180 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ed67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee00c88b0 0x7f3ee0154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ed67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee00c88b0 0x7f3ee0154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58916/0 (socket says 172.21.15.94:58916) 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3ed67fc640 1 -- 172.21.15.94:0/1249760847 learned_addr learned my addr 172.21.15.94:0/1249760847 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.394+0000 7f3edeffd640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c7180 0x7f3ee00c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3edf7fe640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee01527b0 0x7f3ee014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed67fc640 1 -- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c7180 msgr2=0x7f3ee00c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed67fc640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c7180 0x7f3ee00c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed67fc640 1 -- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee01527b0 msgr2=0x7f3ee014b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed67fc640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee01527b0 0x7f3ee014b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed67fc640 1 -- 172.21.15.94:0/1249760847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee0157f20 con 0x7f3ee00c88b0 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed67fc640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee00c88b0 0x7f3ee0154040 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800b3f0 tx=0x7f3ec800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed7fff640 1 -- 172.21.15.94:0/1249760847 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8015020 con 0x7f3ee00c88b0 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed7fff640 1 -- 172.21.15.94:0/1249760847 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec8011830 con 0x7f3ee00c88b0 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee0158200 con 0x7f3ee00c88b0 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ed7fff640 1 -- 172.21.15.94:0/1249760847 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8011c20 con 0x7f3ee00c88b0 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.395+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee0157190 con 0x7f3ee00c88b0 2024-03-06T07:24:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.396+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee004f330 con 0x7f3ee00c88b0 2024-03-06T07:24:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.397+0000 7f3ed7fff640 1 -- 172.21.15.94:0/1249760847 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ec8018020 con 0x7f3ee00c88b0 2024-03-06T07:24:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.398+0000 7f3ed7fff640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.398+0000 7f3ed7fff640 1 -- 172.21.15.94:0/1249760847 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3ec808a140 con 0x7f3ee00c88b0 2024-03-06T07:24:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.398+0000 7f3edeffd640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.399+0000 7f3edeffd640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0036000 tx=0x7f3ed003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.402+0000 7f3ed7fff640 1 -- 172.21.15.94:0/1249760847 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ec801f030 con 0x7f3ee00c88b0 2024-03-06T07:24:20.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.682+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ee006ce00 con 0x7f3ee00c88b0 2024-03-06T07:24:20.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.682+0000 7f3ed7fff640 1 -- 172.21.15.94:0/1249760847 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6275 (secure 0 0 0) 0x7f3ec8058520 con 0x7f3ee00c88b0 2024-03-06T07:24:20.684 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:20.684 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8155,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:05.235078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/1613596341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1613596341},{"type":"v1","addr":"172.21.15.94:6835","nonce":1613596341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:19.614520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:20.685 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 140 2024-03-06T07:24:20.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 msgr2=0x7f3ec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0036000 tx=0x7f3ed003d040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee00c88b0 msgr2=0x7f3ee0154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee00c88b0 0x7f3ee0154040 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800b3f0 tx=0x7f3ec800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 shutdown_connections 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ec006b4d0 0x7f3ec006d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee01527b0 0x7f3ee014b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee00c88b0 0x7f3ee0154040 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 --2- 172.21.15.94:0/1249760847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee00c7180 0x7f3ee00c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.684+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 >> 172.21.15.94:0/1249760847 conn(0x7f3ee00ac560 msgr2=0x7f3ee0158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.685+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 shutdown_connections 2024-03-06T07:24:20.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.685+0000 7f3ee558c640 1 -- 172.21.15.94:0/1249760847 wait complete. 2024-03-06T07:24:20.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.822+0000 7f2b6320e640 1 Processor -- start 2024-03-06T07:24:20.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.822+0000 7f2b6320e640 1 -- start start 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.822+0000 7f2b6320e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2b5c14f360 0x7f2b5c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.822+0000 7f2b6320e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2b5c158b60 0x7f2b5c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.822+0000 7f2b6320e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5c0c73d0 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.822+0000 7f2b6320e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c0c8dd0 con 0x7f2b5c14f360 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.822+0000 7f2b6320e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c0be2e0 con 0x7f2b5c158b60 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b60faa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2b5c14f360 0x7f2b5c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b60faa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2b5c14f360 0x7f2b5c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58922/0 (socket says 172.21.15.94:58922) 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b60faa640 1 -- 172.21.15.94:0/1706027613 learned_addr learned my addr 172.21.15.94:0/1706027613 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b617ab640 1 --2- 172.21.15.94:0/1706027613 >> v2:172.21.15.145:3300/0 conn(0x7f2b5c158b60 0x7f2b5c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b60faa640 1 -- 172.21.15.94:0/1706027613 >> v2:172.21.15.145:3300/0 conn(0x7f2b5c158b60 msgr2=0x7f2b5c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b60faa640 1 --2- 172.21.15.94:0/1706027613 >> v2:172.21.15.145:3300/0 conn(0x7f2b5c158b60 0x7f2b5c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b60faa640 1 -- 172.21.15.94:0/1706027613 >> v1:172.21.15.145:6789/0 conn(0x7f2b5c0c88c0 legacy=0x7f2b5c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.823+0000 7f2b60faa640 1 -- 172.21.15.94:0/1706027613 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0c7670 con 0x7f2b5c14f360 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.824+0000 7f2b60faa640 1 --2- 172.21.15.94:0/1706027613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c14f360 0x7f2b5c155c10 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c152d60 tx=0x7f2b5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c5dbb542a27892 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.824+0000 7f2b5b7fe640 1 -- 172.21.15.94:0/1706027613 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50035020 con 0x7f2b5c14f360 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.824+0000 7f2b5b7fe640 1 -- 172.21.15.94:0/1706027613 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b50031870 con 0x7f2b5c14f360 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.824+0000 7f2b5b7fe640 1 -- 172.21.15.94:0/1706027613 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50031c40 con 0x7f2b5c14f360 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 -- 172.21.15.94:0/1706027613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c14f360 msgr2=0x7f2b5c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 --2- 172.21.15.94:0/1706027613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c14f360 0x7f2b5c155c10 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c152d60 tx=0x7f2b5002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 -- 172.21.15.94:0/1706027613 shutdown_connections 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 --2- 172.21.15.94:0/1706027613 >> v2:172.21.15.145:3300/0 conn(0x7f2b5c158b60 0x7f2b5c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 --2- 172.21.15.94:0/1706027613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c14f360 0x7f2b5c155c10 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 -- 172.21.15.94:0/1706027613 >> 172.21.15.94:0/1706027613 conn(0x7f2b5c0ac570 msgr2=0x7f2b5c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 -- 172.21.15.94:0/1706027613 shutdown_connections 2024-03-06T07:24:20.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.825+0000 7f2b6320e640 1 -- 172.21.15.94:0/1706027613 wait complete. 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.826+0000 7f2b6320e640 1 Processor -- start 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.826+0000 7f2b6320e640 1 -- start start 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c0c88c0 0x7f2b5c1e0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b5c14f360 0x7f2b5c1ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b5c158b60 0x7f2b5c1ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0c5a50 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0c7c30 con 0x7f2b5c158b60 2024-03-06T07:24:20.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0c5bd0 con 0x7f2b5c14f360 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c0c88c0 0x7f2b5c1e0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c0c88c0 0x7f2b5c1e0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58930/0 (socket says 172.21.15.94:58930) 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 -- 172.21.15.94:0/3566662239 learned_addr learned my addr 172.21.15.94:0/3566662239 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 -- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b5c14f360 msgr2=0x7f2b5c1ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b617ab640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b5c158b60 0x7f2b5c1ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b5bfff640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b5c14f360 0x7f2b5c1ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b5c14f360 0x7f2b5c1ef370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 -- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b5c158b60 msgr2=0x7f2b5c1ebd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b5c158b60 0x7f2b5c1ebd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 -- 172.21.15.94:0/3566662239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c158370 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b617ab640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b5c158b60 0x7f2b5c1ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b60faa640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c0c88c0 0x7f2b5c1e0530 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2b50000c00 tx=0x7f2b500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b597fa640 1 -- 172.21.15.94:0/3566662239 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50035040 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b597fa640 1 -- 172.21.15.94:0/3566662239 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b50030d00 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c157ad0 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b597fa640 1 -- 172.21.15.94:0/3566662239 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b500456a0 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.827+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c158090 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.828+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c04f330 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.829+0000 7f2b597fa640 1 -- 172.21.15.94:0/3566662239 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b50038020 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.829+0000 7f2b597fa640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b4006b4d0 0x7f2b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.830+0000 7f2b597fa640 1 -- 172.21.15.94:0/3566662239 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f2b500aca30 con 0x7f2b5c0c88c0 2024-03-06T07:24:20.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.830+0000 7f2b5bfff640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b4006b4d0 0x7f2b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.830+0000 7f2b5bfff640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b4006b4d0 0x7f2b4006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c1efa20 tx=0x7f2b48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:20.834+0000 7f2b597fa640 1 -- 172.21.15.94:0/3566662239 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b5007b460 con 0x7f2b5c0c88c0 2024-03-06T07:24:21.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.112+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b5c0c1640 con 0x7f2b5c0c88c0 2024-03-06T07:24:21.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.115+0000 7f2b597fa640 1 -- 172.21.15.94:0/3566662239 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+20602 (secure 0 0 0) 0x7f2b5007ad60 con 0x7f2b5c0c88c0 2024-03-06T07:24:21.117 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:21.117 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":153,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:18.255304+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.116+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b4006b4d0 msgr2=0x7f2b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.116+0000 7f2b6320e640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b4006b4d0 0x7f2b4006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c1efa20 tx=0x7f2b48009040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.116+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c0c88c0 msgr2=0x7f2b5c1e0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.116+0000 7f2b6320e640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c0c88c0 0x7f2b5c1e0530 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2b50000c00 tx=0x7f2b500046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 shutdown_connections 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b4006b4d0 0x7f2b4006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b5c158b60 0x7f2b5c1ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b5c14f360 0x7f2b5c1ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 --2- 172.21.15.94:0/3566662239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b5c0c88c0 0x7f2b5c1e0530 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 >> 172.21.15.94:0/3566662239 conn(0x7f2b5c0ac570 msgr2=0x7f2b5c0ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 shutdown_connections 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.117+0000 7f2b6320e640 1 -- 172.21.15.94:0/3566662239 wait complete. 2024-03-06T07:24:21.130 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:21.130 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:21.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.228+0000 7f0c5b5e2640 1 Processor -- start 2024-03-06T07:24:21.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.228+0000 7f0c5b5e2640 1 -- start start 2024-03-06T07:24:21.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c5b5e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 0x7f0c5406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c5b5e2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540c1ae0 con 0x7f0c54155c80 2024-03-06T07:24:21.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c5b5e2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5406e450 con 0x7f0c54152520 2024-03-06T07:24:21.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c5b5e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540c1dc0 con 0x7f0c5406ef30 2024-03-06T07:24:21.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c58b7d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0c54155c80 0x7f0c540720a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48490/0 (socket says 172.21.15.94:48490) 2024-03-06T07:24:21.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c58b7d640 1 -- 172.21.15.94:0/4061259546 learned_addr learned my addr 172.21.15.94:0/4061259546 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:21.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c59b7f640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 0x7f0c5406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341980086 0 0) 0x7f0c540c1ae0 con 0x7f0c54155c80 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40003520 con 0x7f0c54155c80 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904348162 0 0) 0x7f0c5406e450 con 0x7f0c54152520 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c540c1ae0 con 0x7f0c54152520 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496638842 0 0) 0x7f0c40003520 con 0x7f0c54155c80 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 msgr2=0x7f0c5406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 0x7f0c5406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 >> v1:172.21.15.145:6789/0 conn(0x7f0c54152520 legacy=0x7f0c540c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5406e730 con 0x7f0c54155c80 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f0c50001670 con 0x7f0c54155c80 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 >> v1:172.21.15.94:6789/0 conn(0x7f0c54155c80 legacy=0x7f0c540720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c40003b70 0x7f0c40003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 0x7f0c40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40004d40 0x7f0c400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540c1dc0 con 0x7f0c40003b70 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4000ba70 con 0x7f0c40004d40 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.230+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4000bc20 con 0x7f0c5406ef30 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c5937e640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c40003b70 0x7f0c40003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c58b7d640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 0x7f0c40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c59b7f640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40004d40 0x7f0c400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c5937e640 1 -- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 msgr2=0x7f0c40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c5937e640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 0x7f0c40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c5937e640 1 -- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40004d40 msgr2=0x7f0c400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c5937e640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40004d40 0x7f0c400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c5937e640 1 -- 172.21.15.94:0/4061259546 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40003520 con 0x7f0c40003b70 2024-03-06T07:24:21.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c58b7d640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c5406ef30 0x7f0c40004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c59b7f640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40004d40 0x7f0c400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.231+0000 7f0c5937e640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c40003b70 0x7f0c40003f70 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f0c480018d0 tx=0x7f0c40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809a60e62c3803f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48037070 con 0x7f0c40003b70 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c480052d0 con 0x7f0c40003b70 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c47fff640 1 -- 172.21.15.94:0/4061259546 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c480014f0 con 0x7f0c40003b70 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/4061259546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c40003b70 msgr2=0x7f0c40003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c5b5e2640 1 --2- 172.21.15.94:0/4061259546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c40003b70 0x7f0c40003f70 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f0c480018d0 tx=0x7f0c40005830 comp rx=0 tx=0).stop 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c59b7f640 1 -- 172.21.15.94:0/4061259546 reap_dead start 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/4061259546 shutdown_connections 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/4061259546 >> 172.21.15.94:0/4061259546 conn(0x7f0c540c76b0 msgr2=0x7f0c540c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/4061259546 shutdown_connections 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.232+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/4061259546 wait complete. 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.233+0000 7f0c5b5e2640 1 Processor -- start 2024-03-06T07:24:21.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.233+0000 7f0c5b5e2640 1 -- start start 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.234+0000 7f0c5b5e2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40003b70 0x7f0c541ea5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.234+0000 7f0c5b5e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c54147790 0x7f0c54148da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.234+0000 7f0c5b5e2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c5414a040 0x7f0c541eba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.234+0000 7f0c5b5e2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54152900 con 0x7f0c5414a040 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.234+0000 7f0c5b5e2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540727b0 con 0x7f0c40003b70 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.234+0000 7f0c5b5e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54152a80 con 0x7f0c54147790 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c5414a040 0x7f0c541eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c5414a040 0x7f0c541eba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58952/0 (socket says 172.21.15.94:58952) 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 -- 172.21.15.94:0/3234472022 learned_addr learned my addr 172.21.15.94:0/3234472022 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c5937e640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40003b70 0x7f0c541ea5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c58b7d640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c54147790 0x7f0c54148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 -- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c54147790 msgr2=0x7f0c54148da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c54147790 0x7f0c54148da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 -- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40003b70 msgr2=0x7f0c541ea5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40003b70 0x7f0c541ea5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 -- 172.21.15.94:0/3234472022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c54147d00 con 0x7f0c5414a040 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c5937e640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40003b70 0x7f0c541ea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c59b7f640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c5414a040 0x7f0c541eba40 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0c540c4180 tx=0x7f0c4c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c45ffb640 1 -- 172.21.15.94:0/3234472022 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c01a030 con 0x7f0c5414a040 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c45ffb640 1 -- 172.21.15.94:0/3234472022 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c010d20 con 0x7f0c5414a040 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c54147fe0 con 0x7f0c5414a040 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.235+0000 7f0c45ffb640 1 -- 172.21.15.94:0/3234472022 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c003bd0 con 0x7f0c5414a040 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.236+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c541485a0 con 0x7f0c5414a040 2024-03-06T07:24:21.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.236+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5404f330 con 0x7f0c5414a040 2024-03-06T07:24:21.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.237+0000 7f0c45ffb640 1 -- 172.21.15.94:0/3234472022 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c4c01d070 con 0x7f0c5414a040 2024-03-06T07:24:21.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.237+0000 7f0c45ffb640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c3806b500 0x7f0c3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.237+0000 7f0c45ffb640 1 -- 172.21.15.94:0/3234472022 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f0c4c08b070 con 0x7f0c5414a040 2024-03-06T07:24:21.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.238+0000 7f0c5937e640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c3806b500 0x7f0c3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.238+0000 7f0c5937e640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c3806b500 0x7f0c3806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0c48034000 tx=0x7f0c48031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.246+0000 7f0c45ffb640 1 -- 172.21.15.94:0/3234472022 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c4c059aa0 con 0x7f0c5414a040 2024-03-06T07:24:21.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.420+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c540c0b70 con 0x7f0c3806b500 2024-03-06T07:24:21.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.433+0000 7f0c45ffb640 1 -- 172.21.15.94:0/3234472022 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+491467 (secure 0 0 0) 0x7f0c540c0b70 con 0x7f0c3806b500 2024-03-06T07:24:21.440 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:21.443 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:24:21.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c3806b500 msgr2=0x7f0c3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c3806b500 0x7f0c3806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0c48034000 tx=0x7f0c48031000 comp rx=0 tx=0).stop 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c5414a040 msgr2=0x7f0c541eba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c5414a040 0x7f0c541eba40 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0c540c4180 tx=0x7f0c4c010590 comp rx=0 tx=0).stop 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 shutdown_connections 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c3806b500 0x7f0c3806d9c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c5414a040 0x7f0c541eba40 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c54147790 0x7f0c54148da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 --2- 172.21.15.94:0/3234472022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c40003b70 0x7f0c541ea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 >> 172.21.15.94:0/3234472022 conn(0x7f0c540c76b0 msgr2=0x7f0c5406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 shutdown_connections 2024-03-06T07:24:21.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.437+0000 7f0c5b5e2640 1 -- 172.21.15.94:0/3234472022 wait complete. 2024-03-06T07:24:21.450 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-06T07:24:20.714092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264504,"kb_used_data":15040,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710216,"statfs":{"total":773094113280,"available":772823261184,"internally_reserved":0,"allocated":15400960,"data_stored":8539140,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.481974"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261571+0000","last_change":"2024-03-06T07:24:17.258953+0000","last_active":"2024-03-06T07:24:17.261571+0000","last_peered":"2024-03-06T07:24:17.261571+0000","last_clean":"2024-03-06T07:24:17.261571+0000","last_became_active":"2024-03-06T07:24:17.258657+0000","last_became_peered":"2024-03-06T07:24:17.258657+0000","last_unstale":"2024-03-06T07:24:17.261571+0000","last_undegraded":"2024-03-06T07:24:17.261571+0000","last_fullsized":"2024-03-06T07:24:17.261571+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:45.917698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251582+0000","last_change":"2024-03-06T07:24:03.878818+0000","last_active":"2024-03-06T07:24:17.251582+0000","last_peered":"2024-03-06T07:24:17.251582+0000","last_clean":"2024-03-06T07:24:17.251582+0000","last_became_active":"2024-03-06T07:24:03.878538+0000","last_became_peered":"2024-03-06T07:24:03.878538+0000","last_unstale":"2024-03-06T07:24:17.251582+0000","last_undegraded":"2024-03-06T07:24:17.251582+0000","last_fullsized":"2024-03-06T07:24:17.251582+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:43.198749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266396+0000","last_change":"2024-03-06T07:24:17.265631+0000","last_active":"2024-03-06T07:24:17.266396+0000","last_peered":"2024-03-06T07:24:17.266396+0000","last_clean":"2024-03-06T07:24:17.266396+0000","last_became_active":"2024-03-06T07:24:17.265372+0000","last_became_peered":"2024-03-06T07:24:17.265372+0000","last_unstale":"2024-03-06T07:24:17.266396+0000","last_undegraded":"2024-03-06T07:24:17.266396+0000","last_fullsized":"2024-03-06T07:24:17.266396+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:14.432264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258133+0000","last_change":"2024-03-06T07:24:03.887970+0000","last_active":"2024-03-06T07:24:17.258133+0000","last_peered":"2024-03-06T07:24:17.258133+0000","last_clean":"2024-03-06T07:24:17.258133+0000","last_became_active":"2024-03-06T07:24:03.887681+0000","last_became_peered":"2024-03-06T07:24:03.887681+0000","last_unstale":"2024-03-06T07:24:17.258133+0000","last_undegraded":"2024-03-06T07:24:17.258133+0000","last_fullsized":"2024-03-06T07:24:17.258133+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:32.211048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272424+0000","last_change":"2024-03-06T07:24:17.270677+0000","last_active":"2024-03-06T07:24:17.272424+0000","last_peered":"2024-03-06T07:24:17.272424+0000","last_clean":"2024-03-06T07:24:17.272424+0000","last_became_active":"2024-03-06T07:24:17.270031+0000","last_became_peered":"2024-03-06T07:24:17.270031+0000","last_unstale":"2024-03-06T07:24:17.272424+0000","last_undegraded":"2024-03-06T07:24:17.272424+0000","last_fullsized":"2024-03-06T07:24:17.272424+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:35.720668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256252+0000","last_change":"2024-03-06T07:24:03.885308+0000","last_active":"2024-03-06T07:24:17.256252+0000","last_peered":"2024-03-06T07:24:17.256252+0000","last_clean":"2024-03-06T07:24:17.256252+0000","last_became_active":"2024-03-06T07:24:03.885013+0000","last_became_peered":"2024-03-06T07:24:03.885013+0000","last_unstale":"2024-03-06T07:24:17.256252+0000","last_undegraded":"2024-03-06T07:24:17.256252+0000","last_fullsized":"2024-03-06T07:24:17.256252+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:45.339308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271835+0000","last_change":"2024-03-06T07:24:17.266889+0000","last_active":"2024-03-06T07:24:17.271835+0000","last_peered":"2024-03-06T07:24:17.271835+0000","last_clean":"2024-03-06T07:24:17.271835+0000","last_became_active":"2024-03-06T07:24:17.266598+0000","last_became_peered":"2024-03-06T07:24:17.266598+0000","last_unstale":"2024-03-06T07:24:17.271835+0000","last_undegraded":"2024-03-06T07:24:17.271835+0000","last_fullsized":"2024-03-06T07:24:17.271835+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:11.169809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256191+0000","last_change":"2024-03-06T07:24:03.885984+0000","last_active":"2024-03-06T07:24:17.256191+0000","last_peered":"2024-03-06T07:24:17.256191+0000","last_clean":"2024-03-06T07:24:17.256191+0000","last_became_active":"2024-03-06T07:24:03.885293+0000","last_became_peered":"2024-03-06T07:24:03.885293+0000","last_unstale":"2024-03-06T07:24:17.256191+0000","last_undegraded":"2024-03-06T07:24:17.256191+0000","last_fullsized":"2024-03-06T07:24:17.256191+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:34.171582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271652+0000","last_change":"2024-03-06T07:24:17.268782+0000","last_active":"2024-03-06T07:24:17.271652+0000","last_peered":"2024-03-06T07:24:17.271652+0000","last_clean":"2024-03-06T07:24:17.271652+0000","last_became_active":"2024-03-06T07:24:17.268442+0000","last_became_peered":"2024-03-06T07:24:17.268442+0000","last_unstale":"2024-03-06T07:24:17.271652+0000","last_undegraded":"2024-03-06T07:24:17.271652+0000","last_fullsized":"2024-03-06T07:24:17.271652+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:11.025284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247374+0000","last_change":"2024-03-06T07:24:03.884021+0000","last_active":"2024-03-06T07:24:17.247374+0000","last_peered":"2024-03-06T07:24:17.247374+0000","last_clean":"2024-03-06T07:24:17.247374+0000","last_became_active":"2024-03-06T07:24:03.883424+0000","last_became_peered":"2024-03-06T07:24:03.883424+0000","last_unstale":"2024-03-06T07:24:17.247374+0000","last_undegraded":"2024-03-06T07:24:17.247374+0000","last_fullsized":"2024-03-06T07:24:17.247374+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276015+0000","last_change":"2024-03-06T07:24:17.271626+0000","last_active":"2024-03-06T07:24:17.276015+0000","last_peered":"2024-03-06T07:24:17.276015+0000","last_clean":"2024-03-06T07:24:17.276015+0000","last_became_active":"2024-03-06T07:24:17.270271+0000","last_became_peered":"2024-03-06T07:24:17.270271+0000","last_unstale":"2024-03-06T07:24:17.276015+0000","last_undegraded":"2024-03-06T07:24:17.276015+0000","last_fullsized":"2024-03-06T07:24:17.276015+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:58.896993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257497+0000","last_change":"2024-03-06T07:24:03.889706+0000","last_active":"2024-03-06T07:24:17.257497+0000","last_peered":"2024-03-06T07:24:17.257497+0000","last_clean":"2024-03-06T07:24:17.257497+0000","last_became_active":"2024-03-06T07:24:03.889392+0000","last_became_peered":"2024-03-06T07:24:03.889392+0000","last_unstale":"2024-03-06T07:24:17.257497+0000","last_undegraded":"2024-03-06T07:24:17.257497+0000","last_fullsized":"2024-03-06T07:24:17.257497+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277226+0000","last_change":"2024-03-06T07:24:17.274041+0000","last_active":"2024-03-06T07:24:17.277226+0000","last_peered":"2024-03-06T07:24:17.277226+0000","last_clean":"2024-03-06T07:24:17.277226+0000","last_became_active":"2024-03-06T07:24:17.273745+0000","last_became_peered":"2024-03-06T07:24:17.273745+0000","last_unstale":"2024-03-06T07:24:17.277226+0000","last_undegraded":"2024-03-06T07:24:17.277226+0000","last_fullsized":"2024-03-06T07:24:17.277226+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:01.165624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262013+0000","last_change":"2024-03-06T07:24:03.889999+0000","last_active":"2024-03-06T07:24:17.262013+0000","last_peered":"2024-03-06T07:24:17.262013+0000","last_clean":"2024-03-06T07:24:17.262013+0000","last_became_active":"2024-03-06T07:24:03.889432+0000","last_became_peered":"2024-03-06T07:24:03.889432+0000","last_unstale":"2024-03-06T07:24:17.262013+0000","last_undegraded":"2024-03-06T07:24:17.262013+0000","last_fullsized":"2024-03-06T07:24:17.262013+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.265703+0000","last_change":"2024-03-06T07:24:17.262868+0000","last_active":"2024-03-06T07:24:17.265703+0000","last_peered":"2024-03-06T07:24:17.265703+0000","last_clean":"2024-03-06T07:24:17.265703+0000","last_became_active":"2024-03-06T07:24:17.262581+0000","last_became_peered":"2024-03-06T07:24:17.262581+0000","last_unstale":"2024-03-06T07:24:17.265703+0000","last_undegraded":"2024-03-06T07:24:17.265703+0000","last_fullsized":"2024-03-06T07:24:17.265703+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:45.980040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255844+0000","last_change":"2024-03-06T07:24:03.895907+0000","last_active":"2024-03-06T07:24:17.255844+0000","last_peered":"2024-03-06T07:24:17.255844+0000","last_clean":"2024-03-06T07:24:17.255844+0000","last_became_active":"2024-03-06T07:24:03.895663+0000","last_became_peered":"2024-03-06T07:24:03.895663+0000","last_unstale":"2024-03-06T07:24:17.255844+0000","last_undegraded":"2024-03-06T07:24:17.255844+0000","last_fullsized":"2024-03-06T07:24:17.255844+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.267806+0000","last_change":"2024-03-06T07:24:17.265846+0000","last_active":"2024-03-06T07:24:17.267806+0000","last_peered":"2024-03-06T07:24:17.267806+0000","last_clean":"2024-03-06T07:24:17.267806+0000","last_became_active":"2024-03-06T07:24:17.265556+0000","last_became_peered":"2024-03-06T07:24:17.265556+0000","last_unstale":"2024-03-06T07:24:17.267806+0000","last_undegraded":"2024-03-06T07:24:17.267806+0000","last_fullsized":"2024-03-06T07:24:17.267806+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:53.042094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258297+0000","last_change":"2024-03-06T07:24:03.887528+0000","last_active":"2024-03-06T07:24:17.258297+0000","last_peered":"2024-03-06T07:24:17.258297+0000","last_clean":"2024-03-06T07:24:17.258297+0000","last_became_active":"2024-03-06T07:24:03.887269+0000","last_became_peered":"2024-03-06T07:24:03.887269+0000","last_unstale":"2024-03-06T07:24:17.258297+0000","last_undegraded":"2024-03-06T07:24:17.258297+0000","last_fullsized":"2024-03-06T07:24:17.258297+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:32.211048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272068+0000","last_change":"2024-03-06T07:24:17.267854+0000","last_active":"2024-03-06T07:24:17.272068+0000","last_peered":"2024-03-06T07:24:17.272068+0000","last_clean":"2024-03-06T07:24:17.272068+0000","last_became_active":"2024-03-06T07:24:17.267442+0000","last_became_peered":"2024-03-06T07:24:17.267442+0000","last_unstale":"2024-03-06T07:24:17.272068+0000","last_undegraded":"2024-03-06T07:24:17.272068+0000","last_fullsized":"2024-03-06T07:24:17.272068+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.451737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256461+0000","last_change":"2024-03-06T07:24:03.897717+0000","last_active":"2024-03-06T07:24:17.256461+0000","last_peered":"2024-03-06T07:24:17.256461+0000","last_clean":"2024-03-06T07:24:17.256461+0000","last_became_active":"2024-03-06T07:24:03.897445+0000","last_became_peered":"2024-03-06T07:24:03.897445+0000","last_unstale":"2024-03-06T07:24:17.256461+0000","last_undegraded":"2024-03-06T07:24:17.256461+0000","last_fullsized":"2024-03-06T07:24:17.256461+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.492678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271829+0000","last_change":"2024-03-06T07:24:17.267888+0000","last_active":"2024-03-06T07:24:17.271829+0000","last_peered":"2024-03-06T07:24:17.271829+0000","last_clean":"2024-03-06T07:24:17.271829+0000","last_became_active":"2024-03-06T07:24:17.267656+0000","last_became_peered":"2024-03-06T07:24:17.267656+0000","last_unstale":"2024-03-06T07:24:17.271829+0000","last_undegraded":"2024-03-06T07:24:17.271829+0000","last_fullsized":"2024-03-06T07:24:17.271829+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:14.432264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258213+0000","last_change":"2024-03-06T07:24:03.892692+0000","last_active":"2024-03-06T07:24:17.258213+0000","last_peered":"2024-03-06T07:24:17.258213+0000","last_clean":"2024-03-06T07:24:17.258213+0000","last_became_active":"2024-03-06T07:24:03.892434+0000","last_became_peered":"2024-03-06T07:24:03.892434+0000","last_unstale":"2024-03-06T07:24:17.258213+0000","last_undegraded":"2024-03-06T07:24:17.258213+0000","last_fullsized":"2024-03-06T07:24:17.258213+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:48.146871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277213+0000","last_change":"2024-03-06T07:24:17.273322+0000","last_active":"2024-03-06T07:24:17.277213+0000","last_peered":"2024-03-06T07:24:17.277213+0000","last_clean":"2024-03-06T07:24:17.277213+0000","last_became_active":"2024-03-06T07:24:17.273061+0000","last_became_peered":"2024-03-06T07:24:17.273061+0000","last_unstale":"2024-03-06T07:24:17.277213+0000","last_undegraded":"2024-03-06T07:24:17.277213+0000","last_fullsized":"2024-03-06T07:24:17.277213+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:22.020279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262127+0000","last_change":"2024-03-06T07:24:03.887559+0000","last_active":"2024-03-06T07:24:17.262127+0000","last_peered":"2024-03-06T07:24:17.262127+0000","last_clean":"2024-03-06T07:24:17.262127+0000","last_became_active":"2024-03-06T07:24:03.887275+0000","last_became_peered":"2024-03-06T07:24:03.887275+0000","last_unstale":"2024-03-06T07:24:17.262127+0000","last_undegraded":"2024-03-06T07:24:17.262127+0000","last_fullsized":"2024-03-06T07:24:17.262127+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.492678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271733+0000","last_change":"2024-03-06T07:24:17.265162+0000","last_active":"2024-03-06T07:24:17.271733+0000","last_peered":"2024-03-06T07:24:17.271733+0000","last_clean":"2024-03-06T07:24:17.271733+0000","last_became_active":"2024-03-06T07:24:17.264894+0000","last_became_peered":"2024-03-06T07:24:17.264894+0000","last_unstale":"2024-03-06T07:24:17.271733+0000","last_undegraded":"2024-03-06T07:24:17.271733+0000","last_fullsized":"2024-03-06T07:24:17.271733+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:06.739402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250535+0000","last_change":"2024-03-06T07:24:03.894465+0000","last_active":"2024-03-06T07:24:17.250535+0000","last_peered":"2024-03-06T07:24:17.250535+0000","last_clean":"2024-03-06T07:24:17.250535+0000","last_became_active":"2024-03-06T07:24:03.894192+0000","last_became_peered":"2024-03-06T07:24:03.894192+0000","last_unstale":"2024-03-06T07:24:17.250535+0000","last_undegraded":"2024-03-06T07:24:17.250535+0000","last_fullsized":"2024-03-06T07:24:17.250535+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:48.848727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277189+0000","last_change":"2024-03-06T07:24:17.274474+0000","last_active":"2024-03-06T07:24:17.277189+0000","last_peered":"2024-03-06T07:24:17.277189+0000","last_clean":"2024-03-06T07:24:17.277189+0000","last_became_active":"2024-03-06T07:24:17.274029+0000","last_became_peered":"2024-03-06T07:24:17.274029+0000","last_unstale":"2024-03-06T07:24:17.277189+0000","last_undegraded":"2024-03-06T07:24:17.277189+0000","last_fullsized":"2024-03-06T07:24:17.277189+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:08.004043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247179+0000","last_change":"2024-03-06T07:24:03.890072+0000","last_active":"2024-03-06T07:24:17.247179+0000","last_peered":"2024-03-06T07:24:17.247179+0000","last_clean":"2024-03-06T07:24:17.247179+0000","last_became_active":"2024-03-06T07:24:03.889801+0000","last_became_peered":"2024-03-06T07:24:03.889801+0000","last_unstale":"2024-03-06T07:24:17.247179+0000","last_undegraded":"2024-03-06T07:24:17.247179+0000","last_fullsized":"2024-03-06T07:24:17.247179+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272030+0000","last_change":"2024-03-06T07:24:17.269417+0000","last_active":"2024-03-06T07:24:17.272030+0000","last_peered":"2024-03-06T07:24:17.272030+0000","last_clean":"2024-03-06T07:24:17.272030+0000","last_became_active":"2024-03-06T07:24:17.269019+0000","last_became_peered":"2024-03-06T07:24:17.269019+0000","last_unstale":"2024-03-06T07:24:17.272030+0000","last_undegraded":"2024-03-06T07:24:17.272030+0000","last_fullsized":"2024-03-06T07:24:17.272030+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.975875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258696+0000","last_change":"2024-03-06T07:24:03.887600+0000","last_active":"2024-03-06T07:24:17.258696+0000","last_peered":"2024-03-06T07:24:17.258696+0000","last_clean":"2024-03-06T07:24:17.258696+0000","last_became_active":"2024-03-06T07:24:03.887214+0000","last_became_peered":"2024-03-06T07:24:03.887214+0000","last_unstale":"2024-03-06T07:24:17.258696+0000","last_undegraded":"2024-03-06T07:24:17.258696+0000","last_fullsized":"2024-03-06T07:24:17.258696+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:16.405620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270626+0000","last_change":"2024-03-06T07:24:17.267764+0000","last_active":"2024-03-06T07:24:17.270626+0000","last_peered":"2024-03-06T07:24:17.270626+0000","last_clean":"2024-03-06T07:24:17.270626+0000","last_became_active":"2024-03-06T07:24:17.267472+0000","last_became_peered":"2024-03-06T07:24:17.267472+0000","last_unstale":"2024-03-06T07:24:17.270626+0000","last_undegraded":"2024-03-06T07:24:17.270626+0000","last_fullsized":"2024-03-06T07:24:17.270626+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:17.556623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251671+0000","last_change":"2024-03-06T07:24:03.884458+0000","last_active":"2024-03-06T07:24:17.251671+0000","last_peered":"2024-03-06T07:24:17.251671+0000","last_clean":"2024-03-06T07:24:17.251671+0000","last_became_active":"2024-03-06T07:24:03.883983+0000","last_became_peered":"2024-03-06T07:24:03.883983+0000","last_unstale":"2024-03-06T07:24:17.251671+0000","last_undegraded":"2024-03-06T07:24:17.251671+0000","last_fullsized":"2024-03-06T07:24:17.251671+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:36.691438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.265418+0000","last_change":"2024-03-06T07:24:17.263338+0000","last_active":"2024-03-06T07:24:17.265418+0000","last_peered":"2024-03-06T07:24:17.265418+0000","last_clean":"2024-03-06T07:24:17.265418+0000","last_became_active":"2024-03-06T07:24:17.263048+0000","last_became_peered":"2024-03-06T07:24:17.263048+0000","last_unstale":"2024-03-06T07:24:17.265418+0000","last_undegraded":"2024-03-06T07:24:17.265418+0000","last_fullsized":"2024-03-06T07:24:17.265418+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:49.453665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258388+0000","last_change":"2024-03-06T07:24:03.888459+0000","last_active":"2024-03-06T07:24:17.258388+0000","last_peered":"2024-03-06T07:24:17.258388+0000","last_clean":"2024-03-06T07:24:17.258388+0000","last_became_active":"2024-03-06T07:24:03.888203+0000","last_became_peered":"2024-03-06T07:24:03.888203+0000","last_unstale":"2024-03-06T07:24:17.258388+0000","last_undegraded":"2024-03-06T07:24:17.258388+0000","last_fullsized":"2024-03-06T07:24:17.258388+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:35.615657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276912+0000","last_change":"2024-03-06T07:24:17.271635+0000","last_active":"2024-03-06T07:24:17.276912+0000","last_peered":"2024-03-06T07:24:17.276912+0000","last_clean":"2024-03-06T07:24:17.276912+0000","last_became_active":"2024-03-06T07:24:17.267039+0000","last_became_peered":"2024-03-06T07:24:17.267039+0000","last_unstale":"2024-03-06T07:24:17.276912+0000","last_undegraded":"2024-03-06T07:24:17.276912+0000","last_fullsized":"2024-03-06T07:24:17.276912+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:26.200535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251262+0000","last_change":"2024-03-06T07:24:03.885816+0000","last_active":"2024-03-06T07:24:17.251262+0000","last_peered":"2024-03-06T07:24:17.251262+0000","last_clean":"2024-03-06T07:24:17.251262+0000","last_became_active":"2024-03-06T07:24:03.885561+0000","last_became_peered":"2024-03-06T07:24:03.885561+0000","last_unstale":"2024-03-06T07:24:17.251262+0000","last_undegraded":"2024-03-06T07:24:17.251262+0000","last_fullsized":"2024-03-06T07:24:17.251262+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:33.243851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276893+0000","last_change":"2024-03-06T07:24:17.270974+0000","last_active":"2024-03-06T07:24:17.276893+0000","last_peered":"2024-03-06T07:24:17.276893+0000","last_clean":"2024-03-06T07:24:17.276893+0000","last_became_active":"2024-03-06T07:24:17.270510+0000","last_became_peered":"2024-03-06T07:24:17.270510+0000","last_unstale":"2024-03-06T07:24:17.276893+0000","last_undegraded":"2024-03-06T07:24:17.276893+0000","last_fullsized":"2024-03-06T07:24:17.276893+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:51.222298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261409+0000","last_change":"2024-03-06T07:24:03.887681+0000","last_active":"2024-03-06T07:24:17.261409+0000","last_peered":"2024-03-06T07:24:17.261409+0000","last_clean":"2024-03-06T07:24:17.261409+0000","last_became_active":"2024-03-06T07:24:03.887375+0000","last_became_peered":"2024-03-06T07:24:03.887375+0000","last_unstale":"2024-03-06T07:24:17.261409+0000","last_undegraded":"2024-03-06T07:24:17.261409+0000","last_fullsized":"2024-03-06T07:24:17.261409+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:06.466254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.281203+0000","last_change":"2024-03-06T07:24:17.277664+0000","last_active":"2024-03-06T07:24:17.281203+0000","last_peered":"2024-03-06T07:24:17.281203+0000","last_clean":"2024-03-06T07:24:17.281203+0000","last_became_active":"2024-03-06T07:24:17.277387+0000","last_became_peered":"2024-03-06T07:24:17.277387+0000","last_unstale":"2024-03-06T07:24:17.281203+0000","last_undegraded":"2024-03-06T07:24:17.281203+0000","last_fullsized":"2024-03-06T07:24:17.281203+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:14.626596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250586+0000","last_change":"2024-03-06T07:24:03.884294+0000","last_active":"2024-03-06T07:24:17.250586+0000","last_peered":"2024-03-06T07:24:17.250586+0000","last_clean":"2024-03-06T07:24:17.250586+0000","last_became_active":"2024-03-06T07:24:03.884050+0000","last_became_peered":"2024-03-06T07:24:03.884050+0000","last_unstale":"2024-03-06T07:24:17.250586+0000","last_undegraded":"2024-03-06T07:24:17.250586+0000","last_fullsized":"2024-03-06T07:24:17.250586+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:01.779181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275578+0000","last_change":"2024-03-06T07:24:17.274560+0000","last_active":"2024-03-06T07:24:17.275578+0000","last_peered":"2024-03-06T07:24:17.275578+0000","last_clean":"2024-03-06T07:24:17.275578+0000","last_became_active":"2024-03-06T07:24:17.272120+0000","last_became_peered":"2024-03-06T07:24:17.272120+0000","last_unstale":"2024-03-06T07:24:17.275578+0000","last_undegraded":"2024-03-06T07:24:17.275578+0000","last_fullsized":"2024-03-06T07:24:17.275578+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:43.160668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250625+0000","last_change":"2024-03-06T07:24:03.894163+0000","last_active":"2024-03-06T07:24:17.250625+0000","last_peered":"2024-03-06T07:24:17.250625+0000","last_clean":"2024-03-06T07:24:17.250625+0000","last_became_active":"2024-03-06T07:24:03.893792+0000","last_became_peered":"2024-03-06T07:24:03.893792+0000","last_unstale":"2024-03-06T07:24:17.250625+0000","last_undegraded":"2024-03-06T07:24:17.250625+0000","last_fullsized":"2024-03-06T07:24:17.250625+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:43.594819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271809+0000","last_change":"2024-03-06T07:24:17.267437+0000","last_active":"2024-03-06T07:24:17.271809+0000","last_peered":"2024-03-06T07:24:17.271809+0000","last_clean":"2024-03-06T07:24:17.271809+0000","last_became_active":"2024-03-06T07:24:17.267195+0000","last_became_peered":"2024-03-06T07:24:17.267195+0000","last_unstale":"2024-03-06T07:24:17.271809+0000","last_undegraded":"2024-03-06T07:24:17.271809+0000","last_fullsized":"2024-03-06T07:24:17.271809+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:32.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257601+0000","last_change":"2024-03-06T07:24:03.892392+0000","last_active":"2024-03-06T07:24:17.257601+0000","last_peered":"2024-03-06T07:24:17.257601+0000","last_clean":"2024-03-06T07:24:17.257601+0000","last_became_active":"2024-03-06T07:24:03.892131+0000","last_became_peered":"2024-03-06T07:24:03.892131+0000","last_unstale":"2024-03-06T07:24:17.257601+0000","last_undegraded":"2024-03-06T07:24:17.257601+0000","last_fullsized":"2024-03-06T07:24:17.257601+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271858+0000","last_change":"2024-03-06T07:24:17.268503+0000","last_active":"2024-03-06T07:24:17.271858+0000","last_peered":"2024-03-06T07:24:17.271858+0000","last_clean":"2024-03-06T07:24:17.271858+0000","last_became_active":"2024-03-06T07:24:17.268253+0000","last_became_peered":"2024-03-06T07:24:17.268253+0000","last_unstale":"2024-03-06T07:24:17.271858+0000","last_undegraded":"2024-03-06T07:24:17.271858+0000","last_fullsized":"2024-03-06T07:24:17.271858+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.451737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.246925+0000","last_change":"2024-03-06T07:24:03.890324+0000","last_active":"2024-03-06T07:24:17.246925+0000","last_peered":"2024-03-06T07:24:17.246925+0000","last_clean":"2024-03-06T07:24:17.246925+0000","last_became_active":"2024-03-06T07:24:03.890037+0000","last_became_peered":"2024-03-06T07:24:03.890037+0000","last_unstale":"2024-03-06T07:24:17.246925+0000","last_undegraded":"2024-03-06T07:24:17.246925+0000","last_fullsized":"2024-03-06T07:24:17.246925+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:57.609556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.274406+0000","last_change":"2024-03-06T07:24:17.272731+0000","last_active":"2024-03-06T07:24:17.274406+0000","last_peered":"2024-03-06T07:24:17.274406+0000","last_clean":"2024-03-06T07:24:17.274406+0000","last_became_active":"2024-03-06T07:24:17.272481+0000","last_became_peered":"2024-03-06T07:24:17.272481+0000","last_unstale":"2024-03-06T07:24:17.274406+0000","last_undegraded":"2024-03-06T07:24:17.274406+0000","last_fullsized":"2024-03-06T07:24:17.274406+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:51.196370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251786+0000","last_change":"2024-03-06T07:24:03.884245+0000","last_active":"2024-03-06T07:24:17.251786+0000","last_peered":"2024-03-06T07:24:17.251786+0000","last_clean":"2024-03-06T07:24:17.251786+0000","last_became_active":"2024-03-06T07:24:03.883649+0000","last_became_peered":"2024-03-06T07:24:03.883649+0000","last_unstale":"2024-03-06T07:24:17.251786+0000","last_undegraded":"2024-03-06T07:24:17.251786+0000","last_fullsized":"2024-03-06T07:24:17.251786+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:36.283613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275551+0000","last_change":"2024-03-06T07:24:17.271088+0000","last_active":"2024-03-06T07:24:17.275551+0000","last_peered":"2024-03-06T07:24:17.275551+0000","last_clean":"2024-03-06T07:24:17.275551+0000","last_became_active":"2024-03-06T07:24:17.270793+0000","last_became_peered":"2024-03-06T07:24:17.270793+0000","last_unstale":"2024-03-06T07:24:17.275551+0000","last_undegraded":"2024-03-06T07:24:17.275551+0000","last_fullsized":"2024-03-06T07:24:17.275551+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.629152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251886+0000","last_change":"2024-03-06T07:24:03.898098+0000","last_active":"2024-03-06T07:24:17.251886+0000","last_peered":"2024-03-06T07:24:17.251886+0000","last_clean":"2024-03-06T07:24:17.251886+0000","last_became_active":"2024-03-06T07:24:03.895994+0000","last_became_peered":"2024-03-06T07:24:03.895994+0000","last_unstale":"2024-03-06T07:24:17.251886+0000","last_undegraded":"2024-03-06T07:24:17.251886+0000","last_fullsized":"2024-03-06T07:24:17.251886+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:31.833931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275076+0000","last_change":"2024-03-06T07:24:17.264851+0000","last_active":"2024-03-06T07:24:17.275076+0000","last_peered":"2024-03-06T07:24:17.275076+0000","last_clean":"2024-03-06T07:24:17.275076+0000","last_became_active":"2024-03-06T07:24:17.264602+0000","last_became_peered":"2024-03-06T07:24:17.264602+0000","last_unstale":"2024-03-06T07:24:17.275076+0000","last_undegraded":"2024-03-06T07:24:17.275076+0000","last_fullsized":"2024-03-06T07:24:17.275076+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.099433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.249857+0000","last_change":"2024-03-06T07:24:03.879536+0000","last_active":"2024-03-06T07:24:17.249857+0000","last_peered":"2024-03-06T07:24:17.249857+0000","last_clean":"2024-03-06T07:24:17.249857+0000","last_became_active":"2024-03-06T07:24:03.879280+0000","last_became_peered":"2024-03-06T07:24:03.879280+0000","last_unstale":"2024-03-06T07:24:17.249857+0000","last_undegraded":"2024-03-06T07:24:17.249857+0000","last_fullsized":"2024-03-06T07:24:17.249857+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:09.372273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266178+0000","last_change":"2024-03-06T07:24:17.262753+0000","last_active":"2024-03-06T07:24:17.266178+0000","last_peered":"2024-03-06T07:24:17.266178+0000","last_clean":"2024-03-06T07:24:17.266178+0000","last_became_active":"2024-03-06T07:24:17.262562+0000","last_became_peered":"2024-03-06T07:24:17.262562+0000","last_unstale":"2024-03-06T07:24:17.266178+0000","last_undegraded":"2024-03-06T07:24:17.266178+0000","last_fullsized":"2024-03-06T07:24:17.266178+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:47.972512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263555+0000","last_change":"2024-03-06T07:24:03.893792+0000","last_active":"2024-03-06T07:24:17.263555+0000","last_peered":"2024-03-06T07:24:17.263555+0000","last_clean":"2024-03-06T07:24:17.263555+0000","last_became_active":"2024-03-06T07:24:03.893240+0000","last_became_peered":"2024-03-06T07:24:03.893240+0000","last_unstale":"2024-03-06T07:24:17.263555+0000","last_undegraded":"2024-03-06T07:24:17.263555+0000","last_fullsized":"2024-03-06T07:24:17.263555+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:51.104075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.283587+0000","last_change":"2024-03-06T07:24:17.278327+0000","last_active":"2024-03-06T07:24:17.283587+0000","last_peered":"2024-03-06T07:24:17.283587+0000","last_clean":"2024-03-06T07:24:17.283587+0000","last_became_active":"2024-03-06T07:24:17.277812+0000","last_became_peered":"2024-03-06T07:24:17.277812+0000","last_unstale":"2024-03-06T07:24:17.283587+0000","last_undegraded":"2024-03-06T07:24:17.283587+0000","last_fullsized":"2024-03-06T07:24:17.283587+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:40.803855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250320+0000","last_change":"2024-03-06T07:24:03.894449+0000","last_active":"2024-03-06T07:24:17.250320+0000","last_peered":"2024-03-06T07:24:17.250320+0000","last_clean":"2024-03-06T07:24:17.250320+0000","last_became_active":"2024-03-06T07:24:03.894007+0000","last_became_peered":"2024-03-06T07:24:03.894007+0000","last_unstale":"2024-03-06T07:24:17.250320+0000","last_undegraded":"2024-03-06T07:24:17.250320+0000","last_fullsized":"2024-03-06T07:24:17.250320+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:37.861026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266260+0000","last_change":"2024-03-06T07:24:17.264381+0000","last_active":"2024-03-06T07:24:17.266260+0000","last_peered":"2024-03-06T07:24:17.266260+0000","last_clean":"2024-03-06T07:24:17.266260+0000","last_became_active":"2024-03-06T07:24:17.264124+0000","last_became_peered":"2024-03-06T07:24:17.264124+0000","last_unstale":"2024-03-06T07:24:17.266260+0000","last_undegraded":"2024-03-06T07:24:17.266260+0000","last_fullsized":"2024-03-06T07:24:17.266260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:46.610169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262851+0000","last_change":"2024-03-06T07:24:03.887453+0000","last_active":"2024-03-06T07:24:17.262851+0000","last_peered":"2024-03-06T07:24:17.262851+0000","last_clean":"2024-03-06T07:24:17.262851+0000","last_became_active":"2024-03-06T07:24:03.887167+0000","last_became_peered":"2024-03-06T07:24:03.887167+0000","last_unstale":"2024-03-06T07:24:17.262851+0000","last_undegraded":"2024-03-06T07:24:17.262851+0000","last_fullsized":"2024-03-06T07:24:17.262851+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:01.897485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277168+0000","last_change":"2024-03-06T07:24:17.274281+0000","last_active":"2024-03-06T07:24:17.277168+0000","last_peered":"2024-03-06T07:24:17.277168+0000","last_clean":"2024-03-06T07:24:17.277168+0000","last_became_active":"2024-03-06T07:24:17.273563+0000","last_became_peered":"2024-03-06T07:24:17.273563+0000","last_unstale":"2024-03-06T07:24:17.277168+0000","last_undegraded":"2024-03-06T07:24:17.277168+0000","last_fullsized":"2024-03-06T07:24:17.277168+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:56.473600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251079+0000","last_change":"2024-03-06T07:24:03.885011+0000","last_active":"2024-03-06T07:24:17.251079+0000","last_peered":"2024-03-06T07:24:17.251079+0000","last_clean":"2024-03-06T07:24:17.251079+0000","last_became_active":"2024-03-06T07:24:03.884571+0000","last_became_peered":"2024-03-06T07:24:03.884571+0000","last_unstale":"2024-03-06T07:24:17.251079+0000","last_undegraded":"2024-03-06T07:24:17.251079+0000","last_fullsized":"2024-03-06T07:24:17.251079+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:12.327628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278283+0000","last_change":"2024-03-06T07:24:17.272899+0000","last_active":"2024-03-06T07:24:17.278283+0000","last_peered":"2024-03-06T07:24:17.278283+0000","last_clean":"2024-03-06T07:24:17.278283+0000","last_became_active":"2024-03-06T07:24:17.272624+0000","last_became_peered":"2024-03-06T07:24:17.272624+0000","last_unstale":"2024-03-06T07:24:17.278283+0000","last_undegraded":"2024-03-06T07:24:17.278283+0000","last_fullsized":"2024-03-06T07:24:17.278283+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:15.051999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258789+0000","last_change":"2024-03-06T07:24:03.891048+0000","last_active":"2024-03-06T07:24:17.258789+0000","last_peered":"2024-03-06T07:24:17.258789+0000","last_clean":"2024-03-06T07:24:17.258789+0000","last_became_active":"2024-03-06T07:24:03.890787+0000","last_became_peered":"2024-03-06T07:24:03.890787+0000","last_unstale":"2024-03-06T07:24:17.258789+0000","last_undegraded":"2024-03-06T07:24:17.258789+0000","last_fullsized":"2024-03-06T07:24:17.258789+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:39.286189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278425+0000","last_change":"2024-03-06T07:24:17.273563+0000","last_active":"2024-03-06T07:24:17.278425+0000","last_peered":"2024-03-06T07:24:17.278425+0000","last_clean":"2024-03-06T07:24:17.278425+0000","last_became_active":"2024-03-06T07:24:17.273293+0000","last_became_peered":"2024-03-06T07:24:17.273293+0000","last_unstale":"2024-03-06T07:24:17.278425+0000","last_undegraded":"2024-03-06T07:24:17.278425+0000","last_fullsized":"2024-03-06T07:24:17.278425+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.546588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252848+0000","last_change":"2024-03-06T07:24:03.884181+0000","last_active":"2024-03-06T07:24:17.252848+0000","last_peered":"2024-03-06T07:24:17.252848+0000","last_clean":"2024-03-06T07:24:17.252848+0000","last_became_active":"2024-03-06T07:24:03.883923+0000","last_became_peered":"2024-03-06T07:24:03.883923+0000","last_unstale":"2024-03-06T07:24:17.252848+0000","last_undegraded":"2024-03-06T07:24:17.252848+0000","last_fullsized":"2024-03-06T07:24:17.252848+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:04.190305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266571+0000","last_change":"2024-03-06T07:24:17.263544+0000","last_active":"2024-03-06T07:24:17.266571+0000","last_peered":"2024-03-06T07:24:17.266571+0000","last_clean":"2024-03-06T07:24:17.266571+0000","last_became_active":"2024-03-06T07:24:17.263162+0000","last_became_peered":"2024-03-06T07:24:17.263162+0000","last_unstale":"2024-03-06T07:24:17.266571+0000","last_undegraded":"2024-03-06T07:24:17.266571+0000","last_fullsized":"2024-03-06T07:24:17.266571+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.502877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.246963+0000","last_change":"2024-03-06T07:24:16.249577+0000","last_active":"2024-03-06T07:24:17.246963+0000","last_peered":"2024-03-06T07:24:17.246963+0000","last_clean":"2024-03-06T07:24:17.246963+0000","last_became_active":"2024-03-06T07:24:16.249271+0000","last_became_peered":"2024-03-06T07:24:16.249271+0000","last_unstale":"2024-03-06T07:24:17.246963+0000","last_undegraded":"2024-03-06T07:24:17.246963+0000","last_fullsized":"2024-03-06T07:24:17.246963+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:39.799276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258184+0000","last_change":"2024-03-06T07:24:03.893008+0000","last_active":"2024-03-06T07:24:17.258184+0000","last_peered":"2024-03-06T07:24:17.258184+0000","last_clean":"2024-03-06T07:24:17.258184+0000","last_became_active":"2024-03-06T07:24:03.892568+0000","last_became_peered":"2024-03-06T07:24:03.892568+0000","last_unstale":"2024-03-06T07:24:17.258184+0000","last_undegraded":"2024-03-06T07:24:17.258184+0000","last_fullsized":"2024-03-06T07:24:17.258184+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":24,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251820+0000","last_change":"2024-03-06T07:24:02.889021+0000","last_active":"2024-03-06T07:24:17.251820+0000","last_peered":"2024-03-06T07:24:17.251820+0000","last_clean":"2024-03-06T07:24:17.251820+0000","last_became_active":"2024-03-06T07:24:02.888760+0000","last_became_peered":"2024-03-06T07:24:02.888760+0000","last_unstale":"2024-03-06T07:24:17.251820+0000","last_undegraded":"2024-03-06T07:24:17.251820+0000","last_fullsized":"2024-03-06T07:24:17.251820+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:56.709899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266608+0000","last_change":"2024-03-06T07:24:17.262650+0000","last_active":"2024-03-06T07:24:17.266608+0000","last_peered":"2024-03-06T07:24:17.266608+0000","last_clean":"2024-03-06T07:24:17.266608+0000","last_became_active":"2024-03-06T07:24:17.262347+0000","last_became_peered":"2024-03-06T07:24:17.262347+0000","last_unstale":"2024-03-06T07:24:17.266608+0000","last_undegraded":"2024-03-06T07:24:17.266608+0000","last_fullsized":"2024-03-06T07:24:17.266608+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:29.771938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"152'8","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.281141+0000","last_change":"2024-03-06T07:24:16.254291+0000","last_active":"2024-03-06T07:24:17.281141+0000","last_peered":"2024-03-06T07:24:17.281141+0000","last_clean":"2024-03-06T07:24:17.281141+0000","last_became_active":"2024-03-06T07:24:16.253983+0000","last_became_peered":"2024-03-06T07:24:16.253983+0000","last_unstale":"2024-03-06T07:24:17.281141+0000","last_undegraded":"2024-03-06T07:24:17.281141+0000","last_fullsized":"2024-03-06T07:24:17.281141+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:47.977396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250274+0000","last_change":"2024-03-06T07:24:03.895263+0000","last_active":"2024-03-06T07:24:17.250274+0000","last_peered":"2024-03-06T07:24:17.250274+0000","last_clean":"2024-03-06T07:24:17.250274+0000","last_became_active":"2024-03-06T07:24:03.894999+0000","last_became_peered":"2024-03-06T07:24:03.894999+0000","last_unstale":"2024-03-06T07:24:17.250274+0000","last_undegraded":"2024-03-06T07:24:17.250274+0000","last_fullsized":"2024-03-06T07:24:17.250274+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:37.830052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.248048+0000","last_change":"2024-03-06T07:24:02.881058+0000","last_active":"2024-03-06T07:24:17.248048+0000","last_peered":"2024-03-06T07:24:17.248048+0000","last_clean":"2024-03-06T07:24:17.248048+0000","last_became_active":"2024-03-06T07:24:02.880664+0000","last_became_peered":"2024-03-06T07:24:02.880664+0000","last_unstale":"2024-03-06T07:24:17.248048+0000","last_undegraded":"2024-03-06T07:24:17.248048+0000","last_fullsized":"2024-03-06T07:24:17.248048+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:06.978957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271616+0000","last_change":"2024-03-06T07:24:17.269631+0000","last_active":"2024-03-06T07:24:17.271616+0000","last_peered":"2024-03-06T07:24:17.271616+0000","last_clean":"2024-03-06T07:24:17.271616+0000","last_became_active":"2024-03-06T07:24:17.269359+0000","last_became_peered":"2024-03-06T07:24:17.269359+0000","last_unstale":"2024-03-06T07:24:17.271616+0000","last_undegraded":"2024-03-06T07:24:17.271616+0000","last_fullsized":"2024-03-06T07:24:17.271616+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:58.820106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278095+0000","last_change":"2024-03-06T07:24:16.254093+0000","last_active":"2024-03-06T07:24:17.278095+0000","last_peered":"2024-03-06T07:24:17.278095+0000","last_clean":"2024-03-06T07:24:17.278095+0000","last_became_active":"2024-03-06T07:24:16.253786+0000","last_became_peered":"2024-03-06T07:24:16.253786+0000","last_unstale":"2024-03-06T07:24:17.278095+0000","last_undegraded":"2024-03-06T07:24:17.278095+0000","last_fullsized":"2024-03-06T07:24:17.278095+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:56.105200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247276+0000","last_change":"2024-03-06T07:24:03.890715+0000","last_active":"2024-03-06T07:24:17.247276+0000","last_peered":"2024-03-06T07:24:17.247276+0000","last_clean":"2024-03-06T07:24:17.247276+0000","last_became_active":"2024-03-06T07:24:03.890464+0000","last_became_peered":"2024-03-06T07:24:03.890464+0000","last_unstale":"2024-03-06T07:24:17.247276+0000","last_undegraded":"2024-03-06T07:24:17.247276+0000","last_fullsized":"2024-03-06T07:24:17.247276+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261609+0000","last_change":"2024-03-06T07:24:02.881481+0000","last_active":"2024-03-06T07:24:17.261609+0000","last_peered":"2024-03-06T07:24:17.261609+0000","last_clean":"2024-03-06T07:24:17.261609+0000","last_became_active":"2024-03-06T07:24:02.881205+0000","last_became_peered":"2024-03-06T07:24:02.881205+0000","last_unstale":"2024-03-06T07:24:17.261609+0000","last_undegraded":"2024-03-06T07:24:17.261609+0000","last_fullsized":"2024-03-06T07:24:17.261609+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:01.647744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262137+0000","last_change":"2024-03-06T07:24:17.258522+0000","last_active":"2024-03-06T07:24:17.262137+0000","last_peered":"2024-03-06T07:24:17.262137+0000","last_clean":"2024-03-06T07:24:17.262137+0000","last_became_active":"2024-03-06T07:24:17.257291+0000","last_became_peered":"2024-03-06T07:24:17.257291+0000","last_unstale":"2024-03-06T07:24:17.262137+0000","last_undegraded":"2024-03-06T07:24:17.262137+0000","last_fullsized":"2024-03-06T07:24:17.262137+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:53.021295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257950+0000","last_change":"2024-03-06T07:24:16.254272+0000","last_active":"2024-03-06T07:24:17.257950+0000","last_peered":"2024-03-06T07:24:17.257950+0000","last_clean":"2024-03-06T07:24:17.257950+0000","last_became_active":"2024-03-06T07:24:16.254030+0000","last_became_peered":"2024-03-06T07:24:16.254030+0000","last_unstale":"2024-03-06T07:24:17.257950+0000","last_undegraded":"2024-03-06T07:24:17.257950+0000","last_fullsized":"2024-03-06T07:24:17.257950+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:39.799276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262934+0000","last_change":"2024-03-06T07:24:03.892672+0000","last_active":"2024-03-06T07:24:17.262934+0000","last_peered":"2024-03-06T07:24:17.262934+0000","last_clean":"2024-03-06T07:24:17.262934+0000","last_became_active":"2024-03-06T07:24:03.891741+0000","last_became_peered":"2024-03-06T07:24:03.891741+0000","last_unstale":"2024-03-06T07:24:17.262934+0000","last_undegraded":"2024-03-06T07:24:17.262934+0000","last_fullsized":"2024-03-06T07:24:17.262934+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:39.675776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250680+0000","last_change":"2024-03-06T07:24:02.880928+0000","last_active":"2024-03-06T07:24:17.250680+0000","last_peered":"2024-03-06T07:24:17.250680+0000","last_clean":"2024-03-06T07:24:17.250680+0000","last_became_active":"2024-03-06T07:24:02.880661+0000","last_became_peered":"2024-03-06T07:24:02.880661+0000","last_unstale":"2024-03-06T07:24:17.250680+0000","last_undegraded":"2024-03-06T07:24:17.250680+0000","last_fullsized":"2024-03-06T07:24:17.250680+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:01.167170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272120+0000","last_change":"2024-03-06T07:24:17.269178+0000","last_active":"2024-03-06T07:24:17.272120+0000","last_peered":"2024-03-06T07:24:17.272120+0000","last_clean":"2024-03-06T07:24:17.272120+0000","last_became_active":"2024-03-06T07:24:17.267893+0000","last_became_peered":"2024-03-06T07:24:17.267893+0000","last_unstale":"2024-03-06T07:24:17.272120+0000","last_undegraded":"2024-03-06T07:24:17.272120+0000","last_fullsized":"2024-03-06T07:24:17.272120+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.828239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"152'3","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277453+0000","last_change":"2024-03-06T07:24:16.250987+0000","last_active":"2024-03-06T07:24:17.277453+0000","last_peered":"2024-03-06T07:24:17.277453+0000","last_clean":"2024-03-06T07:24:17.277453+0000","last_became_active":"2024-03-06T07:24:16.250742+0000","last_became_peered":"2024-03-06T07:24:16.250742+0000","last_unstale":"2024-03-06T07:24:17.277453+0000","last_undegraded":"2024-03-06T07:24:17.277453+0000","last_fullsized":"2024-03-06T07:24:17.277453+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:52.772965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256464+0000","last_change":"2024-03-06T07:24:03.898477+0000","last_active":"2024-03-06T07:24:17.256464+0000","last_peered":"2024-03-06T07:24:17.256464+0000","last_clean":"2024-03-06T07:24:17.256464+0000","last_became_active":"2024-03-06T07:24:03.897814+0000","last_became_peered":"2024-03-06T07:24:03.897814+0000","last_unstale":"2024-03-06T07:24:17.256464+0000","last_undegraded":"2024-03-06T07:24:17.256464+0000","last_fullsized":"2024-03-06T07:24:17.256464+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:51.104075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256172+0000","last_change":"2024-03-06T07:24:02.878433+0000","last_active":"2024-03-06T07:24:17.256172+0000","last_peered":"2024-03-06T07:24:17.256172+0000","last_clean":"2024-03-06T07:24:17.256172+0000","last_became_active":"2024-03-06T07:24:02.878153+0000","last_became_peered":"2024-03-06T07:24:02.878153+0000","last_unstale":"2024-03-06T07:24:17.256172+0000","last_undegraded":"2024-03-06T07:24:17.256172+0000","last_fullsized":"2024-03-06T07:24:17.256172+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:27.611384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272444+0000","last_change":"2024-03-06T07:24:17.270886+0000","last_active":"2024-03-06T07:24:17.272444+0000","last_peered":"2024-03-06T07:24:17.272444+0000","last_clean":"2024-03-06T07:24:17.272444+0000","last_became_active":"2024-03-06T07:24:17.270361+0000","last_became_peered":"2024-03-06T07:24:17.270361+0000","last_unstale":"2024-03-06T07:24:17.272444+0000","last_undegraded":"2024-03-06T07:24:17.272444+0000","last_fullsized":"2024-03-06T07:24:17.272444+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:47.029222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"152'6","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.283925+0000","last_change":"2024-03-06T07:24:16.250814+0000","last_active":"2024-03-06T07:24:17.283925+0000","last_peered":"2024-03-06T07:24:17.283925+0000","last_clean":"2024-03-06T07:24:17.283925+0000","last_became_active":"2024-03-06T07:24:16.250538+0000","last_became_peered":"2024-03-06T07:24:16.250538+0000","last_unstale":"2024-03-06T07:24:17.283925+0000","last_undegraded":"2024-03-06T07:24:17.283925+0000","last_fullsized":"2024-03-06T07:24:17.283925+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:20.346111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251927+0000","last_change":"2024-03-06T07:24:03.891217+0000","last_active":"2024-03-06T07:24:17.251927+0000","last_peered":"2024-03-06T07:24:17.251927+0000","last_clean":"2024-03-06T07:24:17.251927+0000","last_became_active":"2024-03-06T07:24:03.890968+0000","last_became_peered":"2024-03-06T07:24:03.890968+0000","last_unstale":"2024-03-06T07:24:17.251927+0000","last_undegraded":"2024-03-06T07:24:17.251927+0000","last_fullsized":"2024-03-06T07:24:17.251927+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:26.282549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247739+0000","last_change":"2024-03-06T07:24:02.890689+0000","last_active":"2024-03-06T07:24:17.247739+0000","last_peered":"2024-03-06T07:24:17.247739+0000","last_clean":"2024-03-06T07:24:17.247739+0000","last_became_active":"2024-03-06T07:24:02.890424+0000","last_became_peered":"2024-03-06T07:24:02.890424+0000","last_unstale":"2024-03-06T07:24:17.247739+0000","last_undegraded":"2024-03-06T07:24:17.247739+0000","last_fullsized":"2024-03-06T07:24:17.247739+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.280887+0000","last_change":"2024-03-06T07:24:17.276994+0000","last_active":"2024-03-06T07:24:17.280887+0000","last_peered":"2024-03-06T07:24:17.280887+0000","last_clean":"2024-03-06T07:24:17.280887+0000","last_became_active":"2024-03-06T07:24:17.276715+0000","last_became_peered":"2024-03-06T07:24:17.276715+0000","last_unstale":"2024-03-06T07:24:17.280887+0000","last_undegraded":"2024-03-06T07:24:17.280887+0000","last_fullsized":"2024-03-06T07:24:17.280887+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:32.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251679+0000","last_change":"2024-03-06T07:24:16.253614+0000","last_active":"2024-03-06T07:24:17.251679+0000","last_peered":"2024-03-06T07:24:17.251679+0000","last_clean":"2024-03-06T07:24:17.251679+0000","last_became_active":"2024-03-06T07:24:16.253341+0000","last_became_peered":"2024-03-06T07:24:16.253341+0000","last_unstale":"2024-03-06T07:24:17.251679+0000","last_undegraded":"2024-03-06T07:24:17.251679+0000","last_fullsized":"2024-03-06T07:24:17.251679+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:19.541026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252113+0000","last_change":"2024-03-06T07:24:03.878910+0000","last_active":"2024-03-06T07:24:17.252113+0000","last_peered":"2024-03-06T07:24:17.252113+0000","last_clean":"2024-03-06T07:24:17.252113+0000","last_became_active":"2024-03-06T07:24:03.878660+0000","last_became_peered":"2024-03-06T07:24:03.878660+0000","last_unstale":"2024-03-06T07:24:17.252113+0000","last_undegraded":"2024-03-06T07:24:17.252113+0000","last_fullsized":"2024-03-06T07:24:17.252113+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:52.217554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250528+0000","last_change":"2024-03-06T07:24:02.891744+0000","last_active":"2024-03-06T07:24:17.250528+0000","last_peered":"2024-03-06T07:24:17.250528+0000","last_clean":"2024-03-06T07:24:17.250528+0000","last_became_active":"2024-03-06T07:24:02.891480+0000","last_became_peered":"2024-03-06T07:24:02.891480+0000","last_unstale":"2024-03-06T07:24:17.250528+0000","last_undegraded":"2024-03-06T07:24:17.250528+0000","last_fullsized":"2024-03-06T07:24:17.250528+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:07.998051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262167+0000","last_change":"2024-03-06T07:24:17.256920+0000","last_active":"2024-03-06T07:24:17.262167+0000","last_peered":"2024-03-06T07:24:17.262167+0000","last_clean":"2024-03-06T07:24:17.262167+0000","last_became_active":"2024-03-06T07:24:17.256642+0000","last_became_peered":"2024-03-06T07:24:17.256642+0000","last_unstale":"2024-03-06T07:24:17.262167+0000","last_undegraded":"2024-03-06T07:24:17.262167+0000","last_fullsized":"2024-03-06T07:24:17.262167+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:03.211317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257581+0000","last_change":"2024-03-06T07:24:16.257197+0000","last_active":"2024-03-06T07:24:17.257581+0000","last_peered":"2024-03-06T07:24:17.257581+0000","last_clean":"2024-03-06T07:24:17.257581+0000","last_became_active":"2024-03-06T07:24:16.256944+0000","last_became_peered":"2024-03-06T07:24:16.256944+0000","last_unstale":"2024-03-06T07:24:17.257581+0000","last_undegraded":"2024-03-06T07:24:17.257581+0000","last_fullsized":"2024-03-06T07:24:17.257581+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:12.911145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251861+0000","last_change":"2024-03-06T07:24:03.885245+0000","last_active":"2024-03-06T07:24:17.251861+0000","last_peered":"2024-03-06T07:24:17.251861+0000","last_clean":"2024-03-06T07:24:17.251861+0000","last_became_active":"2024-03-06T07:24:03.884959+0000","last_became_peered":"2024-03-06T07:24:03.884959+0000","last_unstale":"2024-03-06T07:24:17.251861+0000","last_undegraded":"2024-03-06T07:24:17.251861+0000","last_fullsized":"2024-03-06T07:24:17.251861+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:44.253526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256036+0000","last_change":"2024-03-06T07:24:02.886573+0000","last_active":"2024-03-06T07:24:17.256036+0000","last_peered":"2024-03-06T07:24:17.256036+0000","last_clean":"2024-03-06T07:24:17.256036+0000","last_became_active":"2024-03-06T07:24:02.885928+0000","last_became_peered":"2024-03-06T07:24:02.885928+0000","last_unstale":"2024-03-06T07:24:17.256036+0000","last_undegraded":"2024-03-06T07:24:17.256036+0000","last_fullsized":"2024-03-06T07:24:17.256036+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271873+0000","last_change":"2024-03-06T07:24:17.265532+0000","last_active":"2024-03-06T07:24:17.271873+0000","last_peered":"2024-03-06T07:24:17.271873+0000","last_clean":"2024-03-06T07:24:17.271873+0000","last_became_active":"2024-03-06T07:24:17.265093+0000","last_became_peered":"2024-03-06T07:24:17.265093+0000","last_unstale":"2024-03-06T07:24:17.271873+0000","last_undegraded":"2024-03-06T07:24:17.271873+0000","last_fullsized":"2024-03-06T07:24:17.271873+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:58.896993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253042+0000","last_change":"2024-03-06T07:24:16.250130+0000","last_active":"2024-03-06T07:24:17.253042+0000","last_peered":"2024-03-06T07:24:17.253042+0000","last_clean":"2024-03-06T07:24:17.253042+0000","last_became_active":"2024-03-06T07:24:16.249680+0000","last_became_peered":"2024-03-06T07:24:16.249680+0000","last_unstale":"2024-03-06T07:24:17.253042+0000","last_undegraded":"2024-03-06T07:24:17.253042+0000","last_fullsized":"2024-03-06T07:24:17.253042+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:28.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257738+0000","last_change":"2024-03-06T07:24:03.892190+0000","last_active":"2024-03-06T07:24:17.257738+0000","last_peered":"2024-03-06T07:24:17.257738+0000","last_clean":"2024-03-06T07:24:17.257738+0000","last_became_active":"2024-03-06T07:24:03.891822+0000","last_became_peered":"2024-03-06T07:24:03.891822+0000","last_unstale":"2024-03-06T07:24:17.257738+0000","last_undegraded":"2024-03-06T07:24:17.257738+0000","last_fullsized":"2024-03-06T07:24:17.257738+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251798+0000","last_change":"2024-03-06T07:24:02.892342+0000","last_active":"2024-03-06T07:24:17.251798+0000","last_peered":"2024-03-06T07:24:17.251798+0000","last_clean":"2024-03-06T07:24:17.251798+0000","last_became_active":"2024-03-06T07:24:02.892089+0000","last_became_peered":"2024-03-06T07:24:02.892089+0000","last_unstale":"2024-03-06T07:24:17.251798+0000","last_undegraded":"2024-03-06T07:24:17.251798+0000","last_fullsized":"2024-03-06T07:24:17.251798+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270328+0000","last_change":"2024-03-06T07:24:17.266037+0000","last_active":"2024-03-06T07:24:17.270328+0000","last_peered":"2024-03-06T07:24:17.270328+0000","last_clean":"2024-03-06T07:24:17.270328+0000","last_became_active":"2024-03-06T07:24:17.265742+0000","last_became_peered":"2024-03-06T07:24:17.265742+0000","last_unstale":"2024-03-06T07:24:17.270328+0000","last_undegraded":"2024-03-06T07:24:17.270328+0000","last_fullsized":"2024-03-06T07:24:17.270328+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:19.832573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.281162+0000","last_change":"2024-03-06T07:24:16.256325+0000","last_active":"2024-03-06T07:24:17.281162+0000","last_peered":"2024-03-06T07:24:17.281162+0000","last_clean":"2024-03-06T07:24:17.281162+0000","last_became_active":"2024-03-06T07:24:16.255909+0000","last_became_peered":"2024-03-06T07:24:16.255909+0000","last_unstale":"2024-03-06T07:24:17.281162+0000","last_undegraded":"2024-03-06T07:24:17.281162+0000","last_fullsized":"2024-03-06T07:24:17.281162+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:41.994723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251854+0000","last_change":"2024-03-06T07:24:03.892347+0000","last_active":"2024-03-06T07:24:17.251854+0000","last_peered":"2024-03-06T07:24:17.251854+0000","last_clean":"2024-03-06T07:24:17.251854+0000","last_became_active":"2024-03-06T07:24:03.892074+0000","last_became_peered":"2024-03-06T07:24:03.892074+0000","last_unstale":"2024-03-06T07:24:17.251854+0000","last_undegraded":"2024-03-06T07:24:17.251854+0000","last_fullsized":"2024-03-06T07:24:17.251854+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:20.749704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":23,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255891+0000","last_change":"2024-03-06T07:24:02.886419+0000","last_active":"2024-03-06T07:24:17.255891+0000","last_peered":"2024-03-06T07:24:17.255891+0000","last_clean":"2024-03-06T07:24:17.255891+0000","last_became_active":"2024-03-06T07:24:02.886064+0000","last_became_peered":"2024-03-06T07:24:02.886064+0000","last_unstale":"2024-03-06T07:24:17.255891+0000","last_undegraded":"2024-03-06T07:24:17.255891+0000","last_fullsized":"2024-03-06T07:24:17.255891+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:29.962503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271712+0000","last_change":"2024-03-06T07:24:17.264408+0000","last_active":"2024-03-06T07:24:17.271712+0000","last_peered":"2024-03-06T07:24:17.271712+0000","last_clean":"2024-03-06T07:24:17.271712+0000","last_became_active":"2024-03-06T07:24:17.263510+0000","last_became_peered":"2024-03-06T07:24:17.263510+0000","last_unstale":"2024-03-06T07:24:17.271712+0000","last_undegraded":"2024-03-06T07:24:17.271712+0000","last_fullsized":"2024-03-06T07:24:17.271712+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.777544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277491+0000","last_change":"2024-03-06T07:24:16.252774+0000","last_active":"2024-03-06T07:24:17.277491+0000","last_peered":"2024-03-06T07:24:17.277491+0000","last_clean":"2024-03-06T07:24:17.277491+0000","last_became_active":"2024-03-06T07:24:16.252533+0000","last_became_peered":"2024-03-06T07:24:16.252533+0000","last_unstale":"2024-03-06T07:24:17.277491+0000","last_undegraded":"2024-03-06T07:24:17.277491+0000","last_fullsized":"2024-03-06T07:24:17.277491+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:19.586046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.249760+0000","last_change":"2024-03-06T07:24:03.886038+0000","last_active":"2024-03-06T07:24:17.249760+0000","last_peered":"2024-03-06T07:24:17.249760+0000","last_clean":"2024-03-06T07:24:17.249760+0000","last_became_active":"2024-03-06T07:24:03.885779+0000","last_became_peered":"2024-03-06T07:24:03.885779+0000","last_unstale":"2024-03-06T07:24:17.249760+0000","last_undegraded":"2024-03-06T07:24:17.249760+0000","last_fullsized":"2024-03-06T07:24:17.249760+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:01.188435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":26,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250914+0000","last_change":"2024-03-06T07:24:02.889795+0000","last_active":"2024-03-06T07:24:17.250914+0000","last_peered":"2024-03-06T07:24:17.250914+0000","last_clean":"2024-03-06T07:24:17.250914+0000","last_became_active":"2024-03-06T07:24:02.889534+0000","last_became_peered":"2024-03-06T07:24:02.889534+0000","last_unstale":"2024-03-06T07:24:17.250914+0000","last_undegraded":"2024-03-06T07:24:17.250914+0000","last_fullsized":"2024-03-06T07:24:17.250914+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:26.432122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.274796+0000","last_change":"2024-03-06T07:24:17.270359+0000","last_active":"2024-03-06T07:24:17.274796+0000","last_peered":"2024-03-06T07:24:17.274796+0000","last_clean":"2024-03-06T07:24:17.274796+0000","last_became_active":"2024-03-06T07:24:17.269845+0000","last_became_peered":"2024-03-06T07:24:17.269845+0000","last_unstale":"2024-03-06T07:24:17.274796+0000","last_undegraded":"2024-03-06T07:24:17.274796+0000","last_fullsized":"2024-03-06T07:24:17.274796+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:42.029712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.280998+0000","last_change":"2024-03-06T07:24:16.251053+0000","last_active":"2024-03-06T07:24:17.280998+0000","last_peered":"2024-03-06T07:24:17.280998+0000","last_clean":"2024-03-06T07:24:17.280998+0000","last_became_active":"2024-03-06T07:24:16.250601+0000","last_became_peered":"2024-03-06T07:24:16.250601+0000","last_unstale":"2024-03-06T07:24:17.280998+0000","last_undegraded":"2024-03-06T07:24:17.280998+0000","last_fullsized":"2024-03-06T07:24:17.280998+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:44.572787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251088+0000","last_change":"2024-03-06T07:24:03.883274+0000","last_active":"2024-03-06T07:24:17.251088+0000","last_peered":"2024-03-06T07:24:17.251088+0000","last_clean":"2024-03-06T07:24:17.251088+0000","last_became_active":"2024-03-06T07:24:03.882970+0000","last_became_peered":"2024-03-06T07:24:03.882970+0000","last_unstale":"2024-03-06T07:24:17.251088+0000","last_undegraded":"2024-03-06T07:24:17.251088+0000","last_fullsized":"2024-03-06T07:24:17.251088+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:34.606194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247467+0000","last_change":"2024-03-06T07:24:02.886772+0000","last_active":"2024-03-06T07:24:17.247467+0000","last_peered":"2024-03-06T07:24:17.247467+0000","last_clean":"2024-03-06T07:24:17.247467+0000","last_became_active":"2024-03-06T07:24:02.886444+0000","last_became_peered":"2024-03-06T07:24:02.886444+0000","last_unstale":"2024-03-06T07:24:17.247467+0000","last_undegraded":"2024-03-06T07:24:17.247467+0000","last_fullsized":"2024-03-06T07:24:17.247467+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:14.833238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271526+0000","last_change":"2024-03-06T07:24:17.267411+0000","last_active":"2024-03-06T07:24:17.271526+0000","last_peered":"2024-03-06T07:24:17.271526+0000","last_clean":"2024-03-06T07:24:17.271526+0000","last_became_active":"2024-03-06T07:24:17.267128+0000","last_became_peered":"2024-03-06T07:24:17.267128+0000","last_unstale":"2024-03-06T07:24:17.271526+0000","last_undegraded":"2024-03-06T07:24:17.271526+0000","last_fullsized":"2024-03-06T07:24:17.271526+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:16.019477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277966+0000","last_change":"2024-03-06T07:24:16.255311+0000","last_active":"2024-03-06T07:24:17.277966+0000","last_peered":"2024-03-06T07:24:17.277966+0000","last_clean":"2024-03-06T07:24:17.277966+0000","last_became_active":"2024-03-06T07:24:16.254966+0000","last_became_peered":"2024-03-06T07:24:16.254966+0000","last_unstale":"2024-03-06T07:24:17.277966+0000","last_undegraded":"2024-03-06T07:24:17.277966+0000","last_fullsized":"2024-03-06T07:24:17.277966+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:51.647928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256068+0000","last_change":"2024-03-06T07:24:03.899462+0000","last_active":"2024-03-06T07:24:17.256068+0000","last_peered":"2024-03-06T07:24:17.256068+0000","last_clean":"2024-03-06T07:24:17.256068+0000","last_became_active":"2024-03-06T07:24:03.899154+0000","last_became_peered":"2024-03-06T07:24:03.899154+0000","last_unstale":"2024-03-06T07:24:17.256068+0000","last_undegraded":"2024-03-06T07:24:17.256068+0000","last_fullsized":"2024-03-06T07:24:17.256068+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:05.985567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252064+0000","last_change":"2024-03-06T07:24:02.886145+0000","last_active":"2024-03-06T07:24:17.252064+0000","last_peered":"2024-03-06T07:24:17.252064+0000","last_clean":"2024-03-06T07:24:17.252064+0000","last_became_active":"2024-03-06T07:24:02.885886+0000","last_became_peered":"2024-03-06T07:24:02.885886+0000","last_unstale":"2024-03-06T07:24:17.252064+0000","last_undegraded":"2024-03-06T07:24:17.252064+0000","last_fullsized":"2024-03-06T07:24:17.252064+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:43.257419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270982+0000","last_change":"2024-03-06T07:24:17.265778+0000","last_active":"2024-03-06T07:24:17.270982+0000","last_peered":"2024-03-06T07:24:17.270982+0000","last_clean":"2024-03-06T07:24:17.270982+0000","last_became_active":"2024-03-06T07:24:17.265517+0000","last_became_peered":"2024-03-06T07:24:17.265517+0000","last_unstale":"2024-03-06T07:24:17.270982+0000","last_undegraded":"2024-03-06T07:24:17.270982+0000","last_fullsized":"2024-03-06T07:24:17.270982+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.777544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"152'4","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275215+0000","last_change":"2024-03-06T07:24:16.242118+0000","last_active":"2024-03-06T07:24:17.275215+0000","last_peered":"2024-03-06T07:24:17.275215+0000","last_clean":"2024-03-06T07:24:17.275215+0000","last_became_active":"2024-03-06T07:24:16.241847+0000","last_became_peered":"2024-03-06T07:24:16.241847+0000","last_unstale":"2024-03-06T07:24:17.275215+0000","last_undegraded":"2024-03-06T07:24:17.275215+0000","last_fullsized":"2024-03-06T07:24:17.275215+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:57.892414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263494+0000","last_change":"2024-03-06T07:24:03.883971+0000","last_active":"2024-03-06T07:24:17.263494+0000","last_peered":"2024-03-06T07:24:17.263494+0000","last_clean":"2024-03-06T07:24:17.263494+0000","last_became_active":"2024-03-06T07:24:03.883705+0000","last_became_peered":"2024-03-06T07:24:03.883705+0000","last_unstale":"2024-03-06T07:24:17.263494+0000","last_undegraded":"2024-03-06T07:24:17.263494+0000","last_fullsized":"2024-03-06T07:24:17.263494+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:34.136559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258878+0000","last_change":"2024-03-06T07:24:02.883880+0000","last_active":"2024-03-06T07:24:17.258878+0000","last_peered":"2024-03-06T07:24:17.258878+0000","last_clean":"2024-03-06T07:24:17.258878+0000","last_became_active":"2024-03-06T07:24:02.883488+0000","last_became_peered":"2024-03-06T07:24:02.883488+0000","last_unstale":"2024-03-06T07:24:17.258878+0000","last_undegraded":"2024-03-06T07:24:17.258878+0000","last_fullsized":"2024-03-06T07:24:17.258878+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:31.545966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271679+0000","last_change":"2024-03-06T07:24:17.270301+0000","last_active":"2024-03-06T07:24:17.271679+0000","last_peered":"2024-03-06T07:24:17.271679+0000","last_clean":"2024-03-06T07:24:17.271679+0000","last_became_active":"2024-03-06T07:24:17.270059+0000","last_became_peered":"2024-03-06T07:24:17.270059+0000","last_unstale":"2024-03-06T07:24:17.271679+0000","last_undegraded":"2024-03-06T07:24:17.271679+0000","last_fullsized":"2024-03-06T07:24:17.271679+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:27.251415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276359+0000","last_change":"2024-03-06T07:24:16.256150+0000","last_active":"2024-03-06T07:24:17.276359+0000","last_peered":"2024-03-06T07:24:17.276359+0000","last_clean":"2024-03-06T07:24:17.276359+0000","last_became_active":"2024-03-06T07:24:16.255892+0000","last_became_peered":"2024-03-06T07:24:16.255892+0000","last_unstale":"2024-03-06T07:24:17.276359+0000","last_undegraded":"2024-03-06T07:24:17.276359+0000","last_fullsized":"2024-03-06T07:24:17.276359+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:25.087329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252485+0000","last_change":"2024-03-06T07:24:03.883761+0000","last_active":"2024-03-06T07:24:17.252485+0000","last_peered":"2024-03-06T07:24:17.252485+0000","last_clean":"2024-03-06T07:24:17.252485+0000","last_became_active":"2024-03-06T07:24:03.882991+0000","last_became_peered":"2024-03-06T07:24:03.882991+0000","last_unstale":"2024-03-06T07:24:17.252485+0000","last_undegraded":"2024-03-06T07:24:17.252485+0000","last_fullsized":"2024-03-06T07:24:17.252485+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:28.659458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":28,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256119+0000","last_change":"2024-03-06T07:24:02.885178+0000","last_active":"2024-03-06T07:24:17.256119+0000","last_peered":"2024-03-06T07:24:17.256119+0000","last_clean":"2024-03-06T07:24:17.256119+0000","last_became_active":"2024-03-06T07:24:02.880268+0000","last_became_peered":"2024-03-06T07:24:02.880268+0000","last_unstale":"2024-03-06T07:24:17.256119+0000","last_undegraded":"2024-03-06T07:24:17.256119+0000","last_fullsized":"2024-03-06T07:24:17.256119+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:14.833238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271854+0000","last_change":"2024-03-06T07:24:17.265332+0000","last_active":"2024-03-06T07:24:17.271854+0000","last_peered":"2024-03-06T07:24:17.271854+0000","last_clean":"2024-03-06T07:24:17.271854+0000","last_became_active":"2024-03-06T07:24:17.264768+0000","last_became_peered":"2024-03-06T07:24:17.264768+0000","last_unstale":"2024-03-06T07:24:17.271854+0000","last_undegraded":"2024-03-06T07:24:17.271854+0000","last_fullsized":"2024-03-06T07:24:17.271854+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.629152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250348+0000","last_change":"2024-03-06T07:24:16.249194+0000","last_active":"2024-03-06T07:24:17.250348+0000","last_peered":"2024-03-06T07:24:17.250348+0000","last_clean":"2024-03-06T07:24:17.250348+0000","last_became_active":"2024-03-06T07:24:16.248943+0000","last_became_peered":"2024-03-06T07:24:16.248943+0000","last_unstale":"2024-03-06T07:24:17.250348+0000","last_undegraded":"2024-03-06T07:24:17.250348+0000","last_fullsized":"2024-03-06T07:24:17.250348+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:51.728831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255948+0000","last_change":"2024-03-06T07:24:03.888753+0000","last_active":"2024-03-06T07:24:17.255948+0000","last_peered":"2024-03-06T07:24:17.255948+0000","last_clean":"2024-03-06T07:24:17.255948+0000","last_became_active":"2024-03-06T07:24:03.888491+0000","last_became_peered":"2024-03-06T07:24:03.888491+0000","last_unstale":"2024-03-06T07:24:17.255948+0000","last_undegraded":"2024-03-06T07:24:17.255948+0000","last_fullsized":"2024-03-06T07:24:17.255948+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251982+0000","last_change":"2024-03-06T07:24:02.886460+0000","last_active":"2024-03-06T07:24:17.251982+0000","last_peered":"2024-03-06T07:24:17.251982+0000","last_clean":"2024-03-06T07:24:17.251982+0000","last_became_active":"2024-03-06T07:24:02.885733+0000","last_became_peered":"2024-03-06T07:24:02.885733+0000","last_unstale":"2024-03-06T07:24:17.251982+0000","last_undegraded":"2024-03-06T07:24:17.251982+0000","last_fullsized":"2024-03-06T07:24:17.251982+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:59.543991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278436+0000","last_change":"2024-03-06T07:24:17.275819+0000","last_active":"2024-03-06T07:24:17.278436+0000","last_peered":"2024-03-06T07:24:17.278436+0000","last_clean":"2024-03-06T07:24:17.278436+0000","last_became_active":"2024-03-06T07:24:17.275562+0000","last_became_peered":"2024-03-06T07:24:17.275562+0000","last_unstale":"2024-03-06T07:24:17.278436+0000","last_undegraded":"2024-03-06T07:24:17.278436+0000","last_fullsized":"2024-03-06T07:24:17.278436+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:11.169809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251745+0000","last_change":"2024-03-06T07:24:16.245437+0000","last_active":"2024-03-06T07:24:17.251745+0000","last_peered":"2024-03-06T07:24:17.251745+0000","last_clean":"2024-03-06T07:24:17.251745+0000","last_became_active":"2024-03-06T07:24:16.245151+0000","last_became_peered":"2024-03-06T07:24:16.245151+0000","last_unstale":"2024-03-06T07:24:17.251745+0000","last_undegraded":"2024-03-06T07:24:17.251745+0000","last_fullsized":"2024-03-06T07:24:17.251745+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:56.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252006+0000","last_change":"2024-03-06T07:24:03.889875+0000","last_active":"2024-03-06T07:24:17.252006+0000","last_peered":"2024-03-06T07:24:17.252006+0000","last_clean":"2024-03-06T07:24:17.252006+0000","last_became_active":"2024-03-06T07:24:03.889040+0000","last_became_peered":"2024-03-06T07:24:03.889040+0000","last_unstale":"2024-03-06T07:24:17.252006+0000","last_undegraded":"2024-03-06T07:24:17.252006+0000","last_fullsized":"2024-03-06T07:24:17.252006+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:57.658966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257692+0000","last_change":"2024-03-06T07:24:02.888816+0000","last_active":"2024-03-06T07:24:17.257692+0000","last_peered":"2024-03-06T07:24:17.257692+0000","last_clean":"2024-03-06T07:24:17.257692+0000","last_became_active":"2024-03-06T07:24:02.888090+0000","last_became_peered":"2024-03-06T07:24:02.888090+0000","last_unstale":"2024-03-06T07:24:17.257692+0000","last_undegraded":"2024-03-06T07:24:17.257692+0000","last_fullsized":"2024-03-06T07:24:17.257692+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:42.738046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271418+0000","last_change":"2024-03-06T07:24:17.265980+0000","last_active":"2024-03-06T07:24:17.271418+0000","last_peered":"2024-03-06T07:24:17.271418+0000","last_clean":"2024-03-06T07:24:17.271418+0000","last_became_active":"2024-03-06T07:24:17.260869+0000","last_became_peered":"2024-03-06T07:24:17.260869+0000","last_unstale":"2024-03-06T07:24:17.271418+0000","last_undegraded":"2024-03-06T07:24:17.271418+0000","last_fullsized":"2024-03-06T07:24:17.271418+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:09.278978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262672+0000","last_change":"2024-03-06T07:24:16.252214+0000","last_active":"2024-03-06T07:24:17.262672+0000","last_peered":"2024-03-06T07:24:17.262672+0000","last_clean":"2024-03-06T07:24:17.262672+0000","last_became_active":"2024-03-06T07:24:16.251529+0000","last_became_peered":"2024-03-06T07:24:16.251529+0000","last_unstale":"2024-03-06T07:24:17.262672+0000","last_undegraded":"2024-03-06T07:24:17.262672+0000","last_fullsized":"2024-03-06T07:24:17.262672+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:19.586046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257436+0000","last_change":"2024-03-06T07:24:03.890038+0000","last_active":"2024-03-06T07:24:17.257436+0000","last_peered":"2024-03-06T07:24:17.257436+0000","last_clean":"2024-03-06T07:24:17.257436+0000","last_became_active":"2024-03-06T07:24:03.889784+0000","last_became_peered":"2024-03-06T07:24:03.889784+0000","last_unstale":"2024-03-06T07:24:17.257436+0000","last_undegraded":"2024-03-06T07:24:17.257436+0000","last_fullsized":"2024-03-06T07:24:17.257436+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:35.461921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247718+0000","last_change":"2024-03-06T07:24:02.887549+0000","last_active":"2024-03-06T07:24:17.247718+0000","last_peered":"2024-03-06T07:24:17.247718+0000","last_clean":"2024-03-06T07:24:17.247718+0000","last_became_active":"2024-03-06T07:24:02.887264+0000","last_became_peered":"2024-03-06T07:24:02.887264+0000","last_unstale":"2024-03-06T07:24:17.247718+0000","last_undegraded":"2024-03-06T07:24:17.247718+0000","last_fullsized":"2024-03-06T07:24:17.247718+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:15.400205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.267075+0000","last_change":"2024-03-06T07:24:17.264034+0000","last_active":"2024-03-06T07:24:17.267075+0000","last_peered":"2024-03-06T07:24:17.267075+0000","last_clean":"2024-03-06T07:24:17.267075+0000","last_became_active":"2024-03-06T07:24:17.263770+0000","last_became_peered":"2024-03-06T07:24:17.263770+0000","last_unstale":"2024-03-06T07:24:17.267075+0000","last_undegraded":"2024-03-06T07:24:17.267075+0000","last_fullsized":"2024-03-06T07:24:17.267075+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:01.513189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257440+0000","last_change":"2024-03-06T07:24:16.251808+0000","last_active":"2024-03-06T07:24:17.257440+0000","last_peered":"2024-03-06T07:24:17.257440+0000","last_clean":"2024-03-06T07:24:17.257440+0000","last_became_active":"2024-03-06T07:24:16.251527+0000","last_became_peered":"2024-03-06T07:24:16.251527+0000","last_unstale":"2024-03-06T07:24:17.257440+0000","last_undegraded":"2024-03-06T07:24:17.257440+0000","last_fullsized":"2024-03-06T07:24:17.257440+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:24.856326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261087+0000","last_change":"2024-03-06T07:24:03.897943+0000","last_active":"2024-03-06T07:24:17.261087+0000","last_peered":"2024-03-06T07:24:17.261087+0000","last_clean":"2024-03-06T07:24:17.261087+0000","last_became_active":"2024-03-06T07:24:03.897665+0000","last_became_peered":"2024-03-06T07:24:03.897665+0000","last_unstale":"2024-03-06T07:24:17.261087+0000","last_undegraded":"2024-03-06T07:24:17.261087+0000","last_fullsized":"2024-03-06T07:24:17.261087+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:36.087346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256551+0000","last_change":"2024-03-06T07:24:02.881968+0000","last_active":"2024-03-06T07:24:17.256551+0000","last_peered":"2024-03-06T07:24:17.256551+0000","last_clean":"2024-03-06T07:24:17.256551+0000","last_became_active":"2024-03-06T07:24:02.881702+0000","last_became_peered":"2024-03-06T07:24:02.881702+0000","last_unstale":"2024-03-06T07:24:17.256551+0000","last_undegraded":"2024-03-06T07:24:17.256551+0000","last_fullsized":"2024-03-06T07:24:17.256551+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:37.173846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275615+0000","last_change":"2024-03-06T07:24:17.268289+0000","last_active":"2024-03-06T07:24:17.275615+0000","last_peered":"2024-03-06T07:24:17.275615+0000","last_clean":"2024-03-06T07:24:17.275615+0000","last_became_active":"2024-03-06T07:24:17.266770+0000","last_became_peered":"2024-03-06T07:24:17.266770+0000","last_unstale":"2024-03-06T07:24:17.275615+0000","last_undegraded":"2024-03-06T07:24:17.275615+0000","last_fullsized":"2024-03-06T07:24:17.275615+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:06.739402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250922+0000","last_change":"2024-03-06T07:24:16.256703+0000","last_active":"2024-03-06T07:24:17.250922+0000","last_peered":"2024-03-06T07:24:17.250922+0000","last_clean":"2024-03-06T07:24:17.250922+0000","last_became_active":"2024-03-06T07:24:16.256327+0000","last_became_peered":"2024-03-06T07:24:16.256327+0000","last_unstale":"2024-03-06T07:24:17.250922+0000","last_undegraded":"2024-03-06T07:24:17.250922+0000","last_fullsized":"2024-03-06T07:24:17.250922+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:34.579168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250844+0000","last_change":"2024-03-06T07:24:03.894250+0000","last_active":"2024-03-06T07:24:17.250844+0000","last_peered":"2024-03-06T07:24:17.250844+0000","last_clean":"2024-03-06T07:24:17.250844+0000","last_became_active":"2024-03-06T07:24:03.893672+0000","last_became_peered":"2024-03-06T07:24:03.893672+0000","last_unstale":"2024-03-06T07:24:17.250844+0000","last_undegraded":"2024-03-06T07:24:17.250844+0000","last_fullsized":"2024-03-06T07:24:17.250844+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:55.684312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247702+0000","last_change":"2024-03-06T07:24:02.883228+0000","last_active":"2024-03-06T07:24:17.247702+0000","last_peered":"2024-03-06T07:24:17.247702+0000","last_clean":"2024-03-06T07:24:17.247702+0000","last_became_active":"2024-03-06T07:24:02.882971+0000","last_became_peered":"2024-03-06T07:24:02.882971+0000","last_unstale":"2024-03-06T07:24:17.247702+0000","last_undegraded":"2024-03-06T07:24:17.247702+0000","last_fullsized":"2024-03-06T07:24:17.247702+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:42.738046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270653+0000","last_change":"2024-03-06T07:24:17.268650+0000","last_active":"2024-03-06T07:24:17.270653+0000","last_peered":"2024-03-06T07:24:17.270653+0000","last_clean":"2024-03-06T07:24:17.270653+0000","last_became_active":"2024-03-06T07:24:17.268386+0000","last_became_peered":"2024-03-06T07:24:17.268386+0000","last_unstale":"2024-03-06T07:24:17.270653+0000","last_undegraded":"2024-03-06T07:24:17.270653+0000","last_fullsized":"2024-03-06T07:24:17.270653+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:25.693947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250861+0000","last_change":"2024-03-06T07:24:16.252130+0000","last_active":"2024-03-06T07:24:17.250861+0000","last_peered":"2024-03-06T07:24:17.250861+0000","last_clean":"2024-03-06T07:24:17.250861+0000","last_became_active":"2024-03-06T07:24:16.251812+0000","last_became_peered":"2024-03-06T07:24:16.251812+0000","last_unstale":"2024-03-06T07:24:17.250861+0000","last_undegraded":"2024-03-06T07:24:17.250861+0000","last_fullsized":"2024-03-06T07:24:17.250861+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:08.639042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247625+0000","last_change":"2024-03-06T07:24:03.889481+0000","last_active":"2024-03-06T07:24:17.247625+0000","last_peered":"2024-03-06T07:24:17.247625+0000","last_clean":"2024-03-06T07:24:17.247625+0000","last_became_active":"2024-03-06T07:24:03.889258+0000","last_became_peered":"2024-03-06T07:24:03.889258+0000","last_unstale":"2024-03-06T07:24:17.247625+0000","last_undegraded":"2024-03-06T07:24:17.247625+0000","last_fullsized":"2024-03-06T07:24:17.247625+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257343+0000","last_change":"2024-03-06T07:24:02.889856+0000","last_active":"2024-03-06T07:24:17.257343+0000","last_peered":"2024-03-06T07:24:17.257343+0000","last_clean":"2024-03-06T07:24:17.257343+0000","last_became_active":"2024-03-06T07:24:02.889589+0000","last_became_peered":"2024-03-06T07:24:02.889589+0000","last_unstale":"2024-03-06T07:24:17.257343+0000","last_undegraded":"2024-03-06T07:24:17.257343+0000","last_fullsized":"2024-03-06T07:24:17.257343+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:09.042646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262891+0000","last_change":"2024-03-06T07:24:17.262167+0000","last_active":"2024-03-06T07:24:17.262891+0000","last_peered":"2024-03-06T07:24:17.262891+0000","last_clean":"2024-03-06T07:24:17.262891+0000","last_became_active":"2024-03-06T07:24:17.261902+0000","last_became_peered":"2024-03-06T07:24:17.261902+0000","last_unstale":"2024-03-06T07:24:17.262891+0000","last_undegraded":"2024-03-06T07:24:17.262891+0000","last_fullsized":"2024-03-06T07:24:17.262891+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:11.081631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276725+0000","last_change":"2024-03-06T07:24:16.250683+0000","last_active":"2024-03-06T07:24:17.276725+0000","last_peered":"2024-03-06T07:24:17.276725+0000","last_clean":"2024-03-06T07:24:17.276725+0000","last_became_active":"2024-03-06T07:24:16.250440+0000","last_became_peered":"2024-03-06T07:24:16.250440+0000","last_unstale":"2024-03-06T07:24:17.276725+0000","last_undegraded":"2024-03-06T07:24:17.276725+0000","last_fullsized":"2024-03-06T07:24:17.276725+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:42.156089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247958+0000","last_change":"2024-03-06T07:24:03.890643+0000","last_active":"2024-03-06T07:24:17.247958+0000","last_peered":"2024-03-06T07:24:17.247958+0000","last_clean":"2024-03-06T07:24:17.247958+0000","last_became_active":"2024-03-06T07:24:03.890406+0000","last_became_peered":"2024-03-06T07:24:03.890406+0000","last_unstale":"2024-03-06T07:24:17.247958+0000","last_undegraded":"2024-03-06T07:24:17.247958+0000","last_fullsized":"2024-03-06T07:24:17.247958+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250657+0000","last_change":"2024-03-06T07:24:02.882070+0000","last_active":"2024-03-06T07:24:17.250657+0000","last_peered":"2024-03-06T07:24:17.250657+0000","last_clean":"2024-03-06T07:24:17.250657+0000","last_became_active":"2024-03-06T07:24:02.881582+0000","last_became_peered":"2024-03-06T07:24:02.881582+0000","last_unstale":"2024-03-06T07:24:17.250657+0000","last_undegraded":"2024-03-06T07:24:17.250657+0000","last_fullsized":"2024-03-06T07:24:17.250657+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:37.816763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263777+0000","last_change":"2024-03-06T07:24:17.258001+0000","last_active":"2024-03-06T07:24:17.263777+0000","last_peered":"2024-03-06T07:24:17.263777+0000","last_clean":"2024-03-06T07:24:17.263777+0000","last_became_active":"2024-03-06T07:24:17.257764+0000","last_became_peered":"2024-03-06T07:24:17.257764+0000","last_unstale":"2024-03-06T07:24:17.263777+0000","last_undegraded":"2024-03-06T07:24:17.263777+0000","last_fullsized":"2024-03-06T07:24:17.263777+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:25.860905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.249807+0000","last_change":"2024-03-06T07:24:16.251351+0000","last_active":"2024-03-06T07:24:17.249807+0000","last_peered":"2024-03-06T07:24:17.249807+0000","last_clean":"2024-03-06T07:24:17.249807+0000","last_became_active":"2024-03-06T07:24:16.251106+0000","last_became_peered":"2024-03-06T07:24:16.251106+0000","last_unstale":"2024-03-06T07:24:17.249807+0000","last_undegraded":"2024-03-06T07:24:17.249807+0000","last_fullsized":"2024-03-06T07:24:17.249807+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:07.372422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263446+0000","last_change":"2024-03-06T07:24:03.890706+0000","last_active":"2024-03-06T07:24:17.263446+0000","last_peered":"2024-03-06T07:24:17.263446+0000","last_clean":"2024-03-06T07:24:17.263446+0000","last_became_active":"2024-03-06T07:24:03.890369+0000","last_became_peered":"2024-03-06T07:24:03.890369+0000","last_unstale":"2024-03-06T07:24:17.263446+0000","last_undegraded":"2024-03-06T07:24:17.263446+0000","last_fullsized":"2024-03-06T07:24:17.263446+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:35.615657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258420+0000","last_change":"2024-03-06T07:24:02.885639+0000","last_active":"2024-03-06T07:24:17.258420+0000","last_peered":"2024-03-06T07:24:17.258420+0000","last_clean":"2024-03-06T07:24:17.258420+0000","last_became_active":"2024-03-06T07:24:02.885357+0000","last_became_peered":"2024-03-06T07:24:02.885357+0000","last_unstale":"2024-03-06T07:24:17.258420+0000","last_undegraded":"2024-03-06T07:24:17.258420+0000","last_fullsized":"2024-03-06T07:24:17.258420+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.265219+0000","last_change":"2024-03-06T07:24:17.264192+0000","last_active":"2024-03-06T07:24:17.265219+0000","last_peered":"2024-03-06T07:24:17.265219+0000","last_clean":"2024-03-06T07:24:17.265219+0000","last_became_active":"2024-03-06T07:24:17.263885+0000","last_became_peered":"2024-03-06T07:24:17.263885+0000","last_unstale":"2024-03-06T07:24:17.265219+0000","last_undegraded":"2024-03-06T07:24:17.265219+0000","last_fullsized":"2024-03-06T07:24:17.265219+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:32.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250218+0000","last_change":"2024-03-06T07:24:16.249778+0000","last_active":"2024-03-06T07:24:17.250218+0000","last_peered":"2024-03-06T07:24:17.250218+0000","last_clean":"2024-03-06T07:24:17.250218+0000","last_became_active":"2024-03-06T07:24:16.249542+0000","last_became_peered":"2024-03-06T07:24:16.249542+0000","last_unstale":"2024-03-06T07:24:17.250218+0000","last_undegraded":"2024-03-06T07:24:17.250218+0000","last_fullsized":"2024-03-06T07:24:17.250218+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:10.769430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.260234+0000","last_change":"2024-03-06T07:24:03.888914+0000","last_active":"2024-03-06T07:24:17.260234+0000","last_peered":"2024-03-06T07:24:17.260234+0000","last_clean":"2024-03-06T07:24:17.260234+0000","last_became_active":"2024-03-06T07:24:03.888605+0000","last_became_peered":"2024-03-06T07:24:03.888605+0000","last_unstale":"2024-03-06T07:24:17.260234+0000","last_undegraded":"2024-03-06T07:24:17.260234+0000","last_fullsized":"2024-03-06T07:24:17.260234+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:07.984372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.248261+0000","last_change":"2024-03-06T07:24:02.884638+0000","last_active":"2024-03-06T07:24:17.248261+0000","last_peered":"2024-03-06T07:24:17.248261+0000","last_clean":"2024-03-06T07:24:17.248261+0000","last_became_active":"2024-03-06T07:24:02.884370+0000","last_became_peered":"2024-03-06T07:24:02.884370+0000","last_unstale":"2024-03-06T07:24:17.248261+0000","last_undegraded":"2024-03-06T07:24:17.248261+0000","last_fullsized":"2024-03-06T07:24:17.248261+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:38.280774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262215+0000","last_change":"2024-03-06T07:24:17.256748+0000","last_active":"2024-03-06T07:24:17.262215+0000","last_peered":"2024-03-06T07:24:17.262215+0000","last_clean":"2024-03-06T07:24:17.262215+0000","last_became_active":"2024-03-06T07:24:17.256505+0000","last_became_peered":"2024-03-06T07:24:17.256505+0000","last_unstale":"2024-03-06T07:24:17.262215+0000","last_undegraded":"2024-03-06T07:24:17.262215+0000","last_fullsized":"2024-03-06T07:24:17.262215+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:51.041492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252967+0000","last_change":"2024-03-06T07:24:16.248612+0000","last_active":"2024-03-06T07:24:17.252967+0000","last_peered":"2024-03-06T07:24:17.252967+0000","last_clean":"2024-03-06T07:24:17.252967+0000","last_became_active":"2024-03-06T07:24:16.247445+0000","last_became_peered":"2024-03-06T07:24:16.247445+0000","last_unstale":"2024-03-06T07:24:17.252967+0000","last_undegraded":"2024-03-06T07:24:17.252967+0000","last_fullsized":"2024-03-06T07:24:17.252967+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:44.856793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262149+0000","last_change":"2024-03-06T07:24:03.889791+0000","last_active":"2024-03-06T07:24:17.262149+0000","last_peered":"2024-03-06T07:24:17.262149+0000","last_clean":"2024-03-06T07:24:17.262149+0000","last_became_active":"2024-03-06T07:24:03.889121+0000","last_became_peered":"2024-03-06T07:24:03.889121+0000","last_unstale":"2024-03-06T07:24:17.262149+0000","last_undegraded":"2024-03-06T07:24:17.262149+0000","last_fullsized":"2024-03-06T07:24:17.262149+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.254200+0000","last_change":"2024-03-06T07:24:02.891610+0000","last_active":"2024-03-06T07:24:17.254200+0000","last_peered":"2024-03-06T07:24:17.254200+0000","last_clean":"2024-03-06T07:24:17.254200+0000","last_became_active":"2024-03-06T07:24:02.891294+0000","last_became_peered":"2024-03-06T07:24:02.891294+0000","last_unstale":"2024-03-06T07:24:17.254200+0000","last_undegraded":"2024-03-06T07:24:17.254200+0000","last_fullsized":"2024-03-06T07:24:17.254200+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:39.405811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271786+0000","last_change":"2024-03-06T07:24:17.264101+0000","last_active":"2024-03-06T07:24:17.271786+0000","last_peered":"2024-03-06T07:24:17.271786+0000","last_clean":"2024-03-06T07:24:17.271786+0000","last_became_active":"2024-03-06T07:24:17.263836+0000","last_became_peered":"2024-03-06T07:24:17.263836+0000","last_unstale":"2024-03-06T07:24:17.271786+0000","last_undegraded":"2024-03-06T07:24:17.271786+0000","last_fullsized":"2024-03-06T07:24:17.271786+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:20.590625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278736+0000","last_change":"2024-03-06T07:24:16.252025+0000","last_active":"2024-03-06T07:24:17.278736+0000","last_peered":"2024-03-06T07:24:17.278736+0000","last_clean":"2024-03-06T07:24:17.278736+0000","last_became_active":"2024-03-06T07:24:16.251755+0000","last_became_peered":"2024-03-06T07:24:16.251755+0000","last_unstale":"2024-03-06T07:24:17.278736+0000","last_undegraded":"2024-03-06T07:24:17.278736+0000","last_fullsized":"2024-03-06T07:24:17.278736+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.265210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253127+0000","last_change":"2024-03-06T07:24:03.881428+0000","last_active":"2024-03-06T07:24:17.253127+0000","last_peered":"2024-03-06T07:24:17.253127+0000","last_clean":"2024-03-06T07:24:17.253127+0000","last_became_active":"2024-03-06T07:24:03.881168+0000","last_became_peered":"2024-03-06T07:24:03.881168+0000","last_unstale":"2024-03-06T07:24:17.253127+0000","last_undegraded":"2024-03-06T07:24:17.253127+0000","last_fullsized":"2024-03-06T07:24:17.253127+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:00.752911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251111+0000","last_change":"2024-03-06T07:24:02.887206+0000","last_active":"2024-03-06T07:24:17.251111+0000","last_peered":"2024-03-06T07:24:17.251111+0000","last_clean":"2024-03-06T07:24:17.251111+0000","last_became_active":"2024-03-06T07:24:02.886946+0000","last_became_peered":"2024-03-06T07:24:02.886946+0000","last_unstale":"2024-03-06T07:24:17.251111+0000","last_undegraded":"2024-03-06T07:24:17.251111+0000","last_fullsized":"2024-03-06T07:24:17.251111+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:11.489172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.264345+0000","last_change":"2024-03-06T07:24:17.263538+0000","last_active":"2024-03-06T07:24:17.264345+0000","last_peered":"2024-03-06T07:24:17.264345+0000","last_clean":"2024-03-06T07:24:17.264345+0000","last_became_active":"2024-03-06T07:24:17.263265+0000","last_became_peered":"2024-03-06T07:24:17.263265+0000","last_unstale":"2024-03-06T07:24:17.264345+0000","last_undegraded":"2024-03-06T07:24:17.264345+0000","last_fullsized":"2024-03-06T07:24:17.264345+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:11.169809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247687+0000","last_change":"2024-03-06T07:24:16.248840+0000","last_active":"2024-03-06T07:24:17.247687+0000","last_peered":"2024-03-06T07:24:17.247687+0000","last_clean":"2024-03-06T07:24:17.247687+0000","last_became_active":"2024-03-06T07:24:16.248116+0000","last_became_peered":"2024-03-06T07:24:16.248116+0000","last_unstale":"2024-03-06T07:24:17.247687+0000","last_undegraded":"2024-03-06T07:24:17.247687+0000","last_fullsized":"2024-03-06T07:24:17.247687+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:05.734823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255719+0000","last_change":"2024-03-06T07:24:03.887606+0000","last_active":"2024-03-06T07:24:17.255719+0000","last_peered":"2024-03-06T07:24:17.255719+0000","last_clean":"2024-03-06T07:24:17.255719+0000","last_became_active":"2024-03-06T07:24:03.887318+0000","last_became_peered":"2024-03-06T07:24:03.887318+0000","last_unstale":"2024-03-06T07:24:17.255719+0000","last_undegraded":"2024-03-06T07:24:17.255719+0000","last_fullsized":"2024-03-06T07:24:17.255719+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250778+0000","last_change":"2024-03-06T07:24:02.891159+0000","last_active":"2024-03-06T07:24:17.250778+0000","last_peered":"2024-03-06T07:24:17.250778+0000","last_clean":"2024-03-06T07:24:17.250778+0000","last_became_active":"2024-03-06T07:24:02.890886+0000","last_became_peered":"2024-03-06T07:24:02.890886+0000","last_unstale":"2024-03-06T07:24:17.250778+0000","last_undegraded":"2024-03-06T07:24:17.250778+0000","last_fullsized":"2024-03-06T07:24:17.250778+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:57.098650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.280932+0000","last_change":"2024-03-06T07:24:17.277737+0000","last_active":"2024-03-06T07:24:17.280932+0000","last_peered":"2024-03-06T07:24:17.280932+0000","last_clean":"2024-03-06T07:24:17.280932+0000","last_became_active":"2024-03-06T07:24:17.277420+0000","last_became_peered":"2024-03-06T07:24:17.277420+0000","last_unstale":"2024-03-06T07:24:17.280932+0000","last_undegraded":"2024-03-06T07:24:17.280932+0000","last_fullsized":"2024-03-06T07:24:17.280932+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:13.915724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277527+0000","last_change":"2024-03-06T07:24:16.252568+0000","last_active":"2024-03-06T07:24:17.277527+0000","last_peered":"2024-03-06T07:24:17.277527+0000","last_clean":"2024-03-06T07:24:17.277527+0000","last_became_active":"2024-03-06T07:24:16.252168+0000","last_became_peered":"2024-03-06T07:24:16.252168+0000","last_unstale":"2024-03-06T07:24:17.277527+0000","last_undegraded":"2024-03-06T07:24:17.277527+0000","last_fullsized":"2024-03-06T07:24:17.277527+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:55.560488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256170+0000","last_change":"2024-03-06T07:24:03.886641+0000","last_active":"2024-03-06T07:24:17.256170+0000","last_peered":"2024-03-06T07:24:17.256170+0000","last_clean":"2024-03-06T07:24:17.256170+0000","last_became_active":"2024-03-06T07:24:03.885674+0000","last_became_peered":"2024-03-06T07:24:03.885674+0000","last_unstale":"2024-03-06T07:24:17.256170+0000","last_undegraded":"2024-03-06T07:24:17.256170+0000","last_fullsized":"2024-03-06T07:24:17.256170+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:32.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253627+0000","last_change":"2024-03-06T07:24:02.888862+0000","last_active":"2024-03-06T07:24:17.253627+0000","last_peered":"2024-03-06T07:24:17.253627+0000","last_clean":"2024-03-06T07:24:17.253627+0000","last_became_active":"2024-03-06T07:24:02.888609+0000","last_became_peered":"2024-03-06T07:24:02.888609+0000","last_unstale":"2024-03-06T07:24:17.253627+0000","last_undegraded":"2024-03-06T07:24:17.253627+0000","last_fullsized":"2024-03-06T07:24:17.253627+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:38.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271780+0000","last_change":"2024-03-06T07:24:17.266303+0000","last_active":"2024-03-06T07:24:17.271780+0000","last_peered":"2024-03-06T07:24:17.271780+0000","last_clean":"2024-03-06T07:24:17.271780+0000","last_became_active":"2024-03-06T07:24:17.265823+0000","last_became_peered":"2024-03-06T07:24:17.265823+0000","last_unstale":"2024-03-06T07:24:17.271780+0000","last_undegraded":"2024-03-06T07:24:17.271780+0000","last_fullsized":"2024-03-06T07:24:17.271780+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:43.160668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258562+0000","last_change":"2024-03-06T07:24:16.248938+0000","last_active":"2024-03-06T07:24:17.258562+0000","last_peered":"2024-03-06T07:24:17.258562+0000","last_clean":"2024-03-06T07:24:17.258562+0000","last_became_active":"2024-03-06T07:24:16.248668+0000","last_became_peered":"2024-03-06T07:24:16.248668+0000","last_unstale":"2024-03-06T07:24:17.258562+0000","last_undegraded":"2024-03-06T07:24:17.258562+0000","last_fullsized":"2024-03-06T07:24:17.258562+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:44.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263654+0000","last_change":"2024-03-06T07:24:03.893828+0000","last_active":"2024-03-06T07:24:17.263654+0000","last_peered":"2024-03-06T07:24:17.263654+0000","last_clean":"2024-03-06T07:24:17.263654+0000","last_became_active":"2024-03-06T07:24:03.893578+0000","last_became_peered":"2024-03-06T07:24:03.893578+0000","last_unstale":"2024-03-06T07:24:17.263654+0000","last_undegraded":"2024-03-06T07:24:17.263654+0000","last_fullsized":"2024-03-06T07:24:17.263654+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:05.985567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251942+0000","last_change":"2024-03-06T07:24:02.884916+0000","last_active":"2024-03-06T07:24:17.251942+0000","last_peered":"2024-03-06T07:24:17.251942+0000","last_clean":"2024-03-06T07:24:17.251942+0000","last_became_active":"2024-03-06T07:24:02.883806+0000","last_became_peered":"2024-03-06T07:24:02.883806+0000","last_unstale":"2024-03-06T07:24:17.251942+0000","last_undegraded":"2024-03-06T07:24:17.251942+0000","last_fullsized":"2024-03-06T07:24:17.251942+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:52.367669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.260115+0000","last_change":"2024-03-06T07:24:17.257264+0000","last_active":"2024-03-06T07:24:17.260115+0000","last_peered":"2024-03-06T07:24:17.260115+0000","last_clean":"2024-03-06T07:24:17.260115+0000","last_became_active":"2024-03-06T07:24:17.257035+0000","last_became_peered":"2024-03-06T07:24:17.257035+0000","last_unstale":"2024-03-06T07:24:17.260115+0000","last_undegraded":"2024-03-06T07:24:17.260115+0000","last_fullsized":"2024-03-06T07:24:17.260115+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:44.334236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278469+0000","last_change":"2024-03-06T07:24:16.254369+0000","last_active":"2024-03-06T07:24:17.278469+0000","last_peered":"2024-03-06T07:24:17.278469+0000","last_clean":"2024-03-06T07:24:17.278469+0000","last_became_active":"2024-03-06T07:24:16.254038+0000","last_became_peered":"2024-03-06T07:24:16.254038+0000","last_unstale":"2024-03-06T07:24:17.278469+0000","last_undegraded":"2024-03-06T07:24:17.278469+0000","last_fullsized":"2024-03-06T07:24:17.278469+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:14.259224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253686+0000","last_change":"2024-03-06T07:24:03.895097+0000","last_active":"2024-03-06T07:24:17.253686+0000","last_peered":"2024-03-06T07:24:17.253686+0000","last_clean":"2024-03-06T07:24:17.253686+0000","last_became_active":"2024-03-06T07:24:03.894829+0000","last_became_peered":"2024-03-06T07:24:03.894829+0000","last_unstale":"2024-03-06T07:24:17.253686+0000","last_undegraded":"2024-03-06T07:24:17.253686+0000","last_fullsized":"2024-03-06T07:24:17.253686+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:52.563689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.259001+0000","last_change":"2024-03-06T07:24:02.883059+0000","last_active":"2024-03-06T07:24:17.259001+0000","last_peered":"2024-03-06T07:24:17.259001+0000","last_clean":"2024-03-06T07:24:17.259001+0000","last_became_active":"2024-03-06T07:24:02.882626+0000","last_became_peered":"2024-03-06T07:24:02.882626+0000","last_unstale":"2024-03-06T07:24:17.259001+0000","last_undegraded":"2024-03-06T07:24:17.259001+0000","last_fullsized":"2024-03-06T07:24:17.259001+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:54.907245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277207+0000","last_change":"2024-03-06T07:24:17.273278+0000","last_active":"2024-03-06T07:24:17.277207+0000","last_peered":"2024-03-06T07:24:17.277207+0000","last_clean":"2024-03-06T07:24:17.277207+0000","last_became_active":"2024-03-06T07:24:17.273036+0000","last_became_peered":"2024-03-06T07:24:17.273036+0000","last_unstale":"2024-03-06T07:24:17.277207+0000","last_undegraded":"2024-03-06T07:24:17.277207+0000","last_fullsized":"2024-03-06T07:24:17.277207+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:38.588859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.274987+0000","last_change":"2024-03-06T07:24:16.250136+0000","last_active":"2024-03-06T07:24:17.274987+0000","last_peered":"2024-03-06T07:24:17.274987+0000","last_clean":"2024-03-06T07:24:17.274987+0000","last_became_active":"2024-03-06T07:24:16.249896+0000","last_became_peered":"2024-03-06T07:24:16.249896+0000","last_unstale":"2024-03-06T07:24:17.274987+0000","last_undegraded":"2024-03-06T07:24:17.274987+0000","last_fullsized":"2024-03-06T07:24:17.274987+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:56.624573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252571+0000","last_change":"2024-03-06T07:24:03.885590+0000","last_active":"2024-03-06T07:24:17.252571+0000","last_peered":"2024-03-06T07:24:17.252571+0000","last_clean":"2024-03-06T07:24:17.252571+0000","last_became_active":"2024-03-06T07:24:03.885328+0000","last_became_peered":"2024-03-06T07:24:03.885328+0000","last_unstale":"2024-03-06T07:24:17.252571+0000","last_undegraded":"2024-03-06T07:24:17.252571+0000","last_fullsized":"2024-03-06T07:24:17.252571+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:32.966742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.246812+0000","last_change":"2024-03-06T07:24:02.885607+0000","last_active":"2024-03-06T07:24:17.246812+0000","last_peered":"2024-03-06T07:24:17.246812+0000","last_clean":"2024-03-06T07:24:17.246812+0000","last_became_active":"2024-03-06T07:24:02.885342+0000","last_became_peered":"2024-03-06T07:24:02.885342+0000","last_unstale":"2024-03-06T07:24:17.246812+0000","last_undegraded":"2024-03-06T07:24:17.246812+0000","last_fullsized":"2024-03-06T07:24:17.246812+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:37.173846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.267837+0000","last_change":"2024-03-06T07:24:17.265577+0000","last_active":"2024-03-06T07:24:17.267837+0000","last_peered":"2024-03-06T07:24:17.267837+0000","last_clean":"2024-03-06T07:24:17.267837+0000","last_became_active":"2024-03-06T07:24:17.265291+0000","last_became_peered":"2024-03-06T07:24:17.265291+0000","last_unstale":"2024-03-06T07:24:17.267837+0000","last_undegraded":"2024-03-06T07:24:17.267837+0000","last_fullsized":"2024-03-06T07:24:17.267837+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:23.414379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247090+0000","last_change":"2024-03-06T07:24:16.240376+0000","last_active":"2024-03-06T07:24:17.247090+0000","last_peered":"2024-03-06T07:24:17.247090+0000","last_clean":"2024-03-06T07:24:17.247090+0000","last_became_active":"2024-03-06T07:24:16.240124+0000","last_became_peered":"2024-03-06T07:24:16.240124+0000","last_unstale":"2024-03-06T07:24:17.247090+0000","last_undegraded":"2024-03-06T07:24:17.247090+0000","last_fullsized":"2024-03-06T07:24:17.247090+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:24.856326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257222+0000","last_change":"2024-03-06T07:24:03.891512+0000","last_active":"2024-03-06T07:24:17.257222+0000","last_peered":"2024-03-06T07:24:17.257222+0000","last_clean":"2024-03-06T07:24:17.257222+0000","last_became_active":"2024-03-06T07:24:03.891241+0000","last_became_peered":"2024-03-06T07:24:03.891241+0000","last_unstale":"2024-03-06T07:24:17.257222+0000","last_undegraded":"2024-03-06T07:24:17.257222+0000","last_fullsized":"2024-03-06T07:24:17.257222+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.254108+0000","last_change":"2024-03-06T07:24:02.891039+0000","last_active":"2024-03-06T07:24:17.254108+0000","last_peered":"2024-03-06T07:24:17.254108+0000","last_clean":"2024-03-06T07:24:17.254108+0000","last_became_active":"2024-03-06T07:24:02.890670+0000","last_became_peered":"2024-03-06T07:24:02.890670+0000","last_unstale":"2024-03-06T07:24:17.254108+0000","last_undegraded":"2024-03-06T07:24:17.254108+0000","last_fullsized":"2024-03-06T07:24:17.254108+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:06.218892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":421,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258479+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:17.258479+0000","last_peered":"2024-03-06T07:24:17.258479+0000","last_clean":"2024-03-06T07:24:17.258479+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:17.258479+0000","last_undegraded":"2024-03-06T07:24:17.258479+0000","last_fullsized":"2024-03-06T07:24:17.258479+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1509566,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":919198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1908736,"data_stored":919322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1916928,"data_stored":919962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1921024,"data_stored":921414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":919198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1912832,"data_stored":920810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":1974272,"data_stored":1509670,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:21.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:21.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.597+0000 7fad12cc9640 1 Processor -- start 2024-03-06T07:24:21.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.597+0000 7fad12cc9640 1 -- start start 2024-03-06T07:24:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad12cc9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad0c0c5330 0x7fad0c0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad12cc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c158db0 0x7fad0c15fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad12cc9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0c155110 con 0x7fad0c0c88b0 2024-03-06T07:24:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad12cc9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0c1559e0 con 0x7fad0c0c5330 2024-03-06T07:24:21.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad12cc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c0c8f40 con 0x7fad0c158db0 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad10a65640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad0c0c5330 0x7fad0c0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad10a65640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad0c0c5330 0x7fad0c0c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58954/0 (socket says 172.21.15.94:58954) 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad10a65640 1 -- 172.21.15.94:0/3485890316 learned_addr learned my addr 172.21.15.94:0/3485890316 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad10a65640 1 -- 172.21.15.94:0/3485890316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c158db0 msgr2=0x7fad0c15fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad10a65640 1 --2- 172.21.15.94:0/3485890316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c158db0 0x7fad0c15fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad10a65640 1 -- 172.21.15.94:0/3485890316 >> v1:172.21.15.145:6789/0 conn(0x7fad0c0c88b0 legacy=0x7fad0c1586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.598+0000 7fad10a65640 1 -- 172.21.15.94:0/3485890316 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c155390 con 0x7fad0c0c5330 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.599+0000 7fad10a65640 1 --2- 172.21.15.94:0/3485890316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c5330 0x7fad0c0c5710 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fad00009870 tx=0x7fad0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a172e4d2765c9ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.599+0000 7fad0b7fe640 1 -- 172.21.15.94:0/3485890316 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000317c0 con 0x7fad0c0c5330 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.599+0000 7fad0b7fe640 1 -- 172.21.15.94:0/3485890316 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad00037070 con 0x7fad0c0c5330 2024-03-06T07:24:21.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.599+0000 7fad0b7fe640 1 -- 172.21.15.94:0/3485890316 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00031b80 con 0x7fad0c0c5330 2024-03-06T07:24:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 -- 172.21.15.94:0/3485890316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c5330 msgr2=0x7fad0c0c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 --2- 172.21.15.94:0/3485890316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c5330 0x7fad0c0c5710 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fad00009870 tx=0x7fad0002b860 comp rx=0 tx=0).stop 2024-03-06T07:24:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 -- 172.21.15.94:0/3485890316 shutdown_connections 2024-03-06T07:24:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 --2- 172.21.15.94:0/3485890316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c158db0 0x7fad0c15fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 --2- 172.21.15.94:0/3485890316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c5330 0x7fad0c0c5710 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 -- 172.21.15.94:0/3485890316 >> 172.21.15.94:0/3485890316 conn(0x7fad0c0ac560 msgr2=0x7fad0c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 -- 172.21.15.94:0/3485890316 shutdown_connections 2024-03-06T07:24:21.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.600+0000 7fad12cc9640 1 -- 172.21.15.94:0/3485890316 wait complete. 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.601+0000 7fad12cc9640 1 Processor -- start 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.601+0000 7fad12cc9640 1 -- start start 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad12cc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c0c5330 0x7fad0c151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad12cc9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c88b0 0x7fad0c1ead40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad12cc9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad0c14c070 0x7fad0c1eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad12cc9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c1529d0 con 0x7fad0c0c88b0 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad12cc9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c06d090 con 0x7fad0c14c070 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad12cc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0c152b50 con 0x7fad0c0c5330 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad0bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c88b0 0x7fad0c1ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad0bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c88b0 0x7fad0c1ead40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58962/0 (socket says 172.21.15.94:58962) 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.602+0000 7fad0bfff640 1 -- 172.21.15.94:0/2342711880 learned_addr learned my addr 172.21.15.94:0/2342711880 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:21.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad10a65640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c0c5330 0x7fad0c151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad11266640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad0c14c070 0x7fad0c1eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad0bfff640 1 -- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c0c5330 msgr2=0x7fad0c151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad0bfff640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c0c5330 0x7fad0c151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad0bfff640 1 -- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad0c14c070 msgr2=0x7fad0c1eb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad0bfff640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad0c14c070 0x7fad0c1eb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad0bfff640 1 -- 172.21.15.94:0/2342711880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c0c1190 con 0x7fad0c0c88b0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad10a65640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c0c5330 0x7fad0c151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad0bfff640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c88b0 0x7fad0c1ead40 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7facf800c430 tx=0x7facf800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad097fa640 1 -- 172.21.15.94:0/2342711880 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8017070 con 0x7fad0c0c88b0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad097fa640 1 -- 172.21.15.94:0/2342711880 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf80035f0 con 0x7fad0c0c88b0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0c1eeca0 con 0x7fad0c0c88b0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad097fa640 1 -- 172.21.15.94:0/2342711880 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf80039e0 con 0x7fad0c0c88b0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.603+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0c157c90 con 0x7fad0c0c88b0 2024-03-06T07:24:21.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.604+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0c04f330 con 0x7fad0c0c88b0 2024-03-06T07:24:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.605+0000 7fad097fa640 1 -- 172.21.15.94:0/2342711880 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facf801a070 con 0x7fad0c0c88b0 2024-03-06T07:24:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.606+0000 7fad097fa640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7facf006b4d0 0x7facf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.606+0000 7fad097fa640 1 -- 172.21.15.94:0/2342711880 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7facf8089910 con 0x7fad0c0c88b0 2024-03-06T07:24:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.606+0000 7fad10a65640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7facf006b4d0 0x7facf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.606+0000 7fad10a65640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7facf006b4d0 0x7facf006d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fad00036000 tx=0x7fad0003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.610+0000 7fad097fa640 1 -- 172.21.15.94:0/2342711880 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7facf8057cf0 con 0x7fad0c0c88b0 2024-03-06T07:24:21.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.783+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad0c06cdc0 con 0x7facf006b4d0 2024-03-06T07:24:21.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.795+0000 7fad097fa640 1 -- 172.21.15.94:0/2342711880 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+491467 (secure 0 0 0) 0x7fad0c06cdc0 con 0x7facf006b4d0 2024-03-06T07:24:21.802 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:21.806 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:24:21.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7facf006b4d0 msgr2=0x7facf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7facf006b4d0 0x7facf006d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fad00036000 tx=0x7fad0003d040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c88b0 msgr2=0x7fad0c1ead40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c88b0 0x7fad0c1ead40 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7facf800c430 tx=0x7facf800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 shutdown_connections 2024-03-06T07:24:21.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7facf006b4d0 0x7facf006d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad0c14c070 0x7fad0c1eb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad0c0c88b0 0x7fad0c1ead40 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.799+0000 7fad12cc9640 1 --2- 172.21.15.94:0/2342711880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0c0c5330 0x7fad0c151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.800+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 >> 172.21.15.94:0/2342711880 conn(0x7fad0c0ac560 msgr2=0x7fad0c14f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.800+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 shutdown_connections 2024-03-06T07:24:21.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.800+0000 7fad12cc9640 1 -- 172.21.15.94:0/2342711880 wait complete. 2024-03-06T07:24:21.812 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-06T07:24:20.714092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264504,"kb_used_data":15040,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710216,"statfs":{"total":773094113280,"available":772823261184,"internally_reserved":0,"allocated":15400960,"data_stored":8539140,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.481974"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261571+0000","last_change":"2024-03-06T07:24:17.258953+0000","last_active":"2024-03-06T07:24:17.261571+0000","last_peered":"2024-03-06T07:24:17.261571+0000","last_clean":"2024-03-06T07:24:17.261571+0000","last_became_active":"2024-03-06T07:24:17.258657+0000","last_became_peered":"2024-03-06T07:24:17.258657+0000","last_unstale":"2024-03-06T07:24:17.261571+0000","last_undegraded":"2024-03-06T07:24:17.261571+0000","last_fullsized":"2024-03-06T07:24:17.261571+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:45.917698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251582+0000","last_change":"2024-03-06T07:24:03.878818+0000","last_active":"2024-03-06T07:24:17.251582+0000","last_peered":"2024-03-06T07:24:17.251582+0000","last_clean":"2024-03-06T07:24:17.251582+0000","last_became_active":"2024-03-06T07:24:03.878538+0000","last_became_peered":"2024-03-06T07:24:03.878538+0000","last_unstale":"2024-03-06T07:24:17.251582+0000","last_undegraded":"2024-03-06T07:24:17.251582+0000","last_fullsized":"2024-03-06T07:24:17.251582+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:43.198749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266396+0000","last_change":"2024-03-06T07:24:17.265631+0000","last_active":"2024-03-06T07:24:17.266396+0000","last_peered":"2024-03-06T07:24:17.266396+0000","last_clean":"2024-03-06T07:24:17.266396+0000","last_became_active":"2024-03-06T07:24:17.265372+0000","last_became_peered":"2024-03-06T07:24:17.265372+0000","last_unstale":"2024-03-06T07:24:17.266396+0000","last_undegraded":"2024-03-06T07:24:17.266396+0000","last_fullsized":"2024-03-06T07:24:17.266396+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:14.432264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258133+0000","last_change":"2024-03-06T07:24:03.887970+0000","last_active":"2024-03-06T07:24:17.258133+0000","last_peered":"2024-03-06T07:24:17.258133+0000","last_clean":"2024-03-06T07:24:17.258133+0000","last_became_active":"2024-03-06T07:24:03.887681+0000","last_became_peered":"2024-03-06T07:24:03.887681+0000","last_unstale":"2024-03-06T07:24:17.258133+0000","last_undegraded":"2024-03-06T07:24:17.258133+0000","last_fullsized":"2024-03-06T07:24:17.258133+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:32.211048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272424+0000","last_change":"2024-03-06T07:24:17.270677+0000","last_active":"2024-03-06T07:24:17.272424+0000","last_peered":"2024-03-06T07:24:17.272424+0000","last_clean":"2024-03-06T07:24:17.272424+0000","last_became_active":"2024-03-06T07:24:17.270031+0000","last_became_peered":"2024-03-06T07:24:17.270031+0000","last_unstale":"2024-03-06T07:24:17.272424+0000","last_undegraded":"2024-03-06T07:24:17.272424+0000","last_fullsized":"2024-03-06T07:24:17.272424+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:35.720668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256252+0000","last_change":"2024-03-06T07:24:03.885308+0000","last_active":"2024-03-06T07:24:17.256252+0000","last_peered":"2024-03-06T07:24:17.256252+0000","last_clean":"2024-03-06T07:24:17.256252+0000","last_became_active":"2024-03-06T07:24:03.885013+0000","last_became_peered":"2024-03-06T07:24:03.885013+0000","last_unstale":"2024-03-06T07:24:17.256252+0000","last_undegraded":"2024-03-06T07:24:17.256252+0000","last_fullsized":"2024-03-06T07:24:17.256252+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:45.339308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271835+0000","last_change":"2024-03-06T07:24:17.266889+0000","last_active":"2024-03-06T07:24:17.271835+0000","last_peered":"2024-03-06T07:24:17.271835+0000","last_clean":"2024-03-06T07:24:17.271835+0000","last_became_active":"2024-03-06T07:24:17.266598+0000","last_became_peered":"2024-03-06T07:24:17.266598+0000","last_unstale":"2024-03-06T07:24:17.271835+0000","last_undegraded":"2024-03-06T07:24:17.271835+0000","last_fullsized":"2024-03-06T07:24:17.271835+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:11.169809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256191+0000","last_change":"2024-03-06T07:24:03.885984+0000","last_active":"2024-03-06T07:24:17.256191+0000","last_peered":"2024-03-06T07:24:17.256191+0000","last_clean":"2024-03-06T07:24:17.256191+0000","last_became_active":"2024-03-06T07:24:03.885293+0000","last_became_peered":"2024-03-06T07:24:03.885293+0000","last_unstale":"2024-03-06T07:24:17.256191+0000","last_undegraded":"2024-03-06T07:24:17.256191+0000","last_fullsized":"2024-03-06T07:24:17.256191+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:34.171582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271652+0000","last_change":"2024-03-06T07:24:17.268782+0000","last_active":"2024-03-06T07:24:17.271652+0000","last_peered":"2024-03-06T07:24:17.271652+0000","last_clean":"2024-03-06T07:24:17.271652+0000","last_became_active":"2024-03-06T07:24:17.268442+0000","last_became_peered":"2024-03-06T07:24:17.268442+0000","last_unstale":"2024-03-06T07:24:17.271652+0000","last_undegraded":"2024-03-06T07:24:17.271652+0000","last_fullsized":"2024-03-06T07:24:17.271652+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:11.025284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247374+0000","last_change":"2024-03-06T07:24:03.884021+0000","last_active":"2024-03-06T07:24:17.247374+0000","last_peered":"2024-03-06T07:24:17.247374+0000","last_clean":"2024-03-06T07:24:17.247374+0000","last_became_active":"2024-03-06T07:24:03.883424+0000","last_became_peered":"2024-03-06T07:24:03.883424+0000","last_unstale":"2024-03-06T07:24:17.247374+0000","last_undegraded":"2024-03-06T07:24:17.247374+0000","last_fullsized":"2024-03-06T07:24:17.247374+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276015+0000","last_change":"2024-03-06T07:24:17.271626+0000","last_active":"2024-03-06T07:24:17.276015+0000","last_peered":"2024-03-06T07:24:17.276015+0000","last_clean":"2024-03-06T07:24:17.276015+0000","last_became_active":"2024-03-06T07:24:17.270271+0000","last_became_peered":"2024-03-06T07:24:17.270271+0000","last_unstale":"2024-03-06T07:24:17.276015+0000","last_undegraded":"2024-03-06T07:24:17.276015+0000","last_fullsized":"2024-03-06T07:24:17.276015+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:58.896993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257497+0000","last_change":"2024-03-06T07:24:03.889706+0000","last_active":"2024-03-06T07:24:17.257497+0000","last_peered":"2024-03-06T07:24:17.257497+0000","last_clean":"2024-03-06T07:24:17.257497+0000","last_became_active":"2024-03-06T07:24:03.889392+0000","last_became_peered":"2024-03-06T07:24:03.889392+0000","last_unstale":"2024-03-06T07:24:17.257497+0000","last_undegraded":"2024-03-06T07:24:17.257497+0000","last_fullsized":"2024-03-06T07:24:17.257497+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277226+0000","last_change":"2024-03-06T07:24:17.274041+0000","last_active":"2024-03-06T07:24:17.277226+0000","last_peered":"2024-03-06T07:24:17.277226+0000","last_clean":"2024-03-06T07:24:17.277226+0000","last_became_active":"2024-03-06T07:24:17.273745+0000","last_became_peered":"2024-03-06T07:24:17.273745+0000","last_unstale":"2024-03-06T07:24:17.277226+0000","last_undegraded":"2024-03-06T07:24:17.277226+0000","last_fullsized":"2024-03-06T07:24:17.277226+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:01.165624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262013+0000","last_change":"2024-03-06T07:24:03.889999+0000","last_active":"2024-03-06T07:24:17.262013+0000","last_peered":"2024-03-06T07:24:17.262013+0000","last_clean":"2024-03-06T07:24:17.262013+0000","last_became_active":"2024-03-06T07:24:03.889432+0000","last_became_peered":"2024-03-06T07:24:03.889432+0000","last_unstale":"2024-03-06T07:24:17.262013+0000","last_undegraded":"2024-03-06T07:24:17.262013+0000","last_fullsized":"2024-03-06T07:24:17.262013+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.265703+0000","last_change":"2024-03-06T07:24:17.262868+0000","last_active":"2024-03-06T07:24:17.265703+0000","last_peered":"2024-03-06T07:24:17.265703+0000","last_clean":"2024-03-06T07:24:17.265703+0000","last_became_active":"2024-03-06T07:24:17.262581+0000","last_became_peered":"2024-03-06T07:24:17.262581+0000","last_unstale":"2024-03-06T07:24:17.265703+0000","last_undegraded":"2024-03-06T07:24:17.265703+0000","last_fullsized":"2024-03-06T07:24:17.265703+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:45.980040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255844+0000","last_change":"2024-03-06T07:24:03.895907+0000","last_active":"2024-03-06T07:24:17.255844+0000","last_peered":"2024-03-06T07:24:17.255844+0000","last_clean":"2024-03-06T07:24:17.255844+0000","last_became_active":"2024-03-06T07:24:03.895663+0000","last_became_peered":"2024-03-06T07:24:03.895663+0000","last_unstale":"2024-03-06T07:24:17.255844+0000","last_undegraded":"2024-03-06T07:24:17.255844+0000","last_fullsized":"2024-03-06T07:24:17.255844+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.267806+0000","last_change":"2024-03-06T07:24:17.265846+0000","last_active":"2024-03-06T07:24:17.267806+0000","last_peered":"2024-03-06T07:24:17.267806+0000","last_clean":"2024-03-06T07:24:17.267806+0000","last_became_active":"2024-03-06T07:24:17.265556+0000","last_became_peered":"2024-03-06T07:24:17.265556+0000","last_unstale":"2024-03-06T07:24:17.267806+0000","last_undegraded":"2024-03-06T07:24:17.267806+0000","last_fullsized":"2024-03-06T07:24:17.267806+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:53.042094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258297+0000","last_change":"2024-03-06T07:24:03.887528+0000","last_active":"2024-03-06T07:24:17.258297+0000","last_peered":"2024-03-06T07:24:17.258297+0000","last_clean":"2024-03-06T07:24:17.258297+0000","last_became_active":"2024-03-06T07:24:03.887269+0000","last_became_peered":"2024-03-06T07:24:03.887269+0000","last_unstale":"2024-03-06T07:24:17.258297+0000","last_undegraded":"2024-03-06T07:24:17.258297+0000","last_fullsized":"2024-03-06T07:24:17.258297+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:32.211048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272068+0000","last_change":"2024-03-06T07:24:17.267854+0000","last_active":"2024-03-06T07:24:17.272068+0000","last_peered":"2024-03-06T07:24:17.272068+0000","last_clean":"2024-03-06T07:24:17.272068+0000","last_became_active":"2024-03-06T07:24:17.267442+0000","last_became_peered":"2024-03-06T07:24:17.267442+0000","last_unstale":"2024-03-06T07:24:17.272068+0000","last_undegraded":"2024-03-06T07:24:17.272068+0000","last_fullsized":"2024-03-06T07:24:17.272068+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.451737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256461+0000","last_change":"2024-03-06T07:24:03.897717+0000","last_active":"2024-03-06T07:24:17.256461+0000","last_peered":"2024-03-06T07:24:17.256461+0000","last_clean":"2024-03-06T07:24:17.256461+0000","last_became_active":"2024-03-06T07:24:03.897445+0000","last_became_peered":"2024-03-06T07:24:03.897445+0000","last_unstale":"2024-03-06T07:24:17.256461+0000","last_undegraded":"2024-03-06T07:24:17.256461+0000","last_fullsized":"2024-03-06T07:24:17.256461+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.492678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271829+0000","last_change":"2024-03-06T07:24:17.267888+0000","last_active":"2024-03-06T07:24:17.271829+0000","last_peered":"2024-03-06T07:24:17.271829+0000","last_clean":"2024-03-06T07:24:17.271829+0000","last_became_active":"2024-03-06T07:24:17.267656+0000","last_became_peered":"2024-03-06T07:24:17.267656+0000","last_unstale":"2024-03-06T07:24:17.271829+0000","last_undegraded":"2024-03-06T07:24:17.271829+0000","last_fullsized":"2024-03-06T07:24:17.271829+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:14.432264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258213+0000","last_change":"2024-03-06T07:24:03.892692+0000","last_active":"2024-03-06T07:24:17.258213+0000","last_peered":"2024-03-06T07:24:17.258213+0000","last_clean":"2024-03-06T07:24:17.258213+0000","last_became_active":"2024-03-06T07:24:03.892434+0000","last_became_peered":"2024-03-06T07:24:03.892434+0000","last_unstale":"2024-03-06T07:24:17.258213+0000","last_undegraded":"2024-03-06T07:24:17.258213+0000","last_fullsized":"2024-03-06T07:24:17.258213+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:48.146871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277213+0000","last_change":"2024-03-06T07:24:17.273322+0000","last_active":"2024-03-06T07:24:17.277213+0000","last_peered":"2024-03-06T07:24:17.277213+0000","last_clean":"2024-03-06T07:24:17.277213+0000","last_became_active":"2024-03-06T07:24:17.273061+0000","last_became_peered":"2024-03-06T07:24:17.273061+0000","last_unstale":"2024-03-06T07:24:17.277213+0000","last_undegraded":"2024-03-06T07:24:17.277213+0000","last_fullsized":"2024-03-06T07:24:17.277213+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:22.020279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262127+0000","last_change":"2024-03-06T07:24:03.887559+0000","last_active":"2024-03-06T07:24:17.262127+0000","last_peered":"2024-03-06T07:24:17.262127+0000","last_clean":"2024-03-06T07:24:17.262127+0000","last_became_active":"2024-03-06T07:24:03.887275+0000","last_became_peered":"2024-03-06T07:24:03.887275+0000","last_unstale":"2024-03-06T07:24:17.262127+0000","last_undegraded":"2024-03-06T07:24:17.262127+0000","last_fullsized":"2024-03-06T07:24:17.262127+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.492678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271733+0000","last_change":"2024-03-06T07:24:17.265162+0000","last_active":"2024-03-06T07:24:17.271733+0000","last_peered":"2024-03-06T07:24:17.271733+0000","last_clean":"2024-03-06T07:24:17.271733+0000","last_became_active":"2024-03-06T07:24:17.264894+0000","last_became_peered":"2024-03-06T07:24:17.264894+0000","last_unstale":"2024-03-06T07:24:17.271733+0000","last_undegraded":"2024-03-06T07:24:17.271733+0000","last_fullsized":"2024-03-06T07:24:17.271733+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:06.739402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250535+0000","last_change":"2024-03-06T07:24:03.894465+0000","last_active":"2024-03-06T07:24:17.250535+0000","last_peered":"2024-03-06T07:24:17.250535+0000","last_clean":"2024-03-06T07:24:17.250535+0000","last_became_active":"2024-03-06T07:24:03.894192+0000","last_became_peered":"2024-03-06T07:24:03.894192+0000","last_unstale":"2024-03-06T07:24:17.250535+0000","last_undegraded":"2024-03-06T07:24:17.250535+0000","last_fullsized":"2024-03-06T07:24:17.250535+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:48.848727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277189+0000","last_change":"2024-03-06T07:24:17.274474+0000","last_active":"2024-03-06T07:24:17.277189+0000","last_peered":"2024-03-06T07:24:17.277189+0000","last_clean":"2024-03-06T07:24:17.277189+0000","last_became_active":"2024-03-06T07:24:17.274029+0000","last_became_peered":"2024-03-06T07:24:17.274029+0000","last_unstale":"2024-03-06T07:24:17.277189+0000","last_undegraded":"2024-03-06T07:24:17.277189+0000","last_fullsized":"2024-03-06T07:24:17.277189+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:08.004043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247179+0000","last_change":"2024-03-06T07:24:03.890072+0000","last_active":"2024-03-06T07:24:17.247179+0000","last_peered":"2024-03-06T07:24:17.247179+0000","last_clean":"2024-03-06T07:24:17.247179+0000","last_became_active":"2024-03-06T07:24:03.889801+0000","last_became_peered":"2024-03-06T07:24:03.889801+0000","last_unstale":"2024-03-06T07:24:17.247179+0000","last_undegraded":"2024-03-06T07:24:17.247179+0000","last_fullsized":"2024-03-06T07:24:17.247179+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272030+0000","last_change":"2024-03-06T07:24:17.269417+0000","last_active":"2024-03-06T07:24:17.272030+0000","last_peered":"2024-03-06T07:24:17.272030+0000","last_clean":"2024-03-06T07:24:17.272030+0000","last_became_active":"2024-03-06T07:24:17.269019+0000","last_became_peered":"2024-03-06T07:24:17.269019+0000","last_unstale":"2024-03-06T07:24:17.272030+0000","last_undegraded":"2024-03-06T07:24:17.272030+0000","last_fullsized":"2024-03-06T07:24:17.272030+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.975875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258696+0000","last_change":"2024-03-06T07:24:03.887600+0000","last_active":"2024-03-06T07:24:17.258696+0000","last_peered":"2024-03-06T07:24:17.258696+0000","last_clean":"2024-03-06T07:24:17.258696+0000","last_became_active":"2024-03-06T07:24:03.887214+0000","last_became_peered":"2024-03-06T07:24:03.887214+0000","last_unstale":"2024-03-06T07:24:17.258696+0000","last_undegraded":"2024-03-06T07:24:17.258696+0000","last_fullsized":"2024-03-06T07:24:17.258696+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:16.405620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270626+0000","last_change":"2024-03-06T07:24:17.267764+0000","last_active":"2024-03-06T07:24:17.270626+0000","last_peered":"2024-03-06T07:24:17.270626+0000","last_clean":"2024-03-06T07:24:17.270626+0000","last_became_active":"2024-03-06T07:24:17.267472+0000","last_became_peered":"2024-03-06T07:24:17.267472+0000","last_unstale":"2024-03-06T07:24:17.270626+0000","last_undegraded":"2024-03-06T07:24:17.270626+0000","last_fullsized":"2024-03-06T07:24:17.270626+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:17.556623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251671+0000","last_change":"2024-03-06T07:24:03.884458+0000","last_active":"2024-03-06T07:24:17.251671+0000","last_peered":"2024-03-06T07:24:17.251671+0000","last_clean":"2024-03-06T07:24:17.251671+0000","last_became_active":"2024-03-06T07:24:03.883983+0000","last_became_peered":"2024-03-06T07:24:03.883983+0000","last_unstale":"2024-03-06T07:24:17.251671+0000","last_undegraded":"2024-03-06T07:24:17.251671+0000","last_fullsized":"2024-03-06T07:24:17.251671+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:36.691438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.265418+0000","last_change":"2024-03-06T07:24:17.263338+0000","last_active":"2024-03-06T07:24:17.265418+0000","last_peered":"2024-03-06T07:24:17.265418+0000","last_clean":"2024-03-06T07:24:17.265418+0000","last_became_active":"2024-03-06T07:24:17.263048+0000","last_became_peered":"2024-03-06T07:24:17.263048+0000","last_unstale":"2024-03-06T07:24:17.265418+0000","last_undegraded":"2024-03-06T07:24:17.265418+0000","last_fullsized":"2024-03-06T07:24:17.265418+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:49.453665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258388+0000","last_change":"2024-03-06T07:24:03.888459+0000","last_active":"2024-03-06T07:24:17.258388+0000","last_peered":"2024-03-06T07:24:17.258388+0000","last_clean":"2024-03-06T07:24:17.258388+0000","last_became_active":"2024-03-06T07:24:03.888203+0000","last_became_peered":"2024-03-06T07:24:03.888203+0000","last_unstale":"2024-03-06T07:24:17.258388+0000","last_undegraded":"2024-03-06T07:24:17.258388+0000","last_fullsized":"2024-03-06T07:24:17.258388+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:35.615657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276912+0000","last_change":"2024-03-06T07:24:17.271635+0000","last_active":"2024-03-06T07:24:17.276912+0000","last_peered":"2024-03-06T07:24:17.276912+0000","last_clean":"2024-03-06T07:24:17.276912+0000","last_became_active":"2024-03-06T07:24:17.267039+0000","last_became_peered":"2024-03-06T07:24:17.267039+0000","last_unstale":"2024-03-06T07:24:17.276912+0000","last_undegraded":"2024-03-06T07:24:17.276912+0000","last_fullsized":"2024-03-06T07:24:17.276912+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:26.200535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251262+0000","last_change":"2024-03-06T07:24:03.885816+0000","last_active":"2024-03-06T07:24:17.251262+0000","last_peered":"2024-03-06T07:24:17.251262+0000","last_clean":"2024-03-06T07:24:17.251262+0000","last_became_active":"2024-03-06T07:24:03.885561+0000","last_became_peered":"2024-03-06T07:24:03.885561+0000","last_unstale":"2024-03-06T07:24:17.251262+0000","last_undegraded":"2024-03-06T07:24:17.251262+0000","last_fullsized":"2024-03-06T07:24:17.251262+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:33.243851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276893+0000","last_change":"2024-03-06T07:24:17.270974+0000","last_active":"2024-03-06T07:24:17.276893+0000","last_peered":"2024-03-06T07:24:17.276893+0000","last_clean":"2024-03-06T07:24:17.276893+0000","last_became_active":"2024-03-06T07:24:17.270510+0000","last_became_peered":"2024-03-06T07:24:17.270510+0000","last_unstale":"2024-03-06T07:24:17.276893+0000","last_undegraded":"2024-03-06T07:24:17.276893+0000","last_fullsized":"2024-03-06T07:24:17.276893+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:51.222298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261409+0000","last_change":"2024-03-06T07:24:03.887681+0000","last_active":"2024-03-06T07:24:17.261409+0000","last_peered":"2024-03-06T07:24:17.261409+0000","last_clean":"2024-03-06T07:24:17.261409+0000","last_became_active":"2024-03-06T07:24:03.887375+0000","last_became_peered":"2024-03-06T07:24:03.887375+0000","last_unstale":"2024-03-06T07:24:17.261409+0000","last_undegraded":"2024-03-06T07:24:17.261409+0000","last_fullsized":"2024-03-06T07:24:17.261409+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:06.466254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.281203+0000","last_change":"2024-03-06T07:24:17.277664+0000","last_active":"2024-03-06T07:24:17.281203+0000","last_peered":"2024-03-06T07:24:17.281203+0000","last_clean":"2024-03-06T07:24:17.281203+0000","last_became_active":"2024-03-06T07:24:17.277387+0000","last_became_peered":"2024-03-06T07:24:17.277387+0000","last_unstale":"2024-03-06T07:24:17.281203+0000","last_undegraded":"2024-03-06T07:24:17.281203+0000","last_fullsized":"2024-03-06T07:24:17.281203+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:14.626596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250586+0000","last_change":"2024-03-06T07:24:03.884294+0000","last_active":"2024-03-06T07:24:17.250586+0000","last_peered":"2024-03-06T07:24:17.250586+0000","last_clean":"2024-03-06T07:24:17.250586+0000","last_became_active":"2024-03-06T07:24:03.884050+0000","last_became_peered":"2024-03-06T07:24:03.884050+0000","last_unstale":"2024-03-06T07:24:17.250586+0000","last_undegraded":"2024-03-06T07:24:17.250586+0000","last_fullsized":"2024-03-06T07:24:17.250586+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:01.779181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275578+0000","last_change":"2024-03-06T07:24:17.274560+0000","last_active":"2024-03-06T07:24:17.275578+0000","last_peered":"2024-03-06T07:24:17.275578+0000","last_clean":"2024-03-06T07:24:17.275578+0000","last_became_active":"2024-03-06T07:24:17.272120+0000","last_became_peered":"2024-03-06T07:24:17.272120+0000","last_unstale":"2024-03-06T07:24:17.275578+0000","last_undegraded":"2024-03-06T07:24:17.275578+0000","last_fullsized":"2024-03-06T07:24:17.275578+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:43.160668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250625+0000","last_change":"2024-03-06T07:24:03.894163+0000","last_active":"2024-03-06T07:24:17.250625+0000","last_peered":"2024-03-06T07:24:17.250625+0000","last_clean":"2024-03-06T07:24:17.250625+0000","last_became_active":"2024-03-06T07:24:03.893792+0000","last_became_peered":"2024-03-06T07:24:03.893792+0000","last_unstale":"2024-03-06T07:24:17.250625+0000","last_undegraded":"2024-03-06T07:24:17.250625+0000","last_fullsized":"2024-03-06T07:24:17.250625+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:43.594819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271809+0000","last_change":"2024-03-06T07:24:17.267437+0000","last_active":"2024-03-06T07:24:17.271809+0000","last_peered":"2024-03-06T07:24:17.271809+0000","last_clean":"2024-03-06T07:24:17.271809+0000","last_became_active":"2024-03-06T07:24:17.267195+0000","last_became_peered":"2024-03-06T07:24:17.267195+0000","last_unstale":"2024-03-06T07:24:17.271809+0000","last_undegraded":"2024-03-06T07:24:17.271809+0000","last_fullsized":"2024-03-06T07:24:17.271809+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:32.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257601+0000","last_change":"2024-03-06T07:24:03.892392+0000","last_active":"2024-03-06T07:24:17.257601+0000","last_peered":"2024-03-06T07:24:17.257601+0000","last_clean":"2024-03-06T07:24:17.257601+0000","last_became_active":"2024-03-06T07:24:03.892131+0000","last_became_peered":"2024-03-06T07:24:03.892131+0000","last_unstale":"2024-03-06T07:24:17.257601+0000","last_undegraded":"2024-03-06T07:24:17.257601+0000","last_fullsized":"2024-03-06T07:24:17.257601+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271858+0000","last_change":"2024-03-06T07:24:17.268503+0000","last_active":"2024-03-06T07:24:17.271858+0000","last_peered":"2024-03-06T07:24:17.271858+0000","last_clean":"2024-03-06T07:24:17.271858+0000","last_became_active":"2024-03-06T07:24:17.268253+0000","last_became_peered":"2024-03-06T07:24:17.268253+0000","last_unstale":"2024-03-06T07:24:17.271858+0000","last_undegraded":"2024-03-06T07:24:17.271858+0000","last_fullsized":"2024-03-06T07:24:17.271858+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.451737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.246925+0000","last_change":"2024-03-06T07:24:03.890324+0000","last_active":"2024-03-06T07:24:17.246925+0000","last_peered":"2024-03-06T07:24:17.246925+0000","last_clean":"2024-03-06T07:24:17.246925+0000","last_became_active":"2024-03-06T07:24:03.890037+0000","last_became_peered":"2024-03-06T07:24:03.890037+0000","last_unstale":"2024-03-06T07:24:17.246925+0000","last_undegraded":"2024-03-06T07:24:17.246925+0000","last_fullsized":"2024-03-06T07:24:17.246925+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:57.609556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.274406+0000","last_change":"2024-03-06T07:24:17.272731+0000","last_active":"2024-03-06T07:24:17.274406+0000","last_peered":"2024-03-06T07:24:17.274406+0000","last_clean":"2024-03-06T07:24:17.274406+0000","last_became_active":"2024-03-06T07:24:17.272481+0000","last_became_peered":"2024-03-06T07:24:17.272481+0000","last_unstale":"2024-03-06T07:24:17.274406+0000","last_undegraded":"2024-03-06T07:24:17.274406+0000","last_fullsized":"2024-03-06T07:24:17.274406+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:51.196370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251786+0000","last_change":"2024-03-06T07:24:03.884245+0000","last_active":"2024-03-06T07:24:17.251786+0000","last_peered":"2024-03-06T07:24:17.251786+0000","last_clean":"2024-03-06T07:24:17.251786+0000","last_became_active":"2024-03-06T07:24:03.883649+0000","last_became_peered":"2024-03-06T07:24:03.883649+0000","last_unstale":"2024-03-06T07:24:17.251786+0000","last_undegraded":"2024-03-06T07:24:17.251786+0000","last_fullsized":"2024-03-06T07:24:17.251786+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:36.283613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275551+0000","last_change":"2024-03-06T07:24:17.271088+0000","last_active":"2024-03-06T07:24:17.275551+0000","last_peered":"2024-03-06T07:24:17.275551+0000","last_clean":"2024-03-06T07:24:17.275551+0000","last_became_active":"2024-03-06T07:24:17.270793+0000","last_became_peered":"2024-03-06T07:24:17.270793+0000","last_unstale":"2024-03-06T07:24:17.275551+0000","last_undegraded":"2024-03-06T07:24:17.275551+0000","last_fullsized":"2024-03-06T07:24:17.275551+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.629152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251886+0000","last_change":"2024-03-06T07:24:03.898098+0000","last_active":"2024-03-06T07:24:17.251886+0000","last_peered":"2024-03-06T07:24:17.251886+0000","last_clean":"2024-03-06T07:24:17.251886+0000","last_became_active":"2024-03-06T07:24:03.895994+0000","last_became_peered":"2024-03-06T07:24:03.895994+0000","last_unstale":"2024-03-06T07:24:17.251886+0000","last_undegraded":"2024-03-06T07:24:17.251886+0000","last_fullsized":"2024-03-06T07:24:17.251886+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:31.833931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275076+0000","last_change":"2024-03-06T07:24:17.264851+0000","last_active":"2024-03-06T07:24:17.275076+0000","last_peered":"2024-03-06T07:24:17.275076+0000","last_clean":"2024-03-06T07:24:17.275076+0000","last_became_active":"2024-03-06T07:24:17.264602+0000","last_became_peered":"2024-03-06T07:24:17.264602+0000","last_unstale":"2024-03-06T07:24:17.275076+0000","last_undegraded":"2024-03-06T07:24:17.275076+0000","last_fullsized":"2024-03-06T07:24:17.275076+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:19.099433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.249857+0000","last_change":"2024-03-06T07:24:03.879536+0000","last_active":"2024-03-06T07:24:17.249857+0000","last_peered":"2024-03-06T07:24:17.249857+0000","last_clean":"2024-03-06T07:24:17.249857+0000","last_became_active":"2024-03-06T07:24:03.879280+0000","last_became_peered":"2024-03-06T07:24:03.879280+0000","last_unstale":"2024-03-06T07:24:17.249857+0000","last_undegraded":"2024-03-06T07:24:17.249857+0000","last_fullsized":"2024-03-06T07:24:17.249857+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:09.372273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266178+0000","last_change":"2024-03-06T07:24:17.262753+0000","last_active":"2024-03-06T07:24:17.266178+0000","last_peered":"2024-03-06T07:24:17.266178+0000","last_clean":"2024-03-06T07:24:17.266178+0000","last_became_active":"2024-03-06T07:24:17.262562+0000","last_became_peered":"2024-03-06T07:24:17.262562+0000","last_unstale":"2024-03-06T07:24:17.266178+0000","last_undegraded":"2024-03-06T07:24:17.266178+0000","last_fullsized":"2024-03-06T07:24:17.266178+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:47.972512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263555+0000","last_change":"2024-03-06T07:24:03.893792+0000","last_active":"2024-03-06T07:24:17.263555+0000","last_peered":"2024-03-06T07:24:17.263555+0000","last_clean":"2024-03-06T07:24:17.263555+0000","last_became_active":"2024-03-06T07:24:03.893240+0000","last_became_peered":"2024-03-06T07:24:03.893240+0000","last_unstale":"2024-03-06T07:24:17.263555+0000","last_undegraded":"2024-03-06T07:24:17.263555+0000","last_fullsized":"2024-03-06T07:24:17.263555+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:51.104075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.283587+0000","last_change":"2024-03-06T07:24:17.278327+0000","last_active":"2024-03-06T07:24:17.283587+0000","last_peered":"2024-03-06T07:24:17.283587+0000","last_clean":"2024-03-06T07:24:17.283587+0000","last_became_active":"2024-03-06T07:24:17.277812+0000","last_became_peered":"2024-03-06T07:24:17.277812+0000","last_unstale":"2024-03-06T07:24:17.283587+0000","last_undegraded":"2024-03-06T07:24:17.283587+0000","last_fullsized":"2024-03-06T07:24:17.283587+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:40.803855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250320+0000","last_change":"2024-03-06T07:24:03.894449+0000","last_active":"2024-03-06T07:24:17.250320+0000","last_peered":"2024-03-06T07:24:17.250320+0000","last_clean":"2024-03-06T07:24:17.250320+0000","last_became_active":"2024-03-06T07:24:03.894007+0000","last_became_peered":"2024-03-06T07:24:03.894007+0000","last_unstale":"2024-03-06T07:24:17.250320+0000","last_undegraded":"2024-03-06T07:24:17.250320+0000","last_fullsized":"2024-03-06T07:24:17.250320+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:37.861026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266260+0000","last_change":"2024-03-06T07:24:17.264381+0000","last_active":"2024-03-06T07:24:17.266260+0000","last_peered":"2024-03-06T07:24:17.266260+0000","last_clean":"2024-03-06T07:24:17.266260+0000","last_became_active":"2024-03-06T07:24:17.264124+0000","last_became_peered":"2024-03-06T07:24:17.264124+0000","last_unstale":"2024-03-06T07:24:17.266260+0000","last_undegraded":"2024-03-06T07:24:17.266260+0000","last_fullsized":"2024-03-06T07:24:17.266260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:46.610169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262851+0000","last_change":"2024-03-06T07:24:03.887453+0000","last_active":"2024-03-06T07:24:17.262851+0000","last_peered":"2024-03-06T07:24:17.262851+0000","last_clean":"2024-03-06T07:24:17.262851+0000","last_became_active":"2024-03-06T07:24:03.887167+0000","last_became_peered":"2024-03-06T07:24:03.887167+0000","last_unstale":"2024-03-06T07:24:17.262851+0000","last_undegraded":"2024-03-06T07:24:17.262851+0000","last_fullsized":"2024-03-06T07:24:17.262851+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:01.897485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277168+0000","last_change":"2024-03-06T07:24:17.274281+0000","last_active":"2024-03-06T07:24:17.277168+0000","last_peered":"2024-03-06T07:24:17.277168+0000","last_clean":"2024-03-06T07:24:17.277168+0000","last_became_active":"2024-03-06T07:24:17.273563+0000","last_became_peered":"2024-03-06T07:24:17.273563+0000","last_unstale":"2024-03-06T07:24:17.277168+0000","last_undegraded":"2024-03-06T07:24:17.277168+0000","last_fullsized":"2024-03-06T07:24:17.277168+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:56.473600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251079+0000","last_change":"2024-03-06T07:24:03.885011+0000","last_active":"2024-03-06T07:24:17.251079+0000","last_peered":"2024-03-06T07:24:17.251079+0000","last_clean":"2024-03-06T07:24:17.251079+0000","last_became_active":"2024-03-06T07:24:03.884571+0000","last_became_peered":"2024-03-06T07:24:03.884571+0000","last_unstale":"2024-03-06T07:24:17.251079+0000","last_undegraded":"2024-03-06T07:24:17.251079+0000","last_fullsized":"2024-03-06T07:24:17.251079+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:12.327628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278283+0000","last_change":"2024-03-06T07:24:17.272899+0000","last_active":"2024-03-06T07:24:17.278283+0000","last_peered":"2024-03-06T07:24:17.278283+0000","last_clean":"2024-03-06T07:24:17.278283+0000","last_became_active":"2024-03-06T07:24:17.272624+0000","last_became_peered":"2024-03-06T07:24:17.272624+0000","last_unstale":"2024-03-06T07:24:17.278283+0000","last_undegraded":"2024-03-06T07:24:17.278283+0000","last_fullsized":"2024-03-06T07:24:17.278283+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:15.051999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258789+0000","last_change":"2024-03-06T07:24:03.891048+0000","last_active":"2024-03-06T07:24:17.258789+0000","last_peered":"2024-03-06T07:24:17.258789+0000","last_clean":"2024-03-06T07:24:17.258789+0000","last_became_active":"2024-03-06T07:24:03.890787+0000","last_became_peered":"2024-03-06T07:24:03.890787+0000","last_unstale":"2024-03-06T07:24:17.258789+0000","last_undegraded":"2024-03-06T07:24:17.258789+0000","last_fullsized":"2024-03-06T07:24:17.258789+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:39.286189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278425+0000","last_change":"2024-03-06T07:24:17.273563+0000","last_active":"2024-03-06T07:24:17.278425+0000","last_peered":"2024-03-06T07:24:17.278425+0000","last_clean":"2024-03-06T07:24:17.278425+0000","last_became_active":"2024-03-06T07:24:17.273293+0000","last_became_peered":"2024-03-06T07:24:17.273293+0000","last_unstale":"2024-03-06T07:24:17.278425+0000","last_undegraded":"2024-03-06T07:24:17.278425+0000","last_fullsized":"2024-03-06T07:24:17.278425+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:50.546588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252848+0000","last_change":"2024-03-06T07:24:03.884181+0000","last_active":"2024-03-06T07:24:17.252848+0000","last_peered":"2024-03-06T07:24:17.252848+0000","last_clean":"2024-03-06T07:24:17.252848+0000","last_became_active":"2024-03-06T07:24:03.883923+0000","last_became_peered":"2024-03-06T07:24:03.883923+0000","last_unstale":"2024-03-06T07:24:17.252848+0000","last_undegraded":"2024-03-06T07:24:17.252848+0000","last_fullsized":"2024-03-06T07:24:17.252848+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:04.190305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266571+0000","last_change":"2024-03-06T07:24:17.263544+0000","last_active":"2024-03-06T07:24:17.266571+0000","last_peered":"2024-03-06T07:24:17.266571+0000","last_clean":"2024-03-06T07:24:17.266571+0000","last_became_active":"2024-03-06T07:24:17.263162+0000","last_became_peered":"2024-03-06T07:24:17.263162+0000","last_unstale":"2024-03-06T07:24:17.266571+0000","last_undegraded":"2024-03-06T07:24:17.266571+0000","last_fullsized":"2024-03-06T07:24:17.266571+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:47.502877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.246963+0000","last_change":"2024-03-06T07:24:16.249577+0000","last_active":"2024-03-06T07:24:17.246963+0000","last_peered":"2024-03-06T07:24:17.246963+0000","last_clean":"2024-03-06T07:24:17.246963+0000","last_became_active":"2024-03-06T07:24:16.249271+0000","last_became_peered":"2024-03-06T07:24:16.249271+0000","last_unstale":"2024-03-06T07:24:17.246963+0000","last_undegraded":"2024-03-06T07:24:17.246963+0000","last_fullsized":"2024-03-06T07:24:17.246963+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:39.799276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258184+0000","last_change":"2024-03-06T07:24:03.893008+0000","last_active":"2024-03-06T07:24:17.258184+0000","last_peered":"2024-03-06T07:24:17.258184+0000","last_clean":"2024-03-06T07:24:17.258184+0000","last_became_active":"2024-03-06T07:24:03.892568+0000","last_became_peered":"2024-03-06T07:24:03.892568+0000","last_unstale":"2024-03-06T07:24:17.258184+0000","last_undegraded":"2024-03-06T07:24:17.258184+0000","last_fullsized":"2024-03-06T07:24:17.258184+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:01.260277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"148'4","reported_seq":24,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251820+0000","last_change":"2024-03-06T07:24:02.889021+0000","last_active":"2024-03-06T07:24:17.251820+0000","last_peered":"2024-03-06T07:24:17.251820+0000","last_clean":"2024-03-06T07:24:17.251820+0000","last_became_active":"2024-03-06T07:24:02.888760+0000","last_became_peered":"2024-03-06T07:24:02.888760+0000","last_unstale":"2024-03-06T07:24:17.251820+0000","last_undegraded":"2024-03-06T07:24:17.251820+0000","last_fullsized":"2024-03-06T07:24:17.251820+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:56.709899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.266608+0000","last_change":"2024-03-06T07:24:17.262650+0000","last_active":"2024-03-06T07:24:17.266608+0000","last_peered":"2024-03-06T07:24:17.266608+0000","last_clean":"2024-03-06T07:24:17.266608+0000","last_became_active":"2024-03-06T07:24:17.262347+0000","last_became_peered":"2024-03-06T07:24:17.262347+0000","last_unstale":"2024-03-06T07:24:17.266608+0000","last_undegraded":"2024-03-06T07:24:17.266608+0000","last_fullsized":"2024-03-06T07:24:17.266608+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:29.771938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"152'8","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.281141+0000","last_change":"2024-03-06T07:24:16.254291+0000","last_active":"2024-03-06T07:24:17.281141+0000","last_peered":"2024-03-06T07:24:17.281141+0000","last_clean":"2024-03-06T07:24:17.281141+0000","last_became_active":"2024-03-06T07:24:16.253983+0000","last_became_peered":"2024-03-06T07:24:16.253983+0000","last_unstale":"2024-03-06T07:24:17.281141+0000","last_undegraded":"2024-03-06T07:24:17.281141+0000","last_fullsized":"2024-03-06T07:24:17.281141+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:47.977396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250274+0000","last_change":"2024-03-06T07:24:03.895263+0000","last_active":"2024-03-06T07:24:17.250274+0000","last_peered":"2024-03-06T07:24:17.250274+0000","last_clean":"2024-03-06T07:24:17.250274+0000","last_became_active":"2024-03-06T07:24:03.894999+0000","last_became_peered":"2024-03-06T07:24:03.894999+0000","last_unstale":"2024-03-06T07:24:17.250274+0000","last_undegraded":"2024-03-06T07:24:17.250274+0000","last_fullsized":"2024-03-06T07:24:17.250274+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:37.830052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.248048+0000","last_change":"2024-03-06T07:24:02.881058+0000","last_active":"2024-03-06T07:24:17.248048+0000","last_peered":"2024-03-06T07:24:17.248048+0000","last_clean":"2024-03-06T07:24:17.248048+0000","last_became_active":"2024-03-06T07:24:02.880664+0000","last_became_peered":"2024-03-06T07:24:02.880664+0000","last_unstale":"2024-03-06T07:24:17.248048+0000","last_undegraded":"2024-03-06T07:24:17.248048+0000","last_fullsized":"2024-03-06T07:24:17.248048+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:06.978957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271616+0000","last_change":"2024-03-06T07:24:17.269631+0000","last_active":"2024-03-06T07:24:17.271616+0000","last_peered":"2024-03-06T07:24:17.271616+0000","last_clean":"2024-03-06T07:24:17.271616+0000","last_became_active":"2024-03-06T07:24:17.269359+0000","last_became_peered":"2024-03-06T07:24:17.269359+0000","last_unstale":"2024-03-06T07:24:17.271616+0000","last_undegraded":"2024-03-06T07:24:17.271616+0000","last_fullsized":"2024-03-06T07:24:17.271616+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:58.820106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278095+0000","last_change":"2024-03-06T07:24:16.254093+0000","last_active":"2024-03-06T07:24:17.278095+0000","last_peered":"2024-03-06T07:24:17.278095+0000","last_clean":"2024-03-06T07:24:17.278095+0000","last_became_active":"2024-03-06T07:24:16.253786+0000","last_became_peered":"2024-03-06T07:24:16.253786+0000","last_unstale":"2024-03-06T07:24:17.278095+0000","last_undegraded":"2024-03-06T07:24:17.278095+0000","last_fullsized":"2024-03-06T07:24:17.278095+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:56.105200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247276+0000","last_change":"2024-03-06T07:24:03.890715+0000","last_active":"2024-03-06T07:24:17.247276+0000","last_peered":"2024-03-06T07:24:17.247276+0000","last_clean":"2024-03-06T07:24:17.247276+0000","last_became_active":"2024-03-06T07:24:03.890464+0000","last_became_peered":"2024-03-06T07:24:03.890464+0000","last_unstale":"2024-03-06T07:24:17.247276+0000","last_undegraded":"2024-03-06T07:24:17.247276+0000","last_fullsized":"2024-03-06T07:24:17.247276+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261609+0000","last_change":"2024-03-06T07:24:02.881481+0000","last_active":"2024-03-06T07:24:17.261609+0000","last_peered":"2024-03-06T07:24:17.261609+0000","last_clean":"2024-03-06T07:24:17.261609+0000","last_became_active":"2024-03-06T07:24:02.881205+0000","last_became_peered":"2024-03-06T07:24:02.881205+0000","last_unstale":"2024-03-06T07:24:17.261609+0000","last_undegraded":"2024-03-06T07:24:17.261609+0000","last_fullsized":"2024-03-06T07:24:17.261609+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:01.647744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262137+0000","last_change":"2024-03-06T07:24:17.258522+0000","last_active":"2024-03-06T07:24:17.262137+0000","last_peered":"2024-03-06T07:24:17.262137+0000","last_clean":"2024-03-06T07:24:17.262137+0000","last_became_active":"2024-03-06T07:24:17.257291+0000","last_became_peered":"2024-03-06T07:24:17.257291+0000","last_unstale":"2024-03-06T07:24:17.262137+0000","last_undegraded":"2024-03-06T07:24:17.262137+0000","last_fullsized":"2024-03-06T07:24:17.262137+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:53.021295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257950+0000","last_change":"2024-03-06T07:24:16.254272+0000","last_active":"2024-03-06T07:24:17.257950+0000","last_peered":"2024-03-06T07:24:17.257950+0000","last_clean":"2024-03-06T07:24:17.257950+0000","last_became_active":"2024-03-06T07:24:16.254030+0000","last_became_peered":"2024-03-06T07:24:16.254030+0000","last_unstale":"2024-03-06T07:24:17.257950+0000","last_undegraded":"2024-03-06T07:24:17.257950+0000","last_fullsized":"2024-03-06T07:24:17.257950+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:39.799276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262934+0000","last_change":"2024-03-06T07:24:03.892672+0000","last_active":"2024-03-06T07:24:17.262934+0000","last_peered":"2024-03-06T07:24:17.262934+0000","last_clean":"2024-03-06T07:24:17.262934+0000","last_became_active":"2024-03-06T07:24:03.891741+0000","last_became_peered":"2024-03-06T07:24:03.891741+0000","last_unstale":"2024-03-06T07:24:17.262934+0000","last_undegraded":"2024-03-06T07:24:17.262934+0000","last_fullsized":"2024-03-06T07:24:17.262934+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:39.675776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250680+0000","last_change":"2024-03-06T07:24:02.880928+0000","last_active":"2024-03-06T07:24:17.250680+0000","last_peered":"2024-03-06T07:24:17.250680+0000","last_clean":"2024-03-06T07:24:17.250680+0000","last_became_active":"2024-03-06T07:24:02.880661+0000","last_became_peered":"2024-03-06T07:24:02.880661+0000","last_unstale":"2024-03-06T07:24:17.250680+0000","last_undegraded":"2024-03-06T07:24:17.250680+0000","last_fullsized":"2024-03-06T07:24:17.250680+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:01.167170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272120+0000","last_change":"2024-03-06T07:24:17.269178+0000","last_active":"2024-03-06T07:24:17.272120+0000","last_peered":"2024-03-06T07:24:17.272120+0000","last_clean":"2024-03-06T07:24:17.272120+0000","last_became_active":"2024-03-06T07:24:17.267893+0000","last_became_peered":"2024-03-06T07:24:17.267893+0000","last_unstale":"2024-03-06T07:24:17.272120+0000","last_undegraded":"2024-03-06T07:24:17.272120+0000","last_fullsized":"2024-03-06T07:24:17.272120+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:48.828239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"152'3","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277453+0000","last_change":"2024-03-06T07:24:16.250987+0000","last_active":"2024-03-06T07:24:17.277453+0000","last_peered":"2024-03-06T07:24:17.277453+0000","last_clean":"2024-03-06T07:24:17.277453+0000","last_became_active":"2024-03-06T07:24:16.250742+0000","last_became_peered":"2024-03-06T07:24:16.250742+0000","last_unstale":"2024-03-06T07:24:17.277453+0000","last_undegraded":"2024-03-06T07:24:17.277453+0000","last_fullsized":"2024-03-06T07:24:17.277453+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:52.772965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256464+0000","last_change":"2024-03-06T07:24:03.898477+0000","last_active":"2024-03-06T07:24:17.256464+0000","last_peered":"2024-03-06T07:24:17.256464+0000","last_clean":"2024-03-06T07:24:17.256464+0000","last_became_active":"2024-03-06T07:24:03.897814+0000","last_became_peered":"2024-03-06T07:24:03.897814+0000","last_unstale":"2024-03-06T07:24:17.256464+0000","last_undegraded":"2024-03-06T07:24:17.256464+0000","last_fullsized":"2024-03-06T07:24:17.256464+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:51.104075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256172+0000","last_change":"2024-03-06T07:24:02.878433+0000","last_active":"2024-03-06T07:24:17.256172+0000","last_peered":"2024-03-06T07:24:17.256172+0000","last_clean":"2024-03-06T07:24:17.256172+0000","last_became_active":"2024-03-06T07:24:02.878153+0000","last_became_peered":"2024-03-06T07:24:02.878153+0000","last_unstale":"2024-03-06T07:24:17.256172+0000","last_undegraded":"2024-03-06T07:24:17.256172+0000","last_fullsized":"2024-03-06T07:24:17.256172+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:27.611384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.272444+0000","last_change":"2024-03-06T07:24:17.270886+0000","last_active":"2024-03-06T07:24:17.272444+0000","last_peered":"2024-03-06T07:24:17.272444+0000","last_clean":"2024-03-06T07:24:17.272444+0000","last_became_active":"2024-03-06T07:24:17.270361+0000","last_became_peered":"2024-03-06T07:24:17.270361+0000","last_unstale":"2024-03-06T07:24:17.272444+0000","last_undegraded":"2024-03-06T07:24:17.272444+0000","last_fullsized":"2024-03-06T07:24:17.272444+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:47.029222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"152'6","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.283925+0000","last_change":"2024-03-06T07:24:16.250814+0000","last_active":"2024-03-06T07:24:17.283925+0000","last_peered":"2024-03-06T07:24:17.283925+0000","last_clean":"2024-03-06T07:24:17.283925+0000","last_became_active":"2024-03-06T07:24:16.250538+0000","last_became_peered":"2024-03-06T07:24:16.250538+0000","last_unstale":"2024-03-06T07:24:17.283925+0000","last_undegraded":"2024-03-06T07:24:17.283925+0000","last_fullsized":"2024-03-06T07:24:17.283925+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:20.346111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251927+0000","last_change":"2024-03-06T07:24:03.891217+0000","last_active":"2024-03-06T07:24:17.251927+0000","last_peered":"2024-03-06T07:24:17.251927+0000","last_clean":"2024-03-06T07:24:17.251927+0000","last_became_active":"2024-03-06T07:24:03.890968+0000","last_became_peered":"2024-03-06T07:24:03.890968+0000","last_unstale":"2024-03-06T07:24:17.251927+0000","last_undegraded":"2024-03-06T07:24:17.251927+0000","last_fullsized":"2024-03-06T07:24:17.251927+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:26.282549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247739+0000","last_change":"2024-03-06T07:24:02.890689+0000","last_active":"2024-03-06T07:24:17.247739+0000","last_peered":"2024-03-06T07:24:17.247739+0000","last_clean":"2024-03-06T07:24:17.247739+0000","last_became_active":"2024-03-06T07:24:02.890424+0000","last_became_peered":"2024-03-06T07:24:02.890424+0000","last_unstale":"2024-03-06T07:24:17.247739+0000","last_undegraded":"2024-03-06T07:24:17.247739+0000","last_fullsized":"2024-03-06T07:24:17.247739+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.280887+0000","last_change":"2024-03-06T07:24:17.276994+0000","last_active":"2024-03-06T07:24:17.280887+0000","last_peered":"2024-03-06T07:24:17.280887+0000","last_clean":"2024-03-06T07:24:17.280887+0000","last_became_active":"2024-03-06T07:24:17.276715+0000","last_became_peered":"2024-03-06T07:24:17.276715+0000","last_unstale":"2024-03-06T07:24:17.280887+0000","last_undegraded":"2024-03-06T07:24:17.280887+0000","last_fullsized":"2024-03-06T07:24:17.280887+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:32.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251679+0000","last_change":"2024-03-06T07:24:16.253614+0000","last_active":"2024-03-06T07:24:17.251679+0000","last_peered":"2024-03-06T07:24:17.251679+0000","last_clean":"2024-03-06T07:24:17.251679+0000","last_became_active":"2024-03-06T07:24:16.253341+0000","last_became_peered":"2024-03-06T07:24:16.253341+0000","last_unstale":"2024-03-06T07:24:17.251679+0000","last_undegraded":"2024-03-06T07:24:17.251679+0000","last_fullsized":"2024-03-06T07:24:17.251679+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:19.541026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252113+0000","last_change":"2024-03-06T07:24:03.878910+0000","last_active":"2024-03-06T07:24:17.252113+0000","last_peered":"2024-03-06T07:24:17.252113+0000","last_clean":"2024-03-06T07:24:17.252113+0000","last_became_active":"2024-03-06T07:24:03.878660+0000","last_became_peered":"2024-03-06T07:24:03.878660+0000","last_unstale":"2024-03-06T07:24:17.252113+0000","last_undegraded":"2024-03-06T07:24:17.252113+0000","last_fullsized":"2024-03-06T07:24:17.252113+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:52.217554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250528+0000","last_change":"2024-03-06T07:24:02.891744+0000","last_active":"2024-03-06T07:24:17.250528+0000","last_peered":"2024-03-06T07:24:17.250528+0000","last_clean":"2024-03-06T07:24:17.250528+0000","last_became_active":"2024-03-06T07:24:02.891480+0000","last_became_peered":"2024-03-06T07:24:02.891480+0000","last_unstale":"2024-03-06T07:24:17.250528+0000","last_undegraded":"2024-03-06T07:24:17.250528+0000","last_fullsized":"2024-03-06T07:24:17.250528+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:07.998051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262167+0000","last_change":"2024-03-06T07:24:17.256920+0000","last_active":"2024-03-06T07:24:17.262167+0000","last_peered":"2024-03-06T07:24:17.262167+0000","last_clean":"2024-03-06T07:24:17.262167+0000","last_became_active":"2024-03-06T07:24:17.256642+0000","last_became_peered":"2024-03-06T07:24:17.256642+0000","last_unstale":"2024-03-06T07:24:17.262167+0000","last_undegraded":"2024-03-06T07:24:17.262167+0000","last_fullsized":"2024-03-06T07:24:17.262167+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:03.211317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257581+0000","last_change":"2024-03-06T07:24:16.257197+0000","last_active":"2024-03-06T07:24:17.257581+0000","last_peered":"2024-03-06T07:24:17.257581+0000","last_clean":"2024-03-06T07:24:17.257581+0000","last_became_active":"2024-03-06T07:24:16.256944+0000","last_became_peered":"2024-03-06T07:24:16.256944+0000","last_unstale":"2024-03-06T07:24:17.257581+0000","last_undegraded":"2024-03-06T07:24:17.257581+0000","last_fullsized":"2024-03-06T07:24:17.257581+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:12.911145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251861+0000","last_change":"2024-03-06T07:24:03.885245+0000","last_active":"2024-03-06T07:24:17.251861+0000","last_peered":"2024-03-06T07:24:17.251861+0000","last_clean":"2024-03-06T07:24:17.251861+0000","last_became_active":"2024-03-06T07:24:03.884959+0000","last_became_peered":"2024-03-06T07:24:03.884959+0000","last_unstale":"2024-03-06T07:24:17.251861+0000","last_undegraded":"2024-03-06T07:24:17.251861+0000","last_fullsized":"2024-03-06T07:24:17.251861+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:44.253526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256036+0000","last_change":"2024-03-06T07:24:02.886573+0000","last_active":"2024-03-06T07:24:17.256036+0000","last_peered":"2024-03-06T07:24:17.256036+0000","last_clean":"2024-03-06T07:24:17.256036+0000","last_became_active":"2024-03-06T07:24:02.885928+0000","last_became_peered":"2024-03-06T07:24:02.885928+0000","last_unstale":"2024-03-06T07:24:17.256036+0000","last_undegraded":"2024-03-06T07:24:17.256036+0000","last_fullsized":"2024-03-06T07:24:17.256036+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271873+0000","last_change":"2024-03-06T07:24:17.265532+0000","last_active":"2024-03-06T07:24:17.271873+0000","last_peered":"2024-03-06T07:24:17.271873+0000","last_clean":"2024-03-06T07:24:17.271873+0000","last_became_active":"2024-03-06T07:24:17.265093+0000","last_became_peered":"2024-03-06T07:24:17.265093+0000","last_unstale":"2024-03-06T07:24:17.271873+0000","last_undegraded":"2024-03-06T07:24:17.271873+0000","last_fullsized":"2024-03-06T07:24:17.271873+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:58.896993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253042+0000","last_change":"2024-03-06T07:24:16.250130+0000","last_active":"2024-03-06T07:24:17.253042+0000","last_peered":"2024-03-06T07:24:17.253042+0000","last_clean":"2024-03-06T07:24:17.253042+0000","last_became_active":"2024-03-06T07:24:16.249680+0000","last_became_peered":"2024-03-06T07:24:16.249680+0000","last_unstale":"2024-03-06T07:24:17.253042+0000","last_undegraded":"2024-03-06T07:24:17.253042+0000","last_fullsized":"2024-03-06T07:24:17.253042+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:28.200391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257738+0000","last_change":"2024-03-06T07:24:03.892190+0000","last_active":"2024-03-06T07:24:17.257738+0000","last_peered":"2024-03-06T07:24:17.257738+0000","last_clean":"2024-03-06T07:24:17.257738+0000","last_became_active":"2024-03-06T07:24:03.891822+0000","last_became_peered":"2024-03-06T07:24:03.891822+0000","last_unstale":"2024-03-06T07:24:17.257738+0000","last_undegraded":"2024-03-06T07:24:17.257738+0000","last_fullsized":"2024-03-06T07:24:17.257738+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251798+0000","last_change":"2024-03-06T07:24:02.892342+0000","last_active":"2024-03-06T07:24:17.251798+0000","last_peered":"2024-03-06T07:24:17.251798+0000","last_clean":"2024-03-06T07:24:17.251798+0000","last_became_active":"2024-03-06T07:24:02.892089+0000","last_became_peered":"2024-03-06T07:24:02.892089+0000","last_unstale":"2024-03-06T07:24:17.251798+0000","last_undegraded":"2024-03-06T07:24:17.251798+0000","last_fullsized":"2024-03-06T07:24:17.251798+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:36.174855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270328+0000","last_change":"2024-03-06T07:24:17.266037+0000","last_active":"2024-03-06T07:24:17.270328+0000","last_peered":"2024-03-06T07:24:17.270328+0000","last_clean":"2024-03-06T07:24:17.270328+0000","last_became_active":"2024-03-06T07:24:17.265742+0000","last_became_peered":"2024-03-06T07:24:17.265742+0000","last_unstale":"2024-03-06T07:24:17.270328+0000","last_undegraded":"2024-03-06T07:24:17.270328+0000","last_fullsized":"2024-03-06T07:24:17.270328+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:19.832573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.281162+0000","last_change":"2024-03-06T07:24:16.256325+0000","last_active":"2024-03-06T07:24:17.281162+0000","last_peered":"2024-03-06T07:24:17.281162+0000","last_clean":"2024-03-06T07:24:17.281162+0000","last_became_active":"2024-03-06T07:24:16.255909+0000","last_became_peered":"2024-03-06T07:24:16.255909+0000","last_unstale":"2024-03-06T07:24:17.281162+0000","last_undegraded":"2024-03-06T07:24:17.281162+0000","last_fullsized":"2024-03-06T07:24:17.281162+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:41.994723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251854+0000","last_change":"2024-03-06T07:24:03.892347+0000","last_active":"2024-03-06T07:24:17.251854+0000","last_peered":"2024-03-06T07:24:17.251854+0000","last_clean":"2024-03-06T07:24:17.251854+0000","last_became_active":"2024-03-06T07:24:03.892074+0000","last_became_peered":"2024-03-06T07:24:03.892074+0000","last_unstale":"2024-03-06T07:24:17.251854+0000","last_undegraded":"2024-03-06T07:24:17.251854+0000","last_fullsized":"2024-03-06T07:24:17.251854+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:20.749704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"148'3","reported_seq":23,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255891+0000","last_change":"2024-03-06T07:24:02.886419+0000","last_active":"2024-03-06T07:24:17.255891+0000","last_peered":"2024-03-06T07:24:17.255891+0000","last_clean":"2024-03-06T07:24:17.255891+0000","last_became_active":"2024-03-06T07:24:02.886064+0000","last_became_peered":"2024-03-06T07:24:02.886064+0000","last_unstale":"2024-03-06T07:24:17.255891+0000","last_undegraded":"2024-03-06T07:24:17.255891+0000","last_fullsized":"2024-03-06T07:24:17.255891+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:29.962503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271712+0000","last_change":"2024-03-06T07:24:17.264408+0000","last_active":"2024-03-06T07:24:17.271712+0000","last_peered":"2024-03-06T07:24:17.271712+0000","last_clean":"2024-03-06T07:24:17.271712+0000","last_became_active":"2024-03-06T07:24:17.263510+0000","last_became_peered":"2024-03-06T07:24:17.263510+0000","last_unstale":"2024-03-06T07:24:17.271712+0000","last_undegraded":"2024-03-06T07:24:17.271712+0000","last_fullsized":"2024-03-06T07:24:17.271712+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.777544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277491+0000","last_change":"2024-03-06T07:24:16.252774+0000","last_active":"2024-03-06T07:24:17.277491+0000","last_peered":"2024-03-06T07:24:17.277491+0000","last_clean":"2024-03-06T07:24:17.277491+0000","last_became_active":"2024-03-06T07:24:16.252533+0000","last_became_peered":"2024-03-06T07:24:16.252533+0000","last_unstale":"2024-03-06T07:24:17.277491+0000","last_undegraded":"2024-03-06T07:24:17.277491+0000","last_fullsized":"2024-03-06T07:24:17.277491+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:19.586046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.249760+0000","last_change":"2024-03-06T07:24:03.886038+0000","last_active":"2024-03-06T07:24:17.249760+0000","last_peered":"2024-03-06T07:24:17.249760+0000","last_clean":"2024-03-06T07:24:17.249760+0000","last_became_active":"2024-03-06T07:24:03.885779+0000","last_became_peered":"2024-03-06T07:24:03.885779+0000","last_unstale":"2024-03-06T07:24:17.249760+0000","last_undegraded":"2024-03-06T07:24:17.249760+0000","last_fullsized":"2024-03-06T07:24:17.249760+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:01.188435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"148'6","reported_seq":26,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250914+0000","last_change":"2024-03-06T07:24:02.889795+0000","last_active":"2024-03-06T07:24:17.250914+0000","last_peered":"2024-03-06T07:24:17.250914+0000","last_clean":"2024-03-06T07:24:17.250914+0000","last_became_active":"2024-03-06T07:24:02.889534+0000","last_became_peered":"2024-03-06T07:24:02.889534+0000","last_unstale":"2024-03-06T07:24:17.250914+0000","last_undegraded":"2024-03-06T07:24:17.250914+0000","last_fullsized":"2024-03-06T07:24:17.250914+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:49:26.432122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.274796+0000","last_change":"2024-03-06T07:24:17.270359+0000","last_active":"2024-03-06T07:24:17.274796+0000","last_peered":"2024-03-06T07:24:17.274796+0000","last_clean":"2024-03-06T07:24:17.274796+0000","last_became_active":"2024-03-06T07:24:17.269845+0000","last_became_peered":"2024-03-06T07:24:17.269845+0000","last_unstale":"2024-03-06T07:24:17.274796+0000","last_undegraded":"2024-03-06T07:24:17.274796+0000","last_fullsized":"2024-03-06T07:24:17.274796+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:31:42.029712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.280998+0000","last_change":"2024-03-06T07:24:16.251053+0000","last_active":"2024-03-06T07:24:17.280998+0000","last_peered":"2024-03-06T07:24:17.280998+0000","last_clean":"2024-03-06T07:24:17.280998+0000","last_became_active":"2024-03-06T07:24:16.250601+0000","last_became_peered":"2024-03-06T07:24:16.250601+0000","last_unstale":"2024-03-06T07:24:17.280998+0000","last_undegraded":"2024-03-06T07:24:17.280998+0000","last_fullsized":"2024-03-06T07:24:17.280998+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:44.572787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251088+0000","last_change":"2024-03-06T07:24:03.883274+0000","last_active":"2024-03-06T07:24:17.251088+0000","last_peered":"2024-03-06T07:24:17.251088+0000","last_clean":"2024-03-06T07:24:17.251088+0000","last_became_active":"2024-03-06T07:24:03.882970+0000","last_became_peered":"2024-03-06T07:24:03.882970+0000","last_unstale":"2024-03-06T07:24:17.251088+0000","last_undegraded":"2024-03-06T07:24:17.251088+0000","last_fullsized":"2024-03-06T07:24:17.251088+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:34.606194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247467+0000","last_change":"2024-03-06T07:24:02.886772+0000","last_active":"2024-03-06T07:24:17.247467+0000","last_peered":"2024-03-06T07:24:17.247467+0000","last_clean":"2024-03-06T07:24:17.247467+0000","last_became_active":"2024-03-06T07:24:02.886444+0000","last_became_peered":"2024-03-06T07:24:02.886444+0000","last_unstale":"2024-03-06T07:24:17.247467+0000","last_undegraded":"2024-03-06T07:24:17.247467+0000","last_fullsized":"2024-03-06T07:24:17.247467+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:14.833238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271526+0000","last_change":"2024-03-06T07:24:17.267411+0000","last_active":"2024-03-06T07:24:17.271526+0000","last_peered":"2024-03-06T07:24:17.271526+0000","last_clean":"2024-03-06T07:24:17.271526+0000","last_became_active":"2024-03-06T07:24:17.267128+0000","last_became_peered":"2024-03-06T07:24:17.267128+0000","last_unstale":"2024-03-06T07:24:17.271526+0000","last_undegraded":"2024-03-06T07:24:17.271526+0000","last_fullsized":"2024-03-06T07:24:17.271526+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:16.019477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277966+0000","last_change":"2024-03-06T07:24:16.255311+0000","last_active":"2024-03-06T07:24:17.277966+0000","last_peered":"2024-03-06T07:24:17.277966+0000","last_clean":"2024-03-06T07:24:17.277966+0000","last_became_active":"2024-03-06T07:24:16.254966+0000","last_became_peered":"2024-03-06T07:24:16.254966+0000","last_unstale":"2024-03-06T07:24:17.277966+0000","last_undegraded":"2024-03-06T07:24:17.277966+0000","last_fullsized":"2024-03-06T07:24:17.277966+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:51.647928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256068+0000","last_change":"2024-03-06T07:24:03.899462+0000","last_active":"2024-03-06T07:24:17.256068+0000","last_peered":"2024-03-06T07:24:17.256068+0000","last_clean":"2024-03-06T07:24:17.256068+0000","last_became_active":"2024-03-06T07:24:03.899154+0000","last_became_peered":"2024-03-06T07:24:03.899154+0000","last_unstale":"2024-03-06T07:24:17.256068+0000","last_undegraded":"2024-03-06T07:24:17.256068+0000","last_fullsized":"2024-03-06T07:24:17.256068+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:05.985567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252064+0000","last_change":"2024-03-06T07:24:02.886145+0000","last_active":"2024-03-06T07:24:17.252064+0000","last_peered":"2024-03-06T07:24:17.252064+0000","last_clean":"2024-03-06T07:24:17.252064+0000","last_became_active":"2024-03-06T07:24:02.885886+0000","last_became_peered":"2024-03-06T07:24:02.885886+0000","last_unstale":"2024-03-06T07:24:17.252064+0000","last_undegraded":"2024-03-06T07:24:17.252064+0000","last_fullsized":"2024-03-06T07:24:17.252064+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:43.257419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270982+0000","last_change":"2024-03-06T07:24:17.265778+0000","last_active":"2024-03-06T07:24:17.270982+0000","last_peered":"2024-03-06T07:24:17.270982+0000","last_clean":"2024-03-06T07:24:17.270982+0000","last_became_active":"2024-03-06T07:24:17.265517+0000","last_became_peered":"2024-03-06T07:24:17.265517+0000","last_unstale":"2024-03-06T07:24:17.270982+0000","last_undegraded":"2024-03-06T07:24:17.270982+0000","last_fullsized":"2024-03-06T07:24:17.270982+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.777544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"152'4","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275215+0000","last_change":"2024-03-06T07:24:16.242118+0000","last_active":"2024-03-06T07:24:17.275215+0000","last_peered":"2024-03-06T07:24:17.275215+0000","last_clean":"2024-03-06T07:24:17.275215+0000","last_became_active":"2024-03-06T07:24:16.241847+0000","last_became_peered":"2024-03-06T07:24:16.241847+0000","last_unstale":"2024-03-06T07:24:17.275215+0000","last_undegraded":"2024-03-06T07:24:17.275215+0000","last_fullsized":"2024-03-06T07:24:17.275215+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:57.892414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263494+0000","last_change":"2024-03-06T07:24:03.883971+0000","last_active":"2024-03-06T07:24:17.263494+0000","last_peered":"2024-03-06T07:24:17.263494+0000","last_clean":"2024-03-06T07:24:17.263494+0000","last_became_active":"2024-03-06T07:24:03.883705+0000","last_became_peered":"2024-03-06T07:24:03.883705+0000","last_unstale":"2024-03-06T07:24:17.263494+0000","last_undegraded":"2024-03-06T07:24:17.263494+0000","last_fullsized":"2024-03-06T07:24:17.263494+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:34.136559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258878+0000","last_change":"2024-03-06T07:24:02.883880+0000","last_active":"2024-03-06T07:24:17.258878+0000","last_peered":"2024-03-06T07:24:17.258878+0000","last_clean":"2024-03-06T07:24:17.258878+0000","last_became_active":"2024-03-06T07:24:02.883488+0000","last_became_peered":"2024-03-06T07:24:02.883488+0000","last_unstale":"2024-03-06T07:24:17.258878+0000","last_undegraded":"2024-03-06T07:24:17.258878+0000","last_fullsized":"2024-03-06T07:24:17.258878+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:31.545966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271679+0000","last_change":"2024-03-06T07:24:17.270301+0000","last_active":"2024-03-06T07:24:17.271679+0000","last_peered":"2024-03-06T07:24:17.271679+0000","last_clean":"2024-03-06T07:24:17.271679+0000","last_became_active":"2024-03-06T07:24:17.270059+0000","last_became_peered":"2024-03-06T07:24:17.270059+0000","last_unstale":"2024-03-06T07:24:17.271679+0000","last_undegraded":"2024-03-06T07:24:17.271679+0000","last_fullsized":"2024-03-06T07:24:17.271679+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:27.251415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276359+0000","last_change":"2024-03-06T07:24:16.256150+0000","last_active":"2024-03-06T07:24:17.276359+0000","last_peered":"2024-03-06T07:24:17.276359+0000","last_clean":"2024-03-06T07:24:17.276359+0000","last_became_active":"2024-03-06T07:24:16.255892+0000","last_became_peered":"2024-03-06T07:24:16.255892+0000","last_unstale":"2024-03-06T07:24:17.276359+0000","last_undegraded":"2024-03-06T07:24:17.276359+0000","last_fullsized":"2024-03-06T07:24:17.276359+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:25.087329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252485+0000","last_change":"2024-03-06T07:24:03.883761+0000","last_active":"2024-03-06T07:24:17.252485+0000","last_peered":"2024-03-06T07:24:17.252485+0000","last_clean":"2024-03-06T07:24:17.252485+0000","last_became_active":"2024-03-06T07:24:03.882991+0000","last_became_peered":"2024-03-06T07:24:03.882991+0000","last_unstale":"2024-03-06T07:24:17.252485+0000","last_undegraded":"2024-03-06T07:24:17.252485+0000","last_fullsized":"2024-03-06T07:24:17.252485+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:28.659458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"148'8","reported_seq":28,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256119+0000","last_change":"2024-03-06T07:24:02.885178+0000","last_active":"2024-03-06T07:24:17.256119+0000","last_peered":"2024-03-06T07:24:17.256119+0000","last_clean":"2024-03-06T07:24:17.256119+0000","last_became_active":"2024-03-06T07:24:02.880268+0000","last_became_peered":"2024-03-06T07:24:02.880268+0000","last_unstale":"2024-03-06T07:24:17.256119+0000","last_undegraded":"2024-03-06T07:24:17.256119+0000","last_fullsized":"2024-03-06T07:24:17.256119+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:14.833238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271854+0000","last_change":"2024-03-06T07:24:17.265332+0000","last_active":"2024-03-06T07:24:17.271854+0000","last_peered":"2024-03-06T07:24:17.271854+0000","last_clean":"2024-03-06T07:24:17.271854+0000","last_became_active":"2024-03-06T07:24:17.264768+0000","last_became_peered":"2024-03-06T07:24:17.264768+0000","last_unstale":"2024-03-06T07:24:17.271854+0000","last_undegraded":"2024-03-06T07:24:17.271854+0000","last_fullsized":"2024-03-06T07:24:17.271854+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:57.629152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250348+0000","last_change":"2024-03-06T07:24:16.249194+0000","last_active":"2024-03-06T07:24:17.250348+0000","last_peered":"2024-03-06T07:24:17.250348+0000","last_clean":"2024-03-06T07:24:17.250348+0000","last_became_active":"2024-03-06T07:24:16.248943+0000","last_became_peered":"2024-03-06T07:24:16.248943+0000","last_unstale":"2024-03-06T07:24:17.250348+0000","last_undegraded":"2024-03-06T07:24:17.250348+0000","last_fullsized":"2024-03-06T07:24:17.250348+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:51.728831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255948+0000","last_change":"2024-03-06T07:24:03.888753+0000","last_active":"2024-03-06T07:24:17.255948+0000","last_peered":"2024-03-06T07:24:17.255948+0000","last_clean":"2024-03-06T07:24:17.255948+0000","last_became_active":"2024-03-06T07:24:03.888491+0000","last_became_peered":"2024-03-06T07:24:03.888491+0000","last_unstale":"2024-03-06T07:24:17.255948+0000","last_undegraded":"2024-03-06T07:24:17.255948+0000","last_fullsized":"2024-03-06T07:24:17.255948+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251982+0000","last_change":"2024-03-06T07:24:02.886460+0000","last_active":"2024-03-06T07:24:17.251982+0000","last_peered":"2024-03-06T07:24:17.251982+0000","last_clean":"2024-03-06T07:24:17.251982+0000","last_became_active":"2024-03-06T07:24:02.885733+0000","last_became_peered":"2024-03-06T07:24:02.885733+0000","last_unstale":"2024-03-06T07:24:17.251982+0000","last_undegraded":"2024-03-06T07:24:17.251982+0000","last_fullsized":"2024-03-06T07:24:17.251982+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:59.543991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278436+0000","last_change":"2024-03-06T07:24:17.275819+0000","last_active":"2024-03-06T07:24:17.278436+0000","last_peered":"2024-03-06T07:24:17.278436+0000","last_clean":"2024-03-06T07:24:17.278436+0000","last_became_active":"2024-03-06T07:24:17.275562+0000","last_became_peered":"2024-03-06T07:24:17.275562+0000","last_unstale":"2024-03-06T07:24:17.278436+0000","last_undegraded":"2024-03-06T07:24:17.278436+0000","last_fullsized":"2024-03-06T07:24:17.278436+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:11.169809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251745+0000","last_change":"2024-03-06T07:24:16.245437+0000","last_active":"2024-03-06T07:24:17.251745+0000","last_peered":"2024-03-06T07:24:17.251745+0000","last_clean":"2024-03-06T07:24:17.251745+0000","last_became_active":"2024-03-06T07:24:16.245151+0000","last_became_peered":"2024-03-06T07:24:16.245151+0000","last_unstale":"2024-03-06T07:24:17.251745+0000","last_undegraded":"2024-03-06T07:24:17.251745+0000","last_fullsized":"2024-03-06T07:24:17.251745+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:56.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252006+0000","last_change":"2024-03-06T07:24:03.889875+0000","last_active":"2024-03-06T07:24:17.252006+0000","last_peered":"2024-03-06T07:24:17.252006+0000","last_clean":"2024-03-06T07:24:17.252006+0000","last_became_active":"2024-03-06T07:24:03.889040+0000","last_became_peered":"2024-03-06T07:24:03.889040+0000","last_unstale":"2024-03-06T07:24:17.252006+0000","last_undegraded":"2024-03-06T07:24:17.252006+0000","last_fullsized":"2024-03-06T07:24:17.252006+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:57.658966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257692+0000","last_change":"2024-03-06T07:24:02.888816+0000","last_active":"2024-03-06T07:24:17.257692+0000","last_peered":"2024-03-06T07:24:17.257692+0000","last_clean":"2024-03-06T07:24:17.257692+0000","last_became_active":"2024-03-06T07:24:02.888090+0000","last_became_peered":"2024-03-06T07:24:02.888090+0000","last_unstale":"2024-03-06T07:24:17.257692+0000","last_undegraded":"2024-03-06T07:24:17.257692+0000","last_fullsized":"2024-03-06T07:24:17.257692+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:42.738046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271418+0000","last_change":"2024-03-06T07:24:17.265980+0000","last_active":"2024-03-06T07:24:17.271418+0000","last_peered":"2024-03-06T07:24:17.271418+0000","last_clean":"2024-03-06T07:24:17.271418+0000","last_became_active":"2024-03-06T07:24:17.260869+0000","last_became_peered":"2024-03-06T07:24:17.260869+0000","last_unstale":"2024-03-06T07:24:17.271418+0000","last_undegraded":"2024-03-06T07:24:17.271418+0000","last_fullsized":"2024-03-06T07:24:17.271418+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:09.278978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262672+0000","last_change":"2024-03-06T07:24:16.252214+0000","last_active":"2024-03-06T07:24:17.262672+0000","last_peered":"2024-03-06T07:24:17.262672+0000","last_clean":"2024-03-06T07:24:17.262672+0000","last_became_active":"2024-03-06T07:24:16.251529+0000","last_became_peered":"2024-03-06T07:24:16.251529+0000","last_unstale":"2024-03-06T07:24:17.262672+0000","last_undegraded":"2024-03-06T07:24:17.262672+0000","last_fullsized":"2024-03-06T07:24:17.262672+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:19.586046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257436+0000","last_change":"2024-03-06T07:24:03.890038+0000","last_active":"2024-03-06T07:24:17.257436+0000","last_peered":"2024-03-06T07:24:17.257436+0000","last_clean":"2024-03-06T07:24:17.257436+0000","last_became_active":"2024-03-06T07:24:03.889784+0000","last_became_peered":"2024-03-06T07:24:03.889784+0000","last_unstale":"2024-03-06T07:24:17.257436+0000","last_undegraded":"2024-03-06T07:24:17.257436+0000","last_fullsized":"2024-03-06T07:24:17.257436+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:35.461921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"148'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247718+0000","last_change":"2024-03-06T07:24:02.887549+0000","last_active":"2024-03-06T07:24:17.247718+0000","last_peered":"2024-03-06T07:24:17.247718+0000","last_clean":"2024-03-06T07:24:17.247718+0000","last_became_active":"2024-03-06T07:24:02.887264+0000","last_became_peered":"2024-03-06T07:24:02.887264+0000","last_unstale":"2024-03-06T07:24:17.247718+0000","last_undegraded":"2024-03-06T07:24:17.247718+0000","last_fullsized":"2024-03-06T07:24:17.247718+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:15.400205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.267075+0000","last_change":"2024-03-06T07:24:17.264034+0000","last_active":"2024-03-06T07:24:17.267075+0000","last_peered":"2024-03-06T07:24:17.267075+0000","last_clean":"2024-03-06T07:24:17.267075+0000","last_became_active":"2024-03-06T07:24:17.263770+0000","last_became_peered":"2024-03-06T07:24:17.263770+0000","last_unstale":"2024-03-06T07:24:17.267075+0000","last_undegraded":"2024-03-06T07:24:17.267075+0000","last_fullsized":"2024-03-06T07:24:17.267075+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:01.513189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257440+0000","last_change":"2024-03-06T07:24:16.251808+0000","last_active":"2024-03-06T07:24:17.257440+0000","last_peered":"2024-03-06T07:24:17.257440+0000","last_clean":"2024-03-06T07:24:17.257440+0000","last_became_active":"2024-03-06T07:24:16.251527+0000","last_became_peered":"2024-03-06T07:24:16.251527+0000","last_unstale":"2024-03-06T07:24:17.257440+0000","last_undegraded":"2024-03-06T07:24:17.257440+0000","last_fullsized":"2024-03-06T07:24:17.257440+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:24.856326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.261087+0000","last_change":"2024-03-06T07:24:03.897943+0000","last_active":"2024-03-06T07:24:17.261087+0000","last_peered":"2024-03-06T07:24:17.261087+0000","last_clean":"2024-03-06T07:24:17.261087+0000","last_became_active":"2024-03-06T07:24:03.897665+0000","last_became_peered":"2024-03-06T07:24:03.897665+0000","last_unstale":"2024-03-06T07:24:17.261087+0000","last_undegraded":"2024-03-06T07:24:17.261087+0000","last_fullsized":"2024-03-06T07:24:17.261087+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:36.087346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256551+0000","last_change":"2024-03-06T07:24:02.881968+0000","last_active":"2024-03-06T07:24:17.256551+0000","last_peered":"2024-03-06T07:24:17.256551+0000","last_clean":"2024-03-06T07:24:17.256551+0000","last_became_active":"2024-03-06T07:24:02.881702+0000","last_became_peered":"2024-03-06T07:24:02.881702+0000","last_unstale":"2024-03-06T07:24:17.256551+0000","last_undegraded":"2024-03-06T07:24:17.256551+0000","last_fullsized":"2024-03-06T07:24:17.256551+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:37.173846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.275615+0000","last_change":"2024-03-06T07:24:17.268289+0000","last_active":"2024-03-06T07:24:17.275615+0000","last_peered":"2024-03-06T07:24:17.275615+0000","last_clean":"2024-03-06T07:24:17.275615+0000","last_became_active":"2024-03-06T07:24:17.266770+0000","last_became_peered":"2024-03-06T07:24:17.266770+0000","last_unstale":"2024-03-06T07:24:17.275615+0000","last_undegraded":"2024-03-06T07:24:17.275615+0000","last_fullsized":"2024-03-06T07:24:17.275615+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:06.739402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250922+0000","last_change":"2024-03-06T07:24:16.256703+0000","last_active":"2024-03-06T07:24:17.250922+0000","last_peered":"2024-03-06T07:24:17.250922+0000","last_clean":"2024-03-06T07:24:17.250922+0000","last_became_active":"2024-03-06T07:24:16.256327+0000","last_became_peered":"2024-03-06T07:24:16.256327+0000","last_unstale":"2024-03-06T07:24:17.250922+0000","last_undegraded":"2024-03-06T07:24:17.250922+0000","last_fullsized":"2024-03-06T07:24:17.250922+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:34.579168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250844+0000","last_change":"2024-03-06T07:24:03.894250+0000","last_active":"2024-03-06T07:24:17.250844+0000","last_peered":"2024-03-06T07:24:17.250844+0000","last_clean":"2024-03-06T07:24:17.250844+0000","last_became_active":"2024-03-06T07:24:03.893672+0000","last_became_peered":"2024-03-06T07:24:03.893672+0000","last_unstale":"2024-03-06T07:24:17.250844+0000","last_undegraded":"2024-03-06T07:24:17.250844+0000","last_fullsized":"2024-03-06T07:24:17.250844+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:55.684312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247702+0000","last_change":"2024-03-06T07:24:02.883228+0000","last_active":"2024-03-06T07:24:17.247702+0000","last_peered":"2024-03-06T07:24:17.247702+0000","last_clean":"2024-03-06T07:24:17.247702+0000","last_became_active":"2024-03-06T07:24:02.882971+0000","last_became_peered":"2024-03-06T07:24:02.882971+0000","last_unstale":"2024-03-06T07:24:17.247702+0000","last_undegraded":"2024-03-06T07:24:17.247702+0000","last_fullsized":"2024-03-06T07:24:17.247702+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:42.738046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.270653+0000","last_change":"2024-03-06T07:24:17.268650+0000","last_active":"2024-03-06T07:24:17.270653+0000","last_peered":"2024-03-06T07:24:17.270653+0000","last_clean":"2024-03-06T07:24:17.270653+0000","last_became_active":"2024-03-06T07:24:17.268386+0000","last_became_peered":"2024-03-06T07:24:17.268386+0000","last_unstale":"2024-03-06T07:24:17.270653+0000","last_undegraded":"2024-03-06T07:24:17.270653+0000","last_fullsized":"2024-03-06T07:24:17.270653+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:25.693947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250861+0000","last_change":"2024-03-06T07:24:16.252130+0000","last_active":"2024-03-06T07:24:17.250861+0000","last_peered":"2024-03-06T07:24:17.250861+0000","last_clean":"2024-03-06T07:24:17.250861+0000","last_became_active":"2024-03-06T07:24:16.251812+0000","last_became_peered":"2024-03-06T07:24:16.251812+0000","last_unstale":"2024-03-06T07:24:17.250861+0000","last_undegraded":"2024-03-06T07:24:17.250861+0000","last_fullsized":"2024-03-06T07:24:17.250861+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:08.639042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247625+0000","last_change":"2024-03-06T07:24:03.889481+0000","last_active":"2024-03-06T07:24:17.247625+0000","last_peered":"2024-03-06T07:24:17.247625+0000","last_clean":"2024-03-06T07:24:17.247625+0000","last_became_active":"2024-03-06T07:24:03.889258+0000","last_became_peered":"2024-03-06T07:24:03.889258+0000","last_unstale":"2024-03-06T07:24:17.247625+0000","last_undegraded":"2024-03-06T07:24:17.247625+0000","last_fullsized":"2024-03-06T07:24:17.247625+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.629710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257343+0000","last_change":"2024-03-06T07:24:02.889856+0000","last_active":"2024-03-06T07:24:17.257343+0000","last_peered":"2024-03-06T07:24:17.257343+0000","last_clean":"2024-03-06T07:24:17.257343+0000","last_became_active":"2024-03-06T07:24:02.889589+0000","last_became_peered":"2024-03-06T07:24:02.889589+0000","last_unstale":"2024-03-06T07:24:17.257343+0000","last_undegraded":"2024-03-06T07:24:17.257343+0000","last_fullsized":"2024-03-06T07:24:17.257343+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:09.042646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262891+0000","last_change":"2024-03-06T07:24:17.262167+0000","last_active":"2024-03-06T07:24:17.262891+0000","last_peered":"2024-03-06T07:24:17.262891+0000","last_clean":"2024-03-06T07:24:17.262891+0000","last_became_active":"2024-03-06T07:24:17.261902+0000","last_became_peered":"2024-03-06T07:24:17.261902+0000","last_unstale":"2024-03-06T07:24:17.262891+0000","last_undegraded":"2024-03-06T07:24:17.262891+0000","last_fullsized":"2024-03-06T07:24:17.262891+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:11.081631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.276725+0000","last_change":"2024-03-06T07:24:16.250683+0000","last_active":"2024-03-06T07:24:17.276725+0000","last_peered":"2024-03-06T07:24:17.276725+0000","last_clean":"2024-03-06T07:24:17.276725+0000","last_became_active":"2024-03-06T07:24:16.250440+0000","last_became_peered":"2024-03-06T07:24:16.250440+0000","last_unstale":"2024-03-06T07:24:17.276725+0000","last_undegraded":"2024-03-06T07:24:17.276725+0000","last_fullsized":"2024-03-06T07:24:17.276725+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:42.156089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247958+0000","last_change":"2024-03-06T07:24:03.890643+0000","last_active":"2024-03-06T07:24:17.247958+0000","last_peered":"2024-03-06T07:24:17.247958+0000","last_clean":"2024-03-06T07:24:17.247958+0000","last_became_active":"2024-03-06T07:24:03.890406+0000","last_became_peered":"2024-03-06T07:24:03.890406+0000","last_unstale":"2024-03-06T07:24:17.247958+0000","last_undegraded":"2024-03-06T07:24:17.247958+0000","last_fullsized":"2024-03-06T07:24:17.247958+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:18.050746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250657+0000","last_change":"2024-03-06T07:24:02.882070+0000","last_active":"2024-03-06T07:24:17.250657+0000","last_peered":"2024-03-06T07:24:17.250657+0000","last_clean":"2024-03-06T07:24:17.250657+0000","last_became_active":"2024-03-06T07:24:02.881582+0000","last_became_peered":"2024-03-06T07:24:02.881582+0000","last_unstale":"2024-03-06T07:24:17.250657+0000","last_undegraded":"2024-03-06T07:24:17.250657+0000","last_fullsized":"2024-03-06T07:24:17.250657+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:37.816763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263777+0000","last_change":"2024-03-06T07:24:17.258001+0000","last_active":"2024-03-06T07:24:17.263777+0000","last_peered":"2024-03-06T07:24:17.263777+0000","last_clean":"2024-03-06T07:24:17.263777+0000","last_became_active":"2024-03-06T07:24:17.257764+0000","last_became_peered":"2024-03-06T07:24:17.257764+0000","last_unstale":"2024-03-06T07:24:17.263777+0000","last_undegraded":"2024-03-06T07:24:17.263777+0000","last_fullsized":"2024-03-06T07:24:17.263777+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:25.860905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.249807+0000","last_change":"2024-03-06T07:24:16.251351+0000","last_active":"2024-03-06T07:24:17.249807+0000","last_peered":"2024-03-06T07:24:17.249807+0000","last_clean":"2024-03-06T07:24:17.249807+0000","last_became_active":"2024-03-06T07:24:16.251106+0000","last_became_peered":"2024-03-06T07:24:16.251106+0000","last_unstale":"2024-03-06T07:24:17.249807+0000","last_undegraded":"2024-03-06T07:24:17.249807+0000","last_fullsized":"2024-03-06T07:24:17.249807+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:07.372422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263446+0000","last_change":"2024-03-06T07:24:03.890706+0000","last_active":"2024-03-06T07:24:17.263446+0000","last_peered":"2024-03-06T07:24:17.263446+0000","last_clean":"2024-03-06T07:24:17.263446+0000","last_became_active":"2024-03-06T07:24:03.890369+0000","last_became_peered":"2024-03-06T07:24:03.890369+0000","last_unstale":"2024-03-06T07:24:17.263446+0000","last_undegraded":"2024-03-06T07:24:17.263446+0000","last_fullsized":"2024-03-06T07:24:17.263446+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:35.615657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258420+0000","last_change":"2024-03-06T07:24:02.885639+0000","last_active":"2024-03-06T07:24:17.258420+0000","last_peered":"2024-03-06T07:24:17.258420+0000","last_clean":"2024-03-06T07:24:17.258420+0000","last_became_active":"2024-03-06T07:24:02.885357+0000","last_became_peered":"2024-03-06T07:24:02.885357+0000","last_unstale":"2024-03-06T07:24:17.258420+0000","last_undegraded":"2024-03-06T07:24:17.258420+0000","last_fullsized":"2024-03-06T07:24:17.258420+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:08.534409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.265219+0000","last_change":"2024-03-06T07:24:17.264192+0000","last_active":"2024-03-06T07:24:17.265219+0000","last_peered":"2024-03-06T07:24:17.265219+0000","last_clean":"2024-03-06T07:24:17.265219+0000","last_became_active":"2024-03-06T07:24:17.263885+0000","last_became_peered":"2024-03-06T07:24:17.263885+0000","last_unstale":"2024-03-06T07:24:17.265219+0000","last_undegraded":"2024-03-06T07:24:17.265219+0000","last_fullsized":"2024-03-06T07:24:17.265219+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:32.119377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250218+0000","last_change":"2024-03-06T07:24:16.249778+0000","last_active":"2024-03-06T07:24:17.250218+0000","last_peered":"2024-03-06T07:24:17.250218+0000","last_clean":"2024-03-06T07:24:17.250218+0000","last_became_active":"2024-03-06T07:24:16.249542+0000","last_became_peered":"2024-03-06T07:24:16.249542+0000","last_unstale":"2024-03-06T07:24:17.250218+0000","last_undegraded":"2024-03-06T07:24:17.250218+0000","last_fullsized":"2024-03-06T07:24:17.250218+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:10.769430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.260234+0000","last_change":"2024-03-06T07:24:03.888914+0000","last_active":"2024-03-06T07:24:17.260234+0000","last_peered":"2024-03-06T07:24:17.260234+0000","last_clean":"2024-03-06T07:24:17.260234+0000","last_became_active":"2024-03-06T07:24:03.888605+0000","last_became_peered":"2024-03-06T07:24:03.888605+0000","last_unstale":"2024-03-06T07:24:17.260234+0000","last_undegraded":"2024-03-06T07:24:17.260234+0000","last_fullsized":"2024-03-06T07:24:17.260234+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:07.984372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"148'2","reported_seq":22,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.248261+0000","last_change":"2024-03-06T07:24:02.884638+0000","last_active":"2024-03-06T07:24:17.248261+0000","last_peered":"2024-03-06T07:24:17.248261+0000","last_clean":"2024-03-06T07:24:17.248261+0000","last_became_active":"2024-03-06T07:24:02.884370+0000","last_became_peered":"2024-03-06T07:24:02.884370+0000","last_unstale":"2024-03-06T07:24:17.248261+0000","last_undegraded":"2024-03-06T07:24:17.248261+0000","last_fullsized":"2024-03-06T07:24:17.248261+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:38.280774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262215+0000","last_change":"2024-03-06T07:24:17.256748+0000","last_active":"2024-03-06T07:24:17.262215+0000","last_peered":"2024-03-06T07:24:17.262215+0000","last_clean":"2024-03-06T07:24:17.262215+0000","last_became_active":"2024-03-06T07:24:17.256505+0000","last_became_peered":"2024-03-06T07:24:17.256505+0000","last_unstale":"2024-03-06T07:24:17.262215+0000","last_undegraded":"2024-03-06T07:24:17.262215+0000","last_fullsized":"2024-03-06T07:24:17.262215+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:51.041492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252967+0000","last_change":"2024-03-06T07:24:16.248612+0000","last_active":"2024-03-06T07:24:17.252967+0000","last_peered":"2024-03-06T07:24:17.252967+0000","last_clean":"2024-03-06T07:24:17.252967+0000","last_became_active":"2024-03-06T07:24:16.247445+0000","last_became_peered":"2024-03-06T07:24:16.247445+0000","last_unstale":"2024-03-06T07:24:17.252967+0000","last_undegraded":"2024-03-06T07:24:17.252967+0000","last_fullsized":"2024-03-06T07:24:17.252967+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:44.856793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.262149+0000","last_change":"2024-03-06T07:24:03.889791+0000","last_active":"2024-03-06T07:24:17.262149+0000","last_peered":"2024-03-06T07:24:17.262149+0000","last_clean":"2024-03-06T07:24:17.262149+0000","last_became_active":"2024-03-06T07:24:03.889121+0000","last_became_peered":"2024-03-06T07:24:03.889121+0000","last_unstale":"2024-03-06T07:24:17.262149+0000","last_undegraded":"2024-03-06T07:24:17.262149+0000","last_fullsized":"2024-03-06T07:24:17.262149+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:51.877262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"149'1","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.254200+0000","last_change":"2024-03-06T07:24:02.891610+0000","last_active":"2024-03-06T07:24:17.254200+0000","last_peered":"2024-03-06T07:24:17.254200+0000","last_clean":"2024-03-06T07:24:17.254200+0000","last_became_active":"2024-03-06T07:24:02.891294+0000","last_became_peered":"2024-03-06T07:24:02.891294+0000","last_unstale":"2024-03-06T07:24:17.254200+0000","last_undegraded":"2024-03-06T07:24:17.254200+0000","last_fullsized":"2024-03-06T07:24:17.254200+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:39.405811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271786+0000","last_change":"2024-03-06T07:24:17.264101+0000","last_active":"2024-03-06T07:24:17.271786+0000","last_peered":"2024-03-06T07:24:17.271786+0000","last_clean":"2024-03-06T07:24:17.271786+0000","last_became_active":"2024-03-06T07:24:17.263836+0000","last_became_peered":"2024-03-06T07:24:17.263836+0000","last_unstale":"2024-03-06T07:24:17.271786+0000","last_undegraded":"2024-03-06T07:24:17.271786+0000","last_fullsized":"2024-03-06T07:24:17.271786+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:20.590625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278736+0000","last_change":"2024-03-06T07:24:16.252025+0000","last_active":"2024-03-06T07:24:17.278736+0000","last_peered":"2024-03-06T07:24:17.278736+0000","last_clean":"2024-03-06T07:24:17.278736+0000","last_became_active":"2024-03-06T07:24:16.251755+0000","last_became_peered":"2024-03-06T07:24:16.251755+0000","last_unstale":"2024-03-06T07:24:17.278736+0000","last_undegraded":"2024-03-06T07:24:17.278736+0000","last_fullsized":"2024-03-06T07:24:17.278736+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:52.265210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253127+0000","last_change":"2024-03-06T07:24:03.881428+0000","last_active":"2024-03-06T07:24:17.253127+0000","last_peered":"2024-03-06T07:24:17.253127+0000","last_clean":"2024-03-06T07:24:17.253127+0000","last_became_active":"2024-03-06T07:24:03.881168+0000","last_became_peered":"2024-03-06T07:24:03.881168+0000","last_unstale":"2024-03-06T07:24:17.253127+0000","last_undegraded":"2024-03-06T07:24:17.253127+0000","last_fullsized":"2024-03-06T07:24:17.253127+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:00.752911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251111+0000","last_change":"2024-03-06T07:24:02.887206+0000","last_active":"2024-03-06T07:24:17.251111+0000","last_peered":"2024-03-06T07:24:17.251111+0000","last_clean":"2024-03-06T07:24:17.251111+0000","last_became_active":"2024-03-06T07:24:02.886946+0000","last_became_peered":"2024-03-06T07:24:02.886946+0000","last_unstale":"2024-03-06T07:24:17.251111+0000","last_undegraded":"2024-03-06T07:24:17.251111+0000","last_fullsized":"2024-03-06T07:24:17.251111+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:11.489172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.264345+0000","last_change":"2024-03-06T07:24:17.263538+0000","last_active":"2024-03-06T07:24:17.264345+0000","last_peered":"2024-03-06T07:24:17.264345+0000","last_clean":"2024-03-06T07:24:17.264345+0000","last_became_active":"2024-03-06T07:24:17.263265+0000","last_became_peered":"2024-03-06T07:24:17.263265+0000","last_unstale":"2024-03-06T07:24:17.264345+0000","last_undegraded":"2024-03-06T07:24:17.264345+0000","last_fullsized":"2024-03-06T07:24:17.264345+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:11.169809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247687+0000","last_change":"2024-03-06T07:24:16.248840+0000","last_active":"2024-03-06T07:24:17.247687+0000","last_peered":"2024-03-06T07:24:17.247687+0000","last_clean":"2024-03-06T07:24:17.247687+0000","last_became_active":"2024-03-06T07:24:16.248116+0000","last_became_peered":"2024-03-06T07:24:16.248116+0000","last_unstale":"2024-03-06T07:24:17.247687+0000","last_undegraded":"2024-03-06T07:24:17.247687+0000","last_fullsized":"2024-03-06T07:24:17.247687+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:05.734823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.255719+0000","last_change":"2024-03-06T07:24:03.887606+0000","last_active":"2024-03-06T07:24:17.255719+0000","last_peered":"2024-03-06T07:24:17.255719+0000","last_clean":"2024-03-06T07:24:17.255719+0000","last_became_active":"2024-03-06T07:24:03.887318+0000","last_became_peered":"2024-03-06T07:24:03.887318+0000","last_unstale":"2024-03-06T07:24:17.255719+0000","last_undegraded":"2024-03-06T07:24:17.255719+0000","last_fullsized":"2024-03-06T07:24:17.255719+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.250778+0000","last_change":"2024-03-06T07:24:02.891159+0000","last_active":"2024-03-06T07:24:17.250778+0000","last_peered":"2024-03-06T07:24:17.250778+0000","last_clean":"2024-03-06T07:24:17.250778+0000","last_became_active":"2024-03-06T07:24:02.890886+0000","last_became_peered":"2024-03-06T07:24:02.890886+0000","last_unstale":"2024-03-06T07:24:17.250778+0000","last_undegraded":"2024-03-06T07:24:17.250778+0000","last_fullsized":"2024-03-06T07:24:17.250778+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:57.098650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.280932+0000","last_change":"2024-03-06T07:24:17.277737+0000","last_active":"2024-03-06T07:24:17.280932+0000","last_peered":"2024-03-06T07:24:17.280932+0000","last_clean":"2024-03-06T07:24:17.280932+0000","last_became_active":"2024-03-06T07:24:17.277420+0000","last_became_peered":"2024-03-06T07:24:17.277420+0000","last_unstale":"2024-03-06T07:24:17.280932+0000","last_undegraded":"2024-03-06T07:24:17.280932+0000","last_fullsized":"2024-03-06T07:24:17.280932+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:13.915724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277527+0000","last_change":"2024-03-06T07:24:16.252568+0000","last_active":"2024-03-06T07:24:17.277527+0000","last_peered":"2024-03-06T07:24:17.277527+0000","last_clean":"2024-03-06T07:24:17.277527+0000","last_became_active":"2024-03-06T07:24:16.252168+0000","last_became_peered":"2024-03-06T07:24:16.252168+0000","last_unstale":"2024-03-06T07:24:17.277527+0000","last_undegraded":"2024-03-06T07:24:17.277527+0000","last_fullsized":"2024-03-06T07:24:17.277527+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:55.560488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.256170+0000","last_change":"2024-03-06T07:24:03.886641+0000","last_active":"2024-03-06T07:24:17.256170+0000","last_peered":"2024-03-06T07:24:17.256170+0000","last_clean":"2024-03-06T07:24:17.256170+0000","last_became_active":"2024-03-06T07:24:03.885674+0000","last_became_peered":"2024-03-06T07:24:03.885674+0000","last_unstale":"2024-03-06T07:24:17.256170+0000","last_undegraded":"2024-03-06T07:24:17.256170+0000","last_fullsized":"2024-03-06T07:24:17.256170+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:32.495053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253627+0000","last_change":"2024-03-06T07:24:02.888862+0000","last_active":"2024-03-06T07:24:17.253627+0000","last_peered":"2024-03-06T07:24:17.253627+0000","last_clean":"2024-03-06T07:24:17.253627+0000","last_became_active":"2024-03-06T07:24:02.888609+0000","last_became_peered":"2024-03-06T07:24:02.888609+0000","last_unstale":"2024-03-06T07:24:17.253627+0000","last_undegraded":"2024-03-06T07:24:17.253627+0000","last_fullsized":"2024-03-06T07:24:17.253627+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:38.361678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.271780+0000","last_change":"2024-03-06T07:24:17.266303+0000","last_active":"2024-03-06T07:24:17.271780+0000","last_peered":"2024-03-06T07:24:17.271780+0000","last_clean":"2024-03-06T07:24:17.271780+0000","last_became_active":"2024-03-06T07:24:17.265823+0000","last_became_peered":"2024-03-06T07:24:17.265823+0000","last_unstale":"2024-03-06T07:24:17.271780+0000","last_undegraded":"2024-03-06T07:24:17.271780+0000","last_fullsized":"2024-03-06T07:24:17.271780+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:43.160668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258562+0000","last_change":"2024-03-06T07:24:16.248938+0000","last_active":"2024-03-06T07:24:17.258562+0000","last_peered":"2024-03-06T07:24:17.258562+0000","last_clean":"2024-03-06T07:24:17.258562+0000","last_became_active":"2024-03-06T07:24:16.248668+0000","last_became_peered":"2024-03-06T07:24:16.248668+0000","last_unstale":"2024-03-06T07:24:17.258562+0000","last_undegraded":"2024-03-06T07:24:17.258562+0000","last_fullsized":"2024-03-06T07:24:17.258562+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:44.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.263654+0000","last_change":"2024-03-06T07:24:03.893828+0000","last_active":"2024-03-06T07:24:17.263654+0000","last_peered":"2024-03-06T07:24:17.263654+0000","last_clean":"2024-03-06T07:24:17.263654+0000","last_became_active":"2024-03-06T07:24:03.893578+0000","last_became_peered":"2024-03-06T07:24:03.893578+0000","last_unstale":"2024-03-06T07:24:17.263654+0000","last_undegraded":"2024-03-06T07:24:17.263654+0000","last_fullsized":"2024-03-06T07:24:17.263654+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:05.985567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.251942+0000","last_change":"2024-03-06T07:24:02.884916+0000","last_active":"2024-03-06T07:24:17.251942+0000","last_peered":"2024-03-06T07:24:17.251942+0000","last_clean":"2024-03-06T07:24:17.251942+0000","last_became_active":"2024-03-06T07:24:02.883806+0000","last_became_peered":"2024-03-06T07:24:02.883806+0000","last_unstale":"2024-03-06T07:24:17.251942+0000","last_undegraded":"2024-03-06T07:24:17.251942+0000","last_fullsized":"2024-03-06T07:24:17.251942+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:52.367669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.260115+0000","last_change":"2024-03-06T07:24:17.257264+0000","last_active":"2024-03-06T07:24:17.260115+0000","last_peered":"2024-03-06T07:24:17.260115+0000","last_clean":"2024-03-06T07:24:17.260115+0000","last_became_active":"2024-03-06T07:24:17.257035+0000","last_became_peered":"2024-03-06T07:24:17.257035+0000","last_unstale":"2024-03-06T07:24:17.260115+0000","last_undegraded":"2024-03-06T07:24:17.260115+0000","last_fullsized":"2024-03-06T07:24:17.260115+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:44.334236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"152'1","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.278469+0000","last_change":"2024-03-06T07:24:16.254369+0000","last_active":"2024-03-06T07:24:17.278469+0000","last_peered":"2024-03-06T07:24:17.278469+0000","last_clean":"2024-03-06T07:24:17.278469+0000","last_became_active":"2024-03-06T07:24:16.254038+0000","last_became_peered":"2024-03-06T07:24:16.254038+0000","last_unstale":"2024-03-06T07:24:17.278469+0000","last_undegraded":"2024-03-06T07:24:17.278469+0000","last_fullsized":"2024-03-06T07:24:17.278469+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:14.259224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.253686+0000","last_change":"2024-03-06T07:24:03.895097+0000","last_active":"2024-03-06T07:24:17.253686+0000","last_peered":"2024-03-06T07:24:17.253686+0000","last_clean":"2024-03-06T07:24:17.253686+0000","last_became_active":"2024-03-06T07:24:03.894829+0000","last_became_peered":"2024-03-06T07:24:03.894829+0000","last_unstale":"2024-03-06T07:24:17.253686+0000","last_undegraded":"2024-03-06T07:24:17.253686+0000","last_fullsized":"2024-03-06T07:24:17.253686+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:52.563689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.259001+0000","last_change":"2024-03-06T07:24:02.883059+0000","last_active":"2024-03-06T07:24:17.259001+0000","last_peered":"2024-03-06T07:24:17.259001+0000","last_clean":"2024-03-06T07:24:17.259001+0000","last_became_active":"2024-03-06T07:24:02.882626+0000","last_became_peered":"2024-03-06T07:24:02.882626+0000","last_unstale":"2024-03-06T07:24:17.259001+0000","last_undegraded":"2024-03-06T07:24:17.259001+0000","last_fullsized":"2024-03-06T07:24:17.259001+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:54.907245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.277207+0000","last_change":"2024-03-06T07:24:17.273278+0000","last_active":"2024-03-06T07:24:17.277207+0000","last_peered":"2024-03-06T07:24:17.277207+0000","last_clean":"2024-03-06T07:24:17.277207+0000","last_became_active":"2024-03-06T07:24:17.273036+0000","last_became_peered":"2024-03-06T07:24:17.273036+0000","last_unstale":"2024-03-06T07:24:17.277207+0000","last_undegraded":"2024-03-06T07:24:17.277207+0000","last_fullsized":"2024-03-06T07:24:17.277207+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:38.588859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"152'2","reported_seq":14,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.274987+0000","last_change":"2024-03-06T07:24:16.250136+0000","last_active":"2024-03-06T07:24:17.274987+0000","last_peered":"2024-03-06T07:24:17.274987+0000","last_clean":"2024-03-06T07:24:17.274987+0000","last_became_active":"2024-03-06T07:24:16.249896+0000","last_became_peered":"2024-03-06T07:24:16.249896+0000","last_unstale":"2024-03-06T07:24:17.274987+0000","last_undegraded":"2024-03-06T07:24:17.274987+0000","last_fullsized":"2024-03-06T07:24:17.274987+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:56.624573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.252571+0000","last_change":"2024-03-06T07:24:03.885590+0000","last_active":"2024-03-06T07:24:17.252571+0000","last_peered":"2024-03-06T07:24:17.252571+0000","last_clean":"2024-03-06T07:24:17.252571+0000","last_became_active":"2024-03-06T07:24:03.885328+0000","last_became_peered":"2024-03-06T07:24:03.885328+0000","last_unstale":"2024-03-06T07:24:17.252571+0000","last_undegraded":"2024-03-06T07:24:17.252571+0000","last_fullsized":"2024-03-06T07:24:17.252571+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:32.966742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.246812+0000","last_change":"2024-03-06T07:24:02.885607+0000","last_active":"2024-03-06T07:24:17.246812+0000","last_peered":"2024-03-06T07:24:17.246812+0000","last_clean":"2024-03-06T07:24:17.246812+0000","last_became_active":"2024-03-06T07:24:02.885342+0000","last_became_peered":"2024-03-06T07:24:02.885342+0000","last_unstale":"2024-03-06T07:24:17.246812+0000","last_undegraded":"2024-03-06T07:24:17.246812+0000","last_fullsized":"2024-03-06T07:24:17.246812+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:37.173846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":9,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.267837+0000","last_change":"2024-03-06T07:24:17.265577+0000","last_active":"2024-03-06T07:24:17.267837+0000","last_peered":"2024-03-06T07:24:17.267837+0000","last_clean":"2024-03-06T07:24:17.267837+0000","last_became_active":"2024-03-06T07:24:17.265291+0000","last_became_peered":"2024-03-06T07:24:17.265291+0000","last_unstale":"2024-03-06T07:24:17.267837+0000","last_undegraded":"2024-03-06T07:24:17.267837+0000","last_fullsized":"2024-03-06T07:24:17.267837+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:16.231261+0000","last_clean_scrub_stamp":"2024-03-06T07:24:16.231261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:23.414379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.247090+0000","last_change":"2024-03-06T07:24:16.240376+0000","last_active":"2024-03-06T07:24:17.247090+0000","last_peered":"2024-03-06T07:24:17.247090+0000","last_clean":"2024-03-06T07:24:17.247090+0000","last_became_active":"2024-03-06T07:24:16.240124+0000","last_became_peered":"2024-03-06T07:24:16.240124+0000","last_unstale":"2024-03-06T07:24:17.247090+0000","last_undegraded":"2024-03-06T07:24:17.247090+0000","last_fullsized":"2024-03-06T07:24:17.247090+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:15.226682+0000","last_clean_scrub_stamp":"2024-03-06T07:24:15.226682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:24.856326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.257222+0000","last_change":"2024-03-06T07:24:03.891512+0000","last_active":"2024-03-06T07:24:17.257222+0000","last_peered":"2024-03-06T07:24:17.257222+0000","last_clean":"2024-03-06T07:24:17.257222+0000","last_became_active":"2024-03-06T07:24:03.891241+0000","last_became_peered":"2024-03-06T07:24:03.891241+0000","last_unstale":"2024-03-06T07:24:17.257222+0000","last_undegraded":"2024-03-06T07:24:17.257222+0000","last_fullsized":"2024-03-06T07:24:17.257222+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":147,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:02.864943+0000","last_clean_scrub_stamp":"2024-03-06T07:24:02.864943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:34.678948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.254108+0000","last_change":"2024-03-06T07:24:02.891039+0000","last_active":"2024-03-06T07:24:17.254108+0000","last_peered":"2024-03-06T07:24:17.254108+0000","last_clean":"2024-03-06T07:24:17.254108+0000","last_became_active":"2024-03-06T07:24:02.890670+0000","last_became_peered":"2024-03-06T07:24:02.890670+0000","last_unstale":"2024-03-06T07:24:17.254108+0000","last_undegraded":"2024-03-06T07:24:17.254108+0000","last_fullsized":"2024-03-06T07:24:17.254108+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":146,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:01.859528+0000","last_clean_scrub_stamp":"2024-03-06T07:24:01.859528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:06.218892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":421,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-06T07:24:17.258479+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:17.258479+0000","last_peered":"2024-03-06T07:24:17.258479+0000","last_clean":"2024-03-06T07:24:17.258479+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:17.258479+0000","last_undegraded":"2024-03-06T07:24:17.258479+0000","last_fullsized":"2024-03-06T07:24:17.258479+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1966080,"data_stored":1509566,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":919198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1908736,"data_stored":919322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1916928,"data_stored":919962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1921024,"data_stored":921414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":919198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33052,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338788,"statfs":{"total":96636764160,"available":96602918912,"internally_reserved":0,"allocated":1912832,"data_stored":920810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":1974272,"data_stored":1509670,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:21.820 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:21.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:21.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.969+0000 7f840837a640 1 Processor -- start 2024-03-06T07:24:21.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.969+0000 7f840837a640 1 -- start start 2024-03-06T07:24:21.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f840837a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f840014f360 0x7f8400155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f840837a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84000c88c0 0x7f84000c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f840837a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84000c73d0 con 0x7f84000c5880 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f840837a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84000c8dd0 con 0x7f84000c88c0 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f840837a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84000be2e0 con 0x7f840014f360 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8405915640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84000c88c0 0x7f84000c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8406917640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f84000c5880 0x7f8400158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48502/0 (socket says 172.21.15.94:48502) 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8406917640 1 -- 172.21.15.94:0/1505103033 learned_addr learned my addr 172.21.15.94:0/1505103033 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8406116640 1 --2- 172.21.15.94:0/1505103033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f840014f360 0x7f8400155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8405915640 1 -- 172.21.15.94:0/1505103033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f840014f360 msgr2=0x7f8400155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8405915640 1 --2- 172.21.15.94:0/1505103033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f840014f360 0x7f8400155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8405915640 1 -- 172.21.15.94:0/1505103033 >> v1:172.21.15.94:6789/0 conn(0x7f84000c5880 legacy=0x7f8400158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8405915640 1 -- 172.21.15.94:0/1505103033 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000c7670 con 0x7f84000c88c0 2024-03-06T07:24:21.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.970+0000 7f8405114640 1 -- 172.21.15.94:0/1505103033 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723056242 0 0) 0x7f84000c73d0 con 0x7f84000c5880 2024-03-06T07:24:21.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.971+0000 7f8405915640 1 --2- 172.21.15.94:0/1505103033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84000c88c0 0x7f84000c5340 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f83f00097a0 tx=0x7f83f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70629558ef282bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.971+0000 7f8405114640 1 -- 172.21.15.94:0/1505103033 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f00315a0 con 0x7f84000c88c0 2024-03-06T07:24:21.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.971+0000 7f8405114640 1 -- 172.21.15.94:0/1505103033 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f0037070 con 0x7f84000c88c0 2024-03-06T07:24:21.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.971+0000 7f8405114640 1 -- 172.21.15.94:0/1505103033 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0031960 con 0x7f84000c88c0 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 -- 172.21.15.94:0/1505103033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84000c88c0 msgr2=0x7f84000c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 --2- 172.21.15.94:0/1505103033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84000c88c0 0x7f84000c5340 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f83f00097a0 tx=0x7f83f002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 -- 172.21.15.94:0/1505103033 shutdown_connections 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 --2- 172.21.15.94:0/1505103033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84000c88c0 0x7f84000c5340 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 --2- 172.21.15.94:0/1505103033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f840014f360 0x7f8400155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 -- 172.21.15.94:0/1505103033 >> 172.21.15.94:0/1505103033 conn(0x7f84000ac570 msgr2=0x7f84000ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 -- 172.21.15.94:0/1505103033 shutdown_connections 2024-03-06T07:24:21.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.972+0000 7f840837a640 1 -- 172.21.15.94:0/1505103033 wait complete. 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.974+0000 7f840837a640 1 Processor -- start 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.974+0000 7f840837a640 1 -- start start 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f840837a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84000c5880 0x7f84000ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f840837a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84000c88c0 0x7f84001ea880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f840837a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f840014f360 0x7f84000af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f840837a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8400159270 con 0x7f840014f360 2024-03-06T07:24:21.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f840837a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84000c7c30 con 0x7f84000c5880 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f840837a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84001593f0 con 0x7f84000c88c0 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f840014f360 0x7f84000af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f840014f360 0x7f84000af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58982/0 (socket says 172.21.15.94:58982) 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 -- 172.21.15.94:0/1402424501 learned_addr learned my addr 172.21.15.94:0/1402424501 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8405915640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84000c88c0 0x7f84001ea880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406116640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84000c5880 0x7f84000ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 -- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84000c88c0 msgr2=0x7f84001ea880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84000c88c0 0x7f84001ea880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 -- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84000c5880 msgr2=0x7f84000ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84000c5880 0x7f84000ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 -- 172.21.15.94:0/1402424501 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000ad970 con 0x7f840014f360 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.975+0000 7f8406917640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f840014f360 0x7f84000af960 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f83f800c7d0 tx=0x7f83f800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.976+0000 7f83f6ffd640 1 -- 172.21.15.94:0/1402424501 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8018070 con 0x7f840014f360 2024-03-06T07:24:21.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.976+0000 7f83f6ffd640 1 -- 172.21.15.94:0/1402424501 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f800a640 con 0x7f840014f360 2024-03-06T07:24:21.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.976+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84000adc50 con 0x7f840014f360 2024-03-06T07:24:21.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.976+0000 7f83f6ffd640 1 -- 172.21.15.94:0/1402424501 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f800aa70 con 0x7f840014f360 2024-03-06T07:24:21.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.976+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84000ae210 con 0x7f840014f360 2024-03-06T07:24:21.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.977+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840004f330 con 0x7f840014f360 2024-03-06T07:24:21.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.977+0000 7f83f6ffd640 1 -- 172.21.15.94:0/1402424501 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83f801b070 con 0x7f840014f360 2024-03-06T07:24:21.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.977+0000 7f83f6ffd640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83e406b4d0 0x7f83e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.978+0000 7f83f6ffd640 1 -- 172.21.15.94:0/1402424501 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f83f8089bc0 con 0x7f840014f360 2024-03-06T07:24:21.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.978+0000 7f8406116640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83e406b4d0 0x7f83e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.978+0000 7f8406116640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83e406b4d0 0x7f83e406d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00aa20 tx=0x7f83fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:21.982+0000 7f83f6ffd640 1 -- 172.21.15.94:0/1402424501 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83f8057fa0 con 0x7f840014f360 2024-03-06T07:24:22.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.213+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84000ca160 con 0x7f840014f360 2024-03-06T07:24:22.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.213+0000 7f83f6ffd640 1 -- 172.21.15.94:0/1402424501 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6275 (secure 0 0 0) 0x7f83f805c040 con 0x7f840014f360 2024-03-06T07:24:22.216 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:22.216 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8155,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:05.235078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/1613596341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1613596341},{"type":"v1","addr":"172.21.15.94:6835","nonce":1613596341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:19.614520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:22.216 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 140 2024-03-06T07:24:22.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83e406b4d0 msgr2=0x7f83e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83e406b4d0 0x7f83e406d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00aa20 tx=0x7f83fc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f840014f360 msgr2=0x7f84000af960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f840014f360 0x7f84000af960 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f83f800c7d0 tx=0x7f83f800cc90 comp rx=0 tx=0).stop 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 shutdown_connections 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83e406b4d0 0x7f83e406d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f840014f360 0x7f84000af960 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84000c88c0 0x7f84001ea880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 --2- 172.21.15.94:0/1402424501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84000c5880 0x7f84000ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.215+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 >> 172.21.15.94:0/1402424501 conn(0x7f84000ac570 msgr2=0x7f840014d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.216+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 shutdown_connections 2024-03-06T07:24:22.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.216+0000 7f840837a640 1 -- 172.21.15.94:0/1402424501 wait complete. 2024-03-06T07:24:22.229 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 140, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:03.229941+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 135, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8772": { "addr": "172.21.15.94:6835/1613596341", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 1613596341, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 1613596341, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8772, "incarnation": 133, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-06T07:24:05.235078+0000", "qdb_cluster": [ 8772 ], "qdb_leader": 8772, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8772 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:16.636350+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8769": { "addr": "172.21.15.94:6834/1222126706", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 1222126706, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 1222126706, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8769, "incarnation": 136, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-06T07:24:19.614520+0000", "qdb_cluster": [ 8769 ], "qdb_leader": 8769, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8769 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/1521427660", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1521427660, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1521427660, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 132, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8152, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/486493869", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 486493869, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 486493869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 132, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8155, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:22.229 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:22.229 INFO:tasks.cephfs.test_failover:current = [{'gid': 8769, 'name': 'a', 'rank': 0, 'incarnation': 136, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.94:6834/1222126706', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 1222126706}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 1222126706}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8155, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 132}, {'gid': 8772, 'name': 'c', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.94:6835/1613596341', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 1613596341}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 1613596341}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8152, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/1521427660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1521427660}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1521427660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 132}] 2024-03-06T07:24:22.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:22.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.329+0000 7fae3ce46640 1 Processor -- start 2024-03-06T07:24:22.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.329+0000 7fae3ce46640 1 -- start start 2024-03-06T07:24:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae3ce46640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fae381506b0 0x7fae38153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae3ce46640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fae38156ea0 0x7fae380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae3ce46640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae380acc00 0x7fae380c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae3ce46640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae38156450 con 0x7fae38156ea0 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae3ce46640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae38156a10 con 0x7fae381506b0 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae3ce46640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae38070800 con 0x7fae380acc00 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fae38156ea0 0x7fae380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fae38156ea0 0x7fae380702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58994/0 (socket says 172.21.15.94:58994) 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 -- 172.21.15.94:0/1771414929 learned_addr learned my addr 172.21.15.94:0/1771414929 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae36d9d640 1 --2- 172.21.15.94:0/1771414929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae380acc00 0x7fae380c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 -- 172.21.15.94:0/1771414929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae380acc00 msgr2=0x7fae380c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae3659c640 1 --2- 172.21.15.94:0/1771414929 >> v2:172.21.15.145:3300/0 conn(0x7fae381506b0 0x7fae38153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 --2- 172.21.15.94:0/1771414929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae380acc00 0x7fae380c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 -- 172.21.15.94:0/1771414929 >> v2:172.21.15.145:3300/0 conn(0x7fae381506b0 msgr2=0x7fae38153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 --2- 172.21.15.94:0/1771414929 >> v2:172.21.15.145:3300/0 conn(0x7fae381506b0 0x7fae38153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.330+0000 7fae2ffff640 1 -- 172.21.15.94:0/1771414929 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38070aa0 con 0x7fae38156ea0 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.331+0000 7fae2ffff640 1 --2- 172.21.15.94:0/1771414929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae38156ea0 0x7fae380702c0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fae380c97e0 tx=0x7fae2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc81d4682d3ea14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.331+0000 7fae35d9b640 1 -- 172.21.15.94:0/1771414929 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20035020 con 0x7fae38156ea0 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.331+0000 7fae35d9b640 1 -- 172.21.15.94:0/1771414929 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae20031870 con 0x7fae38156ea0 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.331+0000 7fae35d9b640 1 -- 172.21.15.94:0/1771414929 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20031c40 con 0x7fae38156ea0 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 -- 172.21.15.94:0/1771414929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae38156ea0 msgr2=0x7fae380702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 --2- 172.21.15.94:0/1771414929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae38156ea0 0x7fae380702c0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fae380c97e0 tx=0x7fae2002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 -- 172.21.15.94:0/1771414929 shutdown_connections 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 --2- 172.21.15.94:0/1771414929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae380acc00 0x7fae380c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 --2- 172.21.15.94:0/1771414929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae38156ea0 0x7fae380702c0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 --2- 172.21.15.94:0/1771414929 >> v2:172.21.15.145:3300/0 conn(0x7fae381506b0 0x7fae38153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 -- 172.21.15.94:0/1771414929 >> 172.21.15.94:0/1771414929 conn(0x7fae3806b1a0 msgr2=0x7fae3806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 -- 172.21.15.94:0/1771414929 shutdown_connections 2024-03-06T07:24:22.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.332+0000 7fae3ce46640 1 -- 172.21.15.94:0/1771414929 wait complete. 2024-03-06T07:24:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.333+0000 7fae3ce46640 1 Processor -- start 2024-03-06T07:24:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.333+0000 7fae3ce46640 1 -- start start 2024-03-06T07:24:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae3ce46640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae380acc00 0x7fae380c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae3ce46640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae381506b0 0x7fae38146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae3ce46640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae38156ea0 0x7fae38147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae3ce46640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae380c59e0 con 0x7fae381506b0 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae3ce46640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae381566a0 con 0x7fae380acc00 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae3ce46640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae380c5b60 con 0x7fae38156ea0 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae381506b0 0x7fae38146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae381506b0 0x7fae38146dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59008/0 (socket says 172.21.15.94:59008) 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 -- 172.21.15.94:0/3986756936 learned_addr learned my addr 172.21.15.94:0/3986756936 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae36d9d640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae38156ea0 0x7fae38147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 -- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae38156ea0 msgr2=0x7fae38147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae38156ea0 0x7fae38147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 -- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae380acc00 msgr2=0x7fae380c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae3659c640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae380acc00 0x7fae380c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae380acc00 0x7fae380c42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 -- 172.21.15.94:0/3986756936 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3806c630 con 0x7fae381506b0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.334+0000 7fae2ffff640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae381506b0 0x7fae38146dd0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fae20000c00 tx=0x7fae200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.335+0000 7fae2effd640 1 -- 172.21.15.94:0/3986756936 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20040070 con 0x7fae381506b0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.335+0000 7fae2effd640 1 -- 172.21.15.94:0/3986756936 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2003e070 con 0x7fae381506b0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.335+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae38150e90 con 0x7fae381506b0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.335+0000 7fae2effd640 1 -- 172.21.15.94:0/3986756936 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20035040 con 0x7fae381506b0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.335+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae380c6ea0 con 0x7fae381506b0 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.336+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3804f330 con 0x7fae381506b0 2024-03-06T07:24:22.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.337+0000 7fae2effd640 1 -- 172.21.15.94:0/3986756936 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae20038020 con 0x7fae381506b0 2024-03-06T07:24:22.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.337+0000 7fae2effd640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae1806b4d0 0x7fae1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.337+0000 7fae2effd640 1 -- 172.21.15.94:0/3986756936 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fae200ac660 con 0x7fae381506b0 2024-03-06T07:24:22.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.337+0000 7fae3659c640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae1806b4d0 0x7fae1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.338+0000 7fae3659c640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae1806b4d0 0x7fae1806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fae28006fd0 tx=0x7fae28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.342+0000 7fae2effd640 1 -- 172.21.15.94:0/3986756936 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae2002f320 con 0x7fae381506b0 2024-03-06T07:24:22.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.625+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae380700a0 con 0x7fae381506b0 2024-03-06T07:24:22.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.625+0000 7fae2effd640 1 -- 172.21.15.94:0/3986756936 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+6275 (secure 0 0 0) 0x7fae2007b000 con 0x7fae381506b0 2024-03-06T07:24:22.627 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:22.627 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":140,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8155,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:05.235078+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"c","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.15.94:6835/1613596341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1613596341},{"type":"v1","addr":"172.21.15.94:6835","nonce":1613596341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:19.614520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:22.627 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 140 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae1806b4d0 msgr2=0x7fae1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae1806b4d0 0x7fae1806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fae28006fd0 tx=0x7fae28008040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae381506b0 msgr2=0x7fae38146dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae381506b0 0x7fae38146dd0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fae20000c00 tx=0x7fae200046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 shutdown_connections 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae1806b4d0 0x7fae1806d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae38156ea0 0x7fae38147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fae381506b0 0x7fae38146dd0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 --2- 172.21.15.94:0/3986756936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae380acc00 0x7fae380c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 >> 172.21.15.94:0/3986756936 conn(0x7fae3806b1a0 msgr2=0x7fae3814d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 shutdown_connections 2024-03-06T07:24:22.630 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.627+0000 7fae3ce46640 1 -- 172.21.15.94:0/3986756936 wait complete. 2024-03-06T07:24:22.641 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 140, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:03.229941+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 135, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8772": { "addr": "172.21.15.94:6835/1613596341", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 1613596341, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 1613596341, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8772, "incarnation": 133, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-06T07:24:05.235078+0000", "qdb_cluster": [ 8772 ], "qdb_leader": 8772, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8772 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:16.636350+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8769": { "addr": "172.21.15.94:6834/1222126706", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 1222126706, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 1222126706, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8769, "incarnation": 136, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-06T07:24:19.614520+0000", "qdb_cluster": [ 8769 ], "qdb_leader": 8769, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8769 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/1521427660", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1521427660, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1521427660, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 132, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8152, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.145:6837/486493869", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 486493869, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 486493869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 132, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8155, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:22.641 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-03-06T07:24:22.641 INFO:tasks.cephfs.test_failover:current = [{'gid': 8769, 'name': 'a', 'rank': 0, 'incarnation': 136, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.94:6834/1222126706', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 1222126706}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 1222126706}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8155, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 132}, {'gid': 8772, 'name': 'c', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.94:6835/1613596341', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 1613596341}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 1613596341}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8152, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/1521427660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1521427660}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1521427660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 132}] 2024-03-06T07:24:22.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-03-06T07:24:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.744+0000 7fbce409d640 1 Processor -- start 2024-03-06T07:24:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.744+0000 7fbce409d640 1 -- start start 2024-03-06T07:24:22.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce409d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbcdc158930 0x7fbcdc1526f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce409d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbcdc152c30 0x7fbcdc158d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce409d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc072950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce409d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc0c2aa0 con 0x7fbcdc152c30 2024-03-06T07:24:22.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce409d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc0c3490 con 0x7fbcdc158930 2024-03-06T07:24:22.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce409d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc0c3770 con 0x7fbcdc159250 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce1638640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbcdc152c30 0x7fbcdc158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce1638640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbcdc152c30 0x7fbcdc158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59012/0 (socket says 172.21.15.94:59012) 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce1638640 1 -- 172.21.15.94:0/535161135 learned_addr learned my addr 172.21.15.94:0/535161135 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce1e39640 1 --2- 172.21.15.94:0/535161135 >> v2:172.21.15.145:3300/0 conn(0x7fbcdc158930 0x7fbcdc1526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce263a640 1 --2- 172.21.15.94:0/535161135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.745+0000 7fbce1638640 1 -- 172.21.15.94:0/535161135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 msgr2=0x7fbcdc072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce1638640 1 --2- 172.21.15.94:0/535161135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc072950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce1638640 1 -- 172.21.15.94:0/535161135 >> v2:172.21.15.145:3300/0 conn(0x7fbcdc158930 msgr2=0x7fbcdc1526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce1638640 1 --2- 172.21.15.94:0/535161135 >> v2:172.21.15.145:3300/0 conn(0x7fbcdc158930 0x7fbcdc1526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce1638640 1 -- 172.21.15.94:0/535161135 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc0c2730 con 0x7fbcdc152c30 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce263a640 1 --2- 172.21.15.94:0/535161135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc072950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce1638640 1 --2- 172.21.15.94:0/535161135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc158d10 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc067c30 tx=0x7fbccc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f28a10c2c55e9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce0e37640 1 -- 172.21.15.94:0/535161135 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc02e070 con 0x7fbcdc152c30 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce0e37640 1 -- 172.21.15.94:0/535161135 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc037070 con 0x7fbcdc152c30 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.746+0000 7fbce0e37640 1 -- 172.21.15.94:0/535161135 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc031a00 con 0x7fbcdc152c30 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 -- 172.21.15.94:0/535161135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 msgr2=0x7fbcdc158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 --2- 172.21.15.94:0/535161135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc158d10 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc067c30 tx=0x7fbccc02b880 comp rx=0 tx=0).stop 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 -- 172.21.15.94:0/535161135 shutdown_connections 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 --2- 172.21.15.94:0/535161135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc072950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 --2- 172.21.15.94:0/535161135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc158d10 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 --2- 172.21.15.94:0/535161135 >> v2:172.21.15.145:3300/0 conn(0x7fbcdc158930 0x7fbcdc1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 -- 172.21.15.94:0/535161135 >> 172.21.15.94:0/535161135 conn(0x7fbcdc0aa6a0 msgr2=0x7fbcdc0aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 -- 172.21.15.94:0/535161135 shutdown_connections 2024-03-06T07:24:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.747+0000 7fbce409d640 1 -- 172.21.15.94:0/535161135 wait complete. 2024-03-06T07:24:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.748+0000 7fbce409d640 1 Processor -- start 2024-03-06T07:24:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.748+0000 7fbce409d640 1 -- start start 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce409d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce409d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc158930 0x7fbcdc06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce409d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce409d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc159810 con 0x7fbcdc152c30 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce409d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc159690 con 0x7fbcdc158930 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce409d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc072e90 con 0x7fbcdc159250 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59016/0 (socket says 172.21.15.94:59016) 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 -- 172.21.15.94:0/2752819470 learned_addr learned my addr 172.21.15.94:0/2752819470 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1638640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc158930 0x7fbcdc06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce263a640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 -- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 msgr2=0x7fbcdc14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc14db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 -- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc158930 msgr2=0x7fbcdc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc158930 0x7fbcdc06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce1e39640 1 -- 172.21.15.94:0/2752819470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc06ddb0 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.749+0000 7fbce263a640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.750+0000 7fbce1e39640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc155c40 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800e9e0 tx=0x7fbcd800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.750+0000 7fbcd27fc640 1 -- 172.21.15.94:0/2752819470 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8017070 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.750+0000 7fbcd27fc640 1 -- 172.21.15.94:0/2752819470 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd800c9d0 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.750+0000 7fbcd27fc640 1 -- 172.21.15.94:0/2752819470 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd800cdc0 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.750+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc1ee1d0 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.750+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc1ee790 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.751+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc04f330 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.751+0000 7fbcd27fc640 1 -- 172.21.15.94:0/2752819470 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcd801a070 con 0x7fbcdc152c30 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.752+0000 7fbcd27fc640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbcc006b4d0 0x7fbcc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.752+0000 7fbcd27fc640 1 -- 172.21.15.94:0/2752819470 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fbcd808ab00 con 0x7fbcdc152c30 2024-03-06T07:24:22.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.752+0000 7fbce1638640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbcc006b4d0 0x7fbcc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.753+0000 7fbce1638640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbcc006b4d0 0x7fbcc006d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fbccc036000 tx=0x7fbccc03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.756+0000 7fbcd27fc640 1 -- 172.21.15.94:0/2752819470 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcd8058dc0 con 0x7fbcdc152c30 2024-03-06T07:24:22.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.936+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fbcdc0adea0 con 0x7fbcdc152c30 2024-03-06T07:24:22.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.940+0000 7fbcd27fc640 1 -- 172.21.15.94:0/2752819470 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v52)=0 v52) v1 ==== 117+0+0 (secure 0 0 0) 0x7fbcd805ce60 con 0x7fbcdc152c30 2024-03-06T07:24:22.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbcc006b4d0 msgr2=0x7fbcc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbcc006b4d0 0x7fbcc006d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fbccc036000 tx=0x7fbccc03d040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 msgr2=0x7fbcdc155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc155c40 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800e9e0 tx=0x7fbcd800eea0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 shutdown_connections 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbcc006b4d0 0x7fbcc006d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbcdc159250 0x7fbcdc14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbcdc158930 0x7fbcdc06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 --2- 172.21.15.94:0/2752819470 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbcdc152c30 0x7fbcdc155c40 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 >> 172.21.15.94:0/2752819470 conn(0x7fbcdc0aa6a0 msgr2=0x7fbcdc0a9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 shutdown_connections 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:22.942+0000 7fbce409d640 1 -- 172.21.15.94:0/2752819470 wait complete. 2024-03-06T07:24:22.958 INFO:tasks.cephfs.test_failover:assigning affinity to cephfs2 for active mds (mds.a) 2024-03-06T07:24:22.958 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.a mds_join_fs cephfs2 2024-03-06T07:24:23.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.075+0000 7f39149f6640 1 Processor -- start 2024-03-06T07:24:23.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.075+0000 7f39149f6640 1 -- start start 2024-03-06T07:24:23.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f39149f6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39101527f0 0x7f39100c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f39149f6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f39100c7370 0x7f39100c7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f39149f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3910155260 con 0x7f391006c0b0 2024-03-06T07:24:23.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f39149f6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3910155540 con 0x7f39101527f0 2024-03-06T07:24:23.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f39149f6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3910155820 con 0x7f39100c7370 2024-03-06T07:24:23.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f390e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39101527f0 0x7f39100c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f390e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f39101527f0 0x7f39100c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59026/0 (socket says 172.21.15.94:59026) 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f390e59c640 1 -- 172.21.15.94:0/195511210 learned_addr learned my addr 172.21.15.94:0/195511210 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.076+0000 7f390dd9b640 1 --2- 172.21.15.94:0/195511210 >> v2:172.21.15.145:3300/0 conn(0x7f39100c7370 0x7f39100c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.077+0000 7f390e59c640 1 -- 172.21.15.94:0/195511210 >> v2:172.21.15.145:3300/0 conn(0x7f39100c7370 msgr2=0x7f39100c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.077+0000 7f390e59c640 1 --2- 172.21.15.94:0/195511210 >> v2:172.21.15.145:3300/0 conn(0x7f39100c7370 0x7f39100c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.077+0000 7f390e59c640 1 -- 172.21.15.94:0/195511210 >> v1:172.21.15.145:6789/0 conn(0x7f391006c0b0 legacy=0x7f391006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.077+0000 7f390e59c640 1 -- 172.21.15.94:0/195511210 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910155b00 con 0x7f39101527f0 2024-03-06T07:24:23.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.077+0000 7f390dd9b640 1 --2- 172.21.15.94:0/195511210 >> v2:172.21.15.145:3300/0 conn(0x7f39100c7370 0x7f39100c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:23.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.077+0000 7f390e59c640 1 --2- 172.21.15.94:0/195511210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39101527f0 0x7f39100c8860 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f39100aa410 tx=0x7f390402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b715fb512c0ffc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.077+0000 7f390d59a640 1 -- 172.21.15.94:0/195511210 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390402e070 con 0x7f39101527f0 2024-03-06T07:24:23.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f390d59a640 1 -- 172.21.15.94:0/195511210 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3904037070 con 0x7f39101527f0 2024-03-06T07:24:23.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f390d59a640 1 -- 172.21.15.94:0/195511210 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3904031a00 con 0x7f39101527f0 2024-03-06T07:24:23.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 -- 172.21.15.94:0/195511210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39101527f0 msgr2=0x7f39100c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 --2- 172.21.15.94:0/195511210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39101527f0 0x7f39100c8860 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f39100aa410 tx=0x7f390402b880 comp rx=0 tx=0).stop 2024-03-06T07:24:23.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 -- 172.21.15.94:0/195511210 shutdown_connections 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 --2- 172.21.15.94:0/195511210 >> v2:172.21.15.145:3300/0 conn(0x7f39100c7370 0x7f39100c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 --2- 172.21.15.94:0/195511210 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39101527f0 0x7f39100c8860 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 -- 172.21.15.94:0/195511210 >> 172.21.15.94:0/195511210 conn(0x7f39100ac4f0 msgr2=0x7f39100ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 -- 172.21.15.94:0/195511210 shutdown_connections 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.078+0000 7f39149f6640 1 -- 172.21.15.94:0/195511210 wait complete. 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 Processor -- start 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 -- start start 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f391006c0b0 0x7f39100b0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39100c7370 0x7f39100bea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39101527f0 0x7f39100bef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39100c8f20 con 0x7f39100c7370 2024-03-06T07:24:23.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39100c8da0 con 0x7f391006c0b0 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f39149f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391006cbd0 con 0x7f39101527f0 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f390dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39100c7370 0x7f39100bea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f390dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39100c7370 0x7f39100bea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59032/0 (socket says 172.21.15.94:59032) 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f390dd9b640 1 -- 172.21.15.94:0/577109570 learned_addr learned my addr 172.21.15.94:0/577109570 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.080+0000 7f390e59c640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f391006c0b0 0x7f39100b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390ed9d640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39101527f0 0x7f39100bef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390dd9b640 1 -- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39101527f0 msgr2=0x7f39100bef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390dd9b640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39101527f0 0x7f39100bef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390dd9b640 1 -- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f391006c0b0 msgr2=0x7f39100b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390dd9b640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f391006c0b0 0x7f39100b0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390dd9b640 1 -- 172.21.15.94:0/577109570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39100aeff0 con 0x7f39100c7370 2024-03-06T07:24:23.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390e59c640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f391006c0b0 0x7f39100b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:23.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f390dd9b640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39100c7370 0x7f39100bea30 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f390000d7d0 tx=0x7f390000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f38feffd640 1 -- 172.21.15.94:0/577109570 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900004030 con 0x7f39100c7370 2024-03-06T07:24:23.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f38feffd640 1 -- 172.21.15.94:0/577109570 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39000041d0 con 0x7f39100c7370 2024-03-06T07:24:23.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f38feffd640 1 -- 172.21.15.94:0/577109570 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000096a0 con 0x7f39100c7370 2024-03-06T07:24:23.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39100ae750 con 0x7f39100c7370 2024-03-06T07:24:23.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.081+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39100aed10 con 0x7f39100c7370 2024-03-06T07:24:23.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.082+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391004f330 con 0x7f39100c7370 2024-03-06T07:24:23.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.083+0000 7f38feffd640 1 -- 172.21.15.94:0/577109570 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3900009840 con 0x7f39100c7370 2024-03-06T07:24:23.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.083+0000 7f38feffd640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38f006b4d0 0x7f38f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.083+0000 7f38feffd640 1 -- 172.21.15.94:0/577109570 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f390008a980 con 0x7f39100c7370 2024-03-06T07:24:23.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.083+0000 7f390e59c640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38f006b4d0 0x7f38f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.084+0000 7f390e59c640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38f006b4d0 0x7f38f006d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3904036000 tx=0x7f390403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.087+0000 7f38feffd640 1 -- 172.21.15.94:0/577109570 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39000593b0 con 0x7f39100c7370 2024-03-06T07:24:23.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.261+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f391006d860 con 0x7f39100c7370 2024-03-06T07:24:23.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.265+0000 7f38feffd640 1 -- 172.21.15.94:0/577109570 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v53)=0 v53) v1 ==== 117+0+0 (secure 0 0 0) 0x7f3900058cb0 con 0x7f39100c7370 2024-03-06T07:24:23.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38f006b4d0 msgr2=0x7f38f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38f006b4d0 0x7f38f006d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3904036000 tx=0x7f390403d040 comp rx=0 tx=0).stop 2024-03-06T07:24:23.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39100c7370 msgr2=0x7f39100bea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39100c7370 0x7f39100bea30 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f390000d7d0 tx=0x7f390000dc90 comp rx=0 tx=0).stop 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 shutdown_connections 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38f006b4d0 0x7f38f006d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39101527f0 0x7f39100bef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f39100c7370 0x7f39100bea30 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 --2- 172.21.15.94:0/577109570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f391006c0b0 0x7f39100b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 >> 172.21.15.94:0/577109570 conn(0x7f39100ac4f0 msgr2=0x7f39100abc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 shutdown_connections 2024-03-06T07:24:23.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.267+0000 7f39149f6640 1 -- 172.21.15.94:0/577109570 wait complete. 2024-03-06T07:24:23.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 11:0 2024-03-06T07:24:23.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.417+0000 7fc6d7d81640 1 Processor -- start 2024-03-06T07:24:23.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.418+0000 7fc6d7d81640 1 -- start start 2024-03-06T07:24:23.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.418+0000 7fc6d7d81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d0155240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.418+0000 7fc6d7d81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc6d00c64f0 0x7fc6d00c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.418+0000 7fc6d7d81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d00c8e30 con 0x7fc6d0152900 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.418+0000 7fc6d7d81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d0152ce0 con 0x7fc6d00c64f0 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.418+0000 7fc6d7d81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c9110 con 0x7fc6d00c8950 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d531c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d0155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d631e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc6d00c64f0 0x7fc6d00c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d5b1d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc6d0152900 0x7fc6d00bf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:40036/0 (socket says 172.21.15.94:40036) 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d5b1d640 1 -- 172.21.15.94:0/22748779 learned_addr learned my addr 172.21.15.94:0/22748779 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d531c640 1 -- 172.21.15.94:0/22748779 >> v2:172.21.15.145:3300/0 conn(0x7fc6d00c64f0 msgr2=0x7fc6d00c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d531c640 1 --2- 172.21.15.94:0/22748779 >> v2:172.21.15.145:3300/0 conn(0x7fc6d00c64f0 0x7fc6d00c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d531c640 1 -- 172.21.15.94:0/22748779 >> v1:172.21.15.145:6789/0 conn(0x7fc6d0152900 legacy=0x7fc6d00bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d531c640 1 -- 172.21.15.94:0/22748779 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00c93f0 con 0x7fc6d00c8950 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.419+0000 7fc6d631e640 1 --2- 172.21.15.94:0/22748779 >> v2:172.21.15.145:3300/0 conn(0x7fc6d00c64f0 0x7fc6d00c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:23.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.420+0000 7fc6d531c640 1 --2- 172.21.15.94:0/22748779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d0155240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00097a0 tx=0x7fc6c002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbedb187f491bddd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.420+0000 7fc6d4b1b640 1 -- 172.21.15.94:0/22748779 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0038070 con 0x7fc6d00c8950 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.420+0000 7fc6d4b1b640 1 -- 172.21.15.94:0/22748779 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c0031700 con 0x7fc6d00c8950 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.420+0000 7fc6d4b1b640 1 -- 172.21.15.94:0/22748779 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0031ad0 con 0x7fc6d00c8950 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 -- 172.21.15.94:0/22748779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 msgr2=0x7fc6d0155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/22748779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d0155240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00097a0 tx=0x7fc6c002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 -- 172.21.15.94:0/22748779 shutdown_connections 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/22748779 >> v2:172.21.15.145:3300/0 conn(0x7fc6d00c64f0 0x7fc6d00c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/22748779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d0155240 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 -- 172.21.15.94:0/22748779 >> 172.21.15.94:0/22748779 conn(0x7fc6d00ac860 msgr2=0x7fc6d00acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 -- 172.21.15.94:0/22748779 shutdown_connections 2024-03-06T07:24:23.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.421+0000 7fc6d7d81640 1 -- 172.21.15.94:0/22748779 wait complete. 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.422+0000 7fc6d7d81640 1 Processor -- start 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.422+0000 7fc6d7d81640 1 -- start start 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.423+0000 7fc6d7d81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d00c64f0 0x7fc6d014ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d7d81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d00ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d7d81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6d0152900 0x7fc6d01eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d7d81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c0180 con 0x7fc6d00c64f0 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d7d81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c0000 con 0x7fc6d0152900 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d7d81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d00c55f0 con 0x7fc6d00c8950 2024-03-06T07:24:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d00c64f0 0x7fc6d014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d00c64f0 0x7fc6d014ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59036/0 (socket says 172.21.15.94:59036) 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 -- 172.21.15.94:0/2092997623 learned_addr learned my addr 172.21.15.94:0/2092997623 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d631e640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6d0152900 0x7fc6d01eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d531c640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d00ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 -- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 msgr2=0x7fc6d00ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d00ab700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 -- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6d0152900 msgr2=0x7fc6d01eabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6d0152900 0x7fc6d01eabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 -- 172.21.15.94:0/2092997623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00c2d10 con 0x7fc6d00c64f0 2024-03-06T07:24:23.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d5b1d640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d00c64f0 0x7fc6d014ca50 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00c400 tx=0x7fc6cc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6c67fc640 1 -- 172.21.15.94:0/2092997623 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc017070 con 0x7fc6d00c64f0 2024-03-06T07:24:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.424+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d00c2530 con 0x7fc6d00c64f0 2024-03-06T07:24:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.425+0000 7fc6c67fc640 1 -- 172.21.15.94:0/2092997623 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc005ce0 con 0x7fc6d00c64f0 2024-03-06T07:24:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.425+0000 7fc6c67fc640 1 -- 172.21.15.94:0/2092997623 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0072f0 con 0x7fc6d00c64f0 2024-03-06T07:24:23.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.425+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d00c2210 con 0x7fc6d00c64f0 2024-03-06T07:24:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.425+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d0153070 con 0x7fc6d00c64f0 2024-03-06T07:24:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.426+0000 7fc6c67fc640 1 -- 172.21.15.94:0/2092997623 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6cc01a070 con 0x7fc6d00c64f0 2024-03-06T07:24:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.427+0000 7fc6c67fc640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6b406b4d0 0x7fc6b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.427+0000 7fc6c67fc640 1 -- 172.21.15.94:0/2092997623 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc6cc0899b0 con 0x7fc6d00c64f0 2024-03-06T07:24:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.427+0000 7fc6d531c640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6b406b4d0 0x7fc6b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.427+0000 7fc6d531c640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6b406b4d0 0x7fc6b406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0035000 tx=0x7fc6c0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.435+0000 7fc6c67fc640 1 -- 172.21.15.94:0/2092997623 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6d0153070 con 0x7fc6d00c64f0 2024-03-06T07:24:23.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.692+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "11:0"} v 0) v1 -- 0x7fc6d004f330 con 0x7fc6d00c64f0 2024-03-06T07:24:23.694 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:24:23.692+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:23.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.694+0000 7fc6c67fc640 1 -- 172.21.15.94:0/2092997623 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "11:0"}]=0 failed mds gid 8772 v141) v1 ==== 98+0+0 (secure 0 0 0) 0x7fc6cc087e20 con 0x7fc6d00c64f0 2024-03-06T07:24:23.696 INFO:teuthology.orchestra.run.smithi094.stderr:failed mds gid 8772 2024-03-06T07:24:23.697 INFO:tasks.ceph.mds.c.smithi094.stderr: -1743> 2024-03-06T07:23:58.764+0000 7f1f60d54640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:24:23.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6b406b4d0 msgr2=0x7fc6b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6b406b4d0 0x7fc6b406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0035000 tx=0x7fc6c0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d00c64f0 msgr2=0x7fc6d014ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d00c64f0 0x7fc6d014ca50 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00c400 tx=0x7fc6cc00c800 comp rx=0 tx=0).stop 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 shutdown_connections 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc6b406b4d0 0x7fc6b406d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6d0152900 0x7fc6d01eabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6d00c8950 0x7fc6d00ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 --2- 172.21.15.94:0/2092997623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6d00c64f0 0x7fc6d014ca50 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.697+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 >> 172.21.15.94:0/2092997623 conn(0x7fc6d00ac860 msgr2=0x7fc6d0150b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.698+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 shutdown_connections 2024-03-06T07:24:23.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.698+0000 7fc6d7d81640 1 -- 172.21.15.94:0/2092997623 wait complete. 2024-03-06T07:24:23.701 INFO:tasks.ceph.mds.c.smithi094.stderr: -482> 2024-03-06T07:24:09.151+0000 7f1f5ad48640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:23.715 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:23.726 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:24:23.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.828+0000 7f122958c640 1 Processor -- start 2024-03-06T07:24:23.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.828+0000 7f122958c640 1 -- start start 2024-03-06T07:24:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f122958c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1224072720 0x7f12241564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f122958c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1224156a00 0x7f1224150290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f122958c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1224156f90 con 0x7f12241507d0 2024-03-06T07:24:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f122958c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12240c38d0 con 0x7f1224156a00 2024-03-06T07:24:23.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f122958c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1224157270 con 0x7f1224072720 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f12227fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1224156a00 0x7f1224150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f12237fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f12241507d0 0x7f12240c3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48528/0 (socket says 172.21.15.94:48528) 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.829+0000 7f12237fe640 1 -- 172.21.15.94:0/4063374513 learned_addr learned my addr 172.21.15.94:0/4063374513 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.830+0000 7f1222ffd640 1 --2- 172.21.15.94:0/4063374513 >> v2:172.21.15.145:3300/0 conn(0x7f1224072720 0x7f12241564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.830+0000 7f12227fc640 1 -- 172.21.15.94:0/4063374513 >> v2:172.21.15.145:3300/0 conn(0x7f1224072720 msgr2=0x7f12241564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.830+0000 7f12227fc640 1 --2- 172.21.15.94:0/4063374513 >> v2:172.21.15.145:3300/0 conn(0x7f1224072720 0x7f12241564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.830+0000 7f12227fc640 1 -- 172.21.15.94:0/4063374513 >> v1:172.21.15.94:6789/0 conn(0x7f12241507d0 legacy=0x7f12240c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.830+0000 7f1221ffb640 1 -- 172.21.15.94:0/4063374513 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415851729 0 0) 0x7f1224156f90 con 0x7f12241507d0 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.830+0000 7f12227fc640 1 -- 172.21.15.94:0/4063374513 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12240c3280 con 0x7f1224156a00 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.830+0000 7f12227fc640 1 --2- 172.21.15.94:0/4063374513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224156a00 0x7f1224150290 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f12240c9850 tx=0x7f121802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78b30abe82772e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f1221ffb640 1 -- 172.21.15.94:0/4063374513 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12180315a0 con 0x7f1224156a00 2024-03-06T07:24:23.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f1221ffb640 1 -- 172.21.15.94:0/4063374513 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1218037070 con 0x7f1224156a00 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f1221ffb640 1 -- 172.21.15.94:0/4063374513 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218031960 con 0x7f1224156a00 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 -- 172.21.15.94:0/4063374513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224156a00 msgr2=0x7f1224150290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 --2- 172.21.15.94:0/4063374513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224156a00 0x7f1224150290 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f12240c9850 tx=0x7f121802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 -- 172.21.15.94:0/4063374513 shutdown_connections 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 --2- 172.21.15.94:0/4063374513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224156a00 0x7f1224150290 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 --2- 172.21.15.94:0/4063374513 >> v2:172.21.15.145:3300/0 conn(0x7f1224072720 0x7f12241564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 -- 172.21.15.94:0/4063374513 >> 172.21.15.94:0/4063374513 conn(0x7f12240aa630 msgr2=0x7f12240aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 -- 172.21.15.94:0/4063374513 shutdown_connections 2024-03-06T07:24:23.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.831+0000 7f122958c640 1 -- 172.21.15.94:0/4063374513 wait complete. 2024-03-06T07:24:23.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.833+0000 7f122958c640 1 Processor -- start 2024-03-06T07:24:23.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.833+0000 7f122958c640 1 -- start start 2024-03-06T07:24:23.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f122958c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224072720 0x7f12241455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f122958c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12241507d0 0x7f12241e7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f122958c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1224156a00 0x7f12241e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f122958c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1224072b00 con 0x7f1224072720 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f122958c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122406f8f0 con 0x7f1224156a00 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f122958c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1224072c80 con 0x7f12241507d0 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224072720 0x7f12241455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224072720 0x7f12241455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59058/0 (socket says 172.21.15.94:59058) 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 -- 172.21.15.94:0/3780368801 learned_addr learned my addr 172.21.15.94:0/3780368801 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f12227fc640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12241507d0 0x7f12241e7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f12237fe640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1224156a00 0x7f12241e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 -- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12241507d0 msgr2=0x7f12241e7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12241507d0 0x7f12241e7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 -- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1224156a00 msgr2=0x7f12241e7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1224156a00 0x7f12241e7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f1222ffd640 1 -- 172.21.15.94:0/3780368801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12241560d0 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.834+0000 7f12227fc640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12241507d0 0x7f12241e7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.835+0000 7f1222ffd640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224072720 0x7f12241455b0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f121c00efc0 tx=0x7f121c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.835+0000 7f120b7fe640 1 -- 172.21.15.94:0/3780368801 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c018020 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.835+0000 7f120b7fe640 1 -- 172.21.15.94:0/3780368801 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c014480 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.835+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1224156390 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.835+0000 7f120b7fe640 1 -- 172.21.15.94:0/3780368801 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c014850 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.835+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12240c5970 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.836+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122404f330 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.836+0000 7f120b7fe640 1 -- 172.21.15.94:0/3780368801 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f121c005d10 con 0x7f1224072720 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.837+0000 7f120b7fe640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f121006b4d0 0x7f121006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.837+0000 7f120b7fe640 1 -- 172.21.15.94:0/3780368801 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f121c08b550 con 0x7f1224072720 2024-03-06T07:24:23.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.837+0000 7f12227fc640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f121006b4d0 0x7f121006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.837+0000 7f12227fc640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f121006b4d0 0x7f121006d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1218036000 tx=0x7f1218008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:23.845+0000 7f120b7fe640 1 -- 172.21.15.94:0/3780368801 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f121c022030 con 0x7f1224072720 2024-03-06T07:24:24.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.061+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f122414c310 con 0x7f1224072720 2024-03-06T07:24:24.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.061+0000 7f120b7fe640 1 -- 172.21.15.94:0/3780368801 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5478 (secure 0 0 0) 0x7f121c010040 con 0x7f1224072720 2024-03-06T07:24:24.063 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:24.063 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:23.697856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:replay","state_seq":1,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:19.614520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:24.063 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 142 2024-03-06T07:24:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f121006b4d0 msgr2=0x7f121006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f121006b4d0 0x7f121006d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1218036000 tx=0x7f1218008030 comp rx=0 tx=0).stop 2024-03-06T07:24:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224072720 msgr2=0x7f12241455b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224072720 0x7f12241455b0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f121c00efc0 tx=0x7f121c00c490 comp rx=0 tx=0).stop 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 shutdown_connections 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f121006b4d0 0x7f121006d990 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1224156a00 0x7f12241e7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12241507d0 0x7f12241e7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 --2- 172.21.15.94:0/3780368801 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1224072720 0x7f12241455b0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 >> 172.21.15.94:0/3780368801 conn(0x7f12240aa630 msgr2=0x7f1224148a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 shutdown_connections 2024-03-06T07:24:24.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:24.063+0000 7f122958c640 1 -- 172.21.15.94:0/3780368801 wait complete. 2024-03-06T07:24:24.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 142, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:03.229941+0000', 'modified': '2024-03-06T07:24:23.697856+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 154, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8155}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8155': {'gid': 8155, 'name': 'b', 'rank': 0, 'incarnation': 142, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:24:24.481 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8155:up:replay 2024-03-06T07:24:25.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.581+0000 7faebc2c9640 1 Processor -- start 2024-03-06T07:24:25.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.581+0000 7faebc2c9640 1 -- start start 2024-03-06T07:24:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faebc2c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faeb4159100 0x7faeb40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faebc2c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb40c7c60 con 0x7faeb40c7180 2024-03-06T07:24:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faebc2c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb40c8060 con 0x7faeb41527b0 2024-03-06T07:24:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faebc2c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40c8df0 con 0x7faeb4159100 2024-03-06T07:24:25.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeba866640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faeb4159100 0x7faeb40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeb9864640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7faeb40c7180 0x7faeb40c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48530/0 (socket says 172.21.15.94:48530) 2024-03-06T07:24:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeba866640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faeb4159100 0x7faeb40c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59070/0 (socket says 172.21.15.94:59070) 2024-03-06T07:24:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeb9864640 1 -- 172.21.15.94:0/1564430333 learned_addr learned my addr 172.21.15.94:0/1564430333 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeba866640 1 -- 172.21.15.94:0/1564430333 >> v1:172.21.15.145:6789/0 conn(0x7faeb41527b0 legacy=0x7faeb4155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeba866640 1 -- 172.21.15.94:0/1564430333 >> v1:172.21.15.94:6789/0 conn(0x7faeb40c7180 legacy=0x7faeb40c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeb9063640 1 -- 172.21.15.94:0/1564430333 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187545688 0 0) 0x7faea4001110 con 0x7faeb40c7180 2024-03-06T07:24:25.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.582+0000 7faeba866640 1 -- 172.21.15.94:0/1564430333 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40c76a0 con 0x7faeb4159100 2024-03-06T07:24:25.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.583+0000 7faeba866640 1 --2- 172.21.15.94:0/1564430333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb4159100 0x7faeb40c88b0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009d90 tx=0x7faeb002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee045ffb0a9b11b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.583+0000 7faeb9063640 1 -- 172.21.15.94:0/1564430333 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0039070 con 0x7faeb4159100 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.583+0000 7faeb9063640 1 -- 172.21.15.94:0/1564430333 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb002cc60 con 0x7faeb4159100 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.583+0000 7faeb9063640 1 -- 172.21.15.94:0/1564430333 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0034660 con 0x7faeb4159100 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.584+0000 7faebc2c9640 1 -- 172.21.15.94:0/1564430333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb4159100 msgr2=0x7faeb40c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.584+0000 7faebc2c9640 1 --2- 172.21.15.94:0/1564430333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb4159100 0x7faeb40c88b0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009d90 tx=0x7faeb002beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.584+0000 7faebc2c9640 1 -- 172.21.15.94:0/1564430333 shutdown_connections 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.584+0000 7faebc2c9640 1 --2- 172.21.15.94:0/1564430333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb4159100 0x7faeb40c88b0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.584+0000 7faebc2c9640 1 -- 172.21.15.94:0/1564430333 >> 172.21.15.94:0/1564430333 conn(0x7faeb40ac560 msgr2=0x7faeb40ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.584+0000 7faebc2c9640 1 -- 172.21.15.94:0/1564430333 shutdown_connections 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.584+0000 7faebc2c9640 1 -- 172.21.15.94:0/1564430333 wait complete. 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.585+0000 7faebc2c9640 1 Processor -- start 2024-03-06T07:24:25.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.585+0000 7faebc2c9640 1 -- start start 2024-03-06T07:24:25.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faeb40c7180 0x7faeb414ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb41527b0 0x7faeb41dedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faeb4159100 0x7faeb414bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4155600 con 0x7faeb41527b0 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4155320 con 0x7faeb40c7180 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4152b90 con 0x7faeb4159100 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb41527b0 0x7faeb41dedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb41527b0 0x7faeb41dedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59078/0 (socket says 172.21.15.94:59078) 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 -- 172.21.15.94:0/3331460442 learned_addr learned my addr 172.21.15.94:0/3331460442 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeba866640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faeb4159100 0x7faeb414bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeba065640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faeb40c7180 0x7faeb414ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 -- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faeb4159100 msgr2=0x7faeb414bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faeb4159100 0x7faeb414bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 -- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faeb40c7180 msgr2=0x7faeb414ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faeb40c7180 0x7faeb414ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 -- 172.21.15.94:0/3331460442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40c3960 con 0x7faeb41527b0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeb9864640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb41527b0 0x7faeb41dedc0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7faea400c780 tx=0x7faea400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeaaffd640 1 -- 172.21.15.94:0/3331460442 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4018070 con 0x7faeb41527b0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeaaffd640 1 -- 172.21.15.94:0/3331460442 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea40072d0 con 0x7faeb41527b0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faeaaffd640 1 -- 172.21.15.94:0/3331460442 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea40047e0 con 0x7faeb41527b0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb41ee2a0 con 0x7faeb41527b0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.586+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb41ee860 con 0x7faeb41527b0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.587+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb40c3b30 con 0x7faeb41527b0 2024-03-06T07:24:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.588+0000 7faeaaffd640 1 -- 172.21.15.94:0/3331460442 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faea401b070 con 0x7faeb41527b0 2024-03-06T07:24:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.588+0000 7faeaaffd640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae9806b4d0 0x7fae9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.588+0000 7faeaaffd640 1 -- 172.21.15.94:0/3331460442 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7faea4089960 con 0x7faeb41527b0 2024-03-06T07:24:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.589+0000 7faeba065640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae9806b4d0 0x7fae9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.589+0000 7faeba065640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae9806b4d0 0x7fae9806d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7faeb40bdf40 tx=0x7faeac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.593+0000 7faeaaffd640 1 -- 172.21.15.94:0/3331460442 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faeb40c7c60 con 0x7faeb41527b0 2024-03-06T07:24:25.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.805+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeb404f330 con 0x7faeb41527b0 2024-03-06T07:24:25.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.805+0000 7faeaaffd640 1 -- 172.21.15.94:0/3331460442 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+6270 (secure 0 0 0) 0x7faea4058330 con 0x7faeb41527b0 2024-03-06T07:24:25.807 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:25.807 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":144,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:24.705391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:25.808 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 144 2024-03-06T07:24:25.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae9806b4d0 msgr2=0x7fae9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae9806b4d0 0x7fae9806d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7faeb40bdf40 tx=0x7faeac009040 comp rx=0 tx=0).stop 2024-03-06T07:24:25.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb41527b0 msgr2=0x7faeb41dedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb41527b0 0x7faeb41dedc0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7faea400c780 tx=0x7faea400cc40 comp rx=0 tx=0).stop 2024-03-06T07:24:25.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 shutdown_connections 2024-03-06T07:24:25.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae9806b4d0 0x7fae9806d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faeb4159100 0x7faeb414bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faeb41527b0 0x7faeb41dedc0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 --2- 172.21.15.94:0/3331460442 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faeb40c7180 0x7faeb414ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 >> 172.21.15.94:0/3331460442 conn(0x7faeb40ac560 msgr2=0x7faeb40c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 shutdown_connections 2024-03-06T07:24:25.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:25.807+0000 7faebc2c9640 1 -- 172.21.15.94:0/3331460442 wait complete. 2024-03-06T07:24:26.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:03.229941+0000', 'modified': '2024-03-06T07:24:24.705391+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 154, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8155}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8155': {'gid': 8155, 'name': 'b', 'rank': 0, 'incarnation': 142, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:24:26.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8155:up:rejoin 2024-03-06T07:24:27.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:27.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.323+0000 7f560c156640 1 Processor -- start 2024-03-06T07:24:27.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.324+0000 7f560c156640 1 -- start start 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.324+0000 7f560c156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.324+0000 7f560c156640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56040c73d0 con 0x7f560414f360 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.324+0000 7f560c156640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56040be2e0 con 0x7f56040c88c0 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.324+0000 7f560c156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5604159700 con 0x7f5604159270 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f5609ef2640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f560414f360 0x7f56040c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48538/0 (socket says 172.21.15.94:48538) 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f5609ef2640 1 -- 172.21.15.94:0/4157587629 learned_addr learned my addr 172.21.15.94:0/4157587629 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f560a6f3640 1 --2- 172.21.15.94:0/4157587629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f5608ef0640 1 -- 172.21.15.94:0/4157587629 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133276859 0 0) 0x7f56040c73d0 con 0x7f560414f360 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f5608ef0640 1 -- 172.21.15.94:0/4157587629 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ec003520 con 0x7f560414f360 2024-03-06T07:24:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f5608ef0640 1 -- 172.21.15.94:0/4157587629 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956105928 0 0) 0x7f56040be2e0 con 0x7f56040c88c0 2024-03-06T07:24:27.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f560a6f3640 1 -- 172.21.15.94:0/4157587629 >> v1:172.21.15.145:6789/0 conn(0x7f56040c88c0 legacy=0x7f5604158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f560a6f3640 1 -- 172.21.15.94:0/4157587629 >> v1:172.21.15.94:6789/0 conn(0x7f560414f360 legacy=0x7f56040c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.325+0000 7f560a6f3640 1 -- 172.21.15.94:0/4157587629 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56040c7670 con 0x7f5604159270 2024-03-06T07:24:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560a6f3640 1 --2- 172.21.15.94:0/4157587629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604155c10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f55f40097d0 tx=0x7f55f402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b1b7b9f864f73f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f5608ef0640 1 -- 172.21.15.94:0/4157587629 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4035020 con 0x7f5604159270 2024-03-06T07:24:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f5608ef0640 1 -- 172.21.15.94:0/4157587629 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f4031810 con 0x7f5604159270 2024-03-06T07:24:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f5608ef0640 1 -- 172.21.15.94:0/4157587629 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4031be0 con 0x7f5604159270 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560c156640 1 -- 172.21.15.94:0/4157587629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 msgr2=0x7f5604155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560c156640 1 --2- 172.21.15.94:0/4157587629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604155c10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f55f40097d0 tx=0x7f55f402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560c156640 1 -- 172.21.15.94:0/4157587629 shutdown_connections 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560c156640 1 --2- 172.21.15.94:0/4157587629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604155c10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560c156640 1 -- 172.21.15.94:0/4157587629 >> 172.21.15.94:0/4157587629 conn(0x7f56040ac570 msgr2=0x7f56040ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560c156640 1 -- 172.21.15.94:0/4157587629 shutdown_connections 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.326+0000 7f560c156640 1 -- 172.21.15.94:0/4157587629 wait complete. 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.327+0000 7f560c156640 1 Processor -- start 2024-03-06T07:24:27.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560c156640 1 -- start start 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560c156640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56040c88c0 0x7f560414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560c156640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f560414f360 0x7f5604157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560c156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560c156640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56040c7c30 con 0x7f56040c88c0 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560c156640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56040c5a50 con 0x7f560414f360 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560c156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56040c5bd0 con 0x7f5604159270 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f5609ef2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56040c88c0 0x7f560414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f5609ef2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56040c88c0 0x7f560414df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59084/0 (socket says 172.21.15.94:59084) 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f5609ef2640 1 -- 172.21.15.94:0/3501578257 learned_addr learned my addr 172.21.15.94:0/3501578257 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f56096f1640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f560414f360 0x7f5604157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.328+0000 7f560a6f3640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f5609ef2640 1 -- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 msgr2=0x7f5604148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f5609ef2640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f5609ef2640 1 -- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f560414f360 msgr2=0x7f5604157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f5609ef2640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f560414f360 0x7f5604157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f5609ef2640 1 -- 172.21.15.94:0/3501578257 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604156d90 con 0x7f56040c88c0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f560a6f3640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f5609ef2640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56040c88c0 0x7f560414df70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c430 tx=0x7f55fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f56027fc640 1 -- 172.21.15.94:0/3501578257 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc017070 con 0x7f56040c88c0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f56027fc640 1 -- 172.21.15.94:0/3501578257 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc005ce0 con 0x7f56040c88c0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5604156ab0 con 0x7f56040c88c0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f56027fc640 1 -- 172.21.15.94:0/3501578257 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc0072f0 con 0x7f56040c88c0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.329+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56041eed50 con 0x7f56040c88c0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.330+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560404f330 con 0x7f56040c88c0 2024-03-06T07:24:27.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.330+0000 7f56027fc640 1 -- 172.21.15.94:0/3501578257 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55fc01a070 con 0x7f56040c88c0 2024-03-06T07:24:27.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.331+0000 7f56027fc640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55f006b4d0 0x7f55f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:27.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.331+0000 7f56027fc640 1 -- 172.21.15.94:0/3501578257 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f55fc089c80 con 0x7f56040c88c0 2024-03-06T07:24:27.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.331+0000 7f56096f1640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55f006b4d0 0x7f55f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:27.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.331+0000 7f56096f1640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55f006b4d0 0x7f55f006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5604156a40 tx=0x7f55f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.335+0000 7f56027fc640 1 -- 172.21.15.94:0/3501578257 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55fc057e60 con 0x7f56040c88c0 2024-03-06T07:24:27.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.547+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56040c8d50 con 0x7f56040c88c0 2024-03-06T07:24:27.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.548+0000 7f56027fc640 1 -- 172.21.15.94:0/3501578257 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6278 (secure 0 0 0) 0x7f55fc05bf00 con 0x7f56040c88c0 2024-03-06T07:24:27.552 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:27.553 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":145,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:26.705762+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8155,"qdb_cluster":[8155]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:27.553 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 145 2024-03-06T07:24:27.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55f006b4d0 msgr2=0x7f55f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55f006b4d0 0x7f55f006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5604156a40 tx=0x7f55f8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56040c88c0 msgr2=0x7f560414df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56040c88c0 0x7f560414df70 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c430 tx=0x7f55fc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 shutdown_connections 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55f006b4d0 0x7f55f006d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5604159270 0x7f5604148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f560414f360 0x7f5604157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 --2- 172.21.15.94:0/3501578257 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56040c88c0 0x7f560414df70 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 >> 172.21.15.94:0/3501578257 conn(0x7f56040ac570 msgr2=0x7f56040ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 shutdown_connections 2024-03-06T07:24:27.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:27.550+0000 7f560c156640 1 -- 172.21.15.94:0/3501578257 wait complete. 2024-03-06T07:24:27.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 145, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:03.229941+0000', 'modified': '2024-03-06T07:24:26.705762+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 154, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8155}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8155': {'gid': 8155, 'name': 'b', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8155, 'qdb_cluster': [8155]} 2024-03-06T07:24:27.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:27.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:24:28.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.065+0000 7f4ff444e640 1 Processor -- start 2024-03-06T07:24:28.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.065+0000 7f4ff444e640 1 -- start start 2024-03-06T07:24:28.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.066+0000 7f4ff444e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4fec0c69f0 0x7f4fec0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.066+0000 7f4ff444e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4fec0c8950 0x7f4fec0acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.066+0000 7f4ff444e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.066+0000 7f4ff444e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec155440 con 0x7f4fec0c8950 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.066+0000 7f4ff444e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec155130 con 0x7f4fec0c69f0 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.066+0000 7f4ff444e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0c5d50 con 0x7f4fec0ad2b0 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.066+0000 7f4ff19e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4fec0c8950 0x7f4fec0acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff19e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4fec0c8950 0x7f4fec0acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59086/0 (socket says 172.21.15.94:59086) 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff19e9640 1 -- 172.21.15.94:0/1494127971 learned_addr learned my addr 172.21.15.94:0/1494127971 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/1494127971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/1494127971 >> v2:172.21.15.145:3300/0 conn(0x7f4fec0c69f0 0x7f4fec0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff19e9640 1 -- 172.21.15.94:0/1494127971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 msgr2=0x7f4fec1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/1494127971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff19e9640 1 -- 172.21.15.94:0/1494127971 >> v2:172.21.15.145:3300/0 conn(0x7f4fec0c69f0 msgr2=0x7f4fec0c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/1494127971 >> v2:172.21.15.145:3300/0 conn(0x7f4fec0c69f0 0x7f4fec0c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.067+0000 7f4ff19e9640 1 -- 172.21.15.94:0/1494127971 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0c6030 con 0x7f4fec0c8950 2024-03-06T07:24:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/1494127971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c8950 0x7f4fec0acd70 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0aa430 tx=0x7f4fe802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff70ea1e9eaf985 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff11e8640 1 -- 172.21.15.94:0/1494127971 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe802e070 con 0x7f4fec0c8950 2024-03-06T07:24:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff11e8640 1 -- 172.21.15.94:0/1494127971 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8037070 con 0x7f4fec0c8950 2024-03-06T07:24:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff11e8640 1 -- 172.21.15.94:0/1494127971 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8031a00 con 0x7f4fec0c8950 2024-03-06T07:24:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 -- 172.21.15.94:0/1494127971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c8950 msgr2=0x7f4fec0acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 --2- 172.21.15.94:0/1494127971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c8950 0x7f4fec0acd70 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0aa430 tx=0x7f4fe802b880 comp rx=0 tx=0).stop 2024-03-06T07:24:28.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 -- 172.21.15.94:0/1494127971 shutdown_connections 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 --2- 172.21.15.94:0/1494127971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 --2- 172.21.15.94:0/1494127971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c8950 0x7f4fec0acd70 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 --2- 172.21.15.94:0/1494127971 >> v2:172.21.15.145:3300/0 conn(0x7f4fec0c69f0 0x7f4fec0c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 -- 172.21.15.94:0/1494127971 >> 172.21.15.94:0/1494127971 conn(0x7f4fec0cae70 msgr2=0x7f4fec0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 -- 172.21.15.94:0/1494127971 shutdown_connections 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.068+0000 7f4ff444e640 1 -- 172.21.15.94:0/1494127971 wait complete. 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.069+0000 7f4ff444e640 1 Processor -- start 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.069+0000 7f4ff444e640 1 -- start start 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff444e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec14a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff444e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c69f0 0x7f4fec152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff444e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fec0c8950 0x7f4fec1534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff444e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec152ba0 con 0x7f4fec0c69f0 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff444e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec152a20 con 0x7f4fec0c8950 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff444e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0c58e0 con 0x7f4fec0ad2b0 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff19e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c69f0 0x7f4fec152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff19e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c69f0 0x7f4fec152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59094/0 (socket says 172.21.15.94:59094) 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff19e9640 1 -- 172.21.15.94:0/2972944619 learned_addr learned my addr 172.21.15.94:0/2972944619 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fec0c8950 0x7f4fec1534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.070+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec14a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff19e9640 1 -- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 msgr2=0x7f4fec14a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec14a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff19e9640 1 -- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fec0c8950 msgr2=0x7f4fec1534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fec0c8950 0x7f4fec1534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff19e9640 1 -- 172.21.15.94:0/2972944619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec1eea00 con 0x7f4fec0c69f0 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c69f0 0x7f4fec152fa0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8000c00 tx=0x7f4fe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4fdeffd640 1 -- 172.21.15.94:0/2972944619 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8032670 con 0x7f4fec0c69f0 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4fdeffd640 1 -- 172.21.15.94:0/2972944619 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8032810 con 0x7f4fec0c69f0 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec1eecc0 con 0x7f4fec0c69f0 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4fdeffd640 1 -- 172.21.15.94:0/2972944619 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8032c00 con 0x7f4fec0c69f0 2024-03-06T07:24:28.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.071+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec1ef280 con 0x7f4fec0c69f0 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.072+0000 7f4fdeffd640 1 -- 172.21.15.94:0/2972944619 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fe804c060 con 0x7f4fec0c69f0 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.073+0000 7f4fdeffd640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.073+0000 7f4fdeffd640 1 -- 172.21.15.94:0/2972944619 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f4fe80abe10 con 0x7f4fec0c69f0 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.073+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.074+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f4fec152f50 tx=0x7f4fe0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.074+0000 7f4ff703c740 1 Processor -- start 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.074+0000 7f4ff703c740 1 -- start start 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x5569953efa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x5569953eff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f04b0 0x5569953ec680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff703c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569953566b0 con 0x5569953f2330 2024-03-06T07:24:28.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569952ffa60 con 0x5569953f1f50 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556995352750 con 0x5569953f04b0 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x5569953eff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff21ea640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x5569953efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x5569953eff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59110/0 (socket says 172.21.15.94:59110) 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff21ea640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x5569953efa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:34088/0 (socket says 172.21.15.94:34088) 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 -- 172.21.15.94:0/4005373176 learned_addr learned my addr 172.21.15.94:0/4005373176 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f04b0 0x5569953ec680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 -- 172.21.15.94:0/4005373176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f04b0 msgr2=0x5569953ec680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f04b0 0x5569953ec680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 -- 172.21.15.94:0/4005373176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 msgr2=0x5569953efa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x5569953efa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 -- 172.21.15.94:0/4005373176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55699531df30 con 0x5569953f2330 2024-03-06T07:24:28.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x5569953eff70 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80a9e90 tx=0x7f4fe8079bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f72eaabe54e3ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.075+0000 7f4fcffff640 1 -- 172.21.15.94:0/4005373176 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe80a7020 con 0x5569953f2330 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4fcffff640 1 -- 172.21.15.94:0/4005373176 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8037040 con 0x5569953f2330 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4fcffff640 1 -- 172.21.15.94:0/4005373176 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe80828b0 con 0x5569953f2330 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 -- 172.21.15.94:0/4005373176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 msgr2=0x5569953eff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x5569953eff70 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80a9e90 tx=0x7f4fe8079bb0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 -- 172.21.15.94:0/4005373176 shutdown_connections 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f04b0 0x5569953ec680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x5569953eff70 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 --2- 172.21.15.94:0/4005373176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x5569953efa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 -- 172.21.15.94:0/4005373176 >> 172.21.15.94:0/4005373176 conn(0x5569953ee7d0 msgr2=0x5569954133c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 -- 172.21.15.94:0/4005373176 shutdown_connections 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.076+0000 7f4ff703c740 1 -- 172.21.15.94:0/4005373176 wait complete. 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.077+0000 7f4ff703c740 1 -- start start 2024-03-06T07:24:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.077+0000 7f4ff703c740 1 Processor -- start 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x556995442a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x55699543e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569954171f0 0x556995426800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff703c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x556995352750 con 0x5569953f2330 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556995312ea0 con 0x5569953f1f50 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5569953566b0 con 0x5569954171f0 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff703c740 10 client.0 _ll_register_callbacks cb 0x556995312870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x55699543e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x55699543e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59126/0 (socket says 172.21.15.94:59126) 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 -- 172.21.15.94:0/3975346547 learned_addr learned my addr 172.21.15.94:0/3975346547 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff21ea640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x556995442a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569954171f0 0x556995426800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 -- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569954171f0 msgr2=0x556995426800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569954171f0 0x556995426800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 -- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 msgr2=0x556995442a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x556995442a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 -- 172.21.15.94:0/3975346547 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556995313270 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x55699543e9c0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80a9ee0 tx=0x7f4fe804b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28c0e3707c5a95b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.078+0000 7f4fcd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/3975346547 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe809d060 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/3975346547 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f4fe8082420 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4fe8037040 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/3975346547 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fe80a3050 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 10 client.9114 fetch_fsmap learned FSMap version 146 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x55699506b2d0 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/3975346547 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8037040 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/3975346547 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 146) v1 ==== 3929+0+0 (secure 0 0 0) 0x7f4fe80796a0 con 0x5569953f2330 2024-03-06T07:24:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 10 client.9114 fetch_fsmap finished waiting for FSMap version 146 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 10 client.9114 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 20 client.9114 populate_metadata read hostname 'smithi094' 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x5569954149f0 0x556995414dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 4 client.9114 mds_command: new command op to 8155 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 --> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556995363bf0 con 0x5569954149f0 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.079+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x5569954149f0 0x556995414dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.080+0000 7f4fcd7fa640 10 client.9114 ms_handle_connect on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.080+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x5569954149f0 0x556995414dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c97390515e44cb30 server_cookie=841b044c8304b6aa in_seq=0 out_seq=0 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/3975346547 <== mds.0 v2:172.21.15.145:6835/486493869 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556995363bf0 con 0x5569954149f0 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4fcd7fa640 10 client.9114 handle_command_reply: tid=0 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 client.9114 shutdown 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 msgr2=0x55699543e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x55699543e9c0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80a9ee0 tx=0x7f4fe804b000 comp rx=0 tx=0).stop 2024-03-06T07:24:28.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 shutdown_connections 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569954171f0 0x556995426800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x5569954149f0 0x556995414dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f2330 0x55699543e9c0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4fcd7fa640 0 client.9114 ms_handle_reset on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 --2- 172.21.15.94:0/3975346547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f1f50 0x556995442a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 >> 172.21.15.94:0/3975346547 conn(0x5569953f04b0 msgr2=0x556995413c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 shutdown_connections 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 1 -- 172.21.15.94:0/3975346547 wait complete. 2024-03-06T07:24:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.081+0000 7f4ff703c740 20 client.9114 trim_cache size 0 max 16384 2024-03-06T07:24:28.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.097+0000 7f4ff703c740 1 Processor -- start 2024-03-06T07:24:28.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.097+0000 7f4ff703c740 1 -- start start 2024-03-06T07:24:28.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f1f50 0x556995414dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f2330 0x55699543e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569954171f0 0x55699543ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff703c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1ff030 con 0x5569954171f0 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1fcc90 con 0x5569953f2330 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec1fd260 con 0x5569953f1f50 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff29eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569954171f0 0x55699543ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff21ea640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f1f50 0x556995414dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff29eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569954171f0 0x55699543ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59140/0 (socket says 172.21.15.94:59140) 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff29eb640 1 -- 172.21.15.94:0/2466960430 learned_addr learned my addr 172.21.15.94:0/2466960430 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f2330 0x55699543e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.098+0000 7f4ff29eb640 1 -- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f1f50 msgr2=0x556995414dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f1f50 0x556995414dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff29eb640 1 -- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f2330 msgr2=0x55699543e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f2330 0x55699543e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff29eb640 1 -- 172.21.15.94:0/2466960430 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569951356b0 con 0x5569954171f0 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f1f50 0x556995414dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f2330 0x55699543e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569954171f0 0x55699543ee80 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40176e0 tx=0x7f4fe400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3700d71756f19a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4fcf7fe640 1 -- 172.21.15.94:0/2466960430 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4025070 con 0x5569954171f0 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4fcf7fe640 1 -- 172.21.15.94:0/2466960430 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4015a10 con 0x5569954171f0 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4fcf7fe640 1 -- 172.21.15.94:0/2466960430 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4015de0 con 0x5569954171f0 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 -- 172.21.15.94:0/2466960430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569954171f0 msgr2=0x55699543ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569954171f0 0x55699543ee80 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40176e0 tx=0x7f4fe400b040 comp rx=0 tx=0).stop 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 -- 172.21.15.94:0/2466960430 shutdown_connections 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569954171f0 0x55699543ee80 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569953f2330 0x55699543e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2466960430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f1f50 0x556995414dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 -- 172.21.15.94:0/2466960430 >> 172.21.15.94:0/2466960430 conn(0x5569953f04b0 msgr2=0x556995412d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 -- 172.21.15.94:0/2466960430 shutdown_connections 2024-03-06T07:24:28.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.099+0000 7f4ff703c740 1 -- 172.21.15.94:0/2466960430 wait complete. 2024-03-06T07:24:28.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.100+0000 7f4ff703c740 1 -- start start 2024-03-06T07:24:28.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.100+0000 7f4ff703c740 1 Processor -- start 2024-03-06T07:24:28.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f1f50 0x55699540fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f2330 0x55699549e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569954171f0 0x55699549b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x556995415860 con 0x5569953f1f50 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569952ffa60 con 0x5569954171f0 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556995416d80 con 0x5569953f2330 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 10 client.0 _ll_register_callbacks cb 0x7f4fe807a0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f1f50 0x55699540fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f1f50 0x55699540fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59144/0 (socket says 172.21.15.94:59144) 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 -- 172.21.15.94:0/2188104980 learned_addr learned my addr 172.21.15.94:0/2188104980 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff29eb640 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569954171f0 0x55699549b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f2330 0x55699549e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 -- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f2330 msgr2=0x55699549e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f2330 0x55699549e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 -- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569954171f0 msgr2=0x55699549b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569954171f0 0x55699549b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 -- 172.21.15.94:0/2188104980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556995414720 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff21ea640 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f1f50 0x55699540fda0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00096a0 tx=0x7f4fe0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732bb2fb2739ebef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4fcd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/2188104980 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0031030 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/2188104980 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f4fe000f040 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.101+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4fe4015a10 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/2188104980 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fe003d050 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 10 client.9120 fetch_fsmap learned FSMap version 146 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x556995415590 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/2188104980 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4015a10 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/2188104980 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 146) v1 ==== 3929+0+0 (secure 0 0 0) 0x7f4fe0040020 con 0x5569953f1f50 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 10 client.9120 fetch_fsmap finished waiting for FSMap version 146 2024-03-06T07:24:28.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 10 client.9120 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 20 client.9120 populate_metadata read hostname 'smithi094' 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x55699543ea10 0x55699543edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 4 client.9120 mds_command: new command op to 8155 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 --> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55699535df40 con 0x55699543ea10 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x55699543ea10 0x55699543edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4ff19e9640 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x55699543ea10 0x55699543edf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3edae56a2f0407 server_cookie=dace3c0a485a2b77 in_seq=0 out_seq=0 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.102+0000 7f4fcd7fa640 10 client.9120 ms_handle_connect on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4fcd7fa640 1 -- 172.21.15.94:0/2188104980 <== mds.0 v2:172.21.15.145:6835/486493869 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55699535df40 con 0x55699543ea10 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4fcd7fa640 10 client.9120 handle_command_reply: tid=0 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 client.9120 shutdown 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f1f50 msgr2=0x55699540fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f1f50 0x55699540fda0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00096a0 tx=0x7f4fe0004730 comp rx=0 tx=0).stop 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 shutdown_connections 2024-03-06T07:24:28.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5569954171f0 0x55699549b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5569953f2330 0x55699549e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x55699543ea10 0x55699543edf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 --2- 172.21.15.94:0/2188104980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5569953f1f50 0x55699540fda0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4fcd7fa640 0 client.9120 ms_handle_reset on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 >> 172.21.15.94:0/2188104980 conn(0x5569953f04b0 msgr2=0x556995412d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 shutdown_connections 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 1 -- 172.21.15.94:0/2188104980 wait complete. 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff703c740 20 client.9120 trim_cache size 0 max 16384 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:24:28.111 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 8155, 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.403243121, 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 145, 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 154, 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 154, 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 29.329128942000001 2024-03-06T07:24:28.112 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:24:28.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4fd006b4d0 msgr2=0x7f4fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff444e640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f4fec152f50 tx=0x7f4fe0009190 comp rx=0 tx=0).stop 2024-03-06T07:24:28.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c69f0 msgr2=0x7f4fec152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:28.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.103+0000 7f4ff444e640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c69f0 0x7f4fec152fa0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8000c00 tx=0x7f4fe80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 shutdown_connections 2024-03-06T07:24:28.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fec0c8950 0x7f4fec1534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fec0c69f0 0x7f4fec152fa0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 --2- 172.21.15.94:0/2972944619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fec0ad2b0 0x7f4fec14a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:28.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 >> 172.21.15.94:0/2972944619 conn(0x7f4fec0cae70 msgr2=0x7f4fec1515c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:28.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 shutdown_connections 2024-03-06T07:24:28.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:28.104+0000 7f4ff444e640 1 -- 172.21.15.94:0/2972944619 wait complete. 2024-03-06T07:24:28.116 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 145, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:03.229941+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 145, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8155": { "addr": "172.21.15.145:6837/486493869", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 486493869, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 486493869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8155, "incarnation": 142, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 154, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-06T07:24:26.705762+0000", "qdb_cluster": [ 8155 ], "qdb_leader": 8155, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8155 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:16.636350+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8769": { "addr": "172.21.15.94:6834/1222126706", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 1222126706, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 1222126706, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8769, "incarnation": 136, "join_fscid": 12, "name": "a", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-06T07:24:25.285179+0000", "qdb_cluster": [ 8769 ], "qdb_leader": 8769, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8769 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/1521427660", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1521427660, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1521427660, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 132, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8152, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.94:6835/3923835266", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3923835266, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3923835266, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 143, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9078, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:28.117 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 12, 'name': 'd'}] 2024-03-06T07:24:28.117 INFO:tasks.cephfs.test_failover:current = [{'gid': 8769, 'name': 'a', 'rank': 0, 'incarnation': 136, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.94:6834/1222126706', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 1222126706}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 1222126706}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8155, 'name': 'b', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 9078, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3923835266', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3923835266}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3923835266}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 143}, {'gid': 8152, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.145:6836/1521427660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1521427660}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1521427660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 132}] 2024-03-06T07:24:28.117 DEBUG:tasks.cephfs.test_failover:12 != -1 2024-03-06T07:24:28.118 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:24:29.154 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:24:29.153+0000 7ff6fea1b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:33.119 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:33.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc336163640 1 Processor -- start 2024-03-06T07:24:33.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc336163640 1 -- start start 2024-03-06T07:24:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc336163640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc3300c88c0 0x7fc330155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc336163640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3300c73d0 con 0x7fc33014f360 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc336163640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3300be2e0 con 0x7fc330158b60 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc336163640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc330159700 con 0x7fc3300c88c0 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc32f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc33014f360 0x7fc3300c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51822/0 (socket says 172.21.15.94:51822) 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.219+0000 7fc32f7fe640 1 -- 172.21.15.94:0/26359713 learned_addr learned my addr 172.21.15.94:0/26359713 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.220+0000 7fc327fff640 1 --2- 172.21.15.94:0/26359713 >> v2:172.21.15.145:3300/0 conn(0x7fc3300c88c0 0x7fc330155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.220+0000 7fc32effd640 1 -- 172.21.15.94:0/26359713 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868103809 0 0) 0x7fc3300c73d0 con 0x7fc33014f360 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.220+0000 7fc32effd640 1 -- 172.21.15.94:0/26359713 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c003520 con 0x7fc33014f360 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.220+0000 7fc327fff640 1 -- 172.21.15.94:0/26359713 >> v1:172.21.15.145:6789/0 conn(0x7fc330158b60 legacy=0x7fc330158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.220+0000 7fc327fff640 1 -- 172.21.15.94:0/26359713 >> v1:172.21.15.94:6789/0 conn(0x7fc33014f360 legacy=0x7fc3300c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.220+0000 7fc327fff640 1 -- 172.21.15.94:0/26359713 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3300c7670 con 0x7fc3300c88c0 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc327fff640 1 --2- 172.21.15.94:0/26359713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3300c88c0 0x7fc330155c10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc3180097a0 tx=0x7fc31802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cfcb322b0b9e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc32effd640 1 -- 172.21.15.94:0/26359713 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318035020 con 0x7fc3300c88c0 2024-03-06T07:24:33.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc32effd640 1 -- 172.21.15.94:0/26359713 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318031830 con 0x7fc3300c88c0 2024-03-06T07:24:33.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc32effd640 1 -- 172.21.15.94:0/26359713 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318031be0 con 0x7fc3300c88c0 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc336163640 1 -- 172.21.15.94:0/26359713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3300c88c0 msgr2=0x7fc330155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc336163640 1 --2- 172.21.15.94:0/26359713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3300c88c0 0x7fc330155c10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc3180097a0 tx=0x7fc31802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc336163640 1 -- 172.21.15.94:0/26359713 shutdown_connections 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc336163640 1 --2- 172.21.15.94:0/26359713 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3300c88c0 0x7fc330155c10 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc336163640 1 -- 172.21.15.94:0/26359713 >> 172.21.15.94:0/26359713 conn(0x7fc3300ac570 msgr2=0x7fc3300ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc336163640 1 -- 172.21.15.94:0/26359713 shutdown_connections 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.221+0000 7fc336163640 1 -- 172.21.15.94:0/26359713 wait complete. 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.223+0000 7fc336163640 1 Processor -- start 2024-03-06T07:24:33.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.223+0000 7fc336163640 1 -- start start 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc336163640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3300c88c0 0x7fc3301eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc336163640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc33014f360 0x7fc330157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc336163640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc330158b60 0x7fc33014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc336163640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300c7c30 con 0x7fc3300c88c0 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc336163640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300c5a50 con 0x7fc33014f360 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc336163640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3300c5bd0 con 0x7fc330158b60 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc32f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3300c88c0 0x7fc3301eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc32f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3300c88c0 0x7fc3301eb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52858/0 (socket says 172.21.15.94:52858) 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc32f7fe640 1 -- 172.21.15.94:0/595678088 learned_addr learned my addr 172.21.15.94:0/595678088 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc327fff640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc33014f360 0x7fc330157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.224+0000 7fc32ffff640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc330158b60 0x7fc33014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32f7fe640 1 -- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc330158b60 msgr2=0x7fc33014b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32f7fe640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc330158b60 0x7fc33014b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32f7fe640 1 -- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc33014f360 msgr2=0x7fc330157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32f7fe640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc33014f360 0x7fc330157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32f7fe640 1 -- 172.21.15.94:0/595678088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3300c1560 con 0x7fc3300c88c0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32f7fe640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3300c88c0 0x7fc3301eb370 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c530 tx=0x7fc32000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32cff9640 1 -- 172.21.15.94:0/595678088 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320017070 con 0x7fc3300c88c0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32cff9640 1 -- 172.21.15.94:0/595678088 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc320005ce0 con 0x7fc3300c88c0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3300c0fe0 con 0x7fc3300c88c0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc32cff9640 1 -- 172.21.15.94:0/595678088 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3200072b0 con 0x7fc3300c88c0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.225+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3300c1280 con 0x7fc3300c88c0 2024-03-06T07:24:33.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.226+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33004f330 con 0x7fc3300c88c0 2024-03-06T07:24:33.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.227+0000 7fc32cff9640 1 -- 172.21.15.94:0/595678088 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc32001a070 con 0x7fc3300c88c0 2024-03-06T07:24:33.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.227+0000 7fc32cff9640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.227+0000 7fc32cff9640 1 -- 172.21.15.94:0/595678088 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fc320089a50 con 0x7fc3300c88c0 2024-03-06T07:24:33.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.227+0000 7fc327fff640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.228+0000 7fc327fff640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc31006b4d0 0x7fc31006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc318036000 tx=0x7fc31803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.231+0000 7fc32cff9640 1 -- 172.21.15.94:0/595678088 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc320057e90 con 0x7fc3300c88c0 2024-03-06T07:24:33.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.448+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc33006cda0 con 0x7fc3300c88c0 2024-03-06T07:24:33.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.449+0000 7fc32cff9640 1 -- 172.21.15.94:0/595678088 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+6278 (secure 0 0 0) 0x7fc32005bf30 con 0x7fc3300c88c0 2024-03-06T07:24:33.451 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:33.451 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":146,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":146},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:26.705762+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8155,"qdb_cluster":[8155]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:33.451 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 146 2024-03-06T07:24:33.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc31006b4d0 msgr2=0x7fc31006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc31006b4d0 0x7fc31006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc318036000 tx=0x7fc31803d040 comp rx=0 tx=0).stop 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3300c88c0 msgr2=0x7fc3301eb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3300c88c0 0x7fc3301eb370 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c530 tx=0x7fc32000c9f0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 shutdown_connections 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc330158b60 0x7fc33014b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc33014f360 0x7fc330157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 --2- 172.21.15.94:0/595678088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3300c88c0 0x7fc3301eb370 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 >> 172.21.15.94:0/595678088 conn(0x7fc3300ac570 msgr2=0x7fc33014e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 shutdown_connections 2024-03-06T07:24:33.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.450+0000 7fc336163640 1 -- 172.21.15.94:0/595678088 wait complete. 2024-03-06T07:24:33.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 145, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:03.229941+0000', 'modified': '2024-03-06T07:24:26.705762+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 154, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8155}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8155': {'gid': 8155, 'name': 'b', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8155, 'qdb_cluster': [8155]} 2024-03-06T07:24:33.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:33.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:24:33.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.966+0000 7fd5279b6640 1 Processor -- start 2024-03-06T07:24:33.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.966+0000 7fd5279b6640 1 -- start start 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd5279b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd52006ff10 0x7fd520153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd5279b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd5200c7170 0x7fd520154160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd5279b6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5200c6c40 0x7fd5200c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd5279b6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5200c7620 con 0x7fd5200c6c40 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd5279b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5200a8390 con 0x7fd5200c7170 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd5279b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200730f0 con 0x7fd52006ff10 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5200c6c40 0x7fd5200c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd5200c6c40 0x7fd5200c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52874/0 (socket says 172.21.15.94:52874) 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 -- 172.21.15.94:0/196816841 learned_addr learned my addr 172.21.15.94:0/196816841 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525752640 1 --2- 172.21.15.94:0/196816841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd52006ff10 0x7fd520153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd524f51640 1 --2- 172.21.15.94:0/196816841 >> v2:172.21.15.145:3300/0 conn(0x7fd5200c7170 0x7fd520154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 -- 172.21.15.94:0/196816841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd52006ff10 msgr2=0x7fd520153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 --2- 172.21.15.94:0/196816841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd52006ff10 0x7fd520153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 -- 172.21.15.94:0/196816841 >> v2:172.21.15.145:3300/0 conn(0x7fd5200c7170 msgr2=0x7fd520154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 --2- 172.21.15.94:0/196816841 >> v2:172.21.15.145:3300/0 conn(0x7fd5200c7170 0x7fd520154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.967+0000 7fd525f53640 1 -- 172.21.15.94:0/196816841 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5200733d0 con 0x7fd5200c6c40 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.968+0000 7fd525f53640 1 --2- 172.21.15.94:0/196816841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c6c40 0x7fd5200c8810 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fd5200707c0 tx=0x7fd51802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bebfaf573a3e269 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.968+0000 7fd517fff640 1 -- 172.21.15.94:0/196816841 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51802c950 con 0x7fd5200c6c40 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.968+0000 7fd517fff640 1 -- 172.21.15.94:0/196816841 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51802caf0 con 0x7fd5200c6c40 2024-03-06T07:24:33.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.968+0000 7fd517fff640 1 -- 172.21.15.94:0/196816841 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180343f0 con 0x7fd5200c6c40 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 -- 172.21.15.94:0/196816841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c6c40 msgr2=0x7fd5200c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 --2- 172.21.15.94:0/196816841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c6c40 0x7fd5200c8810 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fd5200707c0 tx=0x7fd51802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 -- 172.21.15.94:0/196816841 shutdown_connections 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 --2- 172.21.15.94:0/196816841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c6c40 0x7fd5200c8810 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 --2- 172.21.15.94:0/196816841 >> v2:172.21.15.145:3300/0 conn(0x7fd5200c7170 0x7fd520154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 --2- 172.21.15.94:0/196816841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd52006ff10 0x7fd520153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 -- 172.21.15.94:0/196816841 >> 172.21.15.94:0/196816841 conn(0x7fd52006b970 msgr2=0x7fd5200aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 -- 172.21.15.94:0/196816841 shutdown_connections 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.969+0000 7fd5279b6640 1 -- 172.21.15.94:0/196816841 wait complete. 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.970+0000 7fd5279b6640 1 Processor -- start 2024-03-06T07:24:33.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.970+0000 7fd5279b6640 1 -- start start 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.970+0000 7fd5279b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd52006ff10 0x7fd5200acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.970+0000 7fd5279b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5200c6c40 0x7fd5201edc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd5279b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c7170 0x7fd52014ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd5279b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd520070470 con 0x7fd5200c7170 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd5279b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200702f0 con 0x7fd52006ff10 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd5279b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200c6860 con 0x7fd5200c6c40 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c7170 0x7fd52014ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c7170 0x7fd52014ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52878/0 (socket says 172.21.15.94:52878) 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 -- 172.21.15.94:0/2198124961 learned_addr learned my addr 172.21.15.94:0/2198124961 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525752640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd52006ff10 0x7fd5200acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd524f51640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5200c6c40 0x7fd5201edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 -- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5200c6c40 msgr2=0x7fd5201edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5200c6c40 0x7fd5201edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 -- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd52006ff10 msgr2=0x7fd5200acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd52006ff10 0x7fd5200acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 -- 172.21.15.94:0/2198124961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520156f20 con 0x7fd5200c7170 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525752640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd52006ff10 0x7fd5200acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd525f53640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c7170 0x7fd52014ce70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fd518036040 tx=0x7fd518004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd515ffb640 1 -- 172.21.15.94:0/2198124961 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518043020 con 0x7fd5200c7170 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd515ffb640 1 -- 172.21.15.94:0/2198124961 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51803e450 con 0x7fd5200c7170 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd520156c40 con 0x7fd5200c7170 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.971+0000 7fd515ffb640 1 -- 172.21.15.94:0/2198124961 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51803e840 con 0x7fd5200c7170 2024-03-06T07:24:33.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.972+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52006c4e0 con 0x7fd5200c7170 2024-03-06T07:24:33.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.973+0000 7fd515ffb640 1 -- 172.21.15.94:0/2198124961 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd518046020 con 0x7fd5200c7170 2024-03-06T07:24:33.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.973+0000 7fd515ffb640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50406b4d0 0x7fd50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.973+0000 7fd515ffb640 1 -- 172.21.15.94:0/2198124961 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fd5180acd10 con 0x7fd5200c7170 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.973+0000 7fd525752640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50406b4d0 0x7fd50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.974+0000 7fd525752640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50406b4d0 0x7fd50406d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd510009eb0 tx=0x7fd510009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.974+0000 7fd52a597740 1 Processor -- start 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.975+0000 7fd52a597740 1 -- start start 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb18a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd52a597740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb18f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb194b0 0x56259eb15680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd52a597740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259ea7f6b0 con 0x56259eb1b330 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259ea28a60 con 0x56259eb194b0 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56259ea7b750 con 0x56259eb1af50 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd525752640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb18a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd524f51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb18f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd525752640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb18a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:43498/0 (socket says 172.21.15.94:43498) 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd525752640 1 -- 172.21.15.94:0/3488691070 learned_addr learned my addr 172.21.15.94:0/3488691070 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd525f53640 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb194b0 0x56259eb15680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd524f51640 1 -- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 msgr2=0x56259eb18a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd524f51640 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb18a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd524f51640 1 -- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb194b0 msgr2=0x56259eb15680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd524f51640 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb194b0 0x56259eb15680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd524f51640 1 -- 172.21.15.94:0/3488691070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259ea46f30 con 0x56259eb1b330 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd525752640 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb18a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd524f51640 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb18f70 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd520072820 tx=0x7fd51c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7bf66d52ef615f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd50affd640 1 -- 172.21.15.94:0/3488691070 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c019070 con 0x56259eb1b330 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd50affd640 1 -- 172.21.15.94:0/3488691070 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c004300 con 0x56259eb1b330 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.976+0000 7fd50affd640 1 -- 172.21.15.94:0/3488691070 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c011bf0 con 0x56259eb1b330 2024-03-06T07:24:33.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 -- 172.21.15.94:0/3488691070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 msgr2=0x56259eb18f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb18f70 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fd520072820 tx=0x7fd51c009f50 comp rx=0 tx=0).stop 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 -- 172.21.15.94:0/3488691070 shutdown_connections 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb194b0 0x56259eb15680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb18f70 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 --2- 172.21.15.94:0/3488691070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb18a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 -- 172.21.15.94:0/3488691070 >> 172.21.15.94:0/3488691070 conn(0x56259eb177d0 msgr2=0x56259eb3c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 -- 172.21.15.94:0/3488691070 shutdown_connections 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 -- 172.21.15.94:0/3488691070 wait complete. 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.977+0000 7fd52a597740 1 -- start start 2024-03-06T07:24:33.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 1 Processor -- start 2024-03-06T07:24:33.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb3b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb6abb0 0x56259eb37fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259eb3d230 con 0x56259eb1b330 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259ea3bea0 con 0x56259eb6abb0 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56259eb3cc70 con 0x56259eb1af50 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd52a597740 10 client.0 _ll_register_callbacks cb 0x56259ea3b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.978+0000 7fd524f51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52896/0 (socket says 172.21.15.94:52896) 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 -- 172.21.15.94:0/2021732678 learned_addr learned my addr 172.21.15.94:0/2021732678 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd525f53640 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb6abb0 0x56259eb37fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd525752640 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb3b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 -- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 msgr2=0x56259eb3b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb3b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 -- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb6abb0 msgr2=0x56259eb37fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb6abb0 0x56259eb37fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 -- 172.21.15.94:0/2021732678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56259ea3c270 con 0x56259eb1b330 2024-03-06T07:24:33.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd524f51640 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb426d0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fd51c011d60 tx=0x7fd51c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f68f31588c7583 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd4fffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd4fffff640 1 -- 172.21.15.94:0/2021732678 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c019040 con 0x56259eb1b330 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd4fffff640 1 -- 172.21.15.94:0/2021732678 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fd51c003820 con 0x56259eb1b330 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd51c004300 con 0x56259eb1b330 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd4fffff640 1 -- 172.21.15.94:0/2021732678 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd51c031050 con 0x56259eb1b330 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd52a597740 10 client.9138 fetch_fsmap learned FSMap version 146 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x56259e7942d0 con 0x56259eb1b330 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.979+0000 7fd4fffff640 1 -- 172.21.15.94:0/2021732678 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c004300 con 0x56259eb1b330 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd4fffff640 1 -- 172.21.15.94:0/2021732678 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 146) v1 ==== 3929+0+0 (secure 0 0 0) 0x7fd51c034020 con 0x56259eb1b330 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd52a597740 10 client.9138 fetch_fsmap finished waiting for FSMap version 146 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd52a597740 10 client.9138 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd52a597740 20 client.9138 populate_metadata read hostname 'smithi094' 2024-03-06T07:24:33.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd52a597740 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259eb67640 0x56259eb67fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd52a597740 4 client.9138 mds_command: new command op to 8155 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 --> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56259ea8cbf0 con 0x56259eb67640 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd525752640 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259eb67640 0x56259eb67fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd4fffff640 10 client.9138 ms_handle_connect on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.980+0000 7fd525752640 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259eb67640 0x56259eb67fc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ddee85f937d891 server_cookie=db33a0e54845642d in_seq=0 out_seq=0 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd4fffff640 1 -- 172.21.15.94:0/2021732678 <== mds.0 v2:172.21.15.145:6835/486493869 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56259ea8cbf0 con 0x56259eb67640 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd4fffff640 10 client.9138 handle_command_reply: tid=0 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 client.9138 shutdown 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 msgr2=0x56259eb426d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb426d0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fd51c011d60 tx=0x7fd51c004730 comp rx=0 tx=0).stop 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 shutdown_connections 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb6abb0 0x56259eb37fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259eb67640 0x56259eb67fc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1b330 0x56259eb426d0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 --2- 172.21.15.94:0/2021732678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb3b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd4fffff640 0 client.9138 ms_handle_reset on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:33.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 >> 172.21.15.94:0/2021732678 conn(0x56259eb194b0 msgr2=0x56259eb198e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 shutdown_connections 2024-03-06T07:24:33.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 1 -- 172.21.15.94:0/2021732678 wait complete. 2024-03-06T07:24:33.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.981+0000 7fd52a597740 20 client.9138 trim_cache size 0 max 16384 2024-03-06T07:24:33.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.998+0000 7fd52a597740 1 Processor -- start 2024-03-06T07:24:34.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.998+0000 7fd52a597740 1 -- start start 2024-03-06T07:24:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb19890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb41440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd52a597740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb6abb0 0x56259eb41980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd52a597740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5201fae80 con 0x56259eb6abb0 2024-03-06T07:24:34.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5201f8ab0 con 0x56259eb1b330 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5201f90b0 con 0x56259eb1af50 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb6abb0 0x56259eb41980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb6abb0 0x56259eb41980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52912/0 (socket says 172.21.15.94:52912) 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 -- 172.21.15.94:0/2760807986 learned_addr learned my addr 172.21.15.94:0/2760807986 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525752640 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb19890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd524f51640 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb41440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 -- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 msgr2=0x56259eb19890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb19890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 -- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 msgr2=0x56259eb41440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb41440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 -- 172.21.15.94:0/2760807986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259e85e6b0 con 0x56259eb6abb0 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd525f53640 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb6abb0 0x56259eb41980 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fd5180aad90 tx=0x7fd518041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9610eb3f8947e454 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd50a7fc640 1 -- 172.21.15.94:0/2760807986 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51807b090 con 0x56259eb6abb0 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd50a7fc640 1 -- 172.21.15.94:0/2760807986 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51807f130 con 0x56259eb6abb0 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:33.999+0000 7fd50a7fc640 1 -- 172.21.15.94:0/2760807986 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180836f0 con 0x56259eb6abb0 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 -- 172.21.15.94:0/2760807986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb6abb0 msgr2=0x56259eb41980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb6abb0 0x56259eb41980 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fd5180aad90 tx=0x7fd518041040 comp rx=0 tx=0).stop 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 -- 172.21.15.94:0/2760807986 shutdown_connections 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb6abb0 0x56259eb41980 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb41440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 --2- 172.21.15.94:0/2760807986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb1af50 0x56259eb19890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 -- 172.21.15.94:0/2760807986 >> 172.21.15.94:0/2760807986 conn(0x56259eb194b0 msgr2=0x56259eb40a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 -- 172.21.15.94:0/2760807986 shutdown_connections 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.000+0000 7fd52a597740 1 -- 172.21.15.94:0/2760807986 wait complete. 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 -- start start 2024-03-06T07:24:34.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 Processor -- start 2024-03-06T07:24:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1af50 0x56259eb41840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb53d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb6abb0 0x56259eb710e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259eb3f800 con 0x56259eb1af50 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259eb40150 con 0x56259eb1b330 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56259eb67a70 con 0x56259eb6abb0 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.001+0000 7fd52a597740 10 client.0 _ll_register_callbacks cb 0x7fd51c009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1af50 0x56259eb41840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1af50 0x56259eb41840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52914/0 (socket says 172.21.15.94:52914) 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 -- 172.21.15.94:0/3201712422 learned_addr learned my addr 172.21.15.94:0/3201712422 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd524f51640 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb53d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525f53640 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb6abb0 0x56259eb710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 -- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb6abb0 msgr2=0x56259eb710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb6abb0 0x56259eb710e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 -- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 msgr2=0x56259eb53d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb53d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 -- 172.21.15.94:0/3201712422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56259eb1c500 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd525752640 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1af50 0x56259eb41840 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fd510011b20 tx=0x7fd510004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca6b71fd7e8175b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd4fffff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd4fffff640 1 -- 172.21.15.94:0/3201712422 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51002b590 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd4fffff640 1 -- 172.21.15.94:0/3201712422 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fd51002b730 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd51807f130 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd4fffff640 1 -- 172.21.15.94:0/3201712422 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd51003c050 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd52a597740 10 client.9144 fetch_fsmap learned FSMap version 146 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=146,monmap=2+}) v3 -- 0x56259eb71620 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.002+0000 7fd4fffff640 1 -- 172.21.15.94:0/3201712422 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51807f130 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd4fffff640 1 -- 172.21.15.94:0/3201712422 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 146) v1 ==== 3929+0+0 (secure 0 0 0) 0x7fd51003f020 con 0x56259eb1af50 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 10 client.9144 fetch_fsmap finished waiting for FSMap version 146 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 10 client.9144 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 20 client.9144 populate_metadata read hostname 'smithi094' 2024-03-06T07:24:34.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259ebc4750 0x56259ebc4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 4 client.9144 mds_command: new command op to 8155 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 --> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56259ea86f40 con 0x56259ebc4750 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd524f51640 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259ebc4750 0x56259ebc4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd524f51640 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259ebc4750 0x56259ebc4b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3c53aa3e408ab97 server_cookie=5c23b945ea23c558 in_seq=0 out_seq=0 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd4fffff640 10 client.9144 ms_handle_connect on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd4fffff640 1 -- 172.21.15.94:0/3201712422 <== mds.0 v2:172.21.15.145:6835/486493869 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x56259ea86f40 con 0x56259ebc4750 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd4fffff640 10 client.9144 handle_command_reply: tid=0 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 1 client.9144 shutdown 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1af50 msgr2=0x56259eb41840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.003+0000 7fd52a597740 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1af50 0x56259eb41840 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fd510011b20 tx=0x7fd510004730 comp rx=0 tx=0).stop 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 shutdown_connections 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56259eb6abb0 0x56259eb710e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56259eb1b330 0x56259eb53d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.145:6835/486493869,v1:172.21.15.145:6837/486493869] conn(0x56259ebc4750 0x56259ebc4b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 --2- 172.21.15.94:0/3201712422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56259eb1af50 0x56259eb41840 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd4fffff640 0 client.9144 ms_handle_reset on v2:172.21.15.145:6835/486493869 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 >> 172.21.15.94:0/3201712422 conn(0x56259eb194b0 msgr2=0x56259eb40a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 shutdown_connections 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 1 -- 172.21.15.94:0/3201712422 wait complete. 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd52a597740 20 client.9144 trim_cache size 0 max 16384 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:24:34.012 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 8155, 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 10.304013152, 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 145, 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 154, 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 154, 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 35.229902594000002 2024-03-06T07:24:34.013 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50406b4d0 msgr2=0x7fd50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50406b4d0 0x7fd50406d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd510009eb0 tx=0x7fd510009190 comp rx=0 tx=0).stop 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c7170 msgr2=0x7fd52014ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c7170 0x7fd52014ce70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fd518036040 tx=0x7fd518004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 shutdown_connections 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd50406b4d0 0x7fd50406d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5200c7170 0x7fd52014ce70 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5200c6c40 0x7fd5201edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 --2- 172.21.15.94:0/2198124961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd52006ff10 0x7fd5200acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 >> 172.21.15.94:0/2198124961 conn(0x7fd52006b970 msgr2=0x7fd5200c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 shutdown_connections 2024-03-06T07:24:34.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.004+0000 7fd5279b6640 1 -- 172.21.15.94:0/2198124961 wait complete. 2024-03-06T07:24:34.017 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 146, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:03.229941+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 145, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8155": { "addr": "172.21.15.145:6837/486493869", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6835", "nonce": 486493869, "type": "v2" }, { "addr": "172.21.15.145:6837", "nonce": 486493869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8155, "incarnation": 142, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 154, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-03-06T07:24:26.705762+0000", "qdb_cluster": [ 8155 ], "qdb_leader": 8155, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8155 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:24:16.636350+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8769": { "addr": "172.21.15.94:6834/1222126706", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 1222126706, "type": "v2" }, { "addr": "172.21.15.94:6834", "nonce": 1222126706, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8769, "incarnation": 136, "join_fscid": 12, "name": "a", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-03-06T07:24:25.285179+0000", "qdb_cluster": [ 8769 ], "qdb_leader": 8769, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8769 } } } ], "standbys": [ { "addr": "172.21.15.145:6836/1521427660", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 1521427660, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 1521427660, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 146, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8152, "incarnation": 0, "join_fscid": 12, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 8 }, { "addr": "172.21.15.94:6835/3923835266", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6833", "nonce": 3923835266, "type": "v2" }, { "addr": "172.21.15.94:6835", "nonce": 3923835266, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 143, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9078, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-03-06T07:24:34.018 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': 12, 'name': 'a'}, {'join_fscid': -1, 'name': 'b', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 12, 'name': 'd'}] 2024-03-06T07:24:34.018 INFO:tasks.cephfs.test_failover:current = [{'gid': 8769, 'name': 'a', 'rank': 0, 'incarnation': 136, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.94:6834/1222126706', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 1222126706}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 1222126706}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8155, 'name': 'b', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.145:6837/486493869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 486493869}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 486493869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 9078, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.94:6835/3923835266', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 3923835266}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 3923835266}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 143}, {'gid': 8152, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 8, 'addr': '172.21.15.145:6836/1521427660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 1521427660}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1521427660}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 146}] 2024-03-06T07:24:34.019 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-06T07:24:34.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.118+0000 7f41f48a4640 1 Processor -- start 2024-03-06T07:24:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.118+0000 7f41f48a4640 1 -- start start 2024-03-06T07:24:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41f48a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f41f01527b0 0x7f41f0159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41f48a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f00c7180 0x7f41f00c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41f48a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00c7c60 con 0x7f41f0155b20 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41f48a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f0159640 con 0x7f41f01527b0 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41f48a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00c8df0 con 0x7f41f00c7180 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41eed9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f41f0155b20 0x7f41f00c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51828/0 (socket says 172.21.15.94:51828) 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41eed9d640 1 -- 172.21.15.94:0/2812532265 learned_addr learned my addr 172.21.15.94:0/2812532265 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41edd9b640 1 --2- 172.21.15.94:0/2812532265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f00c7180 0x7f41f00c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41ee59c640 1 --2- 172.21.15.94:0/2812532265 >> v2:172.21.15.145:3300/0 conn(0x7f41f01527b0 0x7f41f0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41edd9b640 1 -- 172.21.15.94:0/2812532265 >> v2:172.21.15.145:3300/0 conn(0x7f41f01527b0 msgr2=0x7f41f0159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41edd9b640 1 --2- 172.21.15.94:0/2812532265 >> v2:172.21.15.145:3300/0 conn(0x7f41f01527b0 0x7f41f0159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41ed59a640 1 -- 172.21.15.94:0/2812532265 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242995746 0 0) 0x7f41e8001490 con 0x7f41f0155b20 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41edd9b640 1 -- 172.21.15.94:0/2812532265 >> v1:172.21.15.94:6789/0 conn(0x7f41f0155b20 legacy=0x7f41f00c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.119+0000 7f41edd9b640 1 -- 172.21.15.94:0/2812532265 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00c8060 con 0x7f41f00c7180 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.120+0000 7f41edd9b640 1 --2- 172.21.15.94:0/2812532265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f00c7180 0x7f41f00c88b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f41e40097a0 tx=0x7f41e402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fa1f0f4a31e7eed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41ed59a640 1 -- 172.21.15.94:0/2812532265 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e40315a0 con 0x7f41f00c7180 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41ed59a640 1 -- 172.21.15.94:0/2812532265 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e4037070 con 0x7f41f00c7180 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41ed59a640 1 -- 172.21.15.94:0/2812532265 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4031960 con 0x7f41f00c7180 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 -- 172.21.15.94:0/2812532265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f00c7180 msgr2=0x7f41f00c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 --2- 172.21.15.94:0/2812532265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f00c7180 0x7f41f00c88b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f41e40097a0 tx=0x7f41e402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 -- 172.21.15.94:0/2812532265 shutdown_connections 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 --2- 172.21.15.94:0/2812532265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f00c7180 0x7f41f00c88b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 --2- 172.21.15.94:0/2812532265 >> v2:172.21.15.145:3300/0 conn(0x7f41f01527b0 0x7f41f0159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 -- 172.21.15.94:0/2812532265 >> 172.21.15.94:0/2812532265 conn(0x7f41f00ac560 msgr2=0x7f41f00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 -- 172.21.15.94:0/2812532265 shutdown_connections 2024-03-06T07:24:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.121+0000 7f41f48a4640 1 -- 172.21.15.94:0/2812532265 wait complete. 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.122+0000 7f41f48a4640 1 Processor -- start 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41f48a4640 1 -- start start 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41f48a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41f00c7180 0x7f41f014d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41f48a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f01527b0 0x7f41f01eab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41f48a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41f0155b20 0x7f41f0154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41f48a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f0155320 con 0x7f41f0155b20 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41f48a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00c7980 con 0x7f41f00c7180 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41f48a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f0155600 con 0x7f41f01527b0 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41f0155b20 0x7f41f0154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41edd9b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f01527b0 0x7f41f01eab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41eed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41f0155b20 0x7f41f0154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52922/0 (socket says 172.21.15.94:52922) 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41ee59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41f00c7180 0x7f41f014d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.123+0000 7f41eed9d640 1 -- 172.21.15.94:0/1504907299 learned_addr learned my addr 172.21.15.94:0/1504907299 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41eed9d640 1 -- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f01527b0 msgr2=0x7f41f01eab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41eed9d640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f01527b0 0x7f41f01eab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41eed9d640 1 -- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41f00c7180 msgr2=0x7f41f014d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41eed9d640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41f00c7180 0x7f41f014d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41eed9d640 1 -- 172.21.15.94:0/1504907299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f01ee150 con 0x7f41f0155b20 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41eed9d640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41f0155b20 0x7f41f0154850 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f41e800c730 tx=0x7f41e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41deffd640 1 -- 172.21.15.94:0/1504907299 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8018070 con 0x7f41f0155b20 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41deffd640 1 -- 172.21.15.94:0/1504907299 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e800ce40 con 0x7f41f0155b20 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f014dc30 con 0x7f41f0155b20 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41deffd640 1 -- 172.21.15.94:0/1504907299 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e800a860 con 0x7f41f0155b20 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.124+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f01ee3f0 con 0x7f41f0155b20 2024-03-06T07:24:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.125+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f0069460 con 0x7f41f0155b20 2024-03-06T07:24:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.126+0000 7f41deffd640 1 -- 172.21.15.94:0/1504907299 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41e801b070 con 0x7f41f0155b20 2024-03-06T07:24:34.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.126+0000 7f41deffd640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d006b4d0 0x7f41d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.126+0000 7f41deffd640 1 -- 172.21.15.94:0/1504907299 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f41e8089b60 con 0x7f41f0155b20 2024-03-06T07:24:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.126+0000 7f41ee59c640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d006b4d0 0x7f41d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.127+0000 7f41ee59c640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d006b4d0 0x7f41d006d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009b40 tx=0x7f41e0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.130+0000 7f41deffd640 1 -- 172.21.15.94:0/1504907299 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41f0069460 con 0x7f41f0155b20 2024-03-06T07:24:34.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.338+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41f004f330 con 0x7f41f0155b20 2024-03-06T07:24:34.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.338+0000 7f41deffd640 1 -- 172.21.15.94:0/1504907299 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+6278 (secure 0 0 0) 0x7f41f00c7c60 con 0x7f41f0155b20 2024-03-06T07:24:34.340 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:34.340 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":146,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":146},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:26.705762+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8155,"qdb_cluster":[8155]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:34.340 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 146 2024-03-06T07:24:34.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d006b4d0 msgr2=0x7f41d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d006b4d0 0x7f41d006d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009b40 tx=0x7f41e0009190 comp rx=0 tx=0).stop 2024-03-06T07:24:34.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41f0155b20 msgr2=0x7f41f0154850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41f0155b20 0x7f41f0154850 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f41e800c730 tx=0x7f41e800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 shutdown_connections 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d006b4d0 0x7f41d006d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41f0155b20 0x7f41f0154850 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41f01527b0 0x7f41f01eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 --2- 172.21.15.94:0/1504907299 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41f00c7180 0x7f41f014d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 >> 172.21.15.94:0/1504907299 conn(0x7f41f00ac560 msgr2=0x7f41f00c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 shutdown_connections 2024-03-06T07:24:34.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.340+0000 7f41f48a4640 1 -- 172.21.15.94:0/1504907299 wait complete. 2024-03-06T07:24:34.353 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:34.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:34.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:34.428 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:34.428 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:24:34.429 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:34.443 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:24:34.444 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:24:34.444 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:34.484 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:24:34.484 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:34.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.583+0000 7f7ffdeed640 1 Processor -- start 2024-03-06T07:24:34.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.583+0000 7f7ffdeed640 1 -- start start 2024-03-06T07:24:34.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.584+0000 7f7ffdeed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ff80c5860 0x7f7ff81609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.584+0000 7f7ffdeed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff81591c0 con 0x7f7ff8155ba0 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.584+0000 7f7ffdeed640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff81594a0 con 0x7f7ff80c5480 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.584+0000 7f7ffdeed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff806df40 con 0x7f7ff80c5860 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.584+0000 7f7ff6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7ff8155ba0 0x7f7ff81586f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51842/0 (socket says 172.21.15.94:51842) 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.584+0000 7f7ff6ffd640 1 -- 172.21.15.94:0/1359794133 learned_addr learned my addr 172.21.15.94:0/1359794133 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.584+0000 7f7ff7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ff80c5860 0x7f7ff81609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.585+0000 7f7ff7fff640 1 -- 172.21.15.94:0/1359794133 >> v1:172.21.15.145:6789/0 conn(0x7f7ff80c5480 legacy=0x7f7ff806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.585+0000 7f7ff7fff640 1 -- 172.21.15.94:0/1359794133 >> v1:172.21.15.94:6789/0 conn(0x7f7ff8155ba0 legacy=0x7f7ff81586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.585+0000 7f7ff67fc640 1 -- 172.21.15.94:0/1359794133 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737593423 0 0) 0x7f7ff81591c0 con 0x7f7ff8155ba0 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.585+0000 7f7ff7fff640 1 -- 172.21.15.94:0/1359794133 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff81553a0 con 0x7f7ff80c5860 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.585+0000 7f7ff7fff640 1 --2- 172.21.15.94:0/1359794133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5860 0x7f7ff81609e0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097a0 tx=0x7f7ff002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9329755cc47cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ff67fc640 1 -- 172.21.15.94:0/1359794133 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0035020 con 0x7f7ff80c5860 2024-03-06T07:24:34.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ff67fc640 1 -- 172.21.15.94:0/1359794133 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff0031830 con 0x7f7ff80c5860 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ff67fc640 1 -- 172.21.15.94:0/1359794133 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0031c00 con 0x7f7ff80c5860 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ffdeed640 1 -- 172.21.15.94:0/1359794133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5860 msgr2=0x7f7ff81609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/1359794133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5860 0x7f7ff81609e0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097a0 tx=0x7f7ff002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ffdeed640 1 -- 172.21.15.94:0/1359794133 shutdown_connections 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/1359794133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5860 0x7f7ff81609e0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ffdeed640 1 -- 172.21.15.94:0/1359794133 >> 172.21.15.94:0/1359794133 conn(0x7f7ff806b1a0 msgr2=0x7f7ff806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ffdeed640 1 -- 172.21.15.94:0/1359794133 shutdown_connections 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.586+0000 7f7ffdeed640 1 -- 172.21.15.94:0/1359794133 wait complete. 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.587+0000 7f7ffdeed640 1 Processor -- start 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.588+0000 7f7ffdeed640 1 -- start start 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ffdeed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5480 0x7f7ff8157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ffdeed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ff80c5860 0x7f7ff80ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ffdeed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ff8155ba0 0x7f7ff80ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ffdeed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80c5cf0 con 0x7f7ff80c5480 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ffdeed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff806fdb0 con 0x7f7ff80c5860 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ffdeed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff8156000 con 0x7f7ff8155ba0 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5480 0x7f7ff8157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5480 0x7f7ff8157440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52938/0 (socket says 172.21.15.94:52938) 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 -- 172.21.15.94:0/3543994488 learned_addr learned my addr 172.21.15.94:0/3543994488 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff6ffd640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ff80c5860 0x7f7ff80ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff7fff640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ff8155ba0 0x7f7ff80ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 -- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ff8155ba0 msgr2=0x7f7ff80ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ff8155ba0 0x7f7ff80ac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 -- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ff80c5860 msgr2=0x7f7ff80ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ff80c5860 0x7f7ff80ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 -- 172.21.15.94:0/3543994488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff81ef360 con 0x7f7ff80c5480 2024-03-06T07:24:34.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.589+0000 7f7ff77fe640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5480 0x7f7ff8157440 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800c8a0 tx=0x7f7fe800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.590+0000 7f7fdffff640 1 -- 172.21.15.94:0/3543994488 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8003b70 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.590+0000 7f7fdffff640 1 -- 172.21.15.94:0/3543994488 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8003d30 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.590+0000 7f7fdffff640 1 -- 172.21.15.94:0/3543994488 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe800aa90 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.590+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff80acd10 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.590+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff80ad510 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.591+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff804f330 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.591+0000 7f7fdffff640 1 -- 172.21.15.94:0/3543994488 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fe800ac30 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.592+0000 7f7fdffff640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe406b500 0x7f7fe406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.592+0000 7f7fdffff640 1 -- 172.21.15.94:0/3543994488 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f7fe808a4d0 con 0x7f7ff80c5480 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.592+0000 7f7ff6ffd640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe406b500 0x7f7fe406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.592+0000 7f7ff6ffd640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe406b500 0x7f7fe406d9c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7fec000f60 tx=0x7f7fec007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.596+0000 7f7fdffff640 1 -- 172.21.15.94:0/3543994488 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fe8058850 con 0x7f7ff80c5480 2024-03-06T07:24:34.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.852+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ff80adc70 con 0x7f7ff80c5480 2024-03-06T07:24:34.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.853+0000 7f7fdffff640 1 -- 172.21.15.94:0/3543994488 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+6278 (secure 0 0 0) 0x7f7fe805c8f0 con 0x7f7ff80c5480 2024-03-06T07:24:34.855 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:34.855 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":146,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":146},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:26.705762+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8155,"qdb_cluster":[8155]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:34.855 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 146 2024-03-06T07:24:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.854+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe406b500 msgr2=0x7f7fe406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.854+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe406b500 0x7f7fe406d9c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f7fec000f60 tx=0x7f7fec007e40 comp rx=0 tx=0).stop 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.854+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5480 msgr2=0x7f7ff8157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.854+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5480 0x7f7ff8157440 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800c8a0 tx=0x7f7fe800cd60 comp rx=0 tx=0).stop 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 shutdown_connections 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe406b500 0x7f7fe406d9c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ff8155ba0 0x7f7ff80ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ff80c5860 0x7f7ff80ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 --2- 172.21.15.94:0/3543994488 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ff80c5480 0x7f7ff8157440 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 >> 172.21.15.94:0/3543994488 conn(0x7f7ff806b1a0 msgr2=0x7f7ff806b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 shutdown_connections 2024-03-06T07:24:34.858 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.855+0000 7f7ffdeed640 1 -- 172.21.15.94:0/3543994488 wait complete. 2024-03-06T07:24:34.868 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.969+0000 7f378359e640 1 Processor -- start 2024-03-06T07:24:34.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.969+0000 7f378359e640 1 -- start start 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378359e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378359e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37840c5460 con 0x7f37840c87d0 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378359e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37840c5740 con 0x7f37840c7670 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378359e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840c8ce0 con 0x7f37841526d0 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f377bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f37840c87d0 0x7f37840cabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51846/0 (socket says 172.21.15.94:51846) 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f377bfff640 1 -- 172.21.15.94:0/4032732394 learned_addr learned my addr 172.21.15.94:0/4032732394 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378259c640 1 --2- 172.21.15.94:0/4032732394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f3781d9b640 1 -- 172.21.15.94:0/4032732394 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154368345 0 0) 0x7f37840c5460 con 0x7f37840c87d0 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f3781d9b640 1 -- 172.21.15.94:0/4032732394 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3770003520 con 0x7f37840c87d0 2024-03-06T07:24:34.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f3781d9b640 1 -- 172.21.15.94:0/4032732394 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337280989 0 0) 0x7f37840c5740 con 0x7f37840c7670 2024-03-06T07:24:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378259c640 1 -- 172.21.15.94:0/4032732394 >> v1:172.21.15.145:6789/0 conn(0x7f37840c7670 legacy=0x7f37840c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378259c640 1 -- 172.21.15.94:0/4032732394 >> v1:172.21.15.94:6789/0 conn(0x7f37840c87d0 legacy=0x7f37840cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.970+0000 7f378259c640 1 -- 172.21.15.94:0/4032732394 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840c5a20 con 0x7f37841526d0 2024-03-06T07:24:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.971+0000 7f378259c640 1 --2- 172.21.15.94:0/4032732394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784155bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37740097d0 tx=0x7f377402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92e40352a39e8f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.971+0000 7f3781d9b640 1 -- 172.21.15.94:0/4032732394 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774035020 con 0x7f37841526d0 2024-03-06T07:24:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.971+0000 7f3781d9b640 1 -- 172.21.15.94:0/4032732394 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3774031890 con 0x7f37841526d0 2024-03-06T07:24:34.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.971+0000 7f3781d9b640 1 -- 172.21.15.94:0/4032732394 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774031c40 con 0x7f37841526d0 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.972+0000 7f378359e640 1 -- 172.21.15.94:0/4032732394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 msgr2=0x7f3784155bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.972+0000 7f378359e640 1 --2- 172.21.15.94:0/4032732394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784155bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37740097d0 tx=0x7f377402b930 comp rx=0 tx=0).stop 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.972+0000 7f378359e640 1 -- 172.21.15.94:0/4032732394 shutdown_connections 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.972+0000 7f378359e640 1 --2- 172.21.15.94:0/4032732394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784155bb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.972+0000 7f378359e640 1 -- 172.21.15.94:0/4032732394 >> 172.21.15.94:0/4032732394 conn(0x7f37840728d0 msgr2=0x7f3784072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.972+0000 7f378359e640 1 -- 172.21.15.94:0/4032732394 shutdown_connections 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.972+0000 7f378359e640 1 -- 172.21.15.94:0/4032732394 wait complete. 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.973+0000 7f378359e640 1 Processor -- start 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.973+0000 7f378359e640 1 -- start start 2024-03-06T07:24:34.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37840c7670 0x7f3784158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37840c87d0 0x7f37840c2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3784152ab0 con 0x7f37840c7670 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37840c71e0 con 0x7f37840c87d0 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840cb2f0 con 0x7f37841526d0 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37840c7670 0x7f3784158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37840c7670 0x7f3784158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52944/0 (socket says 172.21.15.94:52944) 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 -- 172.21.15.94:0/679127047 learned_addr learned my addr 172.21.15.94:0/679127047 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f377bfff640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37840c87d0 0x7f37840c2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f3782d9d640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 -- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 msgr2=0x7f3784153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 -- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37840c87d0 msgr2=0x7f37840c2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37840c87d0 0x7f37840c2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 -- 172.21.15.94:0/679127047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3784158d90 con 0x7f37840c7670 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378259c640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37840c7670 0x7f3784158370 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f377402b970 tx=0x7f37740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f377affd640 1 -- 172.21.15.94:0/679127047 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377403ad60 con 0x7f37840c7670 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f377affd640 1 -- 172.21.15.94:0/679127047 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37740443f0 con 0x7f37840c7670 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f377affd640 1 -- 172.21.15.94:0/679127047 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37740447c0 con 0x7f37840c7670 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37840c4570 con 0x7f37840c7670 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.974+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37841e8070 con 0x7f37840c7670 2024-03-06T07:24:34.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.975+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3784158b80 con 0x7f37840c7670 2024-03-06T07:24:34.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.976+0000 7f377affd640 1 -- 172.21.15.94:0/679127047 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f377404b060 con 0x7f37840c7670 2024-03-06T07:24:34.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.976+0000 7f377affd640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f376406b4d0 0x7f376406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.976+0000 7f377affd640 1 -- 172.21.15.94:0/679127047 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f37740acb10 con 0x7f37840c7670 2024-03-06T07:24:34.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.977+0000 7f377bfff640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f376406b4d0 0x7f376406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.977+0000 7f377bfff640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f376406b4d0 0x7f376406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f37840c3ce0 tx=0x7f376c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:34.984+0000 7f377affd640 1 -- 172.21.15.94:0/679127047 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f377403f070 con 0x7f37840c7670 2024-03-06T07:24:35.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.162+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3784069430 con 0x7f37840c7670 2024-03-06T07:24:35.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.164+0000 7f377affd640 1 -- 172.21.15.94:0/679127047 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+20731 (secure 0 0 0) 0x7f377402d040 con 0x7f37840c7670 2024-03-06T07:24:35.166 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:35.166 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":154,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:23.693537+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/1613596341":"2024-03-07T07:24:23.693503+0000","172.21.15.94:6833/1613596341":"2024-03-07T07:24:23.693503+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f376406b4d0 msgr2=0x7f376406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f376406b4d0 0x7f376406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f37840c3ce0 tx=0x7f376c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37840c7670 msgr2=0x7f3784158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37840c7670 0x7f3784158370 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f377402b970 tx=0x7f37740046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 shutdown_connections 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f376406b4d0 0x7f376406d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37841526d0 0x7f3784153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37840c87d0 0x7f37840c2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 --2- 172.21.15.94:0/679127047 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37840c7670 0x7f3784158370 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 >> 172.21.15.94:0/679127047 conn(0x7f37840728d0 msgr2=0x7f37840c5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 shutdown_connections 2024-03-06T07:24:35.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.166+0000 7f378359e640 1 -- 172.21.15.94:0/679127047 wait complete. 2024-03-06T07:24:35.182 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:01.443144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:02.192757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:24:15.094089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:24:15.572143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:35.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:35.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 Processor -- start 2024-03-06T07:24:35.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 -- start start 2024-03-06T07:24:35.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04dc0a8290 0x7f04dc1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0c4740 0x7f04dc0701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f04dc0c69a0 0x7f04dc072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc153f00 con 0x7f04dc0c69a0 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc0c62a0 con 0x7f04dc0a8290 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.285+0000 7f04e44c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0706e0 con 0x7f04dc0c4740 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f04dc0c69a0 0x7f04dc072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f04dc0c69a0 0x7f04dc072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52950/0 (socket says 172.21.15.94:52950) 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 -- 172.21.15.94:0/547491672 learned_addr learned my addr 172.21.15.94:0/547491672 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e1a64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0c4740 0x7f04dc0701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2265640 1 --2- 172.21.15.94:0/547491672 >> v2:172.21.15.145:3300/0 conn(0x7f04dc0a8290 0x7f04dc1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 -- 172.21.15.94:0/547491672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0c4740 msgr2=0x7f04dc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 --2- 172.21.15.94:0/547491672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0c4740 0x7f04dc0701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 -- 172.21.15.94:0/547491672 >> v2:172.21.15.145:3300/0 conn(0x7f04dc0a8290 msgr2=0x7f04dc1539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 --2- 172.21.15.94:0/547491672 >> v2:172.21.15.145:3300/0 conn(0x7f04dc0a8290 0x7f04dc1539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.286+0000 7f04e2a66640 1 -- 172.21.15.94:0/547491672 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0c4c00 con 0x7f04dc0c69a0 2024-03-06T07:24:35.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e2a66640 1 --2- 172.21.15.94:0/547491672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc072680 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0735b0 tx=0x7f04d402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797d202ba5a39595 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e1263640 1 -- 172.21.15.94:0/547491672 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d402c950 con 0x7f04dc0c69a0 2024-03-06T07:24:35.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e1263640 1 -- 172.21.15.94:0/547491672 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d402caf0 con 0x7f04dc0c69a0 2024-03-06T07:24:35.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e1263640 1 -- 172.21.15.94:0/547491672 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d40343f0 con 0x7f04dc0c69a0 2024-03-06T07:24:35.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 -- 172.21.15.94:0/547491672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 msgr2=0x7f04dc072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 --2- 172.21.15.94:0/547491672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc072680 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0735b0 tx=0x7f04d402bdb0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 -- 172.21.15.94:0/547491672 shutdown_connections 2024-03-06T07:24:35.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 --2- 172.21.15.94:0/547491672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc072680 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 --2- 172.21.15.94:0/547491672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0c4740 0x7f04dc0701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 --2- 172.21.15.94:0/547491672 >> v2:172.21.15.145:3300/0 conn(0x7f04dc0a8290 0x7f04dc1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 -- 172.21.15.94:0/547491672 >> 172.21.15.94:0/547491672 conn(0x7f04dc0c5560 msgr2=0x7f04dc0c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 -- 172.21.15.94:0/547491672 shutdown_connections 2024-03-06T07:24:35.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.287+0000 7f04e44c9640 1 -- 172.21.15.94:0/547491672 wait complete. 2024-03-06T07:24:35.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.289+0000 7f04e44c9640 1 Processor -- start 2024-03-06T07:24:35.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.289+0000 7f04e44c9640 1 -- start start 2024-03-06T07:24:35.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e44c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0a8290 0x7f04dc0a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e44c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc0c4740 0x7f04dc0ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e44c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc1ec290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e44c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc06fc80 con 0x7f04dc0c69a0 2024-03-06T07:24:35.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e44c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc06f9a0 con 0x7f04dc0c4740 2024-03-06T07:24:35.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e44c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0c6e10 con 0x7f04dc0a8290 2024-03-06T07:24:35.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e2a66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc1ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e2a66640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc1ec290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52958/0 (socket says 172.21.15.94:52958) 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e2a66640 1 -- 172.21.15.94:0/998691858 learned_addr learned my addr 172.21.15.94:0/998691858 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e2265640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0a8290 0x7f04dc0a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.290+0000 7f04e1a64640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc0c4740 0x7f04dc0ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e2a66640 1 -- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0a8290 msgr2=0x7f04dc0a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e2a66640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0a8290 0x7f04dc0a88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e2a66640 1 -- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc0c4740 msgr2=0x7f04dc0ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e2a66640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc0c4740 0x7f04dc0ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e2a66640 1 -- 172.21.15.94:0/998691858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0ac450 con 0x7f04dc0c69a0 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e2a66640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc1ec290 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f04d402c3c0 tx=0x7f04d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04d2ffd640 1 -- 172.21.15.94:0/998691858 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4043020 con 0x7f04dc0c69a0 2024-03-06T07:24:35.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04d2ffd640 1 -- 172.21.15.94:0/998691858 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d403e450 con 0x7f04dc0c69a0 2024-03-06T07:24:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc0ac730 con 0x7f04dc0c69a0 2024-03-06T07:24:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04d2ffd640 1 -- 172.21.15.94:0/998691858 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4009510 con 0x7f04dc0c69a0 2024-03-06T07:24:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.291+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc0a8e20 con 0x7f04dc0c69a0 2024-03-06T07:24:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.292+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc04f330 con 0x7f04dc0c69a0 2024-03-06T07:24:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.292+0000 7f04d2ffd640 1 -- 172.21.15.94:0/998691858 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04d4046020 con 0x7f04dc0c69a0 2024-03-06T07:24:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.293+0000 7f04d2ffd640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f04c006b4d0 0x7f04c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.293+0000 7f04d2ffd640 1 -- 172.21.15.94:0/998691858 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f04d40acc60 con 0x7f04dc0c69a0 2024-03-06T07:24:35.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.293+0000 7f04e2265640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f04c006b4d0 0x7f04c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.293+0000 7f04e2265640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f04c006b4d0 0x7f04c006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0a90f0 tx=0x7f04d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.297+0000 7f04d2ffd640 1 -- 172.21.15.94:0/998691858 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04d404d030 con 0x7f04dc0c69a0 2024-03-06T07:24:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.474+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f04dc0ac090 con 0x7f04dc0c69a0 2024-03-06T07:24:35.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.475+0000 7f04d2ffd640 1 -- 172.21.15.94:0/998691858 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v154) v1 ==== 93+0+10 (secure 0 0 0) 0x7f04d407af50 con 0x7f04dc0c69a0 2024-03-06T07:24:35.477 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f04c006b4d0 msgr2=0x7f04c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f04c006b4d0 0x7f04c006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0a90f0 tx=0x7f04d8009190 comp rx=0 tx=0).stop 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 msgr2=0x7f04dc1ec290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc1ec290 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f04d402c3c0 tx=0x7f04d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 shutdown_connections 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f04c006b4d0 0x7f04c006d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04dc0c69a0 0x7f04dc1ec290 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc0c4740 0x7f04dc0ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 --2- 172.21.15.94:0/998691858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04dc0a8290 0x7f04dc0a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 >> 172.21.15.94:0/998691858 conn(0x7f04dc0c5560 msgr2=0x7f04dc0a9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 shutdown_connections 2024-03-06T07:24:35.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.477+0000 7f04e44c9640 1 -- 172.21.15.94:0/998691858 wait complete. 2024-03-06T07:24:35.490 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:35.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.592+0000 7f5c92ceb640 1 Processor -- start 2024-03-06T07:24:35.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.592+0000 7f5c92ceb640 1 -- start start 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.592+0000 7f5c92ceb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c8c0b0070 0x7f5c8c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.592+0000 7f5c92ceb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5c8c06ca70 0x7f5c8c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.592+0000 7f5c92ceb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0b0450 con 0x7f5c8c0c5380 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.592+0000 7f5c92ceb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c06c020 con 0x7f5c8c0b0070 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.592+0000 7f5c92ceb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c06cf20 con 0x7f5c8c06ca70 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c90a87640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5c8c0b0070 0x7f5c8c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c91288640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5c8c0c5380 0x7f5c8c14e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51858/0 (socket says 172.21.15.94:51858) 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c91288640 1 -- 172.21.15.94:0/3897077695 learned_addr learned my addr 172.21.15.94:0/3897077695 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:35.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c8bfff640 1 --2- 172.21.15.94:0/3897077695 >> v2:172.21.15.145:3300/0 conn(0x7f5c8c06ca70 0x7f5c8c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c90a87640 1 -- 172.21.15.94:0/3897077695 >> v2:172.21.15.145:3300/0 conn(0x7f5c8c06ca70 msgr2=0x7f5c8c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c90a87640 1 --2- 172.21.15.94:0/3897077695 >> v2:172.21.15.145:3300/0 conn(0x7f5c8c06ca70 0x7f5c8c0c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c90a87640 1 -- 172.21.15.94:0/3897077695 >> v1:172.21.15.94:6789/0 conn(0x7f5c8c0c5380 legacy=0x7f5c8c14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c8b7fe640 1 -- 172.21.15.94:0/3897077695 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659904877 0 0) 0x7f5c7c001110 con 0x7f5c8c0c5380 2024-03-06T07:24:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.593+0000 7f5c90a87640 1 -- 172.21.15.94:0/3897077695 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0c77f0 con 0x7f5c8c0b0070 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c90a87640 1 --2- 172.21.15.94:0/3897077695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0b0070 0x7f5c8c155c30 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06d4c0 tx=0x7f5c8402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4d0bef9964c954 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c8b7fe640 1 -- 172.21.15.94:0/3897077695 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84039070 con 0x7f5c8c0b0070 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c8b7fe640 1 -- 172.21.15.94:0/3897077695 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8402cbf0 con 0x7f5c8c0b0070 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c8b7fe640 1 -- 172.21.15.94:0/3897077695 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84034440 con 0x7f5c8c0b0070 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 -- 172.21.15.94:0/3897077695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0b0070 msgr2=0x7f5c8c155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/3897077695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0b0070 0x7f5c8c155c30 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c06d4c0 tx=0x7f5c8402be40 comp rx=0 tx=0).stop 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 -- 172.21.15.94:0/3897077695 shutdown_connections 2024-03-06T07:24:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/3897077695 >> v2:172.21.15.145:3300/0 conn(0x7f5c8c06ca70 0x7f5c8c0c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/3897077695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c0b0070 0x7f5c8c155c30 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 -- 172.21.15.94:0/3897077695 >> 172.21.15.94:0/3897077695 conn(0x7f5c8c0ac590 msgr2=0x7f5c8c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 -- 172.21.15.94:0/3897077695 shutdown_connections 2024-03-06T07:24:35.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.594+0000 7f5c92ceb640 1 -- 172.21.15.94:0/3897077695 wait complete. 2024-03-06T07:24:35.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.596+0000 7f5c92ceb640 1 Processor -- start 2024-03-06T07:24:35.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.596+0000 7f5c92ceb640 1 -- start start 2024-03-06T07:24:35.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.597+0000 7f5c92ceb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c06ca70 0x7f5c8c1eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.597+0000 7f5c92ceb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c8c0b0070 0x7f5c8c154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.597+0000 7f5c92ceb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c8c0c5380 0x7f5c8c157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.597+0000 7f5c92ceb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c06c550 con 0x7f5c8c06ca70 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.597+0000 7f5c92ceb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c06c270 con 0x7f5c8c0b0070 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.597+0000 7f5c92ceb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c06da80 con 0x7f5c8c0c5380 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c06ca70 0x7f5c8c1eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c06ca70 0x7f5c8c1eafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52982/0 (socket says 172.21.15.94:52982) 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 -- 172.21.15.94:0/606210083 learned_addr learned my addr 172.21.15.94:0/606210083 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c8bfff640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c8c0b0070 0x7f5c8c154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c91288640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c8c0c5380 0x7f5c8c157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 -- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c8c0c5380 msgr2=0x7f5c8c157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c8c0c5380 0x7f5c8c157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 -- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c8c0b0070 msgr2=0x7f5c8c154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c8c0b0070 0x7f5c8c154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 -- 172.21.15.94:0/606210083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0c34c0 con 0x7f5c8c06ca70 2024-03-06T07:24:35.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c90a87640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c06ca70 0x7f5c8c1eafb0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f5c84037040 tx=0x7f5c84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c897fa640 1 -- 172.21.15.94:0/606210083 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84039050 con 0x7f5c8c06ca70 2024-03-06T07:24:35.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c897fa640 1 -- 172.21.15.94:0/606210083 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c84045070 con 0x7f5c8c06ca70 2024-03-06T07:24:35.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c0c37a0 con 0x7f5c8c06ca70 2024-03-06T07:24:35.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c897fa640 1 -- 172.21.15.94:0/606210083 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8403e850 con 0x7f5c8c06ca70 2024-03-06T07:24:35.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.598+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c0c3d60 con 0x7f5c8c06ca70 2024-03-06T07:24:35.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.599+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c0ab3e0 con 0x7f5c8c06ca70 2024-03-06T07:24:35.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.600+0000 7f5c897fa640 1 -- 172.21.15.94:0/606210083 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c8403e9f0 con 0x7f5c8c06ca70 2024-03-06T07:24:35.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.600+0000 7f5c897fa640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c7806b4d0 0x7f5c7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.600+0000 7f5c897fa640 1 -- 172.21.15.94:0/606210083 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f5c840ad2c0 con 0x7f5c8c06ca70 2024-03-06T07:24:35.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.600+0000 7f5c8bfff640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c7806b4d0 0x7f5c7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.601+0000 7f5c8bfff640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c7806b4d0 0x7f5c7806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c1e2900 tx=0x7f5c80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.604+0000 7f5c897fa640 1 -- 172.21.15.94:0/606210083 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c8c0ab3e0 con 0x7f5c8c06ca70 2024-03-06T07:24:35.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.782+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5c8c04f330 con 0x7f5c8c06ca70 2024-03-06T07:24:35.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.783+0000 7f5c897fa640 1 -- 172.21.15.94:0/606210083 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v154) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5c8407b760 con 0x7f5c8c06ca70 2024-03-06T07:24:35.785 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:24:35.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.785+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c7806b4d0 msgr2=0x7f5c7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.785+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c7806b4d0 0x7f5c7806d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c1e2900 tx=0x7f5c80009190 comp rx=0 tx=0).stop 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.785+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c06ca70 msgr2=0x7f5c8c1eafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.785+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c06ca70 0x7f5c8c1eafb0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f5c84037040 tx=0x7f5c84004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 shutdown_connections 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c7806b4d0 0x7f5c7806d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c8c0c5380 0x7f5c8c157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c8c0b0070 0x7f5c8c154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 --2- 172.21.15.94:0/606210083 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5c8c06ca70 0x7f5c8c1eafb0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 >> 172.21.15.94:0/606210083 conn(0x7f5c8c0ac590 msgr2=0x7f5c8c0ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 shutdown_connections 2024-03-06T07:24:35.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.786+0000 7f5c92ceb640 1 -- 172.21.15.94:0/606210083 wait complete. 2024-03-06T07:24:35.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:35.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.903+0000 7f8261917640 1 Processor -- start 2024-03-06T07:24:35.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.903+0000 7f8261917640 1 -- start start 2024-03-06T07:24:35.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261917640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261917640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f825c0c8d50 0x7f825c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261917640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c1552c0 con 0x7f825c06afc0 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261917640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825c0c6d30 con 0x7f825c0c8d50 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261917640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c155580 con 0x7f825c152a20 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261116640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f825c0c8d50 0x7f825c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261116640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f825c0c8d50 0x7f825c06b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52996/0 (socket says 172.21.15.94:52996) 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f825bfff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f825c06afc0 0x7f825c0c8640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:56922/0 (socket says 172.21.15.94:56922) 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8261116640 1 -- 172.21.15.94:0/590222957 learned_addr learned my addr 172.21.15.94:0/590222957 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.904+0000 7f8260915640 1 --2- 172.21.15.94:0/590222957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.905+0000 7f825b7fe640 1 -- 172.21.15.94:0/590222957 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938359678 0 0) 0x7f825c1552c0 con 0x7f825c06afc0 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.905+0000 7f8261116640 1 -- 172.21.15.94:0/590222957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 msgr2=0x7f825c1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.905+0000 7f8261116640 1 --2- 172.21.15.94:0/590222957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.905+0000 7f8261116640 1 -- 172.21.15.94:0/590222957 >> v1:172.21.15.145:6789/0 conn(0x7f825c06afc0 legacy=0x7f825c0c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.905+0000 7f8261116640 1 -- 172.21.15.94:0/590222957 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c155860 con 0x7f825c0c8d50 2024-03-06T07:24:35.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f8261116640 1 --2- 172.21.15.94:0/590222957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c8d50 0x7f825c06b420 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f8254009e50 tx=0x7f825402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90de69b5b09aa95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f825b7fe640 1 -- 172.21.15.94:0/590222957 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825402c9e0 con 0x7f825c0c8d50 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f825b7fe640 1 -- 172.21.15.94:0/590222957 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825402cb80 con 0x7f825c0c8d50 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f825b7fe640 1 -- 172.21.15.94:0/590222957 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82540343f0 con 0x7f825c0c8d50 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f8261917640 1 -- 172.21.15.94:0/590222957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c8d50 msgr2=0x7f825c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f8261917640 1 --2- 172.21.15.94:0/590222957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c8d50 0x7f825c06b420 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f8254009e50 tx=0x7f825402be40 comp rx=0 tx=0).stop 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f8261917640 1 -- 172.21.15.94:0/590222957 shutdown_connections 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f8261917640 1 --2- 172.21.15.94:0/590222957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c8d50 0x7f825c06b420 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f8261917640 1 --2- 172.21.15.94:0/590222957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.906+0000 7f8261917640 1 -- 172.21.15.94:0/590222957 >> 172.21.15.94:0/590222957 conn(0x7f825c0ca740 msgr2=0x7f825c0cab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.907+0000 7f8261917640 1 -- 172.21.15.94:0/590222957 shutdown_connections 2024-03-06T07:24:35.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.907+0000 7f8261917640 1 -- 172.21.15.94:0/590222957 wait complete. 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 Processor -- start 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 -- start start 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c06afc0 0x7f825c0af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1ea590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c1f90 0x7f825c1eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c156060 con 0x7f825c0c1f90 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c152e00 con 0x7f825c06afc0 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.908+0000 7f8261917640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c1edbe0 con 0x7f825c152a20 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c1f90 0x7f825c1eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c1f90 0x7f825c1eaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53000/0 (socket says 172.21.15.94:53000) 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 -- 172.21.15.94:0/3844945658 learned_addr learned my addr 172.21.15.94:0/3844945658 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8260915640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c06afc0 0x7f825c0af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f825bfff640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 -- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 msgr2=0x7f825c1ea590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1ea590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 -- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c06afc0 msgr2=0x7f825c0af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c06afc0 0x7f825c0af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 -- 172.21.15.94:0/3844945658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c1edec0 con 0x7f825c0c1f90 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261116640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c1f90 0x7f825c1eaad0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f8254036040 tx=0x7f8254004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f82597fa640 1 -- 172.21.15.94:0/3844945658 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254043020 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f82597fa640 1 -- 172.21.15.94:0/3844945658 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825403e450 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f82597fa640 1 -- 172.21.15.94:0/3844945658 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825403e820 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c1ee1a0 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.909+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c157bf0 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.910+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c0684b0 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.911+0000 7f82597fa640 1 -- 172.21.15.94:0/3844945658 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8254005d10 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.911+0000 7f82597fa640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f824006b4d0 0x7f824006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.911+0000 7f82597fa640 1 -- 172.21.15.94:0/3844945658 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f82540acd10 con 0x7f825c0c1f90 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.911+0000 7f8260915640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f824006b4d0 0x7f824006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.912+0000 7f8260915640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f824006b4d0 0x7f824006d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8250009aa0 tx=0x7f8250009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:35.916+0000 7f82597fa640 1 -- 172.21.15.94:0/3844945658 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f825404d070 con 0x7f825c0c1f90 2024-03-06T07:24:36.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.092+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f825c0735a0 con 0x7f825c0c1f90 2024-03-06T07:24:36.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.093+0000 7f82597fa640 1 -- 172.21.15.94:0/3844945658 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v154) v1 ==== 100+0+11 (secure 0 0 0) 0x7f825c0735a0 con 0x7f825c0c1f90 2024-03-06T07:24:36.095 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:24:36.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f824006b4d0 msgr2=0x7f824006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f824006b4d0 0x7f824006d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8250009aa0 tx=0x7f8250009190 comp rx=0 tx=0).stop 2024-03-06T07:24:36.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c1f90 msgr2=0x7f825c1eaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c1f90 0x7f825c1eaad0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f8254036040 tx=0x7f8254004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:36.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 shutdown_connections 2024-03-06T07:24:36.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f824006b4d0 0x7f824006d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f825c0c1f90 0x7f825c1eaad0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c152a20 0x7f825c1ea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 --2- 172.21.15.94:0/3844945658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c06afc0 0x7f825c0af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 >> 172.21.15.94:0/3844945658 conn(0x7f825c0ca740 msgr2=0x7f825c0c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 shutdown_connections 2024-03-06T07:24:36.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.095+0000 7f8261917640 1 -- 172.21.15.94:0/3844945658 wait complete. 2024-03-06T07:24:36.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:24:36.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.213+0000 7f0656d0d640 1 Processor -- start 2024-03-06T07:24:36.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.213+0000 7f0656d0d640 1 -- start start 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f0656d0d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f06500a8290 0x7f06501539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f0656d0d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f06500701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f0656d0d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f06500c69a0 0x7f0650072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f0656d0d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0650153f00 con 0x7f06500c69a0 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f0656d0d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06500c62a0 con 0x7f06500a8290 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f0656d0d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06500706e0 con 0x7f06500c4740 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f06500c69a0 0x7f0650072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f064ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f06500701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f06500c69a0 0x7f0650072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53010/0 (socket says 172.21.15.94:53010) 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 -- 172.21.15.94:0/48547205 learned_addr learned my addr 172.21.15.94:0/48547205 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f0654aa9640 1 --2- 172.21.15.94:0/48547205 >> v2:172.21.15.145:3300/0 conn(0x7f06500a8290 0x7f06501539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 -- 172.21.15.94:0/48547205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 msgr2=0x7f06500701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 --2- 172.21.15.94:0/48547205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f06500701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 -- 172.21.15.94:0/48547205 >> v2:172.21.15.145:3300/0 conn(0x7f06500a8290 msgr2=0x7f06501539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 --2- 172.21.15.94:0/48547205 >> v2:172.21.15.145:3300/0 conn(0x7f06500a8290 0x7f06501539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.214+0000 7f06552aa640 1 -- 172.21.15.94:0/48547205 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500c4c00 con 0x7f06500c69a0 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.215+0000 7f06552aa640 1 --2- 172.21.15.94:0/48547205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500c69a0 0x7f0650072680 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f06500735b0 tx=0x7f064002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352e488c8e8197f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.215+0000 7f064f7fe640 1 -- 172.21.15.94:0/48547205 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640038070 con 0x7f06500c69a0 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.215+0000 7f064f7fe640 1 -- 172.21.15.94:0/48547205 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064002f050 con 0x7f06500c69a0 2024-03-06T07:24:36.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.215+0000 7f064f7fe640 1 -- 172.21.15.94:0/48547205 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640031970 con 0x7f06500c69a0 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 -- 172.21.15.94:0/48547205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500c69a0 msgr2=0x7f0650072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 --2- 172.21.15.94:0/48547205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500c69a0 0x7f0650072680 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f06500735b0 tx=0x7f064002b7f0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 -- 172.21.15.94:0/48547205 shutdown_connections 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 --2- 172.21.15.94:0/48547205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500c69a0 0x7f0650072680 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 --2- 172.21.15.94:0/48547205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f06500701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 --2- 172.21.15.94:0/48547205 >> v2:172.21.15.145:3300/0 conn(0x7f06500a8290 0x7f06501539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 -- 172.21.15.94:0/48547205 >> 172.21.15.94:0/48547205 conn(0x7f06500c5560 msgr2=0x7f06500c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 -- 172.21.15.94:0/48547205 shutdown_connections 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.216+0000 7f0656d0d640 1 -- 172.21.15.94:0/48547205 wait complete. 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.217+0000 7f0656d0d640 1 Processor -- start 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.217+0000 7f0656d0d640 1 -- start start 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0656d0d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500a8290 0x7f065014a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0656d0d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f065014c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0656d0d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06500c69a0 0x7f06501ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0656d0d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065006fc80 con 0x7f06500a8290 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0656d0d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065006f9a0 con 0x7f06500c69a0 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0656d0d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06500c6e10 con 0x7f06500c4740 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500a8290 0x7f065014a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500a8290 0x7f065014a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53016/0 (socket says 172.21.15.94:53016) 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 -- 172.21.15.94:0/4170545098 learned_addr learned my addr 172.21.15.94:0/4170545098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f064ffff640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f065014c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f06552aa640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06500c69a0 0x7f06501ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 -- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 msgr2=0x7f065014c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f065014c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 -- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06500c69a0 msgr2=0x7f06501ebf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06500c69a0 0x7f06501ebf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 -- 172.21.15.94:0/4170545098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065014d5f0 con 0x7f06500a8290 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f0654aa9640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500a8290 0x7f065014a2f0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f064800efc0 tx=0x7f064800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f064d7fa640 1 -- 172.21.15.94:0/4170545098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648018020 con 0x7f06500a8290 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.218+0000 7f064d7fa640 1 -- 172.21.15.94:0/4170545098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648014480 con 0x7f06500a8290 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.219+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065014d8d0 con 0x7f06500a8290 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.219+0000 7f064d7fa640 1 -- 172.21.15.94:0/4170545098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0648014870 con 0x7f06500a8290 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.219+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06500aec80 con 0x7f06500a8290 2024-03-06T07:24:36.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.219+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0650150d30 con 0x7f06500a8290 2024-03-06T07:24:36.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.220+0000 7f064d7fa640 1 -- 172.21.15.94:0/4170545098 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f064801b020 con 0x7f06500a8290 2024-03-06T07:24:36.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.221+0000 7f064d7fa640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f063c06b4d0 0x7f063c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.221+0000 7f064ffff640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f063c06b4d0 0x7f063c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.221+0000 7f064d7fa640 1 -- 172.21.15.94:0/4170545098 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f064808b580 con 0x7f06500a8290 2024-03-06T07:24:36.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.221+0000 7f064ffff640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f063c06b4d0 0x7f063c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f06500c2cf0 tx=0x7f0644009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.225+0000 7f064d7fa640 1 -- 172.21.15.94:0/4170545098 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0648022020 con 0x7f06500a8290 2024-03-06T07:24:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.409+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06500ae270 con 0x7f06500a8290 2024-03-06T07:24:36.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.409+0000 7f064d7fa640 1 -- 172.21.15.94:0/4170545098 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v154) v1 ==== 105+0+11 (secure 0 0 0) 0x7f0648010040 con 0x7f06500a8290 2024-03-06T07:24:36.411 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:24:36.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f063c06b4d0 msgr2=0x7f063c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f063c06b4d0 0x7f063c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f06500c2cf0 tx=0x7f0644009190 comp rx=0 tx=0).stop 2024-03-06T07:24:36.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500a8290 msgr2=0x7f065014a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500a8290 0x7f065014a2f0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f064800efc0 tx=0x7f064800c490 comp rx=0 tx=0).stop 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 shutdown_connections 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f063c06b4d0 0x7f063c06d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06500c69a0 0x7f06501ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06500c4740 0x7f065014c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 --2- 172.21.15.94:0/4170545098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f06500a8290 0x7f065014a2f0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 >> 172.21.15.94:0/4170545098 conn(0x7f06500c5560 msgr2=0x7f06500aa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 shutdown_connections 2024-03-06T07:24:36.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.411+0000 7f0656d0d640 1 -- 172.21.15.94:0/4170545098 wait complete. 2024-03-06T07:24:36.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:24:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.539+0000 7f9ee4622640 1 Processor -- start 2024-03-06T07:24:36.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.539+0000 7f9ee4622640 1 -- start start 2024-03-06T07:24:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee4622640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9edc06ca70 0x7f9edc14e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee4622640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc0b0450 con 0x7f9edc155c30 2024-03-06T07:24:36.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee4622640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc1560e0 con 0x7f9edc0b0070 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee4622640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc06c020 con 0x7f9edc06ca70 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee2bbf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9edc06ca70 0x7f9edc14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee1bbd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9edc155c30 0x7f9edc0c5380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51874/0 (socket says 172.21.15.94:51874) 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee1bbd640 1 -- 172.21.15.94:0/1880499865 learned_addr learned my addr 172.21.15.94:0/1880499865 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee13bc640 1 -- 172.21.15.94:0/1880499865 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972708834 0 0) 0x7f9ed8001110 con 0x7f9edc155c30 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee13bc640 1 -- 172.21.15.94:0/1880499865 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec8003520 con 0x7f9edc155c30 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee13bc640 1 -- 172.21.15.94:0/1880499865 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273132436 0 0) 0x7f9edc1560e0 con 0x7f9edc0b0070 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee2bbf640 1 -- 172.21.15.94:0/1880499865 >> v1:172.21.15.145:6789/0 conn(0x7f9edc0b0070 legacy=0x7f9edc0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee2bbf640 1 -- 172.21.15.94:0/1880499865 >> v1:172.21.15.94:6789/0 conn(0x7f9edc155c30 legacy=0x7f9edc0c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.540+0000 7f9ee2bbf640 1 -- 172.21.15.94:0/1880499865 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc0c77f0 con 0x7f9edc06ca70 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.541+0000 7f9ee2bbf640 1 --2- 172.21.15.94:0/1880499865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc06ca70 0x7f9edc14e800 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009d90 tx=0x7f9ed402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418b30b9ac2c4bf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.541+0000 7f9ee13bc640 1 -- 172.21.15.94:0/1880499865 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4039070 con 0x7f9edc06ca70 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.541+0000 7f9ee13bc640 1 -- 172.21.15.94:0/1880499865 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed402cc60 con 0x7f9edc06ca70 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.541+0000 7f9ee13bc640 1 -- 172.21.15.94:0/1880499865 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4034660 con 0x7f9edc06ca70 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.542+0000 7f9ee4622640 1 -- 172.21.15.94:0/1880499865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc06ca70 msgr2=0x7f9edc14e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.542+0000 7f9ee4622640 1 --2- 172.21.15.94:0/1880499865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc06ca70 0x7f9edc14e800 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009d90 tx=0x7f9ed402beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.542+0000 7f9ee4622640 1 -- 172.21.15.94:0/1880499865 shutdown_connections 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.542+0000 7f9ee4622640 1 --2- 172.21.15.94:0/1880499865 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc06ca70 0x7f9edc14e800 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.542+0000 7f9ee4622640 1 -- 172.21.15.94:0/1880499865 >> 172.21.15.94:0/1880499865 conn(0x7f9edc0ac590 msgr2=0x7f9edc0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.542+0000 7f9ee4622640 1 -- 172.21.15.94:0/1880499865 shutdown_connections 2024-03-06T07:24:36.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.542+0000 7f9ee4622640 1 -- 172.21.15.94:0/1880499865 wait complete. 2024-03-06T07:24:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.543+0000 7f9ee4622640 1 Processor -- start 2024-03-06T07:24:36.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.543+0000 7f9ee4622640 1 -- start start 2024-03-06T07:24:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee4622640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9edc06ca70 0x7f9edc156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee4622640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc0b0070 0x7f9edc154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee4622640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9edc155c30 0x7f9edc1edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee4622640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc06c550 con 0x7f9edc0b0070 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee4622640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc06c270 con 0x7f9edc06ca70 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee4622640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc06da80 con 0x7f9edc155c30 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc0b0070 0x7f9edc154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc0b0070 0x7f9edc154200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53046/0 (socket says 172.21.15.94:53046) 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 -- 172.21.15.94:0/2248333046 learned_addr learned my addr 172.21.15.94:0/2248333046 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee23be640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9edc06ca70 0x7f9edc156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee2bbf640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9edc155c30 0x7f9edc1edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 -- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9edc155c30 msgr2=0x7f9edc1edb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9edc155c30 0x7f9edc1edb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 -- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9edc06ca70 msgr2=0x7f9edc156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9edc06ca70 0x7f9edc156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 -- 172.21.15.94:0/2248333046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc1573c0 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ee1bbd640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc0b0070 0x7f9edc154200 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800c760 tx=0x7f9ed800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.544+0000 7f9ed2ffd640 1 -- 172.21.15.94:0/2248333046 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8017070 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.545+0000 7f9ed2ffd640 1 -- 172.21.15.94:0/2248333046 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed8005ad0 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.545+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc1ea830 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.545+0000 7f9ed2ffd640 1 -- 172.21.15.94:0/2248333046 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8007e20 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.545+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc154740 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.546+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc04f330 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.546+0000 7f9ed2ffd640 1 -- 172.21.15.94:0/2248333046 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ed801a070 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.546+0000 7f9ed2ffd640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec006b4d0 0x7f9ec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.546+0000 7f9ed2ffd640 1 -- 172.21.15.94:0/2248333046 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f9ed8089a80 con 0x7f9edc0b0070 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.547+0000 7f9ee23be640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec006b4d0 0x7f9ec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.547+0000 7f9ee23be640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec006b4d0 0x7f9ec006d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc000f60 tx=0x7f9ecc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.551+0000 7f9ed2ffd640 1 -- 172.21.15.94:0/2248333046 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9edc0b0450 con 0x7f9edc0b0070 2024-03-06T07:24:36.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.729+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f9edc06d860 con 0x7f9edc0b0070 2024-03-06T07:24:36.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.729+0000 7f9ed2ffd640 1 -- 172.21.15.94:0/2248333046 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v154) v1 ==== 101+0+11 (secure 0 0 0) 0x7f9ed8057d70 con 0x7f9edc0b0070 2024-03-06T07:24:36.731 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:24:36.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec006b4d0 msgr2=0x7f9ec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec006b4d0 0x7f9ec006d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc000f60 tx=0x7f9ecc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc0b0070 msgr2=0x7f9edc154200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc0b0070 0x7f9edc154200 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800c760 tx=0x7f9ed800cc20 comp rx=0 tx=0).stop 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 shutdown_connections 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9ec006b4d0 0x7f9ec006d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9edc155c30 0x7f9edc1edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc0b0070 0x7f9edc154200 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 --2- 172.21.15.94:0/2248333046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9edc06ca70 0x7f9edc156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 >> 172.21.15.94:0/2248333046 conn(0x7f9edc0ac590 msgr2=0x7f9edc0ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.731+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 shutdown_connections 2024-03-06T07:24:36.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.732+0000 7f9ee4622640 1 -- 172.21.15.94:0/2248333046 wait complete. 2024-03-06T07:24:36.744 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:36.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.843+0000 7f6e8641c640 1 Processor -- start 2024-03-06T07:24:36.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.843+0000 7f6e8641c640 1 -- start start 2024-03-06T07:24:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e8641c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6e8014f360 0x7f6e80155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e8641c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e800c88c0 0x7f6e800c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e8641c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e800c73d0 con 0x7f6e800c5880 2024-03-06T07:24:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e8641c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e800c8dd0 con 0x7f6e800c88c0 2024-03-06T07:24:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e8641c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e800be2e0 con 0x7f6e8014f360 2024-03-06T07:24:36.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e7f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e800c88c0 0x7f6e800c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e7f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e800c88c0 0x7f6e800c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53054/0 (socket says 172.21.15.94:53054) 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e7f7fe640 1 -- 172.21.15.94:0/3584974176 learned_addr learned my addr 172.21.15.94:0/3584974176 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.844+0000 7f6e7ffff640 1 --2- 172.21.15.94:0/3584974176 >> v2:172.21.15.145:3300/0 conn(0x7f6e8014f360 0x7f6e80155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.845+0000 7f6e7f7fe640 1 -- 172.21.15.94:0/3584974176 >> v2:172.21.15.145:3300/0 conn(0x7f6e8014f360 msgr2=0x7f6e80155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.845+0000 7f6e7f7fe640 1 --2- 172.21.15.94:0/3584974176 >> v2:172.21.15.145:3300/0 conn(0x7f6e8014f360 0x7f6e80155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.845+0000 7f6e7f7fe640 1 -- 172.21.15.94:0/3584974176 >> v1:172.21.15.145:6789/0 conn(0x7f6e800c5880 legacy=0x7f6e80158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.845+0000 7f6e7f7fe640 1 -- 172.21.15.94:0/3584974176 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e800c7670 con 0x7f6e800c88c0 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.845+0000 7f6e7f7fe640 1 --2- 172.21.15.94:0/3584974176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e800c88c0 0x7f6e800c5340 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f6e8008c840 tx=0x7f6e7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e8575b45df869f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e7effd640 1 -- 172.21.15.94:0/3584974176 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74035020 con 0x7f6e800c88c0 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e7effd640 1 -- 172.21.15.94:0/3584974176 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e74031870 con 0x7f6e800c88c0 2024-03-06T07:24:36.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e7effd640 1 -- 172.21.15.94:0/3584974176 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74031c40 con 0x7f6e800c88c0 2024-03-06T07:24:36.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 -- 172.21.15.94:0/3584974176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e800c88c0 msgr2=0x7f6e800c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 --2- 172.21.15.94:0/3584974176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e800c88c0 0x7f6e800c5340 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f6e8008c840 tx=0x7f6e7402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 -- 172.21.15.94:0/3584974176 shutdown_connections 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 --2- 172.21.15.94:0/3584974176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e800c88c0 0x7f6e800c5340 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 --2- 172.21.15.94:0/3584974176 >> v2:172.21.15.145:3300/0 conn(0x7f6e8014f360 0x7f6e80155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 -- 172.21.15.94:0/3584974176 >> 172.21.15.94:0/3584974176 conn(0x7f6e800ac570 msgr2=0x7f6e800ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 -- 172.21.15.94:0/3584974176 shutdown_connections 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.846+0000 7f6e8641c640 1 -- 172.21.15.94:0/3584974176 wait complete. 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.847+0000 7f6e8641c640 1 Processor -- start 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e8641c640 1 -- start start 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e8641c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e800c5880 0x7f6e8014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e8641c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e800c88c0 0x7f6e8014bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e8641c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e8014f360 0x7f6e801ee070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e8641c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80159270 con 0x7f6e8014f360 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e8641c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800c7c30 con 0x7f6e800c5880 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e8641c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e801593f0 con 0x7f6e800c88c0 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e849b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e8014f360 0x7f6e801ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e7ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e800c5880 0x7f6e8014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e849b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e8014f360 0x7f6e801ee070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53066/0 (socket says 172.21.15.94:53066) 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e7ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e800c5880 0x7f6e8014b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:53704/0 (socket says 172.21.15.94:53704) 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e849b9640 1 -- 172.21.15.94:0/60379733 learned_addr learned my addr 172.21.15.94:0/60379733 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.848+0000 7f6e7f7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e800c88c0 0x7f6e8014bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7ffff640 1 -- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e800c88c0 msgr2=0x7f6e8014bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7ffff640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e800c88c0 0x7f6e8014bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7ffff640 1 -- 172.21.15.94:0/60379733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e8014f360 msgr2=0x7f6e801ee070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7ffff640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e8014f360 0x7f6e801ee070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7ffff640 1 -- 172.21.15.94:0/60379733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e800c1470 con 0x7f6e800c5880 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e849b9640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e8014f360 0x7f6e801ee070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7ffff640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e800c5880 0x7f6e8014b790 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000e870 tx=0x7f6e7000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7cff9640 1 -- 172.21.15.94:0/60379733 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70017070 con 0x7f6e800c5880 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7cff9640 1 -- 172.21.15.94:0/60379733 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7000ee60 con 0x7f6e800c5880 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e7cff9640 1 -- 172.21.15.94:0/60379733 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000cb60 con 0x7f6e800c5880 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e800af750 con 0x7f6e800c5880 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.849+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e801eecd0 con 0x7f6e800c5880 2024-03-06T07:24:36.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.850+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8004f330 con 0x7f6e800c5880 2024-03-06T07:24:36.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.850+0000 7f6e7cff9640 1 -- 172.21.15.94:0/60379733 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e7001a070 con 0x7f6e800c5880 2024-03-06T07:24:36.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.851+0000 7f6e7cff9640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.851+0000 7f6e7cff9640 1 -- 172.21.15.94:0/60379733 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f6e7008a440 con 0x7f6e800c5880 2024-03-06T07:24:36.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.851+0000 7f6e7f7fe640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.851+0000 7f6e7f7fe640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6e74036000 tx=0x7f6e7403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:36.855+0000 7f6e7cff9640 1 -- 172.21.15.94:0/60379733 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e700587c0 con 0x7f6e800c5880 2024-03-06T07:24:37.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.071+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e8006cda0 con 0x7f6e800c5880 2024-03-06T07:24:37.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.071+0000 7f6e7cff9640 1 -- 172.21.15.94:0/60379733 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+6278 (secure 0 0 0) 0x7f6e7005c860 con 0x7f6e800c5880 2024-03-06T07:24:37.073 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:37.073 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":146,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":146},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:26.705762+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8155,"qdb_cluster":[8155]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:37.074 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 146 2024-03-06T07:24:37.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6c06b4d0 msgr2=0x7f6e6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6e74036000 tx=0x7f6e7403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e800c5880 msgr2=0x7f6e8014b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e800c5880 0x7f6e8014b790 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000e870 tx=0x7f6e7000ed30 comp rx=0 tx=0).stop 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 shutdown_connections 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6c06b4d0 0x7f6e6c06d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e8014f360 0x7f6e801ee070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e800c88c0 0x7f6e8014bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 --2- 172.21.15.94:0/60379733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e800c5880 0x7f6e8014b790 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.073+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 >> 172.21.15.94:0/60379733 conn(0x7f6e800ac570 msgr2=0x7f6e800c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.074+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 shutdown_connections 2024-03-06T07:24:37.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.074+0000 7f6e8641c640 1 -- 172.21.15.94:0/60379733 wait complete. 2024-03-06T07:24:37.086 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:37.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.185+0000 7f133c15e640 1 Processor -- start 2024-03-06T07:24:37.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f133c15e640 1 -- start start 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f133c15e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13341527b0 0x7f1334159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f133c15e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f13340c88b0 0x7f13340c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f133c15e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13340c7c60 con 0x7f13340c7180 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f133c15e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1334159640 con 0x7f13341527b0 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f133c15e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13340c8060 con 0x7f13340c88b0 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f1339efa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13341527b0 0x7f1334159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f1339efa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13341527b0 0x7f1334159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53074/0 (socket says 172.21.15.94:53074) 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.186+0000 7f1339efa640 1 -- 172.21.15.94:0/1968687196 learned_addr learned my addr 172.21.15.94:0/1968687196 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.187+0000 7f133a6fb640 1 --2- 172.21.15.94:0/1968687196 >> v2:172.21.15.145:3300/0 conn(0x7f13340c88b0 0x7f13340c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.187+0000 7f1339efa640 1 -- 172.21.15.94:0/1968687196 >> v2:172.21.15.145:3300/0 conn(0x7f13340c88b0 msgr2=0x7f13340c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.187+0000 7f1339efa640 1 --2- 172.21.15.94:0/1968687196 >> v2:172.21.15.145:3300/0 conn(0x7f13340c88b0 0x7f13340c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.187+0000 7f1339efa640 1 -- 172.21.15.94:0/1968687196 >> v1:172.21.15.145:6789/0 conn(0x7f13340c7180 legacy=0x7f1334155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.187+0000 7f1339efa640 1 -- 172.21.15.94:0/1968687196 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13340c76a0 con 0x7f13341527b0 2024-03-06T07:24:37.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.187+0000 7f1339efa640 1 --2- 172.21.15.94:0/1968687196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f1334159100 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f13340ad600 tx=0x7f133002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14a8abee4ae27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f1338ef8640 1 -- 172.21.15.94:0/1968687196 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330035020 con 0x7f13341527b0 2024-03-06T07:24:37.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f1338ef8640 1 -- 172.21.15.94:0/1968687196 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1330031870 con 0x7f13341527b0 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f1338ef8640 1 -- 172.21.15.94:0/1968687196 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330031c40 con 0x7f13341527b0 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 -- 172.21.15.94:0/1968687196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 msgr2=0x7f1334159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 --2- 172.21.15.94:0/1968687196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f1334159100 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f13340ad600 tx=0x7f133002b910 comp rx=0 tx=0).stop 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 -- 172.21.15.94:0/1968687196 shutdown_connections 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 --2- 172.21.15.94:0/1968687196 >> v2:172.21.15.145:3300/0 conn(0x7f13340c88b0 0x7f13340c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 --2- 172.21.15.94:0/1968687196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f1334159100 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 -- 172.21.15.94:0/1968687196 >> 172.21.15.94:0/1968687196 conn(0x7f13340ac560 msgr2=0x7f13340ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 -- 172.21.15.94:0/1968687196 shutdown_connections 2024-03-06T07:24:37.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.188+0000 7f133c15e640 1 -- 172.21.15.94:0/1968687196 wait complete. 2024-03-06T07:24:37.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.189+0000 7f133c15e640 1 Processor -- start 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.189+0000 7f133c15e640 1 -- start start 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133c15e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13340c7180 0x7f1334154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133c15e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13340c88b0 0x7f13340be6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133c15e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f13341564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133c15e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334155600 con 0x7f13341527b0 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133c15e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334155320 con 0x7f13340c7180 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133c15e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1334152b90 con 0x7f13340c88b0 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f13341564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f13341564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53088/0 (socket says 172.21.15.94:53088) 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 -- 172.21.15.94:0/1601119305 learned_addr learned my addr 172.21.15.94:0/1601119305 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f13396f9640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13340c88b0 0x7f13340be6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f1339efa640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13340c7180 0x7f1334154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 -- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13340c88b0 msgr2=0x7f13340be6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13340c88b0 0x7f13340be6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 -- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13340c7180 msgr2=0x7f1334154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13340c7180 0x7f1334154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.190+0000 7f133a6fb640 1 -- 172.21.15.94:0/1601119305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13341583e0 con 0x7f13341527b0 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.191+0000 7f133a6fb640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f13341564f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f132c00efc0 tx=0x7f132c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.191+0000 7f132a7fc640 1 -- 172.21.15.94:0/1601119305 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c018020 con 0x7f13341527b0 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.191+0000 7f132a7fc640 1 -- 172.21.15.94:0/1601119305 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c014480 con 0x7f13341527b0 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.191+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13341ee9a0 con 0x7f13341527b0 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.191+0000 7f132a7fc640 1 -- 172.21.15.94:0/1601119305 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c014870 con 0x7f13341527b0 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.191+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1334154640 con 0x7f13341527b0 2024-03-06T07:24:37.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.192+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133404f330 con 0x7f13341527b0 2024-03-06T07:24:37.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.192+0000 7f132a7fc640 1 -- 172.21.15.94:0/1601119305 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f132c005d10 con 0x7f13341527b0 2024-03-06T07:24:37.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.193+0000 7f132a7fc640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.193+0000 7f132a7fc640 1 -- 172.21.15.94:0/1601119305 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f132c08afe0 con 0x7f13341527b0 2024-03-06T07:24:37.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.193+0000 7f1339efa640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.193+0000 7f1339efa640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f131806b500 0x7f131806d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1330036000 tx=0x7f133003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.197+0000 7f132a7fc640 1 -- 172.21.15.94:0/1601119305 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f132c022070 con 0x7f13341527b0 2024-03-06T07:24:37.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.375+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f133414f630 con 0x7f13341527b0 2024-03-06T07:24:37.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.376+0000 7f132a7fc640 1 -- 172.21.15.94:0/1601119305 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+20731 (secure 0 0 0) 0x7f132c010040 con 0x7f13341527b0 2024-03-06T07:24:37.378 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:37.378 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":154,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:23.693537+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/1613596341":"2024-03-07T07:24:23.693503+0000","172.21.15.94:6833/1613596341":"2024-03-07T07:24:23.693503+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f131806b500 msgr2=0x7f131806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f131806b500 0x7f131806d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1330036000 tx=0x7f133003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 msgr2=0x7f13341564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f13341564f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f132c00efc0 tx=0x7f132c00c490 comp rx=0 tx=0).stop 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 shutdown_connections 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f131806b500 0x7f131806d9c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13341527b0 0x7f13341564f0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13340c88b0 0x7f13340be6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 --2- 172.21.15.94:0/1601119305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13340c7180 0x7f1334154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 >> 172.21.15.94:0/1601119305 conn(0x7f13340ac560 msgr2=0x7f133414ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 shutdown_connections 2024-03-06T07:24:37.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.378+0000 7f133c15e640 1 -- 172.21.15.94:0/1601119305 wait complete. 2024-03-06T07:24:37.391 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:37.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:37.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.490+0000 7f6247d1e640 1 Processor -- start 2024-03-06T07:24:37.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.490+0000 7f6247d1e640 1 -- start start 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f6247d1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f6240155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f6247d1e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6240158b60 0x7f6240158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f6247d1e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62400c73d0 con 0x7f624014f360 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f6247d1e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62400c8dd0 con 0x7f6240158b60 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f6247d1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62400be2e0 con 0x7f62400c88c0 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62462bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6240158b60 0x7f6240158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62462bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6240158b60 0x7f6240158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53090/0 (socket says 172.21.15.94:53090) 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62462bb640 1 -- 172.21.15.94:0/275807957 learned_addr learned my addr 172.21.15.94:0/275807957 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62452b9640 1 --2- 172.21.15.94:0/275807957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f6240155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62462bb640 1 -- 172.21.15.94:0/275807957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 msgr2=0x7f6240155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62462bb640 1 --2- 172.21.15.94:0/275807957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f6240155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62462bb640 1 -- 172.21.15.94:0/275807957 >> v1:172.21.15.145:6789/0 conn(0x7f624014f360 legacy=0x7f62400c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f6244ab8640 1 -- 172.21.15.94:0/275807957 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726798219 0 0) 0x7f62400c73d0 con 0x7f624014f360 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.491+0000 7f62462bb640 1 -- 172.21.15.94:0/275807957 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62400c7670 con 0x7f6240158b60 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.492+0000 7f62462bb640 1 --2- 172.21.15.94:0/275807957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6240158b60 0x7f6240158ff0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6238009e50 tx=0x7f623802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c483ee5a549a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.492+0000 7f6244ab8640 1 -- 172.21.15.94:0/275807957 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623802ca70 con 0x7f6240158b60 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.492+0000 7f6244ab8640 1 -- 172.21.15.94:0/275807957 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623802cc10 con 0x7f6240158b60 2024-03-06T07:24:37.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.492+0000 7f6244ab8640 1 -- 172.21.15.94:0/275807957 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62380343f0 con 0x7f6240158b60 2024-03-06T07:24:37.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 -- 172.21.15.94:0/275807957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6240158b60 msgr2=0x7f6240158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 --2- 172.21.15.94:0/275807957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6240158b60 0x7f6240158ff0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6238009e50 tx=0x7f623802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 -- 172.21.15.94:0/275807957 shutdown_connections 2024-03-06T07:24:37.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 --2- 172.21.15.94:0/275807957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6240158b60 0x7f6240158ff0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 --2- 172.21.15.94:0/275807957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f6240155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 -- 172.21.15.94:0/275807957 >> 172.21.15.94:0/275807957 conn(0x7f62400ac570 msgr2=0x7f62400ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 -- 172.21.15.94:0/275807957 shutdown_connections 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.493+0000 7f6247d1e640 1 -- 172.21.15.94:0/275807957 wait complete. 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.494+0000 7f6247d1e640 1 Processor -- start 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.494+0000 7f6247d1e640 1 -- start start 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f62400c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624014f360 0x7f624014af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6240158b60 0x7f624014b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400c5a50 con 0x7f624014f360 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62400c7c30 con 0x7f6240158b60 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62400c5bd0 con 0x7f62400c88c0 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624014f360 0x7f624014af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624014f360 0x7f624014af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53102/0 (socket says 172.21.15.94:53102) 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 -- 172.21.15.94:0/3558134842 learned_addr learned my addr 172.21.15.94:0/3558134842 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6245aba640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f62400c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62462bb640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6240158b60 0x7f624014b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 -- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 msgr2=0x7f62400c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f62400c41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 -- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6240158b60 msgr2=0x7f624014b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6240158b60 0x7f624014b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 -- 172.21.15.94:0/3558134842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62401eb790 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62452b9640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624014f360 0x7f624014af90 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f623c00e840 tx=0x7f623c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62367fc640 1 -- 172.21.15.94:0/3558134842 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c017070 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62367fc640 1 -- 172.21.15.94:0/3558134842 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c00c940 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62400c1bc0 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f62367fc640 1 -- 172.21.15.94:0/3558134842 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c00cd10 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.495+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62400c2180 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.496+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624004f330 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.497+0000 7f62367fc640 1 -- 172.21.15.94:0/3558134842 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f623c01a070 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.497+0000 7f62367fc640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f622406b4d0 0x7f622406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.498+0000 7f62367fc640 1 -- 172.21.15.94:0/3558134842 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f623c08a840 con 0x7f624014f360 2024-03-06T07:24:37.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.498+0000 7f6245aba640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f622406b4d0 0x7f622406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.498+0000 7f6245aba640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f622406b4d0 0x7f622406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6230000f60 tx=0x7f6230009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.502+0000 7f62367fc640 1 -- 172.21.15.94:0/3558134842 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f623c058bc0 con 0x7f624014f360 2024-03-06T07:24:37.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.711+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f624006cda0 con 0x7f624014f360 2024-03-06T07:24:37.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.711+0000 7f62367fc640 1 -- 172.21.15.94:0/3558134842 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+6278 (secure 0 0 0) 0x7f623c05cc60 con 0x7f624014f360 2024-03-06T07:24:37.713 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:37.713 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":146,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":146},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:26.705762+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8155,"qdb_cluster":[8155]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:37.714 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 146 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f622406b4d0 msgr2=0x7f622406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f622406b4d0 0x7f622406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6230000f60 tx=0x7f6230009040 comp rx=0 tx=0).stop 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624014f360 msgr2=0x7f624014af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624014f360 0x7f624014af90 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f623c00e840 tx=0x7f623c00ed00 comp rx=0 tx=0).stop 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 shutdown_connections 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f622406b4d0 0x7f622406d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6240158b60 0x7f624014b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f624014f360 0x7f624014af90 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 --2- 172.21.15.94:0/3558134842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62400c88c0 0x7f62400c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 >> 172.21.15.94:0/3558134842 conn(0x7f62400ac570 msgr2=0x7f62400ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 shutdown_connections 2024-03-06T07:24:37.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.713+0000 7f6247d1e640 1 -- 172.21.15.94:0/3558134842 wait complete. 2024-03-06T07:24:37.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:37.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.828+0000 7f6bec68f640 1 Processor -- start 2024-03-06T07:24:37.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.828+0000 7f6bec68f640 1 -- start start 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.829+0000 7f6bec68f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6be4155ba0 0x7f6be40c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.829+0000 7f6bec68f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.829+0000 7f6bec68f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be41591c0 con 0x7f6be40c5480 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.829+0000 7f6bec68f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be406df40 con 0x7f6be4155ba0 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.829+0000 7f6bec68f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be406fdb0 con 0x7f6be41586f0 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.829+0000 7f6be9c2a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6be4155ba0 0x7f6be40c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6bea42b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6be40c5480 0x7f6be406f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51876/0 (socket says 172.21.15.94:51876) 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6be9c2a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6be4155ba0 0x7f6be40c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53114/0 (socket says 172.21.15.94:53114) 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6bea42b640 1 -- 172.21.15.94:0/2722131761 learned_addr learned my addr 172.21.15.94:0/2722131761 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6beac2c640 1 --2- 172.21.15.94:0/2722131761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6be9c2a640 1 -- 172.21.15.94:0/2722131761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 msgr2=0x7f6be4158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6be9c2a640 1 --2- 172.21.15.94:0/2722131761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6be9c2a640 1 -- 172.21.15.94:0/2722131761 >> v1:172.21.15.94:6789/0 conn(0x7f6be40c5480 legacy=0x7f6be406f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6be9429640 1 -- 172.21.15.94:0/2722131761 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951891376 0 0) 0x7f6be41591c0 con 0x7f6be40c5480 2024-03-06T07:24:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.830+0000 7f6be9c2a640 1 -- 172.21.15.94:0/2722131761 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be41594a0 con 0x7f6be4155ba0 2024-03-06T07:24:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.831+0000 7f6be9c2a640 1 --2- 172.21.15.94:0/2722131761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be40c5860 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40097a0 tx=0x7f6bd402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba805cd5189ee2b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.831+0000 7f6be9429640 1 -- 172.21.15.94:0/2722131761 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40315a0 con 0x7f6be4155ba0 2024-03-06T07:24:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.831+0000 7f6be9429640 1 -- 172.21.15.94:0/2722131761 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4037070 con 0x7f6be4155ba0 2024-03-06T07:24:37.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.831+0000 7f6be9429640 1 -- 172.21.15.94:0/2722131761 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4031960 con 0x7f6be4155ba0 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 -- 172.21.15.94:0/2722131761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 msgr2=0x7f6be40c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 --2- 172.21.15.94:0/2722131761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be40c5860 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40097a0 tx=0x7f6bd402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 -- 172.21.15.94:0/2722131761 shutdown_connections 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 --2- 172.21.15.94:0/2722131761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 --2- 172.21.15.94:0/2722131761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be40c5860 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 -- 172.21.15.94:0/2722131761 >> 172.21.15.94:0/2722131761 conn(0x7f6be406b1a0 msgr2=0x7f6be406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 -- 172.21.15.94:0/2722131761 shutdown_connections 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.832+0000 7f6bec68f640 1 -- 172.21.15.94:0/2722131761 wait complete. 2024-03-06T07:24:37.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.834+0000 7f6bec68f640 1 Processor -- start 2024-03-06T07:24:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.834+0000 7f6bec68f640 1 -- start start 2024-03-06T07:24:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bec68f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6be40c5480 0x7f6be41eba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bec68f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be41514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bec68f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bec68f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be4155680 con 0x7f6be4155ba0 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bec68f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be4156000 con 0x7f6be40c5480 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bec68f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be4152a20 con 0x7f6be41586f0 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be41514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be41514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53122/0 (socket says 172.21.15.94:53122) 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 -- 172.21.15.94:0/511215000 learned_addr learned my addr 172.21.15.94:0/511215000 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6beac2c640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bea42b640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6be40c5480 0x7f6be41eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 -- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 msgr2=0x7f6be4156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 -- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6be40c5480 msgr2=0x7f6be41eba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6be40c5480 0x7f6be41eba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 -- 172.21.15.94:0/511215000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be40ad6b0 con 0x7f6be4155ba0 2024-03-06T07:24:37.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6be9c2a640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be41514e0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40098c0 tx=0x7f6bd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bdaffd640 1 -- 172.21.15.94:0/511215000 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4032630 con 0x7f6be4155ba0 2024-03-06T07:24:37.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bdaffd640 1 -- 172.21.15.94:0/511215000 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd40327d0 con 0x7f6be4155ba0 2024-03-06T07:24:37.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bdaffd640 1 -- 172.21.15.94:0/511215000 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4032bc0 con 0x7f6be4155ba0 2024-03-06T07:24:37.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.835+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be41eaf60 con 0x7f6be4155ba0 2024-03-06T07:24:37.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.836+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be4151a20 con 0x7f6be4155ba0 2024-03-06T07:24:37.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.837+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be404f330 con 0x7f6be4155ba0 2024-03-06T07:24:37.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.837+0000 7f6bdaffd640 1 -- 172.21.15.94:0/511215000 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bd404e430 con 0x7f6be4155ba0 2024-03-06T07:24:37.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.838+0000 7f6bdaffd640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6bc806b500 0x7f6bc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.838+0000 7f6bdaffd640 1 -- 172.21.15.94:0/511215000 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f6bd402f030 con 0x7f6be4155ba0 2024-03-06T07:24:37.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.838+0000 7f6bea42b640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6bc806b500 0x7f6bc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.839+0000 7f6bea42b640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6bc806b500 0x7f6bc806d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6be0000f60 tx=0x7f6be0007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:37.842+0000 7f6bdaffd640 1 -- 172.21.15.94:0/511215000 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bd4040070 con 0x7f6be4155ba0 2024-03-06T07:24:38.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.057+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be406d080 con 0x7f6be4155ba0 2024-03-06T07:24:38.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.058+0000 7f6bdaffd640 1 -- 172.21.15.94:0/511215000 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+6278 (secure 0 0 0) 0x7f6bd407a6a0 con 0x7f6be4155ba0 2024-03-06T07:24:38.060 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:38.060 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":146,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8152,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":146},{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:03.229941+0000","modified":"2024-03-06T07:24:26.705762+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8155},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8155":{"gid":8155,"name":"b","rank":0,"incarnation":142,"state":"up:active","state_seq":10,"addr":"172.21.15.145:6837/486493869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":486493869},{"type":"v1","addr":"172.21.15.145:6837","nonce":486493869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8155,"qdb_cluster":[8155]},"id":11},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:25.285179+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8769":{"gid":8769,"name":"a","rank":0,"incarnation":136,"state":"up:active","state_seq":8,"addr":"172.21.15.94:6834/1222126706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1222126706},{"type":"v1","addr":"172.21.15.94:6834","nonce":1222126706}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8769,"qdb_cluster":[8769]},"id":12}]} 2024-03-06T07:24:38.060 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 146 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6bc806b500 msgr2=0x7f6bc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6bc806b500 0x7f6bc806d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6be0000f60 tx=0x7f6be0007d50 comp rx=0 tx=0).stop 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 msgr2=0x7f6be41514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be41514e0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40098c0 tx=0x7f6bd40046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 shutdown_connections 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6bc806b500 0x7f6bc806d9c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6be41586f0 0x7f6be4156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be4155ba0 0x7f6be41514e0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 --2- 172.21.15.94:0/511215000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6be40c5480 0x7f6be41eba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 >> 172.21.15.94:0/511215000 conn(0x7f6be406b1a0 msgr2=0x7f6be4153640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 shutdown_connections 2024-03-06T07:24:38.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.060+0000 7f6bec68f640 1 -- 172.21.15.94:0/511215000 wait complete. 2024-03-06T07:24:38.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.172+0000 7efc7ae81640 1 Processor -- start 2024-03-06T07:24:38.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.172+0000 7efc7ae81640 1 -- start start 2024-03-06T07:24:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc7ae81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc7414f360 0x7efc74155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc7ae81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc740c88c0 0x7efc740c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc7ae81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc740c73d0 con 0x7efc740c5880 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc7ae81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc740c8dd0 con 0x7efc7414f360 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc7ae81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc740be2e0 con 0x7efc740c88c0 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc73fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc740c88c0 0x7efc740c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc78c1d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc7414f360 0x7efc74155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc73fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc740c88c0 0x7efc740c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:43734/0 (socket says 172.21.15.94:43734) 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc78c1d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc7414f360 0x7efc74155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:53752/0 (socket says 172.21.15.94:53752) 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc7941e640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7efc740c5880 0x7efc74158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:56970/0 (socket says 172.21.15.94:56970) 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc73fff640 1 -- 172.21.15.94:0/1993347802 learned_addr learned my addr 172.21.15.94:0/1993347802 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc73fff640 1 -- 172.21.15.94:0/1993347802 >> v2:172.21.15.145:3300/0 conn(0x7efc7414f360 msgr2=0x7efc74155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc73fff640 1 --2- 172.21.15.94:0/1993347802 >> v2:172.21.15.145:3300/0 conn(0x7efc7414f360 0x7efc74155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc73fff640 1 -- 172.21.15.94:0/1993347802 >> v1:172.21.15.145:6789/0 conn(0x7efc740c5880 legacy=0x7efc74158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc73fff640 1 -- 172.21.15.94:0/1993347802 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc740c7670 con 0x7efc740c88c0 2024-03-06T07:24:38.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.173+0000 7efc78c1d640 1 --2- 172.21.15.94:0/1993347802 >> v2:172.21.15.145:3300/0 conn(0x7efc7414f360 0x7efc74155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.174+0000 7efc73fff640 1 --2- 172.21.15.94:0/1993347802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc740c88c0 0x7efc740c5340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc60009870 tx=0x7efc6002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc0d118fcbc735d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc737fe640 1 -- 172.21.15.94:0/1993347802 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60035020 con 0x7efc740c88c0 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc737fe640 1 -- 172.21.15.94:0/1993347802 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc60031640 con 0x7efc740c88c0 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc737fe640 1 -- 172.21.15.94:0/1993347802 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6002d040 con 0x7efc740c88c0 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 -- 172.21.15.94:0/1993347802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc740c88c0 msgr2=0x7efc740c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1993347802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc740c88c0 0x7efc740c5340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc60009870 tx=0x7efc6002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 -- 172.21.15.94:0/1993347802 shutdown_connections 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1993347802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc740c88c0 0x7efc740c5340 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1993347802 >> v2:172.21.15.145:3300/0 conn(0x7efc7414f360 0x7efc74155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 -- 172.21.15.94:0/1993347802 >> 172.21.15.94:0/1993347802 conn(0x7efc740ac570 msgr2=0x7efc740ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 -- 172.21.15.94:0/1993347802 shutdown_connections 2024-03-06T07:24:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.175+0000 7efc7ae81640 1 -- 172.21.15.94:0/1993347802 wait complete. 2024-03-06T07:24:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.176+0000 7efc7ae81640 1 Processor -- start 2024-03-06T07:24:38.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.176+0000 7efc7ae81640 1 -- start start 2024-03-06T07:24:38.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc7ae81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c5880 0x7efc7414db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc7ae81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc740c88c0 0x7efc740c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc7ae81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc7414f360 0x7efc7414abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc7ae81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74159270 con 0x7efc740c88c0 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc7ae81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740c7c30 con 0x7efc740c5880 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc7ae81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc741593f0 con 0x7efc7414f360 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc740c88c0 0x7efc740c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc78c1d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c5880 0x7efc7414db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc740c88c0 0x7efc740c1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53126/0 (socket says 172.21.15.94:53126) 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 -- 172.21.15.94:0/1333411423 learned_addr learned my addr 172.21.15.94:0/1333411423 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc7941e640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc7414f360 0x7efc7414abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 -- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc7414f360 msgr2=0x7efc7414abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc7414f360 0x7efc7414abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 -- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c5880 msgr2=0x7efc7414db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c5880 0x7efc7414db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.177+0000 7efc73fff640 1 -- 172.21.15.94:0/1333411423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc741522c0 con 0x7efc740c88c0 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.178+0000 7efc73fff640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc740c88c0 0x7efc740c1a90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.178+0000 7efc717fa640 1 -- 172.21.15.94:0/1333411423 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60040070 con 0x7efc740c88c0 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.178+0000 7efc717fa640 1 -- 172.21.15.94:0/1333411423 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc600399f0 con 0x7efc740c88c0 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.178+0000 7efc717fa640 1 -- 172.21.15.94:0/1333411423 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60035040 con 0x7efc740c88c0 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.178+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc741525a0 con 0x7efc740c88c0 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.178+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc741eed50 con 0x7efc740c88c0 2024-03-06T07:24:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.179+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7404f330 con 0x7efc740c88c0 2024-03-06T07:24:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.180+0000 7efc717fa640 1 -- 172.21.15.94:0/1333411423 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc6004b060 con 0x7efc740c88c0 2024-03-06T07:24:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.180+0000 7efc717fa640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc5806b4d0 0x7efc5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.180+0000 7efc717fa640 1 -- 172.21.15.94:0/1333411423 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7efc600abe20 con 0x7efc740c88c0 2024-03-06T07:24:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.180+0000 7efc78c1d640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc5806b4d0 0x7efc5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.181+0000 7efc78c1d640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc5806b4d0 0x7efc5806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7efc68009560 tx=0x7efc68009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.184+0000 7efc717fa640 1 -- 172.21.15.94:0/1333411423 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc6007a1a0 con 0x7efc740c88c0 2024-03-06T07:24:38.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.364+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc740acba0 con 0x7efc740c88c0 2024-03-06T07:24:38.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.365+0000 7efc717fa640 1 -- 172.21.15.94:0/1333411423 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+20731 (secure 0 0 0) 0x7efc6007e240 con 0x7efc740c88c0 2024-03-06T07:24:38.367 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:38.367 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":154,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:23.693537+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:01.443144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:02.192757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/1613596341":"2024-03-07T07:24:23.693503+0000","172.21.15.94:6833/1613596341":"2024-03-07T07:24:23.693503+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:38.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc5806b4d0 msgr2=0x7efc5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc5806b4d0 0x7efc5806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7efc68009560 tx=0x7efc68009190 comp rx=0 tx=0).stop 2024-03-06T07:24:38.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc740c88c0 msgr2=0x7efc740c1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc740c88c0 0x7efc740c1a90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc600046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 shutdown_connections 2024-03-06T07:24:38.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc5806b4d0 0x7efc5806d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc7414f360 0x7efc7414abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc740c88c0 0x7efc740c1a90 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 --2- 172.21.15.94:0/1333411423 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c5880 0x7efc7414db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 >> 172.21.15.94:0/1333411423 conn(0x7efc740ac570 msgr2=0x7efc740ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 shutdown_connections 2024-03-06T07:24:38.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.367+0000 7efc7ae81640 1 -- 172.21.15.94:0/1333411423 wait complete. 2024-03-06T07:24:38.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:38.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.484+0000 7fca8c8c9640 1 Processor -- start 2024-03-06T07:24:38.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.484+0000 7fca8c8c9640 1 -- start start 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8c8c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fca880c69f0 0x7fca880c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8c8c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8c8c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca88155130 con 0x7fca880ad2b0 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8c8c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca88155440 con 0x7fca880c69f0 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8c8c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca880c5d50 con 0x7fca880c8950 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca86d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fca880ad2b0 0x7fca881558d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45818/0 (socket says 172.21.15.94:45818) 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca86d9d640 1 -- 172.21.15.94:0/1055296168 learned_addr learned my addr 172.21.15.94:0/1055296168 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8659c640 1 --2- 172.21.15.94:0/1055296168 >> v2:172.21.15.145:3300/0 conn(0x7fca880c69f0 0x7fca880c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca7ffff640 1 --2- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca85d9b640 1 -- 172.21.15.94:0/1055296168 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895332887 0 0) 0x7fca80001490 con 0x7fca880ad2b0 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca85d9b640 1 -- 172.21.15.94:0/1055296168 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca74003520 con 0x7fca880ad2b0 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8659c640 1 -- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 msgr2=0x7fca880acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8659c640 1 --2- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8659c640 1 -- 172.21.15.94:0/1055296168 >> v1:172.21.15.94:6789/0 conn(0x7fca880ad2b0 legacy=0x7fca881558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.485+0000 7fca8659c640 1 -- 172.21.15.94:0/1055296168 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca880c6030 con 0x7fca880c69f0 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8659c640 1 --2- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880c69f0 0x7fca880c6dd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fca780097a0 tx=0x7fca7802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e8e59d68d211946 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca85d9b640 1 -- 172.21.15.94:0/1055296168 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca780315a0 con 0x7fca880c69f0 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca85d9b640 1 -- 172.21.15.94:0/1055296168 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca78037070 con 0x7fca880c69f0 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca85d9b640 1 -- 172.21.15.94:0/1055296168 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78031960 con 0x7fca880c69f0 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 -- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880c69f0 msgr2=0x7fca880c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880c69f0 0x7fca880c6dd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fca780097a0 tx=0x7fca7802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 -- 172.21.15.94:0/1055296168 shutdown_connections 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/1055296168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880c69f0 0x7fca880c6dd0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 -- 172.21.15.94:0/1055296168 >> 172.21.15.94:0/1055296168 conn(0x7fca880cae70 msgr2=0x7fca880cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 -- 172.21.15.94:0/1055296168 shutdown_connections 2024-03-06T07:24:38.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.487+0000 7fca8c8c9640 1 -- 172.21.15.94:0/1055296168 wait complete. 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 Processor -- start 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 -- start start 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880ad2b0 0x7fca88154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca880c69f0 0x7fca880c1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca88152a20 con 0x7fca880c69f0 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca88155fe0 con 0x7fca880ad2b0 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8c8c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca88152ba0 con 0x7fca880c8950 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca880c69f0 0x7fca880c1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca7ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca880c69f0 0x7fca880c1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39060/0 (socket says 172.21.15.94:39060) 2024-03-06T07:24:38.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca7ffff640 1 -- 172.21.15.94:0/975707915 learned_addr learned my addr 172.21.15.94:0/975707915 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca8659c640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880ad2b0 0x7fca88154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.489+0000 7fca86d9d640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7ffff640 1 -- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 msgr2=0x7fca880c22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7ffff640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880c22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7ffff640 1 -- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880ad2b0 msgr2=0x7fca88154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7ffff640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880ad2b0 0x7fca88154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7ffff640 1 -- 172.21.15.94:0/975707915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8806d030 con 0x7fca880c69f0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7ffff640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca880c69f0 0x7fca880c1d70 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e9e0 tx=0x7fca7000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7effd640 1 -- 172.21.15.94:0/975707915 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70017070 con 0x7fca880c69f0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7effd640 1 -- 172.21.15.94:0/975707915 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7000c9d0 con 0x7fca880c69f0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca881edfd0 con 0x7fca880c69f0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca7effd640 1 -- 172.21.15.94:0/975707915 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7000cdc0 con 0x7fca880c69f0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.490+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca881ee590 con 0x7fca880c69f0 2024-03-06T07:24:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.491+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8804f330 con 0x7fca880c69f0 2024-03-06T07:24:38.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.492+0000 7fca7effd640 1 -- 172.21.15.94:0/975707915 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca7001a070 con 0x7fca880c69f0 2024-03-06T07:24:38.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.492+0000 7fca7effd640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca6806b4d0 0x7fca6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.492+0000 7fca7effd640 1 -- 172.21.15.94:0/975707915 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fca7008a640 con 0x7fca880c69f0 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.492+0000 7fca8659c640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca6806b4d0 0x7fca6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.492+0000 7fca8659c640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca6806b4d0 0x7fca6806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fca78036000 tx=0x7fca7803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.500+0000 7fca7effd640 1 -- 172.21.15.94:0/975707915 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca700589c0 con 0x7fca880c69f0 2024-03-06T07:24:38.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.715+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fca880ca650 con 0x7fca880c69f0 2024-03-06T07:24:38.717 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:24:38.715+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:38.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.719+0000 7fca7effd640 1 -- 172.21.15.94:0/975707915 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v147) v1 ==== 161+0+0 (secure 0 0 0) 0x7fca7005ca60 con 0x7fca880c69f0 2024-03-06T07:24:38.720 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca6806b4d0 msgr2=0x7fca6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca6806b4d0 0x7fca6806d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fca78036000 tx=0x7fca7803d040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca880c69f0 msgr2=0x7fca880c1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca880c69f0 0x7fca880c1d70 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e9e0 tx=0x7fca7000eea0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 shutdown_connections 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fca6806b4d0 0x7fca6806d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca880c8950 0x7fca880c22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fca880c69f0 0x7fca880c1d70 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 --2- 172.21.15.94:0/975707915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca880ad2b0 0x7fca88154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 >> 172.21.15.94:0/975707915 conn(0x7fca880cae70 msgr2=0x7fca881558d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 shutdown_connections 2024-03-06T07:24:38.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.722+0000 7fca8c8c9640 1 -- 172.21.15.94:0/975707915 wait complete. 2024-03-06T07:24:38.726 INFO:tasks.ceph.mds.b.smithi145.stderr: -333> 2024-03-06T07:24:29.153+0000 7ff6fea1b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:38.735 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:38.749 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:24:38.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.836+0000 7fd8c1264640 1 Processor -- start 2024-03-06T07:24:38.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.836+0000 7fd8c1264640 1 -- start start 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.837+0000 7fd8c1264640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd8bc06c1f0 0x7fd8bc0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.837+0000 7fd8c1264640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0c2790 con 0x7fd8bc14ef70 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.837+0000 7fd8c1264640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc06cf20 con 0x7fd8bc06cad0 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.837+0000 7fd8c1264640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8bc0c3090 con 0x7fd8bc06c1f0 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.837+0000 7fd8baffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd8bc14ef70 0x7fd8bc0c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45826/0 (socket says 172.21.15.94:45826) 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.837+0000 7fd8baffd640 1 -- 172.21.15.94:0/2990315081 learned_addr learned my addr 172.21.15.94:0/2990315081 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.837+0000 7fd8ba7fc640 1 --2- 172.21.15.94:0/2990315081 >> v2:172.21.15.145:3300/0 conn(0x7fd8bc06c1f0 0x7fd8bc0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.838+0000 7fd8b9ffb640 1 -- 172.21.15.94:0/2990315081 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467692207 0 0) 0x7fd8bc0c2790 con 0x7fd8bc14ef70 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.838+0000 7fd8b9ffb640 1 -- 172.21.15.94:0/2990315081 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a8003440 con 0x7fd8bc14ef70 2024-03-06T07:24:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.838+0000 7fd8ba7fc640 1 -- 172.21.15.94:0/2990315081 >> v1:172.21.15.145:6789/0 conn(0x7fd8bc06cad0 legacy=0x7fd8bc154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.838+0000 7fd8b9ffb640 1 -- 172.21.15.94:0/2990315081 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725010891 0 0) 0x7fd8bc06cf20 con 0x7fd8bc06cad0 2024-03-06T07:24:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.838+0000 7fd8ba7fc640 1 -- 172.21.15.94:0/2990315081 >> v1:172.21.15.94:6789/0 conn(0x7fd8bc14ef70 legacy=0x7fd8bc0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.838+0000 7fd8ba7fc640 1 -- 172.21.15.94:0/2990315081 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0c1540 con 0x7fd8bc06c1f0 2024-03-06T07:24:38.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8ba7fc640 1 --2- 172.21.15.94:0/2990315081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0c1820 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0097a0 tx=0x7fd8ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e9c5c6c1cb1853 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8b9ffb640 1 -- 172.21.15.94:0/2990315081 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac035020 con 0x7fd8bc06c1f0 2024-03-06T07:24:38.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8b9ffb640 1 -- 172.21.15.94:0/2990315081 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac031830 con 0x7fd8bc06c1f0 2024-03-06T07:24:38.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8b9ffb640 1 -- 172.21.15.94:0/2990315081 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac031c00 con 0x7fd8bc06c1f0 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8c1264640 1 -- 172.21.15.94:0/2990315081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 msgr2=0x7fd8bc0c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2990315081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0c1820 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0097a0 tx=0x7fd8ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8c1264640 1 -- 172.21.15.94:0/2990315081 shutdown_connections 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2990315081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0c1820 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8c1264640 1 -- 172.21.15.94:0/2990315081 >> 172.21.15.94:0/2990315081 conn(0x7fd8bc0c6710 msgr2=0x7fd8bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8c1264640 1 -- 172.21.15.94:0/2990315081 shutdown_connections 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.839+0000 7fd8c1264640 1 -- 172.21.15.94:0/2990315081 wait complete. 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 Processor -- start 2024-03-06T07:24:38.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 -- start start 2024-03-06T07:24:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8bc06cad0 0x7fd8bc0a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8bc14ef70 0x7fd8bc072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc155750 con 0x7fd8bc14ef70 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc14f350 con 0x7fd8bc06c1f0 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8c1264640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc155340 con 0x7fd8bc06cad0 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8bb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8bc14ef70 0x7fd8bc072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8bb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8bc14ef70 0x7fd8bc072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39072/0 (socket says 172.21.15.94:39072) 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8ba7fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8bc06cad0 0x7fd8bc0a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/2049632765 learned_addr learned my addr 172.21.15.94:0/2049632765 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.841+0000 7fd8baffd640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8bc06cad0 msgr2=0x7fd8bc0a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8bb7fe640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8bc06cad0 0x7fd8bc0a9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 msgr2=0x7fd8bc0bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8bb7fe640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8bb7fe640 1 -- 172.21.15.94:0/2049632765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc1ee640 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8baffd640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8bb7fe640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8bc14ef70 0x7fd8bc072730 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4001560 tx=0x7fd8b400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8b37fe640 1 -- 172.21.15.94:0/2049632765 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4017070 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8b37fe640 1 -- 172.21.15.94:0/2049632765 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b4003dc0 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8b37fe640 1 -- 172.21.15.94:0/2049632765 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40052b0 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc1ee920 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.842+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc1eeee0 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.843+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc04f330 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.843+0000 7fd8b37fe640 1 -- 172.21.15.94:0/2049632765 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8b401a070 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.844+0000 7fd8b37fe640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b500 0x7fd89c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.844+0000 7fd8b37fe640 1 -- 172.21.15.94:0/2049632765 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fd8b4089ce0 con 0x7fd8bc14ef70 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.844+0000 7fd8baffd640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b500 0x7fd89c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.844+0000 7fd8baffd640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b500 0x7fd89c06d9c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4005880 tx=0x7fd8a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:38.848+0000 7fd8b37fe640 1 -- 172.21.15.94:0/2049632765 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8b4058010 con 0x7fd8bc14ef70 2024-03-06T07:24:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.104+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd8bc153af0 con 0x7fd8bc14ef70 2024-03-06T07:24:39.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.750+0000 7fd8b37fe640 1 -- 172.21.15.94:0/2049632765 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v148) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd8b405c0b0 con 0x7fd8bc14ef70 2024-03-06T07:24:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b500 msgr2=0x7fd89c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b500 0x7fd89c06d9c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4005880 tx=0x7fd8a4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8bc14ef70 msgr2=0x7fd8bc072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8bc14ef70 0x7fd8bc072730 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4001560 tx=0x7fd8b400cc70 comp rx=0 tx=0).stop 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 shutdown_connections 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b500 0x7fd89c06d9c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8bc14ef70 0x7fd8bc072730 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8bc06cad0 0x7fd8bc0a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 --2- 172.21.15.94:0/2049632765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8bc06c1f0 0x7fd8bc0bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 >> 172.21.15.94:0/2049632765 conn(0x7fd8bc0c6710 msgr2=0x7fd8bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 shutdown_connections 2024-03-06T07:24:39.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.752+0000 7fd8c1264640 1 -- 172.21.15.94:0/2049632765 wait complete. 2024-03-06T07:24:39.762 INFO:tasks.ceph.mds.a.smithi094.stderr: -740> 2024-03-06T07:24:19.151+0000 7f3c0e760640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:39.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:39.790 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:24:39.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.872+0000 7f2b3d33a640 1 Processor -- start 2024-03-06T07:24:39.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.872+0000 7f2b3d33a640 1 -- start start 2024-03-06T07:24:39.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b3d33a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b380c8980 0x7f2b381551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b3d33a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2b381566c0 0x7f2b3806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b3d33a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b380c8f60 con 0x7f2b3814f180 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b3d33a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b380c9240 con 0x7f2b381566c0 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b3d33a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b380c9520 con 0x7f2b380c8980 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b377fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2b381566c0 0x7f2b3806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b2ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b380c8980 0x7f2b381551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b36ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2b3814f180 0x7f2b380ca000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49470/0 (socket says 172.21.15.94:49470) 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b36ffd640 1 -- 172.21.15.94:0/4167755133 learned_addr learned my addr 172.21.15.94:0/4167755133 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b377fe640 1 -- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b380c8980 msgr2=0x7f2b381551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b377fe640 1 --2- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b380c8980 0x7f2b381551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b377fe640 1 -- 172.21.15.94:0/4167755133 >> v1:172.21.15.145:6789/0 conn(0x7f2b3814f180 legacy=0x7f2b380ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b377fe640 1 -- 172.21.15.94:0/4167755133 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380c9800 con 0x7f2b381566c0 2024-03-06T07:24:39.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b367fc640 1 -- 172.21.15.94:0/4167755133 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481517558 0 0) 0x7f2b380c8f60 con 0x7f2b3814f180 2024-03-06T07:24:39.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.873+0000 7f2b2ffff640 1 --2- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b380c8980 0x7f2b381551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:39.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.874+0000 7f2b377fe640 1 --2- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b381566c0 0x7f2b3806c990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009d90 tx=0x7f2b3002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aae4efc48d022a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.874+0000 7f2b367fc640 1 -- 172.21.15.94:0/4167755133 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30039070 con 0x7f2b381566c0 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.874+0000 7f2b367fc640 1 -- 172.21.15.94:0/4167755133 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3002cc50 con 0x7f2b381566c0 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.874+0000 7f2b367fc640 1 -- 172.21.15.94:0/4167755133 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30034440 con 0x7f2b381566c0 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 -- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b381566c0 msgr2=0x7f2b3806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b381566c0 0x7f2b3806c990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009d90 tx=0x7f2b3002bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 -- 172.21.15.94:0/4167755133 shutdown_connections 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b381566c0 0x7f2b3806c990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/4167755133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b380c8980 0x7f2b381551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 -- 172.21.15.94:0/4167755133 >> 172.21.15.94:0/4167755133 conn(0x7f2b38072d20 msgr2=0x7f2b380ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 -- 172.21.15.94:0/4167755133 shutdown_connections 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.875+0000 7f2b3d33a640 1 -- 172.21.15.94:0/4167755133 wait complete. 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.876+0000 7f2b3d33a640 1 Processor -- start 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.876+0000 7f2b3d33a640 1 -- start start 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b3d33a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b380c8980 0x7f2b381523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b3d33a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b3814f180 0x7f2b381547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b3d33a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b381566c0 0x7f2b3814a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b3d33a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3814f560 con 0x7f2b3814f180 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b3d33a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b38156b50 con 0x7f2b380c8980 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b3d33a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3814f6e0 con 0x7f2b381566c0 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b3814f180 0x7f2b381547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b3814f180 0x7f2b381547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39098/0 (socket says 172.21.15.94:39098) 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 -- 172.21.15.94:0/3841440267 learned_addr learned my addr 172.21.15.94:0/3841440267 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b36ffd640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b380c8980 0x7f2b381523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b377fe640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b381566c0 0x7f2b3814a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 -- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b381566c0 msgr2=0x7f2b3814a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b381566c0 0x7f2b3814a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 -- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b380c8980 msgr2=0x7f2b381523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b380c8980 0x7f2b381523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 -- 172.21.15.94:0/3841440267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380c5070 con 0x7f2b3814f180 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2ffff640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b3814f180 0x7f2b381547b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000e990 tx=0x7f2b2000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.877+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/3841440267 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20017070 con 0x7f2b3814f180 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.878+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/3841440267 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2000c9e0 con 0x7f2b3814f180 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.878+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3814b590 con 0x7f2b3814f180 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.878+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/3841440267 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2000cdb0 con 0x7f2b3814f180 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.878+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3814b2b0 con 0x7f2b3814f180 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.878+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3804f330 con 0x7f2b3814f180 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.879+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/3841440267 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b2001a070 con 0x7f2b3814f180 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.879+0000 7f2b2f7fe640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b1806b4d0 0x7f2b1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.879+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/3841440267 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f2b2008a540 con 0x7f2b3814f180 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.879+0000 7f2b36ffd640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b1806b4d0 0x7f2b1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.880+0000 7f2b36ffd640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b1806b4d0 0x7f2b1806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007fd0 tx=0x7f2b28007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:39.884+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/3841440267 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b20058820 con 0x7f2b3814f180 2024-03-06T07:24:40.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.057+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2b38155710 con 0x7f2b3814f180 2024-03-06T07:24:40.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.761+0000 7f2b2f7fe640 1 -- 172.21.15.94:0/3841440267 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v157) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2b2005c8c0 con 0x7f2b3814f180 2024-03-06T07:24:40.763 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:40.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b1806b4d0 msgr2=0x7f2b1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b1806b4d0 0x7f2b1806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007fd0 tx=0x7f2b28007de0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b3814f180 msgr2=0x7f2b381547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b3814f180 0x7f2b381547b0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000e990 tx=0x7f2b2000ee50 comp rx=0 tx=0).stop 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 shutdown_connections 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2b1806b4d0 0x7f2b1806d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b381566c0 0x7f2b3814a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2b3814f180 0x7f2b381547b0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 --2- 172.21.15.94:0/3841440267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b380c8980 0x7f2b381523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 >> 172.21.15.94:0/3841440267 conn(0x7f2b38072d20 msgr2=0x7f2b380cb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.765+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 shutdown_connections 2024-03-06T07:24:40.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.766+0000 7f2b3d33a640 1 -- 172.21.15.94:0/3841440267 wait complete. 2024-03-06T07:24:40.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:40.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.931+0000 7fc497a88640 1 Processor -- start 2024-03-06T07:24:40.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.931+0000 7fc497a88640 1 -- start start 2024-03-06T07:24:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.932+0000 7fc497a88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc490155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.932+0000 7fc497a88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc4900c14e0 0x7fc4900c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.932+0000 7fc497a88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4900c1970 con 0x7fc4900a6040 2024-03-06T07:24:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.932+0000 7fc497a88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4900c1ef0 con 0x7fc4900c14e0 2024-03-06T07:24:40.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.932+0000 7fc497a88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc490158f00 con 0x7fc490159390 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc496025640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc4900c14e0 0x7fc4900c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc495824640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc4900a6040 0x7fc4900c5620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49478/0 (socket says 172.21.15.94:49478) 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc495824640 1 -- 172.21.15.94:0/777046411 learned_addr learned my addr 172.21.15.94:0/777046411 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc495023640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc490155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc496025640 1 -- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 msgr2=0x7fc490155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc496025640 1 --2- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc490155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc496025640 1 -- 172.21.15.94:0/777046411 >> v1:172.21.15.145:6789/0 conn(0x7fc4900a6040 legacy=0x7fc4900c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc496025640 1 -- 172.21.15.94:0/777046411 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4900c1c10 con 0x7fc4900c14e0 2024-03-06T07:24:40.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.933+0000 7fc495023640 1 --2- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc490155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.934+0000 7fc496025640 1 --2- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 0x7fc4900c34e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc4900c2070 tx=0x7fc48802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc674e0ad2d00538 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.934+0000 7fc494822640 1 -- 172.21.15.94:0/777046411 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488039070 con 0x7fc4900c14e0 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.934+0000 7fc494822640 1 -- 172.21.15.94:0/777046411 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48802cb20 con 0x7fc4900c14e0 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.934+0000 7fc494822640 1 -- 172.21.15.94:0/777046411 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488034440 con 0x7fc4900c14e0 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 -- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 msgr2=0x7fc4900c34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 --2- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 0x7fc4900c34e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc4900c2070 tx=0x7fc48802bd70 comp rx=0 tx=0).stop 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 -- 172.21.15.94:0/777046411 shutdown_connections 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 --2- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 0x7fc4900c34e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 --2- 172.21.15.94:0/777046411 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc490155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 -- 172.21.15.94:0/777046411 >> 172.21.15.94:0/777046411 conn(0x7fc4900a8f70 msgr2=0x7fc49006afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 -- 172.21.15.94:0/777046411 shutdown_connections 2024-03-06T07:24:40.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.935+0000 7fc497a88640 1 -- 172.21.15.94:0/777046411 wait complete. 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 Processor -- start 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 -- start start 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4900a6040 0x7fc490153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 0x7fc490151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc49009b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4900c5d30 con 0x7fc4900a6040 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc490158b90 con 0x7fc4900c14e0 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.936+0000 7fc497a88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4900a6420 con 0x7fc490159390 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4900a6040 0x7fc490153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4900a6040 0x7fc490153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39110/0 (socket says 172.21.15.94:39110) 2024-03-06T07:24:40.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 -- 172.21.15.94:0/403933074 learned_addr learned my addr 172.21.15.94:0/403933074 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495023640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 0x7fc490151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc496025640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc49009b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 -- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 msgr2=0x7fc49009b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc49009b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 -- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 msgr2=0x7fc490151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 0x7fc490151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 -- 172.21.15.94:0/403933074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4901570d0 con 0x7fc4900a6040 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc495824640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4900a6040 0x7fc490153a90 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00c3f0 tx=0x7fc48c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc4867fc640 1 -- 172.21.15.94:0/403933074 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c017070 con 0x7fc4900a6040 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc490151f40 con 0x7fc4900a6040 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc4867fc640 1 -- 172.21.15.94:0/403933074 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c005ce0 con 0x7fc4900a6040 2024-03-06T07:24:40.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc4867fc640 1 -- 172.21.15.94:0/403933074 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c0072f0 con 0x7fc4900a6040 2024-03-06T07:24:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.937+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc490155070 con 0x7fc4900a6040 2024-03-06T07:24:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.938+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4900c3d80 con 0x7fc4900a6040 2024-03-06T07:24:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.939+0000 7fc4867fc640 1 -- 172.21.15.94:0/403933074 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc48c01a070 con 0x7fc4900a6040 2024-03-06T07:24:40.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.939+0000 7fc4867fc640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47406b4d0 0x7fc47406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.939+0000 7fc4867fc640 1 -- 172.21.15.94:0/403933074 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 6777+0+0 (secure 0 0 0) 0x7fc48c0898f0 con 0x7fc4900a6040 2024-03-06T07:24:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.939+0000 7fc495023640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47406b4d0 0x7fc47406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.940+0000 7fc495023640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47406b4d0 0x7fc47406d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc480009560 tx=0x7fc480009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:40.943+0000 7fc4867fc640 1 -- 172.21.15.94:0/403933074 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4900c3d80 con 0x7fc4900a6040 2024-03-06T07:24:41.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.113+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc49004f330 con 0x7fc4900a6040 2024-03-06T07:24:41.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.765+0000 7fc4867fc640 1 -- 172.21.15.94:0/403933074 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v158) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc48c057d00 con 0x7fc4900a6040 2024-03-06T07:24:41.768 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47406b4d0 msgr2=0x7fc47406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47406b4d0 0x7fc47406d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc480009560 tx=0x7fc480009190 comp rx=0 tx=0).stop 2024-03-06T07:24:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4900a6040 msgr2=0x7fc490153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4900a6040 0x7fc490153a90 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00c3f0 tx=0x7fc48c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 shutdown_connections 2024-03-06T07:24:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc47406b4d0 0x7fc47406d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc490159390 0x7fc49009b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4900c14e0 0x7fc490151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 --2- 172.21.15.94:0/403933074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4900a6040 0x7fc490153a90 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.768+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 >> 172.21.15.94:0/403933074 conn(0x7fc4900a8f70 msgr2=0x7fc4900c3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.769+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 shutdown_connections 2024-03-06T07:24:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.769+0000 7fc497a88640 1 -- 172.21.15.94:0/403933074 wait complete. 2024-03-06T07:24:41.784 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.883+0000 7fcbd49b5640 1 Processor -- start 2024-03-06T07:24:41.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.883+0000 7fcbd49b5640 1 -- start start 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbd49b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcbd014f790 0x7fcbd01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbd49b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c1aa0 0x7fcbd00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbd49b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd00c28d0 con 0x7fcbd00c5c60 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbd49b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd00c2680 con 0x7fcbd014f790 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbd49b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd00c2bb0 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbce59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcbd014f790 0x7fcbd01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbce59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcbd014f790 0x7fcbd01529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39118/0 (socket says 172.21.15.94:39118) 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbce59c640 1 -- 172.21.15.94:0/852313594 learned_addr learned my addr 172.21.15.94:0/852313594 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbce59c640 1 -- 172.21.15.94:0/852313594 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c1aa0 msgr2=0x7fcbd00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbce59c640 1 --2- 172.21.15.94:0/852313594 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c1aa0 0x7fcbd00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbce59c640 1 -- 172.21.15.94:0/852313594 >> v1:172.21.15.145:6789/0 conn(0x7fcbd00c5c60 legacy=0x7fcbd0155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.884+0000 7fcbce59c640 1 -- 172.21.15.94:0/852313594 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd00c2e90 con 0x7fcbd014f790 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbce59c640 1 --2- 172.21.15.94:0/852313594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd014f790 0x7fcbd01529d0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000bb10 tx=0x7fcbc002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee18fdf31d30879b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbcdd9b640 1 -- 172.21.15.94:0/852313594 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc003b070 con 0x7fcbd014f790 2024-03-06T07:24:41.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbcdd9b640 1 -- 172.21.15.94:0/852313594 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc0036440 con 0x7fcbd014f790 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbcdd9b640 1 -- 172.21.15.94:0/852313594 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0036810 con 0x7fcbd014f790 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbd49b5640 1 -- 172.21.15.94:0/852313594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd014f790 msgr2=0x7fcbd01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/852313594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd014f790 0x7fcbd01529d0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000bb10 tx=0x7fcbc002e0c0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbd49b5640 1 -- 172.21.15.94:0/852313594 shutdown_connections 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/852313594 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c1aa0 0x7fcbd00c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.885+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/852313594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd014f790 0x7fcbd01529d0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.886+0000 7fcbd49b5640 1 -- 172.21.15.94:0/852313594 >> 172.21.15.94:0/852313594 conn(0x7fcbd00c6710 msgr2=0x7fcbd00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.886+0000 7fcbd49b5640 1 -- 172.21.15.94:0/852313594 shutdown_connections 2024-03-06T07:24:41.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.886+0000 7fcbd49b5640 1 -- 172.21.15.94:0/852313594 wait complete. 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.887+0000 7fcbd49b5640 1 Processor -- start 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.887+0000 7fcbd49b5640 1 -- start start 2024-03-06T07:24:41.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd00c1aa0 0x7fcbd01ddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c5c60 0x7fcbd01e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbd014f790 0x7fcbd01e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd006cc00 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd00c0d60 con 0x7fcbd014f790 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd006cd80 con 0x7fcbd00c5c60 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd00c1aa0 0x7fcbd01ddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd00c1aa0 0x7fcbd01ddaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39128/0 (socket says 172.21.15.94:39128) 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 -- 172.21.15.94:0/535847682 learned_addr learned my addr 172.21.15.94:0/535847682 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 -- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c5c60 msgr2=0x7fcbd01e7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbced9d640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbd014f790 0x7fcbd01e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbc5d9b640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c5c60 0x7fcbd01e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c5c60 0x7fcbd01e7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 -- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbd014f790 msgr2=0x7fcbd01e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbd014f790 0x7fcbd01e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 -- 172.21.15.94:0/535847682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd01dc040 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbce59c640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd00c1aa0 0x7fcbd01ddaf0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000b2c0 tx=0x7fcbc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbc77fe640 1 -- 172.21.15.94:0/535847682 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0045070 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbc77fe640 1 -- 172.21.15.94:0/535847682 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc0040890 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd01dc2e0 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbc77fe640 1 -- 172.21.15.94:0/535847682 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc003b040 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.888+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd01dc8a0 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.889+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd004f330 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.890+0000 7fcbc77fe640 1 -- 172.21.15.94:0/535847682 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbc0048070 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.890+0000 7fcbc77fe640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbb006b500 0x7fcbb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.890+0000 7fcbc77fe640 1 -- 172.21.15.94:0/535847682 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7fcbc00af180 con 0x7fcbd00c1aa0 2024-03-06T07:24:41.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.890+0000 7fcbc5d9b640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbb006b500 0x7fcbb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.891+0000 7fcbc5d9b640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbb006b500 0x7fcbb006d9c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fcbd01dd5e0 tx=0x7fcbb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:41.895+0000 7fcbc77fe640 1 -- 172.21.15.94:0/535847682 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbc007df20 con 0x7fcbd00c1aa0 2024-03-06T07:24:42.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.075+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbd00c1f40 con 0x7fcbd00c1aa0 2024-03-06T07:24:42.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.076+0000 7fcbc77fe640 1 -- 172.21.15.94:0/535847682 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+17281 (secure 0 0 0) 0x7fcbc007d820 con 0x7fcbd00c1aa0 2024-03-06T07:24:42.079 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:42.079 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":158,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:41.762609+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.94:6832/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.145:6835/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.145:6837/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.94:6835/1613596341":"2024-03-07T07:24:23.693503+0000","172.21.15.94:6833/1613596341":"2024-03-07T07:24:23.693503+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbb006b500 msgr2=0x7fcbb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbb006b500 0x7fcbb006d9c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fcbd01dd5e0 tx=0x7fcbb8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd00c1aa0 msgr2=0x7fcbd01ddaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd00c1aa0 0x7fcbd01ddaf0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000b2c0 tx=0x7fcbc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 shutdown_connections 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbb006b500 0x7fcbb006d9c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbd014f790 0x7fcbd01e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbd00c5c60 0x7fcbd01e7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.078+0000 7fcbd49b5640 1 --2- 172.21.15.94:0/535847682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcbd00c1aa0 0x7fcbd01ddaf0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.079+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 >> 172.21.15.94:0/535847682 conn(0x7fcbd00c6710 msgr2=0x7fcbd00c4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.079+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 shutdown_connections 2024-03-06T07:24:42.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.079+0000 7fcbd49b5640 1 -- 172.21.15.94:0/535847682 wait complete. 2024-03-06T07:24:42.092 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-03-06T07:24:15.094089+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-03-06T07:24:15.572143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:42.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.193+0000 7f89fc59a640 1 Processor -- start 2024-03-06T07:24:42.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.193+0000 7f89fc59a640 1 -- start start 2024-03-06T07:24:42.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fc59a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f89f40c5990 0x7f89f40c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fc59a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40ab170 0x7f89f406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fc59a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f40c3ef0 con 0x7f89f406c730 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fc59a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f406cbe0 con 0x7f89f40c5990 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fc59a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f41556d0 con 0x7f89f40ab170 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fa336640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f89f40c5990 0x7f89f40c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fa336640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f89f40c5990 0x7f89f40c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39136/0 (socket says 172.21.15.94:39136) 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fa336640 1 -- 172.21.15.94:0/2887479804 learned_addr learned my addr 172.21.15.94:0/2887479804 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89f9b35640 1 --2- 172.21.15.94:0/2887479804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40ab170 0x7f89f406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fa336640 1 -- 172.21.15.94:0/2887479804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40ab170 msgr2=0x7f89f406c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fa336640 1 --2- 172.21.15.94:0/2887479804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40ab170 0x7f89f406c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fa336640 1 -- 172.21.15.94:0/2887479804 >> v1:172.21.15.145:6789/0 conn(0x7f89f406c730 legacy=0x7f89f41523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.194+0000 7f89fa336640 1 -- 172.21.15.94:0/2887479804 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f41559b0 con 0x7f89f40c5990 2024-03-06T07:24:42.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.195+0000 7f89fa336640 1 --2- 172.21.15.94:0/2887479804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f40c0ab0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f89f000bb10 tx=0x7f89f002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17eee72cab5b3436 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.195+0000 7f89f9334640 1 -- 172.21.15.94:0/2887479804 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f003b070 con 0x7f89f40c5990 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.195+0000 7f89f9334640 1 -- 172.21.15.94:0/2887479804 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f002ede0 con 0x7f89f40c5990 2024-03-06T07:24:42.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.195+0000 7f89f9334640 1 -- 172.21.15.94:0/2887479804 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0036660 con 0x7f89f40c5990 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.195+0000 7f89fc59a640 1 -- 172.21.15.94:0/2887479804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 msgr2=0x7f89f40c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.195+0000 7f89fc59a640 1 --2- 172.21.15.94:0/2887479804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f40c0ab0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f89f000bb10 tx=0x7f89f002e030 comp rx=0 tx=0).stop 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.196+0000 7f89fc59a640 1 -- 172.21.15.94:0/2887479804 shutdown_connections 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.196+0000 7f89fc59a640 1 --2- 172.21.15.94:0/2887479804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f40ab170 0x7f89f406c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.196+0000 7f89fc59a640 1 --2- 172.21.15.94:0/2887479804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f40c0ab0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.196+0000 7f89fc59a640 1 -- 172.21.15.94:0/2887479804 >> 172.21.15.94:0/2887479804 conn(0x7f89f40c6710 msgr2=0x7f89f40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.196+0000 7f89fc59a640 1 -- 172.21.15.94:0/2887479804 shutdown_connections 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.196+0000 7f89fc59a640 1 -- 172.21.15.94:0/2887479804 wait complete. 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 Processor -- start 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 -- start start 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f406c730 0x7f89f4146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89f40ab170 0x7f89f4153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f41ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4156490 con 0x7f89f40c5990 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4155f70 con 0x7f89f40ab170 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fc59a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f4156610 con 0x7f89f406c730 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fab37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f41ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fab37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f41ee0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39142/0 (socket says 172.21.15.94:39142) 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.197+0000 7f89fab37640 1 -- 172.21.15.94:0/3288100661 learned_addr learned my addr 172.21.15.94:0/3288100661 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:42.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fa336640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f406c730 0x7f89f4146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fab37640 1 -- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f406c730 msgr2=0x7f89f4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89f9b35640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89f40ab170 0x7f89f4153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fab37640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f406c730 0x7f89f4146470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fab37640 1 -- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89f40ab170 msgr2=0x7f89f4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fab37640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89f40ab170 0x7f89f4153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fab37640 1 -- 172.21.15.94:0/3288100661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40a7ca0 con 0x7f89f40c5990 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fab37640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f41ee0c0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c7b0 tx=0x7f89ec00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89eaffd640 1 -- 172.21.15.94:0/3288100661 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec004b60 con 0x7f89f40c5990 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89eaffd640 1 -- 172.21.15.94:0/3288100661 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec004d00 con 0x7f89f40c5990 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89eaffd640 1 -- 172.21.15.94:0/3288100661 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec00aa90 con 0x7f89f40c5990 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f4154f70 con 0x7f89f40c5990 2024-03-06T07:24:42.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.198+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f414e3c0 con 0x7f89f40c5990 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.199+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f414dcd0 con 0x7f89f40c5990 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.199+0000 7f89eaffd640 1 -- 172.21.15.94:0/3288100661 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89ec00ac30 con 0x7f89f40c5990 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.200+0000 7f89eaffd640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.200+0000 7f89eaffd640 1 -- 172.21.15.94:0/3288100661 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f89ec08a670 con 0x7f89f40c5990 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.200+0000 7f89fa336640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.200+0000 7f89fa336640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f89f0038000 tx=0x7f89f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.204+0000 7f89eaffd640 1 -- 172.21.15.94:0/3288100661 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89ec020070 con 0x7f89f40c5990 2024-03-06T07:24:42.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.377+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89f404f330 con 0x7f89f40c5990 2024-03-06T07:24:42.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.378+0000 7f89eaffd640 1 -- 172.21.15.94:0/3288100661 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7f89ec0592f0 con 0x7f89f40c5990 2024-03-06T07:24:42.381 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 msgr2=0x7f89d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f89f0038000 tx=0x7f89f0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 msgr2=0x7f89f41ee0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f41ee0c0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c7b0 tx=0x7f89ec00cc70 comp rx=0 tx=0).stop 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 shutdown_connections 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f89d806b4d0 0x7f89d806d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f89f40c5990 0x7f89f41ee0c0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89f40ab170 0x7f89f4153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 --2- 172.21.15.94:0/3288100661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89f406c730 0x7f89f4146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 >> 172.21.15.94:0/3288100661 conn(0x7f89f40c6710 msgr2=0x7f89f406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 shutdown_connections 2024-03-06T07:24:42.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.380+0000 7f89fc59a640 1 -- 172.21.15.94:0/3288100661 wait complete. 2024-03-06T07:24:42.393 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-03-06T07:24:42.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.493+0000 7f1137fc9640 1 Processor -- start 2024-03-06T07:24:42.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.493+0000 7f1137fc9640 1 -- start start 2024-03-06T07:24:42.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1137fc9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f11300b00d0 0x7f11300c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1137fc9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f113006b8e0 0x7f113006bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1137fc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f11300c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1137fc9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1130156160 con 0x7f113006b8e0 2024-03-06T07:24:42.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1137fc9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11300c5710 con 0x7f11300b00d0 2024-03-06T07:24:42.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1137fc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1130152c50 con 0x7f11300c7210 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1135564640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f113006b8e0 0x7f113006bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1135564640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f113006b8e0 0x7f113006bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39154/0 (socket says 172.21.15.94:39154) 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.494+0000 7f1135564640 1 -- 172.21.15.94:0/2651063835 learned_addr learned my addr 172.21.15.94:0/2651063835 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1136566640 1 --2- 172.21.15.94:0/2651063835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f11300c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1135564640 1 -- 172.21.15.94:0/2651063835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 msgr2=0x7f11300c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1135564640 1 --2- 172.21.15.94:0/2651063835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f11300c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1135564640 1 -- 172.21.15.94:0/2651063835 >> v2:172.21.15.145:3300/0 conn(0x7f11300b00d0 msgr2=0x7f11300c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1135d65640 1 --2- 172.21.15.94:0/2651063835 >> v2:172.21.15.145:3300/0 conn(0x7f11300b00d0 0x7f11300c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1135564640 1 --2- 172.21.15.94:0/2651063835 >> v2:172.21.15.145:3300/0 conn(0x7f11300b00d0 0x7f11300c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1135564640 1 -- 172.21.15.94:0/2651063835 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11300b04b0 con 0x7f113006b8e0 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1135564640 1 --2- 172.21.15.94:0/2651063835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113006b8e0 0x7f113006bd20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f11300c5890 tx=0x7f112002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c409879b8c763d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.495+0000 7f1134d63640 1 -- 172.21.15.94:0/2651063835 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120035020 con 0x7f113006b8e0 2024-03-06T07:24:42.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1134d63640 1 -- 172.21.15.94:0/2651063835 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120031640 con 0x7f113006b8e0 2024-03-06T07:24:42.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1134d63640 1 -- 172.21.15.94:0/2651063835 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112002d040 con 0x7f113006b8e0 2024-03-06T07:24:42.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 -- 172.21.15.94:0/2651063835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113006b8e0 msgr2=0x7f113006bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2651063835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113006b8e0 0x7f113006bd20 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f11300c5890 tx=0x7f112002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 -- 172.21.15.94:0/2651063835 shutdown_connections 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2651063835 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f11300c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2651063835 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f113006b8e0 0x7f113006bd20 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2651063835 >> v2:172.21.15.145:3300/0 conn(0x7f11300b00d0 0x7f11300c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 -- 172.21.15.94:0/2651063835 >> 172.21.15.94:0/2651063835 conn(0x7f11300ac810 msgr2=0x7f11300acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 -- 172.21.15.94:0/2651063835 shutdown_connections 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.496+0000 7f1137fc9640 1 -- 172.21.15.94:0/2651063835 wait complete. 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.497+0000 7f1137fc9640 1 Processor -- start 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.497+0000 7f1137fc9640 1 -- start start 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1137fc9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f113006b8e0 0x7f11300c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1137fc9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11300b00d0 0x7f113014d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1137fc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f113014ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1137fc9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113006c260 con 0x7f11300b00d0 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1137fc9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1130155e80 con 0x7f113006b8e0 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1137fc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11300c5ba0 con 0x7f11300c7210 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11300b00d0 0x7f113014d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11300b00d0 0x7f113014d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39162/0 (socket says 172.21.15.94:39162) 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 -- 172.21.15.94:0/2087374577 learned_addr learned my addr 172.21.15.94:0/2087374577 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135d65640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f113006b8e0 0x7f11300c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1136566640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f113014ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 -- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 msgr2=0x7f113014ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f113014ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 -- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f113006b8e0 msgr2=0x7f11300c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f113006b8e0 0x7f11300c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 -- 172.21.15.94:0/2087374577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11300c45a0 con 0x7f11300b00d0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.498+0000 7f1135564640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11300b00d0 0x7f113014d470 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1120000c00 tx=0x7f11200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.499+0000 7f11267fc640 1 -- 172.21.15.94:0/2087374577 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120035040 con 0x7f11300b00d0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.499+0000 7f11267fc640 1 -- 172.21.15.94:0/2087374577 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120039c10 con 0x7f11300b00d0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.499+0000 7f11267fc640 1 -- 172.21.15.94:0/2087374577 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120045480 con 0x7f11300b00d0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.499+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11300cb740 con 0x7f11300b00d0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.499+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1130159050 con 0x7f11300b00d0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.500+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113004f330 con 0x7f11300b00d0 2024-03-06T07:24:42.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.500+0000 7f11267fc640 1 -- 172.21.15.94:0/2087374577 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1120038020 con 0x7f11300b00d0 2024-03-06T07:24:42.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.500+0000 7f11267fc640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f111406b4d0 0x7f111406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.500+0000 7f11267fc640 1 -- 172.21.15.94:0/2087374577 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f11200ac500 con 0x7f11300b00d0 2024-03-06T07:24:42.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.501+0000 7f1135d65640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f111406b4d0 0x7f111406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.501+0000 7f1135d65640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f111406b4d0 0x7f111406d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f11300c2f80 tx=0x7f112c007580 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.506+0000 7f11267fc640 1 -- 172.21.15.94:0/2087374577 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f112007b180 con 0x7f11300b00d0 2024-03-06T07:24:42.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.674+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11301596f0 con 0x7f11300b00d0 2024-03-06T07:24:42.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.674+0000 7f11267fc640 1 -- 172.21.15.94:0/2087374577 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v158) v1 ==== 105+0+11 (secure 0 0 0) 0x7f112007aa80 con 0x7f11300b00d0 2024-03-06T07:24:42.677 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:24:42.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f111406b4d0 msgr2=0x7f111406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f111406b4d0 0x7f111406d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f11300c2f80 tx=0x7f112c007580 comp rx=0 tx=0).stop 2024-03-06T07:24:42.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11300b00d0 msgr2=0x7f113014d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11300b00d0 0x7f113014d470 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1120000c00 tx=0x7f11200046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 shutdown_connections 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f111406b4d0 0x7f111406d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11300c7210 0x7f113014ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11300b00d0 0x7f113014d470 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 --2- 172.21.15.94:0/2087374577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f113006b8e0 0x7f11300c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 >> 172.21.15.94:0/2087374577 conn(0x7f11300ac810 msgr2=0x7f1130150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 shutdown_connections 2024-03-06T07:24:42.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.676+0000 7f1137fc9640 1 -- 172.21.15.94:0/2087374577 wait complete. 2024-03-06T07:24:42.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-03-06T07:24:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.793+0000 7f263a35c640 1 Processor -- start 2024-03-06T07:24:42.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.793+0000 7f263a35c640 1 -- start start 2024-03-06T07:24:42.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f263a35c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f26340c5990 0x7f26340c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f263a35c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f263406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f263a35c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26340c3ef0 con 0x7f263406c730 2024-03-06T07:24:42.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f263a35c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263406cbe0 con 0x7f26340c5990 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f263a35c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26341556d0 con 0x7f26340ab170 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f26388f9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f263406c730 0x7f26341523f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45828/0 (socket says 172.21.15.94:45828) 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f26388f9640 1 -- 172.21.15.94:0/331541662 learned_addr learned my addr 172.21.15.94:0/331541662 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f26337fe640 1 --2- 172.21.15.94:0/331541662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f263406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f2633fff640 1 --2- 172.21.15.94:0/331541662 >> v2:172.21.15.145:3300/0 conn(0x7f26340c5990 0x7f26340c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f2632ffd640 1 -- 172.21.15.94:0/331541662 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691169134 0 0) 0x7f26340c3ef0 con 0x7f263406c730 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.794+0000 7f2632ffd640 1 -- 172.21.15.94:0/331541662 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c003520 con 0x7f263406c730 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.795+0000 7f2632ffd640 1 -- 172.21.15.94:0/331541662 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 755868496 0 0) 0x7f261c003520 con 0x7f263406c730 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.795+0000 7f26337fe640 1 -- 172.21.15.94:0/331541662 >> v2:172.21.15.145:3300/0 conn(0x7f26340c5990 msgr2=0x7f26340c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.795+0000 7f26337fe640 1 --2- 172.21.15.94:0/331541662 >> v2:172.21.15.145:3300/0 conn(0x7f26340c5990 0x7f26340c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.795+0000 7f26337fe640 1 -- 172.21.15.94:0/331541662 >> v1:172.21.15.94:6789/0 conn(0x7f263406c730 legacy=0x7f26341523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.795+0000 7f26337fe640 1 -- 172.21.15.94:0/331541662 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26341559b0 con 0x7f26340ab170 2024-03-06T07:24:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.795+0000 7f2633fff640 1 --2- 172.21.15.94:0/331541662 >> v2:172.21.15.145:3300/0 conn(0x7f26340c5990 0x7f26340c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.795+0000 7f26337fe640 1 --2- 172.21.15.94:0/331541662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f263406c1f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f26280097a0 tx=0x7f262802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a332d2d92e98b251 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f2632ffd640 1 -- 172.21.15.94:0/331541662 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628037070 con 0x7f26340ab170 2024-03-06T07:24:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f2632ffd640 1 -- 172.21.15.94:0/331541662 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2628030510 con 0x7f26340ab170 2024-03-06T07:24:42.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f2632ffd640 1 -- 172.21.15.94:0/331541662 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c0019f0 con 0x7f26340ab170 2024-03-06T07:24:42.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 -- 172.21.15.94:0/331541662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 msgr2=0x7f263406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 --2- 172.21.15.94:0/331541662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f263406c1f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f26280097a0 tx=0x7f262802b710 comp rx=0 tx=0).stop 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 -- 172.21.15.94:0/331541662 shutdown_connections 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 --2- 172.21.15.94:0/331541662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f263406c1f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 --2- 172.21.15.94:0/331541662 >> v2:172.21.15.145:3300/0 conn(0x7f26340c5990 0x7f26340c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 -- 172.21.15.94:0/331541662 >> 172.21.15.94:0/331541662 conn(0x7f26340c6710 msgr2=0x7f26340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 -- 172.21.15.94:0/331541662 shutdown_connections 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.796+0000 7f263a35c640 1 -- 172.21.15.94:0/331541662 wait complete. 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.797+0000 7f263a35c640 1 Processor -- start 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.797+0000 7f263a35c640 1 -- start start 2024-03-06T07:24:42.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f263a35c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f263406c730 0x7f2634147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f263a35c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f26340a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f263a35c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26340c5990 0x7f26340a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f263a35c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2634156490 con 0x7f26340c5990 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f263a35c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2634155f70 con 0x7f263406c730 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f263a35c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2634156610 con 0x7f26340ab170 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26340c5990 0x7f26340a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26340c5990 0x7f26340a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39170/0 (socket says 172.21.15.94:39170) 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 -- 172.21.15.94:0/3823666763 learned_addr learned my addr 172.21.15.94:0/3823666763 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f2633fff640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f263406c730 0x7f2634147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26337fe640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f26340a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 -- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 msgr2=0x7f26340a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f26340a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 -- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f263406c730 msgr2=0x7f2634147010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f263406c730 0x7f2634147010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.798+0000 7f26388f9640 1 -- 172.21.15.94:0/3823666763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634155050 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.799+0000 7f26388f9640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26340c5990 0x7f26340a6e30 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f262c001890 tx=0x7f262c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.799+0000 7f2630ff9640 1 -- 172.21.15.94:0/3823666763 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c012400 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.799+0000 7f2630ff9640 1 -- 172.21.15.94:0/3823666763 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c0125c0 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.799+0000 7f2630ff9640 1 -- 172.21.15.94:0/3823666763 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c0129b0 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.799+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26341552f0 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.799+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26340c26f0 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.800+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263404f330 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.800+0000 7f2630ff9640 1 -- 172.21.15.94:0/3823666763 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f262c012b50 con 0x7f26340c5990 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.801+0000 7f2630ff9640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f262006b500 0x7f262006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.801+0000 7f2630ff9640 1 -- 172.21.15.94:0/3823666763 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f262c08acb0 con 0x7f26340c5990 2024-03-06T07:24:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.801+0000 7f2633fff640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f262006b500 0x7f262006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.801+0000 7f2633fff640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f262006b500 0x7f262006d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2624009560 tx=0x7f2624009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.806+0000 7f2630ff9640 1 -- 172.21.15.94:0/3823666763 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f262c0591c0 con 0x7f26340c5990 2024-03-06T07:24:42.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.977+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f26340c1740 con 0x7f26340c5990 2024-03-06T07:24:42.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.977+0000 7f2630ff9640 1 -- 172.21.15.94:0/3823666763 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v158) v1 ==== 101+0+11 (secure 0 0 0) 0x7f262c05d260 con 0x7f26340c5990 2024-03-06T07:24:42.979 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:24:42.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f262006b500 msgr2=0x7f262006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f262006b500 0x7f262006d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2624009560 tx=0x7f2624009190 comp rx=0 tx=0).stop 2024-03-06T07:24:42.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26340c5990 msgr2=0x7f26340a6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26340c5990 0x7f26340a6e30 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f262c001890 tx=0x7f262c00a430 comp rx=0 tx=0).stop 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 shutdown_connections 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f262006b500 0x7f262006d9c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f26340c5990 0x7f26340a6e30 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26340ab170 0x7f26340a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 --2- 172.21.15.94:0/3823666763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f263406c730 0x7f2634147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 >> 172.21.15.94:0/3823666763 conn(0x7f26340c6710 msgr2=0x7f26340c1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 shutdown_connections 2024-03-06T07:24:42.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:42.978+0000 7f263a35c640 1 -- 172.21.15.94:0/3823666763 wait complete. 2024-03-06T07:24:42.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:43.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f074e6f4640 1 Processor -- start 2024-03-06T07:24:43.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f074e6f4640 1 -- start start 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f074e6f4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f074814f360 0x7f0748155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f074e6f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07480c88c0 0x7f07480c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f074e6f4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07480c73d0 con 0x7f07480c5880 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f074e6f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07480c8dd0 con 0x7f07480c88c0 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f074e6f4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07480be2e0 con 0x7f074814f360 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f07477fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07480c88c0 0x7f07480c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f07477fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07480c88c0 0x7f07480c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39176/0 (socket says 172.21.15.94:39176) 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.090+0000 7f07477fe640 1 -- 172.21.15.94:0/2452887477 learned_addr learned my addr 172.21.15.94:0/2452887477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.091+0000 7f0747fff640 1 --2- 172.21.15.94:0/2452887477 >> v2:172.21.15.145:3300/0 conn(0x7f074814f360 0x7f0748155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.091+0000 7f07477fe640 1 -- 172.21.15.94:0/2452887477 >> v2:172.21.15.145:3300/0 conn(0x7f074814f360 msgr2=0x7f0748155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.091+0000 7f07477fe640 1 --2- 172.21.15.94:0/2452887477 >> v2:172.21.15.145:3300/0 conn(0x7f074814f360 0x7f0748155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.091+0000 7f07477fe640 1 -- 172.21.15.94:0/2452887477 >> v1:172.21.15.145:6789/0 conn(0x7f07480c5880 legacy=0x7f0748158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.091+0000 7f07477fe640 1 -- 172.21.15.94:0/2452887477 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480c7670 con 0x7f07480c88c0 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.091+0000 7f07477fe640 1 --2- 172.21.15.94:0/2452887477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c88c0 0x7f07480c5340 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f074808c840 tx=0x7f073802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe732b6c7c86ea00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f0746ffd640 1 -- 172.21.15.94:0/2452887477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738035020 con 0x7f07480c88c0 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f0746ffd640 1 -- 172.21.15.94:0/2452887477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738031870 con 0x7f07480c88c0 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f0746ffd640 1 -- 172.21.15.94:0/2452887477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738031c40 con 0x7f07480c88c0 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 -- 172.21.15.94:0/2452887477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c88c0 msgr2=0x7f07480c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 --2- 172.21.15.94:0/2452887477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c88c0 0x7f07480c5340 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f074808c840 tx=0x7f073802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 -- 172.21.15.94:0/2452887477 shutdown_connections 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 --2- 172.21.15.94:0/2452887477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c88c0 0x7f07480c5340 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 --2- 172.21.15.94:0/2452887477 >> v2:172.21.15.145:3300/0 conn(0x7f074814f360 0x7f0748155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 -- 172.21.15.94:0/2452887477 >> 172.21.15.94:0/2452887477 conn(0x7f07480ac570 msgr2=0x7f07480ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 -- 172.21.15.94:0/2452887477 shutdown_connections 2024-03-06T07:24:43.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.092+0000 7f074e6f4640 1 -- 172.21.15.94:0/2452887477 wait complete. 2024-03-06T07:24:43.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.093+0000 7f074e6f4640 1 Processor -- start 2024-03-06T07:24:43.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.093+0000 7f074e6f4640 1 -- start start 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074e6f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c5880 0x7f0748152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074e6f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07480c88c0 0x7f074806bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074e6f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074814f360 0x7f07481ead50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074e6f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0748159270 con 0x7f07480c5880 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074e6f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07480c7c30 con 0x7f07480c88c0 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074e6f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07481593f0 con 0x7f074814f360 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c5880 0x7f0748152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c5880 0x7f0748152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39190/0 (socket says 172.21.15.94:39190) 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 -- 172.21.15.94:0/3170809859 learned_addr learned my addr 172.21.15.94:0/3170809859 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f07477fe640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07480c88c0 0x7f074806bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074cc91640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074814f360 0x7f07481ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 -- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074814f360 msgr2=0x7f07481ead50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074814f360 0x7f07481ead50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 -- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07480c88c0 msgr2=0x7f074806bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07480c88c0 0x7f074806bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f0747fff640 1 -- 172.21.15.94:0/3170809859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480c1110 con 0x7f07480c5880 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.094+0000 7f074cc91640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074814f360 0x7f07481ead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.095+0000 7f0747fff640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c5880 0x7f0748152090 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f073c00e8a0 tx=0x7f073c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.095+0000 7f0744ff9640 1 -- 172.21.15.94:0/3170809859 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c017070 con 0x7f07480c5880 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.095+0000 7f0744ff9640 1 -- 172.21.15.94:0/3170809859 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c00c940 con 0x7f07480c5880 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.095+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07481514e0 con 0x7f07480c5880 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.095+0000 7f0744ff9640 1 -- 172.21.15.94:0/3170809859 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c00cd30 con 0x7f07480c5880 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.095+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0748151aa0 con 0x7f07480c5880 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.096+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074804f330 con 0x7f07480c5880 2024-03-06T07:24:43.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.096+0000 7f0744ff9640 1 -- 172.21.15.94:0/3170809859 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f073c01a070 con 0x7f07480c5880 2024-03-06T07:24:43.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.097+0000 7f0744ff9640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073406b4d0 0x7f073406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.097+0000 7f0744ff9640 1 -- 172.21.15.94:0/3170809859 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f073c08a1e0 con 0x7f07480c5880 2024-03-06T07:24:43.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.097+0000 7f07477fe640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073406b4d0 0x7f073406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.097+0000 7f07477fe640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073406b4d0 0x7f073406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0738036000 tx=0x7f073803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.102+0000 7f0744ff9640 1 -- 172.21.15.94:0/3170809859 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f073c058810 con 0x7f07480c5880 2024-03-06T07:24:43.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.307+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f074806cda0 con 0x7f07480c5880 2024-03-06T07:24:43.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.307+0000 7f0744ff9640 1 -- 172.21.15.94:0/3170809859 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f073c05c8b0 con 0x7f07480c5880 2024-03-06T07:24:43.310 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:43.310 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":152,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1221074247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1221074247},{"type":"v1","addr":"172.21.15.94:6834","nonce":1221074247}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":37370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1409431517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1409431517},{"type":"v1","addr":"172.21.15.145:6837","nonce":1409431517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:42.767355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8152":{"gid":8152,"name":"d","rank":0,"incarnation":149,"state":"up:active","state_seq":14,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8152,"qdb_cluster":[8152]},"id":12}]} 2024-03-06T07:24:43.311 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 152 2024-03-06T07:24:43.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073406b4d0 msgr2=0x7f073406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073406b4d0 0x7f073406d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0738036000 tx=0x7f073803c040 comp rx=0 tx=0).stop 2024-03-06T07:24:43.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c5880 msgr2=0x7f0748152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c5880 0x7f0748152090 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f073c00e8a0 tx=0x7f073c00ed60 comp rx=0 tx=0).stop 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 shutdown_connections 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f073406b4d0 0x7f073406d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f074814f360 0x7f07481ead50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07480c88c0 0x7f074806bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 --2- 172.21.15.94:0/3170809859 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07480c5880 0x7f0748152090 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 >> 172.21.15.94:0/3170809859 conn(0x7f07480ac570 msgr2=0x7f074814e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 shutdown_connections 2024-03-06T07:24:43.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.309+0000 7f074e6f4640 1 -- 172.21.15.94:0/3170809859 wait complete. 2024-03-06T07:24:43.323 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:43.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f7926b3e640 1 Processor -- start 2024-03-06T07:24:43.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f7926b3e640 1 -- start start 2024-03-06T07:24:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f7926b3e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f79200c88c0 0x7f7920155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f7926b3e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7920158b60 0x7f7920158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f7926b3e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79200c73d0 con 0x7f792014f360 2024-03-06T07:24:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f7926b3e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79200c8dd0 con 0x7f7920158b60 2024-03-06T07:24:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f7926b3e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79200be2e0 con 0x7f79200c88c0 2024-03-06T07:24:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f79248da640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f792014f360 0x7f79200c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45830/0 (socket says 172.21.15.94:45830) 2024-03-06T07:24:43.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f79248da640 1 -- 172.21.15.94:0/1079577344 learned_addr learned my addr 172.21.15.94:0/1079577344 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f79250db640 1 --2- 172.21.15.94:0/1079577344 >> v2:172.21.15.94:3300/0 conn(0x7f7920158b60 0x7f7920158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.421+0000 7f791ffff640 1 --2- 172.21.15.94:0/1079577344 >> v2:172.21.15.145:3300/0 conn(0x7f79200c88c0 0x7f7920155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.422+0000 7f79250db640 1 -- 172.21.15.94:0/1079577344 >> v2:172.21.15.145:3300/0 conn(0x7f79200c88c0 msgr2=0x7f7920155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.422+0000 7f79250db640 1 --2- 172.21.15.94:0/1079577344 >> v2:172.21.15.145:3300/0 conn(0x7f79200c88c0 0x7f7920155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.422+0000 7f79250db640 1 -- 172.21.15.94:0/1079577344 >> v1:172.21.15.94:6789/0 conn(0x7f792014f360 legacy=0x7f79200c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.422+0000 7f79250db640 1 -- 172.21.15.94:0/1079577344 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79200c7670 con 0x7f7920158b60 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.422+0000 7f791f7fe640 1 -- 172.21.15.94:0/1079577344 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992784164 0 0) 0x7f79200c73d0 con 0x7f792014f360 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.422+0000 7f79250db640 1 --2- 172.21.15.94:0/1079577344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7920158b60 0x7f7920158ff0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f79200c84c0 tx=0x7f791002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a98697aee12695 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f791f7fe640 1 -- 172.21.15.94:0/1079577344 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79100315a0 con 0x7f7920158b60 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f791f7fe640 1 -- 172.21.15.94:0/1079577344 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7910037070 con 0x7f7920158b60 2024-03-06T07:24:43.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f791f7fe640 1 -- 172.21.15.94:0/1079577344 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910031960 con 0x7f7920158b60 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 -- 172.21.15.94:0/1079577344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7920158b60 msgr2=0x7f7920158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 --2- 172.21.15.94:0/1079577344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7920158b60 0x7f7920158ff0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f79200c84c0 tx=0x7f791002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 -- 172.21.15.94:0/1079577344 shutdown_connections 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 --2- 172.21.15.94:0/1079577344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7920158b60 0x7f7920158ff0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 --2- 172.21.15.94:0/1079577344 >> v2:172.21.15.145:3300/0 conn(0x7f79200c88c0 0x7f7920155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 -- 172.21.15.94:0/1079577344 >> 172.21.15.94:0/1079577344 conn(0x7f79200ac570 msgr2=0x7f79200ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 -- 172.21.15.94:0/1079577344 shutdown_connections 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.423+0000 7f7926b3e640 1 -- 172.21.15.94:0/1079577344 wait complete. 2024-03-06T07:24:43.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.424+0000 7f7926b3e640 1 Processor -- start 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.424+0000 7f7926b3e640 1 -- start start 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f7926b3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79200c88c0 0x7f79200c0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f7926b3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f792014f360 0x7f79201ebf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f7926b3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7920158b60 0x7f79201ec490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f7926b3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79200c5a50 con 0x7f79200c88c0 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f7926b3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79200c7c30 con 0x7f792014f360 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f7926b3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79200c5bd0 con 0x7f7920158b60 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f79248da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79200c88c0 0x7f79200c0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f79248da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79200c88c0 0x7f79200c0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39198/0 (socket says 172.21.15.94:39198) 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f79248da640 1 -- 172.21.15.94:0/2296844608 learned_addr learned my addr 172.21.15.94:0/2296844608 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f791ffff640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f792014f360 0x7f79201ebf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f79250db640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7920158b60 0x7f79201ec490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f79248da640 1 -- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7920158b60 msgr2=0x7f79201ec490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.425+0000 7f79248da640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7920158b60 0x7f79201ec490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f79248da640 1 -- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f792014f360 msgr2=0x7f79201ebf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f79248da640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f792014f360 0x7f79201ebf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f79248da640 1 -- 172.21.15.94:0/2296844608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7920158710 con 0x7f79200c88c0 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f791ffff640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f792014f360 0x7f79201ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f79248da640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79200c88c0 0x7f79200c0b70 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f791800c890 tx=0x7f791800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f791d7fa640 1 -- 172.21.15.94:0/2296844608 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918018070 con 0x7f79200c88c0 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f791d7fa640 1 -- 172.21.15.94:0/2296844608 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791800a640 con 0x7f79200c88c0 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79200befa0 con 0x7f79200c88c0 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f791d7fa640 1 -- 172.21.15.94:0/2296844608 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791800aa30 con 0x7f79200c88c0 2024-03-06T07:24:43.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.426+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79200bf560 con 0x7f79200c88c0 2024-03-06T07:24:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.427+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792004f330 con 0x7f79200c88c0 2024-03-06T07:24:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.427+0000 7f791d7fa640 1 -- 172.21.15.94:0/2296844608 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f791801b070 con 0x7f79200c88c0 2024-03-06T07:24:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.428+0000 7f791d7fa640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f790c06b4d0 0x7f790c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.428+0000 7f791d7fa640 1 -- 172.21.15.94:0/2296844608 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f7918089990 con 0x7f79200c88c0 2024-03-06T07:24:43.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.428+0000 7f791ffff640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f790c06b4d0 0x7f790c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.428+0000 7f791ffff640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f790c06b4d0 0x7f790c06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f792014c3a0 tx=0x7f7914009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.433+0000 7f791d7fa640 1 -- 172.21.15.94:0/2296844608 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7918057fc0 con 0x7f79200c88c0 2024-03-06T07:24:43.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.605+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f792006cda0 con 0x7f79200c88c0 2024-03-06T07:24:43.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.607+0000 7f791d7fa640 1 -- 172.21.15.94:0/2296844608 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+17281 (secure 0 0 0) 0x7f791805c060 con 0x7f79200c88c0 2024-03-06T07:24:43.610 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:43.610 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":158,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:41.762609+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.94:6832/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.145:6835/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.145:6837/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.94:6835/1613596341":"2024-03-07T07:24:23.693503+0000","172.21.15.94:6833/1613596341":"2024-03-07T07:24:23.693503+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:43.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.608+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f790c06b4d0 msgr2=0x7f790c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.608+0000 7f7926b3e640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f790c06b4d0 0x7f790c06d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f792014c3a0 tx=0x7f7914009190 comp rx=0 tx=0).stop 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.608+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79200c88c0 msgr2=0x7f79200c0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.608+0000 7f7926b3e640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79200c88c0 0x7f79200c0b70 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f791800c890 tx=0x7f791800cd50 comp rx=0 tx=0).stop 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 shutdown_connections 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f790c06b4d0 0x7f790c06d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7920158b60 0x7f79201ec490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f792014f360 0x7f79201ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 --2- 172.21.15.94:0/2296844608 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79200c88c0 0x7f79200c0b70 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 >> 172.21.15.94:0/2296844608 conn(0x7f79200ac570 msgr2=0x7f79200c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 shutdown_connections 2024-03-06T07:24:43.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.609+0000 7f7926b3e640 1 -- 172.21.15.94:0/2296844608 wait complete. 2024-03-06T07:24:43.622 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-03-06T07:24:43.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:43.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.720+0000 7f85508c9640 1 Processor -- start 2024-03-06T07:24:43.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.720+0000 7f85508c9640 1 -- start start 2024-03-06T07:24:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f85508c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f854c0c5480 0x7f854c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f85508c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f854c0c5860 0x7f854c06f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f85508c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c0c5cc0 con 0x7f854c1586f0 2024-03-06T07:24:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f85508c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c06df40 con 0x7f854c0c5480 2024-03-06T07:24:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f85508c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c1591c0 con 0x7f854c0c5860 2024-03-06T07:24:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f854a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f854c0c5480 0x7f854c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f854ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f854c1586f0 0x7f854c0724f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45836/0 (socket says 172.21.15.94:45836) 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f854a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f854c0c5480 0x7f854c155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39214/0 (socket says 172.21.15.94:39214) 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f854ad9d640 1 -- 172.21.15.94:0/2158093051 learned_addr learned my addr 172.21.15.94:0/2158093051 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.721+0000 7f8549d9b640 1 --2- 172.21.15.94:0/2158093051 >> v2:172.21.15.145:3300/0 conn(0x7f854c0c5860 0x7f854c06f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854a59c640 1 -- 172.21.15.94:0/2158093051 >> v2:172.21.15.145:3300/0 conn(0x7f854c0c5860 msgr2=0x7f854c06f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854a59c640 1 --2- 172.21.15.94:0/2158093051 >> v2:172.21.15.145:3300/0 conn(0x7f854c0c5860 0x7f854c06f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854a59c640 1 -- 172.21.15.94:0/2158093051 >> v1:172.21.15.94:6789/0 conn(0x7f854c1586f0 legacy=0x7f854c0724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854a59c640 1 -- 172.21.15.94:0/2158093051 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c1594a0 con 0x7f854c0c5480 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854959a640 1 -- 172.21.15.94:0/2158093051 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093160185 0 0) 0x7f854c0c5cc0 con 0x7f854c1586f0 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854a59c640 1 --2- 172.21.15.94:0/2158093051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c0c5480 0x7f854c155ba0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f854c0c9770 tx=0x7f853c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528b2c1e13cf75f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854959a640 1 -- 172.21.15.94:0/2158093051 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c039070 con 0x7f854c0c5480 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854959a640 1 -- 172.21.15.94:0/2158093051 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c02cbf0 con 0x7f854c0c5480 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.722+0000 7f854959a640 1 -- 172.21.15.94:0/2158093051 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c034440 con 0x7f854c0c5480 2024-03-06T07:24:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 -- 172.21.15.94:0/2158093051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c0c5480 msgr2=0x7f854c155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 --2- 172.21.15.94:0/2158093051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c0c5480 0x7f854c155ba0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f854c0c9770 tx=0x7f853c02be40 comp rx=0 tx=0).stop 2024-03-06T07:24:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 -- 172.21.15.94:0/2158093051 shutdown_connections 2024-03-06T07:24:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 --2- 172.21.15.94:0/2158093051 >> v2:172.21.15.145:3300/0 conn(0x7f854c0c5860 0x7f854c06f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 --2- 172.21.15.94:0/2158093051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c0c5480 0x7f854c155ba0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 -- 172.21.15.94:0/2158093051 >> 172.21.15.94:0/2158093051 conn(0x7f854c06b1a0 msgr2=0x7f854c06b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 -- 172.21.15.94:0/2158093051 shutdown_connections 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.723+0000 7f85508c9640 1 -- 172.21.15.94:0/2158093051 wait complete. 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.724+0000 7f85508c9640 1 Processor -- start 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.724+0000 7f85508c9640 1 -- start start 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f85508c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c0c5480 0x7f854c151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f85508c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f854c0c5860 0x7f854c14e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f85508c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c1586f0 0x7f854c1ee090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f85508c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c152a20 con 0x7f854c1586f0 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f85508c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c155680 con 0x7f854c0c5480 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f85508c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c155050 con 0x7f854c0c5860 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c1586f0 0x7f854c1ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c1586f0 0x7f854c1ee090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39224/0 (socket says 172.21.15.94:39224) 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 -- 172.21.15.94:0/1800622132 learned_addr learned my addr 172.21.15.94:0/1800622132 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:43.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f8549d9b640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f854c0c5860 0x7f854c14e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854a59c640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c0c5480 0x7f854c151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 -- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f854c0c5860 msgr2=0x7f854c14e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f854c0c5860 0x7f854c14e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 -- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c0c5480 msgr2=0x7f854c151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c0c5480 0x7f854c151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.725+0000 7f854ad9d640 1 -- 172.21.15.94:0/1800622132 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c14ed50 con 0x7f854c1586f0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.726+0000 7f854ad9d640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c1586f0 0x7f854c1ee090 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f854400c3f0 tx=0x7f854400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.726+0000 7f853affd640 1 -- 172.21.15.94:0/1800622132 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85440074e0 con 0x7f854c1586f0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.726+0000 7f853affd640 1 -- 172.21.15.94:0/1800622132 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85440047e0 con 0x7f854c1586f0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.726+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c14efc0 con 0x7f854c1586f0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.726+0000 7f853affd640 1 -- 172.21.15.94:0/1800622132 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544004bd0 con 0x7f854c1586f0 2024-03-06T07:24:43.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.726+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c14f580 con 0x7f854c1586f0 2024-03-06T07:24:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.727+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c04f330 con 0x7f854c1586f0 2024-03-06T07:24:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.727+0000 7f853affd640 1 -- 172.21.15.94:0/1800622132 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8544011070 con 0x7f854c1586f0 2024-03-06T07:24:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.727+0000 7f853affd640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f852c06b500 0x7f852c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.728+0000 7f853affd640 1 -- 172.21.15.94:0/1800622132 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f8544089e50 con 0x7f854c1586f0 2024-03-06T07:24:43.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.728+0000 7f854a59c640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f852c06b500 0x7f852c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.728+0000 7f854a59c640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f852c06b500 0x7f852c06d9c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f853c036000 tx=0x7f853c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.732+0000 7f853affd640 1 -- 172.21.15.94:0/1800622132 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8544058480 con 0x7f854c1586f0 2024-03-06T07:24:43.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.944+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f854c06e0c0 con 0x7f854c1586f0 2024-03-06T07:24:43.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.945+0000 7f853affd640 1 -- 172.21.15.94:0/1800622132 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f854405c520 con 0x7f854c1586f0 2024-03-06T07:24:43.947 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:43.947 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":152,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1221074247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1221074247},{"type":"v1","addr":"172.21.15.94:6834","nonce":1221074247}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":37370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1409431517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1409431517},{"type":"v1","addr":"172.21.15.145:6837","nonce":1409431517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:42.767355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8152":{"gid":8152,"name":"d","rank":0,"incarnation":149,"state":"up:active","state_seq":14,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8152,"qdb_cluster":[8152]},"id":12}]} 2024-03-06T07:24:43.947 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 152 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.946+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f852c06b500 msgr2=0x7f852c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.946+0000 7f85508c9640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f852c06b500 0x7f852c06d9c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f853c036000 tx=0x7f853c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.946+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c1586f0 msgr2=0x7f854c1ee090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.946+0000 7f85508c9640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c1586f0 0x7f854c1ee090 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f854400c3f0 tx=0x7f854400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 shutdown_connections 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f852c06b500 0x7f852c06d9c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f854c1586f0 0x7f854c1ee090 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f854c0c5860 0x7f854c14e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 --2- 172.21.15.94:0/1800622132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f854c0c5480 0x7f854c151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 >> 172.21.15.94:0/1800622132 conn(0x7f854c06b1a0 msgr2=0x7f854c0702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 shutdown_connections 2024-03-06T07:24:43.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:43.947+0000 7f85508c9640 1 -- 172.21.15.94:0/1800622132 wait complete. 2024-03-06T07:24:43.959 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:44.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.058+0000 7fbe545f6640 1 Processor -- start 2024-03-06T07:24:44.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.058+0000 7fbe545f6640 1 -- start start 2024-03-06T07:24:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.058+0000 7fbe545f6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbe4c150580 0x7fbe4c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.058+0000 7fbe545f6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbe4c0c1320 0x7fbe4c06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.058+0000 7fbe545f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c153a60 con 0x7fbe4c0c6000 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.058+0000 7fbe545f6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4c153d40 con 0x7fbe4c150580 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.058+0000 7fbe545f6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4c06c2c0 con 0x7fbe4c0c1320 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe52392640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbe4c150580 0x7fbe4c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe52392640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbe4c150580 0x7fbe4c06dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39232/0 (socket says 172.21.15.94:39232) 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe52392640 1 -- 172.21.15.94:0/226651418 learned_addr learned my addr 172.21.15.94:0/226651418 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe51b91640 1 --2- 172.21.15.94:0/226651418 >> v2:172.21.15.145:3300/0 conn(0x7fbe4c0c1320 0x7fbe4c06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe52392640 1 -- 172.21.15.94:0/226651418 >> v2:172.21.15.145:3300/0 conn(0x7fbe4c0c1320 msgr2=0x7fbe4c06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe52392640 1 --2- 172.21.15.94:0/226651418 >> v2:172.21.15.145:3300/0 conn(0x7fbe4c0c1320 0x7fbe4c06e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe52392640 1 -- 172.21.15.94:0/226651418 >> v1:172.21.15.145:6789/0 conn(0x7fbe4c0c6000 legacy=0x7fbe4c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.059+0000 7fbe52392640 1 -- 172.21.15.94:0/226651418 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c06c5a0 con 0x7fbe4c150580 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe52392640 1 --2- 172.21.15.94:0/226651418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c150580 0x7fbe4c06dd10 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0c7730 tx=0x7fbe3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb42ba312694a89e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe51390640 1 -- 172.21.15.94:0/226651418 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c035020 con 0x7fbe4c150580 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe51390640 1 -- 172.21.15.94:0/226651418 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3c031870 con 0x7fbe4c150580 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe51390640 1 -- 172.21.15.94:0/226651418 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c031c40 con 0x7fbe4c150580 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 -- 172.21.15.94:0/226651418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c150580 msgr2=0x7fbe4c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 --2- 172.21.15.94:0/226651418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c150580 0x7fbe4c06dd10 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0c7730 tx=0x7fbe3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 -- 172.21.15.94:0/226651418 shutdown_connections 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 --2- 172.21.15.94:0/226651418 >> v2:172.21.15.145:3300/0 conn(0x7fbe4c0c1320 0x7fbe4c06e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 --2- 172.21.15.94:0/226651418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c150580 0x7fbe4c06dd10 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 -- 172.21.15.94:0/226651418 >> 172.21.15.94:0/226651418 conn(0x7fbe4c0c6710 msgr2=0x7fbe4c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 -- 172.21.15.94:0/226651418 shutdown_connections 2024-03-06T07:24:44.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.060+0000 7fbe545f6640 1 -- 172.21.15.94:0/226651418 wait complete. 2024-03-06T07:24:44.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.061+0000 7fbe545f6640 1 Processor -- start 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe545f6640 1 -- start start 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe545f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe4c0c1320 0x7fbe4c0a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe545f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c0c6000 0x7fbe4c1458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe545f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe4c150580 0x7fbe4c145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe545f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c0c0dc0 con 0x7fbe4c0c6000 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe545f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c0c0c40 con 0x7fbe4c150580 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe545f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c156dc0 con 0x7fbe4c0c1320 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe51b91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c0c6000 0x7fbe4c1458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe51b91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c0c6000 0x7fbe4c1458d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39242/0 (socket says 172.21.15.94:39242) 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe51b91640 1 -- 172.21.15.94:0/1991220838 learned_addr learned my addr 172.21.15.94:0/1991220838 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe52b93640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe4c150580 0x7fbe4c145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.062+0000 7fbe52392640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe4c0c1320 0x7fbe4c0a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe51b91640 1 -- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe4c0c1320 msgr2=0x7fbe4c0a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe51b91640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe4c0c1320 0x7fbe4c0a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe51b91640 1 -- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe4c150580 msgr2=0x7fbe4c145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe51b91640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe4c150580 0x7fbe4c145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe51b91640 1 -- 172.21.15.94:0/1991220838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c1eaaa0 con 0x7fbe4c0c6000 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe51b91640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c0c6000 0x7fbe4c1458d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800e9e0 tx=0x7fbe4800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe42ffd640 1 -- 172.21.15.94:0/1991220838 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48017070 con 0x7fbe4c0c6000 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe42ffd640 1 -- 172.21.15.94:0/1991220838 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4800c9d0 con 0x7fbe4c0c6000 2024-03-06T07:24:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c1ead60 con 0x7fbe4c0c6000 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe42ffd640 1 -- 172.21.15.94:0/1991220838 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4800cdc0 con 0x7fbe4c0c6000 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.063+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c1eb320 con 0x7fbe4c0c6000 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.064+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c04f330 con 0x7fbe4c0c6000 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.064+0000 7fbe42ffd640 1 -- 172.21.15.94:0/1991220838 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe4801a070 con 0x7fbe4c0c6000 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.065+0000 7fbe42ffd640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe3006b4d0 0x7fbe3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.065+0000 7fbe42ffd640 1 -- 172.21.15.94:0/1991220838 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7fbe4808a3e0 con 0x7fbe4c0c6000 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.065+0000 7fbe52392640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe3006b4d0 0x7fbe3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.065+0000 7fbe52392640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe3006b4d0 0x7fbe3006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c036000 tx=0x7fbe3c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.069+0000 7fbe42ffd640 1 -- 172.21.15.94:0/1991220838 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe48058970 con 0x7fbe4c0c6000 2024-03-06T07:24:44.154 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:24:44.152+0000 7fd58006c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:44.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.274+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe4c0c5de0 con 0x7fbe4c0c6000 2024-03-06T07:24:44.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.275+0000 7fbe42ffd640 1 -- 172.21.15.94:0/1991220838 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+4998 (secure 0 0 0) 0x7fbe4805ca10 con 0x7fbe4c0c6000 2024-03-06T07:24:44.277 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:44.277 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":152,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1221074247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1221074247},{"type":"v1","addr":"172.21.15.94:6834","nonce":1221074247}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150},{"gid":37370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1409431517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1409431517},{"type":"v1","addr":"172.21.15.145:6837","nonce":1409431517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:16.636350+0000","modified":"2024-03-06T07:24:42.767355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8152},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8152":{"gid":8152,"name":"d","rank":0,"incarnation":149,"state":"up:active","state_seq":14,"addr":"172.21.15.145:6836/1521427660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1521427660},{"type":"v1","addr":"172.21.15.145:6836","nonce":1521427660}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8152,"qdb_cluster":[8152]},"id":12}]} 2024-03-06T07:24:44.277 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 152 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe3006b4d0 msgr2=0x7fbe3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe3006b4d0 0x7fbe3006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c036000 tx=0x7fbe3c03c040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c0c6000 msgr2=0x7fbe4c1458d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c0c6000 0x7fbe4c1458d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800e9e0 tx=0x7fbe4800eea0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 shutdown_connections 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe3006b4d0 0x7fbe3006d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe4c150580 0x7fbe4c145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe4c0c6000 0x7fbe4c1458d0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 --2- 172.21.15.94:0/1991220838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe4c0c1320 0x7fbe4c0a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 >> 172.21.15.94:0/1991220838 conn(0x7fbe4c0c6710 msgr2=0x7fbe4c0c4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 shutdown_connections 2024-03-06T07:24:44.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.277+0000 7fbe545f6640 1 -- 172.21.15.94:0/1991220838 wait complete. 2024-03-06T07:24:44.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:44.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.388+0000 7f85836b8640 1 Processor -- start 2024-03-06T07:24:44.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.388+0000 7f85836b8640 1 -- start start 2024-03-06T07:24:44.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f85836b8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c072e70 0x7f857c0bd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f85836b8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f857c073250 0x7f857c0c3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f85836b8640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f857c0c3950 0x7f857c1589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f85836b8640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857c06c8e0 con 0x7f857c0c3950 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f85836b8640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857c06ce80 con 0x7f857c073250 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f85836b8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0c1b00 con 0x7f857c072e70 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f857c0c3950 0x7f857c1589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f857c0c3950 0x7f857c1589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39248/0 (socket says 172.21.15.94:39248) 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 -- 172.21.15.94:0/1038795462 learned_addr learned my addr 172.21.15.94:0/1038795462 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581454640 1 --2- 172.21.15.94:0/1038795462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c072e70 0x7f857c0bd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8580c53640 1 --2- 172.21.15.94:0/1038795462 >> v2:172.21.15.145:3300/0 conn(0x7f857c073250 0x7f857c0c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 -- 172.21.15.94:0/1038795462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c072e70 msgr2=0x7f857c0bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 --2- 172.21.15.94:0/1038795462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c072e70 0x7f857c0bd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 -- 172.21.15.94:0/1038795462 >> v2:172.21.15.145:3300/0 conn(0x7f857c073250 msgr2=0x7f857c0c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 --2- 172.21.15.94:0/1038795462 >> v2:172.21.15.145:3300/0 conn(0x7f857c073250 0x7f857c0c3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.389+0000 7f8581c55640 1 -- 172.21.15.94:0/1038795462 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c06cba0 con 0x7f857c0c3950 2024-03-06T07:24:44.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.390+0000 7f8581c55640 1 --2- 172.21.15.94:0/1038795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1589b0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f857c068140 tx=0x7f857402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b00b9d22d4ed8e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.390+0000 7f8573fff640 1 -- 172.21.15.94:0/1038795462 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857402e070 con 0x7f857c0c3950 2024-03-06T07:24:44.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.390+0000 7f8573fff640 1 -- 172.21.15.94:0/1038795462 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8574037070 con 0x7f857c0c3950 2024-03-06T07:24:44.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.390+0000 7f8573fff640 1 -- 172.21.15.94:0/1038795462 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8574031a00 con 0x7f857c0c3950 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.390+0000 7f85836b8640 1 -- 172.21.15.94:0/1038795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 msgr2=0x7f857c1589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.390+0000 7f85836b8640 1 --2- 172.21.15.94:0/1038795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1589b0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f857c068140 tx=0x7f857402b880 comp rx=0 tx=0).stop 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.391+0000 7f85836b8640 1 -- 172.21.15.94:0/1038795462 shutdown_connections 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.391+0000 7f85836b8640 1 --2- 172.21.15.94:0/1038795462 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1589b0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.391+0000 7f85836b8640 1 --2- 172.21.15.94:0/1038795462 >> v2:172.21.15.145:3300/0 conn(0x7f857c073250 0x7f857c0c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.391+0000 7f85836b8640 1 --2- 172.21.15.94:0/1038795462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c072e70 0x7f857c0bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.391+0000 7f85836b8640 1 -- 172.21.15.94:0/1038795462 >> 172.21.15.94:0/1038795462 conn(0x7f857c06b1a0 msgr2=0x7f857c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.391+0000 7f85836b8640 1 -- 172.21.15.94:0/1038795462 shutdown_connections 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.391+0000 7f85836b8640 1 -- 172.21.15.94:0/1038795462 wait complete. 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 Processor -- start 2024-03-06T07:24:44.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 -- start start 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f857c072e70 0x7f857c157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c073250 0x7f857c154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1e7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0c4b80 con 0x7f857c0c3950 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0c4a00 con 0x7f857c072e70 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f85836b8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857c0bda50 con 0x7f857c073250 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f8581c55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1e7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f8581c55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1e7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39258/0 (socket says 172.21.15.94:39258) 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f8581c55640 1 -- 172.21.15.94:0/4158342685 learned_addr learned my addr 172.21.15.94:0/4158342685 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f8580c53640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c073250 0x7f857c154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.392+0000 7f8581454640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f857c072e70 0x7f857c157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8581c55640 1 -- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c073250 msgr2=0x7f857c154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8581c55640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c073250 0x7f857c154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8581c55640 1 -- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f857c072e70 msgr2=0x7f857c157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8581c55640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f857c072e70 0x7f857c157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8581c55640 1 -- 172.21.15.94:0/4158342685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c1eb570 con 0x7f857c0c3950 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8581c55640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1e7fa0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f85740090b0 tx=0x7f85740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8571ffb640 1 -- 172.21.15.94:0/4158342685 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8574032670 con 0x7f857c0c3950 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8571ffb640 1 -- 172.21.15.94:0/4158342685 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8574032830 con 0x7f857c0c3950 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f8571ffb640 1 -- 172.21.15.94:0/4158342685 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8574032c20 con 0x7f857c0c3950 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857c153d60 con 0x7f857c0c3950 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.393+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857c157840 con 0x7f857c0c3950 2024-03-06T07:24:44.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.394+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857c0706e0 con 0x7f857c0c3950 2024-03-06T07:24:44.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.395+0000 7f8571ffb640 1 -- 172.21.15.94:0/4158342685 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f857404c060 con 0x7f857c0c3950 2024-03-06T07:24:44.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.395+0000 7f8571ffb640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f856006b4d0 0x7f856006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.395+0000 7f8571ffb640 1 -- 172.21.15.94:0/4158342685 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7f85740abc60 con 0x7f857c0c3950 2024-03-06T07:24:44.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.395+0000 7f8581454640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f856006b4d0 0x7f856006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.395+0000 7f8581454640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f856006b4d0 0x7f856006d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8578011200 tx=0x7f857800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.399+0000 7f8571ffb640 1 -- 172.21.15.94:0/4158342685 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f857c0706e0 con 0x7f857c0c3950 2024-03-06T07:24:44.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.571+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f857c04f330 con 0x7f857c0c3950 2024-03-06T07:24:44.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.572+0000 7f8571ffb640 1 -- 172.21.15.94:0/4158342685 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+17281 (secure 0 0 0) 0x7f8574040070 con 0x7f857c0c3950 2024-03-06T07:24:44.574 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:44.574 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":158,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:41.762609+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-03-06T07:24:15.094089+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-03-06T07:24:15.572143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6834/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.94:6832/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.145:6835/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.145:6837/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.94:6835/1613596341":"2024-03-07T07:24:23.693503+0000","172.21.15.94:6833/1613596341":"2024-03-07T07:24:23.693503+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:44.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f856006b4d0 msgr2=0x7f856006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f856006b4d0 0x7f856006d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8578011200 tx=0x7f857800d040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 msgr2=0x7f857c1e7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1e7fa0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f85740090b0 tx=0x7f85740046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 shutdown_connections 2024-03-06T07:24:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f856006b4d0 0x7f856006d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f857c0c3950 0x7f857c1e7fa0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f857c073250 0x7f857c154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 --2- 172.21.15.94:0/4158342685 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f857c072e70 0x7f857c157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 >> 172.21.15.94:0/4158342685 conn(0x7f857c06b1a0 msgr2=0x7f857c0bba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 shutdown_connections 2024-03-06T07:24:44.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.574+0000 7f85836b8640 1 -- 172.21.15.94:0/4158342685 wait complete. 2024-03-06T07:24:44.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-03-06T07:24:44.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.686+0000 7fe64a35c640 1 Processor -- start 2024-03-06T07:24:44.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.686+0000 7fe64a35c640 1 -- start start 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.686+0000 7fe64a35c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6441561a0 0x7fe6441529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.686+0000 7fe64a35c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6440a9990 con 0x7fe64414f110 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.686+0000 7fe64a35c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6441556c0 con 0x7fe6440c6ab0 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.686+0000 7fe64a35c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe644155d10 con 0x7fe6441561a0 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.686+0000 7fe6437fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6441561a0 0x7fe6441529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.687+0000 7fe6437fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6441561a0 0x7fe6441529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39272/0 (socket says 172.21.15.94:39272) 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.687+0000 7fe6488f9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe64414f110 0x7fe64415c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45844/0 (socket says 172.21.15.94:45844) 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.687+0000 7fe6437fe640 1 -- 172.21.15.94:0/2260489365 learned_addr learned my addr 172.21.15.94:0/2260489365 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.687+0000 7fe6437fe640 1 -- 172.21.15.94:0/2260489365 >> v1:172.21.15.145:6789/0 conn(0x7fe6440c6ab0 legacy=0x7fe6440a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.687+0000 7fe6437fe640 1 -- 172.21.15.94:0/2260489365 >> v1:172.21.15.94:6789/0 conn(0x7fe64414f110 legacy=0x7fe64415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.687+0000 7fe642ffd640 1 -- 172.21.15.94:0/2260489365 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450763284 0 0) 0x7fe6441556c0 con 0x7fe6440c6ab0 2024-03-06T07:24:44.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.687+0000 7fe6437fe640 1 -- 172.21.15.94:0/2260489365 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6441559a0 con 0x7fe6441561a0 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe6437fe640 1 --2- 172.21.15.94:0/2260489365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441529e0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fe634009870 tx=0x7fe63402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118fde103238b44a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe642ffd640 1 -- 172.21.15.94:0/2260489365 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634037070 con 0x7fe6441561a0 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe642ffd640 1 -- 172.21.15.94:0/2260489365 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe634004230 con 0x7fe6441561a0 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe642ffd640 1 -- 172.21.15.94:0/2260489365 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634003610 con 0x7fe6441561a0 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe64a35c640 1 -- 172.21.15.94:0/2260489365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 msgr2=0x7fe6441529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2260489365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441529e0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fe634009870 tx=0x7fe63402b7e0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe64a35c640 1 -- 172.21.15.94:0/2260489365 shutdown_connections 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2260489365 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441529e0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe64a35c640 1 -- 172.21.15.94:0/2260489365 >> 172.21.15.94:0/2260489365 conn(0x7fe64406b1a0 msgr2=0x7fe64406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe64a35c640 1 -- 172.21.15.94:0/2260489365 shutdown_connections 2024-03-06T07:24:44.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.688+0000 7fe64a35c640 1 -- 172.21.15.94:0/2260489365 wait complete. 2024-03-06T07:24:44.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.689+0000 7fe64a35c640 1 Processor -- start 2024-03-06T07:24:44.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe64a35c640 1 -- start start 2024-03-06T07:24:44.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe64a35c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6440c6ab0 0x7fe644146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe64a35c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe64414f110 0x7fe644150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe64a35c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe64a35c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440aa600 con 0x7fe6441561a0 2024-03-06T07:24:44.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe64a35c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6440c2a90 con 0x7fe6440c6ab0 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe64a35c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441e8d80 con 0x7fe64414f110 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe6488f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe6488f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441445d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39274/0 (socket says 172.21.15.94:39274) 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe6488f9640 1 -- 172.21.15.94:0/2416142820 learned_addr learned my addr 172.21.15.94:0/2416142820 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe643fff640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6440c6ab0 0x7fe644146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.690+0000 7fe6437fe640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe64414f110 0x7fe644150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe6488f9640 1 -- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe64414f110 msgr2=0x7fe644150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe6488f9640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe64414f110 0x7fe644150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe6488f9640 1 -- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6440c6ab0 msgr2=0x7fe644146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe6488f9640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6440c6ab0 0x7fe644146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe6488f9640 1 -- 172.21.15.94:0/2416142820 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6441e5ea0 con 0x7fe6441561a0 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe6488f9640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441445d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00cb20 tx=0x7fe63c0020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe640ff9640 1 -- 172.21.15.94:0/2416142820 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c018070 con 0x7fe6441561a0 2024-03-06T07:24:44.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe640ff9640 1 -- 172.21.15.94:0/2416142820 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c00a760 con 0x7fe6441561a0 2024-03-06T07:24:44.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6441e8450 con 0x7fe6441561a0 2024-03-06T07:24:44.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe640ff9640 1 -- 172.21.15.94:0/2416142820 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00ab30 con 0x7fe6441561a0 2024-03-06T07:24:44.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.691+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6441e8a10 con 0x7fe6441561a0 2024-03-06T07:24:44.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.692+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64404f330 con 0x7fe6441561a0 2024-03-06T07:24:44.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.693+0000 7fe640ff9640 1 -- 172.21.15.94:0/2416142820 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe63c01b070 con 0x7fe6441561a0 2024-03-06T07:24:44.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.693+0000 7fe640ff9640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62406b500 0x7fe62406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.693+0000 7fe640ff9640 1 -- 172.21.15.94:0/2416142820 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6363+0+0 (secure 0 0 0) 0x7fe63c089b40 con 0x7fe6441561a0 2024-03-06T07:24:44.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.693+0000 7fe643fff640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62406b500 0x7fe62406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.693+0000 7fe643fff640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62406b500 0x7fe62406d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe62c000f60 tx=0x7fe62c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.697+0000 7fe640ff9640 1 -- 172.21.15.94:0/2416142820 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe63c0580e0 con 0x7fe6441561a0 2024-03-06T07:24:44.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.905+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7fe6440c2690 con 0x7fe6441561a0 2024-03-06T07:24:44.908 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:24:44.906+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:44.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.907+0000 7fe640ff9640 1 -- 172.21.15.94:0/2416142820 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v153) v1 ==== 163+0+0 (secure 0 0 0) 0x7fe63c05c180 con 0x7fe6441561a0 2024-03-06T07:24:44.909 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.909+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62406b500 msgr2=0x7fe62406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.909+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62406b500 0x7fe62406d9c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe62c000f60 tx=0x7fe62c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.909+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 msgr2=0x7fe6441445d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.909+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441445d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00cb20 tx=0x7fe63c0020f0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 shutdown_connections 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62406b500 0x7fe62406d9c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6441561a0 0x7fe6441445d0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.912 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe64414f110 0x7fe644150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 --2- 172.21.15.94:0/2416142820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6440c6ab0 0x7fe644146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 >> 172.21.15.94:0/2416142820 conn(0x7fe64406b1a0 msgr2=0x7fe64406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 shutdown_connections 2024-03-06T07:24:44.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:44.910+0000 7fe64a35c640 1 -- 172.21.15.94:0/2416142820 wait complete. 2024-03-06T07:24:44.914 INFO:tasks.ceph.mds.d.smithi145.stderr: -37> 2024-03-06T07:24:44.152+0000 7fd58006c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:44.922 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-03-06T07:24:44.937 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.024+0000 7fd000ab8640 1 Processor -- start 2024-03-06T07:24:45.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.024+0000 7fd000ab8640 1 -- start start 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fd000ab8640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcffc14ef70 0x7fcffc06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fd000ab8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc0c2790 con 0x7fcffc06cad0 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fd000ab8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc06cf40 con 0x7fcffc0c1820 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fd000ab8640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcffc0c3090 con 0x7fcffc14ef70 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fcffa59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcffc14ef70 0x7fcffc06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fcffa59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcffc14ef70 0x7fcffc06c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39290/0 (socket says 172.21.15.94:39290) 2024-03-06T07:24:45.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fcffad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcffc06cad0 0x7fcffc154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45856/0 (socket says 172.21.15.94:45856) 2024-03-06T07:24:45.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fcffa59c640 1 -- 172.21.15.94:0/3748423321 learned_addr learned my addr 172.21.15.94:0/3748423321 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:45.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fcffa59c640 1 -- 172.21.15.94:0/3748423321 >> v1:172.21.15.145:6789/0 conn(0x7fcffc0c1820 legacy=0x7fcffc0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fcffa59c640 1 -- 172.21.15.94:0/3748423321 >> v1:172.21.15.94:6789/0 conn(0x7fcffc06cad0 legacy=0x7fcffc154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.025+0000 7fcffa59c640 1 -- 172.21.15.94:0/3748423321 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc0c1540 con 0x7fcffc14ef70 2024-03-06T07:24:45.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.026+0000 7fcffa59c640 1 --2- 172.21.15.94:0/3748423321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc06c1f0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009870 tx=0x7fcfec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eae4fff11533f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.026+0000 7fcff959a640 1 -- 172.21.15.94:0/3748423321 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec034070 con 0x7fcffc14ef70 2024-03-06T07:24:45.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.026+0000 7fcff959a640 1 -- 172.21.15.94:0/3748423321 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec030860 con 0x7fcffc14ef70 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.026+0000 7fcff959a640 1 -- 172.21.15.94:0/3748423321 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec030c30 con 0x7fcffc14ef70 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.026+0000 7fd000ab8640 1 -- 172.21.15.94:0/3748423321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 msgr2=0x7fcffc06c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.026+0000 7fd000ab8640 1 --2- 172.21.15.94:0/3748423321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc06c1f0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009870 tx=0x7fcfec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.027+0000 7fd000ab8640 1 -- 172.21.15.94:0/3748423321 shutdown_connections 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.027+0000 7fd000ab8640 1 --2- 172.21.15.94:0/3748423321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc06c1f0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.027+0000 7fd000ab8640 1 -- 172.21.15.94:0/3748423321 >> 172.21.15.94:0/3748423321 conn(0x7fcffc0c6710 msgr2=0x7fcffc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.027+0000 7fd000ab8640 1 -- 172.21.15.94:0/3748423321 shutdown_connections 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.027+0000 7fd000ab8640 1 -- 172.21.15.94:0/3748423321 wait complete. 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 Processor -- start 2024-03-06T07:24:45.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 -- start start 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcffc06cad0 0x7fcffc14f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcffc0c1820 0x7fcffc151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc155750 con 0x7fcffc14ef70 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc14f350 con 0x7fcffc06cad0 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.028+0000 7fd000ab8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc155340 con 0x7fcffc0c1820 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39306/0 (socket says 172.21.15.94:39306) 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 -- 172.21.15.94:0/2454013076 learned_addr learned my addr 172.21.15.94:0/2454013076 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffa59c640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcffc06cad0 0x7fcffc14f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcff9d9b640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcffc0c1820 0x7fcffc151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 -- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcffc0c1820 msgr2=0x7fcffc151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcffc0c1820 0x7fcffc151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 -- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcffc06cad0 msgr2=0x7fcffc14f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcffc06cad0 0x7fcffc14f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 -- 172.21.15.94:0/2454013076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc151750 con 0x7fcffc14ef70 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcffad9d640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc151f10 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c780 tx=0x7fcff400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcfeaffd640 1 -- 172.21.15.94:0/2454013076 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4017070 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcfeaffd640 1 -- 172.21.15.94:0/2454013076 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff400a640 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc0a8590 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fcfeaffd640 1 -- 172.21.15.94:0/2454013076 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff400aa30 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.029+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc0a8280 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.030+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcffc04f330 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.031+0000 7fcfeaffd640 1 -- 172.21.15.94:0/2454013076 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcff401a070 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.031+0000 7fcfeaffd640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.031+0000 7fcfeaffd640 1 -- 172.21.15.94:0/2454013076 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7fcff4089910 con 0x7fcffc14ef70 2024-03-06T07:24:45.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.031+0000 7fcffa59c640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:45.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.032+0000 7fcffa59c640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcfec036000 tx=0x7fcfec03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:45.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.035+0000 7fcfeaffd640 1 -- 172.21.15.94:0/2454013076 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcff4057ee0 con 0x7fcffc14ef70 2024-03-06T07:24:45.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.241+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcffc0c1110 con 0x7fcffc14ef70 2024-03-06T07:24:45.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:45.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:45.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.909+0000 7fcfeaffd640 1 -- 172.21.15.94:0/2454013076 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v154) v1 ==== 105+0+0 (secure 0 0 0) 0x7fcff405bf80 con 0x7fcffc14ef70 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfdc06b4d0 msgr2=0x7fcfdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcfec036000 tx=0x7fcfec03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 msgr2=0x7fcffc151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc151f10 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c780 tx=0x7fcff400cc40 comp rx=0 tx=0).stop 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 shutdown_connections 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc14ef70 0x7fcffc151f10 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcffc0c1820 0x7fcffc151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 --2- 172.21.15.94:0/2454013076 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcffc06cad0 0x7fcffc14f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 >> 172.21.15.94:0/2454013076 conn(0x7fcffc0c6710 msgr2=0x7fcffc0c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 shutdown_connections 2024-03-06T07:24:45.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:45.911+0000 7fd000ab8640 1 -- 172.21.15.94:0/2454013076 wait complete. 2024-03-06T07:24:45.924 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:46.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f270e21c640 1 Processor -- start 2024-03-06T07:24:46.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f270e21c640 1 -- start start 2024-03-06T07:24:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f270e21c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f270814f180 0x7f27080c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f270e21c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27081551d0 0x7f27081566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f270e21c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27080c8f60 con 0x7f27080ca000 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f270e21c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2708156c00 con 0x7f27081551d0 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f270e21c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27080c9240 con 0x7f270814f180 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f2706ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27081551d0 0x7f27081566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.028+0000 7f2707fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f27080ca000 0x7f270806c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45862/0 (socket says 172.21.15.94:45862) 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f2706ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27081551d0 0x7f27081566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39318/0 (socket says 172.21.15.94:39318) 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f2707fff640 1 -- 172.21.15.94:0/1102218291 learned_addr learned my addr 172.21.15.94:0/1102218291 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f27077fe640 1 --2- 172.21.15.94:0/1102218291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f270814f180 0x7f27080c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f2706ffd640 1 -- 172.21.15.94:0/1102218291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f270814f180 msgr2=0x7f27080c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f2706ffd640 1 --2- 172.21.15.94:0/1102218291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f270814f180 0x7f27080c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f2706ffd640 1 -- 172.21.15.94:0/1102218291 >> v1:172.21.15.94:6789/0 conn(0x7f27080ca000 legacy=0x7f270806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f2706ffd640 1 -- 172.21.15.94:0/1102218291 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27080c9520 con 0x7f27081551d0 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.029+0000 7f27067fc640 1 -- 172.21.15.94:0/1102218291 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489493762 0 0) 0x7f27080c8f60 con 0x7f27080ca000 2024-03-06T07:24:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f2706ffd640 1 --2- 172.21.15.94:0/1102218291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f27081566c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0097a0 tx=0x7f26fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac0d52a28d31d5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f27067fc640 1 -- 172.21.15.94:0/1102218291 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035020 con 0x7f27081551d0 2024-03-06T07:24:46.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f27067fc640 1 -- 172.21.15.94:0/1102218291 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc031830 con 0x7f27081551d0 2024-03-06T07:24:46.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f27067fc640 1 -- 172.21.15.94:0/1102218291 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc031c00 con 0x7f27081551d0 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 -- 172.21.15.94:0/1102218291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 msgr2=0x7f27081566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 --2- 172.21.15.94:0/1102218291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f27081566c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0097a0 tx=0x7f26fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 -- 172.21.15.94:0/1102218291 shutdown_connections 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 --2- 172.21.15.94:0/1102218291 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f27081566c0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 --2- 172.21.15.94:0/1102218291 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f270814f180 0x7f27080c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 -- 172.21.15.94:0/1102218291 >> 172.21.15.94:0/1102218291 conn(0x7f2708072d20 msgr2=0x7f27080ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 -- 172.21.15.94:0/1102218291 shutdown_connections 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.030+0000 7f270e21c640 1 -- 172.21.15.94:0/1102218291 wait complete. 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.031+0000 7f270e21c640 1 Processor -- start 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f270e21c640 1 -- start start 2024-03-06T07:24:46.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f270e21c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27080ca000 0x7f270814fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f270e21c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f270814f180 0x7f27081eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f270e21c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f270814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f270e21c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270814f560 con 0x7f27081551d0 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f270e21c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27080c9ae0 con 0x7f270814f180 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f270e21c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270814f6e0 con 0x7f27080ca000 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f2707fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f270814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f2707fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f270814d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39334/0 (socket says 172.21.15.94:39334) 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f2707fff640 1 -- 172.21.15.94:0/2361944244 learned_addr learned my addr 172.21.15.94:0/2361944244 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f27077fe640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27080ca000 0x7f270814fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.032+0000 7f2706ffd640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f270814f180 0x7f27081eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f2707fff640 1 -- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27080ca000 msgr2=0x7f270814fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f2707fff640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27080ca000 0x7f270814fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f2707fff640 1 -- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f270814f180 msgr2=0x7f27081eb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f2707fff640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f270814f180 0x7f27081eb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f2707fff640 1 -- 172.21.15.94:0/2361944244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27080af520 con 0x7f27081551d0 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f27077fe640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27080ca000 0x7f270814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f2706ffd640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f270814f180 0x7f27081eb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f2707fff640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f270814d920 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f270000c700 tx=0x7f270000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f26effff640 1 -- 172.21.15.94:0/2361944244 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2700018070 con 0x7f27081551d0 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f26effff640 1 -- 172.21.15.94:0/2361944244 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270000ce10 con 0x7f27081551d0 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27080af800 con 0x7f27081551d0 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f26effff640 1 -- 172.21.15.94:0/2361944244 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270000a860 con 0x7f27081551d0 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.033+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270814de60 con 0x7f27081551d0 2024-03-06T07:24:46.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.034+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270804f330 con 0x7f27081551d0 2024-03-06T07:24:46.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.034+0000 7f26effff640 1 -- 172.21.15.94:0/2361944244 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f270001b070 con 0x7f27081551d0 2024-03-06T07:24:46.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.035+0000 7f26effff640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26f406b500 0x7f26f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.035+0000 7f26effff640 1 -- 172.21.15.94:0/2361944244 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 6449+0+0 (secure 0 0 0) 0x7f2700089970 con 0x7f27081551d0 2024-03-06T07:24:46.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.035+0000 7f27077fe640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26f406b500 0x7f26f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.036+0000 7f27077fe640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26f406b500 0x7f26f406d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f26f8006fd0 tx=0x7f26f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.039+0000 7f26effff640 1 -- 172.21.15.94:0/2361944244 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2700058590 con 0x7f27081551d0 2024-03-06T07:24:46.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.208+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2708159160 con 0x7f27081551d0 2024-03-06T07:24:46.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.915+0000 7f26effff640 1 -- 172.21.15.94:0/2361944244 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v160) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2700057e90 con 0x7f27081551d0 2024-03-06T07:24:46.918 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_metadata' removed 2024-03-06T07:24:46.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26f406b500 msgr2=0x7f26f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26f406b500 0x7f26f406d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f26f8006fd0 tx=0x7f26f8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 msgr2=0x7f270814d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f270814d920 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f270000c700 tx=0x7f270000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 shutdown_connections 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f26f406b500 0x7f26f406d9c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27081551d0 0x7f270814d920 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f270814f180 0x7f27081eb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.918+0000 7f270e21c640 1 --2- 172.21.15.94:0/2361944244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27080ca000 0x7f270814fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.919+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 >> 172.21.15.94:0/2361944244 conn(0x7f2708072d20 msgr2=0x7f27081546e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.919+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 shutdown_connections 2024-03-06T07:24:46.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:46.919+0000 7f270e21c640 1 -- 172.21.15.94:0/2361944244 wait complete. 2024-03-06T07:24:46.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-03-06T07:24:47.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.035+0000 7fdb2cc1c640 1 Processor -- start 2024-03-06T07:24:47.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.035+0000 7fdb2cc1c640 1 -- start start 2024-03-06T07:24:47.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2cc1c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdb281527a0 0x7fdb28155040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2cc1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb280c1860 0x7fdb2806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2cc1c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb28155580 con 0x7fdb2806c510 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2cc1c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb28158820 con 0x7fdb281527a0 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2cc1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280c2b10 con 0x7fdb280c1860 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdb281527a0 0x7fdb28155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdb281527a0 0x7fdb28155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39344/0 (socket says 172.21.15.94:39344) 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2659c640 1 -- 172.21.15.94:0/2540033986 learned_addr learned my addr 172.21.15.94:0/2540033986 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb25d9b640 1 --2- 172.21.15.94:0/2540033986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb280c1860 0x7fdb2806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2659c640 1 -- 172.21.15.94:0/2540033986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb280c1860 msgr2=0x7fdb2806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2659c640 1 --2- 172.21.15.94:0/2540033986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb280c1860 0x7fdb2806bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.036+0000 7fdb2659c640 1 -- 172.21.15.94:0/2540033986 >> v1:172.21.15.145:6789/0 conn(0x7fdb2806c510 legacy=0x7fdb28158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.037+0000 7fdb2659c640 1 -- 172.21.15.94:0/2540033986 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28155820 con 0x7fdb281527a0 2024-03-06T07:24:47.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.037+0000 7fdb2659c640 1 --2- 172.21.15.94:0/2540033986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb28155040 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb1802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6ff81284529f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.037+0000 7fdb2559a640 1 -- 172.21.15.94:0/2540033986 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1802ca70 con 0x7fdb281527a0 2024-03-06T07:24:47.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.037+0000 7fdb2559a640 1 -- 172.21.15.94:0/2540033986 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1802cc10 con 0x7fdb281527a0 2024-03-06T07:24:47.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.037+0000 7fdb2559a640 1 -- 172.21.15.94:0/2540033986 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb180343f0 con 0x7fdb281527a0 2024-03-06T07:24:47.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2540033986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 msgr2=0x7fdb28155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2540033986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb28155040 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb1802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2540033986 shutdown_connections 2024-03-06T07:24:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2540033986 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb280c1860 0x7fdb2806bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2540033986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb28155040 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2540033986 >> 172.21.15.94:0/2540033986 conn(0x7fdb280c3960 msgr2=0x7fdb280c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2540033986 shutdown_connections 2024-03-06T07:24:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.038+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2540033986 wait complete. 2024-03-06T07:24:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.039+0000 7fdb2cc1c640 1 Processor -- start 2024-03-06T07:24:47.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.039+0000 7fdb2cc1c640 1 -- start start 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2cc1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb2806c510 0x7fdb2814f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2cc1c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb280c1860 0x7fdb280c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2cc1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb281edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2cc1c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28152b80 con 0x7fdb281527a0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2cc1c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28155de0 con 0x7fdb280c1860 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2cc1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28152d00 con 0x7fdb2806c510 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb281edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb281edbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39360/0 (socket says 172.21.15.94:39360) 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 -- 172.21.15.94:0/2241510551 learned_addr learned my addr 172.21.15.94:0/2241510551 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb25d9b640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb280c1860 0x7fdb280c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2659c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb2806c510 0x7fdb2814f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 -- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb2806c510 msgr2=0x7fdb2814f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb2806c510 0x7fdb2814f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 -- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb280c1860 msgr2=0x7fdb280c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb280c1860 0x7fdb280c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb26d9d640 1 -- 172.21.15.94:0/2241510551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280c5500 con 0x7fdb281527a0 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb25d9b640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb280c1860 0x7fdb280c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.040+0000 7fdb2659c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb2806c510 0x7fdb2814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.041+0000 7fdb26d9d640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb281edbe0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdb20001eb0 tx=0x7fdb2000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.041+0000 7fdb16ffd640 1 -- 172.21.15.94:0/2241510551 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20017070 con 0x7fdb281527a0 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.041+0000 7fdb16ffd640 1 -- 172.21.15.94:0/2241510551 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb200052d0 con 0x7fdb281527a0 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.041+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb281ee520 con 0x7fdb281527a0 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.041+0000 7fdb16ffd640 1 -- 172.21.15.94:0/2241510551 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20003810 con 0x7fdb281527a0 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.041+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb281ee800 con 0x7fdb281527a0 2024-03-06T07:24:47.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.042+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2804f330 con 0x7fdb281527a0 2024-03-06T07:24:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.042+0000 7fdb16ffd640 1 -- 172.21.15.94:0/2241510551 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb2001a070 con 0x7fdb281527a0 2024-03-06T07:24:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.043+0000 7fdb16ffd640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0806b4d0 0x7fdb0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.043+0000 7fdb16ffd640 1 -- 172.21.15.94:0/2241510551 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 6009+0+0 (secure 0 0 0) 0x7fdb20089640 con 0x7fdb281527a0 2024-03-06T07:24:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.043+0000 7fdb2659c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0806b4d0 0x7fdb0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.043+0000 7fdb2659c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0806b4d0 0x7fdb0806d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdb18035000 tx=0x7fdb18030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.047+0000 7fdb16ffd640 1 -- 172.21.15.94:0/2241510551 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb20058420 con 0x7fdb281527a0 2024-03-06T07:24:47.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.218+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdb28067590 con 0x7fdb281527a0 2024-03-06T07:24:47.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.917+0000 7fdb16ffd640 1 -- 172.21.15.94:0/2241510551 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v161) v1 ==== 172+0+0 (secure 0 0 0) 0x7fdb20057d20 con 0x7fdb281527a0 2024-03-06T07:24:47.919 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs2_data' removed 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0806b4d0 msgr2=0x7fdb0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0806b4d0 0x7fdb0806d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdb18035000 tx=0x7fdb18030000 comp rx=0 tx=0).stop 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 msgr2=0x7fdb281edbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb281edbe0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fdb20001eb0 tx=0x7fdb2000c890 comp rx=0 tx=0).stop 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 shutdown_connections 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb0806b4d0 0x7fdb0806d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb281527a0 0x7fdb281edbe0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb280c1860 0x7fdb280c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 --2- 172.21.15.94:0/2241510551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb2806c510 0x7fdb2814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 >> 172.21.15.94:0/2241510551 conn(0x7fdb280c3960 msgr2=0x7fdb2806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.920+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 shutdown_connections 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:47.921+0000 7fdb2cc1c640 1 -- 172.21.15.94:0/2241510551 wait complete. 2024-03-06T07:24:47.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:48.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.045+0000 7f52c428d640 1 Processor -- start 2024-03-06T07:24:48.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.045+0000 7f52c428d640 1 -- start start 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c428d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f52bc06d840 0x7f52bc073090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c428d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc0c69a0 0x7f52bc158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c428d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f52bc158bd0 0x7f52bc0c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c428d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52bc0c3460 con 0x7f52bc06d840 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c428d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52bc0c2050 con 0x7f52bc158bd0 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c428d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc159580 con 0x7f52bc0c69a0 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c2029640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f52bc06d840 0x7f52bc073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c2029640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f52bc06d840 0x7f52bc073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39366/0 (socket says 172.21.15.94:39366) 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c2029640 1 -- 172.21.15.94:0/1827306461 learned_addr learned my addr 172.21.15.94:0/1827306461 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.046+0000 7f52c282a640 1 --2- 172.21.15.94:0/1827306461 >> v2:172.21.15.145:3300/0 conn(0x7f52bc158bd0 0x7f52bc0c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c1828640 1 --2- 172.21.15.94:0/1827306461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc0c69a0 0x7f52bc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c2029640 1 -- 172.21.15.94:0/1827306461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc0c69a0 msgr2=0x7f52bc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c2029640 1 --2- 172.21.15.94:0/1827306461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc0c69a0 0x7f52bc158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c2029640 1 -- 172.21.15.94:0/1827306461 >> v2:172.21.15.145:3300/0 conn(0x7f52bc158bd0 msgr2=0x7f52bc0c24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c2029640 1 --2- 172.21.15.94:0/1827306461 >> v2:172.21.15.145:3300/0 conn(0x7f52bc158bd0 0x7f52bc0c24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c2029640 1 -- 172.21.15.94:0/1827306461 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc0c36b0 con 0x7f52bc06d840 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c1828640 1 --2- 172.21.15.94:0/1827306461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc0c69a0 0x7f52bc158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c2029640 1 --2- 172.21.15.94:0/1827306461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc06d840 0x7f52bc073090 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0c6410 tx=0x7f52b802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b44d31bf9eeb47e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c1027640 1 -- 172.21.15.94:0/1827306461 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b802c950 con 0x7f52bc06d840 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c1027640 1 -- 172.21.15.94:0/1827306461 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b802caf0 con 0x7f52bc06d840 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.047+0000 7f52c1027640 1 -- 172.21.15.94:0/1827306461 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80343f0 con 0x7f52bc06d840 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 -- 172.21.15.94:0/1827306461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc06d840 msgr2=0x7f52bc073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 --2- 172.21.15.94:0/1827306461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc06d840 0x7f52bc073090 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0c6410 tx=0x7f52b802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 -- 172.21.15.94:0/1827306461 shutdown_connections 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 --2- 172.21.15.94:0/1827306461 >> v2:172.21.15.145:3300/0 conn(0x7f52bc158bd0 0x7f52bc0c24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 --2- 172.21.15.94:0/1827306461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc0c69a0 0x7f52bc158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 --2- 172.21.15.94:0/1827306461 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc06d840 0x7f52bc073090 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 -- 172.21.15.94:0/1827306461 >> 172.21.15.94:0/1827306461 conn(0x7f52bc0c5560 msgr2=0x7f52bc0c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 -- 172.21.15.94:0/1827306461 shutdown_connections 2024-03-06T07:24:48.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.048+0000 7f52c428d640 1 -- 172.21.15.94:0/1827306461 wait complete. 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.049+0000 7f52c428d640 1 Processor -- start 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.049+0000 7f52c428d640 1 -- start start 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c428d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc06d840 0x7f52bc1447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c428d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc0c69a0 0x7f52bc06fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c428d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52bc158bd0 0x7f52bc0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c428d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc06dc20 con 0x7f52bc0c69a0 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c428d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc0c3c70 con 0x7f52bc158bd0 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c428d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc06dda0 con 0x7f52bc06d840 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc0c69a0 0x7f52bc06fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc0c69a0 0x7f52bc06fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39370/0 (socket says 172.21.15.94:39370) 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 -- 172.21.15.94:0/3294403191 learned_addr learned my addr 172.21.15.94:0/3294403191 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c2029640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc06d840 0x7f52bc1447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c282a640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52bc158bd0 0x7f52bc0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 -- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc06d840 msgr2=0x7f52bc1447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc06d840 0x7f52bc1447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 -- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52bc158bd0 msgr2=0x7f52bc0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52bc158bd0 0x7f52bc0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 -- 172.21.15.94:0/3294403191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc158640 con 0x7f52bc0c69a0 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c282a640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52bc158bd0 0x7f52bc0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c2029640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc06d840 0x7f52bc1447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c1828640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc0c69a0 0x7f52bc06fb90 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c7f0 tx=0x7f52ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52b2ffd640 1 -- 172.21.15.94:0/3294403191 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac015070 con 0x7f52bc0c69a0 2024-03-06T07:24:48.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52b2ffd640 1 -- 172.21.15.94:0/3294403191 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac007a30 con 0x7f52bc0c69a0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.050+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc1ee0e0 con 0x7f52bc0c69a0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.051+0000 7f52b2ffd640 1 -- 172.21.15.94:0/3294403191 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac007e00 con 0x7f52bc0c69a0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.051+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc1ee6a0 con 0x7f52bc0c69a0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.051+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc04f330 con 0x7f52bc0c69a0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.052+0000 7f52b2ffd640 1 -- 172.21.15.94:0/3294403191 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52ac018070 con 0x7f52bc0c69a0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.052+0000 7f52b2ffd640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.052+0000 7f52b2ffd640 1 -- 172.21.15.94:0/3294403191 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f52ac089ff0 con 0x7f52bc0c69a0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.052+0000 7f52c2029640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.053+0000 7f52c2029640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f52a006b4d0 0x7f52a006d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f52b8035000 tx=0x7f52b8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.057+0000 7f52b2ffd640 1 -- 172.21.15.94:0/3294403191 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52bc158640 con 0x7f52bc0c69a0 2024-03-06T07:24:48.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.229+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f52bc152c70 con 0x7f52bc0c69a0 2024-03-06T07:24:48.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.232+0000 7f52b2ffd640 1 -- 172.21.15.94:0/3294403191 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7f52ac058f70 con 0x7f52bc0c69a0 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f52a006b4d0 msgr2=0x7f52a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f52a006b4d0 0x7f52a006d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f52b8035000 tx=0x7f52b8030000 comp rx=0 tx=0).stop 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc0c69a0 msgr2=0x7f52bc06fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc0c69a0 0x7f52bc06fb90 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c7f0 tx=0x7f52ac00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 shutdown_connections 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52bc158bd0 0x7f52bc0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52bc0c69a0 0x7f52bc06fb90 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 --2- 172.21.15.94:0/3294403191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52bc06d840 0x7f52bc1447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 >> 172.21.15.94:0/3294403191 conn(0x7f52bc0c5560 msgr2=0x7f52bc06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 shutdown_connections 2024-03-06T07:24:48.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.234+0000 7f52c428d640 1 -- 172.21.15.94:0/3294403191 wait complete. 2024-03-06T07:24:48.246 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.a mds_join_fs 2024-03-06T07:24:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.347+0000 7f08aee0e640 1 Processor -- start 2024-03-06T07:24:48.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.347+0000 7f08aee0e640 1 -- start start 2024-03-06T07:24:48.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08aee0e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f08a81538c0 0x7f08a81494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08aee0e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f08a80a97c0 0x7f08a81499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08aee0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a8154330 con 0x7f08a80ab310 2024-03-06T07:24:48.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08aee0e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a80c46a0 con 0x7f08a81538c0 2024-03-06T07:24:48.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08aee0e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a8153dc0 con 0x7f08a80a97c0 2024-03-06T07:24:48.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08a7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f08a81538c0 0x7f08a81494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08acbaa640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f08a80ab310 0x7f08a80ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45874/0 (socket says 172.21.15.94:45874) 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08acbaa640 1 -- 172.21.15.94:0/768108122 learned_addr learned my addr 172.21.15.94:0/768108122 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08ad3ab640 1 --2- 172.21.15.94:0/768108122 >> v2:172.21.15.145:3300/0 conn(0x7f08a80a97c0 0x7f08a81499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08a7fff640 1 -- 172.21.15.94:0/768108122 >> v2:172.21.15.145:3300/0 conn(0x7f08a80a97c0 msgr2=0x7f08a81499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08a7fff640 1 --2- 172.21.15.94:0/768108122 >> v2:172.21.15.145:3300/0 conn(0x7f08a80a97c0 0x7f08a81499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08a7fff640 1 -- 172.21.15.94:0/768108122 >> v1:172.21.15.94:6789/0 conn(0x7f08a80ab310 legacy=0x7f08a80ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08a77fe640 1 -- 172.21.15.94:0/768108122 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742671796 0 0) 0x7f08a8154330 con 0x7f08a80ab310 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08a7fff640 1 -- 172.21.15.94:0/768108122 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a80c5d30 con 0x7f08a81538c0 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.348+0000 7f08ad3ab640 1 --2- 172.21.15.94:0/768108122 >> v2:172.21.15.145:3300/0 conn(0x7f08a80a97c0 0x7f08a81499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08a7fff640 1 --2- 172.21.15.94:0/768108122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a81538c0 0x7f08a81494a0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f08a814d490 tx=0x7f089402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de7620b2c0fb128 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08a77fe640 1 -- 172.21.15.94:0/768108122 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08940315a0 con 0x7f08a81538c0 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08a77fe640 1 -- 172.21.15.94:0/768108122 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0894037070 con 0x7f08a81538c0 2024-03-06T07:24:48.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08a77fe640 1 -- 172.21.15.94:0/768108122 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894031960 con 0x7f08a81538c0 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 -- 172.21.15.94:0/768108122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a81538c0 msgr2=0x7f08a81494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 --2- 172.21.15.94:0/768108122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a81538c0 0x7f08a81494a0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f08a814d490 tx=0x7f089402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 -- 172.21.15.94:0/768108122 shutdown_connections 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 --2- 172.21.15.94:0/768108122 >> v2:172.21.15.145:3300/0 conn(0x7f08a80a97c0 0x7f08a81499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 --2- 172.21.15.94:0/768108122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a81538c0 0x7f08a81494a0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 -- 172.21.15.94:0/768108122 >> 172.21.15.94:0/768108122 conn(0x7f08a80c6710 msgr2=0x7f08a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 -- 172.21.15.94:0/768108122 shutdown_connections 2024-03-06T07:24:48.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.349+0000 7f08aee0e640 1 -- 172.21.15.94:0/768108122 wait complete. 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.350+0000 7f08aee0e640 1 Processor -- start 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.350+0000 7f08aee0e640 1 -- start start 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08aee0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a80a97c0 0x7f08a8156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08aee0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08a80ab310 0x7f08a8149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08aee0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08a81538c0 0x7f08a814a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08aee0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80c4b30 con 0x7f08a80a97c0 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08aee0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80c62f0 con 0x7f08a80ab310 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08aee0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80c4cb0 con 0x7f08a81538c0 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08acbaa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a80a97c0 0x7f08a8156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08acbaa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a80a97c0 0x7f08a8156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39388/0 (socket says 172.21.15.94:39388) 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08acbaa640 1 -- 172.21.15.94:0/3731501782 learned_addr learned my addr 172.21.15.94:0/3731501782 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.351+0000 7f08a7fff640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08a80ab310 0x7f08a8149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08ad3ab640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08a81538c0 0x7f08a814a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08acbaa640 1 -- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08a81538c0 msgr2=0x7f08a814a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08acbaa640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08a81538c0 0x7f08a814a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08acbaa640 1 -- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08a80ab310 msgr2=0x7f08a8149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08acbaa640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08a80ab310 0x7f08a8149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08acbaa640 1 -- 172.21.15.94:0/3731501782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a814f870 con 0x7f08a80a97c0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08a7fff640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08a80ab310 0x7f08a8149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08acbaa640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a80a97c0 0x7f08a8156960 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c4d0 tx=0x7f089c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08a57fa640 1 -- 172.21.15.94:0/3731501782 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c018070 con 0x7f08a80a97c0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08a57fa640 1 -- 172.21.15.94:0/3731501782 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c007500 con 0x7f08a80a97c0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a81e8b80 con 0x7f08a80a97c0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08a57fa640 1 -- 172.21.15.94:0/3731501782 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c004a20 con 0x7f08a80a97c0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.352+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a81e9140 con 0x7f08a80a97c0 2024-03-06T07:24:48.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.353+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a804f330 con 0x7f08a80a97c0 2024-03-06T07:24:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.353+0000 7f08a57fa640 1 -- 172.21.15.94:0/3731501782 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f089c01b070 con 0x7f08a80a97c0 2024-03-06T07:24:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.354+0000 7f08a57fa640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.354+0000 7f08a57fa640 1 -- 172.21.15.94:0/3731501782 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f089c0894d0 con 0x7f08a80a97c0 2024-03-06T07:24:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.354+0000 7f08a7fff640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.355+0000 7f08a7fff640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f088c06b4d0 0x7f088c06d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0894036000 tx=0x7f089403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.358+0000 7f08a57fa640 1 -- 172.21.15.94:0/3731501782 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f089c0583c0 con 0x7f08a80a97c0 2024-03-06T07:24:48.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.528+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"} v 0) v1 -- 0x7f08a80703c0 con 0x7f08a80a97c0 2024-03-06T07:24:48.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.531+0000 7f08a57fa640 1 -- 172.21.15.94:0/3731501782 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.a", "name": "mds_join_fs"}]=0 v55) v1 ==== 96+0+0 (secure 0 0 0) 0x7f089c057cc0 con 0x7f08a80a97c0 2024-03-06T07:24:48.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f088c06b4d0 msgr2=0x7f088c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f088c06b4d0 0x7f088c06d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0894036000 tx=0x7f089403d040 comp rx=0 tx=0).stop 2024-03-06T07:24:48.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a80a97c0 msgr2=0x7f08a8156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a80a97c0 0x7f08a8156960 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f089c00c4d0 tx=0x7f089c00c990 comp rx=0 tx=0).stop 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 shutdown_connections 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f088c06b4d0 0x7f088c06d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08a81538c0 0x7f08a814a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08a80ab310 0x7f08a8149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 --2- 172.21.15.94:0/3731501782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f08a80a97c0 0x7f08a8156960 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 >> 172.21.15.94:0/3731501782 conn(0x7f08a80c6710 msgr2=0x7f08a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 shutdown_connections 2024-03-06T07:24:48.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.533+0000 7f08aee0e640 1 -- 172.21.15.94:0/3731501782 wait complete. 2024-03-06T07:24:48.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-03-06T07:24:48.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.646+0000 7f9c2d8df640 1 Processor -- start 2024-03-06T07:24:48.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.646+0000 7f9c2d8df640 1 -- start start 2024-03-06T07:24:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c2d8df640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9c28152730 0x7f9c280c3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c2d8df640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9c280c6710 0x7f9c281554c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c2d8df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c28152b10 con 0x7f9c280c3750 2024-03-06T07:24:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c2d8df640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c28156110 con 0x7f9c28152730 2024-03-06T07:24:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c2d8df640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c28158ad0 con 0x7f9c280c6710 2024-03-06T07:24:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c26ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9c28152730 0x7f9c280c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c26ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9c28152730 0x7f9c280c3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42992/0 (socket says 172.21.15.94:42992) 2024-03-06T07:24:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c26ffd640 1 -- 172.21.15.94:0/477835962 learned_addr learned my addr 172.21.15.94:0/477835962 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c267fc640 1 --2- 172.21.15.94:0/477835962 >> v2:172.21.15.145:3300/0 conn(0x7f9c280c6710 0x7f9c281554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c26ffd640 1 -- 172.21.15.94:0/477835962 >> v2:172.21.15.145:3300/0 conn(0x7f9c280c6710 msgr2=0x7f9c281554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c26ffd640 1 --2- 172.21.15.94:0/477835962 >> v2:172.21.15.145:3300/0 conn(0x7f9c280c6710 0x7f9c281554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c26ffd640 1 -- 172.21.15.94:0/477835962 >> v1:172.21.15.145:6789/0 conn(0x7f9c280c3750 legacy=0x7f9c28155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.647+0000 7f9c26ffd640 1 -- 172.21.15.94:0/477835962 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c28158db0 con 0x7f9c28152730 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.648+0000 7f9c26ffd640 1 --2- 172.21.15.94:0/477835962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c28152730 0x7f9c280c3210 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9c280ab3a0 tx=0x7f9c1802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b9c6aaf860ac35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.648+0000 7f9c25ffb640 1 -- 172.21.15.94:0/477835962 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1802c9f0 con 0x7f9c28152730 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.648+0000 7f9c25ffb640 1 -- 172.21.15.94:0/477835962 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1802cb90 con 0x7f9c28152730 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.648+0000 7f9c25ffb640 1 -- 172.21.15.94:0/477835962 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c180343f0 con 0x7f9c28152730 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 -- 172.21.15.94:0/477835962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c28152730 msgr2=0x7f9c280c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/477835962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c28152730 0x7f9c280c3210 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9c280ab3a0 tx=0x7f9c1802be50 comp rx=0 tx=0).stop 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 -- 172.21.15.94:0/477835962 shutdown_connections 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/477835962 >> v2:172.21.15.145:3300/0 conn(0x7f9c280c6710 0x7f9c281554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/477835962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c28152730 0x7f9c280c3210 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 -- 172.21.15.94:0/477835962 >> 172.21.15.94:0/477835962 conn(0x7f9c280aa5c0 msgr2=0x7f9c280aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 -- 172.21.15.94:0/477835962 shutdown_connections 2024-03-06T07:24:48.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.649+0000 7f9c2d8df640 1 -- 172.21.15.94:0/477835962 wait complete. 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.650+0000 7f9c2d8df640 1 Processor -- start 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.650+0000 7f9c2d8df640 1 -- start start 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c2d8df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c280c3750 0x7f9c2806fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c2d8df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c280c6710 0x7f9c2814f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c2d8df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c28152730 0x7f9c280708a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c2d8df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c281586f0 con 0x7f9c280c6710 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c2d8df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c28159370 con 0x7f9c28152730 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c2d8df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c28158870 con 0x7f9c280c3750 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c280c6710 0x7f9c2814f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c280c6710 0x7f9c2814f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43002/0 (socket says 172.21.15.94:43002) 2024-03-06T07:24:48.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 -- 172.21.15.94:0/178786382 learned_addr learned my addr 172.21.15.94:0/178786382 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c277fe640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c28152730 0x7f9c280708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c26ffd640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c280c3750 0x7f9c2806fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 -- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c280c3750 msgr2=0x7f9c2806fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c280c3750 0x7f9c2806fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 -- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c28152730 msgr2=0x7f9c280708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c28152730 0x7f9c280708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 -- 172.21.15.94:0/178786382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2806eab0 con 0x7f9c280c6710 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c277fe640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c28152730 0x7f9c280708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.651+0000 7f9c267fc640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c280c6710 0x7f9c2814f9c0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00d7d0 tx=0x7f9c1c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.652+0000 7f9c0ffff640 1 -- 172.21.15.94:0/178786382 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c00b6c0 con 0x7f9c280c6710 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.652+0000 7f9c0ffff640 1 -- 172.21.15.94:0/178786382 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c019070 con 0x7f9c280c6710 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.652+0000 7f9c0ffff640 1 -- 172.21.15.94:0/178786382 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c00ba80 con 0x7f9c280c6710 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.652+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2806ed90 con 0x7f9c280c6710 2024-03-06T07:24:48.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.652+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2806f350 con 0x7f9c280c6710 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.653+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2804f330 con 0x7f9c280c6710 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.653+0000 7f9c0ffff640 1 -- 172.21.15.94:0/178786382 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c1c00bc20 con 0x7f9c280c6710 2024-03-06T07:24:48.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.653+0000 7f9c0ffff640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c1406b4d0 0x7f9c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.654+0000 7f9c0ffff640 1 -- 172.21.15.94:0/178786382 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f9c1c08a660 con 0x7f9c280c6710 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.654+0000 7f9c26ffd640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c1406b4d0 0x7f9c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.654+0000 7f9c26ffd640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c1406b4d0 0x7f9c1406d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9c18035000 tx=0x7f9c18030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.658+0000 7f9c0ffff640 1 -- 172.21.15.94:0/178786382 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c1c0594c0 con 0x7f9c280c6710 2024-03-06T07:24:48.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.828+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f9c28155710 con 0x7f9c280c6710 2024-03-06T07:24:48.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.831+0000 7f9c0ffff640 1 -- 172.21.15.94:0/178786382 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v56) v1 ==== 96+0+0 (secure 0 0 0) 0x7f9c1c058dc0 con 0x7f9c280c6710 2024-03-06T07:24:48.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c1406b4d0 msgr2=0x7f9c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c1406b4d0 0x7f9c1406d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9c18035000 tx=0x7f9c18030000 comp rx=0 tx=0).stop 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c280c6710 msgr2=0x7f9c2814f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c280c6710 0x7f9c2814f9c0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00d7d0 tx=0x7f9c1c00dc90 comp rx=0 tx=0).stop 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 shutdown_connections 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9c1406b4d0 0x7f9c1406d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c28152730 0x7f9c280708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c280c6710 0x7f9c2814f9c0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 --2- 172.21.15.94:0/178786382 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c280c3750 0x7f9c2806fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.833+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 >> 172.21.15.94:0/178786382 conn(0x7f9c280aa5c0 msgr2=0x7f9c280c5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.834+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 shutdown_connections 2024-03-06T07:24:48.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.834+0000 7f9c2d8df640 1 -- 172.21.15.94:0/178786382 wait complete. 2024-03-06T07:24:48.846 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-03-06T07:24:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.945+0000 7fe83759e640 1 Processor -- start 2024-03-06T07:24:48.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.945+0000 7fe83759e640 1 -- start start 2024-03-06T07:24:48.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83759e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe8380b0000 0x7fe8380c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83759e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe8380c51f0 0x7fe8380c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83759e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe838155410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83759e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8381594b0 con 0x7fe8380b0000 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83759e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8381586f0 con 0x7fe8380c51f0 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83759e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381589d0 con 0x7fe838154f80 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe8380b0000 0x7fe8380c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe8380b0000 0x7fe8380c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43008/0 (socket says 172.21.15.94:43008) 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 -- 172.21.15.94:0/3646499394 learned_addr learned my addr 172.21.15.94:0/3646499394 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe82ffff640 1 --2- 172.21.15.94:0/3646499394 >> v2:172.21.15.145:3300/0 conn(0x7fe8380c51f0 0x7fe8380c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe836d9d640 1 --2- 172.21.15.94:0/3646499394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe838155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 -- 172.21.15.94:0/3646499394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 msgr2=0x7fe838155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 --2- 172.21.15.94:0/3646499394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe838155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 -- 172.21.15.94:0/3646499394 >> v2:172.21.15.145:3300/0 conn(0x7fe8380c51f0 msgr2=0x7fe8380c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 --2- 172.21.15.94:0/3646499394 >> v2:172.21.15.145:3300/0 conn(0x7fe8380c51f0 0x7fe8380c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe83659c640 1 -- 172.21.15.94:0/3646499394 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838158cb0 con 0x7fe8380b0000 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.946+0000 7fe836d9d640 1 --2- 172.21.15.94:0/3646499394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe838155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.947+0000 7fe83659c640 1 --2- 172.21.15.94:0/3646499394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380b0000 0x7fe8380c8770 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fe83806c290 tx=0x7fe82802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f00adf5af4ed23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.947+0000 7fe835d9b640 1 -- 172.21.15.94:0/3646499394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82802e070 con 0x7fe8380b0000 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.947+0000 7fe835d9b640 1 -- 172.21.15.94:0/3646499394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe828037070 con 0x7fe8380b0000 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.947+0000 7fe835d9b640 1 -- 172.21.15.94:0/3646499394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe828031a00 con 0x7fe8380b0000 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 -- 172.21.15.94:0/3646499394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380b0000 msgr2=0x7fe8380c8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 --2- 172.21.15.94:0/3646499394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380b0000 0x7fe8380c8770 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fe83806c290 tx=0x7fe82802b880 comp rx=0 tx=0).stop 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 -- 172.21.15.94:0/3646499394 shutdown_connections 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 --2- 172.21.15.94:0/3646499394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe838155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 --2- 172.21.15.94:0/3646499394 >> v2:172.21.15.145:3300/0 conn(0x7fe8380c51f0 0x7fe8380c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 --2- 172.21.15.94:0/3646499394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380b0000 0x7fe8380c8770 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 -- 172.21.15.94:0/3646499394 >> 172.21.15.94:0/3646499394 conn(0x7fe8380728d0 msgr2=0x7fe838072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 -- 172.21.15.94:0/3646499394 shutdown_connections 2024-03-06T07:24:48.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.948+0000 7fe83759e640 1 -- 172.21.15.94:0/3646499394 wait complete. 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.949+0000 7fe83759e640 1 Processor -- start 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.949+0000 7fe83759e640 1 -- start start 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83759e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8380b0000 0x7fe838148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380c51f0 0x7fe8381ebcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83759e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe8381ec210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83759e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe838155950 con 0x7fe8380c51f0 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83759e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8380be070 con 0x7fe8380b0000 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83759e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe838155ad0 con 0x7fe838154f80 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380c51f0 0x7fe8381ebcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380c51f0 0x7fe8381ebcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43014/0 (socket says 172.21.15.94:43014) 2024-03-06T07:24:48.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 -- 172.21.15.94:0/329173991 learned_addr learned my addr 172.21.15.94:0/329173991 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83659c640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8380b0000 0x7fe838148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe836d9d640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe8381ec210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 -- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 msgr2=0x7fe8381ec210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe8381ec210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 -- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8380b0000 msgr2=0x7fe838148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8380b0000 0x7fe838148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 -- 172.21.15.94:0/329173991 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8381e8310 con 0x7fe8380c51f0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe836d9d640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe8381ec210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82ffff640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380c51f0 0x7fe8381ebcd0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fe82000d830 tx=0x7fe82000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82effd640 1 -- 172.21.15.94:0/329173991 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820018030 con 0x7fe8380c51f0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82effd640 1 -- 172.21.15.94:0/329173991 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82000be10 con 0x7fe8380c51f0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe82effd640 1 -- 172.21.15.94:0/329173991 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200098b0 con 0x7fe8380c51f0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.950+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8380beee0 con 0x7fe8380c51f0 2024-03-06T07:24:48.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.951+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8380bf4a0 con 0x7fe8380c51f0 2024-03-06T07:24:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.952+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe838069430 con 0x7fe8380c51f0 2024-03-06T07:24:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.952+0000 7fe82effd640 1 -- 172.21.15.94:0/329173991 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe82001b070 con 0x7fe8380c51f0 2024-03-06T07:24:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.952+0000 7fe82effd640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81806b4d0 0x7fe81806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.953+0000 7fe82effd640 1 -- 172.21.15.94:0/329173991 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fe82008a3a0 con 0x7fe8380c51f0 2024-03-06T07:24:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.953+0000 7fe83659c640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81806b4d0 0x7fe81806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.953+0000 7fe83659c640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81806b4d0 0x7fe81806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe828036000 tx=0x7fe82803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:48.957+0000 7fe82effd640 1 -- 172.21.15.94:0/329173991 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe820059290 con 0x7fe8380c51f0 2024-03-06T07:24:49.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.192+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7fe83806cd80 con 0x7fe8380c51f0 2024-03-06T07:24:49.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.927+0000 7fe82effd640 1 -- 172.21.15.94:0/329173991 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v381) v1 ==== 146+0+0 (secure 0 0 0) 0x7fe820058b90 con 0x7fe8380c51f0 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81806b4d0 msgr2=0x7fe81806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81806b4d0 0x7fe81806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe828036000 tx=0x7fe82803d040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380c51f0 msgr2=0x7fe8381ebcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380c51f0 0x7fe8381ebcd0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fe82000d830 tx=0x7fe82000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 shutdown_connections 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81806b4d0 0x7fe81806d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe838154f80 0x7fe8381ec210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe8380c51f0 0x7fe8381ebcd0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 --2- 172.21.15.94:0/329173991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8380b0000 0x7fe838148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 >> 172.21.15.94:0/329173991 conn(0x7fe8380728d0 msgr2=0x7fe838072cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 shutdown_connections 2024-03-06T07:24:49.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:49.930+0000 7fe83759e640 1 -- 172.21.15.94:0/329173991 wait complete. 2024-03-06T07:24:49.944 INFO:tasks.cephfs_test_runner:That a vanilla standby is preferred over others with mds_join_fs set to another fs. ... ok 2024-03-06T07:24:49.945 INFO:tasks.cephfs_test_runner:Starting test: test_down (tasks.cephfs.test_failover.TestClusterResize) That down setting toggles and sets max_mds appropriately. 2024-03-06T07:24:49.945 INFO:tasks.cephfs_test_runner:test_down (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:24:49.945 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.047+0000 7f2923fff640 1 Processor -- start 2024-03-06T07:24:50.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.047+0000 7f2923fff640 1 -- start start 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.048+0000 7f2923fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29240729a0 0x7f29240c15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.048+0000 7f2923fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2924158520 0x7f2924158990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.048+0000 7f2923fff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2924155140 con 0x7f292406e150 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.048+0000 7f2923fff640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29240c1ae0 con 0x7f29240729a0 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.048+0000 7f2923fff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29240c1d10 con 0x7f2924158520 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f29237fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2924158520 0x7f2924158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f2922ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29240729a0 0x7f29240c15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f29237fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2924158520 0x7f2924158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:47126/0 (socket says 172.21.15.94:47126) 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f291bfff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f292406e150 0x7f29240c1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37480/0 (socket says 172.21.15.94:37480) 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f2922ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29240729a0 0x7f29240c15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:35888/0 (socket says 172.21.15.94:35888) 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f29237fe640 1 -- 172.21.15.94:0/873451226 learned_addr learned my addr 172.21.15.94:0/873451226 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f29237fe640 1 -- 172.21.15.94:0/873451226 >> v2:172.21.15.145:3300/0 conn(0x7f29240729a0 msgr2=0x7f29240c15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f29237fe640 1 --2- 172.21.15.94:0/873451226 >> v2:172.21.15.145:3300/0 conn(0x7f29240729a0 0x7f29240c15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f29237fe640 1 -- 172.21.15.94:0/873451226 >> v1:172.21.15.145:6789/0 conn(0x7f292406e150 legacy=0x7f29240c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f29237fe640 1 -- 172.21.15.94:0/873451226 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2924155420 con 0x7f2924158520 2024-03-06T07:24:50.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.049+0000 7f2922ffd640 1 --2- 172.21.15.94:0/873451226 >> v2:172.21.15.145:3300/0 conn(0x7f29240729a0 0x7f29240c15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:50.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.050+0000 7f29237fe640 1 --2- 172.21.15.94:0/873451226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2924158520 0x7f2924158990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f291c009e50 tx=0x7f291c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc74820e983e13d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.050+0000 7f29227fc640 1 -- 172.21.15.94:0/873451226 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c02cb00 con 0x7f2924158520 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.050+0000 7f29227fc640 1 -- 172.21.15.94:0/873451226 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291c02cca0 con 0x7f2924158520 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.050+0000 7f29227fc640 1 -- 172.21.15.94:0/873451226 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c034610 con 0x7f2924158520 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 -- 172.21.15.94:0/873451226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2924158520 msgr2=0x7f2924158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 --2- 172.21.15.94:0/873451226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2924158520 0x7f2924158990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f291c009e50 tx=0x7f291c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 -- 172.21.15.94:0/873451226 shutdown_connections 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 --2- 172.21.15.94:0/873451226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2924158520 0x7f2924158990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 --2- 172.21.15.94:0/873451226 >> v2:172.21.15.145:3300/0 conn(0x7f29240729a0 0x7f29240c15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 -- 172.21.15.94:0/873451226 >> 172.21.15.94:0/873451226 conn(0x7f292406c0d0 msgr2=0x7f292406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 -- 172.21.15.94:0/873451226 shutdown_connections 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.051+0000 7f2923fff640 1 -- 172.21.15.94:0/873451226 wait complete. 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.052+0000 7f2923fff640 1 Processor -- start 2024-03-06T07:24:50.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.052+0000 7f2923fff640 1 -- start start 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.052+0000 7f2923fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f292406e150 0x7f2924154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.052+0000 7f2923fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29240729a0 0x7f292414f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.052+0000 7f2923fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2924158520 0x7f292414fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2923fff640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29240c26c0 con 0x7f29240729a0 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2923fff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2924155a70 con 0x7f2924158520 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2923fff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29240c2840 con 0x7f292406e150 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29240729a0 0x7f292414f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29240729a0 0x7f292414f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43016/0 (socket says 172.21.15.94:43016) 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 -- 172.21.15.94:0/473536468 learned_addr learned my addr 172.21.15.94:0/473536468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2922ffd640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f292406e150 0x7f2924154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f29237fe640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2924158520 0x7f292414fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 -- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f292406e150 msgr2=0x7f2924154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f292406e150 0x7f2924154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 -- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2924158520 msgr2=0x7f292414fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2924158520 0x7f292414fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 -- 172.21.15.94:0/473536468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29241e0640 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f291bfff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29240729a0 0x7f292414f990 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f290c00c3f0 tx=0x7f290c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2920ff9640 1 -- 172.21.15.94:0/473536468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c0074e0 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2920ff9640 1 -- 172.21.15.94:0/473536468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c007680 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29241480b0 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2920ff9640 1 -- 172.21.15.94:0/473536468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c004a00 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.053+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29241546d0 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.054+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29241e0810 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.055+0000 7f2920ff9640 1 -- 172.21.15.94:0/473536468 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f290c01a020 con 0x7f29240729a0 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.055+0000 7f2920ff9640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f290406b4d0 0x7f290406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.055+0000 7f2920ff9640 1 -- 172.21.15.94:0/473536468 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f290c0896f0 con 0x7f29240729a0 2024-03-06T07:24:50.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.055+0000 7f2922ffd640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f290406b4d0 0x7f290406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.056+0000 7f2922ffd640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f290406b4d0 0x7f290406d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2924154df0 tx=0x7f2914009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.059+0000 7f2920ff9640 1 -- 172.21.15.94:0/473536468 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f290c058020 con 0x7f29240729a0 2024-03-06T07:24:50.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.231+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2924069430 con 0x7f29240729a0 2024-03-06T07:24:50.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.232+0000 7f2920ff9640 1 -- 172.21.15.94:0/473536468 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+13695 (secure 0 0 0) 0x7f290c05c0c0 con 0x7f29240729a0 2024-03-06T07:24:50.234 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:50.234 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":161,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:47.916571+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/1521427660":"2024-03-07T07:24:44.907216+0000","172.21.15.145:6834/1521427660":"2024-03-07T07:24:44.907216+0000","172.21.15.94:6834/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.94:6832/1222126706":"2024-03-07T07:24:39.753849+0000","172.21.15.145:6835/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.145:6837/486493869":"2024-03-07T07:24:38.716429+0000","172.21.15.94:6835/1613596341":"2024-03-07T07:24:23.693503+0000","172.21.15.94:6833/1613596341":"2024-03-07T07:24:23.693503+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f290406b4d0 msgr2=0x7f290406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f290406b4d0 0x7f290406d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2924154df0 tx=0x7f2914009190 comp rx=0 tx=0).stop 2024-03-06T07:24:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29240729a0 msgr2=0x7f292414f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29240729a0 0x7f292414f990 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f290c00c3f0 tx=0x7f290c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 shutdown_connections 2024-03-06T07:24:50.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f290406b4d0 0x7f290406d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2924158520 0x7f292414fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29240729a0 0x7f292414f990 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 --2- 172.21.15.94:0/473536468 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f292406e150 0x7f2924154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 >> 172.21.15.94:0/473536468 conn(0x7f292406c0d0 msgr2=0x7f2924157d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 shutdown_connections 2024-03-06T07:24:50.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.234+0000 7f2923fff640 1 -- 172.21.15.94:0/473536468 wait complete. 2024-03-06T07:24:50.247 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:50.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:50.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.348+0000 7f6762379640 1 Processor -- start 2024-03-06T07:24:50.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.348+0000 7f6762379640 1 -- start start 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f6762379640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 0x7f675c0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f6762379640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f675c06c900 0x7f675c0a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f6762379640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675c0c3ef0 con 0x7f675c0ab170 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f6762379640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675c1556d0 con 0x7f675c06c900 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f6762379640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675c1559b0 con 0x7f675c0c5990 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f675c0ab170 0x7f675c06c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52382/0 (socket says 172.21.15.94:52382) 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675b7fe640 1 -- 172.21.15.94:0/48889484 learned_addr learned my addr 172.21.15.94:0/48889484 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160625453 0 0) 0x7f675c0c3ef0 con 0x7f675c0ab170 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6744003520 con 0x7f675c0ab170 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675bfff640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 0x7f675c0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f6760916640 1 --2- 172.21.15.94:0/48889484 >> v2:172.21.15.145:3300/0 conn(0x7f675c06c900 0x7f675c0a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1585402560 0 0) 0x7f6744003520 con 0x7f675c0ab170 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 msgr2=0x7f675c0c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 0x7f675c0c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 >> v2:172.21.15.145:3300/0 conn(0x7f675c06c900 msgr2=0x7f675c0a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 --2- 172.21.15.94:0/48889484 >> v2:172.21.15.145:3300/0 conn(0x7f675c06c900 0x7f675c0a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c155c90 con 0x7f675c0ab170 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f6750001670 con 0x7f675c0ab170 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 >> v1:172.21.15.94:6789/0 conn(0x7f675c0ab170 legacy=0x7f675c06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67440039b0 0x7f6744003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 0x7f6744004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6744004d40 0x7f67440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c1556d0 con 0x7f67440039b0 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c1559b0 con 0x7f6744004d40 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674400ba70 con 0x7f675c0c5990 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.349+0000 7f675bfff640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67440039b0 0x7f6744003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f675b7fe640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 0x7f6744004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f6760916640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6744004d40 0x7f67440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f675bfff640 1 -- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 msgr2=0x7f6744004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f675bfff640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c0c5990 0x7f6744004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f675bfff640 1 -- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6744004d40 msgr2=0x7f67440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f675bfff640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6744004d40 0x7f67440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f675bfff640 1 -- 172.21.15.94:0/48889484 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744003520 con 0x7f67440039b0 2024-03-06T07:24:50.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.350+0000 7f6760916640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6744004d40 0x7f67440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f675bfff640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67440039b0 0x7f6744003dd0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f674c00ba60 tx=0x7f6744005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0adb8f4f7df983d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c036020 con 0x7f67440039b0 2024-03-06T07:24:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c032630 con 0x7f67440039b0 2024-03-06T07:24:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f675affd640 1 -- 172.21.15.94:0/48889484 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c00b600 con 0x7f67440039b0 2024-03-06T07:24:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f6762379640 1 -- 172.21.15.94:0/48889484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67440039b0 msgr2=0x7f6744003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f6762379640 1 --2- 172.21.15.94:0/48889484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67440039b0 0x7f6744003dd0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f674c00ba60 tx=0x7f6744005830 comp rx=0 tx=0).stop 2024-03-06T07:24:50.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f6760916640 1 -- 172.21.15.94:0/48889484 reap_dead start 2024-03-06T07:24:50.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f6762379640 1 -- 172.21.15.94:0/48889484 shutdown_connections 2024-03-06T07:24:50.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f6762379640 1 -- 172.21.15.94:0/48889484 >> 172.21.15.94:0/48889484 conn(0x7f675c0c6710 msgr2=0x7f675c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f6762379640 1 -- 172.21.15.94:0/48889484 shutdown_connections 2024-03-06T07:24:50.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.351+0000 7f6762379640 1 -- 172.21.15.94:0/48889484 wait complete. 2024-03-06T07:24:50.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.352+0000 7f6762379640 1 Processor -- start 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6762379640 1 -- start start 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6762379640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67440039b0 0x7f675c1e7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6762379640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f675c148040 0x7f675c14dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6762379640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c1e9070 0x7f675c14ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6762379640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c156610 con 0x7f675c148040 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6762379640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c156490 con 0x7f67440039b0 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6762379640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675c0c3af0 con 0x7f675c1e9070 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f675b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f675c148040 0x7f675c14dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f675b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f675c148040 0x7f675c14dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43040/0 (socket says 172.21.15.94:43040) 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f675b7fe640 1 -- 172.21.15.94:0/3717926440 learned_addr learned my addr 172.21.15.94:0/3717926440 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:50.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f675bfff640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67440039b0 0x7f675c1e7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.353+0000 7f6760916640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c1e9070 0x7f675c14ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f675b7fe640 1 -- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c1e9070 msgr2=0x7f675c14ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f675b7fe640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c1e9070 0x7f675c14ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f675b7fe640 1 -- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67440039b0 msgr2=0x7f675c1e7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f675b7fe640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67440039b0 0x7f675c1e7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f675b7fe640 1 -- 172.21.15.94:0/3717926440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c0c1ed0 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f6760916640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c1e9070 0x7f675c14ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f675b7fe640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f675c148040 0x7f675c14dfd0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f6750007370 tx=0x7f6750007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f6758ff9640 1 -- 172.21.15.94:0/3717926440 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675000f9c0 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f6758ff9640 1 -- 172.21.15.94:0/3717926440 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6750018070 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c14e510 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f6758ff9640 1 -- 172.21.15.94:0/3717926440 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675000fda0 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.354+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c14d210 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.355+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c04f330 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.355+0000 7f6758ff9640 1 -- 172.21.15.94:0/3717926440 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f675001d460 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.356+0000 7f6758ff9640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674806b500 0x7f674806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.356+0000 7f6758ff9640 1 -- 172.21.15.94:0/3717926440 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f675008a1b0 con 0x7f675c148040 2024-03-06T07:24:50.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.356+0000 7f675bfff640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674806b500 0x7f674806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.356+0000 7f675bfff640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674806b500 0x7f674806d9c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f674c00ba30 tx=0x7f674c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.360+0000 7f6758ff9640 1 -- 172.21.15.94:0/3717926440 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6750059130 con 0x7f675c148040 2024-03-06T07:24:50.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.532+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f675c0a9740 con 0x7f675c148040 2024-03-06T07:24:50.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.532+0000 7f6758ff9640 1 -- 172.21.15.94:0/3717926440 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v161) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6750058a30 con 0x7f675c148040 2024-03-06T07:24:50.535 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:24:50.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674806b500 msgr2=0x7f674806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674806b500 0x7f674806d9c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f674c00ba30 tx=0x7f674c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f675c148040 msgr2=0x7f675c14dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f675c148040 0x7f675c14dfd0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f6750007370 tx=0x7f6750007830 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 shutdown_connections 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f674806b500 0x7f674806d9c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f675c1e9070 0x7f675c14ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f675c148040 0x7f675c14dfd0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 --2- 172.21.15.94:0/3717926440 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67440039b0 0x7f675c1e7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 >> 172.21.15.94:0/3717926440 conn(0x7f675c0c6710 msgr2=0x7f675c06cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 shutdown_connections 2024-03-06T07:24:50.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.534+0000 7f6762379640 1 -- 172.21.15.94:0/3717926440 wait complete. 2024-03-06T07:24:50.547 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-03-06T07:24:50.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.645+0000 7f013ae88640 1 Processor -- start 2024-03-06T07:24:50.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.645+0000 7f013ae88640 1 -- start start 2024-03-06T07:24:50.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f013ae88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f01340aa180 0x7f013414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f013ae88640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013414f5d0 con 0x7f01341550c0 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f013ae88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0134155bb0 con 0x7f013414f140 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f013ae88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01340c4610 con 0x7f01340aa180 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f0138c24640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f01341550c0 0x7f01341554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52390/0 (socket says 172.21.15.94:52390) 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f0138c24640 1 -- 172.21.15.94:0/184905189 learned_addr learned my addr 172.21.15.94:0/184905189 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f0133fff640 1 --2- 172.21.15.94:0/184905189 >> v2:172.21.15.145:3300/0 conn(0x7f01340aa180 0x7f013414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992049128 0 0) 0x7f013414f5d0 con 0x7f01341550c0 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.646+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c004630 con 0x7f01341550c0 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93727462 0 0) 0x7f0134155bb0 con 0x7f013414f140 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013414f5d0 con 0x7f013414f140 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3292602157 0 0) 0x7f011c004630 con 0x7f01341550c0 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 >> v2:172.21.15.145:3300/0 conn(0x7f01340aa180 msgr2=0x7f013414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 --2- 172.21.15.94:0/184905189 >> v2:172.21.15.145:3300/0 conn(0x7f01340aa180 0x7f013414ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 >> v1:172.21.15.145:6789/0 conn(0x7f013414f140 legacy=0x7f01340c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340aa660 con 0x7f01341550c0 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f0124001670 con 0x7f01341550c0 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 >> v1:172.21.15.94:6789/0 conn(0x7f01341550c0 legacy=0x7f01341554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f011c0050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f011c005910 0x7f011c005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f011c006390 0x7f011c00cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c4610 con 0x7f011c004ca0 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c00d2d0 con 0x7f011c005910 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c00d4d0 con 0x7f011c006390 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0138c24640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f011c0050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0133fff640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f011c005910 0x7f011c005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0138c24640 1 -- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f011c006390 msgr2=0x7f011c00cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0139425640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f011c006390 0x7f011c00cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0138c24640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f011c006390 0x7f011c00cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0138c24640 1 -- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f011c005910 msgr2=0x7f011c005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0138c24640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f011c005910 0x7f011c005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0138c24640 1 -- 172.21.15.94:0/184905189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c004630 con 0x7f011c004ca0 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0139425640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f011c006390 0x7f011c00cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0133fff640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f011c005910 0x7f011c005d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.647+0000 7f0139425640 1 -- 172.21.15.94:0/184905189 reap_dead start 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.648+0000 7f0138c24640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f011c0050a0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f012400c0e0 tx=0x7f011c006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c04fedb87dd5286 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.648+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124007c00 con 0x7f011c004ca0 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.648+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0124030070 con 0x7f011c004ca0 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.648+0000 7f01337fe640 1 -- 172.21.15.94:0/184905189 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124005d10 con 0x7f011c004ca0 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.649+0000 7f013ae88640 1 -- 172.21.15.94:0/184905189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 msgr2=0x7f011c0050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.649+0000 7f013ae88640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f011c0050a0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f012400c0e0 tx=0x7f011c006940 comp rx=0 tx=0).stop 2024-03-06T07:24:50.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.649+0000 7f013ae88640 1 -- 172.21.15.94:0/184905189 shutdown_connections 2024-03-06T07:24:50.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.649+0000 7f013ae88640 1 --2- 172.21.15.94:0/184905189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f011c0050a0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.649+0000 7f013ae88640 1 -- 172.21.15.94:0/184905189 >> 172.21.15.94:0/184905189 conn(0x7f01340c6710 msgr2=0x7f01340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.649+0000 7f013ae88640 1 -- 172.21.15.94:0/184905189 shutdown_connections 2024-03-06T07:24:50.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.649+0000 7f013ae88640 1 -- 172.21.15.94:0/184905189 wait complete. 2024-03-06T07:24:50.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.650+0000 7f013ae88640 1 Processor -- start 2024-03-06T07:24:50.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.650+0000 7f013ae88640 1 -- start start 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.650+0000 7f013ae88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f013414bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.650+0000 7f013ae88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0134152620 0x7f01341498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.650+0000 7f013ae88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01341520f0 0x7f0134149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f013ae88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c1310 con 0x7f011c004ca0 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f013ae88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c4860 con 0x7f0134152620 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f013ae88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c1490 con 0x7f01341520f0 2024-03-06T07:24:50.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f013414bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f013414bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43064/0 (socket says 172.21.15.94:43064) 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 -- 172.21.15.94:0/896250040 learned_addr learned my addr 172.21.15.94:0/896250040 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 -- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01341520f0 msgr2=0x7f0134149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01341520f0 0x7f0134149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 -- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0134152620 msgr2=0x7f01341498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0134152620 0x7f01341498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 -- 172.21.15.94:0/896250040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340a8c00 con 0x7f011c004ca0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f0138c24640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f013414bdb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0124000c00 tx=0x7f01340ab0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f01317fa640 1 -- 172.21.15.94:0/896250040 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124007c00 con 0x7f011c004ca0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f01317fa640 1 -- 172.21.15.94:0/896250040 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012402c9d0 con 0x7f011c004ca0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f01317fa640 1 -- 172.21.15.94:0/896250040 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012402cd80 con 0x7f011c004ca0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0134072ba0 con 0x7f011c004ca0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.651+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0134153af0 con 0x7f011c004ca0 2024-03-06T07:24:50.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.652+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013404f330 con 0x7f011c004ca0 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.653+0000 7f01317fa640 1 -- 172.21.15.94:0/896250040 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0124007070 con 0x7f011c004ca0 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.653+0000 7f01317fa640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f012006b500 0x7f012006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.653+0000 7f01317fa640 1 -- 172.21.15.94:0/896250040 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f01240ab1a0 con 0x7f011c004ca0 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.653+0000 7f0133fff640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f012006b500 0x7f012006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.659 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.654+0000 7f0133fff640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f012006b500 0x7f012006d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f01340a6630 tx=0x7f0128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.657+0000 7f01317fa640 1 -- 172.21.15.94:0/896250040 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0124079ad0 con 0x7f011c004ca0 2024-03-06T07:24:50.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.894+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7f01340730c0 con 0x7f011c004ca0 2024-03-06T07:24:50.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.935+0000 7f01317fa640 1 -- 172.21.15.94:0/896250040 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v382) v1 ==== 136+0+0 (secure 0 0 0) 0x7f012407db70 con 0x7f011c004ca0 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.937+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f012006b500 msgr2=0x7f012006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.937+0000 7f013ae88640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f012006b500 0x7f012006d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f01340a6630 tx=0x7f0128008040 comp rx=0 tx=0).stop 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.937+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 msgr2=0x7f013414bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.937+0000 7f013ae88640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f013414bdb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0124000c00 tx=0x7f01340ab0c0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 shutdown_connections 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f012006b500 0x7f012006d9c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01341520f0 0x7f0134149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0134152620 0x7f01341498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 --2- 172.21.15.94:0/896250040 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f011c004ca0 0x7f013414bdb0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 >> 172.21.15.94:0/896250040 conn(0x7f01340c6710 msgr2=0x7f01340a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 shutdown_connections 2024-03-06T07:24:50.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:50.938+0000 7f013ae88640 1 -- 172.21.15.94:0/896250040 wait complete. 2024-03-06T07:24:50.950 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:51.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66bd763640 1 Processor -- start 2024-03-06T07:24:51.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66bd763640 1 -- start start 2024-03-06T07:24:51.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66bd763640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f66b81527c0 0x7f66b80c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66bd763640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b8152ba0 con 0x7f66b80c88c0 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66bd763640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b80c8dd0 con 0x7f66b8155e10 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66bd763640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b80c9390 con 0x7f66b81527c0 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66b6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f66b81527c0 0x7f66b80c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66b77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f66b80c88c0 0x7f66b815faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52404/0 (socket says 172.21.15.94:52404) 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.052+0000 7f66b77fe640 1 -- 172.21.15.94:0/2260334019 learned_addr learned my addr 172.21.15.94:0/2260334019 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.053+0000 7f66b6ffd640 1 -- 172.21.15.94:0/2260334019 >> v1:172.21.15.145:6789/0 conn(0x7f66b8155e10 legacy=0x7f66b806c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.053+0000 7f66b6ffd640 1 -- 172.21.15.94:0/2260334019 >> v1:172.21.15.94:6789/0 conn(0x7f66b80c88c0 legacy=0x7f66b815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.053+0000 7f66b6ffd640 1 -- 172.21.15.94:0/2260334019 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b80c90b0 con 0x7f66b81527c0 2024-03-06T07:24:51.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.053+0000 7f66b5ffb640 1 -- 172.21.15.94:0/2260334019 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135476550 0 0) 0x7f66b0001110 con 0x7f66b80c88c0 2024-03-06T07:24:51.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.053+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/2260334019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b81527c0 0x7f66b80c73d0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009d90 tx=0x7f66a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622790e21e953441 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66b5ffb640 1 -- 172.21.15.94:0/2260334019 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8039070 con 0x7f66b81527c0 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66b5ffb640 1 -- 172.21.15.94:0/2260334019 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a802cc60 con 0x7f66b81527c0 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66b5ffb640 1 -- 172.21.15.94:0/2260334019 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8034660 con 0x7f66b81527c0 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66bd763640 1 -- 172.21.15.94:0/2260334019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b81527c0 msgr2=0x7f66b80c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66bd763640 1 --2- 172.21.15.94:0/2260334019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b81527c0 0x7f66b80c73d0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009d90 tx=0x7f66a802beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66bd763640 1 -- 172.21.15.94:0/2260334019 shutdown_connections 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66bd763640 1 --2- 172.21.15.94:0/2260334019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b81527c0 0x7f66b80c73d0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66bd763640 1 -- 172.21.15.94:0/2260334019 >> 172.21.15.94:0/2260334019 conn(0x7f66b80ac550 msgr2=0x7f66b80ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66bd763640 1 -- 172.21.15.94:0/2260334019 shutdown_connections 2024-03-06T07:24:51.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.054+0000 7f66bd763640 1 -- 172.21.15.94:0/2260334019 wait complete. 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.055+0000 7f66bd763640 1 Processor -- start 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.055+0000 7f66bd763640 1 -- start start 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66bd763640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66b80c88c0 0x7f66b80c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66bd763640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66b81527c0 0x7f66b814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66bd763640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b8155e10 0x7f66b814d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66bd763640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b806d590 con 0x7f66b8155e10 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66bd763640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b8155610 con 0x7f66b80c88c0 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66bd763640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b806da10 con 0x7f66b81527c0 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b8155e10 0x7f66b814d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b8155e10 0x7f66b814d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43092/0 (socket says 172.21.15.94:43092) 2024-03-06T07:24:51.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 -- 172.21.15.94:0/2397482639 learned_addr learned my addr 172.21.15.94:0/2397482639 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66b80c88c0 0x7f66b80c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b67fc640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66b81527c0 0x7f66b814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 -- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66b81527c0 msgr2=0x7f66b814d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66b81527c0 0x7f66b814d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 -- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66b80c88c0 msgr2=0x7f66b80c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66b80c88c0 0x7f66b80c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b77fe640 1 -- 172.21.15.94:0/2397482639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b814f680 con 0x7f66b8155e10 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.056+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66b80c88c0 0x7f66b80c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.057+0000 7f66b77fe640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b8155e10 0x7f66b814d620 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f66b000c780 tx=0x7f66b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.057+0000 7f669f7fe640 1 -- 172.21.15.94:0/2397482639 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0018070 con 0x7f66b8155e10 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.057+0000 7f669f7fe640 1 -- 172.21.15.94:0/2397482639 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00070d0 con 0x7f66b8155e10 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.057+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b814f960 con 0x7f66b8155e10 2024-03-06T07:24:51.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.057+0000 7f669f7fe640 1 -- 172.21.15.94:0/2397482639 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0007500 con 0x7f66b8155e10 2024-03-06T07:24:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.057+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b80c4490 con 0x7f66b8155e10 2024-03-06T07:24:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.058+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b804f330 con 0x7f66b8155e10 2024-03-06T07:24:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.058+0000 7f669f7fe640 1 -- 172.21.15.94:0/2397482639 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66b001b070 con 0x7f66b8155e10 2024-03-06T07:24:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.059+0000 7f669f7fe640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a406b4d0 0x7f66a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.059+0000 7f669f7fe640 1 -- 172.21.15.94:0/2397482639 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f66b0089390 con 0x7f66b8155e10 2024-03-06T07:24:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.059+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a406b4d0 0x7f66a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.059+0000 7f66b6ffd640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a406b4d0 0x7f66a406d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f66a8036000 tx=0x7f66a8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.063+0000 7f669f7fe640 1 -- 172.21.15.94:0/2397482639 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66b8152ba0 con 0x7f66b8155e10 2024-03-06T07:24:51.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.235+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f66b806d7f0 con 0x7f66b8155e10 2024-03-06T07:24:51.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.238+0000 7f669f7fe640 1 -- 172.21.15.94:0/2397482639 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v57)=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7f66b0058310 con 0x7f66b8155e10 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a406b4d0 msgr2=0x7f66a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a406b4d0 0x7f66a406d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f66a8036000 tx=0x7f66a8030000 comp rx=0 tx=0).stop 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b8155e10 msgr2=0x7f66b814d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b8155e10 0x7f66b814d620 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f66b000c780 tx=0x7f66b000cc40 comp rx=0 tx=0).stop 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 shutdown_connections 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66a406b4d0 0x7f66a406d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66b8155e10 0x7f66b814d620 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66b81527c0 0x7f66b814d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 --2- 172.21.15.94:0/2397482639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66b80c88c0 0x7f66b80c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 >> 172.21.15.94:0/2397482639 conn(0x7f66b80ac550 msgr2=0x7f66b80cb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 shutdown_connections 2024-03-06T07:24:51.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.240+0000 7f66bd763640 1 -- 172.21.15.94:0/2397482639 wait complete. 2024-03-06T07:24:51.252 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:51.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:51.294 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:51.337 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:51.337 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:24:51.337 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:51.352 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:24:51.352 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:24:51.352 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:51.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:24:51.395 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:24:51.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:51.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.492+0000 7f66fa1a0640 1 Processor -- start 2024-03-06T07:24:51.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66fa1a0640 1 -- start start 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66fa1a0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f66f414f360 0x7f66f4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66fa1a0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f66f40c88c0 0x7f66f40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66fa1a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c5880 0x7f66f4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66fa1a0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f40c8dd0 con 0x7f66f40c88c0 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66fa1a0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f40c73d0 con 0x7f66f414f360 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66fa1a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f40be2e0 con 0x7f66f40c5880 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66eaffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f66f40c88c0 0x7f66f40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66eaffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f66f40c88c0 0x7f66f40c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43100/0 (socket says 172.21.15.94:43100) 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.493+0000 7f66eaffd640 1 -- 172.21.15.94:0/1646004756 learned_addr learned my addr 172.21.15.94:0/1646004756 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66f37fe640 1 --2- 172.21.15.94:0/1646004756 >> v2:172.21.15.145:3300/0 conn(0x7f66f414f360 0x7f66f4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66f3fff640 1 --2- 172.21.15.94:0/1646004756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c5880 0x7f66f4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66eaffd640 1 -- 172.21.15.94:0/1646004756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c5880 msgr2=0x7f66f4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66eaffd640 1 --2- 172.21.15.94:0/1646004756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c5880 0x7f66f4158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66eaffd640 1 -- 172.21.15.94:0/1646004756 >> v2:172.21.15.145:3300/0 conn(0x7f66f414f360 msgr2=0x7f66f4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66eaffd640 1 --2- 172.21.15.94:0/1646004756 >> v2:172.21.15.145:3300/0 conn(0x7f66f414f360 0x7f66f4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66eaffd640 1 -- 172.21.15.94:0/1646004756 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f40c7670 con 0x7f66f40c88c0 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.494+0000 7f66eaffd640 1 --2- 172.21.15.94:0/1646004756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c88c0 0x7f66f40c5340 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f66f408c840 tx=0x7f66dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3aad31839a3bb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66f2ffd640 1 -- 172.21.15.94:0/1646004756 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc02e070 con 0x7f66f40c88c0 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66f2ffd640 1 -- 172.21.15.94:0/1646004756 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc037070 con 0x7f66f40c88c0 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66f2ffd640 1 -- 172.21.15.94:0/1646004756 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc031a00 con 0x7f66f40c88c0 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 -- 172.21.15.94:0/1646004756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c88c0 msgr2=0x7f66f40c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/1646004756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c88c0 0x7f66f40c5340 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f66f408c840 tx=0x7f66dc02b880 comp rx=0 tx=0).stop 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 -- 172.21.15.94:0/1646004756 shutdown_connections 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/1646004756 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c5880 0x7f66f4158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/1646004756 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c88c0 0x7f66f40c5340 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/1646004756 >> v2:172.21.15.145:3300/0 conn(0x7f66f414f360 0x7f66f4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 -- 172.21.15.94:0/1646004756 >> 172.21.15.94:0/1646004756 conn(0x7f66f40ac570 msgr2=0x7f66f40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 -- 172.21.15.94:0/1646004756 shutdown_connections 2024-03-06T07:24:51.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.495+0000 7f66fa1a0640 1 -- 172.21.15.94:0/1646004756 wait complete. 2024-03-06T07:24:51.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.496+0000 7f66fa1a0640 1 Processor -- start 2024-03-06T07:24:51.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.496+0000 7f66fa1a0640 1 -- start start 2024-03-06T07:24:51.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66fa1a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c5880 0x7f66f40c10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66fa1a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c88c0 0x7f66f40c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66fa1a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66f414f360 0x7f66f40c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66fa1a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f41590a0 con 0x7f66f40c5880 2024-03-06T07:24:51.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66fa1a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f40c7c30 con 0x7f66f414f360 2024-03-06T07:24:51.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66fa1a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f4159220 con 0x7f66f40c88c0 2024-03-06T07:24:51.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c5880 0x7f66f40c10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c5880 0x7f66f40c10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43106/0 (socket says 172.21.15.94:43106) 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 -- 172.21.15.94:0/3083537049 learned_addr learned my addr 172.21.15.94:0/3083537049 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66eaffd640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c88c0 0x7f66f40c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f3fff640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66f414f360 0x7f66f40c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 -- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c88c0 msgr2=0x7f66f40c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c88c0 0x7f66f40c06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 -- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66f414f360 msgr2=0x7f66f40c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66f414f360 0x7f66f40c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f37fe640 1 -- 172.21.15.94:0/3083537049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f40be740 con 0x7f66f40c5880 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66eaffd640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c88c0 0x7f66f40c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.497+0000 7f66f3fff640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66f414f360 0x7f66f40c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.498+0000 7f66f37fe640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c5880 0x7f66f40c10c0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f66e400d5f0 tx=0x7f66e400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.498+0000 7f66f0ff9640 1 -- 172.21.15.94:0/3083537049 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40125f0 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.498+0000 7f66f0ff9640 1 -- 172.21.15.94:0/3083537049 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e4019070 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.498+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f41ee880 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.498+0000 7f66f0ff9640 1 -- 172.21.15.94:0/3083537049 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40129d0 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.498+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f41eeb60 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.499+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f404f330 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.499+0000 7f66f0ff9640 1 -- 172.21.15.94:0/3083537049 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66e4012b70 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.500+0000 7f66f0ff9640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66d406b500 0x7f66d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.500+0000 7f66f0ff9640 1 -- 172.21.15.94:0/3083537049 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f66e408a4c0 con 0x7f66f40c5880 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.500+0000 7f66eaffd640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66d406b500 0x7f66d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.500+0000 7f66eaffd640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66d406b500 0x7f66d406d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f66dc036000 tx=0x7f66dc03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.504+0000 7f66f0ff9640 1 -- 172.21.15.94:0/3083537049 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66e4059440 con 0x7f66f40c5880 2024-03-06T07:24:51.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.713+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66f406cda0 con 0x7f66f40c5880 2024-03-06T07:24:51.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.713+0000 7f66f0ff9640 1 -- 172.21.15.94:0/3083537049 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+3712 (secure 0 0 0) 0x7f66e4058d40 con 0x7f66f40c5880 2024-03-06T07:24:51.715 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:51.715 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":154,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3923835266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3923835266},{"type":"v1","addr":"172.21.15.94:6835","nonce":3923835266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":143},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1221074247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1221074247},{"type":"v1","addr":"172.21.15.94:6834","nonce":1221074247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154},{"gid":37370,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1409431517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1409431517},{"type":"v1","addr":"172.21.15.145:6837","nonce":1409431517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":148},{"gid":37424,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186983770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186983770},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186983770}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":154}],"filesystems":[]} 2024-03-06T07:24:51.715 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 154 2024-03-06T07:24:51.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66d406b500 msgr2=0x7f66d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66d406b500 0x7f66d406d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f66dc036000 tx=0x7f66dc03d040 comp rx=0 tx=0).stop 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c5880 msgr2=0x7f66f40c10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c5880 0x7f66f40c10c0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f66e400d5f0 tx=0x7f66e400dab0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 shutdown_connections 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f66d406b500 0x7f66d406d9c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66f414f360 0x7f66f40c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66f40c88c0 0x7f66f40c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 --2- 172.21.15.94:0/3083537049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f66f40c5880 0x7f66f40c10c0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 >> 172.21.15.94:0/3083537049 conn(0x7f66f40ac570 msgr2=0x7f66f40cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 shutdown_connections 2024-03-06T07:24:51.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.715+0000 7f66fa1a0640 1 -- 172.21.15.94:0/3083537049 wait complete. 2024-03-06T07:24:51.728 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:51.728 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:51.729 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:51.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:51.729 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:51.729 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:51.729 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:51.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:51.729 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:51.729 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:51.730 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:51.730 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:51.730 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:51.730 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:51.730 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:51.730 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:51.738 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:51.738 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:51.738 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:51.739 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:51.741 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:51.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:51.742 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:51.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:51.780 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:51.780 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:51.780 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:51.780 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:51.784 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:51.785 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:51.785 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:51.785 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:51.786 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:51.856 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:24:51.857 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:24:51.866 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:24:51.867 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:24:51.868 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-03-06T07:24:51.866+0000 7f18df309640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6836/0: (98) Address already in use 2024-03-06T07:24:51.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.889+0000 7f04b9fc9640 1 Processor -- start 2024-03-06T07:24:51.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.889+0000 7f04b9fc9640 1 -- start start 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b9fc9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04b4155240 0x7f04b40c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b9fc9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40c8e30 con 0x7f04b4152900 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b9fc9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40c9110 con 0x7f04b40c8950 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b9fc9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b4152ce0 con 0x7f04b4155240 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b37fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f04b4152900 0x7f04b40bf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52434/0 (socket says 172.21.15.94:52434) 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b37fe640 1 -- 172.21.15.94:0/2130189855 learned_addr learned my addr 172.21.15.94:0/2130189855 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563782605 0 0) 0x7f04b40c8e30 con 0x7f04b4152900 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a0003520 con 0x7f04b4152900 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.890+0000 7f04b3fff640 1 --2- 172.21.15.94:0/2130189855 >> v2:172.21.15.145:3300/0 conn(0x7f04b4155240 0x7f04b40c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2509712384 0 0) 0x7f04a0003520 con 0x7f04b4152900 2024-03-06T07:24:51.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 >> v2:172.21.15.145:3300/0 conn(0x7f04b4155240 msgr2=0x7f04b40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 --2- 172.21.15.94:0/2130189855 >> v2:172.21.15.145:3300/0 conn(0x7f04b4155240 0x7f04b40c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 >> v1:172.21.15.145:6789/0 conn(0x7f04b40c8950 legacy=0x7f04b40c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40c93f0 con 0x7f04b4152900 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f04a4001670 con 0x7f04b4152900 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 >> v1:172.21.15.94:6789/0 conn(0x7f04b4152900 legacy=0x7f04b40bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04a0004800 0x7f04a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00052e0 0x7f04a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b4152ce0 con 0x7f04a0004800 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a000c1b0 con 0x7f04a00039b0 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a000c3b0 con 0x7f04a00052e0 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b37fe640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b2ffd640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04a0004800 0x7f04a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b3fff640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00052e0 0x7f04a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b37fe640 1 -- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00052e0 msgr2=0x7f04a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b37fe640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00052e0 0x7f04a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b37fe640 1 -- 172.21.15.94:0/2130189855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04a0004800 msgr2=0x7f04a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b37fe640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04a0004800 0x7f04a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b37fe640 1 -- 172.21.15.94:0/2130189855 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a0003520 con 0x7f04a00039b0 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b3fff640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00052e0 0x7f04a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b2ffd640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04a0004800 0x7f04a0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.891+0000 7f04b3fff640 1 -- 172.21.15.94:0/2130189855 reap_dead start 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.892+0000 7f04b37fe640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04a0003dd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c210 tx=0x7f04a0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=261455436982da5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.892+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4007cf0 con 0x7f04a00039b0 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.892+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4030070 con 0x7f04a00039b0 2024-03-06T07:24:51.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.892+0000 7f04b27fc640 1 -- 172.21.15.94:0/2130189855 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4005d70 con 0x7f04a00039b0 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.893+0000 7f04b9fc9640 1 -- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 msgr2=0x7f04a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.893+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04a0003dd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c210 tx=0x7f04a0005830 comp rx=0 tx=0).stop 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.893+0000 7f04b9fc9640 1 -- 172.21.15.94:0/2130189855 shutdown_connections 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.893+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/2130189855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04a0003dd0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.893+0000 7f04b9fc9640 1 -- 172.21.15.94:0/2130189855 >> 172.21.15.94:0/2130189855 conn(0x7f04b40ac860 msgr2=0x7f04b40acc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.893+0000 7f04b9fc9640 1 -- 172.21.15.94:0/2130189855 shutdown_connections 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.893+0000 7f04b9fc9640 1 -- 172.21.15.94:0/2130189855 wait complete. 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.894+0000 7f04b9fc9640 1 Processor -- start 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.894+0000 7f04b9fc9640 1 -- start start 2024-03-06T07:24:51.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b9fc9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04b41eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b9fc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b41ec180 0x7f04b41566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b9fc9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b4159330 0x7f04b414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b9fc9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40c5770 con 0x7f04b4159330 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b9fc9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40c55f0 con 0x7f04a00039b0 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b9fc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b40bf8f0 con 0x7f04b41ec180 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b4159330 0x7f04b414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b4159330 0x7f04b414fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43144/0 (socket says 172.21.15.94:43144) 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 -- 172.21.15.94:0/3128991363 learned_addr learned my addr 172.21.15.94:0/3128991363 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b37fe640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04b41eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b2ffd640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b41ec180 0x7f04b41566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 -- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b41ec180 msgr2=0x7f04b41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b41ec180 0x7f04b41566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 -- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 msgr2=0x7f04b41eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04b41eb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.895+0000 7f04b3fff640 1 -- 172.21.15.94:0/3128991363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4157020 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.896+0000 7f04b3fff640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b4159330 0x7f04b414fff0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f04b406d290 tx=0x7f04ac00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.896+0000 7f04abfff640 1 -- 172.21.15.94:0/3128991363 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac009c10 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.896+0000 7f04abfff640 1 -- 172.21.15.94:0/3128991363 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac003990 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.896+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b41572c0 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.896+0000 7f04abfff640 1 -- 172.21.15.94:0/3128991363 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac003d80 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.896+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b4151b20 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.897+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b40ca3f0 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.897+0000 7f04abfff640 1 -- 172.21.15.94:0/3128991363 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04ac003f20 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.898+0000 7f04abfff640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049406b4d0 0x7f049406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.898+0000 7f04abfff640 1 -- 172.21.15.94:0/3128991363 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f04ac08ac50 con 0x7f04b4159330 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.898+0000 7f04b37fe640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049406b4d0 0x7f049406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.898+0000 7f04b37fe640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049406b4d0 0x7f049406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f04a4034000 tx=0x7f04a4007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:51.902+0000 7f04abfff640 1 -- 172.21.15.94:0/3128991363 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04b40ca3f0 con 0x7f04b4159330 2024-03-06T07:24:52.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.089+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f04b4067910 con 0x7f04b4159330 2024-03-06T07:24:52.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.955+0000 7f04abfff640 1 -- 172.21.15.94:0/3128991363 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v166) v1 ==== 97+0+0 (secure 0 0 0) 0x7f04b4067910 con 0x7f04b4159330 2024-03-06T07:24:52.957 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:24:52.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049406b4d0 msgr2=0x7f049406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049406b4d0 0x7f049406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f04a4034000 tx=0x7f04a4007000 comp rx=0 tx=0).stop 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b4159330 msgr2=0x7f04b414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b4159330 0x7f04b414fff0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f04b406d290 tx=0x7f04ac00f580 comp rx=0 tx=0).stop 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 shutdown_connections 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049406b4d0 0x7f049406d990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b4159330 0x7f04b414fff0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b41ec180 0x7f04b41566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 --2- 172.21.15.94:0/3128991363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a00039b0 0x7f04b41eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.958+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 >> 172.21.15.94:0/3128991363 conn(0x7f04b40ac860 msgr2=0x7f04b40acc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.959+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 shutdown_connections 2024-03-06T07:24:52.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:52.959+0000 7f04b9fc9640 1 -- 172.21.15.94:0/3128991363 wait complete. 2024-03-06T07:24:52.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:53.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.077+0000 7f0065e6b640 1 Processor -- start 2024-03-06T07:24:53.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.077+0000 7f0065e6b640 1 -- start start 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f0065e6b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0060150580 0x7f006006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f0065e6b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0060153a60 con 0x7f00600c6000 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f0065e6b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0060153d40 con 0x7f00600c1320 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f0065e6b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006006c2c0 con 0x7f0060150580 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0060150580 0x7f006006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f00600c6000 0x7f0060156dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52444/0 (socket says 172.21.15.94:52444) 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0060150580 0x7f006006dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43156/0 (socket says 172.21.15.94:43156) 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005ffff640 1 -- 172.21.15.94:0/3359567924 learned_addr learned my addr 172.21.15.94:0/3359567924 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005f7fe640 1 -- 172.21.15.94:0/3359567924 >> v1:172.21.15.145:6789/0 conn(0x7f00600c1320 legacy=0x7f00601566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005f7fe640 1 -- 172.21.15.94:0/3359567924 >> v1:172.21.15.94:6789/0 conn(0x7f00600c6000 legacy=0x7f0060156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005e7fc640 1 -- 172.21.15.94:0/3359567924 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042459484 0 0) 0x7f0060153a60 con 0x7f00600c6000 2024-03-06T07:24:53.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.078+0000 7f005f7fe640 1 -- 172.21.15.94:0/3359567924 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006006c5a0 con 0x7f0060150580 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.079+0000 7f005f7fe640 1 --2- 172.21.15.94:0/3359567924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f006006dd10 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f00500097a0 tx=0x7f005002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e34a2ffaaa07ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.079+0000 7f005e7fc640 1 -- 172.21.15.94:0/3359567924 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00500315a0 con 0x7f0060150580 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.079+0000 7f005e7fc640 1 -- 172.21.15.94:0/3359567924 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050037070 con 0x7f0060150580 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.079+0000 7f005e7fc640 1 -- 172.21.15.94:0/3359567924 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050031960 con 0x7f0060150580 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.080+0000 7f0065e6b640 1 -- 172.21.15.94:0/3359567924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 msgr2=0x7f006006dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.080+0000 7f0065e6b640 1 --2- 172.21.15.94:0/3359567924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f006006dd10 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f00500097a0 tx=0x7f005002b840 comp rx=0 tx=0).stop 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.080+0000 7f0065e6b640 1 -- 172.21.15.94:0/3359567924 shutdown_connections 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.080+0000 7f0065e6b640 1 --2- 172.21.15.94:0/3359567924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f006006dd10 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.080+0000 7f0065e6b640 1 -- 172.21.15.94:0/3359567924 >> 172.21.15.94:0/3359567924 conn(0x7f00600c6710 msgr2=0x7f00600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.080+0000 7f0065e6b640 1 -- 172.21.15.94:0/3359567924 shutdown_connections 2024-03-06T07:24:53.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.080+0000 7f0065e6b640 1 -- 172.21.15.94:0/3359567924 wait complete. 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.081+0000 7f0065e6b640 1 Processor -- start 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.081+0000 7f0065e6b640 1 -- start start 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00600c1320 0x7f00600c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00600c6000 0x7f0060070720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f0060070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600c0dc0 con 0x7f0060150580 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00600c0c40 con 0x7f00600c6000 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006006e250 con 0x7f00600c1320 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f0060070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f0060070c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43168/0 (socket says 172.21.15.94:43168) 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 -- 172.21.15.94:0/823136791 learned_addr learned my addr 172.21.15.94:0/823136791 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005f7fe640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00600c1320 0x7f00600c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 -- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00600c1320 msgr2=0x7f00600c2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00600c1320 0x7f00600c2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 -- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00600c6000 msgr2=0x7f0060070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005effd640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00600c6000 0x7f0060070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00600c6000 0x7f0060070720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 -- 172.21.15.94:0/823136791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0060072300 con 0x7f0060150580 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005effd640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00600c6000 0x7f0060070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f005ffff640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f0060070c60 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f005800c7d0 tx=0x7f005800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0047fff640 1 -- 172.21.15.94:0/823136791 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058018070 con 0x7f0060150580 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0047fff640 1 -- 172.21.15.94:0/823136791 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005800a640 con 0x7f0060150580 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0060072850 con 0x7f0060150580 2024-03-06T07:24:53.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0047fff640 1 -- 172.21.15.94:0/823136791 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005800aa70 con 0x7f0060150580 2024-03-06T07:24:53.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.082+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0060149840 con 0x7f0060150580 2024-03-06T07:24:53.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.083+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006004f330 con 0x7f0060150580 2024-03-06T07:24:53.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.084+0000 7f0047fff640 1 -- 172.21.15.94:0/823136791 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f005801b070 con 0x7f0060150580 2024-03-06T07:24:53.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.084+0000 7f0047fff640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f004c06b4d0 0x7f004c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.084+0000 7f0047fff640 1 -- 172.21.15.94:0/823136791 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0058089590 con 0x7f0060150580 2024-03-06T07:24:53.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.085+0000 7f005f7fe640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f004c06b4d0 0x7f004c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.085+0000 7f005f7fe640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f004c06b4d0 0x7f004c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0050036000 tx=0x7f0050008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.089+0000 7f0047fff640 1 -- 172.21.15.94:0/823136791 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0058058660 con 0x7f0060150580 2024-03-06T07:24:53.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.325+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f00600c5de0 con 0x7f0060150580 2024-03-06T07:24:53.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.325+0000 7f0047fff640 1 -- 172.21.15.94:0/823136791 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f0058057f60 con 0x7f0060150580 2024-03-06T07:24:53.327 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:53.327 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:24:53.327 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:24:53.328 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:53.329 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:24:53.330 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.331 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:24:53.332 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.333 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.334 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.335 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:24:53.336 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.337 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.338 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:53.339 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:24:53.339 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:24:53.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f004c06b4d0 msgr2=0x7f004c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f004c06b4d0 0x7f004c06d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0050036000 tx=0x7f0050008030 comp rx=0 tx=0).stop 2024-03-06T07:24:53.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 msgr2=0x7f0060070c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f0060070c60 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f005800c7d0 tx=0x7f005800cc90 comp rx=0 tx=0).stop 2024-03-06T07:24:53.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 shutdown_connections 2024-03-06T07:24:53.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f004c06b4d0 0x7f004c06d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0060150580 0x7f0060070c60 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00600c6000 0x7f0060070720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 --2- 172.21.15.94:0/823136791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00600c1320 0x7f00600c2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 >> 172.21.15.94:0/823136791 conn(0x7f00600c6710 msgr2=0x7f00600a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 shutdown_connections 2024-03-06T07:24:53.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.327+0000 7f0065e6b640 1 -- 172.21.15.94:0/823136791 wait complete. 2024-03-06T07:24:53.341 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:53.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.439+0000 7f13cb3f7640 1 Processor -- start 2024-03-06T07:24:53.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.439+0000 7f13cb3f7640 1 -- start start 2024-03-06T07:24:53.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13cb3f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13cb3f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13c40c1aa0 0x7f13c40c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13cb3f7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f13c40c5c60 0x7f13c4155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13cb3f7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c40c2680 con 0x7f13c40c1aa0 2024-03-06T07:24:53.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13cb3f7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c40c28d0 con 0x7f13c40c5c60 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13cb3f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c40c2bb0 con 0x7f13c414f790 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13c40c1aa0 0x7f13c40c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13c40c1aa0 0x7f13c40c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43184/0 (socket says 172.21.15.94:43184) 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 -- 172.21.15.94:0/3425002416 learned_addr learned my addr 172.21.15.94:0/3425002416 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c9193640 1 --2- 172.21.15.94:0/3425002416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c9994640 1 --2- 172.21.15.94:0/3425002416 >> v2:172.21.15.145:3300/0 conn(0x7f13c40c5c60 0x7f13c4155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 -- 172.21.15.94:0/3425002416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 msgr2=0x7f13c41529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 --2- 172.21.15.94:0/3425002416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 -- 172.21.15.94:0/3425002416 >> v2:172.21.15.145:3300/0 conn(0x7f13c40c5c60 msgr2=0x7f13c4155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 --2- 172.21.15.94:0/3425002416 >> v2:172.21.15.145:3300/0 conn(0x7f13c40c5c60 0x7f13c4155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.440+0000 7f13c8992640 1 -- 172.21.15.94:0/3425002416 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c40c2e90 con 0x7f13c40c1aa0 2024-03-06T07:24:53.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.441+0000 7f13c8992640 1 --2- 172.21.15.94:0/3425002416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c5720 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f13c406c960 tx=0x7f13b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1131527fb5796aa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.441+0000 7f13bbfff640 1 -- 172.21.15.94:0/3425002416 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4035020 con 0x7f13c40c1aa0 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.441+0000 7f13bbfff640 1 -- 172.21.15.94:0/3425002416 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b4031870 con 0x7f13c40c1aa0 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.441+0000 7f13bbfff640 1 -- 172.21.15.94:0/3425002416 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4031c40 con 0x7f13c40c1aa0 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 -- 172.21.15.94:0/3425002416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 msgr2=0x7f13c40c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/3425002416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c5720 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f13c406c960 tx=0x7f13b402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 -- 172.21.15.94:0/3425002416 shutdown_connections 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/3425002416 >> v2:172.21.15.145:3300/0 conn(0x7f13c40c5c60 0x7f13c4155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/3425002416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c5720 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/3425002416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 -- 172.21.15.94:0/3425002416 >> 172.21.15.94:0/3425002416 conn(0x7f13c40c6710 msgr2=0x7f13c40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 -- 172.21.15.94:0/3425002416 shutdown_connections 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.442+0000 7f13cb3f7640 1 -- 172.21.15.94:0/3425002416 wait complete. 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.443+0000 7f13cb3f7640 1 Processor -- start 2024-03-06T07:24:53.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.443+0000 7f13cb3f7640 1 -- start start 2024-03-06T07:24:53.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13cb3f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13cb3f7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c40c5c60 0x7f13c41e90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13cb3f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41e9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13cb3f7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c406cc00 con 0x7f13c40c1aa0 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13cb3f7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c40c0d60 con 0x7f13c40c5c60 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13cb3f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c406cd80 con 0x7f13c414f790 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43198/0 (socket says 172.21.15.94:43198) 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 -- 172.21.15.94:0/2229143732 learned_addr learned my addr 172.21.15.94:0/2229143732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 -- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 msgr2=0x7f13c41e9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9994640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c8992640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c40c5c60 0x7f13c41e90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41e9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 -- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c40c5c60 msgr2=0x7f13c41e90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c40c5c60 0x7f13c41e90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 -- 172.21.15.94:0/2229143732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c4158e70 con 0x7f13c40c1aa0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13c9193640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c3410 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f13c000c7f0 tx=0x7f13c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13b9ffb640 1 -- 172.21.15.94:0/2229143732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0015070 con 0x7f13c40c1aa0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13b9ffb640 1 -- 172.21.15.94:0/2229143732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c0004220 con 0x7f13c40c1aa0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c41ed540 con 0x7f13c40c1aa0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.444+0000 7f13b9ffb640 1 -- 172.21.15.94:0/2229143732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0007620 con 0x7f13c40c1aa0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.445+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c41eca60 con 0x7f13c40c1aa0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.446+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c4069460 con 0x7f13c40c1aa0 2024-03-06T07:24:53.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.446+0000 7f13b9ffb640 1 -- 172.21.15.94:0/2229143732 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13c00079d0 con 0x7f13c40c1aa0 2024-03-06T07:24:53.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.446+0000 7f13b9ffb640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f13a806b4d0 0x7f13a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.446+0000 7f13b9ffb640 1 -- 172.21.15.94:0/2229143732 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f13c008a370 con 0x7f13c40c1aa0 2024-03-06T07:24:53.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.446+0000 7f13c8992640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f13a806b4d0 0x7f13a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.447+0000 7f13c8992640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f13a806b4d0 0x7f13a806d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f13b4036000 tx=0x7f13b403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.451+0000 7f13b9ffb640 1 -- 172.21.15.94:0/2229143732 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13c4069460 con 0x7f13c40c1aa0 2024-03-06T07:24:53.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.621+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13c404f330 con 0x7f13c40c1aa0 2024-03-06T07:24:53.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.622+0000 7f13b9ffb640 1 -- 172.21.15.94:0/2229143732 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+13174 (secure 0 0 0) 0x7f13c0059440 con 0x7f13c40c1aa0 2024-03-06T07:24:53.624 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:53.624 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":166,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:52.954300+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:53.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f13a806b4d0 msgr2=0x7f13a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f13a806b4d0 0x7f13a806d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f13b4036000 tx=0x7f13b403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:53.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 msgr2=0x7f13c40c3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c3410 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f13c000c7f0 tx=0x7f13c000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 shutdown_connections 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f13a806b4d0 0x7f13a806d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c414f790 0x7f13c41e9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c40c5c60 0x7f13c41e90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 --2- 172.21.15.94:0/2229143732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13c40c1aa0 0x7f13c40c3410 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 >> 172.21.15.94:0/2229143732 conn(0x7f13c40c6710 msgr2=0x7f13c40a69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 shutdown_connections 2024-03-06T07:24:53.627 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.624+0000 7f13cb3f7640 1 -- 172.21.15.94:0/2229143732 wait complete. 2024-03-06T07:24:53.637 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:53.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:53.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.739+0000 7f7fedc50640 1 Processor -- start 2024-03-06T07:24:53.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.739+0000 7f7fedc50640 1 -- start start 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fedc50640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7fe80c2aa0 0x7f7fe8155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fedc50640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe80670c0 con 0x7f7fe80c0470 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fedc50640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe806b0f0 con 0x7f7fe80c26c0 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fedc50640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe806b740 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7fe80c2aa0 0x7f7fe8155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7fe80c2aa0 0x7f7fe8155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43206/0 (socket says 172.21.15.94:43206) 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe6ffd640 1 -- 172.21.15.94:0/1575059497 learned_addr learned my addr 172.21.15.94:0/1575059497 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:53.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7fe80c0470 0x7f7fe8152350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52448/0 (socket says 172.21.15.94:52448) 2024-03-06T07:24:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe6ffd640 1 -- 172.21.15.94:0/1575059497 >> v1:172.21.15.145:6789/0 conn(0x7f7fe80c26c0 legacy=0x7f7fe806bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe6ffd640 1 -- 172.21.15.94:0/1575059497 >> v1:172.21.15.94:6789/0 conn(0x7f7fe80c0470 legacy=0x7f7fe8152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe6ffd640 1 -- 172.21.15.94:0/1575059497 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe806b3d0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.740+0000 7f7fe67fc640 1 -- 172.21.15.94:0/1575059497 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170202624 0 0) 0x7f7fe80670c0 con 0x7f7fe80c0470 2024-03-06T07:24:53.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.741+0000 7f7fe6ffd640 1 --2- 172.21.15.94:0/1575059497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8155bd0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009870 tx=0x7f7fdc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57b828aa9846979 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.741+0000 7f7fe67fc640 1 -- 172.21.15.94:0/1575059497 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc035020 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.741+0000 7f7fe67fc640 1 -- 172.21.15.94:0/1575059497 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc031870 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.741+0000 7f7fe67fc640 1 -- 172.21.15.94:0/1575059497 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc031c40 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.742+0000 7f7fedc50640 1 -- 172.21.15.94:0/1575059497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 msgr2=0x7f7fe8155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.742+0000 7f7fedc50640 1 --2- 172.21.15.94:0/1575059497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8155bd0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009870 tx=0x7f7fdc02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:53.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.742+0000 7f7fedc50640 1 -- 172.21.15.94:0/1575059497 shutdown_connections 2024-03-06T07:24:53.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.742+0000 7f7fedc50640 1 --2- 172.21.15.94:0/1575059497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8155bd0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.742+0000 7f7fedc50640 1 -- 172.21.15.94:0/1575059497 >> 172.21.15.94:0/1575059497 conn(0x7f7fe80c3b40 msgr2=0x7f7fe80c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.742+0000 7f7fedc50640 1 -- 172.21.15.94:0/1575059497 shutdown_connections 2024-03-06T07:24:53.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.742+0000 7f7fedc50640 1 -- 172.21.15.94:0/1575059497 wait complete. 2024-03-06T07:24:53.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.743+0000 7f7fedc50640 1 Processor -- start 2024-03-06T07:24:53.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.743+0000 7f7fedc50640 1 -- start start 2024-03-06T07:24:53.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fedc50640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7fe80c0470 0x7f7fe81edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fedc50640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe80c26c0 0x7f7fe8146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fedc50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fedc50640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80c20b0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fedc50640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80c1f30 con 0x7f7fe80c26c0 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fedc50640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe8066d00 con 0x7f7fe80c0470 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8147300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43212/0 (socket says 172.21.15.94:43212) 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 -- 172.21.15.94:0/2678170183 learned_addr learned my addr 172.21.15.94:0/2678170183 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe6ffd640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe80c26c0 0x7f7fe8146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe77fe640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7fe80c0470 0x7f7fe81edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 -- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7fe80c0470 msgr2=0x7f7fe81edf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7fe80c0470 0x7f7fe81edf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 -- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe80c26c0 msgr2=0x7f7fe8146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe80c26c0 0x7f7fe8146dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 -- 172.21.15.94:0/2678170183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8154dd0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe77fe640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7fe80c0470 0x7f7fe81edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fe7fff640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8147300 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000c7a0 tx=0x7f7fe000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fcffff640 1 -- 172.21.15.94:0/2678170183 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0018070 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fcffff640 1 -- 172.21.15.94:0/2678170183 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe000a640 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe80aa300 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.744+0000 7f7fcffff640 1 -- 172.21.15.94:0/2678170183 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe000aab0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.745+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe814d0c0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.746+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe804f330 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.746+0000 7f7fcffff640 1 -- 172.21.15.94:0/2678170183 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fe001b070 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.746+0000 7f7fcffff640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fd406b4d0 0x7f7fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.746+0000 7f7fcffff640 1 -- 172.21.15.94:0/2678170183 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7fe00895d0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.746+0000 7f7fe77fe640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fd406b4d0 0x7f7fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.747+0000 7f7fe77fe640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fd406b4d0 0x7f7fd406d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8000f60 tx=0x7f7fd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.750+0000 7f7fcffff640 1 -- 172.21.15.94:0/2678170183 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fe00586a0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.921+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fe806bc70 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.922+0000 7f7fcffff640 1 -- 172.21.15.94:0/2678170183 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v166) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fe0057fa0 con 0x7f7fe80c2aa0 2024-03-06T07:24:53.924 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:24:53.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fd406b4d0 msgr2=0x7f7fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fd406b4d0 0x7f7fd406d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8000f60 tx=0x7f7fd8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 msgr2=0x7f7fe8147300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8147300 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000c7a0 tx=0x7f7fe000cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 shutdown_connections 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fd406b4d0 0x7f7fd406d990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7fe80c2aa0 0x7f7fe8147300 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7fe80c26c0 0x7f7fe8146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 --2- 172.21.15.94:0/2678170183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7fe80c0470 0x7f7fe81edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.923+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 >> 172.21.15.94:0/2678170183 conn(0x7f7fe80c3b40 msgr2=0x7f7fe8152120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.924+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 shutdown_connections 2024-03-06T07:24:53.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:53.924+0000 7f7fedc50640 1 -- 172.21.15.94:0/2678170183 wait complete. 2024-03-06T07:24:53.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.035+0000 7fac60247640 1 Processor -- start 2024-03-06T07:24:54.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac60247640 1 -- start start 2024-03-06T07:24:54.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac60247640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 0x7fac58155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac60247640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fac58158b60 0x7fac58158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac60247640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac580c73d0 con 0x7fac580c88c0 2024-03-06T07:24:54.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac60247640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac580c8dd0 con 0x7fac58158b60 2024-03-06T07:24:54.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac60247640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580be2e0 con 0x7fac5814f360 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac5d7e2640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fac580c88c0 0x7fac580c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52456/0 (socket says 172.21.15.94:52456) 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.036+0000 7fac5d7e2640 1 -- 172.21.15.94:0/2610909542 learned_addr learned my addr 172.21.15.94:0/2610909542 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5dfe3640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 0x7fac58155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5e7e4640 1 --2- 172.21.15.94:0/2610909542 >> v2:172.21.15.145:3300/0 conn(0x7fac58158b60 0x7fac58158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881291127 0 0) 0x7fac580c73d0 con 0x7fac580c88c0 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac44003520 con 0x7fac580c88c0 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745458960 0 0) 0x7fac44003520 con 0x7fac580c88c0 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 msgr2=0x7fac58155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 0x7fac58155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 >> v2:172.21.15.145:3300/0 conn(0x7fac58158b60 msgr2=0x7fac58158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 --2- 172.21.15.94:0/2610909542 >> v2:172.21.15.145:3300/0 conn(0x7fac58158b60 0x7fac58158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580c7670 con 0x7fac580c88c0 2024-03-06T07:24:54.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fac48001670 con 0x7fac580c88c0 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 >> v1:172.21.15.94:6789/0 conn(0x7fac580c88c0 legacy=0x7fac580c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 0x7fac440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac44003f30 0x7fac44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580be2e0 con 0x7fac44004d40 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4400b9c0 con 0x7fac44003f30 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4400bb90 con 0x7fac5814f360 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5e7e4640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5d7e2640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac44003f30 0x7fac44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5dfe3640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 0x7fac440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5e7e4640 1 -- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 msgr2=0x7fac440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5e7e4640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 0x7fac440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5e7e4640 1 -- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac44003f30 msgr2=0x7fac44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.037+0000 7fac5e7e4640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac44003f30 0x7fac44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.038+0000 7fac5e7e4640 1 -- 172.21.15.94:0/2610909542 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44003520 con 0x7fac44004d40 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.038+0000 7fac5dfe3640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac5814f360 0x7fac440039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.038+0000 7fac5e7e4640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac440051c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fac5000e170 tx=0x7fac44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af8dfb046686840 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.038+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50039070 con 0x7fac44004d40 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.038+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5002cc20 con 0x7fac44004d40 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.038+0000 7fac5cfe1640 1 -- 172.21.15.94:0/2610909542 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50009c10 con 0x7fac44004d40 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.039+0000 7fac60247640 1 -- 172.21.15.94:0/2610909542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 msgr2=0x7fac440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.039+0000 7fac60247640 1 --2- 172.21.15.94:0/2610909542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac440051c0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fac5000e170 tx=0x7fac44005830 comp rx=0 tx=0).stop 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.039+0000 7fac5e7e4640 1 -- 172.21.15.94:0/2610909542 reap_dead start 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.039+0000 7fac60247640 1 -- 172.21.15.94:0/2610909542 shutdown_connections 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.039+0000 7fac60247640 1 -- 172.21.15.94:0/2610909542 >> 172.21.15.94:0/2610909542 conn(0x7fac580ac570 msgr2=0x7fac580ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.039+0000 7fac60247640 1 -- 172.21.15.94:0/2610909542 shutdown_connections 2024-03-06T07:24:54.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.039+0000 7fac60247640 1 -- 172.21.15.94:0/2610909542 wait complete. 2024-03-06T07:24:54.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.040+0000 7fac60247640 1 Processor -- start 2024-03-06T07:24:54.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.040+0000 7fac60247640 1 -- start start 2024-03-06T07:24:54.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac60247640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac580c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac60247640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac58158550 0x7fac581521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac60247640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac5814b500 0x7fac5814b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac60247640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580c5a50 con 0x7fac44004d40 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac60247640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580c7c30 con 0x7fac5814b500 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac60247640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580c5bd0 con 0x7fac58158550 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac580c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac580c4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43230/0 (socket says 172.21.15.94:43230) 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 -- 172.21.15.94:0/416152265 learned_addr learned my addr 172.21.15.94:0/416152265 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5d7e2640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac58158550 0x7fac581521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5e7e4640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac5814b500 0x7fac5814b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 -- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac58158550 msgr2=0x7fac581521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac58158550 0x7fac581521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 -- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac5814b500 msgr2=0x7fac5814b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac5814b500 0x7fac5814b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 -- 172.21.15.94:0/416152265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac581eebe0 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5e7e4640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac5814b500 0x7fac5814b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5d7e2640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac58158550 0x7fac581521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.041+0000 7fac5dfe3640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac580c4780 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fac54011660 tx=0x7fac54011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.042+0000 7fac4e7fc640 1 -- 172.21.15.94:0/416152265 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5400f960 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.042+0000 7fac4e7fc640 1 -- 172.21.15.94:0/416152265 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5400fb00 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.042+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac580c1d00 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.042+0000 7fac4e7fc640 1 -- 172.21.15.94:0/416152265 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54015460 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.042+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac581eeec0 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.043+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5804f330 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.043+0000 7fac4e7fc640 1 -- 172.21.15.94:0/416152265 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac54015600 con 0x7fac44004d40 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.043+0000 7fac4e7fc640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fac3c06b4d0 0x7fac3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.043+0000 7fac4e7fc640 1 -- 172.21.15.94:0/416152265 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fac5408a980 con 0x7fac44004d40 2024-03-06T07:24:54.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.044+0000 7fac5d7e2640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fac3c06b4d0 0x7fac3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.044+0000 7fac5d7e2640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fac3c06b4d0 0x7fac3c06d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fac48007fd0 tx=0x7fac48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.048+0000 7fac4e7fc640 1 -- 172.21.15.94:0/416152265 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac54019030 con 0x7fac44004d40 2024-03-06T07:24:54.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.220+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fac5806cda0 con 0x7fac44004d40 2024-03-06T07:24:54.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.220+0000 7fac4e7fc640 1 -- 172.21.15.94:0/416152265 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v166) v1 ==== 84+0+65 (secure 0 0 0) 0x7fac540599c0 con 0x7fac44004d40 2024-03-06T07:24:54.222 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:54.222 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:24:54.222 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:24:54.222 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:24:54.222 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:24:54.223 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:24:54.223 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:24:54.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fac3c06b4d0 msgr2=0x7fac3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fac3c06b4d0 0x7fac3c06d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fac48007fd0 tx=0x7fac48009040 comp rx=0 tx=0).stop 2024-03-06T07:24:54.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 msgr2=0x7fac580c4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac580c4780 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fac54011660 tx=0x7fac54011b20 comp rx=0 tx=0).stop 2024-03-06T07:24:54.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 shutdown_connections 2024-03-06T07:24:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fac3c06b4d0 0x7fac3c06d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac5814b500 0x7fac5814b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac58158550 0x7fac581521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 --2- 172.21.15.94:0/416152265 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fac44004d40 0x7fac580c4780 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 >> 172.21.15.94:0/416152265 conn(0x7fac580ac570 msgr2=0x7fac580ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 shutdown_connections 2024-03-06T07:24:54.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.222+0000 7fac60247640 1 -- 172.21.15.94:0/416152265 wait complete. 2024-03-06T07:24:54.234 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:54.235 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:54.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.337+0000 7efeac363640 1 Processor -- start 2024-03-06T07:24:54.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.337+0000 7efeac363640 1 -- start start 2024-03-06T07:24:54.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeac363640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efea40a9830 0x7efea4151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeac363640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea4153390 0x7efea4153820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeac363640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea406e850 con 0x7efea40c22b0 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeac363640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea406d2b0 con 0x7efea40a9830 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeac363640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea406d590 con 0x7efea4153390 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efea98fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efea40a9830 0x7efea4151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeaa0ff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efea40c22b0 0x7efea414ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52472/0 (socket says 172.21.15.94:52472) 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeaa0ff640 1 -- 172.21.15.94:0/496859051 learned_addr learned my addr 172.21.15.94:0/496859051 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.338+0000 7efeaa900640 1 --2- 172.21.15.94:0/496859051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea4153390 0x7efea4153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea98fe640 1 -- 172.21.15.94:0/496859051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea4153390 msgr2=0x7efea4153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea98fe640 1 --2- 172.21.15.94:0/496859051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea4153390 0x7efea4153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea98fe640 1 -- 172.21.15.94:0/496859051 >> v1:172.21.15.94:6789/0 conn(0x7efea40c22b0 legacy=0x7efea414ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea98fe640 1 -- 172.21.15.94:0/496859051 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea406d870 con 0x7efea40a9830 2024-03-06T07:24:54.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea90fd640 1 -- 172.21.15.94:0/496859051 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763945659 0 0) 0x7efea0001490 con 0x7efea40c22b0 2024-03-06T07:24:54.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea98fe640 1 --2- 172.21.15.94:0/496859051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea40a9830 0x7efea4151e10 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7efe940097d0 tx=0x7efe9402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466a025619d7fdaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea90fd640 1 -- 172.21.15.94:0/496859051 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94035020 con 0x7efea40a9830 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea90fd640 1 -- 172.21.15.94:0/496859051 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe94031800 con 0x7efea40a9830 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.339+0000 7efea90fd640 1 -- 172.21.15.94:0/496859051 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94031bd0 con 0x7efea40a9830 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 -- 172.21.15.94:0/496859051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea40a9830 msgr2=0x7efea4151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 --2- 172.21.15.94:0/496859051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea40a9830 0x7efea4151e10 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7efe940097d0 tx=0x7efe9402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 -- 172.21.15.94:0/496859051 shutdown_connections 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 --2- 172.21.15.94:0/496859051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea4153390 0x7efea4153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 --2- 172.21.15.94:0/496859051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea40a9830 0x7efea4151e10 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 -- 172.21.15.94:0/496859051 >> 172.21.15.94:0/496859051 conn(0x7efea40c65b0 msgr2=0x7efea40c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 -- 172.21.15.94:0/496859051 shutdown_connections 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.340+0000 7efeac363640 1 -- 172.21.15.94:0/496859051 wait complete. 2024-03-06T07:24:54.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.341+0000 7efeac363640 1 Processor -- start 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.341+0000 7efeac363640 1 -- start start 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeac363640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea40a9830 0x7efea40c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeac363640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efea40c22b0 0x7efea4071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeac363640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea4153390 0x7efea41464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeac363640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea414f7e0 con 0x7efea4153390 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeac363640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea414f660 con 0x7efea40c22b0 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeac363640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea40c2690 con 0x7efea40a9830 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea4153390 0x7efea41464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea4153390 0x7efea41464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43256/0 (socket says 172.21.15.94:43256) 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 -- 172.21.15.94:0/1107941681 learned_addr learned my addr 172.21.15.94:0/1107941681 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efea98fe640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efea40c22b0 0x7efea4071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa0ff640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea40a9830 0x7efea40c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 -- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea40a9830 msgr2=0x7efea40c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea40a9830 0x7efea40c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 -- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efea40c22b0 msgr2=0x7efea4071ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efea40c22b0 0x7efea4071ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 -- 172.21.15.94:0/1107941681 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea4147580 con 0x7efea4153390 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efea98fe640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efea40c22b0 0x7efea4071ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa0ff640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea40a9830 0x7efea40c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.342+0000 7efeaa900640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea4153390 0x7efea41464f0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d5d0 tx=0x7efe9c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.343+0000 7efe9affd640 1 -- 172.21.15.94:0/1107941681 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c0125f0 con 0x7efea4153390 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.343+0000 7efe9affd640 1 -- 172.21.15.94:0/1107941681 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c019070 con 0x7efea4153390 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.343+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea40c5810 con 0x7efea4153390 2024-03-06T07:24:54.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.343+0000 7efe9affd640 1 -- 172.21.15.94:0/1107941681 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c0129f0 con 0x7efea4153390 2024-03-06T07:24:54.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.343+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea4147b20 con 0x7efea4153390 2024-03-06T07:24:54.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.343+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea404f330 con 0x7efea4153390 2024-03-06T07:24:54.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.344+0000 7efe9affd640 1 -- 172.21.15.94:0/1107941681 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe9c012b90 con 0x7efea4153390 2024-03-06T07:24:54.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.344+0000 7efe9affd640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe8806b4d0 0x7efe8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.344+0000 7efe9affd640 1 -- 172.21.15.94:0/1107941681 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5249+0+0 (secure 0 0 0) 0x7efe9c08a400 con 0x7efea4153390 2024-03-06T07:24:54.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.345+0000 7efeaa0ff640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe8806b4d0 0x7efe8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.345+0000 7efeaa0ff640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe8806b4d0 0x7efe8806d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7efea0007fd0 tx=0x7efea0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.349+0000 7efe9affd640 1 -- 172.21.15.94:0/1107941681 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe9c059450 con 0x7efea4153390 2024-03-06T07:24:54.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.521+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efea4151f00 con 0x7efea4153390 2024-03-06T07:24:54.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.960+0000 7efe9affd640 1 -- 172.21.15.94:0/1107941681 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v167) v1 ==== 138+0+0 (secure 0 0 0) 0x7efe9c058d50 con 0x7efea4153390 2024-03-06T07:24:54.962 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe8806b4d0 msgr2=0x7efe8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe8806b4d0 0x7efe8806d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7efea0007fd0 tx=0x7efea0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea4153390 msgr2=0x7efea41464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea4153390 0x7efea41464f0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d5d0 tx=0x7efe9c00da90 comp rx=0 tx=0).stop 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 shutdown_connections 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efe8806b4d0 0x7efe8806d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efea4153390 0x7efea41464f0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efea40c22b0 0x7efea4071ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 --2- 172.21.15.94:0/1107941681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efea40a9830 0x7efea40c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 >> 172.21.15.94:0/1107941681 conn(0x7efea40c65b0 msgr2=0x7efea4150930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 shutdown_connections 2024-03-06T07:24:54.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:54.963+0000 7efeac363640 1 -- 172.21.15.94:0/1107941681 wait complete. 2024-03-06T07:24:54.980 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.085+0000 7f95c4466640 1 Processor -- start 2024-03-06T07:24:55.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.085+0000 7f95c4466640 1 -- start start 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.086+0000 7f95c4466640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0c5990 0x7f95bc0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.086+0000 7f95c4466640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f95bc0ab170 0x7f95bc06c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.086+0000 7f95c4466640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc0c3ef0 con 0x7f95bc06c730 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.086+0000 7f95c4466640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95bc06cbe0 con 0x7f95bc0ab170 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.086+0000 7f95c4466640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1556d0 con 0x7f95bc0c5990 2024-03-06T07:24:55.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c1a01640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f95bc0ab170 0x7f95bc06c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c2a03640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f95bc06c730 0x7f95bc1523f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37540/0 (socket says 172.21.15.94:37540) 2024-03-06T07:24:55.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c2202640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0c5990 0x7f95bc0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c2a03640 1 -- 172.21.15.94:0/3819314315 learned_addr learned my addr 172.21.15.94:0/3819314315 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c2202640 1 -- 172.21.15.94:0/3819314315 >> v2:172.21.15.145:3300/0 conn(0x7f95bc0ab170 msgr2=0x7f95bc06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c2202640 1 --2- 172.21.15.94:0/3819314315 >> v2:172.21.15.145:3300/0 conn(0x7f95bc0ab170 0x7f95bc06c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c2202640 1 -- 172.21.15.94:0/3819314315 >> v1:172.21.15.145:6789/0 conn(0x7f95bc06c730 legacy=0x7f95bc1523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c2202640 1 -- 172.21.15.94:0/3819314315 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc1559b0 con 0x7f95bc0c5990 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.087+0000 7f95c1a01640 1 --2- 172.21.15.94:0/3819314315 >> v2:172.21.15.145:3300/0 conn(0x7f95bc0ab170 0x7f95bc06c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.088+0000 7f95c2202640 1 --2- 172.21.15.94:0/3819314315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0c5990 0x7f95bc0c0ab0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ebac0d352aa5596 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.088+0000 7f95c1200640 1 -- 172.21.15.94:0/3819314315 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8039070 con 0x7f95bc0c5990 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.088+0000 7f95c1200640 1 -- 172.21.15.94:0/3819314315 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b802cbc0 con 0x7f95bc0c5990 2024-03-06T07:24:55.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.088+0000 7f95c1200640 1 -- 172.21.15.94:0/3819314315 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8034440 con 0x7f95bc0c5990 2024-03-06T07:24:55.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 -- 172.21.15.94:0/3819314315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0c5990 msgr2=0x7f95bc0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 --2- 172.21.15.94:0/3819314315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0c5990 0x7f95bc0c0ab0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b802be10 comp rx=0 tx=0).stop 2024-03-06T07:24:55.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 -- 172.21.15.94:0/3819314315 shutdown_connections 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 --2- 172.21.15.94:0/3819314315 >> v2:172.21.15.145:3300/0 conn(0x7f95bc0ab170 0x7f95bc06c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 --2- 172.21.15.94:0/3819314315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0c5990 0x7f95bc0c0ab0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 -- 172.21.15.94:0/3819314315 >> 172.21.15.94:0/3819314315 conn(0x7f95bc0c6710 msgr2=0x7f95bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 -- 172.21.15.94:0/3819314315 shutdown_connections 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.089+0000 7f95c4466640 1 -- 172.21.15.94:0/3819314315 wait complete. 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.090+0000 7f95c4466640 1 Processor -- start 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.090+0000 7f95c4466640 1 -- start start 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c4466640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95bc06c730 0x7f95bc1edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c4466640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0ab170 0x7f95bc14dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c4466640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0c5990 0x7f95bc0a6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c4466640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc156490 con 0x7f95bc06c730 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c4466640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc155f70 con 0x7f95bc0c5990 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c4466640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc156610 con 0x7f95bc0ab170 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95bc06c730 0x7f95bc1edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95bc06c730 0x7f95bc1edeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43262/0 (socket says 172.21.15.94:43262) 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 -- 172.21.15.94:0/2754924450 learned_addr learned my addr 172.21.15.94:0/2754924450 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c1a01640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0ab170 0x7f95bc14dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2a03640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0c5990 0x7f95bc0a6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 -- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0ab170 msgr2=0x7f95bc14dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0ab170 0x7f95bc14dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 -- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0c5990 msgr2=0x7f95bc0a6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0c5990 0x7f95bc0a6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 -- 172.21.15.94:0/2754924450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc0a6a10 con 0x7f95bc06c730 2024-03-06T07:24:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2a03640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0c5990 0x7f95bc0a6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:55.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95c2202640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95bc06c730 0x7f95bc1edeb0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95b2ffd640 1 -- 172.21.15.94:0/2754924450 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8039050 con 0x7f95bc06c730 2024-03-06T07:24:55.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.091+0000 7f95b2ffd640 1 -- 172.21.15.94:0/2754924450 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b8045070 con 0x7f95bc06c730 2024-03-06T07:24:55.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.092+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc14ce90 con 0x7f95bc06c730 2024-03-06T07:24:55.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.092+0000 7f95b2ffd640 1 -- 172.21.15.94:0/2754924450 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b803e850 con 0x7f95bc06c730 2024-03-06T07:24:55.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.092+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc14d450 con 0x7f95bc06c730 2024-03-06T07:24:55.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.092+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc0a6be0 con 0x7f95bc06c730 2024-03-06T07:24:55.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.093+0000 7f95b2ffd640 1 -- 172.21.15.94:0/2754924450 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95b803e9f0 con 0x7f95bc06c730 2024-03-06T07:24:55.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.093+0000 7f95b2ffd640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95a006b4d0 0x7f95a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.094+0000 7f95b2ffd640 1 -- 172.21.15.94:0/2754924450 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f95b80acf50 con 0x7f95bc06c730 2024-03-06T07:24:55.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.094+0000 7f95c1a01640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95a006b4d0 0x7f95a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.094+0000 7f95c1a01640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95a006b4d0 0x7f95a006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0c0b80 tx=0x7f95ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.098+0000 7f95b2ffd640 1 -- 172.21.15.94:0/2754924450 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95b807be30 con 0x7f95bc06c730 2024-03-06T07:24:55.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.270+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f95bc04f330 con 0x7f95bc06c730 2024-03-06T07:24:55.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.966+0000 7f95b2ffd640 1 -- 172.21.15.94:0/2754924450 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v168) v1 ==== 170+0+0 (secure 0 0 0) 0x7f95b807b730 con 0x7f95bc06c730 2024-03-06T07:24:55.968 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:24:55.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.970+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95a006b4d0 msgr2=0x7f95a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.970+0000 7f95c4466640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95a006b4d0 0x7f95a006d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0c0b80 tx=0x7f95ac008040 comp rx=0 tx=0).stop 2024-03-06T07:24:55.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.970+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95bc06c730 msgr2=0x7f95bc1edeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.970+0000 7f95c4466640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95bc06c730 0x7f95bc1edeb0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b8004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:55.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.974+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 shutdown_connections 2024-03-06T07:24:55.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.974+0000 7f95c4466640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f95a006b4d0 0x7f95a006d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.974+0000 7f95c4466640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0c5990 0x7f95bc0a6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.974+0000 7f95c4466640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0ab170 0x7f95bc14dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.974+0000 7f95c4466640 1 --2- 172.21.15.94:0/2754924450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95bc06c730 0x7f95bc1edeb0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.974+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 >> 172.21.15.94:0/2754924450 conn(0x7f95bc0c6710 msgr2=0x7f95bc06cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.982+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 shutdown_connections 2024-03-06T07:24:55.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:55.983+0000 7f95c4466640 1 -- 172.21.15.94:0/2754924450 wait complete. 2024-03-06T07:24:56.000 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:56.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.106+0000 7ff0f6913640 1 Processor -- start 2024-03-06T07:24:56.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.106+0000 7ff0f6913640 1 -- start start 2024-03-06T07:24:56.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.106+0000 7ff0f6913640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0f00a8ff0 0x7ff0f00ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.106+0000 7ff0f6913640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a6290 0x7ff0f00a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.106+0000 7ff0f6913640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f00c1f90 con 0x7ff0f00ab730 2024-03-06T07:24:56.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.106+0000 7ff0f6913640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f014f850 con 0x7ff0f00a8ff0 2024-03-06T07:24:56.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.106+0000 7ff0f6913640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f0156260 con 0x7ff0f00a6290 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0effff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0f00a8ff0 0x7ff0f00ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0effff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0f00a8ff0 0x7ff0f00ab1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43266/0 (socket says 172.21.15.94:43266) 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0effff640 1 -- 172.21.15.94:0/3600952293 learned_addr learned my addr 172.21.15.94:0/3600952293 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0ef7fe640 1 --2- 172.21.15.94:0/3600952293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a6290 0x7ff0f00a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0effff640 1 -- 172.21.15.94:0/3600952293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a6290 msgr2=0x7ff0f00a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0effff640 1 --2- 172.21.15.94:0/3600952293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a6290 0x7ff0f00a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0effff640 1 -- 172.21.15.94:0/3600952293 >> v1:172.21.15.145:6789/0 conn(0x7ff0f00ab730 legacy=0x7ff0f00aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0effff640 1 -- 172.21.15.94:0/3600952293 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f00c11d0 con 0x7ff0f00a8ff0 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.107+0000 7ff0ef7fe640 1 --2- 172.21.15.94:0/3600952293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a6290 0x7ff0f00a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0effff640 1 --2- 172.21.15.94:0/3600952293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00a8ff0 0x7ff0f00ab1f0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009870 tx=0x7ff0e002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880d42139ca76b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0eeffd640 1 -- 172.21.15.94:0/3600952293 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0038070 con 0x7ff0f00a8ff0 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0eeffd640 1 -- 172.21.15.94:0/3600952293 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e0031740 con 0x7ff0f00a8ff0 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0eeffd640 1 -- 172.21.15.94:0/3600952293 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e002f050 con 0x7ff0f00a8ff0 2024-03-06T07:24:56.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 -- 172.21.15.94:0/3600952293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00a8ff0 msgr2=0x7ff0f00ab1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3600952293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00a8ff0 0x7ff0f00ab1f0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009870 tx=0x7ff0e002b7e0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 -- 172.21.15.94:0/3600952293 shutdown_connections 2024-03-06T07:24:56.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3600952293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a6290 0x7ff0f00a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3600952293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00a8ff0 0x7ff0f00ab1f0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 -- 172.21.15.94:0/3600952293 >> 172.21.15.94:0/3600952293 conn(0x7ff0f006afc0 msgr2=0x7ff0f006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 -- 172.21.15.94:0/3600952293 shutdown_connections 2024-03-06T07:24:56.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.108+0000 7ff0f6913640 1 -- 172.21.15.94:0/3600952293 wait complete. 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.109+0000 7ff0f6913640 1 Processor -- start 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f6913640 1 -- start start 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f6913640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a8ff0 0x7ff0f014e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f6913640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00ab730 0x7ff0f014eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f6913640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f01ebea0 0x7ff0f01ec350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f6913640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c1a70 con 0x7ff0f00ab730 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f6913640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00c1790 con 0x7ff0f01ebea0 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f6913640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f0155cf0 con 0x7ff0f00a8ff0 2024-03-06T07:24:56.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0ef7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00ab730 0x7ff0f014eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0ef7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00ab730 0x7ff0f014eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43280/0 (socket says 172.21.15.94:43280) 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3160072737 learned_addr learned my addr 172.21.15.94:0/3160072737 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0effff640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a8ff0 0x7ff0f014e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.110+0000 7ff0f4eb0640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f01ebea0 0x7ff0f01ec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a8ff0 msgr2=0x7ff0f014e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ef7fe640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a8ff0 0x7ff0f014e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f01ebea0 msgr2=0x7ff0f01ec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ef7fe640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f01ebea0 0x7ff0f01ec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ef7fe640 1 -- 172.21.15.94:0/3160072737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f014fe30 con 0x7ff0f00ab730 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0f4eb0640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f01ebea0 0x7ff0f01ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ef7fe640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00ab730 0x7ff0f014eba0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400c7f0 tx=0x7ff0e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ecff9640 1 -- 172.21.15.94:0/3160072737 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4015070 con 0x7ff0f00ab730 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ecff9640 1 -- 172.21.15.94:0/3160072737 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e4007a30 con 0x7ff0f00ab730 2024-03-06T07:24:56.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f014adc0 con 0x7ff0f00ab730 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0ecff9640 1 -- 172.21.15.94:0/3160072737 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4007e00 con 0x7ff0f00ab730 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.111+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f00c74c0 con 0x7ff0f00ab730 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.112+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f004f330 con 0x7ff0f00ab730 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.113+0000 7ff0ecff9640 1 -- 172.21.15.94:0/3160072737 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0e4018070 con 0x7ff0f00ab730 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.113+0000 7ff0ecff9640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.113+0000 7ff0ecff9640 1 -- 172.21.15.94:0/3160072737 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6001+0+0 (secure 0 0 0) 0x7ff0e408a1a0 con 0x7ff0f00ab730 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.113+0000 7ff0effff640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.114+0000 7ff0effff640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0035000 tx=0x7ff0e0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.118+0000 7ff0ecff9640 1 -- 172.21.15.94:0/3160072737 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0e4058930 con 0x7ff0f00ab730 2024-03-06T07:24:56.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.325+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff0f01556a0 con 0x7ff0f00ab730 2024-03-06T07:24:56.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.978+0000 7ff0ecff9640 1 -- 172.21.15.94:0/3160072737 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v156) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff0e405c9d0 con 0x7ff0f00ab730 2024-03-06T07:24:56.980 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 27 and data pool 28 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 msgr2=0x7ff0dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0035000 tx=0x7ff0e0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00ab730 msgr2=0x7ff0f014eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00ab730 0x7ff0f014eba0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400c7f0 tx=0x7ff0e400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 shutdown_connections 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff0dc06b4d0 0x7ff0dc06d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0f01ebea0 0x7ff0f01ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff0f00ab730 0x7ff0f014eba0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 --2- 172.21.15.94:0/3160072737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0f00a8ff0 0x7ff0f014e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 >> 172.21.15.94:0/3160072737 conn(0x7ff0f006afc0 msgr2=0x7ff0f006b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 shutdown_connections 2024-03-06T07:24:56.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:56.986+0000 7ff0f6913640 1 -- 172.21.15.94:0/3160072737 wait complete. 2024-03-06T07:24:57.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:57.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.106+0000 7fe53bea3640 1 Processor -- start 2024-03-06T07:24:57.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.106+0000 7fe53bea3640 1 -- start start 2024-03-06T07:24:57.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe53bea3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5341550c0 0x7fe5341554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe53bea3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe5341559e0 0x7fe53414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe53bea3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5340aa660 con 0x7fe5340aa180 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe53bea3640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5340c4610 con 0x7fe5341550c0 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe53bea3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5340aa940 con 0x7fe5341559e0 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe539c3f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5341550c0 0x7fe5341554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe539c3f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe5341550c0 0x7fe5341554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43286/0 (socket says 172.21.15.94:43286) 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe539c3f640 1 -- 172.21.15.94:0/747588806 learned_addr learned my addr 172.21.15.94:0/747588806 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe53943e640 1 --2- 172.21.15.94:0/747588806 >> v2:172.21.15.145:3300/0 conn(0x7fe5341559e0 0x7fe53414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe539c3f640 1 -- 172.21.15.94:0/747588806 >> v2:172.21.15.145:3300/0 conn(0x7fe5341559e0 msgr2=0x7fe53414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe539c3f640 1 --2- 172.21.15.94:0/747588806 >> v2:172.21.15.145:3300/0 conn(0x7fe5341559e0 0x7fe53414ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe539c3f640 1 -- 172.21.15.94:0/747588806 >> v1:172.21.15.94:6789/0 conn(0x7fe5340aa180 legacy=0x7fe5340c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.107+0000 7fe539c3f640 1 -- 172.21.15.94:0/747588806 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5340c4860 con 0x7fe5341550c0 2024-03-06T07:24:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe539c3f640 1 --2- 172.21.15.94:0/747588806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5341554a0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fe5340abe90 tx=0x7fe53002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de481bc727145e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe538c3d640 1 -- 172.21.15.94:0/747588806 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53002e070 con 0x7fe5341550c0 2024-03-06T07:24:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe538c3d640 1 -- 172.21.15.94:0/747588806 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe530037070 con 0x7fe5341550c0 2024-03-06T07:24:57.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe538c3d640 1 -- 172.21.15.94:0/747588806 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530031a00 con 0x7fe5341550c0 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe53bea3640 1 -- 172.21.15.94:0/747588806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 msgr2=0x7fe5341554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe53bea3640 1 --2- 172.21.15.94:0/747588806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5341554a0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fe5340abe90 tx=0x7fe53002b880 comp rx=0 tx=0).stop 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe53bea3640 1 -- 172.21.15.94:0/747588806 shutdown_connections 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe53bea3640 1 --2- 172.21.15.94:0/747588806 >> v2:172.21.15.145:3300/0 conn(0x7fe5341559e0 0x7fe53414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe53bea3640 1 --2- 172.21.15.94:0/747588806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5341554a0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.108+0000 7fe53bea3640 1 -- 172.21.15.94:0/747588806 >> 172.21.15.94:0/747588806 conn(0x7fe5340c6710 msgr2=0x7fe5340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.109+0000 7fe53bea3640 1 -- 172.21.15.94:0/747588806 shutdown_connections 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.109+0000 7fe53bea3640 1 -- 172.21.15.94:0/747588806 wait complete. 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 Processor -- start 2024-03-06T07:24:57.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 -- start start 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5340aa180 0x7fe5340a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5340a89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5341559e0 0x7fe5340a8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53414f140 con 0x7fe5341550c0 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340c1560 con 0x7fe5340aa180 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53bea3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53414f2c0 con 0x7fe5341559e0 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53943e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5340a89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53943e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5340a89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43292/0 (socket says 172.21.15.94:43292) 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53943e640 1 -- 172.21.15.94:0/3922555612 learned_addr learned my addr 172.21.15.94:0/3922555612 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe539c3f640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5340aa180 0x7fe5340a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.110+0000 7fe53a440640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5341559e0 0x7fe5340a8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53943e640 1 -- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5341559e0 msgr2=0x7fe5340a8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53943e640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5341559e0 0x7fe5340a8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53943e640 1 -- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5340aa180 msgr2=0x7fe5340a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53943e640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5340aa180 0x7fe5340a7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53943e640 1 -- 172.21.15.94:0/3922555612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5340a7220 con 0x7fe5341550c0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53943e640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5340a89e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fe52400d5f0 tx=0x7fe52400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe52a7fc640 1 -- 172.21.15.94:0/3922555612 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240125f0 con 0x7fe5341550c0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe52a7fc640 1 -- 172.21.15.94:0/3922555612 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524019070 con 0x7fe5341550c0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5340a8280 con 0x7fe5341550c0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe52a7fc640 1 -- 172.21.15.94:0/3922555612 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240129d0 con 0x7fe5341550c0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.111+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53406d940 con 0x7fe5341550c0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.112+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53404f330 con 0x7fe5341550c0 2024-03-06T07:24:57.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.112+0000 7fe52a7fc640 1 -- 172.21.15.94:0/3922555612 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe524012b70 con 0x7fe5341550c0 2024-03-06T07:24:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.113+0000 7fe52a7fc640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe51806b4d0 0x7fe51806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.113+0000 7fe52a7fc640 1 -- 172.21.15.94:0/3922555612 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe52408a570 con 0x7fe5341550c0 2024-03-06T07:24:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.113+0000 7fe539c3f640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe51806b4d0 0x7fe51806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.113+0000 7fe539c3f640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe51806b4d0 0x7fe51806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe530036000 tx=0x7fe53003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.117+0000 7fe52a7fc640 1 -- 172.21.15.94:0/3922555612 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe524058ca0 con 0x7fe5341550c0 2024-03-06T07:24:57.246 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:24:57.244+0000 7f14836d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:57.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.291+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe534068450 con 0x7fe5341550c0 2024-03-06T07:24:57.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.292+0000 7fe52a7fc640 1 -- 172.21.15.94:0/3922555612 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+16965 (secure 0 0 0) 0x7fe52405cd40 con 0x7fe5341550c0 2024-03-06T07:24:57.294 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:57.294 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":169,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:56.326806+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:54.539968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:55.288852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe51806b4d0 msgr2=0x7fe51806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe51806b4d0 0x7fe51806d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe530036000 tx=0x7fe53003d040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 msgr2=0x7fe5340a89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5340a89e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fe52400d5f0 tx=0x7fe52400dab0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 shutdown_connections 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe51806b4d0 0x7fe51806d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5341559e0 0x7fe5340a8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5341550c0 0x7fe5340a89e0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 --2- 172.21.15.94:0/3922555612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5340aa180 0x7fe5340a7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 >> 172.21.15.94:0/3922555612 conn(0x7fe5340c6710 msgr2=0x7fe53414d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 shutdown_connections 2024-03-06T07:24:57.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.294+0000 7fe53bea3640 1 -- 172.21.15.94:0/3922555612 wait complete. 2024-03-06T07:24:57.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.405+0000 7f1b20081640 1 Processor -- start 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.405+0000 7f1b20081640 1 -- start start 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b20081640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1b180c1aa0 0x7f1b181529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b20081640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b180c28d0 con 0x7f1b18155a40 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b20081640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b180c2bb0 con 0x7f1b1814f790 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b20081640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b180c2680 con 0x7f1b180c1aa0 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1d61c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1b180c1aa0 0x7f1b181529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1d61c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1b180c1aa0 0x7f1b181529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43304/0 (socket says 172.21.15.94:43304) 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1e61e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1b18155a40 0x7f1b18155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52482/0 (socket says 172.21.15.94:52482) 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1d61c640 1 -- 172.21.15.94:0/3580765272 learned_addr learned my addr 172.21.15.94:0/3580765272 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:57.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1d61c640 1 -- 172.21.15.94:0/3580765272 >> v1:172.21.15.145:6789/0 conn(0x7f1b1814f790 legacy=0x7f1b180c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1d61c640 1 -- 172.21.15.94:0/3580765272 >> v1:172.21.15.94:6789/0 conn(0x7f1b18155a40 legacy=0x7f1b18155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1d61c640 1 -- 172.21.15.94:0/3580765272 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b180c2e90 con 0x7f1b180c1aa0 2024-03-06T07:24:57.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.406+0000 7f1b1ce1b640 1 -- 172.21.15.94:0/3580765272 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047801226 0 0) 0x7f1b180c28d0 con 0x7f1b18155a40 2024-03-06T07:24:57.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.407+0000 7f1b1d61c640 1 --2- 172.21.15.94:0/3580765272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b180c1aa0 0x7f1b181529d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f1b180c2800 tx=0x7f1b0802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e456b7e861497d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.407+0000 7f1b1ce1b640 1 -- 172.21.15.94:0/3580765272 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0802f840 con 0x7f1b180c1aa0 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.407+0000 7f1b1ce1b640 1 -- 172.21.15.94:0/3580765272 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0802f9e0 con 0x7f1b180c1aa0 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.407+0000 7f1b1ce1b640 1 -- 172.21.15.94:0/3580765272 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0802fdb0 con 0x7f1b180c1aa0 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.408+0000 7f1b20081640 1 -- 172.21.15.94:0/3580765272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b180c1aa0 msgr2=0x7f1b181529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.408+0000 7f1b20081640 1 --2- 172.21.15.94:0/3580765272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b180c1aa0 0x7f1b181529d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f1b180c2800 tx=0x7f1b0802ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.408+0000 7f1b20081640 1 -- 172.21.15.94:0/3580765272 shutdown_connections 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.408+0000 7f1b20081640 1 --2- 172.21.15.94:0/3580765272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b180c1aa0 0x7f1b181529d0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.408+0000 7f1b20081640 1 -- 172.21.15.94:0/3580765272 >> 172.21.15.94:0/3580765272 conn(0x7f1b180c6710 msgr2=0x7f1b180c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.408+0000 7f1b20081640 1 -- 172.21.15.94:0/3580765272 shutdown_connections 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.408+0000 7f1b20081640 1 -- 172.21.15.94:0/3580765272 wait complete. 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.409+0000 7f1b20081640 1 Processor -- start 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.409+0000 7f1b20081640 1 -- start start 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b180c1aa0 0x7f1b18149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b1814f790 0x7f1b18147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b18155a40 0x7f1b18148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b180c5e30 con 0x7f1b1814f790 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b180c0d60 con 0x7f1b18155a40 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b180c5fb0 con 0x7f1b180c1aa0 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b1814f790 0x7f1b18147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b1814f790 0x7f1b18147e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43314/0 (socket says 172.21.15.94:43314) 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 -- 172.21.15.94:0/559631870 learned_addr learned my addr 172.21.15.94:0/559631870 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1de1d640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b180c1aa0 0x7f1b18149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1e61e640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b18155a40 0x7f1b18148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 -- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b180c1aa0 msgr2=0x7f1b18149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b180c1aa0 0x7f1b18149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 -- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b18155a40 msgr2=0x7f1b18148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b18155a40 0x7f1b18148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 -- 172.21.15.94:0/559631870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18151780 con 0x7f1b1814f790 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1de1d640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b180c1aa0 0x7f1b18149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b1d61c640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b1814f790 0x7f1b18147e40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f1b080090b0 tx=0x7f1b08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b0e7fc640 1 -- 172.21.15.94:0/559631870 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08047070 con 0x7f1b1814f790 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b0e7fc640 1 -- 172.21.15.94:0/559631870 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b08037cc0 con 0x7f1b1814f790 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b0e7fc640 1 -- 172.21.15.94:0/559631870 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08004270 con 0x7f1b1814f790 2024-03-06T07:24:57.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b180c0340 con 0x7f1b1814f790 2024-03-06T07:24:57.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.410+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b180c0900 con 0x7f1b1814f790 2024-03-06T07:24:57.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.411+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1804f330 con 0x7f1b1814f790 2024-03-06T07:24:57.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.412+0000 7f1b0e7fc640 1 -- 172.21.15.94:0/559631870 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b0804a070 con 0x7f1b1814f790 2024-03-06T07:24:57.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.412+0000 7f1b0e7fc640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1afc06b4d0 0x7f1afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.412+0000 7f1b0e7fc640 1 -- 172.21.15.94:0/559631870 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b080ac490 con 0x7f1b1814f790 2024-03-06T07:24:57.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.412+0000 7f1b1de1d640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1afc06b4d0 0x7f1afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.413+0000 7f1b1de1d640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1afc06b4d0 0x7f1afc06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1b14007fd0 tx=0x7f1b14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.417+0000 7f1b0e7fc640 1 -- 172.21.15.94:0/559631870 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b18151780 con 0x7f1b1814f790 2024-03-06T07:24:57.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.589+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b180c0b40 con 0x7f1b1814f790 2024-03-06T07:24:57.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.590+0000 7f1b0e7fc640 1 -- 172.21.15.94:0/559631870 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+16965 (secure 0 0 0) 0x7f1b08033030 con 0x7f1b1814f790 2024-03-06T07:24:57.592 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:57.592 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":169,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:56.326806+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:54.539968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:55.288852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1afc06b4d0 msgr2=0x7f1afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1afc06b4d0 0x7f1afc06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f1b14007fd0 tx=0x7f1b14009040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b1814f790 msgr2=0x7f1b18147e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b1814f790 0x7f1b18147e40 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f1b080090b0 tx=0x7f1b08004730 comp rx=0 tx=0).stop 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 shutdown_connections 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1afc06b4d0 0x7f1afc06d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1b18155a40 0x7f1b18148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1b1814f790 0x7f1b18147e40 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 --2- 172.21.15.94:0/559631870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1b180c1aa0 0x7f1b18149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 >> 172.21.15.94:0/559631870 conn(0x7f1b180c6710 msgr2=0x7f1b1814d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 shutdown_connections 2024-03-06T07:24:57.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.592+0000 7f1b20081640 1 -- 172.21.15.94:0/559631870 wait complete. 2024-03-06T07:24:57.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:57.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.706+0000 7fd84cf8a640 1 Processor -- start 2024-03-06T07:24:57.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.706+0000 7fd84cf8a640 1 -- start start 2024-03-06T07:24:57.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd84cf8a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd84806afc0 0x7fd8480ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd84cf8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480abe90 0x7fd84815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd84cf8a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84806b440 con 0x7fd84814d350 2024-03-06T07:24:57.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd84cf8a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8480ac750 con 0x7fd84806afc0 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd84cf8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480c1a40 con 0x7fd8480abe90 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd845d9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd84806afc0 0x7fd8480ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd846d9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480abe90 0x7fd84815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd84659c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd84814d350 0x7fd8480acbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37560/0 (socket says 172.21.15.94:37560) 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd845d9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd84806afc0 0x7fd8480ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:36156/0 (socket says 172.21.15.94:36156) 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.707+0000 7fd84659c640 1 -- 172.21.15.94:0/2038839644 learned_addr learned my addr 172.21.15.94:0/2038839644 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd845d9b640 1 -- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480abe90 msgr2=0x7fd84815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd845d9b640 1 --2- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480abe90 0x7fd84815d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd845d9b640 1 -- 172.21.15.94:0/2038839644 >> v1:172.21.15.145:6789/0 conn(0x7fd84814d350 legacy=0x7fd8480acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd845d9b640 1 -- 172.21.15.94:0/2038839644 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8480c1c90 con 0x7fd84806afc0 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd84559a640 1 -- 172.21.15.94:0/2038839644 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396388459 0 0) 0x7fd838001490 con 0x7fd84814d350 2024-03-06T07:24:57.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd846d9d640 1 --2- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480abe90 0x7fd84815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd845d9b640 1 --2- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84806afc0 0x7fd8480ab950 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097a0 tx=0x7fd83c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66554f51bc415345 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd84559a640 1 -- 172.21.15.94:0/2038839644 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0315a0 con 0x7fd84806afc0 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd84559a640 1 -- 172.21.15.94:0/2038839644 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c037070 con 0x7fd84806afc0 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.708+0000 7fd84559a640 1 -- 172.21.15.94:0/2038839644 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c031960 con 0x7fd84806afc0 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 -- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84806afc0 msgr2=0x7fd8480ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84806afc0 0x7fd8480ab950 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097a0 tx=0x7fd83c02b840 comp rx=0 tx=0).stop 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 -- 172.21.15.94:0/2038839644 shutdown_connections 2024-03-06T07:24:57.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480abe90 0x7fd84815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/2038839644 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84806afc0 0x7fd8480ab950 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 -- 172.21.15.94:0/2038839644 >> 172.21.15.94:0/2038839644 conn(0x7fd8480c3b40 msgr2=0x7fd8480c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 -- 172.21.15.94:0/2038839644 shutdown_connections 2024-03-06T07:24:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.709+0000 7fd84cf8a640 1 -- 172.21.15.94:0/2038839644 wait complete. 2024-03-06T07:24:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.710+0000 7fd84cf8a640 1 Processor -- start 2024-03-06T07:24:57.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.710+0000 7fd84cf8a640 1 -- start start 2024-03-06T07:24:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84cf8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd84806afc0 0x7fd848146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84cf8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8480abe90 0x7fd848156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84cf8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84814d350 0x7fd848156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84cf8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84806ea70 con 0x7fd8480abe90 2024-03-06T07:24:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84cf8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd848153c00 con 0x7fd84814d350 2024-03-06T07:24:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84cf8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481464a0 con 0x7fd84806afc0 2024-03-06T07:24:57.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd845d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8480abe90 0x7fd848156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd846d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84814d350 0x7fd848156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84659c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd84806afc0 0x7fd848146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd846d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84814d350 0x7fd848156eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:36172/0 (socket says 172.21.15.94:36172) 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84659c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd84806afc0 0x7fd848146780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:47398/0 (socket says 172.21.15.94:47398) 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd845d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8480abe90 0x7fd848156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43322/0 (socket says 172.21.15.94:43322) 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd846d9d640 1 -- 172.21.15.94:0/3430425179 learned_addr learned my addr 172.21.15.94:0/3430425179 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84659c640 1 -- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84814d350 msgr2=0x7fd848156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84659c640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84814d350 0x7fd848156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84659c640 1 -- 172.21.15.94:0/3430425179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8480abe90 msgr2=0x7fd848156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84659c640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8480abe90 0x7fd848156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd84659c640 1 -- 172.21.15.94:0/3430425179 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848154e30 con 0x7fd84806afc0 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.711+0000 7fd846d9d640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84814d350 0x7fd848156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:57.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.712+0000 7fd84659c640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd84806afc0 0x7fd848146780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c780 tx=0x7fd83800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.712+0000 7fd836ffd640 1 -- 172.21.15.94:0/3430425179 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838018070 con 0x7fd84806afc0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.712+0000 7fd836ffd640 1 -- 172.21.15.94:0/3430425179 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83800ce30 con 0x7fd84806afc0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.712+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd848155110 con 0x7fd84806afc0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.712+0000 7fd836ffd640 1 -- 172.21.15.94:0/3430425179 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83800a860 con 0x7fd84806afc0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.712+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8480c6dc0 con 0x7fd84806afc0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.713+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8480c38c0 con 0x7fd84806afc0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.713+0000 7fd836ffd640 1 -- 172.21.15.94:0/3430425179 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd83801b070 con 0x7fd84806afc0 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.714+0000 7fd836ffd640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82806b500 0x7fd82806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.714+0000 7fd836ffd640 1 -- 172.21.15.94:0/3430425179 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd8380897c0 con 0x7fd84806afc0 2024-03-06T07:24:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.714+0000 7fd845d9b640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82806b500 0x7fd82806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.714+0000 7fd845d9b640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82806b500 0x7fd82806d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd83c036000 tx=0x7fd83c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.718+0000 7fd836ffd640 1 -- 172.21.15.94:0/3430425179 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8480c38c0 con 0x7fd84806afc0 2024-03-06T07:24:57.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:57.925+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd84804f330 con 0x7fd84806afc0 2024-03-06T07:24:58.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.223+0000 7fd836ffd640 1 -- 172.21.15.94:0/3430425179 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v158) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd83805ce20 con 0x7fd84806afc0 2024-03-06T07:24:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82806b500 msgr2=0x7fd82806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82806b500 0x7fd82806d9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd83c036000 tx=0x7fd83c03d040 comp rx=0 tx=0).stop 2024-03-06T07:24:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd84806afc0 msgr2=0x7fd848146780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd84806afc0 0x7fd848146780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c780 tx=0x7fd83800cc40 comp rx=0 tx=0).stop 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 shutdown_connections 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd82806b500 0x7fd82806d9c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd84814d350 0x7fd848156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8480abe90 0x7fd848156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 --2- 172.21.15.94:0/3430425179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd84806afc0 0x7fd848146780 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 >> 172.21.15.94:0/3430425179 conn(0x7fd8480c3b40 msgr2=0x7fd8480c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 shutdown_connections 2024-03-06T07:24:58.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.225+0000 7fd84cf8a640 1 -- 172.21.15.94:0/3430425179 wait complete. 2024-03-06T07:24:58.238 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:58.238 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:58.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.338+0000 7fe644c81640 1 Processor -- start 2024-03-06T07:24:58.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe644c81640 1 -- start start 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe644c81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe640155c30 0x7fe64006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe644c81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6400c5380 0x7fe6400c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe644c81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6400b0450 con 0x7fe6400b0070 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe644c81640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64006cfb0 con 0x7fe640155c30 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe644c81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64006c020 con 0x7fe6400c5380 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe637fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe640155c30 0x7fe64006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe637fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe640155c30 0x7fe64006ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43330/0 (socket says 172.21.15.94:43330) 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe637fff640 1 -- 172.21.15.94:0/2911216897 learned_addr learned my addr 172.21.15.94:0/2911216897 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.339+0000 7fe63ed9d640 1 --2- 172.21.15.94:0/2911216897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6400c5380 0x7fe6400c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.340+0000 7fe637fff640 1 -- 172.21.15.94:0/2911216897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6400c5380 msgr2=0x7fe6400c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:58.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.340+0000 7fe637fff640 1 --2- 172.21.15.94:0/2911216897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6400c5380 0x7fe6400c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.340+0000 7fe637fff640 1 -- 172.21.15.94:0/2911216897 >> v1:172.21.15.145:6789/0 conn(0x7fe6400b0070 legacy=0x7fe6400c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.340+0000 7fe637fff640 1 -- 172.21.15.94:0/2911216897 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6401560e0 con 0x7fe640155c30 2024-03-06T07:24:58.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.340+0000 7fe637fff640 1 --2- 172.21.15.94:0/2911216897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe640155c30 0x7fe64006ca70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89c43b533d4f20e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe63dd9b640 1 -- 172.21.15.94:0/2911216897 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628035020 con 0x7fe640155c30 2024-03-06T07:24:58.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe63dd9b640 1 -- 172.21.15.94:0/2911216897 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628031640 con 0x7fe640155c30 2024-03-06T07:24:58.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe63dd9b640 1 -- 172.21.15.94:0/2911216897 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62802d040 con 0x7fe640155c30 2024-03-06T07:24:58.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 -- 172.21.15.94:0/2911216897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe640155c30 msgr2=0x7fe64006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 --2- 172.21.15.94:0/2911216897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe640155c30 0x7fe64006ca70 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 -- 172.21.15.94:0/2911216897 shutdown_connections 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 --2- 172.21.15.94:0/2911216897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6400c5380 0x7fe6400c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 --2- 172.21.15.94:0/2911216897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe640155c30 0x7fe64006ca70 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 -- 172.21.15.94:0/2911216897 >> 172.21.15.94:0/2911216897 conn(0x7fe6400ac590 msgr2=0x7fe6400ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 -- 172.21.15.94:0/2911216897 shutdown_connections 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.341+0000 7fe644c81640 1 -- 172.21.15.94:0/2911216897 wait complete. 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.342+0000 7fe644c81640 1 Processor -- start 2024-03-06T07:24:58.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.342+0000 7fe644c81640 1 -- start start 2024-03-06T07:24:58.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe644c81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6400b0070 0x7fe6401549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe644c81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6400c5380 0x7fe64014ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe644c81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe640155c30 0x7fe64014bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe644c81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64006c270 con 0x7fe6400b0070 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe644c81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6400c7ad0 con 0x7fe6400c5380 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe644c81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64006c550 con 0x7fe640155c30 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6400b0070 0x7fe6401549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63ed9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe640155c30 0x7fe64014bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6400b0070 0x7fe6401549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:43340/0 (socket says 172.21.15.94:43340) 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe637fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6400c5380 0x7fe64014ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63e59c640 1 -- 172.21.15.94:0/3991818523 learned_addr learned my addr 172.21.15.94:0/3991818523 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63e59c640 1 -- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe640155c30 msgr2=0x7fe64014bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63e59c640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe640155c30 0x7fe64014bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63e59c640 1 -- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6400c5380 msgr2=0x7fe64014ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:58.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.343+0000 7fe63e59c640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6400c5380 0x7fe64014ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.344+0000 7fe63e59c640 1 -- 172.21.15.94:0/3991818523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe640155620 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.344+0000 7fe63e59c640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6400b0070 0x7fe6401549b0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c3c0 tx=0x7fe63000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.344+0000 7fe636ffd640 1 -- 172.21.15.94:0/3991818523 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6300052d0 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.344+0000 7fe636ffd640 1 -- 172.21.15.94:0/3991818523 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe630005490 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.344+0000 7fe636ffd640 1 -- 172.21.15.94:0/3991818523 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630007260 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.344+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe640158110 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.344+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6401ec030 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.345+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6400c85d0 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.345+0000 7fe636ffd640 1 -- 172.21.15.94:0/3991818523 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe630007400 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.346+0000 7fe636ffd640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62006b4d0 0x7fe62006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.346+0000 7fe636ffd640 1 -- 172.21.15.94:0/3991818523 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe6300896f0 con 0x7fe6400b0070 2024-03-06T07:24:58.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.346+0000 7fe637fff640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62006b4d0 0x7fe62006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:58.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.346+0000 7fe637fff640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62006b4d0 0x7fe62006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe628036000 tx=0x7fe62803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:58.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.350+0000 7fe636ffd640 1 -- 172.21.15.94:0/3991818523 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6400c85d0 con 0x7fe6400b0070 2024-03-06T07:24:58.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:58.557+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe64006d860 con 0x7fe6400b0070 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.227+0000 7fe636ffd640 1 -- 172.21.15.94:0/3991818523 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v159) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe640155620 con 0x7fe6400b0070 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62006b4d0 msgr2=0x7fe62006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62006b4d0 0x7fe62006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe628036000 tx=0x7fe62803c040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6400b0070 msgr2=0x7fe6401549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6400b0070 0x7fe6401549b0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c3c0 tx=0x7fe63000c880 comp rx=0 tx=0).stop 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 shutdown_connections 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe62006b4d0 0x7fe62006d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe640155c30 0x7fe64014bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6400c5380 0x7fe64014ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 --2- 172.21.15.94:0/3991818523 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6400b0070 0x7fe6401549b0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.229+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 >> 172.21.15.94:0/3991818523 conn(0x7fe6400ac590 msgr2=0x7fe6400ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.230+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 shutdown_connections 2024-03-06T07:24:59.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.230+0000 7fe644c81640 1 -- 172.21.15.94:0/3991818523 wait complete. 2024-03-06T07:24:59.244 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:59.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.346+0000 7ff5f8aaf640 1 Processor -- start 2024-03-06T07:24:59.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.346+0000 7ff5f8aaf640 1 -- start start 2024-03-06T07:24:59.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f8aaf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f41527b0 0x7ff5f4159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f8aaf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5f40c7180 0x7ff5f40c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f8aaf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40c7c60 con 0x7ff5f4155b20 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f8aaf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f4159640 con 0x7ff5f40c7180 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f8aaf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40c8df0 con 0x7ff5f41527b0 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f259c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f41527b0 0x7ff5f4159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f1d9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5f40c7180 0x7ff5f40c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f259c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f41527b0 0x7ff5f4159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:48848/0 (socket says 172.21.15.94:48848) 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f2d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff5f4155b20 0x7ff5f40c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:48736/0 (socket says 172.21.15.94:48736) 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f1d9b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5f40c7180 0x7ff5f40c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:39968/0 (socket says 172.21.15.94:39968) 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.347+0000 7ff5f259c640 1 -- 172.21.15.94:0/402143718 learned_addr learned my addr 172.21.15.94:0/402143718 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.348+0000 7ff5f1d9b640 1 -- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f41527b0 msgr2=0x7ff5f4159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.348+0000 7ff5f1d9b640 1 --2- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f41527b0 0x7ff5f4159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.348+0000 7ff5f1d9b640 1 -- 172.21.15.94:0/402143718 >> v1:172.21.15.145:6789/0 conn(0x7ff5f4155b20 legacy=0x7ff5f40c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.348+0000 7ff5f159a640 1 -- 172.21.15.94:0/402143718 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760462336 0 0) 0x7ff5ec001110 con 0x7ff5f4155b20 2024-03-06T07:24:59.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.348+0000 7ff5f1d9b640 1 -- 172.21.15.94:0/402143718 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40c8060 con 0x7ff5f40c7180 2024-03-06T07:24:59.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.348+0000 7ff5f259c640 1 --2- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f41527b0 0x7ff5f4159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:59.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.348+0000 7ff5f1d9b640 1 --2- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f40c7180 0x7ff5f40c88b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80097a0 tx=0x7ff5e802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b40fd527a6a999ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f159a640 1 -- 172.21.15.94:0/402143718 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e80315a0 con 0x7ff5f40c7180 2024-03-06T07:24:59.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f159a640 1 -- 172.21.15.94:0/402143718 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e8037070 con 0x7ff5f40c7180 2024-03-06T07:24:59.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f159a640 1 -- 172.21.15.94:0/402143718 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8031960 con 0x7ff5f40c7180 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f40c7180 msgr2=0x7ff5f40c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f40c7180 0x7ff5f40c88b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80097a0 tx=0x7ff5e802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/402143718 shutdown_connections 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f40c7180 0x7ff5f40c88b0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/402143718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f41527b0 0x7ff5f4159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/402143718 >> 172.21.15.94:0/402143718 conn(0x7ff5f40ac560 msgr2=0x7ff5f40ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/402143718 shutdown_connections 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.349+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/402143718 wait complete. 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 Processor -- start 2024-03-06T07:24:59.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 -- start start 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f40c7180 0x7ff5f41eb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5f41527b0 0x7ff5f40c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f4155b20 0x7ff5f4151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f4155320 con 0x7ff5f41527b0 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40c7980 con 0x7ff5f4155b20 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f8aaf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f4155600 con 0x7ff5f40c7180 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5f41527b0 0x7ff5f40c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f259c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f40c7180 0x7ff5f41eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5f41527b0 0x7ff5f40c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58990/0 (socket says 172.21.15.94:58990) 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f1d9b640 1 -- 172.21.15.94:0/636600147 learned_addr learned my addr 172.21.15.94:0/636600147 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.351+0000 7ff5f2d9d640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f4155b20 0x7ff5f4151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f1d9b640 1 -- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f40c7180 msgr2=0x7ff5f41eb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f1d9b640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f40c7180 0x7ff5f41eb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f1d9b640 1 -- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f4155b20 msgr2=0x7ff5f4151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f1d9b640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f4155b20 0x7ff5f4151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f1d9b640 1 -- 172.21.15.94:0/636600147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40c1d30 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f1d9b640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5f41527b0 0x7ff5f40c46b0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80097a0 tx=0x7ff5e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5e2ffd640 1 -- 172.21.15.94:0/636600147 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8004230 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5e2ffd640 1 -- 172.21.15.94:0/636600147 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e80043d0 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f41eed20 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5e2ffd640 1 -- 172.21.15.94:0/636600147 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8003850 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.352+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f41eb060 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.353+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f404f330 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.353+0000 7ff5e2ffd640 1 -- 172.21.15.94:0/636600147 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5e80039f0 con 0x7ff5f41527b0 2024-03-06T07:24:59.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.354+0000 7ff5e2ffd640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5d406b4d0 0x7ff5d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.354+0000 7ff5e2ffd640 1 -- 172.21.15.94:0/636600147 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff5e802f030 con 0x7ff5f41527b0 2024-03-06T07:24:59.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.354+0000 7ff5f259c640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5d406b4d0 0x7ff5d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.354+0000 7ff5f259c640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5d406b4d0 0x7ff5d406d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009c90 tx=0x7ff5e400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.358+0000 7ff5e2ffd640 1 -- 172.21.15.94:0/636600147 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5e8041070 con 0x7ff5f41527b0 2024-03-06T07:24:59.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.564+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5f406ce00 con 0x7ff5f41527b0 2024-03-06T07:24:59.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.564+0000 7ff5e2ffd640 1 -- 172.21.15.94:0/636600147 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4998 (secure 0 0 0) 0x7ff5e807aaf0 con 0x7ff5f41527b0 2024-03-06T07:24:59.566 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:59.566 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:24:58.558540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37487,"qdb_cluster":[37487]},"id":13}]} 2024-03-06T07:24:59.566 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 159 2024-03-06T07:24:59.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5d406b4d0 msgr2=0x7ff5d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5d406b4d0 0x7ff5d406d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009c90 tx=0x7ff5e400b040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5f41527b0 msgr2=0x7ff5f40c46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5f41527b0 0x7ff5f40c46b0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80097a0 tx=0x7ff5e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 shutdown_connections 2024-03-06T07:24:59.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5d406b4d0 0x7ff5d406d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5f4155b20 0x7ff5f4151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5f41527b0 0x7ff5f40c46b0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 --2- 172.21.15.94:0/636600147 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5f40c7180 0x7ff5f41eb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 >> 172.21.15.94:0/636600147 conn(0x7ff5f40ac560 msgr2=0x7ff5f414aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 shutdown_connections 2024-03-06T07:24:59.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.566+0000 7ff5f8aaf640 1 -- 172.21.15.94:0/636600147 wait complete. 2024-03-06T07:24:59.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:59.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.677+0000 7f81ee1c9640 1 Processor -- start 2024-03-06T07:24:59.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.677+0000 7f81ee1c9640 1 -- start start 2024-03-06T07:24:59.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.677+0000 7f81ee1c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f81e814f790 0x7f81e81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.677+0000 7f81ee1c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e80c28d0 con 0x7f81e8155a40 2024-03-06T07:24:59.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.677+0000 7f81ee1c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e80c2bb0 con 0x7f81e80c1aa0 2024-03-06T07:24:59.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.677+0000 7f81ee1c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e80c2680 con 0x7f81e814f790 2024-03-06T07:24:59.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f81e814f790 0x7f81e81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f81e814f790 0x7f81e81529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58996/0 (socket says 172.21.15.94:58996) 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f81e8155a40 0x7f81e8155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33108/0 (socket says 172.21.15.94:33108) 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e77fe640 1 -- 172.21.15.94:0/3223577095 learned_addr learned my addr 172.21.15.94:0/3223577095 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e77fe640 1 -- 172.21.15.94:0/3223577095 >> v1:172.21.15.145:6789/0 conn(0x7f81e80c1aa0 legacy=0x7f81e80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e77fe640 1 -- 172.21.15.94:0/3223577095 >> v1:172.21.15.94:6789/0 conn(0x7f81e8155a40 legacy=0x7f81e8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e67fc640 1 -- 172.21.15.94:0/3223577095 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416115472 0 0) 0x7f81e80c28d0 con 0x7f81e8155a40 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.678+0000 7f81e77fe640 1 -- 172.21.15.94:0/3223577095 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e80c2e90 con 0x7f81e814f790 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81e77fe640 1 --2- 172.21.15.94:0/3223577095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e814f790 0x7f81e81529d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f81e80c2800 tx=0x7f81d802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b821af2077fa2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81e67fc640 1 -- 172.21.15.94:0/3223577095 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d802f840 con 0x7f81e814f790 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81e67fc640 1 -- 172.21.15.94:0/3223577095 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d802f9e0 con 0x7f81e814f790 2024-03-06T07:24:59.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81e67fc640 1 -- 172.21.15.94:0/3223577095 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d802fdb0 con 0x7f81e814f790 2024-03-06T07:24:59.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81ee1c9640 1 -- 172.21.15.94:0/3223577095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e814f790 msgr2=0x7f81e81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/3223577095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e814f790 0x7f81e81529d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f81e80c2800 tx=0x7f81d802ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:59.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81ee1c9640 1 -- 172.21.15.94:0/3223577095 shutdown_connections 2024-03-06T07:24:59.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/3223577095 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e814f790 0x7f81e81529d0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81ee1c9640 1 -- 172.21.15.94:0/3223577095 >> 172.21.15.94:0/3223577095 conn(0x7f81e80c6710 msgr2=0x7f81e80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81ee1c9640 1 -- 172.21.15.94:0/3223577095 shutdown_connections 2024-03-06T07:24:59.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.679+0000 7f81ee1c9640 1 -- 172.21.15.94:0/3223577095 wait complete. 2024-03-06T07:24:59.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.680+0000 7f81ee1c9640 1 Processor -- start 2024-03-06T07:24:59.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81ee1c9640 1 -- start start 2024-03-06T07:24:59.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81ee1c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e80c1aa0 0x7f81e81511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81ee1c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81e814f790 0x7f81e814d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81ee1c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81e8155a40 0x7f81e814da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81ee1c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80c5e30 con 0x7f81e80c1aa0 2024-03-06T07:24:59.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81ee1c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80c0d60 con 0x7f81e8155a40 2024-03-06T07:24:59.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81ee1c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80c5fb0 con 0x7f81e814f790 2024-03-06T07:24:59.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81e77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e80c1aa0 0x7f81e81511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81e77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e80c1aa0 0x7f81e81511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59010/0 (socket says 172.21.15.94:59010) 2024-03-06T07:24:59.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81e77fe640 1 -- 172.21.15.94:0/4148833212 learned_addr learned my addr 172.21.15.94:0/4148833212 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:59.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81e7fff640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81e8155a40 0x7f81e814da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81e6ffd640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81e814f790 0x7f81e814d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.681+0000 7f81e77fe640 1 -- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81e814f790 msgr2=0x7f81e814d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81e77fe640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81e814f790 0x7f81e814d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81e77fe640 1 -- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81e8155a40 msgr2=0x7f81e814da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81e77fe640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81e8155a40 0x7f81e814da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81e77fe640 1 -- 172.21.15.94:0/4148833212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e81db950 con 0x7f81e80c1aa0 2024-03-06T07:24:59.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81e77fe640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e80c1aa0 0x7f81e81511b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81cffff640 1 -- 172.21.15.94:0/4148833212 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8041400 con 0x7f81e80c1aa0 2024-03-06T07:24:59.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81cffff640 1 -- 172.21.15.94:0/4148833212 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d8046070 con 0x7f81e80c1aa0 2024-03-06T07:24:59.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81cffff640 1 -- 172.21.15.94:0/4148833212 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8041800 con 0x7f81e80c1aa0 2024-03-06T07:24:59.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e814df80 con 0x7f81e80c1aa0 2024-03-06T07:24:59.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.682+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e81ed140 con 0x7f81e80c1aa0 2024-03-06T07:24:59.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.683+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e804f330 con 0x7f81e80c1aa0 2024-03-06T07:24:59.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.683+0000 7f81cffff640 1 -- 172.21.15.94:0/4148833212 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81d8040070 con 0x7f81e80c1aa0 2024-03-06T07:24:59.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.684+0000 7f81cffff640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f81d406b500 0x7f81d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.684+0000 7f81cffff640 1 -- 172.21.15.94:0/4148833212 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f81d80ac410 con 0x7f81e80c1aa0 2024-03-06T07:24:59.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.684+0000 7f81e6ffd640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f81d406b500 0x7f81d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.684+0000 7f81e6ffd640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f81d406b500 0x7f81d406d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f81e81553a0 tx=0x7f81dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.688+0000 7f81cffff640 1 -- 172.21.15.94:0/4148833212 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81d804f020 con 0x7f81e80c1aa0 2024-03-06T07:24:59.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.864+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81e80c1f40 con 0x7f81e80c1aa0 2024-03-06T07:24:59.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.865+0000 7f81cffff640 1 -- 172.21.15.94:0/4148833212 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+16952 (secure 0 0 0) 0x7f81d8033030 con 0x7f81e80c1aa0 2024-03-06T07:24:59.867 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:24:59.867 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":170,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:57.969058+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:54.539968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:55.288852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f81d406b500 msgr2=0x7f81d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f81d406b500 0x7f81d406d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f81e81553a0 tx=0x7f81dc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e80c1aa0 msgr2=0x7f81e81511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e80c1aa0 0x7f81e81511b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d8004730 comp rx=0 tx=0).stop 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 shutdown_connections 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f81d406b500 0x7f81d406d9c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f81e8155a40 0x7f81e814da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f81e814f790 0x7f81e814d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 --2- 172.21.15.94:0/4148833212 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81e80c1aa0 0x7f81e81511b0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 >> 172.21.15.94:0/4148833212 conn(0x7f81e80c6710 msgr2=0x7f81e806d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 shutdown_connections 2024-03-06T07:24:59.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.867+0000 7f81ee1c9640 1 -- 172.21.15.94:0/4148833212 wait complete. 2024-03-06T07:24:59.880 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:59.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:59.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.978+0000 7fb21afe9640 1 Processor -- start 2024-03-06T07:24:59.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.978+0000 7fb21afe9640 1 -- start start 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb21afe9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140c3370 0x7fb214155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb21afe9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb2140c3750 0x7fb214156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb21afe9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2140c5490 con 0x7fb2140a5e60 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb21afe9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2140c5770 con 0x7fb2140c3750 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb21afe9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140c5a50 con 0x7fb2140c3370 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb213fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb2140c3750 0x7fb214156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb218d85640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140c3370 0x7fb214155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb218d85640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140c3370 0x7fb214155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:48866/0 (socket says 172.21.15.94:48866) 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb213fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb2140c3750 0x7fb214156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:39984/0 (socket says 172.21.15.94:39984) 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb218d85640 1 -- 172.21.15.94:0/2193964141 learned_addr learned my addr 172.21.15.94:0/2193964141 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb219586640 1 --1- 172.21.15.94:0/2193964141 >> v1:172.21.15.145:6789/0 conn(0x7fb2140a5e60 0x7fb2140c1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:48758/0 (socket says 172.21.15.94:48758) 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb218d85640 1 -- 172.21.15.94:0/2193964141 >> v2:172.21.15.145:3300/0 conn(0x7fb2140c3750 msgr2=0x7fb214156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb218d85640 1 --2- 172.21.15.94:0/2193964141 >> v2:172.21.15.145:3300/0 conn(0x7fb2140c3750 0x7fb214156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb218d85640 1 -- 172.21.15.94:0/2193964141 >> v1:172.21.15.145:6789/0 conn(0x7fb2140a5e60 legacy=0x7fb2140c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb218d85640 1 -- 172.21.15.94:0/2193964141 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2140c5d30 con 0x7fb2140c3370 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.979+0000 7fb213fff640 1 --2- 172.21.15.94:0/2193964141 >> v2:172.21.15.145:3300/0 conn(0x7fb2140c3750 0x7fb214156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.980+0000 7fb218d85640 1 --2- 172.21.15.94:0/2193964141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140c3370 0x7fb214155ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb208009870 tx=0x7fb20802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cefb0a244adfedf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.980+0000 7fb2137fe640 1 -- 172.21.15.94:0/2193964141 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2080317c0 con 0x7fb2140c3370 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.980+0000 7fb2137fe640 1 -- 172.21.15.94:0/2193964141 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208037070 con 0x7fb2140c3370 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.980+0000 7fb2137fe640 1 -- 172.21.15.94:0/2193964141 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208031b80 con 0x7fb2140c3370 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 -- 172.21.15.94:0/2193964141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140c3370 msgr2=0x7fb214155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2193964141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140c3370 0x7fb214155ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb208009870 tx=0x7fb20802b860 comp rx=0 tx=0).stop 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 -- 172.21.15.94:0/2193964141 shutdown_connections 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2193964141 >> v2:172.21.15.145:3300/0 conn(0x7fb2140c3750 0x7fb214156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2193964141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140c3370 0x7fb214155ad0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 -- 172.21.15.94:0/2193964141 >> 172.21.15.94:0/2193964141 conn(0x7fb2140c6710 msgr2=0x7fb2140c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 -- 172.21.15.94:0/2193964141 shutdown_connections 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.981+0000 7fb21afe9640 1 -- 172.21.15.94:0/2193964141 wait complete. 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.982+0000 7fb21afe9640 1 Processor -- start 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.982+0000 7fb21afe9640 1 -- start start 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140a5e60 0x7fb2140c2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2140c3370 0x7fb214150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2140c3750 0x7fb214151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214159910 con 0x7fb2140c3370 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214159790 con 0x7fb2140c3750 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140c4ed0 con 0x7fb2140a5e60 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2140c3370 0x7fb214150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2140c3370 0x7fb214150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59014/0 (socket says 172.21.15.94:59014) 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 -- 172.21.15.94:0/2385902394 learned_addr learned my addr 172.21.15.94:0/2385902394 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb218d85640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140a5e60 0x7fb2140c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb219586640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2140c3750 0x7fb214151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 -- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140a5e60 msgr2=0x7fb2140c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140a5e60 0x7fb2140c2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 -- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2140c3750 msgr2=0x7fb214151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2140c3750 0x7fb214151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 -- 172.21.15.94:0/2385902394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2141edff0 con 0x7fb2140c3370 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb213fff640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2140c3370 0x7fb214150b20 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fb20000e990 tx=0x7fb20000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb2117fa640 1 -- 172.21.15.94:0/2385902394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb200017070 con 0x7fb2140c3370 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb2117fa640 1 -- 172.21.15.94:0/2385902394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20000c9e0 con 0x7fb2140c3370 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb2117fa640 1 -- 172.21.15.94:0/2385902394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20000cdb0 con 0x7fb2140c3370 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2141ee2d0 con 0x7fb2140c3370 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.983+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2141ee890 con 0x7fb2140c3370 2024-03-06T07:24:59.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.984+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21404f330 con 0x7fb2140c3370 2024-03-06T07:24:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.985+0000 7fb2117fa640 1 -- 172.21.15.94:0/2385902394 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb20001a070 con 0x7fb2140c3370 2024-03-06T07:24:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.985+0000 7fb2117fa640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb1f806b4d0 0x7fb1f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.985+0000 7fb2117fa640 1 -- 172.21.15.94:0/2385902394 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb20008a500 con 0x7fb2140c3370 2024-03-06T07:24:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.986+0000 7fb218d85640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb1f806b4d0 0x7fb1f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.986+0000 7fb218d85640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb1f806b4d0 0x7fb1f806d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb208036000 tx=0x7fb20803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:24:59.990+0000 7fb2117fa640 1 -- 172.21.15.94:0/2385902394 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb200058bb0 con 0x7fb2140c3370 2024-03-06T07:25:00.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.157+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb214068450 con 0x7fb1f806b4d0 2024-03-06T07:25:00.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.163+0000 7fb2117fa640 1 -- 172.21.15.94:0/2385902394 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252499 (secure 0 0 0) 0x7fb214068450 con 0x7fb1f806b4d0 2024-03-06T07:25:00.166 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:00.170 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb1f806b4d0 msgr2=0x7fb1f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb1f806b4d0 0x7fb1f806d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb208036000 tx=0x7fb20803d040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2140c3370 msgr2=0x7fb214150b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2140c3370 0x7fb214150b20 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fb20000e990 tx=0x7fb20000ee50 comp rx=0 tx=0).stop 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 shutdown_connections 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb1f806b4d0 0x7fb1f806d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2140c3750 0x7fb214151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb2140c3370 0x7fb214150b20 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 --2- 172.21.15.94:0/2385902394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2140a5e60 0x7fb2140c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 >> 172.21.15.94:0/2385902394 conn(0x7fb2140c6710 msgr2=0x7fb214154fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 shutdown_connections 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.166+0000 7fb21afe9640 1 -- 172.21.15.94:0/2385902394 wait complete. 2024-03-06T07:25:00.179 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-03-06T07:24:58.723054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266252,"kb_used_data":16792,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708468,"statfs":{"total":773094113280,"available":772821471232,"internally_reserved":0,"allocated":17195008,"data_stored":9467009,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000908"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004303+0000","last_change":"2024-03-06T07:24:56.997658+0000","last_active":"2024-03-06T07:24:57.004303+0000","last_peered":"2024-03-06T07:24:57.004303+0000","last_clean":"2024-03-06T07:24:57.004303+0000","last_became_active":"2024-03-06T07:24:56.996963+0000","last_became_peered":"2024-03-06T07:24:56.996963+0000","last_unstale":"2024-03-06T07:24:57.004303+0000","last_undegraded":"2024-03-06T07:24:57.004303+0000","last_fullsized":"2024-03-06T07:24:57.004303+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:05.822744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996018+0000","last_change":"2024-03-06T07:24:56.994300+0000","last_active":"2024-03-06T07:24:56.996018+0000","last_peered":"2024-03-06T07:24:56.996018+0000","last_clean":"2024-03-06T07:24:56.996018+0000","last_became_active":"2024-03-06T07:24:56.994019+0000","last_became_peered":"2024-03-06T07:24:56.994019+0000","last_unstale":"2024-03-06T07:24:56.996018+0000","last_undegraded":"2024-03-06T07:24:56.996018+0000","last_fullsized":"2024-03-06T07:24:56.996018+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:39.647212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996030+0000","last_change":"2024-03-06T07:24:56.994367+0000","last_active":"2024-03-06T07:24:56.996030+0000","last_peered":"2024-03-06T07:24:56.996030+0000","last_clean":"2024-03-06T07:24:56.996030+0000","last_became_active":"2024-03-06T07:24:56.994110+0000","last_became_peered":"2024-03-06T07:24:56.994110+0000","last_unstale":"2024-03-06T07:24:56.996030+0000","last_undegraded":"2024-03-06T07:24:56.996030+0000","last_fullsized":"2024-03-06T07:24:56.996030+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:22.730963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996514+0000","last_change":"2024-03-06T07:24:56.988648+0000","last_active":"2024-03-06T07:24:56.996514+0000","last_peered":"2024-03-06T07:24:56.996514+0000","last_clean":"2024-03-06T07:24:56.996514+0000","last_became_active":"2024-03-06T07:24:56.988320+0000","last_became_peered":"2024-03-06T07:24:56.988320+0000","last_unstale":"2024-03-06T07:24:56.996514+0000","last_undegraded":"2024-03-06T07:24:56.996514+0000","last_fullsized":"2024-03-06T07:24:56.996514+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:30.772328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986603+0000","last_change":"2024-03-06T07:24:56.984517+0000","last_active":"2024-03-06T07:24:56.986603+0000","last_peered":"2024-03-06T07:24:56.986603+0000","last_clean":"2024-03-06T07:24:56.986603+0000","last_became_active":"2024-03-06T07:24:56.984245+0000","last_became_peered":"2024-03-06T07:24:56.984245+0000","last_unstale":"2024-03-06T07:24:56.986603+0000","last_undegraded":"2024-03-06T07:24:56.986603+0000","last_fullsized":"2024-03-06T07:24:56.986603+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:25.308202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995970+0000","last_change":"2024-03-06T07:24:56.988686+0000","last_active":"2024-03-06T07:24:56.995970+0000","last_peered":"2024-03-06T07:24:56.995970+0000","last_clean":"2024-03-06T07:24:56.995970+0000","last_became_active":"2024-03-06T07:24:56.988398+0000","last_became_peered":"2024-03-06T07:24:56.988398+0000","last_unstale":"2024-03-06T07:24:56.995970+0000","last_undegraded":"2024-03-06T07:24:56.995970+0000","last_fullsized":"2024-03-06T07:24:56.995970+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:42.942153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984836+0000","last_change":"2024-03-06T07:24:56.983905+0000","last_active":"2024-03-06T07:24:56.984836+0000","last_peered":"2024-03-06T07:24:56.984836+0000","last_clean":"2024-03-06T07:24:56.984836+0000","last_became_active":"2024-03-06T07:24:56.983619+0000","last_became_peered":"2024-03-06T07:24:56.983619+0000","last_unstale":"2024-03-06T07:24:56.984836+0000","last_undegraded":"2024-03-06T07:24:56.984836+0000","last_fullsized":"2024-03-06T07:24:56.984836+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:59.082721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.994107+0000","last_change":"2024-03-06T07:24:56.991468+0000","last_active":"2024-03-06T07:24:56.994107+0000","last_peered":"2024-03-06T07:24:56.994107+0000","last_clean":"2024-03-06T07:24:56.994107+0000","last_became_active":"2024-03-06T07:24:56.990805+0000","last_became_peered":"2024-03-06T07:24:56.990805+0000","last_unstale":"2024-03-06T07:24:56.994107+0000","last_undegraded":"2024-03-06T07:24:56.994107+0000","last_fullsized":"2024-03-06T07:24:56.994107+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:54.782835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995803+0000","last_change":"2024-03-06T07:24:56.994841+0000","last_active":"2024-03-06T07:24:56.995803+0000","last_peered":"2024-03-06T07:24:56.995803+0000","last_clean":"2024-03-06T07:24:56.995803+0000","last_became_active":"2024-03-06T07:24:56.994540+0000","last_became_peered":"2024-03-06T07:24:56.994540+0000","last_unstale":"2024-03-06T07:24:56.995803+0000","last_undegraded":"2024-03-06T07:24:56.995803+0000","last_fullsized":"2024-03-06T07:24:56.995803+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:21.626315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989239+0000","last_change":"2024-03-06T07:24:56.981436+0000","last_active":"2024-03-06T07:24:56.989239+0000","last_peered":"2024-03-06T07:24:56.989239+0000","last_clean":"2024-03-06T07:24:56.989239+0000","last_became_active":"2024-03-06T07:24:56.981153+0000","last_became_peered":"2024-03-06T07:24:56.981153+0000","last_unstale":"2024-03-06T07:24:56.989239+0000","last_undegraded":"2024-03-06T07:24:56.989239+0000","last_fullsized":"2024-03-06T07:24:56.989239+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:44.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.998326+0000","last_change":"2024-03-06T07:24:56.996355+0000","last_active":"2024-03-06T07:24:56.998326+0000","last_peered":"2024-03-06T07:24:56.998326+0000","last_clean":"2024-03-06T07:24:56.998326+0000","last_became_active":"2024-03-06T07:24:56.996096+0000","last_became_peered":"2024-03-06T07:24:56.996096+0000","last_unstale":"2024-03-06T07:24:56.998326+0000","last_undegraded":"2024-03-06T07:24:56.998326+0000","last_fullsized":"2024-03-06T07:24:56.998326+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:27.073130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984328+0000","last_change":"2024-03-06T07:24:56.979974+0000","last_active":"2024-03-06T07:24:56.984328+0000","last_peered":"2024-03-06T07:24:56.984328+0000","last_clean":"2024-03-06T07:24:56.984328+0000","last_became_active":"2024-03-06T07:24:56.979695+0000","last_became_peered":"2024-03-06T07:24:56.979695+0000","last_unstale":"2024-03-06T07:24:56.984328+0000","last_undegraded":"2024-03-06T07:24:56.984328+0000","last_fullsized":"2024-03-06T07:24:56.984328+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:27.776102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996075+0000","last_change":"2024-03-06T07:24:56.990237+0000","last_active":"2024-03-06T07:24:56.996075+0000","last_peered":"2024-03-06T07:24:56.996075+0000","last_clean":"2024-03-06T07:24:56.996075+0000","last_became_active":"2024-03-06T07:24:56.989981+0000","last_became_peered":"2024-03-06T07:24:56.989981+0000","last_unstale":"2024-03-06T07:24:56.996075+0000","last_undegraded":"2024-03-06T07:24:56.996075+0000","last_fullsized":"2024-03-06T07:24:56.996075+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:32.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997558+0000","last_change":"2024-03-06T07:24:56.995253+0000","last_active":"2024-03-06T07:24:56.997558+0000","last_peered":"2024-03-06T07:24:56.997558+0000","last_clean":"2024-03-06T07:24:56.997558+0000","last_became_active":"2024-03-06T07:24:56.994735+0000","last_became_peered":"2024-03-06T07:24:56.994735+0000","last_unstale":"2024-03-06T07:24:56.997558+0000","last_undegraded":"2024-03-06T07:24:56.997558+0000","last_fullsized":"2024-03-06T07:24:56.997558+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:20.534691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983390+0000","last_change":"2024-03-06T07:24:56.982130+0000","last_active":"2024-03-06T07:24:56.983390+0000","last_peered":"2024-03-06T07:24:56.983390+0000","last_clean":"2024-03-06T07:24:56.983390+0000","last_became_active":"2024-03-06T07:24:56.981779+0000","last_became_peered":"2024-03-06T07:24:56.981779+0000","last_unstale":"2024-03-06T07:24:56.983390+0000","last_undegraded":"2024-03-06T07:24:56.983390+0000","last_fullsized":"2024-03-06T07:24:56.983390+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:26.760058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.987188+0000","last_change":"2024-03-06T07:24:56.985435+0000","last_active":"2024-03-06T07:24:56.987188+0000","last_peered":"2024-03-06T07:24:56.987188+0000","last_clean":"2024-03-06T07:24:56.987188+0000","last_became_active":"2024-03-06T07:24:56.985143+0000","last_became_peered":"2024-03-06T07:24:56.985143+0000","last_unstale":"2024-03-06T07:24:56.987188+0000","last_undegraded":"2024-03-06T07:24:56.987188+0000","last_fullsized":"2024-03-06T07:24:56.987188+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:21.626315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989377+0000","last_change":"2024-03-06T07:24:56.988589+0000","last_active":"2024-03-06T07:24:56.989377+0000","last_peered":"2024-03-06T07:24:56.989377+0000","last_clean":"2024-03-06T07:24:56.989377+0000","last_became_active":"2024-03-06T07:24:56.988156+0000","last_became_peered":"2024-03-06T07:24:56.988156+0000","last_unstale":"2024-03-06T07:24:56.989377+0000","last_undegraded":"2024-03-06T07:24:56.989377+0000","last_fullsized":"2024-03-06T07:24:56.989377+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:01.751115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986580+0000","last_change":"2024-03-06T07:24:56.984877+0000","last_active":"2024-03-06T07:24:56.986580+0000","last_peered":"2024-03-06T07:24:56.986580+0000","last_clean":"2024-03-06T07:24:56.986580+0000","last_became_active":"2024-03-06T07:24:56.984621+0000","last_became_peered":"2024-03-06T07:24:56.984621+0000","last_unstale":"2024-03-06T07:24:56.986580+0000","last_undegraded":"2024-03-06T07:24:56.986580+0000","last_fullsized":"2024-03-06T07:24:56.986580+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:05.931371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989442+0000","last_change":"2024-03-06T07:24:56.982689+0000","last_active":"2024-03-06T07:24:56.989442+0000","last_peered":"2024-03-06T07:24:56.989442+0000","last_clean":"2024-03-06T07:24:56.989442+0000","last_became_active":"2024-03-06T07:24:56.982383+0000","last_became_peered":"2024-03-06T07:24:56.982383+0000","last_unstale":"2024-03-06T07:24:56.989442+0000","last_undegraded":"2024-03-06T07:24:56.989442+0000","last_fullsized":"2024-03-06T07:24:56.989442+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:25.308202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.993525+0000","last_change":"2024-03-06T07:24:56.987455+0000","last_active":"2024-03-06T07:24:56.993525+0000","last_peered":"2024-03-06T07:24:56.993525+0000","last_clean":"2024-03-06T07:24:56.993525+0000","last_became_active":"2024-03-06T07:24:56.986523+0000","last_became_peered":"2024-03-06T07:24:56.986523+0000","last_unstale":"2024-03-06T07:24:56.993525+0000","last_undegraded":"2024-03-06T07:24:56.993525+0000","last_fullsized":"2024-03-06T07:24:56.993525+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:54.994639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.993919+0000","last_change":"2024-03-06T07:24:56.989312+0000","last_active":"2024-03-06T07:24:56.993919+0000","last_peered":"2024-03-06T07:24:56.993919+0000","last_clean":"2024-03-06T07:24:56.993919+0000","last_became_active":"2024-03-06T07:24:56.988606+0000","last_became_peered":"2024-03-06T07:24:56.988606+0000","last_unstale":"2024-03-06T07:24:56.993919+0000","last_undegraded":"2024-03-06T07:24:56.993919+0000","last_fullsized":"2024-03-06T07:24:56.993919+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:32.772930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997152+0000","last_change":"2024-03-06T07:24:56.994742+0000","last_active":"2024-03-06T07:24:56.997152+0000","last_peered":"2024-03-06T07:24:56.997152+0000","last_clean":"2024-03-06T07:24:56.997152+0000","last_became_active":"2024-03-06T07:24:56.994376+0000","last_became_peered":"2024-03-06T07:24:56.994376+0000","last_unstale":"2024-03-06T07:24:56.997152+0000","last_undegraded":"2024-03-06T07:24:56.997152+0000","last_fullsized":"2024-03-06T07:24:56.997152+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:53.646560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995382+0000","last_change":"2024-03-06T07:24:56.993067+0000","last_active":"2024-03-06T07:24:56.995382+0000","last_peered":"2024-03-06T07:24:56.995382+0000","last_clean":"2024-03-06T07:24:56.995382+0000","last_became_active":"2024-03-06T07:24:56.992802+0000","last_became_peered":"2024-03-06T07:24:56.992802+0000","last_unstale":"2024-03-06T07:24:56.995382+0000","last_undegraded":"2024-03-06T07:24:56.995382+0000","last_fullsized":"2024-03-06T07:24:56.995382+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:26.114882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.994920+0000","last_change":"2024-03-06T07:24:56.993015+0000","last_active":"2024-03-06T07:24:56.994920+0000","last_peered":"2024-03-06T07:24:56.994920+0000","last_clean":"2024-03-06T07:24:56.994920+0000","last_became_active":"2024-03-06T07:24:56.992714+0000","last_became_peered":"2024-03-06T07:24:56.992714+0000","last_unstale":"2024-03-06T07:24:56.994920+0000","last_undegraded":"2024-03-06T07:24:56.994920+0000","last_fullsized":"2024-03-06T07:24:56.994920+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:25.589832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986574+0000","last_change":"2024-03-06T07:24:56.985846+0000","last_active":"2024-03-06T07:24:56.986574+0000","last_peered":"2024-03-06T07:24:56.986574+0000","last_clean":"2024-03-06T07:24:56.986574+0000","last_became_active":"2024-03-06T07:24:56.985362+0000","last_became_peered":"2024-03-06T07:24:56.985362+0000","last_unstale":"2024-03-06T07:24:56.986574+0000","last_undegraded":"2024-03-06T07:24:56.986574+0000","last_fullsized":"2024-03-06T07:24:56.986574+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:18.319695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.006922+0000","last_change":"2024-03-06T07:24:57.000322+0000","last_active":"2024-03-06T07:24:57.006922+0000","last_peered":"2024-03-06T07:24:57.006922+0000","last_clean":"2024-03-06T07:24:57.006922+0000","last_became_active":"2024-03-06T07:24:57.000042+0000","last_became_peered":"2024-03-06T07:24:57.000042+0000","last_unstale":"2024-03-06T07:24:57.006922+0000","last_undegraded":"2024-03-06T07:24:57.006922+0000","last_fullsized":"2024-03-06T07:24:57.006922+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:30.772328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986679+0000","last_change":"2024-03-06T07:24:56.984716+0000","last_active":"2024-03-06T07:24:56.986679+0000","last_peered":"2024-03-06T07:24:56.986679+0000","last_clean":"2024-03-06T07:24:56.986679+0000","last_became_active":"2024-03-06T07:24:56.984487+0000","last_became_peered":"2024-03-06T07:24:56.984487+0000","last_unstale":"2024-03-06T07:24:56.986679+0000","last_undegraded":"2024-03-06T07:24:56.986679+0000","last_fullsized":"2024-03-06T07:24:56.986679+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:47.734879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984569+0000","last_change":"2024-03-06T07:24:56.982818+0000","last_active":"2024-03-06T07:24:56.984569+0000","last_peered":"2024-03-06T07:24:56.984569+0000","last_clean":"2024-03-06T07:24:56.984569+0000","last_became_active":"2024-03-06T07:24:56.982533+0000","last_became_peered":"2024-03-06T07:24:56.982533+0000","last_unstale":"2024-03-06T07:24:56.984569+0000","last_undegraded":"2024-03-06T07:24:56.984569+0000","last_fullsized":"2024-03-06T07:24:56.984569+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:44.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989722+0000","last_change":"2024-03-06T07:24:56.988856+0000","last_active":"2024-03-06T07:24:56.989722+0000","last_peered":"2024-03-06T07:24:56.989722+0000","last_clean":"2024-03-06T07:24:56.989722+0000","last_became_active":"2024-03-06T07:24:56.988096+0000","last_became_peered":"2024-03-06T07:24:56.988096+0000","last_unstale":"2024-03-06T07:24:56.989722+0000","last_undegraded":"2024-03-06T07:24:56.989722+0000","last_fullsized":"2024-03-06T07:24:56.989722+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:36.204436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983239+0000","last_change":"2024-03-06T07:24:56.980859+0000","last_active":"2024-03-06T07:24:56.983239+0000","last_peered":"2024-03-06T07:24:56.983239+0000","last_clean":"2024-03-06T07:24:56.983239+0000","last_became_active":"2024-03-06T07:24:56.980578+0000","last_became_peered":"2024-03-06T07:24:56.980578+0000","last_unstale":"2024-03-06T07:24:56.983239+0000","last_undegraded":"2024-03-06T07:24:56.983239+0000","last_fullsized":"2024-03-06T07:24:56.983239+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:54.163100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996033+0000","last_change":"2024-03-06T07:24:56.991368+0000","last_active":"2024-03-06T07:24:56.996033+0000","last_peered":"2024-03-06T07:24:56.996033+0000","last_clean":"2024-03-06T07:24:56.996033+0000","last_became_active":"2024-03-06T07:24:56.991027+0000","last_became_peered":"2024-03-06T07:24:56.991027+0000","last_unstale":"2024-03-06T07:24:56.996033+0000","last_undegraded":"2024-03-06T07:24:56.996033+0000","last_fullsized":"2024-03-06T07:24:56.996033+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:30.953134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996104+0000","last_change":"2024-03-06T07:24:56.988597+0000","last_active":"2024-03-06T07:24:56.996104+0000","last_peered":"2024-03-06T07:24:56.996104+0000","last_clean":"2024-03-06T07:24:56.996104+0000","last_became_active":"2024-03-06T07:24:56.988286+0000","last_became_peered":"2024-03-06T07:24:56.988286+0000","last_unstale":"2024-03-06T07:24:56.996104+0000","last_undegraded":"2024-03-06T07:24:56.996104+0000","last_fullsized":"2024-03-06T07:24:56.996104+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:05.822744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.993575+0000","last_change":"2024-03-06T07:24:56.992588+0000","last_active":"2024-03-06T07:24:56.993575+0000","last_peered":"2024-03-06T07:24:56.993575+0000","last_clean":"2024-03-06T07:24:56.993575+0000","last_became_active":"2024-03-06T07:24:56.992317+0000","last_became_peered":"2024-03-06T07:24:56.992317+0000","last_unstale":"2024-03-06T07:24:56.993575+0000","last_undegraded":"2024-03-06T07:24:56.993575+0000","last_fullsized":"2024-03-06T07:24:56.993575+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:43.253268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"169'4","reported_seq":16,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002962+0000","last_change":"2024-03-06T07:24:55.983989+0000","last_active":"2024-03-06T07:24:57.002962+0000","last_peered":"2024-03-06T07:24:57.002962+0000","last_clean":"2024-03-06T07:24:57.002962+0000","last_became_active":"2024-03-06T07:24:55.983710+0000","last_became_peered":"2024-03-06T07:24:55.983710+0000","last_unstale":"2024-03-06T07:24:57.002962+0000","last_undegraded":"2024-03-06T07:24:57.002962+0000","last_fullsized":"2024-03-06T07:24:57.002962+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:49.896080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997412+0000","last_change":"2024-03-06T07:24:56.993002+0000","last_active":"2024-03-06T07:24:56.997412+0000","last_peered":"2024-03-06T07:24:56.997412+0000","last_clean":"2024-03-06T07:24:56.997412+0000","last_became_active":"2024-03-06T07:24:56.992733+0000","last_became_peered":"2024-03-06T07:24:56.992733+0000","last_unstale":"2024-03-06T07:24:56.997412+0000","last_undegraded":"2024-03-06T07:24:56.997412+0000","last_fullsized":"2024-03-06T07:24:56.997412+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:22.730138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004254+0000","last_change":"2024-03-06T07:24:55.988062+0000","last_active":"2024-03-06T07:24:57.004254+0000","last_peered":"2024-03-06T07:24:57.004254+0000","last_clean":"2024-03-06T07:24:57.004254+0000","last_became_active":"2024-03-06T07:24:55.987789+0000","last_became_peered":"2024-03-06T07:24:55.987789+0000","last_unstale":"2024-03-06T07:24:57.004254+0000","last_undegraded":"2024-03-06T07:24:57.004254+0000","last_fullsized":"2024-03-06T07:24:57.004254+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:46.326046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989121+0000","last_change":"2024-03-06T07:24:56.980031+0000","last_active":"2024-03-06T07:24:56.989121+0000","last_peered":"2024-03-06T07:24:56.989121+0000","last_clean":"2024-03-06T07:24:56.989121+0000","last_became_active":"2024-03-06T07:24:56.979758+0000","last_became_peered":"2024-03-06T07:24:56.979758+0000","last_unstale":"2024-03-06T07:24:56.989121+0000","last_undegraded":"2024-03-06T07:24:56.989121+0000","last_fullsized":"2024-03-06T07:24:56.989121+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:44.974416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000295+0000","last_change":"2024-03-06T07:24:55.981064+0000","last_active":"2024-03-06T07:24:57.000295+0000","last_peered":"2024-03-06T07:24:57.000295+0000","last_clean":"2024-03-06T07:24:57.000295+0000","last_became_active":"2024-03-06T07:24:55.980522+0000","last_became_peered":"2024-03-06T07:24:55.980522+0000","last_unstale":"2024-03-06T07:24:57.000295+0000","last_undegraded":"2024-03-06T07:24:57.000295+0000","last_fullsized":"2024-03-06T07:24:57.000295+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:20.101019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983650+0000","last_change":"2024-03-06T07:24:56.978181+0000","last_active":"2024-03-06T07:24:56.983650+0000","last_peered":"2024-03-06T07:24:56.983650+0000","last_clean":"2024-03-06T07:24:56.983650+0000","last_became_active":"2024-03-06T07:24:56.977923+0000","last_became_peered":"2024-03-06T07:24:56.977923+0000","last_unstale":"2024-03-06T07:24:56.983650+0000","last_undegraded":"2024-03-06T07:24:56.983650+0000","last_fullsized":"2024-03-06T07:24:56.983650+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.726864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002188+0000","last_change":"2024-03-06T07:24:55.986832+0000","last_active":"2024-03-06T07:24:57.002188+0000","last_peered":"2024-03-06T07:24:57.002188+0000","last_clean":"2024-03-06T07:24:57.002188+0000","last_became_active":"2024-03-06T07:24:55.986453+0000","last_became_peered":"2024-03-06T07:24:55.986453+0000","last_unstale":"2024-03-06T07:24:57.002188+0000","last_undegraded":"2024-03-06T07:24:57.002188+0000","last_fullsized":"2024-03-06T07:24:57.002188+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:31.747566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983461+0000","last_change":"2024-03-06T07:24:56.980858+0000","last_active":"2024-03-06T07:24:56.983461+0000","last_peered":"2024-03-06T07:24:56.983461+0000","last_clean":"2024-03-06T07:24:56.983461+0000","last_became_active":"2024-03-06T07:24:56.980554+0000","last_became_peered":"2024-03-06T07:24:56.980554+0000","last_unstale":"2024-03-06T07:24:56.983461+0000","last_undegraded":"2024-03-06T07:24:56.983461+0000","last_fullsized":"2024-03-06T07:24:56.983461+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:36.182573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002004+0000","last_change":"2024-03-06T07:24:55.985565+0000","last_active":"2024-03-06T07:24:57.002004+0000","last_peered":"2024-03-06T07:24:57.002004+0000","last_clean":"2024-03-06T07:24:57.002004+0000","last_became_active":"2024-03-06T07:24:55.985278+0000","last_became_peered":"2024-03-06T07:24:55.985278+0000","last_unstale":"2024-03-06T07:24:57.002004+0000","last_undegraded":"2024-03-06T07:24:57.002004+0000","last_fullsized":"2024-03-06T07:24:57.002004+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:48.369892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983186+0000","last_change":"2024-03-06T07:24:56.981707+0000","last_active":"2024-03-06T07:24:56.983186+0000","last_peered":"2024-03-06T07:24:56.983186+0000","last_clean":"2024-03-06T07:24:56.983186+0000","last_became_active":"2024-03-06T07:24:56.981436+0000","last_became_peered":"2024-03-06T07:24:56.981436+0000","last_unstale":"2024-03-06T07:24:56.983186+0000","last_undegraded":"2024-03-06T07:24:56.983186+0000","last_fullsized":"2024-03-06T07:24:56.983186+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:15.451504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.978457+0000","last_change":"2024-03-06T07:24:55.987161+0000","last_active":"2024-03-06T07:24:56.978457+0000","last_peered":"2024-03-06T07:24:56.978457+0000","last_clean":"2024-03-06T07:24:56.978457+0000","last_became_active":"2024-03-06T07:24:55.986697+0000","last_became_peered":"2024-03-06T07:24:55.986697+0000","last_unstale":"2024-03-06T07:24:56.978457+0000","last_undegraded":"2024-03-06T07:24:56.978457+0000","last_fullsized":"2024-03-06T07:24:56.978457+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:21.725573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989695+0000","last_change":"2024-03-06T07:24:56.988844+0000","last_active":"2024-03-06T07:24:56.989695+0000","last_peered":"2024-03-06T07:24:56.989695+0000","last_clean":"2024-03-06T07:24:56.989695+0000","last_became_active":"2024-03-06T07:24:56.988289+0000","last_became_peered":"2024-03-06T07:24:56.988289+0000","last_unstale":"2024-03-06T07:24:56.989695+0000","last_undegraded":"2024-03-06T07:24:56.989695+0000","last_fullsized":"2024-03-06T07:24:56.989695+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:40.896460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.978355+0000","last_change":"2024-03-06T07:24:55.981034+0000","last_active":"2024-03-06T07:24:56.978355+0000","last_peered":"2024-03-06T07:24:56.978355+0000","last_clean":"2024-03-06T07:24:56.978355+0000","last_became_active":"2024-03-06T07:24:55.980755+0000","last_became_peered":"2024-03-06T07:24:55.980755+0000","last_unstale":"2024-03-06T07:24:56.978355+0000","last_undegraded":"2024-03-06T07:24:56.978355+0000","last_fullsized":"2024-03-06T07:24:56.978355+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:00.746550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.991299+0000","last_change":"2024-03-06T07:24:56.987558+0000","last_active":"2024-03-06T07:24:56.991299+0000","last_peered":"2024-03-06T07:24:56.991299+0000","last_clean":"2024-03-06T07:24:56.991299+0000","last_became_active":"2024-03-06T07:24:56.987287+0000","last_became_peered":"2024-03-06T07:24:56.987287+0000","last_unstale":"2024-03-06T07:24:56.991299+0000","last_undegraded":"2024-03-06T07:24:56.991299+0000","last_fullsized":"2024-03-06T07:24:56.991299+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:30.927206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974454+0000","last_change":"2024-03-06T07:24:55.982490+0000","last_active":"2024-03-06T07:24:56.974454+0000","last_peered":"2024-03-06T07:24:56.974454+0000","last_clean":"2024-03-06T07:24:56.974454+0000","last_became_active":"2024-03-06T07:24:55.981754+0000","last_became_peered":"2024-03-06T07:24:55.981754+0000","last_unstale":"2024-03-06T07:24:56.974454+0000","last_undegraded":"2024-03-06T07:24:56.974454+0000","last_fullsized":"2024-03-06T07:24:56.974454+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:10.143335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.987937+0000","last_change":"2024-03-06T07:24:56.987377+0000","last_active":"2024-03-06T07:24:56.987937+0000","last_peered":"2024-03-06T07:24:56.987937+0000","last_clean":"2024-03-06T07:24:56.987937+0000","last_became_active":"2024-03-06T07:24:56.987097+0000","last_became_peered":"2024-03-06T07:24:56.987097+0000","last_unstale":"2024-03-06T07:24:56.987937+0000","last_undegraded":"2024-03-06T07:24:56.987937+0000","last_fullsized":"2024-03-06T07:24:56.987937+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:25.903162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979361+0000","last_change":"2024-03-06T07:24:55.986301+0000","last_active":"2024-03-06T07:24:56.979361+0000","last_peered":"2024-03-06T07:24:56.979361+0000","last_clean":"2024-03-06T07:24:56.979361+0000","last_became_active":"2024-03-06T07:24:55.985972+0000","last_became_peered":"2024-03-06T07:24:55.985972+0000","last_unstale":"2024-03-06T07:24:56.979361+0000","last_undegraded":"2024-03-06T07:24:56.979361+0000","last_fullsized":"2024-03-06T07:24:56.979361+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:59.271876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997353+0000","last_change":"2024-03-06T07:24:56.995552+0000","last_active":"2024-03-06T07:24:56.997353+0000","last_peered":"2024-03-06T07:24:56.997353+0000","last_clean":"2024-03-06T07:24:56.997353+0000","last_became_active":"2024-03-06T07:24:56.995297+0000","last_became_peered":"2024-03-06T07:24:56.995297+0000","last_unstale":"2024-03-06T07:24:56.997353+0000","last_undegraded":"2024-03-06T07:24:56.997353+0000","last_fullsized":"2024-03-06T07:24:56.997353+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:25.903162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"169'3","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.001122+0000","last_change":"2024-03-06T07:24:55.987935+0000","last_active":"2024-03-06T07:24:57.001122+0000","last_peered":"2024-03-06T07:24:57.001122+0000","last_clean":"2024-03-06T07:24:57.001122+0000","last_became_active":"2024-03-06T07:24:55.987636+0000","last_became_peered":"2024-03-06T07:24:55.987636+0000","last_unstale":"2024-03-06T07:24:57.001122+0000","last_undegraded":"2024-03-06T07:24:57.001122+0000","last_fullsized":"2024-03-06T07:24:57.001122+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:41.937588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984659+0000","last_change":"2024-03-06T07:24:56.982361+0000","last_active":"2024-03-06T07:24:56.984659+0000","last_peered":"2024-03-06T07:24:56.984659+0000","last_clean":"2024-03-06T07:24:56.984659+0000","last_became_active":"2024-03-06T07:24:56.982080+0000","last_became_peered":"2024-03-06T07:24:56.982080+0000","last_unstale":"2024-03-06T07:24:56.984659+0000","last_undegraded":"2024-03-06T07:24:56.984659+0000","last_fullsized":"2024-03-06T07:24:56.984659+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:44.974416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"169'6","reported_seq":18,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.010073+0000","last_change":"2024-03-06T07:24:55.982194+0000","last_active":"2024-03-06T07:24:57.010073+0000","last_peered":"2024-03-06T07:24:57.010073+0000","last_clean":"2024-03-06T07:24:57.010073+0000","last_became_active":"2024-03-06T07:24:55.981938+0000","last_became_peered":"2024-03-06T07:24:55.981938+0000","last_unstale":"2024-03-06T07:24:57.010073+0000","last_undegraded":"2024-03-06T07:24:57.010073+0000","last_fullsized":"2024-03-06T07:24:57.010073+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:14.310018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.977721+0000","last_change":"2024-03-06T07:24:56.975877+0000","last_active":"2024-03-06T07:24:56.977721+0000","last_peered":"2024-03-06T07:24:56.977721+0000","last_clean":"2024-03-06T07:24:56.977721+0000","last_became_active":"2024-03-06T07:24:56.975601+0000","last_became_peered":"2024-03-06T07:24:56.975601+0000","last_unstale":"2024-03-06T07:24:56.977721+0000","last_undegraded":"2024-03-06T07:24:56.977721+0000","last_fullsized":"2024-03-06T07:24:56.977721+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:56.173817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004660+0000","last_change":"2024-03-06T07:24:55.985953+0000","last_active":"2024-03-06T07:24:57.004660+0000","last_peered":"2024-03-06T07:24:57.004660+0000","last_clean":"2024-03-06T07:24:57.004660+0000","last_became_active":"2024-03-06T07:24:55.985645+0000","last_became_peered":"2024-03-06T07:24:55.985645+0000","last_unstale":"2024-03-06T07:24:57.004660+0000","last_undegraded":"2024-03-06T07:24:57.004660+0000","last_fullsized":"2024-03-06T07:24:57.004660+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:14.926934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986656+0000","last_change":"2024-03-06T07:24:56.984663+0000","last_active":"2024-03-06T07:24:56.986656+0000","last_peered":"2024-03-06T07:24:56.986656+0000","last_clean":"2024-03-06T07:24:56.986656+0000","last_became_active":"2024-03-06T07:24:56.984371+0000","last_became_peered":"2024-03-06T07:24:56.984371+0000","last_unstale":"2024-03-06T07:24:56.986656+0000","last_undegraded":"2024-03-06T07:24:56.986656+0000","last_fullsized":"2024-03-06T07:24:56.986656+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:36.204436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979918+0000","last_change":"2024-03-06T07:24:55.981303+0000","last_active":"2024-03-06T07:24:56.979918+0000","last_peered":"2024-03-06T07:24:56.979918+0000","last_clean":"2024-03-06T07:24:56.979918+0000","last_became_active":"2024-03-06T07:24:55.981000+0000","last_became_peered":"2024-03-06T07:24:55.981000+0000","last_unstale":"2024-03-06T07:24:56.979918+0000","last_undegraded":"2024-03-06T07:24:56.979918+0000","last_fullsized":"2024-03-06T07:24:56.979918+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:39.891895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.980568+0000","last_change":"2024-03-06T07:24:56.977228+0000","last_active":"2024-03-06T07:24:56.980568+0000","last_peered":"2024-03-06T07:24:56.980568+0000","last_clean":"2024-03-06T07:24:56.980568+0000","last_became_active":"2024-03-06T07:24:56.976967+0000","last_became_peered":"2024-03-06T07:24:56.976967+0000","last_unstale":"2024-03-06T07:24:56.980568+0000","last_undegraded":"2024-03-06T07:24:56.980568+0000","last_fullsized":"2024-03-06T07:24:56.980568+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:58.830269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.973804+0000","last_change":"2024-03-06T07:24:55.983949+0000","last_active":"2024-03-06T07:24:56.973804+0000","last_peered":"2024-03-06T07:24:56.973804+0000","last_clean":"2024-03-06T07:24:56.973804+0000","last_became_active":"2024-03-06T07:24:55.983631+0000","last_became_peered":"2024-03-06T07:24:55.983631+0000","last_unstale":"2024-03-06T07:24:56.973804+0000","last_undegraded":"2024-03-06T07:24:56.973804+0000","last_fullsized":"2024-03-06T07:24:56.973804+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:59.271876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000535+0000","last_change":"2024-03-06T07:24:56.995337+0000","last_active":"2024-03-06T07:24:57.000535+0000","last_peered":"2024-03-06T07:24:57.000535+0000","last_clean":"2024-03-06T07:24:57.000535+0000","last_became_active":"2024-03-06T07:24:56.994857+0000","last_became_peered":"2024-03-06T07:24:56.994857+0000","last_unstale":"2024-03-06T07:24:57.000535+0000","last_undegraded":"2024-03-06T07:24:57.000535+0000","last_fullsized":"2024-03-06T07:24:57.000535+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:18.319695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"169'8","reported_seq":20,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.007191+0000","last_change":"2024-03-06T07:24:55.981434+0000","last_active":"2024-03-06T07:24:57.007191+0000","last_peered":"2024-03-06T07:24:57.007191+0000","last_clean":"2024-03-06T07:24:57.007191+0000","last_became_active":"2024-03-06T07:24:55.981144+0000","last_became_peered":"2024-03-06T07:24:55.981144+0000","last_unstale":"2024-03-06T07:24:57.007191+0000","last_undegraded":"2024-03-06T07:24:57.007191+0000","last_fullsized":"2024-03-06T07:24:57.007191+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:58.558844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997298+0000","last_change":"2024-03-06T07:24:56.991808+0000","last_active":"2024-03-06T07:24:56.997298+0000","last_peered":"2024-03-06T07:24:56.997298+0000","last_clean":"2024-03-06T07:24:56.997298+0000","last_became_active":"2024-03-06T07:24:56.991155+0000","last_became_peered":"2024-03-06T07:24:56.991155+0000","last_unstale":"2024-03-06T07:24:56.997298+0000","last_undegraded":"2024-03-06T07:24:56.997298+0000","last_fullsized":"2024-03-06T07:24:56.997298+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:30.277424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004361+0000","last_change":"2024-03-06T07:24:55.983699+0000","last_active":"2024-03-06T07:24:57.004361+0000","last_peered":"2024-03-06T07:24:57.004361+0000","last_clean":"2024-03-06T07:24:57.004361+0000","last_became_active":"2024-03-06T07:24:55.983409+0000","last_became_peered":"2024-03-06T07:24:55.983409+0000","last_unstale":"2024-03-06T07:24:57.004361+0000","last_undegraded":"2024-03-06T07:24:57.004361+0000","last_fullsized":"2024-03-06T07:24:57.004361+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:44.010529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000437+0000","last_change":"2024-03-06T07:24:56.995382+0000","last_active":"2024-03-06T07:24:57.000437+0000","last_peered":"2024-03-06T07:24:57.000437+0000","last_clean":"2024-03-06T07:24:57.000437+0000","last_became_active":"2024-03-06T07:24:56.995113+0000","last_became_peered":"2024-03-06T07:24:56.995113+0000","last_unstale":"2024-03-06T07:24:57.000437+0000","last_undegraded":"2024-03-06T07:24:57.000437+0000","last_fullsized":"2024-03-06T07:24:57.000437+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:49.374457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004348+0000","last_change":"2024-03-06T07:24:55.987346+0000","last_active":"2024-03-06T07:24:57.004348+0000","last_peered":"2024-03-06T07:24:57.004348+0000","last_clean":"2024-03-06T07:24:57.004348+0000","last_became_active":"2024-03-06T07:24:55.987094+0000","last_became_peered":"2024-03-06T07:24:55.987094+0000","last_unstale":"2024-03-06T07:24:57.004348+0000","last_undegraded":"2024-03-06T07:24:57.004348+0000","last_fullsized":"2024-03-06T07:24:57.004348+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:57.153000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997467+0000","last_change":"2024-03-06T07:24:56.994468+0000","last_active":"2024-03-06T07:24:56.997467+0000","last_peered":"2024-03-06T07:24:56.997467+0000","last_clean":"2024-03-06T07:24:56.997467+0000","last_became_active":"2024-03-06T07:24:56.994185+0000","last_became_peered":"2024-03-06T07:24:56.994185+0000","last_unstale":"2024-03-06T07:24:56.997467+0000","last_undegraded":"2024-03-06T07:24:56.997467+0000","last_fullsized":"2024-03-06T07:24:56.997467+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:15.314583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002062+0000","last_change":"2024-03-06T07:24:55.980697+0000","last_active":"2024-03-06T07:24:57.002062+0000","last_peered":"2024-03-06T07:24:57.002062+0000","last_clean":"2024-03-06T07:24:57.002062+0000","last_became_active":"2024-03-06T07:24:55.980415+0000","last_became_peered":"2024-03-06T07:24:55.980415+0000","last_unstale":"2024-03-06T07:24:57.002062+0000","last_undegraded":"2024-03-06T07:24:57.002062+0000","last_fullsized":"2024-03-06T07:24:57.002062+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:16.584485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.992616+0000","last_change":"2024-03-06T07:24:56.984621+0000","last_active":"2024-03-06T07:24:56.992616+0000","last_peered":"2024-03-06T07:24:56.992616+0000","last_clean":"2024-03-06T07:24:56.992616+0000","last_became_active":"2024-03-06T07:24:56.984359+0000","last_became_peered":"2024-03-06T07:24:56.984359+0000","last_unstale":"2024-03-06T07:24:56.992616+0000","last_undegraded":"2024-03-06T07:24:56.992616+0000","last_fullsized":"2024-03-06T07:24:56.992616+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:28.712811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974380+0000","last_change":"2024-03-06T07:24:55.983500+0000","last_active":"2024-03-06T07:24:56.974380+0000","last_peered":"2024-03-06T07:24:56.974380+0000","last_clean":"2024-03-06T07:24:56.974380+0000","last_became_active":"2024-03-06T07:24:55.983234+0000","last_became_peered":"2024-03-06T07:24:55.983234+0000","last_unstale":"2024-03-06T07:24:56.974380+0000","last_undegraded":"2024-03-06T07:24:56.974380+0000","last_fullsized":"2024-03-06T07:24:56.974380+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:53.352866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986875+0000","last_change":"2024-03-06T07:24:56.983170+0000","last_active":"2024-03-06T07:24:56.986875+0000","last_peered":"2024-03-06T07:24:56.986875+0000","last_clean":"2024-03-06T07:24:56.986875+0000","last_became_active":"2024-03-06T07:24:56.982701+0000","last_became_peered":"2024-03-06T07:24:56.982701+0000","last_unstale":"2024-03-06T07:24:56.986875+0000","last_undegraded":"2024-03-06T07:24:56.986875+0000","last_fullsized":"2024-03-06T07:24:56.986875+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:12.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.001519+0000","last_change":"2024-03-06T07:24:55.974671+0000","last_active":"2024-03-06T07:24:57.001519+0000","last_peered":"2024-03-06T07:24:57.001519+0000","last_clean":"2024-03-06T07:24:57.001519+0000","last_became_active":"2024-03-06T07:24:55.974392+0000","last_became_peered":"2024-03-06T07:24:55.974392+0000","last_unstale":"2024-03-06T07:24:57.001519+0000","last_undegraded":"2024-03-06T07:24:57.001519+0000","last_fullsized":"2024-03-06T07:24:57.001519+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:49.702145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995901+0000","last_change":"2024-03-06T07:24:56.990589+0000","last_active":"2024-03-06T07:24:56.995901+0000","last_peered":"2024-03-06T07:24:56.995901+0000","last_clean":"2024-03-06T07:24:56.995901+0000","last_became_active":"2024-03-06T07:24:56.989245+0000","last_became_peered":"2024-03-06T07:24:56.989245+0000","last_unstale":"2024-03-06T07:24:56.995901+0000","last_undegraded":"2024-03-06T07:24:56.995901+0000","last_fullsized":"2024-03-06T07:24:56.995901+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:25.710876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.970703+0000","last_change":"2024-03-06T07:24:55.983340+0000","last_active":"2024-03-06T07:24:56.970703+0000","last_peered":"2024-03-06T07:24:56.970703+0000","last_clean":"2024-03-06T07:24:56.970703+0000","last_became_active":"2024-03-06T07:24:55.982397+0000","last_became_peered":"2024-03-06T07:24:55.982397+0000","last_unstale":"2024-03-06T07:24:56.970703+0000","last_undegraded":"2024-03-06T07:24:56.970703+0000","last_fullsized":"2024-03-06T07:24:56.970703+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:31.650143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995450+0000","last_change":"2024-03-06T07:24:56.993222+0000","last_active":"2024-03-06T07:24:56.995450+0000","last_peered":"2024-03-06T07:24:56.995450+0000","last_clean":"2024-03-06T07:24:56.995450+0000","last_became_active":"2024-03-06T07:24:56.992958+0000","last_became_peered":"2024-03-06T07:24:56.992958+0000","last_unstale":"2024-03-06T07:24:56.995450+0000","last_undegraded":"2024-03-06T07:24:56.995450+0000","last_fullsized":"2024-03-06T07:24:56.995450+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:47.734879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974360+0000","last_change":"2024-03-06T07:24:55.983772+0000","last_active":"2024-03-06T07:24:56.974360+0000","last_peered":"2024-03-06T07:24:56.974360+0000","last_clean":"2024-03-06T07:24:56.974360+0000","last_became_active":"2024-03-06T07:24:55.983289+0000","last_became_peered":"2024-03-06T07:24:55.983289+0000","last_unstale":"2024-03-06T07:24:56.974360+0000","last_undegraded":"2024-03-06T07:24:56.974360+0000","last_fullsized":"2024-03-06T07:24:56.974360+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:10.845648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989412+0000","last_change":"2024-03-06T07:24:56.982033+0000","last_active":"2024-03-06T07:24:56.989412+0000","last_peered":"2024-03-06T07:24:56.989412+0000","last_clean":"2024-03-06T07:24:56.989412+0000","last_became_active":"2024-03-06T07:24:56.981070+0000","last_became_peered":"2024-03-06T07:24:56.981070+0000","last_unstale":"2024-03-06T07:24:56.989412+0000","last_undegraded":"2024-03-06T07:24:56.989412+0000","last_fullsized":"2024-03-06T07:24:56.989412+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:59.082721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.985139+0000","last_change":"2024-03-06T07:24:55.989533+0000","last_active":"2024-03-06T07:24:56.985139+0000","last_peered":"2024-03-06T07:24:56.985139+0000","last_clean":"2024-03-06T07:24:56.985139+0000","last_became_active":"2024-03-06T07:24:55.989263+0000","last_became_peered":"2024-03-06T07:24:55.989263+0000","last_unstale":"2024-03-06T07:24:56.985139+0000","last_undegraded":"2024-03-06T07:24:56.985139+0000","last_fullsized":"2024-03-06T07:24:56.985139+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:58.332262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.992193+0000","last_change":"2024-03-06T07:24:56.984039+0000","last_active":"2024-03-06T07:24:56.992193+0000","last_peered":"2024-03-06T07:24:56.992193+0000","last_clean":"2024-03-06T07:24:56.992193+0000","last_became_active":"2024-03-06T07:24:56.982838+0000","last_became_peered":"2024-03-06T07:24:56.982838+0000","last_unstale":"2024-03-06T07:24:56.992193+0000","last_undegraded":"2024-03-06T07:24:56.992193+0000","last_fullsized":"2024-03-06T07:24:56.992193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:25.589832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002478+0000","last_change":"2024-03-06T07:24:55.985972+0000","last_active":"2024-03-06T07:24:57.002478+0000","last_peered":"2024-03-06T07:24:57.002478+0000","last_clean":"2024-03-06T07:24:57.002478+0000","last_became_active":"2024-03-06T07:24:55.985205+0000","last_became_peered":"2024-03-06T07:24:55.985205+0000","last_unstale":"2024-03-06T07:24:57.002478+0000","last_undegraded":"2024-03-06T07:24:57.002478+0000","last_fullsized":"2024-03-06T07:24:57.002478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:29.948569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.998460+0000","last_change":"2024-03-06T07:24:56.995014+0000","last_active":"2024-03-06T07:24:56.998460+0000","last_peered":"2024-03-06T07:24:56.998460+0000","last_clean":"2024-03-06T07:24:56.998460+0000","last_became_active":"2024-03-06T07:24:56.994760+0000","last_became_peered":"2024-03-06T07:24:56.994760+0000","last_unstale":"2024-03-06T07:24:56.998460+0000","last_undegraded":"2024-03-06T07:24:56.998460+0000","last_fullsized":"2024-03-06T07:24:56.998460+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:50.595303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979527+0000","last_change":"2024-03-06T07:24:55.981956+0000","last_active":"2024-03-06T07:24:56.979527+0000","last_peered":"2024-03-06T07:24:56.979527+0000","last_clean":"2024-03-06T07:24:56.979527+0000","last_became_active":"2024-03-06T07:24:55.981705+0000","last_became_peered":"2024-03-06T07:24:55.981705+0000","last_unstale":"2024-03-06T07:24:56.979527+0000","last_undegraded":"2024-03-06T07:24:56.979527+0000","last_fullsized":"2024-03-06T07:24:56.979527+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:26.771537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.980590+0000","last_change":"2024-03-06T07:24:56.977134+0000","last_active":"2024-03-06T07:24:56.980590+0000","last_peered":"2024-03-06T07:24:56.980590+0000","last_clean":"2024-03-06T07:24:56.980590+0000","last_became_active":"2024-03-06T07:24:56.976863+0000","last_became_peered":"2024-03-06T07:24:56.976863+0000","last_unstale":"2024-03-06T07:24:56.980590+0000","last_undegraded":"2024-03-06T07:24:56.980590+0000","last_fullsized":"2024-03-06T07:24:56.980590+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:25.710876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974081+0000","last_change":"2024-03-06T07:24:55.982271+0000","last_active":"2024-03-06T07:24:56.974081+0000","last_peered":"2024-03-06T07:24:56.974081+0000","last_clean":"2024-03-06T07:24:56.974081+0000","last_became_active":"2024-03-06T07:24:55.981970+0000","last_became_peered":"2024-03-06T07:24:55.981970+0000","last_unstale":"2024-03-06T07:24:56.974081+0000","last_undegraded":"2024-03-06T07:24:56.974081+0000","last_fullsized":"2024-03-06T07:24:56.974081+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:21.886939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.006909+0000","last_change":"2024-03-06T07:24:57.001495+0000","last_active":"2024-03-06T07:24:57.006909+0000","last_peered":"2024-03-06T07:24:57.006909+0000","last_clean":"2024-03-06T07:24:57.006909+0000","last_became_active":"2024-03-06T07:24:57.001065+0000","last_became_peered":"2024-03-06T07:24:57.001065+0000","last_unstale":"2024-03-06T07:24:57.006909+0000","last_undegraded":"2024-03-06T07:24:57.006909+0000","last_fullsized":"2024-03-06T07:24:57.006909+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:42.942153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.985610+0000","last_change":"2024-03-06T07:24:55.986141+0000","last_active":"2024-03-06T07:24:56.985610+0000","last_peered":"2024-03-06T07:24:56.985610+0000","last_clean":"2024-03-06T07:24:56.985610+0000","last_became_active":"2024-03-06T07:24:55.985225+0000","last_became_peered":"2024-03-06T07:24:55.985225+0000","last_unstale":"2024-03-06T07:24:56.985610+0000","last_undegraded":"2024-03-06T07:24:56.985610+0000","last_fullsized":"2024-03-06T07:24:56.985610+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:30.365068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.994032+0000","last_change":"2024-03-06T07:24:56.989762+0000","last_active":"2024-03-06T07:24:56.994032+0000","last_peered":"2024-03-06T07:24:56.994032+0000","last_clean":"2024-03-06T07:24:56.994032+0000","last_became_active":"2024-03-06T07:24:56.989495+0000","last_became_peered":"2024-03-06T07:24:56.989495+0000","last_unstale":"2024-03-06T07:24:56.994032+0000","last_undegraded":"2024-03-06T07:24:56.994032+0000","last_fullsized":"2024-03-06T07:24:56.994032+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:21.760548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.977029+0000","last_change":"2024-03-06T07:24:55.980753+0000","last_active":"2024-03-06T07:24:56.977029+0000","last_peered":"2024-03-06T07:24:56.977029+0000","last_clean":"2024-03-06T07:24:56.977029+0000","last_became_active":"2024-03-06T07:24:55.980474+0000","last_became_peered":"2024-03-06T07:24:55.980474+0000","last_unstale":"2024-03-06T07:24:56.977029+0000","last_undegraded":"2024-03-06T07:24:56.977029+0000","last_fullsized":"2024-03-06T07:24:56.977029+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:00.722299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995924+0000","last_change":"2024-03-06T07:24:56.990803+0000","last_active":"2024-03-06T07:24:56.995924+0000","last_peered":"2024-03-06T07:24:56.995924+0000","last_clean":"2024-03-06T07:24:56.995924+0000","last_became_active":"2024-03-06T07:24:56.989539+0000","last_became_peered":"2024-03-06T07:24:56.989539+0000","last_unstale":"2024-03-06T07:24:56.995924+0000","last_undegraded":"2024-03-06T07:24:56.995924+0000","last_fullsized":"2024-03-06T07:24:56.995924+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:58.830269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974043+0000","last_change":"2024-03-06T07:24:55.984798+0000","last_active":"2024-03-06T07:24:56.974043+0000","last_peered":"2024-03-06T07:24:56.974043+0000","last_clean":"2024-03-06T07:24:56.974043+0000","last_became_active":"2024-03-06T07:24:55.984387+0000","last_became_peered":"2024-03-06T07:24:55.984387+0000","last_unstale":"2024-03-06T07:24:56.974043+0000","last_undegraded":"2024-03-06T07:24:56.974043+0000","last_fullsized":"2024-03-06T07:24:56.974043+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:29.767763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997093+0000","last_change":"2024-03-06T07:24:56.992237+0000","last_active":"2024-03-06T07:24:56.997093+0000","last_peered":"2024-03-06T07:24:56.997093+0000","last_clean":"2024-03-06T07:24:56.997093+0000","last_became_active":"2024-03-06T07:24:56.991930+0000","last_became_peered":"2024-03-06T07:24:56.991930+0000","last_unstale":"2024-03-06T07:24:56.997093+0000","last_undegraded":"2024-03-06T07:24:56.997093+0000","last_fullsized":"2024-03-06T07:24:56.997093+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:05.931371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.991270+0000","last_change":"2024-03-06T07:24:55.989801+0000","last_active":"2024-03-06T07:24:56.991270+0000","last_peered":"2024-03-06T07:24:56.991270+0000","last_clean":"2024-03-06T07:24:56.991270+0000","last_became_active":"2024-03-06T07:24:55.989517+0000","last_became_peered":"2024-03-06T07:24:55.989517+0000","last_unstale":"2024-03-06T07:24:56.991270+0000","last_undegraded":"2024-03-06T07:24:56.991270+0000","last_fullsized":"2024-03-06T07:24:56.991270+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:04.680394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.998384+0000","last_change":"2024-03-06T07:24:56.995538+0000","last_active":"2024-03-06T07:24:56.998384+0000","last_peered":"2024-03-06T07:24:56.998384+0000","last_clean":"2024-03-06T07:24:56.998384+0000","last_became_active":"2024-03-06T07:24:56.995268+0000","last_became_peered":"2024-03-06T07:24:56.995268+0000","last_unstale":"2024-03-06T07:24:56.998384+0000","last_undegraded":"2024-03-06T07:24:56.998384+0000","last_fullsized":"2024-03-06T07:24:56.998384+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:18.722511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.973438+0000","last_change":"2024-03-06T07:24:55.981966+0000","last_active":"2024-03-06T07:24:56.973438+0000","last_peered":"2024-03-06T07:24:56.973438+0000","last_clean":"2024-03-06T07:24:56.973438+0000","last_became_active":"2024-03-06T07:24:55.981707+0000","last_became_peered":"2024-03-06T07:24:55.981707+0000","last_unstale":"2024-03-06T07:24:56.973438+0000","last_undegraded":"2024-03-06T07:24:56.973438+0000","last_fullsized":"2024-03-06T07:24:56.973438+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:27.554510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997560+0000","last_change":"2024-03-06T07:24:56.995474+0000","last_active":"2024-03-06T07:24:56.997560+0000","last_peered":"2024-03-06T07:24:56.997560+0000","last_clean":"2024-03-06T07:24:56.997560+0000","last_became_active":"2024-03-06T07:24:56.995184+0000","last_became_peered":"2024-03-06T07:24:56.995184+0000","last_unstale":"2024-03-06T07:24:56.997560+0000","last_undegraded":"2024-03-06T07:24:56.997560+0000","last_fullsized":"2024-03-06T07:24:56.997560+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:00.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000774+0000","last_change":"2024-03-06T07:24:55.986704+0000","last_active":"2024-03-06T07:24:57.000774+0000","last_peered":"2024-03-06T07:24:57.000774+0000","last_clean":"2024-03-06T07:24:57.000774+0000","last_became_active":"2024-03-06T07:24:55.986416+0000","last_became_peered":"2024-03-06T07:24:55.986416+0000","last_unstale":"2024-03-06T07:24:57.000774+0000","last_undegraded":"2024-03-06T07:24:57.000774+0000","last_fullsized":"2024-03-06T07:24:57.000774+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:04.818179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":452,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979135+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:56.979135+0000","last_peered":"2024-03-06T07:24:56.979135+0000","last_clean":"2024-03-06T07:24:56.979135+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:56.979135+0000","last_undegraded":"2024-03-06T07:24:56.979135+0000","last_fullsized":"2024-03-06T07:24:56.979135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1623179,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2154496,"data_stored":1042172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574933,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1032811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1032811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574934,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1032811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33276,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338564,"statfs":{"total":96636764160,"available":96602689536,"internally_reserved":0,"allocated":2142208,"data_stored":1040068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1039978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1623179,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:00.183 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.321+0000 7f11e9ec2640 1 Processor -- start 2024-03-06T07:25:00.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.321+0000 7f11e9ec2640 1 -- start start 2024-03-06T07:25:00.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.322+0000 7f11e9ec2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e40c1aa0 0x7f11e41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.322+0000 7f11e9ec2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e40c28d0 con 0x7f11e4155a40 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.322+0000 7f11e9ec2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e40c2bb0 con 0x7f11e414f790 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.322+0000 7f11e9ec2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40c2680 con 0x7f11e40c1aa0 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.322+0000 7f11e3fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f11e4155a40 0x7f11e4155ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33124/0 (socket says 172.21.15.94:33124) 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.322+0000 7f11e3fff640 1 -- 172.21.15.94:0/85100423 learned_addr learned my addr 172.21.15.94:0/85100423 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.322+0000 7f11e2ffd640 1 --2- 172.21.15.94:0/85100423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e40c1aa0 0x7f11e41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e27fc640 1 -- 172.21.15.94:0/85100423 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44336805 0 0) 0x7f11e40c28d0 con 0x7f11e4155a40 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e27fc640 1 -- 172.21.15.94:0/85100423 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11cc003520 con 0x7f11e4155a40 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e27fc640 1 -- 172.21.15.94:0/85100423 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798539386 0 0) 0x7f11e40c2bb0 con 0x7f11e414f790 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e27fc640 1 -- 172.21.15.94:0/85100423 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e40c28d0 con 0x7f11e414f790 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e2ffd640 1 -- 172.21.15.94:0/85100423 >> v1:172.21.15.145:6789/0 conn(0x7f11e414f790 legacy=0x7f11e40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e2ffd640 1 -- 172.21.15.94:0/85100423 >> v1:172.21.15.94:6789/0 conn(0x7f11e4155a40 legacy=0x7f11e4155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e2ffd640 1 -- 172.21.15.94:0/85100423 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e40c2e90 con 0x7f11e40c1aa0 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.323+0000 7f11e2ffd640 1 --2- 172.21.15.94:0/85100423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e40c1aa0 0x7f11e41529d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097d0 tx=0x7f11d802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=788d91a8ca7759c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e27fc640 1 -- 172.21.15.94:0/85100423 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8035020 con 0x7f11e40c1aa0 2024-03-06T07:25:00.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e27fc640 1 -- 172.21.15.94:0/85100423 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8031890 con 0x7f11e40c1aa0 2024-03-06T07:25:00.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e27fc640 1 -- 172.21.15.94:0/85100423 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8031c40 con 0x7f11e40c1aa0 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e9ec2640 1 -- 172.21.15.94:0/85100423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e40c1aa0 msgr2=0x7f11e41529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/85100423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e40c1aa0 0x7f11e41529d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097d0 tx=0x7f11d802b930 comp rx=0 tx=0).stop 2024-03-06T07:25:00.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e9ec2640 1 -- 172.21.15.94:0/85100423 shutdown_connections 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/85100423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e40c1aa0 0x7f11e41529d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e9ec2640 1 -- 172.21.15.94:0/85100423 >> 172.21.15.94:0/85100423 conn(0x7f11e40c6710 msgr2=0x7f11e40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e9ec2640 1 -- 172.21.15.94:0/85100423 shutdown_connections 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.324+0000 7f11e9ec2640 1 -- 172.21.15.94:0/85100423 wait complete. 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.325+0000 7f11e9ec2640 1 Processor -- start 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.325+0000 7f11e9ec2640 1 -- start start 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e9ec2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e40c1aa0 0x7f11e4072760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e9ec2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11e414f790 0x7f11e414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e9ec2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e4155a40 0x7f11e414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e9ec2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40c5e30 con 0x7f11e414f790 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e9ec2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40c0d60 con 0x7f11e40c1aa0 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e9ec2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40c5fb0 con 0x7f11e4155a40 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11e414f790 0x7f11e414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11e414f790 0x7f11e414aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59020/0 (socket says 172.21.15.94:59020) 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e2ffd640 1 -- 172.21.15.94:0/1684149656 learned_addr learned my addr 172.21.15.94:0/1684149656 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e37fe640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e40c1aa0 0x7f11e4072760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e3fff640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e4155a40 0x7f11e414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e2ffd640 1 -- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e4155a40 msgr2=0x7f11e414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e2ffd640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e4155a40 0x7f11e414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.326+0000 7f11e2ffd640 1 -- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e40c1aa0 msgr2=0x7f11e4072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11e2ffd640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e40c1aa0 0x7f11e4072760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11e2ffd640 1 -- 172.21.15.94:0/1684149656 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e40a5e30 con 0x7f11e414f790 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11e2ffd640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11e414f790 0x7f11e414aa90 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f11d80098f0 tx=0x7f11d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11cbfff640 1 -- 172.21.15.94:0/1684149656 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8035040 con 0x7f11e414f790 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11cbfff640 1 -- 172.21.15.94:0/1684149656 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8030b10 con 0x7f11e414f790 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e41e9570 con 0x7f11e414f790 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11cbfff640 1 -- 172.21.15.94:0/1684149656 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8044430 con 0x7f11e414f790 2024-03-06T07:25:00.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.327+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e4073150 con 0x7f11e414f790 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.328+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e404f330 con 0x7f11e414f790 2024-03-06T07:25:00.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.328+0000 7f11cbfff640 1 -- 172.21.15.94:0/1684149656 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11d8038020 con 0x7f11e414f790 2024-03-06T07:25:00.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.329+0000 7f11cbfff640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.329+0000 7f11cbfff640 1 -- 172.21.15.94:0/1684149656 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f11d80ac6e0 con 0x7f11e414f790 2024-03-06T07:25:00.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.329+0000 7f11e37fe640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.329+0000 7f11e37fe640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11d006b4d0 0x7f11d006d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f11d4007fd0 tx=0x7f11d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.333+0000 7f11cbfff640 1 -- 172.21.15.94:0/1684149656 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11d802d040 con 0x7f11e414f790 2024-03-06T07:25:00.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.503+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11e40c1f40 con 0x7f11d006b4d0 2024-03-06T07:25:00.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.513+0000 7f11cbfff640 1 -- 172.21.15.94:0/1684149656 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252499 (secure 0 0 0) 0x7f11e40c1f40 con 0x7f11d006b4d0 2024-03-06T07:25:00.517 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:00.519 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:25:00.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11d006b4d0 msgr2=0x7f11d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11d006b4d0 0x7f11d006d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f11d4007fd0 tx=0x7f11d4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11e414f790 msgr2=0x7f11e414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11e414f790 0x7f11e414aa90 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f11d80098f0 tx=0x7f11d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 shutdown_connections 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11d006b4d0 0x7f11d006d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e4155a40 0x7f11e414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f11e414f790 0x7f11e414aa90 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 --2- 172.21.15.94:0/1684149656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e40c1aa0 0x7f11e4072760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 >> 172.21.15.94:0/1684149656 conn(0x7f11e40c6710 msgr2=0x7f11e40a9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 shutdown_connections 2024-03-06T07:25:00.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.516+0000 7f11e9ec2640 1 -- 172.21.15.94:0/1684149656 wait complete. 2024-03-06T07:25:00.529 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-03-06T07:24:58.723054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":266252,"kb_used_data":16792,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754708468,"statfs":{"total":773094113280,"available":772821471232,"internally_reserved":0,"allocated":17195008,"data_stored":9467009,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000908"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004303+0000","last_change":"2024-03-06T07:24:56.997658+0000","last_active":"2024-03-06T07:24:57.004303+0000","last_peered":"2024-03-06T07:24:57.004303+0000","last_clean":"2024-03-06T07:24:57.004303+0000","last_became_active":"2024-03-06T07:24:56.996963+0000","last_became_peered":"2024-03-06T07:24:56.996963+0000","last_unstale":"2024-03-06T07:24:57.004303+0000","last_undegraded":"2024-03-06T07:24:57.004303+0000","last_fullsized":"2024-03-06T07:24:57.004303+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:05.822744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996018+0000","last_change":"2024-03-06T07:24:56.994300+0000","last_active":"2024-03-06T07:24:56.996018+0000","last_peered":"2024-03-06T07:24:56.996018+0000","last_clean":"2024-03-06T07:24:56.996018+0000","last_became_active":"2024-03-06T07:24:56.994019+0000","last_became_peered":"2024-03-06T07:24:56.994019+0000","last_unstale":"2024-03-06T07:24:56.996018+0000","last_undegraded":"2024-03-06T07:24:56.996018+0000","last_fullsized":"2024-03-06T07:24:56.996018+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:39.647212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996030+0000","last_change":"2024-03-06T07:24:56.994367+0000","last_active":"2024-03-06T07:24:56.996030+0000","last_peered":"2024-03-06T07:24:56.996030+0000","last_clean":"2024-03-06T07:24:56.996030+0000","last_became_active":"2024-03-06T07:24:56.994110+0000","last_became_peered":"2024-03-06T07:24:56.994110+0000","last_unstale":"2024-03-06T07:24:56.996030+0000","last_undegraded":"2024-03-06T07:24:56.996030+0000","last_fullsized":"2024-03-06T07:24:56.996030+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:22.730963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996514+0000","last_change":"2024-03-06T07:24:56.988648+0000","last_active":"2024-03-06T07:24:56.996514+0000","last_peered":"2024-03-06T07:24:56.996514+0000","last_clean":"2024-03-06T07:24:56.996514+0000","last_became_active":"2024-03-06T07:24:56.988320+0000","last_became_peered":"2024-03-06T07:24:56.988320+0000","last_unstale":"2024-03-06T07:24:56.996514+0000","last_undegraded":"2024-03-06T07:24:56.996514+0000","last_fullsized":"2024-03-06T07:24:56.996514+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:30.772328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986603+0000","last_change":"2024-03-06T07:24:56.984517+0000","last_active":"2024-03-06T07:24:56.986603+0000","last_peered":"2024-03-06T07:24:56.986603+0000","last_clean":"2024-03-06T07:24:56.986603+0000","last_became_active":"2024-03-06T07:24:56.984245+0000","last_became_peered":"2024-03-06T07:24:56.984245+0000","last_unstale":"2024-03-06T07:24:56.986603+0000","last_undegraded":"2024-03-06T07:24:56.986603+0000","last_fullsized":"2024-03-06T07:24:56.986603+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:25.308202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995970+0000","last_change":"2024-03-06T07:24:56.988686+0000","last_active":"2024-03-06T07:24:56.995970+0000","last_peered":"2024-03-06T07:24:56.995970+0000","last_clean":"2024-03-06T07:24:56.995970+0000","last_became_active":"2024-03-06T07:24:56.988398+0000","last_became_peered":"2024-03-06T07:24:56.988398+0000","last_unstale":"2024-03-06T07:24:56.995970+0000","last_undegraded":"2024-03-06T07:24:56.995970+0000","last_fullsized":"2024-03-06T07:24:56.995970+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:42.942153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984836+0000","last_change":"2024-03-06T07:24:56.983905+0000","last_active":"2024-03-06T07:24:56.984836+0000","last_peered":"2024-03-06T07:24:56.984836+0000","last_clean":"2024-03-06T07:24:56.984836+0000","last_became_active":"2024-03-06T07:24:56.983619+0000","last_became_peered":"2024-03-06T07:24:56.983619+0000","last_unstale":"2024-03-06T07:24:56.984836+0000","last_undegraded":"2024-03-06T07:24:56.984836+0000","last_fullsized":"2024-03-06T07:24:56.984836+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:59.082721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.994107+0000","last_change":"2024-03-06T07:24:56.991468+0000","last_active":"2024-03-06T07:24:56.994107+0000","last_peered":"2024-03-06T07:24:56.994107+0000","last_clean":"2024-03-06T07:24:56.994107+0000","last_became_active":"2024-03-06T07:24:56.990805+0000","last_became_peered":"2024-03-06T07:24:56.990805+0000","last_unstale":"2024-03-06T07:24:56.994107+0000","last_undegraded":"2024-03-06T07:24:56.994107+0000","last_fullsized":"2024-03-06T07:24:56.994107+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:54.782835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995803+0000","last_change":"2024-03-06T07:24:56.994841+0000","last_active":"2024-03-06T07:24:56.995803+0000","last_peered":"2024-03-06T07:24:56.995803+0000","last_clean":"2024-03-06T07:24:56.995803+0000","last_became_active":"2024-03-06T07:24:56.994540+0000","last_became_peered":"2024-03-06T07:24:56.994540+0000","last_unstale":"2024-03-06T07:24:56.995803+0000","last_undegraded":"2024-03-06T07:24:56.995803+0000","last_fullsized":"2024-03-06T07:24:56.995803+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:21.626315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989239+0000","last_change":"2024-03-06T07:24:56.981436+0000","last_active":"2024-03-06T07:24:56.989239+0000","last_peered":"2024-03-06T07:24:56.989239+0000","last_clean":"2024-03-06T07:24:56.989239+0000","last_became_active":"2024-03-06T07:24:56.981153+0000","last_became_peered":"2024-03-06T07:24:56.981153+0000","last_unstale":"2024-03-06T07:24:56.989239+0000","last_undegraded":"2024-03-06T07:24:56.989239+0000","last_fullsized":"2024-03-06T07:24:56.989239+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:44.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.998326+0000","last_change":"2024-03-06T07:24:56.996355+0000","last_active":"2024-03-06T07:24:56.998326+0000","last_peered":"2024-03-06T07:24:56.998326+0000","last_clean":"2024-03-06T07:24:56.998326+0000","last_became_active":"2024-03-06T07:24:56.996096+0000","last_became_peered":"2024-03-06T07:24:56.996096+0000","last_unstale":"2024-03-06T07:24:56.998326+0000","last_undegraded":"2024-03-06T07:24:56.998326+0000","last_fullsized":"2024-03-06T07:24:56.998326+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:27.073130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984328+0000","last_change":"2024-03-06T07:24:56.979974+0000","last_active":"2024-03-06T07:24:56.984328+0000","last_peered":"2024-03-06T07:24:56.984328+0000","last_clean":"2024-03-06T07:24:56.984328+0000","last_became_active":"2024-03-06T07:24:56.979695+0000","last_became_peered":"2024-03-06T07:24:56.979695+0000","last_unstale":"2024-03-06T07:24:56.984328+0000","last_undegraded":"2024-03-06T07:24:56.984328+0000","last_fullsized":"2024-03-06T07:24:56.984328+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:27.776102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996075+0000","last_change":"2024-03-06T07:24:56.990237+0000","last_active":"2024-03-06T07:24:56.996075+0000","last_peered":"2024-03-06T07:24:56.996075+0000","last_clean":"2024-03-06T07:24:56.996075+0000","last_became_active":"2024-03-06T07:24:56.989981+0000","last_became_peered":"2024-03-06T07:24:56.989981+0000","last_unstale":"2024-03-06T07:24:56.996075+0000","last_undegraded":"2024-03-06T07:24:56.996075+0000","last_fullsized":"2024-03-06T07:24:56.996075+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:32.752131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997558+0000","last_change":"2024-03-06T07:24:56.995253+0000","last_active":"2024-03-06T07:24:56.997558+0000","last_peered":"2024-03-06T07:24:56.997558+0000","last_clean":"2024-03-06T07:24:56.997558+0000","last_became_active":"2024-03-06T07:24:56.994735+0000","last_became_peered":"2024-03-06T07:24:56.994735+0000","last_unstale":"2024-03-06T07:24:56.997558+0000","last_undegraded":"2024-03-06T07:24:56.997558+0000","last_fullsized":"2024-03-06T07:24:56.997558+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:20.534691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983390+0000","last_change":"2024-03-06T07:24:56.982130+0000","last_active":"2024-03-06T07:24:56.983390+0000","last_peered":"2024-03-06T07:24:56.983390+0000","last_clean":"2024-03-06T07:24:56.983390+0000","last_became_active":"2024-03-06T07:24:56.981779+0000","last_became_peered":"2024-03-06T07:24:56.981779+0000","last_unstale":"2024-03-06T07:24:56.983390+0000","last_undegraded":"2024-03-06T07:24:56.983390+0000","last_fullsized":"2024-03-06T07:24:56.983390+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:26.760058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.987188+0000","last_change":"2024-03-06T07:24:56.985435+0000","last_active":"2024-03-06T07:24:56.987188+0000","last_peered":"2024-03-06T07:24:56.987188+0000","last_clean":"2024-03-06T07:24:56.987188+0000","last_became_active":"2024-03-06T07:24:56.985143+0000","last_became_peered":"2024-03-06T07:24:56.985143+0000","last_unstale":"2024-03-06T07:24:56.987188+0000","last_undegraded":"2024-03-06T07:24:56.987188+0000","last_fullsized":"2024-03-06T07:24:56.987188+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:21.626315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989377+0000","last_change":"2024-03-06T07:24:56.988589+0000","last_active":"2024-03-06T07:24:56.989377+0000","last_peered":"2024-03-06T07:24:56.989377+0000","last_clean":"2024-03-06T07:24:56.989377+0000","last_became_active":"2024-03-06T07:24:56.988156+0000","last_became_peered":"2024-03-06T07:24:56.988156+0000","last_unstale":"2024-03-06T07:24:56.989377+0000","last_undegraded":"2024-03-06T07:24:56.989377+0000","last_fullsized":"2024-03-06T07:24:56.989377+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:01.751115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986580+0000","last_change":"2024-03-06T07:24:56.984877+0000","last_active":"2024-03-06T07:24:56.986580+0000","last_peered":"2024-03-06T07:24:56.986580+0000","last_clean":"2024-03-06T07:24:56.986580+0000","last_became_active":"2024-03-06T07:24:56.984621+0000","last_became_peered":"2024-03-06T07:24:56.984621+0000","last_unstale":"2024-03-06T07:24:56.986580+0000","last_undegraded":"2024-03-06T07:24:56.986580+0000","last_fullsized":"2024-03-06T07:24:56.986580+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:05.931371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989442+0000","last_change":"2024-03-06T07:24:56.982689+0000","last_active":"2024-03-06T07:24:56.989442+0000","last_peered":"2024-03-06T07:24:56.989442+0000","last_clean":"2024-03-06T07:24:56.989442+0000","last_became_active":"2024-03-06T07:24:56.982383+0000","last_became_peered":"2024-03-06T07:24:56.982383+0000","last_unstale":"2024-03-06T07:24:56.989442+0000","last_undegraded":"2024-03-06T07:24:56.989442+0000","last_fullsized":"2024-03-06T07:24:56.989442+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:25.308202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.993525+0000","last_change":"2024-03-06T07:24:56.987455+0000","last_active":"2024-03-06T07:24:56.993525+0000","last_peered":"2024-03-06T07:24:56.993525+0000","last_clean":"2024-03-06T07:24:56.993525+0000","last_became_active":"2024-03-06T07:24:56.986523+0000","last_became_peered":"2024-03-06T07:24:56.986523+0000","last_unstale":"2024-03-06T07:24:56.993525+0000","last_undegraded":"2024-03-06T07:24:56.993525+0000","last_fullsized":"2024-03-06T07:24:56.993525+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:54.994639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.993919+0000","last_change":"2024-03-06T07:24:56.989312+0000","last_active":"2024-03-06T07:24:56.993919+0000","last_peered":"2024-03-06T07:24:56.993919+0000","last_clean":"2024-03-06T07:24:56.993919+0000","last_became_active":"2024-03-06T07:24:56.988606+0000","last_became_peered":"2024-03-06T07:24:56.988606+0000","last_unstale":"2024-03-06T07:24:56.993919+0000","last_undegraded":"2024-03-06T07:24:56.993919+0000","last_fullsized":"2024-03-06T07:24:56.993919+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:32:32.772930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997152+0000","last_change":"2024-03-06T07:24:56.994742+0000","last_active":"2024-03-06T07:24:56.997152+0000","last_peered":"2024-03-06T07:24:56.997152+0000","last_clean":"2024-03-06T07:24:56.997152+0000","last_became_active":"2024-03-06T07:24:56.994376+0000","last_became_peered":"2024-03-06T07:24:56.994376+0000","last_unstale":"2024-03-06T07:24:56.997152+0000","last_undegraded":"2024-03-06T07:24:56.997152+0000","last_fullsized":"2024-03-06T07:24:56.997152+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:53.646560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995382+0000","last_change":"2024-03-06T07:24:56.993067+0000","last_active":"2024-03-06T07:24:56.995382+0000","last_peered":"2024-03-06T07:24:56.995382+0000","last_clean":"2024-03-06T07:24:56.995382+0000","last_became_active":"2024-03-06T07:24:56.992802+0000","last_became_peered":"2024-03-06T07:24:56.992802+0000","last_unstale":"2024-03-06T07:24:56.995382+0000","last_undegraded":"2024-03-06T07:24:56.995382+0000","last_fullsized":"2024-03-06T07:24:56.995382+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:26.114882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.994920+0000","last_change":"2024-03-06T07:24:56.993015+0000","last_active":"2024-03-06T07:24:56.994920+0000","last_peered":"2024-03-06T07:24:56.994920+0000","last_clean":"2024-03-06T07:24:56.994920+0000","last_became_active":"2024-03-06T07:24:56.992714+0000","last_became_peered":"2024-03-06T07:24:56.992714+0000","last_unstale":"2024-03-06T07:24:56.994920+0000","last_undegraded":"2024-03-06T07:24:56.994920+0000","last_fullsized":"2024-03-06T07:24:56.994920+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:25.589832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986574+0000","last_change":"2024-03-06T07:24:56.985846+0000","last_active":"2024-03-06T07:24:56.986574+0000","last_peered":"2024-03-06T07:24:56.986574+0000","last_clean":"2024-03-06T07:24:56.986574+0000","last_became_active":"2024-03-06T07:24:56.985362+0000","last_became_peered":"2024-03-06T07:24:56.985362+0000","last_unstale":"2024-03-06T07:24:56.986574+0000","last_undegraded":"2024-03-06T07:24:56.986574+0000","last_fullsized":"2024-03-06T07:24:56.986574+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:18.319695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.006922+0000","last_change":"2024-03-06T07:24:57.000322+0000","last_active":"2024-03-06T07:24:57.006922+0000","last_peered":"2024-03-06T07:24:57.006922+0000","last_clean":"2024-03-06T07:24:57.006922+0000","last_became_active":"2024-03-06T07:24:57.000042+0000","last_became_peered":"2024-03-06T07:24:57.000042+0000","last_unstale":"2024-03-06T07:24:57.006922+0000","last_undegraded":"2024-03-06T07:24:57.006922+0000","last_fullsized":"2024-03-06T07:24:57.006922+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:30.772328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986679+0000","last_change":"2024-03-06T07:24:56.984716+0000","last_active":"2024-03-06T07:24:56.986679+0000","last_peered":"2024-03-06T07:24:56.986679+0000","last_clean":"2024-03-06T07:24:56.986679+0000","last_became_active":"2024-03-06T07:24:56.984487+0000","last_became_peered":"2024-03-06T07:24:56.984487+0000","last_unstale":"2024-03-06T07:24:56.986679+0000","last_undegraded":"2024-03-06T07:24:56.986679+0000","last_fullsized":"2024-03-06T07:24:56.986679+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:47.734879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984569+0000","last_change":"2024-03-06T07:24:56.982818+0000","last_active":"2024-03-06T07:24:56.984569+0000","last_peered":"2024-03-06T07:24:56.984569+0000","last_clean":"2024-03-06T07:24:56.984569+0000","last_became_active":"2024-03-06T07:24:56.982533+0000","last_became_peered":"2024-03-06T07:24:56.982533+0000","last_unstale":"2024-03-06T07:24:56.984569+0000","last_undegraded":"2024-03-06T07:24:56.984569+0000","last_fullsized":"2024-03-06T07:24:56.984569+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:44.201229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989722+0000","last_change":"2024-03-06T07:24:56.988856+0000","last_active":"2024-03-06T07:24:56.989722+0000","last_peered":"2024-03-06T07:24:56.989722+0000","last_clean":"2024-03-06T07:24:56.989722+0000","last_became_active":"2024-03-06T07:24:56.988096+0000","last_became_peered":"2024-03-06T07:24:56.988096+0000","last_unstale":"2024-03-06T07:24:56.989722+0000","last_undegraded":"2024-03-06T07:24:56.989722+0000","last_fullsized":"2024-03-06T07:24:56.989722+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:36.204436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983239+0000","last_change":"2024-03-06T07:24:56.980859+0000","last_active":"2024-03-06T07:24:56.983239+0000","last_peered":"2024-03-06T07:24:56.983239+0000","last_clean":"2024-03-06T07:24:56.983239+0000","last_became_active":"2024-03-06T07:24:56.980578+0000","last_became_peered":"2024-03-06T07:24:56.980578+0000","last_unstale":"2024-03-06T07:24:56.983239+0000","last_undegraded":"2024-03-06T07:24:56.983239+0000","last_fullsized":"2024-03-06T07:24:56.983239+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:54.163100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996033+0000","last_change":"2024-03-06T07:24:56.991368+0000","last_active":"2024-03-06T07:24:56.996033+0000","last_peered":"2024-03-06T07:24:56.996033+0000","last_clean":"2024-03-06T07:24:56.996033+0000","last_became_active":"2024-03-06T07:24:56.991027+0000","last_became_peered":"2024-03-06T07:24:56.991027+0000","last_unstale":"2024-03-06T07:24:56.996033+0000","last_undegraded":"2024-03-06T07:24:56.996033+0000","last_fullsized":"2024-03-06T07:24:56.996033+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:30.953134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.996104+0000","last_change":"2024-03-06T07:24:56.988597+0000","last_active":"2024-03-06T07:24:56.996104+0000","last_peered":"2024-03-06T07:24:56.996104+0000","last_clean":"2024-03-06T07:24:56.996104+0000","last_became_active":"2024-03-06T07:24:56.988286+0000","last_became_peered":"2024-03-06T07:24:56.988286+0000","last_unstale":"2024-03-06T07:24:56.996104+0000","last_undegraded":"2024-03-06T07:24:56.996104+0000","last_fullsized":"2024-03-06T07:24:56.996104+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:05.822744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.993575+0000","last_change":"2024-03-06T07:24:56.992588+0000","last_active":"2024-03-06T07:24:56.993575+0000","last_peered":"2024-03-06T07:24:56.993575+0000","last_clean":"2024-03-06T07:24:56.993575+0000","last_became_active":"2024-03-06T07:24:56.992317+0000","last_became_peered":"2024-03-06T07:24:56.992317+0000","last_unstale":"2024-03-06T07:24:56.993575+0000","last_undegraded":"2024-03-06T07:24:56.993575+0000","last_fullsized":"2024-03-06T07:24:56.993575+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:43.253268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"169'4","reported_seq":16,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002962+0000","last_change":"2024-03-06T07:24:55.983989+0000","last_active":"2024-03-06T07:24:57.002962+0000","last_peered":"2024-03-06T07:24:57.002962+0000","last_clean":"2024-03-06T07:24:57.002962+0000","last_became_active":"2024-03-06T07:24:55.983710+0000","last_became_peered":"2024-03-06T07:24:55.983710+0000","last_unstale":"2024-03-06T07:24:57.002962+0000","last_undegraded":"2024-03-06T07:24:57.002962+0000","last_fullsized":"2024-03-06T07:24:57.002962+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:49.896080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997412+0000","last_change":"2024-03-06T07:24:56.993002+0000","last_active":"2024-03-06T07:24:56.997412+0000","last_peered":"2024-03-06T07:24:56.997412+0000","last_clean":"2024-03-06T07:24:56.997412+0000","last_became_active":"2024-03-06T07:24:56.992733+0000","last_became_peered":"2024-03-06T07:24:56.992733+0000","last_unstale":"2024-03-06T07:24:56.997412+0000","last_undegraded":"2024-03-06T07:24:56.997412+0000","last_fullsized":"2024-03-06T07:24:56.997412+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:22.730138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004254+0000","last_change":"2024-03-06T07:24:55.988062+0000","last_active":"2024-03-06T07:24:57.004254+0000","last_peered":"2024-03-06T07:24:57.004254+0000","last_clean":"2024-03-06T07:24:57.004254+0000","last_became_active":"2024-03-06T07:24:55.987789+0000","last_became_peered":"2024-03-06T07:24:55.987789+0000","last_unstale":"2024-03-06T07:24:57.004254+0000","last_undegraded":"2024-03-06T07:24:57.004254+0000","last_fullsized":"2024-03-06T07:24:57.004254+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:46.326046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989121+0000","last_change":"2024-03-06T07:24:56.980031+0000","last_active":"2024-03-06T07:24:56.989121+0000","last_peered":"2024-03-06T07:24:56.989121+0000","last_clean":"2024-03-06T07:24:56.989121+0000","last_became_active":"2024-03-06T07:24:56.979758+0000","last_became_peered":"2024-03-06T07:24:56.979758+0000","last_unstale":"2024-03-06T07:24:56.989121+0000","last_undegraded":"2024-03-06T07:24:56.989121+0000","last_fullsized":"2024-03-06T07:24:56.989121+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:44.974416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000295+0000","last_change":"2024-03-06T07:24:55.981064+0000","last_active":"2024-03-06T07:24:57.000295+0000","last_peered":"2024-03-06T07:24:57.000295+0000","last_clean":"2024-03-06T07:24:57.000295+0000","last_became_active":"2024-03-06T07:24:55.980522+0000","last_became_peered":"2024-03-06T07:24:55.980522+0000","last_unstale":"2024-03-06T07:24:57.000295+0000","last_undegraded":"2024-03-06T07:24:57.000295+0000","last_fullsized":"2024-03-06T07:24:57.000295+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:49:20.101019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983650+0000","last_change":"2024-03-06T07:24:56.978181+0000","last_active":"2024-03-06T07:24:56.983650+0000","last_peered":"2024-03-06T07:24:56.983650+0000","last_clean":"2024-03-06T07:24:56.983650+0000","last_became_active":"2024-03-06T07:24:56.977923+0000","last_became_peered":"2024-03-06T07:24:56.977923+0000","last_unstale":"2024-03-06T07:24:56.983650+0000","last_undegraded":"2024-03-06T07:24:56.983650+0000","last_fullsized":"2024-03-06T07:24:56.983650+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:01.726864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002188+0000","last_change":"2024-03-06T07:24:55.986832+0000","last_active":"2024-03-06T07:24:57.002188+0000","last_peered":"2024-03-06T07:24:57.002188+0000","last_clean":"2024-03-06T07:24:57.002188+0000","last_became_active":"2024-03-06T07:24:55.986453+0000","last_became_peered":"2024-03-06T07:24:55.986453+0000","last_unstale":"2024-03-06T07:24:57.002188+0000","last_undegraded":"2024-03-06T07:24:57.002188+0000","last_fullsized":"2024-03-06T07:24:57.002188+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:31.747566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983461+0000","last_change":"2024-03-06T07:24:56.980858+0000","last_active":"2024-03-06T07:24:56.983461+0000","last_peered":"2024-03-06T07:24:56.983461+0000","last_clean":"2024-03-06T07:24:56.983461+0000","last_became_active":"2024-03-06T07:24:56.980554+0000","last_became_peered":"2024-03-06T07:24:56.980554+0000","last_unstale":"2024-03-06T07:24:56.983461+0000","last_undegraded":"2024-03-06T07:24:56.983461+0000","last_fullsized":"2024-03-06T07:24:56.983461+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:36.182573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002004+0000","last_change":"2024-03-06T07:24:55.985565+0000","last_active":"2024-03-06T07:24:57.002004+0000","last_peered":"2024-03-06T07:24:57.002004+0000","last_clean":"2024-03-06T07:24:57.002004+0000","last_became_active":"2024-03-06T07:24:55.985278+0000","last_became_peered":"2024-03-06T07:24:55.985278+0000","last_unstale":"2024-03-06T07:24:57.002004+0000","last_undegraded":"2024-03-06T07:24:57.002004+0000","last_fullsized":"2024-03-06T07:24:57.002004+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:48.369892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.983186+0000","last_change":"2024-03-06T07:24:56.981707+0000","last_active":"2024-03-06T07:24:56.983186+0000","last_peered":"2024-03-06T07:24:56.983186+0000","last_clean":"2024-03-06T07:24:56.983186+0000","last_became_active":"2024-03-06T07:24:56.981436+0000","last_became_peered":"2024-03-06T07:24:56.981436+0000","last_unstale":"2024-03-06T07:24:56.983186+0000","last_undegraded":"2024-03-06T07:24:56.983186+0000","last_fullsized":"2024-03-06T07:24:56.983186+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:15.451504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.978457+0000","last_change":"2024-03-06T07:24:55.987161+0000","last_active":"2024-03-06T07:24:56.978457+0000","last_peered":"2024-03-06T07:24:56.978457+0000","last_clean":"2024-03-06T07:24:56.978457+0000","last_became_active":"2024-03-06T07:24:55.986697+0000","last_became_peered":"2024-03-06T07:24:55.986697+0000","last_unstale":"2024-03-06T07:24:56.978457+0000","last_undegraded":"2024-03-06T07:24:56.978457+0000","last_fullsized":"2024-03-06T07:24:56.978457+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:21.725573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989695+0000","last_change":"2024-03-06T07:24:56.988844+0000","last_active":"2024-03-06T07:24:56.989695+0000","last_peered":"2024-03-06T07:24:56.989695+0000","last_clean":"2024-03-06T07:24:56.989695+0000","last_became_active":"2024-03-06T07:24:56.988289+0000","last_became_peered":"2024-03-06T07:24:56.988289+0000","last_unstale":"2024-03-06T07:24:56.989695+0000","last_undegraded":"2024-03-06T07:24:56.989695+0000","last_fullsized":"2024-03-06T07:24:56.989695+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:40.896460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.978355+0000","last_change":"2024-03-06T07:24:55.981034+0000","last_active":"2024-03-06T07:24:56.978355+0000","last_peered":"2024-03-06T07:24:56.978355+0000","last_clean":"2024-03-06T07:24:56.978355+0000","last_became_active":"2024-03-06T07:24:55.980755+0000","last_became_peered":"2024-03-06T07:24:55.980755+0000","last_unstale":"2024-03-06T07:24:56.978355+0000","last_undegraded":"2024-03-06T07:24:56.978355+0000","last_fullsized":"2024-03-06T07:24:56.978355+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:00.746550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.991299+0000","last_change":"2024-03-06T07:24:56.987558+0000","last_active":"2024-03-06T07:24:56.991299+0000","last_peered":"2024-03-06T07:24:56.991299+0000","last_clean":"2024-03-06T07:24:56.991299+0000","last_became_active":"2024-03-06T07:24:56.987287+0000","last_became_peered":"2024-03-06T07:24:56.987287+0000","last_unstale":"2024-03-06T07:24:56.991299+0000","last_undegraded":"2024-03-06T07:24:56.991299+0000","last_fullsized":"2024-03-06T07:24:56.991299+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:30.927206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974454+0000","last_change":"2024-03-06T07:24:55.982490+0000","last_active":"2024-03-06T07:24:56.974454+0000","last_peered":"2024-03-06T07:24:56.974454+0000","last_clean":"2024-03-06T07:24:56.974454+0000","last_became_active":"2024-03-06T07:24:55.981754+0000","last_became_peered":"2024-03-06T07:24:55.981754+0000","last_unstale":"2024-03-06T07:24:56.974454+0000","last_undegraded":"2024-03-06T07:24:56.974454+0000","last_fullsized":"2024-03-06T07:24:56.974454+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:10.143335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.987937+0000","last_change":"2024-03-06T07:24:56.987377+0000","last_active":"2024-03-06T07:24:56.987937+0000","last_peered":"2024-03-06T07:24:56.987937+0000","last_clean":"2024-03-06T07:24:56.987937+0000","last_became_active":"2024-03-06T07:24:56.987097+0000","last_became_peered":"2024-03-06T07:24:56.987097+0000","last_unstale":"2024-03-06T07:24:56.987937+0000","last_undegraded":"2024-03-06T07:24:56.987937+0000","last_fullsized":"2024-03-06T07:24:56.987937+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:25.903162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979361+0000","last_change":"2024-03-06T07:24:55.986301+0000","last_active":"2024-03-06T07:24:56.979361+0000","last_peered":"2024-03-06T07:24:56.979361+0000","last_clean":"2024-03-06T07:24:56.979361+0000","last_became_active":"2024-03-06T07:24:55.985972+0000","last_became_peered":"2024-03-06T07:24:55.985972+0000","last_unstale":"2024-03-06T07:24:56.979361+0000","last_undegraded":"2024-03-06T07:24:56.979361+0000","last_fullsized":"2024-03-06T07:24:56.979361+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:59.271876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997353+0000","last_change":"2024-03-06T07:24:56.995552+0000","last_active":"2024-03-06T07:24:56.997353+0000","last_peered":"2024-03-06T07:24:56.997353+0000","last_clean":"2024-03-06T07:24:56.997353+0000","last_became_active":"2024-03-06T07:24:56.995297+0000","last_became_peered":"2024-03-06T07:24:56.995297+0000","last_unstale":"2024-03-06T07:24:56.997353+0000","last_undegraded":"2024-03-06T07:24:56.997353+0000","last_fullsized":"2024-03-06T07:24:56.997353+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:25.903162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"169'3","reported_seq":15,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.001122+0000","last_change":"2024-03-06T07:24:55.987935+0000","last_active":"2024-03-06T07:24:57.001122+0000","last_peered":"2024-03-06T07:24:57.001122+0000","last_clean":"2024-03-06T07:24:57.001122+0000","last_became_active":"2024-03-06T07:24:55.987636+0000","last_became_peered":"2024-03-06T07:24:55.987636+0000","last_unstale":"2024-03-06T07:24:57.001122+0000","last_undegraded":"2024-03-06T07:24:57.001122+0000","last_fullsized":"2024-03-06T07:24:57.001122+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:41.937588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.984659+0000","last_change":"2024-03-06T07:24:56.982361+0000","last_active":"2024-03-06T07:24:56.984659+0000","last_peered":"2024-03-06T07:24:56.984659+0000","last_clean":"2024-03-06T07:24:56.984659+0000","last_became_active":"2024-03-06T07:24:56.982080+0000","last_became_peered":"2024-03-06T07:24:56.982080+0000","last_unstale":"2024-03-06T07:24:56.984659+0000","last_undegraded":"2024-03-06T07:24:56.984659+0000","last_fullsized":"2024-03-06T07:24:56.984659+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:44.974416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"169'6","reported_seq":18,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.010073+0000","last_change":"2024-03-06T07:24:55.982194+0000","last_active":"2024-03-06T07:24:57.010073+0000","last_peered":"2024-03-06T07:24:57.010073+0000","last_clean":"2024-03-06T07:24:57.010073+0000","last_became_active":"2024-03-06T07:24:55.981938+0000","last_became_peered":"2024-03-06T07:24:55.981938+0000","last_unstale":"2024-03-06T07:24:57.010073+0000","last_undegraded":"2024-03-06T07:24:57.010073+0000","last_fullsized":"2024-03-06T07:24:57.010073+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:14.310018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.977721+0000","last_change":"2024-03-06T07:24:56.975877+0000","last_active":"2024-03-06T07:24:56.977721+0000","last_peered":"2024-03-06T07:24:56.977721+0000","last_clean":"2024-03-06T07:24:56.977721+0000","last_became_active":"2024-03-06T07:24:56.975601+0000","last_became_peered":"2024-03-06T07:24:56.975601+0000","last_unstale":"2024-03-06T07:24:56.977721+0000","last_undegraded":"2024-03-06T07:24:56.977721+0000","last_fullsized":"2024-03-06T07:24:56.977721+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:56.173817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004660+0000","last_change":"2024-03-06T07:24:55.985953+0000","last_active":"2024-03-06T07:24:57.004660+0000","last_peered":"2024-03-06T07:24:57.004660+0000","last_clean":"2024-03-06T07:24:57.004660+0000","last_became_active":"2024-03-06T07:24:55.985645+0000","last_became_peered":"2024-03-06T07:24:55.985645+0000","last_unstale":"2024-03-06T07:24:57.004660+0000","last_undegraded":"2024-03-06T07:24:57.004660+0000","last_fullsized":"2024-03-06T07:24:57.004660+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:14.926934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986656+0000","last_change":"2024-03-06T07:24:56.984663+0000","last_active":"2024-03-06T07:24:56.986656+0000","last_peered":"2024-03-06T07:24:56.986656+0000","last_clean":"2024-03-06T07:24:56.986656+0000","last_became_active":"2024-03-06T07:24:56.984371+0000","last_became_peered":"2024-03-06T07:24:56.984371+0000","last_unstale":"2024-03-06T07:24:56.986656+0000","last_undegraded":"2024-03-06T07:24:56.986656+0000","last_fullsized":"2024-03-06T07:24:56.986656+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:36.204436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979918+0000","last_change":"2024-03-06T07:24:55.981303+0000","last_active":"2024-03-06T07:24:56.979918+0000","last_peered":"2024-03-06T07:24:56.979918+0000","last_clean":"2024-03-06T07:24:56.979918+0000","last_became_active":"2024-03-06T07:24:55.981000+0000","last_became_peered":"2024-03-06T07:24:55.981000+0000","last_unstale":"2024-03-06T07:24:56.979918+0000","last_undegraded":"2024-03-06T07:24:56.979918+0000","last_fullsized":"2024-03-06T07:24:56.979918+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:39.891895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.980568+0000","last_change":"2024-03-06T07:24:56.977228+0000","last_active":"2024-03-06T07:24:56.980568+0000","last_peered":"2024-03-06T07:24:56.980568+0000","last_clean":"2024-03-06T07:24:56.980568+0000","last_became_active":"2024-03-06T07:24:56.976967+0000","last_became_peered":"2024-03-06T07:24:56.976967+0000","last_unstale":"2024-03-06T07:24:56.980568+0000","last_undegraded":"2024-03-06T07:24:56.980568+0000","last_fullsized":"2024-03-06T07:24:56.980568+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:58.830269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.973804+0000","last_change":"2024-03-06T07:24:55.983949+0000","last_active":"2024-03-06T07:24:56.973804+0000","last_peered":"2024-03-06T07:24:56.973804+0000","last_clean":"2024-03-06T07:24:56.973804+0000","last_became_active":"2024-03-06T07:24:55.983631+0000","last_became_peered":"2024-03-06T07:24:55.983631+0000","last_unstale":"2024-03-06T07:24:56.973804+0000","last_undegraded":"2024-03-06T07:24:56.973804+0000","last_fullsized":"2024-03-06T07:24:56.973804+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:59.271876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000535+0000","last_change":"2024-03-06T07:24:56.995337+0000","last_active":"2024-03-06T07:24:57.000535+0000","last_peered":"2024-03-06T07:24:57.000535+0000","last_clean":"2024-03-06T07:24:57.000535+0000","last_became_active":"2024-03-06T07:24:56.994857+0000","last_became_peered":"2024-03-06T07:24:56.994857+0000","last_unstale":"2024-03-06T07:24:57.000535+0000","last_undegraded":"2024-03-06T07:24:57.000535+0000","last_fullsized":"2024-03-06T07:24:57.000535+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:18.319695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"169'8","reported_seq":20,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.007191+0000","last_change":"2024-03-06T07:24:55.981434+0000","last_active":"2024-03-06T07:24:57.007191+0000","last_peered":"2024-03-06T07:24:57.007191+0000","last_clean":"2024-03-06T07:24:57.007191+0000","last_became_active":"2024-03-06T07:24:55.981144+0000","last_became_peered":"2024-03-06T07:24:55.981144+0000","last_unstale":"2024-03-06T07:24:57.007191+0000","last_undegraded":"2024-03-06T07:24:57.007191+0000","last_fullsized":"2024-03-06T07:24:57.007191+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:58.558844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997298+0000","last_change":"2024-03-06T07:24:56.991808+0000","last_active":"2024-03-06T07:24:56.997298+0000","last_peered":"2024-03-06T07:24:56.997298+0000","last_clean":"2024-03-06T07:24:56.997298+0000","last_became_active":"2024-03-06T07:24:56.991155+0000","last_became_peered":"2024-03-06T07:24:56.991155+0000","last_unstale":"2024-03-06T07:24:56.997298+0000","last_undegraded":"2024-03-06T07:24:56.997298+0000","last_fullsized":"2024-03-06T07:24:56.997298+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:30.277424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004361+0000","last_change":"2024-03-06T07:24:55.983699+0000","last_active":"2024-03-06T07:24:57.004361+0000","last_peered":"2024-03-06T07:24:57.004361+0000","last_clean":"2024-03-06T07:24:57.004361+0000","last_became_active":"2024-03-06T07:24:55.983409+0000","last_became_peered":"2024-03-06T07:24:55.983409+0000","last_unstale":"2024-03-06T07:24:57.004361+0000","last_undegraded":"2024-03-06T07:24:57.004361+0000","last_fullsized":"2024-03-06T07:24:57.004361+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:44.010529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000437+0000","last_change":"2024-03-06T07:24:56.995382+0000","last_active":"2024-03-06T07:24:57.000437+0000","last_peered":"2024-03-06T07:24:57.000437+0000","last_clean":"2024-03-06T07:24:57.000437+0000","last_became_active":"2024-03-06T07:24:56.995113+0000","last_became_peered":"2024-03-06T07:24:56.995113+0000","last_unstale":"2024-03-06T07:24:57.000437+0000","last_undegraded":"2024-03-06T07:24:57.000437+0000","last_fullsized":"2024-03-06T07:24:57.000437+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:49.374457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.004348+0000","last_change":"2024-03-06T07:24:55.987346+0000","last_active":"2024-03-06T07:24:57.004348+0000","last_peered":"2024-03-06T07:24:57.004348+0000","last_clean":"2024-03-06T07:24:57.004348+0000","last_became_active":"2024-03-06T07:24:55.987094+0000","last_became_peered":"2024-03-06T07:24:55.987094+0000","last_unstale":"2024-03-06T07:24:57.004348+0000","last_undegraded":"2024-03-06T07:24:57.004348+0000","last_fullsized":"2024-03-06T07:24:57.004348+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:57.153000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997467+0000","last_change":"2024-03-06T07:24:56.994468+0000","last_active":"2024-03-06T07:24:56.997467+0000","last_peered":"2024-03-06T07:24:56.997467+0000","last_clean":"2024-03-06T07:24:56.997467+0000","last_became_active":"2024-03-06T07:24:56.994185+0000","last_became_peered":"2024-03-06T07:24:56.994185+0000","last_unstale":"2024-03-06T07:24:56.997467+0000","last_undegraded":"2024-03-06T07:24:56.997467+0000","last_fullsized":"2024-03-06T07:24:56.997467+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:15.314583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002062+0000","last_change":"2024-03-06T07:24:55.980697+0000","last_active":"2024-03-06T07:24:57.002062+0000","last_peered":"2024-03-06T07:24:57.002062+0000","last_clean":"2024-03-06T07:24:57.002062+0000","last_became_active":"2024-03-06T07:24:55.980415+0000","last_became_peered":"2024-03-06T07:24:55.980415+0000","last_unstale":"2024-03-06T07:24:57.002062+0000","last_undegraded":"2024-03-06T07:24:57.002062+0000","last_fullsized":"2024-03-06T07:24:57.002062+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:16.584485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.992616+0000","last_change":"2024-03-06T07:24:56.984621+0000","last_active":"2024-03-06T07:24:56.992616+0000","last_peered":"2024-03-06T07:24:56.992616+0000","last_clean":"2024-03-06T07:24:56.992616+0000","last_became_active":"2024-03-06T07:24:56.984359+0000","last_became_peered":"2024-03-06T07:24:56.984359+0000","last_unstale":"2024-03-06T07:24:56.992616+0000","last_undegraded":"2024-03-06T07:24:56.992616+0000","last_fullsized":"2024-03-06T07:24:56.992616+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:28.712811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974380+0000","last_change":"2024-03-06T07:24:55.983500+0000","last_active":"2024-03-06T07:24:56.974380+0000","last_peered":"2024-03-06T07:24:56.974380+0000","last_clean":"2024-03-06T07:24:56.974380+0000","last_became_active":"2024-03-06T07:24:55.983234+0000","last_became_peered":"2024-03-06T07:24:55.983234+0000","last_unstale":"2024-03-06T07:24:56.974380+0000","last_undegraded":"2024-03-06T07:24:56.974380+0000","last_fullsized":"2024-03-06T07:24:56.974380+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:53.352866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.986875+0000","last_change":"2024-03-06T07:24:56.983170+0000","last_active":"2024-03-06T07:24:56.986875+0000","last_peered":"2024-03-06T07:24:56.986875+0000","last_clean":"2024-03-06T07:24:56.986875+0000","last_became_active":"2024-03-06T07:24:56.982701+0000","last_became_peered":"2024-03-06T07:24:56.982701+0000","last_unstale":"2024-03-06T07:24:56.986875+0000","last_undegraded":"2024-03-06T07:24:56.986875+0000","last_fullsized":"2024-03-06T07:24:56.986875+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:12.878346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.001519+0000","last_change":"2024-03-06T07:24:55.974671+0000","last_active":"2024-03-06T07:24:57.001519+0000","last_peered":"2024-03-06T07:24:57.001519+0000","last_clean":"2024-03-06T07:24:57.001519+0000","last_became_active":"2024-03-06T07:24:55.974392+0000","last_became_peered":"2024-03-06T07:24:55.974392+0000","last_unstale":"2024-03-06T07:24:57.001519+0000","last_undegraded":"2024-03-06T07:24:57.001519+0000","last_fullsized":"2024-03-06T07:24:57.001519+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:49.702145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995901+0000","last_change":"2024-03-06T07:24:56.990589+0000","last_active":"2024-03-06T07:24:56.995901+0000","last_peered":"2024-03-06T07:24:56.995901+0000","last_clean":"2024-03-06T07:24:56.995901+0000","last_became_active":"2024-03-06T07:24:56.989245+0000","last_became_peered":"2024-03-06T07:24:56.989245+0000","last_unstale":"2024-03-06T07:24:56.995901+0000","last_undegraded":"2024-03-06T07:24:56.995901+0000","last_fullsized":"2024-03-06T07:24:56.995901+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:25.710876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.970703+0000","last_change":"2024-03-06T07:24:55.983340+0000","last_active":"2024-03-06T07:24:56.970703+0000","last_peered":"2024-03-06T07:24:56.970703+0000","last_clean":"2024-03-06T07:24:56.970703+0000","last_became_active":"2024-03-06T07:24:55.982397+0000","last_became_peered":"2024-03-06T07:24:55.982397+0000","last_unstale":"2024-03-06T07:24:56.970703+0000","last_undegraded":"2024-03-06T07:24:56.970703+0000","last_fullsized":"2024-03-06T07:24:56.970703+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:31.650143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995450+0000","last_change":"2024-03-06T07:24:56.993222+0000","last_active":"2024-03-06T07:24:56.995450+0000","last_peered":"2024-03-06T07:24:56.995450+0000","last_clean":"2024-03-06T07:24:56.995450+0000","last_became_active":"2024-03-06T07:24:56.992958+0000","last_became_peered":"2024-03-06T07:24:56.992958+0000","last_unstale":"2024-03-06T07:24:56.995450+0000","last_undegraded":"2024-03-06T07:24:56.995450+0000","last_fullsized":"2024-03-06T07:24:56.995450+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:47.734879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974360+0000","last_change":"2024-03-06T07:24:55.983772+0000","last_active":"2024-03-06T07:24:56.974360+0000","last_peered":"2024-03-06T07:24:56.974360+0000","last_clean":"2024-03-06T07:24:56.974360+0000","last_became_active":"2024-03-06T07:24:55.983289+0000","last_became_peered":"2024-03-06T07:24:55.983289+0000","last_unstale":"2024-03-06T07:24:56.974360+0000","last_undegraded":"2024-03-06T07:24:56.974360+0000","last_fullsized":"2024-03-06T07:24:56.974360+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:10.845648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.989412+0000","last_change":"2024-03-06T07:24:56.982033+0000","last_active":"2024-03-06T07:24:56.989412+0000","last_peered":"2024-03-06T07:24:56.989412+0000","last_clean":"2024-03-06T07:24:56.989412+0000","last_became_active":"2024-03-06T07:24:56.981070+0000","last_became_peered":"2024-03-06T07:24:56.981070+0000","last_unstale":"2024-03-06T07:24:56.989412+0000","last_undegraded":"2024-03-06T07:24:56.989412+0000","last_fullsized":"2024-03-06T07:24:56.989412+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:59.082721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.985139+0000","last_change":"2024-03-06T07:24:55.989533+0000","last_active":"2024-03-06T07:24:56.985139+0000","last_peered":"2024-03-06T07:24:56.985139+0000","last_clean":"2024-03-06T07:24:56.985139+0000","last_became_active":"2024-03-06T07:24:55.989263+0000","last_became_peered":"2024-03-06T07:24:55.989263+0000","last_unstale":"2024-03-06T07:24:56.985139+0000","last_undegraded":"2024-03-06T07:24:56.985139+0000","last_fullsized":"2024-03-06T07:24:56.985139+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:58.332262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.992193+0000","last_change":"2024-03-06T07:24:56.984039+0000","last_active":"2024-03-06T07:24:56.992193+0000","last_peered":"2024-03-06T07:24:56.992193+0000","last_clean":"2024-03-06T07:24:56.992193+0000","last_became_active":"2024-03-06T07:24:56.982838+0000","last_became_peered":"2024-03-06T07:24:56.982838+0000","last_unstale":"2024-03-06T07:24:56.992193+0000","last_undegraded":"2024-03-06T07:24:56.992193+0000","last_fullsized":"2024-03-06T07:24:56.992193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:25.589832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"169'2","reported_seq":14,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.002478+0000","last_change":"2024-03-06T07:24:55.985972+0000","last_active":"2024-03-06T07:24:57.002478+0000","last_peered":"2024-03-06T07:24:57.002478+0000","last_clean":"2024-03-06T07:24:57.002478+0000","last_became_active":"2024-03-06T07:24:55.985205+0000","last_became_peered":"2024-03-06T07:24:55.985205+0000","last_unstale":"2024-03-06T07:24:57.002478+0000","last_undegraded":"2024-03-06T07:24:57.002478+0000","last_fullsized":"2024-03-06T07:24:57.002478+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:29.948569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.998460+0000","last_change":"2024-03-06T07:24:56.995014+0000","last_active":"2024-03-06T07:24:56.998460+0000","last_peered":"2024-03-06T07:24:56.998460+0000","last_clean":"2024-03-06T07:24:56.998460+0000","last_became_active":"2024-03-06T07:24:56.994760+0000","last_became_peered":"2024-03-06T07:24:56.994760+0000","last_unstale":"2024-03-06T07:24:56.998460+0000","last_undegraded":"2024-03-06T07:24:56.998460+0000","last_fullsized":"2024-03-06T07:24:56.998460+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:50.595303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979527+0000","last_change":"2024-03-06T07:24:55.981956+0000","last_active":"2024-03-06T07:24:56.979527+0000","last_peered":"2024-03-06T07:24:56.979527+0000","last_clean":"2024-03-06T07:24:56.979527+0000","last_became_active":"2024-03-06T07:24:55.981705+0000","last_became_peered":"2024-03-06T07:24:55.981705+0000","last_unstale":"2024-03-06T07:24:56.979527+0000","last_undegraded":"2024-03-06T07:24:56.979527+0000","last_fullsized":"2024-03-06T07:24:56.979527+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:26.771537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.980590+0000","last_change":"2024-03-06T07:24:56.977134+0000","last_active":"2024-03-06T07:24:56.980590+0000","last_peered":"2024-03-06T07:24:56.980590+0000","last_clean":"2024-03-06T07:24:56.980590+0000","last_became_active":"2024-03-06T07:24:56.976863+0000","last_became_peered":"2024-03-06T07:24:56.976863+0000","last_unstale":"2024-03-06T07:24:56.980590+0000","last_undegraded":"2024-03-06T07:24:56.980590+0000","last_fullsized":"2024-03-06T07:24:56.980590+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:25.710876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974081+0000","last_change":"2024-03-06T07:24:55.982271+0000","last_active":"2024-03-06T07:24:56.974081+0000","last_peered":"2024-03-06T07:24:56.974081+0000","last_clean":"2024-03-06T07:24:56.974081+0000","last_became_active":"2024-03-06T07:24:55.981970+0000","last_became_peered":"2024-03-06T07:24:55.981970+0000","last_unstale":"2024-03-06T07:24:56.974081+0000","last_undegraded":"2024-03-06T07:24:56.974081+0000","last_fullsized":"2024-03-06T07:24:56.974081+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:21.886939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.006909+0000","last_change":"2024-03-06T07:24:57.001495+0000","last_active":"2024-03-06T07:24:57.006909+0000","last_peered":"2024-03-06T07:24:57.006909+0000","last_clean":"2024-03-06T07:24:57.006909+0000","last_became_active":"2024-03-06T07:24:57.001065+0000","last_became_peered":"2024-03-06T07:24:57.001065+0000","last_unstale":"2024-03-06T07:24:57.006909+0000","last_undegraded":"2024-03-06T07:24:57.006909+0000","last_fullsized":"2024-03-06T07:24:57.006909+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:42.942153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.985610+0000","last_change":"2024-03-06T07:24:55.986141+0000","last_active":"2024-03-06T07:24:56.985610+0000","last_peered":"2024-03-06T07:24:56.985610+0000","last_clean":"2024-03-06T07:24:56.985610+0000","last_became_active":"2024-03-06T07:24:55.985225+0000","last_became_peered":"2024-03-06T07:24:55.985225+0000","last_unstale":"2024-03-06T07:24:56.985610+0000","last_undegraded":"2024-03-06T07:24:56.985610+0000","last_fullsized":"2024-03-06T07:24:56.985610+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:18:30.365068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.994032+0000","last_change":"2024-03-06T07:24:56.989762+0000","last_active":"2024-03-06T07:24:56.994032+0000","last_peered":"2024-03-06T07:24:56.994032+0000","last_clean":"2024-03-06T07:24:56.994032+0000","last_became_active":"2024-03-06T07:24:56.989495+0000","last_became_peered":"2024-03-06T07:24:56.989495+0000","last_unstale":"2024-03-06T07:24:56.994032+0000","last_undegraded":"2024-03-06T07:24:56.994032+0000","last_fullsized":"2024-03-06T07:24:56.994032+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:21.760548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.977029+0000","last_change":"2024-03-06T07:24:55.980753+0000","last_active":"2024-03-06T07:24:56.977029+0000","last_peered":"2024-03-06T07:24:56.977029+0000","last_clean":"2024-03-06T07:24:56.977029+0000","last_became_active":"2024-03-06T07:24:55.980474+0000","last_became_peered":"2024-03-06T07:24:55.980474+0000","last_unstale":"2024-03-06T07:24:56.977029+0000","last_undegraded":"2024-03-06T07:24:56.977029+0000","last_fullsized":"2024-03-06T07:24:56.977029+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:00.722299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.995924+0000","last_change":"2024-03-06T07:24:56.990803+0000","last_active":"2024-03-06T07:24:56.995924+0000","last_peered":"2024-03-06T07:24:56.995924+0000","last_clean":"2024-03-06T07:24:56.995924+0000","last_became_active":"2024-03-06T07:24:56.989539+0000","last_became_peered":"2024-03-06T07:24:56.989539+0000","last_unstale":"2024-03-06T07:24:56.995924+0000","last_undegraded":"2024-03-06T07:24:56.995924+0000","last_fullsized":"2024-03-06T07:24:56.995924+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:58.830269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.974043+0000","last_change":"2024-03-06T07:24:55.984798+0000","last_active":"2024-03-06T07:24:56.974043+0000","last_peered":"2024-03-06T07:24:56.974043+0000","last_clean":"2024-03-06T07:24:56.974043+0000","last_became_active":"2024-03-06T07:24:55.984387+0000","last_became_peered":"2024-03-06T07:24:55.984387+0000","last_unstale":"2024-03-06T07:24:56.974043+0000","last_undegraded":"2024-03-06T07:24:56.974043+0000","last_fullsized":"2024-03-06T07:24:56.974043+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:29.767763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997093+0000","last_change":"2024-03-06T07:24:56.992237+0000","last_active":"2024-03-06T07:24:56.997093+0000","last_peered":"2024-03-06T07:24:56.997093+0000","last_clean":"2024-03-06T07:24:56.997093+0000","last_became_active":"2024-03-06T07:24:56.991930+0000","last_became_peered":"2024-03-06T07:24:56.991930+0000","last_unstale":"2024-03-06T07:24:56.997093+0000","last_undegraded":"2024-03-06T07:24:56.997093+0000","last_fullsized":"2024-03-06T07:24:56.997093+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:05.931371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.991270+0000","last_change":"2024-03-06T07:24:55.989801+0000","last_active":"2024-03-06T07:24:56.991270+0000","last_peered":"2024-03-06T07:24:56.991270+0000","last_clean":"2024-03-06T07:24:56.991270+0000","last_became_active":"2024-03-06T07:24:55.989517+0000","last_became_peered":"2024-03-06T07:24:55.989517+0000","last_unstale":"2024-03-06T07:24:56.991270+0000","last_undegraded":"2024-03-06T07:24:56.991270+0000","last_fullsized":"2024-03-06T07:24:56.991270+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:04.680394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.998384+0000","last_change":"2024-03-06T07:24:56.995538+0000","last_active":"2024-03-06T07:24:56.998384+0000","last_peered":"2024-03-06T07:24:56.998384+0000","last_clean":"2024-03-06T07:24:56.998384+0000","last_became_active":"2024-03-06T07:24:56.995268+0000","last_became_peered":"2024-03-06T07:24:56.995268+0000","last_unstale":"2024-03-06T07:24:56.998384+0000","last_undegraded":"2024-03-06T07:24:56.998384+0000","last_fullsized":"2024-03-06T07:24:56.998384+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:18.722511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.973438+0000","last_change":"2024-03-06T07:24:55.981966+0000","last_active":"2024-03-06T07:24:56.973438+0000","last_peered":"2024-03-06T07:24:56.973438+0000","last_clean":"2024-03-06T07:24:56.973438+0000","last_became_active":"2024-03-06T07:24:55.981707+0000","last_became_peered":"2024-03-06T07:24:55.981707+0000","last_unstale":"2024-03-06T07:24:56.973438+0000","last_undegraded":"2024-03-06T07:24:56.973438+0000","last_fullsized":"2024-03-06T07:24:56.973438+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:27.554510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.997560+0000","last_change":"2024-03-06T07:24:56.995474+0000","last_active":"2024-03-06T07:24:56.997560+0000","last_peered":"2024-03-06T07:24:56.997560+0000","last_clean":"2024-03-06T07:24:56.997560+0000","last_became_active":"2024-03-06T07:24:56.995184+0000","last_became_peered":"2024-03-06T07:24:56.995184+0000","last_unstale":"2024-03-06T07:24:56.997560+0000","last_undegraded":"2024-03-06T07:24:56.997560+0000","last_fullsized":"2024-03-06T07:24:56.997560+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:55.962097+0000","last_clean_scrub_stamp":"2024-03-06T07:24:55.962097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:00.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"169'1","reported_seq":13,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:57.000774+0000","last_change":"2024-03-06T07:24:55.986704+0000","last_active":"2024-03-06T07:24:57.000774+0000","last_peered":"2024-03-06T07:24:57.000774+0000","last_clean":"2024-03-06T07:24:57.000774+0000","last_became_active":"2024-03-06T07:24:55.986416+0000","last_became_peered":"2024-03-06T07:24:55.986416+0000","last_unstale":"2024-03-06T07:24:57.000774+0000","last_undegraded":"2024-03-06T07:24:57.000774+0000","last_fullsized":"2024-03-06T07:24:57.000774+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:54.957532+0000","last_clean_scrub_stamp":"2024-03-06T07:24:54.957532+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:04.818179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":452,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-06T07:24:56.979135+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:24:56.979135+0000","last_peered":"2024-03-06T07:24:56.979135+0000","last_clean":"2024-03-06T07:24:56.979135+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:24:56.979135+0000","last_undegraded":"2024-03-06T07:24:56.979135+0000","last_fullsized":"2024-03-06T07:24:56.979135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1623179,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2154496,"data_stored":1042172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574933,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1032811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1032811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574934,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33256,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338584,"statfs":{"total":96636764160,"available":96602710016,"internally_reserved":0,"allocated":2125824,"data_stored":1032811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33276,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338564,"statfs":{"total":96636764160,"available":96602689536,"internally_reserved":0,"allocated":2142208,"data_stored":1040068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33272,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338568,"statfs":{"total":96636764160,"available":96602693632,"internally_reserved":0,"allocated":2138112,"data_stored":1039978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33324,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338516,"statfs":{"total":96636764160,"available":96602640384,"internally_reserved":0,"allocated":2191360,"data_stored":1623179,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:00.533 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:00.533 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:00.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.675+0000 7f58b2637640 1 Processor -- start 2024-03-06T07:25:00.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.675+0000 7f58b2637640 1 -- start start 2024-03-06T07:25:00.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b2637640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b2637640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f58ac159160 0x7f58ac15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b2637640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac06b800 con 0x7f58ac06afc0 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b2637640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac158790 con 0x7f58ac159160 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b2637640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0c5840 con 0x7f58ac06b3a0 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b0bd4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f58ac159160 0x7f58ac15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b0bd4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f58ac159160 0x7f58ac15fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59034/0 (socket says 172.21.15.94:59034) 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b0bd4640 1 -- 172.21.15.94:0/1327219752 learned_addr learned my addr 172.21.15.94:0/1327219752 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58ab7fe640 1 --2- 172.21.15.94:0/1327219752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b0bd4640 1 -- 172.21.15.94:0/1327219752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 msgr2=0x7f58ac158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b0bd4640 1 --2- 172.21.15.94:0/1327219752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b0bd4640 1 -- 172.21.15.94:0/1327219752 >> v1:172.21.15.145:6789/0 conn(0x7f58ac06afc0 legacy=0x7f58ac0c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.676+0000 7f58b0bd4640 1 -- 172.21.15.94:0/1327219752 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0c5b20 con 0x7f58ac159160 2024-03-06T07:25:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b0bd4640 1 --2- 172.21.15.94:0/1327219752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac159160 0x7f58ac15fac0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009e50 tx=0x7f58a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d225a67c0321c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58aaffd640 1 -- 172.21.15.94:0/1327219752 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a402ca70 con 0x7f58ac159160 2024-03-06T07:25:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58aaffd640 1 -- 172.21.15.94:0/1327219752 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a402cc10 con 0x7f58ac159160 2024-03-06T07:25:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58aaffd640 1 -- 172.21.15.94:0/1327219752 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a40343f0 con 0x7f58ac159160 2024-03-06T07:25:00.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 -- 172.21.15.94:0/1327219752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac159160 msgr2=0x7f58ac15fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 --2- 172.21.15.94:0/1327219752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac159160 0x7f58ac15fac0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009e50 tx=0x7f58a402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 -- 172.21.15.94:0/1327219752 shutdown_connections 2024-03-06T07:25:00.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 --2- 172.21.15.94:0/1327219752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac159160 0x7f58ac15fac0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 --2- 172.21.15.94:0/1327219752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 -- 172.21.15.94:0/1327219752 >> 172.21.15.94:0/1327219752 conn(0x7f58ac0c4220 msgr2=0x7f58ac0c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 -- 172.21.15.94:0/1327219752 shutdown_connections 2024-03-06T07:25:00.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.677+0000 7f58b2637640 1 -- 172.21.15.94:0/1327219752 wait complete. 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.678+0000 7f58b2637640 1 Processor -- start 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58b2637640 1 -- start start 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58b2637640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac06afc0 0x7f58ac06e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58b2637640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58b2637640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58ac159160 0x7f58ac14b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58b2637640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1595f0 con 0x7f58ac06afc0 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58b2637640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0c60e0 con 0x7f58ac159160 2024-03-06T07:25:00.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58b2637640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac155220 con 0x7f58ac06b3a0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac06afc0 0x7f58ac06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58abfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac06afc0 0x7f58ac06e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59044/0 (socket says 172.21.15.94:59044) 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.679+0000 7f58abfff640 1 -- 172.21.15.94:0/2290841349 learned_addr learned my addr 172.21.15.94:0/2290841349 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58ab7fe640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58b0bd4640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58ac159160 0x7f58ac14b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58abfff640 1 -- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 msgr2=0x7f58ac157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58abfff640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58abfff640 1 -- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58ac159160 msgr2=0x7f58ac14b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58abfff640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58ac159160 0x7f58ac14b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58abfff640 1 -- 172.21.15.94:0/2290841349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0715b0 con 0x7f58ac06afc0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58abfff640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac06afc0 0x7f58ac06e390 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f589c00c430 tx=0x7f589c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58a8ff9640 1 -- 172.21.15.94:0/2290841349 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c0052d0 con 0x7f58ac06afc0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58a8ff9640 1 -- 172.21.15.94:0/2290841349 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c005490 con 0x7f58ac06afc0 2024-03-06T07:25:00.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac1ee810 con 0x7f58ac06afc0 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58a8ff9640 1 -- 172.21.15.94:0/2290841349 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c007260 con 0x7f58ac06afc0 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.680+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac1eedd0 con 0x7f58ac06afc0 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.681+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac0c5010 con 0x7f58ac06afc0 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.681+0000 7f58a8ff9640 1 -- 172.21.15.94:0/2290841349 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f589c007400 con 0x7f58ac06afc0 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.682+0000 7f58a8ff9640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f589806b4d0 0x7f589806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.682+0000 7f58a8ff9640 1 -- 172.21.15.94:0/2290841349 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f589c0897d0 con 0x7f58ac06afc0 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.682+0000 7f58ab7fe640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f589806b4d0 0x7f589806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.682+0000 7f58ab7fe640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f589806b4d0 0x7f589806d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f58ac074330 tx=0x7f58a0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.686+0000 7f58a8ff9640 1 -- 172.21.15.94:0/2290841349 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f589c057e70 con 0x7f58ac06afc0 2024-03-06T07:25:00.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.921+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f58ac0c6730 con 0x7f58ac06afc0 2024-03-06T07:25:00.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.922+0000 7f58a8ff9640 1 -- 172.21.15.94:0/2290841349 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f589c05bf10 con 0x7f58ac06afc0 2024-03-06T07:25:00.924 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:25:00.924 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:00.925 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.924+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f589806b4d0 msgr2=0x7f589806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.924+0000 7f58b2637640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f589806b4d0 0x7f589806d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f58ac074330 tx=0x7f58a0009190 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.924+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac06afc0 msgr2=0x7f58ac06e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.924+0000 7f58b2637640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac06afc0 0x7f58ac06e390 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f589c00c430 tx=0x7f589c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 shutdown_connections 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f589806b4d0 0x7f589806d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58ac159160 0x7f58ac14b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f58ac06b3a0 0x7f58ac157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 --2- 172.21.15.94:0/2290841349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f58ac06afc0 0x7f58ac06e390 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 >> 172.21.15.94:0/2290841349 conn(0x7f58ac0c4220 msgr2=0x7f58ac153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 shutdown_connections 2024-03-06T07:25:00.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:00.925+0000 7f58b2637640 1 -- 172.21.15.94:0/2290841349 wait complete. 2024-03-06T07:25:00.937 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:01.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.036+0000 7f1353b1e640 1 Processor -- start 2024-03-06T07:25:01.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.036+0000 7f1353b1e640 1 -- start start 2024-03-06T07:25:01.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f1353b1e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f134c14f790 0x7f134c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f1353b1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f1353b1e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c0c28d0 con 0x7f134c0c5c60 2024-03-06T07:25:01.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f1353b1e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134c0c2680 con 0x7f134c14f790 2024-03-06T07:25:01.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f1353b1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0c2bb0 con 0x7f134c0c1aa0 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13518ba640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f134c14f790 0x7f134c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13518ba640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f134c14f790 0x7f134c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59060/0 (socket says 172.21.15.94:59060) 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13518ba640 1 -- 172.21.15.94:0/813701768 learned_addr learned my addr 172.21.15.94:0/813701768 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13510b9640 1 --2- 172.21.15.94:0/813701768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13518ba640 1 -- 172.21.15.94:0/813701768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 msgr2=0x7f134c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13518ba640 1 --2- 172.21.15.94:0/813701768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13518ba640 1 -- 172.21.15.94:0/813701768 >> v1:172.21.15.145:6789/0 conn(0x7f134c0c5c60 legacy=0x7f134c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.037+0000 7f13518ba640 1 -- 172.21.15.94:0/813701768 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0c2e90 con 0x7f134c14f790 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.038+0000 7f13518ba640 1 --2- 172.21.15.94:0/813701768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c14f790 0x7f134c1529d0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f1348009870 tx=0x7f134802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e83f6c1d96c01a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.038+0000 7f13508b8640 1 -- 172.21.15.94:0/813701768 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348034070 con 0x7f134c14f790 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.038+0000 7f13508b8640 1 -- 172.21.15.94:0/813701768 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1348030840 con 0x7f134c14f790 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.038+0000 7f13508b8640 1 -- 172.21.15.94:0/813701768 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348030c10 con 0x7f134c14f790 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 -- 172.21.15.94:0/813701768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c14f790 msgr2=0x7f134c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 --2- 172.21.15.94:0/813701768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c14f790 0x7f134c1529d0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f1348009870 tx=0x7f134802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 -- 172.21.15.94:0/813701768 shutdown_connections 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 --2- 172.21.15.94:0/813701768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 --2- 172.21.15.94:0/813701768 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c14f790 0x7f134c1529d0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 -- 172.21.15.94:0/813701768 >> 172.21.15.94:0/813701768 conn(0x7f134c0c6710 msgr2=0x7f134c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 -- 172.21.15.94:0/813701768 shutdown_connections 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.039+0000 7f1353b1e640 1 -- 172.21.15.94:0/813701768 wait complete. 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.040+0000 7f1353b1e640 1 Processor -- start 2024-03-06T07:25:01.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.040+0000 7f1353b1e640 1 -- start start 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c1ed400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c0c5c60 0x7f134c1ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f134c14f790 0x7f134c0724c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06cc00 con 0x7f134c0c5c60 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0c0d60 con 0x7f134c14f790 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06cd80 con 0x7f134c0c1aa0 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c0c5c60 0x7f134c1ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c0c5c60 0x7f134c1ea6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59066/0 (socket says 172.21.15.94:59066) 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 -- 172.21.15.94:0/148385825 learned_addr learned my addr 172.21.15.94:0/148385825 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13518ba640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c1ed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13520bb640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f134c14f790 0x7f134c0724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 -- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 msgr2=0x7f134c1ed400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c1ed400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 -- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f134c14f790 msgr2=0x7f134c0724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f134c14f790 0x7f134c0724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 -- 172.21.15.94:0/148385825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0c0b10 con 0x7f134c0c5c60 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13510b9640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c0c5c60 0x7f134c1ea6f0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e9b0 tx=0x7f133c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13427fc640 1 -- 172.21.15.94:0/148385825 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c017070 con 0x7f134c0c5c60 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13427fc640 1 -- 172.21.15.94:0/148385825 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c00c9a0 con 0x7f134c0c5c60 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c1dd700 con 0x7f134c0c5c60 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f13427fc640 1 -- 172.21.15.94:0/148385825 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00cd90 con 0x7f134c0c5c60 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.041+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c1ec920 con 0x7f134c0c5c60 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.042+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c075300 con 0x7f134c0c5c60 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.043+0000 7f13427fc640 1 -- 172.21.15.94:0/148385825 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f133c01a070 con 0x7f134c0c5c60 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.043+0000 7f13427fc640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133006b4d0 0x7f133006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.043+0000 7f13427fc640 1 -- 172.21.15.94:0/148385825 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f133c08a2e0 con 0x7f134c0c5c60 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.043+0000 7f13518ba640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133006b4d0 0x7f133006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.044+0000 7f13518ba640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133006b4d0 0x7f133006d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1348036000 tx=0x7f134803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.047+0000 7f13427fc640 1 -- 172.21.15.94:0/148385825 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f134c075300 con 0x7f134c0c5c60 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.256+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f134c04f330 con 0x7f134c0c5c60 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.257+0000 7f13427fc640 1 -- 172.21.15.94:0/148385825 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f133c058a10 con 0x7f134c0c5c60 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":159,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:24:58.558540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37487,"qdb_cluster":[37487]},"id":13}]} 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 159 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133006b4d0 msgr2=0x7f133006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133006b4d0 0x7f133006d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1348036000 tx=0x7f134803b040 comp rx=0 tx=0).stop 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c0c5c60 msgr2=0x7f134c1ea6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c0c5c60 0x7f134c1ea6f0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e9b0 tx=0x7f133c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 shutdown_connections 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f133006b4d0 0x7f133006d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f134c14f790 0x7f134c0724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f134c0c5c60 0x7f134c1ea6f0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 --2- 172.21.15.94:0/148385825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f134c0c1aa0 0x7f134c1ed400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 >> 172.21.15.94:0/148385825 conn(0x7f134c0c6710 msgr2=0x7f134c0c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 shutdown_connections 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.259+0000 7f1353b1e640 1 -- 172.21.15.94:0/148385825 wait complete. 2024-03-06T07:25:01.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 159, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:24:58.558540+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 37487, 'qdb_cluster': [37487]} 2024-03-06T07:25:01.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:01.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:25:01.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.779+0000 7fccefeb5640 1 Processor -- start 2024-03-06T07:25:01.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.779+0000 7fccefeb5640 1 -- start start 2024-03-06T07:25:01.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccefeb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccefeb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcce814f110 0x7fcce814f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccefeb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce80a9990 con 0x7fcce80c6ab0 2024-03-06T07:25:01.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccefeb5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce8155d10 con 0x7fcce814f110 2024-03-06T07:25:01.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccefeb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce81556c0 con 0x7fcce81561a0 2024-03-06T07:25:01.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccee452640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcce814f110 0x7fcce814f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccee452640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcce814f110 0x7fcce814f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59076/0 (socket says 172.21.15.94:59076) 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccedc51640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcce80c6ab0 0x7fcce80a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33140/0 (socket says 172.21.15.94:33140) 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccee452640 1 -- 172.21.15.94:0/2950058729 learned_addr learned my addr 172.21.15.94:0/2950058729 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fcced450640 1 --2- 172.21.15.94:0/2950058729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccee452640 1 -- 172.21.15.94:0/2950058729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 msgr2=0x7fcce81529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccee452640 1 --2- 172.21.15.94:0/2950058729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.780+0000 7fccee452640 1 -- 172.21.15.94:0/2950058729 >> v1:172.21.15.94:6789/0 conn(0x7fcce80c6ab0 legacy=0x7fcce80a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.781+0000 7fccee452640 1 -- 172.21.15.94:0/2950058729 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce81559a0 con 0x7fcce814f110 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.781+0000 7fccee452640 1 --2- 172.21.15.94:0/2950058729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce814f110 0x7fcce814f570 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009870 tx=0x7fcce002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f247b47ebbd6da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.781+0000 7fccecc4f640 1 -- 172.21.15.94:0/2950058729 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0035020 con 0x7fcce814f110 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.781+0000 7fccecc4f640 1 -- 172.21.15.94:0/2950058729 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce0031640 con 0x7fcce814f110 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.781+0000 7fccecc4f640 1 -- 172.21.15.94:0/2950058729 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce002d040 con 0x7fcce814f110 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 -- 172.21.15.94:0/2950058729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce814f110 msgr2=0x7fcce814f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2950058729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce814f110 0x7fcce814f570 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009870 tx=0x7fcce002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 -- 172.21.15.94:0/2950058729 shutdown_connections 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2950058729 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce814f110 0x7fcce814f570 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2950058729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 -- 172.21.15.94:0/2950058729 >> 172.21.15.94:0/2950058729 conn(0x7fcce806b1a0 msgr2=0x7fcce806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 -- 172.21.15.94:0/2950058729 shutdown_connections 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.782+0000 7fccefeb5640 1 -- 172.21.15.94:0/2950058729 wait complete. 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.783+0000 7fccefeb5640 1 Processor -- start 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.783+0000 7fccefeb5640 1 -- start start 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccefeb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce80c6ab0 0x7fcce8154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccefeb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcce814f110 0x7fcce81dfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccefeb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81eb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccefeb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80aa600 con 0x7fcce80c6ab0 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccefeb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80c2a90 con 0x7fcce814f110 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccefeb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce8153550 con 0x7fcce81561a0 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce80c6ab0 0x7fcce8154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce80c6ab0 0x7fcce8154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59080/0 (socket says 172.21.15.94:59080) 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 -- 172.21.15.94:0/2953805002 learned_addr learned my addr 172.21.15.94:0/2953805002 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fcced450640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcce814f110 0x7fcce81dfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccee452640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 -- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 msgr2=0x7fcce81eb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81eb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 -- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcce814f110 msgr2=0x7fcce81dfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcce814f110 0x7fcce81dfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 -- 172.21.15.94:0/2953805002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce81ee910 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.784+0000 7fccedc51640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce80c6ab0 0x7fcce8154090 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fcce400c730 tx=0x7fcce400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.785+0000 7fccde7fc640 1 -- 172.21.15.94:0/2953805002 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4017070 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.785+0000 7fccde7fc640 1 -- 172.21.15.94:0/2953805002 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce400ce40 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.785+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce81eeb90 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.785+0000 7fccde7fc640 1 -- 172.21.15.94:0/2953805002 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce400a860 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.785+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce81e0110 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.786+0000 7fccde7fc640 1 -- 172.21.15.94:0/2953805002 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcce401a070 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.787+0000 7fccde7fc640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.787+0000 7fccde7fc640 1 -- 172.21.15.94:0/2953805002 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcce4089810 con 0x7fcce80c6ab0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.787+0000 7fcced450640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.787+0000 7fcced450640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcccc06b4d0 0x7fcccc06d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcce814f640 tx=0x7fccd8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.787+0000 7fccf2b7a740 1 Processor -- start 2024-03-06T07:25:01.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccf2b7a740 1 -- start start 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c627776a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c627776f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277774b0 0x55c627773680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6276dd6b0 con 0x55c627778f50 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c627686a60 con 0x55c6277774b0 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6276d9750 con 0x55c627779330 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccedc51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c627776a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccedc51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c627776a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59096/0 (socket says 172.21.15.94:59096) 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccedc51640 1 -- 172.21.15.94:0/758507574 learned_addr learned my addr 172.21.15.94:0/758507574 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fcced450640 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c627776f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccee452640 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277774b0 0x55c627773680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.788+0000 7fccedc51640 1 -- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 msgr2=0x55c627776f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fccedc51640 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c627776f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fccedc51640 1 -- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277774b0 msgr2=0x55c627773680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fccedc51640 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277774b0 0x55c627773680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fccedc51640 1 -- 172.21.15.94:0/758507574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6276a4f30 con 0x55c627778f50 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fccedc51640 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c627776a30 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fcce4060070 tx=0x7fcce4057bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734e5b26ed4d0ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fcccb7fe640 1 -- 172.21.15.94:0/758507574 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4056240 con 0x55c627778f50 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fcccb7fe640 1 -- 172.21.15.94:0/758507574 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce406be50 con 0x55c627778f50 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fcccb7fe640 1 -- 172.21.15.94:0/758507574 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce40581a0 con 0x55c627778f50 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fccf2b7a740 1 -- 172.21.15.94:0/758507574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 msgr2=0x55c627776a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.789+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c627776a30 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fcce4060070 tx=0x7fcce4057bb0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 -- 172.21.15.94:0/758507574 shutdown_connections 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277774b0 0x55c627773680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c627776f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/758507574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c627776a30 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 -- 172.21.15.94:0/758507574 >> 172.21.15.94:0/758507574 conn(0x55c6277757d0 msgr2=0x55c62779a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 -- 172.21.15.94:0/758507574 shutdown_connections 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 -- 172.21.15.94:0/758507574 wait complete. 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 -- start start 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.790+0000 7fccf2b7a740 1 Processor -- start 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c6277c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c62779c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277ac3c0 0x55c6277c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62779b230 con 0x55c627778f50 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c627699ea0 con 0x55c6277ac3c0 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c62779ac70 con 0x55c627779330 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccf2b7a740 10 client.0 _ll_register_callbacks cb 0x55c627699870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c6277c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c6277c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59106/0 (socket says 172.21.15.94:59106) 2024-03-06T07:25:01.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 -- 172.21.15.94:0/3329471014 learned_addr learned my addr 172.21.15.94:0/3329471014 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fcced450640 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c62779c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccee452640 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277ac3c0 0x55c6277c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 -- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 msgr2=0x55c62779c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c62779c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 -- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277ac3c0 msgr2=0x55c6277c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277ac3c0 0x55c6277c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.791+0000 7fccedc51640 1 -- 172.21.15.94:0/3329471014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c62769a270 con 0x55c627778f50 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccedc51640 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c6277c97f0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fcce4060070 tx=0x7fcce4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1510e01e3990bb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccc8ff9640 1 -- 172.21.15.94:0/3329471014 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4074060 con 0x55c627778f50 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccc8ff9640 1 -- 172.21.15.94:0/3329471014 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcce40583a0 con 0x55c627778f50 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcce406be50 con 0x55c627778f50 2024-03-06T07:25:01.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccc8ff9640 1 -- 172.21.15.94:0/3329471014 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=159) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcce4060110 con 0x55c627778f50 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 10 client.9573 fetch_fsmap learned FSMap version 159 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=159,monmap=2+}) v3 -- 0x55c6273f22d0 con 0x55c627778f50 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccc8ff9640 1 -- 172.21.15.94:0/3329471014 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce406be50 con 0x55c627778f50 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccc8ff9640 1 -- 172.21.15.94:0/3329471014 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 159) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcce4057390 con 0x55c627778f50 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 10 client.9573 fetch_fsmap finished waiting for FSMap version 159 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 10 client.9573 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 20 client.9573 populate_metadata read hostname 'smithi094' 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c6277c63b0 0x55c6277c6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 4 client.9573 mds_command: new command op to 37487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 --> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6276eabf0 con 0x55c6277c63b0 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.792+0000 7fcced450640 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c6277c63b0 0x55c6277c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.793+0000 7fccc8ff9640 10 client.9573 ms_handle_connect on v2:172.21.15.145:6835/1570645588 2024-03-06T07:25:01.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.793+0000 7fcced450640 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c6277c63b0 0x55c6277c6790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=312397676ac4701f server_cookie=fc1446c73e1860e0 in_seq=0 out_seq=0 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.793+0000 7fccc8ff9640 1 -- 172.21.15.94:0/3329471014 <== mds.0 v2:172.21.15.145:6835/1570645588 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c6276eabf0 con 0x55c6277c63b0 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.793+0000 7fccc8ff9640 10 client.9573 handle_command_reply: tid=0 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.793+0000 7fccf2b7a740 1 client.9573 shutdown 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.793+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 msgr2=0x55c6277c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.793+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c6277c97f0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fcce4060070 tx=0x7fcce4000f40 comp rx=0 tx=0).stop 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 shutdown_connections 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c6277ac3c0 0x55c6277c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c62779c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c6277c63b0 0x55c6277c6790 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3329471014 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c627778f50 0x55c6277c97f0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 >> 172.21.15.94:0/3329471014 conn(0x55c6277774b0 msgr2=0x55c627777ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccc8ff9640 0 client.9573 ms_handle_reset on v2:172.21.15.145:6835/1570645588 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 shutdown_connections 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3329471014 wait complete. 2024-03-06T07:25:01.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.794+0000 7fccf2b7a740 20 client.9573 trim_cache size 0 max 16384 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.810+0000 7fccf2b7a740 1 Processor -- start 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.810+0000 7fccf2b7a740 1 -- start start 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277d8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c6277d8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce8151850 con 0x55c6277ac3c0 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce8150c50 con 0x55c627778f50 2024-03-06T07:25:01.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce8151250 con 0x55c627779330 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c6277d8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c6277d8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59114/0 (socket says 172.21.15.94:59114) 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 -- 172.21.15.94:0/3637879121 learned_addr learned my addr 172.21.15.94:0/3637879121 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fcced450640 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277d8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccedc51640 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 -- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 msgr2=0x55c6277d8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277d8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 -- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 msgr2=0x55c6277c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 -- 172.21.15.94:0/3637879121 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6274bc6b0 con 0x55c6277ac3c0 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccedc51640 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.811+0000 7fccee452640 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c6277d8ff0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009960 tx=0x7fcce0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b656ee87313b885 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fcccaffd640 1 -- 172.21.15.94:0/3637879121 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0035040 con 0x55c6277ac3c0 2024-03-06T07:25:01.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fcccaffd640 1 -- 172.21.15.94:0/3637879121 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce00318e0 con 0x55c6277ac3c0 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fcccaffd640 1 -- 172.21.15.94:0/3637879121 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0031c90 con 0x55c6277ac3c0 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3637879121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 msgr2=0x55c6277d8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c6277d8ff0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009960 tx=0x7fcce0030000 comp rx=0 tx=0).stop 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3637879121 shutdown_connections 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c6277d8ff0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/3637879121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3637879121 >> 172.21.15.94:0/3637879121 conn(0x55c6277774b0 msgr2=0x55c627799d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3637879121 shutdown_connections 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.812+0000 7fccf2b7a740 1 -- 172.21.15.94:0/3637879121 wait complete. 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.813+0000 7fccf2b7a740 1 -- start start 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.813+0000 7fccf2b7a740 1 Processor -- start 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c627796890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6277c9ff0 con 0x55c6277ac3c0 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c62779b230 con 0x55c627778f50 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6277ca240 con 0x55c627779330 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 10 client.0 _ll_register_callbacks cb 0x7fcce00096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c627796890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c627796890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59122/0 (socket says 172.21.15.94:59122) 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 -- 172.21.15.94:0/2282717794 learned_addr learned my addr 172.21.15.94:0/2282717794 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccedc51640 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fcced450640 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 -- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 msgr2=0x55c6277cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 -- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 msgr2=0x55c6277c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 -- 172.21.15.94:0/2282717794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6277ce660 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccee452640 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c627796890 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009990 tx=0x7fcce0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ab00973abe78ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccc8ff9640 1 -- 172.21.15.94:0/2282717794 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0035040 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccc8ff9640 1 -- 172.21.15.94:0/2282717794 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcce0004030 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcce00318e0 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccc8ff9640 1 -- 172.21.15.94:0/2282717794 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=159) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcce0031e40 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 10 client.9579 fetch_fsmap learned FSMap version 159 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.814+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=159,monmap=2+}) v3 -- 0x55c627774e20 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccc8ff9640 1 -- 172.21.15.94:0/2282717794 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce00318e0 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccc8ff9640 1 -- 172.21.15.94:0/2282717794 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 159) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcce0004230 con 0x55c6277ac3c0 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 10 client.9579 fetch_fsmap finished waiting for FSMap version 159 2024-03-06T07:25:01.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 10 client.9579 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 20 client.9579 populate_metadata read hostname 'smithi094' 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c627797270 0x55c62781c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 4 client.9579 mds_command: new command op to 37487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 --> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c6276e4f40 con 0x55c627797270 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccedc51640 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c627797270 0x55c62781c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccc8ff9640 10 client.9579 ms_handle_connect on v2:172.21.15.145:6835/1570645588 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccedc51640 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c627797270 0x55c62781c420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a209400d9290065 server_cookie=7c5ae43675133a3 in_seq=0 out_seq=0 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccc8ff9640 1 -- 172.21.15.94:0/2282717794 <== mds.0 v2:172.21.15.145:6835/1570645588 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55c6276e4f40 con 0x55c627797270 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccc8ff9640 10 client.9579 handle_command_reply: tid=0 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 client.9579 shutdown 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 msgr2=0x55c627796890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c627796890 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009990 tx=0x7fcce0004730 comp rx=0 tx=0).stop 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 shutdown_connections 2024-03-06T07:25:01.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:6835/1570645588,v1:172.21.15.145:6836/1570645588] conn(0x55c627797270 0x55c62781c420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.815+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c6277ac3c0 0x55c627796890 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c627779330 0x55c6277cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccc8ff9640 0 client.9579 ms_handle_reset on v2:172.21.15.145:6835/1570645588 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccf2b7a740 1 --2- 172.21.15.94:0/2282717794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c627778f50 0x55c6277c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 >> 172.21.15.94:0/2282717794 conn(0x55c6277774b0 msgr2=0x55c627799d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 shutdown_connections 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccf2b7a740 1 -- 172.21.15.94:0/2282717794 wait complete. 2024-03-06T07:25:01.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccf2b7a740 20 client.9579 trim_cache size 0 max 16384 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 37487, 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.836782436, 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 159, 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 170, 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 170, 2024-03-06T07:25:01.827 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 9.9489772189999996 2024-03-06T07:25:01.828 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcccc06b4d0 msgr2=0x7fcccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcccc06b4d0 0x7fcccc06d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcce814f640 tx=0x7fccd8009190 comp rx=0 tx=0).stop 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce80c6ab0 msgr2=0x7fcce8154090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce80c6ab0 0x7fcce8154090 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fcce400c730 tx=0x7fcce400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 shutdown_connections 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcce81561a0 0x7fcce81eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcce814f110 0x7fcce81dfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 --2- 172.21.15.94:0/2953805002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcce80c6ab0 0x7fcce8154090 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 >> 172.21.15.94:0/2953805002 conn(0x7fcce806b1a0 msgr2=0x7fcce806e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 shutdown_connections 2024-03-06T07:25:01.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.816+0000 7fccefeb5640 1 -- 172.21.15.94:0/2953805002 wait complete. 2024-03-06T07:25:01.830 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.931+0000 7f0fa0fac640 1 Processor -- start 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0fa0fac640 1 -- start start 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0fa0fac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0f9c155c30 0x7f0f9c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0fa0fac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0c5380 0x7f0f9c0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0fa0fac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0b0450 con 0x7f0f9c0b0070 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0fa0fac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c06cfb0 con 0x7f0f9c155c30 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0fa0fac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c06c020 con 0x7f0f9c0c5380 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0f9a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0f9c0b0070 0x7f0f9c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33144/0 (socket says 172.21.15.94:33144) 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0f9a59c640 1 -- 172.21.15.94:0/1804557194 learned_addr learned my addr 172.21.15.94:0/1804557194 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0f93fff640 1 --2- 172.21.15.94:0/1804557194 >> v2:172.21.15.94:3300/0 conn(0x7f0f9c155c30 0x7f0f9c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.932+0000 7f0f9ad9d640 1 --2- 172.21.15.94:0/1804557194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0c5380 0x7f0f9c0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f99d9b640 1 -- 172.21.15.94:0/1804557194 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127155999 0 0) 0x7f0f8c001110 con 0x7f0f9c0b0070 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f99d9b640 1 -- 172.21.15.94:0/1804557194 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f88003520 con 0x7f0f9c0b0070 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f93fff640 1 -- 172.21.15.94:0/1804557194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0c5380 msgr2=0x7f0f9c0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f93fff640 1 --2- 172.21.15.94:0/1804557194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0c5380 0x7f0f9c0c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f93fff640 1 -- 172.21.15.94:0/1804557194 >> v1:172.21.15.94:6789/0 conn(0x7f0f9c0b0070 legacy=0x7f0f9c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f93fff640 1 -- 172.21.15.94:0/1804557194 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c1560e0 con 0x7f0f9c155c30 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f9ad9d640 1 --2- 172.21.15.94:0/1804557194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0c5380 0x7f0f9c0c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.933+0000 7f0f93fff640 1 --2- 172.21.15.94:0/1804557194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c155c30 0x7f0f9c06ca70 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f0f840097a0 tx=0x7f0f8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5a7e308abc4f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0f99d9b640 1 -- 172.21.15.94:0/1804557194 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f840315a0 con 0x7f0f9c155c30 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0f99d9b640 1 -- 172.21.15.94:0/1804557194 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f84037070 con 0x7f0f9c155c30 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0f99d9b640 1 -- 172.21.15.94:0/1804557194 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84031960 con 0x7f0f9c155c30 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 -- 172.21.15.94:0/1804557194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c155c30 msgr2=0x7f0f9c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/1804557194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c155c30 0x7f0f9c06ca70 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f0f840097a0 tx=0x7f0f8402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 -- 172.21.15.94:0/1804557194 shutdown_connections 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/1804557194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0c5380 0x7f0f9c0c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/1804557194 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c155c30 0x7f0f9c06ca70 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 -- 172.21.15.94:0/1804557194 >> 172.21.15.94:0/1804557194 conn(0x7f0f9c0ac590 msgr2=0x7f0f9c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 -- 172.21.15.94:0/1804557194 shutdown_connections 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.934+0000 7f0fa0fac640 1 -- 172.21.15.94:0/1804557194 wait complete. 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.935+0000 7f0fa0fac640 1 Processor -- start 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.935+0000 7f0fa0fac640 1 -- start start 2024-03-06T07:25:01.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0fa0fac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0b0070 0x7f0f9c14e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0fa0fac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c0c5380 0x7f0f9c1ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.940 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0fa0fac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f9c155c30 0x7f0f9c1ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0fa0fac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c06c270 con 0x7f0f9c0c5380 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0fa0fac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c0c7ad0 con 0x7f0f9c155c30 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0fa0fac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c06c550 con 0x7f0f9c0b0070 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0f93fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c0c5380 0x7f0f9c1ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0f93fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c0c5380 0x7f0f9c1ee2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59134/0 (socket says 172.21.15.94:59134) 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0f93fff640 1 -- 172.21.15.94:0/2654568447 learned_addr learned my addr 172.21.15.94:0/2654568447 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0f9a59c640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0b0070 0x7f0f9c14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.936+0000 7f0f9ad9d640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f9c155c30 0x7f0f9c1ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f93fff640 1 -- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0b0070 msgr2=0x7f0f9c14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f93fff640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0b0070 0x7f0f9c14e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f93fff640 1 -- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f9c155c30 msgr2=0x7f0f9c1ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f93fff640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f9c155c30 0x7f0f9c1ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f93fff640 1 -- 172.21.15.94:0/2654568447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c14f2c0 con 0x7f0f9c0c5380 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f93fff640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c0c5380 0x7f0f9c1ee2d0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0f84034040 tx=0x7f0f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f92ffd640 1 -- 172.21.15.94:0/2654568447 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84032630 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f92ffd640 1 -- 172.21.15.94:0/2654568447 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f840327f0 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c14efe0 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0f92ffd640 1 -- 172.21.15.94:0/2654568447 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84032be0 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.937+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c1593f0 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.938+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c1ef1e0 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.938+0000 7f0f92ffd640 1 -- 172.21.15.94:0/2654568447 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f8404e430 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.939+0000 7f0f92ffd640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.939+0000 7f0f92ffd640 1 -- 172.21.15.94:0/2654568447 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0f8402f030 con 0x7f0f9c0c5380 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.939+0000 7f0f9a59c640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.939+0000 7f0f9a59c640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c005880 tx=0x7f0f8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:01.943+0000 7f0f92ffd640 1 -- 172.21.15.94:0/2654568447 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f9c1ef1e0 con 0x7f0f9c0c5380 2024-03-06T07:25:02.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.150+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f0f9c04f330 con 0x7f0f9c0c5380 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.221+0000 7f0f92ffd640 1 -- 172.21.15.94:0/2654568447 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v160) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0f84040070 con 0x7f0f9c0c5380 2024-03-06T07:25:02.223 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked down. 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7c06b4d0 msgr2=0x7f0f7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c005880 tx=0x7f0f8c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c0c5380 msgr2=0x7f0f9c1ee2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c0c5380 0x7f0f9c1ee2d0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f0f84034040 tx=0x7f0f840046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 shutdown_connections 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7c06b4d0 0x7f0f7c06d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f9c155c30 0x7f0f9c1ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f9c0c5380 0x7f0f9c1ee2d0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 --2- 172.21.15.94:0/2654568447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f9c0b0070 0x7f0f9c14e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 >> 172.21.15.94:0/2654568447 conn(0x7f0f9c0ac590 msgr2=0x7f0f9c0adc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 shutdown_connections 2024-03-06T07:25:02.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.223+0000 7f0fa0fac640 1 -- 172.21.15.94:0/2654568447 wait complete. 2024-03-06T07:25:02.236 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:02.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.333+0000 7fe425165640 1 Processor -- start 2024-03-06T07:25:02.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe425165640 1 -- start start 2024-03-06T07:25:02.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe425165640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe42014f790 0x7fe4201529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe425165640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe4200c1aa0 0x7fe4200c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe425165640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4200c28d0 con 0x7fe4200c5c60 2024-03-06T07:25:02.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe425165640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4200c2680 con 0x7fe4200c1aa0 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe425165640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4200c2bb0 con 0x7fe42014f790 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe41e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe4200c1aa0 0x7fe4200c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe41f59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe4200c5c60 0x7fe420155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33152/0 (socket says 172.21.15.94:33152) 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.334+0000 7fe41f59e640 1 -- 172.21.15.94:0/2268841549 learned_addr learned my addr 172.21.15.94:0/2268841549 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.335+0000 7fe41ed9d640 1 --2- 172.21.15.94:0/2268841549 >> v2:172.21.15.145:3300/0 conn(0x7fe42014f790 0x7fe4201529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.335+0000 7fe41e59c640 1 -- 172.21.15.94:0/2268841549 >> v2:172.21.15.145:3300/0 conn(0x7fe42014f790 msgr2=0x7fe4201529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.335+0000 7fe41e59c640 1 --2- 172.21.15.94:0/2268841549 >> v2:172.21.15.145:3300/0 conn(0x7fe42014f790 0x7fe4201529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.335+0000 7fe41e59c640 1 -- 172.21.15.94:0/2268841549 >> v1:172.21.15.94:6789/0 conn(0x7fe4200c5c60 legacy=0x7fe420155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.335+0000 7fe41dd9b640 1 -- 172.21.15.94:0/2268841549 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855239812 0 0) 0x7fe4200c28d0 con 0x7fe4200c5c60 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.335+0000 7fe41e59c640 1 -- 172.21.15.94:0/2268841549 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4200c2e90 con 0x7fe4200c1aa0 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.335+0000 7fe41e59c640 1 --2- 172.21.15.94:0/2268841549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c1aa0 0x7fe4200c5720 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fe4200c3170 tx=0x7fe41002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25bcbc1e7cf9e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe41dd9b640 1 -- 172.21.15.94:0/2268841549 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410035020 con 0x7fe4200c1aa0 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe41dd9b640 1 -- 172.21.15.94:0/2268841549 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe410031830 con 0x7fe4200c1aa0 2024-03-06T07:25:02.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe41dd9b640 1 -- 172.21.15.94:0/2268841549 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410031c00 con 0x7fe4200c1aa0 2024-03-06T07:25:02.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 -- 172.21.15.94:0/2268841549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c1aa0 msgr2=0x7fe4200c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 --2- 172.21.15.94:0/2268841549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c1aa0 0x7fe4200c5720 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fe4200c3170 tx=0x7fe41002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 -- 172.21.15.94:0/2268841549 shutdown_connections 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 --2- 172.21.15.94:0/2268841549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c1aa0 0x7fe4200c5720 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 --2- 172.21.15.94:0/2268841549 >> v2:172.21.15.145:3300/0 conn(0x7fe42014f790 0x7fe4201529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 -- 172.21.15.94:0/2268841549 >> 172.21.15.94:0/2268841549 conn(0x7fe4200c6710 msgr2=0x7fe4200c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 -- 172.21.15.94:0/2268841549 shutdown_connections 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.336+0000 7fe425165640 1 -- 172.21.15.94:0/2268841549 wait complete. 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.337+0000 7fe425165640 1 Processor -- start 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe425165640 1 -- start start 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe425165640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4200c1aa0 0x7fe4201ec150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe425165640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c5c60 0x7fe42014bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe425165640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe42014f790 0x7fe420149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe425165640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42006cc00 con 0x7fe4200c5c60 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe425165640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200c0d60 con 0x7fe4200c1aa0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe425165640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42006cd80 con 0x7fe42014f790 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe41e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c5c60 0x7fe42014bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe41e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c5c60 0x7fe42014bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59158/0 (socket says 172.21.15.94:59158) 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe41e59c640 1 -- 172.21.15.94:0/2331243931 learned_addr learned my addr 172.21.15.94:0/2331243931 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe41ed9d640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4200c1aa0 0x7fe4201ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.338+0000 7fe41f59e640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe42014f790 0x7fe420149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe41e59c640 1 -- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe42014f790 msgr2=0x7fe420149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe41e59c640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe42014f790 0x7fe420149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe41e59c640 1 -- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4200c1aa0 msgr2=0x7fe4201ec150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe41e59c640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4200c1aa0 0x7fe4201ec150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe41e59c640 1 -- 172.21.15.94:0/2331243931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe420154610 con 0x7fe4200c5c60 2024-03-06T07:25:02.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe41e59c640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c5c60 0x7fe42014bc70 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fe410000c00 tx=0x7fe4100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe4077fe640 1 -- 172.21.15.94:0/2331243931 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410035070 con 0x7fe4200c5c60 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe4077fe640 1 -- 172.21.15.94:0/2331243931 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe410004250 con 0x7fe4200c5c60 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42014cca0 con 0x7fe4200c5c60 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe4077fe640 1 -- 172.21.15.94:0/2331243931 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41003a6f0 con 0x7fe4200c5c60 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.339+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4201eca20 con 0x7fe4200c5c60 2024-03-06T07:25:02.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.340+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42004f330 con 0x7fe4200c5c60 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.340+0000 7fe4077fe640 1 -- 172.21.15.94:0/2331243931 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe410038020 con 0x7fe4200c5c60 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.341+0000 7fe4077fe640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe40c06b4d0 0x7fe40c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.341+0000 7fe4077fe640 1 -- 172.21.15.94:0/2331243931 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe4100abec0 con 0x7fe4200c5c60 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.341+0000 7fe41ed9d640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe40c06b4d0 0x7fe40c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.341+0000 7fe41ed9d640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe40c06b4d0 0x7fe40c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe4140079a0 tx=0x7fe414008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.345+0000 7fe4077fe640 1 -- 172.21.15.94:0/2331243931 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe410040070 con 0x7fe4200c5c60 2024-03-06T07:25:02.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.553+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4200c1f40 con 0x7fe4200c5c60 2024-03-06T07:25:02.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.553+0000 7fe4077fe640 1 -- 172.21.15.94:0/2331243931 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fe410031800 con 0x7fe4200c5c60 2024-03-06T07:25:02.555 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:02.555 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:02.556 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:02.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe40c06b4d0 msgr2=0x7fe40c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe40c06b4d0 0x7fe40c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fe4140079a0 tx=0x7fe414008040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c5c60 msgr2=0x7fe42014bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c5c60 0x7fe42014bc70 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fe410000c00 tx=0x7fe4100046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 shutdown_connections 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe40c06b4d0 0x7fe40c06d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe42014f790 0x7fe420149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4200c5c60 0x7fe42014bc70 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 --2- 172.21.15.94:0/2331243931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4200c1aa0 0x7fe4201ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 >> 172.21.15.94:0/2331243931 conn(0x7fe4200c6710 msgr2=0x7fe420072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 shutdown_connections 2024-03-06T07:25:02.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:02.555+0000 7fe425165640 1 -- 172.21.15.94:0/2331243931 wait complete. 2024-03-06T07:25:02.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:02.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:03.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:04.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.082+0000 7f9eb5acf640 1 Processor -- start 2024-03-06T07:25:04.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.082+0000 7f9eb5acf640 1 -- start start 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eb5acf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9eb01550c0 0x7f9eb01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eb5acf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb00aa180 0x7f9eb00c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eb5acf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb00aa660 con 0x7f9eb01559e0 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eb5acf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb00c4610 con 0x7f9eb01550c0 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eb5acf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00aa940 con 0x7f9eb00aa180 2024-03-06T07:25:04.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9eb01550c0 0x7f9eb01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9eb01550c0 0x7f9eb01554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59172/0 (socket says 172.21.15.94:59172) 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaeffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9eb01559e0 0x7f9eb014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33162/0 (socket says 172.21.15.94:33162) 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaf7fe640 1 -- 172.21.15.94:0/1225882409 learned_addr learned my addr 172.21.15.94:0/1225882409 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaffff640 1 --2- 172.21.15.94:0/1225882409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb00aa180 0x7f9eb00c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaf7fe640 1 -- 172.21.15.94:0/1225882409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb00aa180 msgr2=0x7f9eb00c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaf7fe640 1 --2- 172.21.15.94:0/1225882409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb00aa180 0x7f9eb00c5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaf7fe640 1 -- 172.21.15.94:0/1225882409 >> v1:172.21.15.94:6789/0 conn(0x7f9eb01559e0 legacy=0x7f9eb014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eaf7fe640 1 -- 172.21.15.94:0/1225882409 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb00c4860 con 0x7f9eb01550c0 2024-03-06T07:25:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.083+0000 7f9eae7fc640 1 -- 172.21.15.94:0/1225882409 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010005166 0 0) 0x7f9eb00aa660 con 0x7f9eb01559e0 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.084+0000 7f9eaf7fe640 1 --2- 172.21.15.94:0/1225882409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01554a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009870 tx=0x7f9ea002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb06f5a5dfef38c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.084+0000 7f9eae7fc640 1 -- 172.21.15.94:0/1225882409 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00317c0 con 0x7f9eb01550c0 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.084+0000 7f9eae7fc640 1 -- 172.21.15.94:0/1225882409 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea0037070 con 0x7f9eb01550c0 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.084+0000 7f9eae7fc640 1 -- 172.21.15.94:0/1225882409 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0031b80 con 0x7f9eb01550c0 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 -- 172.21.15.94:0/1225882409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 msgr2=0x7f9eb01554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/1225882409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01554a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009870 tx=0x7f9ea002b860 comp rx=0 tx=0).stop 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 -- 172.21.15.94:0/1225882409 shutdown_connections 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/1225882409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb00aa180 0x7f9eb00c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/1225882409 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01554a0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 -- 172.21.15.94:0/1225882409 >> 172.21.15.94:0/1225882409 conn(0x7f9eb00c6710 msgr2=0x7f9eb00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 -- 172.21.15.94:0/1225882409 shutdown_connections 2024-03-06T07:25:04.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.085+0000 7f9eb5acf640 1 -- 172.21.15.94:0/1225882409 wait complete. 2024-03-06T07:25:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.086+0000 7f9eb5acf640 1 Processor -- start 2024-03-06T07:25:04.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.086+0000 7f9eb5acf640 1 -- start start 2024-03-06T07:25:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eb5acf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9eb00aa180 0x7f9eb014af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eb5acf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01ec360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eb5acf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb01559e0 0x7f9eb00a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eb5acf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb014f310 con 0x7f9eb01550c0 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eb5acf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00c1560 con 0x7f9eb00aa180 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eb5acf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb014f490 con 0x7f9eb01559e0 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01ec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01ec360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59180/0 (socket says 172.21.15.94:59180) 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 -- 172.21.15.94:0/3919631395 learned_addr learned my addr 172.21.15.94:0/3919631395 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaffff640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb01559e0 0x7f9eb00a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaf7fe640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9eb00aa180 0x7f9eb014af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 -- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb01559e0 msgr2=0x7f9eb00a88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb01559e0 0x7f9eb00a88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 -- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9eb00aa180 msgr2=0x7f9eb014af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9eb00aa180 0x7f9eb014af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaeffd640 1 -- 172.21.15.94:0/3919631395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb01e8380 con 0x7f9eb01550c0 2024-03-06T07:25:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.087+0000 7f9eaf7fe640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9eb00aa180 0x7f9eb014af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.088+0000 7f9eaeffd640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01ec360 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c4b0 tx=0x7f9ea400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.088+0000 7f9e97fff640 1 -- 172.21.15.94:0/3919631395 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4018070 con 0x7f9eb01550c0 2024-03-06T07:25:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.088+0000 7f9e97fff640 1 -- 172.21.15.94:0/3919631395 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea4007480 con 0x7f9eb01550c0 2024-03-06T07:25:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.088+0000 7f9e97fff640 1 -- 172.21.15.94:0/3919631395 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4004820 con 0x7f9eb01550c0 2024-03-06T07:25:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.088+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb01491f0 con 0x7f9eb01550c0 2024-03-06T07:25:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.088+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb01497b0 con 0x7f9eb01550c0 2024-03-06T07:25:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.089+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb004f330 con 0x7f9eb01550c0 2024-03-06T07:25:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.089+0000 7f9e97fff640 1 -- 172.21.15.94:0/3919631395 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ea401b070 con 0x7f9eb01550c0 2024-03-06T07:25:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.090+0000 7f9e97fff640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.090+0000 7f9e97fff640 1 -- 172.21.15.94:0/3919631395 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ea40894a0 con 0x7f9eb01550c0 2024-03-06T07:25:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.090+0000 7f9eaf7fe640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.091+0000 7f9eaf7fe640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0036000 tx=0x7f9ea003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.094+0000 7f9e97fff640 1 -- 172.21.15.94:0/3919631395 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ea4057c90 con 0x7f9eb01550c0 2024-03-06T07:25:04.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.302+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eb0155640 con 0x7f9eb01550c0 2024-03-06T07:25:04.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.303+0000 7f9e97fff640 1 -- 172.21.15.94:0/3919631395 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f9ea405bd30 con 0x7f9eb01550c0 2024-03-06T07:25:04.305 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:04.305 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:04.305 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.304+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e9c06b500 msgr2=0x7f9e9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0036000 tx=0x7f9ea003d040 comp rx=0 tx=0).stop 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 msgr2=0x7f9eb01ec360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01ec360 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400c4b0 tx=0x7f9ea400c970 comp rx=0 tx=0).stop 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 shutdown_connections 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9e9c06b500 0x7f9e9c06d9c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9eb01559e0 0x7f9eb00a88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9eb01550c0 0x7f9eb01ec360 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 --2- 172.21.15.94:0/3919631395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9eb00aa180 0x7f9eb014af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 >> 172.21.15.94:0/3919631395 conn(0x7f9eb00c6710 msgr2=0x7f9eb00c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 shutdown_connections 2024-03-06T07:25:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:04.305+0000 7f9eb5acf640 1 -- 172.21.15.94:0/3919631395 wait complete. 2024-03-06T07:25:04.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:04.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:05.721 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:05.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.834+0000 7fc40538c640 1 Processor -- start 2024-03-06T07:25:05.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.834+0000 7fc40538c640 1 -- start start 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc40538c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 0x7fc4001529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc40538c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc400155a40 0x7fc400155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc40538c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc400156410 con 0x7fc4000c1aa0 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc40538c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4000c2680 con 0x7fc400155a40 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc40538c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c28d0 con 0x7fc40014f790 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc3f7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc4000c1aa0 0x7fc4000c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33174/0 (socket says 172.21.15.94:33174) 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc3f7fff640 1 -- 172.21.15.94:0/3519284036 learned_addr learned my addr 172.21.15.94:0/3519284036 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc3feffd640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 0x7fc4001529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/3519284036 >> v2:172.21.15.145:3300/0 conn(0x7fc400155a40 0x7fc400155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492111429 0 0) 0x7fc400156410 con 0x7fc4000c1aa0 2024-03-06T07:25:05.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.835+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ec005730 con 0x7fc4000c1aa0 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241033217 0 0) 0x7fc3ec005730 con 0x7fc4000c1aa0 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 msgr2=0x7fc4001529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 0x7fc4001529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 >> v2:172.21.15.145:3300/0 conn(0x7fc400155a40 msgr2=0x7fc400155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 --2- 172.21.15.94:0/3519284036 >> v2:172.21.15.145:3300/0 conn(0x7fc400155a40 0x7fc400155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4000c2bb0 con 0x7fc4000c1aa0 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fc3e8001670 con 0x7fc4000c1aa0 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 >> v1:172.21.15.94:6789/0 conn(0x7fc4000c1aa0 legacy=0x7fc4000c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 0x7fc3ec005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3ec006140 0x7fc3ec006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3ec006f50 0x7fc3ec0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c2680 con 0x7fc3ec006f50 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c28d0 con 0x7fc3ec006140 2024-03-06T07:25:05.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec00dbd0 con 0x7fc40014f790 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3ec006f50 0x7fc3ec0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3f7fff640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3ec006140 0x7fc3ec006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3feffd640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 0x7fc3ec005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 msgr2=0x7fc3ec005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc40014f790 0x7fc3ec005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3ec006140 msgr2=0x7fc3ec006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3ec006140 0x7fc3ec006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.836+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/3519284036 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec005730 con 0x7fc3ec006f50 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.837+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3ec006f50 0x7fc3ec0073d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009fc0 tx=0x7fc3ec007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698b9389eed9f866 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.837+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8039070 con 0x7fc3ec006f50 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.837+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f802cc40 con 0x7fc3ec006f50 2024-03-06T07:25:05.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.837+0000 7fc3fe7fc640 1 -- 172.21.15.94:0/3519284036 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8009c10 con 0x7fc3ec006f50 2024-03-06T07:25:05.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.838+0000 7fc40538c640 1 -- 172.21.15.94:0/3519284036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3ec006f50 msgr2=0x7fc3ec0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.838+0000 7fc40538c640 1 --2- 172.21.15.94:0/3519284036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3ec006f50 0x7fc3ec0073d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009fc0 tx=0x7fc3ec007a40 comp rx=0 tx=0).stop 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.838+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/3519284036 reap_dead start 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.838+0000 7fc40538c640 1 -- 172.21.15.94:0/3519284036 shutdown_connections 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.838+0000 7fc40538c640 1 -- 172.21.15.94:0/3519284036 >> 172.21.15.94:0/3519284036 conn(0x7fc4000c6710 msgr2=0x7fc4000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.838+0000 7fc40538c640 1 -- 172.21.15.94:0/3519284036 shutdown_connections 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.838+0000 7fc40538c640 1 -- 172.21.15.94:0/3519284036 wait complete. 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.839+0000 7fc40538c640 1 Processor -- start 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.839+0000 7fc40538c640 1 -- start start 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc40538c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc400150790 0x7fc40014ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc40538c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc40014b270 0x7fc400072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc40538c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc400072fd0 0x7fc400148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc40538c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c0d60 con 0x7fc400072fd0 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc40538c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc400155520 con 0x7fc40014b270 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc40538c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c5e30 con 0x7fc400150790 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc400072fd0 0x7fc400148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc400072fd0 0x7fc400148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59206/0 (socket says 172.21.15.94:59206) 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/1434582386 learned_addr learned my addr 172.21.15.94:0/1434582386 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3feffd640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc400150790 0x7fc40014ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3f7fff640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc40014b270 0x7fc400072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc400150790 msgr2=0x7fc40014ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc400150790 0x7fc40014ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc40014b270 msgr2=0x7fc400072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc40014b270 0x7fc400072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 -- 172.21.15.94:0/1434582386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400073480 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3ff7fe640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc400072fd0 0x7fc400148d50 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc3f802c710 tx=0x7fc3f8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3f77fe640 1 -- 172.21.15.94:0/1434582386 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8039070 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc3f77fe640 1 -- 172.21.15.94:0/1434582386 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f803a040 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.840+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40014daf0 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.841+0000 7fc3f77fe640 1 -- 172.21.15.94:0/1434582386 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8009c10 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.841+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40014e0b0 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.842+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4000c7020 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.842+0000 7fc3f77fe640 1 -- 172.21.15.94:0/1434582386 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3f8045030 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.842+0000 7fc3f77fe640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3e006b4d0 0x7fc3e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.842+0000 7fc3f77fe640 1 -- 172.21.15.94:0/1434582386 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc3f80aba30 con 0x7fc400072fd0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.843+0000 7fc3feffd640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3e006b4d0 0x7fc3e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.843+0000 7fc3feffd640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3e006b4d0 0x7fc3e006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009b00 tx=0x7fc3f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:05.847+0000 7fc3f77fe640 1 -- 172.21.15.94:0/1434582386 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4000c7020 con 0x7fc400072fd0 2024-03-06T07:25:06.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.057+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc40004f330 con 0x7fc400072fd0 2024-03-06T07:25:06.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.057+0000 7fc3f77fe640 1 -- 172.21.15.94:0/1434582386 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fc3f804c060 con 0x7fc400072fd0 2024-03-06T07:25:06.059 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:06.059 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:06.059 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3e006b4d0 msgr2=0x7fc3e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3e006b4d0 0x7fc3e006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009b00 tx=0x7fc3f0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc400072fd0 msgr2=0x7fc400148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc400072fd0 0x7fc400148d50 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc3f802c710 tx=0x7fc3f8009750 comp rx=0 tx=0).stop 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 shutdown_connections 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3e006b4d0 0x7fc3e006d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc400072fd0 0x7fc400148d50 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc40014b270 0x7fc400072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 --2- 172.21.15.94:0/1434582386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc400150790 0x7fc40014ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 >> 172.21.15.94:0/1434582386 conn(0x7fc4000c6710 msgr2=0x7fc40006cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 shutdown_connections 2024-03-06T07:25:06.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:06.059+0000 7fc40538c640 1 -- 172.21.15.94:0/1434582386 wait complete. 2024-03-06T07:25:06.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:06.475 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:07.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:07.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.587+0000 7fe7d570c640 1 Processor -- start 2024-03-06T07:25:07.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7d570c640 1 -- start start 2024-03-06T07:25:07.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7d570c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe7d00c88c0 0x7fe7d0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7d570c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe7d0158b60 0x7fe7d0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7d570c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d00c73d0 con 0x7fe7d014f360 2024-03-06T07:25:07.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7d570c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d00c8dd0 con 0x7fe7d0158b60 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7d570c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d00be2e0 con 0x7fe7d00c88c0 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7cf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe7d0158b60 0x7fe7d0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7cf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe7d0158b60 0x7fe7d0158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59218/0 (socket says 172.21.15.94:59218) 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.588+0000 7fe7cf7fe640 1 -- 172.21.15.94:0/4164504143 learned_addr learned my addr 172.21.15.94:0/4164504143 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.589+0000 7fe7ce7fc640 1 --2- 172.21.15.94:0/4164504143 >> v2:172.21.15.145:3300/0 conn(0x7fe7d00c88c0 0x7fe7d0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.589+0000 7fe7cf7fe640 1 -- 172.21.15.94:0/4164504143 >> v2:172.21.15.145:3300/0 conn(0x7fe7d00c88c0 msgr2=0x7fe7d0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.589+0000 7fe7cf7fe640 1 --2- 172.21.15.94:0/4164504143 >> v2:172.21.15.145:3300/0 conn(0x7fe7d00c88c0 0x7fe7d0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.589+0000 7fe7cf7fe640 1 -- 172.21.15.94:0/4164504143 >> v1:172.21.15.145:6789/0 conn(0x7fe7d014f360 legacy=0x7fe7d00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.589+0000 7fe7cf7fe640 1 -- 172.21.15.94:0/4164504143 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d00c7670 con 0x7fe7d0158b60 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.589+0000 7fe7cf7fe640 1 --2- 172.21.15.94:0/4164504143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d0158b60 0x7fe7d0158ff0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00c84c0 tx=0x7fe7c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cdaa6d2ba9397d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7cdffb640 1 -- 172.21.15.94:0/4164504143 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c802e070 con 0x7fe7d0158b60 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7cdffb640 1 -- 172.21.15.94:0/4164504143 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c8037070 con 0x7fe7d0158b60 2024-03-06T07:25:07.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7cdffb640 1 -- 172.21.15.94:0/4164504143 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8031a00 con 0x7fe7d0158b60 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 -- 172.21.15.94:0/4164504143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d0158b60 msgr2=0x7fe7d0158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 --2- 172.21.15.94:0/4164504143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d0158b60 0x7fe7d0158ff0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00c84c0 tx=0x7fe7c802b880 comp rx=0 tx=0).stop 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 -- 172.21.15.94:0/4164504143 shutdown_connections 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 --2- 172.21.15.94:0/4164504143 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d0158b60 0x7fe7d0158ff0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 --2- 172.21.15.94:0/4164504143 >> v2:172.21.15.145:3300/0 conn(0x7fe7d00c88c0 0x7fe7d0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 -- 172.21.15.94:0/4164504143 >> 172.21.15.94:0/4164504143 conn(0x7fe7d00ac570 msgr2=0x7fe7d00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 -- 172.21.15.94:0/4164504143 shutdown_connections 2024-03-06T07:25:07.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.590+0000 7fe7d570c640 1 -- 172.21.15.94:0/4164504143 wait complete. 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 Processor -- start 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 -- start start 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d00c88c0 0x7fe7d0150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7d014f360 0x7fe7d00be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7d0158b60 0x7fe7d01511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d00c5a50 con 0x7fe7d00c88c0 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d00c7c30 con 0x7fe7d0158b60 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7d570c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d00c5bd0 con 0x7fe7d014f360 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7ceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d00c88c0 0x7fe7d0150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7ceffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d00c88c0 0x7fe7d0150830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59232/0 (socket says 172.21.15.94:59232) 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.592+0000 7fe7ceffd640 1 -- 172.21.15.94:0/998211556 learned_addr learned my addr 172.21.15.94:0/998211556 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7ce7fc640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7d014f360 0x7fe7d00be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7cf7fe640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7d0158b60 0x7fe7d01511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7ceffd640 1 -- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7d014f360 msgr2=0x7fe7d00be4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7ceffd640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7d014f360 0x7fe7d00be4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7ceffd640 1 -- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7d0158b60 msgr2=0x7fe7d01511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7ceffd640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7d0158b60 0x7fe7d01511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7ceffd640 1 -- 172.21.15.94:0/998211556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d0150d70 con 0x7fe7d00c88c0 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7ceffd640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d00c88c0 0x7fe7d0150830 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000c7c0 tx=0x7fe7c000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7b77fe640 1 -- 172.21.15.94:0/998211556 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0016020 con 0x7fe7d00c88c0 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7b77fe640 1 -- 172.21.15.94:0/998211556 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c000aa50 con 0x7fe7d00c88c0 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d00bd630 con 0x7fe7d00c88c0 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7b77fe640 1 -- 172.21.15.94:0/998211556 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0012420 con 0x7fe7d00c88c0 2024-03-06T07:25:07.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.593+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d014fa70 con 0x7fe7d00c88c0 2024-03-06T07:25:07.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.594+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d004f330 con 0x7fe7d00c88c0 2024-03-06T07:25:07.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.594+0000 7fe7b77fe640 1 -- 172.21.15.94:0/998211556 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7c0019020 con 0x7fe7d00c88c0 2024-03-06T07:25:07.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.595+0000 7fe7b77fe640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe7bc06b4d0 0x7fe7bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.595+0000 7fe7b77fe640 1 -- 172.21.15.94:0/998211556 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe7c008a060 con 0x7fe7d00c88c0 2024-03-06T07:25:07.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.595+0000 7fe7ce7fc640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe7bc06b4d0 0x7fe7bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.595+0000 7fe7ce7fc640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe7bc06b4d0 0x7fe7bc06d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.599+0000 7fe7b77fe640 1 -- 172.21.15.94:0/998211556 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7c0020070 con 0x7fe7d00c88c0 2024-03-06T07:25:07.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.806+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7d00c1640 con 0x7fe7d00c88c0 2024-03-06T07:25:07.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.807+0000 7fe7b77fe640 1 -- 172.21.15.94:0/998211556 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fe7c0058de0 con 0x7fe7d00c88c0 2024-03-06T07:25:07.809 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:07.809 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:07.809 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:07.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe7bc06b4d0 msgr2=0x7fe7bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe7bc06b4d0 0x7fe7bc06d990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d00c88c0 msgr2=0x7fe7d0150830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d00c88c0 0x7fe7d0150830 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000c7c0 tx=0x7fe7c000cc80 comp rx=0 tx=0).stop 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 shutdown_connections 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe7bc06b4d0 0x7fe7bc06d990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7d0158b60 0x7fe7d01511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7d014f360 0x7fe7d00be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 --2- 172.21.15.94:0/998211556 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe7d00c88c0 0x7fe7d0150830 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 >> 172.21.15.94:0/998211556 conn(0x7fe7d00ac570 msgr2=0x7fe7d00ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 shutdown_connections 2024-03-06T07:25:07.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:07.809+0000 7fe7d570c640 1 -- 172.21.15.94:0/998211556 wait complete. 2024-03-06T07:25:08.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:08.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:09.224 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:09.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.334+0000 7f47e493a640 1 Processor -- start 2024-03-06T07:25:09.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.334+0000 7f47e493a640 1 -- start start 2024-03-06T07:25:09.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47e493a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f47e0152960 0x7f47e006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47e493a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47e01592d0 0x7f47e01552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47e493a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e0152d40 con 0x7f47e00ac260 2024-03-06T07:25:09.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47e493a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e0066e30 con 0x7f47e01592d0 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47e493a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e00c1dc0 con 0x7f47e0152960 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47ddd9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47e01592d0 0x7f47e01552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47ddd9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47e01592d0 0x7f47e01552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37872/0 (socket says 172.21.15.94:37872) 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47ddd9b640 1 -- 172.21.15.94:0/121724785 learned_addr learned my addr 172.21.15.94:0/121724785 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47ded9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f47e00ac260 0x7f47e0155810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43014/0 (socket says 172.21.15.94:43014) 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.335+0000 7f47de59c640 1 --2- 172.21.15.94:0/121724785 >> v2:172.21.15.145:3300/0 conn(0x7f47e0152960 0x7f47e006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.336+0000 7f47ddd9b640 1 -- 172.21.15.94:0/121724785 >> v2:172.21.15.145:3300/0 conn(0x7f47e0152960 msgr2=0x7f47e006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.336+0000 7f47ddd9b640 1 --2- 172.21.15.94:0/121724785 >> v2:172.21.15.145:3300/0 conn(0x7f47e0152960 0x7f47e006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.336+0000 7f47ddd9b640 1 -- 172.21.15.94:0/121724785 >> v1:172.21.15.94:6789/0 conn(0x7f47e00ac260 legacy=0x7f47e0155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.336+0000 7f47dd59a640 1 -- 172.21.15.94:0/121724785 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490454506 0 0) 0x7f47e0152d40 con 0x7f47e00ac260 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.336+0000 7f47ddd9b640 1 -- 172.21.15.94:0/121724785 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e00c1b50 con 0x7f47e01592d0 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.336+0000 7f47ddd9b640 1 --2- 172.21.15.94:0/121724785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e01592d0 0x7f47e01552d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f47e006b1c0 tx=0x7f47d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b56eac46149f63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47dd59a640 1 -- 172.21.15.94:0/121724785 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4034070 con 0x7f47e01592d0 2024-03-06T07:25:09.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47dd59a640 1 -- 172.21.15.94:0/121724785 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d4030860 con 0x7f47e01592d0 2024-03-06T07:25:09.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47dd59a640 1 -- 172.21.15.94:0/121724785 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4030c30 con 0x7f47e01592d0 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 -- 172.21.15.94:0/121724785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e01592d0 msgr2=0x7f47e01552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 --2- 172.21.15.94:0/121724785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e01592d0 0x7f47e01552d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f47e006b1c0 tx=0x7f47d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 -- 172.21.15.94:0/121724785 shutdown_connections 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 --2- 172.21.15.94:0/121724785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e01592d0 0x7f47e01552d0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 --2- 172.21.15.94:0/121724785 >> v2:172.21.15.145:3300/0 conn(0x7f47e0152960 0x7f47e006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 -- 172.21.15.94:0/121724785 >> 172.21.15.94:0/121724785 conn(0x7f47e00c70b0 msgr2=0x7f47e00c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 -- 172.21.15.94:0/121724785 shutdown_connections 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.337+0000 7f47e493a640 1 -- 172.21.15.94:0/121724785 wait complete. 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.338+0000 7f47e493a640 1 Processor -- start 2024-03-06T07:25:09.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47e493a640 1 -- start start 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47e493a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e00ac260 0x7f47e01de930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47e493a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e0152960 0x7f47e01ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47e493a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e0152e20 0x7f47e01532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47e493a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e00c20a0 con 0x7f47e00ac260 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47e493a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e0159760 con 0x7f47e0152960 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47e493a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e00c2220 con 0x7f47e0152e20 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47de59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e00ac260 0x7f47e01de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47de59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e00ac260 0x7f47e01de930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37882/0 (socket says 172.21.15.94:37882) 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47de59c640 1 -- 172.21.15.94:0/3411488761 learned_addr learned my addr 172.21.15.94:0/3411488761 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47ddd9b640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e0152960 0x7f47e01ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.339+0000 7f47ded9d640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e0152e20 0x7f47e01532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47de59c640 1 -- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e0152e20 msgr2=0x7f47e01532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47de59c640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e0152e20 0x7f47e01532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47de59c640 1 -- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e0152960 msgr2=0x7f47e01ee080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47de59c640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e0152960 0x7f47e01ee080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47de59c640 1 -- 172.21.15.94:0/3411488761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e01dee70 con 0x7f47e00ac260 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47ded9d640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e0152e20 0x7f47e01532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47ddd9b640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e0152960 0x7f47e01ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47de59c640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e00ac260 0x7f47e01de930 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f47d000e9b0 tx=0x7f47d000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47ceffd640 1 -- 172.21.15.94:0/3411488761 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0017070 con 0x7f47e00ac260 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47ceffd640 1 -- 172.21.15.94:0/3411488761 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d000c9a0 con 0x7f47e00ac260 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47ceffd640 1 -- 172.21.15.94:0/3411488761 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d000cd90 con 0x7f47e00ac260 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e01ee950 con 0x7f47e00ac260 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.340+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e01567f0 con 0x7f47e00ac260 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.341+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e004f330 con 0x7f47e00ac260 2024-03-06T07:25:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.341+0000 7f47ceffd640 1 -- 172.21.15.94:0/3411488761 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47d001a070 con 0x7f47e00ac260 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.342+0000 7f47ceffd640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47c006b4d0 0x7f47c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.342+0000 7f47ceffd640 1 -- 172.21.15.94:0/3411488761 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f47d008a2f0 con 0x7f47e00ac260 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.342+0000 7f47ddd9b640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47c006b4d0 0x7f47c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.343+0000 7f47ddd9b640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47c006b4d0 0x7f47c006d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f47d4036000 tx=0x7f47d403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.346+0000 7f47ceffd640 1 -- 172.21.15.94:0/3411488761 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47d0059070 con 0x7f47e00ac260 2024-03-06T07:25:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.553+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47e0156270 con 0x7f47e00ac260 2024-03-06T07:25:09.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.553+0000 7f47ceffd640 1 -- 172.21.15.94:0/3411488761 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f47d0058970 con 0x7f47e00ac260 2024-03-06T07:25:09.556 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:09.556 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:09.556 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:09.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47c006b4d0 msgr2=0x7f47c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47c006b4d0 0x7f47c006d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f47d4036000 tx=0x7f47d403b040 comp rx=0 tx=0).stop 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e00ac260 msgr2=0x7f47e01de930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e00ac260 0x7f47e01de930 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f47d000e9b0 tx=0x7f47d000ee70 comp rx=0 tx=0).stop 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 shutdown_connections 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f47c006b4d0 0x7f47c006d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e0152e20 0x7f47e01532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e0152960 0x7f47e01ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 --2- 172.21.15.94:0/3411488761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47e00ac260 0x7f47e01de930 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.555+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 >> 172.21.15.94:0/3411488761 conn(0x7f47e00c70b0 msgr2=0x7f47e006b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.556+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 shutdown_connections 2024-03-06T07:25:09.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:09.556+0000 7f47e493a640 1 -- 172.21.15.94:0/3411488761 wait complete. 2024-03-06T07:25:09.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:09.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:10.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:11.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.081+0000 7fe1607f7640 1 Processor -- start 2024-03-06T07:25:11.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.081+0000 7fe1607f7640 1 -- start start 2024-03-06T07:25:11.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe1607f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe1581527b0 0x7fe158159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe1607f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe1607f7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1580c7c60 con 0x7fe1580c7180 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe1607f7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe158159640 con 0x7fe1581527b0 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe1607f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580c8060 con 0x7fe1580c88b0 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe15e593640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe1581527b0 0x7fe158159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe15e593640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe1581527b0 0x7fe158159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37894/0 (socket says 172.21.15.94:37894) 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.082+0000 7fe15e593640 1 -- 172.21.15.94:0/3334389447 learned_addr learned my addr 172.21.15.94:0/3334389447 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.083+0000 7fe15e593640 1 -- 172.21.15.94:0/3334389447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 msgr2=0x7fe1580c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.083+0000 7fe15ed94640 1 --2- 172.21.15.94:0/3334389447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.083+0000 7fe15e593640 1 --2- 172.21.15.94:0/3334389447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580c5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.083+0000 7fe15e593640 1 -- 172.21.15.94:0/3334389447 >> v1:172.21.15.145:6789/0 conn(0x7fe1580c7180 legacy=0x7fe158155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.083+0000 7fe15e593640 1 -- 172.21.15.94:0/3334389447 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580c76a0 con 0x7fe1581527b0 2024-03-06T07:25:11.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.083+0000 7fe15e593640 1 --2- 172.21.15.94:0/3334389447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1581527b0 0x7fe158159100 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fe154009e50 tx=0x7fe15402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a89ab469e734196 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe15d591640 1 -- 172.21.15.94:0/3334389447 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15402c9e0 con 0x7fe1581527b0 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe15d591640 1 -- 172.21.15.94:0/3334389447 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15402cb80 con 0x7fe1581527b0 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe15d591640 1 -- 172.21.15.94:0/3334389447 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1540343f0 con 0x7fe1581527b0 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 -- 172.21.15.94:0/3334389447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1581527b0 msgr2=0x7fe158159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 --2- 172.21.15.94:0/3334389447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1581527b0 0x7fe158159100 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fe154009e50 tx=0x7fe15402be40 comp rx=0 tx=0).stop 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 -- 172.21.15.94:0/3334389447 shutdown_connections 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 --2- 172.21.15.94:0/3334389447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 --2- 172.21.15.94:0/3334389447 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1581527b0 0x7fe158159100 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 -- 172.21.15.94:0/3334389447 >> 172.21.15.94:0/3334389447 conn(0x7fe1580ac560 msgr2=0x7fe1580ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 -- 172.21.15.94:0/3334389447 shutdown_connections 2024-03-06T07:25:11.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.084+0000 7fe1607f7640 1 -- 172.21.15.94:0/3334389447 wait complete. 2024-03-06T07:25:11.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.085+0000 7fe1607f7640 1 Processor -- start 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.085+0000 7fe1607f7640 1 -- start start 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe1607f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1580c7180 0x7fe158158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe1607f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe1607f7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1581527b0 0x7fe1580c1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe1607f7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158155600 con 0x7fe1580c7180 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe1607f7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158155320 con 0x7fe1581527b0 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe1607f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580c8d60 con 0x7fe1580c88b0 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1580c7180 0x7fe158158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1580c7180 0x7fe158158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37896/0 (socket says 172.21.15.94:37896) 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 -- 172.21.15.94:0/718265673 learned_addr learned my addr 172.21.15.94:0/718265673 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15dd92640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15ed94640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1581527b0 0x7fe1580c1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 -- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 msgr2=0x7fe1580ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 -- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1581527b0 msgr2=0x7fe1580c1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1581527b0 0x7fe1580c1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.086+0000 7fe15e593640 1 -- 172.21.15.94:0/718265673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580c4a50 con 0x7fe1580c7180 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.087+0000 7fe15ed94640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1581527b0 0x7fe1580c1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.087+0000 7fe15e593640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1580c7180 0x7fe158158440 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe154009e50 tx=0x7fe154004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.087+0000 7fe14effd640 1 -- 172.21.15.94:0/718265673 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154043020 con 0x7fe1580c7180 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.087+0000 7fe14effd640 1 -- 172.21.15.94:0/718265673 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15403e450 con 0x7fe1580c7180 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.087+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1581570a0 con 0x7fe1580c7180 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.087+0000 7fe14effd640 1 -- 172.21.15.94:0/718265673 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15403e840 con 0x7fe1580c7180 2024-03-06T07:25:11.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.087+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1580ca7a0 con 0x7fe1580c7180 2024-03-06T07:25:11.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.088+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15804f330 con 0x7fe1580c7180 2024-03-06T07:25:11.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.088+0000 7fe14effd640 1 -- 172.21.15.94:0/718265673 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe154046020 con 0x7fe1580c7180 2024-03-06T07:25:11.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.089+0000 7fe14effd640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe13c06b4d0 0x7fe13c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.089+0000 7fe14effd640 1 -- 172.21.15.94:0/718265673 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe1540acc70 con 0x7fe1580c7180 2024-03-06T07:25:11.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.089+0000 7fe15dd92640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe13c06b4d0 0x7fe13c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.090+0000 7fe15dd92640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe13c06b4d0 0x7fe13c06d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe1580754e0 tx=0x7fe148009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.093+0000 7fe14effd640 1 -- 172.21.15.94:0/718265673 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe15404d030 con 0x7fe1580c7180 2024-03-06T07:25:11.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.299+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe15806ce00 con 0x7fe1580c7180 2024-03-06T07:25:11.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.300+0000 7fe14effd640 1 -- 172.21.15.94:0/718265673 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fe1580c7c60 con 0x7fe1580c7180 2024-03-06T07:25:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:11.302 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:11.302 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:11.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.301+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe13c06b4d0 msgr2=0x7fe13c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.301+0000 7fe1607f7640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe13c06b4d0 0x7fe13c06d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe1580754e0 tx=0x7fe148009040 comp rx=0 tx=0).stop 2024-03-06T07:25:11.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.301+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1580c7180 msgr2=0x7fe158158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.301+0000 7fe1607f7640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1580c7180 0x7fe158158440 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe154009e50 tx=0x7fe154004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 shutdown_connections 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe13c06b4d0 0x7fe13c06d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1581527b0 0x7fe1580c1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1580c88b0 0x7fe1580ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 --2- 172.21.15.94:0/718265673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1580c7180 0x7fe158158440 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 >> 172.21.15.94:0/718265673 conn(0x7fe1580ac560 msgr2=0x7fe158150a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 shutdown_connections 2024-03-06T07:25:11.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:11.302+0000 7fe1607f7640 1 -- 172.21.15.94:0/718265673 wait complete. 2024-03-06T07:25:11.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:11.717 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:12.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:12.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.828+0000 7f2fb6a84640 1 Processor -- start 2024-03-06T07:25:12.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.828+0000 7f2fb6a84640 1 -- start start 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.828+0000 7f2fb6a84640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2fb014f360 0x7f2fb0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.828+0000 7f2fb6a84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c88c0 0x7f2fb00c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.828+0000 7f2fb6a84640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb00c73d0 con 0x7f2fb00c5880 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.828+0000 7f2fb6a84640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb00c8dd0 con 0x7f2fb014f360 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.828+0000 7f2fb6a84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00be2e0 con 0x7f2fb00c88c0 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2fb014f360 0x7f2fb0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2fb014f360 0x7f2fb0155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37910/0 (socket says 172.21.15.94:37910) 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 -- 172.21.15.94:0/1624513673 learned_addr learned my addr 172.21.15.94:0/1624513673 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 -- 172.21.15.94:0/1624513673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c88c0 msgr2=0x7f2fb00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 --2- 172.21.15.94:0/1624513673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c88c0 0x7f2fb00c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 -- 172.21.15.94:0/1624513673 >> v1:172.21.15.145:6789/0 conn(0x7f2fb00c5880 legacy=0x7f2fb0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:12.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 -- 172.21.15.94:0/1624513673 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00c7670 con 0x7f2fb014f360 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.829+0000 7f2fb4820640 1 --2- 172.21.15.94:0/1624513673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb0155c10 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009e50 tx=0x7f2fa802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17afef3ee048bf29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2faf7fe640 1 -- 172.21.15.94:0/1624513673 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa802c9e0 con 0x7f2fb014f360 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2faf7fe640 1 -- 172.21.15.94:0/1624513673 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa802cb80 con 0x7f2fb014f360 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2faf7fe640 1 -- 172.21.15.94:0/1624513673 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa80343f0 con 0x7f2fb014f360 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 -- 172.21.15.94:0/1624513673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 msgr2=0x7f2fb0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/1624513673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb0155c10 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009e50 tx=0x7f2fa802be40 comp rx=0 tx=0).stop 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 -- 172.21.15.94:0/1624513673 shutdown_connections 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/1624513673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c88c0 0x7f2fb00c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/1624513673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb0155c10 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 -- 172.21.15.94:0/1624513673 >> 172.21.15.94:0/1624513673 conn(0x7f2fb00ac570 msgr2=0x7f2fb00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 -- 172.21.15.94:0/1624513673 shutdown_connections 2024-03-06T07:25:12.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.830+0000 7f2fb6a84640 1 -- 172.21.15.94:0/1624513673 wait complete. 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.831+0000 7f2fb6a84640 1 Processor -- start 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.831+0000 7f2fb6a84640 1 -- start start 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb6a84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c5880 0x7f2fb01ebcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb6a84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fb00c88c0 0x7f2fb01eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb6a84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb00c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb6a84640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0159270 con 0x7f2fb014f360 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb6a84640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00c7c30 con 0x7f2fb00c88c0 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb6a84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb01593f0 con 0x7f2fb00c5880 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb5021640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb00c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb5021640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb00c08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37914/0 (socket says 172.21.15.94:37914) 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.832+0000 7f2fb5021640 1 -- 172.21.15.94:0/2068718289 learned_addr learned my addr 172.21.15.94:0/2068718289 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:12.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb4820640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c5880 0x7f2fb01ebcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2faffff640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fb00c88c0 0x7f2fb01eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb5021640 1 -- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c5880 msgr2=0x7f2fb01ebcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb5021640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c5880 0x7f2fb01ebcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb5021640 1 -- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fb00c88c0 msgr2=0x7f2fb01eb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb5021640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fb00c88c0 0x7f2fb01eb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb5021640 1 -- 172.21.15.94:0/2068718289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00c0f70 con 0x7f2fb014f360 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb5021640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb00c08d0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c390 tx=0x7f2fa000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fad7fa640 1 -- 172.21.15.94:0/2068718289 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0017070 con 0x7f2fb014f360 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fad7fa640 1 -- 172.21.15.94:0/2068718289 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa00035f0 con 0x7f2fb014f360 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb00bf5e0 con 0x7f2fb014f360 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fad7fa640 1 -- 172.21.15.94:0/2068718289 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa00039e0 con 0x7f2fb014f360 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.833+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb00c1250 con 0x7f2fb014f360 2024-03-06T07:25:12.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.834+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb004f330 con 0x7f2fb014f360 2024-03-06T07:25:12.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.834+0000 7f2fad7fa640 1 -- 172.21.15.94:0/2068718289 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fa001a070 con 0x7f2fb014f360 2024-03-06T07:25:12.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.835+0000 7f2fad7fa640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f9c06b4d0 0x7f2f9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:12.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.835+0000 7f2fad7fa640 1 -- 172.21.15.94:0/2068718289 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2fa00896d0 con 0x7f2fb014f360 2024-03-06T07:25:12.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.835+0000 7f2fb4820640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f9c06b4d0 0x7f2f9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:12.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.836+0000 7f2fb4820640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f9c06b4d0 0x7f2f9c06d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8035000 tx=0x7f2fa8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:12.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:12.839+0000 7f2fad7fa640 1 -- 172.21.15.94:0/2068718289 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fa0058450 con 0x7f2fb014f360 2024-03-06T07:25:13.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.051+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fb0158e00 con 0x7f2fb014f360 2024-03-06T07:25:13.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.052+0000 7f2fad7fa640 1 -- 172.21.15.94:0/2068718289 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f2fa0057d50 con 0x7f2fb014f360 2024-03-06T07:25:13.054 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:13.054 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:13.054 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:13.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.053+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f9c06b4d0 msgr2=0x7f2f9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.053+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f9c06b4d0 0x7f2f9c06d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8035000 tx=0x7f2fa8030000 comp rx=0 tx=0).stop 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.053+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 msgr2=0x7f2fb00c08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.053+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb00c08d0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000c390 tx=0x7f2fa000c850 comp rx=0 tx=0).stop 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 shutdown_connections 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2f9c06b4d0 0x7f2f9c06d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fb014f360 0x7f2fb00c08d0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fb00c88c0 0x7f2fb01eb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 --2- 172.21.15.94:0/2068718289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fb00c5880 0x7f2fb01ebcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 >> 172.21.15.94:0/2068718289 conn(0x7f2fb00ac570 msgr2=0x7f2fb0153f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 shutdown_connections 2024-03-06T07:25:13.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:13.054+0000 7f2fb6a84640 1 -- 172.21.15.94:0/2068718289 wait complete. 2024-03-06T07:25:13.471 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:13.472 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:14.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:14.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.583+0000 7f713b9e9640 1 Processor -- start 2024-03-06T07:25:14.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.583+0000 7f713b9e9640 1 -- start start 2024-03-06T07:25:14.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f713b9e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7134072f70 0x7f7134154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f713b9e9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f71340c6560 0x7f7134154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f713b9e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71340bdc30 con 0x7f71340bbf80 2024-03-06T07:25:14.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f713b9e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71340bd690 con 0x7f7134072f70 2024-03-06T07:25:14.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f713b9e9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71340bd950 con 0x7f71340c6560 2024-03-06T07:25:14.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f7139785640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7134072f70 0x7f7134154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f7139785640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7134072f70 0x7f7134154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37918/0 (socket says 172.21.15.94:37918) 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f7139f86640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f71340bbf80 0x7f7134156670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43022/0 (socket says 172.21.15.94:43022) 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f7139785640 1 -- 172.21.15.94:0/3700774071 learned_addr learned my addr 172.21.15.94:0/3700774071 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.584+0000 7f7138f84640 1 --2- 172.21.15.94:0/3700774071 >> v2:172.21.15.145:3300/0 conn(0x7f71340c6560 0x7f7134154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.585+0000 7f7139785640 1 -- 172.21.15.94:0/3700774071 >> v2:172.21.15.145:3300/0 conn(0x7f71340c6560 msgr2=0x7f7134154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.585+0000 7f7139785640 1 --2- 172.21.15.94:0/3700774071 >> v2:172.21.15.145:3300/0 conn(0x7f71340c6560 0x7f7134154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.585+0000 7f7139785640 1 -- 172.21.15.94:0/3700774071 >> v1:172.21.15.94:6789/0 conn(0x7f71340bbf80 legacy=0x7f7134156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.585+0000 7f712bfff640 1 -- 172.21.15.94:0/3700774071 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663344288 0 0) 0x7f71340bdc30 con 0x7f71340bbf80 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.585+0000 7f7139785640 1 -- 172.21.15.94:0/3700774071 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71340c5540 con 0x7f7134072f70 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.585+0000 7f7139785640 1 --2- 172.21.15.94:0/3700774071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7134072f70 0x7f7134154570 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f71340c4650 tx=0x7f713002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc465418e992f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.585+0000 7f712bfff640 1 -- 172.21.15.94:0/3700774071 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130035020 con 0x7f7134072f70 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f712bfff640 1 -- 172.21.15.94:0/3700774071 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7130031870 con 0x7f7134072f70 2024-03-06T07:25:14.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f712bfff640 1 -- 172.21.15.94:0/3700774071 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130031c40 con 0x7f7134072f70 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 -- 172.21.15.94:0/3700774071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7134072f70 msgr2=0x7f7134154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 --2- 172.21.15.94:0/3700774071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7134072f70 0x7f7134154570 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f71340c4650 tx=0x7f713002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 -- 172.21.15.94:0/3700774071 shutdown_connections 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 --2- 172.21.15.94:0/3700774071 >> v2:172.21.15.145:3300/0 conn(0x7f71340c6560 0x7f7134154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 --2- 172.21.15.94:0/3700774071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7134072f70 0x7f7134154570 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 -- 172.21.15.94:0/3700774071 >> 172.21.15.94:0/3700774071 conn(0x7f713406b1a0 msgr2=0x7f713406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 -- 172.21.15.94:0/3700774071 shutdown_connections 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.586+0000 7f713b9e9640 1 -- 172.21.15.94:0/3700774071 wait complete. 2024-03-06T07:25:14.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.587+0000 7f713b9e9640 1 Processor -- start 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.587+0000 7f713b9e9640 1 -- start start 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f713b9e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7134072f70 0x7f71340aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f713b9e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71340bbf80 0x7f7134156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f713b9e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71340c6560 0x7f71340a9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f713b9e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7134073350 con 0x7f71340c6560 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f713b9e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71340bba60 con 0x7f71340bbf80 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f713b9e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71340734d0 con 0x7f7134072f70 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71340c6560 0x7f71340a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71340c6560 0x7f71340a9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37932/0 (socket says 172.21.15.94:37932) 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 -- 172.21.15.94:0/4198115666 learned_addr learned my addr 172.21.15.94:0/4198115666 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7138f84640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71340bbf80 0x7f7134156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139785640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7134072f70 0x7f71340aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 -- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7134072f70 msgr2=0x7f71340aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7134072f70 0x7f71340aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 -- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71340bbf80 msgr2=0x7f7134156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71340bbf80 0x7f7134156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.588+0000 7f7139f86640 1 -- 172.21.15.94:0/4198115666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71341e97f0 con 0x7f71340c6560 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.589+0000 7f7139f86640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71340c6560 0x7f71340a9830 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c780 tx=0x7f712c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.589+0000 7f7129ffb640 1 -- 172.21.15.94:0/4198115666 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c018070 con 0x7f71340c6560 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.589+0000 7f7129ffb640 1 -- 172.21.15.94:0/4198115666 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c00a640 con 0x7f71340c6560 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.589+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71341e9ad0 con 0x7f71340c6560 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.589+0000 7f7129ffb640 1 -- 172.21.15.94:0/4198115666 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c00aab0 con 0x7f71340c6560 2024-03-06T07:25:14.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.589+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71340c0be0 con 0x7f71340c6560 2024-03-06T07:25:14.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.590+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71340c06f0 con 0x7f71340c6560 2024-03-06T07:25:14.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.590+0000 7f7129ffb640 1 -- 172.21.15.94:0/4198115666 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f712c01b070 con 0x7f71340c6560 2024-03-06T07:25:14.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.591+0000 7f7129ffb640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.591+0000 7f7129ffb640 1 -- 172.21.15.94:0/4198115666 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f712c089870 con 0x7f71340c6560 2024-03-06T07:25:14.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.591+0000 7f7139785640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.592+0000 7f7139785640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711806b4d0 0x7f711806d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f7130036000 tx=0x7f713003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.595+0000 7f7129ffb640 1 -- 172.21.15.94:0/4198115666 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f712c057fa0 con 0x7f71340c6560 2024-03-06T07:25:14.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.803+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f713404f330 con 0x7f71340c6560 2024-03-06T07:25:14.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.803+0000 7f7129ffb640 1 -- 172.21.15.94:0/4198115666 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f712c05c040 con 0x7f71340c6560 2024-03-06T07:25:14.805 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:14.806 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:14.806 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.805+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711806b4d0 msgr2=0x7f711806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.805+0000 7f713b9e9640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711806b4d0 0x7f711806d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f7130036000 tx=0x7f713003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.805+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71340c6560 msgr2=0x7f71340a9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.805+0000 7f713b9e9640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71340c6560 0x7f71340a9830 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c780 tx=0x7f712c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 shutdown_connections 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71340c6560 0x7f71340a9830 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71340bbf80 0x7f7134156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 --2- 172.21.15.94:0/4198115666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7134072f70 0x7f71340aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 >> 172.21.15.94:0/4198115666 conn(0x7f713406b1a0 msgr2=0x7f7134150140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 shutdown_connections 2024-03-06T07:25:14.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:14.806+0000 7f713b9e9640 1 -- 172.21.15.94:0/4198115666 wait complete. 2024-03-06T07:25:15.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:15.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:15.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:15.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:16.221 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.332+0000 7f91e6de9640 1 Processor -- start 2024-03-06T07:25:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.332+0000 7f91e6de9640 1 -- start start 2024-03-06T07:25:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.332+0000 7f91e6de9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 0x7f91e0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.332+0000 7f91e6de9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00c73d0 con 0x7f91e014f360 2024-03-06T07:25:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.332+0000 7f91e6de9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00be2e0 con 0x7f91e00c88c0 2024-03-06T07:25:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.332+0000 7f91e6de9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e0159700 con 0x7f91e0159270 2024-03-06T07:25:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91e4b85640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f91e014f360 0x7f91e00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43026/0 (socket says 172.21.15.94:43026) 2024-03-06T07:25:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91e4b85640 1 -- 172.21.15.94:0/4290928799 learned_addr learned my addr 172.21.15.94:0/4290928799 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91e5386640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 0x7f91e0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183020049 0 0) 0x7f91e00c73d0 con 0x7f91e014f360 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d0003520 con 0x7f91e014f360 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3699056644 0 0) 0x7f91d0003520 con 0x7f91e014f360 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 msgr2=0x7f91e0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 0x7f91e0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 >> v1:172.21.15.145:6789/0 conn(0x7f91e00c88c0 legacy=0x7f91e0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00c7670 con 0x7f91e014f360 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f91d4001670 con 0x7f91e014f360 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 >> v1:172.21.15.94:6789/0 conn(0x7f91e014f360 legacy=0x7f91e00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 0x7f91d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91d0003f60 0x7f91d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e0159700 con 0x7f91d0004d40 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d000bb50 con 0x7f91d0003f60 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d000bd50 con 0x7f91e0159270 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91e5386640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91e4b85640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 0x7f91d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.333+0000 7f91dffff640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91d0003f60 0x7f91d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.334+0000 7f91e5386640 1 -- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 msgr2=0x7f91d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.334+0000 7f91e5386640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e0159270 0x7f91d00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.334+0000 7f91e5386640 1 -- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91d0003f60 msgr2=0x7f91d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.334+0000 7f91e5386640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91d0003f60 0x7f91d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.334+0000 7f91e5386640 1 -- 172.21.15.94:0/4290928799 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0003520 con 0x7f91d0004d40 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.334+0000 7f91e5386640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91d00051c0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009fc0 tx=0x7f91d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83b1b8f60342654 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8039070 con 0x7f91d0004d40 2024-03-06T07:25:16.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d802cc20 con 0x7f91d0004d40 2024-03-06T07:25:16.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91df7fe640 1 -- 172.21.15.94:0/4290928799 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8009c10 con 0x7f91d0004d40 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91e6de9640 1 -- 172.21.15.94:0/4290928799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 msgr2=0x7f91d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91e6de9640 1 --2- 172.21.15.94:0/4290928799 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91d00051c0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009fc0 tx=0x7f91d0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91e5386640 1 -- 172.21.15.94:0/4290928799 reap_dead start 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91e6de9640 1 -- 172.21.15.94:0/4290928799 shutdown_connections 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91e6de9640 1 -- 172.21.15.94:0/4290928799 >> 172.21.15.94:0/4290928799 conn(0x7f91e00ac570 msgr2=0x7f91e00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91e6de9640 1 -- 172.21.15.94:0/4290928799 shutdown_connections 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.335+0000 7f91e6de9640 1 -- 172.21.15.94:0/4290928799 wait complete. 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.336+0000 7f91e6de9640 1 Processor -- start 2024-03-06T07:25:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.336+0000 7f91e6de9640 1 -- start start 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e6de9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91e01dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e6de9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e0156910 0x7f91e01574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e6de9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e01536d0 0x7f91e0153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e6de9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00c7c30 con 0x7f91d0004d40 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e6de9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00c5a50 con 0x7f91e0156910 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e6de9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00c5bd0 con 0x7f91e01536d0 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e4b85640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91e01dfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e4b85640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91e01dfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37964/0 (socket says 172.21.15.94:37964) 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e4b85640 1 -- 172.21.15.94:0/752182220 learned_addr learned my addr 172.21.15.94:0/752182220 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91dffff640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e0156910 0x7f91e01574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.337+0000 7f91e5386640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e01536d0 0x7f91e0153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e4b85640 1 -- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e01536d0 msgr2=0x7f91e0153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e4b85640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e01536d0 0x7f91e0153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e4b85640 1 -- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e0156910 msgr2=0x7f91e01574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e4b85640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e0156910 0x7f91e01574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e4b85640 1 -- 172.21.15.94:0/752182220 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e01de7f0 con 0x7f91d0004d40 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e4b85640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91e01dfd90 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f91d4007290 tx=0x7f91d4007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91dd7fa640 1 -- 172.21.15.94:0/752182220 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4000ed0 con 0x7f91d0004d40 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91dd7fa640 1 -- 172.21.15.94:0/752182220 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4018070 con 0x7f91d0004d40 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e01dead0 con 0x7f91d0004d40 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91dd7fa640 1 -- 172.21.15.94:0/752182220 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d400fbe0 con 0x7f91d0004d40 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.338+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e01df090 con 0x7f91d0004d40 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.339+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e006d0b0 con 0x7f91d0004d40 2024-03-06T07:25:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.339+0000 7f91dd7fa640 1 -- 172.21.15.94:0/752182220 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91d401d460 con 0x7f91d0004d40 2024-03-06T07:25:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.340+0000 7f91dd7fa640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91c406b6a0 0x7f91c406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.340+0000 7f91dd7fa640 1 -- 172.21.15.94:0/752182220 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f91d408a4f0 con 0x7f91d0004d40 2024-03-06T07:25:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.340+0000 7f91dffff640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91c406b6a0 0x7f91c406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.341+0000 7f91dffff640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91c406b6a0 0x7f91c406db60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f91e0155240 tx=0x7f91cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:16.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.344+0000 7f91dd7fa640 1 -- 172.21.15.94:0/752182220 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91e006d0b0 con 0x7f91d0004d40 2024-03-06T07:25:16.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.553+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91e00c81a0 con 0x7f91d0004d40 2024-03-06T07:25:16.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.553+0000 7f91dd7fa640 1 -- 172.21.15.94:0/752182220 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f91e00c81a0 con 0x7f91d0004d40 2024-03-06T07:25:16.555 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:16.555 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":160,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:02.221178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37487":{"gid":37487,"name":"b","rank":0,"incarnation":156,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/1570645588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1570645588},{"type":"v1","addr":"172.21.15.145:6836","nonce":1570645588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:16.555 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 160 2024-03-06T07:25:16.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91c406b6a0 msgr2=0x7f91c406db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91c406b6a0 0x7f91c406db60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f91e0155240 tx=0x7f91cc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:16.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 msgr2=0x7f91e01dfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:16.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91e01dfd90 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f91d4007290 tx=0x7f91d4007750 comp rx=0 tx=0).stop 2024-03-06T07:25:16.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 shutdown_connections 2024-03-06T07:25:16.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f91c406b6a0 0x7f91c406db60 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e01536d0 0x7f91e0153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e0156910 0x7f91e01574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 --2- 172.21.15.94:0/752182220 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91d0004d40 0x7f91e01dfd90 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:16.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 >> 172.21.15.94:0/752182220 conn(0x7f91e00ac570 msgr2=0x7f91e00ca050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:16.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 shutdown_connections 2024-03-06T07:25:16.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:16.555+0000 7f91e6de9640 1 -- 172.21.15.94:0/752182220 wait complete. 2024-03-06T07:25:16.970 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 160, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:02.221178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37487': {'gid': 37487, 'name': 'b', 'rank': 0, 'incarnation': 156, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/1570645588', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1570645588}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1570645588}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:16.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37487:up:stopping 2024-03-06T07:25:17.270 INFO:tasks.ceph.mds.b.smithi145.stderr: -822> 2024-03-06T07:24:57.244+0000 7f14836d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:17.294 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:25:17.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.082+0000 7fe4e8cc9640 1 Processor -- start 2024-03-06T07:25:18.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.082+0000 7fe4e8cc9640 1 -- start start 2024-03-06T07:25:18.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e8cc9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe4e40ac360 0x7fe4e40c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e8cc9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e4072db0 con 0x7fe4e40aac70 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e8cc9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e4067250 con 0x7fe4e414f790 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e8cc9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e4072ad0 con 0x7fe4e40ac360 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e1d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe4e40aac70 0x7fe4e40c1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43034/0 (socket says 172.21.15.94:43034) 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e1d9b640 1 -- 172.21.15.94:0/3004424419 learned_addr learned my addr 172.21.15.94:0/3004424419 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e2d9d640 1 --2- 172.21.15.94:0/3004424419 >> v2:172.21.15.145:3300/0 conn(0x7fe4e40ac360 0x7fe4e40c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.083+0000 7fe4e159a640 1 -- 172.21.15.94:0/3004424419 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457032766 0 0) 0x7fe4e4072db0 con 0x7fe4e40aac70 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.084+0000 7fe4e159a640 1 -- 172.21.15.94:0/3004424419 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4cc003520 con 0x7fe4e40aac70 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.084+0000 7fe4e159a640 1 -- 172.21.15.94:0/3004424419 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296144286 0 0) 0x7fe4e4067250 con 0x7fe4e414f790 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.084+0000 7fe4e159a640 1 -- 172.21.15.94:0/3004424419 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e4072db0 con 0x7fe4e414f790 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.084+0000 7fe4e2d9d640 1 -- 172.21.15.94:0/3004424419 >> v1:172.21.15.145:6789/0 conn(0x7fe4e414f790 legacy=0x7fe4e40c3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.084+0000 7fe4e2d9d640 1 -- 172.21.15.94:0/3004424419 >> v1:172.21.15.94:6789/0 conn(0x7fe4e40aac70 legacy=0x7fe4e40c1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.084+0000 7fe4e2d9d640 1 -- 172.21.15.94:0/3004424419 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4152b70 con 0x7fe4e40ac360 2024-03-06T07:25:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.084+0000 7fe4e2d9d640 1 --2- 172.21.15.94:0/3004424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c2a20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009d90 tx=0x7fe4dc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9694deefb549f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e159a640 1 -- 172.21.15.94:0/3004424419 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc039070 con 0x7fe4e40ac360 2024-03-06T07:25:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e159a640 1 -- 172.21.15.94:0/3004424419 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc02cd00 con 0x7fe4e40ac360 2024-03-06T07:25:18.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e159a640 1 -- 172.21.15.94:0/3004424419 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc034660 con 0x7fe4e40ac360 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/3004424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e40ac360 msgr2=0x7fe4e40c2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/3004424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c2a20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009d90 tx=0x7fe4dc02bf50 comp rx=0 tx=0).stop 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/3004424419 shutdown_connections 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/3004424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c2a20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/3004424419 >> 172.21.15.94:0/3004424419 conn(0x7fe4e40c70b0 msgr2=0x7fe4e40c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/3004424419 shutdown_connections 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.085+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/3004424419 wait complete. 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 Processor -- start 2024-03-06T07:25:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 -- start start 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4e40aac70 0x7fe4e415acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e414f790 0x7fe4e40bd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e4066e80 con 0x7fe4e40ac360 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e4066d00 con 0x7fe4e414f790 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e8cc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e415d900 con 0x7fe4e40aac70 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:37970/0 (socket says 172.21.15.94:37970) 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e1d9b640 1 -- 172.21.15.94:0/1707336863 learned_addr learned my addr 172.21.15.94:0/1707336863 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e259c640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4e40aac70 0x7fe4e415acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.087+0000 7fe4e2d9d640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e414f790 0x7fe4e40bd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e1d9b640 1 -- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4e40aac70 msgr2=0x7fe4e415acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e1d9b640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4e40aac70 0x7fe4e415acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e1d9b640 1 -- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e414f790 msgr2=0x7fe4e40bd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e1d9b640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e414f790 0x7fe4e40bd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e1d9b640 1 -- 172.21.15.94:0/1707336863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e415b210 con 0x7fe4e40ac360 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e1d9b640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c62f0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800c500 tx=0x7fe4d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4d2ffd640 1 -- 172.21.15.94:0/1707336863 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8017070 con 0x7fe4e40ac360 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4d2ffd640 1 -- 172.21.15.94:0/1707336863 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8005ce0 con 0x7fe4e40ac360 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4d2ffd640 1 -- 172.21.15.94:0/1707336863 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d80072f0 con 0x7fe4e40ac360 2024-03-06T07:25:18.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e415c4e0 con 0x7fe4e40ac360 2024-03-06T07:25:18.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.088+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e41eb540 con 0x7fe4e40ac360 2024-03-06T07:25:18.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.089+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e404f330 con 0x7fe4e40ac360 2024-03-06T07:25:18.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.090+0000 7fe4d2ffd640 1 -- 172.21.15.94:0/1707336863 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4d801a070 con 0x7fe4e40ac360 2024-03-06T07:25:18.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.090+0000 7fe4d2ffd640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4c406b4d0 0x7fe4c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.090+0000 7fe4d2ffd640 1 -- 172.21.15.94:0/1707336863 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe4d8089660 con 0x7fe4e40ac360 2024-03-06T07:25:18.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.090+0000 7fe4e259c640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4c406b4d0 0x7fe4c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.091+0000 7fe4e259c640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4c406b4d0 0x7fe4c406d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4007fd0 tx=0x7fe4d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.095+0000 7fe4d2ffd640 1 -- 172.21.15.94:0/1707336863 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4d8057d90 con 0x7fe4e40ac360 2024-03-06T07:25:18.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.304+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4e40c1ab0 con 0x7fe4e40ac360 2024-03-06T07:25:18.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.305+0000 7fe4d2ffd640 1 -- 172.21.15.94:0/1707336863 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+4977 (secure 0 0 0) 0x7fe4d8087e20 con 0x7fe4e40ac360 2024-03-06T07:25:18.307 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:18.307 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":162,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":37613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:16.871205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:18.307 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 162 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4c406b4d0 msgr2=0x7fe4c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4c406b4d0 0x7fe4c406d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4007fd0 tx=0x7fe4d4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4e40ac360 msgr2=0x7fe4e40c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c62f0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800c500 tx=0x7fe4d800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 shutdown_connections 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4c406b4d0 0x7fe4c406d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4e414f790 0x7fe4e40bd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4e40ac360 0x7fe4e40c62f0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 --2- 172.21.15.94:0/1707336863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4e40aac70 0x7fe4e415acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 >> 172.21.15.94:0/1707336863 conn(0x7fe4e40c70b0 msgr2=0x7fe4e40c1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 shutdown_connections 2024-03-06T07:25:18.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.307+0000 7fe4e8cc9640 1 -- 172.21.15.94:0/1707336863 wait complete. 2024-03-06T07:25:18.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 161, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:16.871205+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:18.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-03-06T07:25:18.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:18.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.824+0000 7f1dabf8a640 1 Processor -- start 2024-03-06T07:25:18.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.824+0000 7f1dabf8a640 1 -- start start 2024-03-06T07:25:18.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1dabf8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1da414f790 0x7f1da41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1dabf8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 0x7f1da4155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1dabf8a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da41563d0 con 0x7f1da40c1aa0 2024-03-06T07:25:18.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1dabf8a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da40c2680 con 0x7f1da414f790 2024-03-06T07:25:18.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1dabf8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40c28d0 con 0x7f1da4155a40 2024-03-06T07:25:18.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9d26640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1da414f790 0x7f1da41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9d26640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1da414f790 0x7f1da41529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60356/0 (socket says 172.21.15.94:60356) 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9525640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1da40c1aa0 0x7f1da40c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49742/0 (socket says 172.21.15.94:49742) 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9525640 1 -- 172.21.15.94:0/505556668 learned_addr learned my addr 172.21.15.94:0/505556668 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9d26640 1 -- 172.21.15.94:0/505556668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 msgr2=0x7f1da4155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9d26640 1 --2- 172.21.15.94:0/505556668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 0x7f1da4155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9d26640 1 -- 172.21.15.94:0/505556668 >> v1:172.21.15.94:6789/0 conn(0x7f1da40c1aa0 legacy=0x7f1da40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.825+0000 7f1da9d26640 1 -- 172.21.15.94:0/505556668 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40c2bb0 con 0x7f1da414f790 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.826+0000 7f1da9d26640 1 --2- 172.21.15.94:0/505556668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41529d0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097a0 tx=0x7f1da002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118436fbb85cf81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.826+0000 7f1da8d24640 1 -- 172.21.15.94:0/505556668 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0037070 con 0x7f1da414f790 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.826+0000 7f1da8d24640 1 -- 172.21.15.94:0/505556668 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0030710 con 0x7f1da414f790 2024-03-06T07:25:18.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.826+0000 7f1da8d24640 1 -- 172.21.15.94:0/505556668 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0030ae0 con 0x7f1da414f790 2024-03-06T07:25:18.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 -- 172.21.15.94:0/505556668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 msgr2=0x7f1da41529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/505556668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41529d0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097a0 tx=0x7f1da002b710 comp rx=0 tx=0).stop 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 -- 172.21.15.94:0/505556668 shutdown_connections 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/505556668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 0x7f1da4155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/505556668 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41529d0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 -- 172.21.15.94:0/505556668 >> 172.21.15.94:0/505556668 conn(0x7f1da40c6710 msgr2=0x7f1da40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 -- 172.21.15.94:0/505556668 shutdown_connections 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.827+0000 7f1dabf8a640 1 -- 172.21.15.94:0/505556668 wait complete. 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.828+0000 7f1dabf8a640 1 Processor -- start 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.828+0000 7f1dabf8a640 1 -- start start 2024-03-06T07:25:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da40c1aa0 0x7f1da40c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41e93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 0x7f1da41e9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40c0d60 con 0x7f1da414f790 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da4155520 con 0x7f1da40c1aa0 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40c5e30 con 0x7f1da4155a40 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41e93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41e93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60360/0 (socket says 172.21.15.94:60360) 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 -- 172.21.15.94:0/454277719 learned_addr learned my addr 172.21.15.94:0/454277719 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9d26640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da40c1aa0 0x7f1da40c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1daa527640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 0x7f1da41e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 -- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 msgr2=0x7f1da41e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 0x7f1da41e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 -- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da40c1aa0 msgr2=0x7f1da40c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da40c1aa0 0x7f1da40c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 -- 172.21.15.94:0/454277719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da41ecdc0 con 0x7f1da414f790 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1da9525640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41e93d0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400c780 tx=0x7f1d9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1d9a7fc640 1 -- 172.21.15.94:0/454277719 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94018070 con 0x7f1da414f790 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1d9a7fc640 1 -- 172.21.15.94:0/454277719 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d940072f0 con 0x7f1da414f790 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da41ed080 con 0x7f1da414f790 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1d9a7fc640 1 -- 172.21.15.94:0/454277719 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d940047e0 con 0x7f1da414f790 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.829+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da4070c00 con 0x7f1da414f790 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.830+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da4075300 con 0x7f1da414f790 2024-03-06T07:25:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.831+0000 7f1d9a7fc640 1 -- 172.21.15.94:0/454277719 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d9401b070 con 0x7f1da414f790 2024-03-06T07:25:18.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.831+0000 7f1d9a7fc640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d8806b4d0 0x7f1d8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:18.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.831+0000 7f1d9a7fc640 1 -- 172.21.15.94:0/454277719 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1d94089860 con 0x7f1da414f790 2024-03-06T07:25:18.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.831+0000 7f1da9d26640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d8806b4d0 0x7f1d8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:18.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.832+0000 7f1da9d26640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d8806b4d0 0x7f1d8806d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1da0034000 tx=0x7f1da0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:18.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:18.835+0000 7f1d9a7fc640 1 -- 172.21.15.94:0/454277719 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1da4075300 con 0x7f1da414f790 2024-03-06T07:25:19.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.042+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1da404f330 con 0x7f1da414f790 2024-03-06T07:25:19.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.042+0000 7f1d9a7fc640 1 -- 172.21.15.94:0/454277719 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+4977 (secure 0 0 0) 0x7f1d940585e0 con 0x7f1da414f790 2024-03-06T07:25:19.045 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:19.045 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":162,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":37613,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:16.871205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:19.045 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 162 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d8806b4d0 msgr2=0x7f1d8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d8806b4d0 0x7f1d8806d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1da0034000 tx=0x7f1da0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 msgr2=0x7f1da41e93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41e93d0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400c780 tx=0x7f1d9400cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 shutdown_connections 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d8806b4d0 0x7f1d8806d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da4155a40 0x7f1da41e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1da414f790 0x7f1da41e93d0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.044+0000 7f1dabf8a640 1 --2- 172.21.15.94:0/454277719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da40c1aa0 0x7f1da40c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.045+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 >> 172.21.15.94:0/454277719 conn(0x7f1da40c6710 msgr2=0x7f1da40c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.045+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 shutdown_connections 2024-03-06T07:25:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.045+0000 7f1dabf8a640 1 -- 172.21.15.94:0/454277719 wait complete. 2024-03-06T07:25:19.057 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down false 2024-03-06T07:25:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.159+0000 7f0bada50640 1 Processor -- start 2024-03-06T07:25:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.159+0000 7f0bada50640 1 -- start start 2024-03-06T07:25:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0bada50640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba806afc0 0x7f0ba806b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0bada50640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ba80c2120 0x7f0ba8158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0bada50640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0ba80c2580 0x7f0ba8159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0bada50640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba8158790 con 0x7f0ba80c2120 2024-03-06T07:25:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0bada50640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba81596a0 con 0x7f0ba80c2580 2024-03-06T07:25:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0bada50640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80c5840 con 0x7f0ba806afc0 2024-03-06T07:25:19.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ba80c2120 0x7f0ba8158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ba80c2120 0x7f0ba8158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60362/0 (socket says 172.21.15.94:60362) 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 -- 172.21.15.94:0/3098831050 learned_addr learned my addr 172.21.15.94:0/3098831050 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 -- 172.21.15.94:0/3098831050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba806afc0 msgr2=0x7f0ba806b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba77fe640 1 --2- 172.21.15.94:0/3098831050 >> v2:172.21.15.145:3300/0 conn(0x7f0ba80c2580 0x7f0ba8159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba6ffd640 1 --2- 172.21.15.94:0/3098831050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba806afc0 0x7f0ba806b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 --2- 172.21.15.94:0/3098831050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba806afc0 0x7f0ba806b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 -- 172.21.15.94:0/3098831050 >> v2:172.21.15.145:3300/0 conn(0x7f0ba80c2580 msgr2=0x7f0ba8159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 --2- 172.21.15.94:0/3098831050 >> v2:172.21.15.145:3300/0 conn(0x7f0ba80c2580 0x7f0ba8159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.160+0000 7f0ba67fc640 1 -- 172.21.15.94:0/3098831050 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80c5b20 con 0x7f0ba80c2120 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.161+0000 7f0ba67fc640 1 --2- 172.21.15.94:0/3098831050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba80c2120 0x7f0ba8158c20 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80c1e10 tx=0x7f0b9c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab378ea4ec3bffc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.161+0000 7f0ba5ffb640 1 -- 172.21.15.94:0/3098831050 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c02e070 con 0x7f0ba80c2120 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.161+0000 7f0ba5ffb640 1 -- 172.21.15.94:0/3098831050 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9c037070 con 0x7f0ba80c2120 2024-03-06T07:25:19.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.161+0000 7f0ba5ffb640 1 -- 172.21.15.94:0/3098831050 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c031a00 con 0x7f0ba80c2120 2024-03-06T07:25:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 -- 172.21.15.94:0/3098831050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba80c2120 msgr2=0x7f0ba8158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 --2- 172.21.15.94:0/3098831050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba80c2120 0x7f0ba8158c20 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80c1e10 tx=0x7f0b9c02b880 comp rx=0 tx=0).stop 2024-03-06T07:25:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 -- 172.21.15.94:0/3098831050 shutdown_connections 2024-03-06T07:25:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 --2- 172.21.15.94:0/3098831050 >> v2:172.21.15.145:3300/0 conn(0x7f0ba80c2580 0x7f0ba8159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 --2- 172.21.15.94:0/3098831050 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba80c2120 0x7f0ba8158c20 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 --2- 172.21.15.94:0/3098831050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba806afc0 0x7f0ba806b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 -- 172.21.15.94:0/3098831050 >> 172.21.15.94:0/3098831050 conn(0x7f0ba80c4220 msgr2=0x7f0ba80c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 -- 172.21.15.94:0/3098831050 shutdown_connections 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.162+0000 7f0bada50640 1 -- 172.21.15.94:0/3098831050 wait complete. 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.163+0000 7f0bada50640 1 Processor -- start 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.163+0000 7f0bada50640 1 -- start start 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba806afc0 0x7f0ba8151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ba80c2120 0x7f0ba8149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba80c2580 0x7f0ba814cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba8155220 con 0x7f0ba806afc0 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80c60e0 con 0x7f0ba80c2120 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80c2b30 con 0x7f0ba80c2580 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba806afc0 0x7f0ba8151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba806afc0 0x7f0ba8151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60364/0 (socket says 172.21.15.94:60364) 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 -- 172.21.15.94:0/1495522985 learned_addr learned my addr 172.21.15.94:0/1495522985 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba77fe640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba80c2580 0x7f0ba814cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba67fc640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ba80c2120 0x7f0ba8149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 -- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba80c2580 msgr2=0x7f0ba814cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba80c2580 0x7f0ba814cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 -- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ba80c2120 msgr2=0x7f0ba8149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ba80c2120 0x7f0ba8149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 -- 172.21.15.94:0/1495522985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba81ef890 con 0x7f0ba806afc0 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba67fc640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ba80c2120 0x7f0ba8149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0ba6ffd640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba806afc0 0x7f0ba8151b80 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c7f0 tx=0x7f0b9800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0b8ffff640 1 -- 172.21.15.94:0/1495522985 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98015070 con 0x7f0ba806afc0 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0b8ffff640 1 -- 172.21.15.94:0/1495522985 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b98004220 con 0x7f0ba806afc0 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba81ec350 con 0x7f0ba806afc0 2024-03-06T07:25:19.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0b8ffff640 1 -- 172.21.15.94:0/1495522985 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98007620 con 0x7f0ba806afc0 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.164+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba81eb870 con 0x7f0ba806afc0 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.165+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba804f330 con 0x7f0ba806afc0 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.166+0000 7f0b8ffff640 1 -- 172.21.15.94:0/1495522985 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b980077c0 con 0x7f0ba806afc0 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.166+0000 7f0b8ffff640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9406b4d0 0x7f0b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.167+0000 7f0b8ffff640 1 -- 172.21.15.94:0/1495522985 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0b9808a430 con 0x7f0ba806afc0 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.167+0000 7f0ba67fc640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9406b4d0 0x7f0b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.167+0000 7f0ba67fc640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9406b4d0 0x7f0b9406d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c036000 tx=0x7f0b9c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:19.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.171+0000 7f0b8ffff640 1 -- 172.21.15.94:0/1495522985 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b980591b0 con 0x7f0ba806afc0 2024-03-06T07:25:19.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:19.378+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"} v 0) v1 -- 0x7f0ba80c6730 con 0x7f0ba806afc0 2024-03-06T07:25:20.275 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:25:20.274+0000 7f90f7f21640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:20.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.278+0000 7f0b8ffff640 1 -- 172.21.15.94:0/1495522985 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"}]=0 cephfs marked up, max_mds = 1 v163) v1 ==== 135+0+0 (secure 0 0 0) 0x7f0b98058ab0 con 0x7f0ba806afc0 2024-03-06T07:25:20.280 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked up, max_mds = 1 2024-03-06T07:25:20.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9406b4d0 msgr2=0x7f0b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9406b4d0 0x7f0b9406d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c036000 tx=0x7f0b9c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba806afc0 msgr2=0x7f0ba8151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba806afc0 0x7f0ba8151b80 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c7f0 tx=0x7f0b9800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 shutdown_connections 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0b9406b4d0 0x7f0b9406d990 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ba80c2580 0x7f0ba814cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ba80c2120 0x7f0ba8149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 --2- 172.21.15.94:0/1495522985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ba806afc0 0x7f0ba8151b80 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.280+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 >> 172.21.15.94:0/1495522985 conn(0x7f0ba80c4220 msgr2=0x7f0ba8150530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.281+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 shutdown_connections 2024-03-06T07:25:20.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.281+0000 7f0bada50640 1 -- 172.21.15.94:0/1495522985 wait complete. 2024-03-06T07:25:20.294 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:20.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.395+0000 7fadce878640 1 Processor -- start 2024-03-06T07:25:20.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadce878640 1 -- start start 2024-03-06T07:25:20.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadce878640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fadc814f360 0x7fadc8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadce878640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc8158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadce878640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc80c73d0 con 0x7fadc80c88c0 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadce878640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc80c8dd0 con 0x7fadc814f360 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadce878640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80be2e0 con 0x7fadc8158b60 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadc7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fadc814f360 0x7fadc8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadc77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fadc80c88c0 0x7fadc80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49746/0 (socket says 172.21.15.94:49746) 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadc77fe640 1 -- 172.21.15.94:0/172834492 learned_addr learned my addr 172.21.15.94:0/172834492 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.396+0000 7fadc7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fadc814f360 0x7fadc8155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60374/0 (socket says 172.21.15.94:60374) 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.397+0000 7fadcce15640 1 --2- 172.21.15.94:0/172834492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc8158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.397+0000 7fadc7fff640 1 -- 172.21.15.94:0/172834492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 msgr2=0x7fadc8158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.397+0000 7fadc7fff640 1 --2- 172.21.15.94:0/172834492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc8158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.397+0000 7fadc7fff640 1 -- 172.21.15.94:0/172834492 >> v1:172.21.15.94:6789/0 conn(0x7fadc80c88c0 legacy=0x7fadc80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.397+0000 7fadc7fff640 1 -- 172.21.15.94:0/172834492 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc80c7670 con 0x7fadc814f360 2024-03-06T07:25:20.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.397+0000 7fadc6ffd640 1 -- 172.21.15.94:0/172834492 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219168787 0 0) 0x7fadc80c73d0 con 0x7fadc80c88c0 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.397+0000 7fadc7fff640 1 --2- 172.21.15.94:0/172834492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc8155c10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fadb80097a0 tx=0x7fadb802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9688c91982ae886 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadc6ffd640 1 -- 172.21.15.94:0/172834492 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb80315a0 con 0x7fadc814f360 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadc6ffd640 1 -- 172.21.15.94:0/172834492 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb8037070 con 0x7fadc814f360 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadc6ffd640 1 -- 172.21.15.94:0/172834492 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb8031960 con 0x7fadc814f360 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 -- 172.21.15.94:0/172834492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 msgr2=0x7fadc8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 --2- 172.21.15.94:0/172834492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc8155c10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fadb80097a0 tx=0x7fadb802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 -- 172.21.15.94:0/172834492 shutdown_connections 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 --2- 172.21.15.94:0/172834492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc8158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 --2- 172.21.15.94:0/172834492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc8155c10 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 -- 172.21.15.94:0/172834492 >> 172.21.15.94:0/172834492 conn(0x7fadc80ac570 msgr2=0x7fadc80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 -- 172.21.15.94:0/172834492 shutdown_connections 2024-03-06T07:25:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.398+0000 7fadce878640 1 -- 172.21.15.94:0/172834492 wait complete. 2024-03-06T07:25:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.399+0000 7fadce878640 1 Processor -- start 2024-03-06T07:25:20.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.399+0000 7fadce878640 1 -- start start 2024-03-06T07:25:20.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadce878640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc80c88c0 0x7fadc8157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadce878640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc80bfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadce878640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc80c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadce878640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80c5a50 con 0x7fadc814f360 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadce878640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80c7c30 con 0x7fadc80c88c0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadce878640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc80c5bd0 con 0x7fadc8158b60 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc80bfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc80bfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60388/0 (socket says 172.21.15.94:60388) 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 -- 172.21.15.94:0/3227376404 learned_addr learned my addr 172.21.15.94:0/3227376404 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc80c88c0 0x7fadc8157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadcce15640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc80c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 -- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 msgr2=0x7fadc80c0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc80c0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 -- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc80c88c0 msgr2=0x7fadc8157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc80c88c0 0x7fadc8157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.400+0000 7fadc77fe640 1 -- 172.21.15.94:0/3227376404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc81eea30 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.401+0000 7fadc77fe640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc80bfae0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c410 tx=0x7fadbc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.401+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3227376404 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc0074e0 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.401+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3227376404 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc0047e0 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.401+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3227376404 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc004bd0 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.401+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc8151a20 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.401+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc8158480 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.402+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc8151f40 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.402+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3227376404 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadbc011070 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.403+0000 7fadc4ff9640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fadb406b4d0 0x7fadb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.403+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3227376404 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fadbc089f50 con 0x7fadc814f360 2024-03-06T07:25:20.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.403+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fadb406b4d0 0x7fadb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.403+0000 7fadc7fff640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fadb406b4d0 0x7fadb406d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fadb8036000 tx=0x7fadb803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.407+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3227376404 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadc8151f40 con 0x7fadc814f360 2024-03-06T07:25:20.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.614+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadc804f330 con 0x7fadc814f360 2024-03-06T07:25:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.615+0000 7fadc4ff9640 1 -- 172.21.15.94:0/3227376404 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fadbc00e280 con 0x7fadc814f360 2024-03-06T07:25:20.617 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:20.617 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:20.281203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:starting","state_seq":1,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:20.617 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 164 2024-03-06T07:25:20.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fadb406b4d0 msgr2=0x7fadb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fadb406b4d0 0x7fadb406d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fadb8036000 tx=0x7fadb803d040 comp rx=0 tx=0).stop 2024-03-06T07:25:20.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 msgr2=0x7fadc80bfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc80bfae0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c410 tx=0x7fadbc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 shutdown_connections 2024-03-06T07:25:20.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fadb406b4d0 0x7fadb406d990 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fadc8158b60 0x7fadc80c0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fadc814f360 0x7fadc80bfae0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 --2- 172.21.15.94:0/3227376404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fadc80c88c0 0x7fadc8157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 >> 172.21.15.94:0/3227376404 conn(0x7fadc80ac570 msgr2=0x7fadc80ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 shutdown_connections 2024-03-06T07:25:20.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.617+0000 7fadce878640 1 -- 172.21.15.94:0/3227376404 wait complete. 2024-03-06T07:25:20.630 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:20.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.728+0000 7fad3ffff640 1 Processor -- start 2024-03-06T07:25:20.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.728+0000 7fad3ffff640 1 -- start start 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad400b0000 0x7fad400c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c56a0 0x7fad40155690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3ffff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad401594b0 con 0x7fad400c51f0 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3ffff640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad401586f0 con 0x7fad400b0000 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3ffff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad401589d0 con 0x7fad400c56a0 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad400b0000 0x7fad400c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad400b0000 0x7fad400c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60400/0 (socket says 172.21.15.94:60400) 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad367fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fad400c51f0 0x7fad40154f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49760/0 (socket says 172.21.15.94:49760) 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3effd640 1 -- 172.21.15.94:0/1518073914 learned_addr learned my addr 172.21.15.94:0/1518073914 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3effd640 1 -- 172.21.15.94:0/1518073914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c56a0 msgr2=0x7fad40155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3effd640 1 --2- 172.21.15.94:0/1518073914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c56a0 0x7fad40155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3effd640 1 -- 172.21.15.94:0/1518073914 >> v1:172.21.15.94:6789/0 conn(0x7fad400c51f0 legacy=0x7fad40154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3effd640 1 -- 172.21.15.94:0/1518073914 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad40158cb0 con 0x7fad400b0000 2024-03-06T07:25:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.729+0000 7fad3e7fc640 1 -- 172.21.15.94:0/1518073914 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259631169 0 0) 0x7fad401594b0 con 0x7fad400c51f0 2024-03-06T07:25:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.730+0000 7fad3effd640 1 --2- 172.21.15.94:0/1518073914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400b0000 0x7fad400c8770 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fad30009870 tx=0x7fad3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a974d343c45889 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.730+0000 7fad3e7fc640 1 -- 172.21.15.94:0/1518073914 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30035020 con 0x7fad400b0000 2024-03-06T07:25:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.730+0000 7fad3e7fc640 1 -- 172.21.15.94:0/1518073914 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad30031640 con 0x7fad400b0000 2024-03-06T07:25:20.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.730+0000 7fad3e7fc640 1 -- 172.21.15.94:0/1518073914 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3002d040 con 0x7fad400b0000 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.730+0000 7fad3ffff640 1 -- 172.21.15.94:0/1518073914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400b0000 msgr2=0x7fad400c8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.731+0000 7fad3ffff640 1 --2- 172.21.15.94:0/1518073914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400b0000 0x7fad400c8770 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fad30009870 tx=0x7fad3002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.731+0000 7fad3ffff640 1 -- 172.21.15.94:0/1518073914 shutdown_connections 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.731+0000 7fad3ffff640 1 --2- 172.21.15.94:0/1518073914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c56a0 0x7fad40155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.731+0000 7fad3ffff640 1 --2- 172.21.15.94:0/1518073914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400b0000 0x7fad400c8770 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.731+0000 7fad3ffff640 1 -- 172.21.15.94:0/1518073914 >> 172.21.15.94:0/1518073914 conn(0x7fad400728d0 msgr2=0x7fad40072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.731+0000 7fad3ffff640 1 -- 172.21.15.94:0/1518073914 shutdown_connections 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.731+0000 7fad3ffff640 1 -- 172.21.15.94:0/1518073914 wait complete. 2024-03-06T07:25:20.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.732+0000 7fad3ffff640 1 Processor -- start 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.732+0000 7fad3ffff640 1 -- start start 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad400b0000 0x7fad401eee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c51f0 0x7fad4014f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400c56a0 0x7fad401eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4006cfa0 con 0x7fad400c56a0 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad400be070 con 0x7fad400b0000 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4006d120 con 0x7fad400c51f0 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400c56a0 0x7fad401eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400c56a0 0x7fad401eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60410/0 (socket says 172.21.15.94:60410) 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 -- 172.21.15.94:0/843176209 learned_addr learned my addr 172.21.15.94:0/843176209 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 -- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c51f0 msgr2=0x7fad4014f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c51f0 0x7fad4014f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 -- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad400b0000 msgr2=0x7fad401eee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad400b0000 0x7fad401eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 -- 172.21.15.94:0/843176209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad401530d0 con 0x7fad400c56a0 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3f7fe640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400c56a0 0x7fad401eada0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fad3800b9e0 tx=0x7fad3800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3cff9640 1 -- 172.21.15.94:0/843176209 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38019070 con 0x7fad400c56a0 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3cff9640 1 -- 172.21.15.94:0/843176209 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3800cac0 con 0x7fad400c56a0 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad40152df0 con 0x7fad400c56a0 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3cff9640 1 -- 172.21.15.94:0/843176209 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38014440 con 0x7fad400c56a0 2024-03-06T07:25:20.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.733+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad401ee670 con 0x7fad400c56a0 2024-03-06T07:25:20.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.734+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad40156770 con 0x7fad400c56a0 2024-03-06T07:25:20.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.735+0000 7fad3cff9640 1 -- 172.21.15.94:0/843176209 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad3801c070 con 0x7fad400c56a0 2024-03-06T07:25:20.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.735+0000 7fad3cff9640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2006b4d0 0x7fad2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.735+0000 7fad3cff9640 1 -- 172.21.15.94:0/843176209 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad38089fa0 con 0x7fad400c56a0 2024-03-06T07:25:20.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.735+0000 7fad3effd640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2006b4d0 0x7fad2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.736+0000 7fad3effd640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2006b4d0 0x7fad2006d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fad30036000 tx=0x7fad30030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.740+0000 7fad3cff9640 1 -- 172.21.15.94:0/843176209 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad40156770 con 0x7fad400c56a0 2024-03-06T07:25:20.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.947+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad40069430 con 0x7fad400c56a0 2024-03-06T07:25:20.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.948+0000 7fad3cff9640 1 -- 172.21.15.94:0/843176209 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fad38058d20 con 0x7fad400c56a0 2024-03-06T07:25:20.950 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:20.950 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":164,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:20.281203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:starting","state_seq":1,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":13}]} 2024-03-06T07:25:20.950 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 164 2024-03-06T07:25:20.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2006b4d0 msgr2=0x7fad2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2006b4d0 0x7fad2006d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fad30036000 tx=0x7fad30030000 comp rx=0 tx=0).stop 2024-03-06T07:25:20.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400c56a0 msgr2=0x7fad401eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400c56a0 0x7fad401eada0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fad3800b9e0 tx=0x7fad3800bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 shutdown_connections 2024-03-06T07:25:20.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2006b4d0 0x7fad2006d990 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400c56a0 0x7fad401eada0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad400c51f0 0x7fad4014f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 --2- 172.21.15.94:0/843176209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad400b0000 0x7fad401eee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 >> 172.21.15.94:0/843176209 conn(0x7fad400728d0 msgr2=0x7fad40151fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 shutdown_connections 2024-03-06T07:25:20.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:20.950+0000 7fad3ffff640 1 -- 172.21.15.94:0/843176209 wait complete. 2024-03-06T07:25:21.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 164, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:20.281203+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 37613}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37613': {'gid': 37613, 'name': 'b', 'rank': 0, 'incarnation': 164, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.145:6836/1183474661', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1183474661}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1183474661}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:21.365 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37613:up:starting 2024-03-06T07:25:22.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:22.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.477+0000 7f347fc15640 1 Processor -- start 2024-03-06T07:25:22.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.477+0000 7f347fc15640 1 -- start start 2024-03-06T07:25:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347fc15640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3478152960 0x7f347806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347fc15640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34780ac260 0x7f34781559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347fc15640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3478152d40 con 0x7f34781592d0 2024-03-06T07:25:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347fc15640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3478066e30 con 0x7f3478152960 2024-03-06T07:25:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347fc15640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c1dc0 con 0x7f34780ac260 2024-03-06T07:25:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347e1b2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34780ac260 0x7f34781559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347d9b1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3478152960 0x7f347806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347d9b1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3478152960 0x7f347806dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:44346/0 (socket says 172.21.15.94:44346) 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347d9b1640 1 -- 172.21.15.94:0/1486917437 learned_addr learned my addr 172.21.15.94:0/1486917437 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347c9af640 1 -- 172.21.15.94:0/1486917437 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357809361 0 0) 0x7f3478152d40 con 0x7f34781592d0 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347e1b2640 1 -- 172.21.15.94:0/1486917437 >> v2:172.21.15.145:3300/0 conn(0x7f3478152960 msgr2=0x7f347806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347e1b2640 1 --2- 172.21.15.94:0/1486917437 >> v2:172.21.15.145:3300/0 conn(0x7f3478152960 0x7f347806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347e1b2640 1 -- 172.21.15.94:0/1486917437 >> v1:172.21.15.145:6789/0 conn(0x7f34781592d0 legacy=0x7f34781552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347e1b2640 1 -- 172.21.15.94:0/1486917437 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34780c1b50 con 0x7f34780ac260 2024-03-06T07:25:22.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.478+0000 7f347d9b1640 1 --2- 172.21.15.94:0/1486917437 >> v2:172.21.15.145:3300/0 conn(0x7f3478152960 0x7f347806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.479+0000 7f347e1b2640 1 --2- 172.21.15.94:0/1486917437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34780ac260 0x7f34781559e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3470009d90 tx=0x7f347002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a54ad53692bf3bfe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.479+0000 7f347c9af640 1 -- 172.21.15.94:0/1486917437 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470039070 con 0x7f34780ac260 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.479+0000 7f347c9af640 1 -- 172.21.15.94:0/1486917437 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347002cc50 con 0x7f34780ac260 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.479+0000 7f347c9af640 1 -- 172.21.15.94:0/1486917437 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470034440 con 0x7f34780ac260 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 -- 172.21.15.94:0/1486917437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34780ac260 msgr2=0x7f34781559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 --2- 172.21.15.94:0/1486917437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34780ac260 0x7f34781559e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3470009d90 tx=0x7f347002bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 -- 172.21.15.94:0/1486917437 shutdown_connections 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 --2- 172.21.15.94:0/1486917437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34780ac260 0x7f34781559e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 --2- 172.21.15.94:0/1486917437 >> v2:172.21.15.145:3300/0 conn(0x7f3478152960 0x7f347806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 -- 172.21.15.94:0/1486917437 >> 172.21.15.94:0/1486917437 conn(0x7f34780c70b0 msgr2=0x7f34780c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 -- 172.21.15.94:0/1486917437 shutdown_connections 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.480+0000 7f347fc15640 1 -- 172.21.15.94:0/1486917437 wait complete. 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.481+0000 7f347fc15640 1 Processor -- start 2024-03-06T07:25:22.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.481+0000 7f347fc15640 1 -- start start 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347fc15640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34780ac260 0x7f347814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347fc15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3478152960 0x7f347814b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347fc15640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f347814bfa0 0x7f34781ee660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347fc15640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c20a0 con 0x7f3478152960 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347fc15640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3478159760 con 0x7f34780ac260 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347fc15640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c2220 con 0x7f347814bfa0 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3478152960 0x7f347814b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3478152960 0x7f347814b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60412/0 (socket says 172.21.15.94:60412) 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 -- 172.21.15.94:0/1182377396 learned_addr learned my addr 172.21.15.94:0/1182377396 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347e1b2640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f347814bfa0 0x7f34781ee660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d9b1640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34780ac260 0x7f347814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 -- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f347814bfa0 msgr2=0x7f34781ee660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f347814bfa0 0x7f34781ee660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 -- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34780ac260 msgr2=0x7f347814ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34780ac260 0x7f347814ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 -- 172.21.15.94:0/1182377396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34781ef290 con 0x7f3478152960 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347d1b0640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3478152960 0x7f347814b340 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f346800c4e0 tx=0x7f346800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f346e7fc640 1 -- 172.21.15.94:0/1182377396 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468017070 con 0x7f3478152960 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34781584c0 con 0x7f3478152960 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f346e7fc640 1 -- 172.21.15.94:0/1182377396 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34680035f0 con 0x7f3478152960 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.482+0000 7f346e7fc640 1 -- 172.21.15.94:0/1182377396 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680039e0 con 0x7f3478152960 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.483+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3478158a80 con 0x7f3478152960 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.483+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3478069460 con 0x7f3478152960 2024-03-06T07:25:22.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.484+0000 7f346e7fc640 1 -- 172.21.15.94:0/1182377396 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f346801a070 con 0x7f3478152960 2024-03-06T07:25:22.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.484+0000 7f346e7fc640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.484+0000 7f346e7fc640 1 -- 172.21.15.94:0/1182377396 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3468089660 con 0x7f3478152960 2024-03-06T07:25:22.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.485+0000 7f347d9b1640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.485+0000 7f347d9b1640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f345c06b4d0 0x7f345c06d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3474006ca0 tx=0x7f3474009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.489+0000 7f346e7fc640 1 -- 172.21.15.94:0/1182377396 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3478069460 con 0x7f3478152960 2024-03-06T07:25:22.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.695+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f347804f330 con 0x7f3478152960 2024-03-06T07:25:22.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.695+0000 7f346e7fc640 1 -- 172.21.15.94:0/1182377396 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f3468087e20 con 0x7f3478152960 2024-03-06T07:25:22.698 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:22.698 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:21.282376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:active","state_seq":2,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37613,"qdb_cluster":[37613]},"id":13}]} 2024-03-06T07:25:22.698 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 165 2024-03-06T07:25:22.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f345c06b4d0 msgr2=0x7f345c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f345c06b4d0 0x7f345c06d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3474006ca0 tx=0x7f3474009240 comp rx=0 tx=0).stop 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3478152960 msgr2=0x7f347814b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3478152960 0x7f347814b340 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f346800c4e0 tx=0x7f346800c9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 shutdown_connections 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f345c06b4d0 0x7f345c06d990 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f347814bfa0 0x7f34781ee660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3478152960 0x7f347814b340 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 --2- 172.21.15.94:0/1182377396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34780ac260 0x7f347814ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.697+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 >> 172.21.15.94:0/1182377396 conn(0x7f34780c70b0 msgr2=0x7f347806bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.698+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 shutdown_connections 2024-03-06T07:25:22.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:22.698+0000 7f347fc15640 1 -- 172.21.15.94:0/1182377396 wait complete. 2024-03-06T07:25:23.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:56.327121+0000', 'modified': '2024-03-06T07:25:21.282376+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 37613}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37613': {'gid': 37613, 'name': 'b', 'rank': 0, 'incarnation': 164, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.145:6836/1183474661', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 1183474661}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 1183474661}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 37613, 'qdb_cluster': [37613]} 2024-03-06T07:25:23.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:23.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:25:23.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.214+0000 7f43e7a9a640 1 Processor -- start 2024-03-06T07:25:23.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.215+0000 7f43e7a9a640 1 -- start start 2024-03-06T07:25:23.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.215+0000 7f43e7a9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e00c6ab0 0x7f43e01529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.215+0000 7f43e7a9a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f43e01561a0 0x7f43e00a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.215+0000 7f43e7a9a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e00a9990 con 0x7f43e00aa430 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.215+0000 7f43e7a9a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e0155d10 con 0x7f43e01561a0 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.215+0000 7f43e7a9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01556c0 con 0x7f43e00c6ab0 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e5035640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f43e01561a0 0x7f43e00a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e6037640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f43e00aa430 0x7f43e014f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:54748/0 (socket says 172.21.15.94:54748) 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e6037640 1 -- 172.21.15.94:0/462215762 learned_addr learned my addr 172.21.15.94:0/462215762 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e5035640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f43e01561a0 0x7f43e00a9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:44374/0 (socket says 172.21.15.94:44374) 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e5836640 1 --2- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e00c6ab0 0x7f43e01529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e5035640 1 -- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e00c6ab0 msgr2=0x7f43e01529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e5035640 1 --2- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e00c6ab0 0x7f43e01529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e5035640 1 -- 172.21.15.94:0/462215762 >> v1:172.21.15.145:6789/0 conn(0x7f43e00aa430 legacy=0x7f43e014f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.216+0000 7f43e5035640 1 -- 172.21.15.94:0/462215762 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e01559a0 con 0x7f43e01561a0 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e5035640 1 --2- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e01561a0 0x7f43e00a9ef0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0097a0 tx=0x7f43dc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de86dcc4d5f814b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e4834640 1 -- 172.21.15.94:0/462215762 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc038070 con 0x7f43e01561a0 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e4834640 1 -- 172.21.15.94:0/462215762 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc031700 con 0x7f43e01561a0 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e4834640 1 -- 172.21.15.94:0/462215762 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc031ad0 con 0x7f43e01561a0 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e7a9a640 1 -- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e01561a0 msgr2=0x7f43e00a9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e01561a0 0x7f43e00a9ef0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0097a0 tx=0x7f43dc02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e7a9a640 1 -- 172.21.15.94:0/462215762 shutdown_connections 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e01561a0 0x7f43e00a9ef0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/462215762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e00c6ab0 0x7f43e01529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.217+0000 7f43e7a9a640 1 -- 172.21.15.94:0/462215762 >> 172.21.15.94:0/462215762 conn(0x7f43e006b1a0 msgr2=0x7f43e006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.218+0000 7f43e7a9a640 1 -- 172.21.15.94:0/462215762 shutdown_connections 2024-03-06T07:25:23.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.218+0000 7f43e7a9a640 1 -- 172.21.15.94:0/462215762 wait complete. 2024-03-06T07:25:23.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 Processor -- start 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 -- start start 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43e00aa430 0x7f43e0072350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e00c6ab0 0x7f43e0140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e01561a0 0x7f43e01410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e014f820 con 0x7f43e00aa430 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00c2a90 con 0x7f43e00c6ab0 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e7a9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01501f0 con 0x7f43e01561a0 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e5836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43e00aa430 0x7f43e0072350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.219+0000 7f43e5836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43e00aa430 0x7f43e0072350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60420/0 (socket says 172.21.15.94:60420) 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5836640 1 -- 172.21.15.94:0/3755221104 learned_addr learned my addr 172.21.15.94:0/3755221104 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5035640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e00c6ab0 0x7f43e0140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e6037640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e01561a0 0x7f43e01410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5836640 1 -- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e01561a0 msgr2=0x7f43e01410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5836640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e01561a0 0x7f43e01410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5836640 1 -- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e00c6ab0 msgr2=0x7f43e0140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5836640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e00c6ab0 0x7f43e0140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5836640 1 -- 172.21.15.94:0/3755221104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e01512b0 con 0x7f43e00aa430 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e5836640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43e00aa430 0x7f43e0072350 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f43d000e870 tx=0x7f43d000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43d67fc640 1 -- 172.21.15.94:0/3755221104 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0017070 con 0x7f43e00aa430 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43d67fc640 1 -- 172.21.15.94:0/3755221104 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d000c8e0 con 0x7f43e00aa430 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43d67fc640 1 -- 172.21.15.94:0/3755221104 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d000ccd0 con 0x7f43e00aa430 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e0151590 con 0x7f43e00aa430 2024-03-06T07:25:23.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.220+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e0151b50 con 0x7f43e00aa430 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.222+0000 7f43d67fc640 1 -- 172.21.15.94:0/3755221104 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43d001a070 con 0x7f43e00aa430 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.222+0000 7f43d67fc640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43c406b4d0 0x7f43c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.222+0000 7f43d67fc640 1 -- 172.21.15.94:0/3755221104 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43d008a430 con 0x7f43e00aa430 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.222+0000 7f43e5035640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43c406b4d0 0x7f43c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.223+0000 7f43e5035640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43c406b4d0 0x7f43c406d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f43dc035000 tx=0x7f43dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.223+0000 7f43ea322740 1 Processor -- start 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.223+0000 7f43ea322740 1 -- start start 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325395f50 0x55f325393a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f325393f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43ea322740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253944b0 0x55f325390680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43ea322740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3252fa6b0 con 0x55f3253944b0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3252a3a60 con 0x55f325396330 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3252f6750 con 0x55f325395f50 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253944b0 0x55f325390680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e5836640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325395f50 0x55f325393a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253944b0 0x55f325390680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60426/0 (socket says 172.21.15.94:60426) 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 -- 172.21.15.94:0/3383535320 learned_addr learned my addr 172.21.15.94:0/3383535320 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e5836640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325395f50 0x55f325393a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:41318/0 (socket says 172.21.15.94:41318) 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e5035640 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f325393f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 -- 172.21.15.94:0/3383535320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325395f50 msgr2=0x55f325393a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325395f50 0x55f325393a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 -- 172.21.15.94:0/3383535320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 msgr2=0x55f325393f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f325393f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 -- 172.21.15.94:0/3383535320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3252c1f30 con 0x55f3253944b0 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43e6037640 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253944b0 0x55f325390680 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f43d800f1d0 tx=0x7f43d8001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b836c362c5d2edc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43c37fe640 1 -- 172.21.15.94:0/3383535320 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d800fe50 con 0x55f3253944b0 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43c37fe640 1 -- 172.21.15.94:0/3383535320 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8018070 con 0x55f3253944b0 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.224+0000 7f43c37fe640 1 -- 172.21.15.94:0/3383535320 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80074e0 con 0x55f3253944b0 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 -- 172.21.15.94:0/3383535320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253944b0 msgr2=0x55f325390680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253944b0 0x55f325390680 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f43d800f1d0 tx=0x7f43d8001950 comp rx=0 tx=0).stop 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 -- 172.21.15.94:0/3383535320 shutdown_connections 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253944b0 0x55f325390680 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f325393f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 --2- 172.21.15.94:0/3383535320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325395f50 0x55f325393a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 -- 172.21.15.94:0/3383535320 >> 172.21.15.94:0/3383535320 conn(0x55f3253927d0 msgr2=0x55f3253b73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 -- 172.21.15.94:0/3383535320 shutdown_connections 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.225+0000 7f43ea322740 1 -- 172.21.15.94:0/3383535320 wait complete. 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.226+0000 7f43ea322740 1 -- start start 2024-03-06T07:25:23.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.226+0000 7f43ea322740 1 Processor -- start 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325395f50 0x55f3253b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325396330 0x55f3253b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253b9960 0x55f3253ba5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3253b8230 con 0x55f3253b9960 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3252b6ea0 con 0x55f325395f50 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3253b7c70 con 0x55f325396330 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253b9960 0x55f3253ba5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 10 client.0 _ll_register_callbacks cb 0x55f3252b6870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253b9960 0x55f3253ba5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60442/0 (socket says 172.21.15.94:60442) 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 -- 172.21.15.94:0/4277424516 learned_addr learned my addr 172.21.15.94:0/4277424516 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e5836640 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325395f50 0x55f3253b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e5035640 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325396330 0x55f3253b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 -- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325396330 msgr2=0x55f3253b8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325396330 0x55f3253b8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 -- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325395f50 msgr2=0x55f3253b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325395f50 0x55f3253b23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 -- 172.21.15.94:0/4277424516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3252b7270 con 0x55f3253b9960 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43e6037640 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253b9960 0x55f3253ba5a0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f43d80012e0 tx=0x7f43d8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5565ead456a1fd1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43c0ff9640 1 -- 172.21.15.94:0/4277424516 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d801e420 con 0x55f3253b9960 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43c0ff9640 1 -- 172.21.15.94:0/4277424516 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43d8015870 con 0x55f3253b9960 2024-03-06T07:25:23.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43d8018070 con 0x55f3253b9960 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43c0ff9640 1 -- 172.21.15.94:0/4277424516 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43d802e050 con 0x55f3253b9960 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 10 client.9708 fetch_fsmap learned FSMap version 165 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.227+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x55f32500f2d0 con 0x55f3253b9960 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43c0ff9640 1 -- 172.21.15.94:0/4277424516 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8018070 con 0x55f3253b9960 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43c0ff9640 1 -- 172.21.15.94:0/4277424516 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 165) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f43d8031020 con 0x55f3253b9960 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43ea322740 10 client.9708 fetch_fsmap finished waiting for FSMap version 165 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43ea322740 10 client.9708 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43ea322740 20 client.9708 populate_metadata read hostname 'smithi094' 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43ea322740 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253e6a30 0x55f3253f1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43ea322740 4 client.9708 mds_command: new command op to 37613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 --> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f325307bf0 con 0x55f3253e6a30 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43e5836640 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253e6a30 0x55f3253f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43c0ff9640 10 client.9708 ms_handle_connect on v2:172.21.15.145:6835/1183474661 2024-03-06T07:25:23.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.228+0000 7f43e5836640 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253e6a30 0x55f3253f1520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45fd225f43c3dad9 server_cookie=c9eac9a50311184d in_seq=0 out_seq=0 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43c0ff9640 1 -- 172.21.15.94:0/4277424516 <== mds.0 v2:172.21.15.145:6835/1183474661 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f325307bf0 con 0x55f3253e6a30 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43c0ff9640 10 client.9708 handle_command_reply: tid=0 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 client.9708 shutdown 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253b9960 msgr2=0x55f3253ba5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253b9960 0x55f3253ba5a0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f43d80012e0 tx=0x7f43d8001250 comp rx=0 tx=0).stop 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 shutdown_connections 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253e6a30 0x55f3253f1520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f3253b9960 0x55f3253ba5a0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f325396330 0x55f3253b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43c0ff9640 0 client.9708 ms_handle_reset on v2:172.21.15.145:6835/1183474661 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 --2- 172.21.15.94:0/4277424516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325395f50 0x55f3253b23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 >> 172.21.15.94:0/4277424516 conn(0x55f3253944b0 msgr2=0x55f3253948e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 shutdown_connections 2024-03-06T07:25:23.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 1 -- 172.21.15.94:0/4277424516 wait complete. 2024-03-06T07:25:23.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.229+0000 7f43ea322740 20 client.9708 trim_cache size 0 max 16384 2024-03-06T07:25:23.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 Processor -- start 2024-03-06T07:25:23.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 -- start start 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f3253bc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253f6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f3253b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01f4c50 con 0x55f325395f50 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01f40b0 con 0x55f325396330 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e01f4650 con 0x55f3253b9960 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43e5836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f3253bc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43e5836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f3253bc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60452/0 (socket says 172.21.15.94:60452) 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43e5836640 1 -- 172.21.15.94:0/666360085 learned_addr learned my addr 172.21.15.94:0/666360085 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.246+0000 7f43e6037640 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f3253b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5035640 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5836640 1 -- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 msgr2=0x55f3253b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5836640 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f3253b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5836640 1 -- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 msgr2=0x55f3253f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5836640 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253f6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5836640 1 -- 172.21.15.94:0/666360085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3250d96b0 con 0x55f325395f50 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5035640 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43e5836640 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f3253bc9e0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f43d0054fc0 tx=0x7f43d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda5d4a6edc4611e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43c2ffd640 1 -- 172.21.15.94:0/666360085 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0021040 con 0x55f325395f50 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43c2ffd640 1 -- 172.21.15.94:0/666360085 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d00611c0 con 0x55f325395f50 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43c2ffd640 1 -- 172.21.15.94:0/666360085 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0056e60 con 0x55f325395f50 2024-03-06T07:25:23.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 -- 172.21.15.94:0/666360085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 msgr2=0x55f3253bc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f3253bc9e0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f43d0054fc0 tx=0x7f43d0000bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 -- 172.21.15.94:0/666360085 shutdown_connections 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f3253b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 --2- 172.21.15.94:0/666360085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f3253bc9e0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 -- 172.21.15.94:0/666360085 >> 172.21.15.94:0/666360085 conn(0x55f3253944b0 msgr2=0x55f3253bb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 -- 172.21.15.94:0/666360085 shutdown_connections 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.247+0000 7f43ea322740 1 -- 172.21.15.94:0/666360085 wait complete. 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.248+0000 7f43ea322740 1 -- start start 2024-03-06T07:25:23.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.248+0000 7f43ea322740 1 Processor -- start 2024-03-06T07:25:23.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43ea322740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f325393ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43ea322740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f32543fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43ea322740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3253b7570 con 0x55f325395f50 2024-03-06T07:25:23.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3253bbbc0 con 0x55f325396330 2024-03-06T07:25:23.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43ea322740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3253e5e80 con 0x55f3253b9960 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43ea322740 10 client.0 _ll_register_callbacks cb 0x7f43d8002110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f325393ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f325393ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60460/0 (socket says 172.21.15.94:60460) 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 -- 172.21.15.94:0/2032181213 learned_addr learned my addr 172.21.15.94:0/2032181213 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e6037640 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f32543fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5035640 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 -- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 msgr2=0x55f32543fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f32543fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 -- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 msgr2=0x55f3253b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 -- 172.21.15.94:0/2032181213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3253b88e0 con 0x55f325395f50 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43e5836640 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f325393ef0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f43d0058860 tx=0x7f43d0088730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38fd75a22caa71a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.249+0000 7f43c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:25:23.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2032181213 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0074070 con 0x55f325395f50 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2032181213 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43d00551d0 con 0x55f325395f50 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43d00611c0 con 0x55f325395f50 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2032181213 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43d0021070 con 0x55f325395f50 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 10 client.9714 fetch_fsmap learned FSMap version 165 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x55f3253d48a0 con 0x55f325395f50 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2032181213 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d00611c0 con 0x55f325395f50 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2032181213 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 165) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f43d008a0b0 con 0x55f325395f50 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 10 client.9714 fetch_fsmap finished waiting for FSMap version 165 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 10 client.9714 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 20 client.9714 populate_metadata read hostname 'smithi094' 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253def40 0x55f3253df320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 4 client.9714 mds_command: new command op to 37613 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 --> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f325301f40 con 0x55f3253def40 2024-03-06T07:25:23.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43e5035640 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253def40 0x55f3253df320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43c0ff9640 10 client.9714 ms_handle_connect on v2:172.21.15.145:6835/1183474661 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.250+0000 7f43e5035640 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253def40 0x55f3253df320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5db5b2ca9104f667 server_cookie=689afead8d221a32 in_seq=0 out_seq=0 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43c0ff9640 1 -- 172.21.15.94:0/2032181213 <== mds.0 v2:172.21.15.145:6835/1183474661 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55f325301f40 con 0x55f3253def40 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43c0ff9640 10 client.9714 handle_command_reply: tid=0 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 client.9714 shutdown 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 msgr2=0x55f325393ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f325393ef0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f43d0058860 tx=0x7f43d0088730 comp rx=0 tx=0).stop 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 shutdown_connections 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:6835/1183474661,v1:172.21.15.145:6836/1183474661] conn(0x55f3253def40 0x55f3253df320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f3253b9960 0x55f32543fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43c0ff9640 0 client.9714 ms_handle_reset on v2:172.21.15.145:6835/1183474661 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f325396330 0x55f3253b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 --2- 172.21.15.94:0/2032181213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55f325395f50 0x55f325393ef0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 >> 172.21.15.94:0/2032181213 conn(0x55f3253944b0 msgr2=0x55f325397170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 shutdown_connections 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 1 -- 172.21.15.94:0/2032181213 wait complete. 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43ea322740 20 client.9714 trim_cache size 0 max 16384 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 37613, 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:25:23.260 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 2.9689723539999999, 2024-03-06T07:25:23.261 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 165, 2024-03-06T07:25:23.261 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 170, 2024-03-06T07:25:23.261 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 170, 2024-03-06T07:25:23.261 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 5.956751626 2024-03-06T07:25:23.261 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43c406b4d0 msgr2=0x7f43c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43c406b4d0 0x7f43c406d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f43dc035000 tx=0x7f43dc008030 comp rx=0 tx=0).stop 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43e00aa430 msgr2=0x7f43e0072350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.251+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43e00aa430 0x7f43e0072350 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f43d000e870 tx=0x7f43d000ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 shutdown_connections 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f43c406b4d0 0x7f43c406d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43e01561a0 0x7f43e01410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f43e00c6ab0 0x7f43e0140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 --2- 172.21.15.94:0/3755221104 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43e00aa430 0x7f43e0072350 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 >> 172.21.15.94:0/3755221104 conn(0x7f43e006b1a0 msgr2=0x7f43e00ab1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 shutdown_connections 2024-03-06T07:25:23.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.252+0000 7f43e7a9a640 1 -- 172.21.15.94:0/3755221104 wait complete. 2024-03-06T07:25:23.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:23.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.363+0000 7f13624df640 1 Processor -- start 2024-03-06T07:25:23.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.363+0000 7f13624df640 1 -- start start 2024-03-06T07:25:23.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f13624df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c88c0 0x7f135c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f13624df640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f135c158b60 0x7f135c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f13624df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135c0c73d0 con 0x7f135c14f360 2024-03-06T07:25:23.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f13624df640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c0c8dd0 con 0x7f135c158b60 2024-03-06T07:25:23.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f13624df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0be2e0 con 0x7f135c0c88c0 2024-03-06T07:25:23.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f1360a7c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f135c158b60 0x7f135c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f135bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f135c14f360 0x7f135c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49768/0 (socket says 172.21.15.94:49768) 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f135bfff640 1 -- 172.21.15.94:0/725040543 learned_addr learned my addr 172.21.15.94:0/725040543 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.364+0000 7f135b7fe640 1 --2- 172.21.15.94:0/725040543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c88c0 0x7f135c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f1360a7c640 1 -- 172.21.15.94:0/725040543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c88c0 msgr2=0x7f135c155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f1360a7c640 1 --2- 172.21.15.94:0/725040543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c88c0 0x7f135c155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f1360a7c640 1 -- 172.21.15.94:0/725040543 >> v1:172.21.15.94:6789/0 conn(0x7f135c14f360 legacy=0x7f135c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f1360a7c640 1 -- 172.21.15.94:0/725040543 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c0c7670 con 0x7f135c158b60 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f135affd640 1 -- 172.21.15.94:0/725040543 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228736488 0 0) 0x7f135c0c73d0 con 0x7f135c14f360 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f1360a7c640 1 --2- 172.21.15.94:0/725040543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c158ff0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f1354009d90 tx=0x7f135402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9093843147746f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f135affd640 1 -- 172.21.15.94:0/725040543 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354039070 con 0x7f135c158b60 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f135affd640 1 -- 172.21.15.94:0/725040543 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135402cbc0 con 0x7f135c158b60 2024-03-06T07:25:23.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.365+0000 7f135affd640 1 -- 172.21.15.94:0/725040543 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354034440 con 0x7f135c158b60 2024-03-06T07:25:23.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 -- 172.21.15.94:0/725040543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 msgr2=0x7f135c158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 --2- 172.21.15.94:0/725040543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c158ff0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f1354009d90 tx=0x7f135402be10 comp rx=0 tx=0).stop 2024-03-06T07:25:23.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 -- 172.21.15.94:0/725040543 shutdown_connections 2024-03-06T07:25:23.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 --2- 172.21.15.94:0/725040543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c158ff0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 --2- 172.21.15.94:0/725040543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c0c88c0 0x7f135c155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 -- 172.21.15.94:0/725040543 >> 172.21.15.94:0/725040543 conn(0x7f135c0ac570 msgr2=0x7f135c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 -- 172.21.15.94:0/725040543 shutdown_connections 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.366+0000 7f13624df640 1 -- 172.21.15.94:0/725040543 wait complete. 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.367+0000 7f13624df640 1 Processor -- start 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.367+0000 7f13624df640 1 -- start start 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f13624df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f135c0c88c0 0x7f135c1ec0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f13624df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c14f360 0x7f135c0bf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f13624df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c14b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f13624df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0c5a50 con 0x7f135c158b60 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f13624df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0c7c30 con 0x7f135c0c88c0 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f13624df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0c5bd0 con 0x7f135c14f360 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c14b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c14b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60484/0 (socket says 172.21.15.94:60484) 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 -- 172.21.15.94:0/1389825139 learned_addr learned my addr 172.21.15.94:0/1389825139 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f135b7fe640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c14f360 0x7f135c0bf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f135bfff640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f135c0c88c0 0x7f135c1ec0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 -- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c14f360 msgr2=0x7f135c0bf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c14f360 0x7f135c0bf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 -- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f135c0c88c0 msgr2=0x7f135c1ec0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f135c0c88c0 0x7f135c1ec0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 -- 172.21.15.94:0/1389825139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c1ec610 con 0x7f135c158b60 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f135bfff640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f135c0c88c0 0x7f135c1ec0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.368+0000 7f1360a7c640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c14b790 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f1354037040 tx=0x7f1354004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.369+0000 7f1358ff9640 1 -- 172.21.15.94:0/1389825139 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354039050 con 0x7f135c158b60 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.369+0000 7f1358ff9640 1 -- 172.21.15.94:0/1389825139 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354045070 con 0x7f135c158b60 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.369+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c0bfce0 con 0x7f135c158b60 2024-03-06T07:25:23.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.369+0000 7f1358ff9640 1 -- 172.21.15.94:0/1389825139 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354009450 con 0x7f135c158b60 2024-03-06T07:25:23.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.369+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c0c0ca0 con 0x7f135c158b60 2024-03-06T07:25:23.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.370+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c04f330 con 0x7f135c158b60 2024-03-06T07:25:23.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.370+0000 7f1358ff9640 1 -- 172.21.15.94:0/1389825139 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f135403e850 con 0x7f135c158b60 2024-03-06T07:25:23.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.370+0000 7f1358ff9640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f134806b500 0x7f134806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.371+0000 7f1358ff9640 1 -- 172.21.15.94:0/1389825139 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f13540ac8e0 con 0x7f135c158b60 2024-03-06T07:25:23.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.371+0000 7f135bfff640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f134806b500 0x7f134806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.371+0000 7f135bfff640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f134806b500 0x7f134806d9c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f134c000f60 tx=0x7f134c007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.375+0000 7f1358ff9640 1 -- 172.21.15.94:0/1389825139 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f135404e020 con 0x7f135c158b60 2024-03-06T07:25:23.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.582+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f135c155f20 con 0x7f135c158b60 2024-03-06T07:25:23.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.583+0000 7f1358ff9640 1 -- 172.21.15.94:0/1389825139 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f135407b660 con 0x7f135c158b60 2024-03-06T07:25:23.585 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:23.585 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:21.282376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:active","state_seq":2,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37613,"qdb_cluster":[37613]},"id":13}]} 2024-03-06T07:25:23.585 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 165 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f134806b500 msgr2=0x7f134806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f134806b500 0x7f134806d9c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f134c000f60 tx=0x7f134c007ca0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 msgr2=0x7f135c14b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c14b790 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f1354037040 tx=0x7f1354004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 shutdown_connections 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f134806b500 0x7f134806d9c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f135c158b60 0x7f135c14b790 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f135c14f360 0x7f135c0bf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 --2- 172.21.15.94:0/1389825139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f135c0c88c0 0x7f135c1ec0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 >> 172.21.15.94:0/1389825139 conn(0x7f135c0ac570 msgr2=0x7f135c0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 shutdown_connections 2024-03-06T07:25:23.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.585+0000 7f13624df640 1 -- 172.21.15.94:0/1389825139 wait complete. 2024-03-06T07:25:23.597 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:23.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:23.634 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:23.688 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:23.688 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:25:23.688 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:23.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:25:23.703 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:25:23.703 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:23.718 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:25:23.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:23.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.842+0000 7fa873593640 1 Processor -- start 2024-03-06T07:25:23.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.842+0000 7fa873593640 1 -- start start 2024-03-06T07:25:23.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa873593640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa86c150580 0x7fa86c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa873593640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa86c0c6000 0x7fa86c06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa873593640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c153a60 con 0x7fa86c0c1320 2024-03-06T07:25:23.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa873593640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c153d40 con 0x7fa86c150580 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa873593640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c06c2c0 con 0x7fa86c0c6000 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa87132f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa86c150580 0x7fa86c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa87132f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa86c150580 0x7fa86c06dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60488/0 (socket says 172.21.15.94:60488) 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa87132f640 1 -- 172.21.15.94:0/2268357973 learned_addr learned my addr 172.21.15.94:0/2268357973 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa871b30640 1 --2- 172.21.15.94:0/2268357973 >> v2:172.21.15.145:3300/0 conn(0x7fa86c0c6000 0x7fa86c06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa87132f640 1 -- 172.21.15.94:0/2268357973 >> v2:172.21.15.145:3300/0 conn(0x7fa86c0c6000 msgr2=0x7fa86c06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa87132f640 1 --2- 172.21.15.94:0/2268357973 >> v2:172.21.15.145:3300/0 conn(0x7fa86c0c6000 0x7fa86c06e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa87132f640 1 -- 172.21.15.94:0/2268357973 >> v1:172.21.15.145:6789/0 conn(0x7fa86c0c1320 legacy=0x7fa86c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.843+0000 7fa87132f640 1 -- 172.21.15.94:0/2268357973 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c06c5a0 con 0x7fa86c150580 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.844+0000 7fa87132f640 1 --2- 172.21.15.94:0/2268357973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c150580 0x7fa86c06dd10 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0c5960 tx=0x7fa86802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6991a6e00cbac78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.844+0000 7fa863fff640 1 -- 172.21.15.94:0/2268357973 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868035020 con 0x7fa86c150580 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.844+0000 7fa863fff640 1 -- 172.21.15.94:0/2268357973 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868031870 con 0x7fa86c150580 2024-03-06T07:25:23.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.844+0000 7fa863fff640 1 -- 172.21.15.94:0/2268357973 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868031c40 con 0x7fa86c150580 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.844+0000 7fa873593640 1 -- 172.21.15.94:0/2268357973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c150580 msgr2=0x7fa86c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.844+0000 7fa873593640 1 --2- 172.21.15.94:0/2268357973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c150580 0x7fa86c06dd10 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0c5960 tx=0x7fa86802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.845+0000 7fa873593640 1 -- 172.21.15.94:0/2268357973 shutdown_connections 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.845+0000 7fa873593640 1 --2- 172.21.15.94:0/2268357973 >> v2:172.21.15.145:3300/0 conn(0x7fa86c0c6000 0x7fa86c06e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.845+0000 7fa873593640 1 --2- 172.21.15.94:0/2268357973 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c150580 0x7fa86c06dd10 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.845+0000 7fa873593640 1 -- 172.21.15.94:0/2268357973 >> 172.21.15.94:0/2268357973 conn(0x7fa86c0c6710 msgr2=0x7fa86c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.845+0000 7fa873593640 1 -- 172.21.15.94:0/2268357973 shutdown_connections 2024-03-06T07:25:23.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.845+0000 7fa873593640 1 -- 172.21.15.94:0/2268357973 wait complete. 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 Processor -- start 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 -- start start 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c0c1320 0x7fa86c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86c0c6000 0x7fa86c0bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa86c150580 0x7fa86c145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c0c0dc0 con 0x7fa86c0c1320 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c0c0c40 con 0x7fa86c150580 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa873593640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c156dc0 con 0x7fa86c0c6000 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa87132f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c0c1320 0x7fa86c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa87132f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c0c1320 0x7fa86c152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60500/0 (socket says 172.21.15.94:60500) 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.846+0000 7fa87132f640 1 -- 172.21.15.94:0/3812903431 learned_addr learned my addr 172.21.15.94:0/3812903431 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa871b30640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa86c150580 0x7fa86c145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa870b2e640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86c0c6000 0x7fa86c0bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa87132f640 1 -- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86c0c6000 msgr2=0x7fa86c0bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa87132f640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86c0c6000 0x7fa86c0bfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa87132f640 1 -- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa86c150580 msgr2=0x7fa86c145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa87132f640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa86c150580 0x7fa86c145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa87132f640 1 -- 172.21.15.94:0/3812903431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0aade0 con 0x7fa86c0c1320 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa870b2e640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86c0c6000 0x7fa86c0bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa87132f640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c0c1320 0x7fa86c152be0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fa86802b950 tx=0x7fa8680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa861ffb640 1 -- 172.21.15.94:0/3812903431 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868040070 con 0x7fa86c0c1320 2024-03-06T07:25:23.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa861ffb640 1 -- 172.21.15.94:0/3812903431 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868030d00 con 0x7fa86c0c1320 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c0ab0c0 con 0x7fa86c0c1320 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa861ffb640 1 -- 172.21.15.94:0/3812903431 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868035040 con 0x7fa86c0c1320 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.847+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c0ab680 con 0x7fa86c0c1320 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.848+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c06bfa0 con 0x7fa86c0c1320 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.849+0000 7fa861ffb640 1 -- 172.21.15.94:0/3812903431 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa86804b060 con 0x7fa86c0c1320 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.849+0000 7fa861ffb640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa85006b4d0 0x7fa85006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.849+0000 7fa861ffb640 1 -- 172.21.15.94:0/3812903431 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa8680abf50 con 0x7fa86c0c1320 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.849+0000 7fa870b2e640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa85006b4d0 0x7fa85006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.850+0000 7fa870b2e640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa85006b4d0 0x7fa85006d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa86c1e9110 tx=0x7fa85c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:23.853+0000 7fa861ffb640 1 -- 172.21.15.94:0/3812903431 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa86c06bfa0 con 0x7fa86c0c1320 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.062+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa86c04f330 con 0x7fa86c0c1320 2024-03-06T07:25:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.063+0000 7fa861ffb640 1 -- 172.21.15.94:0/3812903431 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4998 (secure 0 0 0) 0x7fa86807acd0 con 0x7fa86c0c1320 2024-03-06T07:25:24.065 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:24.065 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:21.282376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:active","state_seq":2,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37613,"qdb_cluster":[37613]},"id":13}]} 2024-03-06T07:25:24.065 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 165 2024-03-06T07:25:24.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa85006b4d0 msgr2=0x7fa85006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa85006b4d0 0x7fa85006d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa86c1e9110 tx=0x7fa85c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c0c1320 msgr2=0x7fa86c152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c0c1320 0x7fa86c152be0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fa86802b950 tx=0x7fa8680046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 shutdown_connections 2024-03-06T07:25:24.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa85006b4d0 0x7fa85006d990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa86c150580 0x7fa86c145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86c0c6000 0x7fa86c0bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 --2- 172.21.15.94:0/3812903431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa86c0c1320 0x7fa86c152be0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 >> 172.21.15.94:0/3812903431 conn(0x7fa86c0c6710 msgr2=0x7fa86c06dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 shutdown_connections 2024-03-06T07:25:24.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.065+0000 7fa873593640 1 -- 172.21.15.94:0/3812903431 wait complete. 2024-03-06T07:25:24.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:24.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.175+0000 7f07f7ad4640 1 Processor -- start 2024-03-06T07:25:24.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.175+0000 7f07f7ad4640 1 -- start start 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f7ad4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f014f360 0x7f07f0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f7ad4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07f0158b60 0x7f07f0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f7ad4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f00c73d0 con 0x7f07f00c88c0 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f7ad4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f00c8dd0 con 0x7f07f0158b60 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f7ad4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00be2e0 con 0x7f07f014f360 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f6071640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07f0158b60 0x7f07f0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f6071640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f07f0158b60 0x7f07f0158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60514/0 (socket says 172.21.15.94:60514) 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f506f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f07f00c88c0 0x7f07f00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49774/0 (socket says 172.21.15.94:49774) 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f6071640 1 -- 172.21.15.94:0/918216728 learned_addr learned my addr 172.21.15.94:0/918216728 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.176+0000 7f07f5870640 1 --2- 172.21.15.94:0/918216728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f014f360 0x7f07f0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.177+0000 7f07f6071640 1 -- 172.21.15.94:0/918216728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f014f360 msgr2=0x7f07f0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.177+0000 7f07f6071640 1 --2- 172.21.15.94:0/918216728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f014f360 0x7f07f0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.177+0000 7f07f6071640 1 -- 172.21.15.94:0/918216728 >> v1:172.21.15.94:6789/0 conn(0x7f07f00c88c0 legacy=0x7f07f00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.177+0000 7f07f6071640 1 -- 172.21.15.94:0/918216728 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f00c7670 con 0x7f07f0158b60 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.177+0000 7f07f6071640 1 --2- 172.21.15.94:0/918216728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f0158b60 0x7f07f0158ff0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009e50 tx=0x7f07e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72027ca73a837c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f486e640 1 -- 172.21.15.94:0/918216728 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e802ca70 con 0x7f07f0158b60 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f486e640 1 -- 172.21.15.94:0/918216728 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e802cc10 con 0x7f07f0158b60 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f486e640 1 -- 172.21.15.94:0/918216728 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e80343f0 con 0x7f07f0158b60 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 -- 172.21.15.94:0/918216728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f0158b60 msgr2=0x7f07f0158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/918216728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f0158b60 0x7f07f0158ff0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009e50 tx=0x7f07e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 -- 172.21.15.94:0/918216728 shutdown_connections 2024-03-06T07:25:24.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/918216728 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f0158b60 0x7f07f0158ff0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/918216728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f014f360 0x7f07f0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 -- 172.21.15.94:0/918216728 >> 172.21.15.94:0/918216728 conn(0x7f07f00ac570 msgr2=0x7f07f00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 -- 172.21.15.94:0/918216728 shutdown_connections 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.178+0000 7f07f7ad4640 1 -- 172.21.15.94:0/918216728 wait complete. 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.179+0000 7f07f7ad4640 1 Processor -- start 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.179+0000 7f07f7ad4640 1 -- start start 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f7ad4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f00c88c0 0x7f07f00abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f7ad4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f014f360 0x7f07f0148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f7ad4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07f0158b60 0x7f07f0149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f7ad4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00c5a50 con 0x7f07f014f360 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f7ad4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00c7c30 con 0x7f07f0158b60 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f7ad4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f00c5bd0 con 0x7f07f00c88c0 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f014f360 0x7f07f0148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f014f360 0x7f07f0148c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60516/0 (socket says 172.21.15.94:60516) 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 -- 172.21.15.94:0/726908666 learned_addr learned my addr 172.21.15.94:0/726908666 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f5870640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f00c88c0 0x7f07f00abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f6071640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07f0158b60 0x7f07f0149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 -- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f00c88c0 msgr2=0x7f07f00abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f00c88c0 0x7f07f00abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 -- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07f0158b60 msgr2=0x7f07f0149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07f0158b60 0x7f07f0149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f506f640 1 -- 172.21.15.94:0/726908666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f01ec1d0 con 0x7f07f014f360 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.180+0000 7f07f6071640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07f0158b60 0x7f07f0149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.181+0000 7f07f506f640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f014f360 0x7f07f0148c20 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c470 tx=0x7f07e000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.181+0000 7f07e67fc640 1 -- 172.21.15.94:0/726908666 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e00074e0 con 0x7f07f014f360 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.181+0000 7f07e67fc640 1 -- 172.21.15.94:0/726908666 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e0007680 con 0x7f07f014f360 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.181+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f01ec4b0 con 0x7f07f014f360 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.181+0000 7f07e67fc640 1 -- 172.21.15.94:0/726908666 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0004a00 con 0x7f07f014f360 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.181+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f01eca70 con 0x7f07f014f360 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.182+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f004f330 con 0x7f07f014f360 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.182+0000 7f07e67fc640 1 -- 172.21.15.94:0/726908666 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07e001a020 con 0x7f07f014f360 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.183+0000 7f07e67fc640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f07d406b500 0x7f07d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.183+0000 7f07e67fc640 1 -- 172.21.15.94:0/726908666 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f07e0089900 con 0x7f07f014f360 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.183+0000 7f07f5870640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f07d406b500 0x7f07d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.183+0000 7f07f5870640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f07d406b500 0x7f07d406d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009b40 tx=0x7f07ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.187+0000 7f07e67fc640 1 -- 172.21.15.94:0/726908666 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07e0058680 con 0x7f07f014f360 2024-03-06T07:25:24.273 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:25:24.270+0000 7f02403a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:24.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.359+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07f014c710 con 0x7f07f014f360 2024-03-06T07:25:24.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.360+0000 7f07e67fc640 1 -- 172.21.15.94:0/726908666 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+16952 (secure 0 0 0) 0x7f07e0057f80 con 0x7f07f014f360 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":170,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:57.969058+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:54.539968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:55.288852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f07d406b500 msgr2=0x7f07d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f07d406b500 0x7f07d406d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009b40 tx=0x7f07ec009190 comp rx=0 tx=0).stop 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f014f360 msgr2=0x7f07f0148c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f014f360 0x7f07f0148c20 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c470 tx=0x7f07e000c930 comp rx=0 tx=0).stop 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 shutdown_connections 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f07d406b500 0x7f07d406d9c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07f0158b60 0x7f07f0149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f07f014f360 0x7f07f0148c20 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 --2- 172.21.15.94:0/726908666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07f00c88c0 0x7f07f00abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.362+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 >> 172.21.15.94:0/726908666 conn(0x7f07f00ac570 msgr2=0x7f07f006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.363+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 shutdown_connections 2024-03-06T07:25:24.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.363+0000 7f07f7ad4640 1 -- 172.21.15.94:0/726908666 wait complete. 2024-03-06T07:25:24.375 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:54.539968+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '169', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:55.288852+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:24.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.477+0000 7f44de2d4640 1 Processor -- start 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.477+0000 7f44de2d4640 1 -- start start 2024-03-06T07:25:24.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.477+0000 7f44de2d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 0x7f44d8160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.477+0000 7f44de2d4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d806b800 con 0x7f44d806afc0 2024-03-06T07:25:24.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.477+0000 7f44de2d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80c5840 con 0x7f44d806b3a0 2024-03-06T07:25:24.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.477+0000 7f44de2d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8158790 con 0x7f44d8159330 2024-03-06T07:25:24.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f44d806afc0 0x7f44d80c2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49782/0 (socket says 172.21.15.94:49782) 2024-03-06T07:25:24.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d7fff640 1 -- 172.21.15.94:0/3294462659 learned_addr learned my addr 172.21.15.94:0/3294462659 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:24.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44dc871640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 0x7f44d8160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502725259 0 0) 0x7f44d806b800 con 0x7f44d806afc0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c0003520 con 0x7f44d806afc0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364075213 0 0) 0x7f44d80c5840 con 0x7f44d806b3a0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d806b800 con 0x7f44d806b3a0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1803484870 0 0) 0x7f44c0003520 con 0x7f44d806afc0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 msgr2=0x7f44d8160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 0x7f44d8160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 >> v1:172.21.15.145:6789/0 conn(0x7f44d806b3a0 legacy=0x7f44d8158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d80c5b20 con 0x7f44d806afc0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f44c8001670 con 0x7f44d806afc0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 >> v1:172.21.15.94:6789/0 conn(0x7f44d806afc0 legacy=0x7f44d80c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44c0003b70 0x7f44c0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 0x7f44c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0004d40 0x7f44c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8158790 con 0x7f44c0004d40 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c000baf0 con 0x7f44c0003b70 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c000bcf0 con 0x7f44d8159330 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44dc871640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0004d40 0x7f44c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d77fe640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 0x7f44c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.478+0000 7f44d7fff640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44c0003b70 0x7f44c0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44dc871640 1 -- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 msgr2=0x7f44c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44dc871640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 0x7f44c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44dc871640 1 -- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44c0003b70 msgr2=0x7f44c0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44dc871640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44c0003b70 0x7f44c0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44dc871640 1 -- 172.21.15.94:0/3294462659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0003520 con 0x7f44c0004d40 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44d77fe640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d8159330 0x7f44c0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44dc871640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0004d40 0x7f44c00051c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009fc0 tx=0x7f44c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10316645f5cfb60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0039070 con 0x7f44c0004d40 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d002cc20 con 0x7f44c0004d40 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.479+0000 7f44d6ffd640 1 -- 172.21.15.94:0/3294462659 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0009c10 con 0x7f44c0004d40 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.480+0000 7f44de2d4640 1 -- 172.21.15.94:0/3294462659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0004d40 msgr2=0x7f44c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.480+0000 7f44de2d4640 1 --2- 172.21.15.94:0/3294462659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44c0004d40 0x7f44c00051c0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009fc0 tx=0x7f44c0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.480+0000 7f44dc871640 1 -- 172.21.15.94:0/3294462659 reap_dead start 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.480+0000 7f44de2d4640 1 -- 172.21.15.94:0/3294462659 shutdown_connections 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.480+0000 7f44de2d4640 1 -- 172.21.15.94:0/3294462659 >> 172.21.15.94:0/3294462659 conn(0x7f44d80c4220 msgr2=0x7f44d80c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.480+0000 7f44de2d4640 1 -- 172.21.15.94:0/3294462659 shutdown_connections 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.480+0000 7f44de2d4640 1 -- 172.21.15.94:0/3294462659 wait complete. 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.481+0000 7f44de2d4640 1 Processor -- start 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.481+0000 7f44de2d4640 1 -- start start 2024-03-06T07:25:24.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44de2d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d8157c30 0x7f44d8158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44de2d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d81ee440 0x7f44d8156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44de2d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d8156cd0 0x7f44d81490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44de2d4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8155220 con 0x7f44d8157c30 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44de2d4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80c60e0 con 0x7f44d8156cd0 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44de2d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80c2b30 con 0x7f44d81ee440 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d8157c30 0x7f44d8158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d8157c30 0x7f44d8158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60542/0 (socket says 172.21.15.94:60542) 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 -- 172.21.15.94:0/1912934616 learned_addr learned my addr 172.21.15.94:0/1912934616 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d77fe640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d81ee440 0x7f44d8156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44dc871640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d8156cd0 0x7f44d81490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 -- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d81ee440 msgr2=0x7f44d8156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d81ee440 0x7f44d8156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 -- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d8156cd0 msgr2=0x7f44d81490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d8156cd0 0x7f44d81490e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.482+0000 7f44d7fff640 1 -- 172.21.15.94:0/1912934616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d8158550 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.483+0000 7f44d7fff640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d8157c30 0x7f44d8158010 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f44c80017c0 tx=0x7f44c8007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.483+0000 7f44d4ff9640 1 -- 172.21.15.94:0/1912934616 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c800f750 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.483+0000 7f44d4ff9640 1 -- 172.21.15.94:0/1912934616 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c8018070 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.483+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d81eaed0 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.483+0000 7f44d4ff9640 1 -- 172.21.15.94:0/1912934616 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c800fb50 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.483+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81eb470 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.484+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d81ebe00 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.484+0000 7f44d4ff9640 1 -- 172.21.15.94:0/1912934616 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44c800fcf0 con 0x7f44d8157c30 2024-03-06T07:25:24.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.485+0000 7f44d4ff9640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.485+0000 7f44d4ff9640 1 -- 172.21.15.94:0/1912934616 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f44c808a180 con 0x7f44d8157c30 2024-03-06T07:25:24.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.485+0000 7f44d77fe640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.485+0000 7f44d77fe640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44c406b500 0x7f44c406d9c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f44d814a630 tx=0x7f44cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.489+0000 7f44d4ff9640 1 -- 172.21.15.94:0/1912934616 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44c8058f00 con 0x7f44d8157c30 2024-03-06T07:25:24.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.658+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44d80c6730 con 0x7f44d8157c30 2024-03-06T07:25:24.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.658+0000 7f44d4ff9640 1 -- 172.21.15.94:0/1912934616 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v170) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44c8058800 con 0x7f44d8157c30 2024-03-06T07:25:24.660 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44c406b500 msgr2=0x7f44c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44c406b500 0x7f44c406d9c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f44d814a630 tx=0x7f44cc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d8157c30 msgr2=0x7f44d8158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d8157c30 0x7f44d8158010 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f44c80017c0 tx=0x7f44c8007430 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 shutdown_connections 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f44c406b500 0x7f44c406d9c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44d8156cd0 0x7f44d81490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44d81ee440 0x7f44d8156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 --2- 172.21.15.94:0/1912934616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44d8157c30 0x7f44d8158010 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 >> 172.21.15.94:0/1912934616 conn(0x7f44d80c4220 msgr2=0x7f44d806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 shutdown_connections 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.660+0000 7f44de2d4640 1 -- 172.21.15.94:0/1912934616 wait complete. 2024-03-06T07:25:24.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:24.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.774+0000 7f2cef73a640 1 Processor -- start 2024-03-06T07:25:24.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.774+0000 7f2cef73a640 1 -- start start 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cef73a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2ce80ab310 0x7f2ce80ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cef73a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce8154330 con 0x7f2ce81538c0 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cef73a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce8153dc0 con 0x7f2ce80a97c0 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cef73a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80c46a0 con 0x7f2ce80ab310 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2ceccd5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2ce81538c0 0x7f2ce81494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49792/0 (socket says 172.21.15.94:49792) 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2ceccd5640 1 -- 172.21.15.94:0/989269951 learned_addr learned my addr 172.21.15.94:0/989269951 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2ced4d6640 1 --2- 172.21.15.94:0/989269951 >> v2:172.21.15.145:3300/0 conn(0x7f2ce80ab310 0x7f2ce80ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cdffff640 1 -- 172.21.15.94:0/989269951 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577506974 0 0) 0x7f2ce8154330 con 0x7f2ce81538c0 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cdffff640 1 -- 172.21.15.94:0/989269951 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd4003520 con 0x7f2ce81538c0 2024-03-06T07:25:24.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cdffff640 1 -- 172.21.15.94:0/989269951 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583381197 0 0) 0x7f2ce8153dc0 con 0x7f2ce80a97c0 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2cdffff640 1 -- 172.21.15.94:0/989269951 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce8154330 con 0x7f2ce80a97c0 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2ced4d6640 1 -- 172.21.15.94:0/989269951 >> v1:172.21.15.145:6789/0 conn(0x7f2ce80a97c0 legacy=0x7f2ce80c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2ced4d6640 1 -- 172.21.15.94:0/989269951 >> v1:172.21.15.94:6789/0 conn(0x7f2ce81538c0 legacy=0x7f2ce81494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.775+0000 7f2ced4d6640 1 -- 172.21.15.94:0/989269951 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80c5d30 con 0x7f2ce80ab310 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.776+0000 7f2ced4d6640 1 --2- 172.21.15.94:0/989269951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80ab310 0x7f2ce80ab6f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40097a0 tx=0x7f2ce402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a78c3ff1335da2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.776+0000 7f2cdffff640 1 -- 172.21.15.94:0/989269951 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4035020 con 0x7f2ce80ab310 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.776+0000 7f2cdffff640 1 -- 172.21.15.94:0/989269951 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce4031830 con 0x7f2ce80ab310 2024-03-06T07:25:24.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.776+0000 7f2cdffff640 1 -- 172.21.15.94:0/989269951 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4031c00 con 0x7f2ce80ab310 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.777+0000 7f2cef73a640 1 -- 172.21.15.94:0/989269951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80ab310 msgr2=0x7f2ce80ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.777+0000 7f2cef73a640 1 --2- 172.21.15.94:0/989269951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80ab310 0x7f2ce80ab6f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40097a0 tx=0x7f2ce402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.777+0000 7f2cef73a640 1 -- 172.21.15.94:0/989269951 shutdown_connections 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.777+0000 7f2cef73a640 1 --2- 172.21.15.94:0/989269951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80ab310 0x7f2ce80ab6f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.777+0000 7f2cef73a640 1 -- 172.21.15.94:0/989269951 >> 172.21.15.94:0/989269951 conn(0x7f2ce80c6710 msgr2=0x7f2ce80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.777+0000 7f2cef73a640 1 -- 172.21.15.94:0/989269951 shutdown_connections 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.777+0000 7f2cef73a640 1 -- 172.21.15.94:0/989269951 wait complete. 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.778+0000 7f2cef73a640 1 Processor -- start 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.778+0000 7f2cef73a640 1 -- start start 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cef73a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80a97c0 0x7f2ce81e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cef73a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ce80ab310 0x7f2ce8156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cef73a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ce81538c0 0x7f2ce81ec7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cef73a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80abc30 con 0x7f2ce81538c0 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cef73a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80c62f0 con 0x7f2ce80a97c0 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cef73a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8149bb0 con 0x7f2ce80ab310 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ce81538c0 0x7f2ce81ec7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ce81538c0 0x7f2ce81ec7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60546/0 (socket says 172.21.15.94:60546) 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 -- 172.21.15.94:0/3857747716 learned_addr learned my addr 172.21.15.94:0/3857747716 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2ceccd5640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ce80ab310 0x7f2ce8156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2ced4d6640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80a97c0 0x7f2ce81e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 -- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ce80ab310 msgr2=0x7f2ce8156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ce80ab310 0x7f2ce8156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 -- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80a97c0 msgr2=0x7f2ce81e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80a97c0 0x7f2ce81e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 -- 172.21.15.94:0/3857747716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce814c390 con 0x7f2ce81538c0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cedcd7640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ce81538c0 0x7f2ce81ec7a0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000c790 tx=0x7f2ce000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cddffb640 1 -- 172.21.15.94:0/3857747716 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0003b70 con 0x7f2ce81538c0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cddffb640 1 -- 172.21.15.94:0/3857747716 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0003d10 con 0x7f2ce81538c0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cddffb640 1 -- 172.21.15.94:0/3857747716 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce000aa50 con 0x7f2ce81538c0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.779+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce814c670 con 0x7f2ce81538c0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.780+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce81ebfa0 con 0x7f2ce81538c0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.781+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce80c0f90 con 0x7f2ce81538c0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.781+0000 7f2cddffb640 1 -- 172.21.15.94:0/3857747716 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ce000abf0 con 0x7f2ce81538c0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.781+0000 7f2cddffb640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ccc06b500 0x7f2ccc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.781+0000 7f2cddffb640 1 -- 172.21.15.94:0/3857747716 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2ce008a140 con 0x7f2ce81538c0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.781+0000 7f2ced4d6640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ccc06b500 0x7f2ccc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.782+0000 7f2ced4d6640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ccc06b500 0x7f2ccc06d9c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4036000 tx=0x7f2ce4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.786+0000 7f2cddffb640 1 -- 172.21.15.94:0/3857747716 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ce0058870 con 0x7f2ce81538c0 2024-03-06T07:25:24.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.959+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2ce804f330 con 0x7f2ce81538c0 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.960+0000 7f2cddffb640 1 -- 172.21.15.94:0/3857747716 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v170) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2ce005c910 con 0x7f2ce81538c0 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:25:24.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ccc06b500 msgr2=0x7f2ccc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ccc06b500 0x7f2ccc06d9c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4036000 tx=0x7f2ce4008030 comp rx=0 tx=0).stop 2024-03-06T07:25:24.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ce81538c0 msgr2=0x7f2ce81ec7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ce81538c0 0x7f2ce81ec7a0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000c790 tx=0x7f2ce000cc50 comp rx=0 tx=0).stop 2024-03-06T07:25:24.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 shutdown_connections 2024-03-06T07:25:24.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ccc06b500 0x7f2ccc06d9c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ce81538c0 0x7f2ce81ec7a0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ce80ab310 0x7f2ce8156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 --2- 172.21.15.94:0/3857747716 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ce80a97c0 0x7f2ce81e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 >> 172.21.15.94:0/3857747716 conn(0x7f2ce80c6710 msgr2=0x7f2ce80a7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 shutdown_connections 2024-03-06T07:25:24.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:24.962+0000 7f2cef73a640 1 -- 172.21.15.94:0/3857747716 wait complete. 2024-03-06T07:25:24.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:25.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.075+0000 7f3d7bab5640 1 Processor -- start 2024-03-06T07:25:25.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.076+0000 7f3d7bab5640 1 -- start start 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.076+0000 7f3d7bab5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3d740c2aa0 0x7f3d74155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.076+0000 7f3d7bab5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740670c0 con 0x7f3d740c0470 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.076+0000 7f3d7bab5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7406b0f0 con 0x7f3d740c26c0 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.076+0000 7f3d7bab5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7406b740 con 0x7f3d740c2aa0 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d79050640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3d740c2aa0 0x7f3d74155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d79050640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3d740c2aa0 0x7f3d74155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60550/0 (socket says 172.21.15.94:60550) 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d7a052640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3d740c0470 0x7f3d74152350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49806/0 (socket says 172.21.15.94:49806) 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d79050640 1 -- 172.21.15.94:0/2326907560 learned_addr learned my addr 172.21.15.94:0/2326907560 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d79050640 1 -- 172.21.15.94:0/2326907560 >> v1:172.21.15.145:6789/0 conn(0x7f3d740c26c0 legacy=0x7f3d7406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d79050640 1 -- 172.21.15.94:0/2326907560 >> v1:172.21.15.94:6789/0 conn(0x7f3d740c0470 legacy=0x7f3d74152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d7884f640 1 -- 172.21.15.94:0/2326907560 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411544051 0 0) 0x7f3d740670c0 con 0x7f3d740c0470 2024-03-06T07:25:25.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.077+0000 7f3d79050640 1 -- 172.21.15.94:0/2326907560 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7406b3d0 con 0x7f3d740c2aa0 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d79050640 1 --2- 172.21.15.94:0/2326907560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c2aa0 0x7f3d74155bd0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009870 tx=0x7f3d7002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b89fdf852db5a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7884f640 1 -- 172.21.15.94:0/2326907560 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70035020 con 0x7f3d740c2aa0 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7884f640 1 -- 172.21.15.94:0/2326907560 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d70031870 con 0x7f3d740c2aa0 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7884f640 1 -- 172.21.15.94:0/2326907560 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70031c40 con 0x7f3d740c2aa0 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2326907560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c2aa0 msgr2=0x7f3d74155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2326907560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c2aa0 0x7f3d74155bd0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009870 tx=0x7f3d7002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2326907560 shutdown_connections 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2326907560 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c2aa0 0x7f3d74155bd0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.078+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2326907560 >> 172.21.15.94:0/2326907560 conn(0x7f3d740c3b40 msgr2=0x7f3d740c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.079+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2326907560 shutdown_connections 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.079+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2326907560 wait complete. 2024-03-06T07:25:25.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 Processor -- start 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 -- start start 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c0470 0x7f3d741edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d740c26c0 0x7f3d74146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d740c2aa0 0x7f3d74147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740c20b0 con 0x7f3d740c0470 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740c1f30 con 0x7f3d740c26c0 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7bab5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74066d00 con 0x7f3d740c2aa0 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d79851640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c0470 0x7f3d741edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d79851640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c0470 0x7f3d741edf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60556/0 (socket says 172.21.15.94:60556) 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d79851640 1 -- 172.21.15.94:0/2420254922 learned_addr learned my addr 172.21.15.94:0/2420254922 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d79050640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d740c26c0 0x7f3d74146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.080+0000 7f3d7a052640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d740c2aa0 0x7f3d74147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d79851640 1 -- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d740c2aa0 msgr2=0x7f3d74147300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d79851640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d740c2aa0 0x7f3d74147300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d79851640 1 -- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d740c26c0 msgr2=0x7f3d74146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d79851640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d740c26c0 0x7f3d74146dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d79851640 1 -- 172.21.15.94:0/2420254922 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74154dd0 con 0x7f3d740c0470 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d7a052640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d740c2aa0 0x7f3d74147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d79851640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c0470 0x7f3d741edf90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c450 tx=0x7f3d6800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d667fc640 1 -- 172.21.15.94:0/2420254922 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d680052d0 con 0x7f3d740c0470 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d667fc640 1 -- 172.21.15.94:0/2420254922 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d68005470 con 0x7f3d740c0470 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d740aa300 con 0x7f3d740c0470 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d667fc640 1 -- 172.21.15.94:0/2420254922 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68012870 con 0x7f3d740c0470 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.081+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7414d0c0 con 0x7f3d740c0470 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.082+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7404f330 con 0x7f3d740c0470 2024-03-06T07:25:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.082+0000 7f3d667fc640 1 -- 172.21.15.94:0/2420254922 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d68012a10 con 0x7f3d740c0470 2024-03-06T07:25:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.083+0000 7f3d667fc640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d5806b4d0 0x7f3d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.083+0000 7f3d667fc640 1 -- 172.21.15.94:0/2420254922 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d68089e60 con 0x7f3d740c0470 2024-03-06T07:25:25.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.083+0000 7f3d79050640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d5806b4d0 0x7f3d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.083+0000 7f3d79050640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d5806b4d0 0x7f3d5806d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3d70036000 tx=0x7f3d70008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.087+0000 7f3d667fc640 1 -- 172.21.15.94:0/2420254922 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d68058be0 con 0x7f3d740c0470 2024-03-06T07:25:25.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.257+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d7406bc70 con 0x7f3d740c0470 2024-03-06T07:25:25.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.257+0000 7f3d667fc640 1 -- 172.21.15.94:0/2420254922 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v170) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3d680584e0 con 0x7f3d740c0470 2024-03-06T07:25:25.259 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:25:25.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d5806b4d0 msgr2=0x7f3d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d5806b4d0 0x7f3d5806d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3d70036000 tx=0x7f3d70008030 comp rx=0 tx=0).stop 2024-03-06T07:25:25.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c0470 msgr2=0x7f3d741edf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c0470 0x7f3d741edf90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c450 tx=0x7f3d6800c910 comp rx=0 tx=0).stop 2024-03-06T07:25:25.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 shutdown_connections 2024-03-06T07:25:25.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3d5806b4d0 0x7f3d5806d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3d740c2aa0 0x7f3d74147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3d740c26c0 0x7f3d74146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 --2- 172.21.15.94:0/2420254922 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d740c0470 0x7f3d741edf90 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 >> 172.21.15.94:0/2420254922 conn(0x7f3d740c3b40 msgr2=0x7f3d74152120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 shutdown_connections 2024-03-06T07:25:25.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.259+0000 7f3d7bab5640 1 -- 172.21.15.94:0/2420254922 wait complete. 2024-03-06T07:25:25.271 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:25.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.370+0000 7fe96f156640 1 Processor -- start 2024-03-06T07:25:25.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.370+0000 7fe96f156640 1 -- start start 2024-03-06T07:25:25.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.371+0000 7fe96f156640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe968159270 0x7fe968155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.371+0000 7fe96f156640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680c73d0 con 0x7fe96814f360 2024-03-06T07:25:25.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.371+0000 7fe96f156640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680be2e0 con 0x7fe9680c88c0 2024-03-06T07:25:25.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.371+0000 7fe96f156640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe968159700 con 0x7fe968159270 2024-03-06T07:25:25.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.371+0000 7fe96d6f3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe968159270 0x7fe968155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.371+0000 7fe96cef2640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe96814f360 0x7fe9680c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49814/0 (socket says 172.21.15.94:49814) 2024-03-06T07:25:25.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.371+0000 7fe96cef2640 1 -- 172.21.15.94:0/146458234 learned_addr learned my addr 172.21.15.94:0/146458234 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:25.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.372+0000 7fe96d6f3640 1 -- 172.21.15.94:0/146458234 >> v1:172.21.15.145:6789/0 conn(0x7fe9680c88c0 legacy=0x7fe968158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.372+0000 7fe96d6f3640 1 -- 172.21.15.94:0/146458234 >> v1:172.21.15.94:6789/0 conn(0x7fe96814f360 legacy=0x7fe9680c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.372+0000 7fe96d6f3640 1 -- 172.21.15.94:0/146458234 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680c7670 con 0x7fe968159270 2024-03-06T07:25:25.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.372+0000 7fe96d6f3640 1 --2- 172.21.15.94:0/146458234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe968155c10 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fe960009d90 tx=0x7fe96002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d67e8592c14854 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.372+0000 7fe9677fe640 1 -- 172.21.15.94:0/146458234 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960039070 con 0x7fe968159270 2024-03-06T07:25:25.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.372+0000 7fe9677fe640 1 -- 172.21.15.94:0/146458234 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96002cc60 con 0x7fe968159270 2024-03-06T07:25:25.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.372+0000 7fe9677fe640 1 -- 172.21.15.94:0/146458234 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960034660 con 0x7fe968159270 2024-03-06T07:25:25.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.373+0000 7fe96f156640 1 -- 172.21.15.94:0/146458234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 msgr2=0x7fe968155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.373+0000 7fe96f156640 1 --2- 172.21.15.94:0/146458234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe968155c10 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fe960009d90 tx=0x7fe96002beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.373+0000 7fe96f156640 1 -- 172.21.15.94:0/146458234 shutdown_connections 2024-03-06T07:25:25.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.373+0000 7fe96f156640 1 --2- 172.21.15.94:0/146458234 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe968155c10 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.373+0000 7fe96f156640 1 -- 172.21.15.94:0/146458234 >> 172.21.15.94:0/146458234 conn(0x7fe9680ac570 msgr2=0x7fe9680ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.373+0000 7fe96f156640 1 -- 172.21.15.94:0/146458234 shutdown_connections 2024-03-06T07:25:25.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.373+0000 7fe96f156640 1 -- 172.21.15.94:0/146458234 wait complete. 2024-03-06T07:25:25.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.374+0000 7fe96f156640 1 Processor -- start 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.374+0000 7fe96f156640 1 -- start start 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96f156640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9680c88c0 0x7fe9681548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96f156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe96814f360 0x7fe9681eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96f156640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe9681eb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96f156640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680c7c30 con 0x7fe968159270 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96f156640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680c5a50 con 0x7fe9680c88c0 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96f156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680c5bd0 con 0x7fe96814f360 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe9681eb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe9681eb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60574/0 (socket says 172.21.15.94:60574) 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 -- 172.21.15.94:0/1097817395 learned_addr learned my addr 172.21.15.94:0/1097817395 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe967fff640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe96814f360 0x7fe9681eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96cef2640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9680c88c0 0x7fe9681548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 -- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe96814f360 msgr2=0x7fe9681eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe96814f360 0x7fe9681eae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 -- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9680c88c0 msgr2=0x7fe9681548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9680c88c0 0x7fe9681548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 -- 172.21.15.94:0/1097817395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9681ee550 con 0x7fe968159270 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.375+0000 7fe96d6f3640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe9681eb380 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fe960009d90 tx=0x7fe960004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.376+0000 7fe9657fa640 1 -- 172.21.15.94:0/1097817395 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960039050 con 0x7fe968159270 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.376+0000 7fe9657fa640 1 -- 172.21.15.94:0/1097817395 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe960045070 con 0x7fe968159270 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.376+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9681ee830 con 0x7fe968159270 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.376+0000 7fe9657fa640 1 -- 172.21.15.94:0/1097817395 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96003e830 con 0x7fe968159270 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.376+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe968154de0 con 0x7fe968159270 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.377+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96804f330 con 0x7fe968159270 2024-03-06T07:25:25.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.377+0000 7fe9657fa640 1 -- 172.21.15.94:0/1097817395 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe96003e9d0 con 0x7fe968159270 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.377+0000 7fe9657fa640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe95406b4d0 0x7fe95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.378+0000 7fe9657fa640 1 -- 172.21.15.94:0/1097817395 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe9600ad090 con 0x7fe968159270 2024-03-06T07:25:25.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.378+0000 7fe96cef2640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe95406b4d0 0x7fe95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.382 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.378+0000 7fe96cef2640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe95406b4d0 0x7fe95406d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe958005880 tx=0x7fe958009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.382+0000 7fe9657fa640 1 -- 172.21.15.94:0/1097817395 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe96007b8e0 con 0x7fe968159270 2024-03-06T07:25:25.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.590+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9680c8d50 con 0x7fe968159270 2024-03-06T07:25:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.590+0000 7fe9657fa640 1 -- 172.21.15.94:0/1097817395 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4998 (secure 0 0 0) 0x7fe96007f980 con 0x7fe968159270 2024-03-06T07:25:25.592 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:25.592 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:21.282376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:active","state_seq":2,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37613,"qdb_cluster":[37613]},"id":13}]} 2024-03-06T07:25:25.592 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 165 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe95406b4d0 msgr2=0x7fe95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe95406b4d0 0x7fe95406d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe958005880 tx=0x7fe958009040 comp rx=0 tx=0).stop 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 msgr2=0x7fe9681eb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe9681eb380 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fe960009d90 tx=0x7fe960004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 shutdown_connections 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe95406b4d0 0x7fe95406d990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe968159270 0x7fe9681eb380 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe96814f360 0x7fe9681eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 --2- 172.21.15.94:0/1097817395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9680c88c0 0x7fe9681548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 >> 172.21.15.94:0/1097817395 conn(0x7fe9680ac570 msgr2=0x7fe9680ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 shutdown_connections 2024-03-06T07:25:25.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.592+0000 7fe96f156640 1 -- 172.21.15.94:0/1097817395 wait complete. 2024-03-06T07:25:25.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.703+0000 7fcf10f1e640 1 Processor -- start 2024-03-06T07:25:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.703+0000 7fcf10f1e640 1 -- start start 2024-03-06T07:25:25.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf10f1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c14f790 0x7fcf0c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf10f1e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0c0c28d0 con 0x7fcf0c155a40 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf10f1e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0c0c2bb0 con 0x7fcf0c0c1aa0 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf10f1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0c2680 con 0x7fcf0c14f790 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcf0c155a40 0x7fcf0c155ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49820/0 (socket says 172.21.15.94:49820) 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0ad9d640 1 -- 172.21.15.94:0/2721818781 learned_addr learned my addr 172.21.15.94:0/2721818781 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0a59c640 1 --2- 172.21.15.94:0/2721818781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c14f790 0x7fcf0c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0959a640 1 -- 172.21.15.94:0/2721818781 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861995768 0 0) 0x7fcf0c0c28d0 con 0x7fcf0c155a40 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0959a640 1 -- 172.21.15.94:0/2721818781 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef4003520 con 0x7fcf0c155a40 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0959a640 1 -- 172.21.15.94:0/2721818781 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873160915 0 0) 0x7fcf0c0c2bb0 con 0x7fcf0c0c1aa0 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0959a640 1 -- 172.21.15.94:0/2721818781 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf0c0c28d0 con 0x7fcf0c0c1aa0 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0a59c640 1 -- 172.21.15.94:0/2721818781 >> v1:172.21.15.145:6789/0 conn(0x7fcf0c0c1aa0 legacy=0x7fcf0c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0a59c640 1 -- 172.21.15.94:0/2721818781 >> v1:172.21.15.94:6789/0 conn(0x7fcf0c155a40 legacy=0x7fcf0c155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.704+0000 7fcf0a59c640 1 -- 172.21.15.94:0/2721818781 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c0c2e90 con 0x7fcf0c14f790 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.705+0000 7fcf0a59c640 1 --2- 172.21.15.94:0/2721818781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c14f790 0x7fcf0c1529d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097d0 tx=0x7fcefc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d657290e8f5bde8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.705+0000 7fcf0959a640 1 -- 172.21.15.94:0/2721818781 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc035020 con 0x7fcf0c14f790 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.705+0000 7fcf0959a640 1 -- 172.21.15.94:0/2721818781 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc031890 con 0x7fcf0c14f790 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.705+0000 7fcf0959a640 1 -- 172.21.15.94:0/2721818781 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc031c40 con 0x7fcf0c14f790 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.706+0000 7fcf10f1e640 1 -- 172.21.15.94:0/2721818781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c14f790 msgr2=0x7fcf0c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.706+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/2721818781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c14f790 0x7fcf0c1529d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097d0 tx=0x7fcefc02b930 comp rx=0 tx=0).stop 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.706+0000 7fcf10f1e640 1 -- 172.21.15.94:0/2721818781 shutdown_connections 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.706+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/2721818781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c14f790 0x7fcf0c1529d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.706+0000 7fcf10f1e640 1 -- 172.21.15.94:0/2721818781 >> 172.21.15.94:0/2721818781 conn(0x7fcf0c0c6710 msgr2=0x7fcf0c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.706+0000 7fcf10f1e640 1 -- 172.21.15.94:0/2721818781 shutdown_connections 2024-03-06T07:25:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.706+0000 7fcf10f1e640 1 -- 172.21.15.94:0/2721818781 wait complete. 2024-03-06T07:25:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.707+0000 7fcf10f1e640 1 Processor -- start 2024-03-06T07:25:25.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.707+0000 7fcf10f1e640 1 -- start start 2024-03-06T07:25:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf10f1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c0c1aa0 0x7fcf0c14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf10f1e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf0c14f790 0x7fcf0c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf10f1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf0c155a40 0x7fcf0c070df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf10f1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0c5e30 con 0x7fcf0c155a40 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf10f1e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0c0d60 con 0x7fcf0c14f790 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf10f1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c0c5fb0 con 0x7fcf0c0c1aa0 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf0c155a40 0x7fcf0c070df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf0c155a40 0x7fcf0c070df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60580/0 (socket says 172.21.15.94:60580) 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 -- 172.21.15.94:0/993827396 learned_addr learned my addr 172.21.15.94:0/993827396 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf09d9b640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf0c14f790 0x7fcf0c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0a59c640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c0c1aa0 0x7fcf0c14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 -- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c0c1aa0 msgr2=0x7fcf0c14dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c0c1aa0 0x7fcf0c14dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 -- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf0c14f790 msgr2=0x7fcf0c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf0c14f790 0x7fcf0c0708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.708+0000 7fcf0ad9d640 1 -- 172.21.15.94:0/993827396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c0a7930 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.709+0000 7fcf0ad9d640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf0c155a40 0x7fcf0c070df0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400caf0 tx=0x7fcf0400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.709+0000 7fcefaffd640 1 -- 172.21.15.94:0/993827396 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04017070 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.709+0000 7fcefaffd640 1 -- 172.21.15.94:0/993827396 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0400aca0 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.709+0000 7fcefaffd640 1 -- 172.21.15.94:0/993827396 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04012620 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.709+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c072d70 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.709+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c073010 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.710+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c04f330 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.710+0000 7fcefaffd640 1 -- 172.21.15.94:0/993827396 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf0401a070 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.711+0000 7fcefaffd640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fceec06b500 0x7fceec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.711+0000 7fcefaffd640 1 -- 172.21.15.94:0/993827396 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf0408a0a0 con 0x7fcf0c155a40 2024-03-06T07:25:25.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.711+0000 7fcf0a59c640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fceec06b500 0x7fceec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.711+0000 7fcf0a59c640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fceec06b500 0x7fceec06d9c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fcefc036000 tx=0x7fcefc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.715+0000 7fcefaffd640 1 -- 172.21.15.94:0/993827396 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf0c0a7930 con 0x7fcf0c155a40 2024-03-06T07:25:25.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.887+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf0c0c1f40 con 0x7fcf0c155a40 2024-03-06T07:25:25.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.888+0000 7fcefaffd640 1 -- 172.21.15.94:0/993827396 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+16952 (secure 0 0 0) 0x7fcf040586c0 con 0x7fcf0c155a40 2024-03-06T07:25:25.890 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:25.891 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":170,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:57.969058+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:54.539968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:55.288852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.890+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fceec06b500 msgr2=0x7fceec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.890+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fceec06b500 0x7fceec06d9c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fcefc036000 tx=0x7fcefc008030 comp rx=0 tx=0).stop 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf0c155a40 msgr2=0x7fcf0c070df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf0c155a40 0x7fcf0c070df0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400caf0 tx=0x7fcf0400a060 comp rx=0 tx=0).stop 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 shutdown_connections 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fceec06b500 0x7fceec06d9c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf0c155a40 0x7fcf0c070df0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf0c14f790 0x7fcf0c0708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 --2- 172.21.15.94:0/993827396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf0c0c1aa0 0x7fcf0c14dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 >> 172.21.15.94:0/993827396 conn(0x7fcf0c0c6710 msgr2=0x7fcf0c0a9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 shutdown_connections 2024-03-06T07:25:25.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:25.891+0000 7fcf10f1e640 1 -- 172.21.15.94:0/993827396 wait complete. 2024-03-06T07:25:25.903 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:25.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:26.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.003+0000 7f2940429640 1 Processor -- start 2024-03-06T07:25:26.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.003+0000 7f2940429640 1 -- start start 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.003+0000 7f2940429640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29381529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.003+0000 7f2940429640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29380c1aa0 0x7f29380c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.003+0000 7f2940429640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29380c5c60 0x7f2938155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f2940429640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29380c2680 con 0x7f29380c5c60 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f2940429640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29380c28d0 con 0x7f29380c1aa0 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f2940429640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29380c2bb0 con 0x7f293814f790 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29380c5c60 0x7f2938155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29380c5c60 0x7f2938155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60590/0 (socket says 172.21.15.94:60590) 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e1c5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29381529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293d9c4640 1 --2- 172.21.15.94:0/1327537510 >> v2:172.21.15.145:3300/0 conn(0x7f29380c1aa0 0x7f29380c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 -- 172.21.15.94:0/1327537510 learned_addr learned my addr 172.21.15.94:0/1327537510 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 -- 172.21.15.94:0/1327537510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 msgr2=0x7f29381529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 --2- 172.21.15.94:0/1327537510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29381529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 -- 172.21.15.94:0/1327537510 >> v2:172.21.15.145:3300/0 conn(0x7f29380c1aa0 msgr2=0x7f29380c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 --2- 172.21.15.94:0/1327537510 >> v2:172.21.15.145:3300/0 conn(0x7f29380c1aa0 0x7f29380c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e9c6640 1 -- 172.21.15.94:0/1327537510 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29380c2e90 con 0x7f29380c5c60 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293e1c5640 1 --2- 172.21.15.94:0/1327537510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29381529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:26.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.004+0000 7f293d9c4640 1 --2- 172.21.15.94:0/1327537510 >> v2:172.21.15.145:3300/0 conn(0x7f29380c1aa0 0x7f29380c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:26.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f293e9c6640 1 --2- 172.21.15.94:0/1327537510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c5c60 0x7f2938155a40 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f29380c76f0 tx=0x7f293002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba9e4620efdee32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f293d1c3640 1 -- 172.21.15.94:0/1327537510 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293002ca70 con 0x7f29380c5c60 2024-03-06T07:25:26.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f293d1c3640 1 -- 172.21.15.94:0/1327537510 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293002cc10 con 0x7f29380c5c60 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f293d1c3640 1 -- 172.21.15.94:0/1327537510 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29300343f0 con 0x7f29380c5c60 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f2940429640 1 -- 172.21.15.94:0/1327537510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c5c60 msgr2=0x7f2938155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f2940429640 1 --2- 172.21.15.94:0/1327537510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c5c60 0x7f2938155a40 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f29380c76f0 tx=0x7f293002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f2940429640 1 -- 172.21.15.94:0/1327537510 shutdown_connections 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f2940429640 1 --2- 172.21.15.94:0/1327537510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c5c60 0x7f2938155a40 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f2940429640 1 --2- 172.21.15.94:0/1327537510 >> v2:172.21.15.145:3300/0 conn(0x7f29380c1aa0 0x7f29380c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f2940429640 1 --2- 172.21.15.94:0/1327537510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29381529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.005+0000 7f2940429640 1 -- 172.21.15.94:0/1327537510 >> 172.21.15.94:0/1327537510 conn(0x7f29380c6710 msgr2=0x7f29380c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.006+0000 7f2940429640 1 -- 172.21.15.94:0/1327537510 shutdown_connections 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.006+0000 7f2940429640 1 -- 172.21.15.94:0/1327537510 wait complete. 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 Processor -- start 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 -- start start 2024-03-06T07:25:26.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c1aa0 0x7f2938071160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29380c5c60 0x7f29381de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29380a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293806cc00 con 0x7f29380c1aa0 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29380c0d60 con 0x7f29380c5c60 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f2940429640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293806cd80 con 0x7f293814f790 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f293e1c5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c1aa0 0x7f2938071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f293e1c5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c1aa0 0x7f2938071160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60598/0 (socket says 172.21.15.94:60598) 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f293e1c5640 1 -- 172.21.15.94:0/839811355 learned_addr learned my addr 172.21.15.94:0/839811355 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f293e9c6640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29380a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.007+0000 7f293d9c4640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29380c5c60 0x7f29381de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f293e1c5640 1 -- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 msgr2=0x7f29380a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f293e1c5640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29380a9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f293e1c5640 1 -- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29380c5c60 msgr2=0x7f29381de4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f293e1c5640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29380c5c60 0x7f29381de4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f293e1c5640 1 -- 172.21.15.94:0/839811355 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293814cb20 con 0x7f29380c1aa0 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f293e1c5640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c1aa0 0x7f2938071160 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f293400e990 tx=0x7f293400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f292effd640 1 -- 172.21.15.94:0/839811355 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934017070 con 0x7f29380c1aa0 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f292effd640 1 -- 172.21.15.94:0/839811355 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293400c9e0 con 0x7f29380c1aa0 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293814ce00 con 0x7f29380c1aa0 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f292effd640 1 -- 172.21.15.94:0/839811355 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293400cdb0 con 0x7f29380c1aa0 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.008+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29380716a0 con 0x7f29380c1aa0 2024-03-06T07:25:26.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.009+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293804f330 con 0x7f29380c1aa0 2024-03-06T07:25:26.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.009+0000 7f292effd640 1 -- 172.21.15.94:0/839811355 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f293401a070 con 0x7f29380c1aa0 2024-03-06T07:25:26.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.010+0000 7f292effd640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f291c06b500 0x7f291c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.010+0000 7f292effd640 1 -- 172.21.15.94:0/839811355 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f293408a450 con 0x7f29380c1aa0 2024-03-06T07:25:26.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.010+0000 7f293d9c4640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f291c06b500 0x7f291c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.010+0000 7f293d9c4640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f291c06b500 0x7f291c06d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f2928009560 tx=0x7f2928009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.014+0000 7f292effd640 1 -- 172.21.15.94:0/839811355 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2934058b80 con 0x7f29380c1aa0 2024-03-06T07:25:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.222+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29380c1f40 con 0x7f29380c1aa0 2024-03-06T07:25:26.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.223+0000 7f292effd640 1 -- 172.21.15.94:0/839811355 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f293405cc20 con 0x7f29380c1aa0 2024-03-06T07:25:26.225 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:26.225 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:21.282376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:active","state_seq":2,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37613,"qdb_cluster":[37613]},"id":13}]} 2024-03-06T07:25:26.225 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 165 2024-03-06T07:25:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f291c06b500 msgr2=0x7f291c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f291c06b500 0x7f291c06d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f2928009560 tx=0x7f2928009190 comp rx=0 tx=0).stop 2024-03-06T07:25:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c1aa0 msgr2=0x7f2938071160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c1aa0 0x7f2938071160 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f293400e990 tx=0x7f293400ee50 comp rx=0 tx=0).stop 2024-03-06T07:25:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 shutdown_connections 2024-03-06T07:25:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f291c06b500 0x7f291c06d9c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293814f790 0x7f29380a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29380c5c60 0x7f29381de4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 --2- 172.21.15.94:0/839811355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29380c1aa0 0x7f2938071160 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 >> 172.21.15.94:0/839811355 conn(0x7f29380c6710 msgr2=0x7f29381520d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 shutdown_connections 2024-03-06T07:25:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.225+0000 7f2940429640 1 -- 172.21.15.94:0/839811355 wait complete. 2024-03-06T07:25:26.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:26.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.337+0000 7f223cd8a640 1 Processor -- start 2024-03-06T07:25:26.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.337+0000 7f223cd8a640 1 -- start start 2024-03-06T07:25:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223cd8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f22380c1aa0 0x7f22381529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223cd8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2238155a40 0x7f2238155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223cd8a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22381563d0 con 0x7f223814f790 2024-03-06T07:25:26.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223cd8a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22380c2680 con 0x7f22380c1aa0 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223cd8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22380c28d0 con 0x7f2238155a40 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f223814f790 0x7f22380c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49834/0 (socket says 172.21.15.94:49834) 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223659c640 1 -- 172.21.15.94:0/3260224498 learned_addr learned my addr 172.21.15.94:0/3260224498 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f2235d9b640 1 --2- 172.21.15.94:0/3260224498 >> v2:172.21.15.94:3300/0 conn(0x7f22380c1aa0 0x7f22381529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f2236d9d640 1 --2- 172.21.15.94:0/3260224498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2238155a40 0x7f2238155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223559a640 1 -- 172.21.15.94:0/3260224498 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373472771 0 0) 0x7f22381563d0 con 0x7f223814f790 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f223559a640 1 -- 172.21.15.94:0/3260224498 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2220003520 con 0x7f223814f790 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f2235d9b640 1 -- 172.21.15.94:0/3260224498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2238155a40 msgr2=0x7f2238155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f2235d9b640 1 --2- 172.21.15.94:0/3260224498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2238155a40 0x7f2238155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f2235d9b640 1 -- 172.21.15.94:0/3260224498 >> v1:172.21.15.94:6789/0 conn(0x7f223814f790 legacy=0x7f22380c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.338+0000 7f2235d9b640 1 -- 172.21.15.94:0/3260224498 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22380c2bb0 con 0x7f22380c1aa0 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.339+0000 7f2235d9b640 1 --2- 172.21.15.94:0/3260224498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22380c1aa0 0x7f22381529d0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f222c0097a0 tx=0x7f222c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d85e4f66cdfa8df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.339+0000 7f223559a640 1 -- 172.21.15.94:0/3260224498 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c035020 con 0x7f22380c1aa0 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.339+0000 7f223559a640 1 -- 172.21.15.94:0/3260224498 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c031830 con 0x7f22380c1aa0 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.339+0000 7f223559a640 1 -- 172.21.15.94:0/3260224498 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c031be0 con 0x7f22380c1aa0 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 -- 172.21.15.94:0/3260224498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22380c1aa0 msgr2=0x7f22381529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 --2- 172.21.15.94:0/3260224498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22380c1aa0 0x7f22381529d0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f222c0097a0 tx=0x7f222c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 -- 172.21.15.94:0/3260224498 shutdown_connections 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 --2- 172.21.15.94:0/3260224498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2238155a40 0x7f2238155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 --2- 172.21.15.94:0/3260224498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f22380c1aa0 0x7f22381529d0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 -- 172.21.15.94:0/3260224498 >> 172.21.15.94:0/3260224498 conn(0x7f22380c6710 msgr2=0x7f22380c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 -- 172.21.15.94:0/3260224498 shutdown_connections 2024-03-06T07:25:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.340+0000 7f223cd8a640 1 -- 172.21.15.94:0/3260224498 wait complete. 2024-03-06T07:25:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.341+0000 7f223cd8a640 1 Processor -- start 2024-03-06T07:25:26.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.341+0000 7f223cd8a640 1 -- start start 2024-03-06T07:25:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f223cd8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22380c1aa0 0x7f2238151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f223cd8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f223814f790 0x7f223814e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f223cd8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2238155a40 0x7f223814ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f223cd8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22380c0d60 con 0x7f2238155a40 2024-03-06T07:25:26.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f223cd8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2238155520 con 0x7f22380c1aa0 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f223cd8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22380c5e30 con 0x7f223814f790 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2238155a40 0x7f223814ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2238155a40 0x7f223814ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60610/0 (socket says 172.21.15.94:60610) 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 -- 172.21.15.94:0/4179717854 learned_addr learned my addr 172.21.15.94:0/4179717854 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2235d9b640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f223814f790 0x7f223814e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f223659c640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22380c1aa0 0x7f2238151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 -- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f223814f790 msgr2=0x7f223814e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f223814f790 0x7f223814e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 -- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22380c1aa0 msgr2=0x7f2238151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22380c1aa0 0x7f2238151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 -- 172.21.15.94:0/4179717854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238070470 con 0x7f2238155a40 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.342+0000 7f2236d9d640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2238155a40 0x7f223814ec80 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f223000efc0 tx=0x7f223000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.343+0000 7f2226ffd640 1 -- 172.21.15.94:0/4179717854 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230018020 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.343+0000 7f2226ffd640 1 -- 172.21.15.94:0/4179717854 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2230014480 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.343+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2238070710 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.343+0000 7f2226ffd640 1 -- 172.21.15.94:0/4179717854 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230014870 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.343+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2238070cd0 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.344+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223804f330 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.344+0000 7f2226ffd640 1 -- 172.21.15.94:0/4179717854 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2230005d10 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.344+0000 7f2226ffd640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f221806b4d0 0x7f221806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.345+0000 7f2226ffd640 1 -- 172.21.15.94:0/4179717854 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f223008ac50 con 0x7f2238155a40 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.345+0000 7f223659c640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f221806b4d0 0x7f221806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.345+0000 7f223659c640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f221806b4d0 0x7f221806d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2228005880 tx=0x7f2228009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.349+0000 7f2226ffd640 1 -- 172.21.15.94:0/4179717854 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2230090020 con 0x7f2238155a40 2024-03-06T07:25:26.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.557+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f223815c690 con 0x7f2238155a40 2024-03-06T07:25:26.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.558+0000 7f2226ffd640 1 -- 172.21.15.94:0/4179717854 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f2230022070 con 0x7f2238155a40 2024-03-06T07:25:26.560 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:26.560 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:56.327121+0000","modified":"2024-03-06T07:25:21.282376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37613":{"gid":37613,"name":"b","rank":0,"incarnation":164,"state":"up:active","state_seq":2,"addr":"172.21.15.145:6836/1183474661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1183474661},{"type":"v1","addr":"172.21.15.145:6836","nonce":1183474661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37613,"qdb_cluster":[37613]},"id":13}]} 2024-03-06T07:25:26.560 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 165 2024-03-06T07:25:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f221806b4d0 msgr2=0x7f221806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f221806b4d0 0x7f221806d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2228005880 tx=0x7f2228009040 comp rx=0 tx=0).stop 2024-03-06T07:25:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2238155a40 msgr2=0x7f223814ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2238155a40 0x7f223814ec80 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f223000efc0 tx=0x7f223000c490 comp rx=0 tx=0).stop 2024-03-06T07:25:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 shutdown_connections 2024-03-06T07:25:26.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f221806b4d0 0x7f221806d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2238155a40 0x7f223814ec80 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f223814f790 0x7f223814e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 --2- 172.21.15.94:0/4179717854 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22380c1aa0 0x7f2238151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 >> 172.21.15.94:0/4179717854 conn(0x7f22380c6710 msgr2=0x7f22380c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 shutdown_connections 2024-03-06T07:25:26.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.560+0000 7f223cd8a640 1 -- 172.21.15.94:0/4179717854 wait complete. 2024-03-06T07:25:26.573 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.670+0000 7fe01a17a640 1 Processor -- start 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.670+0000 7fe01a17a640 1 -- start start 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe01a17a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe01414f360 0x7fe014155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe01a17a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0140c73d0 con 0x7fe0140c88c0 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe01a17a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0140be2e0 con 0x7fe014158b60 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe01a17a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe014159700 con 0x7fe01414f360 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe012ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe0140c88c0 0x7fe0140c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49836/0 (socket says 172.21.15.94:49836) 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe012ffd640 1 -- 172.21.15.94:0/3334374486 learned_addr learned my addr 172.21.15.94:0/3334374486 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe0137fe640 1 --2- 172.21.15.94:0/3334374486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe01414f360 0x7fe014155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.671+0000 7fe0127fc640 1 -- 172.21.15.94:0/3334374486 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649119379 0 0) 0x7fe0140c73d0 con 0x7fe0140c88c0 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.672+0000 7fe0127fc640 1 -- 172.21.15.94:0/3334374486 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdffc003520 con 0x7fe0140c88c0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.672+0000 7fe0127fc640 1 -- 172.21.15.94:0/3334374486 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821137617 0 0) 0x7fe0140be2e0 con 0x7fe014158b60 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.672+0000 7fe0127fc640 1 -- 172.21.15.94:0/3334374486 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0140c73d0 con 0x7fe014158b60 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.672+0000 7fe0137fe640 1 -- 172.21.15.94:0/3334374486 >> v1:172.21.15.145:6789/0 conn(0x7fe014158b60 legacy=0x7fe014158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.672+0000 7fe0137fe640 1 -- 172.21.15.94:0/3334374486 >> v1:172.21.15.94:6789/0 conn(0x7fe0140c88c0 legacy=0x7fe0140c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.672+0000 7fe0137fe640 1 -- 172.21.15.94:0/3334374486 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140c7670 con 0x7fe01414f360 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.672+0000 7fe0137fe640 1 --2- 172.21.15.94:0/3334374486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe01414f360 0x7fe014155c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe0040097d0 tx=0x7fe00402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc51e01d555db5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe0127fc640 1 -- 172.21.15.94:0/3334374486 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004035020 con 0x7fe01414f360 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe0127fc640 1 -- 172.21.15.94:0/3334374486 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe004031800 con 0x7fe01414f360 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe0127fc640 1 -- 172.21.15.94:0/3334374486 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004031bd0 con 0x7fe01414f360 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe01a17a640 1 -- 172.21.15.94:0/3334374486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe01414f360 msgr2=0x7fe014155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe01a17a640 1 --2- 172.21.15.94:0/3334374486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe01414f360 0x7fe014155c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe0040097d0 tx=0x7fe00402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe01a17a640 1 -- 172.21.15.94:0/3334374486 shutdown_connections 2024-03-06T07:25:26.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe01a17a640 1 --2- 172.21.15.94:0/3334374486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe01414f360 0x7fe014155c10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe01a17a640 1 -- 172.21.15.94:0/3334374486 >> 172.21.15.94:0/3334374486 conn(0x7fe0140ac570 msgr2=0x7fe0140ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe01a17a640 1 -- 172.21.15.94:0/3334374486 shutdown_connections 2024-03-06T07:25:26.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.673+0000 7fe01a17a640 1 -- 172.21.15.94:0/3334374486 wait complete. 2024-03-06T07:25:26.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.674+0000 7fe01a17a640 1 Processor -- start 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.674+0000 7fe01a17a640 1 -- start start 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe01a17a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0140c88c0 0x7fe0140ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe01a17a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe01414f360 0x7fe0141eecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe01a17a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe014158b60 0x7fe0140ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe01a17a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c7c30 con 0x7fe01414f360 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe01a17a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c5a50 con 0x7fe0140c88c0 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe01a17a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140c5bd0 con 0x7fe014158b60 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe01414f360 0x7fe0141eecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe01414f360 0x7fe0141eecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60622/0 (socket says 172.21.15.94:60622) 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 -- 172.21.15.94:0/2795417408 learned_addr learned my addr 172.21.15.94:0/2795417408 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe0137fe640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0140c88c0 0x7fe0140ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe013fff640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe014158b60 0x7fe0140ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 -- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe014158b60 msgr2=0x7fe0140ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe014158b60 0x7fe0140ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 -- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0140c88c0 msgr2=0x7fe0140ad750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0140c88c0 0x7fe0140ad750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.675+0000 7fe012ffd640 1 -- 172.21.15.94:0/2795417408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0140c3ea0 con 0x7fe01414f360 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.676+0000 7fe012ffd640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe01414f360 0x7fe0141eecc0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c3e0 tx=0x7fe00800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.676+0000 7fdffbfff640 1 -- 172.21.15.94:0/2795417408 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008017070 con 0x7fe01414f360 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.676+0000 7fdffbfff640 1 -- 172.21.15.94:0/2795417408 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008005ce0 con 0x7fe01414f360 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.676+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0140c4180 con 0x7fe01414f360 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.676+0000 7fdffbfff640 1 -- 172.21.15.94:0/2795417408 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080072d0 con 0x7fe01414f360 2024-03-06T07:25:26.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.676+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0140abd10 con 0x7fe01414f360 2024-03-06T07:25:26.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.677+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01404f330 con 0x7fe01414f360 2024-03-06T07:25:26.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.677+0000 7fdffbfff640 1 -- 172.21.15.94:0/2795417408 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe00801a070 con 0x7fe01414f360 2024-03-06T07:25:26.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.677+0000 7fdffbfff640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe00006b4d0 0x7fe00006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.678+0000 7fdffbfff640 1 -- 172.21.15.94:0/2795417408 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe0080895a0 con 0x7fe01414f360 2024-03-06T07:25:26.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.678+0000 7fe0137fe640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe00006b4d0 0x7fe00006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.678+0000 7fe0137fe640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe00006b4d0 0x7fe00006d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe004036000 tx=0x7fe004008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.682+0000 7fdffbfff640 1 -- 172.21.15.94:0/2795417408 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe008057d90 con 0x7fe01414f360 2024-03-06T07:25:26.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.857+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0140c8d50 con 0x7fe01414f360 2024-03-06T07:25:26.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.858+0000 7fdffbfff640 1 -- 172.21.15.94:0/2795417408 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+16952 (secure 0 0 0) 0x7fe00805e0e0 con 0x7fe01414f360 2024-03-06T07:25:26.860 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:26.861 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":170,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:24:57.969058+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:54.539968+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:55.288852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:26.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe00006b4d0 msgr2=0x7fe00006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe00006b4d0 0x7fe00006d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe004036000 tx=0x7fe004008030 comp rx=0 tx=0).stop 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe01414f360 msgr2=0x7fe0141eecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe01414f360 0x7fe0141eecc0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c3e0 tx=0x7fe00800c8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 shutdown_connections 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe00006b4d0 0x7fe00006d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe014158b60 0x7fe0140ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe01414f360 0x7fe0141eecc0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 --2- 172.21.15.94:0/2795417408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0140c88c0 0x7fe0140ad750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.860+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 >> 172.21.15.94:0/2795417408 conn(0x7fe0140ac570 msgr2=0x7fe014153870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.861+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 shutdown_connections 2024-03-06T07:25:26.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.861+0000 7fe01a17a640 1 -- 172.21.15.94:0/2795417408 wait complete. 2024-03-06T07:25:26.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:26.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.973+0000 7feed791e640 1 Processor -- start 2024-03-06T07:25:26.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.973+0000 7feed791e640 1 -- start start 2024-03-06T07:25:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed791e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed006e0a0 0x7feed0153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed791e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed0070550 con 0x7feed0072f00 2024-03-06T07:25:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed791e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed0070830 con 0x7feed00c0fa0 2024-03-06T07:25:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed791e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed0070b10 con 0x7feed006e0a0 2024-03-06T07:25:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed56ba640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feed0072f00 0x7feed00c6840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49846/0 (socket says 172.21.15.94:49846) 2024-03-06T07:25:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed56ba640 1 -- 172.21.15.94:0/3110634254 learned_addr learned my addr 172.21.15.94:0/3110634254 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:26.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed5ebb640 1 --2- 172.21.15.94:0/3110634254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed006e0a0 0x7feed0153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feec7fff640 1 -- 172.21.15.94:0/3110634254 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100787028 0 0) 0x7feed0070550 con 0x7feed0072f00 2024-03-06T07:25:26.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feec7fff640 1 -- 172.21.15.94:0/3110634254 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feebc003520 con 0x7feed0072f00 2024-03-06T07:25:26.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feec7fff640 1 -- 172.21.15.94:0/3110634254 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321460180 0 0) 0x7feed0070830 con 0x7feed00c0fa0 2024-03-06T07:25:26.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feec7fff640 1 -- 172.21.15.94:0/3110634254 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed0070550 con 0x7feed00c0fa0 2024-03-06T07:25:26.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed5ebb640 1 -- 172.21.15.94:0/3110634254 >> v1:172.21.15.145:6789/0 conn(0x7feed00c0fa0 legacy=0x7feed006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed5ebb640 1 -- 172.21.15.94:0/3110634254 >> v1:172.21.15.94:6789/0 conn(0x7feed0072f00 legacy=0x7feed00c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.974+0000 7feed5ebb640 1 -- 172.21.15.94:0/3110634254 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed00732e0 con 0x7feed006e0a0 2024-03-06T07:25:26.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.975+0000 7feed5ebb640 1 --2- 172.21.15.94:0/3110634254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed006e0a0 0x7feed0153990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feec8009dc0 tx=0x7feec802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c364f0fda282bab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.975+0000 7feec7fff640 1 -- 172.21.15.94:0/3110634254 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8039070 con 0x7feed006e0a0 2024-03-06T07:25:26.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.975+0000 7feec7fff640 1 -- 172.21.15.94:0/3110634254 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec802ccd0 con 0x7feed006e0a0 2024-03-06T07:25:26.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.975+0000 7feec7fff640 1 -- 172.21.15.94:0/3110634254 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8034660 con 0x7feed006e0a0 2024-03-06T07:25:26.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.975+0000 7feed791e640 1 -- 172.21.15.94:0/3110634254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed006e0a0 msgr2=0x7feed0153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.975+0000 7feed791e640 1 --2- 172.21.15.94:0/3110634254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed006e0a0 0x7feed0153990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feec8009dc0 tx=0x7feec802bf20 comp rx=0 tx=0).stop 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.976+0000 7feed791e640 1 -- 172.21.15.94:0/3110634254 shutdown_connections 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.976+0000 7feed791e640 1 --2- 172.21.15.94:0/3110634254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed006e0a0 0x7feed0153990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.976+0000 7feed791e640 1 -- 172.21.15.94:0/3110634254 >> 172.21.15.94:0/3110634254 conn(0x7feed006b1a0 msgr2=0x7feed006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.976+0000 7feed791e640 1 -- 172.21.15.94:0/3110634254 shutdown_connections 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.976+0000 7feed791e640 1 -- 172.21.15.94:0/3110634254 wait complete. 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 Processor -- start 2024-03-06T07:25:26.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 -- start start 2024-03-06T07:25:26.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feed006e0a0 0x7feed01ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feed0072f00 0x7feed00c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed00c0fa0 0x7feed00ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed0070f70 con 0x7feed006e0a0 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed0070df0 con 0x7feed0072f00 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed791e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed0072400 con 0x7feed00c0fa0 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed56ba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feed006e0a0 0x7feed01ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed56ba640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feed006e0a0 0x7feed01ee050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60634/0 (socket says 172.21.15.94:60634) 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.977+0000 7feed56ba640 1 -- 172.21.15.94:0/3390808180 learned_addr learned my addr 172.21.15.94:0/3390808180 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed5ebb640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed00c0fa0 0x7feed00ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed4eb9640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feed0072f00 0x7feed00c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed56ba640 1 -- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed00c0fa0 msgr2=0x7feed00ab070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed56ba640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed00c0fa0 0x7feed00ab070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed56ba640 1 -- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feed0072f00 msgr2=0x7feed00c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed56ba640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feed0072f00 0x7feed00c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed56ba640 1 -- 172.21.15.94:0/3390808180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0157250 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed56ba640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feed006e0a0 0x7feed01ee050 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c500 tx=0x7feecc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feec5ffb640 1 -- 172.21.15.94:0/3390808180 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc017070 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feec5ffb640 1 -- 172.21.15.94:0/3390808180 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc005ce0 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed0156fb0 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feec5ffb640 1 -- 172.21.15.94:0/3390808180 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc0072f0 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.978+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed00aaaa0 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.979+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed004f330 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.980+0000 7feec5ffb640 1 -- 172.21.15.94:0/3390808180 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feecc01a070 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.980+0000 7feec5ffb640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feeb406b4d0 0x7feeb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.980+0000 7feec5ffb640 1 -- 172.21.15.94:0/3390808180 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feecc089710 con 0x7feed006e0a0 2024-03-06T07:25:26.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.980+0000 7feed4eb9640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feeb406b4d0 0x7feeb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.981+0000 7feed4eb9640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feeb406b4d0 0x7feeb406d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feec0007fd0 tx=0x7feec0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:26.984+0000 7feec5ffb640 1 -- 172.21.15.94:0/3390808180 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feecc057dc0 con 0x7feed006e0a0 2024-03-06T07:25:27.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.191+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7feed00abf40 con 0x7feed006e0a0 2024-03-06T07:25:27.193 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:25:27.192+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:27.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.193+0000 7feec5ffb640 1 -- 172.21.15.94:0/3390808180 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v166) v1 ==== 161+0+0 (secure 0 0 0) 0x7feecc05be60 con 0x7feed006e0a0 2024-03-06T07:25:27.195 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:27.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feeb406b4d0 msgr2=0x7feeb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feeb406b4d0 0x7feeb406d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7feec0007fd0 tx=0x7feec0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feed006e0a0 msgr2=0x7feed01ee050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feed006e0a0 0x7feed01ee050 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c500 tx=0x7feecc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 shutdown_connections 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7feeb406b4d0 0x7feeb406d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feed00c0fa0 0x7feed00ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feed0072f00 0x7feed00c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 --2- 172.21.15.94:0/3390808180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feed006e0a0 0x7feed01ee050 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 >> 172.21.15.94:0/3390808180 conn(0x7feed006b1a0 msgr2=0x7feed006d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 shutdown_connections 2024-03-06T07:25:27.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.195+0000 7feed791e640 1 -- 172.21.15.94:0/3390808180 wait complete. 2024-03-06T07:25:27.199 INFO:tasks.ceph.mds.b.smithi145.stderr: -95> 2024-03-06T07:25:24.270+0000 7f02403a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:27.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:27.221 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:25:27.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.309+0000 7f864d24e640 1 Processor -- start 2024-03-06T07:25:27.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.309+0000 7f864d24e640 1 -- start start 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f864d24e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 0x7f8648155210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f864d24e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86481596f0 con 0x7f86481527f0 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f864d24e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86480c82a0 con 0x7f8648155750 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f864d24e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86480c9130 con 0x7f86480c88f0 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8646d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f86481527f0 0x7f86480c5370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49860/0 (socket says 172.21.15.94:49860) 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8646d9d640 1 -- 172.21.15.94:0/970275823 learned_addr learned my addr 172.21.15.94:0/970275823 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724059742 0 0) 0x7f86481596f0 con 0x7f86481527f0 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8634003520 con 0x7f86481527f0 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f864659c640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 0x7f8648155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1910714688 0 0) 0x7f8634003520 con 0x7f86481527f0 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 msgr2=0x7f8648155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 0x7f8648155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 >> v1:172.21.15.145:6789/0 conn(0x7f8648155750 legacy=0x7f864815fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8648158930 con 0x7f86481527f0 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f86400019f0 con 0x7f86481527f0 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 >> v1:172.21.15.94:6789/0 conn(0x7f86481527f0 legacy=0x7f86480c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.310+0000 7f8645d9b640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86340039e0 0x7f8634003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8645d9b640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8634004800 0x7f8634004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8645d9b640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 0x7f863400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86480c9130 con 0x7f86340039e0 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8634005570 con 0x7f8634004800 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863400bf00 con 0x7f86480c88f0 2024-03-06T07:25:27.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8646d9d640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86340039e0 0x7f8634003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f864659c640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8634004800 0x7f8634004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f864759e640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 0x7f863400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8646d9d640 1 -- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 msgr2=0x7f863400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8646d9d640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86480c88f0 0x7f863400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8646d9d640 1 -- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8634004800 msgr2=0x7f8634004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8646d9d640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8634004800 0x7f8634004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.311+0000 7f8646d9d640 1 -- 172.21.15.94:0/970275823 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634003520 con 0x7f86340039e0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f8646d9d640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86340039e0 0x7f8634003e00 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f864000a850 tx=0x7f8634005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf6e7c8cdfbedf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000a4a0 con 0x7f86340039e0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640034650 con 0x7f86340039e0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f8645d9b640 1 -- 172.21.15.94:0/970275823 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640034a40 con 0x7f86340039e0 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f864d24e640 1 -- 172.21.15.94:0/970275823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86340039e0 msgr2=0x7f8634003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f864d24e640 1 --2- 172.21.15.94:0/970275823 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86340039e0 0x7f8634003e00 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f864000a850 tx=0x7f8634005830 comp rx=0 tx=0).stop 2024-03-06T07:25:27.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f864759e640 1 -- 172.21.15.94:0/970275823 reap_dead start 2024-03-06T07:25:27.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f864d24e640 1 -- 172.21.15.94:0/970275823 shutdown_connections 2024-03-06T07:25:27.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f864d24e640 1 -- 172.21.15.94:0/970275823 >> 172.21.15.94:0/970275823 conn(0x7f86480ac580 msgr2=0x7f86480ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f864d24e640 1 -- 172.21.15.94:0/970275823 shutdown_connections 2024-03-06T07:25:27.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.312+0000 7f864d24e640 1 -- 172.21.15.94:0/970275823 wait complete. 2024-03-06T07:25:27.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 Processor -- start 2024-03-06T07:25:27.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 -- start start 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86340039e0 0x7f86481578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86480c0ff0 0x7f864814bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f864814c0c0 0x7f86481eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86480c6b90 con 0x7f864814c0c0 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8648159260 con 0x7f86480c0ff0 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864d24e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8648158ef0 con 0x7f86340039e0 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.314+0000 7f864759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f864814c0c0 0x7f86481eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f864814c0c0 0x7f86481eb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60650/0 (socket says 172.21.15.94:60650) 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 -- 172.21.15.94:0/3412673782 learned_addr learned my addr 172.21.15.94:0/3412673782 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864659c640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86480c0ff0 0x7f864814bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f8646d9d640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86340039e0 0x7f86481578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 -- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86340039e0 msgr2=0x7f86481578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86340039e0 0x7f86481578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 -- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86480c0ff0 msgr2=0x7f864814bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86480c0ff0 0x7f864814bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 -- 172.21.15.94:0/3412673782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8648155580 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864759e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f864814c0c0 0x7f86481eb280 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f863800f470 tx=0x7f863800f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f863f7fe640 1 -- 172.21.15.94:0/3412673782 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86380053a0 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f863f7fe640 1 -- 172.21.15.94:0/3412673782 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8638018070 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8648154ce0 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f863f7fe640 1 -- 172.21.15.94:0/3412673782 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638003830 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.315+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86481552a0 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.316+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864804f330 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.317+0000 7f863f7fe640 1 -- 172.21.15.94:0/3412673782 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86380039d0 con 0x7f864814c0c0 2024-03-06T07:25:27.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.317+0000 7f863f7fe640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f862806b500 0x7f862806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.317+0000 7f863f7fe640 1 -- 172.21.15.94:0/3412673782 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8638089c90 con 0x7f864814c0c0 2024-03-06T07:25:27.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.317+0000 7f8646d9d640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f862806b500 0x7f862806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.318+0000 7f8646d9d640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f862806b500 0x7f862806d9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8640034000 tx=0x7f864002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.321+0000 7f863f7fe640 1 -- 172.21.15.94:0/3412673782 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8638058370 con 0x7f864814c0c0 2024-03-06T07:25:27.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:27.529+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f864806b6f0 con 0x7f864814c0c0 2024-03-06T07:25:28.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.194+0000 7f863f7fe640 1 -- 172.21.15.94:0/3412673782 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v167) v1 ==== 104+0+0 (secure 0 0 0) 0x7f863805c410 con 0x7f864814c0c0 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f862806b500 msgr2=0x7f862806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f862806b500 0x7f862806d9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8640034000 tx=0x7f864002f000 comp rx=0 tx=0).stop 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f864814c0c0 msgr2=0x7f86481eb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f864814c0c0 0x7f86481eb280 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f863800f470 tx=0x7f863800f930 comp rx=0 tx=0).stop 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 shutdown_connections 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f862806b500 0x7f862806d9c0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f864814c0c0 0x7f86481eb280 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86480c0ff0 0x7f864814bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 --2- 172.21.15.94:0/3412673782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86340039e0 0x7f86481578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 >> 172.21.15.94:0/3412673782 conn(0x7f86480ac580 msgr2=0x7f86480c9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 shutdown_connections 2024-03-06T07:25:28.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.197+0000 7f864d24e640 1 -- 172.21.15.94:0/3412673782 wait complete. 2024-03-06T07:25:28.210 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:28.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.313+0000 7f4d6602a640 1 Processor -- start 2024-03-06T07:25:28.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.313+0000 7f4d6602a640 1 -- start start 2024-03-06T07:25:28.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d6602a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c1d10 0x7f4d60156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d6602a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d600c2920 0x7f4d60156980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d6602a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d60070430 con 0x7f4d600abb60 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d6602a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600711f0 con 0x7f4d600c2920 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d6602a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60070d60 con 0x7f4d600c1d10 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d5effd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c1d10 0x7f4d60156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d5ffff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d600c2920 0x7f4d60156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d5f7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f4d600abb60 0x7f4d601502f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:54836/0 (socket says 172.21.15.94:54836) 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.314+0000 7f4d5f7fe640 1 -- 172.21.15.94:0/3815261667 learned_addr learned my addr 172.21.15.94:0/3815261667 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.315+0000 7f4d5ffff640 1 -- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c1d10 msgr2=0x7f4d60156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.315+0000 7f4d5ffff640 1 --2- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c1d10 0x7f4d60156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.315+0000 7f4d5ffff640 1 -- 172.21.15.94:0/3815261667 >> v1:172.21.15.145:6789/0 conn(0x7f4d600abb60 legacy=0x7f4d601502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.315+0000 7f4d5e7fc640 1 -- 172.21.15.94:0/3815261667 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985022739 0 0) 0x7f4d60070430 con 0x7f4d600abb60 2024-03-06T07:25:28.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.315+0000 7f4d5ffff640 1 -- 172.21.15.94:0/3815261667 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60070710 con 0x7f4d600c2920 2024-03-06T07:25:28.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.315+0000 7f4d5effd640 1 --2- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c1d10 0x7f4d60156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:28.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d5ffff640 1 --2- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600c2920 0x7f4d60156980 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009d90 tx=0x7f4d5802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8887d799fa9a781b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d5e7fc640 1 -- 172.21.15.94:0/3815261667 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58039070 con 0x7f4d600c2920 2024-03-06T07:25:28.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d5e7fc640 1 -- 172.21.15.94:0/3815261667 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5802cbc0 con 0x7f4d600c2920 2024-03-06T07:25:28.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d5e7fc640 1 -- 172.21.15.94:0/3815261667 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58034440 con 0x7f4d600c2920 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 -- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600c2920 msgr2=0x7f4d60156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 --2- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600c2920 0x7f4d60156980 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009d90 tx=0x7f4d5802be10 comp rx=0 tx=0).stop 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 -- 172.21.15.94:0/3815261667 shutdown_connections 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 --2- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600c2920 0x7f4d60156980 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 --2- 172.21.15.94:0/3815261667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d600c1d10 0x7f4d60156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 -- 172.21.15.94:0/3815261667 >> 172.21.15.94:0/3815261667 conn(0x7f4d600c65b0 msgr2=0x7f4d600c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 -- 172.21.15.94:0/3815261667 shutdown_connections 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.316+0000 7f4d6602a640 1 -- 172.21.15.94:0/3815261667 wait complete. 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.317+0000 7f4d6602a640 1 Processor -- start 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d6602a640 1 -- start start 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d6602a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600abb60 0x7f4d601517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d6602a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c2920 0x7f4d6013c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d6602a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60147150 0x7f4d6013ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d6602a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60150b80 con 0x7f4d600c2920 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d6602a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60150a00 con 0x7f4d600abb60 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d6602a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60156ec0 con 0x7f4d60147150 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d5effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c2920 0x7f4d6013c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d5effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c2920 0x7f4d6013c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60654/0 (socket says 172.21.15.94:60654) 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.318+0000 7f4d5effd640 1 -- 172.21.15.94:0/229892403 learned_addr learned my addr 172.21.15.94:0/229892403 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5f7fe640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600abb60 0x7f4d601517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5ffff640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60147150 0x7f4d6013ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5effd640 1 -- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60147150 msgr2=0x7f4d6013ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5effd640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60147150 0x7f4d6013ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5effd640 1 -- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600abb60 msgr2=0x7f4d601517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5effd640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600abb60 0x7f4d601517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5effd640 1 -- 172.21.15.94:0/229892403 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600c6200 con 0x7f4d600c2920 2024-03-06T07:25:28.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5ffff640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60147150 0x7f4d6013ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d5effd640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c2920 0x7f4d6013c790 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400ebe0 tx=0x7f4d5400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d47fff640 1 -- 172.21.15.94:0/229892403 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54017070 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d47fff640 1 -- 172.21.15.94:0/229892403 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5400cc30 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d60155bf0 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d47fff640 1 -- 172.21.15.94:0/229892403 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54013470 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.319+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d60155e90 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.320+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6004f330 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.321+0000 7f4d47fff640 1 -- 172.21.15.94:0/229892403 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d5401a070 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.321+0000 7f4d47fff640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.321+0000 7f4d47fff640 1 -- 172.21.15.94:0/229892403 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4d5408a620 con 0x7f4d600c2920 2024-03-06T07:25:28.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.321+0000 7f4d5f7fe640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.322+0000 7f4d5f7fe640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4d50000f60 tx=0x7f4d50007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.325+0000 7f4d47fff640 1 -- 172.21.15.94:0/229892403 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d600c6200 con 0x7f4d600c2920 2024-03-06T07:25:28.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:28.497+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4d6014d560 con 0x7f4d600c2920 2024-03-06T07:25:29.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.306+0000 7f4d47fff640 1 -- 172.21.15.94:0/229892403 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v172) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4d54058c70 con 0x7f4d600c2920 2024-03-06T07:25:29.308 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:29.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4c06b500 msgr2=0x7f4d4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4d50000f60 tx=0x7f4d50007d50 comp rx=0 tx=0).stop 2024-03-06T07:25:29.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c2920 msgr2=0x7f4d6013c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c2920 0x7f4d6013c790 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400ebe0 tx=0x7f4d5400c520 comp rx=0 tx=0).stop 2024-03-06T07:25:29.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 shutdown_connections 2024-03-06T07:25:29.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60147150 0x7f4d6013ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d600c2920 0x7f4d6013c790 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 --2- 172.21.15.94:0/229892403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600abb60 0x7f4d601517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 >> 172.21.15.94:0/229892403 conn(0x7f4d600c65b0 msgr2=0x7f4d60072b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 shutdown_connections 2024-03-06T07:25:29.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.310+0000 7f4d6602a640 1 -- 172.21.15.94:0/229892403 wait complete. 2024-03-06T07:25:29.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.432+0000 7f936880e640 1 Processor -- start 2024-03-06T07:25:29.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.432+0000 7f936880e640 1 -- start start 2024-03-06T07:25:29.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f936880e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f93640abb60 0x7f93641502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f936880e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f93640c2920 0x7f9364156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f936880e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9364070430 con 0x7f93640c1d10 2024-03-06T07:25:29.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f936880e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93640711f0 con 0x7f93640c2920 2024-03-06T07:25:29.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f936880e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9364070d60 con 0x7f93640abb60 2024-03-06T07:25:29.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9362d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f93640c2920 0x7f9364156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9362d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f93640c2920 0x7f9364156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60042/0 (socket says 172.21.15.94:60042) 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9361d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f93640c1d10 0x7f9364156440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36912/0 (socket says 172.21.15.94:36912) 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9362d9d640 1 -- 172.21.15.94:0/1177652393 learned_addr learned my addr 172.21.15.94:0/1177652393 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9362d9d640 1 -- 172.21.15.94:0/1177652393 >> v2:172.21.15.145:3300/0 conn(0x7f93640abb60 msgr2=0x7f93641502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9362d9d640 1 --2- 172.21.15.94:0/1177652393 >> v2:172.21.15.145:3300/0 conn(0x7f93640abb60 0x7f93641502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9362d9d640 1 -- 172.21.15.94:0/1177652393 >> v1:172.21.15.94:6789/0 conn(0x7f93640c1d10 legacy=0x7f9364156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f9362d9d640 1 -- 172.21.15.94:0/1177652393 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9364070710 con 0x7f93640c2920 2024-03-06T07:25:29.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.433+0000 7f936159a640 1 -- 172.21.15.94:0/1177652393 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400918232 0 0) 0x7f9364070430 con 0x7f93640c1d10 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.434+0000 7f9362d9d640 1 --2- 172.21.15.94:0/1177652393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f9364156b50 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f936406d150 tx=0x7f935c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a96945e1d7556df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.434+0000 7f936159a640 1 -- 172.21.15.94:0/1177652393 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c02c9f0 con 0x7f93640c2920 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.434+0000 7f936159a640 1 -- 172.21.15.94:0/1177652393 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935c02cb90 con 0x7f93640c2920 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.434+0000 7f936159a640 1 -- 172.21.15.94:0/1177652393 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c0343f0 con 0x7f93640c2920 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 -- 172.21.15.94:0/1177652393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 msgr2=0x7f9364156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 --2- 172.21.15.94:0/1177652393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f9364156b50 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f936406d150 tx=0x7f935c02be50 comp rx=0 tx=0).stop 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 -- 172.21.15.94:0/1177652393 shutdown_connections 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 --2- 172.21.15.94:0/1177652393 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f9364156b50 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 --2- 172.21.15.94:0/1177652393 >> v2:172.21.15.145:3300/0 conn(0x7f93640abb60 0x7f93641502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 -- 172.21.15.94:0/1177652393 >> 172.21.15.94:0/1177652393 conn(0x7f93640c65b0 msgr2=0x7f93640c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 -- 172.21.15.94:0/1177652393 shutdown_connections 2024-03-06T07:25:29.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.435+0000 7f936880e640 1 -- 172.21.15.94:0/1177652393 wait complete. 2024-03-06T07:25:29.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.436+0000 7f936880e640 1 Processor -- start 2024-03-06T07:25:29.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.436+0000 7f936880e640 1 -- start start 2024-03-06T07:25:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936880e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93640abb60 0x7f936414bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936880e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f93641dd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936880e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93641dd8e0 0x7f93641490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936880e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93640abf40 con 0x7f93640c2920 2024-03-06T07:25:29.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936880e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9364157090 con 0x7f93640abb60 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936880e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93640ac0c0 con 0x7f93641dd8e0 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f93641dd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f93641dd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60046/0 (socket says 172.21.15.94:60046) 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 -- 172.21.15.94:0/720064698 learned_addr learned my addr 172.21.15.94:0/720064698 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936259c640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93640abb60 0x7f936414bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 -- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93641dd8e0 msgr2=0x7f93641490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9362d9d640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93641dd8e0 0x7f93641490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93641dd8e0 0x7f93641490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 -- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93640abb60 msgr2=0x7f936414bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93640abb60 0x7f936414bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 -- 172.21.15.94:0/720064698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93641538b0 con 0x7f93640c2920 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936259c640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93640abb60 0x7f936414bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9361d9b640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f93641dd3a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f935800c500 tx=0x7f935800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9352ffd640 1 -- 172.21.15.94:0/720064698 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358018070 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9352ffd640 1 -- 172.21.15.94:0/720064698 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9358007580 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9364149cc0 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.437+0000 7f9352ffd640 1 -- 172.21.15.94:0/720064698 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358004a20 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.438+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9364153b70 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.438+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936404f330 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.439+0000 7f9352ffd640 1 -- 172.21.15.94:0/720064698 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f935801b070 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.439+0000 7f9352ffd640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f934406b500 0x7f934406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.439+0000 7f9352ffd640 1 -- 172.21.15.94:0/720064698 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f9358089460 con 0x7f93640c2920 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.440+0000 7f936259c640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f934406b500 0x7f934406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.440+0000 7f936259c640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f934406b500 0x7f934406d9c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9354006fd0 tx=0x7f9354008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.444+0000 7f9352ffd640 1 -- 172.21.15.94:0/720064698 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9358058340 con 0x7f93640c2920 2024-03-06T07:25:29.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:29.614+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9364153690 con 0x7f93640c2920 2024-03-06T07:25:30.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.313+0000 7f9352ffd640 1 -- 172.21.15.94:0/720064698 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v173) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9358057c40 con 0x7f93640c2920 2024-03-06T07:25:30.315 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f934406b500 msgr2=0x7f934406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f934406b500 0x7f934406d9c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9354006fd0 tx=0x7f9354008040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 msgr2=0x7f93641dd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f93641dd3a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f935800c500 tx=0x7f935800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 shutdown_connections 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f934406b500 0x7f934406d9c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93641dd8e0 0x7f93641490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93640c2920 0x7f93641dd3a0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 --2- 172.21.15.94:0/720064698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93640abb60 0x7f936414bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.316+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 >> 172.21.15.94:0/720064698 conn(0x7f93640c65b0 msgr2=0x7f93641500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.317+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 shutdown_connections 2024-03-06T07:25:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.317+0000 7f936880e640 1 -- 172.21.15.94:0/720064698 wait complete. 2024-03-06T07:25:30.331 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:30.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.432+0000 7fd8b5c15640 1 Processor -- start 2024-03-06T07:25:30.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.432+0000 7fd8b5c15640 1 -- start start 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.433+0000 7fd8b5c15640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd8b00a97c0 0x7fd8b00c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.433+0000 7fd8b5c15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b0154330 con 0x7fd8b01538c0 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.433+0000 7fd8b5c15640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b0153dc0 con 0x7fd8b00ab310 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.433+0000 7fd8b5c15640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00c46a0 con 0x7fd8b00a97c0 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.433+0000 7fd8aeffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd8b01538c0 0x7fd8b01494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36924/0 (socket says 172.21.15.94:36924) 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.433+0000 7fd8aeffd640 1 -- 172.21.15.94:0/263727203 learned_addr learned my addr 172.21.15.94:0/263727203 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390915955 0 0) 0x7fd8b0154330 con 0x7fd8b01538c0 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd898003520 con 0x7fd8b01538c0 2024-03-06T07:25:30.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8affff640 1 --2- 172.21.15.94:0/263727203 >> v2:172.21.15.145:3300/0 conn(0x7fd8b00a97c0 0x7fd8b00c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472096971 0 0) 0x7fd898003520 con 0x7fd8b01538c0 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 >> v2:172.21.15.145:3300/0 conn(0x7fd8b00a97c0 msgr2=0x7fd8b00c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 --2- 172.21.15.94:0/263727203 >> v2:172.21.15.145:3300/0 conn(0x7fd8b00a97c0 0x7fd8b00c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 >> v1:172.21.15.145:6789/0 conn(0x7fd8b00ab310 legacy=0x7fd8b00ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00c5d30 con 0x7fd8b01538c0 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fd8a4001670 con 0x7fd8b01538c0 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 >> v1:172.21.15.94:6789/0 conn(0x7fd8b01538c0 legacy=0x7fd8b01494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8980039b0 0x7fd898003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd898004800 0x7fd898004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980052e0 0x7fd89800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c46a0 con 0x7fd8980039b0 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89800c190 con 0x7fd898004800 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89800c390 con 0x7fd8980052e0 2024-03-06T07:25:30.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8980039b0 0x7fd898003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8aeffd640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd898004800 0x7fd898004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8affff640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980052e0 0x7fd89800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8af7fe640 1 -- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980052e0 msgr2=0x7fd89800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980052e0 0x7fd89800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8af7fe640 1 -- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd898004800 msgr2=0x7fd898004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd898004800 0x7fd898004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8af7fe640 1 -- 172.21.15.94:0/263727203 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898003520 con 0x7fd8980039b0 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8affff640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980052e0 0x7fd89800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.434+0000 7fd8affff640 1 -- 172.21.15.94:0/263727203 reap_dead start 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.435+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8980039b0 0x7fd898003db0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0003d80 tx=0x7fd898005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1310a0f00529aa95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.435+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0037070 con 0x7fd8980039b0 2024-03-06T07:25:30.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.435+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0032600 con 0x7fd8980039b0 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.435+0000 7fd8ae7fc640 1 -- 172.21.15.94:0/263727203 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0003980 con 0x7fd8980039b0 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.436+0000 7fd8b5c15640 1 -- 172.21.15.94:0/263727203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8980039b0 msgr2=0x7fd898003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.436+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8980039b0 0x7fd898003db0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0003d80 tx=0x7fd898005830 comp rx=0 tx=0).stop 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.436+0000 7fd8b5c15640 1 -- 172.21.15.94:0/263727203 shutdown_connections 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.436+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/263727203 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8980039b0 0x7fd898003db0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.436+0000 7fd8b5c15640 1 -- 172.21.15.94:0/263727203 >> 172.21.15.94:0/263727203 conn(0x7fd8b00c6710 msgr2=0x7fd8b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.436+0000 7fd8b5c15640 1 -- 172.21.15.94:0/263727203 shutdown_connections 2024-03-06T07:25:30.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.436+0000 7fd8b5c15640 1 -- 172.21.15.94:0/263727203 wait complete. 2024-03-06T07:25:30.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 Processor -- start 2024-03-06T07:25:30.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 -- start start 2024-03-06T07:25:30.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980039b0 0x7fd8b01de5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8b00a5dd0 0x7fd8b01487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8b0072f90 0x7fd8b0148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00ab6f0 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c62f0 con 0x7fd8b0072f90 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.437+0000 7fd8b5c15640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00ab870 con 0x7fd8980039b0 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8b00a5dd0 0x7fd8b01487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8b00a5dd0 0x7fd8b01487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60070/0 (socket says 172.21.15.94:60070) 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 -- 172.21.15.94:0/797877688 learned_addr learned my addr 172.21.15.94:0/797877688 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8affff640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8b0072f90 0x7fd8b0148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980039b0 0x7fd8b01de5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 -- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980039b0 msgr2=0x7fd8b01de5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980039b0 0x7fd8b01de5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 -- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8b0072f90 msgr2=0x7fd8b0148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8b0072f90 0x7fd8b0148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 -- 172.21.15.94:0/797877688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00712c0 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980039b0 0x7fd8b01de5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8aeffd640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8b00a5dd0 0x7fd8b01487e0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400e870 tx=0x7fd8a400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd897fff640 1 -- 172.21.15.94:0/797877688 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400c7f0 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd897fff640 1 -- 172.21.15.94:0/797877688 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a4018070 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b0156fe0 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd897fff640 1 -- 172.21.15.94:0/797877688 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400cbf0 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.438+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b0149260 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.439+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b0075300 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.439+0000 7fd897fff640 1 -- 172.21.15.94:0/797877688 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8a401d460 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.440+0000 7fd897fff640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b4d0 0x7fd89c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.440+0000 7fd897fff640 1 -- 172.21.15.94:0/797877688 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fd8a4089ed0 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.440+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b4d0 0x7fd89c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.441+0000 7fd8af7fe640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b4d0 0x7fd89c06d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0034000 tx=0x7fd8a0009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.444+0000 7fd897fff640 1 -- 172.21.15.94:0/797877688 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8b0075300 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.614+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd8b0071490 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.617+0000 7fd897fff640 1 -- 172.21.15.94:0/797877688 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd8a4058f50 con 0x7fd8b00a5dd0 2024-03-06T07:25:30.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b4d0 msgr2=0x7fd89c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b4d0 0x7fd89c06d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0034000 tx=0x7fd8a0009030 comp rx=0 tx=0).stop 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8b00a5dd0 msgr2=0x7fd8b01487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8b00a5dd0 0x7fd8b01487e0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400e870 tx=0x7fd8a400ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 shutdown_connections 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd89c06b4d0 0x7fd89c06d990 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8b0072f90 0x7fd8b0148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd8b00a5dd0 0x7fd8b01487e0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 --2- 172.21.15.94:0/797877688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8980039b0 0x7fd8b01de5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 >> 172.21.15.94:0/797877688 conn(0x7fd8b00c6710 msgr2=0x7fd8b00c1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 shutdown_connections 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.619+0000 7fd8b5c15640 1 -- 172.21.15.94:0/797877688 wait complete. 2024-03-06T07:25:30.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-03-06T07:25:30.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.730+0000 7fa1924b5640 1 Processor -- start 2024-03-06T07:25:30.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.730+0000 7fa1924b5640 1 -- start start 2024-03-06T07:25:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa1924b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa18c150920 0x7fa18c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa1924b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa1924b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18c067340 con 0x7fa18c0c2e70 2024-03-06T07:25:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa1924b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c150d00 con 0x7fa18c150920 2024-03-06T07:25:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa1924b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0c2010 con 0x7fa18c06c720 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa18c150920 0x7fa18c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa18c150920 0x7fa18c0c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60074/0 (socket says 172.21.15.94:60074) 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18bfff640 1 -- 172.21.15.94:0/2958242513 learned_addr learned my addr 172.21.15.94:0/2958242513 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18b7fe640 1 --2- 172.21.15.94:0/2958242513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18bfff640 1 -- 172.21.15.94:0/2958242513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 msgr2=0x7fa18c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18bfff640 1 --2- 172.21.15.94:0/2958242513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18bfff640 1 -- 172.21.15.94:0/2958242513 >> v1:172.21.15.145:6789/0 conn(0x7fa18c0c2e70 legacy=0x7fa18c1532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18bfff640 1 -- 172.21.15.94:0/2958242513 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c067060 con 0x7fa18c150920 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.731+0000 7fa18b7fe640 1 --2- 172.21.15.94:0/2958242513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa18bfff640 1 --2- 172.21.15.94:0/2958242513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c150920 0x7fa18c0c2930 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa180009870 tx=0x7fa18002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cba034dfd98175 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa18affd640 1 -- 172.21.15.94:0/2958242513 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180035020 con 0x7fa18c150920 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa18affd640 1 -- 172.21.15.94:0/2958242513 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa180031640 con 0x7fa18c150920 2024-03-06T07:25:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa18affd640 1 -- 172.21.15.94:0/2958242513 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18002d040 con 0x7fa18c150920 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa1924b5640 1 -- 172.21.15.94:0/2958242513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c150920 msgr2=0x7fa18c0c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa1924b5640 1 --2- 172.21.15.94:0/2958242513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c150920 0x7fa18c0c2930 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa180009870 tx=0x7fa18002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa1924b5640 1 -- 172.21.15.94:0/2958242513 shutdown_connections 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa1924b5640 1 --2- 172.21.15.94:0/2958242513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa1924b5640 1 --2- 172.21.15.94:0/2958242513 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c150920 0x7fa18c0c2930 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa1924b5640 1 -- 172.21.15.94:0/2958242513 >> 172.21.15.94:0/2958242513 conn(0x7fa18c0c70b0 msgr2=0x7fa18c0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.732+0000 7fa1924b5640 1 -- 172.21.15.94:0/2958242513 shutdown_connections 2024-03-06T07:25:30.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.733+0000 7fa1924b5640 1 -- 172.21.15.94:0/2958242513 wait complete. 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.733+0000 7fa1924b5640 1 Processor -- start 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa1924b5640 1 -- start start 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa1924b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa1924b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c0c2e70 0x7fa18c1ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa1924b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c150920 0x7fa18c14be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa1924b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c073210 con 0x7fa18c0c2e70 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa1924b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c073090 con 0x7fa18c150920 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa1924b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0c1b40 con 0x7fa18c06c720 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa18b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c0c2e70 0x7fa18c1ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa18b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c0c2e70 0x7fa18c1ec6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60086/0 (socket says 172.21.15.94:60086) 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.734+0000 7fa18b7fe640 1 -- 172.21.15.94:0/1883713640 learned_addr learned my addr 172.21.15.94:0/1883713640 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa190a52640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c150920 0x7fa18c14be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa18bfff640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa18b7fe640 1 -- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 msgr2=0x7fa18c152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa18b7fe640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa18b7fe640 1 -- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c150920 msgr2=0x7fa18c14be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa18b7fe640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c150920 0x7fa18c14be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa18b7fe640 1 -- 172.21.15.94:0/1883713640 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c14d0b0 con 0x7fa18c0c2e70 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa18b7fe640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c0c2e70 0x7fa18c1ec6d0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e7e0 tx=0x7fa17c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa188ff9640 1 -- 172.21.15.94:0/1883713640 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c00f040 con 0x7fa18c0c2e70 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa188ff9640 1 -- 172.21.15.94:0/1883713640 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c004030 con 0x7fa18c0c2e70 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18c1ecfa0 con 0x7fa18c0c2e70 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa188ff9640 1 -- 172.21.15.94:0/1883713640 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c004400 con 0x7fa18c0c2e70 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.735+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c1515b0 con 0x7fa18c0c2e70 2024-03-06T07:25:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.736+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c0735b0 con 0x7fa18c0c2e70 2024-03-06T07:25:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.736+0000 7fa188ff9640 1 -- 172.21.15.94:0/1883713640 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa17c00c6d0 con 0x7fa18c0c2e70 2024-03-06T07:25:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.737+0000 7fa188ff9640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa17806b4d0 0x7fa17806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.737+0000 7fa188ff9640 1 -- 172.21.15.94:0/1883713640 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa17c089dc0 con 0x7fa18c0c2e70 2024-03-06T07:25:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.737+0000 7fa18bfff640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa17806b4d0 0x7fa17806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.737+0000 7fa18bfff640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa17806b4d0 0x7fa17806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa180036000 tx=0x7fa18003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.741+0000 7fa188ff9640 1 -- 172.21.15.94:0/1883713640 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa18c0735b0 con 0x7fa18c0c2e70 2024-03-06T07:25:30.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:30.978+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7fa18c04f330 con 0x7fa18c0c2e70 2024-03-06T07:25:31.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.314+0000 7fa188ff9640 1 -- 172.21.15.94:0/1883713640 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v419) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa17c017020 con 0x7fa18c0c2e70 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa17806b4d0 msgr2=0x7fa17806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa17806b4d0 0x7fa17806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa180036000 tx=0x7fa18003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c0c2e70 msgr2=0x7fa18c1ec6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c0c2e70 0x7fa18c1ec6d0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e7e0 tx=0x7fa17c00eca0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 shutdown_connections 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa17806b4d0 0x7fa17806d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c150920 0x7fa18c14be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa18c0c2e70 0x7fa18c1ec6d0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 --2- 172.21.15.94:0/1883713640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c06c720 0x7fa18c152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 >> 172.21.15.94:0/1883713640 conn(0x7fa18c0c70b0 msgr2=0x7fa18c0c7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 shutdown_connections 2024-03-06T07:25:31.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.317+0000 7fa1924b5640 1 -- 172.21.15.94:0/1883713640 wait complete. 2024-03-06T07:25:31.330 INFO:tasks.cephfs_test_runner:That down setting toggles and sets max_mds appropriately. ... ok 2024-03-06T07:25:31.330 INFO:tasks.cephfs_test_runner:Starting test: test_down_grow (tasks.cephfs.test_failover.TestClusterResize) That setting max_mds undoes down. 2024-03-06T07:25:31.330 INFO:tasks.cephfs_test_runner:test_down_grow (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:25:31.331 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:31.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.431+0000 7f847aceb640 1 Processor -- start 2024-03-06T07:25:31.433 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.432+0000 7f847aceb640 1 -- start start 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.432+0000 7f847aceb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f84740c88c0 0x7f8474155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.432+0000 7f847aceb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84740c73d0 con 0x7f8474158b60 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.432+0000 7f847aceb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84740be2e0 con 0x7f847414f360 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.432+0000 7f847aceb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8474159700 con 0x7f84740c88c0 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.432+0000 7f8479288640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8474158b60 0x7f8474158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36938/0 (socket says 172.21.15.94:36938) 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.432+0000 7f8479288640 1 -- 172.21.15.94:0/1372923320 learned_addr learned my addr 172.21.15.94:0/1372923320 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8473fff640 1 --2- 172.21.15.94:0/1372923320 >> v2:172.21.15.145:3300/0 conn(0x7f84740c88c0 0x7f8474155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150615785 0 0) 0x7f84740c73d0 con 0x7f8474158b60 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8464003520 con 0x7f8474158b60 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3013285773 0 0) 0x7f8464003520 con 0x7f8474158b60 2024-03-06T07:25:31.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 >> v2:172.21.15.145:3300/0 conn(0x7f84740c88c0 msgr2=0x7f8474155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 --2- 172.21.15.94:0/1372923320 >> v2:172.21.15.145:3300/0 conn(0x7f84740c88c0 0x7f8474155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 >> v1:172.21.15.145:6789/0 conn(0x7f847414f360 legacy=0x7f84740c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84740c7670 con 0x7f8474158b60 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f846c0019f0 con 0x7f8474158b60 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 >> v1:172.21.15.94:6789/0 conn(0x7f8474158b60 legacy=0x7f8474158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640039b0 0x7f8464003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84640043b0 0x7f8464004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8464005120 0x7f846400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8474159700 con 0x7f84640043b0 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846400c060 con 0x7f84640039b0 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846400c260 con 0x7f8464005120 2024-03-06T07:25:31.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8473fff640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84640043b0 0x7f8464004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8478a87640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640039b0 0x7f8464003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8479288640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8464005120 0x7f846400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8473fff640 1 -- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8464005120 msgr2=0x7f846400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8473fff640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8464005120 0x7f846400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8473fff640 1 -- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640039b0 msgr2=0x7f8464003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8473fff640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640039b0 0x7f8464003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8473fff640 1 -- 172.21.15.94:0/1372923320 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464003520 con 0x7f84640043b0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8479288640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8464005120 0x7f846400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.433+0000 7f8479288640 1 -- 172.21.15.94:0/1372923320 reap_dead start 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.434+0000 7f8473fff640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84640043b0 0x7f8464004880 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f8460009a90 tx=0x7f8464005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b4e2e60469011d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.434+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460039070 con 0x7f84640043b0 2024-03-06T07:25:31.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.434+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846002c680 con 0x7f84640043b0 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.434+0000 7f84737fe640 1 -- 172.21.15.94:0/1372923320 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460009690 con 0x7f84640043b0 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.435+0000 7f847aceb640 1 -- 172.21.15.94:0/1372923320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84640043b0 msgr2=0x7f8464004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.435+0000 7f847aceb640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84640043b0 0x7f8464004880 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f8460009a90 tx=0x7f8464005830 comp rx=0 tx=0).stop 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.435+0000 7f847aceb640 1 -- 172.21.15.94:0/1372923320 shutdown_connections 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.435+0000 7f847aceb640 1 --2- 172.21.15.94:0/1372923320 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84640043b0 0x7f8464004880 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.435+0000 7f847aceb640 1 -- 172.21.15.94:0/1372923320 >> 172.21.15.94:0/1372923320 conn(0x7f84740ac570 msgr2=0x7f84740ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.435+0000 7f847aceb640 1 -- 172.21.15.94:0/1372923320 shutdown_connections 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.435+0000 7f847aceb640 1 -- 172.21.15.94:0/1372923320 wait complete. 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.436+0000 7f847aceb640 1 Processor -- start 2024-03-06T07:25:31.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.436+0000 7f847aceb640 1 -- start start 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640043b0 0x7f8474151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84740bd970 0x7f8474152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847414aa40 0x7f847414aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84740c7c30 con 0x7f847414aa40 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8474155c10 con 0x7f84640043b0 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8474155d90 con 0x7f84740bd970 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847414aa40 0x7f847414aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847414aa40 0x7f847414aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60110/0 (socket says 172.21.15.94:60110) 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 -- 172.21.15.94:0/51631471 learned_addr learned my addr 172.21.15.94:0/51631471 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8473fff640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84740bd970 0x7f8474152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8478a87640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640043b0 0x7f8474151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 -- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84740bd970 msgr2=0x7f8474152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84740bd970 0x7f8474152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 -- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640043b0 msgr2=0x7f8474151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640043b0 0x7f8474151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 -- 172.21.15.94:0/51631471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84741ec230 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f8479288640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847414aa40 0x7f847414aec0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f8474067c30 tx=0x7f846c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f84717fa640 1 -- 172.21.15.94:0/51631471 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00eb90 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f84717fa640 1 -- 172.21.15.94:0/51631471 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c00ed30 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847414f870 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f84717fa640 1 -- 172.21.15.94:0/51631471 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c014630 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.437+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847414fe30 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.438+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84740c8590 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.439+0000 7f84717fa640 1 -- 172.21.15.94:0/51631471 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f846c0147d0 con 0x7f847414aa40 2024-03-06T07:25:31.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.439+0000 7f84717fa640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f845806b4d0 0x7f845806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.439+0000 7f84717fa640 1 -- 172.21.15.94:0/51631471 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f846c089e40 con 0x7f847414aa40 2024-03-06T07:25:31.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.439+0000 7f8478a87640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f845806b4d0 0x7f845806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.440+0000 7f8478a87640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f845806b4d0 0x7f845806d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8468003700 tx=0x7f8468009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.443+0000 7f84717fa640 1 -- 172.21.15.94:0/51631471 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84740c8590 con 0x7f847414aa40 2024-03-06T07:25:31.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.617+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8474075300 con 0x7f847414aa40 2024-03-06T07:25:31.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.619+0000 7f84717fa640 1 -- 172.21.15.94:0/51631471 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+13305 (secure 0 0 0) 0x7f8474075300 con 0x7f847414aa40 2024-03-06T07:25:31.621 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:31.621 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:30.308084+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6836/1183474661":"2024-03-07T07:25:27.193037+0000","172.21.15.145:6835/1183474661":"2024-03-07T07:25:27.193037+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:31.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f845806b4d0 msgr2=0x7f845806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f845806b4d0 0x7f845806d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8468003700 tx=0x7f8468009040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847414aa40 msgr2=0x7f847414aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847414aa40 0x7f847414aec0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f8474067c30 tx=0x7f846c00e530 comp rx=0 tx=0).stop 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 shutdown_connections 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f845806b4d0 0x7f845806d990 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f847414aa40 0x7f847414aec0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84740bd970 0x7f8474152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 --2- 172.21.15.94:0/51631471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84640043b0 0x7f8474151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 >> 172.21.15.94:0/51631471 conn(0x7f84740ac570 msgr2=0x7f84740ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 shutdown_connections 2024-03-06T07:25:31.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.621+0000 7f847aceb640 1 -- 172.21.15.94:0/51631471 wait complete. 2024-03-06T07:25:31.634 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:31.634 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:31.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.736+0000 7f3564302640 1 Processor -- start 2024-03-06T07:25:31.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.736+0000 7f3564302640 1 -- start start 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f3564302640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f355c0ab310 0x7f355c0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f3564302640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f355c0a97c0 0x7f355c0c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f3564302640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c154330 con 0x7f355c1538c0 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f3564302640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c0c46a0 con 0x7f355c0ab310 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f3564302640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c153dc0 con 0x7f355c0a97c0 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356209e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f355c0ab310 0x7f355c0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356189d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f355c1538c0 0x7f355c1494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36954/0 (socket says 172.21.15.94:36954) 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356189d640 1 -- 172.21.15.94:0/4231773772 learned_addr learned my addr 172.21.15.94:0/4231773772 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356209e640 1 -- 172.21.15.94:0/4231773772 >> v2:172.21.15.145:3300/0 conn(0x7f355c0a97c0 msgr2=0x7f355c0c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356209e640 1 --2- 172.21.15.94:0/4231773772 >> v2:172.21.15.145:3300/0 conn(0x7f355c0a97c0 0x7f355c0c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356209e640 1 -- 172.21.15.94:0/4231773772 >> v1:172.21.15.94:6789/0 conn(0x7f355c1538c0 legacy=0x7f355c1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356209e640 1 -- 172.21.15.94:0/4231773772 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0c5d30 con 0x7f355c0ab310 2024-03-06T07:25:31.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.737+0000 7f356109c640 1 -- 172.21.15.94:0/4231773772 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690160505 0 0) 0x7f355c154330 con 0x7f355c1538c0 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.738+0000 7f356209e640 1 --2- 172.21.15.94:0/4231773772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c0ab6f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f355c14d490 tx=0x7f355802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3422b55654d46748 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.738+0000 7f356109c640 1 -- 172.21.15.94:0/4231773772 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35580315a0 con 0x7f355c0ab310 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.738+0000 7f356109c640 1 -- 172.21.15.94:0/4231773772 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558037070 con 0x7f355c0ab310 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.738+0000 7f356109c640 1 -- 172.21.15.94:0/4231773772 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558031960 con 0x7f355c0ab310 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.738+0000 7f3564302640 1 -- 172.21.15.94:0/4231773772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 msgr2=0x7f355c0ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.738+0000 7f3564302640 1 --2- 172.21.15.94:0/4231773772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c0ab6f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f355c14d490 tx=0x7f355802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.739+0000 7f3564302640 1 -- 172.21.15.94:0/4231773772 shutdown_connections 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.739+0000 7f3564302640 1 --2- 172.21.15.94:0/4231773772 >> v2:172.21.15.145:3300/0 conn(0x7f355c0a97c0 0x7f355c0c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.739+0000 7f3564302640 1 --2- 172.21.15.94:0/4231773772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c0ab6f0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.739+0000 7f3564302640 1 -- 172.21.15.94:0/4231773772 >> 172.21.15.94:0/4231773772 conn(0x7f355c0c6710 msgr2=0x7f355c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.739+0000 7f3564302640 1 -- 172.21.15.94:0/4231773772 shutdown_connections 2024-03-06T07:25:31.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.739+0000 7f3564302640 1 -- 172.21.15.94:0/4231773772 wait complete. 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 Processor -- start 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 -- start start 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f355c0a97c0 0x7f355c156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f355c1538c0 0x7f355c148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0abc30 con 0x7f355c0ab310 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0c62f0 con 0x7f355c1538c0 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f3564302640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c149bb0 con 0x7f355c0a97c0 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f356189d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f356189d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c148000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60132/0 (socket says 172.21.15.94:60132) 2024-03-06T07:25:31.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f356189d640 1 -- 172.21.15.94:0/342123842 learned_addr learned my addr 172.21.15.94:0/342123842 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f356289f640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f355c1538c0 0x7f355c148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.740+0000 7f356209e640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f355c0a97c0 0x7f355c156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f356189d640 1 -- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f355c0a97c0 msgr2=0x7f355c156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f356189d640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f355c0a97c0 0x7f355c156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f356189d640 1 -- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f355c1538c0 msgr2=0x7f355c148540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f356189d640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f355c1538c0 0x7f355c148540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f356189d640 1 -- 172.21.15.94:0/342123842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c148a80 con 0x7f355c0ab310 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f356189d640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c148000 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c440 tx=0x7f354c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f3552ffd640 1 -- 172.21.15.94:0/342123842 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c018070 con 0x7f355c0ab310 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f3552ffd640 1 -- 172.21.15.94:0/342123842 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c0074e0 con 0x7f355c0ab310 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c1553d0 con 0x7f355c0ab310 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f3552ffd640 1 -- 172.21.15.94:0/342123842 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c004a20 con 0x7f355c0ab310 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.741+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c155990 con 0x7f355c0ab310 2024-03-06T07:25:31.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.742+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c04f330 con 0x7f355c0ab310 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.742+0000 7f3552ffd640 1 -- 172.21.15.94:0/342123842 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f354c01b070 con 0x7f355c0ab310 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.743+0000 7f3552ffd640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f354006b500 0x7f354006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.743+0000 7f3552ffd640 1 -- 172.21.15.94:0/342123842 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f354c089270 con 0x7f355c0ab310 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.743+0000 7f356209e640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f354006b500 0x7f354006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.743+0000 7f356209e640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f354006b500 0x7f354006d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3558036000 tx=0x7f355803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.747+0000 7f3552ffd640 1 -- 172.21.15.94:0/342123842 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f354c057b80 con 0x7f355c0ab310 2024-03-06T07:25:31.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.924+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f355c0c5a70 con 0x7f355c0ab310 2024-03-06T07:25:31.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.924+0000 7f3552ffd640 1 -- 172.21.15.94:0/342123842 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) v1 ==== 93+0+10 (secure 0 0 0) 0x7f354c05bc20 con 0x7f355c0ab310 2024-03-06T07:25:31.926 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:25:31.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f354006b500 msgr2=0x7f354006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f354006b500 0x7f354006d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3558036000 tx=0x7f355803d040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 msgr2=0x7f355c148000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c148000 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f354c00c440 tx=0x7f354c00c900 comp rx=0 tx=0).stop 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 shutdown_connections 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f354006b500 0x7f354006d9c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f355c1538c0 0x7f355c148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f355c0ab310 0x7f355c148000 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 --2- 172.21.15.94:0/342123842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f355c0a97c0 0x7f355c156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 >> 172.21.15.94:0/342123842 conn(0x7f355c0c6710 msgr2=0x7f355c0c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 shutdown_connections 2024-03-06T07:25:31.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:31.926+0000 7f3564302640 1 -- 172.21.15.94:0/342123842 wait complete. 2024-03-06T07:25:31.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-03-06T07:25:32.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.038+0000 7f2ccd4eb640 1 Processor -- start 2024-03-06T07:25:32.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.038+0000 7f2ccd4eb640 1 -- start start 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2ccd4eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 0x7f2cc80c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2ccd4eb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2cc80c88b0 0x7f2cc81586a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2ccd4eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2cc80c7dd0 0x7f2cc8158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2ccd4eb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc81559e0 con 0x7f2cc80c7dd0 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2ccd4eb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc8155110 con 0x7f2cc80c88b0 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2ccd4eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc80c8f40 con 0x7f2cc80c5330 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2cc80c7dd0 0x7f2cc8158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2cc80c7dd0 0x7f2cc8158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60138/0 (socket says 172.21.15.94:60138) 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 -- 172.21.15.94:0/2127166817 learned_addr learned my addr 172.21.15.94:0/2127166817 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 -- 172.21.15.94:0/2127166817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 msgr2=0x7f2cc80c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 --2- 172.21.15.94:0/2127166817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 0x7f2cc80c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 -- 172.21.15.94:0/2127166817 >> v2:172.21.15.145:3300/0 conn(0x7f2cc80c88b0 msgr2=0x7f2cc81586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 --2- 172.21.15.94:0/2127166817 >> v2:172.21.15.145:3300/0 conn(0x7f2cc80c88b0 0x7f2cc81586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.039+0000 7f2cc77fe640 1 -- 172.21.15.94:0/2127166817 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc8155390 con 0x7f2cc80c7dd0 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2cc77fe640 1 --2- 172.21.15.94:0/2127166817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc8158be0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f2cc808c940 tx=0x7f2cc002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2b3b1dadc1edd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2cc5ffb640 1 -- 172.21.15.94:0/2127166817 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc002c9e0 con 0x7f2cc80c7dd0 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2cc5ffb640 1 -- 172.21.15.94:0/2127166817 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc002cb80 con 0x7f2cc80c7dd0 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2cc5ffb640 1 -- 172.21.15.94:0/2127166817 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc00343f0 con 0x7f2cc80c7dd0 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/2127166817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c7dd0 msgr2=0x7f2cc8158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/2127166817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc8158be0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f2cc808c940 tx=0x7f2cc002be40 comp rx=0 tx=0).stop 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/2127166817 shutdown_connections 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/2127166817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc8158be0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/2127166817 >> v2:172.21.15.145:3300/0 conn(0x7f2cc80c88b0 0x7f2cc81586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/2127166817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 0x7f2cc80c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.040+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/2127166817 >> 172.21.15.94:0/2127166817 conn(0x7f2cc80ac560 msgr2=0x7f2cc80ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.041+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/2127166817 shutdown_connections 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.041+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/2127166817 wait complete. 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 Processor -- start 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 -- start start 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 0x7f2cc81eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc81eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c88b0 0x7f2cc8157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc81529d0 con 0x7f2cc80c88b0 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc806d090 con 0x7f2cc80c7dd0 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.042+0000 7f2ccd4eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc8152b50 con 0x7f2cc80c5330 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c88b0 0x7f2cc8157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c88b0 0x7f2cc8157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60144/0 (socket says 172.21.15.94:60144) 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 -- 172.21.15.94:0/1739912176 learned_addr learned my addr 172.21.15.94:0/1739912176 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc6ffd640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 0x7f2cc81eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc67fc640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc81eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 -- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 msgr2=0x7f2cc81eefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 0x7f2cc81eefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 -- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cc80c7dd0 msgr2=0x7f2cc81eaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc81eaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 -- 172.21.15.94:0/1739912176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc81eb3f0 con 0x7f2cc80c88b0 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc67fc640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc81eaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2cc77fe640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c88b0 0x7f2cc8157a70 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0036040 tx=0x7f2cc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2caf7fe640 1 -- 172.21.15.94:0/1739912176 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0043020 con 0x7f2cc80c88b0 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2caf7fe640 1 -- 172.21.15.94:0/1739912176 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc003e450 con 0x7f2cc80c88b0 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2caf7fe640 1 -- 172.21.15.94:0/1739912176 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0009510 con 0x7f2cc80c88b0 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc81eb670 con 0x7f2cc80c88b0 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.043+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc8151c70 con 0x7f2cc80c88b0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.044+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc8158f40 con 0x7f2cc80c88b0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.045+0000 7f2caf7fe640 1 -- 172.21.15.94:0/1739912176 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cc0046020 con 0x7f2cc80c88b0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.045+0000 7f2caf7fe640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cb406b4d0 0x7f2cb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.045+0000 7f2caf7fe640 1 -- 172.21.15.94:0/1739912176 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f2cc00ac920 con 0x7f2cc80c88b0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.045+0000 7f2cc6ffd640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cb406b4d0 0x7f2cb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.046+0000 7f2cc6ffd640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cb406b4d0 0x7f2cb406d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b810 tx=0x7f2cb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.049+0000 7f2caf7fe640 1 -- 172.21.15.94:0/1739912176 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2cc8158f40 con 0x7f2cc80c88b0 2024-03-06T07:25:32.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.286+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7f2cc8067910 con 0x7f2cc80c88b0 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.316+0000 7f2caf7fe640 1 -- 172.21.15.94:0/1739912176 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v420) v1 ==== 141+0+0 (secure 0 0 0) 0x7f2cc8067910 con 0x7f2cc80c88b0 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cb406b4d0 msgr2=0x7f2cb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cb406b4d0 0x7f2cb406d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b810 tx=0x7f2cb8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c88b0 msgr2=0x7f2cc8157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c88b0 0x7f2cc8157a70 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0036040 tx=0x7f2cc0004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 shutdown_connections 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2cb406b4d0 0x7f2cb406d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc80c88b0 0x7f2cc8157a70 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cc80c7dd0 0x7f2cc81eaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 --2- 172.21.15.94:0/1739912176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cc80c5330 0x7f2cc81eefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 >> 172.21.15.94:0/1739912176 conn(0x7f2cc80ac560 msgr2=0x7f2cc8154120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 shutdown_connections 2024-03-06T07:25:32.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.318+0000 7f2ccd4eb640 1 -- 172.21.15.94:0/1739912176 wait complete. 2024-03-06T07:25:32.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:32.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5ae2c84640 1 Processor -- start 2024-03-06T07:25:32.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5ae2c84640 1 -- start start 2024-03-06T07:25:32.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5ae2c84640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5adc155c30 0x7f5adc06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5ae2c84640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5adc0c5380 0x7f5adc0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5ae2c84640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5adc0b0450 con 0x7f5adc0b0070 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5ae2c84640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5adc06cfb0 con 0x7f5adc155c30 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5ae2c84640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5adc06c020 con 0x7f5adc0c5380 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5adbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5adc155c30 0x7f5adc06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5adbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5adc155c30 0x7f5adc06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60150/0 (socket says 172.21.15.94:60150) 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.433+0000 7f5adbfff640 1 -- 172.21.15.94:0/4177151001 learned_addr learned my addr 172.21.15.94:0/4177151001 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.434+0000 7f5ae1221640 1 --2- 172.21.15.94:0/4177151001 >> v2:172.21.15.145:3300/0 conn(0x7f5adc0c5380 0x7f5adc0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.434+0000 7f5adbfff640 1 -- 172.21.15.94:0/4177151001 >> v2:172.21.15.145:3300/0 conn(0x7f5adc0c5380 msgr2=0x7f5adc0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.434+0000 7f5adbfff640 1 --2- 172.21.15.94:0/4177151001 >> v2:172.21.15.145:3300/0 conn(0x7f5adc0c5380 0x7f5adc0c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.434+0000 7f5adbfff640 1 -- 172.21.15.94:0/4177151001 >> v1:172.21.15.145:6789/0 conn(0x7f5adc0b0070 legacy=0x7f5adc0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.434+0000 7f5adbfff640 1 -- 172.21.15.94:0/4177151001 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc1560e0 con 0x7f5adc155c30 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.434+0000 7f5adbfff640 1 --2- 172.21.15.94:0/4177151001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc06ca70 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0be030 tx=0x7f5ac402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e997525ff5008ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5adb7fe640 1 -- 172.21.15.94:0/4177151001 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac402e070 con 0x7f5adc155c30 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5adb7fe640 1 -- 172.21.15.94:0/4177151001 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac4037070 con 0x7f5adc155c30 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5adb7fe640 1 -- 172.21.15.94:0/4177151001 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4031a00 con 0x7f5adc155c30 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177151001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 msgr2=0x7f5adc06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177151001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc06ca70 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0be030 tx=0x7f5ac402b880 comp rx=0 tx=0).stop 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177151001 shutdown_connections 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177151001 >> v2:172.21.15.145:3300/0 conn(0x7f5adc0c5380 0x7f5adc0c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177151001 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc06ca70 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177151001 >> 172.21.15.94:0/4177151001 conn(0x7f5adc0ac590 msgr2=0x7f5adc0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177151001 shutdown_connections 2024-03-06T07:25:32.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.435+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177151001 wait complete. 2024-03-06T07:25:32.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.436+0000 7f5ae2c84640 1 Processor -- start 2024-03-06T07:25:32.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.436+0000 7f5ae2c84640 1 -- start start 2024-03-06T07:25:32.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae2c84640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5adc0b0070 0x7f5adc1ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae2c84640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5adc0c5380 0x7f5adc0c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae2c84640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc0c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae2c84640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc06c270 con 0x7f5adc155c30 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae2c84640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0c7ad0 con 0x7f5adc0b0070 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae2c84640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc06c550 con 0x7f5adc0c5380 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae1221640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc0c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae1221640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc0c2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60156/0 (socket says 172.21.15.94:60156) 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae1221640 1 -- 172.21.15.94:0/4177906412 learned_addr learned my addr 172.21.15.94:0/4177906412 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae0a20640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5adc0b0070 0x7f5adc1ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5adbfff640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5adc0c5380 0x7f5adc0c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae1221640 1 -- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5adc0c5380 msgr2=0x7f5adc0c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.437+0000 7f5ae1221640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5adc0c5380 0x7f5adc0c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ae1221640 1 -- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5adc0b0070 msgr2=0x7f5adc1ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ae1221640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5adc0b0070 0x7f5adc1ee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ae1221640 1 -- 172.21.15.94:0/4177906412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc1eee00 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ae1221640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc0c2c20 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00d800 tx=0x7f5acc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ad97fa640 1 -- 172.21.15.94:0/4177906412 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00b6c0 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ad97fa640 1 -- 172.21.15.94:0/4177906412 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5acc019070 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc0c4500 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ad97fa640 1 -- 172.21.15.94:0/4177906412 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00bac0 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.438+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc0beb40 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.439+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc04f330 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.439+0000 7f5ad97fa640 1 -- 172.21.15.94:0/4177906412 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5acc00bc60 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.440+0000 7f5ad97fa640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5abc06b4d0 0x7f5abc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.440+0000 7f5ad97fa640 1 -- 172.21.15.94:0/4177906412 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5acc08a470 con 0x7f5adc155c30 2024-03-06T07:25:32.443 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.440+0000 7f5ae0a20640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5abc06b4d0 0x7f5abc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.440+0000 7f5ae0a20640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5abc06b4d0 0x7f5abc06d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009f20 tx=0x7f5ad40094d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.444+0000 7f5ad97fa640 1 -- 172.21.15.94:0/4177906412 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5acc0594f0 con 0x7f5adc155c30 2024-03-06T07:25:32.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.615+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5adc0c4b00 con 0x7f5adc155c30 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.618+0000 7f5ad97fa640 1 -- 172.21.15.94:0/4177906412 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v59)=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5acc058df0 con 0x7f5adc155c30 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5abc06b4d0 msgr2=0x7f5abc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5abc06b4d0 0x7f5abc06d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009f20 tx=0x7f5ad40094d0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 msgr2=0x7f5adc0c2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc0c2c20 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00d800 tx=0x7f5acc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 shutdown_connections 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5abc06b4d0 0x7f5abc06d990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5adc155c30 0x7f5adc0c2c20 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5adc0c5380 0x7f5adc0c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 --2- 172.21.15.94:0/4177906412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5adc0b0070 0x7f5adc1ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 >> 172.21.15.94:0/4177906412 conn(0x7f5adc0ac590 msgr2=0x7f5adc0ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 shutdown_connections 2024-03-06T07:25:32.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.620+0000 7f5ae2c84640 1 -- 172.21.15.94:0/4177906412 wait complete. 2024-03-06T07:25:32.633 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:32.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:32.679 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:32.732 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:32.732 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:25:32.733 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:32.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:25:32.747 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:25:32.748 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:32.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:25:32.765 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:25:32.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:32.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.885+0000 7fd74765c640 1 Processor -- start 2024-03-06T07:25:32.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.885+0000 7fd74765c640 1 -- start start 2024-03-06T07:25:32.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd74765c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd7401550c0 0x7fd7401554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd74765c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7400aa660 con 0x7fd7401559e0 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd74765c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7400aa940 con 0x7fd7400aa180 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd74765c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7400c4610 con 0x7fd7401550c0 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd744bf7640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd7401559e0 0x7fd74014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36962/0 (socket says 172.21.15.94:36962) 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd744bf7640 1 -- 172.21.15.94:0/1048561322 learned_addr learned my addr 172.21.15.94:0/1048561322 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd7453f8640 1 --2- 172.21.15.94:0/1048561322 >> v2:172.21.15.145:3300/0 conn(0x7fd7401550c0 0x7fd7401554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597992956 0 0) 0x7fd7400aa660 con 0x7fd7401559e0 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd72c003520 con 0x7fd7401559e0 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.886+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955277874 0 0) 0x7fd72c003520 con 0x7fd7401559e0 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 >> v2:172.21.15.145:3300/0 conn(0x7fd7401550c0 msgr2=0x7fd7401554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 --2- 172.21.15.94:0/1048561322 >> v2:172.21.15.145:3300/0 conn(0x7fd7401550c0 0x7fd7401554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 >> v1:172.21.15.145:6789/0 conn(0x7fd7400aa180 legacy=0x7fd7400c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7400c4860 con 0x7fd7401559e0 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fd730001670 con 0x7fd7401559e0 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 >> v1:172.21.15.94:6789/0 conn(0x7fd7401559e0 legacy=0x7fd74014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd72c0039b0 0x7fd72c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c004800 0x7fd72c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72c0052e0 0x7fd72c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400c4610 con 0x7fd72c0052e0 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c00c1b0 con 0x7fd72c0039b0 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c00c3b0 con 0x7fd72c004800 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd745bf9640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72c0052e0 0x7fd72c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd7453f8640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd72c0039b0 0x7fd72c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd744bf7640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c004800 0x7fd72c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd745bf9640 1 -- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c004800 msgr2=0x7fd72c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd745bf9640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c004800 0x7fd72c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd745bf9640 1 -- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd72c0039b0 msgr2=0x7fd72c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd745bf9640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd72c0039b0 0x7fd72c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.887+0000 7fd745bf9640 1 -- 172.21.15.94:0/1048561322 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c003520 con 0x7fd72c0052e0 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd745bf9640 1 -- 172.21.15.94:0/1048561322 reap_dead start 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd745bf9640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72c0052e0 0x7fd72c00b9c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd738001e20 tx=0x7fd72c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fde4b8a5702a5c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738001a70 con 0x7fd72c0052e0 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd738003b70 con 0x7fd72c0052e0 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd737fff640 1 -- 172.21.15.94:0/1048561322 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7380050d0 con 0x7fd72c0052e0 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd74765c640 1 -- 172.21.15.94:0/1048561322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72c0052e0 msgr2=0x7fd72c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd74765c640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72c0052e0 0x7fd72c00b9c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd738001e20 tx=0x7fd72c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd74765c640 1 -- 172.21.15.94:0/1048561322 shutdown_connections 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd74765c640 1 --2- 172.21.15.94:0/1048561322 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd72c0052e0 0x7fd72c00b9c0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.888+0000 7fd74765c640 1 -- 172.21.15.94:0/1048561322 >> 172.21.15.94:0/1048561322 conn(0x7fd7400c6710 msgr2=0x7fd7400c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.889+0000 7fd74765c640 1 -- 172.21.15.94:0/1048561322 shutdown_connections 2024-03-06T07:25:32.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.889+0000 7fd74765c640 1 -- 172.21.15.94:0/1048561322 wait complete. 2024-03-06T07:25:32.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 Processor -- start 2024-03-06T07:25:32.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 -- start start 2024-03-06T07:25:32.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c0052e0 0x7fd7401e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7400a5cb0 0x7fd74006d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd740072540 0x7fd7400729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7401554a0 con 0x7fd740072540 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400c1520 con 0x7fd7400a5cb0 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd74765c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd740155620 con 0x7fd72c0052e0 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd745bf9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd740072540 0x7fd7400729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd745bf9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd740072540 0x7fd7400729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60168/0 (socket says 172.21.15.94:60168) 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd745bf9640 1 -- 172.21.15.94:0/4236095059 learned_addr learned my addr 172.21.15.94:0/4236095059 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd7453f8640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c0052e0 0x7fd7401e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.890+0000 7fd744bf7640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7400a5cb0 0x7fd74006d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd745bf9640 1 -- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c0052e0 msgr2=0x7fd7401e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd745bf9640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c0052e0 0x7fd7401e7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd745bf9640 1 -- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7400a5cb0 msgr2=0x7fd74006d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd745bf9640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7400a5cb0 0x7fd74006d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd745bf9640 1 -- 172.21.15.94:0/4236095059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd740151c50 con 0x7fd740072540 2024-03-06T07:25:32.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd745bf9640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd740072540 0x7fd7400729d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fd738001f40 tx=0x7fd7380014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd735ffb640 1 -- 172.21.15.94:0/4236095059 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738006b50 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd735ffb640 1 -- 172.21.15.94:0/4236095059 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd738006cf0 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd740151f30 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd735ffb640 1 -- 172.21.15.94:0/4236095059 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738007250 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.891+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7401524f0 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.892+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74004f330 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.893+0000 7fd735ffb640 1 -- 172.21.15.94:0/4236095059 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7380073f0 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.893+0000 7fd735ffb640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd72406b4d0 0x7fd72406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.893+0000 7fd735ffb640 1 -- 172.21.15.94:0/4236095059 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fd7380ab900 con 0x7fd740072540 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.893+0000 7fd7453f8640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd72406b4d0 0x7fd72406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.894+0000 7fd7453f8640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd72406b4d0 0x7fd72406d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009b40 tx=0x7fd73c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:32.897+0000 7fd735ffb640 1 -- 172.21.15.94:0/4236095059 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd738033030 con 0x7fd740072540 2024-03-06T07:25:33.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.106+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd740068450 con 0x7fd740072540 2024-03-06T07:25:33.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.107+0000 7fd735ffb640 1 -- 172.21.15.94:0/4236095059 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+3711 (secure 0 0 0) 0x7fd738031e90 con 0x7fd740072540 2024-03-06T07:25:33.109 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:33.109 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":167,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8704,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3062923209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3062923209},{"type":"v1","addr":"172.21.15.145:6837","nonce":3062923209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9432,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3109563835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3109563835},{"type":"v1","addr":"172.21.15.94:6834","nonce":3109563835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":9435,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3468378320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3468378320},{"type":"v1","addr":"172.21.15.94:6835","nonce":3468378320}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155},{"gid":37715,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3364424747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3364424747},{"type":"v1","addr":"172.21.15.145:6836","nonce":3364424747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":167}],"filesystems":[]} 2024-03-06T07:25:33.109 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 167 2024-03-06T07:25:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd72406b4d0 msgr2=0x7fd72406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd72406b4d0 0x7fd72406d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009b40 tx=0x7fd73c009190 comp rx=0 tx=0).stop 2024-03-06T07:25:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd740072540 msgr2=0x7fd7400729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd740072540 0x7fd7400729d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fd738001f40 tx=0x7fd7380014a0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 shutdown_connections 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd72406b4d0 0x7fd72406d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd740072540 0x7fd7400729d0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7400a5cb0 0x7fd74006d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 --2- 172.21.15.94:0/4236095059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd72c0052e0 0x7fd7401e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 >> 172.21.15.94:0/4236095059 conn(0x7fd7400c6710 msgr2=0x7fd74014b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 shutdown_connections 2024-03-06T07:25:33.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.109+0000 7fd74765c640 1 -- 172.21.15.94:0/4236095059 wait complete. 2024-03-06T07:25:33.121 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:33.122 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:33.122 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:33.122 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:33.122 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:33.122 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:33.123 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:33.123 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:33.123 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:33.123 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:33.123 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:33.123 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:33.123 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:33.124 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:33.131 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:33.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:33.132 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:33.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:33.135 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:33.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:33.136 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:33.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:33.173 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:33.173 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:33.174 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:33.174 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:33.177 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:33.178 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:33.178 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:33.178 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:33.179 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:33.249 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:25:33.250 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:25:33.265 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:25:33.265 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:25:33.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.283+0000 7f8ed53a0640 1 Processor -- start 2024-03-06T07:25:33.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.283+0000 7f8ed53a0640 1 -- start start 2024-03-06T07:25:33.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.284+0000 7f8ed53a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c8950 0x7f8ed01559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.284+0000 7f8ed53a0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed00c52b0 con 0x7f8ed01523d0 2024-03-06T07:25:33.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.284+0000 7f8ed53a0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed00c5590 con 0x7f8ed00c78f0 2024-03-06T07:25:33.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.284+0000 7f8ed53a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00c5870 con 0x7f8ed00c8950 2024-03-06T07:25:33.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.284+0000 7f8ecffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8ed01523d0 0x7f8ed00cabc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36978/0 (socket says 172.21.15.94:36978) 2024-03-06T07:25:33.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.284+0000 7f8ecffff640 1 -- 172.21.15.94:0/1889560749 learned_addr learned my addr 172.21.15.94:0/1889560749 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:33.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.284+0000 7f8ecf7fe640 1 --2- 172.21.15.94:0/1889560749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c8950 0x7f8ed01559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.285+0000 7f8eceffd640 1 -- 172.21.15.94:0/1889560749 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650037805 0 0) 0x7f8ed00c52b0 con 0x7f8ed01523d0 2024-03-06T07:25:33.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.285+0000 7f8ecf7fe640 1 -- 172.21.15.94:0/1889560749 >> v1:172.21.15.145:6789/0 conn(0x7f8ed00c78f0 legacy=0x7f8ed015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.285+0000 7f8ecf7fe640 1 -- 172.21.15.94:0/1889560749 >> v1:172.21.15.94:6789/0 conn(0x7f8ed01523d0 legacy=0x7f8ed00cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.285+0000 7f8ecf7fe640 1 -- 172.21.15.94:0/1889560749 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed00c73d0 con 0x7f8ed00c8950 2024-03-06T07:25:33.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.285+0000 7f8ecf7fe640 1 --2- 172.21.15.94:0/1889560749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c8950 0x7f8ed01559c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097d0 tx=0x7f8ec402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ad43edcdd86a6a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8eceffd640 1 -- 172.21.15.94:0/1889560749 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4035020 con 0x7f8ed00c8950 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8eceffd640 1 -- 172.21.15.94:0/1889560749 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec4031800 con 0x7f8ed00c8950 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8eceffd640 1 -- 172.21.15.94:0/1889560749 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4031bd0 con 0x7f8ed00c8950 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8ed53a0640 1 -- 172.21.15.94:0/1889560749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c8950 msgr2=0x7f8ed01559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/1889560749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c8950 0x7f8ed01559c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097d0 tx=0x7f8ec402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8ed53a0640 1 -- 172.21.15.94:0/1889560749 shutdown_connections 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/1889560749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c8950 0x7f8ed01559c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8ed53a0640 1 -- 172.21.15.94:0/1889560749 >> 172.21.15.94:0/1889560749 conn(0x7f8ed00728d0 msgr2=0x7f8ed0072ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8ed53a0640 1 -- 172.21.15.94:0/1889560749 shutdown_connections 2024-03-06T07:25:33.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.286+0000 7f8ed53a0640 1 -- 172.21.15.94:0/1889560749 wait complete. 2024-03-06T07:25:33.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.287+0000 7f8ed53a0640 1 Processor -- start 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.288+0000 7f8ed53a0640 1 -- start start 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.288+0000 7f8ed53a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c78f0 0x7f8ed0148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.288+0000 7f8ed53a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed00c8950 0x7f8ed014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.288+0000 7f8ed53a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed01523d0 0x7f8ed014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.288+0000 7f8ed53a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00cb2d0 con 0x7f8ed01523d0 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.288+0000 7f8ed53a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00be0d0 con 0x7f8ed00c8950 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.288+0000 7f8ed53a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00cb450 con 0x7f8ed00c78f0 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed01523d0 0x7f8ed014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed00c8950 0x7f8ed014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ecffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c78f0 0x7f8ed0148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ecf7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed00c8950 0x7f8ed014d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:53182/0 (socket says 172.21.15.94:53182) 2024-03-06T07:25:33.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed01523d0 0x7f8ed014d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60202/0 (socket says 172.21.15.94:60202) 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 -- 172.21.15.94:0/523072833 learned_addr learned my addr 172.21.15.94:0/523072833 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 -- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c78f0 msgr2=0x7f8ed0148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c78f0 0x7f8ed0148630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 -- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed00c8950 msgr2=0x7f8ed014d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed00c8950 0x7f8ed014d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 -- 172.21.15.94:0/523072833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0156460 con 0x7f8ed01523d0 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ecf7fe640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed00c8950 0x7f8ed014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed4b9f640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed01523d0 0x7f8ed014d980 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800c7a0 tx=0x7f8ec800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8eccff9640 1 -- 172.21.15.94:0/523072833 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8003b70 con 0x7f8ed01523d0 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8eccff9640 1 -- 172.21.15.94:0/523072833 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec8003d30 con 0x7f8ed01523d0 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8eccff9640 1 -- 172.21.15.94:0/523072833 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800aa90 con 0x7f8ed01523d0 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed014f0d0 con 0x7f8ed01523d0 2024-03-06T07:25:33.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.289+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed014dec0 con 0x7f8ed01523d0 2024-03-06T07:25:33.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.291+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed00693c0 con 0x7f8ed01523d0 2024-03-06T07:25:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.291+0000 7f8eccff9640 1 -- 172.21.15.94:0/523072833 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ec800ac30 con 0x7f8ed01523d0 2024-03-06T07:25:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.291+0000 7f8eccff9640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ebc06b4d0 0x7f8ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.291+0000 7f8eccff9640 1 -- 172.21.15.94:0/523072833 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f8ec8089e00 con 0x7f8ed01523d0 2024-03-06T07:25:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.291+0000 7f8ecffff640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ebc06b4d0 0x7f8ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.292+0000 7f8ecffff640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ebc06b4d0 0x7f8ebc06d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000f60 tx=0x7f8ec0007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.295+0000 7f8eccff9640 1 -- 172.21.15.94:0/523072833 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ec8058830 con 0x7f8ed01523d0 2024-03-06T07:25:33.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:33.484+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8ed006ccd0 con 0x7f8ed01523d0 2024-03-06T07:25:34.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.332+0000 7f8eccff9640 1 -- 172.21.15.94:0/523072833 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v178) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8ec805c8d0 con 0x7f8ed01523d0 2024-03-06T07:25:34.334 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:25:34.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ebc06b4d0 msgr2=0x7f8ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ebc06b4d0 0x7f8ebc06d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0000f60 tx=0x7f8ec0007e40 comp rx=0 tx=0).stop 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed01523d0 msgr2=0x7f8ed014d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed01523d0 0x7f8ed014d980 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800c7a0 tx=0x7f8ec800cc60 comp rx=0 tx=0).stop 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 shutdown_connections 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ebc06b4d0 0x7f8ebc06d990 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ed01523d0 0x7f8ed014d980 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ed00c8950 0x7f8ed014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 --2- 172.21.15.94:0/523072833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ed00c78f0 0x7f8ed0148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 >> 172.21.15.94:0/523072833 conn(0x7f8ed00728d0 msgr2=0x7f8ed006c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 shutdown_connections 2024-03-06T07:25:34.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.335+0000 7f8ed53a0640 1 -- 172.21.15.94:0/523072833 wait complete. 2024-03-06T07:25:34.348 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:34.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.448+0000 7fd429793640 1 Processor -- start 2024-03-06T07:25:34.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.448+0000 7fd429793640 1 -- start start 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd429793640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4241592d0 0x7fd4241552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd429793640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd424152d40 con 0x7fd424152960 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd429793640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4240c1b50 con 0x7fd4240ac260 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd429793640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd424066e30 con 0x7fd4241592d0 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd4227fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4241592d0 0x7fd4241552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd422ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd424152960 0x7fd42406dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36980/0 (socket says 172.21.15.94:36980) 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd422ffd640 1 -- 172.21.15.94:0/3584502003 learned_addr learned my addr 172.21.15.94:0/3584502003 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd4227fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd4241592d0 0x7fd4241552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60212/0 (socket says 172.21.15.94:60212) 2024-03-06T07:25:34.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd4227fc640 1 -- 172.21.15.94:0/3584502003 >> v1:172.21.15.145:6789/0 conn(0x7fd4240ac260 legacy=0x7fd424155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd4227fc640 1 -- 172.21.15.94:0/3584502003 >> v1:172.21.15.94:6789/0 conn(0x7fd424152960 legacy=0x7fd42406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd4227fc640 1 -- 172.21.15.94:0/3584502003 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4240c1dc0 con 0x7fd4241592d0 2024-03-06T07:25:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.449+0000 7fd421ffb640 1 -- 172.21.15.94:0/3584502003 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912537021 0 0) 0x7fd424152d40 con 0x7fd424152960 2024-03-06T07:25:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.450+0000 7fd4227fc640 1 --2- 172.21.15.94:0/3584502003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4241592d0 0x7fd4241552d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fd4180097a0 tx=0x7fd41802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2aa4959e8c55889 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.450+0000 7fd421ffb640 1 -- 172.21.15.94:0/3584502003 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd418035020 con 0x7fd4241592d0 2024-03-06T07:25:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.450+0000 7fd421ffb640 1 -- 172.21.15.94:0/3584502003 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd418031830 con 0x7fd4241592d0 2024-03-06T07:25:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.450+0000 7fd421ffb640 1 -- 172.21.15.94:0/3584502003 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd418031c00 con 0x7fd4241592d0 2024-03-06T07:25:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.451+0000 7fd429793640 1 -- 172.21.15.94:0/3584502003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4241592d0 msgr2=0x7fd4241552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.451+0000 7fd429793640 1 --2- 172.21.15.94:0/3584502003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4241592d0 0x7fd4241552d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fd4180097a0 tx=0x7fd41802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.451+0000 7fd429793640 1 -- 172.21.15.94:0/3584502003 shutdown_connections 2024-03-06T07:25:34.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.451+0000 7fd429793640 1 --2- 172.21.15.94:0/3584502003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4241592d0 0x7fd4241552d0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.451+0000 7fd429793640 1 -- 172.21.15.94:0/3584502003 >> 172.21.15.94:0/3584502003 conn(0x7fd4240c70b0 msgr2=0x7fd4240c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.451+0000 7fd429793640 1 -- 172.21.15.94:0/3584502003 shutdown_connections 2024-03-06T07:25:34.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.451+0000 7fd429793640 1 -- 172.21.15.94:0/3584502003 wait complete. 2024-03-06T07:25:34.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.452+0000 7fd429793640 1 Processor -- start 2024-03-06T07:25:34.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.452+0000 7fd429793640 1 -- start start 2024-03-06T07:25:34.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4240ac260 0x7fd4241deb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd424152960 0x7fd424152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd42414eda0 0x7fd4241534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42406e3a0 con 0x7fd4240ac260 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd424159760 con 0x7fd424152960 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42406e520 con 0x7fd42414eda0 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4240ac260 0x7fd4241deb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4240ac260 0x7fd4241deb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60218/0 (socket says 172.21.15.94:60218) 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 -- 172.21.15.94:0/1805420398 learned_addr learned my addr 172.21.15.94:0/1805420398 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd4227fc640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd424152960 0x7fd424152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 -- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd42414eda0 msgr2=0x7fd4241534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd42414eda0 0x7fd4241534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 -- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd424152960 msgr2=0x7fd424152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd424152960 0x7fd424152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 -- 172.21.15.94:0/1805420398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424158a00 con 0x7fd4240ac260 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd4227fc640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd424152960 0x7fd424152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd422ffd640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4240ac260 0x7fd4241deb10 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fd41400c4d0 tx=0x7fd41400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd40b7fe640 1 -- 172.21.15.94:0/1805420398 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414018070 con 0x7fd4240ac260 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd40b7fe640 1 -- 172.21.15.94:0/1805420398 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414007500 con 0x7fd4240ac260 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd424153a10 con 0x7fd4240ac260 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd40b7fe640 1 -- 172.21.15.94:0/1805420398 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414004a20 con 0x7fd4240ac260 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.453+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4241569a0 con 0x7fd4240ac260 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.455+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42404f330 con 0x7fd4240ac260 2024-03-06T07:25:34.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.455+0000 7fd40b7fe640 1 -- 172.21.15.94:0/1805420398 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd41401b070 con 0x7fd4240ac260 2024-03-06T07:25:34.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.455+0000 7fd40b7fe640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd41006b4d0 0x7fd41006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.455+0000 7fd40b7fe640 1 -- 172.21.15.94:0/1805420398 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd414089340 con 0x7fd4240ac260 2024-03-06T07:25:34.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.455+0000 7fd4227fc640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd41006b4d0 0x7fd41006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.456+0000 7fd4227fc640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd41006b4d0 0x7fd41006d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd418036000 tx=0x7fd41803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.460+0000 7fd40b7fe640 1 -- 172.21.15.94:0/1805420398 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd414058380 con 0x7fd4240ac260 2024-03-06T07:25:34.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.694+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd424156270 con 0x7fd4240ac260 2024-03-06T07:25:34.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.694+0000 7fd40b7fe640 1 -- 172.21.15.94:0/1805420398 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fd414057c80 con 0x7fd4240ac260 2024-03-06T07:25:34.696 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:34.696 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:25:34.696 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:25:34.697 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:25:34.698 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.699 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.700 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.701 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.702 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.703 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.704 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:25:34.705 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.706 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:34.707 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:25:34.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.708 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:34.708 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:25:34.708 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:25:34.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd41006b4d0 msgr2=0x7fd41006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd41006b4d0 0x7fd41006d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd418036000 tx=0x7fd41803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:34.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4240ac260 msgr2=0x7fd4241deb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4240ac260 0x7fd4241deb10 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fd41400c4d0 tx=0x7fd41400c990 comp rx=0 tx=0).stop 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 shutdown_connections 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd41006b4d0 0x7fd41006d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd42414eda0 0x7fd4241534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd424152960 0x7fd424152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 --2- 172.21.15.94:0/1805420398 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4240ac260 0x7fd4241deb10 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 >> 172.21.15.94:0/1805420398 conn(0x7fd4240c70b0 msgr2=0x7fd4240c5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 shutdown_connections 2024-03-06T07:25:34.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.696+0000 7fd429793640 1 -- 172.21.15.94:0/1805420398 wait complete. 2024-03-06T07:25:34.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.809+0000 7f4bd6593640 1 Processor -- start 2024-03-06T07:25:34.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.809+0000 7f4bd6593640 1 -- start start 2024-03-06T07:25:34.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.809+0000 7f4bd6593640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 0x7f4bd006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.809+0000 7f4bd6593640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4bd00c1320 0x7f4bd006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.809+0000 7f4bd6593640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd0153a60 con 0x7f4bd00c6000 2024-03-06T07:25:34.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.809+0000 7f4bd6593640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd0153d40 con 0x7f4bd00c1320 2024-03-06T07:25:34.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.809+0000 7f4bd6593640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd006c2c0 con 0x7f4bd0150580 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bd4b30640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4bd00c6000 0x7f4bd01566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36996/0 (socket says 172.21.15.94:36996) 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bd4b30640 1 -- 172.21.15.94:0/1580944659 learned_addr learned my addr 172.21.15.94:0/1580944659 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/1580944659 >> v2:172.21.15.145:3300/0 conn(0x7f4bd00c1320 0x7f4bd006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bcffff640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 0x7f4bd006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267445330 0 0) 0x7f4bd0153a60 con 0x7f4bd00c6000 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb8003520 con 0x7f4bd00c6000 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159133532 0 0) 0x7f4bb8003520 con 0x7f4bd00c6000 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 msgr2=0x7f4bd006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 0x7f4bd006dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 >> v2:172.21.15.145:3300/0 conn(0x7f4bd00c1320 msgr2=0x7f4bd006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 --2- 172.21.15.94:0/1580944659 >> v2:172.21.15.145:3300/0 conn(0x7f4bd00c1320 0x7f4bd006e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd006c5a0 con 0x7f4bd00c6000 2024-03-06T07:25:34.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f4bc80019f0 con 0x7f4bd00c6000 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 >> v1:172.21.15.94:6789/0 conn(0x7f4bd00c6000 legacy=0x7f4bd01566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb80039b0 0x7f4bb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 0x7f4bb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd0153d40 con 0x7f4bb8004800 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd006c2c0 con 0x7f4bb80039b0 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb800b9c0 con 0x7f4bd0150580 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.810+0000 7f4bcffff640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb80039b0 0x7f4bb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bd4b30640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 0x7f4bb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bcf7fe640 1 -- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 msgr2=0x7f4bb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0150580 0x7f4bb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bcf7fe640 1 -- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb80039b0 msgr2=0x7f4bb8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb80039b0 0x7f4bb8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bcf7fe640 1 -- 172.21.15.94:0/1580944659 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8003520 con 0x7f4bb8004800 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009af0 tx=0x7f4bb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d5cf9045065e46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.811+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4039070 con 0x7f4bb8004800 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc402c710 con 0x7f4bb8004800 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bceffd640 1 -- 172.21.15.94:0/1580944659 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4009690 con 0x7f4bb8004800 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bd6593640 1 -- 172.21.15.94:0/1580944659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bb8004800 msgr2=0x7f4bb8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bd6593640 1 --2- 172.21.15.94:0/1580944659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bb8004800 0x7f4bb8004c80 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009af0 tx=0x7f4bb8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bd4b30640 1 -- 172.21.15.94:0/1580944659 reap_dead start 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bd6593640 1 -- 172.21.15.94:0/1580944659 shutdown_connections 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bd6593640 1 -- 172.21.15.94:0/1580944659 >> 172.21.15.94:0/1580944659 conn(0x7f4bd00c6710 msgr2=0x7f4bd00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bd6593640 1 -- 172.21.15.94:0/1580944659 shutdown_connections 2024-03-06T07:25:34.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.812+0000 7f4bd6593640 1 -- 172.21.15.94:0/1580944659 wait complete. 2024-03-06T07:25:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.813+0000 7f4bd6593640 1 Processor -- start 2024-03-06T07:25:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.813+0000 7f4bd6593640 1 -- start start 2024-03-06T07:25:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bd6593640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb8004800 0x7f4bd01511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bd6593640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd014e060 0x7f4bd00bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bd6593640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0145a70 0x7f4bd0145f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bd6593640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00c0dc0 con 0x7f4bd014e060 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bd6593640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd00c0c40 con 0x7f4bb8004800 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bd6593640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd006dd10 con 0x7f4bd0145a70 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd014e060 0x7f4bd00bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd014e060 0x7f4bd00bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60236/0 (socket says 172.21.15.94:60236) 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcf7fe640 1 -- 172.21.15.94:0/2414960023 learned_addr learned my addr 172.21.15.94:0/2414960023 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcffff640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb8004800 0x7f4bd01511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bd4b30640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0145a70 0x7f4bd0145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcf7fe640 1 -- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0145a70 msgr2=0x7f4bd0145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0145a70 0x7f4bd0145f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcf7fe640 1 -- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb8004800 msgr2=0x7f4bd01511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.814+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb8004800 0x7f4bd01511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bcf7fe640 1 -- 172.21.15.94:0/2414960023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd014de10 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bcffff640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb8004800 0x7f4bd01511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bcf7fe640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd014e060 0x7f4bd00bfdb0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009af0 tx=0x7f4bc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bccff9640 1 -- 172.21.15.94:0/2414960023 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4039050 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bccff9640 1 -- 172.21.15.94:0/2414960023 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc4045070 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd0151700 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bccff9640 1 -- 172.21.15.94:0/2414960023 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4003a50 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.815+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd01ee0b0 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.816+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd004f330 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.816+0000 7f4bccff9640 1 -- 172.21.15.94:0/2414960023 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bc4003bf0 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.817+0000 7f4bccff9640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.817+0000 7f4bccff9640 1 -- 172.21.15.94:0/2414960023 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4bc40abb80 con 0x7f4bd014e060 2024-03-06T07:25:34.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.817+0000 7f4bcffff640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.817+0000 7f4bcffff640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009ae0 tx=0x7f4bc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.821+0000 7f4bccff9640 1 -- 172.21.15.94:0/2414960023 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bc404e020 con 0x7f4bd014e060 2024-03-06T07:25:34.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.995+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bd00c5de0 con 0x7f4bd014e060 2024-03-06T07:25:34.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.996+0000 7f4bccff9640 1 -- 172.21.15.94:0/2414960023 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+13174 (secure 0 0 0) 0x7f4bd014de10 con 0x7f4bd014e060 2024-03-06T07:25:34.998 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:34.998 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":178,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:34.332293+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bbc06b500 msgr2=0x7f4bbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009ae0 tx=0x7f4bc0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd014e060 msgr2=0x7f4bd00bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd014e060 0x7f4bd00bfdb0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009af0 tx=0x7f4bc4004730 comp rx=0 tx=0).stop 2024-03-06T07:25:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 shutdown_connections 2024-03-06T07:25:35.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd0145a70 0x7f4bd0145f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd014e060 0x7f4bd00bfdb0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 --2- 172.21.15.94:0/2414960023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bb8004800 0x7f4bd01511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 >> 172.21.15.94:0/2414960023 conn(0x7f4bd00c6710 msgr2=0x7f4bd006e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 shutdown_connections 2024-03-06T07:25:35.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:34.998+0000 7f4bd6593640 1 -- 172.21.15.94:0/2414960023 wait complete. 2024-03-06T07:25:35.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:35.011 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:35.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.112+0000 7f9d362c9640 1 Processor -- start 2024-03-06T07:25:35.114 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.112+0000 7f9d362c9640 1 -- start start 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d362c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9d300c48c0 0x7f9d301558e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d362c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d300c6500 con 0x7f9d301527a0 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d362c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d300c5c60 con 0x7f9d300c1b20 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d362c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d300c3dc0 con 0x7f9d300c48c0 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9d301527a0 0x7f9d300c4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37008/0 (socket says 172.21.15.94:37008) 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2ffff640 1 -- 172.21.15.94:0/1696250543 learned_addr learned my addr 172.21.15.94:0/1696250543 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2f7fe640 1 --2- 172.21.15.94:0/1696250543 >> v2:172.21.15.145:3300/0 conn(0x7f9d300c48c0 0x7f9d301558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301092063 0 0) 0x7f9d300c6500 con 0x7f9d301527a0 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d18003520 con 0x7f9d301527a0 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 771302754 0 0) 0x7f9d18003520 con 0x7f9d301527a0 2024-03-06T07:25:35.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 >> v2:172.21.15.145:3300/0 conn(0x7f9d300c48c0 msgr2=0x7f9d301558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 --2- 172.21.15.94:0/1696250543 >> v2:172.21.15.145:3300/0 conn(0x7f9d300c48c0 0x7f9d301558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 >> v1:172.21.15.145:6789/0 conn(0x7f9d300c1b20 legacy=0x7f9d3006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d300c5f40 con 0x7f9d301527a0 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.113+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9d20001670 con 0x7f9d301527a0 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 >> v1:172.21.15.94:6789/0 conn(0x7f9d301527a0 legacy=0x7f9d300c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2effd640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18003a00 0x7f9d18003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2effd640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d18004800 0x7f9d18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2effd640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d18005210 0x7f9d1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d300c3dc0 con 0x7f9d18005210 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1800c160 con 0x7f9d18003a00 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1800c360 con 0x7f9d18004800 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d34866640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d18005210 0x7f9d1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2ffff640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18003a00 0x7f9d18003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d2f7fe640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d18004800 0x7f9d18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d34866640 1 -- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d18004800 msgr2=0x7f9d18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d34866640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d18004800 0x7f9d18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d34866640 1 -- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18003a00 msgr2=0x7f9d18003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d34866640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18003a00 0x7f9d18003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.114+0000 7f9d34866640 1 -- 172.21.15.94:0/1696250543 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d18003520 con 0x7f9d18005210 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d34866640 1 -- 172.21.15.94:0/1696250543 reap_dead start 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d34866640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d18005210 0x7f9d1800b9c0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9d28001e20 tx=0x7f9d18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9faa4f054dad1769 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28001a70 con 0x7f9d18005210 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28003b70 con 0x7f9d18005210 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d2effd640 1 -- 172.21.15.94:0/1696250543 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d280050d0 con 0x7f9d18005210 2024-03-06T07:25:35.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d362c9640 1 -- 172.21.15.94:0/1696250543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d18005210 msgr2=0x7f9d1800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d362c9640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d18005210 0x7f9d1800b9c0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f9d28001e20 tx=0x7f9d18005830 comp rx=0 tx=0).stop 2024-03-06T07:25:35.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d362c9640 1 -- 172.21.15.94:0/1696250543 shutdown_connections 2024-03-06T07:25:35.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d362c9640 1 --2- 172.21.15.94:0/1696250543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d18005210 0x7f9d1800b9c0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d362c9640 1 -- 172.21.15.94:0/1696250543 >> 172.21.15.94:0/1696250543 conn(0x7f9d300c6710 msgr2=0x7f9d300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d362c9640 1 -- 172.21.15.94:0/1696250543 shutdown_connections 2024-03-06T07:25:35.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.115+0000 7f9d362c9640 1 -- 172.21.15.94:0/1696250543 wait complete. 2024-03-06T07:25:35.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.116+0000 7f9d362c9640 1 Processor -- start 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d362c9640 1 -- start start 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d362c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18005210 0x7f9d3014d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d362c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d30154de0 0x7f9d30149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d362c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d30072e60 0x7f9d30149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d362c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30152b80 con 0x7f9d30072e60 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d362c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30155450 con 0x7f9d18005210 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d362c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d300c58d0 con 0x7f9d30154de0 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d34866640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d30072e60 0x7f9d30149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d34866640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d30072e60 0x7f9d30149ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60254/0 (socket says 172.21.15.94:60254) 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d34866640 1 -- 172.21.15.94:0/668747329 learned_addr learned my addr 172.21.15.94:0/668747329 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d2ffff640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18005210 0x7f9d3014d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.117+0000 7f9d2f7fe640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d30154de0 0x7f9d30149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d34866640 1 -- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d30154de0 msgr2=0x7f9d30149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d34866640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d30154de0 0x7f9d30149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d34866640 1 -- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18005210 msgr2=0x7f9d3014d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d34866640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18005210 0x7f9d3014d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d34866640 1 -- 172.21.15.94:0/668747329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3014aa60 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d34866640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d30072e60 0x7f9d30149ca0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9d2802c020 tx=0x7f9d280014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d2cff9640 1 -- 172.21.15.94:0/668747329 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28006b50 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d2cff9640 1 -- 172.21.15.94:0/668747329 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28006cf0 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d30154650 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d2cff9640 1 -- 172.21.15.94:0/668747329 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28007230 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.118+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d301eebf0 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.119+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3004f330 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.119+0000 7f9d2cff9640 1 -- 172.21.15.94:0/668747329 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d280073d0 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.120+0000 7f9d2cff9640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.120+0000 7f9d2cff9640 1 -- 172.21.15.94:0/668747329 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9d280ab900 con 0x7f9d30072e60 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.120+0000 7f9d2ffff640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.121+0000 7f9d2ffff640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9d3006de40 tx=0x7f9d20001df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.124+0000 7f9d2cff9640 1 -- 172.21.15.94:0/668747329 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d28033030 con 0x7f9d30072e60 2024-03-06T07:25:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.294+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d3014ac30 con 0x7f9d30072e60 2024-03-06T07:25:35.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.294+0000 7f9d2cff9640 1 -- 172.21.15.94:0/668747329 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d2807a460 con 0x7f9d30072e60 2024-03-06T07:25:35.296 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:25:35.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d1c06b4d0 msgr2=0x7f9d1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f9d3006de40 tx=0x7f9d20001df0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d30072e60 msgr2=0x7f9d30149ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d30072e60 0x7f9d30149ca0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f9d2802c020 tx=0x7f9d280014a0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 shutdown_connections 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d1c06b4d0 0x7f9d1c06d990 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d30072e60 0x7f9d30149ca0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d30154de0 0x7f9d30149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 --2- 172.21.15.94:0/668747329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d18005210 0x7f9d3014d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 >> 172.21.15.94:0/668747329 conn(0x7f9d300c6710 msgr2=0x7f9d30157670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 shutdown_connections 2024-03-06T07:25:35.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.296+0000 7f9d362c9640 1 -- 172.21.15.94:0/668747329 wait complete. 2024-03-06T07:25:35.309 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:35.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.407+0000 7f4fa5c95640 1 Processor -- start 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.407+0000 7f4fa5c95640 1 -- start start 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4fa5c95640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4fa01508c0 0x7f4fa00c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4fa5c95640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa0157340 con 0x7f4fa0072720 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4fa5c95640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa0156d80 con 0x7f4fa006f7b0 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4fa5c95640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa0156b40 con 0x7f4fa01508c0 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4f9ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4fa01508c0 0x7f4fa00c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4f9f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4fa0072720 0x7f4fa0156430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37024/0 (socket says 172.21.15.94:37024) 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4f9f7fe640 1 -- 172.21.15.94:0/2974977923 learned_addr learned my addr 172.21.15.94:0/2974977923 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4f9ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4fa01508c0 0x7f4fa00c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60264/0 (socket says 172.21.15.94:60264) 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.408+0000 7f4f9ffff640 1 -- 172.21.15.94:0/2974977923 >> v1:172.21.15.145:6789/0 conn(0x7f4fa006f7b0 legacy=0x7f4fa01501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.409+0000 7f4f9ffff640 1 -- 172.21.15.94:0/2974977923 >> v1:172.21.15.94:6789/0 conn(0x7f4fa0072720 legacy=0x7f4fa0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.409+0000 7f4f9ffff640 1 -- 172.21.15.94:0/2974977923 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa0157060 con 0x7f4fa01508c0 2024-03-06T07:25:35.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.409+0000 7f4f9e7fc640 1 -- 172.21.15.94:0/2974977923 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114841900 0 0) 0x7f4fa0157340 con 0x7f4fa0072720 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.409+0000 7f4f9ffff640 1 --2- 172.21.15.94:0/2974977923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa00c3bd0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009d90 tx=0x7f4f9802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0ed81f28e4b338 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.409+0000 7f4f9e7fc640 1 -- 172.21.15.94:0/2974977923 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98039070 con 0x7f4fa01508c0 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.409+0000 7f4f9e7fc640 1 -- 172.21.15.94:0/2974977923 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9802cbd0 con 0x7f4fa01508c0 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.409+0000 7f4f9e7fc640 1 -- 172.21.15.94:0/2974977923 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98034440 con 0x7f4fa01508c0 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.410+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2974977923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 msgr2=0x7f4fa00c3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.410+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2974977923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa00c3bd0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009d90 tx=0x7f4f9802be20 comp rx=0 tx=0).stop 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.410+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2974977923 shutdown_connections 2024-03-06T07:25:35.413 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.410+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2974977923 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa00c3bd0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.410+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2974977923 >> 172.21.15.94:0/2974977923 conn(0x7f4fa00aa630 msgr2=0x7f4fa00aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.410+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2974977923 shutdown_connections 2024-03-06T07:25:35.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.410+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2974977923 wait complete. 2024-03-06T07:25:35.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.411+0000 7f4fa5c95640 1 Processor -- start 2024-03-06T07:25:35.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.411+0000 7f4fa5c95640 1 -- start start 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4fa5c95640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa006f7b0 0x7f4fa00c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4fa5c95640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa0072720 0x7f4fa00ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4fa5c95640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa01df2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4fa5c95640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa0072b00 con 0x7f4fa01508c0 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4fa5c95640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa00c3740 con 0x7f4fa006f7b0 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4fa5c95640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa0072c80 con 0x7f4fa0072720 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa01df2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa01df2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60280/0 (socket says 172.21.15.94:60280) 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 -- 172.21.15.94:0/2018982018 learned_addr learned my addr 172.21.15.94:0/2018982018 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 -- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa0072720 msgr2=0x7f4fa00ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa0072720 0x7f4fa00ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 -- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa006f7b0 msgr2=0x7f4fa00c1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa006f7b0 0x7f4fa00c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 -- 172.21.15.94:0/2018982018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa01eedb0 con 0x7f4fa01508c0 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f9ffff640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa01df2d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009d90 tx=0x7f4f98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f87fff640 1 -- 172.21.15.94:0/2018982018 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98044020 con 0x7f4fa01508c0 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4f87fff640 1 -- 172.21.15.94:0/2018982018 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9803f470 con 0x7f4fa01508c0 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.412+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa006c320 con 0x7f4fa01508c0 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.413+0000 7f4f87fff640 1 -- 172.21.15.94:0/2018982018 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98039040 con 0x7f4fa01508c0 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.413+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa00c2460 con 0x7f4fa01508c0 2024-03-06T07:25:35.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.413+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa00a8330 con 0x7f4fa01508c0 2024-03-06T07:25:35.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.414+0000 7f4f87fff640 1 -- 172.21.15.94:0/2018982018 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f98047020 con 0x7f4fa01508c0 2024-03-06T07:25:35.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.414+0000 7f4f87fff640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.414+0000 7f4f87fff640 1 -- 172.21.15.94:0/2018982018 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4f980acb90 con 0x7f4fa01508c0 2024-03-06T07:25:35.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.415+0000 7f4f9f7fe640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.415+0000 7f4f9f7fe640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4f90007fd0 tx=0x7f4f90007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.419+0000 7f4f87fff640 1 -- 172.21.15.94:0/2018982018 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fa00a8330 con 0x7f4fa01508c0 2024-03-06T07:25:35.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.591+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4fa006e090 con 0x7f4fa01508c0 2024-03-06T07:25:35.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.592+0000 7f4f87fff640 1 -- 172.21.15.94:0/2018982018 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v178) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4f9804e030 con 0x7f4fa01508c0 2024-03-06T07:25:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:35.594 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:25:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:25:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:25:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:25:35.594 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:25:35.594 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.593+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f8c06b4d0 msgr2=0x7f4f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.593+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4f90007fd0 tx=0x7f4f90007920 comp rx=0 tx=0).stop 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.593+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 msgr2=0x7f4fa01df2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.593+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa01df2d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009d90 tx=0x7f4f98004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 shutdown_connections 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f8c06b4d0 0x7f4f8c06d990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4fa01508c0 0x7f4fa01df2d0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa0072720 0x7f4fa00ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 --2- 172.21.15.94:0/2018982018 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa006f7b0 0x7f4fa00c1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 >> 172.21.15.94:0/2018982018 conn(0x7f4fa00aa630 msgr2=0x7f4fa00a9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 shutdown_connections 2024-03-06T07:25:35.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.594+0000 7f4fa5c95640 1 -- 172.21.15.94:0/2018982018 wait complete. 2024-03-06T07:25:35.606 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:35.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.708+0000 7fc11da88640 1 Processor -- start 2024-03-06T07:25:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.708+0000 7fc11da88640 1 -- start start 2024-03-06T07:25:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc11da88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc11814bb20 0x7fc1180c5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc11da88640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1180c2330 0x7fc1180c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc11da88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc118159260 con 0x7fc1181550c0 2024-03-06T07:25:35.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc11da88640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1180c3070 con 0x7fc1180c2330 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc11da88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1180a80e0 con 0x7fc11814bb20 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc10effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1180c2330 0x7fc1180c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc10effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc1180c2330 0x7fc1180c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60296/0 (socket says 172.21.15.94:60296) 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc10effd640 1 -- 172.21.15.94:0/1446649892 learned_addr learned my addr 172.21.15.94:0/1446649892 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc1177fe640 1 --2- 172.21.15.94:0/1446649892 >> v2:172.21.15.145:3300/0 conn(0x7fc11814bb20 0x7fc1180c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc10effd640 1 -- 172.21.15.94:0/1446649892 >> v2:172.21.15.145:3300/0 conn(0x7fc11814bb20 msgr2=0x7fc1180c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.709+0000 7fc10effd640 1 --2- 172.21.15.94:0/1446649892 >> v2:172.21.15.145:3300/0 conn(0x7fc11814bb20 0x7fc1180c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.710+0000 7fc10effd640 1 -- 172.21.15.94:0/1446649892 >> v1:172.21.15.145:6789/0 conn(0x7fc1181550c0 legacy=0x7fc118155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.710+0000 7fc10effd640 1 -- 172.21.15.94:0/1446649892 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180a7d70 con 0x7fc1180c2330 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.710+0000 7fc10effd640 1 --2- 172.21.15.94:0/1446649892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1180c2330 0x7fc1180c5850 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fc1180735a0 tx=0x7fc10002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a220371f5d88be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.710+0000 7fc116ffd640 1 -- 172.21.15.94:0/1446649892 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10002e070 con 0x7fc1180c2330 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.710+0000 7fc116ffd640 1 -- 172.21.15.94:0/1446649892 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc100037070 con 0x7fc1180c2330 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.710+0000 7fc116ffd640 1 -- 172.21.15.94:0/1446649892 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100031a00 con 0x7fc1180c2330 2024-03-06T07:25:35.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 -- 172.21.15.94:0/1446649892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1180c2330 msgr2=0x7fc1180c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 --2- 172.21.15.94:0/1446649892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1180c2330 0x7fc1180c5850 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fc1180735a0 tx=0x7fc10002b880 comp rx=0 tx=0).stop 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 -- 172.21.15.94:0/1446649892 shutdown_connections 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 --2- 172.21.15.94:0/1446649892 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1180c2330 0x7fc1180c5850 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 --2- 172.21.15.94:0/1446649892 >> v2:172.21.15.145:3300/0 conn(0x7fc11814bb20 0x7fc1180c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 -- 172.21.15.94:0/1446649892 >> 172.21.15.94:0/1446649892 conn(0x7fc1180c88f0 msgr2=0x7fc1180c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 -- 172.21.15.94:0/1446649892 shutdown_connections 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.711+0000 7fc11da88640 1 -- 172.21.15.94:0/1446649892 wait complete. 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.712+0000 7fc11da88640 1 Processor -- start 2024-03-06T07:25:35.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.712+0000 7fc11da88640 1 -- start start 2024-03-06T07:25:35.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc11da88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1180c2330 0x7fc11814ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc11da88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc11814bb20 0x7fc1181eef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc11da88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1181550c0 0x7fc1180c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc11da88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc118155c60 con 0x7fc1181550c0 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc11da88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11814bf00 con 0x7fc1180c2330 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc11da88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1180c67f0 con 0x7fc11814bb20 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1181550c0 0x7fc1180c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1181550c0 0x7fc1180c45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60304/0 (socket says 172.21.15.94:60304) 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 -- 172.21.15.94:0/4134337312 learned_addr learned my addr 172.21.15.94:0/4134337312 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc1177fe640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1180c2330 0x7fc11814ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc10effd640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc11814bb20 0x7fc1181eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 -- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc11814bb20 msgr2=0x7fc1181eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc11814bb20 0x7fc1181eef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 -- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1180c2330 msgr2=0x7fc11814ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1180c2330 0x7fc11814ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 -- 172.21.15.94:0/4134337312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11814ca80 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.713+0000 7fc117fff640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1181550c0 0x7fc1180c45e0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c780 tx=0x7fc11000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.714+0000 7fc114ff9640 1 -- 172.21.15.94:0/4134337312 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc110004b60 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.714+0000 7fc114ff9640 1 -- 172.21.15.94:0/4134337312 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc110004d00 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.714+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11814cd60 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.714+0000 7fc114ff9640 1 -- 172.21.15.94:0/4134337312 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11000aa90 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.714+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1181eb3b0 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.715+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1181eb5b0 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.715+0000 7fc114ff9640 1 -- 172.21.15.94:0/4134337312 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc11000ac30 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.715+0000 7fc114ff9640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f806b4d0 0x7fc0f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.715+0000 7fc114ff9640 1 -- 172.21.15.94:0/4134337312 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc11008a5f0 con 0x7fc1181550c0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.716+0000 7fc1177fe640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f806b4d0 0x7fc0f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.716+0000 7fc1177fe640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f806b4d0 0x7fc0f806d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc108006fd0 tx=0x7fc108008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.720+0000 7fc114ff9640 1 -- 172.21.15.94:0/4134337312 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1181eb5b0 con 0x7fc1181550c0 2024-03-06T07:25:35.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:35.889+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc11804f330 con 0x7fc1181550c0 2024-03-06T07:25:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.340+0000 7fc114ff9640 1 -- 172.21.15.94:0/4134337312 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v179) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc110059630 con 0x7fc1181550c0 2024-03-06T07:25:36.342 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:36.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f806b4d0 msgr2=0x7fc0f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f806b4d0 0x7fc0f806d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fc108006fd0 tx=0x7fc108008040 comp rx=0 tx=0).stop 2024-03-06T07:25:36.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1181550c0 msgr2=0x7fc1180c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1181550c0 0x7fc1180c45e0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c780 tx=0x7fc11000cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:36.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 shutdown_connections 2024-03-06T07:25:36.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc0f806b4d0 0x7fc0f806d990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc1181550c0 0x7fc1180c45e0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc11814bb20 0x7fc1181eef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 --2- 172.21.15.94:0/4134337312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1180c2330 0x7fc11814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 >> 172.21.15.94:0/4134337312 conn(0x7fc1180c88f0 msgr2=0x7fc118155510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 shutdown_connections 2024-03-06T07:25:36.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.342+0000 7fc11da88640 1 -- 172.21.15.94:0/4134337312 wait complete. 2024-03-06T07:25:36.358 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.464+0000 7f9db217a640 1 Processor -- start 2024-03-06T07:25:36.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.464+0000 7f9db217a640 1 -- start start 2024-03-06T07:25:36.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9db217a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9dac0ab310 0x7f9dac0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9db217a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac154330 con 0x7f9dac1538c0 2024-03-06T07:25:36.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9db217a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac153dc0 con 0x7f9dac0a97c0 2024-03-06T07:25:36.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9db217a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dac0c46a0 con 0x7f9dac0ab310 2024-03-06T07:25:36.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9daaffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9dac1538c0 0x7f9dac1494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37034/0 (socket says 172.21.15.94:37034) 2024-03-06T07:25:36.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9daaffd640 1 -- 172.21.15.94:0/962480371 learned_addr learned my addr 172.21.15.94:0/962480371 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9dab7fe640 1 --2- 172.21.15.94:0/962480371 >> v2:172.21.15.145:3300/0 conn(0x7f9dac0ab310 0x7f9dac0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9daa7fc640 1 -- 172.21.15.94:0/962480371 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460766872 0 0) 0x7f9dac154330 con 0x7f9dac1538c0 2024-03-06T07:25:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9daa7fc640 1 -- 172.21.15.94:0/962480371 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d94003520 con 0x7f9dac1538c0 2024-03-06T07:25:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9dab7fe640 1 -- 172.21.15.94:0/962480371 >> v1:172.21.15.145:6789/0 conn(0x7f9dac0a97c0 legacy=0x7f9dac0c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9daa7fc640 1 -- 172.21.15.94:0/962480371 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730364137 0 0) 0x7f9d94003520 con 0x7f9dac1538c0 2024-03-06T07:25:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9dab7fe640 1 -- 172.21.15.94:0/962480371 >> v1:172.21.15.94:6789/0 conn(0x7f9dac1538c0 legacy=0x7f9dac1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.465+0000 7f9dab7fe640 1 -- 172.21.15.94:0/962480371 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac0c5d30 con 0x7f9dac0ab310 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.466+0000 7f9dab7fe640 1 --2- 172.21.15.94:0/962480371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac0ab310 0x7f9dac0ab6f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0097a0 tx=0x7f9d9c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2e47f5603522a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.466+0000 7f9daa7fc640 1 -- 172.21.15.94:0/962480371 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c037070 con 0x7f9dac0ab310 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.466+0000 7f9daa7fc640 1 -- 172.21.15.94:0/962480371 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c030510 con 0x7f9dac0ab310 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.466+0000 7f9daa7fc640 1 -- 172.21.15.94:0/962480371 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0001670 con 0x7f9dac0ab310 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.467+0000 7f9db217a640 1 -- 172.21.15.94:0/962480371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac0ab310 msgr2=0x7f9dac0ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.467+0000 7f9db217a640 1 --2- 172.21.15.94:0/962480371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac0ab310 0x7f9dac0ab6f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0097a0 tx=0x7f9d9c02b790 comp rx=0 tx=0).stop 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.467+0000 7f9db217a640 1 -- 172.21.15.94:0/962480371 shutdown_connections 2024-03-06T07:25:36.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.467+0000 7f9db217a640 1 --2- 172.21.15.94:0/962480371 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac0ab310 0x7f9dac0ab6f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.467+0000 7f9db217a640 1 -- 172.21.15.94:0/962480371 >> 172.21.15.94:0/962480371 conn(0x7f9dac0c6710 msgr2=0x7f9dac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.467+0000 7f9db217a640 1 -- 172.21.15.94:0/962480371 shutdown_connections 2024-03-06T07:25:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.467+0000 7f9db217a640 1 -- 172.21.15.94:0/962480371 wait complete. 2024-03-06T07:25:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.468+0000 7f9db217a640 1 Processor -- start 2024-03-06T07:25:36.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.468+0000 7f9db217a640 1 -- start start 2024-03-06T07:25:36.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9db217a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac0a97c0 0x7f9dac0727f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9db217a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dac0ab310 0x7f9dac1e5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9db217a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac1538c0 0x7f9dac1e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9db217a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0abc30 con 0x7f9dac0a97c0 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9db217a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0c62f0 con 0x7f9dac1538c0 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9db217a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac149bb0 con 0x7f9dac0ab310 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac0a97c0 0x7f9dac0727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac0a97c0 0x7f9dac0727f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60314/0 (socket says 172.21.15.94:60314) 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 -- 172.21.15.94:0/2414858877 learned_addr learned my addr 172.21.15.94:0/2414858877 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dabfff640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac1538c0 0x7f9dac1e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9daaffd640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dac0ab310 0x7f9dac1e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 -- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dac0ab310 msgr2=0x7f9dac1e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dac0ab310 0x7f9dac1e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 -- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac1538c0 msgr2=0x7f9dac1e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac1538c0 0x7f9dac1e5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 -- 172.21.15.94:0/2414858877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac1e9190 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dabfff640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac1538c0 0x7f9dac1e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9daaffd640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dac0ab310 0x7f9dac1e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9dab7fe640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac0a97c0 0x7f9dac0727f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c000c00 tx=0x7f9d9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9d93fff640 1 -- 172.21.15.94:0/2414858877 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c042070 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9d93fff640 1 -- 172.21.15.94:0/2414858877 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c03e870 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dac072d30 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.469+0000 7f9d93fff640 1 -- 172.21.15.94:0/2414858877 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c037040 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.470+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dac14d500 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.471+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dac04f330 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.471+0000 7f9d93fff640 1 -- 172.21.15.94:0/2414858877 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d9c045070 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.471+0000 7f9d93fff640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d9806b4d0 0x7f9d9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.472+0000 7f9d93fff640 1 -- 172.21.15.94:0/2414858877 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f9d9c0abd90 con 0x7f9dac0a97c0 2024-03-06T07:25:36.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.472+0000 7f9daaffd640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d9806b4d0 0x7f9d9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.472+0000 7f9daaffd640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d9806b4d0 0x7f9d9806d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001eb0 tx=0x7f9da0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.476+0000 7f9d93fff640 1 -- 172.21.15.94:0/2414858877 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d9c07ae10 con 0x7f9dac0a97c0 2024-03-06T07:25:36.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:36.648+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9dac0c6ef0 con 0x7f9dac0a97c0 2024-03-06T07:25:37.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.342+0000 7f9d93fff640 1 -- 172.21.15.94:0/2414858877 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v180) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9d9c07a710 con 0x7f9dac0a97c0 2024-03-06T07:25:37.344 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d9806b4d0 msgr2=0x7f9d9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d9806b4d0 0x7f9d9806d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9da0001eb0 tx=0x7f9da0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac0a97c0 msgr2=0x7f9dac0727f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac0a97c0 0x7f9dac0727f0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c000c00 tx=0x7f9d9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 shutdown_connections 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d9806b4d0 0x7f9d9806d990 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dac1538c0 0x7f9dac1e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dac0ab310 0x7f9dac1e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 --2- 172.21.15.94:0/2414858877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9dac0a97c0 0x7f9dac0727f0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 >> 172.21.15.94:0/2414858877 conn(0x7f9dac0c6710 msgr2=0x7f9dac0c2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 shutdown_connections 2024-03-06T07:25:37.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.345+0000 7f9db217a640 1 -- 172.21.15.94:0/2414858877 wait complete. 2024-03-06T07:25:37.373 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:37.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.478+0000 7f35de6c2640 1 Processor -- start 2024-03-06T07:25:37.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.478+0000 7f35de6c2640 1 -- start start 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.478+0000 7f35de6c2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35d814ece0 0x7f35d80c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.478+0000 7f35de6c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d80c2810 0x7f35d80aa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.478+0000 7f35de6c2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d81524d0 con 0x7f35d80aae80 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.478+0000 7f35de6c2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d8151710 con 0x7f35d814ece0 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.478+0000 7f35de6c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81519f0 con 0x7f35d80c2810 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35dcc5f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f35d80aae80 0x7f35d8154af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37050/0 (socket says 172.21.15.94:37050) 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35dcc5f640 1 -- 172.21.15.94:0/2817505805 learned_addr learned my addr 172.21.15.94:0/2817505805 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d7fff640 1 --2- 172.21.15.94:0/2817505805 >> v2:172.21.15.145:3300/0 conn(0x7f35d814ece0 0x7f35d80c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d77fe640 1 --2- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d80c2810 0x7f35d80aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d6ffd640 1 -- 172.21.15.94:0/2817505805 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004360085 0 0) 0x7f35d81524d0 con 0x7f35d80aae80 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d6ffd640 1 -- 172.21.15.94:0/2817505805 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c0003520 con 0x7f35d80aae80 2024-03-06T07:25:37.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d7fff640 1 -- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d80c2810 msgr2=0x7f35d80aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d7fff640 1 --2- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d80c2810 0x7f35d80aa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d6ffd640 1 -- 172.21.15.94:0/2817505805 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2367752505 0 0) 0x7f35c0003520 con 0x7f35d80aae80 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d7fff640 1 -- 172.21.15.94:0/2817505805 >> v1:172.21.15.94:6789/0 conn(0x7f35d80aae80 legacy=0x7f35d8154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d7fff640 1 -- 172.21.15.94:0/2817505805 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8151cd0 con 0x7f35d814ece0 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.479+0000 7f35d77fe640 1 --2- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d80c2810 0x7f35d80aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.480+0000 7f35d7fff640 1 --2- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d814ece0 0x7f35d80c22d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f35c80097a0 tx=0x7f35c802b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d99966e06622296 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.480+0000 7f35d6ffd640 1 -- 172.21.15.94:0/2817505805 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8037070 con 0x7f35d814ece0 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.480+0000 7f35d6ffd640 1 -- 172.21.15.94:0/2817505805 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c8030510 con 0x7f35d814ece0 2024-03-06T07:25:37.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.480+0000 7f35d6ffd640 1 -- 172.21.15.94:0/2817505805 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c802be60 con 0x7f35d814ece0 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 -- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d814ece0 msgr2=0x7f35d80c22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 --2- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d814ece0 0x7f35d80c22d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f35c80097a0 tx=0x7f35c802b660 comp rx=0 tx=0).stop 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 -- 172.21.15.94:0/2817505805 shutdown_connections 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 --2- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d80c2810 0x7f35d80aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 --2- 172.21.15.94:0/2817505805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d814ece0 0x7f35d80c22d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 -- 172.21.15.94:0/2817505805 >> 172.21.15.94:0/2817505805 conn(0x7f35d806afc0 msgr2=0x7f35d806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 -- 172.21.15.94:0/2817505805 shutdown_connections 2024-03-06T07:25:37.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.481+0000 7f35de6c2640 1 -- 172.21.15.94:0/2817505805 wait complete. 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.482+0000 7f35de6c2640 1 Processor -- start 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.482+0000 7f35de6c2640 1 -- start start 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.482+0000 7f35de6c2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d80aae80 0x7f35d80bcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.482+0000 7f35de6c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d80c2810 0x7f35d814dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35de6c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d814ece0 0x7f35d814e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35de6c2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d806dd30 con 0x7f35d80aae80 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35de6c2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d8155200 con 0x7f35d80c2810 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35de6c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81ed0c0 con 0x7f35d814ece0 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d80aae80 0x7f35d80bcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d80aae80 0x7f35d80bcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:60324/0 (socket says 172.21.15.94:60324) 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 -- 172.21.15.94:0/1125921916 learned_addr learned my addr 172.21.15.94:0/1125921916 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35dcc5f640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d814ece0 0x7f35d814e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d77fe640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d80c2810 0x7f35d814dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 -- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d814ece0 msgr2=0x7f35d814e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d814ece0 0x7f35d814e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 -- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d80c2810 msgr2=0x7f35d814dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d80c2810 0x7f35d814dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 -- 172.21.15.94:0/1125921916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d80be220 con 0x7f35d80aae80 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35dcc5f640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d814ece0 0x7f35d814e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d7fff640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d80aae80 0x7f35d80bcc80 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f35c802b6a0 tx=0x7f35c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d4ff9640 1 -- 172.21.15.94:0/1125921916 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8037050 con 0x7f35d80aae80 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d4ff9640 1 -- 172.21.15.94:0/1125921916 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c8043070 con 0x7f35d80aae80 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d81ec4b0 con 0x7f35d80aae80 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35d4ff9640 1 -- 172.21.15.94:0/1125921916 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d00019f0 con 0x7f35d80aae80 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.483+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d81eca70 con 0x7f35d80aae80 2024-03-06T07:25:37.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.484+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d80a6760 con 0x7f35d80aae80 2024-03-06T07:25:37.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.485+0000 7f35d4ff9640 1 -- 172.21.15.94:0/1125921916 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35c8003a30 con 0x7f35d80aae80 2024-03-06T07:25:37.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.485+0000 7f35d4ff9640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35c406b4d0 0x7f35c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.485+0000 7f35d4ff9640 1 -- 172.21.15.94:0/1125921916 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f35c80abe80 con 0x7f35d80aae80 2024-03-06T07:25:37.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.486+0000 7f35d77fe640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35c406b4d0 0x7f35c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.486+0000 7f35d77fe640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35c406b4d0 0x7f35c406d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009560 tx=0x7f35cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.490+0000 7f35d4ff9640 1 -- 172.21.15.94:0/1125921916 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35d80a6760 con 0x7f35d80aae80 2024-03-06T07:25:37.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:37.697+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f35d806e550 con 0x7f35d80aae80 2024-03-06T07:25:38.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.361+0000 7f35d4ff9640 1 -- 172.21.15.94:0/1125921916 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 29 and data pool 30 v169) v1 ==== 174+0+0 (secure 0 0 0) 0x7f35d80be220 con 0x7f35d80aae80 2024-03-06T07:25:38.363 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 29 and data pool 30 2024-03-06T07:25:38.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.363+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35c406b4d0 msgr2=0x7f35c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.363+0000 7f35de6c2640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35c406b4d0 0x7f35c406d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009560 tx=0x7f35cc008040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d80aae80 msgr2=0x7f35d80bcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d80aae80 0x7f35d80bcc80 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f35c802b6a0 tx=0x7f35c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 shutdown_connections 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f35c406b4d0 0x7f35c406d990 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35d814ece0 0x7f35d814e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35d80c2810 0x7f35d814dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 --2- 172.21.15.94:0/1125921916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f35d80aae80 0x7f35d80bcc80 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.364+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 >> 172.21.15.94:0/1125921916 conn(0x7f35d806afc0 msgr2=0x7f35d80c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.365+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 shutdown_connections 2024-03-06T07:25:38.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.368+0000 7f35de6c2640 1 -- 172.21.15.94:0/1125921916 wait complete. 2024-03-06T07:25:38.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:38.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.499+0000 7f742b1b5640 1 Processor -- start 2024-03-06T07:25:38.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.499+0000 7f742b1b5640 1 -- start start 2024-03-06T07:25:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f742b1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 0x7f74241559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f742b1b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7424152d40 con 0x7f7424152960 2024-03-06T07:25:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f742b1b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74240c1b50 con 0x7f74241592d0 2024-03-06T07:25:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f742b1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7424066e30 con 0x7f74240ac260 2024-03-06T07:25:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f7428f51640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7424152960 0x7f742406dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58436/0 (socket says 172.21.15.94:58436) 2024-03-06T07:25:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f7428f51640 1 -- 172.21.15.94:0/2240929804 learned_addr learned my addr 172.21.15.94:0/2240929804 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:38.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f7429752640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 0x7f74241559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017922068 0 0) 0x7f7424152d40 con 0x7f7424152960 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7414003520 con 0x7f7424152960 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4038186217 0 0) 0x7f7414003520 con 0x7f7424152960 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 msgr2=0x7f74241559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 0x7f74241559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 >> v1:172.21.15.145:6789/0 conn(0x7f74241592d0 legacy=0x7f74241552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74240c1dc0 con 0x7f7424152960 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f7418001670 con 0x7f7424152960 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 >> v1:172.21.15.94:6789/0 conn(0x7f7424152960 legacy=0x7f742406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74140039e0 0x7f7414003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 0x7f7414004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7414004d40 0x7f74140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424066e30 con 0x7f7414004d40 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741400ba70 con 0x7f74140039e0 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741400bc70 con 0x7f74240ac260 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f7429752640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7414004d40 0x7f74140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.500+0000 7f7428f51640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74140039e0 0x7f7414003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.501+0000 7f7423fff640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 0x7f7414004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.501+0000 7f7429752640 1 -- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 msgr2=0x7f7414004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.501+0000 7f7429752640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74240ac260 0x7f7414004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.501+0000 7f7429752640 1 -- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74140039e0 msgr2=0x7f7414003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.501+0000 7f7429752640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74140039e0 0x7f7414003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.501+0000 7f7429752640 1 -- 172.21.15.94:0/2240929804 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414003520 con 0x7f7414004d40 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.501+0000 7f7429752640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7414004d40 0x7f74140051c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f7424155f20 tx=0x7f7414005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c12b8061195985d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c039070 con 0x7f7414004d40 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c02cb90 con 0x7f7414004d40 2024-03-06T07:25:38.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f74237fe640 1 -- 172.21.15.94:0/2240929804 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c009c10 con 0x7f7414004d40 2024-03-06T07:25:38.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f742b1b5640 1 -- 172.21.15.94:0/2240929804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7414004d40 msgr2=0x7f74140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f742b1b5640 1 --2- 172.21.15.94:0/2240929804 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7414004d40 0x7f74140051c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f7424155f20 tx=0x7f7414005830 comp rx=0 tx=0).stop 2024-03-06T07:25:38.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f7429752640 1 -- 172.21.15.94:0/2240929804 reap_dead start 2024-03-06T07:25:38.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f742b1b5640 1 -- 172.21.15.94:0/2240929804 shutdown_connections 2024-03-06T07:25:38.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f742b1b5640 1 -- 172.21.15.94:0/2240929804 >> 172.21.15.94:0/2240929804 conn(0x7f74240c70b0 msgr2=0x7f74240c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f742b1b5640 1 -- 172.21.15.94:0/2240929804 shutdown_connections 2024-03-06T07:25:38.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.502+0000 7f742b1b5640 1 -- 172.21.15.94:0/2240929804 wait complete. 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.503+0000 7f742b1b5640 1 Processor -- start 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.503+0000 7f742b1b5640 1 -- start start 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f742b1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7414004d40 0x7f742414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f742b1b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7424156f90 0x7f7424157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f742b1b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7424157950 0x7f74241edf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f742b1b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742406e3a0 con 0x7f7424156f90 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f742b1b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424159760 con 0x7f7424157950 2024-03-06T07:25:38.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f742b1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742406e520 con 0x7f7414004d40 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7424156f90 0x7f7424157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7424156f90 0x7f7424157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41694/0 (socket says 172.21.15.94:41694) 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 -- 172.21.15.94:0/598364980 learned_addr learned my addr 172.21.15.94:0/598364980 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7429752640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7424157950 0x7f74241edf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7428f51640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7414004d40 0x7f742414d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 -- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7414004d40 msgr2=0x7f742414d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7414004d40 0x7f742414d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 -- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7424157950 msgr2=0x7f74241edf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7424157950 0x7f74241edf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7423fff640 1 -- 172.21.15.94:0/598364980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424070c50 con 0x7f7424156f90 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.504+0000 7f7428f51640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7414004d40 0x7f742414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.505+0000 7f7423fff640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7424156f90 0x7f7424157410 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f741000f880 tx=0x7f741000fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.505+0000 7f74217fa640 1 -- 172.21.15.94:0/598364980 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410001d20 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.505+0000 7f74217fa640 1 -- 172.21.15.94:0/598364980 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7410018070 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.505+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7424070f30 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.505+0000 7f74217fa640 1 -- 172.21.15.94:0/598364980 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741000d730 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.505+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74240714f0 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.506+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742404f330 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.506+0000 7f74217fa640 1 -- 172.21.15.94:0/598364980 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f741000d8d0 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.507+0000 7f74217fa640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f740806b500 0x7f740806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.507+0000 7f74217fa640 1 -- 172.21.15.94:0/598364980 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f741008a120 con 0x7f7424156f90 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.507+0000 7f7428f51640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f740806b500 0x7f740806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.508+0000 7f7428f51640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f740806b500 0x7f740806d9c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7424072f40 tx=0x7f7418009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.511+0000 7f74217fa640 1 -- 172.21.15.94:0/598364980 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7410058ea0 con 0x7f7424156f90 2024-03-06T07:25:38.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.683+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f742406ebc0 con 0x7f7424156f90 2024-03-06T07:25:38.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.684+0000 7f74217fa640 1 -- 172.21.15.94:0/598364980 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16897 (secure 0 0 0) 0x7f74100587a0 con 0x7f7424156f90 2024-03-06T07:25:38.686 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:38.686 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":181,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:37.698697+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.909545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.667783+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f740806b500 msgr2=0x7f740806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f740806b500 0x7f740806d9c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7424072f40 tx=0x7f7418009040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7424156f90 msgr2=0x7f7424157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7424156f90 0x7f7424157410 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f741000f880 tx=0x7f741000fd40 comp rx=0 tx=0).stop 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 shutdown_connections 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f740806b500 0x7f740806d9c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7424157950 0x7f74241edf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7424156f90 0x7f7424157410 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 --2- 172.21.15.94:0/598364980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7414004d40 0x7f742414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 >> 172.21.15.94:0/598364980 conn(0x7f74240c70b0 msgr2=0x7f742406c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 shutdown_connections 2024-03-06T07:25:38.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.686+0000 7f742b1b5640 1 -- 172.21.15.94:0/598364980 wait complete. 2024-03-06T07:25:38.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:38.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.797+0000 7f802ecd4640 1 Processor -- start 2024-03-06T07:25:38.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.797+0000 7f802ecd4640 1 -- start start 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ecd4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f802814f790 0x7f80281529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ecd4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8028155a40 0x7f8028155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ecd4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80281563d0 con 0x7f80280c1aa0 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ecd4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80280c2680 con 0x7f802814f790 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ecd4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80280c28d0 con 0x7f8028155a40 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ca70640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f802814f790 0x7f80281529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ca70640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f802814f790 0x7f80281529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41708/0 (socket says 172.21.15.94:41708) 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ca70640 1 -- 172.21.15.94:0/3018155195 learned_addr learned my addr 172.21.15.94:0/3018155195 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802d271640 1 --2- 172.21.15.94:0/3018155195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8028155a40 0x7f8028155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ca70640 1 -- 172.21.15.94:0/3018155195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8028155a40 msgr2=0x7f8028155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ca70640 1 --2- 172.21.15.94:0/3018155195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8028155a40 0x7f8028155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ca70640 1 -- 172.21.15.94:0/3018155195 >> v1:172.21.15.145:6789/0 conn(0x7f80280c1aa0 legacy=0x7f80280c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802ca70640 1 -- 172.21.15.94:0/3018155195 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80280c2bb0 con 0x7f802814f790 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.798+0000 7f802d271640 1 --2- 172.21.15.94:0/3018155195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8028155a40 0x7f8028155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.799+0000 7f802ca70640 1 --2- 172.21.15.94:0/3018155195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f802814f790 0x7f80281529d0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f8018009870 tx=0x7f801802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2027d5f92e02e942 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.799+0000 7f80277fe640 1 -- 172.21.15.94:0/3018155195 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018035020 con 0x7f802814f790 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.799+0000 7f80277fe640 1 -- 172.21.15.94:0/3018155195 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8018031640 con 0x7f802814f790 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.799+0000 7f80277fe640 1 -- 172.21.15.94:0/3018155195 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801802d040 con 0x7f802814f790 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 -- 172.21.15.94:0/3018155195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f802814f790 msgr2=0x7f80281529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 --2- 172.21.15.94:0/3018155195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f802814f790 0x7f80281529d0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f8018009870 tx=0x7f801802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 -- 172.21.15.94:0/3018155195 shutdown_connections 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 --2- 172.21.15.94:0/3018155195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8028155a40 0x7f8028155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 --2- 172.21.15.94:0/3018155195 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f802814f790 0x7f80281529d0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 -- 172.21.15.94:0/3018155195 >> 172.21.15.94:0/3018155195 conn(0x7f80280c6710 msgr2=0x7f80280c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 -- 172.21.15.94:0/3018155195 shutdown_connections 2024-03-06T07:25:38.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.800+0000 7f802ecd4640 1 -- 172.21.15.94:0/3018155195 wait complete. 2024-03-06T07:25:38.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.801+0000 7f802ecd4640 1 Processor -- start 2024-03-06T07:25:38.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.801+0000 7f802ecd4640 1 -- start start 2024-03-06T07:25:38.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80280c1aa0 0x7f802814d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f802814f790 0x7f80281e0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8028155a40 0x7f802814b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80280c0d60 con 0x7f8028155a40 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8028155520 con 0x7f802814f790 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80280c5e30 con 0x7f80280c1aa0 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8028155a40 0x7f802814b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8028155a40 0x7f802814b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41724/0 (socket says 172.21.15.94:41724) 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 -- 172.21.15.94:0/408597294 learned_addr learned my addr 172.21.15.94:0/408597294 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f8027fff640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f802814f790 0x7f80281e0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ca70640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80280c1aa0 0x7f802814d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 -- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80280c1aa0 msgr2=0x7f802814d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80280c1aa0 0x7f802814d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 -- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f802814f790 msgr2=0x7f80281e0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f802814f790 0x7f80281e0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 -- 172.21.15.94:0/408597294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80281515a0 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802d271640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8028155a40 0x7f802814b810 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f802000cd80 tx=0x7f802000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f80257fa640 1 -- 172.21.15.94:0/408597294 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802000f680 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f80257fa640 1 -- 172.21.15.94:0/408597294 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802000f840 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802814ca60 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f80257fa640 1 -- 172.21.15.94:0/408597294 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802000fc30 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.802+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802814e6f0 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.803+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802814e240 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.804+0000 7f80257fa640 1 -- 172.21.15.94:0/408597294 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80200213f0 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.804+0000 7f80257fa640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f801406b4d0 0x7f801406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.804+0000 7f80257fa640 1 -- 172.21.15.94:0/408597294 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f802008af40 con 0x7f8028155a40 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.804+0000 7f802ca70640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f801406b4d0 0x7f801406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.805+0000 7f802ca70640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f801406b4d0 0x7f801406d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8018036000 tx=0x7f801803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.809+0000 7f80257fa640 1 -- 172.21.15.94:0/408597294 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8020014030 con 0x7f8028155a40 2024-03-06T07:25:38.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.980+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f802814e8f0 con 0x7f8028155a40 2024-03-06T07:25:38.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.981+0000 7f80257fa640 1 -- 172.21.15.94:0/408597294 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16897 (secure 0 0 0) 0x7f802814e8f0 con 0x7f8028155a40 2024-03-06T07:25:38.984 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:38.984 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":181,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:37.698697+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.909545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.667783+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:38.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.983+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f801406b4d0 msgr2=0x7f801406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.983+0000 7f802ecd4640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f801406b4d0 0x7f801406d990 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8018036000 tx=0x7f801803c040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.983+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8028155a40 msgr2=0x7f802814b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.983+0000 7f802ecd4640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8028155a40 0x7f802814b810 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f802000cd80 tx=0x7f802000a650 comp rx=0 tx=0).stop 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 shutdown_connections 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f801406b4d0 0x7f801406d990 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8028155a40 0x7f802814b810 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f802814f790 0x7f80281e0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 --2- 172.21.15.94:0/408597294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80280c1aa0 0x7f802814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 >> 172.21.15.94:0/408597294 conn(0x7f80280c6710 msgr2=0x7f80280c03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 shutdown_connections 2024-03-06T07:25:38.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:38.984+0000 7f802ecd4640 1 -- 172.21.15.94:0/408597294 wait complete. 2024-03-06T07:25:38.996 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:39.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.098+0000 7f341490f640 1 Processor -- start 2024-03-06T07:25:39.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.098+0000 7f341490f640 1 -- start start 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.098+0000 7f341490f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34100ab310 0x7f34100ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.098+0000 7f341490f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3410154330 con 0x7f34100a97c0 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.098+0000 7f341490f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3410153dc0 con 0x7f34101538c0 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.098+0000 7f341490f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34100c46a0 con 0x7f34100ab310 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34100ab310 0x7f34100ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34100ab310 0x7f34100ab6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41734/0 (socket says 172.21.15.94:41734) 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340ed9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f34100a97c0 0x7f34100c4b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58446/0 (socket says 172.21.15.94:58446) 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340e59c640 1 -- 172.21.15.94:0/2400983586 learned_addr learned my addr 172.21.15.94:0/2400983586 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:39.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340e59c640 1 -- 172.21.15.94:0/2400983586 >> v1:172.21.15.145:6789/0 conn(0x7f34101538c0 legacy=0x7f34101494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340e59c640 1 -- 172.21.15.94:0/2400983586 >> v1:172.21.15.94:6789/0 conn(0x7f34100a97c0 legacy=0x7f34100c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340dd9b640 1 -- 172.21.15.94:0/2400983586 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053699934 0 0) 0x7f3410154330 con 0x7f34100a97c0 2024-03-06T07:25:39.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.099+0000 7f340e59c640 1 -- 172.21.15.94:0/2400983586 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100c5d30 con 0x7f34100ab310 2024-03-06T07:25:39.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f340e59c640 1 --2- 172.21.15.94:0/2400983586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34100ab310 0x7f34100ab6f0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f3400009870 tx=0x7f340002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cf271b42e56f0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f340dd9b640 1 -- 172.21.15.94:0/2400983586 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400034070 con 0x7f34100ab310 2024-03-06T07:25:39.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f340dd9b640 1 -- 172.21.15.94:0/2400983586 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3400030860 con 0x7f34100ab310 2024-03-06T07:25:39.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f340dd9b640 1 -- 172.21.15.94:0/2400983586 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400030c30 con 0x7f34100ab310 2024-03-06T07:25:39.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f341490f640 1 -- 172.21.15.94:0/2400983586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34100ab310 msgr2=0x7f34100ab6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f341490f640 1 --2- 172.21.15.94:0/2400983586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34100ab310 0x7f34100ab6f0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f3400009870 tx=0x7f340002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f341490f640 1 -- 172.21.15.94:0/2400983586 shutdown_connections 2024-03-06T07:25:39.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f341490f640 1 --2- 172.21.15.94:0/2400983586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34100ab310 0x7f34100ab6f0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f341490f640 1 -- 172.21.15.94:0/2400983586 >> 172.21.15.94:0/2400983586 conn(0x7f34100c6710 msgr2=0x7f34100c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f341490f640 1 -- 172.21.15.94:0/2400983586 shutdown_connections 2024-03-06T07:25:39.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.100+0000 7f341490f640 1 -- 172.21.15.94:0/2400983586 wait complete. 2024-03-06T07:25:39.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 Processor -- start 2024-03-06T07:25:39.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 -- start start 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34100a97c0 0x7f3410070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34100ab310 0x7f34101566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34101538c0 0x7f34100bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100abc30 con 0x7f34101538c0 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100c62f0 con 0x7f34100a97c0 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f341490f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3410149bb0 con 0x7f34100ab310 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f340ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34101538c0 0x7f34100bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f340ed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34101538c0 0x7f34100bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41744/0 (socket says 172.21.15.94:41744) 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f340ed9d640 1 -- 172.21.15.94:0/1755998039 learned_addr learned my addr 172.21.15.94:0/1755998039 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f340e59c640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34100a97c0 0x7f3410070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.102+0000 7f3407fff640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34100ab310 0x7f34101566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f340ed9d640 1 -- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34100ab310 msgr2=0x7f34101566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f340ed9d640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34100ab310 0x7f34101566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f340ed9d640 1 -- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34100a97c0 msgr2=0x7f3410070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f340ed9d640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34100a97c0 0x7f3410070dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f340ed9d640 1 -- 172.21.15.94:0/1755998039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34101ee880 con 0x7f34101538c0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f340ed9d640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34101538c0 0x7f34100bfdb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f340800c7d0 tx=0x7f340800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f3406ffd640 1 -- 172.21.15.94:0/1755998039 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408018070 con 0x7f34101538c0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f3406ffd640 1 -- 172.21.15.94:0/1755998039 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340800a640 con 0x7f34101538c0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34101eeb60 con 0x7f34101538c0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f3406ffd640 1 -- 172.21.15.94:0/1755998039 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340800aa70 con 0x7f34101538c0 2024-03-06T07:25:39.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.103+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34101ef120 con 0x7f34101538c0 2024-03-06T07:25:39.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.104+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34100a7c50 con 0x7f34101538c0 2024-03-06T07:25:39.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.105+0000 7f3406ffd640 1 -- 172.21.15.94:0/1755998039 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f340801b070 con 0x7f34101538c0 2024-03-06T07:25:39.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.105+0000 7f3406ffd640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f33f006b4d0 0x7f33f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.105+0000 7f3406ffd640 1 -- 172.21.15.94:0/1755998039 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3408089930 con 0x7f34101538c0 2024-03-06T07:25:39.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.105+0000 7f340e59c640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f33f006b4d0 0x7f33f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.106+0000 7f340e59c640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f33f006b4d0 0x7f33f006d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f3400036000 tx=0x7f3400008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.109+0000 7f3406ffd640 1 -- 172.21.15.94:0/1755998039 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34100a7c50 con 0x7f34101538c0 2024-03-06T07:25:39.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.319+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f34101ef320 con 0x7f34101538c0 2024-03-06T07:25:39.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.372+0000 7f3406ffd640 1 -- 172.21.15.94:0/1755998039 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v170) v1 ==== 118+0+0 (secure 0 0 0) 0x7f34101ef320 con 0x7f34101538c0 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f33f006b4d0 msgr2=0x7f33f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f33f006b4d0 0x7f33f006d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f3400036000 tx=0x7f3400008030 comp rx=0 tx=0).stop 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34101538c0 msgr2=0x7f34100bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34101538c0 0x7f34100bfdb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f340800c7d0 tx=0x7f340800cc90 comp rx=0 tx=0).stop 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 shutdown_connections 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f33f006b4d0 0x7f33f006d990 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34101538c0 0x7f34100bfdb0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34100ab310 0x7f34101566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 --2- 172.21.15.94:0/1755998039 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34100a97c0 0x7f3410070dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 >> 172.21.15.94:0/1755998039 conn(0x7f34100c6710 msgr2=0x7f34100a62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 shutdown_connections 2024-03-06T07:25:39.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.375+0000 7f341490f640 1 -- 172.21.15.94:0/1755998039 wait complete. 2024-03-06T07:25:39.388 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:39.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:39.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.489+0000 7f6fd6702640 1 Processor -- start 2024-03-06T07:25:39.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.489+0000 7f6fd6702640 1 -- start start 2024-03-06T07:25:39.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.489+0000 7f6fd6702640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6fd00ab310 0x7f6fd00ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fd6702640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6fd01538c0 0x7f6fd01494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fd6702640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd0154330 con 0x7f6fd00a97c0 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fd6702640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd00c46a0 con 0x7f6fd01538c0 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fd6702640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd0153dc0 con 0x7f6fd00ab310 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fcf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6fd01538c0 0x7f6fd01494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fd4c9f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6fd00a97c0 0x7f6fd00c4b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58458/0 (socket says 172.21.15.94:58458) 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fd4c9f640 1 -- 172.21.15.94:0/588112772 learned_addr learned my addr 172.21.15.94:0/588112772 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:39.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fcffff640 1 --2- 172.21.15.94:0/588112772 >> v2:172.21.15.145:3300/0 conn(0x7f6fd00ab310 0x7f6fd00ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fcf7fe640 1 -- 172.21.15.94:0/588112772 >> v2:172.21.15.145:3300/0 conn(0x7f6fd00ab310 msgr2=0x7f6fd00ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fcf7fe640 1 --2- 172.21.15.94:0/588112772 >> v2:172.21.15.145:3300/0 conn(0x7f6fd00ab310 0x7f6fd00ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fcf7fe640 1 -- 172.21.15.94:0/588112772 >> v1:172.21.15.94:6789/0 conn(0x7f6fd00a97c0 legacy=0x7f6fd00c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fceffd640 1 -- 172.21.15.94:0/588112772 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864759978 0 0) 0x7f6fd0154330 con 0x7f6fd00a97c0 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.490+0000 7f6fcf7fe640 1 -- 172.21.15.94:0/588112772 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00c5d30 con 0x7f6fd01538c0 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fcf7fe640 1 --2- 172.21.15.94:0/588112772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01494a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00a9550 tx=0x7f6fc402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe72a0ddb577f14f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fceffd640 1 -- 172.21.15.94:0/588112772 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc40315a0 con 0x7f6fd01538c0 2024-03-06T07:25:39.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fceffd640 1 -- 172.21.15.94:0/588112772 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc4037070 con 0x7f6fd01538c0 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fceffd640 1 -- 172.21.15.94:0/588112772 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc4031960 con 0x7f6fd01538c0 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 -- 172.21.15.94:0/588112772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 msgr2=0x7f6fd01494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 --2- 172.21.15.94:0/588112772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01494a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00a9550 tx=0x7f6fc402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 -- 172.21.15.94:0/588112772 shutdown_connections 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 --2- 172.21.15.94:0/588112772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01494a0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 --2- 172.21.15.94:0/588112772 >> v2:172.21.15.145:3300/0 conn(0x7f6fd00ab310 0x7f6fd00ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 -- 172.21.15.94:0/588112772 >> 172.21.15.94:0/588112772 conn(0x7f6fd00c6710 msgr2=0x7f6fd00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 -- 172.21.15.94:0/588112772 shutdown_connections 2024-03-06T07:25:39.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.491+0000 7f6fd6702640 1 -- 172.21.15.94:0/588112772 wait complete. 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.492+0000 7f6fd6702640 1 Processor -- start 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd6702640 1 -- start start 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd6702640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fd00a97c0 0x7f6fd0156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd6702640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fd00ab310 0x7f6fd01e8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd6702640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01e92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd6702640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00abc30 con 0x7f6fd01538c0 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd6702640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00c62f0 con 0x7f6fd00a97c0 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd6702640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00c5240 con 0x7f6fd00ab310 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd4c9f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01e92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fcffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fd00a97c0 0x7f6fd0156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd4c9f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01e92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41772/0 (socket says 172.21.15.94:41772) 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.493+0000 7f6fd4c9f640 1 -- 172.21.15.94:0/2138033878 learned_addr learned my addr 172.21.15.94:0/2138033878 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fcf7fe640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fd00ab310 0x7f6fd01e8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd4c9f640 1 -- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fd00ab310 msgr2=0x7f6fd01e8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd4c9f640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fd00ab310 0x7f6fd01e8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd4c9f640 1 -- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fd00a97c0 msgr2=0x7f6fd0156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd4c9f640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fd00a97c0 0x7f6fd0156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd4c9f640 1 -- 172.21.15.94:0/2138033878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd01ec9b0 con 0x7f6fd01538c0 2024-03-06T07:25:39.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fcffff640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fd00a97c0 0x7f6fd0156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd4c9f640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01e92b0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800c7a0 tx=0x7f6fc800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fccff9640 1 -- 172.21.15.94:0/2138033878 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8018070 con 0x7f6fd01538c0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fccff9640 1 -- 172.21.15.94:0/2138033878 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc800a640 con 0x7f6fd01538c0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd01507e0 con 0x7f6fd01538c0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fccff9640 1 -- 172.21.15.94:0/2138033878 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc800aab0 con 0x7f6fd01538c0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.494+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd0156eb0 con 0x7f6fd01538c0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.495+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd004f330 con 0x7f6fd01538c0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.495+0000 7f6fccff9640 1 -- 172.21.15.94:0/2138033878 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fc801b070 con 0x7f6fd01538c0 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.496+0000 7f6fccff9640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fbc06b500 0x7f6fbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.496+0000 7f6fccff9640 1 -- 172.21.15.94:0/2138033878 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6fc80898e0 con 0x7f6fd01538c0 2024-03-06T07:25:39.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.496+0000 7f6fcffff640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fbc06b500 0x7f6fbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.496+0000 7f6fcffff640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fbc06b500 0x7f6fbc06d9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6fd01e04b0 tx=0x7f6fc0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.500+0000 7f6fccff9640 1 -- 172.21.15.94:0/2138033878 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fc8057f80 con 0x7f6fd01538c0 2024-03-06T07:25:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:39.710+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6fd00aa580 con 0x7f6fd01538c0 2024-03-06T07:25:40.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.382+0000 7f6fccff9640 1 -- 172.21.15.94:0/2138033878 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v171) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6fc805c020 con 0x7f6fd01538c0 2024-03-06T07:25:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fbc06b500 msgr2=0x7f6fbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fbc06b500 0x7f6fbc06d9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6fd01e04b0 tx=0x7f6fc0009190 comp rx=0 tx=0).stop 2024-03-06T07:25:40.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 msgr2=0x7f6fd01e92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01e92b0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800c7a0 tx=0x7f6fc800cc60 comp rx=0 tx=0).stop 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 shutdown_connections 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6fbc06b500 0x7f6fbc06d9c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fd01538c0 0x7f6fd01e92b0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fd00ab310 0x7f6fd01e8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 --2- 172.21.15.94:0/2138033878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fd00a97c0 0x7f6fd0156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 >> 172.21.15.94:0/2138033878 conn(0x7f6fd00c6710 msgr2=0x7f6fd00c2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 shutdown_connections 2024-03-06T07:25:40.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.384+0000 7f6fd6702640 1 -- 172.21.15.94:0/2138033878 wait complete. 2024-03-06T07:25:40.398 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:40.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:40.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.500+0000 7f844553a640 1 Processor -- start 2024-03-06T07:25:40.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.500+0000 7f844553a640 1 -- start start 2024-03-06T07:25:40.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f844553a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84401506b0 0x7f8440153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f844553a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8440156ea0 0x7f84400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f844553a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84400acc00 0x7f84400c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f844553a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8440156450 con 0x7f84401506b0 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f844553a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8440156a10 con 0x7f8440156ea0 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f844553a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8440070800 con 0x7f84400acc00 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f843effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84401506b0 0x7f8440153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f843effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84401506b0 0x7f8440153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41782/0 (socket says 172.21.15.94:41782) 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f843effd640 1 -- 172.21.15.94:0/1089489413 learned_addr learned my addr 172.21.15.94:0/1089489413 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f843e7fc640 1 --2- 172.21.15.94:0/1089489413 >> v2:172.21.15.145:3300/0 conn(0x7f8440156ea0 0x7f84400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.501+0000 7f843f7fe640 1 --2- 172.21.15.94:0/1089489413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84400acc00 0x7f84400c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.502+0000 7f843effd640 1 -- 172.21.15.94:0/1089489413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84400acc00 msgr2=0x7f84400c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.502+0000 7f843effd640 1 --2- 172.21.15.94:0/1089489413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84400acc00 0x7f84400c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.502+0000 7f843effd640 1 -- 172.21.15.94:0/1089489413 >> v2:172.21.15.145:3300/0 conn(0x7f8440156ea0 msgr2=0x7f84400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.502+0000 7f843effd640 1 --2- 172.21.15.94:0/1089489413 >> v2:172.21.15.145:3300/0 conn(0x7f8440156ea0 0x7f84400702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.502+0000 7f843effd640 1 -- 172.21.15.94:0/1089489413 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440070aa0 con 0x7f84401506b0 2024-03-06T07:25:40.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.502+0000 7f843effd640 1 --2- 172.21.15.94:0/1089489413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f8440153990 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f844006bf40 tx=0x7f843002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbeb79207148b2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.502+0000 7f843dffb640 1 -- 172.21.15.94:0/1089489413 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843002ca70 con 0x7f84401506b0 2024-03-06T07:25:40.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f843dffb640 1 -- 172.21.15.94:0/1089489413 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843002cc10 con 0x7f84401506b0 2024-03-06T07:25:40.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f843dffb640 1 -- 172.21.15.94:0/1089489413 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84300343f0 con 0x7f84401506b0 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 -- 172.21.15.94:0/1089489413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 msgr2=0x7f8440153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 --2- 172.21.15.94:0/1089489413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f8440153990 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f844006bf40 tx=0x7f843002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 -- 172.21.15.94:0/1089489413 shutdown_connections 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 --2- 172.21.15.94:0/1089489413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84400acc00 0x7f84400c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 --2- 172.21.15.94:0/1089489413 >> v2:172.21.15.145:3300/0 conn(0x7f8440156ea0 0x7f84400702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 --2- 172.21.15.94:0/1089489413 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f8440153990 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 -- 172.21.15.94:0/1089489413 >> 172.21.15.94:0/1089489413 conn(0x7f844006b1a0 msgr2=0x7f844006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 -- 172.21.15.94:0/1089489413 shutdown_connections 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.503+0000 7f844553a640 1 -- 172.21.15.94:0/1089489413 wait complete. 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.504+0000 7f844553a640 1 Processor -- start 2024-03-06T07:25:40.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.504+0000 7f844553a640 1 -- start start 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f844553a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84400acc00 0x7f8440154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f844553a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f84401ee180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f844553a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8440156ea0 0x7f84400c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f844553a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84400c59e0 con 0x7f84401506b0 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f844553a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84401566a0 con 0x7f84400acc00 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f844553a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84400c5b60 con 0x7f8440156ea0 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f84401ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f84401ee180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41794/0 (socket says 172.21.15.94:41794) 2024-03-06T07:25:40.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 -- 172.21.15.94:0/391566384 learned_addr learned my addr 172.21.15.94:0/391566384 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843effd640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84400acc00 0x7f8440154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843f7fe640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8440156ea0 0x7f84400c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 -- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8440156ea0 msgr2=0x7f84400c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8440156ea0 0x7f84400c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 -- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84400acc00 msgr2=0x7f8440154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84400acc00 0x7f8440154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.505+0000 7f843e7fc640 1 -- 172.21.15.94:0/391566384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440154830 con 0x7f84401506b0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.506+0000 7f843e7fc640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f84401ee180 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f843400e870 tx=0x7f843400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.506+0000 7f84277fe640 1 -- 172.21.15.94:0/391566384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434017070 con 0x7f84401506b0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.506+0000 7f84277fe640 1 -- 172.21.15.94:0/391566384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843400c940 con 0x7f84401506b0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.506+0000 7f84277fe640 1 -- 172.21.15.94:0/391566384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843400cd30 con 0x7f84401506b0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.506+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844006dbd0 con 0x7f84401506b0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.506+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8440154bf0 con 0x7f84401506b0 2024-03-06T07:25:40.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.507+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844004f330 con 0x7f84401506b0 2024-03-06T07:25:40.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.507+0000 7f84277fe640 1 -- 172.21.15.94:0/391566384 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f843401a070 con 0x7f84401506b0 2024-03-06T07:25:40.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.508+0000 7f84277fe640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f842c06b4d0 0x7f842c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.508+0000 7f84277fe640 1 -- 172.21.15.94:0/391566384 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f843408a4d0 con 0x7f84401506b0 2024-03-06T07:25:40.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.508+0000 7f843effd640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f842c06b4d0 0x7f842c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.508+0000 7f843effd640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f842c06b4d0 0x7f842c06d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8430035000 tx=0x7f8430030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.512+0000 7f84277fe640 1 -- 172.21.15.94:0/391566384 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8434058c00 con 0x7f84401506b0 2024-03-06T07:25:40.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.719+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84400700a0 con 0x7f84401506b0 2024-03-06T07:25:40.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.719+0000 7f84277fe640 1 -- 172.21.15.94:0/391566384 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f843405cca0 con 0x7f84401506b0 2024-03-06T07:25:40.721 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:40.721 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":171,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:39.711371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37763,"qdb_cluster":[37763]},"id":14}]} 2024-03-06T07:25:40.722 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 171 2024-03-06T07:25:40.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f842c06b4d0 msgr2=0x7f842c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f842c06b4d0 0x7f842c06d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8430035000 tx=0x7f8430030000 comp rx=0 tx=0).stop 2024-03-06T07:25:40.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 msgr2=0x7f84401ee180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f84401ee180 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f843400e870 tx=0x7f843400ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:40.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 shutdown_connections 2024-03-06T07:25:40.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f842c06b4d0 0x7f842c06d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8440156ea0 0x7f84400c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84401506b0 0x7f84401ee180 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 --2- 172.21.15.94:0/391566384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84400acc00 0x7f8440154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 >> 172.21.15.94:0/391566384 conn(0x7f844006b1a0 msgr2=0x7f84400c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 shutdown_connections 2024-03-06T07:25:40.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.721+0000 7f844553a640 1 -- 172.21.15.94:0/391566384 wait complete. 2024-03-06T07:25:40.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:40.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.833+0000 7fa83bdbf640 1 Processor -- start 2024-03-06T07:25:40.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.833+0000 7fa83bdbf640 1 -- start start 2024-03-06T07:25:40.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83bdbf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8340c1320 0x7fa8341566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83bdbf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c6000 0x7fa834156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83bdbf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa834153a60 con 0x7fa834150580 2024-03-06T07:25:40.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83bdbf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa834153d40 con 0x7fa8340c1320 2024-03-06T07:25:40.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83bdbf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83406c2c0 con 0x7fa8340c6000 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83935a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8340c1320 0x7fa8341566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa839b5b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa834150580 0x7fa83406dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:48160/0 (socket says 172.21.15.94:48160) 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa839b5b640 1 -- 172.21.15.94:0/1542655967 learned_addr learned my addr 172.21.15.94:0/1542655967 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83935a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa8340c1320 0x7fa8341566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41798/0 (socket says 172.21.15.94:41798) 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83a35c640 1 --2- 172.21.15.94:0/1542655967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c6000 0x7fa834156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83935a640 1 -- 172.21.15.94:0/1542655967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c6000 msgr2=0x7fa834156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83935a640 1 --2- 172.21.15.94:0/1542655967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c6000 0x7fa834156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83935a640 1 -- 172.21.15.94:0/1542655967 >> v1:172.21.15.145:6789/0 conn(0x7fa834150580 legacy=0x7fa83406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa838b59640 1 -- 172.21.15.94:0/1542655967 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526796168 0 0) 0x7fa834153a60 con 0x7fa834150580 2024-03-06T07:25:40.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.834+0000 7fa83935a640 1 -- 172.21.15.94:0/1542655967 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83406c5a0 con 0x7fa8340c1320 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.835+0000 7fa83935a640 1 --2- 172.21.15.94:0/1542655967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8341566b0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fa8240097a0 tx=0x7fa82402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9fb740def4b1a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.835+0000 7fa838b59640 1 -- 172.21.15.94:0/1542655967 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8240315a0 con 0x7fa8340c1320 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.835+0000 7fa838b59640 1 -- 172.21.15.94:0/1542655967 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa824037070 con 0x7fa8340c1320 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.835+0000 7fa838b59640 1 -- 172.21.15.94:0/1542655967 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa824031960 con 0x7fa8340c1320 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 -- 172.21.15.94:0/1542655967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 msgr2=0x7fa8341566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/1542655967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8341566b0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fa8240097a0 tx=0x7fa82402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 -- 172.21.15.94:0/1542655967 shutdown_connections 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/1542655967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8340c6000 0x7fa834156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/1542655967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8341566b0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 -- 172.21.15.94:0/1542655967 >> 172.21.15.94:0/1542655967 conn(0x7fa8340c6710 msgr2=0x7fa8340c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 -- 172.21.15.94:0/1542655967 shutdown_connections 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.836+0000 7fa83bdbf640 1 -- 172.21.15.94:0/1542655967 wait complete. 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 Processor -- start 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 -- start start 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8340a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c6000 0x7fa834149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa834150580 0x7fa8341541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83406e5a0 con 0x7fa8340c1320 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83406e420 con 0x7fa8340c6000 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.837+0000 7fa83bdbf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8340c0c40 con 0x7fa834150580 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8340a7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8340a7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41810/0 (socket says 172.21.15.94:41810) 2024-03-06T07:25:40.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 -- 172.21.15.94:0/912835874 learned_addr learned my addr 172.21.15.94:0/912835874 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa83935a640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c6000 0x7fa834149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa83a35c640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa834150580 0x7fa8341541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 -- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa834150580 msgr2=0x7fa8341541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa834150580 0x7fa8341541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 -- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c6000 msgr2=0x7fa834149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c6000 0x7fa834149070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 -- 172.21.15.94:0/912835874 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834155b60 con 0x7fa8340c1320 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa839b5b640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8340a7f10 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fa83000c430 tx=0x7fa83000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa82a7fc640 1 -- 172.21.15.94:0/912835874 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8300074e0 con 0x7fa8340c1320 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa82a7fc640 1 -- 172.21.15.94:0/912835874 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8300047e0 con 0x7fa8340c1320 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa834155f80 con 0x7fa8340c1320 2024-03-06T07:25:40.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa82a7fc640 1 -- 172.21.15.94:0/912835874 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830004bd0 con 0x7fa8340c1320 2024-03-06T07:25:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.838+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa834156220 con 0x7fa8340c1320 2024-03-06T07:25:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.839+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa834156460 con 0x7fa8340c1320 2024-03-06T07:25:40.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.840+0000 7fa82a7fc640 1 -- 172.21.15.94:0/912835874 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa830011070 con 0x7fa8340c1320 2024-03-06T07:25:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.840+0000 7fa82a7fc640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.840+0000 7fa82a7fc640 1 -- 172.21.15.94:0/912835874 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa830089e00 con 0x7fa8340c1320 2024-03-06T07:25:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.840+0000 7fa83935a640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.841+0000 7fa83935a640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa824036000 tx=0x7fa82403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:40.844+0000 7fa82a7fc640 1 -- 172.21.15.94:0/912835874 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa834156460 con 0x7fa8340c1320 2024-03-06T07:25:41.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.016+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8340c3a40 con 0x7fa8340c1320 2024-03-06T07:25:41.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.018+0000 7fa82a7fc640 1 -- 172.21.15.94:0/912835874 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+16884 (secure 0 0 0) 0x7fa830058530 con 0x7fa8340c1320 2024-03-06T07:25:41.020 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:41.020 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":182,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:39.369608+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.909545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.667783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:41.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.019+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 msgr2=0x7fa81806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.019+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa824036000 tx=0x7fa82403d040 comp rx=0 tx=0).stop 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.019+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 msgr2=0x7fa8340a7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.019+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8340a7f10 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fa83000c430 tx=0x7fa83000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 shutdown_connections 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa81806b4d0 0x7fa81806d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa834150580 0x7fa8341541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8340c6000 0x7fa834149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 --2- 172.21.15.94:0/912835874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8340c1320 0x7fa8340a7f10 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 >> 172.21.15.94:0/912835874 conn(0x7fa8340c6710 msgr2=0x7fa834153630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 shutdown_connections 2024-03-06T07:25:41.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.020+0000 7fa83bdbf640 1 -- 172.21.15.94:0/912835874 wait complete. 2024-03-06T07:25:41.032 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:41.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:41.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.129+0000 7fbd2c7bc640 1 Processor -- start 2024-03-06T07:25:41.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.129+0000 7fbd2c7bc640 1 -- start start 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.130+0000 7fbd2c7bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbd24150580 0x7fbd2406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.130+0000 7fbd2c7bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd2406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.130+0000 7fbd2c7bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd24153a60 con 0x7fbd240c6000 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.130+0000 7fbd2c7bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd24153d40 con 0x7fbd24150580 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.130+0000 7fbd2c7bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2406c2c0 con 0x7fbd240c1320 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.130+0000 7fbd2a558640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbd24150580 0x7fbd2406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd2ad59640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fbd240c6000 0x7fbd241566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:48172/0 (socket says 172.21.15.94:48172) 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd2ad59640 1 -- 172.21.15.94:0/3767478504 learned_addr learned my addr 172.21.15.94:0/3767478504 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd2a558640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbd24150580 0x7fbd2406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41826/0 (socket says 172.21.15.94:41826) 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd29d57640 1 --2- 172.21.15.94:0/3767478504 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd2406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd2a558640 1 -- 172.21.15.94:0/3767478504 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 msgr2=0x7fbd2406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd2a558640 1 --2- 172.21.15.94:0/3767478504 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd2406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd29556640 1 -- 172.21.15.94:0/3767478504 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627814399 0 0) 0x7fbd24153a60 con 0x7fbd240c6000 2024-03-06T07:25:41.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd2a558640 1 -- 172.21.15.94:0/3767478504 >> v1:172.21.15.145:6789/0 conn(0x7fbd240c6000 legacy=0x7fbd241566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.131+0000 7fbd2a558640 1 -- 172.21.15.94:0/3767478504 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2406c5a0 con 0x7fbd24150580 2024-03-06T07:25:41.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2a558640 1 --2- 172.21.15.94:0/3767478504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd24150580 0x7fbd2406dd10 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fbd200097a0 tx=0x7fbd2002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b1a7ca1a37343e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd29556640 1 -- 172.21.15.94:0/3767478504 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd200315a0 con 0x7fbd24150580 2024-03-06T07:25:41.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd29556640 1 -- 172.21.15.94:0/3767478504 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd20037070 con 0x7fbd24150580 2024-03-06T07:25:41.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd29556640 1 -- 172.21.15.94:0/3767478504 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd20031960 con 0x7fbd24150580 2024-03-06T07:25:41.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3767478504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd24150580 msgr2=0x7fbd2406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3767478504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd24150580 0x7fbd2406dd10 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fbd200097a0 tx=0x7fbd2002b840 comp rx=0 tx=0).stop 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3767478504 shutdown_connections 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3767478504 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd2406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3767478504 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd24150580 0x7fbd2406dd10 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3767478504 >> 172.21.15.94:0/3767478504 conn(0x7fbd240c6710 msgr2=0x7fbd240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3767478504 shutdown_connections 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.132+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3767478504 wait complete. 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.133+0000 7fbd2c7bc640 1 Processor -- start 2024-03-06T07:25:41.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.133+0000 7fbd2c7bc640 1 -- start start 2024-03-06T07:25:41.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2c7bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd241e92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2c7bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd240c6000 0x7fbd24155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2c7bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd24150580 0x7fbd240a6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2c7bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240c0dc0 con 0x7fbd240c6000 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2c7bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240c0c40 con 0x7fbd24150580 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2c7bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24156dc0 con 0x7fbd240c1320 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd29d57640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd240c6000 0x7fbd24155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd29d57640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd240c6000 0x7fbd24155690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41840/0 (socket says 172.21.15.94:41840) 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd29d57640 1 -- 172.21.15.94:0/3984356319 learned_addr learned my addr 172.21.15.94:0/3984356319 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2ad59640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd24150580 0x7fbd240a6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd2a558640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd241e92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd29d57640 1 -- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 msgr2=0x7fbd241e92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.134+0000 7fbd29d57640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd241e92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd29d57640 1 -- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd24150580 msgr2=0x7fbd240a6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd29d57640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd24150580 0x7fbd240a6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd29d57640 1 -- 172.21.15.94:0/3984356319 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240a63d0 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd29d57640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd240c6000 0x7fbd24155690 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400e9e0 tx=0x7fbd1400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd1affd640 1 -- 172.21.15.94:0/3984356319 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14017070 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd1affd640 1 -- 172.21.15.94:0/3984356319 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1400c9d0 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd24072fa0 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd1affd640 1 -- 172.21.15.94:0/3984356319 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1400cdc0 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.135+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24155bd0 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.136+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2404f330 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.136+0000 7fbd1affd640 1 -- 172.21.15.94:0/3984356319 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd1401a070 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.137+0000 7fbd1affd640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0806b4d0 0x7fbd0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.137+0000 7fbd1affd640 1 -- 172.21.15.94:0/3984356319 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbd1408a2f0 con 0x7fbd240c6000 2024-03-06T07:25:41.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.137+0000 7fbd2a558640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0806b4d0 0x7fbd0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.137+0000 7fbd2a558640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0806b4d0 0x7fbd0806d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbd20036000 tx=0x7fbd20008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.141+0000 7fbd1affd640 1 -- 172.21.15.94:0/3984356319 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd14058990 con 0x7fbd240c6000 2024-03-06T07:25:41.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.310+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd240c5de0 con 0x7fbd0806b4d0 2024-03-06T07:25:41.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.322+0000 7fbd1affd640 1 -- 172.21.15.94:0/3984356319 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7fbd240c5de0 con 0x7fbd0806b4d0 2024-03-06T07:25:41.326 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0806b4d0 msgr2=0x7fbd0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0806b4d0 0x7fbd0806d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbd20036000 tx=0x7fbd20008030 comp rx=0 tx=0).stop 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd240c6000 msgr2=0x7fbd24155690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd240c6000 0x7fbd24155690 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400e9e0 tx=0x7fbd1400eea0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 shutdown_connections 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbd0806b4d0 0x7fbd0806d990 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd24150580 0x7fbd240a6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbd240c6000 0x7fbd24155690 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 --2- 172.21.15.94:0/3984356319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd240c1320 0x7fbd241e92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 >> 172.21.15.94:0/3984356319 conn(0x7fbd240c6710 msgr2=0x7fbd24151410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 shutdown_connections 2024-03-06T07:25:41.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.324+0000 7fbd2c7bc640 1 -- 172.21.15.94:0/3984356319 wait complete. 2024-03-06T07:25:41.337 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-03-06T07:25:40.732723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271508,"kb_used_data":17880,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754703212,"statfs":{"total":773094113280,"available":772816089088,"internally_reserved":0,"allocated":18309120,"data_stored":9988974,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.390499"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353125+0000","last_change":"2024-03-06T07:25:37.353125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353125+0000","last_undegraded":"2024-03-06T07:25:37.353125+0000","last_fullsized":"2024-03-06T07:25:37.353125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351988+0000","last_change":"2024-03-06T07:25:37.351988+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351988+0000","last_undegraded":"2024-03-06T07:25:37.351988+0000","last_fullsized":"2024-03-06T07:25:37.351988+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355468+0000","last_change":"2024-03-06T07:25:37.355468+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355468+0000","last_undegraded":"2024-03-06T07:25:37.355468+0000","last_fullsized":"2024-03-06T07:25:37.355468+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347174+0000","last_change":"2024-03-06T07:25:37.347174+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347174+0000","last_undegraded":"2024-03-06T07:25:37.347174+0000","last_fullsized":"2024-03-06T07:25:37.347174+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351436+0000","last_change":"2024-03-06T07:25:37.351436+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351436+0000","last_undegraded":"2024-03-06T07:25:37.351436+0000","last_fullsized":"2024-03-06T07:25:37.351436+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349668+0000","last_change":"2024-03-06T07:25:37.349668+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349668+0000","last_undegraded":"2024-03-06T07:25:37.349668+0000","last_fullsized":"2024-03-06T07:25:37.349668+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353857+0000","last_change":"2024-03-06T07:25:37.353857+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353857+0000","last_undegraded":"2024-03-06T07:25:37.353857+0000","last_fullsized":"2024-03-06T07:25:37.353857+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352760+0000","last_change":"2024-03-06T07:25:37.352760+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352760+0000","last_undegraded":"2024-03-06T07:25:37.352760+0000","last_fullsized":"2024-03-06T07:25:37.352760+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349674+0000","last_change":"2024-03-06T07:25:37.349674+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349674+0000","last_undegraded":"2024-03-06T07:25:37.349674+0000","last_fullsized":"2024-03-06T07:25:37.349674+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355049+0000","last_change":"2024-03-06T07:25:37.355049+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355049+0000","last_undegraded":"2024-03-06T07:25:37.355049+0000","last_fullsized":"2024-03-06T07:25:37.355049+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348776+0000","last_change":"2024-03-06T07:25:37.348776+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348776+0000","last_undegraded":"2024-03-06T07:25:37.348776+0000","last_fullsized":"2024-03-06T07:25:37.348776+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348316+0000","last_change":"2024-03-06T07:25:37.348316+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348316+0000","last_undegraded":"2024-03-06T07:25:37.348316+0000","last_fullsized":"2024-03-06T07:25:37.348316+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351809+0000","last_change":"2024-03-06T07:25:37.351809+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351809+0000","last_undegraded":"2024-03-06T07:25:37.351809+0000","last_fullsized":"2024-03-06T07:25:37.351809+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350214+0000","last_change":"2024-03-06T07:25:37.350214+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350214+0000","last_undegraded":"2024-03-06T07:25:37.350214+0000","last_fullsized":"2024-03-06T07:25:37.350214+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351435+0000","last_change":"2024-03-06T07:25:37.351435+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351435+0000","last_undegraded":"2024-03-06T07:25:37.351435+0000","last_fullsized":"2024-03-06T07:25:37.351435+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350699+0000","last_change":"2024-03-06T07:25:37.350699+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350699+0000","last_undegraded":"2024-03-06T07:25:37.350699+0000","last_fullsized":"2024-03-06T07:25:37.350699+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354123+0000","last_change":"2024-03-06T07:25:37.354123+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354123+0000","last_undegraded":"2024-03-06T07:25:37.354123+0000","last_fullsized":"2024-03-06T07:25:37.354123+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354755+0000","last_change":"2024-03-06T07:25:37.354755+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354755+0000","last_undegraded":"2024-03-06T07:25:37.354755+0000","last_fullsized":"2024-03-06T07:25:37.354755+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350943+0000","last_change":"2024-03-06T07:25:37.350943+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350943+0000","last_undegraded":"2024-03-06T07:25:37.350943+0000","last_fullsized":"2024-03-06T07:25:37.350943+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353972+0000","last_change":"2024-03-06T07:25:37.353972+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353972+0000","last_undegraded":"2024-03-06T07:25:37.353972+0000","last_fullsized":"2024-03-06T07:25:37.353972+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349753+0000","last_change":"2024-03-06T07:25:37.349753+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349753+0000","last_undegraded":"2024-03-06T07:25:37.349753+0000","last_fullsized":"2024-03-06T07:25:37.349753+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349137+0000","last_change":"2024-03-06T07:25:37.349137+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349137+0000","last_undegraded":"2024-03-06T07:25:37.349137+0000","last_fullsized":"2024-03-06T07:25:37.349137+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355131+0000","last_change":"2024-03-06T07:25:37.355131+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355131+0000","last_undegraded":"2024-03-06T07:25:37.355131+0000","last_fullsized":"2024-03-06T07:25:37.355131+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353675+0000","last_change":"2024-03-06T07:25:37.353675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353675+0000","last_undegraded":"2024-03-06T07:25:37.353675+0000","last_fullsized":"2024-03-06T07:25:37.353675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348241+0000","last_change":"2024-03-06T07:25:37.348241+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348241+0000","last_undegraded":"2024-03-06T07:25:37.348241+0000","last_fullsized":"2024-03-06T07:25:37.348241+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347813+0000","last_change":"2024-03-06T07:25:37.347813+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347813+0000","last_undegraded":"2024-03-06T07:25:37.347813+0000","last_fullsized":"2024-03-06T07:25:37.347813+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351675+0000","last_change":"2024-03-06T07:25:37.351675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351675+0000","last_undegraded":"2024-03-06T07:25:37.351675+0000","last_fullsized":"2024-03-06T07:25:37.351675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350203+0000","last_change":"2024-03-06T07:25:37.350203+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350203+0000","last_undegraded":"2024-03-06T07:25:37.350203+0000","last_fullsized":"2024-03-06T07:25:37.350203+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348621+0000","last_change":"2024-03-06T07:25:37.348621+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348621+0000","last_undegraded":"2024-03-06T07:25:37.348621+0000","last_fullsized":"2024-03-06T07:25:37.348621+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353167+0000","last_change":"2024-03-06T07:25:37.353167+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353167+0000","last_undegraded":"2024-03-06T07:25:37.353167+0000","last_fullsized":"2024-03-06T07:25:37.353167+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361035+0000","last_change":"2024-03-06T07:25:37.360161+0000","last_active":"2024-03-06T07:25:37.361035+0000","last_peered":"2024-03-06T07:25:37.361035+0000","last_clean":"2024-03-06T07:25:37.361035+0000","last_became_active":"2024-03-06T07:25:37.359858+0000","last_became_peered":"2024-03-06T07:25:37.359858+0000","last_unstale":"2024-03-06T07:25:37.361035+0000","last_undegraded":"2024-03-06T07:25:37.361035+0000","last_fullsized":"2024-03-06T07:25:37.361035+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:11.304824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353595+0000","last_change":"2024-03-06T07:25:37.353595+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353595+0000","last_undegraded":"2024-03-06T07:25:37.353595+0000","last_fullsized":"2024-03-06T07:25:37.353595+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.366147+0000","last_change":"2024-03-06T07:25:37.361735+0000","last_active":"2024-03-06T07:25:37.366147+0000","last_peered":"2024-03-06T07:25:37.366147+0000","last_clean":"2024-03-06T07:25:37.366147+0000","last_became_active":"2024-03-06T07:25:37.361071+0000","last_became_peered":"2024-03-06T07:25:37.361071+0000","last_unstale":"2024-03-06T07:25:37.366147+0000","last_undegraded":"2024-03-06T07:25:37.366147+0000","last_fullsized":"2024-03-06T07:25:37.366147+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:38.535183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353456+0000","last_change":"2024-03-06T07:25:37.353456+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353456+0000","last_undegraded":"2024-03-06T07:25:37.353456+0000","last_fullsized":"2024-03-06T07:25:37.353456+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358849+0000","last_change":"2024-03-06T07:25:37.358208+0000","last_active":"2024-03-06T07:25:37.358849+0000","last_peered":"2024-03-06T07:25:37.358849+0000","last_clean":"2024-03-06T07:25:37.358849+0000","last_became_active":"2024-03-06T07:25:37.357941+0000","last_became_peered":"2024-03-06T07:25:37.357941+0000","last_unstale":"2024-03-06T07:25:37.358849+0000","last_undegraded":"2024-03-06T07:25:37.358849+0000","last_fullsized":"2024-03-06T07:25:37.358849+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:04.977160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353298+0000","last_change":"2024-03-06T07:25:37.353298+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353298+0000","last_undegraded":"2024-03-06T07:25:37.353298+0000","last_fullsized":"2024-03-06T07:25:37.353298+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358129+0000","last_change":"2024-03-06T07:25:37.357145+0000","last_active":"2024-03-06T07:25:37.358129+0000","last_peered":"2024-03-06T07:25:37.358129+0000","last_clean":"2024-03-06T07:25:37.358129+0000","last_became_active":"2024-03-06T07:25:37.356819+0000","last_became_peered":"2024-03-06T07:25:37.356819+0000","last_unstale":"2024-03-06T07:25:37.358129+0000","last_undegraded":"2024-03-06T07:25:37.358129+0000","last_fullsized":"2024-03-06T07:25:37.358129+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.694205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350553+0000","last_change":"2024-03-06T07:25:37.350553+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350553+0000","last_undegraded":"2024-03-06T07:25:37.350553+0000","last_fullsized":"2024-03-06T07:25:37.350553+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368484+0000","last_change":"2024-03-06T07:25:37.366475+0000","last_active":"2024-03-06T07:25:37.368484+0000","last_peered":"2024-03-06T07:25:37.368484+0000","last_clean":"2024-03-06T07:25:37.368484+0000","last_became_active":"2024-03-06T07:25:37.366203+0000","last_became_peered":"2024-03-06T07:25:37.366203+0000","last_unstale":"2024-03-06T07:25:37.368484+0000","last_undegraded":"2024-03-06T07:25:37.368484+0000","last_fullsized":"2024-03-06T07:25:37.368484+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:33.314274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349256+0000","last_change":"2024-03-06T07:25:37.349256+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349256+0000","last_undegraded":"2024-03-06T07:25:37.349256+0000","last_fullsized":"2024-03-06T07:25:37.349256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364042+0000","last_change":"2024-03-06T07:25:37.362493+0000","last_active":"2024-03-06T07:25:37.364042+0000","last_peered":"2024-03-06T07:25:37.364042+0000","last_clean":"2024-03-06T07:25:37.364042+0000","last_became_active":"2024-03-06T07:25:37.362019+0000","last_became_peered":"2024-03-06T07:25:37.362019+0000","last_unstale":"2024-03-06T07:25:37.364042+0000","last_undegraded":"2024-03-06T07:25:37.364042+0000","last_fullsized":"2024-03-06T07:25:37.364042+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348628+0000","last_change":"2024-03-06T07:25:37.348628+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348628+0000","last_undegraded":"2024-03-06T07:25:37.348628+0000","last_fullsized":"2024-03-06T07:25:37.348628+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360769+0000","last_change":"2024-03-06T07:25:37.359237+0000","last_active":"2024-03-06T07:25:37.360769+0000","last_peered":"2024-03-06T07:25:37.360769+0000","last_clean":"2024-03-06T07:25:37.360769+0000","last_became_active":"2024-03-06T07:25:37.358927+0000","last_became_peered":"2024-03-06T07:25:37.358927+0000","last_unstale":"2024-03-06T07:25:37.360769+0000","last_undegraded":"2024-03-06T07:25:37.360769+0000","last_fullsized":"2024-03-06T07:25:37.360769+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:08.259298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353004+0000","last_change":"2024-03-06T07:25:37.353004+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353004+0000","last_undegraded":"2024-03-06T07:25:37.353004+0000","last_fullsized":"2024-03-06T07:25:37.353004+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368462+0000","last_change":"2024-03-06T07:25:37.367257+0000","last_active":"2024-03-06T07:25:37.368462+0000","last_peered":"2024-03-06T07:25:37.368462+0000","last_clean":"2024-03-06T07:25:37.368462+0000","last_became_active":"2024-03-06T07:25:37.366971+0000","last_became_peered":"2024-03-06T07:25:37.366971+0000","last_unstale":"2024-03-06T07:25:37.368462+0000","last_undegraded":"2024-03-06T07:25:37.368462+0000","last_fullsized":"2024-03-06T07:25:37.368462+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.083348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355752+0000","last_change":"2024-03-06T07:25:37.355752+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355752+0000","last_undegraded":"2024-03-06T07:25:37.355752+0000","last_fullsized":"2024-03-06T07:25:37.355752+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364834+0000","last_change":"2024-03-06T07:25:37.363334+0000","last_active":"2024-03-06T07:25:37.364834+0000","last_peered":"2024-03-06T07:25:37.364834+0000","last_clean":"2024-03-06T07:25:37.364834+0000","last_became_active":"2024-03-06T07:25:37.362631+0000","last_became_peered":"2024-03-06T07:25:37.362631+0000","last_unstale":"2024-03-06T07:25:37.364834+0000","last_undegraded":"2024-03-06T07:25:37.364834+0000","last_fullsized":"2024-03-06T07:25:37.364834+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352531+0000","last_change":"2024-03-06T07:25:37.352531+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352531+0000","last_undegraded":"2024-03-06T07:25:37.352531+0000","last_fullsized":"2024-03-06T07:25:37.352531+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368576+0000","last_change":"2024-03-06T07:25:37.367405+0000","last_active":"2024-03-06T07:25:37.368576+0000","last_peered":"2024-03-06T07:25:37.368576+0000","last_clean":"2024-03-06T07:25:37.368576+0000","last_became_active":"2024-03-06T07:25:37.367077+0000","last_became_peered":"2024-03-06T07:25:37.367077+0000","last_unstale":"2024-03-06T07:25:37.368576+0000","last_undegraded":"2024-03-06T07:25:37.368576+0000","last_fullsized":"2024-03-06T07:25:37.368576+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348006+0000","last_change":"2024-03-06T07:25:37.348006+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348006+0000","last_undegraded":"2024-03-06T07:25:37.348006+0000","last_fullsized":"2024-03-06T07:25:37.348006+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360396+0000","last_change":"2024-03-06T07:25:37.359803+0000","last_active":"2024-03-06T07:25:37.360396+0000","last_peered":"2024-03-06T07:25:37.360396+0000","last_clean":"2024-03-06T07:25:37.360396+0000","last_became_active":"2024-03-06T07:25:37.359509+0000","last_became_peered":"2024-03-06T07:25:37.359509+0000","last_unstale":"2024-03-06T07:25:37.360396+0000","last_undegraded":"2024-03-06T07:25:37.360396+0000","last_fullsized":"2024-03-06T07:25:37.360396+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:16.038727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368085+0000","last_change":"2024-03-06T07:25:37.363650+0000","last_active":"2024-03-06T07:25:37.368085+0000","last_peered":"2024-03-06T07:25:37.368085+0000","last_clean":"2024-03-06T07:25:37.368085+0000","last_became_active":"2024-03-06T07:25:37.363319+0000","last_became_peered":"2024-03-06T07:25:37.363319+0000","last_unstale":"2024-03-06T07:25:37.368085+0000","last_undegraded":"2024-03-06T07:25:37.368085+0000","last_fullsized":"2024-03-06T07:25:37.368085+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:03.107757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347979+0000","last_change":"2024-03-06T07:25:37.347979+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347979+0000","last_undegraded":"2024-03-06T07:25:37.347979+0000","last_fullsized":"2024-03-06T07:25:37.347979+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359733+0000","last_change":"2024-03-06T07:25:37.357207+0000","last_active":"2024-03-06T07:25:37.359733+0000","last_peered":"2024-03-06T07:25:37.359733+0000","last_clean":"2024-03-06T07:25:37.359733+0000","last_became_active":"2024-03-06T07:25:37.356920+0000","last_became_peered":"2024-03-06T07:25:37.356920+0000","last_unstale":"2024-03-06T07:25:37.359733+0000","last_undegraded":"2024-03-06T07:25:37.359733+0000","last_fullsized":"2024-03-06T07:25:37.359733+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351780+0000","last_change":"2024-03-06T07:25:37.351780+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351780+0000","last_undegraded":"2024-03-06T07:25:37.351780+0000","last_fullsized":"2024-03-06T07:25:37.351780+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361242+0000","last_change":"2024-03-06T07:25:37.360307+0000","last_active":"2024-03-06T07:25:37.361242+0000","last_peered":"2024-03-06T07:25:37.361242+0000","last_clean":"2024-03-06T07:25:37.361242+0000","last_became_active":"2024-03-06T07:25:37.360024+0000","last_became_peered":"2024-03-06T07:25:37.360024+0000","last_unstale":"2024-03-06T07:25:37.361242+0000","last_undegraded":"2024-03-06T07:25:37.361242+0000","last_fullsized":"2024-03-06T07:25:37.361242+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:13.020947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349007+0000","last_change":"2024-03-06T07:25:37.349007+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349007+0000","last_undegraded":"2024-03-06T07:25:37.349007+0000","last_fullsized":"2024-03-06T07:25:37.349007+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368184+0000","last_change":"2024-03-06T07:25:37.364965+0000","last_active":"2024-03-06T07:25:37.368184+0000","last_peered":"2024-03-06T07:25:37.368184+0000","last_clean":"2024-03-06T07:25:37.368184+0000","last_became_active":"2024-03-06T07:25:37.363889+0000","last_became_peered":"2024-03-06T07:25:37.363889+0000","last_unstale":"2024-03-06T07:25:37.368184+0000","last_undegraded":"2024-03-06T07:25:37.368184+0000","last_fullsized":"2024-03-06T07:25:37.368184+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352125+0000","last_change":"2024-03-06T07:25:37.352125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352125+0000","last_undegraded":"2024-03-06T07:25:37.352125+0000","last_fullsized":"2024-03-06T07:25:37.352125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364603+0000","last_change":"2024-03-06T07:25:37.364101+0000","last_active":"2024-03-06T07:25:37.364603+0000","last_peered":"2024-03-06T07:25:37.364603+0000","last_clean":"2024-03-06T07:25:37.364603+0000","last_became_active":"2024-03-06T07:25:37.363814+0000","last_became_peered":"2024-03-06T07:25:37.363814+0000","last_unstale":"2024-03-06T07:25:37.364603+0000","last_undegraded":"2024-03-06T07:25:37.364603+0000","last_fullsized":"2024-03-06T07:25:37.364603+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:08.141356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.356323+0000","last_change":"2024-03-06T07:25:37.356323+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.356323+0000","last_undegraded":"2024-03-06T07:25:37.356323+0000","last_fullsized":"2024-03-06T07:25:37.356323+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359709+0000","last_change":"2024-03-06T07:25:37.358403+0000","last_active":"2024-03-06T07:25:37.359709+0000","last_peered":"2024-03-06T07:25:37.359709+0000","last_clean":"2024-03-06T07:25:37.359709+0000","last_became_active":"2024-03-06T07:25:37.358134+0000","last_became_peered":"2024-03-06T07:25:37.358134+0000","last_unstale":"2024-03-06T07:25:37.359709+0000","last_undegraded":"2024-03-06T07:25:37.359709+0000","last_fullsized":"2024-03-06T07:25:37.359709+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.829122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352078+0000","last_change":"2024-03-06T07:25:37.352078+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352078+0000","last_undegraded":"2024-03-06T07:25:37.352078+0000","last_fullsized":"2024-03-06T07:25:37.352078+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358546+0000","last_change":"2024-03-06T07:25:37.355316+0000","last_active":"2024-03-06T07:25:37.358546+0000","last_peered":"2024-03-06T07:25:37.358546+0000","last_clean":"2024-03-06T07:25:37.358546+0000","last_became_active":"2024-03-06T07:25:37.355060+0000","last_became_peered":"2024-03-06T07:25:37.355060+0000","last_unstale":"2024-03-06T07:25:37.358546+0000","last_undegraded":"2024-03-06T07:25:37.358546+0000","last_fullsized":"2024-03-06T07:25:37.358546+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:27.708230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350295+0000","last_change":"2024-03-06T07:25:37.350295+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350295+0000","last_undegraded":"2024-03-06T07:25:37.350295+0000","last_fullsized":"2024-03-06T07:25:37.350295+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.357802+0000","last_change":"2024-03-06T07:25:37.354804+0000","last_active":"2024-03-06T07:25:37.357802+0000","last_peered":"2024-03-06T07:25:37.357802+0000","last_clean":"2024-03-06T07:25:37.357802+0000","last_became_active":"2024-03-06T07:25:37.354535+0000","last_became_peered":"2024-03-06T07:25:37.354535+0000","last_unstale":"2024-03-06T07:25:37.357802+0000","last_undegraded":"2024-03-06T07:25:37.357802+0000","last_fullsized":"2024-03-06T07:25:37.357802+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:11.747252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351130+0000","last_change":"2024-03-06T07:25:37.351130+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351130+0000","last_undegraded":"2024-03-06T07:25:37.351130+0000","last_fullsized":"2024-03-06T07:25:37.351130+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.367951+0000","last_change":"2024-03-06T07:25:37.362425+0000","last_active":"2024-03-06T07:25:37.367951+0000","last_peered":"2024-03-06T07:25:37.367951+0000","last_clean":"2024-03-06T07:25:37.367951+0000","last_became_active":"2024-03-06T07:25:37.362172+0000","last_became_peered":"2024-03-06T07:25:37.362172+0000","last_unstale":"2024-03-06T07:25:37.367951+0000","last_undegraded":"2024-03-06T07:25:37.367951+0000","last_fullsized":"2024-03-06T07:25:37.367951+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:46.062578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350611+0000","last_change":"2024-03-06T07:25:37.350611+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350611+0000","last_undegraded":"2024-03-06T07:25:37.350611+0000","last_fullsized":"2024-03-06T07:25:37.350611+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356209+0000","last_change":"2024-03-06T07:25:37.354863+0000","last_active":"2024-03-06T07:25:37.356209+0000","last_peered":"2024-03-06T07:25:37.356209+0000","last_clean":"2024-03-06T07:25:37.356209+0000","last_became_active":"2024-03-06T07:25:37.354465+0000","last_became_peered":"2024-03-06T07:25:37.354465+0000","last_unstale":"2024-03-06T07:25:37.356209+0000","last_undegraded":"2024-03-06T07:25:37.356209+0000","last_fullsized":"2024-03-06T07:25:37.356209+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:33.951539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349868+0000","last_change":"2024-03-06T07:25:37.349868+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349868+0000","last_undegraded":"2024-03-06T07:25:37.349868+0000","last_fullsized":"2024-03-06T07:25:37.349868+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362855+0000","last_change":"2024-03-06T07:25:37.361572+0000","last_active":"2024-03-06T07:25:37.362855+0000","last_peered":"2024-03-06T07:25:37.362855+0000","last_clean":"2024-03-06T07:25:37.362855+0000","last_became_active":"2024-03-06T07:25:37.361271+0000","last_became_peered":"2024-03-06T07:25:37.361271+0000","last_unstale":"2024-03-06T07:25:37.362855+0000","last_undegraded":"2024-03-06T07:25:37.362855+0000","last_fullsized":"2024-03-06T07:25:37.362855+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:13.032326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351680+0000","last_change":"2024-03-06T07:25:37.351680+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351680+0000","last_undegraded":"2024-03-06T07:25:37.351680+0000","last_fullsized":"2024-03-06T07:25:37.351680+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360711+0000","last_change":"2024-03-06T07:25:37.359314+0000","last_active":"2024-03-06T07:25:37.360711+0000","last_peered":"2024-03-06T07:25:37.360711+0000","last_clean":"2024-03-06T07:25:37.360711+0000","last_became_active":"2024-03-06T07:25:37.359008+0000","last_became_peered":"2024-03-06T07:25:37.359008+0000","last_unstale":"2024-03-06T07:25:37.360711+0000","last_undegraded":"2024-03-06T07:25:37.360711+0000","last_fullsized":"2024-03-06T07:25:37.360711+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:07.137677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351711+0000","last_change":"2024-03-06T07:25:37.351711+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351711+0000","last_undegraded":"2024-03-06T07:25:37.351711+0000","last_fullsized":"2024-03-06T07:25:37.351711+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368155+0000","last_change":"2024-03-06T07:25:37.364253+0000","last_active":"2024-03-06T07:25:37.368155+0000","last_peered":"2024-03-06T07:25:37.368155+0000","last_clean":"2024-03-06T07:25:37.368155+0000","last_became_active":"2024-03-06T07:25:37.363933+0000","last_became_peered":"2024-03-06T07:25:37.363933+0000","last_unstale":"2024-03-06T07:25:37.368155+0000","last_undegraded":"2024-03-06T07:25:37.368155+0000","last_fullsized":"2024-03-06T07:25:37.368155+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355920+0000","last_change":"2024-03-06T07:25:37.355920+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355920+0000","last_undegraded":"2024-03-06T07:25:37.355920+0000","last_fullsized":"2024-03-06T07:25:37.355920+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364944+0000","last_change":"2024-03-06T07:25:37.364443+0000","last_active":"2024-03-06T07:25:37.364944+0000","last_peered":"2024-03-06T07:25:37.364944+0000","last_clean":"2024-03-06T07:25:37.364944+0000","last_became_active":"2024-03-06T07:25:37.364093+0000","last_became_peered":"2024-03-06T07:25:37.364093+0000","last_unstale":"2024-03-06T07:25:37.364944+0000","last_undegraded":"2024-03-06T07:25:37.364944+0000","last_fullsized":"2024-03-06T07:25:37.364944+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:01.483203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351304+0000","last_change":"2024-03-06T07:25:37.351304+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351304+0000","last_undegraded":"2024-03-06T07:25:37.351304+0000","last_fullsized":"2024-03-06T07:25:37.351304+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.363191+0000","last_change":"2024-03-06T07:25:37.362488+0000","last_active":"2024-03-06T07:25:37.363191+0000","last_peered":"2024-03-06T07:25:37.363191+0000","last_clean":"2024-03-06T07:25:37.363191+0000","last_became_active":"2024-03-06T07:25:37.362203+0000","last_became_peered":"2024-03-06T07:25:37.362203+0000","last_unstale":"2024-03-06T07:25:37.363191+0000","last_undegraded":"2024-03-06T07:25:37.363191+0000","last_fullsized":"2024-03-06T07:25:37.363191+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.309117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348099+0000","last_change":"2024-03-06T07:25:37.348099+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348099+0000","last_undegraded":"2024-03-06T07:25:37.348099+0000","last_fullsized":"2024-03-06T07:25:37.348099+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368386+0000","last_change":"2024-03-06T07:25:37.366204+0000","last_active":"2024-03-06T07:25:37.368386+0000","last_peered":"2024-03-06T07:25:37.368386+0000","last_clean":"2024-03-06T07:25:37.368386+0000","last_became_active":"2024-03-06T07:25:37.365751+0000","last_became_peered":"2024-03-06T07:25:37.365751+0000","last_unstale":"2024-03-06T07:25:37.368386+0000","last_undegraded":"2024-03-06T07:25:37.368386+0000","last_fullsized":"2024-03-06T07:25:37.368386+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350714+0000","last_change":"2024-03-06T07:25:37.350714+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350714+0000","last_undegraded":"2024-03-06T07:25:37.350714+0000","last_fullsized":"2024-03-06T07:25:37.350714+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368066+0000","last_change":"2024-03-06T07:25:37.363581+0000","last_active":"2024-03-06T07:25:37.368066+0000","last_peered":"2024-03-06T07:25:37.368066+0000","last_clean":"2024-03-06T07:25:37.368066+0000","last_became_active":"2024-03-06T07:25:37.362991+0000","last_became_peered":"2024-03-06T07:25:37.362991+0000","last_unstale":"2024-03-06T07:25:37.368066+0000","last_undegraded":"2024-03-06T07:25:37.368066+0000","last_fullsized":"2024-03-06T07:25:37.368066+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:25.392712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351198+0000","last_change":"2024-03-06T07:25:37.351198+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351198+0000","last_undegraded":"2024-03-06T07:25:37.351198+0000","last_fullsized":"2024-03-06T07:25:37.351198+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362479+0000","last_change":"2024-03-06T07:25:37.361352+0000","last_active":"2024-03-06T07:25:37.362479+0000","last_peered":"2024-03-06T07:25:37.362479+0000","last_clean":"2024-03-06T07:25:37.362479+0000","last_became_active":"2024-03-06T07:25:37.361059+0000","last_became_peered":"2024-03-06T07:25:37.361059+0000","last_unstale":"2024-03-06T07:25:37.362479+0000","last_undegraded":"2024-03-06T07:25:37.362479+0000","last_fullsized":"2024-03-06T07:25:37.362479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348043+0000","last_change":"2024-03-06T07:25:37.348043+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348043+0000","last_undegraded":"2024-03-06T07:25:37.348043+0000","last_fullsized":"2024-03-06T07:25:37.348043+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368916+0000","last_change":"2024-03-06T07:25:37.368239+0000","last_active":"2024-03-06T07:25:37.368916+0000","last_peered":"2024-03-06T07:25:37.368916+0000","last_clean":"2024-03-06T07:25:37.368916+0000","last_became_active":"2024-03-06T07:25:37.367979+0000","last_became_peered":"2024-03-06T07:25:37.367979+0000","last_unstale":"2024-03-06T07:25:37.368916+0000","last_undegraded":"2024-03-06T07:25:37.368916+0000","last_fullsized":"2024-03-06T07:25:37.368916+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:03.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349897+0000","last_change":"2024-03-06T07:25:37.349897+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349897+0000","last_undegraded":"2024-03-06T07:25:37.349897+0000","last_fullsized":"2024-03-06T07:25:37.349897+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368412+0000","last_change":"2024-03-06T07:25:37.366881+0000","last_active":"2024-03-06T07:25:37.368412+0000","last_peered":"2024-03-06T07:25:37.368412+0000","last_clean":"2024-03-06T07:25:37.368412+0000","last_became_active":"2024-03-06T07:25:37.366588+0000","last_became_peered":"2024-03-06T07:25:37.366588+0000","last_unstale":"2024-03-06T07:25:37.368412+0000","last_undegraded":"2024-03-06T07:25:37.368412+0000","last_fullsized":"2024-03-06T07:25:37.368412+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:46.200362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354567+0000","last_change":"2024-03-06T07:25:37.354567+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354567+0000","last_undegraded":"2024-03-06T07:25:37.354567+0000","last_fullsized":"2024-03-06T07:25:37.354567+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364364+0000","last_change":"2024-03-06T07:25:37.359284+0000","last_active":"2024-03-06T07:25:37.364364+0000","last_peered":"2024-03-06T07:25:37.364364+0000","last_clean":"2024-03-06T07:25:37.364364+0000","last_became_active":"2024-03-06T07:25:37.358847+0000","last_became_peered":"2024-03-06T07:25:37.358847+0000","last_unstale":"2024-03-06T07:25:37.364364+0000","last_undegraded":"2024-03-06T07:25:37.364364+0000","last_fullsized":"2024-03-06T07:25:37.364364+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.913135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":472,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356621+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:25:37.356621+0000","last_peered":"2024-03-06T07:25:37.356621+0000","last_clean":"2024-03-06T07:25:37.356621+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:25:37.356621+0000","last_undegraded":"2024-03-06T07:25:37.356621+0000","last_fullsized":"2024-03-06T07:25:37.356621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33412,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338428,"statfs":{"total":96636764160,"available":96602550272,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37624,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334216,"statfs":{"total":96636764160,"available":96598237184,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:41.345 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.476+0000 7fa779fa0640 1 Processor -- start 2024-03-06T07:25:41.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.476+0000 7fa779fa0640 1 -- start start 2024-03-06T07:25:41.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa779fa0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa7740c77b0 0x7fa77406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa779fa0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77406e820 0x7fa77414c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa779fa0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7740c2200 con 0x7fa77414ccf0 2024-03-06T07:25:41.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa779fa0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7740c1fb0 con 0x7fa7740c77b0 2024-03-06T07:25:41.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa779fa0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740c2b30 con 0x7fa77406e820 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa778f9e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa7740c77b0 0x7fa77406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa77979f640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa77414ccf0 0x7fa774152ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:48182/0 (socket says 172.21.15.94:48182) 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa773fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77406e820 0x7fa77414c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.477+0000 7fa77979f640 1 -- 172.21.15.94:0/2087674186 learned_addr learned my addr 172.21.15.94:0/2087674186 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa773fff640 1 -- 172.21.15.94:0/2087674186 >> v2:172.21.15.145:3300/0 conn(0x7fa7740c77b0 msgr2=0x7fa77406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa773fff640 1 --2- 172.21.15.94:0/2087674186 >> v2:172.21.15.145:3300/0 conn(0x7fa7740c77b0 0x7fa77406e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa773fff640 1 -- 172.21.15.94:0/2087674186 >> v1:172.21.15.145:6789/0 conn(0x7fa77414ccf0 legacy=0x7fa774152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa773fff640 1 -- 172.21.15.94:0/2087674186 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7740c24e0 con 0x7fa77406e820 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa7737fe640 1 -- 172.21.15.94:0/2087674186 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670096698 0 0) 0x7fa7740c2200 con 0x7fa77414ccf0 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa778f9e640 1 --2- 172.21.15.94:0/2087674186 >> v2:172.21.15.145:3300/0 conn(0x7fa7740c77b0 0x7fa77406e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa773fff640 1 --2- 172.21.15.94:0/2087674186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77406e820 0x7fa77414c7b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa7680097a0 tx=0x7fa76802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f717888a796834 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.478+0000 7fa7737fe640 1 -- 172.21.15.94:0/2087674186 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768031780 con 0x7fa77406e820 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa7737fe640 1 -- 172.21.15.94:0/2087674186 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa768037070 con 0x7fa77406e820 2024-03-06T07:25:41.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa7737fe640 1 -- 172.21.15.94:0/2087674186 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768031b40 con 0x7fa77406e820 2024-03-06T07:25:41.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 -- 172.21.15.94:0/2087674186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77406e820 msgr2=0x7fa77414c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 --2- 172.21.15.94:0/2087674186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77406e820 0x7fa77414c7b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa7680097a0 tx=0x7fa76802b820 comp rx=0 tx=0).stop 2024-03-06T07:25:41.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 -- 172.21.15.94:0/2087674186 shutdown_connections 2024-03-06T07:25:41.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 --2- 172.21.15.94:0/2087674186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77406e820 0x7fa77414c7b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 --2- 172.21.15.94:0/2087674186 >> v2:172.21.15.145:3300/0 conn(0x7fa7740c77b0 0x7fa77406e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 -- 172.21.15.94:0/2087674186 >> 172.21.15.94:0/2087674186 conn(0x7fa7740a9450 msgr2=0x7fa7740a9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 -- 172.21.15.94:0/2087674186 shutdown_connections 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.479+0000 7fa779fa0640 1 -- 172.21.15.94:0/2087674186 wait complete. 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.480+0000 7fa779fa0640 1 Processor -- start 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.480+0000 7fa779fa0640 1 -- start start 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa779fa0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa77406e820 0x7fa7740ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa779fa0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa7740c77b0 0x7fa774151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa779fa0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77414ccf0 0x7fa7740abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa779fa0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77406bb40 con 0x7fa7740c77b0 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa779fa0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77414d160 con 0x7fa77406e820 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa779fa0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77406bcc0 con 0x7fa77414ccf0 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa7740c77b0 0x7fa774151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa7740c77b0 0x7fa774151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41844/0 (socket says 172.21.15.94:41844) 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa778f9e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa77406e820 0x7fa7740ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 -- 172.21.15.94:0/943976019 learned_addr learned my addr 172.21.15.94:0/943976019 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa77979f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77414ccf0 0x7fa7740abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 -- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77414ccf0 msgr2=0x7fa7740abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77414ccf0 0x7fa7740abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 -- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa77406e820 msgr2=0x7fa7740ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa77406e820 0x7fa7740ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.481+0000 7fa773fff640 1 -- 172.21.15.94:0/943976019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7741ec140 con 0x7fa7740c77b0 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.482+0000 7fa773fff640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa7740c77b0 0x7fa774151f20 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fa7680098c0 tx=0x7fa7680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.482+0000 7fa7717fa640 1 -- 172.21.15.94:0/943976019 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768032610 con 0x7fa7740c77b0 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.482+0000 7fa7717fa640 1 -- 172.21.15.94:0/943976019 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7680327b0 con 0x7fa7740c77b0 2024-03-06T07:25:41.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.482+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7741ebe60 con 0x7fa7740c77b0 2024-03-06T07:25:41.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.482+0000 7fa7717fa640 1 -- 172.21.15.94:0/943976019 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768032ba0 con 0x7fa7740c77b0 2024-03-06T07:25:41.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.482+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7740aae20 con 0x7fa7740c77b0 2024-03-06T07:25:41.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.483+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa774069430 con 0x7fa7740c77b0 2024-03-06T07:25:41.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.483+0000 7fa7717fa640 1 -- 172.21.15.94:0/943976019 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa76804e440 con 0x7fa7740c77b0 2024-03-06T07:25:41.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.484+0000 7fa7717fa640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.484+0000 7fa7717fa640 1 -- 172.21.15.94:0/943976019 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa7680abf20 con 0x7fa7740c77b0 2024-03-06T07:25:41.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.484+0000 7fa778f9e640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.484+0000 7fa778f9e640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa76006b4d0 0x7fa76006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa764009560 tx=0x7fa764009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.488+0000 7fa7717fa640 1 -- 172.21.15.94:0/943976019 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa768040070 con 0x7fa7740c77b0 2024-03-06T07:25:41.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.655+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7741ebc10 con 0x7fa76006b4d0 2024-03-06T07:25:41.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.661+0000 7fa7717fa640 1 -- 172.21.15.94:0/943976019 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7fa7741ebc10 con 0x7fa76006b4d0 2024-03-06T07:25:41.665 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:41.666 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa76006b4d0 msgr2=0x7fa76006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa76006b4d0 0x7fa76006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa764009560 tx=0x7fa764009190 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa7740c77b0 msgr2=0x7fa774151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa7740c77b0 0x7fa774151f20 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fa7680098c0 tx=0x7fa7680046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 shutdown_connections 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa77414ccf0 0x7fa7740abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa7740c77b0 0x7fa774151f20 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 --2- 172.21.15.94:0/943976019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa77406e820 0x7fa7740ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 >> 172.21.15.94:0/943976019 conn(0x7fa7740a9450 msgr2=0x7fa774152ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 shutdown_connections 2024-03-06T07:25:41.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.664+0000 7fa779fa0640 1 -- 172.21.15.94:0/943976019 wait complete. 2024-03-06T07:25:41.676 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-03-06T07:25:40.732723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271508,"kb_used_data":17880,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754703212,"statfs":{"total":773094113280,"available":772816089088,"internally_reserved":0,"allocated":18309120,"data_stored":9988974,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.390499"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353125+0000","last_change":"2024-03-06T07:25:37.353125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353125+0000","last_undegraded":"2024-03-06T07:25:37.353125+0000","last_fullsized":"2024-03-06T07:25:37.353125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351988+0000","last_change":"2024-03-06T07:25:37.351988+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351988+0000","last_undegraded":"2024-03-06T07:25:37.351988+0000","last_fullsized":"2024-03-06T07:25:37.351988+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355468+0000","last_change":"2024-03-06T07:25:37.355468+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355468+0000","last_undegraded":"2024-03-06T07:25:37.355468+0000","last_fullsized":"2024-03-06T07:25:37.355468+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347174+0000","last_change":"2024-03-06T07:25:37.347174+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347174+0000","last_undegraded":"2024-03-06T07:25:37.347174+0000","last_fullsized":"2024-03-06T07:25:37.347174+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351436+0000","last_change":"2024-03-06T07:25:37.351436+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351436+0000","last_undegraded":"2024-03-06T07:25:37.351436+0000","last_fullsized":"2024-03-06T07:25:37.351436+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349668+0000","last_change":"2024-03-06T07:25:37.349668+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349668+0000","last_undegraded":"2024-03-06T07:25:37.349668+0000","last_fullsized":"2024-03-06T07:25:37.349668+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353857+0000","last_change":"2024-03-06T07:25:37.353857+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353857+0000","last_undegraded":"2024-03-06T07:25:37.353857+0000","last_fullsized":"2024-03-06T07:25:37.353857+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352760+0000","last_change":"2024-03-06T07:25:37.352760+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352760+0000","last_undegraded":"2024-03-06T07:25:37.352760+0000","last_fullsized":"2024-03-06T07:25:37.352760+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349674+0000","last_change":"2024-03-06T07:25:37.349674+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349674+0000","last_undegraded":"2024-03-06T07:25:37.349674+0000","last_fullsized":"2024-03-06T07:25:37.349674+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355049+0000","last_change":"2024-03-06T07:25:37.355049+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355049+0000","last_undegraded":"2024-03-06T07:25:37.355049+0000","last_fullsized":"2024-03-06T07:25:37.355049+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348776+0000","last_change":"2024-03-06T07:25:37.348776+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348776+0000","last_undegraded":"2024-03-06T07:25:37.348776+0000","last_fullsized":"2024-03-06T07:25:37.348776+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348316+0000","last_change":"2024-03-06T07:25:37.348316+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348316+0000","last_undegraded":"2024-03-06T07:25:37.348316+0000","last_fullsized":"2024-03-06T07:25:37.348316+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351809+0000","last_change":"2024-03-06T07:25:37.351809+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351809+0000","last_undegraded":"2024-03-06T07:25:37.351809+0000","last_fullsized":"2024-03-06T07:25:37.351809+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350214+0000","last_change":"2024-03-06T07:25:37.350214+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350214+0000","last_undegraded":"2024-03-06T07:25:37.350214+0000","last_fullsized":"2024-03-06T07:25:37.350214+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351435+0000","last_change":"2024-03-06T07:25:37.351435+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351435+0000","last_undegraded":"2024-03-06T07:25:37.351435+0000","last_fullsized":"2024-03-06T07:25:37.351435+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350699+0000","last_change":"2024-03-06T07:25:37.350699+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350699+0000","last_undegraded":"2024-03-06T07:25:37.350699+0000","last_fullsized":"2024-03-06T07:25:37.350699+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354123+0000","last_change":"2024-03-06T07:25:37.354123+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354123+0000","last_undegraded":"2024-03-06T07:25:37.354123+0000","last_fullsized":"2024-03-06T07:25:37.354123+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354755+0000","last_change":"2024-03-06T07:25:37.354755+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354755+0000","last_undegraded":"2024-03-06T07:25:37.354755+0000","last_fullsized":"2024-03-06T07:25:37.354755+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350943+0000","last_change":"2024-03-06T07:25:37.350943+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350943+0000","last_undegraded":"2024-03-06T07:25:37.350943+0000","last_fullsized":"2024-03-06T07:25:37.350943+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353972+0000","last_change":"2024-03-06T07:25:37.353972+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353972+0000","last_undegraded":"2024-03-06T07:25:37.353972+0000","last_fullsized":"2024-03-06T07:25:37.353972+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349753+0000","last_change":"2024-03-06T07:25:37.349753+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349753+0000","last_undegraded":"2024-03-06T07:25:37.349753+0000","last_fullsized":"2024-03-06T07:25:37.349753+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349137+0000","last_change":"2024-03-06T07:25:37.349137+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349137+0000","last_undegraded":"2024-03-06T07:25:37.349137+0000","last_fullsized":"2024-03-06T07:25:37.349137+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355131+0000","last_change":"2024-03-06T07:25:37.355131+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355131+0000","last_undegraded":"2024-03-06T07:25:37.355131+0000","last_fullsized":"2024-03-06T07:25:37.355131+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353675+0000","last_change":"2024-03-06T07:25:37.353675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353675+0000","last_undegraded":"2024-03-06T07:25:37.353675+0000","last_fullsized":"2024-03-06T07:25:37.353675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348241+0000","last_change":"2024-03-06T07:25:37.348241+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348241+0000","last_undegraded":"2024-03-06T07:25:37.348241+0000","last_fullsized":"2024-03-06T07:25:37.348241+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347813+0000","last_change":"2024-03-06T07:25:37.347813+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347813+0000","last_undegraded":"2024-03-06T07:25:37.347813+0000","last_fullsized":"2024-03-06T07:25:37.347813+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351675+0000","last_change":"2024-03-06T07:25:37.351675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351675+0000","last_undegraded":"2024-03-06T07:25:37.351675+0000","last_fullsized":"2024-03-06T07:25:37.351675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350203+0000","last_change":"2024-03-06T07:25:37.350203+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350203+0000","last_undegraded":"2024-03-06T07:25:37.350203+0000","last_fullsized":"2024-03-06T07:25:37.350203+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348621+0000","last_change":"2024-03-06T07:25:37.348621+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348621+0000","last_undegraded":"2024-03-06T07:25:37.348621+0000","last_fullsized":"2024-03-06T07:25:37.348621+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353167+0000","last_change":"2024-03-06T07:25:37.353167+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353167+0000","last_undegraded":"2024-03-06T07:25:37.353167+0000","last_fullsized":"2024-03-06T07:25:37.353167+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361035+0000","last_change":"2024-03-06T07:25:37.360161+0000","last_active":"2024-03-06T07:25:37.361035+0000","last_peered":"2024-03-06T07:25:37.361035+0000","last_clean":"2024-03-06T07:25:37.361035+0000","last_became_active":"2024-03-06T07:25:37.359858+0000","last_became_peered":"2024-03-06T07:25:37.359858+0000","last_unstale":"2024-03-06T07:25:37.361035+0000","last_undegraded":"2024-03-06T07:25:37.361035+0000","last_fullsized":"2024-03-06T07:25:37.361035+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:11.304824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353595+0000","last_change":"2024-03-06T07:25:37.353595+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353595+0000","last_undegraded":"2024-03-06T07:25:37.353595+0000","last_fullsized":"2024-03-06T07:25:37.353595+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.366147+0000","last_change":"2024-03-06T07:25:37.361735+0000","last_active":"2024-03-06T07:25:37.366147+0000","last_peered":"2024-03-06T07:25:37.366147+0000","last_clean":"2024-03-06T07:25:37.366147+0000","last_became_active":"2024-03-06T07:25:37.361071+0000","last_became_peered":"2024-03-06T07:25:37.361071+0000","last_unstale":"2024-03-06T07:25:37.366147+0000","last_undegraded":"2024-03-06T07:25:37.366147+0000","last_fullsized":"2024-03-06T07:25:37.366147+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:38.535183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353456+0000","last_change":"2024-03-06T07:25:37.353456+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353456+0000","last_undegraded":"2024-03-06T07:25:37.353456+0000","last_fullsized":"2024-03-06T07:25:37.353456+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358849+0000","last_change":"2024-03-06T07:25:37.358208+0000","last_active":"2024-03-06T07:25:37.358849+0000","last_peered":"2024-03-06T07:25:37.358849+0000","last_clean":"2024-03-06T07:25:37.358849+0000","last_became_active":"2024-03-06T07:25:37.357941+0000","last_became_peered":"2024-03-06T07:25:37.357941+0000","last_unstale":"2024-03-06T07:25:37.358849+0000","last_undegraded":"2024-03-06T07:25:37.358849+0000","last_fullsized":"2024-03-06T07:25:37.358849+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:04.977160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353298+0000","last_change":"2024-03-06T07:25:37.353298+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353298+0000","last_undegraded":"2024-03-06T07:25:37.353298+0000","last_fullsized":"2024-03-06T07:25:37.353298+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358129+0000","last_change":"2024-03-06T07:25:37.357145+0000","last_active":"2024-03-06T07:25:37.358129+0000","last_peered":"2024-03-06T07:25:37.358129+0000","last_clean":"2024-03-06T07:25:37.358129+0000","last_became_active":"2024-03-06T07:25:37.356819+0000","last_became_peered":"2024-03-06T07:25:37.356819+0000","last_unstale":"2024-03-06T07:25:37.358129+0000","last_undegraded":"2024-03-06T07:25:37.358129+0000","last_fullsized":"2024-03-06T07:25:37.358129+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.694205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350553+0000","last_change":"2024-03-06T07:25:37.350553+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350553+0000","last_undegraded":"2024-03-06T07:25:37.350553+0000","last_fullsized":"2024-03-06T07:25:37.350553+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368484+0000","last_change":"2024-03-06T07:25:37.366475+0000","last_active":"2024-03-06T07:25:37.368484+0000","last_peered":"2024-03-06T07:25:37.368484+0000","last_clean":"2024-03-06T07:25:37.368484+0000","last_became_active":"2024-03-06T07:25:37.366203+0000","last_became_peered":"2024-03-06T07:25:37.366203+0000","last_unstale":"2024-03-06T07:25:37.368484+0000","last_undegraded":"2024-03-06T07:25:37.368484+0000","last_fullsized":"2024-03-06T07:25:37.368484+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:33.314274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349256+0000","last_change":"2024-03-06T07:25:37.349256+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349256+0000","last_undegraded":"2024-03-06T07:25:37.349256+0000","last_fullsized":"2024-03-06T07:25:37.349256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364042+0000","last_change":"2024-03-06T07:25:37.362493+0000","last_active":"2024-03-06T07:25:37.364042+0000","last_peered":"2024-03-06T07:25:37.364042+0000","last_clean":"2024-03-06T07:25:37.364042+0000","last_became_active":"2024-03-06T07:25:37.362019+0000","last_became_peered":"2024-03-06T07:25:37.362019+0000","last_unstale":"2024-03-06T07:25:37.364042+0000","last_undegraded":"2024-03-06T07:25:37.364042+0000","last_fullsized":"2024-03-06T07:25:37.364042+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348628+0000","last_change":"2024-03-06T07:25:37.348628+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348628+0000","last_undegraded":"2024-03-06T07:25:37.348628+0000","last_fullsized":"2024-03-06T07:25:37.348628+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360769+0000","last_change":"2024-03-06T07:25:37.359237+0000","last_active":"2024-03-06T07:25:37.360769+0000","last_peered":"2024-03-06T07:25:37.360769+0000","last_clean":"2024-03-06T07:25:37.360769+0000","last_became_active":"2024-03-06T07:25:37.358927+0000","last_became_peered":"2024-03-06T07:25:37.358927+0000","last_unstale":"2024-03-06T07:25:37.360769+0000","last_undegraded":"2024-03-06T07:25:37.360769+0000","last_fullsized":"2024-03-06T07:25:37.360769+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:08.259298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353004+0000","last_change":"2024-03-06T07:25:37.353004+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353004+0000","last_undegraded":"2024-03-06T07:25:37.353004+0000","last_fullsized":"2024-03-06T07:25:37.353004+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368462+0000","last_change":"2024-03-06T07:25:37.367257+0000","last_active":"2024-03-06T07:25:37.368462+0000","last_peered":"2024-03-06T07:25:37.368462+0000","last_clean":"2024-03-06T07:25:37.368462+0000","last_became_active":"2024-03-06T07:25:37.366971+0000","last_became_peered":"2024-03-06T07:25:37.366971+0000","last_unstale":"2024-03-06T07:25:37.368462+0000","last_undegraded":"2024-03-06T07:25:37.368462+0000","last_fullsized":"2024-03-06T07:25:37.368462+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.083348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355752+0000","last_change":"2024-03-06T07:25:37.355752+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355752+0000","last_undegraded":"2024-03-06T07:25:37.355752+0000","last_fullsized":"2024-03-06T07:25:37.355752+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364834+0000","last_change":"2024-03-06T07:25:37.363334+0000","last_active":"2024-03-06T07:25:37.364834+0000","last_peered":"2024-03-06T07:25:37.364834+0000","last_clean":"2024-03-06T07:25:37.364834+0000","last_became_active":"2024-03-06T07:25:37.362631+0000","last_became_peered":"2024-03-06T07:25:37.362631+0000","last_unstale":"2024-03-06T07:25:37.364834+0000","last_undegraded":"2024-03-06T07:25:37.364834+0000","last_fullsized":"2024-03-06T07:25:37.364834+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352531+0000","last_change":"2024-03-06T07:25:37.352531+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352531+0000","last_undegraded":"2024-03-06T07:25:37.352531+0000","last_fullsized":"2024-03-06T07:25:37.352531+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368576+0000","last_change":"2024-03-06T07:25:37.367405+0000","last_active":"2024-03-06T07:25:37.368576+0000","last_peered":"2024-03-06T07:25:37.368576+0000","last_clean":"2024-03-06T07:25:37.368576+0000","last_became_active":"2024-03-06T07:25:37.367077+0000","last_became_peered":"2024-03-06T07:25:37.367077+0000","last_unstale":"2024-03-06T07:25:37.368576+0000","last_undegraded":"2024-03-06T07:25:37.368576+0000","last_fullsized":"2024-03-06T07:25:37.368576+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348006+0000","last_change":"2024-03-06T07:25:37.348006+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348006+0000","last_undegraded":"2024-03-06T07:25:37.348006+0000","last_fullsized":"2024-03-06T07:25:37.348006+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360396+0000","last_change":"2024-03-06T07:25:37.359803+0000","last_active":"2024-03-06T07:25:37.360396+0000","last_peered":"2024-03-06T07:25:37.360396+0000","last_clean":"2024-03-06T07:25:37.360396+0000","last_became_active":"2024-03-06T07:25:37.359509+0000","last_became_peered":"2024-03-06T07:25:37.359509+0000","last_unstale":"2024-03-06T07:25:37.360396+0000","last_undegraded":"2024-03-06T07:25:37.360396+0000","last_fullsized":"2024-03-06T07:25:37.360396+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:16.038727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368085+0000","last_change":"2024-03-06T07:25:37.363650+0000","last_active":"2024-03-06T07:25:37.368085+0000","last_peered":"2024-03-06T07:25:37.368085+0000","last_clean":"2024-03-06T07:25:37.368085+0000","last_became_active":"2024-03-06T07:25:37.363319+0000","last_became_peered":"2024-03-06T07:25:37.363319+0000","last_unstale":"2024-03-06T07:25:37.368085+0000","last_undegraded":"2024-03-06T07:25:37.368085+0000","last_fullsized":"2024-03-06T07:25:37.368085+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:03.107757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347979+0000","last_change":"2024-03-06T07:25:37.347979+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347979+0000","last_undegraded":"2024-03-06T07:25:37.347979+0000","last_fullsized":"2024-03-06T07:25:37.347979+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359733+0000","last_change":"2024-03-06T07:25:37.357207+0000","last_active":"2024-03-06T07:25:37.359733+0000","last_peered":"2024-03-06T07:25:37.359733+0000","last_clean":"2024-03-06T07:25:37.359733+0000","last_became_active":"2024-03-06T07:25:37.356920+0000","last_became_peered":"2024-03-06T07:25:37.356920+0000","last_unstale":"2024-03-06T07:25:37.359733+0000","last_undegraded":"2024-03-06T07:25:37.359733+0000","last_fullsized":"2024-03-06T07:25:37.359733+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351780+0000","last_change":"2024-03-06T07:25:37.351780+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351780+0000","last_undegraded":"2024-03-06T07:25:37.351780+0000","last_fullsized":"2024-03-06T07:25:37.351780+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361242+0000","last_change":"2024-03-06T07:25:37.360307+0000","last_active":"2024-03-06T07:25:37.361242+0000","last_peered":"2024-03-06T07:25:37.361242+0000","last_clean":"2024-03-06T07:25:37.361242+0000","last_became_active":"2024-03-06T07:25:37.360024+0000","last_became_peered":"2024-03-06T07:25:37.360024+0000","last_unstale":"2024-03-06T07:25:37.361242+0000","last_undegraded":"2024-03-06T07:25:37.361242+0000","last_fullsized":"2024-03-06T07:25:37.361242+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:13.020947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349007+0000","last_change":"2024-03-06T07:25:37.349007+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349007+0000","last_undegraded":"2024-03-06T07:25:37.349007+0000","last_fullsized":"2024-03-06T07:25:37.349007+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368184+0000","last_change":"2024-03-06T07:25:37.364965+0000","last_active":"2024-03-06T07:25:37.368184+0000","last_peered":"2024-03-06T07:25:37.368184+0000","last_clean":"2024-03-06T07:25:37.368184+0000","last_became_active":"2024-03-06T07:25:37.363889+0000","last_became_peered":"2024-03-06T07:25:37.363889+0000","last_unstale":"2024-03-06T07:25:37.368184+0000","last_undegraded":"2024-03-06T07:25:37.368184+0000","last_fullsized":"2024-03-06T07:25:37.368184+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352125+0000","last_change":"2024-03-06T07:25:37.352125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352125+0000","last_undegraded":"2024-03-06T07:25:37.352125+0000","last_fullsized":"2024-03-06T07:25:37.352125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364603+0000","last_change":"2024-03-06T07:25:37.364101+0000","last_active":"2024-03-06T07:25:37.364603+0000","last_peered":"2024-03-06T07:25:37.364603+0000","last_clean":"2024-03-06T07:25:37.364603+0000","last_became_active":"2024-03-06T07:25:37.363814+0000","last_became_peered":"2024-03-06T07:25:37.363814+0000","last_unstale":"2024-03-06T07:25:37.364603+0000","last_undegraded":"2024-03-06T07:25:37.364603+0000","last_fullsized":"2024-03-06T07:25:37.364603+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:08.141356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.356323+0000","last_change":"2024-03-06T07:25:37.356323+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.356323+0000","last_undegraded":"2024-03-06T07:25:37.356323+0000","last_fullsized":"2024-03-06T07:25:37.356323+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359709+0000","last_change":"2024-03-06T07:25:37.358403+0000","last_active":"2024-03-06T07:25:37.359709+0000","last_peered":"2024-03-06T07:25:37.359709+0000","last_clean":"2024-03-06T07:25:37.359709+0000","last_became_active":"2024-03-06T07:25:37.358134+0000","last_became_peered":"2024-03-06T07:25:37.358134+0000","last_unstale":"2024-03-06T07:25:37.359709+0000","last_undegraded":"2024-03-06T07:25:37.359709+0000","last_fullsized":"2024-03-06T07:25:37.359709+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.829122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352078+0000","last_change":"2024-03-06T07:25:37.352078+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352078+0000","last_undegraded":"2024-03-06T07:25:37.352078+0000","last_fullsized":"2024-03-06T07:25:37.352078+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358546+0000","last_change":"2024-03-06T07:25:37.355316+0000","last_active":"2024-03-06T07:25:37.358546+0000","last_peered":"2024-03-06T07:25:37.358546+0000","last_clean":"2024-03-06T07:25:37.358546+0000","last_became_active":"2024-03-06T07:25:37.355060+0000","last_became_peered":"2024-03-06T07:25:37.355060+0000","last_unstale":"2024-03-06T07:25:37.358546+0000","last_undegraded":"2024-03-06T07:25:37.358546+0000","last_fullsized":"2024-03-06T07:25:37.358546+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:27.708230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350295+0000","last_change":"2024-03-06T07:25:37.350295+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350295+0000","last_undegraded":"2024-03-06T07:25:37.350295+0000","last_fullsized":"2024-03-06T07:25:37.350295+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.357802+0000","last_change":"2024-03-06T07:25:37.354804+0000","last_active":"2024-03-06T07:25:37.357802+0000","last_peered":"2024-03-06T07:25:37.357802+0000","last_clean":"2024-03-06T07:25:37.357802+0000","last_became_active":"2024-03-06T07:25:37.354535+0000","last_became_peered":"2024-03-06T07:25:37.354535+0000","last_unstale":"2024-03-06T07:25:37.357802+0000","last_undegraded":"2024-03-06T07:25:37.357802+0000","last_fullsized":"2024-03-06T07:25:37.357802+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:11.747252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351130+0000","last_change":"2024-03-06T07:25:37.351130+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351130+0000","last_undegraded":"2024-03-06T07:25:37.351130+0000","last_fullsized":"2024-03-06T07:25:37.351130+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.367951+0000","last_change":"2024-03-06T07:25:37.362425+0000","last_active":"2024-03-06T07:25:37.367951+0000","last_peered":"2024-03-06T07:25:37.367951+0000","last_clean":"2024-03-06T07:25:37.367951+0000","last_became_active":"2024-03-06T07:25:37.362172+0000","last_became_peered":"2024-03-06T07:25:37.362172+0000","last_unstale":"2024-03-06T07:25:37.367951+0000","last_undegraded":"2024-03-06T07:25:37.367951+0000","last_fullsized":"2024-03-06T07:25:37.367951+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:46.062578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350611+0000","last_change":"2024-03-06T07:25:37.350611+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350611+0000","last_undegraded":"2024-03-06T07:25:37.350611+0000","last_fullsized":"2024-03-06T07:25:37.350611+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356209+0000","last_change":"2024-03-06T07:25:37.354863+0000","last_active":"2024-03-06T07:25:37.356209+0000","last_peered":"2024-03-06T07:25:37.356209+0000","last_clean":"2024-03-06T07:25:37.356209+0000","last_became_active":"2024-03-06T07:25:37.354465+0000","last_became_peered":"2024-03-06T07:25:37.354465+0000","last_unstale":"2024-03-06T07:25:37.356209+0000","last_undegraded":"2024-03-06T07:25:37.356209+0000","last_fullsized":"2024-03-06T07:25:37.356209+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:33.951539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349868+0000","last_change":"2024-03-06T07:25:37.349868+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349868+0000","last_undegraded":"2024-03-06T07:25:37.349868+0000","last_fullsized":"2024-03-06T07:25:37.349868+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362855+0000","last_change":"2024-03-06T07:25:37.361572+0000","last_active":"2024-03-06T07:25:37.362855+0000","last_peered":"2024-03-06T07:25:37.362855+0000","last_clean":"2024-03-06T07:25:37.362855+0000","last_became_active":"2024-03-06T07:25:37.361271+0000","last_became_peered":"2024-03-06T07:25:37.361271+0000","last_unstale":"2024-03-06T07:25:37.362855+0000","last_undegraded":"2024-03-06T07:25:37.362855+0000","last_fullsized":"2024-03-06T07:25:37.362855+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:13.032326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351680+0000","last_change":"2024-03-06T07:25:37.351680+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351680+0000","last_undegraded":"2024-03-06T07:25:37.351680+0000","last_fullsized":"2024-03-06T07:25:37.351680+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360711+0000","last_change":"2024-03-06T07:25:37.359314+0000","last_active":"2024-03-06T07:25:37.360711+0000","last_peered":"2024-03-06T07:25:37.360711+0000","last_clean":"2024-03-06T07:25:37.360711+0000","last_became_active":"2024-03-06T07:25:37.359008+0000","last_became_peered":"2024-03-06T07:25:37.359008+0000","last_unstale":"2024-03-06T07:25:37.360711+0000","last_undegraded":"2024-03-06T07:25:37.360711+0000","last_fullsized":"2024-03-06T07:25:37.360711+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:07.137677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351711+0000","last_change":"2024-03-06T07:25:37.351711+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351711+0000","last_undegraded":"2024-03-06T07:25:37.351711+0000","last_fullsized":"2024-03-06T07:25:37.351711+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368155+0000","last_change":"2024-03-06T07:25:37.364253+0000","last_active":"2024-03-06T07:25:37.368155+0000","last_peered":"2024-03-06T07:25:37.368155+0000","last_clean":"2024-03-06T07:25:37.368155+0000","last_became_active":"2024-03-06T07:25:37.363933+0000","last_became_peered":"2024-03-06T07:25:37.363933+0000","last_unstale":"2024-03-06T07:25:37.368155+0000","last_undegraded":"2024-03-06T07:25:37.368155+0000","last_fullsized":"2024-03-06T07:25:37.368155+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355920+0000","last_change":"2024-03-06T07:25:37.355920+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355920+0000","last_undegraded":"2024-03-06T07:25:37.355920+0000","last_fullsized":"2024-03-06T07:25:37.355920+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364944+0000","last_change":"2024-03-06T07:25:37.364443+0000","last_active":"2024-03-06T07:25:37.364944+0000","last_peered":"2024-03-06T07:25:37.364944+0000","last_clean":"2024-03-06T07:25:37.364944+0000","last_became_active":"2024-03-06T07:25:37.364093+0000","last_became_peered":"2024-03-06T07:25:37.364093+0000","last_unstale":"2024-03-06T07:25:37.364944+0000","last_undegraded":"2024-03-06T07:25:37.364944+0000","last_fullsized":"2024-03-06T07:25:37.364944+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:01.483203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351304+0000","last_change":"2024-03-06T07:25:37.351304+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351304+0000","last_undegraded":"2024-03-06T07:25:37.351304+0000","last_fullsized":"2024-03-06T07:25:37.351304+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.363191+0000","last_change":"2024-03-06T07:25:37.362488+0000","last_active":"2024-03-06T07:25:37.363191+0000","last_peered":"2024-03-06T07:25:37.363191+0000","last_clean":"2024-03-06T07:25:37.363191+0000","last_became_active":"2024-03-06T07:25:37.362203+0000","last_became_peered":"2024-03-06T07:25:37.362203+0000","last_unstale":"2024-03-06T07:25:37.363191+0000","last_undegraded":"2024-03-06T07:25:37.363191+0000","last_fullsized":"2024-03-06T07:25:37.363191+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.309117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348099+0000","last_change":"2024-03-06T07:25:37.348099+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348099+0000","last_undegraded":"2024-03-06T07:25:37.348099+0000","last_fullsized":"2024-03-06T07:25:37.348099+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368386+0000","last_change":"2024-03-06T07:25:37.366204+0000","last_active":"2024-03-06T07:25:37.368386+0000","last_peered":"2024-03-06T07:25:37.368386+0000","last_clean":"2024-03-06T07:25:37.368386+0000","last_became_active":"2024-03-06T07:25:37.365751+0000","last_became_peered":"2024-03-06T07:25:37.365751+0000","last_unstale":"2024-03-06T07:25:37.368386+0000","last_undegraded":"2024-03-06T07:25:37.368386+0000","last_fullsized":"2024-03-06T07:25:37.368386+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350714+0000","last_change":"2024-03-06T07:25:37.350714+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350714+0000","last_undegraded":"2024-03-06T07:25:37.350714+0000","last_fullsized":"2024-03-06T07:25:37.350714+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368066+0000","last_change":"2024-03-06T07:25:37.363581+0000","last_active":"2024-03-06T07:25:37.368066+0000","last_peered":"2024-03-06T07:25:37.368066+0000","last_clean":"2024-03-06T07:25:37.368066+0000","last_became_active":"2024-03-06T07:25:37.362991+0000","last_became_peered":"2024-03-06T07:25:37.362991+0000","last_unstale":"2024-03-06T07:25:37.368066+0000","last_undegraded":"2024-03-06T07:25:37.368066+0000","last_fullsized":"2024-03-06T07:25:37.368066+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:25.392712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351198+0000","last_change":"2024-03-06T07:25:37.351198+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351198+0000","last_undegraded":"2024-03-06T07:25:37.351198+0000","last_fullsized":"2024-03-06T07:25:37.351198+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362479+0000","last_change":"2024-03-06T07:25:37.361352+0000","last_active":"2024-03-06T07:25:37.362479+0000","last_peered":"2024-03-06T07:25:37.362479+0000","last_clean":"2024-03-06T07:25:37.362479+0000","last_became_active":"2024-03-06T07:25:37.361059+0000","last_became_peered":"2024-03-06T07:25:37.361059+0000","last_unstale":"2024-03-06T07:25:37.362479+0000","last_undegraded":"2024-03-06T07:25:37.362479+0000","last_fullsized":"2024-03-06T07:25:37.362479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348043+0000","last_change":"2024-03-06T07:25:37.348043+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348043+0000","last_undegraded":"2024-03-06T07:25:37.348043+0000","last_fullsized":"2024-03-06T07:25:37.348043+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368916+0000","last_change":"2024-03-06T07:25:37.368239+0000","last_active":"2024-03-06T07:25:37.368916+0000","last_peered":"2024-03-06T07:25:37.368916+0000","last_clean":"2024-03-06T07:25:37.368916+0000","last_became_active":"2024-03-06T07:25:37.367979+0000","last_became_peered":"2024-03-06T07:25:37.367979+0000","last_unstale":"2024-03-06T07:25:37.368916+0000","last_undegraded":"2024-03-06T07:25:37.368916+0000","last_fullsized":"2024-03-06T07:25:37.368916+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:03.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349897+0000","last_change":"2024-03-06T07:25:37.349897+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349897+0000","last_undegraded":"2024-03-06T07:25:37.349897+0000","last_fullsized":"2024-03-06T07:25:37.349897+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368412+0000","last_change":"2024-03-06T07:25:37.366881+0000","last_active":"2024-03-06T07:25:37.368412+0000","last_peered":"2024-03-06T07:25:37.368412+0000","last_clean":"2024-03-06T07:25:37.368412+0000","last_became_active":"2024-03-06T07:25:37.366588+0000","last_became_peered":"2024-03-06T07:25:37.366588+0000","last_unstale":"2024-03-06T07:25:37.368412+0000","last_undegraded":"2024-03-06T07:25:37.368412+0000","last_fullsized":"2024-03-06T07:25:37.368412+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:46.200362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354567+0000","last_change":"2024-03-06T07:25:37.354567+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354567+0000","last_undegraded":"2024-03-06T07:25:37.354567+0000","last_fullsized":"2024-03-06T07:25:37.354567+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364364+0000","last_change":"2024-03-06T07:25:37.359284+0000","last_active":"2024-03-06T07:25:37.364364+0000","last_peered":"2024-03-06T07:25:37.364364+0000","last_clean":"2024-03-06T07:25:37.364364+0000","last_became_active":"2024-03-06T07:25:37.358847+0000","last_became_peered":"2024-03-06T07:25:37.358847+0000","last_unstale":"2024-03-06T07:25:37.364364+0000","last_undegraded":"2024-03-06T07:25:37.364364+0000","last_fullsized":"2024-03-06T07:25:37.364364+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.913135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":472,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356621+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:25:37.356621+0000","last_peered":"2024-03-06T07:25:37.356621+0000","last_clean":"2024-03-06T07:25:37.356621+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:25:37.356621+0000","last_undegraded":"2024-03-06T07:25:37.356621+0000","last_fullsized":"2024-03-06T07:25:37.356621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33412,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338428,"statfs":{"total":96636764160,"available":96602550272,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37624,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334216,"statfs":{"total":96636764160,"available":96598237184,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:41.680 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.815+0000 7fa9eb439640 1 Processor -- start 2024-03-06T07:25:41.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.815+0000 7fa9eb439640 1 -- start start 2024-03-06T07:25:41.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9eb439640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9e40c1aa0 0x7fa9e41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9eb439640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa9e4155a40 0x7fa9e4155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9eb439640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e40c28d0 con 0x7fa9e414f790 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9eb439640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e40c2680 con 0x7fa9e40c1aa0 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9eb439640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e40c2bb0 con 0x7fa9e4155a40 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e89d4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9e40c1aa0 0x7fa9e41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e91d5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa9e414f790 0x7fa9e40c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58466/0 (socket says 172.21.15.94:58466) 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e91d5640 1 -- 172.21.15.94:0/2232898086 learned_addr learned my addr 172.21.15.94:0/2232898086 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e99d6640 1 --2- 172.21.15.94:0/2232898086 >> v2:172.21.15.145:3300/0 conn(0x7fa9e4155a40 0x7fa9e4155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e89d4640 1 -- 172.21.15.94:0/2232898086 >> v2:172.21.15.145:3300/0 conn(0x7fa9e4155a40 msgr2=0x7fa9e4155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e89d4640 1 --2- 172.21.15.94:0/2232898086 >> v2:172.21.15.145:3300/0 conn(0x7fa9e4155a40 0x7fa9e4155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e89d4640 1 -- 172.21.15.94:0/2232898086 >> v1:172.21.15.94:6789/0 conn(0x7fa9e414f790 legacy=0x7fa9e40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9dbfff640 1 -- 172.21.15.94:0/2232898086 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681762548 0 0) 0x7fa9e40c28d0 con 0x7fa9e414f790 2024-03-06T07:25:41.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.816+0000 7fa9e89d4640 1 -- 172.21.15.94:0/2232898086 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e40c2e90 con 0x7fa9e40c1aa0 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.817+0000 7fa9e89d4640 1 --2- 172.21.15.94:0/2232898086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e40c1aa0 0x7fa9e41529d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fa9e406c960 tx=0x7fa9d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33c24e936493f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.817+0000 7fa9dbfff640 1 -- 172.21.15.94:0/2232898086 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4035020 con 0x7fa9e40c1aa0 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.817+0000 7fa9dbfff640 1 -- 172.21.15.94:0/2232898086 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4031830 con 0x7fa9e40c1aa0 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.817+0000 7fa9dbfff640 1 -- 172.21.15.94:0/2232898086 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4031c00 con 0x7fa9e40c1aa0 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.817+0000 7fa9eb439640 1 -- 172.21.15.94:0/2232898086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e40c1aa0 msgr2=0x7fa9e41529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.817+0000 7fa9eb439640 1 --2- 172.21.15.94:0/2232898086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e40c1aa0 0x7fa9e41529d0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fa9e406c960 tx=0x7fa9d402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.818+0000 7fa9eb439640 1 -- 172.21.15.94:0/2232898086 shutdown_connections 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.818+0000 7fa9eb439640 1 --2- 172.21.15.94:0/2232898086 >> v2:172.21.15.145:3300/0 conn(0x7fa9e4155a40 0x7fa9e4155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.818+0000 7fa9eb439640 1 --2- 172.21.15.94:0/2232898086 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e40c1aa0 0x7fa9e41529d0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.818+0000 7fa9eb439640 1 -- 172.21.15.94:0/2232898086 >> 172.21.15.94:0/2232898086 conn(0x7fa9e40c6710 msgr2=0x7fa9e40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.818+0000 7fa9eb439640 1 -- 172.21.15.94:0/2232898086 shutdown_connections 2024-03-06T07:25:41.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.818+0000 7fa9eb439640 1 -- 172.21.15.94:0/2232898086 wait complete. 2024-03-06T07:25:41.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 Processor -- start 2024-03-06T07:25:41.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 -- start start 2024-03-06T07:25:41.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9e40c1aa0 0x7fa9e4148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9e414f790 0x7fa9e406d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e4155a40 0x7fa9e40a6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e40c0d60 con 0x7fa9e4155a40 2024-03-06T07:25:41.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4156390 con 0x7fa9e40c1aa0 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9eb439640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e40c5e30 con 0x7fa9e414f790 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9e99d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e4155a40 0x7fa9e40a6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9e99d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e4155a40 0x7fa9e40a6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41864/0 (socket says 172.21.15.94:41864) 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9e99d6640 1 -- 172.21.15.94:0/774805208 learned_addr learned my addr 172.21.15.94:0/774805208 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.819+0000 7fa9e89d4640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9e414f790 0x7fa9e406d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9e91d5640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9e40c1aa0 0x7fa9e4148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9e99d6640 1 -- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9e414f790 msgr2=0x7fa9e406d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9e99d6640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9e414f790 0x7fa9e406d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9e99d6640 1 -- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9e40c1aa0 msgr2=0x7fa9e4148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9e99d6640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9e40c1aa0 0x7fa9e4148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9e99d6640 1 -- 172.21.15.94:0/774805208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e41534c0 con 0x7fa9e4155a40 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9e99d6640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e4155a40 0x7fa9e40a6ba0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00efc0 tx=0x7fa9dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9d9ffb640 1 -- 172.21.15.94:0/774805208 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc018020 con 0x7fa9e4155a40 2024-03-06T07:25:41.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9d9ffb640 1 -- 172.21.15.94:0/774805208 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc014480 con 0x7fa9e4155a40 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e41537a0 con 0x7fa9e4155a40 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9d9ffb640 1 -- 172.21.15.94:0/774805208 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc014870 con 0x7fa9e4155a40 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.820+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e406e060 con 0x7fa9e4155a40 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.821+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e404f330 con 0x7fa9e4155a40 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.822+0000 7fa9d9ffb640 1 -- 172.21.15.94:0/774805208 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9dc005d10 con 0x7fa9e4155a40 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.822+0000 7fa9d9ffb640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa9c806b500 0x7fa9c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.822+0000 7fa9d9ffb640 1 -- 172.21.15.94:0/774805208 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa9dc08abd0 con 0x7fa9e4155a40 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.822+0000 7fa9e91d5640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa9c806b500 0x7fa9c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.823+0000 7fa9e91d5640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa9c806b500 0x7fa9c806d9c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0000f60 tx=0x7fa9e0007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.826+0000 7fa9d9ffb640 1 -- 172.21.15.94:0/774805208 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9dc022070 con 0x7fa9e4155a40 2024-03-06T07:25:41.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:41.994+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9e40a5f60 con 0x7fa9c806b500 2024-03-06T07:25:42.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.001+0000 7fa9d9ffb640 1 -- 172.21.15.94:0/774805208 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7fa9e40a5f60 con 0x7fa9c806b500 2024-03-06T07:25:42.005 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:42.006 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:25:42.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa9c806b500 msgr2=0x7fa9c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa9c806b500 0x7fa9c806d9c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0000f60 tx=0x7fa9e0007d50 comp rx=0 tx=0).stop 2024-03-06T07:25:42.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e4155a40 msgr2=0x7fa9e40a6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e4155a40 0x7fa9e40a6ba0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00efc0 tx=0x7fa9dc00c490 comp rx=0 tx=0).stop 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 shutdown_connections 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa9c806b500 0x7fa9c806d9c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9e4155a40 0x7fa9e40a6ba0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9e414f790 0x7fa9e406d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 --2- 172.21.15.94:0/774805208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9e40c1aa0 0x7fa9e4148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 >> 172.21.15.94:0/774805208 conn(0x7fa9e40c6710 msgr2=0x7fa9e406cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 shutdown_connections 2024-03-06T07:25:42.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.003+0000 7fa9eb439640 1 -- 172.21.15.94:0/774805208 wait complete. 2024-03-06T07:25:42.016 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-03-06T07:25:40.732723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271508,"kb_used_data":17880,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754703212,"statfs":{"total":773094113280,"available":772816089088,"internally_reserved":0,"allocated":18309120,"data_stored":9988974,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.390499"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353125+0000","last_change":"2024-03-06T07:25:37.353125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353125+0000","last_undegraded":"2024-03-06T07:25:37.353125+0000","last_fullsized":"2024-03-06T07:25:37.353125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351988+0000","last_change":"2024-03-06T07:25:37.351988+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351988+0000","last_undegraded":"2024-03-06T07:25:37.351988+0000","last_fullsized":"2024-03-06T07:25:37.351988+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355468+0000","last_change":"2024-03-06T07:25:37.355468+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355468+0000","last_undegraded":"2024-03-06T07:25:37.355468+0000","last_fullsized":"2024-03-06T07:25:37.355468+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347174+0000","last_change":"2024-03-06T07:25:37.347174+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347174+0000","last_undegraded":"2024-03-06T07:25:37.347174+0000","last_fullsized":"2024-03-06T07:25:37.347174+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351436+0000","last_change":"2024-03-06T07:25:37.351436+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351436+0000","last_undegraded":"2024-03-06T07:25:37.351436+0000","last_fullsized":"2024-03-06T07:25:37.351436+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349668+0000","last_change":"2024-03-06T07:25:37.349668+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349668+0000","last_undegraded":"2024-03-06T07:25:37.349668+0000","last_fullsized":"2024-03-06T07:25:37.349668+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353857+0000","last_change":"2024-03-06T07:25:37.353857+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353857+0000","last_undegraded":"2024-03-06T07:25:37.353857+0000","last_fullsized":"2024-03-06T07:25:37.353857+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352760+0000","last_change":"2024-03-06T07:25:37.352760+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352760+0000","last_undegraded":"2024-03-06T07:25:37.352760+0000","last_fullsized":"2024-03-06T07:25:37.352760+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349674+0000","last_change":"2024-03-06T07:25:37.349674+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349674+0000","last_undegraded":"2024-03-06T07:25:37.349674+0000","last_fullsized":"2024-03-06T07:25:37.349674+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355049+0000","last_change":"2024-03-06T07:25:37.355049+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355049+0000","last_undegraded":"2024-03-06T07:25:37.355049+0000","last_fullsized":"2024-03-06T07:25:37.355049+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348776+0000","last_change":"2024-03-06T07:25:37.348776+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348776+0000","last_undegraded":"2024-03-06T07:25:37.348776+0000","last_fullsized":"2024-03-06T07:25:37.348776+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348316+0000","last_change":"2024-03-06T07:25:37.348316+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348316+0000","last_undegraded":"2024-03-06T07:25:37.348316+0000","last_fullsized":"2024-03-06T07:25:37.348316+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351809+0000","last_change":"2024-03-06T07:25:37.351809+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351809+0000","last_undegraded":"2024-03-06T07:25:37.351809+0000","last_fullsized":"2024-03-06T07:25:37.351809+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350214+0000","last_change":"2024-03-06T07:25:37.350214+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350214+0000","last_undegraded":"2024-03-06T07:25:37.350214+0000","last_fullsized":"2024-03-06T07:25:37.350214+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351435+0000","last_change":"2024-03-06T07:25:37.351435+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351435+0000","last_undegraded":"2024-03-06T07:25:37.351435+0000","last_fullsized":"2024-03-06T07:25:37.351435+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350699+0000","last_change":"2024-03-06T07:25:37.350699+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350699+0000","last_undegraded":"2024-03-06T07:25:37.350699+0000","last_fullsized":"2024-03-06T07:25:37.350699+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354123+0000","last_change":"2024-03-06T07:25:37.354123+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354123+0000","last_undegraded":"2024-03-06T07:25:37.354123+0000","last_fullsized":"2024-03-06T07:25:37.354123+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354755+0000","last_change":"2024-03-06T07:25:37.354755+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354755+0000","last_undegraded":"2024-03-06T07:25:37.354755+0000","last_fullsized":"2024-03-06T07:25:37.354755+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350943+0000","last_change":"2024-03-06T07:25:37.350943+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350943+0000","last_undegraded":"2024-03-06T07:25:37.350943+0000","last_fullsized":"2024-03-06T07:25:37.350943+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353972+0000","last_change":"2024-03-06T07:25:37.353972+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353972+0000","last_undegraded":"2024-03-06T07:25:37.353972+0000","last_fullsized":"2024-03-06T07:25:37.353972+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349753+0000","last_change":"2024-03-06T07:25:37.349753+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349753+0000","last_undegraded":"2024-03-06T07:25:37.349753+0000","last_fullsized":"2024-03-06T07:25:37.349753+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349137+0000","last_change":"2024-03-06T07:25:37.349137+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349137+0000","last_undegraded":"2024-03-06T07:25:37.349137+0000","last_fullsized":"2024-03-06T07:25:37.349137+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355131+0000","last_change":"2024-03-06T07:25:37.355131+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355131+0000","last_undegraded":"2024-03-06T07:25:37.355131+0000","last_fullsized":"2024-03-06T07:25:37.355131+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353675+0000","last_change":"2024-03-06T07:25:37.353675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353675+0000","last_undegraded":"2024-03-06T07:25:37.353675+0000","last_fullsized":"2024-03-06T07:25:37.353675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348241+0000","last_change":"2024-03-06T07:25:37.348241+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348241+0000","last_undegraded":"2024-03-06T07:25:37.348241+0000","last_fullsized":"2024-03-06T07:25:37.348241+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347813+0000","last_change":"2024-03-06T07:25:37.347813+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347813+0000","last_undegraded":"2024-03-06T07:25:37.347813+0000","last_fullsized":"2024-03-06T07:25:37.347813+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351675+0000","last_change":"2024-03-06T07:25:37.351675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351675+0000","last_undegraded":"2024-03-06T07:25:37.351675+0000","last_fullsized":"2024-03-06T07:25:37.351675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350203+0000","last_change":"2024-03-06T07:25:37.350203+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350203+0000","last_undegraded":"2024-03-06T07:25:37.350203+0000","last_fullsized":"2024-03-06T07:25:37.350203+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348621+0000","last_change":"2024-03-06T07:25:37.348621+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348621+0000","last_undegraded":"2024-03-06T07:25:37.348621+0000","last_fullsized":"2024-03-06T07:25:37.348621+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353167+0000","last_change":"2024-03-06T07:25:37.353167+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353167+0000","last_undegraded":"2024-03-06T07:25:37.353167+0000","last_fullsized":"2024-03-06T07:25:37.353167+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361035+0000","last_change":"2024-03-06T07:25:37.360161+0000","last_active":"2024-03-06T07:25:37.361035+0000","last_peered":"2024-03-06T07:25:37.361035+0000","last_clean":"2024-03-06T07:25:37.361035+0000","last_became_active":"2024-03-06T07:25:37.359858+0000","last_became_peered":"2024-03-06T07:25:37.359858+0000","last_unstale":"2024-03-06T07:25:37.361035+0000","last_undegraded":"2024-03-06T07:25:37.361035+0000","last_fullsized":"2024-03-06T07:25:37.361035+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:11.304824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353595+0000","last_change":"2024-03-06T07:25:37.353595+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353595+0000","last_undegraded":"2024-03-06T07:25:37.353595+0000","last_fullsized":"2024-03-06T07:25:37.353595+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.366147+0000","last_change":"2024-03-06T07:25:37.361735+0000","last_active":"2024-03-06T07:25:37.366147+0000","last_peered":"2024-03-06T07:25:37.366147+0000","last_clean":"2024-03-06T07:25:37.366147+0000","last_became_active":"2024-03-06T07:25:37.361071+0000","last_became_peered":"2024-03-06T07:25:37.361071+0000","last_unstale":"2024-03-06T07:25:37.366147+0000","last_undegraded":"2024-03-06T07:25:37.366147+0000","last_fullsized":"2024-03-06T07:25:37.366147+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:38.535183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353456+0000","last_change":"2024-03-06T07:25:37.353456+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353456+0000","last_undegraded":"2024-03-06T07:25:37.353456+0000","last_fullsized":"2024-03-06T07:25:37.353456+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358849+0000","last_change":"2024-03-06T07:25:37.358208+0000","last_active":"2024-03-06T07:25:37.358849+0000","last_peered":"2024-03-06T07:25:37.358849+0000","last_clean":"2024-03-06T07:25:37.358849+0000","last_became_active":"2024-03-06T07:25:37.357941+0000","last_became_peered":"2024-03-06T07:25:37.357941+0000","last_unstale":"2024-03-06T07:25:37.358849+0000","last_undegraded":"2024-03-06T07:25:37.358849+0000","last_fullsized":"2024-03-06T07:25:37.358849+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:04.977160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353298+0000","last_change":"2024-03-06T07:25:37.353298+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353298+0000","last_undegraded":"2024-03-06T07:25:37.353298+0000","last_fullsized":"2024-03-06T07:25:37.353298+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358129+0000","last_change":"2024-03-06T07:25:37.357145+0000","last_active":"2024-03-06T07:25:37.358129+0000","last_peered":"2024-03-06T07:25:37.358129+0000","last_clean":"2024-03-06T07:25:37.358129+0000","last_became_active":"2024-03-06T07:25:37.356819+0000","last_became_peered":"2024-03-06T07:25:37.356819+0000","last_unstale":"2024-03-06T07:25:37.358129+0000","last_undegraded":"2024-03-06T07:25:37.358129+0000","last_fullsized":"2024-03-06T07:25:37.358129+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.694205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350553+0000","last_change":"2024-03-06T07:25:37.350553+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350553+0000","last_undegraded":"2024-03-06T07:25:37.350553+0000","last_fullsized":"2024-03-06T07:25:37.350553+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368484+0000","last_change":"2024-03-06T07:25:37.366475+0000","last_active":"2024-03-06T07:25:37.368484+0000","last_peered":"2024-03-06T07:25:37.368484+0000","last_clean":"2024-03-06T07:25:37.368484+0000","last_became_active":"2024-03-06T07:25:37.366203+0000","last_became_peered":"2024-03-06T07:25:37.366203+0000","last_unstale":"2024-03-06T07:25:37.368484+0000","last_undegraded":"2024-03-06T07:25:37.368484+0000","last_fullsized":"2024-03-06T07:25:37.368484+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:33.314274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349256+0000","last_change":"2024-03-06T07:25:37.349256+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349256+0000","last_undegraded":"2024-03-06T07:25:37.349256+0000","last_fullsized":"2024-03-06T07:25:37.349256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364042+0000","last_change":"2024-03-06T07:25:37.362493+0000","last_active":"2024-03-06T07:25:37.364042+0000","last_peered":"2024-03-06T07:25:37.364042+0000","last_clean":"2024-03-06T07:25:37.364042+0000","last_became_active":"2024-03-06T07:25:37.362019+0000","last_became_peered":"2024-03-06T07:25:37.362019+0000","last_unstale":"2024-03-06T07:25:37.364042+0000","last_undegraded":"2024-03-06T07:25:37.364042+0000","last_fullsized":"2024-03-06T07:25:37.364042+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348628+0000","last_change":"2024-03-06T07:25:37.348628+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348628+0000","last_undegraded":"2024-03-06T07:25:37.348628+0000","last_fullsized":"2024-03-06T07:25:37.348628+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360769+0000","last_change":"2024-03-06T07:25:37.359237+0000","last_active":"2024-03-06T07:25:37.360769+0000","last_peered":"2024-03-06T07:25:37.360769+0000","last_clean":"2024-03-06T07:25:37.360769+0000","last_became_active":"2024-03-06T07:25:37.358927+0000","last_became_peered":"2024-03-06T07:25:37.358927+0000","last_unstale":"2024-03-06T07:25:37.360769+0000","last_undegraded":"2024-03-06T07:25:37.360769+0000","last_fullsized":"2024-03-06T07:25:37.360769+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:08.259298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353004+0000","last_change":"2024-03-06T07:25:37.353004+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353004+0000","last_undegraded":"2024-03-06T07:25:37.353004+0000","last_fullsized":"2024-03-06T07:25:37.353004+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368462+0000","last_change":"2024-03-06T07:25:37.367257+0000","last_active":"2024-03-06T07:25:37.368462+0000","last_peered":"2024-03-06T07:25:37.368462+0000","last_clean":"2024-03-06T07:25:37.368462+0000","last_became_active":"2024-03-06T07:25:37.366971+0000","last_became_peered":"2024-03-06T07:25:37.366971+0000","last_unstale":"2024-03-06T07:25:37.368462+0000","last_undegraded":"2024-03-06T07:25:37.368462+0000","last_fullsized":"2024-03-06T07:25:37.368462+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.083348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355752+0000","last_change":"2024-03-06T07:25:37.355752+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355752+0000","last_undegraded":"2024-03-06T07:25:37.355752+0000","last_fullsized":"2024-03-06T07:25:37.355752+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364834+0000","last_change":"2024-03-06T07:25:37.363334+0000","last_active":"2024-03-06T07:25:37.364834+0000","last_peered":"2024-03-06T07:25:37.364834+0000","last_clean":"2024-03-06T07:25:37.364834+0000","last_became_active":"2024-03-06T07:25:37.362631+0000","last_became_peered":"2024-03-06T07:25:37.362631+0000","last_unstale":"2024-03-06T07:25:37.364834+0000","last_undegraded":"2024-03-06T07:25:37.364834+0000","last_fullsized":"2024-03-06T07:25:37.364834+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352531+0000","last_change":"2024-03-06T07:25:37.352531+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352531+0000","last_undegraded":"2024-03-06T07:25:37.352531+0000","last_fullsized":"2024-03-06T07:25:37.352531+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368576+0000","last_change":"2024-03-06T07:25:37.367405+0000","last_active":"2024-03-06T07:25:37.368576+0000","last_peered":"2024-03-06T07:25:37.368576+0000","last_clean":"2024-03-06T07:25:37.368576+0000","last_became_active":"2024-03-06T07:25:37.367077+0000","last_became_peered":"2024-03-06T07:25:37.367077+0000","last_unstale":"2024-03-06T07:25:37.368576+0000","last_undegraded":"2024-03-06T07:25:37.368576+0000","last_fullsized":"2024-03-06T07:25:37.368576+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348006+0000","last_change":"2024-03-06T07:25:37.348006+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348006+0000","last_undegraded":"2024-03-06T07:25:37.348006+0000","last_fullsized":"2024-03-06T07:25:37.348006+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360396+0000","last_change":"2024-03-06T07:25:37.359803+0000","last_active":"2024-03-06T07:25:37.360396+0000","last_peered":"2024-03-06T07:25:37.360396+0000","last_clean":"2024-03-06T07:25:37.360396+0000","last_became_active":"2024-03-06T07:25:37.359509+0000","last_became_peered":"2024-03-06T07:25:37.359509+0000","last_unstale":"2024-03-06T07:25:37.360396+0000","last_undegraded":"2024-03-06T07:25:37.360396+0000","last_fullsized":"2024-03-06T07:25:37.360396+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:16.038727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368085+0000","last_change":"2024-03-06T07:25:37.363650+0000","last_active":"2024-03-06T07:25:37.368085+0000","last_peered":"2024-03-06T07:25:37.368085+0000","last_clean":"2024-03-06T07:25:37.368085+0000","last_became_active":"2024-03-06T07:25:37.363319+0000","last_became_peered":"2024-03-06T07:25:37.363319+0000","last_unstale":"2024-03-06T07:25:37.368085+0000","last_undegraded":"2024-03-06T07:25:37.368085+0000","last_fullsized":"2024-03-06T07:25:37.368085+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:03.107757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347979+0000","last_change":"2024-03-06T07:25:37.347979+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347979+0000","last_undegraded":"2024-03-06T07:25:37.347979+0000","last_fullsized":"2024-03-06T07:25:37.347979+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359733+0000","last_change":"2024-03-06T07:25:37.357207+0000","last_active":"2024-03-06T07:25:37.359733+0000","last_peered":"2024-03-06T07:25:37.359733+0000","last_clean":"2024-03-06T07:25:37.359733+0000","last_became_active":"2024-03-06T07:25:37.356920+0000","last_became_peered":"2024-03-06T07:25:37.356920+0000","last_unstale":"2024-03-06T07:25:37.359733+0000","last_undegraded":"2024-03-06T07:25:37.359733+0000","last_fullsized":"2024-03-06T07:25:37.359733+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351780+0000","last_change":"2024-03-06T07:25:37.351780+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351780+0000","last_undegraded":"2024-03-06T07:25:37.351780+0000","last_fullsized":"2024-03-06T07:25:37.351780+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361242+0000","last_change":"2024-03-06T07:25:37.360307+0000","last_active":"2024-03-06T07:25:37.361242+0000","last_peered":"2024-03-06T07:25:37.361242+0000","last_clean":"2024-03-06T07:25:37.361242+0000","last_became_active":"2024-03-06T07:25:37.360024+0000","last_became_peered":"2024-03-06T07:25:37.360024+0000","last_unstale":"2024-03-06T07:25:37.361242+0000","last_undegraded":"2024-03-06T07:25:37.361242+0000","last_fullsized":"2024-03-06T07:25:37.361242+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:13.020947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349007+0000","last_change":"2024-03-06T07:25:37.349007+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349007+0000","last_undegraded":"2024-03-06T07:25:37.349007+0000","last_fullsized":"2024-03-06T07:25:37.349007+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368184+0000","last_change":"2024-03-06T07:25:37.364965+0000","last_active":"2024-03-06T07:25:37.368184+0000","last_peered":"2024-03-06T07:25:37.368184+0000","last_clean":"2024-03-06T07:25:37.368184+0000","last_became_active":"2024-03-06T07:25:37.363889+0000","last_became_peered":"2024-03-06T07:25:37.363889+0000","last_unstale":"2024-03-06T07:25:37.368184+0000","last_undegraded":"2024-03-06T07:25:37.368184+0000","last_fullsized":"2024-03-06T07:25:37.368184+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352125+0000","last_change":"2024-03-06T07:25:37.352125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352125+0000","last_undegraded":"2024-03-06T07:25:37.352125+0000","last_fullsized":"2024-03-06T07:25:37.352125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364603+0000","last_change":"2024-03-06T07:25:37.364101+0000","last_active":"2024-03-06T07:25:37.364603+0000","last_peered":"2024-03-06T07:25:37.364603+0000","last_clean":"2024-03-06T07:25:37.364603+0000","last_became_active":"2024-03-06T07:25:37.363814+0000","last_became_peered":"2024-03-06T07:25:37.363814+0000","last_unstale":"2024-03-06T07:25:37.364603+0000","last_undegraded":"2024-03-06T07:25:37.364603+0000","last_fullsized":"2024-03-06T07:25:37.364603+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:08.141356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.356323+0000","last_change":"2024-03-06T07:25:37.356323+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.356323+0000","last_undegraded":"2024-03-06T07:25:37.356323+0000","last_fullsized":"2024-03-06T07:25:37.356323+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359709+0000","last_change":"2024-03-06T07:25:37.358403+0000","last_active":"2024-03-06T07:25:37.359709+0000","last_peered":"2024-03-06T07:25:37.359709+0000","last_clean":"2024-03-06T07:25:37.359709+0000","last_became_active":"2024-03-06T07:25:37.358134+0000","last_became_peered":"2024-03-06T07:25:37.358134+0000","last_unstale":"2024-03-06T07:25:37.359709+0000","last_undegraded":"2024-03-06T07:25:37.359709+0000","last_fullsized":"2024-03-06T07:25:37.359709+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.829122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352078+0000","last_change":"2024-03-06T07:25:37.352078+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352078+0000","last_undegraded":"2024-03-06T07:25:37.352078+0000","last_fullsized":"2024-03-06T07:25:37.352078+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358546+0000","last_change":"2024-03-06T07:25:37.355316+0000","last_active":"2024-03-06T07:25:37.358546+0000","last_peered":"2024-03-06T07:25:37.358546+0000","last_clean":"2024-03-06T07:25:37.358546+0000","last_became_active":"2024-03-06T07:25:37.355060+0000","last_became_peered":"2024-03-06T07:25:37.355060+0000","last_unstale":"2024-03-06T07:25:37.358546+0000","last_undegraded":"2024-03-06T07:25:37.358546+0000","last_fullsized":"2024-03-06T07:25:37.358546+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:27.708230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350295+0000","last_change":"2024-03-06T07:25:37.350295+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350295+0000","last_undegraded":"2024-03-06T07:25:37.350295+0000","last_fullsized":"2024-03-06T07:25:37.350295+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.357802+0000","last_change":"2024-03-06T07:25:37.354804+0000","last_active":"2024-03-06T07:25:37.357802+0000","last_peered":"2024-03-06T07:25:37.357802+0000","last_clean":"2024-03-06T07:25:37.357802+0000","last_became_active":"2024-03-06T07:25:37.354535+0000","last_became_peered":"2024-03-06T07:25:37.354535+0000","last_unstale":"2024-03-06T07:25:37.357802+0000","last_undegraded":"2024-03-06T07:25:37.357802+0000","last_fullsized":"2024-03-06T07:25:37.357802+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:11.747252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351130+0000","last_change":"2024-03-06T07:25:37.351130+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351130+0000","last_undegraded":"2024-03-06T07:25:37.351130+0000","last_fullsized":"2024-03-06T07:25:37.351130+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.367951+0000","last_change":"2024-03-06T07:25:37.362425+0000","last_active":"2024-03-06T07:25:37.367951+0000","last_peered":"2024-03-06T07:25:37.367951+0000","last_clean":"2024-03-06T07:25:37.367951+0000","last_became_active":"2024-03-06T07:25:37.362172+0000","last_became_peered":"2024-03-06T07:25:37.362172+0000","last_unstale":"2024-03-06T07:25:37.367951+0000","last_undegraded":"2024-03-06T07:25:37.367951+0000","last_fullsized":"2024-03-06T07:25:37.367951+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:46.062578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350611+0000","last_change":"2024-03-06T07:25:37.350611+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350611+0000","last_undegraded":"2024-03-06T07:25:37.350611+0000","last_fullsized":"2024-03-06T07:25:37.350611+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356209+0000","last_change":"2024-03-06T07:25:37.354863+0000","last_active":"2024-03-06T07:25:37.356209+0000","last_peered":"2024-03-06T07:25:37.356209+0000","last_clean":"2024-03-06T07:25:37.356209+0000","last_became_active":"2024-03-06T07:25:37.354465+0000","last_became_peered":"2024-03-06T07:25:37.354465+0000","last_unstale":"2024-03-06T07:25:37.356209+0000","last_undegraded":"2024-03-06T07:25:37.356209+0000","last_fullsized":"2024-03-06T07:25:37.356209+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:33.951539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349868+0000","last_change":"2024-03-06T07:25:37.349868+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349868+0000","last_undegraded":"2024-03-06T07:25:37.349868+0000","last_fullsized":"2024-03-06T07:25:37.349868+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362855+0000","last_change":"2024-03-06T07:25:37.361572+0000","last_active":"2024-03-06T07:25:37.362855+0000","last_peered":"2024-03-06T07:25:37.362855+0000","last_clean":"2024-03-06T07:25:37.362855+0000","last_became_active":"2024-03-06T07:25:37.361271+0000","last_became_peered":"2024-03-06T07:25:37.361271+0000","last_unstale":"2024-03-06T07:25:37.362855+0000","last_undegraded":"2024-03-06T07:25:37.362855+0000","last_fullsized":"2024-03-06T07:25:37.362855+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:13.032326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351680+0000","last_change":"2024-03-06T07:25:37.351680+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351680+0000","last_undegraded":"2024-03-06T07:25:37.351680+0000","last_fullsized":"2024-03-06T07:25:37.351680+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360711+0000","last_change":"2024-03-06T07:25:37.359314+0000","last_active":"2024-03-06T07:25:37.360711+0000","last_peered":"2024-03-06T07:25:37.360711+0000","last_clean":"2024-03-06T07:25:37.360711+0000","last_became_active":"2024-03-06T07:25:37.359008+0000","last_became_peered":"2024-03-06T07:25:37.359008+0000","last_unstale":"2024-03-06T07:25:37.360711+0000","last_undegraded":"2024-03-06T07:25:37.360711+0000","last_fullsized":"2024-03-06T07:25:37.360711+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:07.137677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351711+0000","last_change":"2024-03-06T07:25:37.351711+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351711+0000","last_undegraded":"2024-03-06T07:25:37.351711+0000","last_fullsized":"2024-03-06T07:25:37.351711+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368155+0000","last_change":"2024-03-06T07:25:37.364253+0000","last_active":"2024-03-06T07:25:37.368155+0000","last_peered":"2024-03-06T07:25:37.368155+0000","last_clean":"2024-03-06T07:25:37.368155+0000","last_became_active":"2024-03-06T07:25:37.363933+0000","last_became_peered":"2024-03-06T07:25:37.363933+0000","last_unstale":"2024-03-06T07:25:37.368155+0000","last_undegraded":"2024-03-06T07:25:37.368155+0000","last_fullsized":"2024-03-06T07:25:37.368155+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355920+0000","last_change":"2024-03-06T07:25:37.355920+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355920+0000","last_undegraded":"2024-03-06T07:25:37.355920+0000","last_fullsized":"2024-03-06T07:25:37.355920+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364944+0000","last_change":"2024-03-06T07:25:37.364443+0000","last_active":"2024-03-06T07:25:37.364944+0000","last_peered":"2024-03-06T07:25:37.364944+0000","last_clean":"2024-03-06T07:25:37.364944+0000","last_became_active":"2024-03-06T07:25:37.364093+0000","last_became_peered":"2024-03-06T07:25:37.364093+0000","last_unstale":"2024-03-06T07:25:37.364944+0000","last_undegraded":"2024-03-06T07:25:37.364944+0000","last_fullsized":"2024-03-06T07:25:37.364944+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:01.483203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351304+0000","last_change":"2024-03-06T07:25:37.351304+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351304+0000","last_undegraded":"2024-03-06T07:25:37.351304+0000","last_fullsized":"2024-03-06T07:25:37.351304+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.363191+0000","last_change":"2024-03-06T07:25:37.362488+0000","last_active":"2024-03-06T07:25:37.363191+0000","last_peered":"2024-03-06T07:25:37.363191+0000","last_clean":"2024-03-06T07:25:37.363191+0000","last_became_active":"2024-03-06T07:25:37.362203+0000","last_became_peered":"2024-03-06T07:25:37.362203+0000","last_unstale":"2024-03-06T07:25:37.363191+0000","last_undegraded":"2024-03-06T07:25:37.363191+0000","last_fullsized":"2024-03-06T07:25:37.363191+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.309117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348099+0000","last_change":"2024-03-06T07:25:37.348099+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348099+0000","last_undegraded":"2024-03-06T07:25:37.348099+0000","last_fullsized":"2024-03-06T07:25:37.348099+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368386+0000","last_change":"2024-03-06T07:25:37.366204+0000","last_active":"2024-03-06T07:25:37.368386+0000","last_peered":"2024-03-06T07:25:37.368386+0000","last_clean":"2024-03-06T07:25:37.368386+0000","last_became_active":"2024-03-06T07:25:37.365751+0000","last_became_peered":"2024-03-06T07:25:37.365751+0000","last_unstale":"2024-03-06T07:25:37.368386+0000","last_undegraded":"2024-03-06T07:25:37.368386+0000","last_fullsized":"2024-03-06T07:25:37.368386+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350714+0000","last_change":"2024-03-06T07:25:37.350714+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350714+0000","last_undegraded":"2024-03-06T07:25:37.350714+0000","last_fullsized":"2024-03-06T07:25:37.350714+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368066+0000","last_change":"2024-03-06T07:25:37.363581+0000","last_active":"2024-03-06T07:25:37.368066+0000","last_peered":"2024-03-06T07:25:37.368066+0000","last_clean":"2024-03-06T07:25:37.368066+0000","last_became_active":"2024-03-06T07:25:37.362991+0000","last_became_peered":"2024-03-06T07:25:37.362991+0000","last_unstale":"2024-03-06T07:25:37.368066+0000","last_undegraded":"2024-03-06T07:25:37.368066+0000","last_fullsized":"2024-03-06T07:25:37.368066+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:25.392712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351198+0000","last_change":"2024-03-06T07:25:37.351198+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351198+0000","last_undegraded":"2024-03-06T07:25:37.351198+0000","last_fullsized":"2024-03-06T07:25:37.351198+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362479+0000","last_change":"2024-03-06T07:25:37.361352+0000","last_active":"2024-03-06T07:25:37.362479+0000","last_peered":"2024-03-06T07:25:37.362479+0000","last_clean":"2024-03-06T07:25:37.362479+0000","last_became_active":"2024-03-06T07:25:37.361059+0000","last_became_peered":"2024-03-06T07:25:37.361059+0000","last_unstale":"2024-03-06T07:25:37.362479+0000","last_undegraded":"2024-03-06T07:25:37.362479+0000","last_fullsized":"2024-03-06T07:25:37.362479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348043+0000","last_change":"2024-03-06T07:25:37.348043+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348043+0000","last_undegraded":"2024-03-06T07:25:37.348043+0000","last_fullsized":"2024-03-06T07:25:37.348043+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368916+0000","last_change":"2024-03-06T07:25:37.368239+0000","last_active":"2024-03-06T07:25:37.368916+0000","last_peered":"2024-03-06T07:25:37.368916+0000","last_clean":"2024-03-06T07:25:37.368916+0000","last_became_active":"2024-03-06T07:25:37.367979+0000","last_became_peered":"2024-03-06T07:25:37.367979+0000","last_unstale":"2024-03-06T07:25:37.368916+0000","last_undegraded":"2024-03-06T07:25:37.368916+0000","last_fullsized":"2024-03-06T07:25:37.368916+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:03.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349897+0000","last_change":"2024-03-06T07:25:37.349897+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349897+0000","last_undegraded":"2024-03-06T07:25:37.349897+0000","last_fullsized":"2024-03-06T07:25:37.349897+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368412+0000","last_change":"2024-03-06T07:25:37.366881+0000","last_active":"2024-03-06T07:25:37.368412+0000","last_peered":"2024-03-06T07:25:37.368412+0000","last_clean":"2024-03-06T07:25:37.368412+0000","last_became_active":"2024-03-06T07:25:37.366588+0000","last_became_peered":"2024-03-06T07:25:37.366588+0000","last_unstale":"2024-03-06T07:25:37.368412+0000","last_undegraded":"2024-03-06T07:25:37.368412+0000","last_fullsized":"2024-03-06T07:25:37.368412+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:46.200362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354567+0000","last_change":"2024-03-06T07:25:37.354567+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354567+0000","last_undegraded":"2024-03-06T07:25:37.354567+0000","last_fullsized":"2024-03-06T07:25:37.354567+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364364+0000","last_change":"2024-03-06T07:25:37.359284+0000","last_active":"2024-03-06T07:25:37.364364+0000","last_peered":"2024-03-06T07:25:37.364364+0000","last_clean":"2024-03-06T07:25:37.364364+0000","last_became_active":"2024-03-06T07:25:37.358847+0000","last_became_peered":"2024-03-06T07:25:37.358847+0000","last_unstale":"2024-03-06T07:25:37.364364+0000","last_undegraded":"2024-03-06T07:25:37.364364+0000","last_fullsized":"2024-03-06T07:25:37.364364+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.913135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":472,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356621+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:25:37.356621+0000","last_peered":"2024-03-06T07:25:37.356621+0000","last_clean":"2024-03-06T07:25:37.356621+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:25:37.356621+0000","last_undegraded":"2024-03-06T07:25:37.356621+0000","last_fullsized":"2024-03-06T07:25:37.356621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33412,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338428,"statfs":{"total":96636764160,"available":96602550272,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37624,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334216,"statfs":{"total":96636764160,"available":96598237184,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3f is not active+clean 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353125+0000', 'last_change': '2024-03-06T07:25:37.353125+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353125+0000', 'last_undegraded': '2024-03-06T07:25:37.353125+0000', 'last_fullsized': '2024-03-06T07:25:37.353125+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3e is not active+clean 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351988+0000', 'last_change': '2024-03-06T07:25:37.351988+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351988+0000', 'last_undegraded': '2024-03-06T07:25:37.351988+0000', 'last_fullsized': '2024-03-06T07:25:37.351988+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3d is not active+clean 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.355468+0000', 'last_change': '2024-03-06T07:25:37.355468+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.355468+0000', 'last_undegraded': '2024-03-06T07:25:37.355468+0000', 'last_fullsized': '2024-03-06T07:25:37.355468+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3c is not active+clean 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.347174+0000', 'last_change': '2024-03-06T07:25:37.347174+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.347174+0000', 'last_undegraded': '2024-03-06T07:25:37.347174+0000', 'last_fullsized': '2024-03-06T07:25:37.347174+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3b is not active+clean 2024-03-06T07:25:42.020 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351436+0000', 'last_change': '2024-03-06T07:25:37.351436+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351436+0000', 'last_undegraded': '2024-03-06T07:25:37.351436+0000', 'last_fullsized': '2024-03-06T07:25:37.351436+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3a is not active+clean 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349668+0000', 'last_change': '2024-03-06T07:25:37.349668+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349668+0000', 'last_undegraded': '2024-03-06T07:25:37.349668+0000', 'last_fullsized': '2024-03-06T07:25:37.349668+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.39 is not active+clean 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353857+0000', 'last_change': '2024-03-06T07:25:37.353857+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353857+0000', 'last_undegraded': '2024-03-06T07:25:37.353857+0000', 'last_fullsized': '2024-03-06T07:25:37.353857+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.38 is not active+clean 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.352760+0000', 'last_change': '2024-03-06T07:25:37.352760+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.352760+0000', 'last_undegraded': '2024-03-06T07:25:37.352760+0000', 'last_fullsized': '2024-03-06T07:25:37.352760+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.37 is not active+clean 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349674+0000', 'last_change': '2024-03-06T07:25:37.349674+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349674+0000', 'last_undegraded': '2024-03-06T07:25:37.349674+0000', 'last_fullsized': '2024-03-06T07:25:37.349674+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.36 is not active+clean 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.355049+0000', 'last_change': '2024-03-06T07:25:37.355049+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.355049+0000', 'last_undegraded': '2024-03-06T07:25:37.355049+0000', 'last_fullsized': '2024-03-06T07:25:37.355049+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.35 is not active+clean 2024-03-06T07:25:42.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348776+0000', 'last_change': '2024-03-06T07:25:37.348776+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348776+0000', 'last_undegraded': '2024-03-06T07:25:37.348776+0000', 'last_fullsized': '2024-03-06T07:25:37.348776+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.34 is not active+clean 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348316+0000', 'last_change': '2024-03-06T07:25:37.348316+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348316+0000', 'last_undegraded': '2024-03-06T07:25:37.348316+0000', 'last_fullsized': '2024-03-06T07:25:37.348316+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.33 is not active+clean 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.352728+0000', 'last_change': '2024-03-06T07:25:37.352728+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.352728+0000', 'last_undegraded': '2024-03-06T07:25:37.352728+0000', 'last_fullsized': '2024-03-06T07:25:37.352728+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.32 is not active+clean 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351809+0000', 'last_change': '2024-03-06T07:25:37.351809+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351809+0000', 'last_undegraded': '2024-03-06T07:25:37.351809+0000', 'last_fullsized': '2024-03-06T07:25:37.351809+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.31 is not active+clean 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350214+0000', 'last_change': '2024-03-06T07:25:37.350214+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350214+0000', 'last_undegraded': '2024-03-06T07:25:37.350214+0000', 'last_fullsized': '2024-03-06T07:25:37.350214+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.30 is not active+clean 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351435+0000', 'last_change': '2024-03-06T07:25:37.351435+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351435+0000', 'last_undegraded': '2024-03-06T07:25:37.351435+0000', 'last_fullsized': '2024-03-06T07:25:37.351435+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2f is not active+clean 2024-03-06T07:25:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350699+0000', 'last_change': '2024-03-06T07:25:37.350699+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350699+0000', 'last_undegraded': '2024-03-06T07:25:37.350699+0000', 'last_fullsized': '2024-03-06T07:25:37.350699+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2e is not active+clean 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.354123+0000', 'last_change': '2024-03-06T07:25:37.354123+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.354123+0000', 'last_undegraded': '2024-03-06T07:25:37.354123+0000', 'last_fullsized': '2024-03-06T07:25:37.354123+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2d is not active+clean 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.354755+0000', 'last_change': '2024-03-06T07:25:37.354755+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.354755+0000', 'last_undegraded': '2024-03-06T07:25:37.354755+0000', 'last_fullsized': '2024-03-06T07:25:37.354755+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2c is not active+clean 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350943+0000', 'last_change': '2024-03-06T07:25:37.350943+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350943+0000', 'last_undegraded': '2024-03-06T07:25:37.350943+0000', 'last_fullsized': '2024-03-06T07:25:37.350943+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2b is not active+clean 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353972+0000', 'last_change': '2024-03-06T07:25:37.353972+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353972+0000', 'last_undegraded': '2024-03-06T07:25:37.353972+0000', 'last_fullsized': '2024-03-06T07:25:37.353972+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2a is not active+clean 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349753+0000', 'last_change': '2024-03-06T07:25:37.349753+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349753+0000', 'last_undegraded': '2024-03-06T07:25:37.349753+0000', 'last_fullsized': '2024-03-06T07:25:37.349753+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.29 is not active+clean 2024-03-06T07:25:42.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349137+0000', 'last_change': '2024-03-06T07:25:37.349137+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349137+0000', 'last_undegraded': '2024-03-06T07:25:37.349137+0000', 'last_fullsized': '2024-03-06T07:25:37.349137+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.28 is not active+clean 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.355131+0000', 'last_change': '2024-03-06T07:25:37.355131+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.355131+0000', 'last_undegraded': '2024-03-06T07:25:37.355131+0000', 'last_fullsized': '2024-03-06T07:25:37.355131+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.27 is not active+clean 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.352728+0000', 'last_change': '2024-03-06T07:25:37.352728+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.352728+0000', 'last_undegraded': '2024-03-06T07:25:37.352728+0000', 'last_fullsized': '2024-03-06T07:25:37.352728+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.26 is not active+clean 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353675+0000', 'last_change': '2024-03-06T07:25:37.353675+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353675+0000', 'last_undegraded': '2024-03-06T07:25:37.353675+0000', 'last_fullsized': '2024-03-06T07:25:37.353675+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.25 is not active+clean 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348241+0000', 'last_change': '2024-03-06T07:25:37.348241+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348241+0000', 'last_undegraded': '2024-03-06T07:25:37.348241+0000', 'last_fullsized': '2024-03-06T07:25:37.348241+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.24 is not active+clean 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.347813+0000', 'last_change': '2024-03-06T07:25:37.347813+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.347813+0000', 'last_undegraded': '2024-03-06T07:25:37.347813+0000', 'last_fullsized': '2024-03-06T07:25:37.347813+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.23 is not active+clean 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351675+0000', 'last_change': '2024-03-06T07:25:37.351675+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351675+0000', 'last_undegraded': '2024-03-06T07:25:37.351675+0000', 'last_fullsized': '2024-03-06T07:25:37.351675+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.22 is not active+clean 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350203+0000', 'last_change': '2024-03-06T07:25:37.350203+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350203+0000', 'last_undegraded': '2024-03-06T07:25:37.350203+0000', 'last_fullsized': '2024-03-06T07:25:37.350203+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.21 is not active+clean 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348621+0000', 'last_change': '2024-03-06T07:25:37.348621+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348621+0000', 'last_undegraded': '2024-03-06T07:25:37.348621+0000', 'last_fullsized': '2024-03-06T07:25:37.348621+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.20 is not active+clean 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349549+0000', 'last_change': '2024-03-06T07:25:37.349549+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349549+0000', 'last_undegraded': '2024-03-06T07:25:37.349549+0000', 'last_fullsized': '2024-03-06T07:25:37.349549+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1c is not active+clean 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353167+0000', 'last_change': '2024-03-06T07:25:37.353167+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353167+0000', 'last_undegraded': '2024-03-06T07:25:37.353167+0000', 'last_fullsized': '2024-03-06T07:25:37.353167+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1d is not active+clean 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353595+0000', 'last_change': '2024-03-06T07:25:37.353595+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353595+0000', 'last_undegraded': '2024-03-06T07:25:37.353595+0000', 'last_fullsized': '2024-03-06T07:25:37.353595+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1e is not active+clean 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353456+0000', 'last_change': '2024-03-06T07:25:37.353456+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353456+0000', 'last_undegraded': '2024-03-06T07:25:37.353456+0000', 'last_fullsized': '2024-03-06T07:25:37.353456+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.18 is not active+clean 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353298+0000', 'last_change': '2024-03-06T07:25:37.353298+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353298+0000', 'last_undegraded': '2024-03-06T07:25:37.353298+0000', 'last_fullsized': '2024-03-06T07:25:37.353298+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.19 is not active+clean 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350553+0000', 'last_change': '2024-03-06T07:25:37.350553+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350553+0000', 'last_undegraded': '2024-03-06T07:25:37.350553+0000', 'last_fullsized': '2024-03-06T07:25:37.350553+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1a is not active+clean 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349256+0000', 'last_change': '2024-03-06T07:25:37.349256+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349256+0000', 'last_undegraded': '2024-03-06T07:25:37.349256+0000', 'last_fullsized': '2024-03-06T07:25:37.349256+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1b is not active+clean 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348628+0000', 'last_change': '2024-03-06T07:25:37.348628+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348628+0000', 'last_undegraded': '2024-03-06T07:25:37.348628+0000', 'last_fullsized': '2024-03-06T07:25:37.348628+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.14 is not active+clean 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.353004+0000', 'last_change': '2024-03-06T07:25:37.353004+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.353004+0000', 'last_undegraded': '2024-03-06T07:25:37.353004+0000', 'last_fullsized': '2024-03-06T07:25:37.353004+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.15 is not active+clean 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.355752+0000', 'last_change': '2024-03-06T07:25:37.355752+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.355752+0000', 'last_undegraded': '2024-03-06T07:25:37.355752+0000', 'last_fullsized': '2024-03-06T07:25:37.355752+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.16 is not active+clean 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.352531+0000', 'last_change': '2024-03-06T07:25:37.352531+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.352531+0000', 'last_undegraded': '2024-03-06T07:25:37.352531+0000', 'last_fullsized': '2024-03-06T07:25:37.352531+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.17 is not active+clean 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348006+0000', 'last_change': '2024-03-06T07:25:37.348006+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348006+0000', 'last_undegraded': '2024-03-06T07:25:37.348006+0000', 'last_fullsized': '2024-03-06T07:25:37.348006+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.10 is not active+clean 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349549+0000', 'last_change': '2024-03-06T07:25:37.349549+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349549+0000', 'last_undegraded': '2024-03-06T07:25:37.349549+0000', 'last_fullsized': '2024-03-06T07:25:37.349549+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.11 is not active+clean 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.347979+0000', 'last_change': '2024-03-06T07:25:37.347979+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.347979+0000', 'last_undegraded': '2024-03-06T07:25:37.347979+0000', 'last_fullsized': '2024-03-06T07:25:37.347979+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.12 is not active+clean 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351780+0000', 'last_change': '2024-03-06T07:25:37.351780+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351780+0000', 'last_undegraded': '2024-03-06T07:25:37.351780+0000', 'last_fullsized': '2024-03-06T07:25:37.351780+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.13 is not active+clean 2024-03-06T07:25:42.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349007+0000', 'last_change': '2024-03-06T07:25:37.349007+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349007+0000', 'last_undegraded': '2024-03-06T07:25:37.349007+0000', 'last_fullsized': '2024-03-06T07:25:37.349007+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.c is not active+clean 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.352125+0000', 'last_change': '2024-03-06T07:25:37.352125+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.352125+0000', 'last_undegraded': '2024-03-06T07:25:37.352125+0000', 'last_fullsized': '2024-03-06T07:25:37.352125+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.d is not active+clean 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.356323+0000', 'last_change': '2024-03-06T07:25:37.356323+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.356323+0000', 'last_undegraded': '2024-03-06T07:25:37.356323+0000', 'last_fullsized': '2024-03-06T07:25:37.356323+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.e is not active+clean 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.352078+0000', 'last_change': '2024-03-06T07:25:37.352078+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.352078+0000', 'last_undegraded': '2024-03-06T07:25:37.352078+0000', 'last_fullsized': '2024-03-06T07:25:37.352078+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.f is not active+clean 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350295+0000', 'last_change': '2024-03-06T07:25:37.350295+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350295+0000', 'last_undegraded': '2024-03-06T07:25:37.350295+0000', 'last_fullsized': '2024-03-06T07:25:37.350295+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.8 is not active+clean 2024-03-06T07:25:42.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351130+0000', 'last_change': '2024-03-06T07:25:37.351130+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351130+0000', 'last_undegraded': '2024-03-06T07:25:37.351130+0000', 'last_fullsized': '2024-03-06T07:25:37.351130+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.9 is not active+clean 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350611+0000', 'last_change': '2024-03-06T07:25:37.350611+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350611+0000', 'last_undegraded': '2024-03-06T07:25:37.350611+0000', 'last_fullsized': '2024-03-06T07:25:37.350611+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.a is not active+clean 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349868+0000', 'last_change': '2024-03-06T07:25:37.349868+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349868+0000', 'last_undegraded': '2024-03-06T07:25:37.349868+0000', 'last_fullsized': '2024-03-06T07:25:37.349868+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.b is not active+clean 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351680+0000', 'last_change': '2024-03-06T07:25:37.351680+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351680+0000', 'last_undegraded': '2024-03-06T07:25:37.351680+0000', 'last_fullsized': '2024-03-06T07:25:37.351680+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.4 is not active+clean 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351711+0000', 'last_change': '2024-03-06T07:25:37.351711+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351711+0000', 'last_undegraded': '2024-03-06T07:25:37.351711+0000', 'last_fullsized': '2024-03-06T07:25:37.351711+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.5 is not active+clean 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.355920+0000', 'last_change': '2024-03-06T07:25:37.355920+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.355920+0000', 'last_undegraded': '2024-03-06T07:25:37.355920+0000', 'last_fullsized': '2024-03-06T07:25:37.355920+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:42.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.6 is not active+clean 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351304+0000', 'last_change': '2024-03-06T07:25:37.351304+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351304+0000', 'last_undegraded': '2024-03-06T07:25:37.351304+0000', 'last_fullsized': '2024-03-06T07:25:37.351304+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.7 is not active+clean 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348099+0000', 'last_change': '2024-03-06T07:25:37.348099+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348099+0000', 'last_undegraded': '2024-03-06T07:25:37.348099+0000', 'last_fullsized': '2024-03-06T07:25:37.348099+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.0 is not active+clean 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.350714+0000', 'last_change': '2024-03-06T07:25:37.350714+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.350714+0000', 'last_undegraded': '2024-03-06T07:25:37.350714+0000', 'last_fullsized': '2024-03-06T07:25:37.350714+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1 is not active+clean 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.351198+0000', 'last_change': '2024-03-06T07:25:37.351198+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.351198+0000', 'last_undegraded': '2024-03-06T07:25:37.351198+0000', 'last_fullsized': '2024-03-06T07:25:37.351198+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.2 is not active+clean 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.348043+0000', 'last_change': '2024-03-06T07:25:37.348043+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.348043+0000', 'last_undegraded': '2024-03-06T07:25:37.348043+0000', 'last_fullsized': '2024-03-06T07:25:37.348043+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.3 is not active+clean 2024-03-06T07:25:42.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.349897+0000', 'last_change': '2024-03-06T07:25:37.349897+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.349897+0000', 'last_undegraded': '2024-03-06T07:25:37.349897+0000', 'last_fullsized': '2024-03-06T07:25:37.349897+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:42.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 30.1f is not active+clean 2024-03-06T07:25:42.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '30.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 180, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:37.354567+0000', 'last_change': '2024-03-06T07:25:37.354567+0000', 'last_active': '2024-03-06T07:25:37.341311+0000', 'last_peered': '2024-03-06T07:25:37.341311+0000', 'last_clean': '2024-03-06T07:25:37.341311+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:37.354567+0000', 'last_undegraded': '2024-03-06T07:25:37.354567+0000', 'last_fullsized': '2024-03-06T07:25:37.354567+0000', 'mapping_epoch': 180, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 180, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:37.341311+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:42.031 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:25:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.152+0000 7f1ce5d5e640 1 Processor -- start 2024-03-06T07:25:42.155 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.152+0000 7f1ce5d5e640 1 -- start start 2024-03-06T07:25:42.156 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1ce5d5e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1ce00c7180 0x7f1ce0159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1ce5d5e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00c7c60 con 0x7f1ce00c88b0 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1ce5d5e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00c8060 con 0x7f1ce01527b0 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1ce5d5e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce0159640 con 0x7f1ce00c7180 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1cd6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1ce00c7180 0x7f1ce0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1cdffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1ce00c88b0 0x7f1ce00c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58480/0 (socket says 172.21.15.94:58480) 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1cd6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1ce00c7180 0x7f1ce0159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41868/0 (socket says 172.21.15.94:41868) 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.153+0000 7f1cdffff640 1 -- 172.21.15.94:0/1208572032 learned_addr learned my addr 172.21.15.94:0/1208572032 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.154+0000 7f1cd6ffd640 1 -- 172.21.15.94:0/1208572032 >> v1:172.21.15.145:6789/0 conn(0x7f1ce01527b0 legacy=0x7f1ce0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.154+0000 7f1cd6ffd640 1 -- 172.21.15.94:0/1208572032 >> v1:172.21.15.94:6789/0 conn(0x7f1ce00c88b0 legacy=0x7f1ce00c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.154+0000 7f1cdeffd640 1 -- 172.21.15.94:0/1208572032 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517171196 0 0) 0x7f1cd8001490 con 0x7f1ce00c88b0 2024-03-06T07:25:42.157 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.154+0000 7f1cd6ffd640 1 -- 172.21.15.94:0/1208572032 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00c76a0 con 0x7f1ce00c7180 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.154+0000 7f1cd6ffd640 1 --2- 172.21.15.94:0/1208572032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce0159100 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009720 tx=0x7f1cc802b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b032d766f302bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1cdeffd640 1 -- 172.21.15.94:0/1208572032 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8037070 con 0x7f1ce00c7180 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1cdeffd640 1 -- 172.21.15.94:0/1208572032 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc8030710 con 0x7f1ce00c7180 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1cdeffd640 1 -- 172.21.15.94:0/1208572032 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8030ae0 con 0x7f1ce00c7180 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1208572032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 msgr2=0x7f1ce0159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1208572032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce0159100 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009720 tx=0x7f1cc802b6a0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1208572032 shutdown_connections 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1208572032 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce0159100 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1208572032 >> 172.21.15.94:0/1208572032 conn(0x7f1ce00ac560 msgr2=0x7f1ce00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1208572032 shutdown_connections 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.155+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1208572032 wait complete. 2024-03-06T07:25:42.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.156+0000 7f1ce5d5e640 1 Processor -- start 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.156+0000 7f1ce5d5e640 1 -- start start 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1ce5d5e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce00c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1ce5d5e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ce00c88b0 0x7f1ce014ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1ce5d5e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ce01527b0 0x7f1ce014efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1ce5d5e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce0155600 con 0x7f1ce00c7180 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1ce5d5e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce0155320 con 0x7f1ce01527b0 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1ce5d5e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce0152b90 con 0x7f1ce00c88b0 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce00c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce00c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41870/0 (socket says 172.21.15.94:41870) 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 -- 172.21.15.94:0/1807777872 learned_addr learned my addr 172.21.15.94:0/1807777872 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdffff640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ce01527b0 0x7f1ce014efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cd6ffd640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ce00c88b0 0x7f1ce014ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 -- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ce00c88b0 msgr2=0x7f1ce014ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ce00c88b0 0x7f1ce014ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 -- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ce01527b0 msgr2=0x7f1ce014efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ce01527b0 0x7f1ce014efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.157+0000 7f1cdf7fe640 1 -- 172.21.15.94:0/1807777872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00c4790 con 0x7f1ce00c7180 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.158+0000 7f1cdf7fe640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce00c2890 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000c550 tx=0x7f1cd000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.158+0000 7f1cdcff9640 1 -- 172.21.15.94:0/1807777872 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0017070 con 0x7f1ce00c7180 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.158+0000 7f1cdcff9640 1 -- 172.21.15.94:0/1807777872 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd0005480 con 0x7f1ce00c7180 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.158+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce01ebac0 con 0x7f1ce00c7180 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.158+0000 7f1cdcff9640 1 -- 172.21.15.94:0/1807777872 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd0003830 con 0x7f1ce00c7180 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.158+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce00c2dd0 con 0x7f1ce00c7180 2024-03-06T07:25:42.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.159+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce0075300 con 0x7f1ce00c7180 2024-03-06T07:25:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.159+0000 7f1cdcff9640 1 -- 172.21.15.94:0/1807777872 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cd001a070 con 0x7f1ce00c7180 2024-03-06T07:25:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.160+0000 7f1cdcff9640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cc006b4d0 0x7f1cc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.160+0000 7f1cdcff9640 1 -- 172.21.15.94:0/1807777872 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1cd0089730 con 0x7f1ce00c7180 2024-03-06T07:25:42.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.160+0000 7f1cd6ffd640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cc006b4d0 0x7f1cc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.160+0000 7f1cd6ffd640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cc006b4d0 0x7f1cc006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8034000 tx=0x7f1cc8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.167 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.164+0000 7f1cdcff9640 1 -- 172.21.15.94:0/1807777872 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ce0075300 con 0x7f1ce00c7180 2024-03-06T07:25:42.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.402+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1ce004f330 con 0x7f1ce00c7180 2024-03-06T07:25:42.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.402+0000 7f1cdcff9640 1 -- 172.21.15.94:0/1807777872 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1459 (secure 0 0 0) 0x7f1cd0057dd0 con 0x7f1ce00c7180 2024-03-06T07:25:42.405 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:42.405 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":468,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":182,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709482,"num_in_osds":8,"osd_in_since":1709709474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":278024192,"bytes_avail":772816089088,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":171,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"b","status":"up:active","gid":37763}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-03-06T07:24:40.718982+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cc006b4d0 msgr2=0x7f1cc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cc006b4d0 0x7f1cc006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8034000 tx=0x7f1cc8008030 comp rx=0 tx=0).stop 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 msgr2=0x7f1ce00c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce00c2890 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000c550 tx=0x7f1cd000ca10 comp rx=0 tx=0).stop 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 shutdown_connections 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cc006b4d0 0x7f1cc006d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ce01527b0 0x7f1ce014efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ce00c88b0 0x7f1ce014ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 --2- 172.21.15.94:0/1807777872 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ce00c7180 0x7f1ce00c2890 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 >> 172.21.15.94:0/1807777872 conn(0x7f1ce00ac560 msgr2=0x7f1ce006c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 shutdown_connections 2024-03-06T07:25:42.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.404+0000 7f1ce5d5e640 1 -- 172.21.15.94:0/1807777872 wait complete. 2024-03-06T07:25:42.418 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:25:42.418 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:42.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.516+0000 7f1388a9a640 1 Processor -- start 2024-03-06T07:25:42.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.516+0000 7f1388a9a640 1 -- start start 2024-03-06T07:25:42.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.516+0000 7f1388a9a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13841592d0 0x7f13841552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.516+0000 7f1388a9a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1384152d40 con 0x7f13840ac260 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.516+0000 7f1388a9a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13840c1b50 con 0x7f1384152960 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.516+0000 7f1388a9a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1384066e30 con 0x7f13841592d0 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f137bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13841592d0 0x7f13841552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f1382d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f13840ac260 0x7f1384155810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58482/0 (socket says 172.21.15.94:58482) 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f137bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13841592d0 0x7f13841552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41882/0 (socket says 172.21.15.94:41882) 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f1382d9d640 1 -- 172.21.15.94:0/3388363477 learned_addr learned my addr 172.21.15.94:0/3388363477 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f1381d9b640 1 -- 172.21.15.94:0/3388363477 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271589087 0 0) 0x7f1384152d40 con 0x7f13840ac260 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f1381d9b640 1 -- 172.21.15.94:0/3388363477 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1370003520 con 0x7f13840ac260 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f1381d9b640 1 -- 172.21.15.94:0/3388363477 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139395281 0 0) 0x7f13840c1b50 con 0x7f1384152960 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f137bfff640 1 -- 172.21.15.94:0/3388363477 >> v1:172.21.15.145:6789/0 conn(0x7f1384152960 legacy=0x7f138406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f137bfff640 1 -- 172.21.15.94:0/3388363477 >> v1:172.21.15.94:6789/0 conn(0x7f13840ac260 legacy=0x7f1384155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.517+0000 7f137bfff640 1 -- 172.21.15.94:0/3388363477 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13840c1dc0 con 0x7f13841592d0 2024-03-06T07:25:42.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f137bfff640 1 --2- 172.21.15.94:0/3388363477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13841592d0 0x7f13841552d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f136c0097a0 tx=0x7f136c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831f6fea7d5dab54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1381d9b640 1 -- 172.21.15.94:0/3388363477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c034070 con 0x7f13841592d0 2024-03-06T07:25:42.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1381d9b640 1 -- 172.21.15.94:0/3388363477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136c030610 con 0x7f13841592d0 2024-03-06T07:25:42.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1381d9b640 1 -- 172.21.15.94:0/3388363477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c0309e0 con 0x7f13841592d0 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1388a9a640 1 -- 172.21.15.94:0/3388363477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13841592d0 msgr2=0x7f13841552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3388363477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13841592d0 0x7f13841552d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f136c0097a0 tx=0x7f136c02b960 comp rx=0 tx=0).stop 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1388a9a640 1 -- 172.21.15.94:0/3388363477 shutdown_connections 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3388363477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13841592d0 0x7f13841552d0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1388a9a640 1 -- 172.21.15.94:0/3388363477 >> 172.21.15.94:0/3388363477 conn(0x7f13840c70b0 msgr2=0x7f13840c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1388a9a640 1 -- 172.21.15.94:0/3388363477 shutdown_connections 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.518+0000 7f1388a9a640 1 -- 172.21.15.94:0/3388363477 wait complete. 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 Processor -- start 2024-03-06T07:25:42.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 -- start start 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13840ac260 0x7f1384071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1384152960 0x7f13841548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f138414f3c0 0x7f138414e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138406e3a0 con 0x7f1384152960 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1384159760 con 0x7f13840ac260 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1388a9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138406e520 con 0x7f138414f3c0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f137bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1384152960 0x7f13841548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f137bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1384152960 0x7f13841548e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41896/0 (socket says 172.21.15.94:41896) 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f137bfff640 1 -- 172.21.15.94:0/3618948457 learned_addr learned my addr 172.21.15.94:0/3618948457 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f138259c640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13840ac260 0x7f1384071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.520+0000 7f1382d9d640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f138414f3c0 0x7f138414e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137bfff640 1 -- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f138414f3c0 msgr2=0x7f138414e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137bfff640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f138414f3c0 0x7f138414e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137bfff640 1 -- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13840ac260 msgr2=0x7f1384071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137bfff640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13840ac260 0x7f1384071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137bfff640 1 -- 172.21.15.94:0/3618948457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384072460 con 0x7f1384152960 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137bfff640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1384152960 0x7f13841548e0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f136c02bf70 tx=0x7f136c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137affd640 1 -- 172.21.15.94:0/3618948457 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c049070 con 0x7f1384152960 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137affd640 1 -- 172.21.15.94:0/3618948457 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136c03a890 con 0x7f1384152960 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f137affd640 1 -- 172.21.15.94:0/3618948457 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c034040 con 0x7f1384152960 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138414f0e0 con 0x7f1384152960 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.521+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138414fa50 con 0x7f1384152960 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.522+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138404f330 con 0x7f1384152960 2024-03-06T07:25:42.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.522+0000 7f137affd640 1 -- 172.21.15.94:0/3618948457 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f136c04c070 con 0x7f1384152960 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.523+0000 7f137affd640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f136406b4d0 0x7f136406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.523+0000 7f137affd640 1 -- 172.21.15.94:0/3618948457 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f136c0ac140 con 0x7f1384152960 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.523+0000 7f138259c640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f136406b4d0 0x7f136406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.523+0000 7f138259c640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f136406b4d0 0x7f136406d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1374000f60 tx=0x7f1374009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.527+0000 7f137affd640 1 -- 172.21.15.94:0/3618948457 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f136c03f070 con 0x7f1384152960 2024-03-06T07:25:42.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.695+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13841e0600 con 0x7f136406b4d0 2024-03-06T07:25:42.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.707+0000 7f137affd640 1 -- 172.21.15.94:0/3618948457 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247159 (secure 0 0 0) 0x7f13841e0600 con 0x7f136406b4d0 2024-03-06T07:25:42.711 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:42.713 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:25:42.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f136406b4d0 msgr2=0x7f136406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f136406b4d0 0x7f136406d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1374000f60 tx=0x7f1374009040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1384152960 msgr2=0x7f13841548e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1384152960 0x7f13841548e0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f136c02bf70 tx=0x7f136c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 shutdown_connections 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f136406b4d0 0x7f136406d990 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f138414f3c0 0x7f138414e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1384152960 0x7f13841548e0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 --2- 172.21.15.94:0/3618948457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13840ac260 0x7f1384071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 >> 172.21.15.94:0/3618948457 conn(0x7f13840c70b0 msgr2=0x7f138406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 shutdown_connections 2024-03-06T07:25:42.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:42.709+0000 7f1388a9a640 1 -- 172.21.15.94:0/3618948457 wait complete. 2024-03-06T07:25:42.723 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-03-06T07:25:40.732723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271508,"kb_used_data":17880,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754703212,"statfs":{"total":773094113280,"available":772816089088,"internally_reserved":0,"allocated":18309120,"data_stored":9988974,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.390499"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353125+0000","last_change":"2024-03-06T07:25:37.353125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353125+0000","last_undegraded":"2024-03-06T07:25:37.353125+0000","last_fullsized":"2024-03-06T07:25:37.353125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351988+0000","last_change":"2024-03-06T07:25:37.351988+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351988+0000","last_undegraded":"2024-03-06T07:25:37.351988+0000","last_fullsized":"2024-03-06T07:25:37.351988+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355468+0000","last_change":"2024-03-06T07:25:37.355468+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355468+0000","last_undegraded":"2024-03-06T07:25:37.355468+0000","last_fullsized":"2024-03-06T07:25:37.355468+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347174+0000","last_change":"2024-03-06T07:25:37.347174+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347174+0000","last_undegraded":"2024-03-06T07:25:37.347174+0000","last_fullsized":"2024-03-06T07:25:37.347174+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351436+0000","last_change":"2024-03-06T07:25:37.351436+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351436+0000","last_undegraded":"2024-03-06T07:25:37.351436+0000","last_fullsized":"2024-03-06T07:25:37.351436+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349668+0000","last_change":"2024-03-06T07:25:37.349668+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349668+0000","last_undegraded":"2024-03-06T07:25:37.349668+0000","last_fullsized":"2024-03-06T07:25:37.349668+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353857+0000","last_change":"2024-03-06T07:25:37.353857+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353857+0000","last_undegraded":"2024-03-06T07:25:37.353857+0000","last_fullsized":"2024-03-06T07:25:37.353857+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352760+0000","last_change":"2024-03-06T07:25:37.352760+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352760+0000","last_undegraded":"2024-03-06T07:25:37.352760+0000","last_fullsized":"2024-03-06T07:25:37.352760+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349674+0000","last_change":"2024-03-06T07:25:37.349674+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349674+0000","last_undegraded":"2024-03-06T07:25:37.349674+0000","last_fullsized":"2024-03-06T07:25:37.349674+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355049+0000","last_change":"2024-03-06T07:25:37.355049+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355049+0000","last_undegraded":"2024-03-06T07:25:37.355049+0000","last_fullsized":"2024-03-06T07:25:37.355049+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348776+0000","last_change":"2024-03-06T07:25:37.348776+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348776+0000","last_undegraded":"2024-03-06T07:25:37.348776+0000","last_fullsized":"2024-03-06T07:25:37.348776+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348316+0000","last_change":"2024-03-06T07:25:37.348316+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348316+0000","last_undegraded":"2024-03-06T07:25:37.348316+0000","last_fullsized":"2024-03-06T07:25:37.348316+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351809+0000","last_change":"2024-03-06T07:25:37.351809+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351809+0000","last_undegraded":"2024-03-06T07:25:37.351809+0000","last_fullsized":"2024-03-06T07:25:37.351809+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350214+0000","last_change":"2024-03-06T07:25:37.350214+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350214+0000","last_undegraded":"2024-03-06T07:25:37.350214+0000","last_fullsized":"2024-03-06T07:25:37.350214+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351435+0000","last_change":"2024-03-06T07:25:37.351435+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351435+0000","last_undegraded":"2024-03-06T07:25:37.351435+0000","last_fullsized":"2024-03-06T07:25:37.351435+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350699+0000","last_change":"2024-03-06T07:25:37.350699+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350699+0000","last_undegraded":"2024-03-06T07:25:37.350699+0000","last_fullsized":"2024-03-06T07:25:37.350699+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354123+0000","last_change":"2024-03-06T07:25:37.354123+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354123+0000","last_undegraded":"2024-03-06T07:25:37.354123+0000","last_fullsized":"2024-03-06T07:25:37.354123+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354755+0000","last_change":"2024-03-06T07:25:37.354755+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354755+0000","last_undegraded":"2024-03-06T07:25:37.354755+0000","last_fullsized":"2024-03-06T07:25:37.354755+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350943+0000","last_change":"2024-03-06T07:25:37.350943+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350943+0000","last_undegraded":"2024-03-06T07:25:37.350943+0000","last_fullsized":"2024-03-06T07:25:37.350943+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353972+0000","last_change":"2024-03-06T07:25:37.353972+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353972+0000","last_undegraded":"2024-03-06T07:25:37.353972+0000","last_fullsized":"2024-03-06T07:25:37.353972+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349753+0000","last_change":"2024-03-06T07:25:37.349753+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349753+0000","last_undegraded":"2024-03-06T07:25:37.349753+0000","last_fullsized":"2024-03-06T07:25:37.349753+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349137+0000","last_change":"2024-03-06T07:25:37.349137+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349137+0000","last_undegraded":"2024-03-06T07:25:37.349137+0000","last_fullsized":"2024-03-06T07:25:37.349137+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355131+0000","last_change":"2024-03-06T07:25:37.355131+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355131+0000","last_undegraded":"2024-03-06T07:25:37.355131+0000","last_fullsized":"2024-03-06T07:25:37.355131+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352728+0000","last_change":"2024-03-06T07:25:37.352728+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352728+0000","last_undegraded":"2024-03-06T07:25:37.352728+0000","last_fullsized":"2024-03-06T07:25:37.352728+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353675+0000","last_change":"2024-03-06T07:25:37.353675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353675+0000","last_undegraded":"2024-03-06T07:25:37.353675+0000","last_fullsized":"2024-03-06T07:25:37.353675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348241+0000","last_change":"2024-03-06T07:25:37.348241+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348241+0000","last_undegraded":"2024-03-06T07:25:37.348241+0000","last_fullsized":"2024-03-06T07:25:37.348241+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347813+0000","last_change":"2024-03-06T07:25:37.347813+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347813+0000","last_undegraded":"2024-03-06T07:25:37.347813+0000","last_fullsized":"2024-03-06T07:25:37.347813+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351675+0000","last_change":"2024-03-06T07:25:37.351675+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351675+0000","last_undegraded":"2024-03-06T07:25:37.351675+0000","last_fullsized":"2024-03-06T07:25:37.351675+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350203+0000","last_change":"2024-03-06T07:25:37.350203+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350203+0000","last_undegraded":"2024-03-06T07:25:37.350203+0000","last_fullsized":"2024-03-06T07:25:37.350203+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348621+0000","last_change":"2024-03-06T07:25:37.348621+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348621+0000","last_undegraded":"2024-03-06T07:25:37.348621+0000","last_fullsized":"2024-03-06T07:25:37.348621+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353167+0000","last_change":"2024-03-06T07:25:37.353167+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353167+0000","last_undegraded":"2024-03-06T07:25:37.353167+0000","last_fullsized":"2024-03-06T07:25:37.353167+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361035+0000","last_change":"2024-03-06T07:25:37.360161+0000","last_active":"2024-03-06T07:25:37.361035+0000","last_peered":"2024-03-06T07:25:37.361035+0000","last_clean":"2024-03-06T07:25:37.361035+0000","last_became_active":"2024-03-06T07:25:37.359858+0000","last_became_peered":"2024-03-06T07:25:37.359858+0000","last_unstale":"2024-03-06T07:25:37.361035+0000","last_undegraded":"2024-03-06T07:25:37.361035+0000","last_fullsized":"2024-03-06T07:25:37.361035+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:11.304824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353595+0000","last_change":"2024-03-06T07:25:37.353595+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353595+0000","last_undegraded":"2024-03-06T07:25:37.353595+0000","last_fullsized":"2024-03-06T07:25:37.353595+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.366147+0000","last_change":"2024-03-06T07:25:37.361735+0000","last_active":"2024-03-06T07:25:37.366147+0000","last_peered":"2024-03-06T07:25:37.366147+0000","last_clean":"2024-03-06T07:25:37.366147+0000","last_became_active":"2024-03-06T07:25:37.361071+0000","last_became_peered":"2024-03-06T07:25:37.361071+0000","last_unstale":"2024-03-06T07:25:37.366147+0000","last_undegraded":"2024-03-06T07:25:37.366147+0000","last_fullsized":"2024-03-06T07:25:37.366147+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:38.535183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353456+0000","last_change":"2024-03-06T07:25:37.353456+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353456+0000","last_undegraded":"2024-03-06T07:25:37.353456+0000","last_fullsized":"2024-03-06T07:25:37.353456+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358849+0000","last_change":"2024-03-06T07:25:37.358208+0000","last_active":"2024-03-06T07:25:37.358849+0000","last_peered":"2024-03-06T07:25:37.358849+0000","last_clean":"2024-03-06T07:25:37.358849+0000","last_became_active":"2024-03-06T07:25:37.357941+0000","last_became_peered":"2024-03-06T07:25:37.357941+0000","last_unstale":"2024-03-06T07:25:37.358849+0000","last_undegraded":"2024-03-06T07:25:37.358849+0000","last_fullsized":"2024-03-06T07:25:37.358849+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:04.977160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353298+0000","last_change":"2024-03-06T07:25:37.353298+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353298+0000","last_undegraded":"2024-03-06T07:25:37.353298+0000","last_fullsized":"2024-03-06T07:25:37.353298+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358129+0000","last_change":"2024-03-06T07:25:37.357145+0000","last_active":"2024-03-06T07:25:37.358129+0000","last_peered":"2024-03-06T07:25:37.358129+0000","last_clean":"2024-03-06T07:25:37.358129+0000","last_became_active":"2024-03-06T07:25:37.356819+0000","last_became_peered":"2024-03-06T07:25:37.356819+0000","last_unstale":"2024-03-06T07:25:37.358129+0000","last_undegraded":"2024-03-06T07:25:37.358129+0000","last_fullsized":"2024-03-06T07:25:37.358129+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.694205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350553+0000","last_change":"2024-03-06T07:25:37.350553+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350553+0000","last_undegraded":"2024-03-06T07:25:37.350553+0000","last_fullsized":"2024-03-06T07:25:37.350553+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368484+0000","last_change":"2024-03-06T07:25:37.366475+0000","last_active":"2024-03-06T07:25:37.368484+0000","last_peered":"2024-03-06T07:25:37.368484+0000","last_clean":"2024-03-06T07:25:37.368484+0000","last_became_active":"2024-03-06T07:25:37.366203+0000","last_became_peered":"2024-03-06T07:25:37.366203+0000","last_unstale":"2024-03-06T07:25:37.368484+0000","last_undegraded":"2024-03-06T07:25:37.368484+0000","last_fullsized":"2024-03-06T07:25:37.368484+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:33.314274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349256+0000","last_change":"2024-03-06T07:25:37.349256+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349256+0000","last_undegraded":"2024-03-06T07:25:37.349256+0000","last_fullsized":"2024-03-06T07:25:37.349256+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364042+0000","last_change":"2024-03-06T07:25:37.362493+0000","last_active":"2024-03-06T07:25:37.364042+0000","last_peered":"2024-03-06T07:25:37.364042+0000","last_clean":"2024-03-06T07:25:37.364042+0000","last_became_active":"2024-03-06T07:25:37.362019+0000","last_became_peered":"2024-03-06T07:25:37.362019+0000","last_unstale":"2024-03-06T07:25:37.364042+0000","last_undegraded":"2024-03-06T07:25:37.364042+0000","last_fullsized":"2024-03-06T07:25:37.364042+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348628+0000","last_change":"2024-03-06T07:25:37.348628+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348628+0000","last_undegraded":"2024-03-06T07:25:37.348628+0000","last_fullsized":"2024-03-06T07:25:37.348628+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360769+0000","last_change":"2024-03-06T07:25:37.359237+0000","last_active":"2024-03-06T07:25:37.360769+0000","last_peered":"2024-03-06T07:25:37.360769+0000","last_clean":"2024-03-06T07:25:37.360769+0000","last_became_active":"2024-03-06T07:25:37.358927+0000","last_became_peered":"2024-03-06T07:25:37.358927+0000","last_unstale":"2024-03-06T07:25:37.360769+0000","last_undegraded":"2024-03-06T07:25:37.360769+0000","last_fullsized":"2024-03-06T07:25:37.360769+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:08.259298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.353004+0000","last_change":"2024-03-06T07:25:37.353004+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.353004+0000","last_undegraded":"2024-03-06T07:25:37.353004+0000","last_fullsized":"2024-03-06T07:25:37.353004+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368462+0000","last_change":"2024-03-06T07:25:37.367257+0000","last_active":"2024-03-06T07:25:37.368462+0000","last_peered":"2024-03-06T07:25:37.368462+0000","last_clean":"2024-03-06T07:25:37.368462+0000","last_became_active":"2024-03-06T07:25:37.366971+0000","last_became_peered":"2024-03-06T07:25:37.366971+0000","last_unstale":"2024-03-06T07:25:37.368462+0000","last_undegraded":"2024-03-06T07:25:37.368462+0000","last_fullsized":"2024-03-06T07:25:37.368462+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:31.083348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355752+0000","last_change":"2024-03-06T07:25:37.355752+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355752+0000","last_undegraded":"2024-03-06T07:25:37.355752+0000","last_fullsized":"2024-03-06T07:25:37.355752+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364834+0000","last_change":"2024-03-06T07:25:37.363334+0000","last_active":"2024-03-06T07:25:37.364834+0000","last_peered":"2024-03-06T07:25:37.364834+0000","last_clean":"2024-03-06T07:25:37.364834+0000","last_became_active":"2024-03-06T07:25:37.362631+0000","last_became_peered":"2024-03-06T07:25:37.362631+0000","last_unstale":"2024-03-06T07:25:37.364834+0000","last_undegraded":"2024-03-06T07:25:37.364834+0000","last_fullsized":"2024-03-06T07:25:37.364834+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352531+0000","last_change":"2024-03-06T07:25:37.352531+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352531+0000","last_undegraded":"2024-03-06T07:25:37.352531+0000","last_fullsized":"2024-03-06T07:25:37.352531+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368576+0000","last_change":"2024-03-06T07:25:37.367405+0000","last_active":"2024-03-06T07:25:37.368576+0000","last_peered":"2024-03-06T07:25:37.368576+0000","last_clean":"2024-03-06T07:25:37.368576+0000","last_became_active":"2024-03-06T07:25:37.367077+0000","last_became_peered":"2024-03-06T07:25:37.367077+0000","last_unstale":"2024-03-06T07:25:37.368576+0000","last_undegraded":"2024-03-06T07:25:37.368576+0000","last_fullsized":"2024-03-06T07:25:37.368576+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348006+0000","last_change":"2024-03-06T07:25:37.348006+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348006+0000","last_undegraded":"2024-03-06T07:25:37.348006+0000","last_fullsized":"2024-03-06T07:25:37.348006+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360396+0000","last_change":"2024-03-06T07:25:37.359803+0000","last_active":"2024-03-06T07:25:37.360396+0000","last_peered":"2024-03-06T07:25:37.360396+0000","last_clean":"2024-03-06T07:25:37.360396+0000","last_became_active":"2024-03-06T07:25:37.359509+0000","last_became_peered":"2024-03-06T07:25:37.359509+0000","last_unstale":"2024-03-06T07:25:37.360396+0000","last_undegraded":"2024-03-06T07:25:37.360396+0000","last_fullsized":"2024-03-06T07:25:37.360396+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:16.038727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349549+0000","last_change":"2024-03-06T07:25:37.349549+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349549+0000","last_undegraded":"2024-03-06T07:25:37.349549+0000","last_fullsized":"2024-03-06T07:25:37.349549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368085+0000","last_change":"2024-03-06T07:25:37.363650+0000","last_active":"2024-03-06T07:25:37.368085+0000","last_peered":"2024-03-06T07:25:37.368085+0000","last_clean":"2024-03-06T07:25:37.368085+0000","last_became_active":"2024-03-06T07:25:37.363319+0000","last_became_peered":"2024-03-06T07:25:37.363319+0000","last_unstale":"2024-03-06T07:25:37.368085+0000","last_undegraded":"2024-03-06T07:25:37.368085+0000","last_fullsized":"2024-03-06T07:25:37.368085+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:03.107757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.347979+0000","last_change":"2024-03-06T07:25:37.347979+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.347979+0000","last_undegraded":"2024-03-06T07:25:37.347979+0000","last_fullsized":"2024-03-06T07:25:37.347979+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359733+0000","last_change":"2024-03-06T07:25:37.357207+0000","last_active":"2024-03-06T07:25:37.359733+0000","last_peered":"2024-03-06T07:25:37.359733+0000","last_clean":"2024-03-06T07:25:37.359733+0000","last_became_active":"2024-03-06T07:25:37.356920+0000","last_became_peered":"2024-03-06T07:25:37.356920+0000","last_unstale":"2024-03-06T07:25:37.359733+0000","last_undegraded":"2024-03-06T07:25:37.359733+0000","last_fullsized":"2024-03-06T07:25:37.359733+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:20.703148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351780+0000","last_change":"2024-03-06T07:25:37.351780+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351780+0000","last_undegraded":"2024-03-06T07:25:37.351780+0000","last_fullsized":"2024-03-06T07:25:37.351780+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.361242+0000","last_change":"2024-03-06T07:25:37.360307+0000","last_active":"2024-03-06T07:25:37.361242+0000","last_peered":"2024-03-06T07:25:37.361242+0000","last_clean":"2024-03-06T07:25:37.361242+0000","last_became_active":"2024-03-06T07:25:37.360024+0000","last_became_peered":"2024-03-06T07:25:37.360024+0000","last_unstale":"2024-03-06T07:25:37.361242+0000","last_undegraded":"2024-03-06T07:25:37.361242+0000","last_fullsized":"2024-03-06T07:25:37.361242+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:13.020947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349007+0000","last_change":"2024-03-06T07:25:37.349007+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349007+0000","last_undegraded":"2024-03-06T07:25:37.349007+0000","last_fullsized":"2024-03-06T07:25:37.349007+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368184+0000","last_change":"2024-03-06T07:25:37.364965+0000","last_active":"2024-03-06T07:25:37.368184+0000","last_peered":"2024-03-06T07:25:37.368184+0000","last_clean":"2024-03-06T07:25:37.368184+0000","last_became_active":"2024-03-06T07:25:37.363889+0000","last_became_peered":"2024-03-06T07:25:37.363889+0000","last_unstale":"2024-03-06T07:25:37.368184+0000","last_undegraded":"2024-03-06T07:25:37.368184+0000","last_fullsized":"2024-03-06T07:25:37.368184+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:29.752076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352125+0000","last_change":"2024-03-06T07:25:37.352125+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352125+0000","last_undegraded":"2024-03-06T07:25:37.352125+0000","last_fullsized":"2024-03-06T07:25:37.352125+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364603+0000","last_change":"2024-03-06T07:25:37.364101+0000","last_active":"2024-03-06T07:25:37.364603+0000","last_peered":"2024-03-06T07:25:37.364603+0000","last_clean":"2024-03-06T07:25:37.364603+0000","last_became_active":"2024-03-06T07:25:37.363814+0000","last_became_peered":"2024-03-06T07:25:37.363814+0000","last_unstale":"2024-03-06T07:25:37.364603+0000","last_undegraded":"2024-03-06T07:25:37.364603+0000","last_fullsized":"2024-03-06T07:25:37.364603+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:08.141356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.356323+0000","last_change":"2024-03-06T07:25:37.356323+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.356323+0000","last_undegraded":"2024-03-06T07:25:37.356323+0000","last_fullsized":"2024-03-06T07:25:37.356323+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.359709+0000","last_change":"2024-03-06T07:25:37.358403+0000","last_active":"2024-03-06T07:25:37.359709+0000","last_peered":"2024-03-06T07:25:37.359709+0000","last_clean":"2024-03-06T07:25:37.359709+0000","last_became_active":"2024-03-06T07:25:37.358134+0000","last_became_peered":"2024-03-06T07:25:37.358134+0000","last_unstale":"2024-03-06T07:25:37.359709+0000","last_undegraded":"2024-03-06T07:25:37.359709+0000","last_fullsized":"2024-03-06T07:25:37.359709+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:52:55.829122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.352078+0000","last_change":"2024-03-06T07:25:37.352078+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.352078+0000","last_undegraded":"2024-03-06T07:25:37.352078+0000","last_fullsized":"2024-03-06T07:25:37.352078+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.358546+0000","last_change":"2024-03-06T07:25:37.355316+0000","last_active":"2024-03-06T07:25:37.358546+0000","last_peered":"2024-03-06T07:25:37.358546+0000","last_clean":"2024-03-06T07:25:37.358546+0000","last_became_active":"2024-03-06T07:25:37.355060+0000","last_became_peered":"2024-03-06T07:25:37.355060+0000","last_unstale":"2024-03-06T07:25:37.358546+0000","last_undegraded":"2024-03-06T07:25:37.358546+0000","last_fullsized":"2024-03-06T07:25:37.358546+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:27.708230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350295+0000","last_change":"2024-03-06T07:25:37.350295+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350295+0000","last_undegraded":"2024-03-06T07:25:37.350295+0000","last_fullsized":"2024-03-06T07:25:37.350295+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.357802+0000","last_change":"2024-03-06T07:25:37.354804+0000","last_active":"2024-03-06T07:25:37.357802+0000","last_peered":"2024-03-06T07:25:37.357802+0000","last_clean":"2024-03-06T07:25:37.357802+0000","last_became_active":"2024-03-06T07:25:37.354535+0000","last_became_peered":"2024-03-06T07:25:37.354535+0000","last_unstale":"2024-03-06T07:25:37.357802+0000","last_undegraded":"2024-03-06T07:25:37.357802+0000","last_fullsized":"2024-03-06T07:25:37.357802+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:11.747252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351130+0000","last_change":"2024-03-06T07:25:37.351130+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351130+0000","last_undegraded":"2024-03-06T07:25:37.351130+0000","last_fullsized":"2024-03-06T07:25:37.351130+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.367951+0000","last_change":"2024-03-06T07:25:37.362425+0000","last_active":"2024-03-06T07:25:37.367951+0000","last_peered":"2024-03-06T07:25:37.367951+0000","last_clean":"2024-03-06T07:25:37.367951+0000","last_became_active":"2024-03-06T07:25:37.362172+0000","last_became_peered":"2024-03-06T07:25:37.362172+0000","last_unstale":"2024-03-06T07:25:37.367951+0000","last_undegraded":"2024-03-06T07:25:37.367951+0000","last_fullsized":"2024-03-06T07:25:37.367951+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:46.062578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350611+0000","last_change":"2024-03-06T07:25:37.350611+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350611+0000","last_undegraded":"2024-03-06T07:25:37.350611+0000","last_fullsized":"2024-03-06T07:25:37.350611+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356209+0000","last_change":"2024-03-06T07:25:37.354863+0000","last_active":"2024-03-06T07:25:37.356209+0000","last_peered":"2024-03-06T07:25:37.356209+0000","last_clean":"2024-03-06T07:25:37.356209+0000","last_became_active":"2024-03-06T07:25:37.354465+0000","last_became_peered":"2024-03-06T07:25:37.354465+0000","last_unstale":"2024-03-06T07:25:37.356209+0000","last_undegraded":"2024-03-06T07:25:37.356209+0000","last_fullsized":"2024-03-06T07:25:37.356209+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:33.951539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349868+0000","last_change":"2024-03-06T07:25:37.349868+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349868+0000","last_undegraded":"2024-03-06T07:25:37.349868+0000","last_fullsized":"2024-03-06T07:25:37.349868+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362855+0000","last_change":"2024-03-06T07:25:37.361572+0000","last_active":"2024-03-06T07:25:37.362855+0000","last_peered":"2024-03-06T07:25:37.362855+0000","last_clean":"2024-03-06T07:25:37.362855+0000","last_became_active":"2024-03-06T07:25:37.361271+0000","last_became_peered":"2024-03-06T07:25:37.361271+0000","last_unstale":"2024-03-06T07:25:37.362855+0000","last_undegraded":"2024-03-06T07:25:37.362855+0000","last_fullsized":"2024-03-06T07:25:37.362855+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:13.032326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351680+0000","last_change":"2024-03-06T07:25:37.351680+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351680+0000","last_undegraded":"2024-03-06T07:25:37.351680+0000","last_fullsized":"2024-03-06T07:25:37.351680+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.360711+0000","last_change":"2024-03-06T07:25:37.359314+0000","last_active":"2024-03-06T07:25:37.360711+0000","last_peered":"2024-03-06T07:25:37.360711+0000","last_clean":"2024-03-06T07:25:37.360711+0000","last_became_active":"2024-03-06T07:25:37.359008+0000","last_became_peered":"2024-03-06T07:25:37.359008+0000","last_unstale":"2024-03-06T07:25:37.360711+0000","last_undegraded":"2024-03-06T07:25:37.360711+0000","last_fullsized":"2024-03-06T07:25:37.360711+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:07.137677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351711+0000","last_change":"2024-03-06T07:25:37.351711+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351711+0000","last_undegraded":"2024-03-06T07:25:37.351711+0000","last_fullsized":"2024-03-06T07:25:37.351711+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368155+0000","last_change":"2024-03-06T07:25:37.364253+0000","last_active":"2024-03-06T07:25:37.368155+0000","last_peered":"2024-03-06T07:25:37.368155+0000","last_clean":"2024-03-06T07:25:37.368155+0000","last_became_active":"2024-03-06T07:25:37.363933+0000","last_became_peered":"2024-03-06T07:25:37.363933+0000","last_unstale":"2024-03-06T07:25:37.368155+0000","last_undegraded":"2024-03-06T07:25:37.368155+0000","last_fullsized":"2024-03-06T07:25:37.368155+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.355920+0000","last_change":"2024-03-06T07:25:37.355920+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.355920+0000","last_undegraded":"2024-03-06T07:25:37.355920+0000","last_fullsized":"2024-03-06T07:25:37.355920+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364944+0000","last_change":"2024-03-06T07:25:37.364443+0000","last_active":"2024-03-06T07:25:37.364944+0000","last_peered":"2024-03-06T07:25:37.364944+0000","last_clean":"2024-03-06T07:25:37.364944+0000","last_became_active":"2024-03-06T07:25:37.364093+0000","last_became_peered":"2024-03-06T07:25:37.364093+0000","last_unstale":"2024-03-06T07:25:37.364944+0000","last_undegraded":"2024-03-06T07:25:37.364944+0000","last_fullsized":"2024-03-06T07:25:37.364944+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:01.483203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351304+0000","last_change":"2024-03-06T07:25:37.351304+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351304+0000","last_undegraded":"2024-03-06T07:25:37.351304+0000","last_fullsized":"2024-03-06T07:25:37.351304+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.363191+0000","last_change":"2024-03-06T07:25:37.362488+0000","last_active":"2024-03-06T07:25:37.363191+0000","last_peered":"2024-03-06T07:25:37.363191+0000","last_clean":"2024-03-06T07:25:37.363191+0000","last_became_active":"2024-03-06T07:25:37.362203+0000","last_became_peered":"2024-03-06T07:25:37.362203+0000","last_unstale":"2024-03-06T07:25:37.363191+0000","last_undegraded":"2024-03-06T07:25:37.363191+0000","last_fullsized":"2024-03-06T07:25:37.363191+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.309117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348099+0000","last_change":"2024-03-06T07:25:37.348099+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348099+0000","last_undegraded":"2024-03-06T07:25:37.348099+0000","last_fullsized":"2024-03-06T07:25:37.348099+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368386+0000","last_change":"2024-03-06T07:25:37.366204+0000","last_active":"2024-03-06T07:25:37.368386+0000","last_peered":"2024-03-06T07:25:37.368386+0000","last_clean":"2024-03-06T07:25:37.368386+0000","last_became_active":"2024-03-06T07:25:37.365751+0000","last_became_peered":"2024-03-06T07:25:37.365751+0000","last_unstale":"2024-03-06T07:25:37.368386+0000","last_undegraded":"2024-03-06T07:25:37.368386+0000","last_fullsized":"2024-03-06T07:25:37.368386+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:40.654060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.350714+0000","last_change":"2024-03-06T07:25:37.350714+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.350714+0000","last_undegraded":"2024-03-06T07:25:37.350714+0000","last_fullsized":"2024-03-06T07:25:37.350714+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368066+0000","last_change":"2024-03-06T07:25:37.363581+0000","last_active":"2024-03-06T07:25:37.368066+0000","last_peered":"2024-03-06T07:25:37.368066+0000","last_clean":"2024-03-06T07:25:37.368066+0000","last_became_active":"2024-03-06T07:25:37.362991+0000","last_became_peered":"2024-03-06T07:25:37.362991+0000","last_unstale":"2024-03-06T07:25:37.368066+0000","last_undegraded":"2024-03-06T07:25:37.368066+0000","last_fullsized":"2024-03-06T07:25:37.368066+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:25.392712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.351198+0000","last_change":"2024-03-06T07:25:37.351198+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.351198+0000","last_undegraded":"2024-03-06T07:25:37.351198+0000","last_fullsized":"2024-03-06T07:25:37.351198+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.362479+0000","last_change":"2024-03-06T07:25:37.361352+0000","last_active":"2024-03-06T07:25:37.362479+0000","last_peered":"2024-03-06T07:25:37.362479+0000","last_clean":"2024-03-06T07:25:37.362479+0000","last_became_active":"2024-03-06T07:25:37.361059+0000","last_became_peered":"2024-03-06T07:25:37.361059+0000","last_unstale":"2024-03-06T07:25:37.362479+0000","last_undegraded":"2024-03-06T07:25:37.362479+0000","last_fullsized":"2024-03-06T07:25:37.362479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:40:55.692201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.348043+0000","last_change":"2024-03-06T07:25:37.348043+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.348043+0000","last_undegraded":"2024-03-06T07:25:37.348043+0000","last_fullsized":"2024-03-06T07:25:37.348043+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368916+0000","last_change":"2024-03-06T07:25:37.368239+0000","last_active":"2024-03-06T07:25:37.368916+0000","last_peered":"2024-03-06T07:25:37.368916+0000","last_clean":"2024-03-06T07:25:37.368916+0000","last_became_active":"2024-03-06T07:25:37.367979+0000","last_became_peered":"2024-03-06T07:25:37.367979+0000","last_unstale":"2024-03-06T07:25:37.368916+0000","last_undegraded":"2024-03-06T07:25:37.368916+0000","last_fullsized":"2024-03-06T07:25:37.368916+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:03.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.349897+0000","last_change":"2024-03-06T07:25:37.349897+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.349897+0000","last_undegraded":"2024-03-06T07:25:37.349897+0000","last_fullsized":"2024-03-06T07:25:37.349897+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.368412+0000","last_change":"2024-03-06T07:25:37.366881+0000","last_active":"2024-03-06T07:25:37.368412+0000","last_peered":"2024-03-06T07:25:37.368412+0000","last_clean":"2024-03-06T07:25:37.368412+0000","last_became_active":"2024-03-06T07:25:37.366588+0000","last_became_peered":"2024-03-06T07:25:37.366588+0000","last_unstale":"2024-03-06T07:25:37.368412+0000","last_undegraded":"2024-03-06T07:25:37.368412+0000","last_fullsized":"2024-03-06T07:25:37.368412+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:46.200362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":1,"reported_epoch":180,"state":"creating+peering","last_fresh":"2024-03-06T07:25:37.354567+0000","last_change":"2024-03-06T07:25:37.354567+0000","last_active":"2024-03-06T07:25:37.341311+0000","last_peered":"2024-03-06T07:25:37.341311+0000","last_clean":"2024-03-06T07:25:37.341311+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:37.354567+0000","last_undegraded":"2024-03-06T07:25:37.354567+0000","last_fullsized":"2024-03-06T07:25:37.354567+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":9,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.364364+0000","last_change":"2024-03-06T07:25:37.359284+0000","last_active":"2024-03-06T07:25:37.364364+0000","last_peered":"2024-03-06T07:25:37.364364+0000","last_clean":"2024-03-06T07:25:37.364364+0000","last_became_active":"2024-03-06T07:25:37.358847+0000","last_became_peered":"2024-03-06T07:25:37.358847+0000","last_unstale":"2024-03-06T07:25:37.364364+0000","last_undegraded":"2024-03-06T07:25:37.364364+0000","last_fullsized":"2024-03-06T07:25:37.364364+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:43.913135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":472,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-06T07:25:37.356621+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:25:37.356621+0000","last_peered":"2024-03-06T07:25:37.356621+0000","last_clean":"2024-03-06T07:25:37.356621+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:25:37.356621+0000","last_undegraded":"2024-03-06T07:25:37.356621+0000","last_fullsized":"2024-03-06T07:25:37.356621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33464,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338376,"statfs":{"total":96636764160,"available":96602497024,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33416,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338424,"statfs":{"total":96636764160,"available":96602546176,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33412,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338428,"statfs":{"total":96636764160,"available":96602550272,"internally_reserved":0,"allocated":2281472,"data_stored":1106030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33396,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338444,"statfs":{"total":96636764160,"available":96602566656,"internally_reserved":0,"allocated":2269184,"data_stored":1099363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574941,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37624,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334216,"statfs":{"total":96636764160,"available":96598237184,"internally_reserved":0,"allocated":2334720,"data_stored":1689731,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:43.628 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:25:43.627+0000 7f06fe5f2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:45.597 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:45.600 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:45.728 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:45.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.837+0000 7fa6d0cb5640 1 Processor -- start 2024-03-06T07:25:45.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.837+0000 7fa6d0cb5640 1 -- start start 2024-03-06T07:25:45.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6d0cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 0x7fa6cc156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6d0cb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc153a60 con 0x7fa6cc0c1320 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6d0cb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc153d40 con 0x7fa6cc150580 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6d0cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc06c2c0 con 0x7fa6cc0c6000 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c3fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa6cc0c1320 0x7fa6cc1566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58492/0 (socket says 172.21.15.94:58492) 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c3fff640 1 -- 172.21.15.94:0/3471645663 learned_addr learned my addr 172.21.15.94:0/3471645663 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6cad9d640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 0x7fa6cc156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979329041 0 0) 0x7fa6cc153a60 con 0x7fa6cc0c1320 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b8003520 con 0x7fa6cc0c1320 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687195625 0 0) 0x7fa6cc153d40 con 0x7fa6cc150580 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc153a60 con 0x7fa6cc150580 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740662701 0 0) 0x7fa6b8003520 con 0x7fa6cc0c1320 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 msgr2=0x7fa6cc156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 0x7fa6cc156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 >> v1:172.21.15.145:6789/0 conn(0x7fa6cc150580 legacy=0x7fa6cc06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc06c5a0 con 0x7fa6cc0c1320 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fa6b4001670 con 0x7fa6cc0c1320 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 >> v1:172.21.15.94:6789/0 conn(0x7fa6cc0c1320 legacy=0x7fa6cc1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6b8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 0x7fa6b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8004d40 0x7fa6b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc06c2c0 con 0x7fa6b8003b70 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b800bac0 con 0x7fa6b8004d40 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b800bcc0 con 0x7fa6cc0c6000 2024-03-06T07:25:45.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.838+0000 7fa6ca59c640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6b8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6c3fff640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 0x7fa6b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6cad9d640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8004d40 0x7fa6b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6ca59c640 1 -- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 msgr2=0x7fa6b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6ca59c640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc0c6000 0x7fa6b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6ca59c640 1 -- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8004d40 msgr2=0x7fa6b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6ca59c640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6b8004d40 0x7fa6b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6ca59c640 1 -- 172.21.15.94:0/3471645663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b8003520 con 0x7fa6b8003b70 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.839+0000 7fa6ca59c640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6b8003f70 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0018d0 tx=0x7fa6b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946aa586f98a51ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc037070 con 0x7fa6b8003b70 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc0052d0 con 0x7fa6b8003b70 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6c9d9b640 1 -- 172.21.15.94:0/3471645663 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0014f0 con 0x7fa6b8003b70 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/3471645663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 msgr2=0x7fa6b8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6d0cb5640 1 --2- 172.21.15.94:0/3471645663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6b8003f70 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0018d0 tx=0x7fa6b8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:45.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6cad9d640 1 -- 172.21.15.94:0/3471645663 reap_dead start 2024-03-06T07:25:45.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/3471645663 shutdown_connections 2024-03-06T07:25:45.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/3471645663 >> 172.21.15.94:0/3471645663 conn(0x7fa6cc0c6710 msgr2=0x7fa6cc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/3471645663 shutdown_connections 2024-03-06T07:25:45.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.840+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/3471645663 wait complete. 2024-03-06T07:25:45.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 Processor -- start 2024-03-06T07:25:45.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 -- start start 2024-03-06T07:25:45.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6cc148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6cc0c3650 0x7fa6cc0c1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc154340 0x7fa6cc0c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc06e5a0 con 0x7fa6b8003b70 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc06e420 con 0x7fa6cc0c3650 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6d0cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc0c0c40 con 0x7fa6cc154340 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6ca59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6cc148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6ca59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6cc148ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41914/0 (socket says 172.21.15.94:41914) 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6ca59c640 1 -- 172.21.15.94:0/2757336466 learned_addr learned my addr 172.21.15.94:0/2757336466 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6cad9d640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc154340 0x7fa6cc0c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.842+0000 7fa6c3fff640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6cc0c3650 0x7fa6cc0c1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6ca59c640 1 -- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc154340 msgr2=0x7fa6cc0c21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6ca59c640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc154340 0x7fa6cc0c21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6ca59c640 1 -- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6cc0c3650 msgr2=0x7fa6cc0c1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6ca59c640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6cc0c3650 0x7fa6cc0c1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6ca59c640 1 -- 172.21.15.94:0/2757336466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc1512e0 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6ca59c640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6cc148ee0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0018d0 tx=0x7fa6bc0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6c2ffd640 1 -- 172.21.15.94:0/2757336466 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc037070 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6c2ffd640 1 -- 172.21.15.94:0/2757336466 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc038040 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc147260 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6c2ffd640 1 -- 172.21.15.94:0/2757336466 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0014f0 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.843+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc14c960 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.844+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc146b70 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.845+0000 7fa6c2ffd640 1 -- 172.21.15.94:0/2757336466 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6bc0052d0 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.845+0000 7fa6c2ffd640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.845+0000 7fa6c2ffd640 1 -- 172.21.15.94:0/2757336466 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa6bc0ab9d0 con 0x7fa6b8003b70 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.845+0000 7fa6c3fff640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.846+0000 7fa6c3fff640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0a6130 tx=0x7fa6b4001e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:45.849+0000 7fa6c2ffd640 1 -- 172.21.15.94:0/2757336466 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6cc1512e0 con 0x7fa6b8003b70 2024-03-06T07:25:46.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.018+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa6cc04f330 con 0x7fa6ac06b4d0 2024-03-06T07:25:46.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.024+0000 7fa6c2ffd640 1 -- 172.21.15.94:0/2757336466 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252554 (secure 0 0 0) 0x7fa6cc04f330 con 0x7fa6ac06b4d0 2024-03-06T07:25:46.028 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:46.029 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa6ac06b4d0 msgr2=0x7fa6ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0a6130 tx=0x7fa6b4001e20 comp rx=0 tx=0).stop 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 msgr2=0x7fa6cc148ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6cc148ee0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0018d0 tx=0x7fa6bc0012d0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 shutdown_connections 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6cc154340 0x7fa6cc0c21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6cc0c3650 0x7fa6cc0c1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 --2- 172.21.15.94:0/2757336466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6b8003b70 0x7fa6cc148ee0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 >> 172.21.15.94:0/2757336466 conn(0x7fa6cc0c6710 msgr2=0x7fa6cc0c4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 shutdown_connections 2024-03-06T07:25:46.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.027+0000 7fa6d0cb5640 1 -- 172.21.15.94:0/2757336466 wait complete. 2024-03-06T07:25:46.040 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-03-06T07:25:44.733420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":271848,"kb_used_data":18208,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754702872,"statfs":{"total":773094113280,"available":772815740928,"internally_reserved":0,"allocated":18644992,"data_stored":10144668,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.381614"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669112+0000","last_change":"2024-03-06T07:25:38.385134+0000","last_active":"2024-03-06T07:25:39.669112+0000","last_peered":"2024-03-06T07:25:39.669112+0000","last_clean":"2024-03-06T07:25:39.669112+0000","last_became_active":"2024-03-06T07:25:38.384815+0000","last_became_peered":"2024-03-06T07:25:38.384815+0000","last_unstale":"2024-03-06T07:25:39.669112+0000","last_undegraded":"2024-03-06T07:25:39.669112+0000","last_fullsized":"2024-03-06T07:25:39.669112+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:41.133021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.375427+0000","last_change":"2024-03-06T07:25:38.373861+0000","last_active":"2024-03-06T07:25:39.375427+0000","last_peered":"2024-03-06T07:25:39.375427+0000","last_clean":"2024-03-06T07:25:39.375427+0000","last_became_active":"2024-03-06T07:25:38.373598+0000","last_became_peered":"2024-03-06T07:25:38.373598+0000","last_unstale":"2024-03-06T07:25:39.375427+0000","last_undegraded":"2024-03-06T07:25:39.375427+0000","last_fullsized":"2024-03-06T07:25:39.375427+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:21.026425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668475+0000","last_change":"2024-03-06T07:25:38.374436+0000","last_active":"2024-03-06T07:25:39.668475+0000","last_peered":"2024-03-06T07:25:39.668475+0000","last_clean":"2024-03-06T07:25:39.668475+0000","last_became_active":"2024-03-06T07:25:38.374191+0000","last_became_peered":"2024-03-06T07:25:38.374191+0000","last_unstale":"2024-03-06T07:25:39.668475+0000","last_undegraded":"2024-03-06T07:25:39.668475+0000","last_fullsized":"2024-03-06T07:25:39.668475+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:01.531294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.376867+0000","last_change":"2024-03-06T07:25:38.388213+0000","last_active":"2024-03-06T07:25:39.376867+0000","last_peered":"2024-03-06T07:25:39.376867+0000","last_clean":"2024-03-06T07:25:39.376867+0000","last_became_active":"2024-03-06T07:25:38.387932+0000","last_became_peered":"2024-03-06T07:25:38.387932+0000","last_unstale":"2024-03-06T07:25:39.376867+0000","last_undegraded":"2024-03-06T07:25:39.376867+0000","last_fullsized":"2024-03-06T07:25:39.376867+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:43.130329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668949+0000","last_change":"2024-03-06T07:25:38.384938+0000","last_active":"2024-03-06T07:25:39.668949+0000","last_peered":"2024-03-06T07:25:39.668949+0000","last_clean":"2024-03-06T07:25:39.668949+0000","last_became_active":"2024-03-06T07:25:38.384672+0000","last_became_peered":"2024-03-06T07:25:38.384672+0000","last_unstale":"2024-03-06T07:25:39.668949+0000","last_undegraded":"2024-03-06T07:25:39.668949+0000","last_fullsized":"2024-03-06T07:25:39.668949+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:12.495576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668899+0000","last_change":"2024-03-06T07:25:38.384842+0000","last_active":"2024-03-06T07:25:39.668899+0000","last_peered":"2024-03-06T07:25:39.668899+0000","last_clean":"2024-03-06T07:25:39.668899+0000","last_became_active":"2024-03-06T07:25:38.384598+0000","last_became_peered":"2024-03-06T07:25:38.384598+0000","last_unstale":"2024-03-06T07:25:39.668899+0000","last_undegraded":"2024-03-06T07:25:39.668899+0000","last_fullsized":"2024-03-06T07:25:39.668899+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:37.553031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668494+0000","last_change":"2024-03-06T07:25:38.385750+0000","last_active":"2024-03-06T07:25:39.668494+0000","last_peered":"2024-03-06T07:25:39.668494+0000","last_clean":"2024-03-06T07:25:39.668494+0000","last_became_active":"2024-03-06T07:25:38.385123+0000","last_became_peered":"2024-03-06T07:25:38.385123+0000","last_unstale":"2024-03-06T07:25:39.668494+0000","last_undegraded":"2024-03-06T07:25:39.668494+0000","last_fullsized":"2024-03-06T07:25:39.668494+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:42.000859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668416+0000","last_change":"2024-03-06T07:25:38.392716+0000","last_active":"2024-03-06T07:25:39.668416+0000","last_peered":"2024-03-06T07:25:39.668416+0000","last_clean":"2024-03-06T07:25:39.668416+0000","last_became_active":"2024-03-06T07:25:38.392377+0000","last_became_peered":"2024-03-06T07:25:38.392377+0000","last_unstale":"2024-03-06T07:25:39.668416+0000","last_undegraded":"2024-03-06T07:25:39.668416+0000","last_fullsized":"2024-03-06T07:25:39.668416+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:56.695800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668832+0000","last_change":"2024-03-06T07:25:38.379443+0000","last_active":"2024-03-06T07:25:39.668832+0000","last_peered":"2024-03-06T07:25:39.668832+0000","last_clean":"2024-03-06T07:25:39.668832+0000","last_became_active":"2024-03-06T07:25:38.379168+0000","last_became_peered":"2024-03-06T07:25:38.379168+0000","last_unstale":"2024-03-06T07:25:39.668832+0000","last_undegraded":"2024-03-06T07:25:39.668832+0000","last_fullsized":"2024-03-06T07:25:39.668832+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:47.064173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668407+0000","last_change":"2024-03-06T07:25:38.379051+0000","last_active":"2024-03-06T07:25:39.668407+0000","last_peered":"2024-03-06T07:25:39.668407+0000","last_clean":"2024-03-06T07:25:39.668407+0000","last_became_active":"2024-03-06T07:25:38.378755+0000","last_became_peered":"2024-03-06T07:25:38.378755+0000","last_unstale":"2024-03-06T07:25:39.668407+0000","last_undegraded":"2024-03-06T07:25:39.668407+0000","last_fullsized":"2024-03-06T07:25:39.668407+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:24.283370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377150+0000","last_change":"2024-03-06T07:25:38.377410+0000","last_active":"2024-03-06T07:25:39.377150+0000","last_peered":"2024-03-06T07:25:39.377150+0000","last_clean":"2024-03-06T07:25:39.377150+0000","last_became_active":"2024-03-06T07:25:38.377146+0000","last_became_peered":"2024-03-06T07:25:38.377146+0000","last_unstale":"2024-03-06T07:25:39.377150+0000","last_undegraded":"2024-03-06T07:25:39.377150+0000","last_fullsized":"2024-03-06T07:25:39.377150+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:22.275673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378547+0000","last_change":"2024-03-06T07:25:38.389905+0000","last_active":"2024-03-06T07:25:39.378547+0000","last_peered":"2024-03-06T07:25:39.378547+0000","last_clean":"2024-03-06T07:25:39.378547+0000","last_became_active":"2024-03-06T07:25:38.387616+0000","last_became_peered":"2024-03-06T07:25:38.387616+0000","last_unstale":"2024-03-06T07:25:39.378547+0000","last_undegraded":"2024-03-06T07:25:39.378547+0000","last_fullsized":"2024-03-06T07:25:39.378547+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:45.104907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668586+0000","last_change":"2024-03-06T07:25:38.384227+0000","last_active":"2024-03-06T07:25:39.668586+0000","last_peered":"2024-03-06T07:25:39.668586+0000","last_clean":"2024-03-06T07:25:39.668586+0000","last_became_active":"2024-03-06T07:25:38.383984+0000","last_became_peered":"2024-03-06T07:25:38.383984+0000","last_unstale":"2024-03-06T07:25:39.668586+0000","last_undegraded":"2024-03-06T07:25:39.668586+0000","last_fullsized":"2024-03-06T07:25:39.668586+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:02.484798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377108+0000","last_change":"2024-03-06T07:25:38.389146+0000","last_active":"2024-03-06T07:25:39.377108+0000","last_peered":"2024-03-06T07:25:39.377108+0000","last_clean":"2024-03-06T07:25:39.377108+0000","last_became_active":"2024-03-06T07:25:38.388208+0000","last_became_peered":"2024-03-06T07:25:38.388208+0000","last_unstale":"2024-03-06T07:25:39.377108+0000","last_undegraded":"2024-03-06T07:25:39.377108+0000","last_fullsized":"2024-03-06T07:25:39.377108+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:17.040322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644126+0000","last_change":"2024-03-06T07:25:38.376370+0000","last_active":"2024-03-06T07:25:39.644126+0000","last_peered":"2024-03-06T07:25:39.644126+0000","last_clean":"2024-03-06T07:25:39.644126+0000","last_became_active":"2024-03-06T07:25:38.376106+0000","last_became_peered":"2024-03-06T07:25:38.376106+0000","last_unstale":"2024-03-06T07:25:39.644126+0000","last_undegraded":"2024-03-06T07:25:39.644126+0000","last_fullsized":"2024-03-06T07:25:39.644126+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:41.133021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668448+0000","last_change":"2024-03-06T07:25:38.381229+0000","last_active":"2024-03-06T07:25:39.668448+0000","last_peered":"2024-03-06T07:25:39.668448+0000","last_clean":"2024-03-06T07:25:39.668448+0000","last_became_active":"2024-03-06T07:25:38.380804+0000","last_became_peered":"2024-03-06T07:25:38.380804+0000","last_unstale":"2024-03-06T07:25:39.668448+0000","last_undegraded":"2024-03-06T07:25:39.668448+0000","last_fullsized":"2024-03-06T07:25:39.668448+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:14.033921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378157+0000","last_change":"2024-03-06T07:25:38.390211+0000","last_active":"2024-03-06T07:25:39.378157+0000","last_peered":"2024-03-06T07:25:39.378157+0000","last_clean":"2024-03-06T07:25:39.378157+0000","last_became_active":"2024-03-06T07:25:38.389613+0000","last_became_peered":"2024-03-06T07:25:38.389613+0000","last_unstale":"2024-03-06T07:25:39.378157+0000","last_undegraded":"2024-03-06T07:25:39.378157+0000","last_fullsized":"2024-03-06T07:25:39.378157+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:06.093299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668544+0000","last_change":"2024-03-06T07:25:38.386137+0000","last_active":"2024-03-06T07:25:39.668544+0000","last_peered":"2024-03-06T07:25:39.668544+0000","last_clean":"2024-03-06T07:25:39.668544+0000","last_became_active":"2024-03-06T07:25:38.385876+0000","last_became_peered":"2024-03-06T07:25:38.385876+0000","last_unstale":"2024-03-06T07:25:39.668544+0000","last_undegraded":"2024-03-06T07:25:39.668544+0000","last_fullsized":"2024-03-06T07:25:39.668544+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:53.695724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669201+0000","last_change":"2024-03-06T07:25:38.385492+0000","last_active":"2024-03-06T07:25:39.669201+0000","last_peered":"2024-03-06T07:25:39.669201+0000","last_clean":"2024-03-06T07:25:39.669201+0000","last_became_active":"2024-03-06T07:25:38.384854+0000","last_became_peered":"2024-03-06T07:25:38.384854+0000","last_unstale":"2024-03-06T07:25:39.669201+0000","last_undegraded":"2024-03-06T07:25:39.669201+0000","last_fullsized":"2024-03-06T07:25:39.669201+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:19.493328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668768+0000","last_change":"2024-03-06T07:25:38.383549+0000","last_active":"2024-03-06T07:25:39.668768+0000","last_peered":"2024-03-06T07:25:39.668768+0000","last_clean":"2024-03-06T07:25:39.668768+0000","last_became_active":"2024-03-06T07:25:38.383266+0000","last_became_peered":"2024-03-06T07:25:38.383266+0000","last_unstale":"2024-03-06T07:25:39.668768+0000","last_undegraded":"2024-03-06T07:25:39.668768+0000","last_fullsized":"2024-03-06T07:25:39.668768+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:58.968264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668791+0000","last_change":"2024-03-06T07:25:38.375067+0000","last_active":"2024-03-06T07:25:39.668791+0000","last_peered":"2024-03-06T07:25:39.668791+0000","last_clean":"2024-03-06T07:25:39.668791+0000","last_became_active":"2024-03-06T07:25:38.374794+0000","last_became_peered":"2024-03-06T07:25:38.374794+0000","last_unstale":"2024-03-06T07:25:39.668791+0000","last_undegraded":"2024-03-06T07:25:39.668791+0000","last_fullsized":"2024-03-06T07:25:39.668791+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:00.635952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644179+0000","last_change":"2024-03-06T07:25:38.386358+0000","last_active":"2024-03-06T07:25:39.644179+0000","last_peered":"2024-03-06T07:25:39.644179+0000","last_clean":"2024-03-06T07:25:39.644179+0000","last_became_active":"2024-03-06T07:25:38.386072+0000","last_became_peered":"2024-03-06T07:25:38.386072+0000","last_unstale":"2024-03-06T07:25:39.644179+0000","last_undegraded":"2024-03-06T07:25:39.644179+0000","last_fullsized":"2024-03-06T07:25:39.644179+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:03.304412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668227+0000","last_change":"2024-03-06T07:25:38.377502+0000","last_active":"2024-03-06T07:25:39.668227+0000","last_peered":"2024-03-06T07:25:39.668227+0000","last_clean":"2024-03-06T07:25:39.668227+0000","last_became_active":"2024-03-06T07:25:38.377184+0000","last_became_peered":"2024-03-06T07:25:38.377184+0000","last_unstale":"2024-03-06T07:25:39.668227+0000","last_undegraded":"2024-03-06T07:25:39.668227+0000","last_fullsized":"2024-03-06T07:25:39.668227+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:26:02.911394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668412+0000","last_change":"2024-03-06T07:25:38.384299+0000","last_active":"2024-03-06T07:25:39.668412+0000","last_peered":"2024-03-06T07:25:39.668412+0000","last_clean":"2024-03-06T07:25:39.668412+0000","last_became_active":"2024-03-06T07:25:38.384031+0000","last_became_peered":"2024-03-06T07:25:38.384031+0000","last_unstale":"2024-03-06T07:25:39.668412+0000","last_undegraded":"2024-03-06T07:25:39.668412+0000","last_fullsized":"2024-03-06T07:25:39.668412+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:52.137438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669131+0000","last_change":"2024-03-06T07:25:38.384441+0000","last_active":"2024-03-06T07:25:39.669131+0000","last_peered":"2024-03-06T07:25:39.669131+0000","last_clean":"2024-03-06T07:25:39.669131+0000","last_became_active":"2024-03-06T07:25:38.383802+0000","last_became_peered":"2024-03-06T07:25:38.383802+0000","last_unstale":"2024-03-06T07:25:39.669131+0000","last_undegraded":"2024-03-06T07:25:39.669131+0000","last_fullsized":"2024-03-06T07:25:39.669131+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:58.284018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668785+0000","last_change":"2024-03-06T07:25:38.389481+0000","last_active":"2024-03-06T07:25:39.668785+0000","last_peered":"2024-03-06T07:25:39.668785+0000","last_clean":"2024-03-06T07:25:39.668785+0000","last_became_active":"2024-03-06T07:25:38.389189+0000","last_became_peered":"2024-03-06T07:25:38.389189+0000","last_unstale":"2024-03-06T07:25:39.668785+0000","last_undegraded":"2024-03-06T07:25:39.668785+0000","last_fullsized":"2024-03-06T07:25:39.668785+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:02.791255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377293+0000","last_change":"2024-03-06T07:25:38.382066+0000","last_active":"2024-03-06T07:25:39.377293+0000","last_peered":"2024-03-06T07:25:39.377293+0000","last_clean":"2024-03-06T07:25:39.377293+0000","last_became_active":"2024-03-06T07:25:38.381803+0000","last_became_peered":"2024-03-06T07:25:38.381803+0000","last_unstale":"2024-03-06T07:25:39.377293+0000","last_undegraded":"2024-03-06T07:25:39.377293+0000","last_fullsized":"2024-03-06T07:25:39.377293+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:17.583650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378600+0000","last_change":"2024-03-06T07:25:38.381171+0000","last_active":"2024-03-06T07:25:39.378600+0000","last_peered":"2024-03-06T07:25:39.378600+0000","last_clean":"2024-03-06T07:25:39.378600+0000","last_became_active":"2024-03-06T07:25:38.380919+0000","last_became_peered":"2024-03-06T07:25:38.380919+0000","last_unstale":"2024-03-06T07:25:39.378600+0000","last_undegraded":"2024-03-06T07:25:39.378600+0000","last_fullsized":"2024-03-06T07:25:39.378600+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:46:59.505639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.375528+0000","last_change":"2024-03-06T07:25:38.384100+0000","last_active":"2024-03-06T07:25:39.375528+0000","last_peered":"2024-03-06T07:25:39.375528+0000","last_clean":"2024-03-06T07:25:39.375528+0000","last_became_active":"2024-03-06T07:25:38.383468+0000","last_became_peered":"2024-03-06T07:25:38.383468+0000","last_unstale":"2024-03-06T07:25:39.375528+0000","last_undegraded":"2024-03-06T07:25:39.375528+0000","last_fullsized":"2024-03-06T07:25:39.375528+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:19.265668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669364+0000","last_change":"2024-03-06T07:25:38.388709+0000","last_active":"2024-03-06T07:25:39.669364+0000","last_peered":"2024-03-06T07:25:39.669364+0000","last_clean":"2024-03-06T07:25:39.669364+0000","last_became_active":"2024-03-06T07:25:38.388457+0000","last_became_peered":"2024-03-06T07:25:38.388457+0000","last_unstale":"2024-03-06T07:25:39.669364+0000","last_undegraded":"2024-03-06T07:25:39.669364+0000","last_fullsized":"2024-03-06T07:25:39.669364+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:24.632482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668309+0000","last_change":"2024-03-06T07:25:38.381579+0000","last_active":"2024-03-06T07:25:39.668309+0000","last_peered":"2024-03-06T07:25:39.668309+0000","last_clean":"2024-03-06T07:25:39.668309+0000","last_became_active":"2024-03-06T07:25:38.381292+0000","last_became_peered":"2024-03-06T07:25:38.381292+0000","last_unstale":"2024-03-06T07:25:39.668309+0000","last_undegraded":"2024-03-06T07:25:39.668309+0000","last_fullsized":"2024-03-06T07:25:39.668309+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:40.716041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668576+0000","last_change":"2024-03-06T07:25:38.389915+0000","last_active":"2024-03-06T07:25:39.668576+0000","last_peered":"2024-03-06T07:25:39.668576+0000","last_clean":"2024-03-06T07:25:39.668576+0000","last_became_active":"2024-03-06T07:25:38.389534+0000","last_became_peered":"2024-03-06T07:25:38.389534+0000","last_unstale":"2024-03-06T07:25:39.668576+0000","last_undegraded":"2024-03-06T07:25:39.668576+0000","last_fullsized":"2024-03-06T07:25:39.668576+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:57.310712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668655+0000","last_change":"2024-03-06T07:25:38.385487+0000","last_active":"2024-03-06T07:25:39.668655+0000","last_peered":"2024-03-06T07:25:39.668655+0000","last_clean":"2024-03-06T07:25:39.668655+0000","last_became_active":"2024-03-06T07:25:38.384963+0000","last_became_peered":"2024-03-06T07:25:38.384963+0000","last_unstale":"2024-03-06T07:25:39.668655+0000","last_undegraded":"2024-03-06T07:25:39.668655+0000","last_fullsized":"2024-03-06T07:25:39.668655+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:28.834292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"181'4","reported_seq":18,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668921+0000","last_change":"2024-03-06T07:25:37.360161+0000","last_active":"2024-03-06T07:25:39.668921+0000","last_peered":"2024-03-06T07:25:39.668921+0000","last_clean":"2024-03-06T07:25:39.668921+0000","last_became_active":"2024-03-06T07:25:37.359858+0000","last_became_peered":"2024-03-06T07:25:37.359858+0000","last_unstale":"2024-03-06T07:25:39.668921+0000","last_undegraded":"2024-03-06T07:25:39.668921+0000","last_fullsized":"2024-03-06T07:25:39.668921+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:17.069592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668614+0000","last_change":"2024-03-06T07:25:38.381755+0000","last_active":"2024-03-06T07:25:39.668614+0000","last_peered":"2024-03-06T07:25:39.668614+0000","last_clean":"2024-03-06T07:25:39.668614+0000","last_became_active":"2024-03-06T07:25:38.381521+0000","last_became_peered":"2024-03-06T07:25:38.381521+0000","last_unstale":"2024-03-06T07:25:39.668614+0000","last_undegraded":"2024-03-06T07:25:39.668614+0000","last_fullsized":"2024-03-06T07:25:39.668614+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:53.859903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"181'1","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377781+0000","last_change":"2024-03-06T07:25:37.361735+0000","last_active":"2024-03-06T07:25:39.377781+0000","last_peered":"2024-03-06T07:25:39.377781+0000","last_clean":"2024-03-06T07:25:39.377781+0000","last_became_active":"2024-03-06T07:25:37.361071+0000","last_became_peered":"2024-03-06T07:25:37.361071+0000","last_unstale":"2024-03-06T07:25:39.377781+0000","last_undegraded":"2024-03-06T07:25:39.377781+0000","last_fullsized":"2024-03-06T07:25:39.377781+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:58.697314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377691+0000","last_change":"2024-03-06T07:25:38.388389+0000","last_active":"2024-03-06T07:25:39.377691+0000","last_peered":"2024-03-06T07:25:39.377691+0000","last_clean":"2024-03-06T07:25:39.377691+0000","last_became_active":"2024-03-06T07:25:38.388081+0000","last_became_peered":"2024-03-06T07:25:38.388081+0000","last_unstale":"2024-03-06T07:25:39.377691+0000","last_undegraded":"2024-03-06T07:25:39.377691+0000","last_fullsized":"2024-03-06T07:25:39.377691+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:14.022542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"181'2","reported_seq":16,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668565+0000","last_change":"2024-03-06T07:25:37.358208+0000","last_active":"2024-03-06T07:25:39.668565+0000","last_peered":"2024-03-06T07:25:39.668565+0000","last_clean":"2024-03-06T07:25:39.668565+0000","last_became_active":"2024-03-06T07:25:37.357941+0000","last_became_peered":"2024-03-06T07:25:37.357941+0000","last_unstale":"2024-03-06T07:25:39.668565+0000","last_undegraded":"2024-03-06T07:25:39.668565+0000","last_fullsized":"2024-03-06T07:25:39.668565+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:39.788968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668983+0000","last_change":"2024-03-06T07:25:38.383798+0000","last_active":"2024-03-06T07:25:39.668983+0000","last_peered":"2024-03-06T07:25:39.668983+0000","last_clean":"2024-03-06T07:25:39.668983+0000","last_became_active":"2024-03-06T07:25:38.383562+0000","last_became_peered":"2024-03-06T07:25:38.383562+0000","last_unstale":"2024-03-06T07:25:39.668983+0000","last_undegraded":"2024-03-06T07:25:39.668983+0000","last_fullsized":"2024-03-06T07:25:39.668983+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:51.792062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"181'1","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644234+0000","last_change":"2024-03-06T07:25:37.357145+0000","last_active":"2024-03-06T07:25:39.644234+0000","last_peered":"2024-03-06T07:25:39.644234+0000","last_clean":"2024-03-06T07:25:39.644234+0000","last_became_active":"2024-03-06T07:25:37.356819+0000","last_became_peered":"2024-03-06T07:25:37.356819+0000","last_unstale":"2024-03-06T07:25:39.644234+0000","last_undegraded":"2024-03-06T07:25:39.644234+0000","last_fullsized":"2024-03-06T07:25:39.644234+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:36.551436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.376615+0000","last_change":"2024-03-06T07:25:38.380554+0000","last_active":"2024-03-06T07:25:39.376615+0000","last_peered":"2024-03-06T07:25:39.376615+0000","last_clean":"2024-03-06T07:25:39.376615+0000","last_became_active":"2024-03-06T07:25:38.379008+0000","last_became_peered":"2024-03-06T07:25:38.379008+0000","last_unstale":"2024-03-06T07:25:39.376615+0000","last_undegraded":"2024-03-06T07:25:39.376615+0000","last_fullsized":"2024-03-06T07:25:39.376615+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:11.827074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"181'1","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668797+0000","last_change":"2024-03-06T07:25:37.366475+0000","last_active":"2024-03-06T07:25:39.668797+0000","last_peered":"2024-03-06T07:25:39.668797+0000","last_clean":"2024-03-06T07:25:39.668797+0000","last_became_active":"2024-03-06T07:25:37.366203+0000","last_became_peered":"2024-03-06T07:25:37.366203+0000","last_unstale":"2024-03-06T07:25:39.668797+0000","last_undegraded":"2024-03-06T07:25:39.668797+0000","last_fullsized":"2024-03-06T07:25:39.668797+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:33.951539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644187+0000","last_change":"2024-03-06T07:25:38.385709+0000","last_active":"2024-03-06T07:25:39.644187+0000","last_peered":"2024-03-06T07:25:39.644187+0000","last_clean":"2024-03-06T07:25:39.644187+0000","last_became_active":"2024-03-06T07:25:38.385457+0000","last_became_peered":"2024-03-06T07:25:38.385457+0000","last_unstale":"2024-03-06T07:25:39.644187+0000","last_undegraded":"2024-03-06T07:25:39.644187+0000","last_fullsized":"2024-03-06T07:25:39.644187+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:58.284018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.376808+0000","last_change":"2024-03-06T07:25:37.362493+0000","last_active":"2024-03-06T07:25:39.376808+0000","last_peered":"2024-03-06T07:25:39.376808+0000","last_clean":"2024-03-06T07:25:39.376808+0000","last_became_active":"2024-03-06T07:25:37.362019+0000","last_became_peered":"2024-03-06T07:25:37.362019+0000","last_unstale":"2024-03-06T07:25:39.376808+0000","last_undegraded":"2024-03-06T07:25:39.376808+0000","last_fullsized":"2024-03-06T07:25:39.376808+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:23.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668684+0000","last_change":"2024-03-06T07:25:38.382196+0000","last_active":"2024-03-06T07:25:39.668684+0000","last_peered":"2024-03-06T07:25:39.668684+0000","last_clean":"2024-03-06T07:25:39.668684+0000","last_became_active":"2024-03-06T07:25:38.381931+0000","last_became_peered":"2024-03-06T07:25:38.381931+0000","last_unstale":"2024-03-06T07:25:39.668684+0000","last_undegraded":"2024-03-06T07:25:39.668684+0000","last_fullsized":"2024-03-06T07:25:39.668684+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:28.709825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669122+0000","last_change":"2024-03-06T07:25:37.359237+0000","last_active":"2024-03-06T07:25:39.669122+0000","last_peered":"2024-03-06T07:25:39.669122+0000","last_clean":"2024-03-06T07:25:39.669122+0000","last_became_active":"2024-03-06T07:25:37.358927+0000","last_became_peered":"2024-03-06T07:25:37.358927+0000","last_unstale":"2024-03-06T07:25:39.669122+0000","last_undegraded":"2024-03-06T07:25:39.669122+0000","last_fullsized":"2024-03-06T07:25:39.669122+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:48.589245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644419+0000","last_change":"2024-03-06T07:25:38.383408+0000","last_active":"2024-03-06T07:25:39.644419+0000","last_peered":"2024-03-06T07:25:39.644419+0000","last_clean":"2024-03-06T07:25:39.644419+0000","last_became_active":"2024-03-06T07:25:38.383056+0000","last_became_peered":"2024-03-06T07:25:38.383056+0000","last_unstale":"2024-03-06T07:25:39.644419+0000","last_undegraded":"2024-03-06T07:25:39.644419+0000","last_fullsized":"2024-03-06T07:25:39.644419+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:06.753664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669297+0000","last_change":"2024-03-06T07:25:37.367257+0000","last_active":"2024-03-06T07:25:39.669297+0000","last_peered":"2024-03-06T07:25:39.669297+0000","last_clean":"2024-03-06T07:25:39.669297+0000","last_became_active":"2024-03-06T07:25:37.366971+0000","last_became_peered":"2024-03-06T07:25:37.366971+0000","last_unstale":"2024-03-06T07:25:39.669297+0000","last_undegraded":"2024-03-06T07:25:39.669297+0000","last_fullsized":"2024-03-06T07:25:39.669297+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:55.694205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644390+0000","last_change":"2024-03-06T07:25:38.380428+0000","last_active":"2024-03-06T07:25:39.644390+0000","last_peered":"2024-03-06T07:25:39.644390+0000","last_clean":"2024-03-06T07:25:39.644390+0000","last_became_active":"2024-03-06T07:25:38.378644+0000","last_became_peered":"2024-03-06T07:25:38.378644+0000","last_unstale":"2024-03-06T07:25:39.644390+0000","last_undegraded":"2024-03-06T07:25:39.644390+0000","last_fullsized":"2024-03-06T07:25:39.644390+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:27.036006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.376571+0000","last_change":"2024-03-06T07:25:37.363334+0000","last_active":"2024-03-06T07:25:39.376571+0000","last_peered":"2024-03-06T07:25:39.376571+0000","last_clean":"2024-03-06T07:25:39.376571+0000","last_became_active":"2024-03-06T07:25:37.362631+0000","last_became_peered":"2024-03-06T07:25:37.362631+0000","last_unstale":"2024-03-06T07:25:39.376571+0000","last_undegraded":"2024-03-06T07:25:39.376571+0000","last_fullsized":"2024-03-06T07:25:39.376571+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:13.129749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668912+0000","last_change":"2024-03-06T07:25:38.373547+0000","last_active":"2024-03-06T07:25:39.668912+0000","last_peered":"2024-03-06T07:25:39.668912+0000","last_clean":"2024-03-06T07:25:39.668912+0000","last_became_active":"2024-03-06T07:25:38.373299+0000","last_became_peered":"2024-03-06T07:25:38.373299+0000","last_unstale":"2024-03-06T07:25:39.668912+0000","last_undegraded":"2024-03-06T07:25:39.668912+0000","last_fullsized":"2024-03-06T07:25:39.668912+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:32.322641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"181'3","reported_seq":17,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669274+0000","last_change":"2024-03-06T07:25:37.367405+0000","last_active":"2024-03-06T07:25:39.669274+0000","last_peered":"2024-03-06T07:25:39.669274+0000","last_clean":"2024-03-06T07:25:39.669274+0000","last_became_active":"2024-03-06T07:25:37.367077+0000","last_became_peered":"2024-03-06T07:25:37.367077+0000","last_unstale":"2024-03-06T07:25:39.669274+0000","last_undegraded":"2024-03-06T07:25:39.669274+0000","last_fullsized":"2024-03-06T07:25:39.669274+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:56.309117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.376820+0000","last_change":"2024-03-06T07:25:38.382930+0000","last_active":"2024-03-06T07:25:39.376820+0000","last_peered":"2024-03-06T07:25:39.376820+0000","last_clean":"2024-03-06T07:25:39.376820+0000","last_became_active":"2024-03-06T07:25:38.382654+0000","last_became_peered":"2024-03-06T07:25:38.382654+0000","last_unstale":"2024-03-06T07:25:39.376820+0000","last_undegraded":"2024-03-06T07:25:39.376820+0000","last_fullsized":"2024-03-06T07:25:39.376820+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:02:59.698909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"181'6","reported_seq":20,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668914+0000","last_change":"2024-03-06T07:25:37.359803+0000","last_active":"2024-03-06T07:25:39.668914+0000","last_peered":"2024-03-06T07:25:39.668914+0000","last_clean":"2024-03-06T07:25:39.668914+0000","last_became_active":"2024-03-06T07:25:37.359509+0000","last_became_peered":"2024-03-06T07:25:37.359509+0000","last_unstale":"2024-03-06T07:25:39.668914+0000","last_undegraded":"2024-03-06T07:25:39.668914+0000","last_fullsized":"2024-03-06T07:25:39.668914+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:34.052508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.375250+0000","last_change":"2024-03-06T07:25:38.373185+0000","last_active":"2024-03-06T07:25:39.375250+0000","last_peered":"2024-03-06T07:25:39.375250+0000","last_clean":"2024-03-06T07:25:39.375250+0000","last_became_active":"2024-03-06T07:25:38.372921+0000","last_became_peered":"2024-03-06T07:25:38.372921+0000","last_unstale":"2024-03-06T07:25:39.375250+0000","last_undegraded":"2024-03-06T07:25:39.375250+0000","last_fullsized":"2024-03-06T07:25:39.375250+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:37.553031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"181'2","reported_seq":16,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378361+0000","last_change":"2024-03-06T07:25:37.363650+0000","last_active":"2024-03-06T07:25:39.378361+0000","last_peered":"2024-03-06T07:25:39.378361+0000","last_clean":"2024-03-06T07:25:39.378361+0000","last_became_active":"2024-03-06T07:25:37.363319+0000","last_became_peered":"2024-03-06T07:25:37.363319+0000","last_unstale":"2024-03-06T07:25:39.378361+0000","last_undegraded":"2024-03-06T07:25:39.378361+0000","last_fullsized":"2024-03-06T07:25:39.378361+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:13.129749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.376771+0000","last_change":"2024-03-06T07:25:38.387048+0000","last_active":"2024-03-06T07:25:39.376771+0000","last_peered":"2024-03-06T07:25:39.376771+0000","last_clean":"2024-03-06T07:25:39.376771+0000","last_became_active":"2024-03-06T07:25:38.386767+0000","last_became_peered":"2024-03-06T07:25:38.386767+0000","last_unstale":"2024-03-06T07:25:39.376771+0000","last_undegraded":"2024-03-06T07:25:39.376771+0000","last_fullsized":"2024-03-06T07:25:39.376771+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:05.978755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.375435+0000","last_change":"2024-03-06T07:25:37.357207+0000","last_active":"2024-03-06T07:25:39.375435+0000","last_peered":"2024-03-06T07:25:39.375435+0000","last_clean":"2024-03-06T07:25:39.375435+0000","last_became_active":"2024-03-06T07:25:37.356920+0000","last_became_peered":"2024-03-06T07:25:37.356920+0000","last_unstale":"2024-03-06T07:25:39.375435+0000","last_undegraded":"2024-03-06T07:25:39.375435+0000","last_fullsized":"2024-03-06T07:25:39.375435+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:03.063073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668476+0000","last_change":"2024-03-06T07:25:38.385514+0000","last_active":"2024-03-06T07:25:39.668476+0000","last_peered":"2024-03-06T07:25:39.668476+0000","last_clean":"2024-03-06T07:25:39.668476+0000","last_became_active":"2024-03-06T07:25:38.385158+0000","last_became_peered":"2024-03-06T07:25:38.385158+0000","last_unstale":"2024-03-06T07:25:39.668476+0000","last_undegraded":"2024-03-06T07:25:39.668476+0000","last_fullsized":"2024-03-06T07:25:39.668476+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:41.396084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668358+0000","last_change":"2024-03-06T07:25:37.360307+0000","last_active":"2024-03-06T07:25:39.668358+0000","last_peered":"2024-03-06T07:25:39.668358+0000","last_clean":"2024-03-06T07:25:39.668358+0000","last_became_active":"2024-03-06T07:25:37.360024+0000","last_became_peered":"2024-03-06T07:25:37.360024+0000","last_unstale":"2024-03-06T07:25:39.668358+0000","last_undegraded":"2024-03-06T07:25:39.668358+0000","last_fullsized":"2024-03-06T07:25:39.668358+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:05.091704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644335+0000","last_change":"2024-03-06T07:25:38.374693+0000","last_active":"2024-03-06T07:25:39.644335+0000","last_peered":"2024-03-06T07:25:39.644335+0000","last_clean":"2024-03-06T07:25:39.644335+0000","last_became_active":"2024-03-06T07:25:38.374316+0000","last_became_peered":"2024-03-06T07:25:38.374316+0000","last_unstale":"2024-03-06T07:25:39.644335+0000","last_undegraded":"2024-03-06T07:25:39.644335+0000","last_fullsized":"2024-03-06T07:25:39.644335+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:19.493328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"181'8","reported_seq":22,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377871+0000","last_change":"2024-03-06T07:25:37.364965+0000","last_active":"2024-03-06T07:25:39.377871+0000","last_peered":"2024-03-06T07:25:39.377871+0000","last_clean":"2024-03-06T07:25:39.377871+0000","last_became_active":"2024-03-06T07:25:37.363889+0000","last_became_peered":"2024-03-06T07:25:37.363889+0000","last_unstale":"2024-03-06T07:25:39.377871+0000","last_undegraded":"2024-03-06T07:25:39.377871+0000","last_fullsized":"2024-03-06T07:25:39.377871+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:11.149946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.375393+0000","last_change":"2024-03-06T07:25:38.386970+0000","last_active":"2024-03-06T07:25:39.375393+0000","last_peered":"2024-03-06T07:25:39.375393+0000","last_clean":"2024-03-06T07:25:39.375393+0000","last_became_active":"2024-03-06T07:25:38.386682+0000","last_became_peered":"2024-03-06T07:25:38.386682+0000","last_unstale":"2024-03-06T07:25:39.375393+0000","last_undegraded":"2024-03-06T07:25:39.375393+0000","last_fullsized":"2024-03-06T07:25:39.375393+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:24.632482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"181'1","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668795+0000","last_change":"2024-03-06T07:25:37.364101+0000","last_active":"2024-03-06T07:25:39.668795+0000","last_peered":"2024-03-06T07:25:39.668795+0000","last_clean":"2024-03-06T07:25:39.668795+0000","last_became_active":"2024-03-06T07:25:37.363814+0000","last_became_peered":"2024-03-06T07:25:37.363814+0000","last_unstale":"2024-03-06T07:25:39.668795+0000","last_undegraded":"2024-03-06T07:25:39.668795+0000","last_fullsized":"2024-03-06T07:25:39.668795+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:27.495845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669063+0000","last_change":"2024-03-06T07:25:38.393126+0000","last_active":"2024-03-06T07:25:39.669063+0000","last_peered":"2024-03-06T07:25:39.669063+0000","last_clean":"2024-03-06T07:25:39.669063+0000","last_became_active":"2024-03-06T07:25:38.392539+0000","last_became_peered":"2024-03-06T07:25:38.392539+0000","last_unstale":"2024-03-06T07:25:39.669063+0000","last_undegraded":"2024-03-06T07:25:39.669063+0000","last_fullsized":"2024-03-06T07:25:39.669063+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:26.623201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"181'2","reported_seq":16,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668468+0000","last_change":"2024-03-06T07:25:37.358403+0000","last_active":"2024-03-06T07:25:39.668468+0000","last_peered":"2024-03-06T07:25:39.668468+0000","last_clean":"2024-03-06T07:25:39.668468+0000","last_became_active":"2024-03-06T07:25:37.358134+0000","last_became_peered":"2024-03-06T07:25:37.358134+0000","last_unstale":"2024-03-06T07:25:39.668468+0000","last_undegraded":"2024-03-06T07:25:39.668468+0000","last_fullsized":"2024-03-06T07:25:39.668468+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:22.323500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669043+0000","last_change":"2024-03-06T07:25:38.383050+0000","last_active":"2024-03-06T07:25:39.669043+0000","last_peered":"2024-03-06T07:25:39.669043+0000","last_clean":"2024-03-06T07:25:39.669043+0000","last_became_active":"2024-03-06T07:25:38.382813+0000","last_became_peered":"2024-03-06T07:25:38.382813+0000","last_unstale":"2024-03-06T07:25:39.669043+0000","last_undegraded":"2024-03-06T07:25:39.669043+0000","last_fullsized":"2024-03-06T07:25:39.669043+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:50.499953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"181'1","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377289+0000","last_change":"2024-03-06T07:25:37.355316+0000","last_active":"2024-03-06T07:25:39.377289+0000","last_peered":"2024-03-06T07:25:39.377289+0000","last_clean":"2024-03-06T07:25:39.377289+0000","last_became_active":"2024-03-06T07:25:37.355060+0000","last_became_peered":"2024-03-06T07:25:37.355060+0000","last_unstale":"2024-03-06T07:25:39.377289+0000","last_undegraded":"2024-03-06T07:25:39.377289+0000","last_fullsized":"2024-03-06T07:25:39.377289+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:11.330752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378497+0000","last_change":"2024-03-06T07:25:38.391664+0000","last_active":"2024-03-06T07:25:39.378497+0000","last_peered":"2024-03-06T07:25:39.378497+0000","last_clean":"2024-03-06T07:25:39.378497+0000","last_became_active":"2024-03-06T07:25:38.391330+0000","last_became_peered":"2024-03-06T07:25:38.391330+0000","last_unstale":"2024-03-06T07:25:39.378497+0000","last_undegraded":"2024-03-06T07:25:39.378497+0000","last_fullsized":"2024-03-06T07:25:39.378497+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:35.054103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378741+0000","last_change":"2024-03-06T07:25:37.354804+0000","last_active":"2024-03-06T07:25:39.378741+0000","last_peered":"2024-03-06T07:25:39.378741+0000","last_clean":"2024-03-06T07:25:39.378741+0000","last_became_active":"2024-03-06T07:25:37.354535+0000","last_became_peered":"2024-03-06T07:25:37.354535+0000","last_unstale":"2024-03-06T07:25:39.378741+0000","last_undegraded":"2024-03-06T07:25:39.378741+0000","last_fullsized":"2024-03-06T07:25:39.378741+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:42.128734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377779+0000","last_change":"2024-03-06T07:25:38.388989+0000","last_active":"2024-03-06T07:25:39.377779+0000","last_peered":"2024-03-06T07:25:39.377779+0000","last_clean":"2024-03-06T07:25:39.377779+0000","last_became_active":"2024-03-06T07:25:38.388734+0000","last_became_peered":"2024-03-06T07:25:38.388734+0000","last_unstale":"2024-03-06T07:25:39.377779+0000","last_undegraded":"2024-03-06T07:25:39.377779+0000","last_fullsized":"2024-03-06T07:25:39.377779+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:15.701618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"181'2","reported_seq":16,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378542+0000","last_change":"2024-03-06T07:25:37.362425+0000","last_active":"2024-03-06T07:25:39.378542+0000","last_peered":"2024-03-06T07:25:39.378542+0000","last_clean":"2024-03-06T07:25:39.378542+0000","last_became_active":"2024-03-06T07:25:37.362172+0000","last_became_peered":"2024-03-06T07:25:37.362172+0000","last_unstale":"2024-03-06T07:25:39.378542+0000","last_undegraded":"2024-03-06T07:25:39.378542+0000","last_fullsized":"2024-03-06T07:25:39.378542+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:06.088494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668962+0000","last_change":"2024-03-06T07:25:38.384216+0000","last_active":"2024-03-06T07:25:39.668962+0000","last_peered":"2024-03-06T07:25:39.668962+0000","last_clean":"2024-03-06T07:25:39.668962+0000","last_became_active":"2024-03-06T07:25:38.383934+0000","last_became_peered":"2024-03-06T07:25:38.383934+0000","last_unstale":"2024-03-06T07:25:39.668962+0000","last_undegraded":"2024-03-06T07:25:39.668962+0000","last_fullsized":"2024-03-06T07:25:39.668962+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:18.585007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644273+0000","last_change":"2024-03-06T07:25:37.354863+0000","last_active":"2024-03-06T07:25:39.644273+0000","last_peered":"2024-03-06T07:25:39.644273+0000","last_clean":"2024-03-06T07:25:39.644273+0000","last_became_active":"2024-03-06T07:25:37.354465+0000","last_became_peered":"2024-03-06T07:25:37.354465+0000","last_unstale":"2024-03-06T07:25:39.644273+0000","last_undegraded":"2024-03-06T07:25:39.644273+0000","last_fullsized":"2024-03-06T07:25:39.644273+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:57.966669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.375347+0000","last_change":"2024-03-06T07:25:38.380432+0000","last_active":"2024-03-06T07:25:39.375347+0000","last_peered":"2024-03-06T07:25:39.375347+0000","last_clean":"2024-03-06T07:25:39.375347+0000","last_became_active":"2024-03-06T07:25:38.380176+0000","last_became_peered":"2024-03-06T07:25:38.380176+0000","last_unstale":"2024-03-06T07:25:39.375347+0000","last_undegraded":"2024-03-06T07:25:39.375347+0000","last_fullsized":"2024-03-06T07:25:39.375347+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:54.257560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644148+0000","last_change":"2024-03-06T07:25:37.361572+0000","last_active":"2024-03-06T07:25:39.644148+0000","last_peered":"2024-03-06T07:25:39.644148+0000","last_clean":"2024-03-06T07:25:39.644148+0000","last_became_active":"2024-03-06T07:25:37.361271+0000","last_became_peered":"2024-03-06T07:25:37.361271+0000","last_unstale":"2024-03-06T07:25:39.644148+0000","last_undegraded":"2024-03-06T07:25:39.644148+0000","last_fullsized":"2024-03-06T07:25:39.644148+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:23.630887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668864+0000","last_change":"2024-03-06T07:25:38.389508+0000","last_active":"2024-03-06T07:25:39.668864+0000","last_peered":"2024-03-06T07:25:39.668864+0000","last_clean":"2024-03-06T07:25:39.668864+0000","last_became_active":"2024-03-06T07:25:38.389181+0000","last_became_peered":"2024-03-06T07:25:38.389181+0000","last_unstale":"2024-03-06T07:25:39.668864+0000","last_undegraded":"2024-03-06T07:25:39.668864+0000","last_fullsized":"2024-03-06T07:25:39.668864+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:38.250424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668158+0000","last_change":"2024-03-06T07:25:37.359314+0000","last_active":"2024-03-06T07:25:39.668158+0000","last_peered":"2024-03-06T07:25:39.668158+0000","last_clean":"2024-03-06T07:25:39.668158+0000","last_became_active":"2024-03-06T07:25:37.359008+0000","last_became_peered":"2024-03-06T07:25:37.359008+0000","last_unstale":"2024-03-06T07:25:39.668158+0000","last_undegraded":"2024-03-06T07:25:39.668158+0000","last_fullsized":"2024-03-06T07:25:39.668158+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:54.224477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.375420+0000","last_change":"2024-03-06T07:25:38.375341+0000","last_active":"2024-03-06T07:25:39.375420+0000","last_peered":"2024-03-06T07:25:39.375420+0000","last_clean":"2024-03-06T07:25:39.375420+0000","last_became_active":"2024-03-06T07:25:38.375073+0000","last_became_peered":"2024-03-06T07:25:38.375073+0000","last_unstale":"2024-03-06T07:25:39.375420+0000","last_undegraded":"2024-03-06T07:25:39.375420+0000","last_fullsized":"2024-03-06T07:25:39.375420+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:07.494095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"181'2","reported_seq":16,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378596+0000","last_change":"2024-03-06T07:25:37.364253+0000","last_active":"2024-03-06T07:25:39.378596+0000","last_peered":"2024-03-06T07:25:39.378596+0000","last_clean":"2024-03-06T07:25:39.378596+0000","last_became_active":"2024-03-06T07:25:37.363933+0000","last_became_peered":"2024-03-06T07:25:37.363933+0000","last_unstale":"2024-03-06T07:25:39.378596+0000","last_undegraded":"2024-03-06T07:25:39.378596+0000","last_fullsized":"2024-03-06T07:25:39.378596+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:46.308989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669185+0000","last_change":"2024-03-06T07:25:38.385294+0000","last_active":"2024-03-06T07:25:39.669185+0000","last_peered":"2024-03-06T07:25:39.669185+0000","last_clean":"2024-03-06T07:25:39.669185+0000","last_became_active":"2024-03-06T07:25:38.383985+0000","last_became_peered":"2024-03-06T07:25:38.383985+0000","last_unstale":"2024-03-06T07:25:39.669185+0000","last_undegraded":"2024-03-06T07:25:39.669185+0000","last_fullsized":"2024-03-06T07:25:39.669185+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:03.304412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"182'1","reported_seq":15,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644488+0000","last_change":"2024-03-06T07:25:37.364443+0000","last_active":"2024-03-06T07:25:39.644488+0000","last_peered":"2024-03-06T07:25:39.644488+0000","last_clean":"2024-03-06T07:25:39.644488+0000","last_became_active":"2024-03-06T07:25:37.364093+0000","last_became_peered":"2024-03-06T07:25:37.364093+0000","last_unstale":"2024-03-06T07:25:39.644488+0000","last_undegraded":"2024-03-06T07:25:39.644488+0000","last_fullsized":"2024-03-06T07:25:39.644488+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:53.255965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668767+0000","last_change":"2024-03-06T07:25:38.389757+0000","last_active":"2024-03-06T07:25:39.668767+0000","last_peered":"2024-03-06T07:25:39.668767+0000","last_clean":"2024-03-06T07:25:39.668767+0000","last_became_active":"2024-03-06T07:25:38.389469+0000","last_became_peered":"2024-03-06T07:25:38.389469+0000","last_unstale":"2024-03-06T07:25:39.668767+0000","last_undegraded":"2024-03-06T07:25:39.668767+0000","last_fullsized":"2024-03-06T07:25:39.668767+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:54.257560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.644396+0000","last_change":"2024-03-06T07:25:37.362488+0000","last_active":"2024-03-06T07:25:39.644396+0000","last_peered":"2024-03-06T07:25:39.644396+0000","last_clean":"2024-03-06T07:25:39.644396+0000","last_became_active":"2024-03-06T07:25:37.362203+0000","last_became_peered":"2024-03-06T07:25:37.362203+0000","last_unstale":"2024-03-06T07:25:39.644396+0000","last_undegraded":"2024-03-06T07:25:39.644396+0000","last_fullsized":"2024-03-06T07:25:39.644396+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:18.264073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668701+0000","last_change":"2024-03-06T07:25:38.382870+0000","last_active":"2024-03-06T07:25:39.668701+0000","last_peered":"2024-03-06T07:25:39.668701+0000","last_clean":"2024-03-06T07:25:39.668701+0000","last_became_active":"2024-03-06T07:25:38.382612+0000","last_became_peered":"2024-03-06T07:25:38.382612+0000","last_unstale":"2024-03-06T07:25:39.668701+0000","last_undegraded":"2024-03-06T07:25:39.668701+0000","last_fullsized":"2024-03-06T07:25:39.668701+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:39.536778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378753+0000","last_change":"2024-03-06T07:25:37.366204+0000","last_active":"2024-03-06T07:25:39.378753+0000","last_peered":"2024-03-06T07:25:39.378753+0000","last_clean":"2024-03-06T07:25:39.378753+0000","last_became_active":"2024-03-06T07:25:37.365751+0000","last_became_peered":"2024-03-06T07:25:37.365751+0000","last_unstale":"2024-03-06T07:25:39.378753+0000","last_undegraded":"2024-03-06T07:25:39.378753+0000","last_fullsized":"2024-03-06T07:25:39.378753+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:21.274078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669104+0000","last_change":"2024-03-06T07:25:38.389588+0000","last_active":"2024-03-06T07:25:39.669104+0000","last_peered":"2024-03-06T07:25:39.669104+0000","last_clean":"2024-03-06T07:25:39.669104+0000","last_became_active":"2024-03-06T07:25:38.389316+0000","last_became_peered":"2024-03-06T07:25:38.389316+0000","last_unstale":"2024-03-06T07:25:39.669104+0000","last_undegraded":"2024-03-06T07:25:39.669104+0000","last_fullsized":"2024-03-06T07:25:39.669104+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:21.823427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378553+0000","last_change":"2024-03-06T07:25:37.363581+0000","last_active":"2024-03-06T07:25:39.378553+0000","last_peered":"2024-03-06T07:25:39.378553+0000","last_clean":"2024-03-06T07:25:39.378553+0000","last_became_active":"2024-03-06T07:25:37.362991+0000","last_became_peered":"2024-03-06T07:25:37.362991+0000","last_unstale":"2024-03-06T07:25:39.378553+0000","last_undegraded":"2024-03-06T07:25:39.378553+0000","last_fullsized":"2024-03-06T07:25:39.378553+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:35:39.207887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668618+0000","last_change":"2024-03-06T07:25:38.379228+0000","last_active":"2024-03-06T07:25:39.668618+0000","last_peered":"2024-03-06T07:25:39.668618+0000","last_clean":"2024-03-06T07:25:39.668618+0000","last_became_active":"2024-03-06T07:25:38.378962+0000","last_became_peered":"2024-03-06T07:25:38.378962+0000","last_unstale":"2024-03-06T07:25:39.668618+0000","last_undegraded":"2024-03-06T07:25:39.668618+0000","last_fullsized":"2024-03-06T07:25:39.668618+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:30.871653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377536+0000","last_change":"2024-03-06T07:25:37.361352+0000","last_active":"2024-03-06T07:25:39.377536+0000","last_peered":"2024-03-06T07:25:39.377536+0000","last_clean":"2024-03-06T07:25:39.377536+0000","last_became_active":"2024-03-06T07:25:37.361059+0000","last_became_peered":"2024-03-06T07:25:37.361059+0000","last_unstale":"2024-03-06T07:25:39.377536+0000","last_undegraded":"2024-03-06T07:25:39.377536+0000","last_fullsized":"2024-03-06T07:25:39.377536+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:11.149946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668436+0000","last_change":"2024-03-06T07:25:38.387479+0000","last_active":"2024-03-06T07:25:39.668436+0000","last_peered":"2024-03-06T07:25:39.668436+0000","last_clean":"2024-03-06T07:25:39.668436+0000","last_became_active":"2024-03-06T07:25:38.386409+0000","last_became_peered":"2024-03-06T07:25:38.386409+0000","last_unstale":"2024-03-06T07:25:39.668436+0000","last_undegraded":"2024-03-06T07:25:39.668436+0000","last_fullsized":"2024-03-06T07:25:39.668436+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:12.748847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.669064+0000","last_change":"2024-03-06T07:25:37.368239+0000","last_active":"2024-03-06T07:25:39.669064+0000","last_peered":"2024-03-06T07:25:39.669064+0000","last_clean":"2024-03-06T07:25:39.669064+0000","last_became_active":"2024-03-06T07:25:37.367979+0000","last_became_peered":"2024-03-06T07:25:37.367979+0000","last_unstale":"2024-03-06T07:25:39.669064+0000","last_undegraded":"2024-03-06T07:25:39.669064+0000","last_fullsized":"2024-03-06T07:25:39.669064+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:13.032326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.378248+0000","last_change":"2024-03-06T07:25:38.391437+0000","last_active":"2024-03-06T07:25:39.378248+0000","last_peered":"2024-03-06T07:25:39.378248+0000","last_clean":"2024-03-06T07:25:39.378248+0000","last_became_active":"2024-03-06T07:25:38.391093+0000","last_became_peered":"2024-03-06T07:25:38.391093+0000","last_unstale":"2024-03-06T07:25:39.378248+0000","last_undegraded":"2024-03-06T07:25:39.378248+0000","last_fullsized":"2024-03-06T07:25:39.378248+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:40.790563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.377903+0000","last_change":"2024-03-06T07:25:37.366881+0000","last_active":"2024-03-06T07:25:39.377903+0000","last_peered":"2024-03-06T07:25:39.377903+0000","last_clean":"2024-03-06T07:25:39.377903+0000","last_became_active":"2024-03-06T07:25:37.366588+0000","last_became_peered":"2024-03-06T07:25:37.366588+0000","last_unstale":"2024-03-06T07:25:39.377903+0000","last_undegraded":"2024-03-06T07:25:39.377903+0000","last_fullsized":"2024-03-06T07:25:39.377903+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:23.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668769+0000","last_change":"2024-03-06T07:25:38.389468+0000","last_active":"2024-03-06T07:25:39.668769+0000","last_peered":"2024-03-06T07:25:39.668769+0000","last_clean":"2024-03-06T07:25:39.668769+0000","last_became_active":"2024-03-06T07:25:38.389172+0000","last_became_peered":"2024-03-06T07:25:38.389172+0000","last_unstale":"2024-03-06T07:25:39.668769+0000","last_undegraded":"2024-03-06T07:25:39.668769+0000","last_fullsized":"2024-03-06T07:25:39.668769+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":180,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:37.341311+0000","last_clean_scrub_stamp":"2024-03-06T07:25:37.341311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:02.693505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"181'2","reported_seq":16,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668877+0000","last_change":"2024-03-06T07:25:37.359284+0000","last_active":"2024-03-06T07:25:39.668877+0000","last_peered":"2024-03-06T07:25:39.668877+0000","last_clean":"2024-03-06T07:25:39.668877+0000","last_became_active":"2024-03-06T07:25:37.358847+0000","last_became_peered":"2024-03-06T07:25:37.358847+0000","last_unstale":"2024-03-06T07:25:39.668877+0000","last_undegraded":"2024-03-06T07:25:39.668877+0000","last_fullsized":"2024-03-06T07:25:39.668877+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.339716+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.339716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:01.483203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":476,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.668710+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:25:39.668710+0000","last_peered":"2024-03-06T07:25:39.668710+0000","last_clean":"2024-03-06T07:25:39.668710+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:25:39.668710+0000","last_undegraded":"2024-03-06T07:25:39.668710+0000","last_fullsized":"2024-03-06T07:25:39.668710+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33512,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338328,"statfs":{"total":96636764160,"available":96602447872,"internally_reserved":0,"allocated":2379776,"data_stored":1710929,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33460,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338380,"statfs":{"total":96636764160,"available":96602501120,"internally_reserved":0,"allocated":2330624,"data_stored":1121627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33452,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338388,"statfs":{"total":96636764160,"available":96602509312,"internally_reserved":0,"allocated":2322432,"data_stored":1121571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574942,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33440,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338400,"statfs":{"total":96636764160,"available":96602521600,"internally_reserved":0,"allocated":2310144,"data_stored":1119979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33440,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338400,"statfs":{"total":96636764160,"available":96602521600,"internally_reserved":0,"allocated":2310144,"data_stored":1119911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33444,"kb_used_data":2260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338396,"statfs":{"total":96636764160,"available":96602517504,"internally_reserved":0,"allocated":2314240,"data_stored":1120505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33436,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338404,"statfs":{"total":96636764160,"available":96602525696,"internally_reserved":0,"allocated":2306048,"data_stored":1119889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37664,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":2371584,"data_stored":1710257,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:46.043 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:46.043 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:46.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.184+0000 7f50f48c3640 1 Processor -- start 2024-03-06T07:25:46.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.184+0000 7f50f48c3640 1 -- start start 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.185+0000 7f50f48c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f0155c30 0x7f50f006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.185+0000 7f50f48c3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f00b0450 con 0x7f50f00c5380 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.185+0000 7f50f48c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f01560e0 con 0x7f50f00b0070 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.185+0000 7f50f48c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f006cfb0 con 0x7f50f0155c30 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.185+0000 7f50eed9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f50f00c5380 0x7f50f014e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58498/0 (socket says 172.21.15.94:58498) 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.185+0000 7f50eed9d640 1 -- 172.21.15.94:0/2117994036 learned_addr learned my addr 172.21.15.94:0/2117994036 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.185+0000 7f50edd9b640 1 --2- 172.21.15.94:0/2117994036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f0155c30 0x7f50f006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.186+0000 7f50ed59a640 1 -- 172.21.15.94:0/2117994036 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536885061 0 0) 0x7f50e8001110 con 0x7f50f00c5380 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.186+0000 7f50ed59a640 1 -- 172.21.15.94:0/2117994036 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d8003520 con 0x7f50f00c5380 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.186+0000 7f50ed59a640 1 -- 172.21.15.94:0/2117994036 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581464129 0 0) 0x7f50f01560e0 con 0x7f50f00b0070 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.186+0000 7f50edd9b640 1 -- 172.21.15.94:0/2117994036 >> v1:172.21.15.145:6789/0 conn(0x7f50f00b0070 legacy=0x7f50f00c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.186+0000 7f50edd9b640 1 -- 172.21.15.94:0/2117994036 >> v1:172.21.15.94:6789/0 conn(0x7f50f00c5380 legacy=0x7f50f014e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.186+0000 7f50edd9b640 1 -- 172.21.15.94:0/2117994036 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f00c77f0 con 0x7f50f0155c30 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.186+0000 7f50edd9b640 1 --2- 172.21.15.94:0/2117994036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f0155c30 0x7f50f006ca70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50e40097d0 tx=0x7f50e402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0a5ec2e77de0eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50ed59a640 1 -- 172.21.15.94:0/2117994036 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4035020 con 0x7f50f0155c30 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50ed59a640 1 -- 172.21.15.94:0/2117994036 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e4031800 con 0x7f50f0155c30 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50ed59a640 1 -- 172.21.15.94:0/2117994036 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4031bd0 con 0x7f50f0155c30 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50f48c3640 1 -- 172.21.15.94:0/2117994036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f0155c30 msgr2=0x7f50f006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50f48c3640 1 --2- 172.21.15.94:0/2117994036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f0155c30 0x7f50f006ca70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50e40097d0 tx=0x7f50e402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50f48c3640 1 -- 172.21.15.94:0/2117994036 shutdown_connections 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50f48c3640 1 --2- 172.21.15.94:0/2117994036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f0155c30 0x7f50f006ca70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50f48c3640 1 -- 172.21.15.94:0/2117994036 >> 172.21.15.94:0/2117994036 conn(0x7f50f00ac590 msgr2=0x7f50f00ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50f48c3640 1 -- 172.21.15.94:0/2117994036 shutdown_connections 2024-03-06T07:25:46.190 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.187+0000 7f50f48c3640 1 -- 172.21.15.94:0/2117994036 wait complete. 2024-03-06T07:25:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.188+0000 7f50f48c3640 1 Processor -- start 2024-03-06T07:25:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50f48c3640 1 -- start start 2024-03-06T07:25:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50f48c3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50f00b0070 0x7f50f01ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50f48c3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f00c5380 0x7f50f01ec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50f48c3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50f0155c30 0x7f50f01ec720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50f48c3640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f006c270 con 0x7f50f00b0070 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50f48c3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f006c020 con 0x7f50f0155c30 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50f48c3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f006c550 con 0x7f50f00c5380 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50ee59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50f00b0070 0x7f50f01ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50ee59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50f00b0070 0x7f50f01ee1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41926/0 (socket says 172.21.15.94:41926) 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50ee59c640 1 -- 172.21.15.94:0/3628606448 learned_addr learned my addr 172.21.15.94:0/3628606448 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50edd9b640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f00c5380 0x7f50f01ec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.189+0000 7f50eed9d640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50f0155c30 0x7f50f01ec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50ee59c640 1 -- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f00c5380 msgr2=0x7f50f01ec1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50ee59c640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f00c5380 0x7f50f01ec1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50ee59c640 1 -- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50f0155c30 msgr2=0x7f50f01ec720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50ee59c640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50f0155c30 0x7f50f01ec720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50ee59c640 1 -- 172.21.15.94:0/3628606448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f00c4100 con 0x7f50f00b0070 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50ee59c640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50f00b0070 0x7f50f01ee1e0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f50e000c890 tx=0x7f50e000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50deffd640 1 -- 172.21.15.94:0/3628606448 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e0017070 con 0x7f50f00b0070 2024-03-06T07:25:46.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50deffd640 1 -- 172.21.15.94:0/3628606448 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e0003dc0 con 0x7f50f00b0070 2024-03-06T07:25:46.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50deffd640 1 -- 172.21.15.94:0/3628606448 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e00052b0 con 0x7f50f00b0070 2024-03-06T07:25:46.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f00c43a0 con 0x7f50f00b0070 2024-03-06T07:25:46.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.190+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f00c4b50 con 0x7f50f00b0070 2024-03-06T07:25:46.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.191+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f0069460 con 0x7f50f00b0070 2024-03-06T07:25:46.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.191+0000 7f50deffd640 1 -- 172.21.15.94:0/3628606448 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50e001a070 con 0x7f50f00b0070 2024-03-06T07:25:46.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.192+0000 7f50deffd640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50d006b4d0 0x7f50d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.192+0000 7f50deffd640 1 -- 172.21.15.94:0/3628606448 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50e0089950 con 0x7f50f00b0070 2024-03-06T07:25:46.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.192+0000 7f50edd9b640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50d006b4d0 0x7f50d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.192+0000 7f50edd9b640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50d006b4d0 0x7f50d006d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f50e4036000 tx=0x7f50e4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.196+0000 7f50deffd640 1 -- 172.21.15.94:0/3628606448 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50f0069460 con 0x7f50f00b0070 2024-03-06T07:25:46.439 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.438+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f50f004f330 con 0x7f50f00b0070 2024-03-06T07:25:46.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.440+0000 7f50deffd640 1 -- 172.21.15.94:0/3628606448 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f50e0058080 con 0x7f50f00b0070 2024-03-06T07:25:46.442 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:25:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:25:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:25:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:25:46.442 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:46.442 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:25:46.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50d006b4d0 msgr2=0x7f50d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50d006b4d0 0x7f50d006d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f50e4036000 tx=0x7f50e4008030 comp rx=0 tx=0).stop 2024-03-06T07:25:46.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50f00b0070 msgr2=0x7f50f01ee1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50f00b0070 0x7f50f01ee1e0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f50e000c890 tx=0x7f50e000cd50 comp rx=0 tx=0).stop 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 shutdown_connections 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f50d006b4d0 0x7f50d006d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50f0155c30 0x7f50f01ec720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50f00c5380 0x7f50f01ec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 --2- 172.21.15.94:0/3628606448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50f00b0070 0x7f50f01ee1e0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 >> 172.21.15.94:0/3628606448 conn(0x7f50f00ac590 msgr2=0x7f50f00ad770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 shutdown_connections 2024-03-06T07:25:46.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.442+0000 7f50f48c3640 1 -- 172.21.15.94:0/3628606448 wait complete. 2024-03-06T07:25:46.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:46.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6086916640 1 Processor -- start 2024-03-06T07:25:46.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6086916640 1 -- start start 2024-03-06T07:25:46.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6086916640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f60800c1aa0 0x7f60801529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6086916640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60800c28d0 con 0x7f6080155a40 2024-03-06T07:25:46.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6086916640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60800c2bb0 con 0x7f608014f790 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6086916640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60800c2680 con 0x7f60800c1aa0 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f607f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f60800c1aa0 0x7f60801529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6084eb3640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6080155a40 0x7f6080155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58508/0 (socket says 172.21.15.94:58508) 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f6084eb3640 1 -- 172.21.15.94:0/2499851999 learned_addr learned my addr 172.21.15.94:0/2499851999 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.554+0000 7f607f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f60800c1aa0 0x7f60801529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41940/0 (socket says 172.21.15.94:41940) 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.555+0000 7f607f7fe640 1 -- 172.21.15.94:0/2499851999 >> v1:172.21.15.145:6789/0 conn(0x7f608014f790 legacy=0x7f60800c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.555+0000 7f607f7fe640 1 -- 172.21.15.94:0/2499851999 >> v1:172.21.15.94:6789/0 conn(0x7f6080155a40 legacy=0x7f6080155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.555+0000 7f607f7fe640 1 -- 172.21.15.94:0/2499851999 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60800c2e90 con 0x7f60800c1aa0 2024-03-06T07:25:46.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.555+0000 7f607f7fe640 1 --2- 172.21.15.94:0/2499851999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60800c1aa0 0x7f60801529d0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f60800c2800 tx=0x7f607002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd009505ee83722 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f607effd640 1 -- 172.21.15.94:0/2499851999 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070034070 con 0x7f60800c1aa0 2024-03-06T07:25:46.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f607effd640 1 -- 172.21.15.94:0/2499851999 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6070030610 con 0x7f60800c1aa0 2024-03-06T07:25:46.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f607effd640 1 -- 172.21.15.94:0/2499851999 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60700309e0 con 0x7f60800c1aa0 2024-03-06T07:25:46.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f6086916640 1 -- 172.21.15.94:0/2499851999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60800c1aa0 msgr2=0x7f60801529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f6086916640 1 --2- 172.21.15.94:0/2499851999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60800c1aa0 0x7f60801529d0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f60800c2800 tx=0x7f607002b960 comp rx=0 tx=0).stop 2024-03-06T07:25:46.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f6086916640 1 -- 172.21.15.94:0/2499851999 shutdown_connections 2024-03-06T07:25:46.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f6086916640 1 --2- 172.21.15.94:0/2499851999 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60800c1aa0 0x7f60801529d0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f6086916640 1 -- 172.21.15.94:0/2499851999 >> 172.21.15.94:0/2499851999 conn(0x7f60800c6710 msgr2=0x7f60800c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f6086916640 1 -- 172.21.15.94:0/2499851999 shutdown_connections 2024-03-06T07:25:46.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.556+0000 7f6086916640 1 -- 172.21.15.94:0/2499851999 wait complete. 2024-03-06T07:25:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.557+0000 7f6086916640 1 Processor -- start 2024-03-06T07:25:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.557+0000 7f6086916640 1 -- start start 2024-03-06T07:25:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6086916640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60800c1aa0 0x7f6080151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6086916640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f608014f790 0x7f60801de260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6086916640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6080155a40 0x7f60801de7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6086916640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60800c5e30 con 0x7f6080155a40 2024-03-06T07:25:46.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6086916640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60800c0d60 con 0x7f608014f790 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6086916640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60800c5fb0 con 0x7f60800c1aa0 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6080155a40 0x7f60801de7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6080155a40 0x7f60801de7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41942/0 (socket says 172.21.15.94:41942) 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 -- 172.21.15.94:0/3983701632 learned_addr learned my addr 172.21.15.94:0/3983701632 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f607f7fe640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f608014f790 0x7f60801de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f607ffff640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60800c1aa0 0x7f6080151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 -- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60800c1aa0 msgr2=0x7f6080151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60800c1aa0 0x7f6080151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 -- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f608014f790 msgr2=0x7f60801de260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f608014f790 0x7f60801de260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.558+0000 7f6084eb3640 1 -- 172.21.15.94:0/3983701632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6080151bf0 con 0x7f6080155a40 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.559+0000 7f6084eb3640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6080155a40 0x7f60801de7a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f607800cab0 tx=0x7f607800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.559+0000 7f607cff9640 1 -- 172.21.15.94:0/3983701632 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078017070 con 0x7f6080155a40 2024-03-06T07:25:46.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.559+0000 7f607cff9640 1 -- 172.21.15.94:0/3983701632 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607800ac30 con 0x7f6080155a40 2024-03-06T07:25:46.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.559+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6080151750 con 0x7f6080155a40 2024-03-06T07:25:46.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.559+0000 7f607cff9640 1 -- 172.21.15.94:0/3983701632 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078012400 con 0x7f6080155a40 2024-03-06T07:25:46.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.559+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60801eb6f0 con 0x7f6080155a40 2024-03-06T07:25:46.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.560+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608004f330 con 0x7f6080155a40 2024-03-06T07:25:46.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.560+0000 7f607cff9640 1 -- 172.21.15.94:0/3983701632 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f607801a070 con 0x7f6080155a40 2024-03-06T07:25:46.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.561+0000 7f607cff9640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.561+0000 7f607cff9640 1 -- 172.21.15.94:0/3983701632 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6078089e90 con 0x7f6080155a40 2024-03-06T07:25:46.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.561+0000 7f607ffff640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.561+0000 7f607ffff640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6068001780 tx=0x7f6068009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.565+0000 7f607cff9640 1 -- 172.21.15.94:0/3983701632 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60780585c0 con 0x7f6080155a40 2024-03-06T07:25:46.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.772+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f608014fb70 con 0x7f6080155a40 2024-03-06T07:25:46.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.772+0000 7f607cff9640 1 -- 172.21.15.94:0/3983701632 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+4998 (secure 0 0 0) 0x7f607805c660 con 0x7f6080155a40 2024-03-06T07:25:46.774 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:46.774 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":171,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:39.711371+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":37763,"qdb_cluster":[37763]},"id":14}]} 2024-03-06T07:25:46.774 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 171 2024-03-06T07:25:46.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f606006b4d0 msgr2=0x7f606006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6068001780 tx=0x7f6068009040 comp rx=0 tx=0).stop 2024-03-06T07:25:46.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6080155a40 msgr2=0x7f60801de7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6080155a40 0x7f60801de7a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f607800cab0 tx=0x7f607800cf70 comp rx=0 tx=0).stop 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 shutdown_connections 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6080155a40 0x7f60801de7a0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f608014f790 0x7f60801de260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 --2- 172.21.15.94:0/3983701632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60800c1aa0 0x7f6080151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 >> 172.21.15.94:0/3983701632 conn(0x7f60800c6710 msgr2=0x7f60800a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 shutdown_connections 2024-03-06T07:25:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:46.774+0000 7f6086916640 1 -- 172.21.15.94:0/3983701632 wait complete. 2024-03-06T07:25:47.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 171, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:39.711371+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 37763, 'qdb_cluster': [37763]} 2024-03-06T07:25:47.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:47.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:25:47.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.293+0000 7ff77060e640 1 Processor -- start 2024-03-06T07:25:47.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.293+0000 7ff77060e640 1 -- start start 2024-03-06T07:25:47.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff77060e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff768072f00 0x7ff768153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff77060e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7680c0fa0 0x7ff7680c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff77060e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff768070550 con 0x7ff7680c6d80 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff77060e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff768070830 con 0x7ff7680c0fa0 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff77060e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff768070b10 con 0x7ff768072f00 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76dba9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7680c0fa0 0x7ff7680c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76dba9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7680c0fa0 0x7ff7680c6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41948/0 (socket says 172.21.15.94:41948) 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76dba9640 1 -- 172.21.15.94:0/1412270773 learned_addr learned my addr 172.21.15.94:0/1412270773 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/1412270773 >> v2:172.21.15.145:3300/0 conn(0x7ff768072f00 0x7ff768153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76dba9640 1 -- 172.21.15.94:0/1412270773 >> v2:172.21.15.145:3300/0 conn(0x7ff768072f00 msgr2=0x7ff768153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76dba9640 1 --2- 172.21.15.94:0/1412270773 >> v2:172.21.15.145:3300/0 conn(0x7ff768072f00 0x7ff768153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76dba9640 1 -- 172.21.15.94:0/1412270773 >> v1:172.21.15.145:6789/0 conn(0x7ff7680c6d80 legacy=0x7ff76806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76dba9640 1 -- 172.21.15.94:0/1412270773 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7680732e0 con 0x7ff7680c0fa0 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.294+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/1412270773 >> v2:172.21.15.145:3300/0 conn(0x7ff768072f00 0x7ff768153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.295+0000 7ff76dba9640 1 --2- 172.21.15.94:0/1412270773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7680c0fa0 0x7ff7680c6840 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff7680c74f0 tx=0x7ff76402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b01da94219be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.295+0000 7ff76d3a8640 1 -- 172.21.15.94:0/1412270773 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764034070 con 0x7ff7680c0fa0 2024-03-06T07:25:47.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.295+0000 7ff76d3a8640 1 -- 172.21.15.94:0/1412270773 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff764030860 con 0x7ff7680c0fa0 2024-03-06T07:25:47.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.295+0000 7ff76d3a8640 1 -- 172.21.15.94:0/1412270773 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764030c30 con 0x7ff7680c0fa0 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.295+0000 7ff77060e640 1 -- 172.21.15.94:0/1412270773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7680c0fa0 msgr2=0x7ff7680c6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.295+0000 7ff77060e640 1 --2- 172.21.15.94:0/1412270773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7680c0fa0 0x7ff7680c6840 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff7680c74f0 tx=0x7ff76402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.296+0000 7ff77060e640 1 -- 172.21.15.94:0/1412270773 shutdown_connections 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.296+0000 7ff77060e640 1 --2- 172.21.15.94:0/1412270773 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7680c0fa0 0x7ff7680c6840 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.296+0000 7ff77060e640 1 --2- 172.21.15.94:0/1412270773 >> v2:172.21.15.145:3300/0 conn(0x7ff768072f00 0x7ff768153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.296+0000 7ff77060e640 1 -- 172.21.15.94:0/1412270773 >> 172.21.15.94:0/1412270773 conn(0x7ff76806b1a0 msgr2=0x7ff76806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.296+0000 7ff77060e640 1 -- 172.21.15.94:0/1412270773 shutdown_connections 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.296+0000 7ff77060e640 1 -- 172.21.15.94:0/1412270773 wait complete. 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 Processor -- start 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 -- start start 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff768072f00 0x7ff768156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680c0fa0 0x7ff76814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7680c6d80 0x7ff7681eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff768072580 con 0x7ff768072f00 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff768072400 con 0x7ff7680c0fa0 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff77060e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76806e0a0 con 0x7ff7680c6d80 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff76e3aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff768072f00 0x7ff768156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff76e3aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff768072f00 0x7ff768156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41952/0 (socket says 172.21.15.94:41952) 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4087177674 learned_addr learned my addr 172.21.15.94:0/4087177674 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.297+0000 7ff76dba9640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680c0fa0 0x7ff76814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff76ebab640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7680c6d80 0x7ff7681eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7680c6d80 msgr2=0x7ff7681eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7680c6d80 0x7ff7681eec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680c0fa0 msgr2=0x7ff76814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680c0fa0 0x7ff76814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4087177674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff768157230 con 0x7ff768072f00 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff768072f00 0x7ff768156cf0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00e810 tx=0x7ff75c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff75affd640 1 -- 172.21.15.94:0/4087177674 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c00f040 con 0x7ff768072f00 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff75affd640 1 -- 172.21.15.94:0/4087177674 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c004030 con 0x7ff768072f00 2024-03-06T07:25:47.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff75affd640 1 -- 172.21.15.94:0/4087177674 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c004400 con 0x7ff768072f00 2024-03-06T07:25:47.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7680aaee0 con 0x7ff768072f00 2024-03-06T07:25:47.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.298+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7680ab4a0 con 0x7ff768072f00 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.299+0000 7ff75affd640 1 -- 172.21.15.94:0/4087177674 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff75c00c6d0 con 0x7ff768072f00 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.300+0000 7ff75affd640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74c06b4d0 0x7ff74c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.300+0000 7ff75affd640 1 -- 172.21.15.94:0/4087177674 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff75c08a030 con 0x7ff768072f00 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.300+0000 7ff76dba9640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74c06b4d0 0x7ff74c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.300+0000 7ff76dba9640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74c06b4d0 0x7ff74c06d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff764036000 tx=0x7ff764008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.301+0000 7ff7731f7740 1 Processor -- start 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.301+0000 7ff7731f7740 1 -- start start 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55ef50 0x5596ab55ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55f330 0x5596ab55cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55d4b0 0x5596ab559680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff7731f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ab4c36b0 con 0x5596ab55d4b0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ab46ca60 con 0x5596ab55ef50 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596ab4bf750 con 0x5596ab55f330 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55d4b0 0x5596ab559680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55d4b0 0x5596ab559680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41958/0 (socket says 172.21.15.94:41958) 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76e3aa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55ef50 0x5596ab55ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 -- 172.21.15.94:0/2435356633 learned_addr learned my addr 172.21.15.94:0/2435356633 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76dba9640 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55f330 0x5596ab55cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 -- 172.21.15.94:0/2435356633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55f330 msgr2=0x5596ab55cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55f330 0x5596ab55cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 -- 172.21.15.94:0/2435356633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55ef50 msgr2=0x5596ab55ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55ef50 0x5596ab55ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 -- 172.21.15.94:0/2435356633 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596ab48af30 con 0x5596ab55d4b0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff76ebab640 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55d4b0 0x5596ab559680 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7ff760001bd0 tx=0x7ff760002090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc60b71d31d01ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff74bfff640 1 -- 172.21.15.94:0/2435356633 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760019070 con 0x5596ab55d4b0 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff74bfff640 1 -- 172.21.15.94:0/2435356633 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff760003b80 con 0x5596ab55d4b0 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.302+0000 7ff74bfff640 1 -- 172.21.15.94:0/2435356633 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76000e800 con 0x5596ab55d4b0 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 -- 172.21.15.94:0/2435356633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55d4b0 msgr2=0x5596ab559680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55d4b0 0x5596ab559680 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7ff760001bd0 tx=0x7ff760002090 comp rx=0 tx=0).stop 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 -- 172.21.15.94:0/2435356633 shutdown_connections 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55d4b0 0x5596ab559680 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55f330 0x5596ab55cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2435356633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55ef50 0x5596ab55ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 -- 172.21.15.94:0/2435356633 >> 172.21.15.94:0/2435356633 conn(0x5596ab55b7d0 msgr2=0x5596ab5803c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 -- 172.21.15.94:0/2435356633 shutdown_connections 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.303+0000 7ff7731f7740 1 -- 172.21.15.94:0/2435356633 wait complete. 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.304+0000 7ff7731f7740 1 -- start start 2024-03-06T07:25:47.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.304+0000 7ff7731f7740 1 Processor -- start 2024-03-06T07:25:47.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab57d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab5aba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab57aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ab581230 con 0x5596ab5abf40 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ab47fea0 con 0x5596ab55f330 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596ab580c70 con 0x5596ab55ef50 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 10 client.0 _ll_register_callbacks cb 0x5596ab47f870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab57aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab57aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41962/0 (socket says 172.21.15.94:41962) 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 -- 172.21.15.94:0/2251720431 learned_addr learned my addr 172.21.15.94:0/2251720431 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76dba9640 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab5aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab57d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 -- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 msgr2=0x5596ab57d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab57d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 -- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 msgr2=0x5596ab5aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab5aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 -- 172.21.15.94:0/2251720431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596ab480270 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff76ebab640 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab57aa60 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x5596ab59e5d0 tx=0x7ff760001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2646ece85bbbdd9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7497fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7497fa640 1 -- 172.21.15.94:0/2251720431 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760022060 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7497fa640 1 -- 172.21.15.94:0/2251720431 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff76000e800 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.305+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff760003b80 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7497fa640 1 -- 172.21.15.94:0/2251720431 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=171) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff760028050 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 10 client.10071 fetch_fsmap learned FSMap version 171 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=171,monmap=2+}) v3 -- 0x5596ab1d82d0 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7497fa640 1 -- 172.21.15.94:0/2251720431 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff760003b80 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7497fa640 1 -- 172.21.15.94:0/2251720431 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 171) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff76002b020 con 0x5596ab5abf40 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 10 client.10071 fetch_fsmap finished waiting for FSMap version 171 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 10 client.10071 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 20 client.10071 populate_metadata read hostname 'smithi094' 2024-03-06T07:25:47.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5af5a0 0x5596ab5ba340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 4 client.10071 mds_command: new command op to 37763 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 --> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596ab4d0bf0 con 0x5596ab5af5a0 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5af5a0 0x5596ab5ba340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff7497fa640 10 client.10071 ms_handle_connect on v2:172.21.15.145:6834/2375611510 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.306+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5af5a0 0x5596ab5ba340 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43bb921a8ffd173c server_cookie=f2a366b6c0cf20ec in_seq=0 out_seq=0 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7497fa640 1 -- 172.21.15.94:0/2251720431 <== mds.0 v2:172.21.15.145:6834/2375611510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5596ab4d0bf0 con 0x5596ab5af5a0 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7497fa640 10 client.10071 handle_command_reply: tid=0 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 client.10071 shutdown 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 msgr2=0x5596ab57aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab57aa60 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x5596ab59e5d0 tx=0x7ff760001250 comp rx=0 tx=0).stop 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 shutdown_connections 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5af5a0 0x5596ab5ba340 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab57aa60 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab5aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 --2- 172.21.15.94:0/2251720431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab57d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7497fa640 0 client.10071 ms_handle_reset on v2:172.21.15.145:6834/2375611510 2024-03-06T07:25:47.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 >> 172.21.15.94:0/2251720431 conn(0x5596ab55d4b0 msgr2=0x5596ab55e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 shutdown_connections 2024-03-06T07:25:47.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 1 -- 172.21.15.94:0/2251720431 wait complete. 2024-03-06T07:25:47.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.307+0000 7ff7731f7740 20 client.10071 trim_cache size 0 max 16384 2024-03-06T07:25:47.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.324+0000 7ff7731f7740 1 Processor -- start 2024-03-06T07:25:47.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.324+0000 7ff7731f7740 1 -- start start 2024-03-06T07:25:47.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab55d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab558510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab558a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff7731f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7681f7cd0 con 0x5596ab5abf40 2024-03-06T07:25:47.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7681f6e80 con 0x5596ab55f330 2024-03-06T07:25:47.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680c02e0 con 0x5596ab55ef50 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab558a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab558a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41972/0 (socket says 172.21.15.94:41972) 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 -- 172.21.15.94:0/531896140 learned_addr learned my addr 172.21.15.94:0/531896140 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76dba9640 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab558510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab55d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 -- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 msgr2=0x5596ab55d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab55d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 -- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 msgr2=0x5596ab558510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab558510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 -- 172.21.15.94:0/531896140 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596ab2a26b0 con 0x5596ab5abf40 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff76ebab640 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab558a50 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ff760001580 tx=0x7ff760010070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1713e27dd8a64bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff74b7fe640 1 -- 172.21.15.94:0/531896140 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760031070 con 0x5596ab5abf40 2024-03-06T07:25:47.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff74b7fe640 1 -- 172.21.15.94:0/531896140 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76000ed60 con 0x5596ab5abf40 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.325+0000 7ff74b7fe640 1 -- 172.21.15.94:0/531896140 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760019040 con 0x5596ab5abf40 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 -- 172.21.15.94:0/531896140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 msgr2=0x5596ab558a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab558a50 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ff760001580 tx=0x7ff760010070 comp rx=0 tx=0).stop 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 -- 172.21.15.94:0/531896140 shutdown_connections 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab5abf40 0x5596ab558a50 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab558510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 --2- 172.21.15.94:0/531896140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab55ef50 0x5596ab55d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 -- 172.21.15.94:0/531896140 >> 172.21.15.94:0/531896140 conn(0x5596ab55d4b0 msgr2=0x5596ab55e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 -- 172.21.15.94:0/531896140 shutdown_connections 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 -- 172.21.15.94:0/531896140 wait complete. 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 -- start start 2024-03-06T07:25:47.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.326+0000 7ff7731f7740 1 Processor -- start 2024-03-06T07:25:47.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55ef50 0x5596ab55dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab60e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff7731f7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab5abf40 0x5596ab5a7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff7731f7740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596ab584d30 con 0x5596ab55ef50 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7681f6af0 con 0x5596ab55f330 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff7731f7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596ab584b50 con 0x5596ab5abf40 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff7731f7740 10 client.0 _ll_register_callbacks cb 0x7ff7640094d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff76e3aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55ef50 0x5596ab55dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff76e3aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55ef50 0x5596ab55dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:41984/0 (socket says 172.21.15.94:41984) 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4235221155 learned_addr learned my addr 172.21.15.94:0/4235221155 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff76dba9640 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab60e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.327+0000 7ff76ebab640 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab5abf40 0x5596ab5a7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab5abf40 msgr2=0x5596ab5a7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab5abf40 0x5596ab5a7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 msgr2=0x5596ab60e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab60e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff76e3aa640 1 -- 172.21.15.94:0/4235221155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596ab5596f0 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff76e3aa640 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55ef50 0x5596ab55dc90 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7ff75c004770 tx=0x7ff75c0582a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b5445f0864c591 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7497fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7497fa640 1 -- 172.21.15.94:0/4235221155 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c020040 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7497fa640 1 -- 172.21.15.94:0/4235221155 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff75c054dd0 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff76000ed60 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7497fa640 1 -- 172.21.15.94:0/4235221155 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=171) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff75c057bb0 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 10 client.10077 fetch_fsmap learned FSMap version 171 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=171,monmap=2+}) v3 -- 0x5596ab60ae80 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7497fa640 1 -- 172.21.15.94:0/4235221155 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76000ed60 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7497fa640 1 -- 172.21.15.94:0/4235221155 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 171) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff75c012050 con 0x5596ab55ef50 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 10 client.10077 fetch_fsmap finished waiting for FSMap version 171 2024-03-06T07:25:47.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 10 client.10077 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 20 client.10077 populate_metadata read hostname 'smithi094' 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5bef60 0x5596ab607910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 4 client.10077 mds_command: new command op to 37763 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.328+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 --> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5596ab4caf40 con 0x5596ab5bef60 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff76dba9640 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5bef60 0x5596ab607910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7497fa640 10 client.10077 ms_handle_connect on v2:172.21.15.145:6834/2375611510 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff76dba9640 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5bef60 0x5596ab607910 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b8760dc1bd5c15 server_cookie=d4f3effee25647a3 in_seq=0 out_seq=0 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7497fa640 1 -- 172.21.15.94:0/4235221155 <== mds.0 v2:172.21.15.145:6834/2375611510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x5596ab4caf40 con 0x5596ab5bef60 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7497fa640 10 client.10077 handle_command_reply: tid=0 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 client.10077 shutdown 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55ef50 msgr2=0x5596ab55dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55ef50 0x5596ab55dc90 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7ff75c004770 tx=0x7ff75c0582a0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 shutdown_connections 2024-03-06T07:25:47.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:6834/2375611510,v1:172.21.15.145:6836/2375611510] conn(0x5596ab5bef60 0x5596ab607910 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5596ab5abf40 0x5596ab5a7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5596ab55f330 0x5596ab60e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7497fa640 0 client.10077 ms_handle_reset on v2:172.21.15.145:6834/2375611510 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 --2- 172.21.15.94:0/4235221155 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5596ab55ef50 0x5596ab55dc90 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 >> 172.21.15.94:0/4235221155 conn(0x5596ab55d4b0 msgr2=0x5596ab560050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 shutdown_connections 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.329+0000 7ff7731f7740 1 -- 172.21.15.94:0/4235221155 wait complete. 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff7731f7740 20 client.10077 trim_cache size 0 max 16384 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:25:47.338 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 37763, 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 8.9665845730000004, 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 171, 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 182, 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 182, 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 14.064331562 2024-03-06T07:25:47.339 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:25:47.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74c06b4d0 msgr2=0x7ff74c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74c06b4d0 0x7ff74c06d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff764036000 tx=0x7ff764008030 comp rx=0 tx=0).stop 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff768072f00 msgr2=0x7ff768156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff768072f00 0x7ff768156cf0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00e810 tx=0x7ff75c00ecd0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 shutdown_connections 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74c06b4d0 0x7ff74c06d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7680c6d80 0x7ff7681eec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680c0fa0 0x7ff76814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 --2- 172.21.15.94:0/4087177674 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff768072f00 0x7ff768156cf0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 >> 172.21.15.94:0/4087177674 conn(0x7ff76806b1a0 msgr2=0x7ff768153890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 shutdown_connections 2024-03-06T07:25:47.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.330+0000 7ff77060e640 1 -- 172.21.15.94:0/4087177674 wait complete. 2024-03-06T07:25:47.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:25:47.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.445+0000 7f9f4261c640 1 Processor -- start 2024-03-06T07:25:47.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.445+0000 7f9f4261c640 1 -- start start 2024-03-06T07:25:47.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f4261c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 0x7f9f3c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f4261c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0b0450 con 0x7f9f3c0c5380 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f4261c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c1560e0 con 0x7f9f3c0b0070 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f4261c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06cfb0 con 0x7f9f3c155c30 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f40bb9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9f3c0c5380 0x7f9f3c14e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58522/0 (socket says 172.21.15.94:58522) 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f40bb9640 1 -- 172.21.15.94:0/603080549 learned_addr learned my addr 172.21.15.94:0/603080549 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3b7fe640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 0x7f9f3c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912493751 0 0) 0x7f9f34001490 con 0x7f9f3c0c5380 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f24003520 con 0x7f9f3c0c5380 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45258804 0 0) 0x7f9f3c1560e0 con 0x7f9f3c0b0070 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f34001490 con 0x7f9f3c0b0070 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1582744289 0 0) 0x7f9f24003520 con 0x7f9f3c0c5380 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 msgr2=0x7f9f3c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 0x7f9f3c06ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 >> v1:172.21.15.145:6789/0 conn(0x7f9f3c0b0070 legacy=0x7f9f3c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c0c77f0 con 0x7f9f3c0c5380 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9f34001bc0 con 0x7f9f3c0c5380 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 >> v1:172.21.15.94:6789/0 conn(0x7f9f3c0c5380 legacy=0x7f9f3c14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f24003b70 0x7f9f24003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 0x7f9f24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f24004d40 0x7f9f240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06cfb0 con 0x7f9f24004d40 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2400baf0 con 0x7f9f24003b70 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.446+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2400bcf0 con 0x7f9f3c155c30 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f40bb9640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f24004d40 0x7f9f240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f3bfff640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f24003b70 0x7f9f24003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f3b7fe640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 0x7f9f24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f40bb9640 1 -- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 msgr2=0x7f9f24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f40bb9640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c155c30 0x7f9f24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f40bb9640 1 -- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f24003b70 msgr2=0x7f9f24003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f40bb9640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f24003b70 0x7f9f24003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f40bb9640 1 -- 172.21.15.94:0/603080549 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f24003520 con 0x7f9f24004d40 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.447+0000 7f9f40bb9640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f24004d40 0x7f9f240051c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f9f3400a9c0 tx=0x7f9f24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9261196080101916 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3400a5e0 con 0x7f9f24004d40 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f34035420 con 0x7f9f24004d40 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f3affd640 1 -- 172.21.15.94:0/603080549 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34035830 con 0x7f9f24004d40 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f4261c640 1 -- 172.21.15.94:0/603080549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f24004d40 msgr2=0x7f9f240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f4261c640 1 --2- 172.21.15.94:0/603080549 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f24004d40 0x7f9f240051c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f9f3400a9c0 tx=0x7f9f24005830 comp rx=0 tx=0).stop 2024-03-06T07:25:47.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f40bb9640 1 -- 172.21.15.94:0/603080549 reap_dead start 2024-03-06T07:25:47.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f4261c640 1 -- 172.21.15.94:0/603080549 shutdown_connections 2024-03-06T07:25:47.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f4261c640 1 -- 172.21.15.94:0/603080549 >> 172.21.15.94:0/603080549 conn(0x7f9f3c0ac590 msgr2=0x7f9f3c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f4261c640 1 -- 172.21.15.94:0/603080549 shutdown_connections 2024-03-06T07:25:47.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.448+0000 7f9f4261c640 1 -- 172.21.15.94:0/603080549 wait complete. 2024-03-06T07:25:47.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.449+0000 7f9f4261c640 1 Processor -- start 2024-03-06T07:25:47.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f4261c640 1 -- start start 2024-03-06T07:25:47.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f4261c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c1ee070 0x7f9f3c0c1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f4261c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f3c1ee450 0x7f9f3c0c2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f4261c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f3c0c1000 0x7f9f3c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f4261c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06c270 con 0x7f9f3c1ee450 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f4261c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06c020 con 0x7f9f3c0c1000 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f4261c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c06c550 con 0x7f9f3c1ee070 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f3b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f3c1ee450 0x7f9f3c0c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f3b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f3c1ee450 0x7f9f3c0c2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42006/0 (socket says 172.21.15.94:42006) 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f3bfff640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c1ee070 0x7f9f3c0c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f3b7fe640 1 -- 172.21.15.94:0/2582612568 learned_addr learned my addr 172.21.15.94:0/2582612568 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.450+0000 7f9f40bb9640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f3c0c1000 0x7f9f3c0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f3b7fe640 1 -- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c1ee070 msgr2=0x7f9f3c0c1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f3b7fe640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c1ee070 0x7f9f3c0c1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f3b7fe640 1 -- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f3c0c1000 msgr2=0x7f9f3c0c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f3b7fe640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f3c0c1000 0x7f9f3c0c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f3b7fe640 1 -- 172.21.15.94:0/2582612568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c159100 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f3b7fe640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f3c1ee450 0x7f9f3c0c2630 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f9f300119a0 tx=0x7f9f30011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f38ff9640 1 -- 172.21.15.94:0/2582612568 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3000f9d0 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f38ff9640 1 -- 172.21.15.94:0/2582612568 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3000fb70 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f38ff9640 1 -- 172.21.15.94:0/2582612568 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30014440 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3c1ee9b0 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.451+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3c1eec90 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.452+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3c04f330 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.452+0000 7f9f38ff9640 1 -- 172.21.15.94:0/2582612568 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f30012040 con 0x7f9f3c1ee450 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.453+0000 7f9f38ff9640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f2806b500 0x7f9f2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.453+0000 7f9f38ff9640 1 -- 172.21.15.94:0/2582612568 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9f3008a720 con 0x7f9f3c1ee450 2024-03-06T07:25:47.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.453+0000 7f9f3bfff640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f2806b500 0x7f9f2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.453+0000 7f9f3bfff640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f2806b500 0x7f9f2806d9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0018d0 tx=0x7f9f2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.457+0000 7f9f38ff9640 1 -- 172.21.15.94:0/2582612568 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f30019030 con 0x7f9f3c1ee450 2024-03-06T07:25:47.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:47.664+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f9f3c06d860 con 0x7f9f3c1ee450 2024-03-06T07:25:48.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.442+0000 7f9f38ff9640 1 -- 172.21.15.94:0/2582612568 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v172) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9f300594a0 con 0x7f9f3c1ee450 2024-03-06T07:25:48.444 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked down. 2024-03-06T07:25:48.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.444+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f2806b500 msgr2=0x7f9f2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.444+0000 7f9f4261c640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f2806b500 0x7f9f2806d9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0018d0 tx=0x7f9f2c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.444+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f3c1ee450 msgr2=0x7f9f3c0c2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.444+0000 7f9f4261c640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f3c1ee450 0x7f9f3c0c2630 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f9f300119a0 tx=0x7f9f30011da0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.444+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 shutdown_connections 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.444+0000 7f9f4261c640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f2806b500 0x7f9f2806d9c0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.444+0000 7f9f4261c640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f3c0c1000 0x7f9f3c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.445+0000 7f9f4261c640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f3c1ee450 0x7f9f3c0c2630 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.445+0000 7f9f4261c640 1 --2- 172.21.15.94:0/2582612568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f3c1ee070 0x7f9f3c0c1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.445+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 >> 172.21.15.94:0/2582612568 conn(0x7f9f3c0ac590 msgr2=0x7f9f3c0aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.445+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 shutdown_connections 2024-03-06T07:25:48.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.445+0000 7f9f4261c640 1 -- 172.21.15.94:0/2582612568 wait complete. 2024-03-06T07:25:48.460 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:48.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.564+0000 7f272ba39640 1 Processor -- start 2024-03-06T07:25:48.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f272ba39640 1 -- start start 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f272ba39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f27240c88c0 0x7f2724155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f272ba39640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2724158b60 0x7f2724158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f272ba39640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27240c73d0 con 0x7f272414f360 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f272ba39640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27240c8dd0 con 0x7f2724158b60 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f272ba39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27240be2e0 con 0x7f27240c88c0 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f27297d5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f272414f360 0x7f27240c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50980/0 (socket says 172.21.15.94:50980) 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f27297d5640 1 -- 172.21.15.94:0/2949960905 learned_addr learned my addr 172.21.15.94:0/2949960905 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:48.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.565+0000 7f2729fd6640 1 --2- 172.21.15.94:0/2949960905 >> v2:172.21.15.94:3300/0 conn(0x7f2724158b60 0x7f2724158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2723fff640 1 -- 172.21.15.94:0/2949960905 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785280265 0 0) 0x7f27240c73d0 con 0x7f272414f360 2024-03-06T07:25:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2728fd4640 1 --2- 172.21.15.94:0/2949960905 >> v2:172.21.15.145:3300/0 conn(0x7f27240c88c0 0x7f2724155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2723fff640 1 -- 172.21.15.94:0/2949960905 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c003520 con 0x7f272414f360 2024-03-06T07:25:48.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2729fd6640 1 -- 172.21.15.94:0/2949960905 >> v2:172.21.15.145:3300/0 conn(0x7f27240c88c0 msgr2=0x7f2724155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2729fd6640 1 --2- 172.21.15.94:0/2949960905 >> v2:172.21.15.145:3300/0 conn(0x7f27240c88c0 0x7f2724155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2729fd6640 1 -- 172.21.15.94:0/2949960905 >> v1:172.21.15.94:6789/0 conn(0x7f272414f360 legacy=0x7f27240c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2729fd6640 1 -- 172.21.15.94:0/2949960905 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27240c7670 con 0x7f2724158b60 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.566+0000 7f2729fd6640 1 --2- 172.21.15.94:0/2949960905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2724158b60 0x7f2724158ff0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f27240c84c0 tx=0x7f271402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58f73410374eeed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f2723fff640 1 -- 172.21.15.94:0/2949960905 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140315a0 con 0x7f2724158b60 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f2723fff640 1 -- 172.21.15.94:0/2949960905 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714037070 con 0x7f2724158b60 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f2723fff640 1 -- 172.21.15.94:0/2949960905 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714031940 con 0x7f2724158b60 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 -- 172.21.15.94:0/2949960905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2724158b60 msgr2=0x7f2724158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 --2- 172.21.15.94:0/2949960905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2724158b60 0x7f2724158ff0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f27240c84c0 tx=0x7f271402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 -- 172.21.15.94:0/2949960905 shutdown_connections 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 --2- 172.21.15.94:0/2949960905 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2724158b60 0x7f2724158ff0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 --2- 172.21.15.94:0/2949960905 >> v2:172.21.15.145:3300/0 conn(0x7f27240c88c0 0x7f2724155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 -- 172.21.15.94:0/2949960905 >> 172.21.15.94:0/2949960905 conn(0x7f27240ac570 msgr2=0x7f27240ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 -- 172.21.15.94:0/2949960905 shutdown_connections 2024-03-06T07:25:48.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.567+0000 7f272ba39640 1 -- 172.21.15.94:0/2949960905 wait complete. 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.568+0000 7f272ba39640 1 Processor -- start 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f272ba39640 1 -- start start 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f272ba39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27240c88c0 0x7f272414bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f272ba39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272414f360 0x7f27241ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f272ba39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2724158b60 0x7f27240ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f272ba39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27240c5a50 con 0x7f27240c88c0 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f272ba39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27240c7c30 con 0x7f2724158b60 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f272ba39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27240c5bd0 con 0x7f272414f360 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f27297d5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27240c88c0 0x7f272414bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f27297d5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27240c88c0 0x7f272414bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44054/0 (socket says 172.21.15.94:44054) 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.569+0000 7f27297d5640 1 -- 172.21.15.94:0/2375057627 learned_addr learned my addr 172.21.15.94:0/2375057627 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f2728fd4640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272414f360 0x7f27241ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f2729fd6640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2724158b60 0x7f27240ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f27297d5640 1 -- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272414f360 msgr2=0x7f27241ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f27297d5640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272414f360 0x7f27241ee050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f27297d5640 1 -- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2724158b60 msgr2=0x7f27240ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f27297d5640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2724158b60 0x7f27240ad320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f27297d5640 1 -- 172.21.15.94:0/2375057627 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27241eead0 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f27297d5640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27240c88c0 0x7f272414bab0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f271c00cb40 tx=0x7f271c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f2721ffb640 1 -- 172.21.15.94:0/2375057627 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c017070 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f2721ffb640 1 -- 172.21.15.94:0/2375057627 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c00aca0 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27240bf080 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f2721ffb640 1 -- 172.21.15.94:0/2375057627 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c012620 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.570+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27240c9bb0 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.571+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272404f330 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.571+0000 7f2721ffb640 1 -- 172.21.15.94:0/2375057627 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f271c01a070 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.572+0000 7f2721ffb640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.572+0000 7f2721ffb640 1 -- 172.21.15.94:0/2375057627 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f271c08a130 con 0x7f27240c88c0 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.572+0000 7f2728fd4640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.572+0000 7f2728fd4640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f271006b500 0x7f271006d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2718006fd0 tx=0x7f2718008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.576+0000 7f2721ffb640 1 -- 172.21.15.94:0/2375057627 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f271c058eb0 con 0x7f27240c88c0 2024-03-06T07:25:48.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.782+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f272406cda0 con 0x7f27240c88c0 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.783+0000 7f2721ffb640 1 -- 172.21.15.94:0/2375057627 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f271c0587b0 con 0x7f27240c88c0 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:25:48.785 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f271006b500 msgr2=0x7f271006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f271006b500 0x7f271006d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2718006fd0 tx=0x7f2718008040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27240c88c0 msgr2=0x7f272414bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27240c88c0 0x7f272414bab0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f271c00cb40 tx=0x7f271c00a060 comp rx=0 tx=0).stop 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 shutdown_connections 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f271006b500 0x7f271006d9c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2724158b60 0x7f27240ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272414f360 0x7f27241ee050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 --2- 172.21.15.94:0/2375057627 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27240c88c0 0x7f272414bab0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 >> 172.21.15.94:0/2375057627 conn(0x7f27240ac570 msgr2=0x7f272414d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 shutdown_connections 2024-03-06T07:25:48.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:48.785+0000 7f272ba39640 1 -- 172.21.15.94:0/2375057627 wait complete. 2024-03-06T07:25:49.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:49.200 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:25:50.200 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:50.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.311+0000 7fc3beda0640 1 Processor -- start 2024-03-06T07:25:50.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.311+0000 7fc3beda0640 1 -- start start 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3beda0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3b80c29b0 0x7fc3b80c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3beda0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 0x7fc3b81557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3beda0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b8158f30 con 0x7fc3b80c5590 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3beda0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b80c3620 con 0x7fc3b80c29b0 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3beda0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c5e00 con 0x7fc3b80c5970 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3bd59d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3b80c29b0 0x7fc3b80c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3bd59d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3b80c29b0 0x7fc3b80c4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44062/0 (socket says 172.21.15.94:44062) 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1307161610 learned_addr learned my addr 172.21.15.94:0/1307161610 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1307161610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 msgr2=0x7fc3b81557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3bd59d640 1 --2- 172.21.15.94:0/1307161610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 0x7fc3b81557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1307161610 >> v1:172.21.15.145:6789/0 conn(0x7fc3b80c5590 legacy=0x7fc3b81550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.312+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1307161610 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b81591d0 con 0x7fc3b80c29b0 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3bd59d640 1 --2- 172.21.15.94:0/1307161610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c29b0 0x7fc3b80c4c70 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009870 tx=0x7fc3b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90dbebda69455ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3bcd9c640 1 -- 172.21.15.94:0/1307161610 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4035020 con 0x7fc3b80c29b0 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3bcd9c640 1 -- 172.21.15.94:0/1307161610 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b4031640 con 0x7fc3b80c29b0 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3bcd9c640 1 -- 172.21.15.94:0/1307161610 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b402d040 con 0x7fc3b80c29b0 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3beda0640 1 -- 172.21.15.94:0/1307161610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c29b0 msgr2=0x7fc3b80c4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1307161610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c29b0 0x7fc3b80c4c70 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009870 tx=0x7fc3b402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3beda0640 1 -- 172.21.15.94:0/1307161610 shutdown_connections 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1307161610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 0x7fc3b81557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1307161610 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c29b0 0x7fc3b80c4c70 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.313+0000 7fc3beda0640 1 -- 172.21.15.94:0/1307161610 >> 172.21.15.94:0/1307161610 conn(0x7fc3b80c8b20 msgr2=0x7fc3b80c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.314+0000 7fc3beda0640 1 -- 172.21.15.94:0/1307161610 shutdown_connections 2024-03-06T07:25:50.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.314+0000 7fc3beda0640 1 -- 172.21.15.94:0/1307161610 wait complete. 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 Processor -- start 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 -- start start 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3b80c29b0 0x7fc3b8149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c5590 0x7fc3b8070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 0x7fc3b8157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b814f330 con 0x7fc3b80c5590 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8155d00 con 0x7fc3b80c29b0 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3beda0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c66e0 con 0x7fc3b80c5970 2024-03-06T07:25:50.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3bd59d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c5590 0x7fc3b8070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3bd59d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c5590 0x7fc3b8070500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44076/0 (socket says 172.21.15.94:44076) 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.315+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1518274724 learned_addr learned my addr 172.21.15.94:0/1518274724 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3bdd9e640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3b80c29b0 0x7fc3b8149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3be59f640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 0x7fc3b8157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 msgr2=0x7fc3b8157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3bd59d640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 0x7fc3b8157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3b80c29b0 msgr2=0x7fc3b8149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3bd59d640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3b80c29b0 0x7fc3b8149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3bd59d640 1 -- 172.21.15.94:0/1518274724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b8158260 con 0x7fc3b80c5590 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3bd59d640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c5590 0x7fc3b8070500 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc3b402b930 tx=0x7fc3b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3ae7fc640 1 -- 172.21.15.94:0/1518274724 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b403ac60 con 0x7fc3b80c5590 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3ae7fc640 1 -- 172.21.15.94:0/1518274724 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b403ae20 con 0x7fc3b80c5590 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b81ee220 con 0x7fc3b80c5590 2024-03-06T07:25:50.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3ae7fc640 1 -- 172.21.15.94:0/1518274724 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4044630 con 0x7fc3b80c5590 2024-03-06T07:25:50.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.316+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b8070a40 con 0x7fc3b80c5590 2024-03-06T07:25:50.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.317+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b8069430 con 0x7fc3b80c5590 2024-03-06T07:25:50.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.318+0000 7fc3ae7fc640 1 -- 172.21.15.94:0/1518274724 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3b404b060 con 0x7fc3b80c5590 2024-03-06T07:25:50.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.318+0000 7fc3ae7fc640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc39c06b500 0x7fc39c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.318+0000 7fc3ae7fc640 1 -- 172.21.15.94:0/1518274724 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc3b40ac070 con 0x7fc3b80c5590 2024-03-06T07:25:50.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.318+0000 7fc3bdd9e640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc39c06b500 0x7fc39c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.319+0000 7fc3bdd9e640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc39c06b500 0x7fc39c06d9c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8004ea0 tx=0x7fc3a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.322+0000 7fc3ae7fc640 1 -- 172.21.15.94:0/1518274724 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3b403f070 con 0x7fc3b80c5590 2024-03-06T07:25:50.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.530+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3b80c2790 con 0x7fc3b80c5590 2024-03-06T07:25:50.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.531+0000 7fc3ae7fc640 1 -- 172.21.15.94:0/1518274724 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fc3b407ad60 con 0x7fc3b80c5590 2024-03-06T07:25:50.533 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:50.533 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:25:50.533 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc39c06b500 msgr2=0x7fc39c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc39c06b500 0x7fc39c06d9c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8004ea0 tx=0x7fc3a8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c5590 msgr2=0x7fc3b8070500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c5590 0x7fc3b8070500 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fc3b402b930 tx=0x7fc3b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 shutdown_connections 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc39c06b500 0x7fc39c06d9c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3b80c5970 0x7fc3b8157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3b80c5590 0x7fc3b8070500 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 --2- 172.21.15.94:0/1518274724 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3b80c29b0 0x7fc3b8149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 >> 172.21.15.94:0/1518274724 conn(0x7fc3b80c8b20 msgr2=0x7fc3b814e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 shutdown_connections 2024-03-06T07:25:50.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:50.533+0000 7fc3beda0640 1 -- 172.21.15.94:0/1518274724 wait complete. 2024-03-06T07:25:50.948 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:50.949 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:25:51.949 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:52.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.059+0000 7fd0096c9640 1 Processor -- start 2024-03-06T07:25:52.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.059+0000 7fd0096c9640 1 -- start start 2024-03-06T07:25:52.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0096c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd0040c5330 0x7fd0040c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0096c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd004158db0 0x7fd00415fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0096c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd004155110 con 0x7fd0040c88b0 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0096c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0041559e0 con 0x7fd004158db0 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0096c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0040c8f40 con 0x7fd0040c5330 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0037fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd004158db0 0x7fd00415fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0037fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd004158db0 0x7fd00415fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44080/0 (socket says 172.21.15.94:44080) 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0037fe640 1 -- 172.21.15.94:0/3832617048 learned_addr learned my addr 172.21.15.94:0/3832617048 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd002ffd640 1 --2- 172.21.15.94:0/3832617048 >> v2:172.21.15.145:3300/0 conn(0x7fd0040c5330 0x7fd0040c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0037fe640 1 -- 172.21.15.94:0/3832617048 >> v2:172.21.15.145:3300/0 conn(0x7fd0040c5330 msgr2=0x7fd0040c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0037fe640 1 --2- 172.21.15.94:0/3832617048 >> v2:172.21.15.145:3300/0 conn(0x7fd0040c5330 0x7fd0040c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0037fe640 1 -- 172.21.15.94:0/3832617048 >> v1:172.21.15.145:6789/0 conn(0x7fd0040c88b0 legacy=0x7fd0041586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.060+0000 7fd0037fe640 1 -- 172.21.15.94:0/3832617048 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004155390 con 0x7fd004158db0 2024-03-06T07:25:52.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.061+0000 7fd0037fe640 1 --2- 172.21.15.94:0/3832617048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd004158db0 0x7fd00415fa30 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fd00408c6e0 tx=0x7fcffc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb254597ab65b318 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.061+0000 7fd0027fc640 1 -- 172.21.15.94:0/3832617048 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc02c9f0 con 0x7fd004158db0 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.061+0000 7fd0027fc640 1 -- 172.21.15.94:0/3832617048 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc02cb90 con 0x7fd004158db0 2024-03-06T07:25:52.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.061+0000 7fd0027fc640 1 -- 172.21.15.94:0/3832617048 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc0343f0 con 0x7fd004158db0 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 -- 172.21.15.94:0/3832617048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd004158db0 msgr2=0x7fd00415fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 --2- 172.21.15.94:0/3832617048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd004158db0 0x7fd00415fa30 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fd00408c6e0 tx=0x7fcffc02be50 comp rx=0 tx=0).stop 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 -- 172.21.15.94:0/3832617048 shutdown_connections 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 --2- 172.21.15.94:0/3832617048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd004158db0 0x7fd00415fa30 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 --2- 172.21.15.94:0/3832617048 >> v2:172.21.15.145:3300/0 conn(0x7fd0040c5330 0x7fd0040c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 -- 172.21.15.94:0/3832617048 >> 172.21.15.94:0/3832617048 conn(0x7fd0040ac560 msgr2=0x7fd0040ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 -- 172.21.15.94:0/3832617048 shutdown_connections 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.062+0000 7fd0096c9640 1 -- 172.21.15.94:0/3832617048 wait complete. 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.063+0000 7fd0096c9640 1 Processor -- start 2024-03-06T07:25:52.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.063+0000 7fd0096c9640 1 -- start start 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0040c5330 0x7fd0040c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0040c88b0 0x7fd0041eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd00414ebc0 0x7fd0041514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041529d0 con 0x7fd0040c88b0 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00406d090 con 0x7fd00414ebc0 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd004152b50 con 0x7fd0040c5330 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0040c88b0 0x7fd0041eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0040c88b0 0x7fd0041eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44082/0 (socket says 172.21.15.94:44082) 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 -- 172.21.15.94:0/4278587369 learned_addr learned my addr 172.21.15.94:0/4278587369 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0037fe640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd00414ebc0 0x7fd0041514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd002ffd640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0040c5330 0x7fd0040c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 -- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0040c5330 msgr2=0x7fd0040c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0040c5330 0x7fd0040c1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 -- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd00414ebc0 msgr2=0x7fd0041514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd00414ebc0 0x7fd0041514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 -- 172.21.15.94:0/4278587369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0041eb5c0 con 0x7fd0040c88b0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0037fe640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd00414ebc0 0x7fd0041514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffa7fc640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0040c88b0 0x7fd0041eb080 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00c400 tx=0x7fcfec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffbfff640 1 -- 172.21.15.94:0/4278587369 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec017070 con 0x7fd0040c88b0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffbfff640 1 -- 172.21.15.94:0/4278587369 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec0035f0 con 0x7fd0040c88b0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00414e130 con 0x7fd0040c88b0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fcffbfff640 1 -- 172.21.15.94:0/4278587369 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec003a20 con 0x7fd0040c88b0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.064+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00414cd10 con 0x7fd0040c88b0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.066+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00404f330 con 0x7fd0040c88b0 2024-03-06T07:25:52.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.066+0000 7fcffbfff640 1 -- 172.21.15.94:0/4278587369 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfec01a070 con 0x7fd0040c88b0 2024-03-06T07:25:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.066+0000 7fcffbfff640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.067+0000 7fcffbfff640 1 -- 172.21.15.94:0/4278587369 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcfec089620 con 0x7fd0040c88b0 2024-03-06T07:25:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.067+0000 7fd002ffd640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.067+0000 7fd002ffd640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfe406b4d0 0x7fcfe406d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcff4006fd0 tx=0x7fcff4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.071+0000 7fcffbfff640 1 -- 172.21.15.94:0/4278587369 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfec057d50 con 0x7fd0040c88b0 2024-03-06T07:25:52.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.279+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0041519f0 con 0x7fd0040c88b0 2024-03-06T07:25:52.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.279+0000 7fcffbfff640 1 -- 172.21.15.94:0/4278587369 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fcfec05bdf0 con 0x7fd0040c88b0 2024-03-06T07:25:52.281 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:52.281 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:25:52.281 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfe406b4d0 msgr2=0x7fcfe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfe406b4d0 0x7fcfe406d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcff4006fd0 tx=0x7fcff4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0040c88b0 msgr2=0x7fd0041eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0040c88b0 0x7fd0041eb080 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fcfec00c400 tx=0x7fcfec00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 shutdown_connections 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd00414ebc0 0x7fd0041514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd0040c88b0 0x7fd0041eb080 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 --2- 172.21.15.94:0/4278587369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0040c5330 0x7fd0040c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 >> 172.21.15.94:0/4278587369 conn(0x7fd0040ac560 msgr2=0x7fd00406c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 shutdown_connections 2024-03-06T07:25:52.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:52.281+0000 7fd0096c9640 1 -- 172.21.15.94:0/4278587369 wait complete. 2024-03-06T07:25:52.696 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:52.696 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:25:53.697 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:53.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.806+0000 7f1217a78640 1 Processor -- start 2024-03-06T07:25:53.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.807+0000 7f1217a78640 1 -- start start 2024-03-06T07:25:53.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.807+0000 7f1217a78640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f121014f360 0x7f1210155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.807+0000 7f1217a78640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f12100c88c0 0x7f12100c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.807+0000 7f1217a78640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12100c73d0 con 0x7f12100c5880 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.807+0000 7f1217a78640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12100c8dd0 con 0x7f12100c88c0 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.807+0000 7f1217a78640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12100be2e0 con 0x7f121014f360 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.807+0000 7f1215013640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f12100c88c0 0x7f12100c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.808+0000 7f1215013640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f12100c88c0 0x7f12100c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44084/0 (socket says 172.21.15.94:44084) 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.808+0000 7f1215013640 1 -- 172.21.15.94:0/3984110110 learned_addr learned my addr 172.21.15.94:0/3984110110 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.808+0000 7f1215814640 1 --2- 172.21.15.94:0/3984110110 >> v2:172.21.15.145:3300/0 conn(0x7f121014f360 0x7f1210155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.808+0000 7f1215013640 1 -- 172.21.15.94:0/3984110110 >> v2:172.21.15.145:3300/0 conn(0x7f121014f360 msgr2=0x7f1210155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.808+0000 7f1215013640 1 --2- 172.21.15.94:0/3984110110 >> v2:172.21.15.145:3300/0 conn(0x7f121014f360 0x7f1210155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.808+0000 7f1215013640 1 -- 172.21.15.94:0/3984110110 >> v1:172.21.15.145:6789/0 conn(0x7f12100c5880 legacy=0x7f1210158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.808+0000 7f1215013640 1 -- 172.21.15.94:0/3984110110 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12100c7670 con 0x7f12100c88c0 2024-03-06T07:25:53.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1215013640 1 --2- 172.21.15.94:0/3984110110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100c5340 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f121008c840 tx=0x7f120002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed719a17c4c7510 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1214812640 1 -- 172.21.15.94:0/3984110110 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120002e070 con 0x7f12100c88c0 2024-03-06T07:25:53.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1214812640 1 -- 172.21.15.94:0/3984110110 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1200037070 con 0x7f12100c88c0 2024-03-06T07:25:53.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1214812640 1 -- 172.21.15.94:0/3984110110 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200031a00 con 0x7f12100c88c0 2024-03-06T07:25:53.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 -- 172.21.15.94:0/3984110110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 msgr2=0x7f12100c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 --2- 172.21.15.94:0/3984110110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100c5340 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f121008c840 tx=0x7f120002b880 comp rx=0 tx=0).stop 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 -- 172.21.15.94:0/3984110110 shutdown_connections 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 --2- 172.21.15.94:0/3984110110 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100c5340 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 --2- 172.21.15.94:0/3984110110 >> v2:172.21.15.145:3300/0 conn(0x7f121014f360 0x7f1210155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 -- 172.21.15.94:0/3984110110 >> 172.21.15.94:0/3984110110 conn(0x7f12100ac570 msgr2=0x7f12100ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 -- 172.21.15.94:0/3984110110 shutdown_connections 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.809+0000 7f1217a78640 1 -- 172.21.15.94:0/3984110110 wait complete. 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.810+0000 7f1217a78640 1 Processor -- start 2024-03-06T07:25:53.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1217a78640 1 -- start start 2024-03-06T07:25:53.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1217a78640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12100c5880 0x7f12101eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1217a78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1217a78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f121014f360 0x7f1210153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1217a78640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1210159270 con 0x7f12100c88c0 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1217a78640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100c7c30 con 0x7f12100c5880 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1217a78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12101593f0 con 0x7f121014f360 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1215013640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1215013640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44096/0 (socket says 172.21.15.94:44096) 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1215013640 1 -- 172.21.15.94:0/2276812213 learned_addr learned my addr 172.21.15.94:0/2276812213 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1215814640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12100c5880 0x7f12101eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.811+0000 7f1216015640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f121014f360 0x7f1210153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1215013640 1 -- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f121014f360 msgr2=0x7f1210153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1215013640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f121014f360 0x7f1210153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1215013640 1 -- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12100c5880 msgr2=0x7f12101eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1215013640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12100c5880 0x7f12101eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1215013640 1 -- 172.21.15.94:0/2276812213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12100af380 con 0x7f12100c88c0 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1215013640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100aee40 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f120002b8c0 tx=0x7f12000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f12067fc640 1 -- 172.21.15.94:0/2276812213 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200032670 con 0x7f12100c88c0 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f12067fc640 1 -- 172.21.15.94:0/2276812213 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1200032830 con 0x7f12100c88c0 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1210153330 con 0x7f12100c88c0 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f12067fc640 1 -- 172.21.15.94:0/2276812213 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200032c20 con 0x7f12100c88c0 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.812+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12100aff80 con 0x7f12100c88c0 2024-03-06T07:25:53.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.813+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12100aebd0 con 0x7f12100c88c0 2024-03-06T07:25:53.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.813+0000 7f12067fc640 1 -- 172.21.15.94:0/2276812213 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f120004c060 con 0x7f12100c88c0 2024-03-06T07:25:53.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.814+0000 7f12067fc640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11f406b4d0 0x7f11f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.814+0000 7f12067fc640 1 -- 172.21.15.94:0/2276812213 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f12000abd00 con 0x7f12100c88c0 2024-03-06T07:25:53.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.814+0000 7f1215814640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11f406b4d0 0x7f11f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.815+0000 7f1215814640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11f406b4d0 0x7f11f406d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f120c003aa0 tx=0x7f120c009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:53.818+0000 7f12067fc640 1 -- 172.21.15.94:0/2276812213 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12100aebd0 con 0x7f12100c88c0 2024-03-06T07:25:54.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.026+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f121004f330 con 0x7f12100c88c0 2024-03-06T07:25:54.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.027+0000 7f12067fc640 1 -- 172.21.15.94:0/2276812213 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f1200040070 con 0x7f12100c88c0 2024-03-06T07:25:54.029 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:54.029 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:25:54.029 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:25:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11f406b4d0 msgr2=0x7f11f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11f406b4d0 0x7f11f406d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f120c003aa0 tx=0x7f120c009830 comp rx=0 tx=0).stop 2024-03-06T07:25:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 msgr2=0x7f12100aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100aee40 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f120002b8c0 tx=0x7f12000046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 shutdown_connections 2024-03-06T07:25:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f11f406b4d0 0x7f11f406d990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f121014f360 0x7f1210153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12100c88c0 0x7f12100aee40 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 --2- 172.21.15.94:0/2276812213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12100c5880 0x7f12101eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 >> 172.21.15.94:0/2276812213 conn(0x7f12100ac570 msgr2=0x7f121006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 shutdown_connections 2024-03-06T07:25:54.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:54.029+0000 7f1217a78640 1 -- 172.21.15.94:0/2276812213 wait complete. 2024-03-06T07:25:54.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:54.444 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:25:55.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:55.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.555+0000 7fab63a22640 1 Processor -- start 2024-03-06T07:25:55.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.555+0000 7fab63a22640 1 -- start start 2024-03-06T07:25:55.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab63a22640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab5c14f790 0x7fab5c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab63a22640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5c0c28d0 con 0x7fab5c0c1aa0 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab63a22640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5c0c2bb0 con 0x7fab5c155a40 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab63a22640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5c0c2680 con 0x7fab5c14f790 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab60fbd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fab5c0c1aa0 0x7fab5c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50996/0 (socket says 172.21.15.94:50996) 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab60fbd640 1 -- 172.21.15.94:0/3023575864 learned_addr learned my addr 172.21.15.94:0/3023575864 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab617be640 1 --2- 172.21.15.94:0/3023575864 >> v2:172.21.15.145:3300/0 conn(0x7fab5c14f790 0x7fab5c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286632190 0 0) 0x7fab5c0c28d0 con 0x7fab5c0c1aa0 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab48003520 con 0x7fab5c0c1aa0 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 883122734 0 0) 0x7fab48003520 con 0x7fab5c0c1aa0 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 >> v2:172.21.15.145:3300/0 conn(0x7fab5c14f790 msgr2=0x7fab5c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 --2- 172.21.15.94:0/3023575864 >> v2:172.21.15.145:3300/0 conn(0x7fab5c14f790 0x7fab5c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 >> v1:172.21.15.145:6789/0 conn(0x7fab5c155a40 legacy=0x7fab5c155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5c0c2e90 con 0x7fab5c0c1aa0 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.556+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fab4c001670 con 0x7fab5c0c1aa0 2024-03-06T07:25:55.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 >> v1:172.21.15.94:6789/0 conn(0x7fab5c0c1aa0 legacy=0x7fab5c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab53fff640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480039b0 0x7fab48003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab53fff640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab48004800 0x7fab48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab53fff640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab480051c0 0x7fab4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c0c2680 con 0x7fab480051c0 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4800c090 con 0x7fab48004800 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4800c290 con 0x7fab480039b0 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab61fbf640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab480051c0 0x7fab4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab617be640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480039b0 0x7fab48003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab60fbd640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab48004800 0x7fab48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab61fbf640 1 -- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480039b0 msgr2=0x7fab48003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab61fbf640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480039b0 0x7fab48003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab61fbf640 1 -- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab48004800 msgr2=0x7fab48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab61fbf640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab48004800 0x7fab48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.557+0000 7fab61fbf640 1 -- 172.21.15.94:0/3023575864 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab48003520 con 0x7fab480051c0 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab61fbf640 1 -- 172.21.15.94:0/3023575864 reap_dead start 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab61fbf640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab480051c0 0x7fab4800b9c0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fab54001e50 tx=0x7fab48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86de0e1a9d6affb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54001a70 con 0x7fab480051c0 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab54003b90 con 0x7fab480051c0 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab53fff640 1 -- 172.21.15.94:0/3023575864 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab540050d0 con 0x7fab480051c0 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab63a22640 1 -- 172.21.15.94:0/3023575864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab480051c0 msgr2=0x7fab4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab63a22640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab480051c0 0x7fab4800b9c0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fab54001e50 tx=0x7fab48005830 comp rx=0 tx=0).stop 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab63a22640 1 -- 172.21.15.94:0/3023575864 shutdown_connections 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab63a22640 1 --2- 172.21.15.94:0/3023575864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab480051c0 0x7fab4800b9c0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.558+0000 7fab63a22640 1 -- 172.21.15.94:0/3023575864 >> 172.21.15.94:0/3023575864 conn(0x7fab5c0c6710 msgr2=0x7fab5c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.559+0000 7fab63a22640 1 -- 172.21.15.94:0/3023575864 shutdown_connections 2024-03-06T07:25:55.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.559+0000 7fab63a22640 1 -- 172.21.15.94:0/3023575864 wait complete. 2024-03-06T07:25:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 Processor -- start 2024-03-06T07:25:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 -- start start 2024-03-06T07:25:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480051c0 0x7fab5c0a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab5c1e72d0 0x7fab5c14ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab5c14fdc0 0x7fab5c14b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c06cc00 con 0x7fab5c14fdc0 2024-03-06T07:25:55.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c0c0d60 con 0x7fab5c1e72d0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab63a22640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c06cd80 con 0x7fab480051c0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab61fbf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab5c14fdc0 0x7fab5c14b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab61fbf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab5c14fdc0 0x7fab5c14b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44118/0 (socket says 172.21.15.94:44118) 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab61fbf640 1 -- 172.21.15.94:0/1948314805 learned_addr learned my addr 172.21.15.94:0/1948314805 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.560+0000 7fab617be640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480051c0 0x7fab5c0a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab60fbd640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab5c1e72d0 0x7fab5c14ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab61fbf640 1 -- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480051c0 msgr2=0x7fab5c0a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab61fbf640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480051c0 0x7fab5c0a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab61fbf640 1 -- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab5c1e72d0 msgr2=0x7fab5c14ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab61fbf640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab5c1e72d0 0x7fab5c14ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab61fbf640 1 -- 172.21.15.94:0/1948314805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5c1e77b0 con 0x7fab5c14fdc0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab61fbf640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab5c14fdc0 0x7fab5c14b290 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fab54001f70 tx=0x7fab540014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab51ffb640 1 -- 172.21.15.94:0/1948314805 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54007db0 con 0x7fab5c14fdc0 2024-03-06T07:25:55.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab51ffb640 1 -- 172.21.15.94:0/1948314805 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab54003d40 con 0x7fab5c14fdc0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5c151840 con 0x7fab5c14fdc0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab51ffb640 1 -- 172.21.15.94:0/1948314805 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54006d90 con 0x7fab5c14fdc0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.561+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5c14a830 con 0x7fab5c14fdc0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.562+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5c04f330 con 0x7fab5c14fdc0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.563+0000 7fab51ffb640 1 -- 172.21.15.94:0/1948314805 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab54007030 con 0x7fab5c14fdc0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.563+0000 7fab51ffb640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab4006b4d0 0x7fab4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.563+0000 7fab51ffb640 1 -- 172.21.15.94:0/1948314805 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fab540abaa0 con 0x7fab5c14fdc0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.564+0000 7fab617be640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab4006b4d0 0x7fab4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.564+0000 7fab617be640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab4006b4d0 0x7fab4006d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fab5c1553f0 tx=0x7fab58009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.568+0000 7fab51ffb640 1 -- 172.21.15.94:0/1948314805 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab54049060 con 0x7fab5c14fdc0 2024-03-06T07:25:55.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.775+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab5c14c360 con 0x7fab5c14fdc0 2024-03-06T07:25:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.776+0000 7fab51ffb640 1 -- 172.21.15.94:0/1948314805 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fab54033020 con 0x7fab5c14fdc0 2024-03-06T07:25:55.778 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:55.778 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:25:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab4006b4d0 msgr2=0x7fab4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab4006b4d0 0x7fab4006d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fab5c1553f0 tx=0x7fab58009190 comp rx=0 tx=0).stop 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab5c14fdc0 msgr2=0x7fab5c14b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab5c14fdc0 0x7fab5c14b290 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fab54001f70 tx=0x7fab540014a0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 shutdown_connections 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fab4006b4d0 0x7fab4006d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fab5c14fdc0 0x7fab5c14b290 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab5c1e72d0 0x7fab5c14ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 --2- 172.21.15.94:0/1948314805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab480051c0 0x7fab5c0a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 >> 172.21.15.94:0/1948314805 conn(0x7fab5c0c6710 msgr2=0x7fab5c14da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 shutdown_connections 2024-03-06T07:25:55.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:55.778+0000 7fab63a22640 1 -- 172.21.15.94:0/1948314805 wait complete. 2024-03-06T07:25:56.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:56.193 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:25:57.194 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.305+0000 7f916a8d4640 1 Processor -- start 2024-03-06T07:25:57.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.305+0000 7f916a8d4640 1 -- start start 2024-03-06T07:25:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f916a8d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f916a8d4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f91640c6000 0x7f9164156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f916a8d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9164153a60 con 0x7f9164150580 2024-03-06T07:25:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f916a8d4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9164153d40 con 0x7f91640c6000 2024-03-06T07:25:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f916a8d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916406c2c0 con 0x7f91640c1320 2024-03-06T07:25:57.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f91637fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f9168e71640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f91640c6000 0x7f9164156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f91637fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:57836/0 (socket says 172.21.15.94:57836) 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f9163fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f9164150580 0x7f916406dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:47196/0 (socket says 172.21.15.94:47196) 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f91637fe640 1 -- 172.21.15.94:0/440991451 learned_addr learned my addr 172.21.15.94:0/440991451 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f9162ffd640 1 -- 172.21.15.94:0/440991451 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910136020 0 0) 0x7f9164153a60 con 0x7f9164150580 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f9162ffd640 1 -- 172.21.15.94:0/440991451 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f914c003520 con 0x7f9164150580 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f91637fe640 1 -- 172.21.15.94:0/440991451 >> v2:172.21.15.145:3300/0 conn(0x7f91640c6000 msgr2=0x7f9164156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f91637fe640 1 --2- 172.21.15.94:0/440991451 >> v2:172.21.15.145:3300/0 conn(0x7f91640c6000 0x7f9164156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f91637fe640 1 -- 172.21.15.94:0/440991451 >> v1:172.21.15.145:6789/0 conn(0x7f9164150580 legacy=0x7f916406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f91637fe640 1 -- 172.21.15.94:0/440991451 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916406c5a0 con 0x7f91640c1320 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.306+0000 7f9168e71640 1 --2- 172.21.15.94:0/440991451 >> v2:172.21.15.145:3300/0 conn(0x7f91640c6000 0x7f9164156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.307+0000 7f91637fe640 1 --2- 172.21.15.94:0/440991451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641566b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9158009870 tx=0x7f915802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea40be6ed82d9b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.307+0000 7f9162ffd640 1 -- 172.21.15.94:0/440991451 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158035020 con 0x7f91640c1320 2024-03-06T07:25:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f9162ffd640 1 -- 172.21.15.94:0/440991451 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158031640 con 0x7f91640c1320 2024-03-06T07:25:57.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f9162ffd640 1 -- 172.21.15.94:0/440991451 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915802d040 con 0x7f91640c1320 2024-03-06T07:25:57.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 -- 172.21.15.94:0/440991451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 msgr2=0x7f91641566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 --2- 172.21.15.94:0/440991451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641566b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9158009870 tx=0x7f915802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 -- 172.21.15.94:0/440991451 shutdown_connections 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 --2- 172.21.15.94:0/440991451 >> v2:172.21.15.145:3300/0 conn(0x7f91640c6000 0x7f9164156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 --2- 172.21.15.94:0/440991451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641566b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 -- 172.21.15.94:0/440991451 >> 172.21.15.94:0/440991451 conn(0x7f91640c6710 msgr2=0x7f91640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 -- 172.21.15.94:0/440991451 shutdown_connections 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.308+0000 7f916a8d4640 1 -- 172.21.15.94:0/440991451 wait complete. 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.309+0000 7f916a8d4640 1 Processor -- start 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.309+0000 7f916a8d4640 1 -- start start 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f916a8d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f916a8d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91640c6000 0x7f91640a7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f916a8d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9164150580 0x7f91640bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f916a8d4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916406e5a0 con 0x7f9164150580 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f916a8d4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916406e420 con 0x7f91640c6000 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f916a8d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91640c0c40 con 0x7f91640c1320 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9164150580 0x7f91640bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9164150580 0x7f91640bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44124/0 (socket says 172.21.15.94:44124) 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 -- 172.21.15.94:0/3570202070 learned_addr learned my addr 172.21.15.94:0/3570202070 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9163fff640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f91637fe640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91640c6000 0x7f91640a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 -- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 msgr2=0x7f91641496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 -- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91640c6000 msgr2=0x7f91640a7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91640c6000 0x7f91640a7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.310+0000 7f9168e71640 1 -- 172.21.15.94:0/3570202070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640a7df0 con 0x7f9164150580 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.311+0000 7f9168e71640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9164150580 0x7f91640bfdb0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f915c009f00 tx=0x7f915c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.311+0000 7f9160ff9640 1 -- 172.21.15.94:0/3570202070 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c018030 con 0x7f9164150580 2024-03-06T07:25:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.311+0000 7f9160ff9640 1 -- 172.21.15.94:0/3570202070 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c00ce50 con 0x7f9164150580 2024-03-06T07:25:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.311+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9164151f00 con 0x7f9164150580 2024-03-06T07:25:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.311+0000 7f9160ff9640 1 -- 172.21.15.94:0/3570202070 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c003bb0 con 0x7f9164150580 2024-03-06T07:25:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.311+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91641524c0 con 0x7f9164150580 2024-03-06T07:25:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.312+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916404f330 con 0x7f9164150580 2024-03-06T07:25:57.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.312+0000 7f9160ff9640 1 -- 172.21.15.94:0/3570202070 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f915c01b070 con 0x7f9164150580 2024-03-06T07:25:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.313+0000 7f9160ff9640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f915006b4d0 0x7f915006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.313+0000 7f9160ff9640 1 -- 172.21.15.94:0/3570202070 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f915c08ace0 con 0x7f9164150580 2024-03-06T07:25:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.313+0000 7f9163fff640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f915006b4d0 0x7f915006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.313+0000 7f9163fff640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f915006b4d0 0x7f915006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9154000f60 tx=0x7f9154007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.317+0000 7f9160ff9640 1 -- 172.21.15.94:0/3570202070 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f915c010040 con 0x7f9164150580 2024-03-06T07:25:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.525+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91640c5de0 con 0x7f9164150580 2024-03-06T07:25:57.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.525+0000 7f9160ff9640 1 -- 172.21.15.94:0/3570202070 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f915c059380 con 0x7f9164150580 2024-03-06T07:25:57.527 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:57.527 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:25:57.528 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:25:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f915006b4d0 msgr2=0x7f915006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f915006b4d0 0x7f915006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9154000f60 tx=0x7f9154007cf0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9164150580 msgr2=0x7f91640bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9164150580 0x7f91640bfdb0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f915c009f00 tx=0x7f915c00c490 comp rx=0 tx=0).stop 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 shutdown_connections 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f915006b4d0 0x7f915006d990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9164150580 0x7f91640bfdb0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91640c6000 0x7f91640a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 --2- 172.21.15.94:0/3570202070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91640c1320 0x7f91641496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 >> 172.21.15.94:0/3570202070 conn(0x7f91640c6710 msgr2=0x7f91640a8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 shutdown_connections 2024-03-06T07:25:57.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:57.527+0000 7f916a8d4640 1 -- 172.21.15.94:0/3570202070 wait complete. 2024-03-06T07:25:57.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:57.943 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:25:58.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:59.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.053+0000 7f4072756640 1 Processor -- start 2024-03-06T07:25:59.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.053+0000 7f4072756640 1 -- start start 2024-03-06T07:25:59.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f4072756640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c14f790 0x7f406c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f4072756640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f406c0c1aa0 0x7f406c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f4072756640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c0c28d0 con 0x7f406c0c5c60 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f4072756640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c0c2680 con 0x7f406c0c1aa0 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f4072756640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0c2bb0 con 0x7f406c14f790 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f406bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c14f790 0x7f406c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f406bfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c14f790 0x7f406c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:55820/0 (socket says 172.21.15.94:55820) 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f406bfff640 1 -- 172.21.15.94:0/2498253971 learned_addr learned my addr 172.21.15.94:0/2498253971 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.054+0000 7f406b7fe640 1 --2- 172.21.15.94:0/2498253971 >> v2:172.21.15.145:3300/0 conn(0x7f406c0c1aa0 0x7f406c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.055+0000 7f406b7fe640 1 -- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c14f790 msgr2=0x7f406c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.055+0000 7f406b7fe640 1 --2- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c14f790 0x7f406c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.055+0000 7f406b7fe640 1 -- 172.21.15.94:0/2498253971 >> v1:172.21.15.145:6789/0 conn(0x7f406c0c5c60 legacy=0x7f406c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.055+0000 7f406affd640 1 -- 172.21.15.94:0/2498253971 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925520357 0 0) 0x7f406c0c28d0 con 0x7f406c0c5c60 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.055+0000 7f406b7fe640 1 -- 172.21.15.94:0/2498253971 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0c2e90 con 0x7f406c0c1aa0 2024-03-06T07:25:59.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.055+0000 7f406bfff640 1 --2- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c14f790 0x7f406c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.055+0000 7f406b7fe640 1 --2- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c0c5720 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f40600097a0 tx=0x7f406002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea58f96b455d9c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f406affd640 1 -- 172.21.15.94:0/2498253971 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060035020 con 0x7f406c0c1aa0 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f406affd640 1 -- 172.21.15.94:0/2498253971 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060031830 con 0x7f406c0c1aa0 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f406affd640 1 -- 172.21.15.94:0/2498253971 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060031c00 con 0x7f406c0c1aa0 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 -- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 msgr2=0x7f406c0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 --2- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c0c5720 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f40600097a0 tx=0x7f406002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 -- 172.21.15.94:0/2498253971 shutdown_connections 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 --2- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c0c5720 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 --2- 172.21.15.94:0/2498253971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c14f790 0x7f406c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 -- 172.21.15.94:0/2498253971 >> 172.21.15.94:0/2498253971 conn(0x7f406c0c6710 msgr2=0x7f406c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 -- 172.21.15.94:0/2498253971 shutdown_connections 2024-03-06T07:25:59.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.056+0000 7f4072756640 1 -- 172.21.15.94:0/2498253971 wait complete. 2024-03-06T07:25:59.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.057+0000 7f4072756640 1 Processor -- start 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4072756640 1 -- start start 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4072756640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4072756640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c0c5c60 0x7f406c1e6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4072756640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406c14f790 0x7f406c1e7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4072756640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c06cc00 con 0x7f406c14f790 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4072756640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0c0d60 con 0x7f406c0c1aa0 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4072756640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c06cd80 con 0x7f406c0c5c60 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4070cf3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406c14f790 0x7f406c1e7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4070cf3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406c14f790 0x7f406c1e7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36676/0 (socket says 172.21.15.94:36676) 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f4070cf3640 1 -- 172.21.15.94:0/2467124280 learned_addr learned my addr 172.21.15.94:0/2467124280 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:25:59.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f406bfff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.058+0000 7f406b7fe640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c0c5c60 0x7f406c1e6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4070cf3640 1 -- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c0c5c60 msgr2=0x7f406c1e6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4070cf3640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c0c5c60 0x7f406c1e6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4070cf3640 1 -- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 msgr2=0x7f406c151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4070cf3640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4070cf3640 1 -- 172.21.15.94:0/2467124280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c072f30 con 0x7f406c14f790 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f406bfff640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4070cf3640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406c14f790 0x7f406c1e7330 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f406400c760 tx=0x7f406400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4068ff9640 1 -- 172.21.15.94:0/2467124280 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064003da0 con 0x7f406c14f790 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4068ff9640 1 -- 172.21.15.94:0/2467124280 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4064005070 con 0x7f406c14f790 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c0bee60 con 0x7f406c14f790 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4068ff9640 1 -- 172.21.15.94:0/2467124280 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064005460 con 0x7f406c14f790 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.059+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c0731d0 con 0x7f406c14f790 2024-03-06T07:25:59.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.060+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c04f330 con 0x7f406c14f790 2024-03-06T07:25:59.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.061+0000 7f4068ff9640 1 -- 172.21.15.94:0/2467124280 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4064005600 con 0x7f406c14f790 2024-03-06T07:25:59.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.061+0000 7f4068ff9640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405806b4d0 0x7f405806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.061+0000 7f4068ff9640 1 -- 172.21.15.94:0/2467124280 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f406408a130 con 0x7f406c14f790 2024-03-06T07:25:59.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.061+0000 7f406bfff640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405806b4d0 0x7f405806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.062+0000 7f406bfff640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405806b4d0 0x7f405806d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b8d0 tx=0x7f405c00b450 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.065+0000 7f4068ff9640 1 -- 172.21.15.94:0/2467124280 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40640587f0 con 0x7f406c14f790 2024-03-06T07:25:59.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.290+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f406c0c1f40 con 0x7f406c14f790 2024-03-06T07:25:59.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.290+0000 7f4068ff9640 1 -- 172.21.15.94:0/2467124280 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f406405c890 con 0x7f406c14f790 2024-03-06T07:25:59.292 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:25:59.293 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:25:59.293 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.292+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405806b4d0 msgr2=0x7f405806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.292+0000 7f4072756640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405806b4d0 0x7f405806d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b8d0 tx=0x7f405c00b450 comp rx=0 tx=0).stop 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.292+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406c14f790 msgr2=0x7f406c1e7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.292+0000 7f4072756640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406c14f790 0x7f406c1e7330 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f406400c760 tx=0x7f406400cc20 comp rx=0 tx=0).stop 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 shutdown_connections 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f405806b4d0 0x7f405806d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f406c14f790 0x7f406c1e7330 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f406c0c5c60 0x7f406c1e6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 --2- 172.21.15.94:0/2467124280 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f406c0c1aa0 0x7f406c151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 >> 172.21.15.94:0/2467124280 conn(0x7f406c0c6710 msgr2=0x7f406c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 shutdown_connections 2024-03-06T07:25:59.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:25:59.293+0000 7f4072756640 1 -- 172.21.15.94:0/2467124280 wait complete. 2024-03-06T07:25:59.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:25:59.708 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:26:00.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:00.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.819+0000 7faff9483640 1 Processor -- start 2024-03-06T07:26:00.821 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.819+0000 7faff9483640 1 -- start start 2024-03-06T07:26:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff9483640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faff4150580 0x7faff406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff9483640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c1320 0x7faff406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff9483640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faff40c6000 0x7faff41566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff9483640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff4153a60 con 0x7faff4150580 2024-03-06T07:26:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff9483640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff4153d40 con 0x7faff40c6000 2024-03-06T07:26:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff9483640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff406c2c0 con 0x7faff40c1320 2024-03-06T07:26:00.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faff4150580 0x7faff406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faff4150580 0x7faff406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36690/0 (socket says 172.21.15.94:36690) 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 -- 172.21.15.94:0/3493444484 learned_addr learned my addr 172.21.15.94:0/3493444484 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff37fe640 1 --2- 172.21.15.94:0/3493444484 >> v2:172.21.15.145:3300/0 conn(0x7faff40c6000 0x7faff41566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff27fc640 1 --2- 172.21.15.94:0/3493444484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c1320 0x7faff406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 -- 172.21.15.94:0/3493444484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c1320 msgr2=0x7faff406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 --2- 172.21.15.94:0/3493444484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c1320 0x7faff406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 -- 172.21.15.94:0/3493444484 >> v2:172.21.15.145:3300/0 conn(0x7faff40c6000 msgr2=0x7faff41566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 --2- 172.21.15.94:0/3493444484 >> v2:172.21.15.145:3300/0 conn(0x7faff40c6000 0x7faff41566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff2ffd640 1 -- 172.21.15.94:0/3493444484 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff406c5a0 con 0x7faff4150580 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.820+0000 7faff37fe640 1 --2- 172.21.15.94:0/3493444484 >> v2:172.21.15.145:3300/0 conn(0x7faff40c6000 0x7faff41566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.821+0000 7faff2ffd640 1 --2- 172.21.15.94:0/3493444484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff4150580 0x7faff406dd10 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7faff4152fe0 tx=0x7fafe402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171f1889988bcd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.821+0000 7faff1ffb640 1 -- 172.21.15.94:0/3493444484 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe402e070 con 0x7faff4150580 2024-03-06T07:26:00.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.821+0000 7faff1ffb640 1 -- 172.21.15.94:0/3493444484 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4037070 con 0x7faff4150580 2024-03-06T07:26:00.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.821+0000 7faff1ffb640 1 -- 172.21.15.94:0/3493444484 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4031a00 con 0x7faff4150580 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 -- 172.21.15.94:0/3493444484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff4150580 msgr2=0x7faff406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 --2- 172.21.15.94:0/3493444484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff4150580 0x7faff406dd10 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7faff4152fe0 tx=0x7fafe402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 -- 172.21.15.94:0/3493444484 shutdown_connections 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 --2- 172.21.15.94:0/3493444484 >> v2:172.21.15.145:3300/0 conn(0x7faff40c6000 0x7faff41566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 --2- 172.21.15.94:0/3493444484 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c1320 0x7faff406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 --2- 172.21.15.94:0/3493444484 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff4150580 0x7faff406dd10 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 -- 172.21.15.94:0/3493444484 >> 172.21.15.94:0/3493444484 conn(0x7faff40c6710 msgr2=0x7faff40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 -- 172.21.15.94:0/3493444484 shutdown_connections 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.822+0000 7faff9483640 1 -- 172.21.15.94:0/3493444484 wait complete. 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.823+0000 7faff9483640 1 Processor -- start 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.823+0000 7faff9483640 1 -- start start 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff9483640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff40c1320 0x7faff4145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff9483640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c6000 0x7faff41522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff9483640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faff4150580 0x7faff41559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff9483640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40c0dc0 con 0x7faff40c1320 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff9483640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40c0c40 con 0x7faff4150580 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff9483640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff4156bf0 con 0x7faff40c6000 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff40c1320 0x7faff4145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff40c1320 0x7faff4145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36694/0 (socket says 172.21.15.94:36694) 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 -- 172.21.15.94:0/3990967425 learned_addr learned my addr 172.21.15.94:0/3990967425 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff37fe640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faff4150580 0x7faff41559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff27fc640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c6000 0x7faff41522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 -- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c6000 msgr2=0x7faff41522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c6000 0x7faff41522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 -- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faff4150580 msgr2=0x7faff41559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faff4150580 0x7faff41559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff2ffd640 1 -- 172.21.15.94:0/3990967425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff41ebbe0 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff37fe640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faff4150580 0x7faff41559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.824+0000 7faff27fc640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c6000 0x7faff41522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.825+0000 7faff2ffd640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff40c1320 0x7faff4145710 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fafe4000c00 tx=0x7fafe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.825+0000 7fafdb7fe640 1 -- 172.21.15.94:0/3990967425 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4032880 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.825+0000 7fafdb7fe640 1 -- 172.21.15.94:0/3990967425 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4032a40 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.825+0000 7fafdb7fe640 1 -- 172.21.15.94:0/3990967425 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4032e30 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.825+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff41ebec0 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.825+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff41ec480 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.826+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff404f330 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.826+0000 7fafdb7fe640 1 -- 172.21.15.94:0/3990967425 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafe40453f0 con 0x7faff40c1320 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.827+0000 7fafdb7fe640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fafe006b4d0 0x7fafe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.827+0000 7fafdb7fe640 1 -- 172.21.15.94:0/3990967425 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fafe40ac1a0 con 0x7faff40c1320 2024-03-06T07:26:00.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.827+0000 7faff27fc640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fafe006b4d0 0x7fafe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.827+0000 7faff27fc640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fafe006b4d0 0x7fafe006d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7faff414e020 tx=0x7fafe8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:00.831+0000 7fafdb7fe640 1 -- 172.21.15.94:0/3990967425 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafe4040070 con 0x7faff40c1320 2024-03-06T07:26:01.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.039+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faff40c5de0 con 0x7faff40c1320 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.039+0000 7fafdb7fe640 1 -- 172.21.15.94:0/3990967425 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7fafe407aeb0 con 0x7faff40c1320 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:01.041 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:26:01.042 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:26:01.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fafe006b4d0 msgr2=0x7fafe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fafe006b4d0 0x7fafe006d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7faff414e020 tx=0x7fafe8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff40c1320 msgr2=0x7faff4145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff40c1320 0x7faff4145710 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fafe4000c00 tx=0x7fafe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 shutdown_connections 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fafe006b4d0 0x7fafe006d990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faff4150580 0x7faff41559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faff40c6000 0x7faff41522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 --2- 172.21.15.94:0/3990967425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff40c1320 0x7faff4145710 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 >> 172.21.15.94:0/3990967425 conn(0x7faff40c6710 msgr2=0x7faff406dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 shutdown_connections 2024-03-06T07:26:01.044 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:01.041+0000 7faff9483640 1 -- 172.21.15.94:0/3990967425 wait complete. 2024-03-06T07:26:01.456 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:01.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:26:02.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.565+0000 7f61e51d7640 1 Processor -- start 2024-03-06T07:26:02.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.565+0000 7f61e51d7640 1 -- start start 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61e51d7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f61e014f360 0x7f61e0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61e51d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e00c73d0 con 0x7f61e00c88c0 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61e51d7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e00be2e0 con 0x7f61e0158b60 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61e51d7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e0159700 con 0x7f61e014f360 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61ded9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f61e014f360 0x7f61e0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61de59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f61e00c88c0 0x7f61e00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42872/0 (socket says 172.21.15.94:42872) 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61ded9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f61e014f360 0x7f61e0155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36698/0 (socket says 172.21.15.94:36698) 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.566+0000 7f61de59c640 1 -- 172.21.15.94:0/4152556611 learned_addr learned my addr 172.21.15.94:0/4152556611 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.567+0000 7f61ded9d640 1 -- 172.21.15.94:0/4152556611 >> v1:172.21.15.145:6789/0 conn(0x7f61e0158b60 legacy=0x7f61e0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.567+0000 7f61ded9d640 1 -- 172.21.15.94:0/4152556611 >> v1:172.21.15.94:6789/0 conn(0x7f61e00c88c0 legacy=0x7f61e00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.567+0000 7f61ded9d640 1 -- 172.21.15.94:0/4152556611 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e00c7670 con 0x7f61e014f360 2024-03-06T07:26:02.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.567+0000 7f61ddd9b640 1 -- 172.21.15.94:0/4152556611 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471485545 0 0) 0x7f61e00c73d0 con 0x7f61e00c88c0 2024-03-06T07:26:02.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.567+0000 7f61ded9d640 1 --2- 172.21.15.94:0/4152556611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e014f360 0x7f61e0155c10 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009d90 tx=0x7f61d002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6a56c27525adf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.567+0000 7f61ddd9b640 1 -- 172.21.15.94:0/4152556611 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0039070 con 0x7f61e014f360 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61ddd9b640 1 -- 172.21.15.94:0/4152556611 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d002cbd0 con 0x7f61e014f360 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61ddd9b640 1 -- 172.21.15.94:0/4152556611 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0034440 con 0x7f61e014f360 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61e51d7640 1 -- 172.21.15.94:0/4152556611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e014f360 msgr2=0x7f61e0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61e51d7640 1 --2- 172.21.15.94:0/4152556611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e014f360 0x7f61e0155c10 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009d90 tx=0x7f61d002be20 comp rx=0 tx=0).stop 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61e51d7640 1 -- 172.21.15.94:0/4152556611 shutdown_connections 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61e51d7640 1 --2- 172.21.15.94:0/4152556611 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e014f360 0x7f61e0155c10 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61e51d7640 1 -- 172.21.15.94:0/4152556611 >> 172.21.15.94:0/4152556611 conn(0x7f61e00ac570 msgr2=0x7f61e00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61e51d7640 1 -- 172.21.15.94:0/4152556611 shutdown_connections 2024-03-06T07:26:02.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.568+0000 7f61e51d7640 1 -- 172.21.15.94:0/4152556611 wait complete. 2024-03-06T07:26:02.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 Processor -- start 2024-03-06T07:26:02.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 -- start start 2024-03-06T07:26:02.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61e00c88c0 0x7f61e014c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61e014f360 0x7f61e00be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e0158b60 0x7f61e0150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00c7c30 con 0x7f61e0158b60 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00c5a50 con 0x7f61e014f360 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61e51d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00c5bd0 con 0x7f61e00c88c0 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61df59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e0158b60 0x7f61e0150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61df59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e0158b60 0x7f61e0150ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36700/0 (socket says 172.21.15.94:36700) 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.570+0000 7f61df59e640 1 -- 172.21.15.94:0/3969841003 learned_addr learned my addr 172.21.15.94:0/3969841003 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61de59c640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61e014f360 0x7f61e00be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61ded9d640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61e00c88c0 0x7f61e014c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61df59e640 1 -- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61e00c88c0 msgr2=0x7f61e014c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61df59e640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61e00c88c0 0x7f61e014c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61df59e640 1 -- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61e014f360 msgr2=0x7f61e00be4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61df59e640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61e014f360 0x7f61e00be4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61df59e640 1 -- 172.21.15.94:0/3969841003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e0151570 con 0x7f61e0158b60 2024-03-06T07:26:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61df59e640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e0158b60 0x7f61e0150ab0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f61d800b440 tx=0x7f61d800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61c77fe640 1 -- 172.21.15.94:0/3969841003 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8015020 con 0x7f61e0158b60 2024-03-06T07:26:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61c77fe640 1 -- 172.21.15.94:0/3969841003 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d8011830 con 0x7f61e0158b60 2024-03-06T07:26:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61c77fe640 1 -- 172.21.15.94:0/3969841003 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8011c20 con 0x7f61e0158b60 2024-03-06T07:26:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e01502f0 con 0x7f61e0158b60 2024-03-06T07:26:02.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.571+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e0150590 con 0x7f61e0158b60 2024-03-06T07:26:02.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.572+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e00bea30 con 0x7f61e0158b60 2024-03-06T07:26:02.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.572+0000 7f61c77fe640 1 -- 172.21.15.94:0/3969841003 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61d8018020 con 0x7f61e0158b60 2024-03-06T07:26:02.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.573+0000 7f61c77fe640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f61cc06b4d0 0x7f61cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.573+0000 7f61c77fe640 1 -- 172.21.15.94:0/3969841003 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f61d8089f20 con 0x7f61e0158b60 2024-03-06T07:26:02.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.573+0000 7f61ded9d640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f61cc06b4d0 0x7f61cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.574+0000 7f61ded9d640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f61cc06b4d0 0x7f61cc06d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f61d0036000 tx=0x7f61d0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.577+0000 7f61c77fe640 1 -- 172.21.15.94:0/3969841003 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61d801f030 con 0x7f61e0158b60 2024-03-06T07:26:02.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.785+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e004f330 con 0x7f61e0158b60 2024-03-06T07:26:02.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.786+0000 7f61c77fe640 1 -- 172.21.15.94:0/3969841003 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4991 (secure 0 0 0) 0x7f61e0151570 con 0x7f61e0158b60 2024-03-06T07:26:02.788 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:02.788 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":173,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:25:48.445772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":37763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_37763":{"gid":37763,"name":"b","rank":0,"incarnation":169,"state":"up:stopping","state_seq":3,"addr":"172.21.15.145:6836/2375611510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2375611510},{"type":"v1","addr":"172.21.15.145:6836","nonce":2375611510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:26:02.788 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 173 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f61cc06b4d0 msgr2=0x7f61cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f61cc06b4d0 0x7f61cc06d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f61d0036000 tx=0x7f61d0030000 comp rx=0 tx=0).stop 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e0158b60 msgr2=0x7f61e0150ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e0158b60 0x7f61e0150ab0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f61d800b440 tx=0x7f61d800b900 comp rx=0 tx=0).stop 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 shutdown_connections 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f61cc06b4d0 0x7f61cc06d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f61e0158b60 0x7f61e0150ab0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61e014f360 0x7f61e00be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 --2- 172.21.15.94:0/3969841003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61e00c88c0 0x7f61e014c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 >> 172.21.15.94:0/3969841003 conn(0x7f61e00ac570 msgr2=0x7f61e014df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 shutdown_connections 2024-03-06T07:26:02.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:02.788+0000 7f61e51d7640 1 -- 172.21.15.94:0/3969841003 wait complete. 2024-03-06T07:26:03.204 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:25:48.445772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 37763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_37763': {'gid': 37763, 'name': 'b', 'rank': 0, 'incarnation': 169, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.145:6836/2375611510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2375611510}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2375611510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:03.204 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_37763:up:stopping 2024-03-06T07:26:04.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:04.265 INFO:tasks.ceph.mds.b.smithi145.stderr: -797> 2024-03-06T07:25:43.627+0000 7f06fe5f2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:04.293 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:26:04.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.319+0000 7f3c9d116640 1 Processor -- start 2024-03-06T07:26:04.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.319+0000 7f3c9d116640 1 -- start start 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c9d116640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3c9814f790 0x7f3c981529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c9d116640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3c980c1aa0 0x7f3c980c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c9d116640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c980c28d0 con 0x7f3c980c5c60 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c9d116640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980c2680 con 0x7f3c9814f790 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c9d116640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980c2bb0 con 0x7f3c980c1aa0 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c96d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3c9814f790 0x7f3c981529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c96d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3c9814f790 0x7f3c981529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36716/0 (socket says 172.21.15.94:36716) 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c96d9d640 1 -- 172.21.15.94:0/111866369 learned_addr learned my addr 172.21.15.94:0/111866369 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c96d9d640 1 -- 172.21.15.94:0/111866369 >> v2:172.21.15.145:3300/0 conn(0x7f3c980c1aa0 msgr2=0x7f3c980c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c9659c640 1 --2- 172.21.15.94:0/111866369 >> v2:172.21.15.145:3300/0 conn(0x7f3c980c1aa0 0x7f3c980c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/111866369 >> v2:172.21.15.145:3300/0 conn(0x7f3c980c1aa0 0x7f3c980c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c96d9d640 1 -- 172.21.15.94:0/111866369 >> v1:172.21.15.145:6789/0 conn(0x7f3c980c5c60 legacy=0x7f3c98155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.320+0000 7f3c96d9d640 1 -- 172.21.15.94:0/111866369 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c980c2e90 con 0x7f3c9814f790 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.321+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/111866369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9814f790 0x7f3c981529d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f3c98070280 tx=0x7f3c8802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5510171fd28080f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.321+0000 7f3c95d9b640 1 -- 172.21.15.94:0/111866369 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8802ca80 con 0x7f3c9814f790 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.321+0000 7f3c95d9b640 1 -- 172.21.15.94:0/111866369 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8802cc20 con 0x7f3c9814f790 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.321+0000 7f3c95d9b640 1 -- 172.21.15.94:0/111866369 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c880343f0 con 0x7f3c9814f790 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 -- 172.21.15.94:0/111866369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9814f790 msgr2=0x7f3c981529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 --2- 172.21.15.94:0/111866369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9814f790 0x7f3c981529d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f3c98070280 tx=0x7f3c8802bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 -- 172.21.15.94:0/111866369 shutdown_connections 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 --2- 172.21.15.94:0/111866369 >> v2:172.21.15.145:3300/0 conn(0x7f3c980c1aa0 0x7f3c980c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 --2- 172.21.15.94:0/111866369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c9814f790 0x7f3c981529d0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 -- 172.21.15.94:0/111866369 >> 172.21.15.94:0/111866369 conn(0x7f3c980c6710 msgr2=0x7f3c980c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 -- 172.21.15.94:0/111866369 shutdown_connections 2024-03-06T07:26:04.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.322+0000 7f3c9d116640 1 -- 172.21.15.94:0/111866369 wait complete. 2024-03-06T07:26:04.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.323+0000 7f3c9d116640 1 Processor -- start 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.323+0000 7f3c9d116640 1 -- start start 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9d116640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c980c1aa0 0x7f3c9806dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9d116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c980c5c60 0x7f3c98147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9d116640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9814f790 0x7f3c98148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9d116640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9806cc00 con 0x7f3c980c5c60 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9d116640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980c0d60 con 0x7f3c9814f790 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9d116640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9806cd80 con 0x7f3c980c1aa0 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c980c5c60 0x7f3c98147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c980c5c60 0x7f3c98147b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36718/0 (socket says 172.21.15.94:36718) 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 -- 172.21.15.94:0/76081385 learned_addr learned my addr 172.21.15.94:0/76081385 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:04.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c980c1aa0 0x7f3c9806dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9759e640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9814f790 0x7f3c98148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 -- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c980c1aa0 msgr2=0x7f3c9806dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c980c1aa0 0x7f3c9806dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 -- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9814f790 msgr2=0x7f3c98148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9814f790 0x7f3c98148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.324+0000 7f3c9659c640 1 -- 172.21.15.94:0/76081385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c981ec320 con 0x7f3c980c5c60 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.325+0000 7f3c9659c640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c980c5c60 0x7f3c98147b20 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c820 tx=0x7f3c8c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.325+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/76081385 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c015070 con 0x7f3c980c5c60 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.325+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/76081385 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c004220 con 0x7f3c980c5c60 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.325+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c981ec600 con 0x7f3c980c5c60 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.325+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/76081385 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00ce10 con 0x7f3c980c5c60 2024-03-06T07:26:04.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.325+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c981ecbc0 con 0x7f3c980c5c60 2024-03-06T07:26:04.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.326+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9804f330 con 0x7f3c980c5c60 2024-03-06T07:26:04.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.326+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/76081385 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c8c007830 con 0x7f3c980c5c60 2024-03-06T07:26:04.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.327+0000 7f3c7f7fe640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.327+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/76081385 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f3c8c08a1c0 con 0x7f3c980c5c60 2024-03-06T07:26:04.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.327+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.327+0000 7f3c96d9d640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3c88035000 tx=0x7f3c88030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.331+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/76081385 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c8c058ed0 con 0x7f3c980c5c60 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.538+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c981485a0 con 0x7f3c980c5c60 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.538+0000 7f3c7f7fe640 1 -- 172.21.15.94:0/76081385 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4181 (secure 0 0 0) 0x7f3c8c0587d0 con 0x7f3c980c5c60 2024-03-06T07:26:04.540 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:04.541 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":174,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:03.269525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:26:04.541 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 174 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 msgr2=0x7f3c8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3c88035000 tx=0x7f3c88030000 comp rx=0 tx=0).stop 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c980c5c60 msgr2=0x7f3c98147b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c980c5c60 0x7f3c98147b20 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c820 tx=0x7f3c8c00cce0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 shutdown_connections 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3c8406b500 0x7f3c8406d9c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c9814f790 0x7f3c98148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3c980c5c60 0x7f3c98147b20 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 --2- 172.21.15.94:0/76081385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c980c1aa0 0x7f3c9806dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 >> 172.21.15.94:0/76081385 conn(0x7f3c980c6710 msgr2=0x7f3c980c54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 shutdown_connections 2024-03-06T07:26:04.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:04.540+0000 7f3c9d116640 1 -- 172.21.15.94:0/76081385 wait complete. 2024-03-06T07:26:04.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 174, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:26:03.269525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:04.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-03-06T07:26:04.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:05.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.055+0000 7fd570f93640 1 Processor -- start 2024-03-06T07:26:05.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.055+0000 7fd570f93640 1 -- start start 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd570f93640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd56c0c1aa0 0x7fd56c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd570f93640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd56c155a40 0x7fd56c155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd570f93640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56c0c28d0 con 0x7fd56c14f790 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd570f93640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56c0c2680 con 0x7fd56c155a40 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd570f93640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56c0c2bb0 con 0x7fd56c0c1aa0 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56ad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd56c155a40 0x7fd56c155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd56c14f790 0x7fd56c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42884/0 (socket says 172.21.15.94:42884) 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56ad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd56c155a40 0x7fd56c155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36724/0 (socket says 172.21.15.94:36724) 2024-03-06T07:26:05.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56a59c640 1 -- 172.21.15.94:0/3461658091 learned_addr learned my addr 172.21.15.94:0/3461658091 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56ad9d640 1 -- 172.21.15.94:0/3461658091 >> v2:172.21.15.145:3300/0 conn(0x7fd56c0c1aa0 msgr2=0x7fd56c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56ad9d640 1 --2- 172.21.15.94:0/3461658091 >> v2:172.21.15.145:3300/0 conn(0x7fd56c0c1aa0 0x7fd56c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56ad9d640 1 -- 172.21.15.94:0/3461658091 >> v1:172.21.15.94:6789/0 conn(0x7fd56c14f790 legacy=0x7fd56c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd569d9b640 1 -- 172.21.15.94:0/3461658091 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792376808 0 0) 0x7fd56c0c28d0 con 0x7fd56c14f790 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.056+0000 7fd56ad9d640 1 -- 172.21.15.94:0/3461658091 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0c2e90 con 0x7fd56c155a40 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.057+0000 7fd56ad9d640 1 --2- 172.21.15.94:0/3461658091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c155e50 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fd56c070280 tx=0x7fd56402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d243a162e325b7c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.057+0000 7fd569d9b640 1 -- 172.21.15.94:0/3461658091 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564039070 con 0x7fd56c155a40 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.057+0000 7fd569d9b640 1 -- 172.21.15.94:0/3461658091 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56402cc80 con 0x7fd56c155a40 2024-03-06T07:26:05.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.057+0000 7fd569d9b640 1 -- 172.21.15.94:0/3461658091 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564034660 con 0x7fd56c155a40 2024-03-06T07:26:05.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 -- 172.21.15.94:0/3461658091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 msgr2=0x7fd56c155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 --2- 172.21.15.94:0/3461658091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c155e50 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fd56c070280 tx=0x7fd56402bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 -- 172.21.15.94:0/3461658091 shutdown_connections 2024-03-06T07:26:05.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 --2- 172.21.15.94:0/3461658091 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c155e50 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 --2- 172.21.15.94:0/3461658091 >> v2:172.21.15.145:3300/0 conn(0x7fd56c0c1aa0 0x7fd56c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 -- 172.21.15.94:0/3461658091 >> 172.21.15.94:0/3461658091 conn(0x7fd56c0c6710 msgr2=0x7fd56c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 -- 172.21.15.94:0/3461658091 shutdown_connections 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.058+0000 7fd570f93640 1 -- 172.21.15.94:0/3461658091 wait complete. 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.059+0000 7fd570f93640 1 Processor -- start 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.059+0000 7fd570f93640 1 -- start start 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd570f93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd56c0c1aa0 0x7fd56c0705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd570f93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd56c14f790 0x7fd56c14cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd570f93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c071360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd570f93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c0c0d60 con 0x7fd56c155a40 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd570f93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c156390 con 0x7fd56c0c1aa0 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd570f93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c0c5e30 con 0x7fd56c14f790 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c071360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36732/0 (socket says 172.21.15.94:36732) 2024-03-06T07:26:05.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 -- 172.21.15.94:0/564308149 learned_addr learned my addr 172.21.15.94:0/564308149 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56a59c640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd56c0c1aa0 0x7fd56c0705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd563fff640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd56c14f790 0x7fd56c14cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 -- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd56c14f790 msgr2=0x7fd56c14cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd56c14f790 0x7fd56c14cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 -- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd56c0c1aa0 msgr2=0x7fd56c0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd56c0c1aa0 0x7fd56c0705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 -- 172.21.15.94:0/564308149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c1ecc50 con 0x7fd56c155a40 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56a59c640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd56c0c1aa0 0x7fd56c0705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd563fff640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd56c14f790 0x7fd56c14cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd56ad9d640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c071360 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd564009e80 tx=0x7fd564004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd562ffd640 1 -- 172.21.15.94:0/564308149 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564044020 con 0x7fd56c155a40 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd562ffd640 1 -- 172.21.15.94:0/564308149 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56403f470 con 0x7fd56c155a40 2024-03-06T07:26:05.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56c1e8c20 con 0x7fd56c155a40 2024-03-06T07:26:05.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.060+0000 7fd562ffd640 1 -- 172.21.15.94:0/564308149 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564039040 con 0x7fd56c155a40 2024-03-06T07:26:05.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.061+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56c1e91e0 con 0x7fd56c155a40 2024-03-06T07:26:05.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.061+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56c0c59c0 con 0x7fd56c155a40 2024-03-06T07:26:05.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.062+0000 7fd562ffd640 1 -- 172.21.15.94:0/564308149 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd564047020 con 0x7fd56c155a40 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.062+0000 7fd562ffd640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd54c06b4d0 0x7fd54c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.063+0000 7fd562ffd640 1 -- 172.21.15.94:0/564308149 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fd5640ad0e0 con 0x7fd56c155a40 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.063+0000 7fd56a59c640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd54c06b4d0 0x7fd54c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.063+0000 7fd56a59c640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd54c06b4d0 0x7fd54c06d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd55c000f60 tx=0x7fd55c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.067+0000 7fd562ffd640 1 -- 172.21.15.94:0/564308149 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd56c0c59c0 con 0x7fd56c155a40 2024-03-06T07:26:05.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.277+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd56c04f330 con 0x7fd56c155a40 2024-03-06T07:26:05.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.278+0000 7fd562ffd640 1 -- 172.21.15.94:0/564308149 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+4976 (secure 0 0 0) 0x7fd56404e030 con 0x7fd56c155a40 2024-03-06T07:26:05.280 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:05.280 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":175,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9891,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:03.269525+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:26:05.280 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 175 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd54c06b4d0 msgr2=0x7fd54c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd54c06b4d0 0x7fd54c06d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd55c000f60 tx=0x7fd55c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 msgr2=0x7fd56c071360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c071360 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd564009e80 tx=0x7fd564004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 shutdown_connections 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd54c06b4d0 0x7fd54c06d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd56c155a40 0x7fd56c071360 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd56c14f790 0x7fd56c14cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 --2- 172.21.15.94:0/564308149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd56c0c1aa0 0x7fd56c0705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 >> 172.21.15.94:0/564308149 conn(0x7fd56c0c6710 msgr2=0x7fd56c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 shutdown_connections 2024-03-06T07:26:05.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.280+0000 7fd570f93640 1 -- 172.21.15.94:0/564308149 wait complete. 2024-03-06T07:26:05.292 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:26:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.393+0000 7fec739e9640 1 Processor -- start 2024-03-06T07:26:05.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.393+0000 7fec739e9640 1 -- start start 2024-03-06T07:26:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.394+0000 7fec739e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 0x7fec6c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.394+0000 7fec739e9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec6c0c5380 0x7fec6c0c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.394+0000 7fec739e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6c0b0450 con 0x7fec6c06ca70 2024-03-06T07:26:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.394+0000 7fec739e9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec6c06c020 con 0x7fec6c0c5380 2024-03-06T07:26:05.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.394+0000 7fec739e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c0c77f0 con 0x7fec6c0b0070 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.394+0000 7fec70f84640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fec6c06ca70 0x7fec6c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42894/0 (socket says 172.21.15.94:42894) 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.394+0000 7fec70f84640 1 -- 172.21.15.94:0/3816776274 learned_addr learned my addr 172.21.15.94:0/3816776274 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71785640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 0x7fec6c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71f86640 1 --2- 172.21.15.94:0/3816776274 >> v2:172.21.15.145:3300/0 conn(0x7fec6c0c5380 0x7fec6c0c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247974316 0 0) 0x7fec5c001110 con 0x7fec6c06ca70 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec58003520 con 0x7fec6c06ca70 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1006065552 0 0) 0x7fec58003520 con 0x7fec6c06ca70 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 msgr2=0x7fec6c155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 0x7fec6c155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 >> v2:172.21.15.145:3300/0 conn(0x7fec6c0c5380 msgr2=0x7fec6c0c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 --2- 172.21.15.94:0/3816776274 >> v2:172.21.15.145:3300/0 conn(0x7fec6c0c5380 0x7fec6c0c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c0c7ad0 con 0x7fec6c06ca70 2024-03-06T07:26:05.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fec5c001840 con 0x7fec6c06ca70 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 >> v1:172.21.15.94:6789/0 conn(0x7fec6c06ca70 legacy=0x7fec6c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 0x7fec58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec58004d40 0x7fec580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c06c020 con 0x7fec580039b0 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c0c77f0 con 0x7fec58004d40 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5800b9c0 con 0x7fec6c0b0070 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71785640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec70f84640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 0x7fec58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71f86640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec58004d40 0x7fec580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71785640 1 -- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 msgr2=0x7fec58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71785640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 0x7fec58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71785640 1 -- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec58004d40 msgr2=0x7fec580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71785640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec58004d40 0x7fec580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71785640 1 -- 172.21.15.94:0/3816776274 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec58003520 con 0x7fec580039b0 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec70f84640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0b0070 0x7fec58004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.395+0000 7fec71f86640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec58004d40 0x7fec580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.396+0000 7fec71785640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec58003dd0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fec68009fc0 tx=0x7fec58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565230ae02ac7b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.396+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68039070 con 0x7fec580039b0 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.396+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6802cc20 con 0x7fec580039b0 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.396+0000 7fec63fff640 1 -- 172.21.15.94:0/3816776274 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68009c10 con 0x7fec580039b0 2024-03-06T07:26:05.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.397+0000 7fec739e9640 1 -- 172.21.15.94:0/3816776274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 msgr2=0x7fec58003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.397+0000 7fec739e9640 1 --2- 172.21.15.94:0/3816776274 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec58003dd0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fec68009fc0 tx=0x7fec58005830 comp rx=0 tx=0).stop 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.397+0000 7fec71f86640 1 -- 172.21.15.94:0/3816776274 reap_dead start 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.397+0000 7fec739e9640 1 -- 172.21.15.94:0/3816776274 shutdown_connections 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.397+0000 7fec739e9640 1 -- 172.21.15.94:0/3816776274 >> 172.21.15.94:0/3816776274 conn(0x7fec6c0ac590 msgr2=0x7fec6c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.397+0000 7fec739e9640 1 -- 172.21.15.94:0/3816776274 shutdown_connections 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.397+0000 7fec739e9640 1 -- 172.21.15.94:0/3816776274 wait complete. 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.398+0000 7fec739e9640 1 Processor -- start 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.398+0000 7fec739e9640 1 -- start start 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.398+0000 7fec739e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec6c1ead30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.398+0000 7fec739e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6c14af80 0x7fec6c0c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec739e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0c3f00 0x7fec6c1e0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec739e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c06cef0 con 0x7fec580039b0 2024-03-06T07:26:05.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec739e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c06c550 con 0x7fec6c14af80 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec739e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c06da80 con 0x7fec6c0c3f00 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec6c1ead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec6c1ead30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36746/0 (socket says 172.21.15.94:36746) 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 -- 172.21.15.94:0/3760067306 learned_addr learned my addr 172.21.15.94:0/3760067306 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 -- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0c3f00 msgr2=0x7fec6c1e0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0c3f00 0x7fec6c1e0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 -- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6c14af80 msgr2=0x7fec6c0c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6c14af80 0x7fec6c0c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 -- 172.21.15.94:0/3760067306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c14aa50 con 0x7fec580039b0 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec71785640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec6c1ead30 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fec6800e200 tx=0x7fec68009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec61ffb640 1 -- 172.21.15.94:0/3760067306 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68039050 con 0x7fec580039b0 2024-03-06T07:26:05.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec61ffb640 1 -- 172.21.15.94:0/3760067306 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6803a040 con 0x7fec580039b0 2024-03-06T07:26:05.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6c14ad10 con 0x7fec580039b0 2024-03-06T07:26:05.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec61ffb640 1 -- 172.21.15.94:0/3760067306 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68003bb0 con 0x7fec580039b0 2024-03-06T07:26:05.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.399+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6c155280 con 0x7fec580039b0 2024-03-06T07:26:05.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.400+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6c04f330 con 0x7fec580039b0 2024-03-06T07:26:05.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.401+0000 7fec61ffb640 1 -- 172.21.15.94:0/3760067306 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec68004030 con 0x7fec580039b0 2024-03-06T07:26:05.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.401+0000 7fec61ffb640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec5006b4d0 0x7fec5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.401+0000 7fec61ffb640 1 -- 172.21.15.94:0/3760067306 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fec680acc10 con 0x7fec580039b0 2024-03-06T07:26:05.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.401+0000 7fec70f84640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec5006b4d0 0x7fec5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.402+0000 7fec70f84640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec5006b4d0 0x7fec5006d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0012f0 tx=0x7fec5c000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.409 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.405+0000 7fec61ffb640 1 -- 172.21.15.94:0/3760067306 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec6807b890 con 0x7fec580039b0 2024-03-06T07:26:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:05.611+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fec6c0af690 con 0x7fec580039b0 2024-03-06T07:26:06.265 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:26:06.264+0000 7f90f7f21640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:06.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.265+0000 7fec61ffb640 1 -- 172.21.15.94:0/3760067306 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v176) v1 ==== 105+0+0 (secure 0 0 0) 0x7fec6807b190 con 0x7fec580039b0 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.268+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec5006b4d0 msgr2=0x7fec5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.268+0000 7fec739e9640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec5006b4d0 0x7fec5006d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0012f0 tx=0x7fec5c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.268+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 msgr2=0x7fec6c1ead30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.268+0000 7fec739e9640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec6c1ead30 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fec6800e200 tx=0x7fec68009750 comp rx=0 tx=0).stop 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 shutdown_connections 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec5006b4d0 0x7fec5006d990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6c0c3f00 0x7fec6c1e0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6c14af80 0x7fec6c0c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 --2- 172.21.15.94:0/3760067306 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec580039b0 0x7fec6c1ead30 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 >> 172.21.15.94:0/3760067306 conn(0x7fec6c0ac590 msgr2=0x7fec6c151330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 shutdown_connections 2024-03-06T07:26:06.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.269+0000 7fec739e9640 1 -- 172.21.15.94:0/3760067306 wait complete. 2024-03-06T07:26:06.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.387+0000 7f0d11cfb640 1 Processor -- start 2024-03-06T07:26:06.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.387+0000 7f0d11cfb640 1 -- start start 2024-03-06T07:26:06.390 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.387+0000 7f0d11cfb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14f790 0x7f0d0c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d11cfb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d0c155a40 0x7f0d0c155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d11cfb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c156410 con 0x7f0d0c0c1aa0 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d11cfb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c2680 con 0x7f0d0c155a40 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d11cfb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c28d0 con 0x7f0d0c14f790 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d0c155a40 0x7f0d0c155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0d0c0c1aa0 0x7f0d0c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42906/0 (socket says 172.21.15.94:42906) 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0affd640 1 -- 172.21.15.94:0/4047089828 learned_addr learned my addr 172.21.15.94:0/4047089828 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/4047089828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14f790 0x7f0d0c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0bfff640 1 -- 172.21.15.94:0/4047089828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14f790 msgr2=0x7f0d0c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0bfff640 1 --2- 172.21.15.94:0/4047089828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14f790 0x7f0d0c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0bfff640 1 -- 172.21.15.94:0/4047089828 >> v1:172.21.15.94:6789/0 conn(0x7f0d0c0c1aa0 legacy=0x7f0d0c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0bfff640 1 -- 172.21.15.94:0/4047089828 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0c2bb0 con 0x7f0d0c155a40 2024-03-06T07:26:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/4047089828 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898195719 0 0) 0x7f0d0c156410 con 0x7f0d0c0c1aa0 2024-03-06T07:26:06.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.388+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/4047089828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14f790 0x7f0d0c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:06.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d0bfff640 1 --2- 172.21.15.94:0/4047089828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c155a40 0x7f0d0c155ed0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009d90 tx=0x7f0d0402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d941436532721d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/4047089828 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04039070 con 0x7f0d0c155a40 2024-03-06T07:26:06.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/4047089828 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0402cc50 con 0x7f0d0c155a40 2024-03-06T07:26:06.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d0a7fc640 1 -- 172.21.15.94:0/4047089828 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04034440 con 0x7f0d0c155a40 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 -- 172.21.15.94:0/4047089828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c155a40 msgr2=0x7f0d0c155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/4047089828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c155a40 0x7f0d0c155ed0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009d90 tx=0x7f0d0402bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 -- 172.21.15.94:0/4047089828 shutdown_connections 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/4047089828 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c155a40 0x7f0d0c155ed0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/4047089828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c14f790 0x7f0d0c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 -- 172.21.15.94:0/4047089828 >> 172.21.15.94:0/4047089828 conn(0x7f0d0c0c6710 msgr2=0x7f0d0c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 -- 172.21.15.94:0/4047089828 shutdown_connections 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.389+0000 7f0d11cfb640 1 -- 172.21.15.94:0/4047089828 wait complete. 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.390+0000 7f0d11cfb640 1 Processor -- start 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d11cfb640 1 -- start start 2024-03-06T07:26:06.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d11cfb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0c1aa0 0x7f0d0c14bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d11cfb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14f790 0x7f0d0c1e7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d11cfb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c155a40 0x7f0d0c151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d11cfb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c0d60 con 0x7f0d0c14f790 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d11cfb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c155520 con 0x7f0d0c155a40 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d11cfb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c0c5e30 con 0x7f0d0c0c1aa0 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d0affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14f790 0x7f0d0c1e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d0affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14f790 0x7f0d0c1e7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36766/0 (socket says 172.21.15.94:36766) 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.391+0000 7f0d0affd640 1 -- 172.21.15.94:0/2944023441 learned_addr learned my addr 172.21.15.94:0/2944023441 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0c1aa0 0x7f0d0c14bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0bfff640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c155a40 0x7f0d0c151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0affd640 1 -- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0c1aa0 msgr2=0x7f0d0c14bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0affd640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0c1aa0 0x7f0d0c14bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0affd640 1 -- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c155a40 msgr2=0x7f0d0c151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0affd640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c155a40 0x7f0d0c151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0affd640 1 -- 172.21.15.94:0/2944023441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c14e390 con 0x7f0d0c14f790 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0c1aa0 0x7f0d0c14bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d0affd640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14f790 0x7f0d0c1e7340 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00c4d0 tx=0x7f0cfc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0cf3fff640 1 -- 172.21.15.94:0/2944023441 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc018070 con 0x7f0d0c14f790 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0cf3fff640 1 -- 172.21.15.94:0/2944023441 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc007500 con 0x7f0d0c14f790 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c14e670 con 0x7f0d0c14f790 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0cf3fff640 1 -- 172.21.15.94:0/2944023441 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc004a20 con 0x7f0d0c14f790 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.392+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c072fa0 con 0x7f0d0c14f790 2024-03-06T07:26:06.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.393+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c04f330 con 0x7f0d0c14f790 2024-03-06T07:26:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.394+0000 7f0cf3fff640 1 -- 172.21.15.94:0/2944023441 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cfc01b070 con 0x7f0d0c14f790 2024-03-06T07:26:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.394+0000 7f0cf3fff640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b500 0x7f0cf806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.394+0000 7f0cf3fff640 1 -- 172.21.15.94:0/2944023441 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f0cfc089650 con 0x7f0d0c14f790 2024-03-06T07:26:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.394+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b500 0x7f0cf806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.395+0000 7f0d0b7fe640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b500 0x7f0cf806d9c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0690c0 tx=0x7f0d00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.398+0000 7f0cf3fff640 1 -- 172.21.15.94:0/2944023441 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cfc058360 con 0x7f0d0c14f790 2024-03-06T07:26:06.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.611+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d0c0c1f40 con 0x7f0d0c14f790 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.611+0000 7f0cf3fff640 1 -- 172.21.15.94:0/2944023441 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+4988 (secure 0 0 0) 0x7f0cfc057c60 con 0x7f0d0c14f790 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":177,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175},{"gid":9888,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:06.269543+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0],"up":{"mds_0":9891},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9891":{"gid":9891,"name":"a","rank":0,"incarnation":177,"state":"up:starting","state_seq":1,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-03-06T07:26:06.613 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 177 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b500 msgr2=0x7f0cf806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b500 0x7f0cf806d9c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0690c0 tx=0x7f0d00008040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14f790 msgr2=0x7f0d0c1e7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14f790 0x7f0d0c1e7340 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00c4d0 tx=0x7f0cfc00c990 comp rx=0 tx=0).stop 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 shutdown_connections 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0cf806b500 0x7f0cf806d9c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d0c155a40 0x7f0d0c151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d0c14f790 0x7f0d0c1e7340 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 --2- 172.21.15.94:0/2944023441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d0c0c1aa0 0x7f0d0c14bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 >> 172.21.15.94:0/2944023441 conn(0x7f0d0c0c6710 msgr2=0x7f0d0c0c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 shutdown_connections 2024-03-06T07:26:06.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:06.613+0000 7f0d11cfb640 1 -- 172.21.15.94:0/2944023441 wait complete. 2024-03-06T07:26:07.029 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 177, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:26:06.269543+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 9891}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9891': {'gid': 9891, 'name': 'a', 'rank': 0, 'incarnation': 177, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.94:6834/4225816089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 4225816089}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 4225816089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:07.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9891:up:starting 2024-03-06T07:26:08.029 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.129+0000 7fd613bc7640 1 Processor -- start 2024-03-06T07:26:08.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.129+0000 7fd613bc7640 1 -- start start 2024-03-06T07:26:08.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd613bc7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd60c1592d0 0x7fd60c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd613bc7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd60c0ac260 0x7fd60c155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd613bc7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c152d40 con 0x7fd60c152960 2024-03-06T07:26:08.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd613bc7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c066e30 con 0x7fd60c1592d0 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd613bc7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c155d50 con 0x7fd60c0ac260 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd611162640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd60c1592d0 0x7fd60c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd611963640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd60c152960 0x7fd60c06dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42910/0 (socket says 172.21.15.94:42910) 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd611963640 1 -- 172.21.15.94:0/1330622006 learned_addr learned my addr 172.21.15.94:0/1330622006 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.130+0000 7fd611162640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd60c1592d0 0x7fd60c1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36772/0 (socket says 172.21.15.94:36772) 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd611162640 1 -- 172.21.15.94:0/1330622006 >> v2:172.21.15.145:3300/0 conn(0x7fd60c0ac260 msgr2=0x7fd60c155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd612164640 1 --2- 172.21.15.94:0/1330622006 >> v2:172.21.15.145:3300/0 conn(0x7fd60c0ac260 0x7fd60c155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd610961640 1 -- 172.21.15.94:0/1330622006 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118071012 0 0) 0x7fd60c152d40 con 0x7fd60c152960 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd611162640 1 --2- 172.21.15.94:0/1330622006 >> v2:172.21.15.145:3300/0 conn(0x7fd60c0ac260 0x7fd60c155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd611162640 1 -- 172.21.15.94:0/1330622006 >> v1:172.21.15.94:6789/0 conn(0x7fd60c152960 legacy=0x7fd60c06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd611162640 1 -- 172.21.15.94:0/1330622006 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0c1b50 con 0x7fd60c1592d0 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd612164640 1 --2- 172.21.15.94:0/1330622006 >> v2:172.21.15.145:3300/0 conn(0x7fd60c0ac260 0x7fd60c155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd611162640 1 --2- 172.21.15.94:0/1330622006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c1592d0 0x7fd60c1552d0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fd60c06b380 tx=0x7fd60802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878ca5ee27931761 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd610961640 1 -- 172.21.15.94:0/1330622006 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608035020 con 0x7fd60c1592d0 2024-03-06T07:26:08.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.131+0000 7fd610961640 1 -- 172.21.15.94:0/1330622006 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd608031830 con 0x7fd60c1592d0 2024-03-06T07:26:08.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd610961640 1 -- 172.21.15.94:0/1330622006 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608031c00 con 0x7fd60c1592d0 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 -- 172.21.15.94:0/1330622006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c1592d0 msgr2=0x7fd60c1552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 --2- 172.21.15.94:0/1330622006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c1592d0 0x7fd60c1552d0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fd60c06b380 tx=0x7fd60802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 -- 172.21.15.94:0/1330622006 shutdown_connections 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 --2- 172.21.15.94:0/1330622006 >> v2:172.21.15.145:3300/0 conn(0x7fd60c0ac260 0x7fd60c155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 --2- 172.21.15.94:0/1330622006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c1592d0 0x7fd60c1552d0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 -- 172.21.15.94:0/1330622006 >> 172.21.15.94:0/1330622006 conn(0x7fd60c0c70b0 msgr2=0x7fd60c0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 -- 172.21.15.94:0/1330622006 shutdown_connections 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.132+0000 7fd613bc7640 1 -- 172.21.15.94:0/1330622006 wait complete. 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.133+0000 7fd613bc7640 1 Processor -- start 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.133+0000 7fd613bc7640 1 -- start start 2024-03-06T07:26:08.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd613bc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c0ac260 0x7fd60c1e03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd613bc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd60c152960 0x7fd60c1ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd613bc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd60c1eedf0 0x7fd60c071680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd613bc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c159760 con 0x7fd60c0ac260 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd613bc7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c0672c0 con 0x7fd60c152960 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd613bc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c06e3a0 con 0x7fd60c1eedf0 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c0ac260 0x7fd60c1e03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c0ac260 0x7fd60c1e03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36778/0 (socket says 172.21.15.94:36778) 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 -- 172.21.15.94:0/2259626950 learned_addr learned my addr 172.21.15.94:0/2259626950 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611162640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd60c152960 0x7fd60c1ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd612164640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd60c1eedf0 0x7fd60c071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 -- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd60c1eedf0 msgr2=0x7fd60c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd60c1eedf0 0x7fd60c071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 -- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd60c152960 msgr2=0x7fd60c1ee8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd60c152960 0x7fd60c1ee8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611963640 1 -- 172.21.15.94:0/2259626950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c1df470 con 0x7fd60c0ac260 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd612164640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd60c1eedf0 0x7fd60c071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.134+0000 7fd611162640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd60c152960 0x7fd60c1ee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.135+0000 7fd611963640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c0ac260 0x7fd60c1e03a0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c4d0 tx=0x7fd5fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.135+0000 7fd6027fc640 1 -- 172.21.15.94:0/2259626950 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc018070 con 0x7fd60c0ac260 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.135+0000 7fd6027fc640 1 -- 172.21.15.94:0/2259626950 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc007500 con 0x7fd60c0ac260 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.135+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c070e80 con 0x7fd60c0ac260 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.135+0000 7fd6027fc640 1 -- 172.21.15.94:0/2259626950 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc004a20 con 0x7fd60c0ac260 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.135+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c1e0d80 con 0x7fd60c0ac260 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.136+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c069460 con 0x7fd60c0ac260 2024-03-06T07:26:08.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.136+0000 7fd6027fc640 1 -- 172.21.15.94:0/2259626950 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5fc01b070 con 0x7fd60c0ac260 2024-03-06T07:26:08.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.137+0000 7fd6027fc640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5f006b500 0x7fd5f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.137+0000 7fd6027fc640 1 -- 172.21.15.94:0/2259626950 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fd5fc089630 con 0x7fd60c0ac260 2024-03-06T07:26:08.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.137+0000 7fd611162640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5f006b500 0x7fd5f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.138+0000 7fd611162640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5f006b500 0x7fd5f006d9c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd608036000 tx=0x7fd60803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.141+0000 7fd6027fc640 1 -- 172.21.15.94:0/2259626950 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd60c069460 con 0x7fd60c0ac260 2024-03-06T07:26:08.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.399+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd60c074ac0 con 0x7fd60c0ac260 2024-03-06T07:26:08.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.399+0000 7fd6027fc640 1 -- 172.21.15.94:0/2259626950 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5014 (secure 0 0 0) 0x7fd60c074ac0 con 0x7fd60c0ac260 2024-03-06T07:26:08.401 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:08.401 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":180,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:08.285097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9891,"mds_1":9888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9888":{"gid":9888,"name":"c","rank":1,"incarnation":179,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9891":{"gid":9891,"name":"a","rank":0,"incarnation":177,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9891,"qdb_cluster":[9888,9891]},"id":14}]} 2024-03-06T07:26:08.401 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 180 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5f006b500 msgr2=0x7fd5f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5f006b500 0x7fd5f006d9c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd608036000 tx=0x7fd60803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c0ac260 msgr2=0x7fd60c1e03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c0ac260 0x7fd60c1e03a0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c4d0 tx=0x7fd5fc00c990 comp rx=0 tx=0).stop 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 shutdown_connections 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd5f006b500 0x7fd5f006d9c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd60c1eedf0 0x7fd60c071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd60c152960 0x7fd60c1ee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 --2- 172.21.15.94:0/2259626950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd60c0ac260 0x7fd60c1e03a0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 >> 172.21.15.94:0/2259626950 conn(0x7fd60c0c70b0 msgr2=0x7fd60c06dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 shutdown_connections 2024-03-06T07:26:08.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.402+0000 7fd613bc7640 1 -- 172.21.15.94:0/2259626950 wait complete. 2024-03-06T07:26:08.626 INFO:tasks.ceph.mds.a.smithi094.stderr:2024-03-06T07:26:08.624+0000 7efcf172f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:08.626 INFO:tasks.ceph.mds.a.smithi094.stderr:2024-03-06T07:26:08.624+0000 7efcf172f640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:26:08.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 180, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:26:08.285097+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9891, 'mds_1': 9888}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9888': {'gid': 9888, 'name': 'c', 'rank': 1, 'incarnation': 179, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.94:6835/2744471722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2744471722}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2744471722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9891': {'gid': 9891, 'name': 'a', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.94:6834/4225816089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 4225816089}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 4225816089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9891, 'qdb_cluster': [9888, 9891]} 2024-03-06T07:26:08.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-03-06T07:26:08.818 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:26:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.922+0000 7f79ece81640 1 Processor -- start 2024-03-06T07:26:08.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.922+0000 7f79ece81640 1 -- start start 2024-03-06T07:26:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79ece81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79ece81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f79e80c8950 0x7f79e80acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79ece81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f79e80ad2b0 0x7f79e81558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79ece81640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e8155440 con 0x7f79e80ad2b0 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79ece81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e8155130 con 0x7f79e80c8950 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79ece81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80c5d50 con 0x7f79e80c69f0 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f79e80ad2b0 0x7f79e81558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f79e80ad2b0 0x7f79e81558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34434/0 (socket says 172.21.15.94:34434) 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 -- 172.21.15.94:0/1516549233 learned_addr learned my addr 172.21.15.94:0/1516549233 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 -- 172.21.15.94:0/1516549233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 msgr2=0x7f79e80c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e659c640 1 --2- 172.21.15.94:0/1516549233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/1516549233 >> v2:172.21.15.145:3300/0 conn(0x7f79e80c8950 0x7f79e80acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/1516549233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 -- 172.21.15.94:0/1516549233 >> v2:172.21.15.145:3300/0 conn(0x7f79e80c8950 msgr2=0x7f79e80acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/1516549233 >> v2:172.21.15.145:3300/0 conn(0x7f79e80c8950 0x7f79e80acd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.923+0000 7f79e6d9d640 1 -- 172.21.15.94:0/1516549233 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e80c6030 con 0x7f79e80ad2b0 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.924+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/1516549233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e81558d0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f79e80aa460 tx=0x7f79e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669a33e71123e743 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.924+0000 7f79e559a640 1 -- 172.21.15.94:0/1516549233 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e002c9e0 con 0x7f79e80ad2b0 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.924+0000 7f79e559a640 1 -- 172.21.15.94:0/1516549233 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e002cb80 con 0x7f79e80ad2b0 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.924+0000 7f79e559a640 1 -- 172.21.15.94:0/1516549233 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e00343f0 con 0x7f79e80ad2b0 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 -- 172.21.15.94:0/1516549233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 msgr2=0x7f79e81558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 --2- 172.21.15.94:0/1516549233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e81558d0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f79e80aa460 tx=0x7f79e002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 -- 172.21.15.94:0/1516549233 shutdown_connections 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 --2- 172.21.15.94:0/1516549233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e81558d0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 --2- 172.21.15.94:0/1516549233 >> v2:172.21.15.145:3300/0 conn(0x7f79e80c8950 0x7f79e80acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 --2- 172.21.15.94:0/1516549233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 -- 172.21.15.94:0/1516549233 >> 172.21.15.94:0/1516549233 conn(0x7f79e80cae70 msgr2=0x7f79e80cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 -- 172.21.15.94:0/1516549233 shutdown_connections 2024-03-06T07:26:08.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.925+0000 7f79ece81640 1 -- 172.21.15.94:0/1516549233 wait complete. 2024-03-06T07:26:08.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.926+0000 7f79ece81640 1 Processor -- start 2024-03-06T07:26:08.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.926+0000 7f79ece81640 1 -- start start 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79ece81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e80c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79ece81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80abc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79ece81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79e80c8950 0x7f79e8145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79ece81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8152ba0 con 0x7f79e80ad2b0 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79ece81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8152a20 con 0x7f79e80c8950 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79ece81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80c58e0 con 0x7f79e80c69f0 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e80c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e80c4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34448/0 (socket says 172.21.15.94:34448) 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 -- 172.21.15.94:0/1009422550 learned_addr learned my addr 172.21.15.94:0/1009422550 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:08.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80abc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79e80c8950 0x7f79e8145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 -- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 msgr2=0x7f79e80abc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80abc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 -- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79e80c8950 msgr2=0x7f79e8145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79e80c8950 0x7f79e8145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.927+0000 7f79e659c640 1 -- 172.21.15.94:0/1009422550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e81eabd0 con 0x7f79e80ad2b0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.928+0000 7f79e659c640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e80c4770 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f79dc00c820 tx=0x7f79dc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.928+0000 7f79d6ffd640 1 -- 172.21.15.94:0/1009422550 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc015070 con 0x7f79e80ad2b0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.928+0000 7f79d6ffd640 1 -- 172.21.15.94:0/1009422550 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc004220 con 0x7f79e80ad2b0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.928+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e81ee640 con 0x7f79e80ad2b0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.928+0000 7f79d6ffd640 1 -- 172.21.15.94:0/1009422550 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc00ce10 con 0x7f79e80ad2b0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.928+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e81eec00 con 0x7f79e80ad2b0 2024-03-06T07:26:08.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.929+0000 7f79d6ffd640 1 -- 172.21.15.94:0/1009422550 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79dc007830 con 0x7f79e80ad2b0 2024-03-06T07:26:08.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.930+0000 7f79d6ffd640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79c806b4d0 0x7f79c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.930+0000 7f79d6ffd640 1 -- 172.21.15.94:0/1009422550 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f79dc08a2c0 con 0x7f79e80ad2b0 2024-03-06T07:26:08.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.930+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79c806b4d0 0x7f79c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.930+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79c806b4d0 0x7f79c806d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f79e80c7190 tx=0x7f79d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.931+0000 7f79efa64740 1 Processor -- start 2024-03-06T07:26:08.934 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.932+0000 7f79efa64740 1 -- start start 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd67a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79efa64740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd6a330 0x55ee5bd67f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd684b0 0x55ee5bd64680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79efa64740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bcce6b0 con 0x55ee5bd6a330 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bc77a60 con 0x55ee5bd684b0 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bcca750 con 0x55ee5bd69f50 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd6a330 0x55ee5bd67f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd6a330 0x55ee5bd67f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34454/0 (socket says 172.21.15.94:34454) 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 -- 172.21.15.94:0/1883043572 learned_addr learned my addr 172.21.15.94:0/1883043572 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e659c640 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd67a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd684b0 0x55ee5bd64680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 -- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 msgr2=0x55ee5bd67a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd67a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 -- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd684b0 msgr2=0x55ee5bd64680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd684b0 0x55ee5bd64680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 -- 172.21.15.94:0/1883043572 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee5bc95f30 con 0x55ee5bd6a330 2024-03-06T07:26:08.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd6a330 0x55ee5bd67f70 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f79d800db30 tx=0x7f79d8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277baac9b225268b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.933+0000 7f79c7fff640 1 -- 172.21.15.94:0/1883043572 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8019050 con 0x55ee5bd6a330 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79c7fff640 1 -- 172.21.15.94:0/1883043572 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d8023070 con 0x55ee5bd6a330 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79c7fff640 1 -- 172.21.15.94:0/1883043572 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8012bd0 con 0x55ee5bd6a330 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 -- 172.21.15.94:0/1883043572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd6a330 msgr2=0x55ee5bd67f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd6a330 0x55ee5bd67f70 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f79d800db30 tx=0x7f79d8010000 comp rx=0 tx=0).stop 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 -- 172.21.15.94:0/1883043572 shutdown_connections 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd684b0 0x55ee5bd64680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd6a330 0x55ee5bd67f70 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 --2- 172.21.15.94:0/1883043572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd67a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 -- 172.21.15.94:0/1883043572 >> 172.21.15.94:0/1883043572 conn(0x55ee5bd667d0 msgr2=0x55ee5bd8b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 -- 172.21.15.94:0/1883043572 shutdown_connections 2024-03-06T07:26:08.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.934+0000 7f79efa64740 1 -- 172.21.15.94:0/1883043572 wait complete. 2024-03-06T07:26:08.942 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.935+0000 7f79efa64740 1 -- start start 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.935+0000 7f79efa64740 1 Processor -- start 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd8a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bd8df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5bd86dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bd8c230 con 0x55ee5bd69f50 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bc8aea0 con 0x55ee5bd6a330 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bd8bc70 con 0x55ee5bd8e4b0 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 10 client.0 _ll_register_callbacks cb 0x55ee5bc8a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd8a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd8a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34468/0 (socket says 172.21.15.94:34468) 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 -- 172.21.15.94:0/274659619 learned_addr learned my addr 172.21.15.94:0/274659619 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bd8df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5bd86dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 -- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 msgr2=0x55ee5bd86dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5bd86dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 -- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 msgr2=0x55ee5bd8df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bd8df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 -- 172.21.15.94:0/274659619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee5bc8b270 con 0x55ee5bd69f50 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5bd86dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bd8df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79e659c640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd8a7b0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f79dc058740 tx=0x7f79dc058870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9555ef7824c5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79c57fa640 1 -- 172.21.15.94:0/274659619 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc070020 con 0x55ee5bd69f50 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79c57fa640 1 -- 172.21.15.94:0/274659619 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f79dc0701c0 con 0x55ee5bd69f50 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79d8023070 con 0x55ee5bd69f50 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79c57fa640 1 -- 172.21.15.94:0/274659619 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79dc01e040 con 0x55ee5bd69f50 2024-03-06T07:26:08.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 10 client.10197 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.937+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x55ee5b9e32d0 con 0x55ee5bd69f50 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79c57fa640 1 -- 172.21.15.94:0/274659619 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d8023070 con 0x55ee5bd69f50 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79c57fa640 1 -- 172.21.15.94:0/274659619 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f79dc0880d0 con 0x55ee5bd69f50 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79efa64740 10 client.10197 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79efa64740 10 client.10197 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79efa64740 20 client.10197 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79efa64740 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdba6a0 0x55ee5bdc5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79efa64740 4 client.10197 mds_command: new command op to 9888 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 --> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee5bcdbbf0 con 0x55ee5bdba6a0 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdba6a0 0x55ee5bdc5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79c57fa640 10 client.10197 ms_handle_connect on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.938+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdba6a0 0x55ee5bdc5280 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b32abbd8a87cdb0d server_cookie=61ccf9f6a92b1151 in_seq=0 out_seq=0 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79c57fa640 1 -- 172.21.15.94:0/274659619 <== mds.1 v2:172.21.15.94:6833/2744471722 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ee5bcdbbf0 con 0x55ee5bdba6a0 2024-03-06T07:26:08.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79c57fa640 10 client.10197 handle_command_reply: tid=0 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 client.10197 shutdown 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 msgr2=0x55ee5bd8a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd8a7b0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f79dc058740 tx=0x7f79dc058870 comp rx=0 tx=0).stop 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 shutdown_connections 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5bd86dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bd8df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdba6a0 0x55ee5bdc5280 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 --2- 172.21.15.94:0/274659619 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd8a7b0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79c57fa640 0 client.10197 ms_handle_reset on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 >> 172.21.15.94:0/274659619 conn(0x55ee5bd684b0 msgr2=0x55ee5bd63510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 shutdown_connections 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.939+0000 7f79efa64740 1 -- 172.21.15.94:0/274659619 wait complete. 2024-03-06T07:26:08.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.940+0000 7f79efa64740 20 client.10197 trim_cache size 0 max 16384 2024-03-06T07:26:08.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.956+0000 7f79efa64740 1 Processor -- start 2024-03-06T07:26:08.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.956+0000 7f79efa64740 1 -- start start 2024-03-06T07:26:08.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd88f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bda3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79efa64740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd8e4b0 0x55ee5bda35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79efa64740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81f7e10 con 0x55ee5bd8e4b0 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80ab230 con 0x55ee5bd6a330 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81f6910 con 0x55ee5bd69f50 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd8e4b0 0x55ee5bda35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd8e4b0 0x55ee5bda35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34480/0 (socket says 172.21.15.94:34480) 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 -- 172.21.15.94:0/804937295 learned_addr learned my addr 172.21.15.94:0/804937295 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e659c640 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd88f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 -- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 msgr2=0x55ee5bd88f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bda3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd88f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 -- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 msgr2=0x55ee5bda3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bda3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 -- 172.21.15.94:0/804937295 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee5baad6b0 con 0x55ee5bd8e4b0 2024-03-06T07:26:08.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e659c640 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd88f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bda3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd8e4b0 0x55ee5bda35b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f79e002c450 tx=0x7f79e002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73a7e19c5723898 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79c77fe640 1 -- 172.21.15.94:0/804937295 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e002cb20 con 0x55ee5bd8e4b0 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79c77fe640 1 -- 172.21.15.94:0/804937295 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e0049430 con 0x55ee5bd8e4b0 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.957+0000 7f79c77fe640 1 -- 172.21.15.94:0/804937295 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0049820 con 0x55ee5bd8e4b0 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 -- 172.21.15.94:0/804937295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd8e4b0 msgr2=0x55ee5bda35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd8e4b0 0x55ee5bda35b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f79e002c450 tx=0x7f79e002e000 comp rx=0 tx=0).stop 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 -- 172.21.15.94:0/804937295 shutdown_connections 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd8e4b0 0x55ee5bda35b0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5bda3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 --2- 172.21.15.94:0/804937295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd69f50 0x55ee5bd88f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 -- 172.21.15.94:0/804937295 >> 172.21.15.94:0/804937295 conn(0x55ee5bd684b0 msgr2=0x55ee5bd8ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 -- 172.21.15.94:0/804937295 shutdown_connections 2024-03-06T07:26:08.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.958+0000 7f79efa64740 1 -- 172.21.15.94:0/804937295 wait complete. 2024-03-06T07:26:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.959+0000 7f79efa64740 1 -- start start 2024-03-06T07:26:08.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.959+0000 7f79efa64740 1 Processor -- start 2024-03-06T07:26:08.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.959+0000 7f79efa64740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd87a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.959+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5be164e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.959+0000 7f79efa64740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5be138a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79efa64740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bd8b2a0 con 0x55ee5bd69f50 2024-03-06T07:26:08.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bd8cce0 con 0x55ee5bd6a330 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79efa64740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ee5bda3b70 con 0x55ee5bd8e4b0 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79efa64740 10 client.0 _ll_register_callbacks cb 0x7f79dc058eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd87a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd87a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34494/0 (socket says 172.21.15.94:34494) 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 -- 172.21.15.94:0/2962814878 learned_addr learned my addr 172.21.15.94:0/2962814878 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5be164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5be138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 -- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 msgr2=0x55ee5be138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5be138a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 -- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 msgr2=0x55ee5be164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5be164e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 -- 172.21.15.94:0/2962814878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee5bd89030 con 0x55ee5bd69f50 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5be164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e6d9d640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5be138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79e659c640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd87a90 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f79dc060a20 tx=0x7f79dc0884c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f3c3dc6db0c74e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79c57fa640 1 -- 172.21.15.94:0/2962814878 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc070040 con 0x55ee5bd69f50 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79c57fa640 1 -- 172.21.15.94:0/2962814878 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f79dc0701e0 con 0x55ee5bd69f50 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79e0049430 con 0x55ee5bd69f50 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79c57fa640 1 -- 172.21.15.94:0/2962814878 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79dc05c900 con 0x55ee5bd69f50 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79efa64740 10 client.10203 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.960+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x55ee5bdc9bc0 con 0x55ee5bd69f50 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79c57fa640 1 -- 172.21.15.94:0/2962814878 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e0049430 con 0x55ee5bd69f50 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79c57fa640 1 -- 172.21.15.94:0/2962814878 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f79dc057c00 con 0x55ee5bd69f50 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 10 client.10203 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 10 client.10203 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 20 client.10203 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:08.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdb6630 0x55ee5bdb6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 4 client.10203 mds_command: new command op to 9888 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 --> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ee5bcd5f40 con 0x55ee5bdb6630 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdb6630 0x55ee5bdb6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79c57fa640 10 client.10203 ms_handle_connect on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79e5d9b640 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdb6630 0x55ee5bdb6a10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=177209d82200ba23 server_cookie=799842a79020bfe3 in_seq=0 out_seq=0 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79c57fa640 1 -- 172.21.15.94:0/2962814878 <== mds.1 v2:172.21.15.94:6833/2744471722 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55ee5bcd5f40 con 0x55ee5bdb6630 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79c57fa640 10 client.10203 handle_command_reply: tid=0 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 1 client.10203 shutdown 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.961+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 msgr2=0x55ee5bd87a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd87a90 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f79dc060a20 tx=0x7f79dc0884c0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 shutdown_connections 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ee5bd8e4b0 0x55ee5be138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ee5bd6a330 0x55ee5be164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ee5bdb6630 0x55ee5bdb6a10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 --2- 172.21.15.94:0/2962814878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ee5bd69f50 0x55ee5bd87a90 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79c57fa640 0 client.10203 ms_handle_reset on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 >> 172.21.15.94:0/2962814878 conn(0x55ee5bd684b0 msgr2=0x55ee5bdc92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 shutdown_connections 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 1 -- 172.21.15.94:0/2962814878 wait complete. 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79efa64740 20 client.10203 trim_cache size 0 max 16384 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 9888, 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 1.6770632809999999, 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 180, 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 184, 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 184, 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 35.711904853999997 2024-03-06T07:26:08.971 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:26:08.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79c806b4d0 msgr2=0x7f79c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79c806b4d0 0x7f79c806d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f79e80c7190 tx=0x7f79d8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 msgr2=0x7f79e80c4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e80c4770 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f79dc00c820 tx=0x7f79dc00cce0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 shutdown_connections 2024-03-06T07:26:08.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f79c806b4d0 0x7f79c806d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79e80c8950 0x7f79e8145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79e80c69f0 0x7f79e80abc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 --2- 172.21.15.94:0/1009422550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79e80ad2b0 0x7f79e80c4770 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 >> 172.21.15.94:0/1009422550 conn(0x7f79e80cae70 msgr2=0x7f79e8155720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 shutdown_connections 2024-03-06T07:26:08.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:08.962+0000 7f79ece81640 1 -- 172.21.15.94:0/1009422550 wait complete. 2024-03-06T07:26:08.978 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:09.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb764a4640 1 Processor -- start 2024-03-06T07:26:09.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb764a4640 1 -- start start 2024-03-06T07:26:09.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb764a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb700c8950 0x7ffb700c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb764a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffb701558d0 0x7ffb70155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb764a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb70155130 con 0x7ffb700c69f0 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb764a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb70155440 con 0x7ffb700c8950 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb764a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb700ad480 con 0x7ffb701558d0 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.079+0000 7ffb6f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb700c8950 0x7ffb700c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb6f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb700c8950 0x7ffb700c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34502/0 (socket says 172.21.15.94:34502) 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/3331807423 learned_addr learned my addr 172.21.15.94:0/3331807423 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb74a41640 1 --2- 172.21.15.94:0/3331807423 >> v2:172.21.15.145:3300/0 conn(0x7ffb701558d0 0x7ffb70155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/3331807423 >> v2:172.21.15.145:3300/0 conn(0x7ffb701558d0 msgr2=0x7ffb70155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/3331807423 >> v2:172.21.15.145:3300/0 conn(0x7ffb701558d0 0x7ffb70155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/3331807423 >> v1:172.21.15.94:6789/0 conn(0x7ffb700c69f0 legacy=0x7ffb700acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb6effd640 1 -- 172.21.15.94:0/3331807423 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469414405 0 0) 0x7ffb60001110 con 0x7ffb700c69f0 2024-03-06T07:26:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/3331807423 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700c5d50 con 0x7ffb700c8950 2024-03-06T07:26:09.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.080+0000 7ffb74a41640 1 --2- 172.21.15.94:0/3331807423 >> v2:172.21.15.145:3300/0 conn(0x7ffb701558d0 0x7ffb70155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.081+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/3331807423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c6dd0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7ffb70074a60 tx=0x7ffb6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cd0b83c62c53b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.081+0000 7ffb6effd640 1 -- 172.21.15.94:0/3331807423 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64035020 con 0x7ffb700c8950 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.081+0000 7ffb6effd640 1 -- 172.21.15.94:0/3331807423 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64031870 con 0x7ffb700c8950 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.081+0000 7ffb6effd640 1 -- 172.21.15.94:0/3331807423 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64031c40 con 0x7ffb700c8950 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 -- 172.21.15.94:0/3331807423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 msgr2=0x7ffb700c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 --2- 172.21.15.94:0/3331807423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c6dd0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7ffb70074a60 tx=0x7ffb6402b910 comp rx=0 tx=0).stop 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 -- 172.21.15.94:0/3331807423 shutdown_connections 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 --2- 172.21.15.94:0/3331807423 >> v2:172.21.15.145:3300/0 conn(0x7ffb701558d0 0x7ffb70155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 --2- 172.21.15.94:0/3331807423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c6dd0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 -- 172.21.15.94:0/3331807423 >> 172.21.15.94:0/3331807423 conn(0x7ffb700cae70 msgr2=0x7ffb700cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 -- 172.21.15.94:0/3331807423 shutdown_connections 2024-03-06T07:26:09.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.082+0000 7ffb764a4640 1 -- 172.21.15.94:0/3331807423 wait complete. 2024-03-06T07:26:09.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.084+0000 7ffb764a4640 1 Processor -- start 2024-03-06T07:26:09.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.084+0000 7ffb764a4640 1 -- start start 2024-03-06T07:26:09.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb764a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb700c69f0 0x7ffb70158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb764a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb764a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb701558d0 0x7ffb701eabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb764a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb70152a20 con 0x7ffb700c8950 2024-03-06T07:26:09.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb764a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700c6310 con 0x7ffb701558d0 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb764a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb70152ba0 con 0x7ffb700c69f0 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34512/0 (socket says 172.21.15.94:34512) 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/125587226 learned_addr learned my addr 172.21.15.94:0/125587226 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb700c69f0 0x7ffb70158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.085+0000 7ffb74a41640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb701558d0 0x7ffb701eabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb700c69f0 msgr2=0x7ffb70158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb700c69f0 0x7ffb70158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb701558d0 msgr2=0x7ffb701eabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb701558d0 0x7ffb701eabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/125587226 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb70159060 con 0x7ffb700c8950 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c43f0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ffb64000c00 tx=0x7ffb640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6cff9640 1 -- 172.21.15.94:0/125587226 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64035040 con 0x7ffb700c8950 2024-03-06T07:26:09.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6cff9640 1 -- 172.21.15.94:0/125587226 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64030d00 con 0x7ffb700c8950 2024-03-06T07:26:09.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb6cff9640 1 -- 172.21.15.94:0/125587226 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640456a0 con 0x7ffb700c8950 2024-03-06T07:26:09.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb700c1b30 con 0x7ffb700c8950 2024-03-06T07:26:09.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.086+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb700c20f0 con 0x7ffb700c8950 2024-03-06T07:26:09.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.087+0000 7ffb6cff9640 1 -- 172.21.15.94:0/125587226 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb64038020 con 0x7ffb700c8950 2024-03-06T07:26:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.088+0000 7ffb6cff9640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.088+0000 7ffb6cff9640 1 -- 172.21.15.94:0/125587226 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7ffb640ac5d0 con 0x7ffb700c8950 2024-03-06T07:26:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.088+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.089+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffb60000f60 tx=0x7ffb60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.090+0000 7ffb7916f740 1 Processor -- start 2024-03-06T07:26:09.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.090+0000 7ffb7916f740 1 -- start start 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a77a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2a77f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a784b0 0x55c2b2a74680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb7916f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b29de6b0 con 0x55c2b2a79f50 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b2987a60 con 0x55c2b2a784b0 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2b29da750 con 0x55c2b2a7a330 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a77a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a77a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34516/0 (socket says 172.21.15.94:34516) 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1239369388 learned_addr learned my addr 172.21.15.94:0/1239369388 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2a77f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb74a41640 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a784b0 0x55c2b2a74680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 msgr2=0x55c2b2a77f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2a77f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a784b0 msgr2=0x55c2b2a74680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a784b0 0x55c2b2a74680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1239369388 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2b29a5f30 con 0x55c2b2a79f50 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb74a41640 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a784b0 0x55c2b2a74680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a77a30 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000fb70 tx=0x7ffb60012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73fe17fe8df1c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb55ffb640 1 -- 172.21.15.94:0/1239369388 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60025070 con 0x55c2b2a79f50 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb55ffb640 1 -- 172.21.15.94:0/1239369388 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6001b050 con 0x55c2b2a79f50 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.091+0000 7ffb55ffb640 1 -- 172.21.15.94:0/1239369388 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60014de0 con 0x55c2b2a79f50 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 -- 172.21.15.94:0/1239369388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 msgr2=0x55c2b2a77a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a77a30 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000fb70 tx=0x7ffb60012000 comp rx=0 tx=0).stop 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 -- 172.21.15.94:0/1239369388 shutdown_connections 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a784b0 0x55c2b2a74680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2a77f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1239369388 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a77a30 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 -- 172.21.15.94:0/1239369388 >> 172.21.15.94:0/1239369388 conn(0x55c2b2a767d0 msgr2=0x55c2b2a9b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 -- 172.21.15.94:0/1239369388 shutdown_connections 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 -- 172.21.15.94:0/1239369388 wait complete. 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.092+0000 7ffb7916f740 1 -- start start 2024-03-06T07:26:09.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 1 Processor -- start 2024-03-06T07:26:09.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2a974f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a7a330 0x55c2b2aa00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2ac6a30 0x55c2b2a9de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b2a9c230 con 0x55c2b2a7a330 2024-03-06T07:26:09.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b299aea0 con 0x55c2b2a79f50 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2b2a9bc70 con 0x55c2b2ac6a30 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.093+0000 7ffb7916f740 10 client.0 _ll_register_callbacks cb 0x55c2b299a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a7a330 0x55c2b2aa00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a7a330 0x55c2b2aa00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34528/0 (socket says 172.21.15.94:34528) 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/439738540 learned_addr learned my addr 172.21.15.94:0/439738540 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2a974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb74a41640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2ac6a30 0x55c2b2a9de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2ac6a30 msgr2=0x55c2b2a9de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2ac6a30 0x55c2b2a9de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 msgr2=0x55c2b2a974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2a974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 -- 172.21.15.94:0/439738540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2b299b270 con 0x55c2b2a7a330 2024-03-06T07:26:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb74a41640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2ac6a30 0x55c2b2a9de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2a974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a7a330 0x55c2b2aa00f0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7ffb640770f0 tx=0x7ffb6407a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b4cedce21fee4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb4effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb4effd640 1 -- 172.21.15.94:0/439738540 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64083060 con 0x55c2b2a7a330 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb6001b050 con 0x55c2b2a7a330 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb4effd640 1 -- 172.21.15.94:0/439738540 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7ffb6408eba0 con 0x55c2b2a7a330 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb4effd640 1 -- 172.21.15.94:0/439738540 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb64099050 con 0x55c2b2a7a330 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb7916f740 10 client.10218 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x55c2b26f32d0 con 0x55c2b2a7a330 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.094+0000 7ffb4effd640 1 -- 172.21.15.94:0/439738540 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6001b050 con 0x55c2b2a7a330 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb4effd640 1 -- 172.21.15.94:0/439738540 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7ffb640a5030 con 0x55c2b2a7a330 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb7916f740 10 client.10218 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb7916f740 10 client.10218 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:09.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb7916f740 20 client.10218 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb7916f740 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2a9d1f0 0x55c2b2ad56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb7916f740 4 client.10218 mds_command: new command op to 9891 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 --> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2b29ebbf0 con 0x55c2b2a9d1f0 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2a9d1f0 0x55c2b2ad56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb4effd640 10 client.10218 ms_handle_connect on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.095+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2a9d1f0 0x55c2b2ad56a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59634ede28120a1b server_cookie=ae22131cc5a3db8b in_seq=0 out_seq=0 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb4effd640 1 -- 172.21.15.94:0/439738540 <== mds.0 v2:172.21.15.94:6832/4225816089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2b29ebbf0 con 0x55c2b2a9d1f0 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb4effd640 10 client.10218 handle_command_reply: tid=0 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 client.10218 shutdown 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a7a330 msgr2=0x55c2b2aa00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a7a330 0x55c2b2aa00f0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7ffb640770f0 tx=0x7ffb6407a5f0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 shutdown_connections 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2a9d1f0 0x55c2b2ad56a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2ac6a30 0x55c2b2a9de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb4effd640 0 client.10218 ms_handle_reset on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a7a330 0x55c2b2aa00f0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 --2- 172.21.15.94:0/439738540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2a974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 >> 172.21.15.94:0/439738540 conn(0x55c2b2a784b0 msgr2=0x55c2b2ab6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 shutdown_connections 2024-03-06T07:26:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 1 -- 172.21.15.94:0/439738540 wait complete. 2024-03-06T07:26:09.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.096+0000 7ffb7916f740 20 client.10218 trim_cache size 0 max 16384 2024-03-06T07:26:09.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.124+0000 7ffb7916f740 1 Processor -- start 2024-03-06T07:26:09.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.124+0000 7ffb7916f740 1 -- start start 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2aa0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2ac3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb7916f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb701f7d80 con 0x55c2b2aa0bd0 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7014ce20 con 0x55c2b2a79f50 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb701f6930 con 0x55c2b2a7a330 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb74a41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb74a41640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34538/0 (socket says 172.21.15.94:34538) 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb74a41640 1 -- 172.21.15.94:0/950622690 learned_addr learned my addr 172.21.15.94:0/950622690 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2ac3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2aa0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb74a41640 1 -- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 msgr2=0x55c2b2ac3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb74a41640 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2ac3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb74a41640 1 -- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 msgr2=0x55c2b2aa0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.125+0000 7ffb74a41640 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2aa0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb74a41640 1 -- 172.21.15.94:0/950622690 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2b27bd6b0 con 0x55c2b2aa0bd0 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2ac3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2aa0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb74a41640 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac3e60 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7ffb68003b20 tx=0x7ffb680097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4839b8b86b6ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb557fa640 1 -- 172.21.15.94:0/950622690 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6801b020 con 0x55c2b2aa0bd0 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb557fa640 1 -- 172.21.15.94:0/950622690 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb68004800 con 0x55c2b2aa0bd0 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb557fa640 1 -- 172.21.15.94:0/950622690 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68017880 con 0x55c2b2aa0bd0 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 -- 172.21.15.94:0/950622690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2aa0bd0 msgr2=0x55c2b2ac3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac3e60 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7ffb68003b20 tx=0x7ffb680097d0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 -- 172.21.15.94:0/950622690 shutdown_connections 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac3e60 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2ac3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 --2- 172.21.15.94:0/950622690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2a79f50 0x55c2b2aa0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 -- 172.21.15.94:0/950622690 >> 172.21.15.94:0/950622690 conn(0x55c2b2a784b0 msgr2=0x55c2b2a97300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 -- 172.21.15.94:0/950622690 shutdown_connections 2024-03-06T07:26:09.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.126+0000 7ffb7916f740 1 -- 172.21.15.94:0/950622690 wait complete. 2024-03-06T07:26:09.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.127+0000 7ffb7916f740 1 -- start start 2024-03-06T07:26:09.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.127+0000 7ffb7916f740 1 Processor -- start 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a99f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2b29ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb7916f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb7916f740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b2ac6e40 con 0x55c2b2a79f50 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b2a9ad10 con 0x55c2b2aa0bd0 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb7916f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2b2a9c600 con 0x55c2b2a7a330 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb7916f740 10 client.0 _ll_register_callbacks cb 0x7ffb64077000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a99f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.135 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a99f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34554/0 (socket says 172.21.15.94:34554) 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1495815189 learned_addr learned my addr 172.21.15.94:0/1495815189 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2b29ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 msgr2=0x55c2b2b29ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2b29ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb74a41640 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2aa0bd0 msgr2=0x55c2b2ac32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.128+0000 7ffb6ffff640 1 -- 172.21.15.94:0/1495815189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2b2a9ffe0 con 0x55c2b2a79f50 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb6ffff640 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a99f20 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7ffb60023040 tx=0x7ffb60001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21964f149feaff2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb4effd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb4effd640 1 -- 172.21.15.94:0/1495815189 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60025040 con 0x55c2b2a79f50 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb4effd640 1 -- 172.21.15.94:0/1495815189 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7ffb60007030 con 0x55c2b2a79f50 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb68004800 con 0x55c2b2a79f50 2024-03-06T07:26:09.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb4effd640 1 -- 172.21.15.94:0/1495815189 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb60041050 con 0x55c2b2a79f50 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 10 client.10224 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x55c2b2ac4440 con 0x55c2b2a79f50 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb4effd640 1 -- 172.21.15.94:0/1495815189 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb68004800 con 0x55c2b2a79f50 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb4effd640 1 -- 172.21.15.94:0/1495815189 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7ffb60044020 con 0x55c2b2a79f50 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 10 client.10224 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 10 client.10224 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 20 client.10224 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2b241b0 0x55c2b2ab2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 4 client.10224 mds_command: new command op to 9891 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 --> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c2b29e5f40 con 0x55c2b2b241b0 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.129+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2b241b0 0x55c2b2ab2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb4effd640 10 client.10224 ms_handle_connect on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb6f7fe640 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2b241b0 0x55c2b2ab2710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d388b07217dc8c4 server_cookie=1f8afac13c7de271 in_seq=0 out_seq=0 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb4effd640 1 -- 172.21.15.94:0/1495815189 <== mds.0 v2:172.21.15.94:6832/4225816089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55c2b29e5f40 con 0x55c2b2b241b0 2024-03-06T07:26:09.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb4effd640 10 client.10224 handle_command_reply: tid=0 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 client.10224 shutdown 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 msgr2=0x55c2b2a99f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a99f20 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7ffb60023040 tx=0x7ffb60001080 comp rx=0 tx=0).stop 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 shutdown_connections 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x55c2b2b241b0 0x55c2b2ab2710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb4effd640 0 client.10224 ms_handle_reset on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c2b2aa0bd0 0x55c2b2ac32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c2b2a7a330 0x55c2b2b29ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 --2- 172.21.15.94:0/1495815189 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c2b2a79f50 0x55c2b2a99f20 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 >> 172.21.15.94:0/1495815189 conn(0x55c2b2a784b0 msgr2=0x55c2b2ac6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 shutdown_connections 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 1 -- 172.21.15.94:0/1495815189 wait complete. 2024-03-06T07:26:09.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.130+0000 7ffb7916f740 20 client.10224 trim_cache size 0 max 16384 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 9891, 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 2.8602600869999999, 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 180, 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 184, 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 184, 2024-03-06T07:26:09.139 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 35.880676893999997 2024-03-06T07:26:09.140 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:26:09.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb5c06b4d0 msgr2=0x7ffb5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffb60000f60 tx=0x7ffb60009040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 msgr2=0x7ffb700c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c43f0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ffb64000c00 tx=0x7ffb640046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 shutdown_connections 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb5c06b4d0 0x7ffb5c06d990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb701558d0 0x7ffb701eabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb700c8950 0x7ffb700c43f0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 --2- 172.21.15.94:0/125587226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb700c69f0 0x7ffb70158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 >> 172.21.15.94:0/125587226 conn(0x7ffb700cae70 msgr2=0x7ffb70150240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 shutdown_connections 2024-03-06T07:26:09.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.131+0000 7ffb764a4640 1 -- 172.21.15.94:0/125587226 wait complete. 2024-03-06T07:26:09.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.259+0000 7faea7c0e640 1 Processor -- start 2024-03-06T07:26:09.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.260+0000 7faea7c0e640 1 -- start start 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.260+0000 7faea7c0e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faea0159270 0x7faea0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.260+0000 7faea7c0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea00c73d0 con 0x7faea014f360 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.260+0000 7faea7c0e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea00be2e0 con 0x7faea00c88c0 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.260+0000 7faea7c0e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea0159700 con 0x7faea0159270 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.260+0000 7faea59aa640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7faea014f360 0x7faea00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39220/0 (socket says 172.21.15.94:39220) 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.260+0000 7faea59aa640 1 -- 172.21.15.94:0/3637033271 learned_addr learned my addr 172.21.15.94:0/3637033271 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.261+0000 7faea61ab640 1 --2- 172.21.15.94:0/3637033271 >> v2:172.21.15.145:3300/0 conn(0x7faea0159270 0x7faea0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.261+0000 7faea49a8640 1 -- 172.21.15.94:0/3637033271 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568566711 0 0) 0x7faea00c73d0 con 0x7faea014f360 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.261+0000 7faea49a8640 1 -- 172.21.15.94:0/3637033271 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae88003520 con 0x7faea014f360 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.261+0000 7faea61ab640 1 -- 172.21.15.94:0/3637033271 >> v1:172.21.15.145:6789/0 conn(0x7faea00c88c0 legacy=0x7faea0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.261+0000 7faea61ab640 1 -- 172.21.15.94:0/3637033271 >> v1:172.21.15.94:6789/0 conn(0x7faea014f360 legacy=0x7faea00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.261+0000 7faea49a8640 1 -- 172.21.15.94:0/3637033271 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 267089207 0 0) 0x7fae88003520 con 0x7faea014f360 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.261+0000 7faea61ab640 1 -- 172.21.15.94:0/3637033271 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00c7670 con 0x7faea0159270 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.262+0000 7faea61ab640 1 --2- 172.21.15.94:0/3637033271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea0155c10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fae900097a0 tx=0x7fae9002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f109a2e11b26016b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.262+0000 7faea49a8640 1 -- 172.21.15.94:0/3637033271 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90035020 con 0x7faea0159270 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.262+0000 7faea49a8640 1 -- 172.21.15.94:0/3637033271 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae90031830 con 0x7faea0159270 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.262+0000 7faea49a8640 1 -- 172.21.15.94:0/3637033271 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90031c00 con 0x7faea0159270 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.263+0000 7faea7c0e640 1 -- 172.21.15.94:0/3637033271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 msgr2=0x7faea0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.263+0000 7faea7c0e640 1 --2- 172.21.15.94:0/3637033271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea0155c10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fae900097a0 tx=0x7fae9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.263+0000 7faea7c0e640 1 -- 172.21.15.94:0/3637033271 shutdown_connections 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.263+0000 7faea7c0e640 1 --2- 172.21.15.94:0/3637033271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea0155c10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.263+0000 7faea7c0e640 1 -- 172.21.15.94:0/3637033271 >> 172.21.15.94:0/3637033271 conn(0x7faea00ac570 msgr2=0x7faea00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.263+0000 7faea7c0e640 1 -- 172.21.15.94:0/3637033271 shutdown_connections 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.263+0000 7faea7c0e640 1 -- 172.21.15.94:0/3637033271 wait complete. 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.265+0000 7faea7c0e640 1 Processor -- start 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.265+0000 7faea7c0e640 1 -- start start 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea7c0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faea00c88c0 0x7faea014dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea7c0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faea014f360 0x7faea014ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea7c0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea014b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea7c0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea00c7c30 con 0x7faea00c88c0 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea7c0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea00c5a50 con 0x7faea0159270 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea7c0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea00c5bd0 con 0x7faea014f360 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faea00c88c0 0x7faea014dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faea00c88c0 0x7faea014dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34564/0 (socket says 172.21.15.94:34564) 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 -- 172.21.15.94:0/2697995502 learned_addr learned my addr 172.21.15.94:0/2697995502 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea51a9640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faea014f360 0x7faea014ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea61ab640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea014b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 -- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faea014f360 msgr2=0x7faea014ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faea014f360 0x7faea014ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 -- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 msgr2=0x7faea014b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea014b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea59aa640 1 -- 172.21.15.94:0/2697995502 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea014b5c0 con 0x7faea00c88c0 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea51a9640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faea014f360 0x7faea014ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.266+0000 7faea61ab640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea014b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.267+0000 7faea59aa640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faea00c88c0 0x7faea014dba0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c480 tx=0x7fae9800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.267+0000 7fae9e7fc640 1 -- 172.21.15.94:0/2697995502 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98017070 con 0x7faea00c88c0 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.267+0000 7fae9e7fc640 1 -- 172.21.15.94:0/2697995502 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae98005d00 con 0x7faea00c88c0 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.267+0000 7fae9e7fc640 1 -- 172.21.15.94:0/2697995502 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae980072b0 con 0x7faea00c88c0 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.267+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea014bd70 con 0x7faea00c88c0 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.267+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea00ae1c0 con 0x7faea00c88c0 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.268+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea004f330 con 0x7faea00c88c0 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.269+0000 7fae9e7fc640 1 -- 172.21.15.94:0/2697995502 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae9801a070 con 0x7faea00c88c0 2024-03-06T07:26:09.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.269+0000 7fae9e7fc640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae8c06b500 0x7fae8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.269+0000 7fae9e7fc640 1 -- 172.21.15.94:0/2697995502 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fae98089790 con 0x7faea00c88c0 2024-03-06T07:26:09.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.269+0000 7faea51a9640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae8c06b500 0x7fae8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.270+0000 7faea51a9640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae8c06b500 0x7fae8c06d9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7faea01e85e0 tx=0x7fae94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.273+0000 7fae9e7fc640 1 -- 172.21.15.94:0/2697995502 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae980584a0 con 0x7faea00c88c0 2024-03-06T07:26:09.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.529+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faea01eaa80 con 0x7faea00c88c0 2024-03-06T07:26:09.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.530+0000 7fae9e7fc640 1 -- 172.21.15.94:0/2697995502 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5014 (secure 0 0 0) 0x7fae98057da0 con 0x7faea00c88c0 2024-03-06T07:26:09.532 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:09.532 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":180,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:08.285097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9891,"mds_1":9888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9888":{"gid":9888,"name":"c","rank":1,"incarnation":179,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9891":{"gid":9891,"name":"a","rank":0,"incarnation":177,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9891,"qdb_cluster":[9888,9891]},"id":14}]} 2024-03-06T07:26:09.532 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 180 2024-03-06T07:26:09.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.531+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae8c06b500 msgr2=0x7fae8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.531+0000 7faea7c0e640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae8c06b500 0x7fae8c06d9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7faea01e85e0 tx=0x7fae94009040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.531+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faea00c88c0 msgr2=0x7faea014dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.531+0000 7faea7c0e640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faea00c88c0 0x7faea014dba0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fae9800c480 tx=0x7fae9800c940 comp rx=0 tx=0).stop 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 shutdown_connections 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fae8c06b500 0x7fae8c06d9c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faea0159270 0x7faea014b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faea014f360 0x7faea014ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 --2- 172.21.15.94:0/2697995502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faea00c88c0 0x7faea014dba0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 >> 172.21.15.94:0/2697995502 conn(0x7faea00ac570 msgr2=0x7faea00ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 shutdown_connections 2024-03-06T07:26:09.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:09.532+0000 7faea7c0e640 1 -- 172.21.15.94:0/2697995502 wait complete. 2024-03-06T07:26:09.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 180, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.699027+0000', 'modified': '2024-03-06T07:26:08.285097+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 9891, 'mds_1': 9888}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9888': {'gid': 9888, 'name': 'c', 'rank': 1, 'incarnation': 179, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.94:6835/2744471722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6833', 'nonce': 2744471722}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2744471722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9891': {'gid': 9891, 'name': 'a', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.94:6834/4225816089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 4225816089}, {'type': 'v1', 'addr': '172.21.15.94:6834', 'nonce': 4225816089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9891, 'qdb_cluster': [9888, 9891]} 2024-03-06T07:26:09.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-03-06T07:26:09.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:26:10.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.052+0000 7f6d213eb640 1 Processor -- start 2024-03-06T07:26:10.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.052+0000 7f6d213eb640 1 -- start start 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d213eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 0x7f6d1c0c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d213eb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c152030 con 0x7f6d1c0c30b0 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d213eb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c152db0 con 0x7f6d1c155510 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d213eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c152920 con 0x7f6d1c0c4bb0 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d1a7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6d1c0c30b0 0x7f6d1c14f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39230/0 (socket says 172.21.15.94:39230) 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d1a7fc640 1 -- 172.21.15.94:0/1886364551 learned_addr learned my addr 172.21.15.94:0/1886364551 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156860129 0 0) 0x7f6d1c152030 con 0x7f6d1c0c30b0 2024-03-06T07:26:10.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d1affd640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 0x7f6d1c0c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.053+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d04003520 con 0x7f6d1c0c30b0 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3602214921 0 0) 0x7f6d04003520 con 0x7f6d1c0c30b0 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 msgr2=0x7f6d1c0c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 0x7f6d1c0c3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 >> v1:172.21.15.145:6789/0 conn(0x7f6d1c155510 legacy=0x7f6d1c155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c1522d0 con 0x7f6d1c0c30b0 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f6d10001670 con 0x7f6d1c0c30b0 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 >> v1:172.21.15.94:6789/0 conn(0x7f6d1c0c30b0 legacy=0x7f6d1c14f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d040039e0 0x7f6d04003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 0x7f6d04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d04004d40 0x7f6d040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c152920 con 0x7f6d04004d40 2024-03-06T07:26:10.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0400baf0 con 0x7f6d040039e0 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0400bcf0 con 0x7f6d1c0c4bb0 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d04004d40 0x7f6d040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 0x7f6d04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1affd640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d040039e0 0x7f6d04003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 msgr2=0x7f6d04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c0c4bb0 0x7f6d04004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d040039e0 msgr2=0x7f6d04003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d040039e0 0x7f6d04003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.054+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/1886364551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d04003520 con 0x7f6d04004d40 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.055+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d04004d40 0x7f6d040051c0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f6d14001e20 tx=0x7f6d04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3aa69782356487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.055+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14001a70 con 0x7f6d04004d40 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.055+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d14003b70 con 0x7f6d04004d40 2024-03-06T07:26:10.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.055+0000 7f6d19ffb640 1 -- 172.21.15.94:0/1886364551 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d140050d0 con 0x7f6d04004d40 2024-03-06T07:26:10.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.056+0000 7f6d213eb640 1 -- 172.21.15.94:0/1886364551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d04004d40 msgr2=0x7f6d040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.056+0000 7f6d213eb640 1 --2- 172.21.15.94:0/1886364551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d04004d40 0x7f6d040051c0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f6d14001e20 tx=0x7f6d04005830 comp rx=0 tx=0).stop 2024-03-06T07:26:10.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.056+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/1886364551 reap_dead start 2024-03-06T07:26:10.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.056+0000 7f6d213eb640 1 -- 172.21.15.94:0/1886364551 shutdown_connections 2024-03-06T07:26:10.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.056+0000 7f6d213eb640 1 -- 172.21.15.94:0/1886364551 >> 172.21.15.94:0/1886364551 conn(0x7f6d1c0c7de0 msgr2=0x7f6d1c0c4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.056+0000 7f6d213eb640 1 -- 172.21.15.94:0/1886364551 shutdown_connections 2024-03-06T07:26:10.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.056+0000 7f6d213eb640 1 -- 172.21.15.94:0/1886364551 wait complete. 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.057+0000 7f6d213eb640 1 Processor -- start 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.057+0000 7f6d213eb640 1 -- start start 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d213eb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d1c153c90 0x7f6d1c153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d213eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c1432e0 0x7f6d1c1eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d213eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d1c1ebae0 0x7f6d1c0c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d213eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c14f830 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d213eb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0c3fd0 con 0x7f6d1c153c90 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d213eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c1dfeb0 con 0x7f6d1c1432e0 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d1c1ebae0 0x7f6d1c0c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d1c1ebae0 0x7f6d1c0c6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34572/0 (socket says 172.21.15.94:34572) 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2198992435 learned_addr learned my addr 172.21.15.94:0/2198992435 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c1432e0 0x7f6d1c1eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d1c153c90 0x7f6d1c153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c1432e0 msgr2=0x7f6d1c1eb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c1432e0 0x7f6d1c1eb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d1c153c90 msgr2=0x7f6d1c153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d1c153c90 0x7f6d1c153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2198992435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c142b50 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c1432e0 0x7f6d1c1eb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.058+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d1c1ebae0 0x7f6d1c0c6b20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f6d1402c040 tx=0x7f6d140014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.059+0000 7f6d037fe640 1 -- 172.21.15.94:0/2198992435 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14001a70 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.059+0000 7f6d037fe640 1 -- 172.21.15.94:0/2198992435 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d14003970 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.059+0000 7f6d037fe640 1 -- 172.21.15.94:0/2198992435 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14007dd0 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.059+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1c1eadc0 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.059+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c0a9dd0 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.061+0000 7f6d037fe640 1 -- 172.21.15.94:0/2198992435 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d14041030 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.061+0000 7f6d037fe640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d0806b4d0 0x7f6d0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.061+0000 7f6d037fe640 1 -- 172.21.15.94:0/2198992435 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f6d140ab3c0 con 0x7f6d1c1ebae0 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.061+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d0806b4d0 0x7f6d0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.062+0000 7f6d23fcc740 1 Processor -- start 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.062+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d0806b4d0 0x7f6d0806d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009b00 tx=0x7f6d0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.062+0000 7f6d23fcc740 1 -- start start 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb1fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb1ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb204b0 0x55ae2fb1c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fa866b0 con 0x55ae2fb204b0 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fa2fa60 con 0x55ae2fb21f50 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fa82750 con 0x55ae2fb22330 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb204b0 0x55ae2fb1c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb204b0 0x55ae2fb1c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34584/0 (socket says 172.21.15.94:34584) 2024-03-06T07:26:10.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2362000011 learned_addr learned my addr 172.21.15.94:0/2362000011 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb1ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb1fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 msgr2=0x55ae2fb1ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb1ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 msgr2=0x55ae2fb1fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb1fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/2362000011 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae2fa4df30 con 0x55ae2fb204b0 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb1fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb204b0 0x55ae2fb1c680 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f6d14081b20 tx=0x7f6d14079fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df5012c486cdf37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d00ff9640 1 -- 172.21.15.94:0/2362000011 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14048070 con 0x55ae2fb204b0 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d00ff9640 1 -- 172.21.15.94:0/2362000011 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d14082050 con 0x55ae2fb204b0 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.063+0000 7f6d00ff9640 1 -- 172.21.15.94:0/2362000011 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14077950 con 0x55ae2fb204b0 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2362000011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb204b0 msgr2=0x55ae2fb1c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb204b0 0x55ae2fb1c680 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f6d14081b20 tx=0x7f6d14079fb0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2362000011 shutdown_connections 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb204b0 0x55ae2fb1c680 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb1ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2362000011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb1fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2362000011 >> 172.21.15.94:0/2362000011 conn(0x55ae2fb1e7d0 msgr2=0x55ae2fb433c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2362000011 shutdown_connections 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.064+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2362000011 wait complete. 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.065+0000 7f6d23fcc740 1 -- start start 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.065+0000 7f6d23fcc740 1 Processor -- start 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb21f50 0x55ae2fb46d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb22330 0x55ae2fb45b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb71f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fa866b0 con 0x55ae2fb22330 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fa42ea0 con 0x55ae2fb460a0 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fb44230 con 0x55ae2fb21f50 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 10 client.0 _ll_register_callbacks cb 0x55ae2fa42870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb22330 0x55ae2fb45b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb22330 0x55ae2fb45b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34594/0 (socket says 172.21.15.94:34594) 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 -- 172.21.15.94:0/2296757526 learned_addr learned my addr 172.21.15.94:0/2296757526 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb21f50 0x55ae2fb46d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb71f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 -- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb21f50 msgr2=0x55ae2fb46d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb21f50 0x55ae2fb46d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 -- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 msgr2=0x55ae2fb71f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb71f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 -- 172.21.15.94:0/2296757526 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae2fa43270 con 0x55ae2fb22330 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb22330 0x55ae2fb45b60 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f6d10004840 tx=0x7f6d10007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fcee3edd077578 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6cf9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/2296757526 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1001a070 con 0x55ae2fb22330 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/2296757526 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f6d1001bac0 con 0x55ae2fb22330 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.066+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d14082050 con 0x55ae2fb22330 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/2296757526 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d10026050 con 0x55ae2fb22330 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 10 client.10242 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x55ae2f79b2d0 con 0x55ae2fb22330 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/2296757526 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d14082050 con 0x55ae2fb22330 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/2296757526 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f6d10029020 con 0x55ae2fb22330 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 10 client.10242 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 10 client.10242 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 20 client.10242 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fb45230 0x55ae2fb7d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 4 client.10242 mds_command: new command op to 9888 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:10.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 --> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae2fa93bf0 con 0x55ae2fb45230 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fb45230 0x55ae2fb7d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6cf9ffb640 10 client.10242 ms_handle_connect on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.067+0000 7f6d1affd640 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fb45230 0x55ae2fb7d440 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4cd626f44bd773f1 server_cookie=a24167f75f1e2605 in_seq=0 out_seq=0 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.068+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/2296757526 <== mds.1 v2:172.21.15.94:6833/2744471722 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ae2fa93bf0 con 0x55ae2fb45230 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.068+0000 7f6cf9ffb640 10 client.10242 handle_command_reply: tid=0 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.068+0000 7f6d23fcc740 1 client.10242 shutdown 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.068+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb22330 msgr2=0x55ae2fb45b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.068+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb22330 0x55ae2fb45b60 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f6d10004840 tx=0x7f6d10007430 comp rx=0 tx=0).stop 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.068+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 shutdown_connections 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb71f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fb45230 0x55ae2fb7d440 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb22330 0x55ae2fb45b60 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/2296757526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb21f50 0x55ae2fb46d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 >> 172.21.15.94:0/2296757526 conn(0x55ae2fb204b0 msgr2=0x55ae2fb208b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6cf9ffb640 0 client.10242 ms_handle_reset on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:10.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 shutdown_connections 2024-03-06T07:26:10.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 1 -- 172.21.15.94:0/2296757526 wait complete. 2024-03-06T07:26:10.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.069+0000 7f6d23fcc740 20 client.10242 trim_cache size 0 max 16384 2024-03-06T07:26:10.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.093+0000 7f6d23fcc740 1 Processor -- start 2024-03-06T07:26:10.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.093+0000 7f6d23fcc740 1 -- start start 2024-03-06T07:26:10.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb21f50 0x55ae2fb5fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb6ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb6bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c1f88e0 con 0x55ae2fb21f50 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c1eef50 con 0x55ae2fb460a0 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c1f7390 con 0x55ae2fb22330 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d1affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb21f50 0x55ae2fb5fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d1affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb21f50 0x55ae2fb5fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34598/0 (socket says 172.21.15.94:34598) 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.094+0000 7f6d1affd640 1 -- 172.21.15.94:0/1843878630 learned_addr learned my addr 172.21.15.94:0/1843878630 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb6ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb6bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1affd640 1 -- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 msgr2=0x55ae2fb6ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1affd640 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb6ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1affd640 1 -- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 msgr2=0x55ae2fb6bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1affd640 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb6bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1affd640 1 -- 172.21.15.94:0/1843878630 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae2f8656b0 con 0x55ae2fb21f50 2024-03-06T07:26:10.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb6ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb6bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6d1affd640 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb21f50 0x55ae2fb5fb40 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c004140 tx=0x7f6d0c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4712e2123525deb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6cfbfff640 1 -- 172.21.15.94:0/1843878630 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c032070 con 0x55ae2fb21f50 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6cfbfff640 1 -- 172.21.15.94:0/1843878630 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c003bd0 con 0x55ae2fb21f50 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.095+0000 7f6cfbfff640 1 -- 172.21.15.94:0/1843878630 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c02d520 con 0x55ae2fb21f50 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 -- 172.21.15.94:0/1843878630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb21f50 msgr2=0x55ae2fb5fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb21f50 0x55ae2fb5fb40 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c004140 tx=0x7f6d0c017000 comp rx=0 tx=0).stop 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 -- 172.21.15.94:0/1843878630 shutdown_connections 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb460a0 0x55ae2fb6bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb6ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/1843878630 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb21f50 0x55ae2fb5fb40 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 -- 172.21.15.94:0/1843878630 >> 172.21.15.94:0/1843878630 conn(0x55ae2fb204b0 msgr2=0x55ae2fb1db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 -- 172.21.15.94:0/1843878630 shutdown_connections 2024-03-06T07:26:10.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.096+0000 7f6d23fcc740 1 -- 172.21.15.94:0/1843878630 wait complete. 2024-03-06T07:26:10.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.097+0000 7f6d23fcc740 1 -- start start 2024-03-06T07:26:10.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.097+0000 7f6d23fcc740 1 Processor -- start 2024-03-06T07:26:10.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb78740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb229d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d23fcc740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb460a0 0x55ae2fbcdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fb6f310 con 0x55ae2fb460a0 2024-03-06T07:26:10.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fb6ed50 con 0x55ae2fb21f50 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d23fcc740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ae2fb72a30 con 0x55ae2fb22330 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d23fcc740 10 client.0 _ll_register_callbacks cb 0x7f6d10004ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb460a0 0x55ae2fbcdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb460a0 0x55ae2fbcdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34610/0 (socket says 172.21.15.94:34610) 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/43914337 learned_addr learned my addr 172.21.15.94:0/43914337 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb229d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1affd640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb78740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 msgr2=0x55ae2fb229d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb229d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 msgr2=0x55ae2fb78740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb78740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 -- 172.21.15.94:0/43914337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae2fbcb820 con 0x55ae2fb460a0 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1a7fc640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb229d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6d1b7fe640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb460a0 0x55ae2fbcdd90 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f6d14079840 tx=0x7f6d14000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439d7c31a21486c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.098+0000 7f6cf9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/43914337 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1407da00 con 0x55ae2fb460a0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/43914337 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f6d14003b20 con 0x55ae2fb460a0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d0c003bd0 con 0x55ae2fb460a0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/43914337 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d140ab0c0 con 0x55ae2fb460a0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 10 client.10248 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x55ae2fb1c790 con 0x55ae2fb460a0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/43914337 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c003bd0 con 0x55ae2fb460a0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/43914337 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f6d14081bc0 con 0x55ae2fb460a0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 10 client.10248 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 10 client.10248 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 20 client.10248 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fbce2d0 0x55ae2fb61600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 4 client.10248 mds_command: new command op to 9888 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 --> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ae2fa8df40 con 0x55ae2fbce2d0 2024-03-06T07:26:10.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d1affd640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fbce2d0 0x55ae2fb61600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6cf9ffb640 10 client.10248 ms_handle_connect on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.099+0000 7f6d1affd640 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fbce2d0 0x55ae2fb61600 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12e4bd620308395c server_cookie=2a54aa5c8fa742f5 in_seq=0 out_seq=0 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6cf9ffb640 1 -- 172.21.15.94:0/43914337 <== mds.1 v2:172.21.15.94:6833/2744471722 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55ae2fa8df40 con 0x55ae2fbce2d0 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6cf9ffb640 10 client.10248 handle_command_reply: tid=0 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 client.10248 shutdown 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb460a0 msgr2=0x55ae2fbcdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb460a0 0x55ae2fbcdd90 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f6d14079840 tx=0x7f6d14000f40 comp rx=0 tx=0).stop 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 shutdown_connections 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:6833/2744471722,v1:172.21.15.94:6835/2744471722] conn(0x55ae2fbce2d0 0x55ae2fb61600 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55ae2fb460a0 0x55ae2fbcdd90 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55ae2fb22330 0x55ae2fb229d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6cf9ffb640 0 client.10248 ms_handle_reset on v2:172.21.15.94:6833/2744471722 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 --2- 172.21.15.94:0/43914337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55ae2fb21f50 0x55ae2fb78740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 >> 172.21.15.94:0/43914337 conn(0x55ae2fb204b0 msgr2=0x55ae2fb5fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 shutdown_connections 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 1 -- 172.21.15.94:0/43914337 wait complete. 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d23fcc740 20 client.10248 trim_cache size 0 max 16384 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 9888, 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 2.8153381899999999, 2024-03-06T07:26:10.107 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 180, 2024-03-06T07:26:10.108 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 184, 2024-03-06T07:26:10.108 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 184, 2024-03-06T07:26:10.108 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 36.850179885999999 2024-03-06T07:26:10.108 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d0806b4d0 msgr2=0x7f6d0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d213eb640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d0806b4d0 0x7f6d0806d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009b00 tx=0x7f6d0c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.100+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d1c1ebae0 msgr2=0x7f6d1c0c6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d1c1ebae0 0x7f6d1c0c6b20 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f6d1402c040 tx=0x7f6d140014a0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 shutdown_connections 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d0806b4d0 0x7f6d0806d990 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d1c1ebae0 0x7f6d1c0c6b20 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d1c1432e0 0x7f6d1c1eb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 --2- 172.21.15.94:0/2198992435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d1c153c90 0x7f6d1c153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 >> 172.21.15.94:0/2198992435 conn(0x7f6d1c0c7de0 msgr2=0x7f6d1c14be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 shutdown_connections 2024-03-06T07:26:10.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.101+0000 7f6d213eb640 1 -- 172.21.15.94:0/2198992435 wait complete. 2024-03-06T07:26:10.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.213+0000 7f9ba3313640 1 Processor -- start 2024-03-06T07:26:10.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.214+0000 7f9ba3313640 1 -- start start 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.214+0000 7f9ba3313640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c155510 0x7f9b9c155980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.214+0000 7f9ba3313640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c152030 con 0x7f9b9c0c4bb0 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.214+0000 7f9ba3313640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c152db0 con 0x7f9b9c0c30b0 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.214+0000 7f9ba3313640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c152920 con 0x7f9b9c155510 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.214+0000 7f9ba10af640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9b9c0c4bb0 0x7f9b9c0c3780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39244/0 (socket says 172.21.15.94:39244) 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.214+0000 7f9ba10af640 1 -- 172.21.15.94:0/180319712 learned_addr learned my addr 172.21.15.94:0/180319712 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/180319712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c155510 0x7f9b9c155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9b93fff640 1 -- 172.21.15.94:0/180319712 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351797268 0 0) 0x7f9b9c152030 con 0x7f9b9c0c4bb0 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9b93fff640 1 -- 172.21.15.94:0/180319712 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b88003520 con 0x7f9b9c0c4bb0 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9b93fff640 1 -- 172.21.15.94:0/180319712 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094256582 0 0) 0x7f9b9c152db0 con 0x7f9b9c0c30b0 2024-03-06T07:26:10.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9b93fff640 1 -- 172.21.15.94:0/180319712 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b9c152030 con 0x7f9b9c0c30b0 2024-03-06T07:26:10.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9ba18b0640 1 -- 172.21.15.94:0/180319712 >> v1:172.21.15.145:6789/0 conn(0x7f9b9c0c30b0 legacy=0x7f9b9c14f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9ba18b0640 1 -- 172.21.15.94:0/180319712 >> v1:172.21.15.94:6789/0 conn(0x7f9b9c0c4bb0 legacy=0x7f9b9c0c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.215+0000 7f9ba18b0640 1 -- 172.21.15.94:0/180319712 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c1522d0 con 0x7f9b9c155510 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.216+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/180319712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c155510 0x7f9b9c155980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009dc0 tx=0x7f9b9402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ca78834032160ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.216+0000 7f9b93fff640 1 -- 172.21.15.94:0/180319712 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94039070 con 0x7f9b9c155510 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.216+0000 7f9b93fff640 1 -- 172.21.15.94:0/180319712 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9402cc40 con 0x7f9b9c155510 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.216+0000 7f9b93fff640 1 -- 172.21.15.94:0/180319712 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94034440 con 0x7f9b9c155510 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.216+0000 7f9ba3313640 1 -- 172.21.15.94:0/180319712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c155510 msgr2=0x7f9b9c155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.216+0000 7f9ba3313640 1 --2- 172.21.15.94:0/180319712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c155510 0x7f9b9c155980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009dc0 tx=0x7f9b9402be90 comp rx=0 tx=0).stop 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.217+0000 7f9ba3313640 1 -- 172.21.15.94:0/180319712 shutdown_connections 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.217+0000 7f9ba3313640 1 --2- 172.21.15.94:0/180319712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c155510 0x7f9b9c155980 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.217+0000 7f9ba3313640 1 -- 172.21.15.94:0/180319712 >> 172.21.15.94:0/180319712 conn(0x7f9b9c0c7de0 msgr2=0x7f9b9c0c4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.217+0000 7f9ba3313640 1 -- 172.21.15.94:0/180319712 shutdown_connections 2024-03-06T07:26:10.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.217+0000 7f9ba3313640 1 -- 172.21.15.94:0/180319712 wait complete. 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 Processor -- start 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 -- start start 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c0c30b0 0x7f9b9c14db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c0c4bb0 0x7f9b9c1e84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b9c155510 0x7f9b9c143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c155ec0 con 0x7f9b9c155510 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0c3fd0 con 0x7f9b9c0c30b0 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba3313640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c14f830 con 0x7f9b9c0c4bb0 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba18b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b9c155510 0x7f9b9c143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba18b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b9c155510 0x7f9b9c143a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34618/0 (socket says 172.21.15.94:34618) 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.218+0000 7f9ba18b0640 1 -- 172.21.15.94:0/2707990649 learned_addr learned my addr 172.21.15.94:0/2707990649 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c0c4bb0 0x7f9b9c1e84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba10af640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c0c30b0 0x7f9b9c14db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba18b0640 1 -- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c0c4bb0 msgr2=0x7f9b9c1e84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c0c4bb0 0x7f9b9c1e84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba18b0640 1 -- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c0c30b0 msgr2=0x7f9b9c14db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c0c30b0 0x7f9b9c14db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba18b0640 1 -- 172.21.15.94:0/2707990649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c14d6a0 con 0x7f9b9c155510 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c0c4bb0 0x7f9b9c1e84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b9c155510 0x7f9b9c143a80 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f9b9402c4a0 tx=0x7f9b94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9b91ffb640 1 -- 172.21.15.94:0/2707990649 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94009450 con 0x7f9b9c155510 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9b91ffb640 1 -- 172.21.15.94:0/2707990649 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94003e10 con 0x7f9b9c155510 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c14d3c0 con 0x7f9b9c155510 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9b91ffb640 1 -- 172.21.15.94:0/2707990649 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94004860 con 0x7f9b9c155510 2024-03-06T07:26:10.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.219+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c14cf30 con 0x7f9b9c155510 2024-03-06T07:26:10.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.221+0000 7f9b91ffb640 1 -- 172.21.15.94:0/2707990649 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b9404e440 con 0x7f9b9c155510 2024-03-06T07:26:10.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.222+0000 7f9b91ffb640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b8006b4d0 0x7f9b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.222+0000 7f9b91ffb640 1 -- 172.21.15.94:0/2707990649 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f9b940aec40 con 0x7f9b9c155510 2024-03-06T07:26:10.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.222+0000 7f9ba10af640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b8006b4d0 0x7f9b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.222+0000 7f9ba10af640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b8006b4d0 0x7f9b8006d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f9b98000f60 tx=0x7f9b98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.222+0000 7f9ba5fdb740 1 Processor -- start 2024-03-06T07:26:10.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba5fdb740 1 -- start start 2024-03-06T07:26:10.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf642010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf642550 0x5562cf68bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf67c420 0x5562cf63c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cf5a66b0 con 0x5562cf68b800 2024-03-06T07:26:10.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cf54fa60 con 0x5562cf67c420 2024-03-06T07:26:10.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562cf5a2750 con 0x5562cf642550 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba10af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf642010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba10af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf642010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34634/0 (socket says 172.21.15.94:34634) 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba10af640 1 -- 172.21.15.94:0/3892815731 learned_addr learned my addr 172.21.15.94:0/3892815731 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf642550 0x5562cf68bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.223+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf67c420 0x5562cf63c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba10af640 1 -- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf642550 msgr2=0x5562cf68bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf642550 0x5562cf68bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba10af640 1 -- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf67c420 msgr2=0x5562cf63c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf67c420 0x5562cf63c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba10af640 1 -- 172.21.15.94:0/3892815731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562cf56df30 con 0x5562cf68b800 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf67c420 0x5562cf63c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf642010 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f9b980058d0 tx=0x7f9b98011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cffb49fdb83596 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9b86ffd640 1 -- 172.21.15.94:0/3892815731 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98007710 con 0x5562cf68b800 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9b86ffd640 1 -- 172.21.15.94:0/3892815731 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9800e7b0 con 0x5562cf68b800 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9b86ffd640 1 -- 172.21.15.94:0/3892815731 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9800eb60 con 0x5562cf68b800 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3892815731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 msgr2=0x5562cf642010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.224+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf642010 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f9b980058d0 tx=0x7f9b98011000 comp rx=0 tx=0).stop 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3892815731 shutdown_connections 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf67c420 0x5562cf63c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf642550 0x5562cf68bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3892815731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf642010 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3892815731 >> 172.21.15.94:0/3892815731 conn(0x5562cf63e7d0 msgr2=0x5562cf6633c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3892815731 shutdown_connections 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3892815731 wait complete. 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 -- start start 2024-03-06T07:26:10.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.225+0000 7f9ba5fdb740 1 Processor -- start 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf640c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf696550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf696a90 0x5562cf660170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cf680590 con 0x5562cf696a90 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cf562ea0 con 0x5562cf68b800 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562cf663db0 con 0x5562cf67c420 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba5fdb740 10 client.0 _ll_register_callbacks cb 0x5562cf562870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba18b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf696a90 0x5562cf660170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba18b0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf696a90 0x5562cf660170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34640/0 (socket says 172.21.15.94:34640) 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba18b0640 1 -- 172.21.15.94:0/3486023574 learned_addr learned my addr 172.21.15.94:0/3486023574 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf640c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.226+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf696550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba18b0640 1 -- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 msgr2=0x5562cf640c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf640c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba18b0640 1 -- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 msgr2=0x5562cf696550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf696550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba18b0640 1 -- 172.21.15.94:0/3486023574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562cf563270 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf696a90 0x5562cf660170 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f9b940acfa0 tx=0x7f9b9407d290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea856d6e1194ae7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9b7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3486023574 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94039070 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3486023574 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f9b94079840 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b9800e7b0 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3486023574 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b94043380 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba5fdb740 10 client.10260 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x5562cf2bb2d0 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3486023574 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9800e7b0 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3486023574 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f9b9407d430 con 0x5562cf696a90 2024-03-06T07:26:10.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba5fdb740 10 client.10260 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba5fdb740 10 client.10260 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.227+0000 7f9ba5fdb740 20 client.10260 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.228+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf69f180 0x5562cf6a1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.228+0000 7f9ba5fdb740 4 client.10260 mds_command: new command op to 9891 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.228+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 --> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562cf5b3bf0 con 0x5562cf69f180 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.228+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf69f180 0x5562cf6a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.228+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf69f180 0x5562cf6a1570 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1df6f7f08a445f0e server_cookie=f5cc65c7252e0ec9 in_seq=0 out_seq=0 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.228+0000 7f9b7bfff640 10 client.10260 ms_handle_connect on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3486023574 <== mds.0 v2:172.21.15.94:6832/4225816089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5562cf5b3bf0 con 0x5562cf69f180 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9b7bfff640 10 client.10260 handle_command_reply: tid=0 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 client.10260 shutdown 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf696a90 msgr2=0x5562cf660170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf696a90 0x5562cf660170 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f9b940acfa0 tx=0x7f9b9407d290 comp rx=0 tx=0).stop 2024-03-06T07:26:10.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 shutdown_connections 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf69f180 0x5562cf6a1570 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf696a90 0x5562cf660170 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9b7bfff640 0 client.10260 ms_handle_reset on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf696550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3486023574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf640c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 >> 172.21.15.94:0/3486023574 conn(0x5562cf642550 msgr2=0x5562cf6880a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 shutdown_connections 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3486023574 wait complete. 2024-03-06T07:26:10.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.229+0000 7f9ba5fdb740 20 client.10260 trim_cache size 0 max 16384 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.246+0000 7f9ba5fdb740 1 Processor -- start 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.246+0000 7f9ba5fdb740 1 -- start start 2024-03-06T07:26:10.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf63b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf65d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf696a90 0x5562cf65d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c1f39d0 con 0x5562cf68b800 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c1ebc10 con 0x5562cf696a90 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c06afc0 con 0x5562cf67c420 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf65d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf65d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34650/0 (socket says 172.21.15.94:34650) 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 -- 172.21.15.94:0/1320563583 learned_addr learned my addr 172.21.15.94:0/1320563583 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba10af640 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf63b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf696a90 0x5562cf65d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 -- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 msgr2=0x5562cf63b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf63b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 -- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf696a90 msgr2=0x5562cf65d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf696a90 0x5562cf65d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 -- 172.21.15.94:0/1320563583 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562cf3856b0 con 0x5562cf68b800 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf696a90 0x5562cf65d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf65d160 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c1ec3c0 tx=0x7f9b8c001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59864a6318e38c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9b867fc640 1 -- 172.21.15.94:0/1320563583 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c01a070 con 0x5562cf68b800 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9b867fc640 1 -- 172.21.15.94:0/1320563583 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c001d30 con 0x5562cf68b800 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.247+0000 7f9b867fc640 1 -- 172.21.15.94:0/1320563583 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c014870 con 0x5562cf68b800 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/1320563583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 msgr2=0x5562cf65d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf65d160 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c1ec3c0 tx=0x7f9b8c001a20 comp rx=0 tx=0).stop 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/1320563583 shutdown_connections 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf696a90 0x5562cf65d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf68b800 0x5562cf65d160 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/1320563583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf67c420 0x5562cf63b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/1320563583 >> 172.21.15.94:0/1320563583 conn(0x5562cf642550 msgr2=0x5562cf65ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/1320563583 shutdown_connections 2024-03-06T07:26:10.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.248+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/1320563583 wait complete. 2024-03-06T07:26:10.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.249+0000 7f9ba5fdb740 1 -- start start 2024-03-06T07:26:10.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.249+0000 7f9ba5fdb740 1 Processor -- start 2024-03-06T07:26:10.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf67c420 0x5562cf63b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf6a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba5fdb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf696a90 0x5562cf6830b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cf67cb40 con 0x5562cf67c420 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cf685bd0 con 0x5562cf68b800 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba5fdb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562cf69d4a0 con 0x5562cf696a90 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba5fdb740 10 client.0 _ll_register_callbacks cb 0x7f9b9407d8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba10af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf67c420 0x5562cf63b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba10af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf67c420 0x5562cf63b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34656/0 (socket says 172.21.15.94:34656) 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba10af640 1 -- 172.21.15.94:0/3466797662 learned_addr learned my addr 172.21.15.94:0/3466797662 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf696a90 0x5562cf6830b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.250+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf6a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba10af640 1 -- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf696a90 msgr2=0x5562cf6830b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf696a90 0x5562cf6830b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba10af640 1 -- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 msgr2=0x5562cf6a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf6a5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba10af640 1 -- 172.21.15.94:0/3466797662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562cf674ad0 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba18b0640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf696a90 0x5562cf6830b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba10af640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf67c420 0x5562cf63b910 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f9b98014000 tx=0x7f9b98000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84dfcac2edb2033 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9b7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3466797662 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b98033060 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3466797662 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f9b980047e0 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b8c001d30 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3466797662 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=180) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b98039050 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba5fdb740 10 client.10266 fetch_fsmap learned FSMap version 180 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=180,monmap=2+}) v3 -- 0x5562cf67ab80 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3466797662 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c001d30 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3466797662 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 180) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f9b9803c020 con 0x5562cf67c420 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba5fdb740 10 client.10266 fetch_fsmap finished waiting for FSMap version 180 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba5fdb740 10 client.10266 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba5fdb740 20 client.10266 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.251+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf683c80 0x5562cf6ef940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf683c80 0x5562cf6ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 4 client.10266 mds_command: new command op to 9891 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 --> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5562cf5adf40 con 0x5562cf683c80 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba08ae640 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf683c80 0x5562cf6ef940 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ea91a4a85566c32 server_cookie=ffa970099b535946 in_seq=0 out_seq=0 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9b7bfff640 10 client.10266 ms_handle_connect on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9b7bfff640 1 -- 172.21.15.94:0/3466797662 <== mds.0 v2:172.21.15.94:6832/4225816089 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5562cf5adf40 con 0x5562cf683c80 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9b7bfff640 10 client.10266 handle_command_reply: tid=0 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 client.10266 shutdown 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf67c420 msgr2=0x5562cf63b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf67c420 0x5562cf63b910 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f9b98014000 tx=0x7f9b98000ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 shutdown_connections 2024-03-06T07:26:10.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:6832/4225816089,v1:172.21.15.94:6834/4225816089] conn(0x5562cf683c80 0x5562cf6ef940 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9b7bfff640 0 client.10266 ms_handle_reset on v2:172.21.15.94:6832/4225816089 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562cf696a90 0x5562cf6830b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562cf68b800 0x5562cf6a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 --2- 172.21.15.94:0/3466797662 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5562cf67c420 0x5562cf63b910 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 >> 172.21.15.94:0/3466797662 conn(0x5562cf642550 msgr2=0x5562cf6635d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 shutdown_connections 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 1 -- 172.21.15.94:0/3466797662 wait complete. 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.252+0000 7f9ba5fdb740 20 client.10266 trim_cache size 0 max 16384 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:26:10.260 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 9891, 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 3.9823050289999999, 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 180, 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 184, 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 184, 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 37.002720908999997 2024-03-06T07:26:10.261 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:26:10.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b8006b4d0 msgr2=0x7f9b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b8006b4d0 0x7f9b8006d990 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f9b98000f60 tx=0x7f9b98009040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b9c155510 msgr2=0x7f9b9c143a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b9c155510 0x7f9b9c143a80 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f9b9402c4a0 tx=0x7f9b94004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 shutdown_connections 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b8006b4d0 0x7f9b8006d990 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b9c155510 0x7f9b9c143a80 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c0c4bb0 0x7f9b9c1e84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 --2- 172.21.15.94:0/2707990649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c0c30b0 0x7f9b9c14db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 >> 172.21.15.94:0/2707990649 conn(0x7f9b9c0c7de0 msgr2=0x7f9b9c072570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 shutdown_connections 2024-03-06T07:26:10.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.253+0000 7f9ba3313640 1 -- 172.21.15.94:0/2707990649 wait complete. 2024-03-06T07:26:10.266 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:10.266 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:10.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:10.372 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:10.373 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:26:10.373 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:10.389 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:26:10.390 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:26:10.390 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:10.405 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:26:10.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:10.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.530+0000 7f521f08f640 1 Processor -- start 2024-03-06T07:26:10.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.530+0000 7f521f08f640 1 -- start start 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.531+0000 7f521f08f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180c8a00 0x7f5218158f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.531+0000 7f521f08f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5218159460 0x7f52180c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.531+0000 7f521f08f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52180c7760 con 0x7f5218152920 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521f08f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52181587b0 con 0x7f5218159460 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521f08f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180c7a40 con 0x7f52180c8a00 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521d62c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5218159460 0x7f52180c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521ce2b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5218152920 0x7f5218155c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39248/0 (socket says 172.21.15.94:39248) 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521ce2b640 1 -- 172.21.15.94:0/1864544744 learned_addr learned my addr 172.21.15.94:0/1864544744 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f5217fff640 1 --2- 172.21.15.94:0/1864544744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180c8a00 0x7f5218158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521d62c640 1 -- 172.21.15.94:0/1864544744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180c8a00 msgr2=0x7f5218158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521d62c640 1 --2- 172.21.15.94:0/1864544744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180c8a00 0x7f5218158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521d62c640 1 -- 172.21.15.94:0/1864544744 >> v1:172.21.15.94:6789/0 conn(0x7f5218152920 legacy=0x7f5218155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f521d62c640 1 -- 172.21.15.94:0/1864544744 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180c7d20 con 0x7f5218159460 2024-03-06T07:26:10.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.532+0000 7f52177fe640 1 -- 172.21.15.94:0/1864544744 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377292349 0 0) 0x7f52180c7760 con 0x7f5218152920 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f521d62c640 1 --2- 172.21.15.94:0/1864544744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5218159460 0x7f52180c5330 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f52000097a0 tx=0x7f520002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec728708f4386846 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f52177fe640 1 -- 172.21.15.94:0/1864544744 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52000315a0 con 0x7f5218159460 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f52177fe640 1 -- 172.21.15.94:0/1864544744 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5200037070 con 0x7f5218159460 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f52177fe640 1 -- 172.21.15.94:0/1864544744 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200031960 con 0x7f5218159460 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f521f08f640 1 -- 172.21.15.94:0/1864544744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5218159460 msgr2=0x7f52180c5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f521f08f640 1 --2- 172.21.15.94:0/1864544744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5218159460 0x7f52180c5330 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f52000097a0 tx=0x7f520002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f521f08f640 1 -- 172.21.15.94:0/1864544744 shutdown_connections 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f521f08f640 1 --2- 172.21.15.94:0/1864544744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5218159460 0x7f52180c5330 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f521f08f640 1 --2- 172.21.15.94:0/1864544744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180c8a00 0x7f5218158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.533+0000 7f521f08f640 1 -- 172.21.15.94:0/1864544744 >> 172.21.15.94:0/1864544744 conn(0x7f52180ac560 msgr2=0x7f52180ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.534+0000 7f521f08f640 1 -- 172.21.15.94:0/1864544744 shutdown_connections 2024-03-06T07:26:10.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.534+0000 7f521f08f640 1 -- 172.21.15.94:0/1864544744 wait complete. 2024-03-06T07:26:10.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 Processor -- start 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 -- start start 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52180c8a00 0x7f52180b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5218152920 0x7f52181ea660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180ad650 0x7f52181eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52180c7390 con 0x7f52180c8a00 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521806d030 con 0x7f5218152920 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.535+0000 7f521f08f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180c7510 con 0x7f52180ad650 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52180c8a00 0x7f52180b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52180c8a00 0x7f52180b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34668/0 (socket says 172.21.15.94:34668) 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 -- 172.21.15.94:0/3276985106 learned_addr learned my addr 172.21.15.94:0/3276985106 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521d62c640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180ad650 0x7f52181eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f5217fff640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5218152920 0x7f52181ea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 -- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180ad650 msgr2=0x7f52181eaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180ad650 0x7f52181eaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 -- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5218152920 msgr2=0x7f52181ea660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5218152920 0x7f52181ea660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 -- 172.21.15.94:0/3276985106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180b0920 con 0x7f52180c8a00 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521d62c640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180ad650 0x7f52181eaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521ce2b640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52180c8a00 0x7f52180b03e0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f521000c890 tx=0x7f521000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f52157fa640 1 -- 172.21.15.94:0/3276985106 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210018070 con 0x7f52180c8a00 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f52157fa640 1 -- 172.21.15.94:0/3276985106 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521000a5e0 con 0x7f52180c8a00 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52180c2c80 con 0x7f52180c8a00 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f52157fa640 1 -- 172.21.15.94:0/3276985106 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521000a9d0 con 0x7f52180c8a00 2024-03-06T07:26:10.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.536+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52180c23e0 con 0x7f52180c8a00 2024-03-06T07:26:10.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.538+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521804f330 con 0x7f52180c8a00 2024-03-06T07:26:10.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.538+0000 7f52157fa640 1 -- 172.21.15.94:0/3276985106 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f521000ab70 con 0x7f52180c8a00 2024-03-06T07:26:10.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.539+0000 7f52157fa640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51f806b4d0 0x7f51f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.539+0000 7f52157fa640 1 -- 172.21.15.94:0/3276985106 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f5210089a10 con 0x7f52180c8a00 2024-03-06T07:26:10.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.539+0000 7f5217fff640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51f806b4d0 0x7f51f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.540+0000 7f5217fff640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51f806b4d0 0x7f51f806d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f52180c6ea0 tx=0x7f5208008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.544+0000 7f52157fa640 1 -- 172.21.15.94:0/3276985106 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52100586a0 con 0x7f52180c8a00 2024-03-06T07:26:10.921 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.919+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f521806cda0 con 0x7f52180c8a00 2024-03-06T07:26:10.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.920+0000 7f52157fa640 1 -- 172.21.15.94:0/3276985106 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5014 (secure 0 0 0) 0x7f5210057fa0 con 0x7f52180c8a00 2024-03-06T07:26:10.922 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:10.922 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":180,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:08.285097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9891,"mds_1":9888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9888":{"gid":9888,"name":"c","rank":1,"incarnation":179,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9891":{"gid":9891,"name":"a","rank":0,"incarnation":177,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9891,"qdb_cluster":[9888,9891]},"id":14}]} 2024-03-06T07:26:10.922 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 180 2024-03-06T07:26:10.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51f806b4d0 msgr2=0x7f51f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51f806b4d0 0x7f51f806d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f52180c6ea0 tx=0x7f5208008040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52180c8a00 msgr2=0x7f52180b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52180c8a00 0x7f52180b03e0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f521000c890 tx=0x7f521000cd50 comp rx=0 tx=0).stop 2024-03-06T07:26:10.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 shutdown_connections 2024-03-06T07:26:10.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f51f806b4d0 0x7f51f806d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52180ad650 0x7f52181eaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5218152920 0x7f52181ea660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 --2- 172.21.15.94:0/3276985106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52180c8a00 0x7f52180b03e0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 >> 172.21.15.94:0/3276985106 conn(0x7f52180ac560 msgr2=0x7f5218151b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 shutdown_connections 2024-03-06T07:26:10.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:10.922+0000 7f521f08f640 1 -- 172.21.15.94:0/3276985106 wait complete. 2024-03-06T07:26:10.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:11.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.061+0000 7fc8f44d4640 1 Processor -- start 2024-03-06T07:26:11.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.061+0000 7fc8f44d4640 1 -- start start 2024-03-06T07:26:11.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.062+0000 7fc8f44d4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc8ec0c88c0 0x7fc8ec155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.062+0000 7fc8f44d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec158b60 0x7fc8ec158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.062+0000 7fc8f44d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec0c73d0 con 0x7fc8ec14f360 2024-03-06T07:26:11.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.062+0000 7fc8f44d4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c8dd0 con 0x7fc8ec0c88c0 2024-03-06T07:26:11.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.062+0000 7fc8f44d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0be2e0 con 0x7fc8ec158b60 2024-03-06T07:26:11.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f2a71640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec158b60 0x7fc8ec158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f1a6f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc8ec0c88c0 0x7fc8ec155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f2a71640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec158b60 0x7fc8ec158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:46766/0 (socket says 172.21.15.94:46766) 2024-03-06T07:26:11.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f2a71640 1 -- 172.21.15.94:0/3734555031 learned_addr learned my addr 172.21.15.94:0/3734555031 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f1a6f640 1 -- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec158b60 msgr2=0x7fc8ec158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f1a6f640 1 --2- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec158b60 0x7fc8ec158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f1a6f640 1 -- 172.21.15.94:0/3734555031 >> v1:172.21.15.145:6789/0 conn(0x7fc8ec14f360 legacy=0x7fc8ec0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f1a6f640 1 -- 172.21.15.94:0/3734555031 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec0c7670 con 0x7fc8ec0c88c0 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.063+0000 7fc8f2a71640 1 --2- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec158b60 0x7fc8ec158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.064+0000 7fc8f1a6f640 1 --2- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 0x7fc8ec155c10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0097a0 tx=0x7fc8dc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26e03b13fb0cfaf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.064+0000 7fc8f126e640 1 -- 172.21.15.94:0/3734555031 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc0315a0 con 0x7fc8ec0c88c0 2024-03-06T07:26:11.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.064+0000 7fc8f126e640 1 -- 172.21.15.94:0/3734555031 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc037070 con 0x7fc8ec0c88c0 2024-03-06T07:26:11.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.064+0000 7fc8f126e640 1 -- 172.21.15.94:0/3734555031 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc031960 con 0x7fc8ec0c88c0 2024-03-06T07:26:11.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 -- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 msgr2=0x7fc8ec155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 0x7fc8ec155c10 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0097a0 tx=0x7fc8dc02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:11.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 -- 172.21.15.94:0/3734555031 shutdown_connections 2024-03-06T07:26:11.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec158b60 0x7fc8ec158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/3734555031 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 0x7fc8ec155c10 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 -- 172.21.15.94:0/3734555031 >> 172.21.15.94:0/3734555031 conn(0x7fc8ec0ac570 msgr2=0x7fc8ec0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 -- 172.21.15.94:0/3734555031 shutdown_connections 2024-03-06T07:26:11.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.065+0000 7fc8f44d4640 1 -- 172.21.15.94:0/3734555031 wait complete. 2024-03-06T07:26:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.067+0000 7fc8f44d4640 1 Processor -- start 2024-03-06T07:26:11.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.067+0000 7fc8f44d4640 1 -- start start 2024-03-06T07:26:11.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f44d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 0x7fc8ec0c0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f44d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec14f360 0x7fc8ec0aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f44d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ec158b60 0x7fc8ec0af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f44d4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c5a50 con 0x7fc8ec158b60 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f44d4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c7c30 con 0x7fc8ec0c88c0 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f44d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c5bd0 con 0x7fc8ec14f360 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ec158b60 0x7fc8ec0af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ec158b60 0x7fc8ec0af370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34676/0 (socket says 172.21.15.94:34676) 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 -- 172.21.15.94:0/2929494437 learned_addr learned my addr 172.21.15.94:0/2929494437 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f1a6f640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec14f360 0x7fc8ec0aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2270640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 0x7fc8ec0c0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 -- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec14f360 msgr2=0x7fc8ec0aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec14f360 0x7fc8ec0aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 -- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 msgr2=0x7fc8ec0c0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 0x7fc8ec0c0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 -- 172.21.15.94:0/2929494437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec1eb9e0 con 0x7fc8ec158b60 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.068+0000 7fc8f2a71640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ec158b60 0x7fc8ec0af370 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009f10 tx=0x7fc8e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.069+0000 7fc8e2ffd640 1 -- 172.21.15.94:0/2929494437 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e400cc40 con 0x7fc8ec158b60 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.069+0000 7fc8e2ffd640 1 -- 172.21.15.94:0/2929494437 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e400cde0 con 0x7fc8ec158b60 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.069+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec14c5e0 con 0x7fc8ec158b60 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.069+0000 7fc8e2ffd640 1 -- 172.21.15.94:0/2929494437 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4003bb0 con 0x7fc8ec158b60 2024-03-06T07:26:11.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.069+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec1ebc60 con 0x7fc8ec158b60 2024-03-06T07:26:11.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.070+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec04f330 con 0x7fc8ec158b60 2024-03-06T07:26:11.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.070+0000 7fc8e2ffd640 1 -- 172.21.15.94:0/2929494437 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8e401b030 con 0x7fc8ec158b60 2024-03-06T07:26:11.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.070+0000 7fc8e2ffd640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc8d006b4d0 0x7fc8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.071+0000 7fc8e2ffd640 1 -- 172.21.15.94:0/2929494437 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fc8e408a720 con 0x7fc8ec158b60 2024-03-06T07:26:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.071+0000 7fc8f2270640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc8d006b4d0 0x7fc8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.071+0000 7fc8f2270640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc8d006b4d0 0x7fc8d006d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc8e80016e0 tx=0x7fc8e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.075+0000 7fc8e2ffd640 1 -- 172.21.15.94:0/2929494437 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8e4022020 con 0x7fc8ec158b60 2024-03-06T07:26:11.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.258+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8ec0ace70 con 0x7fc8ec158b60 2024-03-06T07:26:11.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.260+0000 7fc8e2ffd640 1 -- 172.21.15.94:0/2929494437 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+17076 (secure 0 0 0) 0x7fc8ec1eb9e0 con 0x7fc8ec158b60 2024-03-06T07:26:11.262 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:11.262 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":184,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:59.237111+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.909545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.667783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.7","mappings":[{"from":6,"to":0}]},{"pgid":"29.d","mappings":[{"from":6,"to":0}]},{"pgid":"29.13","mappings":[{"from":6,"to":0}]},{"pgid":"29.1e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.261+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc8d006b4d0 msgr2=0x7fc8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.261+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc8d006b4d0 0x7fc8d006d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc8e80016e0 tx=0x7fc8e8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.261+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ec158b60 msgr2=0x7fc8ec0af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.261+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ec158b60 0x7fc8ec0af370 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009f10 tx=0x7fc8e400c490 comp rx=0 tx=0).stop 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 shutdown_connections 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc8d006b4d0 0x7fc8d006d990 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc8ec158b60 0x7fc8ec0af370 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8ec14f360 0x7fc8ec0aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 --2- 172.21.15.94:0/2929494437 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8ec0c88c0 0x7fc8ec0c0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 >> 172.21.15.94:0/2929494437 conn(0x7fc8ec0ac570 msgr2=0x7fc8ec154020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 shutdown_connections 2024-03-06T07:26:11.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.262+0000 7fc8f44d4640 1 -- 172.21.15.94:0/2929494437 wait complete. 2024-03-06T07:26:11.274 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:35.909545+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:36.667783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:11.275 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:11.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.401+0000 7fc62b888640 1 Processor -- start 2024-03-06T07:26:11.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.401+0000 7fc62b888640 1 -- start start 2024-03-06T07:26:11.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc62b888640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc6240b0070 0x7fc624155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc62b888640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc62406ca70 0x7fc6240c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc62b888640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6240b0450 con 0x7fc6240c5380 2024-03-06T07:26:11.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc62b888640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc62406c020 con 0x7fc62406ca70 2024-03-06T07:26:11.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc62b888640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc62406cf20 con 0x7fc6240b0070 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc628e23640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc62406ca70 0x7fc6240c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc628e23640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc62406ca70 0x7fc6240c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34682/0 (socket says 172.21.15.94:34682) 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc628e23640 1 -- 172.21.15.94:0/2502114063 learned_addr learned my addr 172.21.15.94:0/2502114063 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc629624640 1 --2- 172.21.15.94:0/2502114063 >> v2:172.21.15.145:3300/0 conn(0x7fc6240b0070 0x7fc624155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc628e23640 1 -- 172.21.15.94:0/2502114063 >> v2:172.21.15.145:3300/0 conn(0x7fc6240b0070 msgr2=0x7fc624155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc628e23640 1 --2- 172.21.15.94:0/2502114063 >> v2:172.21.15.145:3300/0 conn(0x7fc6240b0070 0x7fc624155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc628e23640 1 -- 172.21.15.94:0/2502114063 >> v1:172.21.15.145:6789/0 conn(0x7fc6240c5380 legacy=0x7fc62414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.402+0000 7fc628e23640 1 -- 172.21.15.94:0/2502114063 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6240c77f0 con 0x7fc62406ca70 2024-03-06T07:26:11.405 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.403+0000 7fc628e23640 1 --2- 172.21.15.94:0/2502114063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62406ca70 0x7fc6240c8900 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc6240aa3d0 tx=0x7fc61402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0859f52272a6a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.403+0000 7fc61bfff640 1 -- 172.21.15.94:0/2502114063 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61402e070 con 0x7fc62406ca70 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.403+0000 7fc61bfff640 1 -- 172.21.15.94:0/2502114063 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc614037070 con 0x7fc62406ca70 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.403+0000 7fc61bfff640 1 -- 172.21.15.94:0/2502114063 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614031a00 con 0x7fc62406ca70 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.403+0000 7fc62b888640 1 -- 172.21.15.94:0/2502114063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62406ca70 msgr2=0x7fc6240c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.403+0000 7fc62b888640 1 --2- 172.21.15.94:0/2502114063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62406ca70 0x7fc6240c8900 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc6240aa3d0 tx=0x7fc61402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.404+0000 7fc62b888640 1 -- 172.21.15.94:0/2502114063 shutdown_connections 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.404+0000 7fc62b888640 1 --2- 172.21.15.94:0/2502114063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc62406ca70 0x7fc6240c8900 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.404+0000 7fc62b888640 1 --2- 172.21.15.94:0/2502114063 >> v2:172.21.15.145:3300/0 conn(0x7fc6240b0070 0x7fc624155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.404+0000 7fc62b888640 1 -- 172.21.15.94:0/2502114063 >> 172.21.15.94:0/2502114063 conn(0x7fc6240ac590 msgr2=0x7fc6240ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.404+0000 7fc62b888640 1 -- 172.21.15.94:0/2502114063 shutdown_connections 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.404+0000 7fc62b888640 1 -- 172.21.15.94:0/2502114063 wait complete. 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 Processor -- start 2024-03-06T07:26:11.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 -- start start 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62406ca70 0x7fc6241541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6240b0070 0x7fc624156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6240c5380 0x7fc62414dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62406c550 con 0x7fc6240b0070 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62406c270 con 0x7fc62406ca70 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.405+0000 7fc62b888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62406da80 con 0x7fc6240c5380 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6240b0070 0x7fc624156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6240b0070 0x7fc624156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34690/0 (socket says 172.21.15.94:34690) 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 -- 172.21.15.94:0/2498305511 learned_addr learned my addr 172.21.15.94:0/2498305511 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc629e25640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6240c5380 0x7fc62414dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc629624640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62406ca70 0x7fc6241541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 -- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6240c5380 msgr2=0x7fc62414dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.410 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6240c5380 0x7fc62414dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 -- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62406ca70 msgr2=0x7fc6241541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62406ca70 0x7fc6241541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 -- 172.21.15.94:0/2498305511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6241ea780 con 0x7fc6240b0070 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc629624640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62406ca70 0x7fc6241541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc628e23640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6240b0070 0x7fc624156e90 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc614000c00 tx=0x7fc6140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc619ffb640 1 -- 172.21.15.94:0/2498305511 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614032880 con 0x7fc6240b0070 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc619ffb640 1 -- 172.21.15.94:0/2498305511 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc614032a40 con 0x7fc6240b0070 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc619ffb640 1 -- 172.21.15.94:0/2498305511 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614032e30 con 0x7fc6240b0070 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.406+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6241eaa20 con 0x7fc6240b0070 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.407+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6241eafe0 con 0x7fc6240b0070 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.407+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62404f330 con 0x7fc6240b0070 2024-03-06T07:26:11.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.408+0000 7fc619ffb640 1 -- 172.21.15.94:0/2498305511 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6140453f0 con 0x7fc6240b0070 2024-03-06T07:26:11.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.408+0000 7fc619ffb640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60806b4d0 0x7fc60806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.409+0000 7fc619ffb640 1 -- 172.21.15.94:0/2498305511 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fc6140ac160 con 0x7fc6240b0070 2024-03-06T07:26:11.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.409+0000 7fc629624640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60806b4d0 0x7fc60806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.410+0000 7fc629624640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60806b4d0 0x7fc60806d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc620006fd0 tx=0x7fc620008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.413+0000 7fc619ffb640 1 -- 172.21.15.94:0/2498305511 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc614040070 con 0x7fc6240b0070 2024-03-06T07:26:11.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.590+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc62406d860 con 0x7fc6240b0070 2024-03-06T07:26:11.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.590+0000 7fc619ffb640 1 -- 172.21.15.94:0/2498305511 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6240b0450 con 0x7fc6240b0070 2024-03-06T07:26:11.592 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:26:11.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60806b4d0 msgr2=0x7fc60806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60806b4d0 0x7fc60806d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc620006fd0 tx=0x7fc620008040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6240b0070 msgr2=0x7fc624156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6240b0070 0x7fc624156e90 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc614000c00 tx=0x7fc6140046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 shutdown_connections 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc60806b4d0 0x7fc60806d990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6240c5380 0x7fc62414dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc6240b0070 0x7fc624156e90 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.592+0000 7fc62b888640 1 --2- 172.21.15.94:0/2498305511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc62406ca70 0x7fc6241541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.593+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 >> 172.21.15.94:0/2498305511 conn(0x7fc6240ac590 msgr2=0x7fc6240ae5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.593+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 shutdown_connections 2024-03-06T07:26:11.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.593+0000 7fc62b888640 1 -- 172.21.15.94:0/2498305511 wait complete. 2024-03-06T07:26:11.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.707+0000 7f281f278640 1 Processor -- start 2024-03-06T07:26:11.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.707+0000 7f281f278640 1 -- start start 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f281f278640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 0x7f28180732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f281f278640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2818156060 con 0x7f28180c3a70 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f281f278640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28180a8430 con 0x7f281806db30 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f281f278640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2818155b30 con 0x7f281806d750 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f281c813640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f28180c3a70 0x7f2818072680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39260/0 (socket says 172.21.15.94:39260) 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f281c813640 1 -- 172.21.15.94:0/819526531 learned_addr learned my addr 172.21.15.94:0/819526531 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f281d014640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 0x7f28180732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491213981 0 0) 0x7f2818156060 con 0x7f28180c3a70 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2804003520 con 0x7f28180c3a70 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2987801634 0 0) 0x7f2804003520 con 0x7f28180c3a70 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 msgr2=0x7f28180732a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 0x7f28180732a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 >> v1:172.21.15.145:6789/0 conn(0x7f281806db30 legacy=0x7f28180c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2818155d80 con 0x7f28180c3a70 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.708+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f2808001670 con 0x7f28180c3a70 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 >> v1:172.21.15.94:6789/0 conn(0x7f28180c3a70 legacy=0x7f2818072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f280ffff640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2804003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f280ffff640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 0x7f2804004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f280ffff640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2804004d40 0x7f28040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2818155b30 con 0x7f28040039e0 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280400bb50 con 0x7f2804004d40 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280400bd20 con 0x7f281806d750 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281d014640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2804003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281d815640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2804004d40 0x7f28040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281c813640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 0x7f2804004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281d014640 1 -- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 msgr2=0x7f2804004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281d014640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806d750 0x7f2804004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281d014640 1 -- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2804004d40 msgr2=0x7f28040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281d014640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2804004d40 0x7f28040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.709+0000 7f281d014640 1 -- 172.21.15.94:0/819526531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2804003520 con 0x7f28040039e0 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.710+0000 7f281d014640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2804003dc0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f2814009af0 tx=0x7f2804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f028997cb24adbf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.710+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814039070 con 0x7f28040039e0 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.710+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281402c6b0 con 0x7f28040039e0 2024-03-06T07:26:11.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.710+0000 7f280ffff640 1 -- 172.21.15.94:0/819526531 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814009690 con 0x7f28040039e0 2024-03-06T07:26:11.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.711+0000 7f281f278640 1 -- 172.21.15.94:0/819526531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 msgr2=0x7f2804003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.711+0000 7f281f278640 1 --2- 172.21.15.94:0/819526531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2804003dc0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f2814009af0 tx=0x7f2804005830 comp rx=0 tx=0).stop 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.711+0000 7f281d815640 1 -- 172.21.15.94:0/819526531 reap_dead start 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.711+0000 7f281f278640 1 -- 172.21.15.94:0/819526531 shutdown_connections 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.711+0000 7f281f278640 1 -- 172.21.15.94:0/819526531 >> 172.21.15.94:0/819526531 conn(0x7f28180c5560 msgr2=0x7f28180c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.711+0000 7f281f278640 1 -- 172.21.15.94:0/819526531 shutdown_connections 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.711+0000 7f281f278640 1 -- 172.21.15.94:0/819526531 wait complete. 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.712+0000 7f281f278640 1 Processor -- start 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.712+0000 7f281f278640 1 -- start start 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.712+0000 7f281f278640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2818152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.712+0000 7f281f278640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f281814d4c0 0x7f281806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.712+0000 7f281f278640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806f020 0x7f28181ede90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281f278640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28180c2720 con 0x7f28040039e0 2024-03-06T07:26:11.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281f278640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28180c25a0 con 0x7f281814d4c0 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281f278640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28180a7cd0 con 0x7f281806f020 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2818152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2818152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34708/0 (socket says 172.21.15.94:34708) 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 -- 172.21.15.94:0/4182395483 learned_addr learned my addr 172.21.15.94:0/4182395483 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281c813640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f281814d4c0 0x7f281806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d815640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806f020 0x7f28181ede90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 -- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806f020 msgr2=0x7f28181ede90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806f020 0x7f28181ede90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 -- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f281814d4c0 msgr2=0x7f281806eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f281814d4c0 0x7f281806eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 -- 172.21.15.94:0/4182395483 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28181eb890 con 0x7f28040039e0 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281c813640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f281814d4c0 0x7f281806eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281d014640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2818152450 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f2814009af0 tx=0x7f2814004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f280dffb640 1 -- 172.21.15.94:0/4182395483 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814042020 con 0x7f28040039e0 2024-03-06T07:26:11.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f280dffb640 1 -- 172.21.15.94:0/4182395483 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281403a040 con 0x7f28040039e0 2024-03-06T07:26:11.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281814fcb0 con 0x7f28040039e0 2024-03-06T07:26:11.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.713+0000 7f280dffb640 1 -- 172.21.15.94:0/4182395483 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814039070 con 0x7f28040039e0 2024-03-06T07:26:11.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.714+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2818152990 con 0x7f28040039e0 2024-03-06T07:26:11.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.715+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28181eb3a0 con 0x7f28040039e0 2024-03-06T07:26:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.716+0000 7f280dffb640 1 -- 172.21.15.94:0/4182395483 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2814045020 con 0x7f28040039e0 2024-03-06T07:26:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.716+0000 7f280dffb640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.716+0000 7f280dffb640 1 -- 172.21.15.94:0/4182395483 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f28140ab6b0 con 0x7f28040039e0 2024-03-06T07:26:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.716+0000 7f281c813640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.717+0000 7f281c813640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27fc06b500 0x7f27fc06d9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2808001120 tx=0x7f2808009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.721+0000 7f280dffb640 1 -- 172.21.15.94:0/4182395483 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f281404c070 con 0x7f28040039e0 2024-03-06T07:26:11.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.897+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f281804f330 con 0x7f28040039e0 2024-03-06T07:26:11.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.897+0000 7f280dffb640 1 -- 172.21.15.94:0/4182395483 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v184) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2814079d70 con 0x7f28040039e0 2024-03-06T07:26:11.899 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27fc06b500 msgr2=0x7f27fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27fc06b500 0x7f27fc06d9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2808001120 tx=0x7f2808009040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 msgr2=0x7f2818152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2818152450 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f2814009af0 tx=0x7f2814004730 comp rx=0 tx=0).stop 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 shutdown_connections 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f281806f020 0x7f28181ede90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f281814d4c0 0x7f281806eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 --2- 172.21.15.94:0/4182395483 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f28040039e0 0x7f2818152450 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 >> 172.21.15.94:0/4182395483 conn(0x7f28180c5560 msgr2=0x7f28180c74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 shutdown_connections 2024-03-06T07:26:11.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:11.899+0000 7f281f278640 1 -- 172.21.15.94:0/4182395483 wait complete. 2024-03-06T07:26:11.912 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:12.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.015+0000 7f87137b6640 1 Processor -- start 2024-03-06T07:26:12.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.015+0000 7f87137b6640 1 -- start start 2024-03-06T07:26:12.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f87137b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f870c0c1050 0x7f870c06e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f87137b6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870c1550d0 con 0x7f870c0c0bc0 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f87137b6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870c152ad0 con 0x7f870c1526f0 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f87137b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f870c155fa0 con 0x7f870c0c1050 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f8710d51640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f870c0c0bc0 0x7f870c06dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39266/0 (socket says 172.21.15.94:39266) 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f8710d51640 1 -- 172.21.15.94:0/3700661204 learned_addr learned my addr 172.21.15.94:0/3700661204 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f8711d53640 1 --2- 172.21.15.94:0/3700661204 >> v2:172.21.15.145:3300/0 conn(0x7f870c0c1050 0x7f870c06e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.016+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507322400 0 0) 0x7f870c1550d0 con 0x7f870c0c0bc0 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f8003520 con 0x7f870c0c0bc0 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4050610858 0 0) 0x7f86f8003520 con 0x7f870c0c0bc0 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 >> v2:172.21.15.145:3300/0 conn(0x7f870c0c1050 msgr2=0x7f870c06e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 --2- 172.21.15.94:0/3700661204 >> v2:172.21.15.145:3300/0 conn(0x7f870c0c1050 0x7f870c06e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 >> v1:172.21.15.145:6789/0 conn(0x7f870c1526f0 legacy=0x7f870c0a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c155390 con 0x7f870c0c0bc0 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f86fc001670 con 0x7f870c0c0bc0 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 >> v1:172.21.15.94:6789/0 conn(0x7f870c0c0bc0 legacy=0x7f870c06dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80039b0 0x7f86f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86f8004800 0x7f86f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86f80051c0 0x7f86f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c155fa0 con 0x7f86f80051c0 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f800c110 con 0x7f86f8004800 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f800c310 con 0x7f86f80039b0 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8711d53640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86f80051c0 0x7f86f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8711552640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80039b0 0x7f86f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8710d51640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86f8004800 0x7f86f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8711d53640 1 -- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80039b0 msgr2=0x7f86f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8711d53640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80039b0 0x7f86f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8711d53640 1 -- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86f8004800 msgr2=0x7f86f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8711d53640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86f8004800 0x7f86f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.017+0000 7f8711d53640 1 -- 172.21.15.94:0/3700661204 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f8003520 con 0x7f86f80051c0 2024-03-06T07:26:12.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.018+0000 7f8711d53640 1 -- 172.21.15.94:0/3700661204 reap_dead start 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f8711d53640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86f80051c0 0x7f86f800b9c0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f8704009fc0 tx=0x7f86f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b9db1cf0db8d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704009c10 con 0x7f86f80051c0 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870402cca0 con 0x7f86f80051c0 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f8703fff640 1 -- 172.21.15.94:0/3700661204 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704003bd0 con 0x7f86f80051c0 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f87137b6640 1 -- 172.21.15.94:0/3700661204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86f80051c0 msgr2=0x7f86f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f87137b6640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86f80051c0 0x7f86f800b9c0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f8704009fc0 tx=0x7f86f8005830 comp rx=0 tx=0).stop 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f87137b6640 1 -- 172.21.15.94:0/3700661204 shutdown_connections 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f87137b6640 1 --2- 172.21.15.94:0/3700661204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f86f80051c0 0x7f86f800b9c0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f87137b6640 1 -- 172.21.15.94:0/3700661204 >> 172.21.15.94:0/3700661204 conn(0x7f870c0a9540 msgr2=0x7f870c0a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f87137b6640 1 -- 172.21.15.94:0/3700661204 shutdown_connections 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.019+0000 7f87137b6640 1 -- 172.21.15.94:0/3700661204 wait complete. 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.020+0000 7f87137b6640 1 Processor -- start 2024-03-06T07:26:12.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f87137b6640 1 -- start start 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f87137b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80051c0 0x7f870c072670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f87137b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f870c06c7d0 0x7f870c1474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f87137b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f870c147a30 0x7f870c14e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f87137b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c155c30 con 0x7f870c06c7d0 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f87137b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c155950 con 0x7f870c147a30 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f87137b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c0c5820 con 0x7f86f80051c0 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f8710d51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f870c06c7d0 0x7f870c1474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f8710d51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f870c06c7d0 0x7f870c1474f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34722/0 (socket says 172.21.15.94:34722) 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f8710d51640 1 -- 172.21.15.94:0/774507795 learned_addr learned my addr 172.21.15.94:0/774507795 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f8711552640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80051c0 0x7f870c072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f8710d51640 1 -- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80051c0 msgr2=0x7f870c072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.021+0000 7f8711d53640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f870c147a30 0x7f870c14e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8710d51640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80051c0 0x7f870c072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8710d51640 1 -- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f870c147a30 msgr2=0x7f870c14e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8710d51640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f870c147a30 0x7f870c14e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8710d51640 1 -- 172.21.15.94:0/774507795 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c1ee860 con 0x7f870c06c7d0 2024-03-06T07:26:12.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8710d51640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f870c06c7d0 0x7f870c1474f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f86fc004ec0 tx=0x7f86fc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8701ffb640 1 -- 172.21.15.94:0/774507795 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc01a070 con 0x7f870c06c7d0 2024-03-06T07:26:12.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8701ffb640 1 -- 172.21.15.94:0/774507795 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc00f750 con 0x7f870c06c7d0 2024-03-06T07:26:12.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870c0a8440 con 0x7f870c06c7d0 2024-03-06T07:26:12.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f8701ffb640 1 -- 172.21.15.94:0/774507795 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00fb40 con 0x7f870c06c7d0 2024-03-06T07:26:12.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.022+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870c14de20 con 0x7f870c06c7d0 2024-03-06T07:26:12.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.023+0000 7f8701ffb640 1 -- 172.21.15.94:0/774507795 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86fc01d070 con 0x7f870c06c7d0 2024-03-06T07:26:12.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.023+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870c04f330 con 0x7f870c06c7d0 2024-03-06T07:26:12.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.024+0000 7f8701ffb640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86f006b500 0x7f86f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.024+0000 7f8701ffb640 1 -- 172.21.15.94:0/774507795 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f86fc089b90 con 0x7f870c06c7d0 2024-03-06T07:26:12.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.024+0000 7f8711552640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86f006b500 0x7f86f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.024+0000 7f8711552640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86f006b500 0x7f86f006d9c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f87080055a0 tx=0x7f8708009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.032+0000 7f8701ffb640 1 -- 172.21.15.94:0/774507795 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86fc0588a0 con 0x7f870c06c7d0 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.220+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f870c06f190 con 0x7f870c06c7d0 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.220+0000 7f8701ffb640 1 -- 172.21.15.94:0/774507795 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v184) v1 ==== 100+0+11 (secure 0 0 0) 0x7f86fc0581a0 con 0x7f870c06c7d0 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86f006b500 msgr2=0x7f86f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86f006b500 0x7f86f006d9c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f87080055a0 tx=0x7f8708009040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f870c06c7d0 msgr2=0x7f870c1474f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f870c06c7d0 0x7f870c1474f0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f86fc004ec0 tx=0x7f86fc007430 comp rx=0 tx=0).stop 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 shutdown_connections 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f86f006b500 0x7f86f006d9c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f870c147a30 0x7f870c14e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f870c06c7d0 0x7f870c1474f0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 --2- 172.21.15.94:0/774507795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86f80051c0 0x7f870c072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 >> 172.21.15.94:0/774507795 conn(0x7f870c0a9540 msgr2=0x7f870c0a9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.222+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 shutdown_connections 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.223+0000 7f87137b6640 1 -- 172.21.15.94:0/774507795 wait complete. 2024-03-06T07:26:12.235 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.350+0000 7fe8a0be2640 1 Processor -- start 2024-03-06T07:26:12.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.350+0000 7fe8a0be2640 1 -- start start 2024-03-06T07:26:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe8a0be2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe89c14f360 0x7fe89c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe8a0be2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c88c0 0x7fe89c0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe8a0be2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0c73d0 con 0x7fe89c0c5880 2024-03-06T07:26:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe8a0be2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c0c8dd0 con 0x7fe89c14f360 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe8a0be2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0be2e0 con 0x7fe89c0c88c0 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe89a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe89c14f360 0x7fe89c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe89a59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe89c14f360 0x7fe89c155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34738/0 (socket says 172.21.15.94:34738) 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.351+0000 7fe89a59c640 1 -- 172.21.15.94:0/3624330983 learned_addr learned my addr 172.21.15.94:0/3624330983 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.352+0000 7fe893fff640 1 --2- 172.21.15.94:0/3624330983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c88c0 0x7fe89c0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.352+0000 7fe89a59c640 1 -- 172.21.15.94:0/3624330983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c88c0 msgr2=0x7fe89c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.352+0000 7fe89a59c640 1 --2- 172.21.15.94:0/3624330983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c88c0 0x7fe89c0c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.352+0000 7fe89a59c640 1 -- 172.21.15.94:0/3624330983 >> v1:172.21.15.145:6789/0 conn(0x7fe89c0c5880 legacy=0x7fe89c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.352+0000 7fe89a59c640 1 -- 172.21.15.94:0/3624330983 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0c7670 con 0x7fe89c14f360 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.352+0000 7fe893fff640 1 --2- 172.21.15.94:0/3624330983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c88c0 0x7fe89c0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.352+0000 7fe89a59c640 1 --2- 172.21.15.94:0/3624330983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c155c10 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009870 tx=0x7fe88c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98203c3578232f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe899d9b640 1 -- 172.21.15.94:0/3624330983 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0317c0 con 0x7fe89c14f360 2024-03-06T07:26:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe899d9b640 1 -- 172.21.15.94:0/3624330983 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c037070 con 0x7fe89c14f360 2024-03-06T07:26:12.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe899d9b640 1 -- 172.21.15.94:0/3624330983 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c031b80 con 0x7fe89c14f360 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 -- 172.21.15.94:0/3624330983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 msgr2=0x7fe89c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/3624330983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c155c10 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009870 tx=0x7fe88c02b860 comp rx=0 tx=0).stop 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 -- 172.21.15.94:0/3624330983 shutdown_connections 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/3624330983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c88c0 0x7fe89c0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/3624330983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c155c10 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 -- 172.21.15.94:0/3624330983 >> 172.21.15.94:0/3624330983 conn(0x7fe89c0ac570 msgr2=0x7fe89c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 -- 172.21.15.94:0/3624330983 shutdown_connections 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.353+0000 7fe8a0be2640 1 -- 172.21.15.94:0/3624330983 wait complete. 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.355+0000 7fe8a0be2640 1 Processor -- start 2024-03-06T07:26:12.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.355+0000 7fe8a0be2640 1 -- start start 2024-03-06T07:26:12.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe8a0be2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c5880 0x7fe89c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe8a0be2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe89c0c88c0 0x7fe89c0bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe8a0be2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c0bfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe8a0be2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c159270 con 0x7fe89c14f360 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe8a0be2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0c7c30 con 0x7fe89c0c88c0 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe8a0be2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c1593f0 con 0x7fe89c0c5880 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe89ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c0bfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.356+0000 7fe89ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c0bfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34750/0 (socket says 172.21.15.94:34750) 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89ad9d640 1 -- 172.21.15.94:0/276456762 learned_addr learned my addr 172.21.15.94:0/276456762 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89a59c640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c5880 0x7fe89c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe893fff640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe89c0c88c0 0x7fe89c0bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89ad9d640 1 -- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c5880 msgr2=0x7fe89c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89ad9d640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c5880 0x7fe89c1579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89ad9d640 1 -- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe89c0c88c0 msgr2=0x7fe89c0bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89ad9d640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe89c0c88c0 0x7fe89c0bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89ad9d640 1 -- 172.21.15.94:0/276456762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0ad5a0 con 0x7fe89c14f360 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe89ad9d640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c0bfe60 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c7b0 tx=0x7fe89400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe892ffd640 1 -- 172.21.15.94:0/276456762 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894004b60 con 0x7fe89c14f360 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe892ffd640 1 -- 172.21.15.94:0/276456762 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894004d00 con 0x7fe89c14f360 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe892ffd640 1 -- 172.21.15.94:0/276456762 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89400aa90 con 0x7fe89c14f360 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c1eb080 con 0x7fe89c14f360 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.357+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c1eb640 con 0x7fe89c14f360 2024-03-06T07:26:12.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.358+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c04f330 con 0x7fe89c14f360 2024-03-06T07:26:12.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.359+0000 7fe892ffd640 1 -- 172.21.15.94:0/276456762 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe89400ac30 con 0x7fe89c14f360 2024-03-06T07:26:12.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.359+0000 7fe892ffd640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87c06b4d0 0x7fe87c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.359+0000 7fe892ffd640 1 -- 172.21.15.94:0/276456762 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7fe89408a640 con 0x7fe89c14f360 2024-03-06T07:26:12.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.359+0000 7fe89a59c640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87c06b4d0 0x7fe87c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.360+0000 7fe89a59c640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87c06b4d0 0x7fe87c06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe88c036000 tx=0x7fe88c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.364+0000 7fe892ffd640 1 -- 172.21.15.94:0/276456762 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe894020070 con 0x7fe89c14f360 2024-03-06T07:26:12.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.579+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe89c0c6950 con 0x7fe89c14f360 2024-03-06T07:26:12.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.579+0000 7fe892ffd640 1 -- 172.21.15.94:0/276456762 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5014 (secure 0 0 0) 0x7fe894059350 con 0x7fe89c14f360 2024-03-06T07:26:12.581 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:12.581 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":180,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:08.285097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9891,"mds_1":9888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9888":{"gid":9888,"name":"c","rank":1,"incarnation":179,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9891":{"gid":9891,"name":"a","rank":0,"incarnation":177,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9891,"qdb_cluster":[9888,9891]},"id":14}]} 2024-03-06T07:26:12.582 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 180 2024-03-06T07:26:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87c06b4d0 msgr2=0x7fe87c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87c06b4d0 0x7fe87c06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe88c036000 tx=0x7fe88c03d040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 msgr2=0x7fe89c0bfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c0bfe60 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c7b0 tx=0x7fe89400cc70 comp rx=0 tx=0).stop 2024-03-06T07:26:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 shutdown_connections 2024-03-06T07:26:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe87c06b4d0 0x7fe87c06d990 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe89c14f360 0x7fe89c0bfe60 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe89c0c88c0 0x7fe89c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 --2- 172.21.15.94:0/276456762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe89c0c5880 0x7fe89c1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 >> 172.21.15.94:0/276456762 conn(0x7fe89c0ac570 msgr2=0x7fe89c0ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 shutdown_connections 2024-03-06T07:26:12.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.582+0000 7fe8a0be2640 1 -- 172.21.15.94:0/276456762 wait complete. 2024-03-06T07:26:12.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:12.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.701+0000 7f6983039640 1 Processor -- start 2024-03-06T07:26:12.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.701+0000 7f6983039640 1 -- start start 2024-03-06T07:26:12.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f6983039640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f697c14f790 0x7f697c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f6983039640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f697c155a40 0x7f697c155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f6983039640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c0c28d0 con 0x7f697c0c1aa0 2024-03-06T07:26:12.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f6983039640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697c0c2680 con 0x7f697c155a40 2024-03-06T07:26:12.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f6983039640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697c0c2bb0 con 0x7f697c14f790 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f69815d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f697c155a40 0x7f697c155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f69815d6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f697c155a40 0x7f697c155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34754/0 (socket says 172.21.15.94:34754) 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f6973fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f697c0c1aa0 0x7f697c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39282/0 (socket says 172.21.15.94:39282) 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f69815d6640 1 -- 172.21.15.94:0/2356478770 learned_addr learned my addr 172.21.15.94:0/2356478770 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.702+0000 7f6980dd5640 1 --2- 172.21.15.94:0/2356478770 >> v2:172.21.15.145:3300/0 conn(0x7f697c14f790 0x7f697c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.703+0000 7f69815d6640 1 -- 172.21.15.94:0/2356478770 >> v2:172.21.15.145:3300/0 conn(0x7f697c14f790 msgr2=0x7f697c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.703+0000 7f69815d6640 1 --2- 172.21.15.94:0/2356478770 >> v2:172.21.15.145:3300/0 conn(0x7f697c14f790 0x7f697c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.703+0000 7f69815d6640 1 -- 172.21.15.94:0/2356478770 >> v1:172.21.15.94:6789/0 conn(0x7f697c0c1aa0 legacy=0x7f697c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.703+0000 7f69815d6640 1 -- 172.21.15.94:0/2356478770 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c0c2e90 con 0x7f697c155a40 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.703+0000 7f69737fe640 1 -- 172.21.15.94:0/2356478770 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626520437 0 0) 0x7f697c0c28d0 con 0x7f697c0c1aa0 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.703+0000 7f6980dd5640 1 --2- 172.21.15.94:0/2356478770 >> v2:172.21.15.145:3300/0 conn(0x7f697c14f790 0x7f697c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.703+0000 7f69815d6640 1 --2- 172.21.15.94:0/2356478770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c155e50 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f697c0c3170 tx=0x7f697402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65822979f2db1ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f69737fe640 1 -- 172.21.15.94:0/2356478770 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697402cb20 con 0x7f697c155a40 2024-03-06T07:26:12.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f69737fe640 1 -- 172.21.15.94:0/2356478770 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697402ccc0 con 0x7f697c155a40 2024-03-06T07:26:12.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f69737fe640 1 -- 172.21.15.94:0/2356478770 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974034610 con 0x7f697c155a40 2024-03-06T07:26:12.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 -- 172.21.15.94:0/2356478770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 msgr2=0x7f697c155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 --2- 172.21.15.94:0/2356478770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c155e50 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f697c0c3170 tx=0x7f697402bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:12.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 -- 172.21.15.94:0/2356478770 shutdown_connections 2024-03-06T07:26:12.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 --2- 172.21.15.94:0/2356478770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c155e50 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 --2- 172.21.15.94:0/2356478770 >> v2:172.21.15.145:3300/0 conn(0x7f697c14f790 0x7f697c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 -- 172.21.15.94:0/2356478770 >> 172.21.15.94:0/2356478770 conn(0x7f697c0c6710 msgr2=0x7f697c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 -- 172.21.15.94:0/2356478770 shutdown_connections 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.704+0000 7f6983039640 1 -- 172.21.15.94:0/2356478770 wait complete. 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.705+0000 7f6983039640 1 Processor -- start 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.705+0000 7f6983039640 1 -- start start 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6983039640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f697c0c1aa0 0x7f697c154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6983039640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f697c14f790 0x7f697c1490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6983039640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6983039640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0c0d60 con 0x7f697c155a40 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6983039640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c156390 con 0x7f697c0c1aa0 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6983039640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697c0c5e30 con 0x7f697c14f790 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c149630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34758/0 (socket says 172.21.15.94:34758) 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 -- 172.21.15.94:0/972567207 learned_addr learned my addr 172.21.15.94:0/972567207 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6980dd5640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f697c0c1aa0 0x7f697c154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6973fff640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f697c14f790 0x7f697c1490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 -- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f697c14f790 msgr2=0x7f697c1490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f697c14f790 0x7f697c1490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 -- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f697c0c1aa0 msgr2=0x7f697c154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f697c0c1aa0 0x7f697c154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f69815d6640 1 -- 172.21.15.94:0/972567207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c154860 con 0x7f697c155a40 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.706+0000 7f6980dd5640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f697c0c1aa0 0x7f697c154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.707+0000 7f69815d6640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c149630 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f697402c590 tx=0x7f6974004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.707+0000 7f69717fa640 1 -- 172.21.15.94:0/972567207 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974043020 con 0x7f697c155a40 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.707+0000 7f69717fa640 1 -- 172.21.15.94:0/972567207 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697403e450 con 0x7f697c155a40 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.707+0000 7f69717fa640 1 -- 172.21.15.94:0/972567207 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974009510 con 0x7f697c155a40 2024-03-06T07:26:12.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.707+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c146ff0 con 0x7f697c155a40 2024-03-06T07:26:12.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.707+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c1534a0 con 0x7f697c155a40 2024-03-06T07:26:12.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.708+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697c067ad0 con 0x7f697c155a40 2024-03-06T07:26:12.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.708+0000 7f69717fa640 1 -- 172.21.15.94:0/972567207 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6974046020 con 0x7f697c155a40 2024-03-06T07:26:12.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.709+0000 7f69717fa640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696806b4d0 0x7f696806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.709+0000 7f69717fa640 1 -- 172.21.15.94:0/972567207 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f69740ac880 con 0x7f697c155a40 2024-03-06T07:26:12.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.709+0000 7f6980dd5640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696806b4d0 0x7f696806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.710+0000 7f6980dd5640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696806b4d0 0x7f696806d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6978006fd0 tx=0x7f6978008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.717+0000 7f69717fa640 1 -- 172.21.15.94:0/972567207 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f697c067ad0 con 0x7f697c155a40 2024-03-06T07:26:12.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.952+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f697c04f330 con 0x7f697c155a40 2024-03-06T07:26:12.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.953+0000 7f69717fa640 1 -- 172.21.15.94:0/972567207 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+17076 (secure 0 0 0) 0x7f697404d070 con 0x7f697c155a40 2024-03-06T07:26:12.955 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:12.955 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":184,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:59.237111+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.909545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.667783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.7","mappings":[{"from":6,"to":0}]},{"pgid":"29.d","mappings":[{"from":6,"to":0}]},{"pgid":"29.13","mappings":[{"from":6,"to":0}]},{"pgid":"29.1e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:12.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696806b4d0 msgr2=0x7f696806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696806b4d0 0x7f696806d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6978006fd0 tx=0x7f6978008040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 msgr2=0x7f697c149630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c149630 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f697402c590 tx=0x7f6974004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:12.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 shutdown_connections 2024-03-06T07:26:12.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f696806b4d0 0x7f696806d990 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f697c155a40 0x7f697c149630 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f697c14f790 0x7f697c1490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 --2- 172.21.15.94:0/972567207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f697c0c1aa0 0x7f697c154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 >> 172.21.15.94:0/972567207 conn(0x7f697c0c6710 msgr2=0x7f697c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 shutdown_connections 2024-03-06T07:26:12.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:12.955+0000 7f6983039640 1 -- 172.21.15.94:0/972567207 wait complete. 2024-03-06T07:26:12.968 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:12.968 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:13.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.066+0000 7f9c18b1e640 1 Processor -- start 2024-03-06T07:26:13.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.066+0000 7f9c18b1e640 1 -- start start 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.067+0000 7f9c18b1e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9c1414f360 0x7f9c14155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.067+0000 7f9c18b1e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140c73d0 con 0x7f9c14158b60 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.067+0000 7f9c18b1e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140be2e0 con 0x7f9c140c88c0 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.067+0000 7f9c18b1e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c14159700 con 0x7f9c1414f360 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.067+0000 7f9c12d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9c14158b60 0x7f9c14158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39290/0 (socket says 172.21.15.94:39290) 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.067+0000 7f9c12d9d640 1 -- 172.21.15.94:0/1295298111 learned_addr learned my addr 172.21.15.94:0/1295298111 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1259c640 1 --2- 172.21.15.94:0/1295298111 >> v2:172.21.15.145:3300/0 conn(0x7f9c1414f360 0x7f9c14155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446630897 0 0) 0x7f9c140c73d0 con 0x7f9c14158b60 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc003520 con 0x7f9c14158b60 2024-03-06T07:26:13.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823529569 0 0) 0x7f9bfc003520 con 0x7f9c14158b60 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 >> v2:172.21.15.145:3300/0 conn(0x7f9c1414f360 msgr2=0x7f9c14155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 --2- 172.21.15.94:0/1295298111 >> v2:172.21.15.145:3300/0 conn(0x7f9c1414f360 0x7f9c14155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 >> v1:172.21.15.145:6789/0 conn(0x7f9c140c88c0 legacy=0x7f9c140c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140c7670 con 0x7f9c14158b60 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9c0c0019f0 con 0x7f9c14158b60 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 >> v1:172.21.15.94:6789/0 conn(0x7f9c14158b60 legacy=0x7f9c14158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc0039b0 0x7f9bfc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bfc0051c0 0x7f9bfc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14159700 con 0x7f9bfc004800 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc00c160 con 0x7f9bfc0051c0 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc00c360 con 0x7f9bfc0039b0 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c11d9b640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c1259c640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc0039b0 0x7f9bfc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c12d9d640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bfc0051c0 0x7f9bfc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c11d9b640 1 -- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc0039b0 msgr2=0x7f9bfc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c11d9b640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc0039b0 0x7f9bfc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c11d9b640 1 -- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bfc0051c0 msgr2=0x7f9bfc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c11d9b640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bfc0051c0 0x7f9bfc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c12d9d640 1 -- 172.21.15.94:0/1295298111 reap_dead start 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.068+0000 7f9c11d9b640 1 -- 172.21.15.94:0/1295298111 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc003520 con 0x7f9bfc004800 2024-03-06T07:26:13.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.069+0000 7f9c11d9b640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f9c08001720 tx=0x7f9bfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42990c9b11b240b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.069+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08001370 con 0x7f9bfc004800 2024-03-06T07:26:13.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.069+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c080052d0 con 0x7f9bfc004800 2024-03-06T07:26:13.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.069+0000 7f9c1159a640 1 -- 172.21.15.94:0/1295298111 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08003830 con 0x7f9bfc004800 2024-03-06T07:26:13.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.070+0000 7f9c18b1e640 1 -- 172.21.15.94:0/1295298111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bfc004800 msgr2=0x7f9bfc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.070+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f9c08001720 tx=0x7f9bfc005830 comp rx=0 tx=0).stop 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.070+0000 7f9c18b1e640 1 -- 172.21.15.94:0/1295298111 shutdown_connections 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.070+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/1295298111 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.070+0000 7f9c18b1e640 1 -- 172.21.15.94:0/1295298111 >> 172.21.15.94:0/1295298111 conn(0x7f9c140ac570 msgr2=0x7f9c140ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.070+0000 7f9c18b1e640 1 -- 172.21.15.94:0/1295298111 shutdown_connections 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.070+0000 7f9c18b1e640 1 -- 172.21.15.94:0/1295298111 wait complete. 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 Processor -- start 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 -- start start 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc004800 0x7f9c140c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c140b00b0 0x7f9c141ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c141ec220 0x7f9c141edd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140c7c30 con 0x7f9c140b00b0 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14155c10 con 0x7f9c141ec220 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c18b1e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14155d90 con 0x7f9bfc004800 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c11d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c140b00b0 0x7f9c141ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.072+0000 7f9c11d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c140b00b0 0x7f9c141ebce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34776/0 (socket says 172.21.15.94:34776) 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c11d9b640 1 -- 172.21.15.94:0/777886675 learned_addr learned my addr 172.21.15.94:0/777886675 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c12d9d640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c141ec220 0x7f9c141edd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c1259c640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc004800 0x7f9c140c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c11d9b640 1 -- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc004800 msgr2=0x7f9c140c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c11d9b640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc004800 0x7f9c140c0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c11d9b640 1 -- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c141ec220 msgr2=0x7f9c141edd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c11d9b640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c141ec220 0x7f9c141edd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c11d9b640 1 -- 172.21.15.94:0/777886675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140c14a0 con 0x7f9c140b00b0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c11d9b640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c140b00b0 0x7f9c141ebce0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f9c08005040 tx=0x7f9c08001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c02ffd640 1 -- 172.21.15.94:0/777886675 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08041070 con 0x7f9c140b00b0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c02ffd640 1 -- 172.21.15.94:0/777886675 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08005480 con 0x7f9c140b00b0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c02ffd640 1 -- 172.21.15.94:0/777886675 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08007a30 con 0x7f9c140b00b0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c140ae110 con 0x7f9c140b00b0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.073+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c140ae6d0 con 0x7f9c140b00b0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.074+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1404f330 con 0x7f9c140b00b0 2024-03-06T07:26:13.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.075+0000 7f9c02ffd640 1 -- 172.21.15.94:0/777886675 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c08044070 con 0x7f9c140b00b0 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.075+0000 7f9c02ffd640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf406b4d0 0x7f9bf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.076+0000 7f9c02ffd640 1 -- 172.21.15.94:0/777886675 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f9c080ac020 con 0x7f9c140b00b0 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.076+0000 7f9c1259c640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf406b4d0 0x7f9bf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.076+0000 7f9c1259c640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf406b4d0 0x7f9bf406d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009a00 tx=0x7f9c04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.084+0000 7f9c02ffd640 1 -- 172.21.15.94:0/777886675 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c0807ae50 con 0x7f9c140b00b0 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.396+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c1406cda0 con 0x7f9c140b00b0 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.397+0000 7f9c02ffd640 1 -- 172.21.15.94:0/777886675 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5014 (secure 0 0 0) 0x7f9c0807a750 con 0x7f9c140b00b0 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":180,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:08.285097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9891,"mds_1":9888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9888":{"gid":9888,"name":"c","rank":1,"incarnation":179,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9891":{"gid":9891,"name":"a","rank":0,"incarnation":177,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9891,"qdb_cluster":[9888,9891]},"id":14}]} 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 180 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf406b4d0 msgr2=0x7f9bf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf406b4d0 0x7f9bf406d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009a00 tx=0x7f9c04008040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c140b00b0 msgr2=0x7f9c141ebce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c140b00b0 0x7f9c141ebce0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f9c08005040 tx=0x7f9c08001120 comp rx=0 tx=0).stop 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 shutdown_connections 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9bf406b4d0 0x7f9bf406d990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c141ec220 0x7f9c141edd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9c140b00b0 0x7f9c141ebce0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 --2- 172.21.15.94:0/777886675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bfc004800 0x7f9c140c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 >> 172.21.15.94:0/777886675 conn(0x7f9c140ac570 msgr2=0x7f9c1414cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 shutdown_connections 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.399+0000 7f9c18b1e640 1 -- 172.21.15.94:0/777886675 wait complete. 2024-03-06T07:26:13.411 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:13.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.519+0000 7f80ab9ae640 1 Processor -- start 2024-03-06T07:26:13.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.519+0000 7f80ab9ae640 1 -- start start 2024-03-06T07:26:13.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80ab9ae640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f80a414f360 0x7f80a4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80ab9ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80ab9ae640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f80a40c5880 0x7f80a4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80ab9ae640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a40c8dd0 con 0x7f80a40c5880 2024-03-06T07:26:13.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80ab9ae640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a40c73d0 con 0x7f80a414f360 2024-03-06T07:26:13.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80ab9ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a40be2e0 con 0x7f80a40c88c0 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f80a40c5880 0x7f80a4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f80a40c5880 0x7f80a4158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34778/0 (socket says 172.21.15.94:34778) 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 -- 172.21.15.94:0/3652536626 learned_addr learned my addr 172.21.15.94:0/3652536626 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a8f49640 1 --2- 172.21.15.94:0/3652536626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a974a640 1 --2- 172.21.15.94:0/3652536626 >> v2:172.21.15.145:3300/0 conn(0x7f80a414f360 0x7f80a4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 -- 172.21.15.94:0/3652536626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 msgr2=0x7f80a40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 --2- 172.21.15.94:0/3652536626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a40c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 -- 172.21.15.94:0/3652536626 >> v2:172.21.15.145:3300/0 conn(0x7f80a414f360 msgr2=0x7f80a4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 --2- 172.21.15.94:0/3652536626 >> v2:172.21.15.145:3300/0 conn(0x7f80a414f360 0x7f80a4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.520+0000 7f80a9f4b640 1 -- 172.21.15.94:0/3652536626 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a40c7670 con 0x7f80a40c5880 2024-03-06T07:26:13.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.521+0000 7f80a9f4b640 1 --2- 172.21.15.94:0/3652536626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a40c5880 0x7f80a4158b60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f80a408c840 tx=0x7f809c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6416f98ab25d2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.521+0000 7f809bfff640 1 -- 172.21.15.94:0/3652536626 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c02c9e0 con 0x7f80a40c5880 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f809bfff640 1 -- 172.21.15.94:0/3652536626 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c02cb80 con 0x7f80a40c5880 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f809bfff640 1 -- 172.21.15.94:0/3652536626 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c0343f0 con 0x7f80a40c5880 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 -- 172.21.15.94:0/3652536626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a40c5880 msgr2=0x7f80a4158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/3652536626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a40c5880 0x7f80a4158b60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f80a408c840 tx=0x7f809c02be40 comp rx=0 tx=0).stop 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 -- 172.21.15.94:0/3652536626 shutdown_connections 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/3652536626 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a40c5880 0x7f80a4158b60 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/3652536626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/3652536626 >> v2:172.21.15.145:3300/0 conn(0x7f80a414f360 0x7f80a4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 -- 172.21.15.94:0/3652536626 >> 172.21.15.94:0/3652536626 conn(0x7f80a40ac570 msgr2=0x7f80a40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 -- 172.21.15.94:0/3652536626 shutdown_connections 2024-03-06T07:26:13.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.522+0000 7f80ab9ae640 1 -- 172.21.15.94:0/3652536626 wait complete. 2024-03-06T07:26:13.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.524+0000 7f80ab9ae640 1 Processor -- start 2024-03-06T07:26:13.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.524+0000 7f80ab9ae640 1 -- start start 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80ab9ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a40c5880 0x7f80a40c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80ab9ae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a414e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80ab9ae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a414f360 0x7f80a4147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80ab9ae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a41590a0 con 0x7f80a414f360 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80ab9ae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a40c7c30 con 0x7f80a40c5880 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80ab9ae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a4159220 con 0x7f80a40c88c0 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a414f360 0x7f80a4147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a414f360 0x7f80a4147ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34788/0 (socket says 172.21.15.94:34788) 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 -- 172.21.15.94:0/108717901 learned_addr learned my addr 172.21.15.94:0/108717901 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:13.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a8f49640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a414e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a974a640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a40c5880 0x7f80a40c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 -- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 msgr2=0x7f80a414e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a414e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 -- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a40c5880 msgr2=0x7f80a40c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a40c5880 0x7f80a40c3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.525+0000 7f80a9f4b640 1 -- 172.21.15.94:0/108717901 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a40adab0 con 0x7f80a414f360 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.526+0000 7f80a9f4b640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a414f360 0x7f80a4147ed0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f809c036040 tx=0x7f809c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.526+0000 7f8099ffb640 1 -- 172.21.15.94:0/108717901 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c043020 con 0x7f80a414f360 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.526+0000 7f8099ffb640 1 -- 172.21.15.94:0/108717901 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c03e450 con 0x7f80a414f360 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.526+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a40add90 con 0x7f80a414f360 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.526+0000 7f8099ffb640 1 -- 172.21.15.94:0/108717901 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c009510 con 0x7f80a414f360 2024-03-06T07:26:13.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.526+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a414ea50 con 0x7f80a414f360 2024-03-06T07:26:13.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.527+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a404f330 con 0x7f80a414f360 2024-03-06T07:26:13.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.528+0000 7f8099ffb640 1 -- 172.21.15.94:0/108717901 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f809c046020 con 0x7f80a414f360 2024-03-06T07:26:13.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.528+0000 7f8099ffb640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f808806b4d0 0x7f808806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.528+0000 7f8099ffb640 1 -- 172.21.15.94:0/108717901 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f809c0aca70 con 0x7f80a414f360 2024-03-06T07:26:13.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.528+0000 7f80a974a640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f808806b4d0 0x7f808806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.529+0000 7f80a974a640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f808806b4d0 0x7f808806d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f80a00079a0 tx=0x7f80a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.532+0000 7f8099ffb640 1 -- 172.21.15.94:0/108717901 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f809c04d030 con 0x7f80a414f360 2024-03-06T07:26:13.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.747+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a40c52e0 con 0x7f80a414f360 2024-03-06T07:26:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.748+0000 7f8099ffb640 1 -- 172.21.15.94:0/108717901 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5014 (secure 0 0 0) 0x7f809c07b130 con 0x7f80a414f360 2024-03-06T07:26:13.750 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:13.750 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":180,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.699027+0000","modified":"2024-03-06T07:26:08.285097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9891,"mds_1":9888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9888":{"gid":9888,"name":"c","rank":1,"incarnation":179,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6835/2744471722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2744471722},{"type":"v1","addr":"172.21.15.94:6835","nonce":2744471722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9891":{"gid":9891,"name":"a","rank":0,"incarnation":177,"state":"up:active","state_seq":10,"addr":"172.21.15.94:6834/4225816089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":4225816089},{"type":"v1","addr":"172.21.15.94:6834","nonce":4225816089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9891,"qdb_cluster":[9888,9891]},"id":14}]} 2024-03-06T07:26:13.750 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 180 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f808806b4d0 msgr2=0x7f808806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f808806b4d0 0x7f808806d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f80a00079a0 tx=0x7f80a0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a414f360 msgr2=0x7f80a4147ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a414f360 0x7f80a4147ed0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f809c036040 tx=0x7f809c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 shutdown_connections 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f808806b4d0 0x7f808806d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80a414f360 0x7f80a4147ed0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a40c88c0 0x7f80a414e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 --2- 172.21.15.94:0/108717901 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a40c5880 0x7f80a40c3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 >> 172.21.15.94:0/108717901 conn(0x7f80a40ac570 msgr2=0x7f80a406c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 shutdown_connections 2024-03-06T07:26:13.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.750+0000 7f80ab9ae640 1 -- 172.21.15.94:0/108717901 wait complete. 2024-03-06T07:26:13.763 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:13.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.898+0000 7f300579a640 1 Processor -- start 2024-03-06T07:26:13.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.898+0000 7f300579a640 1 -- start start 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f300579a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f30001506b0 0x7f3000153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f300579a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3000156ea0 0x7f30000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f300579a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30000acc00 0x7f30000c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f300579a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3000156450 con 0x7f30001506b0 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f300579a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3000156a10 con 0x7f3000156ea0 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f300579a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3000070800 con 0x7f30000acc00 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f2ffeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f30001506b0 0x7f3000153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f2ffeffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f30001506b0 0x7f3000153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34802/0 (socket says 172.21.15.94:34802) 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f2ffeffd640 1 -- 172.21.15.94:0/279677967 learned_addr learned my addr 172.21.15.94:0/279677967 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f2ffe7fc640 1 --2- 172.21.15.94:0/279677967 >> v2:172.21.15.145:3300/0 conn(0x7f3000156ea0 0x7f30000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.899+0000 7f2fff7fe640 1 --2- 172.21.15.94:0/279677967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30000acc00 0x7f30000c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.900+0000 7f2ffeffd640 1 -- 172.21.15.94:0/279677967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30000acc00 msgr2=0x7f30000c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.900+0000 7f2ffeffd640 1 --2- 172.21.15.94:0/279677967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30000acc00 0x7f30000c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.900+0000 7f2ffeffd640 1 -- 172.21.15.94:0/279677967 >> v2:172.21.15.145:3300/0 conn(0x7f3000156ea0 msgr2=0x7f30000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.900+0000 7f2ffeffd640 1 --2- 172.21.15.94:0/279677967 >> v2:172.21.15.145:3300/0 conn(0x7f3000156ea0 0x7f30000702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.900+0000 7f2ffeffd640 1 -- 172.21.15.94:0/279677967 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000070aa0 con 0x7f30001506b0 2024-03-06T07:26:13.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.900+0000 7f2fff7fe640 1 --2- 172.21.15.94:0/279677967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30000acc00 0x7f30000c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.900+0000 7f2ffe7fc640 1 --2- 172.21.15.94:0/279677967 >> v2:172.21.15.145:3300/0 conn(0x7f3000156ea0 0x7f30000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f2ffeffd640 1 --2- 172.21.15.94:0/279677967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30001506b0 0x7f3000153990 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f300006bf40 tx=0x7f2ff002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9c792063385a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f2ffdffb640 1 -- 172.21.15.94:0/279677967 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0035020 con 0x7f30001506b0 2024-03-06T07:26:13.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f2ffdffb640 1 -- 172.21.15.94:0/279677967 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff0031870 con 0x7f30001506b0 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f2ffdffb640 1 -- 172.21.15.94:0/279677967 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0031c40 con 0x7f30001506b0 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f300579a640 1 -- 172.21.15.94:0/279677967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30001506b0 msgr2=0x7f3000153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f300579a640 1 --2- 172.21.15.94:0/279677967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30001506b0 0x7f3000153990 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f300006bf40 tx=0x7f2ff002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f300579a640 1 -- 172.21.15.94:0/279677967 shutdown_connections 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f300579a640 1 --2- 172.21.15.94:0/279677967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30000acc00 0x7f30000c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f300579a640 1 --2- 172.21.15.94:0/279677967 >> v2:172.21.15.145:3300/0 conn(0x7f3000156ea0 0x7f30000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f300579a640 1 --2- 172.21.15.94:0/279677967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30001506b0 0x7f3000153990 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.901+0000 7f300579a640 1 -- 172.21.15.94:0/279677967 >> 172.21.15.94:0/279677967 conn(0x7f300006b1a0 msgr2=0x7f300006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.902+0000 7f300579a640 1 -- 172.21.15.94:0/279677967 shutdown_connections 2024-03-06T07:26:13.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.902+0000 7f300579a640 1 -- 172.21.15.94:0/279677967 wait complete. 2024-03-06T07:26:13.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.903+0000 7f300579a640 1 Processor -- start 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.903+0000 7f300579a640 1 -- start start 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.904+0000 7f300579a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30000acc00 0x7f30001ec7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.904+0000 7f300579a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30001506b0 0x7f3000152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.904+0000 7f300579a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3000156ea0 0x7f3000148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.904+0000 7f300579a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30000c59e0 con 0x7f3000156ea0 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.904+0000 7f300579a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30001566a0 con 0x7f30000acc00 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f300579a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30000c5b60 con 0x7f30001506b0 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3000156ea0 0x7f3000148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3000156ea0 0x7f3000148d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34810/0 (socket says 172.21.15.94:34810) 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 -- 172.21.15.94:0/1690955181 learned_addr learned my addr 172.21.15.94:0/1690955181 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 -- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30001506b0 msgr2=0x7f3000152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30001506b0 0x7f3000152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 -- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30000acc00 msgr2=0x7f30001ec7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2ffeffd640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30000acc00 0x7f30001ec7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30000acc00 0x7f30001ec7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 -- 172.21.15.94:0/1690955181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000152920 con 0x7f3000156ea0 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fff7fe640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3000156ea0 0x7f3000148d70 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8007d90 tx=0x7f2ff800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fe77fe640 1 -- 172.21.15.94:0/1690955181 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff801a070 con 0x7f3000156ea0 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fe77fe640 1 -- 172.21.15.94:0/1690955181 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff8003990 con 0x7f3000156ea0 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f2fe77fe640 1 -- 172.21.15.94:0/1690955181 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8003d80 con 0x7f3000156ea0 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30001ebd00 con 0x7f3000156ea0 2024-03-06T07:26:13.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.905+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30001ec2c0 con 0x7f3000156ea0 2024-03-06T07:26:13.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.906+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30001ec4c0 con 0x7f3000156ea0 2024-03-06T07:26:13.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.907+0000 7f2fe77fe640 1 -- 172.21.15.94:0/1690955181 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ff801d070 con 0x7f3000156ea0 2024-03-06T07:26:13.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.907+0000 7f2fe77fe640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fec06b4d0 0x7f2fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.907+0000 7f2fe77fe640 1 -- 172.21.15.94:0/1690955181 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f2ff808a550 con 0x7f3000156ea0 2024-03-06T07:26:13.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.907+0000 7f2ffeffd640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fec06b4d0 0x7f2fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.908+0000 7f2ffeffd640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fec06b4d0 0x7f2fec06d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0036000 tx=0x7f2ff0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:13.911+0000 7f2fe77fe640 1 -- 172.21.15.94:0/1690955181 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30001ec4c0 con 0x7f3000156ea0 2024-03-06T07:26:14.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.090+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f300004f330 con 0x7f3000156ea0 2024-03-06T07:26:14.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.091+0000 7f2fe77fe640 1 -- 172.21.15.94:0/1690955181 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+17076 (secure 0 0 0) 0x7f2ff8010070 con 0x7f3000156ea0 2024-03-06T07:26:14.093 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:14.093 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":184,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:25:59.237111+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.909545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.667783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"182","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.7","mappings":[{"from":6,"to":0}]},{"pgid":"29.d","mappings":[{"from":6,"to":0}]},{"pgid":"29.13","mappings":[{"from":6,"to":0}]},{"pgid":"29.1e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fec06b4d0 msgr2=0x7f2fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fec06b4d0 0x7f2fec06d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0036000 tx=0x7f2ff0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3000156ea0 msgr2=0x7f3000148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3000156ea0 0x7f3000148d70 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8007d90 tx=0x7f2ff800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 shutdown_connections 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fec06b4d0 0x7f2fec06d990 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3000156ea0 0x7f3000148d70 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30001506b0 0x7f3000152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 --2- 172.21.15.94:0/1690955181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30000acc00 0x7f30001ec7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 >> 172.21.15.94:0/1690955181 conn(0x7f300006b1a0 msgr2=0x7f30000c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 shutdown_connections 2024-03-06T07:26:14.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.093+0000 7f300579a640 1 -- 172.21.15.94:0/1690955181 wait complete. 2024-03-06T07:26:14.107 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:14.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.207+0000 7f0db0cb5640 1 Processor -- start 2024-03-06T07:26:14.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.207+0000 7f0db0cb5640 1 -- start start 2024-03-06T07:26:14.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0db0cb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0dac0c69f0 0x7f0dac0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0db0cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0c8950 0x7f0dac0acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0db0cb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dac155130 con 0x7f0dac0ad2b0 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0db0cb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dac155440 con 0x7f0dac0c69f0 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0db0cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0c5d50 con 0x7f0dac0c8950 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0daad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0dac0ad2b0 0x7f0dac1558d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39298/0 (socket says 172.21.15.94:39298) 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0daad9d640 1 -- 172.21.15.94:0/1609807403 learned_addr learned my addr 172.21.15.94:0/1609807403 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0da3fff640 1 --2- 172.21.15.94:0/1609807403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0c8950 0x7f0dac0acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0daa59c640 1 --2- 172.21.15.94:0/1609807403 >> v2:172.21.15.145:3300/0 conn(0x7f0dac0c69f0 0x7f0dac0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0da9d9b640 1 -- 172.21.15.94:0/1609807403 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027943034 0 0) 0x7f0da4001490 con 0x7f0dac0ad2b0 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.208+0000 7f0da9d9b640 1 -- 172.21.15.94:0/1609807403 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d98003520 con 0x7f0dac0ad2b0 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.209+0000 7f0da3fff640 1 -- 172.21.15.94:0/1609807403 >> v2:172.21.15.145:3300/0 conn(0x7f0dac0c69f0 msgr2=0x7f0dac0c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.209+0000 7f0da3fff640 1 --2- 172.21.15.94:0/1609807403 >> v2:172.21.15.145:3300/0 conn(0x7f0dac0c69f0 0x7f0dac0c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.209+0000 7f0da3fff640 1 -- 172.21.15.94:0/1609807403 >> v1:172.21.15.94:6789/0 conn(0x7f0dac0ad2b0 legacy=0x7f0dac1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.209+0000 7f0da9d9b640 1 -- 172.21.15.94:0/1609807403 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725910018 0 0) 0x7f0d98003520 con 0x7f0dac0ad2b0 2024-03-06T07:26:14.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.209+0000 7f0da3fff640 1 -- 172.21.15.94:0/1609807403 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac0c6030 con 0x7f0dac0c8950 2024-03-06T07:26:14.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.209+0000 7f0daa59c640 1 --2- 172.21.15.94:0/1609807403 >> v2:172.21.15.145:3300/0 conn(0x7f0dac0c69f0 0x7f0dac0c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:14.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.209+0000 7f0da3fff640 1 --2- 172.21.15.94:0/1609807403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0c8950 0x7f0dac0acd70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0d940097a0 tx=0x7f0d9402b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce0b3f8150c303c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0da9d9b640 1 -- 172.21.15.94:0/1609807403 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d94037070 con 0x7f0dac0c8950 2024-03-06T07:26:14.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0da9d9b640 1 -- 172.21.15.94:0/1609807403 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d94030510 con 0x7f0dac0c8950 2024-03-06T07:26:14.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0da9d9b640 1 -- 172.21.15.94:0/1609807403 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4001bc0 con 0x7f0dac0c8950 2024-03-06T07:26:14.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1609807403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0c8950 msgr2=0x7f0dac0acd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1609807403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0c8950 0x7f0dac0acd70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0d940097a0 tx=0x7f0d9402b710 comp rx=0 tx=0).stop 2024-03-06T07:26:14.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1609807403 shutdown_connections 2024-03-06T07:26:14.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1609807403 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0c8950 0x7f0dac0acd70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1609807403 >> v2:172.21.15.145:3300/0 conn(0x7f0dac0c69f0 0x7f0dac0c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1609807403 >> 172.21.15.94:0/1609807403 conn(0x7f0dac0cae70 msgr2=0x7f0dac0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1609807403 shutdown_connections 2024-03-06T07:26:14.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.210+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1609807403 wait complete. 2024-03-06T07:26:14.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.211+0000 7f0db0cb5640 1 Processor -- start 2024-03-06T07:26:14.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.211+0000 7f0db0cb5640 1 -- start start 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0db0cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0ad2b0 0x7f0dac1eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0db0cb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0dac0c69f0 0x7f0dac06c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0db0cb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0dac0c8950 0x7f0dac1ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0db0cb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac152a20 con 0x7f0dac0c8950 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0db0cb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac155fe0 con 0x7f0dac0c69f0 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0db0cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac152ba0 con 0x7f0dac0ad2b0 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0daad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0dac0c8950 0x7f0dac1ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0daad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0dac0c8950 0x7f0dac1ee040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34826/0 (socket says 172.21.15.94:34826) 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0daad9d640 1 -- 172.21.15.94:0/1846216349 learned_addr learned my addr 172.21.15.94:0/1846216349 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0daa59c640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0ad2b0 0x7f0dac1eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.212+0000 7f0da3fff640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0dac0c69f0 0x7f0dac06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0daad9d640 1 -- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0ad2b0 msgr2=0x7f0dac1eb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0daad9d640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0ad2b0 0x7f0dac1eb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0daad9d640 1 -- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0dac0c69f0 msgr2=0x7f0dac06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0daad9d640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0dac0c69f0 0x7f0dac06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0daad9d640 1 -- 172.21.15.94:0/1846216349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac1eaad0 con 0x7f0dac0c8950 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0da3fff640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0dac0c69f0 0x7f0dac06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0daad9d640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0dac0c8950 0x7f0dac1ee040 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f0da400cc10 tx=0x7f0da400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0da2ffd640 1 -- 172.21.15.94:0/1846216349 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da400ce20 con 0x7f0dac0c8950 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0da2ffd640 1 -- 172.21.15.94:0/1846216349 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4012400 con 0x7f0dac0c8950 2024-03-06T07:26:14.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac0c4540 con 0x7f0dac0c8950 2024-03-06T07:26:14.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0da2ffd640 1 -- 172.21.15.94:0/1846216349 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da40127f0 con 0x7f0dac0c8950 2024-03-06T07:26:14.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.213+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dac1ead90 con 0x7f0dac0c8950 2024-03-06T07:26:14.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.214+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dac075300 con 0x7f0dac0c8950 2024-03-06T07:26:14.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.215+0000 7f0da2ffd640 1 -- 172.21.15.94:0/1846216349 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0da4012990 con 0x7f0dac0c8950 2024-03-06T07:26:14.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.215+0000 7f0da2ffd640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.215+0000 7f0da2ffd640 1 -- 172.21.15.94:0/1846216349 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6217+0+0 (secure 0 0 0) 0x7f0da408aab0 con 0x7f0dac0c8950 2024-03-06T07:26:14.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.215+0000 7f0daa59c640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.216+0000 7f0daa59c640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0dac154eb0 tx=0x7f0d9c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.220+0000 7f0da2ffd640 1 -- 172.21.15.94:0/1846216349 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dac075300 con 0x7f0dac0c8950 2024-03-06T07:26:14.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.546+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0dac04f330 con 0x7f0dac0c8950 2024-03-06T07:26:14.548 INFO:tasks.ceph.mon.a.smithi094.stderr:2024-03-06T07:26:14.546+0000 7f90f571c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.548+0000 7f0da2ffd640 1 -- 172.21.15.94:0/1846216349 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v181) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0dac155130 con 0x7f0dac0c8950 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d8c06b4d0 msgr2=0x7f0d8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0dac154eb0 tx=0x7f0d9c009190 comp rx=0 tx=0).stop 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0dac0c8950 msgr2=0x7f0dac1ee040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0dac0c8950 0x7f0dac1ee040 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f0da400cc10 tx=0x7f0da400a430 comp rx=0 tx=0).stop 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 shutdown_connections 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d8c06b4d0 0x7f0d8c06d990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0dac0c8950 0x7f0dac1ee040 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0dac0c69f0 0x7f0dac06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 --2- 172.21.15.94:0/1846216349 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0dac0ad2b0 0x7f0dac1eb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 >> 172.21.15.94:0/1846216349 conn(0x7f0dac0cae70 msgr2=0x7f0dac1558d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 shutdown_connections 2024-03-06T07:26:14.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.551+0000 7f0db0cb5640 1 -- 172.21.15.94:0/1846216349 wait complete. 2024-03-06T07:26:14.554 INFO:tasks.ceph.mds.a.smithi094.stderr: -287> 2024-03-06T07:26:08.624+0000 7efcf172f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:14.554 INFO:tasks.ceph.mds.a.smithi094.stderr: -285> 2024-03-06T07:26:08.624+0000 7efcf172f640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:26:14.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:14.578 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:26:14.578 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:26:14.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.668+0000 7ffa62fae640 1 Processor -- start 2024-03-06T07:26:14.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.669+0000 7ffa62fae640 1 -- start start 2024-03-06T07:26:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.669+0000 7ffa62fae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c06ca70 0x7ffa5c14e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.669+0000 7ffa62fae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c0b0450 con 0x7ffa5c0b0070 2024-03-06T07:26:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.669+0000 7ffa62fae640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c1560e0 con 0x7ffa5c155c30 2024-03-06T07:26:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.669+0000 7ffa62fae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c06c020 con 0x7ffa5c06ca70 2024-03-06T07:26:14.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.669+0000 7ffa60d4a640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ffa5c0b0070 0x7ffa5c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39312/0 (socket says 172.21.15.94:39312) 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.669+0000 7ffa60d4a640 1 -- 172.21.15.94:0/1424167661 learned_addr learned my addr 172.21.15.94:0/1424167661 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa6154b640 1 --2- 172.21.15.94:0/1424167661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c06ca70 0x7ffa5c14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935481730 0 0) 0x7ffa50001110 con 0x7ffa5c0b0070 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c003520 con 0x7ffa5c0b0070 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032027663 0 0) 0x7ffa5c1560e0 con 0x7ffa5c155c30 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa50001110 con 0x7ffa5c155c30 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030540235 0 0) 0x7ffa4c003520 con 0x7ffa5c0b0070 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa6154b640 1 -- 172.21.15.94:0/1424167661 >> v1:172.21.15.145:6789/0 conn(0x7ffa5c155c30 legacy=0x7ffa5c0c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa6154b640 1 -- 172.21.15.94:0/1424167661 >> v1:172.21.15.94:6789/0 conn(0x7ffa5c0b0070 legacy=0x7ffa5c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.670+0000 7ffa6154b640 1 -- 172.21.15.94:0/1424167661 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c0c77f0 con 0x7ffa5c06ca70 2024-03-06T07:26:14.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa6154b640 1 --2- 172.21.15.94:0/1424167661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c06ca70 0x7ffa5c14e800 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c1562b0 tx=0x7ffa5402bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcace75c4461e43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa54039070 con 0x7ffa5c06ca70 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5402c880 con 0x7ffa5c06ca70 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa5b7fe640 1 -- 172.21.15.94:0/1424167661 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa50001840 con 0x7ffa5c06ca70 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa62fae640 1 -- 172.21.15.94:0/1424167661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c06ca70 msgr2=0x7ffa5c14e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa62fae640 1 --2- 172.21.15.94:0/1424167661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c06ca70 0x7ffa5c14e800 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c1562b0 tx=0x7ffa5402bd50 comp rx=0 tx=0).stop 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa62fae640 1 -- 172.21.15.94:0/1424167661 shutdown_connections 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa62fae640 1 --2- 172.21.15.94:0/1424167661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c06ca70 0x7ffa5c14e800 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa62fae640 1 -- 172.21.15.94:0/1424167661 >> 172.21.15.94:0/1424167661 conn(0x7ffa5c0ac590 msgr2=0x7ffa5c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa62fae640 1 -- 172.21.15.94:0/1424167661 shutdown_connections 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.671+0000 7ffa62fae640 1 -- 172.21.15.94:0/1424167661 wait complete. 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 Processor -- start 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 -- start start 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa5c06ca70 0x7ffa5c151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c0b0070 0x7ffa5c14d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c155c30 0x7ffa5c14d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c06c550 con 0x7ffa5c0b0070 2024-03-06T07:26:14.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c06c270 con 0x7ffa5c06ca70 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.673+0000 7ffa62fae640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c06da80 con 0x7ffa5c155c30 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c0b0070 0x7ffa5c14d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c0b0070 0x7ffa5c14d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34880/0 (socket says 172.21.15.94:34880) 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 -- 172.21.15.94:0/81590020 learned_addr learned my addr 172.21.15.94:0/81590020 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa6154b640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c155c30 0x7ffa5c14d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa60d4a640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa5c06ca70 0x7ffa5c151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 -- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c155c30 msgr2=0x7ffa5c14d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c155c30 0x7ffa5c14d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 -- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa5c06ca70 msgr2=0x7ffa5c151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa5c06ca70 0x7ffa5c151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 -- 172.21.15.94:0/81590020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c14dc90 con 0x7ffa5c0b0070 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa5bfff640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c0b0070 0x7ffa5c14d210 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800c430 tx=0x7ffa4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa597fa640 1 -- 172.21.15.94:0/81590020 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48017070 con 0x7ffa5c0b0070 2024-03-06T07:26:14.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa597fa640 1 -- 172.21.15.94:0/81590020 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa480052f0 con 0x7ffa5c0b0070 2024-03-06T07:26:14.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa597fa640 1 -- 172.21.15.94:0/81590020 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48003830 con 0x7ffa5c0b0070 2024-03-06T07:26:14.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5c14e070 con 0x7ffa5c0b0070 2024-03-06T07:26:14.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.674+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5c1587c0 con 0x7ffa5c0b0070 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.675+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5c04f330 con 0x7ffa5c0b0070 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.676+0000 7ffa597fa640 1 -- 172.21.15.94:0/81590020 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa4801a070 con 0x7ffa5c0b0070 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.676+0000 7ffa597fa640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4006b4d0 0x7ffa4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.676+0000 7ffa597fa640 1 -- 172.21.15.94:0/81590020 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6389+0+0 (secure 0 0 0) 0x7ffa48089840 con 0x7ffa5c0b0070 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.676+0000 7ffa60d4a640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4006b4d0 0x7ffa4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.677+0000 7ffa60d4a640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4006b4d0 0x7ffa4006d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ffa50001eb0 tx=0x7ffa50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.680+0000 7ffa597fa640 1 -- 172.21.15.94:0/81590020 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa480584a0 con 0x7ffa5c0b0070 2024-03-06T07:26:14.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:14.887+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffa5c152140 con 0x7ffa5c0b0070 2024-03-06T07:26:15.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.552+0000 7ffa597fa640 1 -- 172.21.15.94:0/81590020 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v182) v1 ==== 104+0+0 (secure 0 0 0) 0x7ffa48057da0 con 0x7ffa5c0b0070 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4006b4d0 msgr2=0x7ffa4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4006b4d0 0x7ffa4006d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ffa50001eb0 tx=0x7ffa50009040 comp rx=0 tx=0).stop 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c0b0070 msgr2=0x7ffa5c14d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c0b0070 0x7ffa5c14d210 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800c430 tx=0x7ffa4800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 shutdown_connections 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa4006b4d0 0x7ffa4006d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5c155c30 0x7ffa5c14d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c0b0070 0x7ffa5c14d210 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 --2- 172.21.15.94:0/81590020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa5c06ca70 0x7ffa5c151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 >> 172.21.15.94:0/81590020 conn(0x7ffa5c0ac590 msgr2=0x7ffa5c0af240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 shutdown_connections 2024-03-06T07:26:15.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.555+0000 7ffa62fae640 1 -- 172.21.15.94:0/81590020 wait complete. 2024-03-06T07:26:15.570 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:15.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.713+0000 7f181fcb5640 1 Processor -- start 2024-03-06T07:26:15.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.713+0000 7f181fcb5640 1 -- start start 2024-03-06T07:26:15.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181fcb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 0x7f1818151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181fcb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18180a9830 0x7f181814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181fcb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181806e850 con 0x7f181814f490 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181fcb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181806d2b0 con 0x7f18180a9830 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181fcb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181806d590 con 0x7f18180c22b0 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181e252640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f181814f490 0x7f1818153390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39328/0 (socket says 172.21.15.94:39328) 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181e252640 1 -- 172.21.15.94:0/1806693022 learned_addr learned my addr 172.21.15.94:0/1806693022 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181da51640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 0x7f1818151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743059377 0 0) 0x7f1810001490 con 0x7f181814f490 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1804003520 con 0x7f181814f490 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.714+0000 7f181d250640 1 --2- 172.21.15.94:0/1806693022 >> v2:172.21.15.145:3300/0 conn(0x7f18180a9830 0x7f181814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3468569329 0 0) 0x7f1804003520 con 0x7f181814f490 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 msgr2=0x7f1818151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 0x7f1818151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 >> v2:172.21.15.145:3300/0 conn(0x7f18180a9830 msgr2=0x7f181814ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 --2- 172.21.15.94:0/1806693022 >> v2:172.21.15.145:3300/0 conn(0x7f18180a9830 0x7f181814ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181806d870 con 0x7f181814f490 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f1810001bc0 con 0x7f181814f490 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 >> v1:172.21.15.94:6789/0 conn(0x7f181814f490 legacy=0x7f1818153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 0x7f18040039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804003f80 0x7f1804004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1804004d40 0x7f18040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181806d2b0 con 0x7f1804004d40 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181806d590 con 0x7f1804003f80 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180400b9c0 con 0x7f18180c22b0 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181e252640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1804004d40 0x7f18040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181da51640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 0x7f18040039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181d250640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804003f80 0x7f1804004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181e252640 1 -- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 msgr2=0x7f18040039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181e252640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180c22b0 0x7f18040039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181e252640 1 -- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804003f80 msgr2=0x7f1804004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181e252640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804003f80 0x7f1804004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181e252640 1 -- 172.21.15.94:0/1806693022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1804003520 con 0x7f1804004d40 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.715+0000 7f181d250640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804003f80 0x7f1804004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.716+0000 7f181e252640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1804004d40 0x7f18040051c0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f181000a9b0 tx=0x7f1804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8eb8913fb48ef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.716+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181000a5a0 con 0x7f1804004d40 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.716+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1810007290 con 0x7f1804004d40 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.716+0000 7f181ca4f640 1 -- 172.21.15.94:0/1806693022 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810007680 con 0x7f1804004d40 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.716+0000 7f181fcb5640 1 -- 172.21.15.94:0/1806693022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1804004d40 msgr2=0x7f18040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.716+0000 7f181fcb5640 1 --2- 172.21.15.94:0/1806693022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1804004d40 0x7f18040051c0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f181000a9b0 tx=0x7f1804005830 comp rx=0 tx=0).stop 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.716+0000 7f181e252640 1 -- 172.21.15.94:0/1806693022 reap_dead start 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.717+0000 7f181fcb5640 1 -- 172.21.15.94:0/1806693022 shutdown_connections 2024-03-06T07:26:15.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.717+0000 7f181fcb5640 1 -- 172.21.15.94:0/1806693022 >> 172.21.15.94:0/1806693022 conn(0x7f18180c65b0 msgr2=0x7f18180c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.717+0000 7f181fcb5640 1 -- 172.21.15.94:0/1806693022 shutdown_connections 2024-03-06T07:26:15.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.717+0000 7f181fcb5640 1 -- 172.21.15.94:0/1806693022 wait complete. 2024-03-06T07:26:15.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 Processor -- start 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 -- start start 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804004d40 0x7f181813f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180aad10 0x7f1818072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18180c4a20 0x7f18180c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180c2810 con 0x7f18180c4a20 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180c2690 con 0x7f1804004d40 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.718+0000 7f181fcb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818151e10 con 0x7f18180aad10 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18180c4a20 0x7f18180c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18180c4a20 0x7f18180c4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34896/0 (socket says 172.21.15.94:34896) 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 -- 172.21.15.94:0/896260342 learned_addr learned my addr 172.21.15.94:0/896260342 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181d250640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180aad10 0x7f1818072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181da51640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804004d40 0x7f181813f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 -- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180aad10 msgr2=0x7f1818072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180aad10 0x7f1818072cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 -- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804004d40 msgr2=0x7f181813f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804004d40 0x7f181813f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 -- 172.21.15.94:0/896260342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18180aba40 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181da51640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804004d40 0x7f181813f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181e252640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18180c4a20 0x7f18180c4ea0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f181002cd80 tx=0x7f1818070db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f180e7fc640 1 -- 172.21.15.94:0/896260342 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810045070 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f180e7fc640 1 -- 172.21.15.94:0/896260342 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1810004d60 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18180abcb0 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f180e7fc640 1 -- 172.21.15.94:0/896260342 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181003a040 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.719+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18180c4220 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.720+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181804f330 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.721+0000 7f180e7fc640 1 -- 172.21.15.94:0/896260342 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1810048070 con 0x7f18180c4a20 2024-03-06T07:26:15.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.721+0000 7f180e7fc640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17fc06b500 0x7f17fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.721+0000 7f180e7fc640 1 -- 172.21.15.94:0/896260342 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f18100ab520 con 0x7f18180c4a20 2024-03-06T07:26:15.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.721+0000 7f181da51640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17fc06b500 0x7f17fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.722+0000 7f181da51640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17fc06b500 0x7f17fc06d9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1814009ab0 tx=0x7f1814008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.725+0000 7f180e7fc640 1 -- 172.21.15.94:0/896260342 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f181806e850 con 0x7f18180c4a20 2024-03-06T07:26:15.752 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:15.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:15.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:15.902+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1818151560 con 0x7f18180c4a20 2024-03-06T07:26:16.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.553+0000 7f180e7fc640 1 -- 172.21.15.94:0/896260342 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v186) v1 ==== 181+0+0 (secure 0 0 0) 0x7f181007a180 con 0x7f18180c4a20 2024-03-06T07:26:16.555 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:16.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17fc06b500 msgr2=0x7f17fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17fc06b500 0x7f17fc06d9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1814009ab0 tx=0x7f1814008040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18180c4a20 msgr2=0x7f18180c4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18180c4a20 0x7f18180c4ea0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f181002cd80 tx=0x7f1818070db0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 shutdown_connections 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17fc06b500 0x7f17fc06d9c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18180c4a20 0x7f18180c4ea0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18180aad10 0x7f1818072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 --2- 172.21.15.94:0/896260342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1804004d40 0x7f181813f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 >> 172.21.15.94:0/896260342 conn(0x7f18180c65b0 msgr2=0x7f18180c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 shutdown_connections 2024-03-06T07:26:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.557+0000 7f181fcb5640 1 -- 172.21.15.94:0/896260342 wait complete. 2024-03-06T07:26:16.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:16.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.675+0000 7f6c565a0640 1 Processor -- start 2024-03-06T07:26:16.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c565a0640 1 -- start start 2024-03-06T07:26:16.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c565a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c500bf700 0x7f6c500bfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c565a0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6c500c51e0 0x7f6c500c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c565a0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6c50154f90 0x7f6c500c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c565a0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c500c7110 con 0x7f6c500c51e0 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c565a0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c500c5e20 con 0x7f6c50154f90 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c565a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c500cb510 con 0x7f6c500bf700 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c54d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6c500c51e0 0x7f6c500c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c54d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6c500c51e0 0x7f6c500c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34908/0 (socket says 172.21.15.94:34908) 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.676+0000 7f6c54d9d640 1 -- 172.21.15.94:0/2972417479 learned_addr learned my addr 172.21.15.94:0/2972417479 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c5559e640 1 --2- 172.21.15.94:0/2972417479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c500bf700 0x7f6c500bfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c55d9f640 1 --2- 172.21.15.94:0/2972417479 >> v2:172.21.15.145:3300/0 conn(0x7f6c50154f90 0x7f6c500c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c54d9d640 1 -- 172.21.15.94:0/2972417479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c500bf700 msgr2=0x7f6c500bfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c54d9d640 1 --2- 172.21.15.94:0/2972417479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c500bf700 0x7f6c500bfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c54d9d640 1 -- 172.21.15.94:0/2972417479 >> v2:172.21.15.145:3300/0 conn(0x7f6c50154f90 msgr2=0x7f6c500c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c54d9d640 1 --2- 172.21.15.94:0/2972417479 >> v2:172.21.15.145:3300/0 conn(0x7f6c50154f90 0x7f6c500c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c54d9d640 1 -- 172.21.15.94:0/2972417479 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c500c6070 con 0x7f6c500c51e0 2024-03-06T07:26:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c5559e640 1 --2- 172.21.15.94:0/2972417479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c500bf700 0x7f6c500bfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c55d9f640 1 --2- 172.21.15.94:0/2972417479 >> v2:172.21.15.145:3300/0 conn(0x7f6c50154f90 0x7f6c500c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.677+0000 7f6c54d9d640 1 --2- 172.21.15.94:0/2972417479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500c51e0 0x7f6c500c8760 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f6c500689f0 tx=0x7f6c4c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892a6444d792618a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c43fff640 1 -- 172.21.15.94:0/2972417479 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c0317d0 con 0x7f6c500c51e0 2024-03-06T07:26:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c43fff640 1 -- 172.21.15.94:0/2972417479 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c037070 con 0x7f6c500c51e0 2024-03-06T07:26:16.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c43fff640 1 -- 172.21.15.94:0/2972417479 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c031b90 con 0x7f6c500c51e0 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 -- 172.21.15.94:0/2972417479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500c51e0 msgr2=0x7f6c500c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 --2- 172.21.15.94:0/2972417479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500c51e0 0x7f6c500c8760 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f6c500689f0 tx=0x7f6c4c02b870 comp rx=0 tx=0).stop 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 -- 172.21.15.94:0/2972417479 shutdown_connections 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 --2- 172.21.15.94:0/2972417479 >> v2:172.21.15.145:3300/0 conn(0x7f6c50154f90 0x7f6c500c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 --2- 172.21.15.94:0/2972417479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500c51e0 0x7f6c500c8760 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 --2- 172.21.15.94:0/2972417479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c500bf700 0x7f6c500bfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 -- 172.21.15.94:0/2972417479 >> 172.21.15.94:0/2972417479 conn(0x7f6c500ca860 msgr2=0x7f6c500cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 -- 172.21.15.94:0/2972417479 shutdown_connections 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.678+0000 7f6c565a0640 1 -- 172.21.15.94:0/2972417479 wait complete. 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.679+0000 7f6c565a0640 1 Processor -- start 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.679+0000 7f6c565a0640 1 -- start start 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c565a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500bf700 0x7f6c5014e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c565a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c500c51e0 0x7f6c500c2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c565a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c50154f90 0x7f6c50158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c565a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c500c6910 con 0x7f6c500bf700 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c565a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c500c6630 con 0x7f6c500c51e0 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c565a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c500c6bf0 con 0x7f6c50154f90 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500bf700 0x7f6c5014e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500bf700 0x7f6c5014e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34922/0 (socket says 172.21.15.94:34922) 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 -- 172.21.15.94:0/3020935616 learned_addr learned my addr 172.21.15.94:0/3020935616 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c55d9f640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c50154f90 0x7f6c50158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c54d9d640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c500c51e0 0x7f6c500c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 -- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c50154f90 msgr2=0x7f6c50158390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c50154f90 0x7f6c50158390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 -- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c500c51e0 msgr2=0x7f6c500c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c500c51e0 0x7f6c500c2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c5559e640 1 -- 172.21.15.94:0/3020935616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c501ef020 con 0x7f6c500bf700 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c55d9f640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c50154f90 0x7f6c50158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.680+0000 7f6c54d9d640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c500c51e0 0x7f6c500c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.681+0000 7f6c5559e640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500bf700 0x7f6c5014e7f0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400c7f0 tx=0x7f6c4400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.681+0000 7f6c41ffb640 1 -- 172.21.15.94:0/3020935616 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44015070 con 0x7f6c500bf700 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.681+0000 7f6c41ffb640 1 -- 172.21.15.94:0/3020935616 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c44007820 con 0x7f6c500bf700 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.681+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c50156500 con 0x7f6c500bf700 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.681+0000 7f6c41ffb640 1 -- 172.21.15.94:0/3020935616 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44007c10 con 0x7f6c500bf700 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.681+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5014e2c0 con 0x7f6c500bf700 2024-03-06T07:26:16.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.682+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c501ef1f0 con 0x7f6c500bf700 2024-03-06T07:26:16.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.682+0000 7f6c41ffb640 1 -- 172.21.15.94:0/3020935616 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c44018070 con 0x7f6c500bf700 2024-03-06T07:26:16.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.683+0000 7f6c41ffb640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6c3406b500 0x7f6c3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.683+0000 7f6c41ffb640 1 -- 172.21.15.94:0/3020935616 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f6c4408a310 con 0x7f6c500bf700 2024-03-06T07:26:16.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.683+0000 7f6c54d9d640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6c3406b500 0x7f6c3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.683+0000 7f6c54d9d640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6c3406b500 0x7f6c3406d9c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c036000 tx=0x7f6c4c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.687+0000 7f6c41ffb640 1 -- 172.21.15.94:0/3020935616 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c44059120 con 0x7f6c500bf700 2024-03-06T07:26:16.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:16.861+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6c500693c0 con 0x7f6c500bf700 2024-03-06T07:26:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.558+0000 7f6c41ffb640 1 -- 172.21.15.94:0/3020935616 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v187) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6c44058a20 con 0x7f6c500bf700 2024-03-06T07:26:17.560 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:17.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6c3406b500 msgr2=0x7f6c3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6c3406b500 0x7f6c3406d9c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c036000 tx=0x7f6c4c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500bf700 msgr2=0x7f6c5014e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500bf700 0x7f6c5014e7f0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400c7f0 tx=0x7f6c4400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 shutdown_connections 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6c3406b500 0x7f6c3406d9c0 secure :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c036000 tx=0x7f6c4c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c50154f90 0x7f6c50158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c500c51e0 0x7f6c500c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 --2- 172.21.15.94:0/3020935616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c500bf700 0x7f6c5014e7f0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.562+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 >> 172.21.15.94:0/3020935616 conn(0x7f6c500ca860 msgr2=0x7f6c501e1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.563+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 shutdown_connections 2024-03-06T07:26:17.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.563+0000 7f6c565a0640 1 -- 172.21.15.94:0/3020935616 wait complete. 2024-03-06T07:26:17.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.683+0000 7f6d90a78640 1 Processor -- start 2024-03-06T07:26:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.683+0000 7f6d90a78640 1 -- start start 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.684+0000 7f6d90a78640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6d8c155c30 0x7f6d8c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.684+0000 7f6d90a78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 0x7f6d8c0c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.684+0000 7f6d90a78640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c0b0450 con 0x7f6d8c0b0070 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.684+0000 7f6d90a78640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8c06cfb0 con 0x7f6d8c155c30 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.684+0000 7f6d90a78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c06c020 con 0x7f6d8c0c5380 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.684+0000 7f6d8a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6d8c0b0070 0x7f6d8c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39334/0 (socket says 172.21.15.94:39334) 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.684+0000 7f6d8a59c640 1 -- 172.21.15.94:0/125728586 learned_addr learned my addr 172.21.15.94:0/125728586 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8ad9d640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 0x7f6d8c0c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712821487 0 0) 0x7f6d7c001110 con 0x7f6d8c0b0070 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d89d9b640 1 --2- 172.21.15.94:0/125728586 >> v2:172.21.15.145:3300/0 conn(0x7f6d8c155c30 0x7f6d8c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d74003520 con 0x7f6d8c0b0070 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2556560096 0 0) 0x7f6d74003520 con 0x7f6d8c0b0070 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 msgr2=0x7f6d8c0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 0x7f6d8c0c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 >> v2:172.21.15.145:3300/0 conn(0x7f6d8c155c30 msgr2=0x7f6d8c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 --2- 172.21.15.94:0/125728586 >> v2:172.21.15.145:3300/0 conn(0x7f6d8c155c30 0x7f6d8c06ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c1560e0 con 0x7f6d8c0b0070 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f6d7c001840 con 0x7f6d8c0b0070 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 >> v1:172.21.15.94:6789/0 conn(0x7f6d8c0b0070 legacy=0x7f6d8c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d740039b0 0x7f6d74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 0x7f6d74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d74004d40 0x7f6d740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c06cfb0 con 0x7f6d740039b0 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c06c020 con 0x7f6d74004d40 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7400b9c0 con 0x7f6d8c0c5380 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d740039b0 0x7f6d74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d89d9b640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 0x7f6d74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8ad9d640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d74004d40 0x7f6d740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8a59c640 1 -- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 msgr2=0x7f6d74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c0c5380 0x7f6d74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8a59c640 1 -- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d74004d40 msgr2=0x7f6d740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d74004d40 0x7f6d740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8a59c640 1 -- 172.21.15.94:0/125728586 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d74003520 con 0x7f6d740039b0 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.685+0000 7f6d8ad9d640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d74004d40 0x7f6d740051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.686+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d740039b0 0x7f6d74003dd0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00c3b0 tx=0x7f6d74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aef822018d4053e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.686+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c00c0e0 con 0x7f6d740039b0 2024-03-06T07:26:17.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.686+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c030070 con 0x7f6d740039b0 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.686+0000 7f6d8959a640 1 -- 172.21.15.94:0/125728586 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c0058d0 con 0x7f6d740039b0 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.687+0000 7f6d90a78640 1 -- 172.21.15.94:0/125728586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d740039b0 msgr2=0x7f6d74003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.687+0000 7f6d90a78640 1 --2- 172.21.15.94:0/125728586 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d740039b0 0x7f6d74003dd0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00c3b0 tx=0x7f6d74005830 comp rx=0 tx=0).stop 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.687+0000 7f6d8ad9d640 1 -- 172.21.15.94:0/125728586 reap_dead start 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.687+0000 7f6d90a78640 1 -- 172.21.15.94:0/125728586 shutdown_connections 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.687+0000 7f6d90a78640 1 -- 172.21.15.94:0/125728586 >> 172.21.15.94:0/125728586 conn(0x7f6d8c0ac590 msgr2=0x7f6d8c0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.687+0000 7f6d90a78640 1 -- 172.21.15.94:0/125728586 shutdown_connections 2024-03-06T07:26:17.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.687+0000 7f6d90a78640 1 -- 172.21.15.94:0/125728586 wait complete. 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.688+0000 7f6d90a78640 1 Processor -- start 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.688+0000 7f6d90a78640 1 -- start start 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d740039b0 0x7f6d8c14f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d8c0c24e0 0x7f6d8c0c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c14e550 0x7f6d8c14e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c06c270 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c0c7ad0 con 0x7f6d740039b0 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c06c550 con 0x7f6d8c14e550 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d8c0c24e0 0x7f6d8c0c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d8c0c24e0 0x7f6d8c0c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34934/0 (socket says 172.21.15.94:34934) 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 -- 172.21.15.94:0/381398942 learned_addr learned my addr 172.21.15.94:0/381398942 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d8ad9d640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c14e550 0x7f6d8c14e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d740039b0 0x7f6d8c14f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 -- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c14e550 msgr2=0x7f6d8c14e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c14e550 0x7f6d8c14e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 -- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d740039b0 msgr2=0x7f6d8c14f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d740039b0 0x7f6d8c14f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 -- 172.21.15.94:0/381398942 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c14f720 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d740039b0 0x7f6d8c14f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d89d9b640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d8c0c24e0 0x7f6d8c0c41f0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f6d80011aa0 tx=0x7f6d80011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d7affd640 1 -- 172.21.15.94:0/381398942 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8000fb80 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d7affd640 1 -- 172.21.15.94:0/381398942 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8000fd20 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c0c2a30 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d7affd640 1 -- 172.21.15.94:0/381398942 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80014660 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.689+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c0c2ff0 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.690+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c04f330 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.691+0000 7f6d7affd640 1 -- 172.21.15.94:0/381398942 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d80012040 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.691+0000 7f6d7affd640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d6c06b4d0 0x7f6d6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.691+0000 7f6d7affd640 1 -- 172.21.15.94:0/381398942 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f6d8008a6c0 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.692+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d6c06b4d0 0x7f6d6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.692+0000 7f6d8a59c640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d6c06b4d0 0x7f6d6c06d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c034000 tx=0x7f6d7c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.696+0000 7f6d7affd640 1 -- 172.21.15.94:0/381398942 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d80019030 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.874+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6d8c06d860 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.877+0000 7f6d7affd640 1 -- 172.21.15.94:0/381398942 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6d800596f0 con 0x7f6d8c0c24e0 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d6c06b4d0 msgr2=0x7f6d6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d6c06b4d0 0x7f6d6c06d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c034000 tx=0x7f6d7c007000 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d8c0c24e0 msgr2=0x7f6d8c0c41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d8c0c24e0 0x7f6d8c0c41f0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f6d80011aa0 tx=0x7f6d80011f60 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 shutdown_connections 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d6c06b4d0 0x7f6d6c06d990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d8c14e550 0x7f6d8c14e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d8c0c24e0 0x7f6d8c0c41f0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 --2- 172.21.15.94:0/381398942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d740039b0 0x7f6d8c14f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 >> 172.21.15.94:0/381398942 conn(0x7f6d8c0ac590 msgr2=0x7f6d8c0aefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 shutdown_connections 2024-03-06T07:26:17.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.879+0000 7f6d90a78640 1 -- 172.21.15.94:0/381398942 wait complete. 2024-03-06T07:26:17.892 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-03-06T07:26:17.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.990+0000 7f8cb21c7640 1 Processor -- start 2024-03-06T07:26:17.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.990+0000 7f8cb21c7640 1 -- start start 2024-03-06T07:26:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8cb21c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 0x7f8cac156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8cb21c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac153a60 con 0x7f8cac150580 2024-03-06T07:26:17.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8cb21c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac153d40 con 0x7f8cac0c1320 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8cb21c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06c2c0 con 0x7f8cac0c6000 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8cab7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8cac150580 0x7f8cac06dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:39336/0 (socket says 172.21.15.94:39336) 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8cab7fe640 1 -- 172.21.15.94:0/2483528812 learned_addr learned my addr 172.21.15.94:0/2483528812 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8cabfff640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 0x7f8cac156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589397058 0 0) 0x7f8cac153a60 con 0x7f8cac150580 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c94003520 con 0x7f8cac150580 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751121292 0 0) 0x7f8c94003520 con 0x7f8cac150580 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 msgr2=0x7f8cac156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 0x7f8cac156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 >> v1:172.21.15.145:6789/0 conn(0x7f8cac0c1320 legacy=0x7f8cac1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac06c5a0 con 0x7f8cac150580 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f8c9c001670 con 0x7f8cac150580 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 >> v1:172.21.15.94:6789/0 conn(0x7f8cac150580 legacy=0x7f8cac06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c940039e0 0x7f8c94003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c94004800 0x7f8c94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 0x7f8c94005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06c2c0 con 0x7f8c940039e0 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9400baf0 con 0x7f8c94004800 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.991+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9400bcf0 con 0x7f8cac0c6000 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cab7fe640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c940039e0 0x7f8c94003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8caaffd640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c94004800 0x7f8c94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cabfff640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 0x7f8c94005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cab7fe640 1 -- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 msgr2=0x7f8c94005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cab7fe640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0c6000 0x7f8c94005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cab7fe640 1 -- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c94004800 msgr2=0x7f8c94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cab7fe640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c94004800 0x7f8c94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cab7fe640 1 -- 172.21.15.94:0/2483528812 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94003520 con 0x7f8c940039e0 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.992+0000 7f8cab7fe640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c940039e0 0x7f8c94003e00 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c240 tx=0x7f8c94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48af4470b2b0ab2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c007cf0 con 0x7f8c940039e0 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c030070 con 0x7f8c940039e0 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8caa7fc640 1 -- 172.21.15.94:0/2483528812 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c005d30 con 0x7f8c940039e0 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8cb21c7640 1 -- 172.21.15.94:0/2483528812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c940039e0 msgr2=0x7f8c94003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8cb21c7640 1 --2- 172.21.15.94:0/2483528812 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c940039e0 0x7f8c94003e00 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c240 tx=0x7f8c94005830 comp rx=0 tx=0).stop 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8cabfff640 1 -- 172.21.15.94:0/2483528812 reap_dead start 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8cb21c7640 1 -- 172.21.15.94:0/2483528812 shutdown_connections 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8cb21c7640 1 -- 172.21.15.94:0/2483528812 >> 172.21.15.94:0/2483528812 conn(0x7f8cac0c6710 msgr2=0x7f8cac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8cb21c7640 1 -- 172.21.15.94:0/2483528812 shutdown_connections 2024-03-06T07:26:17.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.993+0000 7f8cb21c7640 1 -- 172.21.15.94:0/2483528812 wait complete. 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.994+0000 7f8cb21c7640 1 Processor -- start 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.994+0000 7f8cb21c7640 1 -- start start 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cb21c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c940039e0 0x7f8cac1e9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cb21c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cac070560 0x7f8cac14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cb21c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0bf9a0 0x7f8cac0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cb21c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06e5a0 con 0x7f8cac070560 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cb21c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06e420 con 0x7f8c940039e0 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cb21c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac0c0c40 con 0x7f8cac0bf9a0 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cac070560 0x7f8cac14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cac070560 0x7f8cac14b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34944/0 (socket says 172.21.15.94:34944) 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 -- 172.21.15.94:0/4091154225 learned_addr learned my addr 172.21.15.94:0/4091154225 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cabfff640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0bf9a0 0x7f8cac0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8cab7fe640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c940039e0 0x7f8cac1e9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 -- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0bf9a0 msgr2=0x7f8cac0bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0bf9a0 0x7f8cac0bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 -- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c940039e0 msgr2=0x7f8cac1e9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c940039e0 0x7f8cac1e9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.995+0000 7f8caaffd640 1 -- 172.21.15.94:0/4091154225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac14c740 con 0x7f8cac070560 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.996+0000 7f8caaffd640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cac070560 0x7f8cac14b550 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000f650 tx=0x7f8ca000fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.996+0000 7f8c93fff640 1 -- 172.21.15.94:0/4091154225 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca00050d0 con 0x7f8cac070560 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.996+0000 7f8c93fff640 1 -- 172.21.15.94:0/4091154225 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca0018070 con 0x7f8cac070560 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.996+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac0c0360 con 0x7f8cac070560 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.996+0000 7f8c93fff640 1 -- 172.21.15.94:0/4091154225 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0005490 con 0x7f8cac070560 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.996+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac1e9490 con 0x7f8cac070560 2024-03-06T07:26:18.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.997+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac04f330 con 0x7f8cac070560 2024-03-06T07:26:18.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.997+0000 7f8c93fff640 1 -- 172.21.15.94:0/4091154225 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ca000d730 con 0x7f8cac070560 2024-03-06T07:26:18.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.998+0000 7f8c93fff640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c9806b4d0 0x7f8c9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.998+0000 7f8c93fff640 1 -- 172.21.15.94:0/4091154225 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f8ca0089f30 con 0x7f8cac070560 2024-03-06T07:26:18.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.998+0000 7f8cab7fe640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c9806b4d0 0x7f8c9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:17.998+0000 7f8cab7fe640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c9806b4d0 0x7f8c9806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c034000 tx=0x7f8c9c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.002+0000 7f8c93fff640 1 -- 172.21.15.94:0/4091154225 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ca0058ed0 con 0x7f8cac070560 2024-03-06T07:26:18.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.244+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7f8cac0c5de0 con 0x7f8cac070560 2024-03-06T07:26:18.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.559+0000 7f8c93fff640 1 -- 172.21.15.94:0/4091154225 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v462) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8ca00587d0 con 0x7f8cac070560 2024-03-06T07:26:18.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c9806b4d0 msgr2=0x7f8c9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c9806b4d0 0x7f8c9806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c034000 tx=0x7f8c9c007000 comp rx=0 tx=0).stop 2024-03-06T07:26:18.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cac070560 msgr2=0x7f8cac14b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cac070560 0x7f8cac14b550 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000f650 tx=0x7f8ca000fb10 comp rx=0 tx=0).stop 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 shutdown_connections 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c9806b4d0 0x7f8c9806d990 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8cac0bf9a0 0x7f8cac0bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8cac070560 0x7f8cac14b550 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 --2- 172.21.15.94:0/4091154225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c940039e0 0x7f8cac1e9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 >> 172.21.15.94:0/4091154225 conn(0x7f8cac0c6710 msgr2=0x7f8cac14d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 shutdown_connections 2024-03-06T07:26:18.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.561+0000 7f8cb21c7640 1 -- 172.21.15.94:0/4091154225 wait complete. 2024-03-06T07:26:18.574 INFO:tasks.cephfs_test_runner:That setting max_mds undoes down. ... ok 2024-03-06T07:26:18.575 INFO:tasks.cephfs_test_runner:Starting test: test_down_health (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down does not generate a health warning 2024-03-06T07:26:18.575 INFO:tasks.cephfs_test_runner:test_down_health (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:26:18.575 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:18.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.674+0000 7f462735c640 1 Processor -- start 2024-03-06T07:26:18.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f462735c640 1 -- start start 2024-03-06T07:26:18.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f462735c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f46200c88c0 0x7f4620155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f462735c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4620158b60 0x7f4620158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f462735c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46200c73d0 con 0x7f462014f360 2024-03-06T07:26:18.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f462735c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46200c8dd0 con 0x7f4620158b60 2024-03-06T07:26:18.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f462735c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46200be2e0 con 0x7f46200c88c0 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f46258f9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4620158b60 0x7f4620158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.675+0000 7f46250f8640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f462014f360 0x7f46200c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38674/0 (socket says 172.21.15.94:38674) 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f46250f8640 1 -- 172.21.15.94:0/462928942 learned_addr learned my addr 172.21.15.94:0/462928942 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f46248f7640 1 --2- 172.21.15.94:0/462928942 >> v2:172.21.15.145:3300/0 conn(0x7f46200c88c0 0x7f4620155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f46258f9640 1 -- 172.21.15.94:0/462928942 >> v2:172.21.15.145:3300/0 conn(0x7f46200c88c0 msgr2=0x7f4620155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f46258f9640 1 --2- 172.21.15.94:0/462928942 >> v2:172.21.15.145:3300/0 conn(0x7f46200c88c0 0x7f4620155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f46258f9640 1 -- 172.21.15.94:0/462928942 >> v1:172.21.15.94:6789/0 conn(0x7f462014f360 legacy=0x7f46200c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f46258f9640 1 -- 172.21.15.94:0/462928942 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46200c7670 con 0x7f4620158b60 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f4617fff640 1 -- 172.21.15.94:0/462928942 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771666658 0 0) 0x7f46200c73d0 con 0x7f462014f360 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.676+0000 7f46248f7640 1 --2- 172.21.15.94:0/462928942 >> v2:172.21.15.145:3300/0 conn(0x7f46200c88c0 0x7f4620155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f46258f9640 1 --2- 172.21.15.94:0/462928942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f4620158ff0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f46200c84c0 tx=0x7f461802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f8b6ca176fe121 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f4617fff640 1 -- 172.21.15.94:0/462928942 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618039070 con 0x7f4620158b60 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f4617fff640 1 -- 172.21.15.94:0/462928942 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461802cbf0 con 0x7f4620158b60 2024-03-06T07:26:18.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f4617fff640 1 -- 172.21.15.94:0/462928942 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618034440 con 0x7f4620158b60 2024-03-06T07:26:18.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 -- 172.21.15.94:0/462928942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 msgr2=0x7f4620158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 --2- 172.21.15.94:0/462928942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f4620158ff0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f46200c84c0 tx=0x7f461802be40 comp rx=0 tx=0).stop 2024-03-06T07:26:18.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 -- 172.21.15.94:0/462928942 shutdown_connections 2024-03-06T07:26:18.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 --2- 172.21.15.94:0/462928942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f4620158ff0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 --2- 172.21.15.94:0/462928942 >> v2:172.21.15.145:3300/0 conn(0x7f46200c88c0 0x7f4620155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 -- 172.21.15.94:0/462928942 >> 172.21.15.94:0/462928942 conn(0x7f46200ac570 msgr2=0x7f46200ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 -- 172.21.15.94:0/462928942 shutdown_connections 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.677+0000 7f462735c640 1 -- 172.21.15.94:0/462928942 wait complete. 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.678+0000 7f462735c640 1 Processor -- start 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.678+0000 7f462735c640 1 -- start start 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f462735c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46200c88c0 0x7f4620157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f462735c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f462014f360 0x7f46200aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f462735c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f46201e7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f462735c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46200c5a50 con 0x7f4620158b60 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f462735c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46200c7c30 con 0x7f46200c88c0 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f462735c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46200c5bd0 con 0x7f462014f360 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f46201e7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f46201e7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47910/0 (socket says 172.21.15.94:47910) 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 -- 172.21.15.94:0/2338044887 learned_addr learned my addr 172.21.15.94:0/2338044887 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:18.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 -- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f462014f360 msgr2=0x7f46200aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f462014f360 0x7f46200aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46250f8640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46200c88c0 0x7f4620157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 -- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46200c88c0 msgr2=0x7f4620157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46200c88c0 0x7f4620157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.679+0000 7f46258f9640 1 -- 172.21.15.94:0/2338044887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4620155390 con 0x7f4620158b60 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.680+0000 7f46258f9640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f46201e7bb0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f461802c450 tx=0x7f4618004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.680+0000 7f4615ffb640 1 -- 172.21.15.94:0/2338044887 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618039050 con 0x7f4620158b60 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.680+0000 7f4615ffb640 1 -- 172.21.15.94:0/2338044887 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4618045070 con 0x7f4620158b60 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.680+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46201ea920 con 0x7f4620158b60 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.680+0000 7f4615ffb640 1 -- 172.21.15.94:0/2338044887 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461803e830 con 0x7f4620158b60 2024-03-06T07:26:18.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.680+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46201eaea0 con 0x7f4620158b60 2024-03-06T07:26:18.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.681+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462004f330 con 0x7f4620158b60 2024-03-06T07:26:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.681+0000 7f4615ffb640 1 -- 172.21.15.94:0/2338044887 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f461803e9d0 con 0x7f4620158b60 2024-03-06T07:26:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.682+0000 7f4615ffb640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f460406b4d0 0x7f460406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.682+0000 7f4615ffb640 1 -- 172.21.15.94:0/2338044887 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f46180ace70 con 0x7f4620158b60 2024-03-06T07:26:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.682+0000 7f46250f8640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f460406b4d0 0x7f460406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.683+0000 7f46250f8640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f460406b4d0 0x7f460406d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4610007fd0 tx=0x7f4610009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.686+0000 7f4615ffb640 1 -- 172.21.15.94:0/2338044887 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f461807bea0 con 0x7f4620158b60 2024-03-06T07:26:18.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.858+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46200c1520 con 0x7f4620158b60 2024-03-06T07:26:18.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.859+0000 7f4615ffb640 1 -- 172.21.15.94:0/2338044887 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+13433 (secure 0 0 0) 0x7f461807b7a0 con 0x7f4620158b60 2024-03-06T07:26:18.864 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:18.865 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":187,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:17.554904+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/2744471722":"2024-03-07T07:26:14.547379+0000","172.21.15.94:6833/2744471722":"2024-03-07T07:26:14.547379+0000","172.21.15.94:6834/4225816089":"2024-03-07T07:26:14.547392+0000","172.21.15.94:6832/4225816089":"2024-03-07T07:26:14.547392+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:18.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f460406b4d0 msgr2=0x7f460406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f460406b4d0 0x7f460406d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4610007fd0 tx=0x7f4610009040 comp rx=0 tx=0).stop 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 msgr2=0x7f46201e7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f46201e7bb0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f461802c450 tx=0x7f4618004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 shutdown_connections 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f460406b4d0 0x7f460406d990 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4620158b60 0x7f46201e7bb0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f462014f360 0x7f46200aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 --2- 172.21.15.94:0/2338044887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46200c88c0 0x7f4620157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 >> 172.21.15.94:0/2338044887 conn(0x7f46200ac570 msgr2=0x7f46200c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 shutdown_connections 2024-03-06T07:26:18.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.861+0000 7f462735c640 1 -- 172.21.15.94:0/2338044887 wait complete. 2024-03-06T07:26:18.874 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:18.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.975+0000 7fcba92eb640 1 Processor -- start 2024-03-06T07:26:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.975+0000 7fcba92eb640 1 -- start start 2024-03-06T07:26:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba92eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba92eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcba40c69a0 0x7fcba4072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba92eb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba40c62a0 con 0x7fcba40c4740 2024-03-06T07:26:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba92eb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba4153f00 con 0x7fcba40c69a0 2024-03-06T07:26:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba92eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40c4c00 con 0x7fcba40a8290 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcba40c69a0 0x7fcba4072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcba40c69a0 0x7fcba4072680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47924/0 (socket says 172.21.15.94:47924) 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba37fe640 1 -- 172.21.15.94:0/167066401 learned_addr learned my addr 172.21.15.94:0/167066401 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.976+0000 7fcba2ffd640 1 --2- 172.21.15.94:0/167066401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.977+0000 7fcba37fe640 1 -- 172.21.15.94:0/167066401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 msgr2=0x7fcba41539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.977+0000 7fcba37fe640 1 --2- 172.21.15.94:0/167066401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.977+0000 7fcba37fe640 1 -- 172.21.15.94:0/167066401 >> v1:172.21.15.145:6789/0 conn(0x7fcba40c4740 legacy=0x7fcba40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.977+0000 7fcba37fe640 1 -- 172.21.15.94:0/167066401 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba40c4ee0 con 0x7fcba40c69a0 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.977+0000 7fcba37fe640 1 --2- 172.21.15.94:0/167066401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba4072680 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009e50 tx=0x7fcb9c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef36fcba493c44ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.977+0000 7fcba27fc640 1 -- 172.21.15.94:0/167066401 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c02c9e0 con 0x7fcba40c69a0 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba27fc640 1 -- 172.21.15.94:0/167066401 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c02cb80 con 0x7fcba40c69a0 2024-03-06T07:26:18.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba27fc640 1 -- 172.21.15.94:0/167066401 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c0343f0 con 0x7fcba40c69a0 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 -- 172.21.15.94:0/167066401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 msgr2=0x7fcba4072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 --2- 172.21.15.94:0/167066401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba4072680 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009e50 tx=0x7fcb9c02be40 comp rx=0 tx=0).stop 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 -- 172.21.15.94:0/167066401 shutdown_connections 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 --2- 172.21.15.94:0/167066401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba4072680 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 --2- 172.21.15.94:0/167066401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 -- 172.21.15.94:0/167066401 >> 172.21.15.94:0/167066401 conn(0x7fcba40c5560 msgr2=0x7fcba40c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 -- 172.21.15.94:0/167066401 shutdown_connections 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.978+0000 7fcba92eb640 1 -- 172.21.15.94:0/167066401 wait complete. 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.979+0000 7fcba92eb640 1 Processor -- start 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.979+0000 7fcba92eb640 1 -- start start 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba92eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41ec170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba92eb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba40c4740 0x7fcba41ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba92eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba41ea300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba92eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba40c6e50 con 0x7fcba40c69a0 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba92eb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba406fc80 con 0x7fcba40c4740 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba92eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba4072bc0 con 0x7fcba40a8290 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba41ea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba41ea300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47936/0 (socket says 172.21.15.94:47936) 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 -- 172.21.15.94:0/279728061 learned_addr learned my addr 172.21.15.94:0/279728061 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba2ffd640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcb9a7fc640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba40c4740 0x7fcba41ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 -- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 msgr2=0x7fcba41ec170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41ec170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 -- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba40c4740 msgr2=0x7fcba41ec6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba40c4740 0x7fcba41ec6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.980+0000 7fcba37fe640 1 -- 172.21.15.94:0/279728061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba40be250 con 0x7fcba40c69a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.981+0000 7fcba37fe640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba41ea300 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009f70 tx=0x7fcb9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.981+0000 7fcb9bfff640 1 -- 172.21.15.94:0/279728061 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c043020 con 0x7fcba40c69a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.981+0000 7fcb9bfff640 1 -- 172.21.15.94:0/279728061 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c03e450 con 0x7fcba40c69a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.981+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba41ecda0 con 0x7fcba40c69a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.981+0000 7fcb9bfff640 1 -- 172.21.15.94:0/279728061 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c009510 con 0x7fcba40c69a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.981+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba40ab8f0 con 0x7fcba40c69a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.982+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba41e9fe0 con 0x7fcba40c69a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.982+0000 7fcb9bfff640 1 -- 172.21.15.94:0/279728061 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb9c046020 con 0x7fcba40c69a0 2024-03-06T07:26:18.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.982+0000 7fcb9bfff640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8406b500 0x7fcb8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.983+0000 7fcb9bfff640 1 -- 172.21.15.94:0/279728061 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fcb9c0ac6f0 con 0x7fcba40c69a0 2024-03-06T07:26:18.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.983+0000 7fcba2ffd640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8406b500 0x7fcb8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.983+0000 7fcba2ffd640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8406b500 0x7fcb8406d9c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009b40 tx=0x7fcb94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:18.987+0000 7fcb9bfff640 1 -- 172.21.15.94:0/279728061 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcba41e9fe0 con 0x7fcba40c69a0 2024-03-06T07:26:19.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.156+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcba404f330 con 0x7fcba40c69a0 2024-03-06T07:26:19.158 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.157+0000 7fcb9bfff640 1 -- 172.21.15.94:0/279728061 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v187) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcb9c04d030 con 0x7fcba40c69a0 2024-03-06T07:26:19.159 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:26:19.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8406b500 msgr2=0x7fcb8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8406b500 0x7fcb8406d9c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009b40 tx=0x7fcb94009190 comp rx=0 tx=0).stop 2024-03-06T07:26:19.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 msgr2=0x7fcba41ea300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba41ea300 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009f70 tx=0x7fcb9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 shutdown_connections 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8406b500 0x7fcb8406d9c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcba40c69a0 0x7fcba41ea300 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba40c4740 0x7fcba41ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 --2- 172.21.15.94:0/279728061 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba40a8290 0x7fcba41ec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 >> 172.21.15.94:0/279728061 conn(0x7fcba40c5560 msgr2=0x7fcba40c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 shutdown_connections 2024-03-06T07:26:19.161 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.158+0000 7fcba92eb640 1 -- 172.21.15.94:0/279728061 wait complete. 2024-03-06T07:26:19.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.270+0000 7f85c2888640 1 Processor -- start 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.270+0000 7f85c2888640 1 -- start start 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.270+0000 7f85c2888640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f85bc156150 0x7f85bc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.270+0000 7f85c2888640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc0c28d0 con 0x7f85bc14f790 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.270+0000 7f85c2888640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc0c2bb0 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.270+0000 7f85c2888640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc0c2680 con 0x7f85bc156150 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85c0e25640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f85bc156150 0x7f85bc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85c0e25640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f85bc156150 0x7f85bc1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47952/0 (socket says 172.21.15.94:47952) 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85bbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f85bc14f790 0x7f85bc0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38688/0 (socket says 172.21.15.94:38688) 2024-03-06T07:26:19.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85c0e25640 1 -- 172.21.15.94:0/3669664519 learned_addr learned my addr 172.21.15.94:0/3669664519 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85c0e25640 1 -- 172.21.15.94:0/3669664519 >> v1:172.21.15.145:6789/0 conn(0x7f85bc0c1aa0 legacy=0x7f85bc155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85c0e25640 1 -- 172.21.15.94:0/3669664519 >> v1:172.21.15.94:6789/0 conn(0x7f85bc14f790 legacy=0x7f85bc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85c0e25640 1 -- 172.21.15.94:0/3669664519 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc0c2e90 con 0x7f85bc156150 2024-03-06T07:26:19.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.271+0000 7f85baffd640 1 -- 172.21.15.94:0/3669664519 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804830125 0 0) 0x7f85bc0c28d0 con 0x7f85bc14f790 2024-03-06T07:26:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c0e25640 1 --2- 172.21.15.94:0/3669664519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc156150 0x7f85bc1529d0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0c2800 tx=0x7f85b402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236f3f2c2b7df85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85baffd640 1 -- 172.21.15.94:0/3669664519 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b402caf0 con 0x7f85bc156150 2024-03-06T07:26:19.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85baffd640 1 -- 172.21.15.94:0/3669664519 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b402cc90 con 0x7f85bc156150 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85baffd640 1 -- 172.21.15.94:0/3669664519 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4034610 con 0x7f85bc156150 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c2888640 1 -- 172.21.15.94:0/3669664519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc156150 msgr2=0x7f85bc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c2888640 1 --2- 172.21.15.94:0/3669664519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc156150 0x7f85bc1529d0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0c2800 tx=0x7f85b402bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c2888640 1 -- 172.21.15.94:0/3669664519 shutdown_connections 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c2888640 1 --2- 172.21.15.94:0/3669664519 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc156150 0x7f85bc1529d0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c2888640 1 -- 172.21.15.94:0/3669664519 >> 172.21.15.94:0/3669664519 conn(0x7f85bc0c6710 msgr2=0x7f85bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c2888640 1 -- 172.21.15.94:0/3669664519 shutdown_connections 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.272+0000 7f85c2888640 1 -- 172.21.15.94:0/3669664519 wait complete. 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.273+0000 7f85c2888640 1 Processor -- start 2024-03-06T07:26:19.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85c2888640 1 -- start start 2024-03-06T07:26:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85c2888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc0c1aa0 0x7f85bc0a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85c2888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85bc14f790 0x7f85bc1ef060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85c2888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85bc156150 0x7f85bc1ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85c2888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0c5e30 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85c2888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0c0d60 con 0x7f85bc156150 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85c2888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0c5fb0 con 0x7f85bc14f790 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc0c1aa0 0x7f85bc0a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc0c1aa0 0x7f85bc0a75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47956/0 (socket says 172.21.15.94:47956) 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.274+0000 7f85bbfff640 1 -- 172.21.15.94:0/2061285088 learned_addr learned my addr 172.21.15.94:0/2061285088 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85c0e25640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85bc156150 0x7f85bc1ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bbfff640 1 -- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85bc14f790 msgr2=0x7f85bc1ef060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bb7fe640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85bc14f790 0x7f85bc1ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bbfff640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85bc14f790 0x7f85bc1ef060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bbfff640 1 -- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85bc156150 msgr2=0x7f85bc1ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bbfff640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85bc156150 0x7f85bc1ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bbfff640 1 -- 172.21.15.94:0/2061285088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc0c04d0 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bb7fe640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85bc14f790 0x7f85bc1ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85c0e25640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85bc156150 0x7f85bc1ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85bbfff640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc0c1aa0 0x7f85bc0a75f0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c4d0 tx=0x7f85ac00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85b8ff9640 1 -- 172.21.15.94:0/2061285088 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac018070 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85b8ff9640 1 -- 172.21.15.94:0/2061285088 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac007500 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85b8ff9640 1 -- 172.21.15.94:0/2061285088 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac004a20 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc0c07b0 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.275+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc0a7b30 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.276+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc04f330 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.276+0000 7f85b8ff9640 1 -- 172.21.15.94:0/2061285088 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85ac01b070 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.277+0000 7f85b8ff9640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f85a806b4d0 0x7f85a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.277+0000 7f85b8ff9640 1 -- 172.21.15.94:0/2061285088 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f85ac089440 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.277+0000 7f85bb7fe640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f85a806b4d0 0x7f85a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.278+0000 7f85bb7fe640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f85a806b4d0 0x7f85a806d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0a9b30 tx=0x7f85b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.281+0000 7f85b8ff9640 1 -- 172.21.15.94:0/2061285088 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85ac0583e0 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.521+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7f85bc0c5c10 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.563+0000 7f85b8ff9640 1 -- 172.21.15.94:0/2061285088 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v463) v1 ==== 143+0+0 (secure 0 0 0) 0x7f85ac057ce0 con 0x7f85bc0c1aa0 2024-03-06T07:26:19.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f85a806b4d0 msgr2=0x7f85a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f85a806b4d0 0x7f85a806d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f85bc0a9b30 tx=0x7f85b0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc0c1aa0 msgr2=0x7f85bc0a75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc0c1aa0 0x7f85bc0a75f0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00c4d0 tx=0x7f85ac00c990 comp rx=0 tx=0).stop 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 shutdown_connections 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f85a806b4d0 0x7f85a806d990 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85bc156150 0x7f85bc1ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85bc14f790 0x7f85bc1ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 --2- 172.21.15.94:0/2061285088 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f85bc0c1aa0 0x7f85bc0a75f0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.566+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 >> 172.21.15.94:0/2061285088 conn(0x7f85bc0c6710 msgr2=0x7f85bc06cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.567+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 shutdown_connections 2024-03-06T07:26:19.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.567+0000 7f85c2888640 1 -- 172.21.15.94:0/2061285088 wait complete. 2024-03-06T07:26:19.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.680+0000 7f4a77b3e640 1 Processor -- start 2024-03-06T07:26:19.682 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.680+0000 7f4a77b3e640 1 -- start start 2024-03-06T07:26:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a77b3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 0x7f4a701494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a77b3e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4a700a97c0 0x7f4a701499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a77b3e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a70154330 con 0x7f4a700ab310 2024-03-06T07:26:19.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a77b3e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a700c46a0 con 0x7f4a700a97c0 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a77b3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70153dc0 con 0x7f4a701538c0 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a758da640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4a700ab310 0x7f4a700ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38696/0 (socket says 172.21.15.94:38696) 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a758da640 1 -- 172.21.15.94:0/4249498385 learned_addr learned my addr 172.21.15.94:0/4249498385 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a750d9640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 0x7f4a701494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a760db640 1 --2- 172.21.15.94:0/4249498385 >> v2:172.21.15.145:3300/0 conn(0x7f4a700a97c0 0x7f4a701499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198138381 0 0) 0x7f4a70154330 con 0x7f4a700ab310 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.681+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a5c003520 con 0x7f4a700ab310 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3907916722 0 0) 0x7f4a5c003520 con 0x7f4a700ab310 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 msgr2=0x7f4a701494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 0x7f4a701494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 >> v2:172.21.15.145:3300/0 conn(0x7f4a700a97c0 msgr2=0x7f4a701499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 --2- 172.21.15.94:0/4249498385 >> v2:172.21.15.145:3300/0 conn(0x7f4a700a97c0 0x7f4a701499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700c5d30 con 0x7f4a700ab310 2024-03-06T07:26:19.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f4a6c001670 con 0x7f4a700ab310 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 >> v1:172.21.15.94:6789/0 conn(0x7f4a700ab310 legacy=0x7f4a700ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a5c0039b0 0x7f4a5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 0x7f4a5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700c46a0 con 0x7f4a5c004800 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70153dc0 con 0x7f4a5c0039b0 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c00b9f0 con 0x7f4a701538c0 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a750d9640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a758da640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a5c0039b0 0x7f4a5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a760db640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 0x7f4a5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a750d9640 1 -- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 msgr2=0x7f4a5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a750d9640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701538c0 0x7f4a5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.687 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a750d9640 1 -- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a5c0039b0 msgr2=0x7f4a5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a750d9640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a5c0039b0 0x7f4a5c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.682+0000 7f4a750d9640 1 -- 172.21.15.94:0/4249498385 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c003520 con 0x7f4a5c004800 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a750d9640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a5c004c80 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009ac0 tx=0x7f4a5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461c3678b408b6d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60039070 con 0x7f4a5c004800 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6002c6b0 con 0x7f4a5c004800 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a748d8640 1 -- 172.21.15.94:0/4249498385 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60009690 con 0x7f4a5c004800 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a77b3e640 1 -- 172.21.15.94:0/4249498385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 msgr2=0x7f4a5c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a77b3e640 1 --2- 172.21.15.94:0/4249498385 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a5c004c80 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009ac0 tx=0x7f4a5c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a760db640 1 -- 172.21.15.94:0/4249498385 reap_dead start 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a77b3e640 1 -- 172.21.15.94:0/4249498385 shutdown_connections 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.683+0000 7f4a77b3e640 1 -- 172.21.15.94:0/4249498385 >> 172.21.15.94:0/4249498385 conn(0x7f4a700c6710 msgr2=0x7f4a700c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.684+0000 7f4a77b3e640 1 -- 172.21.15.94:0/4249498385 shutdown_connections 2024-03-06T07:26:19.688 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.684+0000 7f4a77b3e640 1 -- 172.21.15.94:0/4249498385 wait complete. 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 Processor -- start 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 -- start start 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a701eedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a7006cce0 0x7f4a701eb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a70154530 0x7f4a700c3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a701494a0 con 0x7f4a5c004800 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700c62f0 con 0x7f4a7006cce0 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a77b3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70149620 con 0x7f4a70154530 2024-03-06T07:26:19.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a758da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a701eedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a758da640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a701eedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47968/0 (socket says 172.21.15.94:47968) 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.685+0000 7f4a758da640 1 -- 172.21.15.94:0/211274574 learned_addr learned my addr 172.21.15.94:0/211274574 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a760db640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a70154530 0x7f4a700c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a758da640 1 -- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a70154530 msgr2=0x7f4a700c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a750d9640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a7006cce0 0x7f4a701eb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a758da640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a70154530 0x7f4a700c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a758da640 1 -- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a7006cce0 msgr2=0x7f4a701eb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a758da640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a7006cce0 0x7f4a701eb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a758da640 1 -- 172.21.15.94:0/211274574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7006d4a0 con 0x7f4a5c004800 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a760db640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a70154530 0x7f4a700c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a758da640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a701eedc0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00e870 tx=0x7f4a6c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a667fc640 1 -- 172.21.15.94:0/211274574 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c00ee60 con 0x7f4a5c004800 2024-03-06T07:26:19.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a667fc640 1 -- 172.21.15.94:0/211274574 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c018070 con 0x7f4a5c004800 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a701ebd90 con 0x7f4a5c004800 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a667fc640 1 -- 172.21.15.94:0/211274574 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c00ca90 con 0x7f4a5c004800 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.686+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7006e080 con 0x7f4a5c004800 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.687+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7004f330 con 0x7f4a5c004800 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.687+0000 7f4a667fc640 1 -- 172.21.15.94:0/211274574 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a6c00cc30 con 0x7f4a5c004800 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.688+0000 7f4a667fc640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5406b500 0x7f4a5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.688+0000 7f4a667fc640 1 -- 172.21.15.94:0/211274574 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4a6c089b10 con 0x7f4a5c004800 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.688+0000 7f4a750d9640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5406b500 0x7f4a5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.688+0000 7f4a750d9640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5406b500 0x7f4a5406d9c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036000 tx=0x7f4a60030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.692+0000 7f4a667fc640 1 -- 172.21.15.94:0/211274574 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a6c058b40 con 0x7f4a5c004800 2024-03-06T07:26:19.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.861+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4a700c5a70 con 0x7f4a5c004800 2024-03-06T07:26:19.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.864+0000 7f4a667fc640 1 -- 172.21.15.94:0/211274574 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v61)=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4a6c058440 con 0x7f4a5c004800 2024-03-06T07:26:19.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5406b500 msgr2=0x7f4a5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5406b500 0x7f4a5406d9c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4a60036000 tx=0x7f4a60030000 comp rx=0 tx=0).stop 2024-03-06T07:26:19.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 msgr2=0x7f4a701eedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a701eedc0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00e870 tx=0x7f4a6c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 shutdown_connections 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5406b500 0x7f4a5406d9c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a70154530 0x7f4a700c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a7006cce0 0x7f4a701eb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 --2- 172.21.15.94:0/211274574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a5c004800 0x7f4a701eedc0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 >> 172.21.15.94:0/211274574 conn(0x7f4a700c6710 msgr2=0x7f4a700a62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 shutdown_connections 2024-03-06T07:26:19.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:19.867+0000 7f4a77b3e640 1 -- 172.21.15.94:0/211274574 wait complete. 2024-03-06T07:26:19.879 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:19.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:19.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:19.982 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:19.982 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:26:19.982 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:19.997 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:26:19.997 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:26:19.998 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:20.015 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:26:20.015 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:26:20.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:20.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.135+0000 7f1c011a0640 1 Processor -- start 2024-03-06T07:26:20.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.135+0000 7f1c011a0640 1 -- start start 2024-03-06T07:26:20.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c011a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc0c2670 0x7f1bfc0c2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c011a0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1bfc06e810 0x7f1bfc06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c011a0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bfc0717f0 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c011a0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc14f7c0 con 0x7f1bfc06e810 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c011a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc071ad0 con 0x7f1bfc0c2670 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c0099f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1bfc06e810 0x7f1bfc06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c0099f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1bfc06e810 0x7f1bfc06eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47976/0 (socket says 172.21.15.94:47976) 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.136+0000 7f1c0099f640 1 -- 172.21.15.94:0/4204437065 learned_addr learned my addr 172.21.15.94:0/4204437065 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.137+0000 7f1bfb7fe640 1 --2- 172.21.15.94:0/4204437065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc0c2670 0x7f1bfc0c2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.137+0000 7f1c0099f640 1 -- 172.21.15.94:0/4204437065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc0c2670 msgr2=0x7f1bfc0c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.137+0000 7f1c0099f640 1 --2- 172.21.15.94:0/4204437065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc0c2670 0x7f1bfc0c2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.137+0000 7f1c0099f640 1 -- 172.21.15.94:0/4204437065 >> v1:172.21.15.145:6789/0 conn(0x7f1bfc14f3e0 legacy=0x7f1bfc152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.140 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.137+0000 7f1c0099f640 1 -- 172.21.15.94:0/4204437065 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc071db0 con 0x7f1bfc06e810 2024-03-06T07:26:20.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.137+0000 7f1c0099f640 1 --2- 172.21.15.94:0/4204437065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc06e810 0x7f1bfc06eca0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400d150 tx=0x7f1bf402dfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e26ba5be890495b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.137+0000 7f1bfaffd640 1 -- 172.21.15.94:0/4204437065 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf402e950 con 0x7f1bfc06e810 2024-03-06T07:26:20.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1bfaffd640 1 -- 172.21.15.94:0/4204437065 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf402eaf0 con 0x7f1bfc06e810 2024-03-06T07:26:20.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1bfaffd640 1 -- 172.21.15.94:0/4204437065 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4009600 con 0x7f1bfc06e810 2024-03-06T07:26:20.141 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 -- 172.21.15.94:0/4204437065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc06e810 msgr2=0x7f1bfc06eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4204437065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc06e810 0x7f1bfc06eca0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400d150 tx=0x7f1bf402dfb0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 -- 172.21.15.94:0/4204437065 shutdown_connections 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4204437065 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc06e810 0x7f1bfc06eca0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4204437065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc0c2670 0x7f1bfc0c2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 -- 172.21.15.94:0/4204437065 >> 172.21.15.94:0/4204437065 conn(0x7f1bfc06c0d0 msgr2=0x7f1bfc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 -- 172.21.15.94:0/4204437065 shutdown_connections 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.138+0000 7f1c011a0640 1 -- 172.21.15.94:0/4204437065 wait complete. 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.139+0000 7f1c011a0640 1 Processor -- start 2024-03-06T07:26:20.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.139+0000 7f1c011a0640 1 -- start start 2024-03-06T07:26:20.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c011a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc06e810 0x7f1bfc0ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c011a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1bfc0c2670 0x7f1bfc1511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c011a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc14f3e0 0x7f1bfc0aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c011a0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc0c37d0 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c011a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc06f1e0 con 0x7f1bfc0c2670 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c011a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc071450 con 0x7f1bfc06e810 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc14f3e0 0x7f1bfc0aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc14f3e0 0x7f1bfc0aa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47988/0 (socket says 172.21.15.94:47988) 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 -- 172.21.15.94:0/4039951245 learned_addr learned my addr 172.21.15.94:0/4039951245 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1bfbfff640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc06e810 0x7f1bfc0ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1bfb7fe640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1bfc0c2670 0x7f1bfc1511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 -- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc06e810 msgr2=0x7f1bfc0ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc06e810 0x7f1bfc0ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 -- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1bfc0c2670 msgr2=0x7f1bfc1511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1bfc0c2670 0x7f1bfc1511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1c0099f640 1 -- 172.21.15.94:0/4039951245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc154da0 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.140+0000 7f1bfbfff640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc06e810 0x7f1bfc0ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.141+0000 7f1c0099f640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc14f3e0 0x7f1bfc0aa660 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400d270 tx=0x7f1bf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.141+0000 7f1bf8ff9640 1 -- 172.21.15.94:0/4039951245 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4045020 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.141+0000 7f1bf8ff9640 1 -- 172.21.15.94:0/4039951245 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf4040660 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.141+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc1550e0 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.141+0000 7f1bf8ff9640 1 -- 172.21.15.94:0/4039951245 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf402e950 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.141+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc155920 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.142+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc151750 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.142+0000 7f1bf8ff9640 1 -- 172.21.15.94:0/4039951245 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bf4048020 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.143+0000 7f1bf8ff9640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1be806b500 0x7f1be806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.143+0000 7f1bf8ff9640 1 -- 172.21.15.94:0/4039951245 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f1bf40aeba0 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.143+0000 7f1bfbfff640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1be806b500 0x7f1be806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.143+0000 7f1bfbfff640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1be806b500 0x7f1be806d9c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f1bec006fd0 tx=0x7f1bec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.147+0000 7f1bf8ff9640 1 -- 172.21.15.94:0/4039951245 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1bf404f030 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.355+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bfc069430 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.355+0000 7f1bf8ff9640 1 -- 172.21.15.94:0/4039951245 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+3712 (secure 0 0 0) 0x7f1bfc1550e0 con 0x7f1bfc14f3e0 2024-03-06T07:26:20.357 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:20.358 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":182,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9064,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3040651455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3040651455},{"type":"v1","addr":"172.21.15.145:6837","nonce":3040651455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":168},{"gid":9265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3186355167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3186355167},{"type":"v1","addr":"172.21.15.145:6836","nonce":3186355167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":175},{"gid":10347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/3863179237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":3863179237},{"type":"v1","addr":"172.21.15.94:6834","nonce":3863179237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182},{"gid":10350,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2795600004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":2795600004},{"type":"v1","addr":"172.21.15.94:6835","nonce":2795600004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":182}],"filesystems":[]} 2024-03-06T07:26:20.358 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 182 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.357+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1be806b500 msgr2=0x7f1be806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.357+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1be806b500 0x7f1be806d9c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f1bec006fd0 tx=0x7f1bec008040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.357+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc14f3e0 msgr2=0x7f1bfc0aa660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.357+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc14f3e0 0x7f1bfc0aa660 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400d270 tx=0x7f1bf4004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 shutdown_connections 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1be806b500 0x7f1be806d9c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bfc14f3e0 0x7f1bfc0aa660 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1bfc0c2670 0x7f1bfc1511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 --2- 172.21.15.94:0/4039951245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1bfc06e810 0x7f1bfc0ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 >> 172.21.15.94:0/4039951245 conn(0x7f1bfc06c0d0 msgr2=0x7f1bfc14cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 shutdown_connections 2024-03-06T07:26:20.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.358+0000 7f1c011a0640 1 -- 172.21.15.94:0/4039951245 wait complete. 2024-03-06T07:26:20.371 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:20.371 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:20.371 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:20.371 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:20.371 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:20.371 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:20.372 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:20.372 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:20.372 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:20.372 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:20.372 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:20.372 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:20.372 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:20.373 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:20.373 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:20.373 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:20.380 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:20.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:20.381 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:20.381 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:20.388 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:20.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:20.389 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:20.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:20.423 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:20.423 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:20.423 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:20.424 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:20.431 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:20.431 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:20.431 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:20.432 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:20.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:20.500 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:26:20.500 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:26:20.516 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:26:20.516 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.535+0000 7fd3447bf640 1 Processor -- start 2024-03-06T07:26:20.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.535+0000 7fd3447bf640 1 -- start start 2024-03-06T07:26:20.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd3447bf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd33c0c8950 0x7fd33c0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd3447bf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c1558d0 0x7fd33c155d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd3447bf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c155130 con 0x7fd33c0c69f0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd3447bf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c155440 con 0x7fd33c0c8950 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd3447bf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0ad480 con 0x7fd33c1558d0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd342d5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c1558d0 0x7fd33c155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd341d5a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd33c0c8950 0x7fd33c0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd342d5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c1558d0 0x7fd33c155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:42084/0 (socket says 172.21.15.94:42084) 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.536+0000 7fd342d5c640 1 -- 172.21.15.94:0/2189741973 learned_addr learned my addr 172.21.15.94:0/2189741973 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.537+0000 7fd341d5a640 1 -- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c1558d0 msgr2=0x7fd33c155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.537+0000 7fd341d5a640 1 --2- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c1558d0 0x7fd33c155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.537+0000 7fd341d5a640 1 -- 172.21.15.94:0/2189741973 >> v1:172.21.15.145:6789/0 conn(0x7fd33c0c69f0 legacy=0x7fd33c0acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.537+0000 7fd341d5a640 1 -- 172.21.15.94:0/2189741973 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0c5d50 con 0x7fd33c0c8950 2024-03-06T07:26:20.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.537+0000 7fd342d5c640 1 --2- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c1558d0 0x7fd33c155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.537+0000 7fd341d5a640 1 --2- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c0c8950 0x7fd33c0c6dd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0097a0 tx=0x7fd32c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f78851b04f450db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd341559640 1 -- 172.21.15.94:0/2189741973 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0315a0 con 0x7fd33c0c8950 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd341559640 1 -- 172.21.15.94:0/2189741973 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c037070 con 0x7fd33c0c8950 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd341559640 1 -- 172.21.15.94:0/2189741973 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c031960 con 0x7fd33c0c8950 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 -- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c0c8950 msgr2=0x7fd33c0c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 --2- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c0c8950 0x7fd33c0c6dd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0097a0 tx=0x7fd32c02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 -- 172.21.15.94:0/2189741973 shutdown_connections 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 --2- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c1558d0 0x7fd33c155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 --2- 172.21.15.94:0/2189741973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c0c8950 0x7fd33c0c6dd0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 -- 172.21.15.94:0/2189741973 >> 172.21.15.94:0/2189741973 conn(0x7fd33c0cae70 msgr2=0x7fd33c0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 -- 172.21.15.94:0/2189741973 shutdown_connections 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.538+0000 7fd3447bf640 1 -- 172.21.15.94:0/2189741973 wait complete. 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.539+0000 7fd3447bf640 1 Processor -- start 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.539+0000 7fd3447bf640 1 -- start start 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd3447bf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c0c69f0 0x7fd33c154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd3447bf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c0c8950 0x7fd33c1ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd3447bf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c1558d0 0x7fd33c1ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd3447bf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c152a20 con 0x7fd33c0c8950 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd3447bf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0c6310 con 0x7fd33c1558d0 2024-03-06T07:26:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd3447bf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c152ba0 con 0x7fd33c0c69f0 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c0c8950 0x7fd33c1ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c0c8950 0x7fd33c1ee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48046/0 (socket says 172.21.15.94:48046) 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 -- 172.21.15.94:0/3743956271 learned_addr learned my addr 172.21.15.94:0/3743956271 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd34255b640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c0c69f0 0x7fd33c154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd342d5c640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c1558d0 0x7fd33c1ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 -- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c0c69f0 msgr2=0x7fd33c154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c0c69f0 0x7fd33c154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 -- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c1558d0 msgr2=0x7fd33c1ee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c1558d0 0x7fd33c1ee7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.540+0000 7fd341d5a640 1 -- 172.21.15.94:0/3743956271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c158aa0 con 0x7fd33c0c8950 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.541+0000 7fd341d5a640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c0c8950 0x7fd33c1ee2b0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098c0 tx=0x7fd32c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.541+0000 7fd332ffd640 1 -- 172.21.15.94:0/3743956271 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c032630 con 0x7fd33c0c8950 2024-03-06T07:26:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.541+0000 7fd332ffd640 1 -- 172.21.15.94:0/3743956271 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c0327d0 con 0x7fd33c0c8950 2024-03-06T07:26:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.541+0000 7fd332ffd640 1 -- 172.21.15.94:0/3743956271 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c032bc0 con 0x7fd33c0c8950 2024-03-06T07:26:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.541+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c158d80 con 0x7fd33c0c8950 2024-03-06T07:26:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.541+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c154660 con 0x7fd33c0c8950 2024-03-06T07:26:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.542+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c04f330 con 0x7fd33c0c8950 2024-03-06T07:26:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.543+0000 7fd332ffd640 1 -- 172.21.15.94:0/3743956271 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd32c04d3f0 con 0x7fd33c0c8950 2024-03-06T07:26:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.543+0000 7fd332ffd640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd32006b500 0x7fd32006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.543+0000 7fd332ffd640 1 -- 172.21.15.94:0/3743956271 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd32c02f030 con 0x7fd33c0c8950 2024-03-06T07:26:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.543+0000 7fd34255b640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd32006b500 0x7fd32006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.544+0000 7fd34255b640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd32006b500 0x7fd32006d9c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fd338007fd0 tx=0x7fd338009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.547+0000 7fd332ffd640 1 -- 172.21.15.94:0/3743956271 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd32c040070 con 0x7fd33c0c8950 2024-03-06T07:26:20.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:20.739+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd33c0ca650 con 0x7fd33c0c8950 2024-03-06T07:26:21.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.581+0000 7fd332ffd640 1 -- 172.21.15.94:0/3743956271 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v192) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd32c07a460 con 0x7fd33c0c8950 2024-03-06T07:26:21.584 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:26:21.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd32006b500 msgr2=0x7fd32006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd32006b500 0x7fd32006d9c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fd338007fd0 tx=0x7fd338009040 comp rx=0 tx=0).stop 2024-03-06T07:26:21.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c0c8950 msgr2=0x7fd33c1ee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c0c8950 0x7fd33c1ee2b0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098c0 tx=0x7fd32c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 shutdown_connections 2024-03-06T07:26:21.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd32006b500 0x7fd32006d9c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd33c1558d0 0x7fd33c1ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd33c0c8950 0x7fd33c1ee2b0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 --2- 172.21.15.94:0/3743956271 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd33c0c69f0 0x7fd33c154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 >> 172.21.15.94:0/3743956271 conn(0x7fd33c0cae70 msgr2=0x7fd33c150af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 shutdown_connections 2024-03-06T07:26:21.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.585+0000 7fd3447bf640 1 -- 172.21.15.94:0/3743956271 wait complete. 2024-03-06T07:26:21.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.703+0000 7f9e1b15e640 1 Processor -- start 2024-03-06T07:26:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.703+0000 7f9e1b15e640 1 -- start start 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e1b15e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9e140c1aa0 0x7f9e141529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e1b15e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 0x7f9e14155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e1b15e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e141563d0 con 0x7f9e1414f790 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e1b15e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140c2680 con 0x7f9e140c1aa0 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e1b15e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e140c28d0 con 0x7f9e14155a40 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e18efa640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9e1414f790 0x7f9e140c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38722/0 (socket says 172.21.15.94:38722) 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e18efa640 1 -- 172.21.15.94:0/1829444909 learned_addr learned my addr 172.21.15.94:0/1829444909 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e13fff640 1 --2- 172.21.15.94:0/1829444909 >> v2:172.21.15.145:3300/0 conn(0x7f9e140c1aa0 0x7f9e141529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e196fb640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 0x7f9e14155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430683767 0 0) 0x7f9e141563d0 con 0x7f9e1414f790 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e04003520 con 0x7f9e1414f790 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3710431602 0 0) 0x7f9e04003520 con 0x7f9e1414f790 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 msgr2=0x7f9e14155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 0x7f9e14155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 >> v2:172.21.15.145:3300/0 conn(0x7f9e140c1aa0 msgr2=0x7f9e141529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 --2- 172.21.15.94:0/1829444909 >> v2:172.21.15.145:3300/0 conn(0x7f9e140c1aa0 0x7f9e141529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e140c2bb0 con 0x7f9e1414f790 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f9e08001670 con 0x7f9e1414f790 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 >> v1:172.21.15.94:6789/0 conn(0x7f9e1414f790 legacy=0x7f9e140c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 0x7f9e04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e04004d40 0x7f9e040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e140c2680 con 0x7f9e040039b0 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e140c28d0 con 0x7f9e04004d40 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0400b9c0 con 0x7f9e14155a40 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.704+0000 7f9e18efa640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e13fff640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 0x7f9e04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e196fb640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e04004d40 0x7f9e040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e18efa640 1 -- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 msgr2=0x7f9e04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e18efa640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e14155a40 0x7f9e04004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e18efa640 1 -- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e04004d40 msgr2=0x7f9e040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e18efa640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e04004d40 0x7f9e040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e18efa640 1 -- 172.21.15.94:0/1829444909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04003520 con 0x7f9e040039b0 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.705+0000 7f9e18efa640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e04003dd0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c210 tx=0x7f9e04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a4a7cb97780a6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08007d50 con 0x7f9e040039b0 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e08030070 con 0x7f9e040039b0 2024-03-06T07:26:21.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e137fe640 1 -- 172.21.15.94:0/1829444909 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0802cd80 con 0x7f9e040039b0 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e1b15e640 1 -- 172.21.15.94:0/1829444909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 msgr2=0x7f9e04003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e1b15e640 1 --2- 172.21.15.94:0/1829444909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e04003dd0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c210 tx=0x7f9e04005830 comp rx=0 tx=0).stop 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e196fb640 1 -- 172.21.15.94:0/1829444909 reap_dead start 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e1b15e640 1 -- 172.21.15.94:0/1829444909 shutdown_connections 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e1b15e640 1 -- 172.21.15.94:0/1829444909 >> 172.21.15.94:0/1829444909 conn(0x7f9e140c6710 msgr2=0x7f9e140c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e1b15e640 1 -- 172.21.15.94:0/1829444909 shutdown_connections 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.706+0000 7f9e1b15e640 1 -- 172.21.15.94:0/1829444909 wait complete. 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.707+0000 7f9e1b15e640 1 Processor -- start 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.707+0000 7f9e1b15e640 1 -- start start 2024-03-06T07:26:21.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e1b15e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e140c4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e1b15e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e1414c560 0x7f9e14070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e1b15e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e140c4b50 0x7f9e141ec1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e1b15e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e140c0d60 con 0x7f9e040039b0 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e1b15e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14155520 con 0x7f9e140c4b50 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e1b15e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e140c5e30 con 0x7f9e1414c560 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e140c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e140c4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48064/0 (socket says 172.21.15.94:48064) 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 -- 172.21.15.94:0/3084466531 learned_addr learned my addr 172.21.15.94:0/3084466531 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 -- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e1414c560 msgr2=0x7f9e14070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e1414c560 0x7f9e14070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 -- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e140c4b50 msgr2=0x7f9e141ec1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e196fb640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e140c4b50 0x7f9e141ec1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e140c4b50 0x7f9e141ec1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.708+0000 7f9e18efa640 1 -- 172.21.15.94:0/3084466531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e140c5550 con 0x7f9e040039b0 2024-03-06T07:26:21.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.709+0000 7f9e18efa640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e140c4610 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c330 tx=0x7f9e08001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.709+0000 7f9e117fa640 1 -- 172.21.15.94:0/3084466531 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08045070 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.709+0000 7f9e117fa640 1 -- 172.21.15.94:0/3084466531 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e08004bf0 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.709+0000 7f9e117fa640 1 -- 172.21.15.94:0/3084466531 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e08001c90 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.709+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e141e8c60 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.709+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e141e9200 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.710+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1404f330 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.710+0000 7f9e117fa640 1 -- 172.21.15.94:0/3084466531 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e08007070 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.711+0000 7f9e117fa640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9df806b500 0x7f9df806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.711+0000 7f9e117fa640 1 -- 172.21.15.94:0/3084466531 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9e080ab110 con 0x7f9e040039b0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.711+0000 7f9e13fff640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9df806b500 0x7f9df806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.711+0000 7f9e13fff640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9df806b500 0x7f9df806d9c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9e1414c9a0 tx=0x7f9e00006c30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.715+0000 7f9e117fa640 1 -- 172.21.15.94:0/3084466531 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e08079be0 con 0x7f9e040039b0 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.952+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9e14068450 con 0x7f9e040039b0 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.952+0000 7f9e117fa640 1 -- 172.21.15.94:0/3084466531 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f9e0807dc80 con 0x7f9e040039b0 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.954 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.955 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.956 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.957 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:26:21.958 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.959 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:26:21.960 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.961 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:26:21.962 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:26:21.963 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.964 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:21.965 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:26:21.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.966 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:21.966 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:26:21.966 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:26:21.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9df806b500 msgr2=0x7f9df806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9df806b500 0x7f9df806d9c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9e1414c9a0 tx=0x7f9e00006c30 comp rx=0 tx=0).stop 2024-03-06T07:26:21.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 msgr2=0x7f9e140c4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e140c4610 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800c330 tx=0x7f9e08001a40 comp rx=0 tx=0).stop 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 shutdown_connections 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9df806b500 0x7f9df806d9c0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e140c4b50 0x7f9e141ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e1414c560 0x7f9e14070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 --2- 172.21.15.94:0/3084466531 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9e040039b0 0x7f9e140c4610 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 >> 172.21.15.94:0/3084466531 conn(0x7f9e140c6710 msgr2=0x7f9e1406df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 shutdown_connections 2024-03-06T07:26:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:21.954+0000 7f9e1b15e640 1 -- 172.21.15.94:0/3084466531 wait complete. 2024-03-06T07:26:21.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:22.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.066+0000 7f91b584e640 1 Processor -- start 2024-03-06T07:26:22.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.066+0000 7f91b584e640 1 -- start start 2024-03-06T07:26:22.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91b584e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f91b00c3370 0x7f91b0155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91b584e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00a5e60 0x7f91b0156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91b584e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00c5490 con 0x7f91b00c3750 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91b584e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b00c5770 con 0x7f91b00c3370 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91b584e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00c5a50 con 0x7f91b00a5e60 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91af7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00a5e60 0x7f91b0156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91aeffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f91b00c3370 0x7f91b0155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91ae7fc640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f91b00c3750 0x7f91b00c1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:35378/0 (socket says 172.21.15.94:35378) 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.067+0000 7f91ae7fc640 1 -- 172.21.15.94:0/3229942034 learned_addr learned my addr 172.21.15.94:0/3229942034 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.068+0000 7f91aeffd640 1 -- 172.21.15.94:0/3229942034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00a5e60 msgr2=0x7f91b0156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.068+0000 7f91aeffd640 1 --2- 172.21.15.94:0/3229942034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00a5e60 0x7f91b0156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.068+0000 7f91aeffd640 1 -- 172.21.15.94:0/3229942034 >> v1:172.21.15.145:6789/0 conn(0x7f91b00c3750 legacy=0x7f91b00c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.068+0000 7f91aeffd640 1 -- 172.21.15.94:0/3229942034 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b00c5d30 con 0x7f91b00c3370 2024-03-06T07:26:22.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.068+0000 7f91adffb640 1 -- 172.21.15.94:0/3229942034 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274892139 0 0) 0x7f91b00c5490 con 0x7f91b00c3750 2024-03-06T07:26:22.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.068+0000 7f91aeffd640 1 --2- 172.21.15.94:0/3229942034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00c3370 0x7f91b0155ad0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f91a00097a0 tx=0x7f91a002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34937bbf2af53b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91adffb640 1 -- 172.21.15.94:0/3229942034 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0038070 con 0x7f91b00c3370 2024-03-06T07:26:22.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91adffb640 1 -- 172.21.15.94:0/3229942034 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a0031700 con 0x7f91b00c3370 2024-03-06T07:26:22.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91adffb640 1 -- 172.21.15.94:0/3229942034 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0031ad0 con 0x7f91b00c3370 2024-03-06T07:26:22.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 -- 172.21.15.94:0/3229942034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00c3370 msgr2=0x7f91b0155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 --2- 172.21.15.94:0/3229942034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00c3370 0x7f91b0155ad0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f91a00097a0 tx=0x7f91a002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 -- 172.21.15.94:0/3229942034 shutdown_connections 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 --2- 172.21.15.94:0/3229942034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00a5e60 0x7f91b0156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 --2- 172.21.15.94:0/3229942034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00c3370 0x7f91b0155ad0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 -- 172.21.15.94:0/3229942034 >> 172.21.15.94:0/3229942034 conn(0x7f91b00c6710 msgr2=0x7f91b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 -- 172.21.15.94:0/3229942034 shutdown_connections 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.069+0000 7f91b584e640 1 -- 172.21.15.94:0/3229942034 wait complete. 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.070+0000 7f91b584e640 1 Processor -- start 2024-03-06T07:26:22.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.070+0000 7f91b584e640 1 -- start start 2024-03-06T07:26:22.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91b584e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00a5e60 0x7f91b0070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91b584e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00c3370 0x7f91b0152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91b584e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91b00c3750 0x7f91b01ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91b584e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0159910 con 0x7f91b00c3750 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91b584e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0159790 con 0x7f91b00a5e60 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91b584e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00c4ed0 con 0x7f91b00c3370 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91af7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91b00c3750 0x7f91b01ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91af7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91b00c3750 0x7f91b01ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48072/0 (socket says 172.21.15.94:48072) 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91af7fe640 1 -- 172.21.15.94:0/2158296233 learned_addr learned my addr 172.21.15.94:0/2158296233 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91aeffd640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00a5e60 0x7f91b0070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91ae7fc640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00c3370 0x7f91b0152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91af7fe640 1 -- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00c3370 msgr2=0x7f91b0152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91af7fe640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00c3370 0x7f91b0152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91af7fe640 1 -- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00a5e60 msgr2=0x7f91b0070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.071+0000 7f91af7fe640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00a5e60 0x7f91b0070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.072+0000 7f91af7fe640 1 -- 172.21.15.94:0/2158296233 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b0070fc0 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.072+0000 7f91af7fe640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91b00c3750 0x7f91b01ee4e0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f91a800efc0 tx=0x7f91a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.072+0000 7f9197fff640 1 -- 172.21.15.94:0/2158296233 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8018020 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.072+0000 7f9197fff640 1 -- 172.21.15.94:0/2158296233 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a8014480 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.072+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b00a8f30 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.072+0000 7f9197fff640 1 -- 172.21.15.94:0/2158296233 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8014870 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.072+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b01534f0 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.073+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b006d550 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.073+0000 7f9197fff640 1 -- 172.21.15.94:0/2158296233 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91a8005d10 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.074+0000 7f9197fff640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f919c06b4d0 0x7f919c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.074+0000 7f9197fff640 1 -- 172.21.15.94:0/2158296233 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f91a808a930 con 0x7f91b00c3750 2024-03-06T07:26:22.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.074+0000 7f91aeffd640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f919c06b4d0 0x7f919c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.074+0000 7f91aeffd640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f919c06b4d0 0x7f919c06d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f91a0035000 tx=0x7f91a0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.078+0000 7f9197fff640 1 -- 172.21.15.94:0/2158296233 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91b006d550 con 0x7f91b00c3750 2024-03-06T07:26:22.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.251+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91b004f330 con 0x7f91b00c3750 2024-03-06T07:26:22.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.252+0000 7f9197fff640 1 -- 172.21.15.94:0/2158296233 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+13174 (secure 0 0 0) 0x7f91a8022070 con 0x7f91b00c3750 2024-03-06T07:26:22.253 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:22.254 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":192,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:21.579374+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:22.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f919c06b4d0 msgr2=0x7f919c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f919c06b4d0 0x7f919c06d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f91a0035000 tx=0x7f91a0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91b00c3750 msgr2=0x7f91b01ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91b00c3750 0x7f91b01ee4e0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f91a800efc0 tx=0x7f91a800c490 comp rx=0 tx=0).stop 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 shutdown_connections 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f919c06b4d0 0x7f919c06d990 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f91b00c3750 0x7f91b01ee4e0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91b00c3370 0x7f91b0152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.253+0000 7f91b584e640 1 --2- 172.21.15.94:0/2158296233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91b00a5e60 0x7f91b0070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.254+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 >> 172.21.15.94:0/2158296233 conn(0x7f91b00c6710 msgr2=0x7f91b0154160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.254+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 shutdown_connections 2024-03-06T07:26:22.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.254+0000 7f91b584e640 1 -- 172.21.15.94:0/2158296233 wait complete. 2024-03-06T07:26:22.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:22.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:22.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.368+0000 7f1a5149a640 1 Processor -- start 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.368+0000 7f1a5149a640 1 -- start start 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.368+0000 7f1a5149a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c1528d0 0x7f1a4c0724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.368+0000 7f1a5149a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1a4c072a10 0x7f1a4c155220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.368+0000 7f1a5149a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c152cb0 con 0x7f1a4c0c67f0 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.368+0000 7f1a5149a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c0c38b0 con 0x7f1a4c072a10 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.368+0000 7f1a5149a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1594f0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1a4c072a10 0x7f1a4c155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1a4c0c67f0 0x7f1a4c155760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38730/0 (socket says 172.21.15.94:38730) 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4b7fe640 1 -- 172.21.15.94:0/534363394 learned_addr learned my addr 172.21.15.94:0/534363394 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:22.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4affd640 1 --2- 172.21.15.94:0/534363394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c1528d0 0x7f1a4c0724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4a7fc640 1 -- 172.21.15.94:0/534363394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c1528d0 msgr2=0x7f1a4c0724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4a7fc640 1 --2- 172.21.15.94:0/534363394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c1528d0 0x7f1a4c0724d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4a7fc640 1 -- 172.21.15.94:0/534363394 >> v1:172.21.15.94:6789/0 conn(0x7f1a4c0c67f0 legacy=0x7f1a4c155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a4a7fc640 1 -- 172.21.15.94:0/534363394 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0a7f70 con 0x7f1a4c072a10 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.369+0000 7f1a49ffb640 1 -- 172.21.15.94:0/534363394 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891422900 0 0) 0x7f1a4c152cb0 con 0x7f1a4c0c67f0 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a4a7fc640 1 --2- 172.21.15.94:0/534363394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c072a10 0x7f1a4c155220 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f1a400097a0 tx=0x7f1a4002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4a6a92760f4ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a49ffb640 1 -- 172.21.15.94:0/534363394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40034070 con 0x7f1a4c072a10 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a49ffb640 1 -- 172.21.15.94:0/534363394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a40030610 con 0x7f1a4c072a10 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a49ffb640 1 -- 172.21.15.94:0/534363394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a400309e0 con 0x7f1a4c072a10 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 -- 172.21.15.94:0/534363394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c072a10 msgr2=0x7f1a4c155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 --2- 172.21.15.94:0/534363394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c072a10 0x7f1a4c155220 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f1a400097a0 tx=0x7f1a4002b960 comp rx=0 tx=0).stop 2024-03-06T07:26:22.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 -- 172.21.15.94:0/534363394 shutdown_connections 2024-03-06T07:26:22.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 --2- 172.21.15.94:0/534363394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c072a10 0x7f1a4c155220 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 --2- 172.21.15.94:0/534363394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c1528d0 0x7f1a4c0724d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 -- 172.21.15.94:0/534363394 >> 172.21.15.94:0/534363394 conn(0x7f1a4c0aa6a0 msgr2=0x7f1a4c0aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 -- 172.21.15.94:0/534363394 shutdown_connections 2024-03-06T07:26:22.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.370+0000 7f1a5149a640 1 -- 172.21.15.94:0/534363394 wait complete. 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 Processor -- start 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 -- start start 2024-03-06T07:26:22.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a4c072a10 0x7f1a4c157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c0c67f0 0x7f1a4c152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c1528d0 0x7f1a4c1eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c1591a0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0a8250 con 0x7f1a4c072a10 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a5149a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c0c4ef0 con 0x7f1a4c0c67f0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a4b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c1528d0 0x7f1a4c1eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a4b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c1528d0 0x7f1a4c1eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48096/0 (socket says 172.21.15.94:48096) 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a4b7fe640 1 -- 172.21.15.94:0/2011222090 learned_addr learned my addr 172.21.15.94:0/2011222090 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a4affd640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a4c072a10 0x7f1a4c157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.372+0000 7f1a4a7fc640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c0c67f0 0x7f1a4c152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a4b7fe640 1 -- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c0c67f0 msgr2=0x7f1a4c152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a4b7fe640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c0c67f0 0x7f1a4c152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a4b7fe640 1 -- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a4c072a10 msgr2=0x7f1a4c157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a4b7fe640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a4c072a10 0x7f1a4c157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a4b7fe640 1 -- 172.21.15.94:0/2011222090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0ab910 con 0x7f1a4c1528d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a4a7fc640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c0c67f0 0x7f1a4c152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a4b7fe640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c1528d0 0x7f1a4c1eaf70 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c7b0 tx=0x7f1a4400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a337fe640 1 -- 172.21.15.94:0/2011222090 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44018070 con 0x7f1a4c1528d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a337fe640 1 -- 172.21.15.94:0/2011222090 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4400a5e0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c0abbf0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a337fe640 1 -- 172.21.15.94:0/2011222090 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4400a9d0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.373+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c152cb0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.374+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4c04f330 con 0x7f1a4c1528d0 2024-03-06T07:26:22.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.374+0000 7f1a337fe640 1 -- 172.21.15.94:0/2011222090 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a4401b070 con 0x7f1a4c1528d0 2024-03-06T07:26:22.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.375+0000 7f1a337fe640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1a3806b4d0 0x7f1a3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.375+0000 7f1a337fe640 1 -- 172.21.15.94:0/2011222090 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1a440894e0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.375+0000 7f1a4affd640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1a3806b4d0 0x7f1a3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.376+0000 7f1a4affd640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1a3806b4d0 0x7f1a3806d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00aa20 tx=0x7f1a3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.379+0000 7f1a337fe640 1 -- 172.21.15.94:0/2011222090 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a44057f40 con 0x7f1a4c1528d0 2024-03-06T07:26:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.550+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1a4c0a7d50 con 0x7f1a4c1528d0 2024-03-06T07:26:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.550+0000 7f1a337fe640 1 -- 172.21.15.94:0/2011222090 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1a4405bfe0 con 0x7f1a4c1528d0 2024-03-06T07:26:22.552 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:26:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1a3806b4d0 msgr2=0x7f1a3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1a3806b4d0 0x7f1a3806d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00aa20 tx=0x7f1a3c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c1528d0 msgr2=0x7f1a4c1eaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c1528d0 0x7f1a4c1eaf70 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c7b0 tx=0x7f1a4400cc70 comp rx=0 tx=0).stop 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 shutdown_connections 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1a3806b4d0 0x7f1a3806d990 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4c1528d0 0x7f1a4c1eaf70 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a4c0c67f0 0x7f1a4c152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 --2- 172.21.15.94:0/2011222090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a4c072a10 0x7f1a4c157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 >> 172.21.15.94:0/2011222090 conn(0x7f1a4c0aa6a0 msgr2=0x7f1a4c0c7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 shutdown_connections 2024-03-06T07:26:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.552+0000 7f1a5149a640 1 -- 172.21.15.94:0/2011222090 wait complete. 2024-03-06T07:26:22.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:22.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.663+0000 7ff46a64e640 1 Processor -- start 2024-03-06T07:26:22.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff46a64e640 1 -- start start 2024-03-06T07:26:22.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff46a64e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4640c88c0 0x7ff464155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff46a64e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4640c73d0 con 0x7ff46414f360 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff46a64e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4640be2e0 con 0x7ff464158b60 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff46a64e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff464159700 con 0x7ff4640c88c0 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff4637fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4640c88c0 0x7ff464155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff4637fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff4640c88c0 0x7ff464155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48104/0 (socket says 172.21.15.94:48104) 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff463fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff46414f360 0x7ff4640c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38734/0 (socket says 172.21.15.94:38734) 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.664+0000 7ff4637fe640 1 -- 172.21.15.94:0/3446046394 learned_addr learned my addr 172.21.15.94:0/3446046394 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.665+0000 7ff4637fe640 1 -- 172.21.15.94:0/3446046394 >> v1:172.21.15.145:6789/0 conn(0x7ff464158b60 legacy=0x7ff464158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.665+0000 7ff4637fe640 1 -- 172.21.15.94:0/3446046394 >> v1:172.21.15.94:6789/0 conn(0x7ff46414f360 legacy=0x7ff4640c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.665+0000 7ff4637fe640 1 -- 172.21.15.94:0/3446046394 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640c7670 con 0x7ff4640c88c0 2024-03-06T07:26:22.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.665+0000 7ff462ffd640 1 -- 172.21.15.94:0/3446046394 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556220961 0 0) 0x7ff4640c73d0 con 0x7ff46414f360 2024-03-06T07:26:22.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.665+0000 7ff4637fe640 1 --2- 172.21.15.94:0/3446046394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4640c88c0 0x7ff464155c10 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7ff454009870 tx=0x7ff45402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7972f0826c4e120b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff462ffd640 1 -- 172.21.15.94:0/3446046394 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454034070 con 0x7ff4640c88c0 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff462ffd640 1 -- 172.21.15.94:0/3446046394 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454030860 con 0x7ff4640c88c0 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff462ffd640 1 -- 172.21.15.94:0/3446046394 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454030c30 con 0x7ff4640c88c0 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff46a64e640 1 -- 172.21.15.94:0/3446046394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4640c88c0 msgr2=0x7ff464155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff46a64e640 1 --2- 172.21.15.94:0/3446046394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4640c88c0 0x7ff464155c10 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7ff454009870 tx=0x7ff45402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff46a64e640 1 -- 172.21.15.94:0/3446046394 shutdown_connections 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff46a64e640 1 --2- 172.21.15.94:0/3446046394 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4640c88c0 0x7ff464155c10 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff46a64e640 1 -- 172.21.15.94:0/3446046394 >> 172.21.15.94:0/3446046394 conn(0x7ff4640ac570 msgr2=0x7ff4640ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff46a64e640 1 -- 172.21.15.94:0/3446046394 shutdown_connections 2024-03-06T07:26:22.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.666+0000 7ff46a64e640 1 -- 172.21.15.94:0/3446046394 wait complete. 2024-03-06T07:26:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.667+0000 7ff46a64e640 1 Processor -- start 2024-03-06T07:26:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.667+0000 7ff46a64e640 1 -- start start 2024-03-06T07:26:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff46a64e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4640c88c0 0x7ff4641eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff46a64e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46414f360 0x7ff46414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff46a64e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff464158b60 0x7ff464151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff46a64e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640c7c30 con 0x7ff46414f360 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff46a64e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640c5a50 con 0x7ff464158b60 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff46a64e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640c5bd0 con 0x7ff4640c88c0 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff4637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46414f360 0x7ff46414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff4637fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46414f360 0x7ff46414f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48108/0 (socket says 172.21.15.94:48108) 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff4637fe640 1 -- 172.21.15.94:0/1177175732 learned_addr learned my addr 172.21.15.94:0/1177175732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff468beb640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff464158b60 0x7ff464151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff463fff640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4640c88c0 0x7ff4641eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff4637fe640 1 -- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4640c88c0 msgr2=0x7ff4641eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.668+0000 7ff4637fe640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4640c88c0 0x7ff4641eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff4637fe640 1 -- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff464158b60 msgr2=0x7ff464151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff4637fe640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff464158b60 0x7ff464151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff4637fe640 1 -- 172.21.15.94:0/1177175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46414fe40 con 0x7ff46414f360 2024-03-06T07:26:22.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff468beb640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff464158b60 0x7ff464151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff4637fe640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46414f360 0x7ff46414f740 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7ff4540090b0 tx=0x7ff4540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff460ff9640 1 -- 172.21.15.94:0/1177175732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454034040 con 0x7ff46414f360 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff460ff9640 1 -- 172.21.15.94:0/1177175732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454039a80 con 0x7ff46414f360 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff460ff9640 1 -- 172.21.15.94:0/1177175732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454004030 con 0x7ff46414f360 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff464150120 con 0x7ff46414f360 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.669+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4641506e0 con 0x7ff46414f360 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.670+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46414ebc0 con 0x7ff46414f360 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.670+0000 7ff460ff9640 1 -- 172.21.15.94:0/1177175732 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff45404b060 con 0x7ff46414f360 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.671+0000 7ff460ff9640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45006b4d0 0x7ff45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.671+0000 7ff460ff9640 1 -- 172.21.15.94:0/1177175732 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff4540abea0 con 0x7ff46414f360 2024-03-06T07:26:22.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.671+0000 7ff463fff640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45006b4d0 0x7ff45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.671+0000 7ff463fff640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45006b4d0 0x7ff45006d990 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff458007fd0 tx=0x7ff458009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.675+0000 7ff460ff9640 1 -- 172.21.15.94:0/1177175732 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff46414ebc0 con 0x7ff46414f360 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.848+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff464069460 con 0x7ff46414f360 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.848+0000 7ff460ff9640 1 -- 172.21.15.94:0/1177175732 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v192) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff464069460 con 0x7ff46414f360 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:26:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:26:22.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45006b4d0 msgr2=0x7ff45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45006b4d0 0x7ff45006d990 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff458007fd0 tx=0x7ff458009040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46414f360 msgr2=0x7ff46414f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46414f360 0x7ff46414f740 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7ff4540090b0 tx=0x7ff4540046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 shutdown_connections 2024-03-06T07:26:22.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff45006b4d0 0x7ff45006d990 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff464158b60 0x7ff464151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff46414f360 0x7ff46414f740 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 --2- 172.21.15.94:0/1177175732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4640c88c0 0x7ff4641eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 >> 172.21.15.94:0/1177175732 conn(0x7ff4640ac570 msgr2=0x7ff46414e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 shutdown_connections 2024-03-06T07:26:22.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.850+0000 7ff46a64e640 1 -- 172.21.15.94:0/1177175732 wait complete. 2024-03-06T07:26:22.863 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:22.863 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:22.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.965+0000 7f945c3f7640 1 Processor -- start 2024-03-06T07:26:22.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.965+0000 7f945c3f7640 1 -- start start 2024-03-06T07:26:22.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945c3f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94540c1d10 0x7f9454156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945c3f7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f94540c2920 0x7f9454156980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945c3f7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9454070430 con 0x7f94540abb60 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945c3f7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94540711f0 con 0x7f94540c2920 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945c3f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9454070d60 con 0x7f94540c1d10 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945a193640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f94540abb60 0x7f94541502f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38746/0 (socket says 172.21.15.94:38746) 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945a193640 1 -- 172.21.15.94:0/3286753081 learned_addr learned my addr 172.21.15.94:0/3286753081 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759882064 0 0) 0x7f9454070430 con 0x7f94540abb60 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943c003520 con 0x7f94540abb60 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f9459992640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94540c1d10 0x7f9454156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f945a994640 1 --2- 172.21.15.94:0/3286753081 >> v2:172.21.15.145:3300/0 conn(0x7f94540c2920 0x7f9454156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.966+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 363789783 0 0) 0x7f943c003520 con 0x7f94540abb60 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94540c1d10 msgr2=0x7f9454156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f94540c1d10 0x7f9454156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 >> v2:172.21.15.145:3300/0 conn(0x7f94540c2920 msgr2=0x7f9454156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 --2- 172.21.15.94:0/3286753081 >> v2:172.21.15.145:3300/0 conn(0x7f94540c2920 0x7f9454156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454070710 con 0x7f94540abb60 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f944c0019f0 con 0x7f94540abb60 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 >> v1:172.21.15.94:6789/0 conn(0x7f94540abb60 legacy=0x7f94541502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f943c004800 0x7f943c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f943c0051c0 0x7f943c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9454070d60 con 0x7f943c004800 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c005630 con 0x7f943c0039b0 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943c00bf30 con 0x7f943c0051c0 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459992640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f943c004800 0x7f943c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f945a193640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f945a994640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f943c0051c0 0x7f943c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459992640 1 -- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f943c0051c0 msgr2=0x7f943c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459992640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f943c0051c0 0x7f943c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459992640 1 -- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c0039b0 msgr2=0x7f943c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459992640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c0039b0 0x7f943c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f9459992640 1 -- 172.21.15.94:0/3286753081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943c003520 con 0x7f943c004800 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.967+0000 7f945a994640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f943c0051c0 0x7f943c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f9459992640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f943c004800 0x7f943c004c80 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f9448009ac0 tx=0x7f943c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96838626cfad2235 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448039070 con 0x7f943c004800 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944802c6b0 con 0x7f943c004800 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f9459191640 1 -- 172.21.15.94:0/3286753081 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448009690 con 0x7f943c004800 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f945c3f7640 1 -- 172.21.15.94:0/3286753081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f943c004800 msgr2=0x7f943c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f945c3f7640 1 --2- 172.21.15.94:0/3286753081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f943c004800 0x7f943c004c80 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f9448009ac0 tx=0x7f943c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f945a994640 1 -- 172.21.15.94:0/3286753081 reap_dead start 2024-03-06T07:26:22.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f945c3f7640 1 -- 172.21.15.94:0/3286753081 shutdown_connections 2024-03-06T07:26:22.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f945c3f7640 1 -- 172.21.15.94:0/3286753081 >> 172.21.15.94:0/3286753081 conn(0x7f94540c65b0 msgr2=0x7f94540c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f945c3f7640 1 -- 172.21.15.94:0/3286753081 shutdown_connections 2024-03-06T07:26:22.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.968+0000 7f945c3f7640 1 -- 172.21.15.94:0/3286753081 wait complete. 2024-03-06T07:26:22.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 Processor -- start 2024-03-06T07:26:22.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 -- start start 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c004800 0x7f945414f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945414d190 0x7f94541488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9454155db0 0x7f9454148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9454150b80 con 0x7f945414d190 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9454150a00 con 0x7f943c004800 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945c3f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9454156ec0 con 0x7f9454155db0 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f9459992640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945414d190 0x7f94541488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f9459992640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945414d190 0x7f94541488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48126/0 (socket says 172.21.15.94:48126) 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f9459992640 1 -- 172.21.15.94:0/2600369085 learned_addr learned my addr 172.21.15.94:0/2600369085 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.970+0000 7f945a193640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c004800 0x7f945414f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f945a994640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9454155db0 0x7f9454148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9459992640 1 -- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9454155db0 msgr2=0x7f9454148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9459992640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9454155db0 0x7f9454148e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9459992640 1 -- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c004800 msgr2=0x7f945414f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9459992640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c004800 0x7f945414f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9459992640 1 -- 172.21.15.94:0/2600369085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94541541b0 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9459992640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945414d190 0x7f94541488e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f9448000c00 tx=0x7f9448004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9452ffd640 1 -- 172.21.15.94:0/2600369085 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448042020 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9452ffd640 1 -- 172.21.15.94:0/2600369085 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944803a040 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9454154490 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f9452ffd640 1 -- 172.21.15.94:0/2600369085 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448039070 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.971+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9454154a50 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.972+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945404f330 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.973+0000 7f9452ffd640 1 -- 172.21.15.94:0/2600369085 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9448045020 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.973+0000 7f9452ffd640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f944006b4d0 0x7f944006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.973+0000 7f9452ffd640 1 -- 172.21.15.94:0/2600369085 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f94480ab6a0 con 0x7f945414d190 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.973+0000 7f945a193640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f944006b4d0 0x7f944006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.974+0000 7f945a193640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f944006b4d0 0x7f944006d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f944c0013d0 tx=0x7f944c00a0e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:22.977+0000 7f9452ffd640 1 -- 172.21.15.94:0/2600369085 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f944804c030 con 0x7f945414d190 2024-03-06T07:26:23.151 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.149+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f945406d460 con 0x7f945414d190 2024-03-06T07:26:23.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.589+0000 7f9452ffd640 1 -- 172.21.15.94:0/2600369085 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v193) v1 ==== 138+0+0 (secure 0 0 0) 0x7f94480ab070 con 0x7f945414d190 2024-03-06T07:26:23.591 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.591+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f944006b4d0 msgr2=0x7f944006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.591+0000 7f945c3f7640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f944006b4d0 0x7f944006d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f944c0013d0 tx=0x7f944c00a0e0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.591+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945414d190 msgr2=0x7f94541488e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.591+0000 7f945c3f7640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945414d190 0x7f94541488e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f9448000c00 tx=0x7f9448004730 comp rx=0 tx=0).stop 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 shutdown_connections 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f944006b4d0 0x7f944006d990 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9454155db0 0x7f9454148e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945414d190 0x7f94541488e0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 --2- 172.21.15.94:0/2600369085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f943c004800 0x7f945414f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 >> 172.21.15.94:0/2600369085 conn(0x7f94540c65b0 msgr2=0x7f94540c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 shutdown_connections 2024-03-06T07:26:23.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.592+0000 7f945c3f7640 1 -- 172.21.15.94:0/2600369085 wait complete. 2024-03-06T07:26:23.609 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:23.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.716+0000 7f100e6c9640 1 Processor -- start 2024-03-06T07:26:23.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.716+0000 7f100e6c9640 1 -- start start 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.716+0000 7f100e6c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f10080c5990 0x7f10080c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.716+0000 7f100e6c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10080c3ef0 con 0x7f100806c900 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.716+0000 7f100e6c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10081556d0 con 0x7f10080ab170 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.716+0000 7f100e6c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10081559b0 con 0x7f10080c5990 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f100cc66640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f100806c900 0x7f10080a9fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38750/0 (socket says 172.21.15.94:38750) 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f100cc66640 1 -- 172.21.15.94:0/1427335297 learned_addr learned my addr 172.21.15.94:0/1427335297 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:23.721 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794294960 0 0) 0x7f10080c3ef0 con 0x7f100806c900 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff0003520 con 0x7f100806c900 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1007fff640 1 --2- 172.21.15.94:0/1427335297 >> v2:172.21.15.145:3300/0 conn(0x7f10080c5990 0x7f10080c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2767222107 0 0) 0x7f0ff0003520 con 0x7f100806c900 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 >> v2:172.21.15.145:3300/0 conn(0x7f10080c5990 msgr2=0x7f10080c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 --2- 172.21.15.94:0/1427335297 >> v2:172.21.15.145:3300/0 conn(0x7f10080c5990 0x7f10080c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 >> v1:172.21.15.145:6789/0 conn(0x7f10080ab170 legacy=0x7f100806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1008155c90 con 0x7f100806c900 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f10000019f0 con 0x7f100806c900 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 >> v1:172.21.15.94:6789/0 conn(0x7f100806c900 legacy=0x7f10080a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ff0003a00 0x7f0ff0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ff0004800 0x7f0ff0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff00051c0 0x7f0ff000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10081559b0 con 0x7f0ff0003a00 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff000c0c0 con 0x7f0ff00051c0 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff000c2c0 con 0x7f0ff0004800 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f1007fff640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ff0003a00 0x7f0ff0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f10077fe640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ff0004800 0x7f0ff0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.717+0000 7f100cc66640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff00051c0 0x7f0ff000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1007fff640 1 -- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ff0004800 msgr2=0x7f0ff0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1007fff640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ff0004800 0x7f0ff0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1007fff640 1 -- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff00051c0 msgr2=0x7f0ff000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1007fff640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff00051c0 0x7f0ff000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f100cc66640 1 -- 172.21.15.94:0/1427335297 reap_dead start 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1007fff640 1 -- 172.21.15.94:0/1427335297 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0003520 con 0x7f0ff0003a00 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1007fff640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ff0003a00 0x7f0ff0003e20 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009ac0 tx=0x7f0ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4388f29c19bd40f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8039070 con 0x7f0ff0003a00 2024-03-06T07:26:23.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff802c6b0 con 0x7f0ff0003a00 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.718+0000 7f1006ffd640 1 -- 172.21.15.94:0/1427335297 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8009690 con 0x7f0ff0003a00 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.719+0000 7f100e6c9640 1 -- 172.21.15.94:0/1427335297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ff0003a00 msgr2=0x7f0ff0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.719+0000 7f100e6c9640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ff0003a00 0x7f0ff0003e20 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009ac0 tx=0x7f0ff0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.719+0000 7f100e6c9640 1 -- 172.21.15.94:0/1427335297 shutdown_connections 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.719+0000 7f100e6c9640 1 --2- 172.21.15.94:0/1427335297 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ff0003a00 0x7f0ff0003e20 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.719+0000 7f100e6c9640 1 -- 172.21.15.94:0/1427335297 >> 172.21.15.94:0/1427335297 conn(0x7f10080c6710 msgr2=0x7f10080c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.719+0000 7f100e6c9640 1 -- 172.21.15.94:0/1427335297 shutdown_connections 2024-03-06T07:26:23.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.719+0000 7f100e6c9640 1 -- 172.21.15.94:0/1427335297 wait complete. 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.720+0000 7f100e6c9640 1 Processor -- start 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.720+0000 7f100e6c9640 1 -- start start 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f100e6c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff0003a00 0x7f10081e8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f100e6c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1008152a70 0x7f100814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f100e6c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1008153a90 0x7f1008153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f100e6c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1008156610 con 0x7f1008152a70 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f100e6c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1008156490 con 0x7f0ff0003a00 2024-03-06T07:26:23.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f100e6c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080c3af0 con 0x7f1008153a90 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1008152a70 0x7f100814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1008152a70 0x7f100814df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48158/0 (socket says 172.21.15.94:48158) 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 -- 172.21.15.94:0/674733788 learned_addr learned my addr 172.21.15.94:0/674733788 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f1007fff640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff0003a00 0x7f10081e8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f100cc66640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1008153a90 0x7f1008153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 -- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1008153a90 msgr2=0x7f1008153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1008153a90 0x7f1008153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 -- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff0003a00 msgr2=0x7f10081e8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff0003a00 0x7f10081e8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f10077fe640 1 -- 172.21.15.94:0/674733788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1008154a70 con 0x7f1008152a70 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.721+0000 7f1007fff640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff0003a00 0x7f10081e8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.722+0000 7f10077fe640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1008152a70 0x7f100814df80 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00f850 tx=0x7f0ffc00fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.722+0000 7f1004ff9640 1 -- 172.21.15.94:0/674733788 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc00fe40 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.722+0000 7f1004ff9640 1 -- 172.21.15.94:0/674733788 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc018070 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.722+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10081e8980 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.722+0000 7f1004ff9640 1 -- 172.21.15.94:0/674733788 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc00d730 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.722+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1008154ff0 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.723+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100804f330 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.723+0000 7f1004ff9640 1 -- 172.21.15.94:0/674733788 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ffc00d8d0 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.724+0000 7f1004ff9640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.724+0000 7f1004ff9640 1 -- 172.21.15.94:0/674733788 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f0ffc089ff0 con 0x7f1008152a70 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.724+0000 7f1007fff640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.724+0000 7f1007fff640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ff406b500 0x7f0ff406d9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8036000 tx=0x7f0ff8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.728+0000 7f1004ff9640 1 -- 172.21.15.94:0/674733788 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ffc058ed0 con 0x7f1008152a70 2024-03-06T07:26:23.902 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:23.899+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f100806c270 con 0x7f1008152a70 2024-03-06T07:26:24.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.591+0000 7f1004ff9640 1 -- 172.21.15.94:0/674733788 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v194) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0ffc0587d0 con 0x7f1008152a70 2024-03-06T07:26:24.594 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:26:24.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ff406b500 msgr2=0x7f0ff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ff406b500 0x7f0ff406d9c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8036000 tx=0x7f0ff8030000 comp rx=0 tx=0).stop 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1008152a70 msgr2=0x7f100814df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1008152a70 0x7f100814df80 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00f850 tx=0x7f0ffc00fd10 comp rx=0 tx=0).stop 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 shutdown_connections 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1008153a90 0x7f1008153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1008152a70 0x7f100814df80 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 --2- 172.21.15.94:0/674733788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ff0003a00 0x7f10081e8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 >> 172.21.15.94:0/674733788 conn(0x7f10080c6710 msgr2=0x7f10080c1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 shutdown_connections 2024-03-06T07:26:24.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.594+0000 7f100e6c9640 1 -- 172.21.15.94:0/674733788 wait complete. 2024-03-06T07:26:24.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:24.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.728+0000 7f9d54233640 1 Processor -- start 2024-03-06T07:26:24.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.728+0000 7f9d54233640 1 -- start start 2024-03-06T07:26:24.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.728+0000 7f9d54233640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9d4c0a8ff0 0x7f9d4c0ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.728+0000 7f9d54233640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9d4c0a6290 0x7f9d4c0a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.728+0000 7f9d54233640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0c1f90 con 0x7f9d4c0ab730 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.728+0000 7f9d54233640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c14f850 con 0x7f9d4c0a8ff0 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.728+0000 7f9d54233640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c156260 con 0x7f9d4c0a6290 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d51fcf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9d4c0a8ff0 0x7f9d4c0ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d51fcf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9d4c0a8ff0 0x7f9d4c0ab1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48172/0 (socket says 172.21.15.94:48172) 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d51fcf640 1 -- 172.21.15.94:0/3819121099 learned_addr learned my addr 172.21.15.94:0/3819121099 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d517ce640 1 --2- 172.21.15.94:0/3819121099 >> v2:172.21.15.145:3300/0 conn(0x7f9d4c0a6290 0x7f9d4c0a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d51fcf640 1 -- 172.21.15.94:0/3819121099 >> v2:172.21.15.145:3300/0 conn(0x7f9d4c0a6290 msgr2=0x7f9d4c0a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d51fcf640 1 --2- 172.21.15.94:0/3819121099 >> v2:172.21.15.145:3300/0 conn(0x7f9d4c0a6290 0x7f9d4c0a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d51fcf640 1 -- 172.21.15.94:0/3819121099 >> v1:172.21.15.145:6789/0 conn(0x7f9d4c0ab730 legacy=0x7f9d4c0aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:24.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.729+0000 7f9d51fcf640 1 -- 172.21.15.94:0/3819121099 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0c11d0 con 0x7f9d4c0a8ff0 2024-03-06T07:26:24.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d51fcf640 1 --2- 172.21.15.94:0/3819121099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0a8ff0 0x7f9d4c0ab1f0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c06e560 tx=0x7f9d4802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2e2a3b70f566ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d50fcd640 1 -- 172.21.15.94:0/3819121099 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4803b070 con 0x7f9d4c0a8ff0 2024-03-06T07:26:24.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d50fcd640 1 -- 172.21.15.94:0/3819121099 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4802ee00 con 0x7f9d4c0a8ff0 2024-03-06T07:26:24.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d50fcd640 1 -- 172.21.15.94:0/3819121099 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d48036660 con 0x7f9d4c0a8ff0 2024-03-06T07:26:24.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 -- 172.21.15.94:0/3819121099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0a8ff0 msgr2=0x7f9d4c0ab1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 --2- 172.21.15.94:0/3819121099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0a8ff0 0x7f9d4c0ab1f0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c06e560 tx=0x7f9d4802e050 comp rx=0 tx=0).stop 2024-03-06T07:26:24.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 -- 172.21.15.94:0/3819121099 shutdown_connections 2024-03-06T07:26:24.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 --2- 172.21.15.94:0/3819121099 >> v2:172.21.15.145:3300/0 conn(0x7f9d4c0a6290 0x7f9d4c0a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 --2- 172.21.15.94:0/3819121099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0a8ff0 0x7f9d4c0ab1f0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 -- 172.21.15.94:0/3819121099 >> 172.21.15.94:0/3819121099 conn(0x7f9d4c06afc0 msgr2=0x7f9d4c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:24.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 -- 172.21.15.94:0/3819121099 shutdown_connections 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.730+0000 7f9d54233640 1 -- 172.21.15.94:0/3819121099 wait complete. 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.731+0000 7f9d54233640 1 Processor -- start 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d54233640 1 -- start start 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d54233640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d4c0a8ff0 0x7f9d4c071060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d54233640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0ab730 0x7f9d4c148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d54233640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d4c14e6b0 0x7f9d4c14eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d54233640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0c1a70 con 0x7f9d4c0ab730 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d54233640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0c1790 con 0x7f9d4c0a8ff0 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d54233640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c155cf0 con 0x7f9d4c14e6b0 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0ab730 0x7f9d4c148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0ab730 0x7f9d4c148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48184/0 (socket says 172.21.15.94:48184) 2024-03-06T07:26:24.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 -- 172.21.15.94:0/429681089 learned_addr learned my addr 172.21.15.94:0/429681089 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d51fcf640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d4c0a8ff0 0x7f9d4c071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d527d0640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d4c14e6b0 0x7f9d4c14eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 -- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d4c14e6b0 msgr2=0x7f9d4c14eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d4c14e6b0 0x7f9d4c14eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 -- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d4c0a8ff0 msgr2=0x7f9d4c071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d4c0a8ff0 0x7f9d4c071060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.732+0000 7f9d517ce640 1 -- 172.21.15.94:0/429681089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c153830 con 0x7f9d4c0ab730 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.733+0000 7f9d527d0640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d4c14e6b0 0x7f9d4c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.733+0000 7f9d517ce640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0ab730 0x7f9d4c148020 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00d830 tx=0x7f9d3c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.733+0000 7f9d427fc640 1 -- 172.21.15.94:0/429681089 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c00b6c0 con 0x7f9d4c0ab730 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.733+0000 7f9d427fc640 1 -- 172.21.15.94:0/429681089 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c019070 con 0x7f9d4c0ab730 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.733+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c1530e0 con 0x7f9d4c0ab730 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.733+0000 7f9d427fc640 1 -- 172.21.15.94:0/429681089 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c00ba80 con 0x7f9d4c0ab730 2024-03-06T07:26:24.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.733+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c072540 con 0x7f9d4c0ab730 2024-03-06T07:26:24.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.734+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c04f330 con 0x7f9d4c0ab730 2024-03-06T07:26:24.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.734+0000 7f9d427fc640 1 -- 172.21.15.94:0/429681089 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d3c01e460 con 0x7f9d4c0ab730 2024-03-06T07:26:24.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.735+0000 7f9d427fc640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d3006b500 0x7f9d3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:24.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.735+0000 7f9d427fc640 1 -- 172.21.15.94:0/429681089 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f9d3c08a850 con 0x7f9d4c0ab730 2024-03-06T07:26:24.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.735+0000 7f9d51fcf640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d3006b500 0x7f9d3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:24.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.735+0000 7f9d51fcf640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d3006b500 0x7f9d3006d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9d48038000 tx=0x7f9d48032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:24.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.739+0000 7f9d427fc640 1 -- 172.21.15.94:0/429681089 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d4c153830 con 0x7f9d4c0ab730 2024-03-06T07:26:24.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:24.948+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9d4c153df0 con 0x7f9d4c0ab730 2024-03-06T07:26:25.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.617+0000 7f9d427fc640 1 -- 172.21.15.94:0/429681089 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v184) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9d3c059630 con 0x7f9d4c0ab730 2024-03-06T07:26:25.619 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 31 and data pool 32 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d3006b500 msgr2=0x7f9d3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d3006b500 0x7f9d3006d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9d48038000 tx=0x7f9d48032000 comp rx=0 tx=0).stop 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0ab730 msgr2=0x7f9d4c148020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0ab730 0x7f9d4c148020 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00d830 tx=0x7f9d3c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 shutdown_connections 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9d3006b500 0x7f9d3006d9c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d4c14e6b0 0x7f9d4c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9d4c0ab730 0x7f9d4c148020 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 --2- 172.21.15.94:0/429681089 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d4c0a8ff0 0x7f9d4c071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 >> 172.21.15.94:0/429681089 conn(0x7f9d4c06afc0 msgr2=0x7f9d4c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.623+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 shutdown_connections 2024-03-06T07:26:25.626 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.624+0000 7f9d54233640 1 -- 172.21.15.94:0/429681089 wait complete. 2024-03-06T07:26:25.645 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:25.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.746+0000 7f574dc64640 1 Processor -- start 2024-03-06T07:26:25.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.746+0000 7f574dc64640 1 -- start start 2024-03-06T07:26:25.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.746+0000 7f574dc64640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f57480c1aa0 0x7f57481529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.746+0000 7f574dc64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f5748155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f574dc64640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57481563d0 con 0x7f574814f790 2024-03-06T07:26:25.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f574dc64640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57480c2680 con 0x7f57480c1aa0 2024-03-06T07:26:25.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f574dc64640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c28d0 con 0x7f5748155a40 2024-03-06T07:26:25.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f57477fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f574814f790 0x7f57480c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38752/0 (socket says 172.21.15.94:38752) 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f57477fe640 1 -- 172.21.15.94:0/1099717275 learned_addr learned my addr 172.21.15.94:0/1099717275 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f5747fff640 1 --2- 172.21.15.94:0/1099717275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f5748155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f573effd640 1 --2- 172.21.15.94:0/1099717275 >> v2:172.21.15.145:3300/0 conn(0x7f57480c1aa0 0x7f57481529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f5746ffd640 1 -- 172.21.15.94:0/1099717275 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919587298 0 0) 0x7f57481563d0 con 0x7f574814f790 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f5746ffd640 1 -- 172.21.15.94:0/1099717275 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5734003520 con 0x7f574814f790 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f5747fff640 1 -- 172.21.15.94:0/1099717275 >> v2:172.21.15.145:3300/0 conn(0x7f57480c1aa0 msgr2=0x7f57481529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f5747fff640 1 --2- 172.21.15.94:0/1099717275 >> v2:172.21.15.145:3300/0 conn(0x7f57480c1aa0 0x7f57481529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f5747fff640 1 -- 172.21.15.94:0/1099717275 >> v1:172.21.15.94:6789/0 conn(0x7f574814f790 legacy=0x7f57480c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f5747fff640 1 -- 172.21.15.94:0/1099717275 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480c2bb0 con 0x7f5748155a40 2024-03-06T07:26:25.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.747+0000 7f573effd640 1 --2- 172.21.15.94:0/1099717275 >> v2:172.21.15.145:3300/0 conn(0x7f57480c1aa0 0x7f57481529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.748+0000 7f5747fff640 1 --2- 172.21.15.94:0/1099717275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f5748155e90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5740009d90 tx=0x7f574002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=178845970900bc70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.748+0000 7f5746ffd640 1 -- 172.21.15.94:0/1099717275 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740039070 con 0x7f5748155a40 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.748+0000 7f5746ffd640 1 -- 172.21.15.94:0/1099717275 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574002cc50 con 0x7f5748155a40 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.748+0000 7f5746ffd640 1 -- 172.21.15.94:0/1099717275 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740034440 con 0x7f5748155a40 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 -- 172.21.15.94:0/1099717275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 msgr2=0x7f5748155e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 --2- 172.21.15.94:0/1099717275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f5748155e90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5740009d90 tx=0x7f574002bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 -- 172.21.15.94:0/1099717275 shutdown_connections 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 --2- 172.21.15.94:0/1099717275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f5748155e90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 --2- 172.21.15.94:0/1099717275 >> v2:172.21.15.145:3300/0 conn(0x7f57480c1aa0 0x7f57481529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 -- 172.21.15.94:0/1099717275 >> 172.21.15.94:0/1099717275 conn(0x7f57480c6710 msgr2=0x7f57480c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 -- 172.21.15.94:0/1099717275 shutdown_connections 2024-03-06T07:26:25.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.749+0000 7f574dc64640 1 -- 172.21.15.94:0/1099717275 wait complete. 2024-03-06T07:26:25.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.750+0000 7f574dc64640 1 Processor -- start 2024-03-06T07:26:25.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.750+0000 7f574dc64640 1 -- start start 2024-03-06T07:26:25.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f574dc64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57480c1aa0 0x7f57481eca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f574dc64640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f574814f790 0x7f57481542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f574dc64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f57481e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f574dc64640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c0d60 con 0x7f57480c1aa0 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f574dc64640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748155520 con 0x7f574814f790 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f574dc64640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c5e30 con 0x7f5748155a40 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57480c1aa0 0x7f57481eca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57480c1aa0 0x7f57481eca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48192/0 (socket says 172.21.15.94:48192) 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 -- 172.21.15.94:0/2595269245 learned_addr learned my addr 172.21.15.94:0/2595269245 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f573effd640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f574814f790 0x7f57481542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f5747fff640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f57481e8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 -- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 msgr2=0x7f57481e8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f57481e8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 -- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f574814f790 msgr2=0x7f57481542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f574814f790 0x7f57481542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 -- 172.21.15.94:0/2595269245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5748072a80 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f573effd640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f574814f790 0x7f57481542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f5747fff640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f57481e8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f57477fe640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57480c1aa0 0x7f57481eca20 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f573800c530 tx=0x7f573800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f5744ff9640 1 -- 172.21.15.94:0/2595269245 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738017070 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f5744ff9640 1 -- 172.21.15.94:0/2595269245 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738005ce0 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.751+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57481ec220 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.752+0000 7f5744ff9640 1 -- 172.21.15.94:0/2595269245 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380072b0 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.752+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57481e8fe0 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.752+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57481ec010 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.753+0000 7f5744ff9640 1 -- 172.21.15.94:0/2595269245 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573801a070 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.753+0000 7f5744ff9640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.753+0000 7f5744ff9640 1 -- 172.21.15.94:0/2595269245 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f57380897d0 con 0x7f57480c1aa0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.754+0000 7f573effd640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.754+0000 7f573effd640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f572806b4d0 0x7f572806d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f574815cc70 tx=0x7f5730008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.758+0000 7f5744ff9640 1 -- 172.21.15.94:0/2595269245 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5738058590 con 0x7f57480c1aa0 2024-03-06T07:26:25.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.929+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f574804f330 con 0x7f57480c1aa0 2024-03-06T07:26:25.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.931+0000 7f5744ff9640 1 -- 172.21.15.94:0/2595269245 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+16965 (secure 0 0 0) 0x7f5738057e90 con 0x7f57480c1aa0 2024-03-06T07:26:25.932 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:25.933 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":195,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:24.949509+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:23.168853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:23.918870+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f572806b4d0 msgr2=0x7f572806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f572806b4d0 0x7f572806d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f574815cc70 tx=0x7f5730008040 comp rx=0 tx=0).stop 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57480c1aa0 msgr2=0x7f57481eca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57480c1aa0 0x7f57481eca20 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f573800c530 tx=0x7f573800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 shutdown_connections 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f572806b4d0 0x7f572806d990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5748155a40 0x7f57481e8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f574814f790 0x7f57481542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 --2- 172.21.15.94:0/2595269245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57480c1aa0 0x7f57481eca20 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 >> 172.21.15.94:0/2595269245 conn(0x7f57480c6710 msgr2=0x7f574806d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 shutdown_connections 2024-03-06T07:26:25.935 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:25.933+0000 7f574dc64640 1 -- 172.21.15.94:0/2595269245 wait complete. 2024-03-06T07:26:25.946 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:26.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.044+0000 7f37628a4640 1 Processor -- start 2024-03-06T07:26:26.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.045+0000 7f37628a4640 1 -- start start 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.045+0000 7f37628a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 0x7f375c159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.045+0000 7f37628a4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c0c7c60 con 0x7f375c1527b0 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.045+0000 7f37628a4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c0c8060 con 0x7f375c0c88b0 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.045+0000 7f37628a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c159640 con 0x7f375c0c7180 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.045+0000 7f375bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f375c1527b0 0x7f375c155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38768/0 (socket says 172.21.15.94:38768) 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.045+0000 7f375bfff640 1 -- 172.21.15.94:0/1628273800 learned_addr learned my addr 172.21.15.94:0/1628273800 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420438573 0 0) 0x7f3744001110 con 0x7f375c1527b0 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375b7fe640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 0x7f375c159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3748003520 con 0x7f375c1527b0 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2457318089 0 0) 0x7f3748003520 con 0x7f375c1527b0 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 msgr2=0x7f375c159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 0x7f375c159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 >> v1:172.21.15.145:6789/0 conn(0x7f375c0c88b0 legacy=0x7f375c0c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c0c76a0 con 0x7f375c1527b0 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f3744001840 con 0x7f375c1527b0 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 >> v1:172.21.15.94:6789/0 conn(0x7f375c1527b0 legacy=0x7f375c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37480039e0 0x7f3748003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 0x7f3748004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f37480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c159640 con 0x7f3748004d40 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374800baf0 con 0x7f37480039e0 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374800bcf0 con 0x7f375c0c7180 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375bfff640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37480039e0 0x7f3748003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f3760e41640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f37480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f375b7fe640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 0x7f3748004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f3760e41640 1 -- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 msgr2=0x7f3748004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f3760e41640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c0c7180 0x7f3748004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f3760e41640 1 -- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37480039e0 msgr2=0x7f3748003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f3760e41640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37480039e0 0x7f3748003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.046+0000 7f3760e41640 1 -- 172.21.15.94:0/1628273800 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3748003520 con 0x7f3748004d40 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.047+0000 7f3760e41640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f37480051c0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f3754001e50 tx=0x7f3748005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59c667887a91667 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.047+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754001a70 con 0x7f3748004d40 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.047+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3754003b70 con 0x7f3748004d40 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.047+0000 7f375affd640 1 -- 172.21.15.94:0/1628273800 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37540050d0 con 0x7f3748004d40 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.048+0000 7f37628a4640 1 -- 172.21.15.94:0/1628273800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 msgr2=0x7f37480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.048+0000 7f37628a4640 1 --2- 172.21.15.94:0/1628273800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f37480051c0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f3754001e50 tx=0x7f3748005830 comp rx=0 tx=0).stop 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.048+0000 7f3760e41640 1 -- 172.21.15.94:0/1628273800 reap_dead start 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.048+0000 7f37628a4640 1 -- 172.21.15.94:0/1628273800 shutdown_connections 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.048+0000 7f37628a4640 1 -- 172.21.15.94:0/1628273800 >> 172.21.15.94:0/1628273800 conn(0x7f375c0ac560 msgr2=0x7f375c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.048+0000 7f37628a4640 1 -- 172.21.15.94:0/1628273800 shutdown_connections 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.048+0000 7f37628a4640 1 -- 172.21.15.94:0/1628273800 wait complete. 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.049+0000 7f37628a4640 1 Processor -- start 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.049+0000 7f37628a4640 1 -- start start 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f37628a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f375c14ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f37628a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c1564d0 0x7f375c1eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f37628a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f375c0ab8a0 0x7f375c152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f37628a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c155600 con 0x7f3748004d40 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f37628a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c155320 con 0x7f375c0ab8a0 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f37628a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0c8d60 con 0x7f375c1564d0 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f375c14ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f375c14ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48210/0 (socket says 172.21.15.94:48210) 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 -- 172.21.15.94:0/2059592623 learned_addr learned my addr 172.21.15.94:0/2059592623 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375b7fe640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c1564d0 0x7f375c1eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f3760e41640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f375c0ab8a0 0x7f375c152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 -- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c1564d0 msgr2=0x7f375c1eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c1564d0 0x7f375c1eb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 -- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f375c0ab8a0 msgr2=0x7f375c152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f375c0ab8a0 0x7f375c152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 -- 172.21.15.94:0/2059592623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c154bc0 con 0x7f3748004d40 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375b7fe640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c1564d0 0x7f375c1eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.050+0000 7f375bfff640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f375c14ed10 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f3744007290 tx=0x7f3744007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.051+0000 7f3758ff9640 1 -- 172.21.15.94:0/2059592623 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374400f9c0 con 0x7f3748004d40 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.051+0000 7f3758ff9640 1 -- 172.21.15.94:0/2059592623 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3744018070 con 0x7f3748004d40 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.051+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c154ea0 con 0x7f3748004d40 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.051+0000 7f3758ff9640 1 -- 172.21.15.94:0/2059592623 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374400fda0 con 0x7f3748004d40 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.051+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c14e510 con 0x7f3748004d40 2024-03-06T07:26:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.052+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c04f330 con 0x7f3748004d40 2024-03-06T07:26:26.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.052+0000 7f3758ff9640 1 -- 172.21.15.94:0/2059592623 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f374401c410 con 0x7f3748004d40 2024-03-06T07:26:26.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.053+0000 7f3758ff9640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f373c06b500 0x7f373c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.053+0000 7f3758ff9640 1 -- 172.21.15.94:0/2059592623 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f374408a090 con 0x7f3748004d40 2024-03-06T07:26:26.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.053+0000 7f375b7fe640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f373c06b500 0x7f373c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.053+0000 7f375b7fe640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f373c06b500 0x7f373c06d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f375c157050 tx=0x7f374c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.057+0000 7f3758ff9640 1 -- 172.21.15.94:0/2059592623 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f375c0c7c60 con 0x7f3748004d40 2024-03-06T07:26:26.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.228+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f375c06ce00 con 0x7f3748004d40 2024-03-06T07:26:26.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.229+0000 7f3758ff9640 1 -- 172.21.15.94:0/2059592623 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+16965 (secure 0 0 0) 0x7f3744081e20 con 0x7f3748004d40 2024-03-06T07:26:26.231 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:26.231 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":195,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:24.949509+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:23.168853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:23.918870+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f373c06b500 msgr2=0x7f373c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f373c06b500 0x7f373c06d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f375c157050 tx=0x7f374c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 msgr2=0x7f375c14ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f375c14ed10 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f3744007290 tx=0x7f3744007750 comp rx=0 tx=0).stop 2024-03-06T07:26:26.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 shutdown_connections 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f373c06b500 0x7f373c06d9c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f375c0ab8a0 0x7f375c152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f375c1564d0 0x7f375c1eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 --2- 172.21.15.94:0/2059592623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3748004d40 0x7f375c14ed10 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 >> 172.21.15.94:0/2059592623 conn(0x7f375c0ac560 msgr2=0x7f375c0c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 shutdown_connections 2024-03-06T07:26:26.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.231+0000 7f37628a4640 1 -- 172.21.15.94:0/2059592623 wait complete. 2024-03-06T07:26:26.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.346+0000 7f430df3e640 1 Processor -- start 2024-03-06T07:26:26.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.346+0000 7f430df3e640 1 -- start start 2024-03-06T07:26:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f430df3e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f430814ef70 0x7f430806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f430df3e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43080c2790 con 0x7f43080c1820 2024-03-06T07:26:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f430df3e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430806cf40 con 0x7f430806cad0 2024-03-06T07:26:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f430df3e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43080c3090 con 0x7f430814ef70 2024-03-06T07:26:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f43077fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f430814ef70 0x7f430806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f4306ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f43080c1820 0x7f43080c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38774/0 (socket says 172.21.15.94:38774) 2024-03-06T07:26:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f4306ffd640 1 -- 172.21.15.94:0/26020376 learned_addr learned my addr 172.21.15.94:0/26020376 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f43077fe640 1 -- 172.21.15.94:0/26020376 >> v1:172.21.15.145:6789/0 conn(0x7f430806cad0 legacy=0x7f4308154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f43077fe640 1 -- 172.21.15.94:0/26020376 >> v1:172.21.15.94:6789/0 conn(0x7f43080c1820 legacy=0x7f43080c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f43067fc640 1 -- 172.21.15.94:0/26020376 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378050 0 0) 0x7f43080c2790 con 0x7f43080c1820 2024-03-06T07:26:26.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.347+0000 7f43077fe640 1 -- 172.21.15.94:0/26020376 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080c1540 con 0x7f430814ef70 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.348+0000 7f43077fe640 1 --2- 172.21.15.94:0/26020376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430814ef70 0x7f430806c1f0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f42f800ba40 tx=0x7f42f802ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1851d68a68713e22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.348+0000 7f43067fc640 1 -- 172.21.15.94:0/26020376 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8034070 con 0x7f430814ef70 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.348+0000 7f43067fc640 1 -- 172.21.15.94:0/26020376 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f802eba0 con 0x7f430814ef70 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.348+0000 7f43067fc640 1 -- 172.21.15.94:0/26020376 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8038400 con 0x7f430814ef70 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.349+0000 7f430df3e640 1 -- 172.21.15.94:0/26020376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430814ef70 msgr2=0x7f430806c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.349+0000 7f430df3e640 1 --2- 172.21.15.94:0/26020376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430814ef70 0x7f430806c1f0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f42f800ba40 tx=0x7f42f802ddf0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.349+0000 7f430df3e640 1 -- 172.21.15.94:0/26020376 shutdown_connections 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.349+0000 7f430df3e640 1 --2- 172.21.15.94:0/26020376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430814ef70 0x7f430806c1f0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.349+0000 7f430df3e640 1 -- 172.21.15.94:0/26020376 >> 172.21.15.94:0/26020376 conn(0x7f43080c6710 msgr2=0x7f43080c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.349+0000 7f430df3e640 1 -- 172.21.15.94:0/26020376 shutdown_connections 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.349+0000 7f430df3e640 1 -- 172.21.15.94:0/26020376 wait complete. 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.350+0000 7f430df3e640 1 Processor -- start 2024-03-06T07:26:26.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.350+0000 7f430df3e640 1 -- start start 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430806cad0 0x7f430814d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43080c1820 0x7f4308140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f430814ef70 0x7f4308140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4308155750 con 0x7f430806cad0 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430814f350 con 0x7f430814ef70 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4308155340 con 0x7f43080c1820 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430806cad0 0x7f430814d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430806cad0 0x7f430814d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48220/0 (socket says 172.21.15.94:48220) 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 -- 172.21.15.94:0/1377272509 learned_addr learned my addr 172.21.15.94:0/1377272509 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 -- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43080c1820 msgr2=0x7f4308140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43080c1820 0x7f4308140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 -- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f430814ef70 msgr2=0x7f4308140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f430814ef70 0x7f4308140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 -- 172.21.15.94:0/1377272509 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308152380 con 0x7f430806cad0 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f43077fe640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430806cad0 0x7f430814d510 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f42f800ba40 tx=0x7f42f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f42effff640 1 -- 172.21.15.94:0/1377272509 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8047030 con 0x7f430806cad0 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f42effff640 1 -- 172.21.15.94:0/1377272509 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f8043680 con 0x7f430806cad0 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43080a8590 con 0x7f430806cad0 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f42effff640 1 -- 172.21.15.94:0/1377272509 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8043a50 con 0x7f430806cad0 2024-03-06T07:26:26.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.351+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430814da50 con 0x7f430806cad0 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.352+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430804f330 con 0x7f430806cad0 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.353+0000 7f42effff640 1 -- 172.21.15.94:0/1377272509 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42f804a070 con 0x7f430806cad0 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.353+0000 7f42effff640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42f406b4d0 0x7f42f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.353+0000 7f42effff640 1 -- 172.21.15.94:0/1377272509 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f42f80af030 con 0x7f430806cad0 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.353+0000 7f4306ffd640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42f406b4d0 0x7f42f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.354+0000 7f4306ffd640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42f406b4d0 0x7f42f406d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f42fc007fd0 tx=0x7f42fc007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.357+0000 7f42effff640 1 -- 172.21.15.94:0/1377272509 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42f8051070 con 0x7f430806cad0 2024-03-06T07:26:26.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.563+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f430806d7c0 con 0x7f430806cad0 2024-03-06T07:26:26.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.618+0000 7f42effff640 1 -- 172.21.15.94:0/1377272509 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v185) v1 ==== 118+0+0 (secure 0 0 0) 0x7f430806cf40 con 0x7f430806cad0 2024-03-06T07:26:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42f406b4d0 msgr2=0x7f42f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42f406b4d0 0x7f42f406d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f42fc007fd0 tx=0x7f42fc007920 comp rx=0 tx=0).stop 2024-03-06T07:26:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430806cad0 msgr2=0x7f430814d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430806cad0 0x7f430814d510 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f42f800ba40 tx=0x7f42f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 shutdown_connections 2024-03-06T07:26:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f42f406b4d0 0x7f42f406d990 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.623 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f430814ef70 0x7f4308140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f43080c1820 0x7f4308140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 --2- 172.21.15.94:0/1377272509 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f430806cad0 0x7f430814d510 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 >> 172.21.15.94:0/1377272509 conn(0x7f43080c6710 msgr2=0x7f4308153dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 shutdown_connections 2024-03-06T07:26:26.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.621+0000 7f430df3e640 1 -- 172.21.15.94:0/1377272509 wait complete. 2024-03-06T07:26:26.634 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:26.634 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.735+0000 7fcdfe239640 1 Processor -- start 2024-03-06T07:26:26.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.735+0000 7fcdfe239640 1 -- start start 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.735+0000 7fcdfe239640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.735+0000 7fcdfe239640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcdf80ab170 0x7fcdf80a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.735+0000 7fcdfe239640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf80c3ef0 con 0x7fcdf80c5990 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.735+0000 7fcdfe239640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf81556d0 con 0x7fcdf80ab170 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.735+0000 7fcdfe239640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf81559b0 con 0x7fcdf806c900 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcdf80ab170 0x7fcdf80a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf7fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcdf80ab170 0x7fcdf80a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48228/0 (socket says 172.21.15.94:48228) 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf7fff640 1 -- 172.21.15.94:0/645836137 learned_addr learned my addr 172.21.15.94:0/645836137 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf6ffd640 1 --2- 172.21.15.94:0/645836137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf7fff640 1 -- 172.21.15.94:0/645836137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 msgr2=0x7fcdf80c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf7fff640 1 --2- 172.21.15.94:0/645836137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf7fff640 1 -- 172.21.15.94:0/645836137 >> v1:172.21.15.145:6789/0 conn(0x7fcdf80c5990 legacy=0x7fcdf806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.736+0000 7fcdf7fff640 1 -- 172.21.15.94:0/645836137 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8155c90 con 0x7fcdf80ab170 2024-03-06T07:26:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdf7fff640 1 --2- 172.21.15.94:0/645836137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80ab170 0x7fcdf80a9fa0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009e50 tx=0x7fcdf002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce3a579177ed729 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdf67fc640 1 -- 172.21.15.94:0/645836137 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf002cb00 con 0x7fcdf80ab170 2024-03-06T07:26:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdf67fc640 1 -- 172.21.15.94:0/645836137 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf002cca0 con 0x7fcdf80ab170 2024-03-06T07:26:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdf67fc640 1 -- 172.21.15.94:0/645836137 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0034610 con 0x7fcdf80ab170 2024-03-06T07:26:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 -- 172.21.15.94:0/645836137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80ab170 msgr2=0x7fcdf80a9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 --2- 172.21.15.94:0/645836137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80ab170 0x7fcdf80a9fa0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009e50 tx=0x7fcdf002bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:26.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 -- 172.21.15.94:0/645836137 shutdown_connections 2024-03-06T07:26:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 --2- 172.21.15.94:0/645836137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80ab170 0x7fcdf80a9fa0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 --2- 172.21.15.94:0/645836137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 -- 172.21.15.94:0/645836137 >> 172.21.15.94:0/645836137 conn(0x7fcdf80c6710 msgr2=0x7fcdf80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 -- 172.21.15.94:0/645836137 shutdown_connections 2024-03-06T07:26:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.737+0000 7fcdfe239640 1 -- 172.21.15.94:0/645836137 wait complete. 2024-03-06T07:26:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.738+0000 7fcdfe239640 1 Processor -- start 2024-03-06T07:26:26.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdfe239640 1 -- start start 2024-03-06T07:26:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdfe239640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80a87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdfe239640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80ab170 0x7fcdf813c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdfe239640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80c5990 0x7fcdf813cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdfe239640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8156610 con 0x7fcdf80c5990 2024-03-06T07:26:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdfe239640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8156490 con 0x7fcdf80ab170 2024-03-06T07:26:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdfe239640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80c3af0 con 0x7fcdf806c900 2024-03-06T07:26:26.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdf7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80c5990 0x7fcdf813cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdf7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80c5990 0x7fcdf813cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48230/0 (socket says 172.21.15.94:48230) 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdf7fff640 1 -- 172.21.15.94:0/4008495266 learned_addr learned my addr 172.21.15.94:0/4008495266 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdf77fe640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80a87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.739+0000 7fcdf6ffd640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80ab170 0x7fcdf813c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdf7fff640 1 -- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 msgr2=0x7fcdf80a87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdf7fff640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80a87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdf7fff640 1 -- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80ab170 msgr2=0x7fcdf813c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdf7fff640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80ab170 0x7fcdf813c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdf7fff640 1 -- 172.21.15.94:0/4008495266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf813d080 con 0x7fcdf80c5990 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdf7fff640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80c5990 0x7fcdf813cb40 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0036040 tx=0x7fcdf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcddffff640 1 -- 172.21.15.94:0/4008495266 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0043020 con 0x7fcdf80c5990 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcddffff640 1 -- 172.21.15.94:0/4008495266 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf002ce50 con 0x7fcdf80c5990 2024-03-06T07:26:26.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf81ef520 con 0x7fcdf80c5990 2024-03-06T07:26:26.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcddffff640 1 -- 172.21.15.94:0/4008495266 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0009510 con 0x7fcdf80c5990 2024-03-06T07:26:26.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.740+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf81eea40 con 0x7fcdf80c5990 2024-03-06T07:26:26.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.741+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf8075300 con 0x7fcdf80c5990 2024-03-06T07:26:26.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.741+0000 7fcddffff640 1 -- 172.21.15.94:0/4008495266 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdf0046020 con 0x7fcdf80c5990 2024-03-06T07:26:26.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.742+0000 7fcddffff640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcde406b4d0 0x7fcde406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.742+0000 7fcddffff640 1 -- 172.21.15.94:0/4008495266 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcdf00ac730 con 0x7fcdf80c5990 2024-03-06T07:26:26.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.742+0000 7fcdf77fe640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcde406b4d0 0x7fcde406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.742+0000 7fcdf77fe640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcde406b4d0 0x7fcde406d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fcde8001780 tx=0x7fcde8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.746+0000 7fcddffff640 1 -- 172.21.15.94:0/4008495266 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf8075300 con 0x7fcdf80c5990 2024-03-06T07:26:26.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:26.954+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcdf80abc20 con 0x7fcdf80c5990 2024-03-06T07:26:27.624 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.622+0000 7fcddffff640 1 -- 172.21.15.94:0/4008495266 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v186) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcdf80abc20 con 0x7fcdf80c5990 2024-03-06T07:26:27.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcde406b4d0 msgr2=0x7fcde406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcde406b4d0 0x7fcde406d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fcde8001780 tx=0x7fcde8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:27.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80c5990 msgr2=0x7fcdf813cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80c5990 0x7fcdf813cb40 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0036040 tx=0x7fcdf0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:27.628 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 shutdown_connections 2024-03-06T07:26:27.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcde406b4d0 0x7fcde406d990 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf80c5990 0x7fcdf813cb40 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80ab170 0x7fcdf813c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 --2- 172.21.15.94:0/4008495266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf806c900 0x7fcdf80a87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 >> 172.21.15.94:0/4008495266 conn(0x7fcdf80c6710 msgr2=0x7fcdf8148d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 shutdown_connections 2024-03-06T07:26:27.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.626+0000 7fcdfe239640 1 -- 172.21.15.94:0/4008495266 wait complete. 2024-03-06T07:26:27.641 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:27.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:27.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24faec9640 1 Processor -- start 2024-03-06T07:26:27.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24faec9640 1 -- start start 2024-03-06T07:26:27.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24faec9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24faec9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24f4155a40 0x7f24f4155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24faec9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f4156410 con 0x7f24f414f790 2024-03-06T07:26:27.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24faec9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f40c2680 con 0x7f24f4155a40 2024-03-06T07:26:27.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24faec9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40c28d0 con 0x7f24f40c1aa0 2024-03-06T07:26:27.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.745+0000 7f24f9466640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24f4155a40 0x7f24f4155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f9466640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24f4155a40 0x7f24f4155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48244/0 (socket says 172.21.15.94:48244) 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f9466640 1 -- 172.21.15.94:0/1561355057 learned_addr learned my addr 172.21.15.94:0/1561355057 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f3fff640 1 --2- 172.21.15.94:0/1561355057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f9466640 1 -- 172.21.15.94:0/1561355057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 msgr2=0x7f24f41529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f9466640 1 --2- 172.21.15.94:0/1561355057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f41529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f9466640 1 -- 172.21.15.94:0/1561355057 >> v1:172.21.15.145:6789/0 conn(0x7f24f414f790 legacy=0x7f24f40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f9466640 1 -- 172.21.15.94:0/1561355057 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f40c2bb0 con 0x7f24f4155a40 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f37fe640 1 -- 172.21.15.94:0/1561355057 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958966491 0 0) 0x7f24f4156410 con 0x7f24f414f790 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.746+0000 7f24f9466640 1 --2- 172.21.15.94:0/1561355057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f4155a40 0x7f24f4155ed0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009870 tx=0x7f24e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c134f41bf48153 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24f37fe640 1 -- 172.21.15.94:0/1561355057 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4034070 con 0x7f24f4155a40 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24f37fe640 1 -- 172.21.15.94:0/1561355057 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e4030840 con 0x7f24f4155a40 2024-03-06T07:26:27.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24f37fe640 1 -- 172.21.15.94:0/1561355057 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4030c10 con 0x7f24f4155a40 2024-03-06T07:26:27.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 -- 172.21.15.94:0/1561355057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f4155a40 msgr2=0x7f24f4155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 --2- 172.21.15.94:0/1561355057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f4155a40 0x7f24f4155ed0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009870 tx=0x7f24e402b980 comp rx=0 tx=0).stop 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 -- 172.21.15.94:0/1561355057 shutdown_connections 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 --2- 172.21.15.94:0/1561355057 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f4155a40 0x7f24f4155ed0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 --2- 172.21.15.94:0/1561355057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f41529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 -- 172.21.15.94:0/1561355057 >> 172.21.15.94:0/1561355057 conn(0x7f24f40c6710 msgr2=0x7f24f40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 -- 172.21.15.94:0/1561355057 shutdown_connections 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.747+0000 7f24faec9640 1 -- 172.21.15.94:0/1561355057 wait complete. 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.748+0000 7f24faec9640 1 Processor -- start 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.748+0000 7f24faec9640 1 -- start start 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24faec9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f414e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24faec9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f414f790 0x7f24f41e9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24faec9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24f4155a40 0x7f24f41ea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24faec9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40c0d60 con 0x7f24f414f790 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24faec9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4155520 con 0x7f24f4155a40 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24faec9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40c5e30 con 0x7f24f40c1aa0 2024-03-06T07:26:27.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f414f790 0x7f24f41e9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24f3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f414f790 0x7f24f41e9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48250/0 (socket says 172.21.15.94:48250) 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24f3fff640 1 -- 172.21.15.94:0/742817705 learned_addr learned my addr 172.21.15.94:0/742817705 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24f8c65640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f414e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.749+0000 7f24f9466640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24f4155a40 0x7f24f41ea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f3fff640 1 -- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 msgr2=0x7f24f414e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f3fff640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f414e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f3fff640 1 -- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24f4155a40 msgr2=0x7f24f41ea4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f3fff640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24f4155a40 0x7f24f41ea4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f3fff640 1 -- 172.21.15.94:0/742817705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f414cd90 con 0x7f24f414f790 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f3fff640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f414f790 0x7f24f41e9f90 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00e9b0 tx=0x7f24dc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f17fa640 1 -- 172.21.15.94:0/742817705 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc017070 con 0x7f24f414f790 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f17fa640 1 -- 172.21.15.94:0/742817705 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc00c9a0 con 0x7f24f414f790 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24f17fa640 1 -- 172.21.15.94:0/742817705 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00cd90 con 0x7f24f414f790 2024-03-06T07:26:27.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f414d070 con 0x7f24f414f790 2024-03-06T07:26:27.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.750+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f414d910 con 0x7f24f414f790 2024-03-06T07:26:27.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.751+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f414d460 con 0x7f24f414f790 2024-03-06T07:26:27.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.751+0000 7f24f17fa640 1 -- 172.21.15.94:0/742817705 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24dc01a070 con 0x7f24f414f790 2024-03-06T07:26:27.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.752+0000 7f24f17fa640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24d406b4d0 0x7f24d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.752+0000 7f24f17fa640 1 -- 172.21.15.94:0/742817705 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24dc08a7b0 con 0x7f24f414f790 2024-03-06T07:26:27.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.752+0000 7f24f8c65640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24d406b4d0 0x7f24d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.752+0000 7f24f8c65640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24d406b4d0 0x7f24d406d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0012e0 tx=0x7f24ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.756+0000 7f24f17fa640 1 -- 172.21.15.94:0/742817705 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24dc058d40 con 0x7f24f414f790 2024-03-06T07:26:27.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.964+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24f404f330 con 0x7f24f414f790 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.964+0000 7f24f17fa640 1 -- 172.21.15.94:0/742817705 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+4995 (secure 0 0 0) 0x7f24dc05cde0 con 0x7f24f414f790 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":186,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10428,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2752830090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2752830090},{"type":"v1","addr":"172.21.15.94:6834","nonce":2752830090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":10431,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/750154668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":750154668},{"type":"v1","addr":"172.21.15.94:6835","nonce":750154668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":38057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/565245371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":565245371},{"type":"v1","addr":"172.21.15.145:6836","nonce":565245371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:24.949828+0000","modified":"2024-03-06T07:26:26.955353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38060},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38060":{"gid":38060,"name":"d","rank":0,"incarnation":184,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/4067343150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":4067343150},{"type":"v1","addr":"172.21.15.145:6837","nonce":4067343150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38060,"qdb_cluster":[38060]},"id":15}]} 2024-03-06T07:26:27.966 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 186 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24d406b4d0 msgr2=0x7f24d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24d406b4d0 0x7f24d406d990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0012e0 tx=0x7f24ec009040 comp rx=0 tx=0).stop 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f414f790 msgr2=0x7f24f41e9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f414f790 0x7f24f41e9f90 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00e9b0 tx=0x7f24dc00ee70 comp rx=0 tx=0).stop 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 shutdown_connections 2024-03-06T07:26:27.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24d406b4d0 0x7f24d406d990 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24f4155a40 0x7f24f41ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24f414f790 0x7f24f41e9f90 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 --2- 172.21.15.94:0/742817705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40c1aa0 0x7f24f414e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 >> 172.21.15.94:0/742817705 conn(0x7f24f40c6710 msgr2=0x7f24f40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 shutdown_connections 2024-03-06T07:26:27.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:27.966+0000 7f24faec9640 1 -- 172.21.15.94:0/742817705 wait complete. 2024-03-06T07:26:27.979 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:28.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.077+0000 7f8635f93640 1 Processor -- start 2024-03-06T07:26:28.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.077+0000 7f8635f93640 1 -- start start 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f8635f93640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f86300c1aa0 0x7f86301529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f8635f93640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300c28d0 con 0x7f863014f790 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f8635f93640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300c2bb0 con 0x7f8630155a40 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f8635f93640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86300c2680 con 0x7f86300c1aa0 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f863014f790 0x7f86300c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38782/0 (socket says 172.21.15.94:38782) 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862f7fe640 1 -- 172.21.15.94:0/3444847304 learned_addr learned my addr 172.21.15.94:0/3444847304 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862effd640 1 --2- 172.21.15.94:0/3444847304 >> v2:172.21.15.145:3300/0 conn(0x7f86300c1aa0 0x7f86301529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862e7fc640 1 -- 172.21.15.94:0/3444847304 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844488826 0 0) 0x7f86300c28d0 con 0x7f863014f790 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862e7fc640 1 -- 172.21.15.94:0/3444847304 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8618003520 con 0x7f863014f790 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862effd640 1 -- 172.21.15.94:0/3444847304 >> v1:172.21.15.145:6789/0 conn(0x7f8630155a40 legacy=0x7f8630155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862e7fc640 1 -- 172.21.15.94:0/3444847304 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542689531 0 0) 0x7f86300c2bb0 con 0x7f8630155a40 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862effd640 1 -- 172.21.15.94:0/3444847304 >> v1:172.21.15.94:6789/0 conn(0x7f863014f790 legacy=0x7f86300c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.078+0000 7f862effd640 1 -- 172.21.15.94:0/3444847304 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300c2e90 con 0x7f86300c1aa0 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.079+0000 7f862effd640 1 --2- 172.21.15.94:0/3444847304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 0x7f86301529d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f86300c2800 tx=0x7f862402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2b6b262a61bc312 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.079+0000 7f862e7fc640 1 -- 172.21.15.94:0/3444847304 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624034070 con 0x7f86300c1aa0 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.079+0000 7f862e7fc640 1 -- 172.21.15.94:0/3444847304 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624030610 con 0x7f86300c1aa0 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.079+0000 7f862e7fc640 1 -- 172.21.15.94:0/3444847304 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86240309e0 con 0x7f86300c1aa0 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.080+0000 7f8635f93640 1 -- 172.21.15.94:0/3444847304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 msgr2=0x7f86301529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.080+0000 7f8635f93640 1 --2- 172.21.15.94:0/3444847304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 0x7f86301529d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f86300c2800 tx=0x7f862402b960 comp rx=0 tx=0).stop 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.080+0000 7f8635f93640 1 -- 172.21.15.94:0/3444847304 shutdown_connections 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.080+0000 7f8635f93640 1 --2- 172.21.15.94:0/3444847304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 0x7f86301529d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.080+0000 7f8635f93640 1 -- 172.21.15.94:0/3444847304 >> 172.21.15.94:0/3444847304 conn(0x7f86300c6710 msgr2=0x7f86300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.080+0000 7f8635f93640 1 -- 172.21.15.94:0/3444847304 shutdown_connections 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.080+0000 7f8635f93640 1 -- 172.21.15.94:0/3444847304 wait complete. 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.081+0000 7f8635f93640 1 Processor -- start 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.081+0000 7f8635f93640 1 -- start start 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 0x7f86300c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f790 0x7f86301eb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8630155a40 0x7f86301eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86300c5e30 con 0x7f863014f790 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86300c0d60 con 0x7f86300c1aa0 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86300c5fb0 con 0x7f8630155a40 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f790 0x7f86301eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f790 0x7f86301eb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48262/0 (socket says 172.21.15.94:48262) 2024-03-06T07:26:28.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 -- 172.21.15.94:0/1847989924 learned_addr learned my addr 172.21.15.94:0/1847989924 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862f7fe640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 0x7f86300c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862ffff640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8630155a40 0x7f86301eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 -- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8630155a40 msgr2=0x7f86301eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8630155a40 0x7f86301eb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 -- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 msgr2=0x7f86300c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 0x7f86300c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 -- 172.21.15.94:0/1847989924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86301e9590 con 0x7f863014f790 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f862effd640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f790 0x7f86301eb000 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f8624009890 tx=0x7f86240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8617fff640 1 -- 172.21.15.94:0/1847989924 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624049070 con 0x7f863014f790 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8617fff640 1 -- 172.21.15.94:0/1847989924 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624030410 con 0x7f863014f790 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8617fff640 1 -- 172.21.15.94:0/1847989924 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624034040 con 0x7f863014f790 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86300705c0 con 0x7f863014f790 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.082+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8630070b80 con 0x7f863014f790 2024-03-06T07:26:28.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.083+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863004f330 con 0x7f863014f790 2024-03-06T07:26:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.084+0000 7f8617fff640 1 -- 172.21.15.94:0/1847989924 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f862404c070 con 0x7f863014f790 2024-03-06T07:26:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.084+0000 7f8617fff640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861c06b500 0x7f861c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.084+0000 7f8617fff640 1 -- 172.21.15.94:0/1847989924 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f86240abe80 con 0x7f863014f790 2024-03-06T07:26:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.084+0000 7f862f7fe640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861c06b500 0x7f861c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.085+0000 7f862f7fe640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861c06b500 0x7f861c06d9c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8620005880 tx=0x7f8620009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.089+0000 7f8617fff640 1 -- 172.21.15.94:0/1847989924 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f862407a5b0 con 0x7f863014f790 2024-03-06T07:26:28.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.260+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86300c1f40 con 0x7f863014f790 2024-03-06T07:26:28.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.261+0000 7f8617fff640 1 -- 172.21.15.94:0/1847989924 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16952 (secure 0 0 0) 0x7f862407e650 con 0x7f863014f790 2024-03-06T07:26:28.263 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:28.264 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":196,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:26.603512+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:23.168853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:23.918870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:28.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.263+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861c06b500 msgr2=0x7f861c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.263+0000 7f8635f93640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861c06b500 0x7f861c06d9c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8620005880 tx=0x7f8620009040 comp rx=0 tx=0).stop 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.263+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f790 msgr2=0x7f86301eb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.263+0000 7f8635f93640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f790 0x7f86301eb000 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f8624009890 tx=0x7f86240046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 shutdown_connections 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f861c06b500 0x7f861c06d9c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8630155a40 0x7f86301eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f863014f790 0x7f86301eb000 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 --2- 172.21.15.94:0/1847989924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86300c1aa0 0x7f86300c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 >> 172.21.15.94:0/1847989924 conn(0x7f86300c6710 msgr2=0x7f86300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 shutdown_connections 2024-03-06T07:26:28.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.264+0000 7f8635f93640 1 -- 172.21.15.94:0/1847989924 wait complete. 2024-03-06T07:26:28.277 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:28.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:28.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.375+0000 7f27a7e83640 1 Processor -- start 2024-03-06T07:26:28.377 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a7e83640 1 -- start start 2024-03-06T07:26:28.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a7e83640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f27a014f360 0x7f27a0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a7e83640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27a00c88c0 0x7f27a00c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a7e83640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a00c73d0 con 0x7f27a00c5880 2024-03-06T07:26:28.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a7e83640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a00c8dd0 con 0x7f27a00c88c0 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a7e83640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a00be2e0 con 0x7f27a014f360 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a541e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27a00c88c0 0x7f27a00c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a541e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27a00c88c0 0x7f27a00c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48264/0 (socket says 172.21.15.94:48264) 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.376+0000 7f27a541e640 1 -- 172.21.15.94:0/3147448537 learned_addr learned my addr 172.21.15.94:0/3147448537 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.377+0000 7f27a5c1f640 1 --2- 172.21.15.94:0/3147448537 >> v2:172.21.15.145:3300/0 conn(0x7f27a014f360 0x7f27a0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.377+0000 7f27a541e640 1 -- 172.21.15.94:0/3147448537 >> v2:172.21.15.145:3300/0 conn(0x7f27a014f360 msgr2=0x7f27a0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.377+0000 7f27a541e640 1 --2- 172.21.15.94:0/3147448537 >> v2:172.21.15.145:3300/0 conn(0x7f27a014f360 0x7f27a0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.377+0000 7f27a541e640 1 -- 172.21.15.94:0/3147448537 >> v1:172.21.15.145:6789/0 conn(0x7f27a00c5880 legacy=0x7f27a0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.377+0000 7f27a541e640 1 -- 172.21.15.94:0/3147448537 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a00c7670 con 0x7f27a00c88c0 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.377+0000 7f27a5c1f640 1 --2- 172.21.15.94:0/3147448537 >> v2:172.21.15.145:3300/0 conn(0x7f27a014f360 0x7f27a0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.377+0000 7f27a541e640 1 --2- 172.21.15.94:0/3147448537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c88c0 0x7f27a00c5340 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f27a008c840 tx=0x7f279c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ab9fde646edad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a4c1d640 1 -- 172.21.15.94:0/3147448537 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c035020 con 0x7f27a00c88c0 2024-03-06T07:26:28.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a4c1d640 1 -- 172.21.15.94:0/3147448537 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279c031870 con 0x7f27a00c88c0 2024-03-06T07:26:28.381 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a4c1d640 1 -- 172.21.15.94:0/3147448537 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c031c40 con 0x7f27a00c88c0 2024-03-06T07:26:28.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 -- 172.21.15.94:0/3147448537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c88c0 msgr2=0x7f27a00c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 --2- 172.21.15.94:0/3147448537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c88c0 0x7f27a00c5340 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f27a008c840 tx=0x7f279c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:28.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 -- 172.21.15.94:0/3147448537 shutdown_connections 2024-03-06T07:26:28.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 --2- 172.21.15.94:0/3147448537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c88c0 0x7f27a00c5340 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 --2- 172.21.15.94:0/3147448537 >> v2:172.21.15.145:3300/0 conn(0x7f27a014f360 0x7f27a0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 -- 172.21.15.94:0/3147448537 >> 172.21.15.94:0/3147448537 conn(0x7f27a00ac570 msgr2=0x7f27a00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 -- 172.21.15.94:0/3147448537 shutdown_connections 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.378+0000 7f27a7e83640 1 -- 172.21.15.94:0/3147448537 wait complete. 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.379+0000 7f27a7e83640 1 Processor -- start 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.379+0000 7f27a7e83640 1 -- start start 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a7e83640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c5880 0x7f27a00ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a7e83640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27a00c88c0 0x7f27a00c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a7e83640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27a014f360 0x7f27a014bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a7e83640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a0159270 con 0x7f27a00c5880 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a7e83640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00c7c30 con 0x7f27a00c88c0 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a7e83640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01593f0 con 0x7f27a014f360 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c5880 0x7f27a00ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c5880 0x7f27a00ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48266/0 (socket says 172.21.15.94:48266) 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 -- 172.21.15.94:0/1342939636 learned_addr learned my addr 172.21.15.94:0/1342939636 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a541e640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27a00c88c0 0x7f27a00c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a6420640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27a014f360 0x7f27a014bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 -- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27a014f360 msgr2=0x7f27a014bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27a014f360 0x7f27a014bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 -- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27a00c88c0 msgr2=0x7f27a00c0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27a00c88c0 0x7f27a00c0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.380+0000 7f27a5c1f640 1 -- 172.21.15.94:0/1342939636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a00c1d30 con 0x7f27a00c5880 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.381+0000 7f27a5c1f640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c5880 0x7f27a00ab7d0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f279400d7d0 tx=0x7f279400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.381+0000 7f27927fc640 1 -- 172.21.15.94:0/1342939636 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794004030 con 0x7f27a00c5880 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.381+0000 7f27927fc640 1 -- 172.21.15.94:0/1342939636 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27940041d0 con 0x7f27a00c5880 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.381+0000 7f27927fc640 1 -- 172.21.15.94:0/1342939636 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940096a0 con 0x7f27a00c5880 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.381+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a01edf40 con 0x7f27a00c5880 2024-03-06T07:26:28.385 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.381+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a01ee4c0 con 0x7f27a00c5880 2024-03-06T07:26:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.382+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a004f330 con 0x7f27a00c5880 2024-03-06T07:26:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.382+0000 7f27927fc640 1 -- 172.21.15.94:0/1342939636 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2794009840 con 0x7f27a00c5880 2024-03-06T07:26:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.382+0000 7f27927fc640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f278406b4d0 0x7f278406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.383+0000 7f27927fc640 1 -- 172.21.15.94:0/1342939636 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f279408a430 con 0x7f27a00c5880 2024-03-06T07:26:28.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.383+0000 7f27a541e640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f278406b4d0 0x7f278406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.383+0000 7f27a541e640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f278406b4d0 0x7f278406d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f279c036000 tx=0x7f279c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.388 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.387+0000 7f27927fc640 1 -- 172.21.15.94:0/1342939636 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27a00c1d30 con 0x7f27a00c5880 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.554+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27a006cda0 con 0x7f278406b4d0 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.560+0000 7f27927fc640 1 -- 172.21.15.94:0/1342939636 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237404 (secure 0 0 0) 0x7f279c081f70 con 0x7f278406b4d0 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.562+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f278406b4d0 msgr2=0x7f278406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.562+0000 7f27a7e83640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f278406b4d0 0x7f278406d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f279c036000 tx=0x7f279c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.562+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c5880 msgr2=0x7f27a00ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.562+0000 7f27a7e83640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c5880 0x7f27a00ab7d0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f279400d7d0 tx=0x7f279400dc90 comp rx=0 tx=0).stop 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 shutdown_connections 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f278406b4d0 0x7f278406d990 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27a014f360 0x7f27a014bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27a00c88c0 0x7f27a00c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 --2- 172.21.15.94:0/1342939636 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a00c5880 0x7f27a00ab7d0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 >> 172.21.15.94:0/1342939636 conn(0x7f27a00ac570 msgr2=0x7f27a01542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 shutdown_connections 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.563+0000 7f27a7e83640 1 -- 172.21.15.94:0/1342939636 wait complete. 2024-03-06T07:26:28.575 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-03-06T07:26:26.743811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":276908,"kb_used_data":19096,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697812,"statfs":{"total":773094113280,"available":772810559488,"internally_reserved":0,"allocated":19554304,"data_stored":10589386,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000883"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:24.590247+0000","last_change":"2024-03-06T07:26:24.590247+0000","last_active":"2024-03-06T07:26:24.590247+0000","last_peered":"2024-03-06T07:26:24.590247+0000","last_clean":"2024-03-06T07:26:24.590247+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:24.590247+0000","last_undegraded":"2024-03-06T07:26:24.590247+0000","last_fullsized":"2024-03-06T07:26:24.590247+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:23.584498+0000","last_change":"2024-03-06T07:26:23.584498+0000","last_active":"2024-03-06T07:26:23.584498+0000","last_peered":"2024-03-06T07:26:23.584498+0000","last_clean":"2024-03-06T07:26:23.584498+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:23.584498+0000","last_undegraded":"2024-03-06T07:26:23.584498+0000","last_fullsized":"2024-03-06T07:26:23.584498+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":492,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-06T07:26:21.424129+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:26:21.424129+0000","last_peered":"2024-03-06T07:26:21.424129+0000","last_clean":"2024-03-06T07:26:21.424129+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:26:21.424129+0000","last_undegraded":"2024-03-06T07:26:21.424129+0000","last_fullsized":"2024-03-06T07:26:21.424129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33612,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338228,"statfs":{"total":96636764160,"available":96602345472,"internally_reserved":0,"allocated":2490368,"data_stored":1765052,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33576,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338264,"statfs":{"total":96636764160,"available":96602382336,"internally_reserved":0,"allocated":2437120,"data_stored":1180273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33552,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338288,"statfs":{"total":96636764160,"available":96602406912,"internally_reserved":0,"allocated":2424832,"data_stored":1174684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33572,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338268,"statfs":{"total":96636764160,"available":96602386432,"internally_reserved":0,"allocated":2437120,"data_stored":1180273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33556,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338284,"statfs":{"total":96636764160,"available":96602402816,"internally_reserved":0,"allocated":2424832,"data_stored":1174684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33556,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338284,"statfs":{"total":96636764160,"available":96602402816,"internally_reserved":0,"allocated":2424832,"data_stored":1174684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37712,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334128,"statfs":{"total":96636764160,"available":96598147072,"internally_reserved":0,"allocated":2424832,"data_stored":1174684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574950,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37772,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334068,"statfs":{"total":96636764160,"available":96598085632,"internally_reserved":0,"allocated":2490368,"data_stored":1765052,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:28.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:28.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.713+0000 7fcb9bdc9640 1 Processor -- start 2024-03-06T07:26:28.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.714+0000 7fcb9bdc9640 1 -- start start 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.714+0000 7fcb9bdc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb941529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.714+0000 7fcb9bdc9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb940c28d0 con 0x7fcb9414f790 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.714+0000 7fcb9bdc9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb940c2bb0 con 0x7fcb940c1aa0 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.714+0000 7fcb9bdc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940c2680 con 0x7fcb94156150 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb99b65640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcb9414f790 0x7fcb940c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47774/0 (socket says 172.21.15.94:47774) 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb99b65640 1 -- 172.21.15.94:0/1273449048 learned_addr learned my addr 172.21.15.94:0/1273449048 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb9a366640 1 --2- 172.21.15.94:0/1273449048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb941529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb98b63640 1 -- 172.21.15.94:0/1273449048 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392375959 0 0) 0x7fcb940c28d0 con 0x7fcb9414f790 2024-03-06T07:26:28.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb98b63640 1 -- 172.21.15.94:0/1273449048 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7c003520 con 0x7fcb9414f790 2024-03-06T07:26:28.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb98b63640 1 -- 172.21.15.94:0/1273449048 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691973262 0 0) 0x7fcb940c2bb0 con 0x7fcb940c1aa0 2024-03-06T07:26:28.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb98b63640 1 -- 172.21.15.94:0/1273449048 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb940c28d0 con 0x7fcb940c1aa0 2024-03-06T07:26:28.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb9a366640 1 -- 172.21.15.94:0/1273449048 >> v1:172.21.15.145:6789/0 conn(0x7fcb940c1aa0 legacy=0x7fcb94155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb9a366640 1 -- 172.21.15.94:0/1273449048 >> v1:172.21.15.94:6789/0 conn(0x7fcb9414f790 legacy=0x7fcb940c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.715+0000 7fcb9a366640 1 -- 172.21.15.94:0/1273449048 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb940c2e90 con 0x7fcb94156150 2024-03-06T07:26:28.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb9a366640 1 --2- 172.21.15.94:0/1273449048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb941529d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcb840097d0 tx=0x7fcb8402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90d5d8994bdd560 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb98b63640 1 -- 172.21.15.94:0/1273449048 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84035020 con 0x7fcb94156150 2024-03-06T07:26:28.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb98b63640 1 -- 172.21.15.94:0/1273449048 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb84031890 con 0x7fcb94156150 2024-03-06T07:26:28.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb98b63640 1 -- 172.21.15.94:0/1273449048 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84031c60 con 0x7fcb94156150 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/1273449048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 msgr2=0x7fcb941529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/1273449048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb941529d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcb840097d0 tx=0x7fcb8402b930 comp rx=0 tx=0).stop 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/1273449048 shutdown_connections 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/1273449048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb941529d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.716+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/1273449048 >> 172.21.15.94:0/1273449048 conn(0x7fcb940c6710 msgr2=0x7fcb940c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.717+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/1273449048 shutdown_connections 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.717+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/1273449048 wait complete. 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 Processor -- start 2024-03-06T07:26:28.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 -- start start 2024-03-06T07:26:28.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb940c1aa0 0x7fcb941de210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9414f790 0x7fcb940c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb9414bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940c5e30 con 0x7fcb9414f790 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940c0d60 con 0x7fcb940c1aa0 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9bdc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940c5fb0 con 0x7fcb94156150 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb99364640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9414f790 0x7fcb940c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb99364640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9414f790 0x7fcb940c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49074/0 (socket says 172.21.15.94:49074) 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb99364640 1 -- 172.21.15.94:0/636961510 learned_addr learned my addr 172.21.15.94:0/636961510 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb99b65640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb940c1aa0 0x7fcb941de210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.718+0000 7fcb9a366640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb9414bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb99364640 1 -- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 msgr2=0x7fcb9414bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb99364640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb9414bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb99364640 1 -- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb940c1aa0 msgr2=0x7fcb941de210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb99364640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb940c1aa0 0x7fcb941de210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb99364640 1 -- 172.21.15.94:0/636961510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb941ec130 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb99364640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9414f790 0x7fcb940c4880 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c400 tx=0x7fcb8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb927fc640 1 -- 172.21.15.94:0/636961510 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88017070 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb927fc640 1 -- 172.21.15.94:0/636961510 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb880052d0 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb941ec410 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb927fc640 1 -- 172.21.15.94:0/636961510 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88003810 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.719+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb941ec9d0 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.720+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9404f330 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.720+0000 7fcb927fc640 1 -- 172.21.15.94:0/636961510 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb8801a070 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.721+0000 7fcb927fc640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8006b4d0 0x7fcb8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.721+0000 7fcb927fc640 1 -- 172.21.15.94:0/636961510 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcb88089720 con 0x7fcb9414f790 2024-03-06T07:26:28.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.721+0000 7fcb99b65640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8006b4d0 0x7fcb8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.721+0000 7fcb99b65640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8006b4d0 0x7fcb8006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c001560 tx=0x7fcb8c001a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.725+0000 7fcb927fc640 1 -- 172.21.15.94:0/636961510 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb88057dc0 con 0x7fcb9414f790 2024-03-06T07:26:28.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.892+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb941ebee0 con 0x7fcb8006b4d0 2024-03-06T07:26:28.900 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.899+0000 7fcb927fc640 1 -- 172.21.15.94:0/636961510 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252551 (secure 0 0 0) 0x7fcb941ebee0 con 0x7fcb8006b4d0 2024-03-06T07:26:28.903 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:28.904 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8006b4d0 msgr2=0x7fcb8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8006b4d0 0x7fcb8006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c001560 tx=0x7fcb8c001a90 comp rx=0 tx=0).stop 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9414f790 msgr2=0x7fcb940c4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9414f790 0x7fcb940c4880 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c400 tx=0x7fcb8800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 shutdown_connections 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8006b4d0 0x7fcb8006d990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb94156150 0x7fcb9414bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb9414f790 0x7fcb940c4880 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 --2- 172.21.15.94:0/636961510 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb940c1aa0 0x7fcb941de210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 >> 172.21.15.94:0/636961510 conn(0x7fcb940c6710 msgr2=0x7fcb940c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 shutdown_connections 2024-03-06T07:26:28.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:28.901+0000 7fcb9bdc9640 1 -- 172.21.15.94:0/636961510 wait complete. 2024-03-06T07:26:28.914 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-03-06T07:26:28.744099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277332,"kb_used_data":19532,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697388,"statfs":{"total":773094113280,"available":772810125312,"internally_reserved":0,"allocated":20000768,"data_stored":10794776,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.152227"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606491+0000","last_change":"2024-03-06T07:26:25.613526+0000","last_active":"2024-03-06T07:26:26.606491+0000","last_peered":"2024-03-06T07:26:26.606491+0000","last_clean":"2024-03-06T07:26:26.606491+0000","last_became_active":"2024-03-06T07:26:25.613095+0000","last_became_peered":"2024-03-06T07:26:25.613095+0000","last_unstale":"2024-03-06T07:26:26.606491+0000","last_undegraded":"2024-03-06T07:26:26.606491+0000","last_fullsized":"2024-03-06T07:26:26.606491+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:47.546312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.621827+0000","last_change":"2024-03-06T07:26:25.621077+0000","last_active":"2024-03-06T07:26:25.621827+0000","last_peered":"2024-03-06T07:26:25.621827+0000","last_clean":"2024-03-06T07:26:25.621827+0000","last_became_active":"2024-03-06T07:26:25.620809+0000","last_became_peered":"2024-03-06T07:26:25.620809+0000","last_unstale":"2024-03-06T07:26:25.621827+0000","last_undegraded":"2024-03-06T07:26:25.621827+0000","last_fullsized":"2024-03-06T07:26:25.621827+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:55.701280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849923+0000","last_change":"2024-03-06T07:26:25.622164+0000","last_active":"2024-03-06T07:26:26.849923+0000","last_peered":"2024-03-06T07:26:26.849923+0000","last_clean":"2024-03-06T07:26:26.849923+0000","last_became_active":"2024-03-06T07:26:25.621848+0000","last_became_peered":"2024-03-06T07:26:25.621848+0000","last_unstale":"2024-03-06T07:26:26.849923+0000","last_undegraded":"2024-03-06T07:26:26.849923+0000","last_fullsized":"2024-03-06T07:26:26.849923+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:39.040998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.850008+0000","last_change":"2024-03-06T07:26:25.628858+0000","last_active":"2024-03-06T07:26:26.850008+0000","last_peered":"2024-03-06T07:26:26.850008+0000","last_clean":"2024-03-06T07:26:26.850008+0000","last_became_active":"2024-03-06T07:26:25.628594+0000","last_became_peered":"2024-03-06T07:26:25.628594+0000","last_unstale":"2024-03-06T07:26:26.850008+0000","last_undegraded":"2024-03-06T07:26:26.850008+0000","last_fullsized":"2024-03-06T07:26:26.850008+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:09.072363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610641+0000","last_change":"2024-03-06T07:26:25.631230+0000","last_active":"2024-03-06T07:26:26.610641+0000","last_peered":"2024-03-06T07:26:26.610641+0000","last_clean":"2024-03-06T07:26:26.610641+0000","last_became_active":"2024-03-06T07:26:25.630944+0000","last_became_peered":"2024-03-06T07:26:25.630944+0000","last_unstale":"2024-03-06T07:26:26.610641+0000","last_undegraded":"2024-03-06T07:26:26.610641+0000","last_fullsized":"2024-03-06T07:26:26.610641+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:36.839776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606338+0000","last_change":"2024-03-06T07:26:25.623860+0000","last_active":"2024-03-06T07:26:26.606338+0000","last_peered":"2024-03-06T07:26:26.606338+0000","last_clean":"2024-03-06T07:26:26.606338+0000","last_became_active":"2024-03-06T07:26:25.623583+0000","last_became_peered":"2024-03-06T07:26:25.623583+0000","last_unstale":"2024-03-06T07:26:26.606338+0000","last_undegraded":"2024-03-06T07:26:26.606338+0000","last_fullsized":"2024-03-06T07:26:26.606338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:08.128069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610834+0000","last_change":"2024-03-06T07:26:25.621879+0000","last_active":"2024-03-06T07:26:26.610834+0000","last_peered":"2024-03-06T07:26:26.610834+0000","last_clean":"2024-03-06T07:26:26.610834+0000","last_became_active":"2024-03-06T07:26:25.621574+0000","last_became_peered":"2024-03-06T07:26:25.621574+0000","last_unstale":"2024-03-06T07:26:26.610834+0000","last_undegraded":"2024-03-06T07:26:26.610834+0000","last_fullsized":"2024-03-06T07:26:26.610834+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:00.147186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610021+0000","last_change":"2024-03-06T07:26:25.627329+0000","last_active":"2024-03-06T07:26:26.610021+0000","last_peered":"2024-03-06T07:26:26.610021+0000","last_clean":"2024-03-06T07:26:26.610021+0000","last_became_active":"2024-03-06T07:26:25.626924+0000","last_became_peered":"2024-03-06T07:26:25.626924+0000","last_unstale":"2024-03-06T07:26:26.610021+0000","last_undegraded":"2024-03-06T07:26:26.610021+0000","last_fullsized":"2024-03-06T07:26:26.610021+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:56.509828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606631+0000","last_change":"2024-03-06T07:26:25.625699+0000","last_active":"2024-03-06T07:26:26.606631+0000","last_peered":"2024-03-06T07:26:26.606631+0000","last_clean":"2024-03-06T07:26:26.606631+0000","last_became_active":"2024-03-06T07:26:25.625418+0000","last_became_peered":"2024-03-06T07:26:25.625418+0000","last_unstale":"2024-03-06T07:26:26.606631+0000","last_undegraded":"2024-03-06T07:26:26.606631+0000","last_fullsized":"2024-03-06T07:26:26.606631+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:37.087296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.850048+0000","last_change":"2024-03-06T07:26:25.622056+0000","last_active":"2024-03-06T07:26:26.850048+0000","last_peered":"2024-03-06T07:26:26.850048+0000","last_clean":"2024-03-06T07:26:26.850048+0000","last_became_active":"2024-03-06T07:26:25.621762+0000","last_became_peered":"2024-03-06T07:26:25.621762+0000","last_unstale":"2024-03-06T07:26:26.850048+0000","last_undegraded":"2024-03-06T07:26:26.850048+0000","last_fullsized":"2024-03-06T07:26:26.850048+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:11.625371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.850025+0000","last_change":"2024-03-06T07:26:25.622742+0000","last_active":"2024-03-06T07:26:26.850025+0000","last_peered":"2024-03-06T07:26:26.850025+0000","last_clean":"2024-03-06T07:26:26.850025+0000","last_became_active":"2024-03-06T07:26:25.622487+0000","last_became_peered":"2024-03-06T07:26:25.622487+0000","last_unstale":"2024-03-06T07:26:26.850025+0000","last_undegraded":"2024-03-06T07:26:26.850025+0000","last_fullsized":"2024-03-06T07:26:26.850025+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:55.340594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606594+0000","last_change":"2024-03-06T07:26:25.616323+0000","last_active":"2024-03-06T07:26:26.606594+0000","last_peered":"2024-03-06T07:26:26.606594+0000","last_clean":"2024-03-06T07:26:26.606594+0000","last_became_active":"2024-03-06T07:26:25.616078+0000","last_became_peered":"2024-03-06T07:26:25.616078+0000","last_unstale":"2024-03-06T07:26:26.606594+0000","last_undegraded":"2024-03-06T07:26:26.606594+0000","last_fullsized":"2024-03-06T07:26:26.606594+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:44:41.800143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254912+0000","last_change":"2024-03-06T07:26:25.627787+0000","last_active":"2024-03-06T07:26:27.254912+0000","last_peered":"2024-03-06T07:26:27.254912+0000","last_clean":"2024-03-06T07:26:27.254912+0000","last_became_active":"2024-03-06T07:26:25.627468+0000","last_became_peered":"2024-03-06T07:26:25.627468+0000","last_unstale":"2024-03-06T07:26:27.254912+0000","last_undegraded":"2024-03-06T07:26:27.254912+0000","last_fullsized":"2024-03-06T07:26:27.254912+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:59.744512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223458+0000","last_change":"2024-03-06T07:26:25.617583+0000","last_active":"2024-03-06T07:26:27.223458+0000","last_peered":"2024-03-06T07:26:27.223458+0000","last_clean":"2024-03-06T07:26:27.223458+0000","last_became_active":"2024-03-06T07:26:25.617318+0000","last_became_peered":"2024-03-06T07:26:25.617318+0000","last_unstale":"2024-03-06T07:26:27.223458+0000","last_undegraded":"2024-03-06T07:26:27.223458+0000","last_fullsized":"2024-03-06T07:26:27.223458+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:06.628233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.634398+0000","last_change":"2024-03-06T07:26:25.629745+0000","last_active":"2024-03-06T07:26:25.634398+0000","last_peered":"2024-03-06T07:26:25.634398+0000","last_clean":"2024-03-06T07:26:25.634398+0000","last_became_active":"2024-03-06T07:26:25.629158+0000","last_became_peered":"2024-03-06T07:26:25.629158+0000","last_unstale":"2024-03-06T07:26:25.634398+0000","last_undegraded":"2024-03-06T07:26:25.634398+0000","last_fullsized":"2024-03-06T07:26:25.634398+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:24:02.836523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849700+0000","last_change":"2024-03-06T07:26:25.619908+0000","last_active":"2024-03-06T07:26:26.849700+0000","last_peered":"2024-03-06T07:26:26.849700+0000","last_clean":"2024-03-06T07:26:26.849700+0000","last_became_active":"2024-03-06T07:26:25.619335+0000","last_became_peered":"2024-03-06T07:26:25.619335+0000","last_unstale":"2024-03-06T07:26:26.849700+0000","last_undegraded":"2024-03-06T07:26:26.849700+0000","last_fullsized":"2024-03-06T07:26:26.849700+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:49.942441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.255204+0000","last_change":"2024-03-06T07:26:25.623801+0000","last_active":"2024-03-06T07:26:27.255204+0000","last_peered":"2024-03-06T07:26:27.255204+0000","last_clean":"2024-03-06T07:26:27.255204+0000","last_became_active":"2024-03-06T07:26:25.621879+0000","last_became_peered":"2024-03-06T07:26:25.621879+0000","last_unstale":"2024-03-06T07:26:27.255204+0000","last_undegraded":"2024-03-06T07:26:27.255204+0000","last_fullsized":"2024-03-06T07:26:27.255204+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:45.532954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.850114+0000","last_change":"2024-03-06T07:26:25.613503+0000","last_active":"2024-03-06T07:26:26.850114+0000","last_peered":"2024-03-06T07:26:26.850114+0000","last_clean":"2024-03-06T07:26:26.850114+0000","last_became_active":"2024-03-06T07:26:25.613219+0000","last_became_peered":"2024-03-06T07:26:25.613219+0000","last_unstale":"2024-03-06T07:26:26.850114+0000","last_undegraded":"2024-03-06T07:26:26.850114+0000","last_fullsized":"2024-03-06T07:26:26.850114+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:58.784850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.626928+0000","last_change":"2024-03-06T07:26:25.623775+0000","last_active":"2024-03-06T07:26:25.626928+0000","last_peered":"2024-03-06T07:26:25.626928+0000","last_clean":"2024-03-06T07:26:25.626928+0000","last_became_active":"2024-03-06T07:26:25.623512+0000","last_became_peered":"2024-03-06T07:26:25.623512+0000","last_unstale":"2024-03-06T07:26:25.626928+0000","last_undegraded":"2024-03-06T07:26:25.626928+0000","last_fullsized":"2024-03-06T07:26:25.626928+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:27.682237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.632206+0000","last_change":"2024-03-06T07:26:25.631348+0000","last_active":"2024-03-06T07:26:25.632206+0000","last_peered":"2024-03-06T07:26:25.632206+0000","last_clean":"2024-03-06T07:26:25.632206+0000","last_became_active":"2024-03-06T07:26:25.631078+0000","last_became_peered":"2024-03-06T07:26:25.631078+0000","last_unstale":"2024-03-06T07:26:25.632206+0000","last_undegraded":"2024-03-06T07:26:25.632206+0000","last_fullsized":"2024-03-06T07:26:25.632206+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:59.076010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.611183+0000","last_change":"2024-03-06T07:26:25.624266+0000","last_active":"2024-03-06T07:26:26.611183+0000","last_peered":"2024-03-06T07:26:26.611183+0000","last_clean":"2024-03-06T07:26:26.611183+0000","last_became_active":"2024-03-06T07:26:25.623777+0000","last_became_peered":"2024-03-06T07:26:25.623777+0000","last_unstale":"2024-03-06T07:26:26.611183+0000","last_undegraded":"2024-03-06T07:26:26.611183+0000","last_fullsized":"2024-03-06T07:26:26.611183+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:36.839776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254821+0000","last_change":"2024-03-06T07:26:25.622001+0000","last_active":"2024-03-06T07:26:27.254821+0000","last_peered":"2024-03-06T07:26:27.254821+0000","last_clean":"2024-03-06T07:26:27.254821+0000","last_became_active":"2024-03-06T07:26:25.621729+0000","last_became_peered":"2024-03-06T07:26:25.621729+0000","last_unstale":"2024-03-06T07:26:27.254821+0000","last_undegraded":"2024-03-06T07:26:27.254821+0000","last_fullsized":"2024-03-06T07:26:27.254821+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:39.386374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.626874+0000","last_change":"2024-03-06T07:26:25.625726+0000","last_active":"2024-03-06T07:26:25.626874+0000","last_peered":"2024-03-06T07:26:25.626874+0000","last_clean":"2024-03-06T07:26:25.626874+0000","last_became_active":"2024-03-06T07:26:25.625347+0000","last_became_peered":"2024-03-06T07:26:25.625347+0000","last_unstale":"2024-03-06T07:26:25.626874+0000","last_undegraded":"2024-03-06T07:26:25.626874+0000","last_fullsized":"2024-03-06T07:26:25.626874+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:37.368716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254687+0000","last_change":"2024-03-06T07:26:25.623299+0000","last_active":"2024-03-06T07:26:27.254687+0000","last_peered":"2024-03-06T07:26:27.254687+0000","last_clean":"2024-03-06T07:26:27.254687+0000","last_became_active":"2024-03-06T07:26:25.623008+0000","last_became_peered":"2024-03-06T07:26:25.623008+0000","last_unstale":"2024-03-06T07:26:27.254687+0000","last_undegraded":"2024-03-06T07:26:27.254687+0000","last_fullsized":"2024-03-06T07:26:27.254687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:06.514604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.616898+0000","last_change":"2024-03-06T07:26:25.614042+0000","last_active":"2024-03-06T07:26:25.616898+0000","last_peered":"2024-03-06T07:26:25.616898+0000","last_clean":"2024-03-06T07:26:25.616898+0000","last_became_active":"2024-03-06T07:26:25.613775+0000","last_became_peered":"2024-03-06T07:26:25.613775+0000","last_unstale":"2024-03-06T07:26:25.616898+0000","last_undegraded":"2024-03-06T07:26:25.616898+0000","last_fullsized":"2024-03-06T07:26:25.616898+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:54.531311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.611242+0000","last_change":"2024-03-06T07:26:25.624551+0000","last_active":"2024-03-06T07:26:26.611242+0000","last_peered":"2024-03-06T07:26:26.611242+0000","last_clean":"2024-03-06T07:26:26.611242+0000","last_became_active":"2024-03-06T07:26:25.624219+0000","last_became_peered":"2024-03-06T07:26:25.624219+0000","last_unstale":"2024-03-06T07:26:26.611242+0000","last_undegraded":"2024-03-06T07:26:26.611242+0000","last_fullsized":"2024-03-06T07:26:26.611242+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:40.889843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610511+0000","last_change":"2024-03-06T07:26:25.619405+0000","last_active":"2024-03-06T07:26:26.610511+0000","last_peered":"2024-03-06T07:26:26.610511+0000","last_clean":"2024-03-06T07:26:26.610511+0000","last_became_active":"2024-03-06T07:26:25.619139+0000","last_became_peered":"2024-03-06T07:26:25.619139+0000","last_unstale":"2024-03-06T07:26:26.610511+0000","last_undegraded":"2024-03-06T07:26:26.610511+0000","last_fullsized":"2024-03-06T07:26:26.610511+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:00.147186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606639+0000","last_change":"2024-03-06T07:26:25.631492+0000","last_active":"2024-03-06T07:26:26.606639+0000","last_peered":"2024-03-06T07:26:26.606639+0000","last_clean":"2024-03-06T07:26:26.606639+0000","last_became_active":"2024-03-06T07:26:25.631213+0000","last_became_peered":"2024-03-06T07:26:25.631213+0000","last_unstale":"2024-03-06T07:26:26.606639+0000","last_undegraded":"2024-03-06T07:26:26.606639+0000","last_fullsized":"2024-03-06T07:26:26.606639+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:15.826944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223558+0000","last_change":"2024-03-06T07:26:25.616375+0000","last_active":"2024-03-06T07:26:27.223558+0000","last_peered":"2024-03-06T07:26:27.223558+0000","last_clean":"2024-03-06T07:26:27.223558+0000","last_became_active":"2024-03-06T07:26:25.616115+0000","last_became_peered":"2024-03-06T07:26:25.616115+0000","last_unstale":"2024-03-06T07:26:27.223558+0000","last_undegraded":"2024-03-06T07:26:27.223558+0000","last_fullsized":"2024-03-06T07:26:27.223558+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:11.532306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223505+0000","last_change":"2024-03-06T07:26:25.625680+0000","last_active":"2024-03-06T07:26:27.223505+0000","last_peered":"2024-03-06T07:26:27.223505+0000","last_clean":"2024-03-06T07:26:27.223505+0000","last_became_active":"2024-03-06T07:26:25.625380+0000","last_became_peered":"2024-03-06T07:26:25.625380+0000","last_unstale":"2024-03-06T07:26:27.223505+0000","last_undegraded":"2024-03-06T07:26:27.223505+0000","last_fullsized":"2024-03-06T07:26:27.223505+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:39.555100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.628270+0000","last_change":"2024-03-06T07:26:25.627872+0000","last_active":"2024-03-06T07:26:25.628270+0000","last_peered":"2024-03-06T07:26:25.628270+0000","last_clean":"2024-03-06T07:26:25.628270+0000","last_became_active":"2024-03-06T07:26:25.627585+0000","last_became_peered":"2024-03-06T07:26:25.627585+0000","last_unstale":"2024-03-06T07:26:25.628270+0000","last_undegraded":"2024-03-06T07:26:25.628270+0000","last_fullsized":"2024-03-06T07:26:25.628270+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:12.920744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606281+0000","last_change":"2024-03-06T07:26:25.622053+0000","last_active":"2024-03-06T07:26:26.606281+0000","last_peered":"2024-03-06T07:26:26.606281+0000","last_clean":"2024-03-06T07:26:26.606281+0000","last_became_active":"2024-03-06T07:26:25.621607+0000","last_became_peered":"2024-03-06T07:26:25.621607+0000","last_unstale":"2024-03-06T07:26:26.606281+0000","last_undegraded":"2024-03-06T07:26:26.606281+0000","last_fullsized":"2024-03-06T07:26:26.606281+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:28.368849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.617331+0000","last_change":"2024-03-06T07:26:25.616483+0000","last_active":"2024-03-06T07:26:25.617331+0000","last_peered":"2024-03-06T07:26:25.617331+0000","last_clean":"2024-03-06T07:26:25.617331+0000","last_became_active":"2024-03-06T07:26:25.616214+0000","last_became_peered":"2024-03-06T07:26:25.616214+0000","last_unstale":"2024-03-06T07:26:25.617331+0000","last_undegraded":"2024-03-06T07:26:25.617331+0000","last_fullsized":"2024-03-06T07:26:25.617331+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:19.223453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"195'4","reported_seq":18,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606478+0000","last_change":"2024-03-06T07:26:24.607992+0000","last_active":"2024-03-06T07:26:26.606478+0000","last_peered":"2024-03-06T07:26:26.606478+0000","last_clean":"2024-03-06T07:26:26.606478+0000","last_became_active":"2024-03-06T07:26:24.607721+0000","last_became_peered":"2024-03-06T07:26:24.607721+0000","last_unstale":"2024-03-06T07:26:26.606478+0000","last_undegraded":"2024-03-06T07:26:26.606478+0000","last_fullsized":"2024-03-06T07:26:26.606478+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:33.954452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.642486+0000","last_change":"2024-03-06T07:26:25.637945+0000","last_active":"2024-03-06T07:26:25.642486+0000","last_peered":"2024-03-06T07:26:25.642486+0000","last_clean":"2024-03-06T07:26:25.642486+0000","last_became_active":"2024-03-06T07:26:25.637683+0000","last_became_peered":"2024-03-06T07:26:25.637683+0000","last_unstale":"2024-03-06T07:26:25.642486+0000","last_undegraded":"2024-03-06T07:26:25.642486+0000","last_fullsized":"2024-03-06T07:26:25.642486+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:45.569205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254824+0000","last_change":"2024-03-06T07:26:24.615689+0000","last_active":"2024-03-06T07:26:27.254824+0000","last_peered":"2024-03-06T07:26:27.254824+0000","last_clean":"2024-03-06T07:26:27.254824+0000","last_became_active":"2024-03-06T07:26:24.615403+0000","last_became_peered":"2024-03-06T07:26:24.615403+0000","last_unstale":"2024-03-06T07:26:27.254824+0000","last_undegraded":"2024-03-06T07:26:27.254824+0000","last_fullsized":"2024-03-06T07:26:27.254824+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:20.559056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606494+0000","last_change":"2024-03-06T07:26:25.623819+0000","last_active":"2024-03-06T07:26:26.606494+0000","last_peered":"2024-03-06T07:26:26.606494+0000","last_clean":"2024-03-06T07:26:26.606494+0000","last_became_active":"2024-03-06T07:26:25.623537+0000","last_became_peered":"2024-03-06T07:26:25.623537+0000","last_unstale":"2024-03-06T07:26:26.606494+0000","last_undegraded":"2024-03-06T07:26:26.606494+0000","last_fullsized":"2024-03-06T07:26:26.606494+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:14.966434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223714+0000","last_change":"2024-03-06T07:26:24.613260+0000","last_active":"2024-03-06T07:26:27.223714+0000","last_peered":"2024-03-06T07:26:27.223714+0000","last_clean":"2024-03-06T07:26:27.223714+0000","last_became_active":"2024-03-06T07:26:24.612985+0000","last_became_peered":"2024-03-06T07:26:24.612985+0000","last_unstale":"2024-03-06T07:26:27.223714+0000","last_undegraded":"2024-03-06T07:26:27.223714+0000","last_fullsized":"2024-03-06T07:26:27.223714+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:43.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223700+0000","last_change":"2024-03-06T07:26:25.614485+0000","last_active":"2024-03-06T07:26:27.223700+0000","last_peered":"2024-03-06T07:26:27.223700+0000","last_clean":"2024-03-06T07:26:27.223700+0000","last_became_active":"2024-03-06T07:26:25.614015+0000","last_became_peered":"2024-03-06T07:26:25.614015+0000","last_unstale":"2024-03-06T07:26:27.223700+0000","last_undegraded":"2024-03-06T07:26:27.223700+0000","last_fullsized":"2024-03-06T07:26:27.223700+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:29.249795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"195'1","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.639846+0000","last_change":"2024-03-06T07:26:24.610050+0000","last_active":"2024-03-06T07:26:25.639846+0000","last_peered":"2024-03-06T07:26:25.639846+0000","last_clean":"2024-03-06T07:26:25.639846+0000","last_became_active":"2024-03-06T07:26:24.609715+0000","last_became_peered":"2024-03-06T07:26:24.609715+0000","last_unstale":"2024-03-06T07:26:25.639846+0000","last_undegraded":"2024-03-06T07:26:25.639846+0000","last_fullsized":"2024-03-06T07:26:25.639846+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:43.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.611282+0000","last_change":"2024-03-06T07:26:25.626213+0000","last_active":"2024-03-06T07:26:26.611282+0000","last_peered":"2024-03-06T07:26:26.611282+0000","last_clean":"2024-03-06T07:26:26.611282+0000","last_became_active":"2024-03-06T07:26:25.625924+0000","last_became_peered":"2024-03-06T07:26:25.625924+0000","last_unstale":"2024-03-06T07:26:26.611282+0000","last_undegraded":"2024-03-06T07:26:26.611282+0000","last_fullsized":"2024-03-06T07:26:26.611282+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:58.466729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223605+0000","last_change":"2024-03-06T07:26:24.608025+0000","last_active":"2024-03-06T07:26:27.223605+0000","last_peered":"2024-03-06T07:26:27.223605+0000","last_clean":"2024-03-06T07:26:27.223605+0000","last_became_active":"2024-03-06T07:26:24.607770+0000","last_became_peered":"2024-03-06T07:26:24.607770+0000","last_unstale":"2024-03-06T07:26:27.223605+0000","last_undegraded":"2024-03-06T07:26:27.223605+0000","last_fullsized":"2024-03-06T07:26:27.223605+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:44.752151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849913+0000","last_change":"2024-03-06T07:26:25.629540+0000","last_active":"2024-03-06T07:26:26.849913+0000","last_peered":"2024-03-06T07:26:26.849913+0000","last_clean":"2024-03-06T07:26:26.849913+0000","last_became_active":"2024-03-06T07:26:25.629270+0000","last_became_peered":"2024-03-06T07:26:25.629270+0000","last_unstale":"2024-03-06T07:26:26.849913+0000","last_undegraded":"2024-03-06T07:26:26.849913+0000","last_fullsized":"2024-03-06T07:26:26.849913+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:19.571576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849671+0000","last_change":"2024-03-06T07:26:24.604971+0000","last_active":"2024-03-06T07:26:26.849671+0000","last_peered":"2024-03-06T07:26:26.849671+0000","last_clean":"2024-03-06T07:26:26.849671+0000","last_became_active":"2024-03-06T07:26:24.604679+0000","last_became_peered":"2024-03-06T07:26:24.604679+0000","last_unstale":"2024-03-06T07:26:26.849671+0000","last_undegraded":"2024-03-06T07:26:26.849671+0000","last_fullsized":"2024-03-06T07:26:26.849671+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:40.485438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610521+0000","last_change":"2024-03-06T07:26:25.629576+0000","last_active":"2024-03-06T07:26:26.610521+0000","last_peered":"2024-03-06T07:26:26.610521+0000","last_clean":"2024-03-06T07:26:26.610521+0000","last_became_active":"2024-03-06T07:26:25.628378+0000","last_became_peered":"2024-03-06T07:26:25.628378+0000","last_unstale":"2024-03-06T07:26:26.610521+0000","last_undegraded":"2024-03-06T07:26:26.610521+0000","last_fullsized":"2024-03-06T07:26:26.610521+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:15.746376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606213+0000","last_change":"2024-03-06T07:26:24.602757+0000","last_active":"2024-03-06T07:26:26.606213+0000","last_peered":"2024-03-06T07:26:26.606213+0000","last_clean":"2024-03-06T07:26:26.606213+0000","last_became_active":"2024-03-06T07:26:24.602477+0000","last_became_peered":"2024-03-06T07:26:24.602477+0000","last_unstale":"2024-03-06T07:26:26.606213+0000","last_undegraded":"2024-03-06T07:26:26.606213+0000","last_fullsized":"2024-03-06T07:26:26.606213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:00.588022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610843+0000","last_change":"2024-03-06T07:26:25.623783+0000","last_active":"2024-03-06T07:26:26.610843+0000","last_peered":"2024-03-06T07:26:26.610843+0000","last_clean":"2024-03-06T07:26:26.610843+0000","last_became_active":"2024-03-06T07:26:25.623477+0000","last_became_peered":"2024-03-06T07:26:25.623477+0000","last_unstale":"2024-03-06T07:26:26.610843+0000","last_undegraded":"2024-03-06T07:26:26.610843+0000","last_fullsized":"2024-03-06T07:26:26.610843+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:54.121632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223701+0000","last_change":"2024-03-06T07:26:24.613410+0000","last_active":"2024-03-06T07:26:27.223701+0000","last_peered":"2024-03-06T07:26:27.223701+0000","last_clean":"2024-03-06T07:26:27.223701+0000","last_became_active":"2024-03-06T07:26:24.613101+0000","last_became_peered":"2024-03-06T07:26:24.613101+0000","last_unstale":"2024-03-06T07:26:27.223701+0000","last_undegraded":"2024-03-06T07:26:27.223701+0000","last_fullsized":"2024-03-06T07:26:27.223701+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:56.800343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254763+0000","last_change":"2024-03-06T07:26:25.622834+0000","last_active":"2024-03-06T07:26:27.254763+0000","last_peered":"2024-03-06T07:26:27.254763+0000","last_clean":"2024-03-06T07:26:27.254763+0000","last_became_active":"2024-03-06T07:26:25.622188+0000","last_became_peered":"2024-03-06T07:26:25.622188+0000","last_unstale":"2024-03-06T07:26:27.254763+0000","last_undegraded":"2024-03-06T07:26:27.254763+0000","last_fullsized":"2024-03-06T07:26:27.254763+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:17.503445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.850104+0000","last_change":"2024-03-06T07:26:24.612496+0000","last_active":"2024-03-06T07:26:26.850104+0000","last_peered":"2024-03-06T07:26:26.850104+0000","last_clean":"2024-03-06T07:26:26.850104+0000","last_became_active":"2024-03-06T07:26:24.612203+0000","last_became_peered":"2024-03-06T07:26:24.612203+0000","last_unstale":"2024-03-06T07:26:26.850104+0000","last_undegraded":"2024-03-06T07:26:26.850104+0000","last_fullsized":"2024-03-06T07:26:26.850104+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:55.281565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849773+0000","last_change":"2024-03-06T07:26:25.628432+0000","last_active":"2024-03-06T07:26:26.849773+0000","last_peered":"2024-03-06T07:26:26.849773+0000","last_clean":"2024-03-06T07:26:26.849773+0000","last_became_active":"2024-03-06T07:26:25.628026+0000","last_became_peered":"2024-03-06T07:26:25.628026+0000","last_unstale":"2024-03-06T07:26:26.849773+0000","last_undegraded":"2024-03-06T07:26:26.849773+0000","last_fullsized":"2024-03-06T07:26:26.849773+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:00:47.884888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"195'3","reported_seq":17,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610619+0000","last_change":"2024-03-06T07:26:24.605736+0000","last_active":"2024-03-06T07:26:26.610619+0000","last_peered":"2024-03-06T07:26:26.610619+0000","last_clean":"2024-03-06T07:26:26.610619+0000","last_became_active":"2024-03-06T07:26:24.605464+0000","last_became_peered":"2024-03-06T07:26:24.605464+0000","last_unstale":"2024-03-06T07:26:26.610619+0000","last_undegraded":"2024-03-06T07:26:26.610619+0000","last_fullsized":"2024-03-06T07:26:26.610619+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:58.738764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.611023+0000","last_change":"2024-03-06T07:26:25.624038+0000","last_active":"2024-03-06T07:26:26.611023+0000","last_peered":"2024-03-06T07:26:26.611023+0000","last_clean":"2024-03-06T07:26:26.611023+0000","last_became_active":"2024-03-06T07:26:25.623388+0000","last_became_peered":"2024-03-06T07:26:25.623388+0000","last_unstale":"2024-03-06T07:26:26.611023+0000","last_undegraded":"2024-03-06T07:26:26.611023+0000","last_fullsized":"2024-03-06T07:26:26.611023+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:15.746376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"195'6","reported_seq":20,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606678+0000","last_change":"2024-03-06T07:26:24.614760+0000","last_active":"2024-03-06T07:26:26.606678+0000","last_peered":"2024-03-06T07:26:26.606678+0000","last_clean":"2024-03-06T07:26:26.606678+0000","last_became_active":"2024-03-06T07:26:24.614374+0000","last_became_peered":"2024-03-06T07:26:24.614374+0000","last_unstale":"2024-03-06T07:26:26.606678+0000","last_undegraded":"2024-03-06T07:26:26.606678+0000","last_fullsized":"2024-03-06T07:26:26.606678+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:25.940196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.616905+0000","last_change":"2024-03-06T07:26:25.613324+0000","last_active":"2024-03-06T07:26:25.616905+0000","last_peered":"2024-03-06T07:26:25.616905+0000","last_clean":"2024-03-06T07:26:25.616905+0000","last_became_active":"2024-03-06T07:26:25.613068+0000","last_became_peered":"2024-03-06T07:26:25.613068+0000","last_unstale":"2024-03-06T07:26:25.616905+0000","last_undegraded":"2024-03-06T07:26:25.616905+0000","last_fullsized":"2024-03-06T07:26:25.616905+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:14:50.254465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.639794+0000","last_change":"2024-03-06T07:26:24.607300+0000","last_active":"2024-03-06T07:26:25.639794+0000","last_peered":"2024-03-06T07:26:25.639794+0000","last_clean":"2024-03-06T07:26:25.639794+0000","last_became_active":"2024-03-06T07:26:24.607038+0000","last_became_peered":"2024-03-06T07:26:24.607038+0000","last_unstale":"2024-03-06T07:26:25.639794+0000","last_undegraded":"2024-03-06T07:26:25.639794+0000","last_fullsized":"2024-03-06T07:26:25.639794+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:40.485438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.611215+0000","last_change":"2024-03-06T07:26:25.626136+0000","last_active":"2024-03-06T07:26:26.611215+0000","last_peered":"2024-03-06T07:26:26.611215+0000","last_clean":"2024-03-06T07:26:26.611215+0000","last_became_active":"2024-03-06T07:26:25.625379+0000","last_became_peered":"2024-03-06T07:26:25.625379+0000","last_unstale":"2024-03-06T07:26:26.611215+0000","last_undegraded":"2024-03-06T07:26:26.611215+0000","last_fullsized":"2024-03-06T07:26:26.611215+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:14.091033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610312+0000","last_change":"2024-03-06T07:26:24.607104+0000","last_active":"2024-03-06T07:26:26.610312+0000","last_peered":"2024-03-06T07:26:26.610312+0000","last_clean":"2024-03-06T07:26:26.610312+0000","last_became_active":"2024-03-06T07:26:24.606842+0000","last_became_peered":"2024-03-06T07:26:24.606842+0000","last_unstale":"2024-03-06T07:26:26.610312+0000","last_undegraded":"2024-03-06T07:26:26.610312+0000","last_fullsized":"2024-03-06T07:26:26.610312+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:12.866389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.637158+0000","last_change":"2024-03-06T07:26:25.633924+0000","last_active":"2024-03-06T07:26:25.637158+0000","last_peered":"2024-03-06T07:26:25.637158+0000","last_clean":"2024-03-06T07:26:25.637158+0000","last_became_active":"2024-03-06T07:26:25.633556+0000","last_became_peered":"2024-03-06T07:26:25.633556+0000","last_unstale":"2024-03-06T07:26:25.637158+0000","last_undegraded":"2024-03-06T07:26:25.637158+0000","last_fullsized":"2024-03-06T07:26:25.637158+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:18.612412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.609982+0000","last_change":"2024-03-06T07:26:24.604195+0000","last_active":"2024-03-06T07:26:26.609982+0000","last_peered":"2024-03-06T07:26:26.609982+0000","last_clean":"2024-03-06T07:26:26.609982+0000","last_became_active":"2024-03-06T07:26:24.603913+0000","last_became_peered":"2024-03-06T07:26:24.603913+0000","last_unstale":"2024-03-06T07:26:26.609982+0000","last_undegraded":"2024-03-06T07:26:26.609982+0000","last_fullsized":"2024-03-06T07:26:26.609982+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:04.828194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610511+0000","last_change":"2024-03-06T07:26:25.624704+0000","last_active":"2024-03-06T07:26:26.610511+0000","last_peered":"2024-03-06T07:26:26.610511+0000","last_clean":"2024-03-06T07:26:26.610511+0000","last_became_active":"2024-03-06T07:26:25.624435+0000","last_became_peered":"2024-03-06T07:26:25.624435+0000","last_unstale":"2024-03-06T07:26:26.610511+0000","last_undegraded":"2024-03-06T07:26:26.610511+0000","last_fullsized":"2024-03-06T07:26:26.610511+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:54.121632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"195'8","reported_seq":20,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.640316+0000","last_change":"2024-03-06T07:26:24.611003+0000","last_active":"2024-03-06T07:26:25.640316+0000","last_peered":"2024-03-06T07:26:25.640316+0000","last_clean":"2024-03-06T07:26:25.640316+0000","last_became_active":"2024-03-06T07:26:24.610705+0000","last_became_peered":"2024-03-06T07:26:24.610705+0000","last_unstale":"2024-03-06T07:26:25.640316+0000","last_undegraded":"2024-03-06T07:26:25.640316+0000","last_fullsized":"2024-03-06T07:26:25.640316+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:15.357280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.609867+0000","last_change":"2024-03-06T07:26:25.625694+0000","last_active":"2024-03-06T07:26:26.609867+0000","last_peered":"2024-03-06T07:26:26.609867+0000","last_clean":"2024-03-06T07:26:26.609867+0000","last_became_active":"2024-03-06T07:26:25.625223+0000","last_became_peered":"2024-03-06T07:26:25.625223+0000","last_unstale":"2024-03-06T07:26:26.609867+0000","last_undegraded":"2024-03-06T07:26:26.609867+0000","last_fullsized":"2024-03-06T07:26:26.609867+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:12.028455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606591+0000","last_change":"2024-03-06T07:26:24.605017+0000","last_active":"2024-03-06T07:26:26.606591+0000","last_peered":"2024-03-06T07:26:26.606591+0000","last_clean":"2024-03-06T07:26:26.606591+0000","last_became_active":"2024-03-06T07:26:24.604708+0000","last_became_peered":"2024-03-06T07:26:24.604708+0000","last_unstale":"2024-03-06T07:26:26.606591+0000","last_undegraded":"2024-03-06T07:26:26.606591+0000","last_fullsized":"2024-03-06T07:26:26.606591+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:14.901206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.609946+0000","last_change":"2024-03-06T07:26:25.631036+0000","last_active":"2024-03-06T07:26:26.609946+0000","last_peered":"2024-03-06T07:26:26.609946+0000","last_clean":"2024-03-06T07:26:26.609946+0000","last_became_active":"2024-03-06T07:26:25.630382+0000","last_became_peered":"2024-03-06T07:26:25.630382+0000","last_unstale":"2024-03-06T07:26:26.609946+0000","last_undegraded":"2024-03-06T07:26:26.609946+0000","last_fullsized":"2024-03-06T07:26:26.609946+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:33.423849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.642660+0000","last_change":"2024-03-06T07:26:24.615300+0000","last_active":"2024-03-06T07:26:25.642660+0000","last_peered":"2024-03-06T07:26:25.642660+0000","last_clean":"2024-03-06T07:26:25.642660+0000","last_became_active":"2024-03-06T07:26:24.615018+0000","last_became_peered":"2024-03-06T07:26:24.615018+0000","last_unstale":"2024-03-06T07:26:25.642660+0000","last_undegraded":"2024-03-06T07:26:25.642660+0000","last_fullsized":"2024-03-06T07:26:25.642660+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:03.826837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.626468+0000","last_change":"2024-03-06T07:26:25.625694+0000","last_active":"2024-03-06T07:26:25.626468+0000","last_peered":"2024-03-06T07:26:25.626468+0000","last_clean":"2024-03-06T07:26:25.626468+0000","last_became_active":"2024-03-06T07:26:25.625433+0000","last_became_peered":"2024-03-06T07:26:25.625433+0000","last_unstale":"2024-03-06T07:26:25.626468+0000","last_undegraded":"2024-03-06T07:26:25.626468+0000","last_fullsized":"2024-03-06T07:26:25.626468+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:17.820155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.255000+0000","last_change":"2024-03-06T07:26:24.615686+0000","last_active":"2024-03-06T07:26:27.255000+0000","last_peered":"2024-03-06T07:26:27.255000+0000","last_clean":"2024-03-06T07:26:27.255000+0000","last_became_active":"2024-03-06T07:26:24.615394+0000","last_became_peered":"2024-03-06T07:26:24.615394+0000","last_unstale":"2024-03-06T07:26:27.255000+0000","last_undegraded":"2024-03-06T07:26:27.255000+0000","last_fullsized":"2024-03-06T07:26:27.255000+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:18.328130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849903+0000","last_change":"2024-03-06T07:26:25.627420+0000","last_active":"2024-03-06T07:26:26.849903+0000","last_peered":"2024-03-06T07:26:26.849903+0000","last_clean":"2024-03-06T07:26:26.849903+0000","last_became_active":"2024-03-06T07:26:25.627088+0000","last_became_peered":"2024-03-06T07:26:25.627088+0000","last_unstale":"2024-03-06T07:26:26.849903+0000","last_undegraded":"2024-03-06T07:26:26.849903+0000","last_fullsized":"2024-03-06T07:26:26.849903+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:25.499360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.619831+0000","last_change":"2024-03-06T07:26:24.613368+0000","last_active":"2024-03-06T07:26:25.619831+0000","last_peered":"2024-03-06T07:26:25.619831+0000","last_clean":"2024-03-06T07:26:25.619831+0000","last_became_active":"2024-03-06T07:26:24.613061+0000","last_became_peered":"2024-03-06T07:26:24.613061+0000","last_unstale":"2024-03-06T07:26:25.619831+0000","last_undegraded":"2024-03-06T07:26:25.619831+0000","last_fullsized":"2024-03-06T07:26:25.619831+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:44.752151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.850001+0000","last_change":"2024-03-06T07:26:25.620056+0000","last_active":"2024-03-06T07:26:26.850001+0000","last_peered":"2024-03-06T07:26:26.850001+0000","last_clean":"2024-03-06T07:26:26.850001+0000","last_became_active":"2024-03-06T07:26:25.619789+0000","last_became_peered":"2024-03-06T07:26:25.619789+0000","last_unstale":"2024-03-06T07:26:26.850001+0000","last_undegraded":"2024-03-06T07:26:26.850001+0000","last_fullsized":"2024-03-06T07:26:26.850001+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:37.748889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849974+0000","last_change":"2024-03-06T07:26:24.608448+0000","last_active":"2024-03-06T07:26:26.849974+0000","last_peered":"2024-03-06T07:26:26.849974+0000","last_clean":"2024-03-06T07:26:26.849974+0000","last_became_active":"2024-03-06T07:26:24.608188+0000","last_became_peered":"2024-03-06T07:26:24.608188+0000","last_unstale":"2024-03-06T07:26:26.849974+0000","last_undegraded":"2024-03-06T07:26:26.849974+0000","last_fullsized":"2024-03-06T07:26:26.849974+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:43.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.617351+0000","last_change":"2024-03-06T07:26:25.615829+0000","last_active":"2024-03-06T07:26:25.617351+0000","last_peered":"2024-03-06T07:26:25.617351+0000","last_clean":"2024-03-06T07:26:25.617351+0000","last_became_active":"2024-03-06T07:26:25.615555+0000","last_became_peered":"2024-03-06T07:26:25.615555+0000","last_unstale":"2024-03-06T07:26:25.617351+0000","last_undegraded":"2024-03-06T07:26:25.617351+0000","last_fullsized":"2024-03-06T07:26:25.617351+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:51.359112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254856+0000","last_change":"2024-03-06T07:26:24.614866+0000","last_active":"2024-03-06T07:26:27.254856+0000","last_peered":"2024-03-06T07:26:27.254856+0000","last_clean":"2024-03-06T07:26:27.254856+0000","last_became_active":"2024-03-06T07:26:24.614530+0000","last_became_peered":"2024-03-06T07:26:24.614530+0000","last_unstale":"2024-03-06T07:26:27.254856+0000","last_undegraded":"2024-03-06T07:26:27.254856+0000","last_fullsized":"2024-03-06T07:26:27.254856+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:41:21.196321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.255122+0000","last_change":"2024-03-06T07:26:25.622782+0000","last_active":"2024-03-06T07:26:27.255122+0000","last_peered":"2024-03-06T07:26:27.255122+0000","last_clean":"2024-03-06T07:26:27.255122+0000","last_became_active":"2024-03-06T07:26:25.622468+0000","last_became_peered":"2024-03-06T07:26:25.622468+0000","last_unstale":"2024-03-06T07:26:27.255122+0000","last_undegraded":"2024-03-06T07:26:27.255122+0000","last_fullsized":"2024-03-06T07:26:27.255122+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:28.381957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254747+0000","last_change":"2024-03-06T07:26:24.614951+0000","last_active":"2024-03-06T07:26:27.254747+0000","last_peered":"2024-03-06T07:26:27.254747+0000","last_clean":"2024-03-06T07:26:27.254747+0000","last_became_active":"2024-03-06T07:26:24.614646+0000","last_became_peered":"2024-03-06T07:26:24.614646+0000","last_unstale":"2024-03-06T07:26:27.254747+0000","last_undegraded":"2024-03-06T07:26:27.254747+0000","last_fullsized":"2024-03-06T07:26:27.254747+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:42.938987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.255119+0000","last_change":"2024-03-06T07:26:25.628358+0000","last_active":"2024-03-06T07:26:27.255119+0000","last_peered":"2024-03-06T07:26:27.255119+0000","last_clean":"2024-03-06T07:26:27.255119+0000","last_became_active":"2024-03-06T07:26:25.628076+0000","last_became_peered":"2024-03-06T07:26:25.628076+0000","last_unstale":"2024-03-06T07:26:27.255119+0000","last_undegraded":"2024-03-06T07:26:27.255119+0000","last_fullsized":"2024-03-06T07:26:27.255119+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:13.872137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223413+0000","last_change":"2024-03-06T07:26:24.607243+0000","last_active":"2024-03-06T07:26:27.223413+0000","last_peered":"2024-03-06T07:26:27.223413+0000","last_clean":"2024-03-06T07:26:27.223413+0000","last_became_active":"2024-03-06T07:26:24.603695+0000","last_became_peered":"2024-03-06T07:26:24.603695+0000","last_unstale":"2024-03-06T07:26:27.223413+0000","last_undegraded":"2024-03-06T07:26:27.223413+0000","last_fullsized":"2024-03-06T07:26:27.223413+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:57.460980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.609830+0000","last_change":"2024-03-06T07:26:25.622339+0000","last_active":"2024-03-06T07:26:26.609830+0000","last_peered":"2024-03-06T07:26:26.609830+0000","last_clean":"2024-03-06T07:26:26.609830+0000","last_became_active":"2024-03-06T07:26:25.622032+0000","last_became_peered":"2024-03-06T07:26:25.622032+0000","last_unstale":"2024-03-06T07:26:26.609830+0000","last_undegraded":"2024-03-06T07:26:26.609830+0000","last_fullsized":"2024-03-06T07:26:26.609830+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.391887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849884+0000","last_change":"2024-03-06T07:26:24.606090+0000","last_active":"2024-03-06T07:26:26.849884+0000","last_peered":"2024-03-06T07:26:26.849884+0000","last_clean":"2024-03-06T07:26:26.849884+0000","last_became_active":"2024-03-06T07:26:24.605799+0000","last_became_peered":"2024-03-06T07:26:24.605799+0000","last_unstale":"2024-03-06T07:26:26.849884+0000","last_undegraded":"2024-03-06T07:26:26.849884+0000","last_fullsized":"2024-03-06T07:26:26.849884+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:36.362968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.634375+0000","last_change":"2024-03-06T07:26:25.629403+0000","last_active":"2024-03-06T07:26:25.634375+0000","last_peered":"2024-03-06T07:26:25.634375+0000","last_clean":"2024-03-06T07:26:25.634375+0000","last_became_active":"2024-03-06T07:26:25.628831+0000","last_became_peered":"2024-03-06T07:26:25.628831+0000","last_unstale":"2024-03-06T07:26:25.634375+0000","last_undegraded":"2024-03-06T07:26:25.634375+0000","last_fullsized":"2024-03-06T07:26:25.634375+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:56.287314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"196'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.633612+0000","last_change":"2024-03-06T07:26:24.604972+0000","last_active":"2024-03-06T07:26:26.633612+0000","last_peered":"2024-03-06T07:26:26.633612+0000","last_clean":"2024-03-06T07:26:26.633612+0000","last_became_active":"2024-03-06T07:26:24.604648+0000","last_became_peered":"2024-03-06T07:26:24.604648+0000","last_unstale":"2024-03-06T07:26:26.633612+0000","last_undegraded":"2024-03-06T07:26:26.633612+0000","last_fullsized":"2024-03-06T07:26:26.633612+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:38.380625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849742+0000","last_change":"2024-03-06T07:26:25.617315+0000","last_active":"2024-03-06T07:26:26.849742+0000","last_peered":"2024-03-06T07:26:26.849742+0000","last_clean":"2024-03-06T07:26:26.849742+0000","last_became_active":"2024-03-06T07:26:25.617040+0000","last_became_peered":"2024-03-06T07:26:25.617040+0000","last_unstale":"2024-03-06T07:26:26.849742+0000","last_undegraded":"2024-03-06T07:26:26.849742+0000","last_fullsized":"2024-03-06T07:26:26.849742+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:15:50.040191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223653+0000","last_change":"2024-03-06T07:26:24.615202+0000","last_active":"2024-03-06T07:26:27.223653+0000","last_peered":"2024-03-06T07:26:27.223653+0000","last_clean":"2024-03-06T07:26:27.223653+0000","last_became_active":"2024-03-06T07:26:24.614946+0000","last_became_peered":"2024-03-06T07:26:24.614946+0000","last_unstale":"2024-03-06T07:26:27.223653+0000","last_undegraded":"2024-03-06T07:26:27.223653+0000","last_fullsized":"2024-03-06T07:26:27.223653+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:13.297005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610492+0000","last_change":"2024-03-06T07:26:25.622513+0000","last_active":"2024-03-06T07:26:26.610492+0000","last_peered":"2024-03-06T07:26:26.610492+0000","last_clean":"2024-03-06T07:26:26.610492+0000","last_became_active":"2024-03-06T07:26:25.621481+0000","last_became_peered":"2024-03-06T07:26:25.621481+0000","last_unstale":"2024-03-06T07:26:26.610492+0000","last_undegraded":"2024-03-06T07:26:26.610492+0000","last_fullsized":"2024-03-06T07:26:26.610492+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:56.391887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223754+0000","last_change":"2024-03-06T07:26:24.608641+0000","last_active":"2024-03-06T07:26:27.223754+0000","last_peered":"2024-03-06T07:26:27.223754+0000","last_clean":"2024-03-06T07:26:27.223754+0000","last_became_active":"2024-03-06T07:26:24.607639+0000","last_became_peered":"2024-03-06T07:26:24.607639+0000","last_unstale":"2024-03-06T07:26:27.223754+0000","last_undegraded":"2024-03-06T07:26:27.223754+0000","last_fullsized":"2024-03-06T07:26:27.223754+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:40.485438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.223616+0000","last_change":"2024-03-06T07:26:25.618602+0000","last_active":"2024-03-06T07:26:27.223616+0000","last_peered":"2024-03-06T07:26:27.223616+0000","last_clean":"2024-03-06T07:26:27.223616+0000","last_became_active":"2024-03-06T07:26:25.618323+0000","last_became_peered":"2024-03-06T07:26:25.618323+0000","last_unstale":"2024-03-06T07:26:27.223616+0000","last_undegraded":"2024-03-06T07:26:27.223616+0000","last_fullsized":"2024-03-06T07:26:27.223616+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:48.780230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254886+0000","last_change":"2024-03-06T07:26:24.614649+0000","last_active":"2024-03-06T07:26:27.254886+0000","last_peered":"2024-03-06T07:26:27.254886+0000","last_clean":"2024-03-06T07:26:27.254886+0000","last_became_active":"2024-03-06T07:26:24.614364+0000","last_became_peered":"2024-03-06T07:26:24.614364+0000","last_unstale":"2024-03-06T07:26:27.254886+0000","last_undegraded":"2024-03-06T07:26:27.254886+0000","last_fullsized":"2024-03-06T07:26:27.254886+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:50.307855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606449+0000","last_change":"2024-03-06T07:26:25.612357+0000","last_active":"2024-03-06T07:26:26.606449+0000","last_peered":"2024-03-06T07:26:26.606449+0000","last_clean":"2024-03-06T07:26:26.606449+0000","last_became_active":"2024-03-06T07:26:25.612098+0000","last_became_peered":"2024-03-06T07:26:25.612098+0000","last_unstale":"2024-03-06T07:26:26.606449+0000","last_undegraded":"2024-03-06T07:26:26.606449+0000","last_fullsized":"2024-03-06T07:26:26.606449+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:49.235962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.610411+0000","last_change":"2024-03-06T07:26:24.613234+0000","last_active":"2024-03-06T07:26:26.610411+0000","last_peered":"2024-03-06T07:26:26.610411+0000","last_clean":"2024-03-06T07:26:26.610411+0000","last_became_active":"2024-03-06T07:26:24.612953+0000","last_became_peered":"2024-03-06T07:26:24.612953+0000","last_unstale":"2024-03-06T07:26:26.610411+0000","last_undegraded":"2024-03-06T07:26:26.610411+0000","last_fullsized":"2024-03-06T07:26:26.610411+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:27.376208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.621655+0000","last_change":"2024-03-06T07:26:25.620832+0000","last_active":"2024-03-06T07:26:25.621655+0000","last_peered":"2024-03-06T07:26:25.621655+0000","last_clean":"2024-03-06T07:26:25.621655+0000","last_became_active":"2024-03-06T07:26:25.620572+0000","last_became_peered":"2024-03-06T07:26:25.620572+0000","last_unstale":"2024-03-06T07:26:25.621655+0000","last_undegraded":"2024-03-06T07:26:25.621655+0000","last_fullsized":"2024-03-06T07:26:25.621655+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:47.350661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.849872+0000","last_change":"2024-03-06T07:26:24.613635+0000","last_active":"2024-03-06T07:26:26.849872+0000","last_peered":"2024-03-06T07:26:26.849872+0000","last_clean":"2024-03-06T07:26:26.849872+0000","last_became_active":"2024-03-06T07:26:24.613368+0000","last_became_peered":"2024-03-06T07:26:24.613368+0000","last_unstale":"2024-03-06T07:26:26.849872+0000","last_undegraded":"2024-03-06T07:26:26.849872+0000","last_fullsized":"2024-03-06T07:26:26.849872+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:26.676488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.255116+0000","last_change":"2024-03-06T07:26:25.623636+0000","last_active":"2024-03-06T07:26:27.255116+0000","last_peered":"2024-03-06T07:26:27.255116+0000","last_clean":"2024-03-06T07:26:27.255116+0000","last_became_active":"2024-03-06T07:26:25.622317+0000","last_became_peered":"2024-03-06T07:26:25.622317+0000","last_unstale":"2024-03-06T07:26:27.255116+0000","last_undegraded":"2024-03-06T07:26:27.255116+0000","last_fullsized":"2024-03-06T07:26:27.255116+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:05.833943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:27.254588+0000","last_change":"2024-03-06T07:26:24.612983+0000","last_active":"2024-03-06T07:26:27.254588+0000","last_peered":"2024-03-06T07:26:27.254588+0000","last_clean":"2024-03-06T07:26:27.254588+0000","last_became_active":"2024-03-06T07:26:24.612720+0000","last_became_peered":"2024-03-06T07:26:24.612720+0000","last_unstale":"2024-03-06T07:26:27.254588+0000","last_undegraded":"2024-03-06T07:26:27.254588+0000","last_fullsized":"2024-03-06T07:26:27.254588+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:07.947930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.606276+0000","last_change":"2024-03-06T07:26:25.614981+0000","last_active":"2024-03-06T07:26:26.606276+0000","last_peered":"2024-03-06T07:26:26.606276+0000","last_clean":"2024-03-06T07:26:26.606276+0000","last_became_active":"2024-03-06T07:26:25.614561+0000","last_became_peered":"2024-03-06T07:26:25.614561+0000","last_unstale":"2024-03-06T07:26:26.606276+0000","last_undegraded":"2024-03-06T07:26:26.606276+0000","last_fullsized":"2024-03-06T07:26:26.606276+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:24.590247+0000","last_clean_scrub_stamp":"2024-03-06T07:26:24.590247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:36.251543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-06T07:26:26.611239+0000","last_change":"2024-03-06T07:26:24.606404+0000","last_active":"2024-03-06T07:26:26.611239+0000","last_peered":"2024-03-06T07:26:26.611239+0000","last_clean":"2024-03-06T07:26:26.611239+0000","last_became_active":"2024-03-06T07:26:24.606048+0000","last_became_peered":"2024-03-06T07:26:24.606048+0000","last_unstale":"2024-03-06T07:26:26.611239+0000","last_undegraded":"2024-03-06T07:26:26.611239+0000","last_fullsized":"2024-03-06T07:26:26.611239+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:23.584498+0000","last_clean_scrub_stamp":"2024-03-06T07:26:23.584498+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:05.736515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":500,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-03-06T07:26:25.620198+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:26:25.620198+0000","last_peered":"2024-03-06T07:26:25.620198+0000","last_clean":"2024-03-06T07:26:25.620198+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:26:25.620198+0000","last_undegraded":"2024-03-06T07:26:25.620198+0000","last_fullsized":"2024-03-06T07:26:25.620198+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33668,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338172,"statfs":{"total":96636764160,"available":96602288128,"internally_reserved":0,"allocated":2547712,"data_stored":1790808,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574951,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33612,"kb_used_data":2424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338228,"statfs":{"total":96636764160,"available":96602345472,"internally_reserved":0,"allocated":2482176,"data_stored":1200428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33604,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338236,"statfs":{"total":96636764160,"available":96602353664,"internally_reserved":0,"allocated":2473984,"data_stored":1200316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574951,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33608,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338232,"statfs":{"total":96636764160,"available":96602349568,"internally_reserved":0,"allocated":2478080,"data_stored":1200386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574952,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2498560,"data_stored":1202566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33612,"kb_used_data":2424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338228,"statfs":{"total":96636764160,"available":96602345472,"internally_reserved":0,"allocated":2482176,"data_stored":1200968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37768,"kb_used_data":2428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334072,"statfs":{"total":96636764160,"available":96598089728,"internally_reserved":0,"allocated":2486272,"data_stored":1207008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37832,"kb_used_data":2492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334008,"statfs":{"total":96636764160,"available":96598024192,"internally_reserved":0,"allocated":2551808,"data_stored":1792296,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:28.918 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:28.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:29.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.059+0000 7f2fdb955640 1 Processor -- start 2024-03-06T07:26:29.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.060+0000 7f2fdb955640 1 -- start start 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.060+0000 7f2fdb955640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2fd414d350 0x7f2fd406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.060+0000 7f2fdb955640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd415dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.060+0000 7f2fdb955640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40c1c90 con 0x7f2fd40acbc0 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.060+0000 7f2fdb955640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd40ac750 con 0x7f2fd414d350 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.060+0000 7f2fdb955640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40c1a40 con 0x7f2fd415b8c0 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd96f1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2fd414d350 0x7f2fd406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd96f1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2fd414d350 0x7f2fd406afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49080/0 (socket says 172.21.15.94:49080) 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd96f1640 1 -- 172.21.15.94:0/2394005958 learned_addr learned my addr 172.21.15.94:0/2394005958 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd9ef2640 1 --2- 172.21.15.94:0/2394005958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd415dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd96f1640 1 -- 172.21.15.94:0/2394005958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 msgr2=0x7f2fd415dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd96f1640 1 --2- 172.21.15.94:0/2394005958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd415dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd96f1640 1 -- 172.21.15.94:0/2394005958 >> v1:172.21.15.145:6789/0 conn(0x7f2fd40acbc0 legacy=0x7f2fd40ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.061+0000 7f2fd96f1640 1 -- 172.21.15.94:0/2394005958 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40ac470 con 0x7f2fd414d350 2024-03-06T07:26:29.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fd96f1640 1 --2- 172.21.15.94:0/2394005958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd406afc0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009870 tx=0x7f2fd002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f2ec422521d229 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fcbfff640 1 -- 172.21.15.94:0/2394005958 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0035020 con 0x7f2fd414d350 2024-03-06T07:26:29.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fcbfff640 1 -- 172.21.15.94:0/2394005958 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0031640 con 0x7f2fd414d350 2024-03-06T07:26:29.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fcbfff640 1 -- 172.21.15.94:0/2394005958 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd002d040 con 0x7f2fd414d350 2024-03-06T07:26:29.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 -- 172.21.15.94:0/2394005958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 msgr2=0x7f2fd406afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2394005958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd406afc0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009870 tx=0x7f2fd002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 -- 172.21.15.94:0/2394005958 shutdown_connections 2024-03-06T07:26:29.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2394005958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd415dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2394005958 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd406afc0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 -- 172.21.15.94:0/2394005958 >> 172.21.15.94:0/2394005958 conn(0x7f2fd40c3b40 msgr2=0x7f2fd40c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 -- 172.21.15.94:0/2394005958 shutdown_connections 2024-03-06T07:26:29.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.062+0000 7f2fdb955640 1 -- 172.21.15.94:0/2394005958 wait complete. 2024-03-06T07:26:29.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.063+0000 7f2fdb955640 1 Processor -- start 2024-03-06T07:26:29.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fdb955640 1 -- start start 2024-03-06T07:26:29.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fdb955640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fd40acbc0 0x7f2fd4156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fdb955640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd41403c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fdb955640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd4140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fdb955640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40aa160 con 0x7f2fd414d350 2024-03-06T07:26:29.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fdb955640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4153c00 con 0x7f2fd40acbc0 2024-03-06T07:26:29.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fdb955640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40aa2e0 con 0x7f2fd415b8c0 2024-03-06T07:26:29.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fd8ef0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd41403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fd8ef0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd41403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49092/0 (socket says 172.21.15.94:49092) 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fd8ef0640 1 -- 172.21.15.94:0/2071469642 learned_addr learned my addr 172.21.15.94:0/2071469642 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fd96f1640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fd40acbc0 0x7f2fd4156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.064+0000 7f2fd9ef2640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd4140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fd8ef0640 1 -- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 msgr2=0x7f2fd4140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fd8ef0640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd4140900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fd8ef0640 1 -- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fd40acbc0 msgr2=0x7f2fd4156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fd8ef0640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fd40acbc0 0x7f2fd4156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fd8ef0640 1 -- 172.21.15.94:0/2071469642 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd41482b0 con 0x7f2fd414d350 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fd8ef0640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd41403c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400c400 tx=0x7f2fc400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fc9ffb640 1 -- 172.21.15.94:0/2071469642 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc40052d0 con 0x7f2fd414d350 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fc9ffb640 1 -- 172.21.15.94:0/2071469642 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc4005490 con 0x7f2fd414d350 2024-03-06T07:26:29.069 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd4149f10 con 0x7f2fd414d350 2024-03-06T07:26:29.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fc9ffb640 1 -- 172.21.15.94:0/2071469642 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4007260 con 0x7f2fd414d350 2024-03-06T07:26:29.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.065+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd414a4d0 con 0x7f2fd414d350 2024-03-06T07:26:29.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.066+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd4148480 con 0x7f2fd414d350 2024-03-06T07:26:29.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.066+0000 7f2fc9ffb640 1 -- 172.21.15.94:0/2071469642 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fc4007400 con 0x7f2fd414d350 2024-03-06T07:26:29.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.067+0000 7f2fc9ffb640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fb806b4d0 0x7f2fb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.067+0000 7f2fc9ffb640 1 -- 172.21.15.94:0/2071469642 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2fc4089780 con 0x7f2fd414d350 2024-03-06T07:26:29.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.067+0000 7f2fd96f1640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fb806b4d0 0x7f2fb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.067+0000 7f2fd96f1640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fb806b4d0 0x7f2fb806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0036000 tx=0x7f2fd003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.071+0000 7f2fc9ffb640 1 -- 172.21.15.94:0/2071469642 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fc4057eb0 con 0x7f2fd414d350 2024-03-06T07:26:29.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.305+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2fd404f330 con 0x7f2fd414d350 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.306+0000 7f2fc9ffb640 1 -- 172.21.15.94:0/2071469642 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2fc405bf50 con 0x7f2fd414d350 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:26:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fb806b4d0 msgr2=0x7f2fb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fb806b4d0 0x7f2fb806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0036000 tx=0x7f2fd003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 msgr2=0x7f2fd41403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd41403c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400c400 tx=0x7f2fc400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 shutdown_connections 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2fb806b4d0 0x7f2fb806d990 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fd415b8c0 0x7f2fd4140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2fd414d350 0x7f2fd41403c0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 --2- 172.21.15.94:0/2071469642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fd40acbc0 0x7f2fd4156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.308+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 >> 172.21.15.94:0/2071469642 conn(0x7f2fd40c3b40 msgr2=0x7f2fd40ab8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.309+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 shutdown_connections 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.309+0000 7f2fdb955640 1 -- 172.21.15.94:0/2071469642 wait complete. 2024-03-06T07:26:29.321 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:29.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.418+0000 7efc1e793640 1 Processor -- start 2024-03-06T07:26:29.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.418+0000 7efc1e793640 1 -- start start 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc1e793640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efc180c1aa0 0x7efc181529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc1e793640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc18155a40 0x7efc18155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc1e793640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc180c28d0 con 0x7efc1814f790 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc1e793640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc180c2680 con 0x7efc180c1aa0 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc1e793640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc180c2bb0 con 0x7efc18155a40 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc177fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efc180c1aa0 0x7efc181529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc17fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efc1814f790 0x7efc180c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47780/0 (socket says 172.21.15.94:47780) 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc177fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efc180c1aa0 0x7efc181529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49104/0 (socket says 172.21.15.94:49104) 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc17fff640 1 -- 172.21.15.94:0/598869016 learned_addr learned my addr 172.21.15.94:0/598869016 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.419+0000 7efc1cd30640 1 --2- 172.21.15.94:0/598869016 >> v2:172.21.15.145:3300/0 conn(0x7efc18155a40 0x7efc18155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.420+0000 7efc177fe640 1 -- 172.21.15.94:0/598869016 >> v2:172.21.15.145:3300/0 conn(0x7efc18155a40 msgr2=0x7efc18155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.420+0000 7efc177fe640 1 --2- 172.21.15.94:0/598869016 >> v2:172.21.15.145:3300/0 conn(0x7efc18155a40 0x7efc18155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.420+0000 7efc177fe640 1 -- 172.21.15.94:0/598869016 >> v1:172.21.15.94:6789/0 conn(0x7efc1814f790 legacy=0x7efc180c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.420+0000 7efc177fe640 1 -- 172.21.15.94:0/598869016 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc180c2e90 con 0x7efc180c1aa0 2024-03-06T07:26:29.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.420+0000 7efc16ffd640 1 -- 172.21.15.94:0/598869016 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653887051 0 0) 0x7efc180c28d0 con 0x7efc1814f790 2024-03-06T07:26:29.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.420+0000 7efc177fe640 1 --2- 172.21.15.94:0/598869016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181529d0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7efc1806c960 tx=0x7efc0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e8e1b514d1377f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc16ffd640 1 -- 172.21.15.94:0/598869016 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c035020 con 0x7efc180c1aa0 2024-03-06T07:26:29.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc16ffd640 1 -- 172.21.15.94:0/598869016 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c031830 con 0x7efc180c1aa0 2024-03-06T07:26:29.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc16ffd640 1 -- 172.21.15.94:0/598869016 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c031c00 con 0x7efc180c1aa0 2024-03-06T07:26:29.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 -- 172.21.15.94:0/598869016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 msgr2=0x7efc181529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 --2- 172.21.15.94:0/598869016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181529d0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7efc1806c960 tx=0x7efc0c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 -- 172.21.15.94:0/598869016 shutdown_connections 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 --2- 172.21.15.94:0/598869016 >> v2:172.21.15.145:3300/0 conn(0x7efc18155a40 0x7efc18155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 --2- 172.21.15.94:0/598869016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181529d0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 -- 172.21.15.94:0/598869016 >> 172.21.15.94:0/598869016 conn(0x7efc180c6710 msgr2=0x7efc180c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 -- 172.21.15.94:0/598869016 shutdown_connections 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.421+0000 7efc1e793640 1 -- 172.21.15.94:0/598869016 wait complete. 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.422+0000 7efc1e793640 1 Processor -- start 2024-03-06T07:26:29.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.422+0000 7efc1e793640 1 -- start start 2024-03-06T07:26:29.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc1e793640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc1e793640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc1814f790 0x7efc1814c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc1e793640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc18155a40 0x7efc180702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc1e793640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc180c0d60 con 0x7efc180c1aa0 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc1e793640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc18156390 con 0x7efc1814f790 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc1e793640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc180c5e30 con 0x7efc18155a40 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181eb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49106/0 (socket says 172.21.15.94:49106) 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 -- 172.21.15.94:0/496604064 learned_addr learned my addr 172.21.15.94:0/496604064 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc177fe640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc1814f790 0x7efc1814c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc1cd30640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc18155a40 0x7efc180702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 -- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc18155a40 msgr2=0x7efc180702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc18155a40 0x7efc180702f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 -- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc1814f790 msgr2=0x7efc1814c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc1814f790 0x7efc1814c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.423+0000 7efc17fff640 1 -- 172.21.15.94:0/496604064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1814cbe0 con 0x7efc180c1aa0 2024-03-06T07:26:29.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.424+0000 7efc17fff640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181eb4f0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7efc0800c500 tx=0x7efc0800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.424+0000 7efc14ff9640 1 -- 172.21.15.94:0/496604064 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08018070 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.424+0000 7efc14ff9640 1 -- 172.21.15.94:0/496604064 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08007580 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.424+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc18150d90 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.424+0000 7efc14ff9640 1 -- 172.21.15.94:0/496604064 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08004a20 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.424+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc181517d0 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.425+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1804f330 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.425+0000 7efc14ff9640 1 -- 172.21.15.94:0/496604064 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc0801b070 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.425+0000 7efc14ff9640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc0406b4d0 0x7efc0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.426+0000 7efc14ff9640 1 -- 172.21.15.94:0/496604064 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efc08089560 con 0x7efc180c1aa0 2024-03-06T07:26:29.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.426+0000 7efc177fe640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc0406b4d0 0x7efc0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.426+0000 7efc177fe640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc0406b4d0 0x7efc0406d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7efc0c036000 tx=0x7efc0c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.430+0000 7efc14ff9640 1 -- 172.21.15.94:0/496604064 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc08057c90 con 0x7efc180c1aa0 2024-03-06T07:26:29.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.636+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc180c1f40 con 0x7efc180c1aa0 2024-03-06T07:26:29.638 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.637+0000 7efc14ff9640 1 -- 172.21.15.94:0/496604064 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+4995 (secure 0 0 0) 0x7efc0805bd30 con 0x7efc180c1aa0 2024-03-06T07:26:29.639 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:29.639 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":186,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10428,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2752830090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2752830090},{"type":"v1","addr":"172.21.15.94:6834","nonce":2752830090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":10431,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/750154668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":750154668},{"type":"v1","addr":"172.21.15.94:6835","nonce":750154668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":38057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/565245371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":565245371},{"type":"v1","addr":"172.21.15.145:6836","nonce":565245371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:24.949828+0000","modified":"2024-03-06T07:26:26.955353+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38060},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38060":{"gid":38060,"name":"d","rank":0,"incarnation":184,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/4067343150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":4067343150},{"type":"v1","addr":"172.21.15.145:6837","nonce":4067343150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38060,"qdb_cluster":[38060]},"id":15}]} 2024-03-06T07:26:29.639 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 186 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc0406b4d0 msgr2=0x7efc0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc0406b4d0 0x7efc0406d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7efc0c036000 tx=0x7efc0c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 msgr2=0x7efc181eb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181eb4f0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7efc0800c500 tx=0x7efc0800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 shutdown_connections 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc0406b4d0 0x7efc0406d990 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc18155a40 0x7efc180702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc1814f790 0x7efc1814c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 --2- 172.21.15.94:0/496604064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc180c1aa0 0x7efc181eb4f0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 >> 172.21.15.94:0/496604064 conn(0x7efc180c6710 msgr2=0x7efc1814d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 shutdown_connections 2024-03-06T07:26:29.641 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:29.639+0000 7efc1e793640 1 -- 172.21.15.94:0/496604064 wait complete. 2024-03-06T07:26:30.055 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 186, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:24.949828+0000', 'modified': '2024-03-06T07:26:26.955353+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 38060}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_38060': {'gid': 38060, 'name': 'd', 'rank': 0, 'incarnation': 184, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/4067343150', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 4067343150}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 4067343150}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38060, 'qdb_cluster': [38060]} 2024-03-06T07:26:30.055 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:30.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.157+0000 7f86b0993640 1 Processor -- start 2024-03-06T07:26:30.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.157+0000 7f86b0993640 1 -- start start 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86b0993640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f86ac158fb0 0x7f86ac0c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86b0993640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac1553c0 con 0x7f86ac0ab860 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86b0993640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac1556a0 con 0x7f86ac06c240 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86b0993640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac155980 con 0x7f86ac158fb0 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86a9d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f86ac0ab860 0x7f86ac1588a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47796/0 (socket says 172.21.15.94:47796) 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86a9d9b640 1 -- 172.21.15.94:0/265242027 learned_addr learned my addr 172.21.15.94:0/265242027 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86aad9d640 1 --2- 172.21.15.94:0/265242027 >> v2:172.21.15.145:3300/0 conn(0x7f86ac158fb0 0x7f86ac0c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809120048 0 0) 0x7f86ac1553c0 con 0x7f86ac0ab860 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.158+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8694003520 con 0x7f86ac0ab860 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031445691 0 0) 0x7f86ac1556a0 con 0x7f86ac06c240 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ac1553c0 con 0x7f86ac06c240 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1084994397 0 0) 0x7f8694003520 con 0x7f86ac0ab860 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 >> v2:172.21.15.145:3300/0 conn(0x7f86ac158fb0 msgr2=0x7f86ac0c4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 --2- 172.21.15.94:0/265242027 >> v2:172.21.15.145:3300/0 conn(0x7f86ac158fb0 0x7f86ac0c4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 >> v1:172.21.15.145:6789/0 conn(0x7f86ac06c240 legacy=0x7f86ac0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac155c60 con 0x7f86ac0ab860 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f86a0001670 con 0x7f86ac0ab860 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86aad9d640 1 --2- 172.21.15.94:0/265242027 >> v2:172.21.15.145:3300/0 conn(0x7f86ac158fb0 0x7f86ac0c4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 >> v1:172.21.15.94:6789/0 conn(0x7f86ac0ab860 legacy=0x7f86ac1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8694003b40 0x7f8694003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f8694004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86940051c0 0x7f869400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac155980 con 0x7f8694004800 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869400c0f0 con 0x7f8694003b40 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869400c2f0 con 0x7f86940051c0 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f8694004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86aa59c640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8694003b40 0x7f8694003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86aad9d640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86940051c0 0x7f869400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a9d9b640 1 -- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86940051c0 msgr2=0x7f869400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86940051c0 0x7f869400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a9d9b640 1 -- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8694003b40 msgr2=0x7f8694003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8694003b40 0x7f8694003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86a9d9b640 1 -- 172.21.15.94:0/265242027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694003520 con 0x7f8694004800 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.159+0000 7f86aad9d640 1 -- 172.21.15.94:0/265242027 reap_dead start 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.160+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f8694004c80 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0abcb0 tx=0x7f8694005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166fa0c61b8a23e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.160+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0007ca0 con 0x7f8694004800 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.160+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a0030070 con 0x7f8694004800 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.160+0000 7f86a959a640 1 -- 172.21.15.94:0/265242027 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0005d70 con 0x7f8694004800 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.160+0000 7f86b0993640 1 -- 172.21.15.94:0/265242027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 msgr2=0x7f8694004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.160+0000 7f86b0993640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f8694004c80 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0abcb0 tx=0x7f8694005830 comp rx=0 tx=0).stop 2024-03-06T07:26:30.165 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.161+0000 7f86b0993640 1 -- 172.21.15.94:0/265242027 shutdown_connections 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.161+0000 7f86b0993640 1 --2- 172.21.15.94:0/265242027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f8694004c80 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.161+0000 7f86b0993640 1 -- 172.21.15.94:0/265242027 >> 172.21.15.94:0/265242027 conn(0x7f86ac0c7670 msgr2=0x7f86ac0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.161+0000 7f86b0993640 1 -- 172.21.15.94:0/265242027 shutdown_connections 2024-03-06T07:26:30.166 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.161+0000 7f86b0993640 1 -- 172.21.15.94:0/265242027 wait complete. 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 Processor -- start 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 -- start start 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f86ac1509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86ac1575e0 0x7f86ac151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86ac152270 0x7f86ac14bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0c4830 con 0x7f8694004800 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac156180 con 0x7f86ac152270 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.162+0000 7f86b0993640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac06c620 con 0x7f86ac1575e0 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f86ac1509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f86ac1509f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49128/0 (socket says 172.21.15.94:49128) 2024-03-06T07:26:30.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 -- 172.21.15.94:0/3965707790 learned_addr learned my addr 172.21.15.94:0/3965707790 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aad9d640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86ac152270 0x7f86ac14bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86ac1575e0 0x7f86ac151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 -- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86ac1575e0 msgr2=0x7f86ac151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86ac1575e0 0x7f86ac151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 -- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86ac152270 msgr2=0x7f86ac14bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86ac152270 0x7f86ac14bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 -- 172.21.15.94:0/3965707790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac14f6e0 con 0x7f8694004800 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86aa59c640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f86ac1509f0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f869c00fb40 tx=0x7f869c002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f869affd640 1 -- 172.21.15.94:0/3965707790 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c01a070 con 0x7f8694004800 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f869affd640 1 -- 172.21.15.94:0/3965707790 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c006d60 con 0x7f8694004800 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac156b70 con 0x7f8694004800 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f869affd640 1 -- 172.21.15.94:0/3965707790 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c003b90 con 0x7f8694004800 2024-03-06T07:26:30.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.163+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac156e50 con 0x7f8694004800 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.165+0000 7f869affd640 1 -- 172.21.15.94:0/3965707790 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f869c01d070 con 0x7f8694004800 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.165+0000 7f869affd640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f868c06b4d0 0x7f868c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.165+0000 7f869affd640 1 -- 172.21.15.94:0/3965707790 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f869c089e00 con 0x7f8694004800 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.165+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f868c06b4d0 0x7f868c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.166+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f868c06b4d0 0x7f868c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f86a0034000 tx=0x7f86a0007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.166+0000 7f86b357c740 1 Processor -- start 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.166+0000 7f86b357c740 1 -- start start 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x56141607aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86b357c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x56141607af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607b4b0 0x561416077680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86b357c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x561415fe16b0 con 0x56141607d330 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561415f8aa60 con 0x56141607b4b0 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561415fdd750 con 0x56141607cf50 2024-03-06T07:26:30.173 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x56141607af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86aad9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607b4b0 0x561416077680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x56141607af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49140/0 (socket says 172.21.15.94:49140) 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 -- 172.21.15.94:0/4026021438 learned_addr learned my addr 172.21.15.94:0/4026021438 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86aa59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x56141607aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 -- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 msgr2=0x56141607aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x56141607aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 -- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607b4b0 msgr2=0x561416077680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607b4b0 0x561416077680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86a9d9b640 1 -- 172.21.15.94:0/4026021438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561415fa8f30 con 0x56141607d330 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86aad9d640 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607b4b0 0x561416077680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.167+0000 7f86aa59c640 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x56141607aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x56141607af70 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f86a0007fd0 tx=0x7f86a002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e70353a37c6db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f868bfff640 1 -- 172.21.15.94:0/4026021438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a002ca90 con 0x56141607d330 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f868bfff640 1 -- 172.21.15.94:0/4026021438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a0030070 con 0x56141607d330 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f868bfff640 1 -- 172.21.15.94:0/4026021438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a002ce30 con 0x56141607d330 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 -- 172.21.15.94:0/4026021438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 msgr2=0x56141607af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x56141607af70 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f86a0007fd0 tx=0x7f86a002e000 comp rx=0 tx=0).stop 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 -- 172.21.15.94:0/4026021438 shutdown_connections 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607b4b0 0x561416077680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x56141607af70 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 --2- 172.21.15.94:0/4026021438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x56141607aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 -- 172.21.15.94:0/4026021438 >> 172.21.15.94:0/4026021438 conn(0x5614160797d0 msgr2=0x56141609e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 -- 172.21.15.94:0/4026021438 shutdown_connections 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.168+0000 7f86b357c740 1 -- 172.21.15.94:0/4026021438 wait complete. 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.169+0000 7f86b357c740 1 -- start start 2024-03-06T07:26:30.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.169+0000 7f86b357c740 1 Processor -- start 2024-03-06T07:26:30.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86b357c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607cf50 0x56141609d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86b357c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56141609f230 con 0x56141607cf50 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561415f9dea0 con 0x56141607d330 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56141609ec70 con 0x5614160a4ad0 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86b357c740 10 client.0 _ll_register_callbacks cb 0x561415f9d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607cf50 0x56141609d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607cf50 0x56141609d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49150/0 (socket says 172.21.15.94:49150) 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 -- 172.21.15.94:0/1148611137 learned_addr learned my addr 172.21.15.94:0/1148611137 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aad9d640 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 -- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 msgr2=0x5614160a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 -- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 msgr2=0x5614160a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 -- 172.21.15.94:0/1148611137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561415f9e270 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86aa59c640 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607cf50 0x56141609d7b0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f869c00d5e0 tx=0x7f869c058690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff9876cee6f558d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.170+0000 7f86897fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86897fa640 1 -- 172.21.15.94:0/1148611137 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c081d90 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86897fa640 1 -- 172.21.15.94:0/1148611137 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f869c057980 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86a0030070 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86897fa640 1 -- 172.21.15.94:0/1148611137 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=186) v2 ==== 24+0+0 (secure 0 0 0) 0x7f869c0568f0 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 10 client.10572 fetch_fsmap learned FSMap version 186 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=186,monmap=2+}) v3 -- 0x561415cf62d0 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86897fa640 1 -- 172.21.15.94:0/1148611137 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a0030070 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86897fa640 1 -- 172.21.15.94:0/1148611137 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 186) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f869c054280 con 0x56141607cf50 2024-03-06T07:26:30.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 10 client.10572 fetch_fsmap finished waiting for FSMap version 186 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 10 client.10572 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 20 client.10572 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160cd8c0 0x5614160d84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 4 client.10572 mds_command: new command op to 38060 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 --> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561415feebf0 con 0x5614160cd8c0 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.171+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160cd8c0 0x5614160d84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160cd8c0 0x5614160d84e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca4679680882b009 server_cookie=80e10f606bf1306 in_seq=0 out_seq=0 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86897fa640 10 client.10572 ms_handle_connect on v2:172.21.15.145:6835/4067343150 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86897fa640 1 -- 172.21.15.94:0/1148611137 <== mds.0 v2:172.21.15.145:6835/4067343150 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561415feebf0 con 0x5614160cd8c0 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86897fa640 10 client.10572 handle_command_reply: tid=0 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 client.10572 shutdown 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607cf50 msgr2=0x56141609d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607cf50 0x56141609d7b0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f869c00d5e0 tx=0x7f869c058690 comp rx=0 tx=0).stop 2024-03-06T07:26:30.182 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 shutdown_connections 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160cd8c0 0x5614160d84e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86897fa640 0 client.10572 ms_handle_reset on v2:172.21.15.145:6835/4067343150 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 --2- 172.21.15.94:0/1148611137 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607cf50 0x56141609d7b0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.172+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 >> 172.21.15.94:0/1148611137 conn(0x56141607b4b0 msgr2=0x56141607c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.173+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 shutdown_connections 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.173+0000 7f86b357c740 1 -- 172.21.15.94:0/1148611137 wait complete. 2024-03-06T07:26:30.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.173+0000 7f86b357c740 20 client.10572 trim_cache size 0 max 16384 2024-03-06T07:26:30.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.189+0000 7f86b357c740 1 Processor -- start 2024-03-06T07:26:30.191 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.189+0000 7f86b357c740 1 -- start start 2024-03-06T07:26:30.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607cf50 0x561416076930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86b357c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86b357c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac1f7500 con 0x56141607d330 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac1f6900 con 0x56141607cf50 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac1f6f00 con 0x5614160a4ad0 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49164/0 (socket says 172.21.15.94:49164) 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 -- 172.21.15.94:0/1441420963 learned_addr learned my addr 172.21.15.94:0/1441420963 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86aa59c640 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607cf50 0x561416076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86aad9d640 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 -- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 msgr2=0x5614160a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 -- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607cf50 msgr2=0x561416076930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607cf50 0x561416076930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 -- 172.21.15.94:0/1441420963 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561415dc06b0 con 0x56141607d330 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x5614160a2a30 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f86a002e040 tx=0x7f86a003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fcefc40bcf41d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f868b7fe640 1 -- 172.21.15.94:0/1441420963 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a005a2d0 con 0x56141607d330 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f868b7fe640 1 -- 172.21.15.94:0/1441420963 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a0030040 con 0x56141607d330 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.190+0000 7f868b7fe640 1 -- 172.21.15.94:0/1441420963 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a002cc90 con 0x56141607d330 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 -- 172.21.15.94:0/1441420963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 msgr2=0x5614160a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x5614160a2a30 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f86a002e040 tx=0x7f86a003d000 comp rx=0 tx=0).stop 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 -- 172.21.15.94:0/1441420963 shutdown_connections 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5614160a4ad0 0x5614160a1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x56141607d330 0x5614160a2a30 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 --2- 172.21.15.94:0/1441420963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607cf50 0x561416076930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 -- 172.21.15.94:0/1441420963 >> 172.21.15.94:0/1441420963 conn(0x56141607b4b0 msgr2=0x56141609d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 -- 172.21.15.94:0/1441420963 shutdown_connections 2024-03-06T07:26:30.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 -- 172.21.15.94:0/1441420963 wait complete. 2024-03-06T07:26:30.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.191+0000 7f86b357c740 1 -- start start 2024-03-06T07:26:30.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 1 Processor -- start 2024-03-06T07:26:30.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x5614160a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x56141609c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5614160a4ad0 0x561416126980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x56141609fbc0 con 0x5614160a4ad0 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56141609f600 con 0x56141607d330 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5614160a0220 con 0x56141607cf50 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.192+0000 7f86b357c740 10 client.0 _ll_register_callbacks cb 0x7f869c0019f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5614160a4ad0 0x561416126980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5614160a4ad0 0x561416126980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49176/0 (socket says 172.21.15.94:49176) 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 -- 172.21.15.94:0/4079089574 learned_addr learned my addr 172.21.15.94:0/4079089574 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86a9d9b640 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x56141609c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aa59c640 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x5614160a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 -- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 msgr2=0x5614160a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x5614160a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 -- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 msgr2=0x56141609c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x56141609c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 -- 172.21.15.94:0/4079089574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5614160b6810 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86aad9d640 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5614160a4ad0 0x561416126980 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f86ac06ef00 tx=0x7f86a4015a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8088cd5528a028f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86897fa640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86897fa640 1 -- 172.21.15.94:0/4079089574 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a40146e0 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86897fa640 1 -- 172.21.15.94:0/4079089574 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f86a4014880 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86a0030040 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86897fa640 1 -- 172.21.15.94:0/4079089574 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=186) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86a402d050 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86b357c740 10 client.10578 fetch_fsmap learned FSMap version 186 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=186,monmap=2+}) v3 -- 0x56141612d090 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.193+0000 7f86897fa640 1 -- 172.21.15.94:0/4079089574 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a0030040 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86897fa640 1 -- 172.21.15.94:0/4079089574 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 186) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f86a4030020 con 0x5614160a4ad0 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 10 client.10578 fetch_fsmap finished waiting for FSMap version 186 2024-03-06T07:26:30.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 10 client.10578 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 20 client.10578 populate_metadata read hostname 'smithi094' 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160dc620 0x5614160dca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 4 client.10578 mds_command: new command op to 38060 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 --> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561415fe8f40 con 0x5614160dc620 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86aa59c640 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160dc620 0x5614160dca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86aa59c640 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160dc620 0x5614160dca00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10f4c10a258b102 server_cookie=9d9d9aa335b2cafb in_seq=0 out_seq=0 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86897fa640 10 client.10578 ms_handle_connect on v2:172.21.15.145:6835/4067343150 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86897fa640 1 -- 172.21.15.94:0/4079089574 <== mds.0 v2:172.21.15.145:6835/4067343150 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x561415fe8f40 con 0x5614160dc620 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86897fa640 10 client.10578 handle_command_reply: tid=0 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 client.10578 shutdown 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5614160a4ad0 msgr2=0x561416126980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5614160a4ad0 0x561416126980 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f86ac06ef00 tx=0x7f86a4015a80 comp rx=0 tx=0).stop 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 shutdown_connections 2024-03-06T07:26:30.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:6835/4067343150,v1:172.21.15.145:6837/4067343150] conn(0x5614160dc620 0x5614160dca00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5614160a4ad0 0x561416126980 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86897fa640 0 client.10578 ms_handle_reset on v2:172.21.15.145:6835/4067343150 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56141607d330 0x56141609c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.194+0000 7f86b357c740 1 --2- 172.21.15.94:0/4079089574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56141607cf50 0x5614160a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 >> 172.21.15.94:0/4079089574 conn(0x56141607b4b0 msgr2=0x56141607b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 shutdown_connections 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b357c740 1 -- 172.21.15.94:0/4079089574 wait complete. 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b357c740 20 client.10578 trim_cache size 0 max 16384 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:26:30.203 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 38060, 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 4.5788002800000003, 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 186, 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 196, 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 196, 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 9.6786775259999995 2024-03-06T07:26:30.204 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:26:30.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f868c06b4d0 msgr2=0x7f868c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f868c06b4d0 0x7f868c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f86a0034000 tx=0x7f86a0007000 comp rx=0 tx=0).stop 2024-03-06T07:26:30.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 msgr2=0x7f86ac1509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f86ac1509f0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f869c00fb40 tx=0x7f869c002170 comp rx=0 tx=0).stop 2024-03-06T07:26:30.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 shutdown_connections 2024-03-06T07:26:30.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f868c06b4d0 0x7f868c06d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86ac152270 0x7f86ac14bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86ac1575e0 0x7f86ac151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 --2- 172.21.15.94:0/3965707790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8694004800 0x7f86ac1509f0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 >> 172.21.15.94:0/3965707790 conn(0x7f86ac0c7670 msgr2=0x7f86ac06dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 shutdown_connections 2024-03-06T07:26:30.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.195+0000 7f86b0993640 1 -- 172.21.15.94:0/3965707790 wait complete. 2024-03-06T07:26:30.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:26:30.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.308+0000 7fdcabfc7640 1 Processor -- start 2024-03-06T07:26:30.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.309+0000 7fdcabfc7640 1 -- start start 2024-03-06T07:26:30.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.309+0000 7fdcabfc7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdca40ab170 0x7fdca40c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.309+0000 7fdcabfc7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40c3ef0 con 0x7fdca40c5990 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.309+0000 7fdcabfc7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca41556d0 con 0x7fdca406c900 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.309+0000 7fdcabfc7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca41559b0 con 0x7fdca40ab170 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.309+0000 7fdcaa564640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdca40ab170 0x7fdca40c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.310+0000 7fdcaa564640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdca40ab170 0x7fdca40c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49192/0 (socket says 172.21.15.94:49192) 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.310+0000 7fdca9d63640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdca40c5990 0x7fdca406c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47800/0 (socket says 172.21.15.94:47800) 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.310+0000 7fdcaa564640 1 -- 172.21.15.94:0/1694338967 learned_addr learned my addr 172.21.15.94:0/1694338967 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.310+0000 7fdcaa564640 1 -- 172.21.15.94:0/1694338967 >> v1:172.21.15.145:6789/0 conn(0x7fdca406c900 legacy=0x7fdca40a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.310+0000 7fdcaa564640 1 -- 172.21.15.94:0/1694338967 >> v1:172.21.15.94:6789/0 conn(0x7fdca40c5990 legacy=0x7fdca406c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.310+0000 7fdca8d61640 1 -- 172.21.15.94:0/1694338967 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105174583 0 0) 0x7fdca40c3ef0 con 0x7fdca40c5990 2024-03-06T07:26:30.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.310+0000 7fdcaa564640 1 -- 172.21.15.94:0/1694338967 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4155c90 con 0x7fdca40ab170 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcaa564640 1 --2- 172.21.15.94:0/1694338967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca40c0ab0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00d180 tx=0x7fdc9c02e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18450f72dadeaaca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdca8d61640 1 -- 172.21.15.94:0/1694338967 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c00b0d0 con 0x7fdca40ab170 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdca8d61640 1 -- 172.21.15.94:0/1694338967 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c00b270 con 0x7fdca40ab170 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdca8d61640 1 -- 172.21.15.94:0/1694338967 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c009600 con 0x7fdca40ab170 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcabfc7640 1 -- 172.21.15.94:0/1694338967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 msgr2=0x7fdca40c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/1694338967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca40c0ab0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00d180 tx=0x7fdc9c02e1f0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcabfc7640 1 -- 172.21.15.94:0/1694338967 shutdown_connections 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/1694338967 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca40c0ab0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcabfc7640 1 -- 172.21.15.94:0/1694338967 >> 172.21.15.94:0/1694338967 conn(0x7fdca40c6710 msgr2=0x7fdca40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcabfc7640 1 -- 172.21.15.94:0/1694338967 shutdown_connections 2024-03-06T07:26:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.311+0000 7fdcabfc7640 1 -- 172.21.15.94:0/1694338967 wait complete. 2024-03-06T07:26:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.312+0000 7fdcabfc7640 1 Processor -- start 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.312+0000 7fdcabfc7640 1 -- start start 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdcabfc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca406c900 0x7fdca4154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdcabfc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca414df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdcabfc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca40c5990 0x7fdca414e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdcabfc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4156610 con 0x7fdca40ab170 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdcabfc7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca4156490 con 0x7fdca406c900 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdcabfc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca40c3af0 con 0x7fdca40c5990 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdca9562640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca414df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdca9562640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca414df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49200/0 (socket says 172.21.15.94:49200) 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdca9562640 1 -- 172.21.15.94:0/3722780891 learned_addr learned my addr 172.21.15.94:0/3722780891 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdca9d63640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca406c900 0x7fdca4154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdcaa564640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca40c5990 0x7fdca414e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdca9562640 1 -- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca40c5990 msgr2=0x7fdca414e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdca9562640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca40c5990 0x7fdca414e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.313+0000 7fdca9562640 1 -- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca406c900 msgr2=0x7fdca4154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdca9562640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca406c900 0x7fdca4154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdca9562640 1 -- 172.21.15.94:0/3722780891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca41e6bf0 con 0x7fdca40ab170 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdca9562640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca414df90 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c480 tx=0x7fdc9800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdc967fc640 1 -- 172.21.15.94:0/3722780891 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc980052d0 con 0x7fdca40ab170 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdc967fc640 1 -- 172.21.15.94:0/3722780891 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc98005470 con 0x7fdca40ab170 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca40c1a00 con 0x7fdca40ab170 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdc967fc640 1 -- 172.21.15.94:0/3722780891 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98012870 con 0x7fdca40ab170 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.314+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca40c1fc0 con 0x7fdca40ab170 2024-03-06T07:26:30.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.315+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca404f330 con 0x7fdca40ab170 2024-03-06T07:26:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.315+0000 7fdc967fc640 1 -- 172.21.15.94:0/3722780891 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc98012a10 con 0x7fdca40ab170 2024-03-06T07:26:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.316+0000 7fdc967fc640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.316+0000 7fdc967fc640 1 -- 172.21.15.94:0/3722780891 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdc98089e50 con 0x7fdca40ab170 2024-03-06T07:26:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.316+0000 7fdca9d63640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.316+0000 7fdca9d63640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8806b4d0 0x7fdc8806d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdca0000f60 tx=0x7fdca0007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.320+0000 7fdc967fc640 1 -- 172.21.15.94:0/3722780891 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc98058580 con 0x7fdca40ab170 2024-03-06T07:26:30.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.527+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7fdca40c1740 con 0x7fdca40ab170 2024-03-06T07:26:30.629 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.628+0000 7fdc967fc640 1 -- 172.21.15.94:0/3722780891 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v187) v1 ==== 125+0+0 (secure 0 0 0) 0x7fdc9805c620 con 0x7fdca40ab170 2024-03-06T07:26:30.630 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked down. 2024-03-06T07:26:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8806b4d0 msgr2=0x7fdc8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8806b4d0 0x7fdc8806d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdca0000f60 tx=0x7fdca0007e40 comp rx=0 tx=0).stop 2024-03-06T07:26:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 msgr2=0x7fdca414df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca414df90 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800c480 tx=0x7fdc9800c940 comp rx=0 tx=0).stop 2024-03-06T07:26:30.632 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 shutdown_connections 2024-03-06T07:26:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca40c5990 0x7fdca414e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca40ab170 0x7fdca414df90 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 --2- 172.21.15.94:0/3722780891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca406c900 0x7fdca4154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 >> 172.21.15.94:0/3722780891 conn(0x7fdca40c6710 msgr2=0x7fdca41dbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 shutdown_connections 2024-03-06T07:26:30.633 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.630+0000 7fdcabfc7640 1 -- 172.21.15.94:0/3722780891 wait complete. 2024-03-06T07:26:30.643 INFO:tasks.ceph_test_case:waiting 30s for health warning matching 2024-03-06T07:26:30.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:26:30.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.740+0000 7f68059f6640 1 Processor -- start 2024-03-06T07:26:30.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.740+0000 7f68059f6640 1 -- start start 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f68059f6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6800153990 0x7f6800156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.743 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f68059f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800070aa0 con 0x7f68001506b0 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f68059f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800070d80 con 0x7f68000acc00 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f68059f6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6800156450 con 0x7f6800153990 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f67feffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f68001506b0 0x7f68000702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47804/0 (socket says 172.21.15.94:47804) 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f67feffd640 1 -- 172.21.15.94:0/2229954890 learned_addr learned my addr 172.21.15.94:0/2229954890 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f67fe7fc640 1 --2- 172.21.15.94:0/2229954890 >> v2:172.21.15.145:3300/0 conn(0x7f6800153990 0x7f6800156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.741+0000 7f67fdffb640 1 -- 172.21.15.94:0/2229954890 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267126529 0 0) 0x7f6800070aa0 con 0x7f68001506b0 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.742+0000 7f67fdffb640 1 -- 172.21.15.94:0/2229954890 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e8003520 con 0x7f68001506b0 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.742+0000 7f67fe7fc640 1 -- 172.21.15.94:0/2229954890 >> v1:172.21.15.145:6789/0 conn(0x7f68000acc00 legacy=0x7f68000c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.742+0000 7f67fdffb640 1 -- 172.21.15.94:0/2229954890 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662943652 0 0) 0x7f67e8003520 con 0x7f68001506b0 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.742+0000 7f67fe7fc640 1 -- 172.21.15.94:0/2229954890 >> v1:172.21.15.94:6789/0 conn(0x7f68001506b0 legacy=0x7f68000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.742+0000 7f67fe7fc640 1 -- 172.21.15.94:0/2229954890 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68001566a0 con 0x7f6800153990 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.742+0000 7f67fe7fc640 1 --2- 172.21.15.94:0/2229954890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 0x7f6800156ea0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097a0 tx=0x7f67f402b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ac79c3342aa02e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f67fdffb640 1 -- 172.21.15.94:0/2229954890 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4037070 con 0x7f6800153990 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f67fdffb640 1 -- 172.21.15.94:0/2229954890 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f4030510 con 0x7f6800153990 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f67fdffb640 1 -- 172.21.15.94:0/2229954890 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00019f0 con 0x7f6800153990 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f68059f6640 1 -- 172.21.15.94:0/2229954890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 msgr2=0x7f6800156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f68059f6640 1 --2- 172.21.15.94:0/2229954890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 0x7f6800156ea0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097a0 tx=0x7f67f402b680 comp rx=0 tx=0).stop 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f68059f6640 1 -- 172.21.15.94:0/2229954890 shutdown_connections 2024-03-06T07:26:30.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f68059f6640 1 --2- 172.21.15.94:0/2229954890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 0x7f6800156ea0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f68059f6640 1 -- 172.21.15.94:0/2229954890 >> 172.21.15.94:0/2229954890 conn(0x7f680006b1a0 msgr2=0x7f680006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f68059f6640 1 -- 172.21.15.94:0/2229954890 shutdown_connections 2024-03-06T07:26:30.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.743+0000 7f68059f6640 1 -- 172.21.15.94:0/2229954890 wait complete. 2024-03-06T07:26:30.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.744+0000 7f68059f6640 1 Processor -- start 2024-03-06T07:26:30.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f68059f6640 1 -- start start 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f68059f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68000acc00 0x7f68000a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f68059f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68001506b0 0x7f68000a8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f68059f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 0x7f6800148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f68059f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000ad000 con 0x7f68001506b0 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f68059f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800156a10 con 0x7f6800153990 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f68059f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000ad180 con 0x7f68000acc00 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f67fe7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68001506b0 0x7f68000a8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f67fe7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68001506b0 0x7f68000a8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49204/0 (socket says 172.21.15.94:49204) 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f67fe7fc640 1 -- 172.21.15.94:0/1203375466 learned_addr learned my addr 172.21.15.94:0/1203375466 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f67ff7fe640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 0x7f6800148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.745+0000 7f67feffd640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68000acc00 0x7f68000a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67fe7fc640 1 -- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68000acc00 msgr2=0x7f68000a85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67fe7fc640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68000acc00 0x7f68000a85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67fe7fc640 1 -- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 msgr2=0x7f6800148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67fe7fc640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 0x7f6800148af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67fe7fc640 1 -- 172.21.15.94:0/1203375466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000aaf00 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67fe7fc640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68001506b0 0x7f68000a8b00 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097a0 tx=0x7f67f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67e7fff640 1 -- 172.21.15.94:0/1203375466 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4042070 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67e7fff640 1 -- 172.21.15.94:0/1203375466 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f403d850 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680014c9e0 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f67e7fff640 1 -- 172.21.15.94:0/1203375466 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f4037040 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.746+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68000a9d90 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.747+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680004f330 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.747+0000 7f67e7fff640 1 -- 172.21.15.94:0/1203375466 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67f4045070 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.748+0000 7f67e7fff640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.748+0000 7f67e7fff640 1 -- 172.21.15.94:0/1203375466 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f67f40abd90 con 0x7f68001506b0 2024-03-06T07:26:30.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.748+0000 7f67feffd640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.748+0000 7f67feffd640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67ec06b4d0 0x7f67ec06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f67f0001560 tx=0x7f67f0001d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.752+0000 7f67e7fff640 1 -- 172.21.15.94:0/1203375466 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68000aaf00 con 0x7f68001506b0 2024-03-06T07:26:30.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.957+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f680006b8a0 con 0x7f68001506b0 2024-03-06T07:26:30.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.958+0000 7f67e7fff640 1 -- 172.21.15.94:0/1203375466 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f67f407a4c0 con 0x7f68001506b0 2024-03-06T07:26:30.960 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:30.960 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.959+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67ec06b4d0 msgr2=0x7f67ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.959+0000 7f68059f6640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67ec06b4d0 0x7f67ec06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f67f0001560 tx=0x7f67f0001d40 comp rx=0 tx=0).stop 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.959+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68001506b0 msgr2=0x7f68000a8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.959+0000 7f68059f6640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68001506b0 0x7f68000a8b00 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f67f40097a0 tx=0x7f67f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 shutdown_connections 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67ec06b4d0 0x7f67ec06d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6800153990 0x7f6800148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68001506b0 0x7f68000a8b00 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 --2- 172.21.15.94:0/1203375466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68000acc00 0x7f68000a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 >> 172.21.15.94:0/1203375466 conn(0x7f680006b1a0 msgr2=0x7f68000702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 shutdown_connections 2024-03-06T07:26:30.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:30.960+0000 7f68059f6640 1 -- 172.21.15.94:0/1203375466 wait complete. 2024-03-06T07:26:30.972 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:26:30.972 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:26:35.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:26:36.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.081+0000 7f75689c7640 1 Processor -- start 2024-03-06T07:26:36.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.081+0000 7f75689c7640 1 -- start start 2024-03-06T07:26:36.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.082+0000 7f75689c7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f756414f360 0x7f7564155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.082+0000 7f75689c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c88c0 0x7f75640c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.082+0000 7f75689c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640c73d0 con 0x7f75640c5880 2024-03-06T07:26:36.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.082+0000 7f75689c7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75640c8dd0 con 0x7f756414f360 2024-03-06T07:26:36.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.082+0000 7f75689c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75640be2e0 con 0x7f75640c88c0 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7561d9b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c88c0 0x7f75640c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f756259c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f756414f360 0x7f7564155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7562d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f75640c5880 0x7f7564158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:60174/0 (socket says 172.21.15.94:60174) 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7562d9d640 1 -- 172.21.15.94:0/1094952645 learned_addr learned my addr 172.21.15.94:0/1094952645 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7561d9b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c88c0 0x7f75640c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:59696/0 (socket says 172.21.15.94:59696) 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7561d9b640 1 -- 172.21.15.94:0/1094952645 >> v2:172.21.15.145:3300/0 conn(0x7f756414f360 msgr2=0x7f7564155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7561d9b640 1 --2- 172.21.15.94:0/1094952645 >> v2:172.21.15.145:3300/0 conn(0x7f756414f360 0x7f7564155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7561d9b640 1 -- 172.21.15.94:0/1094952645 >> v1:172.21.15.145:6789/0 conn(0x7f75640c5880 legacy=0x7f7564158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f7561d9b640 1 -- 172.21.15.94:0/1094952645 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640c7670 con 0x7f75640c88c0 2024-03-06T07:26:36.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.083+0000 7f756259c640 1 --2- 172.21.15.94:0/1094952645 >> v2:172.21.15.145:3300/0 conn(0x7f756414f360 0x7f7564155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:36.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.084+0000 7f7561d9b640 1 --2- 172.21.15.94:0/1094952645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c88c0 0x7f75640c5340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f75540097a0 tx=0x7f755402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69db198537662445 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.084+0000 7f756159a640 1 -- 172.21.15.94:0/1094952645 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75540315a0 con 0x7f75640c88c0 2024-03-06T07:26:36.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.084+0000 7f756159a640 1 -- 172.21.15.94:0/1094952645 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7554037070 con 0x7f75640c88c0 2024-03-06T07:26:36.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.084+0000 7f756159a640 1 -- 172.21.15.94:0/1094952645 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554031960 con 0x7f75640c88c0 2024-03-06T07:26:36.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 -- 172.21.15.94:0/1094952645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c88c0 msgr2=0x7f75640c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 --2- 172.21.15.94:0/1094952645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c88c0 0x7f75640c5340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f75540097a0 tx=0x7f755402b840 comp rx=0 tx=0).stop 2024-03-06T07:26:36.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 -- 172.21.15.94:0/1094952645 shutdown_connections 2024-03-06T07:26:36.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 --2- 172.21.15.94:0/1094952645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c88c0 0x7f75640c5340 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 --2- 172.21.15.94:0/1094952645 >> v2:172.21.15.145:3300/0 conn(0x7f756414f360 0x7f7564155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 -- 172.21.15.94:0/1094952645 >> 172.21.15.94:0/1094952645 conn(0x7f75640ac570 msgr2=0x7f75640ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 -- 172.21.15.94:0/1094952645 shutdown_connections 2024-03-06T07:26:36.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.085+0000 7f75689c7640 1 -- 172.21.15.94:0/1094952645 wait complete. 2024-03-06T07:26:36.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.086+0000 7f75689c7640 1 Processor -- start 2024-03-06T07:26:36.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.086+0000 7f75689c7640 1 -- start start 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f75689c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c5880 0x7f75641ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f75689c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75640c88c0 0x7f756414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f75689c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f756414f360 0x7f756414b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f75689c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564159270 con 0x7f756414f360 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f75689c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75640c7c30 con 0x7f75640c88c0 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f75689c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75641593f0 con 0x7f75640c5880 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f756414f360 0x7f756414b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f756414f360 0x7f756414b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:49214/0 (socket says 172.21.15.94:49214) 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 -- 172.21.15.94:0/3344653420 learned_addr learned my addr 172.21.15.94:0/3344653420 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:36.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7561d9b640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75640c88c0 0x7f756414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f756259c640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c5880 0x7f75641ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 -- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c5880 msgr2=0x7f75641ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c5880 0x7f75641ee1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 -- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75640c88c0 msgr2=0x7f756414b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75640c88c0 0x7f756414b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f7562d9d640 1 -- 172.21.15.94:0/3344653420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75641eeb00 con 0x7f756414f360 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.087+0000 7f756259c640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c5880 0x7f75641ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.088+0000 7f7562d9d640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f756414f360 0x7f756414b7e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f755c00cbc0 tx=0x7f755c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.088+0000 7f755affd640 1 -- 172.21.15.94:0/3344653420 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c017070 con 0x7f756414f360 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.088+0000 7f755affd640 1 -- 172.21.15.94:0/3344653420 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c00ad40 con 0x7f756414f360 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.088+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7564151760 con 0x7f756414f360 2024-03-06T07:26:36.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.088+0000 7f755affd640 1 -- 172.21.15.94:0/3344653420 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c012600 con 0x7f756414f360 2024-03-06T07:26:36.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.088+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75640c1230 con 0x7f756414f360 2024-03-06T07:26:36.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.089+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7564069460 con 0x7f756414f360 2024-03-06T07:26:36.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.089+0000 7f755affd640 1 -- 172.21.15.94:0/3344653420 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f755c01a070 con 0x7f756414f360 2024-03-06T07:26:36.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.090+0000 7f755affd640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f754406b4d0 0x7f754406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.090+0000 7f755affd640 1 -- 172.21.15.94:0/3344653420 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f755c089f70 con 0x7f756414f360 2024-03-06T07:26:36.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.090+0000 7f756259c640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f754406b4d0 0x7f754406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.091+0000 7f756259c640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f754406b4d0 0x7f754406d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f754c009af0 tx=0x7f754c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.095+0000 7f755affd640 1 -- 172.21.15.94:0/3344653420 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7564069460 con 0x7f756414f360 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.302+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f756404f330 con 0x7f756414f360 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.302+0000 7f755affd640 1 -- 172.21.15.94:0/3344653420 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f755c0586a0 con 0x7f756414f360 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f754406b4d0 msgr2=0x7f754406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f754406b4d0 0x7f754406d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f754c009af0 tx=0x7f754c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f756414f360 msgr2=0x7f756414b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f756414f360 0x7f756414b7e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f755c00cbc0 tx=0x7f755c00a430 comp rx=0 tx=0).stop 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 shutdown_connections 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f754406b4d0 0x7f754406d990 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f756414f360 0x7f756414b7e0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75640c88c0 0x7f756414b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 --2- 172.21.15.94:0/3344653420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75640c5880 0x7f75641ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 >> 172.21.15.94:0/3344653420 conn(0x7f75640ac570 msgr2=0x7f75640aaac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 shutdown_connections 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:36.304+0000 7f75689c7640 1 -- 172.21.15.94:0/3344653420 wait complete. 2024-03-06T07:26:36.316 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:26:36.317 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:26:41.317 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:26:41.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.426+0000 7f258e0ed640 1 Processor -- start 2024-03-06T07:26:41.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f258e0ed640 1 -- start start 2024-03-06T07:26:41.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f258e0ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25880a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f258e0ed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25880c3280 0x7f25880c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f258e0ed640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2588158800 con 0x7f25881551e0 2024-03-06T07:26:41.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f258e0ed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2588158ae0 con 0x7f25880c3280 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f258e0ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2588158dc0 con 0x7f258806d9f0 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f2586ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25880c3280 0x7f25880c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f2586ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f25880c3280 0x7f25880c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46998/0 (socket says 172.21.15.94:46998) 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.427+0000 7f2586ffd640 1 -- 172.21.15.94:0/1030710201 learned_addr learned my addr 172.21.15.94:0/1030710201 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.428+0000 7f25877fe640 1 --2- 172.21.15.94:0/1030710201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25880a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.428+0000 7f2586ffd640 1 -- 172.21.15.94:0/1030710201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 msgr2=0x7f25880a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.428+0000 7f2586ffd640 1 --2- 172.21.15.94:0/1030710201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25880a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.428+0000 7f2586ffd640 1 -- 172.21.15.94:0/1030710201 >> v1:172.21.15.145:6789/0 conn(0x7f25881551e0 legacy=0x7f2588155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.428+0000 7f2586ffd640 1 -- 172.21.15.94:0/1030710201 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25881590a0 con 0x7f25880c3280 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.428+0000 7f25877fe640 1 --2- 172.21.15.94:0/1030710201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25880a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.428+0000 7f2586ffd640 1 --2- 172.21.15.94:0/1030710201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25880c3280 0x7f25880c36e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f2578009870 tx=0x7f257802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2170cc1e630f83a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f25867fc640 1 -- 172.21.15.94:0/1030710201 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25780317c0 con 0x7f25880c3280 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f25867fc640 1 -- 172.21.15.94:0/1030710201 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578037070 con 0x7f25880c3280 2024-03-06T07:26:41.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f25867fc640 1 -- 172.21.15.94:0/1030710201 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578031b80 con 0x7f25880c3280 2024-03-06T07:26:41.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 -- 172.21.15.94:0/1030710201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25880c3280 msgr2=0x7f25880c36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 --2- 172.21.15.94:0/1030710201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25880c3280 0x7f25880c36e0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f2578009870 tx=0x7f257802b860 comp rx=0 tx=0).stop 2024-03-06T07:26:41.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 -- 172.21.15.94:0/1030710201 shutdown_connections 2024-03-06T07:26:41.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 --2- 172.21.15.94:0/1030710201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25880c3280 0x7f25880c36e0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 --2- 172.21.15.94:0/1030710201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25880a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.434 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 -- 172.21.15.94:0/1030710201 >> 172.21.15.94:0/1030710201 conn(0x7f25880aa630 msgr2=0x7f25880aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 -- 172.21.15.94:0/1030710201 shutdown_connections 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.429+0000 7f258e0ed640 1 -- 172.21.15.94:0/1030710201 wait complete. 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.430+0000 7f258e0ed640 1 Processor -- start 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f258e0ed640 1 -- start start 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f258e0ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25881eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f258e0ed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25880c3280 0x7f25880c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f258e0ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25881551e0 0x7f2588156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f258e0ed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258806df50 con 0x7f25881551e0 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f258e0ed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258806ddd0 con 0x7f25880c3280 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f258e0ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25880a8210 con 0x7f258806d9f0 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f2587fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25881551e0 0x7f2588156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f2587fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25881551e0 0x7f2588156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47012/0 (socket says 172.21.15.94:47012) 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.431+0000 7f2587fff640 1 -- 172.21.15.94:0/3732940438 learned_addr learned my addr 172.21.15.94:0/3732940438 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:41.435 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f25877fe640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25881eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f2586ffd640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25880c3280 0x7f25880c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f2587fff640 1 -- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 msgr2=0x7f25881eb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f2587fff640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25881eb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f2587fff640 1 -- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25880c3280 msgr2=0x7f25880c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f2587fff640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25880c3280 0x7f25880c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f2587fff640 1 -- 172.21.15.94:0/3732940438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880ac890 con 0x7f25881551e0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f2587fff640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25881551e0 0x7f2588156b20 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f258000c780 tx=0x7f258000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f256ffff640 1 -- 172.21.15.94:0/3732940438 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580004b10 con 0x7f25881551e0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f256ffff640 1 -- 172.21.15.94:0/3732940438 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580004cb0 con 0x7f25881551e0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f256ffff640 1 -- 172.21.15.94:0/3732940438 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258000aa90 con 0x7f25881551e0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2588156360 con 0x7f25881551e0 2024-03-06T07:26:41.436 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.432+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2588157a30 con 0x7f25881551e0 2024-03-06T07:26:41.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.433+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25880c6470 con 0x7f25881551e0 2024-03-06T07:26:41.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.434+0000 7f256ffff640 1 -- 172.21.15.94:0/3732940438 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f258000ac30 con 0x7f25881551e0 2024-03-06T07:26:41.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.434+0000 7f256ffff640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f257406b4d0 0x7f257406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.434+0000 7f256ffff640 1 -- 172.21.15.94:0/3732940438 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f258008a940 con 0x7f25881551e0 2024-03-06T07:26:41.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.434+0000 7f25877fe640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f257406b4d0 0x7f257406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.435+0000 7f25877fe640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f257406b4d0 0x7f257406d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f257c00a950 tx=0x7f257c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.438+0000 7f256ffff640 1 -- 172.21.15.94:0/3732940438 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25880c6470 con 0x7f25881551e0 2024-03-06T07:26:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.648+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2588067910 con 0x7f25881551e0 2024-03-06T07:26:41.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.648+0000 7f256ffff640 1 -- 172.21.15.94:0/3732940438 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2588067910 con 0x7f25881551e0 2024-03-06T07:26:41.651 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:41.651 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f257406b4d0 msgr2=0x7f257406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f257406b4d0 0x7f257406d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f257c00a950 tx=0x7f257c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25881551e0 msgr2=0x7f2588156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25881551e0 0x7f2588156b20 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f258000c780 tx=0x7f258000cc40 comp rx=0 tx=0).stop 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 shutdown_connections 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f257406b4d0 0x7f257406d990 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f25881551e0 0x7f2588156b20 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25880c3280 0x7f25880c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 --2- 172.21.15.94:0/3732940438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f258806d9f0 0x7f25881eb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.650+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 >> 172.21.15.94:0/3732940438 conn(0x7f25880aa630 msgr2=0x7f2588153c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.651+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 shutdown_connections 2024-03-06T07:26:41.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:41.651+0000 7f258e0ed640 1 -- 172.21.15.94:0/3732940438 wait complete. 2024-03-06T07:26:41.663 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:26:41.663 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:26:45.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:45.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:46.299 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:26:46.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:26:46.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.774+0000 7fb34ee88640 1 Processor -- start 2024-03-06T07:26:46.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.774+0000 7fb34ee88640 1 -- start start 2024-03-06T07:26:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.774+0000 7fb34ee88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb348158db0 0x7fb3480c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.774+0000 7fb34ee88640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb3480c88b0 0x7fb34815fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.774+0000 7fb34ee88640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb348155110 con 0x7fb3480c5330 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.774+0000 7fb34ee88640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3481559e0 con 0x7fb3480c88b0 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.774+0000 7fb34ee88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480c8f40 con 0x7fb348158db0 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34d425640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb3480c88b0 0x7fb34815fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34d425640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb3480c88b0 0x7fb34815fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47016/0 (socket says 172.21.15.94:47016) 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34d425640 1 -- 172.21.15.94:0/1970697249 learned_addr learned my addr 172.21.15.94:0/1970697249 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34cc24640 1 --1- 172.21.15.94:0/1970697249 >> v1:172.21.15.94:6789/0 conn(0x7fb3480c5330 0x7fb3481586a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36496/0 (socket says 172.21.15.94:36496) 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34d425640 1 -- 172.21.15.94:0/1970697249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb348158db0 msgr2=0x7fb3480c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb347fff640 1 --2- 172.21.15.94:0/1970697249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb348158db0 0x7fb3480c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34d425640 1 --2- 172.21.15.94:0/1970697249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb348158db0 0x7fb3480c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb3477fe640 1 -- 172.21.15.94:0/1970697249 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087940898 0 0) 0x7fb340001490 con 0x7fb3480c5330 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34d425640 1 -- 172.21.15.94:0/1970697249 >> v1:172.21.15.94:6789/0 conn(0x7fb3480c5330 legacy=0x7fb3481586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.775+0000 7fb34d425640 1 -- 172.21.15.94:0/1970697249 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb348155390 con 0x7fb3480c88b0 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34d425640 1 --2- 172.21.15.94:0/1970697249 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3480c88b0 0x7fb34815fa30 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fb338009870 tx=0x7fb33802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27c612fbdb26c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb3477fe640 1 -- 172.21.15.94:0/1970697249 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338035020 con 0x7fb3480c88b0 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb3477fe640 1 -- 172.21.15.94:0/1970697249 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338031640 con 0x7fb3480c88b0 2024-03-06T07:26:46.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb3477fe640 1 -- 172.21.15.94:0/1970697249 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33802d040 con 0x7fb3480c88b0 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 -- 172.21.15.94:0/1970697249 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3480c88b0 msgr2=0x7fb34815fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 --2- 172.21.15.94:0/1970697249 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3480c88b0 0x7fb34815fa30 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fb338009870 tx=0x7fb33802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 -- 172.21.15.94:0/1970697249 shutdown_connections 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 --2- 172.21.15.94:0/1970697249 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3480c88b0 0x7fb34815fa30 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 --2- 172.21.15.94:0/1970697249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb348158db0 0x7fb3480c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 -- 172.21.15.94:0/1970697249 >> 172.21.15.94:0/1970697249 conn(0x7fb3480ac560 msgr2=0x7fb3480ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 -- 172.21.15.94:0/1970697249 shutdown_connections 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.776+0000 7fb34ee88640 1 -- 172.21.15.94:0/1970697249 wait complete. 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.777+0000 7fb34ee88640 1 Processor -- start 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34ee88640 1 -- start start 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34ee88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3480c5330 0x7fb3481ee770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34ee88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3480c88b0 0x7fb34814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34ee88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb348156480 0x7fb348150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34ee88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3481529d0 con 0x7fb348156480 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34ee88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34806d090 con 0x7fb3480c88b0 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34ee88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb348152b50 con 0x7fb3480c5330 2024-03-06T07:26:46.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34d425640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb348156480 0x7fb348150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34d425640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb348156480 0x7fb348150b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47018/0 (socket says 172.21.15.94:47018) 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.778+0000 7fb34d425640 1 -- 172.21.15.94:0/2723653477 learned_addr learned my addr 172.21.15.94:0/2723653477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34cc24640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3480c5330 0x7fb3481ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb347fff640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3480c88b0 0x7fb34814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34d425640 1 -- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3480c5330 msgr2=0x7fb3481ee770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34d425640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3480c5330 0x7fb3481ee770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34d425640 1 -- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3480c88b0 msgr2=0x7fb34814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34d425640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3480c88b0 0x7fb34814b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34d425640 1 -- 172.21.15.94:0/2723653477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb348156980 con 0x7fb348156480 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34d425640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb348156480 0x7fb348150b50 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fb338009870 tx=0x7fb3380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb3457fa640 1 -- 172.21.15.94:0/2723653477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338040070 con 0x7fb348156480 2024-03-06T07:26:46.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb3457fa640 1 -- 172.21.15.94:0/2723653477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338039c10 con 0x7fb348156480 2024-03-06T07:26:46.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb348156c60 con 0x7fb348156480 2024-03-06T07:26:46.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb3457fa640 1 -- 172.21.15.94:0/2723653477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338035040 con 0x7fb348156480 2024-03-06T07:26:46.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.779+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3481edf70 con 0x7fb348156480 2024-03-06T07:26:46.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.780+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34804f330 con 0x7fb348156480 2024-03-06T07:26:46.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.781+0000 7fb3457fa640 1 -- 172.21.15.94:0/2723653477 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb338038020 con 0x7fb348156480 2024-03-06T07:26:46.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.781+0000 7fb3457fa640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb32806b4d0 0x7fb32806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.781+0000 7fb3457fa640 1 -- 172.21.15.94:0/2723653477 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb3380abe10 con 0x7fb348156480 2024-03-06T07:26:46.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.781+0000 7fb34cc24640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb32806b4d0 0x7fb32806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.782+0000 7fb34cc24640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb32806b4d0 0x7fb32806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb3400012e0 tx=0x7fb340009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.786+0000 7fb3457fa640 1 -- 172.21.15.94:0/2723653477 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb33807ab90 con 0x7fb348156480 2024-03-06T07:26:46.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.996+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb3481ee280 con 0x7fb348156480 2024-03-06T07:26:46.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.996+0000 7fb3457fa640 1 -- 172.21.15.94:0/2723653477 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb33807a490 con 0x7fb348156480 2024-03-06T07:26:46.998 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:46.998 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:26:47.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb32806b4d0 msgr2=0x7fb32806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb32806b4d0 0x7fb32806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb3400012e0 tx=0x7fb340009040 comp rx=0 tx=0).stop 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb348156480 msgr2=0x7fb348150b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb348156480 0x7fb348150b50 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fb338009870 tx=0x7fb3380046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 shutdown_connections 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb32806b4d0 0x7fb32806d990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb348156480 0x7fb348150b50 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3480c88b0 0x7fb34814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 --2- 172.21.15.94:0/2723653477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3480c5330 0x7fb3481ee770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 >> 172.21.15.94:0/2723653477 conn(0x7fb3480ac560 msgr2=0x7fb3480ca940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 shutdown_connections 2024-03-06T07:26:47.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:46.998+0000 7fb34ee88640 1 -- 172.21.15.94:0/2723653477 wait complete. 2024-03-06T07:26:47.011 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:26:47.011 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:26:52.012 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:26:52.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.121+0000 7fdbc7d3e640 1 Processor -- start 2024-03-06T07:26:52.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.121+0000 7fdbc7d3e640 1 -- start start 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc7d3e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdbc01550c0 0x7fdbc01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc7d3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc00aa180 0x7fdbc00c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc7d3e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc00aa660 con 0x7fdbc01559e0 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc7d3e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc00c4610 con 0x7fdbc01550c0 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc7d3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00aa940 con 0x7fdbc00aa180 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc5ada640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdbc01550c0 0x7fdbc01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc5ada640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdbc01550c0 0x7fdbc01554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46692/0 (socket says 172.21.15.94:46692) 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc5ada640 1 -- 172.21.15.94:0/233064825 learned_addr learned my addr 172.21.15.94:0/233064825 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc62db640 1 --2- 172.21.15.94:0/233064825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc00aa180 0x7fdbc00c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc5ada640 1 -- 172.21.15.94:0/233064825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc00aa180 msgr2=0x7fdbc00c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc5ada640 1 --2- 172.21.15.94:0/233064825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc00aa180 0x7fdbc00c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc5ada640 1 -- 172.21.15.94:0/233064825 >> v1:172.21.15.145:6789/0 conn(0x7fdbc01559e0 legacy=0x7fdbc014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.122+0000 7fdbc5ada640 1 -- 172.21.15.94:0/233064825 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00c4860 con 0x7fdbc01550c0 2024-03-06T07:26:52.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.123+0000 7fdbc5ada640 1 --2- 172.21.15.94:0/233064825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc01554a0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00c3fc0 tx=0x7fdbbc02df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e82f3e114dcbc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.123+0000 7fdbc4ad8640 1 -- 172.21.15.94:0/233064825 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc03b070 con 0x7fdbc01550c0 2024-03-06T07:26:52.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.123+0000 7fdbc4ad8640 1 -- 172.21.15.94:0/233064825 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc02ed30 con 0x7fdbc01550c0 2024-03-06T07:26:52.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.123+0000 7fdbc4ad8640 1 -- 172.21.15.94:0/233064825 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc036660 con 0x7fdbc01550c0 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/233064825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 msgr2=0x7fdbc01554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/233064825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc01554a0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00c3fc0 tx=0x7fdbbc02df80 comp rx=0 tx=0).stop 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/233064825 shutdown_connections 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/233064825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc00aa180 0x7fdbc00c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/233064825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc01554a0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/233064825 >> 172.21.15.94:0/233064825 conn(0x7fdbc00c6710 msgr2=0x7fdbc00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/233064825 shutdown_connections 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.124+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/233064825 wait complete. 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.125+0000 7fdbc7d3e640 1 Processor -- start 2024-03-06T07:26:52.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.125+0000 7fdbc7d3e640 1 -- start start 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbc00aa180 0x7fdbc00bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc00a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc01559e0 0x7fdbc01ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc014f310 con 0x7fdbc01550c0 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00c1560 con 0x7fdbc00aa180 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc014f490 con 0x7fdbc01559e0 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc00a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc00a7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46706/0 (socket says 172.21.15.94:46706) 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 -- 172.21.15.94:0/1353775451 learned_addr learned my addr 172.21.15.94:0/1353775451 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc5ada640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbc00aa180 0x7fdbc00bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc62db640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc01559e0 0x7fdbc01ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 -- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc01559e0 msgr2=0x7fdbc01ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc01559e0 0x7fdbc01ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 -- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbc00aa180 msgr2=0x7fdbc00bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbc00aa180 0x7fdbc00bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 -- 172.21.15.94:0/1353775451 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc006c6b0 con 0x7fdbc01550c0 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc52d9640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc00a7a40 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000c460 tx=0x7fdbb000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbb67fc640 1 -- 172.21.15.94:0/1353775451 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0016020 con 0x7fdbc01550c0 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbb67fc640 1 -- 172.21.15.94:0/1353775451 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb0012870 con 0x7fdbc01550c0 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbb67fc640 1 -- 172.21.15.94:0/1353775451 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0012c60 con 0x7fdbc01550c0 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc0072a90 con 0x7fdbc01550c0 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.126+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc00c02f0 con 0x7fdbc01550c0 2024-03-06T07:26:52.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.127+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc00a6bf0 con 0x7fdbc01550c0 2024-03-06T07:26:52.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.128+0000 7fdbb67fc640 1 -- 172.21.15.94:0/1353775451 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbb0019020 con 0x7fdbc01550c0 2024-03-06T07:26:52.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.128+0000 7fdbb67fc640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdba406b500 0x7fdba406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.128+0000 7fdbb67fc640 1 -- 172.21.15.94:0/1353775451 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbb008a040 con 0x7fdbc01550c0 2024-03-06T07:26:52.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.128+0000 7fdbc5ada640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdba406b500 0x7fdba406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.129+0000 7fdbc5ada640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdba406b500 0x7fdba406d9c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc038000 tx=0x7fdbbc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.134 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.133+0000 7fdbb67fc640 1 -- 172.21.15.94:0/1353775451 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbb0020030 con 0x7fdbc01550c0 2024-03-06T07:26:52.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.340+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdbc00a5ef0 con 0x7fdbc01550c0 2024-03-06T07:26:52.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.340+0000 7fdbb67fc640 1 -- 172.21.15.94:0/1353775451 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fdbb00182c0 con 0x7fdbc01550c0 2024-03-06T07:26:52.342 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:52.342 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:26:52.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdba406b500 msgr2=0x7fdba406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdba406b500 0x7fdba406d9c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc038000 tx=0x7fdbbc008040 comp rx=0 tx=0).stop 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 msgr2=0x7fdbc00a7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc00a7a40 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000c460 tx=0x7fdbb000c920 comp rx=0 tx=0).stop 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 shutdown_connections 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdba406b500 0x7fdba406d9c0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbc01559e0 0x7fdbc01ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdbc01550c0 0x7fdbc00a7a40 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 --2- 172.21.15.94:0/1353775451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbc00aa180 0x7fdbc00bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 >> 172.21.15.94:0/1353775451 conn(0x7fdbc00c6710 msgr2=0x7fdbc00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 shutdown_connections 2024-03-06T07:26:52.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:52.342+0000 7fdbc7d3e640 1 -- 172.21.15.94:0/1353775451 wait complete. 2024-03-06T07:26:52.355 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:26:52.355 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:26:57.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:26:57.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.465+0000 7fec6b69d640 1 Processor -- start 2024-03-06T07:26:57.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.465+0000 7fec6b69d640 1 -- start start 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec6b69d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 0x7fec640c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec6b69d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec64155bb0 con 0x7fec640aa180 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec6b69d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640aa660 con 0x7fec641550c0 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec6b69d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c4610 con 0x7fec6414f310 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec68c38640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fec640aa180 0x7fec6414ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42088/0 (socket says 172.21.15.94:42088) 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec68c38640 1 -- 172.21.15.94:0/147739791 learned_addr learned my addr 172.21.15.94:0/147739791 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec69c3a640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 0x7fec640c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466088595 0 0) 0x7fec64155bb0 con 0x7fec640aa180 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.466+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50003520 con 0x7fec640aa180 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024813251 0 0) 0x7fec640aa660 con 0x7fec641550c0 2024-03-06T07:26:57.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec64155bb0 con 0x7fec641550c0 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3394264459 0 0) 0x7fec50003520 con 0x7fec640aa180 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 msgr2=0x7fec640c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 0x7fec640c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 >> v1:172.21.15.145:6789/0 conn(0x7fec641550c0 legacy=0x7fec641554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640aa940 con 0x7fec640aa180 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fec54001670 con 0x7fec640aa180 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec69c3a640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 0x7fec640c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 >> v1:172.21.15.94:6789/0 conn(0x7fec640aa180 legacy=0x7fec6414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec50003b70 0x7fec50003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 0x7fec50005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c4610 con 0x7fec50004800 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5000bb50 con 0x7fec50003b70 2024-03-06T07:26:57.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5000bd20 con 0x7fec6414f310 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec68c38640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec69439640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec50003b70 0x7fec50003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec69c3a640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 0x7fec50005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec68c38640 1 -- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 msgr2=0x7fec50005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec68c38640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec6414f310 0x7fec50005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec68c38640 1 -- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec50003b70 msgr2=0x7fec50003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec68c38640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec50003b70 0x7fec50003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.467+0000 7fec68c38640 1 -- 172.21.15.94:0/147739791 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50003520 con 0x7fec50004800 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.468+0000 7fec68c38640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec50004c80 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fec54007fd0 tx=0x7fec50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae2d8707a56de7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.468+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54007ca0 con 0x7fec50004800 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.468+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec54030070 con 0x7fec50004800 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.468+0000 7fec5bfff640 1 -- 172.21.15.94:0/147739791 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54005d10 con 0x7fec50004800 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.469+0000 7fec6b69d640 1 -- 172.21.15.94:0/147739791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 msgr2=0x7fec50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.469+0000 7fec6b69d640 1 --2- 172.21.15.94:0/147739791 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec50004c80 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fec54007fd0 tx=0x7fec50005830 comp rx=0 tx=0).stop 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.469+0000 7fec69c3a640 1 -- 172.21.15.94:0/147739791 reap_dead start 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.469+0000 7fec6b69d640 1 -- 172.21.15.94:0/147739791 shutdown_connections 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.469+0000 7fec6b69d640 1 -- 172.21.15.94:0/147739791 >> 172.21.15.94:0/147739791 conn(0x7fec640c6710 msgr2=0x7fec640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.469+0000 7fec6b69d640 1 -- 172.21.15.94:0/147739791 shutdown_connections 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.469+0000 7fec6b69d640 1 -- 172.21.15.94:0/147739791 wait complete. 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.470+0000 7fec6b69d640 1 Processor -- start 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.470+0000 7fec6b69d640 1 -- start start 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec6b69d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec640a7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec6b69d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6406e570 0x7fec640a7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec6b69d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec640a7ca0 0x7fec6406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec6b69d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c1490 con 0x7fec50004800 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec6b69d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c1310 con 0x7fec6406e570 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec6b69d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c31b0 con 0x7fec640a7ca0 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec640a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec640a7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46736/0 (socket says 172.21.15.94:46736) 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 -- 172.21.15.94:0/4012296669 learned_addr learned my addr 172.21.15.94:0/4012296669 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 -- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec640a7ca0 msgr2=0x7fec6406c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec640a7ca0 0x7fec6406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec68c38640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6406e570 0x7fec640a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 -- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6406e570 msgr2=0x7fec640a7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6406e570 0x7fec640a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 -- 172.21.15.94:0/4012296669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640c2ae0 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec69439640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec640a7220 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fec60011980 tx=0x7fec60011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec59ffb640 1 -- 172.21.15.94:0/4012296669 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6000f860 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec59ffb640 1 -- 172.21.15.94:0/4012296669 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec60016030 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6414cd60 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.471+0000 7fec59ffb640 1 -- 172.21.15.94:0/4012296669 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6000fc20 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.472+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6413a980 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.473+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6404f330 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.473+0000 7fec59ffb640 1 -- 172.21.15.94:0/4012296669 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec600035f0 con 0x7fec50004800 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.473+0000 7fec59ffb640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec4806b500 0x7fec4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.473+0000 7fec59ffb640 1 -- 172.21.15.94:0/4012296669 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec6008c250 con 0x7fec50004800 2024-03-06T07:26:57.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.473+0000 7fec68c38640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec4806b500 0x7fec4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.474+0000 7fec68c38640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec4806b500 0x7fec4806d9c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fec54034000 tx=0x7fec54039040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.478+0000 7fec59ffb640 1 -- 172.21.15.94:0/4012296669 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec6005afd0 con 0x7fec50004800 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.688+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fec64068450 con 0x7fec50004800 2024-03-06T07:26:57.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.689+0000 7fec59ffb640 1 -- 172.21.15.94:0/4012296669 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fec6005a8d0 con 0x7fec50004800 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:26:57.691 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec4806b500 msgr2=0x7fec4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec4806b500 0x7fec4806d9c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fec54034000 tx=0x7fec54039040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 msgr2=0x7fec640a7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec640a7220 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fec60011980 tx=0x7fec60011e40 comp rx=0 tx=0).stop 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 shutdown_connections 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec4806b500 0x7fec4806d9c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec640a7ca0 0x7fec6406c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec6406e570 0x7fec640a7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 --2- 172.21.15.94:0/4012296669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec50004800 0x7fec640a7220 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 >> 172.21.15.94:0/4012296669 conn(0x7fec640c6710 msgr2=0x7fec6414e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 shutdown_connections 2024-03-06T07:26:57.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:26:57.691+0000 7fec6b69d640 1 -- 172.21.15.94:0/4012296669 wait complete. 2024-03-06T07:26:57.703 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:26:57.703 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-03-06T07:27:02.705 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:27:02.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.814+0000 7fbe07671640 1 Processor -- start 2024-03-06T07:27:02.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.815+0000 7fbe07671640 1 -- start start 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.815+0000 7fbe07671640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe00155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.815+0000 7fbe07671640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000c73d0 con 0x7fbe0014f360 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.815+0000 7fbe07671640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000be2e0 con 0x7fbe00158b60 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.815+0000 7fbe07671640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00159700 con 0x7fbe000c88c0 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbe0540d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbe0014f360 0x7fbe000c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52390/0 (socket says 172.21.15.94:52390) 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbe0540d640 1 -- 172.21.15.94:0/3219003529 learned_addr learned my addr 172.21.15.94:0/3219003529 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbe04c0c640 1 --2- 172.21.15.94:0/3219003529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe00155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849675351 0 0) 0x7fbe000c73d0 con 0x7fbe0014f360 2024-03-06T07:27:02.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec003520 con 0x7fbe0014f360 2024-03-06T07:27:02.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398243420 0 0) 0x7fbe000be2e0 con 0x7fbe00158b60 2024-03-06T07:27:02.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe000c73d0 con 0x7fbe00158b60 2024-03-06T07:27:02.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbe04c0c640 1 -- 172.21.15.94:0/3219003529 >> v1:172.21.15.145:6789/0 conn(0x7fbe00158b60 legacy=0x7fbe00158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbe04c0c640 1 -- 172.21.15.94:0/3219003529 >> v1:172.21.15.94:6789/0 conn(0x7fbe0014f360 legacy=0x7fbe000c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2565406008 0 0) 0x7fbdec003520 con 0x7fbe0014f360 2024-03-06T07:27:02.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.816+0000 7fbe04c0c640 1 -- 172.21.15.94:0/3219003529 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe000c7670 con 0x7fbe000c88c0 2024-03-06T07:27:02.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.817+0000 7fbe04c0c640 1 --2- 172.21.15.94:0/3219003529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe00155c10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0097d0 tx=0x7fbdfc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a474d97f586da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.817+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc035020 con 0x7fbe000c88c0 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.817+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdfc031800 con 0x7fbe000c88c0 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.817+0000 7fbdf7fff640 1 -- 172.21.15.94:0/3219003529 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc031bb0 con 0x7fbe000c88c0 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.818+0000 7fbe07671640 1 -- 172.21.15.94:0/3219003529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 msgr2=0x7fbe00155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.818+0000 7fbe07671640 1 --2- 172.21.15.94:0/3219003529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe00155c10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0097d0 tx=0x7fbdfc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.818+0000 7fbe07671640 1 -- 172.21.15.94:0/3219003529 shutdown_connections 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.818+0000 7fbe07671640 1 --2- 172.21.15.94:0/3219003529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe00155c10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.818+0000 7fbe07671640 1 -- 172.21.15.94:0/3219003529 >> 172.21.15.94:0/3219003529 conn(0x7fbe000ac570 msgr2=0x7fbe000ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.818+0000 7fbe07671640 1 -- 172.21.15.94:0/3219003529 shutdown_connections 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.818+0000 7fbe07671640 1 -- 172.21.15.94:0/3219003529 wait complete. 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.819+0000 7fbe07671640 1 Processor -- start 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.819+0000 7fbe07671640 1 -- start start 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.819+0000 7fbe07671640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe0014a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe07671640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe0014f360 0x7fbe001481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe07671640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00158b60 0x7fbe00148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe07671640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000c7c30 con 0x7fbe00158b60 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe07671640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000c5a50 con 0x7fbe0014f360 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe07671640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000c5bd0 con 0x7fbe000c88c0 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00158b60 0x7fbe00148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00158b60 0x7fbe00148730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47852/0 (socket says 172.21.15.94:47852) 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 -- 172.21.15.94:0/2228403368 learned_addr learned my addr 172.21.15.94:0/2228403368 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe04c0c640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe0014f360 0x7fbe001481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe0540d640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe0014a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 -- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 msgr2=0x7fbe0014a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe0014a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 -- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe0014f360 msgr2=0x7fbe001481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe0014f360 0x7fbe001481f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 -- 172.21.15.94:0/2228403368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0014e0d0 con 0x7fbe00158b60 2024-03-06T07:27:02.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe05c0e640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00158b60 0x7fbe00148730 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800c7c0 tx=0x7fbdf800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbdf5ffb640 1 -- 172.21.15.94:0/2228403368 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8003b70 con 0x7fbe00158b60 2024-03-06T07:27:02.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbdf5ffb640 1 -- 172.21.15.94:0/2228403368 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf8003d10 con 0x7fbe00158b60 2024-03-06T07:27:02.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbdf5ffb640 1 -- 172.21.15.94:0/2228403368 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf800a9f0 con 0x7fbe00158b60 2024-03-06T07:27:02.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.820+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe001eef00 con 0x7fbe00158b60 2024-03-06T07:27:02.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.821+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0014ddf0 con 0x7fbe00158b60 2024-03-06T07:27:02.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.822+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe00069460 con 0x7fbe00158b60 2024-03-06T07:27:02.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.822+0000 7fbdf5ffb640 1 -- 172.21.15.94:0/2228403368 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbdf800ab90 con 0x7fbe00158b60 2024-03-06T07:27:02.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.822+0000 7fbdf5ffb640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbde406b4d0 0x7fbde406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.823+0000 7fbdf5ffb640 1 -- 172.21.15.94:0/2228403368 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbdf808a2d0 con 0x7fbe00158b60 2024-03-06T07:27:02.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.823+0000 7fbe0540d640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbde406b4d0 0x7fbde406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.823+0000 7fbe0540d640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbde406b4d0 0x7fbde406d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00011e0 tx=0x7fbdf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:02.827+0000 7fbdf5ffb640 1 -- 172.21.15.94:0/2228403368 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe00069460 con 0x7fbe00158b60 2024-03-06T07:27:03.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.032+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe0004f330 con 0x7fbe00158b60 2024-03-06T07:27:03.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.032+0000 7fbdf5ffb640 1 -- 172.21.15.94:0/2228403368 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbdf8059020 con 0x7fbe00158b60 2024-03-06T07:27:03.036 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:03.037 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:27:03.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbde406b4d0 msgr2=0x7fbde406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbde406b4d0 0x7fbde406d990 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00011e0 tx=0x7fbdf0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00158b60 msgr2=0x7fbe00148730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00158b60 0x7fbe00148730 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800c7c0 tx=0x7fbdf800cc80 comp rx=0 tx=0).stop 2024-03-06T07:27:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 shutdown_connections 2024-03-06T07:27:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbde406b4d0 0x7fbde406d990 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe00158b60 0x7fbe00148730 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe0014f360 0x7fbe001481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 --2- 172.21.15.94:0/2228403368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe000c88c0 0x7fbe0014a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 >> 172.21.15.94:0/2228403368 conn(0x7fbe000ac570 msgr2=0x7fbe000ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 shutdown_connections 2024-03-06T07:27:03.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.034+0000 7fbe07671640 1 -- 172.21.15.94:0/2228403368 wait complete. 2024-03-06T07:27:03.047 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-03-06T07:27:03.047 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:03.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:03.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:03.156 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:03.156 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:27:03.156 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:03.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:27:03.173 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:27:03.173 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:03.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:27:03.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:03.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.312+0000 7f878c13e640 1 Processor -- start 2024-03-06T07:27:03.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.312+0000 7f878c13e640 1 -- start start 2024-03-06T07:27:03.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.312+0000 7f878c13e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f878414f360 0x7f8784155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878c13e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c88c0 0x7f87840c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878c13e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87840c5880 0x7f8784158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878c13e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87840c8dd0 con 0x7f87840c5880 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878c13e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87840c73d0 con 0x7f878414f360 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878c13e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87840be2e0 con 0x7f87840c88c0 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87840c5880 0x7f8784158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f87840c5880 0x7f8784158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47862/0 (socket says 172.21.15.94:47862) 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 -- 172.21.15.94:0/3024540140 learned_addr learned my addr 172.21.15.94:0/3024540140 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f8789eda640 1 --2- 172.21.15.94:0/3024540140 >> v2:172.21.15.145:3300/0 conn(0x7f878414f360 0x7f8784155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f87896d9640 1 --2- 172.21.15.94:0/3024540140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c88c0 0x7f87840c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 -- 172.21.15.94:0/3024540140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c88c0 msgr2=0x7f87840c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 --2- 172.21.15.94:0/3024540140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c88c0 0x7f87840c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 -- 172.21.15.94:0/3024540140 >> v2:172.21.15.145:3300/0 conn(0x7f878414f360 msgr2=0x7f8784155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 --2- 172.21.15.94:0/3024540140 >> v2:172.21.15.145:3300/0 conn(0x7f878414f360 0x7f8784155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f878a6db640 1 -- 172.21.15.94:0/3024540140 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87840c7670 con 0x7f87840c5880 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.313+0000 7f87896d9640 1 --2- 172.21.15.94:0/3024540140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c88c0 0x7f87840c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:03.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.314+0000 7f878a6db640 1 --2- 172.21.15.94:0/3024540140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c5880 0x7f8784158b60 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f878408c840 tx=0x7f877c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a7cd846eaceb02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.314+0000 7f8788ed8640 1 -- 172.21.15.94:0/3024540140 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c02c9e0 con 0x7f87840c5880 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.314+0000 7f8788ed8640 1 -- 172.21.15.94:0/3024540140 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c02cb80 con 0x7f87840c5880 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.314+0000 7f8788ed8640 1 -- 172.21.15.94:0/3024540140 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c0343f0 con 0x7f87840c5880 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 -- 172.21.15.94:0/3024540140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c5880 msgr2=0x7f8784158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 --2- 172.21.15.94:0/3024540140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c5880 0x7f8784158b60 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f878408c840 tx=0x7f877c02be40 comp rx=0 tx=0).stop 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 -- 172.21.15.94:0/3024540140 shutdown_connections 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 --2- 172.21.15.94:0/3024540140 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c5880 0x7f8784158b60 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 --2- 172.21.15.94:0/3024540140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c88c0 0x7f87840c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 --2- 172.21.15.94:0/3024540140 >> v2:172.21.15.145:3300/0 conn(0x7f878414f360 0x7f8784155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 -- 172.21.15.94:0/3024540140 >> 172.21.15.94:0/3024540140 conn(0x7f87840ac570 msgr2=0x7f87840ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 -- 172.21.15.94:0/3024540140 shutdown_connections 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.315+0000 7f878c13e640 1 -- 172.21.15.94:0/3024540140 wait complete. 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.316+0000 7f878c13e640 1 Processor -- start 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.316+0000 7f878c13e640 1 -- start start 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f878c13e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c5880 0x7f87840c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f878c13e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c88c0 0x7f87841580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f878c13e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f878414f360 0x7f87841ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f878c13e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87841590a0 con 0x7f87840c88c0 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f878c13e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87840c7c30 con 0x7f878414f360 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f878c13e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8784159220 con 0x7f87840c5880 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c88c0 0x7f87841580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c88c0 0x7f87841580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47868/0 (socket says 172.21.15.94:47868) 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 -- 172.21.15.94:0/951075358 learned_addr learned my addr 172.21.15.94:0/951075358 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f8789eda640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c5880 0x7f87840c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f878a6db640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f878414f360 0x7f87841ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 -- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c5880 msgr2=0x7f87840c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c5880 0x7f87840c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 -- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f878414f360 msgr2=0x7f87841ee2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f878414f360 0x7f87841ee2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 -- 172.21.15.94:0/951075358 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87840ab600 con 0x7f87840c88c0 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f8789eda640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c5880 0x7f87840c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.317+0000 7f87896d9640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c88c0 0x7f87841580e0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f877400c7c0 tx=0x7f877400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.318+0000 7f877a7fc640 1 -- 172.21.15.94:0/951075358 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774015070 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.318+0000 7f877a7fc640 1 -- 172.21.15.94:0/951075358 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8774007820 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.318+0000 7f877a7fc640 1 -- 172.21.15.94:0/951075358 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774007bf0 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.318+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87841ede10 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.318+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87840abb80 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.319+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87841edfe0 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.319+0000 7f877a7fc640 1 -- 172.21.15.94:0/951075358 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8774018070 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.319+0000 7f877a7fc640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f876806b4d0 0x7f876806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.320+0000 7f877a7fc640 1 -- 172.21.15.94:0/951075358 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f877408a180 con 0x7f87840c88c0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.320+0000 7f8789eda640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f876806b4d0 0x7f876806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.320+0000 7f8789eda640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f876806b4d0 0x7f876806d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8780006fd0 tx=0x7f8780008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.324+0000 7f877a7fc640 1 -- 172.21.15.94:0/951075358 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8774058ed0 con 0x7f87840c88c0 2024-03-06T07:27:03.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.531+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8784075300 con 0x7f87840c88c0 2024-03-06T07:27:03.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.531+0000 7f877a7fc640 1 -- 172.21.15.94:0/951075358 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+4970 (secure 0 0 0) 0x7f8784075300 con 0x7f87840c88c0 2024-03-06T07:27:03.533 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:03.533 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":190,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/399318570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":399318570},{"type":"v1","addr":"172.21.15.145:6837","nonce":399318570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190},{"gid":10428,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2752830090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2752830090},{"type":"v1","addr":"172.21.15.94:6834","nonce":2752830090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":10431,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/750154668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":750154668},{"type":"v1","addr":"172.21.15.94:6835","nonce":750154668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":38057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/565245371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":565245371},{"type":"v1","addr":"172.21.15.145:6836","nonce":565245371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:24.949828+0000","modified":"2024-03-06T07:26:45.519981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:27:03.534 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 190 2024-03-06T07:27:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f876806b4d0 msgr2=0x7f876806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f876806b4d0 0x7f876806d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8780006fd0 tx=0x7f8780008040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c88c0 msgr2=0x7f87841580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c88c0 0x7f87841580e0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f877400c7c0 tx=0x7f877400cc80 comp rx=0 tx=0).stop 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 shutdown_connections 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f876806b4d0 0x7f876806d990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f878414f360 0x7f87841ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f87840c88c0 0x7f87841580e0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 --2- 172.21.15.94:0/951075358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87840c5880 0x7f87840c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 >> 172.21.15.94:0/951075358 conn(0x7f87840ac570 msgr2=0x7f87840cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 shutdown_connections 2024-03-06T07:27:03.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.533+0000 7f878c13e640 1 -- 172.21.15.94:0/951075358 wait complete. 2024-03-06T07:27:03.546 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14db24e640 1 Processor -- start 2024-03-06T07:27:03.645 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14db24e640 1 -- start start 2024-03-06T07:27:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14db24e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.646 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14db24e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14d4158b60 0x7f14d4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14db24e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d40c73d0 con 0x7f14d414f360 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14db24e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d40c8dd0 con 0x7f14d4158b60 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14db24e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40be2e0 con 0x7f14d40c88c0 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.644+0000 7f14d97eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14d4158b60 0x7f14d4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.645+0000 7f14d97eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f14d4158b60 0x7f14d4158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47878/0 (socket says 172.21.15.94:47878) 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.645+0000 7f14d97eb640 1 -- 172.21.15.94:0/2689537262 learned_addr learned my addr 172.21.15.94:0/2689537262 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.645+0000 7f14d3fff640 1 --2- 172.21.15.94:0/2689537262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.645+0000 7f14d97eb640 1 -- 172.21.15.94:0/2689537262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 msgr2=0x7f14d4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.645+0000 7f14d97eb640 1 --2- 172.21.15.94:0/2689537262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.645+0000 7f14d97eb640 1 -- 172.21.15.94:0/2689537262 >> v1:172.21.15.145:6789/0 conn(0x7f14d414f360 legacy=0x7f14d40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.647 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.645+0000 7f14d97eb640 1 -- 172.21.15.94:0/2689537262 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d40c7670 con 0x7f14d4158b60 2024-03-06T07:27:03.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14d97eb640 1 --2- 172.21.15.94:0/2689537262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d4158b60 0x7f14d4158ff0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009870 tx=0x7f14c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6e906949dc6ef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14d37fe640 1 -- 172.21.15.94:0/2689537262 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c4035020 con 0x7f14d4158b60 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14d37fe640 1 -- 172.21.15.94:0/2689537262 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c4031640 con 0x7f14d4158b60 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14d37fe640 1 -- 172.21.15.94:0/2689537262 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c402d040 con 0x7f14d4158b60 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 -- 172.21.15.94:0/2689537262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d4158b60 msgr2=0x7f14d4158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 --2- 172.21.15.94:0/2689537262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d4158b60 0x7f14d4158ff0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009870 tx=0x7f14c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 -- 172.21.15.94:0/2689537262 shutdown_connections 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 --2- 172.21.15.94:0/2689537262 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d4158b60 0x7f14d4158ff0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 --2- 172.21.15.94:0/2689537262 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 -- 172.21.15.94:0/2689537262 >> 172.21.15.94:0/2689537262 conn(0x7f14d40ac570 msgr2=0x7f14d40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 -- 172.21.15.94:0/2689537262 shutdown_connections 2024-03-06T07:27:03.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.646+0000 7f14db24e640 1 -- 172.21.15.94:0/2689537262 wait complete. 2024-03-06T07:27:03.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.647+0000 7f14db24e640 1 Processor -- start 2024-03-06T07:27:03.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.647+0000 7f14db24e640 1 -- start start 2024-03-06T07:27:03.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14db24e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d40be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14db24e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d414f360 0x7f14d40bf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14db24e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14d4158b60 0x7f14d414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14db24e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40c5a50 con 0x7f14d414f360 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14db24e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40c7c30 con 0x7f14d4158b60 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14db24e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40c5bd0 con 0x7f14d40c88c0 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14d3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d414f360 0x7f14d40bf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14d3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d414f360 0x7f14d40bf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47882/0 (socket says 172.21.15.94:47882) 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.648+0000 7f14d3fff640 1 -- 172.21.15.94:0/1928111433 learned_addr learned my addr 172.21.15.94:0/1928111433 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d97eb640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14d4158b60 0x7f14d414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d8fea640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d40be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d3fff640 1 -- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 msgr2=0x7f14d40be9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d3fff640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d40be9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d3fff640 1 -- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14d4158b60 msgr2=0x7f14d414bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d3fff640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14d4158b60 0x7f14d414bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d3fff640 1 -- 172.21.15.94:0/1928111433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d40c1690 con 0x7f14d414f360 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d8fea640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d40be9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:03.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d97eb640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14d4158b60 0x7f14d414bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:03.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d3fff640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d414f360 0x7f14d40bf610 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00e840 tx=0x7f14bc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d17fa640 1 -- 172.21.15.94:0/1928111433 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc017070 con 0x7f14d414f360 2024-03-06T07:27:03.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d17fa640 1 -- 172.21.15.94:0/1928111433 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc00c940 con 0x7f14d414f360 2024-03-06T07:27:03.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14d17fa640 1 -- 172.21.15.94:0/1928111433 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc00cd10 con 0x7f14d414f360 2024-03-06T07:27:03.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d40c13b0 con 0x7f14d414f360 2024-03-06T07:27:03.653 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.649+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d41e03c0 con 0x7f14d414f360 2024-03-06T07:27:03.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.650+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d404f330 con 0x7f14d414f360 2024-03-06T07:27:03.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.651+0000 7f14d17fa640 1 -- 172.21.15.94:0/1928111433 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14bc01a070 con 0x7f14d414f360 2024-03-06T07:27:03.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.651+0000 7f14d17fa640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.651+0000 7f14d17fa640 1 -- 172.21.15.94:0/1928111433 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f14bc08a180 con 0x7f14d414f360 2024-03-06T07:27:03.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.651+0000 7f14d8fea640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.654 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.652+0000 7f14d8fea640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f14d41ebd40 tx=0x7f14cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.655+0000 7f14d17fa640 1 -- 172.21.15.94:0/1928111433 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14bc058ed0 con 0x7f14d414f360 2024-03-06T07:27:03.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.826+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14d406cda0 con 0x7f14d414f360 2024-03-06T07:27:03.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.828+0000 7f14d17fa640 1 -- 172.21.15.94:0/1928111433 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+17049 (secure 0 0 0) 0x7f14bc0587d0 con 0x7f14d414f360 2024-03-06T07:27:03.829 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:03.830 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":198,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:59.299653+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:23.168853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:23.918870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"32.c","mappings":[{"from":6,"to":5}]},{"pgid":"32.33","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.829+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14b406b4d0 msgr2=0x7f14b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.829+0000 7f14db24e640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f14d41ebd40 tx=0x7f14cc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.829+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d414f360 msgr2=0x7f14d40bf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.829+0000 7f14db24e640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d414f360 0x7f14d40bf610 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00e840 tx=0x7f14bc00ed00 comp rx=0 tx=0).stop 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 shutdown_connections 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f14d4158b60 0x7f14d414bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14d414f360 0x7f14d40bf610 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 --2- 172.21.15.94:0/1928111433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14d40c88c0 0x7f14d40be9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 >> 172.21.15.94:0/1928111433 conn(0x7f14d40ac570 msgr2=0x7f14d414e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 shutdown_connections 2024-03-06T07:27:03.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.830+0000 7f14db24e640 1 -- 172.21.15.94:0/1928111433 wait complete. 2024-03-06T07:27:03.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:23.168853+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:23.918870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:03.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:03.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.944+0000 7f4708a37640 1 Processor -- start 2024-03-06T07:27:03.946 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.945+0000 7f4708a37640 1 -- start start 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.945+0000 7f4708a37640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f47040b0070 0x7f4704155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.945+0000 7f4708a37640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47040b0450 con 0x7f47040c5380 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.945+0000 7f4708a37640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470406cf20 con 0x7f470406ca70 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.945+0000 7f4708a37640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470406c020 con 0x7f47040b0070 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.945+0000 7f4702d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f47040c5380 0x7f470414e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52400/0 (socket says 172.21.15.94:52400) 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.945+0000 7f4702d9d640 1 -- 172.21.15.94:0/4137075912 learned_addr learned my addr 172.21.15.94:0/4137075912 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f470259c640 1 --2- 172.21.15.94:0/4137075912 >> v2:172.21.15.145:3300/0 conn(0x7f47040b0070 0x7f4704155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136463809 0 0) 0x7f46fc001490 con 0x7f47040c5380 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f0003520 con 0x7f47040c5380 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420922050 0 0) 0x7f46f0003520 con 0x7f47040c5380 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 >> v2:172.21.15.145:3300/0 conn(0x7f47040b0070 msgr2=0x7f4704155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 --2- 172.21.15.94:0/4137075912 >> v2:172.21.15.145:3300/0 conn(0x7f47040b0070 0x7f4704155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 >> v1:172.21.15.145:6789/0 conn(0x7f470406ca70 legacy=0x7f47040c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47040c77f0 con 0x7f47040c5380 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f46fc001bc0 con 0x7f47040c5380 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 >> v1:172.21.15.94:6789/0 conn(0x7f47040c5380 legacy=0x7f470414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46f00039b0 0x7f46f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46f0004800 0x7f46f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f46f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470406c020 con 0x7f46f00051c0 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f000c160 con 0x7f46f00039b0 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f000c360 con 0x7f46f0004800 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4702d9d640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f46f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f470259c640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46f00039b0 0x7f46f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f46fbfff640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46f0004800 0x7f46f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4702d9d640 1 -- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46f0004800 msgr2=0x7f46f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4702d9d640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46f0004800 0x7f46f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4702d9d640 1 -- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46f00039b0 msgr2=0x7f46f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4702d9d640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46f00039b0 0x7f46f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.946+0000 7f4702d9d640 1 -- 172.21.15.94:0/4137075912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f0003520 con 0x7f46f00051c0 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.947+0000 7f4702d9d640 1 -- 172.21.15.94:0/4137075912 reap_dead start 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.947+0000 7f4702d9d640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f46f000b9c0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00aa30 tx=0x7f46f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963a394cb8295ce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.947+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc038070 con 0x7f46f00051c0 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.947+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc033620 con 0x7f46f00051c0 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.947+0000 7f4701d9b640 1 -- 172.21.15.94:0/4137075912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46fc00a650 con 0x7f46f00051c0 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.948+0000 7f4708a37640 1 -- 172.21.15.94:0/4137075912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 msgr2=0x7f46f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.948+0000 7f4708a37640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f46f000b9c0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00aa30 tx=0x7f46f0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.948+0000 7f4708a37640 1 -- 172.21.15.94:0/4137075912 shutdown_connections 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.948+0000 7f4708a37640 1 --2- 172.21.15.94:0/4137075912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f46f000b9c0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.948+0000 7f4708a37640 1 -- 172.21.15.94:0/4137075912 >> 172.21.15.94:0/4137075912 conn(0x7f47040ac590 msgr2=0x7f47040ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.948+0000 7f4708a37640 1 -- 172.21.15.94:0/4137075912 shutdown_connections 2024-03-06T07:27:03.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.948+0000 7f4708a37640 1 -- 172.21.15.94:0/4137075912 wait complete. 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.949+0000 7f4708a37640 1 Processor -- start 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.949+0000 7f4708a37640 1 -- start start 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f47040c1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47040c38c0 0x7f47040c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47041e1a70 0x7f47041e1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470406c550 con 0x7f46f00051c0 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470406c270 con 0x7f47040c38c0 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470406da80 con 0x7f47041e1a70 2024-03-06T07:27:03.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f47040c1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f47040c1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47898/0 (socket says 172.21.15.94:47898) 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 -- 172.21.15.94:0/2502365985 learned_addr learned my addr 172.21.15.94:0/2502365985 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 -- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47041e1a70 msgr2=0x7f47041e1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47041e1a70 0x7f47041e1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 -- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47040c38c0 msgr2=0x7f47040c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47040c38c0 0x7f47040c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 -- 172.21.15.94:0/2502365985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4704152040 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f470259c640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f47040c1470 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f46f40118e0 tx=0x7f46f4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f46faffd640 1 -- 172.21.15.94:0/2502365985 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f400f9d0 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f46faffd640 1 -- 172.21.15.94:0/2502365985 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f400fb70 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4704152320 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f46faffd640 1 -- 172.21.15.94:0/2502365985 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4014440 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.950+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47040c29d0 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.951+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470404f330 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.952+0000 7f46faffd640 1 -- 172.21.15.94:0/2502365985 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46f4012040 con 0x7f46f00051c0 2024-03-06T07:27:03.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.952+0000 7f46faffd640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46e406b500 0x7f46e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.952+0000 7f46faffd640 1 -- 172.21.15.94:0/2502365985 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f46f408a830 con 0x7f46f00051c0 2024-03-06T07:27:03.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.952+0000 7f46fbfff640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46e406b500 0x7f46e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.953+0000 7f46fbfff640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46e406b500 0x7f46e406d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f46ec000c00 tx=0x7f46ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:03.956+0000 7f46faffd640 1 -- 172.21.15.94:0/2502365985 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46f4019030 con 0x7f46f00051c0 2024-03-06T07:27:04.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.127+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47040c5760 con 0x7f46f00051c0 2024-03-06T07:27:04.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.127+0000 7f46faffd640 1 -- 172.21.15.94:0/2502365985 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v198) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46f4059580 con 0x7f46f00051c0 2024-03-06T07:27:04.129 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:27:04.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46e406b500 msgr2=0x7f46e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46e406b500 0x7f46e406d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f46ec000c00 tx=0x7f46ec009040 comp rx=0 tx=0).stop 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 msgr2=0x7f47040c1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f47040c1470 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f46f40118e0 tx=0x7f46f4011da0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 shutdown_connections 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46e406b500 0x7f46e406d9c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47041e1a70 0x7f47041e1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47040c38c0 0x7f47040c2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 --2- 172.21.15.94:0/2502365985 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46f00051c0 0x7f47040c1470 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 >> 172.21.15.94:0/2502365985 conn(0x7f47040ac590 msgr2=0x7f47040c92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 shutdown_connections 2024-03-06T07:27:04.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.129+0000 7f4708a37640 1 -- 172.21.15.94:0/2502365985 wait complete. 2024-03-06T07:27:04.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:04.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.243+0000 7f506b70c640 1 Processor -- start 2024-03-06T07:27:04.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.244+0000 7f506b70c640 1 -- start start 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.244+0000 7f506b70c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506406c1f0 0x7f50640c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.244+0000 7f506b70c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640c2790 con 0x7f506414ef70 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.244+0000 7f506b70c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640c1540 con 0x7f506406cad0 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.244+0000 7f506b70c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50640c3090 con 0x7f506406c1f0 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.244+0000 7f50694a8640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f506414ef70 0x7f50640c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52404/0 (socket says 172.21.15.94:52404) 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.244+0000 7f50694a8640 1 -- 172.21.15.94:0/2690894151 learned_addr learned my addr 172.21.15.94:0/2690894151 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.245+0000 7f5068ca7640 1 --2- 172.21.15.94:0/2690894151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506406c1f0 0x7f50640c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.245+0000 7f5063fff640 1 -- 172.21.15.94:0/2690894151 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174142862 0 0) 0x7f50640c2790 con 0x7f506414ef70 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.245+0000 7f5063fff640 1 -- 172.21.15.94:0/2690894151 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c004630 con 0x7f506414ef70 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.245+0000 7f5063fff640 1 -- 172.21.15.94:0/2690894151 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468363163 0 0) 0x7f50640c1540 con 0x7f506406cad0 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.245+0000 7f5068ca7640 1 -- 172.21.15.94:0/2690894151 >> v1:172.21.15.145:6789/0 conn(0x7f506406cad0 legacy=0x7f5064154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.245+0000 7f5068ca7640 1 -- 172.21.15.94:0/2690894151 >> v1:172.21.15.94:6789/0 conn(0x7f506414ef70 legacy=0x7f50640c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.245+0000 7f5068ca7640 1 -- 172.21.15.94:0/2690894151 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640c24b0 con 0x7f506406c1f0 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f5068ca7640 1 --2- 172.21.15.94:0/2690894151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506406c1f0 0x7f50640c1820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f50580097d0 tx=0x7f505802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99897554b8c8d29b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f5063fff640 1 -- 172.21.15.94:0/2690894151 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058035020 con 0x7f506406c1f0 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f5063fff640 1 -- 172.21.15.94:0/2690894151 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5058031800 con 0x7f506406c1f0 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f5063fff640 1 -- 172.21.15.94:0/2690894151 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058031bb0 con 0x7f506406c1f0 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f506b70c640 1 -- 172.21.15.94:0/2690894151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506406c1f0 msgr2=0x7f50640c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f506b70c640 1 --2- 172.21.15.94:0/2690894151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506406c1f0 0x7f50640c1820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f50580097d0 tx=0x7f505802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f506b70c640 1 -- 172.21.15.94:0/2690894151 shutdown_connections 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f506b70c640 1 --2- 172.21.15.94:0/2690894151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506406c1f0 0x7f50640c1820 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.246+0000 7f506b70c640 1 -- 172.21.15.94:0/2690894151 >> 172.21.15.94:0/2690894151 conn(0x7f50640c6710 msgr2=0x7f50640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.247+0000 7f506b70c640 1 -- 172.21.15.94:0/2690894151 shutdown_connections 2024-03-06T07:27:04.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.247+0000 7f506b70c640 1 -- 172.21.15.94:0/2690894151 wait complete. 2024-03-06T07:27:04.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 Processor -- start 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 -- start start 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f506406c1f0 0x7f5064072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f506406cad0 0x7f506414d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506414ef70 0x7f5064149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506406cf60 con 0x7f506406c1f0 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506414f350 con 0x7f506406cad0 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f506b70c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5064155340 con 0x7f506414ef70 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f50694a8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f506406c1f0 0x7f5064072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f50694a8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f506406c1f0 0x7f5064072f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47908/0 (socket says 172.21.15.94:47908) 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f50694a8640 1 -- 172.21.15.94:0/961803704 learned_addr learned my addr 172.21.15.94:0/961803704 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f5068ca7640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f506406cad0 0x7f506414d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.248+0000 7f5069ca9640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506414ef70 0x7f5064149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f50694a8640 1 -- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506414ef70 msgr2=0x7f5064149a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f50694a8640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506414ef70 0x7f5064149a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f50694a8640 1 -- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f506406cad0 msgr2=0x7f506414d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f50694a8640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f506406cad0 0x7f506414d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f50694a8640 1 -- 172.21.15.94:0/961803704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640713b0 con 0x7f506406c1f0 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f50694a8640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f506406c1f0 0x7f5064072f20 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f505c00caa0 tx=0x7f505c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f5061ffb640 1 -- 172.21.15.94:0/961803704 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c017070 con 0x7f506406c1f0 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f5061ffb640 1 -- 172.21.15.94:0/961803704 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c00ac00 con 0x7f506406c1f0 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5064071690 con 0x7f506406c1f0 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f5061ffb640 1 -- 172.21.15.94:0/961803704 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c012400 con 0x7f506406c1f0 2024-03-06T07:27:04.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.249+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506414d8e0 con 0x7f506406c1f0 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.250+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064075300 con 0x7f506406c1f0 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.250+0000 7f5061ffb640 1 -- 172.21.15.94:0/961803704 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f505c01a070 con 0x7f506406c1f0 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.251+0000 7f5061ffb640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f505006b4d0 0x7f505006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.251+0000 7f5061ffb640 1 -- 172.21.15.94:0/961803704 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f505c08a1c0 con 0x7f506406c1f0 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.251+0000 7f5068ca7640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f505006b4d0 0x7f505006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.251+0000 7f5068ca7640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f505006b4d0 0x7f505006d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.255+0000 7f5061ffb640 1 -- 172.21.15.94:0/961803704 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5064075300 con 0x7f506406c1f0 2024-03-06T07:27:04.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.427+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f506404f330 con 0x7f506406c1f0 2024-03-06T07:27:04.429 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.427+0000 7f5061ffb640 1 -- 172.21.15.94:0/961803704 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v198) v1 ==== 104+0+11 (secure 0 0 0) 0x7f505c0587b0 con 0x7f506406c1f0 2024-03-06T07:27:04.429 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f505006b4d0 msgr2=0x7f505006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f505006b4d0 0x7f505006d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5058036000 tx=0x7f5058008030 comp rx=0 tx=0).stop 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f506406c1f0 msgr2=0x7f5064072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f506406c1f0 0x7f5064072f20 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f505c00caa0 tx=0x7f505c00cf60 comp rx=0 tx=0).stop 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 shutdown_connections 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f505006b4d0 0x7f505006d990 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f506414ef70 0x7f5064149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f506406cad0 0x7f506414d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 --2- 172.21.15.94:0/961803704 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f506406c1f0 0x7f5064072f20 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 >> 172.21.15.94:0/961803704 conn(0x7f50640c6710 msgr2=0x7f506414c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 shutdown_connections 2024-03-06T07:27:04.432 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.429+0000 7f506b70c640 1 -- 172.21.15.94:0/961803704 wait complete. 2024-03-06T07:27:04.441 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:04.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.543+0000 7f29ab434640 1 Processor -- start 2024-03-06T07:27:04.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29ab434640 1 -- start start 2024-03-06T07:27:04.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29ab434640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29a40b0070 0x7f29a4155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29ab434640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40b0450 con 0x7f29a40c5380 2024-03-06T07:27:04.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29ab434640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a406cf20 con 0x7f29a406ca70 2024-03-06T07:27:04.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29ab434640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a406c020 con 0x7f29a40b0070 2024-03-06T07:27:04.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29a91d0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f29a40b0070 0x7f29a4155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29a99d1640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f29a40c5380 0x7f29a414e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52414/0 (socket says 172.21.15.94:52414) 2024-03-06T07:27:04.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.544+0000 7f29a99d1640 1 -- 172.21.15.94:0/2895295101 learned_addr learned my addr 172.21.15.94:0/2895295101 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:04.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.545+0000 7f29a91d0640 1 -- 172.21.15.94:0/2895295101 >> v1:172.21.15.145:6789/0 conn(0x7f29a406ca70 legacy=0x7f29a40c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.545+0000 7f29a91d0640 1 -- 172.21.15.94:0/2895295101 >> v1:172.21.15.94:6789/0 conn(0x7f29a40c5380 legacy=0x7f29a414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.545+0000 7f299bfff640 1 -- 172.21.15.94:0/2895295101 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972470872 0 0) 0x7f299c001490 con 0x7f29a40c5380 2024-03-06T07:27:04.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.545+0000 7f29a91d0640 1 -- 172.21.15.94:0/2895295101 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a40c77f0 con 0x7f29a40b0070 2024-03-06T07:27:04.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.545+0000 7f29a91d0640 1 --2- 172.21.15.94:0/2895295101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4155c30 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f29a00097a0 tx=0x7f29a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64eb45af10ac6159 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f299bfff640 1 -- 172.21.15.94:0/2895295101 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0035020 con 0x7f29a40b0070 2024-03-06T07:27:04.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f299bfff640 1 -- 172.21.15.94:0/2895295101 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0031830 con 0x7f29a40b0070 2024-03-06T07:27:04.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f299bfff640 1 -- 172.21.15.94:0/2895295101 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0031c00 con 0x7f29a40b0070 2024-03-06T07:27:04.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f29ab434640 1 -- 172.21.15.94:0/2895295101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 msgr2=0x7f29a4155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f29ab434640 1 --2- 172.21.15.94:0/2895295101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4155c30 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f29a00097a0 tx=0x7f29a002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f29ab434640 1 -- 172.21.15.94:0/2895295101 shutdown_connections 2024-03-06T07:27:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f29ab434640 1 --2- 172.21.15.94:0/2895295101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4155c30 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f29ab434640 1 -- 172.21.15.94:0/2895295101 >> 172.21.15.94:0/2895295101 conn(0x7f29a40ac590 msgr2=0x7f29a40ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f29ab434640 1 -- 172.21.15.94:0/2895295101 shutdown_connections 2024-03-06T07:27:04.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.546+0000 7f29ab434640 1 -- 172.21.15.94:0/2895295101 wait complete. 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.547+0000 7f29ab434640 1 Processor -- start 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.547+0000 7f29ab434640 1 -- start start 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29ab434640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a406ca70 0x7f29a41542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29ab434640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29ab434640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a40c5380 0x7f29a41eb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29ab434640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a406c550 con 0x7f29a40b0070 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29ab434640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a406c270 con 0x7f29a40c5380 2024-03-06T07:27:04.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29ab434640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a406da80 con 0x7f29a406ca70 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47928/0 (socket says 172.21.15.94:47928) 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 -- 172.21.15.94:0/1231672945 learned_addr learned my addr 172.21.15.94:0/1231672945 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a99d1640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a40c5380 0x7f29a41eb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a91d0640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a406ca70 0x7f29a41542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 -- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a406ca70 msgr2=0x7f29a41542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a406ca70 0x7f29a41542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 -- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a40c5380 msgr2=0x7f29a41eb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a40c5380 0x7f29a41eb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a89cf640 1 -- 172.21.15.94:0/1231672945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4157d40 con 0x7f29a40b0070 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.548+0000 7f29a99d1640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a40c5380 0x7f29a41eb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.549+0000 7f29a89cf640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4158980 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f299400c430 tx=0x7f299400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.549+0000 7f2999ffb640 1 -- 172.21.15.94:0/1231672945 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994017070 con 0x7f29a40b0070 2024-03-06T07:27:04.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.549+0000 7f2999ffb640 1 -- 172.21.15.94:0/1231672945 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29940052d0 con 0x7f29a40b0070 2024-03-06T07:27:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.549+0000 7f2999ffb640 1 -- 172.21.15.94:0/1231672945 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994003830 con 0x7f29a40b0070 2024-03-06T07:27:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.549+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a4158020 con 0x7f29a40b0070 2024-03-06T07:27:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.549+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a41547f0 con 0x7f29a40b0070 2024-03-06T07:27:04.553 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.550+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a4158540 con 0x7f29a40b0070 2024-03-06T07:27:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.550+0000 7f2999ffb640 1 -- 172.21.15.94:0/1231672945 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f299401a070 con 0x7f29a40b0070 2024-03-06T07:27:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.550+0000 7f2999ffb640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f298806b4d0 0x7f298806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.551+0000 7f2999ffb640 1 -- 172.21.15.94:0/1231672945 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2994089680 con 0x7f29a40b0070 2024-03-06T07:27:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.551+0000 7f29a91d0640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f298806b4d0 0x7f298806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.551+0000 7f29a91d0640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f298806b4d0 0x7f298806d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f29a0036000 tx=0x7f29a0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.555+0000 7f2999ffb640 1 -- 172.21.15.94:0/1231672945 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29a4158540 con 0x7f29a40b0070 2024-03-06T07:27:04.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.728+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f29a404f330 con 0x7f29a40b0070 2024-03-06T07:27:04.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.728+0000 7f2999ffb640 1 -- 172.21.15.94:0/1231672945 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v198) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29940583d0 con 0x7f29a40b0070 2024-03-06T07:27:04.730 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:27:04.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f298806b4d0 msgr2=0x7f298806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f298806b4d0 0x7f298806d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f29a0036000 tx=0x7f29a0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 msgr2=0x7f29a4158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4158980 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f299400c430 tx=0x7f299400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 shutdown_connections 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f298806b4d0 0x7f298806d990 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a40c5380 0x7f29a41eb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f29a40b0070 0x7f29a4158980 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 --2- 172.21.15.94:0/1231672945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a406ca70 0x7f29a41542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 >> 172.21.15.94:0/1231672945 conn(0x7f29a40ac590 msgr2=0x7f29a40ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 shutdown_connections 2024-03-06T07:27:04.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.730+0000 7f29ab434640 1 -- 172.21.15.94:0/1231672945 wait complete. 2024-03-06T07:27:04.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:04.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.841+0000 7f922710c640 1 Processor -- start 2024-03-06T07:27:04.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.841+0000 7f922710c640 1 -- start start 2024-03-06T07:27:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f922710c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9220150920 0x7f92200c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f922710c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92200c2e70 0x7f922015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f922710c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9220067340 con 0x7f922006c720 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f922710c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9220150d00 con 0x7f9220150920 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f922710c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92200c2010 con 0x7f92200c2e70 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f9224ea8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9220150920 0x7f92200c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f92256a9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92200c2e70 0x7f922015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f9224ea8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9220150920 0x7f92200c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:36198/0 (socket says 172.21.15.94:36198) 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f9224ea8640 1 -- 172.21.15.94:0/3881879212 learned_addr learned my addr 172.21.15.94:0/3881879212 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f9224ea8640 1 -- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92200c2e70 msgr2=0x7f922015dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f9224ea8640 1 --2- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92200c2e70 0x7f922015dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f9224ea8640 1 -- 172.21.15.94:0/3881879212 >> v1:172.21.15.145:6789/0 conn(0x7f922006c720 legacy=0x7f9220071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f9224ea8640 1 -- 172.21.15.94:0/3881879212 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220067060 con 0x7f9220150920 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.842+0000 7f92256a9640 1 --2- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92200c2e70 0x7f922015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:04.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.843+0000 7f9224ea8640 1 --2- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92200c2930 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9214009870 tx=0x7f921402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b5258fe698a023 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.843+0000 7f921f7fe640 1 -- 172.21.15.94:0/3881879212 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214035020 con 0x7f9220150920 2024-03-06T07:27:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.843+0000 7f921f7fe640 1 -- 172.21.15.94:0/3881879212 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9214031640 con 0x7f9220150920 2024-03-06T07:27:04.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.843+0000 7f921f7fe640 1 -- 172.21.15.94:0/3881879212 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921402d040 con 0x7f9220150920 2024-03-06T07:27:04.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 -- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 msgr2=0x7f92200c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 --2- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92200c2930 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9214009870 tx=0x7f921402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 -- 172.21.15.94:0/3881879212 shutdown_connections 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 --2- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92200c2e70 0x7f922015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 --2- 172.21.15.94:0/3881879212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92200c2930 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 -- 172.21.15.94:0/3881879212 >> 172.21.15.94:0/3881879212 conn(0x7f92200c70b0 msgr2=0x7f92200c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 -- 172.21.15.94:0/3881879212 shutdown_connections 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.844+0000 7f922710c640 1 -- 172.21.15.94:0/3881879212 wait complete. 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.845+0000 7f922710c640 1 Processor -- start 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.845+0000 7f922710c640 1 -- start start 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f922710c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f922006c720 0x7f9220154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f922710c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92200c2e70 0x7f92201530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f922710c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92201ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f922710c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9220073210 con 0x7f92200c2e70 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f922710c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9220073090 con 0x7f9220150920 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f922710c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92200c1b40 con 0x7f922006c720 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92200c2e70 0x7f92201530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92200c2e70 0x7f92201530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47934/0 (socket says 172.21.15.94:47934) 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 -- 172.21.15.94:0/3043619996 learned_addr learned my addr 172.21.15.94:0/3043619996 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f9224ea8640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f922006c720 0x7f9220154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f92256a9640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92201ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 -- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f922006c720 msgr2=0x7f9220154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f922006c720 0x7f9220154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 -- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 msgr2=0x7f92201ec0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92201ec0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f921ffff640 1 -- 172.21.15.94:0/3043619996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220154e60 con 0x7f92200c2e70 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f9224ea8640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f922006c720 0x7f9220154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.846+0000 7f92256a9640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92201ec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.847+0000 7f921ffff640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92200c2e70 0x7f92201530e0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c440 tx=0x7f920c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.847+0000 7f921d7fa640 1 -- 172.21.15.94:0/3043619996 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c0052d0 con 0x7f92200c2e70 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.847+0000 7f921d7fa640 1 -- 172.21.15.94:0/3043619996 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c005470 con 0x7f92200c2e70 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.847+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9220152480 con 0x7f92200c2e70 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.847+0000 7f921d7fa640 1 -- 172.21.15.94:0/3043619996 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c007260 con 0x7f92200c2e70 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.847+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9220151220 con 0x7f92200c2e70 2024-03-06T07:27:05.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.848+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9220150180 con 0x7f92200c2e70 2024-03-06T07:27:05.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.848+0000 7f921d7fa640 1 -- 172.21.15.94:0/3043619996 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f920c0074c0 con 0x7f92200c2e70 2024-03-06T07:27:05.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.849+0000 7f921d7fa640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f920406b4d0 0x7f920406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.849+0000 7f921d7fa640 1 -- 172.21.15.94:0/3043619996 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f920c089780 con 0x7f92200c2e70 2024-03-06T07:27:05.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.849+0000 7f9224ea8640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f920406b4d0 0x7f920406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.849+0000 7f9224ea8640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f920406b4d0 0x7f920406d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f9214036000 tx=0x7f9214008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:04.853+0000 7f921d7fa640 1 -- 172.21.15.94:0/3043619996 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9220150180 con 0x7f92200c2e70 2024-03-06T07:27:05.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.064+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f922004f330 con 0x7f92200c2e70 2024-03-06T07:27:05.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.064+0000 7f921d7fa640 1 -- 172.21.15.94:0/3043619996 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+4970 (secure 0 0 0) 0x7f920c0584d0 con 0x7f92200c2e70 2024-03-06T07:27:05.129 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:05.129 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":190,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/399318570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":399318570},{"type":"v1","addr":"172.21.15.145:6837","nonce":399318570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190},{"gid":10428,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2752830090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2752830090},{"type":"v1","addr":"172.21.15.94:6834","nonce":2752830090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":10431,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/750154668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":750154668},{"type":"v1","addr":"172.21.15.94:6835","nonce":750154668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":38057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/565245371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":565245371},{"type":"v1","addr":"172.21.15.145:6836","nonce":565245371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:24.949828+0000","modified":"2024-03-06T07:26:45.519981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 190 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.066+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f920406b4d0 msgr2=0x7f920406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.066+0000 7f922710c640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f920406b4d0 0x7f920406d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f9214036000 tx=0x7f9214008030 comp rx=0 tx=0).stop 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.066+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92200c2e70 msgr2=0x7f92201530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.066+0000 7f922710c640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92200c2e70 0x7f92201530e0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c440 tx=0x7f920c00c900 comp rx=0 tx=0).stop 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 shutdown_connections 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f920406b4d0 0x7f920406d990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9220150920 0x7f92201ec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92200c2e70 0x7f92201530e0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 --2- 172.21.15.94:0/3043619996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f922006c720 0x7f9220154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 >> 172.21.15.94:0/3043619996 conn(0x7f92200c70b0 msgr2=0x7f92200c7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.131 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 shutdown_connections 2024-03-06T07:27:05.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.067+0000 7f922710c640 1 -- 172.21.15.94:0/3043619996 wait complete. 2024-03-06T07:27:05.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.231+0000 7f487e15e640 1 Processor -- start 2024-03-06T07:27:05.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.231+0000 7f487e15e640 1 -- start start 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f487e15e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4878150580 0x7f487806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f487e15e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48780c1320 0x7f487806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f487e15e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4878153a60 con 0x7f48780c6000 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f487e15e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4878153d40 con 0x7f4878150580 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f487e15e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487806c2c0 con 0x7f48780c1320 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f48777fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4878150580 0x7f487806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48780c1320 0x7f487806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f48777fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4878150580 0x7f487806dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:36210/0 (socket says 172.21.15.94:36210) 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48780c1320 0x7f487806e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:35684/0 (socket says 172.21.15.94:35684) 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f48777fe640 1 -- 172.21.15.94:0/2179066665 learned_addr learned my addr 172.21.15.94:0/2179066665 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:05.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f4877fff640 1 --1- 172.21.15.94:0/2179066665 >> v1:172.21.15.145:6789/0 conn(0x7f48780c6000 0x7f48781566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:46510/0 (socket says 172.21.15.94:46510) 2024-03-06T07:27:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f48777fe640 1 -- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48780c1320 msgr2=0x7f487806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f48777fe640 1 --2- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48780c1320 0x7f487806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.232+0000 7f48777fe640 1 -- 172.21.15.94:0/2179066665 >> v1:172.21.15.145:6789/0 conn(0x7f48780c6000 legacy=0x7f48781566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.233+0000 7f48777fe640 1 -- 172.21.15.94:0/2179066665 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487806c5a0 con 0x7f4878150580 2024-03-06T07:27:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.233+0000 7f4876ffd640 1 --2- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48780c1320 0x7f487806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.233+0000 7f48777fe640 1 --2- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4878150580 0x7f487806dd10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4868009870 tx=0x7f486802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d7e27f2d4e9a79e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.233+0000 7f48767fc640 1 -- 172.21.15.94:0/2179066665 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868035020 con 0x7f4878150580 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f48767fc640 1 -- 172.21.15.94:0/2179066665 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4868031640 con 0x7f4878150580 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f48767fc640 1 -- 172.21.15.94:0/2179066665 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486802d040 con 0x7f4878150580 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 -- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4878150580 msgr2=0x7f487806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 --2- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4878150580 0x7f487806dd10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4868009870 tx=0x7f486802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 -- 172.21.15.94:0/2179066665 shutdown_connections 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 --2- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48780c1320 0x7f487806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 --2- 172.21.15.94:0/2179066665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4878150580 0x7f487806dd10 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 -- 172.21.15.94:0/2179066665 >> 172.21.15.94:0/2179066665 conn(0x7f48780c6710 msgr2=0x7f48780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 -- 172.21.15.94:0/2179066665 shutdown_connections 2024-03-06T07:27:05.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.234+0000 7f487e15e640 1 -- 172.21.15.94:0/2179066665 wait complete. 2024-03-06T07:27:05.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.235+0000 7f487e15e640 1 Processor -- start 2024-03-06T07:27:05.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.235+0000 7f487e15e640 1 -- start start 2024-03-06T07:27:05.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f487e15e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48780c1320 0x7f4878073030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f487e15e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48780c6000 0x7f48781541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f487e15e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4878150580 0x7f48780c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f487e15e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48780c0dc0 con 0x7f48780c6000 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f487e15e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48780c0c40 con 0x7f48780c1320 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f487e15e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4878156dc0 con 0x7f4878150580 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48780c6000 0x7f48781541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48780c6000 0x7f48781541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47948/0 (socket says 172.21.15.94:47948) 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 -- 172.21.15.94:0/4108795548 learned_addr learned my addr 172.21.15.94:0/4108795548 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4877fff640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4878150580 0x7f48780c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f48777fe640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48780c1320 0x7f4878073030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 -- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4878150580 msgr2=0x7f48780c2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4878150580 0x7f48780c2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 -- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48780c1320 msgr2=0x7f4878073030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48780c1320 0x7f4878073030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.236+0000 7f4876ffd640 1 -- 172.21.15.94:0/4108795548 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48780c3560 con 0x7f48780c6000 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.237+0000 7f4876ffd640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48780c6000 0x7f48781541b0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f486c00ebe0 tx=0x7f486c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.237+0000 7f485ffff640 1 -- 172.21.15.94:0/4108795548 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c017070 con 0x7f48780c6000 2024-03-06T07:27:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.237+0000 7f485ffff640 1 -- 172.21.15.94:0/4108795548 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c00cc30 con 0x7f48780c6000 2024-03-06T07:27:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.237+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4878073570 con 0x7f48780c6000 2024-03-06T07:27:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.237+0000 7f485ffff640 1 -- 172.21.15.94:0/4108795548 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c013470 con 0x7f48780c6000 2024-03-06T07:27:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.237+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487814e7f0 con 0x7f48780c6000 2024-03-06T07:27:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.238+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4878075300 con 0x7f48780c6000 2024-03-06T07:27:05.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.238+0000 7f485ffff640 1 -- 172.21.15.94:0/4108795548 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f486c01a070 con 0x7f48780c6000 2024-03-06T07:27:05.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.238+0000 7f485ffff640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f486406b4d0 0x7f486406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.239+0000 7f485ffff640 1 -- 172.21.15.94:0/4108795548 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f486c08a210 con 0x7f48780c6000 2024-03-06T07:27:05.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.239+0000 7f48777fe640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f486406b4d0 0x7f486406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.239+0000 7f48777fe640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f486406b4d0 0x7f486406d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4868036000 tx=0x7f486803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.243+0000 7f485ffff640 1 -- 172.21.15.94:0/4108795548 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4878075300 con 0x7f48780c6000 2024-03-06T07:27:05.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.415+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f487804f330 con 0x7f48780c6000 2024-03-06T07:27:05.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.416+0000 7f485ffff640 1 -- 172.21.15.94:0/4108795548 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+17049 (secure 0 0 0) 0x7f48780c3560 con 0x7f48780c6000 2024-03-06T07:27:05.418 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:05.418 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":198,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:59.299653+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:23.168853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:23.918870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"32.c","mappings":[{"from":6,"to":5}]},{"pgid":"32.33","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:05.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f486406b4d0 msgr2=0x7f486406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f486406b4d0 0x7f486406d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4868036000 tx=0x7f486803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48780c6000 msgr2=0x7f48781541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48780c6000 0x7f48781541b0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f486c00ebe0 tx=0x7f486c00c520 comp rx=0 tx=0).stop 2024-03-06T07:27:05.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 shutdown_connections 2024-03-06T07:27:05.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f486406b4d0 0x7f486406d990 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4878150580 0x7f48780c2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48780c6000 0x7f48781541b0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 --2- 172.21.15.94:0/4108795548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f48780c1320 0x7f4878073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 >> 172.21.15.94:0/4108795548 conn(0x7f48780c6710 msgr2=0x7f48781528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 shutdown_connections 2024-03-06T07:27:05.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.418+0000 7f487e15e640 1 -- 172.21.15.94:0/4108795548 wait complete. 2024-03-06T07:27:05.431 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:27:05.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:05.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.530+0000 7f8c2f18c640 1 Processor -- start 2024-03-06T07:27:05.532 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c2f18c640 1 -- start start 2024-03-06T07:27:05.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c2f18c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8c281527b0 0x7f8c28159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c2f18c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8c280c7180 0x7f8c280c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c2f18c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c28155b20 0x7f8c280c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c2f18c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c28159640 con 0x7f8c280c7180 2024-03-06T07:27:05.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c2f18c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c280c8df0 con 0x7f8c281527b0 2024-03-06T07:27:05.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c2f18c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280c7c60 con 0x7f8c28155b20 2024-03-06T07:27:05.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c27fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8c280c7180 0x7f8c280c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c27fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8c280c7180 0x7f8c280c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47954/0 (socket says 172.21.15.94:47954) 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.531+0000 7f8c27fff640 1 -- 172.21.15.94:0/515604643 learned_addr learned my addr 172.21.15.94:0/515604643 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c2d729640 1 --2- 172.21.15.94:0/515604643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c28155b20 0x7f8c280c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c2cf28640 1 --2- 172.21.15.94:0/515604643 >> v2:172.21.15.145:3300/0 conn(0x7f8c281527b0 0x7f8c28159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c27fff640 1 -- 172.21.15.94:0/515604643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c28155b20 msgr2=0x7f8c280c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c27fff640 1 --2- 172.21.15.94:0/515604643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c28155b20 0x7f8c280c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c27fff640 1 -- 172.21.15.94:0/515604643 >> v2:172.21.15.145:3300/0 conn(0x7f8c281527b0 msgr2=0x7f8c28159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c27fff640 1 --2- 172.21.15.94:0/515604643 >> v2:172.21.15.145:3300/0 conn(0x7f8c281527b0 0x7f8c28159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c27fff640 1 -- 172.21.15.94:0/515604643 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280c8060 con 0x7f8c280c7180 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c27fff640 1 --2- 172.21.15.94:0/515604643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c280c7180 0x7f8c280c88b0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f8c280cba80 tx=0x7f8c1402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb8cb06dd51b153 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.532+0000 7f8c277fe640 1 -- 172.21.15.94:0/515604643 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1402e070 con 0x7f8c280c7180 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c277fe640 1 -- 172.21.15.94:0/515604643 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c14037070 con 0x7f8c280c7180 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c277fe640 1 -- 172.21.15.94:0/515604643 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14031a00 con 0x7f8c280c7180 2024-03-06T07:27:05.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 -- 172.21.15.94:0/515604643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c280c7180 msgr2=0x7f8c280c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/515604643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c280c7180 0x7f8c280c88b0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f8c280cba80 tx=0x7f8c1402b880 comp rx=0 tx=0).stop 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 -- 172.21.15.94:0/515604643 shutdown_connections 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/515604643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c28155b20 0x7f8c280c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/515604643 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c280c7180 0x7f8c280c88b0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/515604643 >> v2:172.21.15.145:3300/0 conn(0x7f8c281527b0 0x7f8c28159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 -- 172.21.15.94:0/515604643 >> 172.21.15.94:0/515604643 conn(0x7f8c280ac560 msgr2=0x7f8c280ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 -- 172.21.15.94:0/515604643 shutdown_connections 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.533+0000 7f8c2f18c640 1 -- 172.21.15.94:0/515604643 wait complete. 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.534+0000 7f8c2f18c640 1 Processor -- start 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.534+0000 7f8c2f18c640 1 -- start start 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2f18c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c280c7180 0x7f8c280c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2f18c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c281527b0 0x7f8c281582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2f18c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c28155b20 0x7f8c281eb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2f18c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28155320 con 0x7f8c28155b20 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2f18c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280c7980 con 0x7f8c281527b0 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2f18c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28155600 con 0x7f8c280c7180 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c28155b20 0x7f8c281eb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c28155b20 0x7f8c281eb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47958/0 (socket says 172.21.15.94:47958) 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 -- 172.21.15.94:0/3910128733 learned_addr learned my addr 172.21.15.94:0/3910128733 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2cf28640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c280c7180 0x7f8c280c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c27fff640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c281527b0 0x7f8c281582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 -- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c280c7180 msgr2=0x7f8c280c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c280c7180 0x7f8c280c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 -- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c281527b0 msgr2=0x7f8c281582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c281527b0 0x7f8c281582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2d729640 1 -- 172.21.15.94:0/3910128733 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280cb050 con 0x7f8c28155b20 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c27fff640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c281527b0 0x7f8c281582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:05.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.535+0000 7f8c2cf28640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c280c7180 0x7f8c280c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.536+0000 7f8c2d729640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c28155b20 0x7f8c281eb550 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f8c20007e90 tx=0x7f8c2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.536+0000 7f8c257fa640 1 -- 172.21.15.94:0/3910128733 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20014420 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.536+0000 7f8c257fa640 1 -- 172.21.15.94:0/3910128733 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c20019070 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.536+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c281ead50 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.536+0000 7f8c257fa640 1 -- 172.21.15.94:0/3910128733 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20014800 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.536+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c280cb330 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.537+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2804f330 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.537+0000 7f8c257fa640 1 -- 172.21.15.94:0/3910128733 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c200149a0 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.537+0000 7f8c257fa640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c0c06b4d0 0x7f8c0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.538+0000 7f8c257fa640 1 -- 172.21.15.94:0/3910128733 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8c2008aaa0 con 0x7f8c28155b20 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.538+0000 7f8c2cf28640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c0c06b4d0 0x7f8c0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.538+0000 7f8c2cf28640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c0c06b4d0 0x7f8c0c06d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c006fd0 tx=0x7f8c1c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.542+0000 7f8c257fa640 1 -- 172.21.15.94:0/3910128733 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c200597f0 con 0x7f8c28155b20 2024-03-06T07:27:05.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.753+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c2806ce00 con 0x7f8c28155b20 2024-03-06T07:27:05.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.753+0000 7f8c257fa640 1 -- 172.21.15.94:0/3910128733 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+4970 (secure 0 0 0) 0x7f8c200590f0 con 0x7f8c28155b20 2024-03-06T07:27:05.755 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:05.756 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":190,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/399318570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":399318570},{"type":"v1","addr":"172.21.15.145:6837","nonce":399318570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190},{"gid":10428,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2752830090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2752830090},{"type":"v1","addr":"172.21.15.94:6834","nonce":2752830090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":10431,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/750154668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":750154668},{"type":"v1","addr":"172.21.15.94:6835","nonce":750154668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":38057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/565245371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":565245371},{"type":"v1","addr":"172.21.15.145:6836","nonce":565245371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:24.949828+0000","modified":"2024-03-06T07:26:45.519981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:27:05.756 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 190 2024-03-06T07:27:05.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.755+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c0c06b4d0 msgr2=0x7f8c0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.755+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c0c06b4d0 0x7f8c0c06d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c006fd0 tx=0x7f8c1c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.755+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c28155b20 msgr2=0x7f8c281eb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.755+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c28155b20 0x7f8c281eb550 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f8c20007e90 tx=0x7f8c2000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.755+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 shutdown_connections 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.755+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c0c06b4d0 0x7f8c0c06d990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.756+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c28155b20 0x7f8c281eb550 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.756+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c281527b0 0x7f8c281582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.756+0000 7f8c2f18c640 1 --2- 172.21.15.94:0/3910128733 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c280c7180 0x7f8c280c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.756+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 >> 172.21.15.94:0/3910128733 conn(0x7f8c280ac560 msgr2=0x7f8c28154ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.756+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 shutdown_connections 2024-03-06T07:27:05.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.756+0000 7f8c2f18c640 1 -- 172.21.15.94:0/3910128733 wait complete. 2024-03-06T07:27:05.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:05.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.867+0000 7fa2be071640 1 Processor -- start 2024-03-06T07:27:05.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.867+0000 7fa2be071640 1 -- start start 2024-03-06T07:27:05.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.867+0000 7fa2be071640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa2b80c88c0 0x7fa2b8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.867+0000 7fa2be071640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b8158b60 0x7fa2b8158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.867+0000 7fa2be071640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b80c73d0 con 0x7fa2b814f360 2024-03-06T07:27:05.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.867+0000 7fa2be071640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b80c8dd0 con 0x7fa2b80c88c0 2024-03-06T07:27:05.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.867+0000 7fa2be071640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80be2e0 con 0x7fa2b8158b60 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2affff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa2b80c88c0 0x7fa2b8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2b77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa2b814f360 0x7fa2b80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52424/0 (socket says 172.21.15.94:52424) 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2b77fe640 1 -- 172.21.15.94:0/3356174302 learned_addr learned my addr 172.21.15.94:0/3356174302 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2b7fff640 1 --2- 172.21.15.94:0/3356174302 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b8158b60 0x7fa2b8158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2affff640 1 -- 172.21.15.94:0/3356174302 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b8158b60 msgr2=0x7fa2b8158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2affff640 1 --2- 172.21.15.94:0/3356174302 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b8158b60 0x7fa2b8158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2affff640 1 -- 172.21.15.94:0/3356174302 >> v1:172.21.15.94:6789/0 conn(0x7fa2b814f360 legacy=0x7fa2b80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2affff640 1 -- 172.21.15.94:0/3356174302 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80c7670 con 0x7fa2b80c88c0 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.868+0000 7fa2b6ffd640 1 -- 172.21.15.94:0/3356174302 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265769544 0 0) 0x7fa2b80c73d0 con 0x7fa2b814f360 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2affff640 1 --2- 172.21.15.94:0/3356174302 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b80c88c0 0x7fa2b8155c10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00097a0 tx=0x7fa2a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d5d0c921a94927 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2b6ffd640 1 -- 172.21.15.94:0/3356174302 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00315a0 con 0x7fa2b80c88c0 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2b6ffd640 1 -- 172.21.15.94:0/3356174302 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a0037070 con 0x7fa2b80c88c0 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2b6ffd640 1 -- 172.21.15.94:0/3356174302 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0031960 con 0x7fa2b80c88c0 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 -- 172.21.15.94:0/3356174302 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b80c88c0 msgr2=0x7fa2b8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 --2- 172.21.15.94:0/3356174302 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b80c88c0 0x7fa2b8155c10 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00097a0 tx=0x7fa2a002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 -- 172.21.15.94:0/3356174302 shutdown_connections 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 --2- 172.21.15.94:0/3356174302 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b8158b60 0x7fa2b8158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 --2- 172.21.15.94:0/3356174302 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b80c88c0 0x7fa2b8155c10 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 -- 172.21.15.94:0/3356174302 >> 172.21.15.94:0/3356174302 conn(0x7fa2b80ac570 msgr2=0x7fa2b80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 -- 172.21.15.94:0/3356174302 shutdown_connections 2024-03-06T07:27:05.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.869+0000 7fa2be071640 1 -- 172.21.15.94:0/3356174302 wait complete. 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.870+0000 7fa2be071640 1 Processor -- start 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.870+0000 7fa2be071640 1 -- start start 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2be071640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b80c88c0 0x7fa2b81ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2be071640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b814f360 0x7fa2b8157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2be071640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2b8158b60 0x7fa2b80c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2be071640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80c5a50 con 0x7fa2b814f360 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2be071640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80c7c30 con 0x7fa2b8158b60 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2be071640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80c5bd0 con 0x7fa2b80c88c0 2024-03-06T07:27:05.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b814f360 0x7fa2b8157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b814f360 0x7fa2b8157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47984/0 (socket says 172.21.15.94:47984) 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 -- 172.21.15.94:0/3828919246 learned_addr learned my addr 172.21.15.94:0/3828919246 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2b77fe640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b80c88c0 0x7fa2b81ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2b7fff640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2b8158b60 0x7fa2b80c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 -- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b80c88c0 msgr2=0x7fa2b81ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b80c88c0 0x7fa2b81ee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 -- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2b8158b60 msgr2=0x7fa2b80c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2b8158b60 0x7fa2b80c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.871+0000 7fa2affff640 1 -- 172.21.15.94:0/3828919246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80c0930 con 0x7fa2b814f360 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.872+0000 7fa2affff640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b814f360 0x7fa2b8157c80 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00097a0 tx=0x7fa2a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.872+0000 7fa2b4ff9640 1 -- 172.21.15.94:0/3828919246 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0004230 con 0x7fa2b814f360 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.872+0000 7fa2b4ff9640 1 -- 172.21.15.94:0/3828919246 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a00043f0 con 0x7fa2b814f360 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.872+0000 7fa2b4ff9640 1 -- 172.21.15.94:0/3828919246 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0003850 con 0x7fa2b814f360 2024-03-06T07:27:05.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.872+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b80c0c10 con 0x7fa2b814f360 2024-03-06T07:27:05.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.872+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b81581c0 con 0x7fa2b814f360 2024-03-06T07:27:05.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.873+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b804f330 con 0x7fa2b814f360 2024-03-06T07:27:05.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.873+0000 7fa2b4ff9640 1 -- 172.21.15.94:0/3828919246 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2a00039f0 con 0x7fa2b814f360 2024-03-06T07:27:05.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.874+0000 7fa2b4ff9640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa29806b4d0 0x7fa29806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.874+0000 7fa2b4ff9640 1 -- 172.21.15.94:0/3828919246 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa2a002f030 con 0x7fa2b814f360 2024-03-06T07:27:05.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.874+0000 7fa2b77fe640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa29806b4d0 0x7fa29806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.874+0000 7fa2b77fe640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa29806b4d0 0x7fa29806d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8000f60 tx=0x7fa2a8007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:05.878+0000 7fa2b4ff9640 1 -- 172.21.15.94:0/3828919246 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2a0041070 con 0x7fa2b814f360 2024-03-06T07:27:06.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.084+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2b806cda0 con 0x7fa2b814f360 2024-03-06T07:27:06.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.084+0000 7fa2b4ff9640 1 -- 172.21.15.94:0/3828919246 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+4970 (secure 0 0 0) 0x7fa2a0044340 con 0x7fa2b814f360 2024-03-06T07:27:06.086 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:06.086 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":190,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/399318570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":399318570},{"type":"v1","addr":"172.21.15.145:6837","nonce":399318570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190},{"gid":10428,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2752830090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2752830090},{"type":"v1","addr":"172.21.15.94:6834","nonce":2752830090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":10431,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/750154668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":750154668},{"type":"v1","addr":"172.21.15.94:6835","nonce":750154668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":38057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/565245371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":565245371},{"type":"v1","addr":"172.21.15.145:6836","nonce":565245371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:24.949828+0000","modified":"2024-03-06T07:26:45.519981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":15}]} 2024-03-06T07:27:06.087 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 190 2024-03-06T07:27:06.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa29806b4d0 msgr2=0x7fa29806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa29806b4d0 0x7fa29806d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8000f60 tx=0x7fa2a8007b50 comp rx=0 tx=0).stop 2024-03-06T07:27:06.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b814f360 msgr2=0x7fa2b8157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b814f360 0x7fa2b8157c80 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00097a0 tx=0x7fa2a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 shutdown_connections 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa29806b4d0 0x7fa29806d990 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2b8158b60 0x7fa2b80c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2b814f360 0x7fa2b8157c80 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 --2- 172.21.15.94:0/3828919246 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2b80c88c0 0x7fa2b81ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 >> 172.21.15.94:0/3828919246 conn(0x7fa2b80ac570 msgr2=0x7fa2b80c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 shutdown_connections 2024-03-06T07:27:06.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.086+0000 7fa2be071640 1 -- 172.21.15.94:0/3828919246 wait complete. 2024-03-06T07:27:06.099 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:06.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.196+0000 7fa20a3f7640 1 Processor -- start 2024-03-06T07:27:06.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.197+0000 7fa20a3f7640 1 -- start start 2024-03-06T07:27:06.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.197+0000 7fa20a3f7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa204159270 0x7fa204155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.197+0000 7fa20a3f7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2040c73d0 con 0x7fa2040c88c0 2024-03-06T07:27:06.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.197+0000 7fa20a3f7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2040be2e0 con 0x7fa20414f360 2024-03-06T07:27:06.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.197+0000 7fa20a3f7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa204159700 con 0x7fa204159270 2024-03-06T07:27:06.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.197+0000 7fa2037fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa2040c88c0 0x7fa204158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52428/0 (socket says 172.21.15.94:52428) 2024-03-06T07:27:06.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.197+0000 7fa2037fe640 1 -- 172.21.15.94:0/2933682441 learned_addr learned my addr 172.21.15.94:0/2933682441 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:06.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa208994640 1 --2- 172.21.15.94:0/2933682441 >> v2:172.21.15.145:3300/0 conn(0x7fa204159270 0x7fa204155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467111349 0 0) 0x7fa2040c73d0 con 0x7fa2040c88c0 2024-03-06T07:27:06.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ec003520 con 0x7fa2040c88c0 2024-03-06T07:27:06.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705419643 0 0) 0x7fa1ec003520 con 0x7fa2040c88c0 2024-03-06T07:27:06.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 >> v2:172.21.15.145:3300/0 conn(0x7fa204159270 msgr2=0x7fa204155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 --2- 172.21.15.94:0/2933682441 >> v2:172.21.15.145:3300/0 conn(0x7fa204159270 0x7fa204155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 >> v1:172.21.15.145:6789/0 conn(0x7fa20414f360 legacy=0x7fa2040c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2040c7670 con 0x7fa2040c88c0 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fa1f8001670 con 0x7fa2040c88c0 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 >> v1:172.21.15.94:6789/0 conn(0x7fa2040c88c0 legacy=0x7fa204158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0039b0 0x7fa1ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ec004800 0x7fa1ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0051c0 0x7fa1ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa204159700 con 0x7fa1ec0051c0 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec00c160 con 0x7fa1ec004800 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec00c360 con 0x7fa1ec0039b0 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa208994640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0051c0 0x7fa1ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa203fff640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0039b0 0x7fa1ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa2037fe640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ec004800 0x7fa1ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa208994640 1 -- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0039b0 msgr2=0x7fa1ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa208994640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0039b0 0x7fa1ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa208994640 1 -- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ec004800 msgr2=0x7fa1ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa208994640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1ec004800 0x7fa1ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa208994640 1 -- 172.21.15.94:0/2933682441 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec003520 con 0x7fa1ec0051c0 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.198+0000 7fa203fff640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0039b0 0x7fa1ec003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.199+0000 7fa208994640 1 -- 172.21.15.94:0/2933682441 reap_dead start 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.199+0000 7fa208994640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0051c0 0x7fa1ec00b9c0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009fc0 tx=0x7fa1ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce9fa6f76cac839 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.199+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc009c10 con 0x7fa1ec0051c0 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.199+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc02cc60 con 0x7fa1ec0051c0 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.199+0000 7fa202ffd640 1 -- 172.21.15.94:0/2933682441 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc003bd0 con 0x7fa1ec0051c0 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.200+0000 7fa20a3f7640 1 -- 172.21.15.94:0/2933682441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0051c0 msgr2=0x7fa1ec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.200+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0051c0 0x7fa1ec00b9c0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009fc0 tx=0x7fa1ec005830 comp rx=0 tx=0).stop 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.200+0000 7fa20a3f7640 1 -- 172.21.15.94:0/2933682441 shutdown_connections 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.200+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/2933682441 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0051c0 0x7fa1ec00b9c0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.200+0000 7fa20a3f7640 1 -- 172.21.15.94:0/2933682441 >> 172.21.15.94:0/2933682441 conn(0x7fa2040ac570 msgr2=0x7fa2040ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.200+0000 7fa20a3f7640 1 -- 172.21.15.94:0/2933682441 shutdown_connections 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.200+0000 7fa20a3f7640 1 -- 172.21.15.94:0/2933682441 wait complete. 2024-03-06T07:27:06.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.201+0000 7fa20a3f7640 1 Processor -- start 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.201+0000 7fa20a3f7640 1 -- start start 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0051c0 0x7fa20414fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2041eab60 0x7fa2040c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2040bf7a0 0x7fa2040afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040c7c30 con 0x7fa2041eab60 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa204155c10 con 0x7fa2040bf7a0 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa204155d90 con 0x7fa1ec0051c0 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2041eab60 0x7fa2040c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2041eab60 0x7fa2040c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47998/0 (socket says 172.21.15.94:47998) 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 -- 172.21.15.94:0/3780600520 learned_addr learned my addr 172.21.15.94:0/3780600520 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa208994640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2040bf7a0 0x7fa2040afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa203fff640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0051c0 0x7fa20414fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 -- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0051c0 msgr2=0x7fa20414fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0051c0 0x7fa20414fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 -- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2040bf7a0 msgr2=0x7fa2040afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2040bf7a0 0x7fa2040afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 -- 172.21.15.94:0/3780600520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2040be700 con 0x7fa2041eab60 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa2037fe640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2041eab60 0x7fa2040c09d0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e730 tx=0x7fa1f800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa200ff9640 1 -- 172.21.15.94:0/3780600520 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800c660 con 0x7fa2041eab60 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa200ff9640 1 -- 172.21.15.94:0/3780600520 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f8018070 con 0x7fa2041eab60 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa200ff9640 1 -- 172.21.15.94:0/3780600520 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800ca60 con 0x7fa2041eab60 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2040be9e0 con 0x7fa2041eab60 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.202+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2040befa0 con 0x7fa2041eab60 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.203+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20406d110 con 0x7fa2041eab60 2024-03-06T07:27:06.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.204+0000 7fa200ff9640 1 -- 172.21.15.94:0/3780600520 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1f800cc00 con 0x7fa2041eab60 2024-03-06T07:27:06.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.204+0000 7fa200ff9640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa1f006b500 0x7fa1f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.204+0000 7fa200ff9640 1 -- 172.21.15.94:0/3780600520 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa1f8089e60 con 0x7fa2041eab60 2024-03-06T07:27:06.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.204+0000 7fa203fff640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa1f006b500 0x7fa1f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.205+0000 7fa203fff640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa1f006b500 0x7fa1f006d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4000c00 tx=0x7fa1f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.208+0000 7fa200ff9640 1 -- 172.21.15.94:0/3780600520 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa20406d110 con 0x7fa2041eab60 2024-03-06T07:27:06.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.381+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa204159090 con 0x7fa2041eab60 2024-03-06T07:27:06.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.382+0000 7fa200ff9640 1 -- 172.21.15.94:0/3780600520 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v198) v1 ==== 74+0+17049 (secure 0 0 0) 0x7fa204159090 con 0x7fa2041eab60 2024-03-06T07:27:06.384 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:06.384 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":198,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:26:59.299653+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:23.168853+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:23.918870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"32.c","mappings":[{"from":6,"to":5}]},{"pgid":"32.33","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:06.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa1f006b500 msgr2=0x7fa1f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa1f006b500 0x7fa1f006d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4000c00 tx=0x7fa1f4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:06.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2041eab60 msgr2=0x7fa2040c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2041eab60 0x7fa2040c09d0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e730 tx=0x7fa1f800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 shutdown_connections 2024-03-06T07:27:06.386 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa1f006b500 0x7fa1f006d9c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2040bf7a0 0x7fa2040afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2041eab60 0x7fa2040c09d0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 --2- 172.21.15.94:0/3780600520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1ec0051c0 0x7fa20414fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 >> 172.21.15.94:0/3780600520 conn(0x7fa2040ac570 msgr2=0x7fa2040c98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 shutdown_connections 2024-03-06T07:27:06.387 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.384+0000 7fa20a3f7640 1 -- 172.21.15.94:0/3780600520 wait complete. 2024-03-06T07:27:06.397 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:27:06.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.496+0000 7fa23c5b6640 1 Processor -- start 2024-03-06T07:27:06.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.496+0000 7fa23c5b6640 1 -- start start 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23c5b6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa2340c69f0 0x7fa2340c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23c5b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa2340c8950 0x7fa2340acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23c5b6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa234155130 con 0x7fa2340ad2b0 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23c5b6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa234155440 con 0x7fa2340c69f0 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23c5b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2340c5d50 con 0x7fa2340c8950 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23a352640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa2340c69f0 0x7fa2340c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23a352640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa2340c69f0 0x7fa2340c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48010/0 (socket says 172.21.15.94:48010) 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23ab53640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa2340ad2b0 0x7fa2341558d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52436/0 (socket says 172.21.15.94:52436) 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.497+0000 7fa23a352640 1 -- 172.21.15.94:0/978133106 learned_addr learned my addr 172.21.15.94:0/978133106 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.498+0000 7fa239b51640 1 --2- 172.21.15.94:0/978133106 >> v2:172.21.15.145:3300/0 conn(0x7fa2340c8950 0x7fa2340acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.498+0000 7fa23a352640 1 -- 172.21.15.94:0/978133106 >> v2:172.21.15.145:3300/0 conn(0x7fa2340c8950 msgr2=0x7fa2340acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.498+0000 7fa23a352640 1 --2- 172.21.15.94:0/978133106 >> v2:172.21.15.145:3300/0 conn(0x7fa2340c8950 0x7fa2340acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.498+0000 7fa23a352640 1 -- 172.21.15.94:0/978133106 >> v1:172.21.15.94:6789/0 conn(0x7fa2340ad2b0 legacy=0x7fa2341558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.498+0000 7fa23a352640 1 -- 172.21.15.94:0/978133106 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2340c6030 con 0x7fa2340c69f0 2024-03-06T07:27:06.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.498+0000 7fa239350640 1 -- 172.21.15.94:0/978133106 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074785889 0 0) 0x7fa22c001490 con 0x7fa2340ad2b0 2024-03-06T07:27:06.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23a352640 1 --2- 172.21.15.94:0/978133106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2340c6dd0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa2340c7430 tx=0x7fa23002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1582efb639bb58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa239350640 1 -- 172.21.15.94:0/978133106 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa230035020 con 0x7fa2340c69f0 2024-03-06T07:27:06.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa239350640 1 -- 172.21.15.94:0/978133106 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa230031870 con 0x7fa2340c69f0 2024-03-06T07:27:06.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa239350640 1 -- 172.21.15.94:0/978133106 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa230031c40 con 0x7fa2340c69f0 2024-03-06T07:27:06.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 -- 172.21.15.94:0/978133106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 msgr2=0x7fa2340c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/978133106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2340c6dd0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fa2340c7430 tx=0x7fa23002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 -- 172.21.15.94:0/978133106 shutdown_connections 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/978133106 >> v2:172.21.15.145:3300/0 conn(0x7fa2340c8950 0x7fa2340acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/978133106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2340c6dd0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 -- 172.21.15.94:0/978133106 >> 172.21.15.94:0/978133106 conn(0x7fa2340cae70 msgr2=0x7fa2340cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 -- 172.21.15.94:0/978133106 shutdown_connections 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.499+0000 7fa23c5b6640 1 -- 172.21.15.94:0/978133106 wait complete. 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.500+0000 7fa23c5b6640 1 Processor -- start 2024-03-06T07:27:06.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.500+0000 7fa23c5b6640 1 -- start start 2024-03-06T07:27:06.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23c5b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2340ad2b0 0x7fa23406c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23c5b6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2341eafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23c5b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2340c8950 0x7fa2341eb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23c5b6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234152a20 con 0x7fa2340c69f0 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23c5b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234155fe0 con 0x7fa2340c8950 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23c5b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa234152ba0 con 0x7fa2340ad2b0 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2341eafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2341eafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48026/0 (socket says 172.21.15.94:48026) 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 -- 172.21.15.94:0/4049916650 learned_addr learned my addr 172.21.15.94:0/4049916650 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23a352640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2340ad2b0 0x7fa23406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23ab53640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2340c8950 0x7fa2341eb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 -- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2340ad2b0 msgr2=0x7fa23406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2340ad2b0 0x7fa23406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 -- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2340c8950 msgr2=0x7fa2341eb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2340c8950 0x7fa2341eb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa239b51640 1 -- 172.21.15.94:0/4049916650 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa234159400 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23a352640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2340ad2b0 0x7fa23406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.501+0000 7fa23ab53640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2340c8950 0x7fa2341eb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.502+0000 7fa239b51640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2341eafa0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fa22400e9b0 tx=0x7fa22400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.502+0000 7fa22affd640 1 -- 172.21.15.94:0/4049916650 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224017070 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.502+0000 7fa22affd640 1 -- 172.21.15.94:0/4049916650 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22400c9a0 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.502+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2341ee760 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.502+0000 7fa22affd640 1 -- 172.21.15.94:0/4049916650 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22400cd90 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.502+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23406c7c0 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.503+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2340c6680 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.503+0000 7fa22affd640 1 -- 172.21.15.94:0/4049916650 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa22401a070 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.504+0000 7fa22affd640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa21806b4d0 0x7fa21806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.504+0000 7fa22affd640 1 -- 172.21.15.94:0/4049916650 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa22408a370 con 0x7fa2340c69f0 2024-03-06T07:27:06.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.504+0000 7fa23a352640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa21806b4d0 0x7fa21806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.504+0000 7fa23a352640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa21806b4d0 0x7fa21806d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa230036000 tx=0x7fa230008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.508+0000 7fa22affd640 1 -- 172.21.15.94:0/4049916650 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2340c6680 con 0x7fa2340c69f0 2024-03-06T07:27:06.717 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:06.715+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa234152d20 con 0x7fa2340c69f0 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.324+0000 7fa22affd640 1 -- 172.21.15.94:0/4049916650 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v191) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa234152d20 con 0x7fa2340c69f0 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa21806b4d0 msgr2=0x7fa21806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa21806b4d0 0x7fa21806d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa230036000 tx=0x7fa230008030 comp rx=0 tx=0).stop 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 msgr2=0x7fa2341eafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2341eafa0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fa22400e9b0 tx=0x7fa22400ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 shutdown_connections 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa21806b4d0 0x7fa21806d990 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2340c8950 0x7fa2341eb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa2340c69f0 0x7fa2341eafa0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 --2- 172.21.15.94:0/4049916650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2340ad2b0 0x7fa23406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 >> 172.21.15.94:0/4049916650 conn(0x7fa2340cae70 msgr2=0x7fa234150240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 shutdown_connections 2024-03-06T07:27:07.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.327+0000 7fa23c5b6640 1 -- 172.21.15.94:0/4049916650 wait complete. 2024-03-06T07:27:07.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:27:07.444 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.443+0000 7fedf3b93640 1 Processor -- start 2024-03-06T07:27:07.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.443+0000 7fedf3b93640 1 -- start start 2024-03-06T07:27:07.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf3b93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec0c5990 0x7fedec0c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf3b93640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fedec06c900 0x7fedec0a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf3b93640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec0c3ef0 con 0x7fedec0ab170 2024-03-06T07:27:07.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf3b93640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedec1556d0 con 0x7fedec06c900 2024-03-06T07:27:07.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf3b93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec1559b0 con 0x7fedec0c5990 2024-03-06T07:27:07.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf192f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec0c5990 0x7fedec0c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf2130640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fedec06c900 0x7fedec0a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf192f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec0c5990 0x7fedec0c0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:35764/0 (socket says 172.21.15.94:35764) 2024-03-06T07:27:07.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf192f640 1 -- 172.21.15.94:0/400456391 learned_addr learned my addr 172.21.15.94:0/400456391 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:07.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf192f640 1 -- 172.21.15.94:0/400456391 >> v2:172.21.15.145:3300/0 conn(0x7fedec06c900 msgr2=0x7fedec0a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf192f640 1 --2- 172.21.15.94:0/400456391 >> v2:172.21.15.145:3300/0 conn(0x7fedec06c900 0x7fedec0a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf192f640 1 -- 172.21.15.94:0/400456391 >> v1:172.21.15.145:6789/0 conn(0x7fedec0ab170 legacy=0x7fedec06c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf192f640 1 -- 172.21.15.94:0/400456391 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec155c90 con 0x7fedec0c5990 2024-03-06T07:27:07.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.444+0000 7fedf2130640 1 --2- 172.21.15.94:0/400456391 >> v2:172.21.15.145:3300/0 conn(0x7fedec06c900 0x7fedec0a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:07.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.445+0000 7fedf192f640 1 --2- 172.21.15.94:0/400456391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec0c5990 0x7fedec0c0ab0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fede8009870 tx=0x7fede802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ed46796da6a00fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.445+0000 7fedf092d640 1 -- 172.21.15.94:0/400456391 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8034070 con 0x7fedec0c5990 2024-03-06T07:27:07.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.445+0000 7fedf092d640 1 -- 172.21.15.94:0/400456391 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8030840 con 0x7fedec0c5990 2024-03-06T07:27:07.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.445+0000 7fedf092d640 1 -- 172.21.15.94:0/400456391 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8030c10 con 0x7fedec0c5990 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 -- 172.21.15.94:0/400456391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec0c5990 msgr2=0x7fedec0c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 --2- 172.21.15.94:0/400456391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec0c5990 0x7fedec0c0ab0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fede8009870 tx=0x7fede802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 -- 172.21.15.94:0/400456391 shutdown_connections 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 --2- 172.21.15.94:0/400456391 >> v2:172.21.15.145:3300/0 conn(0x7fedec06c900 0x7fedec0a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 --2- 172.21.15.94:0/400456391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec0c5990 0x7fedec0c0ab0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 -- 172.21.15.94:0/400456391 >> 172.21.15.94:0/400456391 conn(0x7fedec0c6710 msgr2=0x7fedec0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 -- 172.21.15.94:0/400456391 shutdown_connections 2024-03-06T07:27:07.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.446+0000 7fedf3b93640 1 -- 172.21.15.94:0/400456391 wait complete. 2024-03-06T07:27:07.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.447+0000 7fedf3b93640 1 Processor -- start 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.447+0000 7fedf3b93640 1 -- start start 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf3b93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec06c900 0x7fedec154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf3b93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedec0ab170 0x7fedec147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf3b93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedec0c5990 0x7fedec147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf3b93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec156610 con 0x7fedec0c5990 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf3b93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec156490 con 0x7fedec0ab170 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf3b93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec0c3af0 con 0x7fedec06c900 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedec0c5990 0x7fedec147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedec0c5990 0x7fedec147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48038/0 (socket says 172.21.15.94:48038) 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 -- 172.21.15.94:0/3320369006 learned_addr learned my addr 172.21.15.94:0/3320369006 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf192f640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec06c900 0x7fedec154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf112e640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedec0ab170 0x7fedec147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 -- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec06c900 msgr2=0x7fedec154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec06c900 0x7fedec154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 -- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedec0ab170 msgr2=0x7fedec147860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedec0ab170 0x7fedec147860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 -- 172.21.15.94:0/3320369006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec1ef830 con 0x7fedec0c5990 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fedf2130640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedec0c5990 0x7fedec147da0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fede4009e10 tx=0x7fede400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fede27fc640 1 -- 172.21.15.94:0/3320369006 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4018020 con 0x7fedec0c5990 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.448+0000 7fede27fc640 1 -- 172.21.15.94:0/3320369006 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede4014480 con 0x7fedec0c5990 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.449+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec0705a0 con 0x7fedec0c5990 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.449+0000 7fede27fc640 1 -- 172.21.15.94:0/3320369006 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4014850 con 0x7fedec0c5990 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.449+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec071ff0 con 0x7fedec0c5990 2024-03-06T07:27:07.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.449+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedec0c5d70 con 0x7fedec0c5990 2024-03-06T07:27:07.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.450+0000 7fede27fc640 1 -- 172.21.15.94:0/3320369006 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fede4005d10 con 0x7fedec0c5990 2024-03-06T07:27:07.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.450+0000 7fede27fc640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.450+0000 7fede27fc640 1 -- 172.21.15.94:0/3320369006 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fede408b060 con 0x7fedec0c5990 2024-03-06T07:27:07.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.451+0000 7fedf192f640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.451+0000 7fedf192f640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fedd006b4d0 0x7fedd006d990 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fede8036000 tx=0x7fede803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.455+0000 7fede27fc640 1 -- 172.21.15.94:0/3320369006 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedec0c5d70 con 0x7fedec0c5990 2024-03-06T07:27:07.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:07.663+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fedec0735d0 con 0x7fedec0c5990 2024-03-06T07:27:08.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.326+0000 7fede27fc640 1 -- 172.21.15.94:0/3320369006 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v192) v1 ==== 104+0+0 (secure 0 0 0) 0x7fedec0735d0 con 0x7fedec0c5990 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fedd006b4d0 msgr2=0x7fedd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fedd006b4d0 0x7fedd006d990 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fede8036000 tx=0x7fede803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedec0c5990 msgr2=0x7fedec147da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedec0c5990 0x7fedec147da0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fede4009e10 tx=0x7fede400c490 comp rx=0 tx=0).stop 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 shutdown_connections 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fedec0c5990 0x7fedec147da0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedec0ab170 0x7fedec147860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 --2- 172.21.15.94:0/3320369006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedec06c900 0x7fedec154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 >> 172.21.15.94:0/3320369006 conn(0x7fedec0c6710 msgr2=0x7fedec14c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 shutdown_connections 2024-03-06T07:27:08.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.328+0000 7fedf3b93640 1 -- 172.21.15.94:0/3320369006 wait complete. 2024-03-06T07:27:08.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:27:08.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.449+0000 7fdb73cb5640 1 Processor -- start 2024-03-06T07:27:08.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.449+0000 7fdb73cb5640 1 -- start start 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.449+0000 7fdb73cb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb6c152920 0x7fdb6c155970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb73cb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdb6c0c2900 0x7fdb6c15fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb73cb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c159740 con 0x7fdb6c155eb0 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb73cb5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c1551d0 con 0x7fdb6c0c2900 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb73cb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c0c8ab0 con 0x7fdb6c152920 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb72252640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdb6c0c2900 0x7fdb6c15fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb72252640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdb6c0c2900 0x7fdb6c15fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54734/0 (socket says 172.21.15.94:54734) 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb72252640 1 -- 172.21.15.94:0/389227987 learned_addr learned my addr 172.21.15.94:0/389227987 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb72252640 1 -- 172.21.15.94:0/389227987 >> v2:172.21.15.145:3300/0 conn(0x7fdb6c152920 msgr2=0x7fdb6c155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb72252640 1 --2- 172.21.15.94:0/389227987 >> v2:172.21.15.145:3300/0 conn(0x7fdb6c152920 0x7fdb6c155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb72252640 1 -- 172.21.15.94:0/389227987 >> v1:172.21.15.145:6789/0 conn(0x7fdb6c155eb0 legacy=0x7fdb6c0c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.450+0000 7fdb72252640 1 -- 172.21.15.94:0/389227987 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c0c8d90 con 0x7fdb6c0c2900 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb72252640 1 --2- 172.21.15.94:0/389227987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c15fb50 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c073610 tx=0x7fdb6402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3b1c2ade82f2ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb70a4f640 1 -- 172.21.15.94:0/389227987 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6402c9f0 con 0x7fdb6c0c2900 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb70a4f640 1 -- 172.21.15.94:0/389227987 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6402cb90 con 0x7fdb6c0c2900 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb70a4f640 1 -- 172.21.15.94:0/389227987 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb640343f0 con 0x7fdb6c0c2900 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 -- 172.21.15.94:0/389227987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 msgr2=0x7fdb6c15fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/389227987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c15fb50 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c073610 tx=0x7fdb6402be50 comp rx=0 tx=0).stop 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 -- 172.21.15.94:0/389227987 shutdown_connections 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/389227987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c15fb50 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/389227987 >> v2:172.21.15.145:3300/0 conn(0x7fdb6c152920 0x7fdb6c155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 -- 172.21.15.94:0/389227987 >> 172.21.15.94:0/389227987 conn(0x7fdb6c0aa5c0 msgr2=0x7fdb6c0aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 -- 172.21.15.94:0/389227987 shutdown_connections 2024-03-06T07:27:08.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.451+0000 7fdb73cb5640 1 -- 172.21.15.94:0/389227987 wait complete. 2024-03-06T07:27:08.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.452+0000 7fdb73cb5640 1 Processor -- start 2024-03-06T07:27:08.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.452+0000 7fdb73cb5640 1 -- start start 2024-03-06T07:27:08.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb73cb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c06d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb73cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb6c152920 0x7fdb6c1ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb73cb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb6c155eb0 0x7fdb6c1de830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb73cb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0c82a0 con 0x7fdb6c0c2900 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb73cb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c155450 con 0x7fdb6c155eb0 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb73cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0c8420 con 0x7fdb6c152920 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb71a51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c06d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb71a51640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c06d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54736/0 (socket says 172.21.15.94:54736) 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb71a51640 1 -- 172.21.15.94:0/4227119666 learned_addr learned my addr 172.21.15.94:0/4227119666 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb72252640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb6c155eb0 0x7fdb6c1de830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.453+0000 7fdb71250640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb6c152920 0x7fdb6c1ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb71a51640 1 -- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb6c152920 msgr2=0x7fdb6c1ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb71a51640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb6c152920 0x7fdb6c1ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb71a51640 1 -- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb6c155eb0 msgr2=0x7fdb6c1de830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb71a51640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb6c155eb0 0x7fdb6c1de830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb71a51640 1 -- 172.21.15.94:0/4227119666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c1519f0 con 0x7fdb6c0c2900 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb71250640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb6c152920 0x7fdb6c1ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb71a51640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c06d270 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800b4a0 tx=0x7fdb6800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb627fc640 1 -- 172.21.15.94:0/4227119666 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68014070 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb627fc640 1 -- 172.21.15.94:0/4227119666 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb68010610 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c06d7b0 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb627fc640 1 -- 172.21.15.94:0/4227119666 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68010a00 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.454+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c0a9510 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.455+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c04f330 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.455+0000 7fdb627fc640 1 -- 172.21.15.94:0/4227119666 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb68017070 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.456+0000 7fdb627fc640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb5006b4d0 0x7fdb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.456+0000 7fdb627fc640 1 -- 172.21.15.94:0/4227119666 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fdb68089c80 con 0x7fdb6c0c2900 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.456+0000 7fdb71250640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb5006b4d0 0x7fdb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.456+0000 7fdb71250640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb5006b4d0 0x7fdb5006d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c006fd0 tx=0x7fdb5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.460+0000 7fdb627fc640 1 -- 172.21.15.94:0/4227119666 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb68058380 con 0x7fdb6c0c2900 2024-03-06T07:27:08.635 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:08.633+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdb6c0c1fd0 con 0x7fdb6c0c2900 2024-03-06T07:27:09.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.339+0000 7fdb627fc640 1 -- 172.21.15.94:0/4227119666 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v199) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdb6805c420 con 0x7fdb6c0c2900 2024-03-06T07:27:09.342 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb5006b4d0 msgr2=0x7fdb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb5006b4d0 0x7fdb5006d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c006fd0 tx=0x7fdb5c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 msgr2=0x7fdb6c06d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c06d270 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800b4a0 tx=0x7fdb6800b960 comp rx=0 tx=0).stop 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 shutdown_connections 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdb5006b4d0 0x7fdb5006d990 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb6c155eb0 0x7fdb6c1de830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb6c152920 0x7fdb6c1ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 --2- 172.21.15.94:0/4227119666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c0c2900 0x7fdb6c06d270 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 >> 172.21.15.94:0/4227119666 conn(0x7fdb6c0aa5c0 msgr2=0x7fdb6c0aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.343+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 shutdown_connections 2024-03-06T07:27:09.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.344+0000 7fdb73cb5640 1 -- 172.21.15.94:0/4227119666 wait complete. 2024-03-06T07:27:09.358 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:27:09.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.465+0000 7efc7a09d640 1 Processor -- start 2024-03-06T07:27:09.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.465+0000 7efc7a09d640 1 -- start start 2024-03-06T07:27:09.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc7a09d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efc741527a0 0x7efc74155040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc7a09d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc740c1860 0x7efc7406bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc7a09d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc74155580 con 0x7efc7406c510 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc7a09d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc74158820 con 0x7efc741527a0 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc7a09d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc740c2b10 con 0x7efc740c1860 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efc741527a0 0x7efc74155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc737fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7efc741527a0 0x7efc74155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54746/0 (socket says 172.21.15.94:54746) 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc73fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efc7406c510 0x7efc74158cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37806/0 (socket says 172.21.15.94:37806) 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc737fe640 1 -- 172.21.15.94:0/1056738510 learned_addr learned my addr 172.21.15.94:0/1056738510 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc737fe640 1 -- 172.21.15.94:0/1056738510 >> v2:172.21.15.145:3300/0 conn(0x7efc740c1860 msgr2=0x7efc7406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc737fe640 1 --2- 172.21.15.94:0/1056738510 >> v2:172.21.15.145:3300/0 conn(0x7efc740c1860 0x7efc7406bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc737fe640 1 -- 172.21.15.94:0/1056738510 >> v1:172.21.15.94:6789/0 conn(0x7efc7406c510 legacy=0x7efc74158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc727fc640 1 -- 172.21.15.94:0/1056738510 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669371007 0 0) 0x7efc74155580 con 0x7efc7406c510 2024-03-06T07:27:09.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.466+0000 7efc737fe640 1 -- 172.21.15.94:0/1056738510 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74155820 con 0x7efc741527a0 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.467+0000 7efc737fe640 1 --2- 172.21.15.94:0/1056738510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741527a0 0x7efc74155040 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7efc7406f1e0 tx=0x7efc6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a34b4e9ebd15c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.467+0000 7efc727fc640 1 -- 172.21.15.94:0/1056738510 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64035020 con 0x7efc741527a0 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.467+0000 7efc727fc640 1 -- 172.21.15.94:0/1056738510 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc64031870 con 0x7efc741527a0 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.467+0000 7efc727fc640 1 -- 172.21.15.94:0/1056738510 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64031c40 con 0x7efc741527a0 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 -- 172.21.15.94:0/1056738510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741527a0 msgr2=0x7efc74155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 --2- 172.21.15.94:0/1056738510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741527a0 0x7efc74155040 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7efc7406f1e0 tx=0x7efc6402b910 comp rx=0 tx=0).stop 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 -- 172.21.15.94:0/1056738510 shutdown_connections 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 --2- 172.21.15.94:0/1056738510 >> v2:172.21.15.145:3300/0 conn(0x7efc740c1860 0x7efc7406bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 --2- 172.21.15.94:0/1056738510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc741527a0 0x7efc74155040 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 -- 172.21.15.94:0/1056738510 >> 172.21.15.94:0/1056738510 conn(0x7efc740c3960 msgr2=0x7efc740c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 -- 172.21.15.94:0/1056738510 shutdown_connections 2024-03-06T07:27:09.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.468+0000 7efc7a09d640 1 -- 172.21.15.94:0/1056738510 wait complete. 2024-03-06T07:27:09.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.469+0000 7efc7a09d640 1 Processor -- start 2024-03-06T07:27:09.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.469+0000 7efc7a09d640 1 -- start start 2024-03-06T07:27:09.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc7a09d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc7406c510 0x7efc740c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc7a09d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c1860 0x7efc741425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc7a09d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc741527a0 0x7efc74142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc7a09d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74152b80 con 0x7efc7406c510 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc7a09d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74155de0 con 0x7efc740c1860 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc7a09d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc74152d00 con 0x7efc741527a0 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc7406c510 0x7efc740c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc7406c510 0x7efc740c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54756/0 (socket says 172.21.15.94:54756) 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 -- 172.21.15.94:0/4158277098 learned_addr learned my addr 172.21.15.94:0/4158277098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc73fff640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc741527a0 0x7efc74142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc72ffd640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c1860 0x7efc741425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 -- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc741527a0 msgr2=0x7efc74142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc741527a0 0x7efc74142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 -- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c1860 msgr2=0x7efc741425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c1860 0x7efc741425e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 -- 172.21.15.94:0/4158277098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7414d320 con 0x7efc7406c510 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.470+0000 7efc737fe640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc7406c510 0x7efc740c5c70 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7efc64000c00 tx=0x7efc640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.471+0000 7efc5bfff640 1 -- 172.21.15.94:0/4158277098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6403ad60 con 0x7efc7406c510 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.471+0000 7efc5bfff640 1 -- 172.21.15.94:0/4158277098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc640443f0 con 0x7efc7406c510 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.471+0000 7efc5bfff640 1 -- 172.21.15.94:0/4158277098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640447e0 con 0x7efc7406c510 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.471+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc741eaa00 con 0x7efc7406c510 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.471+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc741eafa0 con 0x7efc7406c510 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.472+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7406c1f0 con 0x7efc7406c510 2024-03-06T07:27:09.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.472+0000 7efc5bfff640 1 -- 172.21.15.94:0/4158277098 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc64038020 con 0x7efc7406c510 2024-03-06T07:27:09.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.472+0000 7efc5bfff640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.473+0000 7efc5bfff640 1 -- 172.21.15.94:0/4158277098 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5721+0+0 (secure 0 0 0) 0x7efc640ac3e0 con 0x7efc7406c510 2024-03-06T07:27:09.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.473+0000 7efc72ffd640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.473+0000 7efc72ffd640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc6006b4d0 0x7efc6006d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7efc74155340 tx=0x7efc68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.477+0000 7efc5bfff640 1 -- 172.21.15.94:0/4158277098 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc7406c1f0 con 0x7efc7406c510 2024-03-06T07:27:09.650 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:09.648+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efc740ab150 con 0x7efc7406c510 2024-03-06T07:27:10.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.341+0000 7efc5bfff640 1 -- 172.21.15.94:0/4158277098 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v200) v1 ==== 169+0+0 (secure 0 0 0) 0x7efc740ab150 con 0x7efc7406c510 2024-03-06T07:27:10.343 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:27:10.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc6006b4d0 msgr2=0x7efc6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc6006b4d0 0x7efc6006d990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7efc74155340 tx=0x7efc68008040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc7406c510 msgr2=0x7efc740c5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc7406c510 0x7efc740c5c70 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7efc64000c00 tx=0x7efc640046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 shutdown_connections 2024-03-06T07:27:10.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc741527a0 0x7efc74142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc740c1860 0x7efc741425e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 --2- 172.21.15.94:0/4158277098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efc7406c510 0x7efc740c5c70 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 >> 172.21.15.94:0/4158277098 conn(0x7efc740c3960 msgr2=0x7efc7406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 shutdown_connections 2024-03-06T07:27:10.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.345+0000 7efc7a09d640 1 -- 172.21.15.94:0/4158277098 wait complete. 2024-03-06T07:27:10.362 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:27:10.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.468+0000 7f6ae92b5640 1 Processor -- start 2024-03-06T07:27:10.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.468+0000 7f6ae92b5640 1 -- start start 2024-03-06T07:27:10.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6ae92b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6ae41565b0 0x7f6ae4155b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6ae92b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae40c7e00 con 0x7f6ae40b01e0 2024-03-06T07:27:10.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6ae92b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae40c7840 con 0x7f6ae406c170 2024-03-06T07:27:10.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6ae92b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae40c7b20 con 0x7f6ae41565b0 2024-03-06T07:27:10.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6adbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6ae41565b0 0x7f6ae4155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6adbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6ae41565b0 0x7f6ae4155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54766/0 (socket says 172.21.15.94:54766) 2024-03-06T07:27:10.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6ae2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6ae40b01e0 0x7f6ae40c53d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37808/0 (socket says 172.21.15.94:37808) 2024-03-06T07:27:10.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6adbfff640 1 -- 172.21.15.94:0/2298866317 learned_addr learned my addr 172.21.15.94:0/2298866317 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:10.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6adbfff640 1 -- 172.21.15.94:0/2298866317 >> v1:172.21.15.145:6789/0 conn(0x7f6ae406c170 legacy=0x7f6ae415fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6adbfff640 1 -- 172.21.15.94:0/2298866317 >> v1:172.21.15.94:6789/0 conn(0x7f6ae40b01e0 legacy=0x7f6ae40c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.469+0000 7f6adbfff640 1 -- 172.21.15.94:0/2298866317 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae406b970 con 0x7f6ae41565b0 2024-03-06T07:27:10.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6adbfff640 1 --2- 172.21.15.94:0/2298866317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae41565b0 0x7f6ae4155b10 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009870 tx=0x7f6acc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27c7ac94ad4659d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae27fc640 1 -- 172.21.15.94:0/2298866317 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc034070 con 0x7f6ae41565b0 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae27fc640 1 -- 172.21.15.94:0/2298866317 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc030860 con 0x7f6ae41565b0 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae27fc640 1 -- 172.21.15.94:0/2298866317 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc030c30 con 0x7f6ae41565b0 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae92b5640 1 -- 172.21.15.94:0/2298866317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae41565b0 msgr2=0x7f6ae4155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/2298866317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae41565b0 0x7f6ae4155b10 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009870 tx=0x7f6acc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae92b5640 1 -- 172.21.15.94:0/2298866317 shutdown_connections 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/2298866317 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae41565b0 0x7f6ae4155b10 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.470+0000 7f6ae92b5640 1 -- 172.21.15.94:0/2298866317 >> 172.21.15.94:0/2298866317 conn(0x7f6ae40ac600 msgr2=0x7f6ae40aca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.471+0000 7f6ae92b5640 1 -- 172.21.15.94:0/2298866317 shutdown_connections 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.471+0000 7f6ae92b5640 1 -- 172.21.15.94:0/2298866317 wait complete. 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 Processor -- start 2024-03-06T07:27:10.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 -- start start 2024-03-06T07:27:10.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae406c170 0x7f6ae40c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ae40b01e0 0x7f6ae40c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ae41565b0 0x7f6ae41eaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40c5c60 con 0x7f6ae406c170 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40c5ae0 con 0x7f6ae40b01e0 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae92b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40be250 con 0x7f6ae41565b0 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae406c170 0x7f6ae40c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae406c170 0x7f6ae40c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54780/0 (socket says 172.21.15.94:54780) 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.472+0000 7f6ae2ffd640 1 -- 172.21.15.94:0/1254218861 learned_addr learned my addr 172.21.15.94:0/1254218861 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6adbfff640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ae40b01e0 0x7f6ae40c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae2ffd640 1 -- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ae41565b0 msgr2=0x7f6ae41eaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae37fe640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ae41565b0 0x7f6ae41eaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae2ffd640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ae41565b0 0x7f6ae41eaec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae2ffd640 1 -- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ae40b01e0 msgr2=0x7f6ae40c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae2ffd640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ae40b01e0 0x7f6ae40c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae2ffd640 1 -- 172.21.15.94:0/1254218861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae4155390 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6adbfff640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ae40b01e0 0x7f6ae40c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae37fe640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ae41565b0 0x7f6ae41eaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae2ffd640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae406c170 0x7f6ae40c8e70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400c430 tx=0x7f6ad400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6adb7fe640 1 -- 172.21.15.94:0/1254218861 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad40074e0 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6adb7fe640 1 -- 172.21.15.94:0/1254218861 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad4007680 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae40c88f0 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6adb7fe640 1 -- 172.21.15.94:0/1254218861 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4004a00 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.473+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae40c9cd0 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.474+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae404f330 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.474+0000 7f6adb7fe640 1 -- 172.21.15.94:0/1254218861 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ad401a020 con 0x7f6ae406c170 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.475+0000 7f6adb7fe640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ac406b4d0 0x7f6ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.475+0000 7f6adb7fe640 1 -- 172.21.15.94:0/1254218861 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6ad4089850 con 0x7f6ae406c170 2024-03-06T07:27:10.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.475+0000 7f6adbfff640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ac406b4d0 0x7f6ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.476+0000 7f6adbfff640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ac406b4d0 0x7f6ac406d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6acc036000 tx=0x7f6acc03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.479+0000 7f6adb7fe640 1 -- 172.21.15.94:0/1254218861 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ad4058890 con 0x7f6ae406c170 2024-03-06T07:27:10.657 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.655+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6ae40c4e80 con 0x7f6ae406c170 2024-03-06T07:27:10.660 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.658+0000 7f6adb7fe640 1 -- 172.21.15.94:0/1254218861 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ad4058190 con 0x7f6ae406c170 2024-03-06T07:27:10.662 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ac406b4d0 msgr2=0x7f6ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ac406b4d0 0x7f6ac406d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f6acc036000 tx=0x7f6acc03b040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae406c170 msgr2=0x7f6ae40c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae406c170 0x7f6ae40c8e70 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400c430 tx=0x7f6ad400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 shutdown_connections 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6ac406b4d0 0x7f6ac406d990 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ae41565b0 0x7f6ae41eaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ae40b01e0 0x7f6ae40c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 --2- 172.21.15.94:0/1254218861 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ae406c170 0x7f6ae40c8e70 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.660+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 >> 172.21.15.94:0/1254218861 conn(0x7f6ae40ac600 msgr2=0x7f6ae40c52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.661+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 shutdown_connections 2024-03-06T07:27:10.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.661+0000 7f6ae92b5640 1 -- 172.21.15.94:0/1254218861 wait complete. 2024-03-06T07:27:10.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-03-06T07:27:10.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.772+0000 7fcba5a0e640 1 Processor -- start 2024-03-06T07:27:10.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.772+0000 7fcba5a0e640 1 -- start start 2024-03-06T07:27:10.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcba5a0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 0x7fcba00c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcba5a0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba00c7c60 con 0x7fcba00c7180 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcba5a0e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba00c8060 con 0x7fcba01527b0 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcba5a0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c8df0 con 0x7fcba0159100 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9e7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcba00c7180 0x7fcba00c5330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37822/0 (socket says 172.21.15.94:37822) 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9e7fc640 1 -- 172.21.15.94:0/3588959701 learned_addr learned my addr 172.21.15.94:0/3588959701 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9f7fe640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 0x7fcba00c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194417957 0 0) 0x7fcb94001110 con 0x7fcba00c7180 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb88003520 con 0x7fcba00c7180 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002954574 0 0) 0x7fcb88003520 con 0x7fcba00c7180 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 msgr2=0x7fcba00c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 0x7fcba00c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 >> v1:172.21.15.145:6789/0 conn(0x7fcba01527b0 legacy=0x7fcba0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba01595b0 con 0x7fcba00c7180 2024-03-06T07:27:10.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.773+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fcb94001840 con 0x7fcba00c7180 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 >> v1:172.21.15.94:6789/0 conn(0x7fcba00c7180 legacy=0x7fcba00c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9dffb640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcb88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9dffb640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 0x7fcb88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9dffb640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb88004d40 0x7fcb880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c8df0 con 0x7fcb880039e0 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8800ba70 con 0x7fcb88004d40 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8800bc70 con 0x7fcba0159100 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9effd640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcb88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9e7fc640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 0x7fcb88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9f7fe640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb88004d40 0x7fcb880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9effd640 1 -- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 msgr2=0x7fcb88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9effd640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 0x7fcb88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9effd640 1 -- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb88004d40 msgr2=0x7fcb880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9effd640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb88004d40 0x7fcb880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9effd640 1 -- 172.21.15.94:0/3588959701 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb88003520 con 0x7fcb880039e0 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9e7fc640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0159100 0x7fcb88004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.774+0000 7fcb9f7fe640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb88004d40 0x7fcb880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcb9effd640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcb88003dc0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fcb90001860 tx=0x7fcb88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76fa558dfe262755 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9002f050 con 0x7fcb880039e0 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb90033650 con 0x7fcb880039e0 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcb9dffb640 1 -- 172.21.15.94:0/3588959701 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90001430 con 0x7fcb880039e0 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcba5a0e640 1 -- 172.21.15.94:0/3588959701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 msgr2=0x7fcb88003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcba5a0e640 1 --2- 172.21.15.94:0/3588959701 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcb88003dc0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fcb90001860 tx=0x7fcb88005830 comp rx=0 tx=0).stop 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcb9f7fe640 1 -- 172.21.15.94:0/3588959701 reap_dead start 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcba5a0e640 1 -- 172.21.15.94:0/3588959701 shutdown_connections 2024-03-06T07:27:10.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcba5a0e640 1 -- 172.21.15.94:0/3588959701 >> 172.21.15.94:0/3588959701 conn(0x7fcba00ac560 msgr2=0x7fcba00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcba5a0e640 1 -- 172.21.15.94:0/3588959701 shutdown_connections 2024-03-06T07:27:10.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.775+0000 7fcba5a0e640 1 -- 172.21.15.94:0/3588959701 wait complete. 2024-03-06T07:27:10.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.776+0000 7fcba5a0e640 1 Processor -- start 2024-03-06T07:27:10.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcba5a0e640 1 -- start start 2024-03-06T07:27:10.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcba5a0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcba0154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcba5a0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0158680 0x7fcba014b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcba5a0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba014c100 0x7fcba014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcba5a0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba0155320 con 0x7fcb880039e0 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcba5a0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c7980 con 0x7fcba014c100 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcba5a0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba0155600 con 0x7fcba0158680 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcb9effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcba0154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcb9effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcba0154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54804/0 (socket says 172.21.15.94:54804) 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.777+0000 7fcb9effd640 1 -- 172.21.15.94:0/2473538785 learned_addr learned my addr 172.21.15.94:0/2473538785 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9e7fc640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0158680 0x7fcba014b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9f7fe640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba014c100 0x7fcba014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9effd640 1 -- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0158680 msgr2=0x7fcba014b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9effd640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0158680 0x7fcba014b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9effd640 1 -- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba014c100 msgr2=0x7fcba014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9effd640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba014c100 0x7fcba014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9effd640 1 -- 172.21.15.94:0/2473538785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0154d10 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9e7fc640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0158680 0x7fcba014b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb9effd640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcba0154270 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fcb9002bdf0 tx=0x7fcb90001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb87fff640 1 -- 172.21.15.94:0/2473538785 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb900035f0 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb87fff640 1 -- 172.21.15.94:0/2473538785 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb90003790 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba014be20 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcb87fff640 1 -- 172.21.15.94:0/2473538785 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90003b40 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.778+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba01eb9d0 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.779+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba01ebbd0 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.779+0000 7fcb87fff640 1 -- 172.21.15.94:0/2473538785 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb90003ce0 con 0x7fcb880039e0 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.780+0000 7fcb87fff640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8c06b500 0x7fcb8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.780+0000 7fcb87fff640 1 -- 172.21.15.94:0/2473538785 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcb900abfa0 con 0x7fcb880039e0 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.780+0000 7fcb9e7fc640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8c06b500 0x7fcb8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.781+0000 7fcb9e7fc640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8c06b500 0x7fcb8c06d9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fcb940013d0 tx=0x7fcb94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:10.784+0000 7fcb87fff640 1 -- 172.21.15.94:0/2473538785 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb9004a060 con 0x7fcb880039e0 2024-03-06T07:27:11.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.026+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7fcba004f330 con 0x7fcb880039e0 2024-03-06T07:27:11.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.348+0000 7fcb87fff640 1 -- 172.21.15.94:0/2473538785 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v503) v1 ==== 140+0+0 (secure 0 0 0) 0x7fcb90041030 con 0x7fcb880039e0 2024-03-06T07:27:11.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.350+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8c06b500 msgr2=0x7fcb8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.350+0000 7fcba5a0e640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8c06b500 0x7fcb8c06d9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fcb940013d0 tx=0x7fcb94009040 comp rx=0 tx=0).stop 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.350+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 msgr2=0x7fcba0154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.350+0000 7fcba5a0e640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcba0154270 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fcb9002bdf0 tx=0x7fcb90001230 comp rx=0 tx=0).stop 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 shutdown_connections 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcb8c06b500 0x7fcb8c06d9c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcba014c100 0x7fcba014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcba0158680 0x7fcba014b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 --2- 172.21.15.94:0/2473538785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb880039e0 0x7fcba0154270 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 >> 172.21.15.94:0/2473538785 conn(0x7fcba00ac560 msgr2=0x7fcba014ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 shutdown_connections 2024-03-06T07:27:11.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.351+0000 7fcba5a0e640 1 -- 172.21.15.94:0/2473538785 wait complete. 2024-03-06T07:27:11.364 INFO:tasks.cephfs_test_runner:That marking a FS down does not generate a health warning ... ok 2024-03-06T07:27:11.364 INFO:tasks.cephfs_test_runner:Starting test: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down twice does not wipe old_max_mds. 2024-03-06T07:27:11.364 INFO:tasks.cephfs_test_runner:test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:27:11.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe8364f4640 1 Processor -- start 2024-03-06T07:27:11.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe8364f4640 1 -- start start 2024-03-06T07:27:11.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe8364f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83014f790 0x7fe8301529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe8364f4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe830155a40 0x7fe830155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe8364f4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe830156410 con 0x7fe8300c1aa0 2024-03-06T07:27:11.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe8364f4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8300c2680 con 0x7fe830155a40 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe8364f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c28d0 con 0x7fe83014f790 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.463+0000 7fe834a91640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe830155a40 0x7fe830155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.464+0000 7fe834a91640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe830155a40 0x7fe830155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54818/0 (socket says 172.21.15.94:54818) 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.464+0000 7fe834a91640 1 -- 172.21.15.94:0/2620275852 learned_addr learned my addr 172.21.15.94:0/2620275852 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.464+0000 7fe82ffff640 1 --2- 172.21.15.94:0/2620275852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83014f790 0x7fe8301529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.464+0000 7fe834a91640 1 -- 172.21.15.94:0/2620275852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83014f790 msgr2=0x7fe8301529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.464+0000 7fe834a91640 1 --2- 172.21.15.94:0/2620275852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83014f790 0x7fe8301529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.464+0000 7fe834a91640 1 -- 172.21.15.94:0/2620275852 >> v1:172.21.15.145:6789/0 conn(0x7fe8300c1aa0 legacy=0x7fe8300c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.464+0000 7fe834a91640 1 -- 172.21.15.94:0/2620275852 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8300c2bb0 con 0x7fe830155a40 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe834a91640 1 --2- 172.21.15.94:0/2620275852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe830155a40 0x7fe830155ed0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fe82800d150 tx=0x7fe82802e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9488a848ddc92a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe82effd640 1 -- 172.21.15.94:0/2620275852 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82800b0d0 con 0x7fe830155a40 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe82effd640 1 -- 172.21.15.94:0/2620275852 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82800b270 con 0x7fe830155a40 2024-03-06T07:27:11.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe82effd640 1 -- 172.21.15.94:0/2620275852 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe828009600 con 0x7fe830155a40 2024-03-06T07:27:11.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 -- 172.21.15.94:0/2620275852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe830155a40 msgr2=0x7fe830155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2620275852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe830155a40 0x7fe830155ed0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fe82800d150 tx=0x7fe82802e160 comp rx=0 tx=0).stop 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 -- 172.21.15.94:0/2620275852 shutdown_connections 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2620275852 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe830155a40 0x7fe830155ed0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2620275852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83014f790 0x7fe8301529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 -- 172.21.15.94:0/2620275852 >> 172.21.15.94:0/2620275852 conn(0x7fe8300c6710 msgr2=0x7fe8300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 -- 172.21.15.94:0/2620275852 shutdown_connections 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.465+0000 7fe8364f4640 1 -- 172.21.15.94:0/2620275852 wait complete. 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.466+0000 7fe8364f4640 1 Processor -- start 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe8364f4640 1 -- start start 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe8364f4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8300c1aa0 0x7fe8301e7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe8364f4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe83014f790 0x7fe830151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe8364f4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe830155a40 0x7fe830071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe8364f4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c0d60 con 0x7fe83014f790 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe8364f4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830155520 con 0x7fe830155a40 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe8364f4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c5e30 con 0x7fe8300c1aa0 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe82f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe83014f790 0x7fe830151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe82f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe83014f790 0x7fe830151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54830/0 (socket says 172.21.15.94:54830) 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe82f7fe640 1 -- 172.21.15.94:0/2268269666 learned_addr learned my addr 172.21.15.94:0/2268269666 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe82ffff640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8300c1aa0 0x7fe8301e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.467+0000 7fe834a91640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe830155a40 0x7fe830071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82f7fe640 1 -- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8300c1aa0 msgr2=0x7fe8301e7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82f7fe640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8300c1aa0 0x7fe8301e7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82f7fe640 1 -- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe830155a40 msgr2=0x7fe830071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82f7fe640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe830155a40 0x7fe830071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82f7fe640 1 -- 172.21.15.94:0/2268269666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe830150cf0 con 0x7fe83014f790 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82f7fe640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe83014f790 0x7fe830151a50 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fe82000c430 tx=0x7fe82000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82cff9640 1 -- 172.21.15.94:0/2268269666 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200052d0 con 0x7fe83014f790 2024-03-06T07:27:11.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82cff9640 1 -- 172.21.15.94:0/2268269666 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe820005490 con 0x7fe83014f790 2024-03-06T07:27:11.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe830150fd0 con 0x7fe83014f790 2024-03-06T07:27:11.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe82cff9640 1 -- 172.21.15.94:0/2268269666 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820007260 con 0x7fe83014f790 2024-03-06T07:27:11.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.468+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe830151f90 con 0x7fe83014f790 2024-03-06T07:27:11.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.469+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83004f330 con 0x7fe83014f790 2024-03-06T07:27:11.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.469+0000 7fe82cff9640 1 -- 172.21.15.94:0/2268269666 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe820007400 con 0x7fe83014f790 2024-03-06T07:27:11.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.470+0000 7fe82cff9640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81c06b500 0x7fe81c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.470+0000 7fe82cff9640 1 -- 172.21.15.94:0/2268269666 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe820089430 con 0x7fe83014f790 2024-03-06T07:27:11.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.470+0000 7fe82ffff640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81c06b500 0x7fe81c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.470+0000 7fe82ffff640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81c06b500 0x7fe81c06d9c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe824006fd0 tx=0x7fe824008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.474+0000 7fe82cff9640 1 -- 172.21.15.94:0/2268269666 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe820057eb0 con 0x7fe83014f790 2024-03-06T07:27:11.648 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.647+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe83006d550 con 0x7fe83014f790 2024-03-06T07:27:11.649 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.647+0000 7fe82cff9640 1 -- 172.21.15.94:0/2268269666 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) v1 ==== 74+0+13174 (secure 0 0 0) 0x7fe82005bf50 con 0x7fe83014f790 2024-03-06T07:27:11.649 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:11.649 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:27:10.340663+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:11.651 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81c06b500 msgr2=0x7fe81c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81c06b500 0x7fe81c06d9c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe824006fd0 tx=0x7fe824008040 comp rx=0 tx=0).stop 2024-03-06T07:27:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe83014f790 msgr2=0x7fe830151a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe83014f790 0x7fe830151a50 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fe82000c430 tx=0x7fe82000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 shutdown_connections 2024-03-06T07:27:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe81c06b500 0x7fe81c06d9c0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe830155a40 0x7fe830071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe83014f790 0x7fe830151a50 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 --2- 172.21.15.94:0/2268269666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe8300c1aa0 0x7fe8301e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 >> 172.21.15.94:0/2268269666 conn(0x7fe8300c6710 msgr2=0x7fe8300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 shutdown_connections 2024-03-06T07:27:11.656 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.649+0000 7fe8364f4640 1 -- 172.21.15.94:0/2268269666 wait complete. 2024-03-06T07:27:11.662 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:11.663 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:11.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.763+0000 7f9f5fb8a640 1 Processor -- start 2024-03-06T07:27:11.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.763+0000 7f9f5fb8a640 1 -- start start 2024-03-06T07:27:11.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5fb8a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9f581538c0 0x7f9f581494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5fb8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9f580a97c0 0x7f9f581499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5fb8a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58154330 con 0x7f9f580ab310 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5fb8a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f580c46a0 con 0x7f9f580a97c0 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5fb8a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f58153dc0 con 0x7f9f581538c0 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5e127640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9f580a97c0 0x7f9f581499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5d926640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9f580ab310 0x7f9f580ab6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37834/0 (socket says 172.21.15.94:37834) 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5e127640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9f580a97c0 0x7f9f581499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54836/0 (socket says 172.21.15.94:54836) 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5d926640 1 -- 172.21.15.94:0/404063856 learned_addr learned my addr 172.21.15.94:0/404063856 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5d125640 1 --2- 172.21.15.94:0/404063856 >> v2:172.21.15.145:3300/0 conn(0x7f9f581538c0 0x7f9f581494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.764+0000 7f9f5e127640 1 -- 172.21.15.94:0/404063856 >> v2:172.21.15.145:3300/0 conn(0x7f9f581538c0 msgr2=0x7f9f581494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5e127640 1 --2- 172.21.15.94:0/404063856 >> v2:172.21.15.145:3300/0 conn(0x7f9f581538c0 0x7f9f581494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5e127640 1 -- 172.21.15.94:0/404063856 >> v1:172.21.15.94:6789/0 conn(0x7f9f580ab310 legacy=0x7f9f580ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5e127640 1 -- 172.21.15.94:0/404063856 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f580c5d30 con 0x7f9f580a97c0 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5c924640 1 -- 172.21.15.94:0/404063856 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309773801 0 0) 0x7f9f58154330 con 0x7f9f580ab310 2024-03-06T07:27:11.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5d125640 1 --2- 172.21.15.94:0/404063856 >> v2:172.21.15.145:3300/0 conn(0x7f9f581538c0 0x7f9f581494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5e127640 1 --2- 172.21.15.94:0/404063856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f581499e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f9f580c75b0 tx=0x7f9f5002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91212065e78370 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5c924640 1 -- 172.21.15.94:0/404063856 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f500315a0 con 0x7f9f580a97c0 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5c924640 1 -- 172.21.15.94:0/404063856 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f50037070 con 0x7f9f580a97c0 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.765+0000 7f9f5c924640 1 -- 172.21.15.94:0/404063856 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50031960 con 0x7f9f580a97c0 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/404063856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 msgr2=0x7f9f581499e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/404063856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f581499e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f9f580c75b0 tx=0x7f9f5002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/404063856 shutdown_connections 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/404063856 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f581499e0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/404063856 >> v2:172.21.15.145:3300/0 conn(0x7f9f581538c0 0x7f9f581494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/404063856 >> 172.21.15.94:0/404063856 conn(0x7f9f580c6710 msgr2=0x7f9f580c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/404063856 shutdown_connections 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.766+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/404063856 wait complete. 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.767+0000 7f9f5fb8a640 1 Processor -- start 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.767+0000 7f9f5fb8a640 1 -- start start 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5fb8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f5814f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5fb8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f580ab310 0x7f9f580720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5fb8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f581538c0 0x7f9f581dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5fb8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580c4b30 con 0x7f9f580a97c0 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5fb8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580c62f0 con 0x7f9f580ab310 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5fb8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580c4cb0 con 0x7f9f581538c0 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f5814f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f5814f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54848/0 (socket says 172.21.15.94:54848) 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 -- 172.21.15.94:0/2743149063 learned_addr learned my addr 172.21.15.94:0/2743149063 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:11.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d125640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f580ab310 0x7f9f580720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5e127640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f581538c0 0x7f9f581dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 -- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f581538c0 msgr2=0x7f9f581dceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f581538c0 0x7f9f581dceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 -- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f580ab310 msgr2=0x7f9f580720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f580ab310 0x7f9f580720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 -- 172.21.15.94:0/2743149063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f58073460 con 0x7f9f580a97c0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d125640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f580ab310 0x7f9f580720c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5e127640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f581538c0 0x7f9f581dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f5d926640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f5814f750 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400c890 tx=0x7f9f5400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.768+0000 7f9f4e7fc640 1 -- 172.21.15.94:0/2743149063 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f54018070 con 0x7f9f580a97c0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.769+0000 7f9f4e7fc640 1 -- 172.21.15.94:0/2743149063 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5400a640 con 0x7f9f580a97c0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.769+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f58073180 con 0x7f9f580a97c0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.769+0000 7f9f4e7fc640 1 -- 172.21.15.94:0/2743149063 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5400aa30 con 0x7f9f580a97c0 2024-03-06T07:27:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.769+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f58072600 con 0x7f9f580a97c0 2024-03-06T07:27:11.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.770+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5804f330 con 0x7f9f580a97c0 2024-03-06T07:27:11.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.770+0000 7f9f4e7fc640 1 -- 172.21.15.94:0/2743149063 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f5401b070 con 0x7f9f580a97c0 2024-03-06T07:27:11.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.770+0000 7f9f4e7fc640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f3c06b4d0 0x7f9f3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.770+0000 7f9f4e7fc640 1 -- 172.21.15.94:0/2743149063 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9f540896a0 con 0x7f9f580a97c0 2024-03-06T07:27:11.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.770+0000 7f9f5d125640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f3c06b4d0 0x7f9f3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.771+0000 7f9f5d125640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f3c06b4d0 0x7f9f3c06d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9f48006fd0 tx=0x7f9f48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.775+0000 7f9f4e7fc640 1 -- 172.21.15.94:0/2743149063 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f540586e0 con 0x7f9f580a97c0 2024-03-06T07:27:11.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.946+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f580c5a70 con 0x7f9f580a97c0 2024-03-06T07:27:11.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.947+0000 7f9f4e7fc640 1 -- 172.21.15.94:0/2743149063 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v200) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f54089030 con 0x7f9f580a97c0 2024-03-06T07:27:11.949 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f3c06b4d0 msgr2=0x7f9f3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f3c06b4d0 0x7f9f3c06d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9f48006fd0 tx=0x7f9f48008040 comp rx=0 tx=0).stop 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 msgr2=0x7f9f5814f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f5814f750 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400c890 tx=0x7f9f5400cd50 comp rx=0 tx=0).stop 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 shutdown_connections 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9f3c06b4d0 0x7f9f3c06d990 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f581538c0 0x7f9f581dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f580ab310 0x7f9f580720c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 --2- 172.21.15.94:0/2743149063 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f580a97c0 0x7f9f5814f750 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 >> 172.21.15.94:0/2743149063 conn(0x7f9f580c6710 msgr2=0x7f9f58156560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 shutdown_connections 2024-03-06T07:27:11.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:11.949+0000 7f9f5fb8a640 1 -- 172.21.15.94:0/2743149063 wait complete. 2024-03-06T07:27:11.962 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-03-06T07:27:12.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.060+0000 7fc7a9bb5640 1 Processor -- start 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.060+0000 7fc7a9bb5640 1 -- start start 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a9bb5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc7a40c1aa0 0x7fc7a41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a9bb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a4155a40 0x7fc7a4155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a9bb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a41563d0 con 0x7fc7a414f790 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a9bb5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a40c2680 con 0x7fc7a40c1aa0 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a9bb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40c28d0 con 0x7fc7a4155a40 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a2ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc7a40c1aa0 0x7fc7a41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a4155a40 0x7fc7a4155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a37fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc7a414f790 0x7fc7a40c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:37702/0 (socket says 172.21.15.94:37702) 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a37fe640 1 -- 172.21.15.94:0/3692379443 learned_addr learned my addr 172.21.15.94:0/3692379443 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a2ffd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc7a40c1aa0 0x7fc7a41529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:33518/0 (socket says 172.21.15.94:33518) 2024-03-06T07:27:12.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.061+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a4155a40 0x7fc7a4155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:41370/0 (socket says 172.21.15.94:41370) 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.062+0000 7fc7a2ffd640 1 -- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a4155a40 msgr2=0x7fc7a4155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.062+0000 7fc7a2ffd640 1 --2- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a4155a40 0x7fc7a4155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.062+0000 7fc7a2ffd640 1 -- 172.21.15.94:0/3692379443 >> v1:172.21.15.145:6789/0 conn(0x7fc7a414f790 legacy=0x7fc7a40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.062+0000 7fc7a2ffd640 1 -- 172.21.15.94:0/3692379443 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40c2bb0 con 0x7fc7a40c1aa0 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.062+0000 7fc7a3fff640 1 --2- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a4155a40 0x7fc7a4155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a2ffd640 1 --2- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a40c1aa0 0x7fc7a41529d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc7980097a0 tx=0x7fc79802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef98747945af9f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a27fc640 1 -- 172.21.15.94:0/3692379443 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798035020 con 0x7fc7a40c1aa0 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a27fc640 1 -- 172.21.15.94:0/3692379443 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc798031830 con 0x7fc7a40c1aa0 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a27fc640 1 -- 172.21.15.94:0/3692379443 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798031c00 con 0x7fc7a40c1aa0 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a40c1aa0 msgr2=0x7fc7a41529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a40c1aa0 0x7fc7a41529d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc7980097a0 tx=0x7fc79802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/3692379443 shutdown_connections 2024-03-06T07:27:12.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a4155a40 0x7fc7a4155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/3692379443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a40c1aa0 0x7fc7a41529d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/3692379443 >> 172.21.15.94:0/3692379443 conn(0x7fc7a40c6710 msgr2=0x7fc7a40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/3692379443 shutdown_connections 2024-03-06T07:27:12.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.063+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/3692379443 wait complete. 2024-03-06T07:27:12.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.064+0000 7fc7a9bb5640 1 Processor -- start 2024-03-06T07:27:12.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a9bb5640 1 -- start start 2024-03-06T07:27:12.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a9bb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a40c1aa0 0x7fc7a40c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a9bb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a414f790 0x7fc7a414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a9bb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7a4155a40 0x7fc7a4148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a9bb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40c0d60 con 0x7fc7a4155a40 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a9bb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a4155520 con 0x7fc7a414f790 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a9bb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a40c5e30 con 0x7fc7a40c1aa0 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7a4155a40 0x7fc7a4148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7a4155a40 0x7fc7a4148e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54856/0 (socket says 172.21.15.94:54856) 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a3fff640 1 -- 172.21.15.94:0/345101477 learned_addr learned my addr 172.21.15.94:0/345101477 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a2ffd640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a414f790 0x7fc7a414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.065+0000 7fc7a37fe640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a40c1aa0 0x7fc7a40c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a3fff640 1 -- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a40c1aa0 msgr2=0x7fc7a40c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a3fff640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a40c1aa0 0x7fc7a40c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a3fff640 1 -- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a414f790 msgr2=0x7fc7a414dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a3fff640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a414f790 0x7fc7a414dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a3fff640 1 -- 172.21.15.94:0/345101477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40c53d0 con 0x7fc7a4155a40 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a3fff640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7a4155a40 0x7fc7a4148e30 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009e60 tx=0x7fc79c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc78bfff640 1 -- 172.21.15.94:0/345101477 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c018030 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc78bfff640 1 -- 172.21.15.94:0/345101477 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c00cdf0 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc78bfff640 1 -- 172.21.15.94:0/345101477 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c003bb0 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a4150d50 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.066+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a4151310 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.067+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a40a64a0 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.067+0000 7fc78bfff640 1 -- 172.21.15.94:0/345101477 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc79c005d10 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.068+0000 7fc78bfff640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc79006b4d0 0x7fc79006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.068+0000 7fc78bfff640 1 -- 172.21.15.94:0/345101477 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc79c08a7e0 con 0x7fc7a4155a40 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.068+0000 7fc7a37fe640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc79006b4d0 0x7fc79006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.069+0000 7fc7a37fe640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc79006b4d0 0x7fc79006d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc794007fd0 tx=0x7fc794007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.072+0000 7fc78bfff640 1 -- 172.21.15.94:0/345101477 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7a40a64a0 con 0x7fc7a4155a40 2024-03-06T07:27:12.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.314+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7fc7a404f330 con 0x7fc7a4155a40 2024-03-06T07:27:12.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.350+0000 7fc78bfff640 1 -- 172.21.15.94:0/345101477 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v504) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc79c022070 con 0x7fc7a4155a40 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.351+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc79006b4d0 msgr2=0x7fc79006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.351+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc79006b4d0 0x7fc79006d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc794007fd0 tx=0x7fc794007e80 comp rx=0 tx=0).stop 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7a4155a40 msgr2=0x7fc7a4148e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7a4155a40 0x7fc7a4148e30 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fc79c009e60 tx=0x7fc79c00c490 comp rx=0 tx=0).stop 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 shutdown_connections 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc79006b4d0 0x7fc79006d990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7a4155a40 0x7fc7a4148e30 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7a414f790 0x7fc7a414dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 --2- 172.21.15.94:0/345101477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7a40c1aa0 0x7fc7a40c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 >> 172.21.15.94:0/345101477 conn(0x7fc7a40c6710 msgr2=0x7fc7a40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 shutdown_connections 2024-03-06T07:27:12.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.352+0000 7fc7a9bb5640 1 -- 172.21.15.94:0/345101477 wait complete. 2024-03-06T07:27:12.364 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:27:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.465+0000 7fed0f932640 1 Processor -- start 2024-03-06T07:27:12.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.465+0000 7fed0f932640 1 -- start start 2024-03-06T07:27:12.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0f932640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed080c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0f932640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed080ab170 0x7fed080a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0f932640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed080c3ef0 con 0x7fed080c5990 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0f932640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed081556d0 con 0x7fed080ab170 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0f932640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed081559b0 con 0x7fed0806c900 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0decf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed080ab170 0x7fed080a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0decf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fed080ab170 0x7fed080a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54864/0 (socket says 172.21.15.94:54864) 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0decf640 1 -- 172.21.15.94:0/2820832853 learned_addr learned my addr 172.21.15.94:0/2820832853 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0decf640 1 -- 172.21.15.94:0/2820832853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 msgr2=0x7fed080c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0decf640 1 --2- 172.21.15.94:0/2820832853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed080c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0decf640 1 -- 172.21.15.94:0/2820832853 >> v1:172.21.15.145:6789/0 conn(0x7fed080c5990 legacy=0x7fed0806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.466+0000 7fed0decf640 1 -- 172.21.15.94:0/2820832853 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08155c90 con 0x7fed080ab170 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.467+0000 7fed0decf640 1 --2- 172.21.15.94:0/2820832853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080ab170 0x7fed080a9fa0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d150 tx=0x7fed0002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd4caaa773a33dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.467+0000 7fecfffff640 1 -- 172.21.15.94:0/2820832853 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0002ea80 con 0x7fed080ab170 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.467+0000 7fecfffff640 1 -- 172.21.15.94:0/2820832853 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0002ec20 con 0x7fed080ab170 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.467+0000 7fecfffff640 1 -- 172.21.15.94:0/2820832853 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00009600 con 0x7fed080ab170 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 -- 172.21.15.94:0/2820832853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080ab170 msgr2=0x7fed080a9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 --2- 172.21.15.94:0/2820832853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080ab170 0x7fed080a9fa0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d150 tx=0x7fed0002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 -- 172.21.15.94:0/2820832853 shutdown_connections 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 --2- 172.21.15.94:0/2820832853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080ab170 0x7fed080a9fa0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 --2- 172.21.15.94:0/2820832853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed080c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 -- 172.21.15.94:0/2820832853 >> 172.21.15.94:0/2820832853 conn(0x7fed080c6710 msgr2=0x7fed080c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 -- 172.21.15.94:0/2820832853 shutdown_connections 2024-03-06T07:27:12.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.468+0000 7fed0f932640 1 -- 172.21.15.94:0/2820832853 wait complete. 2024-03-06T07:27:12.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 Processor -- start 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 -- start start 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed081539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed080ab170 0x7fed081518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080c5990 0x7fed08146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08156610 con 0x7fed080c5990 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08156490 con 0x7fed080ab170 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.469+0000 7fed0f932640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed080c3af0 con 0x7fed0806c900 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080c5990 0x7fed08146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080c5990 0x7fed08146830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54872/0 (socket says 172.21.15.94:54872) 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 -- 172.21.15.94:0/511507332 learned_addr learned my addr 172.21.15.94:0/511507332 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0cecd640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed080ab170 0x7fed081518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0d6ce640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed081539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 -- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 msgr2=0x7fed081539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed081539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 -- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed080ab170 msgr2=0x7fed081518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed080ab170 0x7fed081518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 -- 172.21.15.94:0/511507332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed081ea960 con 0x7fed080c5990 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0cecd640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed080ab170 0x7fed081518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0d6ce640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed081539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0decf640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080c5990 0x7fed08146830 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fed00038040 tx=0x7fed00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fecfdffb640 1 -- 172.21.15.94:0/511507332 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000b720 con 0x7fed080c5990 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fecfdffb640 1 -- 172.21.15.94:0/511507332 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00045070 con 0x7fed080c5990 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed081eac20 con 0x7fed080c5990 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fecfdffb640 1 -- 172.21.15.94:0/511507332 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00036a60 con 0x7fed080c5990 2024-03-06T07:27:12.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.470+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed081eb1e0 con 0x7fed080c5990 2024-03-06T07:27:12.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.471+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0804f330 con 0x7fed080c5990 2024-03-06T07:27:12.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.472+0000 7fecfdffb640 1 -- 172.21.15.94:0/511507332 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed00036c00 con 0x7fed080c5990 2024-03-06T07:27:12.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.472+0000 7fecfdffb640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fecec06b4d0 0x7fecec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.472+0000 7fecfdffb640 1 -- 172.21.15.94:0/511507332 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fed000acfb0 con 0x7fed080c5990 2024-03-06T07:27:12.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.472+0000 7fed0d6ce640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fecec06b4d0 0x7fecec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.473+0000 7fed0d6ce640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fecec06b4d0 0x7fecec06d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fed04006fd0 tx=0x7fed04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.477+0000 7fecfdffb640 1 -- 172.21.15.94:0/511507332 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed0007c1a0 con 0x7fed080c5990 2024-03-06T07:27:12.652 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.651+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fed080c1740 con 0x7fed080c5990 2024-03-06T07:27:12.655 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.653+0000 7fecfdffb640 1 -- 172.21.15.94:0/511507332 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v63)=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7fed0007baa0 con 0x7fed080c5990 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.655+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fecec06b4d0 msgr2=0x7fecec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fecec06b4d0 0x7fecec06d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fed04006fd0 tx=0x7fed04008040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080c5990 msgr2=0x7fed08146830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080c5990 0x7fed08146830 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fed00038040 tx=0x7fed00004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 shutdown_connections 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fecec06b4d0 0x7fecec06d990 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fed080c5990 0x7fed08146830 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed080ab170 0x7fed081518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 --2- 172.21.15.94:0/511507332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed0806c900 0x7fed081539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 >> 172.21.15.94:0/511507332 conn(0x7fed080c6710 msgr2=0x7fed080c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 shutdown_connections 2024-03-06T07:27:12.658 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.656+0000 7fed0f932640 1 -- 172.21.15.94:0/511507332 wait complete. 2024-03-06T07:27:12.668 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:12.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:12.705 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:12.773 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:12.773 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:27:12.773 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:12.788 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:27:12.788 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:27:12.788 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:12.806 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:27:12.806 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-03-06T07:27:12.806 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:12.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.925+0000 7f8c60edf640 1 Processor -- start 2024-03-06T07:27:12.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.925+0000 7f8c60edf640 1 -- start start 2024-03-06T07:27:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c60edf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 0x7f8c5c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c60edf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c153a60 con 0x7f8c5c0c1320 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c60edf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c153d40 con 0x7f8c5c150580 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c60edf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c06c2c0 con 0x7f8c5c0c6000 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c53fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8c5c0c1320 0x7f8c5c1566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37844/0 (socket says 172.21.15.94:37844) 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c53fff640 1 -- 172.21.15.94:0/1415769670 learned_addr learned my addr 172.21.15.94:0/1415769670 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967037777 0 0) 0x7f8c5c153a60 con 0x7f8c5c0c1320 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c48003520 con 0x7f8c5c0c1320 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.926+0000 7f8c5ad9d640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 0x7f8c5c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291343944 0 0) 0x7f8c48003520 con 0x7f8c5c0c1320 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 msgr2=0x7f8c5c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 0x7f8c5c156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 >> v1:172.21.15.145:6789/0 conn(0x7f8c5c150580 legacy=0x7f8c5c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c06c5a0 con 0x7f8c5c0c1320 2024-03-06T07:27:12.931 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f8c44001670 con 0x7f8c5c0c1320 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 >> v1:172.21.15.94:6789/0 conn(0x7f8c5c0c1320 legacy=0x7f8c5c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 0x7f8c48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c48004d40 0x7f8c480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c06c2c0 con 0x7f8c480039e0 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4800baf0 con 0x7f8c48004d40 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4800bcf0 con 0x7f8c5c0c6000 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c5a59c640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c5ad9d640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c48004d40 0x7f8c480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c53fff640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 0x7f8c48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c5a59c640 1 -- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 msgr2=0x7f8c48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c5a59c640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c0c6000 0x7f8c48004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c5a59c640 1 -- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c48004d40 msgr2=0x7f8c480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c5a59c640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c48004d40 0x7f8c480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.927+0000 7f8c5a59c640 1 -- 172.21.15.94:0/1415769670 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c48003520 con 0x7f8c480039e0 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.928+0000 7f8c5a59c640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c48003e00 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c001750 tx=0x7f8c48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22161f562c6eb71f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.928+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c001370 con 0x7f8c480039e0 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.928+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c0052d0 con 0x7f8c480039e0 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.928+0000 7f8c59d9b640 1 -- 172.21.15.94:0/1415769670 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c003830 con 0x7f8c480039e0 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.928+0000 7f8c60edf640 1 -- 172.21.15.94:0/1415769670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 msgr2=0x7f8c48003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.928+0000 7f8c60edf640 1 --2- 172.21.15.94:0/1415769670 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c48003e00 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c001750 tx=0x7f8c48005830 comp rx=0 tx=0).stop 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.929+0000 7f8c5ad9d640 1 -- 172.21.15.94:0/1415769670 reap_dead start 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.929+0000 7f8c60edf640 1 -- 172.21.15.94:0/1415769670 shutdown_connections 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.929+0000 7f8c60edf640 1 -- 172.21.15.94:0/1415769670 >> 172.21.15.94:0/1415769670 conn(0x7f8c5c0c6710 msgr2=0x7f8c5c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.929+0000 7f8c60edf640 1 -- 172.21.15.94:0/1415769670 shutdown_connections 2024-03-06T07:27:12.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.929+0000 7f8c60edf640 1 -- 172.21.15.94:0/1415769670 wait complete. 2024-03-06T07:27:12.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.930+0000 7f8c60edf640 1 Processor -- start 2024-03-06T07:27:12.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.930+0000 7f8c60edf640 1 -- start start 2024-03-06T07:27:12.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.930+0000 7f8c60edf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c5c072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.930+0000 7f8c60edf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c14bf80 0x7f8c5c1e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.936 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c60edf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c5c0a7720 0x7f8c5c0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c60edf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c06e5a0 con 0x7f8c480039e0 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c60edf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c06e420 con 0x7f8c5c0a7720 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c60edf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c0c0c40 con 0x7f8c5c14bf80 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c5c072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c5c072d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54884/0 (socket says 172.21.15.94:54884) 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 -- 172.21.15.94:0/2790733953 learned_addr learned my addr 172.21.15.94:0/2790733953 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c53fff640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c14bf80 0x7f8c5c1e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5ad9d640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c5c0a7720 0x7f8c5c0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 -- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c14bf80 msgr2=0x7f8c5c1e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c14bf80 0x7f8c5c1e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 -- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c5c0a7720 msgr2=0x7f8c5c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c5c0a7720 0x7f8c5c0706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.937 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 -- 172.21.15.94:0/2790733953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c1e9280 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c5a59c640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c5c072d00 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0017b0 tx=0x7f8c4c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c52ffd640 1 -- 172.21.15.94:0/2790733953 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c041070 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c52ffd640 1 -- 172.21.15.94:0/2790733953 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c005480 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c14b0f0 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c52ffd640 1 -- 172.21.15.94:0/2790733953 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c001370 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.931+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c14b3b0 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.932+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c04f330 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.933+0000 7f8c52ffd640 1 -- 172.21.15.94:0/2790733953 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c4c044070 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.933+0000 7f8c52ffd640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c3c06b4d0 0x7f8c3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.933+0000 7f8c52ffd640 1 -- 172.21.15.94:0/2790733953 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8c4c0abc40 con 0x7f8c480039e0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.933+0000 7f8c53fff640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c3c06b4d0 0x7f8c3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.938 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.934+0000 7f8c53fff640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c3c06b4d0 0x7f8c3c06d990 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8c44001eb0 tx=0x7f8c44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.939 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:12.937+0000 7f8c52ffd640 1 -- 172.21.15.94:0/2790733953 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c4c07ae30 con 0x7f8c480039e0 2024-03-06T07:27:13.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.144+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c5c0702f0 con 0x7f8c480039e0 2024-03-06T07:27:13.146 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.144+0000 7f8c52ffd640 1 -- 172.21.15.94:0/2790733953 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+3704 (secure 0 0 0) 0x7f8c4c07a730 con 0x7f8c480039e0 2024-03-06T07:27:13.146 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:13.147 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":192,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/399318570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":399318570},{"type":"v1","addr":"172.21.15.145:6837","nonce":399318570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":190},{"gid":10428,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/2752830090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":2752830090},{"type":"v1","addr":"172.21.15.94:6834","nonce":2752830090}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":10431,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/750154668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":750154668},{"type":"v1","addr":"172.21.15.94:6835","nonce":750154668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183},{"gid":38057,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/565245371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":565245371},{"type":"v1","addr":"172.21.15.145:6836","nonce":565245371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":183}],"filesystems":[]} 2024-03-06T07:27:13.147 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 192 2024-03-06T07:27:13.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c3c06b4d0 msgr2=0x7f8c3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c3c06b4d0 0x7f8c3c06d990 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8c44001eb0 tx=0x7f8c44009040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 msgr2=0x7f8c5c072d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c5c072d00 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0017b0 tx=0x7f8c4c001120 comp rx=0 tx=0).stop 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 shutdown_connections 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8c3c06b4d0 0x7f8c3c06d990 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c5c0a7720 0x7f8c5c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c5c14bf80 0x7f8c5c1e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 --2- 172.21.15.94:0/2790733953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c480039e0 0x7f8c5c072d00 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 >> 172.21.15.94:0/2790733953 conn(0x7f8c5c0c6710 msgr2=0x7f8c5c151a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 shutdown_connections 2024-03-06T07:27:13.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.146+0000 7f8c60edf640 1 -- 172.21.15.94:0/2790733953 wait complete. 2024-03-06T07:27:13.159 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:13.159 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:13.160 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:13.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:13.160 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:13.160 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:13.160 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:13.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:13.161 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:13.161 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:13.161 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:13.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:13.161 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:13.161 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:13.162 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:13.162 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:13.168 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:13.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:13.169 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:13.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:13.173 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:13.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:13.173 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:13.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:13.211 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:13.211 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:13.212 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:13.212 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:13.215 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:13.216 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:13.216 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:13.216 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:13.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:13.287 INFO:tasks.ceph.mds.a.smithi094.stdout:starting mds.a at 2024-03-06T07:27:13.288 INFO:tasks.ceph.mds.c.smithi094.stdout:starting mds.c at 2024-03-06T07:27:13.298 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-03-06T07:27:13.299 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:27:13.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.321+0000 7fb9131ac640 1 Processor -- start 2024-03-06T07:27:13.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.321+0000 7fb9131ac640 1 -- start start 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb9131ac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb90c0c69f0 0x7fb90c0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb9131ac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c155130 con 0x7fb90c1558d0 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb911749640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb90c1558d0 0x7fb90c15fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37858/0 (socket says 172.21.15.94:37858) 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb911749640 1 -- 172.21.15.94:0/2103534381 learned_addr learned my addr 172.21.15.94:0/2103534381 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb9131ac640 1 -- 172.21.15.94:0/2103534381 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c0ad480 con 0x7fb90c0c8950 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb9131ac640 1 -- 172.21.15.94:0/2103534381 --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c155440 con 0x7fb90c0c69f0 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102274226 0 0) 0x7fb904001490 con 0x7fb90c1558d0 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8fc003520 con 0x7fb90c1558d0 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb910f48640 1 --2- 172.21.15.94:0/2103534381 >> v2:172.21.15.145:3300/0 conn(0x7fb90c0c69f0 0x7fb90c0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.322+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 172441525 0 0) 0x7fb8fc003520 con 0x7fb90c1558d0 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 >> v2:172.21.15.145:3300/0 conn(0x7fb90c0c69f0 msgr2=0x7fb90c0c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 --2- 172.21.15.94:0/2103534381 >> v2:172.21.15.145:3300/0 conn(0x7fb90c0c69f0 0x7fb90c0c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 >> v1:172.21.15.145:6789/0 conn(0x7fb90c0c8950 legacy=0x7fb90c0acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c0c5d50 con 0x7fb90c1558d0 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fb904001bc0 con 0x7fb90c1558d0 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 >> v1:172.21.15.94:6789/0 conn(0x7fb90c1558d0 legacy=0x7fb90c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8fc0039b0 0x7fb8fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc004800 0x7fb8fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8fc0051c0 0x7fb8fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c155440 con 0x7fb8fc0039b0 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc00c160 con 0x7fb8fc004800 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc00c360 con 0x7fb8fc0051c0 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb910f48640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8fc0039b0 0x7fb8fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb90bfff640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc004800 0x7fb8fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb911749640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8fc0051c0 0x7fb8fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb910f48640 1 -- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8fc0051c0 msgr2=0x7fb8fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb910f48640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8fc0051c0 0x7fb8fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb910f48640 1 -- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc004800 msgr2=0x7fb8fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb910f48640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc004800 0x7fb8fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb910f48640 1 -- 172.21.15.94:0/2103534381 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc003520 con 0x7fb8fc0039b0 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.323+0000 7fb911749640 1 -- 172.21.15.94:0/2103534381 reap_dead start 2024-03-06T07:27:13.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb910f48640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8fc0039b0 0x7fb8fc003dd0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb900009ac0 tx=0x7fb8fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f38320c47db3eca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb900039070 con 0x7fb8fc0039b0 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90002c6b0 con 0x7fb8fc0039b0 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb90b7fe640 1 -- 172.21.15.94:0/2103534381 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb900009690 con 0x7fb8fc0039b0 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb9131ac640 1 -- 172.21.15.94:0/2103534381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8fc0039b0 msgr2=0x7fb8fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb9131ac640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8fc0039b0 0x7fb8fc003dd0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb900009ac0 tx=0x7fb8fc005830 comp rx=0 tx=0).stop 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb9131ac640 1 -- 172.21.15.94:0/2103534381 shutdown_connections 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb9131ac640 1 --2- 172.21.15.94:0/2103534381 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8fc0039b0 0x7fb8fc003dd0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.324+0000 7fb9131ac640 1 -- 172.21.15.94:0/2103534381 >> 172.21.15.94:0/2103534381 conn(0x7fb90c0cae70 msgr2=0x7fb90c0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.325+0000 7fb9131ac640 1 -- 172.21.15.94:0/2103534381 shutdown_connections 2024-03-06T07:27:13.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.325+0000 7fb9131ac640 1 -- 172.21.15.94:0/2103534381 wait complete. 2024-03-06T07:27:13.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.326+0000 7fb9131ac640 1 Processor -- start 2024-03-06T07:27:13.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.326+0000 7fb9131ac640 1 -- start start 2024-03-06T07:27:13.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb9131ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc0039b0 0x7fb90c0ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb9131ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb90c158880 0x7fb90c14d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb9131ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb90c145650 0x7fb90c145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb9131ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c152a20 con 0x7fb90c145650 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb9131ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c0c6310 con 0x7fb8fc0039b0 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb90c145650 0x7fb90c145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb90c145650 0x7fb90c145ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54926/0 (socket says 172.21.15.94:54926) 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 -- 172.21.15.94:0/317358491 learned_addr learned my addr 172.21.15.94:0/317358491 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb910f48640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc0039b0 0x7fb90c0ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c152ba0 con 0x7fb90c158880 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb90bfff640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb90c158880 0x7fb90c14d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 -- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb90c158880 msgr2=0x7fb90c14d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb90c158880 0x7fb90c14d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 -- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc0039b0 msgr2=0x7fb90c0ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc0039b0 0x7fb90c0ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 -- 172.21.15.94:0/317358491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c156800 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb911749640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb90c145650 0x7fb90c145ad0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fb904001d30 tx=0x7fb90400af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.327+0000 7fb9097fa640 1 -- 172.21.15.94:0/317358491 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9040078a0 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.328+0000 7fb9097fa640 1 -- 172.21.15.94:0/317358491 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb904018070 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.328+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c156520 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.328+0000 7fb9097fa640 1 -- 172.21.15.94:0/317358491 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904013630 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.328+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c1592b0 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.329+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c04f330 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.329+0000 7fb9097fa640 1 -- 172.21.15.94:0/317358491 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9040137d0 con 0x7fb90c145650 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.329+0000 7fb9097fa640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8f006b4d0 0x7fb8f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.330+0000 7fb9097fa640 1 -- 172.21.15.94:0/317358491 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb90408a510 con 0x7fb90c145650 2024-03-06T07:27:13.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.330+0000 7fb910f48640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8f006b4d0 0x7fb8f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.330+0000 7fb910f48640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8f006b4d0 0x7fb8f006d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb900036000 tx=0x7fb900030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.334+0000 7fb9097fa640 1 -- 172.21.15.94:0/317358491 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb90c155130 con 0x7fb90c145650 2024-03-06T07:27:13.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:13.521+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb90c159550 con 0x7fb90c145650 2024-03-06T07:27:14.380 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.378+0000 7fb9097fa640 1 -- 172.21.15.94:0/317358491 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v205) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb904058ef0 con 0x7fb90c145650 2024-03-06T07:27:14.380 INFO:teuthology.orchestra.run.smithi094.stderr: removed all blocklist entries 2024-03-06T07:27:14.383 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8f006b4d0 msgr2=0x7fb8f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8f006b4d0 0x7fb8f006d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb900036000 tx=0x7fb900030000 comp rx=0 tx=0).stop 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb90c145650 msgr2=0x7fb90c145ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb90c145650 0x7fb90c145ad0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fb904001d30 tx=0x7fb90400af60 comp rx=0 tx=0).stop 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 shutdown_connections 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8f006b4d0 0x7fb8f006d990 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb90c145650 0x7fb90c145ad0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb90c158880 0x7fb90c14d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 --2- 172.21.15.94:0/317358491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8fc0039b0 0x7fb90c0ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 >> 172.21.15.94:0/317358491 conn(0x7fb90c0cae70 msgr2=0x7fb90c0c7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 shutdown_connections 2024-03-06T07:27:14.384 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.381+0000 7fb9131ac640 1 -- 172.21.15.94:0/317358491 wait complete. 2024-03-06T07:27:14.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:14.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.500+0000 7f690495c640 1 Processor -- start 2024-03-06T07:27:14.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.500+0000 7f690495c640 1 -- start start 2024-03-06T07:27:14.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f690495c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f69001550c0 0x7f69001554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f690495c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f69000aa180 0x7f69000c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f690495c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69000aa660 con 0x7f69001559e0 2024-03-06T07:27:14.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f690495c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69000c4610 con 0x7f69001550c0 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f690495c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69000aa940 con 0x7f69000aa180 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f68fe59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f69001550c0 0x7f69001554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f68fe59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f69001550c0 0x7f69001554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54942/0 (socket says 172.21.15.94:54942) 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f68fe59c640 1 -- 172.21.15.94:0/1124782993 learned_addr learned my addr 172.21.15.94:0/1124782993 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f68fe59c640 1 -- 172.21.15.94:0/1124782993 >> v2:172.21.15.145:3300/0 conn(0x7f69000aa180 msgr2=0x7f69000c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f68fe59c640 1 --2- 172.21.15.94:0/1124782993 >> v2:172.21.15.145:3300/0 conn(0x7f69000aa180 0x7f69000c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f68fe59c640 1 -- 172.21.15.94:0/1124782993 >> v1:172.21.15.145:6789/0 conn(0x7f69001559e0 legacy=0x7f690014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.501+0000 7f68fe59c640 1 -- 172.21.15.94:0/1124782993 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000c4860 con 0x7f69001550c0 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.502+0000 7f68fe59c640 1 --2- 172.21.15.94:0/1124782993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001550c0 0x7f69001554a0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f69000abd80 tx=0x7f68f002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506f0f76950574db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.502+0000 7f68fd59a640 1 -- 172.21.15.94:0/1124782993 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f0038070 con 0x7f69001550c0 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.502+0000 7f68fd59a640 1 -- 172.21.15.94:0/1124782993 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f002f050 con 0x7f69001550c0 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.502+0000 7f68fd59a640 1 -- 172.21.15.94:0/1124782993 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f0031970 con 0x7f69001550c0 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 -- 172.21.15.94:0/1124782993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001550c0 msgr2=0x7f69001554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 --2- 172.21.15.94:0/1124782993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001550c0 0x7f69001554a0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f69000abd80 tx=0x7f68f002b7f0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 -- 172.21.15.94:0/1124782993 shutdown_connections 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 --2- 172.21.15.94:0/1124782993 >> v2:172.21.15.145:3300/0 conn(0x7f69000aa180 0x7f69000c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 --2- 172.21.15.94:0/1124782993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001550c0 0x7f69001554a0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 -- 172.21.15.94:0/1124782993 >> 172.21.15.94:0/1124782993 conn(0x7f69000c6710 msgr2=0x7f69000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 -- 172.21.15.94:0/1124782993 shutdown_connections 2024-03-06T07:27:14.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.503+0000 7f690495c640 1 -- 172.21.15.94:0/1124782993 wait complete. 2024-03-06T07:27:14.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.504+0000 7f690495c640 1 Processor -- start 2024-03-06T07:27:14.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.504+0000 7f690495c640 1 -- start start 2024-03-06T07:27:14.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f690495c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69000aa180 0x7f69000713c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f690495c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69001550c0 0x7f69000a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f690495c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001559e0 0x7f69000a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f690495c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690014f310 con 0x7f69001559e0 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f690495c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69000c1560 con 0x7f69000aa180 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f690495c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690014f490 con 0x7f69001550c0 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001559e0 0x7f69000a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001559e0 0x7f69000a6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54948/0 (socket says 172.21.15.94:54948) 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 -- 172.21.15.94:0/3346730517 learned_addr learned my addr 172.21.15.94:0/3346730517 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fe59c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69000aa180 0x7f69000713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fdd9b640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69001550c0 0x7f69000a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 -- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69001550c0 msgr2=0x7f69000a6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69001550c0 0x7f69000a6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 -- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69000aa180 msgr2=0x7f69000713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69000aa180 0x7f69000713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 -- 172.21.15.94:0/3346730517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000a5eb0 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fe59c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69000aa180 0x7f69000713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.505+0000 7f68fed9d640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001559e0 0x7f69000a6e70 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f68f800b9e0 tx=0x7f68f800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.506+0000 7f68eeffd640 1 -- 172.21.15.94:0/3346730517 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8019070 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.506+0000 7f68eeffd640 1 -- 172.21.15.94:0/3346730517 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f800cb40 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.506+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6900070760 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.506+0000 7f68eeffd640 1 -- 172.21.15.94:0/3346730517 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8014440 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.506+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6900148330 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.507+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69000acab0 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.507+0000 7f68eeffd640 1 -- 172.21.15.94:0/3346730517 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68f801c070 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.507+0000 7f68eeffd640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68e006b4d0 0x7f68e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.508+0000 7f68eeffd640 1 -- 172.21.15.94:0/3346730517 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f68f8089d70 con 0x7f69001559e0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.508+0000 7f68fe59c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68e006b4d0 0x7f68e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.508+0000 7f68fe59c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68e006b4d0 0x7f68e006d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f68f0035000 tx=0x7f68f0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.512+0000 7f68eeffd640 1 -- 172.21.15.94:0/3346730517 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69000acab0 con 0x7f69001559e0 2024-03-06T07:27:14.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.750+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f690014f130 con 0x7f69001559e0 2024-03-06T07:27:14.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.750+0000 7f68eeffd640 1 -- 172.21.15.94:0/3346730517 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v23) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f690014f130 con 0x7f69001559e0 2024-03-06T07:27:14.752 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:14.752 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:27:14.752 INFO:teuthology.orchestra.run.smithi094.stdout: "auth_dump": [ 2024-03-06T07:27:14.752 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.a", 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhllmTpABAAC6PMw80ATbNfXdyjYn0Cwg==", 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.b", 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhl3lZJDRAAcHWFyJHpKzvUbr8iBhhoHA==", 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.753 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.c", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlcW9DCBAADui8X/v0sWkzARFD1nzglA==", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.754 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mds.d", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlxbacFBAA9pQxppUeylRrXFpIjZD3EQ==", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.0", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB9GOhlkQGNLxAA2bTJS02LkkfLlZtaiIWtTA==", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.755 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.1", 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCAGOhlkegSLhAAhYCUEhKqffZ+EWWmQVJ59Q==", 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.756 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.2", 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCDGOhliTlbLBAAZLtGbNteU/KaQQ/dMGHwMg==", 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.3", 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCGGOhl9FyuKhAAgTkbOuLQ3eOVibNQFcWPEQ==", 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.757 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.4", 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCMGOhlh/v9LxAAXyxFLlCo+mWNnaaQVJKJ5Q==", 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.758 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.5", 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCPGOhlFKlVLhAAZcB2/jFZALdq78m883TGXw==", 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.6", 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCSGOhlELmLLBAAUmmpDIU/+Bl5aNVyxqbm/g==", 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.759 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "osd.7", 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQCVGOhlxBM8KxAAgdGBTEapfEfiA1hCNLLi3g==", 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile osd", 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.760 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.0", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ==", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow r", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.1", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB8GOhldDtLHBAA89EDDFdwXQhcGMBG2bvyIg==", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow r", 2024-03-06T07:27:14.761 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow rw", 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow rwx" 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "client.admin", 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhl69BMHxAADvBxscM+gXgKqVc/YNttzw==", 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "mgr": "allow *", 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow *", 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.762 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.x", 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlHGYfORAA0sg5TWRTHhCNX8YMYAZtFA==", 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: }, 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "entity": "mgr.y", 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "key": "AQB7GOhlsVU1NRAAtdHhkkhwvfwjajLmoy/qxg==", 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "caps": { 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "mds": "allow *", 2024-03-06T07:27:14.763 INFO:teuthology.orchestra.run.smithi094.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: "osd": "allow *" 2024-03-06T07:27:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:14.764 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-03-06T07:27:14.764 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:27:14.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68e006b4d0 msgr2=0x7f68e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68e006b4d0 0x7f68e006d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f68f0035000 tx=0x7f68f0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:14.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001559e0 msgr2=0x7f69000a6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001559e0 0x7f69000a6e70 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f68f800b9e0 tx=0x7f68f800bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 shutdown_connections 2024-03-06T07:27:14.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f68e006b4d0 0x7f68e006d990 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f69001559e0 0x7f69000a6e70 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69001550c0 0x7f69000a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 --2- 172.21.15.94:0/3346730517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69000aa180 0x7f69000713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 >> 172.21.15.94:0/3346730517 conn(0x7f69000c6710 msgr2=0x7f69000c31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 shutdown_connections 2024-03-06T07:27:14.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.752+0000 7f690495c640 1 -- 172.21.15.94:0/3346730517 wait complete. 2024-03-06T07:27:14.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.868+0000 7f99d4c9d640 1 Processor -- start 2024-03-06T07:27:14.870 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.868+0000 7f99d4c9d640 1 -- start start 2024-03-06T07:27:14.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99d4c9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99d014f360 0x7f99d0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99d4c9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d00c88c0 0x7f99d00c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99d4c9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f99d00c5880 0x7f99d0158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99d4c9d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d00c8dd0 con 0x7f99d014f360 2024-03-06T07:27:14.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99d4c9d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d00c73d0 con 0x7f99d00c5880 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99d4c9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d00be2e0 con 0x7f99d00c88c0 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99ce59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99d014f360 0x7f99d0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99ce59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f99d014f360 0x7f99d0155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54954/0 (socket says 172.21.15.94:54954) 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99ce59c640 1 -- 172.21.15.94:0/3939089196 learned_addr learned my addr 172.21.15.94:0/3939089196 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.869+0000 7f99ced9d640 1 --2- 172.21.15.94:0/3939089196 >> v2:172.21.15.145:3300/0 conn(0x7f99d00c5880 0x7f99d0158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99c7fff640 1 --2- 172.21.15.94:0/3939089196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d00c88c0 0x7f99d00c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99ce59c640 1 -- 172.21.15.94:0/3939089196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d00c88c0 msgr2=0x7f99d00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99ce59c640 1 --2- 172.21.15.94:0/3939089196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d00c88c0 0x7f99d00c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99ce59c640 1 -- 172.21.15.94:0/3939089196 >> v2:172.21.15.145:3300/0 conn(0x7f99d00c5880 msgr2=0x7f99d0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99ce59c640 1 --2- 172.21.15.94:0/3939089196 >> v2:172.21.15.145:3300/0 conn(0x7f99d00c5880 0x7f99d0158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99ce59c640 1 -- 172.21.15.94:0/3939089196 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d00c7670 con 0x7f99d014f360 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99ced9d640 1 --2- 172.21.15.94:0/3939089196 >> v2:172.21.15.145:3300/0 conn(0x7f99d00c5880 0x7f99d0158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99ce59c640 1 --2- 172.21.15.94:0/3939089196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d014f360 0x7f99d0155c10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f99d008cc10 tx=0x7f99c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34883ca8666c7b64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.870+0000 7f99cdd9b640 1 -- 172.21.15.94:0/3939089196 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c002e070 con 0x7f99d014f360 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99cdd9b640 1 -- 172.21.15.94:0/3939089196 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c0037070 con 0x7f99d014f360 2024-03-06T07:27:14.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99cdd9b640 1 -- 172.21.15.94:0/3939089196 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0031a00 con 0x7f99d014f360 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 -- 172.21.15.94:0/3939089196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d014f360 msgr2=0x7f99d0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/3939089196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d014f360 0x7f99d0155c10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f99d008cc10 tx=0x7f99c002b880 comp rx=0 tx=0).stop 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 -- 172.21.15.94:0/3939089196 shutdown_connections 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/3939089196 >> v2:172.21.15.145:3300/0 conn(0x7f99d00c5880 0x7f99d0158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/3939089196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d00c88c0 0x7f99d00c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/3939089196 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d014f360 0x7f99d0155c10 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 -- 172.21.15.94:0/3939089196 >> 172.21.15.94:0/3939089196 conn(0x7f99d00ac570 msgr2=0x7f99d00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 -- 172.21.15.94:0/3939089196 shutdown_connections 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.871+0000 7f99d4c9d640 1 -- 172.21.15.94:0/3939089196 wait complete. 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.872+0000 7f99d4c9d640 1 Processor -- start 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.872+0000 7f99d4c9d640 1 -- start start 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99d4c9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99d00c5880 0x7f99d014c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99d4c9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d00c88c0 0x7f99d00c0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99d4c9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d014f360 0x7f99d014ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99d4c9d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d01590a0 con 0x7f99d00c88c0 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99d4c9d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d00c7c30 con 0x7f99d00c5880 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99d4c9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d0159220 con 0x7f99d014f360 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d00c88c0 0x7f99d00c0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d00c88c0 0x7f99d00c0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54962/0 (socket says 172.21.15.94:54962) 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 -- 172.21.15.94:0/2456017164 learned_addr learned my addr 172.21.15.94:0/2456017164 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99ced9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d014f360 0x7f99d014ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99ce59c640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99d00c5880 0x7f99d014c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 -- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d014f360 msgr2=0x7f99d014ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d014f360 0x7f99d014ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 -- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99d00c5880 msgr2=0x7f99d014c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99d00c5880 0x7f99d014c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99c7fff640 1 -- 172.21.15.94:0/2456017164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d014c630 con 0x7f99d00c88c0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.873+0000 7f99ced9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d014f360 0x7f99d014ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.874+0000 7f99c7fff640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d00c88c0 0x7f99d00c0db0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f99b800d7d0 tx=0x7f99b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.874+0000 7f99c6ffd640 1 -- 172.21.15.94:0/2456017164 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b800b6c0 con 0x7f99d00c88c0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.874+0000 7f99c6ffd640 1 -- 172.21.15.94:0/2456017164 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b8019070 con 0x7f99d00c88c0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.874+0000 7f99c6ffd640 1 -- 172.21.15.94:0/2456017164 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b800bac0 con 0x7f99d00c88c0 2024-03-06T07:27:14.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.874+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d00bea60 con 0x7f99d00c88c0 2024-03-06T07:27:14.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.874+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d00bf020 con 0x7f99d00c88c0 2024-03-06T07:27:14.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.875+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d004f330 con 0x7f99d00c88c0 2024-03-06T07:27:14.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.875+0000 7f99c6ffd640 1 -- 172.21.15.94:0/2456017164 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99b800bc60 con 0x7f99d00c88c0 2024-03-06T07:27:14.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.875+0000 7f99c6ffd640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99b006b4d0 0x7f99b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.876+0000 7f99c6ffd640 1 -- 172.21.15.94:0/2456017164 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f99b808a4f0 con 0x7f99d00c88c0 2024-03-06T07:27:14.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.876+0000 7f99ce59c640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99b006b4d0 0x7f99b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.876+0000 7f99ce59c640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99b006b4d0 0x7f99b006d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f99c0036000 tx=0x7f99c003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:14.880+0000 7f99c6ffd640 1 -- 172.21.15.94:0/2456017164 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99d014c630 con 0x7f99d00c88c0 2024-03-06T07:27:15.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.050+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99d00c7f10 con 0x7f99d00c88c0 2024-03-06T07:27:15.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.051+0000 7f99c6ffd640 1 -- 172.21.15.94:0/2456017164 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+13174 (secure 0 0 0) 0x7f99b8059530 con 0x7f99d00c88c0 2024-03-06T07:27:15.087 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:15.087 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":205,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:27:14.377889+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:15.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99b006b4d0 msgr2=0x7f99b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99b006b4d0 0x7f99b006d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f99c0036000 tx=0x7f99c003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d00c88c0 msgr2=0x7f99d00c0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d00c88c0 0x7f99d00c0db0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f99b800d7d0 tx=0x7f99b800dc90 comp rx=0 tx=0).stop 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 shutdown_connections 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f99b006b4d0 0x7f99b006d990 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99d014f360 0x7f99d014ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f99d00c88c0 0x7f99d00c0db0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 --2- 172.21.15.94:0/2456017164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99d00c5880 0x7f99d014c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 >> 172.21.15.94:0/2456017164 conn(0x7f99d00ac570 msgr2=0x7f99d014d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 shutdown_connections 2024-03-06T07:27:15.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.053+0000 7f99d4c9d640 1 -- 172.21.15.94:0/2456017164 wait complete. 2024-03-06T07:27:15.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:15.090 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.191+0000 7f7900a15640 1 Processor -- start 2024-03-06T07:27:15.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.191+0000 7f7900a15640 1 -- start start 2024-03-06T07:27:15.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f7900a15640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78fc0a97c0 0x7f78fc0c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f7900a15640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc154330 con 0x7f78fc1538c0 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f7900a15640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc153dc0 con 0x7f78fc0ab310 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f7900a15640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78fc0c46a0 con 0x7f78fc0a97c0 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f1d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f78fc1538c0 0x7f78fc1494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37860/0 (socket says 172.21.15.94:37860) 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f1d9b640 1 -- 172.21.15.94:0/2295292532 learned_addr learned my addr 172.21.15.94:0/2295292532 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071110125 0 0) 0x7f78fc154330 con 0x7f78fc1538c0 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e8003520 con 0x7f78fc1538c0 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78fad9d640 1 --2- 172.21.15.94:0/2295292532 >> v2:172.21.15.145:3300/0 conn(0x7f78fc0a97c0 0x7f78fc0c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 221310870 0 0) 0x7f78e8003520 con 0x7f78fc1538c0 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 >> v2:172.21.15.145:3300/0 conn(0x7f78fc0a97c0 msgr2=0x7f78fc0c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 --2- 172.21.15.94:0/2295292532 >> v2:172.21.15.145:3300/0 conn(0x7f78fc0a97c0 0x7f78fc0c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 >> v1:172.21.15.145:6789/0 conn(0x7f78fc0ab310 legacy=0x7f78fc0ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc0c5d30 con 0x7f78fc1538c0 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f78e4001670 con 0x7f78fc1538c0 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 >> v1:172.21.15.94:6789/0 conn(0x7f78fc1538c0 legacy=0x7f78fc1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e80039b0 0x7f78e8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78e8004800 0x7f78e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78e80052e0 0x7f78e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0c46a0 con 0x7f78e8004800 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e800c190 con 0x7f78e80039b0 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e800c390 con 0x7f78e80052e0 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.192+0000 7f78f1d9b640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78e8004800 0x7f78e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78fa59c640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e80039b0 0x7f78e8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78fad9d640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78e80052e0 0x7f78e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78f1d9b640 1 -- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78e80052e0 msgr2=0x7f78e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78f1d9b640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78e80052e0 0x7f78e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78f1d9b640 1 -- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e80039b0 msgr2=0x7f78e8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78f1d9b640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e80039b0 0x7f78e8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78f1d9b640 1 -- 172.21.15.94:0/2295292532 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e8003520 con 0x7f78e8004800 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78fad9d640 1 -- 172.21.15.94:0/2295292532 reap_dead start 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.193+0000 7f78f1d9b640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78e8004800 0x7f78e8004c80 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f78fc153cf0 tx=0x7f78e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da146f5d7f02e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4007d10 con 0x7f78e8004800 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4030070 con 0x7f78e8004800 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f78f9d9b640 1 -- 172.21.15.94:0/2295292532 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4005d70 con 0x7f78e8004800 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f7900a15640 1 -- 172.21.15.94:0/2295292532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78e8004800 msgr2=0x7f78e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f7900a15640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78e8004800 0x7f78e8004c80 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f78fc153cf0 tx=0x7f78e8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f7900a15640 1 -- 172.21.15.94:0/2295292532 shutdown_connections 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f7900a15640 1 --2- 172.21.15.94:0/2295292532 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78e8004800 0x7f78e8004c80 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f7900a15640 1 -- 172.21.15.94:0/2295292532 >> 172.21.15.94:0/2295292532 conn(0x7f78fc0c6710 msgr2=0x7f78fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f7900a15640 1 -- 172.21.15.94:0/2295292532 shutdown_connections 2024-03-06T07:27:15.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.194+0000 7f7900a15640 1 -- 172.21.15.94:0/2295292532 wait complete. 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 Processor -- start 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 -- start start 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e8004800 0x7f78fc1e0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc1de070 0x7f78fc1de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78fc14a900 0x7f78fc14ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0ab6f0 con 0x7f78fc1de070 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0c62f0 con 0x7f78e8004800 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f7900a15640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc0ab870 con 0x7f78fc14a900 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f78f1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc1de070 0x7f78fc1de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f78f1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc1de070 0x7f78fc1de4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54974/0 (socket says 172.21.15.94:54974) 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f78f1d9b640 1 -- 172.21.15.94:0/1563628607 learned_addr learned my addr 172.21.15.94:0/1563628607 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.196+0000 7f78fa59c640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e8004800 0x7f78fc1e0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78fad9d640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78fc14a900 0x7f78fc14ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f1d9b640 1 -- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78fc14a900 msgr2=0x7f78fc14ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f1d9b640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78fc14a900 0x7f78fc14ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f1d9b640 1 -- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e8004800 msgr2=0x7f78fc1e0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f1d9b640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e8004800 0x7f78fc1e0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f1d9b640 1 -- 172.21.15.94:0/1563628607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc14b2c0 con 0x7f78fc1de070 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f1d9b640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc1de070 0x7f78fc1de4d0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f78e402c440 tx=0x7f78fc0c4610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f37fe640 1 -- 172.21.15.94:0/1563628607 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4045070 con 0x7f78fc1de070 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f37fe640 1 -- 172.21.15.94:0/1563628607 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e402cbe0 con 0x7f78fc1de070 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f78f37fe640 1 -- 172.21.15.94:0/1563628607 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4001c50 con 0x7f78fc1de070 2024-03-06T07:27:15.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc0c1ac0 con 0x7f78fc1de070 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.197+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc0c2080 con 0x7f78fc1de070 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.198+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78fc04f330 con 0x7f78fc1de070 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.198+0000 7f78f37fe640 1 -- 172.21.15.94:0/1563628607 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78e4007070 con 0x7f78fc1de070 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.199+0000 7f78f37fe640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78dc06b4d0 0x7f78dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.199+0000 7f78f37fe640 1 -- 172.21.15.94:0/1563628607 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f78e40aaf20 con 0x7f78fc1de070 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.199+0000 7f78fa59c640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78dc06b4d0 0x7f78dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.200+0000 7f78fa59c640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78dc06b4d0 0x7f78dc06d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f78ec004f80 tx=0x7f78ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.203+0000 7f78f37fe640 1 -- 172.21.15.94:0/1563628607 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78fc14b2c0 con 0x7f78fc1de070 2024-03-06T07:27:15.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.373+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78fc0a65d0 con 0x7f78fc1de070 2024-03-06T07:27:15.376 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.373+0000 7f78f37fe640 1 -- 172.21.15.94:0/1563628607 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v205) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78e40799a0 con 0x7f78fc1de070 2024-03-06T07:27:15.376 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78dc06b4d0 msgr2=0x7f78dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78dc06b4d0 0x7f78dc06d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f78ec004f80 tx=0x7f78ec009040 comp rx=0 tx=0).stop 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc1de070 msgr2=0x7f78fc1de4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc1de070 0x7f78fc1de4d0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f78e402c440 tx=0x7f78fc0c4610 comp rx=0 tx=0).stop 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 shutdown_connections 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78dc06b4d0 0x7f78dc06d990 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78fc14a900 0x7f78fc14ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc1de070 0x7f78fc1de4d0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 --2- 172.21.15.94:0/1563628607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78e8004800 0x7f78fc1e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 >> 172.21.15.94:0/1563628607 conn(0x7f78fc0c6710 msgr2=0x7f78fc14baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.378 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 shutdown_connections 2024-03-06T07:27:15.379 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.375+0000 7f7900a15640 1 -- 172.21.15.94:0/1563628607 wait complete. 2024-03-06T07:27:15.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:15.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.487+0000 7f787bd56640 1 Processor -- start 2024-03-06T07:27:15.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.487+0000 7f787bd56640 1 -- start start 2024-03-06T07:27:15.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787bd56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f787414f790 0x7f78741529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787bd56640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78740c1aa0 0x7f78740c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787bd56640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78740c5c60 0x7f7874155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787bd56640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78740c2680 con 0x7f78740c5c60 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787bd56640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78740c28d0 con 0x7f78740c1aa0 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787bd56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78740c2bb0 con 0x7f787414f790 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787a2f3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78740c5c60 0x7f7874155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787a2f3640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78740c5c60 0x7f7874155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54980/0 (socket says 172.21.15.94:54980) 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787a2f3640 1 -- 172.21.15.94:0/1713741765 learned_addr learned my addr 172.21.15.94:0/1713741765 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.488+0000 7f787a2f3640 1 -- 172.21.15.94:0/1713741765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f787414f790 msgr2=0x7f78741529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f787a2f3640 1 --2- 172.21.15.94:0/1713741765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f787414f790 0x7f78741529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f787a2f3640 1 -- 172.21.15.94:0/1713741765 >> v2:172.21.15.145:3300/0 conn(0x7f78740c1aa0 msgr2=0x7f78740c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f787a2f3640 1 --2- 172.21.15.94:0/1713741765 >> v2:172.21.15.145:3300/0 conn(0x7f78740c1aa0 0x7f78740c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f787a2f3640 1 -- 172.21.15.94:0/1713741765 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78740c2e90 con 0x7f78740c5c60 2024-03-06T07:27:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f787a2f3640 1 --2- 172.21.15.94:0/1713741765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78740c5c60 0x7f7874155a40 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f78740c76f0 tx=0x7f787002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb27e4b403bf626 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f7878af0640 1 -- 172.21.15.94:0/1713741765 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870035020 con 0x7f78740c5c60 2024-03-06T07:27:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f7878af0640 1 -- 172.21.15.94:0/1713741765 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7870031870 con 0x7f78740c5c60 2024-03-06T07:27:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.489+0000 7f7878af0640 1 -- 172.21.15.94:0/1713741765 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870031c40 con 0x7f78740c5c60 2024-03-06T07:27:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 -- 172.21.15.94:0/1713741765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78740c5c60 msgr2=0x7f7874155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 --2- 172.21.15.94:0/1713741765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78740c5c60 0x7f7874155a40 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f78740c76f0 tx=0x7f787002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:15.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 -- 172.21.15.94:0/1713741765 shutdown_connections 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 --2- 172.21.15.94:0/1713741765 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78740c5c60 0x7f7874155a40 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 --2- 172.21.15.94:0/1713741765 >> v2:172.21.15.145:3300/0 conn(0x7f78740c1aa0 0x7f78740c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 --2- 172.21.15.94:0/1713741765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f787414f790 0x7f78741529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 -- 172.21.15.94:0/1713741765 >> 172.21.15.94:0/1713741765 conn(0x7f78740c6710 msgr2=0x7f78740c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 -- 172.21.15.94:0/1713741765 shutdown_connections 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.490+0000 7f787bd56640 1 -- 172.21.15.94:0/1713741765 wait complete. 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.491+0000 7f787bd56640 1 Processor -- start 2024-03-06T07:27:15.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.491+0000 7f787bd56640 1 -- start start 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787bd56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78740c1aa0 0x7f787414ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787bd56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78740c5c60 0x7f787414b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787bd56640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f787414f790 0x7f7874149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787bd56640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787406cc00 con 0x7f787414f790 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787bd56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78740c0d60 con 0x7f78740c5c60 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787bd56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787406cd80 con 0x7f78740c1aa0 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f787414f790 0x7f7874149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f787414f790 0x7f7874149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54988/0 (socket says 172.21.15.94:54988) 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 -- 172.21.15.94:0/2531031109 learned_addr learned my addr 172.21.15.94:0/2531031109 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f78792f1640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78740c5c60 0x7f787414b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 -- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78740c1aa0 msgr2=0x7f787414ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f7879af2640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78740c1aa0 0x7f787414ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78740c1aa0 0x7f787414ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 -- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78740c5c60 msgr2=0x7f787414b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78740c5c60 0x7f787414b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 -- 172.21.15.94:0/2531031109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787414abd0 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f78792f1640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78740c5c60 0x7f787414b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f7879af2640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78740c1aa0 0x7f787414ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.492+0000 7f787a2f3640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f787414f790 0x7f7874149db0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f787002b950 tx=0x7f78700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.493+0000 7f786a7fc640 1 -- 172.21.15.94:0/2531031109 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78700443f0 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.493+0000 7f786a7fc640 1 -- 172.21.15.94:0/2531031109 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78700445b0 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.493+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787414d2f0 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.493+0000 7f786a7fc640 1 -- 172.21.15.94:0/2531031109 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78700449a0 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.493+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787414d8b0 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.494+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787404f330 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.494+0000 7f786a7fc640 1 -- 172.21.15.94:0/2531031109 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7870038020 con 0x7f787414f790 2024-03-06T07:27:15.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.495+0000 7f786a7fc640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f785806b4d0 0x7f785806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.495+0000 7f786a7fc640 1 -- 172.21.15.94:0/2531031109 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f78700ac360 con 0x7f787414f790 2024-03-06T07:27:15.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.495+0000 7f7879af2640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f785806b4d0 0x7f785806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.495+0000 7f7879af2640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f785806b4d0 0x7f785806d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f786c007eb0 tx=0x7f786c0073c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.499+0000 7f786a7fc640 1 -- 172.21.15.94:0/2531031109 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f787003f070 con 0x7f787414f790 2024-03-06T07:27:15.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.672+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f78740aa6d0 con 0x7f787414f790 2024-03-06T07:27:15.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.673+0000 7f786a7fc640 1 -- 172.21.15.94:0/2531031109 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v205) v1 ==== 84+0+65 (secure 0 0 0) 0x7f787007b3a0 con 0x7f787414f790 2024-03-06T07:27:15.675 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:15.675 INFO:teuthology.orchestra.run.smithi094.stdout:[ 2024-03-06T07:27:15.675 INFO:teuthology.orchestra.run.smithi094.stdout: { 2024-03-06T07:27:15.675 INFO:teuthology.orchestra.run.smithi094.stdout: "poolnum": 1, 2024-03-06T07:27:15.675 INFO:teuthology.orchestra.run.smithi094.stdout: "poolname": ".mgr" 2024-03-06T07:27:15.675 INFO:teuthology.orchestra.run.smithi094.stdout: } 2024-03-06T07:27:15.675 INFO:teuthology.orchestra.run.smithi094.stdout:] 2024-03-06T07:27:15.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f785806b4d0 msgr2=0x7f785806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f785806b4d0 0x7f785806d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f786c007eb0 tx=0x7f786c0073c0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f787414f790 msgr2=0x7f7874149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f787414f790 0x7f7874149db0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f787002b950 tx=0x7f78700046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 shutdown_connections 2024-03-06T07:27:15.677 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f785806b4d0 0x7f785806d990 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f787414f790 0x7f7874149db0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78740c5c60 0x7f787414b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 --2- 172.21.15.94:0/2531031109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78740c1aa0 0x7f787414ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 >> 172.21.15.94:0/2531031109 conn(0x7f78740c6710 msgr2=0x7f78740c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 shutdown_connections 2024-03-06T07:27:15.678 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.675+0000 7f787bd56640 1 -- 172.21.15.94:0/2531031109 wait complete. 2024-03-06T07:27:15.687 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:15.688 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:15.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.790+0000 7f760f017640 1 Processor -- start 2024-03-06T07:27:15.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.790+0000 7f760f017640 1 -- start start 2024-03-06T07:27:15.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f760f017640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 0x7f76081521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f760f017640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760814ef90 con 0x7f760814ebb0 2024-03-06T07:27:15.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f760f017640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7608155780 con 0x7f76080c28d0 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f760f017640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7608155530 con 0x7f76080a9980 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f760cdb3640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f760814ebb0 0x7f76080a9fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37870/0 (socket says 172.21.15.94:37870) 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f760cdb3640 1 -- 172.21.15.94:0/4060431593 learned_addr learned my addr 172.21.15.94:0/4060431593 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268388259 0 0) 0x7f75f8001110 con 0x7f760814ebb0 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f7607fff640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 0x7f76081521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f0003520 con 0x7f760814ebb0 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302523757 0 0) 0x7f75f0003520 con 0x7f760814ebb0 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 msgr2=0x7f76081521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.791+0000 7f76077fe640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 0x7f76081521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 >> v1:172.21.15.145:6789/0 conn(0x7f76080c28d0 legacy=0x7f760806dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76081516d0 con 0x7f760814ebb0 2024-03-06T07:27:15.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f75f8001840 con 0x7f760814ebb0 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 >> v1:172.21.15.94:6789/0 conn(0x7f760814ebb0 legacy=0x7f76080a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f75f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 0x7f75f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75f0004d40 0x7f75f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608155530 con 0x7f75f00039e0 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f000baf0 con 0x7f75f0004d40 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f000bcf0 con 0x7f76080a9980 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f760cdb3640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f75f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f7607fff640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 0x7f75f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f760d5b4640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75f0004d40 0x7f75f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f760cdb3640 1 -- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 msgr2=0x7f75f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f760cdb3640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76080a9980 0x7f75f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f760cdb3640 1 -- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75f0004d40 msgr2=0x7f75f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f760cdb3640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75f0004d40 0x7f75f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.792+0000 7f760cdb3640 1 -- 172.21.15.94:0/4060431593 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f0003520 con 0x7f75f00039e0 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760cdb3640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f75f0003e00 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f75f800c4c0 tx=0x7f75f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b86df171925f198 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f800c0e0 con 0x7f75f00039e0 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f8030070 con 0x7f75f00039e0 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f76077fe640 1 -- 172.21.15.94:0/4060431593 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f802ce60 con 0x7f75f00039e0 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760f017640 1 -- 172.21.15.94:0/4060431593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 msgr2=0x7f75f0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760f017640 1 --2- 172.21.15.94:0/4060431593 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f75f0003e00 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f75f800c4c0 tx=0x7f75f0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760d5b4640 1 -- 172.21.15.94:0/4060431593 reap_dead start 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760f017640 1 -- 172.21.15.94:0/4060431593 shutdown_connections 2024-03-06T07:27:15.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760f017640 1 -- 172.21.15.94:0/4060431593 >> 172.21.15.94:0/4060431593 conn(0x7f76080c65b0 msgr2=0x7f76080c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760f017640 1 -- 172.21.15.94:0/4060431593 shutdown_connections 2024-03-06T07:27:15.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.793+0000 7f760f017640 1 -- 172.21.15.94:0/4060431593 wait complete. 2024-03-06T07:27:15.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.794+0000 7f760f017640 1 Processor -- start 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.794+0000 7f760f017640 1 -- start start 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760f017640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f7608071d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760f017640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7608072270 0x7f76080a6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760f017640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f760814e0c0 0x7f76080a7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760f017640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608151c90 con 0x7f75f00039e0 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760f017640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608155a60 con 0x7f760814e0c0 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760f017640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76080c2d80 con 0x7f7608072270 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760cdb3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f7608071d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760cdb3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f7608071d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54994/0 (socket says 172.21.15.94:54994) 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760cdb3640 1 -- 172.21.15.94:0/3796747838 learned_addr learned my addr 172.21.15.94:0/3796747838 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.795+0000 7f760d5b4640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f760814e0c0 0x7f76080a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f7607fff640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7608072270 0x7f76080a6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760cdb3640 1 -- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7608072270 msgr2=0x7f76080a6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760cdb3640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7608072270 0x7f76080a6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760cdb3640 1 -- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f760814e0c0 msgr2=0x7f76080a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760cdb3640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f760814e0c0 0x7f76080a7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760cdb3640 1 -- 172.21.15.94:0/3796747838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76081ec8f0 con 0x7f75f00039e0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760d5b4640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f760814e0c0 0x7f76080a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760cdb3640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f7608071d30 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f75f800c520 tx=0x7f7608155950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f76057fa640 1 -- 172.21.15.94:0/3796747838 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f80049e0 con 0x7f75f00039e0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f76057fa640 1 -- 172.21.15.94:0/3796747838 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f8004b80 con 0x7f75f00039e0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76081ecbd0 con 0x7f75f00039e0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f76057fa640 1 -- 172.21.15.94:0/3796747838 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8007070 con 0x7f75f00039e0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.796+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76081ed190 con 0x7f75f00039e0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.797+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760804f330 con 0x7f75f00039e0 2024-03-06T07:27:15.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.797+0000 7f76057fa640 1 -- 172.21.15.94:0/3796747838 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75f8007210 con 0x7f75f00039e0 2024-03-06T07:27:15.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.798+0000 7f76057fa640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f75f406b500 0x7f75f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.798+0000 7f76057fa640 1 -- 172.21.15.94:0/3796747838 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f75f80abae0 con 0x7f75f00039e0 2024-03-06T07:27:15.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.798+0000 7f7607fff640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f75f406b500 0x7f75f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.799+0000 7f7607fff640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f75f406b500 0x7f75f406d9c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f76081468b0 tx=0x7f75fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.802+0000 7f76057fa640 1 -- 172.21.15.94:0/3796747838 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f760814ef90 con 0x7f75f00039e0 2024-03-06T07:27:15.869 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:15.871 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:15.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:15.980+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f760804f330 con 0x7f75f00039e0 2024-03-06T07:27:16.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.389+0000 7f76057fa640 1 -- 172.21.15.94:0/3796747838 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v206) v1 ==== 138+0+0 (secure 0 0 0) 0x7f75f807ab70 con 0x7f75f00039e0 2024-03-06T07:27:16.391 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f75f406b500 msgr2=0x7f75f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f75f406b500 0x7f75f406d9c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f76081468b0 tx=0x7f75fc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 msgr2=0x7f7608071d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f7608071d30 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f75f800c520 tx=0x7f7608155950 comp rx=0 tx=0).stop 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 shutdown_connections 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f75f406b500 0x7f75f406d9c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f760814e0c0 0x7f76080a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7608072270 0x7f76080a6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 --2- 172.21.15.94:0/3796747838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f75f00039e0 0x7f7608071d30 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.392+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 >> 172.21.15.94:0/3796747838 conn(0x7f76080c65b0 msgr2=0x7f760806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.393+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 shutdown_connections 2024-03-06T07:27:16.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.394+0000 7f760f017640 1 -- 172.21.15.94:0/3796747838 wait complete. 2024-03-06T07:27:16.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:16.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.514+0000 7fb0b4088640 1 Processor -- start 2024-03-06T07:27:16.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.515+0000 7fb0b4088640 1 -- start start 2024-03-06T07:27:16.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.515+0000 7fb0b4088640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 0x7fb0ac155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.515+0000 7fb0b4088640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0b0450 con 0x7fb0ac06ca70 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.515+0000 7fb0b4088640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0c77f0 con 0x7fb0ac0c5380 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.515+0000 7fb0b4088640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac06c020 con 0x7fb0ac0b0070 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.515+0000 7fb0b1623640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb0ac06ca70 0x7fb0ac0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37872/0 (socket says 172.21.15.94:37872) 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.515+0000 7fb0b1623640 1 -- 172.21.15.94:0/4017628589 learned_addr learned my addr 172.21.15.94:0/4017628589 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453273291 0 0) 0x7fb09c001110 con 0x7fb0ac06ca70 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb098003520 con 0x7fb0ac06ca70 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 0x7fb0ac155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705841672 0 0) 0x7fb098003520 con 0x7fb0ac06ca70 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 msgr2=0x7fb0ac155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 0x7fb0ac155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 >> v1:172.21.15.145:6789/0 conn(0x7fb0ac0c5380 legacy=0x7fb0ac14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0c7ad0 con 0x7fb0ac06ca70 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fb09c001840 con 0x7fb0ac06ca70 2024-03-06T07:27:16.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 >> v1:172.21.15.94:6789/0 conn(0x7fb0ac06ca70 legacy=0x7fb0ac0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0980039e0 0x7fb098003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 0x7fb098004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb098004d40 0x7fb0980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac06c020 con 0x7fb0980039e0 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09800bba0 con 0x7fb098004d40 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09800bda0 con 0x7fb0ac0b0070 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0980039e0 0x7fb098003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1623640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 0x7fb098004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b2625640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb098004d40 0x7fb0980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1e24640 1 -- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 msgr2=0x7fb098004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ac0b0070 0x7fb098004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1e24640 1 -- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb098004d40 msgr2=0x7fb0980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb098004d40 0x7fb0980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b1e24640 1 -- 172.21.15.94:0/4017628589 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb098003520 con 0x7fb0980039e0 2024-03-06T07:27:16.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.516+0000 7fb0b2625640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb098004d40 0x7fb0980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.517+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0980039e0 0x7fb098003e00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800ba30 tx=0x7fb098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41488c3e7ebdcf21 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.517+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8036020 con 0x7fb0980039e0 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.517+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a8032640 con 0x7fb0980039e0 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.517+0000 7fb0b0e22640 1 -- 172.21.15.94:0/4017628589 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a800b600 con 0x7fb0980039e0 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.518+0000 7fb0b4088640 1 -- 172.21.15.94:0/4017628589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0980039e0 msgr2=0x7fb098003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.518+0000 7fb0b4088640 1 --2- 172.21.15.94:0/4017628589 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0980039e0 0x7fb098003e00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800ba30 tx=0x7fb098005830 comp rx=0 tx=0).stop 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.518+0000 7fb0b2625640 1 -- 172.21.15.94:0/4017628589 reap_dead start 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.518+0000 7fb0b4088640 1 -- 172.21.15.94:0/4017628589 shutdown_connections 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.518+0000 7fb0b4088640 1 -- 172.21.15.94:0/4017628589 >> 172.21.15.94:0/4017628589 conn(0x7fb0ac0ac590 msgr2=0x7fb0ac0ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.518+0000 7fb0b4088640 1 -- 172.21.15.94:0/4017628589 shutdown_connections 2024-03-06T07:27:16.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.518+0000 7fb0b4088640 1 -- 172.21.15.94:0/4017628589 wait complete. 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.519+0000 7fb0b4088640 1 Processor -- start 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.519+0000 7fb0b4088640 1 -- start start 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0980039e0 0x7fb0ac1eec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ac14ece0 0x7fb0ac158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0ac1ef1a0 0x7fb0ac158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac06cef0 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac06c550 con 0x7fb0ac14ece0 2024-03-06T07:27:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac06da80 con 0x7fb0980039e0 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0ac1ef1a0 0x7fb0ac158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0ac1ef1a0 0x7fb0ac158b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55000/0 (socket says 172.21.15.94:55000) 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 -- 172.21.15.94:0/396223114 learned_addr learned my addr 172.21.15.94:0/396223114 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0980039e0 0x7fb0ac1eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b1623640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ac14ece0 0x7fb0ac158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 -- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0980039e0 msgr2=0x7fb0ac1eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0980039e0 0x7fb0ac1eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 -- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ac14ece0 msgr2=0x7fb0ac158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ac14ece0 0x7fb0ac158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 -- 172.21.15.94:0/396223114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac1ef650 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b1623640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ac14ece0 0x7fb0ac158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b2625640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0ac1ef1a0 0x7fb0ac158b40 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac0b07c0 tx=0x7fb0a400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0a27fc640 1 -- 172.21.15.94:0/396223114 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4003970 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0a27fc640 1 -- 172.21.15.94:0/396223114 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4018070 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac0c4630 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0a27fc640 1 -- 172.21.15.94:0/396223114 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4003d50 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.520+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac0be900 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.521+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac04f330 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.522+0000 7fb0a27fc640 1 -- 172.21.15.94:0/396223114 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0a40050d0 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.522+0000 7fb0a27fc640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09006b4d0 0x7fb09006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.522+0000 7fb0a27fc640 1 -- 172.21.15.94:0/396223114 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fb0a4089ba0 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.522+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09006b4d0 0x7fb09006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.523+0000 7fb0b1e24640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09006b4d0 0x7fb09006d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800ba00 tx=0x7fb0a803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.527+0000 7fb0a27fc640 1 -- 172.21.15.94:0/396223114 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0a4022020 con 0x7fb0ac1ef1a0 2024-03-06T07:27:16.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:16.700+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb0ac06d860 con 0x7fb0ac1ef1a0 2024-03-06T07:27:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.394+0000 7fb0a27fc640 1 -- 172.21.15.94:0/396223114 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v207) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb0a4058b00 con 0x7fb0ac1ef1a0 2024-03-06T07:27:17.396 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' created 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.397+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09006b4d0 msgr2=0x7fb09006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.397+0000 7fb0b4088640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09006b4d0 0x7fb09006d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800ba00 tx=0x7fb0a803d040 comp rx=0 tx=0).stop 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.397+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0ac1ef1a0 msgr2=0x7fb0ac158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.397+0000 7fb0b4088640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0ac1ef1a0 0x7fb0ac158b40 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac0b07c0 tx=0x7fb0a400eb90 comp rx=0 tx=0).stop 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.398+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 shutdown_connections 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.398+0000 7fb0b4088640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09006b4d0 0x7fb09006d990 secure :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800ba00 tx=0x7fb0a803d040 comp rx=0 tx=0).stop 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.398+0000 7fb0b4088640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0ac1ef1a0 0x7fb0ac158b40 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.398+0000 7fb0b4088640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ac14ece0 0x7fb0ac158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.398+0000 7fb0b4088640 1 --2- 172.21.15.94:0/396223114 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0980039e0 0x7fb0ac1eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.398+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 >> 172.21.15.94:0/396223114 conn(0x7fb0ac0ac590 msgr2=0x7fb0ac151660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.400+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 shutdown_connections 2024-03-06T07:27:17.406 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.400+0000 7fb0b4088640 1 -- 172.21.15.94:0/396223114 wait complete. 2024-03-06T07:27:17.428 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:17.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.532+0000 7f38cf4c2640 1 Processor -- start 2024-03-06T07:27:17.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cf4c2640 1 -- start start 2024-03-06T07:27:17.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cf4c2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f38c80c7f60 0x7f38c8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cf4c2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f38c80c6710 0x7f38c80c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cf4c2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c8073580 con 0x7f38c81505f0 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cf4c2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c8070b20 con 0x7f38c80c6710 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cf4c2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c80701f0 con 0x7f38c80c7f60 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cda5f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f38c80c6710 0x7f38c80c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cd25e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f38c81505f0 0x7f38c8072510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37888/0 (socket says 172.21.15.94:37888) 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cd25e640 1 -- 172.21.15.94:0/2363968982 learned_addr learned my addr 172.21.15.94:0/2363968982 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.533+0000 7f38cca5d640 1 --2- 172.21.15.94:0/2363968982 >> v2:172.21.15.145:3300/0 conn(0x7f38c80c7f60 0x7f38c8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.534+0000 7f38cda5f640 1 -- 172.21.15.94:0/2363968982 >> v2:172.21.15.145:3300/0 conn(0x7f38c80c7f60 msgr2=0x7f38c8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.534+0000 7f38cda5f640 1 --2- 172.21.15.94:0/2363968982 >> v2:172.21.15.145:3300/0 conn(0x7f38c80c7f60 0x7f38c8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.534+0000 7f38cda5f640 1 -- 172.21.15.94:0/2363968982 >> v1:172.21.15.94:6789/0 conn(0x7f38c81505f0 legacy=0x7f38c8072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.534+0000 7f38bffff640 1 -- 172.21.15.94:0/2363968982 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659251324 0 0) 0x7f38c8073580 con 0x7f38c81505f0 2024-03-06T07:27:17.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.534+0000 7f38cda5f640 1 -- 172.21.15.94:0/2363968982 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80704d0 con 0x7f38c80c6710 2024-03-06T07:27:17.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.534+0000 7f38cda5f640 1 --2- 172.21.15.94:0/2363968982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c80c6710 0x7f38c80c6b80 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f38c806fbc0 tx=0x7f38c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965a9c1bd4161b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38bffff640 1 -- 172.21.15.94:0/2363968982 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0039070 con 0x7f38c80c6710 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38bffff640 1 -- 172.21.15.94:0/2363968982 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c002cc80 con 0x7f38c80c6710 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38bffff640 1 -- 172.21.15.94:0/2363968982 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0034660 con 0x7f38c80c6710 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 -- 172.21.15.94:0/2363968982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c80c6710 msgr2=0x7f38c80c6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/2363968982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c80c6710 0x7f38c80c6b80 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f38c806fbc0 tx=0x7f38c002bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 -- 172.21.15.94:0/2363968982 shutdown_connections 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/2363968982 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c80c6710 0x7f38c80c6b80 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/2363968982 >> v2:172.21.15.145:3300/0 conn(0x7f38c80c7f60 0x7f38c8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 -- 172.21.15.94:0/2363968982 >> 172.21.15.94:0/2363968982 conn(0x7f38c80aa5c0 msgr2=0x7f38c80aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 -- 172.21.15.94:0/2363968982 shutdown_connections 2024-03-06T07:27:17.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.535+0000 7f38cf4c2640 1 -- 172.21.15.94:0/2363968982 wait complete. 2024-03-06T07:27:17.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.536+0000 7f38cf4c2640 1 Processor -- start 2024-03-06T07:27:17.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.536+0000 7f38cf4c2640 1 -- start start 2024-03-06T07:27:17.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cf4c2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38c80c6710 0x7f38c81536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cf4c2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c80c7f60 0x7f38c8156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cf4c2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c81505f0 0x7f38c814db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cf4c2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8072c20 con 0x7f38c81505f0 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cf4c2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c806fec0 con 0x7f38c80c7f60 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cf4c2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c8072da0 con 0x7f38c80c6710 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c81505f0 0x7f38c814db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c81505f0 0x7f38c814db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:55006/0 (socket says 172.21.15.94:55006) 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 -- 172.21.15.94:0/78079803 learned_addr learned my addr 172.21.15.94:0/78079803 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cca5d640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c80c7f60 0x7f38c8156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 -- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38c80c6710 msgr2=0x7f38c81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38c80c6710 0x7f38c81536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 -- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c80c7f60 msgr2=0x7f38c8156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c80c7f60 0x7f38c8156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 -- 172.21.15.94:0/78079803 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c8153c30 con 0x7f38c81505f0 2024-03-06T07:27:17.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.537+0000 7f38cda5f640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c81505f0 0x7f38c814db10 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f38c0037040 tx=0x7f38c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.538+0000 7f38bdffb640 1 -- 172.21.15.94:0/78079803 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0044020 con 0x7f38c81505f0 2024-03-06T07:27:17.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.538+0000 7f38bdffb640 1 -- 172.21.15.94:0/78079803 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c003a040 con 0x7f38c81505f0 2024-03-06T07:27:17.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.538+0000 7f38bdffb640 1 -- 172.21.15.94:0/78079803 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0039050 con 0x7f38c81505f0 2024-03-06T07:27:17.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.538+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c80c2960 con 0x7f38c81505f0 2024-03-06T07:27:17.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.538+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c80bdbe0 con 0x7f38c81505f0 2024-03-06T07:27:17.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.539+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c80c2e80 con 0x7f38c81505f0 2024-03-06T07:27:17.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.539+0000 7f38bdffb640 1 -- 172.21.15.94:0/78079803 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38c0003b90 con 0x7f38c81505f0 2024-03-06T07:27:17.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.540+0000 7f38bdffb640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38ac06b4d0 0x7f38ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.540+0000 7f38bdffb640 1 -- 172.21.15.94:0/78079803 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f38c00acaa0 con 0x7f38c81505f0 2024-03-06T07:27:17.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.540+0000 7f38cd25e640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38ac06b4d0 0x7f38ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.540+0000 7f38cd25e640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38ac06b4d0 0x7f38ac06d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f38c4007fd0 tx=0x7f38c4007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.544+0000 7f38bdffb640 1 -- 172.21.15.94:0/78079803 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38c004e020 con 0x7f38c81505f0 2024-03-06T07:27:17.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:17.751+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f38c804f330 con 0x7f38c81505f0 2024-03-06T07:27:18.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.396+0000 7f38bdffb640 1 -- 172.21.15.94:0/78079803 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 33 and data pool 34 v194) v1 ==== 174+0+0 (secure 0 0 0) 0x7f38c8153c30 con 0x7f38c81505f0 2024-03-06T07:27:18.399 INFO:teuthology.orchestra.run.smithi094.stderr:new fs with metadata pool 33 and data pool 34 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38ac06b4d0 msgr2=0x7f38ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38ac06b4d0 0x7f38ac06d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f38c4007fd0 tx=0x7f38c4007920 comp rx=0 tx=0).stop 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c81505f0 msgr2=0x7f38c814db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c81505f0 0x7f38c814db10 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f38c0037040 tx=0x7f38c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 shutdown_connections 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f38ac06b4d0 0x7f38ac06d990 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f38c81505f0 0x7f38c814db10 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38c80c7f60 0x7f38c8156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 --2- 172.21.15.94:0/78079803 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38c80c6710 0x7f38c81536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.400+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 >> 172.21.15.94:0/78079803 conn(0x7f38c80aa5c0 msgr2=0x7f38c80aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.401+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 shutdown_connections 2024-03-06T07:27:18.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.401+0000 7f38cf4c2640 1 -- 172.21.15.94:0/78079803 wait complete. 2024-03-06T07:27:18.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.532+0000 7faed0dd0640 1 Processor -- start 2024-03-06T07:27:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.532+0000 7faed0dd0640 1 -- start start 2024-03-06T07:27:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faed0dd0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faecc14f790 0x7faecc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faed0dd0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faecc0c1aa0 0x7faecc0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faed0dd0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0c28d0 con 0x7faecc0c5c60 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faed0dd0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0c2680 con 0x7faecc14f790 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faed0dd0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0c2bb0 con 0x7faecc0c1aa0 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faeca59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faecc14f790 0x7faecc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faeca59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faecc14f790 0x7faecc1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47710/0 (socket says 172.21.15.94:47710) 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faeca59c640 1 -- 172.21.15.94:0/525944418 learned_addr learned my addr 172.21.15.94:0/525944418 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faec9d9b640 1 --2- 172.21.15.94:0/525944418 >> v2:172.21.15.145:3300/0 conn(0x7faecc0c1aa0 0x7faecc0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faeca59c640 1 -- 172.21.15.94:0/525944418 >> v2:172.21.15.145:3300/0 conn(0x7faecc0c1aa0 msgr2=0x7faecc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faeca59c640 1 --2- 172.21.15.94:0/525944418 >> v2:172.21.15.145:3300/0 conn(0x7faecc0c1aa0 0x7faecc0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faeca59c640 1 -- 172.21.15.94:0/525944418 >> v1:172.21.15.145:6789/0 conn(0x7faecc0c5c60 legacy=0x7faecc155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.533+0000 7faeca59c640 1 -- 172.21.15.94:0/525944418 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0c2e90 con 0x7faecc14f790 2024-03-06T07:27:18.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.534+0000 7faeca59c640 1 --2- 172.21.15.94:0/525944418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc14f790 0x7faecc1529d0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7faecc070280 tx=0x7faebc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3eba5ee9809c82f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.534+0000 7faec959a640 1 -- 172.21.15.94:0/525944418 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc035020 con 0x7faecc14f790 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.534+0000 7faec959a640 1 -- 172.21.15.94:0/525944418 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc031870 con 0x7faecc14f790 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.534+0000 7faec959a640 1 -- 172.21.15.94:0/525944418 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc031c40 con 0x7faecc14f790 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 -- 172.21.15.94:0/525944418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc14f790 msgr2=0x7faecc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 --2- 172.21.15.94:0/525944418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc14f790 0x7faecc1529d0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7faecc070280 tx=0x7faebc02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 -- 172.21.15.94:0/525944418 shutdown_connections 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 --2- 172.21.15.94:0/525944418 >> v2:172.21.15.145:3300/0 conn(0x7faecc0c1aa0 0x7faecc0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 --2- 172.21.15.94:0/525944418 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc14f790 0x7faecc1529d0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 -- 172.21.15.94:0/525944418 >> 172.21.15.94:0/525944418 conn(0x7faecc0c6710 msgr2=0x7faecc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 -- 172.21.15.94:0/525944418 shutdown_connections 2024-03-06T07:27:18.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.535+0000 7faed0dd0640 1 -- 172.21.15.94:0/525944418 wait complete. 2024-03-06T07:27:18.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.536+0000 7faed0dd0640 1 Processor -- start 2024-03-06T07:27:18.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.536+0000 7faed0dd0640 1 -- start start 2024-03-06T07:27:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faecc0c1aa0 0x7faecc1513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc0c5c60 0x7faecc153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faecc14f790 0x7faecc1dda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc06cc00 con 0x7faecc0c5c60 2024-03-06T07:27:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0c0d60 con 0x7faecc14f790 2024-03-06T07:27:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc06cd80 con 0x7faecc0c1aa0 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc0c5c60 0x7faecc153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc0c5c60 0x7faecc153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47714/0 (socket says 172.21.15.94:47714) 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 -- 172.21.15.94:0/4098168878 learned_addr learned my addr 172.21.15.94:0/4098168878 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 -- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faecc0c1aa0 msgr2=0x7faecc1513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faecc0c1aa0 0x7faecc1513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 -- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faecc14f790 msgr2=0x7faecc1dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faecc14f790 0x7faecc1dda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 -- 172.21.15.94:0/4098168878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc1519a0 con 0x7faecc0c5c60 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faec9d9b640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc0c5c60 0x7faecc153f40 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7faec000d7d0 tx=0x7faec000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faebaffd640 1 -- 172.21.15.94:0/4098168878 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec000b6c0 con 0x7faecc0c5c60 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faebaffd640 1 -- 172.21.15.94:0/4098168878 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec0019070 con 0x7faecc0c5c60 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faebaffd640 1 -- 172.21.15.94:0/4098168878 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec000bac0 con 0x7faecc0c5c60 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc1ec3e0 con 0x7faecc0c5c60 2024-03-06T07:27:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.537+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc1ddf50 con 0x7faecc0c5c60 2024-03-06T07:27:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.538+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc04f330 con 0x7faecc0c5c60 2024-03-06T07:27:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.539+0000 7faebaffd640 1 -- 172.21.15.94:0/4098168878 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faec000bc60 con 0x7faecc0c5c60 2024-03-06T07:27:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.539+0000 7faebaffd640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faeac06b4d0 0x7faeac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.539+0000 7faebaffd640 1 -- 172.21.15.94:0/4098168878 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faec008a7e0 con 0x7faecc0c5c60 2024-03-06T07:27:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.540+0000 7faeca59c640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faeac06b4d0 0x7faeac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.540+0000 7faeca59c640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faeac06b4d0 0x7faeac06d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7faebc036000 tx=0x7faebc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.544+0000 7faebaffd640 1 -- 172.21.15.94:0/4098168878 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faec0059560 con 0x7faecc0c5c60 2024-03-06T07:27:18.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.717+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faecc0c1f40 con 0x7faecc0c5c60 2024-03-06T07:27:18.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.718+0000 7faebaffd640 1 -- 172.21.15.94:0/4098168878 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16901 (secure 0 0 0) 0x7faec0058e60 con 0x7faecc0c5c60 2024-03-06T07:27:18.720 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:18.720 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":208,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:27:17.753144+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:15.999555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:16.719665+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:18.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faeac06b4d0 msgr2=0x7faeac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faeac06b4d0 0x7faeac06d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7faebc036000 tx=0x7faebc030000 comp rx=0 tx=0).stop 2024-03-06T07:27:18.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc0c5c60 msgr2=0x7faecc153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc0c5c60 0x7faecc153f40 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7faec000d7d0 tx=0x7faec000dc90 comp rx=0 tx=0).stop 2024-03-06T07:27:18.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 shutdown_connections 2024-03-06T07:27:18.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faeac06b4d0 0x7faeac06d990 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faecc14f790 0x7faecc1dda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faecc0c5c60 0x7faecc153f40 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 --2- 172.21.15.94:0/4098168878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faecc0c1aa0 0x7faecc1513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 >> 172.21.15.94:0/4098168878 conn(0x7faecc0c6710 msgr2=0x7faecc0a78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 shutdown_connections 2024-03-06T07:27:18.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.720+0000 7faed0dd0640 1 -- 172.21.15.94:0/4098168878 wait complete. 2024-03-06T07:27:18.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:18.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.830+0000 7ff76b4a4640 1 Processor -- start 2024-03-06T07:27:18.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.830+0000 7ff76b4a4640 1 -- start start 2024-03-06T07:27:18.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff76b4a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff7640c1320 0x7ff7641566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff76b4a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7640c6000 0x7ff764156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff76b4a4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764153a60 con 0x7ff764150580 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff76b4a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff764153d40 con 0x7ff7640c1320 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff76b4a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76406c2c0 con 0x7ff7640c6000 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff768a3f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff7640c1320 0x7ff7641566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff769a41640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7640c6000 0x7ff764156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff769240640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff764150580 0x7ff76406dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:51590/0 (socket says 172.21.15.94:51590) 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.831+0000 7ff769240640 1 -- 172.21.15.94:0/3164554020 learned_addr learned my addr 172.21.15.94:0/3164554020 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.832+0000 7ff768a3f640 1 -- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7640c6000 msgr2=0x7ff764156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.832+0000 7ff768a3f640 1 --2- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7640c6000 0x7ff764156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.832+0000 7ff768a3f640 1 -- 172.21.15.94:0/3164554020 >> v1:172.21.15.145:6789/0 conn(0x7ff764150580 legacy=0x7ff76406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.832+0000 7ff75bfff640 1 -- 172.21.15.94:0/3164554020 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060572941 0 0) 0x7ff764153a60 con 0x7ff764150580 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.832+0000 7ff768a3f640 1 -- 172.21.15.94:0/3164554020 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76406c5a0 con 0x7ff7640c1320 2024-03-06T07:27:18.834 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.832+0000 7ff769a41640 1 --2- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7640c6000 0x7ff764156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff768a3f640 1 --2- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7641566b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff7540097a0 tx=0x7ff75402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=def50d47d8a6adf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff75bfff640 1 -- 172.21.15.94:0/3164554020 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754031780 con 0x7ff7640c1320 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff75bfff640 1 -- 172.21.15.94:0/3164554020 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff754037070 con 0x7ff7640c1320 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff75bfff640 1 -- 172.21.15.94:0/3164554020 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754031b40 con 0x7ff7640c1320 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 msgr2=0x7ff7641566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7641566b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff7540097a0 tx=0x7ff75402b820 comp rx=0 tx=0).stop 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3164554020 shutdown_connections 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7640c6000 0x7ff764156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3164554020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7641566b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3164554020 >> 172.21.15.94:0/3164554020 conn(0x7ff7640c6710 msgr2=0x7ff7640c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3164554020 shutdown_connections 2024-03-06T07:27:18.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.833+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3164554020 wait complete. 2024-03-06T07:27:18.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.834+0000 7ff76b4a4640 1 Processor -- start 2024-03-06T07:27:18.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.834+0000 7ff76b4a4640 1 -- start start 2024-03-06T07:27:18.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff76b4a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7640bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff76b4a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7640c6000 0x7ff764072310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff76b4a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff764150580 0x7ff764072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff76b4a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76406e5a0 con 0x7ff7640c6000 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff76b4a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76406e420 con 0x7ff7640c1320 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff76b4a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7640c0c40 con 0x7ff764150580 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff768a3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7640c6000 0x7ff764072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff768a3f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7640c6000 0x7ff764072310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47728/0 (socket says 172.21.15.94:47728) 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff768a3f640 1 -- 172.21.15.94:0/3428351066 learned_addr learned my addr 172.21.15.94:0/3428351066 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff769a41640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff764150580 0x7ff764072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.835+0000 7ff769240640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7640bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff768a3f640 1 -- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff764150580 msgr2=0x7ff764072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff768a3f640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff764150580 0x7ff764072850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff768a3f640 1 -- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 msgr2=0x7ff7640bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff768a3f640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7640bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff768a3f640 1 -- 172.21.15.94:0/3428351066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7641ef440 con 0x7ff7640c6000 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff769240640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7640bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff768a3f640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7640c6000 0x7ff764072310 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7ff7540097a0 tx=0x7ff7540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff759ffb640 1 -- 172.21.15.94:0/3428351066 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754004230 con 0x7ff7640c6000 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff759ffb640 1 -- 172.21.15.94:0/3428351066 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7540043d0 con 0x7ff7640c6000 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff759ffb640 1 -- 172.21.15.94:0/3428351066 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754003850 con 0x7ff7640c6000 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7641ee680 con 0x7ff7640c6000 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.836+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7641eec40 con 0x7ff7640c6000 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.837+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76414ed60 con 0x7ff7640c6000 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.837+0000 7ff759ffb640 1 -- 172.21.15.94:0/3428351066 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7540039f0 con 0x7ff7640c6000 2024-03-06T07:27:18.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.838+0000 7ff759ffb640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b500 0x7ff74806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.838+0000 7ff759ffb640 1 -- 172.21.15.94:0/3428351066 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff7540ac250 con 0x7ff7640c6000 2024-03-06T07:27:18.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.838+0000 7ff769240640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b500 0x7ff74806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.839+0000 7ff769240640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b500 0x7ff74806d9c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff760000f60 tx=0x7ff760009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:18.842+0000 7ff759ffb640 1 -- 172.21.15.94:0/3428351066 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff76414ed60 con 0x7ff7640c6000 2024-03-06T07:27:19.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.014+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff76404f330 con 0x7ff7640c6000 2024-03-06T07:27:19.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.015+0000 7ff759ffb640 1 -- 172.21.15.94:0/3428351066 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16901 (secure 0 0 0) 0x7ff754041070 con 0x7ff7640c6000 2024-03-06T07:27:19.017 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:19.017 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":208,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:27:17.753144+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:15.999555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:16.719665+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:19.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b500 msgr2=0x7ff74806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b500 0x7ff74806d9c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff760000f60 tx=0x7ff760009040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7640c6000 msgr2=0x7ff764072310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7640c6000 0x7ff764072310 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7ff7540097a0 tx=0x7ff7540046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 shutdown_connections 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b500 0x7ff74806d9c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff764150580 0x7ff764072850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7640c6000 0x7ff764072310 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 --2- 172.21.15.94:0/3428351066 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7640c1320 0x7ff7640bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 >> 172.21.15.94:0/3428351066 conn(0x7ff7640c6710 msgr2=0x7ff76406dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 shutdown_connections 2024-03-06T07:27:19.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.017+0000 7ff76b4a4640 1 -- 172.21.15.94:0/3428351066 wait complete. 2024-03-06T07:27:19.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.131+0000 7f39515bc640 1 Processor -- start 2024-03-06T07:27:19.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.131+0000 7f39515bc640 1 -- start start 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f39515bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f394c14bcc0 0x7f394c155a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f39515bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f39515bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f394c0c5910 0x7f394c158a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f39515bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394c0c7f10 con 0x7f394c0c5910 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f39515bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394c0c75e0 con 0x7f394c14bcc0 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f39515bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c0c78c0 con 0x7f394c0be150 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f394c0c5910 0x7f394c158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.136 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f394c0c5910 0x7f394c158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47744/0 (socket says 172.21.15.94:47744) 2024-03-06T07:27:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 -- 172.21.15.94:0/3422313895 learned_addr learned my addr 172.21.15.94:0/3422313895 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394affd640 1 --2- 172.21.15.94:0/3422313895 >> v2:172.21.15.145:3300/0 conn(0x7f394c14bcc0 0x7f394c155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394a7fc640 1 --2- 172.21.15.94:0/3422313895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 -- 172.21.15.94:0/3422313895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 msgr2=0x7f394c0c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 --2- 172.21.15.94:0/3422313895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0c53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 -- 172.21.15.94:0/3422313895 >> v2:172.21.15.145:3300/0 conn(0x7f394c14bcc0 msgr2=0x7f394c155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.137 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 --2- 172.21.15.94:0/3422313895 >> v2:172.21.15.145:3300/0 conn(0x7f394c14bcc0 0x7f394c155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.132+0000 7f394b7fe640 1 -- 172.21.15.94:0/3422313895 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c0c7ba0 con 0x7f394c0c5910 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.133+0000 7f394b7fe640 1 --2- 172.21.15.94:0/3422313895 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c158a10 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f394c08c180 tx=0x7f394402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e950f297dfe5fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.133+0000 7f3949ffb640 1 -- 172.21.15.94:0/3422313895 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394402cb00 con 0x7f394c0c5910 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.133+0000 7f3949ffb640 1 -- 172.21.15.94:0/3422313895 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394402cca0 con 0x7f394c0c5910 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.133+0000 7f3949ffb640 1 -- 172.21.15.94:0/3422313895 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944034610 con 0x7f394c0c5910 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 -- 172.21.15.94:0/3422313895 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 msgr2=0x7f394c158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 --2- 172.21.15.94:0/3422313895 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c158a10 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f394c08c180 tx=0x7f394402bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 -- 172.21.15.94:0/3422313895 shutdown_connections 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 --2- 172.21.15.94:0/3422313895 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c158a10 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 --2- 172.21.15.94:0/3422313895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 --2- 172.21.15.94:0/3422313895 >> v2:172.21.15.145:3300/0 conn(0x7f394c14bcc0 0x7f394c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 -- 172.21.15.94:0/3422313895 >> 172.21.15.94:0/3422313895 conn(0x7f394c0ac600 msgr2=0x7f394c0aca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.138 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 -- 172.21.15.94:0/3422313895 shutdown_connections 2024-03-06T07:27:19.139 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.134+0000 7f39515bc640 1 -- 172.21.15.94:0/3422313895 wait complete. 2024-03-06T07:27:19.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 Processor -- start 2024-03-06T07:27:19.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 -- start start 2024-03-06T07:27:19.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c0be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f394c14bcc0 0x7f394c0beec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c1590d0 con 0x7f394c0c5910 2024-03-06T07:27:19.142 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c158f50 con 0x7f394c14bcc0 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.135+0000 7f39515bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c159250 con 0x7f394c0be150 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c0be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c0be980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47754/0 (socket says 172.21.15.94:47754) 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 -- 172.21.15.94:0/2039968348 learned_addr learned my addr 172.21.15.94:0/2039968348 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394b7fe640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f394c14bcc0 0x7f394c0beec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394affd640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 -- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 msgr2=0x7f394c0aad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0aad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 -- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f394c14bcc0 msgr2=0x7f394c0beec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f394c14bcc0 0x7f394c0beec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 -- 172.21.15.94:0/2039968348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c1df310 con 0x7f394c0c5910 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394affd640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f394a7fc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c0be980 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f394000d830 tx=0x7f394000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.143 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f39337fe640 1 -- 172.21.15.94:0/2039968348 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940018020 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f39337fe640 1 -- 172.21.15.94:0/2039968348 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39400096a0 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f39337fe640 1 -- 172.21.15.94:0/2039968348 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940009a70 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c1eee50 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.136+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c1df5d0 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.137+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394c0c6a60 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.138+0000 7f39337fe640 1 -- 172.21.15.94:0/2039968348 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f394001b020 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.138+0000 7f39337fe640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393806b4d0 0x7f393806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.138+0000 7f39337fe640 1 -- 172.21.15.94:0/2039968348 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f394008a700 con 0x7f394c0c5910 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.138+0000 7f394affd640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393806b4d0 0x7f393806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.144 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.139+0000 7f394affd640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393806b4d0 0x7f393806d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f394c1e2830 tx=0x7f393c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.145 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.143+0000 7f39337fe640 1 -- 172.21.15.94:0/2039968348 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f394c0c6a60 con 0x7f394c0c5910 2024-03-06T07:27:19.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.351+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f394c04f330 con 0x7f394c0c5910 2024-03-06T07:27:19.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.417+0000 7f39337fe640 1 -- 172.21.15.94:0/2039968348 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v195) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3940022030 con 0x7f394c0c5910 2024-03-06T07:27:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393806b4d0 msgr2=0x7f393806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393806b4d0 0x7f393806d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f394c1e2830 tx=0x7f393c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 msgr2=0x7f394c0be980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c0be980 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f394000d830 tx=0x7f394000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 shutdown_connections 2024-03-06T07:27:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f393806b4d0 0x7f393806d990 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f394c14bcc0 0x7f394c0beec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f394c0c5910 0x7f394c0be980 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 --2- 172.21.15.94:0/2039968348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f394c0be150 0x7f394c0aad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 >> 172.21.15.94:0/2039968348 conn(0x7f394c0ac600 msgr2=0x7f394c0ac9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 shutdown_connections 2024-03-06T07:27:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.419+0000 7f39515bc640 1 -- 172.21.15.94:0/2039968348 wait complete. 2024-03-06T07:27:19.432 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:19.432 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:19.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.533+0000 7fe6e0fc9640 1 Processor -- start 2024-03-06T07:27:19.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.533+0000 7fe6e0fc9640 1 -- start start 2024-03-06T07:27:19.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6e0fc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06b3a0 0x7fe6dc158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6e0fc9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6dc159160 0x7fe6dc15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6e0fc9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6dc06b800 con 0x7fe6dc06afc0 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6e0fc9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6dc158790 con 0x7fe6dc159160 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6e0fc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0c5840 con 0x7fe6dc06b3a0 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6dad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6dc159160 0x7fe6dc15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6dad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe6dc159160 0x7fe6dc15fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47758/0 (socket says 172.21.15.94:47758) 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6dad9d640 1 -- 172.21.15.94:0/4184888485 learned_addr learned my addr 172.21.15.94:0/4184888485 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6d9d9b640 1 --2- 172.21.15.94:0/4184888485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06b3a0 0x7fe6dc158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6dad9d640 1 -- 172.21.15.94:0/4184888485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06b3a0 msgr2=0x7fe6dc158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6dad9d640 1 --2- 172.21.15.94:0/4184888485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06b3a0 0x7fe6dc158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6dad9d640 1 -- 172.21.15.94:0/4184888485 >> v1:172.21.15.145:6789/0 conn(0x7fe6dc06afc0 legacy=0x7fe6dc0c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.534+0000 7fe6dad9d640 1 -- 172.21.15.94:0/4184888485 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc0c5b20 con 0x7fe6dc159160 2024-03-06T07:27:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.535+0000 7fe6dad9d640 1 --2- 172.21.15.94:0/4184888485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc15fac0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009e50 tx=0x7fe6d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bfeac8c09f1e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.535+0000 7fe6d959a640 1 -- 172.21.15.94:0/4184888485 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d402ca70 con 0x7fe6dc159160 2024-03-06T07:27:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.535+0000 7fe6d959a640 1 -- 172.21.15.94:0/4184888485 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d402cc10 con 0x7fe6dc159160 2024-03-06T07:27:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.535+0000 7fe6d959a640 1 -- 172.21.15.94:0/4184888485 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d40343f0 con 0x7fe6dc159160 2024-03-06T07:27:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/4184888485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 msgr2=0x7fe6dc15fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/4184888485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc15fac0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009e50 tx=0x7fe6d402bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/4184888485 shutdown_connections 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/4184888485 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc15fac0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/4184888485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06b3a0 0x7fe6dc158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/4184888485 >> 172.21.15.94:0/4184888485 conn(0x7fe6dc0c4220 msgr2=0x7fe6dc0c4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/4184888485 shutdown_connections 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.536+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/4184888485 wait complete. 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.537+0000 7fe6e0fc9640 1 Processor -- start 2024-03-06T07:27:19.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.537+0000 7fe6e0fc9640 1 -- start start 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6e0fc9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06afc0 0x7fe6dc06e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6e0fc9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6dc06b3a0 0x7fe6dc157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6e0fc9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc14b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6e0fc9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc1595f0 con 0x7fe6dc159160 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6e0fc9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0c60e0 con 0x7fe6dc06b3a0 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6e0fc9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc155220 con 0x7fe6dc06afc0 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc14b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc14b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47774/0 (socket says 172.21.15.94:47774) 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 -- 172.21.15.94:0/3799564885 learned_addr learned my addr 172.21.15.94:0/3799564885 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6d9d9b640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6dc06b3a0 0x7fe6dc157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6da59c640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06afc0 0x7fe6dc06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 -- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06afc0 msgr2=0x7fe6dc06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06afc0 0x7fe6dc06e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 -- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6dc06b3a0 msgr2=0x7fe6dc157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6dc06b3a0 0x7fe6dc157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 -- 172.21.15.94:0/3799564885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc0715b0 con 0x7fe6dc159160 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6dad9d640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc14b910 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4036040 tx=0x7fe6d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6caffd640 1 -- 172.21.15.94:0/3799564885 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4044020 con 0x7fe6dc159160 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6caffd640 1 -- 172.21.15.94:0/3799564885 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d40406b0 con 0x7fe6dc159160 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6dc1ee810 con 0x7fe6dc159160 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.538+0000 7fe6caffd640 1 -- 172.21.15.94:0/3799564885 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4009510 con 0x7fe6dc159160 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.539+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6dc1eedd0 con 0x7fe6dc159160 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.539+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6dc0c5010 con 0x7fe6dc159160 2024-03-06T07:27:19.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.540+0000 7fe6caffd640 1 -- 172.21.15.94:0/3799564885 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6d4047020 con 0x7fe6dc159160 2024-03-06T07:27:19.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.540+0000 7fe6caffd640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.540+0000 7fe6caffd640 1 -- 172.21.15.94:0/3799564885 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe6d40acc70 con 0x7fe6dc159160 2024-03-06T07:27:19.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.540+0000 7fe6da59c640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.541+0000 7fe6da59c640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc001780 tx=0x7fe6cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.545+0000 7fe6caffd640 1 -- 172.21.15.94:0/3799564885 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6d404e030 con 0x7fe6dc159160 2024-03-06T07:27:19.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:19.756+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe6dc0c6730 con 0x7fe6dc159160 2024-03-06T07:27:20.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.422+0000 7fe6caffd640 1 -- 172.21.15.94:0/3799564885 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v196) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe6d407b430 con 0x7fe6dc159160 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe6bc06b4d0 msgr2=0x7fe6bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc001780 tx=0x7fe6cc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 msgr2=0x7fe6dc14b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc14b910 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4036040 tx=0x7fe6d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 shutdown_connections 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe6bc06b4d0 0x7fe6bc06d990 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe6dc159160 0x7fe6dc14b910 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6dc06b3a0 0x7fe6dc157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 --2- 172.21.15.94:0/3799564885 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6dc06afc0 0x7fe6dc06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 >> 172.21.15.94:0/3799564885 conn(0x7fe6dc0c4220 msgr2=0x7fe6dc153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 shutdown_connections 2024-03-06T07:27:20.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.424+0000 7fe6e0fc9640 1 -- 172.21.15.94:0/3799564885 wait complete. 2024-03-06T07:27:20.437 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:20.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:20.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.538+0000 7f7155a39640 1 Processor -- start 2024-03-06T07:27:20.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.538+0000 7f7155a39640 1 -- start start 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f7155a39640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f715006da40 0x7f7150155b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f7155a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71500c6780 0x7f7150158d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f7155a39640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7150156050 con 0x7f71501592d0 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f7155a39640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715006de20 con 0x7f715006da40 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f7155a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7150072630 con 0x7f71500c6780 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f715006da40 0x7f7150155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f715006da40 0x7f7150155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47776/0 (socket says 172.21.15.94:47776) 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714effd640 1 -- 172.21.15.94:0/611370090 learned_addr learned my addr 172.21.15.94:0/611370090 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714e7fc640 1 --2- 172.21.15.94:0/611370090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71500c6780 0x7f7150158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714effd640 1 -- 172.21.15.94:0/611370090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71500c6780 msgr2=0x7f7150158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714effd640 1 --2- 172.21.15.94:0/611370090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71500c6780 0x7f7150158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714effd640 1 -- 172.21.15.94:0/611370090 >> v1:172.21.15.145:6789/0 conn(0x7f71501592d0 legacy=0x7f71500c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.539+0000 7f714effd640 1 -- 172.21.15.94:0/611370090 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150072910 con 0x7f715006da40 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.540+0000 7f714effd640 1 --2- 172.21.15.94:0/611370090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f715006da40 0x7f7150155b10 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f7140009870 tx=0x7f714002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da366748ea98ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.540+0000 7f714dffb640 1 -- 172.21.15.94:0/611370090 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140035020 con 0x7f715006da40 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.540+0000 7f714dffb640 1 -- 172.21.15.94:0/611370090 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7140031640 con 0x7f715006da40 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.540+0000 7f714dffb640 1 -- 172.21.15.94:0/611370090 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714002d040 con 0x7f715006da40 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 -- 172.21.15.94:0/611370090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f715006da40 msgr2=0x7f7150155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 --2- 172.21.15.94:0/611370090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f715006da40 0x7f7150155b10 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f7140009870 tx=0x7f714002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 -- 172.21.15.94:0/611370090 shutdown_connections 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 --2- 172.21.15.94:0/611370090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71500c6780 0x7f7150158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 --2- 172.21.15.94:0/611370090 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f715006da40 0x7f7150155b10 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 -- 172.21.15.94:0/611370090 >> 172.21.15.94:0/611370090 conn(0x7f71500aa630 msgr2=0x7f71500aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 -- 172.21.15.94:0/611370090 shutdown_connections 2024-03-06T07:27:20.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.541+0000 7f7155a39640 1 -- 172.21.15.94:0/611370090 wait complete. 2024-03-06T07:27:20.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.542+0000 7f7155a39640 1 Processor -- start 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.542+0000 7f7155a39640 1 -- start start 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715006da40 0x7f71501de6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71500c6780 0x7f71501575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71501592d0 0x7f7150150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71500c3990 con 0x7f71500c6780 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150158890 con 0x7f715006da40 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71500c3b10 con 0x7f71501592d0 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71500c6780 0x7f71501575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71500c6780 0x7f71501575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47784/0 (socket says 172.21.15.94:47784) 2024-03-06T07:27:20.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 -- 172.21.15.94:0/2698059710 learned_addr learned my addr 172.21.15.94:0/2698059710 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714effd640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715006da40 0x7f71501de6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714f7fe640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71501592d0 0x7f7150150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 -- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71501592d0 msgr2=0x7f7150150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71501592d0 0x7f7150150d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 -- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715006da40 msgr2=0x7f71501de6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715006da40 0x7f71501de6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 -- 172.21.15.94:0/2698059710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71501dfb20 con 0x7f71500c6780 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714f7fe640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71501592d0 0x7f7150150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f714e7fc640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71500c6780 0x7f71501575e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f714400e8b0 tx=0x7f714400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7137fff640 1 -- 172.21.15.94:0/2698059710 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144017070 con 0x7f71500c6780 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7137fff640 1 -- 172.21.15.94:0/2698059710 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714400c940 con 0x7f71500c6780 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7150151e80 con 0x7f71500c6780 2024-03-06T07:27:20.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7137fff640 1 -- 172.21.15.94:0/2698059710 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714400cd30 con 0x7f71500c6780 2024-03-06T07:27:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.543+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7150157b20 con 0x7f71500c6780 2024-03-06T07:27:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.544+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7150151be0 con 0x7f71500c6780 2024-03-06T07:27:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.545+0000 7f7137fff640 1 -- 172.21.15.94:0/2698059710 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f714401a070 con 0x7f71500c6780 2024-03-06T07:27:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.545+0000 7f7137fff640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f713c06b4d0 0x7f713c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.545+0000 7f7137fff640 1 -- 172.21.15.94:0/2698059710 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f714408a5e0 con 0x7f71500c6780 2024-03-06T07:27:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.545+0000 7f714effd640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f713c06b4d0 0x7f713c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.546+0000 7f714effd640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f713c06b4d0 0x7f713c06d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f7140036000 tx=0x7f714003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.550+0000 7f7137fff640 1 -- 172.21.15.94:0/2698059710 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7144058d10 con 0x7f71500c6780 2024-03-06T07:27:20.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.758+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f715004f330 con 0x7f71500c6780 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.758+0000 7f7137fff640 1 -- 172.21.15.94:0/2698059710 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+4995 (secure 0 0 0) 0x7f714405cdb0 con 0x7f71500c6780 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":196,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:19.757407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:20.760 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 196 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f713c06b4d0 msgr2=0x7f713c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f713c06b4d0 0x7f713c06d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f7140036000 tx=0x7f714003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:20.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71500c6780 msgr2=0x7f71501575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71500c6780 0x7f71501575e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f714400e8b0 tx=0x7f714400ed70 comp rx=0 tx=0).stop 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 shutdown_connections 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f713c06b4d0 0x7f713c06d990 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71501592d0 0x7f7150150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f71500c6780 0x7f71501575e0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 --2- 172.21.15.94:0/2698059710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715006da40 0x7f71501de6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 >> 172.21.15.94:0/2698059710 conn(0x7f71500aa630 msgr2=0x7f71500aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 shutdown_connections 2024-03-06T07:27:20.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.760+0000 7f7155a39640 1 -- 172.21.15.94:0/2698059710 wait complete. 2024-03-06T07:27:20.773 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:20.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.871+0000 7f1fab9e4640 1 Processor -- start 2024-03-06T07:27:20.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.871+0000 7f1fab9e4640 1 -- start start 2024-03-06T07:27:20.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fab9e4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa41592d0 0x7f1fa41552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fab9e4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fa40ac260 0x7f1fa4155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fab9e4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa4152d40 con 0x7f1fa4152960 2024-03-06T07:27:20.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fab9e4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa4066e30 con 0x7f1fa40ac260 2024-03-06T07:27:20.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fab9e4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa4155d50 con 0x7f1fa41592d0 2024-03-06T07:27:20.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9f81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fa40ac260 0x7f1fa4155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.875 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9f81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fa40ac260 0x7f1fa4155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47788/0 (socket says 172.21.15.94:47788) 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9f81640 1 -- 172.21.15.94:0/2851885499 learned_addr learned my addr 172.21.15.94:0/2851885499 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9780640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1fa4152960 0x7f1fa406dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42822/0 (socket says 172.21.15.94:42822) 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa8f7f640 1 --2- 172.21.15.94:0/2851885499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa41592d0 0x7f1fa41552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9f81640 1 -- 172.21.15.94:0/2851885499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa41592d0 msgr2=0x7f1fa41552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9f81640 1 --2- 172.21.15.94:0/2851885499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa41592d0 0x7f1fa41552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9f81640 1 -- 172.21.15.94:0/2851885499 >> v1:172.21.15.94:6789/0 conn(0x7f1fa4152960 legacy=0x7f1fa406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1fa9f81640 1 -- 172.21.15.94:0/2851885499 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa40c1b50 con 0x7f1fa40ac260 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.872+0000 7f1f9bfff640 1 -- 172.21.15.94:0/2851885499 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424822647 0 0) 0x7f1fa4152d40 con 0x7f1fa4152960 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.873+0000 7f1fa9f81640 1 --2- 172.21.15.94:0/2851885499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa40ac260 0x7f1fa4155810 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c009870 tx=0x7f1f9c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0cc0c158562d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.873+0000 7f1f9bfff640 1 -- 172.21.15.94:0/2851885499 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c034070 con 0x7f1fa40ac260 2024-03-06T07:27:20.876 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.873+0000 7f1f9bfff640 1 -- 172.21.15.94:0/2851885499 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c030840 con 0x7f1fa40ac260 2024-03-06T07:27:20.877 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.873+0000 7f1f9bfff640 1 -- 172.21.15.94:0/2851885499 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c030c10 con 0x7f1fa40ac260 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 -- 172.21.15.94:0/2851885499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa40ac260 msgr2=0x7f1fa4155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/2851885499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa40ac260 0x7f1fa4155810 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c009870 tx=0x7f1f9c02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 -- 172.21.15.94:0/2851885499 shutdown_connections 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/2851885499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa40ac260 0x7f1fa4155810 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/2851885499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa41592d0 0x7f1fa41552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 -- 172.21.15.94:0/2851885499 >> 172.21.15.94:0/2851885499 conn(0x7f1fa40c70b0 msgr2=0x7f1fa40c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 -- 172.21.15.94:0/2851885499 shutdown_connections 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.874+0000 7f1fab9e4640 1 -- 172.21.15.94:0/2851885499 wait complete. 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.875+0000 7f1fab9e4640 1 Processor -- start 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.875+0000 7f1fab9e4640 1 -- start start 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fab9e4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa40ac260 0x7f1fa4158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fab9e4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa4152960 0x7f1fa4072040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fab9e4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa41eb500 0x7f1fa41eb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fab9e4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa4159760 con 0x7f1fa4152960 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fab9e4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40672c0 con 0x7f1fa41eb500 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fab9e4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa406e3a0 con 0x7f1fa40ac260 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa4152960 0x7f1fa4072040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa4152960 0x7f1fa4072040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47790/0 (socket says 172.21.15.94:47790) 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 -- 172.21.15.94:0/575470313 learned_addr learned my addr 172.21.15.94:0/575470313 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa9f81640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa41eb500 0x7f1fa41eb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa9780640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa40ac260 0x7f1fa4158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 -- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa40ac260 msgr2=0x7f1fa4158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa40ac260 0x7f1fa4158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 -- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa41eb500 msgr2=0x7f1fa41eb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa41eb500 0x7f1fa41eb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 -- 172.21.15.94:0/575470313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa41df680 con 0x7f1fa4152960 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa9f81640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa41eb500 0x7f1fa41eb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fa8f7f640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa4152960 0x7f1fa4072040 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e8d0 tx=0x7f1f9400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1f99ffb640 1 -- 172.21.15.94:0/575470313 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94017070 con 0x7f1fa4152960 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1f99ffb640 1 -- 172.21.15.94:0/575470313 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9400c8e0 con 0x7f1fa4152960 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1f99ffb640 1 -- 172.21.15.94:0/575470313 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400ccd0 con 0x7f1fa4152960 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.876+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa41eb020 con 0x7f1fa4152960 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.877+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa41ef030 con 0x7f1fa4152960 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.877+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa404f330 con 0x7f1fa4152960 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.878+0000 7f1f99ffb640 1 -- 172.21.15.94:0/575470313 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f9401a070 con 0x7f1fa4152960 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.878+0000 7f1f99ffb640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1f8806b500 0x7f1f8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.878+0000 7f1f99ffb640 1 -- 172.21.15.94:0/575470313 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1f9408a060 con 0x7f1fa4152960 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.878+0000 7f1fa9780640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1f8806b500 0x7f1f8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:20.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.879+0000 7f1fa9780640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1f8806b500 0x7f1f8806d9c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00012e0 tx=0x7f1fa0001700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:20.884 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:20.883+0000 7f1f99ffb640 1 -- 172.21.15.94:0/575470313 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fa41df680 con 0x7f1fa4152960 2024-03-06T07:27:21.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.053+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fa406ebc0 con 0x7f1fa4152960 2024-03-06T07:27:21.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.054+0000 7f1f99ffb640 1 -- 172.21.15.94:0/575470313 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16888 (secure 0 0 0) 0x7f1f94058850 con 0x7f1fa4152960 2024-03-06T07:27:21.056 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:21.056 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":209,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:27:19.397641+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:15.999555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:16.719665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:21.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1f8806b500 msgr2=0x7f1f8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1f8806b500 0x7f1f8806d9c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00012e0 tx=0x7f1fa0001700 comp rx=0 tx=0).stop 2024-03-06T07:27:21.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa4152960 msgr2=0x7f1fa4072040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa4152960 0x7f1fa4072040 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e8d0 tx=0x7f1f9400ed90 comp rx=0 tx=0).stop 2024-03-06T07:27:21.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 shutdown_connections 2024-03-06T07:27:21.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1f8806b500 0x7f1f8806d9c0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fa41eb500 0x7f1fa41eb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fa4152960 0x7f1fa4072040 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 --2- 172.21.15.94:0/575470313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fa40ac260 0x7f1fa4158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 >> 172.21.15.94:0/575470313 conn(0x7f1fa40c70b0 msgr2=0x7f1fa40c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 shutdown_connections 2024-03-06T07:27:21.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.056+0000 7f1fab9e4640 1 -- 172.21.15.94:0/575470313 wait complete. 2024-03-06T07:27:21.069 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:21.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:21.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.167+0000 7f4f6342a640 1 Processor -- start 2024-03-06T07:27:21.169 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.167+0000 7f4f6342a640 1 -- start start 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f6342a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4f5c152520 0x7f4f5c06ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f6342a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c0720a0 0x7f4f5c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f6342a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c0c1ae0 con 0x7f4f5c155c80 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f6342a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0c1dc0 con 0x7f4f5c152520 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f6342a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c06e450 con 0x7f4f5c0720a0 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f609c5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4f5c155c80 0x7f4f5c0c12b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42824/0 (socket says 172.21.15.94:42824) 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f609c5640 1 -- 172.21.15.94:0/3098481323 learned_addr learned my addr 172.21.15.94:0/3098481323 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:21.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f611c6640 1 --2- 172.21.15.94:0/3098481323 >> v2:172.21.15.145:3300/0 conn(0x7f4f5c152520 0x7f4f5c06ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f619c7640 1 --2- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c0720a0 0x7f4f5c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f53fff640 1 -- 172.21.15.94:0/3098481323 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362745753 0 0) 0x7f4f5c0c1ae0 con 0x7f4f5c155c80 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f53fff640 1 -- 172.21.15.94:0/3098481323 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f48003520 con 0x7f4f5c155c80 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f611c6640 1 -- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c0720a0 msgr2=0x7f4f5c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f611c6640 1 --2- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c0720a0 0x7f4f5c06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.168+0000 7f4f611c6640 1 -- 172.21.15.94:0/3098481323 >> v1:172.21.15.94:6789/0 conn(0x7f4f5c155c80 legacy=0x7f4f5c0c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.169+0000 7f4f611c6640 1 -- 172.21.15.94:0/3098481323 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c06e730 con 0x7f4f5c152520 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.169+0000 7f4f611c6640 1 --2- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c152520 0x7f4f5c06ef30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4f580097a0 tx=0x7f4f5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a95434d5096dc4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.169+0000 7f4f53fff640 1 -- 172.21.15.94:0/3098481323 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58035020 con 0x7f4f5c152520 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.169+0000 7f4f53fff640 1 -- 172.21.15.94:0/3098481323 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f58031830 con 0x7f4f5c152520 2024-03-06T07:27:21.172 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.169+0000 7f4f53fff640 1 -- 172.21.15.94:0/3098481323 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58031be0 con 0x7f4f5c152520 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 -- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c152520 msgr2=0x7f4f5c06ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 --2- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c152520 0x7f4f5c06ef30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4f580097a0 tx=0x7f4f5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 -- 172.21.15.94:0/3098481323 shutdown_connections 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 --2- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c0720a0 0x7f4f5c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 --2- 172.21.15.94:0/3098481323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c152520 0x7f4f5c06ef30 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 -- 172.21.15.94:0/3098481323 >> 172.21.15.94:0/3098481323 conn(0x7f4f5c0c76b0 msgr2=0x7f4f5c0c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 -- 172.21.15.94:0/3098481323 shutdown_connections 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.170+0000 7f4f6342a640 1 -- 172.21.15.94:0/3098481323 wait complete. 2024-03-06T07:27:21.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.171+0000 7f4f6342a640 1 Processor -- start 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.171+0000 7f4f6342a640 1 -- start start 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f6342a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5c0720a0 0x7f4f5c157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f6342a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c152520 0x7f4f5c0a72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f6342a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c155c80 0x7f4f5c156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f6342a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c152a80 con 0x7f4f5c0720a0 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f6342a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c152900 con 0x7f4f5c155c80 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f6342a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c06b500 con 0x7f4f5c152520 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5c0720a0 0x7f4f5c157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5c0720a0 0x7f4f5c157260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47796/0 (socket says 172.21.15.94:47796) 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 -- 172.21.15.94:0/2916898120 learned_addr learned my addr 172.21.15.94:0/2916898120 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:21.176 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f609c5640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c152520 0x7f4f5c0a72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f619c7640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c155c80 0x7f4f5c156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 -- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c152520 msgr2=0x7f4f5c0a72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c152520 0x7f4f5c0a72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 -- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c155c80 msgr2=0x7f4f5c156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c155c80 0x7f4f5c156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 -- 172.21.15.94:0/2916898120 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c152fe0 con 0x7f4f5c0720a0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.172+0000 7f4f611c6640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5c0720a0 0x7f4f5c157260 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f4f580098c0 tx=0x7f4f580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.173+0000 7f4f51ffb640 1 -- 172.21.15.94:0/2916898120 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58035040 con 0x7f4f5c0720a0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.173+0000 7f4f51ffb640 1 -- 172.21.15.94:0/2916898120 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5803b900 con 0x7f4f5c0720a0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.173+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c1532c0 con 0x7f4f5c0720a0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.173+0000 7f4f51ffb640 1 -- 172.21.15.94:0/2916898120 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5803bcf0 con 0x7f4f5c0720a0 2024-03-06T07:27:21.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.173+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c153880 con 0x7f4f5c0720a0 2024-03-06T07:27:21.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.174+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c04f330 con 0x7f4f5c0720a0 2024-03-06T07:27:21.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.174+0000 7f4f51ffb640 1 -- 172.21.15.94:0/2916898120 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f58038020 con 0x7f4f5c0720a0 2024-03-06T07:27:21.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.175+0000 7f4f51ffb640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4006b4d0 0x7f4f4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.175+0000 7f4f51ffb640 1 -- 172.21.15.94:0/2916898120 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f580ac1e0 con 0x7f4f5c0720a0 2024-03-06T07:27:21.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.175+0000 7f4f609c5640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4006b4d0 0x7f4f4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.179 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.175+0000 7f4f609c5640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4006b4d0 0x7f4f4006d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c1e83b0 tx=0x7f4f4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.180 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.179+0000 7f4f51ffb640 1 -- 172.21.15.94:0/2916898120 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f5807a910 con 0x7f4f5c0720a0 2024-03-06T07:27:21.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.349+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f5c1520c0 con 0x7f4f4006b4d0 2024-03-06T07:27:21.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.355+0000 7f4f51ffb640 1 -- 172.21.15.94:0/2916898120 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247072 (secure 0 0 0) 0x7f4f5c1520c0 con 0x7f4f4006b4d0 2024-03-06T07:27:21.360 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:21.361 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:27:21.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4006b4d0 msgr2=0x7f4f4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4006b4d0 0x7f4f4006d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c1e83b0 tx=0x7f4f4c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:21.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5c0720a0 msgr2=0x7f4f5c157260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5c0720a0 0x7f4f5c157260 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f4f580098c0 tx=0x7f4f580046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 shutdown_connections 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4f4006b4d0 0x7f4f4006d990 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f5c155c80 0x7f4f5c156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f5c152520 0x7f4f5c0a72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 --2- 172.21.15.94:0/2916898120 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f5c0720a0 0x7f4f5c157260 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 >> 172.21.15.94:0/2916898120 conn(0x7f4f5c0c76b0 msgr2=0x7f4f5c0c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 shutdown_connections 2024-03-06T07:27:21.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.358+0000 7f4f6342a640 1 -- 172.21.15.94:0/2916898120 wait complete. 2024-03-06T07:27:21.371 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-03-06T07:27:20.757508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278380,"kb_used_data":20576,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754696340,"statfs":{"total":773094113280,"available":772809052160,"internally_reserved":0,"allocated":21069824,"data_stored":11304944,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.366341"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405718+0000","last_change":"2024-03-06T07:27:17.405718+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405718+0000","last_undegraded":"2024-03-06T07:27:17.405718+0000","last_fullsized":"2024-03-06T07:27:17.405718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405159+0000","last_change":"2024-03-06T07:27:17.405159+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405159+0000","last_undegraded":"2024-03-06T07:27:17.405159+0000","last_fullsized":"2024-03-06T07:27:17.405159+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410596+0000","last_change":"2024-03-06T07:27:17.410596+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410596+0000","last_undegraded":"2024-03-06T07:27:17.410596+0000","last_fullsized":"2024-03-06T07:27:17.410596+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403936+0000","last_change":"2024-03-06T07:27:17.403936+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403936+0000","last_undegraded":"2024-03-06T07:27:17.403936+0000","last_fullsized":"2024-03-06T07:27:17.403936+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406232+0000","last_change":"2024-03-06T07:27:17.406232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406232+0000","last_undegraded":"2024-03-06T07:27:17.406232+0000","last_fullsized":"2024-03-06T07:27:17.406232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404786+0000","last_change":"2024-03-06T07:27:17.404786+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404786+0000","last_undegraded":"2024-03-06T07:27:17.404786+0000","last_fullsized":"2024-03-06T07:27:17.404786+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409357+0000","last_change":"2024-03-06T07:27:17.409357+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409357+0000","last_undegraded":"2024-03-06T07:27:17.409357+0000","last_fullsized":"2024-03-06T07:27:17.409357+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405499+0000","last_change":"2024-03-06T07:27:17.405499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405499+0000","last_undegraded":"2024-03-06T07:27:17.405499+0000","last_fullsized":"2024-03-06T07:27:17.405499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405024+0000","last_change":"2024-03-06T07:27:17.405024+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405024+0000","last_undegraded":"2024-03-06T07:27:17.405024+0000","last_fullsized":"2024-03-06T07:27:17.405024+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401578+0000","last_change":"2024-03-06T07:27:17.401578+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401578+0000","last_undegraded":"2024-03-06T07:27:17.401578+0000","last_fullsized":"2024-03-06T07:27:17.401578+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402332+0000","last_change":"2024-03-06T07:27:17.402332+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402332+0000","last_undegraded":"2024-03-06T07:27:17.402332+0000","last_fullsized":"2024-03-06T07:27:17.402332+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405310+0000","last_change":"2024-03-06T07:27:17.405310+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405310+0000","last_undegraded":"2024-03-06T07:27:17.405310+0000","last_fullsized":"2024-03-06T07:27:17.405310+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406144+0000","last_change":"2024-03-06T07:27:17.406144+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406144+0000","last_undegraded":"2024-03-06T07:27:17.406144+0000","last_fullsized":"2024-03-06T07:27:17.406144+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405327+0000","last_change":"2024-03-06T07:27:17.405327+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405327+0000","last_undegraded":"2024-03-06T07:27:17.405327+0000","last_fullsized":"2024-03-06T07:27:17.405327+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402499+0000","last_change":"2024-03-06T07:27:17.402499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402499+0000","last_undegraded":"2024-03-06T07:27:17.402499+0000","last_fullsized":"2024-03-06T07:27:17.402499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402943+0000","last_change":"2024-03-06T07:27:17.402943+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402943+0000","last_undegraded":"2024-03-06T07:27:17.402943+0000","last_fullsized":"2024-03-06T07:27:17.402943+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400619+0000","last_change":"2024-03-06T07:27:17.400619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400619+0000","last_undegraded":"2024-03-06T07:27:17.400619+0000","last_fullsized":"2024-03-06T07:27:17.400619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404639+0000","last_change":"2024-03-06T07:27:17.404639+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404639+0000","last_undegraded":"2024-03-06T07:27:17.404639+0000","last_fullsized":"2024-03-06T07:27:17.404639+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400272+0000","last_change":"2024-03-06T07:27:17.400272+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400272+0000","last_undegraded":"2024-03-06T07:27:17.400272+0000","last_fullsized":"2024-03-06T07:27:17.400272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404933+0000","last_change":"2024-03-06T07:27:17.404933+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404933+0000","last_undegraded":"2024-03-06T07:27:17.404933+0000","last_fullsized":"2024-03-06T07:27:17.404933+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405809+0000","last_change":"2024-03-06T07:27:17.405809+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405809+0000","last_undegraded":"2024-03-06T07:27:17.405809+0000","last_fullsized":"2024-03-06T07:27:17.405809+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401251+0000","last_change":"2024-03-06T07:27:17.401251+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401251+0000","last_undegraded":"2024-03-06T07:27:17.401251+0000","last_fullsized":"2024-03-06T07:27:17.401251+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402086+0000","last_change":"2024-03-06T07:27:17.402086+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402086+0000","last_undegraded":"2024-03-06T07:27:17.402086+0000","last_fullsized":"2024-03-06T07:27:17.402086+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400623+0000","last_change":"2024-03-06T07:27:17.400623+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400623+0000","last_undegraded":"2024-03-06T07:27:17.400623+0000","last_fullsized":"2024-03-06T07:27:17.400623+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410762+0000","last_change":"2024-03-06T07:27:17.410762+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410762+0000","last_undegraded":"2024-03-06T07:27:17.410762+0000","last_fullsized":"2024-03-06T07:27:17.410762+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401619+0000","last_change":"2024-03-06T07:27:17.401619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401619+0000","last_undegraded":"2024-03-06T07:27:17.401619+0000","last_fullsized":"2024-03-06T07:27:17.401619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401907+0000","last_change":"2024-03-06T07:27:17.401907+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401907+0000","last_undegraded":"2024-03-06T07:27:17.401907+0000","last_fullsized":"2024-03-06T07:27:17.401907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402685+0000","last_change":"2024-03-06T07:27:17.402685+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402685+0000","last_undegraded":"2024-03-06T07:27:17.402685+0000","last_fullsized":"2024-03-06T07:27:17.402685+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404113+0000","last_change":"2024-03-06T07:27:17.404113+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404113+0000","last_undegraded":"2024-03-06T07:27:17.404113+0000","last_fullsized":"2024-03-06T07:27:17.404113+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401612+0000","last_change":"2024-03-06T07:27:17.401612+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401612+0000","last_undegraded":"2024-03-06T07:27:17.401612+0000","last_fullsized":"2024-03-06T07:27:17.401612+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402531+0000","last_change":"2024-03-06T07:27:17.402531+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402531+0000","last_undegraded":"2024-03-06T07:27:17.402531+0000","last_fullsized":"2024-03-06T07:27:17.402531+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401801+0000","last_change":"2024-03-06T07:27:17.401801+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401801+0000","last_undegraded":"2024-03-06T07:27:17.401801+0000","last_fullsized":"2024-03-06T07:27:17.401801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413284+0000","last_change":"2024-03-06T07:27:17.410327+0000","last_active":"2024-03-06T07:27:17.413284+0000","last_peered":"2024-03-06T07:27:17.413284+0000","last_clean":"2024-03-06T07:27:17.413284+0000","last_became_active":"2024-03-06T07:27:17.409826+0000","last_became_peered":"2024-03-06T07:27:17.409826+0000","last_unstale":"2024-03-06T07:27:17.413284+0000","last_undegraded":"2024-03-06T07:27:17.413284+0000","last_fullsized":"2024-03-06T07:27:17.413284+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:31.182027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403864+0000","last_change":"2024-03-06T07:27:17.403864+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403864+0000","last_undegraded":"2024-03-06T07:27:17.403864+0000","last_fullsized":"2024-03-06T07:27:17.403864+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414725+0000","last_change":"2024-03-06T07:27:17.413513+0000","last_active":"2024-03-06T07:27:17.414725+0000","last_peered":"2024-03-06T07:27:17.414725+0000","last_clean":"2024-03-06T07:27:17.414725+0000","last_became_active":"2024-03-06T07:27:17.413247+0000","last_became_peered":"2024-03-06T07:27:17.413247+0000","last_unstale":"2024-03-06T07:27:17.414725+0000","last_undegraded":"2024-03-06T07:27:17.414725+0000","last_fullsized":"2024-03-06T07:27:17.414725+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404232+0000","last_change":"2024-03-06T07:27:17.404232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404232+0000","last_undegraded":"2024-03-06T07:27:17.404232+0000","last_fullsized":"2024-03-06T07:27:17.404232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418162+0000","last_change":"2024-03-06T07:27:17.415933+0000","last_active":"2024-03-06T07:27:17.418162+0000","last_peered":"2024-03-06T07:27:17.418162+0000","last_clean":"2024-03-06T07:27:17.418162+0000","last_became_active":"2024-03-06T07:27:17.415652+0000","last_became_peered":"2024-03-06T07:27:17.415652+0000","last_unstale":"2024-03-06T07:27:17.418162+0000","last_undegraded":"2024-03-06T07:27:17.418162+0000","last_fullsized":"2024-03-06T07:27:17.418162+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:05.886687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400666+0000","last_change":"2024-03-06T07:27:17.400666+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400666+0000","last_undegraded":"2024-03-06T07:27:17.400666+0000","last_fullsized":"2024-03-06T07:27:17.400666+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413520+0000","last_change":"2024-03-06T07:27:17.412515+0000","last_active":"2024-03-06T07:27:17.413520+0000","last_peered":"2024-03-06T07:27:17.413520+0000","last_clean":"2024-03-06T07:27:17.413520+0000","last_became_active":"2024-03-06T07:27:17.412246+0000","last_became_peered":"2024-03-06T07:27:17.412246+0000","last_unstale":"2024-03-06T07:27:17.413520+0000","last_undegraded":"2024-03-06T07:27:17.413520+0000","last_fullsized":"2024-03-06T07:27:17.413520+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400981+0000","last_change":"2024-03-06T07:27:17.400981+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400981+0000","last_undegraded":"2024-03-06T07:27:17.400981+0000","last_fullsized":"2024-03-06T07:27:17.400981+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418993+0000","last_change":"2024-03-06T07:27:17.418064+0000","last_active":"2024-03-06T07:27:17.418993+0000","last_peered":"2024-03-06T07:27:17.418993+0000","last_clean":"2024-03-06T07:27:17.418993+0000","last_became_active":"2024-03-06T07:27:17.417764+0000","last_became_peered":"2024-03-06T07:27:17.417764+0000","last_unstale":"2024-03-06T07:27:17.418993+0000","last_undegraded":"2024-03-06T07:27:17.418993+0000","last_fullsized":"2024-03-06T07:27:17.418993+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:30.997230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400603+0000","last_change":"2024-03-06T07:27:17.400603+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400603+0000","last_undegraded":"2024-03-06T07:27:17.400603+0000","last_fullsized":"2024-03-06T07:27:17.400603+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416413+0000","last_change":"2024-03-06T07:27:17.415618+0000","last_active":"2024-03-06T07:27:17.416413+0000","last_peered":"2024-03-06T07:27:17.416413+0000","last_clean":"2024-03-06T07:27:17.416413+0000","last_became_active":"2024-03-06T07:27:17.415113+0000","last_became_peered":"2024-03-06T07:27:17.415113+0000","last_unstale":"2024-03-06T07:27:17.416413+0000","last_undegraded":"2024-03-06T07:27:17.416413+0000","last_fullsized":"2024-03-06T07:27:17.416413+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409868+0000","last_change":"2024-03-06T07:27:17.409868+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409868+0000","last_undegraded":"2024-03-06T07:27:17.409868+0000","last_fullsized":"2024-03-06T07:27:17.409868+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412010+0000","last_change":"2024-03-06T07:27:17.410657+0000","last_active":"2024-03-06T07:27:17.412010+0000","last_peered":"2024-03-06T07:27:17.412010+0000","last_clean":"2024-03-06T07:27:17.412010+0000","last_became_active":"2024-03-06T07:27:17.410330+0000","last_became_peered":"2024-03-06T07:27:17.410330+0000","last_unstale":"2024-03-06T07:27:17.412010+0000","last_undegraded":"2024-03-06T07:27:17.412010+0000","last_fullsized":"2024-03-06T07:27:17.412010+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:32.740313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401908+0000","last_change":"2024-03-06T07:27:17.401908+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401908+0000","last_undegraded":"2024-03-06T07:27:17.401908+0000","last_fullsized":"2024-03-06T07:27:17.401908+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412932+0000","last_change":"2024-03-06T07:27:17.412182+0000","last_active":"2024-03-06T07:27:17.412932+0000","last_peered":"2024-03-06T07:27:17.412932+0000","last_clean":"2024-03-06T07:27:17.412932+0000","last_became_active":"2024-03-06T07:27:17.411934+0000","last_became_peered":"2024-03-06T07:27:17.411934+0000","last_unstale":"2024-03-06T07:27:17.412932+0000","last_undegraded":"2024-03-06T07:27:17.412932+0000","last_fullsized":"2024-03-06T07:27:17.412932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:18.741598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402610+0000","last_change":"2024-03-06T07:27:17.402610+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402610+0000","last_undegraded":"2024-03-06T07:27:17.402610+0000","last_fullsized":"2024-03-06T07:27:17.402610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415272+0000","last_change":"2024-03-06T07:27:17.412868+0000","last_active":"2024-03-06T07:27:17.415272+0000","last_peered":"2024-03-06T07:27:17.415272+0000","last_clean":"2024-03-06T07:27:17.415272+0000","last_became_active":"2024-03-06T07:27:17.412587+0000","last_became_peered":"2024-03-06T07:27:17.412587+0000","last_unstale":"2024-03-06T07:27:17.415272+0000","last_undegraded":"2024-03-06T07:27:17.415272+0000","last_fullsized":"2024-03-06T07:27:17.415272+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:36.738070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403419+0000","last_change":"2024-03-06T07:27:17.403419+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403419+0000","last_undegraded":"2024-03-06T07:27:17.403419+0000","last_fullsized":"2024-03-06T07:27:17.403419+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411419+0000","last_change":"2024-03-06T07:27:17.410408+0000","last_active":"2024-03-06T07:27:17.411419+0000","last_peered":"2024-03-06T07:27:17.411419+0000","last_clean":"2024-03-06T07:27:17.411419+0000","last_became_active":"2024-03-06T07:27:17.410121+0000","last_became_peered":"2024-03-06T07:27:17.410121+0000","last_unstale":"2024-03-06T07:27:17.411419+0000","last_undegraded":"2024-03-06T07:27:17.411419+0000","last_fullsized":"2024-03-06T07:27:17.411419+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:21.045449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405727+0000","last_change":"2024-03-06T07:27:17.405727+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405727+0000","last_undegraded":"2024-03-06T07:27:17.405727+0000","last_fullsized":"2024-03-06T07:27:17.405727+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416587+0000","last_change":"2024-03-06T07:27:17.414830+0000","last_active":"2024-03-06T07:27:17.416587+0000","last_peered":"2024-03-06T07:27:17.416587+0000","last_clean":"2024-03-06T07:27:17.416587+0000","last_became_active":"2024-03-06T07:27:17.414541+0000","last_became_peered":"2024-03-06T07:27:17.414541+0000","last_unstale":"2024-03-06T07:27:17.416587+0000","last_undegraded":"2024-03-06T07:27:17.416587+0000","last_fullsized":"2024-03-06T07:27:17.416587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:46.438918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405165+0000","last_change":"2024-03-06T07:27:17.405165+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405165+0000","last_undegraded":"2024-03-06T07:27:17.405165+0000","last_fullsized":"2024-03-06T07:27:17.405165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414323+0000","last_change":"2024-03-06T07:27:17.413318+0000","last_active":"2024-03-06T07:27:17.414323+0000","last_peered":"2024-03-06T07:27:17.414323+0000","last_clean":"2024-03-06T07:27:17.414323+0000","last_became_active":"2024-03-06T07:27:17.413042+0000","last_became_peered":"2024-03-06T07:27:17.413042+0000","last_unstale":"2024-03-06T07:27:17.414323+0000","last_undegraded":"2024-03-06T07:27:17.414323+0000","last_fullsized":"2024-03-06T07:27:17.414323+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:57.629596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.407372+0000","last_change":"2024-03-06T07:27:17.407372+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.407372+0000","last_undegraded":"2024-03-06T07:27:17.407372+0000","last_fullsized":"2024-03-06T07:27:17.407372+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412696+0000","last_change":"2024-03-06T07:27:17.412118+0000","last_active":"2024-03-06T07:27:17.412696+0000","last_peered":"2024-03-06T07:27:17.412696+0000","last_clean":"2024-03-06T07:27:17.412696+0000","last_became_active":"2024-03-06T07:27:17.411828+0000","last_became_peered":"2024-03-06T07:27:17.411828+0000","last_unstale":"2024-03-06T07:27:17.412696+0000","last_undegraded":"2024-03-06T07:27:17.412696+0000","last_fullsized":"2024-03-06T07:27:17.412696+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:07.702608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400917+0000","last_change":"2024-03-06T07:27:17.400917+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400917+0000","last_undegraded":"2024-03-06T07:27:17.400917+0000","last_fullsized":"2024-03-06T07:27:17.400917+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419663+0000","last_change":"2024-03-06T07:27:17.416024+0000","last_active":"2024-03-06T07:27:17.419663+0000","last_peered":"2024-03-06T07:27:17.419663+0000","last_clean":"2024-03-06T07:27:17.419663+0000","last_became_active":"2024-03-06T07:27:17.415738+0000","last_became_peered":"2024-03-06T07:27:17.415738+0000","last_unstale":"2024-03-06T07:27:17.419663+0000","last_undegraded":"2024-03-06T07:27:17.419663+0000","last_fullsized":"2024-03-06T07:27:17.419663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:26.216018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.408899+0000","last_change":"2024-03-06T07:27:17.408899+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.408899+0000","last_undegraded":"2024-03-06T07:27:17.408899+0000","last_fullsized":"2024-03-06T07:27:17.408899+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419824+0000","last_change":"2024-03-06T07:27:17.416751+0000","last_active":"2024-03-06T07:27:17.419824+0000","last_peered":"2024-03-06T07:27:17.419824+0000","last_clean":"2024-03-06T07:27:17.419824+0000","last_became_active":"2024-03-06T07:27:17.416475+0000","last_became_peered":"2024-03-06T07:27:17.416475+0000","last_unstale":"2024-03-06T07:27:17.419824+0000","last_undegraded":"2024-03-06T07:27:17.419824+0000","last_fullsized":"2024-03-06T07:27:17.419824+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:34.635712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405423+0000","last_change":"2024-03-06T07:27:17.405423+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405423+0000","last_undegraded":"2024-03-06T07:27:17.405423+0000","last_fullsized":"2024-03-06T07:27:17.405423+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415326+0000","last_change":"2024-03-06T07:27:17.408458+0000","last_active":"2024-03-06T07:27:17.415326+0000","last_peered":"2024-03-06T07:27:17.415326+0000","last_clean":"2024-03-06T07:27:17.415326+0000","last_became_active":"2024-03-06T07:27:17.408204+0000","last_became_peered":"2024-03-06T07:27:17.408204+0000","last_unstale":"2024-03-06T07:27:17.415326+0000","last_undegraded":"2024-03-06T07:27:17.415326+0000","last_fullsized":"2024-03-06T07:27:17.415326+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:31.044701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400820+0000","last_change":"2024-03-06T07:27:17.400820+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400820+0000","last_undegraded":"2024-03-06T07:27:17.400820+0000","last_fullsized":"2024-03-06T07:27:17.400820+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.423682+0000","last_change":"2024-03-06T07:27:17.418167+0000","last_active":"2024-03-06T07:27:17.423682+0000","last_peered":"2024-03-06T07:27:17.423682+0000","last_clean":"2024-03-06T07:27:17.423682+0000","last_became_active":"2024-03-06T07:27:17.417877+0000","last_became_peered":"2024-03-06T07:27:17.417877+0000","last_unstale":"2024-03-06T07:27:17.423682+0000","last_undegraded":"2024-03-06T07:27:17.423682+0000","last_fullsized":"2024-03-06T07:27:17.423682+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401456+0000","last_change":"2024-03-06T07:27:17.401456+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401456+0000","last_undegraded":"2024-03-06T07:27:17.401456+0000","last_fullsized":"2024-03-06T07:27:17.401456+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413865+0000","last_change":"2024-03-06T07:27:17.409593+0000","last_active":"2024-03-06T07:27:17.413865+0000","last_peered":"2024-03-06T07:27:17.413865+0000","last_clean":"2024-03-06T07:27:17.413865+0000","last_became_active":"2024-03-06T07:27:17.409303+0000","last_became_peered":"2024-03-06T07:27:17.409303+0000","last_unstale":"2024-03-06T07:27:17.413865+0000","last_undegraded":"2024-03-06T07:27:17.413865+0000","last_fullsized":"2024-03-06T07:27:17.413865+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:58.310257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403395+0000","last_change":"2024-03-06T07:27:17.403395+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403395+0000","last_undegraded":"2024-03-06T07:27:17.403395+0000","last_fullsized":"2024-03-06T07:27:17.403395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416748+0000","last_change":"2024-03-06T07:27:17.416748+0000","last_active":"2024-03-06T07:27:17.416748+0000","last_peered":"2024-03-06T07:27:17.416748+0000","last_clean":"2024-03-06T07:27:17.416748+0000","last_became_active":"2024-03-06T07:27:17.416473+0000","last_became_peered":"2024-03-06T07:27:17.416473+0000","last_unstale":"2024-03-06T07:27:17.416748+0000","last_undegraded":"2024-03-06T07:27:17.416748+0000","last_fullsized":"2024-03-06T07:27:17.416748+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404729+0000","last_change":"2024-03-06T07:27:17.404729+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404729+0000","last_undegraded":"2024-03-06T07:27:17.404729+0000","last_fullsized":"2024-03-06T07:27:17.404729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411708+0000","last_change":"2024-03-06T07:27:17.409661+0000","last_active":"2024-03-06T07:27:17.411708+0000","last_peered":"2024-03-06T07:27:17.411708+0000","last_clean":"2024-03-06T07:27:17.411708+0000","last_became_active":"2024-03-06T07:27:17.409383+0000","last_became_peered":"2024-03-06T07:27:17.409383+0000","last_unstale":"2024-03-06T07:27:17.411708+0000","last_undegraded":"2024-03-06T07:27:17.411708+0000","last_fullsized":"2024-03-06T07:27:17.411708+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:09.299098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400137+0000","last_change":"2024-03-06T07:27:17.400137+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400137+0000","last_undegraded":"2024-03-06T07:27:17.400137+0000","last_fullsized":"2024-03-06T07:27:17.400137+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.420704+0000","last_change":"2024-03-06T07:27:17.420352+0000","last_active":"2024-03-06T07:27:17.420704+0000","last_peered":"2024-03-06T07:27:17.420704+0000","last_clean":"2024-03-06T07:27:17.420704+0000","last_became_active":"2024-03-06T07:27:17.420070+0000","last_became_peered":"2024-03-06T07:27:17.420070+0000","last_unstale":"2024-03-06T07:27:17.420704+0000","last_undegraded":"2024-03-06T07:27:17.420704+0000","last_fullsized":"2024-03-06T07:27:17.420704+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:19.650711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400097+0000","last_change":"2024-03-06T07:27:17.400097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400097+0000","last_undegraded":"2024-03-06T07:27:17.400097+0000","last_fullsized":"2024-03-06T07:27:17.400097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417219+0000","last_change":"2024-03-06T07:27:17.414539+0000","last_active":"2024-03-06T07:27:17.417219+0000","last_peered":"2024-03-06T07:27:17.417219+0000","last_clean":"2024-03-06T07:27:17.417219+0000","last_became_active":"2024-03-06T07:27:17.414268+0000","last_became_peered":"2024-03-06T07:27:17.414268+0000","last_unstale":"2024-03-06T07:27:17.417219+0000","last_undegraded":"2024-03-06T07:27:17.417219+0000","last_fullsized":"2024-03-06T07:27:17.417219+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:52.205799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405168+0000","last_change":"2024-03-06T07:27:17.405168+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405168+0000","last_undegraded":"2024-03-06T07:27:17.405168+0000","last_fullsized":"2024-03-06T07:27:17.405168+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.410930+0000","last_change":"2024-03-06T07:27:17.408812+0000","last_active":"2024-03-06T07:27:17.410930+0000","last_peered":"2024-03-06T07:27:17.410930+0000","last_clean":"2024-03-06T07:27:17.410930+0000","last_became_active":"2024-03-06T07:27:17.408558+0000","last_became_peered":"2024-03-06T07:27:17.408558+0000","last_unstale":"2024-03-06T07:27:17.410930+0000","last_undegraded":"2024-03-06T07:27:17.410930+0000","last_fullsized":"2024-03-06T07:27:17.410930+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:00.868016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400532+0000","last_change":"2024-03-06T07:27:17.400532+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400532+0000","last_undegraded":"2024-03-06T07:27:17.400532+0000","last_fullsized":"2024-03-06T07:27:17.400532+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415346+0000","last_change":"2024-03-06T07:27:17.408777+0000","last_active":"2024-03-06T07:27:17.415346+0000","last_peered":"2024-03-06T07:27:17.415346+0000","last_clean":"2024-03-06T07:27:17.415346+0000","last_became_active":"2024-03-06T07:27:17.408201+0000","last_became_peered":"2024-03-06T07:27:17.408201+0000","last_unstale":"2024-03-06T07:27:17.415346+0000","last_undegraded":"2024-03-06T07:27:17.415346+0000","last_fullsized":"2024-03-06T07:27:17.415346+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:42.980380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404461+0000","last_change":"2024-03-06T07:27:17.404461+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404461+0000","last_undegraded":"2024-03-06T07:27:17.404461+0000","last_fullsized":"2024-03-06T07:27:17.404461+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415292+0000","last_change":"2024-03-06T07:27:17.413559+0000","last_active":"2024-03-06T07:27:17.415292+0000","last_peered":"2024-03-06T07:27:17.415292+0000","last_clean":"2024-03-06T07:27:17.415292+0000","last_became_active":"2024-03-06T07:27:17.413298+0000","last_became_peered":"2024-03-06T07:27:17.413298+0000","last_unstale":"2024-03-06T07:27:17.415292+0000","last_undegraded":"2024-03-06T07:27:17.415292+0000","last_fullsized":"2024-03-06T07:27:17.415292+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:30.857109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403746+0000","last_change":"2024-03-06T07:27:17.403746+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403746+0000","last_undegraded":"2024-03-06T07:27:17.403746+0000","last_fullsized":"2024-03-06T07:27:17.403746+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416237+0000","last_change":"2024-03-06T07:27:17.415152+0000","last_active":"2024-03-06T07:27:17.416237+0000","last_peered":"2024-03-06T07:27:17.416237+0000","last_clean":"2024-03-06T07:27:17.416237+0000","last_became_active":"2024-03-06T07:27:17.414894+0000","last_became_peered":"2024-03-06T07:27:17.414894+0000","last_unstale":"2024-03-06T07:27:17.416237+0000","last_undegraded":"2024-03-06T07:27:17.416237+0000","last_fullsized":"2024-03-06T07:27:17.416237+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:05.667791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410538+0000","last_change":"2024-03-06T07:27:17.410538+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410538+0000","last_undegraded":"2024-03-06T07:27:17.410538+0000","last_fullsized":"2024-03-06T07:27:17.410538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":3,"reported_epoch":207,"state":"creating+activating","last_fresh":"2024-03-06T07:27:17.409932+0000","last_change":"2024-03-06T07:27:17.409932+0000","last_active":"2024-03-06T07:27:16.385900+0000","last_peered":"2024-03-06T07:27:16.385900+0000","last_clean":"2024-03-06T07:27:16.385900+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409932+0000","last_undegraded":"2024-03-06T07:27:17.409932+0000","last_fullsized":"2024-03-06T07:27:17.409932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404585+0000","last_change":"2024-03-06T07:27:17.404585+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404585+0000","last_undegraded":"2024-03-06T07:27:17.404585+0000","last_fullsized":"2024-03-06T07:27:17.404585+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416845+0000","last_change":"2024-03-06T07:27:17.416845+0000","last_active":"2024-03-06T07:27:17.416845+0000","last_peered":"2024-03-06T07:27:17.416845+0000","last_clean":"2024-03-06T07:27:17.416845+0000","last_became_active":"2024-03-06T07:27:17.416518+0000","last_became_peered":"2024-03-06T07:27:17.416518+0000","last_unstale":"2024-03-06T07:27:17.416845+0000","last_undegraded":"2024-03-06T07:27:17.416845+0000","last_fullsized":"2024-03-06T07:27:17.416845+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:09.916243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405097+0000","last_change":"2024-03-06T07:27:17.405097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405097+0000","last_undegraded":"2024-03-06T07:27:17.405097+0000","last_fullsized":"2024-03-06T07:27:17.405097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408436+0000","last_change":"2024-03-06T07:27:17.407570+0000","last_active":"2024-03-06T07:27:17.408436+0000","last_peered":"2024-03-06T07:27:17.408436+0000","last_clean":"2024-03-06T07:27:17.408436+0000","last_became_active":"2024-03-06T07:27:17.407285+0000","last_became_peered":"2024-03-06T07:27:17.407285+0000","last_unstale":"2024-03-06T07:27:17.408436+0000","last_undegraded":"2024-03-06T07:27:17.408436+0000","last_fullsized":"2024-03-06T07:27:17.408436+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:00.071014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400116+0000","last_change":"2024-03-06T07:27:17.400116+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400116+0000","last_undegraded":"2024-03-06T07:27:17.400116+0000","last_fullsized":"2024-03-06T07:27:17.400116+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417608+0000","last_change":"2024-03-06T07:27:17.415917+0000","last_active":"2024-03-06T07:27:17.417608+0000","last_peered":"2024-03-06T07:27:17.417608+0000","last_clean":"2024-03-06T07:27:17.417608+0000","last_became_active":"2024-03-06T07:27:17.414080+0000","last_became_peered":"2024-03-06T07:27:17.414080+0000","last_unstale":"2024-03-06T07:27:17.417608+0000","last_undegraded":"2024-03-06T07:27:17.417608+0000","last_fullsized":"2024-03-06T07:27:17.417608+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:20.177610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400115+0000","last_change":"2024-03-06T07:27:17.400115+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400115+0000","last_undegraded":"2024-03-06T07:27:17.400115+0000","last_fullsized":"2024-03-06T07:27:17.400115+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415761+0000","last_change":"2024-03-06T07:27:17.415251+0000","last_active":"2024-03-06T07:27:17.415761+0000","last_peered":"2024-03-06T07:27:17.415761+0000","last_clean":"2024-03-06T07:27:17.415761+0000","last_became_active":"2024-03-06T07:27:17.414993+0000","last_became_peered":"2024-03-06T07:27:17.414993+0000","last_unstale":"2024-03-06T07:27:17.415761+0000","last_undegraded":"2024-03-06T07:27:17.415761+0000","last_fullsized":"2024-03-06T07:27:17.415761+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400287+0000","last_change":"2024-03-06T07:27:17.400287+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400287+0000","last_undegraded":"2024-03-06T07:27:17.400287+0000","last_fullsized":"2024-03-06T07:27:17.400287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413567+0000","last_change":"2024-03-06T07:27:17.412587+0000","last_active":"2024-03-06T07:27:17.413567+0000","last_peered":"2024-03-06T07:27:17.413567+0000","last_clean":"2024-03-06T07:27:17.413567+0000","last_became_active":"2024-03-06T07:27:17.412322+0000","last_became_peered":"2024-03-06T07:27:17.412322+0000","last_unstale":"2024-03-06T07:27:17.413567+0000","last_undegraded":"2024-03-06T07:27:17.413567+0000","last_fullsized":"2024-03-06T07:27:17.413567+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:51.540166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402352+0000","last_change":"2024-03-06T07:27:17.402352+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402352+0000","last_undegraded":"2024-03-06T07:27:17.402352+0000","last_fullsized":"2024-03-06T07:27:17.402352+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":522,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408721+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:27:17.408721+0000","last_peered":"2024-03-06T07:27:17.408721+0000","last_clean":"2024-03-06T07:27:17.408721+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:27:17.408721+0000","last_undegraded":"2024-03-06T07:27:17.408721+0000","last_fullsized":"2024-03-06T07:27:17.408721+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33800,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338040,"statfs":{"total":96636764160,"available":96602152960,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37904,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333936,"statfs":{"total":96636764160,"available":96597950464,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37960,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333880,"statfs":{"total":96636764160,"available":96597893120,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:21.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:21.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.511+0000 7f604c71e640 1 Processor -- start 2024-03-06T07:27:21.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.511+0000 7f604c71e640 1 -- start start 2024-03-06T07:27:21.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.511+0000 7f604c71e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f604414f790 0x7f60441529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f604c71e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60440c28d0 con 0x7f60440c1aa0 2024-03-06T07:27:21.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f604c71e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60440c2bb0 con 0x7f6044155a40 2024-03-06T07:27:21.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f604c71e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440c2680 con 0x7f604414f790 2024-03-06T07:27:21.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f6049cb9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f60440c1aa0 0x7f60440c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42834/0 (socket says 172.21.15.94:42834) 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f6049cb9640 1 -- 172.21.15.94:0/2082171772 learned_addr learned my addr 172.21.15.94:0/2082171772 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f604a4ba640 1 --2- 172.21.15.94:0/2082171772 >> v2:172.21.15.145:3300/0 conn(0x7f604414f790 0x7f60441529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998010183 0 0) 0x7f60440c28d0 con 0x7f60440c1aa0 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030003520 con 0x7f60440c1aa0 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187709846 0 0) 0x7f60440c2bb0 con 0x7f6044155a40 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60440c28d0 con 0x7f6044155a40 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 471794907 0 0) 0x7f6030003520 con 0x7f60440c1aa0 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 >> v2:172.21.15.145:3300/0 conn(0x7f604414f790 msgr2=0x7f60441529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 --2- 172.21.15.94:0/2082171772 >> v2:172.21.15.145:3300/0 conn(0x7f604414f790 0x7f60441529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 >> v1:172.21.15.145:6789/0 conn(0x7f6044155a40 legacy=0x7f6044155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440c2e90 con 0x7f60440c1aa0 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f6034001670 con 0x7f60440c1aa0 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 >> v1:172.21.15.94:6789/0 conn(0x7f60440c1aa0 legacy=0x7f60440c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030003b40 0x7f6030003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 0x7f6030004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60300051c0 0x7f603000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440c2680 con 0x7f6030003b40 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603000c060 con 0x7f6030004800 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603000c260 con 0x7f60300051c0 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f604a4ba640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030003b40 0x7f6030003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.512+0000 7f6049cb9640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 0x7f6030004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604acbb640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60300051c0 0x7f603000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604a4ba640 1 -- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60300051c0 msgr2=0x7f603000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604a4ba640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60300051c0 0x7f603000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604a4ba640 1 -- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 msgr2=0x7f6030004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604a4ba640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030004800 0x7f6030004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604a4ba640 1 -- 172.21.15.94:0/2082171772 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030003520 con 0x7f6030003b40 2024-03-06T07:27:21.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604acbb640 1 -- 172.21.15.94:0/2082171772 reap_dead start 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f604a4ba640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030003b40 0x7f6030003f60 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f6040009aa0 tx=0x7f6030005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4481f8724fe8bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.513+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040039070 con 0x7f6030003b40 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604002c6b0 con 0x7f6030003b40 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f60494b8640 1 -- 172.21.15.94:0/2082171772 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040009670 con 0x7f6030003b40 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f604c71e640 1 -- 172.21.15.94:0/2082171772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030003b40 msgr2=0x7f6030003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f604c71e640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030003b40 0x7f6030003f60 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f6040009aa0 tx=0x7f6030005830 comp rx=0 tx=0).stop 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f604c71e640 1 -- 172.21.15.94:0/2082171772 shutdown_connections 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f604c71e640 1 --2- 172.21.15.94:0/2082171772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6030003b40 0x7f6030003f60 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f604c71e640 1 -- 172.21.15.94:0/2082171772 >> 172.21.15.94:0/2082171772 conn(0x7f60440c6710 msgr2=0x7f60440c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f604c71e640 1 -- 172.21.15.94:0/2082171772 shutdown_connections 2024-03-06T07:27:21.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.514+0000 7f604c71e640 1 -- 172.21.15.94:0/2082171772 wait complete. 2024-03-06T07:27:21.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.515+0000 7f604c71e640 1 Processor -- start 2024-03-06T07:27:21.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.515+0000 7f604c71e640 1 -- start start 2024-03-06T07:27:21.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604c71e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030003b40 0x7f604414c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604c71e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6044072b00 0x7f60440a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604c71e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60440a7e50 0x7f6044149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604c71e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604406cc00 con 0x7f6044072b00 2024-03-06T07:27:21.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604c71e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440c0d60 con 0x7f6030003b40 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604c71e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604406cd80 con 0x7f60440a7e50 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6044072b00 0x7f60440a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6044072b00 0x7f60440a7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47816/0 (socket says 172.21.15.94:47816) 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 -- 172.21.15.94:0/244450924 learned_addr learned my addr 172.21.15.94:0/244450924 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604a4ba640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030003b40 0x7f604414c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f604acbb640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60440a7e50 0x7f6044149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 -- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60440a7e50 msgr2=0x7f6044149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60440a7e50 0x7f6044149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 -- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030003b40 msgr2=0x7f604414c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030003b40 0x7f604414c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 -- 172.21.15.94:0/244450924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60441493b0 con 0x7f6044072b00 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.516+0000 7f6049cb9640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6044072b00 0x7f60440a7910 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f6034004f00 tx=0x7f6034007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.517+0000 7f603affd640 1 -- 172.21.15.94:0/244450924 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603400f750 con 0x7f6044072b00 2024-03-06T07:27:21.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.517+0000 7f603affd640 1 -- 172.21.15.94:0/244450924 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034018070 con 0x7f6044072b00 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.517+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60441ecff0 con 0x7f6044072b00 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.517+0000 7f603affd640 1 -- 172.21.15.94:0/244450924 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603400fb50 con 0x7f6044072b00 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.517+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6044149620 con 0x7f6044072b00 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.517+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604406d4e0 con 0x7f6044072b00 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.518+0000 7f603affd640 1 -- 172.21.15.94:0/244450924 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f603400fcf0 con 0x7f6044072b00 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.518+0000 7f603affd640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.519+0000 7f603affd640 1 -- 172.21.15.94:0/244450924 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f603408a080 con 0x7f6044072b00 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.519+0000 7f604a4ba640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.519+0000 7f604a4ba640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f6040036000 tx=0x7f6040030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.523+0000 7f603affd640 1 -- 172.21.15.94:0/244450924 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f604406d4e0 con 0x7f6044072b00 2024-03-06T07:27:21.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.691+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f60440a8230 con 0x7f602806b4d0 2024-03-06T07:27:21.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.697+0000 7f603affd640 1 -- 172.21.15.94:0/244450924 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247072 (secure 0 0 0) 0x7f60440a8230 con 0x7f602806b4d0 2024-03-06T07:27:21.701 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:21.703 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:27:21.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b4d0 msgr2=0x7f602806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b4d0 0x7f602806d990 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f6040036000 tx=0x7f6040030000 comp rx=0 tx=0).stop 2024-03-06T07:27:21.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6044072b00 msgr2=0x7f60440a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6044072b00 0x7f60440a7910 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f6034004f00 tx=0x7f6034007430 comp rx=0 tx=0).stop 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 shutdown_connections 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f602806b4d0 0x7f602806d990 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60440a7e50 0x7f6044149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6044072b00 0x7f60440a7910 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 --2- 172.21.15.94:0/244450924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6030003b40 0x7f604414c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 >> 172.21.15.94:0/244450924 conn(0x7f60440c6710 msgr2=0x7f60440ab070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 shutdown_connections 2024-03-06T07:27:21.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.700+0000 7f604c71e640 1 -- 172.21.15.94:0/244450924 wait complete. 2024-03-06T07:27:21.712 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-03-06T07:27:20.757508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278380,"kb_used_data":20576,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754696340,"statfs":{"total":773094113280,"available":772809052160,"internally_reserved":0,"allocated":21069824,"data_stored":11304944,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.366341"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405718+0000","last_change":"2024-03-06T07:27:17.405718+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405718+0000","last_undegraded":"2024-03-06T07:27:17.405718+0000","last_fullsized":"2024-03-06T07:27:17.405718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405159+0000","last_change":"2024-03-06T07:27:17.405159+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405159+0000","last_undegraded":"2024-03-06T07:27:17.405159+0000","last_fullsized":"2024-03-06T07:27:17.405159+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410596+0000","last_change":"2024-03-06T07:27:17.410596+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410596+0000","last_undegraded":"2024-03-06T07:27:17.410596+0000","last_fullsized":"2024-03-06T07:27:17.410596+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403936+0000","last_change":"2024-03-06T07:27:17.403936+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403936+0000","last_undegraded":"2024-03-06T07:27:17.403936+0000","last_fullsized":"2024-03-06T07:27:17.403936+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406232+0000","last_change":"2024-03-06T07:27:17.406232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406232+0000","last_undegraded":"2024-03-06T07:27:17.406232+0000","last_fullsized":"2024-03-06T07:27:17.406232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404786+0000","last_change":"2024-03-06T07:27:17.404786+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404786+0000","last_undegraded":"2024-03-06T07:27:17.404786+0000","last_fullsized":"2024-03-06T07:27:17.404786+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409357+0000","last_change":"2024-03-06T07:27:17.409357+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409357+0000","last_undegraded":"2024-03-06T07:27:17.409357+0000","last_fullsized":"2024-03-06T07:27:17.409357+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405499+0000","last_change":"2024-03-06T07:27:17.405499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405499+0000","last_undegraded":"2024-03-06T07:27:17.405499+0000","last_fullsized":"2024-03-06T07:27:17.405499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405024+0000","last_change":"2024-03-06T07:27:17.405024+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405024+0000","last_undegraded":"2024-03-06T07:27:17.405024+0000","last_fullsized":"2024-03-06T07:27:17.405024+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401578+0000","last_change":"2024-03-06T07:27:17.401578+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401578+0000","last_undegraded":"2024-03-06T07:27:17.401578+0000","last_fullsized":"2024-03-06T07:27:17.401578+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402332+0000","last_change":"2024-03-06T07:27:17.402332+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402332+0000","last_undegraded":"2024-03-06T07:27:17.402332+0000","last_fullsized":"2024-03-06T07:27:17.402332+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405310+0000","last_change":"2024-03-06T07:27:17.405310+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405310+0000","last_undegraded":"2024-03-06T07:27:17.405310+0000","last_fullsized":"2024-03-06T07:27:17.405310+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406144+0000","last_change":"2024-03-06T07:27:17.406144+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406144+0000","last_undegraded":"2024-03-06T07:27:17.406144+0000","last_fullsized":"2024-03-06T07:27:17.406144+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405327+0000","last_change":"2024-03-06T07:27:17.405327+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405327+0000","last_undegraded":"2024-03-06T07:27:17.405327+0000","last_fullsized":"2024-03-06T07:27:17.405327+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402499+0000","last_change":"2024-03-06T07:27:17.402499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402499+0000","last_undegraded":"2024-03-06T07:27:17.402499+0000","last_fullsized":"2024-03-06T07:27:17.402499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402943+0000","last_change":"2024-03-06T07:27:17.402943+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402943+0000","last_undegraded":"2024-03-06T07:27:17.402943+0000","last_fullsized":"2024-03-06T07:27:17.402943+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400619+0000","last_change":"2024-03-06T07:27:17.400619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400619+0000","last_undegraded":"2024-03-06T07:27:17.400619+0000","last_fullsized":"2024-03-06T07:27:17.400619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404639+0000","last_change":"2024-03-06T07:27:17.404639+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404639+0000","last_undegraded":"2024-03-06T07:27:17.404639+0000","last_fullsized":"2024-03-06T07:27:17.404639+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400272+0000","last_change":"2024-03-06T07:27:17.400272+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400272+0000","last_undegraded":"2024-03-06T07:27:17.400272+0000","last_fullsized":"2024-03-06T07:27:17.400272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404933+0000","last_change":"2024-03-06T07:27:17.404933+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404933+0000","last_undegraded":"2024-03-06T07:27:17.404933+0000","last_fullsized":"2024-03-06T07:27:17.404933+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405809+0000","last_change":"2024-03-06T07:27:17.405809+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405809+0000","last_undegraded":"2024-03-06T07:27:17.405809+0000","last_fullsized":"2024-03-06T07:27:17.405809+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401251+0000","last_change":"2024-03-06T07:27:17.401251+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401251+0000","last_undegraded":"2024-03-06T07:27:17.401251+0000","last_fullsized":"2024-03-06T07:27:17.401251+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402086+0000","last_change":"2024-03-06T07:27:17.402086+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402086+0000","last_undegraded":"2024-03-06T07:27:17.402086+0000","last_fullsized":"2024-03-06T07:27:17.402086+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400623+0000","last_change":"2024-03-06T07:27:17.400623+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400623+0000","last_undegraded":"2024-03-06T07:27:17.400623+0000","last_fullsized":"2024-03-06T07:27:17.400623+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410762+0000","last_change":"2024-03-06T07:27:17.410762+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410762+0000","last_undegraded":"2024-03-06T07:27:17.410762+0000","last_fullsized":"2024-03-06T07:27:17.410762+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401619+0000","last_change":"2024-03-06T07:27:17.401619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401619+0000","last_undegraded":"2024-03-06T07:27:17.401619+0000","last_fullsized":"2024-03-06T07:27:17.401619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401907+0000","last_change":"2024-03-06T07:27:17.401907+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401907+0000","last_undegraded":"2024-03-06T07:27:17.401907+0000","last_fullsized":"2024-03-06T07:27:17.401907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402685+0000","last_change":"2024-03-06T07:27:17.402685+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402685+0000","last_undegraded":"2024-03-06T07:27:17.402685+0000","last_fullsized":"2024-03-06T07:27:17.402685+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404113+0000","last_change":"2024-03-06T07:27:17.404113+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404113+0000","last_undegraded":"2024-03-06T07:27:17.404113+0000","last_fullsized":"2024-03-06T07:27:17.404113+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401612+0000","last_change":"2024-03-06T07:27:17.401612+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401612+0000","last_undegraded":"2024-03-06T07:27:17.401612+0000","last_fullsized":"2024-03-06T07:27:17.401612+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402531+0000","last_change":"2024-03-06T07:27:17.402531+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402531+0000","last_undegraded":"2024-03-06T07:27:17.402531+0000","last_fullsized":"2024-03-06T07:27:17.402531+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401801+0000","last_change":"2024-03-06T07:27:17.401801+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401801+0000","last_undegraded":"2024-03-06T07:27:17.401801+0000","last_fullsized":"2024-03-06T07:27:17.401801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413284+0000","last_change":"2024-03-06T07:27:17.410327+0000","last_active":"2024-03-06T07:27:17.413284+0000","last_peered":"2024-03-06T07:27:17.413284+0000","last_clean":"2024-03-06T07:27:17.413284+0000","last_became_active":"2024-03-06T07:27:17.409826+0000","last_became_peered":"2024-03-06T07:27:17.409826+0000","last_unstale":"2024-03-06T07:27:17.413284+0000","last_undegraded":"2024-03-06T07:27:17.413284+0000","last_fullsized":"2024-03-06T07:27:17.413284+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:31.182027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403864+0000","last_change":"2024-03-06T07:27:17.403864+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403864+0000","last_undegraded":"2024-03-06T07:27:17.403864+0000","last_fullsized":"2024-03-06T07:27:17.403864+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414725+0000","last_change":"2024-03-06T07:27:17.413513+0000","last_active":"2024-03-06T07:27:17.414725+0000","last_peered":"2024-03-06T07:27:17.414725+0000","last_clean":"2024-03-06T07:27:17.414725+0000","last_became_active":"2024-03-06T07:27:17.413247+0000","last_became_peered":"2024-03-06T07:27:17.413247+0000","last_unstale":"2024-03-06T07:27:17.414725+0000","last_undegraded":"2024-03-06T07:27:17.414725+0000","last_fullsized":"2024-03-06T07:27:17.414725+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404232+0000","last_change":"2024-03-06T07:27:17.404232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404232+0000","last_undegraded":"2024-03-06T07:27:17.404232+0000","last_fullsized":"2024-03-06T07:27:17.404232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418162+0000","last_change":"2024-03-06T07:27:17.415933+0000","last_active":"2024-03-06T07:27:17.418162+0000","last_peered":"2024-03-06T07:27:17.418162+0000","last_clean":"2024-03-06T07:27:17.418162+0000","last_became_active":"2024-03-06T07:27:17.415652+0000","last_became_peered":"2024-03-06T07:27:17.415652+0000","last_unstale":"2024-03-06T07:27:17.418162+0000","last_undegraded":"2024-03-06T07:27:17.418162+0000","last_fullsized":"2024-03-06T07:27:17.418162+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:05.886687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400666+0000","last_change":"2024-03-06T07:27:17.400666+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400666+0000","last_undegraded":"2024-03-06T07:27:17.400666+0000","last_fullsized":"2024-03-06T07:27:17.400666+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413520+0000","last_change":"2024-03-06T07:27:17.412515+0000","last_active":"2024-03-06T07:27:17.413520+0000","last_peered":"2024-03-06T07:27:17.413520+0000","last_clean":"2024-03-06T07:27:17.413520+0000","last_became_active":"2024-03-06T07:27:17.412246+0000","last_became_peered":"2024-03-06T07:27:17.412246+0000","last_unstale":"2024-03-06T07:27:17.413520+0000","last_undegraded":"2024-03-06T07:27:17.413520+0000","last_fullsized":"2024-03-06T07:27:17.413520+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400981+0000","last_change":"2024-03-06T07:27:17.400981+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400981+0000","last_undegraded":"2024-03-06T07:27:17.400981+0000","last_fullsized":"2024-03-06T07:27:17.400981+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418993+0000","last_change":"2024-03-06T07:27:17.418064+0000","last_active":"2024-03-06T07:27:17.418993+0000","last_peered":"2024-03-06T07:27:17.418993+0000","last_clean":"2024-03-06T07:27:17.418993+0000","last_became_active":"2024-03-06T07:27:17.417764+0000","last_became_peered":"2024-03-06T07:27:17.417764+0000","last_unstale":"2024-03-06T07:27:17.418993+0000","last_undegraded":"2024-03-06T07:27:17.418993+0000","last_fullsized":"2024-03-06T07:27:17.418993+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:30.997230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400603+0000","last_change":"2024-03-06T07:27:17.400603+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400603+0000","last_undegraded":"2024-03-06T07:27:17.400603+0000","last_fullsized":"2024-03-06T07:27:17.400603+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416413+0000","last_change":"2024-03-06T07:27:17.415618+0000","last_active":"2024-03-06T07:27:17.416413+0000","last_peered":"2024-03-06T07:27:17.416413+0000","last_clean":"2024-03-06T07:27:17.416413+0000","last_became_active":"2024-03-06T07:27:17.415113+0000","last_became_peered":"2024-03-06T07:27:17.415113+0000","last_unstale":"2024-03-06T07:27:17.416413+0000","last_undegraded":"2024-03-06T07:27:17.416413+0000","last_fullsized":"2024-03-06T07:27:17.416413+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409868+0000","last_change":"2024-03-06T07:27:17.409868+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409868+0000","last_undegraded":"2024-03-06T07:27:17.409868+0000","last_fullsized":"2024-03-06T07:27:17.409868+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412010+0000","last_change":"2024-03-06T07:27:17.410657+0000","last_active":"2024-03-06T07:27:17.412010+0000","last_peered":"2024-03-06T07:27:17.412010+0000","last_clean":"2024-03-06T07:27:17.412010+0000","last_became_active":"2024-03-06T07:27:17.410330+0000","last_became_peered":"2024-03-06T07:27:17.410330+0000","last_unstale":"2024-03-06T07:27:17.412010+0000","last_undegraded":"2024-03-06T07:27:17.412010+0000","last_fullsized":"2024-03-06T07:27:17.412010+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:32.740313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401908+0000","last_change":"2024-03-06T07:27:17.401908+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401908+0000","last_undegraded":"2024-03-06T07:27:17.401908+0000","last_fullsized":"2024-03-06T07:27:17.401908+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412932+0000","last_change":"2024-03-06T07:27:17.412182+0000","last_active":"2024-03-06T07:27:17.412932+0000","last_peered":"2024-03-06T07:27:17.412932+0000","last_clean":"2024-03-06T07:27:17.412932+0000","last_became_active":"2024-03-06T07:27:17.411934+0000","last_became_peered":"2024-03-06T07:27:17.411934+0000","last_unstale":"2024-03-06T07:27:17.412932+0000","last_undegraded":"2024-03-06T07:27:17.412932+0000","last_fullsized":"2024-03-06T07:27:17.412932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:18.741598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402610+0000","last_change":"2024-03-06T07:27:17.402610+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402610+0000","last_undegraded":"2024-03-06T07:27:17.402610+0000","last_fullsized":"2024-03-06T07:27:17.402610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415272+0000","last_change":"2024-03-06T07:27:17.412868+0000","last_active":"2024-03-06T07:27:17.415272+0000","last_peered":"2024-03-06T07:27:17.415272+0000","last_clean":"2024-03-06T07:27:17.415272+0000","last_became_active":"2024-03-06T07:27:17.412587+0000","last_became_peered":"2024-03-06T07:27:17.412587+0000","last_unstale":"2024-03-06T07:27:17.415272+0000","last_undegraded":"2024-03-06T07:27:17.415272+0000","last_fullsized":"2024-03-06T07:27:17.415272+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:36.738070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403419+0000","last_change":"2024-03-06T07:27:17.403419+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403419+0000","last_undegraded":"2024-03-06T07:27:17.403419+0000","last_fullsized":"2024-03-06T07:27:17.403419+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411419+0000","last_change":"2024-03-06T07:27:17.410408+0000","last_active":"2024-03-06T07:27:17.411419+0000","last_peered":"2024-03-06T07:27:17.411419+0000","last_clean":"2024-03-06T07:27:17.411419+0000","last_became_active":"2024-03-06T07:27:17.410121+0000","last_became_peered":"2024-03-06T07:27:17.410121+0000","last_unstale":"2024-03-06T07:27:17.411419+0000","last_undegraded":"2024-03-06T07:27:17.411419+0000","last_fullsized":"2024-03-06T07:27:17.411419+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:21.045449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405727+0000","last_change":"2024-03-06T07:27:17.405727+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405727+0000","last_undegraded":"2024-03-06T07:27:17.405727+0000","last_fullsized":"2024-03-06T07:27:17.405727+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416587+0000","last_change":"2024-03-06T07:27:17.414830+0000","last_active":"2024-03-06T07:27:17.416587+0000","last_peered":"2024-03-06T07:27:17.416587+0000","last_clean":"2024-03-06T07:27:17.416587+0000","last_became_active":"2024-03-06T07:27:17.414541+0000","last_became_peered":"2024-03-06T07:27:17.414541+0000","last_unstale":"2024-03-06T07:27:17.416587+0000","last_undegraded":"2024-03-06T07:27:17.416587+0000","last_fullsized":"2024-03-06T07:27:17.416587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:46.438918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405165+0000","last_change":"2024-03-06T07:27:17.405165+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405165+0000","last_undegraded":"2024-03-06T07:27:17.405165+0000","last_fullsized":"2024-03-06T07:27:17.405165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414323+0000","last_change":"2024-03-06T07:27:17.413318+0000","last_active":"2024-03-06T07:27:17.414323+0000","last_peered":"2024-03-06T07:27:17.414323+0000","last_clean":"2024-03-06T07:27:17.414323+0000","last_became_active":"2024-03-06T07:27:17.413042+0000","last_became_peered":"2024-03-06T07:27:17.413042+0000","last_unstale":"2024-03-06T07:27:17.414323+0000","last_undegraded":"2024-03-06T07:27:17.414323+0000","last_fullsized":"2024-03-06T07:27:17.414323+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:57.629596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.407372+0000","last_change":"2024-03-06T07:27:17.407372+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.407372+0000","last_undegraded":"2024-03-06T07:27:17.407372+0000","last_fullsized":"2024-03-06T07:27:17.407372+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412696+0000","last_change":"2024-03-06T07:27:17.412118+0000","last_active":"2024-03-06T07:27:17.412696+0000","last_peered":"2024-03-06T07:27:17.412696+0000","last_clean":"2024-03-06T07:27:17.412696+0000","last_became_active":"2024-03-06T07:27:17.411828+0000","last_became_peered":"2024-03-06T07:27:17.411828+0000","last_unstale":"2024-03-06T07:27:17.412696+0000","last_undegraded":"2024-03-06T07:27:17.412696+0000","last_fullsized":"2024-03-06T07:27:17.412696+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:07.702608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400917+0000","last_change":"2024-03-06T07:27:17.400917+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400917+0000","last_undegraded":"2024-03-06T07:27:17.400917+0000","last_fullsized":"2024-03-06T07:27:17.400917+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419663+0000","last_change":"2024-03-06T07:27:17.416024+0000","last_active":"2024-03-06T07:27:17.419663+0000","last_peered":"2024-03-06T07:27:17.419663+0000","last_clean":"2024-03-06T07:27:17.419663+0000","last_became_active":"2024-03-06T07:27:17.415738+0000","last_became_peered":"2024-03-06T07:27:17.415738+0000","last_unstale":"2024-03-06T07:27:17.419663+0000","last_undegraded":"2024-03-06T07:27:17.419663+0000","last_fullsized":"2024-03-06T07:27:17.419663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:26.216018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.408899+0000","last_change":"2024-03-06T07:27:17.408899+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.408899+0000","last_undegraded":"2024-03-06T07:27:17.408899+0000","last_fullsized":"2024-03-06T07:27:17.408899+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419824+0000","last_change":"2024-03-06T07:27:17.416751+0000","last_active":"2024-03-06T07:27:17.419824+0000","last_peered":"2024-03-06T07:27:17.419824+0000","last_clean":"2024-03-06T07:27:17.419824+0000","last_became_active":"2024-03-06T07:27:17.416475+0000","last_became_peered":"2024-03-06T07:27:17.416475+0000","last_unstale":"2024-03-06T07:27:17.419824+0000","last_undegraded":"2024-03-06T07:27:17.419824+0000","last_fullsized":"2024-03-06T07:27:17.419824+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:34.635712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405423+0000","last_change":"2024-03-06T07:27:17.405423+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405423+0000","last_undegraded":"2024-03-06T07:27:17.405423+0000","last_fullsized":"2024-03-06T07:27:17.405423+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415326+0000","last_change":"2024-03-06T07:27:17.408458+0000","last_active":"2024-03-06T07:27:17.415326+0000","last_peered":"2024-03-06T07:27:17.415326+0000","last_clean":"2024-03-06T07:27:17.415326+0000","last_became_active":"2024-03-06T07:27:17.408204+0000","last_became_peered":"2024-03-06T07:27:17.408204+0000","last_unstale":"2024-03-06T07:27:17.415326+0000","last_undegraded":"2024-03-06T07:27:17.415326+0000","last_fullsized":"2024-03-06T07:27:17.415326+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:31.044701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400820+0000","last_change":"2024-03-06T07:27:17.400820+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400820+0000","last_undegraded":"2024-03-06T07:27:17.400820+0000","last_fullsized":"2024-03-06T07:27:17.400820+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.423682+0000","last_change":"2024-03-06T07:27:17.418167+0000","last_active":"2024-03-06T07:27:17.423682+0000","last_peered":"2024-03-06T07:27:17.423682+0000","last_clean":"2024-03-06T07:27:17.423682+0000","last_became_active":"2024-03-06T07:27:17.417877+0000","last_became_peered":"2024-03-06T07:27:17.417877+0000","last_unstale":"2024-03-06T07:27:17.423682+0000","last_undegraded":"2024-03-06T07:27:17.423682+0000","last_fullsized":"2024-03-06T07:27:17.423682+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401456+0000","last_change":"2024-03-06T07:27:17.401456+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401456+0000","last_undegraded":"2024-03-06T07:27:17.401456+0000","last_fullsized":"2024-03-06T07:27:17.401456+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413865+0000","last_change":"2024-03-06T07:27:17.409593+0000","last_active":"2024-03-06T07:27:17.413865+0000","last_peered":"2024-03-06T07:27:17.413865+0000","last_clean":"2024-03-06T07:27:17.413865+0000","last_became_active":"2024-03-06T07:27:17.409303+0000","last_became_peered":"2024-03-06T07:27:17.409303+0000","last_unstale":"2024-03-06T07:27:17.413865+0000","last_undegraded":"2024-03-06T07:27:17.413865+0000","last_fullsized":"2024-03-06T07:27:17.413865+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:58.310257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403395+0000","last_change":"2024-03-06T07:27:17.403395+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403395+0000","last_undegraded":"2024-03-06T07:27:17.403395+0000","last_fullsized":"2024-03-06T07:27:17.403395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416748+0000","last_change":"2024-03-06T07:27:17.416748+0000","last_active":"2024-03-06T07:27:17.416748+0000","last_peered":"2024-03-06T07:27:17.416748+0000","last_clean":"2024-03-06T07:27:17.416748+0000","last_became_active":"2024-03-06T07:27:17.416473+0000","last_became_peered":"2024-03-06T07:27:17.416473+0000","last_unstale":"2024-03-06T07:27:17.416748+0000","last_undegraded":"2024-03-06T07:27:17.416748+0000","last_fullsized":"2024-03-06T07:27:17.416748+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404729+0000","last_change":"2024-03-06T07:27:17.404729+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404729+0000","last_undegraded":"2024-03-06T07:27:17.404729+0000","last_fullsized":"2024-03-06T07:27:17.404729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411708+0000","last_change":"2024-03-06T07:27:17.409661+0000","last_active":"2024-03-06T07:27:17.411708+0000","last_peered":"2024-03-06T07:27:17.411708+0000","last_clean":"2024-03-06T07:27:17.411708+0000","last_became_active":"2024-03-06T07:27:17.409383+0000","last_became_peered":"2024-03-06T07:27:17.409383+0000","last_unstale":"2024-03-06T07:27:17.411708+0000","last_undegraded":"2024-03-06T07:27:17.411708+0000","last_fullsized":"2024-03-06T07:27:17.411708+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:09.299098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400137+0000","last_change":"2024-03-06T07:27:17.400137+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400137+0000","last_undegraded":"2024-03-06T07:27:17.400137+0000","last_fullsized":"2024-03-06T07:27:17.400137+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.420704+0000","last_change":"2024-03-06T07:27:17.420352+0000","last_active":"2024-03-06T07:27:17.420704+0000","last_peered":"2024-03-06T07:27:17.420704+0000","last_clean":"2024-03-06T07:27:17.420704+0000","last_became_active":"2024-03-06T07:27:17.420070+0000","last_became_peered":"2024-03-06T07:27:17.420070+0000","last_unstale":"2024-03-06T07:27:17.420704+0000","last_undegraded":"2024-03-06T07:27:17.420704+0000","last_fullsized":"2024-03-06T07:27:17.420704+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:19.650711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400097+0000","last_change":"2024-03-06T07:27:17.400097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400097+0000","last_undegraded":"2024-03-06T07:27:17.400097+0000","last_fullsized":"2024-03-06T07:27:17.400097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417219+0000","last_change":"2024-03-06T07:27:17.414539+0000","last_active":"2024-03-06T07:27:17.417219+0000","last_peered":"2024-03-06T07:27:17.417219+0000","last_clean":"2024-03-06T07:27:17.417219+0000","last_became_active":"2024-03-06T07:27:17.414268+0000","last_became_peered":"2024-03-06T07:27:17.414268+0000","last_unstale":"2024-03-06T07:27:17.417219+0000","last_undegraded":"2024-03-06T07:27:17.417219+0000","last_fullsized":"2024-03-06T07:27:17.417219+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:52.205799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405168+0000","last_change":"2024-03-06T07:27:17.405168+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405168+0000","last_undegraded":"2024-03-06T07:27:17.405168+0000","last_fullsized":"2024-03-06T07:27:17.405168+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.410930+0000","last_change":"2024-03-06T07:27:17.408812+0000","last_active":"2024-03-06T07:27:17.410930+0000","last_peered":"2024-03-06T07:27:17.410930+0000","last_clean":"2024-03-06T07:27:17.410930+0000","last_became_active":"2024-03-06T07:27:17.408558+0000","last_became_peered":"2024-03-06T07:27:17.408558+0000","last_unstale":"2024-03-06T07:27:17.410930+0000","last_undegraded":"2024-03-06T07:27:17.410930+0000","last_fullsized":"2024-03-06T07:27:17.410930+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:00.868016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400532+0000","last_change":"2024-03-06T07:27:17.400532+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400532+0000","last_undegraded":"2024-03-06T07:27:17.400532+0000","last_fullsized":"2024-03-06T07:27:17.400532+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415346+0000","last_change":"2024-03-06T07:27:17.408777+0000","last_active":"2024-03-06T07:27:17.415346+0000","last_peered":"2024-03-06T07:27:17.415346+0000","last_clean":"2024-03-06T07:27:17.415346+0000","last_became_active":"2024-03-06T07:27:17.408201+0000","last_became_peered":"2024-03-06T07:27:17.408201+0000","last_unstale":"2024-03-06T07:27:17.415346+0000","last_undegraded":"2024-03-06T07:27:17.415346+0000","last_fullsized":"2024-03-06T07:27:17.415346+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:42.980380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404461+0000","last_change":"2024-03-06T07:27:17.404461+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404461+0000","last_undegraded":"2024-03-06T07:27:17.404461+0000","last_fullsized":"2024-03-06T07:27:17.404461+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415292+0000","last_change":"2024-03-06T07:27:17.413559+0000","last_active":"2024-03-06T07:27:17.415292+0000","last_peered":"2024-03-06T07:27:17.415292+0000","last_clean":"2024-03-06T07:27:17.415292+0000","last_became_active":"2024-03-06T07:27:17.413298+0000","last_became_peered":"2024-03-06T07:27:17.413298+0000","last_unstale":"2024-03-06T07:27:17.415292+0000","last_undegraded":"2024-03-06T07:27:17.415292+0000","last_fullsized":"2024-03-06T07:27:17.415292+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:30.857109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403746+0000","last_change":"2024-03-06T07:27:17.403746+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403746+0000","last_undegraded":"2024-03-06T07:27:17.403746+0000","last_fullsized":"2024-03-06T07:27:17.403746+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416237+0000","last_change":"2024-03-06T07:27:17.415152+0000","last_active":"2024-03-06T07:27:17.416237+0000","last_peered":"2024-03-06T07:27:17.416237+0000","last_clean":"2024-03-06T07:27:17.416237+0000","last_became_active":"2024-03-06T07:27:17.414894+0000","last_became_peered":"2024-03-06T07:27:17.414894+0000","last_unstale":"2024-03-06T07:27:17.416237+0000","last_undegraded":"2024-03-06T07:27:17.416237+0000","last_fullsized":"2024-03-06T07:27:17.416237+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:05.667791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410538+0000","last_change":"2024-03-06T07:27:17.410538+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410538+0000","last_undegraded":"2024-03-06T07:27:17.410538+0000","last_fullsized":"2024-03-06T07:27:17.410538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":3,"reported_epoch":207,"state":"creating+activating","last_fresh":"2024-03-06T07:27:17.409932+0000","last_change":"2024-03-06T07:27:17.409932+0000","last_active":"2024-03-06T07:27:16.385900+0000","last_peered":"2024-03-06T07:27:16.385900+0000","last_clean":"2024-03-06T07:27:16.385900+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409932+0000","last_undegraded":"2024-03-06T07:27:17.409932+0000","last_fullsized":"2024-03-06T07:27:17.409932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404585+0000","last_change":"2024-03-06T07:27:17.404585+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404585+0000","last_undegraded":"2024-03-06T07:27:17.404585+0000","last_fullsized":"2024-03-06T07:27:17.404585+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416845+0000","last_change":"2024-03-06T07:27:17.416845+0000","last_active":"2024-03-06T07:27:17.416845+0000","last_peered":"2024-03-06T07:27:17.416845+0000","last_clean":"2024-03-06T07:27:17.416845+0000","last_became_active":"2024-03-06T07:27:17.416518+0000","last_became_peered":"2024-03-06T07:27:17.416518+0000","last_unstale":"2024-03-06T07:27:17.416845+0000","last_undegraded":"2024-03-06T07:27:17.416845+0000","last_fullsized":"2024-03-06T07:27:17.416845+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:09.916243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405097+0000","last_change":"2024-03-06T07:27:17.405097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405097+0000","last_undegraded":"2024-03-06T07:27:17.405097+0000","last_fullsized":"2024-03-06T07:27:17.405097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408436+0000","last_change":"2024-03-06T07:27:17.407570+0000","last_active":"2024-03-06T07:27:17.408436+0000","last_peered":"2024-03-06T07:27:17.408436+0000","last_clean":"2024-03-06T07:27:17.408436+0000","last_became_active":"2024-03-06T07:27:17.407285+0000","last_became_peered":"2024-03-06T07:27:17.407285+0000","last_unstale":"2024-03-06T07:27:17.408436+0000","last_undegraded":"2024-03-06T07:27:17.408436+0000","last_fullsized":"2024-03-06T07:27:17.408436+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:00.071014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400116+0000","last_change":"2024-03-06T07:27:17.400116+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400116+0000","last_undegraded":"2024-03-06T07:27:17.400116+0000","last_fullsized":"2024-03-06T07:27:17.400116+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417608+0000","last_change":"2024-03-06T07:27:17.415917+0000","last_active":"2024-03-06T07:27:17.417608+0000","last_peered":"2024-03-06T07:27:17.417608+0000","last_clean":"2024-03-06T07:27:17.417608+0000","last_became_active":"2024-03-06T07:27:17.414080+0000","last_became_peered":"2024-03-06T07:27:17.414080+0000","last_unstale":"2024-03-06T07:27:17.417608+0000","last_undegraded":"2024-03-06T07:27:17.417608+0000","last_fullsized":"2024-03-06T07:27:17.417608+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:20.177610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400115+0000","last_change":"2024-03-06T07:27:17.400115+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400115+0000","last_undegraded":"2024-03-06T07:27:17.400115+0000","last_fullsized":"2024-03-06T07:27:17.400115+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415761+0000","last_change":"2024-03-06T07:27:17.415251+0000","last_active":"2024-03-06T07:27:17.415761+0000","last_peered":"2024-03-06T07:27:17.415761+0000","last_clean":"2024-03-06T07:27:17.415761+0000","last_became_active":"2024-03-06T07:27:17.414993+0000","last_became_peered":"2024-03-06T07:27:17.414993+0000","last_unstale":"2024-03-06T07:27:17.415761+0000","last_undegraded":"2024-03-06T07:27:17.415761+0000","last_fullsized":"2024-03-06T07:27:17.415761+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400287+0000","last_change":"2024-03-06T07:27:17.400287+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400287+0000","last_undegraded":"2024-03-06T07:27:17.400287+0000","last_fullsized":"2024-03-06T07:27:17.400287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413567+0000","last_change":"2024-03-06T07:27:17.412587+0000","last_active":"2024-03-06T07:27:17.413567+0000","last_peered":"2024-03-06T07:27:17.413567+0000","last_clean":"2024-03-06T07:27:17.413567+0000","last_became_active":"2024-03-06T07:27:17.412322+0000","last_became_peered":"2024-03-06T07:27:17.412322+0000","last_unstale":"2024-03-06T07:27:17.413567+0000","last_undegraded":"2024-03-06T07:27:17.413567+0000","last_fullsized":"2024-03-06T07:27:17.413567+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:51.540166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402352+0000","last_change":"2024-03-06T07:27:17.402352+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402352+0000","last_undegraded":"2024-03-06T07:27:17.402352+0000","last_fullsized":"2024-03-06T07:27:17.402352+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":522,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408721+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:27:17.408721+0000","last_peered":"2024-03-06T07:27:17.408721+0000","last_clean":"2024-03-06T07:27:17.408721+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:27:17.408721+0000","last_undegraded":"2024-03-06T07:27:17.408721+0000","last_fullsized":"2024-03-06T07:27:17.408721+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33800,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338040,"statfs":{"total":96636764160,"available":96602152960,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37904,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333936,"statfs":{"total":96636764160,"available":96597950464,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37960,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333880,"statfs":{"total":96636764160,"available":96597893120,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:21.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:21.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.851+0000 7f95af45c640 1 Processor -- start 2024-03-06T07:27:21.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.851+0000 7f95af45c640 1 -- start start 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95af45c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f95a8150580 0x7f95a806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95af45c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95a80c1320 0x7f95a806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95af45c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c6000 0x7f95a81566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95af45c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a8153a60 con 0x7f95a80c1320 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95af45c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a8153d40 con 0x7f95a8150580 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95af45c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a806c2c0 con 0x7f95a80c6000 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95a80c1320 0x7f95a806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f95a80c1320 0x7f95a806e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47826/0 (socket says 172.21.15.94:47826) 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 -- 172.21.15.94:0/3707089182 learned_addr learned my addr 172.21.15.94:0/3707089182 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ad9f9640 1 --2- 172.21.15.94:0/3707089182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c6000 0x7f95a81566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ad1f8640 1 --2- 172.21.15.94:0/3707089182 >> v2:172.21.15.145:3300/0 conn(0x7f95a8150580 0x7f95a806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 -- 172.21.15.94:0/3707089182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c6000 msgr2=0x7f95a81566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 --2- 172.21.15.94:0/3707089182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c6000 0x7f95a81566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 -- 172.21.15.94:0/3707089182 >> v2:172.21.15.145:3300/0 conn(0x7f95a8150580 msgr2=0x7f95a806dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 --2- 172.21.15.94:0/3707089182 >> v2:172.21.15.145:3300/0 conn(0x7f95a8150580 0x7f95a806dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ac9f7640 1 -- 172.21.15.94:0/3707089182 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a806c5a0 con 0x7f95a80c1320 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.852+0000 7f95ad1f8640 1 --2- 172.21.15.94:0/3707089182 >> v2:172.21.15.145:3300/0 conn(0x7f95a8150580 0x7f95a806dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.853+0000 7f95ac9f7640 1 --2- 172.21.15.94:0/3707089182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a80c1320 0x7f95a806e250 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f95a80a9550 tx=0x7f959802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337dc5b4be877d70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.853+0000 7f959ffff640 1 -- 172.21.15.94:0/3707089182 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959802e070 con 0x7f95a80c1320 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.853+0000 7f959ffff640 1 -- 172.21.15.94:0/3707089182 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598037070 con 0x7f95a80c1320 2024-03-06T07:27:21.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.853+0000 7f959ffff640 1 -- 172.21.15.94:0/3707089182 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598031a00 con 0x7f95a80c1320 2024-03-06T07:27:21.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 -- 172.21.15.94:0/3707089182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a80c1320 msgr2=0x7f95a806e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 --2- 172.21.15.94:0/3707089182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a80c1320 0x7f95a806e250 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f95a80a9550 tx=0x7f959802b880 comp rx=0 tx=0).stop 2024-03-06T07:27:21.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 -- 172.21.15.94:0/3707089182 shutdown_connections 2024-03-06T07:27:21.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 --2- 172.21.15.94:0/3707089182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c6000 0x7f95a81566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 --2- 172.21.15.94:0/3707089182 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a80c1320 0x7f95a806e250 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 --2- 172.21.15.94:0/3707089182 >> v2:172.21.15.145:3300/0 conn(0x7f95a8150580 0x7f95a806dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 -- 172.21.15.94:0/3707089182 >> 172.21.15.94:0/3707089182 conn(0x7f95a80c6710 msgr2=0x7f95a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 -- 172.21.15.94:0/3707089182 shutdown_connections 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.854+0000 7f95af45c640 1 -- 172.21.15.94:0/3707089182 wait complete. 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 Processor -- start 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 -- start start 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c1320 0x7f95a80a7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95a80c6000 0x7f95a814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a8150580 0x7f95a814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80c0dc0 con 0x7f95a8150580 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80c0c40 con 0x7f95a80c6000 2024-03-06T07:27:21.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.855+0000 7f95af45c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a8156bf0 con 0x7f95a80c1320 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a8150580 0x7f95a814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a8150580 0x7f95a814d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47828/0 (socket says 172.21.15.94:47828) 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 -- 172.21.15.94:0/2443996960 learned_addr learned my addr 172.21.15.94:0/2443996960 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ac9f7640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95a80c6000 0x7f95a814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad1f8640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c1320 0x7f95a80a7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 -- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c1320 msgr2=0x7f95a80a7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c1320 0x7f95a80a7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 -- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95a80c6000 msgr2=0x7f95a814ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95a80c6000 0x7f95a814ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 -- 172.21.15.94:0/2443996960 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a81551e0 con 0x7f95a8150580 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad1f8640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c1320 0x7f95a80a7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95ad9f9640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a8150580 0x7f95a814d3c0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f95a000ef00 tx=0x7f95a000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f959dffb640 1 -- 172.21.15.94:0/2443996960 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0018020 con 0x7f95a8150580 2024-03-06T07:27:21.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f959dffb640 1 -- 172.21.15.94:0/2443996960 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a0014400 con 0x7f95a8150580 2024-03-06T07:27:21.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f959dffb640 1 -- 172.21.15.94:0/2443996960 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a00147f0 con 0x7f95a8150580 2024-03-06T07:27:21.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a814c3a0 con 0x7f95a8150580 2024-03-06T07:27:21.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.856+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a814c960 con 0x7f95a8150580 2024-03-06T07:27:21.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.857+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a804f330 con 0x7f95a8150580 2024-03-06T07:27:21.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.858+0000 7f959dffb640 1 -- 172.21.15.94:0/2443996960 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95a0005d10 con 0x7f95a8150580 2024-03-06T07:27:21.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.858+0000 7f959dffb640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958c06b4d0 0x7f958c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.858+0000 7f959dffb640 1 -- 172.21.15.94:0/2443996960 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95a008a9d0 con 0x7f95a8150580 2024-03-06T07:27:21.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.858+0000 7f95ad1f8640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958c06b4d0 0x7f958c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.859+0000 7f95ad1f8640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958c06b4d0 0x7f958c06d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009af0 tx=0x7f95a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:21.862+0000 7f959dffb640 1 -- 172.21.15.94:0/2443996960 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95a0022070 con 0x7f95a8150580 2024-03-06T07:27:22.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.028+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95a80c3a40 con 0x7f958c06b4d0 2024-03-06T07:27:22.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.034+0000 7f959dffb640 1 -- 172.21.15.94:0/2443996960 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247072 (secure 0 0 0) 0x7f95a80c3a40 con 0x7f958c06b4d0 2024-03-06T07:27:22.039 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:22.040 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:27:22.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958c06b4d0 msgr2=0x7f958c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958c06b4d0 0x7f958c06d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009af0 tx=0x7f95a4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:22.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a8150580 msgr2=0x7f95a814d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a8150580 0x7f95a814d3c0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f95a000ef00 tx=0x7f95a000b480 comp rx=0 tx=0).stop 2024-03-06T07:27:22.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 shutdown_connections 2024-03-06T07:27:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f958c06b4d0 0x7f958c06d990 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95a8150580 0x7f95a814d3c0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95a80c6000 0x7f95a814ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 --2- 172.21.15.94:0/2443996960 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95a80c1320 0x7f95a80a7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 >> 172.21.15.94:0/2443996960 conn(0x7f95a80c6710 msgr2=0x7f95a8148850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 shutdown_connections 2024-03-06T07:27:22.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.037+0000 7f95af45c640 1 -- 172.21.15.94:0/2443996960 wait complete. 2024-03-06T07:27:22.050 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-03-06T07:27:20.757508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278380,"kb_used_data":20576,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754696340,"statfs":{"total":773094113280,"available":772809052160,"internally_reserved":0,"allocated":21069824,"data_stored":11304944,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.366341"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405718+0000","last_change":"2024-03-06T07:27:17.405718+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405718+0000","last_undegraded":"2024-03-06T07:27:17.405718+0000","last_fullsized":"2024-03-06T07:27:17.405718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405159+0000","last_change":"2024-03-06T07:27:17.405159+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405159+0000","last_undegraded":"2024-03-06T07:27:17.405159+0000","last_fullsized":"2024-03-06T07:27:17.405159+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410596+0000","last_change":"2024-03-06T07:27:17.410596+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410596+0000","last_undegraded":"2024-03-06T07:27:17.410596+0000","last_fullsized":"2024-03-06T07:27:17.410596+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403936+0000","last_change":"2024-03-06T07:27:17.403936+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403936+0000","last_undegraded":"2024-03-06T07:27:17.403936+0000","last_fullsized":"2024-03-06T07:27:17.403936+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406232+0000","last_change":"2024-03-06T07:27:17.406232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406232+0000","last_undegraded":"2024-03-06T07:27:17.406232+0000","last_fullsized":"2024-03-06T07:27:17.406232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404786+0000","last_change":"2024-03-06T07:27:17.404786+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404786+0000","last_undegraded":"2024-03-06T07:27:17.404786+0000","last_fullsized":"2024-03-06T07:27:17.404786+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409357+0000","last_change":"2024-03-06T07:27:17.409357+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409357+0000","last_undegraded":"2024-03-06T07:27:17.409357+0000","last_fullsized":"2024-03-06T07:27:17.409357+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405499+0000","last_change":"2024-03-06T07:27:17.405499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405499+0000","last_undegraded":"2024-03-06T07:27:17.405499+0000","last_fullsized":"2024-03-06T07:27:17.405499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405024+0000","last_change":"2024-03-06T07:27:17.405024+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405024+0000","last_undegraded":"2024-03-06T07:27:17.405024+0000","last_fullsized":"2024-03-06T07:27:17.405024+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401578+0000","last_change":"2024-03-06T07:27:17.401578+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401578+0000","last_undegraded":"2024-03-06T07:27:17.401578+0000","last_fullsized":"2024-03-06T07:27:17.401578+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402332+0000","last_change":"2024-03-06T07:27:17.402332+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402332+0000","last_undegraded":"2024-03-06T07:27:17.402332+0000","last_fullsized":"2024-03-06T07:27:17.402332+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405310+0000","last_change":"2024-03-06T07:27:17.405310+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405310+0000","last_undegraded":"2024-03-06T07:27:17.405310+0000","last_fullsized":"2024-03-06T07:27:17.405310+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406144+0000","last_change":"2024-03-06T07:27:17.406144+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406144+0000","last_undegraded":"2024-03-06T07:27:17.406144+0000","last_fullsized":"2024-03-06T07:27:17.406144+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405327+0000","last_change":"2024-03-06T07:27:17.405327+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405327+0000","last_undegraded":"2024-03-06T07:27:17.405327+0000","last_fullsized":"2024-03-06T07:27:17.405327+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402499+0000","last_change":"2024-03-06T07:27:17.402499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402499+0000","last_undegraded":"2024-03-06T07:27:17.402499+0000","last_fullsized":"2024-03-06T07:27:17.402499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402943+0000","last_change":"2024-03-06T07:27:17.402943+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402943+0000","last_undegraded":"2024-03-06T07:27:17.402943+0000","last_fullsized":"2024-03-06T07:27:17.402943+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400619+0000","last_change":"2024-03-06T07:27:17.400619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400619+0000","last_undegraded":"2024-03-06T07:27:17.400619+0000","last_fullsized":"2024-03-06T07:27:17.400619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404639+0000","last_change":"2024-03-06T07:27:17.404639+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404639+0000","last_undegraded":"2024-03-06T07:27:17.404639+0000","last_fullsized":"2024-03-06T07:27:17.404639+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400272+0000","last_change":"2024-03-06T07:27:17.400272+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400272+0000","last_undegraded":"2024-03-06T07:27:17.400272+0000","last_fullsized":"2024-03-06T07:27:17.400272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404933+0000","last_change":"2024-03-06T07:27:17.404933+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404933+0000","last_undegraded":"2024-03-06T07:27:17.404933+0000","last_fullsized":"2024-03-06T07:27:17.404933+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405809+0000","last_change":"2024-03-06T07:27:17.405809+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405809+0000","last_undegraded":"2024-03-06T07:27:17.405809+0000","last_fullsized":"2024-03-06T07:27:17.405809+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401251+0000","last_change":"2024-03-06T07:27:17.401251+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401251+0000","last_undegraded":"2024-03-06T07:27:17.401251+0000","last_fullsized":"2024-03-06T07:27:17.401251+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402086+0000","last_change":"2024-03-06T07:27:17.402086+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402086+0000","last_undegraded":"2024-03-06T07:27:17.402086+0000","last_fullsized":"2024-03-06T07:27:17.402086+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400623+0000","last_change":"2024-03-06T07:27:17.400623+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400623+0000","last_undegraded":"2024-03-06T07:27:17.400623+0000","last_fullsized":"2024-03-06T07:27:17.400623+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410762+0000","last_change":"2024-03-06T07:27:17.410762+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410762+0000","last_undegraded":"2024-03-06T07:27:17.410762+0000","last_fullsized":"2024-03-06T07:27:17.410762+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401619+0000","last_change":"2024-03-06T07:27:17.401619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401619+0000","last_undegraded":"2024-03-06T07:27:17.401619+0000","last_fullsized":"2024-03-06T07:27:17.401619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401907+0000","last_change":"2024-03-06T07:27:17.401907+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401907+0000","last_undegraded":"2024-03-06T07:27:17.401907+0000","last_fullsized":"2024-03-06T07:27:17.401907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402685+0000","last_change":"2024-03-06T07:27:17.402685+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402685+0000","last_undegraded":"2024-03-06T07:27:17.402685+0000","last_fullsized":"2024-03-06T07:27:17.402685+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404113+0000","last_change":"2024-03-06T07:27:17.404113+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404113+0000","last_undegraded":"2024-03-06T07:27:17.404113+0000","last_fullsized":"2024-03-06T07:27:17.404113+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401612+0000","last_change":"2024-03-06T07:27:17.401612+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401612+0000","last_undegraded":"2024-03-06T07:27:17.401612+0000","last_fullsized":"2024-03-06T07:27:17.401612+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402531+0000","last_change":"2024-03-06T07:27:17.402531+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402531+0000","last_undegraded":"2024-03-06T07:27:17.402531+0000","last_fullsized":"2024-03-06T07:27:17.402531+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401801+0000","last_change":"2024-03-06T07:27:17.401801+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401801+0000","last_undegraded":"2024-03-06T07:27:17.401801+0000","last_fullsized":"2024-03-06T07:27:17.401801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413284+0000","last_change":"2024-03-06T07:27:17.410327+0000","last_active":"2024-03-06T07:27:17.413284+0000","last_peered":"2024-03-06T07:27:17.413284+0000","last_clean":"2024-03-06T07:27:17.413284+0000","last_became_active":"2024-03-06T07:27:17.409826+0000","last_became_peered":"2024-03-06T07:27:17.409826+0000","last_unstale":"2024-03-06T07:27:17.413284+0000","last_undegraded":"2024-03-06T07:27:17.413284+0000","last_fullsized":"2024-03-06T07:27:17.413284+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:31.182027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403864+0000","last_change":"2024-03-06T07:27:17.403864+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403864+0000","last_undegraded":"2024-03-06T07:27:17.403864+0000","last_fullsized":"2024-03-06T07:27:17.403864+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414725+0000","last_change":"2024-03-06T07:27:17.413513+0000","last_active":"2024-03-06T07:27:17.414725+0000","last_peered":"2024-03-06T07:27:17.414725+0000","last_clean":"2024-03-06T07:27:17.414725+0000","last_became_active":"2024-03-06T07:27:17.413247+0000","last_became_peered":"2024-03-06T07:27:17.413247+0000","last_unstale":"2024-03-06T07:27:17.414725+0000","last_undegraded":"2024-03-06T07:27:17.414725+0000","last_fullsized":"2024-03-06T07:27:17.414725+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404232+0000","last_change":"2024-03-06T07:27:17.404232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404232+0000","last_undegraded":"2024-03-06T07:27:17.404232+0000","last_fullsized":"2024-03-06T07:27:17.404232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418162+0000","last_change":"2024-03-06T07:27:17.415933+0000","last_active":"2024-03-06T07:27:17.418162+0000","last_peered":"2024-03-06T07:27:17.418162+0000","last_clean":"2024-03-06T07:27:17.418162+0000","last_became_active":"2024-03-06T07:27:17.415652+0000","last_became_peered":"2024-03-06T07:27:17.415652+0000","last_unstale":"2024-03-06T07:27:17.418162+0000","last_undegraded":"2024-03-06T07:27:17.418162+0000","last_fullsized":"2024-03-06T07:27:17.418162+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:05.886687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400666+0000","last_change":"2024-03-06T07:27:17.400666+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400666+0000","last_undegraded":"2024-03-06T07:27:17.400666+0000","last_fullsized":"2024-03-06T07:27:17.400666+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413520+0000","last_change":"2024-03-06T07:27:17.412515+0000","last_active":"2024-03-06T07:27:17.413520+0000","last_peered":"2024-03-06T07:27:17.413520+0000","last_clean":"2024-03-06T07:27:17.413520+0000","last_became_active":"2024-03-06T07:27:17.412246+0000","last_became_peered":"2024-03-06T07:27:17.412246+0000","last_unstale":"2024-03-06T07:27:17.413520+0000","last_undegraded":"2024-03-06T07:27:17.413520+0000","last_fullsized":"2024-03-06T07:27:17.413520+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400981+0000","last_change":"2024-03-06T07:27:17.400981+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400981+0000","last_undegraded":"2024-03-06T07:27:17.400981+0000","last_fullsized":"2024-03-06T07:27:17.400981+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418993+0000","last_change":"2024-03-06T07:27:17.418064+0000","last_active":"2024-03-06T07:27:17.418993+0000","last_peered":"2024-03-06T07:27:17.418993+0000","last_clean":"2024-03-06T07:27:17.418993+0000","last_became_active":"2024-03-06T07:27:17.417764+0000","last_became_peered":"2024-03-06T07:27:17.417764+0000","last_unstale":"2024-03-06T07:27:17.418993+0000","last_undegraded":"2024-03-06T07:27:17.418993+0000","last_fullsized":"2024-03-06T07:27:17.418993+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:30.997230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400603+0000","last_change":"2024-03-06T07:27:17.400603+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400603+0000","last_undegraded":"2024-03-06T07:27:17.400603+0000","last_fullsized":"2024-03-06T07:27:17.400603+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416413+0000","last_change":"2024-03-06T07:27:17.415618+0000","last_active":"2024-03-06T07:27:17.416413+0000","last_peered":"2024-03-06T07:27:17.416413+0000","last_clean":"2024-03-06T07:27:17.416413+0000","last_became_active":"2024-03-06T07:27:17.415113+0000","last_became_peered":"2024-03-06T07:27:17.415113+0000","last_unstale":"2024-03-06T07:27:17.416413+0000","last_undegraded":"2024-03-06T07:27:17.416413+0000","last_fullsized":"2024-03-06T07:27:17.416413+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409868+0000","last_change":"2024-03-06T07:27:17.409868+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409868+0000","last_undegraded":"2024-03-06T07:27:17.409868+0000","last_fullsized":"2024-03-06T07:27:17.409868+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412010+0000","last_change":"2024-03-06T07:27:17.410657+0000","last_active":"2024-03-06T07:27:17.412010+0000","last_peered":"2024-03-06T07:27:17.412010+0000","last_clean":"2024-03-06T07:27:17.412010+0000","last_became_active":"2024-03-06T07:27:17.410330+0000","last_became_peered":"2024-03-06T07:27:17.410330+0000","last_unstale":"2024-03-06T07:27:17.412010+0000","last_undegraded":"2024-03-06T07:27:17.412010+0000","last_fullsized":"2024-03-06T07:27:17.412010+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:32.740313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401908+0000","last_change":"2024-03-06T07:27:17.401908+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401908+0000","last_undegraded":"2024-03-06T07:27:17.401908+0000","last_fullsized":"2024-03-06T07:27:17.401908+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412932+0000","last_change":"2024-03-06T07:27:17.412182+0000","last_active":"2024-03-06T07:27:17.412932+0000","last_peered":"2024-03-06T07:27:17.412932+0000","last_clean":"2024-03-06T07:27:17.412932+0000","last_became_active":"2024-03-06T07:27:17.411934+0000","last_became_peered":"2024-03-06T07:27:17.411934+0000","last_unstale":"2024-03-06T07:27:17.412932+0000","last_undegraded":"2024-03-06T07:27:17.412932+0000","last_fullsized":"2024-03-06T07:27:17.412932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:18.741598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402610+0000","last_change":"2024-03-06T07:27:17.402610+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402610+0000","last_undegraded":"2024-03-06T07:27:17.402610+0000","last_fullsized":"2024-03-06T07:27:17.402610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415272+0000","last_change":"2024-03-06T07:27:17.412868+0000","last_active":"2024-03-06T07:27:17.415272+0000","last_peered":"2024-03-06T07:27:17.415272+0000","last_clean":"2024-03-06T07:27:17.415272+0000","last_became_active":"2024-03-06T07:27:17.412587+0000","last_became_peered":"2024-03-06T07:27:17.412587+0000","last_unstale":"2024-03-06T07:27:17.415272+0000","last_undegraded":"2024-03-06T07:27:17.415272+0000","last_fullsized":"2024-03-06T07:27:17.415272+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:36.738070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403419+0000","last_change":"2024-03-06T07:27:17.403419+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403419+0000","last_undegraded":"2024-03-06T07:27:17.403419+0000","last_fullsized":"2024-03-06T07:27:17.403419+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411419+0000","last_change":"2024-03-06T07:27:17.410408+0000","last_active":"2024-03-06T07:27:17.411419+0000","last_peered":"2024-03-06T07:27:17.411419+0000","last_clean":"2024-03-06T07:27:17.411419+0000","last_became_active":"2024-03-06T07:27:17.410121+0000","last_became_peered":"2024-03-06T07:27:17.410121+0000","last_unstale":"2024-03-06T07:27:17.411419+0000","last_undegraded":"2024-03-06T07:27:17.411419+0000","last_fullsized":"2024-03-06T07:27:17.411419+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:21.045449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405727+0000","last_change":"2024-03-06T07:27:17.405727+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405727+0000","last_undegraded":"2024-03-06T07:27:17.405727+0000","last_fullsized":"2024-03-06T07:27:17.405727+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416587+0000","last_change":"2024-03-06T07:27:17.414830+0000","last_active":"2024-03-06T07:27:17.416587+0000","last_peered":"2024-03-06T07:27:17.416587+0000","last_clean":"2024-03-06T07:27:17.416587+0000","last_became_active":"2024-03-06T07:27:17.414541+0000","last_became_peered":"2024-03-06T07:27:17.414541+0000","last_unstale":"2024-03-06T07:27:17.416587+0000","last_undegraded":"2024-03-06T07:27:17.416587+0000","last_fullsized":"2024-03-06T07:27:17.416587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:46.438918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405165+0000","last_change":"2024-03-06T07:27:17.405165+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405165+0000","last_undegraded":"2024-03-06T07:27:17.405165+0000","last_fullsized":"2024-03-06T07:27:17.405165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414323+0000","last_change":"2024-03-06T07:27:17.413318+0000","last_active":"2024-03-06T07:27:17.414323+0000","last_peered":"2024-03-06T07:27:17.414323+0000","last_clean":"2024-03-06T07:27:17.414323+0000","last_became_active":"2024-03-06T07:27:17.413042+0000","last_became_peered":"2024-03-06T07:27:17.413042+0000","last_unstale":"2024-03-06T07:27:17.414323+0000","last_undegraded":"2024-03-06T07:27:17.414323+0000","last_fullsized":"2024-03-06T07:27:17.414323+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:57.629596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.407372+0000","last_change":"2024-03-06T07:27:17.407372+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.407372+0000","last_undegraded":"2024-03-06T07:27:17.407372+0000","last_fullsized":"2024-03-06T07:27:17.407372+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412696+0000","last_change":"2024-03-06T07:27:17.412118+0000","last_active":"2024-03-06T07:27:17.412696+0000","last_peered":"2024-03-06T07:27:17.412696+0000","last_clean":"2024-03-06T07:27:17.412696+0000","last_became_active":"2024-03-06T07:27:17.411828+0000","last_became_peered":"2024-03-06T07:27:17.411828+0000","last_unstale":"2024-03-06T07:27:17.412696+0000","last_undegraded":"2024-03-06T07:27:17.412696+0000","last_fullsized":"2024-03-06T07:27:17.412696+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:07.702608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400917+0000","last_change":"2024-03-06T07:27:17.400917+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400917+0000","last_undegraded":"2024-03-06T07:27:17.400917+0000","last_fullsized":"2024-03-06T07:27:17.400917+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419663+0000","last_change":"2024-03-06T07:27:17.416024+0000","last_active":"2024-03-06T07:27:17.419663+0000","last_peered":"2024-03-06T07:27:17.419663+0000","last_clean":"2024-03-06T07:27:17.419663+0000","last_became_active":"2024-03-06T07:27:17.415738+0000","last_became_peered":"2024-03-06T07:27:17.415738+0000","last_unstale":"2024-03-06T07:27:17.419663+0000","last_undegraded":"2024-03-06T07:27:17.419663+0000","last_fullsized":"2024-03-06T07:27:17.419663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:26.216018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.408899+0000","last_change":"2024-03-06T07:27:17.408899+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.408899+0000","last_undegraded":"2024-03-06T07:27:17.408899+0000","last_fullsized":"2024-03-06T07:27:17.408899+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419824+0000","last_change":"2024-03-06T07:27:17.416751+0000","last_active":"2024-03-06T07:27:17.419824+0000","last_peered":"2024-03-06T07:27:17.419824+0000","last_clean":"2024-03-06T07:27:17.419824+0000","last_became_active":"2024-03-06T07:27:17.416475+0000","last_became_peered":"2024-03-06T07:27:17.416475+0000","last_unstale":"2024-03-06T07:27:17.419824+0000","last_undegraded":"2024-03-06T07:27:17.419824+0000","last_fullsized":"2024-03-06T07:27:17.419824+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:34.635712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405423+0000","last_change":"2024-03-06T07:27:17.405423+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405423+0000","last_undegraded":"2024-03-06T07:27:17.405423+0000","last_fullsized":"2024-03-06T07:27:17.405423+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415326+0000","last_change":"2024-03-06T07:27:17.408458+0000","last_active":"2024-03-06T07:27:17.415326+0000","last_peered":"2024-03-06T07:27:17.415326+0000","last_clean":"2024-03-06T07:27:17.415326+0000","last_became_active":"2024-03-06T07:27:17.408204+0000","last_became_peered":"2024-03-06T07:27:17.408204+0000","last_unstale":"2024-03-06T07:27:17.415326+0000","last_undegraded":"2024-03-06T07:27:17.415326+0000","last_fullsized":"2024-03-06T07:27:17.415326+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:31.044701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400820+0000","last_change":"2024-03-06T07:27:17.400820+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400820+0000","last_undegraded":"2024-03-06T07:27:17.400820+0000","last_fullsized":"2024-03-06T07:27:17.400820+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.423682+0000","last_change":"2024-03-06T07:27:17.418167+0000","last_active":"2024-03-06T07:27:17.423682+0000","last_peered":"2024-03-06T07:27:17.423682+0000","last_clean":"2024-03-06T07:27:17.423682+0000","last_became_active":"2024-03-06T07:27:17.417877+0000","last_became_peered":"2024-03-06T07:27:17.417877+0000","last_unstale":"2024-03-06T07:27:17.423682+0000","last_undegraded":"2024-03-06T07:27:17.423682+0000","last_fullsized":"2024-03-06T07:27:17.423682+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401456+0000","last_change":"2024-03-06T07:27:17.401456+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401456+0000","last_undegraded":"2024-03-06T07:27:17.401456+0000","last_fullsized":"2024-03-06T07:27:17.401456+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413865+0000","last_change":"2024-03-06T07:27:17.409593+0000","last_active":"2024-03-06T07:27:17.413865+0000","last_peered":"2024-03-06T07:27:17.413865+0000","last_clean":"2024-03-06T07:27:17.413865+0000","last_became_active":"2024-03-06T07:27:17.409303+0000","last_became_peered":"2024-03-06T07:27:17.409303+0000","last_unstale":"2024-03-06T07:27:17.413865+0000","last_undegraded":"2024-03-06T07:27:17.413865+0000","last_fullsized":"2024-03-06T07:27:17.413865+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:58.310257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403395+0000","last_change":"2024-03-06T07:27:17.403395+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403395+0000","last_undegraded":"2024-03-06T07:27:17.403395+0000","last_fullsized":"2024-03-06T07:27:17.403395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416748+0000","last_change":"2024-03-06T07:27:17.416748+0000","last_active":"2024-03-06T07:27:17.416748+0000","last_peered":"2024-03-06T07:27:17.416748+0000","last_clean":"2024-03-06T07:27:17.416748+0000","last_became_active":"2024-03-06T07:27:17.416473+0000","last_became_peered":"2024-03-06T07:27:17.416473+0000","last_unstale":"2024-03-06T07:27:17.416748+0000","last_undegraded":"2024-03-06T07:27:17.416748+0000","last_fullsized":"2024-03-06T07:27:17.416748+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404729+0000","last_change":"2024-03-06T07:27:17.404729+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404729+0000","last_undegraded":"2024-03-06T07:27:17.404729+0000","last_fullsized":"2024-03-06T07:27:17.404729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411708+0000","last_change":"2024-03-06T07:27:17.409661+0000","last_active":"2024-03-06T07:27:17.411708+0000","last_peered":"2024-03-06T07:27:17.411708+0000","last_clean":"2024-03-06T07:27:17.411708+0000","last_became_active":"2024-03-06T07:27:17.409383+0000","last_became_peered":"2024-03-06T07:27:17.409383+0000","last_unstale":"2024-03-06T07:27:17.411708+0000","last_undegraded":"2024-03-06T07:27:17.411708+0000","last_fullsized":"2024-03-06T07:27:17.411708+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:09.299098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400137+0000","last_change":"2024-03-06T07:27:17.400137+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400137+0000","last_undegraded":"2024-03-06T07:27:17.400137+0000","last_fullsized":"2024-03-06T07:27:17.400137+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.420704+0000","last_change":"2024-03-06T07:27:17.420352+0000","last_active":"2024-03-06T07:27:17.420704+0000","last_peered":"2024-03-06T07:27:17.420704+0000","last_clean":"2024-03-06T07:27:17.420704+0000","last_became_active":"2024-03-06T07:27:17.420070+0000","last_became_peered":"2024-03-06T07:27:17.420070+0000","last_unstale":"2024-03-06T07:27:17.420704+0000","last_undegraded":"2024-03-06T07:27:17.420704+0000","last_fullsized":"2024-03-06T07:27:17.420704+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:19.650711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400097+0000","last_change":"2024-03-06T07:27:17.400097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400097+0000","last_undegraded":"2024-03-06T07:27:17.400097+0000","last_fullsized":"2024-03-06T07:27:17.400097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417219+0000","last_change":"2024-03-06T07:27:17.414539+0000","last_active":"2024-03-06T07:27:17.417219+0000","last_peered":"2024-03-06T07:27:17.417219+0000","last_clean":"2024-03-06T07:27:17.417219+0000","last_became_active":"2024-03-06T07:27:17.414268+0000","last_became_peered":"2024-03-06T07:27:17.414268+0000","last_unstale":"2024-03-06T07:27:17.417219+0000","last_undegraded":"2024-03-06T07:27:17.417219+0000","last_fullsized":"2024-03-06T07:27:17.417219+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:52.205799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405168+0000","last_change":"2024-03-06T07:27:17.405168+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405168+0000","last_undegraded":"2024-03-06T07:27:17.405168+0000","last_fullsized":"2024-03-06T07:27:17.405168+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.410930+0000","last_change":"2024-03-06T07:27:17.408812+0000","last_active":"2024-03-06T07:27:17.410930+0000","last_peered":"2024-03-06T07:27:17.410930+0000","last_clean":"2024-03-06T07:27:17.410930+0000","last_became_active":"2024-03-06T07:27:17.408558+0000","last_became_peered":"2024-03-06T07:27:17.408558+0000","last_unstale":"2024-03-06T07:27:17.410930+0000","last_undegraded":"2024-03-06T07:27:17.410930+0000","last_fullsized":"2024-03-06T07:27:17.410930+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:00.868016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400532+0000","last_change":"2024-03-06T07:27:17.400532+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400532+0000","last_undegraded":"2024-03-06T07:27:17.400532+0000","last_fullsized":"2024-03-06T07:27:17.400532+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415346+0000","last_change":"2024-03-06T07:27:17.408777+0000","last_active":"2024-03-06T07:27:17.415346+0000","last_peered":"2024-03-06T07:27:17.415346+0000","last_clean":"2024-03-06T07:27:17.415346+0000","last_became_active":"2024-03-06T07:27:17.408201+0000","last_became_peered":"2024-03-06T07:27:17.408201+0000","last_unstale":"2024-03-06T07:27:17.415346+0000","last_undegraded":"2024-03-06T07:27:17.415346+0000","last_fullsized":"2024-03-06T07:27:17.415346+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:42.980380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404461+0000","last_change":"2024-03-06T07:27:17.404461+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404461+0000","last_undegraded":"2024-03-06T07:27:17.404461+0000","last_fullsized":"2024-03-06T07:27:17.404461+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415292+0000","last_change":"2024-03-06T07:27:17.413559+0000","last_active":"2024-03-06T07:27:17.415292+0000","last_peered":"2024-03-06T07:27:17.415292+0000","last_clean":"2024-03-06T07:27:17.415292+0000","last_became_active":"2024-03-06T07:27:17.413298+0000","last_became_peered":"2024-03-06T07:27:17.413298+0000","last_unstale":"2024-03-06T07:27:17.415292+0000","last_undegraded":"2024-03-06T07:27:17.415292+0000","last_fullsized":"2024-03-06T07:27:17.415292+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:30.857109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403746+0000","last_change":"2024-03-06T07:27:17.403746+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403746+0000","last_undegraded":"2024-03-06T07:27:17.403746+0000","last_fullsized":"2024-03-06T07:27:17.403746+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416237+0000","last_change":"2024-03-06T07:27:17.415152+0000","last_active":"2024-03-06T07:27:17.416237+0000","last_peered":"2024-03-06T07:27:17.416237+0000","last_clean":"2024-03-06T07:27:17.416237+0000","last_became_active":"2024-03-06T07:27:17.414894+0000","last_became_peered":"2024-03-06T07:27:17.414894+0000","last_unstale":"2024-03-06T07:27:17.416237+0000","last_undegraded":"2024-03-06T07:27:17.416237+0000","last_fullsized":"2024-03-06T07:27:17.416237+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:05.667791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410538+0000","last_change":"2024-03-06T07:27:17.410538+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410538+0000","last_undegraded":"2024-03-06T07:27:17.410538+0000","last_fullsized":"2024-03-06T07:27:17.410538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":3,"reported_epoch":207,"state":"creating+activating","last_fresh":"2024-03-06T07:27:17.409932+0000","last_change":"2024-03-06T07:27:17.409932+0000","last_active":"2024-03-06T07:27:16.385900+0000","last_peered":"2024-03-06T07:27:16.385900+0000","last_clean":"2024-03-06T07:27:16.385900+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409932+0000","last_undegraded":"2024-03-06T07:27:17.409932+0000","last_fullsized":"2024-03-06T07:27:17.409932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404585+0000","last_change":"2024-03-06T07:27:17.404585+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404585+0000","last_undegraded":"2024-03-06T07:27:17.404585+0000","last_fullsized":"2024-03-06T07:27:17.404585+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416845+0000","last_change":"2024-03-06T07:27:17.416845+0000","last_active":"2024-03-06T07:27:17.416845+0000","last_peered":"2024-03-06T07:27:17.416845+0000","last_clean":"2024-03-06T07:27:17.416845+0000","last_became_active":"2024-03-06T07:27:17.416518+0000","last_became_peered":"2024-03-06T07:27:17.416518+0000","last_unstale":"2024-03-06T07:27:17.416845+0000","last_undegraded":"2024-03-06T07:27:17.416845+0000","last_fullsized":"2024-03-06T07:27:17.416845+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:09.916243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405097+0000","last_change":"2024-03-06T07:27:17.405097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405097+0000","last_undegraded":"2024-03-06T07:27:17.405097+0000","last_fullsized":"2024-03-06T07:27:17.405097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408436+0000","last_change":"2024-03-06T07:27:17.407570+0000","last_active":"2024-03-06T07:27:17.408436+0000","last_peered":"2024-03-06T07:27:17.408436+0000","last_clean":"2024-03-06T07:27:17.408436+0000","last_became_active":"2024-03-06T07:27:17.407285+0000","last_became_peered":"2024-03-06T07:27:17.407285+0000","last_unstale":"2024-03-06T07:27:17.408436+0000","last_undegraded":"2024-03-06T07:27:17.408436+0000","last_fullsized":"2024-03-06T07:27:17.408436+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:00.071014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400116+0000","last_change":"2024-03-06T07:27:17.400116+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400116+0000","last_undegraded":"2024-03-06T07:27:17.400116+0000","last_fullsized":"2024-03-06T07:27:17.400116+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417608+0000","last_change":"2024-03-06T07:27:17.415917+0000","last_active":"2024-03-06T07:27:17.417608+0000","last_peered":"2024-03-06T07:27:17.417608+0000","last_clean":"2024-03-06T07:27:17.417608+0000","last_became_active":"2024-03-06T07:27:17.414080+0000","last_became_peered":"2024-03-06T07:27:17.414080+0000","last_unstale":"2024-03-06T07:27:17.417608+0000","last_undegraded":"2024-03-06T07:27:17.417608+0000","last_fullsized":"2024-03-06T07:27:17.417608+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:20.177610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400115+0000","last_change":"2024-03-06T07:27:17.400115+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400115+0000","last_undegraded":"2024-03-06T07:27:17.400115+0000","last_fullsized":"2024-03-06T07:27:17.400115+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415761+0000","last_change":"2024-03-06T07:27:17.415251+0000","last_active":"2024-03-06T07:27:17.415761+0000","last_peered":"2024-03-06T07:27:17.415761+0000","last_clean":"2024-03-06T07:27:17.415761+0000","last_became_active":"2024-03-06T07:27:17.414993+0000","last_became_peered":"2024-03-06T07:27:17.414993+0000","last_unstale":"2024-03-06T07:27:17.415761+0000","last_undegraded":"2024-03-06T07:27:17.415761+0000","last_fullsized":"2024-03-06T07:27:17.415761+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400287+0000","last_change":"2024-03-06T07:27:17.400287+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400287+0000","last_undegraded":"2024-03-06T07:27:17.400287+0000","last_fullsized":"2024-03-06T07:27:17.400287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413567+0000","last_change":"2024-03-06T07:27:17.412587+0000","last_active":"2024-03-06T07:27:17.413567+0000","last_peered":"2024-03-06T07:27:17.413567+0000","last_clean":"2024-03-06T07:27:17.413567+0000","last_became_active":"2024-03-06T07:27:17.412322+0000","last_became_peered":"2024-03-06T07:27:17.412322+0000","last_unstale":"2024-03-06T07:27:17.413567+0000","last_undegraded":"2024-03-06T07:27:17.413567+0000","last_fullsized":"2024-03-06T07:27:17.413567+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:51.540166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402352+0000","last_change":"2024-03-06T07:27:17.402352+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402352+0000","last_undegraded":"2024-03-06T07:27:17.402352+0000","last_fullsized":"2024-03-06T07:27:17.402352+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":522,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408721+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:27:17.408721+0000","last_peered":"2024-03-06T07:27:17.408721+0000","last_clean":"2024-03-06T07:27:17.408721+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:27:17.408721+0000","last_undegraded":"2024-03-06T07:27:17.408721+0000","last_fullsized":"2024-03-06T07:27:17.408721+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33800,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338040,"statfs":{"total":96636764160,"available":96602152960,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37904,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333936,"statfs":{"total":96636764160,"available":96597950464,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37960,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333880,"statfs":{"total":96636764160,"available":96597893120,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:22.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3f is not active+clean 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405718+0000', 'last_change': '2024-03-06T07:27:17.405718+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405718+0000', 'last_undegraded': '2024-03-06T07:27:17.405718+0000', 'last_fullsized': '2024-03-06T07:27:17.405718+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3e is not active+clean 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405159+0000', 'last_change': '2024-03-06T07:27:17.405159+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405159+0000', 'last_undegraded': '2024-03-06T07:27:17.405159+0000', 'last_fullsized': '2024-03-06T07:27:17.405159+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3d is not active+clean 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.410596+0000', 'last_change': '2024-03-06T07:27:17.410596+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.410596+0000', 'last_undegraded': '2024-03-06T07:27:17.410596+0000', 'last_fullsized': '2024-03-06T07:27:17.410596+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3c is not active+clean 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.403936+0000', 'last_change': '2024-03-06T07:27:17.403936+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.403936+0000', 'last_undegraded': '2024-03-06T07:27:17.403936+0000', 'last_fullsized': '2024-03-06T07:27:17.403936+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.054 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3b is not active+clean 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.406232+0000', 'last_change': '2024-03-06T07:27:17.406232+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.406232+0000', 'last_undegraded': '2024-03-06T07:27:17.406232+0000', 'last_fullsized': '2024-03-06T07:27:17.406232+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3a is not active+clean 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404786+0000', 'last_change': '2024-03-06T07:27:17.404786+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404786+0000', 'last_undegraded': '2024-03-06T07:27:17.404786+0000', 'last_fullsized': '2024-03-06T07:27:17.404786+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.39 is not active+clean 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.409357+0000', 'last_change': '2024-03-06T07:27:17.409357+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.409357+0000', 'last_undegraded': '2024-03-06T07:27:17.409357+0000', 'last_fullsized': '2024-03-06T07:27:17.409357+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.38 is not active+clean 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405499+0000', 'last_change': '2024-03-06T07:27:17.405499+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405499+0000', 'last_undegraded': '2024-03-06T07:27:17.405499+0000', 'last_fullsized': '2024-03-06T07:27:17.405499+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.37 is not active+clean 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405024+0000', 'last_change': '2024-03-06T07:27:17.405024+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405024+0000', 'last_undegraded': '2024-03-06T07:27:17.405024+0000', 'last_fullsized': '2024-03-06T07:27:17.405024+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.36 is not active+clean 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401578+0000', 'last_change': '2024-03-06T07:27:17.401578+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401578+0000', 'last_undegraded': '2024-03-06T07:27:17.401578+0000', 'last_fullsized': '2024-03-06T07:27:17.401578+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:22.055 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.35 is not active+clean 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402332+0000', 'last_change': '2024-03-06T07:27:17.402332+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402332+0000', 'last_undegraded': '2024-03-06T07:27:17.402332+0000', 'last_fullsized': '2024-03-06T07:27:17.402332+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.34 is not active+clean 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405310+0000', 'last_change': '2024-03-06T07:27:17.405310+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405310+0000', 'last_undegraded': '2024-03-06T07:27:17.405310+0000', 'last_fullsized': '2024-03-06T07:27:17.405310+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.33 is not active+clean 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.406144+0000', 'last_change': '2024-03-06T07:27:17.406144+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.406144+0000', 'last_undegraded': '2024-03-06T07:27:17.406144+0000', 'last_fullsized': '2024-03-06T07:27:17.406144+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.32 is not active+clean 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405327+0000', 'last_change': '2024-03-06T07:27:17.405327+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405327+0000', 'last_undegraded': '2024-03-06T07:27:17.405327+0000', 'last_fullsized': '2024-03-06T07:27:17.405327+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.31 is not active+clean 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402499+0000', 'last_change': '2024-03-06T07:27:17.402499+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402499+0000', 'last_undegraded': '2024-03-06T07:27:17.402499+0000', 'last_fullsized': '2024-03-06T07:27:17.402499+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.056 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.30 is not active+clean 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402943+0000', 'last_change': '2024-03-06T07:27:17.402943+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402943+0000', 'last_undegraded': '2024-03-06T07:27:17.402943+0000', 'last_fullsized': '2024-03-06T07:27:17.402943+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2f is not active+clean 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400619+0000', 'last_change': '2024-03-06T07:27:17.400619+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400619+0000', 'last_undegraded': '2024-03-06T07:27:17.400619+0000', 'last_fullsized': '2024-03-06T07:27:17.400619+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2e is not active+clean 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404639+0000', 'last_change': '2024-03-06T07:27:17.404639+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404639+0000', 'last_undegraded': '2024-03-06T07:27:17.404639+0000', 'last_fullsized': '2024-03-06T07:27:17.404639+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2d is not active+clean 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400272+0000', 'last_change': '2024-03-06T07:27:17.400272+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400272+0000', 'last_undegraded': '2024-03-06T07:27:17.400272+0000', 'last_fullsized': '2024-03-06T07:27:17.400272+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2c is not active+clean 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404933+0000', 'last_change': '2024-03-06T07:27:17.404933+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404933+0000', 'last_undegraded': '2024-03-06T07:27:17.404933+0000', 'last_fullsized': '2024-03-06T07:27:17.404933+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2b is not active+clean 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405809+0000', 'last_change': '2024-03-06T07:27:17.405809+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405809+0000', 'last_undegraded': '2024-03-06T07:27:17.405809+0000', 'last_fullsized': '2024-03-06T07:27:17.405809+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:22.057 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2a is not active+clean 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401251+0000', 'last_change': '2024-03-06T07:27:17.401251+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401251+0000', 'last_undegraded': '2024-03-06T07:27:17.401251+0000', 'last_fullsized': '2024-03-06T07:27:17.401251+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.29 is not active+clean 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402086+0000', 'last_change': '2024-03-06T07:27:17.402086+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402086+0000', 'last_undegraded': '2024-03-06T07:27:17.402086+0000', 'last_fullsized': '2024-03-06T07:27:17.402086+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.28 is not active+clean 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400623+0000', 'last_change': '2024-03-06T07:27:17.400623+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400623+0000', 'last_undegraded': '2024-03-06T07:27:17.400623+0000', 'last_fullsized': '2024-03-06T07:27:17.400623+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.27 is not active+clean 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.410762+0000', 'last_change': '2024-03-06T07:27:17.410762+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.410762+0000', 'last_undegraded': '2024-03-06T07:27:17.410762+0000', 'last_fullsized': '2024-03-06T07:27:17.410762+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.26 is not active+clean 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401619+0000', 'last_change': '2024-03-06T07:27:17.401619+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401619+0000', 'last_undegraded': '2024-03-06T07:27:17.401619+0000', 'last_fullsized': '2024-03-06T07:27:17.401619+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.25 is not active+clean 2024-03-06T07:27:22.058 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401907+0000', 'last_change': '2024-03-06T07:27:17.401907+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401907+0000', 'last_undegraded': '2024-03-06T07:27:17.401907+0000', 'last_fullsized': '2024-03-06T07:27:17.401907+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.24 is not active+clean 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402685+0000', 'last_change': '2024-03-06T07:27:17.402685+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402685+0000', 'last_undegraded': '2024-03-06T07:27:17.402685+0000', 'last_fullsized': '2024-03-06T07:27:17.402685+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.22 is not active+clean 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404113+0000', 'last_change': '2024-03-06T07:27:17.404113+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404113+0000', 'last_undegraded': '2024-03-06T07:27:17.404113+0000', 'last_fullsized': '2024-03-06T07:27:17.404113+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.21 is not active+clean 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401612+0000', 'last_change': '2024-03-06T07:27:17.401612+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401612+0000', 'last_undegraded': '2024-03-06T07:27:17.401612+0000', 'last_fullsized': '2024-03-06T07:27:17.401612+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.20 is not active+clean 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402531+0000', 'last_change': '2024-03-06T07:27:17.402531+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402531+0000', 'last_undegraded': '2024-03-06T07:27:17.402531+0000', 'last_fullsized': '2024-03-06T07:27:17.402531+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1c is not active+clean 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401801+0000', 'last_change': '2024-03-06T07:27:17.401801+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401801+0000', 'last_undegraded': '2024-03-06T07:27:17.401801+0000', 'last_fullsized': '2024-03-06T07:27:17.401801+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1d is not active+clean 2024-03-06T07:27:22.059 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.403864+0000', 'last_change': '2024-03-06T07:27:17.403864+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.403864+0000', 'last_undegraded': '2024-03-06T07:27:17.403864+0000', 'last_fullsized': '2024-03-06T07:27:17.403864+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1e is not active+clean 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404232+0000', 'last_change': '2024-03-06T07:27:17.404232+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404232+0000', 'last_undegraded': '2024-03-06T07:27:17.404232+0000', 'last_fullsized': '2024-03-06T07:27:17.404232+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1f is not active+clean 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400666+0000', 'last_change': '2024-03-06T07:27:17.400666+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400666+0000', 'last_undegraded': '2024-03-06T07:27:17.400666+0000', 'last_fullsized': '2024-03-06T07:27:17.400666+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.18 is not active+clean 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400981+0000', 'last_change': '2024-03-06T07:27:17.400981+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400981+0000', 'last_undegraded': '2024-03-06T07:27:17.400981+0000', 'last_fullsized': '2024-03-06T07:27:17.400981+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.19 is not active+clean 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400603+0000', 'last_change': '2024-03-06T07:27:17.400603+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400603+0000', 'last_undegraded': '2024-03-06T07:27:17.400603+0000', 'last_fullsized': '2024-03-06T07:27:17.400603+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1a is not active+clean 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.409868+0000', 'last_change': '2024-03-06T07:27:17.409868+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.409868+0000', 'last_undegraded': '2024-03-06T07:27:17.409868+0000', 'last_fullsized': '2024-03-06T07:27:17.409868+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:22.060 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1b is not active+clean 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401908+0000', 'last_change': '2024-03-06T07:27:17.401908+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401908+0000', 'last_undegraded': '2024-03-06T07:27:17.401908+0000', 'last_fullsized': '2024-03-06T07:27:17.401908+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.14 is not active+clean 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402610+0000', 'last_change': '2024-03-06T07:27:17.402610+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402610+0000', 'last_undegraded': '2024-03-06T07:27:17.402610+0000', 'last_fullsized': '2024-03-06T07:27:17.402610+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.15 is not active+clean 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.403419+0000', 'last_change': '2024-03-06T07:27:17.403419+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.403419+0000', 'last_undegraded': '2024-03-06T07:27:17.403419+0000', 'last_fullsized': '2024-03-06T07:27:17.403419+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.16 is not active+clean 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405727+0000', 'last_change': '2024-03-06T07:27:17.405727+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405727+0000', 'last_undegraded': '2024-03-06T07:27:17.405727+0000', 'last_fullsized': '2024-03-06T07:27:17.405727+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.17 is not active+clean 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405165+0000', 'last_change': '2024-03-06T07:27:17.405165+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405165+0000', 'last_undegraded': '2024-03-06T07:27:17.405165+0000', 'last_fullsized': '2024-03-06T07:27:17.405165+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.061 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.10 is not active+clean 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.407372+0000', 'last_change': '2024-03-06T07:27:17.407372+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.407372+0000', 'last_undegraded': '2024-03-06T07:27:17.407372+0000', 'last_fullsized': '2024-03-06T07:27:17.407372+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.11 is not active+clean 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400917+0000', 'last_change': '2024-03-06T07:27:17.400917+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400917+0000', 'last_undegraded': '2024-03-06T07:27:17.400917+0000', 'last_fullsized': '2024-03-06T07:27:17.400917+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.12 is not active+clean 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.408899+0000', 'last_change': '2024-03-06T07:27:17.408899+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.408899+0000', 'last_undegraded': '2024-03-06T07:27:17.408899+0000', 'last_fullsized': '2024-03-06T07:27:17.408899+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.13 is not active+clean 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405423+0000', 'last_change': '2024-03-06T07:27:17.405423+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405423+0000', 'last_undegraded': '2024-03-06T07:27:17.405423+0000', 'last_fullsized': '2024-03-06T07:27:17.405423+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.c is not active+clean 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400820+0000', 'last_change': '2024-03-06T07:27:17.400820+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400820+0000', 'last_undegraded': '2024-03-06T07:27:17.400820+0000', 'last_fullsized': '2024-03-06T07:27:17.400820+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.d is not active+clean 2024-03-06T07:27:22.062 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.401456+0000', 'last_change': '2024-03-06T07:27:17.401456+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.401456+0000', 'last_undegraded': '2024-03-06T07:27:17.401456+0000', 'last_fullsized': '2024-03-06T07:27:17.401456+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.e is not active+clean 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.403395+0000', 'last_change': '2024-03-06T07:27:17.403395+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.403395+0000', 'last_undegraded': '2024-03-06T07:27:17.403395+0000', 'last_fullsized': '2024-03-06T07:27:17.403395+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.f is not active+clean 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404729+0000', 'last_change': '2024-03-06T07:27:17.404729+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404729+0000', 'last_undegraded': '2024-03-06T07:27:17.404729+0000', 'last_fullsized': '2024-03-06T07:27:17.404729+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.8 is not active+clean 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400137+0000', 'last_change': '2024-03-06T07:27:17.400137+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400137+0000', 'last_undegraded': '2024-03-06T07:27:17.400137+0000', 'last_fullsized': '2024-03-06T07:27:17.400137+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.9 is not active+clean 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400097+0000', 'last_change': '2024-03-06T07:27:17.400097+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400097+0000', 'last_undegraded': '2024-03-06T07:27:17.400097+0000', 'last_fullsized': '2024-03-06T07:27:17.400097+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.a is not active+clean 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405168+0000', 'last_change': '2024-03-06T07:27:17.405168+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405168+0000', 'last_undegraded': '2024-03-06T07:27:17.405168+0000', 'last_fullsized': '2024-03-06T07:27:17.405168+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.063 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.b is not active+clean 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400532+0000', 'last_change': '2024-03-06T07:27:17.400532+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400532+0000', 'last_undegraded': '2024-03-06T07:27:17.400532+0000', 'last_fullsized': '2024-03-06T07:27:17.400532+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.4 is not active+clean 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404461+0000', 'last_change': '2024-03-06T07:27:17.404461+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404461+0000', 'last_undegraded': '2024-03-06T07:27:17.404461+0000', 'last_fullsized': '2024-03-06T07:27:17.404461+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.5 is not active+clean 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.403746+0000', 'last_change': '2024-03-06T07:27:17.403746+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.403746+0000', 'last_undegraded': '2024-03-06T07:27:17.403746+0000', 'last_fullsized': '2024-03-06T07:27:17.403746+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.6 is not active+clean 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.410538+0000', 'last_change': '2024-03-06T07:27:17.410538+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.410538+0000', 'last_undegraded': '2024-03-06T07:27:17.410538+0000', 'last_fullsized': '2024-03-06T07:27:17.410538+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 33.5 is not active+clean 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '33.5', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 207, 'state': 'creating+activating', 'last_fresh': '2024-03-06T07:27:17.409932+0000', 'last_change': '2024-03-06T07:27:17.409932+0000', 'last_active': '2024-03-06T07:27:16.385900+0000', 'last_peered': '2024-03-06T07:27:16.385900+0000', 'last_clean': '2024-03-06T07:27:16.385900+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.409932+0000', 'last_undegraded': '2024-03-06T07:27:17.409932+0000', 'last_fullsized': '2024-03-06T07:27:17.409932+0000', 'mapping_epoch': 206, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 206, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:16.385900+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:16.385900+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:16.385900+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': ['5', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.7 is not active+clean 2024-03-06T07:27:22.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.404585+0000', 'last_change': '2024-03-06T07:27:17.404585+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.404585+0000', 'last_undegraded': '2024-03-06T07:27:17.404585+0000', 'last_fullsized': '2024-03-06T07:27:17.404585+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:22.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.0 is not active+clean 2024-03-06T07:27:22.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.405097+0000', 'last_change': '2024-03-06T07:27:17.405097+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.405097+0000', 'last_undegraded': '2024-03-06T07:27:17.405097+0000', 'last_fullsized': '2024-03-06T07:27:17.405097+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.1 is not active+clean 2024-03-06T07:27:22.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400116+0000', 'last_change': '2024-03-06T07:27:17.400116+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400116+0000', 'last_undegraded': '2024-03-06T07:27:17.400116+0000', 'last_fullsized': '2024-03-06T07:27:17.400116+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.2 is not active+clean 2024-03-06T07:27:22.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400115+0000', 'last_change': '2024-03-06T07:27:17.400115+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400115+0000', 'last_undegraded': '2024-03-06T07:27:17.400115+0000', 'last_fullsized': '2024-03-06T07:27:17.400115+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:22.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.3 is not active+clean 2024-03-06T07:27:22.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.400287+0000', 'last_change': '2024-03-06T07:27:17.400287+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.400287+0000', 'last_undegraded': '2024-03-06T07:27:17.400287+0000', 'last_fullsized': '2024-03-06T07:27:17.400287+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:22.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 34.23 is not active+clean 2024-03-06T07:27:22.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '34.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 207, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:17.402352+0000', 'last_change': '2024-03-06T07:27:17.402352+0000', 'last_active': '2024-03-06T07:27:17.390880+0000', 'last_peered': '2024-03-06T07:27:17.390880+0000', 'last_clean': '2024-03-06T07:27:17.390880+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:17.402352+0000', 'last_undegraded': '2024-03-06T07:27:17.402352+0000', 'last_fullsized': '2024-03-06T07:27:17.402352+0000', 'mapping_epoch': 207, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 207, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:17.390880+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:22.068 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:27:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.187+0000 7f425ea0e640 1 Processor -- start 2024-03-06T07:27:22.189 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.187+0000 7f425ea0e640 1 -- start start 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425ea0e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4258150580 0x7f425806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425ea0e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f42580c6000 0x7f425806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.192 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425ea0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4258153a60 con 0x7f42580c1320 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425ea0e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4258153d40 con 0x7f42580c6000 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425ea0e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425806c2c0 con 0x7f4258150580 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425cfab640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f42580c6000 0x7f425806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f42577fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f42580c1320 0x7f42581566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42846/0 (socket says 172.21.15.94:42846) 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f42577fe640 1 -- 172.21.15.94:0/2075228308 learned_addr learned my addr 172.21.15.94:0/2075228308 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:22.193 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f4257fff640 1 --2- 172.21.15.94:0/2075228308 >> v2:172.21.15.145:3300/0 conn(0x7f4258150580 0x7f425806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425cfab640 1 -- 172.21.15.94:0/2075228308 >> v2:172.21.15.145:3300/0 conn(0x7f4258150580 msgr2=0x7f425806dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425cfab640 1 --2- 172.21.15.94:0/2075228308 >> v2:172.21.15.145:3300/0 conn(0x7f4258150580 0x7f425806dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425cfab640 1 -- 172.21.15.94:0/2075228308 >> v1:172.21.15.94:6789/0 conn(0x7f42580c1320 legacy=0x7f42581566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f425cfab640 1 -- 172.21.15.94:0/2075228308 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425806c5a0 con 0x7f42580c6000 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.188+0000 7f4256ffd640 1 -- 172.21.15.94:0/2075228308 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396895646 0 0) 0x7f4258153a60 con 0x7f42580c1320 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f425cfab640 1 --2- 172.21.15.94:0/2075228308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42580c6000 0x7f425806e250 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f42580a9550 tx=0x7f425002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4ccac26df787d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f4256ffd640 1 -- 172.21.15.94:0/2075228308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250039070 con 0x7f42580c6000 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f4256ffd640 1 -- 172.21.15.94:0/2075228308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425002cbf0 con 0x7f42580c6000 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f4256ffd640 1 -- 172.21.15.94:0/2075228308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250034440 con 0x7f42580c6000 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f425ea0e640 1 -- 172.21.15.94:0/2075228308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42580c6000 msgr2=0x7f425806e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.194 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f425ea0e640 1 --2- 172.21.15.94:0/2075228308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42580c6000 0x7f425806e250 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f42580a9550 tx=0x7f425002be40 comp rx=0 tx=0).stop 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f425ea0e640 1 -- 172.21.15.94:0/2075228308 shutdown_connections 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f425ea0e640 1 --2- 172.21.15.94:0/2075228308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42580c6000 0x7f425806e250 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f425ea0e640 1 --2- 172.21.15.94:0/2075228308 >> v2:172.21.15.145:3300/0 conn(0x7f4258150580 0x7f425806dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.189+0000 7f425ea0e640 1 -- 172.21.15.94:0/2075228308 >> 172.21.15.94:0/2075228308 conn(0x7f42580c6710 msgr2=0x7f42580c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.190+0000 7f425ea0e640 1 -- 172.21.15.94:0/2075228308 shutdown_connections 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.190+0000 7f425ea0e640 1 -- 172.21.15.94:0/2075228308 wait complete. 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 Processor -- start 2024-03-06T07:27:22.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 -- start start 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42580c1320 0x7f4258151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42580c6000 0x7f4258148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4258150580 0x7f42581ec510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42580c0dc0 con 0x7f4258150580 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42580c0c40 con 0x7f42580c6000 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425ea0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4258156dc0 con 0x7f42580c1320 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425cfab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4258150580 0x7f42581ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.191+0000 7f425cfab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4258150580 0x7f42581ec510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47854/0 (socket says 172.21.15.94:47854) 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425cfab640 1 -- 172.21.15.94:0/1537854151 learned_addr learned my addr 172.21.15.94:0/1537854151 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:22.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f42577fe640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42580c6000 0x7f4258148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f4257fff640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42580c1320 0x7f4258151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425cfab640 1 -- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42580c1320 msgr2=0x7f4258151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425cfab640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42580c1320 0x7f4258151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425cfab640 1 -- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42580c6000 msgr2=0x7f4258148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425cfab640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42580c6000 0x7f4258148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425cfab640 1 -- 172.21.15.94:0/1537854151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42580c2cb0 con 0x7f4258150580 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f4257fff640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42580c1320 0x7f4258151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425cfab640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4258150580 0x7f42581ec510 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f425002c450 tx=0x7f4250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f4254ff9640 1 -- 172.21.15.94:0/1537854151 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250039050 con 0x7f4258150580 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f4254ff9640 1 -- 172.21.15.94:0/1537854151 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250045070 con 0x7f4258150580 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42580c2f90 con 0x7f4258150580 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f4254ff9640 1 -- 172.21.15.94:0/1537854151 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250009450 con 0x7f4258150580 2024-03-06T07:27:22.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.192+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42581494b0 con 0x7f4258150580 2024-03-06T07:27:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.193+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425804f330 con 0x7f4258150580 2024-03-06T07:27:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.194+0000 7f4254ff9640 1 -- 172.21.15.94:0/1537854151 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f425003e850 con 0x7f4258150580 2024-03-06T07:27:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.194+0000 7f4254ff9640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424406b4d0 0x7f424406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.194+0000 7f4254ff9640 1 -- 172.21.15.94:0/1537854151 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f42500ad1f0 con 0x7f4258150580 2024-03-06T07:27:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.194+0000 7f4257fff640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424406b4d0 0x7f424406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.195+0000 7f4257fff640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424406b4d0 0x7f424406d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f42581538b0 tx=0x7f4248008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.198+0000 7f4254ff9640 1 -- 172.21.15.94:0/1537854151 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f425007c090 con 0x7f4258150580 2024-03-06T07:27:22.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.435+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f42580c3d40 con 0x7f4258150580 2024-03-06T07:27:22.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.436+0000 7f4254ff9640 1 -- 172.21.15.94:0/1537854151 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1506 (secure 0 0 0) 0x7f425007b990 con 0x7f4258150580 2024-03-06T07:27:22.437 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:22.438 INFO:teuthology.orchestra.run.smithi094.stdout:{"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":569,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":209,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709482,"num_in_osds":8,"osd_in_since":1709709474,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":32},{"state_name":"creating+activating","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":285061120,"bytes_avail":772809052160,"bytes_total":773094113280,"inactive_pgs_ratio":0.67010307312011719},"fsmap":{"epoch":196,"id":16,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":16,"rank":0,"name":"b","status":"up:active","gid":38273}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-03-06T07:26:46.748809+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424406b4d0 msgr2=0x7f424406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424406b4d0 0x7f424406d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f42581538b0 tx=0x7f4248008040 comp rx=0 tx=0).stop 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4258150580 msgr2=0x7f42581ec510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4258150580 0x7f42581ec510 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f425002c450 tx=0x7f4250004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 shutdown_connections 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f424406b4d0 0x7f424406d990 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4258150580 0x7f42581ec510 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42580c6000 0x7f4258148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 --2- 172.21.15.94:0/1537854151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42580c1320 0x7f4258151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 >> 172.21.15.94:0/1537854151 conn(0x7f42580c6710 msgr2=0x7f42580c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 shutdown_connections 2024-03-06T07:27:22.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.438+0000 7f425ea0e640 1 -- 172.21.15.94:0/1537854151 wait complete. 2024-03-06T07:27:22.451 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:27:22.451 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:22.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.548+0000 7f206e53e640 1 Processor -- start 2024-03-06T07:27:22.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.548+0000 7f206e53e640 1 -- start start 2024-03-06T07:27:22.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f206e53e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20680c1320 0x7f20681566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f206e53e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f20680c6000 0x7f2068156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f206e53e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2068153a60 con 0x7f2068150580 2024-03-06T07:27:22.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f206e53e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2068153d40 con 0x7f20680c1320 2024-03-06T07:27:22.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f206e53e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f206806c2c0 con 0x7f20680c6000 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f20677fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20680c1320 0x7f20681566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f20677fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20680c1320 0x7f20681566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47856/0 (socket says 172.21.15.94:47856) 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f20677fe640 1 -- 172.21.15.94:0/3936564015 learned_addr learned my addr 172.21.15.94:0/3936564015 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f206cadb640 1 --2- 172.21.15.94:0/3936564015 >> v2:172.21.15.145:3300/0 conn(0x7f20680c6000 0x7f2068156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f20677fe640 1 -- 172.21.15.94:0/3936564015 >> v2:172.21.15.145:3300/0 conn(0x7f20680c6000 msgr2=0x7f2068156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f20677fe640 1 --2- 172.21.15.94:0/3936564015 >> v2:172.21.15.145:3300/0 conn(0x7f20680c6000 0x7f2068156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f20677fe640 1 -- 172.21.15.94:0/3936564015 >> v1:172.21.15.145:6789/0 conn(0x7f2068150580 legacy=0x7f206806dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.549+0000 7f20677fe640 1 -- 172.21.15.94:0/3936564015 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206806c5a0 con 0x7f20680c1320 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.550+0000 7f20677fe640 1 --2- 172.21.15.94:0/3936564015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681566b0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f20680c5960 tx=0x7f205c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e530e7ea46f4999 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.550+0000 7f2066ffd640 1 -- 172.21.15.94:0/3936564015 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c035020 con 0x7f20680c1320 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.550+0000 7f2066ffd640 1 -- 172.21.15.94:0/3936564015 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c031870 con 0x7f20680c1320 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.550+0000 7f2066ffd640 1 -- 172.21.15.94:0/3936564015 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c031c40 con 0x7f20680c1320 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 -- 172.21.15.94:0/3936564015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 msgr2=0x7f20681566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 --2- 172.21.15.94:0/3936564015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681566b0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f20680c5960 tx=0x7f205c02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 -- 172.21.15.94:0/3936564015 shutdown_connections 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 --2- 172.21.15.94:0/3936564015 >> v2:172.21.15.145:3300/0 conn(0x7f20680c6000 0x7f2068156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 --2- 172.21.15.94:0/3936564015 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681566b0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 -- 172.21.15.94:0/3936564015 >> 172.21.15.94:0/3936564015 conn(0x7f20680c6710 msgr2=0x7f20680c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 -- 172.21.15.94:0/3936564015 shutdown_connections 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.551+0000 7f206e53e640 1 -- 172.21.15.94:0/3936564015 wait complete. 2024-03-06T07:27:22.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.552+0000 7f206e53e640 1 Processor -- start 2024-03-06T07:27:22.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.552+0000 7f206e53e640 1 -- start start 2024-03-06T07:27:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206e53e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681e9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206e53e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20680c6000 0x7f2068155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206e53e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2068150580 0x7f20680725d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206e53e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206806e5a0 con 0x7f20680c1320 2024-03-06T07:27:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206e53e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206806e420 con 0x7f20680c6000 2024-03-06T07:27:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206e53e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20680c0c40 con 0x7f2068150580 2024-03-06T07:27:22.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681e9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681e9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47870/0 (socket says 172.21.15.94:47870) 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 -- 172.21.15.94:0/4232912868 learned_addr learned my addr 172.21.15.94:0/4232912868 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f20677fe640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20680c6000 0x7f2068155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206cadb640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2068150580 0x7f20680725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 -- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2068150580 msgr2=0x7f20680725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2068150580 0x7f20680725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 -- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20680c6000 msgr2=0x7f2068155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20680c6000 0x7f2068155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 -- 172.21.15.94:0/4232912868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206814bd50 con 0x7f20680c1320 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2067fff640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681e9a70 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f205800c3c0 tx=0x7f205800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2064ff9640 1 -- 172.21.15.94:0/4232912868 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20580052d0 con 0x7f20680c1320 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2064ff9640 1 -- 172.21.15.94:0/4232912868 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2058005490 con 0x7f20680c1320 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f2064ff9640 1 -- 172.21.15.94:0/4232912868 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058007260 con 0x7f20680c1320 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.553+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2068152bb0 con 0x7f20680c1320 2024-03-06T07:27:22.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.554+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2068153170 con 0x7f20680c1320 2024-03-06T07:27:22.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.555+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206804f330 con 0x7f20680c1320 2024-03-06T07:27:22.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.555+0000 7f2064ff9640 1 -- 172.21.15.94:0/4232912868 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2058007400 con 0x7f20680c1320 2024-03-06T07:27:22.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.555+0000 7f2064ff9640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f205406b500 0x7f205406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.555+0000 7f2064ff9640 1 -- 172.21.15.94:0/4232912868 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f20580896a0 con 0x7f20680c1320 2024-03-06T07:27:22.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.555+0000 7f20677fe640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f205406b500 0x7f205406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.556+0000 7f20677fe640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f205406b500 0x7f205406d9c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f205c036000 tx=0x7f205c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.559+0000 7f2064ff9640 1 -- 172.21.15.94:0/4232912868 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2058057dd0 con 0x7f20680c1320 2024-03-06T07:27:22.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.727+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20680c5de0 con 0x7f205406b500 2024-03-06T07:27:22.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.739+0000 7f2064ff9640 1 -- 172.21.15.94:0/4232912868 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247072 (secure 0 0 0) 0x7f205c084530 con 0x7f205406b500 2024-03-06T07:27:22.743 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:22.744 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f205406b500 msgr2=0x7f205406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f205406b500 0x7f205406d9c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f205c036000 tx=0x7f205c03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 msgr2=0x7f20681e9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681e9a70 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f205800c3c0 tx=0x7f205800c880 comp rx=0 tx=0).stop 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 shutdown_connections 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f205406b500 0x7f205406d9c0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2068150580 0x7f20680725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20680c6000 0x7f2068155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 --2- 172.21.15.94:0/4232912868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20680c1320 0x7f20681e9a70 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.741+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 >> 172.21.15.94:0/4232912868 conn(0x7f20680c6710 msgr2=0x7f20680a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.742+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 shutdown_connections 2024-03-06T07:27:22.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:22.742+0000 7f206e53e640 1 -- 172.21.15.94:0/4232912868 wait complete. 2024-03-06T07:27:22.754 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-03-06T07:27:20.757508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278380,"kb_used_data":20576,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754696340,"statfs":{"total":773094113280,"available":772809052160,"internally_reserved":0,"allocated":21069824,"data_stored":11304944,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.366341"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405718+0000","last_change":"2024-03-06T07:27:17.405718+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405718+0000","last_undegraded":"2024-03-06T07:27:17.405718+0000","last_fullsized":"2024-03-06T07:27:17.405718+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405159+0000","last_change":"2024-03-06T07:27:17.405159+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405159+0000","last_undegraded":"2024-03-06T07:27:17.405159+0000","last_fullsized":"2024-03-06T07:27:17.405159+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410596+0000","last_change":"2024-03-06T07:27:17.410596+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410596+0000","last_undegraded":"2024-03-06T07:27:17.410596+0000","last_fullsized":"2024-03-06T07:27:17.410596+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403936+0000","last_change":"2024-03-06T07:27:17.403936+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403936+0000","last_undegraded":"2024-03-06T07:27:17.403936+0000","last_fullsized":"2024-03-06T07:27:17.403936+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406232+0000","last_change":"2024-03-06T07:27:17.406232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406232+0000","last_undegraded":"2024-03-06T07:27:17.406232+0000","last_fullsized":"2024-03-06T07:27:17.406232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404786+0000","last_change":"2024-03-06T07:27:17.404786+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404786+0000","last_undegraded":"2024-03-06T07:27:17.404786+0000","last_fullsized":"2024-03-06T07:27:17.404786+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409357+0000","last_change":"2024-03-06T07:27:17.409357+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409357+0000","last_undegraded":"2024-03-06T07:27:17.409357+0000","last_fullsized":"2024-03-06T07:27:17.409357+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405499+0000","last_change":"2024-03-06T07:27:17.405499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405499+0000","last_undegraded":"2024-03-06T07:27:17.405499+0000","last_fullsized":"2024-03-06T07:27:17.405499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405024+0000","last_change":"2024-03-06T07:27:17.405024+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405024+0000","last_undegraded":"2024-03-06T07:27:17.405024+0000","last_fullsized":"2024-03-06T07:27:17.405024+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401578+0000","last_change":"2024-03-06T07:27:17.401578+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401578+0000","last_undegraded":"2024-03-06T07:27:17.401578+0000","last_fullsized":"2024-03-06T07:27:17.401578+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402332+0000","last_change":"2024-03-06T07:27:17.402332+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402332+0000","last_undegraded":"2024-03-06T07:27:17.402332+0000","last_fullsized":"2024-03-06T07:27:17.402332+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405310+0000","last_change":"2024-03-06T07:27:17.405310+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405310+0000","last_undegraded":"2024-03-06T07:27:17.405310+0000","last_fullsized":"2024-03-06T07:27:17.405310+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.406144+0000","last_change":"2024-03-06T07:27:17.406144+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.406144+0000","last_undegraded":"2024-03-06T07:27:17.406144+0000","last_fullsized":"2024-03-06T07:27:17.406144+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405327+0000","last_change":"2024-03-06T07:27:17.405327+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405327+0000","last_undegraded":"2024-03-06T07:27:17.405327+0000","last_fullsized":"2024-03-06T07:27:17.405327+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402499+0000","last_change":"2024-03-06T07:27:17.402499+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402499+0000","last_undegraded":"2024-03-06T07:27:17.402499+0000","last_fullsized":"2024-03-06T07:27:17.402499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402943+0000","last_change":"2024-03-06T07:27:17.402943+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402943+0000","last_undegraded":"2024-03-06T07:27:17.402943+0000","last_fullsized":"2024-03-06T07:27:17.402943+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400619+0000","last_change":"2024-03-06T07:27:17.400619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400619+0000","last_undegraded":"2024-03-06T07:27:17.400619+0000","last_fullsized":"2024-03-06T07:27:17.400619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404639+0000","last_change":"2024-03-06T07:27:17.404639+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404639+0000","last_undegraded":"2024-03-06T07:27:17.404639+0000","last_fullsized":"2024-03-06T07:27:17.404639+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400272+0000","last_change":"2024-03-06T07:27:17.400272+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400272+0000","last_undegraded":"2024-03-06T07:27:17.400272+0000","last_fullsized":"2024-03-06T07:27:17.400272+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404933+0000","last_change":"2024-03-06T07:27:17.404933+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404933+0000","last_undegraded":"2024-03-06T07:27:17.404933+0000","last_fullsized":"2024-03-06T07:27:17.404933+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405809+0000","last_change":"2024-03-06T07:27:17.405809+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405809+0000","last_undegraded":"2024-03-06T07:27:17.405809+0000","last_fullsized":"2024-03-06T07:27:17.405809+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401251+0000","last_change":"2024-03-06T07:27:17.401251+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401251+0000","last_undegraded":"2024-03-06T07:27:17.401251+0000","last_fullsized":"2024-03-06T07:27:17.401251+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402086+0000","last_change":"2024-03-06T07:27:17.402086+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402086+0000","last_undegraded":"2024-03-06T07:27:17.402086+0000","last_fullsized":"2024-03-06T07:27:17.402086+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400623+0000","last_change":"2024-03-06T07:27:17.400623+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400623+0000","last_undegraded":"2024-03-06T07:27:17.400623+0000","last_fullsized":"2024-03-06T07:27:17.400623+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410762+0000","last_change":"2024-03-06T07:27:17.410762+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410762+0000","last_undegraded":"2024-03-06T07:27:17.410762+0000","last_fullsized":"2024-03-06T07:27:17.410762+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401619+0000","last_change":"2024-03-06T07:27:17.401619+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401619+0000","last_undegraded":"2024-03-06T07:27:17.401619+0000","last_fullsized":"2024-03-06T07:27:17.401619+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401907+0000","last_change":"2024-03-06T07:27:17.401907+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401907+0000","last_undegraded":"2024-03-06T07:27:17.401907+0000","last_fullsized":"2024-03-06T07:27:17.401907+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402685+0000","last_change":"2024-03-06T07:27:17.402685+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402685+0000","last_undegraded":"2024-03-06T07:27:17.402685+0000","last_fullsized":"2024-03-06T07:27:17.402685+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404113+0000","last_change":"2024-03-06T07:27:17.404113+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404113+0000","last_undegraded":"2024-03-06T07:27:17.404113+0000","last_fullsized":"2024-03-06T07:27:17.404113+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401612+0000","last_change":"2024-03-06T07:27:17.401612+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401612+0000","last_undegraded":"2024-03-06T07:27:17.401612+0000","last_fullsized":"2024-03-06T07:27:17.401612+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402531+0000","last_change":"2024-03-06T07:27:17.402531+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402531+0000","last_undegraded":"2024-03-06T07:27:17.402531+0000","last_fullsized":"2024-03-06T07:27:17.402531+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401801+0000","last_change":"2024-03-06T07:27:17.401801+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401801+0000","last_undegraded":"2024-03-06T07:27:17.401801+0000","last_fullsized":"2024-03-06T07:27:17.401801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413284+0000","last_change":"2024-03-06T07:27:17.410327+0000","last_active":"2024-03-06T07:27:17.413284+0000","last_peered":"2024-03-06T07:27:17.413284+0000","last_clean":"2024-03-06T07:27:17.413284+0000","last_became_active":"2024-03-06T07:27:17.409826+0000","last_became_peered":"2024-03-06T07:27:17.409826+0000","last_unstale":"2024-03-06T07:27:17.413284+0000","last_undegraded":"2024-03-06T07:27:17.413284+0000","last_fullsized":"2024-03-06T07:27:17.413284+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:31.182027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403864+0000","last_change":"2024-03-06T07:27:17.403864+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403864+0000","last_undegraded":"2024-03-06T07:27:17.403864+0000","last_fullsized":"2024-03-06T07:27:17.403864+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414725+0000","last_change":"2024-03-06T07:27:17.413513+0000","last_active":"2024-03-06T07:27:17.414725+0000","last_peered":"2024-03-06T07:27:17.414725+0000","last_clean":"2024-03-06T07:27:17.414725+0000","last_became_active":"2024-03-06T07:27:17.413247+0000","last_became_peered":"2024-03-06T07:27:17.413247+0000","last_unstale":"2024-03-06T07:27:17.414725+0000","last_undegraded":"2024-03-06T07:27:17.414725+0000","last_fullsized":"2024-03-06T07:27:17.414725+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404232+0000","last_change":"2024-03-06T07:27:17.404232+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404232+0000","last_undegraded":"2024-03-06T07:27:17.404232+0000","last_fullsized":"2024-03-06T07:27:17.404232+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418162+0000","last_change":"2024-03-06T07:27:17.415933+0000","last_active":"2024-03-06T07:27:17.418162+0000","last_peered":"2024-03-06T07:27:17.418162+0000","last_clean":"2024-03-06T07:27:17.418162+0000","last_became_active":"2024-03-06T07:27:17.415652+0000","last_became_peered":"2024-03-06T07:27:17.415652+0000","last_unstale":"2024-03-06T07:27:17.418162+0000","last_undegraded":"2024-03-06T07:27:17.418162+0000","last_fullsized":"2024-03-06T07:27:17.418162+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:05.886687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400666+0000","last_change":"2024-03-06T07:27:17.400666+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400666+0000","last_undegraded":"2024-03-06T07:27:17.400666+0000","last_fullsized":"2024-03-06T07:27:17.400666+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413520+0000","last_change":"2024-03-06T07:27:17.412515+0000","last_active":"2024-03-06T07:27:17.413520+0000","last_peered":"2024-03-06T07:27:17.413520+0000","last_clean":"2024-03-06T07:27:17.413520+0000","last_became_active":"2024-03-06T07:27:17.412246+0000","last_became_peered":"2024-03-06T07:27:17.412246+0000","last_unstale":"2024-03-06T07:27:17.413520+0000","last_undegraded":"2024-03-06T07:27:17.413520+0000","last_fullsized":"2024-03-06T07:27:17.413520+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400981+0000","last_change":"2024-03-06T07:27:17.400981+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400981+0000","last_undegraded":"2024-03-06T07:27:17.400981+0000","last_fullsized":"2024-03-06T07:27:17.400981+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.418993+0000","last_change":"2024-03-06T07:27:17.418064+0000","last_active":"2024-03-06T07:27:17.418993+0000","last_peered":"2024-03-06T07:27:17.418993+0000","last_clean":"2024-03-06T07:27:17.418993+0000","last_became_active":"2024-03-06T07:27:17.417764+0000","last_became_peered":"2024-03-06T07:27:17.417764+0000","last_unstale":"2024-03-06T07:27:17.418993+0000","last_undegraded":"2024-03-06T07:27:17.418993+0000","last_fullsized":"2024-03-06T07:27:17.418993+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:30.997230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400603+0000","last_change":"2024-03-06T07:27:17.400603+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400603+0000","last_undegraded":"2024-03-06T07:27:17.400603+0000","last_fullsized":"2024-03-06T07:27:17.400603+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416413+0000","last_change":"2024-03-06T07:27:17.415618+0000","last_active":"2024-03-06T07:27:17.416413+0000","last_peered":"2024-03-06T07:27:17.416413+0000","last_clean":"2024-03-06T07:27:17.416413+0000","last_became_active":"2024-03-06T07:27:17.415113+0000","last_became_peered":"2024-03-06T07:27:17.415113+0000","last_unstale":"2024-03-06T07:27:17.416413+0000","last_undegraded":"2024-03-06T07:27:17.416413+0000","last_fullsized":"2024-03-06T07:27:17.416413+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.409868+0000","last_change":"2024-03-06T07:27:17.409868+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409868+0000","last_undegraded":"2024-03-06T07:27:17.409868+0000","last_fullsized":"2024-03-06T07:27:17.409868+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412010+0000","last_change":"2024-03-06T07:27:17.410657+0000","last_active":"2024-03-06T07:27:17.412010+0000","last_peered":"2024-03-06T07:27:17.412010+0000","last_clean":"2024-03-06T07:27:17.412010+0000","last_became_active":"2024-03-06T07:27:17.410330+0000","last_became_peered":"2024-03-06T07:27:17.410330+0000","last_unstale":"2024-03-06T07:27:17.412010+0000","last_undegraded":"2024-03-06T07:27:17.412010+0000","last_fullsized":"2024-03-06T07:27:17.412010+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:32.740313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401908+0000","last_change":"2024-03-06T07:27:17.401908+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401908+0000","last_undegraded":"2024-03-06T07:27:17.401908+0000","last_fullsized":"2024-03-06T07:27:17.401908+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412932+0000","last_change":"2024-03-06T07:27:17.412182+0000","last_active":"2024-03-06T07:27:17.412932+0000","last_peered":"2024-03-06T07:27:17.412932+0000","last_clean":"2024-03-06T07:27:17.412932+0000","last_became_active":"2024-03-06T07:27:17.411934+0000","last_became_peered":"2024-03-06T07:27:17.411934+0000","last_unstale":"2024-03-06T07:27:17.412932+0000","last_undegraded":"2024-03-06T07:27:17.412932+0000","last_fullsized":"2024-03-06T07:27:17.412932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:18.741598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402610+0000","last_change":"2024-03-06T07:27:17.402610+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402610+0000","last_undegraded":"2024-03-06T07:27:17.402610+0000","last_fullsized":"2024-03-06T07:27:17.402610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415272+0000","last_change":"2024-03-06T07:27:17.412868+0000","last_active":"2024-03-06T07:27:17.415272+0000","last_peered":"2024-03-06T07:27:17.415272+0000","last_clean":"2024-03-06T07:27:17.415272+0000","last_became_active":"2024-03-06T07:27:17.412587+0000","last_became_peered":"2024-03-06T07:27:17.412587+0000","last_unstale":"2024-03-06T07:27:17.415272+0000","last_undegraded":"2024-03-06T07:27:17.415272+0000","last_fullsized":"2024-03-06T07:27:17.415272+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:36.738070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403419+0000","last_change":"2024-03-06T07:27:17.403419+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403419+0000","last_undegraded":"2024-03-06T07:27:17.403419+0000","last_fullsized":"2024-03-06T07:27:17.403419+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411419+0000","last_change":"2024-03-06T07:27:17.410408+0000","last_active":"2024-03-06T07:27:17.411419+0000","last_peered":"2024-03-06T07:27:17.411419+0000","last_clean":"2024-03-06T07:27:17.411419+0000","last_became_active":"2024-03-06T07:27:17.410121+0000","last_became_peered":"2024-03-06T07:27:17.410121+0000","last_unstale":"2024-03-06T07:27:17.411419+0000","last_undegraded":"2024-03-06T07:27:17.411419+0000","last_fullsized":"2024-03-06T07:27:17.411419+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:21.045449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405727+0000","last_change":"2024-03-06T07:27:17.405727+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405727+0000","last_undegraded":"2024-03-06T07:27:17.405727+0000","last_fullsized":"2024-03-06T07:27:17.405727+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416587+0000","last_change":"2024-03-06T07:27:17.414830+0000","last_active":"2024-03-06T07:27:17.416587+0000","last_peered":"2024-03-06T07:27:17.416587+0000","last_clean":"2024-03-06T07:27:17.416587+0000","last_became_active":"2024-03-06T07:27:17.414541+0000","last_became_peered":"2024-03-06T07:27:17.414541+0000","last_unstale":"2024-03-06T07:27:17.416587+0000","last_undegraded":"2024-03-06T07:27:17.416587+0000","last_fullsized":"2024-03-06T07:27:17.416587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:46.438918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405165+0000","last_change":"2024-03-06T07:27:17.405165+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405165+0000","last_undegraded":"2024-03-06T07:27:17.405165+0000","last_fullsized":"2024-03-06T07:27:17.405165+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.414323+0000","last_change":"2024-03-06T07:27:17.413318+0000","last_active":"2024-03-06T07:27:17.414323+0000","last_peered":"2024-03-06T07:27:17.414323+0000","last_clean":"2024-03-06T07:27:17.414323+0000","last_became_active":"2024-03-06T07:27:17.413042+0000","last_became_peered":"2024-03-06T07:27:17.413042+0000","last_unstale":"2024-03-06T07:27:17.414323+0000","last_undegraded":"2024-03-06T07:27:17.414323+0000","last_fullsized":"2024-03-06T07:27:17.414323+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:57.629596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.407372+0000","last_change":"2024-03-06T07:27:17.407372+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.407372+0000","last_undegraded":"2024-03-06T07:27:17.407372+0000","last_fullsized":"2024-03-06T07:27:17.407372+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.412696+0000","last_change":"2024-03-06T07:27:17.412118+0000","last_active":"2024-03-06T07:27:17.412696+0000","last_peered":"2024-03-06T07:27:17.412696+0000","last_clean":"2024-03-06T07:27:17.412696+0000","last_became_active":"2024-03-06T07:27:17.411828+0000","last_became_peered":"2024-03-06T07:27:17.411828+0000","last_unstale":"2024-03-06T07:27:17.412696+0000","last_undegraded":"2024-03-06T07:27:17.412696+0000","last_fullsized":"2024-03-06T07:27:17.412696+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:07.702608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400917+0000","last_change":"2024-03-06T07:27:17.400917+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400917+0000","last_undegraded":"2024-03-06T07:27:17.400917+0000","last_fullsized":"2024-03-06T07:27:17.400917+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419663+0000","last_change":"2024-03-06T07:27:17.416024+0000","last_active":"2024-03-06T07:27:17.419663+0000","last_peered":"2024-03-06T07:27:17.419663+0000","last_clean":"2024-03-06T07:27:17.419663+0000","last_became_active":"2024-03-06T07:27:17.415738+0000","last_became_peered":"2024-03-06T07:27:17.415738+0000","last_unstale":"2024-03-06T07:27:17.419663+0000","last_undegraded":"2024-03-06T07:27:17.419663+0000","last_fullsized":"2024-03-06T07:27:17.419663+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:26.216018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.408899+0000","last_change":"2024-03-06T07:27:17.408899+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.408899+0000","last_undegraded":"2024-03-06T07:27:17.408899+0000","last_fullsized":"2024-03-06T07:27:17.408899+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.419824+0000","last_change":"2024-03-06T07:27:17.416751+0000","last_active":"2024-03-06T07:27:17.419824+0000","last_peered":"2024-03-06T07:27:17.419824+0000","last_clean":"2024-03-06T07:27:17.419824+0000","last_became_active":"2024-03-06T07:27:17.416475+0000","last_became_peered":"2024-03-06T07:27:17.416475+0000","last_unstale":"2024-03-06T07:27:17.419824+0000","last_undegraded":"2024-03-06T07:27:17.419824+0000","last_fullsized":"2024-03-06T07:27:17.419824+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:34.635712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405423+0000","last_change":"2024-03-06T07:27:17.405423+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405423+0000","last_undegraded":"2024-03-06T07:27:17.405423+0000","last_fullsized":"2024-03-06T07:27:17.405423+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415326+0000","last_change":"2024-03-06T07:27:17.408458+0000","last_active":"2024-03-06T07:27:17.415326+0000","last_peered":"2024-03-06T07:27:17.415326+0000","last_clean":"2024-03-06T07:27:17.415326+0000","last_became_active":"2024-03-06T07:27:17.408204+0000","last_became_peered":"2024-03-06T07:27:17.408204+0000","last_unstale":"2024-03-06T07:27:17.415326+0000","last_undegraded":"2024-03-06T07:27:17.415326+0000","last_fullsized":"2024-03-06T07:27:17.415326+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:31.044701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400820+0000","last_change":"2024-03-06T07:27:17.400820+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400820+0000","last_undegraded":"2024-03-06T07:27:17.400820+0000","last_fullsized":"2024-03-06T07:27:17.400820+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.423682+0000","last_change":"2024-03-06T07:27:17.418167+0000","last_active":"2024-03-06T07:27:17.423682+0000","last_peered":"2024-03-06T07:27:17.423682+0000","last_clean":"2024-03-06T07:27:17.423682+0000","last_became_active":"2024-03-06T07:27:17.417877+0000","last_became_peered":"2024-03-06T07:27:17.417877+0000","last_unstale":"2024-03-06T07:27:17.423682+0000","last_undegraded":"2024-03-06T07:27:17.423682+0000","last_fullsized":"2024-03-06T07:27:17.423682+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:47.136248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.401456+0000","last_change":"2024-03-06T07:27:17.401456+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.401456+0000","last_undegraded":"2024-03-06T07:27:17.401456+0000","last_fullsized":"2024-03-06T07:27:17.401456+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413865+0000","last_change":"2024-03-06T07:27:17.409593+0000","last_active":"2024-03-06T07:27:17.413865+0000","last_peered":"2024-03-06T07:27:17.413865+0000","last_clean":"2024-03-06T07:27:17.413865+0000","last_became_active":"2024-03-06T07:27:17.409303+0000","last_became_peered":"2024-03-06T07:27:17.409303+0000","last_unstale":"2024-03-06T07:27:17.413865+0000","last_undegraded":"2024-03-06T07:27:17.413865+0000","last_fullsized":"2024-03-06T07:27:17.413865+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:58.310257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403395+0000","last_change":"2024-03-06T07:27:17.403395+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403395+0000","last_undegraded":"2024-03-06T07:27:17.403395+0000","last_fullsized":"2024-03-06T07:27:17.403395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416748+0000","last_change":"2024-03-06T07:27:17.416748+0000","last_active":"2024-03-06T07:27:17.416748+0000","last_peered":"2024-03-06T07:27:17.416748+0000","last_clean":"2024-03-06T07:27:17.416748+0000","last_became_active":"2024-03-06T07:27:17.416473+0000","last_became_peered":"2024-03-06T07:27:17.416473+0000","last_unstale":"2024-03-06T07:27:17.416748+0000","last_undegraded":"2024-03-06T07:27:17.416748+0000","last_fullsized":"2024-03-06T07:27:17.416748+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:32.904492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404729+0000","last_change":"2024-03-06T07:27:17.404729+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404729+0000","last_undegraded":"2024-03-06T07:27:17.404729+0000","last_fullsized":"2024-03-06T07:27:17.404729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.411708+0000","last_change":"2024-03-06T07:27:17.409661+0000","last_active":"2024-03-06T07:27:17.411708+0000","last_peered":"2024-03-06T07:27:17.411708+0000","last_clean":"2024-03-06T07:27:17.411708+0000","last_became_active":"2024-03-06T07:27:17.409383+0000","last_became_peered":"2024-03-06T07:27:17.409383+0000","last_unstale":"2024-03-06T07:27:17.411708+0000","last_undegraded":"2024-03-06T07:27:17.411708+0000","last_fullsized":"2024-03-06T07:27:17.411708+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:09.299098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400137+0000","last_change":"2024-03-06T07:27:17.400137+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400137+0000","last_undegraded":"2024-03-06T07:27:17.400137+0000","last_fullsized":"2024-03-06T07:27:17.400137+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.420704+0000","last_change":"2024-03-06T07:27:17.420352+0000","last_active":"2024-03-06T07:27:17.420704+0000","last_peered":"2024-03-06T07:27:17.420704+0000","last_clean":"2024-03-06T07:27:17.420704+0000","last_became_active":"2024-03-06T07:27:17.420070+0000","last_became_peered":"2024-03-06T07:27:17.420070+0000","last_unstale":"2024-03-06T07:27:17.420704+0000","last_undegraded":"2024-03-06T07:27:17.420704+0000","last_fullsized":"2024-03-06T07:27:17.420704+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:19.650711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400097+0000","last_change":"2024-03-06T07:27:17.400097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400097+0000","last_undegraded":"2024-03-06T07:27:17.400097+0000","last_fullsized":"2024-03-06T07:27:17.400097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417219+0000","last_change":"2024-03-06T07:27:17.414539+0000","last_active":"2024-03-06T07:27:17.417219+0000","last_peered":"2024-03-06T07:27:17.417219+0000","last_clean":"2024-03-06T07:27:17.417219+0000","last_became_active":"2024-03-06T07:27:17.414268+0000","last_became_peered":"2024-03-06T07:27:17.414268+0000","last_unstale":"2024-03-06T07:27:17.417219+0000","last_undegraded":"2024-03-06T07:27:17.417219+0000","last_fullsized":"2024-03-06T07:27:17.417219+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:52.205799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405168+0000","last_change":"2024-03-06T07:27:17.405168+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405168+0000","last_undegraded":"2024-03-06T07:27:17.405168+0000","last_fullsized":"2024-03-06T07:27:17.405168+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.410930+0000","last_change":"2024-03-06T07:27:17.408812+0000","last_active":"2024-03-06T07:27:17.410930+0000","last_peered":"2024-03-06T07:27:17.410930+0000","last_clean":"2024-03-06T07:27:17.410930+0000","last_became_active":"2024-03-06T07:27:17.408558+0000","last_became_peered":"2024-03-06T07:27:17.408558+0000","last_unstale":"2024-03-06T07:27:17.410930+0000","last_undegraded":"2024-03-06T07:27:17.410930+0000","last_fullsized":"2024-03-06T07:27:17.410930+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:00.868016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400532+0000","last_change":"2024-03-06T07:27:17.400532+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400532+0000","last_undegraded":"2024-03-06T07:27:17.400532+0000","last_fullsized":"2024-03-06T07:27:17.400532+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415346+0000","last_change":"2024-03-06T07:27:17.408777+0000","last_active":"2024-03-06T07:27:17.415346+0000","last_peered":"2024-03-06T07:27:17.415346+0000","last_clean":"2024-03-06T07:27:17.415346+0000","last_became_active":"2024-03-06T07:27:17.408201+0000","last_became_peered":"2024-03-06T07:27:17.408201+0000","last_unstale":"2024-03-06T07:27:17.415346+0000","last_undegraded":"2024-03-06T07:27:17.415346+0000","last_fullsized":"2024-03-06T07:27:17.415346+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:01:42.980380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404461+0000","last_change":"2024-03-06T07:27:17.404461+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404461+0000","last_undegraded":"2024-03-06T07:27:17.404461+0000","last_fullsized":"2024-03-06T07:27:17.404461+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415292+0000","last_change":"2024-03-06T07:27:17.413559+0000","last_active":"2024-03-06T07:27:17.415292+0000","last_peered":"2024-03-06T07:27:17.415292+0000","last_clean":"2024-03-06T07:27:17.415292+0000","last_became_active":"2024-03-06T07:27:17.413298+0000","last_became_peered":"2024-03-06T07:27:17.413298+0000","last_unstale":"2024-03-06T07:27:17.415292+0000","last_undegraded":"2024-03-06T07:27:17.415292+0000","last_fullsized":"2024-03-06T07:27:17.415292+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:30.857109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.403746+0000","last_change":"2024-03-06T07:27:17.403746+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.403746+0000","last_undegraded":"2024-03-06T07:27:17.403746+0000","last_fullsized":"2024-03-06T07:27:17.403746+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416237+0000","last_change":"2024-03-06T07:27:17.415152+0000","last_active":"2024-03-06T07:27:17.416237+0000","last_peered":"2024-03-06T07:27:17.416237+0000","last_clean":"2024-03-06T07:27:17.416237+0000","last_became_active":"2024-03-06T07:27:17.414894+0000","last_became_peered":"2024-03-06T07:27:17.414894+0000","last_unstale":"2024-03-06T07:27:17.416237+0000","last_undegraded":"2024-03-06T07:27:17.416237+0000","last_fullsized":"2024-03-06T07:27:17.416237+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:05.667791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.410538+0000","last_change":"2024-03-06T07:27:17.410538+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.410538+0000","last_undegraded":"2024-03-06T07:27:17.410538+0000","last_fullsized":"2024-03-06T07:27:17.410538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":3,"reported_epoch":207,"state":"creating+activating","last_fresh":"2024-03-06T07:27:17.409932+0000","last_change":"2024-03-06T07:27:17.409932+0000","last_active":"2024-03-06T07:27:16.385900+0000","last_peered":"2024-03-06T07:27:16.385900+0000","last_clean":"2024-03-06T07:27:16.385900+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.409932+0000","last_undegraded":"2024-03-06T07:27:17.409932+0000","last_fullsized":"2024-03-06T07:27:17.409932+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.404585+0000","last_change":"2024-03-06T07:27:17.404585+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.404585+0000","last_undegraded":"2024-03-06T07:27:17.404585+0000","last_fullsized":"2024-03-06T07:27:17.404585+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":8,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.416845+0000","last_change":"2024-03-06T07:27:17.416845+0000","last_active":"2024-03-06T07:27:17.416845+0000","last_peered":"2024-03-06T07:27:17.416845+0000","last_clean":"2024-03-06T07:27:17.416845+0000","last_became_active":"2024-03-06T07:27:17.416518+0000","last_became_peered":"2024-03-06T07:27:17.416518+0000","last_unstale":"2024-03-06T07:27:17.416845+0000","last_undegraded":"2024-03-06T07:27:17.416845+0000","last_fullsized":"2024-03-06T07:27:17.416845+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:09.916243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.405097+0000","last_change":"2024-03-06T07:27:17.405097+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.405097+0000","last_undegraded":"2024-03-06T07:27:17.405097+0000","last_fullsized":"2024-03-06T07:27:17.405097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408436+0000","last_change":"2024-03-06T07:27:17.407570+0000","last_active":"2024-03-06T07:27:17.408436+0000","last_peered":"2024-03-06T07:27:17.408436+0000","last_clean":"2024-03-06T07:27:17.408436+0000","last_became_active":"2024-03-06T07:27:17.407285+0000","last_became_peered":"2024-03-06T07:27:17.407285+0000","last_unstale":"2024-03-06T07:27:17.408436+0000","last_undegraded":"2024-03-06T07:27:17.408436+0000","last_fullsized":"2024-03-06T07:27:17.408436+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:00.071014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400116+0000","last_change":"2024-03-06T07:27:17.400116+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400116+0000","last_undegraded":"2024-03-06T07:27:17.400116+0000","last_fullsized":"2024-03-06T07:27:17.400116+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.417608+0000","last_change":"2024-03-06T07:27:17.415917+0000","last_active":"2024-03-06T07:27:17.417608+0000","last_peered":"2024-03-06T07:27:17.417608+0000","last_clean":"2024-03-06T07:27:17.417608+0000","last_became_active":"2024-03-06T07:27:17.414080+0000","last_became_peered":"2024-03-06T07:27:17.414080+0000","last_unstale":"2024-03-06T07:27:17.417608+0000","last_undegraded":"2024-03-06T07:27:17.417608+0000","last_fullsized":"2024-03-06T07:27:17.417608+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:20.177610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400115+0000","last_change":"2024-03-06T07:27:17.400115+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400115+0000","last_undegraded":"2024-03-06T07:27:17.400115+0000","last_fullsized":"2024-03-06T07:27:17.400115+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.415761+0000","last_change":"2024-03-06T07:27:17.415251+0000","last_active":"2024-03-06T07:27:17.415761+0000","last_peered":"2024-03-06T07:27:17.415761+0000","last_clean":"2024-03-06T07:27:17.415761+0000","last_became_active":"2024-03-06T07:27:17.414993+0000","last_became_peered":"2024-03-06T07:27:17.414993+0000","last_unstale":"2024-03-06T07:27:17.415761+0000","last_undegraded":"2024-03-06T07:27:17.415761+0000","last_fullsized":"2024-03-06T07:27:17.415761+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:20.440673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.400287+0000","last_change":"2024-03-06T07:27:17.400287+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.400287+0000","last_undegraded":"2024-03-06T07:27:17.400287+0000","last_fullsized":"2024-03-06T07:27:17.400287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.413567+0000","last_change":"2024-03-06T07:27:17.412587+0000","last_active":"2024-03-06T07:27:17.413567+0000","last_peered":"2024-03-06T07:27:17.413567+0000","last_clean":"2024-03-06T07:27:17.413567+0000","last_became_active":"2024-03-06T07:27:17.412322+0000","last_became_peered":"2024-03-06T07:27:17.412322+0000","last_unstale":"2024-03-06T07:27:17.413567+0000","last_undegraded":"2024-03-06T07:27:17.413567+0000","last_fullsized":"2024-03-06T07:27:17.413567+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:51.540166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":1,"reported_epoch":207,"state":"creating+peering","last_fresh":"2024-03-06T07:27:17.402352+0000","last_change":"2024-03-06T07:27:17.402352+0000","last_active":"2024-03-06T07:27:17.390880+0000","last_peered":"2024-03-06T07:27:17.390880+0000","last_clean":"2024-03-06T07:27:17.390880+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:17.402352+0000","last_undegraded":"2024-03-06T07:27:17.402352+0000","last_fullsized":"2024-03-06T07:27:17.402352+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":522,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-03-06T07:27:17.408721+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:27:17.408721+0000","last_peered":"2024-03-06T07:27:17.408721+0000","last_clean":"2024-03-06T07:27:17.408721+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:27:17.408721+0000","last_undegraded":"2024-03-06T07:27:17.408721+0000","last_fullsized":"2024-03-06T07:27:17.408721+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574961,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33800,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338040,"statfs":{"total":96636764160,"available":96602152960,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33748,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338092,"statfs":{"total":96636764160,"available":96602206208,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37904,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333936,"statfs":{"total":96636764160,"available":96597950464,"internally_reserved":0,"allocated":2617344,"data_stored":1265526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37960,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333880,"statfs":{"total":96636764160,"available":96597893120,"internally_reserved":0,"allocated":2682880,"data_stored":1855894,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:23.669 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-03-06T07:27:23.667+0000 7fd26e6aa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:25.759 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:25.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.870+0000 7ff7de888640 1 Processor -- start 2024-03-06T07:27:25.872 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.870+0000 7ff7de888640 1 -- start start 2024-03-06T07:27:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7de888640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7d80c1aa0 0x7ff7d81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7de888640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff7d8155a40 0x7ff7d8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7de888640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d80c28d0 con 0x7ff7d814f790 2024-03-06T07:27:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7de888640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d80c2680 con 0x7ff7d80c1aa0 2024-03-06T07:27:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7de888640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d80c2bb0 con 0x7ff7d8155a40 2024-03-06T07:27:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7d80c1aa0 0x7ff7d81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.873 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff7d80c1aa0 0x7ff7d81529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47884/0 (socket says 172.21.15.94:47884) 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d7fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff7d814f790 0x7ff7d80c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42850/0 (socket says 172.21.15.94:42850) 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d77fe640 1 -- 172.21.15.94:0/3567326955 learned_addr learned my addr 172.21.15.94:0/3567326955 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7dce25640 1 --2- 172.21.15.94:0/3567326955 >> v2:172.21.15.145:3300/0 conn(0x7ff7d8155a40 0x7ff7d8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d77fe640 1 -- 172.21.15.94:0/3567326955 >> v2:172.21.15.145:3300/0 conn(0x7ff7d8155a40 msgr2=0x7ff7d8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d77fe640 1 --2- 172.21.15.94:0/3567326955 >> v2:172.21.15.145:3300/0 conn(0x7ff7d8155a40 0x7ff7d8155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d77fe640 1 -- 172.21.15.94:0/3567326955 >> v1:172.21.15.94:6789/0 conn(0x7ff7d814f790 legacy=0x7ff7d80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d77fe640 1 -- 172.21.15.94:0/3567326955 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d80c2e90 con 0x7ff7d80c1aa0 2024-03-06T07:27:25.874 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.871+0000 7ff7d6ffd640 1 -- 172.21.15.94:0/3567326955 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593654306 0 0) 0x7ff7d80c28d0 con 0x7ff7d814f790 2024-03-06T07:27:25.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.872+0000 7ff7d77fe640 1 --2- 172.21.15.94:0/3567326955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d80c1aa0 0x7ff7d81529d0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7ff7d806c960 tx=0x7ff7c002dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b43997a9915c32e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.878 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.872+0000 7ff7d6ffd640 1 -- 172.21.15.94:0/3567326955 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c003b070 con 0x7ff7d80c1aa0 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.872+0000 7ff7d6ffd640 1 -- 172.21.15.94:0/3567326955 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c002e960 con 0x7ff7d80c1aa0 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.872+0000 7ff7d6ffd640 1 -- 172.21.15.94:0/3567326955 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c002ed30 con 0x7ff7d80c1aa0 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 -- 172.21.15.94:0/3567326955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d80c1aa0 msgr2=0x7ff7d81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 --2- 172.21.15.94:0/3567326955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d80c1aa0 0x7ff7d81529d0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7ff7d806c960 tx=0x7ff7c002dbb0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 -- 172.21.15.94:0/3567326955 shutdown_connections 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 --2- 172.21.15.94:0/3567326955 >> v2:172.21.15.145:3300/0 conn(0x7ff7d8155a40 0x7ff7d8155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 --2- 172.21.15.94:0/3567326955 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d80c1aa0 0x7ff7d81529d0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 -- 172.21.15.94:0/3567326955 >> 172.21.15.94:0/3567326955 conn(0x7ff7d80c6710 msgr2=0x7ff7d80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 -- 172.21.15.94:0/3567326955 shutdown_connections 2024-03-06T07:27:25.879 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.873+0000 7ff7de888640 1 -- 172.21.15.94:0/3567326955 wait complete. 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.874+0000 7ff7de888640 1 Processor -- start 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.874+0000 7ff7de888640 1 -- start start 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7d80c1aa0 0x7ff7d81e7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d814f790 0x7ff7d8151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7d8155a40 0x7ff7d8149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80c0d60 con 0x7ff7d814f790 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d8156390 con 0x7ff7d80c1aa0 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80c5e30 con 0x7ff7d8155a40 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d814f790 0x7ff7d8151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d814f790 0x7ff7d8151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47886/0 (socket says 172.21.15.94:47886) 2024-03-06T07:27:25.880 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 -- 172.21.15.94:0/4082100343 learned_addr learned my addr 172.21.15.94:0/4082100343 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d7fff640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7d80c1aa0 0x7ff7d81e7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7dce25640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7d8155a40 0x7ff7d8149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 -- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7d8155a40 msgr2=0x7ff7d8149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7d8155a40 0x7ff7d8149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 -- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7d80c1aa0 msgr2=0x7ff7d81e7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7d80c1aa0 0x7ff7d81e7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 -- 172.21.15.94:0/4082100343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d814a1d0 con 0x7ff7d814f790 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d77fe640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d814f790 0x7ff7d8151780 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7ff7c002dbf0 tx=0x7ff7c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d4ff9640 1 -- 172.21.15.94:0/4082100343 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c003b040 con 0x7ff7d814f790 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d4ff9640 1 -- 172.21.15.94:0/4082100343 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c0046070 con 0x7ff7d814f790 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7d4ff9640 1 -- 172.21.15.94:0/4082100343 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0036d90 con 0x7ff7d814f790 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d814a4b0 con 0x7ff7d814f790 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.875+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d8151cc0 con 0x7ff7d814f790 2024-03-06T07:27:25.881 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.877+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d804f330 con 0x7ff7d814f790 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.877+0000 7ff7d4ff9640 1 -- 172.21.15.94:0/4082100343 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7c004b460 con 0x7ff7d814f790 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.877+0000 7ff7d4ff9640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7b806b4d0 0x7ff7b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.877+0000 7ff7d4ff9640 1 -- 172.21.15.94:0/4082100343 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff7c00ae190 con 0x7ff7d814f790 2024-03-06T07:27:25.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.877+0000 7ff7d7fff640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7b806b4d0 0x7ff7b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.878+0000 7ff7d7fff640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7b806b4d0 0x7ff7b806d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8000f60 tx=0x7ff7c8007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:25.881+0000 7ff7d4ff9640 1 -- 172.21.15.94:0/4082100343 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7c007c8c0 con 0x7ff7d814f790 2024-03-06T07:27:26.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.049+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7d8149f80 con 0x7ff7b806b4d0 2024-03-06T07:27:26.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.055+0000 7ff7d4ff9640 1 -- 172.21.15.94:0/4082100343 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252573 (secure 0 0 0) 0x7ff7d8149f80 con 0x7ff7b806b4d0 2024-03-06T07:27:26.059 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:26.061 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7b806b4d0 msgr2=0x7ff7b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7b806b4d0 0x7ff7b806d990 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8000f60 tx=0x7ff7c8007d50 comp rx=0 tx=0).stop 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d814f790 msgr2=0x7ff7d8151780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d814f790 0x7ff7d8151780 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7ff7c002dbf0 tx=0x7ff7c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 shutdown_connections 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff7b806b4d0 0x7ff7b806d990 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff7d8155a40 0x7ff7d8149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff7d814f790 0x7ff7d8151780 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 --2- 172.21.15.94:0/4082100343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7d80c1aa0 0x7ff7d81e7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 >> 172.21.15.94:0/4082100343 conn(0x7ff7d80c6710 msgr2=0x7ff7d80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 shutdown_connections 2024-03-06T07:27:26.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.058+0000 7ff7de888640 1 -- 172.21.15.94:0/4082100343 wait complete. 2024-03-06T07:27:26.071 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-03-06T07:27:24.758251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":282896,"kb_used_data":20928,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754691824,"statfs":{"total":773094113280,"available":772804427776,"internally_reserved":0,"allocated":21430272,"data_stored":11473972,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.360092"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407437+0000","last_change":"2024-03-06T07:27:18.422020+0000","last_active":"2024-03-06T07:27:19.407437+0000","last_peered":"2024-03-06T07:27:19.407437+0000","last_clean":"2024-03-06T07:27:19.407437+0000","last_became_active":"2024-03-06T07:27:18.421770+0000","last_became_peered":"2024-03-06T07:27:18.421770+0000","last_unstale":"2024-03-06T07:27:19.407437+0000","last_undegraded":"2024-03-06T07:27:19.407437+0000","last_fullsized":"2024-03-06T07:27:19.407437+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:33.909472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663342+0000","last_change":"2024-03-06T07:27:18.414492+0000","last_active":"2024-03-06T07:27:19.663342+0000","last_peered":"2024-03-06T07:27:19.663342+0000","last_clean":"2024-03-06T07:27:19.663342+0000","last_became_active":"2024-03-06T07:27:18.414233+0000","last_became_peered":"2024-03-06T07:27:18.414233+0000","last_unstale":"2024-03-06T07:27:19.663342+0000","last_undegraded":"2024-03-06T07:27:19.663342+0000","last_fullsized":"2024-03-06T07:27:19.663342+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:48.192088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662802+0000","last_change":"2024-03-06T07:27:18.422642+0000","last_active":"2024-03-06T07:27:19.662802+0000","last_peered":"2024-03-06T07:27:19.662802+0000","last_clean":"2024-03-06T07:27:19.662802+0000","last_became_active":"2024-03-06T07:27:18.422391+0000","last_became_peered":"2024-03-06T07:27:18.422391+0000","last_unstale":"2024-03-06T07:27:19.662802+0000","last_undegraded":"2024-03-06T07:27:19.662802+0000","last_fullsized":"2024-03-06T07:27:19.662802+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:11.450774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400832+0000","last_change":"2024-03-06T07:27:18.404321+0000","last_active":"2024-03-06T07:27:19.400832+0000","last_peered":"2024-03-06T07:27:19.400832+0000","last_clean":"2024-03-06T07:27:19.400832+0000","last_became_active":"2024-03-06T07:27:18.404076+0000","last_became_peered":"2024-03-06T07:27:18.404076+0000","last_unstale":"2024-03-06T07:27:19.400832+0000","last_undegraded":"2024-03-06T07:27:19.400832+0000","last_fullsized":"2024-03-06T07:27:19.400832+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:40.280690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662658+0000","last_change":"2024-03-06T07:27:18.409896+0000","last_active":"2024-03-06T07:27:19.662658+0000","last_peered":"2024-03-06T07:27:19.662658+0000","last_clean":"2024-03-06T07:27:19.662658+0000","last_became_active":"2024-03-06T07:27:18.409656+0000","last_became_peered":"2024-03-06T07:27:18.409656+0000","last_unstale":"2024-03-06T07:27:19.662658+0000","last_undegraded":"2024-03-06T07:27:19.662658+0000","last_fullsized":"2024-03-06T07:27:19.662658+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:34.600776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662177+0000","last_change":"2024-03-06T07:27:18.416326+0000","last_active":"2024-03-06T07:27:19.662177+0000","last_peered":"2024-03-06T07:27:19.662177+0000","last_clean":"2024-03-06T07:27:19.662177+0000","last_became_active":"2024-03-06T07:27:18.416052+0000","last_became_peered":"2024-03-06T07:27:18.416052+0000","last_unstale":"2024-03-06T07:27:19.662177+0000","last_undegraded":"2024-03-06T07:27:19.662177+0000","last_fullsized":"2024-03-06T07:27:19.662177+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:18.251390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662141+0000","last_change":"2024-03-06T07:27:18.425155+0000","last_active":"2024-03-06T07:27:19.662141+0000","last_peered":"2024-03-06T07:27:19.662141+0000","last_clean":"2024-03-06T07:27:19.662141+0000","last_became_active":"2024-03-06T07:27:18.424864+0000","last_became_peered":"2024-03-06T07:27:18.424864+0000","last_unstale":"2024-03-06T07:27:19.662141+0000","last_undegraded":"2024-03-06T07:27:19.662141+0000","last_fullsized":"2024-03-06T07:27:19.662141+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:51.585483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400492+0000","last_change":"2024-03-06T07:27:18.417091+0000","last_active":"2024-03-06T07:27:19.400492+0000","last_peered":"2024-03-06T07:27:19.400492+0000","last_clean":"2024-03-06T07:27:19.400492+0000","last_became_active":"2024-03-06T07:27:18.416838+0000","last_became_peered":"2024-03-06T07:27:18.416838+0000","last_unstale":"2024-03-06T07:27:19.400492+0000","last_undegraded":"2024-03-06T07:27:19.400492+0000","last_fullsized":"2024-03-06T07:27:19.400492+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:09.133353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662525+0000","last_change":"2024-03-06T07:27:18.411544+0000","last_active":"2024-03-06T07:27:19.662525+0000","last_peered":"2024-03-06T07:27:19.662525+0000","last_clean":"2024-03-06T07:27:19.662525+0000","last_became_active":"2024-03-06T07:27:18.411244+0000","last_became_peered":"2024-03-06T07:27:18.411244+0000","last_unstale":"2024-03-06T07:27:19.662525+0000","last_undegraded":"2024-03-06T07:27:19.662525+0000","last_fullsized":"2024-03-06T07:27:19.662525+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:13.159834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406376+0000","last_change":"2024-03-06T07:27:18.415168+0000","last_active":"2024-03-06T07:27:19.406376+0000","last_peered":"2024-03-06T07:27:19.406376+0000","last_clean":"2024-03-06T07:27:19.406376+0000","last_became_active":"2024-03-06T07:27:18.414464+0000","last_became_peered":"2024-03-06T07:27:18.414464+0000","last_unstale":"2024-03-06T07:27:19.406376+0000","last_undegraded":"2024-03-06T07:27:19.406376+0000","last_fullsized":"2024-03-06T07:27:19.406376+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:04.426004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407415+0000","last_change":"2024-03-06T07:27:18.414260+0000","last_active":"2024-03-06T07:27:19.407415+0000","last_peered":"2024-03-06T07:27:19.407415+0000","last_clean":"2024-03-06T07:27:19.407415+0000","last_became_active":"2024-03-06T07:27:18.412910+0000","last_became_peered":"2024-03-06T07:27:18.412910+0000","last_unstale":"2024-03-06T07:27:19.407415+0000","last_undegraded":"2024-03-06T07:27:19.407415+0000","last_fullsized":"2024-03-06T07:27:19.407415+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:21.445652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662537+0000","last_change":"2024-03-06T07:27:18.416748+0000","last_active":"2024-03-06T07:27:19.662537+0000","last_peered":"2024-03-06T07:27:19.662537+0000","last_clean":"2024-03-06T07:27:19.662537+0000","last_became_active":"2024-03-06T07:27:18.415979+0000","last_became_peered":"2024-03-06T07:27:18.415979+0000","last_unstale":"2024-03-06T07:27:19.662537+0000","last_undegraded":"2024-03-06T07:27:19.662537+0000","last_fullsized":"2024-03-06T07:27:19.662537+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:17.876078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400866+0000","last_change":"2024-03-06T07:27:18.424227+0000","last_active":"2024-03-06T07:27:19.400866+0000","last_peered":"2024-03-06T07:27:19.400866+0000","last_clean":"2024-03-06T07:27:19.400866+0000","last_became_active":"2024-03-06T07:27:18.423923+0000","last_became_peered":"2024-03-06T07:27:18.423923+0000","last_unstale":"2024-03-06T07:27:19.400866+0000","last_undegraded":"2024-03-06T07:27:19.400866+0000","last_fullsized":"2024-03-06T07:27:19.400866+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:48.608535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663460+0000","last_change":"2024-03-06T07:27:18.413502+0000","last_active":"2024-03-06T07:27:19.663460+0000","last_peered":"2024-03-06T07:27:19.663460+0000","last_clean":"2024-03-06T07:27:19.663460+0000","last_became_active":"2024-03-06T07:27:18.413262+0000","last_became_peered":"2024-03-06T07:27:18.413262+0000","last_unstale":"2024-03-06T07:27:19.663460+0000","last_undegraded":"2024-03-06T07:27:19.663460+0000","last_fullsized":"2024-03-06T07:27:19.663460+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:33.814313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662337+0000","last_change":"2024-03-06T07:27:18.418177+0000","last_active":"2024-03-06T07:27:19.662337+0000","last_peered":"2024-03-06T07:27:19.662337+0000","last_clean":"2024-03-06T07:27:19.662337+0000","last_became_active":"2024-03-06T07:27:18.417920+0000","last_became_peered":"2024-03-06T07:27:18.417920+0000","last_unstale":"2024-03-06T07:27:19.662337+0000","last_undegraded":"2024-03-06T07:27:19.662337+0000","last_fullsized":"2024-03-06T07:27:19.662337+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.214607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406550+0000","last_change":"2024-03-06T07:27:18.420686+0000","last_active":"2024-03-06T07:27:19.406550+0000","last_peered":"2024-03-06T07:27:19.406550+0000","last_clean":"2024-03-06T07:27:19.406550+0000","last_became_active":"2024-03-06T07:27:18.420402+0000","last_became_peered":"2024-03-06T07:27:18.420402+0000","last_unstale":"2024-03-06T07:27:19.406550+0000","last_undegraded":"2024-03-06T07:27:19.406550+0000","last_fullsized":"2024-03-06T07:27:19.406550+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:11.450774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406421+0000","last_change":"2024-03-06T07:27:18.420513+0000","last_active":"2024-03-06T07:27:19.406421+0000","last_peered":"2024-03-06T07:27:19.406421+0000","last_clean":"2024-03-06T07:27:19.406421+0000","last_became_active":"2024-03-06T07:27:18.420191+0000","last_became_peered":"2024-03-06T07:27:18.420191+0000","last_unstale":"2024-03-06T07:27:19.406421+0000","last_undegraded":"2024-03-06T07:27:19.406421+0000","last_fullsized":"2024-03-06T07:27:19.406421+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:51.585483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662452+0000","last_change":"2024-03-06T07:27:18.411459+0000","last_active":"2024-03-06T07:27:19.662452+0000","last_peered":"2024-03-06T07:27:19.662452+0000","last_clean":"2024-03-06T07:27:19.662452+0000","last_became_active":"2024-03-06T07:27:18.411212+0000","last_became_peered":"2024-03-06T07:27:18.411212+0000","last_unstale":"2024-03-06T07:27:19.662452+0000","last_undegraded":"2024-03-06T07:27:19.662452+0000","last_fullsized":"2024-03-06T07:27:19.662452+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:52.947819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406364+0000","last_change":"2024-03-06T07:27:18.417958+0000","last_active":"2024-03-06T07:27:19.406364+0000","last_peered":"2024-03-06T07:27:19.406364+0000","last_clean":"2024-03-06T07:27:19.406364+0000","last_became_active":"2024-03-06T07:27:18.415395+0000","last_became_peered":"2024-03-06T07:27:18.415395+0000","last_unstale":"2024-03-06T07:27:19.406364+0000","last_undegraded":"2024-03-06T07:27:19.406364+0000","last_fullsized":"2024-03-06T07:27:19.406364+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:48.141227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662612+0000","last_change":"2024-03-06T07:27:18.417080+0000","last_active":"2024-03-06T07:27:19.662612+0000","last_peered":"2024-03-06T07:27:19.662612+0000","last_clean":"2024-03-06T07:27:19.662612+0000","last_became_active":"2024-03-06T07:27:18.416818+0000","last_became_peered":"2024-03-06T07:27:18.416818+0000","last_unstale":"2024-03-06T07:27:19.662612+0000","last_undegraded":"2024-03-06T07:27:19.662612+0000","last_fullsized":"2024-03-06T07:27:19.662612+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.230471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662809+0000","last_change":"2024-03-06T07:27:18.409321+0000","last_active":"2024-03-06T07:27:19.662809+0000","last_peered":"2024-03-06T07:27:19.662809+0000","last_clean":"2024-03-06T07:27:19.662809+0000","last_became_active":"2024-03-06T07:27:18.409082+0000","last_became_peered":"2024-03-06T07:27:18.409082+0000","last_unstale":"2024-03-06T07:27:19.662809+0000","last_undegraded":"2024-03-06T07:27:19.662809+0000","last_fullsized":"2024-03-06T07:27:19.662809+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:29.887929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407215+0000","last_change":"2024-03-06T07:27:18.415003+0000","last_active":"2024-03-06T07:27:19.407215+0000","last_peered":"2024-03-06T07:27:19.407215+0000","last_clean":"2024-03-06T07:27:19.407215+0000","last_became_active":"2024-03-06T07:27:18.414732+0000","last_became_peered":"2024-03-06T07:27:18.414732+0000","last_unstale":"2024-03-06T07:27:19.407215+0000","last_undegraded":"2024-03-06T07:27:19.407215+0000","last_fullsized":"2024-03-06T07:27:19.407215+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:04.332939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662794+0000","last_change":"2024-03-06T07:27:18.412351+0000","last_active":"2024-03-06T07:27:19.662794+0000","last_peered":"2024-03-06T07:27:19.662794+0000","last_clean":"2024-03-06T07:27:19.662794+0000","last_became_active":"2024-03-06T07:27:18.410313+0000","last_became_peered":"2024-03-06T07:27:18.410313+0000","last_unstale":"2024-03-06T07:27:19.662794+0000","last_undegraded":"2024-03-06T07:27:19.662794+0000","last_fullsized":"2024-03-06T07:27:19.662794+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:08.707588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407102+0000","last_change":"2024-03-06T07:27:18.415153+0000","last_active":"2024-03-06T07:27:19.407102+0000","last_peered":"2024-03-06T07:27:19.407102+0000","last_clean":"2024-03-06T07:27:19.407102+0000","last_became_active":"2024-03-06T07:27:18.414893+0000","last_became_peered":"2024-03-06T07:27:18.414893+0000","last_unstale":"2024-03-06T07:27:19.407102+0000","last_undegraded":"2024-03-06T07:27:19.407102+0000","last_fullsized":"2024-03-06T07:27:19.407102+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:59.428866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662603+0000","last_change":"2024-03-06T07:27:18.421453+0000","last_active":"2024-03-06T07:27:19.662603+0000","last_peered":"2024-03-06T07:27:19.662603+0000","last_clean":"2024-03-06T07:27:19.662603+0000","last_became_active":"2024-03-06T07:27:18.421127+0000","last_became_peered":"2024-03-06T07:27:18.421127+0000","last_unstale":"2024-03-06T07:27:19.662603+0000","last_undegraded":"2024-03-06T07:27:19.662603+0000","last_fullsized":"2024-03-06T07:27:19.662603+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:21.357053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663389+0000","last_change":"2024-03-06T07:27:18.419296+0000","last_active":"2024-03-06T07:27:19.663389+0000","last_peered":"2024-03-06T07:27:19.663389+0000","last_clean":"2024-03-06T07:27:19.663389+0000","last_became_active":"2024-03-06T07:27:18.414676+0000","last_became_peered":"2024-03-06T07:27:18.414676+0000","last_unstale":"2024-03-06T07:27:19.663389+0000","last_undegraded":"2024-03-06T07:27:19.663389+0000","last_fullsized":"2024-03-06T07:27:19.663389+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:46.452961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407443+0000","last_change":"2024-03-06T07:27:18.422837+0000","last_active":"2024-03-06T07:27:19.407443+0000","last_peered":"2024-03-06T07:27:19.407443+0000","last_clean":"2024-03-06T07:27:19.407443+0000","last_became_active":"2024-03-06T07:27:18.422546+0000","last_became_peered":"2024-03-06T07:27:18.422546+0000","last_unstale":"2024-03-06T07:27:19.407443+0000","last_undegraded":"2024-03-06T07:27:19.407443+0000","last_fullsized":"2024-03-06T07:27:19.407443+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:10.921222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407275+0000","last_change":"2024-03-06T07:27:18.409403+0000","last_active":"2024-03-06T07:27:19.407275+0000","last_peered":"2024-03-06T07:27:19.407275+0000","last_clean":"2024-03-06T07:27:19.407275+0000","last_became_active":"2024-03-06T07:27:18.409121+0000","last_became_peered":"2024-03-06T07:27:18.409121+0000","last_unstale":"2024-03-06T07:27:19.407275+0000","last_undegraded":"2024-03-06T07:27:19.407275+0000","last_fullsized":"2024-03-06T07:27:19.407275+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:22.050428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662342+0000","last_change":"2024-03-06T07:27:18.407775+0000","last_active":"2024-03-06T07:27:19.662342+0000","last_peered":"2024-03-06T07:27:19.662342+0000","last_clean":"2024-03-06T07:27:19.662342+0000","last_became_active":"2024-03-06T07:27:18.407498+0000","last_became_peered":"2024-03-06T07:27:18.407498+0000","last_unstale":"2024-03-06T07:27:19.662342+0000","last_undegraded":"2024-03-06T07:27:19.662342+0000","last_fullsized":"2024-03-06T07:27:19.662342+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:48.969850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662934+0000","last_change":"2024-03-06T07:27:18.413080+0000","last_active":"2024-03-06T07:27:19.662934+0000","last_peered":"2024-03-06T07:27:19.662934+0000","last_clean":"2024-03-06T07:27:19.662934+0000","last_became_active":"2024-03-06T07:27:18.412681+0000","last_became_peered":"2024-03-06T07:27:18.412681+0000","last_unstale":"2024-03-06T07:27:19.662934+0000","last_undegraded":"2024-03-06T07:27:19.662934+0000","last_fullsized":"2024-03-06T07:27:19.662934+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:19.746578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662287+0000","last_change":"2024-03-06T07:27:18.414827+0000","last_active":"2024-03-06T07:27:19.662287+0000","last_peered":"2024-03-06T07:27:19.662287+0000","last_clean":"2024-03-06T07:27:19.662287+0000","last_became_active":"2024-03-06T07:27:18.414127+0000","last_became_peered":"2024-03-06T07:27:18.414127+0000","last_unstale":"2024-03-06T07:27:19.662287+0000","last_undegraded":"2024-03-06T07:27:19.662287+0000","last_fullsized":"2024-03-06T07:27:19.662287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:34.230471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406480+0000","last_change":"2024-03-06T07:27:18.414632+0000","last_active":"2024-03-06T07:27:19.406480+0000","last_peered":"2024-03-06T07:27:19.406480+0000","last_clean":"2024-03-06T07:27:19.406480+0000","last_became_active":"2024-03-06T07:27:18.414262+0000","last_became_peered":"2024-03-06T07:27:18.414262+0000","last_unstale":"2024-03-06T07:27:19.406480+0000","last_undegraded":"2024-03-06T07:27:19.406480+0000","last_fullsized":"2024-03-06T07:27:19.406480+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:21.357053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"208'4","reported_seq":18,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663047+0000","last_change":"2024-03-06T07:27:17.410327+0000","last_active":"2024-03-06T07:27:19.663047+0000","last_peered":"2024-03-06T07:27:19.663047+0000","last_clean":"2024-03-06T07:27:19.663047+0000","last_became_active":"2024-03-06T07:27:17.409826+0000","last_became_peered":"2024-03-06T07:27:17.409826+0000","last_unstale":"2024-03-06T07:27:19.663047+0000","last_undegraded":"2024-03-06T07:27:19.663047+0000","last_fullsized":"2024-03-06T07:27:19.663047+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:37.328607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662652+0000","last_change":"2024-03-06T07:27:18.415922+0000","last_active":"2024-03-06T07:27:19.662652+0000","last_peered":"2024-03-06T07:27:19.662652+0000","last_clean":"2024-03-06T07:27:19.662652+0000","last_became_active":"2024-03-06T07:27:18.415342+0000","last_became_peered":"2024-03-06T07:27:18.415342+0000","last_unstale":"2024-03-06T07:27:19.662652+0000","last_undegraded":"2024-03-06T07:27:19.662652+0000","last_fullsized":"2024-03-06T07:27:19.662652+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:08.547009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"208'1","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406512+0000","last_change":"2024-03-06T07:27:17.413513+0000","last_active":"2024-03-06T07:27:19.406512+0000","last_peered":"2024-03-06T07:27:19.406512+0000","last_clean":"2024-03-06T07:27:19.406512+0000","last_became_active":"2024-03-06T07:27:17.413247+0000","last_became_peered":"2024-03-06T07:27:17.413247+0000","last_unstale":"2024-03-06T07:27:19.406512+0000","last_undegraded":"2024-03-06T07:27:19.406512+0000","last_fullsized":"2024-03-06T07:27:19.406512+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:17.295013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662579+0000","last_change":"2024-03-06T07:27:18.411096+0000","last_active":"2024-03-06T07:27:19.662579+0000","last_peered":"2024-03-06T07:27:19.662579+0000","last_clean":"2024-03-06T07:27:19.662579+0000","last_became_active":"2024-03-06T07:27:18.410815+0000","last_became_peered":"2024-03-06T07:27:18.410815+0000","last_unstale":"2024-03-06T07:27:19.662579+0000","last_undegraded":"2024-03-06T07:27:19.662579+0000","last_fullsized":"2024-03-06T07:27:19.662579+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:23:46.922265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"208'2","reported_seq":16,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407591+0000","last_change":"2024-03-06T07:27:17.415933+0000","last_active":"2024-03-06T07:27:19.407591+0000","last_peered":"2024-03-06T07:27:19.407591+0000","last_clean":"2024-03-06T07:27:19.407591+0000","last_became_active":"2024-03-06T07:27:17.415652+0000","last_became_peered":"2024-03-06T07:27:17.415652+0000","last_unstale":"2024-03-06T07:27:19.407591+0000","last_undegraded":"2024-03-06T07:27:19.407591+0000","last_fullsized":"2024-03-06T07:27:19.407591+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:32.685496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663660+0000","last_change":"2024-03-06T07:27:18.418937+0000","last_active":"2024-03-06T07:27:19.663660+0000","last_peered":"2024-03-06T07:27:19.663660+0000","last_clean":"2024-03-06T07:27:19.663660+0000","last_became_active":"2024-03-06T07:27:18.418653+0000","last_became_peered":"2024-03-06T07:27:18.418653+0000","last_unstale":"2024-03-06T07:27:19.663660+0000","last_undegraded":"2024-03-06T07:27:19.663660+0000","last_fullsized":"2024-03-06T07:27:19.663660+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:17.203309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"208'2","reported_seq":16,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406382+0000","last_change":"2024-03-06T07:27:17.412515+0000","last_active":"2024-03-06T07:27:19.406382+0000","last_peered":"2024-03-06T07:27:19.406382+0000","last_clean":"2024-03-06T07:27:19.406382+0000","last_became_active":"2024-03-06T07:27:17.412246+0000","last_became_peered":"2024-03-06T07:27:17.412246+0000","last_unstale":"2024-03-06T07:27:19.406382+0000","last_undegraded":"2024-03-06T07:27:19.406382+0000","last_fullsized":"2024-03-06T07:27:19.406382+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:41.738094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662431+0000","last_change":"2024-03-06T07:27:18.413343+0000","last_active":"2024-03-06T07:27:19.662431+0000","last_peered":"2024-03-06T07:27:19.662431+0000","last_clean":"2024-03-06T07:27:19.662431+0000","last_became_active":"2024-03-06T07:27:18.413070+0000","last_became_peered":"2024-03-06T07:27:18.413070+0000","last_unstale":"2024-03-06T07:27:19.662431+0000","last_undegraded":"2024-03-06T07:27:19.662431+0000","last_fullsized":"2024-03-06T07:27:19.662431+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:54.394404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"208'1","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663163+0000","last_change":"2024-03-06T07:27:17.418064+0000","last_active":"2024-03-06T07:27:19.663163+0000","last_peered":"2024-03-06T07:27:19.663163+0000","last_clean":"2024-03-06T07:27:19.663163+0000","last_became_active":"2024-03-06T07:27:17.417764+0000","last_became_peered":"2024-03-06T07:27:17.417764+0000","last_unstale":"2024-03-06T07:27:19.663163+0000","last_undegraded":"2024-03-06T07:27:19.663163+0000","last_fullsized":"2024-03-06T07:27:19.663163+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:16.198329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663068+0000","last_change":"2024-03-06T07:27:18.420023+0000","last_active":"2024-03-06T07:27:19.663068+0000","last_peered":"2024-03-06T07:27:19.663068+0000","last_clean":"2024-03-06T07:27:19.663068+0000","last_became_active":"2024-03-06T07:27:18.419749+0000","last_became_peered":"2024-03-06T07:27:18.419749+0000","last_unstale":"2024-03-06T07:27:19.663068+0000","last_undegraded":"2024-03-06T07:27:19.663068+0000","last_fullsized":"2024-03-06T07:27:19.663068+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:32.002209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"208'1","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663582+0000","last_change":"2024-03-06T07:27:17.415618+0000","last_active":"2024-03-06T07:27:19.663582+0000","last_peered":"2024-03-06T07:27:19.663582+0000","last_clean":"2024-03-06T07:27:19.663582+0000","last_became_active":"2024-03-06T07:27:17.415113+0000","last_became_peered":"2024-03-06T07:27:17.415113+0000","last_unstale":"2024-03-06T07:27:19.663582+0000","last_undegraded":"2024-03-06T07:27:19.663582+0000","last_fullsized":"2024-03-06T07:27:19.663582+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:50.580503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662929+0000","last_change":"2024-03-06T07:27:18.422188+0000","last_active":"2024-03-06T07:27:19.662929+0000","last_peered":"2024-03-06T07:27:19.662929+0000","last_clean":"2024-03-06T07:27:19.662929+0000","last_became_active":"2024-03-06T07:27:18.421933+0000","last_became_peered":"2024-03-06T07:27:18.421933+0000","last_unstale":"2024-03-06T07:27:19.662929+0000","last_undegraded":"2024-03-06T07:27:19.662929+0000","last_fullsized":"2024-03-06T07:27:19.662929+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:38.257993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406034+0000","last_change":"2024-03-06T07:27:17.410657+0000","last_active":"2024-03-06T07:27:19.406034+0000","last_peered":"2024-03-06T07:27:19.406034+0000","last_clean":"2024-03-06T07:27:19.406034+0000","last_became_active":"2024-03-06T07:27:17.410330+0000","last_became_peered":"2024-03-06T07:27:17.410330+0000","last_unstale":"2024-03-06T07:27:19.406034+0000","last_undegraded":"2024-03-06T07:27:19.406034+0000","last_fullsized":"2024-03-06T07:27:19.406034+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:07.878881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400998+0000","last_change":"2024-03-06T07:27:18.418061+0000","last_active":"2024-03-06T07:27:19.400998+0000","last_peered":"2024-03-06T07:27:19.400998+0000","last_clean":"2024-03-06T07:27:19.400998+0000","last_became_active":"2024-03-06T07:27:18.417548+0000","last_became_peered":"2024-03-06T07:27:18.417548+0000","last_unstale":"2024-03-06T07:27:19.400998+0000","last_undegraded":"2024-03-06T07:27:19.400998+0000","last_fullsized":"2024-03-06T07:27:19.400998+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:15.203014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662188+0000","last_change":"2024-03-06T07:27:17.412182+0000","last_active":"2024-03-06T07:27:19.662188+0000","last_peered":"2024-03-06T07:27:19.662188+0000","last_clean":"2024-03-06T07:27:19.662188+0000","last_became_active":"2024-03-06T07:27:17.411934+0000","last_became_peered":"2024-03-06T07:27:17.411934+0000","last_unstale":"2024-03-06T07:27:19.662188+0000","last_undegraded":"2024-03-06T07:27:19.662188+0000","last_fullsized":"2024-03-06T07:27:19.662188+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:45.447981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400953+0000","last_change":"2024-03-06T07:27:18.416179+0000","last_active":"2024-03-06T07:27:19.400953+0000","last_peered":"2024-03-06T07:27:19.400953+0000","last_clean":"2024-03-06T07:27:19.400953+0000","last_became_active":"2024-03-06T07:27:18.415900+0000","last_became_peered":"2024-03-06T07:27:18.415900+0000","last_unstale":"2024-03-06T07:27:19.400953+0000","last_undegraded":"2024-03-06T07:27:19.400953+0000","last_fullsized":"2024-03-06T07:27:19.400953+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:09.388853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400654+0000","last_change":"2024-03-06T07:27:17.412868+0000","last_active":"2024-03-06T07:27:19.400654+0000","last_peered":"2024-03-06T07:27:19.400654+0000","last_clean":"2024-03-06T07:27:19.400654+0000","last_became_active":"2024-03-06T07:27:17.412587+0000","last_became_peered":"2024-03-06T07:27:17.412587+0000","last_unstale":"2024-03-06T07:27:19.400654+0000","last_undegraded":"2024-03-06T07:27:19.400654+0000","last_fullsized":"2024-03-06T07:27:19.400654+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:16:00.130007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662812+0000","last_change":"2024-03-06T07:27:18.416360+0000","last_active":"2024-03-06T07:27:19.662812+0000","last_peered":"2024-03-06T07:27:19.662812+0000","last_clean":"2024-03-06T07:27:19.662812+0000","last_became_active":"2024-03-06T07:27:18.415678+0000","last_became_peered":"2024-03-06T07:27:18.415678+0000","last_unstale":"2024-03-06T07:27:19.662812+0000","last_undegraded":"2024-03-06T07:27:19.662812+0000","last_fullsized":"2024-03-06T07:27:19.662812+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:29.640409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406602+0000","last_change":"2024-03-06T07:27:17.410408+0000","last_active":"2024-03-06T07:27:19.406602+0000","last_peered":"2024-03-06T07:27:19.406602+0000","last_clean":"2024-03-06T07:27:19.406602+0000","last_became_active":"2024-03-06T07:27:17.410121+0000","last_became_peered":"2024-03-06T07:27:17.410121+0000","last_unstale":"2024-03-06T07:27:19.406602+0000","last_undegraded":"2024-03-06T07:27:19.406602+0000","last_fullsized":"2024-03-06T07:27:19.406602+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:11.367230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662460+0000","last_change":"2024-03-06T07:27:18.408509+0000","last_active":"2024-03-06T07:27:19.662460+0000","last_peered":"2024-03-06T07:27:19.662460+0000","last_clean":"2024-03-06T07:27:19.662460+0000","last_became_active":"2024-03-06T07:27:18.408222+0000","last_became_peered":"2024-03-06T07:27:18.408222+0000","last_unstale":"2024-03-06T07:27:19.662460+0000","last_undegraded":"2024-03-06T07:27:19.662460+0000","last_fullsized":"2024-03-06T07:27:19.662460+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:33.216263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"208'3","reported_seq":17,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400969+0000","last_change":"2024-03-06T07:27:17.414830+0000","last_active":"2024-03-06T07:27:19.400969+0000","last_peered":"2024-03-06T07:27:19.400969+0000","last_clean":"2024-03-06T07:27:19.400969+0000","last_became_active":"2024-03-06T07:27:17.414541+0000","last_became_peered":"2024-03-06T07:27:17.414541+0000","last_unstale":"2024-03-06T07:27:19.400969+0000","last_undegraded":"2024-03-06T07:27:19.400969+0000","last_fullsized":"2024-03-06T07:27:19.400969+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:59.281427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407518+0000","last_change":"2024-03-06T07:27:18.410136+0000","last_active":"2024-03-06T07:27:19.407518+0000","last_peered":"2024-03-06T07:27:19.407518+0000","last_clean":"2024-03-06T07:27:19.407518+0000","last_became_active":"2024-03-06T07:27:18.409843+0000","last_became_peered":"2024-03-06T07:27:18.409843+0000","last_unstale":"2024-03-06T07:27:19.407518+0000","last_undegraded":"2024-03-06T07:27:19.407518+0000","last_fullsized":"2024-03-06T07:27:19.407518+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:08.883861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"208'6","reported_seq":20,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662763+0000","last_change":"2024-03-06T07:27:17.413318+0000","last_active":"2024-03-06T07:27:19.662763+0000","last_peered":"2024-03-06T07:27:19.662763+0000","last_clean":"2024-03-06T07:27:19.662763+0000","last_became_active":"2024-03-06T07:27:17.413042+0000","last_became_peered":"2024-03-06T07:27:17.413042+0000","last_unstale":"2024-03-06T07:27:19.662763+0000","last_undegraded":"2024-03-06T07:27:19.662763+0000","last_fullsized":"2024-03-06T07:27:19.662763+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:56.250689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662108+0000","last_change":"2024-03-06T07:27:18.425272+0000","last_active":"2024-03-06T07:27:19.662108+0000","last_peered":"2024-03-06T07:27:19.662108+0000","last_clean":"2024-03-06T07:27:19.662108+0000","last_became_active":"2024-03-06T07:27:18.425015+0000","last_became_peered":"2024-03-06T07:27:18.425015+0000","last_unstale":"2024-03-06T07:27:19.662108+0000","last_undegraded":"2024-03-06T07:27:19.662108+0000","last_fullsized":"2024-03-06T07:27:19.662108+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:07.767067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"208'2","reported_seq":16,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663031+0000","last_change":"2024-03-06T07:27:17.412118+0000","last_active":"2024-03-06T07:27:19.663031+0000","last_peered":"2024-03-06T07:27:19.663031+0000","last_clean":"2024-03-06T07:27:19.663031+0000","last_became_active":"2024-03-06T07:27:17.411828+0000","last_became_peered":"2024-03-06T07:27:17.411828+0000","last_unstale":"2024-03-06T07:27:19.663031+0000","last_undegraded":"2024-03-06T07:27:19.663031+0000","last_fullsized":"2024-03-06T07:27:19.663031+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:32.809333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407114+0000","last_change":"2024-03-06T07:27:18.415756+0000","last_active":"2024-03-06T07:27:19.407114+0000","last_peered":"2024-03-06T07:27:19.407114+0000","last_clean":"2024-03-06T07:27:19.407114+0000","last_became_active":"2024-03-06T07:27:18.415407+0000","last_became_peered":"2024-03-06T07:27:18.415407+0000","last_unstale":"2024-03-06T07:27:19.407114+0000","last_undegraded":"2024-03-06T07:27:19.407114+0000","last_fullsized":"2024-03-06T07:27:19.407114+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:32.355733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400720+0000","last_change":"2024-03-06T07:27:17.416024+0000","last_active":"2024-03-06T07:27:19.400720+0000","last_peered":"2024-03-06T07:27:19.400720+0000","last_clean":"2024-03-06T07:27:19.400720+0000","last_became_active":"2024-03-06T07:27:17.415738+0000","last_became_peered":"2024-03-06T07:27:17.415738+0000","last_unstale":"2024-03-06T07:27:19.400720+0000","last_undegraded":"2024-03-06T07:27:19.400720+0000","last_fullsized":"2024-03-06T07:27:19.400720+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:09.753820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662045+0000","last_change":"2024-03-06T07:27:18.410319+0000","last_active":"2024-03-06T07:27:19.662045+0000","last_peered":"2024-03-06T07:27:19.662045+0000","last_clean":"2024-03-06T07:27:19.662045+0000","last_became_active":"2024-03-06T07:27:18.410072+0000","last_became_peered":"2024-03-06T07:27:18.410072+0000","last_unstale":"2024-03-06T07:27:19.662045+0000","last_undegraded":"2024-03-06T07:27:19.662045+0000","last_fullsized":"2024-03-06T07:27:19.662045+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:40.346944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400569+0000","last_change":"2024-03-06T07:27:17.416751+0000","last_active":"2024-03-06T07:27:19.400569+0000","last_peered":"2024-03-06T07:27:19.400569+0000","last_clean":"2024-03-06T07:27:19.400569+0000","last_became_active":"2024-03-06T07:27:17.416475+0000","last_became_peered":"2024-03-06T07:27:17.416475+0000","last_unstale":"2024-03-06T07:27:19.400569+0000","last_undegraded":"2024-03-06T07:27:19.400569+0000","last_fullsized":"2024-03-06T07:27:19.400569+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:58.100234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662947+0000","last_change":"2024-03-06T07:27:18.416478+0000","last_active":"2024-03-06T07:27:19.662947+0000","last_peered":"2024-03-06T07:27:19.662947+0000","last_clean":"2024-03-06T07:27:19.662947+0000","last_became_active":"2024-03-06T07:27:18.416171+0000","last_became_peered":"2024-03-06T07:27:18.416171+0000","last_unstale":"2024-03-06T07:27:19.662947+0000","last_undegraded":"2024-03-06T07:27:19.662947+0000","last_fullsized":"2024-03-06T07:27:19.662947+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:08.627577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"208'8","reported_seq":22,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400613+0000","last_change":"2024-03-06T07:27:17.408458+0000","last_active":"2024-03-06T07:27:19.400613+0000","last_peered":"2024-03-06T07:27:19.400613+0000","last_clean":"2024-03-06T07:27:19.400613+0000","last_became_active":"2024-03-06T07:27:17.408204+0000","last_became_peered":"2024-03-06T07:27:17.408204+0000","last_unstale":"2024-03-06T07:27:19.400613+0000","last_undegraded":"2024-03-06T07:27:19.400613+0000","last_fullsized":"2024-03-06T07:27:19.400613+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:21.056279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407410+0000","last_change":"2024-03-06T07:27:18.410532+0000","last_active":"2024-03-06T07:27:19.407410+0000","last_peered":"2024-03-06T07:27:19.407410+0000","last_clean":"2024-03-06T07:27:19.407410+0000","last_became_active":"2024-03-06T07:27:18.410270+0000","last_became_peered":"2024-03-06T07:27:18.410270+0000","last_unstale":"2024-03-06T07:27:19.407410+0000","last_undegraded":"2024-03-06T07:27:19.407410+0000","last_fullsized":"2024-03-06T07:27:19.407410+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:33.745293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"208'1","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662781+0000","last_change":"2024-03-06T07:27:17.418167+0000","last_active":"2024-03-06T07:27:19.662781+0000","last_peered":"2024-03-06T07:27:19.662781+0000","last_clean":"2024-03-06T07:27:19.662781+0000","last_became_active":"2024-03-06T07:27:17.417877+0000","last_became_peered":"2024-03-06T07:27:17.417877+0000","last_unstale":"2024-03-06T07:27:19.662781+0000","last_undegraded":"2024-03-06T07:27:19.662781+0000","last_fullsized":"2024-03-06T07:27:19.662781+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:03.421025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407608+0000","last_change":"2024-03-06T07:27:18.415939+0000","last_active":"2024-03-06T07:27:19.407608+0000","last_peered":"2024-03-06T07:27:19.407608+0000","last_clean":"2024-03-06T07:27:19.407608+0000","last_became_active":"2024-03-06T07:27:18.415545+0000","last_became_peered":"2024-03-06T07:27:18.415545+0000","last_unstale":"2024-03-06T07:27:19.407608+0000","last_undegraded":"2024-03-06T07:27:19.407608+0000","last_fullsized":"2024-03-06T07:27:19.407608+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:42.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"208'2","reported_seq":16,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662628+0000","last_change":"2024-03-06T07:27:17.409593+0000","last_active":"2024-03-06T07:27:19.662628+0000","last_peered":"2024-03-06T07:27:19.662628+0000","last_clean":"2024-03-06T07:27:19.662628+0000","last_became_active":"2024-03-06T07:27:17.409303+0000","last_became_peered":"2024-03-06T07:27:17.409303+0000","last_unstale":"2024-03-06T07:27:19.662628+0000","last_undegraded":"2024-03-06T07:27:19.662628+0000","last_fullsized":"2024-03-06T07:27:19.662628+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:06.080595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400805+0000","last_change":"2024-03-06T07:27:18.409240+0000","last_active":"2024-03-06T07:27:19.400805+0000","last_peered":"2024-03-06T07:27:19.400805+0000","last_clean":"2024-03-06T07:27:19.400805+0000","last_became_active":"2024-03-06T07:27:18.408920+0000","last_became_peered":"2024-03-06T07:27:18.408920+0000","last_unstale":"2024-03-06T07:27:19.400805+0000","last_undegraded":"2024-03-06T07:27:19.400805+0000","last_fullsized":"2024-03-06T07:27:19.400805+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:38.437529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"208'1","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662892+0000","last_change":"2024-03-06T07:27:17.416748+0000","last_active":"2024-03-06T07:27:19.662892+0000","last_peered":"2024-03-06T07:27:19.662892+0000","last_clean":"2024-03-06T07:27:19.662892+0000","last_became_active":"2024-03-06T07:27:17.416473+0000","last_became_peered":"2024-03-06T07:27:17.416473+0000","last_unstale":"2024-03-06T07:27:19.662892+0000","last_undegraded":"2024-03-06T07:27:19.662892+0000","last_fullsized":"2024-03-06T07:27:19.662892+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:41.835845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407627+0000","last_change":"2024-03-06T07:27:18.421920+0000","last_active":"2024-03-06T07:27:19.407627+0000","last_peered":"2024-03-06T07:27:19.407627+0000","last_clean":"2024-03-06T07:27:19.407627+0000","last_became_active":"2024-03-06T07:27:18.421649+0000","last_became_peered":"2024-03-06T07:27:18.421649+0000","last_unstale":"2024-03-06T07:27:19.407627+0000","last_undegraded":"2024-03-06T07:27:19.407627+0000","last_fullsized":"2024-03-06T07:27:19.407627+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:42.840824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662873+0000","last_change":"2024-03-06T07:27:17.409661+0000","last_active":"2024-03-06T07:27:19.662873+0000","last_peered":"2024-03-06T07:27:19.662873+0000","last_clean":"2024-03-06T07:27:19.662873+0000","last_became_active":"2024-03-06T07:27:17.409383+0000","last_became_peered":"2024-03-06T07:27:17.409383+0000","last_unstale":"2024-03-06T07:27:19.662873+0000","last_undegraded":"2024-03-06T07:27:19.662873+0000","last_fullsized":"2024-03-06T07:27:19.662873+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:53.561850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662563+0000","last_change":"2024-03-06T07:27:18.418544+0000","last_active":"2024-03-06T07:27:19.662563+0000","last_peered":"2024-03-06T07:27:19.662563+0000","last_clean":"2024-03-06T07:27:19.662563+0000","last_became_active":"2024-03-06T07:27:18.417622+0000","last_became_peered":"2024-03-06T07:27:18.417622+0000","last_unstale":"2024-03-06T07:27:19.662563+0000","last_undegraded":"2024-03-06T07:27:19.662563+0000","last_fullsized":"2024-03-06T07:27:19.662563+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:27.760834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"208'2","reported_seq":16,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663125+0000","last_change":"2024-03-06T07:27:17.420352+0000","last_active":"2024-03-06T07:27:19.663125+0000","last_peered":"2024-03-06T07:27:19.663125+0000","last_clean":"2024-03-06T07:27:19.663125+0000","last_became_active":"2024-03-06T07:27:17.420070+0000","last_became_peered":"2024-03-06T07:27:17.420070+0000","last_unstale":"2024-03-06T07:27:19.663125+0000","last_undegraded":"2024-03-06T07:27:19.663125+0000","last_fullsized":"2024-03-06T07:27:19.663125+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:31.669225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.407216+0000","last_change":"2024-03-06T07:27:18.410861+0000","last_active":"2024-03-06T07:27:19.407216+0000","last_peered":"2024-03-06T07:27:19.407216+0000","last_clean":"2024-03-06T07:27:19.407216+0000","last_became_active":"2024-03-06T07:27:18.410570+0000","last_became_peered":"2024-03-06T07:27:18.410570+0000","last_unstale":"2024-03-06T07:27:19.407216+0000","last_undegraded":"2024-03-06T07:27:19.407216+0000","last_fullsized":"2024-03-06T07:27:19.407216+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:41.580863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662061+0000","last_change":"2024-03-06T07:27:17.414539+0000","last_active":"2024-03-06T07:27:19.662061+0000","last_peered":"2024-03-06T07:27:19.662061+0000","last_clean":"2024-03-06T07:27:19.662061+0000","last_became_active":"2024-03-06T07:27:17.414268+0000","last_became_peered":"2024-03-06T07:27:17.414268+0000","last_unstale":"2024-03-06T07:27:19.662061+0000","last_undegraded":"2024-03-06T07:27:19.662061+0000","last_fullsized":"2024-03-06T07:27:19.662061+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:47.187108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400861+0000","last_change":"2024-03-06T07:27:18.415540+0000","last_active":"2024-03-06T07:27:19.400861+0000","last_peered":"2024-03-06T07:27:19.400861+0000","last_clean":"2024-03-06T07:27:19.400861+0000","last_became_active":"2024-03-06T07:27:18.415280+0000","last_became_peered":"2024-03-06T07:27:18.415280+0000","last_unstale":"2024-03-06T07:27:19.400861+0000","last_undegraded":"2024-03-06T07:27:19.400861+0000","last_fullsized":"2024-03-06T07:27:19.400861+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:54.730072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663264+0000","last_change":"2024-03-06T07:27:17.408812+0000","last_active":"2024-03-06T07:27:19.663264+0000","last_peered":"2024-03-06T07:27:19.663264+0000","last_clean":"2024-03-06T07:27:19.663264+0000","last_became_active":"2024-03-06T07:27:17.408558+0000","last_became_peered":"2024-03-06T07:27:17.408558+0000","last_unstale":"2024-03-06T07:27:19.663264+0000","last_undegraded":"2024-03-06T07:27:19.663264+0000","last_fullsized":"2024-03-06T07:27:19.663264+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:03.421025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663010+0000","last_change":"2024-03-06T07:27:18.412131+0000","last_active":"2024-03-06T07:27:19.663010+0000","last_peered":"2024-03-06T07:27:19.663010+0000","last_clean":"2024-03-06T07:27:19.663010+0000","last_became_active":"2024-03-06T07:27:18.411845+0000","last_became_peered":"2024-03-06T07:27:18.411845+0000","last_unstale":"2024-03-06T07:27:19.663010+0000","last_undegraded":"2024-03-06T07:27:19.663010+0000","last_fullsized":"2024-03-06T07:27:19.663010+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:53.210778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400673+0000","last_change":"2024-03-06T07:27:17.408777+0000","last_active":"2024-03-06T07:27:19.400673+0000","last_peered":"2024-03-06T07:27:19.400673+0000","last_clean":"2024-03-06T07:27:19.400673+0000","last_became_active":"2024-03-06T07:27:17.408201+0000","last_became_peered":"2024-03-06T07:27:17.408201+0000","last_unstale":"2024-03-06T07:27:19.400673+0000","last_undegraded":"2024-03-06T07:27:19.400673+0000","last_fullsized":"2024-03-06T07:27:19.400673+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:33.553308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662682+0000","last_change":"2024-03-06T07:27:18.416978+0000","last_active":"2024-03-06T07:27:19.662682+0000","last_peered":"2024-03-06T07:27:19.662682+0000","last_clean":"2024-03-06T07:27:19.662682+0000","last_became_active":"2024-03-06T07:27:18.416477+0000","last_became_peered":"2024-03-06T07:27:18.416477+0000","last_unstale":"2024-03-06T07:27:19.662682+0000","last_undegraded":"2024-03-06T07:27:19.662682+0000","last_fullsized":"2024-03-06T07:27:19.662682+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:17.214607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"208'2","reported_seq":16,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400746+0000","last_change":"2024-03-06T07:27:17.413559+0000","last_active":"2024-03-06T07:27:19.400746+0000","last_peered":"2024-03-06T07:27:19.400746+0000","last_clean":"2024-03-06T07:27:19.400746+0000","last_became_active":"2024-03-06T07:27:17.413298+0000","last_became_peered":"2024-03-06T07:27:17.413298+0000","last_unstale":"2024-03-06T07:27:19.400746+0000","last_undegraded":"2024-03-06T07:27:19.400746+0000","last_fullsized":"2024-03-06T07:27:19.400746+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:16.491323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662337+0000","last_change":"2024-03-06T07:27:18.409625+0000","last_active":"2024-03-06T07:27:19.662337+0000","last_peered":"2024-03-06T07:27:19.662337+0000","last_clean":"2024-03-06T07:27:19.662337+0000","last_became_active":"2024-03-06T07:27:18.409356+0000","last_became_peered":"2024-03-06T07:27:18.409356+0000","last_unstale":"2024-03-06T07:27:19.662337+0000","last_undegraded":"2024-03-06T07:27:19.662337+0000","last_fullsized":"2024-03-06T07:27:19.662337+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:30.934018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"209'1","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663387+0000","last_change":"2024-03-06T07:27:17.415152+0000","last_active":"2024-03-06T07:27:19.663387+0000","last_peered":"2024-03-06T07:27:19.663387+0000","last_clean":"2024-03-06T07:27:19.663387+0000","last_became_active":"2024-03-06T07:27:17.414894+0000","last_became_peered":"2024-03-06T07:27:17.414894+0000","last_unstale":"2024-03-06T07:27:19.663387+0000","last_undegraded":"2024-03-06T07:27:19.663387+0000","last_fullsized":"2024-03-06T07:27:19.663387+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:45.798253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662684+0000","last_change":"2024-03-06T07:27:18.421305+0000","last_active":"2024-03-06T07:27:19.662684+0000","last_peered":"2024-03-06T07:27:19.662684+0000","last_clean":"2024-03-06T07:27:19.662684+0000","last_became_active":"2024-03-06T07:27:18.420004+0000","last_became_peered":"2024-03-06T07:27:18.420004+0000","last_unstale":"2024-03-06T07:27:19.662684+0000","last_undegraded":"2024-03-06T07:27:19.662684+0000","last_fullsized":"2024-03-06T07:27:19.662684+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:25.593638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662996+0000","last_change":"2024-03-06T07:27:17.418924+0000","last_active":"2024-03-06T07:27:19.662996+0000","last_peered":"2024-03-06T07:27:19.662996+0000","last_clean":"2024-03-06T07:27:19.662996+0000","last_became_active":"2024-03-06T07:27:17.418627+0000","last_became_peered":"2024-03-06T07:27:17.418627+0000","last_unstale":"2024-03-06T07:27:19.662996+0000","last_undegraded":"2024-03-06T07:27:19.662996+0000","last_fullsized":"2024-03-06T07:27:19.662996+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:13:11.367230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662567+0000","last_change":"2024-03-06T07:27:18.411523+0000","last_active":"2024-03-06T07:27:19.662567+0000","last_peered":"2024-03-06T07:27:19.662567+0000","last_clean":"2024-03-06T07:27:19.662567+0000","last_became_active":"2024-03-06T07:27:18.411228+0000","last_became_peered":"2024-03-06T07:27:18.411228+0000","last_unstale":"2024-03-06T07:27:19.662567+0000","last_undegraded":"2024-03-06T07:27:19.662567+0000","last_fullsized":"2024-03-06T07:27:19.662567+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:37.875812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662962+0000","last_change":"2024-03-06T07:27:17.416845+0000","last_active":"2024-03-06T07:27:19.662962+0000","last_peered":"2024-03-06T07:27:19.662962+0000","last_clean":"2024-03-06T07:27:19.662962+0000","last_became_active":"2024-03-06T07:27:17.416518+0000","last_became_peered":"2024-03-06T07:27:17.416518+0000","last_unstale":"2024-03-06T07:27:19.662962+0000","last_undegraded":"2024-03-06T07:27:19.662962+0000","last_fullsized":"2024-03-06T07:27:19.662962+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:39.680542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400497+0000","last_change":"2024-03-06T07:27:18.411423+0000","last_active":"2024-03-06T07:27:19.400497+0000","last_peered":"2024-03-06T07:27:19.400497+0000","last_clean":"2024-03-06T07:27:19.400497+0000","last_became_active":"2024-03-06T07:27:18.411173+0000","last_became_peered":"2024-03-06T07:27:18.411173+0000","last_unstale":"2024-03-06T07:27:19.400497+0000","last_undegraded":"2024-03-06T07:27:19.400497+0000","last_fullsized":"2024-03-06T07:27:19.400497+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:06.013073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662505+0000","last_change":"2024-03-06T07:27:17.407570+0000","last_active":"2024-03-06T07:27:19.662505+0000","last_peered":"2024-03-06T07:27:19.662505+0000","last_clean":"2024-03-06T07:27:19.662505+0000","last_became_active":"2024-03-06T07:27:17.407285+0000","last_became_peered":"2024-03-06T07:27:17.407285+0000","last_unstale":"2024-03-06T07:27:19.662505+0000","last_undegraded":"2024-03-06T07:27:19.662505+0000","last_fullsized":"2024-03-06T07:27:19.662505+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:42.349001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663189+0000","last_change":"2024-03-06T07:27:18.412893+0000","last_active":"2024-03-06T07:27:19.663189+0000","last_peered":"2024-03-06T07:27:19.663189+0000","last_clean":"2024-03-06T07:27:19.663189+0000","last_became_active":"2024-03-06T07:27:18.412629+0000","last_became_peered":"2024-03-06T07:27:18.412629+0000","last_unstale":"2024-03-06T07:27:19.663189+0000","last_undegraded":"2024-03-06T07:27:19.663189+0000","last_fullsized":"2024-03-06T07:27:19.663189+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:20.655691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662475+0000","last_change":"2024-03-06T07:27:17.415917+0000","last_active":"2024-03-06T07:27:19.662475+0000","last_peered":"2024-03-06T07:27:19.662475+0000","last_clean":"2024-03-06T07:27:19.662475+0000","last_became_active":"2024-03-06T07:27:17.414080+0000","last_became_peered":"2024-03-06T07:27:17.414080+0000","last_unstale":"2024-03-06T07:27:19.662475+0000","last_undegraded":"2024-03-06T07:27:19.662475+0000","last_fullsized":"2024-03-06T07:27:19.662475+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:58.537917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663536+0000","last_change":"2024-03-06T07:27:18.419085+0000","last_active":"2024-03-06T07:27:19.663536+0000","last_peered":"2024-03-06T07:27:19.663536+0000","last_clean":"2024-03-06T07:27:19.663536+0000","last_became_active":"2024-03-06T07:27:18.418817+0000","last_became_peered":"2024-03-06T07:27:18.418817+0000","last_unstale":"2024-03-06T07:27:19.663536+0000","last_undegraded":"2024-03-06T07:27:19.663536+0000","last_fullsized":"2024-03-06T07:27:19.663536+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:32.674204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662538+0000","last_change":"2024-03-06T07:27:17.415251+0000","last_active":"2024-03-06T07:27:19.662538+0000","last_peered":"2024-03-06T07:27:19.662538+0000","last_clean":"2024-03-06T07:27:19.662538+0000","last_became_active":"2024-03-06T07:27:17.414993+0000","last_became_peered":"2024-03-06T07:27:17.414993+0000","last_unstale":"2024-03-06T07:27:19.662538+0000","last_undegraded":"2024-03-06T07:27:19.662538+0000","last_fullsized":"2024-03-06T07:27:19.662538+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:17.295013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.406538+0000","last_change":"2024-03-06T07:27:18.418437+0000","last_active":"2024-03-06T07:27:19.406538+0000","last_peered":"2024-03-06T07:27:19.406538+0000","last_clean":"2024-03-06T07:27:19.406538+0000","last_became_active":"2024-03-06T07:27:18.417533+0000","last_became_peered":"2024-03-06T07:27:18.417533+0000","last_unstale":"2024-03-06T07:27:19.406538+0000","last_undegraded":"2024-03-06T07:27:19.406538+0000","last_fullsized":"2024-03-06T07:27:19.406538+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:25.593638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.663302+0000","last_change":"2024-03-06T07:27:17.412587+0000","last_active":"2024-03-06T07:27:19.663302+0000","last_peered":"2024-03-06T07:27:19.663302+0000","last_clean":"2024-03-06T07:27:19.663302+0000","last_became_active":"2024-03-06T07:27:17.412322+0000","last_became_peered":"2024-03-06T07:27:17.412322+0000","last_unstale":"2024-03-06T07:27:19.663302+0000","last_undegraded":"2024-03-06T07:27:19.663302+0000","last_fullsized":"2024-03-06T07:27:19.663302+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:16.385900+0000","last_clean_scrub_stamp":"2024-03-06T07:27:16.385900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:28.244923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.400895+0000","last_change":"2024-03-06T07:27:18.417804+0000","last_active":"2024-03-06T07:27:19.400895+0000","last_peered":"2024-03-06T07:27:19.400895+0000","last_clean":"2024-03-06T07:27:19.400895+0000","last_became_active":"2024-03-06T07:27:18.416699+0000","last_became_peered":"2024-03-06T07:27:18.416699+0000","last_unstale":"2024-03-06T07:27:19.400895+0000","last_undegraded":"2024-03-06T07:27:19.400895+0000","last_fullsized":"2024-03-06T07:27:19.400895+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.390880+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.390880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:45.349515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":526,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-03-06T07:27:19.662168+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:27:19.662168+0000","last_peered":"2024-03-06T07:27:19.662168+0000","last_clean":"2024-03-06T07:27:19.662168+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:27:19.662168+0000","last_undegraded":"2024-03-06T07:27:19.662168+0000","last_fullsized":"2024-03-06T07:27:19.662168+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574962,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33844,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337996,"statfs":{"total":96636764160,"available":96602107904,"internally_reserved":0,"allocated":2727936,"data_stored":1877024,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2658304,"data_stored":1286086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574962,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2658304,"data_stored":1286122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33792,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338048,"statfs":{"total":96636764160,"available":96602161152,"internally_reserved":0,"allocated":2666496,"data_stored":1287664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574963,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33784,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338056,"statfs":{"total":96636764160,"available":96602169344,"internally_reserved":0,"allocated":2658304,"data_stored":1286142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37960,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333880,"statfs":{"total":96636764160,"available":96597893120,"internally_reserved":0,"allocated":2674688,"data_stored":1288268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37948,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333892,"statfs":{"total":96636764160,"available":96597905408,"internally_reserved":0,"allocated":2662400,"data_stored":1286176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574962,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38000,"kb_used_data":2660,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333840,"statfs":{"total":96636764160,"available":96597852160,"internally_reserved":0,"allocated":2723840,"data_stored":1876490,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:26.075 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:26.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:26.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.218+0000 7fb8f3e50640 1 Processor -- start 2024-03-06T07:27:26.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.218+0000 7fb8f3e50640 1 -- start start 2024-03-06T07:27:26.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f3e50640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06c1f0 0x7fb8ec0c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f3e50640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec0c2790 con 0x7fb8ec06cad0 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f3e50640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec0c1540 con 0x7fb8ec14ef70 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f3e50640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0c3090 con 0x7fb8ec06c1f0 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f23ed640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb8ec06cad0 0x7fb8ec154c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42852/0 (socket says 172.21.15.94:42852) 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f23ed640 1 -- 172.21.15.94:0/1876369025 learned_addr learned my addr 172.21.15.94:0/1876369025 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f13eb640 1 --2- 172.21.15.94:0/1876369025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06c1f0 0x7fb8ec0c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f0bea640 1 -- 172.21.15.94:0/1876369025 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116260864 0 0) 0x7fb8ec0c2790 con 0x7fb8ec06cad0 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f0bea640 1 -- 172.21.15.94:0/1876369025 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d8003520 con 0x7fb8ec06cad0 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f0bea640 1 -- 172.21.15.94:0/1876369025 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590157404 0 0) 0x7fb8ec0c1540 con 0x7fb8ec14ef70 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f0bea640 1 -- 172.21.15.94:0/1876369025 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ec0c2790 con 0x7fb8ec14ef70 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f13eb640 1 -- 172.21.15.94:0/1876369025 >> v1:172.21.15.145:6789/0 conn(0x7fb8ec14ef70 legacy=0x7fb8ec0c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f13eb640 1 -- 172.21.15.94:0/1876369025 >> v1:172.21.15.94:6789/0 conn(0x7fb8ec06cad0 legacy=0x7fb8ec154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.219+0000 7fb8f13eb640 1 -- 172.21.15.94:0/1876369025 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec0c24b0 con 0x7fb8ec06c1f0 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.220+0000 7fb8f13eb640 1 --2- 172.21.15.94:0/1876369025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06c1f0 0x7fb8ec0c1820 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0097d0 tx=0x7fb8dc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=defd1924790a6745 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.220+0000 7fb8f0bea640 1 -- 172.21.15.94:0/1876369025 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc035020 con 0x7fb8ec06c1f0 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.220+0000 7fb8f0bea640 1 -- 172.21.15.94:0/1876369025 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc031800 con 0x7fb8ec06c1f0 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.220+0000 7fb8f0bea640 1 -- 172.21.15.94:0/1876369025 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc031bb0 con 0x7fb8ec06c1f0 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.221+0000 7fb8f3e50640 1 -- 172.21.15.94:0/1876369025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06c1f0 msgr2=0x7fb8ec0c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.221+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/1876369025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06c1f0 0x7fb8ec0c1820 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0097d0 tx=0x7fb8dc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.221+0000 7fb8f3e50640 1 -- 172.21.15.94:0/1876369025 shutdown_connections 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.221+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/1876369025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06c1f0 0x7fb8ec0c1820 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.221+0000 7fb8f3e50640 1 -- 172.21.15.94:0/1876369025 >> 172.21.15.94:0/1876369025 conn(0x7fb8ec0c6710 msgr2=0x7fb8ec0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.221+0000 7fb8f3e50640 1 -- 172.21.15.94:0/1876369025 shutdown_connections 2024-03-06T07:27:26.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.221+0000 7fb8f3e50640 1 -- 172.21.15.94:0/1876369025 wait complete. 2024-03-06T07:27:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 Processor -- start 2024-03-06T07:27:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 -- start start 2024-03-06T07:27:26.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8ec06c1f0 0x7fb8ec1ecc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06cad0 0x7fb8ec151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8ec14ef70 0x7fb8ec140db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec06cf60 con 0x7fb8ec14ef70 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec14f350 con 0x7fb8ec06c1f0 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.222+0000 7fb8f3e50640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec155340 con 0x7fb8ec06cad0 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8ec14ef70 0x7fb8ec140db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8ec14ef70 0x7fb8ec140db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47902/0 (socket says 172.21.15.94:47902) 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 -- 172.21.15.94:0/3060356167 learned_addr learned my addr 172.21.15.94:0/3060356167 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f13eb640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06cad0 0x7fb8ec151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f1bec640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8ec06c1f0 0x7fb8ec1ecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 -- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06cad0 msgr2=0x7fb8ec151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06cad0 0x7fb8ec151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 -- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8ec06c1f0 msgr2=0x7fb8ec1ecc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8ec06c1f0 0x7fb8ec1ecc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 -- 172.21.15.94:0/3060356167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec0703f0 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f23ed640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8ec14ef70 0x7fb8ec140db0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400cb40 tx=0x7fb8e400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8e27fc640 1 -- 172.21.15.94:0/3060356167 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4017070 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8e27fc640 1 -- 172.21.15.94:0/3060356167 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e400ac60 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8e27fc640 1 -- 172.21.15.94:0/3060356167 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4012640 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec0706d0 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.223+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec151e80 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.224+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec070bf0 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.225+0000 7fb8e27fc640 1 -- 172.21.15.94:0/3060356167 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8e401a070 con 0x7fb8ec14ef70 2024-03-06T07:27:26.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.225+0000 7fb8e27fc640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d006b4d0 0x7fb8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.225+0000 7fb8e27fc640 1 -- 172.21.15.94:0/3060356167 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb8e408a080 con 0x7fb8ec14ef70 2024-03-06T07:27:26.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.225+0000 7fb8f1bec640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d006b4d0 0x7fb8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.226+0000 7fb8f1bec640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d006b4d0 0x7fb8d006d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8007fd0 tx=0x7fb8e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.229+0000 7fb8e27fc640 1 -- 172.21.15.94:0/3060356167 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8ec070bf0 con 0x7fb8ec14ef70 2024-03-06T07:27:26.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.470+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb8ec04f330 con 0x7fb8ec14ef70 2024-03-06T07:27:26.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.472+0000 7fb8e27fc640 1 -- 172.21.15.94:0/3060356167 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v24) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb8e40587b0 con 0x7fb8ec14ef70 2024-03-06T07:27:26.474 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-06T07:27:26.474 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQB8GOhlVd3ZGBAAudJYpLX5y0Lp69mmVG3uKQ== 2024-03-06T07:27:26.474 INFO:teuthology.orchestra.run.smithi094.stdout: caps mds = "allow" 2024-03-06T07:27:26.475 INFO:teuthology.orchestra.run.smithi094.stdout: caps mon = "allow r" 2024-03-06T07:27:26.475 INFO:teuthology.orchestra.run.smithi094.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:26.475 INFO:teuthology.orchestra.run.smithi094.stderr:updated caps for client.0 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d006b4d0 msgr2=0x7fb8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d006b4d0 0x7fb8d006d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8007fd0 tx=0x7fb8e8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8ec14ef70 msgr2=0x7fb8ec140db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8ec14ef70 0x7fb8ec140db0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400cb40 tx=0x7fb8e400a060 comp rx=0 tx=0).stop 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 shutdown_connections 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d006b4d0 0x7fb8d006d990 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8ec14ef70 0x7fb8ec140db0 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8ec06cad0 0x7fb8ec151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 --2- 172.21.15.94:0/3060356167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8ec06c1f0 0x7fb8ec1ecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.474+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 >> 172.21.15.94:0/3060356167 conn(0x7fb8ec0c6710 msgr2=0x7fb8ec143770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.475+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 shutdown_connections 2024-03-06T07:27:26.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.475+0000 7fb8f3e50640 1 -- 172.21.15.94:0/3060356167 wait complete. 2024-03-06T07:27:26.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:26.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.585+0000 7f3f522c9640 1 Processor -- start 2024-03-06T07:27:26.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.585+0000 7f3f522c9640 1 -- start start 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f522c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3f4c06d9f0 0x7f3f4c0a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f522c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c0c3280 0x7f3f4c0c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f522c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3f4c1551e0 0x7f3f4c155610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f522c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c158800 con 0x7f3f4c06d9f0 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f522c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c158ae0 con 0x7f3f4c1551e0 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f522c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c158dc0 con 0x7f3f4c0c3280 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3f4c06d9f0 0x7f3f4c0a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3f4c06d9f0 0x7f3f4c0a7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47910/0 (socket says 172.21.15.94:47910) 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 -- 172.21.15.94:0/567363789 learned_addr learned my addr 172.21.15.94:0/567363789 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f50866640 1 --2- 172.21.15.94:0/567363789 >> v2:172.21.15.145:3300/0 conn(0x7f3f4c1551e0 0x7f3f4c155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4b7fe640 1 --2- 172.21.15.94:0/567363789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c0c3280 0x7f3f4c0c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 -- 172.21.15.94:0/567363789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c0c3280 msgr2=0x7f3f4c0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/567363789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c0c3280 0x7f3f4c0c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 -- 172.21.15.94:0/567363789 >> v2:172.21.15.145:3300/0 conn(0x7f3f4c1551e0 msgr2=0x7f3f4c155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/567363789 >> v2:172.21.15.145:3300/0 conn(0x7f3f4c1551e0 0x7f3f4c155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.586+0000 7f3f4bfff640 1 -- 172.21.15.94:0/567363789 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c1590a0 con 0x7f3f4c06d9f0 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.587+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/567363789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06d9f0 0x7f3f4c0a7cd0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0c3130 tx=0x7f3f3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4775107b0f470473 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.587+0000 7f3f4affd640 1 -- 172.21.15.94:0/567363789 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c02e070 con 0x7f3f4c06d9f0 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.587+0000 7f3f4affd640 1 -- 172.21.15.94:0/567363789 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c037070 con 0x7f3f4c06d9f0 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.587+0000 7f3f4affd640 1 -- 172.21.15.94:0/567363789 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c031a00 con 0x7f3f4c06d9f0 2024-03-06T07:27:26.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 -- 172.21.15.94:0/567363789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06d9f0 msgr2=0x7f3f4c0a7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 --2- 172.21.15.94:0/567363789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06d9f0 0x7f3f4c0a7cd0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0c3130 tx=0x7f3f3c02b880 comp rx=0 tx=0).stop 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 -- 172.21.15.94:0/567363789 shutdown_connections 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 --2- 172.21.15.94:0/567363789 >> v2:172.21.15.145:3300/0 conn(0x7f3f4c1551e0 0x7f3f4c155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 --2- 172.21.15.94:0/567363789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c0c3280 0x7f3f4c0c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 --2- 172.21.15.94:0/567363789 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c06d9f0 0x7f3f4c0a7cd0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 -- 172.21.15.94:0/567363789 >> 172.21.15.94:0/567363789 conn(0x7f3f4c0aa630 msgr2=0x7f3f4c0aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 -- 172.21.15.94:0/567363789 shutdown_connections 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.588+0000 7f3f522c9640 1 -- 172.21.15.94:0/567363789 wait complete. 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.589+0000 7f3f522c9640 1 Processor -- start 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.589+0000 7f3f522c9640 1 -- start start 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f4c06d9f0 0x7f3f4c0a9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c0c3280 0x7f3f4c06e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c1551e0 0x7f3f4c06eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c06df50 con 0x7f3f4c0c3280 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c06ddd0 con 0x7f3f4c06d9f0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0a8210 con 0x7f3f4c1551e0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c0c3280 0x7f3f4c06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c0c3280 0x7f3f4c06e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47922/0 (socket says 172.21.15.94:47922) 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 -- 172.21.15.94:0/3023005033 learned_addr learned my addr 172.21.15.94:0/3023005033 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f4c06d9f0 0x7f3f4c0a9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f50866640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c1551e0 0x7f3f4c06eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 -- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c1551e0 msgr2=0x7f3f4c06eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c1551e0 0x7f3f4c06eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 -- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f4c06d9f0 msgr2=0x7f3f4c0a9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f4c06d9f0 0x7f3f4c0a9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 -- 172.21.15.94:0/3023005033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c1eb8d0 con 0x7f3f4c0c3280 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f4b7fe640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c0c3280 0x7f3f4c06e970 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e8a0 tx=0x7f3f4000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f48ff9640 1 -- 172.21.15.94:0/3023005033 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40017070 con 0x7f3f4c0c3280 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f48ff9640 1 -- 172.21.15.94:0/3023005033 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4000c940 con 0x7f3f4c0c3280 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c1ebbb0 con 0x7f3f4c0c3280 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f48ff9640 1 -- 172.21.15.94:0/3023005033 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000cd30 con 0x7f3f4c0c3280 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.590+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c0aa100 con 0x7f3f4c0c3280 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.591+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c04f330 con 0x7f3f4c0c3280 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.592+0000 7f3f48ff9640 1 -- 172.21.15.94:0/3023005033 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f4001a070 con 0x7f3f4c0c3280 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.592+0000 7f3f48ff9640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f3806b500 0x7f3f3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.592+0000 7f3f48ff9640 1 -- 172.21.15.94:0/3023005033 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3f4008a480 con 0x7f3f4c0c3280 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.592+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f3806b500 0x7f3f3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.593+0000 7f3f4bfff640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f3806b500 0x7f3f3806d9c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c036000 tx=0x7f3f3c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.596+0000 7f3f48ff9640 1 -- 172.21.15.94:0/3023005033 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f40058bb0 con 0x7f3f4c0c3280 2024-03-06T07:27:26.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.804+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f4c073240 con 0x7f3f4c0c3280 2024-03-06T07:27:26.807 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.805+0000 7f3f48ff9640 1 -- 172.21.15.94:0/3023005033 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+4995 (secure 0 0 0) 0x7f3f4005cc50 con 0x7f3f4c0c3280 2024-03-06T07:27:26.807 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:26.807 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":196,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:19.757407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:26.807 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 196 2024-03-06T07:27:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f3806b500 msgr2=0x7f3f3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f3806b500 0x7f3f3806d9c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c036000 tx=0x7f3f3c03d040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c0c3280 msgr2=0x7f3f4c06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c0c3280 0x7f3f4c06e970 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e8a0 tx=0x7f3f4000ed60 comp rx=0 tx=0).stop 2024-03-06T07:27:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 shutdown_connections 2024-03-06T07:27:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3f3806b500 0x7f3f3806d9c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f4c1551e0 0x7f3f4c06eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3f4c0c3280 0x7f3f4c06e970 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 --2- 172.21.15.94:0/3023005033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f4c06d9f0 0x7f3f4c0a9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 >> 172.21.15.94:0/3023005033 conn(0x7f3f4c0aa630 msgr2=0x7f3f4c06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 shutdown_connections 2024-03-06T07:27:26.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:26.807+0000 7f3f522c9640 1 -- 172.21.15.94:0/3023005033 wait complete. 2024-03-06T07:27:27.222 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 196, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:19.757407+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 38273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:27.222 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:27.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:27:27.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.323+0000 7fde1bf7a640 1 Processor -- start 2024-03-06T07:27:27.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde1bf7a640 1 -- start start 2024-03-06T07:27:27.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde1bf7a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde141561a0 0x7fde141529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde1bf7a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde140a9990 con 0x7fde1414f110 2024-03-06T07:27:27.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde1bf7a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde141556c0 con 0x7fde140c6ab0 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde1bf7a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde14155d10 con 0x7fde141561a0 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde19515640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde141561a0 0x7fde141529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde19515640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fde141561a0 0x7fde141529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47930/0 (socket says 172.21.15.94:47930) 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde19d16640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fde140c6ab0 0x7fde140a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:51622/0 (socket says 172.21.15.94:51622) 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.324+0000 7fde19515640 1 -- 172.21.15.94:0/3391194440 learned_addr learned my addr 172.21.15.94:0/3391194440 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.325+0000 7fde19515640 1 -- 172.21.15.94:0/3391194440 >> v1:172.21.15.145:6789/0 conn(0x7fde140c6ab0 legacy=0x7fde140a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.325+0000 7fde19515640 1 -- 172.21.15.94:0/3391194440 >> v1:172.21.15.94:6789/0 conn(0x7fde1414f110 legacy=0x7fde1415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.325+0000 7fde18d14640 1 -- 172.21.15.94:0/3391194440 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87954366 0 0) 0x7fde141556c0 con 0x7fde140c6ab0 2024-03-06T07:27:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.325+0000 7fde19515640 1 -- 172.21.15.94:0/3391194440 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde141559a0 con 0x7fde141561a0 2024-03-06T07:27:27.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.325+0000 7fde19515640 1 --2- 172.21.15.94:0/3391194440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde141561a0 0x7fde141529e0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fde04009870 tx=0x7fde0402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0242e744e3d7afc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.325+0000 7fde18d14640 1 -- 172.21.15.94:0/3391194440 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04037070 con 0x7fde141561a0 2024-03-06T07:27:27.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde18d14640 1 -- 172.21.15.94:0/3391194440 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04004230 con 0x7fde141561a0 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde18d14640 1 -- 172.21.15.94:0/3391194440 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04003610 con 0x7fde141561a0 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde1bf7a640 1 -- 172.21.15.94:0/3391194440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde141561a0 msgr2=0x7fde141529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/3391194440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde141561a0 0x7fde141529e0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fde04009870 tx=0x7fde0402b7e0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde1bf7a640 1 -- 172.21.15.94:0/3391194440 shutdown_connections 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/3391194440 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde141561a0 0x7fde141529e0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde1bf7a640 1 -- 172.21.15.94:0/3391194440 >> 172.21.15.94:0/3391194440 conn(0x7fde1406b1a0 msgr2=0x7fde1406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde1bf7a640 1 -- 172.21.15.94:0/3391194440 shutdown_connections 2024-03-06T07:27:27.329 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.326+0000 7fde1bf7a640 1 -- 172.21.15.94:0/3391194440 wait complete. 2024-03-06T07:27:27.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.327+0000 7fde1bf7a640 1 Processor -- start 2024-03-06T07:27:27.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.327+0000 7fde1bf7a640 1 -- start start 2024-03-06T07:27:27.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde1bf7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde140c6ab0 0x7fde1414a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde1bf7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde1414f110 0x7fde141eac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde1bf7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde141561a0 0x7fde14141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde1bf7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140aa600 con 0x7fde140c6ab0 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde1bf7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde140c2a90 con 0x7fde141561a0 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde1bf7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1414a5b0 con 0x7fde1414f110 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde140c6ab0 0x7fde1414a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde140c6ab0 0x7fde1414a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47932/0 (socket says 172.21.15.94:47932) 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19d16640 1 -- 172.21.15.94:0/1945946732 learned_addr learned my addr 172.21.15.94:0/1945946732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19515640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde1414f110 0x7fde141eac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde1a517640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde141561a0 0x7fde14141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19d16640 1 -- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde1414f110 msgr2=0x7fde141eac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19d16640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde1414f110 0x7fde141eac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19d16640 1 -- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde141561a0 msgr2=0x7fde14141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.328+0000 7fde19d16640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde141561a0 0x7fde14141370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.329+0000 7fde19d16640 1 -- 172.21.15.94:0/1945946732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14151cb0 con 0x7fde140c6ab0 2024-03-06T07:27:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.329+0000 7fde19d16640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde140c6ab0 0x7fde1414a070 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c7c0 tx=0x7fde1000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.329+0000 7fde0a7fc640 1 -- 172.21.15.94:0/1945946732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10018070 con 0x7fde140c6ab0 2024-03-06T07:27:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.329+0000 7fde0a7fc640 1 -- 172.21.15.94:0/1945946732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1000a640 con 0x7fde140c6ab0 2024-03-06T07:27:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.329+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1414a770 con 0x7fde140c6ab0 2024-03-06T07:27:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.329+0000 7fde0a7fc640 1 -- 172.21.15.94:0/1945946732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1000aa30 con 0x7fde140c6ab0 2024-03-06T07:27:27.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.329+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1414ad30 con 0x7fde140c6ab0 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.330+0000 7fde0a7fc640 1 -- 172.21.15.94:0/1945946732 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde1001b070 con 0x7fde140c6ab0 2024-03-06T07:27:27.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.331+0000 7fde0a7fc640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fddf806b4d0 0x7fddf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.331+0000 7fde0a7fc640 1 -- 172.21.15.94:0/1945946732 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde10089880 con 0x7fde140c6ab0 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.331+0000 7fde19515640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fddf806b4d0 0x7fddf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.331+0000 7fde19515640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fddf806b4d0 0x7fddf806d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fde04034000 tx=0x7fde04008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 Processor -- start 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 -- start start 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928a5d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a62d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a63280 0x55c928aa9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9289c76b0 con 0x55c928a9fc40 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c928970a60 con 0x55c928a9e8f0 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.332+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c9289c3750 con 0x55c928a63280 2024-03-06T07:27:27.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a62d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a62d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47944/0 (socket says 172.21.15.94:47944) 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 -- 172.21.15.94:0/3170915911 learned_addr learned my addr 172.21.15.94:0/3170915911 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19d16640 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928a5d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde1a517640 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a63280 0x55c928aa9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 -- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a63280 msgr2=0x55c928aa9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a63280 0x55c928aa9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 -- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 msgr2=0x55c928a5d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928a5d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 -- 172.21.15.94:0/3170915911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c92898ef30 con 0x55c928a9fc40 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fde19515640 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a62d40 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fde0402bdf0 tx=0x7fde0402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ccf62745377009 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fddf77fe640 1 -- 172.21.15.94:0/3170915911 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0404e070 con 0x55c928a9fc40 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fddf77fe640 1 -- 172.21.15.94:0/3170915911 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04030510 con 0x55c928a9fc40 2024-03-06T07:27:27.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.333+0000 7fddf77fe640 1 -- 172.21.15.94:0/3170915911 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04037040 con 0x55c928a9fc40 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 -- 172.21.15.94:0/3170915911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 msgr2=0x55c928a62d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a62d40 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fde0402bdf0 tx=0x7fde0402f000 comp rx=0 tx=0).stop 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 -- 172.21.15.94:0/3170915911 shutdown_connections 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a63280 0x55c928aa9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a62d40 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3170915911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928a5d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 -- 172.21.15.94:0/3170915911 >> 172.21.15.94:0/3170915911 conn(0x55c928a5f7d0 msgr2=0x55c928a843c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 -- 172.21.15.94:0/3170915911 shutdown_connections 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 -- 172.21.15.94:0/3170915911 wait complete. 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 -- start start 2024-03-06T07:27:27.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.334+0000 7fde1ec39740 1 Processor -- start 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a9b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9c1b0 0x55c928a83de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1ec39740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c928a9f7b0 con 0x55c928a9fc40 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c928983ea0 con 0x55c928a9c1b0 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c928aa0b70 con 0x55c928a9e8f0 2024-03-06T07:27:27.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde19515640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a9b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1ec39740 10 client.0 _ll_register_callbacks cb 0x55c928983870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde19515640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a9b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47956/0 (socket says 172.21.15.94:47956) 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde19515640 1 -- 172.21.15.94:0/2829731346 learned_addr learned my addr 172.21.15.94:0/2829731346 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde19d16640 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.335+0000 7fde1a517640 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9c1b0 0x55c928a83de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde19515640 1 -- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 msgr2=0x55c928a82f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde19515640 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde19515640 1 -- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9c1b0 msgr2=0x55c928a83de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde19515640 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9c1b0 0x55c928a83de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde19515640 1 -- 172.21.15.94:0/2829731346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c928984270 con 0x55c928a9fc40 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde19515640 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a9b110 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fde0402bdf0 tx=0x7fde04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2f931c28092fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fddf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fddf4ff9640 1 -- 172.21.15.94:0/2829731346 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04037040 con 0x55c928a9fc40 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fddf4ff9640 1 -- 172.21.15.94:0/2829731346 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde04030ad0 con 0x55c928a9fc40 2024-03-06T07:27:27.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde04030510 con 0x55c928a9fc40 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fddf4ff9640 1 -- 172.21.15.94:0/2829731346 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=196) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde04068050 con 0x55c928a9fc40 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 10 client.10950 fetch_fsmap learned FSMap version 196 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=196,monmap=2+}) v3 -- 0x55c9286dc2d0 con 0x55c928a9fc40 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fddf4ff9640 1 -- 172.21.15.94:0/2829731346 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04030510 con 0x55c928a9fc40 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fddf4ff9640 1 -- 172.21.15.94:0/2829731346 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 196) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fde0406b020 con 0x55c928a9fc40 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 10 client.10950 fetch_fsmap finished waiting for FSMap version 196 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 10 client.10950 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 20 client.10950 populate_metadata read hostname 'smithi094' 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928ab5620 0x55c928ac0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 4 client.10950 mds_command: new command op to 38273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.336+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 --> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c9289d4bf0 con 0x55c928ab5620 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.337+0000 7fde19d16640 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928ab5620 0x55c928ac0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.337+0000 7fddf4ff9640 10 client.10950 ms_handle_connect on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:27.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.337+0000 7fde19d16640 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928ab5620 0x55c928ac0280 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=600160a174d9fc42 server_cookie=8389b512fd7962f6 in_seq=0 out_seq=0 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fddf4ff9640 1 -- 172.21.15.94:0/2829731346 <== mds.0 v2:172.21.15.145:6834/2585066812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c9289d4bf0 con 0x55c928ab5620 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fddf4ff9640 10 client.10950 handle_command_reply: tid=0 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 client.10950 shutdown 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 msgr2=0x55c928a9b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a9b110 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fde0402bdf0 tx=0x7fde04004730 comp rx=0 tx=0).stop 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 shutdown_connections 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9c1b0 0x55c928a83de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928ab5620 0x55c928ac0280 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9fc40 0x55c928a9b110 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 --2- 172.21.15.94:0/2829731346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 >> 172.21.15.94:0/2829731346 conn(0x55c928a63280 msgr2=0x55c928a815a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fddf4ff9640 0 client.10950 ms_handle_reset on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 shutdown_connections 2024-03-06T07:27:27.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 1 -- 172.21.15.94:0/2829731346 wait complete. 2024-03-06T07:27:27.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.338+0000 7fde1ec39740 20 client.10950 trim_cache size 0 max 16384 2024-03-06T07:27:27.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.354+0000 7fde1ec39740 1 Processor -- start 2024-03-06T07:27:27.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1ec39740 1 -- start start 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a9f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9fc40 0x55c928a82f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1ec39740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde141e7ec0 con 0x55c928a9c1b0 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde141e5130 con 0x55c928a9fc40 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde141e78c0 con 0x55c928a9e8f0 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a9f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a9f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47966/0 (socket says 172.21.15.94:47966) 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 -- 172.21.15.94:0/280198929 learned_addr learned my addr 172.21.15.94:0/280198929 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19515640 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde1a517640 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9fc40 0x55c928a82f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 -- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 msgr2=0x55c928a82a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 -- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9fc40 msgr2=0x55c928a82f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9fc40 0x55c928a82f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19d16640 1 -- 172.21.15.94:0/280198929 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9287a66b0 con 0x55c928a9c1b0 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.355+0000 7fde19515640 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde19d16640 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a9f410 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fde10054410 tx=0x7fde10004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e31d89b5c40e222 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fddf6ffd640 1 -- 172.21.15.94:0/280198929 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde100562b0 con 0x55c928a9c1b0 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fddf6ffd640 1 -- 172.21.15.94:0/280198929 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde10054620 con 0x55c928a9c1b0 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fddf6ffd640 1 -- 172.21.15.94:0/280198929 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10060380 con 0x55c928a9c1b0 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 -- 172.21.15.94:0/280198929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 msgr2=0x55c928a9f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.364 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a9f410 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fde10054410 tx=0x7fde10004f90 comp rx=0 tx=0).stop 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 -- 172.21.15.94:0/280198929 shutdown_connections 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9fc40 0x55c928a82f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9e8f0 0x55c928a82a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 --2- 172.21.15.94:0/280198929 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a9f410 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 -- 172.21.15.94:0/280198929 >> 172.21.15.94:0/280198929 conn(0x55c928a63280 msgr2=0x55c928a871f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 -- 172.21.15.94:0/280198929 shutdown_connections 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.356+0000 7fde1ec39740 1 -- 172.21.15.94:0/280198929 wait complete. 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.357+0000 7fde1ec39740 1 -- start start 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.357+0000 7fde1ec39740 1 Processor -- start 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a7e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928ab85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9fc40 0x55c928b12130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c928a82440 con 0x55c928a9c1b0 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c928a9b650 con 0x55c928a9e8f0 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c928ab25e0 con 0x55c928a9fc40 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 10 client.0 _ll_register_callbacks cb 0x7fde040096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a7e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a7e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47974/0 (socket says 172.21.15.94:47974) 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 -- 172.21.15.94:0/3252444730 learned_addr learned my addr 172.21.15.94:0/3252444730 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1a517640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9fc40 0x55c928b12130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19515640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928ab85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 -- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9fc40 msgr2=0x55c928b12130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9fc40 0x55c928b12130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 -- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 msgr2=0x55c928ab85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928ab85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 -- 172.21.15.94:0/3252444730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c928a5ce60 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1a517640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9fc40 0x55c928b12130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19515640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928ab85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde19d16640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a7e120 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7fde100012e0 tx=0x7fde10089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d121867ca12fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fddf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fddf4ff9640 1 -- 172.21.15.94:0/3252444730 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10087070 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fddf4ff9640 1 -- 172.21.15.94:0/3252444730 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde1005e2c0 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde10054620 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fddf4ff9640 1 -- 172.21.15.94:0/3252444730 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=196) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde10060180 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 10 client.10956 fetch_fsmap learned FSMap version 196 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.358+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=196,monmap=2+}) v3 -- 0x55c928aa1df0 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fddf4ff9640 1 -- 172.21.15.94:0/3252444730 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde10054620 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fddf4ff9640 1 -- 172.21.15.94:0/3252444730 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 196) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fde10057400 con 0x55c928a9c1b0 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 10 client.10956 fetch_fsmap finished waiting for FSMap version 196 2024-03-06T07:27:27.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 10 client.10956 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 20 client.10956 populate_metadata read hostname 'smithi094' 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928b0f340 0x55c928ac42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 4 client.10956 mds_command: new command op to 38273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 --> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c9289cef40 con 0x55c928b0f340 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde19515640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928b0f340 0x55c928ac42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde19515640 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928b0f340 0x55c928ac42a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48f04d018cdf5859 server_cookie=8bdf26609f26d469 in_seq=0 out_seq=0 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fddf4ff9640 10 client.10956 ms_handle_connect on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fddf4ff9640 1 -- 172.21.15.94:0/3252444730 <== mds.0 v2:172.21.15.145:6834/2585066812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55c9289cef40 con 0x55c928b0f340 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fddf4ff9640 10 client.10956 handle_command_reply: tid=0 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 client.10956 shutdown 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 msgr2=0x55c928a7e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a7e120 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7fde100012e0 tx=0x7fde10089040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 shutdown_connections 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c928a9fc40 0x55c928b12130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c928a9e8f0 0x55c928ab85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x55c928b0f340 0x55c928ac42a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 --2- 172.21.15.94:0/3252444730 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c928a9c1b0 0x55c928a7e120 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fddf4ff9640 0 client.10956 ms_handle_reset on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.359+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 >> 172.21.15.94:0/3252444730 conn(0x55c928a63280 msgr2=0x55c928a5d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 shutdown_connections 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1ec39740 1 -- 172.21.15.94:0/3252444730 wait complete. 2024-03-06T07:27:27.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1ec39740 20 client.10956 trim_cache size 0 max 16384 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 38273, 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 8.9616779599999994, 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 196, 2024-03-06T07:27:27.371 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 209, 2024-03-06T07:27:27.372 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 209, 2024-03-06T07:27:27.372 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 14.061181766000001 2024-03-06T07:27:27.372 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:27:27.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fddf806b4d0 msgr2=0x7fddf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fddf806b4d0 0x7fddf806d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fde04034000 tx=0x7fde04008030 comp rx=0 tx=0).stop 2024-03-06T07:27:27.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde140c6ab0 msgr2=0x7fde1414a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde140c6ab0 0x7fde1414a070 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c7c0 tx=0x7fde1000cc80 comp rx=0 tx=0).stop 2024-03-06T07:27:27.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 shutdown_connections 2024-03-06T07:27:27.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fddf806b4d0 0x7fddf806d990 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde141561a0 0x7fde14141370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fde1414f110 0x7fde141eac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 --2- 172.21.15.94:0/1945946732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde140c6ab0 0x7fde1414a070 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 >> 172.21.15.94:0/1945946732 conn(0x7fde1406b1a0 msgr2=0x7fde140a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 shutdown_connections 2024-03-06T07:27:27.374 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.360+0000 7fde1bf7a640 1 -- 172.21.15.94:0/1945946732 wait complete. 2024-03-06T07:27:27.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:27.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.474+0000 7f147cb93640 1 Processor -- start 2024-03-06T07:27:27.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.474+0000 7f147cb93640 1 -- start start 2024-03-06T07:27:27.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147cb93640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f147814f650 0x7f1478071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147cb93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14780c3150 0x7f14780c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147cb93640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1478072440 con 0x7f1478152c20 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147cb93640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14780c2b10 con 0x7f147814f650 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147cb93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1478152140 con 0x7f14780c3150 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f146ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14780c3150 0x7f14780c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147659c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f147814f650 0x7f1478071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f146ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14780c3150 0x7f14780c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:48502/0 (socket says 172.21.15.94:48502) 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f146ffff640 1 -- 172.21.15.94:0/2684122370 learned_addr learned my addr 172.21.15.94:0/2684122370 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147659c640 1 -- 172.21.15.94:0/2684122370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14780c3150 msgr2=0x7f14780c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.475+0000 7f147659c640 1 --2- 172.21.15.94:0/2684122370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14780c3150 0x7f14780c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.476+0000 7f147659c640 1 -- 172.21.15.94:0/2684122370 >> v1:172.21.15.145:6789/0 conn(0x7f1478152c20 legacy=0x7f14781554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.476+0000 7f147659c640 1 -- 172.21.15.94:0/2684122370 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478152420 con 0x7f147814f650 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.476+0000 7f147659c640 1 --2- 172.21.15.94:0/2684122370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f147814f650 0x7f1478071880 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f14680097a0 tx=0x7f146802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa97953f81347609 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.476+0000 7f1475d9b640 1 -- 172.21.15.94:0/2684122370 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468035020 con 0x7f147814f650 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.476+0000 7f1475d9b640 1 -- 172.21.15.94:0/2684122370 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468031830 con 0x7f147814f650 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.476+0000 7f1475d9b640 1 -- 172.21.15.94:0/2684122370 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468031c00 con 0x7f147814f650 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 -- 172.21.15.94:0/2684122370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f147814f650 msgr2=0x7f1478071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 --2- 172.21.15.94:0/2684122370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f147814f650 0x7f1478071880 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f14680097a0 tx=0x7f146802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 -- 172.21.15.94:0/2684122370 shutdown_connections 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 --2- 172.21.15.94:0/2684122370 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14780c3150 0x7f14780c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 --2- 172.21.15.94:0/2684122370 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f147814f650 0x7f1478071880 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 -- 172.21.15.94:0/2684122370 >> 172.21.15.94:0/2684122370 conn(0x7f14780c4290 msgr2=0x7f14780c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 -- 172.21.15.94:0/2684122370 shutdown_connections 2024-03-06T07:27:27.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.477+0000 7f147cb93640 1 -- 172.21.15.94:0/2684122370 wait complete. 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.478+0000 7f147cb93640 1 Processor -- start 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.478+0000 7f147cb93640 1 -- start start 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147cb93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14780c3150 0x7f14781ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147cb93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f147814f650 0x7f14781dfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147cb93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1478152c20 0x7f1478153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147cb93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780c40c0 con 0x7f14780c3150 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147cb93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780c3f40 con 0x7f1478152c20 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147cb93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1478071dc0 con 0x7f147814f650 2024-03-06T07:27:27.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14780c3150 0x7f14781ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14780c3150 0x7f14781ee150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47980/0 (socket says 172.21.15.94:47980) 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 -- 172.21.15.94:0/637769259 learned_addr learned my addr 172.21.15.94:0/637769259 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f1476d9d640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1478152c20 0x7f1478153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f146ffff640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f147814f650 0x7f14781dfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 -- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f147814f650 msgr2=0x7f14781dfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f147814f650 0x7f14781dfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 -- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1478152c20 msgr2=0x7f1478153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1478152c20 0x7f1478153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.479+0000 7f147659c640 1 -- 172.21.15.94:0/637769259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478155360 con 0x7f14780c3150 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.480+0000 7f147659c640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14780c3150 0x7f14781ee150 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f146802bee0 tx=0x7f14680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.480+0000 7f146effd640 1 -- 172.21.15.94:0/637769259 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468035040 con 0x7f14780c3150 2024-03-06T07:27:27.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.480+0000 7f146effd640 1 -- 172.21.15.94:0/637769259 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146803b920 con 0x7f14780c3150 2024-03-06T07:27:27.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.480+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1478154da0 con 0x7f14780c3150 2024-03-06T07:27:27.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.480+0000 7f146effd640 1 -- 172.21.15.94:0/637769259 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146803bd10 con 0x7f14780c3150 2024-03-06T07:27:27.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.480+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14781537a0 con 0x7f14780c3150 2024-03-06T07:27:27.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.481+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147804f330 con 0x7f14780c3150 2024-03-06T07:27:27.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.481+0000 7f146effd640 1 -- 172.21.15.94:0/637769259 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1468038020 con 0x7f14780c3150 2024-03-06T07:27:27.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.482+0000 7f146effd640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f145806b4d0 0x7f145806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.482+0000 7f146ffff640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f145806b4d0 0x7f145806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.482+0000 7f146effd640 1 -- 172.21.15.94:0/637769259 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f14680ac380 con 0x7f14780c3150 2024-03-06T07:27:27.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.482+0000 7f146ffff640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f145806b4d0 0x7f145806d990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f14780a7090 tx=0x7f1460009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.486+0000 7f146effd640 1 -- 172.21.15.94:0/637769259 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1468031800 con 0x7f14780c3150 2024-03-06T07:27:27.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.692+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1478068460 con 0x7f14780c3150 2024-03-06T07:27:27.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.692+0000 7f146effd640 1 -- 172.21.15.94:0/637769259 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+4995 (secure 0 0 0) 0x7f146807aa30 con 0x7f14780c3150 2024-03-06T07:27:27.694 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:27.694 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":196,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:19.757407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":38273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:27.694 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 196 2024-03-06T07:27:27.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f145806b4d0 msgr2=0x7f145806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f145806b4d0 0x7f145806d990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f14780a7090 tx=0x7f1460009240 comp rx=0 tx=0).stop 2024-03-06T07:27:27.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14780c3150 msgr2=0x7f14781ee150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14780c3150 0x7f14781ee150 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f146802bee0 tx=0x7f14680046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 shutdown_connections 2024-03-06T07:27:27.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f145806b4d0 0x7f145806d990 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1478152c20 0x7f1478153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f147814f650 0x7f14781dfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 --2- 172.21.15.94:0/637769259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14780c3150 0x7f14781ee150 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 >> 172.21.15.94:0/637769259 conn(0x7f14780c4290 msgr2=0x7f14780c5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 shutdown_connections 2024-03-06T07:27:27.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.694+0000 7f147cb93640 1 -- 172.21.15.94:0/637769259 wait complete. 2024-03-06T07:27:27.707 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:27:27.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.808+0000 7fad9fa55640 1 Processor -- start 2024-03-06T07:27:27.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.809+0000 7fad9fa55640 1 -- start start 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.809+0000 7fad9fa55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad9814d350 0x7fad9806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.809+0000 7fad9fa55640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad980acbc0 0x7fad980ab950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.809+0000 7fad9fa55640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad980c1c90 con 0x7fad980abe90 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.809+0000 7fad9fa55640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad980ac750 con 0x7fad980acbc0 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.809+0000 7fad9fa55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980c1a40 con 0x7fad9814d350 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9cff0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad980acbc0 0x7fad980ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9dff2640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fad980abe90 0x7fad98156540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42884/0 (socket says 172.21.15.94:42884) 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9dff2640 1 -- 172.21.15.94:0/3215095460 learned_addr learned my addr 172.21.15.94:0/3215095460 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9d7f1640 1 --2- 172.21.15.94:0/3215095460 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad9814d350 0x7fad9806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9cff0640 1 -- 172.21.15.94:0/3215095460 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad9814d350 msgr2=0x7fad9806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9cff0640 1 --2- 172.21.15.94:0/3215095460 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad9814d350 0x7fad9806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9cff0640 1 -- 172.21.15.94:0/3215095460 >> v1:172.21.15.94:6789/0 conn(0x7fad980abe90 legacy=0x7fad98156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad97fff640 1 -- 172.21.15.94:0/3215095460 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746733651 0 0) 0x7fad980c1c90 con 0x7fad980abe90 2024-03-06T07:27:27.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.810+0000 7fad9cff0640 1 -- 172.21.15.94:0/3215095460 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980ac470 con 0x7fad980acbc0 2024-03-06T07:27:27.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9cff0640 1 --2- 172.21.15.94:0/3215095460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad980acbc0 0x7fad980ab950 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fad8c0097a0 tx=0x7fad8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cc7f5cc04c2a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad97fff640 1 -- 172.21.15.94:0/3215095460 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c0315a0 con 0x7fad980acbc0 2024-03-06T07:27:27.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad97fff640 1 -- 172.21.15.94:0/3215095460 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c037070 con 0x7fad980acbc0 2024-03-06T07:27:27.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad97fff640 1 -- 172.21.15.94:0/3215095460 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c031960 con 0x7fad980acbc0 2024-03-06T07:27:27.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 -- 172.21.15.94:0/3215095460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad980acbc0 msgr2=0x7fad980ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 --2- 172.21.15.94:0/3215095460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad980acbc0 0x7fad980ab950 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fad8c0097a0 tx=0x7fad8c02b840 comp rx=0 tx=0).stop 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 -- 172.21.15.94:0/3215095460 shutdown_connections 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 --2- 172.21.15.94:0/3215095460 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad980acbc0 0x7fad980ab950 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 --2- 172.21.15.94:0/3215095460 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad9814d350 0x7fad9806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 -- 172.21.15.94:0/3215095460 >> 172.21.15.94:0/3215095460 conn(0x7fad980c3b40 msgr2=0x7fad980c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 -- 172.21.15.94:0/3215095460 shutdown_connections 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.811+0000 7fad9fa55640 1 -- 172.21.15.94:0/3215095460 wait complete. 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.812+0000 7fad9fa55640 1 Processor -- start 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9fa55640 1 -- start start 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9fa55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad980abe90 0x7fad980c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9fa55640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad980acbc0 0x7fad9814a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9fa55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad9814d350 0x7fad9814a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9fa55640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980aa160 con 0x7fad9814d350 2024-03-06T07:27:27.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9fa55640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad98153c00 con 0x7fad980acbc0 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9fa55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980aa2e0 con 0x7fad980abe90 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9dff2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad9814d350 0x7fad9814a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9dff2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad9814d350 0x7fad9814a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:48000/0 (socket says 172.21.15.94:48000) 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9dff2640 1 -- 172.21.15.94:0/826857105 learned_addr learned my addr 172.21.15.94:0/826857105 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9d7f1640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad980abe90 0x7fad980c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.813+0000 7fad9cff0640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad980acbc0 0x7fad9814a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9dff2640 1 -- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad980abe90 msgr2=0x7fad980c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9dff2640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad980abe90 0x7fad980c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9dff2640 1 -- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad980acbc0 msgr2=0x7fad9814a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9dff2640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad980acbc0 0x7fad9814a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9dff2640 1 -- 172.21.15.94:0/826857105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad98146a80 con 0x7fad9814d350 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9dff2640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad9814d350 0x7fad9814a670 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fad8800c450 tx=0x7fad8800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad95ffb640 1 -- 172.21.15.94:0/826857105 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88018070 con 0x7fad9814d350 2024-03-06T07:27:27.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad95ffb640 1 -- 172.21.15.94:0/826857105 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad88007500 con 0x7fad9814d350 2024-03-06T07:27:27.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad981ecaf0 con 0x7fad9814d350 2024-03-06T07:27:27.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad95ffb640 1 -- 172.21.15.94:0/826857105 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88004a20 con 0x7fad9814d350 2024-03-06T07:27:27.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.814+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad980c69a0 con 0x7fad9814d350 2024-03-06T07:27:27.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.815+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9804f330 con 0x7fad9814d350 2024-03-06T07:27:27.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.815+0000 7fad95ffb640 1 -- 172.21.15.94:0/826857105 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad8801b070 con 0x7fad9814d350 2024-03-06T07:27:27.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.816+0000 7fad95ffb640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad8406b4d0 0x7fad8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.816+0000 7fad95ffb640 1 -- 172.21.15.94:0/826857105 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad880894c0 con 0x7fad9814d350 2024-03-06T07:27:27.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.816+0000 7fad9d7f1640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad8406b4d0 0x7fad8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.816+0000 7fad9d7f1640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad8406b4d0 0x7fad8406d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fad90011340 tx=0x7fad9000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:27.820+0000 7fad95ffb640 1 -- 172.21.15.94:0/826857105 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad88057bf0 con 0x7fad9814d350 2024-03-06T07:27:28.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.030+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fad980c49e0 con 0x7fad9814d350 2024-03-06T07:27:28.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.477+0000 7fad95ffb640 1 -- 172.21.15.94:0/826857105 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v197) v1 ==== 105+0+0 (secure 0 0 0) 0x7fad8805bc90 con 0x7fad9814d350 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.479+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad8406b4d0 msgr2=0x7fad8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.479+0000 7fad9fa55640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad8406b4d0 0x7fad8406d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fad90011340 tx=0x7fad9000b040 comp rx=0 tx=0).stop 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.479+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad9814d350 msgr2=0x7fad9814a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.479+0000 7fad9fa55640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad9814d350 0x7fad9814a670 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fad8800c450 tx=0x7fad8800c910 comp rx=0 tx=0).stop 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 shutdown_connections 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad8406b4d0 0x7fad8406d990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad9814d350 0x7fad9814a670 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad980acbc0 0x7fad9814a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 --2- 172.21.15.94:0/826857105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad980abe90 0x7fad980c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 >> 172.21.15.94:0/826857105 conn(0x7fad980c3b40 msgr2=0x7fad980c2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 shutdown_connections 2024-03-06T07:27:28.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.480+0000 7fad9fa55640 1 -- 172.21.15.94:0/826857105 wait complete. 2024-03-06T07:27:28.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:28.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.596+0000 7f7768078640 1 Processor -- start 2024-03-06T07:27:28.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.596+0000 7f7768078640 1 -- start start 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7768078640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f77601550c0 0x7f77601554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7768078640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601559e0 0x7f776014ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7768078640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77600aa660 con 0x7f77600aa180 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7768078640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77600c4610 con 0x7f77601550c0 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7768078640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77600aa940 con 0x7f77601559e0 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7766615640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f77600aa180 0x7f77600c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56852/0 (socket says 172.21.15.94:56852) 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7766615640 1 -- 172.21.15.94:0/1358739408 learned_addr learned my addr 172.21.15.94:0/1358739408 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7765e14640 1 --2- 172.21.15.94:0/1358739408 >> v2:172.21.15.145:3300/0 conn(0x7f77601550c0 0x7f77601554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7765613640 1 --2- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601559e0 0x7f776014ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7764e12640 1 -- 172.21.15.94:0/1358739408 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174383439 0 0) 0x7f77600aa660 con 0x7f77600aa180 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7764e12640 1 -- 172.21.15.94:0/1358739408 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f774c003520 con 0x7f77600aa180 2024-03-06T07:27:28.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7765e14640 1 -- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601559e0 msgr2=0x7f776014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7765e14640 1 --2- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601559e0 0x7f776014ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7765e14640 1 -- 172.21.15.94:0/1358739408 >> v1:172.21.15.94:6789/0 conn(0x7f77600aa180 legacy=0x7f77600c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7765e14640 1 -- 172.21.15.94:0/1358739408 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600c4860 con 0x7f77601550c0 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.597+0000 7f7765613640 1 --2- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601559e0 0x7f776014ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.598+0000 7f7765e14640 1 --2- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601550c0 0x7f77601554a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f775c0097a0 tx=0x7f775c02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80de698d90503b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.598+0000 7f7764e12640 1 -- 172.21.15.94:0/1358739408 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c038070 con 0x7f77601550c0 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.598+0000 7f7764e12640 1 -- 172.21.15.94:0/1358739408 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775c031700 con 0x7f77601550c0 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.598+0000 7f7764e12640 1 -- 172.21.15.94:0/1358739408 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c031ab0 con 0x7f77601550c0 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 -- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601550c0 msgr2=0x7f77601554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 --2- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601550c0 0x7f77601554a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f775c0097a0 tx=0x7f775c02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 -- 172.21.15.94:0/1358739408 shutdown_connections 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 --2- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601559e0 0x7f776014ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 --2- 172.21.15.94:0/1358739408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601550c0 0x7f77601554a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 -- 172.21.15.94:0/1358739408 >> 172.21.15.94:0/1358739408 conn(0x7f77600c6710 msgr2=0x7f77600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 -- 172.21.15.94:0/1358739408 shutdown_connections 2024-03-06T07:27:28.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.599+0000 7f7768078640 1 -- 172.21.15.94:0/1358739408 wait complete. 2024-03-06T07:27:28.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 Processor -- start 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 -- start start 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77600aa180 0x7f77600702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601550c0 0x7f77601401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601559e0 0x7f77601475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776014f140 con 0x7f77600aa180 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77600c1520 con 0x7f77601559e0 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7768078640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776014f2c0 con 0x7f77601550c0 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.600+0000 7f7765e14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77600aa180 0x7f77600702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77600aa180 0x7f77600702f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35806/0 (socket says 172.21.15.94:35806) 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 -- 172.21.15.94:0/4019426790 learned_addr learned my addr 172.21.15.94:0/4019426790 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:28.606 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765613640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601550c0 0x7f77601401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7766615640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601559e0 0x7f77601475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 -- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601550c0 msgr2=0x7f77601401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601550c0 0x7f77601401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 -- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601559e0 msgr2=0x7f77601475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601559e0 0x7f77601475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 -- 172.21.15.94:0/4019426790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600a6a30 con 0x7f77600aa180 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7765e14640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77600aa180 0x7f77600702f0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f775c000c00 tx=0x7f775c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f77567fc640 1 -- 172.21.15.94:0/4019426790 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c042070 con 0x7f77600aa180 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f77567fc640 1 -- 172.21.15.94:0/4019426790 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775c03d870 con 0x7f77600aa180 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77600a8d70 con 0x7f77600aa180 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f77567fc640 1 -- 172.21.15.94:0/4019426790 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c038040 con 0x7f77600aa180 2024-03-06T07:27:28.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.601+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7760146da0 con 0x7f77600aa180 2024-03-06T07:27:28.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.602+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7760067ad0 con 0x7f77600aa180 2024-03-06T07:27:28.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.602+0000 7f77567fc640 1 -- 172.21.15.94:0/4019426790 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f775c045070 con 0x7f77600aa180 2024-03-06T07:27:28.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.603+0000 7f77567fc640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774406b4d0 0x7f774406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.603+0000 7f77567fc640 1 -- 172.21.15.94:0/4019426790 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f775c0abf10 con 0x7f77600aa180 2024-03-06T07:27:28.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.603+0000 7f7765613640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774406b4d0 0x7f774406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.603+0000 7f7765613640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774406b4d0 0x7f774406d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f77601ed360 tx=0x7f7750009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.607+0000 7f77567fc640 1 -- 172.21.15.94:0/4019426790 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7760067ad0 con 0x7f77600aa180 2024-03-06T07:27:28.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.814+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f776004f330 con 0x7f77600aa180 2024-03-06T07:27:28.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.815+0000 7f77567fc640 1 -- 172.21.15.94:0/4019426790 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f775c030070 con 0x7f77600aa180 2024-03-06T07:27:28.817 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:28.817 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":198,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:28.480137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:creating","state_seq":1,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:28.817 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 198 2024-03-06T07:27:28.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774406b4d0 msgr2=0x7f774406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774406b4d0 0x7f774406d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f77601ed360 tx=0x7f7750009190 comp rx=0 tx=0).stop 2024-03-06T07:27:28.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77600aa180 msgr2=0x7f77600702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77600aa180 0x7f77600702f0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f775c000c00 tx=0x7f775c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 shutdown_connections 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f774406b4d0 0x7f774406d990 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77601559e0 0x7f77601475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77601550c0 0x7f77601401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 --2- 172.21.15.94:0/4019426790 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f77600aa180 0x7f77600702f0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 >> 172.21.15.94:0/4019426790 conn(0x7f77600c6710 msgr2=0x7f77600c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 shutdown_connections 2024-03-06T07:27:28.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:28.817+0000 7f7768078640 1 -- 172.21.15.94:0/4019426790 wait complete. 2024-03-06T07:27:29.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 198, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:28.480137+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:29.232 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:creating 2024-03-06T07:27:30.234 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:30.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.344+0000 7fad4cd8a640 1 Processor -- start 2024-03-06T07:27:30.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.344+0000 7fad4cd8a640 1 -- start start 2024-03-06T07:27:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad4cd8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad4806c720 0x7fad48071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad4cd8a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fad480c2410 0x7fad48071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad4cd8a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad48067340 con 0x7fad48150920 2024-03-06T07:27:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad4cd8a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad48072400 con 0x7fad4806c720 2024-03-06T07:27:30.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad4cd8a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad48150d00 con 0x7fad480c2410 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad45d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad4806c720 0x7fad48071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad45d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad4806c720 0x7fad48071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35810/0 (socket says 172.21.15.94:35810) 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad45d9b640 1 -- 172.21.15.94:0/541587023 learned_addr learned my addr 172.21.15.94:0/541587023 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad46d9d640 1 --2- 172.21.15.94:0/541587023 >> v2:172.21.15.145:3300/0 conn(0x7fad480c2410 0x7fad48071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad45d9b640 1 -- 172.21.15.94:0/541587023 >> v2:172.21.15.145:3300/0 conn(0x7fad480c2410 msgr2=0x7fad48071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad45d9b640 1 --2- 172.21.15.94:0/541587023 >> v2:172.21.15.145:3300/0 conn(0x7fad480c2410 0x7fad48071ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad45d9b640 1 -- 172.21.15.94:0/541587023 >> v1:172.21.15.145:6789/0 conn(0x7fad48150920 legacy=0x7fad480c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.345+0000 7fad45d9b640 1 -- 172.21.15.94:0/541587023 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad480c2010 con 0x7fad4806c720 2024-03-06T07:27:30.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.346+0000 7fad45d9b640 1 --2- 172.21.15.94:0/541587023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad4806c720 0x7fad48071980 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fad481543f0 tx=0x7fad3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954a909e4f914dff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.346+0000 7fad4559a640 1 -- 172.21.15.94:0/541587023 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c035020 con 0x7fad4806c720 2024-03-06T07:27:30.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.346+0000 7fad4559a640 1 -- 172.21.15.94:0/541587023 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c031870 con 0x7fad4806c720 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.346+0000 7fad4559a640 1 -- 172.21.15.94:0/541587023 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c031c40 con 0x7fad4806c720 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 -- 172.21.15.94:0/541587023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad4806c720 msgr2=0x7fad48071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/541587023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad4806c720 0x7fad48071980 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fad481543f0 tx=0x7fad3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 -- 172.21.15.94:0/541587023 shutdown_connections 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/541587023 >> v2:172.21.15.145:3300/0 conn(0x7fad480c2410 0x7fad48071ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/541587023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad4806c720 0x7fad48071980 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 -- 172.21.15.94:0/541587023 >> 172.21.15.94:0/541587023 conn(0x7fad480c70b0 msgr2=0x7fad480c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 -- 172.21.15.94:0/541587023 shutdown_connections 2024-03-06T07:27:30.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.347+0000 7fad4cd8a640 1 -- 172.21.15.94:0/541587023 wait complete. 2024-03-06T07:27:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.348+0000 7fad4cd8a640 1 Processor -- start 2024-03-06T07:27:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.348+0000 7fad4cd8a640 1 -- start start 2024-03-06T07:27:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad4cd8a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4806c720 0x7fad480bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad4cd8a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad480c2410 0x7fad480c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad4cd8a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad48150920 0x7fad48144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad4cd8a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad480c3040 con 0x7fad48150920 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad4cd8a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad48072b70 con 0x7fad4806c720 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad4cd8a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad480c31c0 con 0x7fad480c2410 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad48150920 0x7fad48144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad48150920 0x7fad48144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35822/0 (socket says 172.21.15.94:35822) 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 -- 172.21.15.94:0/976054191 learned_addr learned my addr 172.21.15.94:0/976054191 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad4659c640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4806c720 0x7fad480bc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad45d9b640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad480c2410 0x7fad480c0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 -- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad480c2410 msgr2=0x7fad480c0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad480c2410 0x7fad480c0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 -- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4806c720 msgr2=0x7fad480bc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4806c720 0x7fad480bc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 -- 172.21.15.94:0/976054191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48157630 con 0x7fad48150920 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad45d9b640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad480c2410 0x7fad480c0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.349+0000 7fad46d9d640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad48150920 0x7fad48144e80 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fad4000efc0 tx=0x7fad4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.350+0000 7fad36ffd640 1 -- 172.21.15.94:0/976054191 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40018020 con 0x7fad48150920 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.350+0000 7fad36ffd640 1 -- 172.21.15.94:0/976054191 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40014480 con 0x7fad48150920 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.350+0000 7fad36ffd640 1 -- 172.21.15.94:0/976054191 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40014870 con 0x7fad48150920 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.350+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad480bcb10 con 0x7fad48150920 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.350+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4814d110 con 0x7fad48150920 2024-03-06T07:27:30.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.351+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4804f330 con 0x7fad48150920 2024-03-06T07:27:30.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.351+0000 7fad36ffd640 1 -- 172.21.15.94:0/976054191 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad40005d10 con 0x7fad48150920 2024-03-06T07:27:30.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.352+0000 7fad36ffd640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2806b4d0 0x7fad2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.352+0000 7fad36ffd640 1 -- 172.21.15.94:0/976054191 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad4008ad30 con 0x7fad48150920 2024-03-06T07:27:30.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.352+0000 7fad4659c640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2806b4d0 0x7fad2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.352+0000 7fad4659c640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2806b4d0 0x7fad2806d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fad380016e0 tx=0x7fad38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.356+0000 7fad36ffd640 1 -- 172.21.15.94:0/976054191 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad40022070 con 0x7fad48150920 2024-03-06T07:27:30.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.565+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad4806eca0 con 0x7fad48150920 2024-03-06T07:27:30.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.566+0000 7fad36ffd640 1 -- 172.21.15.94:0/976054191 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5017 (secure 0 0 0) 0x7fad40010040 con 0x7fad48150920 2024-03-06T07:27:30.568 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:30.568 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":199,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:29.482340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:active","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[10791,38273]},"id":16}]} 2024-03-06T07:27:30.568 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 199 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.567+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2806b4d0 msgr2=0x7fad2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.567+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2806b4d0 0x7fad2806d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fad380016e0 tx=0x7fad38009040 comp rx=0 tx=0).stop 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.567+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad48150920 msgr2=0x7fad48144e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.567+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad48150920 0x7fad48144e80 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fad4000efc0 tx=0x7fad4000c490 comp rx=0 tx=0).stop 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 shutdown_connections 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fad2806b4d0 0x7fad2806d990 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad48150920 0x7fad48144e80 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad480c2410 0x7fad480c0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 --2- 172.21.15.94:0/976054191 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4806c720 0x7fad480bc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 >> 172.21.15.94:0/976054191 conn(0x7fad480c70b0 msgr2=0x7fad480c7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 shutdown_connections 2024-03-06T07:27:30.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:30.568+0000 7fad4cd8a640 1 -- 172.21.15.94:0/976054191 wait complete. 2024-03-06T07:27:30.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 199, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:29.482340+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [10791, 38273]} 2024-03-06T07:27:30.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-03-06T07:27:30.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:27:31.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.086+0000 7f8ffa42a640 1 Processor -- start 2024-03-06T07:27:31.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.086+0000 7f8ffa42a640 1 -- start start 2024-03-06T07:27:31.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.086+0000 7f8ffa42a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff41561a0 0x7f8ff41529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.086+0000 7f8ffa42a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40a9990 con 0x7f8ff414f110 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.086+0000 7f8ffa42a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff41556c0 con 0x7f8ff40c6ab0 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.086+0000 7f8ffa42a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4155d10 con 0x7f8ff41561a0 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff89c7640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8ff414f110 0x7f8ff415c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56866/0 (socket says 172.21.15.94:56866) 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff89c7640 1 -- 172.21.15.94:0/1646159344 learned_addr learned my addr 172.21.15.94:0/1646159344 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/1646159344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff41561a0 0x7f8ff41529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff2ffd640 1 -- 172.21.15.94:0/1646159344 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921721435 0 0) 0x7f8ff40a9990 con 0x7f8ff414f110 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff2ffd640 1 -- 172.21.15.94:0/1646159344 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc003520 con 0x7f8ff414f110 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff2ffd640 1 -- 172.21.15.94:0/1646159344 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914519644 0 0) 0x7f8ff41556c0 con 0x7f8ff40c6ab0 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff2ffd640 1 -- 172.21.15.94:0/1646159344 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff40a9990 con 0x7f8ff40c6ab0 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff37fe640 1 -- 172.21.15.94:0/1646159344 >> v1:172.21.15.145:6789/0 conn(0x7f8ff40c6ab0 legacy=0x7f8ff40a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff37fe640 1 -- 172.21.15.94:0/1646159344 >> v1:172.21.15.94:6789/0 conn(0x7f8ff414f110 legacy=0x7f8ff415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.087+0000 7f8ff37fe640 1 -- 172.21.15.94:0/1646159344 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff41559a0 con 0x7f8ff41561a0 2024-03-06T07:27:31.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/1646159344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff41561a0 0x7f8ff41529e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80097d0 tx=0x7f8fe802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efbc84d7c47cb67c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ff2ffd640 1 -- 172.21.15.94:0/1646159344 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8035020 con 0x7f8ff41561a0 2024-03-06T07:27:31.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ff2ffd640 1 -- 172.21.15.94:0/1646159344 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe8031800 con 0x7f8ff41561a0 2024-03-06T07:27:31.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ff2ffd640 1 -- 172.21.15.94:0/1646159344 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8031bb0 con 0x7f8ff41561a0 2024-03-06T07:27:31.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ffa42a640 1 -- 172.21.15.94:0/1646159344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff41561a0 msgr2=0x7f8ff41529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/1646159344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff41561a0 0x7f8ff41529e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80097d0 tx=0x7f8fe802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ffa42a640 1 -- 172.21.15.94:0/1646159344 shutdown_connections 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/1646159344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff41561a0 0x7f8ff41529e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ffa42a640 1 -- 172.21.15.94:0/1646159344 >> 172.21.15.94:0/1646159344 conn(0x7f8ff406b1a0 msgr2=0x7f8ff406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ffa42a640 1 -- 172.21.15.94:0/1646159344 shutdown_connections 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.088+0000 7f8ffa42a640 1 -- 172.21.15.94:0/1646159344 wait complete. 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.089+0000 7f8ffa42a640 1 Processor -- start 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ffa42a640 1 -- start start 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ffa42a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff40c6ab0 0x7f8ff40c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ffa42a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff414f110 0x7f8ff4151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ffa42a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff41561a0 0x7f8ff41ea9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ffa42a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40aa600 con 0x7f8ff414f110 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ffa42a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40c2a90 con 0x7f8ff41561a0 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ffa42a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff414ebc0 con 0x7f8ff40c6ab0 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff414f110 0x7f8ff4151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff414f110 0x7f8ff4151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35836/0 (socket says 172.21.15.94:35836) 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ff37fe640 1 -- 172.21.15.94:0/4142517503 learned_addr learned my addr 172.21.15.94:0/4142517503 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff40c6ab0 0x7f8ff40c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.090+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff41561a0 0x7f8ff41ea9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff37fe640 1 -- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff40c6ab0 msgr2=0x7f8ff40c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff40c6ab0 0x7f8ff40c5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff37fe640 1 -- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff41561a0 msgr2=0x7f8ff41ea9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff41561a0 0x7f8ff41ea9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff37fe640 1 -- 172.21.15.94:0/4142517503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff41ee140 con 0x7f8ff414f110 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff414f110 0x7f8ff4151530 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff0ff9640 1 -- 172.21.15.94:0/4142517503 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8041070 con 0x7f8ff414f110 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff0ff9640 1 -- 172.21.15.94:0/4142517503 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe8031e30 con 0x7f8ff414f110 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff41ee420 con 0x7f8ff414f110 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ff0ff9640 1 -- 172.21.15.94:0/4142517503 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8035040 con 0x7f8ff414f110 2024-03-06T07:27:31.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.091+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff41ee9e0 con 0x7f8ff414f110 2024-03-06T07:27:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.093+0000 7f8ff0ff9640 1 -- 172.21.15.94:0/4142517503 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fe8038020 con 0x7f8ff414f110 2024-03-06T07:27:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.093+0000 7f8ff0ff9640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe006b500 0x7f8fe006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.093+0000 7f8ff0ff9640 1 -- 172.21.15.94:0/4142517503 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fe80abe00 con 0x7f8ff414f110 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.093+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe006b500 0x7f8fe006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.093+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe006b500 0x7f8fe006d9c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4007fd0 tx=0x7f8fe4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 Processor -- start 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 -- start start 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6bfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c04b0 0x55c3ff6bc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff6266b0 con 0x55c3ff6c2330 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff5cfa60 con 0x55c3ff6c04b0 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.094+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff622750 con 0x55c3ff6c1f50 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35852/0 (socket says 172.21.15.94:35852) 2024-03-06T07:27:31.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 -- 172.21.15.94:0/1209854578 learned_addr learned my addr 172.21.15.94:0/1209854578 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6bfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c04b0 0x55c3ff6bc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 -- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 msgr2=0x55c3ff6bfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6bfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 -- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c04b0 msgr2=0x55c3ff6bc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c04b0 0x55c3ff6bc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 -- 172.21.15.94:0/1209854578 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3ff5edf30 con 0x55c3ff6c2330 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bff70 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8076880 tx=0x7f8fe8079d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d61222f12b7aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8fd9ffb640 1 -- 172.21.15.94:0/1209854578 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8082a80 con 0x55c3ff6c2330 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8fd9ffb640 1 -- 172.21.15.94:0/1209854578 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe8078720 con 0x55c3ff6c2330 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.095+0000 7f8fd9ffb640 1 -- 172.21.15.94:0/1209854578 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe807e0d0 con 0x55c3ff6c2330 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 -- 172.21.15.94:0/1209854578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 msgr2=0x55c3ff6bff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bff70 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8076880 tx=0x7f8fe8079d80 comp rx=0 tx=0).stop 2024-03-06T07:27:31.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 -- 172.21.15.94:0/1209854578 shutdown_connections 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c04b0 0x55c3ff6bc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bff70 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1209854578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6bfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 -- 172.21.15.94:0/1209854578 >> 172.21.15.94:0/1209854578 conn(0x55c3ff6be7d0 msgr2=0x55c3ff6e33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 -- 172.21.15.94:0/1209854578 shutdown_connections 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 -- 172.21.15.94:0/1209854578 wait complete. 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 -- start start 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.096+0000 7f8ffd010740 1 Processor -- start 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c1f50 0x55c3ff6e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6e1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff70ee40 0x55c3ff6fa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ffd010740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff6e4230 con 0x55c3ff6c2330 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff5e2ea0 con 0x55c3ff6c1f50 2024-03-06T07:27:31.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff6e3c70 con 0x55c3ff70ee40 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ffd010740 10 client.0 _ll_register_callbacks cb 0x55c3ff5e2870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6e1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6e1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35862/0 (socket says 172.21.15.94:35862) 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ff37fe640 1 -- 172.21.15.94:0/2999220048 learned_addr learned my addr 172.21.15.94:0/2999220048 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c1f50 0x55c3ff6e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.097+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff70ee40 0x55c3ff6fa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ff37fe640 1 -- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff70ee40 msgr2=0x55c3ff6fa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff70ee40 0x55c3ff6fa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ff37fe640 1 -- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c1f50 msgr2=0x55c3ff6e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c1f50 0x55c3ff6e4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ff37fe640 1 -- 172.21.15.94:0/2999220048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3ff5e3270 con 0x55c3ff6c2330 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6e1480 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80097d0 tx=0x7f8fe8006c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a7623bc7a98179 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8fd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/2999220048 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8097d40 con 0x55c3ff6c2330 2024-03-06T07:27:31.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/2999220048 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fe8079870 con 0x55c3ff6c2330 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fe8078720 con 0x55c3ff6c2330 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/2999220048 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fe80a1050 con 0x55c3ff6c2330 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 10 client.10992 fetch_fsmap learned FSMap version 199 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x55c3ff33b2d0 con 0x55c3ff6c2330 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/2999220048 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe8078720 con 0x55c3ff6c2330 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/2999220048 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f8fe804c070 con 0x55c3ff6c2330 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 10 client.10992 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 10 client.10992 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 20 client.10992 populate_metadata read hostname 'smithi094' 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6dde40 0x55c3ff6de220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 4 client.10992 mds_command: new command op to 10791 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 --> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3ff633bf0 con 0x55c3ff6dde40 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.098+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6dde40 0x55c3ff6de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.099+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6dde40 0x55c3ff6de220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4942eb19eba79543 server_cookie=7943b75c34691adf in_seq=0 out_seq=0 2024-03-06T07:27:31.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.099+0000 7f8fd2ffd640 10 client.10992 ms_handle_connect on v2:172.21.15.145:6835/679350833 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/2999220048 <== mds.1 v2:172.21.15.145:6835/679350833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c3ff633bf0 con 0x55c3ff6dde40 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8fd2ffd640 10 client.10992 handle_command_reply: tid=0 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 client.10992 shutdown 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 msgr2=0x55c3ff6e1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6e1480 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80097d0 tx=0x7f8fe8006c30 comp rx=0 tx=0).stop 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 shutdown_connections 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6dde40 0x55c3ff6de220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff70ee40 0x55c3ff6fa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff6c2330 0x55c3ff6e1480 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 --2- 172.21.15.94:0/2999220048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c1f50 0x55c3ff6e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8fd2ffd640 0 client.10992 ms_handle_reset on v2:172.21.15.145:6835/679350833 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 >> 172.21.15.94:0/2999220048 conn(0x55c3ff6c04b0 msgr2=0x55c3ff6c08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 shutdown_connections 2024-03-06T07:27:31.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 1 -- 172.21.15.94:0/2999220048 wait complete. 2024-03-06T07:27:31.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.100+0000 7f8ffd010740 20 client.10992 trim_cache size 0 max 16384 2024-03-06T07:27:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.116+0000 7f8ffd010740 1 Processor -- start 2024-03-06T07:27:31.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.116+0000 7f8ffd010740 1 -- start start 2024-03-06T07:27:31.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6df440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff6bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ffd010740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41f7b40 con 0x55c3ff70ee40 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40c04d0 con 0x55c3ff6c2330 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41f6720 con 0x55c3ff6c1f50 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff6bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff6bf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35868/0 (socket says 172.21.15.94:35868) 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 -- 172.21.15.94:0/1644654341 learned_addr learned my addr 172.21.15.94:0/1644654341 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6df440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 -- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 msgr2=0x55c3ff6df440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6df440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 -- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 msgr2=0x55c3ff6bf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 -- 172.21.15.94:0/1644654341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3ff4056b0 con 0x55c3ff70ee40 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.117+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff6bf9f0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f8fec002040 tx=0x7f8fec007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8625a1d608b39f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8fd97fa640 1 -- 172.21.15.94:0/1644654341 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec016630 con 0x55c3ff70ee40 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8fd97fa640 1 -- 172.21.15.94:0/1644654341 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec0167d0 con 0x55c3ff70ee40 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8fd97fa640 1 -- 172.21.15.94:0/1644654341 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec016b80 con 0x55c3ff70ee40 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 -- 172.21.15.94:0/1644654341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 msgr2=0x55c3ff6bf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff6bf9f0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f8fec002040 tx=0x7f8fec007410 comp rx=0 tx=0).stop 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 -- 172.21.15.94:0/1644654341 shutdown_connections 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff6bf9f0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 --2- 172.21.15.94:0/1644654341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff6df440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 -- 172.21.15.94:0/1644654341 >> 172.21.15.94:0/1644654341 conn(0x55c3ff6c04b0 msgr2=0x55c3ff6e49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 -- 172.21.15.94:0/1644654341 shutdown_connections 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.118+0000 7f8ffd010740 1 -- 172.21.15.94:0/1644654341 wait complete. 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.119+0000 7f8ffd010740 1 -- start start 2024-03-06T07:27:31.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.119+0000 7f8ffd010740 1 Processor -- start 2024-03-06T07:27:31.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.119+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff76b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.119+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff7011a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff6fba80 con 0x55c3ff70ee40 2024-03-06T07:27:31.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff6e4230 con 0x55c3ff6c2330 2024-03-06T07:27:31.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3ff6fadf0 con 0x55c3ff6c1f50 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 10 client.0 _ll_register_callbacks cb 0x7f8fe807a2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff7011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff7011a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35876/0 (socket says 172.21.15.94:35876) 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 -- 172.21.15.94:0/912314068 learned_addr learned my addr 172.21.15.94:0/912314068 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff37fe640 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff76b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 -- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 msgr2=0x55c3ff76b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff76b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 -- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 msgr2=0x55c3ff6bfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 -- 172.21.15.94:0/912314068 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3ff7025b0 con 0x55c3ff70ee40 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ff89c7640 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff7011a0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f8fec016da0 tx=0x7f8fec001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb2135e8164490f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8fd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/912314068 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec016630 con 0x55c3ff70ee40 2024-03-06T07:27:31.127 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/912314068 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fec005bd0 con 0x55c3ff70ee40 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fec0167d0 con 0x55c3ff70ee40 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/912314068 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fec005dd0 con 0x55c3ff70ee40 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 10 client.10998 fetch_fsmap learned FSMap version 199 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x55c3ff76db80 con 0x55c3ff70ee40 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.120+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/912314068 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec0167d0 con 0x55c3ff70ee40 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/912314068 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f8fec01f420 con 0x55c3ff70ee40 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ffd010740 10 client.10998 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ffd010740 10 client.10998 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ffd010740 20 client.10998 populate_metadata read hostname 'smithi094' 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ffd010740 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6de220 0x55c3ff6de600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ffd010740 4 client.10998 mds_command: new command op to 10791 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 --> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c3ff6c19a0 con 0x55c3ff6de220 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6de220 0x55c3ff6de600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8fd2ffd640 10 client.10998 ms_handle_connect on v2:172.21.15.145:6835/679350833 2024-03-06T07:27:31.128 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ff3fff640 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6de220 0x55c3ff6de600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f4714f28a4466814 server_cookie=1a240f9f026b3d6d in_seq=0 out_seq=0 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8fd2ffd640 1 -- 172.21.15.94:0/912314068 <== mds.1 v2:172.21.15.145:6835/679350833 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55c3ff6c19a0 con 0x55c3ff6de220 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8fd2ffd640 10 client.10998 handle_command_reply: tid=0 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.121+0000 7f8ffd010740 1 client.10998 shutdown 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 msgr2=0x55c3ff7011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff7011a0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f8fec016da0 tx=0x7f8fec001250 comp rx=0 tx=0).stop 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 shutdown_connections 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:6835/679350833,v1:172.21.15.145:6837/679350833] conn(0x55c3ff6de220 0x55c3ff6de600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55c3ff70ee40 0x55c3ff7011a0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8fd2ffd640 0 client.10998 ms_handle_reset on v2:172.21.15.145:6835/679350833 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c3ff6c2330 0x55c3ff6bfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 --2- 172.21.15.94:0/912314068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c3ff6c1f50 0x55c3ff76b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 >> 172.21.15.94:0/912314068 conn(0x55c3ff6c04b0 msgr2=0x55c3ff6e1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 shutdown_connections 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 1 -- 172.21.15.94:0/912314068 wait complete. 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffd010740 20 client.10998 trim_cache size 0 max 16384 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 1, 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 10791, 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:27:31.130 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:27:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:27:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 2.640122232, 2024-03-06T07:27:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 199, 2024-03-06T07:27:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 209, 2024-03-06T07:27:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 209, 2024-03-06T07:27:31.131 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 17.823199057 2024-03-06T07:27:31.131 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe006b500 msgr2=0x7f8fe006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe006b500 0x7f8fe006d9c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4007fd0 tx=0x7f8fe4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff414f110 msgr2=0x7f8ff4151530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff414f110 0x7f8ff4151530 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 shutdown_connections 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8fe006b500 0x7f8fe006d9c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff41561a0 0x7f8ff41ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8ff414f110 0x7f8ff4151530 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 --2- 172.21.15.94:0/4142517503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff40c6ab0 0x7f8ff40c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 >> 172.21.15.94:0/4142517503 conn(0x7f8ff406b1a0 msgr2=0x7f8ff41524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.132 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 shutdown_connections 2024-03-06T07:27:31.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.122+0000 7f8ffa42a640 1 -- 172.21.15.94:0/4142517503 wait complete. 2024-03-06T07:27:31.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:27:31.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.237+0000 7f5d7175e640 1 Processor -- start 2024-03-06T07:27:31.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.237+0000 7f5d7175e640 1 -- start start 2024-03-06T07:27:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d7175e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5d6c06ff10 0x7f5d6c153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d7175e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0a8390 con 0x7f5d6c0c8f20 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d7175e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0730f0 con 0x7f5d6c0c7170 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d7175e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d6c0c7620 con 0x7f5d6c06ff10 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d6b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5d6c0c8f20 0x7f5d6c15dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56880/0 (socket says 172.21.15.94:56880) 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d6b7fe640 1 -- 172.21.15.94:0/1091810522 learned_addr learned my addr 172.21.15.94:0/1091810522 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d6affd640 1 --2- 172.21.15.94:0/1091810522 >> v2:172.21.15.145:3300/0 conn(0x7f5d6c06ff10 0x7f5d6c153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d69ffb640 1 -- 172.21.15.94:0/1091810522 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126444908 0 0) 0x7f5d6c0a8390 con 0x7f5d6c0c8f20 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.238+0000 7f5d69ffb640 1 -- 172.21.15.94:0/1091810522 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d54003520 con 0x7f5d6c0c8f20 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.239+0000 7f5d6affd640 1 -- 172.21.15.94:0/1091810522 >> v1:172.21.15.145:6789/0 conn(0x7f5d6c0c7170 legacy=0x7f5d6c0c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.239+0000 7f5d6affd640 1 -- 172.21.15.94:0/1091810522 >> v1:172.21.15.94:6789/0 conn(0x7f5d6c0c8f20 legacy=0x7f5d6c15dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.239+0000 7f5d69ffb640 1 -- 172.21.15.94:0/1091810522 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295784775 0 0) 0x7f5d6c0730f0 con 0x7f5d6c0c7170 2024-03-06T07:27:31.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.239+0000 7f5d6affd640 1 -- 172.21.15.94:0/1091810522 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c0733d0 con 0x7f5d6c06ff10 2024-03-06T07:27:31.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.239+0000 7f5d6affd640 1 --2- 172.21.15.94:0/1091810522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c06ff10 0x7f5d6c153c20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0c77a0 tx=0x7f5d5c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a5ab5dda94a9fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.239+0000 7f5d69ffb640 1 -- 172.21.15.94:0/1091810522 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c039070 con 0x7f5d6c06ff10 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d69ffb640 1 -- 172.21.15.94:0/1091810522 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c02cbc0 con 0x7f5d6c06ff10 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d69ffb640 1 -- 172.21.15.94:0/1091810522 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c034440 con 0x7f5d6c06ff10 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d7175e640 1 -- 172.21.15.94:0/1091810522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c06ff10 msgr2=0x7f5d6c153c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1091810522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c06ff10 0x7f5d6c153c20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0c77a0 tx=0x7f5d5c02be10 comp rx=0 tx=0).stop 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d7175e640 1 -- 172.21.15.94:0/1091810522 shutdown_connections 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1091810522 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c06ff10 0x7f5d6c153c20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d7175e640 1 -- 172.21.15.94:0/1091810522 >> 172.21.15.94:0/1091810522 conn(0x7f5d6c06b970 msgr2=0x7f5d6c0aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d7175e640 1 -- 172.21.15.94:0/1091810522 shutdown_connections 2024-03-06T07:27:31.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.240+0000 7f5d7175e640 1 -- 172.21.15.94:0/1091810522 wait complete. 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.241+0000 7f5d7175e640 1 Processor -- start 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.241+0000 7f5d7175e640 1 -- start start 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d7175e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d6c06ff10 0x7f5d6c0bd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d7175e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c0c7170 0x7f5d6c156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d7175e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c0c8f20 0x7f5d6c0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d7175e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c070470 con 0x7f5d6c0c7170 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d7175e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c0702f0 con 0x7f5d6c0c8f20 2024-03-06T07:27:31.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d7175e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c0c6860 con 0x7f5d6c06ff10 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c0c7170 0x7f5d6c156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c0c7170 0x7f5d6c156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35890/0 (socket says 172.21.15.94:35890) 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6affd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d6c06ff10 0x7f5d6c0bd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/1970654663 learned_addr learned my addr 172.21.15.94:0/1970654663 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6b7fe640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c0c8f20 0x7f5d6c0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d6c06ff10 msgr2=0x7f5d6c0bd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d6c06ff10 0x7f5d6c0bd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c0c8f20 msgr2=0x7f5d6c0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.242+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c0c8f20 0x7f5d6c0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.243+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/1970654663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c0c5a10 con 0x7f5d6c0c7170 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.243+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c0c7170 0x7f5d6c156470 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c4b0 tx=0x7f5d6000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.243+0000 7f5d537fe640 1 -- 172.21.15.94:0/1970654663 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60017070 con 0x7f5d6c0c7170 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.243+0000 7f5d537fe640 1 -- 172.21.15.94:0/1970654663 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d60005480 con 0x7f5d6c0c7170 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.243+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c14cc90 con 0x7f5d6c0c7170 2024-03-06T07:27:31.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.243+0000 7f5d537fe640 1 -- 172.21.15.94:0/1970654663 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60003830 con 0x7f5d6c0c7170 2024-03-06T07:27:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.243+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c0c2330 con 0x7f5d6c0c7170 2024-03-06T07:27:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.244+0000 7f5d537fe640 1 -- 172.21.15.94:0/1970654663 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d6001a070 con 0x7f5d6c0c7170 2024-03-06T07:27:31.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.245+0000 7f5d537fe640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d5806b500 0x7f5d5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.245+0000 7f5d537fe640 1 -- 172.21.15.94:0/1970654663 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5d600895f0 con 0x7f5d6c0c7170 2024-03-06T07:27:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.245+0000 7f5d6affd640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d5806b500 0x7f5d5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.245+0000 7f5d6affd640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d5806b500 0x7f5d5806d9c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c036000 tx=0x7f5d5c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 Processor -- start 2024-03-06T07:27:31.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 -- start start 2024-03-06T07:27:31.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfd2800 0x5625cbf89010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbf89550 0x5625cbfd2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfc3420 0x5625cbf83680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625cbeed6b0 con 0x5625cbfd2800 2024-03-06T07:27:31.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625cbe96a60 con 0x5625cbfc3420 2024-03-06T07:27:31.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.246+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625cbee9750 con 0x5625cbf89550 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfd2800 0x5625cbf89010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfd2800 0x5625cbf89010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35892/0 (socket says 172.21.15.94:35892) 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 -- 172.21.15.94:0/506841272 learned_addr learned my addr 172.21.15.94:0/506841272 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbf89550 0x5625cbfd2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6b7fe640 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfc3420 0x5625cbf83680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 -- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbf89550 msgr2=0x5625cbfd2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbf89550 0x5625cbfd2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 -- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfc3420 msgr2=0x5625cbf83680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfc3420 0x5625cbf83680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 -- 172.21.15.94:0/506841272 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625cbeb4f30 con 0x5625cbfd2800 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d6affd640 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfd2800 0x5625cbf89010 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c02c420 tx=0x7f5d5c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e900bb463a0eb55f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d50ff9640 1 -- 172.21.15.94:0/506841272 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c039040 con 0x5625cbfd2800 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d50ff9640 1 -- 172.21.15.94:0/506841272 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c03a040 con 0x5625cbfd2800 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.247+0000 7f5d50ff9640 1 -- 172.21.15.94:0/506841272 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c0497f0 con 0x5625cbfd2800 2024-03-06T07:27:31.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 -- 172.21.15.94:0/506841272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfd2800 msgr2=0x5625cbf89010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfd2800 0x5625cbf89010 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c02c420 tx=0x7f5d5c02e000 comp rx=0 tx=0).stop 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 -- 172.21.15.94:0/506841272 shutdown_connections 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfc3420 0x5625cbf83680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbf89550 0x5625cbfd2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 --2- 172.21.15.94:0/506841272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfd2800 0x5625cbf89010 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 -- 172.21.15.94:0/506841272 >> 172.21.15.94:0/506841272 conn(0x5625cbf857d0 msgr2=0x5625cbfaa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 -- 172.21.15.94:0/506841272 shutdown_connections 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 -- 172.21.15.94:0/506841272 wait complete. 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.248+0000 7f5d74428740 1 -- start start 2024-03-06T07:27:31.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 1 Processor -- start 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbf841a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfd2800 0x5625cbfd0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfa7870 0x5625cbfa6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625cbfc7590 con 0x5625cbfc3420 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625cbea9ea0 con 0x5625cbfa7870 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625cbfaadb0 con 0x5625cbfd2800 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbf841a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d74428740 10 client.0 _ll_register_callbacks cb 0x5625cbea9870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbf841a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35908/0 (socket says 172.21.15.94:35908) 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d6affd640 1 -- 172.21.15.94:0/113442899 learned_addr learned my addr 172.21.15.94:0/113442899 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfd2800 0x5625cbfd0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.249+0000 7f5d6b7fe640 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfa7870 0x5625cbfa6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d6affd640 1 -- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfd2800 msgr2=0x5625cbfd0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d6affd640 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfd2800 0x5625cbfd0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d6affd640 1 -- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfa7870 msgr2=0x5625cbfa6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d6affd640 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfa7870 0x5625cbfa6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d6affd640 1 -- 172.21.15.94:0/113442899 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625cbeaa270 con 0x5625cbfc3420 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d6affd640 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbf841a0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c041040 tx=0x7f5d5c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ccc2973ec5da76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d49ffb640 1 -- 172.21.15.94:0/113442899 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c05a020 con 0x5625cbfc3420 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d49ffb640 1 -- 172.21.15.94:0/113442899 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5d5c004240 con 0x5625cbfc3420 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d5c03a040 con 0x5625cbfc3420 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d49ffb640 1 -- 172.21.15.94:0/113442899 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d5c064050 con 0x5625cbfc3420 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 10 client.11010 fetch_fsmap learned FSMap version 199 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x5625cbc022d0 con 0x5625cbfc3420 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d49ffb640 1 -- 172.21.15.94:0/113442899 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c03a040 con 0x5625cbfc3420 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d49ffb640 1 -- 172.21.15.94:0/113442899 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f5d5c067020 con 0x5625cbfc3420 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 10 client.11010 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 10 client.11010 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 20 client.11010 populate_metadata read hostname 'smithi094' 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cbfad970 0x5625cbf87a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 4 client.11010 mds_command: new command op to 38273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.250+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 --> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625cbefabf0 con 0x5625cbfad970 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.251+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cbfad970 0x5625cbf87a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.251+0000 7f5d49ffb640 10 client.11010 ms_handle_connect on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.251+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cbfad970 0x5625cbf87a50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da9f35952a746e75 server_cookie=eeabd052e94877f2 in_seq=0 out_seq=0 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d49ffb640 1 -- 172.21.15.94:0/113442899 <== mds.0 v2:172.21.15.145:6834/2585066812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5625cbefabf0 con 0x5625cbfad970 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d49ffb640 10 client.11010 handle_command_reply: tid=0 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 client.11010 shutdown 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 msgr2=0x5625cbf841a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbf841a0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c041040 tx=0x7f5d5c004ab0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 shutdown_connections 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfa7870 0x5625cbfa6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfd2800 0x5625cbfd0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cbfad970 0x5625cbf87a50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 --2- 172.21.15.94:0/113442899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbf841a0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d49ffb640 0 client.11010 ms_handle_reset on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 >> 172.21.15.94:0/113442899 conn(0x5625cbf89550 msgr2=0x5625cbfcf0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 shutdown_connections 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 1 -- 172.21.15.94:0/113442899 wait complete. 2024-03-06T07:27:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.252+0000 7f5d74428740 20 client.11010 trim_cache size 0 max 16384 2024-03-06T07:27:31.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 Processor -- start 2024-03-06T07:27:31.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 -- start start 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfa7870 0x5625cbfd1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbfe4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cbfe4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c153de0 con 0x5625cbfc3420 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c1ec6a0 con 0x5625cbfd2800 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c06d2e0 con 0x5625cbfa7870 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbfe4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbfe4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35922/0 (socket says 172.21.15.94:35922) 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/964618987 learned_addr learned my addr 172.21.15.94:0/964618987 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d6affd640 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfa7870 0x5625cbfd1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.269+0000 7f5d6b7fe640 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cbfe4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfa7870 msgr2=0x5625cbfd1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfa7870 0x5625cbfd1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 msgr2=0x5625cbfe4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cbfe4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d6a7fc640 1 -- 172.21.15.94:0/964618987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625cbccc6b0 con 0x5625cbfc3420 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d6b7fe640 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cbfe4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbfe4510 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000ce70 tx=0x7f5d60059f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738bb0a161c571f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d4bfff640 1 -- 172.21.15.94:0/964618987 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60056000 con 0x5625cbfc3420 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d4bfff640 1 -- 172.21.15.94:0/964618987 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d60054370 con 0x5625cbfc3420 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d4bfff640 1 -- 172.21.15.94:0/964618987 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d600872f0 con 0x5625cbfc3420 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 -- 172.21.15.94:0/964618987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 msgr2=0x5625cbfe4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbfe4510 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000ce70 tx=0x7f5d60059f90 comp rx=0 tx=0).stop 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 -- 172.21.15.94:0/964618987 shutdown_connections 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cbfe4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfc3420 0x5625cbfe4510 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 --2- 172.21.15.94:0/964618987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfa7870 0x5625cbfd1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 -- 172.21.15.94:0/964618987 >> 172.21.15.94:0/964618987 conn(0x5625cbf89550 msgr2=0x5625cbf88d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 -- 172.21.15.94:0/964618987 shutdown_connections 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.270+0000 7f5d74428740 1 -- 172.21.15.94:0/964618987 wait complete. 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.271+0000 7f5d74428740 1 -- start start 2024-03-06T07:27:31.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.271+0000 7f5d74428740 1 Processor -- start 2024-03-06T07:27:31.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfa7870 0x5625cbfd9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfc3420 0x5625cc03dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cc03a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625cbfda6a0 con 0x5625cbfa7870 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625cbfd9e80 con 0x5625cbfd2800 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625cbfddf40 con 0x5625cbfc3420 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 10 client.0 _ll_register_callbacks cb 0x7f5d5c009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfa7870 0x5625cbfd9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfa7870 0x5625cbfd9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35938/0 (socket says 172.21.15.94:35938) 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 -- 172.21.15.94:0/4056826516 learned_addr learned my addr 172.21.15.94:0/4056826516 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfc3420 0x5625cc03dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6b7fe640 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cc03a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 -- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfc3420 msgr2=0x5625cc03dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfc3420 0x5625cc03dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 -- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 msgr2=0x5625cc03a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cc03a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 -- 172.21.15.94:0/4056826516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625cbfbb6e0 con 0x5625cbfa7870 2024-03-06T07:27:31.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d6affd640 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfa7870 0x5625cbfd9470 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c041040 tx=0x7f5d5c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8a792e880a2fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.94:3300/0 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d49ffb640 1 -- 172.21.15.94:0/4056826516 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c05a070 con 0x5625cbfa7870 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d49ffb640 1 -- 172.21.15.94:0/4056826516 <== mon.0 v2:172.21.15.94:3300/0 2 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5d5c004120 con 0x5625cbfa7870 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d60054370 con 0x5625cbfa7870 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d49ffb640 1 -- 172.21.15.94:0/4056826516 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d5c075050 con 0x5625cbfa7870 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 10 client.11016 fetch_fsmap learned FSMap version 199 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.272+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) v3 -- 0x5625cbfbbd00 con 0x5625cbfa7870 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d49ffb640 1 -- 172.21.15.94:0/4056826516 <== mon.0 v2:172.21.15.94:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d60054370 con 0x5625cbfa7870 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d49ffb640 1 -- 172.21.15.94:0/4056826516 <== mon.0 v2:172.21.15.94:3300/0 5 ==== fsmap(e 199) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f5d5c078020 con 0x5625cbfa7870 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 10 client.11016 fetch_fsmap finished waiting for FSMap version 199 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 10 client.11016 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 20 client.11016 populate_metadata read hostname 'smithi094' 2024-03-06T07:27:31.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cc036df0 0x5625cc0371d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 4 client.11016 mds_command: new command op to 38273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 --> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5625cbfa76c0 con 0x5625cc036df0 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cc036df0 0x5625cc0371d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d6a7fc640 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cc036df0 0x5625cc0371d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6b266c97a2d8aed server_cookie=d339e9313114e451 in_seq=0 out_seq=0 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d49ffb640 10 client.11016 ms_handle_connect on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d49ffb640 1 -- 172.21.15.94:0/4056826516 <== mds.0 v2:172.21.15.145:6834/2585066812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5625cbfa76c0 con 0x5625cc036df0 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d49ffb640 10 client.11016 handle_command_reply: tid=0 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 1 client.11016 shutdown 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfa7870 msgr2=0x5625cbfd9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.273+0000 7f5d74428740 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfa7870 0x5625cbfd9470 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c041040 tx=0x7f5d5c031000 comp rx=0 tx=0).stop 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 shutdown_connections 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5625cbfd2800 0x5625cc03a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5625cbfc3420 0x5625cc03dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.145:6834/2585066812,v1:172.21.15.145:6836/2585066812] conn(0x5625cc036df0 0x5625cc0371d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 --2- 172.21.15.94:0/4056826516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x5625cbfa7870 0x5625cbfd9470 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 >> 172.21.15.94:0/4056826516 conn(0x5625cbf89550 msgr2=0x5625cbfaa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d49ffb640 0 client.11016 ms_handle_reset on v2:172.21.15.145:6834/2585066812 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 shutdown_connections 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 1 -- 172.21.15.94:0/4056826516 wait complete. 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d74428740 20 client.11016 trim_cache size 0 max 16384 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "cluster_fsid": "00dede11-ad16-411b-be77-3a57b1483d8f", 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "whoami": 0, 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "id": 38273, 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "want_state": "up:active", 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "state": "up:active", 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "fs_name": "cephfs", 2024-03-06T07:27:31.282 INFO:teuthology.orchestra.run.smithi094.stdout: "rank_uptime": 12.875727856999999, 2024-03-06T07:27:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "mdsmap_epoch": 199, 2024-03-06T07:27:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch": 209, 2024-03-06T07:27:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "osdmap_epoch_barrier": 209, 2024-03-06T07:27:31.283 INFO:teuthology.orchestra.run.smithi094.stdout: "uptime": 17.975235161000001 2024-03-06T07:27:31.283 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d5806b500 msgr2=0x7f5d5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d5806b500 0x7f5d5806d9c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c036000 tx=0x7f5d5c030000 comp rx=0 tx=0).stop 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c0c7170 msgr2=0x7f5d6c156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c0c7170 0x7f5d6c156470 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c4b0 tx=0x7f5d6000c970 comp rx=0 tx=0).stop 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 shutdown_connections 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d5806b500 0x7f5d5806d9c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d6c0c8f20 0x7f5d6c0ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d6c0c7170 0x7f5d6c156470 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 --2- 172.21.15.94:0/1970654663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d6c06ff10 0x7f5d6c0bd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 >> 172.21.15.94:0/1970654663 conn(0x7f5d6c06b970 msgr2=0x7f5d6c150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 shutdown_connections 2024-03-06T07:27:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.274+0000 7f5d7175e640 1 -- 172.21.15.94:0/1970654663 wait complete. 2024-03-06T07:27:31.287 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-03-06T07:27:31.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.388+0000 7f04bc33e640 1 Processor -- start 2024-03-06T07:27:31.389 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.388+0000 7f04bc33e640 1 -- start start 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.388+0000 7f04bc33e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04b414ef70 0x7f04b406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04bc33e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b40c2790 con 0x7f04b40c1820 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04bc33e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b406cf40 con 0x7f04b406cad0 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04bc33e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b40c3090 con 0x7f04b414ef70 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04b98d9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f04b40c1820 0x7f04b40c2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56896/0 (socket says 172.21.15.94:56896) 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04b98d9640 1 -- 172.21.15.94:0/419337021 learned_addr learned my addr 172.21.15.94:0/419337021 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04ba0da640 1 --2- 172.21.15.94:0/419337021 >> v2:172.21.15.145:3300/0 conn(0x7f04b414ef70 0x7f04b406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04b90d8640 1 -- 172.21.15.94:0/419337021 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714288305 0 0) 0x7f04b40c2790 con 0x7f04b40c1820 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04b90d8640 1 -- 172.21.15.94:0/419337021 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a0003520 con 0x7f04b40c1820 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04ba0da640 1 -- 172.21.15.94:0/419337021 >> v1:172.21.15.145:6789/0 conn(0x7f04b406cad0 legacy=0x7f04b4154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04ba0da640 1 -- 172.21.15.94:0/419337021 >> v1:172.21.15.94:6789/0 conn(0x7f04b40c1820 legacy=0x7f04b40c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.391 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04b90d8640 1 -- 172.21.15.94:0/419337021 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278811250 0 0) 0x7f04ac001110 con 0x7f04b406cad0 2024-03-06T07:27:31.392 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.389+0000 7f04ba0da640 1 -- 172.21.15.94:0/419337021 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40c1540 con 0x7f04b414ef70 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.390+0000 7f04ba0da640 1 --2- 172.21.15.94:0/419337021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b414ef70 0x7f04b406c1f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009d90 tx=0x7f04b002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21ecdce20380256d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.390+0000 7f04b90d8640 1 -- 172.21.15.94:0/419337021 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0039070 con 0x7f04b414ef70 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.390+0000 7f04b90d8640 1 -- 172.21.15.94:0/419337021 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b002cc60 con 0x7f04b414ef70 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.390+0000 7f04b90d8640 1 -- 172.21.15.94:0/419337021 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0034640 con 0x7f04b414ef70 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.390+0000 7f04bc33e640 1 -- 172.21.15.94:0/419337021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b414ef70 msgr2=0x7f04b406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.390+0000 7f04bc33e640 1 --2- 172.21.15.94:0/419337021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b414ef70 0x7f04b406c1f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009d90 tx=0x7f04b002beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.391+0000 7f04bc33e640 1 -- 172.21.15.94:0/419337021 shutdown_connections 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.391+0000 7f04bc33e640 1 --2- 172.21.15.94:0/419337021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b414ef70 0x7f04b406c1f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.391+0000 7f04bc33e640 1 -- 172.21.15.94:0/419337021 >> 172.21.15.94:0/419337021 conn(0x7f04b40c6710 msgr2=0x7f04b40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.391+0000 7f04bc33e640 1 -- 172.21.15.94:0/419337021 shutdown_connections 2024-03-06T07:27:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.391+0000 7f04bc33e640 1 -- 172.21.15.94:0/419337021 wait complete. 2024-03-06T07:27:31.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 Processor -- start 2024-03-06T07:27:31.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 -- start start 2024-03-06T07:27:31.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b406cad0 0x7f04b4151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b40c1820 0x7f04b4145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b414ef70 0x7f04b414f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b4155750 con 0x7f04b414ef70 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b414f350 con 0x7f04b40c1820 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04bc33e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b4155340 con 0x7f04b406cad0 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04ba8db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b414ef70 0x7f04b414f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04ba8db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b414ef70 0x7f04b414f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35952/0 (socket says 172.21.15.94:35952) 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.392+0000 7f04ba8db640 1 -- 172.21.15.94:0/2277540671 learned_addr learned my addr 172.21.15.94:0/2277540671 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04ba0da640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b406cad0 0x7f04b4151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04b98d9640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b40c1820 0x7f04b4145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04ba8db640 1 -- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b406cad0 msgr2=0x7f04b4151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04ba8db640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b406cad0 0x7f04b4151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04ba8db640 1 -- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b40c1820 msgr2=0x7f04b4145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04ba8db640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b40c1820 0x7f04b4145330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04ba8db640 1 -- 172.21.15.94:0/2277540671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4150df0 con 0x7f04b414ef70 2024-03-06T07:27:31.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04ba8db640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b414ef70 0x7f04b414f830 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c3e0 tx=0x7f04ac00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04aaffd640 1 -- 172.21.15.94:0/2277540671 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac017070 con 0x7f04b414ef70 2024-03-06T07:27:31.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04aaffd640 1 -- 172.21.15.94:0/2277540671 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac0035f0 con 0x7f04b414ef70 2024-03-06T07:27:31.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b41510d0 con 0x7f04b414ef70 2024-03-06T07:27:31.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04aaffd640 1 -- 172.21.15.94:0/2277540671 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0039e0 con 0x7f04b414ef70 2024-03-06T07:27:31.397 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.393+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b4143b30 con 0x7f04b414ef70 2024-03-06T07:27:31.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.394+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b404f330 con 0x7f04b414ef70 2024-03-06T07:27:31.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.395+0000 7f04aaffd640 1 -- 172.21.15.94:0/2277540671 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04ac01a070 con 0x7f04b414ef70 2024-03-06T07:27:31.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.395+0000 7f04aaffd640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049806b4d0 0x7f049806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.395+0000 7f04aaffd640 1 -- 172.21.15.94:0/2277540671 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f04ac0895f0 con 0x7f04b414ef70 2024-03-06T07:27:31.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.395+0000 7f04ba0da640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049806b4d0 0x7f049806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.398 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.396+0000 7f04ba0da640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049806b4d0 0x7f049806d990 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f04b0036000 tx=0x7f04b0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.399+0000 7f04aaffd640 1 -- 172.21.15.94:0/2277540671 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04b406cf40 con 0x7f04b414ef70 2024-03-06T07:27:31.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:31.614+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f04b40c1110 con 0x7f04b414ef70 2024-03-06T07:27:32.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.245+0000 7f04aaffd640 1 -- 172.21.15.94:0/2277540671 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v200) v1 ==== 125+0+0 (secure 0 0 0) 0x7f04ac057d20 con 0x7f04b414ef70 2024-03-06T07:27:32.247 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked down. 2024-03-06T07:27:32.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049806b4d0 msgr2=0x7f049806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049806b4d0 0x7f049806d990 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f04b0036000 tx=0x7f04b0030000 comp rx=0 tx=0).stop 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b414ef70 msgr2=0x7f04b414f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b414ef70 0x7f04b414f830 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c3e0 tx=0x7f04ac00c8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 shutdown_connections 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049806b4d0 0x7f049806d990 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b414ef70 0x7f04b414f830 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b40c1820 0x7f04b4145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 --2- 172.21.15.94:0/2277540671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b406cad0 0x7f04b4151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 >> 172.21.15.94:0/2277540671 conn(0x7f04b40c6710 msgr2=0x7f04b40c5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 shutdown_connections 2024-03-06T07:27:32.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.247+0000 7f04bc33e640 1 -- 172.21.15.94:0/2277540671 wait complete. 2024-03-06T07:27:32.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:32.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.361+0000 7fa2a11d6640 1 Processor -- start 2024-03-06T07:27:32.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.361+0000 7fa2a11d6640 1 -- start start 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa2a11d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa2a11d6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa29c155a40 0x7fa29c155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa2a11d6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c156410 con 0x7fa29c14f790 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa2a11d6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c0c2680 con 0x7fa29c155a40 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa2a11d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0c28d0 con 0x7fa29c0c1aa0 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa29ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa29c14f790 0x7fa29c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56908/0 (socket says 172.21.15.94:56908) 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa29ad9d640 1 -- 172.21.15.94:0/333042306 learned_addr learned my addr 172.21.15.94:0/333042306 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa29b59e640 1 --2- 172.21.15.94:0/333042306 >> v2:172.21.15.145:3300/0 conn(0x7fa29c155a40 0x7fa29c155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa29a59c640 1 --2- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.362+0000 7fa299d9b640 1 -- 172.21.15.94:0/333042306 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823498352 0 0) 0x7fa29c156410 con 0x7fa29c14f790 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.363+0000 7fa299d9b640 1 -- 172.21.15.94:0/333042306 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa284003520 con 0x7fa29c14f790 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.363+0000 7fa29b59e640 1 -- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 msgr2=0x7fa29c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.363+0000 7fa29b59e640 1 --2- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.363+0000 7fa29b59e640 1 -- 172.21.15.94:0/333042306 >> v1:172.21.15.94:6789/0 conn(0x7fa29c14f790 legacy=0x7fa29c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.363+0000 7fa29b59e640 1 -- 172.21.15.94:0/333042306 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0c2bb0 con 0x7fa29c155a40 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.363+0000 7fa29a59c640 1 --2- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:32.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.363+0000 7fa29b59e640 1 --2- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c155a40 0x7fa29c155ed0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0097a0 tx=0x7fa28c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8a8ea091099b054 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa299d9b640 1 -- 172.21.15.94:0/333042306 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c035020 con 0x7fa29c155a40 2024-03-06T07:27:32.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa299d9b640 1 -- 172.21.15.94:0/333042306 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c031830 con 0x7fa29c155a40 2024-03-06T07:27:32.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa299d9b640 1 -- 172.21.15.94:0/333042306 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c031c00 con 0x7fa29c155a40 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 -- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c155a40 msgr2=0x7fa29c155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c155a40 0x7fa29c155ed0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0097a0 tx=0x7fa28c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 -- 172.21.15.94:0/333042306 shutdown_connections 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c155a40 0x7fa29c155ed0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/333042306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 -- 172.21.15.94:0/333042306 >> 172.21.15.94:0/333042306 conn(0x7fa29c0c6710 msgr2=0x7fa29c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 -- 172.21.15.94:0/333042306 shutdown_connections 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.364+0000 7fa2a11d6640 1 -- 172.21.15.94:0/333042306 wait complete. 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.365+0000 7fa2a11d6640 1 Processor -- start 2024-03-06T07:27:32.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa2a11d6640 1 -- start start 2024-03-06T07:27:32.370 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa2a11d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c073110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa2a11d6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c14f790 0x7fa29c1dfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa2a11d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa29c155a40 0x7fa29c1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa2a11d6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0c0d60 con 0x7fa29c155a40 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa2a11d6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c155520 con 0x7fa29c14f790 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa2a11d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0c5e30 con 0x7fa29c0c1aa0 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa29ad9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa29a59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c14f790 0x7fa29c1dfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa29b59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa29c155a40 0x7fa29c1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa29a59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c14f790 0x7fa29c1dfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:45932/0 (socket says 172.21.15.94:45932) 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa29ad9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c073110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53952/0 (socket says 172.21.15.94:53952) 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.366+0000 7fa29a59c640 1 -- 172.21.15.94:0/2521363836 learned_addr learned my addr 172.21.15.94:0/2521363836 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:32.371 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa29a59c640 1 -- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 msgr2=0x7fa29c073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa29a59c640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c073110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa29a59c640 1 -- 172.21.15.94:0/2521363836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa29c155a40 msgr2=0x7fa29c1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa29a59c640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa29c155a40 0x7fa29c1541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa29a59c640 1 -- 172.21.15.94:0/2521363836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c14ff60 con 0x7fa29c14f790 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa29ad9d640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa29a59c640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c14f790 0x7fa29c1dfc10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa290009f60 tx=0x7fa290009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa2837fe640 1 -- 172.21.15.94:0/2521363836 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290018020 con 0x7fa29c14f790 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa2837fe640 1 -- 172.21.15.94:0/2521363836 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa290014680 con 0x7fa29c14f790 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa2837fe640 1 -- 172.21.15.94:0/2521363836 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290014a50 con 0x7fa29c14f790 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c153bb0 con 0x7fa29c14f790 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.367+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c150660 con 0x7fa29c14f790 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.368+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c04f330 con 0x7fa29c14f790 2024-03-06T07:27:32.372 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.368+0000 7fa2837fe640 1 -- 172.21.15.94:0/2521363836 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa290005d10 con 0x7fa29c14f790 2024-03-06T07:27:32.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.369+0000 7fa2837fe640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa28806b4d0 0x7fa28806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.369+0000 7fa2837fe640 1 -- 172.21.15.94:0/2521363836 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa29008b060 con 0x7fa29c14f790 2024-03-06T07:27:32.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.369+0000 7fa29ad9d640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa28806b4d0 0x7fa28806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.373 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.369+0000 7fa29ad9d640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa28806b4d0 0x7fa28806d990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa294005880 tx=0x7fa294009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.375 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.374+0000 7fa2837fe640 1 -- 172.21.15.94:0/2521363836 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa290022030 con 0x7fa29c14f790 2024-03-06T07:27:32.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.582+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa29c0c1f40 con 0x7fa29c14f790 2024-03-06T07:27:32.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.583+0000 7fa2837fe640 1 -- 172.21.15.94:0/2521363836 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fa290010040 con 0x7fa29c14f790 2024-03-06T07:27:32.585 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:32.585 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:32.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa28806b4d0 msgr2=0x7fa28806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa28806b4d0 0x7fa28806d990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa294005880 tx=0x7fa294009040 comp rx=0 tx=0).stop 2024-03-06T07:27:32.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c14f790 msgr2=0x7fa29c1dfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c14f790 0x7fa29c1dfc10 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa290009f60 tx=0x7fa290009f90 comp rx=0 tx=0).stop 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 shutdown_connections 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa28806b4d0 0x7fa28806d990 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa29c155a40 0x7fa29c1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa29c14f790 0x7fa29c1dfc10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 --2- 172.21.15.94:0/2521363836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa29c0c1aa0 0x7fa29c073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 >> 172.21.15.94:0/2521363836 conn(0x7fa29c0c6710 msgr2=0x7fa29c06d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 shutdown_connections 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:32.585+0000 7fa2a11d6640 1 -- 172.21.15.94:0/2521363836 wait complete. 2024-03-06T07:27:33.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:33.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:34.001 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.111+0000 7f97c73bc640 1 Processor -- start 2024-03-06T07:27:34.113 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.111+0000 7f97c73bc640 1 -- start start 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c73bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c00a9a30 0x7f97c0156850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c73bc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f97c00ab730 0x7f97c0163fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c73bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c00abca0 con 0x7f97c0156d90 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c73bc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c00c4610 con 0x7f97c00ab730 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c73bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c015a140 con 0x7f97c00a9a30 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c4957640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f97c0156d90 0x7f97c00c4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56912/0 (socket says 172.21.15.94:56912) 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c4957640 1 -- 172.21.15.94:0/3272844848 learned_addr learned my addr 172.21.15.94:0/3272844848 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c5158640 1 --2- 172.21.15.94:0/3272844848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c00a9a30 0x7f97c0156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.112+0000 7f97c5959640 1 --2- 172.21.15.94:0/3272844848 >> v2:172.21.15.145:3300/0 conn(0x7f97c00ab730 0x7f97c0163fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97b7fff640 1 -- 172.21.15.94:0/3272844848 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948379328 0 0) 0x7f97c00abca0 con 0x7f97c0156d90 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97b7fff640 1 -- 172.21.15.94:0/3272844848 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac003520 con 0x7f97c0156d90 2024-03-06T07:27:34.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97c5158640 1 -- 172.21.15.94:0/3272844848 >> v2:172.21.15.145:3300/0 conn(0x7f97c00ab730 msgr2=0x7f97c0163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97c5158640 1 --2- 172.21.15.94:0/3272844848 >> v2:172.21.15.145:3300/0 conn(0x7f97c00ab730 0x7f97c0163fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97c5158640 1 -- 172.21.15.94:0/3272844848 >> v1:172.21.15.94:6789/0 conn(0x7f97c0156d90 legacy=0x7f97c00c4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97c5158640 1 -- 172.21.15.94:0/3272844848 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c015a420 con 0x7f97c00a9a30 2024-03-06T07:27:34.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97c5959640 1 --2- 172.21.15.94:0/3272844848 >> v2:172.21.15.145:3300/0 conn(0x7f97c00ab730 0x7f97c0163fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.113+0000 7f97c5158640 1 --2- 172.21.15.94:0/3272844848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c00a9a30 0x7f97c0156850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e444736682da8b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97b7fff640 1 -- 172.21.15.94:0/3272844848 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc035020 con 0x7f97c00a9a30 2024-03-06T07:27:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97b7fff640 1 -- 172.21.15.94:0/3272844848 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc031830 con 0x7f97c00a9a30 2024-03-06T07:27:34.119 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97b7fff640 1 -- 172.21.15.94:0/3272844848 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc031be0 con 0x7f97c00a9a30 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 -- 172.21.15.94:0/3272844848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c00a9a30 msgr2=0x7f97c0156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 --2- 172.21.15.94:0/3272844848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c00a9a30 0x7f97c0156850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 -- 172.21.15.94:0/3272844848 shutdown_connections 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 --2- 172.21.15.94:0/3272844848 >> v2:172.21.15.145:3300/0 conn(0x7f97c00ab730 0x7f97c0163fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 --2- 172.21.15.94:0/3272844848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c00a9a30 0x7f97c0156850 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 -- 172.21.15.94:0/3272844848 >> 172.21.15.94:0/3272844848 conn(0x7f97c00c6710 msgr2=0x7f97c00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 -- 172.21.15.94:0/3272844848 shutdown_connections 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.114+0000 7f97c73bc640 1 -- 172.21.15.94:0/3272844848 wait complete. 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.115+0000 7f97c73bc640 1 Processor -- start 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.115+0000 7f97c73bc640 1 -- start start 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c73bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97c00a9a30 0x7f97c015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c73bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97c00ab730 0x7f97c0150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.120 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c73bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c0156d90 0x7f97c01f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c73bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00c0e30 con 0x7f97c00a9a30 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c73bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00c0cb0 con 0x7f97c00ab730 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c73bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00c1d80 con 0x7f97c0156d90 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97c00a9a30 0x7f97c015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97c00a9a30 0x7f97c015c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35970/0 (socket says 172.21.15.94:35970) 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 -- 172.21.15.94:0/2864233422 learned_addr learned my addr 172.21.15.94:0/2864233422 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c4957640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97c00ab730 0x7f97c0150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5959640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c0156d90 0x7f97c01f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 -- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c0156d90 msgr2=0x7f97c01f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c0156d90 0x7f97c01f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 -- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97c00ab730 msgr2=0x7f97c0150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.121 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97c00ab730 0x7f97c0150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.116+0000 7f97c5158640 1 -- 172.21.15.94:0/2864233422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c0151980 con 0x7f97c00a9a30 2024-03-06T07:27:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.117+0000 7f97c5158640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97c00a9a30 0x7f97c015c2a0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0098c0 tx=0x7f97bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.117+0000 7f97b5ffb640 1 -- 172.21.15.94:0/2864233422 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc035040 con 0x7f97c00a9a30 2024-03-06T07:27:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.117+0000 7f97b5ffb640 1 -- 172.21.15.94:0/2864233422 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc03b840 con 0x7f97c00a9a30 2024-03-06T07:27:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.117+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c01f29c0 con 0x7f97c00a9a30 2024-03-06T07:27:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.117+0000 7f97b5ffb640 1 -- 172.21.15.94:0/2864233422 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc03bc30 con 0x7f97c00a9a30 2024-03-06T07:27:34.122 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.117+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c015d650 con 0x7f97c00a9a30 2024-03-06T07:27:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.118+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c00aa6a0 con 0x7f97c00a9a30 2024-03-06T07:27:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.118+0000 7f97b5ffb640 1 -- 172.21.15.94:0/2864233422 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97bc038020 con 0x7f97c00a9a30 2024-03-06T07:27:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.119+0000 7f97b5ffb640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97a406b4d0 0x7f97a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.119+0000 7f97b5ffb640 1 -- 172.21.15.94:0/2864233422 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f97bc0ac200 con 0x7f97c00a9a30 2024-03-06T07:27:34.123 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.119+0000 7f97c4957640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97a406b4d0 0x7f97a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.124 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.119+0000 7f97c4957640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97a406b4d0 0x7f97a406d990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f97c0072a10 tx=0x7f97b0000bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.123+0000 7f97b5ffb640 1 -- 172.21.15.94:0/2864233422 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97c00aa6a0 con 0x7f97c00a9a30 2024-03-06T07:27:34.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.330+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97c004f330 con 0x7f97c00a9a30 2024-03-06T07:27:34.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.330+0000 7f97b5ffb640 1 -- 172.21.15.94:0/2864233422 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f97c0151980 con 0x7f97c00a9a30 2024-03-06T07:27:34.333 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:34.333 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:34.333 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:34.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97a406b4d0 msgr2=0x7f97a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97a406b4d0 0x7f97a406d990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f97c0072a10 tx=0x7f97b0000bc0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97c00a9a30 msgr2=0x7f97c015c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97c00a9a30 0x7f97c015c2a0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0098c0 tx=0x7f97bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 shutdown_connections 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f97a406b4d0 0x7f97a406d990 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97c0156d90 0x7f97c01f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97c00ab730 0x7f97c0150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 --2- 172.21.15.94:0/2864233422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f97c00a9a30 0x7f97c015c2a0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.332+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 >> 172.21.15.94:0/2864233422 conn(0x7f97c00c6710 msgr2=0x7f97c00c12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.333+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 shutdown_connections 2024-03-06T07:27:34.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:34.333+0000 7f97c73bc640 1 -- 172.21.15.94:0/2864233422 wait complete. 2024-03-06T07:27:34.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:34.748 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:35.748 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.857+0000 7f1ba4caf640 1 Processor -- start 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.858+0000 7f1ba4caf640 1 -- start start 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.858+0000 7f1ba4caf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ba00c88c0 0x7f1ba0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.858+0000 7f1ba4caf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba0158b60 0x7f1ba0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.858+0000 7f1ba4caf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba00c73d0 con 0x7f1ba014f360 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.858+0000 7f1ba4caf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba00c8dd0 con 0x7f1ba00c88c0 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.858+0000 7f1ba4caf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00be2e0 con 0x7f1ba0158b60 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9e59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1ba014f360 0x7f1ba00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56916/0 (socket says 172.21.15.94:56916) 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9e59c640 1 -- 172.21.15.94:0/1693795495 learned_addr learned my addr 172.21.15.94:0/1693795495 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9ed9d640 1 --2- 172.21.15.94:0/1693795495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba0158b60 0x7f1ba0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9dd9b640 1 --2- 172.21.15.94:0/1693795495 >> v2:172.21.15.145:3300/0 conn(0x7f1ba00c88c0 0x7f1ba0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9d59a640 1 -- 172.21.15.94:0/1693795495 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349331406 0 0) 0x7f1ba00c73d0 con 0x7f1ba014f360 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9d59a640 1 -- 172.21.15.94:0/1693795495 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c003520 con 0x7f1ba014f360 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9ed9d640 1 -- 172.21.15.94:0/1693795495 >> v2:172.21.15.145:3300/0 conn(0x7f1ba00c88c0 msgr2=0x7f1ba0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9ed9d640 1 --2- 172.21.15.94:0/1693795495 >> v2:172.21.15.145:3300/0 conn(0x7f1ba00c88c0 0x7f1ba0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9ed9d640 1 -- 172.21.15.94:0/1693795495 >> v1:172.21.15.94:6789/0 conn(0x7f1ba014f360 legacy=0x7f1ba00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9ed9d640 1 -- 172.21.15.94:0/1693795495 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00c7670 con 0x7f1ba0158b60 2024-03-06T07:27:35.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.859+0000 7f1b9dd9b640 1 --2- 172.21.15.94:0/1693795495 >> v2:172.21.15.145:3300/0 conn(0x7f1ba00c88c0 0x7f1ba0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:35.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.860+0000 7f1b9ed9d640 1 --2- 172.21.15.94:0/1693795495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba0158b60 0x7f1ba0158ff0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009d90 tx=0x7f1b9002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89c63bcdc3e9cac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.860+0000 7f1b9d59a640 1 -- 172.21.15.94:0/1693795495 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90039070 con 0x7f1ba0158b60 2024-03-06T07:27:35.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.860+0000 7f1b9d59a640 1 -- 172.21.15.94:0/1693795495 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9002cbc0 con 0x7f1ba0158b60 2024-03-06T07:27:35.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.860+0000 7f1b9d59a640 1 -- 172.21.15.94:0/1693795495 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90034440 con 0x7f1ba0158b60 2024-03-06T07:27:35.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.860+0000 7f1ba4caf640 1 -- 172.21.15.94:0/1693795495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba0158b60 msgr2=0x7f1ba0158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.860+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/1693795495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba0158b60 0x7f1ba0158ff0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009d90 tx=0x7f1b9002be10 comp rx=0 tx=0).stop 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.861+0000 7f1ba4caf640 1 -- 172.21.15.94:0/1693795495 shutdown_connections 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.861+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/1693795495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba0158b60 0x7f1ba0158ff0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.861+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/1693795495 >> v2:172.21.15.145:3300/0 conn(0x7f1ba00c88c0 0x7f1ba0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.861+0000 7f1ba4caf640 1 -- 172.21.15.94:0/1693795495 >> 172.21.15.94:0/1693795495 conn(0x7f1ba00ac570 msgr2=0x7f1ba00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.861+0000 7f1ba4caf640 1 -- 172.21.15.94:0/1693795495 shutdown_connections 2024-03-06T07:27:35.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.861+0000 7f1ba4caf640 1 -- 172.21.15.94:0/1693795495 wait complete. 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.862+0000 7f1ba4caf640 1 Processor -- start 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.862+0000 7f1ba4caf640 1 -- start start 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba00c88c0 0x7f1ba00c0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ba014f360 0x7f1ba00c1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ba0158b60 0x7f1ba014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00c5a50 con 0x7f1ba014f360 2024-03-06T07:27:35.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00c7c30 con 0x7f1ba0158b60 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00c5bd0 con 0x7f1ba00c88c0 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ba014f360 0x7f1ba00c1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ba014f360 0x7f1ba00c1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35984/0 (socket says 172.21.15.94:35984) 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 -- 172.21.15.94:0/3001947981 learned_addr learned my addr 172.21.15.94:0/3001947981 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9e59c640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba00c88c0 0x7f1ba00c0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9ed9d640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ba0158b60 0x7f1ba014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 -- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba00c88c0 msgr2=0x7f1ba00c0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba00c88c0 0x7f1ba00c0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 -- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ba0158b60 msgr2=0x7f1ba014b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ba0158b60 0x7f1ba014b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 -- 172.21.15.94:0/3001947981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba01eab20 con 0x7f1ba014f360 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b9dd9b640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ba014f360 0x7f1ba00c1200 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800e930 tx=0x7f1b8800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b96ffd640 1 -- 172.21.15.94:0/3001947981 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88017070 con 0x7f1ba014f360 2024-03-06T07:27:35.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b96ffd640 1 -- 172.21.15.94:0/3001947981 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8800ca00 con 0x7f1ba014f360 2024-03-06T07:27:35.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1b96ffd640 1 -- 172.21.15.94:0/3001947981 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8800cdf0 con 0x7f1ba014f360 2024-03-06T07:27:35.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba00be590 con 0x7f1ba014f360 2024-03-06T07:27:35.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.863+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba0152b10 con 0x7f1ba014f360 2024-03-06T07:27:35.868 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.865+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba004f330 con 0x7f1ba014f360 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.865+0000 7f1b96ffd640 1 -- 172.21.15.94:0/3001947981 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b8801a070 con 0x7f1ba014f360 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.865+0000 7f1b96ffd640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b8006b4d0 0x7f1b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.866+0000 7f1b96ffd640 1 -- 172.21.15.94:0/3001947981 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b8808a190 con 0x7f1ba014f360 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.866+0000 7f1b9e59c640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b8006b4d0 0x7f1b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.869 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.866+0000 7f1b9e59c640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b8006b4d0 0x7f1b8006d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1b98005880 tx=0x7f1b98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.871 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:35.870+0000 7f1b96ffd640 1 -- 172.21.15.94:0/3001947981 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b880588c0 con 0x7f1ba014f360 2024-03-06T07:27:36.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.075+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba006cda0 con 0x7f1ba014f360 2024-03-06T07:27:36.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.076+0000 7f1b96ffd640 1 -- 172.21.15.94:0/3001947981 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f1b8805c960 con 0x7f1ba014f360 2024-03-06T07:27:36.078 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:36.078 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:36.078 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:36.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b8006b4d0 msgr2=0x7f1b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b8006b4d0 0x7f1b8006d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1b98005880 tx=0x7f1b98009040 comp rx=0 tx=0).stop 2024-03-06T07:27:36.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ba014f360 msgr2=0x7f1ba00c1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.080 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ba014f360 0x7f1ba00c1200 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800e930 tx=0x7f1b8800edf0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 shutdown_connections 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1b8006b4d0 0x7f1b8006d990 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ba0158b60 0x7f1ba014b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ba014f360 0x7f1ba00c1200 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 --2- 172.21.15.94:0/3001947981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ba00c88c0 0x7f1ba00c0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 >> 172.21.15.94:0/3001947981 conn(0x7f1ba00ac570 msgr2=0x7f1ba006b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 shutdown_connections 2024-03-06T07:27:36.081 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:36.078+0000 7f1ba4caf640 1 -- 172.21.15.94:0/3001947981 wait complete. 2024-03-06T07:27:36.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:36.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:37.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:37.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.603+0000 7f2c33c40640 1 Processor -- start 2024-03-06T07:27:37.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.603+0000 7f2c33c40640 1 -- start start 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.604+0000 7f2c33c40640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2c2c150580 0x7f2c2c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.604+0000 7f2c33c40640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c153a60 con 0x7f2c2c0c1320 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.604+0000 7f2c33c40640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c153d40 con 0x7f2c2c0c6000 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.604+0000 7f2c33c40640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2c06c2c0 con 0x7f2c2c150580 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.604+0000 7f2c311db640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2c2c0c1320 0x7f2c2c1566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:56920/0 (socket says 172.21.15.94:56920) 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.604+0000 7f2c311db640 1 -- 172.21.15.94:0/2573153384 learned_addr learned my addr 172.21.15.94:0/2573153384 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.604+0000 7f2c319dc640 1 --2- 172.21.15.94:0/2573153384 >> v2:172.21.15.145:3300/0 conn(0x7f2c2c150580 0x7f2c2c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.605+0000 7f2c309da640 1 -- 172.21.15.94:0/2573153384 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635334850 0 0) 0x7f2c2c153a60 con 0x7f2c2c0c1320 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.605+0000 7f2c309da640 1 -- 172.21.15.94:0/2573153384 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c18003520 con 0x7f2c2c0c1320 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.605+0000 7f2c309da640 1 -- 172.21.15.94:0/2573153384 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049233278 0 0) 0x7f2c2c153d40 con 0x7f2c2c0c6000 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.605+0000 7f2c309da640 1 -- 172.21.15.94:0/2573153384 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2c153a60 con 0x7f2c2c0c6000 2024-03-06T07:27:37.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.605+0000 7f2c319dc640 1 -- 172.21.15.94:0/2573153384 >> v1:172.21.15.145:6789/0 conn(0x7f2c2c0c6000 legacy=0x7f2c2c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.605+0000 7f2c319dc640 1 -- 172.21.15.94:0/2573153384 >> v1:172.21.15.94:6789/0 conn(0x7f2c2c0c1320 legacy=0x7f2c2c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.605+0000 7f2c319dc640 1 -- 172.21.15.94:0/2573153384 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c06c5a0 con 0x7f2c2c150580 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.606+0000 7f2c319dc640 1 --2- 172.21.15.94:0/2573153384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 0x7f2c2c06dd10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2c280097a0 tx=0x7f2c2802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=137a6879c4d8f6e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.606+0000 7f2c309da640 1 -- 172.21.15.94:0/2573153384 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28035020 con 0x7f2c2c150580 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.606+0000 7f2c309da640 1 -- 172.21.15.94:0/2573153384 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c28031830 con 0x7f2c2c150580 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.606+0000 7f2c309da640 1 -- 172.21.15.94:0/2573153384 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28031c00 con 0x7f2c2c150580 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.606+0000 7f2c33c40640 1 -- 172.21.15.94:0/2573153384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 msgr2=0x7f2c2c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.606+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2573153384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 0x7f2c2c06dd10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2c280097a0 tx=0x7f2c2802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.607+0000 7f2c33c40640 1 -- 172.21.15.94:0/2573153384 shutdown_connections 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.607+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2573153384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 0x7f2c2c06dd10 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.607+0000 7f2c33c40640 1 -- 172.21.15.94:0/2573153384 >> 172.21.15.94:0/2573153384 conn(0x7f2c2c0c6710 msgr2=0x7f2c2c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.607+0000 7f2c33c40640 1 -- 172.21.15.94:0/2573153384 shutdown_connections 2024-03-06T07:27:37.610 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.607+0000 7f2c33c40640 1 -- 172.21.15.94:0/2573153384 wait complete. 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 Processor -- start 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 -- start start 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c2c0c1320 0x7f2c2c1ec550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c2c0c6000 0x7f2c2c149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 0x7f2c2c1e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c0c0dc0 con 0x7f2c2c0c6000 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c0c0c40 con 0x7f2c2c150580 2024-03-06T07:27:37.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.608+0000 7f2c33c40640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c06e250 con 0x7f2c2c0c1320 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c2c0c6000 0x7f2c2c149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c2c0c6000 0x7f2c2c149110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:35990/0 (socket says 172.21.15.94:35990) 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 -- 172.21.15.94:0/2616723517 learned_addr learned my addr 172.21.15.94:0/2616723517 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c319dc640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c2c0c1320 0x7f2c2c1ec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c321dd640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 0x7f2c2c1e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 -- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c2c0c1320 msgr2=0x7f2c2c1ec550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c2c0c1320 0x7f2c2c1ec550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 -- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 msgr2=0x7f2c2c1e8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 0x7f2c2c1e8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 -- 172.21.15.94:0/2616723517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c072a50 con 0x7f2c2c0c6000 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c311db640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c2c0c6000 0x7f2c2c149110 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c530 tx=0x7f2c1c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c227fc640 1 -- 172.21.15.94:0/2616723517 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c017070 con 0x7f2c2c0c6000 2024-03-06T07:27:37.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c227fc640 1 -- 172.21.15.94:0/2616723517 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c005ce0 con 0x7f2c2c0c6000 2024-03-06T07:27:37.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c227fc640 1 -- 172.21.15.94:0/2616723517 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c007270 con 0x7f2c2c0c6000 2024-03-06T07:27:37.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c072d30 con 0x7f2c2c0c6000 2024-03-06T07:27:37.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.609+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2c0732f0 con 0x7f2c2c0c6000 2024-03-06T07:27:37.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.610+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c04f330 con 0x7f2c2c0c6000 2024-03-06T07:27:37.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.611+0000 7f2c227fc640 1 -- 172.21.15.94:0/2616723517 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c1c01a070 con 0x7f2c2c0c6000 2024-03-06T07:27:37.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.611+0000 7f2c227fc640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.611+0000 7f2c227fc640 1 -- 172.21.15.94:0/2616723517 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2c1c089570 con 0x7f2c2c0c6000 2024-03-06T07:27:37.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.611+0000 7f2c319dc640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.612+0000 7f2c319dc640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2c1006b4d0 0x7f2c1006d990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f2c28036000 tx=0x7f2c2803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.617 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.616+0000 7f2c227fc640 1 -- 172.21.15.94:0/2616723517 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c1c057d60 con 0x7f2c2c0c6000 2024-03-06T07:27:37.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.823+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c2c0c5de0 con 0x7f2c2c0c6000 2024-03-06T07:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.823+0000 7f2c227fc640 1 -- 172.21.15.94:0/2616723517 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f2c1c05be00 con 0x7f2c2c0c6000 2024-03-06T07:27:37.825 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:37.825 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:37.825 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2c1006b4d0 msgr2=0x7f2c1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2c1006b4d0 0x7f2c1006d990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f2c28036000 tx=0x7f2c2803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c2c0c6000 msgr2=0x7f2c2c149110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c2c0c6000 0x7f2c2c149110 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c530 tx=0x7f2c1c00c9f0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 shutdown_connections 2024-03-06T07:27:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c2c150580 0x7f2c2c1e8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2c2c0c6000 0x7f2c2c149110 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 --2- 172.21.15.94:0/2616723517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c2c0c1320 0x7f2c2c1ec550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 >> 172.21.15.94:0/2616723517 conn(0x7f2c2c0c6710 msgr2=0x7f2c2c14e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 shutdown_connections 2024-03-06T07:27:37.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:37.825+0000 7f2c33c40640 1 -- 172.21.15.94:0/2616723517 wait complete. 2024-03-06T07:27:38.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:38.241 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:39.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:39.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.351+0000 7f2ef9179640 1 Processor -- start 2024-03-06T07:27:39.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.351+0000 7f2ef9179640 1 -- start start 2024-03-06T07:27:39.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef9179640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef414f360 0x7f2ef4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef9179640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2ef40c88c0 0x7f2ef40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef9179640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40c73d0 con 0x7f2ef40c5880 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef9179640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef40c8dd0 con 0x7f2ef40c88c0 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef9179640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40be2e0 con 0x7f2ef414f360 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef259c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2ef40c88c0 0x7f2ef40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef359e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2ef40c5880 0x7f2ef4158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35812/0 (socket says 172.21.15.94:35812) 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef359e640 1 -- 172.21.15.94:0/1497619950 learned_addr learned my addr 172.21.15.94:0/1497619950 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.352+0000 7f2ef2d9d640 1 --2- 172.21.15.94:0/1497619950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef414f360 0x7f2ef4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef259c640 1 -- 172.21.15.94:0/1497619950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef414f360 msgr2=0x7f2ef4155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef259c640 1 --2- 172.21.15.94:0/1497619950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef414f360 0x7f2ef4155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef259c640 1 -- 172.21.15.94:0/1497619950 >> v1:172.21.15.94:6789/0 conn(0x7f2ef40c5880 legacy=0x7f2ef4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef259c640 1 -- 172.21.15.94:0/1497619950 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40c7670 con 0x7f2ef40c88c0 2024-03-06T07:27:39.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef1d9b640 1 -- 172.21.15.94:0/1497619950 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064129228 0 0) 0x7f2ef40c73d0 con 0x7f2ef40c5880 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef259c640 1 --2- 172.21.15.94:0/1497619950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef40c88c0 0x7f2ef40c5340 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f5e3937489f812 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef1d9b640 1 -- 172.21.15.94:0/1497619950 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee80315a0 con 0x7f2ef40c88c0 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.353+0000 7f2ef1d9b640 1 -- 172.21.15.94:0/1497619950 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee8037070 con 0x7f2ef40c88c0 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef1d9b640 1 -- 172.21.15.94:0/1497619950 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8031960 con 0x7f2ef40c88c0 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 -- 172.21.15.94:0/1497619950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef40c88c0 msgr2=0x7f2ef40c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 --2- 172.21.15.94:0/1497619950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef40c88c0 0x7f2ef40c5340 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 -- 172.21.15.94:0/1497619950 shutdown_connections 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 --2- 172.21.15.94:0/1497619950 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef40c88c0 0x7f2ef40c5340 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 --2- 172.21.15.94:0/1497619950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef414f360 0x7f2ef4155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 -- 172.21.15.94:0/1497619950 >> 172.21.15.94:0/1497619950 conn(0x7f2ef40ac570 msgr2=0x7f2ef40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 -- 172.21.15.94:0/1497619950 shutdown_connections 2024-03-06T07:27:39.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.354+0000 7f2ef9179640 1 -- 172.21.15.94:0/1497619950 wait complete. 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 Processor -- start 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 -- start start 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef40c5880 0x7f2ef40ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ef40c88c0 0x7f2ef40aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef414f360 0x7f2ef414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4159270 con 0x7f2ef414f360 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40c7c30 con 0x7f2ef40c88c0 2024-03-06T07:27:39.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef9179640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef41593f0 con 0x7f2ef40c5880 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef414f360 0x7f2ef414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef414f360 0x7f2ef414b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40354/0 (socket says 172.21.15.94:40354) 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef359e640 1 -- 172.21.15.94:0/2768979016 learned_addr learned my addr 172.21.15.94:0/2768979016 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef2d9d640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef40c5880 0x7f2ef40ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.356+0000 7f2ef259c640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ef40c88c0 0x7f2ef40aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef359e640 1 -- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef40c5880 msgr2=0x7f2ef40ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef359e640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef40c5880 0x7f2ef40ae560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef359e640 1 -- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ef40c88c0 msgr2=0x7f2ef40aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef359e640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ef40c88c0 0x7f2ef40aeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef359e640 1 -- 172.21.15.94:0/2768979016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40adda0 con 0x7f2ef414f360 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef359e640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef414f360 0x7f2ef414b790 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00c480 tx=0x7f2eec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2edb7fe640 1 -- 172.21.15.94:0/2768979016 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec018070 con 0x7f2ef414f360 2024-03-06T07:27:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2edb7fe640 1 -- 172.21.15.94:0/2768979016 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec007500 con 0x7f2ef414f360 2024-03-06T07:27:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef40af800 con 0x7f2ef414f360 2024-03-06T07:27:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2edb7fe640 1 -- 172.21.15.94:0/2768979016 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec004a20 con 0x7f2ef414f360 2024-03-06T07:27:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.357+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef40afdc0 con 0x7f2ef414f360 2024-03-06T07:27:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.358+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef404f330 con 0x7f2ef414f360 2024-03-06T07:27:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.359+0000 7f2edb7fe640 1 -- 172.21.15.94:0/2768979016 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2eec01b070 con 0x7f2ef414f360 2024-03-06T07:27:39.362 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.359+0000 7f2edb7fe640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ee006b4d0 0x7f2ee006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.359+0000 7f2edb7fe640 1 -- 172.21.15.94:0/2768979016 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2eec089530 con 0x7f2ef414f360 2024-03-06T07:27:39.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.359+0000 7f2ef2d9d640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ee006b4d0 0x7f2ee006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.363 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.360+0000 7f2ef2d9d640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ee006b4d0 0x7f2ee006d990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4011340 tx=0x7f2ee400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.363+0000 7f2edb7fe640 1 -- 172.21.15.94:0/2768979016 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2eec057d20 con 0x7f2ef414f360 2024-03-06T07:27:39.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.571+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ef4152b50 con 0x7f2ef414f360 2024-03-06T07:27:39.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.572+0000 7f2edb7fe640 1 -- 172.21.15.94:0/2768979016 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f2eec05bdc0 con 0x7f2ef414f360 2024-03-06T07:27:39.574 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:39.574 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:39.574 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ee006b4d0 msgr2=0x7f2ee006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ee006b4d0 0x7f2ee006d990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4011340 tx=0x7f2ee400b040 comp rx=0 tx=0).stop 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef414f360 msgr2=0x7f2ef414b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef414f360 0x7f2ef414b790 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00c480 tx=0x7f2eec00c940 comp rx=0 tx=0).stop 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 shutdown_connections 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ee006b4d0 0x7f2ee006d990 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2ef414f360 0x7f2ef414b790 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ef40c88c0 0x7f2ef40aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 --2- 172.21.15.94:0/2768979016 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ef40c5880 0x7f2ef40ae560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 >> 172.21.15.94:0/2768979016 conn(0x7f2ef40ac570 msgr2=0x7f2ef40c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 shutdown_connections 2024-03-06T07:27:39.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:39.574+0000 7f2ef9179640 1 -- 172.21.15.94:0/2768979016 wait complete. 2024-03-06T07:27:39.989 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:39.989 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:40.990 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:41.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.099+0000 7f73c371e640 1 Processor -- start 2024-03-06T07:27:41.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.099+0000 7f73c371e640 1 -- start start 2024-03-06T07:27:41.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c371e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f73bc0c1aa0 0x7f73bc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c371e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc155a40 0x7f73bc155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c371e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73bc1563d0 con 0x7f73bc14f790 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c371e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73bc0c2680 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c371e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0c28d0 con 0x7f73bc155a40 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c0cb9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f73bc0c1aa0 0x7f73bc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c14ba640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f73bc14f790 0x7f73bc0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52422/0 (socket says 172.21.15.94:52422) 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c14ba640 1 -- 172.21.15.94:0/2201691069 learned_addr learned my addr 172.21.15.94:0/2201691069 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.100+0000 7f73c1cbb640 1 --2- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc155a40 0x7f73bc155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73b3fff640 1 -- 172.21.15.94:0/2201691069 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369203903 0 0) 0x7f73bc1563d0 con 0x7f73bc14f790 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73b3fff640 1 -- 172.21.15.94:0/2201691069 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a8003520 con 0x7f73bc14f790 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73c0cb9640 1 -- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc155a40 msgr2=0x7f73bc155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73c0cb9640 1 --2- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc155a40 0x7f73bc155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73c0cb9640 1 -- 172.21.15.94:0/2201691069 >> v1:172.21.15.145:6789/0 conn(0x7f73bc14f790 legacy=0x7f73bc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73c0cb9640 1 -- 172.21.15.94:0/2201691069 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc0c2bb0 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73c1cbb640 1 --2- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc155a40 0x7f73bc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:41.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.101+0000 7f73c0cb9640 1 --2- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc0c1aa0 0x7f73bc1529d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097a0 tx=0x7f73ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80aa149339b37b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73b3fff640 1 -- 172.21.15.94:0/2201691069 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac035020 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73b3fff640 1 -- 172.21.15.94:0/2201691069 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac031830 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73b3fff640 1 -- 172.21.15.94:0/2201691069 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac031be0 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 -- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc0c1aa0 msgr2=0x7f73bc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 --2- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc0c1aa0 0x7f73bc1529d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097a0 tx=0x7f73ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 -- 172.21.15.94:0/2201691069 shutdown_connections 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 --2- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc155a40 0x7f73bc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 --2- 172.21.15.94:0/2201691069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc0c1aa0 0x7f73bc1529d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 -- 172.21.15.94:0/2201691069 >> 172.21.15.94:0/2201691069 conn(0x7f73bc0c6710 msgr2=0x7f73bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 -- 172.21.15.94:0/2201691069 shutdown_connections 2024-03-06T07:27:41.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.102+0000 7f73c371e640 1 -- 172.21.15.94:0/2201691069 wait complete. 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.103+0000 7f73c371e640 1 Processor -- start 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.103+0000 7f73c371e640 1 -- start start 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c371e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc0c1aa0 0x7f73bc14db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c371e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc14f790 0x7f73bc151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c371e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc155a40 0x7f73bc14abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c371e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0c0d60 con 0x7f73bc155a40 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c371e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc155520 con 0x7f73bc14f790 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c371e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc0c5e30 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c1cbb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc155a40 0x7f73bc14abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c14ba640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc0c1aa0 0x7f73bc14db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c0cb9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc14f790 0x7f73bc151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c14ba640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc0c1aa0 0x7f73bc14db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53972/0 (socket says 172.21.15.94:53972) 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.104+0000 7f73c14ba640 1 -- 172.21.15.94:0/2644824333 learned_addr learned my addr 172.21.15.94:0/2644824333 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c14ba640 1 -- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc14f790 msgr2=0x7f73bc151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c14ba640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc14f790 0x7f73bc151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c14ba640 1 -- 172.21.15.94:0/2644824333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc155a40 msgr2=0x7f73bc14abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c14ba640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc155a40 0x7f73bc14abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c14ba640 1 -- 172.21.15.94:0/2644824333 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc1dbfa0 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c1cbb640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc155a40 0x7f73bc14abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c14ba640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc0c1aa0 0x7f73bc14db30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c460 tx=0x7f73b800c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73b1ffb640 1 -- 172.21.15.94:0/2644824333 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8017070 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73b1ffb640 1 -- 172.21.15.94:0/2644824333 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b8005ce0 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73bc1dc280 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73b1ffb640 1 -- 172.21.15.94:0/2644824333 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b80072f0 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.105+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73bc14e900 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.106+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc04f330 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.107+0000 7f73b1ffb640 1 -- 172.21.15.94:0/2644824333 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73b801a070 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.107+0000 7f73b1ffb640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f73a006b4d0 0x7f73a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.107+0000 7f73b1ffb640 1 -- 172.21.15.94:0/2644824333 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f73b8089640 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.107+0000 7f73c0cb9640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f73a006b4d0 0x7f73a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.112 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.108+0000 7f73c0cb9640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f73a006b4d0 0x7f73a006d990 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f73ac036000 tx=0x7f73ac008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.116+0000 7f73b1ffb640 1 -- 172.21.15.94:0/2644824333 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b8057d70 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.322+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73bc0c1f40 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.323+0000 7f73b1ffb640 1 -- 172.21.15.94:0/2644824333 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f73b805e0c0 con 0x7f73bc0c1aa0 2024-03-06T07:27:41.325 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:41.325 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:41.325 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f73a006b4d0 msgr2=0x7f73a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f73a006b4d0 0x7f73a006d990 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f73ac036000 tx=0x7f73ac008030 comp rx=0 tx=0).stop 2024-03-06T07:27:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc0c1aa0 msgr2=0x7f73bc14db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc0c1aa0 0x7f73bc14db30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c460 tx=0x7f73b800c860 comp rx=0 tx=0).stop 2024-03-06T07:27:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 shutdown_connections 2024-03-06T07:27:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f73a006b4d0 0x7f73a006d990 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73bc155a40 0x7f73bc14abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73bc14f790 0x7f73bc151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 --2- 172.21.15.94:0/2644824333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73bc0c1aa0 0x7f73bc14db30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 >> 172.21.15.94:0/2644824333 conn(0x7f73bc0c6710 msgr2=0x7f73bc155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 shutdown_connections 2024-03-06T07:27:41.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:41.325+0000 7f73c371e640 1 -- 172.21.15.94:0/2644824333 wait complete. 2024-03-06T07:27:41.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:41.740 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:42.741 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:42.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.851+0000 7f3ae6cdf640 1 Processor -- start 2024-03-06T07:27:42.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.851+0000 7f3ae6cdf640 1 -- start start 2024-03-06T07:27:42.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3ae6cdf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ae00c1aa0 0x7f3ae01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3ae6cdf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae0155a40 0x7f3ae0155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3ae6cdf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae01563d0 con 0x7f3ae014f790 2024-03-06T07:27:42.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3ae6cdf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae00c2680 con 0x7f3ae00c1aa0 2024-03-06T07:27:42.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3ae6cdf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00c28d0 con 0x7f3ae0155a40 2024-03-06T07:27:42.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3adffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ae00c1aa0 0x7f3ae01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3adffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3ae00c1aa0 0x7f3ae01529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40372/0 (socket says 172.21.15.94:40372) 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3adffff640 1 -- 172.21.15.94:0/788755683 learned_addr learned my addr 172.21.15.94:0/788755683 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3ae4a7b640 1 --1- 172.21.15.94:0/788755683 >> v1:172.21.15.145:6789/0 conn(0x7f3ae014f790 0x7f3ae00c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:52436/0 (socket says 172.21.15.94:52436) 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3ae527c640 1 --2- 172.21.15.94:0/788755683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae0155a40 0x7f3ae0155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3adffff640 1 -- 172.21.15.94:0/788755683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae0155a40 msgr2=0x7f3ae0155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3adffff640 1 --2- 172.21.15.94:0/788755683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae0155a40 0x7f3ae0155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.853+0000 7f3adffff640 1 -- 172.21.15.94:0/788755683 >> v1:172.21.15.145:6789/0 conn(0x7f3ae014f790 legacy=0x7f3ae00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.853+0000 7f3adffff640 1 -- 172.21.15.94:0/788755683 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00c2bb0 con 0x7f3ae00c1aa0 2024-03-06T07:27:42.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.852+0000 7f3adf7fe640 1 -- 172.21.15.94:0/788755683 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431185361 0 0) 0x7f3ae01563d0 con 0x7f3ae014f790 2024-03-06T07:27:42.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.853+0000 7f3adffff640 1 --2- 172.21.15.94:0/788755683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae00c1aa0 0x7f3ae01529d0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009870 tx=0x7f3ad402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8808eaf0e2cf93bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.853+0000 7f3adf7fe640 1 -- 172.21.15.94:0/788755683 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4034070 con 0x7f3ae00c1aa0 2024-03-06T07:27:42.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.853+0000 7f3adf7fe640 1 -- 172.21.15.94:0/788755683 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad4030840 con 0x7f3ae00c1aa0 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.853+0000 7f3adf7fe640 1 -- 172.21.15.94:0/788755683 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4030c10 con 0x7f3ae00c1aa0 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/788755683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae00c1aa0 msgr2=0x7f3ae01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/788755683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae00c1aa0 0x7f3ae01529d0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009870 tx=0x7f3ad402b980 comp rx=0 tx=0).stop 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/788755683 shutdown_connections 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/788755683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae0155a40 0x7f3ae0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/788755683 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae00c1aa0 0x7f3ae01529d0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/788755683 >> 172.21.15.94:0/788755683 conn(0x7f3ae00c6710 msgr2=0x7f3ae00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/788755683 shutdown_connections 2024-03-06T07:27:42.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.854+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/788755683 wait complete. 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.855+0000 7f3ae6cdf640 1 Processor -- start 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.855+0000 7f3ae6cdf640 1 -- start start 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae6cdf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ae00c1aa0 0x7f3ae014d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae6cdf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae014f790 0x7f3ae0150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae6cdf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae0155a40 0x7f3ae01e8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae6cdf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00c0d60 con 0x7f3ae0155a40 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae6cdf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae0155520 con 0x7f3ae00c1aa0 2024-03-06T07:27:42.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae6cdf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00c5e30 con 0x7f3ae014f790 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae0155a40 0x7f3ae01e8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae0155a40 0x7f3ae01e8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40386/0 (socket says 172.21.15.94:40386) 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 -- 172.21.15.94:0/924116491 learned_addr learned my addr 172.21.15.94:0/924116491 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae4a7b640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ae00c1aa0 0x7f3ae014d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3adffff640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae014f790 0x7f3ae0150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 -- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae014f790 msgr2=0x7f3ae0150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae014f790 0x7f3ae0150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 -- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ae00c1aa0 msgr2=0x7f3ae014d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ae00c1aa0 0x7f3ae014d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.856+0000 7f3ae527c640 1 -- 172.21.15.94:0/924116491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae0071a70 con 0x7f3ae0155a40 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.857+0000 7f3ae527c640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae0155a40 0x7f3ae01e8cc0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800efc0 tx=0x7f3ad800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.857+0000 7f3add7fa640 1 -- 172.21.15.94:0/924116491 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8018020 con 0x7f3ae0155a40 2024-03-06T07:27:42.860 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.857+0000 7f3add7fa640 1 -- 172.21.15.94:0/924116491 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad8014480 con 0x7f3ae0155a40 2024-03-06T07:27:42.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.857+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae0071d50 con 0x7f3ae0155a40 2024-03-06T07:27:42.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.857+0000 7f3add7fa640 1 -- 172.21.15.94:0/924116491 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8014870 con 0x7f3ae0155a40 2024-03-06T07:27:42.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.857+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae00bb730 con 0x7f3ae0155a40 2024-03-06T07:27:42.861 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.858+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae004f330 con 0x7f3ae0155a40 2024-03-06T07:27:42.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.858+0000 7f3add7fa640 1 -- 172.21.15.94:0/924116491 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ad801b020 con 0x7f3ae0155a40 2024-03-06T07:27:42.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.859+0000 7f3add7fa640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3acc06b4d0 0x7f3acc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.859+0000 7f3add7fa640 1 -- 172.21.15.94:0/924116491 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ad808b040 con 0x7f3ae0155a40 2024-03-06T07:27:42.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.859+0000 7f3ae4a7b640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3acc06b4d0 0x7f3acc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.862 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.859+0000 7f3ae4a7b640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3acc06b4d0 0x7f3acc06d990 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0000f60 tx=0x7f3ad0007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.865 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:42.863+0000 7f3add7fa640 1 -- 172.21.15.94:0/924116491 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ad8022020 con 0x7f3ae0155a40 2024-03-06T07:27:43.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.072+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ae00c1f40 con 0x7f3ae0155a40 2024-03-06T07:27:43.074 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.072+0000 7f3add7fa640 1 -- 172.21.15.94:0/924116491 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f3ad8010040 con 0x7f3ae0155a40 2024-03-06T07:27:43.074 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:43.075 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:43.075 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3acc06b4d0 msgr2=0x7f3acc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3acc06b4d0 0x7f3acc06d990 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0000f60 tx=0x7f3ad0007d50 comp rx=0 tx=0).stop 2024-03-06T07:27:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae0155a40 msgr2=0x7f3ae01e8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae0155a40 0x7f3ae01e8cc0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f3ad800efc0 tx=0x7f3ad800c490 comp rx=0 tx=0).stop 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 shutdown_connections 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3acc06b4d0 0x7f3acc06d990 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ae0155a40 0x7f3ae01e8cc0 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ae014f790 0x7f3ae0150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 --2- 172.21.15.94:0/924116491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ae00c1aa0 0x7f3ae014d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.074+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 >> 172.21.15.94:0/924116491 conn(0x7f3ae00c6710 msgr2=0x7f3ae0155e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.075+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 shutdown_connections 2024-03-06T07:27:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:43.075+0000 7f3ae6cdf640 1 -- 172.21.15.94:0/924116491 wait complete. 2024-03-06T07:27:43.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:43.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:44.491 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:44.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.601+0000 7f399241c640 1 Processor -- start 2024-03-06T07:27:44.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.601+0000 7f399241c640 1 -- start start 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f399241c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f398c14f790 0x7f398c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f399241c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f398c0c1aa0 0x7f398c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f399241c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c5c60 0x7f398c155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f399241c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398c0c2680 con 0x7f398c14f790 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f399241c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398c0c28d0 con 0x7f398c0c1aa0 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f399241c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398c0c2bb0 con 0x7f398c0c5c60 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f398c14f790 0x7f398c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f398c14f790 0x7f398c1529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40392/0 (socket says 172.21.15.94:40392) 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 -- 172.21.15.94:0/1262757327 learned_addr learned my addr 172.21.15.94:0/1262757327 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:44.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398b7fe640 1 --2- 172.21.15.94:0/1262757327 >> v2:172.21.15.145:3300/0 conn(0x7f398c0c1aa0 0x7f398c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f39909b9640 1 --2- 172.21.15.94:0/1262757327 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c5c60 0x7f398c155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 -- 172.21.15.94:0/1262757327 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c5c60 msgr2=0x7f398c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 --2- 172.21.15.94:0/1262757327 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c5c60 0x7f398c155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 -- 172.21.15.94:0/1262757327 >> v2:172.21.15.145:3300/0 conn(0x7f398c0c1aa0 msgr2=0x7f398c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 --2- 172.21.15.94:0/1262757327 >> v2:172.21.15.145:3300/0 conn(0x7f398c0c1aa0 0x7f398c0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f398bfff640 1 -- 172.21.15.94:0/1262757327 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c0c2e90 con 0x7f398c14f790 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.602+0000 7f39909b9640 1 --2- 172.21.15.94:0/1262757327 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c5c60 0x7f398c155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.603+0000 7f398bfff640 1 --2- 172.21.15.94:0/1262757327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1529d0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f398c070280 tx=0x7f398002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73f19b7c26a66f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.608 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.603+0000 7f398affd640 1 -- 172.21.15.94:0/1262757327 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398003b070 con 0x7f398c14f790 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.603+0000 7f398affd640 1 -- 172.21.15.94:0/1262757327 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980036440 con 0x7f398c14f790 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.603+0000 7f398affd640 1 -- 172.21.15.94:0/1262757327 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980036810 con 0x7f398c14f790 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.603+0000 7f399241c640 1 -- 172.21.15.94:0/1262757327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 msgr2=0x7f398c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.603+0000 7f399241c640 1 --2- 172.21.15.94:0/1262757327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1529d0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f398c070280 tx=0x7f398002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.604+0000 7f399241c640 1 -- 172.21.15.94:0/1262757327 shutdown_connections 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.604+0000 7f399241c640 1 --2- 172.21.15.94:0/1262757327 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c5c60 0x7f398c155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.604+0000 7f399241c640 1 --2- 172.21.15.94:0/1262757327 >> v2:172.21.15.145:3300/0 conn(0x7f398c0c1aa0 0x7f398c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.604+0000 7f399241c640 1 --2- 172.21.15.94:0/1262757327 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1529d0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.604+0000 7f399241c640 1 -- 172.21.15.94:0/1262757327 >> 172.21.15.94:0/1262757327 conn(0x7f398c0c6710 msgr2=0x7f398c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.604+0000 7f399241c640 1 -- 172.21.15.94:0/1262757327 shutdown_connections 2024-03-06T07:27:44.609 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.604+0000 7f399241c640 1 -- 172.21.15.94:0/1262757327 wait complete. 2024-03-06T07:27:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.605+0000 7f399241c640 1 Processor -- start 2024-03-06T07:27:44.612 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.605+0000 7f399241c640 1 -- start start 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c1aa0 0x7f398c151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f398c0c5c60 0x7f398c1e6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1e7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c06cc00 con 0x7f398c14f790 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c0c0d60 con 0x7f398c0c5c60 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398c06cd80 con 0x7f398c0c1aa0 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1e7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1e7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40406/0 (socket says 172.21.15.94:40406) 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 -- 172.21.15.94:0/2210756017 learned_addr learned my addr 172.21.15.94:0/2210756017 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f398b7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f398c0c5c60 0x7f398c1e6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f398bfff640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c1aa0 0x7f398c151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 -- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c1aa0 msgr2=0x7f398c151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c1aa0 0x7f398c151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 -- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f398c0c5c60 msgr2=0x7f398c1e6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f398c0c5c60 0x7f398c1e6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 -- 172.21.15.94:0/2210756017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c14be30 con 0x7f398c14f790 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f398b7fe640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f398c0c5c60 0x7f398c1e6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f39909b9640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1e7210 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f3984007ef0 tx=0x7f398400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f3988ff9640 1 -- 172.21.15.94:0/2210756017 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398401a070 con 0x7f398c14f790 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f3988ff9640 1 -- 172.21.15.94:0/2210756017 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3984003b90 con 0x7f398c14f790 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c14c0a0 con 0x7f398c14f790 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f3988ff9640 1 -- 172.21.15.94:0/2210756017 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3984004030 con 0x7f398c14f790 2024-03-06T07:27:44.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.606+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398c14c380 con 0x7f398c14f790 2024-03-06T07:27:44.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.607+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398c0c4040 con 0x7f398c14f790 2024-03-06T07:27:44.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.608+0000 7f3988ff9640 1 -- 172.21.15.94:0/2210756017 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f398401d070 con 0x7f398c14f790 2024-03-06T07:27:44.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.608+0000 7f3988ff9640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f397806b4d0 0x7f397806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.608+0000 7f3988ff9640 1 -- 172.21.15.94:0/2210756017 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f398408a520 con 0x7f398c14f790 2024-03-06T07:27:44.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.608+0000 7f398bfff640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f397806b4d0 0x7f397806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.615 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.609+0000 7f398bfff640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f397806b4d0 0x7f397806d990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3980038000 tx=0x7f3980032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.616 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.612+0000 7f3988ff9640 1 -- 172.21.15.94:0/2210756017 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f398c0c4040 con 0x7f398c14f790 2024-03-06T07:27:44.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.819+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f398c04f330 con 0x7f398c14f790 2024-03-06T07:27:44.822 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.819+0000 7f3988ff9640 1 -- 172.21.15.94:0/2210756017 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f3984010070 con 0x7f398c14f790 2024-03-06T07:27:44.822 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:44.823 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:44.823 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:44.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f397806b4d0 msgr2=0x7f397806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f397806b4d0 0x7f397806d990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3980038000 tx=0x7f3980032000 comp rx=0 tx=0).stop 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 msgr2=0x7f398c1e7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1e7210 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f3984007ef0 tx=0x7f398400d4a0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 shutdown_connections 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f397806b4d0 0x7f397806d990 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f398c14f790 0x7f398c1e7210 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f398c0c5c60 0x7f398c1e6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 --2- 172.21.15.94:0/2210756017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f398c0c1aa0 0x7f398c151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.821+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 >> 172.21.15.94:0/2210756017 conn(0x7f398c0c6710 msgr2=0x7f398c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.822+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 shutdown_connections 2024-03-06T07:27:44.825 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:44.822+0000 7f399241c640 1 -- 172.21.15.94:0/2210756017 wait complete. 2024-03-06T07:27:45.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:45.238 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:45.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:45.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:46.239 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:46.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.347+0000 7f881ddf6640 1 Processor -- start 2024-03-06T07:27:46.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.347+0000 7f881ddf6640 1 -- start start 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f881ddf6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f88181550c0 0x7f88181554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f881ddf6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88181559e0 0x7f881814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f881ddf6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f88180aa180 0x7f881814f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f881ddf6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88180c4610 con 0x7f88180aa180 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f881ddf6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88180aa660 con 0x7f88181550c0 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f881ddf6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88180aa940 con 0x7f88181559e0 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f88180aa180 0x7f881814f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f88180aa180 0x7f881814f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40408/0 (socket says 172.21.15.94:40408) 2024-03-06T07:27:46.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 -- 172.21.15.94:0/4118123289 learned_addr learned my addr 172.21.15.94:0/4118123289 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8816ffd640 1 --2- 172.21.15.94:0/4118123289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88181559e0 0x7f881814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f88177fe640 1 --2- 172.21.15.94:0/4118123289 >> v2:172.21.15.145:3300/0 conn(0x7f88181550c0 0x7f88181554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 -- 172.21.15.94:0/4118123289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88181559e0 msgr2=0x7f881814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 --2- 172.21.15.94:0/4118123289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88181559e0 0x7f881814ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 -- 172.21.15.94:0/4118123289 >> v2:172.21.15.145:3300/0 conn(0x7f88181550c0 msgr2=0x7f88181554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 --2- 172.21.15.94:0/4118123289 >> v2:172.21.15.145:3300/0 conn(0x7f88181550c0 0x7f88181554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 -- 172.21.15.94:0/4118123289 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88180c4860 con 0x7f88180aa180 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f88177fe640 1 --2- 172.21.15.94:0/4118123289 >> v2:172.21.15.145:3300/0 conn(0x7f88181550c0 0x7f88181554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f8817fff640 1 --2- 172.21.15.94:0/4118123289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88180aa180 0x7f881814f140 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f88180c3fc0 tx=0x7f881002dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5cb703ba771299 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f88167fc640 1 -- 172.21.15.94:0/4118123289 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881002e970 con 0x7f88180aa180 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f88167fc640 1 -- 172.21.15.94:0/4118123289 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881002eb10 con 0x7f88180aa180 2024-03-06T07:27:46.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.349+0000 7f88167fc640 1 -- 172.21.15.94:0/4118123289 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810009600 con 0x7f88180aa180 2024-03-06T07:27:46.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 -- 172.21.15.94:0/4118123289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88180aa180 msgr2=0x7f881814f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 --2- 172.21.15.94:0/4118123289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88180aa180 0x7f881814f140 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f88180c3fc0 tx=0x7f881002dfd0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 -- 172.21.15.94:0/4118123289 shutdown_connections 2024-03-06T07:27:46.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 --2- 172.21.15.94:0/4118123289 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88180aa180 0x7f881814f140 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 --2- 172.21.15.94:0/4118123289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88181559e0 0x7f881814ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 --2- 172.21.15.94:0/4118123289 >> v2:172.21.15.145:3300/0 conn(0x7f88181550c0 0x7f88181554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 -- 172.21.15.94:0/4118123289 >> 172.21.15.94:0/4118123289 conn(0x7f88180c6710 msgr2=0x7f88180c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 -- 172.21.15.94:0/4118123289 shutdown_connections 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.350+0000 7f881ddf6640 1 -- 172.21.15.94:0/4118123289 wait complete. 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.351+0000 7f881ddf6640 1 Processor -- start 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.351+0000 7f881ddf6640 1 -- start start 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f881ddf6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88180aa180 0x7f88181ea4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f881ddf6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88181550c0 0x7f8818154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f881ddf6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88181559e0 0x7f88181e64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f881ddf6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88180c3210 con 0x7f88181559e0 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f881ddf6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88180c1520 con 0x7f88181550c0 2024-03-06T07:27:46.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f881ddf6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88180c3390 con 0x7f88180aa180 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88181559e0 0x7f88181e64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88181559e0 0x7f88181e64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40422/0 (socket says 172.21.15.94:40422) 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 -- 172.21.15.94:0/621986646 learned_addr learned my addr 172.21.15.94:0/621986646 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f88177fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88180aa180 0x7f88181ea4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8816ffd640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88181550c0 0x7f8818154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 -- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88180aa180 msgr2=0x7f88181ea4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88180aa180 0x7f88181ea4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 -- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88181550c0 msgr2=0x7f8818154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88181550c0 0x7f8818154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 -- 172.21.15.94:0/621986646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881814bf20 con 0x7f88181559e0 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.352+0000 7f8817fff640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88181559e0 0x7f88181e64a0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f8810038040 tx=0x7f8810004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.353+0000 7f87fffff640 1 -- 172.21.15.94:0/621986646 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810045020 con 0x7f88181559e0 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.353+0000 7f87fffff640 1 -- 172.21.15.94:0/621986646 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8810009e30 con 0x7f88181559e0 2024-03-06T07:27:46.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.353+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8818072c90 con 0x7f88181559e0 2024-03-06T07:27:46.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.353+0000 7f87fffff640 1 -- 172.21.15.94:0/621986646 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881002e970 con 0x7f88181559e0 2024-03-06T07:27:46.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.353+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8818073250 con 0x7f88181559e0 2024-03-06T07:27:46.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.354+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881804f330 con 0x7f88181559e0 2024-03-06T07:27:46.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.354+0000 7f87fffff640 1 -- 172.21.15.94:0/621986646 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8810048020 con 0x7f88181559e0 2024-03-06T07:27:46.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.355+0000 7f87fffff640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f880406b500 0x7f880406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.355+0000 7f87fffff640 1 -- 172.21.15.94:0/621986646 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f88100aec30 con 0x7f88181559e0 2024-03-06T07:27:46.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.355+0000 7f88177fe640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f880406b500 0x7f880406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.355+0000 7f88177fe640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f880406b500 0x7f880406d9c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f8808009950 tx=0x7f8808008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.361 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.360+0000 7f87fffff640 1 -- 172.21.15.94:0/621986646 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f881004f030 con 0x7f88181559e0 2024-03-06T07:27:46.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.566+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8818068450 con 0x7f88181559e0 2024-03-06T07:27:46.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.567+0000 7f87fffff640 1 -- 172.21.15.94:0/621986646 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f881007d400 con 0x7f88181559e0 2024-03-06T07:27:46.570 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:46.570 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:46.570 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f880406b500 msgr2=0x7f880406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f880406b500 0x7f880406d9c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f8808009950 tx=0x7f8808008040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88181559e0 msgr2=0x7f88181e64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88181559e0 0x7f88181e64a0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f8810038040 tx=0x7f8810004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 shutdown_connections 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f880406b500 0x7f880406d9c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88181559e0 0x7f88181e64a0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88181550c0 0x7f8818154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 --2- 172.21.15.94:0/621986646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88180aa180 0x7f88181ea4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 >> 172.21.15.94:0/621986646 conn(0x7f88180c6710 msgr2=0x7f88180c2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 shutdown_connections 2024-03-06T07:27:46.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:46.570+0000 7f881ddf6640 1 -- 172.21.15.94:0/621986646 wait complete. 2024-03-06T07:27:46.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:46.987 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:47.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:48.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.097+0000 7fe4f79ac640 1 Processor -- start 2024-03-06T07:27:48.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.098+0000 7fe4f79ac640 1 -- start start 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.098+0000 7fe4f79ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe4f014f790 0x7fe4f01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.098+0000 7fe4f79ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f0155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.098+0000 7fe4f79ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f01563d0 con 0x7fe4f00c1aa0 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.098+0000 7fe4f79ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f00c2680 con 0x7fe4f014f790 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.098+0000 7fe4f79ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00c28d0 con 0x7fe4f0155a40 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.098+0000 7fe4f5748640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe4f014f790 0x7fe4f01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4f5748640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe4f014f790 0x7fe4f01529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40434/0 (socket says 172.21.15.94:40434) 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4f5748640 1 -- 172.21.15.94:0/2604847853 learned_addr learned my addr 172.21.15.94:0/2604847853 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4f5f49640 1 --2- 172.21.15.94:0/2604847853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f0155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4f5748640 1 -- 172.21.15.94:0/2604847853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 msgr2=0x7fe4f0155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4f5748640 1 --2- 172.21.15.94:0/2604847853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f0155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4f5748640 1 -- 172.21.15.94:0/2604847853 >> v1:172.21.15.145:6789/0 conn(0x7fe4f00c1aa0 legacy=0x7fe4f00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4e7fff640 1 -- 172.21.15.94:0/2604847853 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440742785 0 0) 0x7fe4f01563d0 con 0x7fe4f00c1aa0 2024-03-06T07:27:48.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.099+0000 7fe4f5748640 1 -- 172.21.15.94:0/2604847853 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f00c2bb0 con 0x7fe4f014f790 2024-03-06T07:27:48.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4f5748640 1 --2- 172.21.15.94:0/2604847853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f01529d0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009870 tx=0x7fe4ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa2e701c14d9127 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4e7fff640 1 -- 172.21.15.94:0/2604847853 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec034070 con 0x7fe4f014f790 2024-03-06T07:27:48.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4e7fff640 1 -- 172.21.15.94:0/2604847853 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec030840 con 0x7fe4f014f790 2024-03-06T07:27:48.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4e7fff640 1 -- 172.21.15.94:0/2604847853 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec030c10 con 0x7fe4f014f790 2024-03-06T07:27:48.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4f79ac640 1 -- 172.21.15.94:0/2604847853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 msgr2=0x7fe4f01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/2604847853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f01529d0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009870 tx=0x7fe4ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4f79ac640 1 -- 172.21.15.94:0/2604847853 shutdown_connections 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/2604847853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/2604847853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f01529d0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.100+0000 7fe4f79ac640 1 -- 172.21.15.94:0/2604847853 >> 172.21.15.94:0/2604847853 conn(0x7fe4f00c6710 msgr2=0x7fe4f00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.101+0000 7fe4f79ac640 1 -- 172.21.15.94:0/2604847853 shutdown_connections 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.101+0000 7fe4f79ac640 1 -- 172.21.15.94:0/2604847853 wait complete. 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 Processor -- start 2024-03-06T07:27:48.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 -- start start 2024-03-06T07:27:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4f00c1aa0 0x7fe4f01eca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f00c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f01e9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00c0d60 con 0x7fe4f014f790 2024-03-06T07:27:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0155520 con 0x7fe4f00c1aa0 2024-03-06T07:27:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f79ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00c5e30 con 0x7fe4f0155a40 2024-03-06T07:27:48.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f4f47640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f00c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f4f47640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f00c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:40446/0 (socket says 172.21.15.94:40446) 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f4f47640 1 -- 172.21.15.94:0/312287931 learned_addr learned my addr 172.21.15.94:0/312287931 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f5748640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4f00c1aa0 0x7fe4f01eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.102+0000 7fe4f5f49640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f01e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f4f47640 1 -- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 msgr2=0x7fe4f01e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f4f47640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f01e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f4f47640 1 -- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4f00c1aa0 msgr2=0x7fe4f01eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f4f47640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4f00c1aa0 0x7fe4f01eca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f4f47640 1 -- 172.21.15.94:0/312287931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f01518b0 con 0x7fe4f014f790 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f4f47640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f00c3340 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fe4e000c550 tx=0x7fe4e000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4e5ffb640 1 -- 172.21.15.94:0/312287931 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0017070 con 0x7fe4f014f790 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4e5ffb640 1 -- 172.21.15.94:0/312287931 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e00035f0 con 0x7fe4f014f790 2024-03-06T07:27:48.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f0151b90 con 0x7fe4f014f790 2024-03-06T07:27:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4e5ffb640 1 -- 172.21.15.94:0/312287931 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e00039e0 con 0x7fe4f014f790 2024-03-06T07:27:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.103+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f014d770 con 0x7fe4f014f790 2024-03-06T07:27:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.104+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f00acab0 con 0x7fe4f014f790 2024-03-06T07:27:48.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.105+0000 7fe4e5ffb640 1 -- 172.21.15.94:0/312287931 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4e001a070 con 0x7fe4f014f790 2024-03-06T07:27:48.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.105+0000 7fe4e5ffb640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4d406b4d0 0x7fe4d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.105+0000 7fe4e5ffb640 1 -- 172.21.15.94:0/312287931 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe4e0089690 con 0x7fe4f014f790 2024-03-06T07:27:48.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.105+0000 7fe4f5748640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4d406b4d0 0x7fe4d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.106+0000 7fe4f5748640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4d406b4d0 0x7fe4d406d990 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec036000 tx=0x7fe4ec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.111 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.110+0000 7fe4e5ffb640 1 -- 172.21.15.94:0/312287931 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe4f00acab0 con 0x7fe4f014f790 2024-03-06T07:27:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.317+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4f004f330 con 0x7fe4f014f790 2024-03-06T07:27:48.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.317+0000 7fe4e5ffb640 1 -- 172.21.15.94:0/312287931 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fe4e0057dc0 con 0x7fe4f014f790 2024-03-06T07:27:48.320 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:48.320 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:48.322 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4d406b4d0 msgr2=0x7fe4d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4d406b4d0 0x7fe4d406d990 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec036000 tx=0x7fe4ec008030 comp rx=0 tx=0).stop 2024-03-06T07:27:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 msgr2=0x7fe4f00c3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f00c3340 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fe4e000c550 tx=0x7fe4e000ca10 comp rx=0 tx=0).stop 2024-03-06T07:27:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 shutdown_connections 2024-03-06T07:27:48.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe4d406b4d0 0x7fe4d406d990 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4f0155a40 0x7fe4f01e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe4f014f790 0x7fe4f00c3340 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 --2- 172.21.15.94:0/312287931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4f00c1aa0 0x7fe4f01eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 >> 172.21.15.94:0/312287931 conn(0x7fe4f00c6710 msgr2=0x7fe4f006def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 shutdown_connections 2024-03-06T07:27:48.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:48.319+0000 7fe4f79ac640 1 -- 172.21.15.94:0/312287931 wait complete. 2024-03-06T07:27:48.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:48.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:49.735 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:49.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc958a640 1 Processor -- start 2024-03-06T07:27:49.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc958a640 1 -- start start 2024-03-06T07:27:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc958a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc958a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9fc4158b60 0x7f9fc4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc958a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc40c73d0 con 0x7f9fc414f360 2024-03-06T07:27:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc958a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc40c8dd0 con 0x7f9fc4158b60 2024-03-06T07:27:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc958a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc40be2e0 con 0x7f9fc40c88c0 2024-03-06T07:27:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9fc414f360 0x7f9fc40c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49866/0 (socket says 172.21.15.94:49866) 2024-03-06T07:27:49.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.846+0000 7f9fc2ffd640 1 -- 172.21.15.94:0/3990718318 learned_addr learned my addr 172.21.15.94:0/3990718318 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc27fc640 1 --2- 172.21.15.94:0/3990718318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc37fe640 1 --2- 172.21.15.94:0/3990718318 >> v2:172.21.15.145:3300/0 conn(0x7f9fc4158b60 0x7f9fc4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc1ffb640 1 -- 172.21.15.94:0/3990718318 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379322784 0 0) 0x7f9fc40c73d0 con 0x7f9fc414f360 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc1ffb640 1 -- 172.21.15.94:0/3990718318 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fac003520 con 0x7f9fc414f360 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc27fc640 1 -- 172.21.15.94:0/3990718318 >> v2:172.21.15.145:3300/0 conn(0x7f9fc4158b60 msgr2=0x7f9fc4158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc27fc640 1 --2- 172.21.15.94:0/3990718318 >> v2:172.21.15.145:3300/0 conn(0x7f9fc4158b60 0x7f9fc4158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc27fc640 1 -- 172.21.15.94:0/3990718318 >> v1:172.21.15.94:6789/0 conn(0x7f9fc414f360 legacy=0x7f9fc40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.847+0000 7f9fc27fc640 1 -- 172.21.15.94:0/3990718318 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc40c7670 con 0x7f9fc40c88c0 2024-03-06T07:27:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc27fc640 1 --2- 172.21.15.94:0/3990718318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc4155c10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80097a0 tx=0x7f9fb802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49de9b4d43d29da0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc1ffb640 1 -- 172.21.15.94:0/3990718318 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb80315a0 con 0x7f9fc40c88c0 2024-03-06T07:27:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc1ffb640 1 -- 172.21.15.94:0/3990718318 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb8037070 con 0x7f9fc40c88c0 2024-03-06T07:27:49.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc1ffb640 1 -- 172.21.15.94:0/3990718318 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8031960 con 0x7f9fc40c88c0 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 -- 172.21.15.94:0/3990718318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 msgr2=0x7f9fc4155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 --2- 172.21.15.94:0/3990718318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc4155c10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80097a0 tx=0x7f9fb802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 -- 172.21.15.94:0/3990718318 shutdown_connections 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 --2- 172.21.15.94:0/3990718318 >> v2:172.21.15.145:3300/0 conn(0x7f9fc4158b60 0x7f9fc4158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 --2- 172.21.15.94:0/3990718318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc4155c10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 -- 172.21.15.94:0/3990718318 >> 172.21.15.94:0/3990718318 conn(0x7f9fc40ac570 msgr2=0x7f9fc40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 -- 172.21.15.94:0/3990718318 shutdown_connections 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.848+0000 7f9fc958a640 1 -- 172.21.15.94:0/3990718318 wait complete. 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 Processor -- start 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 -- start start 2024-03-06T07:27:49.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc40c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fc414f360 0x7f9fc40c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9fc4158b60 0x7f9fc41ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc40c5a50 con 0x7f9fc4158b60 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc40c7c30 con 0x7f9fc414f360 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.850+0000 7f9fc958a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc40c5bd0 con 0x7f9fc40c88c0 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9fc4158b60 0x7f9fc41ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9fc4158b60 0x7f9fc41ee220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58320/0 (socket says 172.21.15.94:58320) 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc27fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fc414f360 0x7f9fc40c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 -- 172.21.15.94:0/100336518 learned_addr learned my addr 172.21.15.94:0/100336518 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc2ffd640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc40c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 -- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 msgr2=0x7f9fc40c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc40c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.855 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 -- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fc414f360 msgr2=0x7f9fc40c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fc414f360 0x7f9fc40c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 -- 172.21.15.94:0/100336518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc40c0e80 con 0x7f9fc4158b60 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc37fe640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9fc4158b60 0x7f9fc41ee220 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00e960 tx=0x7f9fbc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fab7fe640 1 -- 172.21.15.94:0/100336518 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc017070 con 0x7f9fc4158b60 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fab7fe640 1 -- 172.21.15.94:0/100336518 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc00c980 con 0x7f9fc4158b60 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc40c1160 con 0x7f9fc4158b60 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fab7fe640 1 -- 172.21.15.94:0/100336518 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc00cd70 con 0x7f9fc4158b60 2024-03-06T07:27:49.856 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.851+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc41ec230 con 0x7f9fc4158b60 2024-03-06T07:27:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.852+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc4069460 con 0x7f9fc4158b60 2024-03-06T07:27:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.853+0000 7f9fab7fe640 1 -- 172.21.15.94:0/100336518 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fbc01a070 con 0x7f9fc4158b60 2024-03-06T07:27:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.853+0000 7f9fab7fe640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9fb006b4d0 0x7f9fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.853+0000 7f9fab7fe640 1 -- 172.21.15.94:0/100336518 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9fbc08a620 con 0x7f9fc4158b60 2024-03-06T07:27:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.853+0000 7f9fc2ffd640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9fb006b4d0 0x7f9fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.854+0000 7f9fc2ffd640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9fb006b4d0 0x7f9fb006d990 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4005c00 tx=0x7f9fb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:49.859 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:49.857+0000 7f9fab7fe640 1 -- 172.21.15.94:0/100336518 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fc4069460 con 0x7f9fc4158b60 2024-03-06T07:27:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.063+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fc404f330 con 0x7f9fc4158b60 2024-03-06T07:27:50.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.064+0000 7f9fab7fe640 1 -- 172.21.15.94:0/100336518 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f9fbc058d50 con 0x7f9fc4158b60 2024-03-06T07:27:50.066 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:50.066 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:50.066 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.065+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9fb006b4d0 msgr2=0x7f9fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.065+0000 7f9fc958a640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9fb006b4d0 0x7f9fb006d990 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4005c00 tx=0x7f9fb4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.065+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9fc4158b60 msgr2=0x7f9fc41ee220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.065+0000 7f9fc958a640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9fc4158b60 0x7f9fc41ee220 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00e960 tx=0x7f9fbc00ee20 comp rx=0 tx=0).stop 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.065+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 shutdown_connections 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.065+0000 7f9fc958a640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9fb006b4d0 0x7f9fb006d990 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.065+0000 7f9fc958a640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9fc4158b60 0x7f9fc41ee220 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.066+0000 7f9fc958a640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fc414f360 0x7f9fc40c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.066+0000 7f9fc958a640 1 --2- 172.21.15.94:0/100336518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fc40c88c0 0x7f9fc40c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.066+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 >> 172.21.15.94:0/100336518 conn(0x7f9fc40ac570 msgr2=0x7f9fc40c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.066+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 shutdown_connections 2024-03-06T07:27:50.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:50.066+0000 7f9fc958a640 1 -- 172.21.15.94:0/100336518 wait complete. 2024-03-06T07:27:50.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:50.482 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:51.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:51.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.593+0000 7fc9436d4640 1 Processor -- start 2024-03-06T07:27:51.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.593+0000 7fc9436d4640 1 -- start start 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.594+0000 7fc9436d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c156850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.594+0000 7fc9436d4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93c0abca0 con 0x7fc93c0ab730 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.594+0000 7fc9436d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93c15a140 con 0x7fc93c156d90 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.594+0000 7fc9436d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0c4610 con 0x7fc93c0a9a30 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.594+0000 7fc941c71640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc93c0ab730 0x7fc93c163fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49872/0 (socket says 172.21.15.94:49872) 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.594+0000 7fc941c71640 1 -- 172.21.15.94:0/7744230 learned_addr learned my addr 172.21.15.94:0/7744230 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.594+0000 7fc941470640 1 --2- 172.21.15.94:0/7744230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.595+0000 7fc933fff640 1 -- 172.21.15.94:0/7744230 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064296029 0 0) 0x7fc93c0abca0 con 0x7fc93c0ab730 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.595+0000 7fc933fff640 1 -- 172.21.15.94:0/7744230 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc928003520 con 0x7fc93c0ab730 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.595+0000 7fc933fff640 1 -- 172.21.15.94:0/7744230 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711941214 0 0) 0x7fc93c15a140 con 0x7fc93c156d90 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.595+0000 7fc933fff640 1 -- 172.21.15.94:0/7744230 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc93c0abca0 con 0x7fc93c156d90 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.595+0000 7fc941470640 1 -- 172.21.15.94:0/7744230 >> v1:172.21.15.145:6789/0 conn(0x7fc93c156d90 legacy=0x7fc93c0c4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.595+0000 7fc941470640 1 -- 172.21.15.94:0/7744230 >> v1:172.21.15.94:6789/0 conn(0x7fc93c0ab730 legacy=0x7fc93c163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.595+0000 7fc941470640 1 -- 172.21.15.94:0/7744230 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c15a420 con 0x7fc93c0a9a30 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc941470640 1 --2- 172.21.15.94:0/7744230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c156850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0c4790 tx=0x7fc93802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c33856bd42af225e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc933fff640 1 -- 172.21.15.94:0/7744230 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938035020 con 0x7fc93c0a9a30 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc933fff640 1 -- 172.21.15.94:0/7744230 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc938031890 con 0x7fc93c0a9a30 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc933fff640 1 -- 172.21.15.94:0/7744230 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938031c40 con 0x7fc93c0a9a30 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc9436d4640 1 -- 172.21.15.94:0/7744230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 msgr2=0x7fc93c156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc9436d4640 1 --2- 172.21.15.94:0/7744230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c156850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0c4790 tx=0x7fc93802b930 comp rx=0 tx=0).stop 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc9436d4640 1 -- 172.21.15.94:0/7744230 shutdown_connections 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc9436d4640 1 --2- 172.21.15.94:0/7744230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c156850 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc9436d4640 1 -- 172.21.15.94:0/7744230 >> 172.21.15.94:0/7744230 conn(0x7fc93c0c6710 msgr2=0x7fc93c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc9436d4640 1 -- 172.21.15.94:0/7744230 shutdown_connections 2024-03-06T07:27:51.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.596+0000 7fc9436d4640 1 -- 172.21.15.94:0/7744230 wait complete. 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.597+0000 7fc9436d4640 1 Processor -- start 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc9436d4640 1 -- start start 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc9436d4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c14a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc9436d4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93c0ab730 0x7fc93c14a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc9436d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93c156d90 0x7fc93c071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc9436d4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0c0e30 con 0x7fc93c156d90 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc9436d4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0c0cb0 con 0x7fc93c0ab730 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc9436d4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0c1d80 con 0x7fc93c0a9a30 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc941c71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93c156d90 0x7fc93c071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc941470640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c14a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc941c71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93c156d90 0x7fc93c071eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58336/0 (socket says 172.21.15.94:58336) 2024-03-06T07:27:51.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc941c71640 1 -- 172.21.15.94:0/1821601332 learned_addr learned my addr 172.21.15.94:0/1821601332 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.598+0000 7fc940c6f640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93c0ab730 0x7fc93c14a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc941c71640 1 -- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 msgr2=0x7fc93c14a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc941c71640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c14a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc941c71640 1 -- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93c0ab730 msgr2=0x7fc93c14a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc941c71640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93c0ab730 0x7fc93c14a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc941c71640 1 -- 172.21.15.94:0/1821601332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c1f3680 con 0x7fc93c156d90 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc941470640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc941c71640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93c156d90 0x7fc93c071eb0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fc93400cbc0 tx=0x7fc93400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc931ffb640 1 -- 172.21.15.94:0/1821601332 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934017070 con 0x7fc93c156d90 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc931ffb640 1 -- 172.21.15.94:0/1821601332 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93400ad60 con 0x7fc93c156d90 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c072ef0 con 0x7fc93c156d90 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc931ffb640 1 -- 172.21.15.94:0/1821601332 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934012640 con 0x7fc93c156d90 2024-03-06T07:27:51.603 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.599+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c1f0790 con 0x7fc93c156d90 2024-03-06T07:27:51.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.600+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c06e0b0 con 0x7fc93c156d90 2024-03-06T07:27:51.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.601+0000 7fc931ffb640 1 -- 172.21.15.94:0/1821601332 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc93401a070 con 0x7fc93c156d90 2024-03-06T07:27:51.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.601+0000 7fc931ffb640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc92006b4d0 0x7fc92006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:51.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.601+0000 7fc931ffb640 1 -- 172.21.15.94:0/1821601332 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc93408a0b0 con 0x7fc93c156d90 2024-03-06T07:27:51.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.601+0000 7fc941470640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc92006b4d0 0x7fc92006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:51.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.602+0000 7fc941470640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc92006b4d0 0x7fc92006d990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc938036000 tx=0x7fc938008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:51.607 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.605+0000 7fc931ffb640 1 -- 172.21.15.94:0/1821601332 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc93c06e0b0 con 0x7fc93c156d90 2024-03-06T07:27:51.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.812+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc93c0c3860 con 0x7fc93c156d90 2024-03-06T07:27:51.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.813+0000 7fc931ffb640 1 -- 172.21.15.94:0/1821601332 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fc93c0c3860 con 0x7fc93c156d90 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:51.815 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc92006b4d0 msgr2=0x7fc92006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc92006b4d0 0x7fc92006d990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc938036000 tx=0x7fc938008030 comp rx=0 tx=0).stop 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93c156d90 msgr2=0x7fc93c071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93c156d90 0x7fc93c071eb0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fc93400cbc0 tx=0x7fc93400a430 comp rx=0 tx=0).stop 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 shutdown_connections 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc92006b4d0 0x7fc92006d990 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc93c156d90 0x7fc93c071eb0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93c0ab730 0x7fc93c14a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 --2- 172.21.15.94:0/1821601332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93c0a9a30 0x7fc93c14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 >> 172.21.15.94:0/1821601332 conn(0x7fc93c0c6710 msgr2=0x7fc93c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 shutdown_connections 2024-03-06T07:27:51.817 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:51.815+0000 7fc9436d4640 1 -- 172.21.15.94:0/1821601332 wait complete. 2024-03-06T07:27:52.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:52.230 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:53.231 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:53.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.342+0000 7f98a2a0e640 1 Processor -- start 2024-03-06T07:27:53.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.342+0000 7f98a2a0e640 1 -- start start 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a2a0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c0c1aa0 0x7f989c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a2a0e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f989c155a40 0x7f989c155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a2a0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989c156410 con 0x7f989c14f790 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a2a0e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f989c0c2680 con 0x7f989c155a40 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a2a0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0c28d0 con 0x7f989c0c1aa0 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f989bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f989c14f790 0x7f989c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49886/0 (socket says 172.21.15.94:49886) 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f989bfff640 1 -- 172.21.15.94:0/477398523 learned_addr learned my addr 172.21.15.94:0/477398523 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a0fab640 1 --2- 172.21.15.94:0/477398523 >> v2:172.21.15.145:3300/0 conn(0x7f989c155a40 0x7f989c155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f989b7fe640 1 --2- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c0c1aa0 0x7f989c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f989affd640 1 -- 172.21.15.94:0/477398523 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293740793 0 0) 0x7f989c156410 con 0x7f989c14f790 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f989affd640 1 -- 172.21.15.94:0/477398523 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9884003520 con 0x7f989c14f790 2024-03-06T07:27:53.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a0fab640 1 -- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c0c1aa0 msgr2=0x7f989c1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a0fab640 1 --2- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c0c1aa0 0x7f989c1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a0fab640 1 -- 172.21.15.94:0/477398523 >> v1:172.21.15.94:6789/0 conn(0x7f989c14f790 legacy=0x7f989c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f98a0fab640 1 -- 172.21.15.94:0/477398523 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c0c2bb0 con 0x7f989c155a40 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.343+0000 7f989b7fe640 1 --2- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c0c1aa0 0x7f989c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.344+0000 7f98a0fab640 1 --2- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c155a40 0x7f989c155ed0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9894009d90 tx=0x7f989402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c9aad73cb7a99a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.344+0000 7f989affd640 1 -- 172.21.15.94:0/477398523 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894039070 con 0x7f989c155a40 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.344+0000 7f989affd640 1 -- 172.21.15.94:0/477398523 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989402cc50 con 0x7f989c155a40 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.344+0000 7f989affd640 1 -- 172.21.15.94:0/477398523 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894034440 con 0x7f989c155a40 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 -- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c155a40 msgr2=0x7f989c155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c155a40 0x7f989c155ed0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9894009d90 tx=0x7f989402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 -- 172.21.15.94:0/477398523 shutdown_connections 2024-03-06T07:27:53.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c155a40 0x7f989c155ed0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/477398523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c0c1aa0 0x7f989c1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 -- 172.21.15.94:0/477398523 >> 172.21.15.94:0/477398523 conn(0x7f989c0c6710 msgr2=0x7f989c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 -- 172.21.15.94:0/477398523 shutdown_connections 2024-03-06T07:27:53.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.345+0000 7f98a2a0e640 1 -- 172.21.15.94:0/477398523 wait complete. 2024-03-06T07:27:53.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.346+0000 7f98a2a0e640 1 Processor -- start 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.346+0000 7f98a2a0e640 1 -- start start 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a2a0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c0c1aa0 0x7f989c0c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a2a0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c14f790 0x7f989c149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a2a0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f989c155a40 0x7f989c1ea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a2a0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0c0d60 con 0x7f989c155a40 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a2a0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c155520 con 0x7f989c0c1aa0 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a2a0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0c5e30 con 0x7f989c14f790 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f989c155a40 0x7f989c1ea9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f989c155a40 0x7f989c1ea9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58340/0 (socket says 172.21.15.94:58340) 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 -- 172.21.15.94:0/4217612350 learned_addr learned my addr 172.21.15.94:0/4217612350 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f989bfff640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c0c1aa0 0x7f989c0c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f989b7fe640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c14f790 0x7f989c149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 -- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c14f790 msgr2=0x7f989c149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c14f790 0x7f989c149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 -- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c0c1aa0 msgr2=0x7f989c0c48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c0c1aa0 0x7f989c0c48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 -- 172.21.15.94:0/4217612350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c1ee1b0 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a0fab640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f989c155a40 0x7f989c1ea9f0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f9894009eb0 tx=0x7f9894004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f9898ff9640 1 -- 172.21.15.94:0/4217612350 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894039050 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f9898ff9640 1 -- 172.21.15.94:0/4217612350 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9894045070 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.347+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c1ee490 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.348+0000 7f9898ff9640 1 -- 172.21.15.94:0/4217612350 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894009450 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.348+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c1eea50 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.349+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989c1eec50 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.349+0000 7f9898ff9640 1 -- 172.21.15.94:0/4217612350 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f989403e610 con 0x7f989c155a40 2024-03-06T07:27:53.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.349+0000 7f9898ff9640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988806b4d0 0x7f988806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:53.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.349+0000 7f9898ff9640 1 -- 172.21.15.94:0/4217612350 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f98940acc70 con 0x7f989c155a40 2024-03-06T07:27:53.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.350+0000 7f989bfff640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988806b4d0 0x7f988806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:53.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.350+0000 7f989bfff640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988806b4d0 0x7f988806d990 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f988c005880 tx=0x7f988c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:53.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.354+0000 7f9898ff9640 1 -- 172.21.15.94:0/4217612350 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f989c1eec50 con 0x7f989c155a40 2024-03-06T07:27:53.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.562+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f989c04f330 con 0x7f989c155a40 2024-03-06T07:27:53.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.563+0000 7f9898ff9640 1 -- 172.21.15.94:0/4217612350 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f989407b500 con 0x7f989c155a40 2024-03-06T07:27:53.565 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:53.565 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:53.565 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:53.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988806b4d0 msgr2=0x7f988806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988806b4d0 0x7f988806d990 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f988c005880 tx=0x7f988c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:53.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f989c155a40 msgr2=0x7f989c1ea9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f989c155a40 0x7f989c1ea9f0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f9894009eb0 tx=0x7f9894004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 shutdown_connections 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f988806b4d0 0x7f988806d990 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f989c155a40 0x7f989c1ea9f0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f989c14f790 0x7f989c149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 --2- 172.21.15.94:0/4217612350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f989c0c1aa0 0x7f989c0c48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 >> 172.21.15.94:0/4217612350 conn(0x7f989c0c6710 msgr2=0x7f989c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 shutdown_connections 2024-03-06T07:27:53.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:53.565+0000 7f98a2a0e640 1 -- 172.21.15.94:0/4217612350 wait complete. 2024-03-06T07:27:53.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:53.981 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:54.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.092+0000 7f956dbf4640 1 Processor -- start 2024-03-06T07:27:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.092+0000 7f956dbf4640 1 -- start start 2024-03-06T07:27:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.092+0000 7f956dbf4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f956814f790 0x7f95681529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.092+0000 7f956dbf4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9568155a40 0x7f9568155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.092+0000 7f956dbf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95680c28d0 con 0x7f95680c1aa0 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.092+0000 7f956dbf4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95680c2680 con 0x7f9568155a40 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.092+0000 7f956dbf4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95680c2bb0 con 0x7f956814f790 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f9567fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9568155a40 0x7f9568155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f9566ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f95680c1aa0 0x7f95680c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49894/0 (socket says 172.21.15.94:49894) 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f9566ffd640 1 -- 172.21.15.94:0/3804931073 learned_addr learned my addr 172.21.15.94:0/3804931073 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f95677fe640 1 --2- 172.21.15.94:0/3804931073 >> v2:172.21.15.145:3300/0 conn(0x7f956814f790 0x7f95681529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f9567fff640 1 -- 172.21.15.94:0/3804931073 >> v2:172.21.15.145:3300/0 conn(0x7f956814f790 msgr2=0x7f95681529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f9567fff640 1 --2- 172.21.15.94:0/3804931073 >> v2:172.21.15.145:3300/0 conn(0x7f956814f790 0x7f95681529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f9567fff640 1 -- 172.21.15.94:0/3804931073 >> v1:172.21.15.94:6789/0 conn(0x7f95680c1aa0 legacy=0x7f95680c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f9567fff640 1 -- 172.21.15.94:0/3804931073 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95680c2e90 con 0x7f9568155a40 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.093+0000 7f95667fc640 1 -- 172.21.15.94:0/3804931073 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367787617 0 0) 0x7f95680c28d0 con 0x7f95680c1aa0 2024-03-06T07:27:55.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f9567fff640 1 --2- 172.21.15.94:0/3804931073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f9568155e50 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f95680c3170 tx=0x7f956002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8dd8de20faebc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f95667fc640 1 -- 172.21.15.94:0/3804931073 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956002ea60 con 0x7f9568155a40 2024-03-06T07:27:55.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f95667fc640 1 -- 172.21.15.94:0/3804931073 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956002ec00 con 0x7f9568155a40 2024-03-06T07:27:55.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f95667fc640 1 -- 172.21.15.94:0/3804931073 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560009540 con 0x7f9568155a40 2024-03-06T07:27:55.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f956dbf4640 1 -- 172.21.15.94:0/3804931073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 msgr2=0x7f9568155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f956dbf4640 1 --2- 172.21.15.94:0/3804931073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f9568155e50 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f95680c3170 tx=0x7f956002e0c0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f956dbf4640 1 -- 172.21.15.94:0/3804931073 shutdown_connections 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f956dbf4640 1 --2- 172.21.15.94:0/3804931073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f9568155e50 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f956dbf4640 1 --2- 172.21.15.94:0/3804931073 >> v2:172.21.15.145:3300/0 conn(0x7f956814f790 0x7f95681529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.094+0000 7f956dbf4640 1 -- 172.21.15.94:0/3804931073 >> 172.21.15.94:0/3804931073 conn(0x7f95680c6710 msgr2=0x7f95680c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.095+0000 7f956dbf4640 1 -- 172.21.15.94:0/3804931073 shutdown_connections 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.095+0000 7f956dbf4640 1 -- 172.21.15.94:0/3804931073 wait complete. 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 Processor -- start 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 -- start start 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95680c1aa0 0x7f9568148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956814f790 0x7f95680c3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f95680727f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95680c0d60 con 0x7f9568155a40 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9568156390 con 0x7f956814f790 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.096+0000 7f956dbf4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95680c5e30 con 0x7f95680c1aa0 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f95680727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f95680727f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58370/0 (socket says 172.21.15.94:58370) 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 -- 172.21.15.94:0/546262584 learned_addr learned my addr 172.21.15.94:0/546262584 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9566ffd640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956814f790 0x7f95680c3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f95677fe640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95680c1aa0 0x7f9568148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 -- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95680c1aa0 msgr2=0x7f9568148ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95680c1aa0 0x7f9568148ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 -- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956814f790 msgr2=0x7f95680c3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956814f790 0x7f95680c3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 -- 172.21.15.94:0/546262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95681489c0 con 0x7f9568155a40 2024-03-06T07:27:55.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f9567fff640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f95680727f0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f9560038040 tx=0x7f9560004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f954ffff640 1 -- 172.21.15.94:0/546262584 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560045020 con 0x7f9568155a40 2024-03-06T07:27:55.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f954ffff640 1 -- 172.21.15.94:0/546262584 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560040450 con 0x7f9568155a40 2024-03-06T07:27:55.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95681e7370 con 0x7f9568155a40 2024-03-06T07:27:55.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f954ffff640 1 -- 172.21.15.94:0/546262584 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956000b660 con 0x7f9568155a40 2024-03-06T07:27:55.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.097+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95681de140 con 0x7f9568155a40 2024-03-06T07:27:55.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.098+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956804f330 con 0x7f9568155a40 2024-03-06T07:27:55.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.099+0000 7f954ffff640 1 -- 172.21.15.94:0/546262584 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9560048020 con 0x7f9568155a40 2024-03-06T07:27:55.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.099+0000 7f954ffff640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955406b500 0x7f955406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.099+0000 7f954ffff640 1 -- 172.21.15.94:0/546262584 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95600ae970 con 0x7f9568155a40 2024-03-06T07:27:55.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.099+0000 7f95677fe640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955406b500 0x7f955406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.100+0000 7f95677fe640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955406b500 0x7f955406d9c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f956814cdd0 tx=0x7f9558009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.103+0000 7f954ffff640 1 -- 172.21.15.94:0/546262584 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f956004f070 con 0x7f9568155a40 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.312+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95680c0b40 con 0x7f9568155a40 2024-03-06T07:27:55.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.313+0000 7f954ffff640 1 -- 172.21.15.94:0/546262584 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f956007d0a0 con 0x7f9568155a40 2024-03-06T07:27:55.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:55.315 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:55.315 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955406b500 msgr2=0x7f955406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955406b500 0x7f955406d9c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f956814cdd0 tx=0x7f9558009190 comp rx=0 tx=0).stop 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 msgr2=0x7f95680727f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f95680727f0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f9560038040 tx=0x7f9560004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 shutdown_connections 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955406b500 0x7f955406d9c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9568155a40 0x7f95680727f0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f956814f790 0x7f95680c3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 --2- 172.21.15.94:0/546262584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95680c1aa0 0x7f9568148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 >> 172.21.15.94:0/546262584 conn(0x7f95680c6710 msgr2=0x7f95680c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 shutdown_connections 2024-03-06T07:27:55.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:55.315+0000 7f956dbf4640 1 -- 172.21.15.94:0/546262584 wait complete. 2024-03-06T07:27:55.729 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:55.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:56.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.840+0000 7fa9bff19640 1 Processor -- start 2024-03-06T07:27:56.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.840+0000 7fa9bff19640 1 -- start start 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.841+0000 7fa9bff19640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9b81592d0 0x7fa9b81552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.841+0000 7fa9bff19640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b8152d40 con 0x7fa9b8152960 2024-03-06T07:27:56.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.841+0000 7fa9bff19640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b80c1b50 con 0x7fa9b80ac260 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.841+0000 7fa9bff19640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b8066e30 con 0x7fa9b81592d0 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.841+0000 7fa9bdcb5640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa9b8152960 0x7fa9b806dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49898/0 (socket says 172.21.15.94:49898) 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.841+0000 7fa9bdcb5640 1 -- 172.21.15.94:0/2395227745 learned_addr learned my addr 172.21.15.94:0/2395227745 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.841+0000 7fa9bd4b4640 1 --2- 172.21.15.94:0/2395227745 >> v2:172.21.15.94:3300/0 conn(0x7fa9b81592d0 0x7fa9b81552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.842+0000 7fa9bccb3640 1 -- 172.21.15.94:0/2395227745 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272918301 0 0) 0x7fa9b8152d40 con 0x7fa9b8152960 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.842+0000 7fa9bccb3640 1 -- 172.21.15.94:0/2395227745 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a4003520 con 0x7fa9b8152960 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.842+0000 7fa9bd4b4640 1 -- 172.21.15.94:0/2395227745 >> v1:172.21.15.145:6789/0 conn(0x7fa9b80ac260 legacy=0x7fa9b8155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.842+0000 7fa9bd4b4640 1 -- 172.21.15.94:0/2395227745 >> v1:172.21.15.94:6789/0 conn(0x7fa9b8152960 legacy=0x7fa9b806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.842+0000 7fa9bccb3640 1 -- 172.21.15.94:0/2395227745 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733411499 0 0) 0x7fa9b80c1b50 con 0x7fa9b80ac260 2024-03-06T07:27:56.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.842+0000 7fa9bd4b4640 1 -- 172.21.15.94:0/2395227745 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b80c1dc0 con 0x7fa9b81592d0 2024-03-06T07:27:56.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.842+0000 7fa9bd4b4640 1 --2- 172.21.15.94:0/2395227745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b81592d0 0x7fa9b81552d0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80097a0 tx=0x7fa9a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3608b38a128dc9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bccb3640 1 -- 172.21.15.94:0/2395227745 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8034070 con 0x7fa9b81592d0 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bccb3640 1 -- 172.21.15.94:0/2395227745 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a8030610 con 0x7fa9b81592d0 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bccb3640 1 -- 172.21.15.94:0/2395227745 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80309e0 con 0x7fa9b81592d0 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bff19640 1 -- 172.21.15.94:0/2395227745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b81592d0 msgr2=0x7fa9b81552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bff19640 1 --2- 172.21.15.94:0/2395227745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b81592d0 0x7fa9b81552d0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80097a0 tx=0x7fa9a802b960 comp rx=0 tx=0).stop 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bff19640 1 -- 172.21.15.94:0/2395227745 shutdown_connections 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bff19640 1 --2- 172.21.15.94:0/2395227745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b81592d0 0x7fa9b81552d0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bff19640 1 -- 172.21.15.94:0/2395227745 >> 172.21.15.94:0/2395227745 conn(0x7fa9b80c70b0 msgr2=0x7fa9b80c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bff19640 1 -- 172.21.15.94:0/2395227745 shutdown_connections 2024-03-06T07:27:56.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.843+0000 7fa9bff19640 1 -- 172.21.15.94:0/2395227745 wait complete. 2024-03-06T07:27:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.844+0000 7fa9bff19640 1 Processor -- start 2024-03-06T07:27:56.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.844+0000 7fa9bff19640 1 -- start start 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bff19640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9b80ac260 0x7fa9b8120b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bff19640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b8152960 0x7fa9b8157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bff19640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9b81585e0 0x7fa9b8152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bff19640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b806e3a0 con 0x7fa9b8152960 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bff19640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b8159760 con 0x7fa9b81585e0 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bff19640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b806e520 con 0x7fa9b80ac260 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b8152960 0x7fa9b8157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b8152960 0x7fa9b8157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58388/0 (socket says 172.21.15.94:58388) 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 -- 172.21.15.94:0/3033821830 learned_addr learned my addr 172.21.15.94:0/3033821830 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bdcb5640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9b80ac260 0x7fa9b8120b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9be4b6640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9b81585e0 0x7fa9b8152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 -- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9b80ac260 msgr2=0x7fa9b8120b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9b80ac260 0x7fa9b8120b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 -- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9b81585e0 msgr2=0x7fa9b8152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9b81585e0 0x7fa9b8152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bd4b4640 1 -- 172.21.15.94:0/3033821830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b8157210 con 0x7fa9b8152960 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9bdcb5640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9b80ac260 0x7fa9b8120b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.845+0000 7fa9be4b6640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9b81585e0 0x7fa9b8152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.846+0000 7fa9bd4b4640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b8152960 0x7fa9b8157a00 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fa9a802bf70 tx=0x7fa9a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.846+0000 7fa9ae7fc640 1 -- 172.21.15.94:0/3033821830 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8049070 con 0x7fa9b8152960 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.846+0000 7fa9ae7fc640 1 -- 172.21.15.94:0/3033821830 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a803a890 con 0x7fa9b8152960 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.846+0000 7fa9ae7fc640 1 -- 172.21.15.94:0/3033821830 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8034040 con 0x7fa9b8152960 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.846+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b81574f0 con 0x7fa9b8152960 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.846+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b81eb0d0 con 0x7fa9b8152960 2024-03-06T07:27:56.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.847+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b804f330 con 0x7fa9b8152960 2024-03-06T07:27:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.847+0000 7fa9ae7fc640 1 -- 172.21.15.94:0/3033821830 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9a804c070 con 0x7fa9b8152960 2024-03-06T07:27:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.848+0000 7fa9ae7fc640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa99c06b4d0 0x7fa99c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.848+0000 7fa9ae7fc640 1 -- 172.21.15.94:0/3033821830 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa9a80ac210 con 0x7fa9b8152960 2024-03-06T07:27:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.848+0000 7fa9bdcb5640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa99c06b4d0 0x7fa99c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:56.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.848+0000 7fa9bdcb5640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa99c06b4d0 0x7fa99c06d990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4005c00 tx=0x7fa9b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:56.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:56.852+0000 7fa9ae7fc640 1 -- 172.21.15.94:0/3033821830 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9a803f070 con 0x7fa9b8152960 2024-03-06T07:27:57.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.058+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9b8156270 con 0x7fa9b8152960 2024-03-06T07:27:57.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.059+0000 7fa9ae7fc640 1 -- 172.21.15.94:0/3033821830 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fa9a807b0b0 con 0x7fa9b8152960 2024-03-06T07:27:57.061 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:57.061 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:57.061 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa99c06b4d0 msgr2=0x7fa99c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa99c06b4d0 0x7fa99c06d990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4005c00 tx=0x7fa9b4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b8152960 msgr2=0x7fa9b8157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b8152960 0x7fa9b8157a00 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fa9a802bf70 tx=0x7fa9a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 shutdown_connections 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa99c06b4d0 0x7fa99c06d990 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9b81585e0 0x7fa9b8152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9b8152960 0x7fa9b8157a00 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 --2- 172.21.15.94:0/3033821830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9b80ac260 0x7fa9b8120b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 >> 172.21.15.94:0/3033821830 conn(0x7fa9b80c70b0 msgr2=0x7fa9b8155620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 shutdown_connections 2024-03-06T07:27:57.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:57.061+0000 7fa9bff19640 1 -- 172.21.15.94:0/3033821830 wait complete. 2024-03-06T07:27:57.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:57.477 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:27:58.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:58.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.588+0000 7f12bcf13640 1 Processor -- start 2024-03-06T07:27:58.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.588+0000 7f12bcf13640 1 -- start start 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.589+0000 7f12bcf13640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b814f360 0x7f12b8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.589+0000 7f12bcf13640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f12b80c88c0 0x7f12b80c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.589+0000 7f12bcf13640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b80c73d0 con 0x7f12b80c5880 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.589+0000 7f12bcf13640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b80c8dd0 con 0x7f12b80c88c0 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.589+0000 7f12bcf13640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80be2e0 con 0x7f12b814f360 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.589+0000 7f12b6d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f12b80c5880 0x7f12b8158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42628/0 (socket says 172.21.15.94:42628) 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.589+0000 7f12b6d9d640 1 -- 172.21.15.94:0/3518417220 learned_addr learned my addr 172.21.15.94:0/3518417220 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:27:58.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b659c640 1 --2- 172.21.15.94:0/3518417220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b814f360 0x7f12b8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b5d9b640 1 --2- 172.21.15.94:0/3518417220 >> v2:172.21.15.145:3300/0 conn(0x7f12b80c88c0 0x7f12b80c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b559a640 1 -- 172.21.15.94:0/3518417220 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977360606 0 0) 0x7f12b80c73d0 con 0x7f12b80c5880 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b559a640 1 -- 172.21.15.94:0/3518417220 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0003520 con 0x7f12b80c5880 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b659c640 1 -- 172.21.15.94:0/3518417220 >> v2:172.21.15.145:3300/0 conn(0x7f12b80c88c0 msgr2=0x7f12b80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b659c640 1 --2- 172.21.15.94:0/3518417220 >> v2:172.21.15.145:3300/0 conn(0x7f12b80c88c0 0x7f12b80c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b659c640 1 -- 172.21.15.94:0/3518417220 >> v1:172.21.15.94:6789/0 conn(0x7f12b80c5880 legacy=0x7f12b8158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b659c640 1 -- 172.21.15.94:0/3518417220 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b80c7670 con 0x7f12b814f360 2024-03-06T07:27:58.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.590+0000 7f12b5d9b640 1 --2- 172.21.15.94:0/3518417220 >> v2:172.21.15.145:3300/0 conn(0x7f12b80c88c0 0x7f12b80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:58.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12b659c640 1 --2- 172.21.15.94:0/3518417220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b814f360 0x7f12b8155c10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009d90 tx=0x7f12a802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e19bf1c403de8cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12b559a640 1 -- 172.21.15.94:0/3518417220 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8039070 con 0x7f12b814f360 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12b559a640 1 -- 172.21.15.94:0/3518417220 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a802cbc0 con 0x7f12b814f360 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12b559a640 1 -- 172.21.15.94:0/3518417220 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8034440 con 0x7f12b814f360 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12bcf13640 1 -- 172.21.15.94:0/3518417220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b814f360 msgr2=0x7f12b8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12bcf13640 1 --2- 172.21.15.94:0/3518417220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b814f360 0x7f12b8155c10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009d90 tx=0x7f12a802be10 comp rx=0 tx=0).stop 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12bcf13640 1 -- 172.21.15.94:0/3518417220 shutdown_connections 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12bcf13640 1 --2- 172.21.15.94:0/3518417220 >> v2:172.21.15.145:3300/0 conn(0x7f12b80c88c0 0x7f12b80c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.591+0000 7f12bcf13640 1 --2- 172.21.15.94:0/3518417220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b814f360 0x7f12b8155c10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.592+0000 7f12bcf13640 1 -- 172.21.15.94:0/3518417220 >> 172.21.15.94:0/3518417220 conn(0x7f12b80ac570 msgr2=0x7f12b80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.592+0000 7f12bcf13640 1 -- 172.21.15.94:0/3518417220 shutdown_connections 2024-03-06T07:27:58.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.592+0000 7f12bcf13640 1 -- 172.21.15.94:0/3518417220 wait complete. 2024-03-06T07:27:58.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.593+0000 7f12bcf13640 1 Processor -- start 2024-03-06T07:27:58.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.593+0000 7f12bcf13640 1 -- start start 2024-03-06T07:27:58.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12b80c5880 0x7f12b80c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b80c88c0 0x7f12b81ebca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12b814f360 0x7f12b80afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8159270 con 0x7f12b80c5880 2024-03-06T07:27:58.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80c7c30 con 0x7f12b814f360 2024-03-06T07:27:58.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b81593f0 con 0x7f12b80c88c0 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12b80c5880 0x7f12b80c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12b80c5880 0x7f12b80c11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58656/0 (socket says 172.21.15.94:58656) 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 -- 172.21.15.94:0/1641757071 learned_addr learned my addr 172.21.15.94:0/1641757071 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b6d9d640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12b814f360 0x7f12b80afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b5d9b640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b80c88c0 0x7f12b81ebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 -- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b80c88c0 msgr2=0x7f12b81ebca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b80c88c0 0x7f12b81ebca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 -- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12b814f360 msgr2=0x7f12b80afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12b814f360 0x7f12b80afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 -- 172.21.15.94:0/1641757071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b80ad530 con 0x7f12b80c5880 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12b659c640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12b80c5880 0x7f12b80c11a0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f12a8037040 tx=0x7f12a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12a6ffd640 1 -- 172.21.15.94:0/1641757071 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8039050 con 0x7f12b80c5880 2024-03-06T07:27:58.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12a6ffd640 1 -- 172.21.15.94:0/1641757071 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a8045070 con 0x7f12b80c5880 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b80ad810 con 0x7f12b80c5880 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12a6ffd640 1 -- 172.21.15.94:0/1641757071 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8009450 con 0x7f12b80c5880 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.594+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b80addd0 con 0x7f12b80c5880 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.595+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b804f330 con 0x7f12b80c5880 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.596+0000 7f12a6ffd640 1 -- 172.21.15.94:0/1641757071 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12a803e850 con 0x7f12b80c5880 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.596+0000 7f12a6ffd640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f129806b4d0 0x7f129806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.596+0000 7f12a6ffd640 1 -- 172.21.15.94:0/1641757071 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f12a80ad050 con 0x7f12b80c5880 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.596+0000 7f12b5d9b640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f129806b4d0 0x7f129806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:58.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.597+0000 7f12b5d9b640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f129806b4d0 0x7f129806d990 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009560 tx=0x7f12ac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:58.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.601+0000 7f12a6ffd640 1 -- 172.21.15.94:0/1641757071 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12a807b8a0 con 0x7f12b80c5880 2024-03-06T07:27:58.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.809+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12b806cda0 con 0x7f12b80c5880 2024-03-06T07:27:58.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.809+0000 7f12a6ffd640 1 -- 172.21.15.94:0/1641757071 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f12a807f940 con 0x7f12b80c5880 2024-03-06T07:27:58.811 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:27:58.811 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:27:58.811 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:27:58.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f129806b4d0 msgr2=0x7f129806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f129806b4d0 0x7f129806d990 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009560 tx=0x7f12ac009190 comp rx=0 tx=0).stop 2024-03-06T07:27:58.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12b80c5880 msgr2=0x7f12b80c11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:58.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12b80c5880 0x7f12b80c11a0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f12a8037040 tx=0x7f12a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:58.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 shutdown_connections 2024-03-06T07:27:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f129806b4d0 0x7f129806d990 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f12b814f360 0x7f12b80afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f12b80c88c0 0x7f12b81ebca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 --2- 172.21.15.94:0/1641757071 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12b80c5880 0x7f12b80c11a0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 >> 172.21.15.94:0/1641757071 conn(0x7f12b80ac570 msgr2=0x7f12b814dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 shutdown_connections 2024-03-06T07:27:58.814 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:27:58.811+0000 7f12bcf13640 1 -- 172.21.15.94:0/1641757071 wait complete. 2024-03-06T07:27:59.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:27:59.226 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:28:00.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:00.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.337+0000 7fdca55e9640 1 Processor -- start 2024-03-06T07:28:00.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.337+0000 7fdca55e9640 1 -- start start 2024-03-06T07:28:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.337+0000 7fdca55e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.337+0000 7fdca55e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca00c73d0 con 0x7fdca0158b60 2024-03-06T07:28:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.337+0000 7fdca55e9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca00be2e0 con 0x7fdca00c88c0 2024-03-06T07:28:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.337+0000 7fdca55e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca0159700 con 0x7fdca014f360 2024-03-06T07:28:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9f7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdca0158b60 0x7fdca0158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42634/0 (socket says 172.21.15.94:42634) 2024-03-06T07:28:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9f7fe640 1 -- 172.21.15.94:0/1279604159 learned_addr learned my addr 172.21.15.94:0/1279604159 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:00.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9effd640 1 --2- 172.21.15.94:0/1279604159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915919364 0 0) 0x7fdca00c73d0 con 0x7fdca0158b60 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc8c003520 con 0x7fdca0158b60 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170631854 0 0) 0x7fdca00be2e0 con 0x7fdca00c88c0 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca00c73d0 con 0x7fdca00c88c0 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9effd640 1 -- 172.21.15.94:0/1279604159 >> v1:172.21.15.145:6789/0 conn(0x7fdca00c88c0 legacy=0x7fdca00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4015801079 0 0) 0x7fdc8c003520 con 0x7fdca0158b60 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9effd640 1 -- 172.21.15.94:0/1279604159 >> v1:172.21.15.94:6789/0 conn(0x7fdca0158b60 legacy=0x7fdca0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.338+0000 7fdc9effd640 1 -- 172.21.15.94:0/1279604159 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca00c7670 con 0x7fdca014f360 2024-03-06T07:28:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.339+0000 7fdc9effd640 1 --2- 172.21.15.94:0/1279604159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca0155c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdc900097d0 tx=0x7fdc9002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c389f2fe1d3b907c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.339+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc90037070 con 0x7fdca014f360 2024-03-06T07:28:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.339+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc90030510 con 0x7fdca014f360 2024-03-06T07:28:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.339+0000 7fdc9dffb640 1 -- 172.21.15.94:0/1279604159 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc980019f0 con 0x7fdca014f360 2024-03-06T07:28:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.340+0000 7fdca55e9640 1 -- 172.21.15.94:0/1279604159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 msgr2=0x7fdca0155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.340+0000 7fdca55e9640 1 --2- 172.21.15.94:0/1279604159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca0155c10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdc900097d0 tx=0x7fdc9002b770 comp rx=0 tx=0).stop 2024-03-06T07:28:00.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.340+0000 7fdca55e9640 1 -- 172.21.15.94:0/1279604159 shutdown_connections 2024-03-06T07:28:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.340+0000 7fdca55e9640 1 --2- 172.21.15.94:0/1279604159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca0155c10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.340+0000 7fdca55e9640 1 -- 172.21.15.94:0/1279604159 >> 172.21.15.94:0/1279604159 conn(0x7fdca00ac570 msgr2=0x7fdca00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.340+0000 7fdca55e9640 1 -- 172.21.15.94:0/1279604159 shutdown_connections 2024-03-06T07:28:00.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.340+0000 7fdca55e9640 1 -- 172.21.15.94:0/1279604159 wait complete. 2024-03-06T07:28:00.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.341+0000 7fdca55e9640 1 Processor -- start 2024-03-06T07:28:00.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.341+0000 7fdca55e9640 1 -- start start 2024-03-06T07:28:00.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdca55e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca00c88c0 0x7fdca00c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdca55e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca014f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdca55e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca0158b60 0x7fdca01ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdca55e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00c7c30 con 0x7fdca00c88c0 2024-03-06T07:28:00.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdca55e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00c5a50 con 0x7fdca0158b60 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdca55e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00c5bd0 con 0x7fdca014f360 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca00c88c0 0x7fdca00c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca00c88c0 0x7fdca00c91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58660/0 (socket says 172.21.15.94:58660) 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 -- 172.21.15.94:0/616856744 learned_addr learned my addr 172.21.15.94:0/616856744 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9f7fe640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca0158b60 0x7fdca01ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9e7fc640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca014f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 -- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 msgr2=0x7fdca014f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca014f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 -- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca0158b60 msgr2=0x7fdca01ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca0158b60 0x7fdca01ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 -- 172.21.15.94:0/616856744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca0152180 con 0x7fdca00c88c0 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc9effd640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca00c88c0 0x7fdca00c91d0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fdc9002b7b0 tx=0x7fdc900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc977fe640 1 -- 172.21.15.94:0/616856744 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc90042070 con 0x7fdca00c88c0 2024-03-06T07:28:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdc977fe640 1 -- 172.21.15.94:0/616856744 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9003e870 con 0x7fdca00c88c0 2024-03-06T07:28:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.342+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca01eb6b0 con 0x7fdca00c88c0 2024-03-06T07:28:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.343+0000 7fdc977fe640 1 -- 172.21.15.94:0/616856744 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc90037040 con 0x7fdca00c88c0 2024-03-06T07:28:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.343+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca0152440 con 0x7fdca00c88c0 2024-03-06T07:28:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.343+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca014d130 con 0x7fdca00c88c0 2024-03-06T07:28:00.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.344+0000 7fdc977fe640 1 -- 172.21.15.94:0/616856744 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc90045070 con 0x7fdca00c88c0 2024-03-06T07:28:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.344+0000 7fdc977fe640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.344+0000 7fdc977fe640 1 -- 172.21.15.94:0/616856744 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6188+0+0 (secure 0 0 0) 0x7fdc900abeb0 con 0x7fdca00c88c0 2024-03-06T07:28:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.345+0000 7fdc9e7fc640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:00.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.345+0000 7fdc9e7fc640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8006b4d0 0x7fdc8006d990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdca0156020 tx=0x7fdc88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:00.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.349+0000 7fdc977fe640 1 -- 172.21.15.94:0/616856744 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdca014d130 con 0x7fdca00c88c0 2024-03-06T07:28:00.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.556+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdca004f330 con 0x7fdca00c88c0 2024-03-06T07:28:00.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.556+0000 7fdc977fe640 1 -- 172.21.15.94:0/616856744 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fdc9007a620 con 0x7fdca00c88c0 2024-03-06T07:28:00.558 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:00.558 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":200,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193},{"gid":10776,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:27:32.245042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10791":{"gid":10791,"name":"d","rank":1,"incarnation":198,"state":"up:stopping","state_seq":5,"addr":"172.21.15.145:6837/679350833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":679350833},{"type":"v1","addr":"172.21.15.145:6837","nonce":679350833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:00.559 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 200 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8006b4d0 msgr2=0x7fdc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8006b4d0 0x7fdc8006d990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdca0156020 tx=0x7fdc88009040 comp rx=0 tx=0).stop 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca00c88c0 msgr2=0x7fdca00c91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca00c88c0 0x7fdca00c91d0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fdc9002b7b0 tx=0x7fdc900046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 shutdown_connections 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdc8006b4d0 0x7fdc8006d990 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdca0158b60 0x7fdca01ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdca014f360 0x7fdca014f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 --2- 172.21.15.94:0/616856744 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdca00c88c0 0x7fdca00c91d0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 >> 172.21.15.94:0/616856744 conn(0x7fdca00ac570 msgr2=0x7fdca00ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:00.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 shutdown_connections 2024-03-06T07:28:00.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:00.559+0000 7fdca55e9640 1 -- 172.21.15.94:0/616856744 wait complete. 2024-03-06T07:28:00.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 200, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:27:32.245042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10791}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10791': {'gid': 10791, 'name': 'd', 'rank': 1, 'incarnation': 198, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.145:6837/679350833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 679350833}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 679350833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:00.973 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10791:up:stopping 2024-03-06T07:28:01.975 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:02.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.085+0000 7f2744bf4640 1 Processor -- start 2024-03-06T07:28:02.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.085+0000 7f2744bf4640 1 -- start start 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.086+0000 7f2744bf4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f274014f360 0x7f2740155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.086+0000 7f2744bf4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400c88c0 0x7f27400c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.086+0000 7f2744bf4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27400c73d0 con 0x7f27400c5880 2024-03-06T07:28:02.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.086+0000 7f2744bf4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27400c8dd0 con 0x7f274014f360 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.086+0000 7f2744bf4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27400be2e0 con 0x7f27400c88c0 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273e59c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f274014f360 0x7f2740155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273ed9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f27400c5880 0x7f2740158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42636/0 (socket says 172.21.15.94:42636) 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273ed9d640 1 -- 172.21.15.94:0/2440619056 learned_addr learned my addr 172.21.15.94:0/2440619056 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273dd9b640 1 --2- 172.21.15.94:0/2440619056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400c88c0 0x7f27400c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273d59a640 1 -- 172.21.15.94:0/2440619056 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653397965 0 0) 0x7f27400c73d0 con 0x7f27400c5880 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273d59a640 1 -- 172.21.15.94:0/2440619056 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2728003520 con 0x7f27400c5880 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273e59c640 1 -- 172.21.15.94:0/2440619056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400c88c0 msgr2=0x7f27400c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273e59c640 1 --2- 172.21.15.94:0/2440619056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400c88c0 0x7f27400c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273e59c640 1 -- 172.21.15.94:0/2440619056 >> v1:172.21.15.94:6789/0 conn(0x7f27400c5880 legacy=0x7f2740158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273e59c640 1 -- 172.21.15.94:0/2440619056 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400c7670 con 0x7f274014f360 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.087+0000 7f273dd9b640 1 --2- 172.21.15.94:0/2440619056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400c88c0 0x7f27400c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.088+0000 7f273e59c640 1 --2- 172.21.15.94:0/2440619056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f274014f360 0x7f2740155c10 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f2730009d90 tx=0x7f273002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70c9db35e5b3957 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.088+0000 7f273d59a640 1 -- 172.21.15.94:0/2440619056 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730039070 con 0x7f274014f360 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.088+0000 7f273d59a640 1 -- 172.21.15.94:0/2440619056 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273002cbc0 con 0x7f274014f360 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.088+0000 7f273d59a640 1 -- 172.21.15.94:0/2440619056 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730034440 con 0x7f274014f360 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.088+0000 7f2744bf4640 1 -- 172.21.15.94:0/2440619056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f274014f360 msgr2=0x7f2740155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.088+0000 7f2744bf4640 1 --2- 172.21.15.94:0/2440619056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f274014f360 0x7f2740155c10 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f2730009d90 tx=0x7f273002be10 comp rx=0 tx=0).stop 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.089+0000 7f2744bf4640 1 -- 172.21.15.94:0/2440619056 shutdown_connections 2024-03-06T07:28:02.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.089+0000 7f2744bf4640 1 --2- 172.21.15.94:0/2440619056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400c88c0 0x7f27400c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.089+0000 7f2744bf4640 1 --2- 172.21.15.94:0/2440619056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f274014f360 0x7f2740155c10 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.089+0000 7f2744bf4640 1 -- 172.21.15.94:0/2440619056 >> 172.21.15.94:0/2440619056 conn(0x7f27400ac570 msgr2=0x7f27400ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.089+0000 7f2744bf4640 1 -- 172.21.15.94:0/2440619056 shutdown_connections 2024-03-06T07:28:02.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.089+0000 7f2744bf4640 1 -- 172.21.15.94:0/2440619056 wait complete. 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.090+0000 7f2744bf4640 1 Processor -- start 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.090+0000 7f2744bf4640 1 -- start start 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.090+0000 7f2744bf4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27400c5880 0x7f27401eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.090+0000 7f2744bf4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400c88c0 0x7f27400c0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.090+0000 7f2744bf4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f274014f360 0x7f27401ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f2744bf4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2740159270 con 0x7f27400c5880 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f2744bf4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400c7c30 con 0x7f27400c88c0 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f2744bf4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27401593f0 con 0x7f274014f360 2024-03-06T07:28:02.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27400c5880 0x7f27401eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27400c5880 0x7f27401eb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58692/0 (socket says 172.21.15.94:58692) 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 -- 172.21.15.94:0/1927155868 learned_addr learned my addr 172.21.15.94:0/1927155868 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273dd9b640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400c88c0 0x7f27400c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273ed9d640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f274014f360 0x7f27401ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 -- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f274014f360 msgr2=0x7f27401ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f274014f360 0x7f27401ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 -- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400c88c0 msgr2=0x7f27400c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400c88c0 0x7f27400c0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 -- 172.21.15.94:0/1927155868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2740158030 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f273e59c640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27400c5880 0x7f27401eb370 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f2730000f80 tx=0x7f2730004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f272effd640 1 -- 172.21.15.94:0/1927155868 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730039050 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f272effd640 1 -- 172.21.15.94:0/1927155868 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2730045070 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2740158310 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f272effd640 1 -- 172.21.15.94:0/1927155868 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730009450 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.091+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27400ad400 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.092+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2740158830 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.093+0000 7f272effd640 1 -- 172.21.15.94:0/1927155868 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f273004a460 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.093+0000 7f272effd640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f272006b4d0 0x7f272006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.093+0000 7f272effd640 1 -- 172.21.15.94:0/1927155868 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6188+0+0 (secure 0 0 0) 0x7f27300acb90 con 0x7f27400c5880 2024-03-06T07:28:02.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.093+0000 7f273dd9b640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f272006b4d0 0x7f272006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.094+0000 7f273dd9b640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f272006b4d0 0x7f272006d990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f2734009560 tx=0x7f2734009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.097+0000 7f272effd640 1 -- 172.21.15.94:0/1927155868 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f273007b270 con 0x7f27400c5880 2024-03-06T07:28:02.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.311+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f274004f330 con 0x7f27400c5880 2024-03-06T07:28:02.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.312+0000 7f272effd640 1 -- 172.21.15.94:0/1927155868 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4220 (secure 0 0 0) 0x7f273007f310 con 0x7f27400c5880 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":201,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:02.249203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:replay","state_seq":1,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:02.314 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 201 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f272006b4d0 msgr2=0x7f272006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f272006b4d0 0x7f272006d990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f2734009560 tx=0x7f2734009190 comp rx=0 tx=0).stop 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27400c5880 msgr2=0x7f27401eb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27400c5880 0x7f27401eb370 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f2730000f80 tx=0x7f2730004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 shutdown_connections 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f272006b4d0 0x7f272006d990 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f274014f360 0x7f27401ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.313+0000 7f2744bf4640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400c88c0 0x7f27400c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.314+0000 7f2744bf4640 1 --2- 172.21.15.94:0/1927155868 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27400c5880 0x7f27401eb370 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.314+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 >> 172.21.15.94:0/1927155868 conn(0x7f27400ac570 msgr2=0x7f27400ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.314+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 shutdown_connections 2024-03-06T07:28:02.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:02.314+0000 7f2744bf4640 1 -- 172.21.15.94:0/1927155868 wait complete. 2024-03-06T07:28:02.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:02.249203+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:02.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:replay 2024-03-06T07:28:03.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.839+0000 7fb99dc5c640 1 Processor -- start 2024-03-06T07:28:03.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.839+0000 7fb99dc5c640 1 -- start start 2024-03-06T07:28:03.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb99dc5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb9980c6560 0x7fb9981530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb99dc5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb998154c80 0x7fb9981535e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb99dc5c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9980bdc30 con 0x7fb998072f70 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb99dc5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9980bd690 con 0x7fb9980c6560 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb99dc5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980bd950 con 0x7fb998154c80 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb9977fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb998072f70 0x7fb998154570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42642/0 (socket says 172.21.15.94:42642) 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb9977fe640 1 -- 172.21.15.94:0/1039183843 learned_addr learned my addr 172.21.15.94:0/1039183843 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb996ffd640 1 --2- 172.21.15.94:0/1039183843 >> v2:172.21.15.145:3300/0 conn(0x7fb9980c6560 0x7fb9981530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb997fff640 1 --2- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb998154c80 0x7fb9981535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb9967fc640 1 -- 172.21.15.94:0/1039183843 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196217566 0 0) 0x7fb9980bdc30 con 0x7fb998072f70 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb9967fc640 1 -- 172.21.15.94:0/1039183843 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb980003520 con 0x7fb998072f70 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb996ffd640 1 -- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb998154c80 msgr2=0x7fb9981535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.840+0000 7fb996ffd640 1 --2- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb998154c80 0x7fb9981535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.841+0000 7fb996ffd640 1 -- 172.21.15.94:0/1039183843 >> v1:172.21.15.94:6789/0 conn(0x7fb998072f70 legacy=0x7fb998154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.841+0000 7fb996ffd640 1 -- 172.21.15.94:0/1039183843 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9980c5540 con 0x7fb9980c6560 2024-03-06T07:28:03.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.841+0000 7fb997fff640 1 --2- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb998154c80 0x7fb9981535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:03.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.841+0000 7fb996ffd640 1 --2- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9980c6560 0x7fb9981530a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0097a0 tx=0x7fb98c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88d2084f06f92bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.841+0000 7fb9967fc640 1 -- 172.21.15.94:0/1039183843 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c0315a0 con 0x7fb9980c6560 2024-03-06T07:28:03.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.841+0000 7fb9967fc640 1 -- 172.21.15.94:0/1039183843 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98c037070 con 0x7fb9980c6560 2024-03-06T07:28:03.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.841+0000 7fb9967fc640 1 -- 172.21.15.94:0/1039183843 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c031960 con 0x7fb9980c6560 2024-03-06T07:28:03.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 -- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9980c6560 msgr2=0x7fb9981530a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9980c6560 0x7fb9981530a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0097a0 tx=0x7fb98c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 -- 172.21.15.94:0/1039183843 shutdown_connections 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb998154c80 0x7fb9981535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/1039183843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9980c6560 0x7fb9981530a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 -- 172.21.15.94:0/1039183843 >> 172.21.15.94:0/1039183843 conn(0x7fb99806b1a0 msgr2=0x7fb99806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 -- 172.21.15.94:0/1039183843 shutdown_connections 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.842+0000 7fb99dc5c640 1 -- 172.21.15.94:0/1039183843 wait complete. 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.843+0000 7fb99dc5c640 1 Processor -- start 2024-03-06T07:28:03.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.843+0000 7fb99dc5c640 1 -- start start 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb99dc5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb998072f70 0x7fb9980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb99dc5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9980c6560 0x7fb99814d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb99dc5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb998154c80 0x7fb9981ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb99dc5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb998073350 con 0x7fb998072f70 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb99dc5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980bba60 con 0x7fb998154c80 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb99dc5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980734d0 con 0x7fb9980c6560 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb9977fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb998072f70 0x7fb9980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb9977fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb998072f70 0x7fb9980a5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58708/0 (socket says 172.21.15.94:58708) 2024-03-06T07:28:03.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb9977fe640 1 -- 172.21.15.94:0/425694149 learned_addr learned my addr 172.21.15.94:0/425694149 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb996ffd640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9980c6560 0x7fb99814d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.844+0000 7fb997fff640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb998154c80 0x7fb9981ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb9977fe640 1 -- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9980c6560 msgr2=0x7fb99814d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb9977fe640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9980c6560 0x7fb99814d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb9977fe640 1 -- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb998154c80 msgr2=0x7fb9981ede30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb9977fe640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb998154c80 0x7fb9981ede30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb9977fe640 1 -- 172.21.15.94:0/425694149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99806d9b0 con 0x7fb998072f70 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb9977fe640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb998072f70 0x7fb9980a5240 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7fb98800c430 tx=0x7fb98800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb97ffff640 1 -- 172.21.15.94:0/425694149 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988017070 con 0x7fb998072f70 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb97ffff640 1 -- 172.21.15.94:0/425694149 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb988005ce0 con 0x7fb998072f70 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9980aa150 con 0x7fb998072f70 2024-03-06T07:28:03.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb97ffff640 1 -- 172.21.15.94:0/425694149 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9880072f0 con 0x7fb998072f70 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.845+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99806dc70 con 0x7fb998072f70 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.846+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99804f330 con 0x7fb998072f70 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.846+0000 7fb97ffff640 1 -- 172.21.15.94:0/425694149 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb98801a070 con 0x7fb998072f70 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.847+0000 7fb97ffff640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb98406b500 0x7fb98406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.847+0000 7fb97ffff640 1 -- 172.21.15.94:0/425694149 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fb988089720 con 0x7fb998072f70 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.847+0000 7fb996ffd640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb98406b500 0x7fb98406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.847+0000 7fb996ffd640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb98406b500 0x7fb98406d9c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb98c036000 tx=0x7fb98c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:03.851+0000 7fb97ffff640 1 -- 172.21.15.94:0/425694149 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb988057d20 con 0x7fb998072f70 2024-03-06T07:28:04.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.059+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9980c6260 con 0x7fb998072f70 2024-03-06T07:28:04.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.060+0000 7fb97ffff640 1 -- 172.21.15.94:0/425694149 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+4222 (secure 0 0 0) 0x7fb98805bdc0 con 0x7fb998072f70 2024-03-06T07:28:04.062 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:04.062 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":202,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:02.264209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:resolve","state_seq":14,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:04.062 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 202 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb98406b500 msgr2=0x7fb98406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb98406b500 0x7fb98406d9c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb98c036000 tx=0x7fb98c03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb998072f70 msgr2=0x7fb9980a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb998072f70 0x7fb9980a5240 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7fb98800c430 tx=0x7fb98800c830 comp rx=0 tx=0).stop 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 shutdown_connections 2024-03-06T07:28:04.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb98406b500 0x7fb98406d9c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb998154c80 0x7fb9981ede30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9980c6560 0x7fb99814d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 --2- 172.21.15.94:0/425694149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb998072f70 0x7fb9980a5240 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 >> 172.21.15.94:0/425694149 conn(0x7fb99806b1a0 msgr2=0x7fb9980c1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 shutdown_connections 2024-03-06T07:28:04.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:04.062+0000 7fb99dc5c640 1 -- 172.21.15.94:0/425694149 wait complete. 2024-03-06T07:28:04.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 202, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:02.264209+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:resolve', 'state_seq': 14, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:04.477 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:resolve 2024-03-06T07:28:05.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:05.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.588+0000 7f46c729d640 1 Processor -- start 2024-03-06T07:28:05.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.588+0000 7f46c729d640 1 -- start start 2024-03-06T07:28:05.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c729d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c729d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f46c01559e0 0x7f46c014ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c729d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f46c00aa180 0x7f46c014f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c729d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c00c4610 con 0x7f46c01559e0 2024-03-06T07:28:05.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c729d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c00aa660 con 0x7f46c00aa180 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c729d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00aa940 con 0x7f46c01550c0 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f46c01559e0 0x7f46c014ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f46c01559e0 0x7f46c014ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58712/0 (socket says 172.21.15.94:58712) 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 -- 172.21.15.94:0/810094450 learned_addr learned my addr 172.21.15.94:0/810094450 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c583a640 1 --2- 172.21.15.94:0/810094450 >> v2:172.21.15.145:3300/0 conn(0x7f46c00aa180 0x7f46c014f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c5039640 1 --2- 172.21.15.94:0/810094450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 -- 172.21.15.94:0/810094450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 msgr2=0x7f46c01554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 --2- 172.21.15.94:0/810094450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 -- 172.21.15.94:0/810094450 >> v2:172.21.15.145:3300/0 conn(0x7f46c00aa180 msgr2=0x7f46c014f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 --2- 172.21.15.94:0/810094450 >> v2:172.21.15.145:3300/0 conn(0x7f46c00aa180 0x7f46c014f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.589+0000 7f46c4838640 1 -- 172.21.15.94:0/810094450 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00c4860 con 0x7f46c01559e0 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.590+0000 7f46c4838640 1 --2- 172.21.15.94:0/810094450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c014ec00 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f46c00c3d10 tx=0x7f46b002dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3190aa735b3e1e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.590+0000 7f46b7fff640 1 -- 172.21.15.94:0/810094450 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b003b070 con 0x7f46c01559e0 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.590+0000 7f46b7fff640 1 -- 172.21.15.94:0/810094450 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b002ed70 con 0x7f46c01559e0 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.590+0000 7f46b7fff640 1 -- 172.21.15.94:0/810094450 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0036660 con 0x7f46c01559e0 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 -- 172.21.15.94:0/810094450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 msgr2=0x7f46c014ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 --2- 172.21.15.94:0/810094450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c014ec00 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f46c00c3d10 tx=0x7f46b002dfc0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 -- 172.21.15.94:0/810094450 shutdown_connections 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 --2- 172.21.15.94:0/810094450 >> v2:172.21.15.145:3300/0 conn(0x7f46c00aa180 0x7f46c014f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 --2- 172.21.15.94:0/810094450 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c014ec00 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 --2- 172.21.15.94:0/810094450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 -- 172.21.15.94:0/810094450 >> 172.21.15.94:0/810094450 conn(0x7f46c00c6710 msgr2=0x7f46c00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 -- 172.21.15.94:0/810094450 shutdown_connections 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.591+0000 7f46c729d640 1 -- 172.21.15.94:0/810094450 wait complete. 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.592+0000 7f46c729d640 1 Processor -- start 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.592+0000 7f46c729d640 1 -- start start 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.592+0000 7f46c729d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c00aa180 0x7f46c014a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.592+0000 7f46c729d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01e9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.592+0000 7f46c729d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c01e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.592+0000 7f46c729d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00c3210 con 0x7f46c01559e0 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c729d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00c1520 con 0x7f46c00aa180 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c729d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00c3390 con 0x7f46c01550c0 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c01e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c01e9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58718/0 (socket says 172.21.15.94:58718) 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 -- 172.21.15.94:0/3412350863 learned_addr learned my addr 172.21.15.94:0/3412350863 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:05.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c5039640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c00aa180 0x7f46c014a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c4838640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01e9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 -- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 msgr2=0x7f46c01e9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01e9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 -- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c00aa180 msgr2=0x7f46c014a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c00aa180 0x7f46c014a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 -- 172.21.15.94:0/3412350863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c01e6930 con 0x7f46c01559e0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c583a640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c01e9770 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f46b800eed0 tx=0x7f46b800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46b5ffb640 1 -- 172.21.15.94:0/3412350863 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8018020 con 0x7f46c01559e0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46b5ffb640 1 -- 172.21.15.94:0/3412350863 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b8014400 con 0x7f46c01559e0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46b5ffb640 1 -- 172.21.15.94:0/3412350863 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b80147f0 con 0x7f46c01559e0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.593+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c0152af0 con 0x7f46c01559e0 2024-03-06T07:28:05.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.594+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c01eac20 con 0x7f46c01559e0 2024-03-06T07:28:05.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.594+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c004f330 con 0x7f46c01559e0 2024-03-06T07:28:05.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.595+0000 7f46b5ffb640 1 -- 172.21.15.94:0/3412350863 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46b8005d10 con 0x7f46c01559e0 2024-03-06T07:28:05.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.595+0000 7f46b5ffb640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.596+0000 7f46b5ffb640 1 -- 172.21.15.94:0/3412350863 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f46b808ac00 con 0x7f46c01559e0 2024-03-06T07:28:05.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.596+0000 7f46c5039640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.596+0000 7f46c5039640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46a406b4d0 0x7f46a406d990 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f46c00c0990 tx=0x7f46bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.600+0000 7f46b5ffb640 1 -- 172.21.15.94:0/3412350863 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46b8091080 con 0x7f46c01559e0 2024-03-06T07:28:05.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.806+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46c00a5ef0 con 0x7f46c01559e0 2024-03-06T07:28:05.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.807+0000 7f46b5ffb640 1 -- 172.21.15.94:0/3412350863 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+4221 (secure 0 0 0) 0x7f46b8022030 con 0x7f46c01559e0 2024-03-06T07:28:05.809 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:05.809 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":204,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:04.264748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:05.809 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 204 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46a406b4d0 msgr2=0x7f46a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46a406b4d0 0x7f46a406d990 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f46c00c0990 tx=0x7f46bc009190 comp rx=0 tx=0).stop 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 msgr2=0x7f46c01e9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c01e9770 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f46b800eed0 tx=0x7f46b800b480 comp rx=0 tx=0).stop 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 shutdown_connections 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f46c01559e0 0x7f46c01e9770 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46c01550c0 0x7f46c01e9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 --2- 172.21.15.94:0/3412350863 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46c00aa180 0x7f46c014a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 >> 172.21.15.94:0/3412350863 conn(0x7f46c00c6710 msgr2=0x7f46c01530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 shutdown_connections 2024-03-06T07:28:05.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:05.809+0000 7f46c729d640 1 -- 172.21.15.94:0/3412350863 wait complete. 2024-03-06T07:28:06.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 204, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:04.264748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:rejoin', 'state_seq': 16, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:06.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:rejoin 2024-03-06T07:28:07.225 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.335+0000 7f9b30193640 1 Processor -- start 2024-03-06T07:28:07.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.336+0000 7f9b30193640 1 -- start start 2024-03-06T07:28:07.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.336+0000 7f9b30193640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b281529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.336+0000 7f9b30193640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9b280c1aa0 0x7f9b280c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.336+0000 7f9b30193640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9b280c5c60 0x7f9b28155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.336+0000 7f9b30193640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b280c2680 con 0x7f9b280c5c60 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.336+0000 7f9b30193640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b280c28d0 con 0x7f9b280c1aa0 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.336+0000 7f9b30193640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280c2bb0 con 0x7f9b2814f790 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9b280c5c60 0x7f9b28155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9b280c5c60 0x7f9b28155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58722/0 (socket says 172.21.15.94:58722) 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 -- 172.21.15.94:0/502729980 learned_addr learned my addr 172.21.15.94:0/502729980 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2d72e640 1 --2- 172.21.15.94:0/502729980 >> v2:172.21.15.145:3300/0 conn(0x7f9b280c1aa0 0x7f9b280c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2df2f640 1 --2- 172.21.15.94:0/502729980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b281529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 -- 172.21.15.94:0/502729980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 msgr2=0x7f9b281529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 --2- 172.21.15.94:0/502729980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b281529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 -- 172.21.15.94:0/502729980 >> v2:172.21.15.145:3300/0 conn(0x7f9b280c1aa0 msgr2=0x7f9b280c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 --2- 172.21.15.94:0/502729980 >> v2:172.21.15.145:3300/0 conn(0x7f9b280c1aa0 0x7f9b280c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2e730640 1 -- 172.21.15.94:0/502729980 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b280c2e90 con 0x7f9b280c5c60 2024-03-06T07:28:07.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.337+0000 7f9b2d72e640 1 --2- 172.21.15.94:0/502729980 >> v2:172.21.15.145:3300/0 conn(0x7f9b280c1aa0 0x7f9b280c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:07.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b2e730640 1 --2- 172.21.15.94:0/502729980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b28155a40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f9b280c76f0 tx=0x7f9b2402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453408792161ab8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b2cf2d640 1 -- 172.21.15.94:0/502729980 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2402ea80 con 0x7f9b280c5c60 2024-03-06T07:28:07.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b2cf2d640 1 -- 172.21.15.94:0/502729980 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2402ec20 con 0x7f9b280c5c60 2024-03-06T07:28:07.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b2cf2d640 1 -- 172.21.15.94:0/502729980 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b24009600 con 0x7f9b280c5c60 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 -- 172.21.15.94:0/502729980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 msgr2=0x7f9b28155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 --2- 172.21.15.94:0/502729980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b28155a40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f9b280c76f0 tx=0x7f9b2402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 -- 172.21.15.94:0/502729980 shutdown_connections 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 --2- 172.21.15.94:0/502729980 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b28155a40 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 --2- 172.21.15.94:0/502729980 >> v2:172.21.15.145:3300/0 conn(0x7f9b280c1aa0 0x7f9b280c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 --2- 172.21.15.94:0/502729980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b281529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 -- 172.21.15.94:0/502729980 >> 172.21.15.94:0/502729980 conn(0x7f9b280c6710 msgr2=0x7f9b280c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 -- 172.21.15.94:0/502729980 shutdown_connections 2024-03-06T07:28:07.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.338+0000 7f9b30193640 1 -- 172.21.15.94:0/502729980 wait complete. 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 Processor -- start 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 -- start start 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b280c1aa0 0x7f9b28150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b281ddd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b28151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2806cc00 con 0x7f9b280c5c60 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280c0d60 con 0x7f9b280c1aa0 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b30193640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2806cd80 con 0x7f9b2814f790 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b2d72e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b281ddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b2d72e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b281ddd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58732/0 (socket says 172.21.15.94:58732) 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b2df2f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b280c1aa0 0x7f9b28150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b2d72e640 1 -- 172.21.15.94:0/4036205019 learned_addr learned my addr 172.21.15.94:0/4036205019 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.340+0000 7f9b2e730640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b28151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b2d72e640 1 -- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 msgr2=0x7f9b28151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b2d72e640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b28151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b2d72e640 1 -- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b280c1aa0 msgr2=0x7f9b28150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b2d72e640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b280c1aa0 0x7f9b28150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b2d72e640 1 -- 172.21.15.94:0/4036205019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2814b1a0 con 0x7f9b280c5c60 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b2df2f640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b280c1aa0 0x7f9b28150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b2d72e640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b281ddd20 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800e9b0 tx=0x7f9b1800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b1e7fc640 1 -- 172.21.15.94:0/4036205019 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b18017070 con 0x7f9b280c5c60 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b1e7fc640 1 -- 172.21.15.94:0/4036205019 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1800ca00 con 0x7f9b280c5c60 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2814acc0 con 0x7f9b280c5c60 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b1e7fc640 1 -- 172.21.15.94:0/4036205019 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1800cdd0 con 0x7f9b280c5c60 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.341+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b281de260 con 0x7f9b280c5c60 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.342+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b281dbf10 con 0x7f9b280c5c60 2024-03-06T07:28:07.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.342+0000 7f9b1e7fc640 1 -- 172.21.15.94:0/4036205019 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b1801a070 con 0x7f9b280c5c60 2024-03-06T07:28:07.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.343+0000 7f9b1e7fc640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.343+0000 7f9b1e7fc640 1 -- 172.21.15.94:0/4036205019 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f9b1808a5c0 con 0x7f9b280c5c60 2024-03-06T07:28:07.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.343+0000 7f9b2df2f640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.343+0000 7f9b2df2f640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9b20009e90 tx=0x7f9b20009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.349 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.347+0000 7f9b1e7fc640 1 -- 172.21.15.94:0/4036205019 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b18058bb0 con 0x7f9b280c5c60 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.558+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b2804f330 con 0x7f9b280c5c60 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.558+0000 7f9b1e7fc640 1 -- 172.21.15.94:0/4036205019 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f9b1805cc50 con 0x7f9b280c5c60 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:07.561 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:07.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b0c06b500 msgr2=0x7f9b0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9b20009e90 tx=0x7f9b20009190 comp rx=0 tx=0).stop 2024-03-06T07:28:07.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 msgr2=0x7f9b281ddd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b281ddd20 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800e9b0 tx=0x7f9b1800ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 shutdown_connections 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9b0c06b500 0x7f9b0c06d9c0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b2814f790 0x7f9b28151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9b280c5c60 0x7f9b281ddd20 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 --2- 172.21.15.94:0/4036205019 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b280c1aa0 0x7f9b28150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 >> 172.21.15.94:0/4036205019 conn(0x7f9b280c6710 msgr2=0x7f9b280c4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 shutdown_connections 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:07.560+0000 7f9b30193640 1 -- 172.21.15.94:0/4036205019 wait complete. 2024-03-06T07:28:07.975 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:07.976 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:08.977 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:09.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.088+0000 7faf80ee0640 1 Processor -- start 2024-03-06T07:28:09.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.088+0000 7faf80ee0640 1 -- start start 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf80ee0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faf7c0c6000 0x7faf7c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf80ee0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c153a60 con 0x7faf7c150580 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf80ee0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c153d40 con 0x7faf7c0c1320 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf80ee0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c06c2c0 con 0x7faf7c0c6000 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf7ad9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faf7c0c6000 0x7faf7c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf7a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7faf7c150580 0x7faf7c06dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42818/0 (socket says 172.21.15.94:42818) 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf7a59c640 1 -- 172.21.15.94:0/4211357256 learned_addr learned my addr 172.21.15.94:0/4211357256 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf7ad9d640 1 -- 172.21.15.94:0/4211357256 >> v1:172.21.15.145:6789/0 conn(0x7faf7c0c1320 legacy=0x7faf7c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf7ad9d640 1 -- 172.21.15.94:0/4211357256 >> v1:172.21.15.94:6789/0 conn(0x7faf7c150580 legacy=0x7faf7c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.089+0000 7faf7ad9d640 1 -- 172.21.15.94:0/4211357256 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c06c5a0 con 0x7faf7c0c6000 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.090+0000 7faf7ad9d640 1 --2- 172.21.15.94:0/4211357256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c6000 0x7faf7c156dc0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7faf74009d90 tx=0x7faf7402bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6685e14ba4fe73bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.090+0000 7faf79d9b640 1 -- 172.21.15.94:0/4211357256 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74039070 con 0x7faf7c0c6000 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.090+0000 7faf79d9b640 1 -- 172.21.15.94:0/4211357256 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7402ca20 con 0x7faf7c0c6000 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.090+0000 7faf79d9b640 1 -- 172.21.15.94:0/4211357256 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7402cdf0 con 0x7faf7c0c6000 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.091+0000 7faf80ee0640 1 -- 172.21.15.94:0/4211357256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c6000 msgr2=0x7faf7c156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.091+0000 7faf80ee0640 1 --2- 172.21.15.94:0/4211357256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c6000 0x7faf7c156dc0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7faf74009d90 tx=0x7faf7402bcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.091+0000 7faf80ee0640 1 -- 172.21.15.94:0/4211357256 shutdown_connections 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.091+0000 7faf80ee0640 1 --2- 172.21.15.94:0/4211357256 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c6000 0x7faf7c156dc0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.091+0000 7faf80ee0640 1 -- 172.21.15.94:0/4211357256 >> 172.21.15.94:0/4211357256 conn(0x7faf7c0c6710 msgr2=0x7faf7c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.091+0000 7faf80ee0640 1 -- 172.21.15.94:0/4211357256 shutdown_connections 2024-03-06T07:28:09.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.091+0000 7faf80ee0640 1 -- 172.21.15.94:0/4211357256 wait complete. 2024-03-06T07:28:09.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.092+0000 7faf80ee0640 1 Processor -- start 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.092+0000 7faf80ee0640 1 -- start start 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf80ee0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c1320 0x7faf7c155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf80ee0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf7c0c6000 0x7faf7c151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf80ee0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf7c150580 0x7faf7c151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf80ee0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c06e420 con 0x7faf7c0c1320 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf80ee0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c150960 con 0x7faf7c150580 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf80ee0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c06e5a0 con 0x7faf7c0c6000 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c1320 0x7faf7c155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c1320 0x7faf7c155730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42778/0 (socket says 172.21.15.94:42778) 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 -- 172.21.15.94:0/67549245 learned_addr learned my addr 172.21.15.94:0/67549245 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf73fff640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf7c0c6000 0x7faf7c151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7ad9d640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf7c150580 0x7faf7c151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 -- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf7c0c6000 msgr2=0x7faf7c151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf7c0c6000 0x7faf7c151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 -- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf7c150580 msgr2=0x7faf7c151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf7c150580 0x7faf7c151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 -- 172.21.15.94:0/67549245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c1ec660 con 0x7faf7c0c1320 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf7a59c640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c1320 0x7faf7c155730 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00c5f0 tx=0x7faf6c00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf72ffd640 1 -- 172.21.15.94:0/67549245 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c018070 con 0x7faf7c0c1320 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf72ffd640 1 -- 172.21.15.94:0/67549245 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c0074e0 con 0x7faf7c0c1320 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf72ffd640 1 -- 172.21.15.94:0/67549245 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c004a00 con 0x7faf7c0c1320 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.093+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c1e91d0 con 0x7faf7c0c1320 2024-03-06T07:28:09.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.094+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c155c70 con 0x7faf7c0c1320 2024-03-06T07:28:09.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.095+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c067ad0 con 0x7faf7c0c1320 2024-03-06T07:28:09.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.095+0000 7faf72ffd640 1 -- 172.21.15.94:0/67549245 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf6c01b070 con 0x7faf7c0c1320 2024-03-06T07:28:09.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.095+0000 7faf72ffd640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf5c06b500 0x7faf5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.095+0000 7faf72ffd640 1 -- 172.21.15.94:0/67549245 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7faf6c089760 con 0x7faf7c0c1320 2024-03-06T07:28:09.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.095+0000 7faf73fff640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf5c06b500 0x7faf5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.096+0000 7faf73fff640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf5c06b500 0x7faf5c06d9c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7faf7c1ea0f0 tx=0x7faf64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.100+0000 7faf72ffd640 1 -- 172.21.15.94:0/67549245 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf7c067ad0 con 0x7faf7c0c1320 2024-03-06T07:28:09.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.305+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7c14e3e0 con 0x7faf7c0c1320 2024-03-06T07:28:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.306+0000 7faf72ffd640 1 -- 172.21.15.94:0/67549245 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7faf7c14e3e0 con 0x7faf7c0c1320 2024-03-06T07:28:09.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:09.308 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:09.308 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf5c06b500 msgr2=0x7faf5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf5c06b500 0x7faf5c06d9c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7faf7c1ea0f0 tx=0x7faf64009040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c1320 msgr2=0x7faf7c155730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c1320 0x7faf7c155730 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00c5f0 tx=0x7faf6c00cab0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 shutdown_connections 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7faf5c06b500 0x7faf5c06d9c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf7c150580 0x7faf7c151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf7c0c6000 0x7faf7c151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 --2- 172.21.15.94:0/67549245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faf7c0c1320 0x7faf7c155730 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 >> 172.21.15.94:0/67549245 conn(0x7faf7c0c6710 msgr2=0x7faf7c0c4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 shutdown_connections 2024-03-06T07:28:09.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:09.308+0000 7faf80ee0640 1 -- 172.21.15.94:0/67549245 wait complete. 2024-03-06T07:28:09.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:09.723 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:10.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:10.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.834+0000 7f1863f8c640 1 Processor -- start 2024-03-06T07:28:10.836 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.834+0000 7f1863f8c640 1 -- start start 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1863f8c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f185c0c5330 0x7f185c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1863f8c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c159630 con 0x7f185c0c8a00 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1863f8c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0c7760 con 0x7f185c152920 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1863f8c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c1587b0 con 0x7f185c0c5330 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1862529640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f185c0c5330 0x7f185c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1861527640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f185c0c8a00 0x7f185c158f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42826/0 (socket says 172.21.15.94:42826) 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1861527640 1 -- 172.21.15.94:0/2729482981 learned_addr learned my addr 172.21.15.94:0/2729482981 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:10.837 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1862529640 1 -- 172.21.15.94:0/2729482981 >> v1:172.21.15.145:6789/0 conn(0x7f185c152920 legacy=0x7f185c155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1862529640 1 -- 172.21.15.94:0/2729482981 >> v1:172.21.15.94:6789/0 conn(0x7f185c0c8a00 legacy=0x7f185c158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1862529640 1 -- 172.21.15.94:0/2729482981 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0c7a40 con 0x7f185c0c5330 2024-03-06T07:28:10.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.835+0000 7f1860d26640 1 -- 172.21.15.94:0/2729482981 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976860246 0 0) 0x7f185c159630 con 0x7f185c0c8a00 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.836+0000 7f1862529640 1 --2- 172.21.15.94:0/2729482981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c0c5330 0x7f185c0c57a0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f1854009d90 tx=0x7f185402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4469e49ff6d099f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.836+0000 7f1860d26640 1 -- 172.21.15.94:0/2729482981 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854039070 con 0x7f185c0c5330 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.836+0000 7f1860d26640 1 -- 172.21.15.94:0/2729482981 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185402cc60 con 0x7f185c0c5330 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.836+0000 7f1860d26640 1 -- 172.21.15.94:0/2729482981 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854034660 con 0x7f185c0c5330 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.837+0000 7f1863f8c640 1 -- 172.21.15.94:0/2729482981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c0c5330 msgr2=0x7f185c0c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.837+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2729482981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c0c5330 0x7f185c0c57a0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f1854009d90 tx=0x7f185402beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.837+0000 7f1863f8c640 1 -- 172.21.15.94:0/2729482981 shutdown_connections 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.837+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2729482981 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c0c5330 0x7f185c0c57a0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.837+0000 7f1863f8c640 1 -- 172.21.15.94:0/2729482981 >> 172.21.15.94:0/2729482981 conn(0x7f185c0ac560 msgr2=0x7f185c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.837+0000 7f1863f8c640 1 -- 172.21.15.94:0/2729482981 shutdown_connections 2024-03-06T07:28:10.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.837+0000 7f1863f8c640 1 -- 172.21.15.94:0/2729482981 wait complete. 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.838+0000 7f1863f8c640 1 Processor -- start 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.838+0000 7f1863f8c640 1 -- start start 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.838+0000 7f1863f8c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f185c0c5330 0x7f185c14d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1863f8c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f185c0c8a00 0x7f185c0cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1863f8c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c152920 0x7f185c1ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1863f8c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c06d030 con 0x7f185c152920 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1863f8c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c158a00 con 0x7f185c0c5330 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1863f8c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0c7390 con 0x7f185c0c8a00 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c152920 0x7f185c1ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c152920 0x7f185c1ee420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42786/0 (socket says 172.21.15.94:42786) 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 -- 172.21.15.94:0/2613704264 learned_addr learned my addr 172.21.15.94:0/2613704264 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1861527640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f185c0c8a00 0x7f185c0cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1861d28640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f185c0c5330 0x7f185c14d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 -- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f185c0c8a00 msgr2=0x7f185c0cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f185c0c8a00 0x7f185c0cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 -- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f185c0c5330 msgr2=0x7f185c14d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f185c0c5330 0x7f185c14d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 -- 172.21.15.94:0/2613704264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c14e5d0 con 0x7f185c152920 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1862529640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c152920 0x7f185c1ee420 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f1854037040 tx=0x7f1854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f18527fc640 1 -- 172.21.15.94:0/2613704264 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854044020 con 0x7f185c152920 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f18527fc640 1 -- 172.21.15.94:0/2613704264 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185403a040 con 0x7f185c152920 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f18527fc640 1 -- 172.21.15.94:0/2613704264 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1854039050 con 0x7f185c152920 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c14e8b0 con 0x7f185c152920 2024-03-06T07:28:10.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.839+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c14eb90 con 0x7f185c152920 2024-03-06T07:28:10.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.840+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185c04f330 con 0x7f185c152920 2024-03-06T07:28:10.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.841+0000 7f18527fc640 1 -- 172.21.15.94:0/2613704264 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f185402cae0 con 0x7f185c152920 2024-03-06T07:28:10.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.841+0000 7f18527fc640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184006b4d0 0x7f184006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.841+0000 7f18527fc640 1 -- 172.21.15.94:0/2613704264 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f18540aca60 con 0x7f185c152920 2024-03-06T07:28:10.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.841+0000 7f1861d28640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184006b4d0 0x7f184006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.842+0000 7f1861d28640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184006b4d0 0x7f184006d990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f1858000f60 tx=0x7f1858009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:10.846+0000 7f18527fc640 1 -- 172.21.15.94:0/2613704264 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f185404e060 con 0x7f185c152920 2024-03-06T07:28:11.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.054+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f185c06cda0 con 0x7f185c152920 2024-03-06T07:28:11.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.055+0000 7f18527fc640 1 -- 172.21.15.94:0/2613704264 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f185407b060 con 0x7f185c152920 2024-03-06T07:28:11.057 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:11.057 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:11.057 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:11.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184006b4d0 msgr2=0x7f184006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184006b4d0 0x7f184006d990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f1858000f60 tx=0x7f1858009040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c152920 msgr2=0x7f185c1ee420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c152920 0x7f185c1ee420 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f1854037040 tx=0x7f1854004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 shutdown_connections 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f184006b4d0 0x7f184006d990 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f185c152920 0x7f185c1ee420 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f185c0c8a00 0x7f185c0cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 --2- 172.21.15.94:0/2613704264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f185c0c5330 0x7f185c14d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 >> 172.21.15.94:0/2613704264 conn(0x7f185c0ac560 msgr2=0x7f185c151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 shutdown_connections 2024-03-06T07:28:11.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:11.057+0000 7f1863f8c640 1 -- 172.21.15.94:0/2613704264 wait complete. 2024-03-06T07:28:11.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:11.473 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:12.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:12.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.584+0000 7fdf0cfd7640 1 Processor -- start 2024-03-06T07:28:12.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.584+0000 7fdf0cfd7640 1 -- start start 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf0cfd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0814f360 0x7fdf08155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf0cfd7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdf08158b60 0x7fdf08158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf0cfd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf080c73d0 con 0x7fdf080c88c0 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf0cfd7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf080c8dd0 con 0x7fdf08158b60 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf0cfd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080be2e0 con 0x7fdf0814f360 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0814f360 0x7fdf08155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf06d9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdf08158b60 0x7fdf08158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf06d9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdf08158b60 0x7fdf08158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:50824/0 (socket says 172.21.15.94:50824) 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdefdd9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fdf080c88c0 0x7fdf080c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:39070/0 (socket says 172.21.15.94:39070) 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0814f360 0x7fdf08155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:34612/0 (socket says 172.21.15.94:34612) 2024-03-06T07:28:12.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.585+0000 7fdf06d9d640 1 -- 172.21.15.94:0/4012221226 learned_addr learned my addr 172.21.15.94:0/4012221226 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:12.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.586+0000 7fdf06d9d640 1 -- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0814f360 msgr2=0x7fdf08155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.586+0000 7fdf06d9d640 1 --2- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0814f360 0x7fdf08155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.586+0000 7fdf06d9d640 1 -- 172.21.15.94:0/4012221226 >> v1:172.21.15.145:6789/0 conn(0x7fdf080c88c0 legacy=0x7fdf080c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.586+0000 7fdf06d9d640 1 -- 172.21.15.94:0/4012221226 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf080c7670 con 0x7fdf08158b60 2024-03-06T07:28:12.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.586+0000 7fdf05d9b640 1 -- 172.21.15.94:0/4012221226 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274845765 0 0) 0x7fdf080c73d0 con 0x7fdf080c88c0 2024-03-06T07:28:12.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.586+0000 7fdf0659c640 1 --2- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0814f360 0x7fdf08155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:12.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf06d9d640 1 --2- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf08158b60 0x7fdf08158ff0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009e50 tx=0x7fdf0002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e65f704a6e10537a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf05d9b640 1 -- 172.21.15.94:0/4012221226 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0002ca70 con 0x7fdf08158b60 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf05d9b640 1 -- 172.21.15.94:0/4012221226 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0002cc10 con 0x7fdf08158b60 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf05d9b640 1 -- 172.21.15.94:0/4012221226 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf000343f0 con 0x7fdf08158b60 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf08158b60 msgr2=0x7fdf08158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf08158b60 0x7fdf08158ff0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009e50 tx=0x7fdf0002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/4012221226 shutdown_connections 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf08158b60 0x7fdf08158ff0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/4012221226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf0814f360 0x7fdf08155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/4012221226 >> 172.21.15.94:0/4012221226 conn(0x7fdf080ac570 msgr2=0x7fdf080ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/4012221226 shutdown_connections 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.587+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/4012221226 wait complete. 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.589+0000 7fdf0cfd7640 1 Processor -- start 2024-03-06T07:28:12.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.589+0000 7fdf0cfd7640 1 -- start start 2024-03-06T07:28:12.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0cfd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf080c88c0 0x7fdf0814e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0cfd7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0814f360 0x7fdf081576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0cfd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf08158b60 0x7fdf08157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0cfd7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080c5a50 con 0x7fdf08158b60 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0cfd7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080c7c30 con 0x7fdf0814f360 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0cfd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf080c5bd0 con 0x7fdf080c88c0 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf08158b60 0x7fdf08157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf08158b60 0x7fdf08157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42796/0 (socket says 172.21.15.94:42796) 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 -- 172.21.15.94:0/1933259211 learned_addr learned my addr 172.21.15.94:0/1933259211 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdefdd9b640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0814f360 0x7fdf081576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0659c640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf080c88c0 0x7fdf0814e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 -- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf080c88c0 msgr2=0x7fdf0814e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf080c88c0 0x7fdf0814e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 -- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0814f360 msgr2=0x7fdf081576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0814f360 0x7fdf081576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 -- 172.21.15.94:0/1933259211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf080c1300 con 0x7fdf08158b60 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf06d9d640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf08158b60 0x7fdf08157bf0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009e50 tx=0x7fdf00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdeff7fe640 1 -- 172.21.15.94:0/1933259211 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00044030 con 0x7fdf08158b60 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdeff7fe640 1 -- 172.21.15.94:0/1933259211 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf00040470 con 0x7fdf08158b60 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdeff7fe640 1 -- 172.21.15.94:0/1933259211 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00009510 con 0x7fdf08158b60 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.590+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf081eb090 con 0x7fdf08158b60 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.591+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf081eb650 con 0x7fdf08158b60 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.591+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0804f330 con 0x7fdf08158b60 2024-03-06T07:28:12.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.592+0000 7fdeff7fe640 1 -- 172.21.15.94:0/1933259211 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf00047070 con 0x7fdf08158b60 2024-03-06T07:28:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.592+0000 7fdeff7fe640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdee806b500 0x7fdee806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.592+0000 7fdeff7fe640 1 -- 172.21.15.94:0/1933259211 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fdf000ac920 con 0x7fdf08158b60 2024-03-06T07:28:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.592+0000 7fdf0659c640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdee806b500 0x7fdee806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.593+0000 7fdf0659c640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdee806b500 0x7fdee806d9c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009560 tx=0x7fdef8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.597+0000 7fdeff7fe640 1 -- 172.21.15.94:0/1933259211 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf0004e070 con 0x7fdf08158b60 2024-03-06T07:28:12.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.803+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf0806cda0 con 0x7fdf08158b60 2024-03-06T07:28:12.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.804+0000 7fdeff7fe640 1 -- 172.21.15.94:0/1933259211 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7fdf080c1300 con 0x7fdf08158b60 2024-03-06T07:28:12.806 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:12.806 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:12.806 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:12.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdee806b500 msgr2=0x7fdee806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdee806b500 0x7fdee806d9c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009560 tx=0x7fdef8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:12.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf08158b60 msgr2=0x7fdf08157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf08158b60 0x7fdf08157bf0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009e50 tx=0x7fdf00004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:12.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 shutdown_connections 2024-03-06T07:28:12.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdee806b500 0x7fdee806d9c0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf08158b60 0x7fdf08157bf0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf0814f360 0x7fdf081576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 --2- 172.21.15.94:0/1933259211 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf080c88c0 0x7fdf0814e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 >> 172.21.15.94:0/1933259211 conn(0x7fdf080ac570 msgr2=0x7fdf0806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 shutdown_connections 2024-03-06T07:28:12.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:12.806+0000 7fdf0cfd7640 1 -- 172.21.15.94:0/1933259211 wait complete. 2024-03-06T07:28:13.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:13.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:14.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:14.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.332+0000 7f5651488640 1 Processor -- start 2024-03-06T07:28:14.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.332+0000 7f5651488640 1 -- start start 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f5651488640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f564c0c7180 0x7f564c159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f5651488640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f564c0c88b0 0x7f564c0c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f5651488640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564c0c7c60 con 0x7f564c1527b0 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f5651488640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c159640 con 0x7f564c0c7180 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f5651488640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c0c8060 con 0x7f564c0c88b0 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f564a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f564c0c7180 0x7f564c159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f564a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f564c0c7180 0x7f564c159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42798/0 (socket says 172.21.15.94:42798) 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f564a7fc640 1 -- 172.21.15.94:0/2067224899 learned_addr learned my addr 172.21.15.94:0/2067224899 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f564affd640 1 --1- 172.21.15.94:0/2067224899 >> v1:172.21.15.145:6789/0 conn(0x7f564c1527b0 0x7f564c155b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:39084/0 (socket says 172.21.15.94:39084) 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.333+0000 7f564b7fe640 1 --2- 172.21.15.94:0/2067224899 >> v2:172.21.15.145:3300/0 conn(0x7f564c0c88b0 0x7f564c0c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.334+0000 7f564a7fc640 1 -- 172.21.15.94:0/2067224899 >> v2:172.21.15.145:3300/0 conn(0x7f564c0c88b0 msgr2=0x7f564c0c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.334+0000 7f564a7fc640 1 --2- 172.21.15.94:0/2067224899 >> v2:172.21.15.145:3300/0 conn(0x7f564c0c88b0 0x7f564c0c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.334+0000 7f564a7fc640 1 -- 172.21.15.94:0/2067224899 >> v1:172.21.15.145:6789/0 conn(0x7f564c1527b0 legacy=0x7f564c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.334+0000 7f564a7fc640 1 -- 172.21.15.94:0/2067224899 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0c76a0 con 0x7f564c0c7180 2024-03-06T07:28:14.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.334+0000 7f564a7fc640 1 --2- 172.21.15.94:0/2067224899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c7180 0x7f564c159100 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f564c0ad600 tx=0x7f564002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f9e67ddb319037 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5649ffb640 1 -- 172.21.15.94:0/2067224899 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640035020 con 0x7f564c0c7180 2024-03-06T07:28:14.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5649ffb640 1 -- 172.21.15.94:0/2067224899 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5640031870 con 0x7f564c0c7180 2024-03-06T07:28:14.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5649ffb640 1 -- 172.21.15.94:0/2067224899 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640031c40 con 0x7f564c0c7180 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 -- 172.21.15.94:0/2067224899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c7180 msgr2=0x7f564c159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 --2- 172.21.15.94:0/2067224899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c7180 0x7f564c159100 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f564c0ad600 tx=0x7f564002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 -- 172.21.15.94:0/2067224899 shutdown_connections 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 --2- 172.21.15.94:0/2067224899 >> v2:172.21.15.145:3300/0 conn(0x7f564c0c88b0 0x7f564c0c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 --2- 172.21.15.94:0/2067224899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c7180 0x7f564c159100 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 -- 172.21.15.94:0/2067224899 >> 172.21.15.94:0/2067224899 conn(0x7f564c0ac560 msgr2=0x7f564c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 -- 172.21.15.94:0/2067224899 shutdown_connections 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.335+0000 7f5651488640 1 -- 172.21.15.94:0/2067224899 wait complete. 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.336+0000 7f5651488640 1 Processor -- start 2024-03-06T07:28:14.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.336+0000 7f5651488640 1 -- start start 2024-03-06T07:28:14.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f5651488640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f564c0c7180 0x7f564c1ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f5651488640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c88b0 0x7f564c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f5651488640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f564c1527b0 0x7f564c1ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f5651488640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c155600 con 0x7f564c0c88b0 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f5651488640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c155320 con 0x7f564c1527b0 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f5651488640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c152b90 con 0x7f564c0c7180 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f564a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c88b0 0x7f564c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f564a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c88b0 0x7f564c0ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42800/0 (socket says 172.21.15.94:42800) 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f564a7fc640 1 -- 172.21.15.94:0/2796961739 learned_addr learned my addr 172.21.15.94:0/2796961739 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f564b7fe640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f564c1527b0 0x7f564c1ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.337+0000 7f564affd640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f564c0c7180 0x7f564c1ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f564a7fc640 1 -- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f564c0c7180 msgr2=0x7f564c1ebc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f564a7fc640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f564c0c7180 0x7f564c1ebc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f564a7fc640 1 -- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f564c1527b0 msgr2=0x7f564c1ec5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f564a7fc640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f564c1527b0 0x7f564c1ec5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f564a7fc640 1 -- 172.21.15.94:0/2796961739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c1ef1f0 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f564a7fc640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c88b0 0x7f564c0ca9d0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f564002b950 tx=0x7f56400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f56337fe640 1 -- 172.21.15.94:0/2796961739 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640040070 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f56337fe640 1 -- 172.21.15.94:0/2796961739 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5640030d00 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c1579e0 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f56337fe640 1 -- 172.21.15.94:0/2796961739 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640035040 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.338+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c1eb490 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.339+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c04f330 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.339+0000 7f56337fe640 1 -- 172.21.15.94:0/2796961739 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f564004b060 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.340+0000 7f56337fe640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f563806b4d0 0x7f563806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.340+0000 7f56337fe640 1 -- 172.21.15.94:0/2796961739 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f56400abf50 con 0x7f564c0c88b0 2024-03-06T07:28:14.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.340+0000 7f564affd640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f563806b4d0 0x7f563806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.340+0000 7f564affd640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f563806b4d0 0x7f563806d990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f564c0c4a80 tx=0x7f563c001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.344+0000 7f56337fe640 1 -- 172.21.15.94:0/2796961739 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f564007a5d0 con 0x7f564c0c88b0 2024-03-06T07:28:14.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.555+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f564c0ace60 con 0x7f564c0c88b0 2024-03-06T07:28:14.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.555+0000 7f56337fe640 1 -- 172.21.15.94:0/2796961739 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f564007e670 con 0x7f564c0c88b0 2024-03-06T07:28:14.557 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:14.557 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:14.558 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:14.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f563806b4d0 msgr2=0x7f563806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f563806b4d0 0x7f563806d990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f564c0c4a80 tx=0x7f563c001630 comp rx=0 tx=0).stop 2024-03-06T07:28:14.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c88b0 msgr2=0x7f564c0ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c88b0 0x7f564c0ca9d0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f564002b950 tx=0x7f56400046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 shutdown_connections 2024-03-06T07:28:14.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f563806b4d0 0x7f563806d990 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f564c1527b0 0x7f564c1ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f564c0c88b0 0x7f564c0ca9d0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 --2- 172.21.15.94:0/2796961739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f564c0c7180 0x7f564c1ebc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 >> 172.21.15.94:0/2796961739 conn(0x7f564c0ac560 msgr2=0x7f564c151180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 shutdown_connections 2024-03-06T07:28:14.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:14.557+0000 7f5651488640 1 -- 172.21.15.94:0/2796961739 wait complete. 2024-03-06T07:28:14.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:14.976 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:15.977 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.087+0000 7f1fc2815640 1 Processor -- start 2024-03-06T07:28:16.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.087+0000 7f1fc2815640 1 -- start start 2024-03-06T07:28:16.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fc2815640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fbc0c1aa0 0x7f1fbc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fc2815640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc155a40 0x7f1fbc155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fc2815640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc1563d0 con 0x7f1fbc14f790 2024-03-06T07:28:16.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fc2815640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc0c2680 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fc2815640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0c28d0 con 0x7f1fbc155a40 2024-03-06T07:28:16.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbb7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1fbc0c1aa0 0x7f1fbc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1fbc14f790 0x7f1fbc0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42828/0 (socket says 172.21.15.94:42828) 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbbfff640 1 -- 172.21.15.94:0/2980963537 learned_addr learned my addr 172.21.15.94:0/2980963537 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fc0db2640 1 --2- 172.21.15.94:0/2980963537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc155a40 0x7f1fbc155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbb7fe640 1 -- 172.21.15.94:0/2980963537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc155a40 msgr2=0x7f1fbc155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbb7fe640 1 --2- 172.21.15.94:0/2980963537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc155a40 0x7f1fbc155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbb7fe640 1 -- 172.21.15.94:0/2980963537 >> v1:172.21.15.94:6789/0 conn(0x7f1fbc14f790 legacy=0x7f1fbc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbb7fe640 1 -- 172.21.15.94:0/2980963537 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc0c2bb0 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fbaffd640 1 -- 172.21.15.94:0/2980963537 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531593993 0 0) 0x7f1fbc1563d0 con 0x7f1fbc14f790 2024-03-06T07:28:16.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.088+0000 7f1fc0db2640 1 --2- 172.21.15.94:0/2980963537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc155a40 0x7f1fbc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:16.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.089+0000 7f1fbb7fe640 1 --2- 172.21.15.94:0/2980963537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc1529d0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00097a0 tx=0x7f1fb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6c991093b851c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.089+0000 7f1fbaffd640 1 -- 172.21.15.94:0/2980963537 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0035020 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.089+0000 7f1fbaffd640 1 -- 172.21.15.94:0/2980963537 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb0031830 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.089+0000 7f1fbaffd640 1 -- 172.21.15.94:0/2980963537 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0031c00 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 -- 172.21.15.94:0/2980963537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 msgr2=0x7f1fbc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 --2- 172.21.15.94:0/2980963537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc1529d0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00097a0 tx=0x7f1fb002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 -- 172.21.15.94:0/2980963537 shutdown_connections 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 --2- 172.21.15.94:0/2980963537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc155a40 0x7f1fbc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 --2- 172.21.15.94:0/2980963537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc1529d0 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 -- 172.21.15.94:0/2980963537 >> 172.21.15.94:0/2980963537 conn(0x7f1fbc0c6710 msgr2=0x7f1fbc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 -- 172.21.15.94:0/2980963537 shutdown_connections 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.090+0000 7f1fc2815640 1 -- 172.21.15.94:0/2980963537 wait complete. 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.091+0000 7f1fc2815640 1 Processor -- start 2024-03-06T07:28:16.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.091+0000 7f1fc2815640 1 -- start start 2024-03-06T07:28:16.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc14f790 0x7f1fbc0a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fbc155a40 0x7f1fbc1466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0c0d60 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc155520 con 0x7f1fbc155a40 2024-03-06T07:28:16.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0c5e30 con 0x7f1fbc14f790 2024-03-06T07:28:16.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42818/0 (socket says 172.21.15.94:42818) 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 -- 172.21.15.94:0/3205014167 learned_addr learned my addr 172.21.15.94:0/3205014167 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbb7fe640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc14f790 0x7f1fbc0a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc0db2640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fbc155a40 0x7f1fbc1466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 -- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc14f790 msgr2=0x7f1fbc0a7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc14f790 0x7f1fbc0a7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 -- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fbc155a40 msgr2=0x7f1fbc1466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fbc155a40 0x7f1fbc1466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 -- 172.21.15.94:0/3205014167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc153ed0 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fbbfff640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc154c10 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00c4d0 tx=0x7f1fac00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fb8ff9640 1 -- 172.21.15.94:0/3205014167 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac018070 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fb8ff9640 1 -- 172.21.15.94:0/3205014167 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac007500 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc1541b0 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fb8ff9640 1 -- 172.21.15.94:0/3205014167 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac004a20 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.092+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc0a81d0 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.093+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc069460 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.094+0000 7f1fb8ff9640 1 -- 172.21.15.94:0/3205014167 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fac01b070 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.094+0000 7f1fb8ff9640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fa806b4d0 0x7f1fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.094+0000 7f1fb8ff9640 1 -- 172.21.15.94:0/3205014167 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f1fac089660 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.094+0000 7f1fbb7fe640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fa806b4d0 0x7f1fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.095+0000 7f1fbb7fe640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fa806b4d0 0x7f1fa806d990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0036000 tx=0x7f1fb0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.099+0000 7f1fb8ff9640 1 -- 172.21.15.94:0/3205014167 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fbc069460 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.180 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:16.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.311+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fbc1e9e80 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.311+0000 7f1fb8ff9640 1 -- 172.21.15.94:0/3205014167 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f1fac057ce0 con 0x7f1fbc0c1aa0 2024-03-06T07:28:16.313 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:16.313 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:16.314 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:16.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fa806b4d0 msgr2=0x7f1fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fa806b4d0 0x7f1fa806d990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0036000 tx=0x7f1fb0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 msgr2=0x7f1fbc154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc154c10 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00c4d0 tx=0x7f1fac00c990 comp rx=0 tx=0).stop 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 shutdown_connections 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fa806b4d0 0x7f1fa806d990 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fbc155a40 0x7f1fbc1466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fbc14f790 0x7f1fbc0a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 --2- 172.21.15.94:0/3205014167 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fbc0c1aa0 0x7f1fbc154c10 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 >> 172.21.15.94:0/3205014167 conn(0x7f1fbc0c6710 msgr2=0x7f1fbc0c31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 shutdown_connections 2024-03-06T07:28:16.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:16.313+0000 7f1fc2815640 1 -- 172.21.15.94:0/3205014167 wait complete. 2024-03-06T07:28:16.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:16.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:17.730 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:17.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.840+0000 7f1db57ac640 1 Processor -- start 2024-03-06T07:28:17.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.840+0000 7f1db57ac640 1 -- start start 2024-03-06T07:28:17.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1db57ac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1db0155c80 0x7f1db006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1db57ac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db00c1ae0 con 0x7f1db00720a0 2024-03-06T07:28:17.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1db57ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db006e450 con 0x7f1db0152520 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1db57ac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db00c1dc0 con 0x7f1db0155c80 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1dae7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1db0155c80 0x7f1db006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1daf7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1db00720a0 0x7f1db006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42832/0 (socket says 172.21.15.94:42832) 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1daf7fe640 1 -- 172.21.15.94:0/2206049312 learned_addr learned my addr 172.21.15.94:0/2206049312 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1dae7fc640 1 -- 172.21.15.94:0/2206049312 >> v1:172.21.15.145:6789/0 conn(0x7f1db0152520 legacy=0x7f1db00c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1dae7fc640 1 -- 172.21.15.94:0/2206049312 >> v1:172.21.15.94:6789/0 conn(0x7f1db00720a0 legacy=0x7f1db006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1dadffb640 1 -- 172.21.15.94:0/2206049312 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293536114 0 0) 0x7f1db006e450 con 0x7f1db0152520 2024-03-06T07:28:17.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.841+0000 7f1dae7fc640 1 -- 172.21.15.94:0/2206049312 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db006e730 con 0x7f1db0155c80 2024-03-06T07:28:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.842+0000 7f1dae7fc640 1 --2- 172.21.15.94:0/2206049312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0155c80 0x7f1db006ef30 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f1da40097a0 tx=0x7f1da402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61f23329406d2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.842+0000 7f1dadffb640 1 -- 172.21.15.94:0/2206049312 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4037070 con 0x7f1db0155c80 2024-03-06T07:28:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.842+0000 7f1dadffb640 1 -- 172.21.15.94:0/2206049312 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4030710 con 0x7f1db0155c80 2024-03-06T07:28:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.842+0000 7f1dadffb640 1 -- 172.21.15.94:0/2206049312 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4030ae0 con 0x7f1db0155c80 2024-03-06T07:28:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.843+0000 7f1db57ac640 1 -- 172.21.15.94:0/2206049312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0155c80 msgr2=0x7f1db006ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.843+0000 7f1db57ac640 1 --2- 172.21.15.94:0/2206049312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0155c80 0x7f1db006ef30 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f1da40097a0 tx=0x7f1da402b790 comp rx=0 tx=0).stop 2024-03-06T07:28:17.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.843+0000 7f1db57ac640 1 -- 172.21.15.94:0/2206049312 shutdown_connections 2024-03-06T07:28:17.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.843+0000 7f1db57ac640 1 --2- 172.21.15.94:0/2206049312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0155c80 0x7f1db006ef30 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.843+0000 7f1db57ac640 1 -- 172.21.15.94:0/2206049312 >> 172.21.15.94:0/2206049312 conn(0x7f1db00c76b0 msgr2=0x7f1db00c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.843+0000 7f1db57ac640 1 -- 172.21.15.94:0/2206049312 shutdown_connections 2024-03-06T07:28:17.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.843+0000 7f1db57ac640 1 -- 172.21.15.94:0/2206049312 wait complete. 2024-03-06T07:28:17.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.844+0000 7f1db57ac640 1 Processor -- start 2024-03-06T07:28:17.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.844+0000 7f1db57ac640 1 -- start start 2024-03-06T07:28:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1db57ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00720a0 0x7f1db00ac540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1db57ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0152520 0x7f1db0157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1db57ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db0155c80 0x7f1db01578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1db57ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db006b850 con 0x7f1db0152520 2024-03-06T07:28:17.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1db57ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db006b6d0 con 0x7f1db0155c80 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1db57ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0152900 con 0x7f1db00720a0 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0152520 0x7f1db0157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0152520 0x7f1db0157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:42842/0 (socket says 172.21.15.94:42842) 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 -- 172.21.15.94:0/3518135384 learned_addr learned my addr 172.21.15.94:0/3518135384 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1daf7fe640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db0155c80 0x7f1db01578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1daeffd640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00720a0 0x7f1db00ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 -- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00720a0 msgr2=0x7f1db00ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00720a0 0x7f1db00ac540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 -- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db0155c80 msgr2=0x7f1db01578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db0155c80 0x7f1db01578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1dae7fc640 1 -- 172.21.15.94:0/3518135384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db01eb500 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.845+0000 7f1daf7fe640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db0155c80 0x7f1db01578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.846+0000 7f1dae7fc640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0152520 0x7f1db0157380 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f1da4035040 tx=0x7f1da40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.846+0000 7f1d977fe640 1 -- 172.21.15.94:0/3518135384 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4037040 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.846+0000 7f1d977fe640 1 -- 172.21.15.94:0/3518135384 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da40329c0 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.846+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db01eb7e0 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.846+0000 7f1d977fe640 1 -- 172.21.15.94:0/3518135384 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4032d90 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.846+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db00c6540 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.847+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db004f330 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.847+0000 7f1d977fe640 1 -- 172.21.15.94:0/3518135384 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1da4004030 con 0x7f1db0152520 2024-03-06T07:28:17.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.848+0000 7f1d977fe640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.848+0000 7f1d977fe640 1 -- 172.21.15.94:0/3518135384 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f1da40ac0b0 con 0x7f1db0152520 2024-03-06T07:28:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.848+0000 7f1daeffd640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.853 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.848+0000 7f1daeffd640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1da0000f60 tx=0x7f1da0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:17.852+0000 7f1d977fe640 1 -- 172.21.15.94:0/3518135384 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1da4042020 con 0x7f1db0152520 2024-03-06T07:28:18.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.059+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1db00c0b70 con 0x7f1db0152520 2024-03-06T07:28:18.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.059+0000 7f1d977fe640 1 -- 172.21.15.94:0/3518135384 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f1da407a850 con 0x7f1db0152520 2024-03-06T07:28:18.061 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:18.061 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:18.062 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 msgr2=0x7f1d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1da0000f60 tx=0x7f1da0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0152520 msgr2=0x7f1db0157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0152520 0x7f1db0157380 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f1da4035040 tx=0x7f1da40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 shutdown_connections 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1d9c06b4d0 0x7f1d9c06d990 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db0155c80 0x7f1db01578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1db0152520 0x7f1db0157380 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 --2- 172.21.15.94:0/3518135384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db00720a0 0x7f1db00ac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 >> 172.21.15.94:0/3518135384 conn(0x7f1db00c76b0 msgr2=0x7f1db0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 shutdown_connections 2024-03-06T07:28:18.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:18.061+0000 7f1db57ac640 1 -- 172.21.15.94:0/3518135384 wait complete. 2024-03-06T07:28:18.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:18.477 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:19.478 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:19.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.589+0000 7f9078e08640 1 Processor -- start 2024-03-06T07:28:19.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.589+0000 7f9078e08640 1 -- start start 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f9078e08640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9074152960 0x7f907406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f9078e08640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f90740ac260 0x7f90741559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f9078e08640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9074152d40 con 0x7f90741592d0 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f9078e08640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9074066e30 con 0x7f9074152960 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f9078e08640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90740c1dc0 con 0x7f90740ac260 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f907259c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9074152960 0x7f907406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f907259c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9074152960 0x7f907406dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50904/0 (socket says 172.21.15.94:50904) 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f907259c640 1 -- 172.21.15.94:0/2442519165 learned_addr learned my addr 172.21.15.94:0/2442519165 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f9072d9d640 1 --2- 172.21.15.94:0/2442519165 >> v2:172.21.15.145:3300/0 conn(0x7f90740ac260 0x7f90741559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f907259c640 1 -- 172.21.15.94:0/2442519165 >> v2:172.21.15.145:3300/0 conn(0x7f90740ac260 msgr2=0x7f90741559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f907259c640 1 --2- 172.21.15.94:0/2442519165 >> v2:172.21.15.145:3300/0 conn(0x7f90740ac260 0x7f90741559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f907259c640 1 -- 172.21.15.94:0/2442519165 >> v1:172.21.15.145:6789/0 conn(0x7f90741592d0 legacy=0x7f90741552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.590+0000 7f907259c640 1 -- 172.21.15.94:0/2442519165 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740c1b50 con 0x7f9074152960 2024-03-06T07:28:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.591+0000 7f907259c640 1 --2- 172.21.15.94:0/2442519165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9074152960 0x7f907406dc90 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f907406b380 tx=0x7f906402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e83763a17807d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.591+0000 7f9071d9b640 1 -- 172.21.15.94:0/2442519165 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064034070 con 0x7f9074152960 2024-03-06T07:28:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.591+0000 7f9071d9b640 1 -- 172.21.15.94:0/2442519165 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064030860 con 0x7f9074152960 2024-03-06T07:28:19.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.591+0000 7f9071d9b640 1 -- 172.21.15.94:0/2442519165 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064030c30 con 0x7f9074152960 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 -- 172.21.15.94:0/2442519165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9074152960 msgr2=0x7f907406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 --2- 172.21.15.94:0/2442519165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9074152960 0x7f907406dc90 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f907406b380 tx=0x7f906402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 -- 172.21.15.94:0/2442519165 shutdown_connections 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 --2- 172.21.15.94:0/2442519165 >> v2:172.21.15.145:3300/0 conn(0x7f90740ac260 0x7f90741559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 --2- 172.21.15.94:0/2442519165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9074152960 0x7f907406dc90 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 -- 172.21.15.94:0/2442519165 >> 172.21.15.94:0/2442519165 conn(0x7f90740c70b0 msgr2=0x7f90740c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 -- 172.21.15.94:0/2442519165 shutdown_connections 2024-03-06T07:28:19.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.592+0000 7f9078e08640 1 -- 172.21.15.94:0/2442519165 wait complete. 2024-03-06T07:28:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.593+0000 7f9078e08640 1 Processor -- start 2024-03-06T07:28:19.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.593+0000 7f9078e08640 1 -- start start 2024-03-06T07:28:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9078e08640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90740ac260 0x7f907414e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9078e08640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9074152960 0x7f907406bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9078e08640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f907414db60 0x7f907413f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9078e08640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90740c20a0 con 0x7f907414db60 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9078e08640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074159760 con 0x7f90740ac260 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9078e08640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90740c2220 con 0x7f9074152960 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f907414db60 0x7f907413f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f907414db60 0x7f907413f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50914/0 (socket says 172.21.15.94:50914) 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 -- 172.21.15.94:0/2322106946 learned_addr learned my addr 172.21.15.94:0/2322106946 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f907259c640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90740ac260 0x7f907414e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f906bfff640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9074152960 0x7f907406bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 -- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9074152960 msgr2=0x7f907406bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9074152960 0x7f907406bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 -- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90740ac260 msgr2=0x7f907414e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90740ac260 0x7f907414e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.599 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 -- 172.21.15.94:0/2322106946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90741556b0 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f907259c640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90740ac260 0x7f907414e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.594+0000 7f9072d9d640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f907414db60 0x7f907413f770 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f906c009cb0 tx=0x7f906c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.595+0000 7f906affd640 1 -- 172.21.15.94:0/2322106946 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c018020 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.595+0000 7f906affd640 1 -- 172.21.15.94:0/2322106946 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c014480 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.595+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90741ef0f0 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.595+0000 7f906affd640 1 -- 172.21.15.94:0/2322106946 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c014870 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.595+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9074155340 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.596+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907404f330 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.596+0000 7f906affd640 1 -- 172.21.15.94:0/2322106946 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f906c005d10 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.597+0000 7f906affd640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f905406b500 0x7f905406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.597+0000 7f906affd640 1 -- 172.21.15.94:0/2322106946 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f906c08af50 con 0x7f907414db60 2024-03-06T07:28:19.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.597+0000 7f907259c640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f905406b500 0x7f905406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.597+0000 7f907259c640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f905406b500 0x7f905406d9c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9064036000 tx=0x7f9064008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.601+0000 7f906affd640 1 -- 172.21.15.94:0/2322106946 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f906c022030 con 0x7f907414db60 2024-03-06T07:28:19.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.807+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9074156270 con 0x7f907414db60 2024-03-06T07:28:19.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.808+0000 7f906affd640 1 -- 172.21.15.94:0/2322106946 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f906c010040 con 0x7f907414db60 2024-03-06T07:28:19.810 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:19.810 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:19.810 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f905406b500 msgr2=0x7f905406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f905406b500 0x7f905406d9c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9064036000 tx=0x7f9064008030 comp rx=0 tx=0).stop 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f907414db60 msgr2=0x7f907413f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f907414db60 0x7f907413f770 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f906c009cb0 tx=0x7f906c00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 shutdown_connections 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f905406b500 0x7f905406d9c0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f907414db60 0x7f907413f770 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9074152960 0x7f907406bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 --2- 172.21.15.94:0/2322106946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90740ac260 0x7f907414e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 >> 172.21.15.94:0/2322106946 conn(0x7f90740c70b0 msgr2=0x7f907406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 shutdown_connections 2024-03-06T07:28:19.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:19.810+0000 7f9078e08640 1 -- 172.21.15.94:0/2322106946 wait complete. 2024-03-06T07:28:20.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:20.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:21.226 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.337+0000 7f7dc3ab5640 1 Processor -- start 2024-03-06T07:28:21.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.337+0000 7f7dc3ab5640 1 -- start start 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc3ab5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc1550c0 0x7f7dbc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc3ab5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7dbc0aa180 0x7f7dbc0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc3ab5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc0aa660 con 0x7f7dbc1559e0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc3ab5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0c4610 con 0x7f7dbc0aa180 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc3ab5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0aa940 con 0x7f7dbc1550c0 2024-03-06T07:28:21.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc1050640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7dbc1559e0 0x7f7dbc14ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40658/0 (socket says 172.21.15.94:40658) 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc1050640 1 -- 172.21.15.94:0/42837950 learned_addr learned my addr 172.21.15.94:0/42837950 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc2052640 1 --2- 172.21.15.94:0/42837950 >> v2:172.21.15.145:3300/0 conn(0x7f7dbc0aa180 0x7f7dbc0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc1851640 1 --2- 172.21.15.94:0/42837950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc1550c0 0x7f7dbc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc084f640 1 -- 172.21.15.94:0/42837950 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724179995 0 0) 0x7f7dbc0aa660 con 0x7f7dbc1559e0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc084f640 1 -- 172.21.15.94:0/42837950 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da8003520 con 0x7f7dbc1559e0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc1851640 1 -- 172.21.15.94:0/42837950 >> v2:172.21.15.145:3300/0 conn(0x7f7dbc0aa180 msgr2=0x7f7dbc0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc1851640 1 --2- 172.21.15.94:0/42837950 >> v2:172.21.15.145:3300/0 conn(0x7f7dbc0aa180 0x7f7dbc0c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc1851640 1 -- 172.21.15.94:0/42837950 >> v1:172.21.15.94:6789/0 conn(0x7f7dbc1559e0 legacy=0x7f7dbc14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc1851640 1 -- 172.21.15.94:0/42837950 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0c4860 con 0x7f7dbc1550c0 2024-03-06T07:28:21.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.338+0000 7f7dc2052640 1 --2- 172.21.15.94:0/42837950 >> v2:172.21.15.145:3300/0 conn(0x7f7dbc0aa180 0x7f7dbc0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.339+0000 7f7dc1851640 1 --2- 172.21.15.94:0/42837950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc1550c0 0x7f7dbc1554a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7db80097a0 tx=0x7f7db802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53b4836b919cbfd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.339+0000 7f7dc084f640 1 -- 172.21.15.94:0/42837950 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8038070 con 0x7f7dbc1550c0 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.339+0000 7f7dc084f640 1 -- 172.21.15.94:0/42837950 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db8031700 con 0x7f7dbc1550c0 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.339+0000 7f7dc084f640 1 -- 172.21.15.94:0/42837950 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8031ad0 con 0x7f7dbc1550c0 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/42837950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc1550c0 msgr2=0x7f7dbc1554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/42837950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc1550c0 0x7f7dbc1554a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7db80097a0 tx=0x7f7db802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/42837950 shutdown_connections 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/42837950 >> v2:172.21.15.145:3300/0 conn(0x7f7dbc0aa180 0x7f7dbc0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/42837950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc1550c0 0x7f7dbc1554a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/42837950 >> 172.21.15.94:0/42837950 conn(0x7f7dbc0c6710 msgr2=0x7f7dbc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/42837950 shutdown_connections 2024-03-06T07:28:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.340+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/42837950 wait complete. 2024-03-06T07:28:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.341+0000 7f7dc3ab5640 1 Processor -- start 2024-03-06T07:28:21.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.341+0000 7f7dc3ab5640 1 -- start start 2024-03-06T07:28:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc3ab5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc0aa180 0x7f7dbc06d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc3ab5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dbc1550c0 0x7f7dbc0bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc3ab5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dbc1559e0 0x7f7dbc1ebec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc3ab5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc14f310 con 0x7f7dbc1550c0 2024-03-06T07:28:21.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc3ab5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0c1520 con 0x7f7dbc1559e0 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc3ab5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc14f490 con 0x7f7dbc0aa180 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dbc1550c0 0x7f7dbc0bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dbc1550c0 0x7f7dbc0bfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50930/0 (socket says 172.21.15.94:50930) 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 -- 172.21.15.94:0/1600161792 learned_addr learned my addr 172.21.15.94:0/1600161792 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1851640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc0aa180 0x7f7dbc06d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc2052640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dbc1559e0 0x7f7dbc1ebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 -- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc0aa180 msgr2=0x7f7dbc06d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc0aa180 0x7f7dbc06d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 -- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dbc1559e0 msgr2=0x7f7dbc1ebec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dbc1559e0 0x7f7dbc1ebec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 -- 172.21.15.94:0/1600161792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc1eb410 con 0x7f7dbc1550c0 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc1050640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dbc1550c0 0x7f7dbc0bfdb0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c390 tx=0x7f7dac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7db27fc640 1 -- 172.21.15.94:0/1600161792 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac017070 con 0x7f7dbc1550c0 2024-03-06T07:28:21.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7db27fc640 1 -- 172.21.15.94:0/1600161792 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac005ce0 con 0x7f7dbc1550c0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc1eb6a0 con 0x7f7dbc1550c0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.342+0000 7f7db27fc640 1 -- 172.21.15.94:0/1600161792 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac0072f0 con 0x7f7dbc1550c0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.343+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc06c890 con 0x7f7dbc1550c0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.344+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc0aa660 con 0x7f7dbc1550c0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.344+0000 7f7db27fc640 1 -- 172.21.15.94:0/1600161792 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dac01a070 con 0x7f7dbc1550c0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.344+0000 7f7db27fc640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7da006b500 0x7f7da006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.345+0000 7f7db27fc640 1 -- 172.21.15.94:0/1600161792 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f7dac0896b0 con 0x7f7dbc1550c0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.345+0000 7f7dc1851640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7da006b500 0x7f7da006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.345+0000 7f7dc1851640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7da006b500 0x7f7da006d9c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7db8035000 tx=0x7f7db8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.349+0000 7f7db27fc640 1 -- 172.21.15.94:0/1600161792 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dbc0aa660 con 0x7f7dbc1550c0 2024-03-06T07:28:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.559+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dbc04f330 con 0x7f7dbc1550c0 2024-03-06T07:28:21.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.559+0000 7f7db27fc640 1 -- 172.21.15.94:0/1600161792 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f7dac057d30 con 0x7f7dbc1550c0 2024-03-06T07:28:21.562 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:21.562 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:21.562 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7da006b500 msgr2=0x7f7da006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7da006b500 0x7f7da006d9c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7db8035000 tx=0x7f7db8008030 comp rx=0 tx=0).stop 2024-03-06T07:28:21.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dbc1550c0 msgr2=0x7f7dbc0bfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dbc1550c0 0x7f7dbc0bfdb0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c390 tx=0x7f7dac00c850 comp rx=0 tx=0).stop 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 shutdown_connections 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7da006b500 0x7f7da006d9c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dbc1559e0 0x7f7dbc1ebec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dbc1550c0 0x7f7dbc0bfdb0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 --2- 172.21.15.94:0/1600161792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dbc0aa180 0x7f7dbc06d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 >> 172.21.15.94:0/1600161792 conn(0x7f7dbc0c6710 msgr2=0x7f7dbc14d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 shutdown_connections 2024-03-06T07:28:21.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:21.561+0000 7f7dc3ab5640 1 -- 172.21.15.94:0/1600161792 wait complete. 2024-03-06T07:28:21.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:21.976 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:22.977 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:23.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.088+0000 7f444ec81640 1 Processor -- start 2024-03-06T07:28:23.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.088+0000 7f444ec81640 1 -- start start 2024-03-06T07:28:23.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ec81640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4448150920 0x7f44480c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ec81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f444806c720 0x7f4448071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ec81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44480c2e70 0x7f4448071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ec81640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4448072400 con 0x7f4448150920 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ec81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4448067340 con 0x7f444806c720 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ec81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4448150d00 con 0x7f44480c2e70 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4448150920 0x7f44480c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4448150920 0x7f44480c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50932/0 (socket says 172.21.15.94:50932) 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 -- 172.21.15.94:0/3404801412 learned_addr learned my addr 172.21.15.94:0/3404801412 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444d21e640 1 --2- 172.21.15.94:0/3404801412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44480c2e70 0x7f4448071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f4447fff640 1 --2- 172.21.15.94:0/3404801412 >> v2:172.21.15.145:3300/0 conn(0x7f444806c720 0x7f4448071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 -- 172.21.15.94:0/3404801412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44480c2e70 msgr2=0x7f4448071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 --2- 172.21.15.94:0/3404801412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44480c2e70 0x7f4448071ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 -- 172.21.15.94:0/3404801412 >> v2:172.21.15.145:3300/0 conn(0x7f444806c720 msgr2=0x7f4448071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 --2- 172.21.15.94:0/3404801412 >> v2:172.21.15.145:3300/0 conn(0x7f444806c720 0x7f4448071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.089+0000 7f444ca1d640 1 -- 172.21.15.94:0/3404801412 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44480c2010 con 0x7f4448150920 2024-03-06T07:28:23.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.090+0000 7f444ca1d640 1 --2- 172.21.15.94:0/3404801412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f44480c2930 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f444806ef00 tx=0x7f443c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129646d3c5af4a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.090+0000 7f44477fe640 1 -- 172.21.15.94:0/3404801412 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c02e070 con 0x7f4448150920 2024-03-06T07:28:23.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.090+0000 7f44477fe640 1 -- 172.21.15.94:0/3404801412 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c037070 con 0x7f4448150920 2024-03-06T07:28:23.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.090+0000 7f44477fe640 1 -- 172.21.15.94:0/3404801412 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c031a00 con 0x7f4448150920 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 -- 172.21.15.94:0/3404801412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 msgr2=0x7f44480c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 --2- 172.21.15.94:0/3404801412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f44480c2930 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f444806ef00 tx=0x7f443c02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 -- 172.21.15.94:0/3404801412 shutdown_connections 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 --2- 172.21.15.94:0/3404801412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f44480c2e70 0x7f4448071ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 --2- 172.21.15.94:0/3404801412 >> v2:172.21.15.145:3300/0 conn(0x7f444806c720 0x7f4448071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 --2- 172.21.15.94:0/3404801412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f44480c2930 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 -- 172.21.15.94:0/3404801412 >> 172.21.15.94:0/3404801412 conn(0x7f44480c70b0 msgr2=0x7f44480c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 -- 172.21.15.94:0/3404801412 shutdown_connections 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.091+0000 7f444ec81640 1 -- 172.21.15.94:0/3404801412 wait complete. 2024-03-06T07:28:23.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.092+0000 7f444ec81640 1 Processor -- start 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.092+0000 7f444ec81640 1 -- start start 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ec81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f444806c720 0x7f44481e99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ec81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44480c2e70 0x7f4448156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ec81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f4448152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ec81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4448073090 con 0x7f4448150920 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ec81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4448072b70 con 0x7f44480c2e70 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ec81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4448073210 con 0x7f444806c720 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f4448152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f4448152c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50946/0 (socket says 172.21.15.94:50946) 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 -- 172.21.15.94:0/3511408112 learned_addr learned my addr 172.21.15.94:0/3511408112 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ca1d640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f444806c720 0x7f44481e99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f4447fff640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44480c2e70 0x7f4448156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 -- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f444806c720 msgr2=0x7f44481e99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f444806c720 0x7f44481e99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 -- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44480c2e70 msgr2=0x7f4448156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44480c2e70 0x7f4448156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 -- 172.21.15.94:0/3511408112 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44481e9ee0 con 0x7f4448150920 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444d21e640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f4448152c00 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f4440009d30 tx=0x7f444000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f44457fa640 1 -- 172.21.15.94:0/3511408112 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440018020 con 0x7f4448150920 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f44457fa640 1 -- 172.21.15.94:0/3511408112 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4440014480 con 0x7f4448150920 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f44457fa640 1 -- 172.21.15.94:0/3511408112 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440014870 con 0x7f4448150920 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.093+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44481e91e0 con 0x7f4448150920 2024-03-06T07:28:23.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.094+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4448156c10 con 0x7f4448150920 2024-03-06T07:28:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.094+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444814f710 con 0x7f4448150920 2024-03-06T07:28:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.095+0000 7f44457fa640 1 -- 172.21.15.94:0/3511408112 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4440005d10 con 0x7f4448150920 2024-03-06T07:28:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.095+0000 7f44457fa640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f442c06b4d0 0x7f442c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.095+0000 7f44457fa640 1 -- 172.21.15.94:0/3511408112 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f444008ad20 con 0x7f4448150920 2024-03-06T07:28:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.096+0000 7f444ca1d640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f442c06b4d0 0x7f442c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.096+0000 7f444ca1d640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f442c06b4d0 0x7f442c06d990 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f443c036000 tx=0x7f443c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.100+0000 7f44457fa640 1 -- 172.21.15.94:0/3511408112 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f444814f710 con 0x7f4448150920 2024-03-06T07:28:23.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.306+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4448069460 con 0x7f4448150920 2024-03-06T07:28:23.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.307+0000 7f44457fa640 1 -- 172.21.15.94:0/3511408112 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f4448069460 con 0x7f4448150920 2024-03-06T07:28:23.309 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:23.309 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:23.309 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f442c06b4d0 msgr2=0x7f442c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f442c06b4d0 0x7f442c06d990 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f443c036000 tx=0x7f443c03d040 comp rx=0 tx=0).stop 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 msgr2=0x7f4448152c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f4448152c00 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f4440009d30 tx=0x7f444000c490 comp rx=0 tx=0).stop 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 shutdown_connections 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f442c06b4d0 0x7f442c06d990 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4448150920 0x7f4448152c00 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f44480c2e70 0x7f4448156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 --2- 172.21.15.94:0/3511408112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f444806c720 0x7f44481e99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 >> 172.21.15.94:0/3511408112 conn(0x7f44480c70b0 msgr2=0x7f444814f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 shutdown_connections 2024-03-06T07:28:23.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:23.309+0000 7f444ec81640 1 -- 172.21.15.94:0/3511408112 wait complete. 2024-03-06T07:28:23.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:23.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:24.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:24.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.838+0000 7fe855e88640 1 Processor -- start 2024-03-06T07:28:24.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.838+0000 7fe855e88640 1 -- start start 2024-03-06T07:28:24.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe855e88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe8500c1aa0 0x7fe8501529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe855e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe850155a40 0x7fe850155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe855e88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8501563d0 con 0x7fe85014f790 2024-03-06T07:28:24.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe855e88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8500c2680 con 0x7fe8500c1aa0 2024-03-06T07:28:24.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe855e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8500c28d0 con 0x7fe850155a40 2024-03-06T07:28:24.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe850155a40 0x7fe850155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84effd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe8500c1aa0 0x7fe8501529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84effd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe8500c1aa0 0x7fe8501529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:55794/0 (socket says 172.21.15.94:55794) 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84f7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fe85014f790 0x7fe8500c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:57264/0 (socket says 172.21.15.94:57264) 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84effd640 1 -- 172.21.15.94:0/3294603831 learned_addr learned my addr 172.21.15.94:0/3294603831 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84effd640 1 -- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe850155a40 msgr2=0x7fe850155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84effd640 1 --2- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe850155a40 0x7fe850155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84effd640 1 -- 172.21.15.94:0/3294603831 >> v1:172.21.15.145:6789/0 conn(0x7fe85014f790 legacy=0x7fe8500c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84effd640 1 -- 172.21.15.94:0/3294603831 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500c2bb0 con 0x7fe8500c1aa0 2024-03-06T07:28:24.842 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.839+0000 7fe84ffff640 1 --2- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe850155a40 0x7fe850155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.840+0000 7fe84effd640 1 --2- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe8501529d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe844009870 tx=0x7fe84402db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd2e7cc9794dfdfd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.840+0000 7fe84e7fc640 1 -- 172.21.15.94:0/3294603831 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84403b070 con 0x7fe8500c1aa0 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.840+0000 7fe84e7fc640 1 -- 172.21.15.94:0/3294603831 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84402e940 con 0x7fe8500c1aa0 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.840+0000 7fe84e7fc640 1 -- 172.21.15.94:0/3294603831 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84402ed10 con 0x7fe8500c1aa0 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 -- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 msgr2=0x7fe8501529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 --2- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe8501529d0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe844009870 tx=0x7fe84402db90 comp rx=0 tx=0).stop 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 -- 172.21.15.94:0/3294603831 shutdown_connections 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 --2- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe850155a40 0x7fe850155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 --2- 172.21.15.94:0/3294603831 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe8501529d0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 -- 172.21.15.94:0/3294603831 >> 172.21.15.94:0/3294603831 conn(0x7fe8500c6710 msgr2=0x7fe8500c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 -- 172.21.15.94:0/3294603831 shutdown_connections 2024-03-06T07:28:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.841+0000 7fe855e88640 1 -- 172.21.15.94:0/3294603831 wait complete. 2024-03-06T07:28:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.842+0000 7fe855e88640 1 Processor -- start 2024-03-06T07:28:24.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.842+0000 7fe855e88640 1 -- start start 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe855e88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe85006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe855e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe85014f790 0x7fe85014c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe855e88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe850155a40 0x7fe85014c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe855e88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8500c0d60 con 0x7fe850155a40 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe855e88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe850155520 con 0x7fe8500c1aa0 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe855e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8500c5e30 con 0x7fe85014f790 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe85006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe850155a40 0x7fe85014c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe85006ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:55808/0 (socket says 172.21.15.94:55808) 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 -- 172.21.15.94:0/1967282407 learned_addr learned my addr 172.21.15.94:0/1967282407 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84effd640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe85014f790 0x7fe85014c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 -- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe85014f790 msgr2=0x7fe85014c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe85014f790 0x7fe85014c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 -- 172.21.15.94:0/1967282407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe850155a40 msgr2=0x7fe85014c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe850155a40 0x7fe85014c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.843+0000 7fe84f7fe640 1 -- 172.21.15.94:0/1967282407 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500c0b90 con 0x7fe8500c1aa0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.844+0000 7fe84f7fe640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe85006ce70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c400 tx=0x7fe84000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.844+0000 7fe837fff640 1 -- 172.21.15.94:0/1967282407 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8400074e0 con 0x7fe8500c1aa0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.844+0000 7fe837fff640 1 -- 172.21.15.94:0/1967282407 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840007680 con 0x7fe8500c1aa0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.844+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe850147eb0 con 0x7fe8500c1aa0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.844+0000 7fe837fff640 1 -- 172.21.15.94:0/1967282407 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840004a00 con 0x7fe8500c1aa0 2024-03-06T07:28:24.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.844+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe850151c60 con 0x7fe8500c1aa0 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.845+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe850151770 con 0x7fe8500c1aa0 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.845+0000 7fe837fff640 1 -- 172.21.15.94:0/1967282407 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe84001a020 con 0x7fe8500c1aa0 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.846+0000 7fe837fff640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.846+0000 7fe837fff640 1 -- 172.21.15.94:0/1967282407 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fe840089a60 con 0x7fe8500c1aa0 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.846+0000 7fe84effd640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.846+0000 7fe84effd640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe83c06b4d0 0x7fe83c06d990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe844038000 tx=0x7fe844032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:24.850+0000 7fe837fff640 1 -- 172.21.15.94:0/1967282407 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe840058730 con 0x7fe8500c1aa0 2024-03-06T07:28:25.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.058+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe85004f330 con 0x7fe8500c1aa0 2024-03-06T07:28:25.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.058+0000 7fe837fff640 1 -- 172.21.15.94:0/1967282407 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7fe840058030 con 0x7fe8500c1aa0 2024-03-06T07:28:25.060 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:25.060 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:25.060 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe83c06b4d0 msgr2=0x7fe83c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe83c06b4d0 0x7fe83c06d990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe844038000 tx=0x7fe844032000 comp rx=0 tx=0).stop 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 msgr2=0x7fe85006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe85006ce70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe84000c400 tx=0x7fe84000c800 comp rx=0 tx=0).stop 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 shutdown_connections 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe850155a40 0x7fe85014c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe85014f790 0x7fe85014c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 --2- 172.21.15.94:0/1967282407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe8500c1aa0 0x7fe85006ce70 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 >> 172.21.15.94:0/1967282407 conn(0x7fe8500c6710 msgr2=0x7fe8500a6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 shutdown_connections 2024-03-06T07:28:25.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:25.060+0000 7fe855e88640 1 -- 172.21.15.94:0/1967282407 wait complete. 2024-03-06T07:28:25.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:25.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:26.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.587+0000 7f4a75d56640 1 Processor -- start 2024-03-06T07:28:26.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.587+0000 7f4a75d56640 1 -- start start 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a75d56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a701554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a75d56640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4a700aa180 0x7f4a700c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a75d56640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700aa660 con 0x7f4a701559e0 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a75d56640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a700c4610 con 0x7f4a700aa180 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a75d56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700aa940 con 0x7f4a701550c0 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a6ffff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4a700aa180 0x7f4a700c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a66ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4a701559e0 0x7f4a7014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40668/0 (socket says 172.21.15.94:40668) 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a66ffd640 1 -- 172.21.15.94:0/305145582 learned_addr learned my addr 172.21.15.94:0/305145582 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a6f7fe640 1 --2- 172.21.15.94:0/305145582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a701554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a6ffff640 1 -- 172.21.15.94:0/305145582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 msgr2=0x7f4a701554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a6ffff640 1 --2- 172.21.15.94:0/305145582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a701554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a6ffff640 1 -- 172.21.15.94:0/305145582 >> v1:172.21.15.94:6789/0 conn(0x7f4a701559e0 legacy=0x7f4a7014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.588+0000 7f4a6ffff640 1 -- 172.21.15.94:0/305145582 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700c4860 con 0x7f4a700aa180 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.589+0000 7f4a6ffff640 1 --2- 172.21.15.94:0/305145582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a700c5060 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009d90 tx=0x7f4a6802bbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cfe06785ebeee51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.589+0000 7f4a6effd640 1 -- 172.21.15.94:0/305145582 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6802d910 con 0x7f4a700aa180 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.589+0000 7f4a6effd640 1 -- 172.21.15.94:0/305145582 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6802dab0 con 0x7f4a700aa180 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.589+0000 7f4a6effd640 1 -- 172.21.15.94:0/305145582 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6802c430 con 0x7f4a700aa180 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 -- 172.21.15.94:0/305145582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 msgr2=0x7f4a700c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 --2- 172.21.15.94:0/305145582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a700c5060 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009d90 tx=0x7f4a6802bbd0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 -- 172.21.15.94:0/305145582 shutdown_connections 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 --2- 172.21.15.94:0/305145582 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a700c5060 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 --2- 172.21.15.94:0/305145582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a701554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 -- 172.21.15.94:0/305145582 >> 172.21.15.94:0/305145582 conn(0x7f4a700c6710 msgr2=0x7f4a700c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 -- 172.21.15.94:0/305145582 shutdown_connections 2024-03-06T07:28:26.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.590+0000 7f4a75d56640 1 -- 172.21.15.94:0/305145582 wait complete. 2024-03-06T07:28:26.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.591+0000 7f4a75d56640 1 Processor -- start 2024-03-06T07:28:26.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.591+0000 7f4a75d56640 1 -- start start 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a75d56640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a70143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a75d56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a7013c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a75d56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a701559e0 0x7f4a7013cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a75d56640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7014f310 con 0x7f4a700aa180 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a75d56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700c1520 con 0x7f4a701559e0 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a75d56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7014f490 con 0x7f4a701550c0 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a70143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a70143180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:50972/0 (socket says 172.21.15.94:50972) 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 -- 172.21.15.94:0/2775172270 learned_addr learned my addr 172.21.15.94:0/2775172270 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a66ffd640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a7013c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6ffff640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a701559e0 0x7f4a7013cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 -- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 msgr2=0x7f4a7013c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a7013c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 -- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a701559e0 msgr2=0x7f4a7013cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a701559e0 0x7f4a7013cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 -- 172.21.15.94:0/2775172270 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a701e91e0 con 0x7f4a700aa180 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.592+0000 7f4a6f7fe640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a70143180 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000e870 tx=0x7f4a6000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.593+0000 7f4a6cff9640 1 -- 172.21.15.94:0/2775172270 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60017070 con 0x7f4a700aa180 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.593+0000 7f4a6cff9640 1 -- 172.21.15.94:0/2775172270 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6000c940 con 0x7f4a700aa180 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.593+0000 7f4a6cff9640 1 -- 172.21.15.94:0/2775172270 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6000cd30 con 0x7f4a700aa180 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.593+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a701e94c0 con 0x7f4a700aa180 2024-03-06T07:28:26.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.593+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a701436c0 con 0x7f4a700aa180 2024-03-06T07:28:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.594+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7004f330 con 0x7f4a700aa180 2024-03-06T07:28:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.594+0000 7f4a6cff9640 1 -- 172.21.15.94:0/2775172270 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a6001a070 con 0x7f4a700aa180 2024-03-06T07:28:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.594+0000 7f4a6cff9640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5006b4d0 0x7f4a5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.594+0000 7f4a6cff9640 1 -- 172.21.15.94:0/2775172270 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f4a6008a5b0 con 0x7f4a700aa180 2024-03-06T07:28:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.595+0000 7f4a66ffd640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5006b4d0 0x7f4a5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.595+0000 7f4a66ffd640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5006b4d0 0x7f4a5006d990 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4a700754e0 tx=0x7f4a58007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.599+0000 7f4a6cff9640 1 -- 172.21.15.94:0/2775172270 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a60058ba0 con 0x7f4a700aa180 2024-03-06T07:28:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.807+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a700c2940 con 0x7f4a700aa180 2024-03-06T07:28:26.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.808+0000 7f4a6cff9640 1 -- 172.21.15.94:0/2775172270 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f4a6005cc40 con 0x7f4a700aa180 2024-03-06T07:28:26.810 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:26.810 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:26.810 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5006b4d0 msgr2=0x7f4a5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5006b4d0 0x7f4a5006d990 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4a700754e0 tx=0x7f4a58007e20 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 msgr2=0x7f4a70143180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a70143180 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000e870 tx=0x7f4a6000ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 shutdown_connections 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4a5006b4d0 0x7f4a5006d990 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a701559e0 0x7f4a7013cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a701550c0 0x7f4a7013c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 --2- 172.21.15.94:0/2775172270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4a700aa180 0x7f4a70143180 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 >> 172.21.15.94:0/2775172270 conn(0x7f4a700c6710 msgr2=0x7f4a7014dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 shutdown_connections 2024-03-06T07:28:26.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:26.810+0000 7f4a75d56640 1 -- 172.21.15.94:0/2775172270 wait complete. 2024-03-06T07:28:27.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:27.229 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:28.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.340+0000 7fb8f8ab5640 1 Processor -- start 2024-03-06T07:28:28.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.340+0000 7fb8f8ab5640 1 -- start start 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f8ab5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f406f120 0x7fb8f41588c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f8ab5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb8f406ebd0 0x7fb8f4158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f8ab5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f40c1910 con 0x7fb8f4152ab0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f8ab5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f40c2cf0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f8ab5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f40c4050 con 0x7fb8f406f120 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb8f4152ab0 0x7fb8f40c4290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40674/0 (socket says 172.21.15.94:40674) 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f259c640 1 -- 172.21.15.94:0/260933876 learned_addr learned my addr 172.21.15.94:0/260933876 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f2d9d640 1 --2- 172.21.15.94:0/260933876 >> v2:172.21.15.94:3300/0 conn(0x7fb8f406ebd0 0x7fb8f4158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.341+0000 7fb8f1d9b640 1 --2- 172.21.15.94:0/260933876 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f406f120 0x7fb8f41588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.342+0000 7fb8f159a640 1 -- 172.21.15.94:0/260933876 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305025988 0 0) 0x7fb8f40c1910 con 0x7fb8f4152ab0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.342+0000 7fb8f159a640 1 -- 172.21.15.94:0/260933876 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8dc003520 con 0x7fb8f4152ab0 2024-03-06T07:28:28.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.342+0000 7fb8f2d9d640 1 -- 172.21.15.94:0/260933876 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f406f120 msgr2=0x7fb8f41588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.342+0000 7fb8f2d9d640 1 --2- 172.21.15.94:0/260933876 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f406f120 0x7fb8f41588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.342+0000 7fb8f2d9d640 1 -- 172.21.15.94:0/260933876 >> v1:172.21.15.94:6789/0 conn(0x7fb8f4152ab0 legacy=0x7fb8f40c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.342+0000 7fb8f2d9d640 1 -- 172.21.15.94:0/260933876 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f40c1bf0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f2d9d640 1 --2- 172.21.15.94:0/260933876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f4158e00 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009dc0 tx=0x7fb8e402be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84887494a31c03ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f159a640 1 -- 172.21.15.94:0/260933876 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4039070 con 0x7fb8f406ebd0 2024-03-06T07:28:28.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f159a640 1 -- 172.21.15.94:0/260933876 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e402cbb0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f159a640 1 -- 172.21.15.94:0/260933876 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4034440 con 0x7fb8f406ebd0 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/260933876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 msgr2=0x7fb8f4158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/260933876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f4158e00 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009dc0 tx=0x7fb8e402be00 comp rx=0 tx=0).stop 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/260933876 shutdown_connections 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/260933876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f4158e00 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/260933876 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f406f120 0x7fb8f41588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/260933876 >> 172.21.15.94:0/260933876 conn(0x7fb8f40c7800 msgr2=0x7fb8f40c7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/260933876 shutdown_connections 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.343+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/260933876 wait complete. 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 Processor -- start 2024-03-06T07:28:28.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 -- start start 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f414f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8f406f120 0x7fb8f4158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f4152ab0 0x7fb8f41eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f40c1380 con 0x7fb8f406ebd0 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f40c6f60 con 0x7fb8f406f120 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f8ab5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f40c1500 con 0x7fb8f4152ab0 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f414f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f414f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:51002/0 (socket says 172.21.15.94:51002) 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.345+0000 7fb8f259c640 1 -- 172.21.15.94:0/2010253324 learned_addr learned my addr 172.21.15.94:0/2010253324 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f1d9b640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8f406f120 0x7fb8f4158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.350 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f2d9d640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f4152ab0 0x7fb8f41eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f259c640 1 -- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f4152ab0 msgr2=0x7fb8f41eb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f259c640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f4152ab0 0x7fb8f41eb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f259c640 1 -- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8f406f120 msgr2=0x7fb8f4158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f259c640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8f406f120 0x7fb8f4158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f259c640 1 -- 172.21.15.94:0/2010253324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4158e20 con 0x7fb8f406ebd0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f259c640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f414f7b0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00c390 tx=0x7fb8ec00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8e2ffd640 1 -- 172.21.15.94:0/2010253324 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec017070 con 0x7fb8f406ebd0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8e2ffd640 1 -- 172.21.15.94:0/2010253324 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec005ce0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f41ef2b0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8e2ffd640 1 -- 172.21.15.94:0/2010253324 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec0072f0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.346+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f41550e0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.347+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f41564b0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.351 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.348+0000 7fb8e2ffd640 1 -- 172.21.15.94:0/2010253324 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8ec01a070 con 0x7fb8f406ebd0 2024-03-06T07:28:28.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.348+0000 7fb8e2ffd640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d406b500 0x7fb8d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.348+0000 7fb8e2ffd640 1 -- 172.21.15.94:0/2010253324 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fb8ec0896d0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.348+0000 7fb8f1d9b640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d406b500 0x7fb8d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.352 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.349+0000 7fb8f1d9b640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d406b500 0x7fb8d406d9c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb8f41543d0 tx=0x7fb8e8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.353+0000 7fb8e2ffd640 1 -- 172.21.15.94:0/2010253324 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb8f41564b0 con 0x7fb8f406ebd0 2024-03-06T07:28:28.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.563+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8f404f330 con 0x7fb8f406ebd0 2024-03-06T07:28:28.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.563+0000 7fb8e2ffd640 1 -- 172.21.15.94:0/2010253324 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7fb8ec057d50 con 0x7fb8f406ebd0 2024-03-06T07:28:28.565 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:28.565 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:28.565 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d406b500 msgr2=0x7fb8d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d406b500 0x7fb8d406d9c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb8f41543d0 tx=0x7fb8e8009190 comp rx=0 tx=0).stop 2024-03-06T07:28:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 msgr2=0x7fb8f414f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f414f7b0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00c390 tx=0x7fb8ec00c850 comp rx=0 tx=0).stop 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 shutdown_connections 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb8d406b500 0x7fb8d406d9c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8f4152ab0 0x7fb8f41eb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8f406f120 0x7fb8f4158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 --2- 172.21.15.94:0/2010253324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb8f406ebd0 0x7fb8f414f7b0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 >> 172.21.15.94:0/2010253324 conn(0x7fb8f40c7800 msgr2=0x7fb8f40c53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 shutdown_connections 2024-03-06T07:28:28.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:28.565+0000 7fb8f8ab5640 1 -- 172.21.15.94:0/2010253324 wait complete. 2024-03-06T07:28:28.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:28.980 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:29.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:30.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.091+0000 7f7907d7a640 1 Processor -- start 2024-03-06T07:28:30.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.091+0000 7f7907d7a640 1 -- start start 2024-03-06T07:28:30.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7907d7a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f790014f310 0x7f79000c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7907d7a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7900155bb0 con 0x7f79001550c0 2024-03-06T07:28:30.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7907d7a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79000aa660 con 0x7f79000aa180 2024-03-06T07:28:30.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7907d7a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79000c4610 con 0x7f790014f310 2024-03-06T07:28:30.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7905b16640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f79001550c0 0x7f79001554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49450/0 (socket says 172.21.15.94:49450) 2024-03-06T07:28:30.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7905b16640 1 -- 172.21.15.94:0/2311541547 learned_addr learned my addr 172.21.15.94:0/2311541547 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:30.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7906317640 1 --2- 172.21.15.94:0/2311541547 >> v2:172.21.15.145:3300/0 conn(0x7f790014f310 0x7f79000c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7904b14640 1 -- 172.21.15.94:0/2311541547 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918414618 0 0) 0x7f7900155bb0 con 0x7f79001550c0 2024-03-06T07:28:30.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.092+0000 7f7904b14640 1 -- 172.21.15.94:0/2311541547 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ec003520 con 0x7f79001550c0 2024-03-06T07:28:30.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.093+0000 7f7906317640 1 -- 172.21.15.94:0/2311541547 >> v1:172.21.15.145:6789/0 conn(0x7f79000aa180 legacy=0x7f790014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.093+0000 7f7906317640 1 -- 172.21.15.94:0/2311541547 >> v1:172.21.15.94:6789/0 conn(0x7f79001550c0 legacy=0x7f79001554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.093+0000 7f7904b14640 1 -- 172.21.15.94:0/2311541547 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579668356 0 0) 0x7f79000aa660 con 0x7f79000aa180 2024-03-06T07:28:30.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.093+0000 7f7906317640 1 -- 172.21.15.94:0/2311541547 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79000aa940 con 0x7f790014f310 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.093+0000 7f7906317640 1 --2- 172.21.15.94:0/2311541547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 0x7f79000c5060 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009d90 tx=0x7f78f802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=589df1016ffcd06e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.093+0000 7f7904b14640 1 -- 172.21.15.94:0/2311541547 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8039070 con 0x7f790014f310 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7904b14640 1 -- 172.21.15.94:0/2311541547 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f802cbd0 con 0x7f790014f310 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7904b14640 1 -- 172.21.15.94:0/2311541547 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f8034440 con 0x7f790014f310 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7907d7a640 1 -- 172.21.15.94:0/2311541547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 msgr2=0x7f79000c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7907d7a640 1 --2- 172.21.15.94:0/2311541547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 0x7f79000c5060 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009d90 tx=0x7f78f802be20 comp rx=0 tx=0).stop 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7907d7a640 1 -- 172.21.15.94:0/2311541547 shutdown_connections 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7907d7a640 1 --2- 172.21.15.94:0/2311541547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 0x7f79000c5060 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7907d7a640 1 -- 172.21.15.94:0/2311541547 >> 172.21.15.94:0/2311541547 conn(0x7f79000c6710 msgr2=0x7f79000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7907d7a640 1 -- 172.21.15.94:0/2311541547 shutdown_connections 2024-03-06T07:28:30.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.094+0000 7f7907d7a640 1 -- 172.21.15.94:0/2311541547 wait complete. 2024-03-06T07:28:30.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.095+0000 7f7907d7a640 1 Processor -- start 2024-03-06T07:28:30.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.095+0000 7f7907d7a640 1 -- start start 2024-03-06T07:28:30.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7907d7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79000aa180 0x7f790014dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7907d7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 0x7f79001e9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7907d7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79001550c0 0x7f79001e9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7907d7a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79000c1490 con 0x7f79000aa180 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7907d7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79000c1310 con 0x7f790014f310 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7907d7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79000c3210 con 0x7f79001550c0 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79000aa180 0x7f790014dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79000aa180 0x7f790014dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47798/0 (socket says 172.21.15.94:47798) 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 -- 172.21.15.94:0/3350805085 learned_addr learned my addr 172.21.15.94:0/3350805085 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7906317640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79001550c0 0x7f79001e9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905315640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 0x7f79001e9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 -- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79001550c0 msgr2=0x7f79001e9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79001550c0 0x7f79001e9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 -- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 msgr2=0x7f79001e9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 0x7f79001e9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 -- 172.21.15.94:0/3350805085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79000bbb10 con 0x7f79000aa180 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.096+0000 7f7905b16640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79000aa180 0x7f790014dbf0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c3c0 tx=0x7f78fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.097+0000 7f78f67fc640 1 -- 172.21.15.94:0/3350805085 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc017070 con 0x7f79000aa180 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.097+0000 7f78f67fc640 1 -- 172.21.15.94:0/3350805085 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc005ce0 con 0x7f79000aa180 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.097+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7900072420 con 0x7f79000aa180 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.097+0000 7f78f67fc640 1 -- 172.21.15.94:0/3350805085 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0072f0 con 0x7f79000aa180 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.097+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79000bbdf0 con 0x7f79000aa180 2024-03-06T07:28:30.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.098+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790004f330 con 0x7f79000aa180 2024-03-06T07:28:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.098+0000 7f78f67fc640 1 -- 172.21.15.94:0/3350805085 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78fc01a070 con 0x7f79000aa180 2024-03-06T07:28:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.099+0000 7f78f67fc640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b500 0x7f78e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.099+0000 7f78f67fc640 1 -- 172.21.15.94:0/3350805085 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f78fc089660 con 0x7f79000aa180 2024-03-06T07:28:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.099+0000 7f7905315640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b500 0x7f78e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.099+0000 7f7905315640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b500 0x7f78e406d9c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f790014ffd0 tx=0x7f78f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.103+0000 7f78f67fc640 1 -- 172.21.15.94:0/3350805085 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78fc057ce0 con 0x7f79000aa180 2024-03-06T07:28:30.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.311+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79000c0900 con 0x7f79000aa180 2024-03-06T07:28:30.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.312+0000 7f78f67fc640 1 -- 172.21.15.94:0/3350805085 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f78fc05bd80 con 0x7f79000aa180 2024-03-06T07:28:30.314 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:30.314 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:30.314 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b500 msgr2=0x7f78e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b500 0x7f78e406d9c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f790014ffd0 tx=0x7f78f0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79000aa180 msgr2=0x7f790014dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79000aa180 0x7f790014dbf0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c3c0 tx=0x7f78fc00c880 comp rx=0 tx=0).stop 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 shutdown_connections 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f78e406b500 0x7f78e406d9c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79001550c0 0x7f79001e9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f790014f310 0x7f79001e9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 --2- 172.21.15.94:0/3350805085 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79000aa180 0x7f790014dbf0 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 >> 172.21.15.94:0/3350805085 conn(0x7f79000c6710 msgr2=0x7f7900154710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 shutdown_connections 2024-03-06T07:28:30.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:30.314+0000 7f7907d7a640 1 -- 172.21.15.94:0/3350805085 wait complete. 2024-03-06T07:28:30.729 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:30.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:31.731 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:31.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.841+0000 7fc3e5cb5640 1 Processor -- start 2024-03-06T07:28:31.843 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.841+0000 7fc3e5cb5640 1 -- start start 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3e5cb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3e006afc0 0x7fc3e006b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3e5cb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e00a9530 con 0x7fc3e00c25c0 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3e5cb5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e00a9810 con 0x7fc3e00c20b0 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3e5cb5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e00a9af0 con 0x7fc3e006afc0 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3df7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3e006afc0 0x7fc3e006b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3dffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc3e00c25c0 0x7fc3e0153890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49452/0 (socket says 172.21.15.94:49452) 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3dffff640 1 -- 172.21.15.94:0/2354437887 learned_addr learned my addr 172.21.15.94:0/2354437887 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3df7fe640 1 -- 172.21.15.94:0/2354437887 >> v1:172.21.15.145:6789/0 conn(0x7fc3e00c20b0 legacy=0x7fc3e0153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3df7fe640 1 -- 172.21.15.94:0/2354437887 >> v1:172.21.15.94:6789/0 conn(0x7fc3e00c25c0 legacy=0x7fc3e0153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3de7fc640 1 -- 172.21.15.94:0/2354437887 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130369432 0 0) 0x7fc3e00a9530 con 0x7fc3e00c25c0 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.842+0000 7fc3df7fe640 1 -- 172.21.15.94:0/2354437887 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e00a9dd0 con 0x7fc3e006afc0 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.843+0000 7fc3df7fe640 1 --2- 172.21.15.94:0/2354437887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e006afc0 0x7fc3e006b3a0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00097a0 tx=0x7fc3d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cccb80f955dc3d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.843+0000 7fc3de7fc640 1 -- 172.21.15.94:0/2354437887 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d00315a0 con 0x7fc3e006afc0 2024-03-06T07:28:31.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.843+0000 7fc3de7fc640 1 -- 172.21.15.94:0/2354437887 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d0037070 con 0x7fc3e006afc0 2024-03-06T07:28:31.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.843+0000 7fc3de7fc640 1 -- 172.21.15.94:0/2354437887 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0031960 con 0x7fc3e006afc0 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.844+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/2354437887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e006afc0 msgr2=0x7fc3e006b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.844+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/2354437887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e006afc0 0x7fc3e006b3a0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00097a0 tx=0x7fc3d002b840 comp rx=0 tx=0).stop 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.844+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/2354437887 shutdown_connections 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.844+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/2354437887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e006afc0 0x7fc3e006b3a0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.844+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/2354437887 >> 172.21.15.94:0/2354437887 conn(0x7fc3e00c76b0 msgr2=0x7fc3e00c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.844+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/2354437887 shutdown_connections 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.844+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/2354437887 wait complete. 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.845+0000 7fc3e5cb5640 1 Processor -- start 2024-03-06T07:28:31.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.845+0000 7fc3e5cb5640 1 -- start start 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3e5cb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3e006afc0 0x7fc3e006c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3e5cb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e00c20b0 0x7fc3e0149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3e5cb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3e00c25c0 0x7fc3e0149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3e5cb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e0150b00 con 0x7fc3e00c20b0 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3e5cb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e0150980 con 0x7fc3e00c25c0 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3e5cb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e00a7d50 con 0x7fc3e006afc0 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e00c20b0 0x7fc3e0149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e00c20b0 0x7fc3e0149970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47814/0 (socket says 172.21.15.94:47814) 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 -- 172.21.15.94:0/3995926769 learned_addr learned my addr 172.21.15.94:0/3995926769 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3df7fe640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3e006afc0 0x7fc3e006c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.850 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3dffff640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3e00c25c0 0x7fc3e0149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 -- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3e006afc0 msgr2=0x7fc3e006c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3e006afc0 0x7fc3e006c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 -- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3e00c25c0 msgr2=0x7fc3e0149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3e00c25c0 0x7fc3e0149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 -- 172.21.15.94:0/3995926769 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e0155260 con 0x7fc3e00c20b0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3deffd640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e00c20b0 0x7fc3e0149970 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400c430 tx=0x7fc3d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3c7fff640 1 -- 172.21.15.94:0/3995926769 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4017070 con 0x7fc3e00c20b0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3c7fff640 1 -- 172.21.15.94:0/3995926769 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d40052d0 con 0x7fc3e00c20b0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3c7fff640 1 -- 172.21.15.94:0/3995926769 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4003830 con 0x7fc3e00c20b0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.846+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e01ece50 con 0x7fc3e00c20b0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.847+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e0155c60 con 0x7fc3e00c20b0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.847+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e004f330 con 0x7fc3e00c20b0 2024-03-06T07:28:31.851 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.848+0000 7fc3c7fff640 1 -- 172.21.15.94:0/3995926769 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3d401a070 con 0x7fc3e00c20b0 2024-03-06T07:28:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.848+0000 7fc3c7fff640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.848+0000 7fc3c7fff640 1 -- 172.21.15.94:0/3995926769 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fc3d40896f0 con 0x7fc3e00c20b0 2024-03-06T07:28:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.849+0000 7fc3df7fe640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.852 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.849+0000 7fc3df7fe640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0036000 tx=0x7fc3d0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.854 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:31.853+0000 7fc3c7fff640 1 -- 172.21.15.94:0/3995926769 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3d4057d70 con 0x7fc3e00c20b0 2024-03-06T07:28:32.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.058+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3e006bc50 con 0x7fc3e00c20b0 2024-03-06T07:28:32.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.059+0000 7fc3c7fff640 1 -- 172.21.15.94:0/3995926769 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7fc3d405be10 con 0x7fc3e00c20b0 2024-03-06T07:28:32.061 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:32.061 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:32.061 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:32.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3cc06b4d0 msgr2=0x7fc3cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0036000 tx=0x7fc3d0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:32.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e00c20b0 msgr2=0x7fc3e0149970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e00c20b0 0x7fc3e0149970 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400c430 tx=0x7fc3d400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 shutdown_connections 2024-03-06T07:28:32.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.063 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3e00c25c0 0x7fc3e0149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3e00c20b0 0x7fc3e0149970 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 --2- 172.21.15.94:0/3995926769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3e006afc0 0x7fc3e006c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 >> 172.21.15.94:0/3995926769 conn(0x7fc3e00c76b0 msgr2=0x7fc3e014ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 shutdown_connections 2024-03-06T07:28:32.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:32.061+0000 7fc3e5cb5640 1 -- 172.21.15.94:0/3995926769 wait complete. 2024-03-06T07:28:32.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:32.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:33.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:33.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.586+0000 7f032b895640 1 Processor -- start 2024-03-06T07:28:33.588 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.586+0000 7f032b895640 1 -- start start 2024-03-06T07:28:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f032b895640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f032414f790 0x7f03241529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f032b895640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03240c1aa0 0x7f03240c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f032b895640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03240c28d0 con 0x7f03240c5c60 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f032b895640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03240c2680 con 0x7f032414f790 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f032b895640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03240c2bb0 con 0x7f03240c1aa0 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0329631640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f032414f790 0x7f03241529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0329631640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f032414f790 0x7f03241529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47826/0 (socket says 172.21.15.94:47826) 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0329631640 1 -- 172.21.15.94:0/2633091448 learned_addr learned my addr 172.21.15.94:0/2633091448 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0328e30640 1 --2- 172.21.15.94:0/2633091448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03240c1aa0 0x7f03240c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f031bfff640 1 -- 172.21.15.94:0/2633091448 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891268084 0 0) 0x7f03240c28d0 con 0x7f03240c5c60 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0329631640 1 -- 172.21.15.94:0/2633091448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03240c1aa0 msgr2=0x7f03240c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0329631640 1 --2- 172.21.15.94:0/2633091448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03240c1aa0 0x7f03240c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0329631640 1 -- 172.21.15.94:0/2633091448 >> v1:172.21.15.145:6789/0 conn(0x7f03240c5c60 legacy=0x7f0324155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.587+0000 7f0329631640 1 -- 172.21.15.94:0/2633091448 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03240c2e90 con 0x7f032414f790 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.588+0000 7f0329631640 1 --2- 172.21.15.94:0/2633091448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f032414f790 0x7f03241529d0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0320009870 tx=0x7f032002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccf210191c45241 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.588+0000 7f031bfff640 1 -- 172.21.15.94:0/2633091448 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0320034070 con 0x7f032414f790 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.588+0000 7f031bfff640 1 -- 172.21.15.94:0/2633091448 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0320030840 con 0x7f032414f790 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.588+0000 7f031bfff640 1 -- 172.21.15.94:0/2633091448 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0320030c10 con 0x7f032414f790 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 -- 172.21.15.94:0/2633091448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f032414f790 msgr2=0x7f03241529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 --2- 172.21.15.94:0/2633091448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f032414f790 0x7f03241529d0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0320009870 tx=0x7f032002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 -- 172.21.15.94:0/2633091448 shutdown_connections 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 --2- 172.21.15.94:0/2633091448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03240c1aa0 0x7f03240c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 --2- 172.21.15.94:0/2633091448 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f032414f790 0x7f03241529d0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 -- 172.21.15.94:0/2633091448 >> 172.21.15.94:0/2633091448 conn(0x7f03240c6710 msgr2=0x7f03240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 -- 172.21.15.94:0/2633091448 shutdown_connections 2024-03-06T07:28:33.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.589+0000 7f032b895640 1 -- 172.21.15.94:0/2633091448 wait complete. 2024-03-06T07:28:33.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.590+0000 7f032b895640 1 Processor -- start 2024-03-06T07:28:33.593 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.590+0000 7f032b895640 1 -- start start 2024-03-06T07:28:33.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f032b895640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03240c1aa0 0x7f0324072770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.595 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f032b895640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03240c5c60 0x7f03241ddca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f032b895640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f032414f790 0x7f032414c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f032b895640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032406cc00 con 0x7f03240c1aa0 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f032b895640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03240c0d60 con 0x7f03240c5c60 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f032b895640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032406cd80 con 0x7f032414f790 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03240c1aa0 0x7f0324072770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03240c1aa0 0x7f0324072770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47828/0 (socket says 172.21.15.94:47828) 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 -- 172.21.15.94:0/1815356897 learned_addr learned my addr 172.21.15.94:0/1815356897 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329e32640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f032414f790 0x7f032414c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0328e30640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03240c5c60 0x7f03241ddca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 -- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f032414f790 msgr2=0x7f032414c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f032414f790 0x7f032414c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 -- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03240c5c60 msgr2=0x7f03241ddca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03240c5c60 0x7f03241ddca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.596 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 -- 172.21.15.94:0/1815356897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324151a10 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0328e30640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03240c5c60 0x7f03241ddca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0329631640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03240c1aa0 0x7f0324072770 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0320009870 tx=0x7f03200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0319ffb640 1 -- 172.21.15.94:0/1815356897 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0320034040 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0319ffb640 1 -- 172.21.15.94:0/1815356897 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0320039d90 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032414d260 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.591+0000 7f0319ffb640 1 -- 172.21.15.94:0/1815356897 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0320043610 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.592+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0324151730 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.593+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0324151270 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.593+0000 7f0319ffb640 1 -- 172.21.15.94:0/1815356897 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f032004a060 con 0x7f03240c1aa0 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.594+0000 7f0319ffb640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f030806b4d0 0x7f030806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.597 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.594+0000 7f0319ffb640 1 -- 172.21.15.94:0/1815356897 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f03200ac380 con 0x7f03240c1aa0 2024-03-06T07:28:33.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.594+0000 7f0328e30640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f030806b4d0 0x7f030806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.598 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.594+0000 7f0328e30640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f030806b4d0 0x7f030806d990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f03240a7a90 tx=0x7f0314008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.600 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.598+0000 7f0319ffb640 1 -- 172.21.15.94:0/1815356897 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0324151270 con 0x7f03240c1aa0 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.803+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f032404f330 con 0x7f03240c1aa0 2024-03-06T07:28:33.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.804+0000 7f0319ffb640 1 -- 172.21.15.94:0/1815356897 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f032003e070 con 0x7f03240c1aa0 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:33.806 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f030806b4d0 msgr2=0x7f030806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f030806b4d0 0x7f030806d990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f03240a7a90 tx=0x7f0314008040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03240c1aa0 msgr2=0x7f0324072770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03240c1aa0 0x7f0324072770 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f0320009870 tx=0x7f03200046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 shutdown_connections 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f030806b4d0 0x7f030806d990 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f032414f790 0x7f032414c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03240c5c60 0x7f03241ddca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 --2- 172.21.15.94:0/1815356897 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f03240c1aa0 0x7f0324072770 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 >> 172.21.15.94:0/1815356897 conn(0x7f03240c6710 msgr2=0x7f032406df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 shutdown_connections 2024-03-06T07:28:33.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:33.806+0000 7f032b895640 1 -- 172.21.15.94:0/1815356897 wait complete. 2024-03-06T07:28:34.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:34.221 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:35.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:35.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.334+0000 7f5d2f0ec640 1 Processor -- start 2024-03-06T07:28:35.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.334+0000 7f5d2f0ec640 1 -- start start 2024-03-06T07:28:35.339 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d2f0ec640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5d2814f310 0x7f5d280c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d2f0ec640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d28155bb0 con 0x7f5d281550c0 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d2f0ec640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d280aa660 con 0x7f5d280aa180 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d2f0ec640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d280c4610 con 0x7f5d2814f310 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d2ce88640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5d281550c0 0x7f5d281554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49462/0 (socket says 172.21.15.94:49462) 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d2ce88640 1 -- 172.21.15.94:0/1908484702 learned_addr learned my addr 172.21.15.94:0/1908484702 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d2d689640 1 --2- 172.21.15.94:0/1908484702 >> v2:172.21.15.145:3300/0 conn(0x7f5d2814f310 0x7f5d280c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299318548 0 0) 0x7f5d28155bb0 con 0x7f5d281550c0 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d10003520 con 0x7f5d281550c0 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122642800 0 0) 0x7f5d280aa660 con 0x7f5d280aa180 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.335+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d28155bb0 con 0x7f5d280aa180 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805819784 0 0) 0x7f5d10003520 con 0x7f5d281550c0 2024-03-06T07:28:35.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 >> v2:172.21.15.145:3300/0 conn(0x7f5d2814f310 msgr2=0x7f5d280c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 --2- 172.21.15.94:0/1908484702 >> v2:172.21.15.145:3300/0 conn(0x7f5d2814f310 0x7f5d280c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 >> v1:172.21.15.145:6789/0 conn(0x7f5d280aa180 legacy=0x7f5d2814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d280aa940 con 0x7f5d281550c0 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f5d18001670 con 0x7f5d281550c0 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2d689640 1 --2- 172.21.15.94:0/1908484702 >> v2:172.21.15.145:3300/0 conn(0x7f5d2814f310 0x7f5d280c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 >> v1:172.21.15.94:6789/0 conn(0x7f5d281550c0 legacy=0x7f5d281554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d10003b40 0x7f5d10003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10004800 0x7f5d10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d10005230 0x7f5d1000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d280c4610 con 0x7f5d10005230 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1000e3a0 con 0x7f5d10003b40 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1000e5a0 con 0x7f5d10004800 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2d689640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d10005230 0x7f5d1000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2ce88640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d10003b40 0x7f5d10003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d27fff640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10004800 0x7f5d10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2d689640 1 -- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10004800 msgr2=0x7f5d10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2d689640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10004800 0x7f5d10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2d689640 1 -- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d10003b40 msgr2=0x7f5d10003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2d689640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d10003b40 0x7f5d10003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.336+0000 7f5d2d689640 1 -- 172.21.15.94:0/1908484702 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10003520 con 0x7f5d10005230 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d2d689640 1 -- 172.21.15.94:0/1908484702 reap_dead start 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d2d689640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d10005230 0x7f5d1000dbd0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000a3f0 tx=0x7f5d10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb144e3e57ec5cd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000a040 con 0x7f5d10005230 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d20036420 con 0x7f5d10005230 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d277fe640 1 -- 172.21.15.94:0/1908484702 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d20036810 con 0x7f5d10005230 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1908484702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d10005230 msgr2=0x7f5d1000dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d10005230 0x7f5d1000dbd0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000a3f0 tx=0x7f5d10005830 comp rx=0 tx=0).stop 2024-03-06T07:28:35.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1908484702 shutdown_connections 2024-03-06T07:28:35.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1908484702 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d10005230 0x7f5d1000dbd0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.337+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1908484702 >> 172.21.15.94:0/1908484702 conn(0x7f5d280c6710 msgr2=0x7f5d280c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.338+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1908484702 shutdown_connections 2024-03-06T07:28:35.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.338+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1908484702 wait complete. 2024-03-06T07:28:35.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 Processor -- start 2024-03-06T07:28:35.345 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 -- start start 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10005230 0x7f5d280bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d280702f0 0x7f5d281ea3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d2806c480 0x7f5d281e7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d280c1490 con 0x7f5d280702f0 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d280c1310 con 0x7f5d2806c480 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.339+0000 7f5d2f0ec640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2814ec00 con 0x7f5d10005230 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d280702f0 0x7f5d281ea3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d280702f0 0x7f5d281ea3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47858/0 (socket says 172.21.15.94:47858) 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 -- 172.21.15.94:0/1246673240 learned_addr learned my addr 172.21.15.94:0/1246673240 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d2ce88640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10005230 0x7f5d280bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d2d689640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d2806c480 0x7f5d281e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 -- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10005230 msgr2=0x7f5d280bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10005230 0x7f5d280bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 -- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d2806c480 msgr2=0x7f5d281e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d2806c480 0x7f5d281e7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 -- 172.21.15.94:0/1246673240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2814c120 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d27fff640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d280702f0 0x7f5d281ea3e0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00f7a0 tx=0x7f5d1c00fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d257fa640 1 -- 172.21.15.94:0/1246673240 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c0050d0 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d257fa640 1 -- 172.21.15.94:0/1246673240 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c018070 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d257fa640 1 -- 172.21.15.94:0/1246673240 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c00d6d0 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2814be40 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.340+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d280c02f0 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.341+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2804f330 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.342+0000 7f5d257fa640 1 -- 172.21.15.94:0/1246673240 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d1c00d870 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.342+0000 7f5d257fa640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d1406b4d0 0x7f5d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.342+0000 7f5d257fa640 1 -- 172.21.15.94:0/1246673240 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f5d1c089fe0 con 0x7f5d280702f0 2024-03-06T07:28:35.347 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.342+0000 7f5d2ce88640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d1406b4d0 0x7f5d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.343+0000 7f5d2ce88640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d1406b4d0 0x7f5d1406d990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007b40 tx=0x7f5d18001e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.348 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.346+0000 7f5d257fa640 1 -- 172.21.15.94:0/1246673240 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d1c058d70 con 0x7f5d280702f0 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.554+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d28068450 con 0x7f5d280702f0 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.554+0000 7f5d257fa640 1 -- 172.21.15.94:0/1246673240 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4223 (secure 0 0 0) 0x7f5d1c058670 con 0x7f5d280702f0 2024-03-06T07:28:35.556 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":206,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10773,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":193}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:06.281184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10776":{"gid":10776,"name":"c","rank":1,"incarnation":201,"state":"up:stopping","state_seq":17,"addr":"172.21.15.94:6835/2402539280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6834","nonce":2402539280},{"type":"v1","addr":"172.21.15.94:6835","nonce":2402539280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:35.557 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 206 2024-03-06T07:28:35.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d1406b4d0 msgr2=0x7f5d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d1406b4d0 0x7f5d1406d990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007b40 tx=0x7f5d18001e50 comp rx=0 tx=0).stop 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d280702f0 msgr2=0x7f5d281ea3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d280702f0 0x7f5d281ea3e0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00f7a0 tx=0x7f5d1c00fba0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 shutdown_connections 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5d1406b4d0 0x7f5d1406d990 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d2806c480 0x7f5d281e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5d280702f0 0x7f5d281ea3e0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 --2- 172.21.15.94:0/1246673240 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d10005230 0x7f5d280bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 >> 172.21.15.94:0/1246673240 conn(0x7f5d280c6710 msgr2=0x7f5d280c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 shutdown_connections 2024-03-06T07:28:35.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:35.556+0000 7f5d2f0ec640 1 -- 172.21.15.94:0/1246673240 wait complete. 2024-03-06T07:28:35.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:06.281184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10776}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10776': {'gid': 10776, 'name': 'c', 'rank': 1, 'incarnation': 201, 'state': 'up:stopping', 'state_seq': 17, 'addr': '172.21.15.94:6835/2402539280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6834', 'nonce': 2402539280}, {'type': 'v1', 'addr': '172.21.15.94:6835', 'nonce': 2402539280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:35.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10776:up:stopping 2024-03-06T07:28:36.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:37.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.082+0000 7fc2c04b5640 1 Processor -- start 2024-03-06T07:28:37.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.082+0000 7fc2c04b5640 1 -- start start 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.083+0000 7fc2c04b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc2b814f360 0x7fc2b8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.083+0000 7fc2c04b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc2b8158b60 0x7fc2b8158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.083+0000 7fc2c04b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b80c73d0 con 0x7fc2b80c88c0 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.083+0000 7fc2c04b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b80c8dd0 con 0x7fc2b814f360 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.083+0000 7fc2c04b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b80be2e0 con 0x7fc2b8158b60 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2be251640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc2b814f360 0x7fc2b8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2bda50640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc2b80c88c0 0x7fc2b80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49474/0 (socket says 172.21.15.94:49474) 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2bda50640 1 -- 172.21.15.94:0/205582416 learned_addr learned my addr 172.21.15.94:0/205582416 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2bea52640 1 --2- 172.21.15.94:0/205582416 >> v2:172.21.15.145:3300/0 conn(0x7fc2b8158b60 0x7fc2b8158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2be251640 1 -- 172.21.15.94:0/205582416 >> v2:172.21.15.145:3300/0 conn(0x7fc2b8158b60 msgr2=0x7fc2b8158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2be251640 1 --2- 172.21.15.94:0/205582416 >> v2:172.21.15.145:3300/0 conn(0x7fc2b8158b60 0x7fc2b8158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2be251640 1 -- 172.21.15.94:0/205582416 >> v1:172.21.15.94:6789/0 conn(0x7fc2b80c88c0 legacy=0x7fc2b80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2bd24f640 1 -- 172.21.15.94:0/205582416 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289305609 0 0) 0x7fc2b80c73d0 con 0x7fc2b80c88c0 2024-03-06T07:28:37.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.084+0000 7fc2be251640 1 -- 172.21.15.94:0/205582416 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b80c7670 con 0x7fc2b814f360 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2be251640 1 --2- 172.21.15.94:0/205582416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b814f360 0x7fc2b8155c10 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8152d60 tx=0x7fc2b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07c6817ed3dc548 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2bd24f640 1 -- 172.21.15.94:0/205582416 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b40315a0 con 0x7fc2b814f360 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2bd24f640 1 -- 172.21.15.94:0/205582416 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b4037070 con 0x7fc2b814f360 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2bd24f640 1 -- 172.21.15.94:0/205582416 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4031960 con 0x7fc2b814f360 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 -- 172.21.15.94:0/205582416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b814f360 msgr2=0x7fc2b8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/205582416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b814f360 0x7fc2b8155c10 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8152d60 tx=0x7fc2b402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 -- 172.21.15.94:0/205582416 shutdown_connections 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/205582416 >> v2:172.21.15.145:3300/0 conn(0x7fc2b8158b60 0x7fc2b8158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/205582416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b814f360 0x7fc2b8155c10 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 -- 172.21.15.94:0/205582416 >> 172.21.15.94:0/205582416 conn(0x7fc2b80ac570 msgr2=0x7fc2b80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 -- 172.21.15.94:0/205582416 shutdown_connections 2024-03-06T07:28:37.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.085+0000 7fc2c04b5640 1 -- 172.21.15.94:0/205582416 wait complete. 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 Processor -- start 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 -- start start 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b80c88c0 0x7fc2b80c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2b814f360 0x7fc2b814f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2b8158b60 0x7fc2b80bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80c5a50 con 0x7fc2b80c88c0 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80c7c30 con 0x7fc2b814f360 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2c04b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b80c5bd0 con 0x7fc2b8158b60 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2be251640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b80c88c0 0x7fc2b80c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2be251640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b80c88c0 0x7fc2b80c3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:47880/0 (socket says 172.21.15.94:47880) 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2be251640 1 -- 172.21.15.94:0/2937016341 learned_addr learned my addr 172.21.15.94:0/2937016341 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:37.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2bda50640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2b814f360 0x7fc2b814f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.087+0000 7fc2bea52640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2b8158b60 0x7fc2b80bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2be251640 1 -- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2b8158b60 msgr2=0x7fc2b80bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2be251640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2b8158b60 0x7fc2b80bf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2be251640 1 -- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2b814f360 msgr2=0x7fc2b814f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2be251640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2b814f360 0x7fc2b814f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2be251640 1 -- 172.21.15.94:0/2937016341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b80bfd60 con 0x7fc2b80c88c0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2bda50640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2b814f360 0x7fc2b814f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2be251640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b80c88c0 0x7fc2b80c3c70 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fc2b402be50 tx=0x7fc2b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2aeffd640 1 -- 172.21.15.94:0/2937016341 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4032630 con 0x7fc2b80c88c0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2aeffd640 1 -- 172.21.15.94:0/2937016341 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b40327d0 con 0x7fc2b80c88c0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b81ebc60 con 0x7fc2b80c88c0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2aeffd640 1 -- 172.21.15.94:0/2937016341 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4032bc0 con 0x7fc2b80c88c0 2024-03-06T07:28:37.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.088+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b80be8d0 con 0x7fc2b80c88c0 2024-03-06T07:28:37.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.089+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b80c9f60 con 0x7fc2b80c88c0 2024-03-06T07:28:37.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.090+0000 7fc2aeffd640 1 -- 172.21.15.94:0/2937016341 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2b404c020 con 0x7fc2b80c88c0 2024-03-06T07:28:37.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.090+0000 7fc2aeffd640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc29c06b4d0 0x7fc29c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.090+0000 7fc2aeffd640 1 -- 172.21.15.94:0/2937016341 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fc2b402f020 con 0x7fc2b80c88c0 2024-03-06T07:28:37.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.090+0000 7fc2bda50640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc29c06b4d0 0x7fc29c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.091+0000 7fc2bda50640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc29c06b4d0 0x7fc29c06d990 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80c1bb0 tx=0x7fc2a8007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.095+0000 7fc2aeffd640 1 -- 172.21.15.94:0/2937016341 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2b80c9f60 con 0x7fc2b80c88c0 2024-03-06T07:28:37.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.306+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2b804f330 con 0x7fc2b80c88c0 2024-03-06T07:28:37.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.306+0000 7fc2aeffd640 1 -- 172.21.15.94:0/2937016341 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+3428 (secure 0 0 0) 0x7fc2b4040070 con 0x7fc2b80c88c0 2024-03-06T07:28:37.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:37.308 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":207,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:37.254483+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:replay","state_seq":1,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:37.309 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 207 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc29c06b4d0 msgr2=0x7fc29c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc29c06b4d0 0x7fc29c06d990 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80c1bb0 tx=0x7fc2a8007de0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b80c88c0 msgr2=0x7fc2b80c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b80c88c0 0x7fc2b80c3c70 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fc2b402be50 tx=0x7fc2b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 shutdown_connections 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc29c06b4d0 0x7fc29c06d990 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2b8158b60 0x7fc2b80bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2b814f360 0x7fc2b814f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 --2- 172.21.15.94:0/2937016341 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc2b80c88c0 0x7fc2b80c3c70 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 >> 172.21.15.94:0/2937016341 conn(0x7fc2b80ac570 msgr2=0x7fc2b814e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 shutdown_connections 2024-03-06T07:28:37.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:37.308+0000 7fc2c04b5640 1 -- 172.21.15.94:0/2937016341 wait complete. 2024-03-06T07:28:37.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 207, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:37.254483+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:37.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:replay 2024-03-06T07:28:38.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:38.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.836+0000 7f505d61c640 1 Processor -- start 2024-03-06T07:28:38.838 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.836+0000 7f505d61c640 1 -- start start 2024-03-06T07:28:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f505d61c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f50580c88c0 0x7f5058155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f505d61c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5058158b60 0x7f5058158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f505d61c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50580c73d0 con 0x7f505814f360 2024-03-06T07:28:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f505d61c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50580c8dd0 con 0x7f50580c88c0 2024-03-06T07:28:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f505d61c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50580be2e0 con 0x7f5058158b60 2024-03-06T07:28:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f50567fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f50580c88c0 0x7f5058155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.840 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f5056ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f505814f360 0x7f50580c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43466/0 (socket says 172.21.15.94:43466) 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f5056ffd640 1 -- 172.21.15.94:0/4221884364 learned_addr learned my addr 172.21.15.94:0/4221884364 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f50577fe640 1 --2- 172.21.15.94:0/4221884364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5058158b60 0x7f5058158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f50567fc640 1 -- 172.21.15.94:0/4221884364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5058158b60 msgr2=0x7f5058158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f50567fc640 1 --2- 172.21.15.94:0/4221884364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5058158b60 0x7f5058158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f50567fc640 1 -- 172.21.15.94:0/4221884364 >> v1:172.21.15.94:6789/0 conn(0x7f505814f360 legacy=0x7f50580c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f50567fc640 1 -- 172.21.15.94:0/4221884364 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50580c7670 con 0x7f50580c88c0 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.837+0000 7f5055ffb640 1 -- 172.21.15.94:0/4221884364 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272443821 0 0) 0x7f50580c73d0 con 0x7f505814f360 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.838+0000 7f50567fc640 1 --2- 172.21.15.94:0/4221884364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50580c88c0 0x7f5058155c10 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f504c0097a0 tx=0x7f504c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4aff306e43e3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.838+0000 7f5055ffb640 1 -- 172.21.15.94:0/4221884364 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c0315a0 con 0x7f50580c88c0 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.838+0000 7f5055ffb640 1 -- 172.21.15.94:0/4221884364 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504c037070 con 0x7f50580c88c0 2024-03-06T07:28:38.841 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.838+0000 7f5055ffb640 1 -- 172.21.15.94:0/4221884364 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c031960 con 0x7f50580c88c0 2024-03-06T07:28:38.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 -- 172.21.15.94:0/4221884364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50580c88c0 msgr2=0x7f5058155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 --2- 172.21.15.94:0/4221884364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50580c88c0 0x7f5058155c10 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f504c0097a0 tx=0x7f504c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 -- 172.21.15.94:0/4221884364 shutdown_connections 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 --2- 172.21.15.94:0/4221884364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5058158b60 0x7f5058158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 --2- 172.21.15.94:0/4221884364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50580c88c0 0x7f5058155c10 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 -- 172.21.15.94:0/4221884364 >> 172.21.15.94:0/4221884364 conn(0x7f50580ac570 msgr2=0x7f50580ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 -- 172.21.15.94:0/4221884364 shutdown_connections 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.839+0000 7f505d61c640 1 -- 172.21.15.94:0/4221884364 wait complete. 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.840+0000 7f505d61c640 1 Processor -- start 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.840+0000 7f505d61c640 1 -- start start 2024-03-06T07:28:38.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f505d61c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50580c88c0 0x7f5058150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f505d61c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f505814f360 0x7f50581ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f505d61c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5058158b60 0x7f505814a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f505d61c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50580c5a50 con 0x7f5058158b60 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f505d61c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50580c7c30 con 0x7f50580c88c0 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f505d61c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50580c5bd0 con 0x7f505814f360 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5058158b60 0x7f505814a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5058158b60 0x7f505814a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38528/0 (socket says 172.21.15.94:38528) 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 -- 172.21.15.94:0/1806368092 learned_addr learned my addr 172.21.15.94:0/1806368092 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50567fc640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f505814f360 0x7f50581ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f5056ffd640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50580c88c0 0x7f5058150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 -- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f505814f360 msgr2=0x7f50581ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f505814f360 0x7f50581ee5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 -- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50580c88c0 msgr2=0x7f5058150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50580c88c0 0x7f5058150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.846 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 -- 172.21.15.94:0/1806368092 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50581eef10 con 0x7f5058158b60 2024-03-06T07:28:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f50577fe640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5058158b60 0x7f505814a780 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f505000efc0 tx=0x7f505000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.841+0000 7f503f7fe640 1 -- 172.21.15.94:0/1806368092 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050018020 con 0x7f5058158b60 2024-03-06T07:28:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.842+0000 7f503f7fe640 1 -- 172.21.15.94:0/1806368092 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5050014480 con 0x7f5058158b60 2024-03-06T07:28:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.842+0000 7f503f7fe640 1 -- 172.21.15.94:0/1806368092 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050014870 con 0x7f5058158b60 2024-03-06T07:28:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.842+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5058150360 con 0x7f5058158b60 2024-03-06T07:28:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.842+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5058150640 con 0x7f5058158b60 2024-03-06T07:28:38.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.843+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5058150150 con 0x7f5058158b60 2024-03-06T07:28:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.843+0000 7f503f7fe640 1 -- 172.21.15.94:0/1806368092 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5050005d10 con 0x7f5058158b60 2024-03-06T07:28:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.843+0000 7f503f7fe640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f504406b4d0 0x7f504406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.844+0000 7f503f7fe640 1 -- 172.21.15.94:0/1806368092 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f505008ae30 con 0x7f5058158b60 2024-03-06T07:28:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.844+0000 7f5056ffd640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f504406b4d0 0x7f504406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.844+0000 7f5056ffd640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f504406b4d0 0x7f504406d990 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:38.848+0000 7f503f7fe640 1 -- 172.21.15.94:0/1806368092 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5050022070 con 0x7f5058158b60 2024-03-06T07:28:39.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.053+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f505804f330 con 0x7f5058158b60 2024-03-06T07:28:39.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.054+0000 7f503f7fe640 1 -- 172.21.15.94:0/1806368092 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+3430 (secure 0 0 0) 0x7f5050010040 con 0x7f5058158b60 2024-03-06T07:28:39.056 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:39.056 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":208,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:37.268952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:resolve","state_seq":22,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:39.056 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 208 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.055+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f504406b4d0 msgr2=0x7f504406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.055+0000 7f505d61c640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f504406b4d0 0x7f504406d990 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048007de0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.055+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5058158b60 msgr2=0x7f505814a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.055+0000 7f505d61c640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5058158b60 0x7f505814a780 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f505000efc0 tx=0x7f505000c490 comp rx=0 tx=0).stop 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 shutdown_connections 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f504406b4d0 0x7f504406d990 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5058158b60 0x7f505814a780 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f505814f360 0x7f50581ee5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 --2- 172.21.15.94:0/1806368092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50580c88c0 0x7f5058150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 >> 172.21.15.94:0/1806368092 conn(0x7f50580ac570 msgr2=0x7f505806c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 shutdown_connections 2024-03-06T07:28:39.058 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:39.056+0000 7f505d61c640 1 -- 172.21.15.94:0/1806368092 wait complete. 2024-03-06T07:28:39.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:37.268952+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:resolve', 'state_seq': 22, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:39.470 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:resolve 2024-03-06T07:28:40.471 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:40.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.580+0000 7fcc0f881640 1 Processor -- start 2024-03-06T07:28:40.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.581+0000 7fcc0f881640 1 -- start start 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.581+0000 7fcc0f881640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcc0814f360 0x7fcc08155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.581+0000 7fcc0f881640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc080c73d0 con 0x7fcc080c88c0 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.581+0000 7fcc0f881640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc080be2e0 con 0x7fcc08158b60 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.581+0000 7fcc0f881640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc08159700 con 0x7fcc0814f360 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcc0ce1c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcc080c88c0 0x7fcc080c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43472/0 (socket says 172.21.15.94:43472) 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcc0ce1c640 1 -- 172.21.15.94:0/702895065 learned_addr learned my addr 172.21.15.94:0/702895065 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcc0d61d640 1 --2- 172.21.15.94:0/702895065 >> v2:172.21.15.145:3300/0 conn(0x7fcc0814f360 0x7fcc08155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcbfffff640 1 -- 172.21.15.94:0/702895065 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730999113 0 0) 0x7fcc080c73d0 con 0x7fcc080c88c0 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcbfffff640 1 -- 172.21.15.94:0/702895065 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf4003520 con 0x7fcc080c88c0 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcbfffff640 1 -- 172.21.15.94:0/702895065 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394744386 0 0) 0x7fcc080be2e0 con 0x7fcc08158b60 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcbfffff640 1 -- 172.21.15.94:0/702895065 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc080c73d0 con 0x7fcc08158b60 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcc0d61d640 1 -- 172.21.15.94:0/702895065 >> v1:172.21.15.145:6789/0 conn(0x7fcc08158b60 legacy=0x7fcc08158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcc0d61d640 1 -- 172.21.15.94:0/702895065 >> v1:172.21.15.94:6789/0 conn(0x7fcc080c88c0 legacy=0x7fcc080c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.582+0000 7fcc0d61d640 1 -- 172.21.15.94:0/702895065 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc080c7670 con 0x7fcc0814f360 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcc0d61d640 1 --2- 172.21.15.94:0/702895065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc0814f360 0x7fcc08155c10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcc040097a0 tx=0x7fcc0402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18795e63ae62cfaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcbfffff640 1 -- 172.21.15.94:0/702895065 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04035020 con 0x7fcc0814f360 2024-03-06T07:28:40.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcbfffff640 1 -- 172.21.15.94:0/702895065 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc04031830 con 0x7fcc0814f360 2024-03-06T07:28:40.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcbfffff640 1 -- 172.21.15.94:0/702895065 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04031c00 con 0x7fcc0814f360 2024-03-06T07:28:40.586 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcc0f881640 1 -- 172.21.15.94:0/702895065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc0814f360 msgr2=0x7fcc08155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcc0f881640 1 --2- 172.21.15.94:0/702895065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc0814f360 0x7fcc08155c10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcc040097a0 tx=0x7fcc0402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcc0f881640 1 -- 172.21.15.94:0/702895065 shutdown_connections 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcc0f881640 1 --2- 172.21.15.94:0/702895065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc0814f360 0x7fcc08155c10 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.583+0000 7fcc0f881640 1 -- 172.21.15.94:0/702895065 >> 172.21.15.94:0/702895065 conn(0x7fcc080ac570 msgr2=0x7fcc080ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.584+0000 7fcc0f881640 1 -- 172.21.15.94:0/702895065 shutdown_connections 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.584+0000 7fcc0f881640 1 -- 172.21.15.94:0/702895065 wait complete. 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 Processor -- start 2024-03-06T07:28:40.587 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 -- start start 2024-03-06T07:28:40.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc080c88c0 0x7fcc080abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc0814f360 0x7fcc080c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.589 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc08158b60 0x7fcc080c0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc080c7c30 con 0x7fcc0814f360 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc080c5a50 con 0x7fcc08158b60 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0f881640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc080c5bd0 con 0x7fcc080c88c0 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0ce1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc0814f360 0x7fcc080c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.585+0000 7fcc0ce1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc0814f360 0x7fcc080c07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38534/0 (socket says 172.21.15.94:38534) 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0ce1c640 1 -- 172.21.15.94:0/2653762877 learned_addr learned my addr 172.21.15.94:0/2653762877 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0de1e640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc08158b60 0x7fcc080c0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0d61d640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc080c88c0 0x7fcc080abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0ce1c640 1 -- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc080c88c0 msgr2=0x7fcc080abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0ce1c640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc080c88c0 0x7fcc080abd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0ce1c640 1 -- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc08158b60 msgr2=0x7fcc080c0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.590 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0ce1c640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc08158b60 0x7fcc080c0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0ce1c640 1 -- 172.21.15.94:0/2653762877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc080ca5a0 con 0x7fcc0814f360 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0de1e640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc08158b60 0x7fcc080c0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0ce1c640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc0814f360 0x7fcc080c07f0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800c4d0 tx=0x7fcbf800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcbfdffb640 1 -- 172.21.15.94:0/2653762877 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8017070 con 0x7fcc0814f360 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcbfdffb640 1 -- 172.21.15.94:0/2653762877 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf8005d00 con 0x7fcc0814f360 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0814fb00 con 0x7fcc0814f360 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcbfdffb640 1 -- 172.21.15.94:0/2653762877 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80072b0 con 0x7fcc0814f360 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.586+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc080bfde0 con 0x7fcc0814f360 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.587+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0804f330 con 0x7fcc0814f360 2024-03-06T07:28:40.591 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.588+0000 7fcbfdffb640 1 -- 172.21.15.94:0/2653762877 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbf801a070 con 0x7fcc0814f360 2024-03-06T07:28:40.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.588+0000 7fcbfdffb640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbec06b4d0 0x7fcbec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.588+0000 7fcbfdffb640 1 -- 172.21.15.94:0/2653762877 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcbf8089730 con 0x7fcc0814f360 2024-03-06T07:28:40.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.588+0000 7fcc0d61d640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbec06b4d0 0x7fcbec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.592 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.589+0000 7fcc0d61d640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbec06b4d0 0x7fcbec06d990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcc04036000 tx=0x7fcc04008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.594 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.593+0000 7fcbfdffb640 1 -- 172.21.15.94:0/2653762877 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc080ca5a0 con 0x7fcc0814f360 2024-03-06T07:28:40.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.798+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc080c8d50 con 0x7fcc0814f360 2024-03-06T07:28:40.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.798+0000 7fcbfdffb640 1 -- 172.21.15.94:0/2653762877 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+3429 (secure 0 0 0) 0x7fcbf8057e20 con 0x7fcc0814f360 2024-03-06T07:28:40.800 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:40.800 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":210,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:39.259889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:40.801 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 210 2024-03-06T07:28:40.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbec06b4d0 msgr2=0x7fcbec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbec06b4d0 0x7fcbec06d990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcc04036000 tx=0x7fcc04008030 comp rx=0 tx=0).stop 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc0814f360 msgr2=0x7fcc080c07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc0814f360 0x7fcc080c07f0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800c4d0 tx=0x7fcbf800c990 comp rx=0 tx=0).stop 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 shutdown_connections 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcbec06b4d0 0x7fcbec06d990 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc08158b60 0x7fcc080c0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcc0814f360 0x7fcc080c07f0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 --2- 172.21.15.94:0/2653762877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcc080c88c0 0x7fcc080abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.800+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 >> 172.21.15.94:0/2653762877 conn(0x7fcc080ac570 msgr2=0x7fcc0806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.801+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 shutdown_connections 2024-03-06T07:28:40.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:40.801+0000 7fcc0f881640 1 -- 172.21.15.94:0/2653762877 wait complete. 2024-03-06T07:28:41.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 210, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:39.259889+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:rejoin', 'state_seq': 24, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:41.215 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:rejoin 2024-03-06T07:28:42.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.326+0000 7f18029dd640 1 Processor -- start 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.326+0000 7f18029dd640 1 -- start start 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f18029dd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f17fc0c1aa0 0x7f17fc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f18029dd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc155a40 0x7f17fc155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f18029dd640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17fc1563d0 con 0x7f17fc14f790 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f18029dd640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17fc0c2680 con 0x7f17fc0c1aa0 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f18029dd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc0c28d0 con 0x7f17fc155a40 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17fbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f17fc14f790 0x7f17fc0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43480/0 (socket says 172.21.15.94:43480) 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17fbfff640 1 -- 172.21.15.94:0/653697746 learned_addr learned my addr 172.21.15.94:0/653697746 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f1800f7a640 1 --2- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc155a40 0x7f17fc155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17fb7fe640 1 --2- 172.21.15.94:0/653697746 >> v2:172.21.15.145:3300/0 conn(0x7f17fc0c1aa0 0x7f17fc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17faffd640 1 -- 172.21.15.94:0/653697746 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041184308 0 0) 0x7f17fc1563d0 con 0x7f17fc14f790 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17faffd640 1 -- 172.21.15.94:0/653697746 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e4003520 con 0x7f17fc14f790 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17fb7fe640 1 -- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc155a40 msgr2=0x7f17fc155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17fb7fe640 1 --2- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc155a40 0x7f17fc155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17fb7fe640 1 -- 172.21.15.94:0/653697746 >> v1:172.21.15.94:6789/0 conn(0x7f17fc14f790 legacy=0x7f17fc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f17fb7fe640 1 -- 172.21.15.94:0/653697746 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc0c2bb0 con 0x7f17fc0c1aa0 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.327+0000 7f1800f7a640 1 --2- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc155a40 0x7f17fc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.328+0000 7f17fb7fe640 1 --2- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 0x7f17fc1529d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097a0 tx=0x7f17f002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fecd8b40d5cfe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.328+0000 7f17faffd640 1 -- 172.21.15.94:0/653697746 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0035020 con 0x7f17fc0c1aa0 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.328+0000 7f17faffd640 1 -- 172.21.15.94:0/653697746 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f0031830 con 0x7f17fc0c1aa0 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.328+0000 7f17faffd640 1 -- 172.21.15.94:0/653697746 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0031be0 con 0x7f17fc0c1aa0 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 -- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 msgr2=0x7f17fc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 --2- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 0x7f17fc1529d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097a0 tx=0x7f17f002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 -- 172.21.15.94:0/653697746 shutdown_connections 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 --2- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc155a40 0x7f17fc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 --2- 172.21.15.94:0/653697746 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 0x7f17fc1529d0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 -- 172.21.15.94:0/653697746 >> 172.21.15.94:0/653697746 conn(0x7f17fc0c6710 msgr2=0x7f17fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 -- 172.21.15.94:0/653697746 shutdown_connections 2024-03-06T07:28:42.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.329+0000 7f18029dd640 1 -- 172.21.15.94:0/653697746 wait complete. 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.330+0000 7f18029dd640 1 Processor -- start 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.330+0000 7f18029dd640 1 -- start start 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f18029dd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 0x7f17fc153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f18029dd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc14f790 0x7f17fc14dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f18029dd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17fc155a40 0x7f17fc1ea2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f18029dd640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc0c0d60 con 0x7f17fc155a40 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f18029dd640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc155520 con 0x7f17fc0c1aa0 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f18029dd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc0c5e30 con 0x7f17fc14f790 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17fc155a40 0x7f17fc1ea2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17fc155a40 0x7f17fc1ea2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38550/0 (socket says 172.21.15.94:38550) 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 -- 172.21.15.94:0/1016374708 learned_addr learned my addr 172.21.15.94:0/1016374708 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:42.335 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f17fbfff640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 0x7f17fc153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f17fb7fe640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc14f790 0x7f17fc14dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 -- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc14f790 msgr2=0x7f17fc14dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc14f790 0x7f17fc14dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 -- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 msgr2=0x7f17fc153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 0x7f17fc153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 -- 172.21.15.94:0/1016374708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc1533c0 con 0x7f17fc155a40 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.331+0000 7f1800f7a640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17fc155a40 0x7f17fc1ea2b0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f17f4009e60 tx=0x7f17f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.332+0000 7f17f8ff9640 1 -- 172.21.15.94:0/1016374708 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4018030 con 0x7f17fc155a40 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.332+0000 7f17f8ff9640 1 -- 172.21.15.94:0/1016374708 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f400cdf0 con 0x7f17fc155a40 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.332+0000 7f17f8ff9640 1 -- 172.21.15.94:0/1016374708 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f4003bb0 con 0x7f17fc155a40 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.332+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17fc1e9a40 con 0x7f17fc155a40 2024-03-06T07:28:42.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.332+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc14e2e0 con 0x7f17fc155a40 2024-03-06T07:28:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.333+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc04f330 con 0x7f17fc155a40 2024-03-06T07:28:42.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.333+0000 7f17f8ff9640 1 -- 172.21.15.94:0/1016374708 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17f4005d10 con 0x7f17fc155a40 2024-03-06T07:28:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.334+0000 7f17f8ff9640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.334+0000 7f17f8ff9640 1 -- 172.21.15.94:0/1016374708 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f17f408ac10 con 0x7f17fc155a40 2024-03-06T07:28:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.334+0000 7f17fbfff640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.338 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.334+0000 7f17fbfff640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17e806b4d0 0x7f17e806d990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f17ec005880 tx=0x7f17ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.338+0000 7f17f8ff9640 1 -- 172.21.15.94:0/1016374708 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17f4022070 con 0x7f17fc155a40 2024-03-06T07:28:42.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.546+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17fc0c0b40 con 0x7f17fc155a40 2024-03-06T07:28:42.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.546+0000 7f17f8ff9640 1 -- 172.21.15.94:0/1016374708 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f17f4010040 con 0x7f17fc155a40 2024-03-06T07:28:42.549 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:42.549 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:42.549 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:42.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.548+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17e806b4d0 msgr2=0x7f17e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.548+0000 7f18029dd640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17e806b4d0 0x7f17e806d990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f17ec005880 tx=0x7f17ec009040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.548+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17fc155a40 msgr2=0x7f17fc1ea2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.548+0000 7f18029dd640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17fc155a40 0x7f17fc1ea2b0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f17f4009e60 tx=0x7f17f400c490 comp rx=0 tx=0).stop 2024-03-06T07:28:42.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 shutdown_connections 2024-03-06T07:28:42.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f17e806b4d0 0x7f17e806d990 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17fc155a40 0x7f17fc1ea2b0 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17fc14f790 0x7f17fc14dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 --2- 172.21.15.94:0/1016374708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f17fc0c1aa0 0x7f17fc153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 >> 172.21.15.94:0/1016374708 conn(0x7f17fc0c6710 msgr2=0x7f17fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 shutdown_connections 2024-03-06T07:28:42.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:42.549+0000 7f18029dd640 1 -- 172.21.15.94:0/1016374708 wait complete. 2024-03-06T07:28:42.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:42.964 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:43.965 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.074+0000 7f43416fb640 1 Processor -- start 2024-03-06T07:28:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.074+0000 7f43416fb640 1 -- start start 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f43416fb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f433c0c5330 0x7f433c0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f43416fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c1586a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f43416fb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433c155110 con 0x7f433c0c7dd0 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f43416fb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433c1559e0 con 0x7f433c0c5330 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f43416fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0c8f40 con 0x7f433c0c88b0 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f433c0c5330 0x7f433c0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f433c0c5330 0x7f433c0c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38558/0 (socket says 172.21.15.94:38558) 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433affd640 1 -- 172.21.15.94:0/3245587709 learned_addr learned my addr 172.21.15.94:0/3245587709 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433a7fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c1586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433b7fe640 1 --1- 172.21.15.94:0/3245587709 >> v1:172.21.15.145:6789/0 conn(0x7f433c0c7dd0 0x7f433c06b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:56940/0 (socket says 172.21.15.94:56940) 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f4339ffb640 1 -- 172.21.15.94:0/3245587709 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252752048 0 0) 0x7f4334001490 con 0x7f433c0c7dd0 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433affd640 1 -- 172.21.15.94:0/3245587709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 msgr2=0x7f433c1586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433affd640 1 --2- 172.21.15.94:0/3245587709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c1586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433affd640 1 -- 172.21.15.94:0/3245587709 >> v1:172.21.15.145:6789/0 conn(0x7f433c0c7dd0 legacy=0x7f433c06b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.075+0000 7f433affd640 1 -- 172.21.15.94:0/3245587709 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c155390 con 0x7f433c0c5330 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.076+0000 7f433affd640 1 --2- 172.21.15.94:0/3245587709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c0c5710 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f432c009870 tx=0x7f432c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd63543e92fff85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.076+0000 7f4339ffb640 1 -- 172.21.15.94:0/3245587709 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c035020 con 0x7f433c0c5330 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.076+0000 7f4339ffb640 1 -- 172.21.15.94:0/3245587709 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c031640 con 0x7f433c0c5330 2024-03-06T07:28:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.076+0000 7f4339ffb640 1 -- 172.21.15.94:0/3245587709 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c02d040 con 0x7f433c0c5330 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 -- 172.21.15.94:0/3245587709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 msgr2=0x7f433c0c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 --2- 172.21.15.94:0/3245587709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c0c5710 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f432c009870 tx=0x7f432c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 -- 172.21.15.94:0/3245587709 shutdown_connections 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 --2- 172.21.15.94:0/3245587709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c1586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 --2- 172.21.15.94:0/3245587709 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c0c5710 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 -- 172.21.15.94:0/3245587709 >> 172.21.15.94:0/3245587709 conn(0x7f433c0ac560 msgr2=0x7f433c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 -- 172.21.15.94:0/3245587709 shutdown_connections 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.077+0000 7f43416fb640 1 -- 172.21.15.94:0/3245587709 wait complete. 2024-03-06T07:28:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.078+0000 7f43416fb640 1 Processor -- start 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.078+0000 7f43416fb640 1 -- start start 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f43416fb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f43416fb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f433c0c7dd0 0x7f433c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f43416fb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f43416fb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c1529d0 con 0x7f433c0c5330 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f43416fb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c06d090 con 0x7f433c0c7dd0 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f43416fb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c152b50 con 0x7f433c0c88b0 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38572/0 (socket says 172.21.15.94:38572) 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 -- 172.21.15.94:0/3229645491 learned_addr learned my addr 172.21.15.94:0/3229645491 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433b7fe640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433a7fc640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f433c0c7dd0 0x7f433c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 -- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 msgr2=0x7f433c147a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c147a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 -- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f433c0c7dd0 msgr2=0x7f433c0ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f433c0c7dd0 0x7f433c0ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.079+0000 7f433affd640 1 -- 172.21.15.94:0/3229645491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0c3730 con 0x7f433c0c5330 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.080+0000 7f433affd640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c159210 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f432c009870 tx=0x7f432c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.080+0000 7f43237fe640 1 -- 172.21.15.94:0/3229645491 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c040070 con 0x7f433c0c5330 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.080+0000 7f43237fe640 1 -- 172.21.15.94:0/3229645491 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c0399f0 con 0x7f433c0c5330 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.080+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c0c3a10 con 0x7f433c0c5330 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.080+0000 7f43237fe640 1 -- 172.21.15.94:0/3229645491 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c035040 con 0x7f433c0c5330 2024-03-06T07:28:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.080+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c157400 con 0x7f433c0c5330 2024-03-06T07:28:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.081+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c155e70 con 0x7f433c0c5330 2024-03-06T07:28:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.081+0000 7f43237fe640 1 -- 172.21.15.94:0/3229645491 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f432c038020 con 0x7f433c0c5330 2024-03-06T07:28:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.082+0000 7f43237fe640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f432806b500 0x7f432806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.082+0000 7f43237fe640 1 -- 172.21.15.94:0/3229645491 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f432c0abcc0 con 0x7f433c0c5330 2024-03-06T07:28:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.082+0000 7f433a7fc640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f432806b500 0x7f432806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.082+0000 7f433a7fc640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f432806b500 0x7f432806d9c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f433c0c4400 tx=0x7f4330009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.086+0000 7f43237fe640 1 -- 172.21.15.94:0/3229645491 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f433c155e70 con 0x7f433c0c5330 2024-03-06T07:28:44.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.293+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f433c067910 con 0x7f433c0c5330 2024-03-06T07:28:44.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.293+0000 7f43237fe640 1 -- 172.21.15.94:0/3229645491 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f433c067910 con 0x7f433c0c5330 2024-03-06T07:28:44.295 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:44.295 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:44.296 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:44.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f432806b500 msgr2=0x7f432806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f432806b500 0x7f432806d9c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f433c0c4400 tx=0x7f4330009190 comp rx=0 tx=0).stop 2024-03-06T07:28:44.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 msgr2=0x7f433c159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c159210 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f432c009870 tx=0x7f432c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 shutdown_connections 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f432806b500 0x7f432806d9c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f433c0c88b0 0x7f433c147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f433c0c7dd0 0x7f433c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 --2- 172.21.15.94:0/3229645491 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f433c0c5330 0x7f433c159210 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 >> 172.21.15.94:0/3229645491 conn(0x7f433c0ac560 msgr2=0x7f433c154840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 shutdown_connections 2024-03-06T07:28:44.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:44.295+0000 7f43416fb640 1 -- 172.21.15.94:0/3229645491 wait complete. 2024-03-06T07:28:44.711 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:44.711 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:45.712 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.821+0000 7f63e6e37640 1 Processor -- start 2024-03-06T07:28:45.824 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.821+0000 7f63e6e37640 1 -- start start 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.821+0000 7f63e6e37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e014f790 0x7f63e01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.821+0000 7f63e6e37640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00c28d0 con 0x7f63e00c1aa0 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.821+0000 7f63e6e37640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00c2bb0 con 0x7f63e0155a40 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.821+0000 7f63e6e37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c2680 con 0x7f63e014f790 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63dffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f63e00c1aa0 0x7f63e00c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43494/0 (socket says 172.21.15.94:43494) 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63dffff640 1 -- 172.21.15.94:0/1605740852 learned_addr learned my addr 172.21.15.94:0/1605740852 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63e4bd3640 1 --2- 172.21.15.94:0/1605740852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e014f790 0x7f63e01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63df7fe640 1 -- 172.21.15.94:0/1605740852 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752761606 0 0) 0x7f63e00c28d0 con 0x7f63e00c1aa0 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63df7fe640 1 -- 172.21.15.94:0/1605740852 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63cc003520 con 0x7f63e00c1aa0 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63df7fe640 1 -- 172.21.15.94:0/1605740852 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555903446 0 0) 0x7f63e00c2bb0 con 0x7f63e0155a40 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63df7fe640 1 -- 172.21.15.94:0/1605740852 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e00c28d0 con 0x7f63e0155a40 2024-03-06T07:28:45.826 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63e4bd3640 1 -- 172.21.15.94:0/1605740852 >> v1:172.21.15.145:6789/0 conn(0x7f63e0155a40 legacy=0x7f63e0155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63e4bd3640 1 -- 172.21.15.94:0/1605740852 >> v1:172.21.15.94:6789/0 conn(0x7f63e00c1aa0 legacy=0x7f63e00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.822+0000 7f63e4bd3640 1 -- 172.21.15.94:0/1605740852 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00c2e90 con 0x7f63e014f790 2024-03-06T07:28:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.823+0000 7f63e4bd3640 1 --2- 172.21.15.94:0/1605740852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e014f790 0x7f63e01529d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009dc0 tx=0x7f63d802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb3676a0dc879f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.823+0000 7f63df7fe640 1 -- 172.21.15.94:0/1605740852 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8039070 con 0x7f63e014f790 2024-03-06T07:28:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.823+0000 7f63df7fe640 1 -- 172.21.15.94:0/1605740852 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d802ccd0 con 0x7f63e014f790 2024-03-06T07:28:45.827 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.823+0000 7f63df7fe640 1 -- 172.21.15.94:0/1605740852 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8034640 con 0x7f63e014f790 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.824+0000 7f63e6e37640 1 -- 172.21.15.94:0/1605740852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e014f790 msgr2=0x7f63e01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.824+0000 7f63e6e37640 1 --2- 172.21.15.94:0/1605740852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e014f790 0x7f63e01529d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009dc0 tx=0x7f63d802bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.824+0000 7f63e6e37640 1 -- 172.21.15.94:0/1605740852 shutdown_connections 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.824+0000 7f63e6e37640 1 --2- 172.21.15.94:0/1605740852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e014f790 0x7f63e01529d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.824+0000 7f63e6e37640 1 -- 172.21.15.94:0/1605740852 >> 172.21.15.94:0/1605740852 conn(0x7f63e00c6710 msgr2=0x7f63e00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.824+0000 7f63e6e37640 1 -- 172.21.15.94:0/1605740852 shutdown_connections 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.824+0000 7f63e6e37640 1 -- 172.21.15.94:0/1605740852 wait complete. 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.825+0000 7f63e6e37640 1 Processor -- start 2024-03-06T07:28:45.830 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.825+0000 7f63e6e37640 1 -- start start 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e6e37640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e00c1aa0 0x7f63e00c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e6e37640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63e014f790 0x7f63e0071c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e6e37640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63e0155a40 0x7f63e0072150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e6e37640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c5e30 con 0x7f63e0155a40 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e6e37640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c0d60 con 0x7f63e014f790 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e6e37640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c5fb0 con 0x7f63e00c1aa0 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63e0155a40 0x7f63e0072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63e0155a40 0x7f63e0072150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38586/0 (socket says 172.21.15.94:38586) 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 -- 172.21.15.94:0/700029259 learned_addr learned my addr 172.21.15.94:0/700029259 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e4bd3640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e00c1aa0 0x7f63e00c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63dffff640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63e014f790 0x7f63e0071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 -- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e00c1aa0 msgr2=0x7f63e00c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e00c1aa0 0x7f63e00c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 -- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63e014f790 msgr2=0x7f63e0071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63e014f790 0x7f63e0071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 -- 172.21.15.94:0/700029259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e01ea9b0 con 0x7f63e0155a40 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e53d4640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63e0155a40 0x7f63e0072150 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f63d0001eb0 tx=0x7f63d000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63dd7fa640 1 -- 172.21.15.94:0/700029259 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0017070 con 0x7f63e0155a40 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63dd7fa640 1 -- 172.21.15.94:0/700029259 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00052d0 con 0x7f63e0155a40 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.826+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e0150010 con 0x7f63e0155a40 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.827+0000 7f63dd7fa640 1 -- 172.21.15.94:0/700029259 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00037f0 con 0x7f63e0155a40 2024-03-06T07:28:45.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.827+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e01505d0 con 0x7f63e0155a40 2024-03-06T07:28:45.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.828+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e004f330 con 0x7f63e0155a40 2024-03-06T07:28:45.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.828+0000 7f63dd7fa640 1 -- 172.21.15.94:0/700029259 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63d001a070 con 0x7f63e0155a40 2024-03-06T07:28:45.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.828+0000 7f63dd7fa640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f63c006b4d0 0x7f63c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.829+0000 7f63dd7fa640 1 -- 172.21.15.94:0/700029259 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f63d00896b0 con 0x7f63e0155a40 2024-03-06T07:28:45.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.829+0000 7f63e4bd3640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f63c006b4d0 0x7f63c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.833 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.829+0000 7f63e4bd3640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f63c006b4d0 0x7f63c006d990 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f63d8036000 tx=0x7f63d8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.835 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:45.833+0000 7f63dd7fa640 1 -- 172.21.15.94:0/700029259 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63d0057c40 con 0x7f63e0155a40 2024-03-06T07:28:46.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.040+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e00c1f40 con 0x7f63e0155a40 2024-03-06T07:28:46.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.041+0000 7f63dd7fa640 1 -- 172.21.15.94:0/700029259 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f63d005bce0 con 0x7f63e0155a40 2024-03-06T07:28:46.044 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:46.044 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:46.044 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:46.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.042+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f63c006b4d0 msgr2=0x7f63c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.042+0000 7f63e6e37640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f63c006b4d0 0x7f63c006d990 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f63d8036000 tx=0x7f63d8030000 comp rx=0 tx=0).stop 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.042+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63e0155a40 msgr2=0x7f63e0072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.042+0000 7f63e6e37640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63e0155a40 0x7f63e0072150 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f63d0001eb0 tx=0x7f63d000c820 comp rx=0 tx=0).stop 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 shutdown_connections 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f63c006b4d0 0x7f63c006d990 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f63e0155a40 0x7f63e0072150 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63e014f790 0x7f63e0071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 --2- 172.21.15.94:0/700029259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63e00c1aa0 0x7f63e00c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 >> 172.21.15.94:0/700029259 conn(0x7f63e00c6710 msgr2=0x7f63e00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 shutdown_connections 2024-03-06T07:28:46.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:46.043+0000 7f63e6e37640 1 -- 172.21.15.94:0/700029259 wait complete. 2024-03-06T07:28:46.247 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:46.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:46.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:46.459 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:47.460 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.569+0000 7f602864e640 1 Processor -- start 2024-03-06T07:28:47.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.569+0000 7f602864e640 1 -- start start 2024-03-06T07:28:47.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.569+0000 7f602864e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f60201554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.569+0000 7f602864e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60200aa660 con 0x7f60200aa180 2024-03-06T07:28:47.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.569+0000 7f602864e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60200aa940 con 0x7f60201559e0 2024-03-06T07:28:47.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.569+0000 7f602864e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60200c4610 con 0x7f60201550c0 2024-03-06T07:28:47.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f6026beb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f60200aa180 0x7f60200c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43504/0 (socket says 172.21.15.94:43504) 2024-03-06T07:28:47.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f6026beb640 1 -- 172.21.15.94:0/3123272419 learned_addr learned my addr 172.21.15.94:0/3123272419 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:47.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60263ea640 1 --2- 172.21.15.94:0/3123272419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f60201554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60253e8640 1 -- 172.21.15.94:0/3123272419 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641305308 0 0) 0x7f60200aa660 con 0x7f60200aa180 2024-03-06T07:28:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60253e8640 1 -- 172.21.15.94:0/3123272419 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6008003520 con 0x7f60200aa180 2024-03-06T07:28:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60253e8640 1 -- 172.21.15.94:0/3123272419 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665496376 0 0) 0x7f60200aa940 con 0x7f60201559e0 2024-03-06T07:28:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60253e8640 1 -- 172.21.15.94:0/3123272419 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60200aa660 con 0x7f60201559e0 2024-03-06T07:28:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60263ea640 1 -- 172.21.15.94:0/3123272419 >> v1:172.21.15.145:6789/0 conn(0x7f60201559e0 legacy=0x7f602014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60263ea640 1 -- 172.21.15.94:0/3123272419 >> v1:172.21.15.94:6789/0 conn(0x7f60200aa180 legacy=0x7f60200c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.570+0000 7f60263ea640 1 -- 172.21.15.94:0/3123272419 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60200c4860 con 0x7f60201550c0 2024-03-06T07:28:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.571+0000 7f60263ea640 1 --2- 172.21.15.94:0/3123272419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f60201554a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6018009dc0 tx=0x7f601802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa2a3b74ccb4b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.571+0000 7f60253e8640 1 -- 172.21.15.94:0/3123272419 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018039070 con 0x7f60201550c0 2024-03-06T07:28:47.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.571+0000 7f60253e8640 1 -- 172.21.15.94:0/3123272419 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601802cbb0 con 0x7f60201550c0 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.571+0000 7f60253e8640 1 -- 172.21.15.94:0/3123272419 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018034440 con 0x7f60201550c0 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.571+0000 7f602864e640 1 -- 172.21.15.94:0/3123272419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 msgr2=0x7f60201554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.571+0000 7f602864e640 1 --2- 172.21.15.94:0/3123272419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f60201554a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6018009dc0 tx=0x7f601802be00 comp rx=0 tx=0).stop 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.572+0000 7f602864e640 1 -- 172.21.15.94:0/3123272419 shutdown_connections 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.572+0000 7f602864e640 1 --2- 172.21.15.94:0/3123272419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f60201554a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.572+0000 7f602864e640 1 -- 172.21.15.94:0/3123272419 >> 172.21.15.94:0/3123272419 conn(0x7f60200c6710 msgr2=0x7f60200c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.572+0000 7f602864e640 1 -- 172.21.15.94:0/3123272419 shutdown_connections 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.572+0000 7f602864e640 1 -- 172.21.15.94:0/3123272419 wait complete. 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 Processor -- start 2024-03-06T07:28:47.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 -- start start 2024-03-06T07:28:47.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60200aa180 0x7f6020144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f6020151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60201559e0 0x7f6020072220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602014f310 con 0x7f60201559e0 2024-03-06T07:28:47.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60200c1520 con 0x7f60200aa180 2024-03-06T07:28:47.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.573+0000 7f602864e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602014f490 con 0x7f60201550c0 2024-03-06T07:28:47.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60201559e0 0x7f6020072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60201559e0 0x7f6020072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38594/0 (socket says 172.21.15.94:38594) 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 -- 172.21.15.94:0/846306021 learned_addr learned my addr 172.21.15.94:0/846306021 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f60263ea640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60200aa180 0x7f6020144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6025be9640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f6020151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 -- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 msgr2=0x7f6020151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f6020151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 -- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60200aa180 msgr2=0x7f6020144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60200aa180 0x7f6020144e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 -- 172.21.15.94:0/846306021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60200730f0 con 0x7f60201559e0 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f6026beb640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60201559e0 0x7f6020072220 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f601000c3f0 tx=0x7f6010001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f601effd640 1 -- 172.21.15.94:0/846306021 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010017070 con 0x7f60201559e0 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f601effd640 1 -- 172.21.15.94:0/846306021 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6010005ce0 con 0x7f60201559e0 2024-03-06T07:28:47.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6020073390 con 0x7f60201559e0 2024-03-06T07:28:47.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f601effd640 1 -- 172.21.15.94:0/846306021 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60100072d0 con 0x7f60201559e0 2024-03-06T07:28:47.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.574+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602006d660 con 0x7f60201559e0 2024-03-06T07:28:47.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.575+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602004f330 con 0x7f60201559e0 2024-03-06T07:28:47.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.576+0000 7f601effd640 1 -- 172.21.15.94:0/846306021 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f601001a070 con 0x7f60201559e0 2024-03-06T07:28:47.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.576+0000 7f601effd640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f600c06b4d0 0x7f600c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.576+0000 7f601effd640 1 -- 172.21.15.94:0/846306021 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f60100895e0 con 0x7f60201559e0 2024-03-06T07:28:47.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.576+0000 7f60263ea640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f600c06b4d0 0x7f600c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.577+0000 7f60263ea640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f600c06b4d0 0x7f600c06d990 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f6018036000 tx=0x7f6018030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.581+0000 7f601effd640 1 -- 172.21.15.94:0/846306021 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6010007e20 con 0x7f60201559e0 2024-03-06T07:28:47.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.785+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6020068450 con 0x7f60201559e0 2024-03-06T07:28:47.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.786+0000 7f601effd640 1 -- 172.21.15.94:0/846306021 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f6010057c10 con 0x7f60201559e0 2024-03-06T07:28:47.789 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:47.789 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:47.789 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f600c06b4d0 msgr2=0x7f600c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f600c06b4d0 0x7f600c06d990 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f6018036000 tx=0x7f6018030000 comp rx=0 tx=0).stop 2024-03-06T07:28:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60201559e0 msgr2=0x7f6020072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60201559e0 0x7f6020072220 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f601000c3f0 tx=0x7f6010001e70 comp rx=0 tx=0).stop 2024-03-06T07:28:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 shutdown_connections 2024-03-06T07:28:47.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f600c06b4d0 0x7f600c06d990 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60201559e0 0x7f6020072220 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60201550c0 0x7f6020151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 --2- 172.21.15.94:0/846306021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60200aa180 0x7f6020144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 >> 172.21.15.94:0/846306021 conn(0x7f60200c6710 msgr2=0x7f60200c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 shutdown_connections 2024-03-06T07:28:47.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:47.788+0000 7f602864e640 1 -- 172.21.15.94:0/846306021 wait complete. 2024-03-06T07:28:48.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:48.204 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:49.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.315+0000 7ffb5bd13640 1 Processor -- start 2024-03-06T07:28:49.317 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.315+0000 7ffb5bd13640 1 -- start start 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5bd13640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb540c25c0 0x7ffb5406b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5bd13640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb540a9530 con 0x7ffb540c20b0 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5bd13640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb540a9810 con 0x7ffb5406afc0 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5bd13640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb540a9af0 con 0x7ffb540c25c0 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5a2b0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb540c25c0 0x7ffb5406b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb592ae640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ffb540c20b0 0x7ffb54153890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55542/0 (socket says 172.21.15.94:55542) 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb592ae640 1 -- 172.21.15.94:0/369725527 learned_addr learned my addr 172.21.15.94:0/369725527 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:49.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5a2b0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb540c25c0 0x7ffb5406b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54704/0 (socket says 172.21.15.94:54704) 2024-03-06T07:28:49.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb58aad640 1 -- 172.21.15.94:0/369725527 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064149276 0 0) 0x7ffb540a9810 con 0x7ffb5406afc0 2024-03-06T07:28:49.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5a2b0640 1 -- 172.21.15.94:0/369725527 >> v1:172.21.15.145:6789/0 conn(0x7ffb5406afc0 legacy=0x7ffb54153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5a2b0640 1 -- 172.21.15.94:0/369725527 >> v1:172.21.15.94:6789/0 conn(0x7ffb540c20b0 legacy=0x7ffb54153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.316+0000 7ffb5a2b0640 1 -- 172.21.15.94:0/369725527 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb540a9dd0 con 0x7ffb540c25c0 2024-03-06T07:28:49.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.317+0000 7ffb5a2b0640 1 --2- 172.21.15.94:0/369725527 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb540c25c0 0x7ffb5406b3a0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009d90 tx=0x7ffb4c02bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba7d8c3e2725945 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.317+0000 7ffb58aad640 1 -- 172.21.15.94:0/369725527 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c039070 con 0x7ffb540c25c0 2024-03-06T07:28:49.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.317+0000 7ffb58aad640 1 -- 172.21.15.94:0/369725527 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c02c990 con 0x7ffb540c25c0 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.317+0000 7ffb58aad640 1 -- 172.21.15.94:0/369725527 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c02cd60 con 0x7ffb540c25c0 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.318+0000 7ffb5bd13640 1 -- 172.21.15.94:0/369725527 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb540c25c0 msgr2=0x7ffb5406b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.318+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/369725527 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb540c25c0 0x7ffb5406b3a0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009d90 tx=0x7ffb4c02bc60 comp rx=0 tx=0).stop 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.318+0000 7ffb5bd13640 1 -- 172.21.15.94:0/369725527 shutdown_connections 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.318+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/369725527 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb540c25c0 0x7ffb5406b3a0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.318+0000 7ffb5bd13640 1 -- 172.21.15.94:0/369725527 >> 172.21.15.94:0/369725527 conn(0x7ffb540c76b0 msgr2=0x7ffb540c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.318+0000 7ffb5bd13640 1 -- 172.21.15.94:0/369725527 shutdown_connections 2024-03-06T07:28:49.321 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.318+0000 7ffb5bd13640 1 -- 172.21.15.94:0/369725527 wait complete. 2024-03-06T07:28:49.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.319+0000 7ffb5bd13640 1 Processor -- start 2024-03-06T07:28:49.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.319+0000 7ffb5bd13640 1 -- start start 2024-03-06T07:28:49.323 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb5bd13640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb5406afc0 0x7ffb54151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb5bd13640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb540c20b0 0x7ffb54148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb5bd13640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb540c25c0 0x7ffb540c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb5bd13640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb54150b00 con 0x7ffb5406afc0 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb5bd13640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb54150980 con 0x7ffb540c25c0 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb5bd13640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb540a7d50 con 0x7ffb540c20b0 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb5406afc0 0x7ffb54151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb5406afc0 0x7ffb54151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54708/0 (socket says 172.21.15.94:54708) 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 -- 172.21.15.94:0/326665672 learned_addr learned my addr 172.21.15.94:0/326665672 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb5a2b0640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb540c25c0 0x7ffb540c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb592ae640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb540c20b0 0x7ffb54148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 -- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb540c20b0 msgr2=0x7ffb54148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.324 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb540c20b0 0x7ffb54148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 -- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb540c25c0 msgr2=0x7ffb540c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb540c25c0 0x7ffb540c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 -- 172.21.15.94:0/326665672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb540c7150 con 0x7ffb5406afc0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb59aaf640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb5406afc0 0x7ffb54151aa0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c4b0 tx=0x7ffb5000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb4a7fc640 1 -- 172.21.15.94:0/326665672 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb50017070 con 0x7ffb5406afc0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.320+0000 7ffb4a7fc640 1 -- 172.21.15.94:0/326665672 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb50005480 con 0x7ffb5406afc0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.321+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb54150ef0 con 0x7ffb5406afc0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.321+0000 7ffb4a7fc640 1 -- 172.21.15.94:0/326665672 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb50003830 con 0x7ffb5406afc0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.321+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb541517c0 con 0x7ffb5406afc0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.321+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5404f330 con 0x7ffb5406afc0 2024-03-06T07:28:49.325 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.322+0000 7ffb4a7fc640 1 -- 172.21.15.94:0/326665672 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb5001a070 con 0x7ffb5406afc0 2024-03-06T07:28:49.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.322+0000 7ffb4a7fc640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3806b4d0 0x7ffb3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.323+0000 7ffb4a7fc640 1 -- 172.21.15.94:0/326665672 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ffb500896e0 con 0x7ffb5406afc0 2024-03-06T07:28:49.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.323+0000 7ffb592ae640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3806b4d0 0x7ffb3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.326 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.323+0000 7ffb592ae640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3806b4d0 0x7ffb3806d990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ffb540adab0 tx=0x7ffb44007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.328 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.327+0000 7ffb4a7fc640 1 -- 172.21.15.94:0/326665672 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb50057d10 con 0x7ffb5406afc0 2024-03-06T07:28:49.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.534+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb540c1a50 con 0x7ffb5406afc0 2024-03-06T07:28:49.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.534+0000 7ffb4a7fc640 1 -- 172.21.15.94:0/326665672 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7ffb5005bdb0 con 0x7ffb5406afc0 2024-03-06T07:28:49.536 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:49.537 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:49.537 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:49.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3806b4d0 msgr2=0x7ffb3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3806b4d0 0x7ffb3806d990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7ffb540adab0 tx=0x7ffb44007e00 comp rx=0 tx=0).stop 2024-03-06T07:28:49.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb5406afc0 msgr2=0x7ffb54151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb5406afc0 0x7ffb54151aa0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c4b0 tx=0x7ffb5000c970 comp rx=0 tx=0).stop 2024-03-06T07:28:49.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 shutdown_connections 2024-03-06T07:28:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffb3806b4d0 0x7ffb3806d990 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb540c25c0 0x7ffb540c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb540c20b0 0x7ffb54148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 --2- 172.21.15.94:0/326665672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb5406afc0 0x7ffb54151aa0 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 >> 172.21.15.94:0/326665672 conn(0x7ffb540c76b0 msgr2=0x7ffb5406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 shutdown_connections 2024-03-06T07:28:49.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:49.536+0000 7ffb5bd13640 1 -- 172.21.15.94:0/326665672 wait complete. 2024-03-06T07:28:49.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:49.951 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:50.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:51.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.062+0000 7f65094af640 1 Processor -- start 2024-03-06T07:28:51.064 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.062+0000 7f65094af640 1 -- start start 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f65094af640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f65040c5330 0x7f65040c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f65094af640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6504155110 con 0x7f6504158db0 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f65094af640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65040c8f40 con 0x7f65040c88b0 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f65094af640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65041559e0 con 0x7f65040c5330 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f6502ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f65040c5330 0x7f65040c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f65037fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6504158db0 0x7f650415fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55548/0 (socket says 172.21.15.94:55548) 2024-03-06T07:28:51.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f65037fe640 1 -- 172.21.15.94:0/3748309521 learned_addr learned my addr 172.21.15.94:0/3748309521 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:51.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f6502ffd640 1 -- 172.21.15.94:0/3748309521 >> v1:172.21.15.145:6789/0 conn(0x7f65040c88b0 legacy=0x7f65041586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f6502ffd640 1 -- 172.21.15.94:0/3748309521 >> v1:172.21.15.94:6789/0 conn(0x7f6504158db0 legacy=0x7f650415fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.066 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.063+0000 7f6502ffd640 1 -- 172.21.15.94:0/3748309521 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504155390 con 0x7f65040c5330 2024-03-06T07:28:51.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.064+0000 7f6502ffd640 1 --2- 172.21.15.94:0/3748309521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65040c5330 0x7f65040c5710 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f64f40097a0 tx=0x7f64f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e312ec25d2532e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.064+0000 7f65027fc640 1 -- 172.21.15.94:0/3748309521 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4035020 con 0x7f65040c5330 2024-03-06T07:28:51.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.064+0000 7f65027fc640 1 -- 172.21.15.94:0/3748309521 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f4031830 con 0x7f65040c5330 2024-03-06T07:28:51.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.064+0000 7f65027fc640 1 -- 172.21.15.94:0/3748309521 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4031c00 con 0x7f65040c5330 2024-03-06T07:28:51.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.065+0000 7f65094af640 1 -- 172.21.15.94:0/3748309521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65040c5330 msgr2=0x7f65040c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.065+0000 7f65094af640 1 --2- 172.21.15.94:0/3748309521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65040c5330 0x7f65040c5710 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f64f40097a0 tx=0x7f64f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.067 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.065+0000 7f65094af640 1 -- 172.21.15.94:0/3748309521 shutdown_connections 2024-03-06T07:28:51.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.065+0000 7f65094af640 1 --2- 172.21.15.94:0/3748309521 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f65040c5330 0x7f65040c5710 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.065+0000 7f65094af640 1 -- 172.21.15.94:0/3748309521 >> 172.21.15.94:0/3748309521 conn(0x7f65040ac560 msgr2=0x7f65040ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.065+0000 7f65094af640 1 -- 172.21.15.94:0/3748309521 shutdown_connections 2024-03-06T07:28:51.068 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.065+0000 7f65094af640 1 -- 172.21.15.94:0/3748309521 wait complete. 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.066+0000 7f65094af640 1 Processor -- start 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.066+0000 7f65094af640 1 -- start start 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65040c5330 0x7f65041514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65040c88b0 0x7f65041eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6504158420 0x7f65041eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.070 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65041529d0 con 0x7f6504158420 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650406d090 con 0x7f65040c88b0 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6504152b50 con 0x7f65040c5330 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6504158420 0x7f65041eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6504158420 0x7f65041eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54730/0 (socket says 172.21.15.94:54730) 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 -- 172.21.15.94:0/4274044623 learned_addr learned my addr 172.21.15.94:0/4274044623 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f6502ffd640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65040c5330 0x7f65041514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f64fbfff640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65040c88b0 0x7f65041eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 -- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65040c5330 msgr2=0x7f65041514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65040c5330 0x7f65041514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 -- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65040c88b0 msgr2=0x7f65041eacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65040c88b0 0x7f65041eacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 -- 172.21.15.94:0/4274044623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6504151d50 con 0x7f6504158420 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65037fe640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6504158420 0x7f65041eb210 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00cad0 tx=0x7f64fc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.071 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f64fb7fe640 1 -- 172.21.15.94:0/4274044623 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc017070 con 0x7f6504158420 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f64fb7fe640 1 -- 172.21.15.94:0/4274044623 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64fc00ac30 con 0x7f6504158420 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650414be40 con 0x7f6504158420 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f64fb7fe640 1 -- 172.21.15.94:0/4274044623 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc012400 con 0x7f6504158420 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.067+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65041ee520 con 0x7f6504158420 2024-03-06T07:28:51.072 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.069+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6504075300 con 0x7f6504158420 2024-03-06T07:28:51.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.069+0000 7f64fb7fe640 1 -- 172.21.15.94:0/4274044623 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64fc01a070 con 0x7f6504158420 2024-03-06T07:28:51.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.069+0000 7f64fb7fe640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f64e406b4d0 0x7f64e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.069+0000 7f64fb7fe640 1 -- 172.21.15.94:0/4274044623 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f64fc089f70 con 0x7f6504158420 2024-03-06T07:28:51.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.070+0000 7f6502ffd640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f64e406b4d0 0x7f64e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.073 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.070+0000 7f6502ffd640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f64e406b4d0 0x7f64e406d990 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f64f4036000 tx=0x7f64f4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.075 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.074+0000 7f64fb7fe640 1 -- 172.21.15.94:0/4274044623 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6504075300 con 0x7f6504158420 2024-03-06T07:28:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.280+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f650404f330 con 0x7f6504158420 2024-03-06T07:28:51.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.281+0000 7f64fb7fe640 1 -- 172.21.15.94:0/4274044623 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f6504155110 con 0x7f6504158420 2024-03-06T07:28:51.283 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:51.283 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:51.283 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f64e406b4d0 msgr2=0x7f64e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f64e406b4d0 0x7f64e406d990 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f64f4036000 tx=0x7f64f4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6504158420 msgr2=0x7f65041eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6504158420 0x7f65041eb210 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00cad0 tx=0x7f64fc00cf90 comp rx=0 tx=0).stop 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 shutdown_connections 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f64e406b4d0 0x7f64e406d990 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6504158420 0x7f65041eb210 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65040c88b0 0x7f65041eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 --2- 172.21.15.94:0/4274044623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65040c5330 0x7f65041514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 >> 172.21.15.94:0/4274044623 conn(0x7f65040ac560 msgr2=0x7f65041549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 shutdown_connections 2024-03-06T07:28:51.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:51.283+0000 7f65094af640 1 -- 172.21.15.94:0/4274044623 wait complete. 2024-03-06T07:28:51.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:51.699 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:52.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:52.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.809+0000 7f76ad6d6640 1 Processor -- start 2024-03-06T07:28:52.811 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.809+0000 7f76ad6d6640 1 -- start start 2024-03-06T07:28:52.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76ad6d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a8158db0 0x7f76a80c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76ad6d6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f76a80c88b0 0x7f76a815fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.812 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76ad6d6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a8155110 con 0x7f76a80c5330 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76ad6d6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a81559e0 con 0x7f76a80c88b0 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76ad6d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a80c8f40 con 0x7f76a8158db0 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f76a80c5330 0x7f76a81586a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55550/0 (socket says 172.21.15.94:55550) 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a6ffd640 1 -- 172.21.15.94:0/1884755332 learned_addr learned my addr 172.21.15.94:0/1884755332 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a67fc640 1 --2- 172.21.15.94:0/1884755332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a8158db0 0x7f76a80c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a77fe640 1 --2- 172.21.15.94:0/1884755332 >> v2:172.21.15.145:3300/0 conn(0x7f76a80c88b0 0x7f76a815fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a5ffb640 1 -- 172.21.15.94:0/1884755332 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322983984 0 0) 0x7f7698001110 con 0x7f76a80c5330 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a5ffb640 1 -- 172.21.15.94:0/1884755332 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7690003520 con 0x7f76a80c5330 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a67fc640 1 -- 172.21.15.94:0/1884755332 >> v2:172.21.15.145:3300/0 conn(0x7f76a80c88b0 msgr2=0x7f76a815fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a67fc640 1 --2- 172.21.15.94:0/1884755332 >> v2:172.21.15.145:3300/0 conn(0x7f76a80c88b0 0x7f76a815fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a67fc640 1 -- 172.21.15.94:0/1884755332 >> v1:172.21.15.94:6789/0 conn(0x7f76a80c5330 legacy=0x7f76a81586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.810+0000 7f76a67fc640 1 -- 172.21.15.94:0/1884755332 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8155390 con 0x7f76a8158db0 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.811+0000 7f76a67fc640 1 --2- 172.21.15.94:0/1884755332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a8158db0 0x7f76a80c5710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f769c0097a0 tx=0x7f769c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6abf4238ba62973 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.811+0000 7f76a5ffb640 1 -- 172.21.15.94:0/1884755332 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c0315a0 con 0x7f76a8158db0 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.811+0000 7f76a5ffb640 1 -- 172.21.15.94:0/1884755332 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769c037070 con 0x7f76a8158db0 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.811+0000 7f76a5ffb640 1 -- 172.21.15.94:0/1884755332 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c031940 con 0x7f76a8158db0 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 -- 172.21.15.94:0/1884755332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a8158db0 msgr2=0x7f76a80c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/1884755332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a8158db0 0x7f76a80c5710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f769c0097a0 tx=0x7f769c02b840 comp rx=0 tx=0).stop 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 -- 172.21.15.94:0/1884755332 shutdown_connections 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/1884755332 >> v2:172.21.15.145:3300/0 conn(0x7f76a80c88b0 0x7f76a815fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/1884755332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a8158db0 0x7f76a80c5710 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.815 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 -- 172.21.15.94:0/1884755332 >> 172.21.15.94:0/1884755332 conn(0x7f76a80ac560 msgr2=0x7f76a80ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 -- 172.21.15.94:0/1884755332 shutdown_connections 2024-03-06T07:28:52.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.812+0000 7f76ad6d6640 1 -- 172.21.15.94:0/1884755332 wait complete. 2024-03-06T07:28:52.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.813+0000 7f76ad6d6640 1 Processor -- start 2024-03-06T07:28:52.816 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.813+0000 7f76ad6d6640 1 -- start start 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76ad6d6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a80c5330 0x7f76a81514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76ad6d6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76a80c88b0 0x7f76a8157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76ad6d6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76a814ea60 0x7f76a8157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76ad6d6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a81529d0 con 0x7f76a814ea60 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76ad6d6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a806d090 con 0x7f76a80c88b0 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76ad6d6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a8152b50 con 0x7f76a80c5330 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76a814ea60 0x7f76a8157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76a814ea60 0x7f76a8157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54734/0 (socket says 172.21.15.94:54734) 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 -- 172.21.15.94:0/2874650817 learned_addr learned my addr 172.21.15.94:0/2874650817 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a6ffd640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a80c5330 0x7f76a81514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.818 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a67fc640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76a80c88b0 0x7f76a8157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 -- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a80c5330 msgr2=0x7f76a81514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a80c5330 0x7f76a81514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 -- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76a80c88b0 msgr2=0x7f76a8157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76a80c88b0 0x7f76a8157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.814+0000 7f76a77fe640 1 -- 172.21.15.94:0/2874650817 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a814ce10 con 0x7f76a814ea60 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.815+0000 7f76a77fe640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76a814ea60 0x7f76a8157cb0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f76a000efc0 tx=0x7f76a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.815+0000 7f768f7fe640 1 -- 172.21.15.94:0/2874650817 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a0018020 con 0x7f76a814ea60 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.815+0000 7f768f7fe640 1 -- 172.21.15.94:0/2874650817 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a0014480 con 0x7f76a814ea60 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.815+0000 7f768f7fe640 1 -- 172.21.15.94:0/2874650817 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a0014870 con 0x7f76a814ea60 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.815+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a81eb110 con 0x7f76a814ea60 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.815+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a814d0b0 con 0x7f76a814ea60 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.816+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a804f330 con 0x7f76a814ea60 2024-03-06T07:28:52.819 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.816+0000 7f768f7fe640 1 -- 172.21.15.94:0/2874650817 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76a001b020 con 0x7f76a814ea60 2024-03-06T07:28:52.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.817+0000 7f768f7fe640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f769406b4d0 0x7f769406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.817+0000 7f76a6ffd640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f769406b4d0 0x7f769406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.817+0000 7f768f7fe640 1 -- 172.21.15.94:0/2874650817 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f76a008ae00 con 0x7f76a814ea60 2024-03-06T07:28:52.820 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.817+0000 7f76a6ffd640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f769406b4d0 0x7f769406d990 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7698005880 tx=0x7f7698009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.823 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:52.821+0000 7f768f7fe640 1 -- 172.21.15.94:0/2874650817 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76a0022030 con 0x7f76a814ea60 2024-03-06T07:28:53.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.027+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76a806cdc0 con 0x7f76a814ea60 2024-03-06T07:28:53.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.027+0000 7f768f7fe640 1 -- 172.21.15.94:0/2874650817 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f76a0010040 con 0x7f76a814ea60 2024-03-06T07:28:53.029 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:53.029 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:53.029 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:53.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f769406b4d0 msgr2=0x7f769406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f769406b4d0 0x7f769406d990 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7698005880 tx=0x7f7698009040 comp rx=0 tx=0).stop 2024-03-06T07:28:53.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76a814ea60 msgr2=0x7f76a8157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76a814ea60 0x7f76a8157cb0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f76a000efc0 tx=0x7f76a000c490 comp rx=0 tx=0).stop 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 shutdown_connections 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f769406b4d0 0x7f769406d990 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f76a814ea60 0x7f76a8157cb0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76a80c88b0 0x7f76a8157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 --2- 172.21.15.94:0/2874650817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76a80c5330 0x7f76a81514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 >> 172.21.15.94:0/2874650817 conn(0x7f76a80ac560 msgr2=0x7f76a80c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 shutdown_connections 2024-03-06T07:28:53.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:53.029+0000 7f76ad6d6640 1 -- 172.21.15.94:0/2874650817 wait complete. 2024-03-06T07:28:53.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:53.445 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:54.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:54.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.555+0000 7fa97fe8f640 1 Processor -- start 2024-03-06T07:28:54.557 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.555+0000 7fa97fe8f640 1 -- start start 2024-03-06T07:28:54.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97fe8f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa97806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.558 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97fe8f640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9780c1320 0x7fa97806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97fe8f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa978153a60 con 0x7fa9780c6000 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97fe8f640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa978153d40 con 0x7fa9780c1320 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97fe8f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97806c2c0 con 0x7fa978150580 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97d42a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa9780c1320 0x7fa97806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97e42c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa9780c6000 0x7fa9781566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55554/0 (socket says 172.21.15.94:55554) 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97e42c640 1 -- 172.21.15.94:0/2960486489 learned_addr learned my addr 172.21.15.94:0/2960486489 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.556+0000 7fa97dc2b640 1 --2- 172.21.15.94:0/2960486489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa97806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.557+0000 7fa97d42a640 1 -- 172.21.15.94:0/2960486489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 msgr2=0x7fa97806dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.557+0000 7fa97d42a640 1 --2- 172.21.15.94:0/2960486489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa97806dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.557+0000 7fa97d42a640 1 -- 172.21.15.94:0/2960486489 >> v1:172.21.15.94:6789/0 conn(0x7fa9780c6000 legacy=0x7fa9781566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.557+0000 7fa97cc29640 1 -- 172.21.15.94:0/2960486489 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055041991 0 0) 0x7fa978153a60 con 0x7fa9780c6000 2024-03-06T07:28:54.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.557+0000 7fa97d42a640 1 -- 172.21.15.94:0/2960486489 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97806c5a0 con 0x7fa9780c1320 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.557+0000 7fa97d42a640 1 --2- 172.21.15.94:0/2960486489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa97806e250 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fa9680097a0 tx=0x7fa96802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c8cbcbc254a3f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.557+0000 7fa97cc29640 1 -- 172.21.15.94:0/2960486489 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680315a0 con 0x7fa9780c1320 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97cc29640 1 -- 172.21.15.94:0/2960486489 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa968037070 con 0x7fa9780c1320 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97cc29640 1 -- 172.21.15.94:0/2960486489 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968031960 con 0x7fa9780c1320 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2960486489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 msgr2=0x7fa97806e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2960486489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa97806e250 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fa9680097a0 tx=0x7fa96802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2960486489 shutdown_connections 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2960486489 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa97806e250 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2960486489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa97806dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2960486489 >> 172.21.15.94:0/2960486489 conn(0x7fa9780c6710 msgr2=0x7fa9780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2960486489 shutdown_connections 2024-03-06T07:28:54.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.558+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2960486489 wait complete. 2024-03-06T07:28:54.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.559+0000 7fa97fe8f640 1 Processor -- start 2024-03-06T07:28:54.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.559+0000 7fa97fe8f640 1 -- start start 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97fe8f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa9780a8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97fe8f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9780c6000 0x7fa9781ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97fe8f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa978154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97fe8f640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780c0dc0 con 0x7fa9780c1320 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97fe8f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9780c0c40 con 0x7fa9780c6000 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97fe8f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa978156dc0 con 0x7fa978150580 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa9780a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa9780a8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54756/0 (socket says 172.21.15.94:54756) 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 -- 172.21.15.94:0/2379565912 learned_addr learned my addr 172.21.15.94:0/2379565912 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:54.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97d42a640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9780c6000 0x7fa9781ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97e42c640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa978154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 -- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 msgr2=0x7fa978154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa978154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 -- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9780c6000 msgr2=0x7fa9781ec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9780c6000 0x7fa9781ec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.560+0000 7fa97dc2b640 1 -- 172.21.15.94:0/2379565912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9781ec950 con 0x7fa9780c1320 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.561+0000 7fa97dc2b640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa9780a8400 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fa97400e870 tx=0x7fa97400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.561+0000 7fa96e7fc640 1 -- 172.21.15.94:0/2379565912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974017070 con 0x7fa9780c1320 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.561+0000 7fa96e7fc640 1 -- 172.21.15.94:0/2379565912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97400c940 con 0x7fa9780c1320 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.561+0000 7fa96e7fc640 1 -- 172.21.15.94:0/2379565912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97400cd30 con 0x7fa9780c1320 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.561+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9781ebc50 con 0x7fa9780c1320 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.561+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9780a8fe0 con 0x7fa9780c1320 2024-03-06T07:28:54.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.562+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa978069460 con 0x7fa9780c1320 2024-03-06T07:28:54.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.562+0000 7fa96e7fc640 1 -- 172.21.15.94:0/2379565912 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa97401a070 con 0x7fa9780c1320 2024-03-06T07:28:54.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.563+0000 7fa96e7fc640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa95c06b500 0x7fa95c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.563+0000 7fa96e7fc640 1 -- 172.21.15.94:0/2379565912 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa97408a1c0 con 0x7fa9780c1320 2024-03-06T07:28:54.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.563+0000 7fa97d42a640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa95c06b500 0x7fa95c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.563+0000 7fa97d42a640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa95c06b500 0x7fa95c06d9c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa968036000 tx=0x7fa968008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.567+0000 7fa96e7fc640 1 -- 172.21.15.94:0/2379565912 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa978069460 con 0x7fa9780c1320 2024-03-06T07:28:54.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.775+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9780a9220 con 0x7fa9780c1320 2024-03-06T07:28:54.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.776+0000 7fa96e7fc640 1 -- 172.21.15.94:0/2379565912 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7fa9740587f0 con 0x7fa9780c1320 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:54.778 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:54.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa95c06b500 msgr2=0x7fa95c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa95c06b500 0x7fa95c06d9c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa968036000 tx=0x7fa968008030 comp rx=0 tx=0).stop 2024-03-06T07:28:54.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 msgr2=0x7fa9780a8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa9780a8400 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fa97400e870 tx=0x7fa97400ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:54.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 shutdown_connections 2024-03-06T07:28:54.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa95c06b500 0x7fa95c06d9c0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa978150580 0x7fa978154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9780c6000 0x7fa9781ec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 --2- 172.21.15.94:0/2379565912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa9780c1320 0x7fa9780a8400 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 >> 172.21.15.94:0/2379565912 conn(0x7fa9780c6710 msgr2=0x7fa9780c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 shutdown_connections 2024-03-06T07:28:54.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:54.778+0000 7fa97fe8f640 1 -- 172.21.15.94:0/2379565912 wait complete. 2024-03-06T07:28:55.192 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:55.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:56.194 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:56.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.304+0000 7f09e8e40640 1 Processor -- start 2024-03-06T07:28:56.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.305+0000 7f09e8e40640 1 -- start start 2024-03-06T07:28:56.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.305+0000 7f09e8e40640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09e4159270 0x7f09e4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.305+0000 7f09e8e40640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e40c73d0 con 0x7f09e414f360 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.305+0000 7f09e8e40640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e40be2e0 con 0x7f09e40c88c0 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.305+0000 7f09e8e40640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e4159700 con 0x7f09e4159270 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.305+0000 7f09e2d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f09e4159270 0x7f09e4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.305+0000 7f09e259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f09e414f360 0x7f09e40c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55564/0 (socket says 172.21.15.94:55564) 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.306+0000 7f09e259c640 1 -- 172.21.15.94:0/3787341416 learned_addr learned my addr 172.21.15.94:0/3787341416 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.306+0000 7f09e2d9d640 1 -- 172.21.15.94:0/3787341416 >> v1:172.21.15.145:6789/0 conn(0x7f09e40c88c0 legacy=0x7f09e4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.306+0000 7f09e2d9d640 1 -- 172.21.15.94:0/3787341416 >> v1:172.21.15.94:6789/0 conn(0x7f09e414f360 legacy=0x7f09e40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.306+0000 7f09e159a640 1 -- 172.21.15.94:0/3787341416 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647790218 0 0) 0x7f09e40be2e0 con 0x7f09e40c88c0 2024-03-06T07:28:56.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.306+0000 7f09e2d9d640 1 -- 172.21.15.94:0/3787341416 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e40c7670 con 0x7f09e4159270 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e2d9d640 1 --2- 172.21.15.94:0/3787341416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4155c10 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009d90 tx=0x7f09dc02bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de90921fd0941209 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e159a640 1 -- 172.21.15.94:0/3787341416 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc039070 con 0x7f09e4159270 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e159a640 1 -- 172.21.15.94:0/3787341416 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc02ca20 con 0x7f09e4159270 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e159a640 1 -- 172.21.15.94:0/3787341416 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc02cdf0 con 0x7f09e4159270 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e8e40640 1 -- 172.21.15.94:0/3787341416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 msgr2=0x7f09e4155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e8e40640 1 --2- 172.21.15.94:0/3787341416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4155c10 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009d90 tx=0x7f09dc02bcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e8e40640 1 -- 172.21.15.94:0/3787341416 shutdown_connections 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e8e40640 1 --2- 172.21.15.94:0/3787341416 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4155c10 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e8e40640 1 -- 172.21.15.94:0/3787341416 >> 172.21.15.94:0/3787341416 conn(0x7f09e40ac570 msgr2=0x7f09e40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e8e40640 1 -- 172.21.15.94:0/3787341416 shutdown_connections 2024-03-06T07:28:56.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.307+0000 7f09e8e40640 1 -- 172.21.15.94:0/3787341416 wait complete. 2024-03-06T07:28:56.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 Processor -- start 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 -- start start 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09e40c88c0 0x7f09e40c0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09e414f360 0x7f09e40be760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40c7c30 con 0x7f09e4159270 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40c5a50 con 0x7f09e40c88c0 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e8e40640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40c5bd0 con 0x7f09e414f360 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e2d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54786/0 (socket says 172.21.15.94:54786) 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.309+0000 7f09e2d9d640 1 -- 172.21.15.94:0/1213743886 learned_addr learned my addr 172.21.15.94:0/1213743886 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e1d9b640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09e414f360 0x7f09e40be760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e259c640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09e40c88c0 0x7f09e40c0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e2d9d640 1 -- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09e414f360 msgr2=0x7f09e40be760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e2d9d640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09e414f360 0x7f09e40be760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e2d9d640 1 -- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09e40c88c0 msgr2=0x7f09e40c0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e2d9d640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09e40c88c0 0x7f09e40c0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e2d9d640 1 -- 172.21.15.94:0/1213743886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e41eb5f0 con 0x7f09e4159270 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e2d9d640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4153a40 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009d90 tx=0x7f09dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09daffd640 1 -- 172.21.15.94:0/1213743886 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc044020 con 0x7f09e4159270 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09daffd640 1 -- 172.21.15.94:0/1213743886 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc034c10 con 0x7f09e4159270 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e40bf380 con 0x7f09e4159270 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09daffd640 1 -- 172.21.15.94:0/1213743886 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc039040 con 0x7f09e4159270 2024-03-06T07:28:56.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.310+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e40ac000 con 0x7f09e4159270 2024-03-06T07:28:56.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.311+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e404f330 con 0x7f09e4159270 2024-03-06T07:28:56.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.312+0000 7f09daffd640 1 -- 172.21.15.94:0/1213743886 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09dc047020 con 0x7f09e4159270 2024-03-06T07:28:56.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.312+0000 7f09daffd640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f09c406b4d0 0x7f09c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.312+0000 7f09daffd640 1 -- 172.21.15.94:0/1213743886 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f09dc0acb60 con 0x7f09e4159270 2024-03-06T07:28:56.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.312+0000 7f09e259c640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f09c406b4d0 0x7f09c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.316 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.312+0000 7f09e259c640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f09c406b4d0 0x7f09c406d990 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f09cc000f60 tx=0x7f09cc007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.316+0000 7f09daffd640 1 -- 172.21.15.94:0/1213743886 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09dc04e030 con 0x7f09e4159270 2024-03-06T07:28:56.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.524+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09e40c9ed0 con 0x7f09e4159270 2024-03-06T07:28:56.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.524+0000 7f09daffd640 1 -- 172.21.15.94:0/1213743886 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f09dc07b190 con 0x7f09e4159270 2024-03-06T07:28:56.526 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:56.526 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:56.527 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:56.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f09c406b4d0 msgr2=0x7f09c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f09c406b4d0 0x7f09c406d990 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f09cc000f60 tx=0x7f09cc007e20 comp rx=0 tx=0).stop 2024-03-06T07:28:56.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 msgr2=0x7f09e4153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4153a40 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009d90 tx=0x7f09dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 shutdown_connections 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f09c406b4d0 0x7f09c406d990 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f09e4159270 0x7f09e4153a40 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09e414f360 0x7f09e40be760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 --2- 172.21.15.94:0/1213743886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09e40c88c0 0x7f09e40c0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 >> 172.21.15.94:0/1213743886 conn(0x7f09e40ac570 msgr2=0x7f09e40ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 shutdown_connections 2024-03-06T07:28:56.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:56.526+0000 7f09e8e40640 1 -- 172.21.15.94:0/1213743886 wait complete. 2024-03-06T07:28:56.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:56.940 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:57.941 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:58.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a188df640 1 Processor -- start 2024-03-06T07:28:58.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a188df640 1 -- start start 2024-03-06T07:28:58.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a188df640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1a1414f360 0x7f1a14155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a188df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a140c73d0 con 0x7f1a14158b60 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a188df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a140be2e0 con 0x7f1a140c88c0 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a188df640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a14159700 con 0x7f1a1414f360 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a1259c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1a1414f360 0x7f1a14155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a12d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1a14158b60 0x7f1a14158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55568/0 (socket says 172.21.15.94:55568) 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.052+0000 7f1a12d9d640 1 -- 172.21.15.94:0/4097950910 learned_addr learned my addr 172.21.15.94:0/4097950910 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.053+0000 7f1a1259c640 1 -- 172.21.15.94:0/4097950910 >> v1:172.21.15.145:6789/0 conn(0x7f1a140c88c0 legacy=0x7f1a140c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.053+0000 7f1a1259c640 1 -- 172.21.15.94:0/4097950910 >> v1:172.21.15.94:6789/0 conn(0x7f1a14158b60 legacy=0x7f1a14158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.053+0000 7f1a1259c640 1 -- 172.21.15.94:0/4097950910 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a140c7670 con 0x7f1a1414f360 2024-03-06T07:28:58.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.053+0000 7f1a1259c640 1 --2- 172.21.15.94:0/4097950910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a14155c10 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f1a040097a0 tx=0x7f1a0402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e773f22c3c744444 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.056 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a11d9b640 1 -- 172.21.15.94:0/4097950910 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04035020 con 0x7f1a1414f360 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a11d9b640 1 -- 172.21.15.94:0/4097950910 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04031830 con 0x7f1a1414f360 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a11d9b640 1 -- 172.21.15.94:0/4097950910 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04031c00 con 0x7f1a1414f360 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a188df640 1 -- 172.21.15.94:0/4097950910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 msgr2=0x7f1a14155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a188df640 1 --2- 172.21.15.94:0/4097950910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a14155c10 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f1a040097a0 tx=0x7f1a0402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a188df640 1 -- 172.21.15.94:0/4097950910 shutdown_connections 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a188df640 1 --2- 172.21.15.94:0/4097950910 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a14155c10 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a188df640 1 -- 172.21.15.94:0/4097950910 >> 172.21.15.94:0/4097950910 conn(0x7f1a140ac570 msgr2=0x7f1a140ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a188df640 1 -- 172.21.15.94:0/4097950910 shutdown_connections 2024-03-06T07:28:58.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.054+0000 7f1a188df640 1 -- 172.21.15.94:0/4097950910 wait complete. 2024-03-06T07:28:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 Processor -- start 2024-03-06T07:28:58.059 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 -- start start 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a140c88c0 0x7f1a140c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a141ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a14158b60 0x7f1a140aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140c7c30 con 0x7f1a1414f360 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140c5a50 con 0x7f1a14158b60 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a188df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a140c5bd0 con 0x7f1a140c88c0 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a0bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a141ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a0bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a141ee130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:54798/0 (socket says 172.21.15.94:54798) 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a1259c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a140c88c0 0x7f1a140c0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a0bfff640 1 -- 172.21.15.94:0/1385421239 learned_addr learned my addr 172.21.15.94:0/1385421239 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.056+0000 7f1a12d9d640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a14158b60 0x7f1a140aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0bfff640 1 -- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a140c88c0 msgr2=0x7f1a140c0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.060 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0bfff640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a140c88c0 0x7f1a140c0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0bfff640 1 -- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a14158b60 msgr2=0x7f1a140aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0bfff640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a14158b60 0x7f1a140aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0bfff640 1 -- 172.21.15.94:0/1385421239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a141ee670 con 0x7f1a1414f360 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0bfff640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a141ee130 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00c430 tx=0x7f19fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0affd640 1 -- 172.21.15.94:0/1385421239 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc017070 con 0x7f1a1414f360 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0affd640 1 -- 172.21.15.94:0/1385421239 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc0052d0 con 0x7f1a1414f360 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a0affd640 1 -- 172.21.15.94:0/1385421239 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc003830 con 0x7f1a1414f360 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a14157ec0 con 0x7f1a1414f360 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.057+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a14157be0 con 0x7f1a1414f360 2024-03-06T07:28:58.061 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.058+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1404f330 con 0x7f1a1414f360 2024-03-06T07:28:58.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.058+0000 7f1a0affd640 1 -- 172.21.15.94:0/1385421239 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19fc01a070 con 0x7f1a1414f360 2024-03-06T07:28:58.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.059+0000 7f1a0affd640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:58.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.059+0000 7f1a0affd640 1 -- 172.21.15.94:0/1385421239 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f19fc089750 con 0x7f1a1414f360 2024-03-06T07:28:58.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.059+0000 7f1a1259c640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:58.062 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.059+0000 7f1a1259c640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f19f406b4d0 0x7f19f406d990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1a04036000 tx=0x7f1a04008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:58.065 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.063+0000 7f1a0affd640 1 -- 172.21.15.94:0/1385421239 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19fc057d80 con 0x7f1a1414f360 2024-03-06T07:28:58.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.271+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a140c8d50 con 0x7f1a1414f360 2024-03-06T07:28:58.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.271+0000 7f1a0affd640 1 -- 172.21.15.94:0/1385421239 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f19fc05e0d0 con 0x7f1a1414f360 2024-03-06T07:28:58.273 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:28:58.273 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:28:58.273 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:28:58.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f19f406b4d0 msgr2=0x7f19f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f19f406b4d0 0x7f19f406d990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1a04036000 tx=0x7f1a04008030 comp rx=0 tx=0).stop 2024-03-06T07:28:58.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 msgr2=0x7f1a141ee130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a141ee130 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00c430 tx=0x7f19fc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 shutdown_connections 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f19f406b4d0 0x7f19f406d990 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1a14158b60 0x7f1a140aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a1414f360 0x7f1a141ee130 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 --2- 172.21.15.94:0/1385421239 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1a140c88c0 0x7f1a140c0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 >> 172.21.15.94:0/1385421239 conn(0x7f1a140ac570 msgr2=0x7f1a140ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 shutdown_connections 2024-03-06T07:28:58.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:58.273+0000 7f1a188df640 1 -- 172.21.15.94:0/1385421239 wait complete. 2024-03-06T07:28:58.688 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:28:58.688 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:28:59.689 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:59.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.799+0000 7fa894902640 1 Processor -- start 2024-03-06T07:28:59.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.799+0000 7fa894902640 1 -- start start 2024-03-06T07:28:59.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa894902640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa8900c88c0 0x7fa890155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa894902640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa890158b60 0x7fa890158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa894902640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900c73d0 con 0x7fa89014f360 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa894902640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8900c8dd0 con 0x7fa890158b60 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa894902640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8900be2e0 con 0x7fa8900c88c0 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa88ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa890158b60 0x7fa890158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa88e59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa89014f360 0x7fa8900c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52858/0 (socket says 172.21.15.94:52858) 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.800+0000 7fa88e59c640 1 -- 172.21.15.94:0/3686423135 learned_addr learned my addr 172.21.15.94:0/3686423135 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.801+0000 7fa88dd9b640 1 --2- 172.21.15.94:0/3686423135 >> v2:172.21.15.145:3300/0 conn(0x7fa8900c88c0 0x7fa890155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.801+0000 7fa88ed9d640 1 -- 172.21.15.94:0/3686423135 >> v2:172.21.15.145:3300/0 conn(0x7fa8900c88c0 msgr2=0x7fa890155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.801+0000 7fa88ed9d640 1 --2- 172.21.15.94:0/3686423135 >> v2:172.21.15.145:3300/0 conn(0x7fa8900c88c0 0x7fa890155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.801+0000 7fa88ed9d640 1 -- 172.21.15.94:0/3686423135 >> v1:172.21.15.94:6789/0 conn(0x7fa89014f360 legacy=0x7fa8900c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.801+0000 7fa88ed9d640 1 -- 172.21.15.94:0/3686423135 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8900c7670 con 0x7fa890158b60 2024-03-06T07:28:59.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.801+0000 7fa88d59a640 1 -- 172.21.15.94:0/3686423135 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51056645 0 0) 0x7fa8900c73d0 con 0x7fa89014f360 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.801+0000 7fa88ed9d640 1 --2- 172.21.15.94:0/3686423135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa890158b60 0x7fa890158ff0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7fa8900c84c0 tx=0x7fa88802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0cd6336d551378 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa88d59a640 1 -- 172.21.15.94:0/3686423135 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8880315a0 con 0x7fa890158b60 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa88d59a640 1 -- 172.21.15.94:0/3686423135 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa888037070 con 0x7fa890158b60 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa88d59a640 1 -- 172.21.15.94:0/3686423135 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888031960 con 0x7fa890158b60 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 -- 172.21.15.94:0/3686423135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa890158b60 msgr2=0x7fa890158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 --2- 172.21.15.94:0/3686423135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa890158b60 0x7fa890158ff0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7fa8900c84c0 tx=0x7fa88802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 -- 172.21.15.94:0/3686423135 shutdown_connections 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 --2- 172.21.15.94:0/3686423135 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa890158b60 0x7fa890158ff0 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 --2- 172.21.15.94:0/3686423135 >> v2:172.21.15.145:3300/0 conn(0x7fa8900c88c0 0x7fa890155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 -- 172.21.15.94:0/3686423135 >> 172.21.15.94:0/3686423135 conn(0x7fa8900ac570 msgr2=0x7fa8900ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 -- 172.21.15.94:0/3686423135 shutdown_connections 2024-03-06T07:28:59.806 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.802+0000 7fa894902640 1 -- 172.21.15.94:0/3686423135 wait complete. 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 Processor -- start 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 -- start start 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8900c88c0 0x7fa8900c0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa89014f360 0x7fa8901ebf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa890158b60 0x7fa8901ec490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8900c5a50 con 0x7fa8900c88c0 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8900c7c30 con 0x7fa890158b60 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa894902640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8900c5bd0 con 0x7fa89014f360 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa88e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8900c88c0 0x7fa8900c0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa88e59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8900c88c0 0x7fa8900c0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36220/0 (socket says 172.21.15.94:36220) 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa88e59c640 1 -- 172.21.15.94:0/863467761 learned_addr learned my addr 172.21.15.94:0/863467761 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:28:59.808 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa88ed9d640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa890158b60 0x7fa8901ec490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.804+0000 7fa88dd9b640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa89014f360 0x7fa8901ebf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa88e59c640 1 -- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa89014f360 msgr2=0x7fa8901ebf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa88e59c640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa89014f360 0x7fa8901ebf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa88e59c640 1 -- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa890158b60 msgr2=0x7fa8901ec490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa88e59c640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa890158b60 0x7fa8901ec490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa88e59c640 1 -- 172.21.15.94:0/863467761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa890158710 con 0x7fa8900c88c0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa88e59c640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8900c88c0 0x7fa8900c0b70 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fa88000c890 tx=0x7fa88000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa87effd640 1 -- 172.21.15.94:0/863467761 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa880018070 con 0x7fa8900c88c0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa87effd640 1 -- 172.21.15.94:0/863467761 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88000a640 con 0x7fa8900c88c0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8900befa0 con 0x7fa8900c88c0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa87effd640 1 -- 172.21.15.94:0/863467761 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88000aa30 con 0x7fa8900c88c0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.805+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8900bf560 con 0x7fa8900c88c0 2024-03-06T07:28:59.809 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.806+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89004f330 con 0x7fa8900c88c0 2024-03-06T07:28:59.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.807+0000 7fa87effd640 1 -- 172.21.15.94:0/863467761 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa88001b070 con 0x7fa8900c88c0 2024-03-06T07:28:59.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.807+0000 7fa87effd640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa87006b4d0 0x7fa87006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.807+0000 7fa87effd640 1 -- 172.21.15.94:0/863467761 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa880089990 con 0x7fa8900c88c0 2024-03-06T07:28:59.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.807+0000 7fa88dd9b640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa87006b4d0 0x7fa87006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.810 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.808+0000 7fa88dd9b640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa87006b4d0 0x7fa87006d990 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa89014c3a0 tx=0x7fa884009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.813 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:28:59.811+0000 7fa87effd640 1 -- 172.21.15.94:0/863467761 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa880057fc0 con 0x7fa8900c88c0 2024-03-06T07:29:00.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.019+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa89006cda0 con 0x7fa8900c88c0 2024-03-06T07:29:00.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.020+0000 7fa87effd640 1 -- 172.21.15.94:0/863467761 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7fa88005c060 con 0x7fa8900c88c0 2024-03-06T07:29:00.022 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:00.022 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:00.022 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa87006b4d0 msgr2=0x7fa87006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa87006b4d0 0x7fa87006d990 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa89014c3a0 tx=0x7fa884009190 comp rx=0 tx=0).stop 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8900c88c0 msgr2=0x7fa8900c0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8900c88c0 0x7fa8900c0b70 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fa88000c890 tx=0x7fa88000cd50 comp rx=0 tx=0).stop 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 shutdown_connections 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa87006b4d0 0x7fa87006d990 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa890158b60 0x7fa8901ec490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa89014f360 0x7fa8901ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 --2- 172.21.15.94:0/863467761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8900c88c0 0x7fa8900c0b70 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 >> 172.21.15.94:0/863467761 conn(0x7fa8900ac570 msgr2=0x7fa8900c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 shutdown_connections 2024-03-06T07:29:00.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:00.022+0000 7fa894902640 1 -- 172.21.15.94:0/863467761 wait complete. 2024-03-06T07:29:00.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:00.437 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:01.438 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.546+0000 7f2ee256c640 1 Processor -- start 2024-03-06T07:29:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.546+0000 7f2ee256c640 1 -- start start 2024-03-06T07:29:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2ee256c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2edc1550c0 0x7f2edc1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2ee256c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2ee256c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc0aa660 con 0x7f2edc0aa180 2024-03-06T07:29:01.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2ee256c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc0c4610 con 0x7f2edc1550c0 2024-03-06T07:29:01.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2ee256c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0aa940 con 0x7f2edc1559e0 2024-03-06T07:29:01.559 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2edb7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2edbfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2edc1550c0 0x7f2edc1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2ee0b09640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2edc0aa180 0x7f2edc0c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49956/0 (socket says 172.21.15.94:49956) 2024-03-06T07:29:01.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2ee0b09640 1 -- 172.21.15.94:0/3925492295 learned_addr learned my addr 172.21.15.94:0/3925492295 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:01.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2edbfff640 1 -- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 msgr2=0x7f2edc14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:01.560 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2edbfff640 1 --2- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc14ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2edbfff640 1 -- 172.21.15.94:0/3925492295 >> v1:172.21.15.145:6789/0 conn(0x7f2edc0aa180 legacy=0x7f2edc0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2edbfff640 1 -- 172.21.15.94:0/3925492295 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0c4860 con 0x7f2edc1550c0 2024-03-06T07:29:01.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.547+0000 7f2edb7fe640 1 --2- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:01.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.548+0000 7f2edbfff640 1 --2- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 0x7f2edc1554a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0097a0 tx=0x7f2ecc02b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b63e296e7d311fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.548+0000 7f2edaffd640 1 -- 172.21.15.94:0/3925492295 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc038070 con 0x7f2edc1550c0 2024-03-06T07:29:01.561 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.548+0000 7f2edaffd640 1 -- 172.21.15.94:0/3925492295 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc031700 con 0x7f2edc1550c0 2024-03-06T07:29:01.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.548+0000 7f2edaffd640 1 -- 172.21.15.94:0/3925492295 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc031ad0 con 0x7f2edc1550c0 2024-03-06T07:29:01.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 -- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 msgr2=0x7f2edc1554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 --2- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 0x7f2edc1554a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0097a0 tx=0x7f2ecc02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 -- 172.21.15.94:0/3925492295 shutdown_connections 2024-03-06T07:29:01.562 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 --2- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 --2- 172.21.15.94:0/3925492295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 0x7f2edc1554a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 -- 172.21.15.94:0/3925492295 >> 172.21.15.94:0/3925492295 conn(0x7f2edc0c6710 msgr2=0x7f2edc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 -- 172.21.15.94:0/3925492295 shutdown_connections 2024-03-06T07:29:01.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.549+0000 7f2ee256c640 1 -- 172.21.15.94:0/3925492295 wait complete. 2024-03-06T07:29:01.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.550+0000 7f2ee256c640 1 Processor -- start 2024-03-06T07:29:01.563 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.550+0000 7f2ee256c640 1 -- start start 2024-03-06T07:29:01.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee256c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2edc0aa180 0x7f2edc0a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee256c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 0x7f2edc1411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee256c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee256c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc14f140 con 0x7f2edc0aa180 2024-03-06T07:29:01.564 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee256c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0c1520 con 0x7f2edc1550c0 2024-03-06T07:29:01.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee256c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc14f2c0 con 0x7f2edc1559e0 2024-03-06T07:29:01.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2edc0aa180 0x7f2edc0a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2edc0aa180 0x7f2edc0a4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36226/0 (socket says 172.21.15.94:36226) 2024-03-06T07:29:01.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 -- 172.21.15.94:0/151997734 learned_addr learned my addr 172.21.15.94:0/151997734 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:01.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edb7fe640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 0x7f2edc1411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee0b09640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 -- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 msgr2=0x7f2edc0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc0702f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 -- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 msgr2=0x7f2edc1411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 0x7f2edc1411d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 -- 172.21.15.94:0/151997734 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0a3eb0 con 0x7f2edc0aa180 2024-03-06T07:29:01.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2edbfff640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2edc0aa180 0x7f2edc0a4c70 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0098c0 tx=0x7f2ecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ed8ff9640 1 -- 172.21.15.94:0/151997734 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc038040 con 0x7f2edc0aa180 2024-03-06T07:29:01.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ed8ff9640 1 -- 172.21.15.94:0/151997734 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc043070 con 0x7f2edc0aa180 2024-03-06T07:29:01.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.551+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc0a4190 con 0x7f2edc0aa180 2024-03-06T07:29:01.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.552+0000 7f2ed8ff9640 1 -- 172.21.15.94:0/151997734 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc03ca70 con 0x7f2edc0aa180 2024-03-06T07:29:01.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.552+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc141710 con 0x7f2edc0aa180 2024-03-06T07:29:01.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.553+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc04f330 con 0x7f2edc0aa180 2024-03-06T07:29:01.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.553+0000 7f2ed8ff9640 1 -- 172.21.15.94:0/151997734 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ecc03b070 con 0x7f2edc0aa180 2024-03-06T07:29:01.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.553+0000 7f2ed8ff9640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ec806b4d0 0x7f2ec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.553+0000 7f2ed8ff9640 1 -- 172.21.15.94:0/151997734 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2ecc0ab750 con 0x7f2edc0aa180 2024-03-06T07:29:01.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.553+0000 7f2edb7fe640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ec806b4d0 0x7f2ec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.554+0000 7f2edb7fe640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ec806b4d0 0x7f2ec806d990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009560 tx=0x7f2ed0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.569 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.558+0000 7f2ed8ff9640 1 -- 172.21.15.94:0/151997734 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ecc030070 con 0x7f2edc0aa180 2024-03-06T07:29:01.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.766+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2edc068450 con 0x7f2edc0aa180 2024-03-06T07:29:01.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.767+0000 7f2ed8ff9640 1 -- 172.21.15.94:0/151997734 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f2ecc079e10 con 0x7f2edc0aa180 2024-03-06T07:29:01.769 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:01.769 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:01.769 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ec806b4d0 msgr2=0x7f2ec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ec806b4d0 0x7f2ec806d990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009560 tx=0x7f2ed0009190 comp rx=0 tx=0).stop 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2edc0aa180 msgr2=0x7f2edc0a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2edc0aa180 0x7f2edc0a4c70 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0098c0 tx=0x7f2ecc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 shutdown_connections 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f2ec806b4d0 0x7f2ec806d990 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2edc1559e0 0x7f2edc0702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2edc1550c0 0x7f2edc1411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 --2- 172.21.15.94:0/151997734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2edc0aa180 0x7f2edc0a4c70 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 >> 172.21.15.94:0/151997734 conn(0x7f2edc0c6710 msgr2=0x7f2edc0c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 shutdown_connections 2024-03-06T07:29:01.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:01.769+0000 7f2ee256c640 1 -- 172.21.15.94:0/151997734 wait complete. 2024-03-06T07:29:02.184 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:02.184 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:03.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:03.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.295+0000 7fddfe3f6640 1 Processor -- start 2024-03-06T07:29:03.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddfe3f6640 1 -- start start 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddfe3f6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fddf80c1aa0 0x7fddf81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddfe3f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf80c28d0 con 0x7fddf8155a40 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddfe3f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf80c2bb0 con 0x7fddf814f790 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddfe3f6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf80c2680 con 0x7fddf80c1aa0 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddf77fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fddf80c1aa0 0x7fddf81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddfc993640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fddf8155a40 0x7fddf8155e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52864/0 (socket says 172.21.15.94:52864) 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.296+0000 7fddfc993640 1 -- 172.21.15.94:0/3852251225 learned_addr learned my addr 172.21.15.94:0/3852251225 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.297+0000 7fddf77fe640 1 -- 172.21.15.94:0/3852251225 >> v1:172.21.15.145:6789/0 conn(0x7fddf814f790 legacy=0x7fddf80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.297+0000 7fddf77fe640 1 -- 172.21.15.94:0/3852251225 >> v1:172.21.15.94:6789/0 conn(0x7fddf8155a40 legacy=0x7fddf8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.297+0000 7fddf77fe640 1 -- 172.21.15.94:0/3852251225 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf80c2e90 con 0x7fddf80c1aa0 2024-03-06T07:29:03.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.297+0000 7fddf6ffd640 1 -- 172.21.15.94:0/3852251225 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297420008 0 0) 0x7fddf80c28d0 con 0x7fddf8155a40 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.297+0000 7fddf77fe640 1 --2- 172.21.15.94:0/3852251225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf80c1aa0 0x7fddf81529d0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fddf80c2800 tx=0x7fdde802e110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e964a2c476655bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddf6ffd640 1 -- 172.21.15.94:0/3852251225 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde803b070 con 0x7fddf80c1aa0 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddf6ffd640 1 -- 172.21.15.94:0/3852251225 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde8036440 con 0x7fddf80c1aa0 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddf6ffd640 1 -- 172.21.15.94:0/3852251225 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde8036810 con 0x7fddf80c1aa0 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddfe3f6640 1 -- 172.21.15.94:0/3852251225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf80c1aa0 msgr2=0x7fddf81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/3852251225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf80c1aa0 0x7fddf81529d0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fddf80c2800 tx=0x7fdde802e110 comp rx=0 tx=0).stop 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddfe3f6640 1 -- 172.21.15.94:0/3852251225 shutdown_connections 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/3852251225 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf80c1aa0 0x7fddf81529d0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddfe3f6640 1 -- 172.21.15.94:0/3852251225 >> 172.21.15.94:0/3852251225 conn(0x7fddf80c6710 msgr2=0x7fddf80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddfe3f6640 1 -- 172.21.15.94:0/3852251225 shutdown_connections 2024-03-06T07:29:03.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.298+0000 7fddfe3f6640 1 -- 172.21.15.94:0/3852251225 wait complete. 2024-03-06T07:29:03.302 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.299+0000 7fddfe3f6640 1 Processor -- start 2024-03-06T07:29:03.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfe3f6640 1 -- start start 2024-03-06T07:29:03.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfe3f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fddf80c1aa0 0x7fddf80bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfe3f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fddf814f790 0x7fddf814b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfe3f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf8155a40 0x7fddf814bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfe3f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80c5e30 con 0x7fddf8155a40 2024-03-06T07:29:03.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfe3f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80c0d60 con 0x7fddf814f790 2024-03-06T07:29:03.304 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfe3f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80c5fb0 con 0x7fddf80c1aa0 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfc993640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf8155a40 0x7fddf814bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfc993640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf8155a40 0x7fddf814bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36238/0 (socket says 172.21.15.94:36238) 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddfc993640 1 -- 172.21.15.94:0/4241436053 learned_addr learned my addr 172.21.15.94:0/4241436053 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddf7fff640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fddf80c1aa0 0x7fddf80bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.300+0000 7fddf77fe640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fddf814f790 0x7fddf814b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfc993640 1 -- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fddf80c1aa0 msgr2=0x7fddf80bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfc993640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fddf80c1aa0 0x7fddf80bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfc993640 1 -- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fddf814f790 msgr2=0x7fddf814b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfc993640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fddf814f790 0x7fddf814b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfc993640 1 -- 172.21.15.94:0/4241436053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf81de3d0 con 0x7fddf8155a40 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfc993640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf8155a40 0x7fddf814bee0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c730 tx=0x7fddf000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddf4ff9640 1 -- 172.21.15.94:0/4241436053 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0018070 con 0x7fddf8155a40 2024-03-06T07:29:03.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddf4ff9640 1 -- 172.21.15.94:0/4241436053 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf000ce40 con 0x7fddf8155a40 2024-03-06T07:29:03.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddf4ff9640 1 -- 172.21.15.94:0/4241436053 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf000a860 con 0x7fddf8155a40 2024-03-06T07:29:03.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf81ddaa0 con 0x7fddf8155a40 2024-03-06T07:29:03.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.301+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf81de060 con 0x7fddf8155a40 2024-03-06T07:29:03.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.302+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf804f330 con 0x7fddf8155a40 2024-03-06T07:29:03.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.302+0000 7fddf4ff9640 1 -- 172.21.15.94:0/4241436053 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddf001b070 con 0x7fddf8155a40 2024-03-06T07:29:03.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.303+0000 7fddf4ff9640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.303+0000 7fddf4ff9640 1 -- 172.21.15.94:0/4241436053 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fddf00898b0 con 0x7fddf8155a40 2024-03-06T07:29:03.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.303+0000 7fddf7fff640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.303+0000 7fddf7fff640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fddec000f60 tx=0x7fddec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.307+0000 7fddf4ff9640 1 -- 172.21.15.94:0/4241436053 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddf0057ee0 con 0x7fddf8155a40 2024-03-06T07:29:03.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.515+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddf8150fe0 con 0x7fddf8155a40 2024-03-06T07:29:03.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.516+0000 7fddf4ff9640 1 -- 172.21.15.94:0/4241436053 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7fddf005bf80 con 0x7fddf8155a40 2024-03-06T07:29:03.518 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:03.518 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:03.518 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.517+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 msgr2=0x7fdde406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.517+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fddec000f60 tx=0x7fddec009040 comp rx=0 tx=0).stop 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.517+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf8155a40 msgr2=0x7fddf814bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.517+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf8155a40 0x7fddf814bee0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fddf000c730 tx=0x7fddf000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.517+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 shutdown_connections 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.518+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fdde406b4d0 0x7fdde406d990 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.518+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fddf8155a40 0x7fddf814bee0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.518+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fddf814f790 0x7fddf814b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.518+0000 7fddfe3f6640 1 --2- 172.21.15.94:0/4241436053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fddf80c1aa0 0x7fddf80bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.518+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 >> 172.21.15.94:0/4241436053 conn(0x7fddf80c6710 msgr2=0x7fddf814d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.518+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 shutdown_connections 2024-03-06T07:29:03.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:03.518+0000 7fddfe3f6640 1 -- 172.21.15.94:0/4241436053 wait complete. 2024-03-06T07:29:03.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:03.932 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:04.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:05.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.043+0000 7fd4c8a8d640 1 Processor -- start 2024-03-06T07:29:05.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.043+0000 7fd4c8a8d640 1 -- start start 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.044+0000 7fd4c8a8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c414f360 0x7fd4c4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.044+0000 7fd4c8a8d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40c73d0 con 0x7fd4c4158b60 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.044+0000 7fd4c8a8d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40be2e0 con 0x7fd4c40c88c0 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.044+0000 7fd4c8a8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c4159700 con 0x7fd4c414f360 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.044+0000 7fd4c2d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd4c4158b60 0x7fd4c4158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52878/0 (socket says 172.21.15.94:52878) 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.044+0000 7fd4c2d9d640 1 -- 172.21.15.94:0/1959236558 learned_addr learned my addr 172.21.15.94:0/1959236558 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.044+0000 7fd4c259c640 1 --2- 172.21.15.94:0/1959236558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c414f360 0x7fd4c4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.045+0000 7fd4c159a640 1 -- 172.21.15.94:0/1959236558 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749154992 0 0) 0x7fd4c40c73d0 con 0x7fd4c4158b60 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.045+0000 7fd4c159a640 1 -- 172.21.15.94:0/1959236558 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac003520 con 0x7fd4c4158b60 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.045+0000 7fd4c159a640 1 -- 172.21.15.94:0/1959236558 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625562389 0 0) 0x7fd4c40be2e0 con 0x7fd4c40c88c0 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.045+0000 7fd4c259c640 1 -- 172.21.15.94:0/1959236558 >> v1:172.21.15.145:6789/0 conn(0x7fd4c40c88c0 legacy=0x7fd4c40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.045+0000 7fd4c259c640 1 -- 172.21.15.94:0/1959236558 >> v1:172.21.15.94:6789/0 conn(0x7fd4c4158b60 legacy=0x7fd4c4158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.045+0000 7fd4c259c640 1 -- 172.21.15.94:0/1959236558 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40c7670 con 0x7fd4c414f360 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c259c640 1 --2- 172.21.15.94:0/1959236558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c414f360 0x7fd4c4155c10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009dc0 tx=0x7fd4b402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7ba3a8aa7cd7af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c159a640 1 -- 172.21.15.94:0/1959236558 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4039070 con 0x7fd4c414f360 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c159a640 1 -- 172.21.15.94:0/1959236558 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b402cc40 con 0x7fd4c414f360 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c159a640 1 -- 172.21.15.94:0/1959236558 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4034440 con 0x7fd4c414f360 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/1959236558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c414f360 msgr2=0x7fd4c4155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/1959236558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c414f360 0x7fd4c4155c10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009dc0 tx=0x7fd4b402be90 comp rx=0 tx=0).stop 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/1959236558 shutdown_connections 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/1959236558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c414f360 0x7fd4c4155c10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/1959236558 >> 172.21.15.94:0/1959236558 conn(0x7fd4c40ac570 msgr2=0x7fd4c40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.046+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/1959236558 shutdown_connections 2024-03-06T07:29:05.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.047+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/1959236558 wait complete. 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 Processor -- start 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 -- start start 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4c40c88c0 0x7fd4c40ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4c414f360 0x7fd4c4156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c4158b60 0x7fd4c4156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40c7c30 con 0x7fd4c414f360 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40c5a50 con 0x7fd4c40c88c0 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.048+0000 7fd4c8a8d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40c5bd0 con 0x7fd4c4158b60 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4c414f360 0x7fd4c4156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4c414f360 0x7fd4c4156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36240/0 (socket says 172.21.15.94:36240) 2024-03-06T07:29:05.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 -- 172.21.15.94:0/2855719628 learned_addr learned my addr 172.21.15.94:0/2855719628 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c259c640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4c40c88c0 0x7fd4c40ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c2d9d640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c4158b60 0x7fd4c4156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 -- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c4158b60 msgr2=0x7fd4c4156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c4158b60 0x7fd4c4156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 -- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4c40c88c0 msgr2=0x7fd4c40ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4c40c88c0 0x7fd4c40ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 -- 172.21.15.94:0/2855719628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c41eaf80 con 0x7fd4c414f360 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c1d9b640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4c414f360 0x7fd4c4156860 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c3e0 tx=0x7fd4b800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4b2ffd640 1 -- 172.21.15.94:0/2855719628 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8017070 con 0x7fd4c414f360 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4b2ffd640 1 -- 172.21.15.94:0/2855719628 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b80035f0 con 0x7fd4c414f360 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c41eb260 con 0x7fd4c414f360 2024-03-06T07:29:05.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4b2ffd640 1 -- 172.21.15.94:0/2855719628 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8003a00 con 0x7fd4c414f360 2024-03-06T07:29:05.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.049+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c41eb820 con 0x7fd4c414f360 2024-03-06T07:29:05.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.050+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c404f330 con 0x7fd4c414f360 2024-03-06T07:29:05.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.051+0000 7fd4b2ffd640 1 -- 172.21.15.94:0/2855719628 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4b801a070 con 0x7fd4c414f360 2024-03-06T07:29:05.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.051+0000 7fd4b2ffd640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.051+0000 7fd4b2ffd640 1 -- 172.21.15.94:0/2855719628 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd4b8089790 con 0x7fd4c414f360 2024-03-06T07:29:05.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.051+0000 7fd4c259c640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.052+0000 7fd4c259c640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4a406b4d0 0x7fd4a406d990 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4036000 tx=0x7fd4b4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.057 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.056+0000 7fd4b2ffd640 1 -- 172.21.15.94:0/2855719628 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4b8057dc0 con 0x7fd4c414f360 2024-03-06T07:29:05.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.261+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4c40c8d50 con 0x7fd4c414f360 2024-03-06T07:29:05.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.261+0000 7fd4b2ffd640 1 -- 172.21.15.94:0/2855719628 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7fd4b805be60 con 0x7fd4c414f360 2024-03-06T07:29:05.263 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:05.263 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:05.263 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:29:05.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4a406b4d0 msgr2=0x7fd4a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4a406b4d0 0x7fd4a406d990 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4036000 tx=0x7fd4b4030000 comp rx=0 tx=0).stop 2024-03-06T07:29:05.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4c414f360 msgr2=0x7fd4c4156860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4c414f360 0x7fd4c4156860 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c3e0 tx=0x7fd4b800c8a0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 shutdown_connections 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4c4158b60 0x7fd4c4156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4c414f360 0x7fd4c4156860 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 --2- 172.21.15.94:0/2855719628 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4c40c88c0 0x7fd4c40ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 >> 172.21.15.94:0/2855719628 conn(0x7fd4c40ac570 msgr2=0x7fd4c4158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 shutdown_connections 2024-03-06T07:29:05.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:05.263+0000 7fd4c8a8d640 1 -- 172.21.15.94:0/2855719628 wait complete. 2024-03-06T07:29:05.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:05.678 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:06.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.789+0000 7f0f98772640 1 Processor -- start 2024-03-06T07:29:06.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.789+0000 7f0f98772640 1 -- start start 2024-03-06T07:29:06.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f98772640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f98772640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0f900c6000 0x7f0f90156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f98772640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f90153a60 con 0x7f0f90150580 2024-03-06T07:29:06.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f98772640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f90153d40 con 0x7f0f900c6000 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f98772640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9006c2c0 con 0x7f0f900c1320 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f96d0f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0f900c6000 0x7f0f90156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f9650e640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0f90150580 0x7f0f9006dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:49972/0 (socket says 172.21.15.94:49972) 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f9650e640 1 -- 172.21.15.94:0/1915396136 learned_addr learned my addr 172.21.15.94:0/1915396136 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.790+0000 7f0f95d0d640 1 --2- 172.21.15.94:0/1915396136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f95d0d640 1 -- 172.21.15.94:0/1915396136 >> v2:172.21.15.145:3300/0 conn(0x7f0f900c6000 msgr2=0x7f0f90156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f95d0d640 1 --2- 172.21.15.94:0/1915396136 >> v2:172.21.15.145:3300/0 conn(0x7f0f900c6000 0x7f0f90156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f95d0d640 1 -- 172.21.15.94:0/1915396136 >> v1:172.21.15.145:6789/0 conn(0x7f0f90150580 legacy=0x7f0f9006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f95d0d640 1 -- 172.21.15.94:0/1915396136 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9006c5a0 con 0x7f0f900c1320 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f9550c640 1 -- 172.21.15.94:0/1915396136 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184020106 0 0) 0x7f0f90153a60 con 0x7f0f90150580 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f96d0f640 1 --2- 172.21.15.94:0/1915396136 >> v2:172.21.15.145:3300/0 conn(0x7f0f900c6000 0x7f0f90156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:06.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f95d0d640 1 --2- 172.21.15.94:0/1915396136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901566b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f800097a0 tx=0x7f0f8002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57d1b978b489e8a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.791+0000 7f0f9550c640 1 -- 172.21.15.94:0/1915396136 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f800315a0 con 0x7f0f900c1320 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f9550c640 1 -- 172.21.15.94:0/1915396136 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f80037070 con 0x7f0f900c1320 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f9550c640 1 -- 172.21.15.94:0/1915396136 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80031960 con 0x7f0f900c1320 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 -- 172.21.15.94:0/1915396136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 msgr2=0x7f0f901566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 --2- 172.21.15.94:0/1915396136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901566b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f800097a0 tx=0x7f0f8002b840 comp rx=0 tx=0).stop 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 -- 172.21.15.94:0/1915396136 shutdown_connections 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 --2- 172.21.15.94:0/1915396136 >> v2:172.21.15.145:3300/0 conn(0x7f0f900c6000 0x7f0f90156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 --2- 172.21.15.94:0/1915396136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901566b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 -- 172.21.15.94:0/1915396136 >> 172.21.15.94:0/1915396136 conn(0x7f0f900c6710 msgr2=0x7f0f900c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 -- 172.21.15.94:0/1915396136 shutdown_connections 2024-03-06T07:29:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.792+0000 7f0f98772640 1 -- 172.21.15.94:0/1915396136 wait complete. 2024-03-06T07:29:06.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.793+0000 7f0f98772640 1 Processor -- start 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.793+0000 7f0f98772640 1 -- start start 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f98772640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f98772640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f900c6000 0x7f0f901471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f98772640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f90150580 0x7f0f901ec530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f98772640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9006e5a0 con 0x7f0f90150580 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f98772640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9006e420 con 0x7f0f900c6000 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f98772640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f900c0c40 con 0x7f0f900c1320 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f90150580 0x7f0f901ec530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f90150580 0x7f0f901ec530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36252/0 (socket says 172.21.15.94:36252) 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f9650e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f95d0d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f900c6000 0x7f0f901471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 -- 172.21.15.94:0/1567792222 learned_addr learned my addr 172.21.15.94:0/1567792222 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 -- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 msgr2=0x7f0f901546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 -- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f900c6000 msgr2=0x7f0f901471a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f900c6000 0x7f0f901471a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f96d0f640 1 -- 172.21.15.94:0/1567792222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f900a8090 con 0x7f0f90150580 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f9650e640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.794+0000 7f0f95d0d640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f900c6000 0x7f0f901471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.795+0000 7f0f96d0f640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f90150580 0x7f0f901ec530 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009f00 tx=0x7f0f8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.795+0000 7f0f86ffd640 1 -- 172.21.15.94:0/1567792222 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88018030 con 0x7f0f90150580 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.795+0000 7f0f86ffd640 1 -- 172.21.15.94:0/1567792222 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8800ce50 con 0x7f0f90150580 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.795+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f90151cd0 con 0x7f0f90150580 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.795+0000 7f0f86ffd640 1 -- 172.21.15.94:0/1567792222 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88003bb0 con 0x7f0f90150580 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.795+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f90155f10 con 0x7f0f90150580 2024-03-06T07:29:06.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.796+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9004f330 con 0x7f0f90150580 2024-03-06T07:29:06.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.796+0000 7f0f86ffd640 1 -- 172.21.15.94:0/1567792222 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f8801b070 con 0x7f0f90150580 2024-03-06T07:29:06.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.797+0000 7f0f86ffd640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7406b4d0 0x7f0f7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.797+0000 7f0f86ffd640 1 -- 172.21.15.94:0/1567792222 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0f8808acd0 con 0x7f0f90150580 2024-03-06T07:29:06.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.797+0000 7f0f9650e640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7406b4d0 0x7f0f7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.797+0000 7f0f9650e640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7406b4d0 0x7f0f7406d990 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c007fd0 tx=0x7f0f8c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:06.801+0000 7f0f86ffd640 1 -- 172.21.15.94:0/1567792222 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f88010040 con 0x7f0f90150580 2024-03-06T07:29:07.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.008+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f9006cfe0 con 0x7f0f90150580 2024-03-06T07:29:07.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.008+0000 7f0f86ffd640 1 -- 172.21.15.94:0/1567792222 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f0f88059300 con 0x7f0f90150580 2024-03-06T07:29:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:07.011 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:07.011 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:29:07.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7406b4d0 msgr2=0x7f0f7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7406b4d0 0x7f0f7406d990 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c007fd0 tx=0x7f0f8c007de0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f90150580 msgr2=0x7f0f901ec530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f90150580 0x7f0f901ec530 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009f00 tx=0x7f0f8800c490 comp rx=0 tx=0).stop 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 shutdown_connections 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0f7406b4d0 0x7f0f7406d990 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0f90150580 0x7f0f901ec530 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f900c6000 0x7f0f901471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 --2- 172.21.15.94:0/1567792222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f900c1320 0x7f0f901546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 >> 172.21.15.94:0/1567792222 conn(0x7f0f900c6710 msgr2=0x7f0f9014f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 shutdown_connections 2024-03-06T07:29:07.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:07.010+0000 7f0f98772640 1 -- 172.21.15.94:0/1567792222 wait complete. 2024-03-06T07:29:07.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:07.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:08.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:08.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.537+0000 7fcdffa5c640 1 Processor -- start 2024-03-06T07:29:08.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.537+0000 7fcdffa5c640 1 -- start start 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.537+0000 7fcdffa5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf80c88c0 0x7fcdf8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.537+0000 7fcdffa5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcdf8158b60 0x7fcdf8158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdffa5c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf80c73d0 con 0x7fcdf814f360 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdffa5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf80c8dd0 con 0x7fcdf8158b60 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdffa5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80be2e0 con 0x7fcdf80c88c0 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfd7f8640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcdf814f360 0x7fcdf80c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47928/0 (socket says 172.21.15.94:47928) 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfd7f8640 1 -- 172.21.15.94:0/3425602098 learned_addr learned my addr 172.21.15.94:0/3425602098 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:08.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfcff7640 1 --2- 172.21.15.94:0/3425602098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf80c88c0 0x7fcdf8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfdff9640 1 --2- 172.21.15.94:0/3425602098 >> v2:172.21.15.145:3300/0 conn(0x7fcdf8158b60 0x7fcdf8158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdebfff640 1 -- 172.21.15.94:0/3425602098 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839438573 0 0) 0x7fcdf80c73d0 con 0x7fcdf814f360 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdebfff640 1 -- 172.21.15.94:0/3425602098 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4003520 con 0x7fcdf814f360 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfcff7640 1 -- 172.21.15.94:0/3425602098 >> v2:172.21.15.145:3300/0 conn(0x7fcdf8158b60 msgr2=0x7fcdf8158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfcff7640 1 --2- 172.21.15.94:0/3425602098 >> v2:172.21.15.145:3300/0 conn(0x7fcdf8158b60 0x7fcdf8158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfcff7640 1 -- 172.21.15.94:0/3425602098 >> v1:172.21.15.94:6789/0 conn(0x7fcdf814f360 legacy=0x7fcdf80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfcff7640 1 -- 172.21.15.94:0/3425602098 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf80c7670 con 0x7fcdf80c88c0 2024-03-06T07:29:08.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.538+0000 7fcdfdff9640 1 --2- 172.21.15.94:0/3425602098 >> v2:172.21.15.145:3300/0 conn(0x7fcdf8158b60 0x7fcdf8158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:08.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.539+0000 7fcdfcff7640 1 --2- 172.21.15.94:0/3425602098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf80c88c0 0x7fcdf8155c10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40097a0 tx=0x7fcdf402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ea32048d4e1e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.539+0000 7fcdebfff640 1 -- 172.21.15.94:0/3425602098 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40315a0 con 0x7fcdf80c88c0 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.539+0000 7fcdebfff640 1 -- 172.21.15.94:0/3425602098 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4037070 con 0x7fcdf80c88c0 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.539+0000 7fcdebfff640 1 -- 172.21.15.94:0/3425602098 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4031960 con 0x7fcdf80c88c0 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 -- 172.21.15.94:0/3425602098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf80c88c0 msgr2=0x7fcdf8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/3425602098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf80c88c0 0x7fcdf8155c10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40097a0 tx=0x7fcdf402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 -- 172.21.15.94:0/3425602098 shutdown_connections 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/3425602098 >> v2:172.21.15.145:3300/0 conn(0x7fcdf8158b60 0x7fcdf8158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/3425602098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf80c88c0 0x7fcdf8155c10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 -- 172.21.15.94:0/3425602098 >> 172.21.15.94:0/3425602098 conn(0x7fcdf80ac570 msgr2=0x7fcdf80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 -- 172.21.15.94:0/3425602098 shutdown_connections 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.540+0000 7fcdffa5c640 1 -- 172.21.15.94:0/3425602098 wait complete. 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 Processor -- start 2024-03-06T07:29:08.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 -- start start 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80c88c0 0x7fcdf81eb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf814f360 0x7fcdf80aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf8158b60 0x7fcdf814b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80c5a50 con 0x7fcdf8158b60 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80c7c30 con 0x7fcdf80c88c0 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.541+0000 7fcdffa5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80c5bd0 con 0x7fcdf814f360 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf8158b60 0x7fcdf814b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf8158b60 0x7fcdf814b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39472/0 (socket says 172.21.15.94:39472) 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 -- 172.21.15.94:0/1512807363 learned_addr learned my addr 172.21.15.94:0/1512807363 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:08.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfd7f8640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80c88c0 0x7fcdf81eb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfcff7640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf814f360 0x7fcdf80aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 -- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf814f360 msgr2=0x7fcdf80aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf814f360 0x7fcdf80aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 -- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80c88c0 msgr2=0x7fcdf81eb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80c88c0 0x7fcdf81eb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 -- 172.21.15.94:0/1512807363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf80c1490 con 0x7fcdf8158b60 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdfdff9640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf8158b60 0x7fcdf814b9c0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000efc0 tx=0x7fcdf0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcde9ffb640 1 -- 172.21.15.94:0/1512807363 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0018030 con 0x7fcdf8158b60 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcde9ffb640 1 -- 172.21.15.94:0/1512807363 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf000cde0 con 0x7fcdf8158b60 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf80c0ed0 con 0x7fcdf8158b60 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.542+0000 7fcde9ffb640 1 -- 172.21.15.94:0/1512807363 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0003bb0 con 0x7fcdf8158b60 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.543+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf80ad660 con 0x7fcdf8158b60 2024-03-06T07:29:08.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.544+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf804f330 con 0x7fcdf8158b60 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.544+0000 7fcde9ffb640 1 -- 172.21.15.94:0/1512807363 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdf0005d10 con 0x7fcdf8158b60 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.544+0000 7fcde9ffb640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcddc06b500 0x7fcddc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.544+0000 7fcde9ffb640 1 -- 172.21.15.94:0/1512807363 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcdf008ad40 con 0x7fcdf8158b60 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.544+0000 7fcdfd7f8640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcddc06b500 0x7fcddc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.545+0000 7fcdfd7f8640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcddc06b500 0x7fcddc06d9c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fcdec005880 tx=0x7fcdec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.549+0000 7fcde9ffb640 1 -- 172.21.15.94:0/1512807363 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf0022070 con 0x7fcdf8158b60 2024-03-06T07:29:08.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.757+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdf806cda0 con 0x7fcdf8158b60 2024-03-06T07:29:08.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.758+0000 7fcde9ffb640 1 -- 172.21.15.94:0/1512807363 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7fcdf0010040 con 0x7fcdf8158b60 2024-03-06T07:29:08.760 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:08.760 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:08.760 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:29:08.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcddc06b500 msgr2=0x7fcddc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcddc06b500 0x7fcddc06d9c0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fcdec005880 tx=0x7fcdec009040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf8158b60 msgr2=0x7fcdf814b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf8158b60 0x7fcdf814b9c0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000efc0 tx=0x7fcdf0009f90 comp rx=0 tx=0).stop 2024-03-06T07:29:08.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 shutdown_connections 2024-03-06T07:29:08.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcddc06b500 0x7fcddc06d9c0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcdf8158b60 0x7fcdf814b9c0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdf814f360 0x7fcdf80aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 --2- 172.21.15.94:0/1512807363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdf80c88c0 0x7fcdf81eb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 >> 172.21.15.94:0/1512807363 conn(0x7fcdf80ac570 msgr2=0x7fcdf80c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 shutdown_connections 2024-03-06T07:29:08.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:08.760+0000 7fcdffa5c640 1 -- 172.21.15.94:0/1512807363 wait complete. 2024-03-06T07:29:09.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:09.175 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:10.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.285+0000 7f02044b5640 1 Processor -- start 2024-03-06T07:29:10.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.285+0000 7f02044b5640 1 -- start start 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f02044b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f02044b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01fc0c1aa0 0x7f01fc0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f02044b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f01fc0c5c60 0x7f01fc155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f02044b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01fc0c2680 con 0x7f01fc0c1aa0 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f02044b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01fc0c28d0 con 0x7f01fc0c5c60 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f02044b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc0c2bb0 con 0x7f01fc14f790 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f0201a50640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01fc0c1aa0 0x7f01fc0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f0201a50640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01fc0c1aa0 0x7f01fc0c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39482/0 (socket says 172.21.15.94:39482) 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.286+0000 7f0201a50640 1 -- 172.21.15.94:0/3869213949 learned_addr learned my addr 172.21.15.94:0/3869213949 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.287+0000 7f0202251640 1 --2- 172.21.15.94:0/3869213949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.287+0000 7f0202a52640 1 --2- 172.21.15.94:0/3869213949 >> v2:172.21.15.145:3300/0 conn(0x7f01fc0c5c60 0x7f01fc155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.287+0000 7f0201a50640 1 -- 172.21.15.94:0/3869213949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 msgr2=0x7f01fc1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.287+0000 7f0201a50640 1 --2- 172.21.15.94:0/3869213949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.287+0000 7f0201a50640 1 -- 172.21.15.94:0/3869213949 >> v2:172.21.15.145:3300/0 conn(0x7f01fc0c5c60 msgr2=0x7f01fc155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.287+0000 7f0201a50640 1 --2- 172.21.15.94:0/3869213949 >> v2:172.21.15.145:3300/0 conn(0x7f01fc0c5c60 0x7f01fc155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.287+0000 7f0201a50640 1 -- 172.21.15.94:0/3869213949 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc0c2e90 con 0x7f01fc0c1aa0 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f0201a50640 1 --2- 172.21.15.94:0/3869213949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c1aa0 0x7f01fc0c5720 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f01fc06c960 tx=0x7f01f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57f7adcb2bb623f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f020124f640 1 -- 172.21.15.94:0/3869213949 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8035020 con 0x7f01fc0c1aa0 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f020124f640 1 -- 172.21.15.94:0/3869213949 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f8031870 con 0x7f01fc0c1aa0 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f020124f640 1 -- 172.21.15.94:0/3869213949 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8031c40 con 0x7f01fc0c1aa0 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 -- 172.21.15.94:0/3869213949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c1aa0 msgr2=0x7f01fc0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 --2- 172.21.15.94:0/3869213949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c1aa0 0x7f01fc0c5720 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f01fc06c960 tx=0x7f01f802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 -- 172.21.15.94:0/3869213949 shutdown_connections 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 --2- 172.21.15.94:0/3869213949 >> v2:172.21.15.145:3300/0 conn(0x7f01fc0c5c60 0x7f01fc155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 --2- 172.21.15.94:0/3869213949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c1aa0 0x7f01fc0c5720 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 --2- 172.21.15.94:0/3869213949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 -- 172.21.15.94:0/3869213949 >> 172.21.15.94:0/3869213949 conn(0x7f01fc0c6710 msgr2=0x7f01fc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 -- 172.21.15.94:0/3869213949 shutdown_connections 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.288+0000 7f02044b5640 1 -- 172.21.15.94:0/3869213949 wait complete. 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.289+0000 7f02044b5640 1 Processor -- start 2024-03-06T07:29:10.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f02044b5640 1 -- start start 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f02044b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01fc0c1aa0 0x7f01fc1eab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f02044b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c5c60 0x7f01fc14e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f02044b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f02044b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc06cc00 con 0x7f01fc0c5c60 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f02044b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc0c0d60 con 0x7f01fc0c1aa0 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f02044b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc06cd80 con 0x7f01fc14f790 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f0201a50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c5c60 0x7f01fc14e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f0201a50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c5c60 0x7f01fc14e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39486/0 (socket says 172.21.15.94:39486) 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f0201a50640 1 -- 172.21.15.94:0/2762575806 learned_addr learned my addr 172.21.15.94:0/2762575806 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f0202251640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01fc0c1aa0 0x7f01fc1eab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.290+0000 7f0202a52640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f0201a50640 1 -- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 msgr2=0x7f01fc072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f0201a50640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f0201a50640 1 -- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01fc0c1aa0 msgr2=0x7f01fc1eab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f0201a50640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01fc0c1aa0 0x7f01fc1eab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f0201a50640 1 -- 172.21.15.94:0/2762575806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc0c0a50 con 0x7f01fc0c5c60 2024-03-06T07:29:10.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f0201a50640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c5c60 0x7f01fc14e3d0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f01f8000c00 tx=0x7f01f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f01eeffd640 1 -- 172.21.15.94:0/2762575806 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8031e10 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f01eeffd640 1 -- 172.21.15.94:0/2762575806 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f803ad60 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01fc0c01b0 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f01eeffd640 1 -- 172.21.15.94:0/2762575806 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8044610 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.291+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01fc0c0770 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.292+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01fc071a20 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.292+0000 7f01eeffd640 1 -- 172.21.15.94:0/2762575806 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01f804b060 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.293+0000 7f01eeffd640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01e006b4d0 0x7f01e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.293+0000 7f01eeffd640 1 -- 172.21.15.94:0/2762575806 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f01f80ac2a0 con 0x7f01fc0c5c60 2024-03-06T07:29:10.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.293+0000 7f0202251640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01e006b4d0 0x7f01e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.293+0000 7f0202251640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01e006b4d0 0x7f01e006d990 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f01fc1ebca0 tx=0x7f01f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.298 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.297+0000 7f01eeffd640 1 -- 172.21.15.94:0/2762575806 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01fc071a20 con 0x7f01fc0c5c60 2024-03-06T07:29:10.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.512+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01fc04f330 con 0x7f01fc0c5c60 2024-03-06T07:29:10.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.512+0000 7f01eeffd640 1 -- 172.21.15.94:0/2762575806 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+3431 (secure 0 0 0) 0x7f01fc0c0a50 con 0x7f01fc0c5c60 2024-03-06T07:29:10.514 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:10.514 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":212,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:28:41.262586+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:10.515 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 212 2024-03-06T07:29:10.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01e006b4d0 msgr2=0x7f01e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01e006b4d0 0x7f01e006d990 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f01fc1ebca0 tx=0x7f01f0009190 comp rx=0 tx=0).stop 2024-03-06T07:29:10.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c5c60 msgr2=0x7f01fc14e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c5c60 0x7f01fc14e3d0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f01f8000c00 tx=0x7f01f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 shutdown_connections 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f01e006b4d0 0x7f01e006d990 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f01fc14f790 0x7f01fc072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01fc0c5c60 0x7f01fc14e3d0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 --2- 172.21.15.94:0/2762575806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f01fc0c1aa0 0x7f01fc1eab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 >> 172.21.15.94:0/2762575806 conn(0x7f01fc0c6710 msgr2=0x7f01fc0a9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 shutdown_connections 2024-03-06T07:29:10.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:10.514+0000 7f02044b5640 1 -- 172.21.15.94:0/2762575806 wait complete. 2024-03-06T07:29:10.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 212, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:28:41.262586+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:10.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:11.931 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:12.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.041+0000 7fc4bf60e640 1 Processor -- start 2024-03-06T07:29:12.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.041+0000 7fc4bf60e640 1 -- start start 2024-03-06T07:29:12.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bf60e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc4b8150580 0x7fc4b806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bf60e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc4b80c1320 0x7fc4b806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bf60e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b80c6000 0x7fc4b81566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bf60e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b8153a60 con 0x7fc4b8150580 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bf60e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b8153d40 con 0x7fc4b80c1320 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bf60e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b806c2c0 con 0x7fc4b80c6000 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bd3aa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc4b8150580 0x7fc4b806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bd3aa640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc4b8150580 0x7fc4b806dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39500/0 (socket says 172.21.15.94:39500) 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/319512387 learned_addr learned my addr 172.21.15.94:0/319512387 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bdbab640 1 --2- 172.21.15.94:0/319512387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b80c6000 0x7fc4b81566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.042+0000 7fc4bcba9640 1 --2- 172.21.15.94:0/319512387 >> v2:172.21.15.145:3300/0 conn(0x7fc4b80c1320 0x7fc4b806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/319512387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b80c6000 msgr2=0x7fc4b81566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4bd3aa640 1 --2- 172.21.15.94:0/319512387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b80c6000 0x7fc4b81566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/319512387 >> v2:172.21.15.145:3300/0 conn(0x7fc4b80c1320 msgr2=0x7fc4b806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4bd3aa640 1 --2- 172.21.15.94:0/319512387 >> v2:172.21.15.145:3300/0 conn(0x7fc4b80c1320 0x7fc4b806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/319512387 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b806c5a0 con 0x7fc4b8150580 2024-03-06T07:29:12.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4bd3aa640 1 --2- 172.21.15.94:0/319512387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b8150580 0x7fc4b806dd10 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80c7730 tx=0x7fc4b402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44d0ac38ac9ed25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4affff640 1 -- 172.21.15.94:0/319512387 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b402e070 con 0x7fc4b8150580 2024-03-06T07:29:12.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.043+0000 7fc4affff640 1 -- 172.21.15.94:0/319512387 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b4037070 con 0x7fc4b8150580 2024-03-06T07:29:12.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4affff640 1 -- 172.21.15.94:0/319512387 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4031a00 con 0x7fc4b8150580 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 -- 172.21.15.94:0/319512387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b8150580 msgr2=0x7fc4b806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/319512387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b8150580 0x7fc4b806dd10 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80c7730 tx=0x7fc4b402b880 comp rx=0 tx=0).stop 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 -- 172.21.15.94:0/319512387 shutdown_connections 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/319512387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b80c6000 0x7fc4b81566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/319512387 >> v2:172.21.15.145:3300/0 conn(0x7fc4b80c1320 0x7fc4b806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/319512387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b8150580 0x7fc4b806dd10 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 -- 172.21.15.94:0/319512387 >> 172.21.15.94:0/319512387 conn(0x7fc4b80c6710 msgr2=0x7fc4b80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.050 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 -- 172.21.15.94:0/319512387 shutdown_connections 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.044+0000 7fc4bf60e640 1 -- 172.21.15.94:0/319512387 wait complete. 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.045+0000 7fc4bf60e640 1 Processor -- start 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.045+0000 7fc4bf60e640 1 -- start start 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bf60e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b80c1320 0x7fc4b80c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bf60e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4b80c6000 0x7fc4b81eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bf60e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b8150580 0x7fc4b81eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bf60e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b80c0dc0 con 0x7fc4b80c1320 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bf60e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b80c0c40 con 0x7fc4b80c6000 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bf60e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b8156bf0 con 0x7fc4b8150580 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b80c1320 0x7fc4b80c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b80c1320 0x7fc4b80c2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39512/0 (socket says 172.21.15.94:39512) 2024-03-06T07:29:12.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/2031632833 learned_addr learned my addr 172.21.15.94:0/2031632833 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bcba9640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4b80c6000 0x7fc4b81eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bdbab640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b8150580 0x7fc4b81eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b8150580 msgr2=0x7fc4b81eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b8150580 0x7fc4b81eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4b80c6000 msgr2=0x7fc4b81eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4b80c6000 0x7fc4b81eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.046+0000 7fc4bd3aa640 1 -- 172.21.15.94:0/2031632833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b8145b00 con 0x7fc4b80c1320 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.047+0000 7fc4bd3aa640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b80c1320 0x7fc4b80c2ce0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7fc4b402b8c0 tx=0x7fc4b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.047+0000 7fc4adffb640 1 -- 172.21.15.94:0/2031632833 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4032670 con 0x7fc4b80c1320 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.047+0000 7fc4adffb640 1 -- 172.21.15.94:0/2031632833 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b4032830 con 0x7fc4b80c1320 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.047+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b80a7fd0 con 0x7fc4b80c1320 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.047+0000 7fc4adffb640 1 -- 172.21.15.94:0/2031632833 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4032c20 con 0x7fc4b80c1320 2024-03-06T07:29:12.052 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.047+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b80c3220 con 0x7fc4b80c1320 2024-03-06T07:29:12.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.048+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b804f330 con 0x7fc4b80c1320 2024-03-06T07:29:12.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.048+0000 7fc4adffb640 1 -- 172.21.15.94:0/2031632833 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4b404c060 con 0x7fc4b80c1320 2024-03-06T07:29:12.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.049+0000 7fc4adffb640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.049+0000 7fc4adffb640 1 -- 172.21.15.94:0/2031632833 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc4b40abc50 con 0x7fc4b80c1320 2024-03-06T07:29:12.053 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.049+0000 7fc4bcba9640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.054 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.049+0000 7fc4bcba9640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49c06b4d0 0x7fc49c06d990 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc4b814db60 tx=0x7fc4a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.055 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.053+0000 7fc4adffb640 1 -- 172.21.15.94:0/2031632833 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4b40b1020 con 0x7fc4b80c1320 2024-03-06T07:29:12.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.261+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4b80c5de0 con 0x7fc4b80c1320 2024-03-06T07:29:12.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.262+0000 7fc4adffb640 1 -- 172.21.15.94:0/2031632833 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fc4b4040070 con 0x7fc4b80c1320 2024-03-06T07:29:12.264 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:12.264 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:12.264 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49c06b4d0 msgr2=0x7fc49c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49c06b4d0 0x7fc49c06d990 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc4b814db60 tx=0x7fc4a8009190 comp rx=0 tx=0).stop 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b80c1320 msgr2=0x7fc4b80c2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b80c1320 0x7fc4b80c2ce0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7fc4b402b8c0 tx=0x7fc4b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 shutdown_connections 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc49c06b4d0 0x7fc49c06d990 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4b8150580 0x7fc4b81eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4b80c6000 0x7fc4b81eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 --2- 172.21.15.94:0/2031632833 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4b80c1320 0x7fc4b80c2ce0 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 >> 172.21.15.94:0/2031632833 conn(0x7fc4b80c6710 msgr2=0x7fc4b80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 shutdown_connections 2024-03-06T07:29:12.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:12.264+0000 7fc4bf60e640 1 -- 172.21.15.94:0/2031632833 wait complete. 2024-03-06T07:29:12.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:12.679 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:13.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:13.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c2eb5640 1 Processor -- start 2024-03-06T07:29:13.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c2eb5640 1 -- start start 2024-03-06T07:29:13.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c2eb5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24bc14f790 0x7f24bc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c2eb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc0c1aa0 0x7f24bc0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c2eb5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24bc0c28d0 con 0x7f24bc0c5c60 2024-03-06T07:29:13.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c2eb5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24bc0c2680 con 0x7f24bc14f790 2024-03-06T07:29:13.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c2eb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0c2bb0 con 0x7f24bc0c1aa0 2024-03-06T07:29:13.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c0c51640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f24bc14f790 0x7f24bc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c1452640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f24bc0c5c60 0x7f24bc155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47934/0 (socket says 172.21.15.94:47934) 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.790+0000 7f24c1452640 1 -- 172.21.15.94:0/1878800397 learned_addr learned my addr 172.21.15.94:0/1878800397 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.791+0000 7f24bbfff640 1 --2- 172.21.15.94:0/1878800397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc0c1aa0 0x7f24bc0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.791+0000 7f24c0c51640 1 -- 172.21.15.94:0/1878800397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc0c1aa0 msgr2=0x7f24bc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.791+0000 7f24c0c51640 1 --2- 172.21.15.94:0/1878800397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc0c1aa0 0x7f24bc0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.791+0000 7f24c0c51640 1 -- 172.21.15.94:0/1878800397 >> v1:172.21.15.94:6789/0 conn(0x7f24bc0c5c60 legacy=0x7f24bc155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.791+0000 7f24bb7fe640 1 -- 172.21.15.94:0/1878800397 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536276066 0 0) 0x7f24bc0c28d0 con 0x7f24bc0c5c60 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.791+0000 7f24c0c51640 1 -- 172.21.15.94:0/1878800397 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0c2e90 con 0x7f24bc14f790 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c0c51640 1 --2- 172.21.15.94:0/1878800397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc14f790 0x7f24bc1529d0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f24b00097a0 tx=0x7f24b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea08fb533f37e130 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24bb7fe640 1 -- 172.21.15.94:0/1878800397 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0035020 con 0x7f24bc14f790 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24bb7fe640 1 -- 172.21.15.94:0/1878800397 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b0031830 con 0x7f24bc14f790 2024-03-06T07:29:13.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24bb7fe640 1 -- 172.21.15.94:0/1878800397 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b0031c00 con 0x7f24bc14f790 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1878800397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc14f790 msgr2=0x7f24bc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1878800397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc14f790 0x7f24bc1529d0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f24b00097a0 tx=0x7f24b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1878800397 shutdown_connections 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1878800397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc0c1aa0 0x7f24bc0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1878800397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc14f790 0x7f24bc1529d0 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1878800397 >> 172.21.15.94:0/1878800397 conn(0x7f24bc0c6710 msgr2=0x7f24bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1878800397 shutdown_connections 2024-03-06T07:29:13.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.792+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1878800397 wait complete. 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.793+0000 7f24c2eb5640 1 Processor -- start 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24c2eb5640 1 -- start start 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24c2eb5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24bc0c1aa0 0x7f24bc15bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24c2eb5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc0c5c60 0x7f24bc14e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24c2eb5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc14f790 0x7f24bc14eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24c2eb5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc06cc00 con 0x7f24bc0c5c60 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24c2eb5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0c0d60 con 0x7f24bc0c1aa0 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24c2eb5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc06cd80 con 0x7f24bc14f790 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.794+0000 7f24bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc0c5c60 0x7f24bc14e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.799 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc0c5c60 0x7f24bc14e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39542/0 (socket says 172.21.15.94:39542) 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 -- 172.21.15.94:0/1678708166 learned_addr learned my addr 172.21.15.94:0/1678708166 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24c0c51640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24bc0c1aa0 0x7f24bc15bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24c1452640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc14f790 0x7f24bc14eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 -- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc14f790 msgr2=0x7f24bc14eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc14f790 0x7f24bc14eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 -- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24bc0c1aa0 msgr2=0x7f24bc15bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24bc0c1aa0 0x7f24bc15bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 -- 172.21.15.94:0/1678708166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc1e8d40 con 0x7f24bc0c5c60 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24bbfff640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc0c5c60 0x7f24bc14e860 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f24a800e9b0 tx=0x7f24a800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24b97fa640 1 -- 172.21.15.94:0/1678708166 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8017070 con 0x7f24bc0c5c60 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24b97fa640 1 -- 172.21.15.94:0/1678708166 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a800c9a0 con 0x7f24bc0c5c60 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc1e9020 con 0x7f24bc0c5c60 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24b97fa640 1 -- 172.21.15.94:0/1678708166 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a800cd90 con 0x7f24bc0c5c60 2024-03-06T07:29:13.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.795+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24bc1e95e0 con 0x7f24bc0c5c60 2024-03-06T07:29:13.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.796+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24bc04f330 con 0x7f24bc0c5c60 2024-03-06T07:29:13.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.797+0000 7f24b97fa640 1 -- 172.21.15.94:0/1678708166 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24a801a070 con 0x7f24bc0c5c60 2024-03-06T07:29:13.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.797+0000 7f24b97fa640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24a006b4d0 0x7f24a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.797+0000 7f24b97fa640 1 -- 172.21.15.94:0/1678708166 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f24a808a730 con 0x7f24bc0c5c60 2024-03-06T07:29:13.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.797+0000 7f24c0c51640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24a006b4d0 0x7f24a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.798+0000 7f24c0c51640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24a006b4d0 0x7f24a006d990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f24b0036000 tx=0x7f24b003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:13.801+0000 7f24b97fa640 1 -- 172.21.15.94:0/1678708166 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24a8058d60 con 0x7f24bc0c5c60 2024-03-06T07:29:14.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.007+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24bc0c1f40 con 0x7f24bc0c5c60 2024-03-06T07:29:14.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.007+0000 7f24b97fa640 1 -- 172.21.15.94:0/1678708166 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f24a805ce00 con 0x7f24bc0c5c60 2024-03-06T07:29:14.012 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:14.012 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:14.013 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:14.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24a006b4d0 msgr2=0x7f24a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24a006b4d0 0x7f24a006d990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f24b0036000 tx=0x7f24b003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:14.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc0c5c60 msgr2=0x7f24bc14e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc0c5c60 0x7f24bc14e860 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f24a800e9b0 tx=0x7f24a800ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 shutdown_connections 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f24a006b4d0 0x7f24a006d990 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24bc14f790 0x7f24bc14eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f24bc0c5c60 0x7f24bc14e860 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 --2- 172.21.15.94:0/1678708166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24bc0c1aa0 0x7f24bc15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 >> 172.21.15.94:0/1678708166 conn(0x7f24bc0c6710 msgr2=0x7f24bc0c4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 shutdown_connections 2024-03-06T07:29:14.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:14.009+0000 7f24c2eb5640 1 -- 172.21.15.94:0/1678708166 wait complete. 2024-03-06T07:29:14.424 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:14.424 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:15.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:15.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.535+0000 7fb0b6de2640 1 Processor -- start 2024-03-06T07:29:15.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.535+0000 7fb0b6de2640 1 -- start start 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b6de2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb0b014f790 0x7fb0b01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b6de2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb0b00c1aa0 0x7fb0b00c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b6de2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b00c28d0 con 0x7fb0b00c5c60 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b6de2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b00c2680 con 0x7fb0b014f790 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b6de2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b00c2bb0 con 0x7fb0b00c1aa0 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b4b7e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb0b014f790 0x7fb0b01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b537f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb0b00c5c60 0x7fb0b0155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47946/0 (socket says 172.21.15.94:47946) 2024-03-06T07:29:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b537f640 1 -- 172.21.15.94:0/45162235 learned_addr learned my addr 172.21.15.94:0/45162235 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:15.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0affff640 1 --2- 172.21.15.94:0/45162235 >> v2:172.21.15.145:3300/0 conn(0x7fb0b00c1aa0 0x7fb0b00c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b4b7e640 1 -- 172.21.15.94:0/45162235 >> v2:172.21.15.145:3300/0 conn(0x7fb0b00c1aa0 msgr2=0x7fb0b00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b4b7e640 1 --2- 172.21.15.94:0/45162235 >> v2:172.21.15.145:3300/0 conn(0x7fb0b00c1aa0 0x7fb0b00c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b4b7e640 1 -- 172.21.15.94:0/45162235 >> v1:172.21.15.94:6789/0 conn(0x7fb0b00c5c60 legacy=0x7fb0b0155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0af7fe640 1 -- 172.21.15.94:0/45162235 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420032222 0 0) 0x7fb0b00c28d0 con 0x7fb0b00c5c60 2024-03-06T07:29:15.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.536+0000 7fb0b4b7e640 1 -- 172.21.15.94:0/45162235 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b00c2e90 con 0x7fb0b014f790 2024-03-06T07:29:15.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.537+0000 7fb0b4b7e640 1 --2- 172.21.15.94:0/45162235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b01529d0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0070280 tx=0x7fb0a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ba37bd5e5d2ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.537+0000 7fb0af7fe640 1 -- 172.21.15.94:0/45162235 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4035020 con 0x7fb0b014f790 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.537+0000 7fb0af7fe640 1 -- 172.21.15.94:0/45162235 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4031830 con 0x7fb0b014f790 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.537+0000 7fb0af7fe640 1 -- 172.21.15.94:0/45162235 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4031c00 con 0x7fb0b014f790 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.537+0000 7fb0b6de2640 1 -- 172.21.15.94:0/45162235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 msgr2=0x7fb0b01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.537+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/45162235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b01529d0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0070280 tx=0x7fb0a402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.538+0000 7fb0b6de2640 1 -- 172.21.15.94:0/45162235 shutdown_connections 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.538+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/45162235 >> v2:172.21.15.145:3300/0 conn(0x7fb0b00c1aa0 0x7fb0b00c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.538+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/45162235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b01529d0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.538+0000 7fb0b6de2640 1 -- 172.21.15.94:0/45162235 >> 172.21.15.94:0/45162235 conn(0x7fb0b00c6710 msgr2=0x7fb0b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.538+0000 7fb0b6de2640 1 -- 172.21.15.94:0/45162235 shutdown_connections 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.538+0000 7fb0b6de2640 1 -- 172.21.15.94:0/45162235 wait complete. 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.539+0000 7fb0b6de2640 1 Processor -- start 2024-03-06T07:29:15.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.539+0000 7fb0b6de2640 1 -- start start 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0b00c1aa0 0x7fb0b014c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0b00c5c60 0x7fb0b01507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b0149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b006cc00 con 0x7fb0b014f790 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b00c0d60 con 0x7fb0b00c5c60 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b006cd80 con 0x7fb0b00c1aa0 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b0149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b0149910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39558/0 (socket says 172.21.15.94:39558) 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 -- 172.21.15.94:0/752040606 learned_addr learned my addr 172.21.15.94:0/752040606 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0affff640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0b00c5c60 0x7fb0b01507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b4b7e640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0b00c1aa0 0x7fb0b014c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 -- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0b00c1aa0 msgr2=0x7fb0b014c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0b00c1aa0 0x7fb0b014c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 -- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0b00c5c60 msgr2=0x7fb0b01507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0b00c5c60 0x7fb0b01507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 -- 172.21.15.94:0/752040606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0070510 con 0x7fb0b014f790 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b537f640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b0149910 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800c7d0 tx=0x7fb0a800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0ad7fa640 1 -- 172.21.15.94:0/752040606 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8018070 con 0x7fb0b014f790 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0ad7fa640 1 -- 172.21.15.94:0/752040606 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a800a640 con 0x7fb0b014f790 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0ad7fa640 1 -- 172.21.15.94:0/752040606 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a800aa30 con 0x7fb0b014f790 2024-03-06T07:29:15.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b0151600 con 0x7fb0b014f790 2024-03-06T07:29:15.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.540+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b0151bc0 con 0x7fb0b014f790 2024-03-06T07:29:15.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.541+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b004f330 con 0x7fb0b014f790 2024-03-06T07:29:15.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.542+0000 7fb0ad7fa640 1 -- 172.21.15.94:0/752040606 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0a801b070 con 0x7fb0b014f790 2024-03-06T07:29:15.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.542+0000 7fb0ad7fa640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09406b4d0 0x7fb09406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.542+0000 7fb0ad7fa640 1 -- 172.21.15.94:0/752040606 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb0a8089a40 con 0x7fb0b014f790 2024-03-06T07:29:15.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.543+0000 7fb0b4b7e640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09406b4d0 0x7fb09406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.546 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.543+0000 7fb0b4b7e640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09406b4d0 0x7fb09406d990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4036000 tx=0x7fb0a403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.547+0000 7fb0ad7fa640 1 -- 172.21.15.94:0/752040606 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0a8057f50 con 0x7fb0b014f790 2024-03-06T07:29:15.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.753+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0b00c1f40 con 0x7fb0b014f790 2024-03-06T07:29:15.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.754+0000 7fb0ad7fa640 1 -- 172.21.15.94:0/752040606 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fb0a8089e20 con 0x7fb0b014f790 2024-03-06T07:29:15.756 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:15.756 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:15.756 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09406b4d0 msgr2=0x7fb09406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09406b4d0 0x7fb09406d990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4036000 tx=0x7fb0a403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 msgr2=0x7fb0b0149910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b0149910 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800c7d0 tx=0x7fb0a800cc90 comp rx=0 tx=0).stop 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 shutdown_connections 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb09406b4d0 0x7fb09406d990 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb0b014f790 0x7fb0b0149910 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.758 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0b00c5c60 0x7fb0b01507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 --2- 172.21.15.94:0/752040606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0b00c1aa0 0x7fb0b014c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 >> 172.21.15.94:0/752040606 conn(0x7fb0b00c6710 msgr2=0x7fb0b006d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 shutdown_connections 2024-03-06T07:29:15.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:15.756+0000 7fb0b6de2640 1 -- 172.21.15.94:0/752040606 wait complete. 2024-03-06T07:29:16.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:16.171 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:16.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:16.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:17.172 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:17.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.282+0000 7fcca4a39640 1 Processor -- start 2024-03-06T07:29:17.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.282+0000 7fcca4a39640 1 -- start start 2024-03-06T07:29:17.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcca4a39640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcca00c1aa0 0x7fcca01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcca4a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca0155a40 0x7fcca0155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcca4a39640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca01563d0 con 0x7fcca014f790 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcca4a39640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca00c2680 con 0x7fcca00c1aa0 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcca4a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca00c28d0 con 0x7fcca0155a40 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcc9e59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcca014f790 0x7fcca00c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47954/0 (socket says 172.21.15.94:47954) 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcc9e59c640 1 -- 172.21.15.94:0/148299866 learned_addr learned my addr 172.21.15.94:0/148299866 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcc9ed9d640 1 --2- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca0155a40 0x7fcca0155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcc9dd9b640 1 --2- 172.21.15.94:0/148299866 >> v2:172.21.15.145:3300/0 conn(0x7fcca00c1aa0 0x7fcca01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcc9d59a640 1 -- 172.21.15.94:0/148299866 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658455638 0 0) 0x7fcca01563d0 con 0x7fcca014f790 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.283+0000 7fcc9d59a640 1 -- 172.21.15.94:0/148299866 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc88003520 con 0x7fcca014f790 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.284+0000 7fcc9dd9b640 1 -- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca0155a40 msgr2=0x7fcca0155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.284+0000 7fcc9dd9b640 1 --2- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca0155a40 0x7fcca0155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.284+0000 7fcc9dd9b640 1 -- 172.21.15.94:0/148299866 >> v1:172.21.15.94:6789/0 conn(0x7fcca014f790 legacy=0x7fcca00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.287 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.284+0000 7fcc9dd9b640 1 -- 172.21.15.94:0/148299866 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca00c2bb0 con 0x7fcca00c1aa0 2024-03-06T07:29:17.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.284+0000 7fcc9ed9d640 1 --2- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca0155a40 0x7fcca0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.284+0000 7fcc9dd9b640 1 --2- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca00c1aa0 0x7fcca01529d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcc940097a0 tx=0x7fcc9402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce71cac7e7ade40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcc9d59a640 1 -- 172.21.15.94:0/148299866 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94035020 con 0x7fcca00c1aa0 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcc9d59a640 1 -- 172.21.15.94:0/148299866 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94031830 con 0x7fcca00c1aa0 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcc9d59a640 1 -- 172.21.15.94:0/148299866 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94031be0 con 0x7fcca00c1aa0 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 -- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca00c1aa0 msgr2=0x7fcca01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 --2- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca00c1aa0 0x7fcca01529d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcc940097a0 tx=0x7fcc9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 -- 172.21.15.94:0/148299866 shutdown_connections 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 --2- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca0155a40 0x7fcca0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 --2- 172.21.15.94:0/148299866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca00c1aa0 0x7fcca01529d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 -- 172.21.15.94:0/148299866 >> 172.21.15.94:0/148299866 conn(0x7fcca00c6710 msgr2=0x7fcca00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 -- 172.21.15.94:0/148299866 shutdown_connections 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.285+0000 7fcca4a39640 1 -- 172.21.15.94:0/148299866 wait complete. 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 Processor -- start 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 -- start start 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca00c1aa0 0x7fcca0148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca014f790 0x7fcca01ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca0155a40 0x7fcca0154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca00c0d60 con 0x7fcca014f790 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca0155520 con 0x7fcca0155a40 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcca4a39640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca00c5e30 con 0x7fcca00c1aa0 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcc9dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca014f790 0x7fcca01ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcc9e59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca00c1aa0 0x7fcca0148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcc9dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca014f790 0x7fcca01ea3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39564/0 (socket says 172.21.15.94:39564) 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcc9dd9b640 1 -- 172.21.15.94:0/2973508185 learned_addr learned my addr 172.21.15.94:0/2973508185 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.287+0000 7fcc9ed9d640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca0155a40 0x7fcca0154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc9dd9b640 1 -- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca00c1aa0 msgr2=0x7fcca0148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc9dd9b640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca00c1aa0 0x7fcca0148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc9dd9b640 1 -- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca0155a40 msgr2=0x7fcca0154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc9dd9b640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca0155a40 0x7fcca0154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc9dd9b640 1 -- 172.21.15.94:0/2973508185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca0150750 con 0x7fcca014f790 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc9dd9b640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca014f790 0x7fcca01ea3d0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fcc9402bee0 tx=0x7fcc940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc8effd640 1 -- 172.21.15.94:0/2973508185 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94035040 con 0x7fcca014f790 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc8effd640 1 -- 172.21.15.94:0/2973508185 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9403b7d0 con 0x7fcca014f790 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcc8effd640 1 -- 172.21.15.94:0/2973508185 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9403bbc0 con 0x7fcca014f790 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca0150a30 con 0x7fcca014f790 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.288+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca00709b0 con 0x7fcca014f790 2024-03-06T07:29:17.293 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.289+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca0152760 con 0x7fcca014f790 2024-03-06T07:29:17.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.290+0000 7fcc8effd640 1 -- 172.21.15.94:0/2973508185 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc94038020 con 0x7fcca014f790 2024-03-06T07:29:17.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.290+0000 7fcc8effd640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.290+0000 7fcc8effd640 1 -- 172.21.15.94:0/2973508185 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcc940adb20 con 0x7fcca014f790 2024-03-06T07:29:17.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.290+0000 7fcc9e59c640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.291+0000 7fcc9e59c640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcc90005880 tx=0x7fcc90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.294+0000 7fcc8effd640 1 -- 172.21.15.94:0/2973508185 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcca0152760 con 0x7fcca014f790 2024-03-06T07:29:17.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.507+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcca004f330 con 0x7fcca014f790 2024-03-06T07:29:17.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.508+0000 7fcc8effd640 1 -- 172.21.15.94:0/2973508185 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fcc9407c7a0 con 0x7fcca014f790 2024-03-06T07:29:17.510 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:17.510 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:17.510 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc8006b4d0 msgr2=0x7fcc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcc90005880 tx=0x7fcc90009040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca014f790 msgr2=0x7fcca01ea3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca014f790 0x7fcca01ea3d0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fcc9402bee0 tx=0x7fcc940046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 shutdown_connections 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca0155a40 0x7fcca0154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.512 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcca014f790 0x7fcca01ea3d0 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 --2- 172.21.15.94:0/2973508185 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca00c1aa0 0x7fcca0148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 >> 172.21.15.94:0/2973508185 conn(0x7fcca00c6710 msgr2=0x7fcca00c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 shutdown_connections 2024-03-06T07:29:17.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:17.510+0000 7fcca4a39640 1 -- 172.21.15.94:0/2973508185 wait complete. 2024-03-06T07:29:17.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:17.926 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:18.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.037+0000 7f4bda684640 1 Processor -- start 2024-03-06T07:29:19.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.037+0000 7f4bda684640 1 -- start start 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bda684640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4bd41592d0 0x7f4bd41552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bda684640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd40ac260 0x7f4bd4155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bda684640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd4152d40 con 0x7f4bd4152960 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bda684640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd4066e30 con 0x7f4bd41592d0 2024-03-06T07:29:19.041 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bda684640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd4155d50 con 0x7f4bd40ac260 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bd3fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4bd4152960 0x7f4bd406dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45730/0 (socket says 172.21.15.94:45730) 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bd3fff640 1 -- 172.21.15.94:0/1648310487 learned_addr learned my addr 172.21.15.94:0/1648310487 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bd37fe640 1 --2- 172.21.15.94:0/1648310487 >> v2:172.21.15.145:3300/0 conn(0x7f4bd41592d0 0x7f4bd41552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bd8c21640 1 --2- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd40ac260 0x7f4bd4155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.038+0000 7f4bd2ffd640 1 -- 172.21.15.94:0/1648310487 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904385425 0 0) 0x7f4bd4152d40 con 0x7f4bd4152960 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.039+0000 7f4bd2ffd640 1 -- 172.21.15.94:0/1648310487 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bbc003520 con 0x7f4bd4152960 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.039+0000 7f4bd37fe640 1 -- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd40ac260 msgr2=0x7f4bd4155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.039+0000 7f4bd37fe640 1 --2- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd40ac260 0x7f4bd4155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.039+0000 7f4bd37fe640 1 -- 172.21.15.94:0/1648310487 >> v1:172.21.15.94:6789/0 conn(0x7f4bd4152960 legacy=0x7f4bd406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.039+0000 7f4bd37fe640 1 -- 172.21.15.94:0/1648310487 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd40c1b50 con 0x7f4bd41592d0 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.039+0000 7f4bd37fe640 1 --2- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41592d0 0x7f4bd41552d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80097a0 tx=0x7f4bc802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e5c5006e3a51498 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bd2ffd640 1 -- 172.21.15.94:0/1648310487 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8035020 con 0x7f4bd41592d0 2024-03-06T07:29:19.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bd2ffd640 1 -- 172.21.15.94:0/1648310487 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc8031830 con 0x7f4bd41592d0 2024-03-06T07:29:19.043 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bd2ffd640 1 -- 172.21.15.94:0/1648310487 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8031be0 con 0x7f4bd41592d0 2024-03-06T07:29:19.045 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 -- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41592d0 msgr2=0x7f4bd41552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 --2- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41592d0 0x7f4bd41552d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80097a0 tx=0x7f4bc802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 -- 172.21.15.94:0/1648310487 shutdown_connections 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 --2- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd40ac260 0x7f4bd4155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 --2- 172.21.15.94:0/1648310487 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41592d0 0x7f4bd41552d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 -- 172.21.15.94:0/1648310487 >> 172.21.15.94:0/1648310487 conn(0x7f4bd40c70b0 msgr2=0x7f4bd40c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 -- 172.21.15.94:0/1648310487 shutdown_connections 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.040+0000 7f4bda684640 1 -- 172.21.15.94:0/1648310487 wait complete. 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.041+0000 7f4bda684640 1 Processor -- start 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.041+0000 7f4bda684640 1 -- start start 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bda684640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd40ac260 0x7f4bd4154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bda684640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd4152960 0x7f4bd41eae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bda684640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41deeb0 0x7f4bd41eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.046 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bda684640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd4159760 con 0x7f4bd40ac260 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bda684640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd40672c0 con 0x7f4bd41deeb0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bda684640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd406e3a0 con 0x7f4bd4152960 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bd3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd40ac260 0x7f4bd4154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bd8c21640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41deeb0 0x7f4bd41eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bd3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd40ac260 0x7f4bd4154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46620/0 (socket says 172.21.15.94:46620) 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bd37fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd4152960 0x7f4bd41eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.042+0000 7f4bd3fff640 1 -- 172.21.15.94:0/109594492 learned_addr learned my addr 172.21.15.94:0/109594492 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd3fff640 1 -- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd4152960 msgr2=0x7f4bd41eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd3fff640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd4152960 0x7f4bd41eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd3fff640 1 -- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41deeb0 msgr2=0x7f4bd41eb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd3fff640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41deeb0 0x7f4bd41eb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd3fff640 1 -- 172.21.15.94:0/109594492 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd41df470 con 0x7f4bd40ac260 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd8c21640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41deeb0 0x7f4bd41eb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd3fff640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd40ac260 0x7f4bd4154530 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c460 tx=0x7f4bc400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd0ff9640 1 -- 172.21.15.94:0/109594492 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4017070 con 0x7f4bd40ac260 2024-03-06T07:29:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd0ff9640 1 -- 172.21.15.94:0/109594492 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc4005ce0 con 0x7f4bd40ac260 2024-03-06T07:29:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd41eef50 con 0x7f4bd40ac260 2024-03-06T07:29:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bd0ff9640 1 -- 172.21.15.94:0/109594492 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4007290 con 0x7f4bd40ac260 2024-03-06T07:29:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.043+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd41ef230 con 0x7f4bd40ac260 2024-03-06T07:29:19.048 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.044+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd404f330 con 0x7f4bd40ac260 2024-03-06T07:29:19.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.044+0000 7f4bd0ff9640 1 -- 172.21.15.94:0/109594492 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bc401a070 con 0x7f4bd40ac260 2024-03-06T07:29:19.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.045+0000 7f4bd0ff9640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bc006b4d0 0x7f4bc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.045+0000 7f4bd0ff9640 1 -- 172.21.15.94:0/109594492 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f4bc4089730 con 0x7f4bd40ac260 2024-03-06T07:29:19.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.045+0000 7f4bd37fe640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bc006b4d0 0x7f4bc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.049 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.045+0000 7f4bd37fe640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bc006b4d0 0x7f4bc006d990 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8036000 tx=0x7f4bc803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.051 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.049+0000 7f4bd0ff9640 1 -- 172.21.15.94:0/109594492 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bc4057ce0 con 0x7f4bd40ac260 2024-03-06T07:29:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.257+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bd4155b30 con 0x7f4bd40ac260 2024-03-06T07:29:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.257+0000 7f4bd0ff9640 1 -- 172.21.15.94:0/109594492 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f4bc405bd80 con 0x7f4bd40ac260 2024-03-06T07:29:19.259 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:19.259 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:19.259 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bc006b4d0 msgr2=0x7f4bc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bc006b4d0 0x7f4bc006d990 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8036000 tx=0x7f4bc803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd40ac260 msgr2=0x7f4bd4154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd40ac260 0x7f4bd4154530 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c460 tx=0x7f4bc400c920 comp rx=0 tx=0).stop 2024-03-06T07:29:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 shutdown_connections 2024-03-06T07:29:19.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4bc006b4d0 0x7f4bc006d990 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bd41deeb0 0x7f4bd41eb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bd4152960 0x7f4bd41eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 --2- 172.21.15.94:0/109594492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bd40ac260 0x7f4bd4154530 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 >> 172.21.15.94:0/109594492 conn(0x7f4bd40c70b0 msgr2=0x7f4bd40aa140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 shutdown_connections 2024-03-06T07:29:19.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:19.259+0000 7f4bda684640 1 -- 172.21.15.94:0/109594492 wait complete. 2024-03-06T07:29:19.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:19.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:20.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:20.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.784+0000 7f53ec713640 1 Processor -- start 2024-03-06T07:29:20.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.784+0000 7f53ec713640 1 -- start start 2024-03-06T07:29:20.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53ec713640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53ec713640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53e4155a40 0x7f53e4155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53ec713640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e4156410 con 0x7f53e40c1aa0 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53ec713640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e40c2680 con 0x7f53e4155a40 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53ec713640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40c28d0 con 0x7f53e414f790 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53eacb0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53e4155a40 0x7f53e4155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53ea4af640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53e9cae640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f53e40c1aa0 0x7f53e40c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:54812/0 (socket says 172.21.15.94:54812) 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.785+0000 7f53e9cae640 1 -- 172.21.15.94:0/3094172053 learned_addr learned my addr 172.21.15.94:0/3094172053 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.786+0000 7f53eacb0640 1 -- 172.21.15.94:0/3094172053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 msgr2=0x7f53e41529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.786+0000 7f53eacb0640 1 --2- 172.21.15.94:0/3094172053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e41529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.786+0000 7f53eacb0640 1 -- 172.21.15.94:0/3094172053 >> v1:172.21.15.145:6789/0 conn(0x7f53e40c1aa0 legacy=0x7f53e40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.786+0000 7f53eacb0640 1 -- 172.21.15.94:0/3094172053 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e40c2bb0 con 0x7f53e4155a40 2024-03-06T07:29:20.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.786+0000 7f53e94ad640 1 -- 172.21.15.94:0/3094172053 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124002264 0 0) 0x7f53e4156410 con 0x7f53e40c1aa0 2024-03-06T07:29:20.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.786+0000 7f53eacb0640 1 --2- 172.21.15.94:0/3094172053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e4155ed0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009d90 tx=0x7f53dc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ac52a4a01a36aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53e94ad640 1 -- 172.21.15.94:0/3094172053 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc039070 con 0x7f53e4155a40 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53e94ad640 1 -- 172.21.15.94:0/3094172053 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc02cc50 con 0x7f53e4155a40 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53e94ad640 1 -- 172.21.15.94:0/3094172053 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc034440 con 0x7f53e4155a40 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 -- 172.21.15.94:0/3094172053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 msgr2=0x7f53e4155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 --2- 172.21.15.94:0/3094172053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e4155ed0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009d90 tx=0x7f53dc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 -- 172.21.15.94:0/3094172053 shutdown_connections 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 --2- 172.21.15.94:0/3094172053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e4155ed0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 --2- 172.21.15.94:0/3094172053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e41529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 -- 172.21.15.94:0/3094172053 >> 172.21.15.94:0/3094172053 conn(0x7f53e40c6710 msgr2=0x7f53e40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 -- 172.21.15.94:0/3094172053 shutdown_connections 2024-03-06T07:29:20.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.787+0000 7f53ec713640 1 -- 172.21.15.94:0/3094172053 wait complete. 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.788+0000 7f53ec713640 1 Processor -- start 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.788+0000 7f53ec713640 1 -- start start 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ec713640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53e40c1aa0 0x7f53e41df440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ec713640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e414cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ec713640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ec713640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40c0d60 con 0x7f53e40c1aa0 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ec713640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e4155520 con 0x7f53e4155a40 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ec713640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e40c5e30 con 0x7f53e414f790 2024-03-06T07:29:20.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ea4af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53e40c1aa0 0x7f53e41df440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ea4af640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53e40c1aa0 0x7f53e41df440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46628/0 (socket says 172.21.15.94:46628) 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53eacb0640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e414d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ea4af640 1 -- 172.21.15.94:0/3698555013 learned_addr learned my addr 172.21.15.94:0/3698555013 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53e9cae640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e414cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ea4af640 1 -- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 msgr2=0x7f53e414cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ea4af640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e414cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ea4af640 1 -- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 msgr2=0x7f53e414d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.789+0000 7f53ea4af640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e414d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53ea4af640 1 -- 172.21.15.94:0/3698555013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e414d560 con 0x7f53e40c1aa0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53eacb0640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53ea4af640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53e40c1aa0 0x7f53e41df440 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f53e000e9e0 tx=0x7f53e000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53daffd640 1 -- 172.21.15.94:0/3698555013 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0017070 con 0x7f53e40c1aa0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53daffd640 1 -- 172.21.15.94:0/3698555013 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e000c9d0 con 0x7f53e40c1aa0 2024-03-06T07:29:20.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e4072420 con 0x7f53e40c1aa0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53daffd640 1 -- 172.21.15.94:0/3698555013 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e000cdc0 con 0x7f53e40c1aa0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.790+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e40720b0 con 0x7f53e40c1aa0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.791+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e404f330 con 0x7f53e40c1aa0 2024-03-06T07:29:20.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.791+0000 7f53daffd640 1 -- 172.21.15.94:0/3698555013 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53e001a070 con 0x7f53e40c1aa0 2024-03-06T07:29:20.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.792+0000 7f53daffd640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f53c806b500 0x7f53c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.792+0000 7f53daffd640 1 -- 172.21.15.94:0/3698555013 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f53e008a790 con 0x7f53e40c1aa0 2024-03-06T07:29:20.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.792+0000 7f53e9cae640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f53c806b500 0x7f53c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.792+0000 7f53e9cae640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f53c806b500 0x7f53c806d9c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f53e4071e20 tx=0x7f53d4007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.798 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:20.796+0000 7f53daffd640 1 -- 172.21.15.94:0/3698555013 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53e0058d40 con 0x7f53e40c1aa0 2024-03-06T07:29:21.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.002+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53e41e9830 con 0x7f53e40c1aa0 2024-03-06T07:29:21.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.003+0000 7f53daffd640 1 -- 172.21.15.94:0/3698555013 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f53e005cde0 con 0x7f53e40c1aa0 2024-03-06T07:29:21.005 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:21.005 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:21.005 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f53c806b500 msgr2=0x7f53c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f53c806b500 0x7f53c806d9c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f53e4071e20 tx=0x7f53d4007d50 comp rx=0 tx=0).stop 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53e40c1aa0 msgr2=0x7f53e41df440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53e40c1aa0 0x7f53e41df440 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f53e000e9e0 tx=0x7f53e000eea0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 shutdown_connections 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f53c806b500 0x7f53c806d9c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53e4155a40 0x7f53e414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53e414f790 0x7f53e414cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 --2- 172.21.15.94:0/3698555013 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53e40c1aa0 0x7f53e41df440 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 >> 172.21.15.94:0/3698555013 conn(0x7f53e40c6710 msgr2=0x7f53e40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 shutdown_connections 2024-03-06T07:29:21.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:21.005+0000 7f53ec713640 1 -- 172.21.15.94:0/3698555013 wait complete. 2024-03-06T07:29:21.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:21.419 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:22.420 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.531+0000 7f961a6cf640 1 Processor -- start 2024-03-06T07:29:22.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.531+0000 7f961a6cf640 1 -- start start 2024-03-06T07:29:22.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f961a6cf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9614150580 0x7f961406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f961a6cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96140c6000 0x7f961406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f961a6cf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9614153a60 con 0x7f96140c1320 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f961a6cf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9614153d40 con 0x7f9614150580 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f961a6cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961406c2c0 con 0x7f96140c6000 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f96137fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f96140c1320 0x7f96141566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45742/0 (socket says 172.21.15.94:45742) 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f96137fe640 1 -- 172.21.15.94:0/1898739364 learned_addr learned my addr 172.21.15.94:0/1898739364 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f9618c6c640 1 --2- 172.21.15.94:0/1898739364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96140c6000 0x7f961406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f9613fff640 1 --2- 172.21.15.94:0/1898739364 >> v2:172.21.15.145:3300/0 conn(0x7f9614150580 0x7f961406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f9612ffd640 1 -- 172.21.15.94:0/1898739364 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823148152 0 0) 0x7f9614153a60 con 0x7f96140c1320 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f9612ffd640 1 -- 172.21.15.94:0/1898739364 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc003520 con 0x7f96140c1320 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f9618c6c640 1 -- 172.21.15.94:0/1898739364 >> v2:172.21.15.145:3300/0 conn(0x7f9614150580 msgr2=0x7f961406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.532+0000 7f9618c6c640 1 --2- 172.21.15.94:0/1898739364 >> v2:172.21.15.145:3300/0 conn(0x7f9614150580 0x7f961406dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.533+0000 7f9618c6c640 1 -- 172.21.15.94:0/1898739364 >> v1:172.21.15.94:6789/0 conn(0x7f96140c1320 legacy=0x7f96141566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.533+0000 7f9618c6c640 1 -- 172.21.15.94:0/1898739364 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961406c5a0 con 0x7f96140c6000 2024-03-06T07:29:22.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.533+0000 7f9613fff640 1 --2- 172.21.15.94:0/1898739364 >> v2:172.21.15.145:3300/0 conn(0x7f9614150580 0x7f961406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.533+0000 7f9618c6c640 1 --2- 172.21.15.94:0/1898739364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96140c6000 0x7f961406e250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f960c009d90 tx=0x7f960c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e05a3d51077fcaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.533+0000 7f9612ffd640 1 -- 172.21.15.94:0/1898739364 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c039070 con 0x7f96140c6000 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f9612ffd640 1 -- 172.21.15.94:0/1898739364 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c02cbc0 con 0x7f96140c6000 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f9612ffd640 1 -- 172.21.15.94:0/1898739364 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c034440 con 0x7f96140c6000 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 -- 172.21.15.94:0/1898739364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96140c6000 msgr2=0x7f961406e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 --2- 172.21.15.94:0/1898739364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96140c6000 0x7f961406e250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f960c009d90 tx=0x7f960c02be10 comp rx=0 tx=0).stop 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 -- 172.21.15.94:0/1898739364 shutdown_connections 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 --2- 172.21.15.94:0/1898739364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96140c6000 0x7f961406e250 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 --2- 172.21.15.94:0/1898739364 >> v2:172.21.15.145:3300/0 conn(0x7f9614150580 0x7f961406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 -- 172.21.15.94:0/1898739364 >> 172.21.15.94:0/1898739364 conn(0x7f96140c6710 msgr2=0x7f96140c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 -- 172.21.15.94:0/1898739364 shutdown_connections 2024-03-06T07:29:22.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.534+0000 7f961a6cf640 1 -- 172.21.15.94:0/1898739364 wait complete. 2024-03-06T07:29:22.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.535+0000 7f961a6cf640 1 Processor -- start 2024-03-06T07:29:22.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.535+0000 7f961a6cf640 1 -- start start 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f961a6cf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96140c1320 0x7f96140c2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f961a6cf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96140c6000 0x7f9614144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f961a6cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9614150580 0x7f96141450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f961a6cf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140c0dc0 con 0x7f96140c1320 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f961a6cf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140c0c40 con 0x7f96140c6000 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f961a6cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9614156dc0 con 0x7f9614150580 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f9613fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96140c1320 0x7f96140c2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f9613fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96140c1320 0x7f96140c2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46640/0 (socket says 172.21.15.94:46640) 2024-03-06T07:29:22.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f9613fff640 1 -- 172.21.15.94:0/140183168 learned_addr learned my addr 172.21.15.94:0/140183168 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f96137fe640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96140c6000 0x7f9614144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f9618c6c640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9614150580 0x7f96141450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.536+0000 7f9613fff640 1 -- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9614150580 msgr2=0x7f96141450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9613fff640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9614150580 0x7f96141450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9613fff640 1 -- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96140c6000 msgr2=0x7f9614144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9613fff640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96140c6000 0x7f9614144520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9613fff640 1 -- 172.21.15.94:0/140183168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96141eb920 con 0x7f96140c1320 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9613fff640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96140c1320 0x7f96140c2e10 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f960400e990 tx=0x7f960400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9610ff9640 1 -- 172.21.15.94:0/140183168 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604017070 con 0x7f96140c1320 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9610ff9640 1 -- 172.21.15.94:0/140183168 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960400c9e0 con 0x7f96140c1320 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96141ebbc0 con 0x7f96140c1320 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f9610ff9640 1 -- 172.21.15.94:0/140183168 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960400cdb0 con 0x7f96140c1320 2024-03-06T07:29:22.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.537+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96140c3350 con 0x7f96140c1320 2024-03-06T07:29:22.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.538+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961404f330 con 0x7f96140c1320 2024-03-06T07:29:22.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.538+0000 7f9610ff9640 1 -- 172.21.15.94:0/140183168 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f960401a070 con 0x7f96140c1320 2024-03-06T07:29:22.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.539+0000 7f9610ff9640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f960006b500 0x7f960006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.539+0000 7f9610ff9640 1 -- 172.21.15.94:0/140183168 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f960408a580 con 0x7f96140c1320 2024-03-06T07:29:22.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.539+0000 7f96137fe640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f960006b500 0x7f960006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.539+0000 7f96137fe640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f960006b500 0x7f960006d9c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f961414e010 tx=0x7f9608009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.543+0000 7f9610ff9640 1 -- 172.21.15.94:0/140183168 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9604058bb0 con 0x7f96140c1320 2024-03-06T07:29:22.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.750+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96140c5de0 con 0x7f96140c1320 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.751+0000 7f9610ff9640 1 -- 172.21.15.94:0/140183168 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f960405cc50 con 0x7f96140c1320 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:22.753 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f960006b500 msgr2=0x7f960006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f960006b500 0x7f960006d9c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f961414e010 tx=0x7f9608009040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96140c1320 msgr2=0x7f96140c2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96140c1320 0x7f96140c2e10 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f960400e990 tx=0x7f960400ee50 comp rx=0 tx=0).stop 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 shutdown_connections 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f960006b500 0x7f960006d9c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9614150580 0x7f96141450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96140c6000 0x7f9614144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 --2- 172.21.15.94:0/140183168 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f96140c1320 0x7f96140c2e10 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 >> 172.21.15.94:0/140183168 conn(0x7f96140c6710 msgr2=0x7f96140c4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 shutdown_connections 2024-03-06T07:29:22.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:22.753+0000 7f961a6cf640 1 -- 172.21.15.94:0/140183168 wait complete. 2024-03-06T07:29:23.167 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:23.168 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:24.168 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:24.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.278+0000 7fd1a095e640 1 Processor -- start 2024-03-06T07:29:24.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.278+0000 7fd1a095e640 1 -- start start 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.279+0000 7fd1a095e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c0c88c0 0x7fd19c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.279+0000 7fd1a095e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19c0c73d0 con 0x7fd19c14f360 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.279+0000 7fd1a095e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19c0be2e0 con 0x7fd19c158b60 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.279+0000 7fd1a095e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c159700 con 0x7fd19c0c88c0 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.279+0000 7fd19a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd19c14f360 0x7fd19c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45744/0 (socket says 172.21.15.94:45744) 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.279+0000 7fd19a59c640 1 -- 172.21.15.94:0/2106971100 learned_addr learned my addr 172.21.15.94:0/2106971100 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd199d9b640 1 --2- 172.21.15.94:0/2106971100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c0c88c0 0x7fd19c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd19959a640 1 -- 172.21.15.94:0/2106971100 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441633272 0 0) 0x7fd19c0c73d0 con 0x7fd19c14f360 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd19959a640 1 -- 172.21.15.94:0/2106971100 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184003520 con 0x7fd19c14f360 2024-03-06T07:29:24.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd19959a640 1 -- 172.21.15.94:0/2106971100 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638115880 0 0) 0x7fd19c0be2e0 con 0x7fd19c158b60 2024-03-06T07:29:24.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd19959a640 1 -- 172.21.15.94:0/2106971100 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19c0c73d0 con 0x7fd19c158b60 2024-03-06T07:29:24.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd199d9b640 1 -- 172.21.15.94:0/2106971100 >> v1:172.21.15.145:6789/0 conn(0x7fd19c158b60 legacy=0x7fd19c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd199d9b640 1 -- 172.21.15.94:0/2106971100 >> v1:172.21.15.94:6789/0 conn(0x7fd19c14f360 legacy=0x7fd19c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.280+0000 7fd199d9b640 1 -- 172.21.15.94:0/2106971100 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c0c7670 con 0x7fd19c0c88c0 2024-03-06T07:29:24.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd199d9b640 1 --2- 172.21.15.94:0/2106971100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c0c88c0 0x7fd19c155c10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd1900097d0 tx=0x7fd19002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bba8b61b941377d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd19959a640 1 -- 172.21.15.94:0/2106971100 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190035020 con 0x7fd19c0c88c0 2024-03-06T07:29:24.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd19959a640 1 -- 172.21.15.94:0/2106971100 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd190031800 con 0x7fd19c0c88c0 2024-03-06T07:29:24.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd19959a640 1 -- 172.21.15.94:0/2106971100 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd190031bd0 con 0x7fd19c0c88c0 2024-03-06T07:29:24.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd1a095e640 1 -- 172.21.15.94:0/2106971100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c0c88c0 msgr2=0x7fd19c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd1a095e640 1 --2- 172.21.15.94:0/2106971100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c0c88c0 0x7fd19c155c10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd1900097d0 tx=0x7fd19002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd1a095e640 1 -- 172.21.15.94:0/2106971100 shutdown_connections 2024-03-06T07:29:24.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd1a095e640 1 --2- 172.21.15.94:0/2106971100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c0c88c0 0x7fd19c155c10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.281+0000 7fd1a095e640 1 -- 172.21.15.94:0/2106971100 >> 172.21.15.94:0/2106971100 conn(0x7fd19c0ac570 msgr2=0x7fd19c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.282+0000 7fd1a095e640 1 -- 172.21.15.94:0/2106971100 shutdown_connections 2024-03-06T07:29:24.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.282+0000 7fd1a095e640 1 -- 172.21.15.94:0/2106971100 wait complete. 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 Processor -- start 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 -- start start 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19c0c88c0 0x7fd19c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c14f360 0x7fd19c0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd19c158b60 0x7fd19c144d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c0c7c30 con 0x7fd19c158b60 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c0c5a50 con 0x7fd19c0c88c0 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.283+0000 7fd1a095e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c0c5bd0 con 0x7fd19c14f360 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd19c158b60 0x7fd19c144d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd19c158b60 0x7fd19c144d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46656/0 (socket says 172.21.15.94:46656) 2024-03-06T07:29:24.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19a59c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19c0c88c0 0x7fd19c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 -- 172.21.15.94:0/1057090179 learned_addr learned my addr 172.21.15.94:0/1057090179 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd199d9b640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c14f360 0x7fd19c0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 -- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c14f360 msgr2=0x7fd19c0ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c14f360 0x7fd19c0ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 -- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19c0c88c0 msgr2=0x7fd19c1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19c0c88c0 0x7fd19c1573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 -- 172.21.15.94:0/1057090179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c0b0470 con 0x7fd19c158b60 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd19ad9d640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd19c158b60 0x7fd19c144d30 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fd19400c400 tx=0x7fd19400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd18affd640 1 -- 172.21.15.94:0/1057090179 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd194017070 con 0x7fd19c158b60 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd18affd640 1 -- 172.21.15.94:0/1057090179 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1940052d0 con 0x7fd19c158b60 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19c0ae590 con 0x7fd19c158b60 2024-03-06T07:29:24.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd18affd640 1 -- 172.21.15.94:0/1057090179 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1940037f0 con 0x7fd19c158b60 2024-03-06T07:29:24.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.284+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c157920 con 0x7fd19c158b60 2024-03-06T07:29:24.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.285+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19c069460 con 0x7fd19c158b60 2024-03-06T07:29:24.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.286+0000 7fd18affd640 1 -- 172.21.15.94:0/1057090179 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd19401a070 con 0x7fd19c158b60 2024-03-06T07:29:24.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.286+0000 7fd18affd640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd17c06b4d0 0x7fd17c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.286+0000 7fd18affd640 1 -- 172.21.15.94:0/1057090179 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd194089770 con 0x7fd19c158b60 2024-03-06T07:29:24.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.286+0000 7fd19a59c640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd17c06b4d0 0x7fd17c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.287+0000 7fd19a59c640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd17c06b4d0 0x7fd17c06d990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0012e0 tx=0x7fd18c001ab0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.292 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.290+0000 7fd18affd640 1 -- 172.21.15.94:0/1057090179 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd19c069460 con 0x7fd19c158b60 2024-03-06T07:29:24.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.500+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd19c04f330 con 0x7fd19c158b60 2024-03-06T07:29:24.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.500+0000 7fd18affd640 1 -- 172.21.15.94:0/1057090179 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fd194057d20 con 0x7fd19c158b60 2024-03-06T07:29:24.502 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:24.503 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:24.503 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:24.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd17c06b4d0 msgr2=0x7fd17c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd17c06b4d0 0x7fd17c06d990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0012e0 tx=0x7fd18c001ab0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd19c158b60 msgr2=0x7fd19c144d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd19c158b60 0x7fd19c144d30 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fd19400c400 tx=0x7fd19400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 shutdown_connections 2024-03-06T07:29:24.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd17c06b4d0 0x7fd17c06d990 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd19c158b60 0x7fd19c144d30 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd19c14f360 0x7fd19c0ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 --2- 172.21.15.94:0/1057090179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19c0c88c0 0x7fd19c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 >> 172.21.15.94:0/1057090179 conn(0x7fd19c0ac570 msgr2=0x7fd19c0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 shutdown_connections 2024-03-06T07:29:24.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:24.502+0000 7fd1a095e640 1 -- 172.21.15.94:0/1057090179 wait complete. 2024-03-06T07:29:24.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:24.918 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:25.919 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:26.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.029+0000 7f88bf6df640 1 Processor -- start 2024-03-06T07:29:26.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.029+0000 7f88bf6df640 1 -- start start 2024-03-06T07:29:26.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bf6df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b8156150 0x7f88b81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bf6df640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b80c28d0 con 0x7f88b814f790 2024-03-06T07:29:26.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bf6df640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b80c2bb0 con 0x7f88b80c1aa0 2024-03-06T07:29:26.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bf6df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80c2680 con 0x7f88b8156150 2024-03-06T07:29:26.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bd47b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f88b814f790 0x7f88b80c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45754/0 (socket says 172.21.15.94:45754) 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bd47b640 1 -- 172.21.15.94:0/3523692254 learned_addr learned my addr 172.21.15.94:0/3523692254 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bdc7c640 1 --2- 172.21.15.94:0/3523692254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b8156150 0x7f88b81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88affff640 1 -- 172.21.15.94:0/3523692254 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245698817 0 0) 0x7f88b80c28d0 con 0x7f88b814f790 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88affff640 1 -- 172.21.15.94:0/3523692254 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a4003520 con 0x7f88b814f790 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88affff640 1 -- 172.21.15.94:0/3523692254 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895236006 0 0) 0x7f88b80c2bb0 con 0x7f88b80c1aa0 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88affff640 1 -- 172.21.15.94:0/3523692254 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b80c28d0 con 0x7f88b80c1aa0 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bdc7c640 1 -- 172.21.15.94:0/3523692254 >> v1:172.21.15.145:6789/0 conn(0x7f88b80c1aa0 legacy=0x7f88b8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bdc7c640 1 -- 172.21.15.94:0/3523692254 >> v1:172.21.15.94:6789/0 conn(0x7f88b814f790 legacy=0x7f88b80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.030+0000 7f88bdc7c640 1 -- 172.21.15.94:0/3523692254 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b80c2e90 con 0x7f88b8156150 2024-03-06T07:29:26.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.031+0000 7f88bdc7c640 1 --2- 172.21.15.94:0/3523692254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b8156150 0x7f88b81529d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009dc0 tx=0x7f88b002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=333ff2f509d051b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.031+0000 7f88affff640 1 -- 172.21.15.94:0/3523692254 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0039070 con 0x7f88b8156150 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.031+0000 7f88affff640 1 -- 172.21.15.94:0/3523692254 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b002ccd0 con 0x7f88b8156150 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.031+0000 7f88affff640 1 -- 172.21.15.94:0/3523692254 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0034640 con 0x7f88b8156150 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.032+0000 7f88bf6df640 1 -- 172.21.15.94:0/3523692254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b8156150 msgr2=0x7f88b81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.032+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3523692254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b8156150 0x7f88b81529d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009dc0 tx=0x7f88b002bf20 comp rx=0 tx=0).stop 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.032+0000 7f88bf6df640 1 -- 172.21.15.94:0/3523692254 shutdown_connections 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.032+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3523692254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b8156150 0x7f88b81529d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.032+0000 7f88bf6df640 1 -- 172.21.15.94:0/3523692254 >> 172.21.15.94:0/3523692254 conn(0x7f88b80c6710 msgr2=0x7f88b80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.032+0000 7f88bf6df640 1 -- 172.21.15.94:0/3523692254 shutdown_connections 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.032+0000 7f88bf6df640 1 -- 172.21.15.94:0/3523692254 wait complete. 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.033+0000 7f88bf6df640 1 Processor -- start 2024-03-06T07:29:26.035 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.033+0000 7f88bf6df640 1 -- start start 2024-03-06T07:29:26.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bf6df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b80c1aa0 0x7f88b80719b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bf6df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88b814f790 0x7f88b80a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bf6df640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88b8156150 0x7f88b814d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bf6df640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80c5e30 con 0x7f88b8156150 2024-03-06T07:29:26.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bf6df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80c0d60 con 0x7f88b814f790 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bf6df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80c5fb0 con 0x7f88b80c1aa0 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88b8156150 0x7f88b814d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88b8156150 0x7f88b814d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46664/0 (socket says 172.21.15.94:46664) 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bd47b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b80c1aa0 0x7f88b80719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 -- 172.21.15.94:0/3852156342 learned_addr learned my addr 172.21.15.94:0/3852156342 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bcc7a640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88b814f790 0x7f88b80a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 -- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b80c1aa0 msgr2=0x7f88b80719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b80c1aa0 0x7f88b80719b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 -- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88b814f790 msgr2=0x7f88b80a97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88b814f790 0x7f88b80a97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 -- 172.21.15.94:0/3852156342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b814e4f0 con 0x7f88b8156150 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bd47b640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b80c1aa0 0x7f88b80719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bdc7c640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88b8156150 0x7f88b814d1c0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f88b002c530 tx=0x7f88b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88adffb640 1 -- 172.21.15.94:0/3852156342 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0039050 con 0x7f88b8156150 2024-03-06T07:29:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88adffb640 1 -- 172.21.15.94:0/3852156342 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b0045070 con 0x7f88b8156150 2024-03-06T07:29:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b814e7d0 con 0x7f88b8156150 2024-03-06T07:29:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.034+0000 7f88adffb640 1 -- 172.21.15.94:0/3852156342 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0009450 con 0x7f88b8156150 2024-03-06T07:29:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.035+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b80a9d00 con 0x7f88b8156150 2024-03-06T07:29:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.036+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b804f330 con 0x7f88b8156150 2024-03-06T07:29:26.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.036+0000 7f88adffb640 1 -- 172.21.15.94:0/3852156342 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88b003e850 con 0x7f88b8156150 2024-03-06T07:29:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.037+0000 7f88adffb640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f889c06b4d0 0x7f889c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.037+0000 7f88adffb640 1 -- 172.21.15.94:0/3852156342 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f88b00ad2e0 con 0x7f88b8156150 2024-03-06T07:29:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.037+0000 7f88bd47b640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f889c06b4d0 0x7f889c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.037+0000 7f88bd47b640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f889c06b4d0 0x7f889c06d990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f88b4000f60 tx=0x7f88b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.042 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.041+0000 7f88adffb640 1 -- 172.21.15.94:0/3852156342 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88b007ba70 con 0x7f88b8156150 2024-03-06T07:29:26.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.251+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b80c0b40 con 0x7f88b8156150 2024-03-06T07:29:26.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.251+0000 7f88adffb640 1 -- 172.21.15.94:0/3852156342 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f88b007fb10 con 0x7f88b8156150 2024-03-06T07:29:26.253 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:26.253 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:26.254 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f889c06b4d0 msgr2=0x7f889c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f889c06b4d0 0x7f889c06d990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f88b4000f60 tx=0x7f88b4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88b8156150 msgr2=0x7f88b814d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88b8156150 0x7f88b814d1c0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f88b002c530 tx=0x7f88b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 shutdown_connections 2024-03-06T07:29:26.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f889c06b4d0 0x7f889c06d990 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88b8156150 0x7f88b814d1c0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88b814f790 0x7f88b80a97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 --2- 172.21.15.94:0/3852156342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88b80c1aa0 0x7f88b80719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 >> 172.21.15.94:0/3852156342 conn(0x7f88b80c6710 msgr2=0x7f88b80c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 shutdown_connections 2024-03-06T07:29:26.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:26.253+0000 7f88bf6df640 1 -- 172.21.15.94:0/3852156342 wait complete. 2024-03-06T07:29:26.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:26.668 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:27.668 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:27.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.778+0000 7fe09a1c9640 1 Processor -- start 2024-03-06T07:29:27.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.778+0000 7fe09a1c9640 1 -- start start 2024-03-06T07:29:27.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe09a1c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe0941592d0 0x7fe0941552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe09a1c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe094152d40 con 0x7fe0940ac260 2024-03-06T07:29:27.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe09a1c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0940c1b50 con 0x7fe094152960 2024-03-06T07:29:27.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe09a1c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe094066e30 con 0x7fe0941592d0 2024-03-06T07:29:27.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe092ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe0941592d0 0x7fe0941552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe093fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe0940ac260 0x7fe094155810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45766/0 (socket says 172.21.15.94:45766) 2024-03-06T07:29:27.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe093fff640 1 -- 172.21.15.94:0/999116943 learned_addr learned my addr 172.21.15.94:0/999116943 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:27.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe092ffd640 1 -- 172.21.15.94:0/999116943 >> v1:172.21.15.145:6789/0 conn(0x7fe094152960 legacy=0x7fe09406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe092ffd640 1 -- 172.21.15.94:0/999116943 >> v1:172.21.15.94:6789/0 conn(0x7fe0940ac260 legacy=0x7fe094155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe0927fc640 1 -- 172.21.15.94:0/999116943 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140503232 0 0) 0x7fe094152d40 con 0x7fe0940ac260 2024-03-06T07:29:27.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.779+0000 7fe092ffd640 1 -- 172.21.15.94:0/999116943 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940c1dc0 con 0x7fe0941592d0 2024-03-06T07:29:27.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.780+0000 7fe092ffd640 1 --2- 172.21.15.94:0/999116943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0941592d0 0x7fe0941552d0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fe088009720 tx=0x7fe08802b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4333587b331fd015 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.780+0000 7fe0927fc640 1 -- 172.21.15.94:0/999116943 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088037070 con 0x7fe0941592d0 2024-03-06T07:29:27.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.780+0000 7fe0927fc640 1 -- 172.21.15.94:0/999116943 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe088030710 con 0x7fe0941592d0 2024-03-06T07:29:27.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.780+0000 7fe0927fc640 1 -- 172.21.15.94:0/999116943 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088030ae0 con 0x7fe0941592d0 2024-03-06T07:29:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.781+0000 7fe09a1c9640 1 -- 172.21.15.94:0/999116943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0941592d0 msgr2=0x7fe0941552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.781+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/999116943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0941592d0 0x7fe0941552d0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fe088009720 tx=0x7fe08802b730 comp rx=0 tx=0).stop 2024-03-06T07:29:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.781+0000 7fe09a1c9640 1 -- 172.21.15.94:0/999116943 shutdown_connections 2024-03-06T07:29:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.781+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/999116943 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0941592d0 0x7fe0941552d0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.781+0000 7fe09a1c9640 1 -- 172.21.15.94:0/999116943 >> 172.21.15.94:0/999116943 conn(0x7fe0940c70b0 msgr2=0x7fe0940c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.781+0000 7fe09a1c9640 1 -- 172.21.15.94:0/999116943 shutdown_connections 2024-03-06T07:29:27.784 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.781+0000 7fe09a1c9640 1 -- 172.21.15.94:0/999116943 wait complete. 2024-03-06T07:29:27.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.782+0000 7fe09a1c9640 1 Processor -- start 2024-03-06T07:29:27.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.782+0000 7fe09a1c9640 1 -- start start 2024-03-06T07:29:27.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe09a1c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0940ac260 0x7fe0941deba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe09a1c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe094152960 0x7fe0941408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe09a1c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe09414ece0 0x7fe094140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe09a1c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09406e3a0 con 0x7fe09414ece0 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe09a1c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094159760 con 0x7fe094152960 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe09a1c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09406e520 con 0x7fe0940ac260 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe09414ece0 0x7fe094140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe092ffd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe094152960 0x7fe0941408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe09414ece0 0x7fe094140e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:46678/0 (socket says 172.21.15.94:46678) 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 -- 172.21.15.94:0/2394059880 learned_addr learned my addr 172.21.15.94:0/2394059880 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe0937fe640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0940ac260 0x7fe0941deba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 -- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0940ac260 msgr2=0x7fe0941deba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0940ac260 0x7fe0941deba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 -- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe094152960 msgr2=0x7fe0941408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe094152960 0x7fe0941408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe093fff640 1 -- 172.21.15.94:0/2394059880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0941eb9a0 con 0x7fe09414ece0 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.783+0000 7fe092ffd640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe094152960 0x7fe0941408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.784+0000 7fe093fff640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe09414ece0 0x7fe094140e30 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c7e0 tx=0x7fe08c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.784+0000 7fe07bfff640 1 -- 172.21.15.94:0/2394059880 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c018070 con 0x7fe09414ece0 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.784+0000 7fe07bfff640 1 -- 172.21.15.94:0/2394059880 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c00a5e0 con 0x7fe09414ece0 2024-03-06T07:29:27.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.784+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0941ebc80 con 0x7fe09414ece0 2024-03-06T07:29:27.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.784+0000 7fe07bfff640 1 -- 172.21.15.94:0/2394059880 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00a9d0 con 0x7fe09414ece0 2024-03-06T07:29:27.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.784+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe094153650 con 0x7fe09414ece0 2024-03-06T07:29:27.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.785+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09404f330 con 0x7fe09414ece0 2024-03-06T07:29:27.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.785+0000 7fe07bfff640 1 -- 172.21.15.94:0/2394059880 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe08c01b070 con 0x7fe09414ece0 2024-03-06T07:29:27.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.786+0000 7fe07bfff640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe08006b4d0 0x7fe08006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.786+0000 7fe07bfff640 1 -- 172.21.15.94:0/2394059880 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe08c089900 con 0x7fe09414ece0 2024-03-06T07:29:27.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.786+0000 7fe0937fe640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe08006b4d0 0x7fe08006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.786+0000 7fe0937fe640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe08006b4d0 0x7fe08006d990 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe084000f60 tx=0x7fe084009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:27.790+0000 7fe07bfff640 1 -- 172.21.15.94:0/2394059880 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe08c057f30 con 0x7fe09414ece0 2024-03-06T07:29:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.001+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe09406ebc0 con 0x7fe09414ece0 2024-03-06T07:29:28.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.002+0000 7fe07bfff640 1 -- 172.21.15.94:0/2394059880 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fe08c05bfd0 con 0x7fe09414ece0 2024-03-06T07:29:28.004 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:28.005 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:28.005 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe08006b4d0 msgr2=0x7fe08006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe08006b4d0 0x7fe08006d990 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe084000f60 tx=0x7fe084009040 comp rx=0 tx=0).stop 2024-03-06T07:29:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe09414ece0 msgr2=0x7fe094140e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe09414ece0 0x7fe094140e30 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c7e0 tx=0x7fe08c00cca0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 shutdown_connections 2024-03-06T07:29:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fe08006b4d0 0x7fe08006d990 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe09414ece0 0x7fe094140e30 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe094152960 0x7fe0941408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 --2- 172.21.15.94:0/2394059880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0940ac260 0x7fe0941deba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 >> 172.21.15.94:0/2394059880 conn(0x7fe0940c70b0 msgr2=0x7fe09406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 shutdown_connections 2024-03-06T07:29:28.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:28.004+0000 7fe09a1c9640 1 -- 172.21.15.94:0/2394059880 wait complete. 2024-03-06T07:29:28.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:28.419 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:29.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:29.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.531+0000 7f54e21d0640 1 Processor -- start 2024-03-06T07:29:29.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.531+0000 7f54e21d0640 1 -- start start 2024-03-06T07:29:29.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54e21d0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54e21d0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0c28d0 con 0x7f54dc14f790 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54e21d0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0c2bb0 con 0x7f54dc155a40 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54e21d0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c2680 con 0x7f54dc0c1aa0 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54db7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f54dc14f790 0x7f54dc0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41048/0 (socket says 172.21.15.94:41048) 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54db7fe640 1 -- 172.21.15.94:0/4229950059 learned_addr learned my addr 172.21.15.94:0/4229950059 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54daffd640 1 --2- 172.21.15.94:0/4229950059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54da7fc640 1 -- 172.21.15.94:0/4229950059 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416754861 0 0) 0x7f54dc0c28d0 con 0x7f54dc14f790 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54da7fc640 1 -- 172.21.15.94:0/4229950059 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c4003520 con 0x7f54dc14f790 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54da7fc640 1 -- 172.21.15.94:0/4229950059 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244673694 0 0) 0x7f54dc0c2bb0 con 0x7f54dc155a40 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.532+0000 7f54da7fc640 1 -- 172.21.15.94:0/4229950059 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54dc0c28d0 con 0x7f54dc155a40 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.533+0000 7f54daffd640 1 -- 172.21.15.94:0/4229950059 >> v1:172.21.15.145:6789/0 conn(0x7f54dc155a40 legacy=0x7f54dc155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.533+0000 7f54daffd640 1 -- 172.21.15.94:0/4229950059 >> v1:172.21.15.94:6789/0 conn(0x7f54dc14f790 legacy=0x7f54dc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.533+0000 7f54daffd640 1 -- 172.21.15.94:0/4229950059 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0c2e90 con 0x7f54dc0c1aa0 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.533+0000 7f54daffd640 1 --2- 172.21.15.94:0/4229950059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc1529d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f54d00097d0 tx=0x7f54d002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ace207a5a34c21e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.533+0000 7f54da7fc640 1 -- 172.21.15.94:0/4229950059 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0035020 con 0x7f54dc0c1aa0 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54da7fc640 1 -- 172.21.15.94:0/4229950059 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d0031890 con 0x7f54dc0c1aa0 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54da7fc640 1 -- 172.21.15.94:0/4229950059 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0031c40 con 0x7f54dc0c1aa0 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54e21d0640 1 -- 172.21.15.94:0/4229950059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 msgr2=0x7f54dc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54e21d0640 1 --2- 172.21.15.94:0/4229950059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc1529d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f54d00097d0 tx=0x7f54d002b930 comp rx=0 tx=0).stop 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54e21d0640 1 -- 172.21.15.94:0/4229950059 shutdown_connections 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54e21d0640 1 --2- 172.21.15.94:0/4229950059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc1529d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54e21d0640 1 -- 172.21.15.94:0/4229950059 >> 172.21.15.94:0/4229950059 conn(0x7f54dc0c6710 msgr2=0x7f54dc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54e21d0640 1 -- 172.21.15.94:0/4229950059 shutdown_connections 2024-03-06T07:29:29.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.534+0000 7f54e21d0640 1 -- 172.21.15.94:0/4229950059 wait complete. 2024-03-06T07:29:29.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.535+0000 7f54e21d0640 1 Processor -- start 2024-03-06T07:29:29.538 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.535+0000 7f54e21d0640 1 -- start start 2024-03-06T07:29:29.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54e21d0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc14c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54e21d0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54dc14f790 0x7f54dc14c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54e21d0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54dc155a40 0x7f54dc145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54e21d0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c5e30 con 0x7f54dc14f790 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54e21d0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c0d60 con 0x7f54dc155a40 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54e21d0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c5fb0 con 0x7f54dc0c1aa0 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54daffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54dc14f790 0x7f54dc14c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54daffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54dc14f790 0x7f54dc14c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39856/0 (socket says 172.21.15.94:39856) 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54daffd640 1 -- 172.21.15.94:0/3481329332 learned_addr learned my addr 172.21.15.94:0/3481329332 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54db7fe640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc14c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.536+0000 7f54dbfff640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54dc155a40 0x7f54dc145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54daffd640 1 -- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 msgr2=0x7f54dc14c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54daffd640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc14c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54daffd640 1 -- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54dc155a40 msgr2=0x7f54dc145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54daffd640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54dc155a40 0x7f54dc145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54daffd640 1 -- 172.21.15.94:0/3481329332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc1e9b80 con 0x7f54dc14f790 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54daffd640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54dc14f790 0x7f54dc14c830 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f54d00098f0 tx=0x7f54d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54c3fff640 1 -- 172.21.15.94:0/3481329332 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0035040 con 0x7f54dc14f790 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54c3fff640 1 -- 172.21.15.94:0/3481329332 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d0030b10 con 0x7f54dc14f790 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc149a00 con 0x7f54dc14f790 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54c3fff640 1 -- 172.21.15.94:0/3481329332 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0044430 con 0x7f54dc14f790 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.537+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc1e9870 con 0x7f54dc14f790 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.538+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc04f330 con 0x7f54dc14f790 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.538+0000 7f54c3fff640 1 -- 172.21.15.94:0/3481329332 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54d0038020 con 0x7f54dc14f790 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.539+0000 7f54c3fff640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f54c806b4d0 0x7f54c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.539+0000 7f54c3fff640 1 -- 172.21.15.94:0/3481329332 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f54d00ac870 con 0x7f54dc14f790 2024-03-06T07:29:29.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.539+0000 7f54db7fe640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f54c806b4d0 0x7f54c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.540+0000 7f54db7fe640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f54c806b4d0 0x7f54c806d990 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f54cc000f60 tx=0x7f54cc007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.545 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.543+0000 7f54c3fff640 1 -- 172.21.15.94:0/3481329332 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54d002d040 con 0x7f54dc14f790 2024-03-06T07:29:29.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.752+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54dc0c1010 con 0x7f54dc14f790 2024-03-06T07:29:29.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.752+0000 7f54c3fff640 1 -- 172.21.15.94:0/3481329332 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f54d003de20 con 0x7f54dc14f790 2024-03-06T07:29:29.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:29.754 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:29.754 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:29.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f54c806b4d0 msgr2=0x7f54c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f54c806b4d0 0x7f54c806d990 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f54cc000f60 tx=0x7f54cc007d70 comp rx=0 tx=0).stop 2024-03-06T07:29:29.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54dc14f790 msgr2=0x7f54dc14c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54dc14f790 0x7f54dc14c830 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f54d00098f0 tx=0x7f54d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 shutdown_connections 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f54c806b4d0 0x7f54c806d990 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54dc155a40 0x7f54dc145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54dc14f790 0x7f54dc14c830 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 --2- 172.21.15.94:0/3481329332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54dc0c1aa0 0x7f54dc14c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 >> 172.21.15.94:0/3481329332 conn(0x7f54dc0c6710 msgr2=0x7f54dc0a9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 shutdown_connections 2024-03-06T07:29:29.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:29.754+0000 7f54e21d0640 1 -- 172.21.15.94:0/3481329332 wait complete. 2024-03-06T07:29:30.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:30.169 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:31.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:31.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.280+0000 7fd39b82a640 1 Processor -- start 2024-03-06T07:29:31.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.280+0000 7fd39b82a640 1 -- start start 2024-03-06T07:29:31.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd39b82a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd3940c1320 0x7fd3941566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd39b82a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd394153a60 con 0x7fd3940c6000 2024-03-06T07:29:31.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd39b82a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd394153d40 con 0x7fd394150580 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd39b82a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39406c2c0 con 0x7fd3940c1320 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd399dc7640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd3940c6000 0x7fd394156bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41056/0 (socket says 172.21.15.94:41056) 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd399dc7640 1 -- 172.21.15.94:0/2156031315 learned_addr learned my addr 172.21.15.94:0/2156031315 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd398dc5640 1 --2- 172.21.15.94:0/2156031315 >> v2:172.21.15.145:3300/0 conn(0x7fd3940c1320 0x7fd3941566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd38bfff640 1 -- 172.21.15.94:0/2156031315 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792846326 0 0) 0x7fd394153a60 con 0x7fd3940c6000 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd38bfff640 1 -- 172.21.15.94:0/2156031315 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd380003520 con 0x7fd3940c6000 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd38bfff640 1 -- 172.21.15.94:0/2156031315 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708702663 0 0) 0x7fd394153d40 con 0x7fd394150580 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.281+0000 7fd38bfff640 1 -- 172.21.15.94:0/2156031315 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd394153a60 con 0x7fd394150580 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.282+0000 7fd398dc5640 1 -- 172.21.15.94:0/2156031315 >> v1:172.21.15.145:6789/0 conn(0x7fd394150580 legacy=0x7fd39406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.282+0000 7fd398dc5640 1 -- 172.21.15.94:0/2156031315 >> v1:172.21.15.94:6789/0 conn(0x7fd3940c6000 legacy=0x7fd394156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.282+0000 7fd398dc5640 1 -- 172.21.15.94:0/2156031315 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39406c5a0 con 0x7fd3940c1320 2024-03-06T07:29:31.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.282+0000 7fd398dc5640 1 --2- 172.21.15.94:0/2156031315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3940c1320 0x7fd3941566b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd3840097a0 tx=0x7fd38402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=499482bc43d5506a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.282+0000 7fd38bfff640 1 -- 172.21.15.94:0/2156031315 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384035020 con 0x7fd3940c1320 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd38bfff640 1 -- 172.21.15.94:0/2156031315 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd384031830 con 0x7fd3940c1320 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd38bfff640 1 -- 172.21.15.94:0/2156031315 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384031c00 con 0x7fd3940c1320 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd39b82a640 1 -- 172.21.15.94:0/2156031315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3940c1320 msgr2=0x7fd3941566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2156031315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3940c1320 0x7fd3941566b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd3840097a0 tx=0x7fd38402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd39b82a640 1 -- 172.21.15.94:0/2156031315 shutdown_connections 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2156031315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3940c1320 0x7fd3941566b0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd39b82a640 1 -- 172.21.15.94:0/2156031315 >> 172.21.15.94:0/2156031315 conn(0x7fd3940c6710 msgr2=0x7fd3940c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd39b82a640 1 -- 172.21.15.94:0/2156031315 shutdown_connections 2024-03-06T07:29:31.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.283+0000 7fd39b82a640 1 -- 172.21.15.94:0/2156031315 wait complete. 2024-03-06T07:29:31.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.284+0000 7fd39b82a640 1 Processor -- start 2024-03-06T07:29:31.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.284+0000 7fd39b82a640 1 -- start start 2024-03-06T07:29:31.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd39b82a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3940c1320 0x7fd394149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd39b82a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3940c6000 0x7fd394149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.288 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd39b82a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd394150580 0x7fd3941545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd39b82a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39406e420 con 0x7fd3940c1320 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd39b82a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394150960 con 0x7fd394150580 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd39b82a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39406e5a0 con 0x7fd3940c6000 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3940c1320 0x7fd394149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3940c1320 0x7fd394149910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39866/0 (socket says 172.21.15.94:39866) 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 -- 172.21.15.94:0/2408397830 learned_addr learned my addr 172.21.15.94:0/2408397830 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd398dc5640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3940c6000 0x7fd394149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd399dc7640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd394150580 0x7fd3941545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 -- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3940c6000 msgr2=0x7fd394149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3940c6000 0x7fd394149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.289 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 -- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd394150580 msgr2=0x7fd3941545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd394150580 0x7fd3941545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.285+0000 7fd3995c6640 1 -- 172.21.15.94:0/2408397830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39414a390 con 0x7fd3940c1320 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.286+0000 7fd3995c6640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3940c1320 0x7fd394149910 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c460 tx=0x7fd39000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.286+0000 7fd389ffb640 1 -- 172.21.15.94:0/2408397830 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390017070 con 0x7fd3940c1320 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.286+0000 7fd389ffb640 1 -- 172.21.15.94:0/2408397830 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3900052f0 con 0x7fd3940c1320 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.286+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3940ab3a0 con 0x7fd3940c1320 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.286+0000 7fd389ffb640 1 -- 172.21.15.94:0/2408397830 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390003830 con 0x7fd3940c1320 2024-03-06T07:29:31.290 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.286+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3940ab660 con 0x7fd3940c1320 2024-03-06T07:29:31.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.287+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39406c7a0 con 0x7fd3940c1320 2024-03-06T07:29:31.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.288+0000 7fd389ffb640 1 -- 172.21.15.94:0/2408397830 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd39001a070 con 0x7fd3940c1320 2024-03-06T07:29:31.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.288+0000 7fd389ffb640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd37806b500 0x7fd37806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.288+0000 7fd389ffb640 1 -- 172.21.15.94:0/2408397830 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd390089770 con 0x7fd3940c1320 2024-03-06T07:29:31.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.288+0000 7fd398dc5640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd37806b500 0x7fd37806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.291 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.289+0000 7fd398dc5640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd37806b500 0x7fd37806d9c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fd384036000 tx=0x7fd384008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.294 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.292+0000 7fd389ffb640 1 -- 172.21.15.94:0/2408397830 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd39406c7a0 con 0x7fd3940c1320 2024-03-06T07:29:31.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.499+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd39404f330 con 0x7fd3940c1320 2024-03-06T07:29:31.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.500+0000 7fd389ffb640 1 -- 172.21.15.94:0/2408397830 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fd390057d10 con 0x7fd3940c1320 2024-03-06T07:29:31.502 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:31.502 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:31.502 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:31.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd37806b500 msgr2=0x7fd37806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd37806b500 0x7fd37806d9c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fd384036000 tx=0x7fd384008030 comp rx=0 tx=0).stop 2024-03-06T07:29:31.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3940c1320 msgr2=0x7fd394149910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3940c1320 0x7fd394149910 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c460 tx=0x7fd39000c920 comp rx=0 tx=0).stop 2024-03-06T07:29:31.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 shutdown_connections 2024-03-06T07:29:31.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd37806b500 0x7fd37806d9c0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd394150580 0x7fd3941545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3940c6000 0x7fd394149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 --2- 172.21.15.94:0/2408397830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3940c1320 0x7fd394149910 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 >> 172.21.15.94:0/2408397830 conn(0x7fd3940c6710 msgr2=0x7fd3940c03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 shutdown_connections 2024-03-06T07:29:31.505 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:31.502+0000 7fd39b82a640 1 -- 172.21.15.94:0/2408397830 wait complete. 2024-03-06T07:29:31.917 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:31.917 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:32.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:33.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.027+0000 7f4735e88640 1 Processor -- start 2024-03-06T07:29:33.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.027+0000 7f4735e88640 1 -- start start 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f4735e88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4730150580 0x7f473006dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f4735e88640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47300c1320 0x7f473006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f4735e88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4730153a60 con 0x7f47300c6000 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f4735e88640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4730153d40 con 0x7f47300c1320 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f4735e88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473006c2c0 con 0x7f4730150580 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47300c1320 0x7f473006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472effd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f47300c1320 0x7f473006e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39874/0 (socket says 172.21.15.94:39874) 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472effd640 1 -- 172.21.15.94:0/3058825752 learned_addr learned my addr 172.21.15.94:0/3058825752 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472f7fe640 1 --2- 172.21.15.94:0/3058825752 >> v2:172.21.15.145:3300/0 conn(0x7f4730150580 0x7f473006dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472effd640 1 -- 172.21.15.94:0/3058825752 >> v2:172.21.15.145:3300/0 conn(0x7f4730150580 msgr2=0x7f473006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472effd640 1 --2- 172.21.15.94:0/3058825752 >> v2:172.21.15.145:3300/0 conn(0x7f4730150580 0x7f473006dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472effd640 1 -- 172.21.15.94:0/3058825752 >> v1:172.21.15.145:6789/0 conn(0x7f47300c6000 legacy=0x7f47301566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.028+0000 7f472effd640 1 -- 172.21.15.94:0/3058825752 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473006c5a0 con 0x7f47300c1320 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.029+0000 7f472effd640 1 --2- 172.21.15.94:0/3058825752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f473006e250 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f47300a9550 tx=0x7f472402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e24e4ac57b02900 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.029+0000 7f472e7fc640 1 -- 172.21.15.94:0/3058825752 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724035020 con 0x7f47300c1320 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.029+0000 7f472e7fc640 1 -- 172.21.15.94:0/3058825752 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4724031870 con 0x7f47300c1320 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.029+0000 7f472e7fc640 1 -- 172.21.15.94:0/3058825752 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724031c40 con 0x7f47300c1320 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 -- 172.21.15.94:0/3058825752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 msgr2=0x7f473006e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 --2- 172.21.15.94:0/3058825752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f473006e250 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f47300a9550 tx=0x7f472402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 -- 172.21.15.94:0/3058825752 shutdown_connections 2024-03-06T07:29:33.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 --2- 172.21.15.94:0/3058825752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f473006e250 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 --2- 172.21.15.94:0/3058825752 >> v2:172.21.15.145:3300/0 conn(0x7f4730150580 0x7f473006dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 -- 172.21.15.94:0/3058825752 >> 172.21.15.94:0/3058825752 conn(0x7f47300c6710 msgr2=0x7f47300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 -- 172.21.15.94:0/3058825752 shutdown_connections 2024-03-06T07:29:33.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.030+0000 7f4735e88640 1 -- 172.21.15.94:0/3058825752 wait complete. 2024-03-06T07:29:33.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.031+0000 7f4735e88640 1 Processor -- start 2024-03-06T07:29:33.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.031+0000 7f4735e88640 1 -- start start 2024-03-06T07:29:33.037 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f4735e88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f47301511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f4735e88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47300c6000 0x7f4730154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f4735e88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4730150580 0x7f4730155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f4735e88640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47300c0dc0 con 0x7f47300c1320 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f4735e88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47300c0c40 con 0x7f47300c6000 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f4735e88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4730156dc0 con 0x7f4730150580 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f47301511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f47301511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39880/0 (socket says 172.21.15.94:39880) 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 -- 172.21.15.94:0/1265270634 learned_addr learned my addr 172.21.15.94:0/1265270634 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472effd640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47300c6000 0x7f4730154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472ffff640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4730150580 0x7f4730155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 -- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4730150580 msgr2=0x7f4730155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.038 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4730150580 0x7f4730155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 -- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47300c6000 msgr2=0x7f4730154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47300c6000 0x7f4730154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 -- 172.21.15.94:0/1265270634 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4730154e20 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.032+0000 7f472f7fe640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f47301511c0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f472000e870 tx=0x7f472000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.033+0000 7f4717fff640 1 -- 172.21.15.94:0/1265270634 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720017070 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.033+0000 7f4717fff640 1 -- 172.21.15.94:0/1265270634 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472000c940 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.033+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4730072fe0 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.033+0000 7f4717fff640 1 -- 172.21.15.94:0/1265270634 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472000cd30 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.033+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4730150ca0 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.034+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4730073500 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.034+0000 7f4717fff640 1 -- 172.21.15.94:0/1265270634 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f472001a070 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.034+0000 7f4717fff640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f471c06b4d0 0x7f471c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.035+0000 7f4717fff640 1 -- 172.21.15.94:0/1265270634 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f472008a5d0 con 0x7f47300c1320 2024-03-06T07:29:33.039 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.035+0000 7f472effd640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f471c06b4d0 0x7f471c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.035+0000 7f472effd640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f471c06b4d0 0x7f471c06d990 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4724036000 tx=0x7f472403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.040 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.039+0000 7f4717fff640 1 -- 172.21.15.94:0/1265270634 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4730073500 con 0x7f47300c1320 2024-03-06T07:29:33.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.248+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f473004f330 con 0x7f47300c1320 2024-03-06T07:29:33.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.248+0000 7f4717fff640 1 -- 172.21.15.94:0/1265270634 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f4720058c00 con 0x7f47300c1320 2024-03-06T07:29:33.250 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:33.250 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:33.250 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:33.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f471c06b4d0 msgr2=0x7f471c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.252 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f471c06b4d0 0x7f471c06d990 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4724036000 tx=0x7f472403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 msgr2=0x7f47301511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f47301511c0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f472000e870 tx=0x7f472000ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 shutdown_connections 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f471c06b4d0 0x7f471c06d990 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4730150580 0x7f4730155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47300c6000 0x7f4730154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 --2- 172.21.15.94:0/1265270634 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47300c1320 0x7f47301511c0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 >> 172.21.15.94:0/1265270634 conn(0x7f47300c6710 msgr2=0x7f47300c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 shutdown_connections 2024-03-06T07:29:33.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:33.250+0000 7f4735e88640 1 -- 172.21.15.94:0/1265270634 wait complete. 2024-03-06T07:29:33.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:33.665 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:34.666 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:34.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.775+0000 7f9a8f39a640 1 Processor -- start 2024-03-06T07:29:34.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.776+0000 7f9a8f39a640 1 -- start start 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.776+0000 7f9a8f39a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9a8814f360 0x7f9a88155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.776+0000 7f9a8f39a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a88158b60 0x7f9a88158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.776+0000 7f9a8f39a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880c73d0 con 0x7f9a880c88c0 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.776+0000 7f9a8f39a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a880c8dd0 con 0x7f9a8814f360 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.776+0000 7f9a8f39a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880be2e0 con 0x7f9a88158b60 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.776+0000 7f9a8d136640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9a8814f360 0x7f9a88155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a8d136640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9a8814f360 0x7f9a88155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39894/0 (socket says 172.21.15.94:39894) 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a8d136640 1 -- 172.21.15.94:0/2264402323 learned_addr learned my addr 172.21.15.94:0/2264402323 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:34.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a8d937640 1 --2- 172.21.15.94:0/2264402323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a88158b60 0x7f9a88158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a8d136640 1 -- 172.21.15.94:0/2264402323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a88158b60 msgr2=0x7f9a88158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a8d136640 1 --2- 172.21.15.94:0/2264402323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a88158b60 0x7f9a88158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a8d136640 1 -- 172.21.15.94:0/2264402323 >> v1:172.21.15.145:6789/0 conn(0x7f9a880c88c0 legacy=0x7f9a880c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a8d136640 1 -- 172.21.15.94:0/2264402323 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880c7670 con 0x7f9a8814f360 2024-03-06T07:29:34.780 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.777+0000 7f9a7ffff640 1 -- 172.21.15.94:0/2264402323 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666064388 0 0) 0x7f9a880c73d0 con 0x7f9a880c88c0 2024-03-06T07:29:34.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8d136640 1 --2- 172.21.15.94:0/2264402323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a8814f360 0x7f9a88155c10 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009e50 tx=0x7f9a8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5449d88629e217 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a7ffff640 1 -- 172.21.15.94:0/2264402323 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8402ca70 con 0x7f9a8814f360 2024-03-06T07:29:34.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a7ffff640 1 -- 172.21.15.94:0/2264402323 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8402cc10 con 0x7f9a8814f360 2024-03-06T07:29:34.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a7ffff640 1 -- 172.21.15.94:0/2264402323 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a840343f0 con 0x7f9a8814f360 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 -- 172.21.15.94:0/2264402323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a8814f360 msgr2=0x7f9a88155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/2264402323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a8814f360 0x7f9a88155c10 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009e50 tx=0x7f9a8402bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 -- 172.21.15.94:0/2264402323 shutdown_connections 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/2264402323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a88158b60 0x7f9a88158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/2264402323 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a8814f360 0x7f9a88155c10 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 -- 172.21.15.94:0/2264402323 >> 172.21.15.94:0/2264402323 conn(0x7f9a880ac570 msgr2=0x7f9a880ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 -- 172.21.15.94:0/2264402323 shutdown_connections 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.778+0000 7f9a8f39a640 1 -- 172.21.15.94:0/2264402323 wait complete. 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 Processor -- start 2024-03-06T07:29:34.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 -- start start 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a880c88c0 0x7f9a8814bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8814f360 0x7f9a881ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a88158b60 0x7f9a881ed930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880c5a50 con 0x7f9a88158b60 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880c7c30 con 0x7f9a8814f360 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8f39a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880c5bd0 con 0x7f9a880c88c0 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8d937640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a88158b60 0x7f9a881ed930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.785 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8d136640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a880c88c0 0x7f9a8814bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8d937640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a88158b60 0x7f9a881ed930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39902/0 (socket says 172.21.15.94:39902) 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8c935640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8814f360 0x7f9a881ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.780+0000 7f9a8d937640 1 -- 172.21.15.94:0/3412375805 learned_addr learned my addr 172.21.15.94:0/3412375805 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8d937640 1 -- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a880c88c0 msgr2=0x7f9a8814bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8d937640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a880c88c0 0x7f9a8814bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8d937640 1 -- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8814f360 msgr2=0x7f9a881ed3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8d937640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8814f360 0x7f9a881ed3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8d937640 1 -- 172.21.15.94:0/3412375805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880c0df0 con 0x7f9a88158b60 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8d937640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a88158b60 0x7f9a881ed930 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000e960 tx=0x7f9a8000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a7dffb640 1 -- 172.21.15.94:0/3412375805 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80017070 con 0x7f9a88158b60 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a7dffb640 1 -- 172.21.15.94:0/3412375805 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8000c980 con 0x7f9a88158b60 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a880c1390 con 0x7f9a88158b60 2024-03-06T07:29:34.786 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a7dffb640 1 -- 172.21.15.94:0/3412375805 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8000cd70 con 0x7f9a88158b60 2024-03-06T07:29:34.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.781+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a880c10b0 con 0x7f9a88158b60 2024-03-06T07:29:34.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.782+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8804f330 con 0x7f9a88158b60 2024-03-06T07:29:34.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.783+0000 7f9a7dffb640 1 -- 172.21.15.94:0/3412375805 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a8001a070 con 0x7f9a88158b60 2024-03-06T07:29:34.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.783+0000 7f9a7dffb640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9a6c06b500 0x7f9a6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.783+0000 7f9a7dffb640 1 -- 172.21.15.94:0/3412375805 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9a8008a400 con 0x7f9a88158b60 2024-03-06T07:29:34.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.783+0000 7f9a8d136640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9a6c06b500 0x7f9a6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.784+0000 7f9a8d136640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9a6c06b500 0x7f9a6c06d9c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f9a84035000 tx=0x7f9a84030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.787+0000 7f9a7dffb640 1 -- 172.21.15.94:0/3412375805 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a800589a0 con 0x7f9a88158b60 2024-03-06T07:29:34.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.995+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a88159660 con 0x7f9a88158b60 2024-03-06T07:29:34.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.995+0000 7f9a7dffb640 1 -- 172.21.15.94:0/3412375805 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f9a8005ca40 con 0x7f9a88158b60 2024-03-06T07:29:34.997 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:34.997 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:34.998 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9a6c06b500 msgr2=0x7f9a6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9a6c06b500 0x7f9a6c06d9c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f9a84035000 tx=0x7f9a84030000 comp rx=0 tx=0).stop 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a88158b60 msgr2=0x7f9a881ed930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a88158b60 0x7f9a881ed930 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000e960 tx=0x7f9a8000ee20 comp rx=0 tx=0).stop 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 shutdown_connections 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9a6c06b500 0x7f9a6c06d9c0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a88158b60 0x7f9a881ed930 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8814f360 0x7f9a881ed3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 --2- 172.21.15.94:0/3412375805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a880c88c0 0x7f9a8814bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 >> 172.21.15.94:0/3412375805 conn(0x7f9a880ac570 msgr2=0x7f9a8806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.997+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 shutdown_connections 2024-03-06T07:29:35.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:34.998+0000 7f9a8f39a640 1 -- 172.21.15.94:0/3412375805 wait complete. 2024-03-06T07:29:35.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:35.412 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:36.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:36.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.524+0000 7f164e1b5640 1 Processor -- start 2024-03-06T07:29:36.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.524+0000 7f164e1b5640 1 -- start start 2024-03-06T07:29:36.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f164e1b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f164806afc0 0x7f164806b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f164e1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16480c25c0 0x7f1648153890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f164e1b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1648153dd0 con 0x7f16480c20b0 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f164e1b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16480a9530 con 0x7f164806afc0 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f164e1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16480a9810 con 0x7f16480c25c0 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1647fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16480c25c0 0x7f1648153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f16477fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f164806afc0 0x7f164806b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1647fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16480c25c0 0x7f1648153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:35296/0 (socket says 172.21.15.94:35296) 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1647fff640 1 -- 172.21.15.94:0/967455771 learned_addr learned my addr 172.21.15.94:0/967455771 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1646ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f16480c20b0 0x7f1648153180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:48392/0 (socket says 172.21.15.94:48392) 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1647fff640 1 -- 172.21.15.94:0/967455771 >> v2:172.21.15.145:3300/0 conn(0x7f164806afc0 msgr2=0x7f164806b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1647fff640 1 --2- 172.21.15.94:0/967455771 >> v2:172.21.15.145:3300/0 conn(0x7f164806afc0 0x7f164806b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1647fff640 1 -- 172.21.15.94:0/967455771 >> v1:172.21.15.145:6789/0 conn(0x7f16480c20b0 legacy=0x7f1648153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f1647fff640 1 -- 172.21.15.94:0/967455771 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16480a9af0 con 0x7f16480c25c0 2024-03-06T07:29:36.529 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.525+0000 7f16467fc640 1 -- 172.21.15.94:0/967455771 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403400277 0 0) 0x7f1648153dd0 con 0x7f16480c20b0 2024-03-06T07:29:36.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.526+0000 7f1647fff640 1 --2- 172.21.15.94:0/967455771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16480c25c0 0x7f1648153890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1640009e50 tx=0x7f164002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50e23495224ccba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.526+0000 7f16467fc640 1 -- 172.21.15.94:0/967455771 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164002c9e0 con 0x7f16480c25c0 2024-03-06T07:29:36.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.526+0000 7f16467fc640 1 -- 172.21.15.94:0/967455771 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164002cb80 con 0x7f16480c25c0 2024-03-06T07:29:36.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.526+0000 7f16467fc640 1 -- 172.21.15.94:0/967455771 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16400343f0 con 0x7f16480c25c0 2024-03-06T07:29:36.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 -- 172.21.15.94:0/967455771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16480c25c0 msgr2=0x7f1648153890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 --2- 172.21.15.94:0/967455771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16480c25c0 0x7f1648153890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1640009e50 tx=0x7f164002be40 comp rx=0 tx=0).stop 2024-03-06T07:29:36.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 -- 172.21.15.94:0/967455771 shutdown_connections 2024-03-06T07:29:36.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 --2- 172.21.15.94:0/967455771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16480c25c0 0x7f1648153890 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 --2- 172.21.15.94:0/967455771 >> v2:172.21.15.145:3300/0 conn(0x7f164806afc0 0x7f164806b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 -- 172.21.15.94:0/967455771 >> 172.21.15.94:0/967455771 conn(0x7f16480c76b0 msgr2=0x7f16480c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 -- 172.21.15.94:0/967455771 shutdown_connections 2024-03-06T07:29:36.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.527+0000 7f164e1b5640 1 -- 172.21.15.94:0/967455771 wait complete. 2024-03-06T07:29:36.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.528+0000 7f164e1b5640 1 Processor -- start 2024-03-06T07:29:36.531 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.528+0000 7f164e1b5640 1 -- start start 2024-03-06T07:29:36.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f164e1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f164806afc0 0x7f1648154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f164e1b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16480c20b0 0x7f16480c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f164e1b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16480c25c0 0x7f16481ec0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f164e1b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1648150980 con 0x7f16480c20b0 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f164e1b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16480aa0b0 con 0x7f16480c25c0 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f164e1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1648150b00 con 0x7f164806afc0 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16480c20b0 0x7f16480c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16480c20b0 0x7f16480c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39904/0 (socket says 172.21.15.94:39904) 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 -- 172.21.15.94:0/2147153069 learned_addr learned my addr 172.21.15.94:0/2147153069 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1647fff640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16480c25c0 0x7f16481ec0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f16477fe640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f164806afc0 0x7f1648154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 -- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f164806afc0 msgr2=0x7f1648154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f164806afc0 0x7f1648154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 -- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16480c25c0 msgr2=0x7f16481ec0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16480c25c0 0x7f16481ec0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.529+0000 7f1646ffd640 1 -- 172.21.15.94:0/2147153069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648156210 con 0x7f16480c20b0 2024-03-06T07:29:36.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.530+0000 7f1646ffd640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16480c20b0 0x7f16480c5500 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c4e0 tx=0x7f163c00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.530+0000 7f162ffff640 1 -- 172.21.15.94:0/2147153069 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c018070 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.530+0000 7f162ffff640 1 -- 172.21.15.94:0/2147153069 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c007500 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.530+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16481564f0 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.530+0000 7f162ffff640 1 -- 172.21.15.94:0/2147153069 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c004a20 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.530+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1648156ab0 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.531+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164804f330 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.531+0000 7f162ffff640 1 -- 172.21.15.94:0/2147153069 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f163c01b070 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.531+0000 7f162ffff640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.532+0000 7f162ffff640 1 -- 172.21.15.94:0/2147153069 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f163c089760 con 0x7f16480c20b0 2024-03-06T07:29:36.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.532+0000 7f16477fe640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.532+0000 7f16477fe640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f163406b4d0 0x7f163406d990 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f16480c3fe0 tx=0x7f1638009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.537 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.536+0000 7f162ffff640 1 -- 172.21.15.94:0/2147153069 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f163c057dc0 con 0x7f16480c20b0 2024-03-06T07:29:36.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.742+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16480c1a50 con 0x7f16480c20b0 2024-03-06T07:29:36.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.743+0000 7f162ffff640 1 -- 172.21.15.94:0/2147153069 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f163c087e20 con 0x7f16480c20b0 2024-03-06T07:29:36.745 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:36.745 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:36.745 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f163406b4d0 msgr2=0x7f163406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f163406b4d0 0x7f163406d990 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f16480c3fe0 tx=0x7f1638009190 comp rx=0 tx=0).stop 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16480c20b0 msgr2=0x7f16480c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16480c20b0 0x7f16480c5500 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c4e0 tx=0x7f163c00c9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 shutdown_connections 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f163406b4d0 0x7f163406d990 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16480c25c0 0x7f16481ec0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16480c20b0 0x7f16480c5500 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 --2- 172.21.15.94:0/2147153069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f164806afc0 0x7f1648154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 >> 172.21.15.94:0/2147153069 conn(0x7f16480c76b0 msgr2=0x7f164806ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 shutdown_connections 2024-03-06T07:29:36.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:36.745+0000 7f164e1b5640 1 -- 172.21.15.94:0/2147153069 wait complete. 2024-03-06T07:29:37.160 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:37.160 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:38.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.272+0000 7ff141202640 1 Processor -- start 2024-03-06T07:29:38.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.272+0000 7ff141202640 1 -- start start 2024-03-06T07:29:38.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff141202640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff13c1527c0 0x7ff13c158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff141202640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c0c6fa0 0x7ff13c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff141202640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c0c5620 con 0x7ff13c06b500 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff141202640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13c06d450 con 0x7ff13c1527c0 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff141202640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c0c5900 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13b59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff13c06b500 0x7ff13c154fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41070/0 (socket says 172.21.15.94:41070) 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13b59e640 1 -- 172.21.15.94:0/3496268975 learned_addr learned my addr 172.21.15.94:0/3496268975 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13a59c640 1 --2- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c0c6fa0 0x7ff13c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13ad9d640 1 --2- 172.21.15.94:0/3496268975 >> v2:172.21.15.145:3300/0 conn(0x7ff13c1527c0 0x7ff13c158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff139d9b640 1 -- 172.21.15.94:0/3496268975 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770018632 0 0) 0x7ff13c0c5620 con 0x7ff13c06b500 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff139d9b640 1 -- 172.21.15.94:0/3496268975 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff128003520 con 0x7ff13c06b500 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13ad9d640 1 -- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c0c6fa0 msgr2=0x7ff13c06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13ad9d640 1 --2- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c0c6fa0 0x7ff13c06afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13ad9d640 1 -- 172.21.15.94:0/3496268975 >> v1:172.21.15.94:6789/0 conn(0x7ff13c06b500 legacy=0x7ff13c154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13ad9d640 1 -- 172.21.15.94:0/3496268975 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c06d0e0 con 0x7ff13c1527c0 2024-03-06T07:29:38.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.273+0000 7ff13a59c640 1 --2- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c0c6fa0 0x7ff13c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.274+0000 7ff13ad9d640 1 --2- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c158fb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff1240097a0 tx=0x7ff12402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3828a21e86b53538 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.274+0000 7ff139d9b640 1 -- 172.21.15.94:0/3496268975 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1240315a0 con 0x7ff13c1527c0 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.274+0000 7ff139d9b640 1 -- 172.21.15.94:0/3496268975 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff124037070 con 0x7ff13c1527c0 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.274+0000 7ff139d9b640 1 -- 172.21.15.94:0/3496268975 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124031960 con 0x7ff13c1527c0 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 -- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 msgr2=0x7ff13c158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 --2- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c158fb0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff1240097a0 tx=0x7ff12402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 -- 172.21.15.94:0/3496268975 shutdown_connections 2024-03-06T07:29:38.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 --2- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c0c6fa0 0x7ff13c06afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 --2- 172.21.15.94:0/3496268975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c158fb0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 -- 172.21.15.94:0/3496268975 >> 172.21.15.94:0/3496268975 conn(0x7ff13c0c4010 msgr2=0x7ff13c0c4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 -- 172.21.15.94:0/3496268975 shutdown_connections 2024-03-06T07:29:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.275+0000 7ff141202640 1 -- 172.21.15.94:0/3496268975 wait complete. 2024-03-06T07:29:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.276+0000 7ff141202640 1 Processor -- start 2024-03-06T07:29:38.279 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.276+0000 7ff141202640 1 -- start start 2024-03-06T07:29:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff141202640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c06b500 0x7ff13c1ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff141202640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff13c0c6fa0 0x7ff13c157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff141202640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c1509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff141202640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c155840 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff141202640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c1556c0 con 0x7ff13c1527c0 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff141202640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c1559c0 con 0x7ff13c06b500 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff13a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff13c0c6fa0 0x7ff13c157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff13a59c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff13c0c6fa0 0x7ff13c157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:39906/0 (socket says 172.21.15.94:39906) 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff13ad9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c06b500 0x7ff13c1ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff13a59c640 1 -- 172.21.15.94:0/4143693364 learned_addr learned my addr 172.21.15.94:0/4143693364 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.277+0000 7ff13b59e640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c1509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff13a59c640 1 -- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c06b500 msgr2=0x7ff13c1ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff13a59c640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c06b500 0x7ff13c1ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff13a59c640 1 -- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 msgr2=0x7ff13c1509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff13a59c640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c1509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff13a59c640 1 -- 172.21.15.94:0/4143693364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0aab60 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff13b59e640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c1509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff13a59c640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff13c0c6fa0 0x7ff13c157c30 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00e9b0 tx=0x7ff12c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff1337fe640 1 -- 172.21.15.94:0/4143693364 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c017070 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff1337fe640 1 -- 172.21.15.94:0/4143693364 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c00ca00 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c151860 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff1337fe640 1 -- 172.21.15.94:0/4143693364 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c00cdf0 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.278+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c1ee770 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.279+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c0693c0 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.279+0000 7ff1337fe640 1 -- 172.21.15.94:0/4143693364 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff12c01a070 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.280+0000 7ff1337fe640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff11c06b4d0 0x7ff11c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.280+0000 7ff1337fe640 1 -- 172.21.15.94:0/4143693364 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff12c089f40 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.280+0000 7ff13ad9d640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff11c06b4d0 0x7ff11c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.280+0000 7ff13ad9d640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff11c06b4d0 0x7ff11c06d990 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff124036000 tx=0x7ff12403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.284+0000 7ff1337fe640 1 -- 172.21.15.94:0/4143693364 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff12c0584f0 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.493+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff13c06fb20 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.495 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.493+0000 7ff1337fe640 1 -- 172.21.15.94:0/4143693364 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7ff12c05c590 con 0x7ff13c0c6fa0 2024-03-06T07:29:38.495 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:38.495 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:38.495 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:38.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff11c06b4d0 msgr2=0x7ff11c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff11c06b4d0 0x7ff11c06d990 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff124036000 tx=0x7ff12403d040 comp rx=0 tx=0).stop 2024-03-06T07:29:38.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff13c0c6fa0 msgr2=0x7ff13c157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff13c0c6fa0 0x7ff13c157c30 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00e9b0 tx=0x7ff12c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:38.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 shutdown_connections 2024-03-06T07:29:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff11c06b4d0 0x7ff11c06d990 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13c1527c0 0x7ff13c1509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff13c0c6fa0 0x7ff13c157c30 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 --2- 172.21.15.94:0/4143693364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13c06b500 0x7ff13c1ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 >> 172.21.15.94:0/4143693364 conn(0x7ff13c0c4010 msgr2=0x7ff13c154c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 shutdown_connections 2024-03-06T07:29:38.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:38.495+0000 7ff141202640 1 -- 172.21.15.94:0/4143693364 wait complete. 2024-03-06T07:29:38.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:38.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:39.912 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.022+0000 7f0d2f9f6640 1 Processor -- start 2024-03-06T07:29:40.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.022+0000 7f0d2f9f6640 1 -- start start 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2f9f6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d2814f360 0x7f0d28155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2f9f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d28158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2f9f6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d280c73d0 con 0x7f0d280c88c0 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2f9f6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d280c8dd0 con 0x7f0d2814f360 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2f9f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d280be2e0 con 0x7f0d28158b60 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2d792640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d2814f360 0x7f0d28155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2cf91640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0d280c88c0 0x7f0d280c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51212/0 (socket says 172.21.15.94:51212) 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2cf91640 1 -- 172.21.15.94:0/297973510 learned_addr learned my addr 172.21.15.94:0/297973510 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.023+0000 7f0d2df93640 1 --2- 172.21.15.94:0/297973510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d28158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.024+0000 7f0d2d792640 1 -- 172.21.15.94:0/297973510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 msgr2=0x7f0d28158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.024+0000 7f0d2d792640 1 --2- 172.21.15.94:0/297973510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d28158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.024+0000 7f0d2d792640 1 -- 172.21.15.94:0/297973510 >> v1:172.21.15.94:6789/0 conn(0x7f0d280c88c0 legacy=0x7f0d280c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.024+0000 7f0d2d792640 1 -- 172.21.15.94:0/297973510 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d280c7670 con 0x7f0d2814f360 2024-03-06T07:29:40.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.024+0000 7f0d2d792640 1 --2- 172.21.15.94:0/297973510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d28155c10 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009d90 tx=0x7f0d2402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed6ec9e85394871 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d1ffff640 1 -- 172.21.15.94:0/297973510 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24039070 con 0x7f0d2814f360 2024-03-06T07:29:40.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d1ffff640 1 -- 172.21.15.94:0/297973510 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2402cbc0 con 0x7f0d2814f360 2024-03-06T07:29:40.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d1ffff640 1 -- 172.21.15.94:0/297973510 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24034440 con 0x7f0d2814f360 2024-03-06T07:29:40.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/297973510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 msgr2=0x7f0d28155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/297973510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d28155c10 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009d90 tx=0x7f0d2402be10 comp rx=0 tx=0).stop 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/297973510 shutdown_connections 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/297973510 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d28158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/297973510 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d28155c10 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/297973510 >> 172.21.15.94:0/297973510 conn(0x7f0d280ac570 msgr2=0x7f0d280ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/297973510 shutdown_connections 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.025+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/297973510 wait complete. 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.026+0000 7f0d2f9f6640 1 Processor -- start 2024-03-06T07:29:40.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.026+0000 7f0d2f9f6640 1 -- start start 2024-03-06T07:29:40.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2f9f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d280c88c0 0x7f0d280c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2f9f6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d280bf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2f9f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d281ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2f9f6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d280c5a50 con 0x7f0d2814f360 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2f9f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d280c7c30 con 0x7f0d280c88c0 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2f9f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d280c5bd0 con 0x7f0d28158b60 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d280bf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d280bf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53150/0 (socket says 172.21.15.94:53150) 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 -- 172.21.15.94:0/2602621646 learned_addr learned my addr 172.21.15.94:0/2602621646 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2d792640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d280c88c0 0x7f0d280c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2df93640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d281ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 -- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 msgr2=0x7f0d281ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d281ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 -- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d280c88c0 msgr2=0x7f0d280c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d280c88c0 0x7f0d280c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.027+0000 7f0d2cf91640 1 -- 172.21.15.94:0/2602621646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d280c2bd0 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.028+0000 7f0d2cf91640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d280bf7a0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800c440 tx=0x7f0d1800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.028+0000 7f0d1dffb640 1 -- 172.21.15.94:0/2602621646 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18017070 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.028+0000 7f0d1dffb640 1 -- 172.21.15.94:0/2602621646 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d18005ce0 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.028+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d281ef320 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.028+0000 7f0d1dffb640 1 -- 172.21.15.94:0/2602621646 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d180072f0 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.028+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d280befa0 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.029+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2804f330 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.029+0000 7f0d1dffb640 1 -- 172.21.15.94:0/2602621646 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d1801a070 con 0x7f0d2814f360 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.030+0000 7f0d1dffb640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.033 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.030+0000 7f0d1dffb640 1 -- 172.21.15.94:0/2602621646 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0d18089700 con 0x7f0d2814f360 2024-03-06T07:29:40.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.030+0000 7f0d2d792640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.034 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.030+0000 7f0d2d792640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f0d24036000 tx=0x7f0d24030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.036 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.034+0000 7f0d1dffb640 1 -- 172.21.15.94:0/2602621646 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d18057ca0 con 0x7f0d2814f360 2024-03-06T07:29:40.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.239+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d280bdfb0 con 0x7f0d2814f360 2024-03-06T07:29:40.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.240+0000 7f0d1dffb640 1 -- 172.21.15.94:0/2602621646 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f0d1805bd40 con 0x7f0d2814f360 2024-03-06T07:29:40.242 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:40.242 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:40.242 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d0c06b4d0 msgr2=0x7f0d0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f0d24036000 tx=0x7f0d24030000 comp rx=0 tx=0).stop 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 msgr2=0x7f0d280bf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d280bf7a0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800c440 tx=0x7f0d1800c900 comp rx=0 tx=0).stop 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 shutdown_connections 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0d0c06b4d0 0x7f0d0c06d990 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d28158b60 0x7f0d281ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d2814f360 0x7f0d280bf7a0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 --2- 172.21.15.94:0/2602621646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d280c88c0 0x7f0d280c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 >> 172.21.15.94:0/2602621646 conn(0x7f0d280ac570 msgr2=0x7f0d28154900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 shutdown_connections 2024-03-06T07:29:40.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:40.242+0000 7f0d2f9f6640 1 -- 172.21.15.94:0/2602621646 wait complete. 2024-03-06T07:29:40.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:40.656 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:41.658 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:41.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.768+0000 7ff00384e640 1 Processor -- start 2024-03-06T07:29:41.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.768+0000 7ff00384e640 1 -- start start 2024-03-06T07:29:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff00384e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc14f310 0x7feffc0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff00384e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc155bb0 con 0x7feffc0aa180 2024-03-06T07:29:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff00384e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0aa660 con 0x7feffc1550c0 2024-03-06T07:29:41.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff00384e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0c4610 con 0x7feffc14f310 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff000de9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7feffc0aa180 0x7feffc14ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51218/0 (socket says 172.21.15.94:51218) 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff000de9640 1 -- 172.21.15.94:0/2362789207 learned_addr learned my addr 172.21.15.94:0/2362789207 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff001deb640 1 --2- 172.21.15.94:0/2362789207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc14f310 0x7feffc0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7feff3fff640 1 -- 172.21.15.94:0/2362789207 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776530228 0 0) 0x7feffc155bb0 con 0x7feffc0aa180 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7feff3fff640 1 -- 172.21.15.94:0/2362789207 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe8003520 con 0x7feffc0aa180 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7feff3fff640 1 -- 172.21.15.94:0/2362789207 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069134970 0 0) 0x7feffc0aa660 con 0x7feffc1550c0 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7feff3fff640 1 -- 172.21.15.94:0/2362789207 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feffc155bb0 con 0x7feffc1550c0 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff001deb640 1 -- 172.21.15.94:0/2362789207 >> v1:172.21.15.145:6789/0 conn(0x7feffc1550c0 legacy=0x7feffc1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff001deb640 1 -- 172.21.15.94:0/2362789207 >> v1:172.21.15.94:6789/0 conn(0x7feffc0aa180 legacy=0x7feffc14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.769+0000 7ff001deb640 1 -- 172.21.15.94:0/2362789207 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc0aa940 con 0x7feffc14f310 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.770+0000 7ff001deb640 1 --2- 172.21.15.94:0/2362789207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc14f310 0x7feffc0c5060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7feff400aed0 tx=0x7feff402cf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6e4f26166474c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.770+0000 7feff3fff640 1 -- 172.21.15.94:0/2362789207 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff402dab0 con 0x7feffc14f310 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.770+0000 7feff3fff640 1 -- 172.21.15.94:0/2362789207 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff402dc50 con 0x7feffc14f310 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.770+0000 7feff3fff640 1 -- 172.21.15.94:0/2362789207 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4008440 con 0x7feffc14f310 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.771+0000 7ff00384e640 1 -- 172.21.15.94:0/2362789207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc14f310 msgr2=0x7feffc0c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.771+0000 7ff00384e640 1 --2- 172.21.15.94:0/2362789207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc14f310 0x7feffc0c5060 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7feff400aed0 tx=0x7feff402cf10 comp rx=0 tx=0).stop 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.771+0000 7ff00384e640 1 -- 172.21.15.94:0/2362789207 shutdown_connections 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.771+0000 7ff00384e640 1 --2- 172.21.15.94:0/2362789207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc14f310 0x7feffc0c5060 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.771+0000 7ff00384e640 1 -- 172.21.15.94:0/2362789207 >> 172.21.15.94:0/2362789207 conn(0x7feffc0c6710 msgr2=0x7feffc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.771+0000 7ff00384e640 1 -- 172.21.15.94:0/2362789207 shutdown_connections 2024-03-06T07:29:41.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.771+0000 7ff00384e640 1 -- 172.21.15.94:0/2362789207 wait complete. 2024-03-06T07:29:41.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.772+0000 7ff00384e640 1 Processor -- start 2024-03-06T07:29:41.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.772+0000 7ff00384e640 1 -- start start 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff00384e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feffc0aa180 0x7feffc151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff00384e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feffc14f310 0x7feffc149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff00384e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc1550c0 0x7feffc149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff00384e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0c1490 con 0x7feffc14f310 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff00384e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0c1310 con 0x7feffc0aa180 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff00384e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0c3210 con 0x7feffc1550c0 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feffc14f310 0x7feffc149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feffc14f310 0x7feffc149610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53164/0 (socket says 172.21.15.94:53164) 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 -- 172.21.15.94:0/1551607432 learned_addr learned my addr 172.21.15.94:0/1551607432 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff0015ea640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feffc0aa180 0x7feffc151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff001deb640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc1550c0 0x7feffc149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 -- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc1550c0 msgr2=0x7feffc149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc1550c0 0x7feffc149b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 -- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feffc0aa180 msgr2=0x7feffc151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feffc0aa180 0x7feffc151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.773+0000 7ff000de9640 1 -- 172.21.15.94:0/1551607432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc14f840 con 0x7feffc14f310 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.774+0000 7ff000de9640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feffc14f310 0x7feffc149610 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c500 tx=0x7fefec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.774+0000 7feff1ffb640 1 -- 172.21.15.94:0/1551607432 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec017070 con 0x7feffc14f310 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.774+0000 7feff1ffb640 1 -- 172.21.15.94:0/1551607432 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec005ce0 con 0x7feffc14f310 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.774+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc14fae0 con 0x7feffc14f310 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.774+0000 7feff1ffb640 1 -- 172.21.15.94:0/1551607432 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec0072f0 con 0x7feffc14f310 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.774+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc14a280 con 0x7feffc14f310 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.775+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc04f330 con 0x7feffc14f310 2024-03-06T07:29:41.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.775+0000 7feff1ffb640 1 -- 172.21.15.94:0/1551607432 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefec01a070 con 0x7feffc14f310 2024-03-06T07:29:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.775+0000 7feff1ffb640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fefe006b500 0x7fefe006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.776+0000 7feff1ffb640 1 -- 172.21.15.94:0/1551607432 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fefec089730 con 0x7feffc14f310 2024-03-06T07:29:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.776+0000 7ff0015ea640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fefe006b500 0x7fefe006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.779 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.776+0000 7ff0015ea640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fefe006b500 0x7fefe006d9c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7feff8007fd0 tx=0x7feff8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.780+0000 7feff1ffb640 1 -- 172.21.15.94:0/1551607432 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fefec057ce0 con 0x7feffc14f310 2024-03-06T07:29:41.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.986+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feffc152840 con 0x7feffc14f310 2024-03-06T07:29:41.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.986+0000 7feff1ffb640 1 -- 172.21.15.94:0/1551607432 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fefec05bd80 con 0x7feffc14f310 2024-03-06T07:29:41.988 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:41.988 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:41.988 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fefe006b500 msgr2=0x7fefe006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fefe006b500 0x7fefe006d9c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7feff8007fd0 tx=0x7feff8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feffc14f310 msgr2=0x7feffc149610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feffc14f310 0x7feffc149610 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c500 tx=0x7fefec00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 shutdown_connections 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fefe006b500 0x7fefe006d9c0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feffc1550c0 0x7feffc149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feffc14f310 0x7feffc149610 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 --2- 172.21.15.94:0/1551607432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feffc0aa180 0x7feffc151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 >> 172.21.15.94:0/1551607432 conn(0x7feffc0c6710 msgr2=0x7feffc0a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 shutdown_connections 2024-03-06T07:29:41.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:41.988+0000 7ff00384e640 1 -- 172.21.15.94:0/1551607432 wait complete. 2024-03-06T07:29:42.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:42.403 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:43.404 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:43.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.514+0000 7fd208579640 1 Processor -- start 2024-03-06T07:29:43.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.514+0000 7fd208579640 1 -- start start 2024-03-06T07:29:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd208579640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd20014f360 0x7fd200155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd208579640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd200158b60 0x7fd200158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd208579640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2000c73d0 con 0x7fd2000c88c0 2024-03-06T07:29:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd208579640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2000c8dd0 con 0x7fd20014f360 2024-03-06T07:29:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd208579640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2000be2e0 con 0x7fd200158b60 2024-03-06T07:29:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd206315640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd20014f360 0x7fd200155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd206315640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd20014f360 0x7fd200155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53176/0 (socket says 172.21.15.94:53176) 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd206315640 1 -- 172.21.15.94:0/2400056432 learned_addr learned my addr 172.21.15.94:0/2400056432 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.515+0000 7fd206b16640 1 --2- 172.21.15.94:0/2400056432 >> v2:172.21.15.145:3300/0 conn(0x7fd200158b60 0x7fd200158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd206315640 1 -- 172.21.15.94:0/2400056432 >> v2:172.21.15.145:3300/0 conn(0x7fd200158b60 msgr2=0x7fd200158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd206315640 1 --2- 172.21.15.94:0/2400056432 >> v2:172.21.15.145:3300/0 conn(0x7fd200158b60 0x7fd200158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd206315640 1 -- 172.21.15.94:0/2400056432 >> v1:172.21.15.145:6789/0 conn(0x7fd2000c88c0 legacy=0x7fd2000c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd206315640 1 -- 172.21.15.94:0/2400056432 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2000c7670 con 0x7fd20014f360 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd206315640 1 --2- 172.21.15.94:0/2400056432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd200155c10 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fd200152d60 tx=0x7fd1fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d06d83544d6166 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd205313640 1 -- 172.21.15.94:0/2400056432 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc035020 con 0x7fd20014f360 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd205313640 1 -- 172.21.15.94:0/2400056432 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1fc031870 con 0x7fd20014f360 2024-03-06T07:29:43.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.516+0000 7fd205313640 1 -- 172.21.15.94:0/2400056432 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc031c40 con 0x7fd20014f360 2024-03-06T07:29:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 -- 172.21.15.94:0/2400056432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 msgr2=0x7fd200155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 --2- 172.21.15.94:0/2400056432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd200155c10 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fd200152d60 tx=0x7fd1fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:29:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 -- 172.21.15.94:0/2400056432 shutdown_connections 2024-03-06T07:29:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 --2- 172.21.15.94:0/2400056432 >> v2:172.21.15.145:3300/0 conn(0x7fd200158b60 0x7fd200158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 --2- 172.21.15.94:0/2400056432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd200155c10 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 -- 172.21.15.94:0/2400056432 >> 172.21.15.94:0/2400056432 conn(0x7fd2000ac570 msgr2=0x7fd2000ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 -- 172.21.15.94:0/2400056432 shutdown_connections 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.517+0000 7fd208579640 1 -- 172.21.15.94:0/2400056432 wait complete. 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.518+0000 7fd208579640 1 Processor -- start 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.518+0000 7fd208579640 1 -- start start 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd208579640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2000c88c0 0x7fd2000ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd208579640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd20014bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd208579640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd200158b60 0x7fd20014c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd208579640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000c5a50 con 0x7fd20014f360 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd208579640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000c7c30 con 0x7fd200158b60 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd208579640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000c5bd0 con 0x7fd2000c88c0 2024-03-06T07:29:43.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd20014bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd20014bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53186/0 (socket says 172.21.15.94:53186) 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 -- 172.21.15.94:0/640776051 learned_addr learned my addr 172.21.15.94:0/640776051 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd206315640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2000c88c0 0x7fd2000ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd206b16640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd200158b60 0x7fd20014c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 -- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2000c88c0 msgr2=0x7fd2000ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2000c88c0 0x7fd2000ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 -- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd200158b60 msgr2=0x7fd20014c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd200158b60 0x7fd20014c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.519+0000 7fd205b14640 1 -- 172.21.15.94:0/640776051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2000befa0 con 0x7fd20014f360 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.520+0000 7fd205b14640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd20014bc30 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000c470 tx=0x7fd1f000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.520+0000 7fd1f6ffd640 1 -- 172.21.15.94:0/640776051 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f00052d0 con 0x7fd20014f360 2024-03-06T07:29:43.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.520+0000 7fd1f6ffd640 1 -- 172.21.15.94:0/640776051 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f0005490 con 0x7fd20014f360 2024-03-06T07:29:43.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.520+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2000bf280 con 0x7fd20014f360 2024-03-06T07:29:43.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.520+0000 7fd1f6ffd640 1 -- 172.21.15.94:0/640776051 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0007260 con 0x7fd20014f360 2024-03-06T07:29:43.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.520+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2000adad0 con 0x7fd20014f360 2024-03-06T07:29:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.521+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20004f330 con 0x7fd20014f360 2024-03-06T07:29:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.521+0000 7fd1f6ffd640 1 -- 172.21.15.94:0/640776051 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1f0007400 con 0x7fd20014f360 2024-03-06T07:29:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.522+0000 7fd1f6ffd640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd1e406b500 0x7fd1e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.522+0000 7fd1f6ffd640 1 -- 172.21.15.94:0/640776051 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd1f00897c0 con 0x7fd20014f360 2024-03-06T07:29:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.522+0000 7fd206315640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd1e406b500 0x7fd1e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.522+0000 7fd206315640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd1e406b500 0x7fd1e406d9c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc036000 tx=0x7fd1fc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.526+0000 7fd1f6ffd640 1 -- 172.21.15.94:0/640776051 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1f0057df0 con 0x7fd20014f360 2024-03-06T07:29:43.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.733+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd20006cda0 con 0x7fd20014f360 2024-03-06T07:29:43.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.734+0000 7fd1f6ffd640 1 -- 172.21.15.94:0/640776051 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fd1f005be90 con 0x7fd20014f360 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:43.736 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd1e406b500 msgr2=0x7fd1e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd1e406b500 0x7fd1e406d9c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc036000 tx=0x7fd1fc03c040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 msgr2=0x7fd20014bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd20014bc30 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000c470 tx=0x7fd1f000c930 comp rx=0 tx=0).stop 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 shutdown_connections 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd1e406b500 0x7fd1e406d9c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd200158b60 0x7fd20014c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20014f360 0x7fd20014bc30 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 --2- 172.21.15.94:0/640776051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2000c88c0 0x7fd2000ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 >> 172.21.15.94:0/640776051 conn(0x7fd2000ac570 msgr2=0x7fd2000ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 shutdown_connections 2024-03-06T07:29:43.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:43.736+0000 7fd208579640 1 -- 172.21.15.94:0/640776051 wait complete. 2024-03-06T07:29:44.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:44.151 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:45.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:45.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.261+0000 7f681920e640 1 Processor -- start 2024-03-06T07:29:45.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.261+0000 7f681920e640 1 -- start start 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681920e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6814152960 0x7f681406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681920e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68141592d0 0x7f68141552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681920e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6814152d40 con 0x7f68140ac260 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681920e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6814066e30 con 0x7f68141592d0 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681920e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68140c1dc0 con 0x7f6814152960 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681259c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68141592d0 0x7f68141552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681259c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f68141592d0 0x7f68141552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53196/0 (socket says 172.21.15.94:53196) 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681259c640 1 -- 172.21.15.94:0/2437230554 learned_addr learned my addr 172.21.15.94:0/2437230554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f681359e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f68140ac260 0x7f6814155810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51234/0 (socket says 172.21.15.94:51234) 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.262+0000 7f6812d9d640 1 --2- 172.21.15.94:0/2437230554 >> v2:172.21.15.145:3300/0 conn(0x7f6814152960 0x7f681406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.263+0000 7f681259c640 1 -- 172.21.15.94:0/2437230554 >> v2:172.21.15.145:3300/0 conn(0x7f6814152960 msgr2=0x7f681406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.263+0000 7f681259c640 1 --2- 172.21.15.94:0/2437230554 >> v2:172.21.15.145:3300/0 conn(0x7f6814152960 0x7f681406dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.263+0000 7f681259c640 1 -- 172.21.15.94:0/2437230554 >> v1:172.21.15.94:6789/0 conn(0x7f68140ac260 legacy=0x7f6814155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.263+0000 7f681259c640 1 -- 172.21.15.94:0/2437230554 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68140c1b50 con 0x7f68141592d0 2024-03-06T07:29:45.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.263+0000 7f6811d9b640 1 -- 172.21.15.94:0/2437230554 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169561764 0 0) 0x7f6814152d40 con 0x7f68140ac260 2024-03-06T07:29:45.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.263+0000 7f681259c640 1 --2- 172.21.15.94:0/2437230554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68141592d0 0x7f68141552d0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f681406b1c0 tx=0x7f680402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d4f2ca47cd2cad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f6811d9b640 1 -- 172.21.15.94:0/2437230554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804034070 con 0x7f68141592d0 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f6811d9b640 1 -- 172.21.15.94:0/2437230554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6804030860 con 0x7f68141592d0 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f6811d9b640 1 -- 172.21.15.94:0/2437230554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804030c30 con 0x7f68141592d0 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 -- 172.21.15.94:0/2437230554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68141592d0 msgr2=0x7f68141552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 --2- 172.21.15.94:0/2437230554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68141592d0 0x7f68141552d0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f681406b1c0 tx=0x7f680402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 -- 172.21.15.94:0/2437230554 shutdown_connections 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 --2- 172.21.15.94:0/2437230554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68141592d0 0x7f68141552d0 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 --2- 172.21.15.94:0/2437230554 >> v2:172.21.15.145:3300/0 conn(0x7f6814152960 0x7f681406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 -- 172.21.15.94:0/2437230554 >> 172.21.15.94:0/2437230554 conn(0x7f68140c70b0 msgr2=0x7f68140c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 -- 172.21.15.94:0/2437230554 shutdown_connections 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.264+0000 7f681920e640 1 -- 172.21.15.94:0/2437230554 wait complete. 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 Processor -- start 2024-03-06T07:29:45.268 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 -- start start 2024-03-06T07:29:45.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68140ac260 0x7f68141de930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6814152960 0x7f68141ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6814152e20 0x7f68141532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68140c20a0 con 0x7f68140ac260 2024-03-06T07:29:45.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6814159760 con 0x7f6814152e20 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f681920e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68140c2220 con 0x7f6814152960 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f6812d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68140ac260 0x7f68141de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f6812d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68140ac260 0x7f68141de930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53202/0 (socket says 172.21.15.94:53202) 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.266+0000 7f6812d9d640 1 -- 172.21.15.94:0/3714098909 learned_addr learned my addr 172.21.15.94:0/3714098909 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f681359e640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6814152e20 0x7f68141532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f681259c640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6814152960 0x7f68141ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f6812d9d640 1 -- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6814152960 msgr2=0x7f68141ee080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f6812d9d640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6814152960 0x7f68141ee080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f6812d9d640 1 -- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6814152e20 msgr2=0x7f68141532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f6812d9d640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6814152e20 0x7f68141532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f6812d9d640 1 -- 172.21.15.94:0/3714098909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68141dee70 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f6812d9d640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68140ac260 0x7f68141de930 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e9b0 tx=0x7f67fc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f680b7fe640 1 -- 172.21.15.94:0/3714098909 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc017070 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f680b7fe640 1 -- 172.21.15.94:0/3714098909 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc00c9a0 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68141ee950 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f680b7fe640 1 -- 172.21.15.94:0/3714098909 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc00cd90 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.267+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68141567f0 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.268+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681404f330 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.269+0000 7f680b7fe640 1 -- 172.21.15.94:0/3714098909 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67fc01a070 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.269+0000 7f680b7fe640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67f406b4d0 0x7f67f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.269+0000 7f680b7fe640 1 -- 172.21.15.94:0/3714098909 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f67fc08a340 con 0x7f68140ac260 2024-03-06T07:29:45.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.269+0000 7f681259c640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67f406b4d0 0x7f67f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.270+0000 7f681259c640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67f406b4d0 0x7f67f406d990 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6804036000 tx=0x7f6804008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.274+0000 7f680b7fe640 1 -- 172.21.15.94:0/3714098909 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67fc058970 con 0x7f68140ac260 2024-03-06T07:29:45.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.482+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6814156270 con 0x7f68140ac260 2024-03-06T07:29:45.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.483+0000 7f680b7fe640 1 -- 172.21.15.94:0/3714098909 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f67fc05ca10 con 0x7f68140ac260 2024-03-06T07:29:45.485 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:45.485 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:45.485 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67f406b4d0 msgr2=0x7f67f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67f406b4d0 0x7f67f406d990 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6804036000 tx=0x7f6804008030 comp rx=0 tx=0).stop 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68140ac260 msgr2=0x7f68141de930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68140ac260 0x7f68141de930 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e9b0 tx=0x7f67fc00ee70 comp rx=0 tx=0).stop 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 shutdown_connections 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f67f406b4d0 0x7f67f406d990 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6814152e20 0x7f68141532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6814152960 0x7f68141ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 --2- 172.21.15.94:0/3714098909 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68140ac260 0x7f68141de930 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 >> 172.21.15.94:0/3714098909 conn(0x7f68140c70b0 msgr2=0x7f681406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 shutdown_connections 2024-03-06T07:29:45.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:45.485+0000 7f681920e640 1 -- 172.21.15.94:0/3714098909 wait complete. 2024-03-06T07:29:45.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:45.900 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:46.440 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:46.443 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:46.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:47.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.010+0000 7f1ad23d7640 1 Processor -- start 2024-03-06T07:29:47.013 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.010+0000 7f1ad23d7640 1 -- start start 2024-03-06T07:29:47.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1ad23d7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1acc14f790 0x7f1acc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1ad23d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1ad23d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc1563d0 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1ad23d7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc0c2680 con 0x7f1acc14f790 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1ad23d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0c28d0 con 0x7f1acc155a40 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1acc0c1aa0 0x7f1acc0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51250/0 (socket says 172.21.15.94:51250) 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acb7fe640 1 -- 172.21.15.94:0/931511735 learned_addr learned my addr 172.21.15.94:0/931511735 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acbfff640 1 --2- 172.21.15.94:0/931511735 >> v2:172.21.15.145:3300/0 conn(0x7f1acc14f790 0x7f1acc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1ad0974640 1 --2- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acaffd640 1 -- 172.21.15.94:0/931511735 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389616766 0 0) 0x7f1acc1563d0 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acaffd640 1 -- 172.21.15.94:0/931511735 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab4003520 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acbfff640 1 -- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 msgr2=0x7f1acc155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acbfff640 1 --2- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acbfff640 1 -- 172.21.15.94:0/931511735 >> v1:172.21.15.94:6789/0 conn(0x7f1acc0c1aa0 legacy=0x7f1acc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1acbfff640 1 -- 172.21.15.94:0/931511735 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0c2bb0 con 0x7f1acc14f790 2024-03-06T07:29:47.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.011+0000 7f1ad0974640 1 --2- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.012+0000 7f1acbfff640 1 --2- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 0x7f1acc1529d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0097a0 tx=0x7f1abc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36bb4f812bd4fb60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.012+0000 7f1acaffd640 1 -- 172.21.15.94:0/931511735 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc035020 con 0x7f1acc14f790 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.012+0000 7f1acaffd640 1 -- 172.21.15.94:0/931511735 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc031830 con 0x7f1acc14f790 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.012+0000 7f1acaffd640 1 -- 172.21.15.94:0/931511735 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc031be0 con 0x7f1acc14f790 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 -- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 msgr2=0x7f1acc1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 0x7f1acc1529d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0097a0 tx=0x7f1abc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 -- 172.21.15.94:0/931511735 shutdown_connections 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/931511735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 0x7f1acc1529d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 -- 172.21.15.94:0/931511735 >> 172.21.15.94:0/931511735 conn(0x7f1acc0c6710 msgr2=0x7f1acc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 -- 172.21.15.94:0/931511735 shutdown_connections 2024-03-06T07:29:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.013+0000 7f1ad23d7640 1 -- 172.21.15.94:0/931511735 wait complete. 2024-03-06T07:29:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.014+0000 7f1ad23d7640 1 Processor -- start 2024-03-06T07:29:47.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.014+0000 7f1ad23d7640 1 -- start start 2024-03-06T07:29:47.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1ad23d7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1acc0c1aa0 0x7f1acc14db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1ad23d7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 0x7f1acc1e8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1ad23d7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc1e9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1ad23d7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0c0d60 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1ad23d7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc155520 con 0x7f1acc14f790 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1ad23d7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0c5e30 con 0x7f1acc155a40 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1acc0c1aa0 0x7f1acc14db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1acc0c1aa0 0x7f1acc14db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:53216/0 (socket says 172.21.15.94:53216) 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 -- 172.21.15.94:0/2928936113 learned_addr learned my addr 172.21.15.94:0/2928936113 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acb7fe640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 0x7f1acc1e8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1ad0974640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc1e9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 -- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 msgr2=0x7f1acc1e9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc1e9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 -- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 msgr2=0x7f1acc1e8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 0x7f1acc1e8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 -- 172.21.15.94:0/2928936113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc154480 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.015+0000 7f1acbfff640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1acc0c1aa0 0x7f1acc14db70 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0098c0 tx=0x7f1abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.016+0000 7f1ac8ff9640 1 -- 172.21.15.94:0/2928936113 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc035040 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.016+0000 7f1ac8ff9640 1 -- 172.21.15.94:0/2928936113 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc004450 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.016+0000 7f1ac8ff9640 1 -- 172.21.15.94:0/2928936113 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc003830 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.016+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc154720 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.016+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc14e0b0 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.017+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc04f330 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.017+0000 7f1ac8ff9640 1 -- 172.21.15.94:0/2928936113 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1abc038020 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.017+0000 7f1ac8ff9640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ab806b4d0 0x7f1ab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.018+0000 7f1ac8ff9640 1 -- 172.21.15.94:0/2928936113 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1abc0ac390 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.018+0000 7f1acb7fe640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ab806b4d0 0x7f1ab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.018+0000 7f1acb7fe640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ab806b4d0 0x7f1ab806d990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f1acc06cfd0 tx=0x7f1ac0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.022+0000 7f1ac8ff9640 1 -- 172.21.15.94:0/2928936113 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1abc07b010 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.230+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1acc1e88e0 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.231+0000 7f1ac8ff9640 1 -- 172.21.15.94:0/2928936113 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f1abc03de20 con 0x7f1acc0c1aa0 2024-03-06T07:29:47.234 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:47.234 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:47.234 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:47.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ab806b4d0 msgr2=0x7f1ab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ab806b4d0 0x7f1ab806d990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f1acc06cfd0 tx=0x7f1ac0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:47.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1acc0c1aa0 msgr2=0x7f1acc14db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1acc0c1aa0 0x7f1acc14db70 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0098c0 tx=0x7f1abc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 shutdown_connections 2024-03-06T07:29:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1ab806b4d0 0x7f1ab806d990 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1acc155a40 0x7f1acc1e9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1acc14f790 0x7f1acc1e8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 --2- 172.21.15.94:0/2928936113 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1acc0c1aa0 0x7f1acc14db70 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 >> 172.21.15.94:0/2928936113 conn(0x7f1acc0c6710 msgr2=0x7f1acc155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 shutdown_connections 2024-03-06T07:29:47.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:47.233+0000 7f1ad23d7640 1 -- 172.21.15.94:0/2928936113 wait complete. 2024-03-06T07:29:47.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:47.649 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:48.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:48.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.758+0000 7ffa319ac640 1 Processor -- start 2024-03-06T07:29:48.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.758+0000 7ffa319ac640 1 -- start start 2024-03-06T07:29:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa319ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c159270 0x7ffa2c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa319ac640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0c73d0 con 0x7ffa2c14f360 2024-03-06T07:29:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa319ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0be2e0 con 0x7ffa2c0c88c0 2024-03-06T07:29:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa319ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c159700 con 0x7ffa2c159270 2024-03-06T07:29:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa2affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ffa2c14f360 0x7ffa2c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57422/0 (socket says 172.21.15.94:57422) 2024-03-06T07:29:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa2affd640 1 -- 172.21.15.94:0/60080188 learned_addr learned my addr 172.21.15.94:0/60080188 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:48.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa2b7fe640 1 --2- 172.21.15.94:0/60080188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c159270 0x7ffa2c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa29ffb640 1 -- 172.21.15.94:0/60080188 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000643247 0 0) 0x7ffa2c0c73d0 con 0x7ffa2c14f360 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa29ffb640 1 -- 172.21.15.94:0/60080188 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa14003520 con 0x7ffa2c14f360 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa29ffb640 1 -- 172.21.15.94:0/60080188 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993879885 0 0) 0x7ffa2c0be2e0 con 0x7ffa2c0c88c0 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa29ffb640 1 -- 172.21.15.94:0/60080188 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa2c0c73d0 con 0x7ffa2c0c88c0 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa2b7fe640 1 -- 172.21.15.94:0/60080188 >> v1:172.21.15.145:6789/0 conn(0x7ffa2c0c88c0 legacy=0x7ffa2c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.759+0000 7ffa2b7fe640 1 -- 172.21.15.94:0/60080188 >> v1:172.21.15.94:6789/0 conn(0x7ffa2c14f360 legacy=0x7ffa2c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.760+0000 7ffa2b7fe640 1 -- 172.21.15.94:0/60080188 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0c7670 con 0x7ffa2c159270 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.760+0000 7ffa2b7fe640 1 --2- 172.21.15.94:0/60080188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c159270 0x7ffa2c155c10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ffa240097d0 tx=0x7ffa2402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5957aeff31e6b699 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.760+0000 7ffa29ffb640 1 -- 172.21.15.94:0/60080188 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24035020 con 0x7ffa2c159270 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.760+0000 7ffa29ffb640 1 -- 172.21.15.94:0/60080188 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24031800 con 0x7ffa2c159270 2024-03-06T07:29:48.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.760+0000 7ffa29ffb640 1 -- 172.21.15.94:0/60080188 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24031bb0 con 0x7ffa2c159270 2024-03-06T07:29:48.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.761+0000 7ffa319ac640 1 -- 172.21.15.94:0/60080188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c159270 msgr2=0x7ffa2c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.761+0000 7ffa319ac640 1 --2- 172.21.15.94:0/60080188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c159270 0x7ffa2c155c10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ffa240097d0 tx=0x7ffa2402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.761+0000 7ffa319ac640 1 -- 172.21.15.94:0/60080188 shutdown_connections 2024-03-06T07:29:48.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.761+0000 7ffa319ac640 1 --2- 172.21.15.94:0/60080188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c159270 0x7ffa2c155c10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.761+0000 7ffa319ac640 1 -- 172.21.15.94:0/60080188 >> 172.21.15.94:0/60080188 conn(0x7ffa2c0ac570 msgr2=0x7ffa2c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.761+0000 7ffa319ac640 1 -- 172.21.15.94:0/60080188 shutdown_connections 2024-03-06T07:29:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.761+0000 7ffa319ac640 1 -- 172.21.15.94:0/60080188 wait complete. 2024-03-06T07:29:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.762+0000 7ffa319ac640 1 Processor -- start 2024-03-06T07:29:48.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.762+0000 7ffa319ac640 1 -- start start 2024-03-06T07:29:48.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa319ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c0c88c0 0x7ffa2c1eae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa319ac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa2c14f360 0x7ffa2c0c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa319ac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa2c159270 0x7ffa2c1eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa319ac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0c7c30 con 0x7ffa2c159270 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa319ac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0c5a50 con 0x7ffa2c14f360 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa319ac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0c5bd0 con 0x7ffa2c0c88c0 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa2c159270 0x7ffa2c1eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa2affd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c0c88c0 0x7ffa2c1eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa2b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa2c159270 0x7ffa2c1eeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58334/0 (socket says 172.21.15.94:58334) 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa2b7fe640 1 -- 172.21.15.94:0/979878293 learned_addr learned my addr 172.21.15.94:0/979878293 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.763+0000 7ffa2a7fc640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa2c14f360 0x7ffa2c0c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa2b7fe640 1 -- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c0c88c0 msgr2=0x7ffa2c1eae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa2b7fe640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c0c88c0 0x7ffa2c1eae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa2b7fe640 1 -- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa2c14f360 msgr2=0x7ffa2c0c27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa2b7fe640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa2c14f360 0x7ffa2c0c27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa2b7fe640 1 -- 172.21.15.94:0/979878293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0c44b0 con 0x7ffa2c159270 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa2b7fe640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa2c159270 0x7ffa2c1eeb60 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7ffa2402b8e0 tx=0x7ffa240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa13fff640 1 -- 172.21.15.94:0/979878293 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24041070 con 0x7ffa2c159270 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa13fff640 1 -- 172.21.15.94:0/979878293 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24031e30 con 0x7ffa2c159270 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c0c4790 con 0x7ffa2c159270 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa13fff640 1 -- 172.21.15.94:0/979878293 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24035040 con 0x7ffa2c159270 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.764+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c0c2d30 con 0x7ffa2c159270 2024-03-06T07:29:48.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.765+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c069460 con 0x7ffa2c159270 2024-03-06T07:29:48.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.765+0000 7ffa13fff640 1 -- 172.21.15.94:0/979878293 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa2404c060 con 0x7ffa2c159270 2024-03-06T07:29:48.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.766+0000 7ffa13fff640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa1806b4d0 0x7ffa1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.766+0000 7ffa13fff640 1 -- 172.21.15.94:0/979878293 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ffa240abcd0 con 0x7ffa2c159270 2024-03-06T07:29:48.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.766+0000 7ffa2affd640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa1806b4d0 0x7ffa1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.770 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.766+0000 7ffa2affd640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa1806b4d0 0x7ffa1806d990 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0012e0 tx=0x7ffa1c001ad0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.770+0000 7ffa13fff640 1 -- 172.21.15.94:0/979878293 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa2c069460 con 0x7ffa2c159270 2024-03-06T07:29:48.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.981+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa2c04f330 con 0x7ffa2c159270 2024-03-06T07:29:48.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.982+0000 7ffa13fff640 1 -- 172.21.15.94:0/979878293 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7ffa2407a300 con 0x7ffa2c159270 2024-03-06T07:29:48.985 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:48.985 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:48.985 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.983+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa1806b4d0 msgr2=0x7ffa1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.983+0000 7ffa319ac640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa1806b4d0 0x7ffa1806d990 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0012e0 tx=0x7ffa1c001ad0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.983+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa2c159270 msgr2=0x7ffa2c1eeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.983+0000 7ffa319ac640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa2c159270 0x7ffa2c1eeb60 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7ffa2402b8e0 tx=0x7ffa240046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 shutdown_connections 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffa1806b4d0 0x7ffa1806d990 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa2c159270 0x7ffa2c1eeb60 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa2c14f360 0x7ffa2c0c27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 --2- 172.21.15.94:0/979878293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa2c0c88c0 0x7ffa2c1eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 >> 172.21.15.94:0/979878293 conn(0x7ffa2c0ac570 msgr2=0x7ffa2c0c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 shutdown_connections 2024-03-06T07:29:48.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:48.984+0000 7ffa319ac640 1 -- 172.21.15.94:0/979878293 wait complete. 2024-03-06T07:29:49.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:49.400 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:50.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:50.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.514+0000 7f0ea9116640 1 Processor -- start 2024-03-06T07:29:50.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.514+0000 7f0ea9116640 1 -- start start 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea9116640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ea41592d0 0x7f0ea41552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea9116640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea4152d40 con 0x7f0ea4152960 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea9116640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea40c1b50 con 0x7f0ea40ac260 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea9116640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea4066e30 con 0x7f0ea41592d0 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0e9bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ea41592d0 0x7f0ea41552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea2d9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0ea4152960 0x7f0ea406dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57430/0 (socket says 172.21.15.94:57430) 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea2d9d640 1 -- 172.21.15.94:0/56001160 learned_addr learned my addr 172.21.15.94:0/56001160 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea259c640 1 -- 172.21.15.94:0/56001160 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547409051 0 0) 0x7f0ea4152d40 con 0x7f0ea4152960 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea259c640 1 -- 172.21.15.94:0/56001160 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e90003520 con 0x7f0ea4152960 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea259c640 1 -- 172.21.15.94:0/56001160 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682599520 0 0) 0x7f0ea40c1b50 con 0x7f0ea40ac260 2024-03-06T07:29:50.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0ea259c640 1 -- 172.21.15.94:0/56001160 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea4152d40 con 0x7f0ea40ac260 2024-03-06T07:29:50.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0e9bfff640 1 -- 172.21.15.94:0/56001160 >> v1:172.21.15.145:6789/0 conn(0x7f0ea40ac260 legacy=0x7f0ea4155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0e9bfff640 1 -- 172.21.15.94:0/56001160 >> v1:172.21.15.94:6789/0 conn(0x7f0ea4152960 legacy=0x7f0ea406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.515+0000 7f0e9bfff640 1 -- 172.21.15.94:0/56001160 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea40c1dc0 con 0x7f0ea41592d0 2024-03-06T07:29:50.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.516+0000 7f0e9bfff640 1 --2- 172.21.15.94:0/56001160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41592d0 0x7f0ea41552d0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d402d492bb9c158 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.516+0000 7f0ea259c640 1 -- 172.21.15.94:0/56001160 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c034070 con 0x7f0ea41592d0 2024-03-06T07:29:50.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.516+0000 7f0ea259c640 1 -- 172.21.15.94:0/56001160 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c030610 con 0x7f0ea41592d0 2024-03-06T07:29:50.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.516+0000 7f0ea259c640 1 -- 172.21.15.94:0/56001160 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0309e0 con 0x7f0ea41592d0 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.517+0000 7f0ea9116640 1 -- 172.21.15.94:0/56001160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41592d0 msgr2=0x7f0ea41552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.517+0000 7f0ea9116640 1 --2- 172.21.15.94:0/56001160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41592d0 0x7f0ea41552d0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097a0 tx=0x7f0e8c02b960 comp rx=0 tx=0).stop 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.517+0000 7f0ea9116640 1 -- 172.21.15.94:0/56001160 shutdown_connections 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.517+0000 7f0ea9116640 1 --2- 172.21.15.94:0/56001160 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41592d0 0x7f0ea41552d0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.517+0000 7f0ea9116640 1 -- 172.21.15.94:0/56001160 >> 172.21.15.94:0/56001160 conn(0x7f0ea40c70b0 msgr2=0x7f0ea40c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.517+0000 7f0ea9116640 1 -- 172.21.15.94:0/56001160 shutdown_connections 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.517+0000 7f0ea9116640 1 -- 172.21.15.94:0/56001160 wait complete. 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.518+0000 7f0ea9116640 1 Processor -- start 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.518+0000 7f0ea9116640 1 -- start start 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea9116640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ea40ac260 0x7f0ea4150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea9116640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ea4152960 0x7f0ea41e0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea9116640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41eb320 0x7f0ea41eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea9116640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea406e3a0 con 0x7f0ea41eb320 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea9116640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea4159760 con 0x7f0ea4152960 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea9116640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea406e520 con 0x7f0ea40ac260 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41eb320 0x7f0ea41eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41eb320 0x7f0ea41eb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58346/0 (socket says 172.21.15.94:58346) 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 -- 172.21.15.94:0/2075803008 learned_addr learned my addr 172.21.15.94:0/2075803008 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea2d9d640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ea40ac260 0x7f0ea4150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0e9bfff640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ea4152960 0x7f0ea41e0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 -- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ea40ac260 msgr2=0x7f0ea4150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ea40ac260 0x7f0ea4150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 -- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ea4152960 msgr2=0x7f0ea41e0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ea4152960 0x7f0ea41e0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.523 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.519+0000 7f0ea359e640 1 -- 172.21.15.94:0/2075803008 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea41e1760 con 0x7f0ea41eb320 2024-03-06T07:29:50.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.520+0000 7f0ea359e640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41eb320 0x7f0ea41eb7d0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00c7d0 tx=0x7f0e9c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.520+0000 7f0e9b7fe640 1 -- 172.21.15.94:0/2075803008 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c003b70 con 0x7f0ea41eb320 2024-03-06T07:29:50.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.520+0000 7f0e9b7fe640 1 -- 172.21.15.94:0/2075803008 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c003d10 con 0x7f0ea41eb320 2024-03-06T07:29:50.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.520+0000 7f0e9b7fe640 1 -- 172.21.15.94:0/2075803008 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c00aa90 con 0x7f0ea41eb320 2024-03-06T07:29:50.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.520+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea41dfe80 con 0x7f0ea41eb320 2024-03-06T07:29:50.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.520+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea41e0440 con 0x7f0ea41eb320 2024-03-06T07:29:50.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.521+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea404f330 con 0x7f0ea41eb320 2024-03-06T07:29:50.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.521+0000 7f0e9b7fe640 1 -- 172.21.15.94:0/2075803008 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e9c00ac30 con 0x7f0ea41eb320 2024-03-06T07:29:50.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.521+0000 7f0e9b7fe640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0e8406b4d0 0x7f0e8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.522+0000 7f0e9b7fe640 1 -- 172.21.15.94:0/2075803008 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0e9c08a330 con 0x7f0ea41eb320 2024-03-06T07:29:50.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.522+0000 7f0ea2d9d640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0e8406b4d0 0x7f0e8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.522+0000 7f0ea2d9d640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0e8406b4d0 0x7f0e8406d990 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f0e940011e0 tx=0x7f0e94007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.526+0000 7f0e9b7fe640 1 -- 172.21.15.94:0/2075803008 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e9c0588d0 con 0x7f0ea41eb320 2024-03-06T07:29:50.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.735+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ea4156270 con 0x7f0ea41eb320 2024-03-06T07:29:50.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.735+0000 7f0e9b7fe640 1 -- 172.21.15.94:0/2075803008 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f0e9c05c970 con 0x7f0ea41eb320 2024-03-06T07:29:50.737 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:50.738 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:50.738 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:50.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0e8406b4d0 msgr2=0x7f0e8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0e8406b4d0 0x7f0e8406d990 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f0e940011e0 tx=0x7f0e94007d30 comp rx=0 tx=0).stop 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41eb320 msgr2=0x7f0ea41eb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41eb320 0x7f0ea41eb7d0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00c7d0 tx=0x7f0e9c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 shutdown_connections 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0e8406b4d0 0x7f0e8406d990 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ea41eb320 0x7f0ea41eb7d0 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ea4152960 0x7f0ea41e0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 --2- 172.21.15.94:0/2075803008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ea40ac260 0x7f0ea4150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 >> 172.21.15.94:0/2075803008 conn(0x7f0ea40c70b0 msgr2=0x7f0ea40c3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 shutdown_connections 2024-03-06T07:29:50.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:50.737+0000 7f0ea9116640 1 -- 172.21.15.94:0/2075803008 wait complete. 2024-03-06T07:29:51.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:51.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:52.154 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:52.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.263+0000 7f8b32afb640 1 Processor -- start 2024-03-06T07:29:52.265 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.263+0000 7f8b32afb640 1 -- start start 2024-03-06T07:29:52.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b32afb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8b2c1550c0 0x7f8b2c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b32afb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c1559e0 0x7f8b2c14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b32afb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2c0aa660 con 0x7f8b2c0aa180 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b32afb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b2c0c4610 con 0x7f8b2c1550c0 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b32afb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c0aa940 con 0x7f8b2c1559e0 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b30897640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8b2c1550c0 0x7f8b2c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b31098640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8b2c0aa180 0x7f8b2c0c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57436/0 (socket says 172.21.15.94:57436) 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b31098640 1 -- 172.21.15.94:0/1181352442 learned_addr learned my addr 172.21.15.94:0/1181352442 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.264+0000 7f8b2bfff640 1 --2- 172.21.15.94:0/1181352442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c1559e0 0x7f8b2c14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b30897640 1 -- 172.21.15.94:0/1181352442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c1559e0 msgr2=0x7f8b2c14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b30897640 1 --2- 172.21.15.94:0/1181352442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c1559e0 0x7f8b2c14ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b30897640 1 -- 172.21.15.94:0/1181352442 >> v1:172.21.15.94:6789/0 conn(0x7f8b2c0aa180 legacy=0x7f8b2c0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b30897640 1 -- 172.21.15.94:0/1181352442 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c0c4860 con 0x7f8b2c1550c0 2024-03-06T07:29:52.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1181352442 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924805141 0 0) 0x7f8b2c0aa660 con 0x7f8b2c0aa180 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b30897640 1 --2- 172.21.15.94:0/1181352442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c1554a0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c0097a0 tx=0x7f8b1c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8c7db32ef2001 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1181352442 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c038070 con 0x7f8b2c1550c0 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.265+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1181352442 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1c031710 con 0x7f8b2c1550c0 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b2b7fe640 1 -- 172.21.15.94:0/1181352442 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c031ae0 con 0x7f8b2c1550c0 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 -- 172.21.15.94:0/1181352442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 msgr2=0x7f8b2c1554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 --2- 172.21.15.94:0/1181352442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c1554a0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c0097a0 tx=0x7f8b1c02b7b0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 -- 172.21.15.94:0/1181352442 shutdown_connections 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 --2- 172.21.15.94:0/1181352442 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c1559e0 0x7f8b2c14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.269 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 --2- 172.21.15.94:0/1181352442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c1554a0 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 -- 172.21.15.94:0/1181352442 >> 172.21.15.94:0/1181352442 conn(0x7f8b2c0c6710 msgr2=0x7f8b2c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 -- 172.21.15.94:0/1181352442 shutdown_connections 2024-03-06T07:29:52.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.266+0000 7f8b32afb640 1 -- 172.21.15.94:0/1181352442 wait complete. 2024-03-06T07:29:52.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.267+0000 7f8b32afb640 1 Processor -- start 2024-03-06T07:29:52.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.267+0000 7f8b32afb640 1 -- start start 2024-03-06T07:29:52.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b32afb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c0aa180 0x7f8b2c144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b32afb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c14bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b32afb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b2c1559e0 0x7f8b2c143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b32afb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c14f140 con 0x7f8b2c1550c0 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b32afb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c0c1520 con 0x7f8b2c1559e0 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b32afb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c14f2c0 con 0x7f8b2c0aa180 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b2bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c14bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b2bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c14bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58352/0 (socket says 172.21.15.94:58352) 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b2bfff640 1 -- 172.21.15.94:0/588543776 learned_addr learned my addr 172.21.15.94:0/588543776 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b30897640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c0aa180 0x7f8b2c144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b31098640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b2c1559e0 0x7f8b2c143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b2bfff640 1 -- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c0aa180 msgr2=0x7f8b2c144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b2bfff640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c0aa180 0x7f8b2c144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b2bfff640 1 -- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b2c1559e0 msgr2=0x7f8b2c143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.268+0000 7f8b2bfff640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b2c1559e0 0x7f8b2c143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.269+0000 7f8b2bfff640 1 -- 172.21.15.94:0/588543776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c145180 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.269+0000 7f8b2bfff640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c14bef0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000e8a0 tx=0x7f8b2000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.269+0000 7f8b297fa640 1 -- 172.21.15.94:0/588543776 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20017070 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.269+0000 7f8b297fa640 1 -- 172.21.15.94:0/588543776 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2000c940 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.269+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2c152b70 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.269+0000 7f8b297fa640 1 -- 172.21.15.94:0/588543776 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000cd30 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.269+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2c144bc0 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.270+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b2c1554a0 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.270+0000 7f8b297fa640 1 -- 172.21.15.94:0/588543776 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b2001a070 con 0x7f8b2c1550c0 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.271+0000 7f8b297fa640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b1806b4d0 0x7f8b1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.274 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.271+0000 7f8b297fa640 1 -- 172.21.15.94:0/588543776 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8b2008a1e0 con 0x7f8b2c1550c0 2024-03-06T07:29:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.271+0000 7f8b30897640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b1806b4d0 0x7f8b1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.271+0000 7f8b30897640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b1806b4d0 0x7f8b1806d990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c035000 tx=0x7f8b1c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.275+0000 7f8b297fa640 1 -- 172.21.15.94:0/588543776 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b2c1554a0 con 0x7f8b2c1550c0 2024-03-06T07:29:52.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.485+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b2c04f330 con 0x7f8b2c1550c0 2024-03-06T07:29:52.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.486+0000 7f8b297fa640 1 -- 172.21.15.94:0/588543776 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f8b2c145180 con 0x7f8b2c1550c0 2024-03-06T07:29:52.488 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:52.488 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:52.488 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b1806b4d0 msgr2=0x7f8b1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b1806b4d0 0x7f8b1806d990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c035000 tx=0x7f8b1c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 msgr2=0x7f8b2c14bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c14bef0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000e8a0 tx=0x7f8b2000ed60 comp rx=0 tx=0).stop 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 shutdown_connections 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8b1806b4d0 0x7f8b1806d990 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b2c1559e0 0x7f8b2c143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8b2c1550c0 0x7f8b2c14bef0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 --2- 172.21.15.94:0/588543776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b2c0aa180 0x7f8b2c144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 >> 172.21.15.94:0/588543776 conn(0x7f8b2c0c6710 msgr2=0x7f8b2c0c3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 shutdown_connections 2024-03-06T07:29:52.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:52.488+0000 7f8b32afb640 1 -- 172.21.15.94:0/588543776 wait complete. 2024-03-06T07:29:52.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:52.905 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:53.906 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.016+0000 7f4d66e0e640 1 Processor -- start 2024-03-06T07:29:54.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.016+0000 7f4d66e0e640 1 -- start start 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.016+0000 7f4d66e0e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d6014f310 0x7f4d600c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.016+0000 7f4d66e0e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d60155bb0 con 0x7f4d601550c0 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.016+0000 7f4d66e0e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600aa660 con 0x7f4d600aa180 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.016+0000 7f4d66e0e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600c4610 con 0x7f4d6014f310 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d64baa640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4d601550c0 0x7f4d601554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57450/0 (socket says 172.21.15.94:57450) 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d64baa640 1 -- 172.21.15.94:0/3804735848 learned_addr learned my addr 172.21.15.94:0/3804735848 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d653ab640 1 --2- 172.21.15.94:0/3804735848 >> v2:172.21.15.145:3300/0 conn(0x7f4d6014f310 0x7f4d600c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d5f7fe640 1 -- 172.21.15.94:0/3804735848 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874562832 0 0) 0x7f4d60155bb0 con 0x7f4d601550c0 2024-03-06T07:29:54.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d5f7fe640 1 -- 172.21.15.94:0/3804735848 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d4c003520 con 0x7f4d601550c0 2024-03-06T07:29:54.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d653ab640 1 -- 172.21.15.94:0/3804735848 >> v1:172.21.15.145:6789/0 conn(0x7f4d600aa180 legacy=0x7f4d6014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d653ab640 1 -- 172.21.15.94:0/3804735848 >> v1:172.21.15.94:6789/0 conn(0x7f4d601550c0 legacy=0x7f4d601554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.017+0000 7f4d653ab640 1 -- 172.21.15.94:0/3804735848 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600aa940 con 0x7f4d6014f310 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.018+0000 7f4d653ab640 1 --2- 172.21.15.94:0/3804735848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6014f310 0x7f4d600c5060 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4d500097a0 tx=0x7f4d5002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e91bf3d0afe31b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.018+0000 7f4d5f7fe640 1 -- 172.21.15.94:0/3804735848 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500315a0 con 0x7f4d6014f310 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.018+0000 7f4d5f7fe640 1 -- 172.21.15.94:0/3804735848 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50037070 con 0x7f4d6014f310 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.018+0000 7f4d5f7fe640 1 -- 172.21.15.94:0/3804735848 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50031940 con 0x7f4d6014f310 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.018+0000 7f4d66e0e640 1 -- 172.21.15.94:0/3804735848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6014f310 msgr2=0x7f4d600c5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.018+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/3804735848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6014f310 0x7f4d600c5060 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4d500097a0 tx=0x7f4d5002b840 comp rx=0 tx=0).stop 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.019+0000 7f4d66e0e640 1 -- 172.21.15.94:0/3804735848 shutdown_connections 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.019+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/3804735848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6014f310 0x7f4d600c5060 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.019+0000 7f4d66e0e640 1 -- 172.21.15.94:0/3804735848 >> 172.21.15.94:0/3804735848 conn(0x7f4d600c6710 msgr2=0x7f4d600c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.019+0000 7f4d66e0e640 1 -- 172.21.15.94:0/3804735848 shutdown_connections 2024-03-06T07:29:54.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.019+0000 7f4d66e0e640 1 -- 172.21.15.94:0/3804735848 wait complete. 2024-03-06T07:29:54.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.020+0000 7f4d66e0e640 1 Processor -- start 2024-03-06T07:29:54.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.020+0000 7f4d66e0e640 1 -- start start 2024-03-06T07:29:54.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.020+0000 7f4d66e0e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600aa180 0x7f4d601542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d66e0e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6014f310 0x7f4d600a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d66e0e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d601550c0 0x7f4d60160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d66e0e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c1490 con 0x7f4d601550c0 2024-03-06T07:29:54.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d66e0e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c1310 con 0x7f4d600aa180 2024-03-06T07:29:54.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d66e0e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c3210 con 0x7f4d6014f310 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d601550c0 0x7f4d60160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d601550c0 0x7f4d60160c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58354/0 (socket says 172.21.15.94:58354) 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 -- 172.21.15.94:0/440981081 learned_addr learned my addr 172.21.15.94:0/440981081 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d64baa640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600aa180 0x7f4d601542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d5ffff640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6014f310 0x7f4d600a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 -- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6014f310 msgr2=0x7f4d600a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6014f310 0x7f4d600a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 -- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600aa180 msgr2=0x7f4d601542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600aa180 0x7f4d601542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 -- 172.21.15.94:0/440981081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6014d660 con 0x7f4d601550c0 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d653ab640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d601550c0 0x7f4d60160c70 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f4d500097a0 tx=0x7f4d500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d5d7fa640 1 -- 172.21.15.94:0/440981081 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50004230 con 0x7f4d601550c0 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d5d7fa640 1 -- 172.21.15.94:0/440981081 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d500043f0 con 0x7f4d601550c0 2024-03-06T07:29:54.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d600a9da0 con 0x7f4d601550c0 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d5d7fa640 1 -- 172.21.15.94:0/440981081 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50003850 con 0x7f4d601550c0 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.021+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6006dd20 con 0x7f4d601550c0 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.023+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6004f330 con 0x7f4d601550c0 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.023+0000 7f4d5d7fa640 1 -- 172.21.15.94:0/440981081 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d500039f0 con 0x7f4d601550c0 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.023+0000 7f4d5d7fa640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.023+0000 7f4d5d7fa640 1 -- 172.21.15.94:0/440981081 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f4d5002f030 con 0x7f4d601550c0 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.023+0000 7f4d64baa640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.024+0000 7f4d64baa640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f4d58001560 tx=0x7f4d580019f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.028+0000 7f4d5d7fa640 1 -- 172.21.15.94:0/440981081 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d50041070 con 0x7f4d601550c0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.235+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d6014ed10 con 0x7f4d601550c0 2024-03-06T07:29:54.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.235+0000 7f4d5d7fa640 1 -- 172.21.15.94:0/440981081 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f4d5007ab70 con 0x7f4d601550c0 2024-03-06T07:29:54.238 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:54.238 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:54.238 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:54.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 msgr2=0x7f4d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f4d58001560 tx=0x7f4d580019f0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d601550c0 msgr2=0x7f4d60160c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d601550c0 0x7f4d60160c70 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f4d500097a0 tx=0x7f4d500046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 shutdown_connections 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4d4006b4d0 0x7f4d4006d990 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d601550c0 0x7f4d60160c70 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6014f310 0x7f4d600a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 --2- 172.21.15.94:0/440981081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600aa180 0x7f4d601542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 >> 172.21.15.94:0/440981081 conn(0x7f4d600c6710 msgr2=0x7f4d60144780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 shutdown_connections 2024-03-06T07:29:54.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:54.237+0000 7f4d66e0e640 1 -- 172.21.15.94:0/440981081 wait complete. 2024-03-06T07:29:54.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:54.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:55.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:55.765 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.764+0000 7f9ded7e9640 1 Processor -- start 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.764+0000 7f9ded7e9640 1 -- start start 2024-03-06T07:29:55.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.764+0000 7f9ded7e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9de814f360 0x7f9de8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.764+0000 7f9ded7e9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80c73d0 con 0x7f9de8158b60 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.764+0000 7f9ded7e9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de80be2e0 con 0x7f9de80c88c0 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.764+0000 7f9ded7e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de8159700 con 0x7f9de814f360 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.765+0000 7f9de6ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9de814f360 0x7f9de8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.765+0000 7f9de77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9de8158b60 0x7f9de8158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57464/0 (socket says 172.21.15.94:57464) 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.765+0000 7f9de77fe640 1 -- 172.21.15.94:0/996909309 learned_addr learned my addr 172.21.15.94:0/996909309 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.765+0000 7f9de6ffd640 1 -- 172.21.15.94:0/996909309 >> v1:172.21.15.145:6789/0 conn(0x7f9de80c88c0 legacy=0x7f9de80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.765+0000 7f9de6ffd640 1 -- 172.21.15.94:0/996909309 >> v1:172.21.15.94:6789/0 conn(0x7f9de8158b60 legacy=0x7f9de8158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.765+0000 7f9de6ffd640 1 -- 172.21.15.94:0/996909309 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de80c7670 con 0x7f9de814f360 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9de6ffd640 1 --2- 172.21.15.94:0/996909309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8155c10 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80097a0 tx=0x7f9dd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6224a0b59cd501e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9de5ffb640 1 -- 172.21.15.94:0/996909309 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8035020 con 0x7f9de814f360 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9de5ffb640 1 -- 172.21.15.94:0/996909309 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd8031830 con 0x7f9de814f360 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9de5ffb640 1 -- 172.21.15.94:0/996909309 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8031c00 con 0x7f9de814f360 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9ded7e9640 1 -- 172.21.15.94:0/996909309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 msgr2=0x7f9de8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/996909309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8155c10 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80097a0 tx=0x7f9dd802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9ded7e9640 1 -- 172.21.15.94:0/996909309 shutdown_connections 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/996909309 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8155c10 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9ded7e9640 1 -- 172.21.15.94:0/996909309 >> 172.21.15.94:0/996909309 conn(0x7f9de80ac570 msgr2=0x7f9de80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9ded7e9640 1 -- 172.21.15.94:0/996909309 shutdown_connections 2024-03-06T07:29:55.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.766+0000 7f9ded7e9640 1 -- 172.21.15.94:0/996909309 wait complete. 2024-03-06T07:29:55.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 Processor -- start 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 -- start start 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9de80c88c0 0x7f9de80ad660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9de8158b60 0x7f9de81e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80c7c30 con 0x7f9de814f360 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80c5a50 con 0x7f9de8158b60 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9ded7e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80c5bd0 con 0x7f9de80c88c0 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9de67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9de67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58370/0 (socket says 172.21.15.94:58370) 2024-03-06T07:29:55.772 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.768+0000 7f9de67fc640 1 -- 172.21.15.94:0/1725081678 learned_addr learned my addr 172.21.15.94:0/1725081678 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de6ffd640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9de80c88c0 0x7f9de80ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de77fe640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9de8158b60 0x7f9de81e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de67fc640 1 -- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9de80c88c0 msgr2=0x7f9de80ad660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de67fc640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9de80c88c0 0x7f9de80ad660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de67fc640 1 -- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9de8158b60 msgr2=0x7f9de81e21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de67fc640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9de8158b60 0x7f9de81e21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de67fc640 1 -- 172.21.15.94:0/1725081678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de81ee270 con 0x7f9de814f360 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9de67fc640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8157970 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c7a0 tx=0x7f9dd000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9ddf7fe640 1 -- 172.21.15.94:0/1725081678 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0016020 con 0x7f9de814f360 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9ddf7fe640 1 -- 172.21.15.94:0/1725081678 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd0006d50 con 0x7f9de814f360 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de81e16e0 con 0x7f9de814f360 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9ddf7fe640 1 -- 172.21.15.94:0/1725081678 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0003bb0 con 0x7f9de814f360 2024-03-06T07:29:55.773 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.769+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de8157eb0 con 0x7f9de814f360 2024-03-06T07:29:55.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.770+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de804f330 con 0x7f9de814f360 2024-03-06T07:29:55.774 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.771+0000 7f9ddf7fe640 1 -- 172.21.15.94:0/1725081678 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dd000a7f0 con 0x7f9de814f360 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.771+0000 7f9ddf7fe640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.771+0000 7f9ddf7fe640 1 -- 172.21.15.94:0/1725081678 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9dd0089f20 con 0x7f9de814f360 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.771+0000 7f9de6ffd640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.772+0000 7f9de6ffd640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dc806b4d0 0x7f9dc806d990 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8036000 tx=0x7f9dd8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.776+0000 7f9ddf7fe640 1 -- 172.21.15.94:0/1725081678 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dd0020020 con 0x7f9de814f360 2024-03-06T07:29:55.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.981+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9de80c8d50 con 0x7f9de814f360 2024-03-06T07:29:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.982+0000 7f9ddf7fe640 1 -- 172.21.15.94:0/1725081678 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f9dd0058550 con 0x7f9de814f360 2024-03-06T07:29:55.984 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:55.984 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:55.984 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dc806b4d0 msgr2=0x7f9dc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dc806b4d0 0x7f9dc806d990 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8036000 tx=0x7f9dd8008030 comp rx=0 tx=0).stop 2024-03-06T07:29:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 msgr2=0x7f9de8157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8157970 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c7a0 tx=0x7f9dd000cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 shutdown_connections 2024-03-06T07:29:55.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f9dc806b4d0 0x7f9dc806d990 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9de8158b60 0x7f9de81e21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9de814f360 0x7f9de8157970 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 --2- 172.21.15.94:0/1725081678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9de80c88c0 0x7f9de80ad660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 >> 172.21.15.94:0/1725081678 conn(0x7f9de80ac570 msgr2=0x7f9de80ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 shutdown_connections 2024-03-06T07:29:55.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:55.984+0000 7f9ded7e9640 1 -- 172.21.15.94:0/1725081678 wait complete. 2024-03-06T07:29:56.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:56.399 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:57.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:57.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.511+0000 7fa28d313640 1 Processor -- start 2024-03-06T07:29:57.513 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.511+0000 7fa28d313640 1 -- start start 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa28d313640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa28806d9f0 0x7fa2880a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa28d313640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa2880c3280 0x7fa2880c3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa28d313640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa288158800 con 0x7fa2881551e0 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa28d313640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa288158ae0 con 0x7fa28806d9f0 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa28d313640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa288158dc0 con 0x7fa2880c3280 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa286ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa28806d9f0 0x7fa2880a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa286ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa28806d9f0 0x7fa2880a7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58384/0 (socket says 172.21.15.94:58384) 2024-03-06T07:29:57.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa286ffd640 1 -- 172.21.15.94:0/1962643468 learned_addr learned my addr 172.21.15.94:0/1962643468 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:57.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa27ffff640 1 --2- 172.21.15.94:0/1962643468 >> v2:172.21.15.145:3300/0 conn(0x7fa2880c3280 0x7fa2880c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa286ffd640 1 -- 172.21.15.94:0/1962643468 >> v2:172.21.15.145:3300/0 conn(0x7fa2880c3280 msgr2=0x7fa2880c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa286ffd640 1 --2- 172.21.15.94:0/1962643468 >> v2:172.21.15.145:3300/0 conn(0x7fa2880c3280 0x7fa2880c3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa286ffd640 1 -- 172.21.15.94:0/1962643468 >> v1:172.21.15.145:6789/0 conn(0x7fa2881551e0 legacy=0x7fa2881555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.512+0000 7fa286ffd640 1 -- 172.21.15.94:0/1962643468 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2881590a0 con 0x7fa28806d9f0 2024-03-06T07:29:57.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.513+0000 7fa286ffd640 1 --2- 172.21.15.94:0/1962643468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa2880a7cd0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fa2880721a0 tx=0x7fa27802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c4acbe9142d340 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.513+0000 7fa2867fc640 1 -- 172.21.15.94:0/1962643468 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278035020 con 0x7fa28806d9f0 2024-03-06T07:29:57.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.513+0000 7fa2867fc640 1 -- 172.21.15.94:0/1962643468 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa278031870 con 0x7fa28806d9f0 2024-03-06T07:29:57.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.513+0000 7fa2867fc640 1 -- 172.21.15.94:0/1962643468 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278031c40 con 0x7fa28806d9f0 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 -- 172.21.15.94:0/1962643468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 msgr2=0x7fa2880a7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 --2- 172.21.15.94:0/1962643468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa2880a7cd0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fa2880721a0 tx=0x7fa27802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 -- 172.21.15.94:0/1962643468 shutdown_connections 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 --2- 172.21.15.94:0/1962643468 >> v2:172.21.15.145:3300/0 conn(0x7fa2880c3280 0x7fa2880c3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 --2- 172.21.15.94:0/1962643468 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa2880a7cd0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 -- 172.21.15.94:0/1962643468 >> 172.21.15.94:0/1962643468 conn(0x7fa2880aa630 msgr2=0x7fa2880aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 -- 172.21.15.94:0/1962643468 shutdown_connections 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.514+0000 7fa28d313640 1 -- 172.21.15.94:0/1962643468 wait complete. 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 Processor -- start 2024-03-06T07:29:57.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 -- start start 2024-03-06T07:29:57.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa288157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2880c3280 0x7fa28814eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2881551e0 0x7fa28814f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28806df50 con 0x7fa28806d9f0 2024-03-06T07:29:57.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28806ddd0 con 0x7fa2880c3280 2024-03-06T07:29:57.519 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.515+0000 7fa28d313640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880a8210 con 0x7fa2881551e0 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa288157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa288157cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:58398/0 (socket says 172.21.15.94:58398) 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 -- 172.21.15.94:0/3560608545 learned_addr learned my addr 172.21.15.94:0/3560608545 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa27ffff640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2880c3280 0x7fa28814eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa2877fe640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2881551e0 0x7fa28814f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 -- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2881551e0 msgr2=0x7fa28814f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2881551e0 0x7fa28814f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 -- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2880c3280 msgr2=0x7fa28814eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2880c3280 0x7fa28814eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 -- 172.21.15.94:0/3560608545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2881512c0 con 0x7fa28806d9f0 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa286ffd640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa288157cc0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fa278000c00 tx=0x7fa2780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa27f7fe640 1 -- 172.21.15.94:0/3560608545 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278035040 con 0x7fa28806d9f0 2024-03-06T07:29:57.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa27f7fe640 1 -- 172.21.15.94:0/3560608545 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa278030d00 con 0x7fa28806d9f0 2024-03-06T07:29:57.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa288070210 con 0x7fa28806d9f0 2024-03-06T07:29:57.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa27f7fe640 1 -- 172.21.15.94:0/3560608545 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2780456a0 con 0x7fa28806d9f0 2024-03-06T07:29:57.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.516+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2881ee010 con 0x7fa28806d9f0 2024-03-06T07:29:57.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.517+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28804f330 con 0x7fa28806d9f0 2024-03-06T07:29:57.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.518+0000 7fa27f7fe640 1 -- 172.21.15.94:0/3560608545 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa278038020 con 0x7fa28806d9f0 2024-03-06T07:29:57.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.518+0000 7fa27f7fe640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26806b500 0x7fa26806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.518+0000 7fa27f7fe640 1 -- 172.21.15.94:0/3560608545 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa2780ac5d0 con 0x7fa28806d9f0 2024-03-06T07:29:57.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.518+0000 7fa27ffff640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26806b500 0x7fa26806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.522 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.519+0000 7fa27ffff640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26806b500 0x7fa26806d9c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa2880c3c10 tx=0x7fa270009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.524 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.522+0000 7fa27f7fe640 1 -- 172.21.15.94:0/3560608545 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa27807ac00 con 0x7fa28806d9f0 2024-03-06T07:29:57.733 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.731+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa288073240 con 0x7fa28806d9f0 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.732+0000 7fa27f7fe640 1 -- 172.21.15.94:0/3560608545 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fa27807eca0 con 0x7fa28806d9f0 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:57.734 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:57.735 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:57.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26806b500 msgr2=0x7fa26806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26806b500 0x7fa26806d9c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa2880c3c10 tx=0x7fa270009190 comp rx=0 tx=0).stop 2024-03-06T07:29:57.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 msgr2=0x7fa288157cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa288157cc0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fa278000c00 tx=0x7fa2780046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 shutdown_connections 2024-03-06T07:29:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa26806b500 0x7fa26806d9c0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2881551e0 0x7fa28814f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2880c3280 0x7fa28814eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 --2- 172.21.15.94:0/3560608545 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28806d9f0 0x7fa288157cc0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 >> 172.21.15.94:0/3560608545 conn(0x7fa2880aa630 msgr2=0x7fa2880c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 shutdown_connections 2024-03-06T07:29:57.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:57.734+0000 7fa28d313640 1 -- 172.21.15.94:0/3560608545 wait complete. 2024-03-06T07:29:58.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:58.149 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:29:59.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:59.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.259+0000 7fec37c64640 1 Processor -- start 2024-03-06T07:29:59.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.260+0000 7fec37c64640 1 -- start start 2024-03-06T07:29:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.260+0000 7fec37c64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec3006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.260+0000 7fec37c64640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec30155c80 0x7fec300c12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.260+0000 7fec37c64640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fec300720a0 0x7fec3006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.260+0000 7fec37c64640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec300c1dc0 con 0x7fec300720a0 2024-03-06T07:29:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.260+0000 7fec37c64640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec300c1ae0 con 0x7fec30155c80 2024-03-06T07:29:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.260+0000 7fec37c64640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3006e450 con 0x7fec30152520 2024-03-06T07:29:59.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec3006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec351ff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec30155c80 0x7fec300c12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec3006ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:53330/0 (socket says 172.21.15.94:53330) 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 -- 172.21.15.94:0/2057722578 learned_addr learned my addr 172.21.15.94:0/2057722578 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec36201640 1 --2- 172.21.15.94:0/2057722578 >> v2:172.21.15.94:3300/0 conn(0x7fec300720a0 0x7fec3006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 -- 172.21.15.94:0/2057722578 >> v2:172.21.15.145:3300/0 conn(0x7fec30155c80 msgr2=0x7fec300c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 --2- 172.21.15.94:0/2057722578 >> v2:172.21.15.145:3300/0 conn(0x7fec30155c80 0x7fec300c12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 -- 172.21.15.94:0/2057722578 >> v2:172.21.15.94:3300/0 conn(0x7fec300720a0 msgr2=0x7fec3006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 --2- 172.21.15.94:0/2057722578 >> v2:172.21.15.94:3300/0 conn(0x7fec300720a0 0x7fec3006afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec35a00640 1 -- 172.21.15.94:0/2057722578 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3006e730 con 0x7fec30152520 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec351ff640 1 --2- 172.21.15.94:0/2057722578 >> v2:172.21.15.145:3300/0 conn(0x7fec30155c80 0x7fec300c12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:59.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.261+0000 7fec36201640 1 --2- 172.21.15.94:0/2057722578 >> v2:172.21.15.94:3300/0 conn(0x7fec300720a0 0x7fec3006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.262+0000 7fec35a00640 1 --2- 172.21.15.94:0/2057722578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec3006ef30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fec300c5230 tx=0x7fec2402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46cdeb3e56e80925 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.262+0000 7fec349fe640 1 -- 172.21.15.94:0/2057722578 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2402ca70 con 0x7fec30152520 2024-03-06T07:29:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.262+0000 7fec349fe640 1 -- 172.21.15.94:0/2057722578 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2402cc10 con 0x7fec30152520 2024-03-06T07:29:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.262+0000 7fec349fe640 1 -- 172.21.15.94:0/2057722578 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec240343f0 con 0x7fec30152520 2024-03-06T07:29:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 -- 172.21.15.94:0/2057722578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 msgr2=0x7fec3006ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 --2- 172.21.15.94:0/2057722578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec3006ef30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fec300c5230 tx=0x7fec2402bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 -- 172.21.15.94:0/2057722578 shutdown_connections 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 --2- 172.21.15.94:0/2057722578 >> v2:172.21.15.94:3300/0 conn(0x7fec300720a0 0x7fec3006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 --2- 172.21.15.94:0/2057722578 >> v2:172.21.15.145:3300/0 conn(0x7fec30155c80 0x7fec300c12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 --2- 172.21.15.94:0/2057722578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec3006ef30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 -- 172.21.15.94:0/2057722578 >> 172.21.15.94:0/2057722578 conn(0x7fec300c76b0 msgr2=0x7fec300c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 -- 172.21.15.94:0/2057722578 shutdown_connections 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.263+0000 7fec37c64640 1 -- 172.21.15.94:0/2057722578 wait complete. 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.264+0000 7fec37c64640 1 Processor -- start 2024-03-06T07:29:59.267 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.264+0000 7fec37c64640 1 -- start start 2024-03-06T07:29:59.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec300720a0 0x7fec30154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.270 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec301e1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec30155c80 0x7fec301e2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec30152900 con 0x7fec30155c80 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec300c17f0 con 0x7fec300720a0 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec30152a80 con 0x7fec30152520 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec30155c80 0x7fec301e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec30155c80 0x7fec301e2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44674/0 (socket says 172.21.15.94:44674) 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 -- 172.21.15.94:0/2966248912 learned_addr learned my addr 172.21.15.94:0/2966248912 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec35a00640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec300720a0 0x7fec30154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec351ff640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec301e1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.271 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 -- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 msgr2=0x7fec301e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec301e1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 -- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec300720a0 msgr2=0x7fec30154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec300720a0 0x7fec30154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 -- 172.21.15.94:0/2966248912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3014d1b0 con 0x7fec30155c80 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec36201640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec30155c80 0x7fec301e2130 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fec2800e8b0 tx=0x7fec2800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec227fc640 1 -- 172.21.15.94:0/2966248912 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28017070 con 0x7fec30155c80 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec227fc640 1 -- 172.21.15.94:0/2966248912 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec28004020 con 0x7fec30155c80 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec301ead60 con 0x7fec30155c80 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec227fc640 1 -- 172.21.15.94:0/2966248912 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec280043f0 con 0x7fec30155c80 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.265+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec301eb320 con 0x7fec30155c80 2024-03-06T07:29:59.272 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.266+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3004f330 con 0x7fec30155c80 2024-03-06T07:29:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.267+0000 7fec227fc640 1 -- 172.21.15.94:0/2966248912 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec2800c940 con 0x7fec30155c80 2024-03-06T07:29:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.267+0000 7fec227fc640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec1406b4d0 0x7fec1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.267+0000 7fec227fc640 1 -- 172.21.15.94:0/2966248912 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fec2808a0c0 con 0x7fec30155c80 2024-03-06T07:29:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.267+0000 7fec35a00640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec1406b4d0 0x7fec1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.268+0000 7fec35a00640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec1406b4d0 0x7fec1406d990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fec24035000 tx=0x7fec24030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.272+0000 7fec227fc640 1 -- 172.21.15.94:0/2966248912 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec3014d1b0 con 0x7fec30155c80 2024-03-06T07:29:59.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.477+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec300c0b70 con 0x7fec30155c80 2024-03-06T07:29:59.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.478+0000 7fec227fc640 1 -- 172.21.15.94:0/2966248912 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fec280586f0 con 0x7fec30155c80 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec1406b4d0 msgr2=0x7fec1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec1406b4d0 0x7fec1406d990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fec24035000 tx=0x7fec24030000 comp rx=0 tx=0).stop 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec30155c80 msgr2=0x7fec301e2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec30155c80 0x7fec301e2130 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fec2800e8b0 tx=0x7fec2800ed70 comp rx=0 tx=0).stop 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 shutdown_connections 2024-03-06T07:29:59.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec1406b4d0 0x7fec1406d990 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec30155c80 0x7fec301e2130 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec30152520 0x7fec301e1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 --2- 172.21.15.94:0/2966248912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec300720a0 0x7fec30154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 >> 172.21.15.94:0/2966248912 conn(0x7fec300c76b0 msgr2=0x7fec300c6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 shutdown_connections 2024-03-06T07:29:59.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:29:59.480+0000 7fec37c64640 1 -- 172.21.15.94:0/2966248912 wait complete. 2024-03-06T07:29:59.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:29:59.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:00.897 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:01.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.008+0000 7f11949e2640 1 Processor -- start 2024-03-06T07:30:01.010 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.008+0000 7f11949e2640 1 -- start start 2024-03-06T07:30:01.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.009+0000 7f11949e2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f11900c8a00 0x7f1190158f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.009+0000 7f11949e2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1190159460 0x7f11900c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.009+0000 7f11949e2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11900c7760 con 0x7f1190152920 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.009+0000 7f11949e2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11901587b0 con 0x7f1190159460 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.009+0000 7f11949e2640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11900c7a40 con 0x7f11900c8a00 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.009+0000 7f118ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1190159460 0x7f11900c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.010+0000 7f118ed9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1190159460 0x7f11900c5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44688/0 (socket says 172.21.15.94:44688) 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.010+0000 7f118ed9d640 1 -- 172.21.15.94:0/2424944761 learned_addr learned my addr 172.21.15.94:0/2424944761 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.010+0000 7f118dd9b640 1 --2- 172.21.15.94:0/2424944761 >> v2:172.21.15.145:3300/0 conn(0x7f11900c8a00 0x7f1190158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.010+0000 7f118ed9d640 1 -- 172.21.15.94:0/2424944761 >> v2:172.21.15.145:3300/0 conn(0x7f11900c8a00 msgr2=0x7f1190158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.010+0000 7f118ed9d640 1 --2- 172.21.15.94:0/2424944761 >> v2:172.21.15.145:3300/0 conn(0x7f11900c8a00 0x7f1190158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.010+0000 7f118ed9d640 1 -- 172.21.15.94:0/2424944761 >> v1:172.21.15.145:6789/0 conn(0x7f1190152920 legacy=0x7f1190155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.010+0000 7f118ed9d640 1 -- 172.21.15.94:0/2424944761 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11900c7d20 con 0x7f1190159460 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f118ed9d640 1 --2- 172.21.15.94:0/2424944761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190159460 0x7f11900c5330 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f1190067c30 tx=0x7f118802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a32c2f80c73ad66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f118d59a640 1 -- 172.21.15.94:0/2424944761 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118802ca80 con 0x7f1190159460 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f118d59a640 1 -- 172.21.15.94:0/2424944761 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118802cc20 con 0x7f1190159460 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f118d59a640 1 -- 172.21.15.94:0/2424944761 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11880343f0 con 0x7f1190159460 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 -- 172.21.15.94:0/2424944761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190159460 msgr2=0x7f11900c5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 --2- 172.21.15.94:0/2424944761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190159460 0x7f11900c5330 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f1190067c30 tx=0x7f118802bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 -- 172.21.15.94:0/2424944761 shutdown_connections 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 --2- 172.21.15.94:0/2424944761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190159460 0x7f11900c5330 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 --2- 172.21.15.94:0/2424944761 >> v2:172.21.15.145:3300/0 conn(0x7f11900c8a00 0x7f1190158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 -- 172.21.15.94:0/2424944761 >> 172.21.15.94:0/2424944761 conn(0x7f11900ac560 msgr2=0x7f11900ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 -- 172.21.15.94:0/2424944761 shutdown_connections 2024-03-06T07:30:01.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.011+0000 7f11949e2640 1 -- 172.21.15.94:0/2424944761 wait complete. 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 Processor -- start 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 -- start start 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11900c8a00 0x7f1190155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190152920 0x7f11900c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1190147290 0x7f11901450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11900c7390 con 0x7f1190152920 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119006d030 con 0x7f1190147290 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f11949e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11900c7510 con 0x7f11900c8a00 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f118dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190152920 0x7f11900c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f118dd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190152920 0x7f11900c4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44692/0 (socket says 172.21.15.94:44692) 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.013+0000 7f118dd9b640 1 -- 172.21.15.94:0/276819878 learned_addr learned my addr 172.21.15.94:0/276819878 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118e59c640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11900c8a00 0x7f1190155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118ed9d640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1190147290 0x7f11901450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118dd9b640 1 -- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11900c8a00 msgr2=0x7f1190155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118dd9b640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11900c8a00 0x7f1190155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118dd9b640 1 -- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1190147290 msgr2=0x7f11901450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118dd9b640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1190147290 0x7f11901450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118dd9b640 1 -- 172.21.15.94:0/276819878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11901530d0 con 0x7f1190152920 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f118dd9b640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190152920 0x7f11900c4620 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f117800e9b0 tx=0x7f117800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f1186ffd640 1 -- 172.21.15.94:0/276819878 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178017070 con 0x7f1190152920 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f1186ffd640 1 -- 172.21.15.94:0/276819878 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117800c9a0 con 0x7f1190152920 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1190156d40 con 0x7f1190152920 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f1186ffd640 1 -- 172.21.15.94:0/276819878 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117800cd90 con 0x7f1190152920 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.014+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1190157000 con 0x7f1190152920 2024-03-06T07:30:01.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.015+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119004f330 con 0x7f1190152920 2024-03-06T07:30:01.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.016+0000 7f1186ffd640 1 -- 172.21.15.94:0/276819878 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f117801a070 con 0x7f1190152920 2024-03-06T07:30:01.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.016+0000 7f1186ffd640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f117006b4d0 0x7f117006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.016+0000 7f1186ffd640 1 -- 172.21.15.94:0/276819878 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f117808a380 con 0x7f1190152920 2024-03-06T07:30:01.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.016+0000 7f118e59c640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f117006b4d0 0x7f117006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.017+0000 7f118e59c640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f117006b4d0 0x7f117006d990 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1180001780 tx=0x7f1180009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.020+0000 7f1186ffd640 1 -- 172.21.15.94:0/276819878 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11780589b0 con 0x7f1190152920 2024-03-06T07:30:01.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.228+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1190153ba0 con 0x7f1190152920 2024-03-06T07:30:01.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.228+0000 7f1186ffd640 1 -- 172.21.15.94:0/276819878 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f117805ca50 con 0x7f1190152920 2024-03-06T07:30:01.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:01.230 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:01.231 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:01.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f117006b4d0 msgr2=0x7f117006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f117006b4d0 0x7f117006d990 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1180001780 tx=0x7f1180009040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190152920 msgr2=0x7f11900c4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190152920 0x7f11900c4620 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f117800e9b0 tx=0x7f117800ee70 comp rx=0 tx=0).stop 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 shutdown_connections 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f117006b4d0 0x7f117006d990 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1190147290 0x7f11901450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1190152920 0x7f11900c4620 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 --2- 172.21.15.94:0/276819878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11900c8a00 0x7f1190155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 >> 172.21.15.94:0/276819878 conn(0x7f11900ac560 msgr2=0x7f11900ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 shutdown_connections 2024-03-06T07:30:01.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:01.230+0000 7f11949e2640 1 -- 172.21.15.94:0/276819878 wait complete. 2024-03-06T07:30:01.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:01.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:02.647 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:02.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.757+0000 7f165995e640 1 Processor -- start 2024-03-06T07:30:02.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f165995e640 1 -- start start 2024-03-06T07:30:02.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f165995e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f16541506b0 0x7f1654153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f165995e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1654070aa0 con 0x7f1654156ea0 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f165995e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1654070d80 con 0x7f16540acc00 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f165995e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1654156450 con 0x7f16541506b0 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f1652ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f16541506b0 0x7f1654153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f16527fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1654156ea0 0x7f16540702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50198/0 (socket says 172.21.15.94:50198) 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.758+0000 7f16527fc640 1 -- 172.21.15.94:0/3087338962 learned_addr learned my addr 172.21.15.94:0/3087338962 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.759+0000 7f1652ffd640 1 -- 172.21.15.94:0/3087338962 >> v1:172.21.15.145:6789/0 conn(0x7f16540acc00 legacy=0x7f16540c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.759+0000 7f1652ffd640 1 -- 172.21.15.94:0/3087338962 >> v1:172.21.15.94:6789/0 conn(0x7f1654156ea0 legacy=0x7f16540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.759+0000 7f1651ffb640 1 -- 172.21.15.94:0/3087338962 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544484352 0 0) 0x7f1654070d80 con 0x7f16540acc00 2024-03-06T07:30:02.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.759+0000 7f1652ffd640 1 -- 172.21.15.94:0/3087338962 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16541566a0 con 0x7f16541506b0 2024-03-06T07:30:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.759+0000 7f1652ffd640 1 --2- 172.21.15.94:0/3087338962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16541506b0 0x7f1654153990 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f1644009d90 tx=0x7f164402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922a08c942a80838 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f1651ffb640 1 -- 172.21.15.94:0/3087338962 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644039070 con 0x7f16541506b0 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f1651ffb640 1 -- 172.21.15.94:0/3087338962 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164402cab0 con 0x7f16541506b0 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f1651ffb640 1 -- 172.21.15.94:0/3087338962 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644034440 con 0x7f16541506b0 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f165995e640 1 -- 172.21.15.94:0/3087338962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16541506b0 msgr2=0x7f1654153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f165995e640 1 --2- 172.21.15.94:0/3087338962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16541506b0 0x7f1654153990 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f1644009d90 tx=0x7f164402bd80 comp rx=0 tx=0).stop 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f165995e640 1 -- 172.21.15.94:0/3087338962 shutdown_connections 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f165995e640 1 --2- 172.21.15.94:0/3087338962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16541506b0 0x7f1654153990 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f165995e640 1 -- 172.21.15.94:0/3087338962 >> 172.21.15.94:0/3087338962 conn(0x7f165406b1a0 msgr2=0x7f165406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f165995e640 1 -- 172.21.15.94:0/3087338962 shutdown_connections 2024-03-06T07:30:02.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.760+0000 7f165995e640 1 -- 172.21.15.94:0/3087338962 wait complete. 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.761+0000 7f165995e640 1 Processor -- start 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f165995e640 1 -- start start 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f165995e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16540acc00 0x7f16540a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f165995e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16541506b0 0x7f1654152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f165995e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1654156ea0 0x7f1654152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f165995e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540ad050 con 0x7f16540acc00 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f165995e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1654156a10 con 0x7f1654156ea0 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f165995e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16540ad1d0 con 0x7f16541506b0 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f1652ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16540acc00 0x7f16540a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f1652ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16540acc00 0x7f16540a8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44710/0 (socket says 172.21.15.94:44710) 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f1652ffd640 1 -- 172.21.15.94:0/3948376552 learned_addr learned my addr 172.21.15.94:0/3948376552 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.762+0000 7f16527fc640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16541506b0 0x7f1654152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f16537fe640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1654156ea0 0x7f1654152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f1652ffd640 1 -- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16541506b0 msgr2=0x7f1654152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f1652ffd640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16541506b0 0x7f1654152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f1652ffd640 1 -- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1654156ea0 msgr2=0x7f1654152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f1652ffd640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1654156ea0 0x7f1654152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f1652ffd640 1 -- 172.21.15.94:0/3948376552 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165414b390 con 0x7f16540acc00 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f1652ffd640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16540acc00 0x7f16540a8db0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f1644009d90 tx=0x7f1644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f163bfff640 1 -- 172.21.15.94:0/3948376552 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644044030 con 0x7f16540acc00 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f163bfff640 1 -- 172.21.15.94:0/3948376552 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164403f470 con 0x7f16540acc00 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165414b670 con 0x7f16540acc00 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f163bfff640 1 -- 172.21.15.94:0/3948376552 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644039040 con 0x7f16540acc00 2024-03-06T07:30:02.767 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.763+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165414bcf0 con 0x7f16540acc00 2024-03-06T07:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.764+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165414b840 con 0x7f16540acc00 2024-03-06T07:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.765+0000 7f163bfff640 1 -- 172.21.15.94:0/3948376552 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1644047070 con 0x7f16540acc00 2024-03-06T07:30:02.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.765+0000 7f163bfff640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164006b4d0 0x7f164006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.765+0000 7f163bfff640 1 -- 172.21.15.94:0/3948376552 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f16440acbf0 con 0x7f16540acc00 2024-03-06T07:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.765+0000 7f16527fc640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164006b4d0 0x7f164006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.766+0000 7f16527fc640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164006b4d0 0x7f164006d990 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f165414eae0 tx=0x7f1648007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.771 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.769+0000 7f163bfff640 1 -- 172.21.15.94:0/3948376552 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f164404e070 con 0x7f16540acc00 2024-03-06T07:30:02.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.977+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1654069460 con 0x7f16540acc00 2024-03-06T07:30:02.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.977+0000 7f163bfff640 1 -- 172.21.15.94:0/3948376552 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f1654069460 con 0x7f16540acc00 2024-03-06T07:30:02.979 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:02.979 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:02.979 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164006b4d0 msgr2=0x7f164006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164006b4d0 0x7f164006d990 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f165414eae0 tx=0x7f1648007e00 comp rx=0 tx=0).stop 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16540acc00 msgr2=0x7f16540a8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16540acc00 0x7f16540a8db0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f1644009d90 tx=0x7f1644004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 shutdown_connections 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f164006b4d0 0x7f164006d990 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1654156ea0 0x7f1654152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16541506b0 0x7f1654152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 --2- 172.21.15.94:0/3948376552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16540acc00 0x7f16540a8db0 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 >> 172.21.15.94:0/3948376552 conn(0x7f165406b1a0 msgr2=0x7f16540ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 shutdown_connections 2024-03-06T07:30:02.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:02.979+0000 7f165995e640 1 -- 172.21.15.94:0/3948376552 wait complete. 2024-03-06T07:30:03.393 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:03.394 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:04.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.504+0000 7ffbd1d55640 1 Processor -- start 2024-03-06T07:30:04.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.504+0000 7ffbd1d55640 1 -- start start 2024-03-06T07:30:04.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbd1d55640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffbcc152e00 0x7ffbcc158d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbd1d55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc159250 0x7ffbcc072950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbd1d55640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc0c2aa0 con 0x7ffbcc158930 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbd1d55640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc0c3490 con 0x7ffbcc152e00 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbd1d55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc0c3770 con 0x7ffbcc159250 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcaffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffbcc152e00 0x7ffbcc158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ffbcc158930 0x7ffbcc1526f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50214/0 (socket says 172.21.15.94:50214) 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcb7fe640 1 -- 172.21.15.94:0/88649122 learned_addr learned my addr 172.21.15.94:0/88649122 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcbfff640 1 --2- 172.21.15.94:0/88649122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc159250 0x7ffbcc072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcaffd640 1 -- 172.21.15.94:0/88649122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc159250 msgr2=0x7ffbcc072950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcaffd640 1 --2- 172.21.15.94:0/88649122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc159250 0x7ffbcc072950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcaffd640 1 -- 172.21.15.94:0/88649122 >> v1:172.21.15.94:6789/0 conn(0x7ffbcc158930 legacy=0x7ffbcc1526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbca7fc640 1 -- 172.21.15.94:0/88649122 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292109698 0 0) 0x7ffbcc0c2aa0 con 0x7ffbcc158930 2024-03-06T07:30:04.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.505+0000 7ffbcaffd640 1 -- 172.21.15.94:0/88649122 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc0c2730 con 0x7ffbcc152e00 2024-03-06T07:30:04.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.506+0000 7ffbcaffd640 1 --2- 172.21.15.94:0/88649122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc158d10 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40097a0 tx=0x7ffbb402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2c87b9fa551eeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.506+0000 7ffbca7fc640 1 -- 172.21.15.94:0/88649122 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40315a0 con 0x7ffbcc152e00 2024-03-06T07:30:04.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.506+0000 7ffbca7fc640 1 -- 172.21.15.94:0/88649122 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb4037070 con 0x7ffbcc152e00 2024-03-06T07:30:04.510 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.506+0000 7ffbca7fc640 1 -- 172.21.15.94:0/88649122 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4031960 con 0x7ffbcc152e00 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 -- 172.21.15.94:0/88649122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 msgr2=0x7ffbcc158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/88649122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc158d10 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40097a0 tx=0x7ffbb402b840 comp rx=0 tx=0).stop 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 -- 172.21.15.94:0/88649122 shutdown_connections 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/88649122 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc159250 0x7ffbcc072950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/88649122 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc158d10 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 -- 172.21.15.94:0/88649122 >> 172.21.15.94:0/88649122 conn(0x7ffbcc0aa6a0 msgr2=0x7ffbcc0aaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 -- 172.21.15.94:0/88649122 shutdown_connections 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.507+0000 7ffbd1d55640 1 -- 172.21.15.94:0/88649122 wait complete. 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.508+0000 7ffbd1d55640 1 Processor -- start 2024-03-06T07:30:04.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.508+0000 7ffbd1d55640 1 -- start start 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbd1d55640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc1ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbd1d55640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc158930 0x7ffbcc154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbd1d55640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbcc159250 0x7ffbcc06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbd1d55640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc159860 con 0x7ffbcc152e00 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbd1d55640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc1596e0 con 0x7ffbcc159250 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbd1d55640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc072e90 con 0x7ffbcc158930 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc1ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc1ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44724/0 (socket says 172.21.15.94:44724) 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 -- 172.21.15.94:0/3988918401 learned_addr learned my addr 172.21.15.94:0/3988918401 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:04.514 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcbfff640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbcc159250 0x7ffbcc06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcaffd640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc158930 0x7ffbcc154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 -- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc158930 msgr2=0x7ffbcc154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc158930 0x7ffbcc154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 -- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbcc159250 msgr2=0x7ffbcc06ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbcc159250 0x7ffbcc06ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.509+0000 7ffbcb7fe640 1 -- 172.21.15.94:0/3988918401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc155f00 con 0x7ffbcc152e00 2024-03-06T07:30:04.515 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.510+0000 7ffbcb7fe640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc1ee890 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00c4d0 tx=0x7ffbbc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.510+0000 7ffbc3fff640 1 -- 172.21.15.94:0/3988918401 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc018070 con 0x7ffbcc152e00 2024-03-06T07:30:04.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.510+0000 7ffbc3fff640 1 -- 172.21.15.94:0/3988918401 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc007500 con 0x7ffbcc152e00 2024-03-06T07:30:04.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.510+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc1eddb0 con 0x7ffbcc152e00 2024-03-06T07:30:04.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.510+0000 7ffbc3fff640 1 -- 172.21.15.94:0/3988918401 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc004a20 con 0x7ffbcc152e00 2024-03-06T07:30:04.516 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.510+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc1561e0 con 0x7ffbcc152e00 2024-03-06T07:30:04.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.511+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbcc04f330 con 0x7ffbcc152e00 2024-03-06T07:30:04.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.511+0000 7ffbc3fff640 1 -- 172.21.15.94:0/3988918401 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbbc01b070 con 0x7ffbcc152e00 2024-03-06T07:30:04.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.512+0000 7ffbc3fff640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbac06b4d0 0x7ffbac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.512+0000 7ffbc3fff640 1 -- 172.21.15.94:0/3988918401 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ffbbc089690 con 0x7ffbcc152e00 2024-03-06T07:30:04.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.512+0000 7ffbcaffd640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbac06b4d0 0x7ffbac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.512+0000 7ffbcaffd640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbac06b4d0 0x7ffbac06d990 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4036000 tx=0x7ffbb403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.516+0000 7ffbc3fff640 1 -- 172.21.15.94:0/3988918401 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbbc057cc0 con 0x7ffbcc152e00 2024-03-06T07:30:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.724+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbcc0c6c40 con 0x7ffbcc152e00 2024-03-06T07:30:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.724+0000 7ffbc3fff640 1 -- 172.21.15.94:0/3988918401 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7ffbbc05bd60 con 0x7ffbcc152e00 2024-03-06T07:30:04.726 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:04.726 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:04.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.726+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbac06b4d0 msgr2=0x7ffbac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.726+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbac06b4d0 0x7ffbac06d990 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4036000 tx=0x7ffbb403d040 comp rx=0 tx=0).stop 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.726+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 msgr2=0x7ffbcc1ee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.726+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc1ee890 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00c4d0 tx=0x7ffbbc00c990 comp rx=0 tx=0).stop 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 shutdown_connections 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ffbac06b4d0 0x7ffbac06d990 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffbcc159250 0x7ffbcc06ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffbcc158930 0x7ffbcc154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 --2- 172.21.15.94:0/3988918401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffbcc152e00 0x7ffbcc1ee890 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 >> 172.21.15.94:0/3988918401 conn(0x7ffbcc0aa6a0 msgr2=0x7ffbcc0c8050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 shutdown_connections 2024-03-06T07:30:04.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:04.727+0000 7ffbd1d55640 1 -- 172.21.15.94:0/3988918401 wait complete. 2024-03-06T07:30:05.143 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:05.143 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:06.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.253+0000 7f0c5a015640 1 Processor -- start 2024-03-06T07:30:06.255 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.253+0000 7f0c5a015640 1 -- start start 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c5a015640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0c5414f790 0x7f0c541529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c5a015640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c54155a40 0x7f0c54155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c5a015640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540c28d0 con 0x7f0c540c1aa0 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c5a015640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c540c2680 con 0x7f0c54155a40 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c5a015640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c540c2bb0 con 0x7f0c5414f790 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c53fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c54155a40 0x7f0c54155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c53fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c54155a40 0x7f0c54155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44732/0 (socket says 172.21.15.94:44732) 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c53fff640 1 -- 172.21.15.94:0/678112921 learned_addr learned my addr 172.21.15.94:0/678112921 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c537fe640 1 --2- 172.21.15.94:0/678112921 >> v2:172.21.15.145:3300/0 conn(0x7f0c5414f790 0x7f0c541529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c53fff640 1 -- 172.21.15.94:0/678112921 >> v2:172.21.15.145:3300/0 conn(0x7f0c5414f790 msgr2=0x7f0c541529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c53fff640 1 --2- 172.21.15.94:0/678112921 >> v2:172.21.15.145:3300/0 conn(0x7f0c5414f790 0x7f0c541529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c53fff640 1 -- 172.21.15.94:0/678112921 >> v1:172.21.15.145:6789/0 conn(0x7f0c540c1aa0 legacy=0x7f0c540c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.254+0000 7f0c53fff640 1 -- 172.21.15.94:0/678112921 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540c2e90 con 0x7f0c54155a40 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.255+0000 7f0c53fff640 1 --2- 172.21.15.94:0/678112921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54155e50 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f0c540c3170 tx=0x7f0c4c02e190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afb87cd2333d7a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.255+0000 7f0c527fc640 1 -- 172.21.15.94:0/678112921 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c02ed30 con 0x7f0c54155a40 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.255+0000 7f0c527fc640 1 -- 172.21.15.94:0/678112921 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c0363f0 con 0x7f0c54155a40 2024-03-06T07:30:06.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.255+0000 7f0c527fc640 1 -- 172.21.15.94:0/678112921 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0367c0 con 0x7f0c54155a40 2024-03-06T07:30:06.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 -- 172.21.15.94:0/678112921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 msgr2=0x7f0c54155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 --2- 172.21.15.94:0/678112921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54155e50 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f0c540c3170 tx=0x7f0c4c02e190 comp rx=0 tx=0).stop 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 -- 172.21.15.94:0/678112921 shutdown_connections 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 --2- 172.21.15.94:0/678112921 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54155e50 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 --2- 172.21.15.94:0/678112921 >> v2:172.21.15.145:3300/0 conn(0x7f0c5414f790 0x7f0c541529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 -- 172.21.15.94:0/678112921 >> 172.21.15.94:0/678112921 conn(0x7f0c540c6710 msgr2=0x7f0c540c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 -- 172.21.15.94:0/678112921 shutdown_connections 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.256+0000 7f0c5a015640 1 -- 172.21.15.94:0/678112921 wait complete. 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.257+0000 7f0c5a015640 1 Processor -- start 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.257+0000 7f0c5a015640 1 -- start start 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c5a015640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c540c1aa0 0x7f0c54153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c5a015640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c5414f790 0x7f0c5414bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c5a015640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.261 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c5a015640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540c0d60 con 0x7f0c54155a40 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c5a015640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54156390 con 0x7f0c5414f790 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c5a015640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540c5e30 con 0x7f0c540c1aa0 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54149810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44734/0 (socket says 172.21.15.94:44734) 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 -- 172.21.15.94:0/1338286435 learned_addr learned my addr 172.21.15.94:0/1338286435 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c52ffd640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c5414f790 0x7f0c5414bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c537fe640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c540c1aa0 0x7f0c54153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 -- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c540c1aa0 msgr2=0x7f0c54153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c540c1aa0 0x7f0c54153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 -- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c5414f790 msgr2=0x7f0c5414bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c5414f790 0x7f0c5414bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 -- 172.21.15.94:0/1338286435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c54153c30 con 0x7f0c54155a40 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.258+0000 7f0c53fff640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54149810 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c02e7a0 tx=0x7f0c4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.262 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.259+0000 7f0c3bfff640 1 -- 172.21.15.94:0/1338286435 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c045020 con 0x7f0c54155a40 2024-03-06T07:30:06.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.259+0000 7f0c3bfff640 1 -- 172.21.15.94:0/1338286435 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c040450 con 0x7f0c54155a40 2024-03-06T07:30:06.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.259+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c54153f10 con 0x7f0c54155a40 2024-03-06T07:30:06.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.259+0000 7f0c3bfff640 1 -- 172.21.15.94:0/1338286435 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c009510 con 0x7f0c54155a40 2024-03-06T07:30:06.263 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.259+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5414c010 con 0x7f0c54155a40 2024-03-06T07:30:06.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.260+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5404f330 con 0x7f0c54155a40 2024-03-06T07:30:06.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.260+0000 7f0c3bfff640 1 -- 172.21.15.94:0/1338286435 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c4c048020 con 0x7f0c54155a40 2024-03-06T07:30:06.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.260+0000 7f0c3bfff640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c4006b4d0 0x7f0c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.261+0000 7f0c3bfff640 1 -- 172.21.15.94:0/1338286435 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0c4c0aeab0 con 0x7f0c54155a40 2024-03-06T07:30:06.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.261+0000 7f0c537fe640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c4006b4d0 0x7f0c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.264 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.261+0000 7f0c537fe640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c4006b4d0 0x7f0c4006d990 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0c540abb90 tx=0x7f0c44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.266 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.265+0000 7f0c3bfff640 1 -- 172.21.15.94:0/1338286435 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c4c04f030 con 0x7f0c54155a40 2024-03-06T07:30:06.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.477+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c540a5f60 con 0x7f0c54155a40 2024-03-06T07:30:06.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.477+0000 7f0c3bfff640 1 -- 172.21.15.94:0/1338286435 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f0c4c07d0e0 con 0x7f0c54155a40 2024-03-06T07:30:06.479 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:06.479 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:06.480 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c4006b4d0 msgr2=0x7f0c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c4006b4d0 0x7f0c4006d990 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0c540abb90 tx=0x7f0c44009190 comp rx=0 tx=0).stop 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 msgr2=0x7f0c54149810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54149810 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c02e7a0 tx=0x7f0c4c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 shutdown_connections 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c4006b4d0 0x7f0c4006d990 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c54155a40 0x7f0c54149810 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c5414f790 0x7f0c5414bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 --2- 172.21.15.94:0/1338286435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c540c1aa0 0x7f0c54153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 >> 172.21.15.94:0/1338286435 conn(0x7f0c540c6710 msgr2=0x7f0c540c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 shutdown_connections 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:06.479+0000 7f0c5a015640 1 -- 172.21.15.94:0/1338286435 wait complete. 2024-03-06T07:30:06.894 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:06.894 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:07.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:08.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.006+0000 7f8a2a9bc640 1 Processor -- start 2024-03-06T07:30:08.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.006+0000 7f8a2a9bc640 1 -- start start 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.006+0000 7f8a2a9bc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8a24156150 0x7f8a241529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.006+0000 7f8a2a9bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a240c28d0 con 0x7f8a240c1aa0 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.006+0000 7f8a2a9bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a240c2bb0 con 0x7f8a2414f790 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.006+0000 7f8a2a9bc640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a240c2680 con 0x7f8a24156150 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.007+0000 7f8a28f59640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8a24156150 0x7f8a241529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.007+0000 7f8a237fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8a240c1aa0 0x7f8a24155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50220/0 (socket says 172.21.15.94:50220) 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.007+0000 7f8a237fe640 1 -- 172.21.15.94:0/4183522601 learned_addr learned my addr 172.21.15.94:0/4183522601 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.007+0000 7f8a28f59640 1 -- 172.21.15.94:0/4183522601 >> v1:172.21.15.145:6789/0 conn(0x7f8a2414f790 legacy=0x7f8a240c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.007+0000 7f8a28f59640 1 -- 172.21.15.94:0/4183522601 >> v1:172.21.15.94:6789/0 conn(0x7f8a240c1aa0 legacy=0x7f8a24155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.007+0000 7f8a28f59640 1 -- 172.21.15.94:0/4183522601 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240c2e90 con 0x7f8a24156150 2024-03-06T07:30:08.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.007+0000 7f8a22ffd640 1 -- 172.21.15.94:0/4183522601 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005728856 0 0) 0x7f8a240c2bb0 con 0x7f8a2414f790 2024-03-06T07:30:08.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a28f59640 1 --2- 172.21.15.94:0/4183522601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a24156150 0x7f8a241529d0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f8a240c2800 tx=0x7f8a1c02bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979dd6e218982a64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a22ffd640 1 -- 172.21.15.94:0/4183522601 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c039070 con 0x7f8a24156150 2024-03-06T07:30:08.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a22ffd640 1 -- 172.21.15.94:0/4183522601 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c02caa0 con 0x7f8a24156150 2024-03-06T07:30:08.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a22ffd640 1 -- 172.21.15.94:0/4183522601 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c034440 con 0x7f8a24156150 2024-03-06T07:30:08.011 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/4183522601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a24156150 msgr2=0x7f8a241529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/4183522601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a24156150 0x7f8a241529d0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f8a240c2800 tx=0x7f8a1c02bd70 comp rx=0 tx=0).stop 2024-03-06T07:30:08.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/4183522601 shutdown_connections 2024-03-06T07:30:08.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/4183522601 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a24156150 0x7f8a241529d0 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/4183522601 >> 172.21.15.94:0/4183522601 conn(0x7f8a240c6710 msgr2=0x7f8a240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/4183522601 shutdown_connections 2024-03-06T07:30:08.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.008+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/4183522601 wait complete. 2024-03-06T07:30:08.012 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 Processor -- start 2024-03-06T07:30:08.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 -- start start 2024-03-06T07:30:08.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a240c1aa0 0x7f8a2414b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.014 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a2414f790 0x7f8a241466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a24156150 0x7f8a24146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240c5e30 con 0x7f8a2414f790 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240c0d60 con 0x7f8a240c1aa0 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.010+0000 7f8a2a9bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240c5fb0 con 0x7f8a24156150 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a2414f790 0x7f8a241466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a2414f790 0x7f8a241466c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44754/0 (socket says 172.21.15.94:44754) 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 -- 172.21.15.94:0/3327678781 learned_addr learned my addr 172.21.15.94:0/3327678781 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a23fff640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a240c1aa0 0x7f8a2414b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a28f59640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a24156150 0x7f8a24146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 -- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a24156150 msgr2=0x7f8a24146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a24156150 0x7f8a24146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 -- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a240c1aa0 msgr2=0x7f8a2414b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a240c1aa0 0x7f8a2414b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.015 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 -- 172.21.15.94:0/3327678781 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24151d30 con 0x7f8a2414f790 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a237fe640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a2414f790 0x7f8a241466c0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800c780 tx=0x7f8a1800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a20ff9640 1 -- 172.21.15.94:0/3327678781 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a18018070 con 0x7f8a2414f790 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a20ff9640 1 -- 172.21.15.94:0/3327678781 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a180072d0 con 0x7f8a2414f790 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a20ff9640 1 -- 172.21.15.94:0/3327678781 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a180047e0 con 0x7f8a2414f790 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2414a640 con 0x7f8a2414f790 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.011+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2414dbe0 con 0x7f8a2414f790 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.012+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2404f330 con 0x7f8a2414f790 2024-03-06T07:30:08.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.013+0000 7f8a20ff9640 1 -- 172.21.15.94:0/3327678781 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a1801b070 con 0x7f8a2414f790 2024-03-06T07:30:08.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.013+0000 7f8a20ff9640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a1006b4d0 0x7f8a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.013+0000 7f8a20ff9640 1 -- 172.21.15.94:0/3327678781 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8a180896f0 con 0x7f8a2414f790 2024-03-06T07:30:08.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.013+0000 7f8a23fff640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a1006b4d0 0x7f8a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.017 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.014+0000 7f8a23fff640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a1006b4d0 0x7f8a1006d990 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f8a14001120 tx=0x7f8a14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.017+0000 7f8a20ff9640 1 -- 172.21.15.94:0/3327678781 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a18001230 con 0x7f8a2414f790 2024-03-06T07:30:08.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.223+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a240c1f40 con 0x7f8a2414f790 2024-03-06T07:30:08.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.223+0000 7f8a20ff9640 1 -- 172.21.15.94:0/3327678781 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f8a18057d20 con 0x7f8a2414f790 2024-03-06T07:30:08.225 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:08.226 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:08.226 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:08.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a1006b4d0 msgr2=0x7f8a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a1006b4d0 0x7f8a1006d990 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f8a14001120 tx=0x7f8a14009040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a2414f790 msgr2=0x7f8a241466c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a2414f790 0x7f8a241466c0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800c780 tx=0x7f8a1800cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 shutdown_connections 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a1006b4d0 0x7f8a1006d990 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a24156150 0x7f8a24146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a2414f790 0x7f8a241466c0 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 --2- 172.21.15.94:0/3327678781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a240c1aa0 0x7f8a2414b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.225+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 >> 172.21.15.94:0/3327678781 conn(0x7f8a240c6710 msgr2=0x7f8a240a9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.226+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 shutdown_connections 2024-03-06T07:30:08.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:08.226+0000 7f8a2a9bc640 1 -- 172.21.15.94:0/3327678781 wait complete. 2024-03-06T07:30:08.640 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:08.640 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:09.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:09.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.751+0000 7f6d5601c640 1 Processor -- start 2024-03-06T07:30:09.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.751+0000 7f6d5601c640 1 -- start start 2024-03-06T07:30:09.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d5601c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6d501550c0 0x7f6d501554a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d5601c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d500aa180 0x7f6d500c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d5601c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500aa660 con 0x7f6d501559e0 2024-03-06T07:30:09.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d5601c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d500c4610 con 0x7f6d501550c0 2024-03-06T07:30:09.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d5601c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500aa940 con 0x7f6d500aa180 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4f7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6d501550c0 0x7f6d501554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4effd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6d501559e0 0x7f6d5014ec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46108/0 (socket says 172.21.15.94:46108) 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4effd640 1 -- 172.21.15.94:0/447640235 learned_addr learned my addr 172.21.15.94:0/447640235 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4ffff640 1 --2- 172.21.15.94:0/447640235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d500aa180 0x7f6d500c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4f7fe640 1 -- 172.21.15.94:0/447640235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d500aa180 msgr2=0x7f6d500c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4f7fe640 1 --2- 172.21.15.94:0/447640235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d500aa180 0x7f6d500c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4f7fe640 1 -- 172.21.15.94:0/447640235 >> v1:172.21.15.94:6789/0 conn(0x7f6d501559e0 legacy=0x7f6d5014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4f7fe640 1 -- 172.21.15.94:0/447640235 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500c4860 con 0x7f6d501550c0 2024-03-06T07:30:09.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.752+0000 7f6d4e7fc640 1 -- 172.21.15.94:0/447640235 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31806844 0 0) 0x7f6d500aa660 con 0x7f6d501559e0 2024-03-06T07:30:09.756 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.753+0000 7f6d4f7fe640 1 --2- 172.21.15.94:0/447640235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501550c0 0x7f6d501554a0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f6d400097a0 tx=0x7f6d4002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4e729947540a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.753+0000 7f6d4e7fc640 1 -- 172.21.15.94:0/447640235 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40038070 con 0x7f6d501550c0 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.753+0000 7f6d4e7fc640 1 -- 172.21.15.94:0/447640235 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d40031700 con 0x7f6d501550c0 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.753+0000 7f6d4e7fc640 1 -- 172.21.15.94:0/447640235 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40031ad0 con 0x7f6d501550c0 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 -- 172.21.15.94:0/447640235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501550c0 msgr2=0x7f6d501554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 --2- 172.21.15.94:0/447640235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501550c0 0x7f6d501554a0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f6d400097a0 tx=0x7f6d4002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 -- 172.21.15.94:0/447640235 shutdown_connections 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 --2- 172.21.15.94:0/447640235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d500aa180 0x7f6d500c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 --2- 172.21.15.94:0/447640235 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501550c0 0x7f6d501554a0 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 -- 172.21.15.94:0/447640235 >> 172.21.15.94:0/447640235 conn(0x7f6d500c6710 msgr2=0x7f6d500c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 -- 172.21.15.94:0/447640235 shutdown_connections 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.754+0000 7f6d5601c640 1 -- 172.21.15.94:0/447640235 wait complete. 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.755+0000 7f6d5601c640 1 Processor -- start 2024-03-06T07:30:09.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.755+0000 7f6d5601c640 1 -- start start 2024-03-06T07:30:09.759 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d5601c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d500aa180 0x7f6d501dee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d5601c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d501550c0 0x7f6d501df360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d5601c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501559e0 0x7f6d501480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d5601c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5014f310 con 0x7f6d501559e0 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d5601c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500c1560 con 0x7f6d500aa180 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d5601c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5014f490 con 0x7f6d501550c0 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501559e0 0x7f6d501480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501559e0 0x7f6d501480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45590/0 (socket says 172.21.15.94:45590) 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 -- 172.21.15.94:0/3089331688 learned_addr learned my addr 172.21.15.94:0/3089331688 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4f7fe640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d500aa180 0x7f6d501dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4effd640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d501550c0 0x7f6d501df360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.760 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 -- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d501550c0 msgr2=0x7f6d501df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d501550c0 0x7f6d501df360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 -- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d500aa180 msgr2=0x7f6d501dee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d500aa180 0x7f6d501dee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 -- 172.21.15.94:0/3089331688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5006e270 con 0x7f6d501559e0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d4ffff640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501559e0 0x7f6d501480f0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800ef90 tx=0x7f6d4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.756+0000 7f6d37fff640 1 -- 172.21.15.94:0/3089331688 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48018020 con 0x7f6d501559e0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.757+0000 7f6d37fff640 1 -- 172.21.15.94:0/3089331688 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48014480 con 0x7f6d501559e0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.757+0000 7f6d37fff640 1 -- 172.21.15.94:0/3089331688 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48009e60 con 0x7f6d501559e0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.757+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5006e550 con 0x7f6d501559e0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.757+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d50147bd0 con 0x7f6d501559e0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.758+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d50147dd0 con 0x7f6d501559e0 2024-03-06T07:30:09.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.758+0000 7f6d37fff640 1 -- 172.21.15.94:0/3089331688 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d48005d10 con 0x7f6d501559e0 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.758+0000 7f6d37fff640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.758+0000 7f6d37fff640 1 -- 172.21.15.94:0/3089331688 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6d4808af40 con 0x7f6d501559e0 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.758+0000 7f6d4f7fe640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.762 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.759+0000 7f6d4f7fe640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f6d40035000 tx=0x7f6d40008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.764 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.763+0000 7f6d37fff640 1 -- 172.21.15.94:0/3089331688 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d50147dd0 con 0x7f6d501559e0 2024-03-06T07:30:09.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.969+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d50068450 con 0x7f6d501559e0 2024-03-06T07:30:09.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.970+0000 7f6d37fff640 1 -- 172.21.15.94:0/3089331688 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f6d48022030 con 0x7f6d501559e0 2024-03-06T07:30:09.972 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:09.972 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:09.972 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d3c06b500 msgr2=0x7f6d3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f6d40035000 tx=0x7f6d40008030 comp rx=0 tx=0).stop 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501559e0 msgr2=0x7f6d501480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501559e0 0x7f6d501480f0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800ef90 tx=0x7f6d4800c490 comp rx=0 tx=0).stop 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 shutdown_connections 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d3c06b500 0x7f6d3c06d9c0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d501559e0 0x7f6d501480f0 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d501550c0 0x7f6d501df360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 --2- 172.21.15.94:0/3089331688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d500aa180 0x7f6d501dee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.974 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 >> 172.21.15.94:0/3089331688 conn(0x7f6d500c6710 msgr2=0x7f6d5014d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 shutdown_connections 2024-03-06T07:30:09.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:09.972+0000 7f6d5601c640 1 -- 172.21.15.94:0/3089331688 wait complete. 2024-03-06T07:30:10.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:10.387 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:11.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:11.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.498+0000 7f6dc27c7640 1 Processor -- start 2024-03-06T07:30:11.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.498+0000 7f6dc27c7640 1 -- start start 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dc27c7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6dbc072f70 0x7f6dbc154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dc27c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc157490 con 0x7f6dbc0bbf80 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dc27c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dbc0bdc30 con 0x7f6dbc0c6560 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dc27c7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dbc0bd690 con 0x7f6dbc072f70 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dbbfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6dbc072f70 0x7f6dbc154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dc0d64640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6dbc0bbf80 0x7f6dbc156d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46116/0 (socket says 172.21.15.94:46116) 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dc0d64640 1 -- 172.21.15.94:0/569901870 learned_addr learned my addr 172.21.15.94:0/569901870 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:11.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dbbfff640 1 -- 172.21.15.94:0/569901870 >> v1:172.21.15.145:6789/0 conn(0x7f6dbc0c6560 legacy=0x7f6dbc156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dbbfff640 1 -- 172.21.15.94:0/569901870 >> v1:172.21.15.94:6789/0 conn(0x7f6dbc0bbf80 legacy=0x7f6dbc156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dbbfff640 1 -- 172.21.15.94:0/569901870 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc0bd950 con 0x7f6dbc072f70 2024-03-06T07:30:11.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.499+0000 7f6dbaffd640 1 -- 172.21.15.94:0/569901870 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045989873 0 0) 0x7f6dbc157490 con 0x7f6dbc0bbf80 2024-03-06T07:30:11.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.500+0000 7f6dbbfff640 1 --2- 172.21.15.94:0/569901870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc072f70 0x7f6dbc154570 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0097a0 tx=0x7f6dac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192257ccd6dba3b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.500+0000 7f6dbaffd640 1 -- 172.21.15.94:0/569901870 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac035020 con 0x7f6dbc072f70 2024-03-06T07:30:11.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.500+0000 7f6dbaffd640 1 -- 172.21.15.94:0/569901870 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dac031830 con 0x7f6dbc072f70 2024-03-06T07:30:11.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.500+0000 7f6dbaffd640 1 -- 172.21.15.94:0/569901870 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dac031c00 con 0x7f6dbc072f70 2024-03-06T07:30:11.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.501+0000 7f6dc27c7640 1 -- 172.21.15.94:0/569901870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc072f70 msgr2=0x7f6dbc154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.501+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/569901870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc072f70 0x7f6dbc154570 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0097a0 tx=0x7f6dac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.503 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.501+0000 7f6dc27c7640 1 -- 172.21.15.94:0/569901870 shutdown_connections 2024-03-06T07:30:11.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.501+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/569901870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc072f70 0x7f6dbc154570 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.501+0000 7f6dc27c7640 1 -- 172.21.15.94:0/569901870 >> 172.21.15.94:0/569901870 conn(0x7f6dbc06b1a0 msgr2=0x7f6dbc06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.501+0000 7f6dc27c7640 1 -- 172.21.15.94:0/569901870 shutdown_connections 2024-03-06T07:30:11.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.501+0000 7f6dc27c7640 1 -- 172.21.15.94:0/569901870 wait complete. 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.502+0000 7f6dc27c7640 1 Processor -- start 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.502+0000 7f6dc27c7640 1 -- start start 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc27c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dbc072f70 0x7f6dbc0a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc27c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dbc0bbf80 0x7f6dbc155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc27c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc0c6560 0x7f6dbc1eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc27c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0bba60 con 0x7f6dbc0c6560 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc27c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc0c5820 con 0x7f6dbc0bbf80 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc27c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc073350 con 0x7f6dbc072f70 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc0c6560 0x7f6dbc1eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc0c6560 0x7f6dbc1eb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45602/0 (socket says 172.21.15.94:45602) 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 -- 172.21.15.94:0/1218203067 learned_addr learned my addr 172.21.15.94:0/1218203067 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dbb7fe640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dbc0bbf80 0x7f6dbc155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dbbfff640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dbc072f70 0x7f6dbc0a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 -- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dbc072f70 msgr2=0x7f6dbc0a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dbc072f70 0x7f6dbc0a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 -- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dbc0bbf80 msgr2=0x7f6dbc155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dbc0bbf80 0x7f6dbc155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 -- 172.21.15.94:0/1218203067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc0bef10 con 0x7f6dbc0c6560 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6dc0d64640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc0c6560 0x7f6dbc1eb4f0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f6db400c7b0 tx=0x7f6db400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.503+0000 7f6db8ff9640 1 -- 172.21.15.94:0/1218203067 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db4018070 con 0x7f6dbc0c6560 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.504+0000 7f6db8ff9640 1 -- 172.21.15.94:0/1218203067 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db400a640 con 0x7f6dbc0c6560 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.504+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc1eec90 con 0x7f6dbc0c6560 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.504+0000 7f6db8ff9640 1 -- 172.21.15.94:0/1218203067 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db400aa70 con 0x7f6dbc0c6560 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.504+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc1eafd0 con 0x7f6dbc0c6560 2024-03-06T07:30:11.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.505+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dbc1eb1d0 con 0x7f6dbc0c6560 2024-03-06T07:30:11.508 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.505+0000 7f6db8ff9640 1 -- 172.21.15.94:0/1218203067 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6db401b070 con 0x7f6dbc0c6560 2024-03-06T07:30:11.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.505+0000 7f6db8ff9640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.506+0000 7f6db8ff9640 1 -- 172.21.15.94:0/1218203067 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6db4089980 con 0x7f6dbc0c6560 2024-03-06T07:30:11.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.506+0000 7f6dbbfff640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.509 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.506+0000 7f6dbbfff640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f6dac036000 tx=0x7f6dac03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.510+0000 7f6db8ff9640 1 -- 172.21.15.94:0/1218203067 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6dbc1eb1d0 con 0x7f6dbc0c6560 2024-03-06T07:30:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.718+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dbc04f330 con 0x7f6dbc0c6560 2024-03-06T07:30:11.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.719+0000 7f6db8ff9640 1 -- 172.21.15.94:0/1218203067 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f6db4057fb0 con 0x7f6dbc0c6560 2024-03-06T07:30:11.721 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:11.721 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:11.721 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d9c06b4d0 msgr2=0x7f6d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f6dac036000 tx=0x7f6dac03d040 comp rx=0 tx=0).stop 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc0c6560 msgr2=0x7f6dbc1eb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc0c6560 0x7f6dbc1eb4f0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f6db400c7b0 tx=0x7f6db400cc70 comp rx=0 tx=0).stop 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 shutdown_connections 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6d9c06b4d0 0x7f6d9c06d990 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dbc0c6560 0x7f6dbc1eb4f0 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6dbc0bbf80 0x7f6dbc155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 --2- 172.21.15.94:0/1218203067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6dbc072f70 0x7f6dbc0a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 >> 172.21.15.94:0/1218203067 conn(0x7f6dbc06b1a0 msgr2=0x7f6dbc0c48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 shutdown_connections 2024-03-06T07:30:11.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:11.721+0000 7f6dc27c7640 1 -- 172.21.15.94:0/1218203067 wait complete. 2024-03-06T07:30:12.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:12.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:13.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.247+0000 7ff507a50640 1 Processor -- start 2024-03-06T07:30:13.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.247+0000 7ff507a50640 1 -- start start 2024-03-06T07:30:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff507a50640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5000c1900 0x7ff500159120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff507a50640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5000c1d90 con 0x7ff50006e490 2024-03-06T07:30:13.250 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff507a50640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5001563e0 con 0x7ff500156000 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff507a50640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5000c4dd0 con 0x7ff5000c1900 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff505fed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5000c1900 0x7ff500159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff504feb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff50006e490 0x7ff500158a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46128/0 (socket says 172.21.15.94:46128) 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff504feb640 1 -- 172.21.15.94:0/1898240383 learned_addr learned my addr 172.21.15.94:0/1898240383 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff4f7fff640 1 -- 172.21.15.94:0/1898240383 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389728141 0 0) 0x7ff5000c1d90 con 0x7ff50006e490 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff4f7fff640 1 -- 172.21.15.94:0/1898240383 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec003520 con 0x7ff50006e490 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff4f7fff640 1 -- 172.21.15.94:0/1898240383 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549641207 0 0) 0x7ff5001563e0 con 0x7ff500156000 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff505fed640 1 -- 172.21.15.94:0/1898240383 >> v1:172.21.15.145:6789/0 conn(0x7ff500156000 legacy=0x7ff5000c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff505fed640 1 -- 172.21.15.94:0/1898240383 >> v1:172.21.15.94:6789/0 conn(0x7ff50006e490 legacy=0x7ff500158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.248+0000 7ff505fed640 1 -- 172.21.15.94:0/1898240383 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff500159660 con 0x7ff5000c1900 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.249+0000 7ff505fed640 1 --2- 172.21.15.94:0/1898240383 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5000c1900 0x7ff500159120 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009d90 tx=0x7ff4f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e279366b80a6a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.249+0000 7ff4f7fff640 1 -- 172.21.15.94:0/1898240383 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8039070 con 0x7ff5000c1900 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.249+0000 7ff4f7fff640 1 -- 172.21.15.94:0/1898240383 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f802cc60 con 0x7ff5000c1900 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.249+0000 7ff4f7fff640 1 -- 172.21.15.94:0/1898240383 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8034640 con 0x7ff5000c1900 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.250+0000 7ff507a50640 1 -- 172.21.15.94:0/1898240383 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5000c1900 msgr2=0x7ff500159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.250+0000 7ff507a50640 1 --2- 172.21.15.94:0/1898240383 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5000c1900 0x7ff500159120 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009d90 tx=0x7ff4f802beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.250+0000 7ff507a50640 1 -- 172.21.15.94:0/1898240383 shutdown_connections 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.250+0000 7ff507a50640 1 --2- 172.21.15.94:0/1898240383 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5000c1900 0x7ff500159120 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.253 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.250+0000 7ff507a50640 1 -- 172.21.15.94:0/1898240383 >> 172.21.15.94:0/1898240383 conn(0x7ff5000c6710 msgr2=0x7ff5000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.250+0000 7ff507a50640 1 -- 172.21.15.94:0/1898240383 shutdown_connections 2024-03-06T07:30:13.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.250+0000 7ff507a50640 1 -- 172.21.15.94:0/1898240383 wait complete. 2024-03-06T07:30:13.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.251+0000 7ff507a50640 1 Processor -- start 2024-03-06T07:30:13.254 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.251+0000 7ff507a50640 1 -- start start 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff507a50640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff50006e490 0x7ff5001573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff507a50640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5000c1900 0x7ff5000a9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff507a50640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff500156000 0x7ff5000a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff507a50640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000c22e0 con 0x7ff50006e490 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff507a50640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000c6260 con 0x7ff500156000 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff507a50640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000c5c10 con 0x7ff5000c1900 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff50006e490 0x7ff5001573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.256 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff50006e490 0x7ff5001573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45622/0 (socket says 172.21.15.94:45622) 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 -- 172.21.15.94:0/3482925920 learned_addr learned my addr 172.21.15.94:0/3482925920 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff505fed640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff500156000 0x7ff5000a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff504feb640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5000c1900 0x7ff5000a9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 -- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5000c1900 msgr2=0x7ff5000a9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5000c1900 0x7ff5000a9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 -- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff500156000 msgr2=0x7ff5000a6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff500156000 0x7ff5000a6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 -- 172.21.15.94:0/3482925920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff500072d30 con 0x7ff50006e490 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff5057ec640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff50006e490 0x7ff5001573e0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00c410 tx=0x7ff4fc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.252+0000 7ff4f5ffb640 1 -- 172.21.15.94:0/3482925920 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc017070 con 0x7ff50006e490 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.253+0000 7ff4f5ffb640 1 -- 172.21.15.94:0/3482925920 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0035f0 con 0x7ff50006e490 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.253+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff500073010 con 0x7ff50006e490 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.253+0000 7ff4f5ffb640 1 -- 172.21.15.94:0/3482925920 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0039e0 con 0x7ff50006e490 2024-03-06T07:30:13.257 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.253+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5001591b0 con 0x7ff50006e490 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.254+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50004f330 con 0x7ff50006e490 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.254+0000 7ff4f5ffb640 1 -- 172.21.15.94:0/3482925920 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4fc01a070 con 0x7ff50006e490 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.254+0000 7ff4f5ffb640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4e406b4d0 0x7ff4e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.254+0000 7ff4f5ffb640 1 -- 172.21.15.94:0/3482925920 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff4fc0897a0 con 0x7ff50006e490 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.255+0000 7ff504feb640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4e406b4d0 0x7ff4e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.258 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.255+0000 7ff504feb640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4e406b4d0 0x7ff4e406d990 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ff500154c90 tx=0x7ff4f0007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.260 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.259+0000 7ff4f5ffb640 1 -- 172.21.15.94:0/3482925920 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4fc057dd0 con 0x7ff50006e490 2024-03-06T07:30:13.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.467+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5001563e0 con 0x7ff50006e490 2024-03-06T07:30:13.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.467+0000 7ff4f5ffb640 1 -- 172.21.15.94:0/3482925920 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7ff4fc05be70 con 0x7ff50006e490 2024-03-06T07:30:13.469 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:13.470 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:13.470 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:13.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4e406b4d0 msgr2=0x7ff4e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4e406b4d0 0x7ff4e406d990 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ff500154c90 tx=0x7ff4f0007d30 comp rx=0 tx=0).stop 2024-03-06T07:30:13.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff50006e490 msgr2=0x7ff5001573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff50006e490 0x7ff5001573e0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00c410 tx=0x7ff4fc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 shutdown_connections 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff4e406b4d0 0x7ff4e406d990 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff500156000 0x7ff5000a6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5000c1900 0x7ff5000a9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 --2- 172.21.15.94:0/3482925920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff50006e490 0x7ff5001573e0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 >> 172.21.15.94:0/3482925920 conn(0x7ff5000c6710 msgr2=0x7ff50006d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 shutdown_connections 2024-03-06T07:30:13.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:13.469+0000 7ff507a50640 1 -- 172.21.15.94:0/3482925920 wait complete. 2024-03-06T07:30:13.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:13.884 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:14.886 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:14.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.994+0000 7f93fce4e640 1 Processor -- start 2024-03-06T07:30:14.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.995+0000 7f93fce4e640 1 -- start start 2024-03-06T07:30:14.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.995+0000 7f93fce4e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f93f8073250 0x7f93f80c3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.995+0000 7f93fce4e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80bdc20 con 0x7f93f8072e70 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.995+0000 7f93fce4e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f806c8e0 con 0x7f93f80c3950 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.995+0000 7f93fce4e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f806ce80 con 0x7f93f8073250 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.996+0000 7f93f5d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f93f8073250 0x7f93f80c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.996+0000 7f93f659c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f93f8072e70 0x7f93f80bd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46134/0 (socket says 172.21.15.94:46134) 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.996+0000 7f93f659c640 1 -- 172.21.15.94:0/4201957983 learned_addr learned my addr 172.21.15.94:0/4201957983 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.996+0000 7f93f5d9b640 1 -- 172.21.15.94:0/4201957983 >> v1:172.21.15.145:6789/0 conn(0x7f93f80c3950 legacy=0x7f93f815fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.996+0000 7f93f5d9b640 1 -- 172.21.15.94:0/4201957983 >> v1:172.21.15.94:6789/0 conn(0x7f93f8072e70 legacy=0x7f93f80bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.996+0000 7f93f5d9b640 1 -- 172.21.15.94:0/4201957983 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80c1b00 con 0x7f93f8073250 2024-03-06T07:30:14.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93f5d9b640 1 --2- 172.21.15.94:0/4201957983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f80c3410 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f93ec009d90 tx=0x7f93ec02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5ad79e2c98e18b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93f559a640 1 -- 172.21.15.94:0/4201957983 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec039070 con 0x7f93f8073250 2024-03-06T07:30:14.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93f559a640 1 -- 172.21.15.94:0/4201957983 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec02cc60 con 0x7f93f8073250 2024-03-06T07:30:14.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93f559a640 1 -- 172.21.15.94:0/4201957983 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec034660 con 0x7f93f8073250 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93fce4e640 1 -- 172.21.15.94:0/4201957983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 msgr2=0x7f93f80c3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93fce4e640 1 --2- 172.21.15.94:0/4201957983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f80c3410 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f93ec009d90 tx=0x7f93ec02beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93fce4e640 1 -- 172.21.15.94:0/4201957983 shutdown_connections 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93fce4e640 1 --2- 172.21.15.94:0/4201957983 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f80c3410 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.997+0000 7f93fce4e640 1 -- 172.21.15.94:0/4201957983 >> 172.21.15.94:0/4201957983 conn(0x7f93f806b1a0 msgr2=0x7f93f806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.998+0000 7f93fce4e640 1 -- 172.21.15.94:0/4201957983 shutdown_connections 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.998+0000 7f93fce4e640 1 -- 172.21.15.94:0/4201957983 wait complete. 2024-03-06T07:30:15.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 Processor -- start 2024-03-06T07:30:15.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 -- start start 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93f8072e70 0x7f93f8154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f81eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93f80c3950 0x7f93f80bf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80c6760 con 0x7f93f8073250 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80c4a00 con 0x7f93f80c3950 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:14.999+0000 7f93fce4e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f8071c30 con 0x7f93f8072e70 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f81eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f81eb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45654/0 (socket says 172.21.15.94:45654) 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 -- 172.21.15.94:0/655991180 learned_addr learned my addr 172.21.15.94:0/655991180 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:15.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f6d9d640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93f80c3950 0x7f93f80bf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f659c640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93f8072e70 0x7f93f8154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 -- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93f8072e70 msgr2=0x7f93f8154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93f8072e70 0x7f93f8154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 -- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93f80c3950 msgr2=0x7f93f80bf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93f80c3950 0x7f93f80bf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 -- 172.21.15.94:0/655991180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f8071e80 con 0x7f93f8073250 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93f5d9b640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f81eb3d0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f93ec037040 tx=0x7f93ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93e6ffd640 1 -- 172.21.15.94:0/655991180 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec039050 con 0x7f93f8073250 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93e6ffd640 1 -- 172.21.15.94:0/655991180 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec045070 con 0x7f93f8073250 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f8072160 con 0x7f93f8073250 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93e6ffd640 1 -- 172.21.15.94:0/655991180 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec03ea50 con 0x7f93f8073250 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.000+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f81edf20 con 0x7f93f8073250 2024-03-06T07:30:15.004 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.001+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f804f330 con 0x7f93f8073250 2024-03-06T07:30:15.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.002+0000 7f93e6ffd640 1 -- 172.21.15.94:0/655991180 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93ec02cae0 con 0x7f93f8073250 2024-03-06T07:30:15.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.002+0000 7f93e6ffd640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f93d806b4d0 0x7f93d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.002+0000 7f93e6ffd640 1 -- 172.21.15.94:0/655991180 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f93ec0ad2b0 con 0x7f93f8073250 2024-03-06T07:30:15.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.002+0000 7f93f659c640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f93d806b4d0 0x7f93d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.003+0000 7f93f659c640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f93d806b4d0 0x7f93d806d990 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f93f0005880 tx=0x7f93f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.006+0000 7f93e6ffd640 1 -- 172.21.15.94:0/655991180 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93ec07b980 con 0x7f93f8073250 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.212+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93f80c6340 con 0x7f93f8073250 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.212+0000 7f93e6ffd640 1 -- 172.21.15.94:0/655991180 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f93ec07fa20 con 0x7f93f8073250 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:15.215 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:15.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f93d806b4d0 msgr2=0x7f93d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f93d806b4d0 0x7f93d806d990 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f93f0005880 tx=0x7f93f0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 msgr2=0x7f93f81eb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f81eb3d0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f93ec037040 tx=0x7f93ec004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 shutdown_connections 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f93d806b4d0 0x7f93d806d990 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93f80c3950 0x7f93f80bf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f8073250 0x7f93f81eb3d0 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 --2- 172.21.15.94:0/655991180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93f8072e70 0x7f93f8154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.214+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 >> 172.21.15.94:0/655991180 conn(0x7f93f806b1a0 msgr2=0x7f93f80c7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.215+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 shutdown_connections 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:15.215+0000 7f93fce4e640 1 -- 172.21.15.94:0/655991180 wait complete. 2024-03-06T07:30:15.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:15.630 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:16.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:16.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:16.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.743+0000 7fc598be9640 1 Processor -- start 2024-03-06T07:30:16.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.743+0000 7fc598be9640 1 -- start start 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc598be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5940c6780 0x7fc594158d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc598be9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc5941592d0 0x7fc5940c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc598be9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc594072630 con 0x7fc59406da40 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc598be9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59406de20 con 0x7fc5941592d0 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc598be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc594072910 con 0x7fc5940c6780 2024-03-06T07:30:16.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc592d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc5941592d0 0x7fc5940c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc59259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc59406da40 0x7fc594155b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46140/0 (socket says 172.21.15.94:46140) 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc59259c640 1 -- 172.21.15.94:0/1997435800 learned_addr learned my addr 172.21.15.94:0/1997435800 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc591d9b640 1 --2- 172.21.15.94:0/1997435800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5940c6780 0x7fc594158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc592d9d640 1 -- 172.21.15.94:0/1997435800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5940c6780 msgr2=0x7fc594158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc592d9d640 1 --2- 172.21.15.94:0/1997435800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5940c6780 0x7fc594158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc592d9d640 1 -- 172.21.15.94:0/1997435800 >> v1:172.21.15.94:6789/0 conn(0x7fc59406da40 legacy=0x7fc594155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc59159a640 1 -- 172.21.15.94:0/1997435800 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091872099 0 0) 0x7fc594072630 con 0x7fc59406da40 2024-03-06T07:30:16.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.744+0000 7fc592d9d640 1 -- 172.21.15.94:0/1997435800 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc594072bf0 con 0x7fc5941592d0 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.745+0000 7fc592d9d640 1 --2- 172.21.15.94:0/1997435800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc5940c3280 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009d90 tx=0x7fc58c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3399bd6f966188f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.745+0000 7fc59159a640 1 -- 172.21.15.94:0/1997435800 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c039070 con 0x7fc5941592d0 2024-03-06T07:30:16.748 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.745+0000 7fc59159a640 1 -- 172.21.15.94:0/1997435800 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c02cbc0 con 0x7fc5941592d0 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.745+0000 7fc59159a640 1 -- 172.21.15.94:0/1997435800 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c034440 con 0x7fc5941592d0 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 -- 172.21.15.94:0/1997435800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 msgr2=0x7fc5940c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 --2- 172.21.15.94:0/1997435800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc5940c3280 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009d90 tx=0x7fc58c02be10 comp rx=0 tx=0).stop 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 -- 172.21.15.94:0/1997435800 shutdown_connections 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 --2- 172.21.15.94:0/1997435800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc5940c3280 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 --2- 172.21.15.94:0/1997435800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5940c6780 0x7fc594158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 -- 172.21.15.94:0/1997435800 >> 172.21.15.94:0/1997435800 conn(0x7fc5940aa630 msgr2=0x7fc5940aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 -- 172.21.15.94:0/1997435800 shutdown_connections 2024-03-06T07:30:16.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.746+0000 7fc598be9640 1 -- 172.21.15.94:0/1997435800 wait complete. 2024-03-06T07:30:16.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.747+0000 7fc598be9640 1 Processor -- start 2024-03-06T07:30:16.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.747+0000 7fc598be9640 1 -- start start 2024-03-06T07:30:16.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc598be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc59406da40 0x7fc5940ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc598be9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5940c6780 0x7fc59414b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc598be9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc59414b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc598be9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940c3940 con 0x7fc5941592d0 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc598be9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940c37c0 con 0x7fc5940c6780 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc598be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940c3ac0 con 0x7fc59406da40 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc59414b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc59414b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45682/0 (socket says 172.21.15.94:45682) 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 -- 172.21.15.94:0/1432029562 learned_addr learned my addr 172.21.15.94:0/1432029562 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc59259c640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc59406da40 0x7fc5940ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc591d9b640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5940c6780 0x7fc59414b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 -- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc59406da40 msgr2=0x7fc5940ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc59406da40 0x7fc5940ae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 -- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5940c6780 msgr2=0x7fc59414b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5940c6780 0x7fc59414b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.748+0000 7fc592d9d640 1 -- 172.21.15.94:0/1432029562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59406f810 con 0x7fc5941592d0 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.749+0000 7fc592d9d640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc59414b5e0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fc58c037040 tx=0x7fc58c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.749+0000 7fc582ffd640 1 -- 172.21.15.94:0/1432029562 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c039050 con 0x7fc5941592d0 2024-03-06T07:30:16.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.749+0000 7fc582ffd640 1 -- 172.21.15.94:0/1432029562 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c045070 con 0x7fc5941592d0 2024-03-06T07:30:16.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.749+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59406faf0 con 0x7fc5941592d0 2024-03-06T07:30:16.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.749+0000 7fc582ffd640 1 -- 172.21.15.94:0/1432029562 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c009450 con 0x7fc5941592d0 2024-03-06T07:30:16.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.749+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5940ae7f0 con 0x7fc5941592d0 2024-03-06T07:30:16.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.750+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59404f330 con 0x7fc5941592d0 2024-03-06T07:30:16.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.750+0000 7fc582ffd640 1 -- 172.21.15.94:0/1432029562 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc58c03e850 con 0x7fc5941592d0 2024-03-06T07:30:16.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.751+0000 7fc582ffd640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc57406b4d0 0x7fc57406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.751+0000 7fc582ffd640 1 -- 172.21.15.94:0/1432029562 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc58c0ad2b0 con 0x7fc5941592d0 2024-03-06T07:30:16.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.751+0000 7fc59259c640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc57406b4d0 0x7fc57406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.751+0000 7fc59259c640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc57406b4d0 0x7fc57406d990 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc584007fd0 tx=0x7fc584007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.757 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.755+0000 7fc582ffd640 1 -- 172.21.15.94:0/1432029562 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc58c07ba00 con 0x7fc5941592d0 2024-03-06T07:30:16.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.960+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5941584a0 con 0x7fc5941592d0 2024-03-06T07:30:16.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.961+0000 7fc582ffd640 1 -- 172.21.15.94:0/1432029562 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fc58c07faa0 con 0x7fc5941592d0 2024-03-06T07:30:16.963 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:16.963 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:16.963 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:16.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc57406b4d0 msgr2=0x7fc57406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc57406b4d0 0x7fc57406d990 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc584007fd0 tx=0x7fc584007de0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 msgr2=0x7fc59414b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc59414b5e0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fc58c037040 tx=0x7fc58c004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:16.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 shutdown_connections 2024-03-06T07:30:16.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc57406b4d0 0x7fc57406d990 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc5941592d0 0x7fc59414b5e0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5940c6780 0x7fc59414b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 --2- 172.21.15.94:0/1432029562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc59406da40 0x7fc5940ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 >> 172.21.15.94:0/1432029562 conn(0x7fc5940aa630 msgr2=0x7fc594153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 shutdown_connections 2024-03-06T07:30:16.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:16.963+0000 7fc598be9640 1 -- 172.21.15.94:0/1432029562 wait complete. 2024-03-06T07:30:17.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:17.379 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:18.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.490+0000 7f15d613e640 1 Processor -- start 2024-03-06T07:30:18.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.490+0000 7f15d613e640 1 -- start start 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15d613e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15d01527b0 0x7f15d0159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15d613e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d00c88b0 0x7f15d00c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15d613e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d00c7c60 con 0x7f15d00c7180 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15d613e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d0159640 con 0x7f15d01527b0 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15d613e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00c8060 con 0x7f15d00c88b0 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15cf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15d01527b0 0x7f15d0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15cf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15d01527b0 0x7f15d0159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34776/0 (socket says 172.21.15.94:34776) 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15cf7fe640 1 -- 172.21.15.94:0/1110391942 learned_addr learned my addr 172.21.15.94:0/1110391942 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.491+0000 7f15cffff640 1 --2- 172.21.15.94:0/1110391942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d00c88b0 0x7f15d00c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.492+0000 7f15cf7fe640 1 -- 172.21.15.94:0/1110391942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d00c88b0 msgr2=0x7f15d00c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.492+0000 7f15cf7fe640 1 --2- 172.21.15.94:0/1110391942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d00c88b0 0x7f15d00c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.492+0000 7f15cf7fe640 1 -- 172.21.15.94:0/1110391942 >> v1:172.21.15.145:6789/0 conn(0x7f15d00c7180 legacy=0x7f15d0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.494 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.492+0000 7f15cf7fe640 1 -- 172.21.15.94:0/1110391942 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d00c76a0 con 0x7f15d01527b0 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.492+0000 7f15cf7fe640 1 --2- 172.21.15.94:0/1110391942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d01527b0 0x7f15d0159100 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009870 tx=0x7f15c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c951dc77957e4009 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15ceffd640 1 -- 172.21.15.94:0/1110391942 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0035020 con 0x7f15d01527b0 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15ceffd640 1 -- 172.21.15.94:0/1110391942 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c0031640 con 0x7f15d01527b0 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15ceffd640 1 -- 172.21.15.94:0/1110391942 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c002d040 con 0x7f15d01527b0 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 -- 172.21.15.94:0/1110391942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d01527b0 msgr2=0x7f15d0159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 --2- 172.21.15.94:0/1110391942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d01527b0 0x7f15d0159100 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009870 tx=0x7f15c002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 -- 172.21.15.94:0/1110391942 shutdown_connections 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 --2- 172.21.15.94:0/1110391942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d00c88b0 0x7f15d00c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 --2- 172.21.15.94:0/1110391942 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d01527b0 0x7f15d0159100 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.496 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 -- 172.21.15.94:0/1110391942 >> 172.21.15.94:0/1110391942 conn(0x7f15d00ac560 msgr2=0x7f15d00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 -- 172.21.15.94:0/1110391942 shutdown_connections 2024-03-06T07:30:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.493+0000 7f15d613e640 1 -- 172.21.15.94:0/1110391942 wait complete. 2024-03-06T07:30:18.497 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.494+0000 7f15d613e640 1 Processor -- start 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15d613e640 1 -- start start 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15d613e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d00c7180 0x7f15d00c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15d613e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15d00c88b0 0x7f15d0153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15d613e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d01527b0 0x7f15d01544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15d613e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d0155600 con 0x7f15d00c7180 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15d613e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d0155320 con 0x7f15d00c88b0 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15d613e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00c8d60 con 0x7f15d01527b0 2024-03-06T07:30:18.499 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d00c7180 0x7f15d00c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15cf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d00c7180 0x7f15d00c4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34778/0 (socket says 172.21.15.94:34778) 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15cf7fe640 1 -- 172.21.15.94:0/2833931177 learned_addr learned my addr 172.21.15.94:0/2833931177 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15cffff640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d01527b0 0x7f15d01544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.495+0000 7f15c6ffd640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15d00c88b0 0x7f15d0153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15cf7fe640 1 -- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d01527b0 msgr2=0x7f15d01544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15cf7fe640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d01527b0 0x7f15d01544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15cf7fe640 1 -- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15d00c88b0 msgr2=0x7f15d0153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15cf7fe640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15d00c88b0 0x7f15d0153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15cf7fe640 1 -- 172.21.15.94:0/2833931177 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d01531d0 con 0x7f15d00c7180 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15cf7fe640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d00c7180 0x7f15d00c4280 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f15c002b930 tx=0x7f15c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15ccff9640 1 -- 172.21.15.94:0/2833931177 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0040070 con 0x7f15d00c7180 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15ccff9640 1 -- 172.21.15.94:0/2833931177 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c003e070 con 0x7f15d00c7180 2024-03-06T07:30:18.500 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d01534b0 con 0x7f15d00c7180 2024-03-06T07:30:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15ccff9640 1 -- 172.21.15.94:0/2833931177 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0035040 con 0x7f15d00c7180 2024-03-06T07:30:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.496+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d00c47c0 con 0x7f15d00c7180 2024-03-06T07:30:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.497+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d004f330 con 0x7f15d00c7180 2024-03-06T07:30:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.497+0000 7f15ccff9640 1 -- 172.21.15.94:0/2833931177 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15c0038020 con 0x7f15d00c7180 2024-03-06T07:30:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.498+0000 7f15ccff9640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15b006b4d0 0x7f15b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.498+0000 7f15ccff9640 1 -- 172.21.15.94:0/2833931177 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f15c00abc50 con 0x7f15d00c7180 2024-03-06T07:30:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.498+0000 7f15c6ffd640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15b006b4d0 0x7f15b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.498+0000 7f15c6ffd640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15b006b4d0 0x7f15b006d990 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f15d00be100 tx=0x7f15b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.504 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.502+0000 7f15ccff9640 1 -- 172.21.15.94:0/2833931177 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15c00b1020 con 0x7f15d00c7180 2024-03-06T07:30:18.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.708+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15d006ce00 con 0x7f15d00c7180 2024-03-06T07:30:18.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.709+0000 7f15ccff9640 1 -- 172.21.15.94:0/2833931177 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f15c007a280 con 0x7f15d00c7180 2024-03-06T07:30:18.711 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:18.711 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:18.711 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15b006b4d0 msgr2=0x7f15b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15b006b4d0 0x7f15b006d990 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f15d00be100 tx=0x7f15b8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d00c7180 msgr2=0x7f15d00c4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d00c7180 0x7f15d00c4280 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f15c002b930 tx=0x7f15c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 shutdown_connections 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15b006b4d0 0x7f15b006d990 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15d01527b0 0x7f15d01544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15d00c88b0 0x7f15d0153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 --2- 172.21.15.94:0/2833931177 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15d00c7180 0x7f15d00c4280 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 >> 172.21.15.94:0/2833931177 conn(0x7f15d00ac560 msgr2=0x7f15d014d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 shutdown_connections 2024-03-06T07:30:18.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:18.711+0000 7f15d613e640 1 -- 172.21.15.94:0/2833931177 wait complete. 2024-03-06T07:30:19.126 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:19.126 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:20.128 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.237+0000 7f1fe7df7640 1 Processor -- start 2024-03-06T07:30:20.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.237+0000 7f1fe7df7640 1 -- start start 2024-03-06T07:30:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe7df7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1fe014f790 0x7f1fe01529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe7df7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00c28d0 con 0x7f1fe00c1aa0 2024-03-06T07:30:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe7df7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00c2bb0 con 0x7f1fe0155a40 2024-03-06T07:30:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe7df7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00c2680 con 0x7f1fe014f790 2024-03-06T07:30:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe5392640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1fe00c1aa0 0x7f1fe00c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37036/0 (socket says 172.21.15.94:37036) 2024-03-06T07:30:20.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe5392640 1 -- 172.21.15.94:0/1056497714 learned_addr learned my addr 172.21.15.94:0/1056497714 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe5b93640 1 --2- 172.21.15.94:0/1056497714 >> v2:172.21.15.145:3300/0 conn(0x7f1fe014f790 0x7f1fe01529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe4b91640 1 -- 172.21.15.94:0/1056497714 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829185278 0 0) 0x7f1fe00c28d0 con 0x7f1fe00c1aa0 2024-03-06T07:30:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe4b91640 1 -- 172.21.15.94:0/1056497714 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc005730 con 0x7f1fe00c1aa0 2024-03-06T07:30:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe5b93640 1 -- 172.21.15.94:0/1056497714 >> v1:172.21.15.145:6789/0 conn(0x7f1fe0155a40 legacy=0x7f1fe0155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe5b93640 1 -- 172.21.15.94:0/1056497714 >> v1:172.21.15.94:6789/0 conn(0x7f1fe00c1aa0 legacy=0x7f1fe00c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe4b91640 1 -- 172.21.15.94:0/1056497714 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986790622 0 0) 0x7f1fe00c2bb0 con 0x7f1fe0155a40 2024-03-06T07:30:20.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.238+0000 7f1fe5b93640 1 -- 172.21.15.94:0/1056497714 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00c2e90 con 0x7f1fe014f790 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.239+0000 7f1fe5b93640 1 --2- 172.21.15.94:0/1056497714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe01529d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00c2800 tx=0x7f1fdc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9815bd3e1621d577 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.239+0000 7f1fe4b91640 1 -- 172.21.15.94:0/1056497714 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc034070 con 0x7f1fe014f790 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.239+0000 7f1fe4b91640 1 -- 172.21.15.94:0/1056497714 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc030610 con 0x7f1fe014f790 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.239+0000 7f1fe4b91640 1 -- 172.21.15.94:0/1056497714 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0309e0 con 0x7f1fe014f790 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.240+0000 7f1fe7df7640 1 -- 172.21.15.94:0/1056497714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 msgr2=0x7f1fe01529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.240+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/1056497714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe01529d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00c2800 tx=0x7f1fdc02b960 comp rx=0 tx=0).stop 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.240+0000 7f1fe7df7640 1 -- 172.21.15.94:0/1056497714 shutdown_connections 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.240+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/1056497714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe01529d0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.240+0000 7f1fe7df7640 1 -- 172.21.15.94:0/1056497714 >> 172.21.15.94:0/1056497714 conn(0x7f1fe00c6710 msgr2=0x7f1fe00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.240+0000 7f1fe7df7640 1 -- 172.21.15.94:0/1056497714 shutdown_connections 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.240+0000 7f1fe7df7640 1 -- 172.21.15.94:0/1056497714 wait complete. 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.241+0000 7f1fe7df7640 1 Processor -- start 2024-03-06T07:30:20.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.241+0000 7f1fe7df7640 1 -- start start 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe7df7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00c1aa0 0x7f1fe014b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe7df7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe00bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe7df7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe0155a40 0x7f1fe0148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe7df7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00c5e30 con 0x7f1fe0155a40 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe7df7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00c0d60 con 0x7f1fe014f790 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe7df7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00c5fb0 con 0x7f1fe00c1aa0 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe0155a40 0x7f1fe0148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe0155a40 0x7f1fe0148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34786/0 (socket says 172.21.15.94:34786) 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe5392640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe00bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 -- 172.21.15.94:0/2686944932 learned_addr learned my addr 172.21.15.94:0/2686944932 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe5b93640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00c1aa0 0x7f1fe014b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 -- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00c1aa0 msgr2=0x7f1fe014b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00c1aa0 0x7f1fe014b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 -- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 msgr2=0x7f1fe00bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe00bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 -- 172.21.15.94:0/2686944932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0155330 con 0x7f1fe0155a40 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe5392640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe00bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe6394640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe0155a40 0x7f1fe0148d50 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c7b0 tx=0x7f1fd800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fd67fc640 1 -- 172.21.15.94:0/2686944932 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8003ba0 con 0x7f1fe0155a40 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fd67fc640 1 -- 172.21.15.94:0/2686944932 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8003d40 con 0x7f1fe0155a40 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe0154a90 con 0x7f1fe0155a40 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.242+0000 7f1fd67fc640 1 -- 172.21.15.94:0/2686944932 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8005270 con 0x7f1fe0155a40 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.243+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe0155050 con 0x7f1fe0155a40 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.243+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe00c5c50 con 0x7f1fe0155a40 2024-03-06T07:30:20.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.244+0000 7f1fd67fc640 1 -- 172.21.15.94:0/2686944932 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd8005410 con 0x7f1fe0155a40 2024-03-06T07:30:20.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.244+0000 7f1fd67fc640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fc406b500 0x7f1fc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.245+0000 7f1fe5b93640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fc406b500 0x7f1fc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.245+0000 7f1fd67fc640 1 -- 172.21.15.94:0/2686944932 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1fd808a160 con 0x7f1fe0155a40 2024-03-06T07:30:20.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.245+0000 7f1fe5b93640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fc406b500 0x7f1fc406d9c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc036000 tx=0x7f1fdc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.249+0000 7f1fd67fc640 1 -- 172.21.15.94:0/2686944932 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fe00c5c50 con 0x7f1fe0155a40 2024-03-06T07:30:20.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.457+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fe004f330 con 0x7f1fe0155a40 2024-03-06T07:30:20.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.458+0000 7f1fd67fc640 1 -- 172.21.15.94:0/2686944932 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f1fd8058790 con 0x7f1fe0155a40 2024-03-06T07:30:20.460 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:20.460 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:20.460 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:20.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fc406b500 msgr2=0x7f1fc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fc406b500 0x7f1fc406d9c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc036000 tx=0x7f1fdc008030 comp rx=0 tx=0).stop 2024-03-06T07:30:20.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe0155a40 msgr2=0x7f1fe0148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe0155a40 0x7f1fe0148d50 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c7b0 tx=0x7f1fd800cbb0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 shutdown_connections 2024-03-06T07:30:20.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1fc406b500 0x7f1fc406d9c0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe0155a40 0x7f1fe0148d50 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe014f790 0x7f1fe00bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 --2- 172.21.15.94:0/2686944932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00c1aa0 0x7f1fe014b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 >> 172.21.15.94:0/2686944932 conn(0x7f1fe00c6710 msgr2=0x7f1fe0155e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 shutdown_connections 2024-03-06T07:30:20.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:20.460+0000 7f1fe7df7640 1 -- 172.21.15.94:0/2686944932 wait complete. 2024-03-06T07:30:20.875 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:20.875 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:21.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:21.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.986+0000 7efdf9be9640 1 Processor -- start 2024-03-06T07:30:21.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.986+0000 7efdf9be9640 1 -- start start 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.986+0000 7efdf9be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf4155ba0 0x7efdf40c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.986+0000 7efdf9be9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efdf41586f0 0x7efdf4158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.986+0000 7efdf9be9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf41591c0 con 0x7efdf40c5480 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.986+0000 7efdf9be9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf406df40 con 0x7efdf41586f0 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.986+0000 7efdf9be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf406fdb0 con 0x7efdf4155ba0 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdebfff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf4155ba0 0x7efdf40c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdf3fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efdf41586f0 0x7efdf4158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdf37fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7efdf40c5480 0x7efdf406f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:60424/0 (socket says 172.21.15.94:60424) 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdf37fe640 1 -- 172.21.15.94:0/2122837067 learned_addr learned my addr 172.21.15.94:0/2122837067 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdf2ffd640 1 -- 172.21.15.94:0/2122837067 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602688098 0 0) 0x7efdf41591c0 con 0x7efdf40c5480 2024-03-06T07:30:21.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdebfff640 1 -- 172.21.15.94:0/2122837067 >> v2:172.21.15.145:3300/0 conn(0x7efdf41586f0 msgr2=0x7efdf4158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdebfff640 1 --2- 172.21.15.94:0/2122837067 >> v2:172.21.15.145:3300/0 conn(0x7efdf41586f0 0x7efdf4158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdebfff640 1 -- 172.21.15.94:0/2122837067 >> v1:172.21.15.145:6789/0 conn(0x7efdf40c5480 legacy=0x7efdf406f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.987+0000 7efdebfff640 1 -- 172.21.15.94:0/2122837067 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf41594a0 con 0x7efdf4155ba0 2024-03-06T07:30:21.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.988+0000 7efdebfff640 1 --2- 172.21.15.94:0/2122837067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf4155ba0 0x7efdf40c5860 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efddc0097a0 tx=0x7efddc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f079e6c964599b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.988+0000 7efdf2ffd640 1 -- 172.21.15.94:0/2122837067 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc0315a0 con 0x7efdf4155ba0 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.988+0000 7efdf2ffd640 1 -- 172.21.15.94:0/2122837067 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc037070 con 0x7efdf4155ba0 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.988+0000 7efdf2ffd640 1 -- 172.21.15.94:0/2122837067 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc031960 con 0x7efdf4155ba0 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 -- 172.21.15.94:0/2122837067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf4155ba0 msgr2=0x7efdf40c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 --2- 172.21.15.94:0/2122837067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf4155ba0 0x7efdf40c5860 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efddc0097a0 tx=0x7efddc02b840 comp rx=0 tx=0).stop 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 -- 172.21.15.94:0/2122837067 shutdown_connections 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 --2- 172.21.15.94:0/2122837067 >> v2:172.21.15.145:3300/0 conn(0x7efdf41586f0 0x7efdf4158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 --2- 172.21.15.94:0/2122837067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf4155ba0 0x7efdf40c5860 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 -- 172.21.15.94:0/2122837067 >> 172.21.15.94:0/2122837067 conn(0x7efdf406b1a0 msgr2=0x7efdf406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 -- 172.21.15.94:0/2122837067 shutdown_connections 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.989+0000 7efdf9be9640 1 -- 172.21.15.94:0/2122837067 wait complete. 2024-03-06T07:30:21.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 Processor -- start 2024-03-06T07:30:21.993 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 -- start start 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf40c5480 0x7efdf414f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdf4155ba0 0x7efdf40c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdf41586f0 0x7efdf40abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4155680 con 0x7efdf41586f0 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4156000 con 0x7efdf4155ba0 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.990+0000 7efdf9be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4152a20 con 0x7efdf40c5480 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdf41586f0 0x7efdf40abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdf41586f0 0x7efdf40abf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34792/0 (socket says 172.21.15.94:34792) 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 -- 172.21.15.94:0/906326401 learned_addr learned my addr 172.21.15.94:0/906326401 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf37fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf40c5480 0x7efdf414f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdebfff640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdf4155ba0 0x7efdf40c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 -- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf40c5480 msgr2=0x7efdf414f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf40c5480 0x7efdf414f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 -- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdf4155ba0 msgr2=0x7efdf40c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdf4155ba0 0x7efdf40c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 -- 172.21.15.94:0/906326401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4156690 con 0x7efdf41586f0 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdebfff640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdf4155ba0 0x7efdf40c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf37fe640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf40c5480 0x7efdf414f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf3fff640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdf41586f0 0x7efdf40abf00 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7efdec00ec10 tx=0x7efdec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf0ff9640 1 -- 172.21.15.94:0/906326401 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec018020 con 0x7efdf41586f0 2024-03-06T07:30:21.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf0ff9640 1 -- 172.21.15.94:0/906326401 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec00cd20 con 0x7efdf41586f0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf4156970 con 0x7efdf41586f0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf0ff9640 1 -- 172.21.15.94:0/906326401 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec0146c0 con 0x7efdf41586f0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.991+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf4156f30 con 0x7efdf41586f0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.992+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf404f330 con 0x7efdf41586f0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.993+0000 7efdf0ff9640 1 -- 172.21.15.94:0/906326401 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdec01b020 con 0x7efdf41586f0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.993+0000 7efdf0ff9640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efdd406b500 0x7efdd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.993+0000 7efdf0ff9640 1 -- 172.21.15.94:0/906326401 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7efdec08af80 con 0x7efdf41586f0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.993+0000 7efdf37fe640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efdd406b500 0x7efdd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.994+0000 7efdf37fe640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efdd406b500 0x7efdd406d9c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efde4000f60 tx=0x7efde4007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:21.998+0000 7efdf0ff9640 1 -- 172.21.15.94:0/906326401 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdec022030 con 0x7efdf41586f0 2024-03-06T07:30:22.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.205+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf414fa60 con 0x7efdf41586f0 2024-03-06T07:30:22.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.206+0000 7efdf0ff9640 1 -- 172.21.15.94:0/906326401 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7efdec010040 con 0x7efdf41586f0 2024-03-06T07:30:22.208 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:22.208 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:22.208 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:22.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efdd406b500 msgr2=0x7efdd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efdd406b500 0x7efdd406d9c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7efde4000f60 tx=0x7efde4007d50 comp rx=0 tx=0).stop 2024-03-06T07:30:22.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdf41586f0 msgr2=0x7efdf40abf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdf41586f0 0x7efdf40abf00 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7efdec00ec10 tx=0x7efdec00c490 comp rx=0 tx=0).stop 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 shutdown_connections 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efdd406b500 0x7efdd406d9c0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdf41586f0 0x7efdf40abf00 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdf4155ba0 0x7efdf40c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 --2- 172.21.15.94:0/906326401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdf40c5480 0x7efdf414f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 >> 172.21.15.94:0/906326401 conn(0x7efdf406b1a0 msgr2=0x7efdf4154b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 shutdown_connections 2024-03-06T07:30:22.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:22.208+0000 7efdf9be9640 1 -- 172.21.15.94:0/906326401 wait complete. 2024-03-06T07:30:22.624 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:22.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:23.625 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.734+0000 7fc310393640 1 Processor -- start 2024-03-06T07:30:23.736 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.734+0000 7fc310393640 1 -- start start 2024-03-06T07:30:23.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc310393640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc308152960 0x7fc30806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc310393640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3081592d0 0x7fc3081552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc310393640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3080ac260 0x7fc308155810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc310393640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc308066e30 con 0x7fc3081592d0 2024-03-06T07:30:23.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc310393640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc308152d40 con 0x7fc308152960 2024-03-06T07:30:23.737 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc310393640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080c1b50 con 0x7fc3080ac260 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3081592d0 0x7fc3081552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3081592d0 0x7fc3081552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34796/0 (socket says 172.21.15.94:34796) 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 -- 172.21.15.94:0/1735314024 learned_addr learned my addr 172.21.15.94:0/1735314024 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30e12f640 1 --2- 172.21.15.94:0/1735314024 >> v2:172.21.15.145:3300/0 conn(0x7fc308152960 0x7fc30806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30e930640 1 --2- 172.21.15.94:0/1735314024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3080ac260 0x7fc308155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 -- 172.21.15.94:0/1735314024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3080ac260 msgr2=0x7fc308155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 --2- 172.21.15.94:0/1735314024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3080ac260 0x7fc308155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 -- 172.21.15.94:0/1735314024 >> v2:172.21.15.145:3300/0 conn(0x7fc308152960 msgr2=0x7fc30806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 --2- 172.21.15.94:0/1735314024 >> v2:172.21.15.145:3300/0 conn(0x7fc308152960 0x7fc30806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.735+0000 7fc30d92e640 1 -- 172.21.15.94:0/1735314024 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308155d50 con 0x7fc3081592d0 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.736+0000 7fc30d92e640 1 --2- 172.21.15.94:0/1735314024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3081592d0 0x7fc3081552d0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fc30806b1c0 tx=0x7fc2f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347fbcdb0a916174 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.736+0000 7fc30d12d640 1 -- 172.21.15.94:0/1735314024 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8035020 con 0x7fc3081592d0 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.736+0000 7fc30d12d640 1 -- 172.21.15.94:0/1735314024 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f8031870 con 0x7fc3081592d0 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.736+0000 7fc30d12d640 1 -- 172.21.15.94:0/1735314024 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8031c40 con 0x7fc3081592d0 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 -- 172.21.15.94:0/1735314024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3081592d0 msgr2=0x7fc3081552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 --2- 172.21.15.94:0/1735314024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3081592d0 0x7fc3081552d0 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7fc30806b1c0 tx=0x7fc2f802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 -- 172.21.15.94:0/1735314024 shutdown_connections 2024-03-06T07:30:23.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 --2- 172.21.15.94:0/1735314024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3080ac260 0x7fc308155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 --2- 172.21.15.94:0/1735314024 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3081592d0 0x7fc3081552d0 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 --2- 172.21.15.94:0/1735314024 >> v2:172.21.15.145:3300/0 conn(0x7fc308152960 0x7fc30806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 -- 172.21.15.94:0/1735314024 >> 172.21.15.94:0/1735314024 conn(0x7fc3080c70b0 msgr2=0x7fc3080c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 -- 172.21.15.94:0/1735314024 shutdown_connections 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.737+0000 7fc310393640 1 -- 172.21.15.94:0/1735314024 wait complete. 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.738+0000 7fc310393640 1 Processor -- start 2024-03-06T07:30:23.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.738+0000 7fc310393640 1 -- start start 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc310393640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3080ac260 0x7fc30806b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc310393640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc308152960 0x7fc30814af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc310393640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc30814bea0 0x7fc3081539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc310393640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc308159760 con 0x7fc30814bea0 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc310393640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080672c0 con 0x7fc3080ac260 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc310393640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080c20a0 con 0x7fc308152960 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc30814bea0 0x7fc3081539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30d92e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc308152960 0x7fc30814af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc30814bea0 0x7fc3081539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34804/0 (socket says 172.21.15.94:34804) 2024-03-06T07:30:23.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e12f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3080ac260 0x7fc30806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 -- 172.21.15.94:0/1413100948 learned_addr learned my addr 172.21.15.94:0/1413100948 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 -- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc308152960 msgr2=0x7fc30814af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc308152960 0x7fc30814af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 -- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3080ac260 msgr2=0x7fc30806b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3080ac260 0x7fc30806b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 -- 172.21.15.94:0/1413100948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3081df520 con 0x7fc30814bea0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc30e930640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc30814bea0 0x7fc3081539a0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fc30000eed0 tx=0x7fc30000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.739+0000 7fc2feffd640 1 -- 172.21.15.94:0/1413100948 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300018020 con 0x7fc30814bea0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.740+0000 7fc2feffd640 1 -- 172.21.15.94:0/1413100948 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc300014400 con 0x7fc30814bea0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.740+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3081531e0 con 0x7fc30814bea0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.740+0000 7fc2feffd640 1 -- 172.21.15.94:0/1413100948 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000147f0 con 0x7fc30814bea0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.740+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc308153480 con 0x7fc30814bea0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.741+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3080c2630 con 0x7fc30814bea0 2024-03-06T07:30:23.745 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.741+0000 7fc2feffd640 1 -- 172.21.15.94:0/1413100948 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc300005d10 con 0x7fc30814bea0 2024-03-06T07:30:23.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.741+0000 7fc2feffd640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc2ec06b4d0 0x7fc2ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.741+0000 7fc2feffd640 1 -- 172.21.15.94:0/1413100948 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc30008a960 con 0x7fc30814bea0 2024-03-06T07:30:23.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.742+0000 7fc30e12f640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc2ec06b4d0 0x7fc2ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.746 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.742+0000 7fc30e12f640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc2ec06b4d0 0x7fc2ec06d990 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc308154a20 tx=0x7fc304009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.747 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.746+0000 7fc2feffd640 1 -- 172.21.15.94:0/1413100948 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3080c2630 con 0x7fc30814bea0 2024-03-06T07:30:23.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.952+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc30804f330 con 0x7fc30814bea0 2024-03-06T07:30:23.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.952+0000 7fc2feffd640 1 -- 172.21.15.94:0/1413100948 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fc300022070 con 0x7fc30814bea0 2024-03-06T07:30:23.954 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:23.955 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:23.955 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:23.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc2ec06b4d0 msgr2=0x7fc2ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc2ec06b4d0 0x7fc2ec06d990 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc308154a20 tx=0x7fc304009190 comp rx=0 tx=0).stop 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc30814bea0 msgr2=0x7fc3081539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc30814bea0 0x7fc3081539a0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fc30000eed0 tx=0x7fc30000b480 comp rx=0 tx=0).stop 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 shutdown_connections 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc2ec06b4d0 0x7fc2ec06d990 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc30814bea0 0x7fc3081539a0 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc308152960 0x7fc30814af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 --2- 172.21.15.94:0/1413100948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3080ac260 0x7fc30806b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 >> 172.21.15.94:0/1413100948 conn(0x7fc3080c70b0 msgr2=0x7fc3080c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 shutdown_connections 2024-03-06T07:30:23.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:23.954+0000 7fc310393640 1 -- 172.21.15.94:0/1413100948 wait complete. 2024-03-06T07:30:24.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:24.369 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:25.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:25.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.480+0000 7fa11998a640 1 Processor -- start 2024-03-06T07:30:25.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.480+0000 7fa11998a640 1 -- start start 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.480+0000 7fa11998a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa11414f360 0x7fa114155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa11998a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1140c73d0 con 0x7fa114158b60 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa11998a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1140be2e0 con 0x7fa1140c88c0 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa11998a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa114159700 con 0x7fa11414f360 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa112ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa11414f360 0x7fa114155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa1137fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa114158b60 0x7fa114158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37050/0 (socket says 172.21.15.94:37050) 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa1137fe640 1 -- 172.21.15.94:0/325300066 learned_addr learned my addr 172.21.15.94:0/325300066 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa112ffd640 1 -- 172.21.15.94:0/325300066 >> v1:172.21.15.145:6789/0 conn(0x7fa1140c88c0 legacy=0x7fa1140c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa112ffd640 1 -- 172.21.15.94:0/325300066 >> v1:172.21.15.94:6789/0 conn(0x7fa114158b60 legacy=0x7fa114158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.484 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.481+0000 7fa112ffd640 1 -- 172.21.15.94:0/325300066 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140c7670 con 0x7fa11414f360 2024-03-06T07:30:25.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa112ffd640 1 --2- 172.21.15.94:0/325300066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa11414f360 0x7fa114155c10 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097a0 tx=0x7fa10402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca5d6e524eb01a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa111ffb640 1 -- 172.21.15.94:0/325300066 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104035020 con 0x7fa11414f360 2024-03-06T07:30:25.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa111ffb640 1 -- 172.21.15.94:0/325300066 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104031830 con 0x7fa11414f360 2024-03-06T07:30:25.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa111ffb640 1 -- 172.21.15.94:0/325300066 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104031c00 con 0x7fa11414f360 2024-03-06T07:30:25.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa11998a640 1 -- 172.21.15.94:0/325300066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa11414f360 msgr2=0x7fa114155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa11998a640 1 --2- 172.21.15.94:0/325300066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa11414f360 0x7fa114155c10 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fa1040097a0 tx=0x7fa10402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa11998a640 1 -- 172.21.15.94:0/325300066 shutdown_connections 2024-03-06T07:30:25.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa11998a640 1 --2- 172.21.15.94:0/325300066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa11414f360 0x7fa114155c10 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa11998a640 1 -- 172.21.15.94:0/325300066 >> 172.21.15.94:0/325300066 conn(0x7fa1140ac570 msgr2=0x7fa1140ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa11998a640 1 -- 172.21.15.94:0/325300066 shutdown_connections 2024-03-06T07:30:25.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.482+0000 7fa11998a640 1 -- 172.21.15.94:0/325300066 wait complete. 2024-03-06T07:30:25.486 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.483+0000 7fa11998a640 1 Processor -- start 2024-03-06T07:30:25.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa11998a640 1 -- start start 2024-03-06T07:30:25.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa11998a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c88c0 0x7fa1140ad660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa11998a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa11414f360 0x7fa114157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa11998a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114158b60 0x7fa1141e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa11998a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140c7c30 con 0x7fa1140c88c0 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa11998a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140c5a50 con 0x7fa114158b60 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa11998a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140c5bd0 con 0x7fa11414f360 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa112ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c88c0 0x7fa1140ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.484+0000 7fa112ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c88c0 0x7fa1140ad660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34826/0 (socket says 172.21.15.94:34826) 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa112ffd640 1 -- 172.21.15.94:0/1436204700 learned_addr learned my addr 172.21.15.94:0/1436204700 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa1137fe640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114158b60 0x7fa1141e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa1127fc640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa11414f360 0x7fa114157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa112ffd640 1 -- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa11414f360 msgr2=0x7fa114157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa112ffd640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa11414f360 0x7fa114157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa112ffd640 1 -- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114158b60 msgr2=0x7fa1141e21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa112ffd640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114158b60 0x7fa1141e21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa112ffd640 1 -- 172.21.15.94:0/1436204700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1141ee270 con 0x7fa1140c88c0 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa112ffd640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c88c0 0x7fa1140ad660 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fa10402bee0 tx=0x7fa1040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa0fbfff640 1 -- 172.21.15.94:0/1436204700 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104041070 con 0x7fa1140c88c0 2024-03-06T07:30:25.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa0fbfff640 1 -- 172.21.15.94:0/1436204700 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10403a950 con 0x7fa1140c88c0 2024-03-06T07:30:25.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1141e16e0 con 0x7fa1140c88c0 2024-03-06T07:30:25.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa0fbfff640 1 -- 172.21.15.94:0/1436204700 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104035040 con 0x7fa1140c88c0 2024-03-06T07:30:25.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.485+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa114157eb0 con 0x7fa1140c88c0 2024-03-06T07:30:25.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.486+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11404f330 con 0x7fa1140c88c0 2024-03-06T07:30:25.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.487+0000 7fa0fbfff640 1 -- 172.21.15.94:0/1436204700 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa10404c060 con 0x7fa1140c88c0 2024-03-06T07:30:25.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.487+0000 7fa0fbfff640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b4d0 0x7fa10006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.487+0000 7fa0fbfff640 1 -- 172.21.15.94:0/1436204700 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa1040abe30 con 0x7fa1140c88c0 2024-03-06T07:30:25.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.487+0000 7fa1127fc640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b4d0 0x7fa10006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.491 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.488+0000 7fa1127fc640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b4d0 0x7fa10006d990 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa114158960 tx=0x7fa108009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.493 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.491+0000 7fa0fbfff640 1 -- 172.21.15.94:0/1436204700 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa10407a3e0 con 0x7fa1140c88c0 2024-03-06T07:30:25.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.702+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1140c8d50 con 0x7fa1140c88c0 2024-03-06T07:30:25.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.703+0000 7fa0fbfff640 1 -- 172.21.15.94:0/1436204700 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fa10403de20 con 0x7fa1140c88c0 2024-03-06T07:30:25.705 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:25.705 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:25.705 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b4d0 msgr2=0x7fa10006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b4d0 0x7fa10006d990 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa114158960 tx=0x7fa108009040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c88c0 msgr2=0x7fa1140ad660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c88c0 0x7fa1140ad660 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fa10402bee0 tx=0x7fa1040046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 shutdown_connections 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa10006b4d0 0x7fa10006d990 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa114158b60 0x7fa1141e21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa11414f360 0x7fa114157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 --2- 172.21.15.94:0/1436204700 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1140c88c0 0x7fa1140ad660 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 >> 172.21.15.94:0/1436204700 conn(0x7fa1140ac570 msgr2=0x7fa1140ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 shutdown_connections 2024-03-06T07:30:25.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:25.705+0000 7fa11998a640 1 -- 172.21.15.94:0/1436204700 wait complete. 2024-03-06T07:30:26.120 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:26.120 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:27.122 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.232+0000 7efdbd381640 1 Processor -- start 2024-03-06T07:30:27.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.232+0000 7efdbd381640 1 -- start start 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdbd381640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80bd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdbd381640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efdb8073250 0x7efdb80c3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdbd381640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb806c8e0 con 0x7efdb80c3950 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdbd381640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb806ce80 con 0x7efdb8073250 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdbd381640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80c1b00 con 0x7efdb8072e70 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7efdb80c3950 0x7efdb81589b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37054/0 (socket says 172.21.15.94:37054) 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb77fe640 1 -- 172.21.15.94:0/3643263128 learned_addr learned my addr 172.21.15.94:0/3643263128 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:27.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb67fc640 1 --2- 172.21.15.94:0/3643263128 >> v2:172.21.15.145:3300/0 conn(0x7efdb8073250 0x7efdb80c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb6ffd640 1 --2- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80bd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb5ffb640 1 -- 172.21.15.94:0/3643263128 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500473264 0 0) 0x7efdb806c8e0 con 0x7efdb80c3950 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb5ffb640 1 -- 172.21.15.94:0/3643263128 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda0003520 con 0x7efdb80c3950 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb67fc640 1 -- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 msgr2=0x7efdb80bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb67fc640 1 --2- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80bd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb67fc640 1 -- 172.21.15.94:0/3643263128 >> v1:172.21.15.94:6789/0 conn(0x7efdb80c3950 legacy=0x7efdb81589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb67fc640 1 -- 172.21.15.94:0/3643263128 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb806cba0 con 0x7efdb8073250 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.233+0000 7efdb6ffd640 1 --2- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.234+0000 7efdb67fc640 1 --2- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 0x7efdb80c3410 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7efdac0097a0 tx=0x7efdac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f67cc11f16ef78a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.234+0000 7efdb5ffb640 1 -- 172.21.15.94:0/3643263128 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac0315a0 con 0x7efdb8073250 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.234+0000 7efdb5ffb640 1 -- 172.21.15.94:0/3643263128 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac037070 con 0x7efdb8073250 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.234+0000 7efdb5ffb640 1 -- 172.21.15.94:0/3643263128 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac031940 con 0x7efdb8073250 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 -- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 msgr2=0x7efdb80c3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 --2- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 0x7efdb80c3410 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7efdac0097a0 tx=0x7efdac02b840 comp rx=0 tx=0).stop 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 -- 172.21.15.94:0/3643263128 shutdown_connections 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 --2- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 0x7efdb80c3410 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 --2- 172.21.15.94:0/3643263128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 -- 172.21.15.94:0/3643263128 >> 172.21.15.94:0/3643263128 conn(0x7efdb806b1a0 msgr2=0x7efdb806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 -- 172.21.15.94:0/3643263128 shutdown_connections 2024-03-06T07:30:27.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.235+0000 7efdbd381640 1 -- 172.21.15.94:0/3643263128 wait complete. 2024-03-06T07:30:27.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.236+0000 7efdbd381640 1 Processor -- start 2024-03-06T07:30:27.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.236+0000 7efdbd381640 1 -- start start 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdbd381640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80c00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdbd381640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 0x7efdb80c0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdbd381640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdb80c3950 0x7efdb81ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdbd381640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80c4b80 con 0x7efdb80c3950 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdbd381640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80c4a00 con 0x7efdb8073250 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdbd381640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80bda50 con 0x7efdb8072e70 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdb80c3950 0x7efdb81ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdb80c3950 0x7efdb81ee740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:34830/0 (socket says 172.21.15.94:34830) 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 -- 172.21.15.94:0/873829598 learned_addr learned my addr 172.21.15.94:0/873829598 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb6ffd640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80c00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb67fc640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 0x7efdb80c0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 -- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 msgr2=0x7efdb80c00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80c00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 -- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 msgr2=0x7efdb80c0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 0x7efdb80c0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 -- 172.21.15.94:0/873829598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb81eec80 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.237+0000 7efdb77fe640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdb80c3950 0x7efdb81ee740 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7efdb000cb40 tx=0x7efdb000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.238+0000 7efd9f7fe640 1 -- 172.21.15.94:0/873829598 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0017070 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.238+0000 7efd9f7fe640 1 -- 172.21.15.94:0/873829598 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb000aca0 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.238+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb80c0e40 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.238+0000 7efd9f7fe640 1 -- 172.21.15.94:0/873829598 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0012620 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.238+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb8154b50 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.239+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb804f330 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.239+0000 7efd9f7fe640 1 -- 172.21.15.94:0/873829598 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdb001a070 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.240+0000 7efd9f7fe640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efda406b500 0x7efda406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.240+0000 7efd9f7fe640 1 -- 172.21.15.94:0/873829598 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7efdb008a1d0 con 0x7efdb80c3950 2024-03-06T07:30:27.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.240+0000 7efdb6ffd640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efda406b500 0x7efda406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.240+0000 7efdb6ffd640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efda406b500 0x7efda406d9c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7efda8009560 tx=0x7efda8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.244+0000 7efd9f7fe640 1 -- 172.21.15.94:0/873829598 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdb0058770 con 0x7efdb80c3950 2024-03-06T07:30:27.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.457+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdb80c6340 con 0x7efdb80c3950 2024-03-06T07:30:27.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.458+0000 7efd9f7fe640 1 -- 172.21.15.94:0/873829598 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7efdb005c810 con 0x7efdb80c3950 2024-03-06T07:30:27.460 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:27.460 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:27.460 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:27.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efda406b500 msgr2=0x7efda406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efda406b500 0x7efda406d9c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7efda8009560 tx=0x7efda8009190 comp rx=0 tx=0).stop 2024-03-06T07:30:27.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdb80c3950 msgr2=0x7efdb81ee740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdb80c3950 0x7efdb81ee740 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7efdb000cb40 tx=0x7efdb000a060 comp rx=0 tx=0).stop 2024-03-06T07:30:27.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 shutdown_connections 2024-03-06T07:30:27.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7efda406b500 0x7efda406d9c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdb80c3950 0x7efdb81ee740 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdb8073250 0x7efdb80c0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 --2- 172.21.15.94:0/873829598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdb8072e70 0x7efdb80c00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 >> 172.21.15.94:0/873829598 conn(0x7efdb806b1a0 msgr2=0x7efdb80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 shutdown_connections 2024-03-06T07:30:27.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:27.460+0000 7efdbd381640 1 -- 172.21.15.94:0/873829598 wait complete. 2024-03-06T07:30:27.875 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:27.875 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:28.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.985+0000 7f20b1a1c640 1 Processor -- start 2024-03-06T07:30:28.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.985+0000 7f20b1a1c640 1 -- start start 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20b1a1c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20ac0c3370 0x7f20ac155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20b1a1c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f20ac0a5e60 0x7f20ac156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20b1a1c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac0c5490 con 0x7f20ac0c3750 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20b1a1c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac0c5770 con 0x7f20ac0c3370 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20b1a1c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac0c5a50 con 0x7f20ac0a5e60 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20aaffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20ac0c3370 0x7f20ac155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20aaffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f20ac0c3370 0x7f20ac155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36552/0 (socket says 172.21.15.94:36552) 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20aaffd640 1 -- 172.21.15.94:0/2805504308 learned_addr learned my addr 172.21.15.94:0/2805504308 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.986+0000 7f20ab7fe640 1 --2- 172.21.15.94:0/2805504308 >> v2:172.21.15.145:3300/0 conn(0x7f20ac0a5e60 0x7f20ac156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.987+0000 7f20aaffd640 1 -- 172.21.15.94:0/2805504308 >> v2:172.21.15.145:3300/0 conn(0x7f20ac0a5e60 msgr2=0x7f20ac156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.987+0000 7f20aaffd640 1 --2- 172.21.15.94:0/2805504308 >> v2:172.21.15.145:3300/0 conn(0x7f20ac0a5e60 0x7f20ac156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.987+0000 7f20aaffd640 1 -- 172.21.15.94:0/2805504308 >> v1:172.21.15.145:6789/0 conn(0x7f20ac0c3750 legacy=0x7f20ac0c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.987+0000 7f20aaffd640 1 -- 172.21.15.94:0/2805504308 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac0c5d30 con 0x7f20ac0c3370 2024-03-06T07:30:28.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.987+0000 7f20aaffd640 1 --2- 172.21.15.94:0/2805504308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac155ad0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0c4190 tx=0x7f209c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f04d8c9928bd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20a9ffb640 1 -- 172.21.15.94:0/2805504308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c02c9f0 con 0x7f20ac0c3370 2024-03-06T07:30:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20a9ffb640 1 -- 172.21.15.94:0/2805504308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c02cb90 con 0x7f20ac0c3370 2024-03-06T07:30:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20a9ffb640 1 -- 172.21.15.94:0/2805504308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c0343f0 con 0x7f20ac0c3370 2024-03-06T07:30:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2805504308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 msgr2=0x7f20ac155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.991 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2805504308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac155ad0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0c4190 tx=0x7f209c02be50 comp rx=0 tx=0).stop 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2805504308 shutdown_connections 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2805504308 >> v2:172.21.15.145:3300/0 conn(0x7f20ac0a5e60 0x7f20ac156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2805504308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac155ad0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2805504308 >> 172.21.15.94:0/2805504308 conn(0x7f20ac0c6710 msgr2=0x7f20ac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2805504308 shutdown_connections 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.988+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2805504308 wait complete. 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.989+0000 7f20b1a1c640 1 Processor -- start 2024-03-06T07:30:28.992 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20b1a1c640 1 -- start start 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20b1a1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20ac0a5e60 0x7f20ac071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20b1a1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac14ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20b1a1c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20ac0c3750 0x7f20ac1ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20b1a1c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac159910 con 0x7f20ac0c3370 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20b1a1c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac159790 con 0x7f20ac0c3750 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20b1a1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac0c4ed0 con 0x7f20ac0a5e60 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20aa7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac14ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20aa7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac14ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36568/0 (socket says 172.21.15.94:36568) 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20aa7fc640 1 -- 172.21.15.94:0/2129721508 learned_addr learned my addr 172.21.15.94:0/2129721508 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:28.995 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20ab7fe640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20ac0c3750 0x7f20ac1ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.990+0000 7f20aaffd640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20ac0a5e60 0x7f20ac071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20aa7fc640 1 -- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20ac0a5e60 msgr2=0x7f20ac071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20aa7fc640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20ac0a5e60 0x7f20ac071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20aa7fc640 1 -- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20ac0c3750 msgr2=0x7f20ac1ee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20aa7fc640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20ac0c3750 0x7f20ac1ee280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20aa7fc640 1 -- 172.21.15.94:0/2129721508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac1edac0 con 0x7f20ac0c3370 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20aa7fc640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac14ffc0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c430 tx=0x7f20a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f2093fff640 1 -- 172.21.15.94:0/2129721508 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0017070 con 0x7f20ac0c3370 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f2093fff640 1 -- 172.21.15.94:0/2129721508 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a00052d0 con 0x7f20ac0c3370 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac1eb090 con 0x7f20ac0c3370 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f2093fff640 1 -- 172.21.15.94:0/2129721508 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0003830 con 0x7f20ac0c3370 2024-03-06T07:30:28.996 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.991+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac150500 con 0x7f20ac0c3370 2024-03-06T07:30:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.992+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac04f330 con 0x7f20ac0c3370 2024-03-06T07:30:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.992+0000 7f2093fff640 1 -- 172.21.15.94:0/2129721508 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20a001a070 con 0x7f20ac0c3370 2024-03-06T07:30:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.993+0000 7f2093fff640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.993+0000 7f2093fff640 1 -- 172.21.15.94:0/2129721508 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f20a0089760 con 0x7f20ac0c3370 2024-03-06T07:30:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.993+0000 7f20aaffd640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.997 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.993+0000 7f20aaffd640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f209806b4d0 0x7f209806d990 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f209c035000 tx=0x7f209c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:28.997+0000 7f2093fff640 1 -- 172.21.15.94:0/2129721508 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20a0057d00 con 0x7f20ac0c3370 2024-03-06T07:30:29.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.202+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20ac068450 con 0x7f20ac0c3370 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.202+0000 7f2093fff640 1 -- 172.21.15.94:0/2129721508 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f20a0087e20 con 0x7f20ac0c3370 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:29.204 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:29.205 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f209806b4d0 msgr2=0x7f209806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f209806b4d0 0x7f209806d990 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f209c035000 tx=0x7f209c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 msgr2=0x7f20ac14ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac14ffc0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c430 tx=0x7f20a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 shutdown_connections 2024-03-06T07:30:29.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20ac0c3750 0x7f20ac1ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f20ac0c3370 0x7f20ac14ffc0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 --2- 172.21.15.94:0/2129721508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20ac0a5e60 0x7f20ac071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 >> 172.21.15.94:0/2129721508 conn(0x7f20ac0c6710 msgr2=0x7f20ac0a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 shutdown_connections 2024-03-06T07:30:29.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:29.204+0000 7f20b1a1c640 1 -- 172.21.15.94:0/2129721508 wait complete. 2024-03-06T07:30:29.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:29.619 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:30.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:30.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.730+0000 7fa5ad62a640 1 Processor -- start 2024-03-06T07:30:30.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.730+0000 7fa5ad62a640 1 -- start start 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5ad62a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a8158db0 0x7fa5a80c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5ad62a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a8155110 con 0x7fa5a80c88b0 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5ad62a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a80c8f40 con 0x7fa5a80c5330 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5ad62a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a81559e0 con 0x7fa5a8158db0 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa5a80c88b0 0x7fa5a815fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37108/0 (socket says 172.21.15.94:37108) 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a77fe640 1 -- 172.21.15.94:0/3563422102 learned_addr learned my addr 172.21.15.94:0/3563422102 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a67fc640 1 --2- 172.21.15.94:0/3563422102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a8158db0 0x7fa5a80c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.734 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a5ffb640 1 -- 172.21.15.94:0/3563422102 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910826670 0 0) 0x7fa5a0001490 con 0x7fa5a80c88b0 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a5ffb640 1 -- 172.21.15.94:0/3563422102 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa590003520 con 0x7fa5a80c88b0 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a5ffb640 1 -- 172.21.15.94:0/3563422102 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897191592 0 0) 0x7fa5a80c8f40 con 0x7fa5a80c5330 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a5ffb640 1 -- 172.21.15.94:0/3563422102 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a0001490 con 0x7fa5a80c5330 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a67fc640 1 -- 172.21.15.94:0/3563422102 >> v1:172.21.15.145:6789/0 conn(0x7fa5a80c5330 legacy=0x7fa5a81586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a67fc640 1 -- 172.21.15.94:0/3563422102 >> v1:172.21.15.94:6789/0 conn(0x7fa5a80c88b0 legacy=0x7fa5a815fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.731+0000 7fa5a67fc640 1 -- 172.21.15.94:0/3563422102 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a8155390 con 0x7fa5a8158db0 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.732+0000 7fa5a67fc640 1 --2- 172.21.15.94:0/3563422102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a8158db0 0x7fa5a80c5710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8155b60 tx=0x7fa59c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f81c49b231e775bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.732+0000 7fa5a5ffb640 1 -- 172.21.15.94:0/3563422102 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c035020 con 0x7fa5a8158db0 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.732+0000 7fa5a5ffb640 1 -- 172.21.15.94:0/3563422102 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c031800 con 0x7fa5a8158db0 2024-03-06T07:30:30.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.732+0000 7fa5a5ffb640 1 -- 172.21.15.94:0/3563422102 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c031bb0 con 0x7fa5a8158db0 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.733+0000 7fa5ad62a640 1 -- 172.21.15.94:0/3563422102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a8158db0 msgr2=0x7fa5a80c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.733+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/3563422102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a8158db0 0x7fa5a80c5710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8155b60 tx=0x7fa59c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.733+0000 7fa5ad62a640 1 -- 172.21.15.94:0/3563422102 shutdown_connections 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.733+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/3563422102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a8158db0 0x7fa5a80c5710 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.733+0000 7fa5ad62a640 1 -- 172.21.15.94:0/3563422102 >> 172.21.15.94:0/3563422102 conn(0x7fa5a80ac560 msgr2=0x7fa5a80ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.733+0000 7fa5ad62a640 1 -- 172.21.15.94:0/3563422102 shutdown_connections 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.733+0000 7fa5ad62a640 1 -- 172.21.15.94:0/3563422102 wait complete. 2024-03-06T07:30:30.738 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.734+0000 7fa5ad62a640 1 Processor -- start 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.734+0000 7fa5ad62a640 1 -- start start 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5ad62a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5a80c5330 0x7fa5a814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5ad62a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a80c88b0 0x7fa5a8157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5ad62a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa5a814c090 0x7fa5a814d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5ad62a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a81529d0 con 0x7fa5a814c090 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5ad62a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a806d090 con 0x7fa5a80c5330 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5ad62a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a8152b50 con 0x7fa5a80c88b0 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa5a814c090 0x7fa5a814d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa5a814c090 0x7fa5a814d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36570/0 (socket says 172.21.15.94:36570) 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 -- 172.21.15.94:0/878164914 learned_addr learned my addr 172.21.15.94:0/878164914 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a67fc640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a80c88b0 0x7fa5a8157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a6ffd640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5a80c5330 0x7fa5a814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.739 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 -- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a80c88b0 msgr2=0x7fa5a8157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a80c88b0 0x7fa5a8157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 -- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5a80c5330 msgr2=0x7fa5a814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5a80c5330 0x7fa5a814b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 -- 172.21.15.94:0/878164914 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80c3c70 con 0x7fa5a814c090 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.735+0000 7fa5a77fe640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa5a814c090 0x7fa5a814d100 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001fc0 tx=0x7fa5a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.736+0000 7fa58f7fe640 1 -- 172.21.15.94:0/878164914 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0017070 con 0x7fa5a814c090 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.736+0000 7fa58f7fe640 1 -- 172.21.15.94:0/878164914 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0012400 con 0x7fa5a814c090 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.736+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a81eb160 con 0x7fa5a814c090 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.736+0000 7fa58f7fe640 1 -- 172.21.15.94:0/878164914 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00127b0 con 0x7fa5a814c090 2024-03-06T07:30:30.740 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.736+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a81eb720 con 0x7fa5a814c090 2024-03-06T07:30:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.737+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a804f330 con 0x7fa5a814c090 2024-03-06T07:30:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.737+0000 7fa58f7fe640 1 -- 172.21.15.94:0/878164914 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5a001a070 con 0x7fa5a814c090 2024-03-06T07:30:30.741 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.738+0000 7fa58f7fe640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa59406b500 0x7fa59406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:30.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.738+0000 7fa58f7fe640 1 -- 172.21.15.94:0/878164914 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa5a008a260 con 0x7fa5a814c090 2024-03-06T07:30:30.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.738+0000 7fa5a6ffd640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa59406b500 0x7fa59406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:30.742 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.738+0000 7fa5a6ffd640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa59406b500 0x7fa59406d9c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa5980010e0 tx=0x7fa598009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:30.744 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.742+0000 7fa58f7fe640 1 -- 172.21.15.94:0/878164914 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5a80c3c70 con 0x7fa5a814c090 2024-03-06T07:30:30.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.949+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5a806cdc0 con 0x7fa5a814c090 2024-03-06T07:30:30.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.950+0000 7fa58f7fe640 1 -- 172.21.15.94:0/878164914 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fa5a8155110 con 0x7fa5a814c090 2024-03-06T07:30:30.952 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:30.952 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:30.952 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.951+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa59406b500 msgr2=0x7fa59406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.951+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa59406b500 0x7fa59406d9c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa5980010e0 tx=0x7fa598009040 comp rx=0 tx=0).stop 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.951+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa5a814c090 msgr2=0x7fa5a814d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.951+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa5a814c090 0x7fa5a814d100 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0001fc0 tx=0x7fa5a000a430 comp rx=0 tx=0).stop 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 shutdown_connections 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa59406b500 0x7fa59406d9c0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa5a814c090 0x7fa5a814d100 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5a80c88b0 0x7fa5a8157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 --2- 172.21.15.94:0/878164914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5a80c5330 0x7fa5a814b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 >> 172.21.15.94:0/878164914 conn(0x7fa5a80ac560 msgr2=0x7fa5a80ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 shutdown_connections 2024-03-06T07:30:30.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:30.952+0000 7fa5ad62a640 1 -- 172.21.15.94:0/878164914 wait complete. 2024-03-06T07:30:31.367 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:31.367 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:32.368 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:32.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.479+0000 7f19ad95c640 1 Processor -- start 2024-03-06T07:30:32.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.479+0000 7f19ad95c640 1 -- start start 2024-03-06T07:30:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.479+0000 7f19ad95c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a8159270 0x7f19a8155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.479+0000 7f19ad95c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a80c73d0 con 0x7f19a80c88c0 2024-03-06T07:30:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.479+0000 7f19ad95c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a80be2e0 con 0x7f19a814f360 2024-03-06T07:30:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.479+0000 7f19ad95c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8159700 con 0x7f19a8159270 2024-03-06T07:30:32.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f199e7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f19a80c88c0 0x7f19a8158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37112/0 (socket says 172.21.15.94:37112) 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f199e7fc640 1 -- 172.21.15.94:0/875318991 learned_addr learned my addr 172.21.15.94:0/875318991 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a77fe640 1 --2- 172.21.15.94:0/875318991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a8159270 0x7f19a8155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a67fc640 1 -- 172.21.15.94:0/875318991 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021757408 0 0) 0x7f19a80c73d0 con 0x7f19a80c88c0 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a67fc640 1 -- 172.21.15.94:0/875318991 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1994003520 con 0x7f19a80c88c0 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a67fc640 1 -- 172.21.15.94:0/875318991 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113318984 0 0) 0x7f19a80be2e0 con 0x7f19a814f360 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a67fc640 1 -- 172.21.15.94:0/875318991 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a80c73d0 con 0x7f19a814f360 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a77fe640 1 -- 172.21.15.94:0/875318991 >> v1:172.21.15.145:6789/0 conn(0x7f19a814f360 legacy=0x7f19a80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a77fe640 1 -- 172.21.15.94:0/875318991 >> v1:172.21.15.94:6789/0 conn(0x7f19a80c88c0 legacy=0x7f19a8158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.480+0000 7f19a77fe640 1 -- 172.21.15.94:0/875318991 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a80c7670 con 0x7f19a8159270 2024-03-06T07:30:32.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.481+0000 7f19a77fe640 1 --2- 172.21.15.94:0/875318991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a8159270 0x7f19a8155c10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009dc0 tx=0x7f19a002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=604ebb0b207a5d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.481+0000 7f19a67fc640 1 -- 172.21.15.94:0/875318991 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0039070 con 0x7f19a8159270 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.481+0000 7f19a67fc640 1 -- 172.21.15.94:0/875318991 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a002cc40 con 0x7f19a8159270 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.481+0000 7f19a67fc640 1 -- 172.21.15.94:0/875318991 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0034440 con 0x7f19a8159270 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.481+0000 7f19ad95c640 1 -- 172.21.15.94:0/875318991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a8159270 msgr2=0x7f19a8155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.481+0000 7f19ad95c640 1 --2- 172.21.15.94:0/875318991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a8159270 0x7f19a8155c10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009dc0 tx=0x7f19a002be90 comp rx=0 tx=0).stop 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.482+0000 7f19ad95c640 1 -- 172.21.15.94:0/875318991 shutdown_connections 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.482+0000 7f19ad95c640 1 --2- 172.21.15.94:0/875318991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a8159270 0x7f19a8155c10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.482+0000 7f19ad95c640 1 -- 172.21.15.94:0/875318991 >> 172.21.15.94:0/875318991 conn(0x7f19a80ac570 msgr2=0x7f19a80ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.482+0000 7f19ad95c640 1 -- 172.21.15.94:0/875318991 shutdown_connections 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.482+0000 7f19ad95c640 1 -- 172.21.15.94:0/875318991 wait complete. 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.483+0000 7f19ad95c640 1 Processor -- start 2024-03-06T07:30:32.485 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.483+0000 7f19ad95c640 1 -- start start 2024-03-06T07:30:32.487 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.483+0000 7f19ad95c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a80c88c0 0x7f19a80af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.483+0000 7f19ad95c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a814f360 0x7f19a814b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.483+0000 7f19ad95c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19a8159270 0x7f19a814bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19ad95c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a80c7c30 con 0x7f19a8159270 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19ad95c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a80c5a50 con 0x7f19a80c88c0 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19ad95c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a80c5bd0 con 0x7f19a814f360 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19a8159270 0x7f19a814bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19a8159270 0x7f19a814bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36582/0 (socket says 172.21.15.94:36582) 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 -- 172.21.15.94:0/1517789366 learned_addr learned my addr 172.21.15.94:0/1517789366 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a6ffd640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a80c88c0 0x7f19a80af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f199e7fc640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a814f360 0x7f19a814b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.488 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 -- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a814f360 msgr2=0x7f19a814b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a814f360 0x7f19a814b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 -- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a80c88c0 msgr2=0x7f19a80af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a80c88c0 0x7f19a80af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 -- 172.21.15.94:0/1517789366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a81eb8a0 con 0x7f19a8159270 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a77fe640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19a8159270 0x7f19a814bcd0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009ee0 tx=0x7f19a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a4ff9640 1 -- 172.21.15.94:0/1517789366 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0039050 con 0x7f19a8159270 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a4ff9640 1 -- 172.21.15.94:0/1517789366 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a0045070 con 0x7f19a8159270 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19a4ff9640 1 -- 172.21.15.94:0/1517789366 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0040690 con 0x7f19a8159270 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a80b0280 con 0x7f19a8159270 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.484+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a80ad320 con 0x7f19a8159270 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.485+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a804f330 con 0x7f19a8159270 2024-03-06T07:30:32.489 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.486+0000 7f19a4ff9640 1 -- 172.21.15.94:0/1517789366 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19a0040830 con 0x7f19a8159270 2024-03-06T07:30:32.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.486+0000 7f19a4ff9640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.486+0000 7f19a4ff9640 1 -- 172.21.15.94:0/1517789366 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f19a00ac5b0 con 0x7f19a8159270 2024-03-06T07:30:32.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.486+0000 7f19a6ffd640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.490 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.487+0000 7f19a6ffd640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1998007fd0 tx=0x7f1998009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.492 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.491+0000 7f19a4ff9640 1 -- 172.21.15.94:0/1517789366 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19a004e020 con 0x7f19a8159270 2024-03-06T07:30:32.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.697+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a80c8d50 con 0x7f19a8159270 2024-03-06T07:30:32.700 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.698+0000 7f19a4ff9640 1 -- 172.21.15.94:0/1517789366 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f19a007abe0 con 0x7f19a8159270 2024-03-06T07:30:32.700 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:32.700 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:32.700 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:32.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 msgr2=0x7f198806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1998007fd0 tx=0x7f1998009040 comp rx=0 tx=0).stop 2024-03-06T07:30:32.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19a8159270 msgr2=0x7f19a814bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19a8159270 0x7f19a814bcd0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009ee0 tx=0x7f19a0004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:32.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 shutdown_connections 2024-03-06T07:30:32.702 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f198806b500 0x7f198806d9c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f19a8159270 0x7f19a814bcd0 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19a814f360 0x7f19a814b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 --2- 172.21.15.94:0/1517789366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19a80c88c0 0x7f19a80af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 >> 172.21.15.94:0/1517789366 conn(0x7f19a80ac570 msgr2=0x7f19a8154a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 shutdown_connections 2024-03-06T07:30:32.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:32.700+0000 7f19ad95c640 1 -- 172.21.15.94:0/1517789366 wait complete. 2024-03-06T07:30:33.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:33.115 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:34.116 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:34.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.227+0000 7fa06bfe9640 1 Processor -- start 2024-03-06T07:30:34.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.227+0000 7fa06bfe9640 1 -- start start 2024-03-06T07:30:34.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa06bfe9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa064150920 0x7fa0640c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa06bfe9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa06406c720 0x7fa064071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa06bfe9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c2e70 0x7fa064071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa06bfe9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa064072400 con 0x7fa064150920 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa06bfe9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa064067340 con 0x7fa06406c720 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa06bfe9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa064150d00 con 0x7fa0640c2e70 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa064150920 0x7fa0640c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa064150920 0x7fa0640c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36584/0 (socket says 172.21.15.94:36584) 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 -- 172.21.15.94:0/2661337691 learned_addr learned my addr 172.21.15.94:0/2661337691 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa06a586640 1 --2- 172.21.15.94:0/2661337691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c2e70 0x7fa064071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069584640 1 --2- 172.21.15.94:0/2661337691 >> v2:172.21.15.145:3300/0 conn(0x7fa06406c720 0x7fa064071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 -- 172.21.15.94:0/2661337691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c2e70 msgr2=0x7fa064071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 --2- 172.21.15.94:0/2661337691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c2e70 0x7fa064071ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 -- 172.21.15.94:0/2661337691 >> v2:172.21.15.145:3300/0 conn(0x7fa06406c720 msgr2=0x7fa064071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 --2- 172.21.15.94:0/2661337691 >> v2:172.21.15.145:3300/0 conn(0x7fa06406c720 0x7fa064071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.228+0000 7fa069d85640 1 -- 172.21.15.94:0/2661337691 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640c2010 con 0x7fa064150920 2024-03-06T07:30:34.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.229+0000 7fa069d85640 1 --2- 172.21.15.94:0/2661337691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa064150920 0x7fa0640c2930 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fa06406ef00 tx=0x7fa06002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cad0865f588731 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.229+0000 7fa068d83640 1 -- 172.21.15.94:0/2661337691 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06002e070 con 0x7fa064150920 2024-03-06T07:30:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.229+0000 7fa068d83640 1 -- 172.21.15.94:0/2661337691 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060037070 con 0x7fa064150920 2024-03-06T07:30:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.229+0000 7fa068d83640 1 -- 172.21.15.94:0/2661337691 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060031a00 con 0x7fa064150920 2024-03-06T07:30:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2661337691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa064150920 msgr2=0x7fa0640c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2661337691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa064150920 0x7fa0640c2930 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7fa06406ef00 tx=0x7fa06002b880 comp rx=0 tx=0).stop 2024-03-06T07:30:34.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2661337691 shutdown_connections 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2661337691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0640c2e70 0x7fa064071ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2661337691 >> v2:172.21.15.145:3300/0 conn(0x7fa06406c720 0x7fa064071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2661337691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa064150920 0x7fa0640c2930 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2661337691 >> 172.21.15.94:0/2661337691 conn(0x7fa0640c70b0 msgr2=0x7fa0640c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2661337691 shutdown_connections 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.230+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2661337691 wait complete. 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.231+0000 7fa06bfe9640 1 Processor -- start 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.231+0000 7fa06bfe9640 1 -- start start 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.231+0000 7fa06bfe9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa06406c720 0x7fa0640c0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.231+0000 7fa06bfe9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0640c2e70 0x7fa0641ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.231+0000 7fa06bfe9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064150920 0x7fa0641eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa06bfe9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064073090 con 0x7fa0640c2e70 2024-03-06T07:30:34.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa06bfe9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa064072b70 con 0x7fa064150920 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa06bfe9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa064073210 con 0x7fa06406c720 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0640c2e70 0x7fa0641ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0640c2e70 0x7fa0641ee5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36596/0 (socket says 172.21.15.94:36596) 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 -- 172.21.15.94:0/2568717797 learned_addr learned my addr 172.21.15.94:0/2568717797 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa06a586640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064150920 0x7fa0641eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069d85640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa06406c720 0x7fa0640c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 -- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa06406c720 msgr2=0x7fa0640c0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa06406c720 0x7fa0640c0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 -- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064150920 msgr2=0x7fa0641eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064150920 0x7fa0641eeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 -- 172.21.15.94:0/2568717797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064149250 con 0x7fa0640c2e70 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa069584640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0640c2e70 0x7fa0641ee5d0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fa05400e870 tx=0x7fa05400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa05a7fc640 1 -- 172.21.15.94:0/2568717797 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054017070 con 0x7fa0640c2e70 2024-03-06T07:30:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa05a7fc640 1 -- 172.21.15.94:0/2568717797 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05400ee60 con 0x7fa0640c2e70 2024-03-06T07:30:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06414ada0 con 0x7fa0640c2e70 2024-03-06T07:30:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa05a7fc640 1 -- 172.21.15.94:0/2568717797 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400cb80 con 0x7fa0640c2e70 2024-03-06T07:30:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.232+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0640aa3d0 con 0x7fa0640c2e70 2024-03-06T07:30:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.233+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06414b620 con 0x7fa0640c2e70 2024-03-06T07:30:34.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.234+0000 7fa05a7fc640 1 -- 172.21.15.94:0/2568717797 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa05401a070 con 0x7fa0640c2e70 2024-03-06T07:30:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.234+0000 7fa05a7fc640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa04806b500 0x7fa04806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.234+0000 7fa05a7fc640 1 -- 172.21.15.94:0/2568717797 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa05408a160 con 0x7fa0640c2e70 2024-03-06T07:30:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.234+0000 7fa069d85640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa04806b500 0x7fa04806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.235+0000 7fa069d85640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa04806b500 0x7fa04806d9c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa060036000 tx=0x7fa06003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.238+0000 7fa05a7fc640 1 -- 172.21.15.94:0/2568717797 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa06414b620 con 0x7fa0640c2e70 2024-03-06T07:30:34.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.447+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa06404f330 con 0x7fa0640c2e70 2024-03-06T07:30:34.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.448+0000 7fa05a7fc640 1 -- 172.21.15.94:0/2568717797 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fa054058790 con 0x7fa0640c2e70 2024-03-06T07:30:34.450 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:34.450 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:34.450 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa04806b500 msgr2=0x7fa04806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa04806b500 0x7fa04806d9c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa060036000 tx=0x7fa06003d040 comp rx=0 tx=0).stop 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0640c2e70 msgr2=0x7fa0641ee5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0640c2e70 0x7fa0641ee5d0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7fa05400e870 tx=0x7fa05400ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 shutdown_connections 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa04806b500 0x7fa04806d9c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa064150920 0x7fa0641eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0640c2e70 0x7fa0641ee5d0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 --2- 172.21.15.94:0/2568717797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa06406c720 0x7fa0640c0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 >> 172.21.15.94:0/2568717797 conn(0x7fa0640c70b0 msgr2=0x7fa0640718d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 shutdown_connections 2024-03-06T07:30:34.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:34.450+0000 7fa06bfe9640 1 -- 172.21.15.94:0/2568717797 wait complete. 2024-03-06T07:30:34.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:34.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:35.867 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:35.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.977+0000 7fc47664e640 1 Processor -- start 2024-03-06T07:30:35.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.977+0000 7fc47664e640 1 -- start start 2024-03-06T07:30:35.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.977+0000 7fc47664e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc470159270 0x7fc470155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.977+0000 7fc47664e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4700c73d0 con 0x7fc4700c88c0 2024-03-06T07:30:35.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.977+0000 7fc47664e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4700be2e0 con 0x7fc47014f360 2024-03-06T07:30:35.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.977+0000 7fc47664e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc470159700 con 0x7fc470159270 2024-03-06T07:30:35.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.977+0000 7fc474beb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc470159270 0x7fc470155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.978+0000 7fc474beb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc470159270 0x7fc470155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36612/0 (socket says 172.21.15.94:36612) 2024-03-06T07:30:35.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.978+0000 7fc474beb640 1 -- 172.21.15.94:0/725318743 learned_addr learned my addr 172.21.15.94:0/725318743 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:35.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.978+0000 7fc46f7fe640 1 --1- 172.21.15.94:0/725318743 >> v1:172.21.15.94:6789/0 conn(0x7fc4700c88c0 0x7fc470158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37128/0 (socket says 172.21.15.94:37128) 2024-03-06T07:30:35.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.978+0000 7fc474beb640 1 -- 172.21.15.94:0/725318743 >> v1:172.21.15.145:6789/0 conn(0x7fc47014f360 legacy=0x7fc4700c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.978+0000 7fc474beb640 1 -- 172.21.15.94:0/725318743 >> v1:172.21.15.94:6789/0 conn(0x7fc4700c88c0 legacy=0x7fc470158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.978+0000 7fc46effd640 1 -- 172.21.15.94:0/725318743 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023590235 0 0) 0x7fc4700be2e0 con 0x7fc47014f360 2024-03-06T07:30:35.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.978+0000 7fc474beb640 1 -- 172.21.15.94:0/725318743 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700c7670 con 0x7fc470159270 2024-03-06T07:30:35.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc474beb640 1 --2- 172.21.15.94:0/725318743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc470159270 0x7fc470155c10 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fc468009e50 tx=0x7fc46802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128e44322a05a123 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc46effd640 1 -- 172.21.15.94:0/725318743 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46802c8d0 con 0x7fc470159270 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc46effd640 1 -- 172.21.15.94:0/725318743 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46802ca70 con 0x7fc470159270 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc46effd640 1 -- 172.21.15.94:0/725318743 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46802ce40 con 0x7fc470159270 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc47664e640 1 -- 172.21.15.94:0/725318743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc470159270 msgr2=0x7fc470155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc47664e640 1 --2- 172.21.15.94:0/725318743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc470159270 0x7fc470155c10 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fc468009e50 tx=0x7fc46802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc47664e640 1 -- 172.21.15.94:0/725318743 shutdown_connections 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc47664e640 1 --2- 172.21.15.94:0/725318743 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc470159270 0x7fc470155c10 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc47664e640 1 -- 172.21.15.94:0/725318743 >> 172.21.15.94:0/725318743 conn(0x7fc4700ac570 msgr2=0x7fc4700ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc47664e640 1 -- 172.21.15.94:0/725318743 shutdown_connections 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.979+0000 7fc47664e640 1 -- 172.21.15.94:0/725318743 wait complete. 2024-03-06T07:30:35.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 Processor -- start 2024-03-06T07:30:35.984 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 -- start start 2024-03-06T07:30:35.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4700c88c0 0x7fc4700add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc47014f360 0x7fc470157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc470159270 0x7fc47014acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700c7c30 con 0x7fc4700c88c0 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700c5a50 con 0x7fc470159270 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc47664e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4700c5bd0 con 0x7fc47014f360 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.981+0000 7fc46ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4700c88c0 0x7fc4700add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4700c88c0 0x7fc4700add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36618/0 (socket says 172.21.15.94:36618) 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 -- 172.21.15.94:0/3598629216 learned_addr learned my addr 172.21.15.94:0/3598629216 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc474beb640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc470159270 0x7fc47014acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46f7fe640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc47014f360 0x7fc470157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 -- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc47014f360 msgr2=0x7fc470157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc47014f360 0x7fc470157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 -- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc470159270 msgr2=0x7fc47014acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc470159270 0x7fc47014acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 -- 172.21.15.94:0/3598629216 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4701ea820 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46ffff640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4700c88c0 0x7fc4700add30 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c440 tx=0x7fc45800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46cff9640 1 -- 172.21.15.94:0/3598629216 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458017070 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46cff9640 1 -- 172.21.15.94:0/3598629216 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458005480 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc46cff9640 1 -- 172.21.15.94:0/3598629216 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458003830 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4701eaac0 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.982+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4701eb080 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.983+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47004f330 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.983+0000 7fc46cff9640 1 -- 172.21.15.94:0/3598629216 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc45801a070 con 0x7fc4700c88c0 2024-03-06T07:30:35.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.984+0000 7fc46cff9640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.984+0000 7fc46cff9640 1 -- 172.21.15.94:0/3598629216 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc4580896c0 con 0x7fc4700c88c0 2024-03-06T07:30:35.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.984+0000 7fc46f7fe640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.985+0000 7fc46f7fe640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc45006b4d0 0x7fc45006d990 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc4700c4fe0 tx=0x7fc460007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:35.988+0000 7fc46cff9640 1 -- 172.21.15.94:0/3598629216 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc458057c60 con 0x7fc4700c88c0 2024-03-06T07:30:36.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.194+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4700c8d50 con 0x7fc4700c88c0 2024-03-06T07:30:36.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.195+0000 7fc46cff9640 1 -- 172.21.15.94:0/3598629216 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fc45805bd00 con 0x7fc4700c88c0 2024-03-06T07:30:36.197 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:36.197 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:36.197 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc45006b4d0 msgr2=0x7fc45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc45006b4d0 0x7fc45006d990 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc4700c4fe0 tx=0x7fc460007e20 comp rx=0 tx=0).stop 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4700c88c0 msgr2=0x7fc4700add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4700c88c0 0x7fc4700add30 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c440 tx=0x7fc45800c900 comp rx=0 tx=0).stop 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 shutdown_connections 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fc45006b4d0 0x7fc45006d990 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc470159270 0x7fc47014acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc47014f360 0x7fc470157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 --2- 172.21.15.94:0/3598629216 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc4700c88c0 0x7fc4700add30 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 >> 172.21.15.94:0/3598629216 conn(0x7fc4700ac570 msgr2=0x7fc4700c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 shutdown_connections 2024-03-06T07:30:36.200 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:36.197+0000 7fc47664e640 1 -- 172.21.15.94:0/3598629216 wait complete. 2024-03-06T07:30:36.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:36.612 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:37.612 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:37.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.721+0000 7fcea8ef4640 1 Processor -- start 2024-03-06T07:30:37.723 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.721+0000 7fcea8ef4640 1 -- start start 2024-03-06T07:30:37.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea8ef4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea4159270 0x7fcea4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea8ef4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40c73d0 con 0x7fcea40c88c0 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea8ef4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40be2e0 con 0x7fcea414f360 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea8ef4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4159700 con 0x7fcea4159270 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fce99d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcea40c88c0 0x7fcea4158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:37144/0 (socket says 172.21.15.94:37144) 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fce99d9b640 1 -- 172.21.15.94:0/4264330025 learned_addr learned my addr 172.21.15.94:0/4264330025 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea2d9d640 1 --2- 172.21.15.94:0/4264330025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea4159270 0x7fcea4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea1d9b640 1 -- 172.21.15.94:0/4264330025 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404406460 0 0) 0x7fcea40c73d0 con 0x7fcea40c88c0 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea1d9b640 1 -- 172.21.15.94:0/4264330025 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce90003520 con 0x7fcea40c88c0 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea1d9b640 1 -- 172.21.15.94:0/4264330025 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146028725 0 0) 0x7fcea40be2e0 con 0x7fcea414f360 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea1d9b640 1 -- 172.21.15.94:0/4264330025 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea40c73d0 con 0x7fcea414f360 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea2d9d640 1 -- 172.21.15.94:0/4264330025 >> v1:172.21.15.145:6789/0 conn(0x7fcea414f360 legacy=0x7fcea40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea2d9d640 1 -- 172.21.15.94:0/4264330025 >> v1:172.21.15.94:6789/0 conn(0x7fcea40c88c0 legacy=0x7fcea4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.722+0000 7fcea2d9d640 1 -- 172.21.15.94:0/4264330025 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40c7670 con 0x7fcea4159270 2024-03-06T07:30:37.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.723+0000 7fcea2d9d640 1 --2- 172.21.15.94:0/4264330025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea4159270 0x7fcea4155c10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009dc0 tx=0x7fce9c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b29cc2ffd9255dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.723+0000 7fcea1d9b640 1 -- 172.21.15.94:0/4264330025 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c039070 con 0x7fcea4159270 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.723+0000 7fcea1d9b640 1 -- 172.21.15.94:0/4264330025 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c02cc40 con 0x7fcea4159270 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea1d9b640 1 -- 172.21.15.94:0/4264330025 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c034440 con 0x7fcea4159270 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea8ef4640 1 -- 172.21.15.94:0/4264330025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea4159270 msgr2=0x7fcea4155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/4264330025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea4159270 0x7fcea4155c10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009dc0 tx=0x7fce9c02be90 comp rx=0 tx=0).stop 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea8ef4640 1 -- 172.21.15.94:0/4264330025 shutdown_connections 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/4264330025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea4159270 0x7fcea4155c10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea8ef4640 1 -- 172.21.15.94:0/4264330025 >> 172.21.15.94:0/4264330025 conn(0x7fcea40ac570 msgr2=0x7fcea40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea8ef4640 1 -- 172.21.15.94:0/4264330025 shutdown_connections 2024-03-06T07:30:37.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.724+0000 7fcea8ef4640 1 -- 172.21.15.94:0/4264330025 wait complete. 2024-03-06T07:30:37.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 Processor -- start 2024-03-06T07:30:37.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 -- start start 2024-03-06T07:30:37.729 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40c88c0 0x7fcea40af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcea414f360 0x7fcea414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea4159270 0x7fcea414bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c7c30 con 0x7fcea414f360 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c5a50 con 0x7fcea4159270 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea8ef4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40c5bd0 con 0x7fcea40c88c0 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fce99d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcea414f360 0x7fcea414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea259c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40c88c0 0x7fcea40af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fce99d9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcea414f360 0x7fcea414b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:36620/0 (socket says 172.21.15.94:36620) 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fce99d9b640 1 -- 172.21.15.94:0/2570198296 learned_addr learned my addr 172.21.15.94:0/2570198296 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.726+0000 7fcea2d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea4159270 0x7fcea414bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea259c640 1 -- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea4159270 msgr2=0x7fcea414bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea259c640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea4159270 0x7fcea414bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea259c640 1 -- 172.21.15.94:0/2570198296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcea414f360 msgr2=0x7fcea414b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea259c640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcea414f360 0x7fcea414b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea259c640 1 -- 172.21.15.94:0/2570198296 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea41eb8a0 con 0x7fcea40c88c0 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fce99d9b640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcea414f360 0x7fcea414b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea259c640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40c88c0 0x7fcea40af4a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fce9400c3f0 tx=0x7fce9400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fce9b7fe640 1 -- 172.21.15.94:0/2570198296 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94017070 con 0x7fcea40c88c0 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fce9b7fe640 1 -- 172.21.15.94:0/2570198296 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce940052f0 con 0x7fcea40c88c0 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea40b0280 con 0x7fcea40c88c0 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fce9b7fe640 1 -- 172.21.15.94:0/2570198296 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94003830 con 0x7fcea40c88c0 2024-03-06T07:30:37.731 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.727+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea40ad320 con 0x7fcea40c88c0 2024-03-06T07:30:37.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.728+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea404f330 con 0x7fcea40c88c0 2024-03-06T07:30:37.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.729+0000 7fce9b7fe640 1 -- 172.21.15.94:0/2570198296 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce9401a070 con 0x7fcea40c88c0 2024-03-06T07:30:37.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.729+0000 7fce9b7fe640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce8406b500 0x7fce8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.729+0000 7fce9b7fe640 1 -- 172.21.15.94:0/2570198296 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fce940896e0 con 0x7fcea40c88c0 2024-03-06T07:30:37.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.729+0000 7fce99d9b640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce8406b500 0x7fce8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.732 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.730+0000 7fce99d9b640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce8406b500 0x7fce8406d9c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fce8c000f60 tx=0x7fce8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.735 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.734+0000 7fce9b7fe640 1 -- 172.21.15.94:0/2570198296 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce94057cf0 con 0x7fcea40c88c0 2024-03-06T07:30:37.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.942+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea40c8d50 con 0x7fcea40c88c0 2024-03-06T07:30:37.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.942+0000 7fce9b7fe640 1 -- 172.21.15.94:0/2570198296 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fce9405bd90 con 0x7fcea40c88c0 2024-03-06T07:30:37.944 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:37.945 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:37.945 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce8406b500 msgr2=0x7fce8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce8406b500 0x7fce8406d9c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fce8c000f60 tx=0x7fce8c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40c88c0 msgr2=0x7fcea40af4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40c88c0 0x7fcea40af4a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fce9400c3f0 tx=0x7fce9400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 shutdown_connections 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fce8406b500 0x7fce8406d9c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea4159270 0x7fcea414bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcea414f360 0x7fcea414b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 --2- 172.21.15.94:0/2570198296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40c88c0 0x7fcea40af4a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 >> 172.21.15.94:0/2570198296 conn(0x7fcea40ac570 msgr2=0x7fcea4154a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 shutdown_connections 2024-03-06T07:30:37.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:37.944+0000 7fcea8ef4640 1 -- 172.21.15.94:0/2570198296 wait complete. 2024-03-06T07:30:38.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:38.360 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:39.361 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.470+0000 7ff5da6eb640 1 Processor -- start 2024-03-06T07:30:39.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.470+0000 7ff5da6eb640 1 -- start start 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5da6eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5d414f790 0x7ff5d41529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5da6eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d40c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5da6eb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5d40c5c60 0x7ff5d4155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5da6eb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d40c2680 con 0x7ff5d414f790 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5da6eb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d40c28d0 con 0x7ff5d40c5c60 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5da6eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40c2bb0 con 0x7ff5d40c1aa0 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d3fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5d414f790 0x7ff5d41529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d3fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5d414f790 0x7ff5d41529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45560/0 (socket says 172.21.15.94:45560) 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d3fff640 1 -- 172.21.15.94:0/1121565640 learned_addr learned my addr 172.21.15.94:0/1121565640 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:39.474 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d8c88640 1 --2- 172.21.15.94:0/1121565640 >> v2:172.21.15.145:3300/0 conn(0x7ff5d40c5c60 0x7ff5d4155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d37fe640 1 --2- 172.21.15.94:0/1121565640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d40c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d3fff640 1 -- 172.21.15.94:0/1121565640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 msgr2=0x7ff5d40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d3fff640 1 --2- 172.21.15.94:0/1121565640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d40c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d3fff640 1 -- 172.21.15.94:0/1121565640 >> v2:172.21.15.145:3300/0 conn(0x7ff5d40c5c60 msgr2=0x7ff5d4155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.471+0000 7ff5d3fff640 1 --2- 172.21.15.94:0/1121565640 >> v2:172.21.15.145:3300/0 conn(0x7ff5d40c5c60 0x7ff5d4155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.472+0000 7ff5d3fff640 1 -- 172.21.15.94:0/1121565640 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d40c2e90 con 0x7ff5d414f790 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.472+0000 7ff5d3fff640 1 --2- 172.21.15.94:0/1121565640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d414f790 0x7ff5d41529d0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7ff5d406c960 tx=0x7ff5c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8015c7b0ab7d5972 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.472+0000 7ff5d2ffd640 1 -- 172.21.15.94:0/1121565640 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4035020 con 0x7ff5d414f790 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.472+0000 7ff5d2ffd640 1 -- 172.21.15.94:0/1121565640 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c4031870 con 0x7ff5d414f790 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.472+0000 7ff5d2ffd640 1 -- 172.21.15.94:0/1121565640 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4031c40 con 0x7ff5d414f790 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 -- 172.21.15.94:0/1121565640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d414f790 msgr2=0x7ff5d41529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/1121565640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d414f790 0x7ff5d41529d0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7ff5d406c960 tx=0x7ff5c402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 -- 172.21.15.94:0/1121565640 shutdown_connections 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/1121565640 >> v2:172.21.15.145:3300/0 conn(0x7ff5d40c5c60 0x7ff5d4155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/1121565640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d40c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/1121565640 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d414f790 0x7ff5d41529d0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 -- 172.21.15.94:0/1121565640 >> 172.21.15.94:0/1121565640 conn(0x7ff5d40c6710 msgr2=0x7ff5d40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 -- 172.21.15.94:0/1121565640 shutdown_connections 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.473+0000 7ff5da6eb640 1 -- 172.21.15.94:0/1121565640 wait complete. 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.474+0000 7ff5da6eb640 1 Processor -- start 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.474+0000 7ff5da6eb640 1 -- start start 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5da6eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d4072db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5da6eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d40c5c60 0x7ff5d41dde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5da6eb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5d414f790 0x7ff5d4150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5da6eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d406cc00 con 0x7ff5d40c5c60 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5da6eb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d40c0d60 con 0x7ff5d414f790 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5da6eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d406cd80 con 0x7ff5d40c1aa0 2024-03-06T07:30:39.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d40c5c60 0x7ff5d41dde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d40c5c60 0x7ff5d41dde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45574/0 (socket says 172.21.15.94:45574) 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 -- 172.21.15.94:0/2617837179 learned_addr learned my addr 172.21.15.94:0/2617837179 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d8c88640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5d414f790 0x7ff5d4150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d3fff640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d4072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 -- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 msgr2=0x7ff5d4072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d4072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 -- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5d414f790 msgr2=0x7ff5d4150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5d414f790 0x7ff5d4150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.475+0000 7ff5d37fe640 1 -- 172.21.15.94:0/2617837179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d41dc280 con 0x7ff5d40c5c60 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.476+0000 7ff5d37fe640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d40c5c60 0x7ff5d41dde00 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800e8c0 tx=0x7ff5c800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.476+0000 7ff5d0ff9640 1 -- 172.21.15.94:0/2617837179 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8017070 con 0x7ff5d40c5c60 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.476+0000 7ff5d0ff9640 1 -- 172.21.15.94:0/2617837179 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c800c940 con 0x7ff5d40c5c60 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.476+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d41e69f0 con 0x7ff5d40c5c60 2024-03-06T07:30:39.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.476+0000 7ff5d0ff9640 1 -- 172.21.15.94:0/2617837179 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c800cd10 con 0x7ff5d40c5c60 2024-03-06T07:30:39.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.476+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d41de340 con 0x7ff5d40c5c60 2024-03-06T07:30:39.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.477+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d404f330 con 0x7ff5d40c5c60 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.477+0000 7ff5d0ff9640 1 -- 172.21.15.94:0/2617837179 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5c801a070 con 0x7ff5d40c5c60 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.478+0000 7ff5d0ff9640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5c006b500 0x7ff5c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.478+0000 7ff5d0ff9640 1 -- 172.21.15.94:0/2617837179 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff5c808a6a0 con 0x7ff5d40c5c60 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.478+0000 7ff5d3fff640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5c006b500 0x7ff5c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.482 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.478+0000 7ff5d3fff640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5c006b500 0x7ff5c006d9c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4036000 tx=0x7ff5c403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.483 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.482+0000 7ff5d0ff9640 1 -- 172.21.15.94:0/2617837179 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5c8058cd0 con 0x7ff5d40c5c60 2024-03-06T07:30:39.689 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.687+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5d40c1f40 con 0x7ff5d40c5c60 2024-03-06T07:30:39.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.688+0000 7ff5d0ff9640 1 -- 172.21.15.94:0/2617837179 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7ff5c805cd70 con 0x7ff5d40c5c60 2024-03-06T07:30:39.690 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:39.690 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:39.690 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5c006b500 msgr2=0x7ff5c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5c006b500 0x7ff5c006d9c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4036000 tx=0x7ff5c403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d40c5c60 msgr2=0x7ff5d41dde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d40c5c60 0x7ff5d41dde00 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800e8c0 tx=0x7ff5c800ed80 comp rx=0 tx=0).stop 2024-03-06T07:30:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 shutdown_connections 2024-03-06T07:30:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff5c006b500 0x7ff5c006d9c0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5d414f790 0x7ff5d4150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d40c5c60 0x7ff5d41dde00 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 --2- 172.21.15.94:0/2617837179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5d40c1aa0 0x7ff5d4072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 >> 172.21.15.94:0/2617837179 conn(0x7ff5d40c6710 msgr2=0x7ff5d40c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 shutdown_connections 2024-03-06T07:30:39.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:39.690+0000 7ff5da6eb640 1 -- 172.21.15.94:0/2617837179 wait complete. 2024-03-06T07:30:40.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:40.105 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:41.106 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:41.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.216+0000 7f172c55e640 1 Processor -- start 2024-03-06T07:30:41.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.216+0000 7f172c55e640 1 -- start start 2024-03-06T07:30:41.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172c55e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172c55e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f172414f140 0x7f17240c5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172c55e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1724155bb0 con 0x7f17241550c0 2024-03-06T07:30:41.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172c55e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17240c4610 con 0x7f172414f140 2024-03-06T07:30:41.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172c55e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172414f5f0 con 0x7f17240aa180 2024-03-06T07:30:41.219 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f1729af9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172aafb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f172414f140 0x7f17240c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f1729af9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172414ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:39152/0 (socket says 172.21.15.94:39152) 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f1729af9640 1 -- 172.21.15.94:0/3617505191 learned_addr learned my addr 172.21.15.94:0/3617505191 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172a2fa640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f17241550c0 0x7f17241554a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:56862/0 (socket says 172.21.15.94:56862) 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f1729af9640 1 -- 172.21.15.94:0/3617505191 >> v2:172.21.15.145:3300/0 conn(0x7f172414f140 msgr2=0x7f17240c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f1729af9640 1 --2- 172.21.15.94:0/3617505191 >> v2:172.21.15.145:3300/0 conn(0x7f172414f140 0x7f17240c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f1729af9640 1 -- 172.21.15.94:0/3617505191 >> v1:172.21.15.145:6789/0 conn(0x7f17241550c0 legacy=0x7f17241554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f1729af9640 1 -- 172.21.15.94:0/3617505191 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17240aa660 con 0x7f17240aa180 2024-03-06T07:30:41.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.217+0000 7f172aafb640 1 --2- 172.21.15.94:0/3617505191 >> v2:172.21.15.145:3300/0 conn(0x7f172414f140 0x7f17240c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:41.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.218+0000 7f1729af9640 1 --2- 172.21.15.94:0/3617505191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172414ec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1720009870 tx=0x7f172002da70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b92fc343f8bd08cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.218+0000 7f17292f8640 1 -- 172.21.15.94:0/3617505191 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172002f830 con 0x7f17240aa180 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.218+0000 7f17292f8640 1 -- 172.21.15.94:0/3617505191 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172002f9d0 con 0x7f17240aa180 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.218+0000 7f17292f8640 1 -- 172.21.15.94:0/3617505191 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172002fda0 con 0x7f17240aa180 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 -- 172.21.15.94:0/3617505191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 msgr2=0x7f172414ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 --2- 172.21.15.94:0/3617505191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172414ec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1720009870 tx=0x7f172002da70 comp rx=0 tx=0).stop 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 -- 172.21.15.94:0/3617505191 shutdown_connections 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 --2- 172.21.15.94:0/3617505191 >> v2:172.21.15.145:3300/0 conn(0x7f172414f140 0x7f17240c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 --2- 172.21.15.94:0/3617505191 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172414ec00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 -- 172.21.15.94:0/3617505191 >> 172.21.15.94:0/3617505191 conn(0x7f17240c6710 msgr2=0x7f17240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 -- 172.21.15.94:0/3617505191 shutdown_connections 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.219+0000 7f172c55e640 1 -- 172.21.15.94:0/3617505191 wait complete. 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.220+0000 7f172c55e640 1 Processor -- start 2024-03-06T07:30:41.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.220+0000 7f172c55e640 1 -- start start 2024-03-06T07:30:41.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172c55e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172c55e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f172414f140 0x7f17241eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172c55e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17241550c0 0x7f17240bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172c55e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17240c1310 con 0x7f17241550c0 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172c55e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17240c4860 con 0x7f172414f140 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172c55e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17240c1490 con 0x7f17240aa180 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172406dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.94:39164/0 (socket says 172.21.15.94:39164) 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 -- 172.21.15.94:0/640111316 learned_addr learned my addr 172.21.15.94:0/640111316 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f1729af9640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f172414f140 0x7f17241eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172aafb640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17241550c0 0x7f17240bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 -- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f172414f140 msgr2=0x7f17241eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f172414f140 0x7f17241eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 -- 172.21.15.94:0/640111316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17241550c0 msgr2=0x7f17240bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.225 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17241550c0 0x7f17240bfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 -- 172.21.15.94:0/640111316 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17240c0730 con 0x7f17240aa180 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f1729af9640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f172414f140 0x7f17241eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172aafb640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17241550c0 0x7f17240bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.221+0000 7f172a2fa640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172406dac0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f171400c430 tx=0x7f171400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.222+0000 7f171affd640 1 -- 172.21.15.94:0/640111316 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17140074e0 con 0x7f17240aa180 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.222+0000 7f171affd640 1 -- 172.21.15.94:0/640111316 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1714007680 con 0x7f17240aa180 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.222+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17241ef150 con 0x7f17240aa180 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.222+0000 7f171affd640 1 -- 172.21.15.94:0/640111316 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714004a00 con 0x7f17240aa180 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.222+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172406e4b0 con 0x7f17240aa180 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.223+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172414e630 con 0x7f17240aa180 2024-03-06T07:30:41.226 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.223+0000 7f171affd640 1 -- 172.21.15.94:0/640111316 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f171401a020 con 0x7f17240aa180 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.223+0000 7f171affd640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f170806b500 0x7f170806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.224+0000 7f171affd640 1 -- 172.21.15.94:0/640111316 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1714089aa0 con 0x7f17240aa180 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.224+0000 7f1729af9640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f170806b500 0x7f170806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:41.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.224+0000 7f1729af9640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f170806b500 0x7f170806d9c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1720038000 tx=0x7f172003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:41.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.228+0000 7f171affd640 1 -- 172.21.15.94:0/640111316 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f172414e630 con 0x7f17240aa180 2024-03-06T07:30:41.437 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.436+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f172404f330 con 0x7f17240aa180 2024-03-06T07:30:41.438 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.436+0000 7f171affd640 1 -- 172.21.15.94:0/640111316 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f1714058720 con 0x7f17240aa180 2024-03-06T07:30:41.438 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:41.438 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:41.438 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:41.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f170806b500 msgr2=0x7f170806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f170806b500 0x7f170806d9c0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1720038000 tx=0x7f172003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:41.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 msgr2=0x7f172406dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:41.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172406dac0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f171400c430 tx=0x7f171400c830 comp rx=0 tx=0).stop 2024-03-06T07:30:41.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 shutdown_connections 2024-03-06T07:30:41.440 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f170806b500 0x7f170806d9c0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17241550c0 0x7f17240bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f172414f140 0x7f17241eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 --2- 172.21.15.94:0/640111316 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f17240aa180 0x7f172406dac0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:41.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 >> 172.21.15.94:0/640111316 conn(0x7f17240c6710 msgr2=0x7f17240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:41.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 shutdown_connections 2024-03-06T07:30:41.441 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:41.438+0000 7f172c55e640 1 -- 172.21.15.94:0/640111316 wait complete. 2024-03-06T07:30:41.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:41.853 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:42.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.963+0000 7f9569fac640 1 Processor -- start 2024-03-06T07:30:42.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.963+0000 7f9569fac640 1 -- start start 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.963+0000 7f9569fac640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9564152960 0x7f956406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.963+0000 7f9569fac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f95641592d0 0x7f95641552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.963+0000 7f9569fac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9564152d40 con 0x7f95640ac260 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.963+0000 7f9569fac640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9564066e30 con 0x7f9564152960 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.963+0000 7f9569fac640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95640c1dc0 con 0x7f95641592d0 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f95637fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9564152960 0x7f956406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f95637fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9564152960 0x7f956406dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45596/0 (socket says 172.21.15.94:45596) 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f95637fe640 1 -- 172.21.15.94:0/1983610354 learned_addr learned my addr 172.21.15.94:0/1983610354 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f9562ffd640 1 --2- 172.21.15.94:0/1983610354 >> v2:172.21.15.145:3300/0 conn(0x7f95641592d0 0x7f95641552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f95637fe640 1 -- 172.21.15.94:0/1983610354 >> v2:172.21.15.145:3300/0 conn(0x7f95641592d0 msgr2=0x7f95641552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f95637fe640 1 --2- 172.21.15.94:0/1983610354 >> v2:172.21.15.145:3300/0 conn(0x7f95641592d0 0x7f95641552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f95637fe640 1 -- 172.21.15.94:0/1983610354 >> v1:172.21.15.145:6789/0 conn(0x7f95640ac260 legacy=0x7f9564155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.966 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.964+0000 7f95637fe640 1 -- 172.21.15.94:0/1983610354 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95640c1b50 con 0x7f9564152960 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f95637fe640 1 --2- 172.21.15.94:0/1983610354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9564152960 0x7f956406dc90 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f956406ee90 tx=0x7f955402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c4075ef0706779 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f95627fc640 1 -- 172.21.15.94:0/1983610354 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554034070 con 0x7f9564152960 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f95627fc640 1 -- 172.21.15.94:0/1983610354 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9554030860 con 0x7f9564152960 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f95627fc640 1 -- 172.21.15.94:0/1983610354 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554030c30 con 0x7f9564152960 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f9569fac640 1 -- 172.21.15.94:0/1983610354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9564152960 msgr2=0x7f956406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f9569fac640 1 --2- 172.21.15.94:0/1983610354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9564152960 0x7f956406dc90 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f956406ee90 tx=0x7f955402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f9569fac640 1 -- 172.21.15.94:0/1983610354 shutdown_connections 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f9569fac640 1 --2- 172.21.15.94:0/1983610354 >> v2:172.21.15.145:3300/0 conn(0x7f95641592d0 0x7f95641552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f9569fac640 1 --2- 172.21.15.94:0/1983610354 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9564152960 0x7f956406dc90 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.965+0000 7f9569fac640 1 -- 172.21.15.94:0/1983610354 >> 172.21.15.94:0/1983610354 conn(0x7f95640c70b0 msgr2=0x7f95640c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.966+0000 7f9569fac640 1 -- 172.21.15.94:0/1983610354 shutdown_connections 2024-03-06T07:30:42.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.966+0000 7f9569fac640 1 -- 172.21.15.94:0/1983610354 wait complete. 2024-03-06T07:30:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.967+0000 7f9569fac640 1 Processor -- start 2024-03-06T07:30:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.967+0000 7f9569fac640 1 -- start start 2024-03-06T07:30:42.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.967+0000 7f9569fac640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640ac260 0x7f956414d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.967+0000 7f9569fac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564152960 0x7f956414c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9569fac640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414ef50 0x7f9564157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9569fac640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640c20a0 con 0x7f956414ef50 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9569fac640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564159760 con 0x7f95640ac260 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9569fac640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95640c2220 con 0x7f9564152960 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414ef50 0x7f9564157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414ef50 0x7f9564157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45600/0 (socket says 172.21.15.94:45600) 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 -- 172.21.15.94:0/2812117147 learned_addr learned my addr 172.21.15.94:0/2812117147 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9562ffd640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564152960 0x7f956414c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f95637fe640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640ac260 0x7f956414d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 -- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564152960 msgr2=0x7f956414c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564152960 0x7f956414c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 -- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640ac260 msgr2=0x7f956414d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640ac260 0x7f956414d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 -- 172.21.15.94:0/2812117147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95641568d0 con 0x7f956414ef50 2024-03-06T07:30:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9563fff640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414ef50 0x7f9564157fc0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c680 tx=0x7f955c00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f954bfff640 1 -- 172.21.15.94:0/2812117147 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c003b70 con 0x7f956414ef50 2024-03-06T07:30:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f954bfff640 1 -- 172.21.15.94:0/2812117147 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c003d10 con 0x7f956414ef50 2024-03-06T07:30:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9564156bb0 con 0x7f956414ef50 2024-03-06T07:30:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f954bfff640 1 -- 172.21.15.94:0/2812117147 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c005310 con 0x7f956414ef50 2024-03-06T07:30:42.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.968+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95641577c0 con 0x7f956414ef50 2024-03-06T07:30:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.969+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956414deb0 con 0x7f956414ef50 2024-03-06T07:30:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.970+0000 7f954bfff640 1 -- 172.21.15.94:0/2812117147 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f955c0054b0 con 0x7f956414ef50 2024-03-06T07:30:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.970+0000 7f954bfff640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.970+0000 7f954bfff640 1 -- 172.21.15.94:0/2812117147 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f955c08a160 con 0x7f956414ef50 2024-03-06T07:30:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.970+0000 7f95637fe640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.971+0000 7f95637fe640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9554036000 tx=0x7f9554008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:42.974+0000 7f954bfff640 1 -- 172.21.15.94:0/2812117147 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f955c058700 con 0x7f956414ef50 2024-03-06T07:30:43.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.181+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f956404f330 con 0x7f956414ef50 2024-03-06T07:30:43.183 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.181+0000 7f954bfff640 1 -- 172.21.15.94:0/2812117147 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f955c05c7a0 con 0x7f956414ef50 2024-03-06T07:30:43.183 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:43.183 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:43.183 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:43.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 msgr2=0x7f955006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f9554036000 tx=0x7f9554008030 comp rx=0 tx=0).stop 2024-03-06T07:30:43.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414ef50 msgr2=0x7f9564157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414ef50 0x7f9564157fc0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c680 tx=0x7f955c00cb40 comp rx=0 tx=0).stop 2024-03-06T07:30:43.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 shutdown_connections 2024-03-06T07:30:43.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f955006b4d0 0x7f955006d990 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.185 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f956414ef50 0x7f9564157fc0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9564152960 0x7f956414c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 --2- 172.21.15.94:0/2812117147 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95640ac260 0x7f956414d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 >> 172.21.15.94:0/2812117147 conn(0x7f95640c70b0 msgr2=0x7f95640c55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 shutdown_connections 2024-03-06T07:30:43.186 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:43.183+0000 7f9569fac640 1 -- 172.21.15.94:0/2812117147 wait complete. 2024-03-06T07:30:43.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:43.598 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:44.598 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:44.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.709+0000 7fbe2178a640 1 Processor -- start 2024-03-06T07:30:44.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.709+0000 7fbe2178a640 1 -- start start 2024-03-06T07:30:44.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe2178a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbe1c158db0 0x7fbe1c0c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe2178a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbe1c0c88b0 0x7fbe1c15fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe2178a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1c155110 con 0x7fbe1c0c5330 2024-03-06T07:30:44.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe2178a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1c1559e0 con 0x7fbe1c158db0 2024-03-06T07:30:44.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe2178a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1c0c8f40 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1a7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbe1c158db0 0x7fbe1c0c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1affd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbe1c0c5330 0x7fbe1c1586a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58454/0 (socket says 172.21.15.94:58454) 2024-03-06T07:30:44.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1affd640 1 -- 172.21.15.94:0/3248921313 learned_addr learned my addr 172.21.15.94:0/3248921313 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:44.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1b7fe640 1 --2- 172.21.15.94:0/3248921313 >> v2:172.21.15.145:3300/0 conn(0x7fbe1c0c88b0 0x7fbe1c15fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1a7fc640 1 -- 172.21.15.94:0/3248921313 >> v2:172.21.15.145:3300/0 conn(0x7fbe1c0c88b0 msgr2=0x7fbe1c15fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:44.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1a7fc640 1 --2- 172.21.15.94:0/3248921313 >> v2:172.21.15.145:3300/0 conn(0x7fbe1c0c88b0 0x7fbe1c15fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1a7fc640 1 -- 172.21.15.94:0/3248921313 >> v1:172.21.15.94:6789/0 conn(0x7fbe1c0c5330 legacy=0x7fbe1c1586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.710+0000 7fbe1a7fc640 1 -- 172.21.15.94:0/3248921313 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c155390 con 0x7fbe1c158db0 2024-03-06T07:30:44.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.711+0000 7fbe1a7fc640 1 --2- 172.21.15.94:0/3248921313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c158db0 0x7fbe1c0c5710 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0ca870 tx=0x7fbe1002b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52029e8d0c11f55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.711+0000 7fbe19ffb640 1 -- 172.21.15.94:0/3248921313 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10037070 con 0x7fbe1c158db0 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.711+0000 7fbe19ffb640 1 -- 172.21.15.94:0/3248921313 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe10030710 con 0x7fbe1c158db0 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.711+0000 7fbe19ffb640 1 -- 172.21.15.94:0/3248921313 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10030ae0 con 0x7fbe1c158db0 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.711+0000 7fbe2178a640 1 -- 172.21.15.94:0/3248921313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c158db0 msgr2=0x7fbe1c0c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.711+0000 7fbe2178a640 1 --2- 172.21.15.94:0/3248921313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c158db0 0x7fbe1c0c5710 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0ca870 tx=0x7fbe1002b680 comp rx=0 tx=0).stop 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.711+0000 7fbe2178a640 1 -- 172.21.15.94:0/3248921313 shutdown_connections 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.712+0000 7fbe2178a640 1 --2- 172.21.15.94:0/3248921313 >> v2:172.21.15.145:3300/0 conn(0x7fbe1c0c88b0 0x7fbe1c15fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.712+0000 7fbe2178a640 1 --2- 172.21.15.94:0/3248921313 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c158db0 0x7fbe1c0c5710 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.712+0000 7fbe2178a640 1 -- 172.21.15.94:0/3248921313 >> 172.21.15.94:0/3248921313 conn(0x7fbe1c0ac560 msgr2=0x7fbe1c0ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.712+0000 7fbe2178a640 1 -- 172.21.15.94:0/3248921313 shutdown_connections 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.712+0000 7fbe2178a640 1 -- 172.21.15.94:0/3248921313 wait complete. 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.713+0000 7fbe2178a640 1 Processor -- start 2024-03-06T07:30:44.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.713+0000 7fbe2178a640 1 -- start start 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe1c0c5330 0x7fbe1c0aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c0c88b0 0x7fbe1c157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe1c151e40 0x7fbe1c1ea940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c1529d0 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c06d090 con 0x7fbe1c0c5330 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c152b50 con 0x7fbe1c151e40 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c0c88b0 0x7fbe1c157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c0c88b0 0x7fbe1c157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45606/0 (socket says 172.21.15.94:45606) 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 -- 172.21.15.94:0/920006762 learned_addr learned my addr 172.21.15.94:0/920006762 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1affd640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe1c0c5330 0x7fbe1c0aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1b7fe640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe1c151e40 0x7fbe1c1ea940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 -- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe1c151e40 msgr2=0x7fbe1c1ea940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe1c151e40 0x7fbe1c1ea940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 -- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe1c0c5330 msgr2=0x7fbe1c0aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe1c0c5330 0x7fbe1c0aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 -- 172.21.15.94:0/920006762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c0abf80 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe1a7fc640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c0c88b0 0x7fbe1c157a60 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009890 tx=0x7fbe100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe037fe640 1 -- 172.21.15.94:0/920006762 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10042070 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe037fe640 1 -- 172.21.15.94:0/920006762 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1003e890 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe037fe640 1 -- 172.21.15.94:0/920006762 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe10037040 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c0ac250 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.714+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c1ee190 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.715+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1c04f330 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.716+0000 7fbe037fe640 1 -- 172.21.15.94:0/920006762 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe10045070 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.716+0000 7fbe037fe640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe0806b500 0x7fbe0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.716+0000 7fbe037fe640 1 -- 172.21.15.94:0/920006762 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbe100ac1d0 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.716+0000 7fbe1affd640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe0806b500 0x7fbe0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.717+0000 7fbe1affd640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe0806b500 0x7fbe0806d9c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c000f60 tx=0x7fbe0c007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.720+0000 7fbe037fe640 1 -- 172.21.15.94:0/920006762 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe1007a920 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.929 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.927+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe1c0c97b0 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.928+0000 7fbe037fe640 1 -- 172.21.15.94:0/920006762 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fbe1007e9c0 con 0x7fbe1c0c88b0 2024-03-06T07:30:44.930 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:44.930 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:44.930 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe0806b500 msgr2=0x7fbe0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.932 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe0806b500 0x7fbe0806d9c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c000f60 tx=0x7fbe0c007e20 comp rx=0 tx=0).stop 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c0c88b0 msgr2=0x7fbe1c157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c0c88b0 0x7fbe1c157a60 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009890 tx=0x7fbe100046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 shutdown_connections 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbe0806b500 0x7fbe0806d9c0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe1c151e40 0x7fbe1c1ea940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbe1c0c88b0 0x7fbe1c157a60 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 --2- 172.21.15.94:0/920006762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe1c0c5330 0x7fbe1c0aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 >> 172.21.15.94:0/920006762 conn(0x7fbe1c0ac560 msgr2=0x7fbe1c0ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 shutdown_connections 2024-03-06T07:30:44.933 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:44.930+0000 7fbe2178a640 1 -- 172.21.15.94:0/920006762 wait complete. 2024-03-06T07:30:45.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:45.346 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:46.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:46.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.459+0000 7f7ddbf77640 1 Processor -- start 2024-03-06T07:30:46.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.459+0000 7f7ddbf77640 1 -- start start 2024-03-06T07:30:46.462 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7ddbf77640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd4155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7ddbf77640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7dd40c88c0 0x7f7dd40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7ddbf77640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd40c73d0 con 0x7f7dd40c5880 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7ddbf77640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd40c8dd0 con 0x7f7dd40c88c0 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7ddbf77640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40be2e0 con 0x7f7dd414f360 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7dda514640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7dd40c5880 0x7f7dd4158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58468/0 (socket says 172.21.15.94:58468) 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7dda514640 1 -- 172.21.15.94:0/4290690626 learned_addr learned my addr 172.21.15.94:0/4290690626 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7dd9512640 1 --2- 172.21.15.94:0/4290690626 >> v2:172.21.15.145:3300/0 conn(0x7f7dd40c88c0 0x7f7dd40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7dd9d13640 1 --2- 172.21.15.94:0/4290690626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd4155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7dd8d11640 1 -- 172.21.15.94:0/4290690626 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103752452 0 0) 0x7f7dd40c73d0 con 0x7f7dd40c5880 2024-03-06T07:30:46.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.460+0000 7f7dd8d11640 1 -- 172.21.15.94:0/4290690626 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc0003520 con 0x7f7dd40c5880 2024-03-06T07:30:46.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd9d13640 1 -- 172.21.15.94:0/4290690626 >> v2:172.21.15.145:3300/0 conn(0x7f7dd40c88c0 msgr2=0x7f7dd40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:46.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd9d13640 1 --2- 172.21.15.94:0/4290690626 >> v2:172.21.15.145:3300/0 conn(0x7f7dd40c88c0 0x7f7dd40c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd9d13640 1 -- 172.21.15.94:0/4290690626 >> v1:172.21.15.94:6789/0 conn(0x7f7dd40c5880 legacy=0x7f7dd4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd9d13640 1 -- 172.21.15.94:0/4290690626 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd40c7670 con 0x7f7dd414f360 2024-03-06T07:30:46.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd9512640 1 --2- 172.21.15.94:0/4290690626 >> v2:172.21.15.145:3300/0 conn(0x7f7dd40c88c0 0x7f7dd40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:46.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd9d13640 1 --2- 172.21.15.94:0/4290690626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd4155c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00097a0 tx=0x7f7dd002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53452a0af6832431 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd8d11640 1 -- 172.21.15.94:0/4290690626 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00315a0 con 0x7f7dd414f360 2024-03-06T07:30:46.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.461+0000 7f7dd8d11640 1 -- 172.21.15.94:0/4290690626 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd0037070 con 0x7f7dd414f360 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7dd8d11640 1 -- 172.21.15.94:0/4290690626 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0031940 con 0x7f7dd414f360 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 -- 172.21.15.94:0/4290690626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 msgr2=0x7f7dd4155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/4290690626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd4155c10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00097a0 tx=0x7f7dd002b840 comp rx=0 tx=0).stop 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 -- 172.21.15.94:0/4290690626 shutdown_connections 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/4290690626 >> v2:172.21.15.145:3300/0 conn(0x7f7dd40c88c0 0x7f7dd40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/4290690626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd4155c10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 -- 172.21.15.94:0/4290690626 >> 172.21.15.94:0/4290690626 conn(0x7f7dd40ac570 msgr2=0x7f7dd40ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 -- 172.21.15.94:0/4290690626 shutdown_connections 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.462+0000 7f7ddbf77640 1 -- 172.21.15.94:0/4290690626 wait complete. 2024-03-06T07:30:46.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.463+0000 7f7ddbf77640 1 Processor -- start 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.463+0000 7f7ddbf77640 1 -- start start 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7ddbf77640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40c5880 0x7f7dd40c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7ddbf77640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dd40c88c0 0x7f7dd41ebca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7ddbf77640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd40afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7ddbf77640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4159270 con 0x7f7dd40c88c0 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7ddbf77640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40c7c30 con 0x7f7dd40c5880 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7ddbf77640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd41593f0 con 0x7f7dd414f360 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dd40c88c0 0x7f7dd41ebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.469 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dd40c88c0 0x7f7dd41ebca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45608/0 (socket says 172.21.15.94:45608) 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 -- 172.21.15.94:0/3406510949 learned_addr learned my addr 172.21.15.94:0/3406510949 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dda514640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd40afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9d13640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40c5880 0x7f7dd40c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 -- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 msgr2=0x7f7dd40afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd40afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 -- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40c5880 msgr2=0x7f7dd40c11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40c5880 0x7f7dd40c11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.464+0000 7f7dd9512640 1 -- 172.21.15.94:0/3406510949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd40ad530 con 0x7f7dd40c88c0 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.465+0000 7f7dd9512640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dd40c88c0 0x7f7dd41ebca0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e9b0 tx=0x7f7dc400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.465+0000 7f7dca7fc640 1 -- 172.21.15.94:0/3406510949 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4017070 con 0x7f7dd40c88c0 2024-03-06T07:30:46.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.465+0000 7f7dca7fc640 1 -- 172.21.15.94:0/3406510949 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc400ca00 con 0x7f7dd40c88c0 2024-03-06T07:30:46.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.465+0000 7f7dca7fc640 1 -- 172.21.15.94:0/3406510949 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400cdf0 con 0x7f7dd40c88c0 2024-03-06T07:30:46.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.465+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd40ad810 con 0x7f7dd40c88c0 2024-03-06T07:30:46.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.465+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd40addd0 con 0x7f7dd40c88c0 2024-03-06T07:30:46.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.466+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd404f330 con 0x7f7dd40c88c0 2024-03-06T07:30:46.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.466+0000 7f7dca7fc640 1 -- 172.21.15.94:0/3406510949 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dc401a070 con 0x7f7dd40c88c0 2024-03-06T07:30:46.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.467+0000 7f7dca7fc640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.467+0000 7f7dca7fc640 1 -- 172.21.15.94:0/3406510949 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7dc408a300 con 0x7f7dd40c88c0 2024-03-06T07:30:46.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.467+0000 7f7dd9d13640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:46.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.467+0000 7f7dd9d13640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7db806b4d0 0x7f7db806d990 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0036000 tx=0x7f7dd003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:46.473 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.471+0000 7f7dca7fc640 1 -- 172.21.15.94:0/3406510949 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dc4058930 con 0x7f7dd40c88c0 2024-03-06T07:30:46.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.679+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dd406cda0 con 0x7f7dd40c88c0 2024-03-06T07:30:46.681 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.679+0000 7f7dca7fc640 1 -- 172.21.15.94:0/3406510949 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f7dc405c9d0 con 0x7f7dd40c88c0 2024-03-06T07:30:46.681 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:46.681 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:46.682 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:46.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7db806b4d0 msgr2=0x7f7db806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7db806b4d0 0x7f7db806d990 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0036000 tx=0x7f7dd003d040 comp rx=0 tx=0).stop 2024-03-06T07:30:46.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dd40c88c0 msgr2=0x7f7dd41ebca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:46.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dd40c88c0 0x7f7dd41ebca0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e9b0 tx=0x7f7dc400ee70 comp rx=0 tx=0).stop 2024-03-06T07:30:46.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 shutdown_connections 2024-03-06T07:30:46.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd414f360 0x7f7dd40afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7dd40c88c0 0x7f7dd41ebca0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 --2- 172.21.15.94:0/3406510949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40c5880 0x7f7dd40c11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:46.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 >> 172.21.15.94:0/3406510949 conn(0x7f7dd40ac570 msgr2=0x7f7dd414dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:46.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 shutdown_connections 2024-03-06T07:30:46.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:46.681+0000 7f7ddbf77640 1 -- 172.21.15.94:0/3406510949 wait complete. 2024-03-06T07:30:46.811 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:46.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:47.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:47.097 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:48.097 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.205+0000 7f34655e9640 1 Processor -- start 2024-03-06T07:30:48.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.205+0000 7f34655e9640 1 -- start start 2024-03-06T07:30:48.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f34655e9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34600c7180 0x7f3460159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f34655e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c88b0 0x7f34600c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f34655e9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600c7c60 con 0x7f34601527b0 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f34655e9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3460159640 con 0x7f34600c7180 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f34655e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600c8060 con 0x7f34600c88b0 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f345e7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34600c7180 0x7f3460159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f345e7fc640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f34600c7180 0x7f3460159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45620/0 (socket says 172.21.15.94:45620) 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f345e7fc640 1 -- 172.21.15.94:0/3995292098 learned_addr learned my addr 172.21.15.94:0/3995292098 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.206+0000 7f345f7fe640 1 --2- 172.21.15.94:0/3995292098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c88b0 0x7f34600c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.207+0000 7f345e7fc640 1 -- 172.21.15.94:0/3995292098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c88b0 msgr2=0x7f34600c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.207+0000 7f345e7fc640 1 --2- 172.21.15.94:0/3995292098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c88b0 0x7f34600c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.207+0000 7f345e7fc640 1 -- 172.21.15.94:0/3995292098 >> v1:172.21.15.145:6789/0 conn(0x7f34601527b0 legacy=0x7f3460155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.207+0000 7f345e7fc640 1 -- 172.21.15.94:0/3995292098 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600c76a0 con 0x7f34600c7180 2024-03-06T07:30:48.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.207+0000 7f345dffb640 1 -- 172.21.15.94:0/3995292098 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521940425 0 0) 0x7f3450001490 con 0x7f34601527b0 2024-03-06T07:30:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.207+0000 7f345e7fc640 1 --2- 172.21.15.94:0/3995292098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c7180 0x7f3460159100 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f3454009870 tx=0x7f345402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8b320270d58331 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f345dffb640 1 -- 172.21.15.94:0/3995292098 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454035020 con 0x7f34600c7180 2024-03-06T07:30:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f345dffb640 1 -- 172.21.15.94:0/3995292098 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454031640 con 0x7f34600c7180 2024-03-06T07:30:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f345dffb640 1 -- 172.21.15.94:0/3995292098 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345402d040 con 0x7f34600c7180 2024-03-06T07:30:48.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 -- 172.21.15.94:0/3995292098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c7180 msgr2=0x7f3460159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 --2- 172.21.15.94:0/3995292098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c7180 0x7f3460159100 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f3454009870 tx=0x7f345402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 -- 172.21.15.94:0/3995292098 shutdown_connections 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 --2- 172.21.15.94:0/3995292098 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c88b0 0x7f34600c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 --2- 172.21.15.94:0/3995292098 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c7180 0x7f3460159100 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 -- 172.21.15.94:0/3995292098 >> 172.21.15.94:0/3995292098 conn(0x7f34600ac560 msgr2=0x7f34600ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 -- 172.21.15.94:0/3995292098 shutdown_connections 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.208+0000 7f34655e9640 1 -- 172.21.15.94:0/3995292098 wait complete. 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 Processor -- start 2024-03-06T07:30:48.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 -- start start 2024-03-06T07:30:48.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c7180 0x7f346014b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c88b0 0x7f346014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34601527b0 0x7f34601ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460155600 con 0x7f34600c88b0 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460155320 con 0x7f34601527b0 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f34655e9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600c8d60 con 0x7f34600c7180 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f345e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c88b0 0x7f346014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f345e7fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c88b0 0x7f346014be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:45624/0 (socket says 172.21.15.94:45624) 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f345e7fc640 1 -- 172.21.15.94:0/1944109223 learned_addr learned my addr 172.21.15.94:0/1944109223 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f345effd640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c7180 0x7f346014b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.210+0000 7f345f7fe640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34601527b0 0x7f34601ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f345e7fc640 1 -- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c7180 msgr2=0x7f346014b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f345e7fc640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c7180 0x7f346014b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f345e7fc640 1 -- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34601527b0 msgr2=0x7f34601ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.216 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f345e7fc640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34601527b0 0x7f34601ee1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f345e7fc640 1 -- 172.21.15.94:0/1944109223 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600b0110 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f345e7fc640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c88b0 0x7f346014be60 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f3454000c00 tx=0x7f34540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f34477fe640 1 -- 172.21.15.94:0/1944109223 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454040070 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f34477fe640 1 -- 172.21.15.94:0/1944109223 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34540399f0 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34600c22f0 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f34477fe640 1 -- 172.21.15.94:0/1944109223 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454035040 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.211+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34600c28b0 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.212+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34600c2ab0 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.212+0000 7f34477fe640 1 -- 172.21.15.94:0/1944109223 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f345404b060 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.213+0000 7f34477fe640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f344c06b4d0 0x7f344c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.213+0000 7f34477fe640 1 -- 172.21.15.94:0/1944109223 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f34540abae0 con 0x7f34600c88b0 2024-03-06T07:30:48.217 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.213+0000 7f345effd640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f344c06b4d0 0x7f344c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.218 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.213+0000 7f345effd640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f344c06b4d0 0x7f344c06d990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f34500012e0 tx=0x7f3450009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.220 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.218+0000 7f34477fe640 1 -- 172.21.15.94:0/1944109223 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34600c2ab0 con 0x7f34600c88b0 2024-03-06T07:30:48.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.425+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f346004f330 con 0x7f34600c88b0 2024-03-06T07:30:48.428 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.425+0000 7f34477fe640 1 -- 172.21.15.94:0/1944109223 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f345403ce20 con 0x7f34600c88b0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f344c06b4d0 msgr2=0x7f344c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f344c06b4d0 0x7f344c06d990 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f34500012e0 tx=0x7f3450009040 comp rx=0 tx=0).stop 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c88b0 msgr2=0x7f346014be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c88b0 0x7f346014be60 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f3454000c00 tx=0x7f34540046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 shutdown_connections 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f344c06b4d0 0x7f344c06d990 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34601527b0 0x7f34601ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34600c88b0 0x7f346014be60 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 --2- 172.21.15.94:0/1944109223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34600c7180 0x7f346014b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.427+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 >> 172.21.15.94:0/1944109223 conn(0x7f34600ac560 msgr2=0x7f3460150ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.428+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 shutdown_connections 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:48.428+0000 7f34655e9640 1 -- 172.21.15.94:0/1944109223 wait complete. 2024-03-06T07:30:48.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:48.843 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:49.844 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.954+0000 7f8e620cf640 1 Processor -- start 2024-03-06T07:30:49.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.954+0000 7f8e620cf640 1 -- start start 2024-03-06T07:30:49.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e620cf640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8e5c0c1aa0 0x7f8e5c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e620cf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0c28d0 con 0x7f8e5c14f790 2024-03-06T07:30:49.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e620cf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0c2bb0 con 0x7f8e5c155a40 2024-03-06T07:30:49.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e620cf640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c0c2680 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e5affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8e5c0c1aa0 0x7f8e5c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e5b7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8e5c14f790 0x7f8e5c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40082/0 (socket says 172.21.15.94:40082) 2024-03-06T07:30:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e5b7fe640 1 -- 172.21.15.94:0/2315180918 learned_addr learned my addr 172.21.15.94:0/2315180918 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e5affd640 1 -- 172.21.15.94:0/2315180918 >> v1:172.21.15.145:6789/0 conn(0x7f8e5c155a40 legacy=0x7f8e5c155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e5affd640 1 -- 172.21.15.94:0/2315180918 >> v1:172.21.15.94:6789/0 conn(0x7f8e5c14f790 legacy=0x7f8e5c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e5affd640 1 -- 172.21.15.94:0/2315180918 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0c2e90 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.958 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.955+0000 7f8e5a7fc640 1 -- 172.21.15.94:0/2315180918 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228080352 0 0) 0x7f8e5c0c28d0 con 0x7f8e5c14f790 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.956+0000 7f8e5affd640 1 --2- 172.21.15.94:0/2315180918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c1529d0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0c2800 tx=0x7f8e5002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa33cc9c34afb4de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.956+0000 7f8e5a7fc640 1 -- 172.21.15.94:0/2315180918 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50034070 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.956+0000 7f8e5a7fc640 1 -- 172.21.15.94:0/2315180918 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e50030610 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.956+0000 7f8e5a7fc640 1 -- 172.21.15.94:0/2315180918 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500309e0 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.957+0000 7f8e620cf640 1 -- 172.21.15.94:0/2315180918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 msgr2=0x7f8e5c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.957+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2315180918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c1529d0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0c2800 tx=0x7f8e5002b960 comp rx=0 tx=0).stop 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.957+0000 7f8e620cf640 1 -- 172.21.15.94:0/2315180918 shutdown_connections 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.957+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2315180918 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c1529d0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.957+0000 7f8e620cf640 1 -- 172.21.15.94:0/2315180918 >> 172.21.15.94:0/2315180918 conn(0x7f8e5c0c6710 msgr2=0x7f8e5c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.957+0000 7f8e620cf640 1 -- 172.21.15.94:0/2315180918 shutdown_connections 2024-03-06T07:30:49.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.957+0000 7f8e620cf640 1 -- 172.21.15.94:0/2315180918 wait complete. 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.958+0000 7f8e620cf640 1 Processor -- start 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.958+0000 7f8e620cf640 1 -- start start 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e620cf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c0a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e620cf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c14f790 0x7f8e5c149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e620cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c155a40 0x7f8e5c153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.962 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e620cf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0c5e30 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e620cf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0c0d60 con 0x7f8e5c14f790 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e620cf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0c5fb0 con 0x7f8e5c155a40 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c0a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c0a9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33814/0 (socket says 172.21.15.94:33814) 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 -- 172.21.15.94:0/2656267402 learned_addr learned my addr 172.21.15.94:0/2656267402 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5affd640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c14f790 0x7f8e5c149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5bfff640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c155a40 0x7f8e5c153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 -- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c155a40 msgr2=0x7f8e5c153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c155a40 0x7f8e5c153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 -- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c14f790 msgr2=0x7f8e5c149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c14f790 0x7f8e5c149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 -- 172.21.15.94:0/2656267402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c150f80 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.959+0000 7f8e5b7fe640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c0a9f80 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00c4d0 tx=0x7f8e4c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.960+0000 7f8e43fff640 1 -- 172.21.15.94:0/2656267402 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c018070 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.960+0000 7f8e43fff640 1 -- 172.21.15.94:0/2656267402 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c0074e0 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.960+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c14fee0 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.960+0000 7f8e43fff640 1 -- 172.21.15.94:0/2656267402 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c004a20 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.960+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c150780 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.961+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c1502d0 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.961+0000 7f8e43fff640 1 -- 172.21.15.94:0/2656267402 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e4c01b070 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.961+0000 7f8e43fff640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e4806b4d0 0x7f8e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.962+0000 7f8e43fff640 1 -- 172.21.15.94:0/2656267402 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8e4c0895d0 con 0x7f8e5c0c1aa0 2024-03-06T07:30:49.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.962+0000 7f8e5affd640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e4806b4d0 0x7f8e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.965 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.962+0000 7f8e5affd640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e4806b4d0 0x7f8e4806d990 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f8e50036000 tx=0x7f8e5003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:49.966+0000 7f8e43fff640 1 -- 172.21.15.94:0/2656267402 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e4c057c00 con 0x7f8e5c0c1aa0 2024-03-06T07:30:50.174 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.172+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e5c04f330 con 0x7f8e5c0c1aa0 2024-03-06T07:30:50.175 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.173+0000 7f8e43fff640 1 -- 172.21.15.94:0/2656267402 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f8e4c05bca0 con 0x7f8e5c0c1aa0 2024-03-06T07:30:50.175 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:50.175 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:50.175 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e4806b4d0 msgr2=0x7f8e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e4806b4d0 0x7f8e4806d990 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f8e50036000 tx=0x7f8e5003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 msgr2=0x7f8e5c0a9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c0a9f80 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00c4d0 tx=0x7f8e4c00c990 comp rx=0 tx=0).stop 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 shutdown_connections 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e4806b4d0 0x7f8e4806d990 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c155a40 0x7f8e5c153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.177 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c14f790 0x7f8e5c149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 --2- 172.21.15.94:0/2656267402 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e5c0c1aa0 0x7f8e5c0a9f80 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 >> 172.21.15.94:0/2656267402 conn(0x7f8e5c0c6710 msgr2=0x7f8e5c072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 shutdown_connections 2024-03-06T07:30:50.178 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:50.175+0000 7f8e620cf640 1 -- 172.21.15.94:0/2656267402 wait complete. 2024-03-06T07:30:50.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:50.590 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:51.591 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:51.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.701+0000 7f4e821a2640 1 Processor -- start 2024-03-06T07:30:51.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.701+0000 7f4e821a2640 1 -- start start 2024-03-06T07:30:51.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.702+0000 7f4e821a2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c0c5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.702+0000 7f4e821a2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7c158f30 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.702+0000 7f4e821a2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7c1591d0 con 0x7f4e7c0c5590 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.702+0000 7f4e821a2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0c3620 con 0x7f4e7c0c4c70 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.702+0000 7f4e8099f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4e7c0c29b0 0x7f4e7c1557c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40090/0 (socket says 172.21.15.94:40090) 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.702+0000 7f4e8099f640 1 -- 172.21.15.94:0/2267655515 learned_addr learned my addr 172.21.15.94:0/2267655515 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.702+0000 7f4e819a1640 1 --2- 172.21.15.94:0/2267655515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c0c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.703+0000 7f4e73fff640 1 -- 172.21.15.94:0/2267655515 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186072867 0 0) 0x7f4e7c158f30 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.703+0000 7f4e73fff640 1 -- 172.21.15.94:0/2267655515 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e68003520 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.703+0000 7f4e73fff640 1 -- 172.21.15.94:0/2267655515 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365331932 0 0) 0x7f4e7c1591d0 con 0x7f4e7c0c5590 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.703+0000 7f4e73fff640 1 -- 172.21.15.94:0/2267655515 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e7c158f30 con 0x7f4e7c0c5590 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.703+0000 7f4e819a1640 1 -- 172.21.15.94:0/2267655515 >> v1:172.21.15.145:6789/0 conn(0x7f4e7c0c5590 legacy=0x7f4e7c1550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.703+0000 7f4e819a1640 1 -- 172.21.15.94:0/2267655515 >> v1:172.21.15.94:6789/0 conn(0x7f4e7c0c29b0 legacy=0x7f4e7c1557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.703+0000 7f4e819a1640 1 -- 172.21.15.94:0/2267655515 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c0c32b0 con 0x7f4e7c0c4c70 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e819a1640 1 --2- 172.21.15.94:0/2267655515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c0c5970 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009dc0 tx=0x7f4e7402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=592359a65b197fa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e73fff640 1 -- 172.21.15.94:0/2267655515 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74039070 con 0x7f4e7c0c4c70 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e73fff640 1 -- 172.21.15.94:0/2267655515 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7402cc40 con 0x7f4e7c0c4c70 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e73fff640 1 -- 172.21.15.94:0/2267655515 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74034440 con 0x7f4e7c0c4c70 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e821a2640 1 -- 172.21.15.94:0/2267655515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 msgr2=0x7f4e7c0c5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e821a2640 1 --2- 172.21.15.94:0/2267655515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c0c5970 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009dc0 tx=0x7f4e7402be90 comp rx=0 tx=0).stop 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e821a2640 1 -- 172.21.15.94:0/2267655515 shutdown_connections 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e821a2640 1 --2- 172.21.15.94:0/2267655515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c0c5970 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e821a2640 1 -- 172.21.15.94:0/2267655515 >> 172.21.15.94:0/2267655515 conn(0x7f4e7c0c8b20 msgr2=0x7f4e7c0c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e821a2640 1 -- 172.21.15.94:0/2267655515 shutdown_connections 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.704+0000 7f4e821a2640 1 -- 172.21.15.94:0/2267655515 wait complete. 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.705+0000 7f4e821a2640 1 Processor -- start 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.705+0000 7f4e821a2640 1 -- start start 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e821a2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e7c0c29b0 0x7f4e7c152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e821a2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e821a2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e7c0c5590 0x7f4e7c148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e821a2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0c6860 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e821a2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0c66e0 con 0x7f4e7c0c5590 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e821a2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c160820 con 0x7f4e7c0c4c70 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e811a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e7c0c29b0 0x7f4e7c152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e811a0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e7c0c29b0 0x7f4e7c152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33830/0 (socket says 172.21.15.94:33830) 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e811a0640 1 -- 172.21.15.94:0/1609929300 learned_addr learned my addr 172.21.15.94:0/1609929300 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e8099f640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.706+0000 7f4e819a1640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e7c0c5590 0x7f4e7c148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e811a0640 1 -- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 msgr2=0x7f4e7c157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e811a0640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e811a0640 1 -- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e7c0c5590 msgr2=0x7f4e7c148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e811a0640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e7c0c5590 0x7f4e7c148970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e811a0640 1 -- 172.21.15.94:0/1609929300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c14e3f0 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e811a0640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e7c0c29b0 0x7f4e7c152110 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c460 tx=0x7f4e7800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e71ffb640 1 -- 172.21.15.94:0/1609929300 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78017070 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e71ffb640 1 -- 172.21.15.94:0/1609929300 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e780052f0 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7c14aa50 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e71ffb640 1 -- 172.21.15.94:0/1609929300 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78003830 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.707+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7c14acf0 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.708+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7c069430 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.709+0000 7f4e71ffb640 1 -- 172.21.15.94:0/1609929300 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e7801a070 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.709+0000 7f4e71ffb640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e6006b500 0x7f4e6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.709+0000 7f4e71ffb640 1 -- 172.21.15.94:0/1609929300 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f4e780896e0 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.709+0000 7f4e8099f640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e6006b500 0x7f4e6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.710+0000 7f4e8099f640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e6006b500 0x7f4e6006d9c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c154fc0 tx=0x7f4e6c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.713+0000 7f4e71ffb640 1 -- 172.21.15.94:0/1609929300 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e78057d10 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.921+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e7c14eb50 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.921+0000 7f4e71ffb640 1 -- 172.21.15.94:0/1609929300 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f4e7805bdb0 con 0x7f4e7c0c29b0 2024-03-06T07:30:51.923 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:51.923 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:51.923 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e6006b500 msgr2=0x7f4e6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e6006b500 0x7f4e6006d9c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c154fc0 tx=0x7f4e6c007d30 comp rx=0 tx=0).stop 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e7c0c29b0 msgr2=0x7f4e7c152110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e7c0c29b0 0x7f4e7c152110 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c460 tx=0x7f4e7800c920 comp rx=0 tx=0).stop 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 shutdown_connections 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4e6006b500 0x7f4e6006d9c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e7c0c5590 0x7f4e7c148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e7c0c4c70 0x7f4e7c157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 --2- 172.21.15.94:0/1609929300 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4e7c0c29b0 0x7f4e7c152110 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 >> 172.21.15.94:0/1609929300 conn(0x7f4e7c0c8b20 msgr2=0x7f4e7c0c6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 shutdown_connections 2024-03-06T07:30:51.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:51.923+0000 7f4e821a2640 1 -- 172.21.15.94:0/1609929300 wait complete. 2024-03-06T07:30:52.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:52.338 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:53.340 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:53.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.454+0000 7f6fbca6b640 1 Processor -- start 2024-03-06T07:30:53.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.454+0000 7f6fbca6b640 1 -- start start 2024-03-06T07:30:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fbca6b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6fb8156150 0x7f6fb81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fbca6b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80c28d0 con 0x7f6fb80c1aa0 2024-03-06T07:30:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fbca6b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80c2bb0 con 0x7f6fb814f790 2024-03-06T07:30:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fbca6b640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb80c2680 con 0x7f6fb8156150 2024-03-06T07:30:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fb6d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6fb8156150 0x7f6fb81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fb5d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6fb80c1aa0 0x7f6fb8155a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40102/0 (socket says 172.21.15.94:40102) 2024-03-06T07:30:53.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fb5d9b640 1 -- 172.21.15.94:0/4294339834 learned_addr learned my addr 172.21.15.94:0/4294339834 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:53.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fb6d9d640 1 -- 172.21.15.94:0/4294339834 >> v1:172.21.15.145:6789/0 conn(0x7f6fb814f790 legacy=0x7f6fb80c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fb6d9d640 1 -- 172.21.15.94:0/4294339834 >> v1:172.21.15.94:6789/0 conn(0x7f6fb80c1aa0 legacy=0x7f6fb8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.455+0000 7f6fb6d9d640 1 -- 172.21.15.94:0/4294339834 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80c2e90 con 0x7f6fb8156150 2024-03-06T07:30:53.459 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.456+0000 7f6fb6d9d640 1 --2- 172.21.15.94:0/4294339834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb81529d0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80c2800 tx=0x7f6fb002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20ef454df5d8787 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.456+0000 7f6fb559a640 1 -- 172.21.15.94:0/4294339834 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0039070 con 0x7f6fb8156150 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.456+0000 7f6fb559a640 1 -- 172.21.15.94:0/4294339834 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb002cce0 con 0x7f6fb8156150 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.456+0000 7f6fb559a640 1 -- 172.21.15.94:0/4294339834 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0034660 con 0x7f6fb8156150 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.456+0000 7f6fbca6b640 1 -- 172.21.15.94:0/4294339834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 msgr2=0x7f6fb81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.456+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/4294339834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb81529d0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80c2800 tx=0x7f6fb002bf30 comp rx=0 tx=0).stop 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.457+0000 7f6fbca6b640 1 -- 172.21.15.94:0/4294339834 shutdown_connections 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.457+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/4294339834 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb81529d0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.457+0000 7f6fbca6b640 1 -- 172.21.15.94:0/4294339834 >> 172.21.15.94:0/4294339834 conn(0x7f6fb80c6710 msgr2=0x7f6fb80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.457+0000 7f6fbca6b640 1 -- 172.21.15.94:0/4294339834 shutdown_connections 2024-03-06T07:30:53.460 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.457+0000 7f6fbca6b640 1 -- 172.21.15.94:0/4294339834 wait complete. 2024-03-06T07:30:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.458+0000 7f6fbca6b640 1 Processor -- start 2024-03-06T07:30:53.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.458+0000 7f6fbca6b640 1 -- start start 2024-03-06T07:30:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb80c1aa0 0x7f6fb81466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb814f790 0x7f6fb813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb813eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80c5e30 con 0x7f6fb8156150 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80c0d60 con 0x7f6fb814f790 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80c5fb0 con 0x7f6fb80c1aa0 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb813eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb813eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33844/0 (socket says 172.21.15.94:33844) 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 -- 172.21.15.94:0/1230243915 learned_addr learned my addr 172.21.15.94:0/1230243915 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb5d9b640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb814f790 0x7f6fb813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb659c640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb80c1aa0 0x7f6fb81466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 -- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb80c1aa0 msgr2=0x7f6fb81466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb80c1aa0 0x7f6fb81466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 -- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb814f790 msgr2=0x7f6fb813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb814f790 0x7f6fb813e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 -- 172.21.15.94:0/1230243915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb81e94b0 con 0x7f6fb8156150 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fb6d9d640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb813eca0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009e80 tx=0x7f6fb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fa6ffd640 1 -- 172.21.15.94:0/1230243915 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0039050 con 0x7f6fb8156150 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fa6ffd640 1 -- 172.21.15.94:0/1230243915 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb0045070 con 0x7f6fb8156150 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb81e9790 con 0x7f6fb8156150 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fa6ffd640 1 -- 172.21.15.94:0/1230243915 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb003e830 con 0x7f6fb8156150 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.459+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb81440a0 con 0x7f6fb8156150 2024-03-06T07:30:53.465 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.460+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb804f330 con 0x7f6fb8156150 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.461+0000 7f6fa6ffd640 1 -- 172.21.15.94:0/1230243915 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fb003e9d0 con 0x7f6fb8156150 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.461+0000 7f6fa6ffd640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f9806b4d0 0x7f6f9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.461+0000 7f6fa6ffd640 1 -- 172.21.15.94:0/1230243915 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6fb00ad190 con 0x7f6fb8156150 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.462+0000 7f6fb659c640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f9806b4d0 0x7f6f9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.466 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.462+0000 7f6fb659c640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f9806b4d0 0x7f6f9806d990 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f6fb815f030 tx=0x7f6fac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.467 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.466+0000 7f6fa6ffd640 1 -- 172.21.15.94:0/1230243915 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fb007b8e0 con 0x7f6fb8156150 2024-03-06T07:30:53.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.674+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fb80c1f40 con 0x7f6fb8156150 2024-03-06T07:30:53.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.675+0000 7f6fa6ffd640 1 -- 172.21.15.94:0/1230243915 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f6fb007f980 con 0x7f6fb8156150 2024-03-06T07:30:53.677 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:53.677 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:53.677 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f9806b4d0 msgr2=0x7f6f9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f9806b4d0 0x7f6f9806d990 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f6fb815f030 tx=0x7f6fac009040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 msgr2=0x7f6fb813eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb813eca0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009e80 tx=0x7f6fb0004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 shutdown_connections 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6f9806b4d0 0x7f6f9806d990 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6fb8156150 0x7f6fb813eca0 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb814f790 0x7f6fb813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 --2- 172.21.15.94:0/1230243915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb80c1aa0 0x7f6fb81466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.679 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 >> 172.21.15.94:0/1230243915 conn(0x7f6fb80c6710 msgr2=0x7f6fb80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 shutdown_connections 2024-03-06T07:30:53.680 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:53.677+0000 7f6fbca6b640 1 -- 172.21.15.94:0/1230243915 wait complete. 2024-03-06T07:30:54.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:54.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:55.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:55.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.202+0000 7f5cb42bb640 1 Processor -- start 2024-03-06T07:30:55.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.202+0000 7f5cb42bb640 1 -- start start 2024-03-06T07:30:55.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb42bb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5cac150580 0x7f5cac06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb42bb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5cac0c1320 0x7f5cac06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb42bb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac153a60 con 0x7f5cac0c6000 2024-03-06T07:30:55.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb42bb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac153d40 con 0x7f5cac0c1320 2024-03-06T07:30:55.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb42bb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cac06c2c0 con 0x7f5cac150580 2024-03-06T07:30:55.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb1856640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5cac0c1320 0x7f5cac06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb1856640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5cac0c1320 0x7f5cac06e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33852/0 (socket says 172.21.15.94:33852) 2024-03-06T07:30:55.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb1856640 1 -- 172.21.15.94:0/164367004 learned_addr learned my addr 172.21.15.94:0/164367004 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:55.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb2057640 1 --2- 172.21.15.94:0/164367004 >> v2:172.21.15.145:3300/0 conn(0x7f5cac150580 0x7f5cac06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb1856640 1 -- 172.21.15.94:0/164367004 >> v2:172.21.15.145:3300/0 conn(0x7f5cac150580 msgr2=0x7f5cac06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb1856640 1 --2- 172.21.15.94:0/164367004 >> v2:172.21.15.145:3300/0 conn(0x7f5cac150580 0x7f5cac06dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb1856640 1 -- 172.21.15.94:0/164367004 >> v1:172.21.15.145:6789/0 conn(0x7f5cac0c6000 legacy=0x7f5cac1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.203+0000 7f5cb1856640 1 -- 172.21.15.94:0/164367004 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac06c5a0 con 0x7f5cac0c1320 2024-03-06T07:30:55.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.204+0000 7f5cb1856640 1 --2- 172.21.15.94:0/164367004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac0c1320 0x7f5cac06e250 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0a9550 tx=0x7f5ca802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e5b09a0bc79357 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.204+0000 7f5cb1055640 1 -- 172.21.15.94:0/164367004 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8035020 con 0x7f5cac0c1320 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.204+0000 7f5cb1055640 1 -- 172.21.15.94:0/164367004 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca8031870 con 0x7f5cac0c1320 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.204+0000 7f5cb1055640 1 -- 172.21.15.94:0/164367004 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8031c40 con 0x7f5cac0c1320 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 -- 172.21.15.94:0/164367004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac0c1320 msgr2=0x7f5cac06e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/164367004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac0c1320 0x7f5cac06e250 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f5cac0a9550 tx=0x7f5ca802b910 comp rx=0 tx=0).stop 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 -- 172.21.15.94:0/164367004 shutdown_connections 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/164367004 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac0c1320 0x7f5cac06e250 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/164367004 >> v2:172.21.15.145:3300/0 conn(0x7f5cac150580 0x7f5cac06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 -- 172.21.15.94:0/164367004 >> 172.21.15.94:0/164367004 conn(0x7f5cac0c6710 msgr2=0x7f5cac0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 -- 172.21.15.94:0/164367004 shutdown_connections 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.205+0000 7f5cb42bb640 1 -- 172.21.15.94:0/164367004 wait complete. 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.206+0000 7f5cb42bb640 1 Processor -- start 2024-03-06T07:30:55.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.206+0000 7f5cb42bb640 1 -- start start 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.206+0000 7f5cb42bb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5cac0c1320 0x7f5cac1ef150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.206+0000 7f5cb42bb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5cac0c6000 0x7f5cac1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.206+0000 7f5cb42bb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac150580 0x7f5cac149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.206+0000 7f5cb42bb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac0c0dc0 con 0x7f5cac150580 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb42bb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac0c0c40 con 0x7f5cac0c1320 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb42bb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac156dc0 con 0x7f5cac0c6000 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac150580 0x7f5cac149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac150580 0x7f5cac149810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33862/0 (socket says 172.21.15.94:33862) 2024-03-06T07:30:55.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 -- 172.21.15.94:0/2657780976 learned_addr learned my addr 172.21.15.94:0/2657780976 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2057640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5cac0c1320 0x7f5cac1ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb1856640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5cac0c6000 0x7f5cac1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 -- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5cac0c6000 msgr2=0x7f5cac1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5cac0c6000 0x7f5cac1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 -- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5cac0c1320 msgr2=0x7f5cac1ef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5cac0c1320 0x7f5cac1ef150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 -- 172.21.15.94:0/2657780976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac154a00 con 0x7f5cac150580 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb2858640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac150580 0x7f5cac149810 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c7a0 tx=0x7f5ca400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5ca2ffd640 1 -- 172.21.15.94:0/2657780976 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4003b70 con 0x7f5cac150580 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5ca2ffd640 1 -- 172.21.15.94:0/2657780976 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca4003d30 con 0x7f5cac150580 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5ca2ffd640 1 -- 172.21.15.94:0/2657780976 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400aa90 con 0x7f5cac150580 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac0ab2f0 con 0x7f5cac150580 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.207+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac154440 con 0x7f5cac150580 2024-03-06T07:30:55.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.208+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cac0ab5e0 con 0x7f5cac150580 2024-03-06T07:30:55.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.209+0000 7f5ca2ffd640 1 -- 172.21.15.94:0/2657780976 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ca400ac30 con 0x7f5cac150580 2024-03-06T07:30:55.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.209+0000 7f5ca2ffd640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c9006b4d0 0x7f5c9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.209+0000 7f5ca2ffd640 1 -- 172.21.15.94:0/2657780976 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5ca408a290 con 0x7f5cac150580 2024-03-06T07:30:55.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.209+0000 7f5cb2057640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c9006b4d0 0x7f5c9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.213 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.210+0000 7f5cb2057640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c9006b4d0 0x7f5c9006d990 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009aa0 tx=0x7f5c9c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.213+0000 7f5ca2ffd640 1 -- 172.21.15.94:0/2657780976 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ca4058830 con 0x7f5cac150580 2024-03-06T07:30:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.422+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cac04f330 con 0x7f5cac150580 2024-03-06T07:30:55.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.422+0000 7f5ca2ffd640 1 -- 172.21.15.94:0/2657780976 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f5ca405c8d0 con 0x7f5cac150580 2024-03-06T07:30:55.424 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:55.424 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:55.425 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c9006b4d0 msgr2=0x7f5c9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c9006b4d0 0x7f5c9006d990 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c009aa0 tx=0x7f5c9c009190 comp rx=0 tx=0).stop 2024-03-06T07:30:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac150580 msgr2=0x7f5cac149810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac150580 0x7f5cac149810 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400c7a0 tx=0x7f5ca400cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 shutdown_connections 2024-03-06T07:30:55.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f5c9006b4d0 0x7f5c9006d990 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5cac150580 0x7f5cac149810 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5cac0c6000 0x7f5cac1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 --2- 172.21.15.94:0/2657780976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5cac0c1320 0x7f5cac1ef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 >> 172.21.15.94:0/2657780976 conn(0x7f5cac0c6710 msgr2=0x7f5cac06dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 shutdown_connections 2024-03-06T07:30:55.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:55.424+0000 7f5cb42bb640 1 -- 172.21.15.94:0/2657780976 wait complete. 2024-03-06T07:30:55.839 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:55.839 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:56.840 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:56.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.950+0000 7fbb28d79640 1 Processor -- start 2024-03-06T07:30:56.952 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.950+0000 7fbb28d79640 1 -- start start 2024-03-06T07:30:56.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb28d79640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbb240c3750 0x7fbb240c1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb28d79640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbb240a5e60 0x7fbb240c1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb28d79640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb240c5490 con 0x7fbb240c3370 2024-03-06T07:30:56.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb28d79640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb240c5770 con 0x7fbb240a5e60 2024-03-06T07:30:56.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb28d79640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb240c5a50 con 0x7fbb240c3750 2024-03-06T07:30:56.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb22d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbb240a5e60 0x7fbb240c1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb22d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbb240a5e60 0x7fbb240c1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33876/0 (socket says 172.21.15.94:33876) 2024-03-06T07:30:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb22d9d640 1 -- 172.21.15.94:0/1765216149 learned_addr learned my addr 172.21.15.94:0/1765216149 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb1bfff640 1 --2- 172.21.15.94:0/1765216149 >> v2:172.21.15.145:3300/0 conn(0x7fbb240c3750 0x7fbb240c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb22d9d640 1 -- 172.21.15.94:0/1765216149 >> v2:172.21.15.145:3300/0 conn(0x7fbb240c3750 msgr2=0x7fbb240c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb22d9d640 1 --2- 172.21.15.94:0/1765216149 >> v2:172.21.15.145:3300/0 conn(0x7fbb240c3750 0x7fbb240c1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb22d9d640 1 -- 172.21.15.94:0/1765216149 >> v1:172.21.15.145:6789/0 conn(0x7fbb240c3370 legacy=0x7fbb24155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.954 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.951+0000 7fbb22d9d640 1 -- 172.21.15.94:0/1765216149 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb240c5d30 con 0x7fbb240a5e60 2024-03-06T07:30:56.955 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.952+0000 7fbb22d9d640 1 --2- 172.21.15.94:0/1765216149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240a5e60 0x7fbb240c1650 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fbb2406e6b0 tx=0x7fbb1c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28727b8a465000ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.952+0000 7fbb21d9b640 1 -- 172.21.15.94:0/1765216149 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c02c9f0 con 0x7fbb240a5e60 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.952+0000 7fbb21d9b640 1 -- 172.21.15.94:0/1765216149 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c02cb90 con 0x7fbb240a5e60 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.952+0000 7fbb21d9b640 1 -- 172.21.15.94:0/1765216149 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c0343f0 con 0x7fbb240a5e60 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 -- 172.21.15.94:0/1765216149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240a5e60 msgr2=0x7fbb240c1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 --2- 172.21.15.94:0/1765216149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240a5e60 0x7fbb240c1650 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fbb2406e6b0 tx=0x7fbb1c02be50 comp rx=0 tx=0).stop 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 -- 172.21.15.94:0/1765216149 shutdown_connections 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 --2- 172.21.15.94:0/1765216149 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240a5e60 0x7fbb240c1650 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 --2- 172.21.15.94:0/1765216149 >> v2:172.21.15.145:3300/0 conn(0x7fbb240c3750 0x7fbb240c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 -- 172.21.15.94:0/1765216149 >> 172.21.15.94:0/1765216149 conn(0x7fbb240c6710 msgr2=0x7fbb240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 -- 172.21.15.94:0/1765216149 shutdown_connections 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.953+0000 7fbb28d79640 1 -- 172.21.15.94:0/1765216149 wait complete. 2024-03-06T07:30:56.956 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.954+0000 7fbb28d79640 1 Processor -- start 2024-03-06T07:30:56.957 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.954+0000 7fbb28d79640 1 -- start start 2024-03-06T07:30:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb28d79640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb240a5e60 0x7fbb241eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb28d79640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240c3370 0x7fbb24156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb28d79640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb240c3750 0x7fbb2406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb28d79640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb24159910 con 0x7fbb240c3370 2024-03-06T07:30:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb28d79640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb24159790 con 0x7fbb240c3750 2024-03-06T07:30:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb28d79640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb240c1b90 con 0x7fbb240a5e60 2024-03-06T07:30:56.959 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240c3370 0x7fbb24156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240c3370 0x7fbb24156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:33882/0 (socket says 172.21.15.94:33882) 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 -- 172.21.15.94:0/3120224077 learned_addr learned my addr 172.21.15.94:0/3120224077 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb22d9d640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb240c3750 0x7fbb2406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb2259c640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb240a5e60 0x7fbb241eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 -- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb240a5e60 msgr2=0x7fbb241eaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb240a5e60 0x7fbb241eaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 -- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb240c3750 msgr2=0x7fbb2406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb240c3750 0x7fbb2406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 -- 172.21.15.94:0/3120224077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2414de80 con 0x7fbb240c3370 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1bfff640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240c3370 0x7fbb24156cf0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00e8a0 tx=0x7fbb0c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1affd640 1 -- 172.21.15.94:0/3120224077 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c017070 con 0x7fbb240c3370 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1affd640 1 -- 172.21.15.94:0/3120224077 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c00c940 con 0x7fbb240c3370 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2414e160 con 0x7fbb240c3370 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.955+0000 7fbb1affd640 1 -- 172.21.15.94:0/3120224077 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00cd30 con 0x7fbb240c3370 2024-03-06T07:30:56.960 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.956+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb24158750 con 0x7fbb240c3370 2024-03-06T07:30:56.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.956+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2404f330 con 0x7fbb240c3370 2024-03-06T07:30:56.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.957+0000 7fbb1affd640 1 -- 172.21.15.94:0/3120224077 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb0c01a070 con 0x7fbb240c3370 2024-03-06T07:30:56.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.957+0000 7fbb1affd640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbb0406b4d0 0x7fbb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.957+0000 7fbb1affd640 1 -- 172.21.15.94:0/3120224077 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbb0c08a580 con 0x7fbb240c3370 2024-03-06T07:30:56.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.957+0000 7fbb2259c640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbb0406b4d0 0x7fbb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.961 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.958+0000 7fbb2259c640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbb0406b4d0 0x7fbb0406d990 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fbb14001780 tx=0x7fbb14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:56.962+0000 7fbb1affd640 1 -- 172.21.15.94:0/3120224077 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb0c058bb0 con 0x7fbb240c3370 2024-03-06T07:30:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.166+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb24068450 con 0x7fbb240c3370 2024-03-06T07:30:57.168 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.167+0000 7fbb1affd640 1 -- 172.21.15.94:0/3120224077 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fbb0c05cc50 con 0x7fbb240c3370 2024-03-06T07:30:57.169 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:57.169 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:57.169 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:57.170 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbb0406b4d0 msgr2=0x7fbb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbb0406b4d0 0x7fbb0406d990 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fbb14001780 tx=0x7fbb14009040 comp rx=0 tx=0).stop 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240c3370 msgr2=0x7fbb24156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240c3370 0x7fbb24156cf0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00e8a0 tx=0x7fbb0c00ed60 comp rx=0 tx=0).stop 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 shutdown_connections 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbb0406b4d0 0x7fbb0406d990 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb240c3750 0x7fbb2406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbb240c3370 0x7fbb24156cf0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 --2- 172.21.15.94:0/3120224077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb240a5e60 0x7fbb241eaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.168+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 >> 172.21.15.94:0/3120224077 conn(0x7fbb240c6710 msgr2=0x7fbb240c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.169+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 shutdown_connections 2024-03-06T07:30:57.171 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:57.169+0000 7fbb28d79640 1 -- 172.21.15.94:0/3120224077 wait complete. 2024-03-06T07:30:57.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:57.584 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:30:58.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:58.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.695+0000 7fbbe301c640 1 Processor -- start 2024-03-06T07:30:58.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.695+0000 7fbbe301c640 1 -- start start 2024-03-06T07:30:58.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe301c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbbdc150580 0x7fbbdc06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe301c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbdc153a60 con 0x7fbbdc0c1320 2024-03-06T07:30:58.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe301c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbdc153d40 con 0x7fbbdc0c6000 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe301c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbdc06c2c0 con 0x7fbbdc150580 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbdbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbbdc0c1320 0x7fbbdc1566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45716/0 (socket says 172.21.15.94:45716) 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbdbfff640 1 -- 172.21.15.94:0/1091650225 learned_addr learned my addr 172.21.15.94:0/1091650225 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe0db8640 1 --2- 172.21.15.94:0/1091650225 >> v2:172.21.15.145:3300/0 conn(0x7fbbdc150580 0x7fbbdc06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbdb7fe640 1 -- 172.21.15.94:0/1091650225 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308241726 0 0) 0x7fbbdc153a60 con 0x7fbbdc0c1320 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbdb7fe640 1 -- 172.21.15.94:0/1091650225 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc003520 con 0x7fbbdc0c1320 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe0db8640 1 -- 172.21.15.94:0/1091650225 >> v1:172.21.15.145:6789/0 conn(0x7fbbdc0c6000 legacy=0x7fbbdc156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe0db8640 1 -- 172.21.15.94:0/1091650225 >> v1:172.21.15.94:6789/0 conn(0x7fbbdc0c1320 legacy=0x7fbbdc1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbdb7fe640 1 -- 172.21.15.94:0/1091650225 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268515067 0 0) 0x7fbbdc153d40 con 0x7fbbdc0c6000 2024-03-06T07:30:58.699 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.696+0000 7fbbe0db8640 1 -- 172.21.15.94:0/1091650225 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc06c5a0 con 0x7fbbdc150580 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.697+0000 7fbbe0db8640 1 --2- 172.21.15.94:0/1091650225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc150580 0x7fbbdc06dd10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00097a0 tx=0x7fbbd002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=770125b67ca30c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.697+0000 7fbbdb7fe640 1 -- 172.21.15.94:0/1091650225 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0035020 con 0x7fbbdc150580 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.697+0000 7fbbdb7fe640 1 -- 172.21.15.94:0/1091650225 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd0031830 con 0x7fbbdc150580 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.697+0000 7fbbdb7fe640 1 -- 172.21.15.94:0/1091650225 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0031c00 con 0x7fbbdc150580 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.698+0000 7fbbe301c640 1 -- 172.21.15.94:0/1091650225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc150580 msgr2=0x7fbbdc06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.698+0000 7fbbe301c640 1 --2- 172.21.15.94:0/1091650225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc150580 0x7fbbdc06dd10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00097a0 tx=0x7fbbd002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.698+0000 7fbbe301c640 1 -- 172.21.15.94:0/1091650225 shutdown_connections 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.698+0000 7fbbe301c640 1 --2- 172.21.15.94:0/1091650225 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc150580 0x7fbbdc06dd10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.698+0000 7fbbe301c640 1 -- 172.21.15.94:0/1091650225 >> 172.21.15.94:0/1091650225 conn(0x7fbbdc0c6710 msgr2=0x7fbbdc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.698+0000 7fbbe301c640 1 -- 172.21.15.94:0/1091650225 shutdown_connections 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.698+0000 7fbbe301c640 1 -- 172.21.15.94:0/1091650225 wait complete. 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.699+0000 7fbbe301c640 1 Processor -- start 2024-03-06T07:30:58.701 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.699+0000 7fbbe301c640 1 -- start start 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe301c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc0c1320 0x7fbbdc154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe301c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbbdc0c6000 0x7fbbdc0a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe301c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbbdc150580 0x7fbbdc149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe301c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc0c0dc0 con 0x7fbbdc150580 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe301c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc0c0c40 con 0x7fbbdc0c1320 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe301c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc06e250 con 0x7fbbdc0c6000 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbbdc150580 0x7fbbdc149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbbdc150580 0x7fbbdc149910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38276/0 (socket says 172.21.15.94:38276) 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 -- 172.21.15.94:0/3165809387 learned_addr learned my addr 172.21.15.94:0/3165809387 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:30:58.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe0db8640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc0c1320 0x7fbbdc154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbdbfff640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbbdc0c6000 0x7fbbdc0a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 -- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbbdc0c6000 msgr2=0x7fbbdc0a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbbdc0c6000 0x7fbbdc0a7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 -- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc0c1320 msgr2=0x7fbbdc154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc0c1320 0x7fbbdc154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.700+0000 7fbbe15b9640 1 -- 172.21.15.94:0/3165809387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc155160 con 0x7fbbdc150580 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.701+0000 7fbbe15b9640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbbdc150580 0x7fbbdc149910 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400c760 tx=0x7fbbd400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.701+0000 7fbbd97fa640 1 -- 172.21.15.94:0/3165809387 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4003da0 con 0x7fbbdc150580 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.701+0000 7fbbd97fa640 1 -- 172.21.15.94:0/3165809387 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd4005070 con 0x7fbbdc150580 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.701+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbdc0a95f0 con 0x7fbbdc150580 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.701+0000 7fbbd97fa640 1 -- 172.21.15.94:0/3165809387 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4005460 con 0x7fbbdc150580 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.701+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbdc0a9bb0 con 0x7fbbdc150580 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.702+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbdc04f330 con 0x7fbbdc150580 2024-03-06T07:30:58.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.702+0000 7fbbd97fa640 1 -- 172.21.15.94:0/3165809387 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbd4005600 con 0x7fbbdc150580 2024-03-06T07:30:58.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.703+0000 7fbbd97fa640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbbc006b4d0 0x7fbbc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.703+0000 7fbbd97fa640 1 -- 172.21.15.94:0/3165809387 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbbd408a220 con 0x7fbbdc150580 2024-03-06T07:30:58.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.703+0000 7fbbe0db8640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbbc006b4d0 0x7fbbc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.703+0000 7fbbe0db8640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbbc006b4d0 0x7fbbc006d990 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0036000 tx=0x7fbbd003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.707+0000 7fbbd97fa640 1 -- 172.21.15.94:0/3165809387 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbd4058850 con 0x7fbbdc150580 2024-03-06T07:30:58.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.915+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbdc0a9ff0 con 0x7fbbdc150580 2024-03-06T07:30:58.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.915+0000 7fbbd97fa640 1 -- 172.21.15.94:0/3165809387 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fbbd405c8f0 con 0x7fbbdc150580 2024-03-06T07:30:58.918 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:30:58.918 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:30:58.918 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:30:58.919 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbbc006b4d0 msgr2=0x7fbbc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbbc006b4d0 0x7fbbc006d990 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0036000 tx=0x7fbbd003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbbdc150580 msgr2=0x7fbbdc149910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbbdc150580 0x7fbbdc149910 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400c760 tx=0x7fbbd400cc20 comp rx=0 tx=0).stop 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 shutdown_connections 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbbc006b4d0 0x7fbbc006d990 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbbdc150580 0x7fbbdc149910 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbbdc0c6000 0x7fbbdc0a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 --2- 172.21.15.94:0/3165809387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbbdc0c1320 0x7fbbdc154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 >> 172.21.15.94:0/3165809387 conn(0x7fbbdc0c6710 msgr2=0x7fbbdc0bfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 shutdown_connections 2024-03-06T07:30:58.920 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:30:58.917+0000 7fbbe301c640 1 -- 172.21.15.94:0/3165809387 wait complete. 2024-03-06T07:30:59.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:30:59.332 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:31:00.333 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:00.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.444+0000 7f3eed379640 1 Processor -- start 2024-03-06T07:31:00.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.444+0000 7f3eed379640 1 -- start start 2024-03-06T07:31:00.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3eed379640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8152520 0x7f3ee806ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3eed379640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3ee80720a0 0x7f3ee806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3eed379640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee80c1ae0 con 0x7f3ee8155c80 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3eed379640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee80c1dc0 con 0x7f3ee80720a0 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3eed379640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee806e450 con 0x7f3ee8152520 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee6ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8152520 0x7f3ee806ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee77fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3ee80720a0 0x7f3ee806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee67fc640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f3ee8155c80 0x7f3ee80c12b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:46286/0 (socket says 172.21.15.94:46286) 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee67fc640 1 -- 172.21.15.94:0/556273638 learned_addr learned my addr 172.21.15.94:0/556273638 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee77fe640 1 -- 172.21.15.94:0/556273638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8152520 msgr2=0x7f3ee806ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee77fe640 1 --2- 172.21.15.94:0/556273638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8152520 0x7f3ee806ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee77fe640 1 -- 172.21.15.94:0/556273638 >> v1:172.21.15.145:6789/0 conn(0x7f3ee8155c80 legacy=0x7f3ee80c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee5ffb640 1 -- 172.21.15.94:0/556273638 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745009782 0 0) 0x7f3ee80c1ae0 con 0x7f3ee8155c80 2024-03-06T07:31:00.448 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.445+0000 7f3ee77fe640 1 -- 172.21.15.94:0/556273638 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee806e730 con 0x7f3ee80720a0 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.446+0000 7f3ee77fe640 1 --2- 172.21.15.94:0/556273638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee806afc0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009d90 tx=0x7f3ee002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb0732efbf285669 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.446+0000 7f3ee5ffb640 1 -- 172.21.15.94:0/556273638 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0039070 con 0x7f3ee80720a0 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.446+0000 7f3ee5ffb640 1 -- 172.21.15.94:0/556273638 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee002cc50 con 0x7f3ee80720a0 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.446+0000 7f3ee5ffb640 1 -- 172.21.15.94:0/556273638 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0034440 con 0x7f3ee80720a0 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 -- 172.21.15.94:0/556273638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 msgr2=0x7f3ee806afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 --2- 172.21.15.94:0/556273638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee806afc0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009d90 tx=0x7f3ee002bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 -- 172.21.15.94:0/556273638 shutdown_connections 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 --2- 172.21.15.94:0/556273638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee806afc0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 --2- 172.21.15.94:0/556273638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8152520 0x7f3ee806ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 -- 172.21.15.94:0/556273638 >> 172.21.15.94:0/556273638 conn(0x7f3ee80c76b0 msgr2=0x7f3ee80c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.450 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 -- 172.21.15.94:0/556273638 shutdown_connections 2024-03-06T07:31:00.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.447+0000 7f3eed379640 1 -- 172.21.15.94:0/556273638 wait complete. 2024-03-06T07:31:00.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.448+0000 7f3eed379640 1 Processor -- start 2024-03-06T07:31:00.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.448+0000 7f3eed379640 1 -- start start 2024-03-06T07:31:00.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3eed379640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee81ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3eed379640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee8152520 0x7f3ee80a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3eed379640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8155c80 0x7f3ee80a88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3eed379640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8152a80 con 0x7f3ee8152520 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3eed379640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8152900 con 0x7f3ee80720a0 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3eed379640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee806b500 con 0x7f3ee8155c80 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee8152520 0x7f3ee80a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee8152520 0x7f3ee80a97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38280/0 (socket says 172.21.15.94:38280) 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 -- 172.21.15.94:0/4109295987 learned_addr learned my addr 172.21.15.94:0/4109295987 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee6ffd640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee81ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee77fe640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8155c80 0x7f3ee80a88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 -- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8155c80 msgr2=0x7f3ee80a88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8155c80 0x7f3ee80a88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 -- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 msgr2=0x7f3ee81ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee81ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee67fc640 1 -- 172.21.15.94:0/4109295987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80a6df0 con 0x7f3ee8152520 2024-03-06T07:31:00.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.449+0000 7f3ee6ffd640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee81ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.450+0000 7f3ee67fc640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee8152520 0x7f3ee80a97e0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c500 tx=0x7f3edc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.450+0000 7f3ecf7fe640 1 -- 172.21.15.94:0/4109295987 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc018070 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.450+0000 7f3ecf7fe640 1 -- 172.21.15.94:0/4109295987 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc007580 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.450+0000 7f3ecf7fe640 1 -- 172.21.15.94:0/4109295987 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc004a20 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.450+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee81eebf0 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.450+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee80a7370 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.451+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee804f330 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.451+0000 7f3ecf7fe640 1 -- 172.21.15.94:0/4109295987 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3edc01b070 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.452+0000 7f3ecf7fe640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed406b4d0 0x7f3ed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.452+0000 7f3ecf7fe640 1 -- 172.21.15.94:0/4109295987 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3edc089680 con 0x7f3ee8152520 2024-03-06T07:31:00.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.452+0000 7f3ee6ffd640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed406b4d0 0x7f3ed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.452+0000 7f3ee6ffd640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed406b4d0 0x7f3ed406d990 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3ee81e8510 tx=0x7f3ed8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.456+0000 7f3ecf7fe640 1 -- 172.21.15.94:0/4109295987 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3edc057cb0 con 0x7f3ee8152520 2024-03-06T07:31:00.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.666+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ee80c0b70 con 0x7f3ee8152520 2024-03-06T07:31:00.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.666+0000 7f3ecf7fe640 1 -- 172.21.15.94:0/4109295987 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f3edc05bd50 con 0x7f3ee8152520 2024-03-06T07:31:00.668 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:00.669 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:31:00.669 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:31:00.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed406b4d0 msgr2=0x7f3ed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed406b4d0 0x7f3ed406d990 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f3ee81e8510 tx=0x7f3ed8009190 comp rx=0 tx=0).stop 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee8152520 msgr2=0x7f3ee80a97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee8152520 0x7f3ee80a97e0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c500 tx=0x7f3edc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 shutdown_connections 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3ed406b4d0 0x7f3ed406d990 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ee8155c80 0x7f3ee80a88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ee8152520 0x7f3ee80a97e0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 --2- 172.21.15.94:0/4109295987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ee80720a0 0x7f3ee81ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 >> 172.21.15.94:0/4109295987 conn(0x7f3ee80c76b0 msgr2=0x7f3ee8151d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 shutdown_connections 2024-03-06T07:31:00.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:00.668+0000 7f3eed379640 1 -- 172.21.15.94:0/4109295987 wait complete. 2024-03-06T07:31:01.083 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:31:01.084 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:31:02.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:02.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.195+0000 7f02c2c5c640 1 Processor -- start 2024-03-06T07:31:02.197 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.195+0000 7f02c2c5c640 1 -- start start 2024-03-06T07:31:02.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02c2c5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f02bc150580 0x7f02bc06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02c2c5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc0c6000 0x7f02bc06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02c2c5c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02bc153a60 con 0x7f02bc0c1320 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02c2c5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02bc153d40 con 0x7f02bc150580 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02c2c5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc06c2c0 con 0x7f02bc0c6000 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02c09f8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f02bc150580 0x7f02bc06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02bbfff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f02bc0c1320 0x7f02bc1566b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:46302/0 (socket says 172.21.15.94:46302) 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02bbfff640 1 -- 172.21.15.94:0/1256033524 learned_addr learned my addr 172.21.15.94:0/1256033524 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.196+0000 7f02c11f9640 1 --2- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc0c6000 0x7f02bc06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.197+0000 7f02c09f8640 1 -- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc0c6000 msgr2=0x7f02bc06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.197+0000 7f02c09f8640 1 --2- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc0c6000 0x7f02bc06e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.197+0000 7f02c09f8640 1 -- 172.21.15.94:0/1256033524 >> v1:172.21.15.145:6789/0 conn(0x7f02bc0c1320 legacy=0x7f02bc1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.197+0000 7f02c09f8640 1 -- 172.21.15.94:0/1256033524 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc06c5a0 con 0x7f02bc150580 2024-03-06T07:31:02.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.197+0000 7f02c09f8640 1 --2- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc150580 0x7f02bc06dd10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0097a0 tx=0x7f02ac02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997851eaeeb557e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.197+0000 7f02bb7fe640 1 -- 172.21.15.94:0/1256033524 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0315a0 con 0x7f02bc150580 2024-03-06T07:31:02.201 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02bb7fe640 1 -- 172.21.15.94:0/1256033524 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac037070 con 0x7f02bc150580 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02bb7fe640 1 -- 172.21.15.94:0/1256033524 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac031960 con 0x7f02bc150580 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 -- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc150580 msgr2=0x7f02bc06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc150580 0x7f02bc06dd10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0097a0 tx=0x7f02ac02b840 comp rx=0 tx=0).stop 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 -- 172.21.15.94:0/1256033524 shutdown_connections 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc0c6000 0x7f02bc06e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/1256033524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc150580 0x7f02bc06dd10 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 -- 172.21.15.94:0/1256033524 >> 172.21.15.94:0/1256033524 conn(0x7f02bc0c6710 msgr2=0x7f02bc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 -- 172.21.15.94:0/1256033524 shutdown_connections 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.198+0000 7f02c2c5c640 1 -- 172.21.15.94:0/1256033524 wait complete. 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.199+0000 7f02c2c5c640 1 Processor -- start 2024-03-06T07:31:02.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.199+0000 7f02c2c5c640 1 -- start start 2024-03-06T07:31:02.205 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c2c5c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc0c1320 0x7f02bc149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c2c5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc0c6000 0x7f02bc070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c2c5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc150580 0x7f02bc0729a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c2c5c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0c0dc0 con 0x7f02bc0c1320 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c2c5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0c0c40 con 0x7f02bc0c6000 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c2c5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc156dc0 con 0x7f02bc150580 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c09f8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc0c1320 0x7f02bc149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c09f8640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc0c1320 0x7f02bc149ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38282/0 (socket says 172.21.15.94:38282) 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c09f8640 1 -- 172.21.15.94:0/2696602030 learned_addr learned my addr 172.21.15.94:0/2696602030 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02bbfff640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc0c6000 0x7f02bc070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.200+0000 7f02c11f9640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc150580 0x7f02bc0729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c09f8640 1 -- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc150580 msgr2=0x7f02bc0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c09f8640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc150580 0x7f02bc0729a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c09f8640 1 -- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc0c6000 msgr2=0x7f02bc070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c09f8640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc0c6000 0x7f02bc070770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c09f8640 1 -- 172.21.15.94:0/2696602030 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc1511b0 con 0x7f02bc0c1320 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c09f8640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc0c1320 0x7f02bc149ca0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f02ac034040 tx=0x7f02ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02b97fa640 1 -- 172.21.15.94:0/2696602030 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac032630 con 0x7f02bc0c1320 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02b97fa640 1 -- 172.21.15.94:0/2696602030 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac0327f0 con 0x7f02bc0c1320 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02b97fa640 1 -- 172.21.15.94:0/2696602030 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac032be0 con 0x7f02bc0c1320 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02bc1e9aa0 con 0x7f02bc0c1320 2024-03-06T07:31:02.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.201+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc153030 con 0x7f02bc0c1320 2024-03-06T07:31:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.202+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02bc04f330 con 0x7f02bc0c1320 2024-03-06T07:31:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.203+0000 7f02b97fa640 1 -- 172.21.15.94:0/2696602030 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02ac04e430 con 0x7f02bc0c1320 2024-03-06T07:31:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.203+0000 7f02b97fa640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f02a806b500 0x7f02a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.203+0000 7f02b97fa640 1 -- 172.21.15.94:0/2696602030 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f02ac02f030 con 0x7f02bc0c1320 2024-03-06T07:31:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.203+0000 7f02bbfff640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f02a806b500 0x7f02a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.208 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.204+0000 7f02bbfff640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f02a806b500 0x7f02a806d9c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f02bc06d4e0 tx=0x7f02b0007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.207+0000 7f02b97fa640 1 -- 172.21.15.94:0/2696602030 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02bc1511b0 con 0x7f02bc0c1320 2024-03-06T07:31:02.417 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.416+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02bc0c3a40 con 0x7f02bc0c1320 2024-03-06T07:31:02.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.416+0000 7f02b97fa640 1 -- 172.21.15.94:0/2696602030 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f02ac040070 con 0x7f02bc0c1320 2024-03-06T07:31:02.418 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:02.418 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:31:02.418 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:31:02.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f02a806b500 msgr2=0x7f02a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f02a806b500 0x7f02a806d9c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f02bc06d4e0 tx=0x7f02b0007e80 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc0c1320 msgr2=0x7f02bc149ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc0c1320 0x7f02bc149ca0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f02ac034040 tx=0x7f02ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 shutdown_connections 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f02a806b500 0x7f02a806d9c0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02bc150580 0x7f02bc0729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02bc0c6000 0x7f02bc070770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 --2- 172.21.15.94:0/2696602030 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02bc0c1320 0x7f02bc149ca0 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.418+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 >> 172.21.15.94:0/2696602030 conn(0x7f02bc0c6710 msgr2=0x7f02bc14c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.419+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 shutdown_connections 2024-03-06T07:31:02.421 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:02.419+0000 7f02c2c5c640 1 -- 172.21.15.94:0/2696602030 wait complete. 2024-03-06T07:31:02.834 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:17.753463+0000', 'modified': '2024-03-06T07:29:12.259362+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 213, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 38273, 'mds_1': 10773}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10773': {'gid': 10773, 'name': 'a', 'rank': 1, 'incarnation': 207, 'state': 'up:stopping', 'state_seq': 25, 'addr': '172.21.15.94:6833/341111452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.94:6832', 'nonce': 341111452}, {'type': 'v1', 'addr': '172.21.15.94:6833', 'nonce': 341111452}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:12.259362+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_38273': {'gid': 38273, 'name': 'b', 'rank': 0, 'incarnation': 194, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6836/2585066812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2585066812}, {'type': 'v1', 'addr': '172.21.15.145:6836', 'nonce': 2585066812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 38273, 'qdb_cluster': [38273]} 2024-03-06T07:31:02.834 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10773:up:stopping 2024-03-06T07:31:03.835 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 16, "epoch": 213, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 16, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:27:17.753463+0000", "damaged": [], "data_pools": [ 34 ], "enabled": true, "epoch": 213, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_10773": { "addr": "172.21.15.94:6833/341111452", "addrs": { "addrvec": [ { "addr": "172.21.15.94:6832", "nonce": 341111452, "type": "v2" }, { "addr": "172.21.15.94:6833", "nonce": 341111452, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 10773, "incarnation": 207, "join_fscid": -1, "laggy_since": "2024-03-06T07:29:12.259362+0000", "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 25 }, "gid_38273": { "addr": "172.21.15.145:6836/2585066812", "addrs": { "addrvec": [ { "addr": "172.21.15.145:6834", "nonce": 2585066812, "type": "v2" }, { "addr": "172.21.15.145:6836", "nonce": 2585066812, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 38273, "incarnation": 194, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 213, "max_file_size": 1099511627776, "max_mds": 0, "max_xattr_size": 65536, "metadata_pool": 33, "modified": "2024-03-06T07:29:12.259362+0000", "qdb_cluster": [ 38273 ], "qdb_leader": 38273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 38273, "mds_1": 10773 } } } ], "standbys": [] } 2024-03-06T07:31:03.836 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:03.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:03.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:03.919 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:03.920 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:31:03.920 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:03.938 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:31:03.938 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:31:03.938 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:03.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:31:03.976 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:04.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.082+0000 7f8a5bcdf640 1 Processor -- start 2024-03-06T07:31:04.084 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.082+0000 7f8a5bcdf640 1 -- start start 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5bcdf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a54155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5bcdf640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8a54158b60 0x7f8a54158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5bcdf640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540c73d0 con 0x7f8a540c88c0 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5bcdf640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a540c8dd0 con 0x7f8a54158b60 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5bcdf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540be2e0 con 0x7f8a5414f360 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5a27c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8a54158b60 0x7f8a54158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a59a7b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a54155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5927a640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f8a540c88c0 0x7f8a540c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:46308/0 (socket says 172.21.15.94:46308) 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5927a640 1 -- 172.21.15.94:0/250344972 learned_addr learned my addr 172.21.15.94:0/250344972 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.083+0000 7f8a5a27c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8a54158b60 0x7f8a54158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.94:38830/0 (socket says 172.21.15.94:38830) 2024-03-06T07:31:04.086 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.084+0000 7f8a5a27c640 1 -- 172.21.15.94:0/250344972 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 msgr2=0x7f8a54155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.084+0000 7f8a5a27c640 1 --2- 172.21.15.94:0/250344972 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a54155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.084+0000 7f8a5a27c640 1 -- 172.21.15.94:0/250344972 >> v1:172.21.15.145:6789/0 conn(0x7f8a540c88c0 legacy=0x7f8a540c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.084+0000 7f8a5a27c640 1 -- 172.21.15.94:0/250344972 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540c7670 con 0x7f8a54158b60 2024-03-06T07:31:04.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.084+0000 7f8a58a79640 1 -- 172.21.15.94:0/250344972 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991862137 0 0) 0x7f8a540c73d0 con 0x7f8a540c88c0 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5a27c640 1 --2- 172.21.15.94:0/250344972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a54158ff0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0097a0 tx=0x7f8a4c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bdfb5de08bb8a6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a58a79640 1 -- 172.21.15.94:0/250344972 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0315a0 con 0x7f8a54158b60 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a58a79640 1 -- 172.21.15.94:0/250344972 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c037070 con 0x7f8a54158b60 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a58a79640 1 -- 172.21.15.94:0/250344972 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c031960 con 0x7f8a54158b60 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/250344972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 msgr2=0x7f8a54158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/250344972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a54158ff0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0097a0 tx=0x7f8a4c02b840 comp rx=0 tx=0).stop 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/250344972 shutdown_connections 2024-03-06T07:31:04.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/250344972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a54158ff0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/250344972 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a54155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/250344972 >> 172.21.15.94:0/250344972 conn(0x7f8a540ac570 msgr2=0x7f8a540ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/250344972 shutdown_connections 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.085+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/250344972 wait complete. 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.086+0000 7f8a5bcdf640 1 Processor -- start 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.086+0000 7f8a5bcdf640 1 -- start start 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5bcdf640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a540c88c0 0x7f8a540c3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5bcdf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a540c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5bcdf640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a540c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5bcdf640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c5a50 con 0x7f8a540c88c0 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5bcdf640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c7c30 con 0x7f8a54158b60 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5bcdf640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c5bd0 con 0x7f8a5414f360 2024-03-06T07:31:04.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a59a7b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a540c88c0 0x7f8a540c3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5a27c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a540c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a59a7b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a540c88c0 0x7f8a540c3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38288/0 (socket says 172.21.15.94:38288) 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a5927a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a540c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.087+0000 7f8a59a7b640 1 -- 172.21.15.94:0/869940020 learned_addr learned my addr 172.21.15.94:0/869940020 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5a27c640 1 -- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 msgr2=0x7f8a540c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5a27c640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a540c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5a27c640 1 -- 172.21.15.94:0/869940020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a540c88c0 msgr2=0x7f8a540c3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5a27c640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a540c88c0 0x7f8a540c3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5a27c640 1 -- 172.21.15.94:0/869940020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a541eee80 con 0x7f8a54158b60 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a59a7b640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a540c88c0 0x7f8a540c3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5a27c640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a540c2ee0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c033000 tx=0x7f8a4c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a4a7fc640 1 -- 172.21.15.94:0/869940020 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c004230 con 0x7f8a54158b60 2024-03-06T07:31:04.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a4a7fc640 1 -- 172.21.15.94:0/869940020 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c0043d0 con 0x7f8a54158b60 2024-03-06T07:31:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a4a7fc640 1 -- 172.21.15.94:0/869940020 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c003850 con 0x7f8a54158b60 2024-03-06T07:31:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a541ef160 con 0x7f8a54158b60 2024-03-06T07:31:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.088+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a540b0170 con 0x7f8a54158b60 2024-03-06T07:31:04.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.089+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5404f330 con 0x7f8a54158b60 2024-03-06T07:31:04.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.089+0000 7f8a4a7fc640 1 -- 172.21.15.94:0/869940020 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a4c0039f0 con 0x7f8a54158b60 2024-03-06T07:31:04.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.090+0000 7f8a4a7fc640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a3806b4d0 0x7f8a3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.090+0000 7f8a4a7fc640 1 -- 172.21.15.94:0/869940020 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8a4c02f030 con 0x7f8a54158b60 2024-03-06T07:31:04.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.090+0000 7f8a59a7b640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a3806b4d0 0x7f8a3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.091+0000 7f8a59a7b640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a3806b4d0 0x7f8a3806d990 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009a40 tx=0x7f8a5000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.094+0000 7f8a4a7fc640 1 -- 172.21.15.94:0/869940020 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a4c041070 con 0x7f8a54158b60 2024-03-06T07:31:04.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.301+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a540b03b0 con 0x7f8a54158b60 2024-03-06T07:31:04.303 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.302+0000 7f8a4a7fc640 1 -- 172.21.15.94:0/869940020 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f8a4c07abf0 con 0x7f8a54158b60 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:31:04.304 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:31:04.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a3806b4d0 msgr2=0x7f8a3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a3806b4d0 0x7f8a3806d990 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009a40 tx=0x7f8a5000b040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 msgr2=0x7f8a540c2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a540c2ee0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c033000 tx=0x7f8a4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 shutdown_connections 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8a3806b4d0 0x7f8a3806d990 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a54158b60 0x7f8a540c2ee0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a5414f360 0x7f8a540c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 --2- 172.21.15.94:0/869940020 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8a540c88c0 0x7f8a540c3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 >> 172.21.15.94:0/869940020 conn(0x7f8a540ac570 msgr2=0x7f8a5414e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 shutdown_connections 2024-03-06T07:31:04.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.304+0000 7f8a5bcdf640 1 -- 172.21.15.94:0/869940020 wait complete. 2024-03-06T07:31:04.317 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:04.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 Processor -- start 2024-03-06T07:31:04.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 -- start start 2024-03-06T07:31:04.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f41ec14f790 0x7f41ec1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c1aa0 0x7f41ec0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.418 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41ec0c5c60 0x7f41ec155a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ec0c2680 con 0x7f41ec0c5c60 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ec0c28d0 con 0x7f41ec14f790 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.415+0000 7f41f22a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec0c2bb0 con 0x7f41ec0c1aa0 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41ec0c5c60 0x7f41ec155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f41ec0c5c60 0x7f41ec155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38296/0 (socket says 172.21.15.94:38296) 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 -- 172.21.15.94:0/980164423 learned_addr learned my addr 172.21.15.94:0/980164423 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41ebfff640 1 --2- 172.21.15.94:0/980164423 >> v2:172.21.15.145:3300/0 conn(0x7f41ec14f790 0x7f41ec1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41eb7fe640 1 --2- 172.21.15.94:0/980164423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c1aa0 0x7f41ec0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 -- 172.21.15.94:0/980164423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c1aa0 msgr2=0x7f41ec0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 --2- 172.21.15.94:0/980164423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c1aa0 0x7f41ec0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 -- 172.21.15.94:0/980164423 >> v2:172.21.15.145:3300/0 conn(0x7f41ec14f790 msgr2=0x7f41ec1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 --2- 172.21.15.94:0/980164423 >> v2:172.21.15.145:3300/0 conn(0x7f41ec14f790 0x7f41ec1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.419 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.416+0000 7f41f0841640 1 -- 172.21.15.94:0/980164423 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec0c2e90 con 0x7f41ec0c5c60 2024-03-06T07:31:04.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f0841640 1 --2- 172.21.15.94:0/980164423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec0c5c60 0x7f41ec155a40 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0c3170 tx=0x7f41e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441572bba7ea6922 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41eaffd640 1 -- 172.21.15.94:0/980164423 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e402ca70 con 0x7f41ec0c5c60 2024-03-06T07:31:04.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41eaffd640 1 -- 172.21.15.94:0/980164423 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e402cc10 con 0x7f41ec0c5c60 2024-03-06T07:31:04.420 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41eaffd640 1 -- 172.21.15.94:0/980164423 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e40343f0 con 0x7f41ec0c5c60 2024-03-06T07:31:04.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 -- 172.21.15.94:0/980164423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec0c5c60 msgr2=0x7f41ec155a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 --2- 172.21.15.94:0/980164423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec0c5c60 0x7f41ec155a40 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0c3170 tx=0x7f41e402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 -- 172.21.15.94:0/980164423 shutdown_connections 2024-03-06T07:31:04.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 --2- 172.21.15.94:0/980164423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec0c5c60 0x7f41ec155a40 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 --2- 172.21.15.94:0/980164423 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c1aa0 0x7f41ec0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 --2- 172.21.15.94:0/980164423 >> v2:172.21.15.145:3300/0 conn(0x7f41ec14f790 0x7f41ec1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 -- 172.21.15.94:0/980164423 >> 172.21.15.94:0/980164423 conn(0x7f41ec0c6710 msgr2=0x7f41ec0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 -- 172.21.15.94:0/980164423 shutdown_connections 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.417+0000 7f41f22a4640 1 -- 172.21.15.94:0/980164423 wait complete. 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.418+0000 7f41f22a4640 1 Processor -- start 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f22a4640 1 -- start start 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f22a4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41ec0c1aa0 0x7f41ec1dfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f22a4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c5c60 0x7f41ec0bf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f22a4640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec14f790 0x7f41ec071340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f22a4640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec06cc00 con 0x7f41ec14f790 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f22a4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec0c0d60 con 0x7f41ec0c1aa0 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f22a4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec06cd80 con 0x7f41ec0c5c60 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f0841640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec14f790 0x7f41ec071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.424 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f0841640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec14f790 0x7f41ec071340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38308/0 (socket says 172.21.15.94:38308) 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41f0841640 1 -- 172.21.15.94:0/1813713027 learned_addr learned my addr 172.21.15.94:0/1813713027 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41ebfff640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41ec0c1aa0 0x7f41ec1dfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.419+0000 7f41eb7fe640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c5c60 0x7f41ec0bf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f0841640 1 -- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c5c60 msgr2=0x7f41ec0bf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f0841640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c5c60 0x7f41ec0bf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f0841640 1 -- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41ec0c1aa0 msgr2=0x7f41ec1dfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f0841640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41ec0c1aa0 0x7f41ec1dfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f0841640 1 -- 172.21.15.94:0/1813713027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec1ee370 con 0x7f41ec14f790 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f0841640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec14f790 0x7f41ec071340 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f41e402c4e0 tx=0x7f41e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41e8ff9640 1 -- 172.21.15.94:0/1813713027 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4043020 con 0x7f41ec14f790 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41e8ff9640 1 -- 172.21.15.94:0/1813713027 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e403e450 con 0x7f41ec14f790 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41e8ff9640 1 -- 172.21.15.94:0/1813713027 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4009510 con 0x7f41ec14f790 2024-03-06T07:31:04.425 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec1ee650 con 0x7f41ec14f790 2024-03-06T07:31:04.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.420+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec1eec10 con 0x7f41ec14f790 2024-03-06T07:31:04.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.421+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec0acab0 con 0x7f41ec14f790 2024-03-06T07:31:04.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.422+0000 7f41e8ff9640 1 -- 172.21.15.94:0/1813713027 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41e4046020 con 0x7f41ec14f790 2024-03-06T07:31:04.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.422+0000 7f41e8ff9640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d806b4d0 0x7f41d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.422+0000 7f41e8ff9640 1 -- 172.21.15.94:0/1813713027 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f41e40ac9e0 con 0x7f41ec14f790 2024-03-06T07:31:04.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.422+0000 7f41ebfff640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d806b4d0 0x7f41d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.423+0000 7f41ebfff640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d806b4d0 0x7f41d806d990 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009aa0 tx=0x7f41dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.430 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.428+0000 7f41e8ff9640 1 -- 172.21.15.94:0/1813713027 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41ec0acab0 con 0x7f41ec14f790 2024-03-06T07:31:04.601 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.599+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41ec04f330 con 0x7f41ec14f790 2024-03-06T07:31:04.602 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.600+0000 7f41e8ff9640 1 -- 172.21.15.94:0/1813713027 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+17288 (secure 0 0 0) 0x7f41e404d030 con 0x7f41ec14f790 2024-03-06T07:31:04.603 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:04.603 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:28:37.254511+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:15.999555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:16.719665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.2","mappings":[{"from":0,"to":2}]},{"pgid":"34.13","mappings":[{"from":0,"to":2}]},{"pgid":"34.31","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6837/679350833":"2024-03-07T07:28:02.249189+0000","172.21.15.94:6834/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6835/679350833":"2024-03-07T07:28:02.249189+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:04.604 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d806b4d0 msgr2=0x7f41d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d806b4d0 0x7f41d806d990 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009aa0 tx=0x7f41dc009190 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec14f790 msgr2=0x7f41ec071340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec14f790 0x7f41ec071340 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f41e402c4e0 tx=0x7f41e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 shutdown_connections 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f41d806b4d0 0x7f41d806d990 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f41ec14f790 0x7f41ec071340 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41ec0c5c60 0x7f41ec0bf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 --2- 172.21.15.94:0/1813713027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41ec0c1aa0 0x7f41ec1dfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 >> 172.21.15.94:0/1813713027 conn(0x7f41ec0c6710 msgr2=0x7f41ec0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 shutdown_connections 2024-03-06T07:31:04.605 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.602+0000 7f41f22a4640 1 -- 172.21.15.94:0/1813713027 wait complete. 2024-03-06T07:31:04.616 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:18:02.678312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:15.999555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:16.719665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:04.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:04.718 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.717+0000 7f514a89d640 1 Processor -- start 2024-03-06T07:31:04.719 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.717+0000 7f514a89d640 1 -- start start 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.717+0000 7f514a89d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51440b0070 0x7f5144155c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.717+0000 7f514a89d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51440b0450 con 0x7f514406ca70 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.717+0000 7f514a89d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514406cf20 con 0x7f51440c5380 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f514a89d640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514406c020 con 0x7f51440b0070 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f5143fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51440b0070 0x7f5144155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f51437fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f514406ca70 0x7f51440c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45728/0 (socket says 172.21.15.94:45728) 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f5143fff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f51440b0070 0x7f5144155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38322/0 (socket says 172.21.15.94:38322) 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f51437fe640 1 -- 172.21.15.94:0/3229088376 learned_addr learned my addr 172.21.15.94:0/3229088376 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f5143fff640 1 -- 172.21.15.94:0/3229088376 >> v1:172.21.15.145:6789/0 conn(0x7f51440c5380 legacy=0x7f514414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f5143fff640 1 -- 172.21.15.94:0/3229088376 >> v1:172.21.15.94:6789/0 conn(0x7f514406ca70 legacy=0x7f51440c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.720 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.718+0000 7f5143fff640 1 -- 172.21.15.94:0/3229088376 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51440c77f0 con 0x7f51440b0070 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f5143fff640 1 --2- 172.21.15.94:0/3229088376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440b0070 0x7f5144155c30 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f51340097a0 tx=0x7f513402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4927b39b7663cf16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f5142ffd640 1 -- 172.21.15.94:0/3229088376 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134035020 con 0x7f51440b0070 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f5142ffd640 1 -- 172.21.15.94:0/3229088376 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5134031830 con 0x7f51440b0070 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f5142ffd640 1 -- 172.21.15.94:0/3229088376 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134031c00 con 0x7f51440b0070 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f514a89d640 1 -- 172.21.15.94:0/3229088376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440b0070 msgr2=0x7f5144155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f514a89d640 1 --2- 172.21.15.94:0/3229088376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440b0070 0x7f5144155c30 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f51340097a0 tx=0x7f513402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f514a89d640 1 -- 172.21.15.94:0/3229088376 shutdown_connections 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f514a89d640 1 --2- 172.21.15.94:0/3229088376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440b0070 0x7f5144155c30 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f514a89d640 1 -- 172.21.15.94:0/3229088376 >> 172.21.15.94:0/3229088376 conn(0x7f51440ac590 msgr2=0x7f51440ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f514a89d640 1 -- 172.21.15.94:0/3229088376 shutdown_connections 2024-03-06T07:31:04.722 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.719+0000 7f514a89d640 1 -- 172.21.15.94:0/3229088376 wait complete. 2024-03-06T07:31:04.724 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.720+0000 7f514a89d640 1 Processor -- start 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f514a89d640 1 -- start start 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f514a89d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f514406ca70 0x7f514414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f514a89d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51440b0070 0x7f51440c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f514a89d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440c5380 0x7f5144154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f514a89d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514406c550 con 0x7f51440c5380 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f514a89d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514406c270 con 0x7f514406ca70 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f514a89d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514406da80 con 0x7f51440b0070 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f5148e3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440c5380 0x7f5144154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f5148e3a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440c5380 0x7f5144154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38330/0 (socket says 172.21.15.94:38330) 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f5143fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f514406ca70 0x7f514414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.725 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f5148e3a640 1 -- 172.21.15.94:0/3027171906 learned_addr learned my addr 172.21.15.94:0/3027171906 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.721+0000 7f51437fe640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51440b0070 0x7f51440c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5148e3a640 1 -- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51440b0070 msgr2=0x7f51440c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5148e3a640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51440b0070 0x7f51440c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5148e3a640 1 -- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f514406ca70 msgr2=0x7f514414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5148e3a640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f514406ca70 0x7f514414b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5148e3a640 1 -- 172.21.15.94:0/3027171906 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5144158080 con 0x7f51440c5380 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5148e3a640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440c5380 0x7f5144154220 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f513c00c7a0 tx=0x7f513c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5140ff9640 1 -- 172.21.15.94:0/3027171906 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c003b70 con 0x7f51440c5380 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5140ff9640 1 -- 172.21.15.94:0/3027171906 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c003d30 con 0x7f51440c5380 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5144158360 con 0x7f51440c5380 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f5140ff9640 1 -- 172.21.15.94:0/3027171906 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c00aa90 con 0x7f51440c5380 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.722+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51441ee360 con 0x7f51440c5380 2024-03-06T07:31:04.726 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.723+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51441ee560 con 0x7f51440c5380 2024-03-06T07:31:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.723+0000 7f5140ff9640 1 -- 172.21.15.94:0/3027171906 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f513c00ac30 con 0x7f51440c5380 2024-03-06T07:31:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.724+0000 7f5140ff9640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f513006b4d0 0x7f513006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.724+0000 7f5140ff9640 1 -- 172.21.15.94:0/3027171906 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f513c08a290 con 0x7f51440c5380 2024-03-06T07:31:04.727 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.724+0000 7f5143fff640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f513006b4d0 0x7f513006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.728 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.725+0000 7f5143fff640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f513006b4d0 0x7f513006d990 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f5134036000 tx=0x7f5134008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.730 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.728+0000 7f5140ff9640 1 -- 172.21.15.94:0/3027171906 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51441ee560 con 0x7f51440c5380 2024-03-06T07:31:04.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.899+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f514404f330 con 0x7f51440c5380 2024-03-06T07:31:04.901 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.899+0000 7f5140ff9640 1 -- 172.21.15.94:0/3027171906 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v213) v1 ==== 93+0+10 (secure 0 0 0) 0x7f513c058830 con 0x7f51440c5380 2024-03-06T07:31:04.901 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2024-03-06T07:31:04.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f513006b4d0 msgr2=0x7f513006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f513006b4d0 0x7f513006d990 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f5134036000 tx=0x7f5134008030 comp rx=0 tx=0).stop 2024-03-06T07:31:04.903 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440c5380 msgr2=0x7f5144154220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440c5380 0x7f5144154220 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f513c00c7a0 tx=0x7f513c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 shutdown_connections 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f513006b4d0 0x7f513006d990 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51440c5380 0x7f5144154220 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51440b0070 0x7f51440c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 --2- 172.21.15.94:0/3027171906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f514406ca70 0x7f514414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 >> 172.21.15.94:0/3027171906 conn(0x7f51440ac590 msgr2=0x7f51441513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 shutdown_connections 2024-03-06T07:31:04.904 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:04.901+0000 7f514a89d640 1 -- 172.21.15.94:0/3027171906 wait complete. 2024-03-06T07:31:04.914 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:05.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.014+0000 7f478e31e640 1 Processor -- start 2024-03-06T07:31:05.016 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.014+0000 7f478e31e640 1 -- start start 2024-03-06T07:31:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f478e31e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47880ab310 0x7f47880ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f478e31e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f47881538c0 0x7f47881494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f478e31e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4788154330 con 0x7f47880a97c0 2024-03-06T07:31:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f478e31e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880c46a0 con 0x7f47881538c0 2024-03-06T07:31:05.018 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f478e31e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4788153dc0 con 0x7f47880ab310 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f478c8bb640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f47880a97c0 0x7f47880c4b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45742/0 (socket says 172.21.15.94:45742) 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f478c8bb640 1 -- 172.21.15.94:0/1283402538 learned_addr learned my addr 172.21.15.94:0/1283402538 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f47877fe640 1 --2- 172.21.15.94:0/1283402538 >> v2:172.21.15.145:3300/0 conn(0x7f47881538c0 0x7f47881494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.015+0000 7f4787fff640 1 --2- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47880ab310 0x7f47880ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.016+0000 7f4786ffd640 1 -- 172.21.15.94:0/1283402538 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665367134 0 0) 0x7f4788154330 con 0x7f47880a97c0 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.016+0000 7f4786ffd640 1 -- 172.21.15.94:0/1283402538 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4770003520 con 0x7f47880a97c0 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.016+0000 7f47877fe640 1 -- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47880ab310 msgr2=0x7f47880ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.016+0000 7f47877fe640 1 --2- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47880ab310 0x7f47880ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.016+0000 7f47877fe640 1 -- 172.21.15.94:0/1283402538 >> v1:172.21.15.94:6789/0 conn(0x7f47880a97c0 legacy=0x7f47880c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.016+0000 7f47877fe640 1 -- 172.21.15.94:0/1283402538 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880c5d30 con 0x7f47881538c0 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.016+0000 7f47877fe640 1 --2- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47881538c0 0x7f47881494a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f47880c4820 tx=0x7f477c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=152ba40a738f21e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.019 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f4786ffd640 1 -- 172.21.15.94:0/1283402538 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c031770 con 0x7f47881538c0 2024-03-06T07:31:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f4786ffd640 1 -- 172.21.15.94:0/1283402538 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477c037070 con 0x7f47881538c0 2024-03-06T07:31:05.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f4786ffd640 1 -- 172.21.15.94:0/1283402538 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477c031b30 con 0x7f47881538c0 2024-03-06T07:31:05.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 -- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47881538c0 msgr2=0x7f47881494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 --2- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47881538c0 0x7f47881494a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f47880c4820 tx=0x7f477c02b810 comp rx=0 tx=0).stop 2024-03-06T07:31:05.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 -- 172.21.15.94:0/1283402538 shutdown_connections 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 --2- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47881538c0 0x7f47881494a0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 --2- 172.21.15.94:0/1283402538 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47880ab310 0x7f47880ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 -- 172.21.15.94:0/1283402538 >> 172.21.15.94:0/1283402538 conn(0x7f47880c6710 msgr2=0x7f47880c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 -- 172.21.15.94:0/1283402538 shutdown_connections 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.017+0000 7f478e31e640 1 -- 172.21.15.94:0/1283402538 wait complete. 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.018+0000 7f478e31e640 1 Processor -- start 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.018+0000 7f478e31e640 1 -- start start 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f478e31e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47880a97c0 0x7f4788072790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f478e31e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47880ab310 0x7f478814dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f478e31e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47881538c0 0x7f478814e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f478e31e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880abc30 con 0x7f47880a97c0 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f478e31e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880c62f0 con 0x7f47880ab310 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f478e31e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880c5240 con 0x7f47881538c0 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47880a97c0 0x7f4788072790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47880a97c0 0x7f4788072790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38342/0 (socket says 172.21.15.94:38342) 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 -- 172.21.15.94:0/3645957890 learned_addr learned my addr 172.21.15.94:0/3645957890 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f478c8bb640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47881538c0 0x7f478814e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f47877fe640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47880ab310 0x7f478814dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 -- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47881538c0 msgr2=0x7f478814e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47881538c0 0x7f478814e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 -- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47880ab310 msgr2=0x7f478814dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47880ab310 0x7f478814dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.019+0000 7f4787fff640 1 -- 172.21.15.94:0/3645957890 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47881e96e0 con 0x7f47880a97c0 2024-03-06T07:31:05.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.020+0000 7f4787fff640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47880a97c0 0x7f4788072790 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f477800e9b0 tx=0x7f477800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.020+0000 7f4784ff9640 1 -- 172.21.15.94:0/3645957890 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4778017070 con 0x7f47880a97c0 2024-03-06T07:31:05.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.020+0000 7f4784ff9640 1 -- 172.21.15.94:0/3645957890 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477800c9a0 con 0x7f47880a97c0 2024-03-06T07:31:05.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.020+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47881e9980 con 0x7f47880a97c0 2024-03-06T07:31:05.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.020+0000 7f4784ff9640 1 -- 172.21.15.94:0/3645957890 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477800cd90 con 0x7f47880a97c0 2024-03-06T07:31:05.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.020+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478814d7e0 con 0x7f47880a97c0 2024-03-06T07:31:05.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.021+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4788075300 con 0x7f47880a97c0 2024-03-06T07:31:05.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.021+0000 7f4784ff9640 1 -- 172.21.15.94:0/3645957890 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f477801a070 con 0x7f47880a97c0 2024-03-06T07:31:05.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.021+0000 7f4784ff9640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f477406b4d0 0x7f477406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.022+0000 7f4784ff9640 1 -- 172.21.15.94:0/3645957890 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f477808a380 con 0x7f47880a97c0 2024-03-06T07:31:05.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.022+0000 7f47877fe640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f477406b4d0 0x7f477406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.026 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.022+0000 7f47877fe640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f477406b4d0 0x7f477406d990 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f477c036000 tx=0x7f477c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.026+0000 7f4784ff9640 1 -- 172.21.15.94:0/3645957890 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4788075300 con 0x7f47880a97c0 2024-03-06T07:31:05.195 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.194+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f47880a95b0 con 0x7f47880a97c0 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.194+0000 7f4784ff9640 1 -- 172.21.15.94:0/3645957890 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v213) v1 ==== 104+0+11 (secure 0 0 0) 0x7f47880a95b0 con 0x7f47880a97c0 2024-03-06T07:31:05.196 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 32 2024-03-06T07:31:05.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f477406b4d0 msgr2=0x7f477406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f477406b4d0 0x7f477406d990 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f477c036000 tx=0x7f477c03d040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47880a97c0 msgr2=0x7f4788072790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47880a97c0 0x7f4788072790 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f477800e9b0 tx=0x7f477800ee70 comp rx=0 tx=0).stop 2024-03-06T07:31:05.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 shutdown_connections 2024-03-06T07:31:05.198 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f477406b4d0 0x7f477406d990 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47881538c0 0x7f478814e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47880ab310 0x7f478814dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 --2- 172.21.15.94:0/3645957890 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47880a97c0 0x7f4788072790 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 >> 172.21.15.94:0/3645957890 conn(0x7f47880c6710 msgr2=0x7f47880c4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 shutdown_connections 2024-03-06T07:31:05.199 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.196+0000 7f478e31e640 1 -- 172.21.15.94:0/3645957890 wait complete. 2024-03-06T07:31:05.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.309+0000 7f55e812c640 1 Processor -- start 2024-03-06T07:31:05.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.309+0000 7f55e812c640 1 -- start start 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e812c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55e00c5990 0x7f55e00c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e812c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e00c3ef0 con 0x7f55e00ab170 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e812c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e01556d0 con 0x7f55e006c900 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e812c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e01559b0 con 0x7f55e00c5990 2024-03-06T07:31:05.312 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e5ec8640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55e00c5990 0x7f55e00c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e56c7640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f55e00ab170 0x7f55e006c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45756/0 (socket says 172.21.15.94:45756) 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e56c7640 1 -- 172.21.15.94:0/739447817 learned_addr learned my addr 172.21.15.94:0/739447817 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e5ec8640 1 -- 172.21.15.94:0/739447817 >> v1:172.21.15.145:6789/0 conn(0x7f55e006c900 legacy=0x7f55e00a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e5ec8640 1 -- 172.21.15.94:0/739447817 >> v1:172.21.15.94:6789/0 conn(0x7f55e00ab170 legacy=0x7f55e006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e4ec6640 1 -- 172.21.15.94:0/739447817 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825177331 0 0) 0x7f55e00c3ef0 con 0x7f55e00ab170 2024-03-06T07:31:05.313 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.310+0000 7f55e5ec8640 1 -- 172.21.15.94:0/739447817 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0155c90 con 0x7f55e00c5990 2024-03-06T07:31:05.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.311+0000 7f55e5ec8640 1 --2- 172.21.15.94:0/739447817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e00c0ab0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0097d0 tx=0x7f55dc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b4ff2902e68781 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.314 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.311+0000 7f55e4ec6640 1 -- 172.21.15.94:0/739447817 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc034070 con 0x7f55e00c5990 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.311+0000 7f55e4ec6640 1 -- 172.21.15.94:0/739447817 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc030610 con 0x7f55e00c5990 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.311+0000 7f55e4ec6640 1 -- 172.21.15.94:0/739447817 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc0309e0 con 0x7f55e00c5990 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.312+0000 7f55e812c640 1 -- 172.21.15.94:0/739447817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 msgr2=0x7f55e00c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.312+0000 7f55e812c640 1 --2- 172.21.15.94:0/739447817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e00c0ab0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0097d0 tx=0x7f55dc02b960 comp rx=0 tx=0).stop 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.312+0000 7f55e812c640 1 -- 172.21.15.94:0/739447817 shutdown_connections 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.312+0000 7f55e812c640 1 --2- 172.21.15.94:0/739447817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e00c0ab0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.312+0000 7f55e812c640 1 -- 172.21.15.94:0/739447817 >> 172.21.15.94:0/739447817 conn(0x7f55e00c6710 msgr2=0x7f55e00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.312+0000 7f55e812c640 1 -- 172.21.15.94:0/739447817 shutdown_connections 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.312+0000 7f55e812c640 1 -- 172.21.15.94:0/739447817 wait complete. 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.313+0000 7f55e812c640 1 Processor -- start 2024-03-06T07:31:05.315 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.313+0000 7f55e812c640 1 -- start start 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55e006c900 0x7f55e00a7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55e00ab170 0x7f55e01ec160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e01ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e0156610 con 0x7f55e00c5990 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e0156490 con 0x7f55e006c900 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e00c3af0 con 0x7f55e00ab170 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e01ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e01ec6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38360/0 (socket says 172.21.15.94:38360) 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 -- 172.21.15.94:0/451510732 learned_addr learned my addr 172.21.15.94:0/451510732 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e5ec8640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55e006c900 0x7f55e00a7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e56c7640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55e00ab170 0x7f55e01ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.318 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 -- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55e00ab170 msgr2=0x7f55e01ec160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55e00ab170 0x7f55e01ec160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 -- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55e006c900 msgr2=0x7f55e00a7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55e006c900 0x7f55e00a7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 -- 172.21.15.94:0/451510732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0148c50 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e66c9640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e01ec6a0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f55d800c7d0 tx=0x7f55d800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55d67fc640 1 -- 172.21.15.94:0/451510732 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d8003b70 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55d67fc640 1 -- 172.21.15.94:0/451510732 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d8003d30 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e0148970 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55d67fc640 1 -- 172.21.15.94:0/451510732 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d800aa90 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.314+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e00a77b0 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.315+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e004f330 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.316+0000 7f55d67fc640 1 -- 172.21.15.94:0/451510732 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55d800ac30 con 0x7f55e00c5990 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.316+0000 7f55d67fc640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55c406b4d0 0x7f55c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.316+0000 7f55e5ec8640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55c406b4d0 0x7f55c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.319 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.316+0000 7f55d67fc640 1 -- 172.21.15.94:0/451510732 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f55d808a1d0 con 0x7f55e00c5990 2024-03-06T07:31:05.320 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.317+0000 7f55e5ec8640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55c406b4d0 0x7f55c406d990 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f55dc036000 tx=0x7f55dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.322 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.321+0000 7f55d67fc640 1 -- 172.21.15.94:0/451510732 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55d8058800 con 0x7f55e00c5990 2024-03-06T07:31:05.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.496+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55e00c1740 con 0x7f55e00c5990 2024-03-06T07:31:05.498 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.496+0000 7f55d67fc640 1 -- 172.21.15.94:0/451510732 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v213) v1 ==== 100+0+11 (secure 0 0 0) 0x7f55d805c8a0 con 0x7f55e00c5990 2024-03-06T07:31:05.498 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 64 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.498+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55c406b4d0 msgr2=0x7f55c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.498+0000 7f55e812c640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55c406b4d0 0x7f55c406d990 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f55dc036000 tx=0x7f55dc008030 comp rx=0 tx=0).stop 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.498+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 msgr2=0x7f55e01ec6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.498+0000 7f55e812c640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e01ec6a0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f55d800c7d0 tx=0x7f55d800cc90 comp rx=0 tx=0).stop 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 shutdown_connections 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f55c406b4d0 0x7f55c406d990 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55e00c5990 0x7f55e01ec6a0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55e00ab170 0x7f55e01ec160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 --2- 172.21.15.94:0/451510732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55e006c900 0x7f55e00a7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 >> 172.21.15.94:0/451510732 conn(0x7f55e00c6710 msgr2=0x7f55e006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.501 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 shutdown_connections 2024-03-06T07:31:05.502 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.499+0000 7f55e812c640 1 -- 172.21.15.94:0/451510732 wait complete. 2024-03-06T07:31:05.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.609+0000 7fcfa3f56640 1 Processor -- start 2024-03-06T07:31:05.611 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.609+0000 7fcfa3f56640 1 -- start start 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa3f56640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcf9c0c1aa0 0x7fcf9c1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa3f56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c155a40 0x7fcf9c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa3f56640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c1563d0 con 0x7fcf9c14f790 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa3f56640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c2680 con 0x7fcf9c0c1aa0 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa3f56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c28d0 con 0x7fcf9c155a40 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa14f1640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcf9c0c1aa0 0x7fcf9c1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa1cf2640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcf9c14f790 0x7fcf9c0c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45772/0 (socket says 172.21.15.94:45772) 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa1cf2640 1 -- 172.21.15.94:0/1187747102 learned_addr learned my addr 172.21.15.94:0/1187747102 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa24f3640 1 --2- 172.21.15.94:0/1187747102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c155a40 0x7fcf9c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa14f1640 1 -- 172.21.15.94:0/1187747102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c155a40 msgr2=0x7fcf9c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.613 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa14f1640 1 --2- 172.21.15.94:0/1187747102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c155a40 0x7fcf9c155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa14f1640 1 -- 172.21.15.94:0/1187747102 >> v1:172.21.15.94:6789/0 conn(0x7fcf9c14f790 legacy=0x7fcf9c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa14f1640 1 -- 172.21.15.94:0/1187747102 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0c2bb0 con 0x7fcf9c0c1aa0 2024-03-06T07:31:05.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.610+0000 7fcfa0cf0640 1 -- 172.21.15.94:0/1187747102 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344847866 0 0) 0x7fcf9c1563d0 con 0x7fcf9c14f790 2024-03-06T07:31:05.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa14f1640 1 --2- 172.21.15.94:0/1187747102 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c0c1aa0 0x7fcf9c1529d0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0097a0 tx=0x7fcf8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fcaa6a66410d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa0cf0640 1 -- 172.21.15.94:0/1187747102 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c035020 con 0x7fcf9c0c1aa0 2024-03-06T07:31:05.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa0cf0640 1 -- 172.21.15.94:0/1187747102 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c031830 con 0x7fcf9c0c1aa0 2024-03-06T07:31:05.614 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa0cf0640 1 -- 172.21.15.94:0/1187747102 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c031c00 con 0x7fcf9c0c1aa0 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa3f56640 1 -- 172.21.15.94:0/1187747102 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c0c1aa0 msgr2=0x7fcf9c1529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/1187747102 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c0c1aa0 0x7fcf9c1529d0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0097a0 tx=0x7fcf8c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa3f56640 1 -- 172.21.15.94:0/1187747102 shutdown_connections 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/1187747102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c155a40 0x7fcf9c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/1187747102 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c0c1aa0 0x7fcf9c1529d0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.611+0000 7fcfa3f56640 1 -- 172.21.15.94:0/1187747102 >> 172.21.15.94:0/1187747102 conn(0x7fcf9c0c6710 msgr2=0x7fcf9c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.612+0000 7fcfa3f56640 1 -- 172.21.15.94:0/1187747102 shutdown_connections 2024-03-06T07:31:05.618 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.612+0000 7fcfa3f56640 1 -- 172.21.15.94:0/1187747102 wait complete. 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 Processor -- start 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 -- start start 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c0c1aa0 0x7fcf9c148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9c14f790 0x7fcf9c14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c155a40 0x7fcf9c147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c0d60 con 0x7fcf9c155a40 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c155520 con 0x7fcf9c14f790 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa3f56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c5e30 con 0x7fcf9c0c1aa0 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa24f3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c155a40 0x7fcf9c147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa24f3640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c155a40 0x7fcf9c147cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38370/0 (socket says 172.21.15.94:38370) 2024-03-06T07:31:05.619 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa24f3640 1 -- 172.21.15.94:0/3041718087 learned_addr learned my addr 172.21.15.94:0/3041718087 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa14f1640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9c14f790 0x7fcf9c14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.613+0000 7fcfa1cf2640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c0c1aa0 0x7fcf9c148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa24f3640 1 -- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c0c1aa0 msgr2=0x7fcf9c148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa24f3640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c0c1aa0 0x7fcf9c148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa24f3640 1 -- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9c14f790 msgr2=0x7fcf9c14e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa24f3640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9c14f790 0x7fcf9c14e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa24f3640 1 -- 172.21.15.94:0/3041718087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c148900 con 0x7fcf9c155a40 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa24f3640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c155a40 0x7fcf9c147cb0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007e60 tx=0x7fcf9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcf927fc640 1 -- 172.21.15.94:0/3041718087 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94014420 con 0x7fcf9c155a40 2024-03-06T07:31:05.620 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcf927fc640 1 -- 172.21.15.94:0/3041718087 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94019070 con 0x7fcf9c155a40 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c072370 con 0x7fcf9c155a40 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcf927fc640 1 -- 172.21.15.94:0/3041718087 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94014820 con 0x7fcf9c155a40 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.614+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c072930 con 0x7fcf9c155a40 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.615+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9c04f330 con 0x7fcf9c155a40 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.615+0000 7fcf927fc640 1 -- 172.21.15.94:0/3041718087 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf940149c0 con 0x7fcf9c155a40 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.616+0000 7fcf927fc640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf8006b4d0 0x7fcf8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.616+0000 7fcf927fc640 1 -- 172.21.15.94:0/3041718087 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcf9408ab10 con 0x7fcf9c155a40 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.616+0000 7fcfa1cf2640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf8006b4d0 0x7fcf8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.621 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.616+0000 7fcfa1cf2640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf8006b4d0 0x7fcf8006d990 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fcf98000f60 tx=0x7fcf98007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.622 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.620+0000 7fcf927fc640 1 -- 172.21.15.94:0/3041718087 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf94059140 con 0x7fcf9c155a40 2024-03-06T07:31:05.828 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.826+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf9c0c1f40 con 0x7fcf9c155a40 2024-03-06T07:31:05.829 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.827+0000 7fcf927fc640 1 -- 172.21.15.94:0/3041718087 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7fcf9405d1e0 con 0x7fcf9c155a40 2024-03-06T07:31:05.829 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:05.829 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:31:05.829 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:31:05.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf8006b4d0 msgr2=0x7fcf8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf8006b4d0 0x7fcf8006d990 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fcf98000f60 tx=0x7fcf98007d50 comp rx=0 tx=0).stop 2024-03-06T07:31:05.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c155a40 msgr2=0x7fcf9c147cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c155a40 0x7fcf9c147cb0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007e60 tx=0x7fcf9400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 shutdown_connections 2024-03-06T07:31:05.831 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcf8006b4d0 0x7fcf8006d990 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcf9c155a40 0x7fcf9c147cb0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf9c14f790 0x7fcf9c14e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 --2- 172.21.15.94:0/3041718087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf9c0c1aa0 0x7fcf9c148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 >> 172.21.15.94:0/3041718087 conn(0x7fcf9c0c6710 msgr2=0x7fcf9c0c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 shutdown_connections 2024-03-06T07:31:05.832 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.829+0000 7fcfa3f56640 1 -- 172.21.15.94:0/3041718087 wait complete. 2024-03-06T07:31:05.841 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.939+0000 7f5399202640 1 Processor -- start 2024-03-06T07:31:05.941 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5399202640 1 -- start start 2024-03-06T07:31:05.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5399202640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53940c5470 0x7f539414ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.943 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5399202640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f539414f110 0x7f53940c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5399202640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5394155140 con 0x7f53940c6710 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5399202640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53941553b0 con 0x7f53940c5470 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5399202640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539406b900 con 0x7f539414f110 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5392d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53940c5470 0x7f539414ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5392d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53940c5470 0x7f539414ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38382/0 (socket says 172.21.15.94:38382) 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f5392d9d640 1 -- 172.21.15.94:0/3425912456 learned_addr learned my addr 172.21.15.94:0/3425912456 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.940+0000 7f539259c640 1 --2- 172.21.15.94:0/3425912456 >> v2:172.21.15.145:3300/0 conn(0x7f539414f110 0x7f53940c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.941+0000 7f5392d9d640 1 -- 172.21.15.94:0/3425912456 >> v2:172.21.15.145:3300/0 conn(0x7f539414f110 msgr2=0x7f53940c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.941+0000 7f5392d9d640 1 --2- 172.21.15.94:0/3425912456 >> v2:172.21.15.145:3300/0 conn(0x7f539414f110 0x7f53940c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.941+0000 7f5392d9d640 1 -- 172.21.15.94:0/3425912456 >> v1:172.21.15.145:6789/0 conn(0x7f53940c6710 legacy=0x7f539406c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.941+0000 7f5391d9b640 1 -- 172.21.15.94:0/3425912456 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63821773 0 0) 0x7f5394155140 con 0x7f53940c6710 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.941+0000 7f5392d9d640 1 -- 172.21.15.94:0/3425912456 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539406bbe0 con 0x7f53940c5470 2024-03-06T07:31:05.944 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.941+0000 7f5392d9d640 1 --2- 172.21.15.94:0/3425912456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53940c5470 0x7f539414ebd0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f53940aa2d0 tx=0x7f538c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751f0d15beadcabe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5391d9b640 1 -- 172.21.15.94:0/3425912456 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c02e070 con 0x7f53940c5470 2024-03-06T07:31:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5391d9b640 1 -- 172.21.15.94:0/3425912456 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c037070 con 0x7f53940c5470 2024-03-06T07:31:05.945 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5391d9b640 1 -- 172.21.15.94:0/3425912456 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c031a00 con 0x7f53940c5470 2024-03-06T07:31:05.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 -- 172.21.15.94:0/3425912456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53940c5470 msgr2=0x7f539414ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.947 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 --2- 172.21.15.94:0/3425912456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53940c5470 0x7f539414ebd0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f53940aa2d0 tx=0x7f538c02b880 comp rx=0 tx=0).stop 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 -- 172.21.15.94:0/3425912456 shutdown_connections 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 --2- 172.21.15.94:0/3425912456 >> v2:172.21.15.145:3300/0 conn(0x7f539414f110 0x7f53940c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 --2- 172.21.15.94:0/3425912456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53940c5470 0x7f539414ebd0 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 -- 172.21.15.94:0/3425912456 >> 172.21.15.94:0/3425912456 conn(0x7f53940ca880 msgr2=0x7f53940cac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 -- 172.21.15.94:0/3425912456 shutdown_connections 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.942+0000 7f5399202640 1 -- 172.21.15.94:0/3425912456 wait complete. 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 Processor -- start 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 -- start start 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53940c5470 0x7f53941ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53940c6710 0x7f539414fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539414f110 0x7f53940c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.948 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53940cb4f0 con 0x7f539414f110 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53940cb370 con 0x7f53940c5470 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5399202640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53940c6b20 con 0x7f53940c6710 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f539359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539414f110 0x7f53940c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f539359e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539414f110 0x7f53940c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38394/0 (socket says 172.21.15.94:38394) 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f539359e640 1 -- 172.21.15.94:0/1849072825 learned_addr learned my addr 172.21.15.94:0/1849072825 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f5392d9d640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53940c5470 0x7f53941ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.944+0000 7f539259c640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53940c6710 0x7f539414fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f539359e640 1 -- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53940c6710 msgr2=0x7f539414fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f539359e640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53940c6710 0x7f539414fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f539359e640 1 -- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53940c5470 msgr2=0x7f53941ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f539359e640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53940c5470 0x7f53941ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f539359e640 1 -- 172.21.15.94:0/1849072825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53941edc90 con 0x7f539414f110 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f539359e640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539414f110 0x7f53940c3d90 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f538400c460 tx=0x7f538400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.949 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f537b7fe640 1 -- 172.21.15.94:0/1849072825 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384017070 con 0x7f539414f110 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f537b7fe640 1 -- 172.21.15.94:0/1849072825 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53840035f0 con 0x7f539414f110 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5394150330 con 0x7f539414f110 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f537b7fe640 1 -- 172.21.15.94:0/1849072825 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53840039e0 con 0x7f539414f110 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.945+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53941edf30 con 0x7f539414f110 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.946+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53940693c0 con 0x7f539414f110 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.946+0000 7f537b7fe640 1 -- 172.21.15.94:0/1849072825 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f538401a070 con 0x7f539414f110 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.947+0000 7f537b7fe640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f538006b500 0x7f538006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.947+0000 7f537b7fe640 1 -- 172.21.15.94:0/1849072825 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5384089650 con 0x7f539414f110 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.947+0000 7f5392d9d640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f538006b500 0x7f538006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.947+0000 7f5392d9d640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f538006b500 0x7f538006d9c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f538c036000 tx=0x7f538c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.953 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:05.951+0000 7f537b7fe640 1 -- 172.21.15.94:0/1849072825 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5384057c80 con 0x7f539414f110 2024-03-06T07:31:06.125 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.123+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53940cb120 con 0x7f539414f110 2024-03-06T07:31:06.126 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.125+0000 7f537b7fe640 1 -- 172.21.15.94:0/1849072825 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+17288 (secure 0 0 0) 0x7f538405bd20 con 0x7f539414f110 2024-03-06T07:31:06.127 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:06.127 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:28:37.254511+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:15.999555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:16.719665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.2","mappings":[{"from":0,"to":2}]},{"pgid":"34.13","mappings":[{"from":0,"to":2}]},{"pgid":"34.31","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6837/679350833":"2024-03-07T07:28:02.249189+0000","172.21.15.94:6834/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6835/679350833":"2024-03-07T07:28:02.249189+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.126+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f538006b500 msgr2=0x7f538006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.126+0000 7f5399202640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f538006b500 0x7f538006d9c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f538c036000 tx=0x7f538c008030 comp rx=0 tx=0).stop 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.126+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539414f110 msgr2=0x7f53940c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.126+0000 7f5399202640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539414f110 0x7f53940c3d90 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f538400c460 tx=0x7f538400c920 comp rx=0 tx=0).stop 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 shutdown_connections 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f538006b500 0x7f538006d9c0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f539414f110 0x7f53940c3d90 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53940c6710 0x7f539414fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 --2- 172.21.15.94:0/1849072825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53940c5470 0x7f53941ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 >> 172.21.15.94:0/1849072825 conn(0x7f53940ca880 msgr2=0x7f53940aaa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 shutdown_connections 2024-03-06T07:31:06.129 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.127+0000 7f5399202640 1 -- 172.21.15.94:0/1849072825 wait complete. 2024-03-06T07:31:06.140 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:06.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.238+0000 7f88716c9640 1 Processor -- start 2024-03-06T07:31:06.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.238+0000 7f88716c9640 1 -- start start 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.238+0000 7f88716c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f886c14f360 0x7f886c155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f88716c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c158b60 0x7f886c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f88716c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886c0c73d0 con 0x7f886c0c88c0 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f88716c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f886c0c8dd0 con 0x7f886c14f360 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f88716c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0be2e0 con 0x7f886c158b60 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f88627fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f886c0c88c0 0x7f886c0c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45780/0 (socket says 172.21.15.94:45780) 2024-03-06T07:31:06.241 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f88627fc640 1 -- 172.21.15.94:0/1108401479 learned_addr learned my addr 172.21.15.94:0/1108401479 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886affd640 1 --2- 172.21.15.94:0/1108401479 >> v2:172.21.15.145:3300/0 conn(0x7f886c14f360 0x7f886c155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886b7fe640 1 --2- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c158b60 0x7f886c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886a7fc640 1 -- 172.21.15.94:0/1108401479 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162736799 0 0) 0x7f886c0c73d0 con 0x7f886c0c88c0 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886a7fc640 1 -- 172.21.15.94:0/1108401479 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8858003520 con 0x7f886c0c88c0 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886affd640 1 -- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c158b60 msgr2=0x7f886c158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886affd640 1 --2- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c158b60 0x7f886c158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886affd640 1 -- 172.21.15.94:0/1108401479 >> v1:172.21.15.94:6789/0 conn(0x7f886c0c88c0 legacy=0x7f886c0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.239+0000 7f886affd640 1 -- 172.21.15.94:0/1108401479 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c0c7670 con 0x7f886c14f360 2024-03-06T07:31:06.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.240+0000 7f886affd640 1 --2- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 0x7f886c155c10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f885c0097a0 tx=0x7f885c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fc15b9d5fc6d8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.240+0000 7f886a7fc640 1 -- 172.21.15.94:0/1108401479 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c0315a0 con 0x7f886c14f360 2024-03-06T07:31:06.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.240+0000 7f886a7fc640 1 -- 172.21.15.94:0/1108401479 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885c037070 con 0x7f886c14f360 2024-03-06T07:31:06.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.240+0000 7f886a7fc640 1 -- 172.21.15.94:0/1108401479 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c031940 con 0x7f886c14f360 2024-03-06T07:31:06.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.240+0000 7f88716c9640 1 -- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 msgr2=0x7f886c155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.241+0000 7f88716c9640 1 --2- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 0x7f886c155c10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f885c0097a0 tx=0x7f885c02b840 comp rx=0 tx=0).stop 2024-03-06T07:31:06.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.241+0000 7f88716c9640 1 -- 172.21.15.94:0/1108401479 shutdown_connections 2024-03-06T07:31:06.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.241+0000 7f88716c9640 1 --2- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c158b60 0x7f886c158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.241+0000 7f88716c9640 1 --2- 172.21.15.94:0/1108401479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 0x7f886c155c10 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.241+0000 7f88716c9640 1 -- 172.21.15.94:0/1108401479 >> 172.21.15.94:0/1108401479 conn(0x7f886c0ac570 msgr2=0x7f886c0ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.241+0000 7f88716c9640 1 -- 172.21.15.94:0/1108401479 shutdown_connections 2024-03-06T07:31:06.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.241+0000 7f88716c9640 1 -- 172.21.15.94:0/1108401479 wait complete. 2024-03-06T07:31:06.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 Processor -- start 2024-03-06T07:31:06.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 -- start start 2024-03-06T07:31:06.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c0c88c0 0x7f886c1ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 0x7f886c1eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c158b60 0x7f886c14b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0c5a50 con 0x7f886c158b60 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0c7c30 con 0x7f886c14f360 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.242+0000 7f88716c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0c5bd0 con 0x7f886c0c88c0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c158b60 0x7f886c14b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c158b60 0x7f886c14b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38410/0 (socket says 172.21.15.94:38410) 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 -- 172.21.15.94:0/3594099568 learned_addr learned my addr 172.21.15.94:0/3594099568 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886affd640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c0c88c0 0x7f886c1ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f88627fc640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 0x7f886c1eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 -- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c0c88c0 msgr2=0x7f886c1ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c0c88c0 0x7f886c1ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 -- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 msgr2=0x7f886c1eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 0x7f886c1eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 -- 172.21.15.94:0/3594099568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c14ff40 con 0x7f886c158b60 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f886b7fe640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c158b60 0x7f886c14b9a0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f886400efc0 tx=0x7f886400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.247 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f8863fff640 1 -- 172.21.15.94:0/3594099568 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864018020 con 0x7f886c158b60 2024-03-06T07:31:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f8863fff640 1 -- 172.21.15.94:0/3594099568 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8864014480 con 0x7f886c158b60 2024-03-06T07:31:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f8863fff640 1 -- 172.21.15.94:0/3594099568 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864014870 con 0x7f886c158b60 2024-03-06T07:31:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886c1eefa0 con 0x7f886c158b60 2024-03-06T07:31:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.243+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886c1501e0 con 0x7f886c158b60 2024-03-06T07:31:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.244+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886c075300 con 0x7f886c158b60 2024-03-06T07:31:06.248 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.245+0000 7f8863fff640 1 -- 172.21.15.94:0/3594099568 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8864005d10 con 0x7f886c158b60 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.245+0000 7f8863fff640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f884c06b500 0x7f884c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.245+0000 7f8863fff640 1 -- 172.21.15.94:0/3594099568 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f886408adc0 con 0x7f886c158b60 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.245+0000 7f886affd640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f884c06b500 0x7f884c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.249 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.246+0000 7f886affd640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f884c06b500 0x7f884c06d9c0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f885c036000 tx=0x7f885c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.251 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.249+0000 7f8863fff640 1 -- 172.21.15.94:0/3594099568 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f886c075300 con 0x7f886c158b60 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.459+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f886c04f330 con 0x7f886c158b60 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.459+0000 7f8863fff640 1 -- 172.21.15.94:0/3594099568 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f8864022070 con 0x7f886c158b60 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:06.461 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:31:06.462 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:31:06.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f884c06b500 msgr2=0x7f884c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f884c06b500 0x7f884c06d9c0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f885c036000 tx=0x7f885c03d040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c158b60 msgr2=0x7f886c14b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c158b60 0x7f886c14b9a0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f886400efc0 tx=0x7f886400c490 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 shutdown_connections 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f884c06b500 0x7f884c06d9c0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c158b60 0x7f886c14b9a0 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f886c14f360 0x7f886c1eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 --2- 172.21.15.94:0/3594099568 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f886c0c88c0 0x7f886c1ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 >> 172.21.15.94:0/3594099568 conn(0x7f886c0ac570 msgr2=0x7f886c14da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 shutdown_connections 2024-03-06T07:31:06.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.461+0000 7f88716c9640 1 -- 172.21.15.94:0/3594099568 wait complete. 2024-03-06T07:31:06.474 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:06.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.571+0000 7f0ca0da0640 1 Processor -- start 2024-03-06T07:31:06.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.571+0000 7f0ca0da0640 1 -- start start 2024-03-06T07:31:06.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0ca0da0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c9c14f3e0 0x7f0c9c152550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0ca0da0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0c9c0c2670 0x7f0c9c0c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0ca0da0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c06e810 0x7f0c9c06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0ca0da0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9c14f7c0 con 0x7f0c9c14f3e0 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0ca0da0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9c152a90 con 0x7f0c9c0c2670 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0ca0da0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c0717f0 con 0x7f0c9c06e810 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0c9b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c9c14f3e0 0x7f0c9c152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0c9b7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c9c14f3e0 0x7f0c9c152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38412/0 (socket says 172.21.15.94:38412) 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/3255577269 learned_addr learned my addr 172.21.15.94:0/3255577269 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0c9affd640 1 --2- 172.21.15.94:0/3255577269 >> v2:172.21.15.145:3300/0 conn(0x7f0c9c0c2670 0x7f0c9c0c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.572+0000 7f0c9bfff640 1 --2- 172.21.15.94:0/3255577269 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c06e810 0x7f0c9c06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.573+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/3255577269 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c06e810 msgr2=0x7f0c9c06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.573+0000 7f0c9b7fe640 1 --2- 172.21.15.94:0/3255577269 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c06e810 0x7f0c9c06eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.573+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/3255577269 >> v2:172.21.15.145:3300/0 conn(0x7f0c9c0c2670 msgr2=0x7f0c9c0c2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.573+0000 7f0c9b7fe640 1 --2- 172.21.15.94:0/3255577269 >> v2:172.21.15.145:3300/0 conn(0x7f0c9c0c2670 0x7f0c9c0c2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.573+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/3255577269 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c071ad0 con 0x7f0c9c14f3e0 2024-03-06T07:31:06.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.573+0000 7f0c9b7fe640 1 --2- 172.21.15.94:0/3255577269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c14f3e0 0x7f0c9c152550 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00bb10 tx=0x7f0c8c02dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42a39d02fce345c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0c9a7fc640 1 -- 172.21.15.94:0/3255577269 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c03b070 con 0x7f0c9c14f3e0 2024-03-06T07:31:06.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0c9a7fc640 1 -- 172.21.15.94:0/3255577269 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c02ed50 con 0x7f0c9c14f3e0 2024-03-06T07:31:06.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0c9a7fc640 1 -- 172.21.15.94:0/3255577269 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c036660 con 0x7f0c9c14f3e0 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 -- 172.21.15.94:0/3255577269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c14f3e0 msgr2=0x7f0c9c152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/3255577269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c14f3e0 0x7f0c9c152550 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00bb10 tx=0x7f0c8c02dfa0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 -- 172.21.15.94:0/3255577269 shutdown_connections 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/3255577269 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c06e810 0x7f0c9c06eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/3255577269 >> v2:172.21.15.145:3300/0 conn(0x7f0c9c0c2670 0x7f0c9c0c2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/3255577269 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c14f3e0 0x7f0c9c152550 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 -- 172.21.15.94:0/3255577269 >> 172.21.15.94:0/3255577269 conn(0x7f0c9c06c0d0 msgr2=0x7f0c9c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 -- 172.21.15.94:0/3255577269 shutdown_connections 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.574+0000 7f0ca0da0640 1 -- 172.21.15.94:0/3255577269 wait complete. 2024-03-06T07:31:06.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.575+0000 7f0ca0da0640 1 Processor -- start 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.575+0000 7f0ca0da0640 1 -- start start 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0ca0da0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c06e810 0x7f0c9c0c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0ca0da0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c9c0c2670 0x7f0c9c1507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0ca0da0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c14f3e0 0x7f0c9c150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0ca0da0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c06f1e0 con 0x7f0c9c06e810 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0ca0da0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c06e2f0 con 0x7f0c9c0c2670 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0ca0da0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c0c37d0 con 0x7f0c9c14f3e0 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0c9b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c06e810 0x7f0c9c0c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0c9b7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c06e810 0x7f0c9c0c0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38428/0 (socket says 172.21.15.94:38428) 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/491191329 learned_addr learned my addr 172.21.15.94:0/491191329 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0c9affd640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c9c0c2670 0x7f0c9c1507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.576+0000 7f0c9bfff640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c14f3e0 0x7f0c9c150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c14f3e0 msgr2=0x7f0c9c150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c9b7fe640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c14f3e0 0x7f0c9c150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c9c0c2670 msgr2=0x7f0c9c1507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c9b7fe640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c9c0c2670 0x7f0c9c1507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c9b7fe640 1 -- 172.21.15.94:0/491191329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c1ea810 con 0x7f0c9c06e810 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c9affd640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c9c0c2670 0x7f0c9c1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c9b7fe640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c06e810 0x7f0c9c0c0c50 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c02dfe0 tx=0x7f0c8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c83fff640 1 -- 172.21.15.94:0/491191329 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c03b030 con 0x7f0c9c06e810 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c83fff640 1 -- 172.21.15.94:0/491191329 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c03d070 con 0x7f0c9c06e810 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0c83fff640 1 -- 172.21.15.94:0/491191329 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c034550 con 0x7f0c9c06e810 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9c0ab900 con 0x7f0c9c06e810 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.577+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9c0abec0 con 0x7f0c9c06e810 2024-03-06T07:31:06.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.578+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9c069430 con 0x7f0c9c06e810 2024-03-06T07:31:06.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.578+0000 7f0c83fff640 1 -- 172.21.15.94:0/491191329 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c8c048060 con 0x7f0c9c06e810 2024-03-06T07:31:06.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.579+0000 7f0c83fff640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c8806b500 0x7f0c8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.579+0000 7f0c83fff640 1 -- 172.21.15.94:0/491191329 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0c8c0ab920 con 0x7f0c9c06e810 2024-03-06T07:31:06.583 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.579+0000 7f0c9affd640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c8806b500 0x7f0c8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.579+0000 7f0c9affd640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c8806b500 0x7f0c8806d9c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c14e4b0 tx=0x7f0c90009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.585 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.583+0000 7f0c83fff640 1 -- 172.21.15.94:0/491191329 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c8c079ed0 con 0x7f0c9c06e810 2024-03-06T07:31:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.791+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c9c14c000 con 0x7f0c9c06e810 2024-03-06T07:31:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.791+0000 7f0c83fff640 1 -- 172.21.15.94:0/491191329 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+3479 (secure 0 0 0) 0x7f0c8c07df70 con 0x7f0c9c06e810 2024-03-06T07:31:06.793 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:06.793 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:17.753463+0000","modified":"2024-03-06T07:29:12.259362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":213,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":38273,"mds_1":10773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10773":{"gid":10773,"name":"a","rank":1,"incarnation":207,"state":"up:stopping","state_seq":25,"addr":"172.21.15.94:6833/341111452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":341111452},{"type":"v1","addr":"172.21.15.94:6833","nonce":341111452}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:12.259362+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_38273":{"gid":38273,"name":"b","rank":0,"incarnation":194,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6836/2585066812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2585066812},{"type":"v1","addr":"172.21.15.145:6836","nonce":2585066812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":38273,"qdb_cluster":[38273]},"id":16}]} 2024-03-06T07:31:06.793 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 213 2024-03-06T07:31:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c8806b500 msgr2=0x7f0c8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c8806b500 0x7f0c8806d9c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c14e4b0 tx=0x7f0c90009190 comp rx=0 tx=0).stop 2024-03-06T07:31:06.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c06e810 msgr2=0x7f0c9c0c0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c06e810 0x7f0c9c0c0c50 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c02dfe0 tx=0x7f0c8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 shutdown_connections 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0c8806b500 0x7f0c8806d9c0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c9c14f3e0 0x7f0c9c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c9c0c2670 0x7f0c9c1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 --2- 172.21.15.94:0/491191329 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c06e810 0x7f0c9c0c0c50 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 >> 172.21.15.94:0/491191329 conn(0x7f0c9c06c0d0 msgr2=0x7f0c9c072b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 shutdown_connections 2024-03-06T07:31:06.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.793+0000 7f0ca0da0640 1 -- 172.21.15.94:0/491191329 wait complete. 2024-03-06T07:31:06.806 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:06.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.903+0000 7fccb68ed640 1 Processor -- start 2024-03-06T07:31:06.905 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccb68ed640 1 -- start start 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccb68ed640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fccb00c1320 0x7fccb01566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccb68ed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fccb00c6000 0x7fccb0156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccb68ed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb0153a60 con 0x7fccb0150580 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccb68ed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb0153d40 con 0x7fccb00c6000 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccb68ed640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb006c2c0 con 0x7fccb00c1320 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccb4e8a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fccb00c6000 0x7fccb0156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccaffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fccb0150580 0x7fccb006dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45796/0 (socket says 172.21.15.94:45796) 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.904+0000 7fccaffff640 1 -- 172.21.15.94:0/753645270 learned_addr learned my addr 172.21.15.94:0/753645270 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.905+0000 7fccaf7fe640 1 --2- 172.21.15.94:0/753645270 >> v2:172.21.15.145:3300/0 conn(0x7fccb00c1320 0x7fccb01566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.905+0000 7fccb4e8a640 1 -- 172.21.15.94:0/753645270 >> v2:172.21.15.145:3300/0 conn(0x7fccb00c1320 msgr2=0x7fccb01566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.905+0000 7fccb4e8a640 1 --2- 172.21.15.94:0/753645270 >> v2:172.21.15.145:3300/0 conn(0x7fccb00c1320 0x7fccb01566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.907 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.905+0000 7fccb4e8a640 1 -- 172.21.15.94:0/753645270 >> v1:172.21.15.94:6789/0 conn(0x7fccb0150580 legacy=0x7fccb006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.905+0000 7fccaeffd640 1 -- 172.21.15.94:0/753645270 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708449226 0 0) 0x7fccb0153a60 con 0x7fccb0150580 2024-03-06T07:31:06.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.905+0000 7fccb4e8a640 1 -- 172.21.15.94:0/753645270 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb006c5a0 con 0x7fccb00c6000 2024-03-06T07:31:06.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb4e8a640 1 --2- 172.21.15.94:0/753645270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c6000 0x7fccb0156bf0 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fccb00c7730 tx=0x7fcca802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee31eca7e68c2707 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccaeffd640 1 -- 172.21.15.94:0/753645270 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8039070 con 0x7fccb00c6000 2024-03-06T07:31:06.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccaeffd640 1 -- 172.21.15.94:0/753645270 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca802cbf0 con 0x7fccb00c6000 2024-03-06T07:31:06.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccaeffd640 1 -- 172.21.15.94:0/753645270 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8034440 con 0x7fccb00c6000 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 -- 172.21.15.94:0/753645270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c6000 msgr2=0x7fccb0156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 --2- 172.21.15.94:0/753645270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c6000 0x7fccb0156bf0 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fccb00c7730 tx=0x7fcca802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 -- 172.21.15.94:0/753645270 shutdown_connections 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 --2- 172.21.15.94:0/753645270 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c6000 0x7fccb0156bf0 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 --2- 172.21.15.94:0/753645270 >> v2:172.21.15.145:3300/0 conn(0x7fccb00c1320 0x7fccb01566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 -- 172.21.15.94:0/753645270 >> 172.21.15.94:0/753645270 conn(0x7fccb00c6710 msgr2=0x7fccb00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 -- 172.21.15.94:0/753645270 shutdown_connections 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.906+0000 7fccb68ed640 1 -- 172.21.15.94:0/753645270 wait complete. 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.907+0000 7fccb68ed640 1 Processor -- start 2024-03-06T07:31:06.910 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.907+0000 7fccb68ed640 1 -- start start 2024-03-06T07:31:06.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccb68ed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c1320 0x7fccb0152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccb68ed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccb00c6000 0x7fccb01ec710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccb68ed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccb0150580 0x7fccb00a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccb68ed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb006e5a0 con 0x7fccb00c1320 2024-03-06T07:31:06.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccb68ed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb006e420 con 0x7fccb0150580 2024-03-06T07:31:06.913 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccb68ed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00c0c40 con 0x7fccb00c6000 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c1320 0x7fccb0152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c1320 0x7fccb0152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38454/0 (socket says 172.21.15.94:38454) 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 -- 172.21.15.94:0/2702962312 learned_addr learned my addr 172.21.15.94:0/2702962312 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccb4e8a640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccb0150580 0x7fccb00a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaf7fe640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccb00c6000 0x7fccb01ec710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 -- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccb00c6000 msgr2=0x7fccb01ec710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccb00c6000 0x7fccb01ec710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 -- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccb0150580 msgr2=0x7fccb00a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccb0150580 0x7fccb00a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.908+0000 7fccaffff640 1 -- 172.21.15.94:0/2702962312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb00a8160 con 0x7fccb00c1320 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.909+0000 7fccaffff640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c1320 0x7fccb0152b00 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800c460 tx=0x7fcc9800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.909+0000 7fccacff9640 1 -- 172.21.15.94:0/2702962312 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc98018070 con 0x7fccb00c1320 2024-03-06T07:31:06.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.909+0000 7fccacff9640 1 -- 172.21.15.94:0/2702962312 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc98007500 con 0x7fccb00c1320 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.909+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb0149160 con 0x7fccb00c1320 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.909+0000 7fccacff9640 1 -- 172.21.15.94:0/2702962312 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc98004a20 con 0x7fccb00c1320 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.909+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb0149720 con 0x7fccb00c1320 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.910+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb004f330 con 0x7fccb00c1320 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.910+0000 7fccacff9640 1 -- 172.21.15.94:0/2702962312 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc9801b070 con 0x7fccb00c1320 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.911+0000 7fccacff9640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc9006b500 0x7fcc9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.911+0000 7fccacff9640 1 -- 172.21.15.94:0/2702962312 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcc980895b0 con 0x7fccb00c1320 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.911+0000 7fccaf7fe640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc9006b500 0x7fcc9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.915 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.911+0000 7fccaf7fe640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc9006b500 0x7fcc9006d9c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fccb014e380 tx=0x7fcca0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:06.915+0000 7fccacff9640 1 -- 172.21.15.94:0/2702962312 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc98057be0 con 0x7fccb00c1320 2024-03-06T07:31:07.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.086+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb014b930 con 0x7fccb00c1320 2024-03-06T07:31:07.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.087+0000 7fccacff9640 1 -- 172.21.15.94:0/2702962312 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+17288 (secure 0 0 0) 0x7fcc9805bc80 con 0x7fccb00c1320 2024-03-06T07:31:07.089 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:07.090 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":213,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:28:37.254511+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:15.999555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:16.719665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.2","mappings":[{"from":0,"to":2}]},{"pgid":"34.13","mappings":[{"from":0,"to":2}]},{"pgid":"34.31","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6835/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6837/679350833":"2024-03-07T07:28:02.249189+0000","172.21.15.94:6834/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6835/679350833":"2024-03-07T07:28:02.249189+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:07.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc9006b500 msgr2=0x7fcc9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc9006b500 0x7fcc9006d9c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fccb014e380 tx=0x7fcca0009190 comp rx=0 tx=0).stop 2024-03-06T07:31:07.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c1320 msgr2=0x7fccb0152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c1320 0x7fccb0152b00 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800c460 tx=0x7fcc9800c920 comp rx=0 tx=0).stop 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 shutdown_connections 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcc9006b500 0x7fcc9006d9c0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccb0150580 0x7fccb00a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccb00c6000 0x7fccb01ec710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 --2- 172.21.15.94:0/2702962312 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fccb00c1320 0x7fccb0152b00 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 >> 172.21.15.94:0/2702962312 conn(0x7fccb00c6710 msgr2=0x7fccb0153630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 shutdown_connections 2024-03-06T07:31:07.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.089+0000 7fccb68ed640 1 -- 172.21.15.94:0/2702962312 wait complete. 2024-03-06T07:31:07.102 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:31:07.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.200+0000 7fec89e81640 1 Processor -- start 2024-03-06T07:31:07.202 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.201+0000 7fec89e81640 1 -- start start 2024-03-06T07:31:07.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.201+0000 7fec89e81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec84072f00 0x7fec84153990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.201+0000 7fec89e81640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec8406d990 0x7fec8406de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.203 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.201+0000 7fec89e81640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec84070550 con 0x7fec840c0fa0 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.201+0000 7fec89e81640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8406e360 con 0x7fec8406d990 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.201+0000 7fec89e81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec84070830 con 0x7fec84072f00 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.201+0000 7fec83fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec8406d990 0x7fec8406de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.202+0000 7fec82ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fec840c0fa0 0x7fec840c6840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:46344/0 (socket says 172.21.15.94:46344) 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.202+0000 7fec82ffd640 1 -- 172.21.15.94:0/4277665780 learned_addr learned my addr 172.21.15.94:0/4277665780 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.202+0000 7fec837fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec84072f00 0x7fec84153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.202+0000 7fec83fff640 1 -- 172.21.15.94:0/4277665780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec84072f00 msgr2=0x7fec84153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.202+0000 7fec83fff640 1 --2- 172.21.15.94:0/4277665780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec84072f00 0x7fec84153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.202+0000 7fec83fff640 1 -- 172.21.15.94:0/4277665780 >> v1:172.21.15.145:6789/0 conn(0x7fec840c0fa0 legacy=0x7fec840c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.204 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.202+0000 7fec83fff640 1 -- 172.21.15.94:0/4277665780 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84070b10 con 0x7fec8406d990 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec83fff640 1 --2- 172.21.15.94:0/4277665780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 0x7fec8406de20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009d90 tx=0x7fec7c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f68fc989d9b5aef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec827fc640 1 -- 172.21.15.94:0/4277665780 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c039070 con 0x7fec8406d990 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec827fc640 1 -- 172.21.15.94:0/4277665780 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c02cc50 con 0x7fec8406d990 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec827fc640 1 -- 172.21.15.94:0/4277665780 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c034440 con 0x7fec8406d990 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 -- 172.21.15.94:0/4277665780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 msgr2=0x7fec8406de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 --2- 172.21.15.94:0/4277665780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 0x7fec8406de20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009d90 tx=0x7fec7c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 -- 172.21.15.94:0/4277665780 shutdown_connections 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 --2- 172.21.15.94:0/4277665780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 0x7fec8406de20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.206 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 --2- 172.21.15.94:0/4277665780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec84072f00 0x7fec84153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 -- 172.21.15.94:0/4277665780 >> 172.21.15.94:0/4277665780 conn(0x7fec8406b1a0 msgr2=0x7fec8406b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 -- 172.21.15.94:0/4277665780 shutdown_connections 2024-03-06T07:31:07.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.203+0000 7fec89e81640 1 -- 172.21.15.94:0/4277665780 wait complete. 2024-03-06T07:31:07.207 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.204+0000 7fec89e81640 1 Processor -- start 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec89e81640 1 -- start start 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec89e81640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 0x7fec84147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec89e81640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec84072f00 0x7fec8414afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec89e81640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec840c0fa0 0x7fec8414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec89e81640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c70d0 con 0x7fec84072f00 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec89e81640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c6f50 con 0x7fec8406d990 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec89e81640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec84070df0 con 0x7fec840c0fa0 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec82ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec84072f00 0x7fec8414afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec82ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec84072f00 0x7fec8414afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38462/0 (socket says 172.21.15.94:38462) 2024-03-06T07:31:07.209 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec837fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 0x7fec84147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec82ffd640 1 -- 172.21.15.94:0/3447759848 learned_addr learned my addr 172.21.15.94:0/3447759848 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.205+0000 7fec83fff640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec840c0fa0 0x7fec8414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec82ffd640 1 -- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec840c0fa0 msgr2=0x7fec8414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec82ffd640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec840c0fa0 0x7fec8414f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec82ffd640 1 -- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 msgr2=0x7fec84147010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec82ffd640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 0x7fec84147010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec82ffd640 1 -- 172.21.15.94:0/3447759848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec841e95c0 con 0x7fec84072f00 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec82ffd640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec84072f00 0x7fec8414afa0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fec7400c470 tx=0x7fec7400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec6bfff640 1 -- 172.21.15.94:0/3447759848 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74017070 con 0x7fec84072f00 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec6bfff640 1 -- 172.21.15.94:0/3447759848 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74005ce0 con 0x7fec84072f00 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec841e9860 con 0x7fec84072f00 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec6bfff640 1 -- 172.21.15.94:0/3447759848 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740072d0 con 0x7fec84072f00 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.206+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8414b4e0 con 0x7fec84072f00 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.207+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec84071170 con 0x7fec84072f00 2024-03-06T07:31:07.210 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.207+0000 7fec6bfff640 1 -- 172.21.15.94:0/3447759848 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec7401a070 con 0x7fec84072f00 2024-03-06T07:31:07.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.208+0000 7fec6bfff640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.208+0000 7fec6bfff640 1 -- 172.21.15.94:0/3447759848 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fec74089630 con 0x7fec84072f00 2024-03-06T07:31:07.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.208+0000 7fec837fe640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.208+0000 7fec837fe640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec7006b4d0 0x7fec7006d990 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fec78009b40 tx=0x7fec78009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.212+0000 7fec6bfff640 1 -- 172.21.15.94:0/3447759848 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec84071170 con 0x7fec84072f00 2024-03-06T07:31:07.422 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.420+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fec8404f330 con 0x7fec84072f00 2024-03-06T07:31:07.423 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.422+0000 7fec6bfff640 1 -- 172.21.15.94:0/3447759848 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v214) v1 ==== 161+0+0 (secure 0 0 0) 0x7fec74057be0 con 0x7fec84072f00 2024-03-06T07:31:07.424 INFO:teuthology.orchestra.run.smithi094.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:31:07.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec7006b4d0 msgr2=0x7fec7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec7006b4d0 0x7fec7006d990 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fec78009b40 tx=0x7fec78009190 comp rx=0 tx=0).stop 2024-03-06T07:31:07.426 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec84072f00 msgr2=0x7fec8414afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec84072f00 0x7fec8414afa0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7fec7400c470 tx=0x7fec7400c930 comp rx=0 tx=0).stop 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 shutdown_connections 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fec7006b4d0 0x7fec7006d990 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec840c0fa0 0x7fec8414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fec84072f00 0x7fec8414afa0 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 --2- 172.21.15.94:0/3447759848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8406d990 0x7fec84147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.424+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 >> 172.21.15.94:0/3447759848 conn(0x7fec8406b1a0 msgr2=0x7fec8406b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.425+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 shutdown_connections 2024-03-06T07:31:07.427 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.425+0000 7fec89e81640 1 -- 172.21.15.94:0/3447759848 wait complete. 2024-03-06T07:31:07.434 INFO:tasks.ceph.mds.b.smithi145.stderr: -8505> 2024-03-06T07:27:23.667+0000 7fd26e6aa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:07.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:31:07.466 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-03-06T07:31:07.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.537+0000 7ff76ce9a640 1 Processor -- start 2024-03-06T07:31:07.539 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.537+0000 7ff76ce9a640 1 -- start start 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.538+0000 7ff76ce9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76806ca70 0x7ff76814e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.538+0000 7ff76ce9a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7680b0450 con 0x7ff768155c30 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.538+0000 7ff76ce9a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7681560e0 con 0x7ff7680b0070 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.538+0000 7ff76ce9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76806c020 con 0x7ff76806ca70 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.538+0000 7ff765d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff768155c30 0x7ff7680c5380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:45808/0 (socket says 172.21.15.94:45808) 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff765d9b640 1 -- 172.21.15.94:0/3565305602 learned_addr learned my addr 172.21.15.94:0/3565305602 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff766d9d640 1 --2- 172.21.15.94:0/3565305602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76806ca70 0x7ff76814e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff76559a640 1 -- 172.21.15.94:0/3565305602 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834813193 0 0) 0x7ff758001110 con 0x7ff768155c30 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff76559a640 1 -- 172.21.15.94:0/3565305602 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff754003520 con 0x7ff768155c30 2024-03-06T07:31:07.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff76559a640 1 -- 172.21.15.94:0/3565305602 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971060002 0 0) 0x7ff7681560e0 con 0x7ff7680b0070 2024-03-06T07:31:07.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff76559a640 1 -- 172.21.15.94:0/3565305602 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff758001110 con 0x7ff7680b0070 2024-03-06T07:31:07.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff766d9d640 1 -- 172.21.15.94:0/3565305602 >> v1:172.21.15.145:6789/0 conn(0x7ff7680b0070 legacy=0x7ff7680c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff766d9d640 1 -- 172.21.15.94:0/3565305602 >> v1:172.21.15.94:6789/0 conn(0x7ff768155c30 legacy=0x7ff7680c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.539+0000 7ff766d9d640 1 -- 172.21.15.94:0/3565305602 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7680c77f0 con 0x7ff76806ca70 2024-03-06T07:31:07.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff766d9d640 1 --2- 172.21.15.94:0/3565305602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76806ca70 0x7ff76814e800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff7681562b0 tx=0x7ff76002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=931affa356a0f233 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76559a640 1 -- 172.21.15.94:0/3565305602 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760039070 con 0x7ff76806ca70 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76559a640 1 -- 172.21.15.94:0/3565305602 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76002cc40 con 0x7ff76806ca70 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76559a640 1 -- 172.21.15.94:0/3565305602 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760034440 con 0x7ff76806ca70 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76ce9a640 1 -- 172.21.15.94:0/3565305602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76806ca70 msgr2=0x7ff76814e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/3565305602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76806ca70 0x7ff76814e800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff7681562b0 tx=0x7ff76002be90 comp rx=0 tx=0).stop 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76ce9a640 1 -- 172.21.15.94:0/3565305602 shutdown_connections 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/3565305602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76806ca70 0x7ff76814e800 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.540+0000 7ff76ce9a640 1 -- 172.21.15.94:0/3565305602 >> 172.21.15.94:0/3565305602 conn(0x7ff7680ac590 msgr2=0x7ff7680ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.541+0000 7ff76ce9a640 1 -- 172.21.15.94:0/3565305602 shutdown_connections 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.541+0000 7ff76ce9a640 1 -- 172.21.15.94:0/3565305602 wait complete. 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 Processor -- start 2024-03-06T07:31:07.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 -- start start 2024-03-06T07:31:07.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff76806ca70 0x7ff768156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680b0070 0x7ff7680c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff768155c30 0x7ff76814dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76806c550 con 0x7ff76806ca70 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76806c270 con 0x7ff7680b0070 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.542+0000 7ff76ce9a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76806da80 con 0x7ff768155c30 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff76806ca70 0x7ff768156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff76806ca70 0x7ff768156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:38478/0 (socket says 172.21.15.94:38478) 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 -- 172.21.15.94:0/1265587209 learned_addr learned my addr 172.21.15.94:0/1265587209 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff765d9b640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680b0070 0x7ff7680c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff766d9d640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff768155c30 0x7ff76814dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 -- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff768155c30 msgr2=0x7ff76814dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff768155c30 0x7ff76814dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 -- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680b0070 msgr2=0x7ff7680c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680b0070 0x7ff7680c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 -- 172.21.15.94:0/1265587209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff768158660 con 0x7ff76806ca70 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff765d9b640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680b0070 0x7ff7680c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff766d9d640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff768155c30 0x7ff76814dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76659c640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff76806ca70 0x7ff768156820 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7ff75000c500 tx=0x7ff75000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff75effd640 1 -- 172.21.15.94:0/1265587209 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7500052d0 con 0x7ff76806ca70 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff75effd640 1 -- 172.21.15.94:0/1265587209 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff750005490 con 0x7ff76806ca70 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff768158940 con 0x7ff76806ca70 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff75effd640 1 -- 172.21.15.94:0/1265587209 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750012870 con 0x7ff76806ca70 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.543+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff768158f00 con 0x7ff76806ca70 2024-03-06T07:31:07.549 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.544+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff768159100 con 0x7ff76806ca70 2024-03-06T07:31:07.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.545+0000 7ff75effd640 1 -- 172.21.15.94:0/1265587209 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff750012a10 con 0x7ff76806ca70 2024-03-06T07:31:07.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.545+0000 7ff75effd640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b4d0 0x7ff74806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.545+0000 7ff75effd640 1 -- 172.21.15.94:0/1265587209 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff75008a220 con 0x7ff76806ca70 2024-03-06T07:31:07.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.545+0000 7ff765d9b640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b4d0 0x7ff74806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.550 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.546+0000 7ff765d9b640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b4d0 0x7ff74806d990 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff758007fd0 tx=0x7ff758009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.549+0000 7ff75effd640 1 -- 172.21.15.94:0/1265587209 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff768159100 con 0x7ff76806ca70 2024-03-06T07:31:07.761 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:07.759+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff76804f330 con 0x7ff76806ca70 2024-03-06T07:31:08.442 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.440+0000 7ff75effd640 1 -- 172.21.15.94:0/1265587209 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v215) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff750058720 con 0x7ff76806ca70 2024-03-06T07:31:08.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b4d0 msgr2=0x7ff74806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b4d0 0x7ff74806d990 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff758007fd0 tx=0x7ff758009040 comp rx=0 tx=0).stop 2024-03-06T07:31:08.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff76806ca70 msgr2=0x7ff768156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff76806ca70 0x7ff768156820 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7ff75000c500 tx=0x7ff75000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 shutdown_connections 2024-03-06T07:31:08.445 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7ff74806b4d0 0x7ff74806d990 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff768155c30 0x7ff76814dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff7680b0070 0x7ff7680c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 --2- 172.21.15.94:0/1265587209 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff76806ca70 0x7ff768156820 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 >> 172.21.15.94:0/1265587209 conn(0x7ff7680ac590 msgr2=0x7ff7680ae4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 shutdown_connections 2024-03-06T07:31:08.446 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.443+0000 7ff76ce9a640 1 -- 172.21.15.94:0/1265587209 wait complete. 2024-03-06T07:31:08.457 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:31:08.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.562+0000 7fbaa9eec640 1 Processor -- start 2024-03-06T07:31:08.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.562+0000 7fbaa9eec640 1 -- start start 2024-03-06T07:31:08.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa9eec640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbaa40c22b0 0x7fbaa4151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa9eec640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbaa4153390 0x7fbaa4153800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.565 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa9eec640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa406e850 con 0x7fbaa40a9830 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa9eec640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa406d2b0 con 0x7fbaa40c22b0 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa9eec640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa406d590 con 0x7fbaa4153390 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbaa40c22b0 0x7fbaa4151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbaa40a9830 0x7fbaa414ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47076/0 (socket says 172.21.15.94:47076) 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fbaa40c22b0 0x7fbaa4151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44570/0 (socket says 172.21.15.94:44570) 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa2ffd640 1 -- 172.21.15.94:0/3695602787 learned_addr learned my addr 172.21.15.94:0/3695602787 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa3fff640 1 --2- 172.21.15.94:0/3695602787 >> v2:172.21.15.145:3300/0 conn(0x7fbaa4153390 0x7fbaa4153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa37fe640 1 -- 172.21.15.94:0/3695602787 >> v2:172.21.15.145:3300/0 conn(0x7fbaa4153390 msgr2=0x7fbaa4153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa37fe640 1 --2- 172.21.15.94:0/3695602787 >> v2:172.21.15.145:3300/0 conn(0x7fbaa4153390 0x7fbaa4153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa37fe640 1 -- 172.21.15.94:0/3695602787 >> v1:172.21.15.94:6789/0 conn(0x7fbaa40a9830 legacy=0x7fbaa414ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa37fe640 1 -- 172.21.15.94:0/3695602787 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa406d870 con 0x7fbaa40c22b0 2024-03-06T07:31:08.566 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.563+0000 7fbaa27fc640 1 -- 172.21.15.94:0/3695602787 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411744084 0 0) 0x7fba98001110 con 0x7fbaa40a9830 2024-03-06T07:31:08.567 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa37fe640 1 --2- 172.21.15.94:0/3695602787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa4151e10 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fba940097a0 tx=0x7fba9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a444372663f318ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa27fc640 1 -- 172.21.15.94:0/3695602787 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba94035020 con 0x7fbaa40c22b0 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa27fc640 1 -- 172.21.15.94:0/3695602787 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba94031830 con 0x7fbaa40c22b0 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa27fc640 1 -- 172.21.15.94:0/3695602787 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba94031c00 con 0x7fbaa40c22b0 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa9eec640 1 -- 172.21.15.94:0/3695602787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 msgr2=0x7fbaa4151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/3695602787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa4151e10 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fba940097a0 tx=0x7fba9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa9eec640 1 -- 172.21.15.94:0/3695602787 shutdown_connections 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/3695602787 >> v2:172.21.15.145:3300/0 conn(0x7fbaa4153390 0x7fbaa4153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/3695602787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa4151e10 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.564+0000 7fbaa9eec640 1 -- 172.21.15.94:0/3695602787 >> 172.21.15.94:0/3695602787 conn(0x7fbaa40c65b0 msgr2=0x7fbaa40c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.565+0000 7fbaa9eec640 1 -- 172.21.15.94:0/3695602787 shutdown_connections 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.565+0000 7fbaa9eec640 1 -- 172.21.15.94:0/3695602787 wait complete. 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 Processor -- start 2024-03-06T07:31:08.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 -- start start 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa40a9830 0x7fbaa41e9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa40c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa4153390 0x7fbaa40c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa414f7e0 con 0x7fbaa40c22b0 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa414f660 con 0x7fbaa40a9830 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa9eec640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa40c2690 con 0x7fbaa4153390 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.566+0000 7fbaa2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa40c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa40c4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44576/0 (socket says 172.21.15.94:44576) 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 -- 172.21.15.94:0/2715147783 learned_addr learned my addr 172.21.15.94:0/2715147783 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:08.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa3fff640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa4153390 0x7fbaa40c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa37fe640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa40a9830 0x7fbaa41e9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 -- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa4153390 msgr2=0x7fbaa40c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa4153390 0x7fbaa40c5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 -- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa40a9830 msgr2=0x7fbaa41e9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa40a9830 0x7fbaa41e9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 -- 172.21.15.94:0/2715147783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa41e87e0 con 0x7fbaa40c22b0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa2ffd640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa40c4180 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c7b0 tx=0x7fba9800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fba8bfff640 1 -- 172.21.15.94:0/2715147783 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98018070 con 0x7fbaa40c22b0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fba8bfff640 1 -- 172.21.15.94:0/2715147783 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba980072d0 con 0x7fbaa40c22b0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa41e97b0 con 0x7fbaa40c22b0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fba8bfff640 1 -- 172.21.15.94:0/2715147783 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba980047e0 con 0x7fbaa40c22b0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.567+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa41df5d0 con 0x7fbaa40c22b0 2024-03-06T07:31:08.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.568+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa4073510 con 0x7fbaa40c22b0 2024-03-06T07:31:08.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.569+0000 7fba8bfff640 1 -- 172.21.15.94:0/2715147783 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba98004980 con 0x7fbaa40c22b0 2024-03-06T07:31:08.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.569+0000 7fba8bfff640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba9006b4d0 0x7fba9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.569+0000 7fba8bfff640 1 -- 172.21.15.94:0/2715147783 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fba98089880 con 0x7fbaa40c22b0 2024-03-06T07:31:08.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.569+0000 7fbaa37fe640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba9006b4d0 0x7fba9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.570+0000 7fbaa37fe640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba9006b4d0 0x7fba9006d990 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fba94036000 tx=0x7fba94008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.573+0000 7fba8bfff640 1 -- 172.21.15.94:0/2715147783 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaa406e850 con 0x7fbaa40c22b0 2024-03-06T07:31:08.749 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:08.747+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbaa40ac1f0 con 0x7fbaa40c22b0 2024-03-06T07:31:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.447+0000 7fba8bfff640 1 -- 172.21.15.94:0/2715147783 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v215) v1 ==== 181+0+0 (secure 0 0 0) 0x7fba98004c70 con 0x7fbaa40c22b0 2024-03-06T07:31:09.449 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:31:09.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba9006b4d0 msgr2=0x7fba9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba9006b4d0 0x7fba9006d990 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fba94036000 tx=0x7fba94008030 comp rx=0 tx=0).stop 2024-03-06T07:31:09.452 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 msgr2=0x7fbaa40c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa40c4180 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7fba9800c7b0 tx=0x7fba9800cc70 comp rx=0 tx=0).stop 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 shutdown_connections 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fba9006b4d0 0x7fba9006d990 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbaa4153390 0x7fbaa40c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbaa40c22b0 0x7fbaa40c4180 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 --2- 172.21.15.94:0/2715147783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbaa40a9830 0x7fbaa41e9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.450+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 >> 172.21.15.94:0/2715147783 conn(0x7fbaa40c65b0 msgr2=0x7fbaa4154be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.451+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 shutdown_connections 2024-03-06T07:31:09.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.451+0000 7fbaa9eec640 1 -- 172.21.15.94:0/2715147783 wait complete. 2024-03-06T07:31:09.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:31:09.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.569+0000 7f939e18a640 1 Processor -- start 2024-03-06T07:31:09.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.569+0000 7f939e18a640 1 -- start start 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.569+0000 7f939e18a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f939814ebb0 0x7f93981521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f939e18a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f93980c28d0 0x7f939806dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f939e18a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939814ef90 con 0x7f93980a9980 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f939e18a640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9398155530 con 0x7f939814ebb0 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f939e18a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9398155a60 con 0x7f93980c28d0 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f93977fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f939814ebb0 0x7f93981521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f9396ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f93980a9980 0x7f93980a9fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47082/0 (socket says 172.21.15.94:47082) 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f9396ffd640 1 -- 172.21.15.94:0/1851711758 learned_addr learned my addr 172.21.15.94:0/1851711758 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:09.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f9397fff640 1 --2- 172.21.15.94:0/1851711758 >> v2:172.21.15.145:3300/0 conn(0x7f93980c28d0 0x7f939806dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f93977fe640 1 -- 172.21.15.94:0/1851711758 >> v2:172.21.15.145:3300/0 conn(0x7f93980c28d0 msgr2=0x7f939806dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f93977fe640 1 --2- 172.21.15.94:0/1851711758 >> v2:172.21.15.145:3300/0 conn(0x7f93980c28d0 0x7f939806dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f93977fe640 1 -- 172.21.15.94:0/1851711758 >> v1:172.21.15.94:6789/0 conn(0x7f93980a9980 legacy=0x7f93980a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f93977fe640 1 -- 172.21.15.94:0/1851711758 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398155780 con 0x7f939814ebb0 2024-03-06T07:31:09.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.570+0000 7f93967fc640 1 -- 172.21.15.94:0/1851711758 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722309792 0 0) 0x7f938c001110 con 0x7f93980a9980 2024-03-06T07:31:09.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f93977fe640 1 --2- 172.21.15.94:0/1851711758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f93981521b0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f939806d290 tx=0x7f938802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7414386566b5fd27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f93967fc640 1 -- 172.21.15.94:0/1851711758 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880315a0 con 0x7f939814ebb0 2024-03-06T07:31:09.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f93967fc640 1 -- 172.21.15.94:0/1851711758 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9388037070 con 0x7f939814ebb0 2024-03-06T07:31:09.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f93967fc640 1 -- 172.21.15.94:0/1851711758 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9388031960 con 0x7f939814ebb0 2024-03-06T07:31:09.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 -- 172.21.15.94:0/1851711758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 msgr2=0x7f93981521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 --2- 172.21.15.94:0/1851711758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f93981521b0 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f939806d290 tx=0x7f938802b840 comp rx=0 tx=0).stop 2024-03-06T07:31:09.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 -- 172.21.15.94:0/1851711758 shutdown_connections 2024-03-06T07:31:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 --2- 172.21.15.94:0/1851711758 >> v2:172.21.15.145:3300/0 conn(0x7f93980c28d0 0x7f939806dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 --2- 172.21.15.94:0/1851711758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f93981521b0 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 -- 172.21.15.94:0/1851711758 >> 172.21.15.94:0/1851711758 conn(0x7f93980c65b0 msgr2=0x7f93980c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 -- 172.21.15.94:0/1851711758 shutdown_connections 2024-03-06T07:31:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.571+0000 7f939e18a640 1 -- 172.21.15.94:0/1851711758 wait complete. 2024-03-06T07:31:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.572+0000 7f939e18a640 1 Processor -- start 2024-03-06T07:31:09.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f939e18a640 1 -- start start 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f939e18a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93980a9980 0x7f939815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f939e18a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93980c28d0 0x7f9398156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f939e18a640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f9398157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f939e18a640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9398151c90 con 0x7f939814ebb0 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f939e18a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93981519b0 con 0x7f93980a9980 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f939e18a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93980c2d80 con 0x7f93980c28d0 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f9397fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f9398157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f9397fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f9398157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44584/0 (socket says 172.21.15.94:44584) 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.573+0000 7f9397fff640 1 -- 172.21.15.94:0/1721815201 learned_addr learned my addr 172.21.15.94:0/1721815201 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:09.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f93977fe640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93980a9980 0x7f939815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f9396ffd640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93980c28d0 0x7f9398156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f9397fff640 1 -- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93980c28d0 msgr2=0x7f9398156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f9397fff640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93980c28d0 0x7f9398156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f9397fff640 1 -- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93980a9980 msgr2=0x7f939815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f9397fff640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93980a9980 0x7f939815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f9397fff640 1 -- 172.21.15.94:0/1721815201 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93981ec6d0 con 0x7f939814ebb0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f9397fff640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f9398157320 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f9390007e70 tx=0x7f939000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f937ffff640 1 -- 172.21.15.94:0/1721815201 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390014620 con 0x7f939814ebb0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f937ffff640 1 -- 172.21.15.94:0/1721815201 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9390019070 con 0x7f939814ebb0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f937ffff640 1 -- 172.21.15.94:0/1721815201 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390014a20 con 0x7f939814ebb0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93981ec9b0 con 0x7f939814ebb0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.574+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93981ecf70 con 0x7f939814ebb0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.575+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93981ed170 con 0x7f939814ebb0 2024-03-06T07:31:09.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.575+0000 7f937ffff640 1 -- 172.21.15.94:0/1721815201 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9390014bc0 con 0x7f939814ebb0 2024-03-06T07:31:09.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.576+0000 7f937ffff640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.576+0000 7f937ffff640 1 -- 172.21.15.94:0/1721815201 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f939008abf0 con 0x7f939814ebb0 2024-03-06T07:31:09.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.576+0000 7f93977fe640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.577+0000 7f93977fe640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b4d0 0x7f938406d990 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f9388036000 tx=0x7f9388008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.580+0000 7f937ffff640 1 -- 172.21.15.94:0/1721815201 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93981ed170 con 0x7f939814ebb0 2024-03-06T07:31:09.752 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:09.750+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f939804f330 con 0x7f939814ebb0 2024-03-06T07:31:10.451 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.449+0000 7f937ffff640 1 -- 172.21.15.94:0/1721815201 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v216) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9390059980 con 0x7f939814ebb0 2024-03-06T07:31:10.451 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'cephfs_data' removed 2024-03-06T07:31:10.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.452+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b4d0 msgr2=0x7f938406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.452+0000 7f939e18a640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b4d0 0x7f938406d990 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f9388036000 tx=0x7f9388008030 comp rx=0 tx=0).stop 2024-03-06T07:31:10.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.452+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 msgr2=0x7f9398157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.452+0000 7f939e18a640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f9398157320 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f9390007e70 tx=0x7f939000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.453+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 shutdown_connections 2024-03-06T07:31:10.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.453+0000 7f939e18a640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f938406b4d0 0x7f938406d990 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.453+0000 7f939e18a640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f939814ebb0 0x7f9398157320 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.455 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.453+0000 7f939e18a640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93980c28d0 0x7f9398156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.453+0000 7f939e18a640 1 --2- 172.21.15.94:0/1721815201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93980a9980 0x7f939815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.453+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 >> 172.21.15.94:0/1721815201 conn(0x7f93980c65b0 msgr2=0x7f939814da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.453+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 shutdown_connections 2024-03-06T07:31:10.456 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.454+0000 7f939e18a640 1 -- 172.21.15.94:0/1721815201 wait complete. 2024-03-06T07:31:10.469 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.571+0000 7f4f0d666640 1 Processor -- start 2024-03-06T07:31:10.573 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.571+0000 7f4f0d666640 1 -- start start 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f0d666640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f08152730 0x7f4f080c3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f0d666640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f08152b10 con 0x7f4f080c6710 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f0d666640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f08158ad0 con 0x7f4f080c3750 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f0d666640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f08158db0 con 0x7f4f08152730 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f06ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f08152730 0x7f4f080c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f067fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f4f080c6710 0x7f4f081554c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47090/0 (socket says 172.21.15.94:47090) 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f06ffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4f08152730 0x7f4f080c3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44592/0 (socket says 172.21.15.94:44592) 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f067fc640 1 -- 172.21.15.94:0/1143903718 learned_addr learned my addr 172.21.15.94:0/1143903718 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f06ffd640 1 -- 172.21.15.94:0/1143903718 >> v1:172.21.15.145:6789/0 conn(0x7f4f080c3750 legacy=0x7f4f0815fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f06ffd640 1 -- 172.21.15.94:0/1143903718 >> v1:172.21.15.94:6789/0 conn(0x7f4f080c6710 legacy=0x7f4f081554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f06ffd640 1 -- 172.21.15.94:0/1143903718 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08159090 con 0x7f4f08152730 2024-03-06T07:31:10.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.572+0000 7f4f05ffb640 1 -- 172.21.15.94:0/1143903718 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900236947 0 0) 0x7f4f08152b10 con 0x7f4f080c6710 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.573+0000 7f4f06ffd640 1 --2- 172.21.15.94:0/1143903718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f08152730 0x7f4f080c3210 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f4f000097a0 tx=0x7f4f0002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b2f966855de0d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.573+0000 7f4f05ffb640 1 -- 172.21.15.94:0/1143903718 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f000315a0 con 0x7f4f08152730 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.573+0000 7f4f05ffb640 1 -- 172.21.15.94:0/1143903718 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f00037070 con 0x7f4f08152730 2024-03-06T07:31:10.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.573+0000 7f4f05ffb640 1 -- 172.21.15.94:0/1143903718 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00031960 con 0x7f4f08152730 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.574+0000 7f4f0d666640 1 -- 172.21.15.94:0/1143903718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f08152730 msgr2=0x7f4f080c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.574+0000 7f4f0d666640 1 --2- 172.21.15.94:0/1143903718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f08152730 0x7f4f080c3210 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f4f000097a0 tx=0x7f4f0002b840 comp rx=0 tx=0).stop 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.574+0000 7f4f0d666640 1 -- 172.21.15.94:0/1143903718 shutdown_connections 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.574+0000 7f4f0d666640 1 --2- 172.21.15.94:0/1143903718 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f08152730 0x7f4f080c3210 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.574+0000 7f4f0d666640 1 -- 172.21.15.94:0/1143903718 >> 172.21.15.94:0/1143903718 conn(0x7f4f080aa5c0 msgr2=0x7f4f080aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.574+0000 7f4f0d666640 1 -- 172.21.15.94:0/1143903718 shutdown_connections 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.574+0000 7f4f0d666640 1 -- 172.21.15.94:0/1143903718 wait complete. 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.575+0000 7f4f0d666640 1 Processor -- start 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.575+0000 7f4f0d666640 1 -- start start 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f080c3750 0x7f4f0806d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f080c6710 0x7f4f081ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f08152730 0x7f4f081ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f08158870 con 0x7f4f080c6710 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f081586f0 con 0x7f4f080c3750 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080a7cd0 con 0x7f4f08152730 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f080c6710 0x7f4f081ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f080c6710 0x7f4f081ee3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44604/0 (socket says 172.21.15.94:44604) 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 -- 172.21.15.94:0/2956433652 learned_addr learned my addr 172.21.15.94:0/2956433652 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f06ffd640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f080c3750 0x7f4f0806d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 -- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f08152730 msgr2=0x7f4f081ee8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f08152730 0x7f4f081ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 -- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f080c3750 msgr2=0x7f4f0806d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f080c3750 0x7f4f0806d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 -- 172.21.15.94:0/2956433652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0806fa80 con 0x7f4f080c6710 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f06ffd640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f080c3750 0x7f4f0806d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f067fc640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f080c6710 0x7f4f081ee3b0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00c820 tx=0x7f4efc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4eef7fe640 1 -- 172.21.15.94:0/2956433652 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc018070 con 0x7f4f080c6710 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4eef7fe640 1 -- 172.21.15.94:0/2956433652 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc00a640 con 0x7f4f080c6710 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4eef7fe640 1 -- 172.21.15.94:0/2956433652 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc00aa30 con 0x7f4f080c6710 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0814c180 con 0x7f4f080c6710 2024-03-06T07:31:10.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.576+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0806dc60 con 0x7f4f080c6710 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.578+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0804f330 con 0x7f4f080c6710 2024-03-06T07:31:10.581 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.578+0000 7f4eef7fe640 1 -- 172.21.15.94:0/2956433652 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4efc01b070 con 0x7f4f080c6710 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.578+0000 7f4eef7fe640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ef406b500 0x7f4ef406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.579+0000 7f4eef7fe640 1 -- 172.21.15.94:0/2956433652 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f4efc089720 con 0x7f4f080c6710 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.579+0000 7f4f06ffd640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ef406b500 0x7f4ef406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.582 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.579+0000 7f4f06ffd640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ef406b500 0x7f4ef406d9c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4f00036000 tx=0x7f4f00008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.584 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.583+0000 7f4eef7fe640 1 -- 172.21.15.94:0/2956433652 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f0806fa80 con 0x7f4f080c6710 2024-03-06T07:31:10.763 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.761+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4f08073330 con 0x7f4f080c6710 2024-03-06T07:31:10.766 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.765+0000 7f4eef7fe640 1 -- 172.21.15.94:0/2956433652 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4efc0586a0 con 0x7f4f080c6710 2024-03-06T07:31:10.768 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.766+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ef406b500 msgr2=0x7f4ef406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.766+0000 7f4f0d666640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ef406b500 0x7f4ef406d9c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f4f00036000 tx=0x7f4f00008030 comp rx=0 tx=0).stop 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f080c6710 msgr2=0x7f4f081ee3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f080c6710 0x7f4f081ee3b0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00c820 tx=0x7f4efc00cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 shutdown_connections 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f4ef406b500 0x7f4ef406d9c0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4f08152730 0x7f4f081ee8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4f080c6710 0x7f4f081ee3b0 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 --2- 172.21.15.94:0/2956433652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4f080c3750 0x7f4f0806d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 >> 172.21.15.94:0/2956433652 conn(0x7f4f080aa5c0 msgr2=0x7f4f080c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 shutdown_connections 2024-03-06T07:31:10.769 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.767+0000 7f4f0d666640 1 -- 172.21.15.94:0/2956433652 wait complete. 2024-03-06T07:31:10.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-03-06T07:31:10.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.880+0000 7f04b69e2640 1 Processor -- start 2024-03-06T07:31:10.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.880+0000 7f04b69e2640 1 -- start start 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04b69e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 0x7f04b014ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04b69e2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b0155bb0 con 0x7f04b014f140 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04b69e2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b00aa660 con 0x7f04b01550c0 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04b69e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00c4610 con 0x7f04b00aa180 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04b4f7f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f04b014f140 0x7f04b00c5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47094/0 (socket says 172.21.15.94:47094) 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04b4f7f640 1 -- 172.21.15.94:0/1754380588 learned_addr learned my addr 172.21.15.94:0/1754380588 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04af7fe640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 0x7f04b014ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339371651 0 0) 0x7f04b0155bb0 con 0x7f04b014f140 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.881+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0498003520 con 0x7f04b014f140 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 764494862 0 0) 0x7f0498003520 con 0x7f04b014f140 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 msgr2=0x7f04b014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 0x7f04b014ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 >> v1:172.21.15.145:6789/0 conn(0x7f04b01550c0 legacy=0x7f04b01554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b00aa940 con 0x7f04b014f140 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f04a80019f0 con 0x7f04b014f140 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 >> v1:172.21.15.94:6789/0 conn(0x7f04b014f140 legacy=0x7f04b00c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 0x7f04980039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0498003f60 0x7f0498004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0498004d40 0x7f04980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00c4610 con 0x7f0498004d40 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049800de10 con 0x7f0498003f60 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049800e010 con 0x7f04b00aa180 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04b4f7f640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0498004d40 0x7f04980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04affff640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 0x7f04980039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04af7fe640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0498003f60 0x7f0498004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04b4f7f640 1 -- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 msgr2=0x7f04980039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04b4f7f640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b00aa180 0x7f04980039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04b4f7f640 1 -- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0498003f60 msgr2=0x7f0498004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04b4f7f640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0498003f60 0x7f0498004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.882+0000 7f04b4f7f640 1 -- 172.21.15.94:0/1754380588 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0498003520 con 0x7f0498004d40 2024-03-06T07:31:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.883+0000 7f04b4f7f640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0498004d40 0x7f04980051c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f04a800a830 tx=0x7f0498005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10fb7ca999b7907 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.883+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a800a480 con 0x7f0498004d40 2024-03-06T07:31:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.883+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8034650 con 0x7f0498004d40 2024-03-06T07:31:10.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.883+0000 7f04aeffd640 1 -- 172.21.15.94:0/1754380588 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8034a40 con 0x7f0498004d40 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.884+0000 7f04b69e2640 1 -- 172.21.15.94:0/1754380588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0498004d40 msgr2=0x7f04980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.884+0000 7f04b69e2640 1 --2- 172.21.15.94:0/1754380588 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0498004d40 0x7f04980051c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f04a800a830 tx=0x7f0498005830 comp rx=0 tx=0).stop 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.884+0000 7f04b4f7f640 1 -- 172.21.15.94:0/1754380588 reap_dead start 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.884+0000 7f04b69e2640 1 -- 172.21.15.94:0/1754380588 shutdown_connections 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.884+0000 7f04b69e2640 1 -- 172.21.15.94:0/1754380588 >> 172.21.15.94:0/1754380588 conn(0x7f04b00c6710 msgr2=0x7f04b00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.884+0000 7f04b69e2640 1 -- 172.21.15.94:0/1754380588 shutdown_connections 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.884+0000 7f04b69e2640 1 -- 172.21.15.94:0/1754380588 wait complete. 2024-03-06T07:31:10.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.885+0000 7f04b69e2640 1 Processor -- start 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.885+0000 7f04b69e2640 1 -- start start 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b0152240 0x7f04b01e9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0148060 0x7f04b01e9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b00a8e70 0x7f04b01e8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00c1310 con 0x7f04b00a8e70 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b014f5f0 con 0x7f04b0152240 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b00c1490 con 0x7f04b0148060 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b00a8e70 0x7f04b01e8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b00a8e70 0x7f04b01e8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44616/0 (socket says 172.21.15.94:44616) 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 -- 172.21.15.94:0/4103035222 learned_addr learned my addr 172.21.15.94:0/4103035222 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04affff640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b0152240 0x7f04b01e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04af7fe640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0148060 0x7f04b01e9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 -- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0148060 msgr2=0x7f04b01e9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0148060 0x7f04b01e9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 -- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b0152240 msgr2=0x7f04b01e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b0152240 0x7f04b01e9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 -- 172.21.15.94:0/4103035222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b0150500 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b4f7f640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b00a8e70 0x7f04b01e8130 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f04a800a830 tx=0x7f04b00c3d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04acff9640 1 -- 172.21.15.94:0/4103035222 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8043070 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04acff9640 1 -- 172.21.15.94:0/4103035222 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a80019f0 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b01507e0 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04acff9640 1 -- 172.21.15.94:0/4103035222 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8039040 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.886+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b0150da0 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.887+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b004f330 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.888+0000 7f04acff9640 1 -- 172.21.15.94:0/4103035222 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04a8046070 con 0x7f04b00a8e70 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.888+0000 7f04acff9640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049c06b4d0 0x7f049c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.892 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.888+0000 7f04acff9640 1 -- 172.21.15.94:0/4103035222 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f04a80ab2e0 con 0x7f04b00a8e70 2024-03-06T07:31:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.889+0000 7f04affff640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049c06b4d0 0x7f049c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.893 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.889+0000 7f04affff640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049c06b4d0 0x7f049c06d990 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f04a00017e0 tx=0x7f04a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.894 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:10.893+0000 7f04acff9640 1 -- 172.21.15.94:0/4103035222 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04a8079c40 con 0x7f04b00a8e70 2024-03-06T07:31:11.133 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.131+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f04b0154be0 con 0x7f04b00a8e70 2024-03-06T07:31:11.461 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.459+0000 7f04acff9640 1 -- 172.21.15.94:0/4103035222 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v701) v1 ==== 139+0+0 (secure 0 0 0) 0x7f04a807dce0 con 0x7f04b00a8e70 2024-03-06T07:31:11.463 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.461+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049c06b4d0 msgr2=0x7f049c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.461+0000 7f04b69e2640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049c06b4d0 0x7f049c06d990 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f04a00017e0 tx=0x7f04a0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.461+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b00a8e70 msgr2=0x7f04b01e8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.461+0000 7f04b69e2640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b00a8e70 0x7f04b01e8130 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f04a800a830 tx=0x7f04b00c3d10 comp rx=0 tx=0).stop 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 shutdown_connections 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f049c06b4d0 0x7f049c06d990 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f04b00a8e70 0x7f04b01e8130 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0148060 0x7f04b01e9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 --2- 172.21.15.94:0/4103035222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04b0152240 0x7f04b01e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 >> 172.21.15.94:0/4103035222 conn(0x7f04b00c6710 msgr2=0x7f04b0153940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 shutdown_connections 2024-03-06T07:31:11.464 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:11.462+0000 7f04b69e2640 1 -- 172.21.15.94:0/4103035222 wait complete. 2024-03-06T07:31:11.476 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. ... ERROR 2024-03-06T07:31:11.476 INFO:tasks.cephfs_test_runner: 2024-03-06T07:31:11.476 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:31:11.476 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:31:11.476 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-03-06T07:31:11.476 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:31:11.476 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:Ran 12 tests in 740.296s 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-03-06T07:31:11.477 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-03-06T07:31:11.478 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:31:11.478 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:31:11.478 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:31:11.478 INFO:tasks.cephfs_test_runner: 2024-03-06T07:31:11.478 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:31:13.090 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=41b432f0aa724dafa3ad74d039e80be1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-03-06T07:31:13.092 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:31:13.101 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-06T07:31:13.110 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-06T07:31:13.110 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-06T07:31:13.110 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:13.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19702 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-03-06T07:31:13.133 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-06T07:31:13.133 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:13.152 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056361,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-03-06T07:31:13.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:13.153 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2024-03-06T07:31:13.192 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:13.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:13.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:13.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:13.264 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2024-03-06T07:31:13.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:13.357 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-03-06T07:31:13.375 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:31:13.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:13.375 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:31:13.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:13.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link delete ceph-brx 2024-03-06T07:31:13.567 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:31:13.577 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:31:13.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:13.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.681+0000 7f664433e640 1 Processor -- start 2024-03-06T07:31:13.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.681+0000 7f664433e640 1 -- start start 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.682+0000 7f664433e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.682+0000 7f664433e640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f663c0c6000 0x7f663c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.682+0000 7f664433e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c153a60 con 0x7f663c150580 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.682+0000 7f664433e640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663c153d40 con 0x7f663c0c6000 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.682+0000 7f664433e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c06c2c0 con 0x7f663c0c1320 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.682+0000 7f66428db640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f663c0c6000 0x7f663c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.682+0000 7f66420da640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f663c150580 0x7f663c06dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47098/0 (socket says 172.21.15.94:47098) 2024-03-06T07:31:13.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66420da640 1 -- 172.21.15.94:0/2506174454 learned_addr learned my addr 172.21.15.94:0/2506174454 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:13.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66428db640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f663c0c6000 0x7f663c156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44624/0 (socket says 172.21.15.94:44624) 2024-03-06T07:31:13.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66418d9640 1 --2- 172.21.15.94:0/2506174454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66428db640 1 -- 172.21.15.94:0/2506174454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 msgr2=0x7f663c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66428db640 1 --2- 172.21.15.94:0/2506174454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66428db640 1 -- 172.21.15.94:0/2506174454 >> v1:172.21.15.94:6789/0 conn(0x7f663c150580 legacy=0x7f663c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66428db640 1 -- 172.21.15.94:0/2506174454 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c06c5a0 con 0x7f663c0c6000 2024-03-06T07:31:13.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66410d8640 1 -- 172.21.15.94:0/2506174454 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092168125 0 0) 0x7f663c153a60 con 0x7f663c150580 2024-03-06T07:31:13.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.683+0000 7f66428db640 1 --2- 172.21.15.94:0/2506174454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c156bf0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f66380097a0 tx=0x7f663802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f203131a87addf8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f66410d8640 1 -- 172.21.15.94:0/2506174454 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380315a0 con 0x7f663c0c6000 2024-03-06T07:31:13.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f66410d8640 1 -- 172.21.15.94:0/2506174454 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6638037070 con 0x7f663c0c6000 2024-03-06T07:31:13.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f66410d8640 1 -- 172.21.15.94:0/2506174454 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638031960 con 0x7f663c0c6000 2024-03-06T07:31:13.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 -- 172.21.15.94:0/2506174454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 msgr2=0x7f663c156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.690 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 --2- 172.21.15.94:0/2506174454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c156bf0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f66380097a0 tx=0x7f663802b840 comp rx=0 tx=0).stop 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 -- 172.21.15.94:0/2506174454 shutdown_connections 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 --2- 172.21.15.94:0/2506174454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c156bf0 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 --2- 172.21.15.94:0/2506174454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 -- 172.21.15.94:0/2506174454 >> 172.21.15.94:0/2506174454 conn(0x7f663c0c6710 msgr2=0x7f663c0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 -- 172.21.15.94:0/2506174454 shutdown_connections 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.684+0000 7f664433e640 1 -- 172.21.15.94:0/2506174454 wait complete. 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.685+0000 7f664433e640 1 Processor -- start 2024-03-06T07:31:13.691 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f664433e640 1 -- start start 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f664433e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f664433e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c1474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f664433e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f663c150580 0x7f663c0bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f664433e640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c06e5a0 con 0x7f663c0c6000 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f664433e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c06e420 con 0x7f663c150580 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f664433e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0c0c40 con 0x7f663c0c1320 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f66418d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c1474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f66418d9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c1474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44628/0 (socket says 172.21.15.94:44628) 2024-03-06T07:31:13.692 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f66418d9640 1 -- 172.21.15.94:0/3274516389 learned_addr learned my addr 172.21.15.94:0/3274516389 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f66420da640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.686+0000 7f66418d9640 1 -- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 msgr2=0x7f663c1ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f66418d9640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1ee700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f66418d9640 1 -- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f663c150580 msgr2=0x7f663c0bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f66428db640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f663c150580 0x7f663c0bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f66418d9640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f663c150580 0x7f663c0bfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f66418d9640 1 -- 172.21.15.94:0/3274516389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c150f70 con 0x7f663c0c6000 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f66418d9640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c1474a0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f662c00e9b0 tx=0x7f662c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.693 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f6632ffd640 1 -- 172.21.15.94:0/3274516389 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c017070 con 0x7f663c0c6000 2024-03-06T07:31:13.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f6632ffd640 1 -- 172.21.15.94:0/3274516389 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c00c9a0 con 0x7f663c0c6000 2024-03-06T07:31:13.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c148d10 con 0x7f663c0c6000 2024-03-06T07:31:13.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f6632ffd640 1 -- 172.21.15.94:0/3274516389 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c00cd90 con 0x7f663c0c6000 2024-03-06T07:31:13.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.687+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c1492d0 con 0x7f663c0c6000 2024-03-06T07:31:13.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.688+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c04f330 con 0x7f663c0c6000 2024-03-06T07:31:13.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.688+0000 7f6632ffd640 1 -- 172.21.15.94:0/3274516389 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f662c01a070 con 0x7f663c0c6000 2024-03-06T07:31:13.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.689+0000 7f6632ffd640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f662006b500 0x7f662006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.689+0000 7f6632ffd640 1 -- 172.21.15.94:0/3274516389 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f662c08a180 con 0x7f663c0c6000 2024-03-06T07:31:13.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.689+0000 7f66420da640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f662006b500 0x7f662006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.689+0000 7f66420da640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f662006b500 0x7f662006d9c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f66340012e0 tx=0x7f6634009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.693+0000 7f6632ffd640 1 -- 172.21.15.94:0/3274516389 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f663c150f70 con 0x7f663c0c6000 2024-03-06T07:31:13.863 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.861+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f663c0c5de0 con 0x7f662006b500 2024-03-06T07:31:13.864 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.862+0000 7f6632ffd640 1 -- 172.21.15.94:0/3274516389 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13564 (secure 0 0 0) 0x7f663c0c5de0 con 0x7f662006b500 2024-03-06T07:31:13.864 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:13.864 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:31:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f662006b500 msgr2=0x7f662006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f662006b500 0x7f662006d9c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f66340012e0 tx=0x7f6634009040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.866 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 msgr2=0x7f663c1474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c1474a0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f662c00e9b0 tx=0x7f662c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 shutdown_connections 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f662006b500 0x7f662006d9c0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f663c150580 0x7f663c0bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f663c0c6000 0x7f663c1474a0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 --2- 172.21.15.94:0/3274516389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f663c0c1320 0x7f663c1ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 >> 172.21.15.94:0/3274516389 conn(0x7f663c0c6710 msgr2=0x7f663c153630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 shutdown_connections 2024-03-06T07:31:13.867 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.864+0000 7f664433e640 1 -- 172.21.15.94:0/3274516389 wait complete. 2024-03-06T07:31:13.876 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-03-06T07:31:12.798565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":283548,"kb_used_data":21600,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754691172,"statfs":{"total":773094113280,"available":772803760128,"internally_reserved":0,"allocated":22118400,"data_stored":11837112,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-67,"num_read_kb":-73,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001554"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":542,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.801349+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:31:10.801349+0000","last_peered":"2024-03-06T07:31:10.801349+0000","last_clean":"2024-03-06T07:31:10.801349+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:31:10.801349+0000","last_undegraded":"2024-03-06T07:31:10.801349+0000","last_fullsized":"2024-03-06T07:31:10.801349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33928,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337912,"statfs":{"total":96636764160,"available":96602021888,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38092,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333748,"statfs":{"total":96636764160,"available":96597757952,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:13.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:13.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.975+0000 7f793cdf6640 1 Processor -- start 2024-03-06T07:31:13.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.975+0000 7f793cdf6640 1 -- start start 2024-03-06T07:31:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.976+0000 7f793cdf6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f793814f790 0x7f79381529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.976+0000 7f793cdf6640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7938155a40 0x7f7938155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.976+0000 7f793cdf6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380c28d0 con 0x7f79380c1aa0 2024-03-06T07:31:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.976+0000 7f793cdf6640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79380c2680 con 0x7f7938155a40 2024-03-06T07:31:13.980 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.976+0000 7f793cdf6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79380c2bb0 con 0x7f793814f790 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.976+0000 7f7936d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7938155a40 0x7f7938155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.977+0000 7f7936d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7938155a40 0x7f7938155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44638/0 (socket says 172.21.15.94:44638) 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.977+0000 7f7936d9d640 1 -- 172.21.15.94:0/2321337321 learned_addr learned my addr 172.21.15.94:0/2321337321 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.977+0000 7f793659c640 1 --2- 172.21.15.94:0/2321337321 >> v2:172.21.15.145:3300/0 conn(0x7f793814f790 0x7f79381529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.977+0000 7f7936d9d640 1 -- 172.21.15.94:0/2321337321 >> v2:172.21.15.145:3300/0 conn(0x7f793814f790 msgr2=0x7f79381529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.977+0000 7f7936d9d640 1 --2- 172.21.15.94:0/2321337321 >> v2:172.21.15.145:3300/0 conn(0x7f793814f790 0x7f79381529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.977+0000 7f7936d9d640 1 -- 172.21.15.94:0/2321337321 >> v1:172.21.15.145:6789/0 conn(0x7f79380c1aa0 legacy=0x7f79380c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.981 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.977+0000 7f7936d9d640 1 -- 172.21.15.94:0/2321337321 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380c2e90 con 0x7f7938155a40 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f7936d9d640 1 --2- 172.21.15.94:0/2321337321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f7938155e50 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f79380c3170 tx=0x7f792802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bca39fac420f42f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793559a640 1 -- 172.21.15.94:0/2321337321 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792803b070 con 0x7f7938155a40 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793559a640 1 -- 172.21.15.94:0/2321337321 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7928036440 con 0x7f7938155a40 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793559a640 1 -- 172.21.15.94:0/2321337321 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928036810 con 0x7f7938155a40 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 -- 172.21.15.94:0/2321337321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 msgr2=0x7f7938155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 --2- 172.21.15.94:0/2321337321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f7938155e50 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f79380c3170 tx=0x7f792802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 -- 172.21.15.94:0/2321337321 shutdown_connections 2024-03-06T07:31:13.982 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 --2- 172.21.15.94:0/2321337321 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f7938155e50 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 --2- 172.21.15.94:0/2321337321 >> v2:172.21.15.145:3300/0 conn(0x7f793814f790 0x7f79381529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 -- 172.21.15.94:0/2321337321 >> 172.21.15.94:0/2321337321 conn(0x7f79380c6710 msgr2=0x7f79380c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 -- 172.21.15.94:0/2321337321 shutdown_connections 2024-03-06T07:31:13.983 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.978+0000 7f793cdf6640 1 -- 172.21.15.94:0/2321337321 wait complete. 2024-03-06T07:31:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.979+0000 7f793cdf6640 1 Processor -- start 2024-03-06T07:31:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.979+0000 7f793cdf6640 1 -- start start 2024-03-06T07:31:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f793cdf6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79380c1aa0 0x7f7938154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f793cdf6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f793814f790 0x7f7938147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f793cdf6640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f79381480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f793cdf6640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79380c0d60 con 0x7f7938155a40 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f793cdf6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7938156390 con 0x7f79380c1aa0 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f793cdf6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79380c5e30 con 0x7f793814f790 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f7936d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f79381480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f7936d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f79381480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44648/0 (socket says 172.21.15.94:44648) 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f7936d9d640 1 -- 172.21.15.94:0/1428375443 learned_addr learned my addr 172.21.15.94:0/1428375443 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f793659c640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79380c1aa0 0x7f7938154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f7935d9b640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f793814f790 0x7f7938147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.987 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f7936d9d640 1 -- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f793814f790 msgr2=0x7f7938147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.980+0000 7f7936d9d640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f793814f790 0x7f7938147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f7936d9d640 1 -- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79380c1aa0 msgr2=0x7f7938154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f7936d9d640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79380c1aa0 0x7f7938154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f7936d9d640 1 -- 172.21.15.94:0/1428375443 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79381546d0 con 0x7f7938155a40 2024-03-06T07:31:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f7936d9d640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f79381480f0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f792800b2c0 tx=0x7f79280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f7926ffd640 1 -- 172.21.15.94:0/1428375443 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792803b040 con 0x7f7938155a40 2024-03-06T07:31:13.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f7926ffd640 1 -- 172.21.15.94:0/1428375443 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7928046070 con 0x7f7938155a40 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79381e6ad0 con 0x7f7938155a40 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f7926ffd640 1 -- 172.21.15.94:0/1428375443 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928034920 con 0x7f7938155a40 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.981+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79381e7070 con 0x7f7938155a40 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.982+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79380c44b0 con 0x7f7938155a40 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.982+0000 7f7926ffd640 1 -- 172.21.15.94:0/1428375443 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7928034ac0 con 0x7f7938155a40 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.983+0000 7f7926ffd640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f791806b4d0 0x7f791806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.983+0000 7f7926ffd640 1 -- 172.21.15.94:0/1428375443 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f79280aeb60 con 0x7f7938155a40 2024-03-06T07:31:13.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.983+0000 7f793659c640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f791806b4d0 0x7f791806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.983+0000 7f793659c640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f791806b4d0 0x7f791806d990 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7930006fd0 tx=0x7f7930008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.990 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:13.987+0000 7f7926ffd640 1 -- 172.21.15.94:0/1428375443 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79380c44b0 con 0x7f7938155a40 2024-03-06T07:31:14.159 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.157+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f793804f330 con 0x7f791806b4d0 2024-03-06T07:31:14.160 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.158+0000 7f7926ffd640 1 -- 172.21.15.94:0/1428375443 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13564 (secure 0 0 0) 0x7f793804f330 con 0x7f791806b4d0 2024-03-06T07:31:14.160 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:14.161 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:31:14.162 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f791806b4d0 msgr2=0x7f791806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f791806b4d0 0x7f791806d990 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7930006fd0 tx=0x7f7930008040 comp rx=0 tx=0).stop 2024-03-06T07:31:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 msgr2=0x7f79381480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f79381480f0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f792800b2c0 tx=0x7f79280046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 shutdown_connections 2024-03-06T07:31:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f791806b4d0 0x7f791806d990 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7938155a40 0x7f79381480f0 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.163 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f793814f790 0x7f7938147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 --2- 172.21.15.94:0/1428375443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79380c1aa0 0x7f7938154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.160+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 >> 172.21.15.94:0/1428375443 conn(0x7f79380c6710 msgr2=0x7f79380c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.161+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 shutdown_connections 2024-03-06T07:31:14.164 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.161+0000 7f793cdf6640 1 -- 172.21.15.94:0/1428375443 wait complete. 2024-03-06T07:31:14.173 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-03-06T07:31:12.798565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":283548,"kb_used_data":21600,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754691172,"statfs":{"total":773094113280,"available":772803760128,"internally_reserved":0,"allocated":22118400,"data_stored":11837112,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-67,"num_read_kb":-73,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001554"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":542,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.801349+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:31:10.801349+0000","last_peered":"2024-03-06T07:31:10.801349+0000","last_clean":"2024-03-06T07:31:10.801349+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:31:10.801349+0000","last_undegraded":"2024-03-06T07:31:10.801349+0000","last_fullsized":"2024-03-06T07:31:10.801349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33928,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337912,"statfs":{"total":96636764160,"available":96602021888,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38092,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333748,"statfs":{"total":96636764160,"available":96597757952,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:14.173 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:31:14.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:14.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.271+0000 7fb36b1b5640 1 Processor -- start 2024-03-06T07:31:14.273 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.271+0000 7fb36b1b5640 1 -- start start 2024-03-06T07:31:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.272+0000 7fb36b1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb36414f360 0x7fb364155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.275 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.272+0000 7fb36b1b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3640c73d0 con 0x7fb3640c88c0 2024-03-06T07:31:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.272+0000 7fb36b1b5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3640be2e0 con 0x7fb364158b60 2024-03-06T07:31:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.272+0000 7fb36b1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb364159700 con 0x7fb36414f360 2024-03-06T07:31:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.272+0000 7fb363fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb3640c88c0 0x7fb3640c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47102/0 (socket says 172.21.15.94:47102) 2024-03-06T07:31:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.272+0000 7fb363fff640 1 -- 172.21.15.94:0/3392769532 learned_addr learned my addr 172.21.15.94:0/3392769532 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.272+0000 7fb368f51640 1 --2- 172.21.15.94:0/3392769532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb36414f360 0x7fb364155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.276 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.273+0000 7fb3637fe640 1 -- 172.21.15.94:0/3392769532 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172610889 0 0) 0x7fb3640c73d0 con 0x7fb3640c88c0 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.273+0000 7fb3637fe640 1 -- 172.21.15.94:0/3392769532 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb34c003520 con 0x7fb3640c88c0 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.273+0000 7fb3637fe640 1 -- 172.21.15.94:0/3392769532 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962708631 0 0) 0x7fb3640be2e0 con 0x7fb364158b60 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.273+0000 7fb368f51640 1 -- 172.21.15.94:0/3392769532 >> v1:172.21.15.145:6789/0 conn(0x7fb364158b60 legacy=0x7fb364158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.273+0000 7fb368f51640 1 -- 172.21.15.94:0/3392769532 >> v1:172.21.15.94:6789/0 conn(0x7fb3640c88c0 legacy=0x7fb3640c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.273+0000 7fb368f51640 1 -- 172.21.15.94:0/3392769532 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3640c7670 con 0x7fb36414f360 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb368f51640 1 --2- 172.21.15.94:0/3392769532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb36414f360 0x7fb364155c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb3540097d0 tx=0x7fb35402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ce17551186001e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb3637fe640 1 -- 172.21.15.94:0/3392769532 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354035020 con 0x7fb36414f360 2024-03-06T07:31:14.277 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb3637fe640 1 -- 172.21.15.94:0/3392769532 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb354031800 con 0x7fb36414f360 2024-03-06T07:31:14.278 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb3637fe640 1 -- 172.21.15.94:0/3392769532 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354031bb0 con 0x7fb36414f360 2024-03-06T07:31:14.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb36b1b5640 1 -- 172.21.15.94:0/3392769532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb36414f360 msgr2=0x7fb364155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/3392769532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb36414f360 0x7fb364155c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb3540097d0 tx=0x7fb35402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.281 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb36b1b5640 1 -- 172.21.15.94:0/3392769532 shutdown_connections 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/3392769532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb36414f360 0x7fb364155c10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb36b1b5640 1 -- 172.21.15.94:0/3392769532 >> 172.21.15.94:0/3392769532 conn(0x7fb3640ac570 msgr2=0x7fb3640ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb36b1b5640 1 -- 172.21.15.94:0/3392769532 shutdown_connections 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.274+0000 7fb36b1b5640 1 -- 172.21.15.94:0/3392769532 wait complete. 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.275+0000 7fb36b1b5640 1 Processor -- start 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.275+0000 7fb36b1b5640 1 -- start start 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb36b1b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3640c88c0 0x7fb3641eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb36b1b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb364158b60 0x7fb36414d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb36b1b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3640ae710 0x7fb3641eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb36b1b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640c7c30 con 0x7fb364158b60 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb36b1b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640c5a50 con 0x7fb3640c88c0 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb36b1b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3640c5bd0 con 0x7fb3640ae710 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb363fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb364158b60 0x7fb36414d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb363fff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb364158b60 0x7fb36414d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44650/0 (socket says 172.21.15.94:44650) 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb363fff640 1 -- 172.21.15.94:0/2203902911 learned_addr learned my addr 172.21.15.94:0/2203902911 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:14.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb368f51640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3640c88c0 0x7fb3641eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.276+0000 7fb369752640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3640ae710 0x7fb3641eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb363fff640 1 -- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3640ae710 msgr2=0x7fb3641eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb363fff640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3640ae710 0x7fb3641eb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb363fff640 1 -- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3640c88c0 msgr2=0x7fb3641eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb363fff640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3640c88c0 0x7fb3641eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb363fff640 1 -- 172.21.15.94:0/2203902911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3640ad450 con 0x7fb364158b60 2024-03-06T07:31:14.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb363fff640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb364158b60 0x7fb36414d290 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c400 tx=0x7fb35800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb3617fa640 1 -- 172.21.15.94:0/2203902911 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358017070 con 0x7fb364158b60 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb3617fa640 1 -- 172.21.15.94:0/2203902911 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb358005d00 con 0x7fb364158b60 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3640ad730 con 0x7fb364158b60 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb3617fa640 1 -- 172.21.15.94:0/2203902911 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358007310 con 0x7fb364158b60 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.277+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3641eaf70 con 0x7fb364158b60 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.278+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3641ead80 con 0x7fb364158b60 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.278+0000 7fb3617fa640 1 -- 172.21.15.94:0/2203902911 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb35801a070 con 0x7fb364158b60 2024-03-06T07:31:14.285 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.279+0000 7fb3617fa640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb35006b4d0 0x7fb35006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.279+0000 7fb3617fa640 1 -- 172.21.15.94:0/2203902911 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fb3580894f0 con 0x7fb364158b60 2024-03-06T07:31:14.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.279+0000 7fb368f51640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb35006b4d0 0x7fb35006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.279+0000 7fb368f51640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb35006b4d0 0x7fb35006d990 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fb354036000 tx=0x7fb354008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.286 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.283+0000 7fb3617fa640 1 -- 172.21.15.94:0/2203902911 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb358057e20 con 0x7fb364158b60 2024-03-06T07:31:14.453 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.452+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3641eb170 con 0x7fb35006b4d0 2024-03-06T07:31:14.454 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.453+0000 7fb3617fa640 1 -- 172.21.15.94:0/2203902911 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13564 (secure 0 0 0) 0x7fb35404d510 con 0x7fb35006b4d0 2024-03-06T07:31:14.455 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:14.455 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:31:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb35006b4d0 msgr2=0x7fb35006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb35006b4d0 0x7fb35006d990 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fb354036000 tx=0x7fb354008030 comp rx=0 tx=0).stop 2024-03-06T07:31:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb364158b60 msgr2=0x7fb36414d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb364158b60 0x7fb36414d290 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c400 tx=0x7fb35800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 shutdown_connections 2024-03-06T07:31:14.457 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fb35006b4d0 0x7fb35006d990 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3640ae710 0x7fb3641eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb364158b60 0x7fb36414d290 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 --2- 172.21.15.94:0/2203902911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3640c88c0 0x7fb3641eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 >> 172.21.15.94:0/2203902911 conn(0x7fb3640ac570 msgr2=0x7fb3640c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 shutdown_connections 2024-03-06T07:31:14.458 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.455+0000 7fb36b1b5640 1 -- 172.21.15.94:0/2203902911 wait complete. 2024-03-06T07:31:14.467 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-03-06T07:31:12.798565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":283548,"kb_used_data":21600,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754691172,"statfs":{"total":773094113280,"available":772803760128,"internally_reserved":0,"allocated":22118400,"data_stored":11837112,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-67,"num_read_kb":-73,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001554"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":542,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.801349+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:31:10.801349+0000","last_peered":"2024-03-06T07:31:10.801349+0000","last_clean":"2024-03-06T07:31:10.801349+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:31:10.801349+0000","last_undegraded":"2024-03-06T07:31:10.801349+0000","last_fullsized":"2024-03-06T07:31:10.801349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33928,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337912,"statfs":{"total":96636764160,"available":96602021888,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575008,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575009,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38092,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333748,"statfs":{"total":96636764160,"available":96597757952,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:14.468 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.566+0000 7f0ab58b5640 1 Processor -- start 2024-03-06T07:31:14.568 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.566+0000 7f0ab58b5640 1 -- start start 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0ab58b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 0x7f0ab0158f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0ab58b5640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ab0159460 0x7f0ab00c5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0ab58b5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab00c7760 con 0x7f0ab0152920 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0ab58b5640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab01587b0 con 0x7f0ab0159460 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0ab58b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00c7a40 con 0x7f0ab00c8a00 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aaf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ab0159460 0x7f0ab00c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aaeffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0ab0152920 0x7f0ab0155c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47110/0 (socket says 172.21.15.94:47110) 2024-03-06T07:31:14.570 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aaeffd640 1 -- 172.21.15.94:0/973417480 learned_addr learned my addr 172.21.15.94:0/973417480 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aaf7fe640 1 -- 172.21.15.94:0/973417480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 msgr2=0x7f0ab0158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aaf7fe640 1 --2- 172.21.15.94:0/973417480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 0x7f0ab0158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aaf7fe640 1 -- 172.21.15.94:0/973417480 >> v1:172.21.15.94:6789/0 conn(0x7f0ab0152920 legacy=0x7f0ab0155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aadffb640 1 -- 172.21.15.94:0/973417480 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068947815 0 0) 0x7f0ab00c7760 con 0x7f0ab0152920 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.567+0000 7f0aaf7fe640 1 -- 172.21.15.94:0/973417480 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00c7d20 con 0x7f0ab0159460 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.568+0000 7f0aaf7fe640 1 --2- 172.21.15.94:0/973417480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab0159460 0x7f0ab00c5330 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009d90 tx=0x7f0aa802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06469d95f33a77f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.568+0000 7f0aadffb640 1 -- 172.21.15.94:0/973417480 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8039070 con 0x7f0ab0159460 2024-03-06T07:31:14.571 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.568+0000 7f0aadffb640 1 -- 172.21.15.94:0/973417480 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa802cbc0 con 0x7f0ab0159460 2024-03-06T07:31:14.572 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.568+0000 7f0aadffb640 1 -- 172.21.15.94:0/973417480 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8034440 con 0x7f0ab0159460 2024-03-06T07:31:14.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.568+0000 7f0ab58b5640 1 -- 172.21.15.94:0/973417480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab0159460 msgr2=0x7f0ab00c5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.569+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/973417480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab0159460 0x7f0ab00c5330 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009d90 tx=0x7f0aa802be10 comp rx=0 tx=0).stop 2024-03-06T07:31:14.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.569+0000 7f0ab58b5640 1 -- 172.21.15.94:0/973417480 shutdown_connections 2024-03-06T07:31:14.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.569+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/973417480 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab0159460 0x7f0ab00c5330 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.569+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/973417480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 0x7f0ab0158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.569+0000 7f0ab58b5640 1 -- 172.21.15.94:0/973417480 >> 172.21.15.94:0/973417480 conn(0x7f0ab00ac560 msgr2=0x7f0ab00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.575 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.569+0000 7f0ab58b5640 1 -- 172.21.15.94:0/973417480 shutdown_connections 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.569+0000 7f0ab58b5640 1 -- 172.21.15.94:0/973417480 wait complete. 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 Processor -- start 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 -- start start 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 0x7f0ab01dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ab0152920 0x7f0ab014d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab014dad0 0x7f0ab01e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00c7390 con 0x7f0ab014dad0 2024-03-06T07:31:14.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab006d030 con 0x7f0ab0152920 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.570+0000 7f0ab58b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab00c7510 con 0x7f0ab00c8a00 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab014dad0 0x7f0ab01e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab014dad0 0x7f0ab01e1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44670/0 (socket says 172.21.15.94:44670) 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 -- 172.21.15.94:0/2302199924 learned_addr learned my addr 172.21.15.94:0/2302199924 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aae7fc640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ab0152920 0x7f0ab014d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaeffd640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 0x7f0ab01dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 -- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 msgr2=0x7f0ab01dea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 0x7f0ab01dea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.577 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 -- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ab0152920 msgr2=0x7f0ab014d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ab0152920 0x7f0ab014d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 -- 172.21.15.94:0/2302199924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab014e200 con 0x7f0ab014dad0 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aae7fc640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ab0152920 0x7f0ab014d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0aaf7fe640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab014dad0 0x7f0ab01e1a60 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8037040 tx=0x7f0aa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0a97fff640 1 -- 172.21.15.94:0/2302199924 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8044030 con 0x7f0ab014dad0 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0a97fff640 1 -- 172.21.15.94:0/2302199924 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa802c9c0 con 0x7f0ab014dad0 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab014e620 con 0x7f0ab014dad0 2024-03-06T07:31:14.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0a97fff640 1 -- 172.21.15.94:0/2302199924 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8039040 con 0x7f0ab014dad0 2024-03-06T07:31:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.571+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab01eede0 con 0x7f0ab014dad0 2024-03-06T07:31:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.572+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab004f330 con 0x7f0ab014dad0 2024-03-06T07:31:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.573+0000 7f0a97fff640 1 -- 172.21.15.94:0/2302199924 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0aa8047070 con 0x7f0ab014dad0 2024-03-06T07:31:14.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.573+0000 7f0a97fff640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.573+0000 7f0a97fff640 1 -- 172.21.15.94:0/2302199924 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f0aa80ac990 con 0x7f0ab014dad0 2024-03-06T07:31:14.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.573+0000 7f0aaeffd640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.574+0000 7f0aaeffd640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007fd0 tx=0x7f0aa0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.580 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.578+0000 7f0a97fff640 1 -- 172.21.15.94:0/2302199924 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0aa804e070 con 0x7f0ab014dad0 2024-03-06T07:31:14.750 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.749+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ab01defd0 con 0x7f0ab014dad0 2024-03-06T07:31:14.751 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.749+0000 7f0a97fff640 1 -- 172.21.15.94:0/2302199924 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) v1 ==== 74+0+13693 (secure 0 0 0) 0x7f0aa807b910 con 0x7f0ab014dad0 2024-03-06T07:31:14.751 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:14.751 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":216,"fsid":"00dede11-ad16-411b-be77-3a57b1483d8f","created":"2024-03-06T07:17:52.402777+0000","modified":"2024-03-06T07:31:10.448589+0000","last_up_change":"2024-03-06T07:18:02.299336+0000","last_in_change":"2024-03-06T07:17:54.623758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:18:02.678312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51d664c3-2eb9-4a1e-b855-ad028426728c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6811","nonce":4051752422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6813","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6815","nonce":4051752422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6821","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6823","nonce":4051752422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6817","nonce":4051752422},{"type":"v1","addr":"172.21.15.94:6819","nonce":4051752422}]},"public_addr":"172.21.15.94:6811/4051752422","cluster_addr":"172.21.15.94:6815/4051752422","heartbeat_back_addr":"172.21.15.94:6823/4051752422","heartbeat_front_addr":"172.21.15.94:6819/4051752422","state":["exists","up"]},{"osd":1,"uuid":"a739cd3d-4564-4d3d-8624-a4d4ddbc0e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6810","nonce":2571975958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6814","nonce":2571975958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6822","nonce":2571975958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":2571975958},{"type":"v1","addr":"172.21.15.94:6818","nonce":2571975958}]},"public_addr":"172.21.15.94:6810/2571975958","cluster_addr":"172.21.15.94:6814/2571975958","heartbeat_back_addr":"172.21.15.94:6822/2571975958","heartbeat_front_addr":"172.21.15.94:6818/2571975958","state":["exists","up"]},{"osd":2,"uuid":"93c1acd5-d24b-42b9-8654-dfec597d57b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6801","nonce":516902551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6803","nonce":516902551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6807","nonce":516902551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":516902551},{"type":"v1","addr":"172.21.15.94:6805","nonce":516902551}]},"public_addr":"172.21.15.94:6801/516902551","cluster_addr":"172.21.15.94:6803/516902551","heartbeat_back_addr":"172.21.15.94:6807/516902551","heartbeat_front_addr":"172.21.15.94:6805/516902551","state":["exists","up"]},{"osd":3,"uuid":"e268a931-5385-498e-8bc4-28a44c6bf4cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6825","nonce":3669301969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6827","nonce":3669301969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6831","nonce":3669301969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3669301969},{"type":"v1","addr":"172.21.15.94:6829","nonce":3669301969}]},"public_addr":"172.21.15.94:6825/3669301969","cluster_addr":"172.21.15.94:6827/3669301969","heartbeat_back_addr":"172.21.15.94:6831/3669301969","heartbeat_front_addr":"172.21.15.94:6829/3669301969","state":["exists","up"]},{"osd":4,"uuid":"f3040779-55db-4733-a4d0-cc47e70d3ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6801","nonce":4137837373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6803","nonce":4137837373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6807","nonce":4137837373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":4137837373},{"type":"v1","addr":"172.21.15.145:6805","nonce":4137837373}]},"public_addr":"172.21.15.145:6801/4137837373","cluster_addr":"172.21.15.145:6803/4137837373","heartbeat_back_addr":"172.21.15.145:6807/4137837373","heartbeat_front_addr":"172.21.15.145:6805/4137837373","state":["exists","up"]},{"osd":5,"uuid":"b95cc0c1-7cbe-4f68-b823-7e3eace28e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6809","nonce":2538796010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6811","nonce":2538796010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6815","nonce":2538796010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2538796010},{"type":"v1","addr":"172.21.15.145:6813","nonce":2538796010}]},"public_addr":"172.21.15.145:6809/2538796010","cluster_addr":"172.21.15.145:6811/2538796010","heartbeat_back_addr":"172.21.15.145:6815/2538796010","heartbeat_front_addr":"172.21.15.145:6813/2538796010","state":["exists","up"]},{"osd":6,"uuid":"c57d10da-d807-4079-91b6-0a0db0c33c60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6817","nonce":1405026475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6819","nonce":1405026475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6823","nonce":1405026475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":1405026475},{"type":"v1","addr":"172.21.15.145:6821","nonce":1405026475}]},"public_addr":"172.21.15.145:6817/1405026475","cluster_addr":"172.21.15.145:6819/1405026475","heartbeat_back_addr":"172.21.15.145:6823/1405026475","heartbeat_front_addr":"172.21.15.145:6821/1405026475","state":["exists","up"]},{"osd":7,"uuid":"6b61b374-fd8d-4b4a-a15b-cc008337e57c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6825","nonce":1323110445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6827","nonce":1323110445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6831","nonce":1323110445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1323110445},{"type":"v1","addr":"172.21.15.145:6829","nonce":1323110445}]},"public_addr":"172.21.15.145:6825/1323110445","cluster_addr":"172.21.15.145:6827/1323110445","heartbeat_back_addr":"172.21.15.145:6831/1323110445","heartbeat_front_addr":"172.21.15.145:6829/1323110445","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.730191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.762550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.790435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.829495+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.836002+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.817819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.802994+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:17:59.799159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.94:6833/341111452":"2024-03-07T07:31:07.421906+0000","172.21.15.94:6832/341111452":"2024-03-07T07:31:07.421906+0000","172.21.15.145:6834/2585066812":"2024-03-07T07:31:07.421918+0000","172.21.15.94:6835/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6836/2585066812":"2024-03-07T07:31:07.421918+0000","172.21.15.145:6837/679350833":"2024-03-07T07:28:02.249189+0000","172.21.15.94:6834/2402539280":"2024-03-07T07:28:37.254469+0000","172.21.15.145:6835/679350833":"2024-03-07T07:28:02.249189+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:14.753 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a9c06b4d0 msgr2=0x7f0a9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007fd0 tx=0x7f0aa0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:14.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab014dad0 msgr2=0x7f0ab01e1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab014dad0 0x7f0ab01e1a60 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8037040 tx=0x7f0aa8004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:14.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 shutdown_connections 2024-03-06T07:31:14.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0a9c06b4d0 0x7f0a9c06d990 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ab014dad0 0x7f0ab01e1a60 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.754 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ab0152920 0x7f0ab014d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 --2- 172.21.15.94:0/2302199924 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ab00c8a00 0x7f0ab01dea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.751+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 >> 172.21.15.94:0/2302199924 conn(0x7f0ab00ac560 msgr2=0x7f0ab00c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.752+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 shutdown_connections 2024-03-06T07:31:14.755 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:14.752+0000 7f0ab58b5640 1 -- 172.21.15.94:0/2302199924 wait complete. 2024-03-06T07:31:15.765 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:31:15.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:15.882 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.881+0000 7fbfb1165640 1 Processor -- start 2024-03-06T07:31:15.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.881+0000 7fbfb1165640 1 -- start start 2024-03-06T07:31:15.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.881+0000 7fbfb1165640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 0x7fbfac156540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.881+0000 7fbfb1165640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbfac0a7340 0x7fbfac06f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.881+0000 7fbfb1165640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac156a80 con 0x7fbfac0a9ea0 2024-03-06T07:31:15.886 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.881+0000 7fbfb1165640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfac0a7e70 con 0x7fbfac0a7340 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.881+0000 7fbfb1165640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac150b90 con 0x7fbfac1507b0 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfaa59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fbfac0a9ea0 0x7fbfac1538f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47126/0 (socket says 172.21.15.94:47126) 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfaa59c640 1 -- 172.21.15.94:0/66143358 learned_addr learned my addr 172.21.15.94:0/66143358 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfab59e640 1 --2- 172.21.15.94:0/66143358 >> v2:172.21.15.145:3300/0 conn(0x7fbfac0a7340 0x7fbfac06f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054309053 0 0) 0x7fbfac156a80 con 0x7fbfac0a9ea0 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf94003520 con 0x7fbfac0a9ea0 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 0x7fbfac156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1780782201 0 0) 0x7fbf94003520 con 0x7fbfac0a9ea0 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 msgr2=0x7fbfac156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 0x7fbfac156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 >> v2:172.21.15.145:3300/0 conn(0x7fbfac0a7340 msgr2=0x7fbfac06f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 --2- 172.21.15.94:0/66143358 >> v2:172.21.15.145:3300/0 conn(0x7fbfac0a7340 0x7fbfac06f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0a78b0 con 0x7fbfac0a9ea0 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7fbfa0001670 con 0x7fbfac0a9ea0 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 >> v1:172.21.15.94:6789/0 conn(0x7fbfac0a9ea0 legacy=0x7fbfac1538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf940039b0 0x7fbf94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 0x7fbf94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf94004d40 0x7fbf940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0a7e70 con 0x7fbf940039b0 2024-03-06T07:31:15.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac150b90 con 0x7fbf94004d40 2024-03-06T07:31:15.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9400b9c0 con 0x7fbfac1507b0 2024-03-06T07:31:15.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.882+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf940039b0 0x7fbf94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.889 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfab59e640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf94004d40 0x7fbf940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfaa59c640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 0x7fbf94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfaad9d640 1 -- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 msgr2=0x7fbf94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbfac1507b0 0x7fbf94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfaad9d640 1 -- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf94004d40 msgr2=0x7fbf940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf94004d40 0x7fbf940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfaad9d640 1 -- 172.21.15.94:0/66143358 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94003520 con 0x7fbf940039b0 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.883+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf940039b0 0x7fbf94003dd0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009ac0 tx=0x7fbf94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b394730ce18024 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c039070 con 0x7fbf940039b0 2024-03-06T07:31:15.890 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c02c6b0 con 0x7fbf940039b0 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfa9d9b640 1 -- 172.21.15.94:0/66143358 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c009690 con 0x7fbf940039b0 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfb1165640 1 -- 172.21.15.94:0/66143358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf940039b0 msgr2=0x7fbf94003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfb1165640 1 --2- 172.21.15.94:0/66143358 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbf940039b0 0x7fbf94003dd0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c009ac0 tx=0x7fbf94005830 comp rx=0 tx=0).stop 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfab59e640 1 -- 172.21.15.94:0/66143358 reap_dead start 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfb1165640 1 -- 172.21.15.94:0/66143358 shutdown_connections 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfb1165640 1 -- 172.21.15.94:0/66143358 >> 172.21.15.94:0/66143358 conn(0x7fbfac06c0d0 msgr2=0x7fbfac06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfb1165640 1 -- 172.21.15.94:0/66143358 shutdown_connections 2024-03-06T07:31:15.891 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.884+0000 7fbfb1165640 1 -- 172.21.15.94:0/66143358 wait complete. 2024-03-06T07:31:15.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 Processor -- start 2024-03-06T07:31:15.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 -- start start 2024-03-06T07:31:15.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf940039b0 0x7fbfac1ec240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbfac1ec780 0x7fbfac146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfac146c60 0x7fbfac144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.895 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac1533d0 con 0x7fbfac146c60 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac1530f0 con 0x7fbfac1ec780 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.886+0000 7fbfb1165640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0c1c00 con 0x7fbf940039b0 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfac146c60 0x7fbfac144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfac146c60 0x7fbfac144480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44696/0 (socket says 172.21.15.94:44696) 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 -- 172.21.15.94:0/2750212205 learned_addr learned my addr 172.21.15.94:0/2750212205 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfaa59c640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbfac1ec780 0x7fbfac146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf940039b0 0x7fbfac1ec240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.896 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 -- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf940039b0 msgr2=0x7fbfac1ec240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf940039b0 0x7fbfac1ec240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 -- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbfac1ec780 msgr2=0x7fbfac146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbfac1ec780 0x7fbfac146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 -- 172.21.15.94:0/2750212205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac151970 con 0x7fbfac146c60 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfab59e640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfac146c60 0x7fbfac144480 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fbfac157660 tx=0x7fbfa400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa400fce0 con 0x7fbfac146c60 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa401d070 con 0x7fbfac146c60 2024-03-06T07:31:15.897 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac151c50 con 0x7fbfac146c60 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4015670 con 0x7fbfac146c60 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.887+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac150200 con 0x7fbfac146c60 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.888+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbfac04f330 con 0x7fbfac146c60 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.889+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfa4005d10 con 0x7fbfac146c60 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.890+0000 7fbf937fe640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf9806b4d0 0x7fbf9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.890+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fbfa408b320 con 0x7fbfac146c60 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.890+0000 7fbf937fe640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7fbf98071b50 0x7fbf98073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.898 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.890+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 --> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfa401eec0 con 0x7fbf98071b50 2024-03-06T07:31:15.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.890+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfa408b7b0 con 0x7fbfac146c60 2024-03-06T07:31:15.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.890+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf9806b4d0 0x7fbf9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.890+0000 7fbfaa59c640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7fbf98071b50 0x7fbf98073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.891+0000 7fbfaa59c640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7fbf98071b50 0x7fbf98073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.891+0000 7fbfaad9d640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf9806b4d0 0x7fbf9806d990 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c036000 tx=0x7fbf9c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.899 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.891+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== osd.0 v2:172.21.15.94:6809/4051752422 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbfa401eec0 con 0x7fbf98071b50 2024-03-06T07:31:15.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.904+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 --> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbfac06f570 con 0x7fbf98071b50 2024-03-06T07:31:15.906 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbf937fe640 1 -- 172.21.15.94:0/2750212205 <== osd.0 v2:172.21.15.94:6809/4051752422 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbfac06f570 con 0x7fbf98071b50 2024-03-06T07:31:15.907 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:15.907 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:15.907 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:15.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7fbf98071b50 msgr2=0x7fbf98073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:6809/4051752422,v1:172.21.15.94:6811/4051752422] conn(0x7fbf98071b50 0x7fbf98073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf9806b4d0 msgr2=0x7fbf9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.908 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fbf9806b4d0 0x7fbf9806d990 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c036000 tx=0x7fbf9c030000 comp rx=0 tx=0).stop 2024-03-06T07:31:15.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfac146c60 msgr2=0x7fbfac144480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 --2- 172.21.15.94:0/2750212205 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fbfac146c60 0x7fbfac144480 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fbfac157660 tx=0x7fbfa400f1b0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfab59e640 1 -- 172.21.15.94:0/2750212205 reap_dead start 2024-03-06T07:31:15.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 shutdown_connections 2024-03-06T07:31:15.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 >> 172.21.15.94:0/2750212205 conn(0x7fbfac06c0d0 msgr2=0x7fbfac153a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 shutdown_connections 2024-03-06T07:31:15.909 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:15.905+0000 7fbfb1165640 1 -- 172.21.15.94:0/2750212205 wait complete. 2024-03-06T07:31:15.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:31:16.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.018+0000 7fa4979eb640 1 Processor -- start 2024-03-06T07:31:16.020 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.018+0000 7fa4979eb640 1 -- start start 2024-03-06T07:31:16.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.019+0000 7fa4979eb640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa4900c6dd0 0x7fa490160ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.019+0000 7fa4979eb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa490155130 con 0x7fa4900c69f0 2024-03-06T07:31:16.021 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.019+0000 7fa4979eb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4900c7200 con 0x7fa4900c8950 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.019+0000 7fa4979eb640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa490155440 con 0x7fa4900c6dd0 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.019+0000 7fa495f88640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa4900c6dd0 0x7fa490160ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.019+0000 7fa495787640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa4900c69f0 0x7fa4900acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47138/0 (socket says 172.21.15.94:47138) 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.019+0000 7fa495787640 1 -- 172.21.15.94:0/2732077911 learned_addr learned my addr 172.21.15.94:0/2732077911 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa495f88640 1 -- 172.21.15.94:0/2732077911 >> v1:172.21.15.145:6789/0 conn(0x7fa4900c8950 legacy=0x7fa4901558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa495f88640 1 -- 172.21.15.94:0/2732077911 >> v1:172.21.15.94:6789/0 conn(0x7fa4900c69f0 legacy=0x7fa4900acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa495f88640 1 -- 172.21.15.94:0/2732077911 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4900ad480 con 0x7fa4900c6dd0 2024-03-06T07:31:16.022 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa483fff640 1 -- 172.21.15.94:0/2732077911 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155883002 0 0) 0x7fa484001110 con 0x7fa4900c69f0 2024-03-06T07:31:16.023 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa495f88640 1 --2- 172.21.15.94:0/2732077911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c6dd0 0x7fa490160ac0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7fa4901555c0 tx=0x7fa48802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abe7f0d9616fc43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa483fff640 1 -- 172.21.15.94:0/2732077911 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488039070 con 0x7fa4900c6dd0 2024-03-06T07:31:16.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa483fff640 1 -- 172.21.15.94:0/2732077911 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48802cc50 con 0x7fa4900c6dd0 2024-03-06T07:31:16.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.020+0000 7fa483fff640 1 -- 172.21.15.94:0/2732077911 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488034440 con 0x7fa4900c6dd0 2024-03-06T07:31:16.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.021+0000 7fa4979eb640 1 -- 172.21.15.94:0/2732077911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c6dd0 msgr2=0x7fa490160ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.021+0000 7fa4979eb640 1 --2- 172.21.15.94:0/2732077911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c6dd0 0x7fa490160ac0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7fa4901555c0 tx=0x7fa48802bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.021+0000 7fa4979eb640 1 -- 172.21.15.94:0/2732077911 shutdown_connections 2024-03-06T07:31:16.024 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.021+0000 7fa4979eb640 1 --2- 172.21.15.94:0/2732077911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c6dd0 0x7fa490160ac0 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.021+0000 7fa4979eb640 1 -- 172.21.15.94:0/2732077911 >> 172.21.15.94:0/2732077911 conn(0x7fa4900cae70 msgr2=0x7fa4900cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.021+0000 7fa4979eb640 1 -- 172.21.15.94:0/2732077911 shutdown_connections 2024-03-06T07:31:16.025 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.021+0000 7fa4979eb640 1 -- 172.21.15.94:0/2732077911 wait complete. 2024-03-06T07:31:16.027 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.022+0000 7fa4979eb640 1 Processor -- start 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.022+0000 7fa4979eb640 1 -- start start 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa4979eb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4900c69f0 0x7fa49014cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa4979eb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4900c6dd0 0x7fa4900b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa4979eb640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c8950 0x7fa49006c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa4979eb640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4900c6310 con 0x7fa4900c8950 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa4979eb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4900c6030 con 0x7fa4900c6dd0 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa4979eb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa490155fe0 con 0x7fa4900c69f0 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c8950 0x7fa49006c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c8950 0x7fa49006c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44726/0 (socket says 172.21.15.94:44726) 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 -- 172.21.15.94:0/1914330855 learned_addr learned my addr 172.21.15.94:0/1914330855 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa494f86640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4900c6dd0 0x7fa4900b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495787640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4900c69f0 0x7fa49014cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 -- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4900c69f0 msgr2=0x7fa49014cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4900c69f0 0x7fa49014cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 -- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4900c6dd0 msgr2=0x7fa4900b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4900c6dd0 0x7fa4900b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.023+0000 7fa495f88640 1 -- 172.21.15.94:0/1914330855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4900c37b0 con 0x7fa4900c8950 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.024+0000 7fa495f88640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c8950 0x7fa49006c520 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7fa488037040 tx=0x7fa488004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.024+0000 7fa481ffb640 1 -- 172.21.15.94:0/1914330855 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488039050 con 0x7fa4900c8950 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.024+0000 7fa481ffb640 1 -- 172.21.15.94:0/1914330855 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa488045070 con 0x7fa4900c8950 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.024+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa490157630 con 0x7fa4900c8950 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.024+0000 7fa481ffb640 1 -- 172.21.15.94:0/1914330855 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48803e610 con 0x7fa4900c8950 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.024+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4901eb2b0 con 0x7fa4900c8950 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.025+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4900bf910 con 0x7fa4900c8950 2024-03-06T07:31:16.030 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.025+0000 7fa481ffb640 1 -- 172.21.15.94:0/1914330855 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa48803e7b0 con 0x7fa4900c8950 2024-03-06T07:31:16.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.026+0000 7fa481ffb640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47406b4d0 0x7fa47406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.026+0000 7fa481ffb640 1 -- 172.21.15.94:0/1914330855 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa4880acdb0 con 0x7fa4900c8950 2024-03-06T07:31:16.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.026+0000 7fa495787640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47406b4d0 0x7fa47406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.031 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.027+0000 7fa495787640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47406b4d0 0x7fa47406d990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa484000f60 tx=0x7fa484007d00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.032 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.030+0000 7fa481ffb640 1 -- 172.21.15.94:0/1914330855 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4900bf910 con 0x7fa4900c8950 2024-03-06T07:31:16.211 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.209+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa49004f330 con 0x7fa47406b4d0 2024-03-06T07:31:16.212 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.210+0000 7fa481ffb640 1 -- 172.21.15.94:0/1914330855 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa49004f330 con 0x7fa47406b4d0 2024-03-06T07:31:16.212 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:31:16.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47406b4d0 msgr2=0x7fa47406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47406b4d0 0x7fa47406d990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa484000f60 tx=0x7fa484007d00 comp rx=0 tx=0).stop 2024-03-06T07:31:16.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c8950 msgr2=0x7fa49006c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.214 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c8950 0x7fa49006c520 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7fa488037040 tx=0x7fa488004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 shutdown_connections 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa47406b4d0 0x7fa47406d990 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa4900c8950 0x7fa49006c520 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4900c6dd0 0x7fa4900b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 --2- 172.21.15.94:0/1914330855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4900c69f0 0x7fa49014cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 >> 172.21.15.94:0/1914330855 conn(0x7fa4900cae70 msgr2=0x7fa490154690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 shutdown_connections 2024-03-06T07:31:16.215 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.212+0000 7fa4979eb640 1 -- 172.21.15.94:0/1914330855 wait complete. 2024-03-06T07:31:16.224 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:31:16.225 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:16.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.330+0000 7fd7a955c640 1 Processor -- start 2024-03-06T07:31:16.332 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.330+0000 7fd7a955c640 1 -- start start 2024-03-06T07:31:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a955c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd7a4151e10 0x7fd7a4153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a955c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a406e850 con 0x7fd7a40c22b0 2024-03-06T07:31:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a955c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a406d2b0 con 0x7fd7a40a9830 2024-03-06T07:31:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a955c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a406d590 con 0x7fd7a4151e10 2024-03-06T07:31:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a37fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd7a4151e10 0x7fd7a4153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.333 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd7a40c22b0 0x7fd7a414ef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47144/0 (socket says 172.21.15.94:47144) 2024-03-06T07:31:16.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a2ffd640 1 -- 172.21.15.94:0/4048452497 learned_addr learned my addr 172.21.15.94:0/4048452497 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:16.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a37fe640 1 -- 172.21.15.94:0/4048452497 >> v1:172.21.15.145:6789/0 conn(0x7fd7a40a9830 legacy=0x7fd7a4153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a37fe640 1 -- 172.21.15.94:0/4048452497 >> v1:172.21.15.94:6789/0 conn(0x7fd7a40c22b0 legacy=0x7fd7a414ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a37fe640 1 -- 172.21.15.94:0/4048452497 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a406d870 con 0x7fd7a4151e10 2024-03-06T07:31:16.334 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.331+0000 7fd7a1ffb640 1 -- 172.21.15.94:0/4048452497 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456949651 0 0) 0x7fd794001110 con 0x7fd7a40c22b0 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.332+0000 7fd7a37fe640 1 --2- 172.21.15.94:0/4048452497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a4151e10 0x7fd7a4153aa0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009dc0 tx=0x7fd79c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881c7ac66b11680a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.332+0000 7fd7a1ffb640 1 -- 172.21.15.94:0/4048452497 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c039070 con 0x7fd7a4151e10 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.332+0000 7fd7a1ffb640 1 -- 172.21.15.94:0/4048452497 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c02cc40 con 0x7fd7a4151e10 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.332+0000 7fd7a1ffb640 1 -- 172.21.15.94:0/4048452497 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c034440 con 0x7fd7a4151e10 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.333+0000 7fd7a955c640 1 -- 172.21.15.94:0/4048452497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a4151e10 msgr2=0x7fd7a4153aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.333+0000 7fd7a955c640 1 --2- 172.21.15.94:0/4048452497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a4151e10 0x7fd7a4153aa0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009dc0 tx=0x7fd79c02be90 comp rx=0 tx=0).stop 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.333+0000 7fd7a955c640 1 -- 172.21.15.94:0/4048452497 shutdown_connections 2024-03-06T07:31:16.336 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.333+0000 7fd7a955c640 1 --2- 172.21.15.94:0/4048452497 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a4151e10 0x7fd7a4153aa0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.333+0000 7fd7a955c640 1 -- 172.21.15.94:0/4048452497 >> 172.21.15.94:0/4048452497 conn(0x7fd7a40c65b0 msgr2=0x7fd7a40c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.333+0000 7fd7a955c640 1 -- 172.21.15.94:0/4048452497 shutdown_connections 2024-03-06T07:31:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.333+0000 7fd7a955c640 1 -- 172.21.15.94:0/4048452497 wait complete. 2024-03-06T07:31:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.334+0000 7fd7a955c640 1 Processor -- start 2024-03-06T07:31:16.337 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.334+0000 7fd7a955c640 1 -- start start 2024-03-06T07:31:16.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7a40a9830 0x7fd7a414da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a40c22b0 0x7fd7a41495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7a4151e10 0x7fd7a4149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.340 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a414f660 con 0x7fd7a40c22b0 2024-03-06T07:31:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a4153fe0 con 0x7fd7a40a9830 2024-03-06T07:31:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a414f7e0 con 0x7fd7a4151e10 2024-03-06T07:31:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a40c22b0 0x7fd7a41495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a40c22b0 0x7fd7a41495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44740/0 (socket says 172.21.15.94:44740) 2024-03-06T07:31:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 -- 172.21.15.94:0/161733977 learned_addr learned my addr 172.21.15.94:0/161733977 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a2ffd640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7a40a9830 0x7fd7a414da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.341 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a37fe640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7a4151e10 0x7fd7a4149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 -- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7a4151e10 msgr2=0x7fd7a4149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7a4151e10 0x7fd7a4149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 -- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7a40a9830 msgr2=0x7fd7a414da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd7a40a9830 0x7fd7a414da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 -- 172.21.15.94:0/161733977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40c5330 con 0x7fd7a40c22b0 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a27fc640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a40c22b0 0x7fd7a41495d0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fd79800c460 tx=0x7fd79800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798016020 con 0x7fd7a40c22b0 2024-03-06T07:31:16.342 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd798012870 con 0x7fd7a40c22b0 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a41e9360 con 0x7fd7a40c22b0 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798012c60 con 0x7fd7a40c22b0 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.335+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a40c58d0 con 0x7fd7a40c22b0 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.336+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7a404f330 con 0x7fd7a40c22b0 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.337+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd798019020 con 0x7fd7a40c22b0 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.337+0000 7fd78b7fe640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd79006b4d0 0x7fd79006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.337+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fd79808a140 con 0x7fd7a40c22b0 2024-03-06T07:31:16.343 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.337+0000 7fd78b7fe640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7fd790071b50 0x7fd790073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.337+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 --> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd798021ec0 con 0x7fd790071b50 2024-03-06T07:31:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.337+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd798091050 con 0x7fd7a40c22b0 2024-03-06T07:31:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.337+0000 7fd7a2ffd640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd79006b4d0 0x7fd79006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.338+0000 7fd7a37fe640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7fd790071b50 0x7fd790073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.338+0000 7fd7a37fe640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7fd790071b50 0x7fd790073f90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.338+0000 7fd7a2ffd640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd79006b4d0 0x7fd79006d990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd794000f60 tx=0x7fd794007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.344 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.338+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== osd.1 v2:172.21.15.94:6808/2571975958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd798021ec0 con 0x7fd790071b50 2024-03-06T07:31:16.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 --> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd7a4155100 con 0x7fd790071b50 2024-03-06T07:31:16.353 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd78b7fe640 1 -- 172.21.15.94:0/161733977 <== osd.1 v2:172.21.15.94:6808/2571975958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd7a4155100 con 0x7fd790071b50 2024-03-06T07:31:16.354 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:16.354 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:16.354 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:16.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7fd790071b50 msgr2=0x7fd790073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:6808/2571975958,v1:172.21.15.94:6810/2571975958] conn(0x7fd790071b50 0x7fd790073f90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd79006b4d0 msgr2=0x7fd79006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fd79006b4d0 0x7fd79006d990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd794000f60 tx=0x7fd794007970 comp rx=0 tx=0).stop 2024-03-06T07:31:16.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a40c22b0 msgr2=0x7fd7a41495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 --2- 172.21.15.94:0/161733977 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd7a40c22b0 0x7fd7a41495d0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fd79800c460 tx=0x7fd79800c920 comp rx=0 tx=0).stop 2024-03-06T07:31:16.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a37fe640 1 -- 172.21.15.94:0/161733977 reap_dead start 2024-03-06T07:31:16.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 shutdown_connections 2024-03-06T07:31:16.356 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 >> 172.21.15.94:0/161733977 conn(0x7fd7a40c65b0 msgr2=0x7fd7a40c6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 shutdown_connections 2024-03-06T07:31:16.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.352+0000 7fd7a955c640 1 -- 172.21.15.94:0/161733977 wait complete. 2024-03-06T07:31:16.365 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:31:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.466+0000 7fa0c53c9640 1 Processor -- start 2024-03-06T07:31:16.468 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0c53c9640 1 -- start start 2024-03-06T07:31:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0c53c9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa0c006d680 0x7fa0c0152060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0c53c9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa0c0072f40 0x7fa0c00c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.470 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0c53c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c00c1140 con 0x7fa0c00c2720 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0c53c9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c00c1d50 con 0x7fa0c006d680 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0c53c9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c01525a0 con 0x7fa0c0072f40 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0beffd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa0c006d680 0x7fa0c0152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0bf7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa0c00c2720 0x7fa0c0154bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47146/0 (socket says 172.21.15.94:47146) 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0bf7fe640 1 -- 172.21.15.94:0/255061787 learned_addr learned my addr 172.21.15.94:0/255061787 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.467+0000 7fa0b7fff640 1 --2- 172.21.15.94:0/255061787 >> v2:172.21.15.145:3300/0 conn(0x7fa0c0072f40 0x7fa0c00c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.468+0000 7fa0beffd640 1 -- 172.21.15.94:0/255061787 >> v2:172.21.15.145:3300/0 conn(0x7fa0c0072f40 msgr2=0x7fa0c00c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.468+0000 7fa0beffd640 1 --2- 172.21.15.94:0/255061787 >> v2:172.21.15.145:3300/0 conn(0x7fa0c0072f40 0x7fa0c00c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.471 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.468+0000 7fa0beffd640 1 -- 172.21.15.94:0/255061787 >> v1:172.21.15.94:6789/0 conn(0x7fa0c00c2720 legacy=0x7fa0c0154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.468+0000 7fa0be7fc640 1 -- 172.21.15.94:0/255061787 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910497970 0 0) 0x7fa0c00c1140 con 0x7fa0c00c2720 2024-03-06T07:31:16.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.468+0000 7fa0beffd640 1 -- 172.21.15.94:0/255061787 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c00c1420 con 0x7fa0c006d680 2024-03-06T07:31:16.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.468+0000 7fa0b7fff640 1 --2- 172.21.15.94:0/255061787 >> v2:172.21.15.145:3300/0 conn(0x7fa0c0072f40 0x7fa0c00c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:16.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.468+0000 7fa0beffd640 1 --2- 172.21.15.94:0/255061787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c006d680 0x7fa0c0152060 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00097d0 tx=0x7fa0b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903ac071c1785671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0be7fc640 1 -- 172.21.15.94:0/255061787 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b00315a0 con 0x7fa0c006d680 2024-03-06T07:31:16.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0be7fc640 1 -- 172.21.15.94:0/255061787 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b0037070 con 0x7fa0c006d680 2024-03-06T07:31:16.472 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0be7fc640 1 -- 172.21.15.94:0/255061787 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0031960 con 0x7fa0c006d680 2024-03-06T07:31:16.475 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 -- 172.21.15.94:0/255061787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c006d680 msgr2=0x7fa0c0152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/255061787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c006d680 0x7fa0c0152060 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00097d0 tx=0x7fa0b002b840 comp rx=0 tx=0).stop 2024-03-06T07:31:16.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 -- 172.21.15.94:0/255061787 shutdown_connections 2024-03-06T07:31:16.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/255061787 >> v2:172.21.15.145:3300/0 conn(0x7fa0c0072f40 0x7fa0c00c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/255061787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c006d680 0x7fa0c0152060 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 -- 172.21.15.94:0/255061787 >> 172.21.15.94:0/255061787 conn(0x7fa0c006b1a0 msgr2=0x7fa0c006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 -- 172.21.15.94:0/255061787 shutdown_connections 2024-03-06T07:31:16.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.469+0000 7fa0c53c9640 1 -- 172.21.15.94:0/255061787 wait complete. 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.471+0000 7fa0c53c9640 1 Processor -- start 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.471+0000 7fa0c53c9640 1 -- start start 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0c53c9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c006d680 0x7fa0c00a9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0c53c9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c0072f40 0x7fa0c01d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0c53c9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c00c2720 0x7fa0c01d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0c53c9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c006da60 con 0x7fa0c00c2720 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0c53c9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00c19e0 con 0x7fa0c006d680 2024-03-06T07:31:16.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0c53c9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c006dbe0 con 0x7fa0c0072f40 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c00c2720 0x7fa0c01d7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c00c2720 0x7fa0c01d7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44766/0 (socket says 172.21.15.94:44766) 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 -- 172.21.15.94:0/3431954425 learned_addr learned my addr 172.21.15.94:0/3431954425 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0b7fff640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c0072f40 0x7fa0c01d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0beffd640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c006d680 0x7fa0c00a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 -- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c0072f40 msgr2=0x7fa0c01d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c0072f40 0x7fa0c01d77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.478 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 -- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c006d680 msgr2=0x7fa0c00a9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c006d680 0x7fa0c00a9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.472+0000 7fa0bf7fe640 1 -- 172.21.15.94:0/3431954425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0150020 con 0x7fa0c00c2720 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.473+0000 7fa0bf7fe640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c00c2720 0x7fa0c01d7d20 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800c730 tx=0x7fa0b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.473+0000 7fa0b77fe640 1 -- 172.21.15.94:0/3431954425 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8018070 con 0x7fa0c00c2720 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.473+0000 7fa0b77fe640 1 -- 172.21.15.94:0/3431954425 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b800ce40 con 0x7fa0c00c2720 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.473+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c0150300 con 0x7fa0c00c2720 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.473+0000 7fa0b77fe640 1 -- 172.21.15.94:0/3431954425 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800a860 con 0x7fa0c00c2720 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.473+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c0151830 con 0x7fa0c00c2720 2024-03-06T07:31:16.479 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.474+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c004f330 con 0x7fa0c00c2720 2024-03-06T07:31:16.480 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.475+0000 7fa0b77fe640 1 -- 172.21.15.94:0/3431954425 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0b800aa00 con 0x7fa0c00c2720 2024-03-06T07:31:16.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.475+0000 7fa0b77fe640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a006b4d0 0x7fa0a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.475+0000 7fa0b77fe640 1 -- 172.21.15.94:0/3431954425 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa0b8089600 con 0x7fa0c00c2720 2024-03-06T07:31:16.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.476+0000 7fa0beffd640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a006b4d0 0x7fa0a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.476+0000 7fa0beffd640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a006b4d0 0x7fa0a006d990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0036000 tx=0x7fa0b003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.481 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.480+0000 7fa0b77fe640 1 -- 172.21.15.94:0/3431954425 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0b8057f30 con 0x7fa0c00c2720 2024-03-06T07:31:16.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.659+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0c01eeb10 con 0x7fa0a006b4d0 2024-03-06T07:31:16.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.659+0000 7fa0b77fe640 1 -- 172.21.15.94:0/3431954425 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa0c01eeb10 con 0x7fa0a006b4d0 2024-03-06T07:31:16.661 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:31:16.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a006b4d0 msgr2=0x7fa0a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a006b4d0 0x7fa0a006d990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0036000 tx=0x7fa0b003d040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c00c2720 msgr2=0x7fa0c01d7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c00c2720 0x7fa0c01d7d20 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800c730 tx=0x7fa0b800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 shutdown_connections 2024-03-06T07:31:16.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa0a006b4d0 0x7fa0a006d990 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa0c00c2720 0x7fa0c01d7d20 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c0072f40 0x7fa0c01d77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 --2- 172.21.15.94:0/3431954425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c006d680 0x7fa0c00a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 >> 172.21.15.94:0/3431954425 conn(0x7fa0c006b1a0 msgr2=0x7fa0c00c6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 shutdown_connections 2024-03-06T07:31:16.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.661+0000 7fa0c53c9640 1 -- 172.21.15.94:0/3431954425 wait complete. 2024-03-06T07:31:16.674 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:31:16.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.777+0000 7f3a36655640 1 Processor -- start 2024-03-06T07:31:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.777+0000 7f3a36655640 1 -- start start 2024-03-06T07:31:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.778+0000 7f3a36655640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a301551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.781 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.778+0000 7f3a36655640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3a301566c0 0x7f3a3006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.778+0000 7f3a36655640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a300c8f60 con 0x7f3a3014f180 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.778+0000 7f3a36655640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a300c9240 con 0x7f3a301566c0 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.778+0000 7f3a36655640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a300c9520 con 0x7f3a300c8980 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.778+0000 7f3a2f7fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a301551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.778+0000 7f3a34bf2640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3a301566c0 0x7f3a3006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.779+0000 7f3a2ffff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f3a3014f180 0x7f3a300ca000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.94:46840/0 (socket says 172.21.15.94:46840) 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.779+0000 7f3a2ffff640 1 -- 172.21.15.94:0/3118589345 learned_addr learned my addr 172.21.15.94:0/3118589345 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:16.782 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.779+0000 7f3a2f7fe640 1 -- 172.21.15.94:0/3118589345 >> v2:172.21.15.145:3300/0 conn(0x7f3a301566c0 msgr2=0x7f3a3006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.779+0000 7f3a2f7fe640 1 --2- 172.21.15.94:0/3118589345 >> v2:172.21.15.145:3300/0 conn(0x7f3a301566c0 0x7f3a3006c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.779+0000 7f3a2f7fe640 1 -- 172.21.15.94:0/3118589345 >> v1:172.21.15.145:6789/0 conn(0x7f3a3014f180 legacy=0x7f3a300ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.783 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.779+0000 7f3a2f7fe640 1 -- 172.21.15.94:0/3118589345 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a300c9800 con 0x7f3a300c8980 2024-03-06T07:31:16.787 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a2f7fe640 1 --2- 172.21.15.94:0/3118589345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a301551d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a200097a0 tx=0x7f3a2002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae1650f53de38594 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a2effd640 1 -- 172.21.15.94:0/3118589345 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20035020 con 0x7f3a300c8980 2024-03-06T07:31:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a2effd640 1 -- 172.21.15.94:0/3118589345 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a20031830 con 0x7f3a300c8980 2024-03-06T07:31:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a2effd640 1 -- 172.21.15.94:0/3118589345 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20031c00 con 0x7f3a300c8980 2024-03-06T07:31:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 -- 172.21.15.94:0/3118589345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 msgr2=0x7f3a301551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 --2- 172.21.15.94:0/3118589345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a301551d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3a200097a0 tx=0x7f3a2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 -- 172.21.15.94:0/3118589345 shutdown_connections 2024-03-06T07:31:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 --2- 172.21.15.94:0/3118589345 >> v2:172.21.15.145:3300/0 conn(0x7f3a301566c0 0x7f3a3006c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 --2- 172.21.15.94:0/3118589345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a301551d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 -- 172.21.15.94:0/3118589345 >> 172.21.15.94:0/3118589345 conn(0x7f3a30072d20 msgr2=0x7f3a300ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 -- 172.21.15.94:0/3118589345 shutdown_connections 2024-03-06T07:31:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.780+0000 7f3a36655640 1 -- 172.21.15.94:0/3118589345 wait complete. 2024-03-06T07:31:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.781+0000 7f3a36655640 1 Processor -- start 2024-03-06T07:31:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a36655640 1 -- start start 2024-03-06T07:31:16.789 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a36655640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a30154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a36655640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a3014f180 0x7f3a301de5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a36655640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a301566c0 0x7f3a3014b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a36655640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3014f560 con 0x7f3a3014f180 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a36655640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a30156b50 con 0x7f3a301566c0 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a36655640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3014f6e0 con 0x7f3a300c8980 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a2f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a3014f180 0x7f3a301de5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a2f7fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a3014f180 0x7f3a301de5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44782/0 (socket says 172.21.15.94:44782) 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a2f7fe640 1 -- 172.21.15.94:0/620817554 learned_addr learned my addr 172.21.15.94:0/620817554 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:16.790 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a2ffff640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a30154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.782+0000 7f3a34bf2640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a301566c0 0x7f3a3014b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2f7fe640 1 -- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 msgr2=0x7f3a30154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2f7fe640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a300c8980 0x7f3a30154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2f7fe640 1 -- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a301566c0 msgr2=0x7f3a3014b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2f7fe640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a301566c0 0x7f3a3014b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2f7fe640 1 -- 172.21.15.94:0/620817554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a301df110 con 0x7f3a3014f180 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2f7fe640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a3014f180 0x7f3a301de5c0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f3a20034040 tx=0x7f3a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.791 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20040070 con 0x7f3a3014f180 2024-03-06T07:31:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2003a910 con 0x7f3a3014f180 2024-03-06T07:31:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a301df3f0 con 0x7f3a3014f180 2024-03-06T07:31:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a20035040 con 0x7f3a3014f180 2024-03-06T07:31:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.783+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a301dee10 con 0x7f3a3014f180 2024-03-06T07:31:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.784+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a3004f330 con 0x7f3a3014f180 2024-03-06T07:31:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.785+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a20038020 con 0x7f3a3014f180 2024-03-06T07:31:16.792 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.785+0000 7f3a2cff9640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a1c06b4d0 0x7f3a1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.785+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f3a200ac2c0 con 0x7f3a3014f180 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.785+0000 7f3a2ffff640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a1c06b4d0 0x7f3a1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.785+0000 7f3a2cff9640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f3a1c071b50 0x7f3a1c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.785+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 --> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a200067f0 con 0x7f3a1c071b50 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.785+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a200ac750 con 0x7f3a3014f180 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.786+0000 7f3a34bf2640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f3a1c071b50 0x7f3a1c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.786+0000 7f3a2ffff640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a1c06b4d0 0x7f3a1c06d990 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3a301e0fc0 tx=0x7f3a24007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.786+0000 7f3a34bf2640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f3a1c071b50 0x7f3a1c073f90 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.794 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.786+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== osd.2 v2:172.21.15.94:6800/516902551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a200067f0 con 0x7f3a1c071b50 2024-03-06T07:31:16.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.799+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 --> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3a300afe70 con 0x7f3a1c071b50 2024-03-06T07:31:16.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a2cff9640 1 -- 172.21.15.94:0/620817554 <== osd.2 v2:172.21.15.94:6800/516902551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3a300afe70 con 0x7f3a1c071b50 2024-03-06T07:31:16.802 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:16.802 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:16.802 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:16.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f3a1c071b50 msgr2=0x7f3a1c073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:6800/516902551,v1:172.21.15.94:6801/516902551] conn(0x7f3a1c071b50 0x7f3a1c073f90 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a1c06b4d0 msgr2=0x7f3a1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f3a1c06b4d0 0x7f3a1c06d990 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3a301e0fc0 tx=0x7f3a24007e80 comp rx=0 tx=0).stop 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a3014f180 msgr2=0x7f3a301de5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 --2- 172.21.15.94:0/620817554 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a3014f180 0x7f3a301de5c0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f3a20034040 tx=0x7f3a200046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a34bf2640 1 -- 172.21.15.94:0/620817554 reap_dead start 2024-03-06T07:31:16.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 shutdown_connections 2024-03-06T07:31:16.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 >> 172.21.15.94:0/620817554 conn(0x7f3a30072d20 msgr2=0x7f3a30157510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 shutdown_connections 2024-03-06T07:31:16.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.800+0000 7f3a36655640 1 -- 172.21.15.94:0/620817554 wait complete. 2024-03-06T07:31:16.812 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:31:16.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.912+0000 7f83c1fa0640 1 Processor -- start 2024-03-06T07:31:16.914 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.912+0000 7f83c1fa0640 1 -- start start 2024-03-06T07:31:16.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83c1fa0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc150040 0x7f83bc1504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.916 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83c1fa0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc06f350 con 0x7f83bc0c7820 2024-03-06T07:31:16.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83c1fa0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc0c41e0 con 0x7f83bc152ed0 2024-03-06T07:31:16.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83c1fa0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0c4fa0 con 0x7f83bc150040 2024-03-06T07:31:16.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83c0f9e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f83bc0c7820 0x7f83bc06e890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47152/0 (socket says 172.21.15.94:47152) 2024-03-06T07:31:16.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83c0f9e640 1 -- 172.21.15.94:0/4175088626 learned_addr learned my addr 172.21.15.94:0/4175088626 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:16.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83bbfff640 1 --2- 172.21.15.94:0/4175088626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc150040 0x7f83bc1504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83bb7fe640 1 -- 172.21.15.94:0/4175088626 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338191953 0 0) 0x7f83bc06f350 con 0x7f83bc0c7820 2024-03-06T07:31:16.917 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.913+0000 7f83bb7fe640 1 -- 172.21.15.94:0/4175088626 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ac003520 con 0x7f83bc0c7820 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.914+0000 7f83bb7fe640 1 -- 172.21.15.94:0/4175088626 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3690508641 0 0) 0x7f83ac003520 con 0x7f83bc0c7820 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.914+0000 7f83bbfff640 1 -- 172.21.15.94:0/4175088626 >> v1:172.21.15.145:6789/0 conn(0x7f83bc152ed0 legacy=0x7f83bc153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.914+0000 7f83bbfff640 1 -- 172.21.15.94:0/4175088626 >> v1:172.21.15.94:6789/0 conn(0x7f83bc0c7820 legacy=0x7f83bc06e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.914+0000 7f83bbfff640 1 -- 172.21.15.94:0/4175088626 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0c44c0 con 0x7f83bc150040 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83bbfff640 1 --2- 172.21.15.94:0/4175088626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc150040 0x7f83bc1504a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f83a80097d0 tx=0x7f83a802b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04cc0b61ccf883a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83bb7fe640 1 -- 172.21.15.94:0/4175088626 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8037070 con 0x7f83bc150040 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83bb7fe640 1 -- 172.21.15.94:0/4175088626 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8030510 con 0x7f83bc150040 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83bb7fe640 1 -- 172.21.15.94:0/4175088626 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a802be60 con 0x7f83bc150040 2024-03-06T07:31:16.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83c1fa0640 1 -- 172.21.15.94:0/4175088626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc150040 msgr2=0x7f83bc1504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/4175088626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc150040 0x7f83bc1504a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f83a80097d0 tx=0x7f83a802b660 comp rx=0 tx=0).stop 2024-03-06T07:31:16.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83c1fa0640 1 -- 172.21.15.94:0/4175088626 shutdown_connections 2024-03-06T07:31:16.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/4175088626 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc150040 0x7f83bc1504a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.922 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83c1fa0640 1 -- 172.21.15.94:0/4175088626 >> 172.21.15.94:0/4175088626 conn(0x7f83bc06daf0 msgr2=0x7f83bc06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83c1fa0640 1 -- 172.21.15.94:0/4175088626 shutdown_connections 2024-03-06T07:31:16.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.915+0000 7f83c1fa0640 1 -- 172.21.15.94:0/4175088626 wait complete. 2024-03-06T07:31:16.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.916+0000 7f83c1fa0640 1 Processor -- start 2024-03-06T07:31:16.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.916+0000 7f83c1fa0640 1 -- start start 2024-03-06T07:31:16.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83c1fa0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc0c7820 0x7f83bc149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83c1fa0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83bc150040 0x7f83bc0c14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.923 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83c1fa0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83bc152ed0 0x7f83bc14f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83c1fa0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc153a30 con 0x7f83bc150040 2024-03-06T07:31:16.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83c1fa0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0c4a80 con 0x7f83bc152ed0 2024-03-06T07:31:16.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83c1fa0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc06efa0 con 0x7f83bc0c7820 2024-03-06T07:31:16.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83bc150040 0x7f83bc0c14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83bbfff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83bc150040 0x7f83bc0c14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44798/0 (socket says 172.21.15.94:44798) 2024-03-06T07:31:16.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83bbfff640 1 -- 172.21.15.94:0/2522719081 learned_addr learned my addr 172.21.15.94:0/2522719081 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:16.924 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.917+0000 7f83c0f9e640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc0c7820 0x7f83bc149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83c179f640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83bc152ed0 0x7f83bc14f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83bbfff640 1 -- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc0c7820 msgr2=0x7f83bc149e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83bbfff640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc0c7820 0x7f83bc149e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83bbfff640 1 -- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83bc152ed0 msgr2=0x7f83bc14f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83bbfff640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83bc152ed0 0x7f83bc14f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83bbfff640 1 -- 172.21.15.94:0/2522719081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0c0a30 con 0x7f83bc150040 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83bbfff640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83bc150040 0x7f83bc0c14b0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f83a802b6a0 tx=0x7f83a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83b97fa640 1 -- 172.21.15.94:0/2522719081 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8042070 con 0x7f83bc150040 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83b97fa640 1 -- 172.21.15.94:0/2522719081 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a803e870 con 0x7f83bc150040 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc1df9f0 con 0x7f83bc150040 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83b97fa640 1 -- 172.21.15.94:0/2522719081 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8037040 con 0x7f83bc150040 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.918+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc0c0cb0 con 0x7f83bc150040 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.919+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc154160 con 0x7f83bc150040 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.919+0000 7f83b97fa640 1 -- 172.21.15.94:0/2522719081 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83a8045070 con 0x7f83bc150040 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.920+0000 7f83b97fa640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.920+0000 7f83b97fa640 1 -- 172.21.15.94:0/2522719081 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f83a80abf90 con 0x7f83bc150040 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.920+0000 7f83c0f9e640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.921+0000 7f83c0f9e640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f83b00017e0 tx=0x7f83b0001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.928 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:16.924+0000 7f83b97fa640 1 -- 172.21.15.94:0/2522719081 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83bc154160 con 0x7f83bc150040 2024-03-06T07:31:16.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:16.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:17.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.101+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f83bc0693c0 con 0x7f83a006b4d0 2024-03-06T07:31:17.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.102+0000 7f83b97fa640 1 -- 172.21.15.94:0/2522719081 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83bc0693c0 con 0x7f83a006b4d0 2024-03-06T07:31:17.104 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:31:17.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83a006b4d0 msgr2=0x7f83a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f83b00017e0 tx=0x7f83b0001630 comp rx=0 tx=0).stop 2024-03-06T07:31:17.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83bc150040 msgr2=0x7f83bc0c14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83bc150040 0x7f83bc0c14b0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f83a802b6a0 tx=0x7f83a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 shutdown_connections 2024-03-06T07:31:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83bc152ed0 0x7f83bc14f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f83bc150040 0x7f83bc0c14b0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 --2- 172.21.15.94:0/2522719081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83bc0c7820 0x7f83bc149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 >> 172.21.15.94:0/2522719081 conn(0x7f83bc06daf0 msgr2=0x7f83bc074080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 shutdown_connections 2024-03-06T07:31:17.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.104+0000 7f83c1fa0640 1 -- 172.21.15.94:0/2522719081 wait complete. 2024-03-06T07:31:17.116 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:31:17.117 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.219+0000 7f8e89be9640 1 Processor -- start 2024-03-06T07:31:17.221 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.219+0000 7f8e89be9640 1 -- start start 2024-03-06T07:31:17.222 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e89be9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8e8414f180 0x7f8e840c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e89be9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8e841551d0 0x7f8e841566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e89be9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e840c8f60 con 0x7f8e840ca000 2024-03-06T07:31:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e89be9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e84156c00 con 0x7f8e841551d0 2024-03-06T07:31:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e89be9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e840c9240 con 0x7f8e8414f180 2024-03-06T07:31:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e7affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8e841551d0 0x7f8e841566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e7affd640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8e841551d0 0x7f8e841566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44808/0 (socket says 172.21.15.94:44808) 2024-03-06T07:31:17.223 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e7affd640 1 -- 172.21.15.94:0/2535406957 learned_addr learned my addr 172.21.15.94:0/2535406957 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.220+0000 7f8e837fe640 1 --2- 172.21.15.94:0/2535406957 >> v2:172.21.15.145:3300/0 conn(0x7f8e8414f180 0x7f8e840c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.221+0000 7f8e7affd640 1 -- 172.21.15.94:0/2535406957 >> v2:172.21.15.145:3300/0 conn(0x7f8e8414f180 msgr2=0x7f8e840c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.221+0000 7f8e7affd640 1 --2- 172.21.15.94:0/2535406957 >> v2:172.21.15.145:3300/0 conn(0x7f8e8414f180 0x7f8e840c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.221+0000 7f8e7affd640 1 -- 172.21.15.94:0/2535406957 >> v1:172.21.15.145:6789/0 conn(0x7f8e840ca000 legacy=0x7f8e8406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.224 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.221+0000 7f8e7affd640 1 -- 172.21.15.94:0/2535406957 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e840c9520 con 0x7f8e841551d0 2024-03-06T07:31:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.221+0000 7f8e7affd640 1 --2- 172.21.15.94:0/2535406957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e841551d0 0x7f8e841566c0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f8e84073630 tx=0x7f8e6c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8543da1ebe3cfc8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.227 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.221+0000 7f8e82ffd640 1 -- 172.21.15.94:0/2535406957 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c034070 con 0x7f8e841551d0 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e82ffd640 1 -- 172.21.15.94:0/2535406957 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c030860 con 0x7f8e841551d0 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e82ffd640 1 -- 172.21.15.94:0/2535406957 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c030c30 con 0x7f8e841551d0 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 -- 172.21.15.94:0/2535406957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e841551d0 msgr2=0x7f8e841566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 --2- 172.21.15.94:0/2535406957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e841551d0 0x7f8e841566c0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f8e84073630 tx=0x7f8e6c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 -- 172.21.15.94:0/2535406957 shutdown_connections 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 --2- 172.21.15.94:0/2535406957 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e841551d0 0x7f8e841566c0 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 --2- 172.21.15.94:0/2535406957 >> v2:172.21.15.145:3300/0 conn(0x7f8e8414f180 0x7f8e840c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.228 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 -- 172.21.15.94:0/2535406957 >> 172.21.15.94:0/2535406957 conn(0x7f8e84072d20 msgr2=0x7f8e840ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 -- 172.21.15.94:0/2535406957 shutdown_connections 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.222+0000 7f8e89be9640 1 -- 172.21.15.94:0/2535406957 wait complete. 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.223+0000 7f8e89be9640 1 Processor -- start 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.223+0000 7f8e89be9640 1 -- start start 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e89be9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e840ca000 0x7f8e840c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e89be9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e8414f180 0x7f8e8414d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e89be9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e8414e350 0x7f8e8414dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e89be9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8414f560 con 0x7f8e840ca000 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e89be9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e840c9ae0 con 0x7f8e8414e350 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e89be9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8414f6e0 con 0x7f8e8414f180 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e840ca000 0x7f8e840c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e840ca000 0x7f8e840c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44824/0 (socket says 172.21.15.94:44824) 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 -- 172.21.15.94:0/2947846064 learned_addr learned my addr 172.21.15.94:0/2947846064 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 -- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e8414f180 msgr2=0x7f8e8414d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e8414f180 0x7f8e8414d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e83fff640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e8414e350 0x7f8e8414dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 -- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e8414e350 msgr2=0x7f8e8414dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e8414e350 0x7f8e8414dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.224+0000 7f8e837fe640 1 -- 172.21.15.94:0/2947846064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84152900 con 0x7f8e840ca000 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.225+0000 7f8e837fe640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e840ca000 0x7f8e840c5e30 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400d5d0 tx=0x7f8e7400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.225+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e740125f0 con 0x7f8e840ca000 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.225+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e74019070 con 0x7f8e840ca000 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.225+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e84152be0 con 0x7f8e840ca000 2024-03-06T07:31:17.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.225+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e740129f0 con 0x7f8e840ca000 2024-03-06T07:31:17.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.225+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e84151920 con 0x7f8e840ca000 2024-03-06T07:31:17.232 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.226+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8e8404f330 con 0x7f8e840ca000 2024-03-06T07:31:17.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.226+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e74012b90 con 0x7f8e840ca000 2024-03-06T07:31:17.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e80ff9640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e6406b4d0 0x7f8e6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8e7408a570 con 0x7f8e840ca000 2024-03-06T07:31:17.233 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e80ff9640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7f8e64071b50 0x7f8e64073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 --> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e74001d30 con 0x7f8e64071b50 2024-03-06T07:31:17.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e74091050 con 0x7f8e840ca000 2024-03-06T07:31:17.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e7affd640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e6406b4d0 0x7f8e6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e83fff640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7f8e64071b50 0x7f8e64073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e83fff640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7f8e64071b50 0x7f8e64073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.227+0000 7f8e7affd640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e6406b4d0 0x7f8e6406d990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c036000 tx=0x7f8e6c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.228+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== osd.3 v2:172.21.15.94:6824/3669301969 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8e74001d30 con 0x7f8e64071b50 2024-03-06T07:31:17.242 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 --> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8e840afe70 con 0x7f8e64071b50 2024-03-06T07:31:17.243 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e80ff9640 1 -- 172.21.15.94:0/2947846064 <== osd.3 v2:172.21.15.94:6824/3669301969 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8e840afe70 con 0x7f8e64071b50 2024-03-06T07:31:17.243 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:17.243 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:17.243 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:17.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7f8e64071b50 msgr2=0x7f8e64073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.244 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:6824/3669301969,v1:172.21.15.94:6825/3669301969] conn(0x7f8e64071b50 0x7f8e64073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e6406b4d0 msgr2=0x7f8e6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8e6406b4d0 0x7f8e6406d990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c036000 tx=0x7f8e6c03b040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e840ca000 msgr2=0x7f8e840c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 --2- 172.21.15.94:0/2947846064 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e840ca000 0x7f8e840c5e30 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400d5d0 tx=0x7f8e7400da90 comp rx=0 tx=0).stop 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e83fff640 1 -- 172.21.15.94:0/2947846064 reap_dead start 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 shutdown_connections 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 >> 172.21.15.94:0/2947846064 conn(0x7f8e84072d20 msgr2=0x7f8e84154850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.245 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 shutdown_connections 2024-03-06T07:31:17.246 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.241+0000 7f8e89be9640 1 -- 172.21.15.94:0/2947846064 wait complete. 2024-03-06T07:31:17.253 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:31:17.354 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.352+0000 7f15c1555640 1 Processor -- start 2024-03-06T07:31:17.355 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15c1555640 1 -- start start 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15c1555640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc06ff10 0x7f15bc153c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15c1555640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15bc0c8f20 0x7f15bc154160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15c1555640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15bc0a8390 con 0x7f15bc0c7170 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15c1555640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15bc0c7620 con 0x7f15bc0c8f20 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15c1555640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0730f0 con 0x7f15bc06ff10 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15bb7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15bc0c8f20 0x7f15bc154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15bb7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15bc0c8f20 0x7f15bc154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44840/0 (socket says 172.21.15.94:44840) 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.353+0000 7f15bb7fe640 1 -- 172.21.15.94:0/537209420 learned_addr learned my addr 172.21.15.94:0/537209420 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.354+0000 7f15baffd640 1 --2- 172.21.15.94:0/537209420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc06ff10 0x7f15bc153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.354+0000 7f15bb7fe640 1 -- 172.21.15.94:0/537209420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc06ff10 msgr2=0x7f15bc153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.354+0000 7f15bb7fe640 1 --2- 172.21.15.94:0/537209420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc06ff10 0x7f15bc153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.354+0000 7f15bb7fe640 1 -- 172.21.15.94:0/537209420 >> v1:172.21.15.145:6789/0 conn(0x7f15bc0c7170 legacy=0x7f15bc0c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.354+0000 7f15bb7fe640 1 -- 172.21.15.94:0/537209420 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc0733d0 con 0x7f15bc0c8f20 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.354+0000 7f15bb7fe640 1 --2- 172.21.15.94:0/537209420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc0c8f20 0x7f15bc154160 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009e50 tx=0x7f15b402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51948d4f1941a01d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15b9ffb640 1 -- 172.21.15.94:0/537209420 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b402c9e0 con 0x7f15bc0c8f20 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15b9ffb640 1 -- 172.21.15.94:0/537209420 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b402cb80 con 0x7f15bc0c8f20 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15b9ffb640 1 -- 172.21.15.94:0/537209420 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b40343f0 con 0x7f15bc0c8f20 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 -- 172.21.15.94:0/537209420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc0c8f20 msgr2=0x7f15bc154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 --2- 172.21.15.94:0/537209420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc0c8f20 0x7f15bc154160 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009e50 tx=0x7f15b402be40 comp rx=0 tx=0).stop 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 -- 172.21.15.94:0/537209420 shutdown_connections 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 --2- 172.21.15.94:0/537209420 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc0c8f20 0x7f15bc154160 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 --2- 172.21.15.94:0/537209420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc06ff10 0x7f15bc153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 -- 172.21.15.94:0/537209420 >> 172.21.15.94:0/537209420 conn(0x7f15bc06b970 msgr2=0x7f15bc0aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 -- 172.21.15.94:0/537209420 shutdown_connections 2024-03-06T07:31:17.360 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.355+0000 7f15c1555640 1 -- 172.21.15.94:0/537209420 wait complete. 2024-03-06T07:31:17.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.356+0000 7f15c1555640 1 Processor -- start 2024-03-06T07:31:17.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.356+0000 7f15c1555640 1 -- start start 2024-03-06T07:31:17.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15c1555640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc06ff10 0x7f15bc1546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15c1555640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc0c7170 0x7f15bc0a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15c1555640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15bc0c8f20 0x7f15bc1e9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.365 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15c1555640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc070470 con 0x7f15bc06ff10 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15c1555640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0702f0 con 0x7f15bc0c8f20 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15c1555640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc0c6860 con 0x7f15bc0c7170 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15baffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc06ff10 0x7f15bc1546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15baffd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc06ff10 0x7f15bc1546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44850/0 (socket says 172.21.15.94:44850) 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15baffd640 1 -- 172.21.15.94:0/2806401250 learned_addr learned my addr 172.21.15.94:0/2806401250 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15bb7fe640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15bc0c8f20 0x7f15bc1e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15ba7fc640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc0c7170 0x7f15bc0a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.366 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15baffd640 1 -- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc0c7170 msgr2=0x7f15bc0a9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15baffd640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc0c7170 0x7f15bc0a9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15baffd640 1 -- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15bc0c8f20 msgr2=0x7f15bc1e9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.357+0000 7f15baffd640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15bc0c8f20 0x7f15bc1e9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15baffd640 1 -- 172.21.15.94:0/2806401250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc1e8da0 con 0x7f15bc06ff10 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15ba7fc640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc0c7170 0x7f15bc0a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15baffd640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc06ff10 0x7f15bc1546e0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f15b000d600 tx=0x7f15b000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15a37fe640 1 -- 172.21.15.94:0/2806401250 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b00125f0 con 0x7f15bc06ff10 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15a37fe640 1 -- 172.21.15.94:0/2806401250 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b0019070 con 0x7f15bc06ff10 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15a37fe640 1 -- 172.21.15.94:0/2806401250 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b00129f0 con 0x7f15bc06ff10 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15bc0bda40 con 0x7f15bc06ff10 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.358+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15bc1e9060 con 0x7f15bc06ff10 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.359+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15bc1e9260 con 0x7f15bc06ff10 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.359+0000 7f15a37fe640 1 -- 172.21.15.94:0/2806401250 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15b0012b90 con 0x7f15bc06ff10 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.360+0000 7f15a37fe640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15a806b4d0 0x7f15a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.360+0000 7f15a37fe640 1 -- 172.21.15.94:0/2806401250 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f15b008a450 con 0x7f15bc06ff10 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.360+0000 7f15ba7fc640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15a806b4d0 0x7f15a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.361+0000 7f15ba7fc640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15a806b4d0 0x7f15a806d990 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f15bc150aa0 tx=0x7f15ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.364+0000 7f15a37fe640 1 -- 172.21.15.94:0/2806401250 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15bc1e9260 con 0x7f15bc06ff10 2024-03-06T07:31:17.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.542+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f15bc04f330 con 0x7f15a806b4d0 2024-03-06T07:31:17.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.542+0000 7f15a37fe640 1 -- 172.21.15.94:0/2806401250 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f15bc04f330 con 0x7f15a806b4d0 2024-03-06T07:31:17.544 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:31:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15a806b4d0 msgr2=0x7f15a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15a806b4d0 0x7f15a806d990 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f15bc150aa0 tx=0x7f15ac009040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc06ff10 msgr2=0x7f15bc1546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc06ff10 0x7f15bc1546e0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f15b000d600 tx=0x7f15b000dac0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.547 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 shutdown_connections 2024-03-06T07:31:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f15a806b4d0 0x7f15a806d990 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15bc0c8f20 0x7f15bc1e9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15bc0c7170 0x7f15bc0a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 --2- 172.21.15.94:0/2806401250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15bc06ff10 0x7f15bc1546e0 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 >> 172.21.15.94:0/2806401250 conn(0x7f15bc06b970 msgr2=0x7f15bc0c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 shutdown_connections 2024-03-06T07:31:17.548 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.545+0000 7f15c1555640 1 -- 172.21.15.94:0/2806401250 wait complete. 2024-03-06T07:31:17.557 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:31:17.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:17.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.659+0000 7f0021fe9640 1 Processor -- start 2024-03-06T07:31:17.661 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.659+0000 7f0021fe9640 1 -- start start 2024-03-06T07:31:17.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f0021fe9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 0x7f001c1502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f0021fe9640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f001c0c2920 0x7f001c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f0021fe9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c070430 con 0x7f001c0c1d10 2024-03-06T07:31:17.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f0021fe9640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001c0711f0 con 0x7f001c0c2920 2024-03-06T07:31:17.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f0021fe9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c070d60 con 0x7f001c0abb60 2024-03-06T07:31:17.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f001c0c2920 0x7f001c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001bfff640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f001c0c2920 0x7f001c156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44866/0 (socket says 172.21.15.94:44866) 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001bfff640 1 -- 172.21.15.94:0/980856031 learned_addr learned my addr 172.21.15.94:0/980856031 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001b7fe640 1 --2- 172.21.15.94:0/980856031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 0x7f001c1502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001bfff640 1 -- 172.21.15.94:0/980856031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 msgr2=0x7f001c1502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001bfff640 1 --2- 172.21.15.94:0/980856031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 0x7f001c1502f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001bfff640 1 -- 172.21.15.94:0/980856031 >> v1:172.21.15.145:6789/0 conn(0x7f001c0c1d10 legacy=0x7f001c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.660+0000 7f001bfff640 1 -- 172.21.15.94:0/980856031 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c070710 con 0x7f001c0c2920 2024-03-06T07:31:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.661+0000 7f001bfff640 1 --2- 172.21.15.94:0/980856031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c156b50 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f001402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9478f5fad116396 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.661+0000 7f001a7fc640 1 -- 172.21.15.94:0/980856031 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001402c9e0 con 0x7f001c0c2920 2024-03-06T07:31:17.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.661+0000 7f001a7fc640 1 -- 172.21.15.94:0/980856031 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001402cb80 con 0x7f001c0c2920 2024-03-06T07:31:17.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.661+0000 7f001a7fc640 1 -- 172.21.15.94:0/980856031 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00140343f0 con 0x7f001c0c2920 2024-03-06T07:31:17.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 -- 172.21.15.94:0/980856031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 msgr2=0x7f001c156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 --2- 172.21.15.94:0/980856031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c156b50 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f001402be40 comp rx=0 tx=0).stop 2024-03-06T07:31:17.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 -- 172.21.15.94:0/980856031 shutdown_connections 2024-03-06T07:31:17.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 --2- 172.21.15.94:0/980856031 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c156b50 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 --2- 172.21.15.94:0/980856031 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 0x7f001c1502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 -- 172.21.15.94:0/980856031 >> 172.21.15.94:0/980856031 conn(0x7f001c0c65b0 msgr2=0x7f001c0c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 -- 172.21.15.94:0/980856031 shutdown_connections 2024-03-06T07:31:17.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.662+0000 7f0021fe9640 1 -- 172.21.15.94:0/980856031 wait complete. 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.663+0000 7f0021fe9640 1 Processor -- start 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.663+0000 7f0021fe9640 1 -- start start 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 0x7f001c0c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c0bbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f001c149ff0 0x7f001c1488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c0abf40 con 0x7f001c0c2920 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c157090 con 0x7f001c149ff0 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c0ac0c0 con 0x7f001c0abb60 2024-03-06T07:31:17.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c0bbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c0bbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44878/0 (socket says 172.21.15.94:44878) 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 -- 172.21.15.94:0/3907316131 learned_addr learned my addr 172.21.15.94:0/3907316131 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001bfff640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f001c149ff0 0x7f001c1488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 -- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 msgr2=0x7f001c0c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f001c0abb60 0x7f001c0c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 -- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f001c149ff0 msgr2=0x7f001c1488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f001c149ff0 0x7f001c1488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.670 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 -- 172.21.15.94:0/3907316131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c1496d0 con 0x7f001c0c2920 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f001affd640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c0bbed0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f001000c460 tx=0x7f001000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010016020 con 0x7f001c0c2920 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0010012870 con 0x7f001c0c2920 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c155e40 con 0x7f001c0c2920 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010012c60 con 0x7f001c0c2920 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.664+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c0c4f10 con 0x7f001c0c2920 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.665+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f001c155c30 con 0x7f001c0c2920 2024-03-06T07:31:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.666+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0010019020 con 0x7f001c0c2920 2024-03-06T07:31:17.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.666+0000 7f0003fff640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f000806b4d0 0x7f000806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.666+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f0010089e70 con 0x7f001c0c2920 2024-03-06T07:31:17.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.666+0000 7f0003fff640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f0008071b50 0x7f0008073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.666+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 --> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f001001ae70 con 0x7f0008071b50 2024-03-06T07:31:17.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.666+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7f001008c570 con 0x7f001c0c2920 2024-03-06T07:31:17.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.667+0000 7f001b7fe640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f000806b4d0 0x7f000806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.667+0000 7f001bfff640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f0008071b50 0x7f0008073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.667+0000 7f001b7fe640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f000806b4d0 0x7f000806d990 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f000c006fd0 tx=0x7f000c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.667+0000 7f001bfff640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f0008071b50 0x7f0008073f90 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.668+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== osd.4 v2:172.21.15.145:6800/4137837373 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f001001ae70 con 0x7f0008071b50 2024-03-06T07:31:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 --> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f001c04f330 con 0x7f0008071b50 2024-03-06T07:31:17.683 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0003fff640 1 -- 172.21.15.94:0/3907316131 <== osd.4 v2:172.21.15.145:6800/4137837373 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f001c04f330 con 0x7f0008071b50 2024-03-06T07:31:17.683 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:17.683 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:17.683 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:17.684 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f0008071b50 msgr2=0x7f0008073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6800/4137837373,v1:172.21.15.145:6801/4137837373] conn(0x7f0008071b50 0x7f0008073f90 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f000806b4d0 msgr2=0x7f000806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f000806b4d0 0x7f000806d990 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f000c006fd0 tx=0x7f000c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 msgr2=0x7f001c0bbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 --2- 172.21.15.94:0/3907316131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f001c0c2920 0x7f001c0bbed0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f001000c460 tx=0x7f001000c920 comp rx=0 tx=0).stop 2024-03-06T07:31:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f001bfff640 1 -- 172.21.15.94:0/3907316131 reap_dead start 2024-03-06T07:31:17.685 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 shutdown_connections 2024-03-06T07:31:17.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.681+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 >> 172.21.15.94:0/3907316131 conn(0x7f001c0c65b0 msgr2=0x7f001c06d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.682+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 shutdown_connections 2024-03-06T07:31:17.686 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.682+0000 7f0021fe9640 1 -- 172.21.15.94:0/3907316131 wait complete. 2024-03-06T07:31:17.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:31:17.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.791+0000 7f0ee7eed640 1 Processor -- start 2024-03-06T07:31:17.793 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.791+0000 7f0ee7eed640 1 -- start start 2024-03-06T07:31:17.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee7eed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee01529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee7eed640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ee014f110 0x7f0ee014f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee7eed640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00a9990 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee7eed640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee0155d10 con 0x7f0ee014f110 2024-03-06T07:31:17.795 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee7eed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee01556c0 con 0x7f0ee01561a0 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee648a640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0ee014f110 0x7f0ee014f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee5c89640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0ee00c6ab0 0x7f0ee00a9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47160/0 (socket says 172.21.15.94:47160) 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee5c89640 1 -- 172.21.15.94:0/3771414903 learned_addr learned my addr 172.21.15.94:0/3771414903 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee648a640 1 -- 172.21.15.94:0/3771414903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 msgr2=0x7f0ee01529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee648a640 1 --2- 172.21.15.94:0/3771414903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee01529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee648a640 1 -- 172.21.15.94:0/3771414903 >> v1:172.21.15.94:6789/0 conn(0x7f0ee00c6ab0 legacy=0x7f0ee00a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee648a640 1 -- 172.21.15.94:0/3771414903 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee01559a0 con 0x7f0ee014f110 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.792+0000 7f0ee4c87640 1 -- 172.21.15.94:0/3771414903 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433702754 0 0) 0x7f0ee00a9990 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.793+0000 7f0ee648a640 1 --2- 172.21.15.94:0/3771414903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee014f110 0x7f0ee014f570 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f0ed80097a0 tx=0x7f0ed802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa59bf66cae6dbb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.793+0000 7f0ee4c87640 1 -- 172.21.15.94:0/3771414903 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80315a0 con 0x7f0ee014f110 2024-03-06T07:31:17.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.793+0000 7f0ee4c87640 1 -- 172.21.15.94:0/3771414903 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed8037070 con 0x7f0ee014f110 2024-03-06T07:31:17.797 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.793+0000 7f0ee4c87640 1 -- 172.21.15.94:0/3771414903 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8031960 con 0x7f0ee014f110 2024-03-06T07:31:17.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 -- 172.21.15.94:0/3771414903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee014f110 msgr2=0x7f0ee014f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.800 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/3771414903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee014f110 0x7f0ee014f570 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f0ed80097a0 tx=0x7f0ed802b840 comp rx=0 tx=0).stop 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 -- 172.21.15.94:0/3771414903 shutdown_connections 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/3771414903 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee014f110 0x7f0ee014f570 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/3771414903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee01529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 -- 172.21.15.94:0/3771414903 >> 172.21.15.94:0/3771414903 conn(0x7f0ee006b1a0 msgr2=0x7f0ee006b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 -- 172.21.15.94:0/3771414903 shutdown_connections 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.794+0000 7f0ee7eed640 1 -- 172.21.15.94:0/3771414903 wait complete. 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.795+0000 7f0ee7eed640 1 Processor -- start 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.795+0000 7f0ee7eed640 1 -- start start 2024-03-06T07:31:17.801 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee00c6ab0 0x7f0ee0143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ee014f110 0x7f0ee00bfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee00716c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00aa600 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00c2a90 con 0x7f0ee014f110 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0142cf0 con 0x7f0ee01561a0 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee00c6ab0 0x7f0ee0143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee00c6ab0 0x7f0ee0143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44900/0 (socket says 172.21.15.94:44900) 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 -- 172.21.15.94:0/1911354298 learned_addr learned my addr 172.21.15.94:0/1911354298 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:17.802 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 -- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 msgr2=0x7f0ee00716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee648a640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee00716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee00716c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 -- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ee014f110 msgr2=0x7f0ee00bfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5488640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ee014f110 0x7f0ee00bfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ee014f110 0x7f0ee00bfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 -- 172.21.15.94:0/1911354298 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee01ee210 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee648a640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee00716c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5488640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ee014f110 0x7f0ee00bfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee5c89640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee00c6ab0 0x7f0ee0143310 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00c7c0 tx=0x7f0edc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.803 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ed67fc640 1 -- 172.21.15.94:0/1911354298 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc018070 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ed67fc640 1 -- 172.21.15.94:0/1911354298 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0edc00a640 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee01ee4f0 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ed67fc640 1 -- 172.21.15.94:0/1911354298 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc00aa30 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.796+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee00c0180 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.797+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee004f330 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.798+0000 7f0ed67fc640 1 -- 172.21.15.94:0/1911354298 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0edc01b070 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.798+0000 7f0ed67fc640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.804 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.798+0000 7f0ed67fc640 1 -- 172.21.15.94:0/1911354298 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f0edc0897c0 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.799+0000 7f0ee5488640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.799+0000 7f0ee5488640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ec406b4d0 0x7f0ec406d990 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0006fd0 tx=0x7f0ed0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.805 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.803+0000 7f0ed67fc640 1 -- 172.21.15.94:0/1911354298 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0edc0586b0 con 0x7f0ee00c6ab0 2024-03-06T07:31:17.975 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.973+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ee00abcc0 con 0x7f0ec406b4d0 2024-03-06T07:31:17.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.973+0000 7f0ed67fc640 1 -- 172.21.15.94:0/1911354298 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ee00abcc0 con 0x7f0ec406b4d0 2024-03-06T07:31:17.976 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ec406b4d0 msgr2=0x7f0ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ec406b4d0 0x7f0ec406d990 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0006fd0 tx=0x7f0ed0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee00c6ab0 msgr2=0x7f0ee0143310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee00c6ab0 0x7f0ee0143310 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00c7c0 tx=0x7f0edc00cc80 comp rx=0 tx=0).stop 2024-03-06T07:31:17.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 shutdown_connections 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f0ec406b4d0 0x7f0ec406d990 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ee01561a0 0x7f0ee00716c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ee014f110 0x7f0ee00bfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 --2- 172.21.15.94:0/1911354298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0ee00c6ab0 0x7f0ee0143310 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 >> 172.21.15.94:0/1911354298 conn(0x7f0ee006b1a0 msgr2=0x7f0ee0148dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 shutdown_connections 2024-03-06T07:31:17.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:17.975+0000 7f0ee7eed640 1 -- 172.21.15.94:0/1911354298 wait complete. 2024-03-06T07:31:17.987 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:31:17.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.091+0000 7f8002f93640 1 Processor -- start 2024-03-06T07:31:18.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.091+0000 7f8002f93640 1 -- start start 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f8002f93640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ffc1551d0 0x7f7ffc1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f8002f93640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc0c8f60 con 0x7f7ffc0c8980 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f8002f93640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc0c9240 con 0x7f7ffc14f180 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f8002f93640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc156c00 con 0x7f7ffc1551d0 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f8001530640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ffc1551d0 0x7f7ffc1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f7ffbfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7ffc0c8980 0x7f7ffc06c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47176/0 (socket says 172.21.15.94:47176) 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f8001530640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7ffc1551d0 0x7f7ffc1566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44908/0 (socket says 172.21.15.94:44908) 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.092+0000 7f7ffbfff640 1 -- 172.21.15.94:0/1172193644 learned_addr learned my addr 172.21.15.94:0/1172193644 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f8001530640 1 -- 172.21.15.94:0/1172193644 >> v1:172.21.15.145:6789/0 conn(0x7f7ffc14f180 legacy=0x7f7ffc0ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f8001530640 1 -- 172.21.15.94:0/1172193644 >> v1:172.21.15.94:6789/0 conn(0x7f7ffc0c8980 legacy=0x7f7ffc06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f8001530640 1 -- 172.21.15.94:0/1172193644 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc0c9520 con 0x7f7ffc1551d0 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f7ffb7fe640 1 -- 172.21.15.94:0/1172193644 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232519929 0 0) 0x7f7ffc0c8f60 con 0x7f7ffc0c8980 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f8001530640 1 --2- 172.21.15.94:0/1172193644 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc1551d0 0x7f7ffc1566c0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc156d80 tx=0x7f7ff402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947c81c2a2666d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f7ffb7fe640 1 -- 172.21.15.94:0/1172193644 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4039070 con 0x7f7ffc1551d0 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f7ffb7fe640 1 -- 172.21.15.94:0/1172193644 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff402cce0 con 0x7f7ffc1551d0 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.093+0000 7f7ffb7fe640 1 -- 172.21.15.94:0/1172193644 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4034660 con 0x7f7ffc1551d0 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.094+0000 7f8002f93640 1 -- 172.21.15.94:0/1172193644 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc1551d0 msgr2=0x7f7ffc1566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.094+0000 7f8002f93640 1 --2- 172.21.15.94:0/1172193644 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc1551d0 0x7f7ffc1566c0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc156d80 tx=0x7f7ff402bf30 comp rx=0 tx=0).stop 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.094+0000 7f8002f93640 1 -- 172.21.15.94:0/1172193644 shutdown_connections 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.094+0000 7f8002f93640 1 --2- 172.21.15.94:0/1172193644 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc1551d0 0x7f7ffc1566c0 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.094+0000 7f8002f93640 1 -- 172.21.15.94:0/1172193644 >> 172.21.15.94:0/1172193644 conn(0x7f7ffc072d20 msgr2=0x7f7ffc0ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.094+0000 7f8002f93640 1 -- 172.21.15.94:0/1172193644 shutdown_connections 2024-03-06T07:31:18.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.094+0000 7f8002f93640 1 -- 172.21.15.94:0/1172193644 wait complete. 2024-03-06T07:31:18.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.095+0000 7f8002f93640 1 Processor -- start 2024-03-06T07:31:18.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.095+0000 7f8002f93640 1 -- start start 2024-03-06T07:31:18.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8002f93640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc0c8980 0x7f7ffc1ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8002f93640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ffc14f180 0x7f7ffc1ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8002f93640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ffc1551d0 0x7f7ffc1ec080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8002f93640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc14f560 con 0x7f7ffc0c8980 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8002f93640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc0c9ae0 con 0x7f7ffc1551d0 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8002f93640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc14f6e0 con 0x7f7ffc14f180 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc0c8980 0x7f7ffc1ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc0c8980 0x7f7ffc1ede60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44916/0 (socket says 172.21.15.94:44916) 2024-03-06T07:31:18.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 -- 172.21.15.94:0/1059862332 learned_addr learned my addr 172.21.15.94:0/1059862332 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f7ffbfff640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ffc14f180 0x7f7ffc1ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8001530640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ffc1551d0 0x7f7ffc1ec080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 -- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ffc14f180 msgr2=0x7f7ffc1ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ffc14f180 0x7f7ffc1ebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 -- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ffc1551d0 msgr2=0x7f7ffc1ec080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ffc1551d0 0x7f7ffc1ec080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8000d2f640 1 -- 172.21.15.94:0/1059862332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc1e2310 con 0x7f7ffc0c8980 2024-03-06T07:31:18.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f7ffbfff640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ffc14f180 0x7f7ffc1ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.096+0000 7f8001530640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ffc1551d0 0x7f7ffc1ec080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.097+0000 7f8000d2f640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc0c8980 0x7f7ffc1ede60 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c480 tx=0x7f7fec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.097+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0052d0 con 0x7f7ffc0c8980 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.097+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec005470 con 0x7f7ffc0c8980 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.097+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc14b1c0 con 0x7f7ffc0c8980 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.097+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec012870 con 0x7f7ffc0c8980 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.097+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc14b570 con 0x7f7ffc0c8980 2024-03-06T07:31:18.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.097+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ffc04f330 con 0x7f7ffc0c8980 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.098+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fec012a10 con 0x7f7ffc0c8980 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.098+0000 7f7ff97fa640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe806b4d0 0x7f7fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f7fec089d90 con 0x7f7ffc0c8980 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f7ff97fa640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f7fe8071b50 0x7f7fe8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 --> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fe8074660 con 0x7f7fe8071b50 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fec08c490 con 0x7f7ffc0c8980 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f7ffbfff640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe806b4d0 0x7f7fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f8001530640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f7fe8071b50 0x7f7fe8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f7ffbfff640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe806b4d0 0x7f7fe806d990 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc14cd90 tx=0x7f7ff0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.099+0000 7f8001530640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f7fe8071b50 0x7f7fe8073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.100+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== osd.5 v2:172.21.15.145:6808/2538796010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7fe8074660 con 0x7f7fe8071b50 2024-03-06T07:31:18.115 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.113+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 --> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7ffc0afe70 con 0x7f7fe8071b50 2024-03-06T07:31:18.116 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f7ff97fa640 1 -- 172.21.15.94:0/1059862332 <== osd.5 v2:172.21.15.145:6808/2538796010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7ffc0afe70 con 0x7f7fe8071b50 2024-03-06T07:31:18.116 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:18.116 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:18.116 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f7fe8071b50 msgr2=0x7f7fe8073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6808/2538796010,v1:172.21.15.145:6809/2538796010] conn(0x7f7fe8071b50 0x7f7fe8073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.117 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe806b4d0 msgr2=0x7f7fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f7fe806b4d0 0x7f7fe806d990 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc14cd90 tx=0x7f7ff0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc0c8980 msgr2=0x7f7ffc1ede60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 --2- 172.21.15.94:0/1059862332 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7ffc0c8980 0x7f7ffc1ede60 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c480 tx=0x7f7fec00c940 comp rx=0 tx=0).stop 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8001530640 1 -- 172.21.15.94:0/1059862332 reap_dead start 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 shutdown_connections 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 >> 172.21.15.94:0/1059862332 conn(0x7f7ffc072d20 msgr2=0x7f7ffc14c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 shutdown_connections 2024-03-06T07:31:18.118 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.114+0000 7f8002f93640 1 -- 172.21.15.94:0/1059862332 wait complete. 2024-03-06T07:31:18.126 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:31:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.224+0000 7f80e8fd0640 1 Processor -- start 2024-03-06T07:31:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.225+0000 7f80e8fd0640 1 -- start start 2024-03-06T07:31:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.225+0000 7f80e8fd0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f80e406c240 0x7f80e4150210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.225+0000 7f80e8fd0640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f80e40c4740 0x7f80e4150750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.225+0000 7f80e8fd0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e4072390 0x7f80e4153320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.229 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.225+0000 7f80e8fd0640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e4073130 con 0x7f80e40c4740 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.225+0000 7f80e8fd0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e40c2f50 con 0x7f80e406c240 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.225+0000 7f80e8fd0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40c2c70 con 0x7f80e4072390 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f80e40c4740 0x7f80e4150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f80e40c4740 0x7f80e4150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44918/0 (socket says 172.21.15.94:44918) 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 -- 172.21.15.94:0/726702713 learned_addr learned my addr 172.21.15.94:0/726702713 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e259c640 1 --2- 172.21.15.94:0/726702713 >> v2:172.21.15.145:3300/0 conn(0x7f80e406c240 0x7f80e4150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e2d9d640 1 --2- 172.21.15.94:0/726702713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e4072390 0x7f80e4153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.230 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 -- 172.21.15.94:0/726702713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e4072390 msgr2=0x7f80e4153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 --2- 172.21.15.94:0/726702713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e4072390 0x7f80e4153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 -- 172.21.15.94:0/726702713 >> v2:172.21.15.145:3300/0 conn(0x7f80e406c240 msgr2=0x7f80e4150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 --2- 172.21.15.94:0/726702713 >> v2:172.21.15.145:3300/0 conn(0x7f80e406c240 0x7f80e4150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e1d9b640 1 -- 172.21.15.94:0/726702713 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e4072ae0 con 0x7f80e40c4740 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.226+0000 7f80e259c640 1 --2- 172.21.15.94:0/726702713 >> v2:172.21.15.145:3300/0 conn(0x7f80e406c240 0x7f80e4150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e1d9b640 1 --2- 172.21.15.94:0/726702713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e40c4740 0x7f80e4150750 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f80e40c2e80 tx=0x7f80d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98727221a17ecbbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e159a640 1 -- 172.21.15.94:0/726702713 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d80317c0 con 0x7f80e40c4740 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e159a640 1 -- 172.21.15.94:0/726702713 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d8037070 con 0x7f80e40c4740 2024-03-06T07:31:18.231 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e159a640 1 -- 172.21.15.94:0/726702713 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8031b80 con 0x7f80e40c4740 2024-03-06T07:31:18.234 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 -- 172.21.15.94:0/726702713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e40c4740 msgr2=0x7f80e4150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/726702713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e40c4740 0x7f80e4150750 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f80e40c2e80 tx=0x7f80d802b860 comp rx=0 tx=0).stop 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 -- 172.21.15.94:0/726702713 shutdown_connections 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/726702713 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e4072390 0x7f80e4153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/726702713 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e40c4740 0x7f80e4150750 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/726702713 >> v2:172.21.15.145:3300/0 conn(0x7f80e406c240 0x7f80e4150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 -- 172.21.15.94:0/726702713 >> 172.21.15.94:0/726702713 conn(0x7f80e40c7670 msgr2=0x7f80e40c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 -- 172.21.15.94:0/726702713 shutdown_connections 2024-03-06T07:31:18.235 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.227+0000 7f80e8fd0640 1 -- 172.21.15.94:0/726702713 wait complete. 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 Processor -- start 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 -- start start 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e406c240 0x7f80e41ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80e4072390 0x7f80e41ee820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e40c5290 0x7f80e40c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e406c7a0 con 0x7f80e406c240 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e406c620 con 0x7f80e4072390 2024-03-06T07:31:18.236 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e8fd0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40c1570 con 0x7f80e40c5290 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e406c240 0x7f80e41ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e259c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e406c240 0x7f80e41ee2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44932/0 (socket says 172.21.15.94:44932) 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.229+0000 7f80e259c640 1 -- 172.21.15.94:0/1805527887 learned_addr learned my addr 172.21.15.94:0/1805527887 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e1d9b640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80e4072390 0x7f80e41ee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e2d9d640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e40c5290 0x7f80e40c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e259c640 1 -- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e40c5290 msgr2=0x7f80e40c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e259c640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e40c5290 0x7f80e40c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e259c640 1 -- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80e4072390 msgr2=0x7f80e41ee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.237 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e259c640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80e4072390 0x7f80e41ee820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e259c640 1 -- 172.21.15.94:0/1805527887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40abef0 con 0x7f80e406c240 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e2d9d640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e40c5290 0x7f80e40c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e259c640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e406c240 0x7f80e41ee2e0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f80d400e810 tx=0x7f80d400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80d2ffd640 1 -- 172.21.15.94:0/1805527887 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d400f040 con 0x7f80e406c240 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80d2ffd640 1 -- 172.21.15.94:0/1805527887 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d4004030 con 0x7f80e406c240 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e414d1d0 con 0x7f80e406c240 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80d2ffd640 1 -- 172.21.15.94:0/1805527887 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d4004400 con 0x7f80e406c240 2024-03-06T07:31:18.238 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.230+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e414b7b0 con 0x7f80e406c240 2024-03-06T07:31:18.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.231+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e404f330 con 0x7f80e406c240 2024-03-06T07:31:18.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.232+0000 7f80d2ffd640 1 -- 172.21.15.94:0/1805527887 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80d400c670 con 0x7f80e406c240 2024-03-06T07:31:18.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.232+0000 7f80d2ffd640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f80c406b4d0 0x7f80c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.232+0000 7f80d2ffd640 1 -- 172.21.15.94:0/1805527887 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f80d4089d40 con 0x7f80e406c240 2024-03-06T07:31:18.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.232+0000 7f80e1d9b640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f80c406b4d0 0x7f80c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.239 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.233+0000 7f80e1d9b640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f80c406b4d0 0x7f80c406d990 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f80d8036000 tx=0x7f80d803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.240 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.236+0000 7f80d2ffd640 1 -- 172.21.15.94:0/1805527887 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80d4017020 con 0x7f80e406c240 2024-03-06T07:31:18.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.409+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f80e4150c40 con 0x7f80c406b4d0 2024-03-06T07:31:18.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.410+0000 7f80d2ffd640 1 -- 172.21.15.94:0/1805527887 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f80e4150c40 con 0x7f80c406b4d0 2024-03-06T07:31:18.412 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:31:18.414 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f80c406b4d0 msgr2=0x7f80c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f80c406b4d0 0x7f80c406d990 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f80d8036000 tx=0x7f80d803d040 comp rx=0 tx=0).stop 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e406c240 msgr2=0x7f80e41ee2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e406c240 0x7f80e41ee2e0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f80d400e810 tx=0x7f80d400ecd0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 shutdown_connections 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f80c406b4d0 0x7f80c406d990 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80e40c5290 0x7f80e40c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80e4072390 0x7f80e41ee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 --2- 172.21.15.94:0/1805527887 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80e406c240 0x7f80e41ee2e0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.415 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 >> 172.21.15.94:0/1805527887 conn(0x7f80e40c7670 msgr2=0x7f80e406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 shutdown_connections 2024-03-06T07:31:18.416 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.412+0000 7f80e8fd0640 1 -- 172.21.15.94:0/1805527887 wait complete. 2024-03-06T07:31:18.425 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:31:18.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:18.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.528+0000 7f783cc1c640 1 Processor -- start 2024-03-06T07:31:18.530 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.528+0000 7f783cc1c640 1 -- start start 2024-03-06T07:31:18.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f783cc1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380abb60 0x7f78381502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f783cc1c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78380c1d10 0x7f7838156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f783cc1c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78380c2920 0x7f7838156980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f783cc1c640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78380711f0 con 0x7f78380c2920 2024-03-06T07:31:18.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f783cc1c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7838070d60 con 0x7f78380c1d10 2024-03-06T07:31:18.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f783cc1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7838070430 con 0x7f78380abb60 2024-03-06T07:31:18.533 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78380c2920 0x7f7838156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f78380c2920 0x7f7838156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59508/0 (socket says 172.21.15.94:59508) 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 -- 172.21.15.94:0/276993844 learned_addr learned my addr 172.21.15.94:0/276993844 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f782ffff640 1 --2- 172.21.15.94:0/276993844 >> v2:172.21.15.145:3300/0 conn(0x7f78380c1d10 0x7f7838156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f783659c640 1 --2- 172.21.15.94:0/276993844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380abb60 0x7f78381502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 -- 172.21.15.94:0/276993844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380abb60 msgr2=0x7f78381502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 --2- 172.21.15.94:0/276993844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380abb60 0x7f78381502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 -- 172.21.15.94:0/276993844 >> v2:172.21.15.145:3300/0 conn(0x7f78380c1d10 msgr2=0x7f7838156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.534 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 --2- 172.21.15.94:0/276993844 >> v2:172.21.15.145:3300/0 conn(0x7f78380c1d10 0x7f7838156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.529+0000 7f7836d9d640 1 -- 172.21.15.94:0/276993844 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838070710 con 0x7f78380c2920 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.530+0000 7f7836d9d640 1 --2- 172.21.15.94:0/276993844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c2920 0x7f7838156980 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f78380ac3f0 tx=0x7f783002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45477e6e9c5e651a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.530+0000 7f7835d9b640 1 -- 172.21.15.94:0/276993844 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783002c9e0 con 0x7f78380c2920 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.530+0000 7f7835d9b640 1 -- 172.21.15.94:0/276993844 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783002cb80 con 0x7f78380c2920 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.530+0000 7f7835d9b640 1 -- 172.21.15.94:0/276993844 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78300343f0 con 0x7f78380c2920 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.530+0000 7f783cc1c640 1 -- 172.21.15.94:0/276993844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c2920 msgr2=0x7f7838156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.530+0000 7f783cc1c640 1 --2- 172.21.15.94:0/276993844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c2920 0x7f7838156980 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f78380ac3f0 tx=0x7f783002be40 comp rx=0 tx=0).stop 2024-03-06T07:31:18.535 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.531+0000 7f783cc1c640 1 -- 172.21.15.94:0/276993844 shutdown_connections 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.531+0000 7f783cc1c640 1 --2- 172.21.15.94:0/276993844 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c2920 0x7f7838156980 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.531+0000 7f783cc1c640 1 --2- 172.21.15.94:0/276993844 >> v2:172.21.15.145:3300/0 conn(0x7f78380c1d10 0x7f7838156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.531+0000 7f783cc1c640 1 --2- 172.21.15.94:0/276993844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380abb60 0x7f78381502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.531+0000 7f783cc1c640 1 -- 172.21.15.94:0/276993844 >> 172.21.15.94:0/276993844 conn(0x7f78380c65b0 msgr2=0x7f78380c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.531+0000 7f783cc1c640 1 -- 172.21.15.94:0/276993844 shutdown_connections 2024-03-06T07:31:18.536 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.531+0000 7f783cc1c640 1 -- 172.21.15.94:0/276993844 wait complete. 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.532+0000 7f783cc1c640 1 Processor -- start 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.532+0000 7f783cc1c640 1 -- start start 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.532+0000 7f783cc1c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78380abb60 0x7f7838151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.532+0000 7f783cc1c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380c2920 0x7f78381e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.532+0000 7f783cc1c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c54b0 0x7f78380bbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.532+0000 7f783cc1c640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7838157040 con 0x7f78380c54b0 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f783cc1c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7838156ec0 con 0x7f78380abb60 2024-03-06T07:31:18.540 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f783cc1c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78380abf40 con 0x7f78380c2920 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c54b0 0x7f78380bbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c54b0 0x7f78380bbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59512/0 (socket says 172.21.15.94:59512) 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 -- 172.21.15.94:0/2021239260 learned_addr learned my addr 172.21.15.94:0/2021239260 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f782ffff640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380c2920 0x7f78381e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 -- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380c2920 msgr2=0x7f78381e94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78380c2920 0x7f78381e94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f783659c640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78380abb60 0x7f7838151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 -- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78380abb60 msgr2=0x7f7838151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.541 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78380abb60 0x7f7838151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 -- 172.21.15.94:0/2021239260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78380bd770 con 0x7f78380c54b0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f7836d9d640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c54b0 0x7f78380bbfb0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f7830009f40 tx=0x7f7830004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7830043020 con 0x7f78380c54b0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783003e450 con 0x7f78380c54b0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78380c5970 con 0x7f78380c54b0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7830009510 con 0x7f78380c54b0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.533+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7838150f40 con 0x7f78380c54b0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.534+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f783804f330 con 0x7f78380c54b0 2024-03-06T07:31:18.542 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.535+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7830046020 con 0x7f78380c54b0 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.535+0000 7f782effd640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f781806b4d0 0x7f781806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.535+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f78300ace90 con 0x7f78380c54b0 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.535+0000 7f782effd640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f7818071b50 0x7f7818073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.535+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 --> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78300020b0 con 0x7f7818071b50 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.535+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78300ad320 con 0x7f78380c54b0 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.535+0000 7f783659c640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f781806b4d0 0x7f781806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.536+0000 7f782ffff640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f7818071b50 0x7f7818073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.543 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.536+0000 7f783659c640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f781806b4d0 0x7f781806d990 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7828006fd0 tx=0x7f7828008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.536+0000 7f782ffff640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f7818071b50 0x7f7818073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.544 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.537+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== osd.6 v2:172.21.15.145:6816/1405026475 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f78300020b0 con 0x7f7818071b50 2024-03-06T07:31:18.551 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.550+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 --> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f783814ea00 con 0x7f7818071b50 2024-03-06T07:31:18.552 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.550+0000 7f782effd640 1 -- 172.21.15.94:0/2021239260 <== osd.6 v2:172.21.15.145:6816/1405026475 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f783814ea00 con 0x7f7818071b50 2024-03-06T07:31:18.552 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:18.553 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:18.553 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:18.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f7818071b50 msgr2=0x7f7818073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6816/1405026475,v1:172.21.15.145:6817/1405026475] conn(0x7f7818071b50 0x7f7818073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f781806b4d0 msgr2=0x7f781806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f781806b4d0 0x7f781806d990 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7828006fd0 tx=0x7f7828008040 comp rx=0 tx=0).stop 2024-03-06T07:31:18.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c54b0 msgr2=0x7f78380bbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 --2- 172.21.15.94:0/2021239260 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78380c54b0 0x7f78380bbfb0 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f7830009f40 tx=0x7f7830004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:18.554 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f7836d9d640 1 -- 172.21.15.94:0/2021239260 reap_dead start 2024-03-06T07:31:18.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 shutdown_connections 2024-03-06T07:31:18.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 >> 172.21.15.94:0/2021239260 conn(0x7f78380c65b0 msgr2=0x7f78381500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 shutdown_connections 2024-03-06T07:31:18.555 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.551+0000 7f783cc1c640 1 -- 172.21.15.94:0/2021239260 wait complete. 2024-03-06T07:31:18.563 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:31:18.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.661+0000 7f1cd4fd7640 1 Processor -- start 2024-03-06T07:31:18.663 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.661+0000 7f1cd4fd7640 1 -- start start 2024-03-06T07:31:18.664 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1cd4fd7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1cd00c7170 0x7f1cd00c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1cd4fd7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd00a8390 con 0x7f1cd00c6c40 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1cd4fd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd00730f0 con 0x7f1cd006ff10 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1cd4fd7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd00c7620 con 0x7f1cd00c7170 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1ccdd9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1cd00c7170 0x7f1cd00c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1cced9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1cd00c6c40 0x7f1cd015dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35670/0 (socket says 172.21.15.94:35670) 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1ccdd9b640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f1cd00c7170 0x7f1cd00c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59516/0 (socket says 172.21.15.94:59516) 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.662+0000 7f1cced9d640 1 -- 172.21.15.94:0/1212823820 learned_addr learned my addr 172.21.15.94:0/1212823820 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:18.665 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.663+0000 7f1ccdd9b640 1 -- 172.21.15.94:0/1212823820 >> v1:172.21.15.145:6789/0 conn(0x7f1cd006ff10 legacy=0x7f1cd0153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.663+0000 7f1ccdd9b640 1 -- 172.21.15.94:0/1212823820 >> v1:172.21.15.94:6789/0 conn(0x7f1cd00c6c40 legacy=0x7f1cd015dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.666 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.663+0000 7f1ccdd9b640 1 -- 172.21.15.94:0/1212823820 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd00733d0 con 0x7f1cd00c7170 2024-03-06T07:31:18.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.663+0000 7f1ccdd9b640 1 --2- 172.21.15.94:0/1212823820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c7170 0x7f1cd00c8810 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40097a0 tx=0x7f1cc402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5317d975395ed68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.663+0000 7f1ccd59a640 1 -- 172.21.15.94:0/1212823820 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4038070 con 0x7f1cd00c7170 2024-03-06T07:31:18.667 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.663+0000 7f1ccd59a640 1 -- 172.21.15.94:0/1212823820 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc4031700 con 0x7f1cd00c7170 2024-03-06T07:31:18.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.663+0000 7f1ccd59a640 1 -- 172.21.15.94:0/1212823820 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4031ad0 con 0x7f1cd00c7170 2024-03-06T07:31:18.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.664+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/1212823820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c7170 msgr2=0x7f1cd00c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.664+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/1212823820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c7170 0x7f1cd00c8810 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40097a0 tx=0x7f1cc402b7a0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.664+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/1212823820 shutdown_connections 2024-03-06T07:31:18.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.664+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/1212823820 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c7170 0x7f1cd00c8810 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.668 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.664+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/1212823820 >> 172.21.15.94:0/1212823820 conn(0x7f1cd006b970 msgr2=0x7f1cd00aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.664+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/1212823820 shutdown_connections 2024-03-06T07:31:18.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.664+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/1212823820 wait complete. 2024-03-06T07:31:18.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.665+0000 7f1cd4fd7640 1 Processor -- start 2024-03-06T07:31:18.669 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.665+0000 7f1cd4fd7640 1 -- start start 2024-03-06T07:31:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cd4fd7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cd006ff10 0x7f1cd01ec070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cd4fd7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c6c40 0x7f1cd014e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cd4fd7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cd00c7170 0x7f1cd014eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cd4fd7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd01544b0 con 0x7f1cd00c6c40 2024-03-06T07:31:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cd4fd7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd0154330 con 0x7f1cd006ff10 2024-03-06T07:31:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cd4fd7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd00702f0 con 0x7f1cd00c7170 2024-03-06T07:31:18.672 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c6c40 0x7f1cd014e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c6c40 0x7f1cd014e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59526/0 (socket says 172.21.15.94:59526) 2024-03-06T07:31:18.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 -- 172.21.15.94:0/3218829689 learned_addr learned my addr 172.21.15.94:0/3218829689 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:18.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cce59c640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cd006ff10 0x7f1cd01ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cced9d640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cd00c7170 0x7f1cd014eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 -- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cd00c7170 msgr2=0x7f1cd014eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cd00c7170 0x7f1cd014eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.673 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 -- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cd006ff10 msgr2=0x7f1cd01ec070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cd006ff10 0x7f1cd01ec070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1ccdd9b640 1 -- 172.21.15.94:0/3218829689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd014efe0 con 0x7f1cd00c6c40 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.666+0000 7f1cced9d640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cd00c7170 0x7f1cd014eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.667+0000 7f1ccdd9b640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c6c40 0x7f1cd014e560 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4000c00 tx=0x7f1cc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.667+0000 7f1cbeffd640 1 -- 172.21.15.94:0/3218829689 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4042070 con 0x7f1cd00c6c40 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.667+0000 7f1cbeffd640 1 -- 172.21.15.94:0/3218829689 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc4033a30 con 0x7f1cd00c6c40 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.667+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd01ec8b0 con 0x7f1cd00c6c40 2024-03-06T07:31:18.674 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.667+0000 7f1cbeffd640 1 -- 172.21.15.94:0/3218829689 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4038040 con 0x7f1cd00c6c40 2024-03-06T07:31:18.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.667+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd01ecb50 con 0x7f1cd00c6c40 2024-03-06T07:31:18.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.668+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd004f330 con 0x7f1cd00c6c40 2024-03-06T07:31:18.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.668+0000 7f1cbeffd640 1 -- 172.21.15.94:0/3218829689 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cc4045070 con 0x7f1cd00c6c40 2024-03-06T07:31:18.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.669+0000 7f1cbeffd640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cb006b500 0x7f1cb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.669+0000 7f1cbeffd640 1 -- 172.21.15.94:0/3218829689 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f1cc40abdb0 con 0x7f1cd00c6c40 2024-03-06T07:31:18.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.669+0000 7f1cce59c640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cb006b500 0x7f1cb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.675 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.669+0000 7f1cce59c640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cb006b500 0x7f1cb006d9c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0005520 tx=0x7f1cc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.676 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.673+0000 7f1cbeffd640 1 -- 172.21.15.94:0/3218829689 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1cd014efe0 con 0x7f1cd00c6c40 2024-03-06T07:31:18.844 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.842+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cd006fa80 con 0x7f1cb006b500 2024-03-06T07:31:18.845 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.843+0000 7f1cbeffd640 1 -- 172.21.15.94:0/3218829689 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1cd006fa80 con 0x7f1cb006b500 2024-03-06T07:31:18.845 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:31:18.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cb006b500 msgr2=0x7f1cb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cb006b500 0x7f1cb006d9c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0005520 tx=0x7f1cc0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:18.847 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c6c40 msgr2=0x7f1cd014e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c6c40 0x7f1cd014e560 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4000c00 tx=0x7f1cc40046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 shutdown_connections 2024-03-06T07:31:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f1cb006b500 0x7f1cb006d9c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cd00c7170 0x7f1cd014eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1cd00c6c40 0x7f1cd014e560 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 --2- 172.21.15.94:0/3218829689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cd006ff10 0x7f1cd01ec070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.848 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 >> 172.21.15.94:0/3218829689 conn(0x7f1cd006b970 msgr2=0x7f1cd0152ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 shutdown_connections 2024-03-06T07:31:18.849 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.845+0000 7f1cd4fd7640 1 -- 172.21.15.94:0/3218829689 wait complete. 2024-03-06T07:31:18.858 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:31:18.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:31:18.963 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.962+0000 7f8f1990d640 1 Processor -- start 2024-03-06T07:31:18.964 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.962+0000 7f8f1990d640 1 -- start start 2024-03-06T07:31:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.962+0000 7f8f1990d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8f1414f180 0x7f8f140c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.967 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f1990d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140c8f60 con 0x7f8f141566c0 2024-03-06T07:31:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f1990d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140c9240 con 0x7f8f141551d0 2024-03-06T07:31:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f1990d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f140c9520 con 0x7f8f1414f180 2024-03-06T07:31:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f137fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8f141566c0 0x7f8f1406c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35672/0 (socket says 172.21.15.94:35672) 2024-03-06T07:31:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f137fe640 1 -- 172.21.15.94:0/309935109 learned_addr learned my addr 172.21.15.94:0/309935109 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267324154 0 0) 0x7f8f140c8f60 con 0x7f8f141566c0 2024-03-06T07:31:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f00003520 con 0x7f8f141566c0 2024-03-06T07:31:18.968 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f12ffd640 1 --2- 172.21.15.94:0/309935109 >> v2:172.21.15.145:3300/0 conn(0x7f8f1414f180 0x7f8f140c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1091302139 0 0) 0x7f8f00003520 con 0x7f8f141566c0 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 >> v2:172.21.15.145:3300/0 conn(0x7f8f1414f180 msgr2=0x7f8f140c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 --2- 172.21.15.94:0/309935109 >> v2:172.21.15.145:3300/0 conn(0x7f8f1414f180 0x7f8f140c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 >> v1:172.21.15.145:6789/0 conn(0x7f8f141551d0 legacy=0x7f8f140ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140c9800 con 0x7f8f141566c0 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2442109139 0 0) 0x7f8f0c0019f0 con 0x7f8f141566c0 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 >> v1:172.21.15.94:6789/0 conn(0x7f8f141566c0 legacy=0x7f8f1406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.969 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f00004800 0x7f8f00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140c9520 con 0x7f8f000039b0 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0000c110 con 0x7f8f000051c0 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0000c310 con 0x7f8f00004800 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f12ffd640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.963+0000 7f8f0bfff640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f00004800 0x7f8f00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f137fe640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f12ffd640 1 -- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f00004800 msgr2=0x7f8f00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.970 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f12ffd640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f00004800 0x7f8f00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f12ffd640 1 -- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000051c0 msgr2=0x7f8f0000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f12ffd640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f12ffd640 1 -- 172.21.15.94:0/309935109 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f00003520 con 0x7f8f000039b0 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f137fe640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000051c0 0x7f8f0000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f137fe640 1 -- 172.21.15.94:0/309935109 reap_dead start 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.964+0000 7f8f12ffd640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400ba30 tx=0x7f8f00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d979effd43118758 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04036020 con 0x7f8f000039b0 2024-03-06T07:31:18.971 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f04032640 con 0x7f8f000039b0 2024-03-06T07:31:18.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f127fc640 1 -- 172.21.15.94:0/309935109 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0400b600 con 0x7f8f000039b0 2024-03-06T07:31:18.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f1990d640 1 -- 172.21.15.94:0/309935109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f000039b0 msgr2=0x7f8f00003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f1990d640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400ba30 tx=0x7f8f00005830 comp rx=0 tx=0).stop 2024-03-06T07:31:18.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f1990d640 1 -- 172.21.15.94:0/309935109 shutdown_connections 2024-03-06T07:31:18.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f1990d640 1 --2- 172.21.15.94:0/309935109 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f000039b0 0x7f8f00003dd0 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.972 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f1990d640 1 -- 172.21.15.94:0/309935109 >> 172.21.15.94:0/309935109 conn(0x7f8f14072d20 msgr2=0x7f8f140ac690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f1990d640 1 -- 172.21.15.94:0/309935109 shutdown_connections 2024-03-06T07:31:18.973 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.965+0000 7f8f1990d640 1 -- 172.21.15.94:0/309935109 wait complete. 2024-03-06T07:31:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.966+0000 7f8f1990d640 1 Processor -- start 2024-03-06T07:31:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.966+0000 7f8f1990d640 1 -- start start 2024-03-06T07:31:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f1990d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000039b0 0x7f8f14152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f1990d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f1414ba50 0x7f8f1414dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f1990d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f1414f860 0x7f8f1414fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f1990d640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1414f560 con 0x7f8f1414f860 2024-03-06T07:31:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f1990d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14156af0 con 0x7f8f000039b0 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f1990d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1414f6e0 con 0x7f8f1414ba50 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f1414f860 0x7f8f1414fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f1414f860 0x7f8f1414fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59548/0 (socket says 172.21.15.94:59548) 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 -- 172.21.15.94:0/252267340 learned_addr learned my addr 172.21.15.94:0/252267340 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 -- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f1414ba50 msgr2=0x7f8f1414dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f1414ba50 0x7f8f1414dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 -- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000039b0 msgr2=0x7f8f14152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f000039b0 0x7f8f14152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.977 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.967+0000 7f8f137fe640 1 -- 172.21.15.94:0/252267340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1414b770 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.968+0000 7f8f137fe640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f1414f860 0x7f8f1414fce0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f8f1406c0b0 tx=0x7f8f0c007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.968+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c00ea80 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.968+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c018070 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.968+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f14157800 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.968+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c00ee60 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.968+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f14157dc0 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.969+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f1404f330 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.969+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f0c01d460 con 0x7f8f1414f860 2024-03-06T07:31:18.978 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f10ff9640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ef406b4d0 0x7f8ef406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8f0c08a460 con 0x7f8f1414f860 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f10ff9640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f8ef4071b50 0x7f8ef4073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 --> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f0c019ec0 con 0x7f8ef4071b50 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=216) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f0c091050 con 0x7f8f1414f860 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f12ffd640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ef406b4d0 0x7f8ef406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f0bfff640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f8ef4071b50 0x7f8ef4073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f12ffd640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ef406b4d0 0x7f8ef406d990 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400ba00 tx=0x7f8f04032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.970+0000 7f8f0bfff640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f8ef4071b50 0x7f8ef4073f90 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.979 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.971+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== osd.7 v2:172.21.15.145:6824/1323110445 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8f0c019ec0 con 0x7f8ef4071b50 2024-03-06T07:31:18.985 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.984+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 --> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8f140af750 con 0x7f8ef4071b50 2024-03-06T07:31:18.986 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f10ff9640 1 -- 172.21.15.94:0/252267340 <== osd.7 v2:172.21.15.145:6824/1323110445 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8f140af750 con 0x7f8ef4071b50 2024-03-06T07:31:18.987 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-03-06T07:31:18.987 INFO:teuthology.orchestra.run.smithi094.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:31:18.987 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-03-06T07:31:18.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f8ef4071b50 msgr2=0x7f8ef4073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6824/1323110445,v1:172.21.15.145:6825/1323110445] conn(0x7f8ef4071b50 0x7f8ef4073f90 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ef406b4d0 msgr2=0x7f8ef406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.988 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f8ef406b4d0 0x7f8ef406d990 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400ba00 tx=0x7f8f04032000 comp rx=0 tx=0).stop 2024-03-06T07:31:18.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f1414f860 msgr2=0x7f8f1414fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 --2- 172.21.15.94:0/252267340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8f1414f860 0x7f8f1414fce0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f8f1406c0b0 tx=0x7f8f0c007bc0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f137fe640 1 -- 172.21.15.94:0/252267340 reap_dead start 2024-03-06T07:31:18.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 shutdown_connections 2024-03-06T07:31:18.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 >> 172.21.15.94:0/252267340 conn(0x7f8f14072d20 msgr2=0x7f8f14153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 shutdown_connections 2024-03-06T07:31:18.989 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:18.985+0000 7f8f1990d640 1 -- 172.21.15.94:0/252267340 wait complete. 2024-03-06T07:31:18.997 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:31:19.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.096+0000 7f10d3a71640 1 Processor -- start 2024-03-06T07:31:19.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.096+0000 7f10d3a71640 1 -- start start 2024-03-06T07:31:19.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d3a71640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f10cc158fa0 0x7f10cc0ac280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d3a71640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc1552a0 con 0x7f10cc0c4250 2024-03-06T07:31:19.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d3a71640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0c4730 con 0x7f10cc06c240 2024-03-06T07:31:19.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d3a71640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10cc14f7e0 con 0x7f10cc158fa0 2024-03-06T07:31:19.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d100c640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f10cc158fa0 0x7f10cc0ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d200e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f10cc0c4250 0x7f10cc0ab150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35680/0 (socket says 172.21.15.94:35680) 2024-03-06T07:31:19.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d200e640 1 -- 172.21.15.94:0/2121220931 learned_addr learned my addr 172.21.15.94:0/2121220931 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:19.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d100c640 1 -- 172.21.15.94:0/2121220931 >> v1:172.21.15.145:6789/0 conn(0x7f10cc06c240 legacy=0x7f10cc158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d100c640 1 -- 172.21.15.94:0/2121220931 >> v1:172.21.15.94:6789/0 conn(0x7f10cc0c4250 legacy=0x7f10cc0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d100c640 1 -- 172.21.15.94:0/2121220931 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc155560 con 0x7f10cc158fa0 2024-03-06T07:31:19.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.097+0000 7f10d080b640 1 -- 172.21.15.94:0/2121220931 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838403512 0 0) 0x7f10cc1552a0 con 0x7f10cc0c4250 2024-03-06T07:31:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d100c640 1 --2- 172.21.15.94:0/2121220931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc158fa0 0x7f10cc0ac280 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0097a0 tx=0x7f10bc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b6f878f00c16c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d080b640 1 -- 172.21.15.94:0/2121220931 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc038070 con 0x7f10cc158fa0 2024-03-06T07:31:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d080b640 1 -- 172.21.15.94:0/2121220931 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc031700 con 0x7f10cc158fa0 2024-03-06T07:31:19.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d080b640 1 -- 172.21.15.94:0/2121220931 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc031ad0 con 0x7f10cc158fa0 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d3a71640 1 -- 172.21.15.94:0/2121220931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc158fa0 msgr2=0x7f10cc0ac280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d3a71640 1 --2- 172.21.15.94:0/2121220931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc158fa0 0x7f10cc0ac280 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0097a0 tx=0x7f10bc02b7a0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d3a71640 1 -- 172.21.15.94:0/2121220931 shutdown_connections 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d3a71640 1 --2- 172.21.15.94:0/2121220931 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc158fa0 0x7f10cc0ac280 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.098+0000 7f10d3a71640 1 -- 172.21.15.94:0/2121220931 >> 172.21.15.94:0/2121220931 conn(0x7f10cc0c7670 msgr2=0x7f10cc0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.099+0000 7f10d3a71640 1 -- 172.21.15.94:0/2121220931 shutdown_connections 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.099+0000 7f10d3a71640 1 -- 172.21.15.94:0/2121220931 wait complete. 2024-03-06T07:31:19.105 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 Processor -- start 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 -- start start 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10cc06c240 0x7f10cc14e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc0c4250 0x7f10cc148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10cc072d20 0x7f10cc148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0c1c00 con 0x7f10cc0c4250 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc155bb0 con 0x7f10cc06c240 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d3a71640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0c1d80 con 0x7f10cc072d20 2024-03-06T07:31:19.106 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d100c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc0c4250 0x7f10cc148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d100c640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc0c4250 0x7f10cc148440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59578/0 (socket says 172.21.15.94:59578) 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.100+0000 7f10d100c640 1 -- 172.21.15.94:0/3133286288 learned_addr learned my addr 172.21.15.94:0/3133286288 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d180d640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10cc06c240 0x7f10cc14e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d200e640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10cc072d20 0x7f10cc148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d100c640 1 -- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10cc072d20 msgr2=0x7f10cc148980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d100c640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10cc072d20 0x7f10cc148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d100c640 1 -- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10cc06c240 msgr2=0x7f10cc14e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d100c640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10cc06c240 0x7f10cc14e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d100c640 1 -- 172.21.15.94:0/3133286288 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc1eeaa0 con 0x7f10cc0c4250 2024-03-06T07:31:19.107 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d180d640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10cc06c240 0x7f10cc14e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d100c640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc0c4250 0x7f10cc148440 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0097a0 tx=0x7f10bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10c27fc640 1 -- 172.21.15.94:0/3133286288 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc043070 con 0x7f10cc0c4250 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10c27fc640 1 -- 172.21.15.94:0/3133286288 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc033a30 con 0x7f10cc0c4250 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10c27fc640 1 -- 172.21.15.94:0/3133286288 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc038040 con 0x7f10cc0c4250 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc1494a0 con 0x7f10cc0c4250 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.101+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc1ee7c0 con 0x7f10cc0c4250 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.102+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc04f330 con 0x7f10cc0c4250 2024-03-06T07:31:19.108 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.102+0000 7f10c27fc640 1 -- 172.21.15.94:0/3133286288 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10bc046070 con 0x7f10cc0c4250 2024-03-06T07:31:19.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.103+0000 7f10c27fc640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10b006b500 0x7f10b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.103+0000 7f10c27fc640 1 -- 172.21.15.94:0/3133286288 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f10bc0abf80 con 0x7f10cc0c4250 2024-03-06T07:31:19.109 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.103+0000 7f10d180d640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10b006b500 0x7f10b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.104+0000 7f10d180d640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10b006b500 0x7f10b006d9c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f10cc072980 tx=0x7f10c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.110 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.107+0000 7f10c27fc640 1 -- 172.21.15.94:0/3133286288 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10bc030070 con 0x7f10cc0c4250 2024-03-06T07:31:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.278+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f10cc072a80 con 0x7f10b006b500 2024-03-06T07:31:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.279+0000 7f10c27fc640 1 -- 172.21.15.94:0/3133286288 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f10cc072a80 con 0x7f10b006b500 2024-03-06T07:31:19.280 INFO:teuthology.orchestra.run.smithi094.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:31:19.282 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10b006b500 msgr2=0x7f10b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10b006b500 0x7f10b006d9c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f10cc072980 tx=0x7f10c8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc0c4250 msgr2=0x7f10cc148440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc0c4250 0x7f10cc148440 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0097a0 tx=0x7f10bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 shutdown_connections 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f10b006b500 0x7f10b006d9c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10cc072d20 0x7f10cc148980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f10cc0c4250 0x7f10cc148440 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.283 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 --2- 172.21.15.94:0/3133286288 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10cc06c240 0x7f10cc14e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.280+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 >> 172.21.15.94:0/3133286288 conn(0x7f10cc0c7670 msgr2=0x7f10cc153800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.281+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 shutdown_connections 2024-03-06T07:31:19.284 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.281+0000 7f10d3a71640 1 -- 172.21.15.94:0/3133286288 wait complete. 2024-03-06T07:31:19.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:19.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.391+0000 7fcfc83c7640 1 Processor -- start 2024-03-06T07:31:19.393 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.391+0000 7fcfc83c7640 1 -- start start 2024-03-06T07:31:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc83c7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcfc00c88c0 0x7fcfc0155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.394 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc83c7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcfc0158b60 0x7fcfc0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc83c7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc00c73d0 con 0x7fcfc014f360 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc83c7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00c8dd0 con 0x7fcfc0158b60 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc83c7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00be2e0 con 0x7fcfc00c88c0 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc6964640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcfc0158b60 0x7fcfc0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc6163640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcfc014f360 0x7fcfc00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35684/0 (socket says 172.21.15.94:35684) 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.392+0000 7fcfc6163640 1 -- 172.21.15.94:0/3388531666 learned_addr learned my addr 172.21.15.94:0/3388531666 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.393+0000 7fcfc5962640 1 --2- 172.21.15.94:0/3388531666 >> v2:172.21.15.145:3300/0 conn(0x7fcfc00c88c0 0x7fcfc0155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.395 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.393+0000 7fcfc6964640 1 -- 172.21.15.94:0/3388531666 >> v2:172.21.15.145:3300/0 conn(0x7fcfc00c88c0 msgr2=0x7fcfc0155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.393+0000 7fcfc6964640 1 --2- 172.21.15.94:0/3388531666 >> v2:172.21.15.145:3300/0 conn(0x7fcfc00c88c0 0x7fcfc0155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.393+0000 7fcfc6964640 1 -- 172.21.15.94:0/3388531666 >> v1:172.21.15.94:6789/0 conn(0x7fcfc014f360 legacy=0x7fcfc00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.393+0000 7fcfc6964640 1 -- 172.21.15.94:0/3388531666 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00c7670 con 0x7fcfc0158b60 2024-03-06T07:31:19.396 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.393+0000 7fcfc5161640 1 -- 172.21.15.94:0/3388531666 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431585602 0 0) 0x7fcfc00c73d0 con 0x7fcfc014f360 2024-03-06T07:31:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.393+0000 7fcfc6964640 1 --2- 172.21.15.94:0/3388531666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc0158b60 0x7fcfc0158ff0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00c84c0 tx=0x7fcfb802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cb12ef0d646f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc5161640 1 -- 172.21.15.94:0/3388531666 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8039070 con 0x7fcfc0158b60 2024-03-06T07:31:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc5161640 1 -- 172.21.15.94:0/3388531666 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb802cbf0 con 0x7fcfc0158b60 2024-03-06T07:31:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc5161640 1 -- 172.21.15.94:0/3388531666 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8034440 con 0x7fcfc0158b60 2024-03-06T07:31:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 -- 172.21.15.94:0/3388531666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc0158b60 msgr2=0x7fcfc0158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.399 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/3388531666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc0158b60 0x7fcfc0158ff0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00c84c0 tx=0x7fcfb802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 -- 172.21.15.94:0/3388531666 shutdown_connections 2024-03-06T07:31:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/3388531666 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc0158b60 0x7fcfc0158ff0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/3388531666 >> v2:172.21.15.145:3300/0 conn(0x7fcfc00c88c0 0x7fcfc0155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 -- 172.21.15.94:0/3388531666 >> 172.21.15.94:0/3388531666 conn(0x7fcfc00ac570 msgr2=0x7fcfc00ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 -- 172.21.15.94:0/3388531666 shutdown_connections 2024-03-06T07:31:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.394+0000 7fcfc83c7640 1 -- 172.21.15.94:0/3388531666 wait complete. 2024-03-06T07:31:19.400 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.395+0000 7fcfc83c7640 1 Processor -- start 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.395+0000 7fcfc83c7640 1 -- start start 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc83c7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc00c88c0 0x7fcfc00c08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc83c7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc014f360 0x7fcfc014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc83c7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc0158b60 0x7fcfc014bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc83c7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c5a50 con 0x7fcfc00c88c0 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc83c7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c7c30 con 0x7fcfc0158b60 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc83c7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c5bd0 con 0x7fcfc014f360 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc00c88c0 0x7fcfc00c08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.401 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc00c88c0 0x7fcfc00c08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:59600/0 (socket says 172.21.15.94:59600) 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 -- 172.21.15.94:0/2027963673 learned_addr learned my addr 172.21.15.94:0/2027963673 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6964640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc0158b60 0x7fcfc014bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc5962640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc014f360 0x7fcfc014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 -- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc014f360 msgr2=0x7fcfc014b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc014f360 0x7fcfc014b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 -- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc0158b60 msgr2=0x7fcfc014bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc0158b60 0x7fcfc014bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.396+0000 7fcfc6163640 1 -- 172.21.15.94:0/2027963673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc01eaaa0 con 0x7fcfc00c88c0 2024-03-06T07:31:19.402 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.397+0000 7fcfc6163640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc00c88c0 0x7fcfc00c08e0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00c500 tx=0x7fcfbc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.397+0000 7fcfb2ffd640 1 -- 172.21.15.94:0/2027963673 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc018070 con 0x7fcfc00c88c0 2024-03-06T07:31:19.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.397+0000 7fcfb2ffd640 1 -- 172.21.15.94:0/2027963673 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc007580 con 0x7fcfc00c88c0 2024-03-06T07:31:19.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.397+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc00be590 con 0x7fcfc00c88c0 2024-03-06T07:31:19.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.397+0000 7fcfb2ffd640 1 -- 172.21.15.94:0/2027963673 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc004a20 con 0x7fcfc00c88c0 2024-03-06T07:31:19.403 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.397+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc00beb50 con 0x7fcfc00c88c0 2024-03-06T07:31:19.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.398+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc004f330 con 0x7fcfc00c88c0 2024-03-06T07:31:19.404 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.398+0000 7fcfb2ffd640 1 -- 172.21.15.94:0/2027963673 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfbc01b070 con 0x7fcfc00c88c0 2024-03-06T07:31:19.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.399+0000 7fcfb2ffd640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfa406b4d0 0x7fcfa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.399+0000 7fcfb2ffd640 1 -- 172.21.15.94:0/2027963673 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fcfbc089430 con 0x7fcfc00c88c0 2024-03-06T07:31:19.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.399+0000 7fcfc5962640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfa406b4d0 0x7fcfa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.407 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.399+0000 7fcfc5962640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfa406b4d0 0x7fcfa406d990 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4006fd0 tx=0x7fcfb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.408 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.403+0000 7fcfb2ffd640 1 -- 172.21.15.94:0/2027963673 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfbc0583b0 con 0x7fcfc00c88c0 2024-03-06T07:31:19.574 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.573+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcfc00c1520 con 0x7fcfa406b4d0 2024-03-06T07:31:19.576 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.574+0000 7fcfb2ffd640 1 -- 172.21.15.94:0/2027963673 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13453 (secure 0 0 0) 0x7fcfc00c1520 con 0x7fcfa406b4d0 2024-03-06T07:31:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:19.576 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:31:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfa406b4d0 msgr2=0x7fcfa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfa406b4d0 0x7fcfa406d990 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4006fd0 tx=0x7fcfb4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc00c88c0 msgr2=0x7fcfc00c08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc00c88c0 0x7fcfc00c08e0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00c500 tx=0x7fcfbc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 shutdown_connections 2024-03-06T07:31:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fcfa406b4d0 0x7fcfa406d990 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.578 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfc0158b60 0x7fcfc014bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfc014f360 0x7fcfc014b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 --2- 172.21.15.94:0/2027963673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcfc00c88c0 0x7fcfc00c08e0 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 >> 172.21.15.94:0/2027963673 conn(0x7fcfc00ac570 msgr2=0x7fcfc00c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 shutdown_connections 2024-03-06T07:31:19.579 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:19.576+0000 7fcfc83c7640 1 -- 172.21.15.94:0/2027963673 wait complete. 2024-03-06T07:31:19.588 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2024-03-06T07:31:18.799821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":283548,"kb_used_data":21600,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754691172,"statfs":{"total":773094113280,"available":772803760128,"internally_reserved":0,"allocated":22118400,"data_stored":11837112,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.349515"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":542,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:31:10.801349+0000","last_change":"2024-03-06T07:18:04.418860+0000","last_active":"2024-03-06T07:31:10.801349+0000","last_peered":"2024-03-06T07:31:10.801349+0000","last_clean":"2024-03-06T07:31:10.801349+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:31:10.801349+0000","last_undegraded":"2024-03-06T07:31:10.801349+0000","last_fullsized":"2024-03-06T07:31:10.801349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:18:03.380542+0000","last_clean_scrub_stamp":"2024-03-06T07:18:03.380542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:54.767800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33928,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337912,"statfs":{"total":96636764160,"available":96602021888,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575009,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38092,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333748,"statfs":{"total":96636764160,"available":96597757952,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:19.591 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:18:03.380542+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=18, tm_sec=3, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=31, tm_sec=14, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:31:19.591 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:31:39.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:39.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.702+0000 7fa30efc7640 1 Processor -- start 2024-03-06T07:31:39.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.702+0000 7fa30efc7640 1 -- start start 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30efc7640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa30814f360 0x7fa308155c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30efc7640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa308158b60 0x7fa308158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30efc7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3080c73d0 con 0x7fa3080c88c0 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30efc7640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080c8dd0 con 0x7fa30814f360 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30efc7640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080be2e0 con 0x7fa308158b60 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30cd63640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa30814f360 0x7fa308155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30cd63640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa30814f360 0x7fa308155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44708/0 (socket says 172.21.15.94:44708) 2024-03-06T07:31:39.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa30cd63640 1 -- 172.21.15.94:0/3205087406 learned_addr learned my addr 172.21.15.94:0/3205087406 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.703+0000 7fa307fff640 1 --1- 172.21.15.94:0/3205087406 >> v1:172.21.15.94:6789/0 conn(0x7fa3080c88c0 0x7fa3080c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36156/0 (socket says 172.21.15.94:36156) 2024-03-06T07:31:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.704+0000 7fa30d564640 1 --2- 172.21.15.94:0/3205087406 >> v2:172.21.15.145:3300/0 conn(0x7fa308158b60 0x7fa308158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.704+0000 7fa30cd63640 1 -- 172.21.15.94:0/3205087406 >> v2:172.21.15.145:3300/0 conn(0x7fa308158b60 msgr2=0x7fa308158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.704+0000 7fa30cd63640 1 --2- 172.21.15.94:0/3205087406 >> v2:172.21.15.145:3300/0 conn(0x7fa308158b60 0x7fa308158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.704+0000 7fa30cd63640 1 -- 172.21.15.94:0/3205087406 >> v1:172.21.15.94:6789/0 conn(0x7fa3080c88c0 legacy=0x7fa3080c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.704+0000 7fa30cd63640 1 -- 172.21.15.94:0/3205087406 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080c7670 con 0x7fa30814f360 2024-03-06T07:31:39.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.704+0000 7fa3077fe640 1 -- 172.21.15.94:0/3205087406 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273438971 0 0) 0x7fa3080c73d0 con 0x7fa3080c88c0 2024-03-06T07:31:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.704+0000 7fa30cd63640 1 --2- 172.21.15.94:0/3205087406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa30814f360 0x7fa308155c10 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fa308152d60 tx=0x7fa2f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab60ffba2d058f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa3077fe640 1 -- 172.21.15.94:0/3205087406 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8035020 con 0x7fa30814f360 2024-03-06T07:31:39.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa3077fe640 1 -- 172.21.15.94:0/3205087406 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8031870 con 0x7fa30814f360 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa3077fe640 1 -- 172.21.15.94:0/3205087406 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8031c40 con 0x7fa30814f360 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 -- 172.21.15.94:0/3205087406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa30814f360 msgr2=0x7fa308155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3205087406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa30814f360 0x7fa308155c10 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7fa308152d60 tx=0x7fa2f802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 -- 172.21.15.94:0/3205087406 shutdown_connections 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3205087406 >> v2:172.21.15.145:3300/0 conn(0x7fa308158b60 0x7fa308158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3205087406 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa30814f360 0x7fa308155c10 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 -- 172.21.15.94:0/3205087406 >> 172.21.15.94:0/3205087406 conn(0x7fa3080ac570 msgr2=0x7fa3080ac980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 -- 172.21.15.94:0/3205087406 shutdown_connections 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.705+0000 7fa30efc7640 1 -- 172.21.15.94:0/3205087406 wait complete. 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.706+0000 7fa30efc7640 1 Processor -- start 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.706+0000 7fa30efc7640 1 -- start start 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30efc7640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3080c88c0 0x7fa3080c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30efc7640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa30814f360 0x7fa30814f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30efc7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa308158b60 0x7fa3080bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30efc7640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080c5a50 con 0x7fa3080c88c0 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30efc7640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080c7c30 con 0x7fa30814f360 2024-03-06T07:31:39.712 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30efc7640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080c5bd0 con 0x7fa308158b60 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30cd63640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3080c88c0 0x7fa3080c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30cd63640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3080c88c0 0x7fa3080c3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44714/0 (socket says 172.21.15.94:44714) 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa30cd63640 1 -- 172.21.15.94:0/3503583551 learned_addr learned my addr 172.21.15.94:0/3503583551 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.707+0000 7fa307fff640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa30814f360 0x7fa30814f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30d564640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa308158b60 0x7fa3080bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30cd63640 1 -- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa308158b60 msgr2=0x7fa3080bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30cd63640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa308158b60 0x7fa3080bf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30cd63640 1 -- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa30814f360 msgr2=0x7fa30814f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.713 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30cd63640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa30814f360 0x7fa30814f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30cd63640 1 -- 172.21.15.94:0/3503583551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080bfd60 con 0x7fa3080c88c0 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa307fff640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa30814f360 0x7fa30814f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30d564640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa308158b60 0x7fa3080bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30cd63640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3080c88c0 0x7fa3080c3c70 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8000c00 tx=0x7fa2f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa3057fa640 1 -- 172.21.15.94:0/3503583551 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8035040 con 0x7fa3080c88c0 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa3057fa640 1 -- 172.21.15.94:0/3503583551 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8030d00 con 0x7fa3080c88c0 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa3057fa640 1 -- 172.21.15.94:0/3503583551 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80456a0 con 0x7fa3080c88c0 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3081ebc60 con 0x7fa3080c88c0 2024-03-06T07:31:39.714 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.708+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3080be8d0 con 0x7fa3080c88c0 2024-03-06T07:31:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.709+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3080c9f60 con 0x7fa3080c88c0 2024-03-06T07:31:39.715 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.710+0000 7fa3057fa640 1 -- 172.21.15.94:0/3503583551 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2f8038020 con 0x7fa3080c88c0 2024-03-06T07:31:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.710+0000 7fa3057fa640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.710+0000 7fa3057fa640 1 -- 172.21.15.94:0/3503583551 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa2f80ac4c0 con 0x7fa3080c88c0 2024-03-06T07:31:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.710+0000 7fa307fff640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.711+0000 7fa307fff640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa2f406b4d0 0x7fa2f406d990 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa3080c1bb0 tx=0x7fa2fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.716 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.714+0000 7fa3057fa640 1 -- 172.21.15.94:0/3503583551 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3080c9f60 con 0x7fa3080c88c0 2024-03-06T07:31:39.883 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.882+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 --> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3081ec180 con 0x7fa2f406b4d0 2024-03-06T07:31:39.885 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.884+0000 7fa3057fa640 1 -- 172.21.15.94:0/3503583551 <== mgr.4105 v2:172.21.15.145:6832/2240241265 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13463 (secure 0 0 0) 0x7fa2fc001c90 con 0x7fa2f406b4d0 2024-03-06T07:31:39.885 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:39.885 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2024-03-06T07:31:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa2f406b4d0 msgr2=0x7fa2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa2f406b4d0 0x7fa2f406d990 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa3080c1bb0 tx=0x7fa2fc009040 comp rx=0 tx=0).stop 2024-03-06T07:31:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3080c88c0 msgr2=0x7fa3080c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.887 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3080c88c0 0x7fa3080c3c70 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8000c00 tx=0x7fa2f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 shutdown_connections 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7fa2f406b4d0 0x7fa2f406d990 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa308158b60 0x7fa3080bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa30814f360 0x7fa30814f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 --2- 172.21.15.94:0/3503583551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa3080c88c0 0x7fa3080c3c70 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 >> 172.21.15.94:0/3503583551 conn(0x7fa3080ac570 msgr2=0x7fa30814e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 shutdown_connections 2024-03-06T07:31:39.888 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.885+0000 7fa30efc7640 1 -- 172.21.15.94:0/3503583551 wait complete. 2024-03-06T07:31:39.897 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-03-06T07:31:38.803949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":283536,"kb_used_data":21600,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754691184,"statfs":{"total":773094113280,"available":772803772416,"internally_reserved":0,"allocated":22118400,"data_stored":11837112,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":268095024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002539"},"pg_stats":[{"pgid":"1.0","version":"216'119","reported_seq":553,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-03-06T07:31:19.335499+0000","last_change":"2024-03-06T07:31:19.335580+0000","last_active":"2024-03-06T07:31:19.335499+0000","last_peered":"2024-03-06T07:31:19.335499+0000","last_clean":"2024-03-06T07:31:19.335499+0000","last_became_active":"2024-03-06T07:18:04.417998+0000","last_became_peered":"2024-03-06T07:18:04.417998+0000","last_unstale":"2024-03-06T07:31:19.335499+0000","last_undegraded":"2024-03-06T07:31:19.335499+0000","last_fullsized":"2024-03-06T07:31:19.335499+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'119","last_scrub_stamp":"2024-03-06T07:31:19.335415+0000","last_deep_scrub":"216'119","last_deep_scrub_stamp":"2024-03-06T07:31:19.335415+0000","last_clean_scrub_stamp":"2024-03-06T07:31:19.335415+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:47.294050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65110,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575013,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33924,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337916,"statfs":{"total":96636764160,"available":96602025984,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33868,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337972,"statfs":{"total":96636764160,"available":96602083328,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38028,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333812,"statfs":{"total":96636764160,"available":96597823488,"internally_reserved":0,"allocated":2748416,"data_stored":1332047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575013,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38084,"kb_used_data":2748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333756,"statfs":{"total":96636764160,"available":96597766144,"internally_reserved":0,"allocated":2813952,"data_stored":1922415,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32555,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:39.898 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.996+0000 7f72d6a33640 1 Processor -- start 2024-03-06T07:31:39.998 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.996+0000 7f72d6a33640 1 -- start start 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72d6a33640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f72d006c900 0x7f72d00c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72d6a33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d00c3ef0 con 0x7f72d00c5990 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72d6a33640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d01556d0 con 0x7f72d00ab170 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72d6a33640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d01559b0 con 0x7f72d006c900 2024-03-06T07:31:39.999 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72cf7fe640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f72d006c900 0x7f72d00c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72cffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f72d00c5990 0x7f72d006c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36164/0 (socket says 172.21.15.94:36164) 2024-03-06T07:31:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72cffff640 1 -- 172.21.15.94:0/402699308 learned_addr learned my addr 172.21.15.94:0/402699308 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72cf7fe640 1 -- 172.21.15.94:0/402699308 >> v1:172.21.15.145:6789/0 conn(0x7f72d00ab170 legacy=0x7f72d00a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72cf7fe640 1 -- 172.21.15.94:0/402699308 >> v1:172.21.15.94:6789/0 conn(0x7f72d00c5990 legacy=0x7f72d006c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.000 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.997+0000 7f72cf7fe640 1 -- 172.21.15.94:0/402699308 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0155c90 con 0x7f72d006c900 2024-03-06T07:31:40.001 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.998+0000 7f72cf7fe640 1 --2- 172.21.15.94:0/402699308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d00c0ab0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f72c400ba70 tx=0x7f72c402e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606f91f7f7f38334 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.998+0000 7f72ceffd640 1 -- 172.21.15.94:0/402699308 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c403b070 con 0x7f72d006c900 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.998+0000 7f72ceffd640 1 -- 172.21.15.94:0/402699308 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c402ee30 con 0x7f72d006c900 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.998+0000 7f72ceffd640 1 -- 172.21.15.94:0/402699308 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4036640 con 0x7f72d006c900 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.999+0000 7f72d6a33640 1 -- 172.21.15.94:0/402699308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 msgr2=0x7f72d00c0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.999+0000 7f72d6a33640 1 --2- 172.21.15.94:0/402699308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d00c0ab0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f72c400ba70 tx=0x7f72c402e080 comp rx=0 tx=0).stop 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.999+0000 7f72d6a33640 1 -- 172.21.15.94:0/402699308 shutdown_connections 2024-03-06T07:31:40.002 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.999+0000 7f72d6a33640 1 --2- 172.21.15.94:0/402699308 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d00c0ab0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.999+0000 7f72d6a33640 1 -- 172.21.15.94:0/402699308 >> 172.21.15.94:0/402699308 conn(0x7f72d00c6710 msgr2=0x7f72d00c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.999+0000 7f72d6a33640 1 -- 172.21.15.94:0/402699308 shutdown_connections 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:39.999+0000 7f72d6a33640 1 -- 172.21.15.94:0/402699308 wait complete. 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.000+0000 7f72d6a33640 1 Processor -- start 2024-03-06T07:31:40.003 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.000+0000 7f72d6a33640 1 -- start start 2024-03-06T07:31:40.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d0145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.005 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72d00ab170 0x7f72d00bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72d00c5990 0x7f72d01eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d0156610 con 0x7f72d006c900 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d0156490 con 0x7f72d00ab170 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d00c3af0 con 0x7f72d00c5990 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d0145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d0145460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44728/0 (socket says 172.21.15.94:44728) 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 -- 172.21.15.94:0/3893395800 learned_addr learned my addr 172.21.15.94:0/3893395800 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:40.006 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cf7fe640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72d00ab170 0x7f72d00bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d4fd0640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72d00c5990 0x7f72d01eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 -- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72d00c5990 msgr2=0x7f72d01eaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72d00c5990 0x7f72d01eaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 -- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72d00ab170 msgr2=0x7f72d00bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72d00ab170 0x7f72d00bfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 -- 172.21.15.94:0/3893395800 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0147680 con 0x7f72d006c900 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d4fd0640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72d00c5990 0x7f72d01eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:40.007 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72cffff640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d0145460 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c3c0 tx=0x7f72c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0017070 con 0x7f72d006c900 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c0005ce0 con 0x7f72d006c900 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d01552c0 con 0x7f72d006c900 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c00072f0 con 0x7f72d006c900 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.001+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d01459a0 con 0x7f72d006c900 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.002+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d01471c0 con 0x7f72d006c900 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.003+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72c001a070 con 0x7f72d006c900 2024-03-06T07:31:40.008 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.003+0000 7f72ccff9640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72bc06b500 0x7f72bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.003+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f72c0089460 con 0x7f72d006c900 2024-03-06T07:31:40.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.003+0000 7f72cf7fe640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72bc06b500 0x7f72bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.004+0000 7f72cf7fe640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72bc06b500 0x7f72bc06d9c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f72c4038000 tx=0x7f72c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.009 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.008+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72d01471c0 con 0x7f72d006c900 2024-03-06T07:31:40.181 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.179+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f72d004f330 con 0x7f72d006c900 2024-03-06T07:31:40.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.182+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v65)=0 v65) v1 ==== 123+0+0 (secure 0 0 0) 0x7f72c0087e20 con 0x7f72d006c900 2024-03-06T07:31:40.184 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.183+0000 7f72ccff9640 1 -- 172.21.15.94:0/3893395800 <== mon.0 v2:172.21.15.94:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f72c00583e0 con 0x7f72d006c900 2024-03-06T07:31:40.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72bc06b500 msgr2=0x7f72bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72bc06b500 0x7f72bc06d9c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f72c4038000 tx=0x7f72c4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:40.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 msgr2=0x7f72d0145460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d0145460 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f72c000c3c0 tx=0x7f72c000c880 comp rx=0 tx=0).stop 2024-03-06T07:31:40.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 shutdown_connections 2024-03-06T07:31:40.187 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f72bc06b500 0x7f72bc06d9c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72d00c5990 0x7f72d01eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72d00ab170 0x7f72d00bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 --2- 172.21.15.94:0/3893395800 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72d006c900 0x7f72d0145460 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 >> 172.21.15.94:0/3893395800 conn(0x7f72d00c6710 msgr2=0x7f72d0149740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 shutdown_connections 2024-03-06T07:31:40.188 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.185+0000 7f72d6a33640 1 -- 172.21.15.94:0/3893395800 wait complete. 2024-03-06T07:31:40.197 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:40.197 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.295+0000 7f6e88ac2640 1 Processor -- start 2024-03-06T07:31:40.297 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e88ac2640 1 -- start start 2024-03-06T07:31:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e88ac2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e840c1aa0 0x7f6e841529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e88ac2640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e84155a40 0x7f6e84155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e88ac2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e84156410 con 0x7f6e8414f790 2024-03-06T07:31:40.299 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e88ac2640 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840c2680 con 0x7f6e84155a40 2024-03-06T07:31:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e88ac2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840c28d0 con 0x7f6e840c1aa0 2024-03-06T07:31:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e82d9d640 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6e84155a40 0x7f6e84155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e8259c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6e8414f790 0x7f6e840c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36178/0 (socket says 172.21.15.94:36178) 2024-03-06T07:31:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.296+0000 7f6e8259c640 1 -- 172.21.15.94:0/3652823478 learned_addr learned my addr 172.21.15.94:0/3652823478 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-06T07:31:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e7bfff640 1 --2- 172.21.15.94:0/3652823478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e840c1aa0 0x7f6e841529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e82d9d640 1 -- 172.21.15.94:0/3652823478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e840c1aa0 msgr2=0x7f6e841529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.300 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e82d9d640 1 --2- 172.21.15.94:0/3652823478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e840c1aa0 0x7f6e841529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e82d9d640 1 -- 172.21.15.94:0/3652823478 >> v1:172.21.15.94:6789/0 conn(0x7f6e8414f790 legacy=0x7f6e840c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e81d9b640 1 -- 172.21.15.94:0/3652823478 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846162349 0 0) 0x7f6e84156410 con 0x7f6e8414f790 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e82d9d640 1 -- 172.21.15.94:0/3652823478 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840c2bb0 con 0x7f6e84155a40 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e7bfff640 1 --2- 172.21.15.94:0/3652823478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e840c1aa0 0x7f6e841529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e82d9d640 1 --2- 172.21.15.94:0/3652823478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84155ed0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009d90 tx=0x7f6e7c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ae15d251807e54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e81d9b640 1 -- 172.21.15.94:0/3652823478 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c039070 con 0x7f6e84155a40 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e81d9b640 1 -- 172.21.15.94:0/3652823478 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f6e7c02cc50 con 0x7f6e84155a40 2024-03-06T07:31:40.301 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.297+0000 7f6e81d9b640 1 -- 172.21.15.94:0/3652823478 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c034440 con 0x7f6e84155a40 2024-03-06T07:31:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 -- 172.21.15.94:0/3652823478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 msgr2=0x7f6e84155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.305 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/3652823478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84155ed0 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009d90 tx=0x7f6e7c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 -- 172.21.15.94:0/3652823478 shutdown_connections 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/3652823478 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84155ed0 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/3652823478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e840c1aa0 0x7f6e841529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 -- 172.21.15.94:0/3652823478 >> 172.21.15.94:0/3652823478 conn(0x7f6e840c6710 msgr2=0x7f6e840c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 -- 172.21.15.94:0/3652823478 shutdown_connections 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.298+0000 7f6e88ac2640 1 -- 172.21.15.94:0/3652823478 wait complete. 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.299+0000 7f6e88ac2640 1 Processor -- start 2024-03-06T07:31:40.306 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.299+0000 7f6e88ac2640 1 -- start start 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e88ac2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e840c1aa0 0x7f6e8414af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e88ac2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8414f790 0x7f6e841506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e88ac2640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e88ac2640 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840c0d60 con 0x7f6e84155a40 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e88ac2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84155520 con 0x7f6e840c1aa0 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e88ac2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840c5e30 con 0x7f6e8414f790 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.307 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:44744/0 (socket says 172.21.15.94:44744) 2024-03-06T07:31:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 -- 172.21.15.94:0/2282425550 learned_addr learned my addr 172.21.15.94:0/2282425550 (peer_addr_for_me v2:172.21.15.94:0/0) 2024-03-06T07:31:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e7bfff640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8414f790 0x7f6e841506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e8259c640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e840c1aa0 0x7f6e8414af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 -- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8414f790 msgr2=0x7f6e841506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8414f790 0x7f6e841506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 -- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e840c1aa0 msgr2=0x7f6e8414af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.308 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e840c1aa0 0x7f6e8414af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e82d9d640 1 -- 172.21.15.94:0/2282425550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84073050 con 0x7f6e84155a40 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e8259c640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e840c1aa0 0x7f6e8414af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.300+0000 7f6e7bfff640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8414f790 0x7f6e841506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.301+0000 7f6e82d9d640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84071030 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009d90 tx=0x7f6e7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.301+0000 7f6e7affd640 1 -- 172.21.15.94:0/2282425550 <== mon.0 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c039050 con 0x7f6e84155a40 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.301+0000 7f6e7affd640 1 -- 172.21.15.94:0/2282425550 <== mon.0 v2:172.21.15.94:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f6e7c045070 con 0x7f6e84155a40 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.301+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e84073310 con 0x7f6e84155a40 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.301+0000 7f6e7affd640 1 -- 172.21.15.94:0/2282425550 <== mon.0 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c03e870 con 0x7f6e84155a40 2024-03-06T07:31:40.309 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.301+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e84151b10 con 0x7f6e84155a40 2024-03-06T07:31:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.302+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8404f330 con 0x7f6e84155a40 2024-03-06T07:31:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.302+0000 7f6e7affd640 1 -- 172.21.15.94:0/2282425550 <== mon.0 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e7c02cad0 con 0x7f6e84155a40 2024-03-06T07:31:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.302+0000 7f6e7affd640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6406b4d0 0x7f6e6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.303+0000 7f6e7affd640 1 -- 172.21.15.94:0/2282425550 <== mon.0 v2:172.21.15.94:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f6e7c0ace30 con 0x7f6e84155a40 2024-03-06T07:31:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.303+0000 7f6e8259c640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6406b4d0 0x7f6e6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.310 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.303+0000 7f6e8259c640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6406b4d0 0x7f6e6406d990 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000f60 tx=0x7f6e74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.311 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.307+0000 7f6e7affd640 1 -- 172.21.15.94:0/2282425550 <== mon.0 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e7c032e20 con 0x7f6e84155a40 2024-03-06T07:31:40.517 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.515+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e8406c770 con 0x7f6e84155a40 2024-03-06T07:31:40.518 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.516+0000 7f6e7affd640 1 -- 172.21.15.94:0/2282425550 <== mon.0 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+1332 (secure 0 0 0) 0x7f6e7c07bed0 con 0x7f6e84155a40 2024-03-06T07:31:40.518 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-06T07:31:40.518 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":215,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":39083,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/1228995575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1228995575},{"type":"v1","addr":"172.21.15.145:6836","nonce":1228995575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":215}],"filesystems":[]} 2024-03-06T07:31:40.518 INFO:teuthology.orchestra.run.smithi094.stderr:dumped fsmap epoch 215 2024-03-06T07:31:40.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6406b4d0 msgr2=0x7f6e6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6406b4d0 0x7f6e6406d990 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000f60 tx=0x7f6e74009040 comp rx=0 tx=0).stop 2024-03-06T07:31:40.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 msgr2=0x7f6e84071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84071030 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009d90 tx=0x7f6e7c004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 shutdown_connections 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:6832/2240241265,v1:172.21.15.145:6833/2240241265] conn(0x7f6e6406b4d0 0x7f6e6406d990 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6e84155a40 0x7f6e84071030 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e8414f790 0x7f6e841506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 --2- 172.21.15.94:0/2282425550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e840c1aa0 0x7f6e8414af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 >> 172.21.15.94:0/2282425550 conn(0x7f6e840c6710 msgr2=0x7f6e840c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 shutdown_connections 2024-03-06T07:31:40.521 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-06T07:31:40.518+0000 7f6e88ac2640 1 -- 172.21.15.94:0/2282425550 wait complete. 2024-03-06T07:31:40.530 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:31:40.531 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:31:40.531 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:31:40.531 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.540 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:31:40.540 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:31:40.540 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.591 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:31:40.592 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:31:40.592 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.603 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:31:40.603 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:31:40.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.655 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:31:40.655 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:31:40.656 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:31:40.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.685 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:31:40.686 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:31:40.686 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.757 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:31:40.757 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:31:40.757 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.824 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:31:40.824 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:31:40.824 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.893 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:31:40.893 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:31:40.893 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:40.925 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:31:40.925 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:31:40.925 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.005 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:31:41.005 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:31:41.005 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.092 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:31:41.093 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:31:41.093 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.135 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:31:41.135 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:31:41.135 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:31:41.135 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.160 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:31:41.160 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:31:41.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.189 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:31:41.189 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:31:41.189 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:31:41.189 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.212 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:31:41.212 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:31:41.212 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.241 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:31:41.242 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:31:41.242 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:41.294 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:31:41.294 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:31:41.295 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'Replacing daemon mds' | egrep -v '\(MDS_INSUFFICIENT_STANDBY\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(FS_DEGRADED\)' | head -n 1 2024-03-06T07:31:41.339 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:31:41.340 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:31:41.483 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:31:41.484 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:31:41.570 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:31:41.570 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:31:41.652 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:31:41.653 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:31:41.710 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:31:41.711 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:31:41.844 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:31:41.845 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:31:41.908 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:31:41.908 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:31:41.956 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:31:41.956 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:31:42.006 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:31:42.007 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/data/mon.a.tgz 2024-03-06T07:31:42.007 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-06T07:31:42.022 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.WcSra6lc7g 2024-03-06T07:31:42.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WcSra6lc7g 2024-03-06T07:31:42.206 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.WcSra6lc7g 2024-03-06T07:31:42.291 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.WcSra6lc7g is 545KB 2024-03-06T07:31:42.389 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.WcSra6lc7g 2024-03-06T07:31:42.404 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/data/mon.b.tgz 2024-03-06T07:31:42.404 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-06T07:31:42.421 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.hpLVPs13br 2024-03-06T07:31:42.421 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.hpLVPs13br 2024-03-06T07:31:42.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.hpLVPs13br 2024-03-06T07:31:42.689 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.hpLVPs13br is 559KB 2024-03-06T07:31:42.787 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.hpLVPs13br 2024-03-06T07:31:42.802 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/data/mon.c.tgz 2024-03-06T07:31:42.802 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-06T07:31:42.856 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.5TEpe6JnCv 2024-03-06T07:31:42.856 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.5TEpe6JnCv 2024-03-06T07:31:43.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.5TEpe6JnCv 2024-03-06T07:31:43.109 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.5TEpe6JnCv is 559KB 2024-03-06T07:31:43.178 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.5TEpe6JnCv 2024-03-06T07:31:43.192 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:31:43.192 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:31:43.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:31:43.277 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:31:43.314 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:31:43.314 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:31:43.316 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/remote/smithi094/crash 2024-03-06T07:31:43.317 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:31:43.346 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/remote/smithi145/crash 2024-03-06T07:31:43.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:31:43.380 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:31:43.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:43.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:43.476 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:43.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:43.529 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:31:43.530 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:31:43.546 DEBUG:teuthology.orchestra.run.smithi145:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:31:43.617 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38163.log 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38123.log 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37963.log 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38123.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38163.log.gz 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38083.log 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38123.log.gz 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.37963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42405.log 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38083.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37963.log.gz 2024-03-06T07:31:43.618 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.1.42405.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38083.log.gz 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42405.log.gz 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38003.log 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37923.log 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38003.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.37923.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38003.log.gz 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:31:43.619 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.7.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37923.log.gz 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37883.log 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.37883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38043.log 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37883.log.gz 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mgr.x.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38043.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:31:43.620 INFO:teuthology.orchestra.run.smithi145.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38043.log.gz 2024-03-06T07:31:43.625 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:31:43.625 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:31:43.625 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:31:43.749 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:31:43.761 INFO:teuthology.orchestra.run.smithi145.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:31:43.807 INFO:teuthology.orchestra.run.smithi145.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:31:43.974 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87513.log 2024-03-06T07:31:43.974 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49232.log 2024-03-06T07:31:43.974 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55831.log 2024-03-06T07:31:43.974 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62994.log 2024-03-06T07:31:43.974 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2%/var/log/ceph/ceph-client.admin.49232.log: -- replaced with /var/log/ceph/ceph-client.admin.87513.log.gz 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72465.log 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60123.log 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55831.log.gz 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49232.log.gz 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62994.log.gz 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73678.log 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54816.log 2024-03-06T07:31:43.975 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72465.log.gz 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60123.log.gz 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62886.log 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54816.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.73678.log.gz 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64106.log 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54816.log.gz 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46177.log 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69522.log 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3%/var/log/ceph/ceph-client.admin.46177.log: -- replaced with /var/log/ceph/ceph-client.admin.64106.log.gz 2024-03-06T07:31:43.976 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76258.log 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.62886.log.gz 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69522.log: gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.46177.log.gz --verbose 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.81987.log 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51209.log 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69522.log.gz 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76258.log.gz 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52993.log 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64311.log 2024-03-06T07:31:43.977 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81987.log.gz 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64557.log 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51209.log.gz 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64311.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52993.log.gz 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72507.log 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64557.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64311.log.gz 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47702.log 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77721.log 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64557.log.gz 2024-03-06T07:31:43.978 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72507.log.gz 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82113.log 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77721.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61919.log 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82113.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77721.log.gz 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67398.log 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76426.log 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82113.log.gz 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61919.log.gz 2024-03-06T07:31:43.979 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86736.log 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67398.log.gz/var/log/ceph/ceph-client.admin.76426.log: 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81158.log 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86736.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76426.log.gz 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68109.log 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81158.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86736.log.gz 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81158.log.gz 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87723.log 2024-03-06T07:31:43.980 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68109.log.gz 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63625.log 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80113.log 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63625.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87723.log.gz 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64232.log 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63625.log.gz 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58122.log 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80113.log.gz 2024-03-06T07:31:43.981 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83041.log 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89778.log 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64232.log.gz 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58122.log.gz 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64064.log 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83041.log.gz 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86256.log 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64064.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89778.log.gz 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55961.log 2024-03-06T07:31:43.982 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64064.log.gz 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76090.log 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7%/var/log/ceph/ceph-client.admin.55961.log: -- replaced with /var/log/ceph/ceph-client.admin.86256.log.gz 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66170.log 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84255.log 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55961.log.gz 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66170.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76090.log.gz 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51293.log 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66170.log.gz 2024-03-06T07:31:43.983 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76585.log 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64685.log 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84255.log.gz 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51293.log.gz 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58080.log 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76585.log.gz 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82999.log 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48410.log 2024-03-06T07:31:43.984 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64685.log.gz 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58080.log.gz 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87429.log 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82999.log.gz 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47492.log 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53999.log 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48410.log.gz 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87429.log.gz 2024-03-06T07:31:43.985 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82219.log 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47492.log.gz 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48116.log 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82219.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53999.log.gz 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67186.log 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82219.log.gz 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81704.log 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48116.log.gz 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88954.log 2024-03-06T07:31:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81704.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67186.log.gz 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58683.log 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81704.log.gz 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61300.log 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58683.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.88954.log.gz --verbose 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.61342.log 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61300.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58683.log.gz 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82957.log 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61300.log.gz 2024-03-06T07:31:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59871.log 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55404.log 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61342.log.gz 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59871.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82957.log.gz 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74830.log 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59871.log.gz 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89862.log 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74830.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55404.log.gz 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46919.log 2024-03-06T07:31:43.988 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74830.log.gz 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69354.log 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46919.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89862.log.gz 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69606.log 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69354.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46919.log.gz 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74638.log 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69354.log.gz 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89122.log 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69606.log.gz 2024-03-06T07:31:43.989 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72549.log 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89122.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74638.log.gz 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76132.log 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89122.log.gz 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46373.log 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72549.log.gz 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43284.log 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46373.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.76132.log.gz -- 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.58038.log 2024-03-06T07:31:43.990 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43284.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46373.log.gz 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59248.log 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43284.log.gz 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52084.log 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59248.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58038.log.gz 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44690.log 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59248.log.gz 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69690.log 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4%/var/log/ceph/ceph-client.admin.44690.log: -- replaced with /var/log/ceph/ceph-client.admin.52084.log.gz 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59741.log 2024-03-06T07:31:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68536.log 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69690.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44690.log.gz 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59741.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.69690.log.gz -5 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78248.log 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59741.log.gz 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75712.log 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78057.log 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75712.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78248.log.gz 2024-03-06T07:31:43.992 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76342.log 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.68536.log.gz 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75712.log.gz 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58911.log 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78057.log.gz 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59533.log 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58911.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76342.log.gz 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70368.log 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59533.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58911.log.gz 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71874.log 2024-03-06T07:31:43.993 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59533.log.gz 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66422.log 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83821.log 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70368.log.gz 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71874.log.gz 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48284.log 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66422.log.gz 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74915.log 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48284.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.83821.log.gz /var/log/ceph/ceph-client.admin.42370.log 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:43.994 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48284.log.gz 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53915.log 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42370.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74915.log.gz 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62175.log 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53915.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42370.log.gz 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77889.log 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62175.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53915.log.gz 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75356.log 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0%/var/log/ceph/ceph-client.admin.77889.log: -- replaced with /var/log/ceph/ceph-client.admin.62175.log.gz 2024-03-06T07:31:43.995 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65527.log 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63331.log 88.1% 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77889.log.gz 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75356.log.gz 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50397.log 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65527.log.gz 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77217.log 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50397.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63331.log.gz 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88455.log 2024-03-06T07:31:43.996 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52571.log 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.50397.log.gz 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88455.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77217.log.gz 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49274.log 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88455.log.gz 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61753.log 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49274.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52571.log.gz 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58595.log 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49274.log.gz 2024-03-06T07:31:43.997 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75441.log 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61753.log: /var/log/ceph/ceph-client.admin.58595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88660.log 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61753.log.gz 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75441.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58595.log.gz 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79448.log 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75441.log.gz 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79815.log 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64957.log 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88660.log.gz 2024-03-06T07:31:43.998 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79815.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79448.log.gz 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71168.log 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79815.log.gz 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55320.log 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72675.log 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64957.log.gz 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55320.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71168.log.gz 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55194.log 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55320.log.gz 2024-03-06T07:31:43.999 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47213.log 2024-03-06T07:31:44.000 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73763.log 2024-03-06T07:31:44.000 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72675.log.gz 2024-03-06T07:31:44.000 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47213.log: gzip -5 89.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.55194.log.gz 2024-03-06T07:31:44.000 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.88828.log 2024-03-06T07:31:44.000 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47213.log.gz 2024-03-06T07:31:44.000 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54732.log 2024-03-06T07:31:44.000 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88828.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73763.log.gz 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60291.log 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88828.log.gz 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50289.log 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60291.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54732.log.gz 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64897.log 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50289.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60291.log.gz 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83947.log 2024-03-06T07:31:44.001 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56297.log 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64897.log.gz 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83947.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.50289.log.gz 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75398.log 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83947.log.gz 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78636.log 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75398.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56297.log.gz 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51377.log 2024-03-06T07:31:44.002 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60754.log 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75398.log.gz 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51377.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78636.log.gz 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87933.log 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51377.log.gz 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46457.log 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87933.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60754.log.gz 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47297.log 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87933.log.gz 2024-03-06T07:31:44.003 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56381.log 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46457.log.gz 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51125.log 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89286.log 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47297.log.gz 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56381.log.gz 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60165.log 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51125.log.gz 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44932.log 2024-03-06T07:31:44.004 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82029.log 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89286.log.gz 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60165.log.gz 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78193.log 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44932.log.gz 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43636.log 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53572.log 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82029.log.gz 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78193.log.gz 2024-03-06T07:31:44.005 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45310.log 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4%/var/log/ceph/ceph-client.admin.53572.log: -- replaced with /var/log/ceph/ceph-client.admin.43636.log.gz 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74722.log 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45310.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53572.log.gz 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75880.log 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45310.log.gz 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57333.log 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70578.log 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57333.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75880.log.gz 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.74722.log.gz 2024-03-06T07:31:44.006 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55278.log 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57333.log.gz 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75838.log 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65251.log 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55278.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70578.log.gz 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50584.log 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55278.log.gz 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8%/var/log/ceph/ceph-client.admin.65251.log: -- replaced with /var/log/ceph/ceph-client.admin.75838.log.gz 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86046.log 2024-03-06T07:31:44.007 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65251.log.gz 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38143.log 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73384.log 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50584.log.gz 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38143.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86046.log.gz 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48937.log 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38143.log.gz 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82345.log 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54167.log 2024-03-06T07:31:44.008 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73384.log.gz 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82345.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48937.log.gz 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67623.log 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82345.log.gz 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46247.log 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67623.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.44675.log 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54167.log.gz 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67623.log.gz 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77847.log 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46247.log.gz 2024-03-06T07:31:44.009 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75483.log 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77847.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44675.log.gz 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49190.log 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9%/var/log/ceph/ceph-client.admin.75483.log: -- replaced with /var/log/ceph/ceph-client.admin.77847.log.gz 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45562.log 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55068.log 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45562.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.75483.log.gz 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62003.log 2024-03-06T07:31:44.010 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49190.log.gz 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45562.log.gz 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54377.log 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55068.log.gz 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73552.log 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54377.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62003.log.gz 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79322.log 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54377.log.gz 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89038.log 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73552.log.gz 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82555.log 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83551.log 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79322.log.gz 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82555.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89038.log.gz 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51744.log 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82555.log.gz 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83779.log 2024-03-06T07:31:44.012 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51744.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.83551.log.gz 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41884.log 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51744.log.gz 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63541.log 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83779.log.gz 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63078.log 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89547.log 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41884.log.gz 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63541.log.gz 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70452.log 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63078.log.gz 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65899.log 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70452.log: gzip -5 --verbose 87.3% -- /var/log/ceph/ceph-client.admin.50836.log -- replaced with /var/log/ceph/ceph-client.admin.89547.log.gz 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70452.log.gz 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57828.log 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65899.log.gz 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61468.log 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57828.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50836.log.gz 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80948.log 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57828.log.gz 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69227.log 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61468.log.gz 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47660.log 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69227.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.80948.log.gz --verbose 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.72255.log 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47660.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69227.log.gz 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47660.log.gz 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75796.log 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51958.log: gzip 87.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.72255.log.gz -- 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.38234.log 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51874.log 2024-03-06T07:31:44.016 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75796.log.gz 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89988.log 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81578.log 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38234.log.gz 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89988.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51874.log.gz 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68620.log 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89988.log.gz 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84651.log 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68620.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.81578.log.gz --verbose 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.66506.log 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68620.log.gz 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70494.log 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84651.log.gz 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43200.log 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70494.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66506.log.gz 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63036.log 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70494.log.gz 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62259.log 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43200.log.gz 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46793.log 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84171.log 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63036.log.gz 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62259.log.gz 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56129.log 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46793.log.gz 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84171.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84171.log.gz 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70938.log 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45394.log 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56129.log.gz 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69101.log 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65209.log 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70938.log.gz 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69101.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45394.log.gz 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86863.log 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69101.log.gz 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62469.log 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86863.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65209.log.gz 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55446.log 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62469.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86863.log.gz 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65167.log 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62469.log.gz 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80701.log 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65167.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55446.log.gz 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75128.log 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80701.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.65167.log.gz -5 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78293.log 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80701.log.gz 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56489.log 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75128.log.gz/var/log/ceph/ceph-client.admin.78293.log: 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76965.log 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51536.log 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56489.log.gz 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67270.log 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.78293.log.gz 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86948.log 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51536.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76965.log.gz 2024-03-06T07:31:44.022 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63120.log 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51536.log.gz 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86948.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67270.log.gz 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88119.log 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86948.log.gz 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74216.log 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46046.log 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88119.log.gz 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83737.log 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.63120.log.gz 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74216.log.gz 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.46046.log: -- /var/log/ceph/ceph-client.admin.48782.log 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.46046.log.gz 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69849.log 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48782.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83737.log.gz 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81536.log 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48782.log.gz 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65569.log 2024-03-06T07:31:44.024 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81536.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69849.log.gz 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70746.log 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9%/var/log/ceph/ceph-client.admin.65569.log: -- replaced with /var/log/ceph/ceph-client.admin.81536.log.gz 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87201.log 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77571.log 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70746.log.gz 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87201.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.65569.log.gzgzip 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53512.log 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62133.log 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87201.log.gz 2024-03-06T07:31:44.025 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77571.log: /var/log/ceph/ceph-client.admin.53512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47129.log 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62133.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.53512.log.gz -5 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.86991.log 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.77571.log.gz 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62133.log.gz 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81032.log 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47129.log.gz 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr:gzip/var/log/ceph/ceph-client.admin.86991.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59290.log 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83257.log 2024-03-06T07:31:44.026 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86991.log.gz 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59290.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81032.log.gz 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62952.log 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59290.log.gz 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80557.log 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83257.log.gz 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85692.log 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80557.log: gzip 89.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.62952.log.gz -- 2024-03-06T07:31:44.027 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.84819.log 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85692.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80557.log.gz 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85692.log.gz 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52294.log 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70896.log 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84819.log.gz 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52294.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50500.log 2024-03-06T07:31:44.028 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52294.log.gz 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52907.log 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68025.log 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70896.log.gz 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52907.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50500.log.gzgzip 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71561.log 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52907.log.gz 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44092.log 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71561.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68025.log.gzgzip 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69648.log 2024-03-06T07:31:44.029 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44092.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71561.log.gz 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54125.log 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44092.log.gz 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67983.log 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69648.log.gz 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57081.log 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54125.log: /var/log/ceph/ceph-client.admin.67983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45268.log 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54125.log.gz 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67983.log.gz 2024-03-06T07:31:44.030 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77679.log 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57081.log.gz 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71435.log 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69312.log 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45268.log.gz 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77679.log.gz 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52949.log 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71435.log.gz 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53037.log 2024-03-06T07:31:44.031 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85878.log 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69312.log.gz 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52949.log.gz 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87681.log 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53037.log.gz 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84483.log 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48979.log 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85878.log.gz 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87681.log.gz 2024-03-06T07:31:44.032 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88245.log 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84483.log.gz 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48824.log 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88245.log: gzip -5 87.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.48979.log.gz -- 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.49824.log 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88245.log.gz 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56045.log 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48824.log.gz 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88413.log 2024-03-06T07:31:44.033 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44693.log 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56045.log.gz 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87033.log 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44693.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88413.log.gz 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67875.log 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44693.log.gz 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87033.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2024-03-06T07:31:44.034 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87075.log 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67875.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87033.log.gz 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54606.log 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87075.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67875.log.gz 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72717.log 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87075.log.gz 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66801.log 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85818.log 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54606.log.gz 2024-03-06T07:31:44.035 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66801.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72717.log.gz 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84777.log 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66801.log.gz 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72133.log 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45604.log 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85818.log.gz 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72133.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84777.log.gz 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55917.log 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72133.log.gz 2024-03-06T07:31:44.036 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81945.log 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46835.log 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81945.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55917.log.gz 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47087.log 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45604.log.gz 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81945.log.gz 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81452.log 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72339.log 2024-03-06T07:31:44.037 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46835.log.gz 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47087.log.gz 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61552.log 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81452.log.gz 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44353.log 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61552.log: gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.72339.log.gz 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.63751.log 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44353.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61552.log.gz 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68933.log 2024-03-06T07:31:44.038 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44353.log.gz 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63667.log 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68933.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63751.log.gz 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83383.log 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63667.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68933.log.gz 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86466.log 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63667.log.gz 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51660.log 2024-03-06T07:31:44.039 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86466.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83383.log.gz -5 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79857.log 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6%/var/log/ceph/ceph-client.admin.51660.log: -- replaced with /var/log/ceph/ceph-client.admin.86466.log.gz 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54479.log 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51660.log.gz 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54858.log 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54479.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.79857.log.gz -5 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77487.log 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54858.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54479.log.gz 2024-03-06T07:31:44.040 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59332.log 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54858.log.gz 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67707.log 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77487.log.gz 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62619.log 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79238.log 88.7% 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59332.log.gz 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62619.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67707.log.gz 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62619.log.gz 2024-03-06T07:31:44.041 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58725.log 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72213.log 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60964.log 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.79238.log.gz 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72213.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58725.log.gz 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61006.log 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72213.log.gz 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57501.log 2024-03-06T07:31:44.042 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60964.log: /var/log/ceph/ceph-client.admin.61006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49318.log 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60964.log.gz 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61006.log.gz 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83467.log 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57501.log.gz 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77049.log 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63457.log 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49318.log.gz 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83467.log.gz 2024-03-06T07:31:44.043 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83863.log 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77049.log.gz 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76384.log 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71477.log 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63457.log.gz 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83863.log.gz 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51456.log 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76384.log.gz 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75628.log 2024-03-06T07:31:44.044 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86652.log 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71477.log.gz 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51456.log.gz 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79569.log 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75628.log.gz 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72067.log 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83215.log 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86652.log.gz 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79569.log.gz 2024-03-06T07:31:44.045 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55747.log 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53667.log 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55747.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.72067.log.gz 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83215.log.gz 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74872.log 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55747.log.gz 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53667.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31931.log 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53667.log.gz 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57291.log 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.tmp-client.admin.31931.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74872.log.gz -5 2024-03-06T07:31:44.046 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56171.log 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31931.log.gz 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89675.log 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57291.log.gz 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67354.log 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56171.log.gz 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43966.log 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89675.log: /var/log/ceph/ceph-client.admin.67354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49932.log 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89675.log.gz 2024-03-06T07:31:44.047 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43966.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67354.log.gz 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42454.log 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43966.log.gz 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr:gzip/var/log/ceph/ceph-client.admin.49932.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.80641.log 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89736.log 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49932.log.gz 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80641.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42454.log.gz 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55236.log 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80641.log.gz 2024-03-06T07:31:44.048 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44008.log 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62745.log 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89736.log.gz 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44008.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55236.log.gz 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87555.log 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44008.log.gz 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66633.log 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73198.log 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66633.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87555.log.gz 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60543.log 2024-03-06T07:31:44.049 INFO:teuthology.orchestra.run.smithi094.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.62745.log.gz 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66633.log.gz 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74046.log 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73198.log.gz 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68704.log 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74046.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.74046.log.gz 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56783.log 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60543.log.gz 2024-03-06T07:31:44.050 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72423.log 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.56783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66128.log 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.68704.log.gz 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72423.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56783.log.gz 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49890.log 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2%/var/log/ceph/ceph-client.admin.66128.log: -- replaced with /var/log/ceph/ceph-client.admin.72423.log.gz 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60880.log 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66128.log.gz 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85920.log 2024-03-06T07:31:44.051 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49890.log: /var/log/ceph/ceph-client.admin.60880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90156.log 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49890.log.gz 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85920.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60880.log.gz 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56867.log 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73072.log 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85920.log.gz 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.90156.log: /var/log/ceph/ceph-client.admin.56867.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.90156.log.gz 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71212.log 2024-03-06T07:31:44.052 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56867.log.gz 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45142.log 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73072.log: /var/log/ceph/ceph-client.admin.71212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52463.log 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73072.log.gz 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45142.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71212.log.gz 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83299.log 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71959.log 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61132.log 2024-03-06T07:31:44.053 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52463.log.gz 2024-03-06T07:31:44.054 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83299.log: /var/log/ceph/ceph-client.admin.71959.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83299.log.gz 2024-03-06T07:31:44.054 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53260.log 2024-03-06T07:31:44.054 INFO:teuthology.orchestra.run.smithi094.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:31:44.054 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.61132.log: -- /var/log/ceph/ceph-client.admin.50097.log 2024-03-06T07:31:44.054 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53260.log: 88.2% 91.4% -- replaced with /var/log/ceph/ceph-client.admin.61132.log.gz -- replaced with /var/log/ceph/ceph-client.admin.71959.log.gz 2024-03-06T07:31:44.054 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.054 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69769.log 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88059.log 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50097.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53260.log.gz 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51251.log 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50097.log.gz 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88059.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69769.log.gz 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71106.log 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51251.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88059.log.gz 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53614.log 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51251.log.gz 2024-03-06T07:31:44.055 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87387.log 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54564.log 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71106.log.gz 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87387.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53614.log.gz 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86088.log 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54564.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87387.log.gz 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82387.log 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54564.log.gz 2024-03-06T07:31:44.056 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62703.log 2024-03-06T07:31:44.057 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58290.log 2024-03-06T07:31:44.057 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86088.log.gz 2024-03-06T07:31:44.057 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62703.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82387.log.gz 2024-03-06T07:31:44.057 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60838.log 2024-03-06T07:31:44.057 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62703.log.gz 2024-03-06T07:31:44.057 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76006.log 2024-03-06T07:31:44.057 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42496.log 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58290.log.gz 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76006.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60838.log.gz 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66338.log 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76006.log.gz 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78099.log 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82849.log 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42496.log.gz 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78099.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66338.log.gz 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79649.log 2024-03-06T07:31:44.058 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78099.log.gz 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57996.log 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85776.log 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57996.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79649.log.gz 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47990.log 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.82849.log.gz 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57996.log.gz 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45732.log 2024-03-06T07:31:44.059 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85776.log.gz 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49530.log 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56213.log 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47990.log.gz 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49530.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45732.log.gz 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66590.log 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49530.log.gz 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59453.log 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66590.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.81242.log 2024-03-06T07:31:44.060 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56213.log.gz 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59453.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66590.log.gz 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68386.log 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59453.log.gz 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73240.log 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48494.log 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81242.log.gz 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89080.log 2024-03-06T07:31:44.061 INFO:teuthology.orchestra.run.smithi094.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.73240.log.gz 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.68386.log.gz 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79731.log 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87975.log 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48494.log.gz 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89080.log.gz 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45016.log 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79731.log.gz 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62511.log 2024-03-06T07:31:44.062 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87975.log: /var/log/ceph/ceph-client.admin.45016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89904.log 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87975.log.gz 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62511.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45016.log.gz 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72025.log 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62511.log.gz 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51786.log 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45226.log 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89904.log.gz 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72025.log.gz 2024-03-06T07:31:44.063 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65743.log 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51786.log.gz 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56255.log 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87471.log 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45226.log.gz 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65743.log.gz 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56573.log 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56255.log.gz 2024-03-06T07:31:44.064 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77133.log 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80785.log 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87471.log.gz 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56573.log.gz 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42054.log 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77133.log.gz 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52613.log 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70704.log 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80785.log.gz 2024-03-06T07:31:44.065 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42054.log.gz 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50016.log 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52613.log.gz 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71771.log 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56657.log 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70704.log.gz 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54984.log 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71771.log.gz 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56657.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.50016.log.gz 2024-03-06T07:31:44.066 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44356.log 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54984.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56657.log.gz 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88912.log 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54984.log.gz 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89164.log 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44356.log.gz 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81494.log 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75044.log 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88912.log.gz 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89164.log: /var/log/ceph/ceph-client.admin.81494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62661.log 2024-03-06T07:31:44.067 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89164.log.gz 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75044.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81494.log.gz 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58954.log 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75044.log.gz 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81116.log 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58954.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62661.log.gz 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63228.log 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58954.log.gz 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61384.log 2024-03-06T07:31:44.068 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63228.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81116.log.gz 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55525.log 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69480.log 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61384.log.gz 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55525.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.63228.log.gz 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71645.log 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55525.log.gz 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71645.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69480.log.gz 2024-03-06T07:31:44.069 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67442.log 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71645.log.gz 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67024.log 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67442.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz -5 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84441.log 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67024.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67442.log.gz 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60922.log 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67024.log.gz 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82765.log 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60922.log: gzip -5 88.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.84441.log.gz -- /var/log/ceph/ceph-client.admin.44357.log 2024-03-06T07:31:44.070 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50752.log 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60922.log.gz 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82765.log.gz 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71917.log 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44357.log.gz 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67833.log 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71917.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50752.log.gz 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85608.log 2024-03-06T07:31:44.071 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71917.log.gz 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78720.log 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67833.log.gz 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81284.log 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78720.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.85608.log.gz /var/log/ceph/ceph-client.admin.47003.log 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78720.log.gz 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86214.log 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47003.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81284.log.gz 2024-03-06T07:31:44.072 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67104.log 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47003.log.gz 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81074.log 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67104.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86214.log.gz 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83341.log 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81074.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67104.log.gz 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56911.log 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81074.log.gz 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62091.log 2024-03-06T07:31:44.073 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56911.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83341.log.gz 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46961.log 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62091.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56911.log.gz 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70057.log 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46961.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62091.log.gzgzip 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55789.log 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46961.log.gz 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53957.log 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70057.log.gz 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52739.log 2024-03-06T07:31:44.074 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53957.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.55789.log.gz -5 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.70980.log 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52739.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53957.log.gz 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50980.log 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52739.log.gz 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85230.log 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70980.log: /var/log/ceph/ceph-client.admin.50980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86821.log 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70980.log.gz 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50980.log.gz 2024-03-06T07:31:44.075 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43552.log 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85230.log.gz 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64641.log 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43552.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86821.log.gz 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83611.log 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43552.log.gz 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52781.log 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83611.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64641.log.gzgzip 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78467.log 2024-03-06T07:31:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52781.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.83611.log.gz -5 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.76879.log 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52781.log.gz 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44134.log 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50794.log 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44134.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76879.log.gz 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58553.log 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.78467.log.gz 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44134.log.gz 2024-03-06T07:31:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68807.log 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42604.log 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50794.log.gz 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68807.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58553.log.gz 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45058.log 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42604.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68807.log.gz 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56615.log 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42604.log.gz 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65377.log 2024-03-06T07:31:44.078 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45058.log.gz 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52865.log 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65377.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56615.log.gz 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72988.log 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55873.log 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52865.log.gz 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76216.log 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.65377.log.gz 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72988.log.gz 2024-03-06T07:31:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58164.log 2024-03-06T07:31:44.080 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88287.log 2024-03-06T07:31:44.080 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55873.log.gz 2024-03-06T07:31:44.080 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58164.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76216.log.gzgzip 2024-03-06T07:31:44.080 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.83653.log 2024-03-06T07:31:44.080 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63938.log 2024-03-06T07:31:44.080 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58164.log.gz 2024-03-06T07:31:44.080 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88287.log.gz 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42010.log 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83653.log.gz 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75002.log 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49022.log 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63938.log.gz 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62385.log 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42010.log.gz 2024-03-06T07:31:44.081 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49022.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75002.log.gz 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59657.log 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49022.log.gz 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88329.log 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59657.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.62385.log.gz 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44694.log 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59657.log.gz 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68236.log 2024-03-06T07:31:44.082 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44694.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88329.log.gz 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46541.log 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68236.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44694.log.gz 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74428.log 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46541.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68236.log.gz 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53344.log 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46541.log.gz 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77385.log 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74428.log.gz 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72381.log 2024-03-06T07:31:44.083 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77385.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53344.log.gz 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54942.log 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77385.log.gz 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72381.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72381.log.gz 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60796.log 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54942.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61877.log 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54942.log.gz 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74596.log 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61877.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60796.log.gz 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43510.log 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61877.log.gz 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60039.log 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74596.log.gz 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52421.log 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43510.log: /var/log/ceph/ceph-client.admin.60039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81410.log 2024-03-06T07:31:44.085 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43510.log.gz 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60039.log.gz 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49572.log 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58206.log 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81410.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52421.log.gz 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84084.log 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81410.log.gz 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49572.log.gz 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66717.log 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58206.log: /var/log/ceph/ceph-client.admin.84084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56699.log 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58206.log.gz 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66717.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84084.log.gz 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65941.log 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66717.log.gz 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86694.log 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83509.log 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56699.log.gz 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65941.log: /var/log/ceph/ceph-client.admin.86694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86424.log 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65941.log.gz 2024-03-06T07:31:44.087 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86694.log.gz 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82681.log 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83509.log: /var/log/ceph/ceph-client.admin.86424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71393.log 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83509.log.gz 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82681.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86424.log.gz 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44050.log 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82681.log.gz 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70238.log 2024-03-06T07:31:44.088 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54209.log 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71393.log.gz 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70238.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44050.log.gz 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49656.log 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46415.log 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70238.log.gz/var/log/ceph/ceph-client.admin.54209.log: 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49656.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54209.log.gz 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46667.log 2024-03-06T07:31:44.089 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81200.log 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46415.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49656.log.gz 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79056.log 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46415.log.gz 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71519.log 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81200.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46667.log.gz 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79056.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81200.log.gz 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84129.log 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77091.log 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71519.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79056.log.gz 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44974.log 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71519.log.gz 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59615.log 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77091.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.84129.log.gz 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44974.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77091.log.gz 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48578.log 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66843.log 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59615.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44974.log.gz 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64391.log 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59615.log.gz 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42328.log 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66843.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48578.log.gz 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64391.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.66843.log.gz /var/log/ceph/ceph-client.admin.75188.log 2024-03-06T07:31:44.092 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64148.log 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42328.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64391.log.gz 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73922.log 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42328.log.gz 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69059.log 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64148.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75188.log.gz 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67577.log 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64148.log.gz 2024-03-06T07:31:44.093 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53729.log 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69059.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73922.log.gz 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60609.log 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69059.log.gz 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47381.log 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53729.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67577.log.gz 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60609.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53729.log.gz 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58641.log 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65796.log 2024-03-06T07:31:44.094 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47381.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60609.log.gz 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44890.log 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47381.log.gz 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41926.log 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5%/var/log/ceph/ceph-client.admin.65796.log: -- replaced with /var/log/ceph/ceph-client.admin.58641.log.gz 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44890.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65796.log.gz 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61961.log 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72819.log 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44890.log.gz 2024-03-06T07:31:44.095 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41926.log: /var/log/ceph/ceph-client.admin.61961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76300.log 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41926.log.gz 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.73030.log -- replaced with /var/log/ceph/ceph-client.admin.61961.log.gz/var/log/ceph/ceph-client.admin.72819.log: 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76300.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72819.log.gz 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48032.log 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65485.log 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73030.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76300.log.gz 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88744.log 2024-03-06T07:31:44.096 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73030.log.gz 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66927.log 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6%/var/log/ceph/ceph-client.admin.65485.log: -- replaced with /var/log/ceph/ceph-client.admin.48032.log.gz 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88744.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65485.log.gz 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48452.log 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88534.log 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88744.log.gz 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66927.log: /var/log/ceph/ceph-client.admin.48452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86550.log 2024-03-06T07:31:44.097 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66927.log.gz 2024-03-06T07:31:44.098 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77343.log 2024-03-06T07:31:44.098 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88534.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48452.log.gz 2024-03-06T07:31:44.098 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86550.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88534.log.gz 2024-03-06T07:31:44.098 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52697.log 2024-03-06T07:31:44.098 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50626.log 2024-03-06T07:31:44.098 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86550.log.gz 2024-03-06T07:31:44.098 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77343.log: /var/log/ceph/ceph-client.admin.52697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81326.log 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77343.log.gz 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45352.log 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52697.log.gz 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50626.log: /var/log/ceph/ceph-client.admin.81326.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50626.log.gz 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46625.log 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80990.log 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81326.log.gz 2024-03-06T07:31:44.099 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45352.log: /var/log/ceph/ceph-client.admin.46625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79943.log 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45352.log.gz 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74260.log 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46625.log.gz 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80990.log: /var/log/ceph/ceph-client.admin.79943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63709.log 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80990.log.gz 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48536.log 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74260.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79943.log.gz 2024-03-06T07:31:44.100 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43242.log 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74260.log.gz 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77805.log 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48536.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63709.log.gz 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43242.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48536.log.gz 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54041.log 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87765.log 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43242.log.gz 2024-03-06T07:31:44.101 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77805.log: /var/log/ceph/ceph-client.admin.54041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45436.log 2024-03-06T07:31:44.102 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77805.log.gz 2024-03-06T07:31:44.102 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80239.log 2024-03-06T07:31:44.102 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87765.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54041.log.gz 2024-03-06T07:31:44.102 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.87765.log.gz 2024-03-06T07:31:44.102 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49404.log 2024-03-06T07:31:44.102 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77931.log 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80239.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45436.log.gz 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70015.log 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80239.log.gz 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85044.log 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77931.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49404.log.gz 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70015.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77931.log.gz 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57207.log 2024-03-06T07:31:44.103 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70015.log.gz 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75230.log 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85044.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.57207.log: -- /var/log/ceph/ceph-client.admin.79364.log 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85044.log.gz 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75230.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57207.log.gz 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43678.log 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75230.log.gz 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44686.log 2024-03-06T07:31:44.104 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43678.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79364.log.gz 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49488.log 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44686.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43678.log.gz 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71347.log 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44686.log.gz 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88618.log 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49488.log.gz 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87285.log 2024-03-06T07:31:44.105 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54648.log 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71347.log.gz 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88618.log: /var/log/ceph/ceph-client.admin.87285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47171.log 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88618.log.gz 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56955.log 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54648.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87285.log.gz 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66254.log 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54648.log.gz 2024-03-06T07:31:44.106 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52000.log 88.3% 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47171.log.gz 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66254.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56955.log.gz 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63896.log 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61835.log 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1%/var/log/ceph/ceph-client.admin.52000.log: -- replaced with /var/log/ceph/ceph-client.admin.66254.log.gz 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89455.log 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52000.log.gz 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71064.log 2024-03-06T07:31:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61835.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63896.log.gz 2024-03-06T07:31:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89455.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61835.log.gz 2024-03-06T07:31:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58511.log 2024-03-06T07:31:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61258.log 2024-03-06T07:31:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71064.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89455.log.gz 2024-03-06T07:31:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52126.log 2024-03-06T07:31:44.108 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71064.log.gz 2024-03-06T07:31:44.108 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71022.log 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58511.log.gz 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61258.log: /var/log/ceph/ceph-client.admin.52126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59997.log 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61258.log.gz 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84735.log 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52126.log.gz 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71022.log: /var/log/ceph/ceph-client.admin.59997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89370.log 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71022.log.gz 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68278.log 2024-03-06T07:31:44.109 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84735.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59997.log.gz 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89370.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84735.log.gz 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74130.log 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54900.log 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89370.log.gz 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68278.log: /var/log/ceph/ceph-client.admin.74130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78425.log 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58851.log 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74130.log.gz 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.68278.log.gz 2024-03-06T07:31:44.110 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54900.log.gz 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73114.log 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78425.log.gz 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80155.log 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58851.log: /var/log/ceph/ceph-client.admin.73114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80827.log 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58851.log.gz 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80155.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73114.log.gz 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48662.log 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80155.log.gz 2024-03-06T07:31:44.111 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75272.log 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80827.log.gz 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77259.log 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48662.log: /var/log/ceph/ceph-client.admin.75272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60399.log 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48662.log.gz 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77259.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75272.log.gz 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79134.log 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77259.log.gz 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66212.log 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65335.log 2024-03-06T07:31:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79134.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60399.log.gz 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46289.log 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79134.log.gz 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66212.log.gz 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86508.log 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65335.log: /var/log/ceph/ceph-client.admin.46289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74512.log 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65335.log.gz 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61174.log 2024-03-06T07:31:44.113 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46289.log.gz 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86508.log.gz 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80323.log 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61174.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74512.log.gz 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53428.log 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61174.log.gz 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80323.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80323.log.gz 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87891.log 2024-03-06T07:31:44.114 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53428.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53428.log.gz 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63854.log 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.87891.log.gz 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69143.log 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63854.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84693.log 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78141.log 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63854.log.gz 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69143.log.gz 2024-03-06T07:31:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84693.log: /var/log/ceph/ceph-client.admin.78141.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84693.log.gz 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78533.log 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78141.log.gz 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80449.log 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59913.log 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71603.log 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74554.log 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.78533.log.gz 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60459.log 2024-03-06T07:31:44.116 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% 91.1% -- replaced with /var/log/ceph/ceph-client.admin.80449.log.gz 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59913.log.gz 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84339.log 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71603.log.gz 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60459.log: /var/log/ceph/ceph-client.admin.74554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47534.log 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60459.log.gz 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74554.log.gz 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85524.log 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47534.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84339.log.gz 2024-03-06T07:31:44.117 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58385.log 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85524.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47534.log.gz 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57375.log 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85524.log.gz 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81863.log 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58385.log: /var/log/ceph/ceph-client.admin.57375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50878.log 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58385.log.gz 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70187.log 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57375.log.gz 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50878.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81863.log.gz 2024-03-06T07:31:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81368.log 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88371.log 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70187.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50878.log.gz 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71729.log 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70187.log.gz 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88371.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81368.log.gz 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69185.log 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88371.log.gz 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61426.log 2024-03-06T07:31:44.119 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83083.log 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69185.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71729.log.gz 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61048.log 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69185.log.gz 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61426.log.gz 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53088.log 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64729.log 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61048.log: /var/log/ceph/ceph-client.admin.53088.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61048.log.gz 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- 90.9% /var/log/ceph/ceph-client.admin.80365.log 2024-03-06T07:31:44.120 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83083.log.gz 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7%/var/log/ceph/ceph-client.admin.64729.log: -- replaced with /var/log/ceph/ceph-client.admin.53088.log.gz 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76835.log 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64729.log.gz 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66759.log 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80365.log: /var/log/ceph/ceph-client.admin.76835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56003.log 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80365.log.gz 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76835.log.gz 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75670.log 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43756.log 2024-03-06T07:31:44.121 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56003.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66759.log.gz 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68975.log 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56003.log.gz 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43756.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75670.log.gz 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67665.log 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43756.log.gz 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87597.log 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72759.log 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67665.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68975.log.gz 2024-03-06T07:31:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57912.log 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67665.log.gz 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72759.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87597.log.gz 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46583.log 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72759.log.gz 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69396.log 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52379.log 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57912.log.gz 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69396.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46583.log.gz 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72946.log 2024-03-06T07:31:44.123 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3%/var/log/ceph/ceph-client.admin.52379.log: -- replaced with /var/log/ceph/ceph-client.admin.69396.log.gz 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88996.log 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86382.log 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52379.log.gz 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77007.log 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72946.log.gz 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88996.log.gz 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66296.log 2024-03-06T07:31:44.124 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85272.log 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77007.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86382.log.gz 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5/var/log/ceph/ceph-client.admin.66296.log: --verbose -- /var/log/ceph/ceph-client.admin.53794.log 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77007.log.gz 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64999.log 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66296.log.gz 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85272.log.gz 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47948.log 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62301.log 2024-03-06T07:31:44.125 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64999.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53794.log.gz 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83149.log 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64999.log.gz 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62301.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47948.log.gz 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57039.log 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83149.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.69564.log 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62301.log.gz 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59122.log 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69564.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.83149.log.gz 2024-03-06T07:31:44.126 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57039.log.gz 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80071.log 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59122.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69564.log.gz 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48620.log 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59122.log.gz 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75754.log 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48620.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80071.log.gz 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56531.log 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75754.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48620.log.gz 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85398.log 2024-03-06T07:31:44.127 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75754.log.gz 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45478.log 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56531.log.gz 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64813.log 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85398.log.gz 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62427.log 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67535.log 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64813.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45478.log.gz 2024-03-06T07:31:44.128 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65041.log 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64813.log.gz 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67535.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62427.log.gz 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62343.log 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67535.log.gz 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57786.log 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58248.log 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62343.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65041.log.gz 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78015.log 2024-03-06T07:31:44.129 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62343.log.gz 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58248.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57786.log.gz 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75314.log 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58248.log.gz 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43009.log 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78015.log.gz 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78930.log 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43009.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75314.log.gz 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80743.log 2024-03-06T07:31:44.130 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75922.log 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78930.log.gz 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66548.log 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80743.log.gz 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50205.log 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.43009.log.gz 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66548.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75922.log.gz 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68067.log 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66548.log.gz 2024-03-06T07:31:44.131 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70788.log 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50205.log: /var/log/ceph/ceph-client.admin.68067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76505.log 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50205.log.gz 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68067.log.gz 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53873.log 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74088.log 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76505.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70788.log.gz 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63373.log 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76505.log.gz 2024-03-06T07:31:44.132 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3%/var/log/ceph/ceph-client.admin.74088.log: -- replaced with /var/log/ceph/ceph-client.admin.53873.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69438.log 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63373.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74088.log.gz 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82175.log 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63373.log.gz 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56741.log 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr:gzip/var/log/ceph/ceph-client.admin.82175.log: -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69438.log.gz 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.49362.log 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56741.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82175.log.gz 2024-03-06T07:31:44.133 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50710.log 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56741.log.gz 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37899.log 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50710.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49362.log.gz 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76667.log 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50710.log.gz 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.37899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64599.log 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73426.log 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76667.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.37899.log.gz 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49974.log 2024-03-06T07:31:44.134 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76667.log.gz 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose 88.4% -- /var/log/ceph/ceph-client.admin.89328.log -- replaced with /var/log/ceph/ceph-client.admin.64599.log.gz 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73426.log: /var/log/ceph/ceph-client.admin.49974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54293.log 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73426.log.gz 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89328.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49974.log.gz 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82261.log 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74004.log 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89328.log.gz 2024-03-06T07:31:44.135 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82261.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54293.log.gz 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47339.log 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41582.log 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74004.log: -- replaced with /var/log/ceph/ceph-client.admin.82261.log.gz 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43882.log 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74004.log.gz 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71305.log 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47339.log.gz 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41582.log.gz 2024-03-06T07:31:44.136 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63415.log 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42226.log 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71305.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43882.log.gz 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73324.log 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71305.log.gz 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42226.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63415.log.gz 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72633.log 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42226.log.gz 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48368.log 2024-03-06T07:31:44.137 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59699.log 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73324.log.gz 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72633.log: /var/log/ceph/ceph-client.admin.48368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43368.log 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72633.log.gz 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79280.log 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48368.log.gz 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59699.log.gz 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68151.log 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85188.log 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79280.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43368.log.gz 2024-03-06T07:31:44.138 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87327.log 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79280.log.gz 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75086.log 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.68151.log.gz 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85188.log.gz 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42098.log 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87327.log.gz 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75964.log 2024-03-06T07:31:44.139 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79014.log 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42098.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75086.log.gz 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.75964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48242.log 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42098.log.gz 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79014.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75964.log.gz 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67493.log 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90199.log 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.79014.log.gz 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67493.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:31:44.140 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48242.log.gz 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0%/var/log/ceph/ceph-client.admin.90199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41842.log 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67493.log.gz 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72297.log 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90199.log.gz 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64771.log 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72297.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41842.log.gz 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63980.log 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72297.log.gz 2024-03-06T07:31:44.141 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43326.log 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64771.log.gz 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82723.log 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63980.log.gz 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87117.log 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43326.log: /var/log/ceph/ceph-client.admin.82723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47255.log 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43326.log.gz 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73510.log 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82723.log.gz 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87117.log: /var/log/ceph/ceph-client.admin.47255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70410.log 2024-03-06T07:31:44.142 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% 88.1%/var/log/ceph/ceph-client.admin.73510.log: -- replaced with /var/log/ceph/ceph-client.admin.47255.log.gz 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82429.log 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87117.log.gz 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73510.log.gz 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:gzip/var/log/ceph/ceph-client.admin.70410.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.80515.log 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82429.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70410.log.gz 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58767.log 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82429.log.gz 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90239.log 2024-03-06T07:31:44.143 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58767.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80515.log.gz 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45184.log 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.90239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57707.log 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58767.log.gz 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60501.log 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90239.log.gz 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45184.log.gz 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73721.log 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57707.log: /var/log/ceph/ceph-client.admin.60501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82915.log 2024-03-06T07:31:44.144 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73721.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87159.log 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60501.log.gz 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57707.log.gz 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82915.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73721.log.gz 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85146.log 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82915.log.gz 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84213.log 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87159.log: /var/log/ceph/ceph-client.admin.85146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80906.log 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87159.log.gzgzip 2024-03-06T07:31:44.145 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.82303.log 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7%/var/log/ceph/ceph-client.admin.84213.log: -- replaced with /var/log/ceph/ceph-client.admin.85146.log.gz 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5/var/log/ceph/ceph-client.admin.80906.log: --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84213.log.gz 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.63583.log 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82303.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80906.log.gz 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44691.log 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82303.log.gz 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47618.log 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63583.log.gz 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71831.log 2024-03-06T07:31:44.146 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44691.log.gz 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90072.log 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71831.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47618.log.gz 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80029.log 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.90072.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71831.log.gz 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46499.log 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90072.log.gz 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78359.log 2024-03-06T07:31:44.147 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80029.log.gz 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49064.log 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65635.log 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46499.log.gz 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79406.log 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65635.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49064.log.gz 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr: -5 91.2% -- replaced with /var/log/ceph/ceph-client.admin.78359.log.gz 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.86130.log 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79406.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65635.log.gz 2024-03-06T07:31:44.148 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55362.log 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79406.log.gz 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74344.log 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55362.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86130.log.gz 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58996.log 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55362.log.gz 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.74344.log: /var/log/ceph/ceph-client.admin.64190.log 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58996.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74344.log.gz 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50139.log 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58996.log.gz 2024-03-06T07:31:44.149 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55663.log 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64190.log.gz 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76174.log 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57165.log 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55663.log: /var/log/ceph/ceph-client.admin.76174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84042.log 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55663.log.gz 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.50139.log.gz 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67228.log 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76174.log.gz 2024-03-06T07:31:44.150 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57165.log: /var/log/ceph/ceph-client.admin.84042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86298.log 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57165.log.gz 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67228.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84042.log.gz 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65443.log 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:gzip/var/log/ceph/ceph-client.admin.86298.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60249.log 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67228.log.gz 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50920.log 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86298.log.gz 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65443.log.gz 2024-03-06T07:31:44.151 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55110.log 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59080.log 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50920.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60249.log.gz 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87807.log/var/log/ceph/ceph-client.admin.55110.log: 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50920.log.gz 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90281.log 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55110.log.gz 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59080.log.gz 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59955.log 2024-03-06T07:31:44.152 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54335.log 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.90281.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87807.log.gz 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59955.log: gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90281.log.gz 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.59827.log 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54335.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59955.log.gz 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74172.log 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54335.log.gz 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80407.log 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59827.log: /var/log/ceph/ceph-client.admin.74172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85482.log 2024-03-06T07:31:44.153 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59827.log.gz 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74172.log.gz 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41759.log 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71687.log 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85482.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80407.log.gz 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79899.log 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85482.log.gz 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:gzip/var/log/ceph/ceph-client.admin.71687.log: -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41759.log.gz 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.82807.log 2024-03-06T07:31:44.154 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79899.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.71687.log.gz 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73842.log 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42538.log 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79899.log.gz 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82807.log: /var/log/ceph/ceph-client.admin.73842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53172.log 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82807.log.gz 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73842.log.gz 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81620.log 2024-03-06T07:31:44.155 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64022.log 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53172.log: /var/log/ceph/ceph-client.admin.81620.log: gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53172.log.gz 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.85566.log 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42538.log.gz 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81620.log.gz 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51830.log 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64022.log.gz 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61594.log 2024-03-06T07:31:44.156 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85566.log.gz 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62217.log 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79987.log 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61594.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51830.log.gz 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62217.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61594.log.gz 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.54690.log 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62217.log.gz 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53302.log 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79987.log.gz 2024-03-06T07:31:44.157 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83905.log 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54690.log.gz 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49782.log 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57249.log 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83905.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53302.log.gz 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49782.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83905.log.gz 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.78846.log 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74386.log 2024-03-06T07:31:44.158 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1%/var/log/ceph/ceph-client.admin.78846.log: -- replaced with /var/log/ceph/ceph-client.admin.57249.log.gz 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69973.log 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78846.log.gz 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56339.log 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69973.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74386.log.gz 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70322.log 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69973.log.gz 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56339.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56339.log.gz 2024-03-06T07:31:44.159 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46751.log 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79773.log 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70322.log.gz 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57585.log 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73636.log 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46751.log.gz 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.79773.log: /var/log/ceph/ceph-client.admin.57585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82639.log 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73594.log 2024-03-06T07:31:44.160 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57585.log.gz 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79773.log.gz 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82639.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73636.log.gz 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72904.log 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82639.log.gz 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43468.log 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4%/var/log/ceph/ceph-client.admin.72904.log: -- replaced with /var/log/ceph/ceph-client.admin.73594.log.gz 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67917.log 2024-03-06T07:31:44.161 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43468.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72904.log.gz 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57417.log 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43468.log.gz 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57123.log 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51023.log 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr: 91.0%/var/log/ceph/ceph-client.admin.57123.log: -- replaced with /var/log/ceph/ceph-client.admin.67917.log.gz 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57417.log.gz 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44358.log 2024-03-06T07:31:44.162 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57123.log.gz 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85962.log 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51023.log: /var/log/ceph/ceph-client.admin.44358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89497.log 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61090.log 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44358.log.gz 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51023.log.gz 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89497.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85962.log.gz 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81662.log 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89497.log.gz 2024-03-06T07:31:44.163 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77175.log 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61090.log.gz 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68662.log 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81662.log.gz 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51618.log 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68662.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85734.log 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77175.log.gz 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68662.log.gz 2024-03-06T07:31:44.164 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88017.log 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr:gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51618.log.gz 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr: -5/var/log/ceph/ceph-client.admin.85734.log: --verbose -- /var/log/ceph/ceph-client.admin.78678.log 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88017.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85734.log.gz 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51167.log 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88017.log.gz 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82071.log 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78678.log.gz 2024-03-06T07:31:44.165 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70620.log 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82071.log: gzip 89.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.70536.log 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51167.log.gz 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70620.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82071.log.gz 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58809.log 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70620.log.gz 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr: -5/var/log/ceph/ceph-client.admin.70536.log: --verbose -- /var/log/ceph/ceph-client.admin.86340.log 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58809.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70536.log.gz 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73282.log 2024-03-06T07:31:44.166 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58809.log.gz 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78762.log 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86340.log.gz 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73156.log 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73282.log.gz 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60333.log 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78762.log.gz 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77529.log 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60333.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73156.log.gz 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84609.log 2024-03-06T07:31:44.167 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65293.log 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.60333.log.gz 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77529.log.gz 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49148.log 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84609.log.gz 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54419.log 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49148.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65293.log.gz 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48200.log 2024-03-06T07:31:44.168 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49148.log.gz 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66086.log 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54419.log.gz 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61673.log 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48200.log: /var/log/ceph/ceph-client.admin.66086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59038.log 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61673.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66086.log.gz 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54083.log 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48200.log.gz 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86172.log 2024-03-06T07:31:44.169 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61673.log.gz 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72591.log 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59038.log.gz 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54083.log.gz 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69269.log 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86172.log: /var/log/ceph/ceph-client.admin.72591.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44176.log 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86172.log.gz 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72591.log.gz 2024-03-06T07:31:44.170 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69269.log.gz 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86004.log 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78972.log 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86004.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44176.log.gz 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86004.log.gz 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90030.log 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78972.log: /var/log/ceph/ceph-client.admin.43924.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78972.log.gz 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2024-03-06T07:31:44.171 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52505.log 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.90030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90114.log 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52505.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90030.log.gzgzip 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80281.log 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.90114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41678.log 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.52505.log.gz/var/log/ceph/ceph-client.admin.80281.log: 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65677.log 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90114.log.gz 88.4% 2024-03-06T07:31:44.172 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41678.log: -- replaced with /var/log/ceph/ceph-client.admin.80281.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54774.log 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41678.log.gz 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49614.log 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52336.log 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65677.log.gz 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54774.log.gz/var/log/ceph/ceph-client.admin.49614.log: 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65125.log 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49614.log.gz 2024-03-06T07:31:44.173 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57459.log 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52336.log: gzip/var/log/ceph/ceph-client.admin.65125.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.84914.log 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52336.log.gz 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57459.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65125.log.gz 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.57647.log 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84914.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57459.log.gz 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82471.log 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84914.log.gz 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57647.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:31:44.174 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57647.log.gz 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89820.log 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.2.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82471.log.gz 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65983.log 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85314.log 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89820.log.gz/var/log/ceph/ceph-client.admin.65983.log: 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43594.log 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48888.log 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85314.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.65983.log.gz 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43594.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85314.log.gz 2024-03-06T07:31:44.175 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68494.log 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43594.log.gz 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89946.log 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63499.log 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48888.log.gz 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68494.log.gz/var/log/ceph/ceph-client.admin.89946.log: 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50355.log 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77445.log 2024-03-06T07:31:44.176 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89946.log.gz 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50355.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63499.log.gz 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74470.log 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50355.log.gz 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64473.log 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77445.log: /var/log/ceph/ceph-client.admin.74470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67749.log 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64473.log: 88.2% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77445.log.gz 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74470.log.gz 2024-03-06T07:31:44.177 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64473.log.gz 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78804.log 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43410.log 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76709.log 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78804.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67749.log.gz 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43410.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78804.log.gz 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58469.log 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43410.log.gz 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70280.log 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58469.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76709.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79178.log 2024-03-06T07:31:44.178 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58469.log.gz 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70280.log: /var/log/ceph/ceph-client.admin.79178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86610.log 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70280.log.gz 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mds.c.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.79178.log.gz 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55026.log 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60207.log 2024-03-06T07:31:44.179 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86610.log.gz 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55026.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55026.log.gz 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67791.log 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79490.log 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60207.log: /var/log/ceph/ceph-client.admin.67791.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60207.log.gz 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67791.log.gz 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50542.log 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47822.log 2024-03-06T07:31:44.180 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50542.log: /var/log/ceph/ceph-client.admin.79490.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50542.log.gz 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79490.log.gz 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75549.log 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87639.log 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47822.log: /var/log/ceph/ceph-client.admin.75549.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47822.log.gz 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75549.log.gz 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52252.log 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63811.log 2024-03-06T07:31:44.181 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52252.log: /var/log/ceph/ceph-client.admin.87639.log: 88.9% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87639.log.gz 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52252.log.gz 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86779.log 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80197.log 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63811.log: /var/log/ceph/ceph-client.admin.86779.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63811.log.gz 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86779.log.gz 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54522.log 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69931.log 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80197.log: /var/log/ceph/ceph-client.admin.54522.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80197.log.gz 2024-03-06T07:31:44.182 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54522.log.gz 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49106.log 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44355.log 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.69931.log: /var/log/ceph/ceph-client.admin.49106.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69931.log.gz 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49106.log.gz 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48326.log 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73468.log 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48326.log: /var/log/ceph/ceph-client.admin.44355.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48326.log.gz 2024-03-06T07:31:44.183 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44355.log.gz 2024-03-06T07:31:44.184 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68891.log 2024-03-06T07:31:44.184 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89412.log 2024-03-06T07:31:44.184 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.73468.log: /var/log/ceph/ceph-client.admin.68891.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.73468.log.gz 2024-03-06T07:31:44.184 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68891.log.gz 2024-03-06T07:31:44.184 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84525.log 2024-03-06T07:31:44.184 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49698.log 2024-03-06T07:31:44.184 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84525.log: /var/log/ceph/ceph-client.admin.89412.log: 88.0% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.84525.log.gz 2024-03-06T07:31:44.185 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89412.log.gz 2024-03-06T07:31:44.185 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66885.log 2024-03-06T07:31:44.185 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76793.log 2024-03-06T07:31:44.185 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49698.log: /var/log/ceph/ceph-client.admin.66885.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49698.log.gz 2024-03-06T07:31:44.185 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66885.log.gz 2024-03-06T07:31:44.185 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83425.log 2024-03-06T07:31:44.185 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76793.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76793.log.gz 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82513.log 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87849.log 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83425.log.gz 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74680.log 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87849.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82513.log.gz 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87849.log.gz 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43840.log 2024-03-06T07:31:44.186 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77973.log 2024-03-06T07:31:44.187 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74680.log.gz 2024-03-06T07:31:44.187 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80599.log 2024-03-06T07:31:44.187 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77973.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43840.log.gz 2024-03-06T07:31:44.187 INFO:teuthology.orchestra.run.smithi094.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.77973.log.gz 2024-03-06T07:31:44.187 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74960.log 2024-03-06T07:31:44.187 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.80599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86905.log 2024-03-06T07:31:44.187 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80599.log.gz 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66464.log 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.86905.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74960.log.gz 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86905.log.gz 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55705.log 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77637.log 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66464.log: /var/log/ceph/ceph-client.admin.55705.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66464.log.gz 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88576.log 2024-03-06T07:31:44.188 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55705.log.gz 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44349.log 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88576.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.88576.log.gz 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77637.log.gz 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62047.log 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.44349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67312.log 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz/var/log/ceph/ceph-client.admin.62047.log: 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55152.log 2024-03-06T07:31:44.189 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.67312.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62047.log.gz 2024-03-06T07:31:44.190 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67312.log.gz 2024-03-06T07:31:44.190 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42182.log 2024-03-06T07:31:44.190 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.55152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64515.log 2024-03-06T07:31:44.190 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55152.log.gz 2024-03-06T07:31:44.190 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78888.log 2024-03-06T07:31:44.190 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42182.log.gz 2024-03-06T07:31:44.190 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71263.log 2024-03-06T07:31:44.191 INFO:teuthology.orchestra.run.smithi094.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64515.log.gz 2024-03-06T07:31:44.191 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.78888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61510.log 2024-03-06T07:31:44.191 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.71263.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78888.log.gz 2024-03-06T07:31:44.191 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71263.log.gz 2024-03-06T07:31:44.191 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88786.log 2024-03-06T07:31:44.191 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52823.log 2024-03-06T07:31:44.191 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.61510.log: /var/log/ceph/ceph-client.admin.88786.log: 88.6% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61510.log.gz 2024-03-06T07:31:44.192 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88786.log.gz 2024-03-06T07:31:44.192 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70830.log 2024-03-06T07:31:44.192 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47864.log 2024-03-06T07:31:44.192 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52823.log: /var/log/ceph/ceph-client.admin.70830.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52823.log.gz 2024-03-06T07:31:44.192 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70662.log 2024-03-06T07:31:44.192 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.70830.log.gz 2024-03-06T07:31:44.192 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44354.log 2024-03-06T07:31:44.193 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5%/var/log/ceph/ceph-client.admin.70662.log: -- replaced with /var/log/ceph/ceph-client.admin.47864.log.gz 2024-03-06T07:31:44.193 INFO:teuthology.orchestra.run.smithi094.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74788.log 2024-03-06T07:31:44.193 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70662.log.gz/var/log/ceph/ceph-client.admin.44354.log: 2024-03-06T07:31:44.193 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44354.log.gz 2024-03-06T07:31:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51916.log 2024-03-06T07:31:44.193 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88203.log 2024-03-06T07:31:44.193 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74788.log.gz 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51083.log 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88203.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51916.log.gz 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88203.log.gz 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57543.log 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56087.log 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51083.log.gz 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44687.log 2024-03-06T07:31:44.194 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56087.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57543.log.gz 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56087.log.gz 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58427.log 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44687.log: -- /var/log/ceph/ceph-client.admin.50247.log 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44687.log.gz 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48158.log 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.50247.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58427.log.gz 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50247.log.gz 2024-03-06T07:31:44.195 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84567.log 2024-03-06T07:31:44.196 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62553.log 2024-03-06T07:31:44.196 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.48158.log: /var/log/ceph/ceph-client.admin.84567.log: 88.4% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48158.log.gz 2024-03-06T07:31:44.196 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84567.log.gz 2024-03-06T07:31:44.196 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42286.log 2024-03-06T07:31:44.196 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.62553.log: -- /var/log/ceph/ceph-client.admin.85086.log 2024-03-06T07:31:44.196 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42286.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.62553.log.gz 2024-03-06T07:31:44.196 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42286.log.gz 2024-03-06T07:31:44.197 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60081.log 2024-03-06T07:31:44.197 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51702.log 2024-03-06T07:31:44.197 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.85086.log.gz 2024-03-06T07:31:44.197 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76751.log 2024-03-06T07:31:44.197 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51702.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60081.log.gz 2024-03-06T07:31:44.197 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53386.log 2024-03-06T07:31:44.197 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51702.log.gz 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68849.log 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53386.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76751.log.gz 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53386.log.gz 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59206.log 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87243.log 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68849.log: /var/log/ceph/ceph-client.admin.59206.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68849.log.gz 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59206.log.gz 2024-03-06T07:31:44.198 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76923.log 2024-03-06T07:31:44.199 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46877.log 2024-03-06T07:31:44.199 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.87243.log: /var/log/ceph/ceph-client.admin.76923.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87243.log.gz 2024-03-06T07:31:44.199 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52655.log 2024-03-06T07:31:44.199 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76923.log.gz 2024-03-06T07:31:44.199 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83695.log 2024-03-06T07:31:44.199 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52655.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46877.log.gz 2024-03-06T07:31:44.199 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52655.log.gz 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63186.log 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70099.log 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83695.log.gz 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.63186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77301.log 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.70099.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63186.log.gz 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70099.log.gz 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 2024-03-06T07:31:44.200 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45100.log 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77301.log: /var/log/ceph/ceph-client.admin.44692.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77301.log.gz 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76048.log 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77763.log 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45100.log.gz 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.76048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57954.log 90.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:44.201 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.77763.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76048.log.gz 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50668.log 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77763.log.gz 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84297.log 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57954.log: /var/log/ceph/ceph-client.admin.50668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51335.log 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84297.log: 88.5% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50668.log.gz 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57954.log.gz 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62844.log 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84297.log.gz 2024-03-06T07:31:44.202 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.51335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53130.log 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47045.log 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51335.log.gz 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.62844.log: /var/log/ceph/ceph-client.admin.53130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47780.log 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62844.log.gz 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47045.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53130.log.gz 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66675.log 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47045.log.gz 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88161.log 2024-03-06T07:31:44.203 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47780.log: /var/log/ceph/ceph-client.admin.66675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47576.log 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88161.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47780.log.gz 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66675.log.gz 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr:gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88161.log.gz 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69017.log 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81783.log 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47576.log.gz/var/log/ceph/ceph-client.admin.69017.log: 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52168.log 2024-03-06T07:31:44.204 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69017.log.gz 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.81783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59164.log 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81783.log.gz 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53218.log 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52168.log: /var/log/ceph/ceph-client.admin.59164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47906.log 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53218.log: 87.7% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59164.log.gz 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52168.log.gz 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53218.log.gz 2024-03-06T07:31:44.205 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68344.log 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.47906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43798.log 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47906.log.gz 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.68344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85440.log 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68344.log.gz 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89589.log 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.43798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45520.log 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85440.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43798.log.gz 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.89589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60651.log 2024-03-06T07:31:44.206 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85440.log.gz 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89589.log.gz 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.45520.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45520.log.gz 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.66380.log 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.60651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84399.log 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.66380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53470.log 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60651.log.gz 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66380.log.gz 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68452.log 2024-03-06T07:31:44.207 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84399.log.gz 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88702.log 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.53470.log: /var/log/ceph/ceph-client.admin.68452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64855.log 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88702.log: 87.5% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53470.log.gz 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68452.log.gz 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88702.log.gz 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59374.log 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82597.log 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.64855.log: gzip -5/var/log/ceph/ceph-client.admin.59374.log: --verbose -- /var/log/ceph/ceph-client.admin.74302.log 2024-03-06T07:31:44.208 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59374.log.gz 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.82597.log: -- replaced with /var/log/ceph/ceph-client.admin.64855.log.gz 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 89.2% --verbose -- /var/log/ceph/ceph-client.admin.70143.log 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82597.log.gz 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.74302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85650.log 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7%/var/log/ceph/ceph-client.admin.70143.log: -- replaced with /var/log/ceph/ceph-client.admin.74302.log.gz 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42140.log 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70143.log.gz 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84861.log 2024-03-06T07:31:44.209 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65083.log 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42140.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85650.log.gz 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83989.log 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr: 88.9% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84861.log.gz 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42140.log.gz/var/log/ceph/ceph-client.admin.65083.log: 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65083.log.gz 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-mds.a.log 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.83989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88870.log 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83989.log.gz 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85356.log 2024-03-06T07:31:44.210 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.88870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65838.log 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88870.log.gz 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.85356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54251.log 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85356.log.gz 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.65838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56825.log 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.54251.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65838.log.gz 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54251.log.gz 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89633.log 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2024-03-06T07:31:44.211 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56825.log: /var/log/ceph/ceph-client.admin.89633.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56825.log.gz 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89633.log.gz 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56997.log 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.42412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61216.log 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72862.log 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7%/var/log/ceph/ceph-client.admin.61216.log: -- replaced with /var/log/ceph/ceph-client.admin.56997.log.gz 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61216.log.gz 2024-03-06T07:31:44.212 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58332.log 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.72862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57870.log 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72862.log.gz 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.58332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59783.log 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.57870.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58332.log.gz 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57870.log.gz 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84956.log 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.59783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46709.log 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59783.log.gz 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.84956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49446.log 2024-03-06T07:31:44.213 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6%/var/log/ceph/ceph-client.admin.46709.log: -- replaced with /var/log/ceph/ceph-client.admin.84956.log.gz 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46709.log.gz 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49740.log 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46331.log 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49446.log.gz 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.49740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52210.log 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.46331.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49740.log.gz 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46331.log.gz 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56423.log 2024-03-06T07:31:44.214 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.52210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85001.log 2024-03-06T07:31:44.215 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52210.log.gz 2024-03-06T07:31:44.215 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.56423.log: /var/log/ceph/ceph-client.admin.85001.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.56423.log.gz 2024-03-06T07:31:44.215 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85001.log.gz 2024-03-06T07:31:44.215 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:31:44.216 INFO:teuthology.orchestra.run.smithi145.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:31:44.216 INFO:teuthology.orchestra.run.smithi145.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:31:44.549 INFO:teuthology.orchestra.run.smithi094.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:31:44.967 INFO:teuthology.orchestra.run.smithi145.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:31:45.135 INFO:teuthology.orchestra.run.smithi145.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:31:45.145 INFO:teuthology.orchestra.run.smithi094.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:31:45.200 INFO:teuthology.orchestra.run.smithi145.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:31:45.226 INFO:teuthology.orchestra.run.smithi145.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:31:45.228 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-06T07:31:45.228 INFO:teuthology.orchestra.run.smithi145.stderr:real 0m1.638s 2024-03-06T07:31:45.228 INFO:teuthology.orchestra.run.smithi145.stderr:user 0m7.197s 2024-03-06T07:31:45.228 INFO:teuthology.orchestra.run.smithi145.stderr:sys 0m0.266s 2024-03-06T07:31:45.251 INFO:teuthology.orchestra.run.smithi094.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:31:45.392 INFO:teuthology.orchestra.run.smithi094.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:31:45.482 INFO:teuthology.orchestra.run.smithi094.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:31:45.485 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-06T07:31:45.485 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m1.927s 2024-03-06T07:31:45.485 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m7.489s 2024-03-06T07:31:45.485 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m0.807s 2024-03-06T07:31:45.485 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:31:45.486 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/remote/smithi094/log 2024-03-06T07:31:45.487 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:31:47.449 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/remote/smithi145/log 2024-03-06T07:31:47.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:31:48.495 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:31:48.504 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:31:48.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:31:48.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:31:48.587 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:31:48.596 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:31:48.597 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:31:48.599 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:31:48.613 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:31:48.613 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:31:48.617 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:31:48.617 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-06T07:31:48.617 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +1783us[+1783us] +/- 97ms 2024-03-06T07:31:48.617 INFO:teuthology.orchestra.run.smithi145.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -1093us[-1099us] +/- 52ms 2024-03-06T07:31:48.618 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +124us[ +118us] +/- 88ms 2024-03-06T07:31:48.618 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:31:48.619 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:31:48.619 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-06T07:31:48.619 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +1791us[+1791us] +/- 97ms 2024-03-06T07:31:48.619 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 7 377 34 -1074us[-1081us] +/- 52ms 2024-03-06T07:31:48.619 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +146us[ +140us] +/- 88ms 2024-03-06T07:31:48.619 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:31:48.619 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:31:48.628 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:31:48.628 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:31:48.693 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:31:48.790 DEBUG:teuthology.orchestra.run.smithi145:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:31:48.857 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:31:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:48.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:31:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:48.919 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:31:48.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:31:48.983 INFO:teuthology.task.internal:Duration was 1267.168054 seconds 2024-03-06T07:31:48.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:31:48.994 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:31:48.994 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:31:48.997 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:31:49.038 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:31:49.040 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:31:49.346 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:31:49.346 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-03-06T07:31:49.347 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:31:49.366 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-03-06T07:31:49.366 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:31:49.395 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:31:49.396 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:31:49.409 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:31:49.465 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:31:49.465 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:31:49.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:31:49.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:31:49.620 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:31:49.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:31:49.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:31:49.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:31:49.683 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:31:49.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:31:49.712 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-03-06T07:31:49.739 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-03-06T07:31:49.755 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:31:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:49.782 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:31:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:49.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:31:49.820 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:31:49.821 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/remote/smithi094 2024-03-06T07:31:49.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:31:49.860 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042/remote/smithi145 2024-03-06T07:31:49.861 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:31:49.896 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:31:49.896 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:31:49.898 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:31:49.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:31:49.956 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:31:49.956 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:31:49.968 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:31:49.968 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:31:49.971 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:31:49.983 INFO:teuthology.orchestra.run.smithi094.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:31 /home/ubuntu/cephtest 2024-03-06T07:31:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: 266039 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:31 /home/ubuntu/cephtest 2024-03-06T07:31:50.000 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:31:50.010 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:31:50.039 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:31:50.065 DEBUG:teuthology.nuke:shortname: smithi094 2024-03-06T07:31:50.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:31:50.091 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:53.122975', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUptUXOT7IP+rlzPS9z+dJBf1CjRETBgacSwiKFjPRRFP2vQO1vvZ/eNWBQcHv29vpje+bcaZkurfh9o6Rui0E='} 2024-03-06T07:31:50.092 DEBUG:teuthology.nuke:shortname: smithi145 2024-03-06T07:31:50.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:31:50.115 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:00:27-fs-wip-lusov-qdb-distro-default-smithi/7583042', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:01:53.124550', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLs9x8j8hoye4v7cxEJMQ6KkZsfY8EpCDIT4JcwJqJWa2nvNoegFCUiOVIwOGvganVWn+4aJUZjR+AISPQXVEc4='} 2024-03-06T07:31:50.143 INFO:teuthology.orchestra.console.smithi094:Power off 2024-03-06T07:31:50.143 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:31:50.172 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-03-06T07:31:50.173 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:31:50.188 INFO:teuthology.orchestra.console.smithi145:Power off 2024-03-06T07:31:50.189 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:31:50.199 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-06T07:31:50.213 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-03-06T07:31:50.214 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:31:50.235 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-06T07:31:54.200 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:31:54.237 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:31:54.311 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-06T07:31:54.345 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-06T07:31:58.315 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:31:58.346 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:31:58.427 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-03-06T07:31:58.456 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-06T07:31:58.528 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-03-06T07:32:02.458 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:32:02.570 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-06T07:32:02.670 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-03-06T07:32:02.775 INFO:teuthology.run:Summary data: description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} duration: 1267.16805434227 failure_reason: 'Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=41b432f0aa724dafa3ad74d039e80be1 status: fail success: false 2024-03-06T07:32:02.776 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:32:02.877 INFO:teuthology.run:FAIL